2024-03-11T20:26:28.610 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-11T20:26:28.610 DEBUG:teuthology.run:Teuthology command: teuthology --name teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi --owner scheduled_teuthology@teuthology --description orch/cephadm/thrash/{0-distro/ubuntu_22.04 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async root} --archive /home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591745 --verbose -- /home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591745/orig.config.yaml 2024-03-11T20:26:28.643 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-11T20:26:28.721 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591745 branch: main description: orch/cephadm/thrash/{0-distro/ubuntu_22.04 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async root} email: ceph-qa@ceph.com first_in_suite: false job_id: '7591745' kernel: kdb: true sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: main ceph: conf: client.0: debug ms: 1 debug objecter: 20 debug rados: 20 global: ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 paxos service trim min: 10 osd: debug ms: 1 debug osd: 20 osd debug reject backfill probability: 0.3 osd delete sleep: 1 osd max backfills: 3 osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true osd snap trim sleep: 2 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - but it is still running - objects unfound and apparently lost - \(POOL_APP_NOT_ENABLED\) sha1: 2b0101eb74d60610d544cb8a473e23a35ac1cc0b ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} cephadm: cephadm_mode: root install: ceph: flavor: default sha1: 2b0101eb74d60610d544cb8a473e23a35ac1cc0b workunit: branch: main sha1: 1c349451176cc5b4ebfb24b22eaaa754e05cff6c owner: scheduled_teuthology@teuthology priority: 200 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - ceph.rgw.foo.a - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b - ceph.iscsi.iscsi.a seed: 6815 sha1: 2b0101eb74d60610d544cb8a473e23a35ac1cc0b sleep_before_teardown: 0 subset: 10/64 suite: orch suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph_1c349451176cc5b4ebfb24b22eaaa754e05cff6c/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 1c349451176cc5b4ebfb24b22eaaa754e05cff6c targets: smithi023.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJktKkTPugdekffXcRoSm8ht+LzmjuwT01HtyylK2Lkt4rGHd5AhDfOFO5u8Qxy8J+yrw66+sXAXaphPkzYzA2o= smithi146.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLU0075hNvzK2/URwJNjADYTv8KNoaw+s3EBLHJ9uB5xFJFgRPGgtjwlpMbn7H+EVSEXyoMcqEK+wNcEL+QGLXo= tasks: - install: null - cephadm: conf: mgr: debug mgr: 20 debug ms: 1 - thrashosds: chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 timeout: 1200 - full_sequential: - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-11_20:08:13 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-11T20:26:28.721 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_1c349451176cc5b4ebfb24b22eaaa754e05cff6c/qa; will attempt to use it 2024-03-11T20:26:28.722 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_1c349451176cc5b4ebfb24b22eaaa754e05cff6c/qa/tasks 2024-03-11T20:26:28.722 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-11T20:26:28.723 INFO:teuthology.task.internal:Checking packages... 2024-03-11T20:26:28.747 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '2b0101eb74d60610d544cb8a473e23a35ac1cc0b' 2024-03-11T20:26:28.747 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-11T20:26:28.747 INFO:teuthology.packaging:ref: None 2024-03-11T20:26:28.747 INFO:teuthology.packaging:tag: None 2024-03-11T20:26:28.747 INFO:teuthology.packaging:branch: main 2024-03-11T20:26:28.747 INFO:teuthology.packaging:sha1: 2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-11T20:26:28.748 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=main 2024-03-11T20:26:29.236 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1979-g2b0101eb-1jammy 2024-03-11T20:26:29.237 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-11T20:26:29.243 INFO:teuthology.task.internal:no buildpackages task found 2024-03-11T20:26:29.243 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-11T20:26:29.250 INFO:teuthology.task.internal:Saving configuration 2024-03-11T20:26:29.265 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-11T20:26:29.270 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-11T20:26:29.296 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi023.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591745', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-11 20:19:40.036957', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAILT8+hdGrdVJLpNe5j2IUTsy+PPwm2ZhGNwjRCOj3slm'} 2024-03-11T20:26:29.318 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi146.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591745', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-11 20:19:40.038821', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIMFcbv0qT6g1BgYQt32KKiyrkGKiZKqPmzovxwC4vkmf'} 2024-03-11T20:26:29.318 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-11T20:26:29.324 INFO:teuthology.task.internal:roles: ubuntu@smithi023.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'ceph.rgw.foo.a', 'node-exporter.a', 'alertmanager.a'] 2024-03-11T20:26:29.324 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b', 'ceph.iscsi.iscsi.a'] 2024-03-11T20:26:29.324 INFO:teuthology.run_tasks:Running task console_log... 2024-03-11T20:26:29.398 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f82067023a0>, signals=[15]) 2024-03-11T20:26:29.399 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-11T20:26:29.407 INFO:teuthology.task.internal:Opening connections... 2024-03-11T20:26:29.407 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi023.front.sepia.ceph.com 2024-03-11T20:26:29.409 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi023.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T20:26:29.493 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi146.front.sepia.ceph.com 2024-03-11T20:26:29.495 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T20:26:29.577 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-11T20:26:29.584 DEBUG:teuthology.orchestra.run.smithi023:> uname -m 2024-03-11T20:26:29.589 INFO:teuthology.orchestra.run.smithi023.stdout:x86_64 2024-03-11T20:26:29.590 DEBUG:teuthology.orchestra.run.smithi023:> cat /etc/os-release 2024-03-11T20:26:29.637 INFO:teuthology.orchestra.run.smithi023.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-03-11T20:26:29.637 INFO:teuthology.orchestra.run.smithi023.stdout:NAME="Ubuntu" 2024-03-11T20:26:29.637 INFO:teuthology.orchestra.run.smithi023.stdout:VERSION_ID="22.04" 2024-03-11T20:26:29.637 INFO:teuthology.orchestra.run.smithi023.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-03-11T20:26:29.637 INFO:teuthology.orchestra.run.smithi023.stdout:VERSION_CODENAME=jammy 2024-03-11T20:26:29.637 INFO:teuthology.orchestra.run.smithi023.stdout:ID=ubuntu 2024-03-11T20:26:29.638 INFO:teuthology.orchestra.run.smithi023.stdout:ID_LIKE=debian 2024-03-11T20:26:29.638 INFO:teuthology.orchestra.run.smithi023.stdout:HOME_URL="https://www.ubuntu.com/" 2024-03-11T20:26:29.638 INFO:teuthology.orchestra.run.smithi023.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-03-11T20:26:29.638 INFO:teuthology.orchestra.run.smithi023.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-03-11T20:26:29.638 INFO:teuthology.orchestra.run.smithi023.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-03-11T20:26:29.638 INFO:teuthology.orchestra.run.smithi023.stdout:UBUNTU_CODENAME=jammy 2024-03-11T20:26:29.638 INFO:teuthology.lock.ops:Updating smithi023.front.sepia.ceph.com on lock server 2024-03-11T20:26:29.656 DEBUG:teuthology.orchestra.run.smithi146:> uname -m 2024-03-11T20:26:29.662 INFO:teuthology.orchestra.run.smithi146.stdout:x86_64 2024-03-11T20:26:29.662 DEBUG:teuthology.orchestra.run.smithi146:> cat /etc/os-release 2024-03-11T20:26:29.707 INFO:teuthology.orchestra.run.smithi146.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-03-11T20:26:29.708 INFO:teuthology.orchestra.run.smithi146.stdout:NAME="Ubuntu" 2024-03-11T20:26:29.708 INFO:teuthology.orchestra.run.smithi146.stdout:VERSION_ID="22.04" 2024-03-11T20:26:29.708 INFO:teuthology.orchestra.run.smithi146.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-03-11T20:26:29.708 INFO:teuthology.orchestra.run.smithi146.stdout:VERSION_CODENAME=jammy 2024-03-11T20:26:29.708 INFO:teuthology.orchestra.run.smithi146.stdout:ID=ubuntu 2024-03-11T20:26:29.708 INFO:teuthology.orchestra.run.smithi146.stdout:ID_LIKE=debian 2024-03-11T20:26:29.708 INFO:teuthology.orchestra.run.smithi146.stdout:HOME_URL="https://www.ubuntu.com/" 2024-03-11T20:26:29.708 INFO:teuthology.orchestra.run.smithi146.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-03-11T20:26:29.708 INFO:teuthology.orchestra.run.smithi146.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-03-11T20:26:29.708 INFO:teuthology.orchestra.run.smithi146.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-03-11T20:26:29.708 INFO:teuthology.orchestra.run.smithi146.stdout:UBUNTU_CODENAME=jammy 2024-03-11T20:26:29.709 INFO:teuthology.lock.ops:Updating smithi146.front.sepia.ceph.com on lock server 2024-03-11T20:26:29.731 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-11T20:26:29.740 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-11T20:26:29.746 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-11T20:26:29.747 DEBUG:teuthology.orchestra.run.smithi023:> test '!' -e /home/ubuntu/cephtest 2024-03-11T20:26:29.749 DEBUG:teuthology.orchestra.run.smithi146:> test '!' -e /home/ubuntu/cephtest 2024-03-11T20:26:29.755 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-11T20:26:29.761 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-11T20:26:29.762 DEBUG:teuthology.orchestra.run.smithi023:> test -z $(ls -A /var/lib/ceph) 2024-03-11T20:26:29.794 DEBUG:teuthology.orchestra.run.smithi146:> test -z $(ls -A /var/lib/ceph) 2024-03-11T20:26:30.049 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-11T20:26:30.104 INFO:teuthology.run_tasks:Running task kernel... 2024-03-11T20:26:30.132 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-11T20:26:30.132 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-11T20:26:30.132 DEBUG:teuthology.orchestra.run.smithi023:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-11T20:26:30.133 DEBUG:teuthology.orchestra.run.smithi146:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-11T20:26:30.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:26:30.138 DEBUG:teuthology.orchestra.run.smithi023:> uname -r 2024-03-11T20:26:30.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:26:30.139 DEBUG:teuthology.orchestra.run.smithi146:> uname -r 2024-03-11T20:26:30.185 INFO:teuthology.orchestra.run.smithi146.stdout:5.15.0-56-generic 2024-03-11T20:26:30.185 INFO:teuthology.task.kernel:Running kernel on smithi146: 5.15.0-56-generic 2024-03-11T20:26:30.185 DEBUG:teuthology.orchestra.run.smithi146:> sudo apt-get clean 2024-03-11T20:26:30.191 INFO:teuthology.orchestra.run.smithi023.stdout:5.15.0-56-generic 2024-03-11T20:26:30.191 INFO:teuthology.task.kernel:Running kernel on smithi023: 5.15.0-56-generic 2024-03-11T20:26:30.191 DEBUG:teuthology.orchestra.run.smithi023:> sudo apt-get clean 2024-03-11T20:26:30.310 DEBUG:teuthology.orchestra.run.smithi146:> sudo apt-get update 2024-03-11T20:26:30.342 DEBUG:teuthology.orchestra.run.smithi023:> sudo apt-get update 2024-03-11T20:26:30.551 INFO:teuthology.orchestra.run.smithi023.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-03-11T20:26:30.598 INFO:teuthology.orchestra.run.smithi023.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-03-11T20:26:30.615 INFO:teuthology.orchestra.run.smithi146.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-03-11T20:26:30.615 INFO:teuthology.orchestra.run.smithi146.stdout:Get:2 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-03-11T20:26:30.637 INFO:teuthology.orchestra.run.smithi146.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-03-11T20:26:30.638 INFO:teuthology.orchestra.run.smithi023.stdout:Get:3 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-03-11T20:26:30.807 INFO:teuthology.orchestra.run.smithi023.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2024-03-11T20:26:30.884 INFO:teuthology.orchestra.run.smithi146.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,241 kB] 2024-03-11T20:26:30.901 INFO:teuthology.orchestra.run.smithi023.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,460 kB] 2024-03-11T20:26:31.004 INFO:teuthology.orchestra.run.smithi146.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2024-03-11T20:26:31.103 INFO:teuthology.orchestra.run.smithi023.stdout:Get:6 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [419 kB] 2024-03-11T20:26:31.138 INFO:teuthology.orchestra.run.smithi146.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,460 kB] 2024-03-11T20:26:31.140 INFO:teuthology.orchestra.run.smithi146.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [419 kB] 2024-03-11T20:26:31.152 INFO:teuthology.orchestra.run.smithi023.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [585 kB] 2024-03-11T20:26:31.155 INFO:teuthology.orchestra.run.smithi146.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [223 kB] 2024-03-11T20:26:31.167 INFO:teuthology.orchestra.run.smithi146.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-03-11T20:26:31.167 INFO:teuthology.orchestra.run.smithi146.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,526 kB] 2024-03-11T20:26:31.178 INFO:teuthology.orchestra.run.smithi023.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [283 kB] 2024-03-11T20:26:31.189 INFO:teuthology.orchestra.run.smithi023.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-03-11T20:26:31.190 INFO:teuthology.orchestra.run.smithi023.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,559 kB] 2024-03-11T20:26:31.296 INFO:teuthology.orchestra.run.smithi146.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [33.7 kB] 2024-03-11T20:26:31.299 INFO:teuthology.orchestra.run.smithi146.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [253 kB] 2024-03-11T20:26:31.301 INFO:teuthology.orchestra.run.smithi023.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [35.3 kB] 2024-03-11T20:26:31.302 INFO:teuthology.orchestra.run.smithi023.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [259 kB] 2024-03-11T20:26:31.304 INFO:teuthology.orchestra.run.smithi146.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-03-11T20:26:31.304 INFO:teuthology.orchestra.run.smithi146.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [597 kB] 2024-03-11T20:26:31.309 INFO:teuthology.orchestra.run.smithi023.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-03-11T20:26:31.309 INFO:teuthology.orchestra.run.smithi023.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,054 kB] 2024-03-11T20:26:31.315 INFO:teuthology.orchestra.run.smithi146.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [848 kB] 2024-03-11T20:26:31.323 INFO:teuthology.orchestra.run.smithi146.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [162 kB] 2024-03-11T20:26:31.329 INFO:teuthology.orchestra.run.smithi023.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [693 kB] 2024-03-11T20:26:31.340 INFO:teuthology.orchestra.run.smithi023.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [238 kB] 2024-03-11T20:26:31.343 INFO:teuthology.orchestra.run.smithi023.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-03-11T20:26:31.344 INFO:teuthology.orchestra.run.smithi023.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,160 B] 2024-03-11T20:26:31.350 INFO:teuthology.orchestra.run.smithi146.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-03-11T20:26:31.350 INFO:teuthology.orchestra.run.smithi146.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-03-11T20:26:31.350 INFO:teuthology.orchestra.run.smithi146.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.1 kB] 2024-03-11T20:26:31.363 INFO:teuthology.orchestra.run.smithi023.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [42.1 kB] 2024-03-11T20:26:31.363 INFO:teuthology.orchestra.run.smithi023.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.1 kB] 2024-03-11T20:26:31.371 INFO:teuthology.orchestra.run.smithi023.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-03-11T20:26:31.372 INFO:teuthology.orchestra.run.smithi023.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [33.9 kB] 2024-03-11T20:26:31.372 INFO:teuthology.orchestra.run.smithi023.stdout:Get:23 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,241 kB] 2024-03-11T20:26:31.372 INFO:teuthology.orchestra.run.smithi023.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [41.7 kB] 2024-03-11T20:26:31.373 INFO:teuthology.orchestra.run.smithi023.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [10.5 kB] 2024-03-11T20:26:31.403 INFO:teuthology.orchestra.run.smithi146.stdout:Get:20 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,476 B] 2024-03-11T20:26:31.403 INFO:teuthology.orchestra.run.smithi146.stdout:Get:21 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-03-11T20:26:31.403 INFO:teuthology.orchestra.run.smithi023.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-03-11T20:26:31.404 INFO:teuthology.orchestra.run.smithi023.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [24.3 kB] 2024-03-11T20:26:31.405 INFO:teuthology.orchestra.run.smithi023.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [13.4 kB] 2024-03-11T20:26:31.405 INFO:teuthology.orchestra.run.smithi023.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB] 2024-03-11T20:26:31.406 INFO:teuthology.orchestra.run.smithi023.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-03-11T20:26:31.438 INFO:teuthology.orchestra.run.smithi146.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [585 kB] 2024-03-11T20:26:31.477 INFO:teuthology.orchestra.run.smithi146.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [283 kB] 2024-03-11T20:26:31.492 INFO:teuthology.orchestra.run.smithi146.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-03-11T20:26:31.493 INFO:teuthology.orchestra.run.smithi146.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,559 kB] 2024-03-11T20:26:31.537 INFO:teuthology.orchestra.run.smithi023.stdout:Get:31 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [223 kB] 2024-03-11T20:26:31.549 INFO:teuthology.orchestra.run.smithi023.stdout:Get:32 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-03-11T20:26:31.550 INFO:teuthology.orchestra.run.smithi023.stdout:Get:33 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,526 kB] 2024-03-11T20:26:31.552 INFO:teuthology.orchestra.run.smithi146.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [35.3 kB] 2024-03-11T20:26:31.553 INFO:teuthology.orchestra.run.smithi146.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [259 kB] 2024-03-11T20:26:31.559 INFO:teuthology.orchestra.run.smithi146.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-03-11T20:26:31.629 INFO:teuthology.orchestra.run.smithi023.stdout:Get:34 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [33.7 kB] 2024-03-11T20:26:31.630 INFO:teuthology.orchestra.run.smithi023.stdout:Get:35 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [253 kB] 2024-03-11T20:26:31.638 INFO:teuthology.orchestra.run.smithi023.stdout:Get:36 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-03-11T20:26:31.639 INFO:teuthology.orchestra.run.smithi023.stdout:Get:37 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [848 kB] 2024-03-11T20:26:31.665 INFO:teuthology.orchestra.run.smithi023.stdout:Get:38 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [597 kB] 2024-03-11T20:26:31.723 INFO:teuthology.orchestra.run.smithi146.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [693 kB] 2024-03-11T20:26:31.842 INFO:teuthology.orchestra.run.smithi023.stdout:Get:39 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [162 kB] 2024-03-11T20:26:32.245 INFO:teuthology.orchestra.run.smithi023.stdout:Get:40 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-03-11T20:26:32.249 INFO:teuthology.orchestra.run.smithi023.stdout:Get:41 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.1 kB] 2024-03-11T20:26:32.263 INFO:teuthology.orchestra.run.smithi023.stdout:Get:42 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-03-11T20:26:32.265 INFO:teuthology.orchestra.run.smithi023.stdout:Get:43 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,476 B] 2024-03-11T20:26:32.266 INFO:teuthology.orchestra.run.smithi023.stdout:Get:44 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-03-11T20:26:32.307 INFO:teuthology.orchestra.run.smithi146.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,054 kB] 2024-03-11T20:26:32.418 INFO:teuthology.orchestra.run.smithi146.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [238 kB] 2024-03-11T20:26:32.438 INFO:teuthology.orchestra.run.smithi146.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-03-11T20:26:32.440 INFO:teuthology.orchestra.run.smithi146.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [42.1 kB] 2024-03-11T20:26:32.443 INFO:teuthology.orchestra.run.smithi146.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,160 B] 2024-03-11T20:26:32.443 INFO:teuthology.orchestra.run.smithi146.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.1 kB] 2024-03-11T20:26:32.444 INFO:teuthology.orchestra.run.smithi146.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-03-11T20:26:32.444 INFO:teuthology.orchestra.run.smithi146.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [41.7 kB] 2024-03-11T20:26:32.447 INFO:teuthology.orchestra.run.smithi146.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [33.9 kB] 2024-03-11T20:26:32.449 INFO:teuthology.orchestra.run.smithi146.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [10.5 kB] 2024-03-11T20:26:32.615 INFO:teuthology.orchestra.run.smithi146.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-03-11T20:26:32.697 INFO:teuthology.orchestra.run.smithi146.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [24.3 kB] 2024-03-11T20:26:32.798 INFO:teuthology.orchestra.run.smithi146.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [13.4 kB] 2024-03-11T20:26:32.827 INFO:teuthology.orchestra.run.smithi146.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB] 2024-03-11T20:26:32.880 INFO:teuthology.orchestra.run.smithi146.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-03-11T20:26:37.102 INFO:teuthology.orchestra.run.smithi023.stdout:Fetched 12.1 MB in 2s (5,791 kB/s) 2024-03-11T20:26:37.240 INFO:teuthology.orchestra.run.smithi146.stdout:Fetched 12.1 MB in 3s (4,821 kB/s) 2024-03-11T20:26:38.449 INFO:teuthology.orchestra.run.smithi023.stdout:Reading package lists... 2024-03-11T20:26:38.467 DEBUG:teuthology.orchestra.run.smithi023:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-03-11T20:26:38.531 INFO:teuthology.orchestra.run.smithi023.stdout:Reading package lists... 2024-03-11T20:26:38.638 INFO:teuthology.orchestra.run.smithi146.stdout:Reading package lists... 2024-03-11T20:26:38.655 DEBUG:teuthology.orchestra.run.smithi146:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-03-11T20:26:38.713 INFO:teuthology.orchestra.run.smithi146.stdout:Reading package lists... 2024-03-11T20:26:38.717 INFO:teuthology.orchestra.run.smithi023.stdout:Building dependency tree... 2024-03-11T20:26:38.717 INFO:teuthology.orchestra.run.smithi023.stdout:Reading state information... 2024-03-11T20:26:38.876 INFO:teuthology.orchestra.run.smithi023.stdout:The following packages were automatically installed and are no longer required: 2024-03-11T20:26:38.876 INFO:teuthology.orchestra.run.smithi023.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-03-11T20:26:38.877 INFO:teuthology.orchestra.run.smithi023.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-03-11T20:26:38.877 INFO:teuthology.orchestra.run.smithi023.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-03-11T20:26:38.877 INFO:teuthology.orchestra.run.smithi023.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-03-11T20:26:38.877 INFO:teuthology.orchestra.run.smithi023.stdout: python2.7-minimal 2024-03-11T20:26:38.877 INFO:teuthology.orchestra.run.smithi023.stdout:Use 'sudo apt autoremove' to remove them. 2024-03-11T20:26:38.879 INFO:teuthology.orchestra.run.smithi023.stdout:The following additional packages will be installed: 2024-03-11T20:26:38.879 INFO:teuthology.orchestra.run.smithi023.stdout: linux-generic linux-headers-5.15.0-100 linux-headers-5.15.0-100-generic 2024-03-11T20:26:38.879 INFO:teuthology.orchestra.run.smithi023.stdout: linux-headers-generic linux-image-5.15.0-100-generic 2024-03-11T20:26:38.879 INFO:teuthology.orchestra.run.smithi023.stdout: linux-modules-5.15.0-100-generic linux-modules-extra-5.15.0-100-generic 2024-03-11T20:26:38.880 INFO:teuthology.orchestra.run.smithi023.stdout:Suggested packages: 2024-03-11T20:26:38.880 INFO:teuthology.orchestra.run.smithi023.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-03-11T20:26:38.880 INFO:teuthology.orchestra.run.smithi023.stdout:Recommended packages: 2024-03-11T20:26:38.880 INFO:teuthology.orchestra.run.smithi023.stdout: thermald 2024-03-11T20:26:38.906 INFO:teuthology.orchestra.run.smithi146.stdout:Building dependency tree... 2024-03-11T20:26:38.907 INFO:teuthology.orchestra.run.smithi146.stdout:Reading state information... 2024-03-11T20:26:38.923 INFO:teuthology.orchestra.run.smithi023.stdout:The following NEW packages will be installed: 2024-03-11T20:26:38.923 INFO:teuthology.orchestra.run.smithi023.stdout: linux-headers-5.15.0-100 linux-headers-5.15.0-100-generic 2024-03-11T20:26:38.923 INFO:teuthology.orchestra.run.smithi023.stdout: linux-image-5.15.0-100-generic linux-modules-5.15.0-100-generic 2024-03-11T20:26:38.923 INFO:teuthology.orchestra.run.smithi023.stdout: linux-modules-extra-5.15.0-100-generic 2024-03-11T20:26:38.924 INFO:teuthology.orchestra.run.smithi023.stdout:The following packages will be upgraded: 2024-03-11T20:26:38.924 INFO:teuthology.orchestra.run.smithi023.stdout: linux-generic linux-headers-generic linux-image-generic 2024-03-11T20:26:39.064 INFO:teuthology.orchestra.run.smithi146.stdout:The following packages were automatically installed and are no longer required: 2024-03-11T20:26:39.064 INFO:teuthology.orchestra.run.smithi146.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-03-11T20:26:39.064 INFO:teuthology.orchestra.run.smithi146.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-03-11T20:26:39.065 INFO:teuthology.orchestra.run.smithi146.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-03-11T20:26:39.065 INFO:teuthology.orchestra.run.smithi146.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-03-11T20:26:39.065 INFO:teuthology.orchestra.run.smithi146.stdout: python2.7-minimal 2024-03-11T20:26:39.065 INFO:teuthology.orchestra.run.smithi146.stdout:Use 'sudo apt autoremove' to remove them. 2024-03-11T20:26:39.066 INFO:teuthology.orchestra.run.smithi146.stdout:The following additional packages will be installed: 2024-03-11T20:26:39.066 INFO:teuthology.orchestra.run.smithi146.stdout: linux-generic linux-headers-5.15.0-100 linux-headers-5.15.0-100-generic 2024-03-11T20:26:39.066 INFO:teuthology.orchestra.run.smithi146.stdout: linux-headers-generic linux-image-5.15.0-100-generic 2024-03-11T20:26:39.066 INFO:teuthology.orchestra.run.smithi146.stdout: linux-modules-5.15.0-100-generic linux-modules-extra-5.15.0-100-generic 2024-03-11T20:26:39.067 INFO:teuthology.orchestra.run.smithi146.stdout:Suggested packages: 2024-03-11T20:26:39.068 INFO:teuthology.orchestra.run.smithi146.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-03-11T20:26:39.068 INFO:teuthology.orchestra.run.smithi146.stdout:Recommended packages: 2024-03-11T20:26:39.068 INFO:teuthology.orchestra.run.smithi146.stdout: thermald 2024-03-11T20:26:39.103 INFO:teuthology.orchestra.run.smithi023.stdout:3 upgraded, 5 newly installed, 0 to remove and 288 not upgraded. 2024-03-11T20:26:39.103 INFO:teuthology.orchestra.run.smithi023.stdout:Need to get 113 MB of archives. 2024-03-11T20:26:39.104 INFO:teuthology.orchestra.run.smithi023.stdout:After this operation, 582 MB of additional disk space will be used. 2024-03-11T20:26:39.104 INFO:teuthology.orchestra.run.smithi023.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-100-generic amd64 5.15.0-100.110 [22.6 MB] 2024-03-11T20:26:39.122 INFO:teuthology.orchestra.run.smithi146.stdout:The following NEW packages will be installed: 2024-03-11T20:26:39.122 INFO:teuthology.orchestra.run.smithi146.stdout: linux-headers-5.15.0-100 linux-headers-5.15.0-100-generic 2024-03-11T20:26:39.122 INFO:teuthology.orchestra.run.smithi146.stdout: linux-image-5.15.0-100-generic linux-modules-5.15.0-100-generic 2024-03-11T20:26:39.122 INFO:teuthology.orchestra.run.smithi146.stdout: linux-modules-extra-5.15.0-100-generic 2024-03-11T20:26:39.123 INFO:teuthology.orchestra.run.smithi146.stdout:The following packages will be upgraded: 2024-03-11T20:26:39.124 INFO:teuthology.orchestra.run.smithi146.stdout: linux-generic linux-headers-generic linux-image-generic 2024-03-11T20:26:39.220 INFO:teuthology.orchestra.run.smithi146.stdout:3 upgraded, 5 newly installed, 0 to remove and 288 not upgraded. 2024-03-11T20:26:39.220 INFO:teuthology.orchestra.run.smithi146.stdout:Need to get 113 MB of archives. 2024-03-11T20:26:39.221 INFO:teuthology.orchestra.run.smithi146.stdout:After this operation, 582 MB of additional disk space will be used. 2024-03-11T20:26:39.221 INFO:teuthology.orchestra.run.smithi146.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-100-generic amd64 5.15.0-100.110 [22.6 MB] 2024-03-11T20:26:40.137 INFO:teuthology.orchestra.run.smithi146.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-100-generic amd64 5.15.0-100.110 [11.5 MB] 2024-03-11T20:26:40.369 INFO:teuthology.orchestra.run.smithi146.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-100-generic amd64 5.15.0-100.110 [63.8 MB] 2024-03-11T20:26:40.509 INFO:teuthology.orchestra.run.smithi023.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-100-generic amd64 5.15.0-100.110 [11.5 MB] 2024-03-11T20:26:40.851 INFO:teuthology.orchestra.run.smithi023.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-100-generic amd64 5.15.0-100.110 [63.8 MB] 2024-03-11T20:26:42.145 INFO:teuthology.orchestra.run.smithi146.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.100.97 [1,696 B] 2024-03-11T20:26:42.146 INFO:teuthology.orchestra.run.smithi146.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.100.97 [2,454 B] 2024-03-11T20:26:42.146 INFO:teuthology.orchestra.run.smithi146.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-100 all 5.15.0-100.110 [12.3 MB] 2024-03-11T20:26:42.385 INFO:teuthology.orchestra.run.smithi146.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-100-generic amd64 5.15.0-100.110 [2,858 kB] 2024-03-11T20:26:42.441 INFO:teuthology.orchestra.run.smithi146.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.100.97 [2,292 B] 2024-03-11T20:26:42.828 INFO:teuthology.orchestra.run.smithi146.stdout:Fetched 113 MB in 3s (34.2 MB/s) 2024-03-11T20:26:42.972 INFO:teuthology.orchestra.run.smithi023.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.100.97 [1,696 B] 2024-03-11T20:26:42.972 INFO:teuthology.orchestra.run.smithi023.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.100.97 [2,454 B] 2024-03-11T20:26:42.973 INFO:teuthology.orchestra.run.smithi023.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-100 all 5.15.0-100.110 [12.3 MB] 2024-03-11T20:26:43.014 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package linux-modules-5.15.0-100-generic. 2024-03-11T20:26:43.641 INFO:teuthology.orchestra.run.smithi023.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-100-generic amd64 5.15.0-100.110 [2,858 kB] 2024-03-11T20:26:43.722 INFO:teuthology.orchestra.run.smithi023.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.100.97 [2,292 B] 2024-03-11T20:26:44.127 INFO:teuthology.orchestra.run.smithi023.stdout:Fetched 113 MB in 5s (23.6 MB/s) 2024-03-11T20:26:44.304 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package linux-modules-5.15.0-100-generic. 2024-03-11T20:26:44.955 INFO:teuthology.orchestra.run.smithi146.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-03-11T20:26:44.959 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../0-linux-modules-5.15.0-100-generic_5.15.0-100.110_amd64.deb ... 2024-03-11T20:26:45.035 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking linux-modules-5.15.0-100-generic (5.15.0-100.110) ... 2024-03-11T20:26:46.292 INFO:teuthology.orchestra.run.smithi023.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-03-11T20:26:46.296 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../0-linux-modules-5.15.0-100-generic_5.15.0-100.110_amd64.deb ... 2024-03-11T20:26:46.361 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking linux-modules-5.15.0-100-generic (5.15.0-100.110) ... 2024-03-11T20:26:47.021 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package linux-image-5.15.0-100-generic. 2024-03-11T20:26:47.043 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../1-linux-image-5.15.0-100-generic_5.15.0-100.110_amd64.deb ... 2024-03-11T20:26:47.133 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking linux-image-5.15.0-100-generic (5.15.0-100.110) ... 2024-03-11T20:26:47.774 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-100-generic. 2024-03-11T20:26:47.795 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-100-generic_5.15.0-100.110_amd64.deb ... 2024-03-11T20:26:47.833 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking linux-modules-extra-5.15.0-100-generic (5.15.0-100.110) ... 2024-03-11T20:26:48.654 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package linux-image-5.15.0-100-generic. 2024-03-11T20:26:48.676 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../1-linux-image-5.15.0-100-generic_5.15.0-100.110_amd64.deb ... 2024-03-11T20:26:48.769 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking linux-image-5.15.0-100-generic (5.15.0-100.110) ... 2024-03-11T20:26:49.207 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-100-generic. 2024-03-11T20:26:49.228 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-100-generic_5.15.0-100.110_amd64.deb ... 2024-03-11T20:26:49.257 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking linux-modules-extra-5.15.0-100-generic (5.15.0-100.110) ... 2024-03-11T20:26:54.338 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../3-linux-generic_5.15.0.100.97_amd64.deb ... 2024-03-11T20:26:54.465 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking linux-generic (5.15.0.100.97) over (5.15.0.56.54) ... 2024-03-11T20:26:54.764 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.100.97_amd64.deb ... 2024-03-11T20:26:54.860 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking linux-image-generic (5.15.0.100.97) over (5.15.0.56.54) ... 2024-03-11T20:26:55.146 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package linux-headers-5.15.0-100. 2024-03-11T20:26:55.168 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../5-linux-headers-5.15.0-100_5.15.0-100.110_all.deb ... 2024-03-11T20:26:55.212 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking linux-headers-5.15.0-100 (5.15.0-100.110) ... 2024-03-11T20:26:55.615 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../3-linux-generic_5.15.0.100.97_amd64.deb ... 2024-03-11T20:26:55.741 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking linux-generic (5.15.0.100.97) over (5.15.0.56.54) ... 2024-03-11T20:26:56.024 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.100.97_amd64.deb ... 2024-03-11T20:26:56.136 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking linux-image-generic (5.15.0.100.97) over (5.15.0.56.54) ... 2024-03-11T20:26:56.448 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package linux-headers-5.15.0-100. 2024-03-11T20:26:56.470 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../5-linux-headers-5.15.0-100_5.15.0-100.110_all.deb ... 2024-03-11T20:26:56.514 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking linux-headers-5.15.0-100 (5.15.0-100.110) ... 2024-03-11T20:27:00.873 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package linux-headers-5.15.0-100-generic. 2024-03-11T20:27:00.910 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../6-linux-headers-5.15.0-100-generic_5.15.0-100.110_amd64.deb ... 2024-03-11T20:27:00.948 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking linux-headers-5.15.0-100-generic (5.15.0-100.110) ... 2024-03-11T20:27:02.179 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package linux-headers-5.15.0-100-generic. 2024-03-11T20:27:02.216 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../6-linux-headers-5.15.0-100-generic_5.15.0-100.110_amd64.deb ... 2024-03-11T20:27:02.254 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking linux-headers-5.15.0-100-generic (5.15.0-100.110) ... 2024-03-11T20:27:03.476 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.100.97_amd64.deb ... 2024-03-11T20:27:03.600 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking linux-headers-generic (5.15.0.100.97) over (5.15.0.56.54) ... 2024-03-11T20:27:03.941 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up linux-headers-5.15.0-100 (5.15.0-100.110) ... 2024-03-11T20:27:04.062 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up linux-headers-5.15.0-100-generic (5.15.0-100.110) ... 2024-03-11T20:27:04.189 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up linux-headers-generic (5.15.0.100.97) ... 2024-03-11T20:27:04.318 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up linux-image-5.15.0-100-generic (5.15.0-100.110) ... 2024-03-11T20:27:04.964 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.100.97_amd64.deb ... 2024-03-11T20:27:05.089 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking linux-headers-generic (5.15.0.100.97) over (5.15.0.56.54) ... 2024-03-11T20:27:05.423 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up linux-headers-5.15.0-100 (5.15.0-100.110) ... 2024-03-11T20:27:05.544 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up linux-headers-5.15.0-100-generic (5.15.0-100.110) ... 2024-03-11T20:27:05.651 INFO:teuthology.orchestra.run.smithi146.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-03-11T20:27:05.651 INFO:teuthology.orchestra.run.smithi146.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-03-11T20:27:05.651 INFO:teuthology.orchestra.run.smithi146.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-100-generic 2024-03-11T20:27:05.651 INFO:teuthology.orchestra.run.smithi146.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-100-generic 2024-03-11T20:27:05.977 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up linux-modules-5.15.0-100-generic (5.15.0-100.110) ... 2024-03-11T20:27:06.446 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up linux-headers-generic (5.15.0.100.97) ... 2024-03-11T20:27:06.566 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up linux-image-5.15.0-100-generic (5.15.0-100.110) ... 2024-03-11T20:27:07.862 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up linux-modules-extra-5.15.0-100-generic (5.15.0-100.110) ... 2024-03-11T20:27:07.896 INFO:teuthology.orchestra.run.smithi023.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-03-11T20:27:07.896 INFO:teuthology.orchestra.run.smithi023.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-03-11T20:27:07.896 INFO:teuthology.orchestra.run.smithi023.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-100-generic 2024-03-11T20:27:07.896 INFO:teuthology.orchestra.run.smithi023.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-100-generic 2024-03-11T20:27:08.043 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up linux-modules-5.15.0-100-generic (5.15.0-100.110) ... 2024-03-11T20:27:09.272 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up linux-image-generic (5.15.0.100.97) ... 2024-03-11T20:27:09.372 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up linux-generic (5.15.0.100.97) ... 2024-03-11T20:27:09.477 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up linux-modules-extra-5.15.0-100-generic (5.15.0-100.110) ... 2024-03-11T20:27:09.490 INFO:teuthology.orchestra.run.smithi146.stdout:Processing triggers for linux-image-5.15.0-100-generic (5.15.0-100.110) ... 2024-03-11T20:27:09.558 INFO:teuthology.orchestra.run.smithi146.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-03-11T20:27:09.558 INFO:teuthology.orchestra.run.smithi146.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-100-generic 2024-03-11T20:27:10.912 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up linux-image-generic (5.15.0.100.97) ... 2024-03-11T20:27:11.037 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up linux-generic (5.15.0.100.97) ... 2024-03-11T20:27:11.155 INFO:teuthology.orchestra.run.smithi023.stdout:Processing triggers for linux-image-5.15.0-100-generic (5.15.0-100.110) ... 2024-03-11T20:27:11.214 INFO:teuthology.orchestra.run.smithi023.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-03-11T20:27:11.214 INFO:teuthology.orchestra.run.smithi023.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-100-generic 2024-03-11T20:27:23.459 INFO:teuthology.orchestra.run.smithi146.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-03-11T20:27:23.459 INFO:teuthology.orchestra.run.smithi146.stdout:Sourcing file `/etc/default/grub' 2024-03-11T20:27:23.474 INFO:teuthology.orchestra.run.smithi146.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-03-11T20:27:23.487 INFO:teuthology.orchestra.run.smithi146.stdout:Generating grub configuration file ... 2024-03-11T20:27:23.754 INFO:teuthology.orchestra.run.smithi146.stdout:Found linux image: /boot/vmlinuz-5.15.0-100-generic 2024-03-11T20:27:23.769 INFO:teuthology.orchestra.run.smithi146.stdout:Found initrd image: /boot/initrd.img-5.15.0-100-generic 2024-03-11T20:27:24.081 INFO:teuthology.orchestra.run.smithi146.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-03-11T20:27:24.087 INFO:teuthology.orchestra.run.smithi146.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-03-11T20:27:24.156 INFO:teuthology.orchestra.run.smithi146.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-03-11T20:27:24.162 INFO:teuthology.orchestra.run.smithi146.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-03-11T20:27:24.323 INFO:teuthology.orchestra.run.smithi146.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-03-11T20:27:24.323 INFO:teuthology.orchestra.run.smithi146.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-03-11T20:27:24.323 INFO:teuthology.orchestra.run.smithi146.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-03-11T20:27:24.371 INFO:teuthology.orchestra.run.smithi146.stdout:done 2024-03-11T20:27:24.837 DEBUG:teuthology.orchestra.run.smithi146:> dpkg -s linux-image-generic 2024-03-11T20:27:24.854 INFO:teuthology.orchestra.run.smithi146.stdout:Package: linux-image-generic 2024-03-11T20:27:24.854 INFO:teuthology.orchestra.run.smithi146.stdout:Status: install ok installed 2024-03-11T20:27:24.854 INFO:teuthology.orchestra.run.smithi146.stdout:Priority: optional 2024-03-11T20:27:24.854 INFO:teuthology.orchestra.run.smithi146.stdout:Section: kernel 2024-03-11T20:27:24.854 INFO:teuthology.orchestra.run.smithi146.stdout:Installed-Size: 21 2024-03-11T20:27:24.854 INFO:teuthology.orchestra.run.smithi146.stdout:Maintainer: Ubuntu Kernel Team 2024-03-11T20:27:24.854 INFO:teuthology.orchestra.run.smithi146.stdout:Architecture: amd64 2024-03-11T20:27:24.854 INFO:teuthology.orchestra.run.smithi146.stdout:Source: linux-meta 2024-03-11T20:27:24.854 INFO:teuthology.orchestra.run.smithi146.stdout:Version: 5.15.0.100.97 2024-03-11T20:27:24.855 INFO:teuthology.orchestra.run.smithi146.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.2), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-100), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.2) 2024-03-11T20:27:24.855 INFO:teuthology.orchestra.run.smithi146.stdout:Depends: linux-image-5.15.0-100-generic, linux-modules-extra-5.15.0-100-generic, linux-firmware, intel-microcode, amd64-microcode 2024-03-11T20:27:24.855 INFO:teuthology.orchestra.run.smithi146.stdout:Recommends: thermald 2024-03-11T20:27:24.855 INFO:teuthology.orchestra.run.smithi146.stdout:Description: Generic Linux kernel image 2024-03-11T20:27:24.855 INFO:teuthology.orchestra.run.smithi146.stdout: This package will always depend on the latest generic kernel image 2024-03-11T20:27:24.855 INFO:teuthology.orchestra.run.smithi146.stdout: available. 2024-03-11T20:27:24.855 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-100-generic 2024-03-11T20:27:24.855 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-11T20:27:24.855 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-11T20:27:24.856 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-11T20:27:24.856 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi146.front.sepia.ceph.com, path=None, version=distro) 2024-03-11T20:27:24.856 DEBUG:teuthology.orchestra.run.smithi146:> sudo apt-get clean 2024-03-11T20:27:25.006 DEBUG:teuthology.orchestra.run.smithi146:> sudo apt-get update 2024-03-11T20:27:25.147 INFO:teuthology.orchestra.run.smithi146.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-03-11T20:27:25.147 INFO:teuthology.orchestra.run.smithi146.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-03-11T20:27:25.167 INFO:teuthology.orchestra.run.smithi146.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-03-11T20:27:25.200 INFO:teuthology.orchestra.run.smithi146.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-03-11T20:27:25.458 INFO:teuthology.orchestra.run.smithi023.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-03-11T20:27:25.458 INFO:teuthology.orchestra.run.smithi023.stdout:Sourcing file `/etc/default/grub' 2024-03-11T20:27:25.479 INFO:teuthology.orchestra.run.smithi023.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-03-11T20:27:25.495 INFO:teuthology.orchestra.run.smithi023.stdout:Generating grub configuration file ... 2024-03-11T20:27:25.736 INFO:teuthology.orchestra.run.smithi023.stdout:Found linux image: /boot/vmlinuz-5.15.0-100-generic 2024-03-11T20:27:25.768 INFO:teuthology.orchestra.run.smithi023.stdout:Found initrd image: /boot/initrd.img-5.15.0-100-generic 2024-03-11T20:27:26.059 INFO:teuthology.orchestra.run.smithi023.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-03-11T20:27:26.065 INFO:teuthology.orchestra.run.smithi023.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-03-11T20:27:26.131 INFO:teuthology.orchestra.run.smithi023.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-03-11T20:27:26.137 INFO:teuthology.orchestra.run.smithi023.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-03-11T20:27:26.323 INFO:teuthology.orchestra.run.smithi023.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-03-11T20:27:26.323 INFO:teuthology.orchestra.run.smithi023.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-03-11T20:27:26.323 INFO:teuthology.orchestra.run.smithi023.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-03-11T20:27:26.372 INFO:teuthology.orchestra.run.smithi023.stdout:done 2024-03-11T20:27:26.501 INFO:teuthology.orchestra.run.smithi146.stdout:Reading package lists... 2024-03-11T20:27:26.519 DEBUG:teuthology.orchestra.run.smithi146:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-03-11T20:27:26.570 INFO:teuthology.orchestra.run.smithi146.stdout:Reading package lists... 2024-03-11T20:27:26.752 INFO:teuthology.orchestra.run.smithi146.stdout:Building dependency tree... 2024-03-11T20:27:26.752 INFO:teuthology.orchestra.run.smithi146.stdout:Reading state information... 2024-03-11T20:27:26.843 DEBUG:teuthology.orchestra.run.smithi023:> dpkg -s linux-image-generic 2024-03-11T20:27:26.865 INFO:teuthology.orchestra.run.smithi023.stdout:Package: linux-image-generic 2024-03-11T20:27:26.865 INFO:teuthology.orchestra.run.smithi023.stdout:Status: install ok installed 2024-03-11T20:27:26.865 INFO:teuthology.orchestra.run.smithi023.stdout:Priority: optional 2024-03-11T20:27:26.865 INFO:teuthology.orchestra.run.smithi023.stdout:Section: kernel 2024-03-11T20:27:26.865 INFO:teuthology.orchestra.run.smithi023.stdout:Installed-Size: 21 2024-03-11T20:27:26.866 INFO:teuthology.orchestra.run.smithi023.stdout:Maintainer: Ubuntu Kernel Team 2024-03-11T20:27:26.866 INFO:teuthology.orchestra.run.smithi023.stdout:Architecture: amd64 2024-03-11T20:27:26.866 INFO:teuthology.orchestra.run.smithi023.stdout:Source: linux-meta 2024-03-11T20:27:26.866 INFO:teuthology.orchestra.run.smithi023.stdout:Version: 5.15.0.100.97 2024-03-11T20:27:26.866 INFO:teuthology.orchestra.run.smithi023.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.2), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-100), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.2) 2024-03-11T20:27:26.866 INFO:teuthology.orchestra.run.smithi023.stdout:Depends: linux-image-5.15.0-100-generic, linux-modules-extra-5.15.0-100-generic, linux-firmware, intel-microcode, amd64-microcode 2024-03-11T20:27:26.866 INFO:teuthology.orchestra.run.smithi023.stdout:Recommends: thermald 2024-03-11T20:27:26.866 INFO:teuthology.orchestra.run.smithi023.stdout:Description: Generic Linux kernel image 2024-03-11T20:27:26.866 INFO:teuthology.orchestra.run.smithi023.stdout: This package will always depend on the latest generic kernel image 2024-03-11T20:27:26.866 INFO:teuthology.orchestra.run.smithi023.stdout: available. 2024-03-11T20:27:26.866 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-100-generic 2024-03-11T20:27:26.866 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-11T20:27:26.866 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-11T20:27:26.867 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-11T20:27:26.867 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi023.front.sepia.ceph.com, path=None, version=distro) 2024-03-11T20:27:26.867 DEBUG:teuthology.orchestra.run.smithi023:> sudo apt-get clean 2024-03-11T20:27:26.910 INFO:teuthology.orchestra.run.smithi146.stdout:linux-image-generic is already the newest version (5.15.0.100.97). 2024-03-11T20:27:26.910 INFO:teuthology.orchestra.run.smithi146.stdout:The following packages were automatically installed and are no longer required: 2024-03-11T20:27:26.910 INFO:teuthology.orchestra.run.smithi146.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-03-11T20:27:26.910 INFO:teuthology.orchestra.run.smithi146.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-03-11T20:27:26.910 INFO:teuthology.orchestra.run.smithi146.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-03-11T20:27:26.911 INFO:teuthology.orchestra.run.smithi146.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-03-11T20:27:26.911 INFO:teuthology.orchestra.run.smithi146.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-03-11T20:27:26.911 INFO:teuthology.orchestra.run.smithi146.stdout:Use 'sudo apt autoremove' to remove them. 2024-03-11T20:27:26.949 INFO:teuthology.orchestra.run.smithi146.stdout:0 upgraded, 0 newly installed, 0 to remove and 288 not upgraded. 2024-03-11T20:27:26.951 DEBUG:teuthology.orchestra.run.smithi146:> dpkg -s linux-image-generic 2024-03-11T20:27:27.005 DEBUG:teuthology.orchestra.run.smithi023:> sudo apt-get update 2024-03-11T20:27:27.010 INFO:teuthology.orchestra.run.smithi146.stdout:Package: linux-image-generic 2024-03-11T20:27:27.010 INFO:teuthology.orchestra.run.smithi146.stdout:Status: install ok installed 2024-03-11T20:27:27.010 INFO:teuthology.orchestra.run.smithi146.stdout:Priority: optional 2024-03-11T20:27:27.010 INFO:teuthology.orchestra.run.smithi146.stdout:Section: kernel 2024-03-11T20:27:27.010 INFO:teuthology.orchestra.run.smithi146.stdout:Installed-Size: 21 2024-03-11T20:27:27.010 INFO:teuthology.orchestra.run.smithi146.stdout:Maintainer: Ubuntu Kernel Team 2024-03-11T20:27:27.010 INFO:teuthology.orchestra.run.smithi146.stdout:Architecture: amd64 2024-03-11T20:27:27.010 INFO:teuthology.orchestra.run.smithi146.stdout:Source: linux-meta 2024-03-11T20:27:27.011 INFO:teuthology.orchestra.run.smithi146.stdout:Version: 5.15.0.100.97 2024-03-11T20:27:27.011 INFO:teuthology.orchestra.run.smithi146.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.2), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-100), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.2) 2024-03-11T20:27:27.011 INFO:teuthology.orchestra.run.smithi146.stdout:Depends: linux-image-5.15.0-100-generic, linux-modules-extra-5.15.0-100-generic, linux-firmware, intel-microcode, amd64-microcode 2024-03-11T20:27:27.011 INFO:teuthology.orchestra.run.smithi146.stdout:Recommends: thermald 2024-03-11T20:27:27.011 INFO:teuthology.orchestra.run.smithi146.stdout:Description: Generic Linux kernel image 2024-03-11T20:27:27.011 INFO:teuthology.orchestra.run.smithi146.stdout: This package will always depend on the latest generic kernel image 2024-03-11T20:27:27.011 INFO:teuthology.orchestra.run.smithi146.stdout: available. 2024-03-11T20:27:27.011 DEBUG:teuthology.orchestra.run.smithi146:> mktemp 2024-03-11T20:27:27.055 INFO:teuthology.orchestra.run.smithi146.stdout:/tmp/tmp.YcoM5zSujr 2024-03-11T20:27:27.056 DEBUG:teuthology.orchestra.run.smithi146:> sudo cp /boot/grub/grub.cfg /tmp/tmp.YcoM5zSujr 2024-03-11T20:27:27.111 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod 0666 /tmp/tmp.YcoM5zSujr 2024-03-11T20:27:27.176 INFO:teuthology.orchestra.run.smithi023.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-03-11T20:27:27.199 INFO:teuthology.orchestra.run.smithi023.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-03-11T20:27:27.228 DEBUG:teuthology.orchestra.remote:smithi146:/tmp/tmp.YcoM5zSujr is 10KB 2024-03-11T20:27:27.236 INFO:teuthology.orchestra.run.smithi023.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-03-11T20:27:27.241 DEBUG:teuthology.orchestra.run.smithi146:> rm -fr /tmp/tmp.YcoM5zSujr 2024-03-11T20:27:27.246 DEBUG:teuthology.orchestra.run.smithi146:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-03-11T20:27:27.248 INFO:teuthology.orchestra.run.smithi023.stdout:Hit:4 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-03-11T20:27:27.299 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-11T20:27:27.299 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-03-11T20:27:27.299 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-03-11T20:27:27.367 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-100-generic 2024-03-11T20:27:27.368 DEBUG:teuthology.orchestra.run.smithi146:> sudo update-grub 2024-03-11T20:27:28.544 INFO:teuthology.orchestra.run.smithi146.stderr:Sourcing file `/etc/default/grub' 2024-03-11T20:27:28.545 INFO:teuthology.orchestra.run.smithi146.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-03-11T20:27:28.547 INFO:teuthology.orchestra.run.smithi146.stderr:Generating grub configuration file ... 2024-03-11T20:27:28.563 INFO:teuthology.orchestra.run.smithi023.stdout:Reading package lists... 2024-03-11T20:27:28.581 DEBUG:teuthology.orchestra.run.smithi023:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-03-11T20:27:28.634 INFO:teuthology.orchestra.run.smithi023.stdout:Reading package lists... 2024-03-11T20:27:28.719 INFO:teuthology.orchestra.run.smithi146.stderr:Found linux image: /boot/vmlinuz-5.15.0-100-generic 2024-03-11T20:27:28.726 INFO:teuthology.orchestra.run.smithi146.stderr:Found initrd image: /boot/initrd.img-5.15.0-100-generic 2024-03-11T20:27:28.809 INFO:teuthology.orchestra.run.smithi023.stdout:Building dependency tree... 2024-03-11T20:27:28.810 INFO:teuthology.orchestra.run.smithi023.stdout:Reading state information... 2024-03-11T20:27:28.967 INFO:teuthology.orchestra.run.smithi023.stdout:linux-image-generic is already the newest version (5.15.0.100.97). 2024-03-11T20:27:28.967 INFO:teuthology.orchestra.run.smithi023.stdout:The following packages were automatically installed and are no longer required: 2024-03-11T20:27:28.967 INFO:teuthology.orchestra.run.smithi023.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-03-11T20:27:28.968 INFO:teuthology.orchestra.run.smithi023.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-03-11T20:27:28.968 INFO:teuthology.orchestra.run.smithi023.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-03-11T20:27:28.968 INFO:teuthology.orchestra.run.smithi023.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-03-11T20:27:28.968 INFO:teuthology.orchestra.run.smithi023.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-03-11T20:27:28.968 INFO:teuthology.orchestra.run.smithi023.stdout:Use 'sudo apt autoremove' to remove them. 2024-03-11T20:27:28.973 INFO:teuthology.orchestra.run.smithi146.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-03-11T20:27:28.978 INFO:teuthology.orchestra.run.smithi146.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-03-11T20:27:29.007 INFO:teuthology.orchestra.run.smithi023.stdout:0 upgraded, 0 newly installed, 0 to remove and 288 not upgraded. 2024-03-11T20:27:29.009 DEBUG:teuthology.orchestra.run.smithi023:> dpkg -s linux-image-generic 2024-03-11T20:27:29.023 INFO:teuthology.orchestra.run.smithi146.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-03-11T20:27:29.029 INFO:teuthology.orchestra.run.smithi146.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-03-11T20:27:29.067 INFO:teuthology.orchestra.run.smithi023.stdout:Package: linux-image-generic 2024-03-11T20:27:29.068 INFO:teuthology.orchestra.run.smithi023.stdout:Status: install ok installed 2024-03-11T20:27:29.068 INFO:teuthology.orchestra.run.smithi023.stdout:Priority: optional 2024-03-11T20:27:29.068 INFO:teuthology.orchestra.run.smithi023.stdout:Section: kernel 2024-03-11T20:27:29.068 INFO:teuthology.orchestra.run.smithi023.stdout:Installed-Size: 21 2024-03-11T20:27:29.068 INFO:teuthology.orchestra.run.smithi023.stdout:Maintainer: Ubuntu Kernel Team 2024-03-11T20:27:29.068 INFO:teuthology.orchestra.run.smithi023.stdout:Architecture: amd64 2024-03-11T20:27:29.068 INFO:teuthology.orchestra.run.smithi023.stdout:Source: linux-meta 2024-03-11T20:27:29.068 INFO:teuthology.orchestra.run.smithi023.stdout:Version: 5.15.0.100.97 2024-03-11T20:27:29.068 INFO:teuthology.orchestra.run.smithi023.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.2), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-100), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.2) 2024-03-11T20:27:29.068 INFO:teuthology.orchestra.run.smithi023.stdout:Depends: linux-image-5.15.0-100-generic, linux-modules-extra-5.15.0-100-generic, linux-firmware, intel-microcode, amd64-microcode 2024-03-11T20:27:29.068 INFO:teuthology.orchestra.run.smithi023.stdout:Recommends: thermald 2024-03-11T20:27:29.068 INFO:teuthology.orchestra.run.smithi023.stdout:Description: Generic Linux kernel image 2024-03-11T20:27:29.068 INFO:teuthology.orchestra.run.smithi023.stdout: This package will always depend on the latest generic kernel image 2024-03-11T20:27:29.068 INFO:teuthology.orchestra.run.smithi023.stdout: available. 2024-03-11T20:27:29.069 DEBUG:teuthology.orchestra.run.smithi023:> mktemp 2024-03-11T20:27:29.116 INFO:teuthology.orchestra.run.smithi023.stdout:/tmp/tmp.3rBPpKB5qT 2024-03-11T20:27:29.116 DEBUG:teuthology.orchestra.run.smithi023:> sudo cp /boot/grub/grub.cfg /tmp/tmp.3rBPpKB5qT 2024-03-11T20:27:29.153 INFO:teuthology.orchestra.run.smithi146.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-03-11T20:27:29.153 INFO:teuthology.orchestra.run.smithi146.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-03-11T20:27:29.154 INFO:teuthology.orchestra.run.smithi146.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-03-11T20:27:29.174 INFO:teuthology.orchestra.run.smithi146.stderr:done 2024-03-11T20:27:29.176 DEBUG:teuthology.orchestra.run.smithi023:> sudo chmod 0666 /tmp/tmp.3rBPpKB5qT 2024-03-11T20:27:29.192 DEBUG:teuthology.orchestra.run.smithi146:> sudo shutdown -r now 2024-03-11T20:27:29.289 DEBUG:teuthology.orchestra.remote:smithi023:/tmp/tmp.3rBPpKB5qT is 10KB 2024-03-11T20:27:29.302 DEBUG:teuthology.orchestra.run.smithi023:> rm -fr /tmp/tmp.3rBPpKB5qT 2024-03-11T20:27:29.307 DEBUG:teuthology.orchestra.run.smithi023:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-03-11T20:27:29.360 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-11T20:27:29.360 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-03-11T20:27:29.360 DEBUG:teuthology.orchestra.run.smithi023:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-03-11T20:27:29.423 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-100-generic 2024-03-11T20:27:29.423 DEBUG:teuthology.orchestra.run.smithi023:> sudo update-grub 2024-03-11T20:27:30.620 INFO:teuthology.orchestra.run.smithi023.stderr:Sourcing file `/etc/default/grub' 2024-03-11T20:27:30.621 INFO:teuthology.orchestra.run.smithi023.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-03-11T20:27:30.623 INFO:teuthology.orchestra.run.smithi023.stderr:Generating grub configuration file ... 2024-03-11T20:27:30.776 INFO:teuthology.orchestra.run.smithi023.stderr:Found linux image: /boot/vmlinuz-5.15.0-100-generic 2024-03-11T20:27:30.783 INFO:teuthology.orchestra.run.smithi023.stderr:Found initrd image: /boot/initrd.img-5.15.0-100-generic 2024-03-11T20:27:31.002 INFO:teuthology.orchestra.run.smithi023.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-03-11T20:27:31.007 INFO:teuthology.orchestra.run.smithi023.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-03-11T20:27:31.045 INFO:teuthology.orchestra.run.smithi023.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-03-11T20:27:31.050 INFO:teuthology.orchestra.run.smithi023.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-03-11T20:27:31.194 INFO:teuthology.orchestra.run.smithi023.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-03-11T20:27:31.194 INFO:teuthology.orchestra.run.smithi023.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-03-11T20:27:31.194 INFO:teuthology.orchestra.run.smithi023.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-03-11T20:27:31.214 INFO:teuthology.orchestra.run.smithi023.stderr:done 2024-03-11T20:27:31.233 DEBUG:teuthology.orchestra.run.smithi023:> sudo shutdown -r now 2024-03-11T20:27:59.223 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-11T20:27:59.223 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-03-11T20:27:59.224 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T20:28:01.237 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-11T20:28:01.237 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi023.front.sepia.ceph.com' 2024-03-11T20:28:01.238 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi023.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T20:28:17.566 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.146 2024-03-11T20:28:19.614 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.23 2024-03-11T20:28:26.574 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-03-11T20:28:26.575 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T20:28:28.617 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi023.front.sepia.ceph.com' 2024-03-11T20:28:28.618 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi023.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T20:28:29.630 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.146 2024-03-11T20:28:31.678 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.23 2024-03-11T20:28:41.637 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-03-11T20:28:41.637 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T20:28:43.682 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi023.front.sepia.ceph.com' 2024-03-11T20:28:43.682 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi023.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T20:28:46.750 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.23 2024-03-11T20:29:01.762 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi023.front.sepia.ceph.com' 2024-03-11T20:29:01.763 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi023.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T20:29:04.830 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.23 2024-03-11T20:29:22.846 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi023.front.sepia.ceph.com' 2024-03-11T20:29:22.847 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi023.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T20:29:41.658 DEBUG:teuthology.orchestra.remote:timed out 2024-03-11T20:29:54.533 DEBUG:teuthology.orchestra.run.smithi023:> true 2024-03-11T20:29:55.210 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi023.front.sepia.ceph.com' 2024-03-11T20:29:55.210 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-100-generic"... 2024-03-11T20:29:55.211 DEBUG:teuthology.orchestra.run.smithi023:> uname -r 2024-03-11T20:29:55.257 INFO:teuthology.orchestra.run.smithi023.stdout:5.15.0-100-generic 2024-03-11T20:29:55.257 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-100-generic vs 5.15.0-100-generic 2024-03-11T20:29:55.257 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-11T20:29:55.257 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-03-11T20:29:56.258 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-11T20:29:56.258 DEBUG:teuthology.orchestra.run.smithi023:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-11T20:29:56.337 INFO:teuthology.orchestra.run.smithi023.stdout:ttyS1 2024-03-11T20:29:56.383 DEBUG:teuthology.parallel:result is None 2024-03-11T20:29:56.659 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-03-11T20:29:56.660 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T20:29:56.886 DEBUG:teuthology.orchestra.run.smithi146:> true 2024-03-11T20:29:57.446 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-03-11T20:29:57.446 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.15.0-100-generic"... 2024-03-11T20:29:57.446 DEBUG:teuthology.orchestra.run.smithi146:> uname -r 2024-03-11T20:29:57.491 INFO:teuthology.orchestra.run.smithi146.stdout:5.15.0-100-generic 2024-03-11T20:29:57.491 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-100-generic vs 5.15.0-100-generic 2024-03-11T20:29:57.491 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-11T20:29:57.492 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-03-11T20:29:58.492 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-11T20:29:58.493 DEBUG:teuthology.orchestra.run.smithi146:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-11T20:29:58.568 INFO:teuthology.orchestra.run.smithi146.stdout:ttyS1 2024-03-11T20:29:58.593 DEBUG:teuthology.parallel:result is None 2024-03-11T20:29:58.593 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-11T20:29:58.600 INFO:teuthology.task.internal:Creating test directory... 2024-03-11T20:29:58.600 DEBUG:teuthology.orchestra.run.smithi023:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-11T20:29:58.603 DEBUG:teuthology.orchestra.run.smithi146:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-11T20:29:58.609 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-11T20:29:58.617 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-11T20:29:58.628 INFO:teuthology.task.internal:Creating archive directory... 2024-03-11T20:29:58.629 DEBUG:teuthology.orchestra.run.smithi023:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-11T20:29:58.650 DEBUG:teuthology.orchestra.run.smithi146:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-11T20:29:58.670 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-11T20:29:58.678 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-11T20:29:58.678 DEBUG:teuthology.orchestra.run.smithi023:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-11T20:29:58.710 DEBUG:teuthology.orchestra.run.smithi146:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-11T20:29:58.727 INFO:teuthology.orchestra.run.smithi023.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-11T20:29:58.728 INFO:teuthology.orchestra.run.smithi146.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-11T20:29:58.735 INFO:teuthology.orchestra.run.smithi023.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-11T20:29:58.736 INFO:teuthology.orchestra.run.smithi146.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-11T20:29:58.738 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-11T20:29:58.755 INFO:teuthology.task.internal:Configuring sudo... 2024-03-11T20:29:58.756 DEBUG:teuthology.orchestra.run.smithi023:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-11T20:29:58.782 DEBUG:teuthology.orchestra.run.smithi146:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-11T20:29:58.796 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-11T20:29:58.809 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-11T20:29:58.809 DEBUG:teuthology.orchestra.run.smithi023:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-11T20:29:58.838 DEBUG:teuthology.orchestra.run.smithi146:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-11T20:29:58.843 DEBUG:teuthology.orchestra.run.smithi023:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-11T20:29:58.889 DEBUG:teuthology.orchestra.run.smithi023:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-11T20:29:58.938 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-11T20:29:58.938 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-11T20:29:58.997 DEBUG:teuthology.orchestra.run.smithi146:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-11T20:29:59.002 DEBUG:teuthology.orchestra.run.smithi146:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-11T20:29:59.047 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-11T20:29:59.048 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-11T20:29:59.115 DEBUG:teuthology.orchestra.run.smithi023:> sudo service rsyslog restart 2024-03-11T20:29:59.118 DEBUG:teuthology.orchestra.run.smithi146:> sudo service rsyslog restart 2024-03-11T20:29:59.192 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-11T20:29:59.201 INFO:teuthology.task.internal:Starting timer... 2024-03-11T20:29:59.201 INFO:teuthology.run_tasks:Running task pcp... 2024-03-11T20:29:59.218 INFO:teuthology.run_tasks:Running task selinux... 2024-03-11T20:29:59.234 DEBUG:teuthology.task.selinux:Excluding smithi023: OS 'ubuntu' does not support SELinux 2024-03-11T20:29:59.234 DEBUG:teuthology.task.selinux:Excluding smithi146: OS 'ubuntu' does not support SELinux 2024-03-11T20:29:59.234 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-11T20:29:59.234 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-03-11T20:29:59.234 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-11T20:29:59.234 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-11T20:29:59.250 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-11T20:29:59.267 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-11T20:29:59.268 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi023.front.sepia.ceph.com,smithi146.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-11T20:37:09.253 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi023.front.sepia.ceph.com'), Remote(name='ubuntu@smithi146.front.sepia.ceph.com')] 2024-03-11T20:37:09.254 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi023.front.sepia.ceph.com' 2024-03-11T20:37:09.255 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi023.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T20:37:09.337 DEBUG:teuthology.orchestra.run.smithi023:> true 2024-03-11T20:37:09.410 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi023.front.sepia.ceph.com' 2024-03-11T20:37:09.410 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-03-11T20:37:09.411 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T20:37:09.490 DEBUG:teuthology.orchestra.run.smithi146:> true 2024-03-11T20:37:09.574 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-03-11T20:37:09.574 INFO:teuthology.run_tasks:Running task clock... 2024-03-11T20:37:09.585 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-11T20:37:09.585 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-11T20:37:09.585 DEBUG:teuthology.orchestra.run.smithi023:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-11T20:37:09.588 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-11T20:37:09.588 DEBUG:teuthology.orchestra.run.smithi146:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-11T20:37:09.617 INFO:teuthology.orchestra.run.smithi023.stdout:11 Mar 20:37:09 ntpd[17065]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-03-11T20:37:09.617 INFO:teuthology.orchestra.run.smithi023.stdout:11 Mar 20:37:09 ntpd[17065]: Command line: ntpd -gq 2024-03-11T20:37:09.617 INFO:teuthology.orchestra.run.smithi023.stdout:11 Mar 20:37:09 ntpd[17065]: ---------------------------------------------------- 2024-03-11T20:37:09.617 INFO:teuthology.orchestra.run.smithi023.stdout:11 Mar 20:37:09 ntpd[17065]: ntp-4 is maintained by Network Time Foundation, 2024-03-11T20:37:09.617 INFO:teuthology.orchestra.run.smithi023.stdout:11 Mar 20:37:09 ntpd[17065]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-03-11T20:37:09.617 INFO:teuthology.orchestra.run.smithi023.stdout:11 Mar 20:37:09 ntpd[17065]: corporation. Support and training for ntp-4 are 2024-03-11T20:37:09.617 INFO:teuthology.orchestra.run.smithi023.stdout:11 Mar 20:37:09 ntpd[17065]: available at https://www.nwtime.org/support 2024-03-11T20:37:09.617 INFO:teuthology.orchestra.run.smithi023.stdout:11 Mar 20:37:09 ntpd[17065]: ---------------------------------------------------- 2024-03-11T20:37:09.618 INFO:teuthology.orchestra.run.smithi023.stdout:11 Mar 20:37:09 ntpd[17065]: proto: precision = 0.066 usec (-24) 2024-03-11T20:37:09.618 INFO:teuthology.orchestra.run.smithi023.stdout:11 Mar 20:37:09 ntpd[17065]: basedate set to 2022-02-04 2024-03-11T20:37:09.618 INFO:teuthology.orchestra.run.smithi023.stdout:11 Mar 20:37:09 ntpd[17065]: gps base set to 2022-02-06 (week 2196) 2024-03-11T20:37:09.619 INFO:teuthology.orchestra.run.smithi023.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-03-11T20:37:09.619 INFO:teuthology.orchestra.run.smithi023.stdout:11 Mar 20:37:09 ntpd[17065]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-03-11T20:37:09.619 INFO:teuthology.orchestra.run.smithi023.stderr:restrict ::: KOD does nothing without LIMITED. 2024-03-11T20:37:09.620 INFO:teuthology.orchestra.run.smithi023.stdout:11 Mar 20:37:09 ntpd[17065]: restrict ::: KOD does nothing without LIMITED. 2024-03-11T20:37:09.620 INFO:teuthology.orchestra.run.smithi023.stdout:11 Mar 20:37:09 ntpd[17065]: Listen and drop on 0 v6wildcard [::]:123 2024-03-11T20:37:09.620 INFO:teuthology.orchestra.run.smithi023.stdout:11 Mar 20:37:09 ntpd[17065]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-03-11T20:37:09.620 INFO:teuthology.orchestra.run.smithi023.stdout:11 Mar 20:37:09 ntpd[17065]: Listen normally on 2 lo 127.0.0.1:123 2024-03-11T20:37:09.620 INFO:teuthology.orchestra.run.smithi023.stdout:11 Mar 20:37:09 ntpd[17065]: Listen normally on 3 ens1f0 172.21.15.23:123 2024-03-11T20:37:09.620 INFO:teuthology.orchestra.run.smithi023.stdout:11 Mar 20:37:09 ntpd[17065]: Listen normally on 4 lo [::1]:123 2024-03-11T20:37:09.621 INFO:teuthology.orchestra.run.smithi023.stdout:11 Mar 20:37:09 ntpd[17065]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:fe59:ce42%4]:123 2024-03-11T20:37:09.621 INFO:teuthology.orchestra.run.smithi023.stdout:11 Mar 20:37:09 ntpd[17065]: Listening on routing socket on fd #22 for interface updates 2024-03-11T20:37:09.642 INFO:teuthology.orchestra.run.smithi146.stdout:11 Mar 20:37:09 ntpd[17180]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-03-11T20:37:09.643 INFO:teuthology.orchestra.run.smithi146.stdout:11 Mar 20:37:09 ntpd[17180]: Command line: ntpd -gq 2024-03-11T20:37:09.643 INFO:teuthology.orchestra.run.smithi146.stdout:11 Mar 20:37:09 ntpd[17180]: ---------------------------------------------------- 2024-03-11T20:37:09.643 INFO:teuthology.orchestra.run.smithi146.stdout:11 Mar 20:37:09 ntpd[17180]: ntp-4 is maintained by Network Time Foundation, 2024-03-11T20:37:09.643 INFO:teuthology.orchestra.run.smithi146.stdout:11 Mar 20:37:09 ntpd[17180]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-03-11T20:37:09.643 INFO:teuthology.orchestra.run.smithi146.stdout:11 Mar 20:37:09 ntpd[17180]: corporation. Support and training for ntp-4 are 2024-03-11T20:37:09.643 INFO:teuthology.orchestra.run.smithi146.stdout:11 Mar 20:37:09 ntpd[17180]: available at https://www.nwtime.org/support 2024-03-11T20:37:09.643 INFO:teuthology.orchestra.run.smithi146.stdout:11 Mar 20:37:09 ntpd[17180]: ---------------------------------------------------- 2024-03-11T20:37:09.644 INFO:teuthology.orchestra.run.smithi146.stdout:11 Mar 20:37:09 ntpd[17180]: proto: precision = 0.059 usec (-24) 2024-03-11T20:37:09.644 INFO:teuthology.orchestra.run.smithi146.stdout:11 Mar 20:37:09 ntpd[17180]: basedate set to 2022-02-04 2024-03-11T20:37:09.644 INFO:teuthology.orchestra.run.smithi146.stdout:11 Mar 20:37:09 ntpd[17180]: gps base set to 2022-02-06 (week 2196) 2024-03-11T20:37:09.644 INFO:teuthology.orchestra.run.smithi146.stdout:11 Mar 20:37:09 ntpd[17180]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-03-11T20:37:09.644 INFO:teuthology.orchestra.run.smithi146.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-03-11T20:37:09.644 INFO:teuthology.orchestra.run.smithi146.stderr:restrict ::: KOD does nothing without LIMITED. 2024-03-11T20:37:09.645 INFO:teuthology.orchestra.run.smithi146.stdout:11 Mar 20:37:09 ntpd[17180]: restrict ::: KOD does nothing without LIMITED. 2024-03-11T20:37:09.645 INFO:teuthology.orchestra.run.smithi146.stdout:11 Mar 20:37:09 ntpd[17180]: Listen and drop on 0 v6wildcard [::]:123 2024-03-11T20:37:09.645 INFO:teuthology.orchestra.run.smithi146.stdout:11 Mar 20:37:09 ntpd[17180]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-03-11T20:37:09.645 INFO:teuthology.orchestra.run.smithi146.stdout:11 Mar 20:37:09 ntpd[17180]: Listen normally on 2 lo 127.0.0.1:123 2024-03-11T20:37:09.645 INFO:teuthology.orchestra.run.smithi146.stdout:11 Mar 20:37:09 ntpd[17180]: Listen normally on 3 enp3s0f1 172.21.15.146:123 2024-03-11T20:37:09.645 INFO:teuthology.orchestra.run.smithi146.stdout:11 Mar 20:37:09 ntpd[17180]: Listen normally on 4 lo [::1]:123 2024-03-11T20:37:09.645 INFO:teuthology.orchestra.run.smithi146.stdout:11 Mar 20:37:09 ntpd[17180]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:76fd%5]:123 2024-03-11T20:37:09.646 INFO:teuthology.orchestra.run.smithi146.stdout:11 Mar 20:37:09 ntpd[17180]: Listening on routing socket on fd #22 for interface updates 2024-03-11T20:37:10.618 INFO:teuthology.orchestra.run.smithi023.stderr:11 Mar 20:37:10 ntpd[17065]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-11T20:37:10.618 INFO:teuthology.orchestra.run.smithi023.stderr:11 Mar 20:37:10 ntpd[17065]: can't open /var/log/ntpstats/rawstats.20240311: Permission denied 2024-03-11T20:37:10.619 INFO:teuthology.orchestra.run.smithi023.stderr:11 Mar 20:37:10 ntpd[17065]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-11T20:37:10.619 INFO:teuthology.orchestra.run.smithi023.stderr:11 Mar 20:37:10 ntpd[17065]: can't open /var/log/ntpstats/rawstats.20240311: Permission denied 2024-03-11T20:37:10.619 INFO:teuthology.orchestra.run.smithi023.stderr:11 Mar 20:37:10 ntpd[17065]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-11T20:37:10.619 INFO:teuthology.orchestra.run.smithi023.stderr:11 Mar 20:37:10 ntpd[17065]: can't open /var/log/ntpstats/rawstats.20240311: Permission denied 2024-03-11T20:37:10.619 INFO:teuthology.orchestra.run.smithi023.stderr:11 Mar 20:37:10 ntpd[17065]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-11T20:37:10.619 INFO:teuthology.orchestra.run.smithi023.stderr:11 Mar 20:37:10 ntpd[17065]: can't open /var/log/ntpstats/peerstats.20240311: Permission denied 2024-03-11T20:37:10.619 INFO:teuthology.orchestra.run.smithi023.stderr:11 Mar 20:37:10 ntpd[17065]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-11T20:37:10.619 INFO:teuthology.orchestra.run.smithi023.stderr:11 Mar 20:37:10 ntpd[17065]: can't open /var/log/ntpstats/rawstats.20240311: Permission denied 2024-03-11T20:37:10.619 INFO:teuthology.orchestra.run.smithi023.stderr:11 Mar 20:37:10 ntpd[17065]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-11T20:37:10.619 INFO:teuthology.orchestra.run.smithi023.stderr:11 Mar 20:37:10 ntpd[17065]: can't open /var/log/ntpstats/peerstats.20240311: Permission denied 2024-03-11T20:37:10.642 INFO:teuthology.orchestra.run.smithi146.stderr:11 Mar 20:37:10 ntpd[17180]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-11T20:37:10.643 INFO:teuthology.orchestra.run.smithi146.stderr:11 Mar 20:37:10 ntpd[17180]: can't open /var/log/ntpstats/rawstats.20240311: Permission denied 2024-03-11T20:37:11.618 INFO:teuthology.orchestra.run.smithi023.stderr:11 Mar 20:37:11 ntpd[17065]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-11T20:37:11.618 INFO:teuthology.orchestra.run.smithi023.stderr:11 Mar 20:37:11 ntpd[17065]: can't open /var/log/ntpstats/rawstats.20240311: Permission denied 2024-03-11T20:37:12.620 INFO:teuthology.orchestra.run.smithi023.stderr:11 Mar 20:37:12 ntpd[17065]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-11T20:37:12.620 INFO:teuthology.orchestra.run.smithi023.stderr:11 Mar 20:37:12 ntpd[17065]: can't open /var/log/ntpstats/rawstats.20240311: Permission denied 2024-03-11T20:37:12.620 INFO:teuthology.orchestra.run.smithi023.stderr:11 Mar 20:37:12 ntpd[17065]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-11T20:37:12.620 INFO:teuthology.orchestra.run.smithi023.stderr:11 Mar 20:37:12 ntpd[17065]: can't open /var/log/ntpstats/rawstats.20240311: Permission denied 2024-03-11T20:37:12.620 INFO:teuthology.orchestra.run.smithi023.stderr:11 Mar 20:37:12 ntpd[17065]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-11T20:37:12.620 INFO:teuthology.orchestra.run.smithi023.stderr:11 Mar 20:37:12 ntpd[17065]: can't open /var/log/ntpstats/rawstats.20240311: Permission denied 2024-03-11T20:37:12.620 INFO:teuthology.orchestra.run.smithi023.stderr:11 Mar 20:37:12 ntpd[17065]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-11T20:37:12.620 INFO:teuthology.orchestra.run.smithi023.stderr:11 Mar 20:37:12 ntpd[17065]: can't open /var/log/ntpstats/rawstats.20240311: Permission denied 2024-03-11T20:37:12.620 INFO:teuthology.orchestra.run.smithi023.stderr:11 Mar 20:37:12 ntpd[17065]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-11T20:37:12.620 INFO:teuthology.orchestra.run.smithi023.stderr:11 Mar 20:37:12 ntpd[17065]: can't open /var/log/ntpstats/peerstats.20240311: Permission denied 2024-03-11T20:37:12.621 INFO:teuthology.orchestra.run.smithi023.stderr:11 Mar 20:37:12 ntpd[17065]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-11T20:37:12.621 INFO:teuthology.orchestra.run.smithi023.stderr:11 Mar 20:37:12 ntpd[17065]: can't open /var/log/ntpstats/rawstats.20240311: Permission denied 2024-03-11T20:37:12.621 INFO:teuthology.orchestra.run.smithi023.stderr:11 Mar 20:37:12 ntpd[17065]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-11T20:37:12.621 INFO:teuthology.orchestra.run.smithi023.stderr:11 Mar 20:37:12 ntpd[17065]: can't open /var/log/ntpstats/peerstats.20240311: Permission denied 2024-03-11T20:37:12.621 INFO:teuthology.orchestra.run.smithi023.stderr:11 Mar 20:37:12 ntpd[17065]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-11T20:37:12.621 INFO:teuthology.orchestra.run.smithi023.stderr:11 Mar 20:37:12 ntpd[17065]: can't open /var/log/ntpstats/rawstats.20240311: Permission denied 2024-03-11T20:37:12.621 INFO:teuthology.orchestra.run.smithi023.stderr:11 Mar 20:37:12 ntpd[17065]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-11T20:37:12.621 INFO:teuthology.orchestra.run.smithi023.stderr:11 Mar 20:37:12 ntpd[17065]: can't open /var/log/ntpstats/peerstats.20240311: Permission denied 2024-03-11T20:37:12.644 INFO:teuthology.orchestra.run.smithi146.stderr:11 Mar 20:37:12 ntpd[17180]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-11T20:37:12.644 INFO:teuthology.orchestra.run.smithi146.stderr:11 Mar 20:37:12 ntpd[17180]: can't open /var/log/ntpstats/rawstats.20240311: Permission denied 2024-03-11T20:37:12.644 INFO:teuthology.orchestra.run.smithi146.stderr:11 Mar 20:37:12 ntpd[17180]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-11T20:37:12.644 INFO:teuthology.orchestra.run.smithi146.stderr:11 Mar 20:37:12 ntpd[17180]: can't open /var/log/ntpstats/rawstats.20240311: Permission denied 2024-03-11T20:37:12.644 INFO:teuthology.orchestra.run.smithi146.stderr:11 Mar 20:37:12 ntpd[17180]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-11T20:37:12.644 INFO:teuthology.orchestra.run.smithi146.stderr:11 Mar 20:37:12 ntpd[17180]: can't open /var/log/ntpstats/rawstats.20240311: Permission denied 2024-03-11T20:37:12.645 INFO:teuthology.orchestra.run.smithi146.stderr:11 Mar 20:37:12 ntpd[17180]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-11T20:37:12.645 INFO:teuthology.orchestra.run.smithi146.stderr:11 Mar 20:37:12 ntpd[17180]: can't open /var/log/ntpstats/rawstats.20240311: Permission denied 2024-03-11T20:37:12.645 INFO:teuthology.orchestra.run.smithi146.stderr:11 Mar 20:37:12 ntpd[17180]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-11T20:37:12.645 INFO:teuthology.orchestra.run.smithi146.stderr:11 Mar 20:37:12 ntpd[17180]: can't open /var/log/ntpstats/peerstats.20240311: Permission denied 2024-03-11T20:37:12.645 INFO:teuthology.orchestra.run.smithi146.stderr:11 Mar 20:37:12 ntpd[17180]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-11T20:37:12.645 INFO:teuthology.orchestra.run.smithi146.stderr:11 Mar 20:37:12 ntpd[17180]: can't open /var/log/ntpstats/rawstats.20240311: Permission denied 2024-03-11T20:37:12.645 INFO:teuthology.orchestra.run.smithi146.stderr:11 Mar 20:37:12 ntpd[17180]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-11T20:37:12.645 INFO:teuthology.orchestra.run.smithi146.stderr:11 Mar 20:37:12 ntpd[17180]: can't open /var/log/ntpstats/peerstats.20240311: Permission denied 2024-03-11T20:37:12.645 INFO:teuthology.orchestra.run.smithi146.stderr:11 Mar 20:37:12 ntpd[17180]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-11T20:37:12.645 INFO:teuthology.orchestra.run.smithi146.stderr:11 Mar 20:37:12 ntpd[17180]: can't open /var/log/ntpstats/rawstats.20240311: Permission denied 2024-03-11T20:37:12.645 INFO:teuthology.orchestra.run.smithi146.stderr:11 Mar 20:37:12 ntpd[17180]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-11T20:37:12.645 INFO:teuthology.orchestra.run.smithi146.stderr:11 Mar 20:37:12 ntpd[17180]: can't open /var/log/ntpstats/peerstats.20240311: Permission denied 2024-03-11T20:37:14.619 INFO:teuthology.orchestra.run.smithi023.stderr:11 Mar 20:37:14 ntpd[17065]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-11T20:37:14.619 INFO:teuthology.orchestra.run.smithi023.stderr:11 Mar 20:37:14 ntpd[17065]: can't open /var/log/ntpstats/rawstats.20240311: Permission denied 2024-03-11T20:37:14.620 INFO:teuthology.orchestra.run.smithi023.stderr:11 Mar 20:37:14 ntpd[17065]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-11T20:37:14.620 INFO:teuthology.orchestra.run.smithi023.stderr:11 Mar 20:37:14 ntpd[17065]: can't open /var/log/ntpstats/rawstats.20240311: Permission denied 2024-03-11T20:37:14.620 INFO:teuthology.orchestra.run.smithi023.stderr:11 Mar 20:37:14 ntpd[17065]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-11T20:37:14.620 INFO:teuthology.orchestra.run.smithi023.stderr:11 Mar 20:37:14 ntpd[17065]: can't open /var/log/ntpstats/rawstats.20240311: Permission denied 2024-03-11T20:37:14.620 INFO:teuthology.orchestra.run.smithi023.stderr:11 Mar 20:37:14 ntpd[17065]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-11T20:37:14.620 INFO:teuthology.orchestra.run.smithi023.stderr:11 Mar 20:37:14 ntpd[17065]: can't open /var/log/ntpstats/rawstats.20240311: Permission denied 2024-03-11T20:37:14.620 INFO:teuthology.orchestra.run.smithi023.stderr:11 Mar 20:37:14 ntpd[17065]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-11T20:37:14.620 INFO:teuthology.orchestra.run.smithi023.stderr:11 Mar 20:37:14 ntpd[17065]: can't open /var/log/ntpstats/peerstats.20240311: Permission denied 2024-03-11T20:37:14.620 INFO:teuthology.orchestra.run.smithi023.stderr:11 Mar 20:37:14 ntpd[17065]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-11T20:37:14.620 INFO:teuthology.orchestra.run.smithi023.stderr:11 Mar 20:37:14 ntpd[17065]: can't open /var/log/ntpstats/rawstats.20240311: Permission denied 2024-03-11T20:37:14.620 INFO:teuthology.orchestra.run.smithi023.stderr:11 Mar 20:37:14 ntpd[17065]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-11T20:37:14.620 INFO:teuthology.orchestra.run.smithi023.stderr:11 Mar 20:37:14 ntpd[17065]: can't open /var/log/ntpstats/peerstats.20240311: Permission denied 2024-03-11T20:37:14.620 INFO:teuthology.orchestra.run.smithi023.stderr:11 Mar 20:37:14 ntpd[17065]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-11T20:37:14.620 INFO:teuthology.orchestra.run.smithi023.stderr:11 Mar 20:37:14 ntpd[17065]: can't open /var/log/ntpstats/rawstats.20240311: Permission denied 2024-03-11T20:37:14.620 INFO:teuthology.orchestra.run.smithi023.stderr:11 Mar 20:37:14 ntpd[17065]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-11T20:37:14.621 INFO:teuthology.orchestra.run.smithi023.stderr:11 Mar 20:37:14 ntpd[17065]: can't open /var/log/ntpstats/peerstats.20240311: Permission denied 2024-03-11T20:37:14.644 INFO:teuthology.orchestra.run.smithi146.stderr:11 Mar 20:37:14 ntpd[17180]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-11T20:37:14.645 INFO:teuthology.orchestra.run.smithi146.stderr:11 Mar 20:37:14 ntpd[17180]: can't open /var/log/ntpstats/rawstats.20240311: Permission denied 2024-03-11T20:37:14.645 INFO:teuthology.orchestra.run.smithi146.stderr:11 Mar 20:37:14 ntpd[17180]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-11T20:37:14.645 INFO:teuthology.orchestra.run.smithi146.stderr:11 Mar 20:37:14 ntpd[17180]: can't open /var/log/ntpstats/rawstats.20240311: Permission denied 2024-03-11T20:37:14.645 INFO:teuthology.orchestra.run.smithi146.stderr:11 Mar 20:37:14 ntpd[17180]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-11T20:37:14.645 INFO:teuthology.orchestra.run.smithi146.stderr:11 Mar 20:37:14 ntpd[17180]: can't open /var/log/ntpstats/rawstats.20240311: Permission denied 2024-03-11T20:37:14.645 INFO:teuthology.orchestra.run.smithi146.stderr:11 Mar 20:37:14 ntpd[17180]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-11T20:37:14.645 INFO:teuthology.orchestra.run.smithi146.stderr:11 Mar 20:37:14 ntpd[17180]: can't open /var/log/ntpstats/rawstats.20240311: Permission denied 2024-03-11T20:37:14.645 INFO:teuthology.orchestra.run.smithi146.stderr:11 Mar 20:37:14 ntpd[17180]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-11T20:37:14.645 INFO:teuthology.orchestra.run.smithi146.stderr:11 Mar 20:37:14 ntpd[17180]: can't open /var/log/ntpstats/peerstats.20240311: Permission denied 2024-03-11T20:37:14.645 INFO:teuthology.orchestra.run.smithi146.stderr:11 Mar 20:37:14 ntpd[17180]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-11T20:37:14.645 INFO:teuthology.orchestra.run.smithi146.stderr:11 Mar 20:37:14 ntpd[17180]: can't open /var/log/ntpstats/rawstats.20240311: Permission denied 2024-03-11T20:37:14.645 INFO:teuthology.orchestra.run.smithi146.stderr:11 Mar 20:37:14 ntpd[17180]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-11T20:37:14.645 INFO:teuthology.orchestra.run.smithi146.stderr:11 Mar 20:37:14 ntpd[17180]: can't open /var/log/ntpstats/peerstats.20240311: Permission denied 2024-03-11T20:37:14.645 INFO:teuthology.orchestra.run.smithi146.stderr:11 Mar 20:37:14 ntpd[17180]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-11T20:37:14.646 INFO:teuthology.orchestra.run.smithi146.stderr:11 Mar 20:37:14 ntpd[17180]: can't open /var/log/ntpstats/rawstats.20240311: Permission denied 2024-03-11T20:37:14.646 INFO:teuthology.orchestra.run.smithi146.stderr:11 Mar 20:37:14 ntpd[17180]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-11T20:37:14.646 INFO:teuthology.orchestra.run.smithi146.stderr:11 Mar 20:37:14 ntpd[17180]: can't open /var/log/ntpstats/peerstats.20240311: Permission denied 2024-03-11T20:37:16.619 INFO:teuthology.orchestra.run.smithi023.stderr:11 Mar 20:37:16 ntpd[17065]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-11T20:37:16.619 INFO:teuthology.orchestra.run.smithi023.stderr:11 Mar 20:37:16 ntpd[17065]: can't open /var/log/ntpstats/rawstats.20240311: Permission denied 2024-03-11T20:37:16.619 INFO:teuthology.orchestra.run.smithi023.stderr:11 Mar 20:37:16 ntpd[17065]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-11T20:37:16.619 INFO:teuthology.orchestra.run.smithi023.stderr:11 Mar 20:37:16 ntpd[17065]: can't open /var/log/ntpstats/rawstats.20240311: Permission denied 2024-03-11T20:37:16.619 INFO:teuthology.orchestra.run.smithi023.stderr:11 Mar 20:37:16 ntpd[17065]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-11T20:37:16.619 INFO:teuthology.orchestra.run.smithi023.stderr:11 Mar 20:37:16 ntpd[17065]: can't open /var/log/ntpstats/rawstats.20240311: Permission denied 2024-03-11T20:37:16.619 INFO:teuthology.orchestra.run.smithi023.stderr:11 Mar 20:37:16 ntpd[17065]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-11T20:37:16.619 INFO:teuthology.orchestra.run.smithi023.stderr:11 Mar 20:37:16 ntpd[17065]: can't open /var/log/ntpstats/rawstats.20240311: Permission denied 2024-03-11T20:37:16.619 INFO:teuthology.orchestra.run.smithi023.stderr:11 Mar 20:37:16 ntpd[17065]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-11T20:37:16.619 INFO:teuthology.orchestra.run.smithi023.stderr:11 Mar 20:37:16 ntpd[17065]: can't open /var/log/ntpstats/peerstats.20240311: Permission denied 2024-03-11T20:37:16.620 INFO:teuthology.orchestra.run.smithi023.stdout:11 Mar 20:37:16 ntpd[17065]: ntpd: time slew -0.001522 s 2024-03-11T20:37:16.620 INFO:teuthology.orchestra.run.smithi023.stdout:ntpd: time slew -0.001522s 2024-03-11T20:37:16.620 INFO:teuthology.orchestra.run.smithi023.stderr:11 Mar 20:37:16 ntpd[17065]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-03-11T20:37:16.620 INFO:teuthology.orchestra.run.smithi023.stderr:11 Mar 20:37:16 ntpd[17065]: can't open /var/log/ntpstats/loopstats.20240311: Permission denied 2024-03-11T20:37:16.644 INFO:teuthology.orchestra.run.smithi146.stderr:11 Mar 20:37:16 ntpd[17180]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-11T20:37:16.644 INFO:teuthology.orchestra.run.smithi146.stderr:11 Mar 20:37:16 ntpd[17180]: can't open /var/log/ntpstats/rawstats.20240311: Permission denied 2024-03-11T20:37:16.645 INFO:teuthology.orchestra.run.smithi146.stderr:11 Mar 20:37:16 ntpd[17180]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-11T20:37:16.645 INFO:teuthology.orchestra.run.smithi146.stderr:11 Mar 20:37:16 ntpd[17180]: can't open /var/log/ntpstats/rawstats.20240311: Permission denied 2024-03-11T20:37:16.645 INFO:teuthology.orchestra.run.smithi146.stderr:11 Mar 20:37:16 ntpd[17180]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-11T20:37:16.645 INFO:teuthology.orchestra.run.smithi146.stderr:11 Mar 20:37:16 ntpd[17180]: can't open /var/log/ntpstats/rawstats.20240311: Permission denied 2024-03-11T20:37:16.645 INFO:teuthology.orchestra.run.smithi146.stderr:11 Mar 20:37:16 ntpd[17180]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-11T20:37:16.645 INFO:teuthology.orchestra.run.smithi146.stderr:11 Mar 20:37:16 ntpd[17180]: can't open /var/log/ntpstats/rawstats.20240311: Permission denied 2024-03-11T20:37:16.645 INFO:teuthology.orchestra.run.smithi146.stderr:11 Mar 20:37:16 ntpd[17180]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-11T20:37:16.645 INFO:teuthology.orchestra.run.smithi146.stderr:11 Mar 20:37:16 ntpd[17180]: can't open /var/log/ntpstats/peerstats.20240311: Permission denied 2024-03-11T20:37:16.645 INFO:teuthology.orchestra.run.smithi146.stderr:11 Mar 20:37:16 ntpd[17180]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-11T20:37:16.645 INFO:teuthology.orchestra.run.smithi146.stderr:11 Mar 20:37:16 ntpd[17180]: can't open /var/log/ntpstats/rawstats.20240311: Permission denied 2024-03-11T20:37:16.645 INFO:teuthology.orchestra.run.smithi146.stderr:11 Mar 20:37:16 ntpd[17180]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-11T20:37:16.645 INFO:teuthology.orchestra.run.smithi146.stderr:11 Mar 20:37:16 ntpd[17180]: can't open /var/log/ntpstats/peerstats.20240311: Permission denied 2024-03-11T20:37:16.645 INFO:teuthology.orchestra.run.smithi146.stderr:11 Mar 20:37:16 ntpd[17180]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-11T20:37:16.646 INFO:teuthology.orchestra.run.smithi146.stderr:11 Mar 20:37:16 ntpd[17180]: can't open /var/log/ntpstats/rawstats.20240311: Permission denied 2024-03-11T20:37:16.646 INFO:teuthology.orchestra.run.smithi146.stderr:11 Mar 20:37:16 ntpd[17180]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-11T20:37:16.646 INFO:teuthology.orchestra.run.smithi146.stderr:11 Mar 20:37:16 ntpd[17180]: can't open /var/log/ntpstats/peerstats.20240311: Permission denied 2024-03-11T20:37:16.673 INFO:teuthology.orchestra.run.smithi023.stdout: remote refid st t when poll reach delay offset jitter 2024-03-11T20:37:16.674 INFO:teuthology.orchestra.run.smithi023.stdout:============================================================================== 2024-03-11T20:37:16.674 INFO:teuthology.orchestra.run.smithi023.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-11T20:37:16.674 INFO:teuthology.orchestra.run.smithi023.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-11T20:37:16.674 INFO:teuthology.orchestra.run.smithi023.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-11T20:37:16.674 INFO:teuthology.orchestra.run.smithi023.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-11T20:37:18.644 INFO:teuthology.orchestra.run.smithi146.stderr:11 Mar 20:37:18 ntpd[17180]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-11T20:37:18.644 INFO:teuthology.orchestra.run.smithi146.stderr:11 Mar 20:37:18 ntpd[17180]: can't open /var/log/ntpstats/rawstats.20240311: Permission denied 2024-03-11T20:37:18.644 INFO:teuthology.orchestra.run.smithi146.stderr:11 Mar 20:37:18 ntpd[17180]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-11T20:37:18.644 INFO:teuthology.orchestra.run.smithi146.stderr:11 Mar 20:37:18 ntpd[17180]: can't open /var/log/ntpstats/rawstats.20240311: Permission denied 2024-03-11T20:37:18.644 INFO:teuthology.orchestra.run.smithi146.stderr:11 Mar 20:37:18 ntpd[17180]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-11T20:37:18.644 INFO:teuthology.orchestra.run.smithi146.stderr:11 Mar 20:37:18 ntpd[17180]: can't open /var/log/ntpstats/rawstats.20240311: Permission denied 2024-03-11T20:37:18.645 INFO:teuthology.orchestra.run.smithi146.stderr:11 Mar 20:37:18 ntpd[17180]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-11T20:37:18.645 INFO:teuthology.orchestra.run.smithi146.stderr:11 Mar 20:37:18 ntpd[17180]: can't open /var/log/ntpstats/rawstats.20240311: Permission denied 2024-03-11T20:37:18.645 INFO:teuthology.orchestra.run.smithi146.stderr:11 Mar 20:37:18 ntpd[17180]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-11T20:37:18.645 INFO:teuthology.orchestra.run.smithi146.stderr:11 Mar 20:37:18 ntpd[17180]: can't open /var/log/ntpstats/peerstats.20240311: Permission denied 2024-03-11T20:37:18.645 INFO:teuthology.orchestra.run.smithi146.stdout:11 Mar 20:37:18 ntpd[17180]: ntpd: time slew -0.002822 s 2024-03-11T20:37:18.645 INFO:teuthology.orchestra.run.smithi146.stdout:ntpd: time slew -0.002822s 2024-03-11T20:37:18.646 INFO:teuthology.orchestra.run.smithi146.stderr:11 Mar 20:37:18 ntpd[17180]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-03-11T20:37:18.646 INFO:teuthology.orchestra.run.smithi146.stderr:11 Mar 20:37:18 ntpd[17180]: can't open /var/log/ntpstats/loopstats.20240311: Permission denied 2024-03-11T20:37:18.703 INFO:teuthology.orchestra.run.smithi146.stdout: remote refid st t when poll reach delay offset jitter 2024-03-11T20:37:18.703 INFO:teuthology.orchestra.run.smithi146.stdout:============================================================================== 2024-03-11T20:37:18.703 INFO:teuthology.orchestra.run.smithi146.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-11T20:37:18.703 INFO:teuthology.orchestra.run.smithi146.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-11T20:37:18.703 INFO:teuthology.orchestra.run.smithi146.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-11T20:37:18.703 INFO:teuthology.orchestra.run.smithi146.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-11T20:37:18.704 INFO:teuthology.run_tasks:Running task install... 2024-03-11T20:37:18.713 DEBUG:teuthology.task.install:project ceph 2024-03-11T20:37:18.713 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '2b0101eb74d60610d544cb8a473e23a35ac1cc0b'}} 2024-03-11T20:37:18.713 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '2b0101eb74d60610d544cb8a473e23a35ac1cc0b'} 2024-03-11T20:37:18.713 INFO:teuthology.task.install:Using flavor: default 2024-03-11T20:37:18.722 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-11T20:37:18.722 INFO:teuthology.task.install:extra packages: [] 2024-03-11T20:37:18.723 DEBUG:teuthology.orchestra.run.smithi023:> sudo apt-key list | grep Ceph 2024-03-11T20:37:18.723 DEBUG:teuthology.orchestra.run.smithi146:> sudo apt-key list | grep Ceph 2024-03-11T20:37:18.786 INFO:teuthology.orchestra.run.smithi023.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-03-11T20:37:18.798 INFO:teuthology.orchestra.run.smithi146.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-03-11T20:37:18.814 INFO:teuthology.orchestra.run.smithi023.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-03-11T20:37:18.814 INFO:teuthology.orchestra.run.smithi023.stdout:uid [ unknown] Ceph.com (release key) 2024-03-11T20:37:18.815 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-03-11T20:37:18.816 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-11T20:37:18.825 INFO:teuthology.orchestra.run.smithi146.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-03-11T20:37:18.825 INFO:teuthology.orchestra.run.smithi146.stdout:uid [ unknown] Ceph.com (release key) 2024-03-11T20:37:18.825 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-03-11T20:37:18.826 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-11T20:37:18.963 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default/ 2024-03-11T20:37:18.963 INFO:teuthology.task.install.deb:Package version is 19.0.0-1979-g2b0101eb-1jammy 2024-03-11T20:37:18.966 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default/ 2024-03-11T20:37:18.966 INFO:teuthology.task.install.deb:Package version is 19.0.0-1979-g2b0101eb-1jammy 2024-03-11T20:37:19.076 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-11T20:37:19.076 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-03-11T20:37:19.090 DEBUG:teuthology.orchestra.run.smithi146:> sudo apt-get update 2024-03-11T20:37:19.154 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-11T20:37:19.154 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-03-11T20:37:19.168 DEBUG:teuthology.orchestra.run.smithi023:> sudo apt-get update 2024-03-11T20:37:19.313 INFO:teuthology.orchestra.run.smithi146.stdout:Ign:1 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy InRelease 2024-03-11T20:37:19.334 INFO:teuthology.orchestra.run.smithi146.stdout:Get:2 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-03-11T20:37:19.357 INFO:teuthology.orchestra.run.smithi146.stdout:Ign:3 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy Release.gpg 2024-03-11T20:37:19.378 INFO:teuthology.orchestra.run.smithi146.stdout:Get:4 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.8 kB] 2024-03-11T20:37:19.394 INFO:teuthology.orchestra.run.smithi023.stdout:Ign:1 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy InRelease 2024-03-11T20:37:19.410 INFO:teuthology.orchestra.run.smithi146.stdout:Get:5 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,401 B] 2024-03-11T20:37:19.415 INFO:teuthology.orchestra.run.smithi023.stdout:Get:2 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-03-11T20:37:19.439 INFO:teuthology.orchestra.run.smithi023.stdout:Ign:3 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy Release.gpg 2024-03-11T20:37:19.460 INFO:teuthology.orchestra.run.smithi023.stdout:Get:4 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.8 kB] 2024-03-11T20:37:19.492 INFO:teuthology.orchestra.run.smithi023.stdout:Get:5 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,401 B] 2024-03-11T20:37:19.853 INFO:teuthology.orchestra.run.smithi146.stdout:Hit:6 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-03-11T20:37:19.853 INFO:teuthology.orchestra.run.smithi023.stdout:Hit:6 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-03-11T20:37:20.141 INFO:teuthology.orchestra.run.smithi146.stdout:Hit:7 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-03-11T20:37:20.160 INFO:teuthology.orchestra.run.smithi146.stdout:Hit:8 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-03-11T20:37:20.180 INFO:teuthology.orchestra.run.smithi146.stdout:Hit:9 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-03-11T20:37:20.264 INFO:teuthology.orchestra.run.smithi023.stdout:Hit:7 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-03-11T20:37:20.345 INFO:teuthology.orchestra.run.smithi023.stdout:Hit:8 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-03-11T20:37:20.427 INFO:teuthology.orchestra.run.smithi023.stdout:Hit:9 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-03-11T20:37:20.459 INFO:teuthology.orchestra.run.smithi146.stdout:Fetched 28.9 kB in 1s (24.4 kB/s) 2024-03-11T20:37:20.605 INFO:teuthology.orchestra.run.smithi023.stdout:Fetched 28.9 kB in 1s (23.1 kB/s) 2024-03-11T20:37:21.445 INFO:teuthology.orchestra.run.smithi146.stdout:Reading package lists... 2024-03-11T20:37:21.460 DEBUG:teuthology.orchestra.run.smithi146:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-1979-g2b0101eb-1jammy cephadm=19.0.0-1979-g2b0101eb-1jammy ceph-mds=19.0.0-1979-g2b0101eb-1jammy ceph-mgr=19.0.0-1979-g2b0101eb-1jammy ceph-common=19.0.0-1979-g2b0101eb-1jammy ceph-fuse=19.0.0-1979-g2b0101eb-1jammy ceph-test=19.0.0-1979-g2b0101eb-1jammy ceph-volume=19.0.0-1979-g2b0101eb-1jammy radosgw=19.0.0-1979-g2b0101eb-1jammy python3-rados=19.0.0-1979-g2b0101eb-1jammy python3-rgw=19.0.0-1979-g2b0101eb-1jammy python3-cephfs=19.0.0-1979-g2b0101eb-1jammy python3-rbd=19.0.0-1979-g2b0101eb-1jammy libcephfs2=19.0.0-1979-g2b0101eb-1jammy libcephfs-dev=19.0.0-1979-g2b0101eb-1jammy librados2=19.0.0-1979-g2b0101eb-1jammy librbd1=19.0.0-1979-g2b0101eb-1jammy rbd-fuse=19.0.0-1979-g2b0101eb-1jammy 2024-03-11T20:37:21.522 INFO:teuthology.orchestra.run.smithi146.stdout:Reading package lists... 2024-03-11T20:37:21.564 INFO:teuthology.orchestra.run.smithi023.stdout:Reading package lists... 2024-03-11T20:37:21.580 DEBUG:teuthology.orchestra.run.smithi023:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-1979-g2b0101eb-1jammy cephadm=19.0.0-1979-g2b0101eb-1jammy ceph-mds=19.0.0-1979-g2b0101eb-1jammy ceph-mgr=19.0.0-1979-g2b0101eb-1jammy ceph-common=19.0.0-1979-g2b0101eb-1jammy ceph-fuse=19.0.0-1979-g2b0101eb-1jammy ceph-test=19.0.0-1979-g2b0101eb-1jammy ceph-volume=19.0.0-1979-g2b0101eb-1jammy radosgw=19.0.0-1979-g2b0101eb-1jammy python3-rados=19.0.0-1979-g2b0101eb-1jammy python3-rgw=19.0.0-1979-g2b0101eb-1jammy python3-cephfs=19.0.0-1979-g2b0101eb-1jammy python3-rbd=19.0.0-1979-g2b0101eb-1jammy libcephfs2=19.0.0-1979-g2b0101eb-1jammy libcephfs-dev=19.0.0-1979-g2b0101eb-1jammy librados2=19.0.0-1979-g2b0101eb-1jammy librbd1=19.0.0-1979-g2b0101eb-1jammy rbd-fuse=19.0.0-1979-g2b0101eb-1jammy 2024-03-11T20:37:21.643 INFO:teuthology.orchestra.run.smithi023.stdout:Reading package lists... 2024-03-11T20:37:21.716 INFO:teuthology.orchestra.run.smithi146.stdout:Building dependency tree... 2024-03-11T20:37:21.717 INFO:teuthology.orchestra.run.smithi146.stdout:Reading state information... 2024-03-11T20:37:21.841 INFO:teuthology.orchestra.run.smithi023.stdout:Building dependency tree... 2024-03-11T20:37:21.841 INFO:teuthology.orchestra.run.smithi023.stdout:Reading state information... 2024-03-11T20:37:21.899 INFO:teuthology.orchestra.run.smithi146.stdout:The following packages were automatically installed and are no longer required: 2024-03-11T20:37:21.899 INFO:teuthology.orchestra.run.smithi146.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-03-11T20:37:21.899 INFO:teuthology.orchestra.run.smithi146.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-03-11T20:37:21.899 INFO:teuthology.orchestra.run.smithi146.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-03-11T20:37:21.900 INFO:teuthology.orchestra.run.smithi146.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-03-11T20:37:21.900 INFO:teuthology.orchestra.run.smithi146.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-03-11T20:37:21.900 INFO:teuthology.orchestra.run.smithi146.stdout:Use 'sudo apt autoremove' to remove them. 2024-03-11T20:37:21.900 INFO:teuthology.orchestra.run.smithi146.stdout:The following additional packages will be installed: 2024-03-11T20:37:21.900 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-03-11T20:37:21.900 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-03-11T20:37:21.902 INFO:teuthology.orchestra.run.smithi146.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-03-11T20:37:21.902 INFO:teuthology.orchestra.run.smithi146.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-03-11T20:37:21.902 INFO:teuthology.orchestra.run.smithi146.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-03-11T20:37:21.902 INFO:teuthology.orchestra.run.smithi146.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-03-11T20:37:21.902 INFO:teuthology.orchestra.run.smithi146.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-03-11T20:37:21.902 INFO:teuthology.orchestra.run.smithi146.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-03-11T20:37:21.902 INFO:teuthology.orchestra.run.smithi146.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-03-11T20:37:21.902 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-03-11T20:37:21.903 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-03-11T20:37:21.903 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-03-11T20:37:21.903 INFO:teuthology.orchestra.run.smithi146.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-03-11T20:37:21.903 INFO:teuthology.orchestra.run.smithi146.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-03-11T20:37:21.903 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-03-11T20:37:21.903 INFO:teuthology.orchestra.run.smithi146.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-03-11T20:37:21.903 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-03-11T20:37:21.903 INFO:teuthology.orchestra.run.smithi146.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-03-11T20:37:21.903 INFO:teuthology.orchestra.run.smithi146.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-03-11T20:37:21.903 INFO:teuthology.orchestra.run.smithi146.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-03-11T20:37:21.903 INFO:teuthology.orchestra.run.smithi146.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-03-11T20:37:21.903 INFO:teuthology.orchestra.run.smithi146.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-03-11T20:37:21.903 INFO:teuthology.orchestra.run.smithi146.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-03-11T20:37:21.904 INFO:teuthology.orchestra.run.smithi146.stdout: xmlstarlet zip 2024-03-11T20:37:21.904 INFO:teuthology.orchestra.run.smithi146.stdout:Suggested packages: 2024-03-11T20:37:21.904 INFO:teuthology.orchestra.run.smithi146.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-03-11T20:37:21.904 INFO:teuthology.orchestra.run.smithi146.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-03-11T20:37:21.904 INFO:teuthology.orchestra.run.smithi146.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-03-11T20:37:21.904 INFO:teuthology.orchestra.run.smithi146.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-03-11T20:37:21.904 INFO:teuthology.orchestra.run.smithi146.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-03-11T20:37:21.904 INFO:teuthology.orchestra.run.smithi146.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-03-11T20:37:21.904 INFO:teuthology.orchestra.run.smithi146.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-03-11T20:37:21.904 INFO:teuthology.orchestra.run.smithi146.stdout:Recommended packages: 2024-03-11T20:37:21.905 INFO:teuthology.orchestra.run.smithi146.stdout: btrfs-tools 2024-03-11T20:37:22.007 INFO:teuthology.orchestra.run.smithi146.stdout:The following NEW packages will be installed: 2024-03-11T20:37:22.007 INFO:teuthology.orchestra.run.smithi146.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-03-11T20:37:22.007 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-03-11T20:37:22.007 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-03-11T20:37:22.007 INFO:teuthology.orchestra.run.smithi146.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-03-11T20:37:22.007 INFO:teuthology.orchestra.run.smithi146.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-03-11T20:37:22.007 INFO:teuthology.orchestra.run.smithi146.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-03-11T20:37:22.007 INFO:teuthology.orchestra.run.smithi146.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-03-11T20:37:22.009 INFO:teuthology.orchestra.run.smithi146.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-03-11T20:37:22.009 INFO:teuthology.orchestra.run.smithi146.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-03-11T20:37:22.009 INFO:teuthology.orchestra.run.smithi146.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-03-11T20:37:22.009 INFO:teuthology.orchestra.run.smithi146.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-03-11T20:37:22.009 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-03-11T20:37:22.009 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-03-11T20:37:22.009 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-03-11T20:37:22.009 INFO:teuthology.orchestra.run.smithi146.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-03-11T20:37:22.009 INFO:teuthology.orchestra.run.smithi146.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-03-11T20:37:22.009 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-03-11T20:37:22.009 INFO:teuthology.orchestra.run.smithi146.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-03-11T20:37:22.009 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-03-11T20:37:22.010 INFO:teuthology.orchestra.run.smithi146.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-03-11T20:37:22.010 INFO:teuthology.orchestra.run.smithi146.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-03-11T20:37:22.010 INFO:teuthology.orchestra.run.smithi146.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-03-11T20:37:22.010 INFO:teuthology.orchestra.run.smithi146.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-03-11T20:37:22.010 INFO:teuthology.orchestra.run.smithi146.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-03-11T20:37:22.010 INFO:teuthology.orchestra.run.smithi146.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-03-11T20:37:22.010 INFO:teuthology.orchestra.run.smithi146.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-03-11T20:37:22.028 INFO:teuthology.orchestra.run.smithi023.stdout:The following packages were automatically installed and are no longer required: 2024-03-11T20:37:22.028 INFO:teuthology.orchestra.run.smithi023.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-03-11T20:37:22.028 INFO:teuthology.orchestra.run.smithi023.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-03-11T20:37:22.028 INFO:teuthology.orchestra.run.smithi023.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-03-11T20:37:22.028 INFO:teuthology.orchestra.run.smithi023.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-03-11T20:37:22.029 INFO:teuthology.orchestra.run.smithi023.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-03-11T20:37:22.029 INFO:teuthology.orchestra.run.smithi023.stdout:Use 'sudo apt autoremove' to remove them. 2024-03-11T20:37:22.029 INFO:teuthology.orchestra.run.smithi023.stdout:The following additional packages will be installed: 2024-03-11T20:37:22.029 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-03-11T20:37:22.029 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-03-11T20:37:22.031 INFO:teuthology.orchestra.run.smithi023.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-03-11T20:37:22.031 INFO:teuthology.orchestra.run.smithi023.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-03-11T20:37:22.031 INFO:teuthology.orchestra.run.smithi023.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-03-11T20:37:22.031 INFO:teuthology.orchestra.run.smithi023.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-03-11T20:37:22.031 INFO:teuthology.orchestra.run.smithi023.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-03-11T20:37:22.031 INFO:teuthology.orchestra.run.smithi023.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-03-11T20:37:22.031 INFO:teuthology.orchestra.run.smithi023.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-03-11T20:37:22.031 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-03-11T20:37:22.031 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-03-11T20:37:22.031 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-03-11T20:37:22.032 INFO:teuthology.orchestra.run.smithi023.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-03-11T20:37:22.032 INFO:teuthology.orchestra.run.smithi023.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-03-11T20:37:22.032 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-03-11T20:37:22.032 INFO:teuthology.orchestra.run.smithi023.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-03-11T20:37:22.032 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-03-11T20:37:22.032 INFO:teuthology.orchestra.run.smithi023.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-03-11T20:37:22.032 INFO:teuthology.orchestra.run.smithi023.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-03-11T20:37:22.032 INFO:teuthology.orchestra.run.smithi023.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-03-11T20:37:22.032 INFO:teuthology.orchestra.run.smithi023.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-03-11T20:37:22.032 INFO:teuthology.orchestra.run.smithi023.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-03-11T20:37:22.032 INFO:teuthology.orchestra.run.smithi023.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-03-11T20:37:22.032 INFO:teuthology.orchestra.run.smithi023.stdout: xmlstarlet zip 2024-03-11T20:37:22.033 INFO:teuthology.orchestra.run.smithi023.stdout:Suggested packages: 2024-03-11T20:37:22.033 INFO:teuthology.orchestra.run.smithi023.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-03-11T20:37:22.033 INFO:teuthology.orchestra.run.smithi023.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-03-11T20:37:22.033 INFO:teuthology.orchestra.run.smithi023.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-03-11T20:37:22.033 INFO:teuthology.orchestra.run.smithi023.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-03-11T20:37:22.033 INFO:teuthology.orchestra.run.smithi023.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-03-11T20:37:22.033 INFO:teuthology.orchestra.run.smithi023.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-03-11T20:37:22.033 INFO:teuthology.orchestra.run.smithi023.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-03-11T20:37:22.033 INFO:teuthology.orchestra.run.smithi023.stdout:Recommended packages: 2024-03-11T20:37:22.033 INFO:teuthology.orchestra.run.smithi023.stdout: btrfs-tools 2024-03-11T20:37:22.139 INFO:teuthology.orchestra.run.smithi023.stdout:The following NEW packages will be installed: 2024-03-11T20:37:22.139 INFO:teuthology.orchestra.run.smithi023.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-03-11T20:37:22.139 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-03-11T20:37:22.139 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-03-11T20:37:22.139 INFO:teuthology.orchestra.run.smithi023.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-03-11T20:37:22.139 INFO:teuthology.orchestra.run.smithi023.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-03-11T20:37:22.139 INFO:teuthology.orchestra.run.smithi023.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-03-11T20:37:22.140 INFO:teuthology.orchestra.run.smithi023.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-03-11T20:37:22.140 INFO:teuthology.orchestra.run.smithi023.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-03-11T20:37:22.141 INFO:teuthology.orchestra.run.smithi023.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-03-11T20:37:22.141 INFO:teuthology.orchestra.run.smithi023.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-03-11T20:37:22.141 INFO:teuthology.orchestra.run.smithi023.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-03-11T20:37:22.141 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-03-11T20:37:22.141 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-03-11T20:37:22.141 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-03-11T20:37:22.141 INFO:teuthology.orchestra.run.smithi023.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-03-11T20:37:22.141 INFO:teuthology.orchestra.run.smithi023.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-03-11T20:37:22.141 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-03-11T20:37:22.141 INFO:teuthology.orchestra.run.smithi023.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-03-11T20:37:22.141 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-03-11T20:37:22.141 INFO:teuthology.orchestra.run.smithi023.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-03-11T20:37:22.141 INFO:teuthology.orchestra.run.smithi023.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-03-11T20:37:22.141 INFO:teuthology.orchestra.run.smithi023.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-03-11T20:37:22.141 INFO:teuthology.orchestra.run.smithi023.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-03-11T20:37:22.142 INFO:teuthology.orchestra.run.smithi023.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-03-11T20:37:22.142 INFO:teuthology.orchestra.run.smithi023.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-03-11T20:37:22.142 INFO:teuthology.orchestra.run.smithi023.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-03-11T20:37:22.153 INFO:teuthology.orchestra.run.smithi146.stdout:0 upgraded, 118 newly installed, 0 to remove and 275 not upgraded. 2024-03-11T20:37:22.153 INFO:teuthology.orchestra.run.smithi146.stdout:Need to get 176 MB of archives. 2024-03-11T20:37:22.153 INFO:teuthology.orchestra.run.smithi146.stdout:After this operation, 731 MB of additional disk space will be used. 2024-03-11T20:37:22.153 INFO:teuthology.orchestra.run.smithi146.stdout:Get:1 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-1979-g2b0101eb-1jammy [3,546 kB] 2024-03-11T20:37:22.197 INFO:teuthology.orchestra.run.smithi146.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-03-11T20:37:22.206 INFO:teuthology.orchestra.run.smithi023.stdout:0 upgraded, 118 newly installed, 0 to remove and 275 not upgraded. 2024-03-11T20:37:22.206 INFO:teuthology.orchestra.run.smithi023.stdout:Need to get 176 MB of archives. 2024-03-11T20:37:22.206 INFO:teuthology.orchestra.run.smithi023.stdout:After this operation, 731 MB of additional disk space will be used. 2024-03-11T20:37:22.206 INFO:teuthology.orchestra.run.smithi023.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-03-11T20:37:22.283 INFO:teuthology.orchestra.run.smithi023.stdout:Get:2 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-1979-g2b0101eb-1jammy [3,546 kB] 2024-03-11T20:37:22.306 INFO:teuthology.orchestra.run.smithi023.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-03-11T20:37:22.309 INFO:teuthology.orchestra.run.smithi023.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-03-11T20:37:22.329 INFO:teuthology.orchestra.run.smithi023.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-03-11T20:37:22.398 INFO:teuthology.orchestra.run.smithi023.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-03-11T20:37:22.402 INFO:teuthology.orchestra.run.smithi023.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-03-11T20:37:22.415 INFO:teuthology.orchestra.run.smithi023.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-03-11T20:37:22.419 INFO:teuthology.orchestra.run.smithi023.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-03-11T20:37:22.419 INFO:teuthology.orchestra.run.smithi023.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-03-11T20:37:22.420 INFO:teuthology.orchestra.run.smithi023.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-03-11T20:37:22.428 INFO:teuthology.orchestra.run.smithi023.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-03-11T20:37:22.430 INFO:teuthology.orchestra.run.smithi023.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-03-11T20:37:22.433 INFO:teuthology.orchestra.run.smithi023.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-03-11T20:37:22.434 INFO:teuthology.orchestra.run.smithi023.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-03-11T20:37:22.443 INFO:teuthology.orchestra.run.smithi023.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-03-11T20:37:22.446 INFO:teuthology.orchestra.run.smithi023.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-03-11T20:37:22.448 INFO:teuthology.orchestra.run.smithi023.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-03-11T20:37:22.450 INFO:teuthology.orchestra.run.smithi023.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-03-11T20:37:22.451 INFO:teuthology.orchestra.run.smithi023.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-03-11T20:37:22.463 INFO:teuthology.orchestra.run.smithi023.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-03-11T20:37:22.464 INFO:teuthology.orchestra.run.smithi023.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-03-11T20:37:22.464 INFO:teuthology.orchestra.run.smithi023.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-03-11T20:37:22.465 INFO:teuthology.orchestra.run.smithi023.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-03-11T20:37:22.465 INFO:teuthology.orchestra.run.smithi023.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-03-11T20:37:22.482 INFO:teuthology.orchestra.run.smithi023.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-03-11T20:37:22.483 INFO:teuthology.orchestra.run.smithi023.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-03-11T20:37:22.483 INFO:teuthology.orchestra.run.smithi023.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-03-11T20:37:22.486 INFO:teuthology.orchestra.run.smithi023.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-03-11T20:37:22.486 INFO:teuthology.orchestra.run.smithi023.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-03-11T20:37:22.502 INFO:teuthology.orchestra.run.smithi023.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-03-11T20:37:22.502 INFO:teuthology.orchestra.run.smithi023.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-03-11T20:37:22.503 INFO:teuthology.orchestra.run.smithi023.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-03-11T20:37:22.504 INFO:teuthology.orchestra.run.smithi023.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-03-11T20:37:22.504 INFO:teuthology.orchestra.run.smithi023.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-03-11T20:37:22.521 INFO:teuthology.orchestra.run.smithi023.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-03-11T20:37:22.522 INFO:teuthology.orchestra.run.smithi023.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-03-11T20:37:22.523 INFO:teuthology.orchestra.run.smithi023.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-03-11T20:37:22.529 INFO:teuthology.orchestra.run.smithi023.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-03-11T20:37:22.529 INFO:teuthology.orchestra.run.smithi023.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-03-11T20:37:22.541 INFO:teuthology.orchestra.run.smithi023.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-03-11T20:37:22.542 INFO:teuthology.orchestra.run.smithi023.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-03-11T20:37:22.544 INFO:teuthology.orchestra.run.smithi023.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-03-11T20:37:22.546 INFO:teuthology.orchestra.run.smithi023.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-03-11T20:37:22.546 INFO:teuthology.orchestra.run.smithi023.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-03-11T20:37:22.561 INFO:teuthology.orchestra.run.smithi023.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-03-11T20:37:22.562 INFO:teuthology.orchestra.run.smithi023.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-03-11T20:37:22.564 INFO:teuthology.orchestra.run.smithi023.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-03-11T20:37:22.565 INFO:teuthology.orchestra.run.smithi023.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-03-11T20:37:22.565 INFO:teuthology.orchestra.run.smithi023.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-03-11T20:37:22.580 INFO:teuthology.orchestra.run.smithi023.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-03-11T20:37:22.618 INFO:teuthology.orchestra.run.smithi146.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-03-11T20:37:22.631 INFO:teuthology.orchestra.run.smithi146.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-03-11T20:37:22.637 INFO:teuthology.orchestra.run.smithi023.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-03-11T20:37:22.638 INFO:teuthology.orchestra.run.smithi023.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-03-11T20:37:22.640 INFO:teuthology.orchestra.run.smithi023.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-03-11T20:37:22.640 INFO:teuthology.orchestra.run.smithi023.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-03-11T20:37:22.641 INFO:teuthology.orchestra.run.smithi023.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-03-11T20:37:22.660 INFO:teuthology.orchestra.run.smithi023.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-03-11T20:37:22.662 INFO:teuthology.orchestra.run.smithi023.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-03-11T20:37:22.662 INFO:teuthology.orchestra.run.smithi023.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-03-11T20:37:22.677 INFO:teuthology.orchestra.run.smithi023.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-03-11T20:37:22.677 INFO:teuthology.orchestra.run.smithi023.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-03-11T20:37:22.678 INFO:teuthology.orchestra.run.smithi023.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-03-11T20:37:22.678 INFO:teuthology.orchestra.run.smithi023.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-03-11T20:37:22.679 INFO:teuthology.orchestra.run.smithi023.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-03-11T20:37:22.679 INFO:teuthology.orchestra.run.smithi023.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-03-11T20:37:22.697 INFO:teuthology.orchestra.run.smithi023.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-03-11T20:37:22.698 INFO:teuthology.orchestra.run.smithi023.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-03-11T20:37:22.702 INFO:teuthology.orchestra.run.smithi023.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-03-11T20:37:22.703 INFO:teuthology.orchestra.run.smithi023.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-03-11T20:37:22.705 INFO:teuthology.orchestra.run.smithi023.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-03-11T20:37:22.705 INFO:teuthology.orchestra.run.smithi023.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-03-11T20:37:22.712 INFO:teuthology.orchestra.run.smithi146.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-03-11T20:37:22.715 INFO:teuthology.orchestra.run.smithi146.stdout:Get:6 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-1979-g2b0101eb-1jammy [3,220 kB] 2024-03-11T20:37:22.717 INFO:teuthology.orchestra.run.smithi023.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-03-11T20:37:22.720 INFO:teuthology.orchestra.run.smithi023.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-03-11T20:37:22.721 INFO:teuthology.orchestra.run.smithi023.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-03-11T20:37:22.721 INFO:teuthology.orchestra.run.smithi023.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-03-11T20:37:22.737 INFO:teuthology.orchestra.run.smithi023.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-03-11T20:37:22.737 INFO:teuthology.orchestra.run.smithi023.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-03-11T20:37:22.740 INFO:teuthology.orchestra.run.smithi023.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-03-11T20:37:22.741 INFO:teuthology.orchestra.run.smithi023.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-03-11T20:37:22.741 INFO:teuthology.orchestra.run.smithi023.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-03-11T20:37:22.741 INFO:teuthology.orchestra.run.smithi023.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-03-11T20:37:22.756 INFO:teuthology.orchestra.run.smithi023.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-03-11T20:37:22.757 INFO:teuthology.orchestra.run.smithi023.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-03-11T20:37:22.764 INFO:teuthology.orchestra.run.smithi023.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-03-11T20:37:22.764 INFO:teuthology.orchestra.run.smithi023.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-03-11T20:37:22.775 INFO:teuthology.orchestra.run.smithi023.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-03-11T20:37:22.777 INFO:teuthology.orchestra.run.smithi023.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-03-11T20:37:22.778 INFO:teuthology.orchestra.run.smithi023.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-03-11T20:37:22.834 INFO:teuthology.orchestra.run.smithi023.stdout:Get:89 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-1979-g2b0101eb-1jammy [3,220 kB] 2024-03-11T20:37:22.926 INFO:teuthology.orchestra.run.smithi146.stdout:Get:7 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-1979-g2b0101eb-1jammy [711 kB] 2024-03-11T20:37:22.941 INFO:teuthology.orchestra.run.smithi146.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-03-11T20:37:22.953 INFO:teuthology.orchestra.run.smithi146.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-03-11T20:37:22.970 INFO:teuthology.orchestra.run.smithi146.stdout:Get:10 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-1979-g2b0101eb-1jammy [356 kB] 2024-03-11T20:37:22.983 INFO:teuthology.orchestra.run.smithi146.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-03-11T20:37:22.993 INFO:teuthology.orchestra.run.smithi146.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-03-11T20:37:22.993 INFO:teuthology.orchestra.run.smithi146.stdout:Get:13 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-1979-g2b0101eb-1jammy [32.8 kB] 2024-03-11T20:37:22.994 INFO:teuthology.orchestra.run.smithi146.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-03-11T20:37:22.994 INFO:teuthology.orchestra.run.smithi146.stdout:Get:15 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-1979-g2b0101eb-1jammy [180 kB] 2024-03-11T20:37:22.995 INFO:teuthology.orchestra.run.smithi146.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-03-11T20:37:23.000 INFO:teuthology.orchestra.run.smithi146.stdout:Get:17 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-1979-g2b0101eb-1jammy [64.6 kB] 2024-03-11T20:37:23.002 INFO:teuthology.orchestra.run.smithi146.stdout:Get:18 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-1979-g2b0101eb-1jammy [330 kB] 2024-03-11T20:37:23.014 INFO:teuthology.orchestra.run.smithi146.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-03-11T20:37:23.026 INFO:teuthology.orchestra.run.smithi146.stdout:Get:20 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-1979-g2b0101eb-1jammy [6,613 kB] 2024-03-11T20:37:23.064 INFO:teuthology.orchestra.run.smithi023.stdout:Get:90 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-1979-g2b0101eb-1jammy [711 kB] 2024-03-11T20:37:23.103 INFO:teuthology.orchestra.run.smithi023.stdout:Get:91 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-1979-g2b0101eb-1jammy [356 kB] 2024-03-11T20:37:23.132 INFO:teuthology.orchestra.run.smithi023.stdout:Get:92 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-1979-g2b0101eb-1jammy [32.8 kB] 2024-03-11T20:37:23.134 INFO:teuthology.orchestra.run.smithi023.stdout:Get:93 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-1979-g2b0101eb-1jammy [180 kB] 2024-03-11T20:37:23.157 INFO:teuthology.orchestra.run.smithi023.stdout:Get:94 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-1979-g2b0101eb-1jammy [64.6 kB] 2024-03-11T20:37:23.160 INFO:teuthology.orchestra.run.smithi023.stdout:Get:95 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-1979-g2b0101eb-1jammy [330 kB] 2024-03-11T20:37:23.170 INFO:teuthology.orchestra.run.smithi023.stdout:Get:96 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-1979-g2b0101eb-1jammy [6,613 kB] 2024-03-11T20:37:23.181 INFO:teuthology.orchestra.run.smithi146.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-03-11T20:37:23.428 INFO:teuthology.orchestra.run.smithi146.stdout:Get:22 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-1979-g2b0101eb-1jammy [112 kB] 2024-03-11T20:37:23.433 INFO:teuthology.orchestra.run.smithi146.stdout:Get:23 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-1979-g2b0101eb-1jammy [467 kB] 2024-03-11T20:37:23.465 INFO:teuthology.orchestra.run.smithi146.stdout:Get:24 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-1979-g2b0101eb-1jammy [24.2 MB] 2024-03-11T20:37:23.581 INFO:teuthology.orchestra.run.smithi146.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-03-11T20:37:23.616 INFO:teuthology.orchestra.run.smithi146.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-03-11T20:37:23.618 INFO:teuthology.orchestra.run.smithi146.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-03-11T20:37:23.665 INFO:teuthology.orchestra.run.smithi023.stdout:Get:97 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-1979-g2b0101eb-1jammy [112 kB] 2024-03-11T20:37:23.667 INFO:teuthology.orchestra.run.smithi023.stdout:Get:98 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-1979-g2b0101eb-1jammy [467 kB] 2024-03-11T20:37:23.698 INFO:teuthology.orchestra.run.smithi146.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-03-11T20:37:23.715 INFO:teuthology.orchestra.run.smithi023.stdout:Get:99 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-1979-g2b0101eb-1jammy [24.2 MB] 2024-03-11T20:37:23.749 INFO:teuthology.orchestra.run.smithi146.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-03-11T20:37:23.771 INFO:teuthology.orchestra.run.smithi146.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-03-11T20:37:23.777 INFO:teuthology.orchestra.run.smithi146.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-03-11T20:37:23.778 INFO:teuthology.orchestra.run.smithi146.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-03-11T20:37:23.789 INFO:teuthology.orchestra.run.smithi146.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-03-11T20:37:23.790 INFO:teuthology.orchestra.run.smithi146.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-03-11T20:37:23.956 INFO:teuthology.orchestra.run.smithi146.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-03-11T20:37:24.039 INFO:teuthology.orchestra.run.smithi146.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-03-11T20:37:24.067 INFO:teuthology.orchestra.run.smithi146.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-03-11T20:37:24.084 INFO:teuthology.orchestra.run.smithi146.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-03-11T20:37:24.128 INFO:teuthology.orchestra.run.smithi146.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-03-11T20:37:24.340 INFO:teuthology.orchestra.run.smithi146.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-03-11T20:37:24.354 INFO:teuthology.orchestra.run.smithi146.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-03-11T20:37:24.361 INFO:teuthology.orchestra.run.smithi146.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-03-11T20:37:24.366 INFO:teuthology.orchestra.run.smithi146.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-03-11T20:37:24.387 INFO:teuthology.orchestra.run.smithi146.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-03-11T20:37:24.391 INFO:teuthology.orchestra.run.smithi146.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-03-11T20:37:24.558 INFO:teuthology.orchestra.run.smithi146.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-03-11T20:37:24.874 INFO:teuthology.orchestra.run.smithi146.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-03-11T20:37:24.964 INFO:teuthology.orchestra.run.smithi146.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-03-11T20:37:24.964 INFO:teuthology.orchestra.run.smithi146.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-03-11T20:37:24.966 INFO:teuthology.orchestra.run.smithi146.stdout:Get:50 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-1979-g2b0101eb-1jammy [5,081 kB] 2024-03-11T20:37:25.149 INFO:teuthology.orchestra.run.smithi146.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-03-11T20:37:25.150 INFO:teuthology.orchestra.run.smithi146.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-03-11T20:37:25.156 INFO:teuthology.orchestra.run.smithi146.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-03-11T20:37:25.162 INFO:teuthology.orchestra.run.smithi146.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-03-11T20:37:25.177 INFO:teuthology.orchestra.run.smithi146.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-03-11T20:37:25.200 INFO:teuthology.orchestra.run.smithi146.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-03-11T20:37:25.206 INFO:teuthology.orchestra.run.smithi146.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-03-11T20:37:25.210 INFO:teuthology.orchestra.run.smithi146.stdout:Get:58 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-1979-g2b0101eb-1jammy [243 kB] 2024-03-11T20:37:25.217 INFO:teuthology.orchestra.run.smithi146.stdout:Get:59 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-1979-g2b0101eb-1jammy [124 kB] 2024-03-11T20:37:25.229 INFO:teuthology.orchestra.run.smithi146.stdout:Get:60 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-1979-g2b0101eb-1jammy [1,522 kB] 2024-03-11T20:37:25.330 INFO:teuthology.orchestra.run.smithi146.stdout:Get:61 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-1979-g2b0101eb-1jammy [6,247 kB] 2024-03-11T20:37:25.381 INFO:teuthology.orchestra.run.smithi146.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-03-11T20:37:25.547 INFO:teuthology.orchestra.run.smithi023.stdout:Get:100 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-1979-g2b0101eb-1jammy [5,081 kB] 2024-03-11T20:37:25.635 INFO:teuthology.orchestra.run.smithi146.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-03-11T20:37:25.686 INFO:teuthology.orchestra.run.smithi146.stdout:Get:64 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-1979-g2b0101eb-1jammy [22.8 MB] 2024-03-11T20:37:25.812 INFO:teuthology.orchestra.run.smithi146.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-03-11T20:37:25.826 INFO:teuthology.orchestra.run.smithi146.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-03-11T20:37:25.846 INFO:teuthology.orchestra.run.smithi146.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-03-11T20:37:25.857 INFO:teuthology.orchestra.run.smithi023.stdout:Get:101 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-1979-g2b0101eb-1jammy [243 kB] 2024-03-11T20:37:25.864 INFO:teuthology.orchestra.run.smithi023.stdout:Get:102 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-1979-g2b0101eb-1jammy [124 kB] 2024-03-11T20:37:25.868 INFO:teuthology.orchestra.run.smithi023.stdout:Get:103 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-1979-g2b0101eb-1jammy [1,522 kB] 2024-03-11T20:37:25.910 INFO:teuthology.orchestra.run.smithi146.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-03-11T20:37:25.996 INFO:teuthology.orchestra.run.smithi023.stdout:Get:104 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-1979-g2b0101eb-1jammy [6,247 kB] 2024-03-11T20:37:26.199 INFO:teuthology.orchestra.run.smithi146.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-03-11T20:37:26.201 INFO:teuthology.orchestra.run.smithi146.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-03-11T20:37:26.204 INFO:teuthology.orchestra.run.smithi146.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-03-11T20:37:26.205 INFO:teuthology.orchestra.run.smithi146.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-03-11T20:37:26.207 INFO:teuthology.orchestra.run.smithi146.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-03-11T20:37:26.432 INFO:teuthology.orchestra.run.smithi146.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-03-11T20:37:26.439 INFO:teuthology.orchestra.run.smithi023.stdout:Get:105 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-1979-g2b0101eb-1jammy [22.8 MB] 2024-03-11T20:37:26.862 INFO:teuthology.orchestra.run.smithi146.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-03-11T20:37:26.868 INFO:teuthology.orchestra.run.smithi146.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-03-11T20:37:27.161 INFO:teuthology.orchestra.run.smithi146.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-03-11T20:37:27.161 INFO:teuthology.orchestra.run.smithi146.stdout:Get:78 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-1979-g2b0101eb-1jammy [14.0 kB] 2024-03-11T20:37:27.161 INFO:teuthology.orchestra.run.smithi146.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-03-11T20:37:27.161 INFO:teuthology.orchestra.run.smithi146.stdout:Get:80 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-1979-g2b0101eb-1jammy [867 kB] 2024-03-11T20:37:27.163 INFO:teuthology.orchestra.run.smithi146.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-03-11T20:37:27.164 INFO:teuthology.orchestra.run.smithi146.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-03-11T20:37:27.165 INFO:teuthology.orchestra.run.smithi146.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-03-11T20:37:27.166 INFO:teuthology.orchestra.run.smithi146.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-03-11T20:37:27.167 INFO:teuthology.orchestra.run.smithi146.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-03-11T20:37:27.169 INFO:teuthology.orchestra.run.smithi146.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-03-11T20:37:27.178 INFO:teuthology.orchestra.run.smithi146.stdout:Get:87 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-1979-g2b0101eb-1jammy [2,192 kB] 2024-03-11T20:37:27.232 INFO:teuthology.orchestra.run.smithi146.stdout:Get:88 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-1979-g2b0101eb-1jammy [777 kB] 2024-03-11T20:37:27.285 INFO:teuthology.orchestra.run.smithi146.stdout:Get:89 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-1979-g2b0101eb-1jammy [145 kB] 2024-03-11T20:37:27.290 INFO:teuthology.orchestra.run.smithi146.stdout:Get:90 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-1979-g2b0101eb-1jammy [1,934 kB] 2024-03-11T20:37:27.350 INFO:teuthology.orchestra.run.smithi146.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-03-11T20:37:27.415 INFO:teuthology.orchestra.run.smithi146.stdout:Get:92 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-1979-g2b0101eb-1jammy [8,625 kB] 2024-03-11T20:37:27.756 INFO:teuthology.orchestra.run.smithi146.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-03-11T20:37:27.802 INFO:teuthology.orchestra.run.smithi146.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-03-11T20:37:27.837 INFO:teuthology.orchestra.run.smithi146.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-03-11T20:37:27.918 INFO:teuthology.orchestra.run.smithi146.stdout:Get:96 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-1979-g2b0101eb-1jammy [14.2 kB] 2024-03-11T20:37:27.919 INFO:teuthology.orchestra.run.smithi146.stdout:Get:97 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-1979-g2b0101eb-1jammy [50.0 MB] 2024-03-11T20:37:27.933 INFO:teuthology.orchestra.run.smithi146.stdout:Get:98 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-03-11T20:37:27.973 INFO:teuthology.orchestra.run.smithi146.stdout:Get:99 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-03-11T20:37:27.985 INFO:teuthology.orchestra.run.smithi146.stdout:Get:100 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-03-11T20:37:27.990 INFO:teuthology.orchestra.run.smithi146.stdout:Get:101 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-03-11T20:37:28.037 INFO:teuthology.orchestra.run.smithi146.stdout:Get:102 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-03-11T20:37:28.041 INFO:teuthology.orchestra.run.smithi146.stdout:Get:103 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-03-11T20:37:28.062 INFO:teuthology.orchestra.run.smithi146.stdout:Get:104 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-03-11T20:37:28.175 INFO:teuthology.orchestra.run.smithi023.stdout:Get:106 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-1979-g2b0101eb-1jammy [14.0 kB] 2024-03-11T20:37:28.176 INFO:teuthology.orchestra.run.smithi023.stdout:Get:107 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-1979-g2b0101eb-1jammy [867 kB] 2024-03-11T20:37:28.193 INFO:teuthology.orchestra.run.smithi023.stdout:Get:108 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-1979-g2b0101eb-1jammy [2,192 kB] 2024-03-11T20:37:28.226 INFO:teuthology.orchestra.run.smithi146.stdout:Get:105 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-03-11T20:37:28.307 INFO:teuthology.orchestra.run.smithi023.stdout:Get:109 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-1979-g2b0101eb-1jammy [777 kB] 2024-03-11T20:37:28.347 INFO:teuthology.orchestra.run.smithi023.stdout:Get:110 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-1979-g2b0101eb-1jammy [145 kB] 2024-03-11T20:37:28.367 INFO:teuthology.orchestra.run.smithi023.stdout:Get:111 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-1979-g2b0101eb-1jammy [1,934 kB] 2024-03-11T20:37:28.477 INFO:teuthology.orchestra.run.smithi023.stdout:Get:112 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-1979-g2b0101eb-1jammy [8,625 kB] 2024-03-11T20:37:28.489 INFO:teuthology.orchestra.run.smithi146.stdout:Get:106 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-03-11T20:37:28.507 INFO:teuthology.orchestra.run.smithi146.stdout:Get:107 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-03-11T20:37:28.659 INFO:teuthology.orchestra.run.smithi146.stdout:Get:108 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-03-11T20:37:28.687 INFO:teuthology.orchestra.run.smithi146.stdout:Get:109 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-03-11T20:37:28.844 INFO:teuthology.orchestra.run.smithi146.stdout:Get:110 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-03-11T20:37:28.847 INFO:teuthology.orchestra.run.smithi146.stdout:Get:111 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-03-11T20:37:28.849 INFO:teuthology.orchestra.run.smithi146.stdout:Get:112 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-03-11T20:37:28.871 INFO:teuthology.orchestra.run.smithi146.stdout:Get:113 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-03-11T20:37:28.876 INFO:teuthology.orchestra.run.smithi146.stdout:Get:114 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-03-11T20:37:29.015 INFO:teuthology.orchestra.run.smithi023.stdout:Get:113 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-1979-g2b0101eb-1jammy [14.2 kB] 2024-03-11T20:37:29.016 INFO:teuthology.orchestra.run.smithi023.stdout:Get:114 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-1979-g2b0101eb-1jammy [50.0 MB] 2024-03-11T20:37:31.558 INFO:teuthology.orchestra.run.smithi146.stdout:Get:115 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-1979-g2b0101eb-1jammy [121 kB] 2024-03-11T20:37:31.561 INFO:teuthology.orchestra.run.smithi146.stdout:Get:116 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-1979-g2b0101eb-1jammy [38.4 kB] 2024-03-11T20:37:31.561 INFO:teuthology.orchestra.run.smithi146.stdout:Get:117 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-1979-g2b0101eb-1jammy [13.1 MB] 2024-03-11T20:37:32.227 INFO:teuthology.orchestra.run.smithi146.stdout:Get:118 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-1979-g2b0101eb-1jammy [91.7 kB] 2024-03-11T20:37:32.340 INFO:teuthology.orchestra.run.smithi023.stdout:Get:115 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-1979-g2b0101eb-1jammy [121 kB] 2024-03-11T20:37:32.343 INFO:teuthology.orchestra.run.smithi023.stdout:Get:116 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-1979-g2b0101eb-1jammy [38.4 kB] 2024-03-11T20:37:32.344 INFO:teuthology.orchestra.run.smithi023.stdout:Get:117 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-1979-g2b0101eb-1jammy [13.1 MB] 2024-03-11T20:37:32.661 INFO:teuthology.orchestra.run.smithi023.stdout:Get:118 https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-1979-g2b0101eb-1jammy [91.7 kB] 2024-03-11T20:37:32.779 INFO:teuthology.orchestra.run.smithi146.stdout:Fetched 176 MB in 10s (17.3 MB/s) 2024-03-11T20:37:32.937 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-03-11T20:37:33.037 INFO:teuthology.orchestra.run.smithi146.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165997 files and directories currently installed.) 2024-03-11T20:37:33.042 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-03-11T20:37:33.135 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-03-11T20:37:33.241 INFO:teuthology.orchestra.run.smithi023.stdout:Fetched 176 MB in 11s (16.8 MB/s) 2024-03-11T20:37:33.402 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-03-11T20:37:33.455 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-03-11T20:37:33.481 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-03-11T20:37:33.505 INFO:teuthology.orchestra.run.smithi023.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165997 files and directories currently installed.) 2024-03-11T20:37:33.510 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-03-11T20:37:33.522 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-03-11T20:37:33.592 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-03-11T20:37:33.808 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-03-11T20:37:33.834 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-03-11T20:37:33.875 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-03-11T20:37:33.937 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-03-11T20:37:33.964 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-03-11T20:37:34.003 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-03-11T20:37:34.290 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-03-11T20:37:34.316 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-03-11T20:37:34.320 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-03-11T20:37:34.346 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-03-11T20:37:34.348 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-03-11T20:37:34.395 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-11T20:37:34.739 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-03-11T20:37:34.765 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-03-11T20:37:34.785 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-03-11T20:37:34.797 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-11T20:37:34.811 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-03-11T20:37:34.860 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-11T20:37:35.109 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-03-11T20:37:35.135 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-03-11T20:37:35.175 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-11T20:37:35.263 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-03-11T20:37:35.289 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-03-11T20:37:35.329 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-11T20:37:35.570 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-03-11T20:37:35.596 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-03-11T20:37:35.637 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-03-11T20:37:35.660 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-03-11T20:37:35.686 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-03-11T20:37:35.726 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-11T20:37:35.956 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package librados2. 2024-03-11T20:37:35.982 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../007-librados2_19.0.0-1979-g2b0101eb-1jammy_amd64.deb ... 2024-03-11T20:37:36.023 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking librados2 (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:37:36.062 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-03-11T20:37:36.089 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-03-11T20:37:36.129 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-03-11T20:37:36.433 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package librbd1. 2024-03-11T20:37:36.448 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package librados2. 2024-03-11T20:37:36.449 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../008-librbd1_19.0.0-1979-g2b0101eb-1jammy_amd64.deb ... 2024-03-11T20:37:36.475 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../007-librados2_19.0.0-1979-g2b0101eb-1jammy_amd64.deb ... 2024-03-11T20:37:36.483 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking librbd1 (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:37:36.515 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking librados2 (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:37:36.901 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package librbd1. 2024-03-11T20:37:36.903 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package libcephfs2. 2024-03-11T20:37:36.928 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../008-librbd1_19.0.0-1979-g2b0101eb-1jammy_amd64.deb ... 2024-03-11T20:37:36.929 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../009-libcephfs2_19.0.0-1979-g2b0101eb-1jammy_amd64.deb ... 2024-03-11T20:37:36.968 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking librbd1 (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:37:36.969 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking libcephfs2 (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:37:37.247 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-rados. 2024-03-11T20:37:37.263 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../010-python3-rados_19.0.0-1979-g2b0101eb-1jammy_amd64.deb ... 2024-03-11T20:37:37.305 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-rados (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:37:37.404 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package libcephfs2. 2024-03-11T20:37:37.430 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../009-libcephfs2_19.0.0-1979-g2b0101eb-1jammy_amd64.deb ... 2024-03-11T20:37:37.470 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking libcephfs2 (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:37:37.764 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-rados. 2024-03-11T20:37:37.780 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../010-python3-rados_19.0.0-1979-g2b0101eb-1jammy_amd64.deb ... 2024-03-11T20:37:37.823 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-rados (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:37:38.083 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-03-11T20:37:38.099 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.0.0-1979-g2b0101eb-1jammy_all.deb ... 2024-03-11T20:37:38.133 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-ceph-argparse (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:37:38.227 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-03-11T20:37:38.254 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.0.0-1979-g2b0101eb-1jammy_all.deb ... 2024-03-11T20:37:38.369 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-cephfs. 2024-03-11T20:37:38.384 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../012-python3-cephfs_19.0.0-1979-g2b0101eb-1jammy_amd64.deb ... 2024-03-11T20:37:38.421 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-ceph-argparse (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:37:38.427 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-cephfs (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:37:38.670 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-cephfs. 2024-03-11T20:37:38.697 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../012-python3-cephfs_19.0.0-1979-g2b0101eb-1jammy_amd64.deb ... 2024-03-11T20:37:38.736 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-cephfs (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:37:38.747 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-ceph-common. 2024-03-11T20:37:38.773 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../013-python3-ceph-common_19.0.0-1979-g2b0101eb-1jammy_all.deb ... 2024-03-11T20:37:38.813 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-ceph-common (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:37:39.047 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-ceph-common. 2024-03-11T20:37:39.075 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../013-python3-ceph-common_19.0.0-1979-g2b0101eb-1jammy_all.deb ... 2024-03-11T20:37:39.114 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-ceph-common (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:37:39.150 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-wcwidth. 2024-03-11T20:37:39.176 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-03-11T20:37:39.216 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-03-11T20:37:39.409 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-wcwidth. 2024-03-11T20:37:39.436 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-03-11T20:37:39.477 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-prettytable. 2024-03-11T20:37:39.484 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-03-11T20:37:39.504 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-03-11T20:37:39.535 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-03-11T20:37:39.788 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-rbd. 2024-03-11T20:37:39.795 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-prettytable. 2024-03-11T20:37:39.803 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../016-python3-rbd_19.0.0-1979-g2b0101eb-1jammy_amd64.deb ... 2024-03-11T20:37:39.822 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-03-11T20:37:39.846 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-rbd (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:37:39.861 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-03-11T20:37:40.122 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-rbd. 2024-03-11T20:37:40.138 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../016-python3-rbd_19.0.0-1979-g2b0101eb-1jammy_amd64.deb ... 2024-03-11T20:37:40.180 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-rbd (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:37:40.191 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-03-11T20:37:40.217 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-03-11T20:37:40.257 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-03-11T20:37:40.517 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-03-11T20:37:40.535 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-03-11T20:37:40.544 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-03-11T20:37:40.550 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-03-11T20:37:40.575 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-03-11T20:37:40.593 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-03-11T20:37:40.836 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-03-11T20:37:40.859 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-03-11T20:37:40.871 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-03-11T20:37:40.897 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-03-11T20:37:40.902 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-03-11T20:37:40.929 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-03-11T20:37:41.214 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-03-11T20:37:41.241 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-03-11T20:37:41.249 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package lua5.1. 2024-03-11T20:37:41.276 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-03-11T20:37:41.280 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-03-11T20:37:41.315 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-03-11T20:37:41.617 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package lua5.1. 2024-03-11T20:37:41.626 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package lua-any. 2024-03-11T20:37:41.643 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-03-11T20:37:41.653 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-03-11T20:37:41.683 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-03-11T20:37:41.693 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking lua-any (27ubuntu1) ... 2024-03-11T20:37:41.948 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package zip. 2024-03-11T20:37:41.961 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package lua-any. 2024-03-11T20:37:41.963 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-03-11T20:37:41.977 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-03-11T20:37:42.006 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking zip (3.0-12build2) ... 2024-03-11T20:37:42.011 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking lua-any (27ubuntu1) ... 2024-03-11T20:37:42.255 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package zip. 2024-03-11T20:37:42.282 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-03-11T20:37:42.322 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking zip (3.0-12build2) ... 2024-03-11T20:37:42.326 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package unzip. 2024-03-11T20:37:42.352 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-03-11T20:37:42.392 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-03-11T20:37:42.633 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package unzip. 2024-03-11T20:37:42.661 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-03-11T20:37:42.700 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-03-11T20:37:42.813 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package luarocks. 2024-03-11T20:37:42.839 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-03-11T20:37:42.879 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-03-11T20:37:43.071 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package luarocks. 2024-03-11T20:37:43.098 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-03-11T20:37:43.137 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-03-11T20:37:43.237 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package librgw2. 2024-03-11T20:37:43.263 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../025-librgw2_19.0.0-1979-g2b0101eb-1jammy_amd64.deb ... 2024-03-11T20:37:43.297 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking librgw2 (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:37:43.531 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package librgw2. 2024-03-11T20:37:43.558 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../025-librgw2_19.0.0-1979-g2b0101eb-1jammy_amd64.deb ... 2024-03-11T20:37:43.745 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking librgw2 (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:37:43.831 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-rgw. 2024-03-11T20:37:43.857 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../026-python3-rgw_19.0.0-1979-g2b0101eb-1jammy_amd64.deb ... 2024-03-11T20:37:43.897 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-rgw (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:37:44.208 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package liboath0:amd64. 2024-03-11T20:37:44.234 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-03-11T20:37:44.275 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-03-11T20:37:44.280 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-rgw. 2024-03-11T20:37:44.296 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../026-python3-rgw_19.0.0-1979-g2b0101eb-1jammy_amd64.deb ... 2024-03-11T20:37:44.339 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-rgw (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:37:44.569 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package libradosstriper1. 2024-03-11T20:37:44.595 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../028-libradosstriper1_19.0.0-1979-g2b0101eb-1jammy_amd64.deb ... 2024-03-11T20:37:44.636 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking libradosstriper1 (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:37:44.666 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package liboath0:amd64. 2024-03-11T20:37:44.693 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-03-11T20:37:44.733 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-03-11T20:37:44.972 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package ceph-common. 2024-03-11T20:37:44.999 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../029-ceph-common_19.0.0-1979-g2b0101eb-1jammy_amd64.deb ... 2024-03-11T20:37:45.002 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package libradosstriper1. 2024-03-11T20:37:45.030 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../028-libradosstriper1_19.0.0-1979-g2b0101eb-1jammy_amd64.deb ... 2024-03-11T20:37:45.030 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking ceph-common (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:37:45.061 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking libradosstriper1 (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:37:45.497 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package ceph-common. 2024-03-11T20:37:45.525 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../029-ceph-common_19.0.0-1979-g2b0101eb-1jammy_amd64.deb ... 2024-03-11T20:37:45.564 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking ceph-common (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:37:46.336 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package ceph-base. 2024-03-11T20:37:46.362 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../030-ceph-base_19.0.0-1979-g2b0101eb-1jammy_amd64.deb ... 2024-03-11T20:37:46.411 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking ceph-base (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:37:46.878 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package ceph-base. 2024-03-11T20:37:46.904 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../030-ceph-base_19.0.0-1979-g2b0101eb-1jammy_amd64.deb ... 2024-03-11T20:37:46.953 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking ceph-base (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:37:47.015 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-03-11T20:37:47.041 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-03-11T20:37:47.081 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-03-11T20:37:47.384 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-cheroot. 2024-03-11T20:37:47.410 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-03-11T20:37:47.450 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-03-11T20:37:47.507 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-03-11T20:37:47.533 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-03-11T20:37:47.564 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-03-11T20:37:47.753 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-03-11T20:37:47.768 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-03-11T20:37:47.803 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-03-11T20:37:47.817 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-cheroot. 2024-03-11T20:37:47.844 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-03-11T20:37:47.901 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-03-11T20:37:48.112 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-jaraco.text. 2024-03-11T20:37:48.138 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-03-11T20:37:48.212 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-03-11T20:37:48.239 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-03-11T20:37:48.278 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-03-11T20:37:48.281 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-03-11T20:37:48.567 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-03-11T20:37:48.581 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-jaraco.text. 2024-03-11T20:37:48.594 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-03-11T20:37:48.608 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-03-11T20:37:48.633 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-03-11T20:37:48.648 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-03-11T20:37:48.892 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-03-11T20:37:48.908 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-03-11T20:37:48.936 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-tempora. 2024-03-11T20:37:48.942 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-03-11T20:37:48.963 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-03-11T20:37:49.003 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-03-11T20:37:49.212 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-tempora. 2024-03-11T20:37:49.239 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-03-11T20:37:49.272 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-portend. 2024-03-11T20:37:49.278 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-03-11T20:37:49.299 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-03-11T20:37:49.339 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-03-11T20:37:49.581 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-portend. 2024-03-11T20:37:49.608 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-03-11T20:37:49.633 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-03-11T20:37:49.660 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-03-11T20:37:49.664 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-03-11T20:37:49.700 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-03-11T20:37:49.967 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-03-11T20:37:49.994 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-03-11T20:37:50.020 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-cherrypy3. 2024-03-11T20:37:50.033 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-03-11T20:37:50.046 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-03-11T20:37:50.086 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-03-11T20:37:50.287 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-cherrypy3. 2024-03-11T20:37:50.314 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-03-11T20:37:50.345 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-03-11T20:37:50.423 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-natsort. 2024-03-11T20:37:50.449 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-03-11T20:37:50.489 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-03-11T20:37:50.715 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-natsort. 2024-03-11T20:37:50.742 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-03-11T20:37:50.782 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-03-11T20:37:50.801 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-logutils. 2024-03-11T20:37:50.827 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-03-11T20:37:50.867 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-03-11T20:37:51.059 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-logutils. 2024-03-11T20:37:51.087 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-03-11T20:37:51.117 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-03-11T20:37:51.119 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-markupsafe. 2024-03-11T20:37:51.135 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-03-11T20:37:51.169 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-03-11T20:37:51.387 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-markupsafe. 2024-03-11T20:37:51.414 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-03-11T20:37:51.447 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-mako. 2024-03-11T20:37:51.454 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-03-11T20:37:51.474 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-03-11T20:37:51.514 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-03-11T20:37:51.757 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-mako. 2024-03-11T20:37:51.784 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-03-11T20:37:51.823 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-03-11T20:37:51.859 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-simplegeneric. 2024-03-11T20:37:51.886 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-03-11T20:37:51.917 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-03-11T20:37:52.135 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-simplegeneric. 2024-03-11T20:37:52.151 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-03-11T20:37:52.185 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-03-11T20:37:52.203 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-singledispatch. 2024-03-11T20:37:52.230 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-03-11T20:37:52.269 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-03-11T20:37:52.438 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-singledispatch. 2024-03-11T20:37:52.464 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-03-11T20:37:52.505 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-03-11T20:37:52.572 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-webob. 2024-03-11T20:37:52.598 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-03-11T20:37:52.639 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-03-11T20:37:52.807 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-webob. 2024-03-11T20:37:52.834 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-03-11T20:37:52.873 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-03-11T20:37:52.983 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-waitress. 2024-03-11T20:37:53.010 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-03-11T20:37:53.053 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-03-11T20:37:53.218 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-waitress. 2024-03-11T20:37:53.245 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-03-11T20:37:53.288 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-03-11T20:37:53.361 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-tempita. 2024-03-11T20:37:53.388 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-03-11T20:37:53.427 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-03-11T20:37:53.596 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-tempita. 2024-03-11T20:37:53.623 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-03-11T20:37:53.662 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-03-11T20:37:53.705 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-paste. 2024-03-11T20:37:53.721 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-03-11T20:37:53.755 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-03-11T20:37:53.923 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-paste. 2024-03-11T20:37:53.936 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-03-11T20:37:53.965 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-03-11T20:37:54.067 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-03-11T20:37:54.093 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-03-11T20:37:54.133 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-03-11T20:37:54.301 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-03-11T20:37:54.328 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-03-11T20:37:54.368 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-03-11T20:37:54.394 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-pastedeploy. 2024-03-11T20:37:54.421 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-03-11T20:37:54.461 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-03-11T20:37:54.629 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-pastedeploy. 2024-03-11T20:37:54.656 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-03-11T20:37:54.696 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-03-11T20:37:54.982 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-webtest. 2024-03-11T20:37:54.999 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-03-11T20:37:55.040 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-03-11T20:37:55.351 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-pecan. 2024-03-11T20:37:55.378 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-03-11T20:37:55.418 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-03-11T20:37:55.472 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-webtest. 2024-03-11T20:37:55.499 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-03-11T20:37:55.539 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-03-11T20:37:55.781 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-certifi. 2024-03-11T20:37:55.808 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-03-11T20:37:55.847 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-03-11T20:37:55.850 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-pecan. 2024-03-11T20:37:55.876 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-03-11T20:37:55.916 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-03-11T20:37:56.167 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-idna. 2024-03-11T20:37:56.194 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../055-python3-idna_3.3-1_all.deb ... 2024-03-11T20:37:56.225 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-idna (3.3-1) ... 2024-03-11T20:37:56.271 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-certifi. 2024-03-11T20:37:56.298 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-03-11T20:37:56.337 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-03-11T20:37:56.503 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-urllib3. 2024-03-11T20:37:56.530 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-03-11T20:37:56.569 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-03-11T20:37:56.666 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-idna. 2024-03-11T20:37:56.692 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../055-python3-idna_3.3-1_all.deb ... 2024-03-11T20:37:56.732 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-idna (3.3-1) ... 2024-03-11T20:37:56.873 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-requests. 2024-03-11T20:37:56.900 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-03-11T20:37:56.931 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-03-11T20:37:57.034 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-urllib3. 2024-03-11T20:37:57.061 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-03-11T20:37:57.101 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-03-11T20:37:57.200 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-werkzeug. 2024-03-11T20:37:57.227 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-03-11T20:37:57.259 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-03-11T20:37:57.413 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-requests. 2024-03-11T20:37:57.501 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-03-11T20:37:57.501 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-03-11T20:37:57.537 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-03-11T20:37:57.553 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.0.0-1979-g2b0101eb-1jammy_all.deb ... 2024-03-11T20:37:57.595 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking ceph-mgr-modules-core (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:37:57.790 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-werkzeug. 2024-03-11T20:37:57.817 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-03-11T20:37:57.857 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-03-11T20:37:58.000 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-03-11T20:37:58.026 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.0.0-1979-g2b0101eb-1jammy_amd64.deb ... 2024-03-11T20:37:58.066 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:37:58.143 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-03-11T20:37:58.158 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.0.0-1979-g2b0101eb-1jammy_all.deb ... 2024-03-11T20:37:58.201 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking ceph-mgr-modules-core (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:37:58.394 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-bcrypt. 2024-03-11T20:37:58.421 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-03-11T20:37:58.469 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-03-11T20:37:58.722 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-openssl. 2024-03-11T20:37:58.749 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-03-11T20:37:58.764 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-03-11T20:37:58.779 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.0.0-1979-g2b0101eb-1jammy_amd64.deb ... 2024-03-11T20:37:58.780 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-03-11T20:37:58.814 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:37:59.109 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-bcrypt. 2024-03-11T20:37:59.135 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-03-11T20:37:59.175 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-03-11T20:37:59.341 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package ceph-mgr. 2024-03-11T20:37:59.368 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../063-ceph-mgr_19.0.0-1979-g2b0101eb-1jammy_amd64.deb ... 2024-03-11T20:37:59.408 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking ceph-mgr (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:37:59.462 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-openssl. 2024-03-11T20:37:59.487 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-03-11T20:37:59.527 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-03-11T20:37:59.794 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package ceph-mon. 2024-03-11T20:37:59.821 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../064-ceph-mon_19.0.0-1979-g2b0101eb-1jammy_amd64.deb ... 2024-03-11T20:37:59.870 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking ceph-mon (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:37:59.880 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package ceph-mgr. 2024-03-11T20:37:59.907 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../063-ceph-mgr_19.0.0-1979-g2b0101eb-1jammy_amd64.deb ... 2024-03-11T20:37:59.947 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking ceph-mgr (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:38:00.283 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package ceph-mon. 2024-03-11T20:38:00.310 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../064-ceph-mon_19.0.0-1979-g2b0101eb-1jammy_amd64.deb ... 2024-03-11T20:38:00.341 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking ceph-mon (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:38:00.423 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package ceph-osd. 2024-03-11T20:38:00.450 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../065-ceph-osd_19.0.0-1979-g2b0101eb-1jammy_amd64.deb ... 2024-03-11T20:38:00.489 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking ceph-osd (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:38:00.937 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package ceph-osd. 2024-03-11T20:38:00.963 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../065-ceph-osd_19.0.0-1979-g2b0101eb-1jammy_amd64.deb ... 2024-03-11T20:38:01.003 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking ceph-osd (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:38:01.584 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package ceph. 2024-03-11T20:38:01.600 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../066-ceph_19.0.0-1979-g2b0101eb-1jammy_amd64.deb ... 2024-03-11T20:38:01.642 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking ceph (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:38:01.962 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package ceph-fuse. 2024-03-11T20:38:01.989 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../067-ceph-fuse_19.0.0-1979-g2b0101eb-1jammy_amd64.deb ... 2024-03-11T20:38:02.028 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking ceph-fuse (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:38:02.047 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package ceph. 2024-03-11T20:38:02.063 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../066-ceph_19.0.0-1979-g2b0101eb-1jammy_amd64.deb ... 2024-03-11T20:38:02.114 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking ceph (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:38:02.306 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package ceph-mds. 2024-03-11T20:38:02.333 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../068-ceph-mds_19.0.0-1979-g2b0101eb-1jammy_amd64.deb ... 2024-03-11T20:38:02.372 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking ceph-mds (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:38:02.392 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package ceph-fuse. 2024-03-11T20:38:02.419 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../067-ceph-fuse_19.0.0-1979-g2b0101eb-1jammy_amd64.deb ... 2024-03-11T20:38:02.450 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking ceph-fuse (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:38:02.784 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package cephadm. 2024-03-11T20:38:02.803 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package ceph-mds. 2024-03-11T20:38:02.811 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../069-cephadm_19.0.0-1979-g2b0101eb-1jammy_amd64.deb ... 2024-03-11T20:38:02.830 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../068-ceph-mds_19.0.0-1979-g2b0101eb-1jammy_amd64.deb ... 2024-03-11T20:38:02.850 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking cephadm (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:38:02.869 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking ceph-mds (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:38:03.178 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-asyncssh. 2024-03-11T20:38:03.205 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-03-11T20:38:03.231 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package cephadm. 2024-03-11T20:38:03.244 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-03-11T20:38:03.257 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../069-cephadm_19.0.0-1979-g2b0101eb-1jammy_amd64.deb ... 2024-03-11T20:38:03.297 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking cephadm (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:38:03.547 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python-babel-localedata. 2024-03-11T20:38:03.574 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-03-11T20:38:03.613 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-03-11T20:38:03.634 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-asyncssh. 2024-03-11T20:38:03.660 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-03-11T20:38:03.700 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-03-11T20:38:04.019 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python-babel-localedata. 2024-03-11T20:38:04.035 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-03-11T20:38:04.077 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-03-11T20:38:04.643 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-babel. 2024-03-11T20:38:04.671 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-03-11T20:38:04.710 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-03-11T20:38:05.021 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-jinja2. 2024-03-11T20:38:05.048 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-03-11T20:38:05.049 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-babel. 2024-03-11T20:38:05.076 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-03-11T20:38:05.088 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-03-11T20:38:05.115 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-03-11T20:38:05.393 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-jinja2. 2024-03-11T20:38:05.408 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-03-11T20:38:05.443 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-03-11T20:38:05.449 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-03-11T20:38:05.476 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.0.0-1979-g2b0101eb-1jammy_all.deb ... 2024-03-11T20:38:05.516 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking ceph-mgr-cephadm (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:38:05.779 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-03-11T20:38:05.806 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.0.0-1979-g2b0101eb-1jammy_all.deb ... 2024-03-11T20:38:05.837 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking ceph-mgr-cephadm (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:38:05.860 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-repoze.lru. 2024-03-11T20:38:05.888 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-03-11T20:38:05.927 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-03-11T20:38:06.140 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-repoze.lru. 2024-03-11T20:38:06.156 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-03-11T20:38:06.188 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-routes. 2024-03-11T20:38:06.190 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-03-11T20:38:06.204 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-03-11T20:38:06.238 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-03-11T20:38:06.468 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-routes. 2024-03-11T20:38:06.495 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-03-11T20:38:06.524 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-03-11T20:38:06.534 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-03-11T20:38:06.552 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.0.0-1979-g2b0101eb-1jammy_all.deb ... 2024-03-11T20:38:06.582 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking ceph-mgr-dashboard (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:38:06.838 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-03-11T20:38:06.864 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.0.0-1979-g2b0101eb-1jammy_all.deb ... 2024-03-11T20:38:06.904 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking ceph-mgr-dashboard (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:38:06.994 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-03-11T20:38:07.022 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-03-11T20:38:07.077 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-03-11T20:38:07.357 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-03-11T20:38:07.373 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-03-11T20:38:07.415 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-03-11T20:38:07.556 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-joblib. 2024-03-11T20:38:07.584 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-03-11T20:38:07.622 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-03-11T20:38:07.894 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-joblib. 2024-03-11T20:38:07.921 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-03-11T20:38:07.960 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-03-11T20:38:07.976 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-03-11T20:38:08.003 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-03-11T20:38:08.042 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-03-11T20:38:08.330 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-03-11T20:38:08.357 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-03-11T20:38:08.362 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-sklearn. 2024-03-11T20:38:08.390 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-03-11T20:38:08.396 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-03-11T20:38:08.428 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-03-11T20:38:08.699 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-sklearn. 2024-03-11T20:38:08.726 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-03-11T20:38:08.766 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-03-11T20:38:09.112 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-03-11T20:38:09.140 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.0.0-1979-g2b0101eb-1jammy_all.deb ... 2024-03-11T20:38:09.264 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:38:09.633 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-03-11T20:38:09.660 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.0.0-1979-g2b0101eb-1jammy_all.deb ... 2024-03-11T20:38:09.784 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:38:10.376 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-cachetools. 2024-03-11T20:38:10.403 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-03-11T20:38:10.442 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-03-11T20:38:10.686 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-pyasn1. 2024-03-11T20:38:10.703 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-03-11T20:38:10.704 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-cachetools. 2024-03-11T20:38:10.720 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-03-11T20:38:10.753 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-03-11T20:38:10.754 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-03-11T20:38:11.049 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-pyasn1. 2024-03-11T20:38:11.076 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-03-11T20:38:11.081 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-03-11T20:38:11.109 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-03-11T20:38:11.115 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-03-11T20:38:11.148 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-03-11T20:38:11.444 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-03-11T20:38:11.471 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-03-11T20:38:11.476 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-rsa. 2024-03-11T20:38:11.503 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-03-11T20:38:11.510 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-03-11T20:38:11.542 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-rsa (4.8-1) ... 2024-03-11T20:38:11.813 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-rsa. 2024-03-11T20:38:11.814 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-google-auth. 2024-03-11T20:38:11.831 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-03-11T20:38:11.840 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-03-11T20:38:11.864 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-03-11T20:38:11.879 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-rsa (4.8-1) ... 2024-03-11T20:38:12.159 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-03-11T20:38:12.176 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-03-11T20:38:12.191 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-google-auth. 2024-03-11T20:38:12.217 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-03-11T20:38:12.218 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-03-11T20:38:12.258 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-03-11T20:38:12.529 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-websocket. 2024-03-11T20:38:12.544 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-03-11T20:38:12.556 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-03-11T20:38:12.571 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-03-11T20:38:12.595 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-03-11T20:38:12.602 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-03-11T20:38:12.847 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-websocket. 2024-03-11T20:38:12.874 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-03-11T20:38:12.913 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-03-11T20:38:12.915 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-kubernetes. 2024-03-11T20:38:12.943 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-03-11T20:38:12.982 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-03-11T20:38:13.233 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-kubernetes. 2024-03-11T20:38:13.260 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-03-11T20:38:13.299 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-03-11T20:38:13.598 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-03-11T20:38:13.612 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.0.0-1979-g2b0101eb-1jammy_all.deb ... 2024-03-11T20:38:13.653 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:38:14.013 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-03-11T20:38:14.041 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.0.0-1979-g2b0101eb-1jammy_all.deb ... 2024-03-11T20:38:14.062 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package libonig5:amd64. 2024-03-11T20:38:14.079 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:38:14.090 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-03-11T20:38:14.156 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-03-11T20:38:14.451 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package libjq1:amd64. 2024-03-11T20:38:14.479 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-03-11T20:38:14.517 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-03-11T20:38:14.778 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package jq. 2024-03-11T20:38:14.782 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package libonig5:amd64. 2024-03-11T20:38:14.794 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-03-11T20:38:14.810 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-03-11T20:38:14.836 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-03-11T20:38:14.849 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-03-11T20:38:15.098 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package socat. 2024-03-11T20:38:15.125 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-03-11T20:38:15.135 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package libjq1:amd64. 2024-03-11T20:38:15.162 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-03-11T20:38:15.164 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-03-11T20:38:15.202 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-03-11T20:38:15.463 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package jq. 2024-03-11T20:38:15.467 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package xmlstarlet. 2024-03-11T20:38:15.490 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-03-11T20:38:15.495 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-03-11T20:38:15.529 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-03-11T20:38:15.533 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-03-11T20:38:15.781 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package socat. 2024-03-11T20:38:15.797 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-03-11T20:38:15.840 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-03-11T20:38:15.913 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package ceph-test. 2024-03-11T20:38:15.929 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../097-ceph-test_19.0.0-1979-g2b0101eb-1jammy_amd64.deb ... 2024-03-11T20:38:15.971 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking ceph-test (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:38:16.135 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package xmlstarlet. 2024-03-11T20:38:16.150 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-03-11T20:38:16.192 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-03-11T20:38:16.547 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package ceph-test. 2024-03-11T20:38:16.562 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../097-ceph-test_19.0.0-1979-g2b0101eb-1jammy_amd64.deb ... 2024-03-11T20:38:16.605 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking ceph-test (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:38:18.225 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package ceph-volume. 2024-03-11T20:38:18.253 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../098-ceph-volume_19.0.0-1979-g2b0101eb-1jammy_all.deb ... 2024-03-11T20:38:18.283 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking ceph-volume (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:38:18.554 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package libcephfs-dev. 2024-03-11T20:38:18.582 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../099-libcephfs-dev_19.0.0-1979-g2b0101eb-1jammy_amd64.deb ... 2024-03-11T20:38:18.612 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking libcephfs-dev (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:38:18.826 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package ceph-volume. 2024-03-11T20:38:18.853 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../098-ceph-volume_19.0.0-1979-g2b0101eb-1jammy_all.deb ... 2024-03-11T20:38:18.892 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking ceph-volume (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:38:18.898 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package lua-socket:amd64. 2024-03-11T20:38:18.926 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-03-11T20:38:18.964 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-03-11T20:38:19.187 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package libcephfs-dev. 2024-03-11T20:38:19.203 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../099-libcephfs-dev_19.0.0-1979-g2b0101eb-1jammy_amd64.deb ... 2024-03-11T20:38:19.246 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking libcephfs-dev (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:38:19.286 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package lua-sec:amd64. 2024-03-11T20:38:19.314 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-03-11T20:38:19.352 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-03-11T20:38:19.557 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package lua-socket:amd64. 2024-03-11T20:38:19.584 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-03-11T20:38:19.623 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-03-11T20:38:19.723 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package nvme-cli. 2024-03-11T20:38:19.751 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-03-11T20:38:19.789 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-03-11T20:38:19.928 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package lua-sec:amd64. 2024-03-11T20:38:19.955 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-03-11T20:38:19.994 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-03-11T20:38:20.228 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package pkg-config. 2024-03-11T20:38:20.256 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-03-11T20:38:20.294 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-03-11T20:38:20.365 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package nvme-cli. 2024-03-11T20:38:20.393 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-03-11T20:38:20.431 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-03-11T20:38:20.555 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-03-11T20:38:20.571 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-03-11T20:38:20.613 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-03-11T20:38:20.763 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package pkg-config. 2024-03-11T20:38:20.791 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-03-11T20:38:20.821 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-03-11T20:38:21.025 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-iniconfig. 2024-03-11T20:38:21.053 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-03-11T20:38:21.091 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-03-11T20:38:21.099 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-03-11T20:38:21.115 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-03-11T20:38:21.157 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-03-11T20:38:21.352 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-pastescript. 2024-03-11T20:38:21.380 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-03-11T20:38:21.410 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-03-11T20:38:21.561 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-iniconfig. 2024-03-11T20:38:21.588 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-03-11T20:38:21.627 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-03-11T20:38:21.706 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-pluggy. 2024-03-11T20:38:21.734 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-03-11T20:38:21.772 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-03-11T20:38:21.896 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-pastescript. 2024-03-11T20:38:21.924 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-03-11T20:38:21.963 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-03-11T20:38:22.025 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-psutil. 2024-03-11T20:38:22.053 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-03-11T20:38:22.091 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-03-11T20:38:22.283 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-pluggy. 2024-03-11T20:38:22.310 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-03-11T20:38:22.349 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-03-11T20:38:22.411 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-py. 2024-03-11T20:38:22.439 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-03-11T20:38:22.477 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-py (1.10.0-1) ... 2024-03-11T20:38:22.669 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-psutil. 2024-03-11T20:38:22.696 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-03-11T20:38:22.735 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-03-11T20:38:22.764 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-pygments. 2024-03-11T20:38:22.792 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-03-11T20:38:22.822 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-03-11T20:38:23.055 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-py. 2024-03-11T20:38:23.082 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-03-11T20:38:23.121 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-py (1.10.0-1) ... 2024-03-11T20:38:23.293 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-pyinotify. 2024-03-11T20:38:23.321 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-03-11T20:38:23.359 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-03-11T20:38:23.450 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-pygments. 2024-03-11T20:38:23.477 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-03-11T20:38:23.516 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-03-11T20:38:23.620 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-toml. 2024-03-11T20:38:23.649 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-03-11T20:38:23.678 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-03-11T20:38:23.945 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-pyinotify. 2024-03-11T20:38:23.956 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-pytest. 2024-03-11T20:38:23.973 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-03-11T20:38:23.973 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-03-11T20:38:24.003 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-03-11T20:38:24.023 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-03-11T20:38:24.326 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-simplejson. 2024-03-11T20:38:24.339 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-toml. 2024-03-11T20:38:24.354 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-03-11T20:38:24.367 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-03-11T20:38:24.384 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-03-11T20:38:24.406 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-03-11T20:38:24.662 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-03-11T20:38:24.678 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-03-11T20:38:24.708 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-pytest. 2024-03-11T20:38:24.711 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-03-11T20:38:24.736 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-03-11T20:38:24.775 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-03-11T20:38:25.104 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-simplejson. 2024-03-11T20:38:25.131 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-03-11T20:38:25.161 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-03-11T20:38:25.389 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-03-11T20:38:25.405 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-03-11T20:38:25.432 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package radosgw. 2024-03-11T20:38:25.447 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-03-11T20:38:25.461 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../116-radosgw_19.0.0-1979-g2b0101eb-1jammy_amd64.deb ... 2024-03-11T20:38:25.726 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking radosgw (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:38:26.151 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package radosgw. 2024-03-11T20:38:26.179 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../116-radosgw_19.0.0-1979-g2b0101eb-1jammy_amd64.deb ... 2024-03-11T20:38:26.217 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking radosgw (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:38:26.418 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package rbd-fuse. 2024-03-11T20:38:26.435 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../117-rbd-fuse_19.0.0-1979-g2b0101eb-1jammy_amd64.deb ... 2024-03-11T20:38:26.468 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking rbd-fuse (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:38:26.731 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-03-11T20:38:26.973 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-03-11T20:38:26.996 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package rbd-fuse. 2024-03-11T20:38:27.012 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../117-rbd-fuse_19.0.0-1979-g2b0101eb-1jammy_amd64.deb ... 2024-03-11T20:38:27.054 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking rbd-fuse (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:38:27.091 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-03-11T20:38:27.341 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-03-11T20:38:27.362 INFO:teuthology.orchestra.run.smithi023.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-03-11T20:38:27.600 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-03-11T20:38:27.701 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-03-11T20:38:27.722 INFO:teuthology.orchestra.run.smithi023.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-03-11T20:38:27.960 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-03-11T20:38:28.154 INFO:teuthology.orchestra.run.smithi023.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-03-11T20:38:28.218 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up cephadm (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:38:28.311 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-03-11T20:38:28.692 INFO:teuthology.orchestra.run.smithi023.stdout:Adding system user cephadm....done 2024-03-11T20:38:28.783 INFO:teuthology.orchestra.run.smithi146.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-03-11T20:38:28.845 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up cephadm (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:38:28.882 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-03-11T20:38:29.158 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-03-11T20:38:29.343 INFO:teuthology.orchestra.run.smithi146.stdout:Adding system user cephadm....done 2024-03-11T20:38:29.418 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-03-11T20:38:29.458 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-03-11T20:38:29.544 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-03-11T20:38:29.727 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-03-11T20:38:29.812 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-03-11T20:38:30.062 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-03-11T20:38:30.145 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-03-11T20:38:30.172 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-py (1.10.0-1) ... 2024-03-11T20:38:30.271 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-03-11T20:38:30.482 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-03-11T20:38:30.522 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-03-11T20:38:30.781 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-03-11T20:38:30.867 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-03-11T20:38:30.907 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-py (1.10.0-1) ... 2024-03-11T20:38:31.127 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-03-11T20:38:31.209 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-03-11T20:38:31.403 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-03-11T20:38:31.553 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-03-11T20:38:31.621 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-03-11T20:38:31.804 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-03-11T20:38:31.880 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-03-11T20:38:32.080 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-03-11T20:38:32.140 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-03-11T20:38:32.323 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-03-11T20:38:32.449 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-ceph-argparse (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:38:32.582 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-03-11T20:38:32.725 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-03-11T20:38:32.833 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-03-11T20:38:32.851 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-03-11T20:38:32.977 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-03-11T20:38:33.068 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-ceph-argparse (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:38:33.095 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-03-11T20:38:33.354 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-03-11T20:38:33.427 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-03-11T20:38:33.553 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-03-11T20:38:33.679 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-03-11T20:38:33.728 INFO:teuthology.orchestra.run.smithi023.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-03-11T20:38:33.753 INFO:teuthology.orchestra.run.smithi023.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-03-11T20:38:33.805 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-03-11T20:38:33.816 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-03-11T20:38:33.942 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-03-11T20:38:34.056 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-03-11T20:38:34.310 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-03-11T20:38:34.436 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-03-11T20:38:34.443 INFO:teuthology.orchestra.run.smithi146.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-03-11T20:38:34.479 INFO:teuthology.orchestra.run.smithi146.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-03-11T20:38:34.534 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-03-11T20:38:34.635 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-03-11T20:38:34.712 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-03-11T20:38:34.971 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-03-11T20:38:35.028 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-03-11T20:38:35.154 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-03-11T20:38:35.272 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up zip (3.0-12build2) ... 2024-03-11T20:38:35.398 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-03-11T20:38:35.447 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-03-11T20:38:35.714 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-03-11T20:38:36.024 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up zip (3.0-12build2) ... 2024-03-11T20:38:36.075 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-03-11T20:38:36.125 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-03-11T20:38:36.334 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-03-11T20:38:36.460 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-03-11T20:38:36.578 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-03-11T20:38:36.810 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-03-11T20:38:36.871 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-03-11T20:38:37.070 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-03-11T20:38:37.196 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-03-11T20:38:37.264 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-03-11T20:38:37.322 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-03-11T20:38:37.565 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-03-11T20:38:37.623 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-03-11T20:38:37.966 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-03-11T20:38:38.007 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-03-11T20:38:38.268 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-03-11T20:38:38.275 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-03-11T20:38:38.619 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-03-11T20:38:38.643 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-03-11T20:38:38.870 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-03-11T20:38:38.936 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-03-11T20:38:38.971 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-ceph-common (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:38:39.264 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-idna (3.3-1) ... 2024-03-11T20:38:39.279 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-03-11T20:38:39.530 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-03-11T20:38:39.607 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-03-11T20:38:39.656 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-ceph-common (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:38:39.957 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-idna (3.3-1) ... 2024-03-11T20:38:40.267 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-03-11T20:38:40.837 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-03-11T20:38:41.055 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-11T20:38:41.481 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-toml (0.10.2-1) ... 2024-03-11T20:38:41.689 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-03-11T20:38:41.757 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-03-11T20:38:41.908 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-03-11T20:38:41.941 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-11T20:38:42.034 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-03-11T20:38:42.050 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-toml (0.10.2-1) ... 2024-03-11T20:38:42.301 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-03-11T20:38:42.336 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-03-11T20:38:42.427 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-03-11T20:38:42.553 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-03-11T20:38:42.603 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-03-11T20:38:42.838 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-11T20:38:42.846 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-03-11T20:38:42.939 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-03-11T20:38:43.097 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-03-11T20:38:43.257 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-03-11T20:38:43.356 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-11T20:38:43.482 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-03-11T20:38:43.500 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-03-11T20:38:43.758 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-03-11T20:38:43.767 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-03-11T20:38:44.010 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-03-11T20:38:44.026 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-03-11T20:38:44.261 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-03-11T20:38:44.286 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-03-11T20:38:44.512 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-03-11T20:38:44.537 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-03-11T20:38:44.663 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-03-11T20:38:44.763 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-03-11T20:38:44.949 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-03-11T20:38:45.022 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-03-11T20:38:45.074 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-03-11T20:38:45.148 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-03-11T20:38:45.358 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-03-11T20:38:45.433 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-03-11T20:38:45.534 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-03-11T20:38:45.659 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-03-11T20:38:45.810 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-03-11T20:38:45.918 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-03-11T20:38:46.044 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-03-11T20:38:46.103 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-03-11T20:38:46.154 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-03-11T20:38:46.242 INFO:teuthology.orchestra.run.smithi023.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-03-11T20:38:46.354 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-03-11T20:38:46.480 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-03-11T20:38:46.489 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-03-11T20:38:46.605 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-03-11T20:38:46.623 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-03-11T20:38:46.695 INFO:teuthology.orchestra.run.smithi146.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-03-11T20:38:46.957 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-03-11T20:38:47.025 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-03-11T20:38:47.058 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-03-11T20:38:47.284 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up lua-any (27ubuntu1) ... 2024-03-11T20:38:47.410 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-portend (3.0.0-1) ... 2024-03-11T20:38:47.435 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-03-11T20:38:47.669 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-11T20:38:47.694 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up lua-any (27ubuntu1) ... 2024-03-11T20:38:47.795 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-03-11T20:38:47.820 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-portend (3.0.0-1) ... 2024-03-11T20:38:48.080 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-11T20:38:48.163 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-03-11T20:38:48.214 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-03-11T20:38:48.447 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-03-11T20:38:48.548 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-03-11T20:38:48.732 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-03-11T20:38:48.849 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-03-11T20:38:48.858 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-rsa (4.8-1) ... 2024-03-11T20:38:49.109 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-03-11T20:38:49.126 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-03-11T20:38:49.218 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-rsa (4.8-1) ... 2024-03-11T20:38:49.402 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-03-11T20:38:49.494 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-03-11T20:38:49.746 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-03-11T20:38:49.762 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-03-11T20:38:50.098 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-03-11T20:38:50.123 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-03-11T20:38:50.408 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-03-11T20:38:50.492 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-03-11T20:38:50.735 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-03-11T20:38:50.768 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-03-11T20:38:50.861 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up librados2 (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:38:50.962 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up libsqlite3-mod-ceph (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:38:51.063 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-03-11T20:38:51.104 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-03-11T20:38:51.230 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up librados2 (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:38:51.356 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up libsqlite3-mod-ceph (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:38:51.391 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up libcephfs2 (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:38:51.482 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-03-11T20:38:51.518 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up libradosstriper1 (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:38:51.643 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-03-11T20:38:51.768 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up libcephfs2 (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:38:51.911 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up libradosstriper1 (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:38:51.928 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up librbd1 (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:38:52.037 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-03-11T20:38:52.054 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up ceph-mgr-modules-core (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:38:52.180 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up ceph-fuse (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:38:52.313 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up librbd1 (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:38:52.399 INFO:teuthology.orchestra.run.smithi023.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-03-11T20:38:52.399 INFO:teuthology.orchestra.run.smithi023.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-03-11T20:38:52.440 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up ceph-mgr-modules-core (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:38:52.557 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up ceph-fuse (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:38:52.762 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-03-11T20:38:52.762 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-03-11T20:38:52.848 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up libcephfs-dev (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:38:52.949 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-rados (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:38:53.050 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up librgw2 (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:38:53.178 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-03-11T20:38:53.250 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up libcephfs-dev (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:38:53.376 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-rados (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:38:53.494 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up librgw2 (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:38:53.620 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-03-11T20:38:54.480 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-rbd (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:38:54.580 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up rbd-fuse (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:38:54.707 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-rgw (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:38:54.832 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-rbd (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:38:54.832 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-cephfs (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:38:54.941 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up rbd-fuse (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:38:54.975 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up ceph-common (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:38:55.042 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-rgw (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:38:55.193 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-cephfs (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:38:55.319 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up ceph-common (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:38:55.519 INFO:teuthology.orchestra.run.smithi023.stdout:Adding group ceph....done 2024-03-11T20:38:55.811 INFO:teuthology.orchestra.run.smithi023.stdout:Adding system user ceph....done 2024-03-11T20:38:55.861 INFO:teuthology.orchestra.run.smithi146.stdout:Adding group ceph....done 2024-03-11T20:38:55.955 INFO:teuthology.orchestra.run.smithi023.stdout:Setting system user ceph properties....done 2024-03-11T20:38:55.973 INFO:teuthology.orchestra.run.smithi023.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-03-11T20:38:56.115 INFO:teuthology.orchestra.run.smithi023.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-03-11T20:38:56.339 INFO:teuthology.orchestra.run.smithi146.stdout:Adding system user ceph....done 2024-03-11T20:38:56.425 INFO:teuthology.orchestra.run.smithi023.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-03-11T20:38:56.474 INFO:teuthology.orchestra.run.smithi146.stdout:Setting system user ceph properties....done 2024-03-11T20:38:56.491 INFO:teuthology.orchestra.run.smithi146.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-03-11T20:38:56.632 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-03-11T20:38:56.884 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up ceph-test (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:38:56.966 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-03-11T20:38:57.010 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up radosgw (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:38:57.446 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up ceph-test (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:38:57.503 INFO:teuthology.orchestra.run.smithi023.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-03-11T20:38:57.503 INFO:teuthology.orchestra.run.smithi023.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-03-11T20:38:57.571 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up radosgw (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:38:57.979 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up ceph-base (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:38:58.027 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-03-11T20:38:58.027 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-03-11T20:38:58.514 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up ceph-base (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:38:58.522 INFO:teuthology.orchestra.run.smithi023.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-03-11T20:38:58.852 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-03-11T20:38:58.972 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up ceph-mds (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:38:59.192 INFO:teuthology.orchestra.run.smithi023.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-03-11T20:38:59.192 INFO:teuthology.orchestra.run.smithi023.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-03-11T20:38:59.341 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up ceph-mds (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:38:59.563 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-03-11T20:38:59.563 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-03-11T20:38:59.659 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up ceph-mgr (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:38:59.895 INFO:teuthology.orchestra.run.smithi023.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-03-11T20:38:59.895 INFO:teuthology.orchestra.run.smithi023.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-03-11T20:39:00.076 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up ceph-mgr (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:39:00.303 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-03-11T20:39:00.303 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-03-11T20:39:00.360 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up ceph-osd (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:39:00.642 INFO:teuthology.orchestra.run.smithi023.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-03-11T20:39:00.642 INFO:teuthology.orchestra.run.smithi023.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-03-11T20:39:00.786 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up ceph-osd (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:39:01.069 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-03-11T20:39:01.069 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-03-11T20:39:01.170 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up ceph-mgr-k8sevents (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:39:01.271 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:39:01.414 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up ceph-mon (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:39:01.563 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up ceph-mgr-k8sevents (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:39:01.618 INFO:teuthology.orchestra.run.smithi023.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-03-11T20:39:01.618 INFO:teuthology.orchestra.run.smithi023.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-03-11T20:39:01.689 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:39:02.026 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up ceph-mon (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:39:02.166 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up ceph-mgr-cephadm (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:39:02.271 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-03-11T20:39:02.271 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-03-11T20:39:02.317 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up ceph (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:39:02.443 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up ceph-mgr-dashboard (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:39:02.594 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up ceph-volume (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:39:02.786 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up ceph-mgr-cephadm (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:39:02.929 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up ceph (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:39:02.947 INFO:teuthology.orchestra.run.smithi023.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-03-11T20:39:03.029 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up ceph-mgr-dashboard (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:39:03.056 INFO:teuthology.orchestra.run.smithi023.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-03-11T20:39:03.200 INFO:teuthology.orchestra.run.smithi023.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-03-11T20:39:03.331 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up ceph-volume (19.0.0-1979-g2b0101eb-1jammy) ... 2024-03-11T20:39:03.658 INFO:teuthology.orchestra.run.smithi146.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-03-11T20:39:03.767 INFO:teuthology.orchestra.run.smithi146.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-03-11T20:39:03.920 INFO:teuthology.orchestra.run.smithi146.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-03-11T20:39:04.334 INFO:teuthology.orchestra.run.smithi023.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-03-11T20:39:04.954 INFO:teuthology.orchestra.run.smithi146.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-03-11T20:39:04.958 INFO:teuthology.orchestra.run.smithi023.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-03-11T20:39:04.964 DEBUG:teuthology.parallel:result is None 2024-03-11T20:39:05.585 INFO:teuthology.orchestra.run.smithi146.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-03-11T20:39:05.589 DEBUG:teuthology.parallel:result is None 2024-03-11T20:39:05.590 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-11T20:39:05.819 DEBUG:teuthology.orchestra.run.smithi023:> dpkg-query -W -f '${Version}' ceph 2024-03-11T20:39:05.839 INFO:teuthology.orchestra.run.smithi023.stdout:19.0.0-1979-g2b0101eb-1jammy 2024-03-11T20:39:05.839 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1979-g2b0101eb-1jammy 2024-03-11T20:39:05.839 INFO:teuthology.task.install:The correct ceph version 19.0.0-1979-g2b0101eb-1jammy is installed. 2024-03-11T20:39:05.841 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-11T20:39:05.979 DEBUG:teuthology.orchestra.run.smithi146:> dpkg-query -W -f '${Version}' ceph 2024-03-11T20:39:05.998 INFO:teuthology.orchestra.run.smithi146.stdout:19.0.0-1979-g2b0101eb-1jammy 2024-03-11T20:39:05.998 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1979-g2b0101eb-1jammy 2024-03-11T20:39:05.998 INFO:teuthology.task.install:The correct ceph version 19.0.0-1979-g2b0101eb-1jammy is installed. 2024-03-11T20:39:06.000 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-11T20:39:06.000 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-11T20:39:06.000 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-11T20:39:06.014 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-11T20:39:06.015 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-11T20:39:06.054 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-11T20:39:06.054 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-11T20:39:06.055 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd of=/usr/bin/daemon-helper 2024-03-11T20:39:06.068 DEBUG:teuthology.orchestra.run.smithi023:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-11T20:39:06.124 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-11T20:39:06.124 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/usr/bin/daemon-helper 2024-03-11T20:39:06.138 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-11T20:39:06.192 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-11T20:39:06.193 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-11T20:39:06.193 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-11T20:39:06.206 DEBUG:teuthology.orchestra.run.smithi023:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-11T20:39:06.261 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-11T20:39:06.261 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-11T20:39:06.273 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-11T20:39:06.328 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-11T20:39:06.328 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-11T20:39:06.328 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd of=/usr/bin/stdin-killer 2024-03-11T20:39:06.341 DEBUG:teuthology.orchestra.run.smithi023:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-11T20:39:06.396 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-11T20:39:06.396 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/usr/bin/stdin-killer 2024-03-11T20:39:06.409 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-11T20:39:06.466 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-11T20:39:06.574 INFO:tasks.cephadm:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'client.0': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'ms type': 'async'}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '2b0101eb74d60610d544cb8a473e23a35ac1cc0b', 'cephadm_mode': 'root'} 2024-03-11T20:39:06.574 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-11T20:39:06.574 INFO:tasks.cephadm:Cluster fsid is 67ba00f8-dfe7-11ee-95c8-87774f69a715 2024-03-11T20:39:06.574 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-11T20:39:06.575 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.23', 'mon.c': '[v2:172.21.15.23:3301,v1:172.21.15.23:6790]', 'mon.b': '172.21.15.146'} 2024-03-11T20:39:06.575 INFO:tasks.cephadm:First mon is mon.a on smithi023 2024-03-11T20:39:06.575 INFO:tasks.cephadm:First mgr is y 2024-03-11T20:39:06.575 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-11T20:39:06.575 DEBUG:teuthology.orchestra.run.smithi023:> sudo hostname $(hostname -s) 2024-03-11T20:39:06.590 DEBUG:teuthology.orchestra.run.smithi146:> sudo hostname $(hostname -s) 2024-03-11T20:39:06.605 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-03-11T20:39:06.606 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-11T20:39:06.837 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '2b0101eb74d60610d544cb8a473e23a35ac1cc0b', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=jammy,DIST=jammy,MACHINE_SIZE=gigantic/54565/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-1979-g2b0101eb', 'node_name': '172.21.2.16+braggi16', 'job_name': 'ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=jammy,DIST=jammy,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-1979-g2b0101eb-1jammy'}, 'url': 'https://1.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default/', 'distro_codename': 'jammy', 'modified': '2024-03-11 04:01:31.529734', 'distro_version': '22.04', 'project': 'ceph', 'flavor': 'default', 'ref': 'main', 'chacra_url': 'https://1.chacra.ceph.com/repos/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/flavors/default/', 'archs': ['x86_64'], 'distro': 'ubuntu'}] 2024-03-11T20:39:06.976 INFO:tasks.util.chacra:got chacra host 1.chacra.ceph.com, ref main, sha1 2b0101eb74d60610d544cb8a473e23a35ac1cc0b from https://shaman.ceph.com/api/search/?project=ceph&distros=ubuntu%2F22.04%2Fx86_64&flavor=default&sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-11T20:39:06.977 INFO:tasks.cephadm:Discovered cachra url: https://1.chacra.ceph.com/binaries/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/x86_64/flavors/default/cephadm 2024-03-11T20:39:06.978 DEBUG:teuthology.orchestra.run.smithi023:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-11T20:39:07.463 INFO:teuthology.orchestra.run.smithi023.stdout:-rw-rw-r-- 1 ubuntu ubuntu 774557 Mar 11 20:39 /home/ubuntu/cephtest/cephadm 2024-03-11T20:39:07.463 DEBUG:teuthology.orchestra.run.smithi146:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/ubuntu/jammy/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-11T20:39:08.108 INFO:teuthology.orchestra.run.smithi146.stdout:-rw-rw-r-- 1 ubuntu ubuntu 774557 Mar 11 20:39 /home/ubuntu/cephtest/cephadm 2024-03-11T20:39:08.108 DEBUG:teuthology.orchestra.run.smithi023:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-11T20:39:08.116 DEBUG:teuthology.orchestra.run.smithi146:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-11T20:39:08.136 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b on all hosts... 2024-03-11T20:39:08.137 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b pull 2024-03-11T20:39:08.162 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b pull 2024-03-11T20:39:08.330 INFO:teuthology.orchestra.run.smithi146.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b... 2024-03-11T20:39:08.334 INFO:teuthology.orchestra.run.smithi023.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b... 2024-03-11T20:39:58.174 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-03-11T20:39:58.175 INFO:teuthology.orchestra.run.smithi146.stdout: "ceph_version": "ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev)", 2024-03-11T20:39:58.175 INFO:teuthology.orchestra.run.smithi146.stdout: "image_id": "4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87", 2024-03-11T20:39:58.175 INFO:teuthology.orchestra.run.smithi146.stdout: "repo_digests": [ 2024-03-11T20:39:58.175 INFO:teuthology.orchestra.run.smithi146.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379" 2024-03-11T20:39:58.175 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T20:39:58.175 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-03-11T20:40:01.396 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-11T20:40:01.397 INFO:teuthology.orchestra.run.smithi023.stdout: "ceph_version": "ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev)", 2024-03-11T20:40:01.397 INFO:teuthology.orchestra.run.smithi023.stdout: "image_id": "4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87", 2024-03-11T20:40:01.397 INFO:teuthology.orchestra.run.smithi023.stdout: "repo_digests": [ 2024-03-11T20:40:01.397 INFO:teuthology.orchestra.run.smithi023.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379" 2024-03-11T20:40:01.397 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T20:40:01.397 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-11T20:40:01.445 DEBUG:teuthology.orchestra.run.smithi023:> sudo mkdir -p /etc/ceph 2024-03-11T20:40:01.459 DEBUG:teuthology.orchestra.run.smithi146:> sudo mkdir -p /etc/ceph 2024-03-11T20:40:01.474 DEBUG:teuthology.orchestra.run.smithi023:> sudo chmod 777 /etc/ceph 2024-03-11T20:40:01.517 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod 777 /etc/ceph 2024-03-11T20:40:01.530 INFO:tasks.cephadm:Writing seed config... 2024-03-11T20:40:01.531 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-11T20:40:01.531 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-11T20:40:01.532 INFO:tasks.cephadm: override: [client.0] debug ms = 1 2024-03-11T20:40:01.532 INFO:tasks.cephadm: override: [client.0] debug objecter = 20 2024-03-11T20:40:01.532 INFO:tasks.cephadm: override: [client.0] debug rados = 20 2024-03-11T20:40:01.532 INFO:tasks.cephadm: override: [global] ms type = async 2024-03-11T20:40:01.532 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-11T20:40:01.532 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-11T20:40:01.532 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-11T20:40:01.532 INFO:tasks.cephadm: override: [mon] mon min osdmap epochs = 50 2024-03-11T20:40:01.532 INFO:tasks.cephadm: override: [mon] mon osdmap full prune interval = 2 2024-03-11T20:40:01.532 INFO:tasks.cephadm: override: [mon] mon osdmap full prune min = 15 2024-03-11T20:40:01.532 INFO:tasks.cephadm: override: [mon] mon osdmap full prune txsize = 2 2024-03-11T20:40:01.532 INFO:tasks.cephadm: override: [mon] paxos service trim min = 10 2024-03-11T20:40:01.532 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-11T20:40:01.532 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-11T20:40:01.533 INFO:tasks.cephadm: override: [osd] osd debug reject backfill probability = 0.3 2024-03-11T20:40:01.533 INFO:tasks.cephadm: override: [osd] osd delete sleep = 1 2024-03-11T20:40:01.533 INFO:tasks.cephadm: override: [osd] osd max backfills = 3 2024-03-11T20:40:01.533 INFO:tasks.cephadm: override: [osd] osd scrub max interval = 120 2024-03-11T20:40:01.533 INFO:tasks.cephadm: override: [osd] osd scrub min interval = 60 2024-03-11T20:40:01.533 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-03-11T20:40:01.533 INFO:tasks.cephadm: override: [osd] osd snap trim sleep = 2 2024-03-11T20:40:01.534 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-11T20:40:01.534 DEBUG:teuthology.orchestra.run.smithi023:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-11T20:40:01.568 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 67ba00f8-dfe7-11ee-95c8-87774f69a715 ms type = async [osd] osd scrub load threshold = 5.0 osd scrub max interval = 120 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 osd debug reject backfill probability = 0.3 osd delete sleep = 1 osd max backfills = 3 osd scrub min interval = 60 osd snap trim sleep = 2 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon min osdmap epochs = 50 mon osdmap full prune interval = 2 mon osdmap full prune min = 15 mon osdmap full prune txsize = 2 paxos service trim min = 10 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client.0] debug ms = 1 debug objecter = 20 debug rados = 20 2024-03-11T20:40:01.568 DEBUG:teuthology.orchestra.run.smithi023:mon.a> sudo journalctl -f -n 0 -u ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@mon.a.service 2024-03-11T20:40:01.610 DEBUG:teuthology.orchestra.run.smithi023:mgr.y> sudo journalctl -f -n 0 -u ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@mgr.y.service 2024-03-11T20:40:01.654 INFO:tasks.cephadm:Bootstrapping... 2024-03-11T20:40:01.654 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b -v bootstrap --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.23 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-11T20:40:01.862 INFO:teuthology.orchestra.run.smithi023.stdout:-------------------------------------------------------------------------------- 2024-03-11T20:40:01.862 INFO:teuthology.orchestra.run.smithi023.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b', '-v', 'bootstrap', '--fsid', '67ba00f8-dfe7-11ee-95c8-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.23', '--skip-admin-label'] 2024-03-11T20:40:01.862 INFO:teuthology.orchestra.run.smithi023.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-11T20:40:01.863 INFO:teuthology.orchestra.run.smithi023.stdout:Verifying podman|docker is present... 2024-03-11T20:40:01.863 INFO:teuthology.orchestra.run.smithi023.stdout:Verifying lvm2 is present... 2024-03-11T20:40:01.864 INFO:teuthology.orchestra.run.smithi023.stdout:Verifying time synchronization is in place... 2024-03-11T20:40:01.869 INFO:teuthology.orchestra.run.smithi023.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-11T20:40:01.869 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-11T20:40:01.873 INFO:teuthology.orchestra.run.smithi023.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-11T20:40:01.873 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stdout inactive 2024-03-11T20:40:01.877 INFO:teuthology.orchestra.run.smithi023.stdout:Non-zero exit code 1 from systemctl is-enabled chronyd.service 2024-03-11T20:40:01.877 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stderr Failed to get unit file state for chronyd.service: No such file or directory 2024-03-11T20:40:01.881 INFO:teuthology.orchestra.run.smithi023.stdout:Non-zero exit code 3 from systemctl is-active chronyd.service 2024-03-11T20:40:01.881 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stdout inactive 2024-03-11T20:40:01.885 INFO:teuthology.orchestra.run.smithi023.stdout:Non-zero exit code 1 from systemctl is-enabled systemd-timesyncd.service 2024-03-11T20:40:01.885 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stdout masked 2024-03-11T20:40:01.888 INFO:teuthology.orchestra.run.smithi023.stdout:Non-zero exit code 3 from systemctl is-active systemd-timesyncd.service 2024-03-11T20:40:01.889 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stdout inactive 2024-03-11T20:40:01.892 INFO:teuthology.orchestra.run.smithi023.stdout:Non-zero exit code 1 from systemctl is-enabled ntpd.service 2024-03-11T20:40:01.892 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stderr Failed to get unit file state for ntpd.service: No such file or directory 2024-03-11T20:40:01.896 INFO:teuthology.orchestra.run.smithi023.stdout:Non-zero exit code 3 from systemctl is-active ntpd.service 2024-03-11T20:40:01.896 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stdout inactive 2024-03-11T20:40:01.901 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stdout enabled 2024-03-11T20:40:01.905 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stdout active 2024-03-11T20:40:01.905 INFO:teuthology.orchestra.run.smithi023.stdout:Unit ntp.service is enabled and running 2024-03-11T20:40:01.905 INFO:teuthology.orchestra.run.smithi023.stdout:Repeating the final host check... 2024-03-11T20:40:01.905 INFO:teuthology.orchestra.run.smithi023.stdout:docker (/usr/bin/docker) is present 2024-03-11T20:40:01.905 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl is present 2024-03-11T20:40:01.905 INFO:teuthology.orchestra.run.smithi023.stdout:lvcreate is present 2024-03-11T20:40:01.908 INFO:teuthology.orchestra.run.smithi023.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-11T20:40:01.908 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-11T20:40:01.912 INFO:teuthology.orchestra.run.smithi023.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-11T20:40:01.912 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stdout inactive 2024-03-11T20:40:01.916 INFO:teuthology.orchestra.run.smithi023.stdout:Non-zero exit code 1 from systemctl is-enabled chronyd.service 2024-03-11T20:40:01.916 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stderr Failed to get unit file state for chronyd.service: No such file or directory 2024-03-11T20:40:01.919 INFO:teuthology.orchestra.run.smithi023.stdout:Non-zero exit code 3 from systemctl is-active chronyd.service 2024-03-11T20:40:01.919 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stdout inactive 2024-03-11T20:40:01.922 INFO:teuthology.orchestra.run.smithi023.stdout:Non-zero exit code 1 from systemctl is-enabled systemd-timesyncd.service 2024-03-11T20:40:01.922 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stdout masked 2024-03-11T20:40:01.925 INFO:teuthology.orchestra.run.smithi023.stdout:Non-zero exit code 3 from systemctl is-active systemd-timesyncd.service 2024-03-11T20:40:01.925 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stdout inactive 2024-03-11T20:40:01.928 INFO:teuthology.orchestra.run.smithi023.stdout:Non-zero exit code 1 from systemctl is-enabled ntpd.service 2024-03-11T20:40:01.928 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stderr Failed to get unit file state for ntpd.service: No such file or directory 2024-03-11T20:40:01.932 INFO:teuthology.orchestra.run.smithi023.stdout:Non-zero exit code 3 from systemctl is-active ntpd.service 2024-03-11T20:40:01.932 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stdout inactive 2024-03-11T20:40:01.936 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stdout enabled 2024-03-11T20:40:01.940 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stdout active 2024-03-11T20:40:01.940 INFO:teuthology.orchestra.run.smithi023.stdout:Unit ntp.service is enabled and running 2024-03-11T20:40:01.941 INFO:teuthology.orchestra.run.smithi023.stdout:Host looks OK 2024-03-11T20:40:01.941 INFO:teuthology.orchestra.run.smithi023.stdout:Cluster fsid: 67ba00f8-dfe7-11ee-95c8-87774f69a715 2024-03-11T20:40:01.941 INFO:teuthology.orchestra.run.smithi023.stdout:Acquiring lock 139957959258032 on /run/cephadm/67ba00f8-dfe7-11ee-95c8-87774f69a715.lock 2024-03-11T20:40:01.941 INFO:teuthology.orchestra.run.smithi023.stdout:Lock 139957959258032 acquired on /run/cephadm/67ba00f8-dfe7-11ee-95c8-87774f69a715.lock 2024-03-11T20:40:01.941 INFO:teuthology.orchestra.run.smithi023.stdout:Verifying IP 172.21.15.23 port 3300 ... 2024-03-11T20:40:01.941 INFO:teuthology.orchestra.run.smithi023.stdout:Verifying IP 172.21.15.23 port 6789 ... 2024-03-11T20:40:01.941 INFO:teuthology.orchestra.run.smithi023.stdout:Base mon IP(s) is [172.21.15.23:3300, 172.21.15.23:6789], mon addrv is [v2:172.21.15.23:3300,v1:172.21.15.23:6789] 2024-03-11T20:40:01.943 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 2024-03-11T20:40:01.943 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/sbin/ip: stdout 172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown 2024-03-11T20:40:01.943 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.23 2024-03-11T20:40:01.945 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-11T20:40:01.945 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 256 pref medium 2024-03-11T20:40:01.945 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 1024 expires 1756sec hoplimit 64 pref medium 2024-03-11T20:40:01.947 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-11T20:40:01.948 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-11T20:40:01.948 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-11T20:40:01.948 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-03-11T20:40:01.948 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/sbin/ip: stdout inet6 fe80::ec4:7aff:fe59:ce42/64 scope link 2024-03-11T20:40:01.948 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-11T20:40:01.948 INFO:teuthology.orchestra.run.smithi023.stdout:Mon IP `172.21.15.23` is in CIDR network `172.21.0.0/20` 2024-03-11T20:40:01.948 INFO:teuthology.orchestra.run.smithi023.stdout:Mon IP `172.21.15.23` is in CIDR network `172.21.0.0/20` 2024-03-11T20:40:01.949 INFO:teuthology.orchestra.run.smithi023.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-03-11T20:40:01.949 INFO:teuthology.orchestra.run.smithi023.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-11T20:40:01.949 INFO:teuthology.orchestra.run.smithi023.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b... 2024-03-11T20:40:02.313 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/docker: stdout 2b0101eb74d60610d544cb8a473e23a35ac1cc0b: Pulling from ceph-ci/ceph 2024-03-11T20:40:02.313 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/docker: stdout Digest: sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379 2024-03-11T20:40:02.314 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/docker: stdout Status: Image is up to date for quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-11T20:40:02.314 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/docker: stdout quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-11T20:40:03.701 INFO:teuthology.orchestra.run.smithi023.stdout:ceph: stdout ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev) 2024-03-11T20:40:03.702 INFO:teuthology.orchestra.run.smithi023.stdout:Ceph version: ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev) 2024-03-11T20:40:03.702 INFO:teuthology.orchestra.run.smithi023.stdout:Extracting ceph user uid/gid from container image... 2024-03-11T20:40:04.905 INFO:teuthology.orchestra.run.smithi023.stdout:stat: stdout 167 167 2024-03-11T20:40:04.906 INFO:teuthology.orchestra.run.smithi023.stdout:Creating initial keys... 2024-03-11T20:40:06.194 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-authtool: stdout AQAlbO9lzqXyJhAAcqCHwtm9OW6Gle9yacGv0Q== 2024-03-11T20:40:07.373 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-authtool: stdout AQAmbO9l0OB0MhAAdm/ROvN4hWl8fVQGxlWQUQ== 2024-03-11T20:40:08.628 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-authtool: stdout AQAobO9lA8VMBhAAWqeUS6kEFMQHyn7CfceOEA== 2024-03-11T20:40:08.629 INFO:teuthology.orchestra.run.smithi023.stdout:Creating initial monmap... 2024-03-11T20:40:09.885 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-11T20:40:09.885 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-03-11T20:40:09.885 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 67ba00f8-dfe7-11ee-95c8-87774f69a715 2024-03-11T20:40:09.886 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-11T20:40:09.886 INFO:teuthology.orchestra.run.smithi023.stdout:monmaptool for a [v2:172.21.15.23:3300,v1:172.21.15.23:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-11T20:40:09.886 INFO:teuthology.orchestra.run.smithi023.stdout:setting min_mon_release = quincy 2024-03-11T20:40:09.886 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/monmaptool: set fsid to 67ba00f8-dfe7-11ee-95c8-87774f69a715 2024-03-11T20:40:09.886 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-11T20:40:09.886 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-11T20:40:09.886 INFO:teuthology.orchestra.run.smithi023.stdout:Creating mon... 2024-03-11T20:40:11.212 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.611+0000 7fb79b3d3c80 0 set uid:gid to 167:167 (ceph:ceph) 2024-03-11T20:40:11.212 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.611+0000 7fb79b3d3c80 1 imported monmap: 2024-03-11T20:40:11.212 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr epoch 0 2024-03-11T20:40:11.212 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 2024-03-11T20:40:11.213 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr last_changed 2024-03-11T20:40:09.377861+0000 2024-03-11T20:40:11.213 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr created 2024-03-11T20:40:09.377861+0000 2024-03-11T20:40:11.213 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr min_mon_release 17 (quincy) 2024-03-11T20:40:11.213 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr election_strategy: 1 2024-03-11T20:40:11.213 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr 0: [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] mon.a 2024-03-11T20:40:11.213 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr 2024-03-11T20:40:11.213 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.611+0000 7fb79b3d3c80 0 /usr/bin/ceph-mon: set fsid to 67ba00f8-dfe7-11ee-95c8-87774f69a715 2024-03-11T20:40:11.213 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.615+0000 7fb79b3d3c80 4 rocksdb: RocksDB version: 7.9.2 2024-03-11T20:40:11.213 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr 2024-03-11T20:40:11.213 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.615+0000 7fb79b3d3c80 4 rocksdb: Git sha 0 2024-03-11T20:40:11.213 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.615+0000 7fb79b3d3c80 4 rocksdb: Compile date 2024-03-10 23:25:48 2024-03-11T20:40:11.213 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.615+0000 7fb79b3d3c80 4 rocksdb: DB SUMMARY 2024-03-11T20:40:11.213 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr 2024-03-11T20:40:11.213 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.615+0000 7fb79b3d3c80 4 rocksdb: DB Session ID: X63EVZ81VE4CYK8QJHBA 2024-03-11T20:40:11.214 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr 2024-03-11T20:40:11.214 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.615+0000 7fb79b3d3c80 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 0, files: 2024-03-11T20:40:11.214 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr 2024-03-11T20:40:11.214 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.615+0000 7fb79b3d3c80 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 2024-03-11T20:40:11.214 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr 2024-03-11T20:40:11.214 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.615+0000 7fb79b3d3c80 4 rocksdb: Options.error_if_exists: 0 2024-03-11T20:40:11.214 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.615+0000 7fb79b3d3c80 4 rocksdb: Options.create_if_missing: 1 2024-03-11T20:40:11.214 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.615+0000 7fb79b3d3c80 4 rocksdb: Options.paranoid_checks: 1 2024-03-11T20:40:11.214 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.615+0000 7fb79b3d3c80 4 rocksdb: Options.flush_verify_memtable_count: 1 2024-03-11T20:40:11.214 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.615+0000 7fb79b3d3c80 4 rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-11T20:40:11.214 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.615+0000 7fb79b3d3c80 4 rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-11T20:40:11.214 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.615+0000 7fb79b3d3c80 4 rocksdb: Options.env: 0x5642990f9b20 2024-03-11T20:40:11.214 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.615+0000 7fb79b3d3c80 4 rocksdb: Options.fs: PosixFileSystem 2024-03-11T20:40:11.214 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.615+0000 7fb79b3d3c80 4 rocksdb: Options.info_log: 0x56429a966b40 2024-03-11T20:40:11.215 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.615+0000 7fb79b3d3c80 4 rocksdb: Options.max_file_opening_threads: 16 2024-03-11T20:40:11.215 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.615+0000 7fb79b3d3c80 4 rocksdb: Options.statistics: (nil) 2024-03-11T20:40:11.215 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.615+0000 7fb79b3d3c80 4 rocksdb: Options.use_fsync: 0 2024-03-11T20:40:11.215 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.615+0000 7fb79b3d3c80 4 rocksdb: Options.max_log_file_size: 0 2024-03-11T20:40:11.215 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.615+0000 7fb79b3d3c80 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-11T20:40:11.215 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.615+0000 7fb79b3d3c80 4 rocksdb: Options.log_file_time_to_roll: 0 2024-03-11T20:40:11.215 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.615+0000 7fb79b3d3c80 4 rocksdb: Options.keep_log_file_num: 1000 2024-03-11T20:40:11.215 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.615+0000 7fb79b3d3c80 4 rocksdb: Options.recycle_log_file_num: 0 2024-03-11T20:40:11.215 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.615+0000 7fb79b3d3c80 4 rocksdb: Options.allow_fallocate: 1 2024-03-11T20:40:11.215 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.615+0000 7fb79b3d3c80 4 rocksdb: Options.allow_mmap_reads: 0 2024-03-11T20:40:11.215 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.615+0000 7fb79b3d3c80 4 rocksdb: Options.allow_mmap_writes: 0 2024-03-11T20:40:11.215 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.615+0000 7fb79b3d3c80 4 rocksdb: Options.use_direct_reads: 0 2024-03-11T20:40:11.215 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.615+0000 7fb79b3d3c80 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-11T20:40:11.215 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.615+0000 7fb79b3d3c80 4 rocksdb: Options.create_missing_column_families: 0 2024-03-11T20:40:11.216 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.615+0000 7fb79b3d3c80 4 rocksdb: Options.db_log_dir: 2024-03-11T20:40:11.216 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.615+0000 7fb79b3d3c80 4 rocksdb: Options.wal_dir: 2024-03-11T20:40:11.216 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.615+0000 7fb79b3d3c80 4 rocksdb: Options.table_cache_numshardbits: 6 2024-03-11T20:40:11.216 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.615+0000 7fb79b3d3c80 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-03-11T20:40:11.216 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.615+0000 7fb79b3d3c80 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-03-11T20:40:11.216 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.615+0000 7fb79b3d3c80 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-11T20:40:11.216 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.615+0000 7fb79b3d3c80 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-11T20:40:11.216 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.615+0000 7fb79b3d3c80 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-03-11T20:40:11.216 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.615+0000 7fb79b3d3c80 4 rocksdb: Options.advise_random_on_open: 1 2024-03-11T20:40:11.216 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.615+0000 7fb79b3d3c80 4 rocksdb: Options.db_write_buffer_size: 0 2024-03-11T20:40:11.216 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.615+0000 7fb79b3d3c80 4 rocksdb: Options.write_buffer_manager: 0x56429a93b360 2024-03-11T20:40:11.216 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.615+0000 7fb79b3d3c80 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-11T20:40:11.216 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.615+0000 7fb79b3d3c80 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-11T20:40:11.217 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.615+0000 7fb79b3d3c80 4 rocksdb: Options.use_adaptive_mutex: 0 2024-03-11T20:40:11.217 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.615+0000 7fb79b3d3c80 4 rocksdb: Options.rate_limiter: (nil) 2024-03-11T20:40:11.217 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.615+0000 7fb79b3d3c80 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-11T20:40:11.217 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.615+0000 7fb79b3d3c80 4 rocksdb: Options.wal_recovery_mode: 2 2024-03-11T20:40:11.217 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.615+0000 7fb79b3d3c80 4 rocksdb: Options.enable_thread_tracking: 0 2024-03-11T20:40:11.217 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.615+0000 7fb79b3d3c80 4 rocksdb: Options.enable_pipelined_write: 0 2024-03-11T20:40:11.217 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.615+0000 7fb79b3d3c80 4 rocksdb: Options.unordered_write: 0 2024-03-11T20:40:11.217 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.615+0000 7fb79b3d3c80 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-11T20:40:11.217 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.615+0000 7fb79b3d3c80 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-11T20:40:11.217 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.615+0000 7fb79b3d3c80 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-11T20:40:11.217 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.615+0000 7fb79b3d3c80 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-11T20:40:11.217 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.615+0000 7fb79b3d3c80 4 rocksdb: Options.row_cache: None 2024-03-11T20:40:11.217 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.615+0000 7fb79b3d3c80 4 rocksdb: Options.wal_filter: None 2024-03-11T20:40:11.218 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.615+0000 7fb79b3d3c80 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-11T20:40:11.218 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.615+0000 7fb79b3d3c80 4 rocksdb: Options.allow_ingest_behind: 0 2024-03-11T20:40:11.218 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.615+0000 7fb79b3d3c80 4 rocksdb: Options.two_write_queues: 0 2024-03-11T20:40:11.218 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.615+0000 7fb79b3d3c80 4 rocksdb: Options.manual_wal_flush: 0 2024-03-11T20:40:11.218 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.615+0000 7fb79b3d3c80 4 rocksdb: Options.wal_compression: 0 2024-03-11T20:40:11.218 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.615+0000 7fb79b3d3c80 4 rocksdb: Options.atomic_flush: 0 2024-03-11T20:40:11.218 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.615+0000 7fb79b3d3c80 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-11T20:40:11.218 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.615+0000 7fb79b3d3c80 4 rocksdb: Options.persist_stats_to_disk: 0 2024-03-11T20:40:11.218 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.615+0000 7fb79b3d3c80 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-03-11T20:40:11.218 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.615+0000 7fb79b3d3c80 4 rocksdb: Options.log_readahead_size: 0 2024-03-11T20:40:11.218 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.615+0000 7fb79b3d3c80 4 rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-11T20:40:11.218 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.615+0000 7fb79b3d3c80 4 rocksdb: Options.best_efforts_recovery: 0 2024-03-11T20:40:11.218 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.615+0000 7fb79b3d3c80 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-11T20:40:11.218 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.615+0000 7fb79b3d3c80 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-11T20:40:11.219 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.615+0000 7fb79b3d3c80 4 rocksdb: Options.allow_data_in_errors: 0 2024-03-11T20:40:11.219 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.615+0000 7fb79b3d3c80 4 rocksdb: Options.db_host_id: __hostname__ 2024-03-11T20:40:11.219 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.615+0000 7fb79b3d3c80 4 rocksdb: Options.enforce_single_del_contracts: true 2024-03-11T20:40:11.219 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.615+0000 7fb79b3d3c80 4 rocksdb: Options.max_background_jobs: 2 2024-03-11T20:40:11.219 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.615+0000 7fb79b3d3c80 4 rocksdb: Options.max_background_compactions: -1 2024-03-11T20:40:11.219 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.615+0000 7fb79b3d3c80 4 rocksdb: Options.max_subcompactions: 1 2024-03-11T20:40:11.219 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.615+0000 7fb79b3d3c80 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-11T20:40:11.219 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.615+0000 7fb79b3d3c80 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-11T20:40:11.219 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.615+0000 7fb79b3d3c80 4 rocksdb: Options.delayed_write_rate : 16777216 2024-03-11T20:40:11.219 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.615+0000 7fb79b3d3c80 4 rocksdb: Options.max_total_wal_size: 0 2024-03-11T20:40:11.219 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.615+0000 7fb79b3d3c80 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-11T20:40:11.219 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.615+0000 7fb79b3d3c80 4 rocksdb: Options.stats_dump_period_sec: 600 2024-03-11T20:40:11.219 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.615+0000 7fb79b3d3c80 4 rocksdb: Options.stats_persist_period_sec: 600 2024-03-11T20:40:11.220 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.615+0000 7fb79b3d3c80 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-11T20:40:11.220 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.615+0000 7fb79b3d3c80 4 rocksdb: Options.max_open_files: -1 2024-03-11T20:40:11.220 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.615+0000 7fb79b3d3c80 4 rocksdb: Options.bytes_per_sync: 0 2024-03-11T20:40:11.220 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.615+0000 7fb79b3d3c80 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-03-11T20:40:11.220 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.615+0000 7fb79b3d3c80 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-03-11T20:40:11.220 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.615+0000 7fb79b3d3c80 4 rocksdb: Options.compaction_readahead_size: 0 2024-03-11T20:40:11.220 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.615+0000 7fb79b3d3c80 4 rocksdb: Options.max_background_flushes: -1 2024-03-11T20:40:11.220 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.615+0000 7fb79b3d3c80 4 rocksdb: Compression algorithms supported: 2024-03-11T20:40:11.220 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.615+0000 7fb79b3d3c80 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-11T20:40:11.220 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.615+0000 7fb79b3d3c80 4 rocksdb: kZSTD supported: 0 2024-03-11T20:40:11.220 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.615+0000 7fb79b3d3c80 4 rocksdb: kXpressCompression supported: 0 2024-03-11T20:40:11.221 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.615+0000 7fb79b3d3c80 4 rocksdb: kLZ4HCCompression supported: 1 2024-03-11T20:40:11.221 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.615+0000 7fb79b3d3c80 4 rocksdb: kZlibCompression supported: 1 2024-03-11T20:40:11.221 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.615+0000 7fb79b3d3c80 4 rocksdb: kSnappyCompression supported: 1 2024-03-11T20:40:11.221 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.615+0000 7fb79b3d3c80 4 rocksdb: kLZ4Compression supported: 1 2024-03-11T20:40:11.221 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.615+0000 7fb79b3d3c80 4 rocksdb: kBZip2Compression supported: 0 2024-03-11T20:40:11.221 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.615+0000 7fb79b3d3c80 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-03-11T20:40:11.221 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.615+0000 7fb79b3d3c80 4 rocksdb: DMutex implementation: pthread_mutex_t 2024-03-11T20:40:11.221 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.615+0000 7fb79b3d3c80 4 rocksdb: [db/db_impl/db_impl_open.cc:317] Creating manifest 1 2024-03-11T20:40:11.221 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr 2024-03-11T20:40:11.221 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.623+0000 7fb79b3d3c80 4 rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000001 2024-03-11T20:40:11.221 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr 2024-03-11T20:40:11.221 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.623+0000 7fb79b3d3c80 4 rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-11T20:40:11.221 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr 2024-03-11T20:40:11.221 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.623+0000 7fb79b3d3c80 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-11T20:40:11.221 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.623+0000 7fb79b3d3c80 4 rocksdb: Options.merge_operator: 2024-03-11T20:40:11.222 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.623+0000 7fb79b3d3c80 4 rocksdb: Options.compaction_filter: None 2024-03-11T20:40:11.222 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.623+0000 7fb79b3d3c80 4 rocksdb: Options.compaction_filter_factory: None 2024-03-11T20:40:11.222 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.623+0000 7fb79b3d3c80 4 rocksdb: Options.sst_partitioner_factory: None 2024-03-11T20:40:11.222 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.623+0000 7fb79b3d3c80 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-03-11T20:40:11.222 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.623+0000 7fb79b3d3c80 4 rocksdb: Options.table_factory: BlockBasedTable 2024-03-11T20:40:11.222 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.623+0000 7fb79b3d3c80 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x56429a966c60) 2024-03-11T20:40:11.222 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr cache_index_and_filter_blocks: 1 2024-03-11T20:40:11.222 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr cache_index_and_filter_blocks_with_high_priority: 0 2024-03-11T20:40:11.222 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-11T20:40:11.222 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr pin_top_level_index_and_filter: 1 2024-03-11T20:40:11.222 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr index_type: 0 2024-03-11T20:40:11.222 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr data_block_index_type: 0 2024-03-11T20:40:11.222 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr index_shortening: 1 2024-03-11T20:40:11.222 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr data_block_hash_table_util_ratio: 0.750000 2024-03-11T20:40:11.223 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr checksum: 4 2024-03-11T20:40:11.223 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr no_block_cache: 0 2024-03-11T20:40:11.223 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr block_cache: 0x56429a96d090 2024-03-11T20:40:11.223 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr block_cache_name: BinnedLRUCache 2024-03-11T20:40:11.223 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr block_cache_options: 2024-03-11T20:40:11.223 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr capacity : 536870912 2024-03-11T20:40:11.223 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr num_shard_bits : 4 2024-03-11T20:40:11.223 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr strict_capacity_limit : 0 2024-03-11T20:40:11.223 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr high_pri_pool_ratio: 0.000 2024-03-11T20:40:11.223 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr block_cache_compressed: (nil) 2024-03-11T20:40:11.223 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr persistent_cache: (nil) 2024-03-11T20:40:11.223 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr block_size: 4096 2024-03-11T20:40:11.223 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr block_size_deviation: 10 2024-03-11T20:40:11.223 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr block_restart_interval: 16 2024-03-11T20:40:11.224 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr index_block_restart_interval: 1 2024-03-11T20:40:11.224 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr metadata_block_size: 4096 2024-03-11T20:40:11.224 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr partition_filters: 0 2024-03-11T20:40:11.224 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr use_delta_encoding: 1 2024-03-11T20:40:11.224 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr filter_policy: bloomfilter 2024-03-11T20:40:11.224 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr whole_key_filtering: 1 2024-03-11T20:40:11.224 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr verify_compression: 0 2024-03-11T20:40:11.224 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr read_amp_bytes_per_bit: 0 2024-03-11T20:40:11.224 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr format_version: 5 2024-03-11T20:40:11.224 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr enable_index_compression: 1 2024-03-11T20:40:11.224 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr block_align: 0 2024-03-11T20:40:11.224 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr max_auto_readahead_size: 262144 2024-03-11T20:40:11.224 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr prepopulate_block_cache: 0 2024-03-11T20:40:11.224 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr initial_auto_readahead_size: 8192 2024-03-11T20:40:11.225 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr num_file_reads_for_auto_readahead: 2 2024-03-11T20:40:11.225 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr 2024-03-11T20:40:11.225 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.623+0000 7fb79b3d3c80 4 rocksdb: Options.write_buffer_size: 33554432 2024-03-11T20:40:11.225 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.623+0000 7fb79b3d3c80 4 rocksdb: Options.max_write_buffer_number: 2 2024-03-11T20:40:11.225 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.623+0000 7fb79b3d3c80 4 rocksdb: Options.compression: NoCompression 2024-03-11T20:40:11.225 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.623+0000 7fb79b3d3c80 4 rocksdb: Options.bottommost_compression: Disabled 2024-03-11T20:40:11.225 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.623+0000 7fb79b3d3c80 4 rocksdb: Options.prefix_extractor: nullptr 2024-03-11T20:40:11.225 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.623+0000 7fb79b3d3c80 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-11T20:40:11.225 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.623+0000 7fb79b3d3c80 4 rocksdb: Options.num_levels: 7 2024-03-11T20:40:11.225 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.623+0000 7fb79b3d3c80 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-11T20:40:11.225 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.623+0000 7fb79b3d3c80 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-11T20:40:11.225 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.623+0000 7fb79b3d3c80 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-11T20:40:11.225 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.623+0000 7fb79b3d3c80 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-11T20:40:11.225 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.623+0000 7fb79b3d3c80 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-11T20:40:11.225 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.623+0000 7fb79b3d3c80 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-11T20:40:11.226 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.623+0000 7fb79b3d3c80 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-11T20:40:11.226 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.623+0000 7fb79b3d3c80 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-11T20:40:11.226 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.623+0000 7fb79b3d3c80 4 rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-11T20:40:11.226 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.623+0000 7fb79b3d3c80 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-11T20:40:11.226 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.623+0000 7fb79b3d3c80 4 rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-11T20:40:11.226 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.623+0000 7fb79b3d3c80 4 rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-11T20:40:11.226 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.623+0000 7fb79b3d3c80 4 rocksdb: Options.compression_opts.window_bits: -14 2024-03-11T20:40:11.226 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.623+0000 7fb79b3d3c80 4 rocksdb: Options.compression_opts.level: 32767 2024-03-11T20:40:11.226 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.623+0000 7fb79b3d3c80 4 rocksdb: Options.compression_opts.strategy: 0 2024-03-11T20:40:11.226 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.623+0000 7fb79b3d3c80 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-11T20:40:11.226 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.623+0000 7fb79b3d3c80 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-11T20:40:11.226 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.623+0000 7fb79b3d3c80 4 rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-11T20:40:11.226 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.623+0000 7fb79b3d3c80 4 rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-11T20:40:11.226 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.623+0000 7fb79b3d3c80 4 rocksdb: Options.compression_opts.enabled: false 2024-03-11T20:40:11.227 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.623+0000 7fb79b3d3c80 4 rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-11T20:40:11.227 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.623+0000 7fb79b3d3c80 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-11T20:40:11.227 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.623+0000 7fb79b3d3c80 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-11T20:40:11.227 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.623+0000 7fb79b3d3c80 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-11T20:40:11.227 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.623+0000 7fb79b3d3c80 4 rocksdb: Options.target_file_size_base: 67108864 2024-03-11T20:40:11.227 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.623+0000 7fb79b3d3c80 4 rocksdb: Options.target_file_size_multiplier: 1 2024-03-11T20:40:11.227 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.623+0000 7fb79b3d3c80 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-11T20:40:11.227 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.623+0000 7fb79b3d3c80 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-11T20:40:11.227 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.623+0000 7fb79b3d3c80 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-11T20:40:11.227 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.623+0000 7fb79b3d3c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-11T20:40:11.227 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.623+0000 7fb79b3d3c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-11T20:40:11.227 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.623+0000 7fb79b3d3c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-11T20:40:11.227 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.623+0000 7fb79b3d3c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-11T20:40:11.227 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.623+0000 7fb79b3d3c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-11T20:40:11.228 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.623+0000 7fb79b3d3c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-11T20:40:11.228 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.623+0000 7fb79b3d3c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-11T20:40:11.228 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.623+0000 7fb79b3d3c80 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-11T20:40:11.228 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.623+0000 7fb79b3d3c80 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-11T20:40:11.228 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.623+0000 7fb79b3d3c80 4 rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-11T20:40:11.228 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.623+0000 7fb79b3d3c80 4 rocksdb: Options.arena_block_size: 1048576 2024-03-11T20:40:11.228 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.623+0000 7fb79b3d3c80 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-11T20:40:11.228 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.623+0000 7fb79b3d3c80 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-11T20:40:11.228 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.623+0000 7fb79b3d3c80 4 rocksdb: Options.disable_auto_compactions: 0 2024-03-11T20:40:11.228 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.623+0000 7fb79b3d3c80 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-11T20:40:11.228 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.623+0000 7fb79b3d3c80 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-11T20:40:11.228 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.623+0000 7fb79b3d3c80 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-11T20:40:11.228 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.623+0000 7fb79b3d3c80 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-11T20:40:11.228 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.623+0000 7fb79b3d3c80 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-11T20:40:11.229 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.623+0000 7fb79b3d3c80 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-11T20:40:11.229 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.623+0000 7fb79b3d3c80 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-11T20:40:11.229 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.623+0000 7fb79b3d3c80 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-11T20:40:11.229 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.623+0000 7fb79b3d3c80 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-11T20:40:11.229 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.623+0000 7fb79b3d3c80 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-11T20:40:11.229 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.623+0000 7fb79b3d3c80 4 rocksdb: Options.table_properties_collectors: 2024-03-11T20:40:11.229 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.623+0000 7fb79b3d3c80 4 rocksdb: Options.inplace_update_support: 0 2024-03-11T20:40:11.229 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.623+0000 7fb79b3d3c80 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-03-11T20:40:11.229 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.623+0000 7fb79b3d3c80 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-11T20:40:11.229 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.623+0000 7fb79b3d3c80 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-11T20:40:11.229 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.623+0000 7fb79b3d3c80 4 rocksdb: Options.memtable_huge_page_size: 0 2024-03-11T20:40:11.229 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.623+0000 7fb79b3d3c80 4 rocksdb: Options.bloom_locality: 0 2024-03-11T20:40:11.229 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.623+0000 7fb79b3d3c80 4 rocksdb: Options.max_successive_merges: 0 2024-03-11T20:40:11.229 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.623+0000 7fb79b3d3c80 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-03-11T20:40:11.229 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.623+0000 7fb79b3d3c80 4 rocksdb: Options.paranoid_file_checks: 0 2024-03-11T20:40:11.230 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.623+0000 7fb79b3d3c80 4 rocksdb: Options.force_consistency_checks: 1 2024-03-11T20:40:11.230 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.623+0000 7fb79b3d3c80 4 rocksdb: Options.report_bg_io_stats: 0 2024-03-11T20:40:11.230 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.623+0000 7fb79b3d3c80 4 rocksdb: Options.ttl: 2592000 2024-03-11T20:40:11.230 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.623+0000 7fb79b3d3c80 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-03-11T20:40:11.230 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.623+0000 7fb79b3d3c80 4 rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-11T20:40:11.230 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.623+0000 7fb79b3d3c80 4 rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-11T20:40:11.230 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.623+0000 7fb79b3d3c80 4 rocksdb: Options.enable_blob_files: false 2024-03-11T20:40:11.230 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.623+0000 7fb79b3d3c80 4 rocksdb: Options.min_blob_size: 0 2024-03-11T20:40:11.230 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.623+0000 7fb79b3d3c80 4 rocksdb: Options.blob_file_size: 268435456 2024-03-11T20:40:11.230 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.623+0000 7fb79b3d3c80 4 rocksdb: Options.blob_compression_type: NoCompression 2024-03-11T20:40:11.230 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.623+0000 7fb79b3d3c80 4 rocksdb: Options.enable_blob_garbage_collection: false 2024-03-11T20:40:11.230 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.623+0000 7fb79b3d3c80 4 rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-11T20:40:11.230 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.623+0000 7fb79b3d3c80 4 rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-11T20:40:11.231 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.623+0000 7fb79b3d3c80 4 rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-11T20:40:11.231 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.623+0000 7fb79b3d3c80 4 rocksdb: Options.blob_file_starting_level: 0 2024-03-11T20:40:11.231 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.623+0000 7fb79b3d3c80 4 rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-11T20:40:11.231 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.623+0000 7fb79b3d3c80 4 rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-11T20:40:11.231 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr 2024-03-11T20:40:11.231 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.623+0000 7fb79b3d3c80 4 rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-11T20:40:11.231 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr 2024-03-11T20:40:11.231 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.623+0000 7fb79b3d3c80 4 rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: dcba1c13-b081-4fc0-ad11-9a30e8d94779 2024-03-11T20:40:11.231 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr 2024-03-11T20:40:11.231 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.623+0000 7fb79b3d3c80 4 rocksdb: [db/version_set.cc:5047] Creating manifest 5 2024-03-11T20:40:11.231 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr 2024-03-11T20:40:11.231 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.635+0000 7fb79b3d3c80 4 rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x56429aa3a000 2024-03-11T20:40:11.231 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.635+0000 7fb79b3d3c80 4 rocksdb: DB pointer 0x56429aa2a000 2024-03-11T20:40:11.231 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.635+0000 7fb78b544700 4 rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-11T20:40:11.231 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.635+0000 7fb78b544700 4 rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-11T20:40:11.232 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr ** DB Stats ** 2024-03-11T20:40:11.232 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr Uptime(secs): 0.0 total, 0.0 interval 2024-03-11T20:40:11.232 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-11T20:40:11.232 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-11T20:40:11.232 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-11T20:40:11.232 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-11T20:40:11.232 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-11T20:40:11.232 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-11T20:40:11.232 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr 2024-03-11T20:40:11.232 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr ** Compaction Stats [default] ** 2024-03-11T20:40:11.232 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-11T20:40:11.232 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-11T20:40:11.232 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-03-11T20:40:11.232 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-03-11T20:40:11.233 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr 2024-03-11T20:40:11.233 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr ** Compaction Stats [default] ** 2024-03-11T20:40:11.233 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-11T20:40:11.233 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-11T20:40:11.233 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr 2024-03-11T20:40:11.233 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-11T20:40:11.233 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr 2024-03-11T20:40:11.233 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr Uptime(secs): 0.0 total, 0.0 interval 2024-03-11T20:40:11.233 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr Flush(GB): cumulative 0.000, interval 0.000 2024-03-11T20:40:11.233 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr AddFile(GB): cumulative 0.000, interval 0.000 2024-03-11T20:40:11.233 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr AddFile(Total Files): cumulative 0, interval 0 2024-03-11T20:40:11.233 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr AddFile(L0 Files): cumulative 0, interval 0 2024-03-11T20:40:11.233 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr AddFile(Keys): cumulative 0, interval 0 2024-03-11T20:40:11.233 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-11T20:40:11.234 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-11T20:40:11.234 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-11T20:40:11.234 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr Block cache BinnedLRUCache@0x56429a96d090#7 capacity: 512.00 MB usage: 0.00 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.2e-05 secs_since: 0 2024-03-11T20:40:11.234 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr Block cache entry stats(count,size,portion): Misc(1,0.00 KB,0%) 2024-03-11T20:40:11.234 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr 2024-03-11T20:40:11.234 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr ** File Read Latency Histogram By Level [default] ** 2024-03-11T20:40:11.234 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr 2024-03-11T20:40:11.234 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.639+0000 7fb79b3d3c80 4 rocksdb: [db/db_impl/db_impl.cc:496] Shutdown: canceling all background work 2024-03-11T20:40:11.234 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.639+0000 7fb79b3d3c80 4 rocksdb: [db/db_impl/db_impl.cc:704] Shutdown complete 2024-03-11T20:40:11.234 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-mon: stderr debug 2024-03-11T20:40:10.639+0000 7fb79b3d3c80 0 /usr/bin/ceph-mon: created monfs at /var/lib/ceph/mon/ceph-a for mon.a 2024-03-11T20:40:11.235 INFO:teuthology.orchestra.run.smithi023.stdout:create mon.a on 2024-03-11T20:40:11.467 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stderr Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2024-03-11T20:40:11.682 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /etc/systemd/system/ceph.target. 2024-03-11T20:40:11.894 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715.target -> /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715.target. 2024-03-11T20:40:11.895 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715.target -> /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715.target. 2024-03-11T20:40:12.168 INFO:teuthology.orchestra.run.smithi023.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@mon.a 2024-03-11T20:40:12.168 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stderr Failed to reset failed state of unit ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@mon.a.service: Unit ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@mon.a.service not loaded. 2024-03-11T20:40:12.386 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715.target.wants/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@mon.a.service -> /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service. 2024-03-11T20:40:12.396 INFO:teuthology.orchestra.run.smithi023.stdout:firewalld does not appear to be present 2024-03-11T20:40:12.396 INFO:teuthology.orchestra.run.smithi023.stdout:Not possible to enable service . firewalld.service is not available 2024-03-11T20:40:12.396 INFO:teuthology.orchestra.run.smithi023.stdout:Waiting for mon to start... 2024-03-11T20:40:12.396 INFO:teuthology.orchestra.run.smithi023.stdout:Waiting for mon... 2024-03-11T20:40:12.411 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:12 smithi023 systemd[1]: Started Ceph mon.a for 67ba00f8-dfe7-11ee-95c8-87774f69a715. 2024-03-11T20:40:13.782 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:13 smithi023 bash[20885]: cluster 2024-03-11T20:40:13.664976+0000 mon.a (mon.0) 0 : cluster 1 mkfs 67ba00f8-dfe7-11ee-95c8-87774f69a715 2024-03-11T20:40:13.782 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:13 smithi023 bash[20885]: cluster 2024-03-11T20:40:13.635875+0000 mon.a (mon.0) 1 : cluster 1 mon.a is new leader, mons a in quorum (ranks 0) 2024-03-11T20:40:14.392 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout cluster: 2024-03-11T20:40:14.392 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout id: 67ba00f8-dfe7-11ee-95c8-87774f69a715 2024-03-11T20:40:14.392 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-11T20:40:14.392 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 2024-03-11T20:40:14.392 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout services: 2024-03-11T20:40:14.392 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.194192s) 2024-03-11T20:40:14.392 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-11T20:40:14.392 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-11T20:40:14.392 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 2024-03-11T20:40:14.393 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout data: 2024-03-11T20:40:14.393 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-11T20:40:14.393 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-11T20:40:14.393 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-11T20:40:14.393 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout pgs: 2024-03-11T20:40:14.393 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 2024-03-11T20:40:14.393 INFO:teuthology.orchestra.run.smithi023.stdout:mon is available 2024-03-11T20:40:14.393 INFO:teuthology.orchestra.run.smithi023.stdout:Assimilating anything we can from ceph.conf... 2024-03-11T20:40:14.985 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:14 smithi023 bash[20885]: cluster 2024-03-11T20:40:13.663850+0000 mon.a (mon.0) 2 : cluster 1 mon.a is new leader, mons a in quorum (ranks 0) 2024-03-11T20:40:14.985 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:14 smithi023 bash[20885]: cluster 2024-03-11T20:40:13.664157+0000 mon.a (mon.0) 3 : cluster 0 monmap e1: 1 mons at {a=[v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T20:40:14.985 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:14 smithi023 bash[20885]: cluster 2024-03-11T20:40:13.666657+0000 mon.a (mon.0) 4 : cluster 0 fsmap 2024-03-11T20:40:14.985 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:14 smithi023 bash[20885]: cluster 2024-03-11T20:40:13.669064+0000 mon.a (mon.0) 5 : cluster 0 osdmap e1: 0 total, 0 up, 0 in 2024-03-11T20:40:14.985 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:14 smithi023 bash[20885]: cluster 2024-03-11T20:40:13.669548+0000 mon.a (mon.0) 6 : cluster 0 mgrmap e1: no daemons active 2024-03-11T20:40:14.985 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:14 smithi023 bash[20885]: audit 2024-03-11T20:40:13.857994+0000 mon.a (mon.0) 7 : audit 0 from='client.? 172.21.15.23:0/2713205701' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-11T20:40:15.898 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 2024-03-11T20:40:15.898 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout [global] 2024-03-11T20:40:15.898 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout fsid = 67ba00f8-dfe7-11ee-95c8-87774f69a715 2024-03-11T20:40:15.898 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.23:3300,v1:172.21.15.23:6789] 2024-03-11T20:40:15.898 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-11T20:40:15.898 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-11T20:40:15.898 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-11T20:40:15.898 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-11T20:40:15.899 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 2024-03-11T20:40:15.899 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-11T20:40:15.899 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-11T20:40:15.899 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 2024-03-11T20:40:15.899 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout [osd] 2024-03-11T20:40:15.899 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-11T20:40:15.899 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-11T20:40:15.899 INFO:teuthology.orchestra.run.smithi023.stdout:Generating new minimal ceph.conf... 2024-03-11T20:40:16.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:15 smithi023 bash[20885]: audit 2024-03-11T20:40:15.329419+0000 mon.a (mon.0) 8 : audit 1 from='client.? 172.21.15.23:0/613220886' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-11T20:40:16.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:15 smithi023 bash[20885]: audit 2024-03-11T20:40:15.334970+0000 mon.a (mon.0) 9 : audit 1 from='client.? 172.21.15.23:0/613220886' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-11T20:40:17.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:16 smithi023 bash[20885]: audit 2024-03-11T20:40:16.825437+0000 mon.a (mon.0) 10 : audit 0 from='client.? 172.21.15.23:0/254802394' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:40:17.352 INFO:teuthology.orchestra.run.smithi023.stdout:Restarting the monitor... 2024-03-11T20:40:17.740 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:17 smithi023 systemd[1]: Stopping Ceph mon.a for 67ba00f8-dfe7-11ee-95c8-87774f69a715... 2024-03-11T20:40:17.740 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:17 smithi023 bash[20885]: debug 2024-03-11T20:40:17.539+0000 7fc7d40de700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-03-11T20:40:17.740 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:17 smithi023 bash[20885]: debug 2024-03-11T20:40:17.539+0000 7fc7d40de700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-11T20:40:18.305 INFO:teuthology.orchestra.run.smithi023.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-11T20:40:18.532 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:18 smithi023 bash[21272]: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-mon-a 2024-03-11T20:40:18.532 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:18 smithi023 bash[21334]: Error response from daemon: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-mon-a 2024-03-11T20:40:18.532 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:18 smithi023 systemd[1]: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-11T20:40:18.532 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:18 smithi023 systemd[1]: Stopped Ceph mon.a for 67ba00f8-dfe7-11ee-95c8-87774f69a715. 2024-03-11T20:40:18.532 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:18 smithi023 systemd[1]: Started Ceph mon.a for 67ba00f8-dfe7-11ee-95c8-87774f69a715. 2024-03-11T20:40:19.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.375+0000 7f553316ec80 0 set uid:gid to 167:167 (ceph:ceph) 2024-03-11T20:40:19.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.375+0000 7f553316ec80 0 ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev), process ceph-mon, pid 7 2024-03-11T20:40:19.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.375+0000 7f553316ec80 0 pidfile_write: ignore empty --pid-file 2024-03-11T20:40:19.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.379+0000 7f553316ec80 0 load: jerasure load: lrc 2024-03-11T20:40:19.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: RocksDB version: 7.9.2 2024-03-11T20:40:19.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Git sha 0 2024-03-11T20:40:19.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Compile date 2024-03-10 23:25:48 2024-03-11T20:40:19.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: DB SUMMARY 2024-03-11T20:40:19.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: DB Session ID: 44CNM7V0MXLUE5BR8DL3 2024-03-11T20:40:19.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: CURRENT file: CURRENT 2024-03-11T20:40:19.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: IDENTITY file: IDENTITY 2024-03-11T20:40:19.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-03-11T20:40:19.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-03-11T20:40:19.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 90797 ; 2024-03-11T20:40:19.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.error_if_exists: 0 2024-03-11T20:40:19.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.create_if_missing: 0 2024-03-11T20:40:19.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.paranoid_checks: 1 2024-03-11T20:40:19.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.flush_verify_memtable_count: 1 2024-03-11T20:40:19.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-11T20:40:19.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-11T20:40:19.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.env: 0x55b8578ceb20 2024-03-11T20:40:19.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.fs: PosixFileSystem 2024-03-11T20:40:19.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.info_log: 0x55b859f17920 2024-03-11T20:40:19.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.max_file_opening_threads: 16 2024-03-11T20:40:19.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.statistics: (nil) 2024-03-11T20:40:19.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.use_fsync: 0 2024-03-11T20:40:19.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.max_log_file_size: 0 2024-03-11T20:40:19.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-11T20:40:19.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.log_file_time_to_roll: 0 2024-03-11T20:40:19.536 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.keep_log_file_num: 1000 2024-03-11T20:40:19.536 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.recycle_log_file_num: 0 2024-03-11T20:40:19.536 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.allow_fallocate: 1 2024-03-11T20:40:19.536 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.allow_mmap_reads: 0 2024-03-11T20:40:19.536 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.allow_mmap_writes: 0 2024-03-11T20:40:19.536 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.use_direct_reads: 0 2024-03-11T20:40:19.536 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-11T20:40:19.536 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.create_missing_column_families: 0 2024-03-11T20:40:19.536 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.db_log_dir: 2024-03-11T20:40:19.536 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.wal_dir: 2024-03-11T20:40:19.536 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.table_cache_numshardbits: 6 2024-03-11T20:40:19.536 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-03-11T20:40:19.536 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-03-11T20:40:19.537 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-11T20:40:19.537 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-11T20:40:19.537 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-03-11T20:40:19.537 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.advise_random_on_open: 1 2024-03-11T20:40:19.537 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.db_write_buffer_size: 0 2024-03-11T20:40:19.537 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.write_buffer_manager: 0x55b859eeb400 2024-03-11T20:40:19.537 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-11T20:40:19.537 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-11T20:40:19.537 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.use_adaptive_mutex: 0 2024-03-11T20:40:19.537 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.rate_limiter: (nil) 2024-03-11T20:40:19.537 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-11T20:40:19.537 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.wal_recovery_mode: 2 2024-03-11T20:40:19.537 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.enable_thread_tracking: 0 2024-03-11T20:40:19.538 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.enable_pipelined_write: 0 2024-03-11T20:40:19.538 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.unordered_write: 0 2024-03-11T20:40:19.538 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-11T20:40:19.538 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-11T20:40:19.538 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-11T20:40:19.538 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-11T20:40:19.538 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.row_cache: None 2024-03-11T20:40:19.538 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.wal_filter: None 2024-03-11T20:40:19.538 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-11T20:40:19.538 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.allow_ingest_behind: 0 2024-03-11T20:40:19.538 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.two_write_queues: 0 2024-03-11T20:40:19.538 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.manual_wal_flush: 0 2024-03-11T20:40:19.538 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.wal_compression: 0 2024-03-11T20:40:19.538 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.atomic_flush: 0 2024-03-11T20:40:19.539 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-11T20:40:19.539 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.persist_stats_to_disk: 0 2024-03-11T20:40:19.539 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-03-11T20:40:19.539 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.log_readahead_size: 0 2024-03-11T20:40:19.539 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-11T20:40:19.539 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.best_efforts_recovery: 0 2024-03-11T20:40:19.539 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-11T20:40:19.539 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-11T20:40:19.539 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.allow_data_in_errors: 0 2024-03-11T20:40:19.539 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.db_host_id: __hostname__ 2024-03-11T20:40:19.539 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.enforce_single_del_contracts: true 2024-03-11T20:40:19.539 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.max_background_jobs: 2 2024-03-11T20:40:19.539 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.max_background_compactions: -1 2024-03-11T20:40:19.539 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.max_subcompactions: 1 2024-03-11T20:40:19.539 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-11T20:40:19.540 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-11T20:40:19.540 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.delayed_write_rate : 16777216 2024-03-11T20:40:19.540 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.max_total_wal_size: 0 2024-03-11T20:40:19.540 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-11T20:40:19.540 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.stats_dump_period_sec: 600 2024-03-11T20:40:19.540 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.stats_persist_period_sec: 600 2024-03-11T20:40:19.540 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-11T20:40:19.540 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.max_open_files: -1 2024-03-11T20:40:19.540 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.bytes_per_sync: 0 2024-03-11T20:40:19.540 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-03-11T20:40:19.540 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-03-11T20:40:19.540 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.compaction_readahead_size: 0 2024-03-11T20:40:19.540 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.max_background_flushes: -1 2024-03-11T20:40:19.540 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Compression algorithms supported: 2024-03-11T20:40:19.540 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-11T20:40:19.541 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: kZSTD supported: 0 2024-03-11T20:40:19.541 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: kXpressCompression supported: 0 2024-03-11T20:40:19.541 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: kLZ4HCCompression supported: 1 2024-03-11T20:40:19.541 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: kZlibCompression supported: 1 2024-03-11T20:40:19.541 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: kSnappyCompression supported: 1 2024-03-11T20:40:19.541 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: kLZ4Compression supported: 1 2024-03-11T20:40:19.541 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: kBZip2Compression supported: 0 2024-03-11T20:40:19.541 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-03-11T20:40:19.541 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: DMutex implementation: pthread_mutex_t 2024-03-11T20:40:19.541 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-03-11T20:40:19.542 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-11T20:40:19.542 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-11T20:40:19.543 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.merge_operator: 2024-03-11T20:40:19.543 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.compaction_filter: None 2024-03-11T20:40:19.543 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.compaction_filter_factory: None 2024-03-11T20:40:19.543 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.sst_partitioner_factory: None 2024-03-11T20:40:19.543 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-03-11T20:40:19.543 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.table_factory: BlockBasedTable 2024-03-11T20:40:19.543 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55b859f17a20) 2024-03-11T20:40:19.543 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: cache_index_and_filter_blocks: 1 2024-03-11T20:40:19.543 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-11T20:40:19.543 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-11T20:40:19.543 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: pin_top_level_index_and_filter: 1 2024-03-11T20:40:19.543 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: index_type: 0 2024-03-11T20:40:19.543 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: data_block_index_type: 0 2024-03-11T20:40:19.543 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: index_shortening: 1 2024-03-11T20:40:19.543 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: data_block_hash_table_util_ratio: 0.750000 2024-03-11T20:40:19.544 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: checksum: 4 2024-03-11T20:40:19.544 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: no_block_cache: 0 2024-03-11T20:40:19.544 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: block_cache: 0x55b859f1b090 2024-03-11T20:40:19.544 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: block_cache_name: BinnedLRUCache 2024-03-11T20:40:19.544 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: block_cache_options: 2024-03-11T20:40:19.544 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: capacity : 536870912 2024-03-11T20:40:19.544 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: num_shard_bits : 4 2024-03-11T20:40:19.544 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: strict_capacity_limit : 0 2024-03-11T20:40:19.544 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: high_pri_pool_ratio: 0.000 2024-03-11T20:40:19.544 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: block_cache_compressed: (nil) 2024-03-11T20:40:19.544 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: persistent_cache: (nil) 2024-03-11T20:40:19.544 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: block_size: 4096 2024-03-11T20:40:19.544 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: block_size_deviation: 10 2024-03-11T20:40:19.544 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: block_restart_interval: 16 2024-03-11T20:40:19.545 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: index_block_restart_interval: 1 2024-03-11T20:40:19.545 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: metadata_block_size: 4096 2024-03-11T20:40:19.545 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: partition_filters: 0 2024-03-11T20:40:19.545 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: use_delta_encoding: 1 2024-03-11T20:40:19.545 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: filter_policy: bloomfilter 2024-03-11T20:40:19.545 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: whole_key_filtering: 1 2024-03-11T20:40:19.545 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: verify_compression: 0 2024-03-11T20:40:19.545 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: read_amp_bytes_per_bit: 0 2024-03-11T20:40:19.545 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: format_version: 5 2024-03-11T20:40:19.545 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: enable_index_compression: 1 2024-03-11T20:40:19.545 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: block_align: 0 2024-03-11T20:40:19.545 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: max_auto_readahead_size: 262144 2024-03-11T20:40:19.545 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: prepopulate_block_cache: 0 2024-03-11T20:40:19.545 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: initial_auto_readahead_size: 8192 2024-03-11T20:40:19.546 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: num_file_reads_for_auto_readahead: 2 2024-03-11T20:40:19.546 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.write_buffer_size: 33554432 2024-03-11T20:40:19.546 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.max_write_buffer_number: 2 2024-03-11T20:40:19.546 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.compression: NoCompression 2024-03-11T20:40:19.546 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.bottommost_compression: Disabled 2024-03-11T20:40:19.546 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.prefix_extractor: nullptr 2024-03-11T20:40:19.546 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-11T20:40:19.546 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.num_levels: 7 2024-03-11T20:40:19.546 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-11T20:40:19.546 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-11T20:40:19.546 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-11T20:40:19.546 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-11T20:40:19.546 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-11T20:40:19.546 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-11T20:40:19.547 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-11T20:40:19.547 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-11T20:40:19.547 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-11T20:40:19.547 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-11T20:40:19.547 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-11T20:40:19.547 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-11T20:40:19.547 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.compression_opts.window_bits: -14 2024-03-11T20:40:19.547 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.compression_opts.level: 32767 2024-03-11T20:40:19.547 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.compression_opts.strategy: 0 2024-03-11T20:40:19.547 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-11T20:40:19.547 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-11T20:40:19.547 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-11T20:40:19.547 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-11T20:40:19.547 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.compression_opts.enabled: false 2024-03-11T20:40:19.547 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-11T20:40:19.548 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-11T20:40:19.548 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-11T20:40:19.548 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-11T20:40:19.548 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.target_file_size_base: 67108864 2024-03-11T20:40:19.548 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.target_file_size_multiplier: 1 2024-03-11T20:40:19.548 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-11T20:40:19.548 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-11T20:40:19.548 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-11T20:40:19.548 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-11T20:40:19.548 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-11T20:40:19.548 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-11T20:40:19.548 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-11T20:40:19.548 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-11T20:40:19.548 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-11T20:40:19.548 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-11T20:40:19.549 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-11T20:40:19.549 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-11T20:40:19.549 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-11T20:40:19.549 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.arena_block_size: 1048576 2024-03-11T20:40:19.549 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-11T20:40:19.549 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-11T20:40:19.549 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.disable_auto_compactions: 0 2024-03-11T20:40:19.550 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-11T20:40:19.550 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-11T20:40:19.550 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-11T20:40:19.550 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-11T20:40:19.550 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-11T20:40:19.550 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-11T20:40:19.550 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-11T20:40:19.550 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-11T20:40:19.550 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-11T20:40:19.550 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-11T20:40:19.550 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.table_properties_collectors: 2024-03-11T20:40:19.550 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.inplace_update_support: 0 2024-03-11T20:40:19.550 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-03-11T20:40:19.550 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-11T20:40:19.551 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-11T20:40:19.551 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.memtable_huge_page_size: 0 2024-03-11T20:40:19.551 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.bloom_locality: 0 2024-03-11T20:40:19.551 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.max_successive_merges: 0 2024-03-11T20:40:19.551 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-03-11T20:40:19.551 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.paranoid_file_checks: 0 2024-03-11T20:40:19.551 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.force_consistency_checks: 1 2024-03-11T20:40:19.551 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.report_bg_io_stats: 0 2024-03-11T20:40:19.551 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.ttl: 2592000 2024-03-11T20:40:19.551 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-03-11T20:40:19.551 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-11T20:40:19.551 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-11T20:40:19.551 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.enable_blob_files: false 2024-03-11T20:40:19.551 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.min_blob_size: 0 2024-03-11T20:40:19.551 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.blob_file_size: 268435456 2024-03-11T20:40:19.552 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.blob_compression_type: NoCompression 2024-03-11T20:40:19.552 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.enable_blob_garbage_collection: false 2024-03-11T20:40:19.552 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-11T20:40:19.552 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-11T20:40:19.552 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-11T20:40:19.552 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.blob_file_starting_level: 0 2024-03-11T20:40:19.552 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.383+0000 7f553316ec80 4 rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-11T20:40:19.552 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.387+0000 7f553316ec80 4 rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-03-11T20:40:19.552 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.387+0000 7f553316ec80 4 rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-03-11T20:40:19.552 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.387+0000 7f553316ec80 4 rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: dcba1c13-b081-4fc0-ad11-9a30e8d94779 2024-03-11T20:40:19.552 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.387+0000 7f553316ec80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1710189619387703, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-03-11T20:40:19.552 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.387+0000 7f553316ec80 4 rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-03-11T20:40:19.552 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.387+0000 7f553316ec80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1710189619388776, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 86792, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 278, "table_properties": {"data_size": 84874, "index_size": 264, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 645, "raw_key_size": 11580, "raw_average_key_size": 48, "raw_value_size": 78495, "raw_average_value_size": 331, "num_data_blocks": 11, "num_entries": 237, "num_filter_entries": 237, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1710189619, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "dcba1c13-b081-4fc0-ad11-9a30e8d94779", "db_session_id": "44CNM7V0MXLUE5BR8DL3", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-03-11T20:40:19.552 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.387+0000 7f553316ec80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1710189619388856, "job": 1, "event": "recovery_finished"} 2024-03-11T20:40:19.553 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.387+0000 7f553316ec80 4 rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-03-11T20:40:19.553 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.387+0000 7f553316ec80 4 rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-11T20:40:19.553 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.387+0000 7f553316ec80 4 rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55b85a012000 2024-03-11T20:40:19.553 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.387+0000 7f553316ec80 4 rocksdb: DB pointer 0x55b859ffc000 2024-03-11T20:40:19.553 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.387+0000 7f55218d7700 4 rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-11T20:40:19.553 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.387+0000 7f55218d7700 4 rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-11T20:40:19.553 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: ** DB Stats ** 2024-03-11T20:40:19.553 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: Uptime(secs): 0.0 total, 0.0 interval 2024-03-11T20:40:19.553 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-11T20:40:19.553 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-11T20:40:19.553 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-11T20:40:19.553 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-11T20:40:19.553 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-11T20:40:19.553 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-11T20:40:19.553 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: ** Compaction Stats [default] ** 2024-03-11T20:40:19.554 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-11T20:40:19.554 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-11T20:40:19.554 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: L0 2/0 86.59 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 124.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-11T20:40:19.554 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: Sum 2/0 86.59 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 124.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-11T20:40:19.554 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 124.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-11T20:40:19.554 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: ** Compaction Stats [default] ** 2024-03-11T20:40:19.554 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-11T20:40:19.554 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-11T20:40:19.554 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 124.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-11T20:40:19.554 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-11T20:40:19.554 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: Uptime(secs): 0.0 total, 0.0 interval 2024-03-11T20:40:19.554 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: Flush(GB): cumulative 0.000, interval 0.000 2024-03-11T20:40:19.554 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-11T20:40:19.554 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: AddFile(Total Files): cumulative 0, interval 0 2024-03-11T20:40:19.555 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: AddFile(L0 Files): cumulative 0, interval 0 2024-03-11T20:40:19.555 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: AddFile(Keys): cumulative 0, interval 0 2024-03-11T20:40:19.555 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: Cumulative compaction: 0.00 GB write, 12.66 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-11T20:40:19.555 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: Interval compaction: 0.00 GB write, 12.66 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-11T20:40:19.555 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-11T20:40:19.555 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: Block cache BinnedLRUCache@0x55b859f1b090#7 capacity: 512.00 MB usage: 1.30 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.1e-05 secs_since: 0 2024-03-11T20:40:19.555 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: Block cache entry stats(count,size,portion): FilterBlock(2,0.83 KB,0.000157952%) IndexBlock(2,0.47 KB,8.9407e-05%) Misc(1,0.00 KB,0%) 2024-03-11T20:40:19.555 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: ** File Read Latency Histogram By Level [default] ** 2024-03-11T20:40:19.555 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.387+0000 7f553316ec80 0 starting mon.a rank 0 at public addrs [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] at bind addrs [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 2024-03-11T20:40:19.555 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.391+0000 7f553316ec80 1 mon.a@-1(???) e1 preinit fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 2024-03-11T20:40:19.555 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.391+0000 7f553316ec80 0 mon.a@-1(???).mds e1 new map 2024-03-11T20:40:19.555 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.391+0000 7f553316ec80 0 mon.a@-1(???).mds e1 print_map 2024-03-11T20:40:19.555 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: e1 2024-03-11T20:40:19.555 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: enable_multiple, ever_enabled_multiple: 1,1 2024-03-11T20:40:19.556 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-11T20:40:19.556 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: legacy client fscid: -1 2024-03-11T20:40:19.556 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: 2024-03-11T20:40:19.556 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: No filesystems configured 2024-03-11T20:40:19.556 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.391+0000 7f553316ec80 0 mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-11T20:40:19.556 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.391+0000 7f553316ec80 0 mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T20:40:19.556 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.391+0000 7f553316ec80 0 mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T20:40:19.556 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.391+0000 7f553316ec80 0 mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T20:40:19.556 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: debug 2024-03-11T20:40:19.391+0000 7f553316ec80 1 mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-11T20:40:19.556 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: cluster 2024-03-11T20:40:19.396497+0000 mon.a (mon.0) 1 : cluster 1 mon.a is new leader, mons a in quorum (ranks 0) 2024-03-11T20:40:19.556 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: cluster 2024-03-11T20:40:19.396585+0000 mon.a (mon.0) 2 : cluster 0 monmap e1: 1 mons at {a=[v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T20:40:19.556 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: cluster 2024-03-11T20:40:19.397175+0000 mon.a (mon.0) 3 : cluster 0 fsmap 2024-03-11T20:40:19.556 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: cluster 2024-03-11T20:40:19.397232+0000 mon.a (mon.0) 4 : cluster 0 osdmap e1: 0 total, 0 up, 0 in 2024-03-11T20:40:19.556 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:19 smithi023 bash[21372]: cluster 2024-03-11T20:40:19.397786+0000 mon.a (mon.0) 5 : cluster 0 mgrmap e1: no daemons active 2024-03-11T20:40:20.136 INFO:teuthology.orchestra.run.smithi023.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-11T20:40:20.162 INFO:teuthology.orchestra.run.smithi023.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-11T20:40:20.162 INFO:teuthology.orchestra.run.smithi023.stdout:Creating mgr... 2024-03-11T20:40:20.162 INFO:teuthology.orchestra.run.smithi023.stdout:Verifying port 0.0.0.0:9283 ... 2024-03-11T20:40:20.162 INFO:teuthology.orchestra.run.smithi023.stdout:Verifying port 0.0.0.0:8765 ... 2024-03-11T20:40:20.512 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:20 smithi023 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:40:20.521 INFO:teuthology.orchestra.run.smithi023.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@mgr.y 2024-03-11T20:40:20.521 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stderr Failed to reset failed state of unit ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@mgr.y.service: Unit ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@mgr.y.service not loaded. 2024-03-11T20:40:20.729 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715.target.wants/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@mgr.y.service -> /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service. 2024-03-11T20:40:20.736 INFO:teuthology.orchestra.run.smithi023.stdout:firewalld does not appear to be present 2024-03-11T20:40:20.736 INFO:teuthology.orchestra.run.smithi023.stdout:Not possible to enable service . firewalld.service is not available 2024-03-11T20:40:20.736 INFO:teuthology.orchestra.run.smithi023.stdout:firewalld does not appear to be present 2024-03-11T20:40:20.736 INFO:teuthology.orchestra.run.smithi023.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-03-11T20:40:20.736 INFO:teuthology.orchestra.run.smithi023.stdout:Waiting for mgr to start... 2024-03-11T20:40:20.737 INFO:teuthology.orchestra.run.smithi023.stdout:Waiting for mgr... 2024-03-11T20:40:20.774 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:20 smithi023 bash[21372]: audit 2024-03-11T20:40:19.582419+0000 mon.a (mon.0) 6 : audit 1 from='client.? 172.21.15.23:0/594073799' entity='client.admin' 2024-03-11T20:40:20.774 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:20 smithi023 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:40:22.532 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:22 smithi023 bash[21372]: audit 2024-03-11T20:40:22.113203+0000 mon.a (mon.0) 7 : audit 0 from='client.? 172.21.15.23:0/2105442738' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-11T20:40:22.532 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:40:22 smithi023 bash[21627]: debug 2024-03-11T20:40:22.067+0000 7f27b3713200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T20:40:22.633 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 2024-03-11T20:40:22.633 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout { 2024-03-11T20:40:22.633 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "fsid": "67ba00f8-dfe7-11ee-95c8-87774f69a715", 2024-03-11T20:40:22.633 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "health": { 2024-03-11T20:40:22.634 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-11T20:40:22.634 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-11T20:40:22.634 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-11T20:40:22.634 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-11T20:40:22.634 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-11T20:40:22.634 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-11T20:40:22.634 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 0 2024-03-11T20:40:22.634 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout ], 2024-03-11T20:40:22.635 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-11T20:40:22.635 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "a" 2024-03-11T20:40:22.635 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout ], 2024-03-11T20:40:22.635 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-03-11T20:40:22.635 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-11T20:40:22.635 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T20:40:22.635 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-11T20:40:22.635 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-11T20:40:22.635 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-11T20:40:22.635 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-11T20:40:22.635 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T20:40:22.635 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-11T20:40:22.635 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-11T20:40:22.635 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-11T20:40:22.636 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-11T20:40:22.636 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-11T20:40:22.636 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-11T20:40:22.636 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-11T20:40:22.636 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-11T20:40:22.636 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-11T20:40:22.636 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-11T20:40:22.636 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-11T20:40:22.636 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-11T20:40:22.636 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-11T20:40:22.636 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-11T20:40:22.636 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-11T20:40:22.637 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-11T20:40:22.637 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-11T20:40:22.637 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-11T20:40:22.637 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T20:40:22.637 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-11T20:40:22.637 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-11T20:40:22.637 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-11T20:40:22.637 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-11T20:40:22.637 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-11T20:40:22.637 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-11T20:40:22.637 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-11T20:40:22.637 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-11T20:40:22.637 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-11T20:40:22.637 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "restful" 2024-03-11T20:40:22.637 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout ], 2024-03-11T20:40:22.638 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T20:40:22.638 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-11T20:40:22.638 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-11T20:40:22.638 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T20:40:22.638 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "modified": "2024-03-11T20:40:13.665433+0000", 2024-03-11T20:40:22.638 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T20:40:22.638 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-11T20:40:22.638 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-11T20:40:22.638 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout } 2024-03-11T20:40:22.638 INFO:teuthology.orchestra.run.smithi023.stdout:mgr not available, waiting (1/15)... 2024-03-11T20:40:24.032 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:40:23 smithi023 bash[21627]: debug 2024-03-11T20:40:23.699+0000 7f27b3713200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T20:40:24.032 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:40:23 smithi023 bash[21627]: debug 2024-03-11T20:40:23.799+0000 7f27b3713200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T20:40:24.532 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:40:24 smithi023 bash[21627]: debug 2024-03-11T20:40:24.051+0000 7f27b3713200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T20:40:24.989 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:40:24 smithi023 bash[21627]: debug 2024-03-11T20:40:24.879+0000 7f27b3713200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T20:40:25.279 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:40:24 smithi023 bash[21627]: debug 2024-03-11T20:40:24.991+0000 7f27b3713200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T20:40:25.280 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:40:25 smithi023 bash[21627]: debug 2024-03-11T20:40:25.099+0000 7f27b3713200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T20:40:25.532 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:40:25 smithi023 bash[21627]: debug 2024-03-11T20:40:25.467+0000 7f27b3713200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-11T20:40:26.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:25 smithi023 bash[21372]: audit 2024-03-11T20:40:25.614598+0000 mon.a (mon.0) 8 : audit 0 from='client.? 172.21.15.23:0/3045087363' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-11T20:40:26.033 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:40:25 smithi023 bash[21627]: debug 2024-03-11T20:40:25.583+0000 7f27b3713200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T20:40:26.033 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:40:25 smithi023 bash[21627]: debug 2024-03-11T20:40:25.691+0000 7f27b3713200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T20:40:26.239 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 2024-03-11T20:40:26.239 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout { 2024-03-11T20:40:26.239 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "fsid": "67ba00f8-dfe7-11ee-95c8-87774f69a715", 2024-03-11T20:40:26.239 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "health": { 2024-03-11T20:40:26.239 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-11T20:40:26.239 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-11T20:40:26.239 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-11T20:40:26.239 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-11T20:40:26.239 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-11T20:40:26.239 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-11T20:40:26.239 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 0 2024-03-11T20:40:26.239 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout ], 2024-03-11T20:40:26.239 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-11T20:40:26.240 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "a" 2024-03-11T20:40:26.240 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout ], 2024-03-11T20:40:26.240 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "quorum_age": 6, 2024-03-11T20:40:26.240 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-11T20:40:26.240 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T20:40:26.240 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-11T20:40:26.240 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-11T20:40:26.240 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-11T20:40:26.240 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-11T20:40:26.240 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T20:40:26.240 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-11T20:40:26.240 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-11T20:40:26.240 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-11T20:40:26.240 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-11T20:40:26.240 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-11T20:40:26.241 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-11T20:40:26.241 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-11T20:40:26.241 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-11T20:40:26.241 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-11T20:40:26.241 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-11T20:40:26.241 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-11T20:40:26.241 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-11T20:40:26.241 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-11T20:40:26.241 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-11T20:40:26.241 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-11T20:40:26.241 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-11T20:40:26.241 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-11T20:40:26.242 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-11T20:40:26.242 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T20:40:26.242 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-11T20:40:26.242 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-11T20:40:26.242 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-11T20:40:26.242 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-11T20:40:26.242 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-11T20:40:26.242 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-11T20:40:26.242 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-11T20:40:26.242 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-11T20:40:26.242 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-11T20:40:26.242 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "restful" 2024-03-11T20:40:26.242 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout ], 2024-03-11T20:40:26.243 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T20:40:26.243 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-11T20:40:26.243 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-11T20:40:26.243 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T20:40:26.243 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "modified": "2024-03-11T20:40:13.665433+0000", 2024-03-11T20:40:26.243 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T20:40:26.243 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-11T20:40:26.243 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-11T20:40:26.243 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout } 2024-03-11T20:40:26.243 INFO:teuthology.orchestra.run.smithi023.stdout:mgr not available, waiting (2/15)... 2024-03-11T20:40:26.532 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:40:26 smithi023 bash[21627]: debug 2024-03-11T20:40:26.059+0000 7f27b3713200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T20:40:26.532 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:40:26 smithi023 bash[21627]: debug 2024-03-11T20:40:26.303+0000 7f27b3713200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T20:40:26.962 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:40:26 smithi023 bash[21627]: debug 2024-03-11T20:40:26.571+0000 7f27b3713200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T20:40:26.962 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:40:26 smithi023 bash[21627]: debug 2024-03-11T20:40:26.683+0000 7f27b3713200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T20:40:27.282 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:40:26 smithi023 bash[21627]: debug 2024-03-11T20:40:26.967+0000 7f27b3713200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T20:40:27.282 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:40:27 smithi023 bash[21627]: debug 2024-03-11T20:40:27.215+0000 7f27b3713200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T20:40:28.002 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:40:27 smithi023 bash[21627]: debug 2024-03-11T20:40:27.631+0000 7f27b3713200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T20:40:28.002 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:40:27 smithi023 bash[21627]: debug 2024-03-11T20:40:27.751+0000 7f27b3713200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T20:40:28.258 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:40:28 smithi023 bash[21627]: debug 2024-03-11T20:40:28.007+0000 7f27b3713200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T20:40:29.160 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:40:29 smithi023 bash[21627]: debug 2024-03-11T20:40:29.043+0000 7f27b3713200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T20:40:29.462 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:40:29 smithi023 bash[21627]: debug 2024-03-11T20:40:29.163+0000 7f27b3713200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T20:40:29.463 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:29 smithi023 bash[21372]: audit 2024-03-11T20:40:29.223862+0000 mon.a (mon.0) 9 : audit 0 from='client.? 172.21.15.23:0/1189564660' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-11T20:40:29.749 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 2024-03-11T20:40:29.749 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout { 2024-03-11T20:40:29.749 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "fsid": "67ba00f8-dfe7-11ee-95c8-87774f69a715", 2024-03-11T20:40:29.749 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "health": { 2024-03-11T20:40:29.749 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-11T20:40:29.750 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-11T20:40:29.750 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-11T20:40:29.750 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-11T20:40:29.750 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-11T20:40:29.750 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-11T20:40:29.750 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 0 2024-03-11T20:40:29.750 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout ], 2024-03-11T20:40:29.750 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-11T20:40:29.750 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "a" 2024-03-11T20:40:29.750 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout ], 2024-03-11T20:40:29.750 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "quorum_age": 9, 2024-03-11T20:40:29.750 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-11T20:40:29.751 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T20:40:29.751 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-11T20:40:29.751 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-11T20:40:29.751 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-11T20:40:29.751 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-11T20:40:29.751 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T20:40:29.751 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-11T20:40:29.751 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-11T20:40:29.751 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-11T20:40:29.751 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-11T20:40:29.751 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-11T20:40:29.751 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-11T20:40:29.755 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-11T20:40:29.756 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-11T20:40:29.756 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-11T20:40:29.756 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-11T20:40:29.756 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-11T20:40:29.756 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-11T20:40:29.756 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-11T20:40:29.756 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-11T20:40:29.756 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-11T20:40:29.756 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-11T20:40:29.756 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-11T20:40:29.756 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-11T20:40:29.757 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T20:40:29.757 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-11T20:40:29.757 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-11T20:40:29.757 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-11T20:40:29.757 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-11T20:40:29.757 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-11T20:40:29.757 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-11T20:40:29.757 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-11T20:40:29.757 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-11T20:40:29.757 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-11T20:40:29.757 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "restful" 2024-03-11T20:40:29.757 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout ], 2024-03-11T20:40:29.757 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T20:40:29.758 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-11T20:40:29.758 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-11T20:40:29.758 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T20:40:29.758 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "modified": "2024-03-11T20:40:13.665433+0000", 2024-03-11T20:40:29.758 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T20:40:29.758 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-11T20:40:29.758 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-11T20:40:29.758 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout } 2024-03-11T20:40:29.758 INFO:teuthology.orchestra.run.smithi023.stdout:mgr not available, waiting (3/15)... 2024-03-11T20:40:29.782 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:40:29 smithi023 bash[21627]: debug 2024-03-11T20:40:29.467+0000 7f27b3713200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T20:40:29.782 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:40:29 smithi023 bash[21627]: debug 2024-03-11T20:40:29.583+0000 7f27b3713200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T20:40:30.532 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:40:30 smithi023 bash[21627]: debug 2024-03-11T20:40:30.055+0000 7f27b3713200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T20:40:30.532 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:40:30 smithi023 bash[21627]: debug 2024-03-11T20:40:30.179+0000 7f27b3713200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T20:40:30.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:30 smithi023 bash[21372]: cluster 2024-03-11T20:40:30.201762+0000 mon.a (mon.0) 10 : cluster 1 Activating manager daemon y 2024-03-11T20:40:30.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:30 smithi023 bash[21372]: cluster 2024-03-11T20:40:30.205484+0000 mon.a (mon.0) 11 : cluster 0 mgrmap e2: y(active, starting, since 0.004253s) 2024-03-11T20:40:30.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:30 smithi023 bash[21372]: audit 2024-03-11T20:40:30.208090+0000 mon.a (mon.0) 12 : audit 0 from='mgr.14100 172.21.15.23:0/3515681013' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T20:40:30.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:30 smithi023 bash[21372]: audit 2024-03-11T20:40:30.208307+0000 mon.a (mon.0) 13 : audit 0 from='mgr.14100 172.21.15.23:0/3515681013' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T20:40:30.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:30 smithi023 bash[21372]: audit 2024-03-11T20:40:30.208493+0000 mon.a (mon.0) 14 : audit 0 from='mgr.14100 172.21.15.23:0/3515681013' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T20:40:30.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:30 smithi023 bash[21372]: audit 2024-03-11T20:40:30.208676+0000 mon.a (mon.0) 15 : audit 0 from='mgr.14100 172.21.15.23:0/3515681013' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T20:40:30.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:30 smithi023 bash[21372]: audit 2024-03-11T20:40:30.208966+0000 mon.a (mon.0) 16 : audit 0 from='mgr.14100 172.21.15.23:0/3515681013' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-11T20:40:30.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:30 smithi023 bash[21372]: cluster 2024-03-11T20:40:30.233676+0000 mon.a (mon.0) 17 : cluster 1 Manager daemon y is now available 2024-03-11T20:40:30.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:30 smithi023 bash[21372]: audit 2024-03-11T20:40:30.246729+0000 mon.a (mon.0) 18 : audit 1 from='mgr.14100 172.21.15.23:0/3515681013' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T20:40:30.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:30 smithi023 bash[21372]: audit 2024-03-11T20:40:30.249268+0000 mon.a (mon.0) 19 : audit 1 from='mgr.14100 172.21.15.23:0/3515681013' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T20:40:30.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:30 smithi023 bash[21372]: audit 2024-03-11T20:40:30.252247+0000 mon.a (mon.0) 20 : audit 1 from='mgr.14100 172.21.15.23:0/3515681013' entity='mgr.y' 2024-03-11T20:40:30.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:30 smithi023 bash[21372]: audit 2024-03-11T20:40:30.254239+0000 mon.a (mon.0) 21 : audit 1 from='mgr.14100 172.21.15.23:0/3515681013' entity='mgr.y' 2024-03-11T20:40:30.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:30 smithi023 bash[21372]: audit 2024-03-11T20:40:30.256433+0000 mon.a (mon.0) 22 : audit 1 from='mgr.14100 172.21.15.23:0/3515681013' entity='mgr.y' 2024-03-11T20:40:32.474 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:32 smithi023 bash[21372]: cluster 2024-03-11T20:40:31.217516+0000 mon.a (mon.0) 23 : cluster 0 mgrmap e3: y(active, since 1.01628s) 2024-03-11T20:40:33.913 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 2024-03-11T20:40:33.913 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout { 2024-03-11T20:40:33.913 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "fsid": "67ba00f8-dfe7-11ee-95c8-87774f69a715", 2024-03-11T20:40:33.913 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "health": { 2024-03-11T20:40:33.913 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-11T20:40:33.913 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-11T20:40:33.913 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-11T20:40:33.913 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-11T20:40:33.913 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-11T20:40:33.913 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-11T20:40:33.913 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 0 2024-03-11T20:40:33.913 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout ], 2024-03-11T20:40:33.914 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-11T20:40:33.914 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "a" 2024-03-11T20:40:33.914 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout ], 2024-03-11T20:40:33.914 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "quorum_age": 13, 2024-03-11T20:40:33.914 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-11T20:40:33.914 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T20:40:33.914 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-11T20:40:33.914 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-11T20:40:33.914 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-11T20:40:33.914 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-11T20:40:33.914 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T20:40:33.914 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-11T20:40:33.914 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-11T20:40:33.914 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-11T20:40:33.915 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-11T20:40:33.915 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-11T20:40:33.915 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-11T20:40:33.918 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-11T20:40:33.918 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-11T20:40:33.919 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-11T20:40:33.919 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-11T20:40:33.919 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-11T20:40:33.919 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-11T20:40:33.919 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-11T20:40:33.919 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-11T20:40:33.919 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-11T20:40:33.919 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-11T20:40:33.920 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-11T20:40:33.920 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-11T20:40:33.920 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T20:40:33.920 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-11T20:40:33.920 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-11T20:40:33.920 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-11T20:40:33.920 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-11T20:40:33.920 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-11T20:40:33.920 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-11T20:40:33.920 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-11T20:40:33.920 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-11T20:40:33.920 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-11T20:40:33.920 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "restful" 2024-03-11T20:40:33.920 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout ], 2024-03-11T20:40:33.920 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T20:40:33.921 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-11T20:40:33.921 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-11T20:40:33.921 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T20:40:33.921 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "modified": "2024-03-11T20:40:13.665433+0000", 2024-03-11T20:40:33.921 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T20:40:33.921 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-11T20:40:33.921 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-11T20:40:33.921 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout } 2024-03-11T20:40:33.921 INFO:teuthology.orchestra.run.smithi023.stdout:mgr is available 2024-03-11T20:40:34.524 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:34 smithi023 bash[21372]: cluster 2024-03-11T20:40:33.217755+0000 mon.a (mon.0) 24 : cluster 0 mgrmap e4: y(active, since 3s) 2024-03-11T20:40:34.524 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:34 smithi023 bash[21372]: audit 2024-03-11T20:40:33.339182+0000 mon.a (mon.0) 25 : audit 0 from='client.? 172.21.15.23:0/896406973' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-11T20:40:35.532 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:35 smithi023 bash[21372]: audit 2024-03-11T20:40:35.121251+0000 mon.a (mon.0) 26 : audit 1 from='client.? 172.21.15.23:0/3825713821' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-11T20:40:35.675 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 2024-03-11T20:40:35.676 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout [global] 2024-03-11T20:40:35.676 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout fsid = 67ba00f8-dfe7-11ee-95c8-87774f69a715 2024-03-11T20:40:35.676 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-11T20:40:35.676 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-11T20:40:35.676 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-11T20:40:35.676 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-11T20:40:35.676 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 2024-03-11T20:40:35.676 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-11T20:40:35.676 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-11T20:40:35.676 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 2024-03-11T20:40:35.676 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout [osd] 2024-03-11T20:40:35.676 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-11T20:40:35.677 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-11T20:40:35.677 INFO:teuthology.orchestra.run.smithi023.stdout:Enabling cephadm module... 2024-03-11T20:40:37.532 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:37 smithi023 bash[21372]: audit 2024-03-11T20:40:37.003965+0000 mon.a (mon.0) 27 : audit 1 from='client.? 172.21.15.23:0/101119663' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-11T20:40:37.533 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:40:37 smithi023 bash[21627]: ignoring --setuser ceph since I am not root 2024-03-11T20:40:37.533 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:40:37 smithi023 bash[21627]: ignoring --setgroup ceph since I am not root 2024-03-11T20:40:37.533 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:40:37 smithi023 bash[21627]: debug 2024-03-11T20:40:37.243+0000 7f0c8f6b4200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T20:40:38.532 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:38 smithi023 bash[21372]: audit 2024-03-11T20:40:37.057439+0000 mon.a (mon.0) 28 : audit 1 from='client.? 172.21.15.23:0/101119663' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-11T20:40:38.532 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:38 smithi023 bash[21372]: cluster 2024-03-11T20:40:37.062799+0000 mon.a (mon.0) 29 : cluster 0 mgrmap e5: y(active, since 6s) 2024-03-11T20:40:39.130 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:39 smithi023 bash[21372]: audit 2024-03-11T20:40:38.950520+0000 mon.a (mon.0) 30 : audit 0 from='client.? 172.21.15.23:0/3226126606' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-11T20:40:39.130 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:40:38 smithi023 bash[21627]: debug 2024-03-11T20:40:38.859+0000 7f0c8f6b4200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T20:40:39.130 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:40:38 smithi023 bash[21627]: debug 2024-03-11T20:40:38.963+0000 7f0c8f6b4200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T20:40:39.532 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:40:39 smithi023 bash[21627]: debug 2024-03-11T20:40:39.219+0000 7f0c8f6b4200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T20:40:39.534 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout { 2024-03-11T20:40:39.534 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-11T20:40:39.534 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-11T20:40:39.534 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-03-11T20:40:39.534 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-11T20:40:39.534 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout } 2024-03-11T20:40:39.534 INFO:teuthology.orchestra.run.smithi023.stdout:Waiting for the mgr to restart... 2024-03-11T20:40:39.534 INFO:teuthology.orchestra.run.smithi023.stdout:Waiting for mgr epoch 5... 2024-03-11T20:40:40.532 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:40:40 smithi023 bash[21627]: debug 2024-03-11T20:40:40.043+0000 7f0c8f6b4200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T20:40:40.532 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:40:40 smithi023 bash[21627]: debug 2024-03-11T20:40:40.155+0000 7f0c8f6b4200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T20:40:40.532 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:40:40 smithi023 bash[21627]: debug 2024-03-11T20:40:40.267+0000 7f0c8f6b4200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T20:40:41.032 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:40:40 smithi023 bash[21627]: debug 2024-03-11T20:40:40.631+0000 7f0c8f6b4200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-11T20:40:41.032 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:40:40 smithi023 bash[21627]: debug 2024-03-11T20:40:40.747+0000 7f0c8f6b4200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T20:40:41.032 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:40:40 smithi023 bash[21627]: debug 2024-03-11T20:40:40.855+0000 7f0c8f6b4200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T20:40:41.532 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:40:41 smithi023 bash[21627]: debug 2024-03-11T20:40:41.223+0000 7f0c8f6b4200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T20:40:41.532 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:40:41 smithi023 bash[21627]: debug 2024-03-11T20:40:41.467+0000 7f0c8f6b4200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T20:40:42.032 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:40:41 smithi023 bash[21627]: debug 2024-03-11T20:40:41.739+0000 7f0c8f6b4200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T20:40:42.032 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:40:41 smithi023 bash[21627]: debug 2024-03-11T20:40:41.851+0000 7f0c8f6b4200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T20:40:42.381 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:40:42 smithi023 bash[21627]: debug 2024-03-11T20:40:42.135+0000 7f0c8f6b4200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T20:40:42.782 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:40:42 smithi023 bash[21627]: debug 2024-03-11T20:40:42.383+0000 7f0c8f6b4200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T20:40:43.173 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:40:42 smithi023 bash[21627]: debug 2024-03-11T20:40:42.803+0000 7f0c8f6b4200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T20:40:43.174 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:40:42 smithi023 bash[21627]: debug 2024-03-11T20:40:42.923+0000 7f0c8f6b4200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T20:40:43.174 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:40:43 smithi023 bash[21627]: debug 2024-03-11T20:40:43.179+0000 7f0c8f6b4200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T20:40:44.532 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:40:44 smithi023 bash[21627]: debug 2024-03-11T20:40:44.211+0000 7f0c8f6b4200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T20:40:44.532 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:40:44 smithi023 bash[21627]: debug 2024-03-11T20:40:44.331+0000 7f0c8f6b4200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T20:40:45.032 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:40:44 smithi023 bash[21627]: debug 2024-03-11T20:40:44.631+0000 7f0c8f6b4200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T20:40:45.032 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:40:44 smithi023 bash[21627]: debug 2024-03-11T20:40:44.747+0000 7f0c8f6b4200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T20:40:45.532 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:40:45 smithi023 bash[21627]: debug 2024-03-11T20:40:45.215+0000 7f0c8f6b4200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T20:40:45.532 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:40:45 smithi023 bash[21627]: debug 2024-03-11T20:40:45.339+0000 7f0c8f6b4200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T20:40:45.532 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:45 smithi023 bash[21372]: cluster 2024-03-11T20:40:45.361501+0000 mon.a (mon.0) 31 : cluster 1 Active manager daemon y restarted 2024-03-11T20:40:45.532 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:45 smithi023 bash[21372]: cluster 2024-03-11T20:40:45.361987+0000 mon.a (mon.0) 32 : cluster 1 Activating manager daemon y 2024-03-11T20:40:45.532 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:45 smithi023 bash[21372]: cluster 2024-03-11T20:40:45.370614+0000 mon.a (mon.0) 33 : cluster 0 osdmap e2: 0 total, 0 up, 0 in 2024-03-11T20:40:45.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:45 smithi023 bash[21372]: cluster 2024-03-11T20:40:45.370853+0000 mon.a (mon.0) 34 : cluster 0 mgrmap e6: y(active, starting, since 0.00904346s) 2024-03-11T20:40:45.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:45 smithi023 bash[21372]: audit 2024-03-11T20:40:45.372385+0000 mon.a (mon.0) 35 : audit 0 from='mgr.14120 172.21.15.23:0/1704051914' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T20:40:45.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:45 smithi023 bash[21372]: audit 2024-03-11T20:40:45.374124+0000 mon.a (mon.0) 36 : audit 0 from='mgr.14120 172.21.15.23:0/1704051914' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-11T20:40:45.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:45 smithi023 bash[21372]: audit 2024-03-11T20:40:45.375292+0000 mon.a (mon.0) 37 : audit 0 from='mgr.14120 172.21.15.23:0/1704051914' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T20:40:45.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:45 smithi023 bash[21372]: audit 2024-03-11T20:40:45.375468+0000 mon.a (mon.0) 38 : audit 0 from='mgr.14120 172.21.15.23:0/1704051914' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T20:40:45.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:45 smithi023 bash[21372]: audit 2024-03-11T20:40:45.375603+0000 mon.a (mon.0) 39 : audit 0 from='mgr.14120 172.21.15.23:0/1704051914' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T20:40:45.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:45 smithi023 bash[21372]: cluster 2024-03-11T20:40:45.401822+0000 mon.a (mon.0) 40 : cluster 1 Manager daemon y is now available 2024-03-11T20:40:46.782 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:46 smithi023 bash[21372]: cephadm 2024-03-11T20:40:45.423285+0000 mgr.y (mgr.14120) 1 : cephadm 1 Found migration_current of "None". Setting to last migration. 2024-03-11T20:40:46.782 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:46 smithi023 bash[21372]: audit 2024-03-11T20:40:45.425496+0000 mon.a (mon.0) 41 : audit 1 from='mgr.14120 172.21.15.23:0/1704051914' entity='mgr.y' 2024-03-11T20:40:46.782 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:46 smithi023 bash[21372]: audit 2024-03-11T20:40:45.430193+0000 mon.a (mon.0) 42 : audit 1 from='mgr.14120 172.21.15.23:0/1704051914' entity='mgr.y' 2024-03-11T20:40:46.782 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:46 smithi023 bash[21372]: audit 2024-03-11T20:40:45.435436+0000 mon.a (mon.0) 43 : audit 0 from='mgr.14120 172.21.15.23:0/1704051914' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:40:46.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:46 smithi023 bash[21372]: audit 2024-03-11T20:40:45.437499+0000 mon.a (mon.0) 44 : audit 0 from='mgr.14120 172.21.15.23:0/1704051914' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:40:46.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:46 smithi023 bash[21372]: audit 2024-03-11T20:40:45.443234+0000 mon.a (mon.0) 45 : audit 0 from='mgr.14120 172.21.15.23:0/1704051914' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:40:46.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:46 smithi023 bash[21372]: audit 2024-03-11T20:40:45.451661+0000 mon.a (mon.0) 46 : audit 1 from='mgr.14120 172.21.15.23:0/1704051914' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T20:40:46.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:46 smithi023 bash[21372]: audit 2024-03-11T20:40:45.454433+0000 mon.a (mon.0) 47 : audit 1 from='mgr.14120 172.21.15.23:0/1704051914' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T20:40:46.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:46 smithi023 bash[21372]: audit 2024-03-11T20:40:46.312541+0000 mon.a (mon.0) 48 : audit 1 from='mgr.14120 172.21.15.23:0/1704051914' entity='mgr.y' 2024-03-11T20:40:46.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:46 smithi023 bash[21372]: audit 2024-03-11T20:40:46.317999+0000 mon.a (mon.0) 49 : audit 1 from='mgr.14120 172.21.15.23:0/1704051914' entity='mgr.y' 2024-03-11T20:40:46.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:46 smithi023 bash[21372]: cluster 2024-03-11T20:40:46.373767+0000 mon.a (mon.0) 50 : cluster 0 mgrmap e7: y(active, since 1.01195s) 2024-03-11T20:40:46.981 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout { 2024-03-11T20:40:46.981 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-11T20:40:46.981 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-11T20:40:46.981 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout } 2024-03-11T20:40:46.981 INFO:teuthology.orchestra.run.smithi023.stdout:mgr epoch 5 is available 2024-03-11T20:40:46.981 INFO:teuthology.orchestra.run.smithi023.stdout:Setting orchestrator backend to cephadm... 2024-03-11T20:40:47.634 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:47 smithi023 bash[21372]: audit 2024-03-11T20:40:46.371275+0000 mgr.y (mgr.14120) 2 : audit 0 from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-11T20:40:47.634 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:47 smithi023 bash[21372]: audit 2024-03-11T20:40:46.386469+0000 mgr.y (mgr.14120) 3 : audit 0 from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-11T20:40:47.634 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:47 smithi023 bash[21372]: cephadm 2024-03-11T20:40:46.677464+0000 mgr.y (mgr.14120) 4 : cephadm 1 [11/Mar/2024:20:40:46] ENGINE Bus STARTING 2024-03-11T20:40:47.634 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:47 smithi023 bash[21372]: cephadm 2024-03-11T20:40:46.779545+0000 mgr.y (mgr.14120) 5 : cephadm 1 [11/Mar/2024:20:40:46] ENGINE Serving on http://172.21.15.23:8765 2024-03-11T20:40:47.634 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:47 smithi023 bash[21372]: cephadm 2024-03-11T20:40:46.900803+0000 mgr.y (mgr.14120) 6 : cephadm 1 [11/Mar/2024:20:40:46] ENGINE Serving on https://172.21.15.23:7150 2024-03-11T20:40:47.634 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:47 smithi023 bash[21372]: cephadm 2024-03-11T20:40:46.901008+0000 mgr.y (mgr.14120) 7 : cephadm 1 [11/Mar/2024:20:40:46] ENGINE Bus STARTED 2024-03-11T20:40:47.634 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:47 smithi023 bash[21372]: audit 2024-03-11T20:40:46.914213+0000 mon.a (mon.0) 51 : audit 0 from='mgr.14120 172.21.15.23:0/1704051914' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:40:48.532 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:48 smithi023 bash[21372]: cluster 2024-03-11T20:40:47.440186+0000 mon.a (mon.0) 52 : cluster 0 mgrmap e8: y(active, since 2s) 2024-03-11T20:40:48.532 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:48 smithi023 bash[21372]: audit 2024-03-11T20:40:48.225840+0000 mon.a (mon.0) 53 : audit 1 from='mgr.14120 172.21.15.23:0/1704051914' entity='mgr.y' 2024-03-11T20:40:48.532 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:48 smithi023 bash[21372]: audit 2024-03-11T20:40:48.284831+0000 mon.a (mon.0) 54 : audit 0 from='mgr.14120 172.21.15.23:0/1704051914' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:40:49.782 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:49 smithi023 bash[21372]: audit 2024-03-11T20:40:48.217029+0000 mgr.y (mgr.14120) 8 : audit 0 from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:40:50.633 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-11T20:40:50.633 INFO:teuthology.orchestra.run.smithi023.stdout:Generating ssh key... 2024-03-11T20:40:50.782 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:50 smithi023 bash[21372]: audit 2024-03-11T20:40:50.054529+0000 mgr.y (mgr.14120) 9 : audit 0 from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:40:52.532 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:52 smithi023 bash[21372]: audit 2024-03-11T20:40:51.919560+0000 mgr.y (mgr.14120) 10 : audit 0 from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:40:52.532 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:52 smithi023 bash[21372]: cephadm 2024-03-11T20:40:51.919984+0000 mgr.y (mgr.14120) 11 : cephadm 1 Generating ssh key... 2024-03-11T20:40:52.532 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:40:52 smithi023 bash[21627]: Generating public/private rsa key pair. 2024-03-11T20:40:52.533 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:40:52 smithi023 bash[21627]: Your identification has been saved in /tmp/tmp54iw_a5w/key. 2024-03-11T20:40:52.533 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:40:52 smithi023 bash[21627]: Your public key has been saved in /tmp/tmp54iw_a5w/key.pub. 2024-03-11T20:40:52.533 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:40:52 smithi023 bash[21627]: The key fingerprint is: 2024-03-11T20:40:52.533 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:40:52 smithi023 bash[21627]: SHA256:PG/y0zAjonUmkBRMTA+hsmEJABJYm6+Rbqb0F/rA7Gc ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715 2024-03-11T20:40:52.533 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:40:52 smithi023 bash[21627]: The key's randomart image is: 2024-03-11T20:40:52.533 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:40:52 smithi023 bash[21627]: +---[RSA 3072]----+ 2024-03-11T20:40:52.533 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:40:52 smithi023 bash[21627]: |Xo.=*o | 2024-03-11T20:40:52.533 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:40:52 smithi023 bash[21627]: |+ .++o | 2024-03-11T20:40:52.533 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:40:52 smithi023 bash[21627]: |oo+. .. | 2024-03-11T20:40:52.533 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:40:52 smithi023 bash[21627]: |.+ oo . | 2024-03-11T20:40:52.533 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:40:52 smithi023 bash[21627]: |. o .. S | 2024-03-11T20:40:52.533 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:40:52 smithi023 bash[21627]: | .oo .+ +o+ | 2024-03-11T20:40:52.533 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:40:52 smithi023 bash[21627]: | .=+.o.=..o= | 2024-03-11T20:40:52.533 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:40:52 smithi023 bash[21627]: |.+o.oE +. . | 2024-03-11T20:40:52.533 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:40:52 smithi023 bash[21627]: |. o=. .. | 2024-03-11T20:40:52.534 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:40:52 smithi023 bash[21627]: +----[SHA256]-----+ 2024-03-11T20:40:53.532 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:53 smithi023 bash[21372]: audit 2024-03-11T20:40:52.221828+0000 mon.a (mon.0) 55 : audit 1 from='mgr.14120 172.21.15.23:0/1704051914' entity='mgr.y' 2024-03-11T20:40:53.532 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:53 smithi023 bash[21372]: audit 2024-03-11T20:40:52.226713+0000 mon.a (mon.0) 56 : audit 1 from='mgr.14120 172.21.15.23:0/1704051914' entity='mgr.y' 2024-03-11T20:40:54.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:54 smithi023 bash[21372]: audit 2024-03-11T20:40:54.001345+0000 mgr.y (mgr.14120) 12 : audit 0 from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:40:54.552 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC/3X8stHmtjvuVWGX46FgHurXUnMqdG7HhyMbUtqscT4ERzjA8oKD/H/oaIhfPGBUwLN9fnlxjZsaUK284ZEZY+aiNbWZNiXUbcMjDmiQLi4jJP2O+T/vjQxBIlw83iVIuriboUJdsx+nNB7ffuy+hBlEqJryq/5f/vuYZEAqM5XbsZax5PL6YfsG7laFnql14cneLAOAJRDPB2VaajsUXoeCv9va7CnDMUDCiEBtj0HkBddHUog2HC8eqzmJFwpPYjEC/G9DYXSD4K+EP0PPYiMr4L2g7qgasMa22mgoet7s69mRMoGBLde5WF9pra3JuWXzqphARprQYqu+Am+YPvAt7zXosy/y1q76N6dfDKtFms5SoCBpvJ12RTCprlqUktz0wm3WKi1KAtjwqBUdm58gy0Zspdn94Ksc+L4dpnXiRkaxRbp4exl+V61uKw+8JTv7o59EBtW55CZTuKsp9ow9T4FduKMjEM2et+h6zwkADRcIKYRNKQedh8ZowWq8= ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715 2024-03-11T20:40:54.552 INFO:teuthology.orchestra.run.smithi023.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-11T20:40:54.552 INFO:teuthology.orchestra.run.smithi023.stdout:Adding key to root@localhost authorized_keys... 2024-03-11T20:40:54.552 INFO:teuthology.orchestra.run.smithi023.stdout:Adding host smithi023... 2024-03-11T20:40:56.214 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:56 smithi023 bash[21372]: audit 2024-03-11T20:40:55.920675+0000 mgr.y (mgr.14120) 13 : audit 0 from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi023", "addr": "172.21.15.23", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:40:57.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:40:57 smithi023 bash[21372]: cephadm 2024-03-11T20:40:56.592047+0000 mgr.y (mgr.14120) 14 : cephadm 1 Deploying cephadm binary to smithi023 2024-03-11T20:41:02.508 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout Added host 'smithi023' with addr '172.21.15.23' 2024-03-11T20:41:02.508 INFO:teuthology.orchestra.run.smithi023.stdout:Deploying unmanaged mon service... 2024-03-11T20:41:02.782 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:41:02 smithi023 bash[21372]: audit 2024-03-11T20:41:01.746601+0000 mon.a (mon.0) 57 : audit 1 from='mgr.14120 172.21.15.23:0/1704051914' entity='mgr.y' 2024-03-11T20:41:02.782 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:41:02 smithi023 bash[21372]: cephadm 2024-03-11T20:41:01.747163+0000 mgr.y (mgr.14120) 15 : cephadm 1 Added host smithi023 2024-03-11T20:41:02.782 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:41:02 smithi023 bash[21372]: audit 2024-03-11T20:41:01.768678+0000 mon.a (mon.0) 58 : audit 0 from='mgr.14120 172.21.15.23:0/1704051914' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:41:05.009 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-11T20:41:05.009 INFO:teuthology.orchestra.run.smithi023.stdout:Deploying unmanaged mgr service... 2024-03-11T20:41:05.699 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:41:05 smithi023 bash[21372]: audit 2024-03-11T20:41:04.411120+0000 mgr.y (mgr.14120) 16 : audit 0 from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:41:05.699 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:41:05 smithi023 bash[21372]: cephadm 2024-03-11T20:41:04.414381+0000 mgr.y (mgr.14120) 17 : cephadm 1 Saving service mon spec with placement count:5 2024-03-11T20:41:05.699 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:41:05 smithi023 bash[21372]: audit 2024-03-11T20:41:04.417480+0000 mon.a (mon.0) 59 : audit 1 from='mgr.14120 172.21.15.23:0/1704051914' entity='mgr.y' 2024-03-11T20:41:05.699 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:41:05 smithi023 bash[21372]: audit 2024-03-11T20:41:04.696567+0000 mon.a (mon.0) 60 : audit 1 from='mgr.14120 172.21.15.23:0/1704051914' entity='mgr.y' 2024-03-11T20:41:05.700 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:41:05 smithi023 bash[21372]: audit 2024-03-11T20:41:05.247412+0000 mon.a (mon.0) 61 : audit 1 from='mgr.14120 172.21.15.23:0/1704051914' entity='mgr.y' 2024-03-11T20:41:06.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:41:06 smithi023 bash[21372]: cluster 2024-03-11T20:41:05.376907+0000 mgr.y (mgr.14120) 18 : cluster 0 pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:41:06.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:41:06 smithi023 bash[21372]: audit 2024-03-11T20:41:06.277037+0000 mon.a (mon.0) 62 : audit 1 from='mgr.14120 172.21.15.23:0/1704051914' entity='mgr.y' 2024-03-11T20:41:06.868 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-11T20:41:07.782 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:41:07 smithi023 bash[21372]: audit 2024-03-11T20:41:06.269697+0000 mgr.y (mgr.14120) 19 : audit 0 from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:41:07.782 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:41:07 smithi023 bash[21372]: cephadm 2024-03-11T20:41:06.272320+0000 mgr.y (mgr.14120) 20 : cephadm 1 Saving service mgr spec with placement count:2 2024-03-11T20:41:08.743 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:41:08 smithi023 bash[21372]: cluster 2024-03-11T20:41:07.377449+0000 mgr.y (mgr.14120) 21 : cluster 0 pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:41:08.743 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:41:08 smithi023 bash[21372]: audit 2024-03-11T20:41:08.120584+0000 mon.a (mon.0) 63 : audit 1 from='client.? 172.21.15.23:0/2349040068' entity='client.admin' 2024-03-11T20:41:10.726 INFO:teuthology.orchestra.run.smithi023.stdout:Enabling the dashboard module... 2024-03-11T20:41:11.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:41:10 smithi023 bash[21372]: cluster 2024-03-11T20:41:09.377841+0000 mgr.y (mgr.14120) 22 : cluster 0 pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:41:11.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:41:10 smithi023 bash[21372]: audit 2024-03-11T20:41:09.929490+0000 mon.a (mon.0) 64 : audit 1 from='client.? 172.21.15.23:0/1619079908' entity='client.admin' 2024-03-11T20:41:13.125 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:41:12 smithi023 bash[21627]: ignoring --setuser ceph since I am not root 2024-03-11T20:41:13.125 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:41:12 smithi023 bash[21627]: ignoring --setgroup ceph since I am not root 2024-03-11T20:41:13.125 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:41:12 smithi023 bash[21372]: cluster 2024-03-11T20:41:11.378282+0000 mgr.y (mgr.14120) 23 : cluster 0 pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:41:13.125 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:41:12 smithi023 bash[21372]: audit 2024-03-11T20:41:12.361087+0000 mon.a (mon.0) 65 : audit 1 from='client.? 172.21.15.23:0/3272725278' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-11T20:41:13.125 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:41:12 smithi023 bash[21372]: audit 2024-03-11T20:41:12.482553+0000 mon.a (mon.0) 66 : audit 1 from='mgr.14120 172.21.15.23:0/1704051914' entity='mgr.y' 2024-03-11T20:41:13.532 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:41:13 smithi023 bash[21627]: debug 2024-03-11T20:41:13.131+0000 7f0404707200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T20:41:14.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:41:13 smithi023 bash[21372]: audit 2024-03-11T20:41:12.933400+0000 mon.a (mon.0) 67 : audit 1 from='client.? 172.21.15.23:0/3272725278' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-11T20:41:14.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:41:13 smithi023 bash[21372]: cluster 2024-03-11T20:41:12.937393+0000 mon.a (mon.0) 68 : cluster 0 mgrmap e9: y(active, since 27s) 2024-03-11T20:41:15.032 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:41:14 smithi023 bash[21627]: debug 2024-03-11T20:41:14.755+0000 7f0404707200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T20:41:15.032 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:41:14 smithi023 bash[21627]: debug 2024-03-11T20:41:14.855+0000 7f0404707200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T20:41:15.532 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:41:15 smithi023 bash[21372]: audit 2024-03-11T20:41:15.082043+0000 mon.a (mon.0) 69 : audit 0 from='client.? 172.21.15.23:0/539650229' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-11T20:41:15.532 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:41:15 smithi023 bash[21627]: debug 2024-03-11T20:41:15.115+0000 7f0404707200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T20:41:15.714 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout { 2024-03-11T20:41:15.714 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-11T20:41:15.714 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-11T20:41:15.714 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-03-11T20:41:15.714 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-11T20:41:15.714 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout } 2024-03-11T20:41:15.714 INFO:teuthology.orchestra.run.smithi023.stdout:Waiting for the mgr to restart... 2024-03-11T20:41:15.714 INFO:teuthology.orchestra.run.smithi023.stdout:Waiting for mgr epoch 9... 2024-03-11T20:41:16.031 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:41:15 smithi023 bash[21627]: debug 2024-03-11T20:41:15.947+0000 7f0404707200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T20:41:16.357 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:41:16 smithi023 bash[21627]: debug 2024-03-11T20:41:16.059+0000 7f0404707200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T20:41:16.357 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:41:16 smithi023 bash[21627]: debug 2024-03-11T20:41:16.167+0000 7f0404707200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T20:41:16.653 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:41:16 smithi023 bash[21627]: debug 2024-03-11T20:41:16.535+0000 7f0404707200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-11T20:41:16.653 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:41:16 smithi023 bash[21627]: debug 2024-03-11T20:41:16.651+0000 7f0404707200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T20:41:17.032 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:41:16 smithi023 bash[21627]: debug 2024-03-11T20:41:16.759+0000 7f0404707200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T20:41:17.532 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:41:17 smithi023 bash[21627]: debug 2024-03-11T20:41:17.127+0000 7f0404707200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T20:41:17.532 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:41:17 smithi023 bash[21627]: debug 2024-03-11T20:41:17.375+0000 7f0404707200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T20:41:18.032 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:41:17 smithi023 bash[21627]: debug 2024-03-11T20:41:17.647+0000 7f0404707200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T20:41:18.032 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:41:17 smithi023 bash[21627]: debug 2024-03-11T20:41:17.763+0000 7f0404707200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T20:41:18.300 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:41:18 smithi023 bash[21627]: debug 2024-03-11T20:41:18.047+0000 7f0404707200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T20:41:18.721 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:41:18 smithi023 bash[21627]: debug 2024-03-11T20:41:18.299+0000 7f0404707200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T20:41:19.032 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:41:18 smithi023 bash[21627]: debug 2024-03-11T20:41:18.719+0000 7f0404707200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T20:41:19.032 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:41:18 smithi023 bash[21627]: debug 2024-03-11T20:41:18.843+0000 7f0404707200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T20:41:19.532 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:41:19 smithi023 bash[21627]: debug 2024-03-11T20:41:19.103+0000 7f0404707200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T20:41:20.532 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:41:20 smithi023 bash[21627]: debug 2024-03-11T20:41:20.139+0000 7f0404707200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T20:41:20.532 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:41:20 smithi023 bash[21627]: debug 2024-03-11T20:41:20.259+0000 7f0404707200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T20:41:21.032 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:41:20 smithi023 bash[21627]: debug 2024-03-11T20:41:20.559+0000 7f0404707200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T20:41:21.032 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:41:20 smithi023 bash[21627]: debug 2024-03-11T20:41:20.675+0000 7f0404707200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T20:41:21.411 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:41:21 smithi023 bash[21627]: debug 2024-03-11T20:41:21.147+0000 7f0404707200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T20:41:21.411 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:41:21 smithi023 bash[21627]: debug 2024-03-11T20:41:21.271+0000 7f0404707200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T20:41:21.411 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:41:21 smithi023 bash[21372]: cluster 2024-03-11T20:41:21.301491+0000 mon.a (mon.0) 70 : cluster 1 Active manager daemon y restarted 2024-03-11T20:41:21.412 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:41:21 smithi023 bash[21372]: cluster 2024-03-11T20:41:21.301745+0000 mon.a (mon.0) 71 : cluster 1 Activating manager daemon y 2024-03-11T20:41:21.412 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:41:21 smithi023 bash[21372]: cluster 2024-03-11T20:41:21.310195+0000 mon.a (mon.0) 72 : cluster 0 osdmap e3: 0 total, 0 up, 0 in 2024-03-11T20:41:21.412 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:41:21 smithi023 bash[21372]: cluster 2024-03-11T20:41:21.310432+0000 mon.a (mon.0) 73 : cluster 0 mgrmap e10: y(active, starting, since 0.00879186s) 2024-03-11T20:41:21.412 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:41:21 smithi023 bash[21372]: audit 2024-03-11T20:41:21.312088+0000 mon.a (mon.0) 74 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T20:41:21.412 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:41:21 smithi023 bash[21372]: audit 2024-03-11T20:41:21.313078+0000 mon.a (mon.0) 75 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-11T20:41:21.412 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:41:21 smithi023 bash[21372]: audit 2024-03-11T20:41:21.314228+0000 mon.a (mon.0) 76 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T20:41:21.412 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:41:21 smithi023 bash[21372]: audit 2024-03-11T20:41:21.314391+0000 mon.a (mon.0) 77 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T20:41:21.412 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:41:21 smithi023 bash[21372]: audit 2024-03-11T20:41:21.314518+0000 mon.a (mon.0) 78 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T20:41:21.412 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:41:21 smithi023 bash[21372]: cluster 2024-03-11T20:41:21.341897+0000 mon.a (mon.0) 79 : cluster 1 Manager daemon y is now available 2024-03-11T20:41:22.782 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:41:22 smithi023 bash[21372]: audit 2024-03-11T20:41:21.366293+0000 mon.a (mon.0) 80 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:41:22.782 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:41:22 smithi023 bash[21372]: audit 2024-03-11T20:41:21.369058+0000 mon.a (mon.0) 81 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:41:22.782 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:41:22 smithi023 bash[21372]: audit 2024-03-11T20:41:21.387176+0000 mon.a (mon.0) 82 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T20:41:22.782 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:41:22 smithi023 bash[21372]: audit 2024-03-11T20:41:21.390533+0000 mon.a (mon.0) 83 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T20:41:22.782 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:41:22 smithi023 bash[21372]: cluster 2024-03-11T20:41:22.315816+0000 mon.a (mon.0) 84 : cluster 0 mgrmap e11: y(active, since 1.01417s) 2024-03-11T20:41:23.014 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout { 2024-03-11T20:41:23.014 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-11T20:41:23.014 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-11T20:41:23.014 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout } 2024-03-11T20:41:23.015 INFO:teuthology.orchestra.run.smithi023.stdout:mgr epoch 9 is available 2024-03-11T20:41:23.015 INFO:teuthology.orchestra.run.smithi023.stdout:Generating a dashboard self-signed certificate... 2024-03-11T20:41:23.666 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:41:23 smithi023 bash[21372]: cephadm 2024-03-11T20:41:22.888379+0000 mgr.y (mgr.14152) 3 : cephadm 1 [11/Mar/2024:20:41:22] ENGINE Bus STARTING 2024-03-11T20:41:23.667 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:41:23 smithi023 bash[21372]: cephadm 2024-03-11T20:41:22.990284+0000 mgr.y (mgr.14152) 4 : cephadm 1 [11/Mar/2024:20:41:22] ENGINE Serving on http://172.21.15.23:8765 2024-03-11T20:41:24.616 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:41:24 smithi023 bash[21372]: cephadm 2024-03-11T20:41:23.113411+0000 mgr.y (mgr.14152) 5 : cephadm 1 [11/Mar/2024:20:41:23] ENGINE Serving on https://172.21.15.23:7150 2024-03-11T20:41:24.616 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:41:24 smithi023 bash[21372]: cephadm 2024-03-11T20:41:23.113723+0000 mgr.y (mgr.14152) 6 : cephadm 1 [11/Mar/2024:20:41:23] ENGINE Bus STARTED 2024-03-11T20:41:24.616 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:41:24 smithi023 bash[21372]: cluster 2024-03-11T20:41:23.368817+0000 mon.a (mon.0) 85 : cluster 0 mgrmap e12: y(active, since 2s) 2024-03-11T20:41:25.007 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-11T20:41:25.007 INFO:teuthology.orchestra.run.smithi023.stdout:Creating initial admin user... 2024-03-11T20:41:25.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:41:25 smithi023 bash[21372]: audit 2024-03-11T20:41:24.273021+0000 mgr.y (mgr.14152) 7 : audit 0 from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:41:25.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:41:25 smithi023 bash[21372]: audit 2024-03-11T20:41:24.418008+0000 mon.a (mon.0) 86 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:41:25.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:41:25 smithi023 bash[21372]: audit 2024-03-11T20:41:24.423010+0000 mon.a (mon.0) 87 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:41:27.105 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$cnjHdk0gYIvDjP0l2m/jJ.oSSm0yXRFV4N0p5UPd1saJW3szAYV4a", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1710189686, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-11T20:41:27.105 INFO:teuthology.orchestra.run.smithi023.stdout:Fetching dashboard port number... 2024-03-11T20:41:27.782 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:41:27 smithi023 bash[21372]: audit 2024-03-11T20:41:26.272293+0000 mgr.y (mgr.14152) 8 : audit 0 from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:41:27.782 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:41:27 smithi023 bash[21372]: audit 2024-03-11T20:41:26.537776+0000 mon.a (mon.0) 88 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:41:28.782 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:41:28 smithi023 bash[21372]: cluster 2024-03-11T20:41:27.550901+0000 mon.a (mon.0) 89 : cluster 0 mgrmap e13: y(active, since 6s) 2024-03-11T20:41:28.782 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:41:28 smithi023 bash[21372]: audit 2024-03-11T20:41:28.300622+0000 mon.a (mon.0) 90 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:41:28.782 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:41:28 smithi023 bash[21372]: audit 2024-03-11T20:41:28.399789+0000 mon.a (mon.0) 91 : audit 0 from='client.? 172.21.15.23:0/1012348945' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-11T20:41:28.984 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 8443 2024-03-11T20:41:28.984 INFO:teuthology.orchestra.run.smithi023.stdout:firewalld does not appear to be present 2024-03-11T20:41:28.984 INFO:teuthology.orchestra.run.smithi023.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-11T20:41:28.985 INFO:teuthology.orchestra.run.smithi023.stdout:Ceph Dashboard is now available at: 2024-03-11T20:41:28.986 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-11T20:41:28.986 INFO:teuthology.orchestra.run.smithi023.stdout: URL: https://smithi023:8443/ 2024-03-11T20:41:28.986 INFO:teuthology.orchestra.run.smithi023.stdout: User: admin 2024-03-11T20:41:28.986 INFO:teuthology.orchestra.run.smithi023.stdout: Password: bv2g2budfn 2024-03-11T20:41:28.986 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-11T20:41:28.986 INFO:teuthology.orchestra.run.smithi023.stdout:Saving cluster configuration to /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/config directory 2024-03-11T20:41:28.986 INFO:teuthology.orchestra.run.smithi023.stdout:Enabling autotune for osd_memory_target 2024-03-11T20:41:30.532 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:41:30 smithi023 bash[21372]: audit 2024-03-11T20:41:29.288585+0000 mon.a (mon.0) 92 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:41:32.754 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-11T20:41:32.754 INFO:teuthology.orchestra.run.smithi023.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-11T20:41:32.755 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-11T20:41:32.755 INFO:teuthology.orchestra.run.smithi023.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-11T20:41:32.755 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-11T20:41:32.755 INFO:teuthology.orchestra.run.smithi023.stdout:Or, if you are only running a single cluster on this host: 2024-03-11T20:41:32.755 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-11T20:41:32.755 INFO:teuthology.orchestra.run.smithi023.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-03-11T20:41:32.755 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-11T20:41:32.755 INFO:teuthology.orchestra.run.smithi023.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-11T20:41:32.755 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-11T20:41:32.755 INFO:teuthology.orchestra.run.smithi023.stdout: ceph telemetry on 2024-03-11T20:41:32.755 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-11T20:41:32.755 INFO:teuthology.orchestra.run.smithi023.stdout:For more information see: 2024-03-11T20:41:32.755 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-11T20:41:32.755 INFO:teuthology.orchestra.run.smithi023.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-03-11T20:41:32.755 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-11T20:41:32.756 INFO:teuthology.orchestra.run.smithi023.stdout:Bootstrap complete. 2024-03-11T20:41:32.809 INFO:tasks.cephadm:Fetching config... 2024-03-11T20:41:32.809 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-11T20:41:32.809 DEBUG:teuthology.orchestra.run.smithi023:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-11T20:41:32.815 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-11T20:41:32.815 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-11T20:41:32.816 DEBUG:teuthology.orchestra.run.smithi023:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-11T20:41:32.861 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-11T20:41:32.861 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-11T20:41:32.861 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd if=/var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/keyring of=/dev/stdout 2024-03-11T20:41:32.918 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-11T20:41:32.918 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-11T20:41:32.918 DEBUG:teuthology.orchestra.run.smithi023:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-11T20:41:32.965 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-11T20:41:32.965 DEBUG:teuthology.orchestra.run.smithi023:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC/3X8stHmtjvuVWGX46FgHurXUnMqdG7HhyMbUtqscT4ERzjA8oKD/H/oaIhfPGBUwLN9fnlxjZsaUK284ZEZY+aiNbWZNiXUbcMjDmiQLi4jJP2O+T/vjQxBIlw83iVIuriboUJdsx+nNB7ffuy+hBlEqJryq/5f/vuYZEAqM5XbsZax5PL6YfsG7laFnql14cneLAOAJRDPB2VaajsUXoeCv9va7CnDMUDCiEBtj0HkBddHUog2HC8eqzmJFwpPYjEC/G9DYXSD4K+EP0PPYiMr4L2g7qgasMa22mgoet7s69mRMoGBLde5WF9pra3JuWXzqphARprQYqu+Am+YPvAt7zXosy/y1q76N6dfDKtFms5SoCBpvJ12RTCprlqUktz0wm3WKi1KAtjwqBUdm58gy0Zspdn94Ksc+L4dpnXiRkaxRbp4exl+V61uKw+8JTv7o59EBtW55CZTuKsp9ow9T4FduKMjEM2et+h6zwkADRcIKYRNKQedh8ZowWq8= ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-11T20:41:33.029 INFO:teuthology.orchestra.run.smithi023.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC/3X8stHmtjvuVWGX46FgHurXUnMqdG7HhyMbUtqscT4ERzjA8oKD/H/oaIhfPGBUwLN9fnlxjZsaUK284ZEZY+aiNbWZNiXUbcMjDmiQLi4jJP2O+T/vjQxBIlw83iVIuriboUJdsx+nNB7ffuy+hBlEqJryq/5f/vuYZEAqM5XbsZax5PL6YfsG7laFnql14cneLAOAJRDPB2VaajsUXoeCv9va7CnDMUDCiEBtj0HkBddHUog2HC8eqzmJFwpPYjEC/G9DYXSD4K+EP0PPYiMr4L2g7qgasMa22mgoet7s69mRMoGBLde5WF9pra3JuWXzqphARprQYqu+Am+YPvAt7zXosy/y1q76N6dfDKtFms5SoCBpvJ12RTCprlqUktz0wm3WKi1KAtjwqBUdm58gy0Zspdn94Ksc+L4dpnXiRkaxRbp4exl+V61uKw+8JTv7o59EBtW55CZTuKsp9ow9T4FduKMjEM2et+h6zwkADRcIKYRNKQedh8ZowWq8= ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715 2024-03-11T20:41:33.038 DEBUG:teuthology.orchestra.run.smithi146:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC/3X8stHmtjvuVWGX46FgHurXUnMqdG7HhyMbUtqscT4ERzjA8oKD/H/oaIhfPGBUwLN9fnlxjZsaUK284ZEZY+aiNbWZNiXUbcMjDmiQLi4jJP2O+T/vjQxBIlw83iVIuriboUJdsx+nNB7ffuy+hBlEqJryq/5f/vuYZEAqM5XbsZax5PL6YfsG7laFnql14cneLAOAJRDPB2VaajsUXoeCv9va7CnDMUDCiEBtj0HkBddHUog2HC8eqzmJFwpPYjEC/G9DYXSD4K+EP0PPYiMr4L2g7qgasMa22mgoet7s69mRMoGBLde5WF9pra3JuWXzqphARprQYqu+Am+YPvAt7zXosy/y1q76N6dfDKtFms5SoCBpvJ12RTCprlqUktz0wm3WKi1KAtjwqBUdm58gy0Zspdn94Ksc+L4dpnXiRkaxRbp4exl+V61uKw+8JTv7o59EBtW55CZTuKsp9ow9T4FduKMjEM2et+h6zwkADRcIKYRNKQedh8ZowWq8= ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-11T20:41:33.062 INFO:teuthology.orchestra.run.smithi146.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC/3X8stHmtjvuVWGX46FgHurXUnMqdG7HhyMbUtqscT4ERzjA8oKD/H/oaIhfPGBUwLN9fnlxjZsaUK284ZEZY+aiNbWZNiXUbcMjDmiQLi4jJP2O+T/vjQxBIlw83iVIuriboUJdsx+nNB7ffuy+hBlEqJryq/5f/vuYZEAqM5XbsZax5PL6YfsG7laFnql14cneLAOAJRDPB2VaajsUXoeCv9va7CnDMUDCiEBtj0HkBddHUog2HC8eqzmJFwpPYjEC/G9DYXSD4K+EP0PPYiMr4L2g7qgasMa22mgoet7s69mRMoGBLde5WF9pra3JuWXzqphARprQYqu+Am+YPvAt7zXosy/y1q76N6dfDKtFms5SoCBpvJ12RTCprlqUktz0wm3WKi1KAtjwqBUdm58gy0Zspdn94Ksc+L4dpnXiRkaxRbp4exl+V61uKw+8JTv7o59EBtW55CZTuKsp9ow9T4FduKMjEM2et+h6zwkADRcIKYRNKQedh8ZowWq8= ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715 2024-03-11T20:41:33.071 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-11T20:41:33.532 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:41:33 smithi023 bash[21372]: audit 2024-03-11T20:41:32.181872+0000 mon.a (mon.0) 93 : audit 1 from='client.? 172.21.15.23:0/2841046255' entity='client.admin' 2024-03-11T20:41:37.761 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:41:41.576 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-11T20:41:41.576 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-11T20:41:42.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:41:41 smithi023 bash[21372]: audit 2024-03-11T20:41:40.583685+0000 mon.a (mon.0) 94 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:41:42.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:41:41 smithi023 bash[21372]: audit 2024-03-11T20:41:40.588329+0000 mon.a (mon.0) 95 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:41:42.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:41:41 smithi023 bash[21372]: audit 2024-03-11T20:41:40.589282+0000 mon.a (mon.0) 96 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi023", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:41:42.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:41:41 smithi023 bash[21372]: audit 2024-03-11T20:41:40.595221+0000 mon.a (mon.0) 97 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:41:42.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:41:41 smithi023 bash[21372]: audit 2024-03-11T20:41:40.612772+0000 mon.a (mon.0) 98 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:41:42.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:41:41 smithi023 bash[21372]: audit 2024-03-11T20:41:40.617890+0000 mon.a (mon.0) 99 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:41:42.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:41:41 smithi023 bash[21372]: audit 2024-03-11T20:41:40.935674+0000 mon.a (mon.0) 100 : audit 1 from='client.? 172.21.15.23:0/2638230701' entity='client.admin' 2024-03-11T20:41:42.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:41:41 smithi023 bash[21372]: audit 2024-03-11T20:41:40.949303+0000 mon.a (mon.0) 101 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:41:42.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:41:41 smithi023 bash[21372]: audit 2024-03-11T20:41:40.955769+0000 mon.a (mon.0) 102 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:41:43.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:41:42 smithi023 bash[21372]: cluster 2024-03-11T20:41:41.315996+0000 mgr.y (mgr.14152) 9 : cluster 0 pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:41:45.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:41:44 smithi023 bash[21372]: cluster 2024-03-11T20:41:43.316433+0000 mgr.y (mgr.14152) 10 : cluster 0 pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:41:46.296 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:41:47.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:41:46 smithi023 bash[21372]: cluster 2024-03-11T20:41:45.316855+0000 mgr.y (mgr.14152) 11 : cluster 0 pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:41:49.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:41:48 smithi023 bash[21372]: cluster 2024-03-11T20:41:47.317264+0000 mgr.y (mgr.14152) 12 : cluster 0 pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:41:49.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:41:48 smithi023 bash[21372]: audit 2024-03-11T20:41:48.726205+0000 mon.a (mon.0) 103 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:41:49.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:41:48 smithi023 bash[21372]: audit 2024-03-11T20:41:48.755505+0000 mon.a (mon.0) 104 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:41:49.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:41:48 smithi023 bash[21372]: audit 2024-03-11T20:41:48.756938+0000 mon.a (mon.0) 105 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:41:49.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:41:48 smithi023 bash[21372]: audit 2024-03-11T20:41:48.757561+0000 mon.a (mon.0) 106 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:41:49.399 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi146 2024-03-11T20:41:49.400 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-11T20:41:49.400 DEBUG:teuthology.orchestra.run.smithi146:> dd of=/etc/ceph/ceph.conf 2024-03-11T20:41:49.406 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-11T20:41:49.407 DEBUG:teuthology.orchestra.run.smithi146:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-11T20:41:49.453 INFO:tasks.cephadm:Adding host smithi146 to orchestrator... 2024-03-11T20:41:49.454 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph orch host add smithi146 2024-03-11T20:41:50.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:41:49 smithi023 bash[21372]: audit 2024-03-11T20:41:48.719515+0000 mgr.y (mgr.14152) 13 : audit 0 from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:41:50.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:41:49 smithi023 bash[21372]: cephadm 2024-03-11T20:41:48.758472+0000 mgr.y (mgr.14152) 14 : cephadm 1 Updating smithi023:/etc/ceph/ceph.conf 2024-03-11T20:41:50.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:41:49 smithi023 bash[21372]: cephadm 2024-03-11T20:41:48.872247+0000 mgr.y (mgr.14152) 15 : cephadm 1 Updating smithi023:/var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T20:41:50.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:41:49 smithi023 bash[21372]: audit 2024-03-11T20:41:49.422059+0000 mon.a (mon.0) 107 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:41:50.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:41:49 smithi023 bash[21372]: audit 2024-03-11T20:41:49.425766+0000 mon.a (mon.0) 108 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:41:50.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:41:49 smithi023 bash[21372]: audit 2024-03-11T20:41:49.447691+0000 mon.a (mon.0) 109 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:41:51.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:41:50 smithi023 bash[21372]: cephadm 2024-03-11T20:41:49.087585+0000 mgr.y (mgr.14152) 16 : cephadm 1 Updating smithi023:/etc/ceph/ceph.client.admin.keyring 2024-03-11T20:41:51.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:41:50 smithi023 bash[21372]: cephadm 2024-03-11T20:41:49.253182+0000 mgr.y (mgr.14152) 17 : cephadm 1 Updating smithi023:/var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T20:41:51.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:41:50 smithi023 bash[21372]: cluster 2024-03-11T20:41:49.317664+0000 mgr.y (mgr.14152) 18 : cluster 0 pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:41:53.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:41:52 smithi023 bash[21372]: cluster 2024-03-11T20:41:51.318141+0000 mgr.y (mgr.14152) 19 : cluster 0 pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:41:54.166 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:41:55.245 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:41:54 smithi023 bash[21372]: cluster 2024-03-11T20:41:53.318594+0000 mgr.y (mgr.14152) 20 : cluster 0 pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:41:57.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:41:56 smithi023 bash[21372]: cluster 2024-03-11T20:41:55.319090+0000 mgr.y (mgr.14152) 21 : cluster 0 pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:41:58.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:41:58 smithi023 bash[21372]: audit 2024-03-11T20:41:56.681616+0000 mgr.y (mgr.14152) 22 : audit 0 from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi146", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:41:59.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:41:59 smithi023 bash[21372]: cluster 2024-03-11T20:41:57.319591+0000 mgr.y (mgr.14152) 23 : cluster 0 pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:41:59.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:41:59 smithi023 bash[21372]: cephadm 2024-03-11T20:41:57.481320+0000 mgr.y (mgr.14152) 24 : cephadm 1 Deploying cephadm binary to smithi146 2024-03-11T20:42:01.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:01 smithi023 bash[21372]: cluster 2024-03-11T20:41:59.319867+0000 mgr.y (mgr.14152) 25 : cluster 0 pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:42:02.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:02 smithi023 bash[21372]: cluster 2024-03-11T20:42:01.320223+0000 mgr.y (mgr.14152) 26 : cluster 0 pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:42:02.736 INFO:teuthology.orchestra.run.smithi023.stdout:Added host 'smithi146' with addr '172.21.15.146' 2024-03-11T20:42:03.374 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph orch host ls --format=json 2024-03-11T20:42:04.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:03 smithi023 bash[21372]: audit 2024-03-11T20:42:02.741155+0000 mon.a (mon.0) 110 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:42:04.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:03 smithi023 bash[21372]: cephadm 2024-03-11T20:42:02.742147+0000 mgr.y (mgr.14152) 27 : cephadm 1 Added host smithi146 2024-03-11T20:42:04.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:03 smithi023 bash[21372]: audit 2024-03-11T20:42:02.765512+0000 mon.a (mon.0) 111 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:42:04.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:03 smithi023 bash[21372]: audit 2024-03-11T20:42:03.238584+0000 mon.a (mon.0) 112 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:42:05.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:04 smithi023 bash[21372]: cluster 2024-03-11T20:42:03.320621+0000 mgr.y (mgr.14152) 28 : cluster 0 pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:42:05.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:04 smithi023 bash[21372]: audit 2024-03-11T20:42:04.714196+0000 mon.a (mon.0) 113 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:42:07.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:06 smithi023 bash[21372]: cluster 2024-03-11T20:42:05.320964+0000 mgr.y (mgr.14152) 29 : cluster 0 pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:42:07.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:06 smithi023 bash[21372]: audit 2024-03-11T20:42:05.865649+0000 mon.a (mon.0) 114 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:42:08.091 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:42:09.112 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:08 smithi023 bash[21372]: cluster 2024-03-11T20:42:07.321358+0000 mgr.y (mgr.14152) 30 : cluster 0 pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:42:10.517 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-11T20:42:10.517 INFO:teuthology.orchestra.run.smithi023.stdout:[{"addr": "172.21.15.23", "hostname": "smithi023", "labels": [], "status": ""}, {"addr": "172.21.15.146", "hostname": "smithi146", "labels": [], "status": ""}] 2024-03-11T20:42:11.183 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:10 smithi023 bash[21372]: cluster 2024-03-11T20:42:09.321770+0000 mgr.y (mgr.14152) 31 : cluster 0 pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:42:11.185 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-11T20:42:11.185 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd crush tunables default 2024-03-11T20:42:12.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:11 smithi023 bash[21372]: audit 2024-03-11T20:42:10.523146+0000 mgr.y (mgr.14152) 32 : audit 0 from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T20:42:12.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:11 smithi023 bash[21372]: audit 2024-03-11T20:42:10.950857+0000 mon.a (mon.0) 115 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:42:12.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:11 smithi023 bash[21372]: audit 2024-03-11T20:42:10.956344+0000 mon.a (mon.0) 116 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:42:12.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:11 smithi023 bash[21372]: audit 2024-03-11T20:42:10.963468+0000 mon.a (mon.0) 117 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:42:12.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:11 smithi023 bash[21372]: audit 2024-03-11T20:42:10.969292+0000 mon.a (mon.0) 118 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:42:12.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:11 smithi023 bash[21372]: audit 2024-03-11T20:42:10.970805+0000 mon.a (mon.0) 119 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi146", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:42:12.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:11 smithi023 bash[21372]: audit 2024-03-11T20:42:10.972771+0000 mon.a (mon.0) 120 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:42:12.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:11 smithi023 bash[21372]: audit 2024-03-11T20:42:10.974673+0000 mon.a (mon.0) 121 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:42:12.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:11 smithi023 bash[21372]: cephadm 2024-03-11T20:42:10.977382+0000 mgr.y (mgr.14152) 33 : cephadm 1 Updating smithi146:/etc/ceph/ceph.conf 2024-03-11T20:42:12.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:11 smithi023 bash[21372]: audit 2024-03-11T20:42:11.698489+0000 mon.a (mon.0) 122 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:42:12.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:11 smithi023 bash[21372]: audit 2024-03-11T20:42:11.703948+0000 mon.a (mon.0) 123 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:42:12.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:11 smithi023 bash[21372]: audit 2024-03-11T20:42:11.709846+0000 mon.a (mon.0) 124 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:42:13.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:12 smithi023 bash[21372]: cephadm 2024-03-11T20:42:11.194265+0000 mgr.y (mgr.14152) 34 : cephadm 1 Updating smithi146:/var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T20:42:13.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:12 smithi023 bash[21372]: cluster 2024-03-11T20:42:11.322145+0000 mgr.y (mgr.14152) 35 : cluster 0 pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:42:13.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:12 smithi023 bash[21372]: cephadm 2024-03-11T20:42:11.358882+0000 mgr.y (mgr.14152) 36 : cephadm 1 Updating smithi146:/etc/ceph/ceph.client.admin.keyring 2024-03-11T20:42:13.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:12 smithi023 bash[21372]: cephadm 2024-03-11T20:42:11.523363+0000 mgr.y (mgr.14152) 37 : cephadm 1 Updating smithi146:/var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T20:42:15.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:14 smithi023 bash[21372]: cluster 2024-03-11T20:42:13.322610+0000 mgr.y (mgr.14152) 38 : cluster 0 pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:42:15.903 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:42:17.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:16 smithi023 bash[21372]: cluster 2024-03-11T20:42:15.323045+0000 mgr.y (mgr.14152) 39 : cluster 0 pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:42:18.991 INFO:teuthology.orchestra.run.smithi023.stderr:adjusted tunables profile to default 2024-03-11T20:42:19.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:18 smithi023 bash[21372]: cluster 2024-03-11T20:42:17.323347+0000 mgr.y (mgr.14152) 40 : cluster 0 pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:42:19.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:18 smithi023 bash[21372]: audit 2024-03-11T20:42:18.372894+0000 mon.a (mon.0) 125 : audit 1 from='client.? 172.21.15.23:0/265517557' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-11T20:42:19.590 INFO:tasks.cephadm:Adding mon.a on smithi023 2024-03-11T20:42:19.590 INFO:tasks.cephadm:Adding mon.c on smithi023 2024-03-11T20:42:19.590 INFO:tasks.cephadm:Adding mon.b on smithi146 2024-03-11T20:42:19.590 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph orch apply mon '3;smithi023:172.21.15.23=a;smithi023:[v2:172.21.15.23:3301,v1:172.21.15.23:6790]=c;smithi146:172.21.15.146=b' 2024-03-11T20:42:20.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:20 smithi023 bash[21372]: audit 2024-03-11T20:42:18.992763+0000 mon.a (mon.0) 126 : audit 1 from='client.? 172.21.15.23:0/265517557' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-11T20:42:20.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:20 smithi023 bash[21372]: cluster 2024-03-11T20:42:18.995412+0000 mon.a (mon.0) 127 : cluster 0 osdmap e4: 0 total, 0 up, 0 in 2024-03-11T20:42:20.799 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T20:42:21.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:21 smithi023 bash[21372]: cluster 2024-03-11T20:42:19.323777+0000 mgr.y (mgr.14152) 41 : cluster 0 pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:42:23.260 INFO:teuthology.orchestra.run.smithi146.stdout:Scheduled mon update... 2024-03-11T20:42:23.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:23 smithi023 bash[21372]: cluster 2024-03-11T20:42:21.324142+0000 mgr.y (mgr.14152) 42 : cluster 0 pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:42:23.940 DEBUG:teuthology.orchestra.run.smithi023:mon.c> sudo journalctl -f -n 0 -u ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@mon.c.service 2024-03-11T20:42:23.943 DEBUG:teuthology.orchestra.run.smithi146:mon.b> sudo journalctl -f -n 0 -u ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@mon.b.service 2024-03-11T20:42:23.945 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-11T20:42:23.945 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph mon dump -f json 2024-03-11T20:42:24.532 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:24 smithi023 bash[21372]: audit 2024-03-11T20:42:23.253859+0000 mgr.y (mgr.14152) 43 : audit 0 from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi023:172.21.15.23=a;smithi023:[v2:172.21.15.23:3301,v1:172.21.15.23:6790]=c;smithi146:172.21.15.146=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:42:24.532 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:24 smithi023 bash[21372]: cephadm 2024-03-11T20:42:23.259937+0000 mgr.y (mgr.14152) 44 : cephadm 1 Saving service mon spec with placement smithi023:172.21.15.23=a;smithi023:[v2:172.21.15.23:3301,v1:172.21.15.23:6790]=c;smithi146:172.21.15.146=b;count:3 2024-03-11T20:42:24.532 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:24 smithi023 bash[21372]: audit 2024-03-11T20:42:23.265502+0000 mon.a (mon.0) 128 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:42:24.532 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:24 smithi023 bash[21372]: audit 2024-03-11T20:42:23.268884+0000 mon.a (mon.0) 129 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:42:24.532 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:24 smithi023 bash[21372]: audit 2024-03-11T20:42:23.272853+0000 mon.a (mon.0) 130 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:42:24.532 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:24 smithi023 bash[21372]: audit 2024-03-11T20:42:23.274813+0000 mon.a (mon.0) 131 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:42:24.532 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:24 smithi023 bash[21372]: audit 2024-03-11T20:42:23.282792+0000 mon.a (mon.0) 132 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:42:24.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:24 smithi023 bash[21372]: audit 2024-03-11T20:42:23.286867+0000 mon.a (mon.0) 133 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T20:42:24.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:24 smithi023 bash[21372]: audit 2024-03-11T20:42:23.288672+0000 mon.a (mon.0) 134 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:42:24.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:24 smithi023 bash[21372]: cephadm 2024-03-11T20:42:23.290842+0000 mgr.y (mgr.14152) 45 : cephadm 1 Deploying daemon mon.b on smithi146 2024-03-11T20:42:24.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:24 smithi023 bash[21372]: cluster 2024-03-11T20:42:23.324669+0000 mgr.y (mgr.14152) 46 : cluster 0 pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:42:25.761 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T20:42:26.781 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:26 smithi023 bash[21372]: cluster 2024-03-11T20:42:25.325189+0000 mgr.y (mgr.14152) 47 : cluster 0 pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:42:28.769 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-11T20:42:28.769 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"67ba00f8-dfe7-11ee-95c8-87774f69a715","modified":"2024-03-11T20:40:09.377861Z","created":"2024-03-11T20:40:09.377861Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-11T20:42:28.772 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-03-11T20:42:28.781 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:28 smithi023 bash[21372]: cluster 2024-03-11T20:42:27.325625+0000 mgr.y (mgr.14152) 48 : cluster 0 pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:42:29.893 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:29 smithi023 bash[21372]: audit 2024-03-11T20:42:28.629219+0000 mon.a (mon.0) 135 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:42:29.893 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:29 smithi023 bash[21372]: audit 2024-03-11T20:42:28.639481+0000 mon.a (mon.0) 136 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:42:29.893 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:29 smithi023 bash[21372]: audit 2024-03-11T20:42:28.673401+0000 mon.a (mon.0) 137 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:42:29.893 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:29 smithi023 bash[21372]: audit 2024-03-11T20:42:28.674784+0000 mon.a (mon.0) 138 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T20:42:29.894 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:29 smithi023 bash[21372]: audit 2024-03-11T20:42:28.676547+0000 mon.a (mon.0) 139 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:42:29.894 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:29 smithi023 bash[21372]: cephadm 2024-03-11T20:42:28.678742+0000 mgr.y (mgr.14152) 49 : cephadm 1 Deploying daemon mon.c on smithi023 2024-03-11T20:42:29.894 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:29 smithi023 bash[21372]: audit 2024-03-11T20:42:28.775584+0000 mon.a (mon.0) 140 : audit 0 from='client.? 172.21.15.146:0/1391876344' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T20:42:30.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:42:29 smithi146 bash[23809]: debug 2024-03-11T20:42:29.832+0000 7f82d13e4700 1 mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-11T20:42:30.650 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-11T20:42:30.650 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph mon dump -f json 2024-03-11T20:42:32.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:32 smithi023 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:42:32.281 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:42:32 smithi023 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:42:32.781 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:32 smithi023 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:42:32.782 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:42:32 smithi023 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:42:33.781 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:42:33 smithi023 bash[29146]: debug 2024-03-11T20:42:33.411+0000 7fedb11a9700 1 mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-11T20:42:34.352 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T20:42:35.226 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:42:34 smithi146 bash[23809]: cluster 2024-03-11T20:42:29.326051+0000 mgr.y (mgr.14152) 50 : cluster 0 pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:42:35.227 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:42:34 smithi146 bash[23809]: audit 2024-03-11T20:42:29.839961+0000 mon.a (mon.0) 141 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T20:42:35.227 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:42:34 smithi146 bash[23809]: audit 2024-03-11T20:42:29.844189+0000 mon.a (mon.0) 142 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T20:42:35.227 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:42:34 smithi146 bash[23809]: cluster 2024-03-11T20:42:29.844580+0000 mon.a (mon.0) 143 : cluster 1 mon.a calling monitor election 2024-03-11T20:42:35.227 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:42:34 smithi146 bash[23809]: audit 2024-03-11T20:42:29.851501+0000 mon.a (mon.0) 144 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T20:42:35.227 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:42:34 smithi146 bash[23809]: audit 2024-03-11T20:42:30.839241+0000 mon.a (mon.0) 145 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T20:42:35.228 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:42:34 smithi146 bash[23809]: cluster 2024-03-11T20:42:31.326383+0000 mgr.y (mgr.14152) 51 : cluster 0 pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:42:35.228 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:42:34 smithi146 bash[23809]: audit 2024-03-11T20:42:31.839909+0000 mon.a (mon.0) 146 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T20:42:35.228 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:42:34 smithi146 bash[23809]: cluster 2024-03-11T20:42:31.841963+0000 mon.b (mon.1) 1 : cluster 1 mon.b calling monitor election 2024-03-11T20:42:35.228 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:42:34 smithi146 bash[23809]: audit 2024-03-11T20:42:32.840228+0000 mon.a (mon.0) 147 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T20:42:35.228 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:42:34 smithi146 bash[23809]: cluster 2024-03-11T20:42:33.326695+0000 mgr.y (mgr.14152) 52 : cluster 0 pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:42:35.228 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:42:34 smithi146 bash[23809]: audit 2024-03-11T20:42:33.421983+0000 mon.a (mon.0) 148 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T20:42:35.229 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:42:34 smithi146 bash[23809]: audit 2024-03-11T20:42:33.840066+0000 mon.a (mon.0) 149 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T20:42:35.229 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:42:34 smithi146 bash[23809]: audit 2024-03-11T20:42:34.422743+0000 mon.a (mon.0) 150 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T20:42:35.229 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:42:34 smithi146 bash[23809]: audit 2024-03-11T20:42:34.840277+0000 mon.a (mon.0) 151 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T20:42:35.229 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:42:34 smithi146 bash[23809]: cluster 2024-03-11T20:42:34.874296+0000 mon.a (mon.0) 152 : cluster 1 mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-03-11T20:42:35.229 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:42:34 smithi146 bash[23809]: cluster 2024-03-11T20:42:34.881003+0000 mon.a (mon.0) 153 : cluster 0 monmap e2: 2 mons at {a=[v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0],b=[v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T20:42:35.229 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:42:34 smithi146 bash[23809]: cluster 2024-03-11T20:42:34.900335+0000 mon.a (mon.0) 154 : cluster 0 fsmap 2024-03-11T20:42:35.230 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:42:34 smithi146 bash[23809]: cluster 2024-03-11T20:42:34.900429+0000 mon.a (mon.0) 155 : cluster 0 osdmap e4: 0 total, 0 up, 0 in 2024-03-11T20:42:35.230 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:42:34 smithi146 bash[23809]: cluster 2024-03-11T20:42:34.900790+0000 mon.a (mon.0) 156 : cluster 0 mgrmap e13: y(active, since 73s) 2024-03-11T20:42:35.230 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:42:34 smithi146 bash[23809]: cluster 2024-03-11T20:42:34.901122+0000 mon.a (mon.0) 157 : cluster 1 overall HEALTH_OK 2024-03-11T20:42:35.230 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:42:34 smithi146 bash[23809]: audit 2024-03-11T20:42:34.909553+0000 mon.a (mon.0) 158 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:42:35.230 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:42:34 smithi146 bash[23809]: audit 2024-03-11T20:42:34.917343+0000 mon.a (mon.0) 159 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:42:35.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:34 smithi023 bash[21372]: cluster 2024-03-11T20:42:29.326051+0000 mgr.y (mgr.14152) 50 : cluster 0 pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:42:35.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:34 smithi023 bash[21372]: audit 2024-03-11T20:42:29.839961+0000 mon.a (mon.0) 141 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T20:42:35.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:34 smithi023 bash[21372]: audit 2024-03-11T20:42:29.844189+0000 mon.a (mon.0) 142 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T20:42:35.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:34 smithi023 bash[21372]: cluster 2024-03-11T20:42:29.844580+0000 mon.a (mon.0) 143 : cluster 1 mon.a calling monitor election 2024-03-11T20:42:35.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:34 smithi023 bash[21372]: audit 2024-03-11T20:42:29.851501+0000 mon.a (mon.0) 144 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T20:42:35.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:34 smithi023 bash[21372]: audit 2024-03-11T20:42:30.839241+0000 mon.a (mon.0) 145 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T20:42:35.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:34 smithi023 bash[21372]: cluster 2024-03-11T20:42:31.326383+0000 mgr.y (mgr.14152) 51 : cluster 0 pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:42:35.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:34 smithi023 bash[21372]: audit 2024-03-11T20:42:31.839909+0000 mon.a (mon.0) 146 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T20:42:35.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:34 smithi023 bash[21372]: cluster 2024-03-11T20:42:31.841963+0000 mon.b (mon.1) 1 : cluster 1 mon.b calling monitor election 2024-03-11T20:42:35.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:34 smithi023 bash[21372]: audit 2024-03-11T20:42:32.840228+0000 mon.a (mon.0) 147 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T20:42:35.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:34 smithi023 bash[21372]: cluster 2024-03-11T20:42:33.326695+0000 mgr.y (mgr.14152) 52 : cluster 0 pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:42:35.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:34 smithi023 bash[21372]: audit 2024-03-11T20:42:33.421983+0000 mon.a (mon.0) 148 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T20:42:35.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:34 smithi023 bash[21372]: audit 2024-03-11T20:42:33.840066+0000 mon.a (mon.0) 149 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T20:42:35.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:34 smithi023 bash[21372]: audit 2024-03-11T20:42:34.422743+0000 mon.a (mon.0) 150 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T20:42:35.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:34 smithi023 bash[21372]: audit 2024-03-11T20:42:34.840277+0000 mon.a (mon.0) 151 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T20:42:35.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:34 smithi023 bash[21372]: cluster 2024-03-11T20:42:34.874296+0000 mon.a (mon.0) 152 : cluster 1 mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-03-11T20:42:35.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:34 smithi023 bash[21372]: cluster 2024-03-11T20:42:34.881003+0000 mon.a (mon.0) 153 : cluster 0 monmap e2: 2 mons at {a=[v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0],b=[v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T20:42:35.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:34 smithi023 bash[21372]: cluster 2024-03-11T20:42:34.900335+0000 mon.a (mon.0) 154 : cluster 0 fsmap 2024-03-11T20:42:35.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:34 smithi023 bash[21372]: cluster 2024-03-11T20:42:34.900429+0000 mon.a (mon.0) 155 : cluster 0 osdmap e4: 0 total, 0 up, 0 in 2024-03-11T20:42:35.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:34 smithi023 bash[21372]: cluster 2024-03-11T20:42:34.900790+0000 mon.a (mon.0) 156 : cluster 0 mgrmap e13: y(active, since 73s) 2024-03-11T20:42:35.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:34 smithi023 bash[21372]: cluster 2024-03-11T20:42:34.901122+0000 mon.a (mon.0) 157 : cluster 1 overall HEALTH_OK 2024-03-11T20:42:35.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:34 smithi023 bash[21372]: audit 2024-03-11T20:42:34.909553+0000 mon.a (mon.0) 158 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:42:35.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:34 smithi023 bash[21372]: audit 2024-03-11T20:42:34.917343+0000 mon.a (mon.0) 159 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:42:36.281 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:42:35 smithi023 bash[21627]: debug 2024-03-11T20:42:35.835+0000 7f03c2e76700 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-11T20:42:40.782 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:40 smithi023 bash[21372]: cluster 2024-03-11T20:42:35.327071+0000 mgr.y (mgr.14152) 53 : cluster 0 pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:42:40.782 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:40 smithi023 bash[21372]: audit 2024-03-11T20:42:35.430612+0000 mon.a (mon.0) 164 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T20:42:40.782 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:40 smithi023 bash[21372]: cluster 2024-03-11T20:42:35.431063+0000 mon.a (mon.0) 165 : cluster 1 mon.a calling monitor election 2024-03-11T20:42:40.782 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:40 smithi023 bash[21372]: cluster 2024-03-11T20:42:35.432663+0000 mon.b (mon.1) 2 : cluster 1 mon.b calling monitor election 2024-03-11T20:42:40.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:40 smithi023 bash[21372]: audit 2024-03-11T20:42:35.452951+0000 mon.a (mon.0) 166 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T20:42:40.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:40 smithi023 bash[21372]: audit 2024-03-11T20:42:35.453286+0000 mon.a (mon.0) 167 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T20:42:40.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:40 smithi023 bash[21372]: audit 2024-03-11T20:42:36.423191+0000 mon.a (mon.0) 168 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T20:42:40.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:40 smithi023 bash[21372]: cluster 2024-03-11T20:42:37.327571+0000 mgr.y (mgr.14152) 54 : cluster 0 pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:42:40.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:40 smithi023 bash[21372]: audit 2024-03-11T20:42:37.423336+0000 mon.a (mon.0) 169 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T20:42:40.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:40 smithi023 bash[21372]: cluster 2024-03-11T20:42:37.425917+0000 mon.c (mon.2) 1 : cluster 1 mon.c calling monitor election 2024-03-11T20:42:40.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:40 smithi023 bash[21372]: audit 2024-03-11T20:42:38.423622+0000 mon.a (mon.0) 170 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T20:42:40.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:40 smithi023 bash[21372]: cluster 2024-03-11T20:42:39.327992+0000 mgr.y (mgr.14152) 55 : cluster 0 pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:42:40.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:40 smithi023 bash[21372]: audit 2024-03-11T20:42:39.423523+0000 mon.a (mon.0) 171 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T20:42:40.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:40 smithi023 bash[21372]: audit 2024-03-11T20:42:40.423533+0000 mon.a (mon.0) 172 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T20:42:40.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:40 smithi023 bash[21372]: cluster 2024-03-11T20:42:40.435315+0000 mon.a (mon.0) 173 : cluster 1 mon.a calling monitor election 2024-03-11T20:42:40.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:40 smithi023 bash[21372]: cluster 2024-03-11T20:42:40.474337+0000 mon.a (mon.0) 174 : cluster 1 mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-03-11T20:42:40.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:40 smithi023 bash[21372]: cluster 2024-03-11T20:42:40.483006+0000 mon.a (mon.0) 175 : cluster 0 monmap e3: 3 mons at {a=[v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0],b=[v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0],c=[v2:172.21.15.23:3301/0,v1:172.21.15.23:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T20:42:40.784 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:40 smithi023 bash[21372]: cluster 2024-03-11T20:42:40.500343+0000 mon.a (mon.0) 176 : cluster 0 fsmap 2024-03-11T20:42:40.784 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:40 smithi023 bash[21372]: cluster 2024-03-11T20:42:40.500433+0000 mon.a (mon.0) 177 : cluster 0 osdmap e4: 0 total, 0 up, 0 in 2024-03-11T20:42:40.784 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:40 smithi023 bash[21372]: cluster 2024-03-11T20:42:40.500707+0000 mon.a (mon.0) 178 : cluster 0 mgrmap e13: y(active, since 79s) 2024-03-11T20:42:40.784 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:40 smithi023 bash[21372]: cluster 2024-03-11T20:42:40.500947+0000 mon.a (mon.0) 179 : cluster 1 overall HEALTH_OK 2024-03-11T20:42:40.784 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:40 smithi023 bash[21372]: audit 2024-03-11T20:42:40.508703+0000 mon.a (mon.0) 180 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:42:40.784 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:40 smithi023 bash[21372]: audit 2024-03-11T20:42:40.514933+0000 mon.a (mon.0) 181 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:42:40.784 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:40 smithi023 bash[21372]: audit 2024-03-11T20:42:40.522904+0000 mon.a (mon.0) 182 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:42:40.784 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:40 smithi023 bash[21372]: audit 2024-03-11T20:42:40.529493+0000 mon.a (mon.0) 183 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:42:40.784 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:40 smithi023 bash[21372]: audit 2024-03-11T20:42:40.536190+0000 mon.a (mon.0) 184 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:42:40.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:42:40 smithi023 bash[29146]: cluster 2024-03-11T20:42:29.326051+0000 mgr.y (mgr.14152) 50 : cluster 0 pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:42:40.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:42:40 smithi023 bash[29146]: audit 2024-03-11T20:42:29.839961+0000 mon.a (mon.0) 141 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T20:42:40.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:42:40 smithi023 bash[29146]: audit 2024-03-11T20:42:29.844189+0000 mon.a (mon.0) 142 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T20:42:40.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:42:40 smithi023 bash[29146]: cluster 2024-03-11T20:42:29.844580+0000 mon.a (mon.0) 143 : cluster 1 mon.a calling monitor election 2024-03-11T20:42:40.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:42:40 smithi023 bash[29146]: audit 2024-03-11T20:42:29.851501+0000 mon.a (mon.0) 144 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T20:42:40.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:42:40 smithi023 bash[29146]: audit 2024-03-11T20:42:30.839241+0000 mon.a (mon.0) 145 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T20:42:40.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:42:40 smithi023 bash[29146]: cluster 2024-03-11T20:42:31.326383+0000 mgr.y (mgr.14152) 51 : cluster 0 pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:42:40.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:42:40 smithi023 bash[29146]: audit 2024-03-11T20:42:31.839909+0000 mon.a (mon.0) 146 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T20:42:40.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:42:40 smithi023 bash[29146]: cluster 2024-03-11T20:42:31.841963+0000 mon.b (mon.1) 1 : cluster 1 mon.b calling monitor election 2024-03-11T20:42:40.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:42:40 smithi023 bash[29146]: audit 2024-03-11T20:42:32.840228+0000 mon.a (mon.0) 147 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T20:42:40.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:42:40 smithi023 bash[29146]: cluster 2024-03-11T20:42:33.326695+0000 mgr.y (mgr.14152) 52 : cluster 0 pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:42:40.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:42:40 smithi023 bash[29146]: audit 2024-03-11T20:42:33.421983+0000 mon.a (mon.0) 148 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T20:42:40.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:42:40 smithi023 bash[29146]: audit 2024-03-11T20:42:33.840066+0000 mon.a (mon.0) 149 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T20:42:40.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:42:40 smithi023 bash[29146]: audit 2024-03-11T20:42:34.422743+0000 mon.a (mon.0) 150 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T20:42:40.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:42:40 smithi023 bash[29146]: audit 2024-03-11T20:42:34.840277+0000 mon.a (mon.0) 151 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T20:42:40.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:42:40 smithi023 bash[29146]: cluster 2024-03-11T20:42:34.874296+0000 mon.a (mon.0) 152 : cluster 1 mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-03-11T20:42:40.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:42:40 smithi023 bash[29146]: cluster 2024-03-11T20:42:34.881003+0000 mon.a (mon.0) 153 : cluster 0 monmap e2: 2 mons at {a=[v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0],b=[v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T20:42:40.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:42:40 smithi023 bash[29146]: cluster 2024-03-11T20:42:34.900335+0000 mon.a (mon.0) 154 : cluster 0 fsmap 2024-03-11T20:42:40.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:42:40 smithi023 bash[29146]: cluster 2024-03-11T20:42:34.900429+0000 mon.a (mon.0) 155 : cluster 0 osdmap e4: 0 total, 0 up, 0 in 2024-03-11T20:42:40.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:42:40 smithi023 bash[29146]: cluster 2024-03-11T20:42:34.900790+0000 mon.a (mon.0) 156 : cluster 0 mgrmap e13: y(active, since 73s) 2024-03-11T20:42:40.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:42:40 smithi023 bash[29146]: cluster 2024-03-11T20:42:34.901122+0000 mon.a (mon.0) 157 : cluster 1 overall HEALTH_OK 2024-03-11T20:42:40.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:42:40 smithi023 bash[29146]: audit 2024-03-11T20:42:34.909553+0000 mon.a (mon.0) 158 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:42:40.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:42:40 smithi023 bash[29146]: audit 2024-03-11T20:42:34.917343+0000 mon.a (mon.0) 159 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:42:40.787 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:42:40 smithi023 bash[29146]: cluster 2024-03-11T20:42:35.327071+0000 mgr.y (mgr.14152) 53 : cluster 0 pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:42:40.787 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:42:40 smithi023 bash[29146]: audit 2024-03-11T20:42:35.430612+0000 mon.a (mon.0) 164 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T20:42:40.787 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:42:40 smithi023 bash[29146]: cluster 2024-03-11T20:42:35.431063+0000 mon.a (mon.0) 165 : cluster 1 mon.a calling monitor election 2024-03-11T20:42:40.787 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:42:40 smithi023 bash[29146]: cluster 2024-03-11T20:42:35.432663+0000 mon.b (mon.1) 2 : cluster 1 mon.b calling monitor election 2024-03-11T20:42:40.787 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:42:40 smithi023 bash[29146]: audit 2024-03-11T20:42:35.452951+0000 mon.a (mon.0) 166 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T20:42:40.787 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:42:40 smithi023 bash[29146]: audit 2024-03-11T20:42:35.453286+0000 mon.a (mon.0) 167 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T20:42:40.787 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:42:40 smithi023 bash[29146]: audit 2024-03-11T20:42:36.423191+0000 mon.a (mon.0) 168 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T20:42:40.787 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:42:40 smithi023 bash[29146]: cluster 2024-03-11T20:42:37.327571+0000 mgr.y (mgr.14152) 54 : cluster 0 pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:42:40.787 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:42:40 smithi023 bash[29146]: audit 2024-03-11T20:42:37.423336+0000 mon.a (mon.0) 169 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T20:42:40.787 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:42:40 smithi023 bash[29146]: cluster 2024-03-11T20:42:37.425917+0000 mon.c (mon.2) 1 : cluster 1 mon.c calling monitor election 2024-03-11T20:42:40.787 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:42:40 smithi023 bash[29146]: audit 2024-03-11T20:42:38.423622+0000 mon.a (mon.0) 170 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T20:42:40.787 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:42:40 smithi023 bash[29146]: cluster 2024-03-11T20:42:39.327992+0000 mgr.y (mgr.14152) 55 : cluster 0 pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:42:40.787 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:42:40 smithi023 bash[29146]: audit 2024-03-11T20:42:39.423523+0000 mon.a (mon.0) 171 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T20:42:40.787 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:42:40 smithi023 bash[29146]: audit 2024-03-11T20:42:40.423533+0000 mon.a (mon.0) 172 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T20:42:40.788 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:42:40 smithi023 bash[29146]: cluster 2024-03-11T20:42:40.435315+0000 mon.a (mon.0) 173 : cluster 1 mon.a calling monitor election 2024-03-11T20:42:40.788 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:42:40 smithi023 bash[29146]: cluster 2024-03-11T20:42:40.474337+0000 mon.a (mon.0) 174 : cluster 1 mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-03-11T20:42:40.788 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:42:40 smithi023 bash[29146]: cluster 2024-03-11T20:42:40.483006+0000 mon.a (mon.0) 175 : cluster 0 monmap e3: 3 mons at {a=[v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0],b=[v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0],c=[v2:172.21.15.23:3301/0,v1:172.21.15.23:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T20:42:40.788 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:42:40 smithi023 bash[29146]: cluster 2024-03-11T20:42:40.500343+0000 mon.a (mon.0) 176 : cluster 0 fsmap 2024-03-11T20:42:40.788 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:42:40 smithi023 bash[29146]: cluster 2024-03-11T20:42:40.500433+0000 mon.a (mon.0) 177 : cluster 0 osdmap e4: 0 total, 0 up, 0 in 2024-03-11T20:42:40.788 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:42:40 smithi023 bash[29146]: cluster 2024-03-11T20:42:40.500707+0000 mon.a (mon.0) 178 : cluster 0 mgrmap e13: y(active, since 79s) 2024-03-11T20:42:40.788 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:42:40 smithi023 bash[29146]: cluster 2024-03-11T20:42:40.500947+0000 mon.a (mon.0) 179 : cluster 1 overall HEALTH_OK 2024-03-11T20:42:40.788 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:42:40 smithi023 bash[29146]: audit 2024-03-11T20:42:40.508703+0000 mon.a (mon.0) 180 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:42:40.788 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:42:40 smithi023 bash[29146]: audit 2024-03-11T20:42:40.514933+0000 mon.a (mon.0) 181 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:42:40.788 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:42:40 smithi023 bash[29146]: audit 2024-03-11T20:42:40.522904+0000 mon.a (mon.0) 182 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:42:40.788 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:42:40 smithi023 bash[29146]: audit 2024-03-11T20:42:40.529493+0000 mon.a (mon.0) 183 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:42:40.788 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:42:40 smithi023 bash[29146]: audit 2024-03-11T20:42:40.536190+0000 mon.a (mon.0) 184 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:42:41.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:42:40 smithi146 bash[23809]: cluster 2024-03-11T20:42:35.327071+0000 mgr.y (mgr.14152) 53 : cluster 0 pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:42:41.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:42:40 smithi146 bash[23809]: audit 2024-03-11T20:42:35.430612+0000 mon.a (mon.0) 164 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T20:42:41.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:42:40 smithi146 bash[23809]: cluster 2024-03-11T20:42:35.431063+0000 mon.a (mon.0) 165 : cluster 1 mon.a calling monitor election 2024-03-11T20:42:41.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:42:40 smithi146 bash[23809]: cluster 2024-03-11T20:42:35.432663+0000 mon.b (mon.1) 2 : cluster 1 mon.b calling monitor election 2024-03-11T20:42:41.013 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:42:40 smithi146 bash[23809]: audit 2024-03-11T20:42:35.452951+0000 mon.a (mon.0) 166 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T20:42:41.013 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:42:40 smithi146 bash[23809]: audit 2024-03-11T20:42:35.453286+0000 mon.a (mon.0) 167 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T20:42:41.013 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:42:40 smithi146 bash[23809]: audit 2024-03-11T20:42:36.423191+0000 mon.a (mon.0) 168 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T20:42:41.013 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:42:40 smithi146 bash[23809]: cluster 2024-03-11T20:42:37.327571+0000 mgr.y (mgr.14152) 54 : cluster 0 pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:42:41.013 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:42:40 smithi146 bash[23809]: audit 2024-03-11T20:42:37.423336+0000 mon.a (mon.0) 169 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T20:42:41.013 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:42:40 smithi146 bash[23809]: cluster 2024-03-11T20:42:37.425917+0000 mon.c (mon.2) 1 : cluster 1 mon.c calling monitor election 2024-03-11T20:42:41.013 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:42:40 smithi146 bash[23809]: audit 2024-03-11T20:42:38.423622+0000 mon.a (mon.0) 170 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T20:42:41.013 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:42:40 smithi146 bash[23809]: cluster 2024-03-11T20:42:39.327992+0000 mgr.y (mgr.14152) 55 : cluster 0 pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:42:41.013 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:42:40 smithi146 bash[23809]: audit 2024-03-11T20:42:39.423523+0000 mon.a (mon.0) 171 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T20:42:41.013 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:42:40 smithi146 bash[23809]: audit 2024-03-11T20:42:40.423533+0000 mon.a (mon.0) 172 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T20:42:41.013 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:42:40 smithi146 bash[23809]: cluster 2024-03-11T20:42:40.435315+0000 mon.a (mon.0) 173 : cluster 1 mon.a calling monitor election 2024-03-11T20:42:41.013 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:42:40 smithi146 bash[23809]: cluster 2024-03-11T20:42:40.474337+0000 mon.a (mon.0) 174 : cluster 1 mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-03-11T20:42:41.013 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:42:40 smithi146 bash[23809]: cluster 2024-03-11T20:42:40.483006+0000 mon.a (mon.0) 175 : cluster 0 monmap e3: 3 mons at {a=[v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0],b=[v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0],c=[v2:172.21.15.23:3301/0,v1:172.21.15.23:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T20:42:41.014 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:42:40 smithi146 bash[23809]: cluster 2024-03-11T20:42:40.500343+0000 mon.a (mon.0) 176 : cluster 0 fsmap 2024-03-11T20:42:41.014 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:42:40 smithi146 bash[23809]: cluster 2024-03-11T20:42:40.500433+0000 mon.a (mon.0) 177 : cluster 0 osdmap e4: 0 total, 0 up, 0 in 2024-03-11T20:42:41.014 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:42:40 smithi146 bash[23809]: cluster 2024-03-11T20:42:40.500707+0000 mon.a (mon.0) 178 : cluster 0 mgrmap e13: y(active, since 79s) 2024-03-11T20:42:41.014 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:42:40 smithi146 bash[23809]: cluster 2024-03-11T20:42:40.500947+0000 mon.a (mon.0) 179 : cluster 1 overall HEALTH_OK 2024-03-11T20:42:41.014 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:42:40 smithi146 bash[23809]: audit 2024-03-11T20:42:40.508703+0000 mon.a (mon.0) 180 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:42:41.014 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:42:40 smithi146 bash[23809]: audit 2024-03-11T20:42:40.514933+0000 mon.a (mon.0) 181 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:42:41.014 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:42:40 smithi146 bash[23809]: audit 2024-03-11T20:42:40.522904+0000 mon.a (mon.0) 182 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:42:41.014 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:42:40 smithi146 bash[23809]: audit 2024-03-11T20:42:40.529493+0000 mon.a (mon.0) 183 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:42:41.014 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:42:40 smithi146 bash[23809]: audit 2024-03-11T20:42:40.536190+0000 mon.a (mon.0) 184 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:42:41.333 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-11T20:42:41.333 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":3,"fsid":"67ba00f8-dfe7-11ee-95c8-87774f69a715","modified":"2024-03-11T20:42:35.423732Z","created":"2024-03-11T20:40:09.377861Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:3300","nonce":0},{"type":"v1","addr":"172.21.15.146:6789","nonce":0}]},"addr":"172.21.15.146:6789/0","public_addr":"172.21.15.146:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3301","nonce":0},{"type":"v1","addr":"172.21.15.23:6790","nonce":0}]},"addr":"172.21.15.23:6790/0","public_addr":"172.21.15.23:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-03-11T20:42:41.336 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 3 2024-03-11T20:42:41.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:42:41 smithi146 bash[23809]: audit 2024-03-11T20:42:41.212686+0000 mon.a (mon.0) 185 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:42:41.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:42:41 smithi146 bash[23809]: audit 2024-03-11T20:42:41.214619+0000 mon.a (mon.0) 186 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:42:41.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:42:41 smithi146 bash[23809]: audit 2024-03-11T20:42:41.339202+0000 mon.a (mon.0) 187 : audit 0 from='client.? 172.21.15.146:0/849851159' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T20:42:41.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:42:41 smithi146 bash[23809]: audit 2024-03-11T20:42:41.423861+0000 mon.a (mon.0) 188 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T20:42:42.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:41 smithi023 bash[21372]: audit 2024-03-11T20:42:41.212686+0000 mon.a (mon.0) 185 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:42:42.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:41 smithi023 bash[21372]: audit 2024-03-11T20:42:41.214619+0000 mon.a (mon.0) 186 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:42:42.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:41 smithi023 bash[21372]: audit 2024-03-11T20:42:41.339202+0000 mon.a (mon.0) 187 : audit 0 from='client.? 172.21.15.146:0/849851159' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T20:42:42.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:41 smithi023 bash[21372]: audit 2024-03-11T20:42:41.423861+0000 mon.a (mon.0) 188 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T20:42:42.033 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:42:41 smithi023 bash[29146]: audit 2024-03-11T20:42:41.212686+0000 mon.a (mon.0) 185 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:42:42.033 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:42:41 smithi023 bash[29146]: audit 2024-03-11T20:42:41.214619+0000 mon.a (mon.0) 186 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:42:42.033 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:42:41 smithi023 bash[29146]: audit 2024-03-11T20:42:41.339202+0000 mon.a (mon.0) 187 : audit 0 from='client.? 172.21.15.146:0/849851159' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T20:42:42.033 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:42:41 smithi023 bash[29146]: audit 2024-03-11T20:42:41.423861+0000 mon.a (mon.0) 188 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T20:42:42.085 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-11T20:42:42.085 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph config generate-minimal-conf 2024-03-11T20:42:43.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:42:42 smithi146 bash[23809]: cephadm 2024-03-11T20:42:41.217227+0000 mgr.y (mgr.14152) 56 : cephadm 1 Updating smithi023:/etc/ceph/ceph.conf 2024-03-11T20:42:43.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:42:42 smithi146 bash[23809]: cephadm 2024-03-11T20:42:41.217916+0000 mgr.y (mgr.14152) 57 : cephadm 1 Updating smithi146:/etc/ceph/ceph.conf 2024-03-11T20:42:43.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:42:42 smithi146 bash[23809]: cluster 2024-03-11T20:42:41.328351+0000 mgr.y (mgr.14152) 58 : cluster 0 pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:42:43.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:42:42 smithi146 bash[23809]: cephadm 2024-03-11T20:42:41.434049+0000 mgr.y (mgr.14152) 59 : cephadm 1 Updating smithi023:/var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T20:42:43.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:42:42 smithi146 bash[23809]: cephadm 2024-03-11T20:42:41.434809+0000 mgr.y (mgr.14152) 60 : cephadm 1 Updating smithi146:/var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T20:42:43.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:42:42 smithi146 bash[23809]: audit 2024-03-11T20:42:41.603013+0000 mon.a (mon.0) 189 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:42:43.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:42:42 smithi146 bash[23809]: audit 2024-03-11T20:42:41.608656+0000 mon.a (mon.0) 190 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:42:43.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:42:42 smithi146 bash[23809]: audit 2024-03-11T20:42:41.614476+0000 mon.a (mon.0) 191 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:42:43.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:42:42 smithi146 bash[23809]: audit 2024-03-11T20:42:41.619914+0000 mon.a (mon.0) 192 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:42:43.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:42:42 smithi146 bash[23809]: audit 2024-03-11T20:42:41.625260+0000 mon.a (mon.0) 193 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:42:43.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:42:42 smithi146 bash[23809]: audit 2024-03-11T20:42:41.698149+0000 mon.a (mon.0) 194 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:42:43.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:42:42 smithi146 bash[23809]: audit 2024-03-11T20:42:41.703903+0000 mon.a (mon.0) 195 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:42:43.013 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:42:42 smithi146 bash[23809]: audit 2024-03-11T20:42:41.709877+0000 mon.a (mon.0) 196 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:42:43.013 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:42:42 smithi146 bash[23809]: audit 2024-03-11T20:42:41.715870+0000 mon.a (mon.0) 197 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:42:43.013 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:42:42 smithi146 bash[23809]: cephadm 2024-03-11T20:42:41.716559+0000 mgr.y (mgr.14152) 61 : cephadm 1 Reconfiguring mon.a (unknown last config time)... 2024-03-11T20:42:43.013 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:42:42 smithi146 bash[23809]: audit 2024-03-11T20:42:41.717097+0000 mon.a (mon.0) 198 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T20:42:43.013 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:42:42 smithi146 bash[23809]: audit 2024-03-11T20:42:41.718506+0000 mon.a (mon.0) 199 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T20:42:43.013 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:42:42 smithi146 bash[23809]: audit 2024-03-11T20:42:41.719855+0000 mon.a (mon.0) 200 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:42:43.013 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:42:42 smithi146 bash[23809]: cephadm 2024-03-11T20:42:41.721560+0000 mgr.y (mgr.14152) 62 : cephadm 1 Reconfiguring daemon mon.a on smithi023 2024-03-11T20:42:43.032 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:42:42 smithi023 bash[29146]: cephadm 2024-03-11T20:42:41.217227+0000 mgr.y (mgr.14152) 56 : cephadm 1 Updating smithi023:/etc/ceph/ceph.conf 2024-03-11T20:42:43.032 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:42:42 smithi023 bash[29146]: cephadm 2024-03-11T20:42:41.217916+0000 mgr.y (mgr.14152) 57 : cephadm 1 Updating smithi146:/etc/ceph/ceph.conf 2024-03-11T20:42:43.032 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:42:42 smithi023 bash[29146]: cluster 2024-03-11T20:42:41.328351+0000 mgr.y (mgr.14152) 58 : cluster 0 pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:42:43.033 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:42:42 smithi023 bash[29146]: cephadm 2024-03-11T20:42:41.434049+0000 mgr.y (mgr.14152) 59 : cephadm 1 Updating smithi023:/var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T20:42:43.033 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:42:42 smithi023 bash[29146]: cephadm 2024-03-11T20:42:41.434809+0000 mgr.y (mgr.14152) 60 : cephadm 1 Updating smithi146:/var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T20:42:43.033 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:42:42 smithi023 bash[29146]: audit 2024-03-11T20:42:41.603013+0000 mon.a (mon.0) 189 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:42:43.033 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:42:42 smithi023 bash[29146]: audit 2024-03-11T20:42:41.608656+0000 mon.a (mon.0) 190 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:42:43.033 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:42:42 smithi023 bash[29146]: audit 2024-03-11T20:42:41.614476+0000 mon.a (mon.0) 191 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:42:43.033 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:42:42 smithi023 bash[29146]: audit 2024-03-11T20:42:41.619914+0000 mon.a (mon.0) 192 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:42:43.033 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:42:42 smithi023 bash[29146]: audit 2024-03-11T20:42:41.625260+0000 mon.a (mon.0) 193 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:42:43.033 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:42:42 smithi023 bash[29146]: audit 2024-03-11T20:42:41.698149+0000 mon.a (mon.0) 194 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:42:43.033 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:42:42 smithi023 bash[29146]: audit 2024-03-11T20:42:41.703903+0000 mon.a (mon.0) 195 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:42:43.033 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:42:42 smithi023 bash[29146]: audit 2024-03-11T20:42:41.709877+0000 mon.a (mon.0) 196 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:42:43.033 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:42:42 smithi023 bash[29146]: audit 2024-03-11T20:42:41.715870+0000 mon.a (mon.0) 197 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:42:43.033 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:42:42 smithi023 bash[29146]: cephadm 2024-03-11T20:42:41.716559+0000 mgr.y (mgr.14152) 61 : cephadm 1 Reconfiguring mon.a (unknown last config time)... 2024-03-11T20:42:43.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:42:42 smithi023 bash[29146]: audit 2024-03-11T20:42:41.717097+0000 mon.a (mon.0) 198 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T20:42:43.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:42:42 smithi023 bash[29146]: audit 2024-03-11T20:42:41.718506+0000 mon.a (mon.0) 199 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T20:42:43.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:42:42 smithi023 bash[29146]: audit 2024-03-11T20:42:41.719855+0000 mon.a (mon.0) 200 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:42:43.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:42:42 smithi023 bash[29146]: cephadm 2024-03-11T20:42:41.721560+0000 mgr.y (mgr.14152) 62 : cephadm 1 Reconfiguring daemon mon.a on smithi023 2024-03-11T20:42:43.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:42 smithi023 bash[21372]: cephadm 2024-03-11T20:42:41.217227+0000 mgr.y (mgr.14152) 56 : cephadm 1 Updating smithi023:/etc/ceph/ceph.conf 2024-03-11T20:42:43.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:42 smithi023 bash[21372]: cephadm 2024-03-11T20:42:41.217916+0000 mgr.y (mgr.14152) 57 : cephadm 1 Updating smithi146:/etc/ceph/ceph.conf 2024-03-11T20:42:43.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:42 smithi023 bash[21372]: cluster 2024-03-11T20:42:41.328351+0000 mgr.y (mgr.14152) 58 : cluster 0 pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:42:43.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:42 smithi023 bash[21372]: cephadm 2024-03-11T20:42:41.434049+0000 mgr.y (mgr.14152) 59 : cephadm 1 Updating smithi023:/var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T20:42:43.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:42 smithi023 bash[21372]: cephadm 2024-03-11T20:42:41.434809+0000 mgr.y (mgr.14152) 60 : cephadm 1 Updating smithi146:/var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T20:42:43.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:42 smithi023 bash[21372]: audit 2024-03-11T20:42:41.603013+0000 mon.a (mon.0) 189 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:42:43.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:42 smithi023 bash[21372]: audit 2024-03-11T20:42:41.608656+0000 mon.a (mon.0) 190 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:42:43.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:42 smithi023 bash[21372]: audit 2024-03-11T20:42:41.614476+0000 mon.a (mon.0) 191 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:42:43.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:42 smithi023 bash[21372]: audit 2024-03-11T20:42:41.619914+0000 mon.a (mon.0) 192 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:42:43.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:42 smithi023 bash[21372]: audit 2024-03-11T20:42:41.625260+0000 mon.a (mon.0) 193 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:42:43.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:42 smithi023 bash[21372]: audit 2024-03-11T20:42:41.698149+0000 mon.a (mon.0) 194 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:42:43.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:42 smithi023 bash[21372]: audit 2024-03-11T20:42:41.703903+0000 mon.a (mon.0) 195 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:42:43.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:42 smithi023 bash[21372]: audit 2024-03-11T20:42:41.709877+0000 mon.a (mon.0) 196 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:42:43.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:42 smithi023 bash[21372]: audit 2024-03-11T20:42:41.715870+0000 mon.a (mon.0) 197 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:42:43.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:42 smithi023 bash[21372]: cephadm 2024-03-11T20:42:41.716559+0000 mgr.y (mgr.14152) 61 : cephadm 1 Reconfiguring mon.a (unknown last config time)... 2024-03-11T20:42:43.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:42 smithi023 bash[21372]: audit 2024-03-11T20:42:41.717097+0000 mon.a (mon.0) 198 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T20:42:43.036 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:42 smithi023 bash[21372]: audit 2024-03-11T20:42:41.718506+0000 mon.a (mon.0) 199 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T20:42:43.036 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:42 smithi023 bash[21372]: audit 2024-03-11T20:42:41.719855+0000 mon.a (mon.0) 200 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:42:43.036 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:42 smithi023 bash[21372]: cephadm 2024-03-11T20:42:41.721560+0000 mgr.y (mgr.14152) 62 : cephadm 1 Reconfiguring daemon mon.a on smithi023 2024-03-11T20:42:45.093 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:42:44 smithi023 bash[29146]: cluster 2024-03-11T20:42:43.328763+0000 mgr.y (mgr.14152) 63 : cluster 0 pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:42:45.093 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:42:44 smithi023 bash[29146]: audit 2024-03-11T20:42:43.789704+0000 mon.a (mon.0) 201 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:42:45.093 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:42:44 smithi023 bash[29146]: audit 2024-03-11T20:42:43.797094+0000 mon.a (mon.0) 202 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:42:45.093 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:42:44 smithi023 bash[29146]: cephadm 2024-03-11T20:42:43.839908+0000 mgr.y (mgr.14152) 64 : cephadm 1 Reconfiguring mon.c (monmap changed)... 2024-03-11T20:42:45.093 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:42:44 smithi023 bash[29146]: audit 2024-03-11T20:42:43.840568+0000 mon.a (mon.0) 203 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T20:42:45.093 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:42:44 smithi023 bash[29146]: audit 2024-03-11T20:42:43.842395+0000 mon.a (mon.0) 204 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T20:42:45.093 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:42:44 smithi023 bash[29146]: audit 2024-03-11T20:42:43.844105+0000 mon.a (mon.0) 205 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:42:45.093 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:42:44 smithi023 bash[29146]: cephadm 2024-03-11T20:42:43.846162+0000 mgr.y (mgr.14152) 65 : cephadm 1 Reconfiguring daemon mon.c on smithi023 2024-03-11T20:42:45.094 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:44 smithi023 bash[21372]: cluster 2024-03-11T20:42:43.328763+0000 mgr.y (mgr.14152) 63 : cluster 0 pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:42:45.094 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:44 smithi023 bash[21372]: audit 2024-03-11T20:42:43.789704+0000 mon.a (mon.0) 201 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:42:45.094 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:44 smithi023 bash[21372]: audit 2024-03-11T20:42:43.797094+0000 mon.a (mon.0) 202 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:42:45.094 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:44 smithi023 bash[21372]: cephadm 2024-03-11T20:42:43.839908+0000 mgr.y (mgr.14152) 64 : cephadm 1 Reconfiguring mon.c (monmap changed)... 2024-03-11T20:42:45.094 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:44 smithi023 bash[21372]: audit 2024-03-11T20:42:43.840568+0000 mon.a (mon.0) 203 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T20:42:45.094 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:44 smithi023 bash[21372]: audit 2024-03-11T20:42:43.842395+0000 mon.a (mon.0) 204 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T20:42:45.094 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:44 smithi023 bash[21372]: audit 2024-03-11T20:42:43.844105+0000 mon.a (mon.0) 205 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:42:45.094 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:44 smithi023 bash[21372]: cephadm 2024-03-11T20:42:43.846162+0000 mgr.y (mgr.14152) 65 : cephadm 1 Reconfiguring daemon mon.c on smithi023 2024-03-11T20:42:45.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:42:44 smithi146 bash[23809]: cluster 2024-03-11T20:42:43.328763+0000 mgr.y (mgr.14152) 63 : cluster 0 pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:42:45.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:42:44 smithi146 bash[23809]: audit 2024-03-11T20:42:43.789704+0000 mon.a (mon.0) 201 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:42:45.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:42:44 smithi146 bash[23809]: audit 2024-03-11T20:42:43.797094+0000 mon.a (mon.0) 202 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:42:45.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:42:44 smithi146 bash[23809]: cephadm 2024-03-11T20:42:43.839908+0000 mgr.y (mgr.14152) 64 : cephadm 1 Reconfiguring mon.c (monmap changed)... 2024-03-11T20:42:45.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:42:44 smithi146 bash[23809]: audit 2024-03-11T20:42:43.840568+0000 mon.a (mon.0) 203 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T20:42:45.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:42:44 smithi146 bash[23809]: audit 2024-03-11T20:42:43.842395+0000 mon.a (mon.0) 204 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T20:42:45.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:42:44 smithi146 bash[23809]: audit 2024-03-11T20:42:43.844105+0000 mon.a (mon.0) 205 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:42:45.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:42:44 smithi146 bash[23809]: cephadm 2024-03-11T20:42:43.846162+0000 mgr.y (mgr.14152) 65 : cephadm 1 Reconfiguring daemon mon.c on smithi023 2024-03-11T20:42:46.808 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:42:46.914 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:42:46 smithi146 bash[23809]: cluster 2024-03-11T20:42:45.329219+0000 mgr.y (mgr.14152) 66 : cluster 0 pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:42:46.915 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:42:46 smithi146 bash[23809]: audit 2024-03-11T20:42:45.650628+0000 mon.a (mon.0) 206 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:42:46.915 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:42:46 smithi146 bash[23809]: audit 2024-03-11T20:42:45.657920+0000 mon.a (mon.0) 207 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:42:46.915 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:42:46 smithi146 bash[23809]: cephadm 2024-03-11T20:42:45.664818+0000 mgr.y (mgr.14152) 67 : cephadm 1 Reconfiguring mon.b (monmap changed)... 2024-03-11T20:42:46.915 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:42:46 smithi146 bash[23809]: audit 2024-03-11T20:42:45.665313+0000 mon.a (mon.0) 208 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T20:42:46.915 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:42:46 smithi146 bash[23809]: audit 2024-03-11T20:42:45.666675+0000 mon.a (mon.0) 209 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T20:42:46.915 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:42:46 smithi146 bash[23809]: audit 2024-03-11T20:42:45.667986+0000 mon.a (mon.0) 210 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:42:46.915 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:42:46 smithi146 bash[23809]: cephadm 2024-03-11T20:42:45.669628+0000 mgr.y (mgr.14152) 68 : cephadm 1 Reconfiguring daemon mon.b on smithi146 2024-03-11T20:42:47.031 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:42:46 smithi023 bash[29146]: cluster 2024-03-11T20:42:45.329219+0000 mgr.y (mgr.14152) 66 : cluster 0 pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:42:47.031 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:42:46 smithi023 bash[29146]: audit 2024-03-11T20:42:45.650628+0000 mon.a (mon.0) 206 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:42:47.032 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:42:46 smithi023 bash[29146]: audit 2024-03-11T20:42:45.657920+0000 mon.a (mon.0) 207 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:42:47.032 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:42:46 smithi023 bash[29146]: cephadm 2024-03-11T20:42:45.664818+0000 mgr.y (mgr.14152) 67 : cephadm 1 Reconfiguring mon.b (monmap changed)... 2024-03-11T20:42:47.032 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:42:46 smithi023 bash[29146]: audit 2024-03-11T20:42:45.665313+0000 mon.a (mon.0) 208 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T20:42:47.032 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:42:46 smithi023 bash[29146]: audit 2024-03-11T20:42:45.666675+0000 mon.a (mon.0) 209 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T20:42:47.032 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:42:46 smithi023 bash[29146]: audit 2024-03-11T20:42:45.667986+0000 mon.a (mon.0) 210 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:42:47.032 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:42:46 smithi023 bash[29146]: cephadm 2024-03-11T20:42:45.669628+0000 mgr.y (mgr.14152) 68 : cephadm 1 Reconfiguring daemon mon.b on smithi146 2024-03-11T20:42:47.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:46 smithi023 bash[21372]: cluster 2024-03-11T20:42:45.329219+0000 mgr.y (mgr.14152) 66 : cluster 0 pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:42:47.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:46 smithi023 bash[21372]: audit 2024-03-11T20:42:45.650628+0000 mon.a (mon.0) 206 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:42:47.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:46 smithi023 bash[21372]: audit 2024-03-11T20:42:45.657920+0000 mon.a (mon.0) 207 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:42:47.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:46 smithi023 bash[21372]: cephadm 2024-03-11T20:42:45.664818+0000 mgr.y (mgr.14152) 67 : cephadm 1 Reconfiguring mon.b (monmap changed)... 2024-03-11T20:42:47.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:46 smithi023 bash[21372]: audit 2024-03-11T20:42:45.665313+0000 mon.a (mon.0) 208 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T20:42:47.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:46 smithi023 bash[21372]: audit 2024-03-11T20:42:45.666675+0000 mon.a (mon.0) 209 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T20:42:47.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:46 smithi023 bash[21372]: audit 2024-03-11T20:42:45.667986+0000 mon.a (mon.0) 210 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:42:47.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:46 smithi023 bash[21372]: cephadm 2024-03-11T20:42:45.669628+0000 mgr.y (mgr.14152) 68 : cephadm 1 Reconfiguring daemon mon.b on smithi146 2024-03-11T20:42:48.532 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:48 smithi023 bash[21372]: cluster 2024-03-11T20:42:47.329583+0000 mgr.y (mgr.14152) 69 : cluster 0 pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:42:48.532 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:48 smithi023 bash[21372]: audit 2024-03-11T20:42:47.422915+0000 mon.a (mon.0) 211 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:42:48.532 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:48 smithi023 bash[21372]: audit 2024-03-11T20:42:47.432652+0000 mon.a (mon.0) 212 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:42:48.532 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:48 smithi023 bash[21372]: audit 2024-03-11T20:42:47.440193+0000 mon.a (mon.0) 213 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:42:48.532 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:48 smithi023 bash[21372]: audit 2024-03-11T20:42:47.443846+0000 mon.a (mon.0) 214 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:42:48.532 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:48 smithi023 bash[21372]: audit 2024-03-11T20:42:47.445557+0000 mon.a (mon.0) 215 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:42:48.532 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:48 smithi023 bash[21372]: audit 2024-03-11T20:42:47.455493+0000 mon.a (mon.0) 216 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:42:48.532 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:42:48 smithi023 bash[29146]: cluster 2024-03-11T20:42:47.329583+0000 mgr.y (mgr.14152) 69 : cluster 0 pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:42:48.533 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:42:48 smithi023 bash[29146]: audit 2024-03-11T20:42:47.422915+0000 mon.a (mon.0) 211 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:42:48.533 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:42:48 smithi023 bash[29146]: audit 2024-03-11T20:42:47.432652+0000 mon.a (mon.0) 212 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:42:48.533 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:42:48 smithi023 bash[29146]: audit 2024-03-11T20:42:47.440193+0000 mon.a (mon.0) 213 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:42:48.533 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:42:48 smithi023 bash[29146]: audit 2024-03-11T20:42:47.443846+0000 mon.a (mon.0) 214 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:42:48.533 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:42:48 smithi023 bash[29146]: audit 2024-03-11T20:42:47.445557+0000 mon.a (mon.0) 215 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:42:48.533 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:42:48 smithi023 bash[29146]: audit 2024-03-11T20:42:47.455493+0000 mon.a (mon.0) 216 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:42:48.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:42:48 smithi146 bash[23809]: cluster 2024-03-11T20:42:47.329583+0000 mgr.y (mgr.14152) 69 : cluster 0 pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:42:48.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:42:48 smithi146 bash[23809]: audit 2024-03-11T20:42:47.422915+0000 mon.a (mon.0) 211 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:42:48.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:42:48 smithi146 bash[23809]: audit 2024-03-11T20:42:47.432652+0000 mon.a (mon.0) 212 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:42:48.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:42:48 smithi146 bash[23809]: audit 2024-03-11T20:42:47.440193+0000 mon.a (mon.0) 213 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:42:48.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:42:48 smithi146 bash[23809]: audit 2024-03-11T20:42:47.443846+0000 mon.a (mon.0) 214 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:42:48.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:42:48 smithi146 bash[23809]: audit 2024-03-11T20:42:47.445557+0000 mon.a (mon.0) 215 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:42:48.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:42:48 smithi146 bash[23809]: audit 2024-03-11T20:42:47.455493+0000 mon.a (mon.0) 216 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:42:49.451 INFO:teuthology.orchestra.run.smithi023.stdout:# minimal ceph.conf for 67ba00f8-dfe7-11ee-95c8-87774f69a715 2024-03-11T20:42:49.452 INFO:teuthology.orchestra.run.smithi023.stdout:[global] 2024-03-11T20:42:49.452 INFO:teuthology.orchestra.run.smithi023.stdout: fsid = 67ba00f8-dfe7-11ee-95c8-87774f69a715 2024-03-11T20:42:49.452 INFO:teuthology.orchestra.run.smithi023.stdout: mon_host = [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] [v2:172.21.15.23:3301/0,v1:172.21.15.23:6790/0] 2024-03-11T20:42:50.080 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-11T20:42:50.080 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-11T20:42:50.080 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd of=/etc/ceph/ceph.conf 2024-03-11T20:42:50.094 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-11T20:42:50.094 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-11T20:42:50.152 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-11T20:42:50.153 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/etc/ceph/ceph.conf 2024-03-11T20:42:50.167 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-11T20:42:50.167 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-11T20:42:50.222 INFO:tasks.cephadm:Adding mgr.y on smithi023 2024-03-11T20:42:50.222 INFO:tasks.cephadm:Adding mgr.x on smithi146 2024-03-11T20:42:50.222 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph orch apply mgr '2;smithi023=y;smithi146=x' 2024-03-11T20:42:50.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:42:50 smithi146 bash[23809]: cluster 2024-03-11T20:42:49.330058+0000 mgr.y (mgr.14152) 70 : cluster 0 pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:42:50.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:42:50 smithi146 bash[23809]: audit 2024-03-11T20:42:49.458182+0000 mon.a (mon.0) 217 : audit 0 from='client.? 172.21.15.23:0/2837354503' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:42:50.781 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:50 smithi023 bash[21372]: cluster 2024-03-11T20:42:49.330058+0000 mgr.y (mgr.14152) 70 : cluster 0 pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:42:50.782 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:50 smithi023 bash[21372]: audit 2024-03-11T20:42:49.458182+0000 mon.a (mon.0) 217 : audit 0 from='client.? 172.21.15.23:0/2837354503' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:42:50.782 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:42:50 smithi023 bash[29146]: cluster 2024-03-11T20:42:49.330058+0000 mgr.y (mgr.14152) 70 : cluster 0 pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:42:50.782 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:42:50 smithi023 bash[29146]: audit 2024-03-11T20:42:49.458182+0000 mon.a (mon.0) 217 : audit 0 from='client.? 172.21.15.23:0/2837354503' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:42:52.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:42:52 smithi146 bash[23809]: cluster 2024-03-11T20:42:51.330590+0000 mgr.y (mgr.14152) 71 : cluster 0 pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:42:52.781 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:52 smithi023 bash[21372]: cluster 2024-03-11T20:42:51.330590+0000 mgr.y (mgr.14152) 71 : cluster 0 pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:42:52.782 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:42:52 smithi023 bash[29146]: cluster 2024-03-11T20:42:51.330590+0000 mgr.y (mgr.14152) 71 : cluster 0 pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:42:53.965 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T20:42:54.710 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:42:54 smithi146 bash[23809]: cluster 2024-03-11T20:42:53.331048+0000 mgr.y (mgr.14152) 72 : cluster 0 pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:42:54.781 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:54 smithi023 bash[21372]: cluster 2024-03-11T20:42:53.331048+0000 mgr.y (mgr.14152) 72 : cluster 0 pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:42:54.782 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:42:54 smithi023 bash[29146]: cluster 2024-03-11T20:42:53.331048+0000 mgr.y (mgr.14152) 72 : cluster 0 pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:42:56.396 INFO:teuthology.orchestra.run.smithi146.stdout:Scheduled mgr update... 2024-03-11T20:42:56.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:42:56 smithi146 bash[23809]: cluster 2024-03-11T20:42:55.331503+0000 mgr.y (mgr.14152) 73 : cluster 0 pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:42:56.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:42:56 smithi146 bash[23809]: audit 2024-03-11T20:42:56.401588+0000 mon.a (mon.0) 218 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:42:56.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:42:56 smithi146 bash[23809]: audit 2024-03-11T20:42:56.411836+0000 mon.a (mon.0) 219 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:42:56.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:42:56 smithi146 bash[23809]: audit 2024-03-11T20:42:56.415429+0000 mon.a (mon.0) 220 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:42:56.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:42:56 smithi146 bash[23809]: audit 2024-03-11T20:42:56.417261+0000 mon.a (mon.0) 221 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:42:56.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:42:56 smithi146 bash[23809]: audit 2024-03-11T20:42:56.427050+0000 mon.a (mon.0) 222 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:42:56.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:42:56 smithi146 bash[23809]: audit 2024-03-11T20:42:56.430360+0000 mon.a (mon.0) 223 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T20:42:56.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:42:56 smithi146 bash[23809]: audit 2024-03-11T20:42:56.432994+0000 mon.a (mon.0) 224 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-11T20:42:56.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:42:56 smithi146 bash[23809]: audit 2024-03-11T20:42:56.438242+0000 mon.a (mon.0) 225 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T20:42:56.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:42:56 smithi146 bash[23809]: audit 2024-03-11T20:42:56.440221+0000 mon.a (mon.0) 226 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:42:56.782 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:56 smithi023 bash[21372]: cluster 2024-03-11T20:42:55.331503+0000 mgr.y (mgr.14152) 73 : cluster 0 pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:42:56.782 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:56 smithi023 bash[21372]: audit 2024-03-11T20:42:56.401588+0000 mon.a (mon.0) 218 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:42:56.782 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:56 smithi023 bash[21372]: audit 2024-03-11T20:42:56.411836+0000 mon.a (mon.0) 219 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:42:56.782 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:56 smithi023 bash[21372]: audit 2024-03-11T20:42:56.415429+0000 mon.a (mon.0) 220 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:42:56.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:56 smithi023 bash[21372]: audit 2024-03-11T20:42:56.417261+0000 mon.a (mon.0) 221 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:42:56.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:56 smithi023 bash[21372]: audit 2024-03-11T20:42:56.427050+0000 mon.a (mon.0) 222 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:42:56.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:56 smithi023 bash[21372]: audit 2024-03-11T20:42:56.430360+0000 mon.a (mon.0) 223 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T20:42:56.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:56 smithi023 bash[21372]: audit 2024-03-11T20:42:56.432994+0000 mon.a (mon.0) 224 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-11T20:42:56.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:56 smithi023 bash[21372]: audit 2024-03-11T20:42:56.438242+0000 mon.a (mon.0) 225 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T20:42:56.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:56 smithi023 bash[21372]: audit 2024-03-11T20:42:56.440221+0000 mon.a (mon.0) 226 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:42:56.783 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:42:56 smithi023 bash[29146]: cluster 2024-03-11T20:42:55.331503+0000 mgr.y (mgr.14152) 73 : cluster 0 pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:42:56.783 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:42:56 smithi023 bash[29146]: audit 2024-03-11T20:42:56.401588+0000 mon.a (mon.0) 218 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:42:56.783 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:42:56 smithi023 bash[29146]: audit 2024-03-11T20:42:56.411836+0000 mon.a (mon.0) 219 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:42:56.783 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:42:56 smithi023 bash[29146]: audit 2024-03-11T20:42:56.415429+0000 mon.a (mon.0) 220 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:42:56.783 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:42:56 smithi023 bash[29146]: audit 2024-03-11T20:42:56.417261+0000 mon.a (mon.0) 221 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:42:56.784 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:42:56 smithi023 bash[29146]: audit 2024-03-11T20:42:56.427050+0000 mon.a (mon.0) 222 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:42:56.784 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:42:56 smithi023 bash[29146]: audit 2024-03-11T20:42:56.430360+0000 mon.a (mon.0) 223 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T20:42:56.784 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:42:56 smithi023 bash[29146]: audit 2024-03-11T20:42:56.432994+0000 mon.a (mon.0) 224 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-11T20:42:56.784 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:42:56 smithi023 bash[29146]: audit 2024-03-11T20:42:56.438242+0000 mon.a (mon.0) 225 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T20:42:56.784 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:42:56 smithi023 bash[29146]: audit 2024-03-11T20:42:56.440221+0000 mon.a (mon.0) 226 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:42:57.308 DEBUG:teuthology.orchestra.run.smithi146:mgr.x> sudo journalctl -f -n 0 -u ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@mgr.x.service 2024-03-11T20:42:57.311 INFO:tasks.cephadm:Deploying OSDs... 2024-03-11T20:42:57.311 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-11T20:42:57.311 DEBUG:teuthology.orchestra.run.smithi023:> dd if=/scratch_devs of=/dev/stdout 2024-03-11T20:42:57.317 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-11T20:42:57.317 DEBUG:teuthology.orchestra.run.smithi023:> stat /dev/vg_nvme/lv_1 2024-03-11T20:42:57.365 INFO:teuthology.orchestra.run.smithi023.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-11T20:42:57.365 INFO:teuthology.orchestra.run.smithi023.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T20:42:57.366 INFO:teuthology.orchestra.run.smithi023.stdout:Device: 5h/5d Inode: 865 Links: 1 2024-03-11T20:42:57.366 INFO:teuthology.orchestra.run.smithi023.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T20:42:57.366 INFO:teuthology.orchestra.run.smithi023.stdout:Access: 2024-03-11 20:33:45.869836376 +0000 2024-03-11T20:42:57.366 INFO:teuthology.orchestra.run.smithi023.stdout:Modify: 2024-03-11 20:33:45.597838802 +0000 2024-03-11T20:42:57.366 INFO:teuthology.orchestra.run.smithi023.stdout:Change: 2024-03-11 20:33:45.597838802 +0000 2024-03-11T20:42:57.366 INFO:teuthology.orchestra.run.smithi023.stdout: Birth: - 2024-03-11T20:42:57.366 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-11T20:42:57.420 INFO:teuthology.orchestra.run.smithi023.stderr:1+0 records in 2024-03-11T20:42:57.421 INFO:teuthology.orchestra.run.smithi023.stderr:1+0 records out 2024-03-11T20:42:57.421 INFO:teuthology.orchestra.run.smithi023.stderr:512 bytes copied, 0.000496584 s, 1.0 MB/s 2024-03-11T20:42:57.422 DEBUG:teuthology.orchestra.run.smithi023:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-11T20:42:57.472 DEBUG:teuthology.orchestra.run.smithi023:> stat /dev/vg_nvme/lv_2 2024-03-11T20:42:57.525 INFO:teuthology.orchestra.run.smithi023.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-11T20:42:57.526 INFO:teuthology.orchestra.run.smithi023.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T20:42:57.526 INFO:teuthology.orchestra.run.smithi023.stdout:Device: 5h/5d Inode: 882 Links: 1 2024-03-11T20:42:57.526 INFO:teuthology.orchestra.run.smithi023.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T20:42:57.526 INFO:teuthology.orchestra.run.smithi023.stdout:Access: 2024-03-11 20:33:46.353832058 +0000 2024-03-11T20:42:57.526 INFO:teuthology.orchestra.run.smithi023.stdout:Modify: 2024-03-11 20:33:46.117834163 +0000 2024-03-11T20:42:57.526 INFO:teuthology.orchestra.run.smithi023.stdout:Change: 2024-03-11 20:33:46.117834163 +0000 2024-03-11T20:42:57.527 INFO:teuthology.orchestra.run.smithi023.stdout: Birth: - 2024-03-11T20:42:57.527 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-11T20:42:57.582 INFO:teuthology.orchestra.run.smithi023.stderr:1+0 records in 2024-03-11T20:42:57.582 INFO:teuthology.orchestra.run.smithi023.stderr:1+0 records out 2024-03-11T20:42:57.582 INFO:teuthology.orchestra.run.smithi023.stderr:512 bytes copied, 0.000428188 s, 1.2 MB/s 2024-03-11T20:42:57.583 DEBUG:teuthology.orchestra.run.smithi023:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-11T20:42:57.634 DEBUG:teuthology.orchestra.run.smithi023:> stat /dev/vg_nvme/lv_3 2024-03-11T20:42:57.681 INFO:teuthology.orchestra.run.smithi023.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-11T20:42:57.681 INFO:teuthology.orchestra.run.smithi023.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T20:42:57.681 INFO:teuthology.orchestra.run.smithi023.stdout:Device: 5h/5d Inode: 902 Links: 1 2024-03-11T20:42:57.681 INFO:teuthology.orchestra.run.smithi023.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T20:42:57.682 INFO:teuthology.orchestra.run.smithi023.stdout:Access: 2024-03-11 20:33:46.877827383 +0000 2024-03-11T20:42:57.682 INFO:teuthology.orchestra.run.smithi023.stdout:Modify: 2024-03-11 20:33:46.625829632 +0000 2024-03-11T20:42:57.682 INFO:teuthology.orchestra.run.smithi023.stdout:Change: 2024-03-11 20:33:46.625829632 +0000 2024-03-11T20:42:57.682 INFO:teuthology.orchestra.run.smithi023.stdout: Birth: - 2024-03-11T20:42:57.683 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-11T20:42:57.735 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:57 smithi023 bash[21372]: audit 2024-03-11T20:42:56.388682+0000 mgr.y (mgr.14152) 74 : audit 0 from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi023=y;smithi146=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:42:57.735 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:57 smithi023 bash[21372]: cephadm 2024-03-11T20:42:56.392271+0000 mgr.y (mgr.14152) 75 : cephadm 1 Saving service mgr spec with placement smithi023=y;smithi146=x;count:2 2024-03-11T20:42:57.735 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:57 smithi023 bash[21372]: cephadm 2024-03-11T20:42:56.442323+0000 mgr.y (mgr.14152) 76 : cephadm 1 Deploying daemon mgr.x on smithi146 2024-03-11T20:42:57.735 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:42:57 smithi023 bash[29146]: audit 2024-03-11T20:42:56.388682+0000 mgr.y (mgr.14152) 74 : audit 0 from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi023=y;smithi146=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:42:57.735 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:42:57 smithi023 bash[29146]: cephadm 2024-03-11T20:42:56.392271+0000 mgr.y (mgr.14152) 75 : cephadm 1 Saving service mgr spec with placement smithi023=y;smithi146=x;count:2 2024-03-11T20:42:57.736 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:42:57 smithi023 bash[29146]: cephadm 2024-03-11T20:42:56.442323+0000 mgr.y (mgr.14152) 76 : cephadm 1 Deploying daemon mgr.x on smithi146 2024-03-11T20:42:57.736 INFO:teuthology.orchestra.run.smithi023.stderr:1+0 records in 2024-03-11T20:42:57.737 INFO:teuthology.orchestra.run.smithi023.stderr:1+0 records out 2024-03-11T20:42:57.737 INFO:teuthology.orchestra.run.smithi023.stderr:512 bytes copied, 0.000470727 s, 1.1 MB/s 2024-03-11T20:42:57.738 DEBUG:teuthology.orchestra.run.smithi023:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-11T20:42:57.738 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:42:57 smithi146 bash[23809]: audit 2024-03-11T20:42:56.388682+0000 mgr.y (mgr.14152) 74 : audit 0 from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi023=y;smithi146=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:42:57.738 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:42:57 smithi146 bash[23809]: cephadm 2024-03-11T20:42:56.392271+0000 mgr.y (mgr.14152) 75 : cephadm 1 Saving service mgr spec with placement smithi023=y;smithi146=x;count:2 2024-03-11T20:42:57.738 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:42:57 smithi146 bash[23809]: cephadm 2024-03-11T20:42:56.442323+0000 mgr.y (mgr.14152) 76 : cephadm 1 Deploying daemon mgr.x on smithi146 2024-03-11T20:42:57.787 DEBUG:teuthology.orchestra.run.smithi023:> stat /dev/vg_nvme/lv_4 2024-03-11T20:42:57.837 INFO:teuthology.orchestra.run.smithi023.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-11T20:42:57.837 INFO:teuthology.orchestra.run.smithi023.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T20:42:57.837 INFO:teuthology.orchestra.run.smithi023.stdout:Device: 5h/5d Inode: 919 Links: 1 2024-03-11T20:42:57.837 INFO:teuthology.orchestra.run.smithi023.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T20:42:57.837 INFO:teuthology.orchestra.run.smithi023.stdout:Access: 2024-03-11 20:33:47.369822994 +0000 2024-03-11T20:42:57.837 INFO:teuthology.orchestra.run.smithi023.stdout:Modify: 2024-03-11 20:33:47.117825242 +0000 2024-03-11T20:42:57.837 INFO:teuthology.orchestra.run.smithi023.stdout:Change: 2024-03-11 20:33:47.117825242 +0000 2024-03-11T20:42:57.837 INFO:teuthology.orchestra.run.smithi023.stdout: Birth: - 2024-03-11T20:42:57.837 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-11T20:42:57.892 INFO:teuthology.orchestra.run.smithi023.stderr:1+0 records in 2024-03-11T20:42:57.892 INFO:teuthology.orchestra.run.smithi023.stderr:1+0 records out 2024-03-11T20:42:57.892 INFO:teuthology.orchestra.run.smithi023.stderr:512 bytes copied, 0.000527376 s, 971 kB/s 2024-03-11T20:42:57.893 DEBUG:teuthology.orchestra.run.smithi023:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-11T20:42:57.943 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-11T20:42:57.943 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/scratch_devs of=/dev/stdout 2024-03-11T20:42:57.948 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-11T20:42:57.948 DEBUG:teuthology.orchestra.run.smithi146:> stat /dev/vg_nvme/lv_1 2024-03-11T20:42:57.997 INFO:teuthology.orchestra.run.smithi146.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-11T20:42:57.997 INFO:teuthology.orchestra.run.smithi146.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T20:42:57.997 INFO:teuthology.orchestra.run.smithi146.stdout:Device: 5h/5d Inode: 882 Links: 1 2024-03-11T20:42:57.997 INFO:teuthology.orchestra.run.smithi146.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T20:42:57.997 INFO:teuthology.orchestra.run.smithi146.stdout:Access: 2024-03-11 20:33:41.554904109 +0000 2024-03-11T20:42:57.997 INFO:teuthology.orchestra.run.smithi146.stdout:Modify: 2024-03-11 20:33:41.274906611 +0000 2024-03-11T20:42:57.997 INFO:teuthology.orchestra.run.smithi146.stdout:Change: 2024-03-11 20:33:41.274906611 +0000 2024-03-11T20:42:57.997 INFO:teuthology.orchestra.run.smithi146.stdout: Birth: - 2024-03-11T20:42:57.998 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-11T20:42:58.052 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records in 2024-03-11T20:42:58.053 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records out 2024-03-11T20:42:58.053 INFO:teuthology.orchestra.run.smithi146.stderr:512 bytes copied, 0.000443212 s, 1.2 MB/s 2024-03-11T20:42:58.054 DEBUG:teuthology.orchestra.run.smithi146:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-11T20:42:58.103 DEBUG:teuthology.orchestra.run.smithi146:> stat /dev/vg_nvme/lv_2 2024-03-11T20:42:58.149 INFO:teuthology.orchestra.run.smithi146.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-11T20:42:58.149 INFO:teuthology.orchestra.run.smithi146.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T20:42:58.149 INFO:teuthology.orchestra.run.smithi146.stdout:Device: 5h/5d Inode: 899 Links: 1 2024-03-11T20:42:58.149 INFO:teuthology.orchestra.run.smithi146.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T20:42:58.149 INFO:teuthology.orchestra.run.smithi146.stdout:Access: 2024-03-11 20:33:42.050899679 +0000 2024-03-11T20:42:58.149 INFO:teuthology.orchestra.run.smithi146.stdout:Modify: 2024-03-11 20:33:41.790902002 +0000 2024-03-11T20:42:58.149 INFO:teuthology.orchestra.run.smithi146.stdout:Change: 2024-03-11 20:33:41.790902002 +0000 2024-03-11T20:42:58.149 INFO:teuthology.orchestra.run.smithi146.stdout: Birth: - 2024-03-11T20:42:58.149 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-11T20:42:58.204 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records in 2024-03-11T20:42:58.205 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records out 2024-03-11T20:42:58.205 INFO:teuthology.orchestra.run.smithi146.stderr:512 bytes copied, 0.000367593 s, 1.4 MB/s 2024-03-11T20:42:58.206 DEBUG:teuthology.orchestra.run.smithi146:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-11T20:42:58.255 DEBUG:teuthology.orchestra.run.smithi146:> stat /dev/vg_nvme/lv_3 2024-03-11T20:42:58.302 INFO:teuthology.orchestra.run.smithi146.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-11T20:42:58.302 INFO:teuthology.orchestra.run.smithi146.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T20:42:58.302 INFO:teuthology.orchestra.run.smithi146.stdout:Device: 5h/5d Inode: 915 Links: 1 2024-03-11T20:42:58.303 INFO:teuthology.orchestra.run.smithi146.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T20:42:58.303 INFO:teuthology.orchestra.run.smithi146.stdout:Access: 2024-03-11 20:33:42.534895355 +0000 2024-03-11T20:42:58.303 INFO:teuthology.orchestra.run.smithi146.stdout:Modify: 2024-03-11 20:33:42.266897750 +0000 2024-03-11T20:42:58.303 INFO:teuthology.orchestra.run.smithi146.stdout:Change: 2024-03-11 20:33:42.266897750 +0000 2024-03-11T20:42:58.303 INFO:teuthology.orchestra.run.smithi146.stdout: Birth: - 2024-03-11T20:42:58.303 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-11T20:42:58.357 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records in 2024-03-11T20:42:58.357 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records out 2024-03-11T20:42:58.357 INFO:teuthology.orchestra.run.smithi146.stderr:512 bytes copied, 0.000368204 s, 1.4 MB/s 2024-03-11T20:42:58.358 DEBUG:teuthology.orchestra.run.smithi146:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-11T20:42:58.412 DEBUG:teuthology.orchestra.run.smithi146:> stat /dev/vg_nvme/lv_4 2024-03-11T20:42:58.458 INFO:teuthology.orchestra.run.smithi146.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-11T20:42:58.458 INFO:teuthology.orchestra.run.smithi146.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T20:42:58.458 INFO:teuthology.orchestra.run.smithi146.stdout:Device: 5h/5d Inode: 935 Links: 1 2024-03-11T20:42:58.458 INFO:teuthology.orchestra.run.smithi146.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T20:42:58.459 INFO:teuthology.orchestra.run.smithi146.stdout:Access: 2024-03-11 20:33:43.038890854 +0000 2024-03-11T20:42:58.459 INFO:teuthology.orchestra.run.smithi146.stdout:Modify: 2024-03-11 20:33:42.778893177 +0000 2024-03-11T20:42:58.459 INFO:teuthology.orchestra.run.smithi146.stdout:Change: 2024-03-11 20:33:42.778893177 +0000 2024-03-11T20:42:58.459 INFO:teuthology.orchestra.run.smithi146.stdout: Birth: - 2024-03-11T20:42:58.459 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-11T20:42:58.512 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records in 2024-03-11T20:42:58.512 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records out 2024-03-11T20:42:58.512 INFO:teuthology.orchestra.run.smithi146.stderr:512 bytes copied, 0.000343664 s, 1.5 MB/s 2024-03-11T20:42:58.513 DEBUG:teuthology.orchestra.run.smithi146:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-11T20:42:58.563 INFO:tasks.cephadm:Deploying osd.0 on smithi023 with /dev/vg_nvme/lv_4... 2024-03-11T20:42:58.563 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-11T20:42:58.688 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:42:58 smithi146 bash[23809]: cluster 2024-03-11T20:42:57.331842+0000 mgr.y (mgr.14152) 77 : cluster 0 pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:42:58.781 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:42:58 smithi023 bash[21372]: cluster 2024-03-11T20:42:57.331842+0000 mgr.y (mgr.14152) 77 : cluster 0 pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:42:58.782 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:42:58 smithi023 bash[29146]: cluster 2024-03-11T20:42:57.331842+0000 mgr.y (mgr.14152) 77 : cluster 0 pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:42:59.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:42:58 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:42:59.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:42:58 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:42:59.262 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Mar 11 20:42:58 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:42:59.262 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Mar 11 20:42:58 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:42:59.262 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Mar 11 20:42:58 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:42:59.262 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Mar 11 20:42:58 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:42:59.262 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Mar 11 20:42:59 smithi146 systemd[1]: Started Ceph mgr.x for 67ba00f8-dfe7-11ee-95c8-87774f69a715. 2024-03-11T20:43:00.062 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Mar 11 20:42:59 smithi146 bash[24737]: debug 2024-03-11T20:42:59.883+0000 7f645d83a700 1 -- 172.21.15.146:0/1321463761 <== mon.1 v2:172.21.15.146:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55d8205325a0 con 0x55d820638c00 2024-03-11T20:43:00.062 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Mar 11 20:43:00 smithi146 bash[24737]: debug 2024-03-11T20:43:00.027+0000 7f6467a3c200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T20:43:00.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:43:00 smithi146 bash[23809]: audit 2024-03-11T20:42:59.065232+0000 mon.a (mon.0) 227 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:43:00.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:43:00 smithi146 bash[23809]: audit 2024-03-11T20:42:59.072812+0000 mon.a (mon.0) 228 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:43:00.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:43:00 smithi146 bash[23809]: audit 2024-03-11T20:42:59.118763+0000 mon.a (mon.0) 229 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:43:00.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:43:00 smithi146 bash[23809]: audit 2024-03-11T20:42:59.125950+0000 mon.a (mon.0) 230 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:43:00.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:43:00 smithi146 bash[23809]: audit 2024-03-11T20:42:59.195823+0000 mon.a (mon.0) 231 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:43:00.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:43:00 smithi146 bash[23809]: cluster 2024-03-11T20:42:59.332200+0000 mgr.y (mgr.14152) 78 : cluster 0 pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:43:00.532 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:43:00 smithi023 bash[21372]: audit 2024-03-11T20:42:59.065232+0000 mon.a (mon.0) 227 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:43:00.532 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:43:00 smithi023 bash[21372]: audit 2024-03-11T20:42:59.072812+0000 mon.a (mon.0) 228 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:43:00.532 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:43:00 smithi023 bash[21372]: audit 2024-03-11T20:42:59.118763+0000 mon.a (mon.0) 229 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:43:00.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:43:00 smithi023 bash[21372]: audit 2024-03-11T20:42:59.125950+0000 mon.a (mon.0) 230 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:43:00.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:43:00 smithi023 bash[21372]: audit 2024-03-11T20:42:59.195823+0000 mon.a (mon.0) 231 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:43:00.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:43:00 smithi023 bash[21372]: cluster 2024-03-11T20:42:59.332200+0000 mgr.y (mgr.14152) 78 : cluster 0 pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:43:00.533 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:43:00 smithi023 bash[29146]: audit 2024-03-11T20:42:59.065232+0000 mon.a (mon.0) 227 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:43:00.533 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:43:00 smithi023 bash[29146]: audit 2024-03-11T20:42:59.072812+0000 mon.a (mon.0) 228 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:43:00.533 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:43:00 smithi023 bash[29146]: audit 2024-03-11T20:42:59.118763+0000 mon.a (mon.0) 229 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:43:00.533 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:43:00 smithi023 bash[29146]: audit 2024-03-11T20:42:59.125950+0000 mon.a (mon.0) 230 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:43:00.533 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:43:00 smithi023 bash[29146]: audit 2024-03-11T20:42:59.195823+0000 mon.a (mon.0) 231 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:43:00.533 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:43:00 smithi023 bash[29146]: cluster 2024-03-11T20:42:59.332200+0000 mgr.y (mgr.14152) 78 : cluster 0 pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:43:01.782 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:43:01 smithi023 bash[21372]: audit 2024-03-11T20:43:00.517048+0000 mon.a (mon.0) 232 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:43:01.782 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:43:01 smithi023 bash[29146]: audit 2024-03-11T20:43:00.517048+0000 mon.a (mon.0) 232 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:43:01.937 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:43:01 smithi146 bash[23809]: audit 2024-03-11T20:43:00.517048+0000 mon.a (mon.0) 232 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:43:01.937 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Mar 11 20:43:01 smithi146 bash[24737]: debug 2024-03-11T20:43:01.595+0000 7f6467a3c200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T20:43:01.937 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Mar 11 20:43:01 smithi146 bash[24737]: debug 2024-03-11T20:43:01.695+0000 7f6467a3c200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T20:43:01.937 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Mar 11 20:43:01 smithi146 bash[24737]: debug 2024-03-11T20:43:01.939+0000 7f6467a3c200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T20:43:02.843 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:43:02 smithi146 bash[23809]: cluster 2024-03-11T20:43:01.332516+0000 mgr.y (mgr.14152) 79 : cluster 0 pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:43:02.843 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Mar 11 20:43:02 smithi146 bash[24737]: debug 2024-03-11T20:43:02.735+0000 7f6467a3c200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T20:43:03.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:43:02 smithi023 bash[21372]: cluster 2024-03-11T20:43:01.332516+0000 mgr.y (mgr.14152) 79 : cluster 0 pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:43:03.032 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:43:02 smithi023 bash[29146]: cluster 2024-03-11T20:43:01.332516+0000 mgr.y (mgr.14152) 79 : cluster 0 pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:43:03.261 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Mar 11 20:43:02 smithi146 bash[24737]: debug 2024-03-11T20:43:02.847+0000 7f6467a3c200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T20:43:03.261 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Mar 11 20:43:02 smithi146 bash[24737]: debug 2024-03-11T20:43:02.947+0000 7f6467a3c200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T20:43:03.287 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:43:03.761 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Mar 11 20:43:03 smithi146 bash[24737]: debug 2024-03-11T20:43:03.299+0000 7f6467a3c200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-11T20:43:03.761 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Mar 11 20:43:03 smithi146 bash[24737]: debug 2024-03-11T20:43:03.411+0000 7f6467a3c200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T20:43:03.761 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Mar 11 20:43:03 smithi146 bash[24737]: debug 2024-03-11T20:43:03.511+0000 7f6467a3c200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T20:43:04.261 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Mar 11 20:43:03 smithi146 bash[24737]: debug 2024-03-11T20:43:03.867+0000 7f6467a3c200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T20:43:04.261 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Mar 11 20:43:04 smithi146 bash[24737]: debug 2024-03-11T20:43:04.107+0000 7f6467a3c200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T20:43:04.753 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Mar 11 20:43:04 smithi146 bash[24737]: debug 2024-03-11T20:43:04.371+0000 7f6467a3c200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T20:43:04.754 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Mar 11 20:43:04 smithi146 bash[24737]: debug 2024-03-11T20:43:04.487+0000 7f6467a3c200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T20:43:04.754 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:43:04 smithi146 bash[23809]: cluster 2024-03-11T20:43:03.332842+0000 mgr.y (mgr.14152) 80 : cluster 0 pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:43:04.754 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:43:04 smithi146 bash[23809]: audit 2024-03-11T20:43:04.393244+0000 mon.a (mon.0) 233 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:43:04.754 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:43:04 smithi146 bash[23809]: audit 2024-03-11T20:43:04.410124+0000 mon.a (mon.0) 234 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:43:04.754 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:43:04 smithi146 bash[23809]: audit 2024-03-11T20:43:04.411492+0000 mon.a (mon.0) 235 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:43:04.754 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:43:04 smithi146 bash[23809]: audit 2024-03-11T20:43:04.412815+0000 mon.a (mon.0) 236 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:43:04.754 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:43:04 smithi146 bash[23809]: audit 2024-03-11T20:43:04.418573+0000 mon.a (mon.0) 237 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:43:04.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:43:04 smithi146 bash[23809]: audit 2024-03-11T20:43:04.485837+0000 mon.a (mon.0) 238 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T20:43:04.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:43:04 smithi146 bash[23809]: audit 2024-03-11T20:43:04.486865+0000 mon.a (mon.0) 239 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T20:43:04.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:43:04 smithi146 bash[23809]: audit 2024-03-11T20:43:04.487641+0000 mon.a (mon.0) 240 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:43:04.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:43:04 smithi023 bash[21372]: cluster 2024-03-11T20:43:03.332842+0000 mgr.y (mgr.14152) 80 : cluster 0 pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:43:04.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:43:04 smithi023 bash[21372]: audit 2024-03-11T20:43:04.393244+0000 mon.a (mon.0) 233 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:43:04.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:43:04 smithi023 bash[21372]: audit 2024-03-11T20:43:04.410124+0000 mon.a (mon.0) 234 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:43:04.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:43:04 smithi023 bash[21372]: audit 2024-03-11T20:43:04.411492+0000 mon.a (mon.0) 235 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:43:04.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:43:04 smithi023 bash[21372]: audit 2024-03-11T20:43:04.412815+0000 mon.a (mon.0) 236 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:43:04.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:43:04 smithi023 bash[21372]: audit 2024-03-11T20:43:04.418573+0000 mon.a (mon.0) 237 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:43:04.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:43:04 smithi023 bash[21372]: audit 2024-03-11T20:43:04.485837+0000 mon.a (mon.0) 238 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T20:43:04.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:43:04 smithi023 bash[21372]: audit 2024-03-11T20:43:04.486865+0000 mon.a (mon.0) 239 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T20:43:04.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:43:04 smithi023 bash[21372]: audit 2024-03-11T20:43:04.487641+0000 mon.a (mon.0) 240 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:43:04.784 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:43:04 smithi023 bash[29146]: cluster 2024-03-11T20:43:03.332842+0000 mgr.y (mgr.14152) 80 : cluster 0 pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:43:04.784 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:43:04 smithi023 bash[29146]: audit 2024-03-11T20:43:04.393244+0000 mon.a (mon.0) 233 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:43:04.784 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:43:04 smithi023 bash[29146]: audit 2024-03-11T20:43:04.410124+0000 mon.a (mon.0) 234 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:43:04.784 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:43:04 smithi023 bash[29146]: audit 2024-03-11T20:43:04.411492+0000 mon.a (mon.0) 235 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:43:04.784 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:43:04 smithi023 bash[29146]: audit 2024-03-11T20:43:04.412815+0000 mon.a (mon.0) 236 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:43:04.784 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:43:04 smithi023 bash[29146]: audit 2024-03-11T20:43:04.418573+0000 mon.a (mon.0) 237 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:43:04.784 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:43:04 smithi023 bash[29146]: audit 2024-03-11T20:43:04.485837+0000 mon.a (mon.0) 238 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T20:43:04.784 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:43:04 smithi023 bash[29146]: audit 2024-03-11T20:43:04.486865+0000 mon.a (mon.0) 239 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T20:43:04.784 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:43:04 smithi023 bash[29146]: audit 2024-03-11T20:43:04.487641+0000 mon.a (mon.0) 240 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:43:05.011 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Mar 11 20:43:04 smithi146 bash[24737]: debug 2024-03-11T20:43:04.755+0000 7f6467a3c200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T20:43:05.011 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Mar 11 20:43:05 smithi146 bash[24737]: debug 2024-03-11T20:43:04.999+0000 7f6467a3c200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T20:43:05.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:43:05 smithi146 bash[23809]: cephadm 2024-03-11T20:43:04.485423+0000 mgr.y (mgr.14152) 81 : cephadm 1 Reconfiguring mgr.y (unknown last config time)... 2024-03-11T20:43:05.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:43:05 smithi146 bash[23809]: cephadm 2024-03-11T20:43:04.488587+0000 mgr.y (mgr.14152) 82 : cephadm 1 Reconfiguring daemon mgr.y on smithi023 2024-03-11T20:43:05.763 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Mar 11 20:43:05 smithi146 bash[24737]: debug 2024-03-11T20:43:05.407+0000 7f6467a3c200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T20:43:05.763 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Mar 11 20:43:05 smithi146 bash[24737]: debug 2024-03-11T20:43:05.523+0000 7f6467a3c200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T20:43:05.902 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:43:05 smithi023 bash[29146]: cephadm 2024-03-11T20:43:04.485423+0000 mgr.y (mgr.14152) 81 : cephadm 1 Reconfiguring mgr.y (unknown last config time)... 2024-03-11T20:43:05.902 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:43:05 smithi023 bash[29146]: cephadm 2024-03-11T20:43:04.488587+0000 mgr.y (mgr.14152) 82 : cephadm 1 Reconfiguring daemon mgr.y on smithi023 2024-03-11T20:43:05.902 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:43:05 smithi023 bash[21372]: cephadm 2024-03-11T20:43:04.485423+0000 mgr.y (mgr.14152) 81 : cephadm 1 Reconfiguring mgr.y (unknown last config time)... 2024-03-11T20:43:05.902 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:43:05 smithi023 bash[21372]: cephadm 2024-03-11T20:43:04.488587+0000 mgr.y (mgr.14152) 82 : cephadm 1 Reconfiguring daemon mgr.y on smithi023 2024-03-11T20:43:06.261 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Mar 11 20:43:05 smithi146 bash[24737]: debug 2024-03-11T20:43:05.767+0000 7f6467a3c200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T20:43:06.893 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Mar 11 20:43:06 smithi146 bash[24737]: debug 2024-03-11T20:43:06.779+0000 7f6467a3c200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T20:43:06.893 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:43:06 smithi146 bash[23809]: cluster 2024-03-11T20:43:05.333235+0000 mgr.y (mgr.14152) 83 : cluster 0 pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:43:06.934 INFO:teuthology.orchestra.run.smithi023.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-11T20:43:06.935 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-11T20:43:06.935 INFO:teuthology.orchestra.run.smithi023.stderr: stderr: 10+0 records in 2024-03-11T20:43:06.935 INFO:teuthology.orchestra.run.smithi023.stderr:10+0 records out 2024-03-11T20:43:06.935 INFO:teuthology.orchestra.run.smithi023.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0127423 s, 823 MB/s 2024-03-11T20:43:06.935 INFO:teuthology.orchestra.run.smithi023.stderr:--> Zapping successful for: 2024-03-11T20:43:06.973 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:43:06 smithi023 bash[21372]: cluster 2024-03-11T20:43:05.333235+0000 mgr.y (mgr.14152) 83 : cluster 0 pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:43:06.973 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:43:06 smithi023 bash[29146]: cluster 2024-03-11T20:43:05.333235+0000 mgr.y (mgr.14152) 83 : cluster 0 pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:43:07.194 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Mar 11 20:43:06 smithi146 bash[24737]: debug 2024-03-11T20:43:06.895+0000 7f6467a3c200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T20:43:07.511 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Mar 11 20:43:07 smithi146 bash[24737]: debug 2024-03-11T20:43:07.195+0000 7f6467a3c200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T20:43:07.511 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Mar 11 20:43:07 smithi146 bash[24737]: debug 2024-03-11T20:43:07.311+0000 7f6467a3c200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T20:43:07.600 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi023:vg_nvme/lv_4 2024-03-11T20:43:08.012 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Mar 11 20:43:07 smithi146 bash[24737]: debug 2024-03-11T20:43:07.763+0000 7f6467a3c200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T20:43:08.012 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Mar 11 20:43:07 smithi146 bash[24737]: debug 2024-03-11T20:43:07.883+0000 7f6467a3c200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T20:43:08.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:43:07 smithi146 bash[23809]: audit 2024-03-11T20:43:06.767285+0000 mon.a (mon.0) 241 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:43:08.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:43:07 smithi146 bash[23809]: audit 2024-03-11T20:43:06.773009+0000 mon.a (mon.0) 242 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:43:08.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:43:07 smithi146 bash[23809]: audit 2024-03-11T20:43:06.808220+0000 mon.a (mon.0) 243 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:43:08.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:43:07 smithi146 bash[23809]: audit 2024-03-11T20:43:07.487556+0000 mon.a (mon.0) 244 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:43:08.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:43:07 smithi146 bash[23809]: audit 2024-03-11T20:43:07.489530+0000 mon.a (mon.0) 245 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:43:08.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:43:07 smithi146 bash[23809]: audit 2024-03-11T20:43:07.498978+0000 mon.a (mon.0) 246 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:43:08.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:43:07 smithi023 bash[21372]: audit 2024-03-11T20:43:06.767285+0000 mon.a (mon.0) 241 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:43:08.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:43:07 smithi023 bash[21372]: audit 2024-03-11T20:43:06.773009+0000 mon.a (mon.0) 242 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:43:08.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:43:07 smithi023 bash[21372]: audit 2024-03-11T20:43:06.808220+0000 mon.a (mon.0) 243 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:43:08.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:43:07 smithi023 bash[21372]: audit 2024-03-11T20:43:07.487556+0000 mon.a (mon.0) 244 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:43:08.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:43:07 smithi023 bash[21372]: audit 2024-03-11T20:43:07.489530+0000 mon.a (mon.0) 245 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:43:08.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:43:07 smithi023 bash[21372]: audit 2024-03-11T20:43:07.498978+0000 mon.a (mon.0) 246 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:43:08.032 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:43:07 smithi023 bash[29146]: audit 2024-03-11T20:43:06.767285+0000 mon.a (mon.0) 241 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:43:08.032 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:43:07 smithi023 bash[29146]: audit 2024-03-11T20:43:06.773009+0000 mon.a (mon.0) 242 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:43:08.032 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:43:07 smithi023 bash[29146]: audit 2024-03-11T20:43:06.808220+0000 mon.a (mon.0) 243 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:43:08.032 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:43:07 smithi023 bash[29146]: audit 2024-03-11T20:43:07.487556+0000 mon.a (mon.0) 244 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:43:08.033 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:43:07 smithi023 bash[29146]: audit 2024-03-11T20:43:07.489530+0000 mon.a (mon.0) 245 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:43:08.033 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:43:07 smithi023 bash[29146]: audit 2024-03-11T20:43:07.498978+0000 mon.a (mon.0) 246 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:43:09.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:43:08 smithi146 bash[23809]: cluster 2024-03-11T20:43:07.333632+0000 mgr.y (mgr.14152) 84 : cluster 0 pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:43:09.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:43:08 smithi146 bash[23809]: cluster 2024-03-11T20:43:07.907216+0000 mon.a (mon.0) 247 : cluster 0 Standby manager daemon x started 2024-03-11T20:43:09.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:43:08 smithi146 bash[23809]: audit 2024-03-11T20:43:07.909610+0000 mon.b (mon.1) 3 : audit 0 from='mgr.? 172.21.15.146:0/2933105015' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-11T20:43:09.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:43:08 smithi146 bash[23809]: audit 2024-03-11T20:43:07.910552+0000 mon.b (mon.1) 4 : audit 0 from='mgr.? 172.21.15.146:0/2933105015' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T20:43:09.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:43:08 smithi146 bash[23809]: audit 2024-03-11T20:43:07.911900+0000 mon.b (mon.1) 5 : audit 0 from='mgr.? 172.21.15.146:0/2933105015' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-11T20:43:09.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:43:08 smithi146 bash[23809]: audit 2024-03-11T20:43:07.912354+0000 mon.b (mon.1) 6 : audit 0 from='mgr.? 172.21.15.146:0/2933105015' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T20:43:09.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:43:08 smithi023 bash[21372]: cluster 2024-03-11T20:43:07.333632+0000 mgr.y (mgr.14152) 84 : cluster 0 pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:43:09.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:43:08 smithi023 bash[21372]: cluster 2024-03-11T20:43:07.907216+0000 mon.a (mon.0) 247 : cluster 0 Standby manager daemon x started 2024-03-11T20:43:09.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:43:08 smithi023 bash[21372]: audit 2024-03-11T20:43:07.909610+0000 mon.b (mon.1) 3 : audit 0 from='mgr.? 172.21.15.146:0/2933105015' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-11T20:43:09.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:43:08 smithi023 bash[21372]: audit 2024-03-11T20:43:07.910552+0000 mon.b (mon.1) 4 : audit 0 from='mgr.? 172.21.15.146:0/2933105015' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T20:43:09.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:43:08 smithi023 bash[21372]: audit 2024-03-11T20:43:07.911900+0000 mon.b (mon.1) 5 : audit 0 from='mgr.? 172.21.15.146:0/2933105015' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-11T20:43:09.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:43:08 smithi023 bash[21372]: audit 2024-03-11T20:43:07.912354+0000 mon.b (mon.1) 6 : audit 0 from='mgr.? 172.21.15.146:0/2933105015' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T20:43:09.282 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:43:08 smithi023 bash[29146]: cluster 2024-03-11T20:43:07.333632+0000 mgr.y (mgr.14152) 84 : cluster 0 pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:43:09.282 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:43:08 smithi023 bash[29146]: cluster 2024-03-11T20:43:07.907216+0000 mon.a (mon.0) 247 : cluster 0 Standby manager daemon x started 2024-03-11T20:43:09.282 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:43:08 smithi023 bash[29146]: audit 2024-03-11T20:43:07.909610+0000 mon.b (mon.1) 3 : audit 0 from='mgr.? 172.21.15.146:0/2933105015' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-11T20:43:09.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:43:08 smithi023 bash[29146]: audit 2024-03-11T20:43:07.910552+0000 mon.b (mon.1) 4 : audit 0 from='mgr.? 172.21.15.146:0/2933105015' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T20:43:09.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:43:08 smithi023 bash[29146]: audit 2024-03-11T20:43:07.911900+0000 mon.b (mon.1) 5 : audit 0 from='mgr.? 172.21.15.146:0/2933105015' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-11T20:43:09.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:43:08 smithi023 bash[29146]: audit 2024-03-11T20:43:07.912354+0000 mon.b (mon.1) 6 : audit 0 from='mgr.? 172.21.15.146:0/2933105015' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T20:43:10.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:43:09 smithi146 bash[23809]: cluster 2024-03-11T20:43:08.822960+0000 mon.a (mon.0) 248 : cluster 0 mgrmap e14: y(active, since 107s), standbys: x 2024-03-11T20:43:10.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:43:09 smithi146 bash[23809]: audit 2024-03-11T20:43:08.823160+0000 mon.a (mon.0) 249 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-11T20:43:10.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:43:09 smithi023 bash[21372]: cluster 2024-03-11T20:43:08.822960+0000 mon.a (mon.0) 248 : cluster 0 mgrmap e14: y(active, since 107s), standbys: x 2024-03-11T20:43:10.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:43:09 smithi023 bash[21372]: audit 2024-03-11T20:43:08.823160+0000 mon.a (mon.0) 249 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-11T20:43:10.282 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:43:09 smithi023 bash[29146]: cluster 2024-03-11T20:43:08.822960+0000 mon.a (mon.0) 248 : cluster 0 mgrmap e14: y(active, since 107s), standbys: x 2024-03-11T20:43:10.282 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:43:09 smithi023 bash[29146]: audit 2024-03-11T20:43:08.823160+0000 mon.a (mon.0) 249 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-11T20:43:11.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:43:10 smithi146 bash[23809]: cluster 2024-03-11T20:43:09.334068+0000 mgr.y (mgr.14152) 85 : cluster 0 pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:43:11.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:43:10 smithi023 bash[21372]: cluster 2024-03-11T20:43:09.334068+0000 mgr.y (mgr.14152) 85 : cluster 0 pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:43:11.282 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:43:10 smithi023 bash[29146]: cluster 2024-03-11T20:43:09.334068+0000 mgr.y (mgr.14152) 85 : cluster 0 pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:43:12.331 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:43:13.086 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:43:12 smithi023 bash[21372]: cluster 2024-03-11T20:43:11.334534+0000 mgr.y (mgr.14152) 86 : cluster 0 pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:43:13.086 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:43:12 smithi023 bash[29146]: cluster 2024-03-11T20:43:11.334534+0000 mgr.y (mgr.14152) 86 : cluster 0 pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:43:13.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:43:12 smithi146 bash[23809]: cluster 2024-03-11T20:43:11.334534+0000 mgr.y (mgr.14152) 86 : cluster 0 pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:43:15.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:43:14 smithi146 bash[23809]: cluster 2024-03-11T20:43:13.335071+0000 mgr.y (mgr.14152) 87 : cluster 0 pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:43:15.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:43:14 smithi023 bash[21372]: cluster 2024-03-11T20:43:13.335071+0000 mgr.y (mgr.14152) 87 : cluster 0 pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:43:15.282 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:43:14 smithi023 bash[29146]: cluster 2024-03-11T20:43:13.335071+0000 mgr.y (mgr.14152) 87 : cluster 0 pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:43:16.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:43:15 smithi146 bash[23809]: audit 2024-03-11T20:43:14.947636+0000 mgr.y (mgr.14152) 88 : audit 0 from='client.24110 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:43:16.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:43:15 smithi146 bash[23809]: audit 2024-03-11T20:43:14.955014+0000 mon.a (mon.0) 250 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T20:43:16.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:43:15 smithi146 bash[23809]: audit 2024-03-11T20:43:14.959654+0000 mon.a (mon.0) 251 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T20:43:16.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:43:15 smithi146 bash[23809]: audit 2024-03-11T20:43:14.960669+0000 mon.a (mon.0) 252 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:43:16.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:43:15 smithi023 bash[21372]: audit 2024-03-11T20:43:14.947636+0000 mgr.y (mgr.14152) 88 : audit 0 from='client.24110 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:43:16.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:43:15 smithi023 bash[21372]: audit 2024-03-11T20:43:14.955014+0000 mon.a (mon.0) 250 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T20:43:16.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:43:15 smithi023 bash[21372]: audit 2024-03-11T20:43:14.959654+0000 mon.a (mon.0) 251 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T20:43:16.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:43:15 smithi023 bash[21372]: audit 2024-03-11T20:43:14.960669+0000 mon.a (mon.0) 252 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:43:16.282 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:43:15 smithi023 bash[29146]: audit 2024-03-11T20:43:14.947636+0000 mgr.y (mgr.14152) 88 : audit 0 from='client.24110 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:43:16.282 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:43:15 smithi023 bash[29146]: audit 2024-03-11T20:43:14.955014+0000 mon.a (mon.0) 250 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T20:43:16.282 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:43:15 smithi023 bash[29146]: audit 2024-03-11T20:43:14.959654+0000 mon.a (mon.0) 251 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T20:43:16.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:43:15 smithi023 bash[29146]: audit 2024-03-11T20:43:14.960669+0000 mon.a (mon.0) 252 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:43:17.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:43:16 smithi146 bash[23809]: cluster 2024-03-11T20:43:15.335545+0000 mgr.y (mgr.14152) 89 : cluster 0 pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:43:17.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:43:16 smithi023 bash[21372]: cluster 2024-03-11T20:43:15.335545+0000 mgr.y (mgr.14152) 89 : cluster 0 pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:43:17.282 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:43:16 smithi023 bash[29146]: cluster 2024-03-11T20:43:15.335545+0000 mgr.y (mgr.14152) 89 : cluster 0 pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:43:19.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:43:18 smithi146 bash[23809]: cluster 2024-03-11T20:43:17.335984+0000 mgr.y (mgr.14152) 90 : cluster 0 pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:43:19.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:43:18 smithi023 bash[21372]: cluster 2024-03-11T20:43:17.335984+0000 mgr.y (mgr.14152) 90 : cluster 0 pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:43:19.282 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:43:18 smithi023 bash[29146]: cluster 2024-03-11T20:43:17.335984+0000 mgr.y (mgr.14152) 90 : cluster 0 pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:43:20.983 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:43:20 smithi023 bash[29146]: cluster 2024-03-11T20:43:19.336409+0000 mgr.y (mgr.14152) 91 : cluster 0 pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:43:20.984 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:43:20 smithi023 bash[21372]: cluster 2024-03-11T20:43:19.336409+0000 mgr.y (mgr.14152) 91 : cluster 0 pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:43:21.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:43:20 smithi146 bash[23809]: cluster 2024-03-11T20:43:19.336409+0000 mgr.y (mgr.14152) 91 : cluster 0 pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:43:23.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:43:22 smithi146 bash[23809]: cluster 2024-03-11T20:43:21.336731+0000 mgr.y (mgr.14152) 92 : cluster 0 pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:43:23.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:43:22 smithi023 bash[21372]: cluster 2024-03-11T20:43:21.336731+0000 mgr.y (mgr.14152) 92 : cluster 0 pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:43:23.282 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:43:22 smithi023 bash[29146]: cluster 2024-03-11T20:43:21.336731+0000 mgr.y (mgr.14152) 92 : cluster 0 pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:43:24.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:43:23 smithi146 bash[23809]: audit 2024-03-11T20:43:23.020281+0000 mon.c (mon.2) 2 : audit 1 from='client.? 172.21.15.23:0/3079261945' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3df72b4a-8fe4-4265-b4c6-af022e1e23e1"}]: dispatch 2024-03-11T20:43:24.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:43:23 smithi146 bash[23809]: audit 2024-03-11T20:43:23.020976+0000 mon.a (mon.0) 253 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3df72b4a-8fe4-4265-b4c6-af022e1e23e1"}]: dispatch 2024-03-11T20:43:24.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:43:23 smithi146 bash[23809]: audit 2024-03-11T20:43:23.024123+0000 mon.a (mon.0) 254 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3df72b4a-8fe4-4265-b4c6-af022e1e23e1"}]': finished 2024-03-11T20:43:24.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:43:23 smithi146 bash[23809]: cluster 2024-03-11T20:43:23.037646+0000 mon.a (mon.0) 255 : cluster 0 osdmap e5: 1 total, 0 up, 1 in 2024-03-11T20:43:24.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:43:23 smithi146 bash[23809]: audit 2024-03-11T20:43:23.037985+0000 mon.a (mon.0) 256 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T20:43:24.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:43:23 smithi146 bash[23809]: audit 2024-03-11T20:43:23.743304+0000 mon.a (mon.0) 257 : audit 0 from='client.? 172.21.15.23:0/1211711742' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T20:43:24.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:43:23 smithi023 bash[21372]: audit 2024-03-11T20:43:23.020281+0000 mon.c (mon.2) 2 : audit 1 from='client.? 172.21.15.23:0/3079261945' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3df72b4a-8fe4-4265-b4c6-af022e1e23e1"}]: dispatch 2024-03-11T20:43:24.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:43:23 smithi023 bash[21372]: audit 2024-03-11T20:43:23.020976+0000 mon.a (mon.0) 253 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3df72b4a-8fe4-4265-b4c6-af022e1e23e1"}]: dispatch 2024-03-11T20:43:24.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:43:23 smithi023 bash[21372]: audit 2024-03-11T20:43:23.024123+0000 mon.a (mon.0) 254 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3df72b4a-8fe4-4265-b4c6-af022e1e23e1"}]': finished 2024-03-11T20:43:24.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:43:23 smithi023 bash[21372]: cluster 2024-03-11T20:43:23.037646+0000 mon.a (mon.0) 255 : cluster 0 osdmap e5: 1 total, 0 up, 1 in 2024-03-11T20:43:24.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:43:23 smithi023 bash[21372]: audit 2024-03-11T20:43:23.037985+0000 mon.a (mon.0) 256 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T20:43:24.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:43:23 smithi023 bash[21372]: audit 2024-03-11T20:43:23.743304+0000 mon.a (mon.0) 257 : audit 0 from='client.? 172.21.15.23:0/1211711742' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T20:43:24.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:43:23 smithi023 bash[29146]: audit 2024-03-11T20:43:23.020281+0000 mon.c (mon.2) 2 : audit 1 from='client.? 172.21.15.23:0/3079261945' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3df72b4a-8fe4-4265-b4c6-af022e1e23e1"}]: dispatch 2024-03-11T20:43:24.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:43:23 smithi023 bash[29146]: audit 2024-03-11T20:43:23.020976+0000 mon.a (mon.0) 253 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3df72b4a-8fe4-4265-b4c6-af022e1e23e1"}]: dispatch 2024-03-11T20:43:24.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:43:23 smithi023 bash[29146]: audit 2024-03-11T20:43:23.024123+0000 mon.a (mon.0) 254 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3df72b4a-8fe4-4265-b4c6-af022e1e23e1"}]': finished 2024-03-11T20:43:24.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:43:23 smithi023 bash[29146]: cluster 2024-03-11T20:43:23.037646+0000 mon.a (mon.0) 255 : cluster 0 osdmap e5: 1 total, 0 up, 1 in 2024-03-11T20:43:24.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:43:23 smithi023 bash[29146]: audit 2024-03-11T20:43:23.037985+0000 mon.a (mon.0) 256 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T20:43:24.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:43:23 smithi023 bash[29146]: audit 2024-03-11T20:43:23.743304+0000 mon.a (mon.0) 257 : audit 0 from='client.? 172.21.15.23:0/1211711742' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T20:43:25.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:43:24 smithi146 bash[23809]: cluster 2024-03-11T20:43:23.337033+0000 mgr.y (mgr.14152) 93 : cluster 0 pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:43:25.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:43:24 smithi023 bash[21372]: cluster 2024-03-11T20:43:23.337033+0000 mgr.y (mgr.14152) 93 : cluster 0 pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:43:25.282 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:43:24 smithi023 bash[29146]: cluster 2024-03-11T20:43:23.337033+0000 mgr.y (mgr.14152) 93 : cluster 0 pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:43:27.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:43:26 smithi146 bash[23809]: cluster 2024-03-11T20:43:25.337445+0000 mgr.y (mgr.14152) 94 : cluster 0 pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:43:27.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:43:26 smithi023 bash[21372]: cluster 2024-03-11T20:43:25.337445+0000 mgr.y (mgr.14152) 94 : cluster 0 pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:43:27.282 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:43:26 smithi023 bash[29146]: cluster 2024-03-11T20:43:25.337445+0000 mgr.y (mgr.14152) 94 : cluster 0 pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:43:29.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:43:28 smithi146 bash[23809]: cluster 2024-03-11T20:43:27.337839+0000 mgr.y (mgr.14152) 95 : cluster 0 pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:43:29.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:43:28 smithi023 bash[21372]: cluster 2024-03-11T20:43:27.337839+0000 mgr.y (mgr.14152) 95 : cluster 0 pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:43:29.282 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:43:28 smithi023 bash[29146]: cluster 2024-03-11T20:43:27.337839+0000 mgr.y (mgr.14152) 95 : cluster 0 pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:43:31.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:43:30 smithi146 bash[23809]: cluster 2024-03-11T20:43:29.338283+0000 mgr.y (mgr.14152) 96 : cluster 0 pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:43:31.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:43:30 smithi023 bash[21372]: cluster 2024-03-11T20:43:29.338283+0000 mgr.y (mgr.14152) 96 : cluster 0 pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:43:31.282 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:43:30 smithi023 bash[29146]: cluster 2024-03-11T20:43:29.338283+0000 mgr.y (mgr.14152) 96 : cluster 0 pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:43:33.252 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:43:32 smithi023 bash[21372]: cluster 2024-03-11T20:43:31.338707+0000 mgr.y (mgr.14152) 97 : cluster 0 pgmap v60: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:43:33.252 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:43:32 smithi023 bash[29146]: cluster 2024-03-11T20:43:31.338707+0000 mgr.y (mgr.14152) 97 : cluster 0 pgmap v60: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:43:33.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:43:32 smithi146 bash[23809]: cluster 2024-03-11T20:43:31.338707+0000 mgr.y (mgr.14152) 97 : cluster 0 pgmap v60: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:43:35.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:43:34 smithi146 bash[23809]: cluster 2024-03-11T20:43:33.339152+0000 mgr.y (mgr.14152) 98 : cluster 0 pgmap v61: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:43:35.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:43:34 smithi023 bash[21372]: cluster 2024-03-11T20:43:33.339152+0000 mgr.y (mgr.14152) 98 : cluster 0 pgmap v61: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:43:35.282 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:43:34 smithi023 bash[29146]: cluster 2024-03-11T20:43:33.339152+0000 mgr.y (mgr.14152) 98 : cluster 0 pgmap v61: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:43:36.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:43:35 smithi146 bash[23809]: audit 2024-03-11T20:43:35.173059+0000 mon.a (mon.0) 258 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-11T20:43:36.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:43:35 smithi146 bash[23809]: audit 2024-03-11T20:43:35.174851+0000 mon.a (mon.0) 259 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:43:36.265 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:43:35 smithi023 bash[21372]: audit 2024-03-11T20:43:35.173059+0000 mon.a (mon.0) 258 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-11T20:43:36.265 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:43:35 smithi023 bash[21372]: audit 2024-03-11T20:43:35.174851+0000 mon.a (mon.0) 259 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:43:36.266 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:43:35 smithi023 bash[29146]: audit 2024-03-11T20:43:35.173059+0000 mon.a (mon.0) 258 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-11T20:43:36.266 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:43:35 smithi023 bash[29146]: audit 2024-03-11T20:43:35.174851+0000 mon.a (mon.0) 259 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:43:37.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:43:36 smithi146 bash[23809]: cephadm 2024-03-11T20:43:35.176556+0000 mgr.y (mgr.14152) 99 : cephadm 1 Deploying daemon osd.0 on smithi023 2024-03-11T20:43:37.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:43:36 smithi146 bash[23809]: cluster 2024-03-11T20:43:35.339487+0000 mgr.y (mgr.14152) 100 : cluster 0 pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:43:37.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:43:36 smithi023 bash[21372]: cephadm 2024-03-11T20:43:35.176556+0000 mgr.y (mgr.14152) 99 : cephadm 1 Deploying daemon osd.0 on smithi023 2024-03-11T20:43:37.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:43:36 smithi023 bash[21372]: cluster 2024-03-11T20:43:35.339487+0000 mgr.y (mgr.14152) 100 : cluster 0 pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:43:37.282 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:43:36 smithi023 bash[29146]: cephadm 2024-03-11T20:43:35.176556+0000 mgr.y (mgr.14152) 99 : cephadm 1 Deploying daemon osd.0 on smithi023 2024-03-11T20:43:37.282 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:43:36 smithi023 bash[29146]: cluster 2024-03-11T20:43:35.339487+0000 mgr.y (mgr.14152) 100 : cluster 0 pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:43:39.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:43:38 smithi146 bash[23809]: cluster 2024-03-11T20:43:37.339879+0000 mgr.y (mgr.14152) 101 : cluster 0 pgmap v63: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:43:39.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:43:38 smithi023 bash[21372]: cluster 2024-03-11T20:43:37.339879+0000 mgr.y (mgr.14152) 101 : cluster 0 pgmap v63: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:43:39.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:43:39 smithi023 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:43:39.274 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:43:39 smithi023 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:43:39.275 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:43:38 smithi023 bash[29146]: cluster 2024-03-11T20:43:37.339879+0000 mgr.y (mgr.14152) 101 : cluster 0 pgmap v63: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:43:39.275 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:43:39 smithi023 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:43:39.532 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:43:39 smithi023 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:43:39.532 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:43:39 smithi023 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:43:39.532 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:43:39 smithi023 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:43:40.782 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:43:40 smithi023 bash[21372]: cluster 2024-03-11T20:43:39.340187+0000 mgr.y (mgr.14152) 102 : cluster 0 pgmap v64: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:43:40.782 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:43:40 smithi023 bash[21372]: audit 2024-03-11T20:43:39.562462+0000 mon.a (mon.0) 260 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:43:40.782 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:43:40 smithi023 bash[21372]: audit 2024-03-11T20:43:39.565071+0000 mon.a (mon.0) 261 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:43:40.782 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:43:40 smithi023 bash[21372]: audit 2024-03-11T20:43:39.567371+0000 mon.a (mon.0) 262 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:43:40.782 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:43:40 smithi023 bash[29146]: cluster 2024-03-11T20:43:39.340187+0000 mgr.y (mgr.14152) 102 : cluster 0 pgmap v64: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:43:40.782 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:43:40 smithi023 bash[29146]: audit 2024-03-11T20:43:39.562462+0000 mon.a (mon.0) 260 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:43:40.782 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:43:40 smithi023 bash[29146]: audit 2024-03-11T20:43:39.565071+0000 mon.a (mon.0) 261 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:43:40.782 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:43:40 smithi023 bash[29146]: audit 2024-03-11T20:43:39.567371+0000 mon.a (mon.0) 262 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:43:41.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:43:40 smithi146 bash[23809]: cluster 2024-03-11T20:43:39.340187+0000 mgr.y (mgr.14152) 102 : cluster 0 pgmap v64: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:43:41.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:43:40 smithi146 bash[23809]: audit 2024-03-11T20:43:39.562462+0000 mon.a (mon.0) 260 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:43:41.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:43:40 smithi146 bash[23809]: audit 2024-03-11T20:43:39.565071+0000 mon.a (mon.0) 261 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:43:41.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:43:40 smithi146 bash[23809]: audit 2024-03-11T20:43:39.567371+0000 mon.a (mon.0) 262 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:43:42.835 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:43:42 smithi023 bash[21372]: cluster 2024-03-11T20:43:41.340568+0000 mgr.y (mgr.14152) 103 : cluster 0 pgmap v65: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:43:42.835 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:43:42 smithi023 bash[29146]: cluster 2024-03-11T20:43:41.340568+0000 mgr.y (mgr.14152) 103 : cluster 0 pgmap v65: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:43:43.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:43:42 smithi146 bash[23809]: cluster 2024-03-11T20:43:41.340568+0000 mgr.y (mgr.14152) 103 : cluster 0 pgmap v65: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:43:45.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:43:44 smithi146 bash[23809]: cluster 2024-03-11T20:43:43.340979+0000 mgr.y (mgr.14152) 104 : cluster 0 pgmap v66: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:43:45.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:43:44 smithi023 bash[21372]: cluster 2024-03-11T20:43:43.340979+0000 mgr.y (mgr.14152) 104 : cluster 0 pgmap v66: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:43:45.032 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:43:44 smithi023 bash[29146]: cluster 2024-03-11T20:43:43.340979+0000 mgr.y (mgr.14152) 104 : cluster 0 pgmap v66: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:43:46.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:43:45 smithi023 bash[21372]: audit 2024-03-11T20:43:44.853082+0000 mon.a (mon.0) 263 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:43:46.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:43:45 smithi023 bash[21372]: audit 2024-03-11T20:43:44.863390+0000 mon.a (mon.0) 264 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:43:46.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:43:45 smithi023 bash[21372]: audit 2024-03-11T20:43:45.490518+0000 mon.a (mon.0) 265 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:43:46.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:43:45 smithi023 bash[21372]: audit 2024-03-11T20:43:45.491453+0000 mon.a (mon.0) 266 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:43:46.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:43:45 smithi023 bash[21372]: audit 2024-03-11T20:43:45.498631+0000 mon.a (mon.0) 267 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:43:46.032 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:43:45 smithi023 bash[29146]: audit 2024-03-11T20:43:44.853082+0000 mon.a (mon.0) 263 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:43:46.033 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:43:45 smithi023 bash[29146]: audit 2024-03-11T20:43:44.863390+0000 mon.a (mon.0) 264 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:43:46.033 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:43:45 smithi023 bash[29146]: audit 2024-03-11T20:43:45.490518+0000 mon.a (mon.0) 265 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:43:46.033 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:43:45 smithi023 bash[29146]: audit 2024-03-11T20:43:45.491453+0000 mon.a (mon.0) 266 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:43:46.033 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:43:45 smithi023 bash[29146]: audit 2024-03-11T20:43:45.498631+0000 mon.a (mon.0) 267 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:43:46.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:43:45 smithi146 bash[23809]: audit 2024-03-11T20:43:44.853082+0000 mon.a (mon.0) 263 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:43:46.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:43:45 smithi146 bash[23809]: audit 2024-03-11T20:43:44.863390+0000 mon.a (mon.0) 264 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:43:46.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:43:45 smithi146 bash[23809]: audit 2024-03-11T20:43:45.490518+0000 mon.a (mon.0) 265 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:43:46.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:43:45 smithi146 bash[23809]: audit 2024-03-11T20:43:45.491453+0000 mon.a (mon.0) 266 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:43:46.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:43:45 smithi146 bash[23809]: audit 2024-03-11T20:43:45.498631+0000 mon.a (mon.0) 267 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:43:47.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:43:46 smithi146 bash[23809]: cluster 2024-03-11T20:43:45.341315+0000 mgr.y (mgr.14152) 105 : cluster 0 pgmap v67: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:43:47.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:43:46 smithi146 bash[23809]: audit 2024-03-11T20:43:45.932705+0000 mon.c (mon.2) 3 : audit 1 from='osd.0 [v2:172.21.15.23:6802/2154627393,v1:172.21.15.23:6803/2154627393]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-11T20:43:47.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:43:46 smithi146 bash[23809]: audit 2024-03-11T20:43:45.933079+0000 mon.a (mon.0) 268 : audit 1 from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-11T20:43:47.281 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:43:46 smithi023 bash[29146]: cluster 2024-03-11T20:43:45.341315+0000 mgr.y (mgr.14152) 105 : cluster 0 pgmap v67: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:43:47.282 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:43:46 smithi023 bash[29146]: audit 2024-03-11T20:43:45.932705+0000 mon.c (mon.2) 3 : audit 1 from='osd.0 [v2:172.21.15.23:6802/2154627393,v1:172.21.15.23:6803/2154627393]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-11T20:43:47.282 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:43:46 smithi023 bash[29146]: audit 2024-03-11T20:43:45.933079+0000 mon.a (mon.0) 268 : audit 1 from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-11T20:43:47.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:43:46 smithi023 bash[21372]: cluster 2024-03-11T20:43:45.341315+0000 mgr.y (mgr.14152) 105 : cluster 0 pgmap v67: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:43:47.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:43:46 smithi023 bash[21372]: audit 2024-03-11T20:43:45.932705+0000 mon.c (mon.2) 3 : audit 1 from='osd.0 [v2:172.21.15.23:6802/2154627393,v1:172.21.15.23:6803/2154627393]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-11T20:43:47.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:43:46 smithi023 bash[21372]: audit 2024-03-11T20:43:45.933079+0000 mon.a (mon.0) 268 : audit 1 from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-11T20:43:48.204 INFO:teuthology.orchestra.run.smithi023.stdout:Created osd(s) 0 on host 'smithi023' 2024-03-11T20:43:48.230 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:43:47 smithi023 bash[29146]: audit 2024-03-11T20:43:46.879999+0000 mon.a (mon.0) 269 : audit 1 from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-11T20:43:48.230 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:43:47 smithi023 bash[29146]: cluster 2024-03-11T20:43:46.884955+0000 mon.a (mon.0) 270 : cluster 0 osdmap e6: 1 total, 0 up, 1 in 2024-03-11T20:43:48.230 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:43:47 smithi023 bash[29146]: audit 2024-03-11T20:43:46.885591+0000 mon.a (mon.0) 271 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T20:43:48.230 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:43:47 smithi023 bash[29146]: audit 2024-03-11T20:43:46.885877+0000 mon.c (mon.2) 4 : audit 1 from='osd.0 [v2:172.21.15.23:6802/2154627393,v1:172.21.15.23:6803/2154627393]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2024-03-11T20:43:48.230 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:43:47 smithi023 bash[29146]: audit 2024-03-11T20:43:46.886637+0000 mon.a (mon.0) 272 : audit 1 from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2024-03-11T20:43:48.230 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:43:47 smithi023 bash[21372]: audit 2024-03-11T20:43:46.879999+0000 mon.a (mon.0) 269 : audit 1 from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-11T20:43:48.231 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:43:47 smithi023 bash[21372]: cluster 2024-03-11T20:43:46.884955+0000 mon.a (mon.0) 270 : cluster 0 osdmap e6: 1 total, 0 up, 1 in 2024-03-11T20:43:48.231 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:43:47 smithi023 bash[21372]: audit 2024-03-11T20:43:46.885591+0000 mon.a (mon.0) 271 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T20:43:48.231 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:43:47 smithi023 bash[21372]: audit 2024-03-11T20:43:46.885877+0000 mon.c (mon.2) 4 : audit 1 from='osd.0 [v2:172.21.15.23:6802/2154627393,v1:172.21.15.23:6803/2154627393]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2024-03-11T20:43:48.231 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:43:47 smithi023 bash[21372]: audit 2024-03-11T20:43:46.886637+0000 mon.a (mon.0) 272 : audit 1 from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2024-03-11T20:43:48.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:43:47 smithi146 bash[23809]: audit 2024-03-11T20:43:46.879999+0000 mon.a (mon.0) 269 : audit 1 from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-11T20:43:48.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:43:47 smithi146 bash[23809]: cluster 2024-03-11T20:43:46.884955+0000 mon.a (mon.0) 270 : cluster 0 osdmap e6: 1 total, 0 up, 1 in 2024-03-11T20:43:48.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:43:47 smithi146 bash[23809]: audit 2024-03-11T20:43:46.885591+0000 mon.a (mon.0) 271 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T20:43:48.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:43:47 smithi146 bash[23809]: audit 2024-03-11T20:43:46.885877+0000 mon.c (mon.2) 4 : audit 1 from='osd.0 [v2:172.21.15.23:6802/2154627393,v1:172.21.15.23:6803/2154627393]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2024-03-11T20:43:48.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:43:47 smithi146 bash[23809]: audit 2024-03-11T20:43:46.886637+0000 mon.a (mon.0) 272 : audit 1 from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2024-03-11T20:43:48.823 DEBUG:teuthology.orchestra.run.smithi023:osd.0> sudo journalctl -f -n 0 -u ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@osd.0.service 2024-03-11T20:43:48.825 INFO:tasks.cephadm:Deploying osd.1 on smithi023 with /dev/vg_nvme/lv_3... 2024-03-11T20:43:48.825 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-11T20:43:49.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:43:48 smithi146 bash[23809]: cluster 2024-03-11T20:43:47.341594+0000 mgr.y (mgr.14152) 106 : cluster 0 pgmap v69: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:43:49.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:43:48 smithi146 bash[23809]: audit 2024-03-11T20:43:47.893326+0000 mon.a (mon.0) 273 : audit 1 from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]': finished 2024-03-11T20:43:49.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:43:48 smithi146 bash[23809]: cluster 2024-03-11T20:43:47.900498+0000 mon.a (mon.0) 274 : cluster 0 osdmap e7: 1 total, 0 up, 1 in 2024-03-11T20:43:49.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:43:48 smithi146 bash[23809]: audit 2024-03-11T20:43:47.901382+0000 mon.a (mon.0) 275 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T20:43:49.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:43:48 smithi146 bash[23809]: audit 2024-03-11T20:43:47.910842+0000 mon.a (mon.0) 276 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T20:43:49.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:43:48 smithi146 bash[23809]: audit 2024-03-11T20:43:48.185653+0000 mon.a (mon.0) 277 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:43:49.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:43:48 smithi146 bash[23809]: audit 2024-03-11T20:43:48.191435+0000 mon.a (mon.0) 278 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:43:49.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:43:48 smithi146 bash[23809]: audit 2024-03-11T20:43:48.219159+0000 mon.a (mon.0) 279 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:43:49.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:43:48 smithi146 bash[23809]: audit 2024-03-11T20:43:48.284838+0000 mon.a (mon.0) 280 : audit 1 from='osd.0 ' entity='osd.0' 2024-03-11T20:43:49.282 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:43:48 smithi023 bash[29146]: cluster 2024-03-11T20:43:47.341594+0000 mgr.y (mgr.14152) 106 : cluster 0 pgmap v69: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:43:49.282 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:43:48 smithi023 bash[29146]: audit 2024-03-11T20:43:47.893326+0000 mon.a (mon.0) 273 : audit 1 from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]': finished 2024-03-11T20:43:49.282 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:43:48 smithi023 bash[29146]: cluster 2024-03-11T20:43:47.900498+0000 mon.a (mon.0) 274 : cluster 0 osdmap e7: 1 total, 0 up, 1 in 2024-03-11T20:43:49.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:43:48 smithi023 bash[29146]: audit 2024-03-11T20:43:47.901382+0000 mon.a (mon.0) 275 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T20:43:49.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:43:48 smithi023 bash[29146]: audit 2024-03-11T20:43:47.910842+0000 mon.a (mon.0) 276 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T20:43:49.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:43:48 smithi023 bash[29146]: audit 2024-03-11T20:43:48.185653+0000 mon.a (mon.0) 277 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:43:49.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:43:48 smithi023 bash[29146]: audit 2024-03-11T20:43:48.191435+0000 mon.a (mon.0) 278 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:43:49.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:43:48 smithi023 bash[29146]: audit 2024-03-11T20:43:48.219159+0000 mon.a (mon.0) 279 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:43:49.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:43:48 smithi023 bash[29146]: audit 2024-03-11T20:43:48.284838+0000 mon.a (mon.0) 280 : audit 1 from='osd.0 ' entity='osd.0' 2024-03-11T20:43:49.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:43:48 smithi023 bash[21372]: cluster 2024-03-11T20:43:47.341594+0000 mgr.y (mgr.14152) 106 : cluster 0 pgmap v69: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:43:49.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:43:48 smithi023 bash[21372]: audit 2024-03-11T20:43:47.893326+0000 mon.a (mon.0) 273 : audit 1 from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]': finished 2024-03-11T20:43:49.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:43:48 smithi023 bash[21372]: cluster 2024-03-11T20:43:47.900498+0000 mon.a (mon.0) 274 : cluster 0 osdmap e7: 1 total, 0 up, 1 in 2024-03-11T20:43:49.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:43:48 smithi023 bash[21372]: audit 2024-03-11T20:43:47.901382+0000 mon.a (mon.0) 275 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T20:43:49.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:43:48 smithi023 bash[21372]: audit 2024-03-11T20:43:47.910842+0000 mon.a (mon.0) 276 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T20:43:49.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:43:48 smithi023 bash[21372]: audit 2024-03-11T20:43:48.185653+0000 mon.a (mon.0) 277 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:43:49.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:43:48 smithi023 bash[21372]: audit 2024-03-11T20:43:48.191435+0000 mon.a (mon.0) 278 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:43:49.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:43:48 smithi023 bash[21372]: audit 2024-03-11T20:43:48.219159+0000 mon.a (mon.0) 279 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:43:49.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:43:48 smithi023 bash[21372]: audit 2024-03-11T20:43:48.284838+0000 mon.a (mon.0) 280 : audit 1 from='osd.0 ' entity='osd.0' 2024-03-11T20:43:50.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:43:49 smithi146 bash[23809]: cluster 2024-03-11T20:43:46.909714+0000 osd.0 (osd.0) 1 : cluster 0 purged_snaps scrub starts 2024-03-11T20:43:50.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:43:49 smithi146 bash[23809]: cluster 2024-03-11T20:43:46.909844+0000 osd.0 (osd.0) 2 : cluster 0 purged_snaps scrub ok 2024-03-11T20:43:50.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:43:49 smithi146 bash[23809]: audit 2024-03-11T20:43:48.908912+0000 mon.a (mon.0) 281 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T20:43:50.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:43:49 smithi146 bash[23809]: cluster 2024-03-11T20:43:49.293206+0000 mon.a (mon.0) 282 : cluster 1 osd.0 [v2:172.21.15.23:6802/2154627393,v1:172.21.15.23:6803/2154627393] boot 2024-03-11T20:43:50.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:43:49 smithi146 bash[23809]: cluster 2024-03-11T20:43:49.293313+0000 mon.a (mon.0) 283 : cluster 0 osdmap e8: 1 total, 1 up, 1 in 2024-03-11T20:43:50.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:43:49 smithi146 bash[23809]: audit 2024-03-11T20:43:49.293668+0000 mon.a (mon.0) 284 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T20:43:50.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:43:49 smithi023 bash[21372]: cluster 2024-03-11T20:43:46.909714+0000 osd.0 (osd.0) 1 : cluster 0 purged_snaps scrub starts 2024-03-11T20:43:50.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:43:49 smithi023 bash[21372]: cluster 2024-03-11T20:43:46.909844+0000 osd.0 (osd.0) 2 : cluster 0 purged_snaps scrub ok 2024-03-11T20:43:50.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:43:49 smithi023 bash[21372]: audit 2024-03-11T20:43:48.908912+0000 mon.a (mon.0) 281 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T20:43:50.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:43:49 smithi023 bash[21372]: cluster 2024-03-11T20:43:49.293206+0000 mon.a (mon.0) 282 : cluster 1 osd.0 [v2:172.21.15.23:6802/2154627393,v1:172.21.15.23:6803/2154627393] boot 2024-03-11T20:43:50.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:43:49 smithi023 bash[21372]: cluster 2024-03-11T20:43:49.293313+0000 mon.a (mon.0) 283 : cluster 0 osdmap e8: 1 total, 1 up, 1 in 2024-03-11T20:43:50.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:43:49 smithi023 bash[21372]: audit 2024-03-11T20:43:49.293668+0000 mon.a (mon.0) 284 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T20:43:50.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:43:49 smithi023 bash[29146]: cluster 2024-03-11T20:43:46.909714+0000 osd.0 (osd.0) 1 : cluster 0 purged_snaps scrub starts 2024-03-11T20:43:50.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:43:49 smithi023 bash[29146]: cluster 2024-03-11T20:43:46.909844+0000 osd.0 (osd.0) 2 : cluster 0 purged_snaps scrub ok 2024-03-11T20:43:50.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:43:49 smithi023 bash[29146]: audit 2024-03-11T20:43:48.908912+0000 mon.a (mon.0) 281 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T20:43:50.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:43:49 smithi023 bash[29146]: cluster 2024-03-11T20:43:49.293206+0000 mon.a (mon.0) 282 : cluster 1 osd.0 [v2:172.21.15.23:6802/2154627393,v1:172.21.15.23:6803/2154627393] boot 2024-03-11T20:43:50.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:43:49 smithi023 bash[29146]: cluster 2024-03-11T20:43:49.293313+0000 mon.a (mon.0) 283 : cluster 0 osdmap e8: 1 total, 1 up, 1 in 2024-03-11T20:43:50.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:43:49 smithi023 bash[29146]: audit 2024-03-11T20:43:49.293668+0000 mon.a (mon.0) 284 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T20:43:51.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:43:50 smithi146 bash[23809]: cluster 2024-03-11T20:43:49.341968+0000 mgr.y (mgr.14152) 107 : cluster 0 pgmap v72: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:43:51.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:43:50 smithi023 bash[21372]: cluster 2024-03-11T20:43:49.341968+0000 mgr.y (mgr.14152) 107 : cluster 0 pgmap v72: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:43:51.282 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:43:50 smithi023 bash[29146]: cluster 2024-03-11T20:43:49.341968+0000 mgr.y (mgr.14152) 107 : cluster 0 pgmap v72: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:43:52.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:43:51 smithi146 bash[23809]: cluster 2024-03-11T20:43:50.920266+0000 mon.a (mon.0) 285 : cluster 0 osdmap e9: 1 total, 1 up, 1 in 2024-03-11T20:43:52.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:43:51 smithi023 bash[21372]: cluster 2024-03-11T20:43:50.920266+0000 mon.a (mon.0) 285 : cluster 0 osdmap e9: 1 total, 1 up, 1 in 2024-03-11T20:43:52.282 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:43:51 smithi023 bash[29146]: cluster 2024-03-11T20:43:50.920266+0000 mon.a (mon.0) 285 : cluster 0 osdmap e9: 1 total, 1 up, 1 in 2024-03-11T20:43:53.176 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:43:53.199 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:43:52 smithi023 bash[29146]: cluster 2024-03-11T20:43:51.342456+0000 mgr.y (mgr.14152) 108 : cluster 0 pgmap v74: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T20:43:53.200 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:43:52 smithi023 bash[21372]: cluster 2024-03-11T20:43:51.342456+0000 mgr.y (mgr.14152) 108 : cluster 0 pgmap v74: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T20:43:53.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:43:52 smithi146 bash[23809]: cluster 2024-03-11T20:43:51.342456+0000 mgr.y (mgr.14152) 108 : cluster 0 pgmap v74: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T20:43:55.198 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:43:54 smithi023 bash[21372]: cluster 2024-03-11T20:43:53.342881+0000 mgr.y (mgr.14152) 109 : cluster 0 pgmap v75: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T20:43:55.198 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:43:54 smithi023 bash[29146]: cluster 2024-03-11T20:43:53.342881+0000 mgr.y (mgr.14152) 109 : cluster 0 pgmap v75: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T20:43:55.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:43:54 smithi146 bash[23809]: cluster 2024-03-11T20:43:53.342881+0000 mgr.y (mgr.14152) 109 : cluster 0 pgmap v75: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T20:43:56.918 INFO:teuthology.orchestra.run.smithi023.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-11T20:43:56.918 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-11T20:43:56.918 INFO:teuthology.orchestra.run.smithi023.stderr: stderr: 10+0 records in 2024-03-11T20:43:56.918 INFO:teuthology.orchestra.run.smithi023.stderr:10+0 records out 2024-03-11T20:43:56.919 INFO:teuthology.orchestra.run.smithi023.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0103888 s, 1.0 GB/s 2024-03-11T20:43:56.919 INFO:teuthology.orchestra.run.smithi023.stderr:--> Zapping successful for: 2024-03-11T20:43:57.230 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:43:56 smithi023 bash[21372]: cluster 2024-03-11T20:43:55.343312+0000 mgr.y (mgr.14152) 110 : cluster 0 pgmap v76: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T20:43:57.230 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:43:56 smithi023 bash[29146]: cluster 2024-03-11T20:43:55.343312+0000 mgr.y (mgr.14152) 110 : cluster 0 pgmap v76: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T20:43:57.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:43:56 smithi146 bash[23809]: cluster 2024-03-11T20:43:55.343312+0000 mgr.y (mgr.14152) 110 : cluster 0 pgmap v76: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T20:43:57.702 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi023:vg_nvme/lv_3 2024-03-11T20:43:59.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:43:58 smithi146 bash[23809]: cluster 2024-03-11T20:43:57.343783+0000 mgr.y (mgr.14152) 111 : cluster 0 pgmap v77: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T20:43:59.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:43:58 smithi146 bash[23809]: audit 2024-03-11T20:43:58.073955+0000 mon.a (mon.0) 286 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:43:59.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:43:58 smithi146 bash[23809]: audit 2024-03-11T20:43:58.087153+0000 mon.a (mon.0) 287 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:43:59.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:43:58 smithi146 bash[23809]: audit 2024-03-11T20:43:58.089090+0000 mon.a (mon.0) 288 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:43:59.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:43:58 smithi146 bash[23809]: audit 2024-03-11T20:43:58.101422+0000 mon.a (mon.0) 289 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:43:59.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:43:58 smithi146 bash[23809]: audit 2024-03-11T20:43:58.106049+0000 mon.a (mon.0) 290 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:43:59.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:43:58 smithi146 bash[23809]: audit 2024-03-11T20:43:58.107928+0000 mon.a (mon.0) 291 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:43:59.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:43:58 smithi146 bash[23809]: audit 2024-03-11T20:43:58.117143+0000 mon.a (mon.0) 292 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:43:59.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:43:58 smithi023 bash[21372]: cluster 2024-03-11T20:43:57.343783+0000 mgr.y (mgr.14152) 111 : cluster 0 pgmap v77: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T20:43:59.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:43:58 smithi023 bash[21372]: audit 2024-03-11T20:43:58.073955+0000 mon.a (mon.0) 286 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:43:59.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:43:58 smithi023 bash[21372]: audit 2024-03-11T20:43:58.087153+0000 mon.a (mon.0) 287 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:43:59.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:43:58 smithi023 bash[21372]: audit 2024-03-11T20:43:58.089090+0000 mon.a (mon.0) 288 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:43:59.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:43:58 smithi023 bash[21372]: audit 2024-03-11T20:43:58.101422+0000 mon.a (mon.0) 289 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:43:59.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:43:58 smithi023 bash[21372]: audit 2024-03-11T20:43:58.106049+0000 mon.a (mon.0) 290 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:43:59.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:43:58 smithi023 bash[21372]: audit 2024-03-11T20:43:58.107928+0000 mon.a (mon.0) 291 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:43:59.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:43:58 smithi023 bash[21372]: audit 2024-03-11T20:43:58.117143+0000 mon.a (mon.0) 292 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:43:59.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:43:58 smithi023 bash[29146]: cluster 2024-03-11T20:43:57.343783+0000 mgr.y (mgr.14152) 111 : cluster 0 pgmap v77: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T20:43:59.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:43:58 smithi023 bash[29146]: audit 2024-03-11T20:43:58.073955+0000 mon.a (mon.0) 286 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:43:59.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:43:58 smithi023 bash[29146]: audit 2024-03-11T20:43:58.087153+0000 mon.a (mon.0) 287 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:43:59.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:43:58 smithi023 bash[29146]: audit 2024-03-11T20:43:58.089090+0000 mon.a (mon.0) 288 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:43:59.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:43:58 smithi023 bash[29146]: audit 2024-03-11T20:43:58.101422+0000 mon.a (mon.0) 289 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:43:59.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:43:58 smithi023 bash[29146]: audit 2024-03-11T20:43:58.106049+0000 mon.a (mon.0) 290 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:43:59.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:43:58 smithi023 bash[29146]: audit 2024-03-11T20:43:58.107928+0000 mon.a (mon.0) 291 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:43:59.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:43:58 smithi023 bash[29146]: audit 2024-03-11T20:43:58.117143+0000 mon.a (mon.0) 292 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:44:00.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:43:59 smithi146 bash[23809]: cephadm 2024-03-11T20:43:58.059903+0000 mgr.y (mgr.14152) 112 : cephadm 1 Detected new or changed devices on smithi023 2024-03-11T20:44:00.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:43:59 smithi146 bash[23809]: cephadm 2024-03-11T20:43:58.090343+0000 mgr.y (mgr.14152) 113 : cephadm 1 Adjusting osd_memory_target on smithi023 to 16241M 2024-03-11T20:44:00.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:43:59 smithi023 bash[21372]: cephadm 2024-03-11T20:43:58.059903+0000 mgr.y (mgr.14152) 112 : cephadm 1 Detected new or changed devices on smithi023 2024-03-11T20:44:00.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:43:59 smithi023 bash[21372]: cephadm 2024-03-11T20:43:58.090343+0000 mgr.y (mgr.14152) 113 : cephadm 1 Adjusting osd_memory_target on smithi023 to 16241M 2024-03-11T20:44:00.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:43:59 smithi023 bash[29146]: cephadm 2024-03-11T20:43:58.059903+0000 mgr.y (mgr.14152) 112 : cephadm 1 Detected new or changed devices on smithi023 2024-03-11T20:44:00.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:43:59 smithi023 bash[29146]: cephadm 2024-03-11T20:43:58.090343+0000 mgr.y (mgr.14152) 113 : cephadm 1 Adjusting osd_memory_target on smithi023 to 16241M 2024-03-11T20:44:01.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:44:00 smithi146 bash[23809]: cluster 2024-03-11T20:43:59.344193+0000 mgr.y (mgr.14152) 114 : cluster 0 pgmap v78: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T20:44:01.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:44:00 smithi023 bash[21372]: cluster 2024-03-11T20:43:59.344193+0000 mgr.y (mgr.14152) 114 : cluster 0 pgmap v78: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T20:44:01.282 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:44:00 smithi023 bash[29146]: cluster 2024-03-11T20:43:59.344193+0000 mgr.y (mgr.14152) 114 : cluster 0 pgmap v78: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T20:44:02.441 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:44:03.244 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:44:02 smithi023 bash[21372]: cluster 2024-03-11T20:44:01.344656+0000 mgr.y (mgr.14152) 115 : cluster 0 pgmap v79: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T20:44:03.245 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:44:02 smithi023 bash[29146]: cluster 2024-03-11T20:44:01.344656+0000 mgr.y (mgr.14152) 115 : cluster 0 pgmap v79: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T20:44:03.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:44:02 smithi146 bash[23809]: cluster 2024-03-11T20:44:01.344656+0000 mgr.y (mgr.14152) 115 : cluster 0 pgmap v79: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T20:44:04.991 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:44:04 smithi023 bash[21372]: cluster 2024-03-11T20:44:03.345111+0000 mgr.y (mgr.14152) 116 : cluster 0 pgmap v80: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T20:44:05.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:44:04 smithi146 bash[23809]: cluster 2024-03-11T20:44:03.345111+0000 mgr.y (mgr.14152) 116 : cluster 0 pgmap v80: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T20:44:05.282 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:44:04 smithi023 bash[29146]: cluster 2024-03-11T20:44:03.345111+0000 mgr.y (mgr.14152) 116 : cluster 0 pgmap v80: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T20:44:06.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:44:06 smithi146 bash[23809]: audit 2024-03-11T20:44:05.175215+0000 mon.a (mon.0) 293 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T20:44:06.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:44:06 smithi146 bash[23809]: audit 2024-03-11T20:44:05.181066+0000 mon.a (mon.0) 294 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T20:44:06.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:44:06 smithi146 bash[23809]: audit 2024-03-11T20:44:05.182586+0000 mon.a (mon.0) 295 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:44:06.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:44:06 smithi023 bash[21372]: audit 2024-03-11T20:44:05.175215+0000 mon.a (mon.0) 293 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T20:44:06.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:44:06 smithi023 bash[21372]: audit 2024-03-11T20:44:05.181066+0000 mon.a (mon.0) 294 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T20:44:06.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:44:06 smithi023 bash[21372]: audit 2024-03-11T20:44:05.182586+0000 mon.a (mon.0) 295 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:44:06.282 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:44:06 smithi023 bash[29146]: audit 2024-03-11T20:44:05.175215+0000 mon.a (mon.0) 293 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T20:44:06.282 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:44:06 smithi023 bash[29146]: audit 2024-03-11T20:44:05.181066+0000 mon.a (mon.0) 294 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T20:44:06.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:44:06 smithi023 bash[29146]: audit 2024-03-11T20:44:05.182586+0000 mon.a (mon.0) 295 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:44:07.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:44:07 smithi023 bash[21372]: audit 2024-03-11T20:44:05.168868+0000 mgr.y (mgr.14152) 117 : audit 0 from='client.14229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:44:07.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:44:07 smithi023 bash[21372]: cluster 2024-03-11T20:44:05.345569+0000 mgr.y (mgr.14152) 118 : cluster 0 pgmap v81: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T20:44:07.282 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:44:07 smithi023 bash[29146]: audit 2024-03-11T20:44:05.168868+0000 mgr.y (mgr.14152) 117 : audit 0 from='client.14229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:44:07.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:44:07 smithi023 bash[29146]: cluster 2024-03-11T20:44:05.345569+0000 mgr.y (mgr.14152) 118 : cluster 0 pgmap v81: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T20:44:07.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:44:07 smithi146 bash[23809]: audit 2024-03-11T20:44:05.168868+0000 mgr.y (mgr.14152) 117 : audit 0 from='client.14229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:44:07.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:44:07 smithi146 bash[23809]: cluster 2024-03-11T20:44:05.345569+0000 mgr.y (mgr.14152) 118 : cluster 0 pgmap v81: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T20:44:09.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:44:09 smithi023 bash[21372]: cluster 2024-03-11T20:44:07.346030+0000 mgr.y (mgr.14152) 119 : cluster 0 pgmap v82: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T20:44:09.282 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:44:09 smithi023 bash[29146]: cluster 2024-03-11T20:44:07.346030+0000 mgr.y (mgr.14152) 119 : cluster 0 pgmap v82: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T20:44:09.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:44:09 smithi146 bash[23809]: cluster 2024-03-11T20:44:07.346030+0000 mgr.y (mgr.14152) 119 : cluster 0 pgmap v82: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T20:44:10.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:44:10 smithi146 bash[23809]: cluster 2024-03-11T20:44:09.346535+0000 mgr.y (mgr.14152) 120 : cluster 0 pgmap v83: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T20:44:10.532 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:44:10 smithi023 bash[29146]: cluster 2024-03-11T20:44:09.346535+0000 mgr.y (mgr.14152) 120 : cluster 0 pgmap v83: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T20:44:10.532 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:44:10 smithi023 bash[21372]: cluster 2024-03-11T20:44:09.346535+0000 mgr.y (mgr.14152) 120 : cluster 0 pgmap v83: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T20:44:12.532 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:44:12 smithi023 bash[21372]: cluster 2024-03-11T20:44:11.347018+0000 mgr.y (mgr.14152) 121 : cluster 0 pgmap v84: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T20:44:12.532 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:44:12 smithi023 bash[29146]: cluster 2024-03-11T20:44:11.347018+0000 mgr.y (mgr.14152) 121 : cluster 0 pgmap v84: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T20:44:12.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:44:12 smithi146 bash[23809]: cluster 2024-03-11T20:44:11.347018+0000 mgr.y (mgr.14152) 121 : cluster 0 pgmap v84: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T20:44:13.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:44:13 smithi146 bash[23809]: audit 2024-03-11T20:44:13.174250+0000 mon.c (mon.2) 5 : audit 1 from='client.? 172.21.15.23:0/2868217210' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e8fd7500-9081-41a1-8077-a4d0f0e12a2c"}]: dispatch 2024-03-11T20:44:13.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:44:13 smithi146 bash[23809]: audit 2024-03-11T20:44:13.174839+0000 mon.a (mon.0) 296 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e8fd7500-9081-41a1-8077-a4d0f0e12a2c"}]: dispatch 2024-03-11T20:44:13.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:44:13 smithi146 bash[23809]: audit 2024-03-11T20:44:13.181547+0000 mon.a (mon.0) 297 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e8fd7500-9081-41a1-8077-a4d0f0e12a2c"}]': finished 2024-03-11T20:44:13.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:44:13 smithi146 bash[23809]: cluster 2024-03-11T20:44:13.189311+0000 mon.a (mon.0) 298 : cluster 0 osdmap e10: 2 total, 1 up, 2 in 2024-03-11T20:44:13.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:44:13 smithi146 bash[23809]: audit 2024-03-11T20:44:13.189841+0000 mon.a (mon.0) 299 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T20:44:13.782 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:44:13 smithi023 bash[29146]: audit 2024-03-11T20:44:13.174250+0000 mon.c (mon.2) 5 : audit 1 from='client.? 172.21.15.23:0/2868217210' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e8fd7500-9081-41a1-8077-a4d0f0e12a2c"}]: dispatch 2024-03-11T20:44:13.782 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:44:13 smithi023 bash[29146]: audit 2024-03-11T20:44:13.174839+0000 mon.a (mon.0) 296 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e8fd7500-9081-41a1-8077-a4d0f0e12a2c"}]: dispatch 2024-03-11T20:44:13.782 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:44:13 smithi023 bash[29146]: audit 2024-03-11T20:44:13.181547+0000 mon.a (mon.0) 297 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e8fd7500-9081-41a1-8077-a4d0f0e12a2c"}]': finished 2024-03-11T20:44:13.782 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:44:13 smithi023 bash[29146]: cluster 2024-03-11T20:44:13.189311+0000 mon.a (mon.0) 298 : cluster 0 osdmap e10: 2 total, 1 up, 2 in 2024-03-11T20:44:13.782 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:44:13 smithi023 bash[29146]: audit 2024-03-11T20:44:13.189841+0000 mon.a (mon.0) 299 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T20:44:13.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:44:13 smithi023 bash[21372]: audit 2024-03-11T20:44:13.174250+0000 mon.c (mon.2) 5 : audit 1 from='client.? 172.21.15.23:0/2868217210' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e8fd7500-9081-41a1-8077-a4d0f0e12a2c"}]: dispatch 2024-03-11T20:44:13.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:44:13 smithi023 bash[21372]: audit 2024-03-11T20:44:13.174839+0000 mon.a (mon.0) 296 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e8fd7500-9081-41a1-8077-a4d0f0e12a2c"}]: dispatch 2024-03-11T20:44:13.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:44:13 smithi023 bash[21372]: audit 2024-03-11T20:44:13.181547+0000 mon.a (mon.0) 297 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e8fd7500-9081-41a1-8077-a4d0f0e12a2c"}]': finished 2024-03-11T20:44:13.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:44:13 smithi023 bash[21372]: cluster 2024-03-11T20:44:13.189311+0000 mon.a (mon.0) 298 : cluster 0 osdmap e10: 2 total, 1 up, 2 in 2024-03-11T20:44:13.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:44:13 smithi023 bash[21372]: audit 2024-03-11T20:44:13.189841+0000 mon.a (mon.0) 299 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T20:44:14.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:44:14 smithi146 bash[23809]: cluster 2024-03-11T20:44:13.347343+0000 mgr.y (mgr.14152) 122 : cluster 0 pgmap v86: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T20:44:14.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:44:14 smithi146 bash[23809]: audit 2024-03-11T20:44:13.920870+0000 mon.a (mon.0) 300 : audit 0 from='client.? 172.21.15.23:0/559519339' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T20:44:14.782 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:44:14 smithi023 bash[21372]: cluster 2024-03-11T20:44:13.347343+0000 mgr.y (mgr.14152) 122 : cluster 0 pgmap v86: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T20:44:14.782 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:44:14 smithi023 bash[21372]: audit 2024-03-11T20:44:13.920870+0000 mon.a (mon.0) 300 : audit 0 from='client.? 172.21.15.23:0/559519339' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T20:44:14.782 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:44:14 smithi023 bash[29146]: cluster 2024-03-11T20:44:13.347343+0000 mgr.y (mgr.14152) 122 : cluster 0 pgmap v86: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T20:44:14.782 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:44:14 smithi023 bash[29146]: audit 2024-03-11T20:44:13.920870+0000 mon.a (mon.0) 300 : audit 0 from='client.? 172.21.15.23:0/559519339' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T20:44:16.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:44:16 smithi146 bash[23809]: cluster 2024-03-11T20:44:15.347683+0000 mgr.y (mgr.14152) 123 : cluster 0 pgmap v87: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T20:44:16.782 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:44:16 smithi023 bash[21372]: cluster 2024-03-11T20:44:15.347683+0000 mgr.y (mgr.14152) 123 : cluster 0 pgmap v87: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T20:44:16.782 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:44:16 smithi023 bash[29146]: cluster 2024-03-11T20:44:15.347683+0000 mgr.y (mgr.14152) 123 : cluster 0 pgmap v87: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T20:44:18.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:44:18 smithi146 bash[23809]: cluster 2024-03-11T20:44:17.348066+0000 mgr.y (mgr.14152) 124 : cluster 0 pgmap v88: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T20:44:18.782 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:44:18 smithi023 bash[21372]: cluster 2024-03-11T20:44:17.348066+0000 mgr.y (mgr.14152) 124 : cluster 0 pgmap v88: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T20:44:18.782 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:44:18 smithi023 bash[29146]: cluster 2024-03-11T20:44:17.348066+0000 mgr.y (mgr.14152) 124 : cluster 0 pgmap v88: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T20:44:20.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:44:20 smithi146 bash[23809]: cluster 2024-03-11T20:44:19.348542+0000 mgr.y (mgr.14152) 125 : cluster 0 pgmap v89: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T20:44:20.782 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:44:20 smithi023 bash[29146]: cluster 2024-03-11T20:44:19.348542+0000 mgr.y (mgr.14152) 125 : cluster 0 pgmap v89: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T20:44:20.782 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:44:20 smithi023 bash[21372]: cluster 2024-03-11T20:44:19.348542+0000 mgr.y (mgr.14152) 125 : cluster 0 pgmap v89: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T20:44:22.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:44:22 smithi146 bash[23809]: cluster 2024-03-11T20:44:21.348979+0000 mgr.y (mgr.14152) 126 : cluster 0 pgmap v90: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T20:44:22.782 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:44:22 smithi023 bash[29146]: cluster 2024-03-11T20:44:21.348979+0000 mgr.y (mgr.14152) 126 : cluster 0 pgmap v90: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T20:44:22.782 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:44:22 smithi023 bash[21372]: cluster 2024-03-11T20:44:21.348979+0000 mgr.y (mgr.14152) 126 : cluster 0 pgmap v90: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T20:44:24.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:44:24 smithi146 bash[23809]: cluster 2024-03-11T20:44:23.349447+0000 mgr.y (mgr.14152) 127 : cluster 0 pgmap v91: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T20:44:24.777 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:44:24 smithi023 bash[29146]: cluster 2024-03-11T20:44:23.349447+0000 mgr.y (mgr.14152) 127 : cluster 0 pgmap v91: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T20:44:24.777 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:44:24 smithi023 bash[21372]: cluster 2024-03-11T20:44:23.349447+0000 mgr.y (mgr.14152) 127 : cluster 0 pgmap v91: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T20:44:25.782 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:44:25 smithi023 bash[21372]: audit 2024-03-11T20:44:25.409441+0000 mon.a (mon.0) 301 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-11T20:44:25.782 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:44:25 smithi023 bash[21372]: audit 2024-03-11T20:44:25.411133+0000 mon.a (mon.0) 302 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:44:25.782 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:44:25 smithi023 bash[29146]: audit 2024-03-11T20:44:25.409441+0000 mon.a (mon.0) 301 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-11T20:44:25.782 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:44:25 smithi023 bash[29146]: audit 2024-03-11T20:44:25.411133+0000 mon.a (mon.0) 302 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:44:26.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:44:25 smithi146 bash[23809]: audit 2024-03-11T20:44:25.409441+0000 mon.a (mon.0) 301 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-11T20:44:26.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:44:25 smithi146 bash[23809]: audit 2024-03-11T20:44:25.411133+0000 mon.a (mon.0) 302 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:44:26.782 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:44:26 smithi023 bash[21372]: cluster 2024-03-11T20:44:25.349941+0000 mgr.y (mgr.14152) 128 : cluster 0 pgmap v92: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T20:44:26.782 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:44:26 smithi023 bash[21372]: cephadm 2024-03-11T20:44:25.412611+0000 mgr.y (mgr.14152) 129 : cephadm 1 Deploying daemon osd.1 on smithi023 2024-03-11T20:44:26.782 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:44:26 smithi023 bash[29146]: cluster 2024-03-11T20:44:25.349941+0000 mgr.y (mgr.14152) 128 : cluster 0 pgmap v92: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T20:44:26.782 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:44:26 smithi023 bash[29146]: cephadm 2024-03-11T20:44:25.412611+0000 mgr.y (mgr.14152) 129 : cephadm 1 Deploying daemon osd.1 on smithi023 2024-03-11T20:44:27.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:44:26 smithi146 bash[23809]: cluster 2024-03-11T20:44:25.349941+0000 mgr.y (mgr.14152) 128 : cluster 0 pgmap v92: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T20:44:27.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:44:26 smithi146 bash[23809]: cephadm 2024-03-11T20:44:25.412611+0000 mgr.y (mgr.14152) 129 : cephadm 1 Deploying daemon osd.1 on smithi023 2024-03-11T20:44:28.782 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:44:28 smithi023 bash[21372]: cluster 2024-03-11T20:44:27.350314+0000 mgr.y (mgr.14152) 130 : cluster 0 pgmap v93: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T20:44:28.782 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:44:28 smithi023 bash[29146]: cluster 2024-03-11T20:44:27.350314+0000 mgr.y (mgr.14152) 130 : cluster 0 pgmap v93: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T20:44:29.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:44:28 smithi146 bash[23809]: cluster 2024-03-11T20:44:27.350314+0000 mgr.y (mgr.14152) 130 : cluster 0 pgmap v93: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T20:44:29.523 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:44:29 smithi023 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:44:29.524 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:44:29 smithi023 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:44:29.524 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:44:29 smithi023 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:44:29.524 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 11 20:44:29 smithi023 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:44:29.782 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:44:29 smithi023 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:44:29.782 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:44:29 smithi023 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:44:29.782 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:44:29 smithi023 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:44:29.783 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 11 20:44:29 smithi023 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:44:31.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:44:30 smithi023 bash[21372]: cluster 2024-03-11T20:44:29.350706+0000 mgr.y (mgr.14152) 131 : cluster 0 pgmap v94: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T20:44:31.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:44:30 smithi023 bash[21372]: audit 2024-03-11T20:44:29.818093+0000 mon.a (mon.0) 303 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:44:31.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:44:30 smithi023 bash[21372]: audit 2024-03-11T20:44:29.822880+0000 mon.a (mon.0) 304 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:44:31.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:44:30 smithi023 bash[21372]: audit 2024-03-11T20:44:29.836140+0000 mon.a (mon.0) 305 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:44:31.033 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:44:30 smithi023 bash[29146]: cluster 2024-03-11T20:44:29.350706+0000 mgr.y (mgr.14152) 131 : cluster 0 pgmap v94: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T20:44:31.033 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:44:30 smithi023 bash[29146]: audit 2024-03-11T20:44:29.818093+0000 mon.a (mon.0) 303 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:44:31.033 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:44:30 smithi023 bash[29146]: audit 2024-03-11T20:44:29.822880+0000 mon.a (mon.0) 304 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:44:31.033 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:44:30 smithi023 bash[29146]: audit 2024-03-11T20:44:29.836140+0000 mon.a (mon.0) 305 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:44:31.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:44:30 smithi146 bash[23809]: cluster 2024-03-11T20:44:29.350706+0000 mgr.y (mgr.14152) 131 : cluster 0 pgmap v94: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T20:44:31.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:44:30 smithi146 bash[23809]: audit 2024-03-11T20:44:29.818093+0000 mon.a (mon.0) 303 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:44:31.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:44:30 smithi146 bash[23809]: audit 2024-03-11T20:44:29.822880+0000 mon.a (mon.0) 304 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:44:31.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:44:30 smithi146 bash[23809]: audit 2024-03-11T20:44:29.836140+0000 mon.a (mon.0) 305 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:44:33.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:44:32 smithi023 bash[21372]: cluster 2024-03-11T20:44:31.351048+0000 mgr.y (mgr.14152) 132 : cluster 0 pgmap v95: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T20:44:33.091 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:44:32 smithi023 bash[29146]: cluster 2024-03-11T20:44:31.351048+0000 mgr.y (mgr.14152) 132 : cluster 0 pgmap v95: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T20:44:33.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:44:32 smithi146 bash[23809]: cluster 2024-03-11T20:44:31.351048+0000 mgr.y (mgr.14152) 132 : cluster 0 pgmap v95: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T20:44:35.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:44:34 smithi146 bash[23809]: cluster 2024-03-11T20:44:33.351515+0000 mgr.y (mgr.14152) 133 : cluster 0 pgmap v96: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T20:44:35.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:44:34 smithi023 bash[21372]: cluster 2024-03-11T20:44:33.351515+0000 mgr.y (mgr.14152) 133 : cluster 0 pgmap v96: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T20:44:35.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:44:34 smithi023 bash[29146]: cluster 2024-03-11T20:44:33.351515+0000 mgr.y (mgr.14152) 133 : cluster 0 pgmap v96: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T20:44:37.255 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:44:36 smithi023 bash[21372]: cluster 2024-03-11T20:44:35.351948+0000 mgr.y (mgr.14152) 134 : cluster 0 pgmap v97: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T20:44:37.255 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:44:36 smithi023 bash[21372]: audit 2024-03-11T20:44:36.022769+0000 mon.a (mon.0) 306 : audit 1 from='osd.1 [v2:172.21.15.23:6810/2506707705,v1:172.21.15.23:6811/2506707705]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-11T20:44:37.255 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:44:36 smithi023 bash[21372]: audit 2024-03-11T20:44:36.737799+0000 mon.a (mon.0) 307 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:44:37.255 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:44:36 smithi023 bash[21372]: audit 2024-03-11T20:44:36.747659+0000 mon.a (mon.0) 308 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:44:37.255 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:44:36 smithi023 bash[21372]: audit 2024-03-11T20:44:36.749052+0000 mon.a (mon.0) 309 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:44:37.256 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:44:36 smithi023 bash[21372]: audit 2024-03-11T20:44:36.750124+0000 mon.a (mon.0) 310 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:44:37.256 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:44:36 smithi023 bash[21372]: audit 2024-03-11T20:44:36.754985+0000 mon.a (mon.0) 311 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:44:37.256 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:44:36 smithi023 bash[29146]: cluster 2024-03-11T20:44:35.351948+0000 mgr.y (mgr.14152) 134 : cluster 0 pgmap v97: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T20:44:37.256 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:44:36 smithi023 bash[29146]: audit 2024-03-11T20:44:36.022769+0000 mon.a (mon.0) 306 : audit 1 from='osd.1 [v2:172.21.15.23:6810/2506707705,v1:172.21.15.23:6811/2506707705]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-11T20:44:37.256 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:44:36 smithi023 bash[29146]: audit 2024-03-11T20:44:36.737799+0000 mon.a (mon.0) 307 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:44:37.256 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:44:36 smithi023 bash[29146]: audit 2024-03-11T20:44:36.747659+0000 mon.a (mon.0) 308 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:44:37.256 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:44:36 smithi023 bash[29146]: audit 2024-03-11T20:44:36.749052+0000 mon.a (mon.0) 309 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:44:37.256 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:44:36 smithi023 bash[29146]: audit 2024-03-11T20:44:36.750124+0000 mon.a (mon.0) 310 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:44:37.256 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:44:36 smithi023 bash[29146]: audit 2024-03-11T20:44:36.754985+0000 mon.a (mon.0) 311 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:44:37.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:44:36 smithi146 bash[23809]: cluster 2024-03-11T20:44:35.351948+0000 mgr.y (mgr.14152) 134 : cluster 0 pgmap v97: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T20:44:37.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:44:36 smithi146 bash[23809]: audit 2024-03-11T20:44:36.022769+0000 mon.a (mon.0) 306 : audit 1 from='osd.1 [v2:172.21.15.23:6810/2506707705,v1:172.21.15.23:6811/2506707705]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-11T20:44:37.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:44:36 smithi146 bash[23809]: audit 2024-03-11T20:44:36.737799+0000 mon.a (mon.0) 307 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:44:37.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:44:36 smithi146 bash[23809]: audit 2024-03-11T20:44:36.747659+0000 mon.a (mon.0) 308 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:44:37.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:44:36 smithi146 bash[23809]: audit 2024-03-11T20:44:36.749052+0000 mon.a (mon.0) 309 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:44:37.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:44:36 smithi146 bash[23809]: audit 2024-03-11T20:44:36.750124+0000 mon.a (mon.0) 310 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:44:37.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:44:36 smithi146 bash[23809]: audit 2024-03-11T20:44:36.754985+0000 mon.a (mon.0) 311 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:44:38.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:44:37 smithi146 bash[23809]: audit 2024-03-11T20:44:36.852863+0000 mon.a (mon.0) 312 : audit 1 from='osd.1 [v2:172.21.15.23:6810/2506707705,v1:172.21.15.23:6811/2506707705]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-11T20:44:38.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:44:37 smithi146 bash[23809]: cluster 2024-03-11T20:44:36.856629+0000 mon.a (mon.0) 313 : cluster 0 osdmap e11: 2 total, 1 up, 2 in 2024-03-11T20:44:38.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:44:37 smithi146 bash[23809]: audit 2024-03-11T20:44:36.857129+0000 mon.a (mon.0) 314 : audit 1 from='osd.1 [v2:172.21.15.23:6810/2506707705,v1:172.21.15.23:6811/2506707705]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2024-03-11T20:44:38.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:44:37 smithi146 bash[23809]: audit 2024-03-11T20:44:36.857633+0000 mon.a (mon.0) 315 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T20:44:38.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:44:37 smithi023 bash[21372]: audit 2024-03-11T20:44:36.852863+0000 mon.a (mon.0) 312 : audit 1 from='osd.1 [v2:172.21.15.23:6810/2506707705,v1:172.21.15.23:6811/2506707705]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-11T20:44:38.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:44:37 smithi023 bash[21372]: cluster 2024-03-11T20:44:36.856629+0000 mon.a (mon.0) 313 : cluster 0 osdmap e11: 2 total, 1 up, 2 in 2024-03-11T20:44:38.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:44:37 smithi023 bash[21372]: audit 2024-03-11T20:44:36.857129+0000 mon.a (mon.0) 314 : audit 1 from='osd.1 [v2:172.21.15.23:6810/2506707705,v1:172.21.15.23:6811/2506707705]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2024-03-11T20:44:38.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:44:37 smithi023 bash[21372]: audit 2024-03-11T20:44:36.857633+0000 mon.a (mon.0) 315 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T20:44:38.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:44:37 smithi023 bash[29146]: audit 2024-03-11T20:44:36.852863+0000 mon.a (mon.0) 312 : audit 1 from='osd.1 [v2:172.21.15.23:6810/2506707705,v1:172.21.15.23:6811/2506707705]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-11T20:44:38.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:44:37 smithi023 bash[29146]: cluster 2024-03-11T20:44:36.856629+0000 mon.a (mon.0) 313 : cluster 0 osdmap e11: 2 total, 1 up, 2 in 2024-03-11T20:44:38.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:44:37 smithi023 bash[29146]: audit 2024-03-11T20:44:36.857129+0000 mon.a (mon.0) 314 : audit 1 from='osd.1 [v2:172.21.15.23:6810/2506707705,v1:172.21.15.23:6811/2506707705]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2024-03-11T20:44:38.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:44:37 smithi023 bash[29146]: audit 2024-03-11T20:44:36.857633+0000 mon.a (mon.0) 315 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T20:44:39.032 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:44:38 smithi023 bash[29146]: cluster 2024-03-11T20:44:37.352365+0000 mgr.y (mgr.14152) 135 : cluster 0 pgmap v99: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T20:44:39.032 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:44:38 smithi023 bash[29146]: audit 2024-03-11T20:44:37.863918+0000 mon.a (mon.0) 316 : audit 1 from='osd.1 [v2:172.21.15.23:6810/2506707705,v1:172.21.15.23:6811/2506707705]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]': finished 2024-03-11T20:44:39.032 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:44:38 smithi023 bash[29146]: cluster 2024-03-11T20:44:37.872456+0000 mon.a (mon.0) 317 : cluster 0 osdmap e12: 2 total, 1 up, 2 in 2024-03-11T20:44:39.032 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:44:38 smithi023 bash[29146]: audit 2024-03-11T20:44:37.874068+0000 mon.a (mon.0) 318 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T20:44:39.032 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:44:38 smithi023 bash[29146]: audit 2024-03-11T20:44:37.880970+0000 mon.a (mon.0) 319 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T20:44:39.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:44:38 smithi023 bash[21372]: cluster 2024-03-11T20:44:37.352365+0000 mgr.y (mgr.14152) 135 : cluster 0 pgmap v99: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T20:44:39.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:44:38 smithi023 bash[21372]: audit 2024-03-11T20:44:37.863918+0000 mon.a (mon.0) 316 : audit 1 from='osd.1 [v2:172.21.15.23:6810/2506707705,v1:172.21.15.23:6811/2506707705]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]': finished 2024-03-11T20:44:39.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:44:38 smithi023 bash[21372]: cluster 2024-03-11T20:44:37.872456+0000 mon.a (mon.0) 317 : cluster 0 osdmap e12: 2 total, 1 up, 2 in 2024-03-11T20:44:39.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:44:38 smithi023 bash[21372]: audit 2024-03-11T20:44:37.874068+0000 mon.a (mon.0) 318 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T20:44:39.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:44:38 smithi023 bash[21372]: audit 2024-03-11T20:44:37.880970+0000 mon.a (mon.0) 319 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T20:44:39.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:44:38 smithi146 bash[23809]: cluster 2024-03-11T20:44:37.352365+0000 mgr.y (mgr.14152) 135 : cluster 0 pgmap v99: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T20:44:39.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:44:38 smithi146 bash[23809]: audit 2024-03-11T20:44:37.863918+0000 mon.a (mon.0) 316 : audit 1 from='osd.1 [v2:172.21.15.23:6810/2506707705,v1:172.21.15.23:6811/2506707705]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]': finished 2024-03-11T20:44:39.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:44:38 smithi146 bash[23809]: cluster 2024-03-11T20:44:37.872456+0000 mon.a (mon.0) 317 : cluster 0 osdmap e12: 2 total, 1 up, 2 in 2024-03-11T20:44:39.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:44:38 smithi146 bash[23809]: audit 2024-03-11T20:44:37.874068+0000 mon.a (mon.0) 318 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T20:44:39.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:44:38 smithi146 bash[23809]: audit 2024-03-11T20:44:37.880970+0000 mon.a (mon.0) 319 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T20:44:39.929 INFO:teuthology.orchestra.run.smithi023.stdout:Created osd(s) 1 on host 'smithi023' 2024-03-11T20:44:40.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:44:39 smithi146 bash[23809]: cluster 2024-03-11T20:44:36.993808+0000 osd.1 (osd.1) 1 : cluster 0 purged_snaps scrub starts 2024-03-11T20:44:40.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:44:39 smithi146 bash[23809]: cluster 2024-03-11T20:44:36.993963+0000 osd.1 (osd.1) 2 : cluster 0 purged_snaps scrub ok 2024-03-11T20:44:40.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:44:39 smithi146 bash[23809]: cluster 2024-03-11T20:44:38.874255+0000 mon.a (mon.0) 320 : cluster 1 osd.1 [v2:172.21.15.23:6810/2506707705,v1:172.21.15.23:6811/2506707705] boot 2024-03-11T20:44:40.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:44:39 smithi146 bash[23809]: cluster 2024-03-11T20:44:38.874342+0000 mon.a (mon.0) 321 : cluster 0 osdmap e13: 2 total, 2 up, 2 in 2024-03-11T20:44:40.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:44:39 smithi146 bash[23809]: audit 2024-03-11T20:44:38.875236+0000 mon.a (mon.0) 322 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T20:44:40.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:44:39 smithi023 bash[21372]: cluster 2024-03-11T20:44:36.993808+0000 osd.1 (osd.1) 1 : cluster 0 purged_snaps scrub starts 2024-03-11T20:44:40.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:44:39 smithi023 bash[21372]: cluster 2024-03-11T20:44:36.993963+0000 osd.1 (osd.1) 2 : cluster 0 purged_snaps scrub ok 2024-03-11T20:44:40.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:44:39 smithi023 bash[21372]: cluster 2024-03-11T20:44:38.874255+0000 mon.a (mon.0) 320 : cluster 1 osd.1 [v2:172.21.15.23:6810/2506707705,v1:172.21.15.23:6811/2506707705] boot 2024-03-11T20:44:40.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:44:39 smithi023 bash[21372]: cluster 2024-03-11T20:44:38.874342+0000 mon.a (mon.0) 321 : cluster 0 osdmap e13: 2 total, 2 up, 2 in 2024-03-11T20:44:40.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:44:39 smithi023 bash[21372]: audit 2024-03-11T20:44:38.875236+0000 mon.a (mon.0) 322 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T20:44:40.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:44:39 smithi023 bash[29146]: cluster 2024-03-11T20:44:36.993808+0000 osd.1 (osd.1) 1 : cluster 0 purged_snaps scrub starts 2024-03-11T20:44:40.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:44:39 smithi023 bash[29146]: cluster 2024-03-11T20:44:36.993963+0000 osd.1 (osd.1) 2 : cluster 0 purged_snaps scrub ok 2024-03-11T20:44:40.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:44:39 smithi023 bash[29146]: cluster 2024-03-11T20:44:38.874255+0000 mon.a (mon.0) 320 : cluster 1 osd.1 [v2:172.21.15.23:6810/2506707705,v1:172.21.15.23:6811/2506707705] boot 2024-03-11T20:44:40.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:44:39 smithi023 bash[29146]: cluster 2024-03-11T20:44:38.874342+0000 mon.a (mon.0) 321 : cluster 0 osdmap e13: 2 total, 2 up, 2 in 2024-03-11T20:44:40.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:44:39 smithi023 bash[29146]: audit 2024-03-11T20:44:38.875236+0000 mon.a (mon.0) 322 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T20:44:40.545 DEBUG:teuthology.orchestra.run.smithi023:osd.1> sudo journalctl -f -n 0 -u ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@osd.1.service 2024-03-11T20:44:40.548 INFO:tasks.cephadm:Deploying osd.2 on smithi023 with /dev/vg_nvme/lv_2... 2024-03-11T20:44:40.548 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-11T20:44:41.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:44:40 smithi146 bash[23809]: cluster 2024-03-11T20:44:39.352893+0000 mgr.y (mgr.14152) 136 : cluster 0 pgmap v102: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:44:41.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:44:40 smithi146 bash[23809]: cluster 2024-03-11T20:44:39.910405+0000 mon.a (mon.0) 323 : cluster 0 osdmap e14: 2 total, 2 up, 2 in 2024-03-11T20:44:41.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:44:40 smithi146 bash[23809]: audit 2024-03-11T20:44:39.919940+0000 mon.a (mon.0) 324 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:44:41.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:44:40 smithi146 bash[23809]: audit 2024-03-11T20:44:39.929305+0000 mon.a (mon.0) 325 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:44:41.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:44:40 smithi146 bash[23809]: audit 2024-03-11T20:44:39.930449+0000 mon.a (mon.0) 326 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:44:41.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:44:40 smithi023 bash[21372]: cluster 2024-03-11T20:44:39.352893+0000 mgr.y (mgr.14152) 136 : cluster 0 pgmap v102: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:44:41.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:44:40 smithi023 bash[21372]: cluster 2024-03-11T20:44:39.910405+0000 mon.a (mon.0) 323 : cluster 0 osdmap e14: 2 total, 2 up, 2 in 2024-03-11T20:44:41.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:44:40 smithi023 bash[21372]: audit 2024-03-11T20:44:39.919940+0000 mon.a (mon.0) 324 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:44:41.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:44:40 smithi023 bash[21372]: audit 2024-03-11T20:44:39.929305+0000 mon.a (mon.0) 325 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:44:41.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:44:40 smithi023 bash[21372]: audit 2024-03-11T20:44:39.930449+0000 mon.a (mon.0) 326 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:44:41.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:44:40 smithi023 bash[29146]: cluster 2024-03-11T20:44:39.352893+0000 mgr.y (mgr.14152) 136 : cluster 0 pgmap v102: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:44:41.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:44:40 smithi023 bash[29146]: cluster 2024-03-11T20:44:39.910405+0000 mon.a (mon.0) 323 : cluster 0 osdmap e14: 2 total, 2 up, 2 in 2024-03-11T20:44:41.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:44:40 smithi023 bash[29146]: audit 2024-03-11T20:44:39.919940+0000 mon.a (mon.0) 324 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:44:41.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:44:40 smithi023 bash[29146]: audit 2024-03-11T20:44:39.929305+0000 mon.a (mon.0) 325 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:44:41.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:44:40 smithi023 bash[29146]: audit 2024-03-11T20:44:39.930449+0000 mon.a (mon.0) 326 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:44:43.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:44:42 smithi146 bash[23809]: cluster 2024-03-11T20:44:41.353415+0000 mgr.y (mgr.14152) 137 : cluster 0 pgmap v104: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:44:43.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:44:42 smithi023 bash[21372]: cluster 2024-03-11T20:44:41.353415+0000 mgr.y (mgr.14152) 137 : cluster 0 pgmap v104: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:44:43.282 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:44:42 smithi023 bash[29146]: cluster 2024-03-11T20:44:41.353415+0000 mgr.y (mgr.14152) 137 : cluster 0 pgmap v104: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:44:44.895 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:44:45.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:44:44 smithi146 bash[23809]: cluster 2024-03-11T20:44:43.353898+0000 mgr.y (mgr.14152) 138 : cluster 0 pgmap v105: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:44:45.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:44:44 smithi023 bash[21372]: cluster 2024-03-11T20:44:43.353898+0000 mgr.y (mgr.14152) 138 : cluster 0 pgmap v105: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:44:45.282 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:44:44 smithi023 bash[29146]: cluster 2024-03-11T20:44:43.353898+0000 mgr.y (mgr.14152) 138 : cluster 0 pgmap v105: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:44:47.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:44:46 smithi146 bash[23809]: cluster 2024-03-11T20:44:45.354238+0000 mgr.y (mgr.14152) 139 : cluster 0 pgmap v106: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:44:47.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:44:46 smithi023 bash[21372]: cluster 2024-03-11T20:44:45.354238+0000 mgr.y (mgr.14152) 139 : cluster 0 pgmap v106: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:44:47.282 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:44:46 smithi023 bash[29146]: cluster 2024-03-11T20:44:45.354238+0000 mgr.y (mgr.14152) 139 : cluster 0 pgmap v106: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:44:48.600 INFO:teuthology.orchestra.run.smithi023.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-11T20:44:48.600 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-11T20:44:48.600 INFO:teuthology.orchestra.run.smithi023.stderr: stderr: 10+0 records in 2024-03-11T20:44:48.600 INFO:teuthology.orchestra.run.smithi023.stderr:10+0 records out 2024-03-11T20:44:48.600 INFO:teuthology.orchestra.run.smithi023.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00948925 s, 1.1 GB/s 2024-03-11T20:44:48.600 INFO:teuthology.orchestra.run.smithi023.stderr:--> Zapping successful for: 2024-03-11T20:44:49.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:44:48 smithi146 bash[23809]: cluster 2024-03-11T20:44:47.354651+0000 mgr.y (mgr.14152) 140 : cluster 0 pgmap v107: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:44:49.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:44:48 smithi023 bash[21372]: cluster 2024-03-11T20:44:47.354651+0000 mgr.y (mgr.14152) 140 : cluster 0 pgmap v107: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:44:49.282 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:44:48 smithi023 bash[29146]: cluster 2024-03-11T20:44:47.354651+0000 mgr.y (mgr.14152) 140 : cluster 0 pgmap v107: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:44:49.373 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi023:vg_nvme/lv_2 2024-03-11T20:44:51.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:44:50 smithi146 bash[23809]: cluster 2024-03-11T20:44:49.104313+0000 osd.0 (osd.0) 3 : cluster 0 purged_snaps scrub starts 2024-03-11T20:44:51.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:44:50 smithi146 bash[23809]: cluster 2024-03-11T20:44:49.104410+0000 osd.0 (osd.0) 4 : cluster 0 purged_snaps scrub ok 2024-03-11T20:44:51.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:44:50 smithi146 bash[23809]: cluster 2024-03-11T20:44:49.355080+0000 mgr.y (mgr.14152) 141 : cluster 0 pgmap v108: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:44:51.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:44:50 smithi146 bash[23809]: cephadm 2024-03-11T20:44:49.824416+0000 mgr.y (mgr.14152) 142 : cephadm 1 Detected new or changed devices on smithi023 2024-03-11T20:44:51.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:44:50 smithi146 bash[23809]: audit 2024-03-11T20:44:49.841250+0000 mon.a (mon.0) 327 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:44:51.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:44:50 smithi146 bash[23809]: audit 2024-03-11T20:44:49.858653+0000 mon.a (mon.0) 328 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:44:51.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:44:50 smithi146 bash[23809]: audit 2024-03-11T20:44:49.861332+0000 mon.a (mon.0) 329 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:44:51.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:44:50 smithi146 bash[23809]: audit 2024-03-11T20:44:49.863408+0000 mon.a (mon.0) 330 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:44:51.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:44:50 smithi146 bash[23809]: cephadm 2024-03-11T20:44:49.865018+0000 mgr.y (mgr.14152) 143 : cephadm 1 Adjusting osd_memory_target on smithi023 to 8120M 2024-03-11T20:44:51.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:44:50 smithi146 bash[23809]: audit 2024-03-11T20:44:49.871256+0000 mon.a (mon.0) 331 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:44:51.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:44:50 smithi146 bash[23809]: audit 2024-03-11T20:44:49.875743+0000 mon.a (mon.0) 332 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:44:51.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:44:50 smithi146 bash[23809]: audit 2024-03-11T20:44:49.877640+0000 mon.a (mon.0) 333 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:44:51.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:44:50 smithi146 bash[23809]: audit 2024-03-11T20:44:49.887501+0000 mon.a (mon.0) 334 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:44:51.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:44:50 smithi146 bash[23809]: cluster 2024-03-11T20:44:49.944927+0000 mon.a (mon.0) 335 : cluster 0 osdmap e15: 2 total, 2 up, 2 in 2024-03-11T20:44:51.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:44:50 smithi023 bash[21372]: cluster 2024-03-11T20:44:49.104313+0000 osd.0 (osd.0) 3 : cluster 0 purged_snaps scrub starts 2024-03-11T20:44:51.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:44:50 smithi023 bash[21372]: cluster 2024-03-11T20:44:49.104410+0000 osd.0 (osd.0) 4 : cluster 0 purged_snaps scrub ok 2024-03-11T20:44:51.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:44:50 smithi023 bash[21372]: cluster 2024-03-11T20:44:49.355080+0000 mgr.y (mgr.14152) 141 : cluster 0 pgmap v108: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:44:51.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:44:50 smithi023 bash[21372]: cephadm 2024-03-11T20:44:49.824416+0000 mgr.y (mgr.14152) 142 : cephadm 1 Detected new or changed devices on smithi023 2024-03-11T20:44:51.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:44:50 smithi023 bash[21372]: audit 2024-03-11T20:44:49.841250+0000 mon.a (mon.0) 327 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:44:51.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:44:50 smithi023 bash[21372]: audit 2024-03-11T20:44:49.858653+0000 mon.a (mon.0) 328 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:44:51.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:44:50 smithi023 bash[21372]: audit 2024-03-11T20:44:49.861332+0000 mon.a (mon.0) 329 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:44:51.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:44:50 smithi023 bash[21372]: audit 2024-03-11T20:44:49.863408+0000 mon.a (mon.0) 330 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:44:51.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:44:50 smithi023 bash[21372]: cephadm 2024-03-11T20:44:49.865018+0000 mgr.y (mgr.14152) 143 : cephadm 1 Adjusting osd_memory_target on smithi023 to 8120M 2024-03-11T20:44:51.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:44:50 smithi023 bash[21372]: audit 2024-03-11T20:44:49.871256+0000 mon.a (mon.0) 331 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:44:51.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:44:50 smithi023 bash[21372]: audit 2024-03-11T20:44:49.875743+0000 mon.a (mon.0) 332 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:44:51.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:44:50 smithi023 bash[21372]: audit 2024-03-11T20:44:49.877640+0000 mon.a (mon.0) 333 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:44:51.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:44:50 smithi023 bash[21372]: audit 2024-03-11T20:44:49.887501+0000 mon.a (mon.0) 334 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:44:51.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:44:50 smithi023 bash[21372]: cluster 2024-03-11T20:44:49.944927+0000 mon.a (mon.0) 335 : cluster 0 osdmap e15: 2 total, 2 up, 2 in 2024-03-11T20:44:51.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:44:50 smithi023 bash[29146]: cluster 2024-03-11T20:44:49.104313+0000 osd.0 (osd.0) 3 : cluster 0 purged_snaps scrub starts 2024-03-11T20:44:51.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:44:50 smithi023 bash[29146]: cluster 2024-03-11T20:44:49.104410+0000 osd.0 (osd.0) 4 : cluster 0 purged_snaps scrub ok 2024-03-11T20:44:51.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:44:50 smithi023 bash[29146]: cluster 2024-03-11T20:44:49.355080+0000 mgr.y (mgr.14152) 141 : cluster 0 pgmap v108: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:44:51.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:44:50 smithi023 bash[29146]: cephadm 2024-03-11T20:44:49.824416+0000 mgr.y (mgr.14152) 142 : cephadm 1 Detected new or changed devices on smithi023 2024-03-11T20:44:51.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:44:50 smithi023 bash[29146]: audit 2024-03-11T20:44:49.841250+0000 mon.a (mon.0) 327 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:44:51.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:44:50 smithi023 bash[29146]: audit 2024-03-11T20:44:49.858653+0000 mon.a (mon.0) 328 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:44:51.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:44:50 smithi023 bash[29146]: audit 2024-03-11T20:44:49.861332+0000 mon.a (mon.0) 329 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:44:51.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:44:50 smithi023 bash[29146]: audit 2024-03-11T20:44:49.863408+0000 mon.a (mon.0) 330 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:44:51.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:44:50 smithi023 bash[29146]: cephadm 2024-03-11T20:44:49.865018+0000 mgr.y (mgr.14152) 143 : cephadm 1 Adjusting osd_memory_target on smithi023 to 8120M 2024-03-11T20:44:51.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:44:50 smithi023 bash[29146]: audit 2024-03-11T20:44:49.871256+0000 mon.a (mon.0) 331 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:44:51.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:44:50 smithi023 bash[29146]: audit 2024-03-11T20:44:49.875743+0000 mon.a (mon.0) 332 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:44:51.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:44:50 smithi023 bash[29146]: audit 2024-03-11T20:44:49.877640+0000 mon.a (mon.0) 333 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:44:51.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:44:50 smithi023 bash[29146]: audit 2024-03-11T20:44:49.887501+0000 mon.a (mon.0) 334 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:44:51.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:44:50 smithi023 bash[29146]: cluster 2024-03-11T20:44:49.944927+0000 mon.a (mon.0) 335 : cluster 0 osdmap e15: 2 total, 2 up, 2 in 2024-03-11T20:44:53.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:44:52 smithi146 bash[23809]: cluster 2024-03-11T20:44:51.355577+0000 mgr.y (mgr.14152) 144 : cluster 0 pgmap v110: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:44:53.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:44:52 smithi023 bash[21372]: cluster 2024-03-11T20:44:51.355577+0000 mgr.y (mgr.14152) 144 : cluster 0 pgmap v110: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:44:53.282 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:44:52 smithi023 bash[29146]: cluster 2024-03-11T20:44:51.355577+0000 mgr.y (mgr.14152) 144 : cluster 0 pgmap v110: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:44:54.126 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:44:55.179 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:44:54 smithi023 bash[21372]: cluster 2024-03-11T20:44:53.355978+0000 mgr.y (mgr.14152) 145 : cluster 0 pgmap v111: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:44:55.179 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:44:54 smithi023 bash[29146]: cluster 2024-03-11T20:44:53.355978+0000 mgr.y (mgr.14152) 145 : cluster 0 pgmap v111: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:44:55.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:44:54 smithi146 bash[23809]: cluster 2024-03-11T20:44:53.355978+0000 mgr.y (mgr.14152) 145 : cluster 0 pgmap v111: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:44:57.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:44:56 smithi146 bash[23809]: cluster 2024-03-11T20:44:55.356465+0000 mgr.y (mgr.14152) 146 : cluster 0 pgmap v112: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:44:57.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:44:56 smithi146 bash[23809]: audit 2024-03-11T20:44:56.726376+0000 mon.a (mon.0) 336 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T20:44:57.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:44:56 smithi146 bash[23809]: audit 2024-03-11T20:44:56.732134+0000 mon.a (mon.0) 337 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T20:44:57.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:44:56 smithi146 bash[23809]: audit 2024-03-11T20:44:56.733075+0000 mon.a (mon.0) 338 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:44:57.282 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:44:56 smithi023 bash[29146]: cluster 2024-03-11T20:44:55.356465+0000 mgr.y (mgr.14152) 146 : cluster 0 pgmap v112: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:44:57.282 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:44:56 smithi023 bash[29146]: audit 2024-03-11T20:44:56.726376+0000 mon.a (mon.0) 336 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T20:44:57.282 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:44:56 smithi023 bash[29146]: audit 2024-03-11T20:44:56.732134+0000 mon.a (mon.0) 337 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T20:44:57.282 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:44:56 smithi023 bash[29146]: audit 2024-03-11T20:44:56.733075+0000 mon.a (mon.0) 338 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:44:57.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:44:56 smithi023 bash[21372]: cluster 2024-03-11T20:44:55.356465+0000 mgr.y (mgr.14152) 146 : cluster 0 pgmap v112: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:44:57.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:44:56 smithi023 bash[21372]: audit 2024-03-11T20:44:56.726376+0000 mon.a (mon.0) 336 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T20:44:57.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:44:56 smithi023 bash[21372]: audit 2024-03-11T20:44:56.732134+0000 mon.a (mon.0) 337 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T20:44:57.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:44:56 smithi023 bash[21372]: audit 2024-03-11T20:44:56.733075+0000 mon.a (mon.0) 338 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:44:58.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:44:57 smithi146 bash[23809]: audit 2024-03-11T20:44:56.719620+0000 mgr.y (mgr.14152) 147 : audit 0 from='client.24164 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:44:58.282 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:44:57 smithi023 bash[29146]: audit 2024-03-11T20:44:56.719620+0000 mgr.y (mgr.14152) 147 : audit 0 from='client.24164 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:44:58.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:44:57 smithi023 bash[21372]: audit 2024-03-11T20:44:56.719620+0000 mgr.y (mgr.14152) 147 : audit 0 from='client.24164 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:44:59.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:44:58 smithi146 bash[23809]: cluster 2024-03-11T20:44:57.356943+0000 mgr.y (mgr.14152) 148 : cluster 0 pgmap v113: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:44:59.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:44:58 smithi023 bash[21372]: cluster 2024-03-11T20:44:57.356943+0000 mgr.y (mgr.14152) 148 : cluster 0 pgmap v113: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:44:59.282 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:44:58 smithi023 bash[29146]: cluster 2024-03-11T20:44:57.356943+0000 mgr.y (mgr.14152) 148 : cluster 0 pgmap v113: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:45:01.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:00 smithi146 bash[23809]: cluster 2024-03-11T20:44:59.357501+0000 mgr.y (mgr.14152) 149 : cluster 0 pgmap v114: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:45:01.282 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:00 smithi023 bash[29146]: cluster 2024-03-11T20:44:59.357501+0000 mgr.y (mgr.14152) 149 : cluster 0 pgmap v114: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:45:01.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:00 smithi023 bash[21372]: cluster 2024-03-11T20:44:59.357501+0000 mgr.y (mgr.14152) 149 : cluster 0 pgmap v114: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:45:03.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:02 smithi146 bash[23809]: cluster 2024-03-11T20:45:01.358101+0000 mgr.y (mgr.14152) 150 : cluster 0 pgmap v115: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:45:03.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:02 smithi023 bash[21372]: cluster 2024-03-11T20:45:01.358101+0000 mgr.y (mgr.14152) 150 : cluster 0 pgmap v115: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:45:03.273 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:02 smithi023 bash[29146]: cluster 2024-03-11T20:45:01.358101+0000 mgr.y (mgr.14152) 150 : cluster 0 pgmap v115: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:45:05.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:04 smithi146 bash[23809]: cluster 2024-03-11T20:45:03.358690+0000 mgr.y (mgr.14152) 151 : cluster 0 pgmap v116: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:45:05.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:04 smithi023 bash[21372]: cluster 2024-03-11T20:45:03.358690+0000 mgr.y (mgr.14152) 151 : cluster 0 pgmap v116: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:45:05.282 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:04 smithi023 bash[29146]: cluster 2024-03-11T20:45:03.358690+0000 mgr.y (mgr.14152) 151 : cluster 0 pgmap v116: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:45:06.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:05 smithi146 bash[23809]: audit 2024-03-11T20:45:05.019829+0000 mon.a (mon.0) 339 : audit 1 from='client.? 172.21.15.23:0/1756528687' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dd85d52a-df48-4b03-8612-cecfd82b5987"}]: dispatch 2024-03-11T20:45:06.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:05 smithi146 bash[23809]: audit 2024-03-11T20:45:05.026827+0000 mon.a (mon.0) 340 : audit 1 from='client.? 172.21.15.23:0/1756528687' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dd85d52a-df48-4b03-8612-cecfd82b5987"}]': finished 2024-03-11T20:45:06.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:05 smithi146 bash[23809]: cluster 2024-03-11T20:45:05.038625+0000 mon.a (mon.0) 341 : cluster 0 osdmap e16: 3 total, 2 up, 3 in 2024-03-11T20:45:06.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:05 smithi146 bash[23809]: audit 2024-03-11T20:45:05.038962+0000 mon.a (mon.0) 342 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T20:45:06.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:05 smithi146 bash[23809]: audit 2024-03-11T20:45:05.797796+0000 mon.a (mon.0) 343 : audit 0 from='client.? 172.21.15.23:0/2537574242' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T20:45:06.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:05 smithi023 bash[21372]: audit 2024-03-11T20:45:05.019829+0000 mon.a (mon.0) 339 : audit 1 from='client.? 172.21.15.23:0/1756528687' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dd85d52a-df48-4b03-8612-cecfd82b5987"}]: dispatch 2024-03-11T20:45:06.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:05 smithi023 bash[21372]: audit 2024-03-11T20:45:05.026827+0000 mon.a (mon.0) 340 : audit 1 from='client.? 172.21.15.23:0/1756528687' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dd85d52a-df48-4b03-8612-cecfd82b5987"}]': finished 2024-03-11T20:45:06.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:05 smithi023 bash[21372]: cluster 2024-03-11T20:45:05.038625+0000 mon.a (mon.0) 341 : cluster 0 osdmap e16: 3 total, 2 up, 3 in 2024-03-11T20:45:06.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:05 smithi023 bash[21372]: audit 2024-03-11T20:45:05.038962+0000 mon.a (mon.0) 342 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T20:45:06.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:05 smithi023 bash[21372]: audit 2024-03-11T20:45:05.797796+0000 mon.a (mon.0) 343 : audit 0 from='client.? 172.21.15.23:0/2537574242' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T20:45:06.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:05 smithi023 bash[29146]: audit 2024-03-11T20:45:05.019829+0000 mon.a (mon.0) 339 : audit 1 from='client.? 172.21.15.23:0/1756528687' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dd85d52a-df48-4b03-8612-cecfd82b5987"}]: dispatch 2024-03-11T20:45:06.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:05 smithi023 bash[29146]: audit 2024-03-11T20:45:05.026827+0000 mon.a (mon.0) 340 : audit 1 from='client.? 172.21.15.23:0/1756528687' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dd85d52a-df48-4b03-8612-cecfd82b5987"}]': finished 2024-03-11T20:45:06.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:05 smithi023 bash[29146]: cluster 2024-03-11T20:45:05.038625+0000 mon.a (mon.0) 341 : cluster 0 osdmap e16: 3 total, 2 up, 3 in 2024-03-11T20:45:06.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:05 smithi023 bash[29146]: audit 2024-03-11T20:45:05.038962+0000 mon.a (mon.0) 342 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T20:45:06.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:05 smithi023 bash[29146]: audit 2024-03-11T20:45:05.797796+0000 mon.a (mon.0) 343 : audit 0 from='client.? 172.21.15.23:0/2537574242' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T20:45:07.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:07 smithi023 bash[21372]: cluster 2024-03-11T20:45:05.359215+0000 mgr.y (mgr.14152) 152 : cluster 0 pgmap v118: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:45:07.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:07 smithi023 bash[29146]: cluster 2024-03-11T20:45:05.359215+0000 mgr.y (mgr.14152) 152 : cluster 0 pgmap v118: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:45:07.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:07 smithi146 bash[23809]: cluster 2024-03-11T20:45:05.359215+0000 mgr.y (mgr.14152) 152 : cluster 0 pgmap v118: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:45:09.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:09 smithi146 bash[23809]: cluster 2024-03-11T20:45:07.359665+0000 mgr.y (mgr.14152) 153 : cluster 0 pgmap v119: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:45:09.532 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:09 smithi023 bash[21372]: cluster 2024-03-11T20:45:07.359665+0000 mgr.y (mgr.14152) 153 : cluster 0 pgmap v119: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:45:09.532 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:09 smithi023 bash[29146]: cluster 2024-03-11T20:45:07.359665+0000 mgr.y (mgr.14152) 153 : cluster 0 pgmap v119: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:45:10.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:10 smithi146 bash[23809]: cluster 2024-03-11T20:45:09.360078+0000 mgr.y (mgr.14152) 154 : cluster 0 pgmap v120: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:45:10.532 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:10 smithi023 bash[21372]: cluster 2024-03-11T20:45:09.360078+0000 mgr.y (mgr.14152) 154 : cluster 0 pgmap v120: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:45:10.532 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:10 smithi023 bash[29146]: cluster 2024-03-11T20:45:09.360078+0000 mgr.y (mgr.14152) 154 : cluster 0 pgmap v120: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:45:12.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:12 smithi146 bash[23809]: cluster 2024-03-11T20:45:11.360526+0000 mgr.y (mgr.14152) 155 : cluster 0 pgmap v121: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:45:12.782 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:12 smithi023 bash[21372]: cluster 2024-03-11T20:45:11.360526+0000 mgr.y (mgr.14152) 155 : cluster 0 pgmap v121: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:45:12.782 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:12 smithi023 bash[29146]: cluster 2024-03-11T20:45:11.360526+0000 mgr.y (mgr.14152) 155 : cluster 0 pgmap v121: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:45:14.701 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:14 smithi023 bash[29146]: cluster 2024-03-11T20:45:13.360933+0000 mgr.y (mgr.14152) 156 : cluster 0 pgmap v122: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:45:14.701 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:14 smithi023 bash[21372]: cluster 2024-03-11T20:45:13.360933+0000 mgr.y (mgr.14152) 156 : cluster 0 pgmap v122: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:45:14.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:14 smithi146 bash[23809]: cluster 2024-03-11T20:45:13.360933+0000 mgr.y (mgr.14152) 156 : cluster 0 pgmap v122: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:45:16.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:16 smithi146 bash[23809]: cluster 2024-03-11T20:45:15.361349+0000 mgr.y (mgr.14152) 157 : cluster 0 pgmap v123: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:45:16.782 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:16 smithi023 bash[21372]: cluster 2024-03-11T20:45:15.361349+0000 mgr.y (mgr.14152) 157 : cluster 0 pgmap v123: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:45:16.782 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:16 smithi023 bash[29146]: cluster 2024-03-11T20:45:15.361349+0000 mgr.y (mgr.14152) 157 : cluster 0 pgmap v123: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:45:17.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:17 smithi146 bash[23809]: audit 2024-03-11T20:45:17.249436+0000 mon.a (mon.0) 344 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-11T20:45:17.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:17 smithi146 bash[23809]: audit 2024-03-11T20:45:17.251226+0000 mon.a (mon.0) 345 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:45:17.788 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:17 smithi023 bash[21372]: audit 2024-03-11T20:45:17.249436+0000 mon.a (mon.0) 344 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-11T20:45:17.789 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:17 smithi023 bash[21372]: audit 2024-03-11T20:45:17.251226+0000 mon.a (mon.0) 345 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:45:17.789 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:17 smithi023 bash[29146]: audit 2024-03-11T20:45:17.249436+0000 mon.a (mon.0) 344 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-11T20:45:17.789 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:17 smithi023 bash[29146]: audit 2024-03-11T20:45:17.251226+0000 mon.a (mon.0) 345 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:45:18.742 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:18 smithi023 bash[21372]: cephadm 2024-03-11T20:45:17.252866+0000 mgr.y (mgr.14152) 158 : cephadm 1 Deploying daemon osd.2 on smithi023 2024-03-11T20:45:18.743 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:18 smithi023 bash[21372]: cluster 2024-03-11T20:45:17.361815+0000 mgr.y (mgr.14152) 159 : cluster 0 pgmap v124: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:45:18.743 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:18 smithi023 bash[29146]: cephadm 2024-03-11T20:45:17.252866+0000 mgr.y (mgr.14152) 158 : cephadm 1 Deploying daemon osd.2 on smithi023 2024-03-11T20:45:18.743 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:18 smithi023 bash[29146]: cluster 2024-03-11T20:45:17.361815+0000 mgr.y (mgr.14152) 159 : cluster 0 pgmap v124: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:45:18.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:18 smithi146 bash[23809]: cephadm 2024-03-11T20:45:17.252866+0000 mgr.y (mgr.14152) 158 : cephadm 1 Deploying daemon osd.2 on smithi023 2024-03-11T20:45:18.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:18 smithi146 bash[23809]: cluster 2024-03-11T20:45:17.361815+0000 mgr.y (mgr.14152) 159 : cluster 0 pgmap v124: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:45:20.532 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:20 smithi023 bash[21372]: cluster 2024-03-11T20:45:19.362288+0000 mgr.y (mgr.14152) 160 : cluster 0 pgmap v125: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:45:20.533 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:20 smithi023 bash[29146]: cluster 2024-03-11T20:45:19.362288+0000 mgr.y (mgr.14152) 160 : cluster 0 pgmap v125: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:45:20.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:20 smithi146 bash[23809]: cluster 2024-03-11T20:45:19.362288+0000 mgr.y (mgr.14152) 160 : cluster 0 pgmap v125: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:45:21.299 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:45:21 smithi023 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:45:21.300 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:21 smithi023 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:45:21.300 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:21 smithi023 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:45:21.300 INFO:journalctl@ceph.osd.1.smithi023.stdout:Mar 11 20:45:21 smithi023 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:45:21.301 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 11 20:45:21 smithi023 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:45:21.585 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:21 smithi023 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:45:21.585 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:45:21 smithi023 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:45:21.585 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:21 smithi023 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:45:21.586 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 11 20:45:21 smithi023 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:45:21.586 INFO:journalctl@ceph.osd.1.smithi023.stdout:Mar 11 20:45:21 smithi023 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:45:22.782 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:22 smithi023 bash[29146]: cluster 2024-03-11T20:45:21.362579+0000 mgr.y (mgr.14152) 161 : cluster 0 pgmap v126: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:45:22.782 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:22 smithi023 bash[29146]: audit 2024-03-11T20:45:21.586492+0000 mon.a (mon.0) 346 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:45:22.783 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:22 smithi023 bash[29146]: audit 2024-03-11T20:45:21.592291+0000 mon.a (mon.0) 347 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:45:22.783 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:22 smithi023 bash[29146]: audit 2024-03-11T20:45:21.597507+0000 mon.a (mon.0) 348 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:45:22.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:22 smithi023 bash[21372]: cluster 2024-03-11T20:45:21.362579+0000 mgr.y (mgr.14152) 161 : cluster 0 pgmap v126: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:45:22.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:22 smithi023 bash[21372]: audit 2024-03-11T20:45:21.586492+0000 mon.a (mon.0) 346 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:45:22.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:22 smithi023 bash[21372]: audit 2024-03-11T20:45:21.592291+0000 mon.a (mon.0) 347 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:45:22.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:22 smithi023 bash[21372]: audit 2024-03-11T20:45:21.597507+0000 mon.a (mon.0) 348 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:45:23.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:22 smithi146 bash[23809]: cluster 2024-03-11T20:45:21.362579+0000 mgr.y (mgr.14152) 161 : cluster 0 pgmap v126: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:45:23.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:22 smithi146 bash[23809]: audit 2024-03-11T20:45:21.586492+0000 mon.a (mon.0) 346 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:45:23.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:22 smithi146 bash[23809]: audit 2024-03-11T20:45:21.592291+0000 mon.a (mon.0) 347 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:45:23.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:22 smithi146 bash[23809]: audit 2024-03-11T20:45:21.597507+0000 mon.a (mon.0) 348 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:45:25.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:24 smithi146 bash[23809]: cluster 2024-03-11T20:45:23.363000+0000 mgr.y (mgr.14152) 162 : cluster 0 pgmap v127: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:45:25.032 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:24 smithi023 bash[29146]: cluster 2024-03-11T20:45:23.363000+0000 mgr.y (mgr.14152) 162 : cluster 0 pgmap v127: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:45:25.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:24 smithi023 bash[21372]: cluster 2024-03-11T20:45:23.363000+0000 mgr.y (mgr.14152) 162 : cluster 0 pgmap v127: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:45:27.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:26 smithi146 bash[23809]: cluster 2024-03-11T20:45:25.363509+0000 mgr.y (mgr.14152) 163 : cluster 0 pgmap v128: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:45:27.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:26 smithi023 bash[21372]: cluster 2024-03-11T20:45:25.363509+0000 mgr.y (mgr.14152) 163 : cluster 0 pgmap v128: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:45:27.032 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:26 smithi023 bash[29146]: cluster 2024-03-11T20:45:25.363509+0000 mgr.y (mgr.14152) 163 : cluster 0 pgmap v128: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:45:28.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:27 smithi023 bash[21372]: audit 2024-03-11T20:45:26.938155+0000 mon.a (mon.0) 349 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:45:28.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:27 smithi023 bash[21372]: audit 2024-03-11T20:45:26.944556+0000 mon.a (mon.0) 350 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:45:28.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:27 smithi023 bash[21372]: audit 2024-03-11T20:45:27.515583+0000 mon.a (mon.0) 351 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:45:28.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:27 smithi023 bash[21372]: audit 2024-03-11T20:45:27.516406+0000 mon.a (mon.0) 352 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:45:28.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:27 smithi023 bash[21372]: audit 2024-03-11T20:45:27.522778+0000 mon.a (mon.0) 353 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:45:28.033 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:27 smithi023 bash[29146]: audit 2024-03-11T20:45:26.938155+0000 mon.a (mon.0) 349 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:45:28.033 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:27 smithi023 bash[29146]: audit 2024-03-11T20:45:26.944556+0000 mon.a (mon.0) 350 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:45:28.033 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:27 smithi023 bash[29146]: audit 2024-03-11T20:45:27.515583+0000 mon.a (mon.0) 351 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:45:28.033 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:27 smithi023 bash[29146]: audit 2024-03-11T20:45:27.516406+0000 mon.a (mon.0) 352 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:45:28.033 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:27 smithi023 bash[29146]: audit 2024-03-11T20:45:27.522778+0000 mon.a (mon.0) 353 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:45:28.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:27 smithi146 bash[23809]: audit 2024-03-11T20:45:26.938155+0000 mon.a (mon.0) 349 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:45:28.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:27 smithi146 bash[23809]: audit 2024-03-11T20:45:26.944556+0000 mon.a (mon.0) 350 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:45:28.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:27 smithi146 bash[23809]: audit 2024-03-11T20:45:27.515583+0000 mon.a (mon.0) 351 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:45:28.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:27 smithi146 bash[23809]: audit 2024-03-11T20:45:27.516406+0000 mon.a (mon.0) 352 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:45:28.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:27 smithi146 bash[23809]: audit 2024-03-11T20:45:27.522778+0000 mon.a (mon.0) 353 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:45:29.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:28 smithi146 bash[23809]: cluster 2024-03-11T20:45:27.363815+0000 mgr.y (mgr.14152) 164 : cluster 0 pgmap v129: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:45:29.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:28 smithi146 bash[23809]: audit 2024-03-11T20:45:28.193664+0000 mon.c (mon.2) 6 : audit 1 from='osd.2 [v2:172.21.15.23:6818/1770748056,v1:172.21.15.23:6819/1770748056]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-11T20:45:29.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:28 smithi146 bash[23809]: audit 2024-03-11T20:45:28.194047+0000 mon.a (mon.0) 354 : audit 1 from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-11T20:45:29.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:28 smithi023 bash[21372]: cluster 2024-03-11T20:45:27.363815+0000 mgr.y (mgr.14152) 164 : cluster 0 pgmap v129: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:45:29.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:28 smithi023 bash[21372]: audit 2024-03-11T20:45:28.193664+0000 mon.c (mon.2) 6 : audit 1 from='osd.2 [v2:172.21.15.23:6818/1770748056,v1:172.21.15.23:6819/1770748056]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-11T20:45:29.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:28 smithi023 bash[21372]: audit 2024-03-11T20:45:28.194047+0000 mon.a (mon.0) 354 : audit 1 from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-11T20:45:29.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:28 smithi023 bash[29146]: cluster 2024-03-11T20:45:27.363815+0000 mgr.y (mgr.14152) 164 : cluster 0 pgmap v129: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:45:29.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:28 smithi023 bash[29146]: audit 2024-03-11T20:45:28.193664+0000 mon.c (mon.2) 6 : audit 1 from='osd.2 [v2:172.21.15.23:6818/1770748056,v1:172.21.15.23:6819/1770748056]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-11T20:45:29.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:28 smithi023 bash[29146]: audit 2024-03-11T20:45:28.194047+0000 mon.a (mon.0) 354 : audit 1 from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-11T20:45:30.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:29 smithi146 bash[23809]: audit 2024-03-11T20:45:28.951359+0000 mon.a (mon.0) 355 : audit 1 from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-11T20:45:30.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:29 smithi146 bash[23809]: cluster 2024-03-11T20:45:28.954810+0000 mon.a (mon.0) 356 : cluster 0 osdmap e17: 3 total, 2 up, 3 in 2024-03-11T20:45:30.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:29 smithi146 bash[23809]: audit 2024-03-11T20:45:28.955994+0000 mon.a (mon.0) 357 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T20:45:30.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:29 smithi146 bash[23809]: audit 2024-03-11T20:45:28.956920+0000 mon.c (mon.2) 7 : audit 1 from='osd.2 [v2:172.21.15.23:6818/1770748056,v1:172.21.15.23:6819/1770748056]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2024-03-11T20:45:30.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:29 smithi146 bash[23809]: audit 2024-03-11T20:45:28.965776+0000 mon.a (mon.0) 358 : audit 1 from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2024-03-11T20:45:30.275 INFO:teuthology.orchestra.run.smithi023.stdout:Created osd(s) 2 on host 'smithi023' 2024-03-11T20:45:30.282 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:29 smithi023 bash[29146]: audit 2024-03-11T20:45:28.951359+0000 mon.a (mon.0) 355 : audit 1 from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-11T20:45:30.282 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:29 smithi023 bash[29146]: cluster 2024-03-11T20:45:28.954810+0000 mon.a (mon.0) 356 : cluster 0 osdmap e17: 3 total, 2 up, 3 in 2024-03-11T20:45:30.282 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:29 smithi023 bash[29146]: audit 2024-03-11T20:45:28.955994+0000 mon.a (mon.0) 357 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T20:45:30.282 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:29 smithi023 bash[29146]: audit 2024-03-11T20:45:28.956920+0000 mon.c (mon.2) 7 : audit 1 from='osd.2 [v2:172.21.15.23:6818/1770748056,v1:172.21.15.23:6819/1770748056]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2024-03-11T20:45:30.282 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:29 smithi023 bash[29146]: audit 2024-03-11T20:45:28.965776+0000 mon.a (mon.0) 358 : audit 1 from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2024-03-11T20:45:30.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:29 smithi023 bash[21372]: audit 2024-03-11T20:45:28.951359+0000 mon.a (mon.0) 355 : audit 1 from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-11T20:45:30.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:29 smithi023 bash[21372]: cluster 2024-03-11T20:45:28.954810+0000 mon.a (mon.0) 356 : cluster 0 osdmap e17: 3 total, 2 up, 3 in 2024-03-11T20:45:30.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:29 smithi023 bash[21372]: audit 2024-03-11T20:45:28.955994+0000 mon.a (mon.0) 357 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T20:45:30.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:29 smithi023 bash[21372]: audit 2024-03-11T20:45:28.956920+0000 mon.c (mon.2) 7 : audit 1 from='osd.2 [v2:172.21.15.23:6818/1770748056,v1:172.21.15.23:6819/1770748056]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2024-03-11T20:45:30.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:29 smithi023 bash[21372]: audit 2024-03-11T20:45:28.965776+0000 mon.a (mon.0) 358 : audit 1 from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2024-03-11T20:45:30.925 DEBUG:teuthology.orchestra.run.smithi023:osd.2> sudo journalctl -f -n 0 -u ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@osd.2.service 2024-03-11T20:45:30.928 INFO:tasks.cephadm:Deploying osd.3 on smithi023 with /dev/vg_nvme/lv_1... 2024-03-11T20:45:30.928 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-11T20:45:31.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:30 smithi146 bash[23809]: cluster 2024-03-11T20:45:29.364063+0000 mgr.y (mgr.14152) 165 : cluster 0 pgmap v131: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:45:31.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:30 smithi146 bash[23809]: audit 2024-03-11T20:45:29.961633+0000 mon.a (mon.0) 359 : audit 1 from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]': finished 2024-03-11T20:45:31.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:30 smithi146 bash[23809]: cluster 2024-03-11T20:45:29.973287+0000 mon.a (mon.0) 360 : cluster 0 osdmap e18: 3 total, 2 up, 3 in 2024-03-11T20:45:31.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:30 smithi146 bash[23809]: audit 2024-03-11T20:45:29.974422+0000 mon.a (mon.0) 361 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T20:45:31.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:30 smithi146 bash[23809]: audit 2024-03-11T20:45:29.982090+0000 mon.a (mon.0) 362 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T20:45:31.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:30 smithi146 bash[23809]: audit 2024-03-11T20:45:30.221282+0000 mon.a (mon.0) 363 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:45:31.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:30 smithi146 bash[23809]: audit 2024-03-11T20:45:30.234441+0000 mon.a (mon.0) 364 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:45:31.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:30 smithi146 bash[23809]: audit 2024-03-11T20:45:30.244873+0000 mon.a (mon.0) 365 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:45:31.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:30 smithi146 bash[23809]: audit 2024-03-11T20:45:30.377248+0000 mon.a (mon.0) 366 : audit 1 from='osd.2 ' entity='osd.2' 2024-03-11T20:45:31.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:30 smithi146 bash[23809]: audit 2024-03-11T20:45:30.981191+0000 mon.a (mon.0) 367 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T20:45:31.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:30 smithi023 bash[21372]: cluster 2024-03-11T20:45:29.364063+0000 mgr.y (mgr.14152) 165 : cluster 0 pgmap v131: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:45:31.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:30 smithi023 bash[21372]: audit 2024-03-11T20:45:29.961633+0000 mon.a (mon.0) 359 : audit 1 from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]': finished 2024-03-11T20:45:31.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:30 smithi023 bash[21372]: cluster 2024-03-11T20:45:29.973287+0000 mon.a (mon.0) 360 : cluster 0 osdmap e18: 3 total, 2 up, 3 in 2024-03-11T20:45:31.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:30 smithi023 bash[21372]: audit 2024-03-11T20:45:29.974422+0000 mon.a (mon.0) 361 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T20:45:31.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:30 smithi023 bash[21372]: audit 2024-03-11T20:45:29.982090+0000 mon.a (mon.0) 362 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T20:45:31.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:30 smithi023 bash[21372]: audit 2024-03-11T20:45:30.221282+0000 mon.a (mon.0) 363 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:45:31.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:30 smithi023 bash[21372]: audit 2024-03-11T20:45:30.234441+0000 mon.a (mon.0) 364 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:45:31.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:30 smithi023 bash[21372]: audit 2024-03-11T20:45:30.244873+0000 mon.a (mon.0) 365 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:45:31.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:30 smithi023 bash[21372]: audit 2024-03-11T20:45:30.377248+0000 mon.a (mon.0) 366 : audit 1 from='osd.2 ' entity='osd.2' 2024-03-11T20:45:31.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:30 smithi023 bash[21372]: audit 2024-03-11T20:45:30.981191+0000 mon.a (mon.0) 367 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T20:45:31.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:30 smithi023 bash[29146]: cluster 2024-03-11T20:45:29.364063+0000 mgr.y (mgr.14152) 165 : cluster 0 pgmap v131: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:45:31.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:30 smithi023 bash[29146]: audit 2024-03-11T20:45:29.961633+0000 mon.a (mon.0) 359 : audit 1 from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]': finished 2024-03-11T20:45:31.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:30 smithi023 bash[29146]: cluster 2024-03-11T20:45:29.973287+0000 mon.a (mon.0) 360 : cluster 0 osdmap e18: 3 total, 2 up, 3 in 2024-03-11T20:45:31.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:30 smithi023 bash[29146]: audit 2024-03-11T20:45:29.974422+0000 mon.a (mon.0) 361 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T20:45:31.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:30 smithi023 bash[29146]: audit 2024-03-11T20:45:29.982090+0000 mon.a (mon.0) 362 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T20:45:31.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:30 smithi023 bash[29146]: audit 2024-03-11T20:45:30.221282+0000 mon.a (mon.0) 363 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:45:31.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:30 smithi023 bash[29146]: audit 2024-03-11T20:45:30.234441+0000 mon.a (mon.0) 364 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:45:31.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:30 smithi023 bash[29146]: audit 2024-03-11T20:45:30.244873+0000 mon.a (mon.0) 365 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:45:31.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:30 smithi023 bash[29146]: audit 2024-03-11T20:45:30.377248+0000 mon.a (mon.0) 366 : audit 1 from='osd.2 ' entity='osd.2' 2024-03-11T20:45:31.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:30 smithi023 bash[29146]: audit 2024-03-11T20:45:30.981191+0000 mon.a (mon.0) 367 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T20:45:32.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:31 smithi146 bash[23809]: cluster 2024-03-11T20:45:29.238542+0000 osd.2 (osd.2) 1 : cluster 0 purged_snaps scrub starts 2024-03-11T20:45:32.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:31 smithi146 bash[23809]: cluster 2024-03-11T20:45:29.238590+0000 osd.2 (osd.2) 2 : cluster 0 purged_snaps scrub ok 2024-03-11T20:45:32.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:31 smithi146 bash[23809]: cluster 2024-03-11T20:45:31.392622+0000 mon.a (mon.0) 368 : cluster 1 osd.2 [v2:172.21.15.23:6818/1770748056,v1:172.21.15.23:6819/1770748056] boot 2024-03-11T20:45:32.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:31 smithi146 bash[23809]: cluster 2024-03-11T20:45:31.392687+0000 mon.a (mon.0) 369 : cluster 0 osdmap e19: 3 total, 3 up, 3 in 2024-03-11T20:45:32.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:31 smithi146 bash[23809]: audit 2024-03-11T20:45:31.392870+0000 mon.a (mon.0) 370 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T20:45:32.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:31 smithi146 bash[23809]: audit 2024-03-11T20:45:31.510026+0000 mon.a (mon.0) 371 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:45:32.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:31 smithi023 bash[21372]: cluster 2024-03-11T20:45:29.238542+0000 osd.2 (osd.2) 1 : cluster 0 purged_snaps scrub starts 2024-03-11T20:45:32.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:31 smithi023 bash[21372]: cluster 2024-03-11T20:45:29.238590+0000 osd.2 (osd.2) 2 : cluster 0 purged_snaps scrub ok 2024-03-11T20:45:32.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:31 smithi023 bash[21372]: cluster 2024-03-11T20:45:31.392622+0000 mon.a (mon.0) 368 : cluster 1 osd.2 [v2:172.21.15.23:6818/1770748056,v1:172.21.15.23:6819/1770748056] boot 2024-03-11T20:45:32.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:31 smithi023 bash[21372]: cluster 2024-03-11T20:45:31.392687+0000 mon.a (mon.0) 369 : cluster 0 osdmap e19: 3 total, 3 up, 3 in 2024-03-11T20:45:32.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:31 smithi023 bash[21372]: audit 2024-03-11T20:45:31.392870+0000 mon.a (mon.0) 370 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T20:45:32.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:31 smithi023 bash[21372]: audit 2024-03-11T20:45:31.510026+0000 mon.a (mon.0) 371 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:45:32.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:31 smithi023 bash[29146]: cluster 2024-03-11T20:45:29.238542+0000 osd.2 (osd.2) 1 : cluster 0 purged_snaps scrub starts 2024-03-11T20:45:32.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:31 smithi023 bash[29146]: cluster 2024-03-11T20:45:29.238590+0000 osd.2 (osd.2) 2 : cluster 0 purged_snaps scrub ok 2024-03-11T20:45:32.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:31 smithi023 bash[29146]: cluster 2024-03-11T20:45:31.392622+0000 mon.a (mon.0) 368 : cluster 1 osd.2 [v2:172.21.15.23:6818/1770748056,v1:172.21.15.23:6819/1770748056] boot 2024-03-11T20:45:32.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:31 smithi023 bash[29146]: cluster 2024-03-11T20:45:31.392687+0000 mon.a (mon.0) 369 : cluster 0 osdmap e19: 3 total, 3 up, 3 in 2024-03-11T20:45:32.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:31 smithi023 bash[29146]: audit 2024-03-11T20:45:31.392870+0000 mon.a (mon.0) 370 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T20:45:32.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:31 smithi023 bash[29146]: audit 2024-03-11T20:45:31.510026+0000 mon.a (mon.0) 371 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:45:33.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:33 smithi023 bash[21372]: cluster 2024-03-11T20:45:31.364472+0000 mgr.y (mgr.14152) 166 : cluster 0 pgmap v133: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:45:33.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:33 smithi023 bash[21372]: audit 2024-03-11T20:45:32.392509+0000 mon.a (mon.0) 372 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-11T20:45:33.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:33 smithi023 bash[21372]: cluster 2024-03-11T20:45:32.402816+0000 mon.a (mon.0) 373 : cluster 0 osdmap e20: 3 total, 3 up, 3 in 2024-03-11T20:45:33.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:33 smithi023 bash[21372]: audit 2024-03-11T20:45:32.405075+0000 mon.a (mon.0) 374 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:45:33.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:33 smithi023 bash[29146]: cluster 2024-03-11T20:45:31.364472+0000 mgr.y (mgr.14152) 166 : cluster 0 pgmap v133: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:45:33.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:33 smithi023 bash[29146]: audit 2024-03-11T20:45:32.392509+0000 mon.a (mon.0) 372 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-11T20:45:33.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:33 smithi023 bash[29146]: cluster 2024-03-11T20:45:32.402816+0000 mon.a (mon.0) 373 : cluster 0 osdmap e20: 3 total, 3 up, 3 in 2024-03-11T20:45:33.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:33 smithi023 bash[29146]: audit 2024-03-11T20:45:32.405075+0000 mon.a (mon.0) 374 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:45:33.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:33 smithi146 bash[23809]: cluster 2024-03-11T20:45:31.364472+0000 mgr.y (mgr.14152) 166 : cluster 0 pgmap v133: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:45:33.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:33 smithi146 bash[23809]: audit 2024-03-11T20:45:32.392509+0000 mon.a (mon.0) 372 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-11T20:45:33.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:33 smithi146 bash[23809]: cluster 2024-03-11T20:45:32.402816+0000 mon.a (mon.0) 373 : cluster 0 osdmap e20: 3 total, 3 up, 3 in 2024-03-11T20:45:33.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:33 smithi146 bash[23809]: audit 2024-03-11T20:45:32.405075+0000 mon.a (mon.0) 374 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:45:34.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:34 smithi146 bash[23809]: cluster 2024-03-11T20:45:33.365135+0000 mgr.y (mgr.14152) 167 : cluster 0 pgmap v136: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-11T20:45:34.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:34 smithi146 bash[23809]: audit 2024-03-11T20:45:33.396219+0000 mon.a (mon.0) 375 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-11T20:45:34.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:34 smithi146 bash[23809]: cluster 2024-03-11T20:45:33.405405+0000 mon.a (mon.0) 376 : cluster 0 osdmap e21: 3 total, 3 up, 3 in 2024-03-11T20:45:34.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:34 smithi146 bash[23809]: cluster 2024-03-11T20:45:34.022059+0000 mon.a (mon.0) 377 : cluster 3 Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:45:34.782 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:34 smithi023 bash[29146]: cluster 2024-03-11T20:45:33.365135+0000 mgr.y (mgr.14152) 167 : cluster 0 pgmap v136: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-11T20:45:34.782 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:34 smithi023 bash[29146]: audit 2024-03-11T20:45:33.396219+0000 mon.a (mon.0) 375 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-11T20:45:34.782 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:34 smithi023 bash[29146]: cluster 2024-03-11T20:45:33.405405+0000 mon.a (mon.0) 376 : cluster 0 osdmap e21: 3 total, 3 up, 3 in 2024-03-11T20:45:34.783 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:34 smithi023 bash[29146]: cluster 2024-03-11T20:45:34.022059+0000 mon.a (mon.0) 377 : cluster 3 Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:45:34.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:34 smithi023 bash[21372]: cluster 2024-03-11T20:45:33.365135+0000 mgr.y (mgr.14152) 167 : cluster 0 pgmap v136: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-11T20:45:34.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:34 smithi023 bash[21372]: audit 2024-03-11T20:45:33.396219+0000 mon.a (mon.0) 375 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-11T20:45:34.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:34 smithi023 bash[21372]: cluster 2024-03-11T20:45:33.405405+0000 mon.a (mon.0) 376 : cluster 0 osdmap e21: 3 total, 3 up, 3 in 2024-03-11T20:45:34.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:34 smithi023 bash[21372]: cluster 2024-03-11T20:45:34.022059+0000 mon.a (mon.0) 377 : cluster 3 Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:45:35.238 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:45:35.532 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:35 smithi023 bash[21372]: cluster 2024-03-11T20:45:34.425422+0000 mon.a (mon.0) 378 : cluster 0 osdmap e22: 3 total, 3 up, 3 in 2024-03-11T20:45:35.532 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:35 smithi023 bash[21372]: audit 2024-03-11T20:45:34.558009+0000 mon.a (mon.0) 379 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:45:35.533 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:35 smithi023 bash[29146]: cluster 2024-03-11T20:45:34.425422+0000 mon.a (mon.0) 378 : cluster 0 osdmap e22: 3 total, 3 up, 3 in 2024-03-11T20:45:35.533 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:35 smithi023 bash[29146]: audit 2024-03-11T20:45:34.558009+0000 mon.a (mon.0) 379 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:45:35.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:35 smithi146 bash[23809]: cluster 2024-03-11T20:45:34.425422+0000 mon.a (mon.0) 378 : cluster 0 osdmap e22: 3 total, 3 up, 3 in 2024-03-11T20:45:35.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:35 smithi146 bash[23809]: audit 2024-03-11T20:45:34.558009+0000 mon.a (mon.0) 379 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:45:37.532 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:37 smithi023 bash[29146]: cluster 2024-03-11T20:45:35.365655+0000 mgr.y (mgr.14152) 168 : cluster 0 pgmap v139: 1 pgs: 1 active+clean; 449 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-11T20:45:37.532 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:37 smithi023 bash[29146]: cluster 2024-03-11T20:45:35.555883+0000 mon.a (mon.0) 380 : cluster 1 Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-11T20:45:37.532 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:37 smithi023 bash[29146]: cluster 2024-03-11T20:45:35.555939+0000 mon.a (mon.0) 381 : cluster 1 Cluster is now healthy 2024-03-11T20:45:37.533 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:37 smithi023 bash[29146]: cluster 2024-03-11T20:45:35.577515+0000 mon.a (mon.0) 382 : cluster 0 mgrmap e15: y(active, since 4m), standbys: x 2024-03-11T20:45:37.533 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:37 smithi023 bash[29146]: audit 2024-03-11T20:45:36.383217+0000 mon.a (mon.0) 383 : audit 1 from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T20:45:37.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:37 smithi023 bash[21372]: cluster 2024-03-11T20:45:35.365655+0000 mgr.y (mgr.14152) 168 : cluster 0 pgmap v139: 1 pgs: 1 active+clean; 449 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-11T20:45:37.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:37 smithi023 bash[21372]: cluster 2024-03-11T20:45:35.555883+0000 mon.a (mon.0) 380 : cluster 1 Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-11T20:45:37.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:37 smithi023 bash[21372]: cluster 2024-03-11T20:45:35.555939+0000 mon.a (mon.0) 381 : cluster 1 Cluster is now healthy 2024-03-11T20:45:37.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:37 smithi023 bash[21372]: cluster 2024-03-11T20:45:35.577515+0000 mon.a (mon.0) 382 : cluster 0 mgrmap e15: y(active, since 4m), standbys: x 2024-03-11T20:45:37.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:37 smithi023 bash[21372]: audit 2024-03-11T20:45:36.383217+0000 mon.a (mon.0) 383 : audit 1 from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T20:45:37.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:37 smithi146 bash[23809]: cluster 2024-03-11T20:45:35.365655+0000 mgr.y (mgr.14152) 168 : cluster 0 pgmap v139: 1 pgs: 1 active+clean; 449 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-11T20:45:37.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:37 smithi146 bash[23809]: cluster 2024-03-11T20:45:35.555883+0000 mon.a (mon.0) 380 : cluster 1 Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-11T20:45:37.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:37 smithi146 bash[23809]: cluster 2024-03-11T20:45:35.555939+0000 mon.a (mon.0) 381 : cluster 1 Cluster is now healthy 2024-03-11T20:45:37.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:37 smithi146 bash[23809]: cluster 2024-03-11T20:45:35.577515+0000 mon.a (mon.0) 382 : cluster 0 mgrmap e15: y(active, since 4m), standbys: x 2024-03-11T20:45:37.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:37 smithi146 bash[23809]: audit 2024-03-11T20:45:36.383217+0000 mon.a (mon.0) 383 : audit 1 from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T20:45:39.201 INFO:teuthology.orchestra.run.smithi023.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-11T20:45:39.202 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-11T20:45:39.202 INFO:teuthology.orchestra.run.smithi023.stderr: stderr: 10+0 records in 2024-03-11T20:45:39.202 INFO:teuthology.orchestra.run.smithi023.stderr:10+0 records out 2024-03-11T20:45:39.202 INFO:teuthology.orchestra.run.smithi023.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0130887 s, 801 MB/s 2024-03-11T20:45:39.202 INFO:teuthology.orchestra.run.smithi023.stderr:--> Zapping successful for: 2024-03-11T20:45:39.500 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:39 smithi023 bash[29146]: audit 2024-03-11T20:45:37.325601+0000 mon.a (mon.0) 384 : audit 1 from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T20:45:39.500 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:39 smithi023 bash[29146]: audit 2024-03-11T20:45:37.326724+0000 mon.a (mon.0) 385 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T20:45:39.501 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:39 smithi023 bash[29146]: audit 2024-03-11T20:45:37.326941+0000 mon.a (mon.0) 386 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T20:45:39.501 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:39 smithi023 bash[29146]: audit 2024-03-11T20:45:37.327132+0000 mon.a (mon.0) 387 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T20:45:39.501 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:39 smithi023 bash[29146]: audit 2024-03-11T20:45:37.335231+0000 mon.b (mon.1) 7 : audit 1 from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T20:45:39.501 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:39 smithi023 bash[29146]: audit 2024-03-11T20:45:37.335923+0000 mon.a (mon.0) 388 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T20:45:39.501 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:39 smithi023 bash[29146]: audit 2024-03-11T20:45:37.336084+0000 mon.a (mon.0) 389 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T20:45:39.501 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:39 smithi023 bash[29146]: audit 2024-03-11T20:45:37.336281+0000 mon.a (mon.0) 390 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T20:45:39.501 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:39 smithi023 bash[29146]: cluster 2024-03-11T20:45:37.366106+0000 mgr.y (mgr.14152) 169 : cluster 0 pgmap v140: 1 pgs: 1 active+clean; 449 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-11T20:45:39.501 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:39 smithi023 bash[29146]: audit 2024-03-11T20:45:38.229680+0000 mon.b (mon.1) 8 : audit 1 from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T20:45:39.501 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:39 smithi023 bash[29146]: audit 2024-03-11T20:45:38.283568+0000 mon.a (mon.0) 391 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:45:39.501 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:39 smithi023 bash[29146]: audit 2024-03-11T20:45:38.289250+0000 mon.a (mon.0) 392 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T20:45:39.501 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:39 smithi023 bash[29146]: audit 2024-03-11T20:45:38.289517+0000 mon.a (mon.0) 393 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T20:45:39.501 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:39 smithi023 bash[29146]: audit 2024-03-11T20:45:38.289757+0000 mon.a (mon.0) 394 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T20:45:39.501 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:39 smithi023 bash[21372]: audit 2024-03-11T20:45:37.325601+0000 mon.a (mon.0) 384 : audit 1 from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T20:45:39.502 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:39 smithi023 bash[21372]: audit 2024-03-11T20:45:37.326724+0000 mon.a (mon.0) 385 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T20:45:39.502 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:39 smithi023 bash[21372]: audit 2024-03-11T20:45:37.326941+0000 mon.a (mon.0) 386 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T20:45:39.502 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:39 smithi023 bash[21372]: audit 2024-03-11T20:45:37.327132+0000 mon.a (mon.0) 387 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T20:45:39.502 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:39 smithi023 bash[21372]: audit 2024-03-11T20:45:37.335231+0000 mon.b (mon.1) 7 : audit 1 from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T20:45:39.502 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:39 smithi023 bash[21372]: audit 2024-03-11T20:45:37.335923+0000 mon.a (mon.0) 388 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T20:45:39.502 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:39 smithi023 bash[21372]: audit 2024-03-11T20:45:37.336084+0000 mon.a (mon.0) 389 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T20:45:39.502 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:39 smithi023 bash[21372]: audit 2024-03-11T20:45:37.336281+0000 mon.a (mon.0) 390 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T20:45:39.502 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:39 smithi023 bash[21372]: cluster 2024-03-11T20:45:37.366106+0000 mgr.y (mgr.14152) 169 : cluster 0 pgmap v140: 1 pgs: 1 active+clean; 449 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-11T20:45:39.502 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:39 smithi023 bash[21372]: audit 2024-03-11T20:45:38.229680+0000 mon.b (mon.1) 8 : audit 1 from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T20:45:39.502 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:39 smithi023 bash[21372]: audit 2024-03-11T20:45:38.283568+0000 mon.a (mon.0) 391 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:45:39.502 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:39 smithi023 bash[21372]: audit 2024-03-11T20:45:38.289250+0000 mon.a (mon.0) 392 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T20:45:39.503 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:39 smithi023 bash[21372]: audit 2024-03-11T20:45:38.289517+0000 mon.a (mon.0) 393 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T20:45:39.503 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:39 smithi023 bash[21372]: audit 2024-03-11T20:45:38.289757+0000 mon.a (mon.0) 394 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T20:45:39.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:39 smithi146 bash[23809]: audit 2024-03-11T20:45:37.325601+0000 mon.a (mon.0) 384 : audit 1 from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T20:45:39.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:39 smithi146 bash[23809]: audit 2024-03-11T20:45:37.326724+0000 mon.a (mon.0) 385 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T20:45:39.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:39 smithi146 bash[23809]: audit 2024-03-11T20:45:37.326941+0000 mon.a (mon.0) 386 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T20:45:39.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:39 smithi146 bash[23809]: audit 2024-03-11T20:45:37.327132+0000 mon.a (mon.0) 387 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T20:45:39.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:39 smithi146 bash[23809]: audit 2024-03-11T20:45:37.335231+0000 mon.b (mon.1) 7 : audit 1 from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T20:45:39.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:39 smithi146 bash[23809]: audit 2024-03-11T20:45:37.335923+0000 mon.a (mon.0) 388 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T20:45:39.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:39 smithi146 bash[23809]: audit 2024-03-11T20:45:37.336084+0000 mon.a (mon.0) 389 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T20:45:39.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:39 smithi146 bash[23809]: audit 2024-03-11T20:45:37.336281+0000 mon.a (mon.0) 390 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T20:45:39.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:39 smithi146 bash[23809]: cluster 2024-03-11T20:45:37.366106+0000 mgr.y (mgr.14152) 169 : cluster 0 pgmap v140: 1 pgs: 1 active+clean; 449 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-11T20:45:39.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:39 smithi146 bash[23809]: audit 2024-03-11T20:45:38.229680+0000 mon.b (mon.1) 8 : audit 1 from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T20:45:39.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:39 smithi146 bash[23809]: audit 2024-03-11T20:45:38.283568+0000 mon.a (mon.0) 391 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:45:39.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:39 smithi146 bash[23809]: audit 2024-03-11T20:45:38.289250+0000 mon.a (mon.0) 392 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T20:45:39.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:39 smithi146 bash[23809]: audit 2024-03-11T20:45:38.289517+0000 mon.a (mon.0) 393 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T20:45:39.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:39 smithi146 bash[23809]: audit 2024-03-11T20:45:38.289757+0000 mon.a (mon.0) 394 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T20:45:40.197 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi023:vg_nvme/lv_1 2024-03-11T20:45:40.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:40 smithi146 bash[23809]: audit 2024-03-11T20:45:38.288619+0000 mon.c (mon.2) 8 : audit 1 from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T20:45:40.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:40 smithi146 bash[23809]: cluster 2024-03-11T20:45:38.476519+0000 osd.1 (osd.1) 3 : cluster 0 purged_snaps scrub starts 2024-03-11T20:45:40.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:40 smithi146 bash[23809]: cluster 2024-03-11T20:45:38.476657+0000 osd.1 (osd.1) 4 : cluster 0 purged_snaps scrub ok 2024-03-11T20:45:40.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:40 smithi146 bash[23809]: audit 2024-03-11T20:45:39.222403+0000 mon.c (mon.2) 9 : audit 1 from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T20:45:40.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:40 smithi146 bash[23809]: cluster 2024-03-11T20:45:39.283658+0000 mon.a (mon.0) 395 : cluster 0 osdmap e23: 3 total, 3 up, 3 in 2024-03-11T20:45:40.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:40 smithi146 bash[23809]: cluster 2024-03-11T20:45:39.366367+0000 mgr.y (mgr.14152) 170 : cluster 0 pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 3.6 KiB/s rd, 211 KiB/s wr, 13 op/s 2024-03-11T20:45:40.532 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:40 smithi023 bash[29146]: audit 2024-03-11T20:45:38.288619+0000 mon.c (mon.2) 8 : audit 1 from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T20:45:40.533 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:40 smithi023 bash[29146]: cluster 2024-03-11T20:45:38.476519+0000 osd.1 (osd.1) 3 : cluster 0 purged_snaps scrub starts 2024-03-11T20:45:40.533 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:40 smithi023 bash[29146]: cluster 2024-03-11T20:45:38.476657+0000 osd.1 (osd.1) 4 : cluster 0 purged_snaps scrub ok 2024-03-11T20:45:40.533 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:40 smithi023 bash[29146]: audit 2024-03-11T20:45:39.222403+0000 mon.c (mon.2) 9 : audit 1 from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T20:45:40.533 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:40 smithi023 bash[29146]: cluster 2024-03-11T20:45:39.283658+0000 mon.a (mon.0) 395 : cluster 0 osdmap e23: 3 total, 3 up, 3 in 2024-03-11T20:45:40.533 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:40 smithi023 bash[29146]: cluster 2024-03-11T20:45:39.366367+0000 mgr.y (mgr.14152) 170 : cluster 0 pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 3.6 KiB/s rd, 211 KiB/s wr, 13 op/s 2024-03-11T20:45:40.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:40 smithi023 bash[21372]: audit 2024-03-11T20:45:38.288619+0000 mon.c (mon.2) 8 : audit 1 from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T20:45:40.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:40 smithi023 bash[21372]: cluster 2024-03-11T20:45:38.476519+0000 osd.1 (osd.1) 3 : cluster 0 purged_snaps scrub starts 2024-03-11T20:45:40.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:40 smithi023 bash[21372]: cluster 2024-03-11T20:45:38.476657+0000 osd.1 (osd.1) 4 : cluster 0 purged_snaps scrub ok 2024-03-11T20:45:40.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:40 smithi023 bash[21372]: audit 2024-03-11T20:45:39.222403+0000 mon.c (mon.2) 9 : audit 1 from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T20:45:40.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:40 smithi023 bash[21372]: cluster 2024-03-11T20:45:39.283658+0000 mon.a (mon.0) 395 : cluster 0 osdmap e23: 3 total, 3 up, 3 in 2024-03-11T20:45:40.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:40 smithi023 bash[21372]: cluster 2024-03-11T20:45:39.366367+0000 mgr.y (mgr.14152) 170 : cluster 0 pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 3.6 KiB/s rd, 211 KiB/s wr, 13 op/s 2024-03-11T20:45:42.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:41 smithi146 bash[23809]: cephadm 2024-03-11T20:45:40.558282+0000 mgr.y (mgr.14152) 171 : cephadm 1 Detected new or changed devices on smithi023 2024-03-11T20:45:42.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:41 smithi146 bash[23809]: audit 2024-03-11T20:45:40.569254+0000 mon.a (mon.0) 396 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:45:42.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:41 smithi146 bash[23809]: audit 2024-03-11T20:45:40.581150+0000 mon.a (mon.0) 397 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:45:42.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:41 smithi146 bash[23809]: audit 2024-03-11T20:45:40.584014+0000 mon.a (mon.0) 398 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:45:42.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:41 smithi146 bash[23809]: audit 2024-03-11T20:45:40.585782+0000 mon.a (mon.0) 399 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:45:42.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:41 smithi146 bash[23809]: audit 2024-03-11T20:45:40.587417+0000 mon.a (mon.0) 400 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:45:42.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:41 smithi146 bash[23809]: cephadm 2024-03-11T20:45:40.588698+0000 mgr.y (mgr.14152) 172 : cephadm 1 Adjusting osd_memory_target on smithi023 to 5413M 2024-03-11T20:45:42.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:41 smithi146 bash[23809]: audit 2024-03-11T20:45:40.595277+0000 mon.a (mon.0) 401 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:45:42.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:41 smithi146 bash[23809]: audit 2024-03-11T20:45:40.600108+0000 mon.a (mon.0) 402 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:45:42.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:41 smithi146 bash[23809]: audit 2024-03-11T20:45:40.601999+0000 mon.a (mon.0) 403 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:45:42.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:41 smithi146 bash[23809]: audit 2024-03-11T20:45:40.612270+0000 mon.a (mon.0) 404 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:45:42.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:41 smithi023 bash[21372]: cephadm 2024-03-11T20:45:40.558282+0000 mgr.y (mgr.14152) 171 : cephadm 1 Detected new or changed devices on smithi023 2024-03-11T20:45:42.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:41 smithi023 bash[21372]: audit 2024-03-11T20:45:40.569254+0000 mon.a (mon.0) 396 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:45:42.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:41 smithi023 bash[21372]: audit 2024-03-11T20:45:40.581150+0000 mon.a (mon.0) 397 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:45:42.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:41 smithi023 bash[21372]: audit 2024-03-11T20:45:40.584014+0000 mon.a (mon.0) 398 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:45:42.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:41 smithi023 bash[21372]: audit 2024-03-11T20:45:40.585782+0000 mon.a (mon.0) 399 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:45:42.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:41 smithi023 bash[21372]: audit 2024-03-11T20:45:40.587417+0000 mon.a (mon.0) 400 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:45:42.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:41 smithi023 bash[21372]: cephadm 2024-03-11T20:45:40.588698+0000 mgr.y (mgr.14152) 172 : cephadm 1 Adjusting osd_memory_target on smithi023 to 5413M 2024-03-11T20:45:42.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:41 smithi023 bash[21372]: audit 2024-03-11T20:45:40.595277+0000 mon.a (mon.0) 401 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:45:42.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:41 smithi023 bash[21372]: audit 2024-03-11T20:45:40.600108+0000 mon.a (mon.0) 402 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:45:42.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:41 smithi023 bash[21372]: audit 2024-03-11T20:45:40.601999+0000 mon.a (mon.0) 403 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:45:42.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:41 smithi023 bash[21372]: audit 2024-03-11T20:45:40.612270+0000 mon.a (mon.0) 404 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:45:42.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:41 smithi023 bash[29146]: cephadm 2024-03-11T20:45:40.558282+0000 mgr.y (mgr.14152) 171 : cephadm 1 Detected new or changed devices on smithi023 2024-03-11T20:45:42.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:41 smithi023 bash[29146]: audit 2024-03-11T20:45:40.569254+0000 mon.a (mon.0) 396 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:45:42.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:41 smithi023 bash[29146]: audit 2024-03-11T20:45:40.581150+0000 mon.a (mon.0) 397 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:45:42.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:41 smithi023 bash[29146]: audit 2024-03-11T20:45:40.584014+0000 mon.a (mon.0) 398 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:45:42.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:41 smithi023 bash[29146]: audit 2024-03-11T20:45:40.585782+0000 mon.a (mon.0) 399 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:45:42.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:41 smithi023 bash[29146]: audit 2024-03-11T20:45:40.587417+0000 mon.a (mon.0) 400 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:45:42.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:41 smithi023 bash[29146]: cephadm 2024-03-11T20:45:40.588698+0000 mgr.y (mgr.14152) 172 : cephadm 1 Adjusting osd_memory_target on smithi023 to 5413M 2024-03-11T20:45:42.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:41 smithi023 bash[29146]: audit 2024-03-11T20:45:40.595277+0000 mon.a (mon.0) 401 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:45:42.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:41 smithi023 bash[29146]: audit 2024-03-11T20:45:40.600108+0000 mon.a (mon.0) 402 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:45:42.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:41 smithi023 bash[29146]: audit 2024-03-11T20:45:40.601999+0000 mon.a (mon.0) 403 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:45:42.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:41 smithi023 bash[29146]: audit 2024-03-11T20:45:40.612270+0000 mon.a (mon.0) 404 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:45:43.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:42 smithi146 bash[23809]: cluster 2024-03-11T20:45:41.366868+0000 mgr.y (mgr.14152) 173 : cluster 0 pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.6 KiB/s rd, 150 KiB/s wr, 9 op/s 2024-03-11T20:45:43.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:42 smithi023 bash[21372]: cluster 2024-03-11T20:45:41.366868+0000 mgr.y (mgr.14152) 173 : cluster 0 pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.6 KiB/s rd, 150 KiB/s wr, 9 op/s 2024-03-11T20:45:43.033 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:42 smithi023 bash[29146]: cluster 2024-03-11T20:45:41.366868+0000 mgr.y (mgr.14152) 173 : cluster 0 pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.6 KiB/s rd, 150 KiB/s wr, 9 op/s 2024-03-11T20:45:44.955 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:45:44.978 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:44 smithi023 bash[21372]: cluster 2024-03-11T20:45:43.367505+0000 mgr.y (mgr.14152) 174 : cluster 0 pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.0 KiB/s rd, 117 KiB/s wr, 7 op/s 2024-03-11T20:45:44.978 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:44 smithi023 bash[29146]: cluster 2024-03-11T20:45:43.367505+0000 mgr.y (mgr.14152) 174 : cluster 0 pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.0 KiB/s rd, 117 KiB/s wr, 7 op/s 2024-03-11T20:45:45.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:44 smithi146 bash[23809]: cluster 2024-03-11T20:45:43.367505+0000 mgr.y (mgr.14152) 174 : cluster 0 pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.0 KiB/s rd, 117 KiB/s wr, 7 op/s 2024-03-11T20:45:46.782 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:46 smithi023 bash[21372]: cluster 2024-03-11T20:45:45.368053+0000 mgr.y (mgr.14152) 175 : cluster 0 pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.7 KiB/s rd, 131 KiB/s wr, 8 op/s 2024-03-11T20:45:46.783 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:46 smithi023 bash[29146]: cluster 2024-03-11T20:45:45.368053+0000 mgr.y (mgr.14152) 175 : cluster 0 pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.7 KiB/s rd, 131 KiB/s wr, 8 op/s 2024-03-11T20:45:47.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:46 smithi146 bash[23809]: cluster 2024-03-11T20:45:45.368053+0000 mgr.y (mgr.14152) 175 : cluster 0 pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.7 KiB/s rd, 131 KiB/s wr, 8 op/s 2024-03-11T20:45:48.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:47 smithi146 bash[23809]: audit 2024-03-11T20:45:47.584816+0000 mon.a (mon.0) 405 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T20:45:48.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:47 smithi146 bash[23809]: audit 2024-03-11T20:45:47.587970+0000 mon.a (mon.0) 406 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T20:45:48.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:47 smithi146 bash[23809]: audit 2024-03-11T20:45:47.589163+0000 mon.a (mon.0) 407 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:45:48.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:47 smithi023 bash[21372]: audit 2024-03-11T20:45:47.584816+0000 mon.a (mon.0) 405 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T20:45:48.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:47 smithi023 bash[21372]: audit 2024-03-11T20:45:47.587970+0000 mon.a (mon.0) 406 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T20:45:48.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:47 smithi023 bash[21372]: audit 2024-03-11T20:45:47.589163+0000 mon.a (mon.0) 407 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:45:48.033 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:47 smithi023 bash[29146]: audit 2024-03-11T20:45:47.584816+0000 mon.a (mon.0) 405 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T20:45:48.033 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:47 smithi023 bash[29146]: audit 2024-03-11T20:45:47.587970+0000 mon.a (mon.0) 406 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T20:45:48.033 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:47 smithi023 bash[29146]: audit 2024-03-11T20:45:47.589163+0000 mon.a (mon.0) 407 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:45:49.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:48 smithi146 bash[23809]: cluster 2024-03-11T20:45:47.368427+0000 mgr.y (mgr.14152) 176 : cluster 0 pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.7 KiB/s rd, 131 KiB/s wr, 8 op/s 2024-03-11T20:45:49.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:48 smithi146 bash[23809]: audit 2024-03-11T20:45:47.581902+0000 mgr.y (mgr.14152) 177 : audit 0 from='client.14292 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:45:49.032 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:48 smithi023 bash[29146]: cluster 2024-03-11T20:45:47.368427+0000 mgr.y (mgr.14152) 176 : cluster 0 pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.7 KiB/s rd, 131 KiB/s wr, 8 op/s 2024-03-11T20:45:49.032 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:48 smithi023 bash[29146]: audit 2024-03-11T20:45:47.581902+0000 mgr.y (mgr.14152) 177 : audit 0 from='client.14292 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:45:49.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:48 smithi023 bash[21372]: cluster 2024-03-11T20:45:47.368427+0000 mgr.y (mgr.14152) 176 : cluster 0 pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.7 KiB/s rd, 131 KiB/s wr, 8 op/s 2024-03-11T20:45:49.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:48 smithi023 bash[21372]: audit 2024-03-11T20:45:47.581902+0000 mgr.y (mgr.14152) 177 : audit 0 from='client.14292 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:45:51.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:50 smithi146 bash[23809]: cluster 2024-03-11T20:45:49.369004+0000 mgr.y (mgr.14152) 178 : cluster 0 pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 913 B/s rd, 26 KiB/s wr, 2 op/s 2024-03-11T20:45:51.032 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:50 smithi023 bash[29146]: cluster 2024-03-11T20:45:49.369004+0000 mgr.y (mgr.14152) 178 : cluster 0 pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 913 B/s rd, 26 KiB/s wr, 2 op/s 2024-03-11T20:45:51.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:50 smithi023 bash[21372]: cluster 2024-03-11T20:45:49.369004+0000 mgr.y (mgr.14152) 178 : cluster 0 pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 913 B/s rd, 26 KiB/s wr, 2 op/s 2024-03-11T20:45:53.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:52 smithi146 bash[23809]: cluster 2024-03-11T20:45:51.369590+0000 mgr.y (mgr.14152) 179 : cluster 0 pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-03-11T20:45:53.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:52 smithi023 bash[21372]: cluster 2024-03-11T20:45:51.369590+0000 mgr.y (mgr.14152) 179 : cluster 0 pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-03-11T20:45:53.032 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:52 smithi023 bash[29146]: cluster 2024-03-11T20:45:51.369590+0000 mgr.y (mgr.14152) 179 : cluster 0 pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-03-11T20:45:55.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:54 smithi146 bash[23809]: cluster 2024-03-11T20:45:53.370060+0000 mgr.y (mgr.14152) 180 : cluster 0 pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-03-11T20:45:55.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:54 smithi023 bash[21372]: cluster 2024-03-11T20:45:53.370060+0000 mgr.y (mgr.14152) 180 : cluster 0 pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-03-11T20:45:55.032 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:54 smithi023 bash[29146]: cluster 2024-03-11T20:45:53.370060+0000 mgr.y (mgr.14152) 180 : cluster 0 pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-03-11T20:45:57.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:56 smithi146 bash[23809]: cluster 2024-03-11T20:45:55.370559+0000 mgr.y (mgr.14152) 181 : cluster 0 pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-03-11T20:45:57.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:56 smithi146 bash[23809]: audit 2024-03-11T20:45:55.823611+0000 mon.c (mon.2) 10 : audit 1 from='client.? 172.21.15.23:0/4104437618' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e3021106-3779-4b94-9bbf-a7724544110f"}]: dispatch 2024-03-11T20:45:57.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:56 smithi146 bash[23809]: audit 2024-03-11T20:45:55.824181+0000 mon.a (mon.0) 408 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e3021106-3779-4b94-9bbf-a7724544110f"}]: dispatch 2024-03-11T20:45:57.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:56 smithi146 bash[23809]: audit 2024-03-11T20:45:55.830192+0000 mon.a (mon.0) 409 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e3021106-3779-4b94-9bbf-a7724544110f"}]': finished 2024-03-11T20:45:57.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:56 smithi146 bash[23809]: cluster 2024-03-11T20:45:55.835730+0000 mon.a (mon.0) 410 : cluster 0 osdmap e24: 4 total, 3 up, 4 in 2024-03-11T20:45:57.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:56 smithi146 bash[23809]: audit 2024-03-11T20:45:55.836011+0000 mon.a (mon.0) 411 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T20:45:57.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:56 smithi146 bash[23809]: audit 2024-03-11T20:45:56.574101+0000 mon.c (mon.2) 11 : audit 0 from='client.? 172.21.15.23:0/3821681927' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T20:45:57.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:56 smithi023 bash[21372]: cluster 2024-03-11T20:45:55.370559+0000 mgr.y (mgr.14152) 181 : cluster 0 pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-03-11T20:45:57.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:56 smithi023 bash[21372]: audit 2024-03-11T20:45:55.823611+0000 mon.c (mon.2) 10 : audit 1 from='client.? 172.21.15.23:0/4104437618' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e3021106-3779-4b94-9bbf-a7724544110f"}]: dispatch 2024-03-11T20:45:57.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:56 smithi023 bash[21372]: audit 2024-03-11T20:45:55.824181+0000 mon.a (mon.0) 408 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e3021106-3779-4b94-9bbf-a7724544110f"}]: dispatch 2024-03-11T20:45:57.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:56 smithi023 bash[21372]: audit 2024-03-11T20:45:55.830192+0000 mon.a (mon.0) 409 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e3021106-3779-4b94-9bbf-a7724544110f"}]': finished 2024-03-11T20:45:57.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:56 smithi023 bash[21372]: cluster 2024-03-11T20:45:55.835730+0000 mon.a (mon.0) 410 : cluster 0 osdmap e24: 4 total, 3 up, 4 in 2024-03-11T20:45:57.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:56 smithi023 bash[21372]: audit 2024-03-11T20:45:55.836011+0000 mon.a (mon.0) 411 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T20:45:57.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:56 smithi023 bash[21372]: audit 2024-03-11T20:45:56.574101+0000 mon.c (mon.2) 11 : audit 0 from='client.? 172.21.15.23:0/3821681927' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T20:45:57.033 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:56 smithi023 bash[29146]: cluster 2024-03-11T20:45:55.370559+0000 mgr.y (mgr.14152) 181 : cluster 0 pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-03-11T20:45:57.033 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:56 smithi023 bash[29146]: audit 2024-03-11T20:45:55.823611+0000 mon.c (mon.2) 10 : audit 1 from='client.? 172.21.15.23:0/4104437618' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e3021106-3779-4b94-9bbf-a7724544110f"}]: dispatch 2024-03-11T20:45:57.033 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:56 smithi023 bash[29146]: audit 2024-03-11T20:45:55.824181+0000 mon.a (mon.0) 408 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e3021106-3779-4b94-9bbf-a7724544110f"}]: dispatch 2024-03-11T20:45:57.033 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:56 smithi023 bash[29146]: audit 2024-03-11T20:45:55.830192+0000 mon.a (mon.0) 409 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e3021106-3779-4b94-9bbf-a7724544110f"}]': finished 2024-03-11T20:45:57.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:56 smithi023 bash[29146]: cluster 2024-03-11T20:45:55.835730+0000 mon.a (mon.0) 410 : cluster 0 osdmap e24: 4 total, 3 up, 4 in 2024-03-11T20:45:57.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:56 smithi023 bash[29146]: audit 2024-03-11T20:45:55.836011+0000 mon.a (mon.0) 411 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T20:45:57.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:56 smithi023 bash[29146]: audit 2024-03-11T20:45:56.574101+0000 mon.c (mon.2) 11 : audit 0 from='client.? 172.21.15.23:0/3821681927' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T20:45:59.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:45:58 smithi146 bash[23809]: cluster 2024-03-11T20:45:57.371097+0000 mgr.y (mgr.14152) 182 : cluster 0 pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-11T20:45:59.030 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:45:58 smithi023 bash[29146]: cluster 2024-03-11T20:45:57.371097+0000 mgr.y (mgr.14152) 182 : cluster 0 pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-11T20:45:59.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:45:58 smithi023 bash[21372]: cluster 2024-03-11T20:45:57.371097+0000 mgr.y (mgr.14152) 182 : cluster 0 pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-11T20:46:01.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:46:00 smithi146 bash[23809]: cluster 2024-03-11T20:45:59.371687+0000 mgr.y (mgr.14152) 183 : cluster 0 pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-11T20:46:01.032 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:46:00 smithi023 bash[29146]: cluster 2024-03-11T20:45:59.371687+0000 mgr.y (mgr.14152) 183 : cluster 0 pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-11T20:46:01.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:46:00 smithi023 bash[21372]: cluster 2024-03-11T20:45:59.371687+0000 mgr.y (mgr.14152) 183 : cluster 0 pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-11T20:46:03.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:46:02 smithi146 bash[23809]: cluster 2024-03-11T20:46:01.372194+0000 mgr.y (mgr.14152) 184 : cluster 0 pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-11T20:46:03.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:46:02 smithi023 bash[21372]: cluster 2024-03-11T20:46:01.372194+0000 mgr.y (mgr.14152) 184 : cluster 0 pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-11T20:46:03.033 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:46:02 smithi023 bash[29146]: cluster 2024-03-11T20:46:01.372194+0000 mgr.y (mgr.14152) 184 : cluster 0 pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-11T20:46:05.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:46:04 smithi146 bash[23809]: cluster 2024-03-11T20:46:03.372769+0000 mgr.y (mgr.14152) 185 : cluster 0 pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-11T20:46:05.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:46:04 smithi023 bash[21372]: cluster 2024-03-11T20:46:03.372769+0000 mgr.y (mgr.14152) 185 : cluster 0 pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-11T20:46:05.033 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:46:04 smithi023 bash[29146]: cluster 2024-03-11T20:46:03.372769+0000 mgr.y (mgr.14152) 185 : cluster 0 pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-11T20:46:07.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:46:06 smithi146 bash[23809]: cluster 2024-03-11T20:46:05.373222+0000 mgr.y (mgr.14152) 186 : cluster 0 pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-11T20:46:07.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:46:06 smithi023 bash[21372]: cluster 2024-03-11T20:46:05.373222+0000 mgr.y (mgr.14152) 186 : cluster 0 pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-11T20:46:07.028 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:46:06 smithi023 bash[29146]: cluster 2024-03-11T20:46:05.373222+0000 mgr.y (mgr.14152) 186 : cluster 0 pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-11T20:46:09.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:46:08 smithi023 bash[21372]: cluster 2024-03-11T20:46:07.334653+0000 osd.0 (osd.0) 5 : cluster 0 purged_snaps scrub starts 2024-03-11T20:46:09.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:46:08 smithi023 bash[21372]: cluster 2024-03-11T20:46:07.334720+0000 osd.0 (osd.0) 6 : cluster 0 purged_snaps scrub ok 2024-03-11T20:46:09.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:46:08 smithi023 bash[21372]: cluster 2024-03-11T20:46:07.373671+0000 mgr.y (mgr.14152) 187 : cluster 0 pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-11T20:46:09.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:46:08 smithi023 bash[21372]: cluster 2024-03-11T20:46:07.784371+0000 mon.a (mon.0) 412 : cluster 0 osdmap e25: 4 total, 3 up, 4 in 2024-03-11T20:46:09.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:46:08 smithi023 bash[21372]: audit 2024-03-11T20:46:07.784726+0000 mon.a (mon.0) 413 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T20:46:09.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:46:08 smithi023 bash[21372]: audit 2024-03-11T20:46:08.184559+0000 mon.a (mon.0) 414 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-11T20:46:09.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:46:08 smithi023 bash[21372]: audit 2024-03-11T20:46:08.186421+0000 mon.a (mon.0) 415 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:46:09.033 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:46:08 smithi023 bash[29146]: cluster 2024-03-11T20:46:07.334653+0000 osd.0 (osd.0) 5 : cluster 0 purged_snaps scrub starts 2024-03-11T20:46:09.033 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:46:08 smithi023 bash[29146]: cluster 2024-03-11T20:46:07.334720+0000 osd.0 (osd.0) 6 : cluster 0 purged_snaps scrub ok 2024-03-11T20:46:09.033 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:46:08 smithi023 bash[29146]: cluster 2024-03-11T20:46:07.373671+0000 mgr.y (mgr.14152) 187 : cluster 0 pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-11T20:46:09.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:46:08 smithi023 bash[29146]: cluster 2024-03-11T20:46:07.784371+0000 mon.a (mon.0) 412 : cluster 0 osdmap e25: 4 total, 3 up, 4 in 2024-03-11T20:46:09.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:46:08 smithi023 bash[29146]: audit 2024-03-11T20:46:07.784726+0000 mon.a (mon.0) 413 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T20:46:09.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:46:08 smithi023 bash[29146]: audit 2024-03-11T20:46:08.184559+0000 mon.a (mon.0) 414 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-11T20:46:09.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:46:08 smithi023 bash[29146]: audit 2024-03-11T20:46:08.186421+0000 mon.a (mon.0) 415 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:46:09.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:46:08 smithi146 bash[23809]: cluster 2024-03-11T20:46:07.334653+0000 osd.0 (osd.0) 5 : cluster 0 purged_snaps scrub starts 2024-03-11T20:46:09.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:46:08 smithi146 bash[23809]: cluster 2024-03-11T20:46:07.334720+0000 osd.0 (osd.0) 6 : cluster 0 purged_snaps scrub ok 2024-03-11T20:46:09.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:46:08 smithi146 bash[23809]: cluster 2024-03-11T20:46:07.373671+0000 mgr.y (mgr.14152) 187 : cluster 0 pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-11T20:46:09.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:46:08 smithi146 bash[23809]: cluster 2024-03-11T20:46:07.784371+0000 mon.a (mon.0) 412 : cluster 0 osdmap e25: 4 total, 3 up, 4 in 2024-03-11T20:46:09.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:46:08 smithi146 bash[23809]: audit 2024-03-11T20:46:07.784726+0000 mon.a (mon.0) 413 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T20:46:09.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:46:08 smithi146 bash[23809]: audit 2024-03-11T20:46:08.184559+0000 mon.a (mon.0) 414 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-11T20:46:09.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:46:08 smithi146 bash[23809]: audit 2024-03-11T20:46:08.186421+0000 mon.a (mon.0) 415 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:46:10.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:46:09 smithi023 bash[21372]: cephadm 2024-03-11T20:46:08.188068+0000 mgr.y (mgr.14152) 188 : cephadm 1 Deploying daemon osd.3 on smithi023 2024-03-11T20:46:10.033 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:46:09 smithi023 bash[29146]: cephadm 2024-03-11T20:46:08.188068+0000 mgr.y (mgr.14152) 188 : cephadm 1 Deploying daemon osd.3 on smithi023 2024-03-11T20:46:10.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:46:09 smithi146 bash[23809]: cephadm 2024-03-11T20:46:08.188068+0000 mgr.y (mgr.14152) 188 : cephadm 1 Deploying daemon osd.3 on smithi023 2024-03-11T20:46:11.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:46:10 smithi146 bash[23809]: cluster 2024-03-11T20:46:09.374232+0000 mgr.y (mgr.14152) 189 : cluster 0 pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-11T20:46:11.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:46:10 smithi023 bash[21372]: cluster 2024-03-11T20:46:09.374232+0000 mgr.y (mgr.14152) 189 : cluster 0 pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-11T20:46:11.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:46:10 smithi023 bash[29146]: cluster 2024-03-11T20:46:09.374232+0000 mgr.y (mgr.14152) 189 : cluster 0 pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-11T20:46:12.475 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:46:12 smithi023 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:46:12.475 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:46:12 smithi023 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:46:12.475 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:46:12 smithi023 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:46:12.476 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 11 20:46:12 smithi023 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:46:12.476 INFO:journalctl@ceph.osd.1.smithi023.stdout:Mar 11 20:46:12 smithi023 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:46:12.476 INFO:journalctl@ceph.osd.2.smithi023.stdout:Mar 11 20:46:12 smithi023 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:46:12.782 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 11 20:46:12 smithi023 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:46:12.783 INFO:journalctl@ceph.osd.2.smithi023.stdout:Mar 11 20:46:12 smithi023 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:46:12.783 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:46:12 smithi023 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:46:12.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:46:12 smithi023 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:46:12.784 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:46:12 smithi023 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:46:12.784 INFO:journalctl@ceph.osd.1.smithi023.stdout:Mar 11 20:46:12 smithi023 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:46:13.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:46:12 smithi146 bash[23809]: cluster 2024-03-11T20:46:11.374744+0000 mgr.y (mgr.14152) 190 : cluster 0 pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-11T20:46:13.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:46:12 smithi146 bash[23809]: audit 2024-03-11T20:46:12.629108+0000 mon.a (mon.0) 416 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:46:13.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:46:12 smithi146 bash[23809]: audit 2024-03-11T20:46:12.634645+0000 mon.a (mon.0) 417 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:46:13.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:46:12 smithi146 bash[23809]: audit 2024-03-11T20:46:12.635194+0000 mon.a (mon.0) 418 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:46:13.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:46:12 smithi023 bash[21372]: cluster 2024-03-11T20:46:11.374744+0000 mgr.y (mgr.14152) 190 : cluster 0 pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-11T20:46:13.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:46:12 smithi023 bash[21372]: audit 2024-03-11T20:46:12.629108+0000 mon.a (mon.0) 416 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:46:13.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:46:12 smithi023 bash[21372]: audit 2024-03-11T20:46:12.634645+0000 mon.a (mon.0) 417 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:46:13.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:46:12 smithi023 bash[21372]: audit 2024-03-11T20:46:12.635194+0000 mon.a (mon.0) 418 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:46:13.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:46:12 smithi023 bash[29146]: cluster 2024-03-11T20:46:11.374744+0000 mgr.y (mgr.14152) 190 : cluster 0 pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-11T20:46:13.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:46:12 smithi023 bash[29146]: audit 2024-03-11T20:46:12.629108+0000 mon.a (mon.0) 416 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:46:13.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:46:12 smithi023 bash[29146]: audit 2024-03-11T20:46:12.634645+0000 mon.a (mon.0) 417 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:46:13.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:46:12 smithi023 bash[29146]: audit 2024-03-11T20:46:12.635194+0000 mon.a (mon.0) 418 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:46:15.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:46:14 smithi146 bash[23809]: cluster 2024-03-11T20:46:13.375284+0000 mgr.y (mgr.14152) 191 : cluster 0 pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-11T20:46:15.282 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:46:14 smithi023 bash[29146]: cluster 2024-03-11T20:46:13.375284+0000 mgr.y (mgr.14152) 191 : cluster 0 pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-11T20:46:15.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:46:14 smithi023 bash[21372]: cluster 2024-03-11T20:46:13.375284+0000 mgr.y (mgr.14152) 191 : cluster 0 pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-11T20:46:17.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:46:16 smithi146 bash[23809]: cluster 2024-03-11T20:46:15.375719+0000 mgr.y (mgr.14152) 192 : cluster 0 pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-11T20:46:17.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:46:16 smithi023 bash[21372]: cluster 2024-03-11T20:46:15.375719+0000 mgr.y (mgr.14152) 192 : cluster 0 pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-11T20:46:17.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:46:16 smithi023 bash[29146]: cluster 2024-03-11T20:46:15.375719+0000 mgr.y (mgr.14152) 192 : cluster 0 pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-11T20:46:19.286 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:46:19 smithi023 bash[21372]: cluster 2024-03-11T20:46:17.376175+0000 mgr.y (mgr.14152) 193 : cluster 0 pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-11T20:46:19.286 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:46:19 smithi023 bash[21372]: audit 2024-03-11T20:46:18.030948+0000 mon.a (mon.0) 419 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:46:19.286 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:46:19 smithi023 bash[21372]: audit 2024-03-11T20:46:18.040333+0000 mon.a (mon.0) 420 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:46:19.286 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:46:19 smithi023 bash[21372]: audit 2024-03-11T20:46:18.041637+0000 mon.a (mon.0) 421 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:46:19.286 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:46:19 smithi023 bash[21372]: audit 2024-03-11T20:46:18.042912+0000 mon.a (mon.0) 422 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:46:19.286 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:46:19 smithi023 bash[21372]: audit 2024-03-11T20:46:18.049507+0000 mon.a (mon.0) 423 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:46:19.286 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:46:19 smithi023 bash[29146]: cluster 2024-03-11T20:46:17.376175+0000 mgr.y (mgr.14152) 193 : cluster 0 pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-11T20:46:19.286 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:46:19 smithi023 bash[29146]: audit 2024-03-11T20:46:18.030948+0000 mon.a (mon.0) 419 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:46:19.287 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:46:19 smithi023 bash[29146]: audit 2024-03-11T20:46:18.040333+0000 mon.a (mon.0) 420 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:46:19.287 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:46:19 smithi023 bash[29146]: audit 2024-03-11T20:46:18.041637+0000 mon.a (mon.0) 421 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:46:19.287 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:46:19 smithi023 bash[29146]: audit 2024-03-11T20:46:18.042912+0000 mon.a (mon.0) 422 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:46:19.287 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:46:19 smithi023 bash[29146]: audit 2024-03-11T20:46:18.049507+0000 mon.a (mon.0) 423 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:46:19.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:46:19 smithi146 bash[23809]: cluster 2024-03-11T20:46:17.376175+0000 mgr.y (mgr.14152) 193 : cluster 0 pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-11T20:46:19.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:46:19 smithi146 bash[23809]: audit 2024-03-11T20:46:18.030948+0000 mon.a (mon.0) 419 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:46:19.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:46:19 smithi146 bash[23809]: audit 2024-03-11T20:46:18.040333+0000 mon.a (mon.0) 420 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:46:19.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:46:19 smithi146 bash[23809]: audit 2024-03-11T20:46:18.041637+0000 mon.a (mon.0) 421 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:46:19.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:46:19 smithi146 bash[23809]: audit 2024-03-11T20:46:18.042912+0000 mon.a (mon.0) 422 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:46:19.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:46:19 smithi146 bash[23809]: audit 2024-03-11T20:46:18.049507+0000 mon.a (mon.0) 423 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:46:20.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:46:20 smithi023 bash[21372]: audit 2024-03-11T20:46:19.229636+0000 mon.c (mon.2) 12 : audit 1 from='osd.3 [v2:172.21.15.23:6826/938496814,v1:172.21.15.23:6827/938496814]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-11T20:46:20.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:46:20 smithi023 bash[21372]: audit 2024-03-11T20:46:19.230000+0000 mon.a (mon.0) 424 : audit 1 from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-11T20:46:20.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:46:20 smithi023 bash[29146]: audit 2024-03-11T20:46:19.229636+0000 mon.c (mon.2) 12 : audit 1 from='osd.3 [v2:172.21.15.23:6826/938496814,v1:172.21.15.23:6827/938496814]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-11T20:46:20.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:46:20 smithi023 bash[29146]: audit 2024-03-11T20:46:19.230000+0000 mon.a (mon.0) 424 : audit 1 from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-11T20:46:20.514 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:46:20 smithi146 bash[23809]: audit 2024-03-11T20:46:19.229636+0000 mon.c (mon.2) 12 : audit 1 from='osd.3 [v2:172.21.15.23:6826/938496814,v1:172.21.15.23:6827/938496814]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-11T20:46:20.514 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:46:20 smithi146 bash[23809]: audit 2024-03-11T20:46:19.230000+0000 mon.a (mon.0) 424 : audit 1 from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-11T20:46:21.287 INFO:teuthology.orchestra.run.smithi023.stdout:Created osd(s) 3 on host 'smithi023' 2024-03-11T20:46:21.338 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:46:21 smithi023 bash[29146]: cluster 2024-03-11T20:46:19.376688+0000 mgr.y (mgr.14152) 194 : cluster 0 pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-11T20:46:21.339 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:46:21 smithi023 bash[29146]: audit 2024-03-11T20:46:20.044755+0000 mon.a (mon.0) 425 : audit 1 from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-11T20:46:21.339 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:46:21 smithi023 bash[29146]: audit 2024-03-11T20:46:20.048275+0000 mon.c (mon.2) 13 : audit 1 from='osd.3 [v2:172.21.15.23:6826/938496814,v1:172.21.15.23:6827/938496814]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2024-03-11T20:46:21.339 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:46:21 smithi023 bash[29146]: cluster 2024-03-11T20:46:20.048845+0000 mon.a (mon.0) 426 : cluster 0 osdmap e26: 4 total, 3 up, 4 in 2024-03-11T20:46:21.339 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:46:21 smithi023 bash[29146]: audit 2024-03-11T20:46:20.049215+0000 mon.a (mon.0) 427 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T20:46:21.339 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:46:21 smithi023 bash[29146]: audit 2024-03-11T20:46:20.049612+0000 mon.a (mon.0) 428 : audit 1 from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2024-03-11T20:46:21.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:46:21 smithi023 bash[21372]: cluster 2024-03-11T20:46:19.376688+0000 mgr.y (mgr.14152) 194 : cluster 0 pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-11T20:46:21.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:46:21 smithi023 bash[21372]: audit 2024-03-11T20:46:20.044755+0000 mon.a (mon.0) 425 : audit 1 from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-11T20:46:21.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:46:21 smithi023 bash[21372]: audit 2024-03-11T20:46:20.048275+0000 mon.c (mon.2) 13 : audit 1 from='osd.3 [v2:172.21.15.23:6826/938496814,v1:172.21.15.23:6827/938496814]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2024-03-11T20:46:21.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:46:21 smithi023 bash[21372]: cluster 2024-03-11T20:46:20.048845+0000 mon.a (mon.0) 426 : cluster 0 osdmap e26: 4 total, 3 up, 4 in 2024-03-11T20:46:21.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:46:21 smithi023 bash[21372]: audit 2024-03-11T20:46:20.049215+0000 mon.a (mon.0) 427 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T20:46:21.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:46:21 smithi023 bash[21372]: audit 2024-03-11T20:46:20.049612+0000 mon.a (mon.0) 428 : audit 1 from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2024-03-11T20:46:21.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:46:21 smithi146 bash[23809]: cluster 2024-03-11T20:46:19.376688+0000 mgr.y (mgr.14152) 194 : cluster 0 pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-11T20:46:21.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:46:21 smithi146 bash[23809]: audit 2024-03-11T20:46:20.044755+0000 mon.a (mon.0) 425 : audit 1 from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-11T20:46:21.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:46:21 smithi146 bash[23809]: audit 2024-03-11T20:46:20.048275+0000 mon.c (mon.2) 13 : audit 1 from='osd.3 [v2:172.21.15.23:6826/938496814,v1:172.21.15.23:6827/938496814]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2024-03-11T20:46:21.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:46:21 smithi146 bash[23809]: cluster 2024-03-11T20:46:20.048845+0000 mon.a (mon.0) 426 : cluster 0 osdmap e26: 4 total, 3 up, 4 in 2024-03-11T20:46:21.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:46:21 smithi146 bash[23809]: audit 2024-03-11T20:46:20.049215+0000 mon.a (mon.0) 427 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T20:46:21.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:46:21 smithi146 bash[23809]: audit 2024-03-11T20:46:20.049612+0000 mon.a (mon.0) 428 : audit 1 from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2024-03-11T20:46:21.984 DEBUG:teuthology.orchestra.run.smithi023:osd.3> sudo journalctl -f -n 0 -u ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@osd.3.service 2024-03-11T20:46:21.986 INFO:tasks.cephadm:Deploying osd.4 on smithi146 with /dev/vg_nvme/lv_4... 2024-03-11T20:46:21.986 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-11T20:46:22.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:46:22 smithi146 bash[23809]: audit 2024-03-11T20:46:21.055737+0000 mon.a (mon.0) 429 : audit 1 from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]': finished 2024-03-11T20:46:22.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:46:22 smithi146 bash[23809]: cluster 2024-03-11T20:46:21.065496+0000 mon.a (mon.0) 430 : cluster 0 osdmap e27: 4 total, 3 up, 4 in 2024-03-11T20:46:22.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:46:22 smithi146 bash[23809]: audit 2024-03-11T20:46:21.066597+0000 mon.a (mon.0) 431 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T20:46:22.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:46:22 smithi146 bash[23809]: audit 2024-03-11T20:46:21.074930+0000 mon.a (mon.0) 432 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T20:46:22.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:46:22 smithi146 bash[23809]: audit 2024-03-11T20:46:21.269582+0000 mon.a (mon.0) 433 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:46:22.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:46:22 smithi146 bash[23809]: audit 2024-03-11T20:46:21.281143+0000 mon.a (mon.0) 434 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:46:22.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:46:22 smithi146 bash[23809]: audit 2024-03-11T20:46:21.287941+0000 mon.a (mon.0) 435 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:46:22.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:46:22 smithi146 bash[23809]: cluster 2024-03-11T20:46:21.377057+0000 mgr.y (mgr.14152) 195 : cluster 0 pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-11T20:46:22.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:46:22 smithi023 bash[21372]: audit 2024-03-11T20:46:21.055737+0000 mon.a (mon.0) 429 : audit 1 from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]': finished 2024-03-11T20:46:22.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:46:22 smithi023 bash[21372]: cluster 2024-03-11T20:46:21.065496+0000 mon.a (mon.0) 430 : cluster 0 osdmap e27: 4 total, 3 up, 4 in 2024-03-11T20:46:22.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:46:22 smithi023 bash[21372]: audit 2024-03-11T20:46:21.066597+0000 mon.a (mon.0) 431 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T20:46:22.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:46:22 smithi023 bash[21372]: audit 2024-03-11T20:46:21.074930+0000 mon.a (mon.0) 432 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T20:46:22.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:46:22 smithi023 bash[21372]: audit 2024-03-11T20:46:21.269582+0000 mon.a (mon.0) 433 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:46:22.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:46:22 smithi023 bash[21372]: audit 2024-03-11T20:46:21.281143+0000 mon.a (mon.0) 434 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:46:22.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:46:22 smithi023 bash[21372]: audit 2024-03-11T20:46:21.287941+0000 mon.a (mon.0) 435 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:46:22.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:46:22 smithi023 bash[21372]: cluster 2024-03-11T20:46:21.377057+0000 mgr.y (mgr.14152) 195 : cluster 0 pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-11T20:46:22.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:46:22 smithi023 bash[29146]: audit 2024-03-11T20:46:21.055737+0000 mon.a (mon.0) 429 : audit 1 from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]': finished 2024-03-11T20:46:22.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:46:22 smithi023 bash[29146]: cluster 2024-03-11T20:46:21.065496+0000 mon.a (mon.0) 430 : cluster 0 osdmap e27: 4 total, 3 up, 4 in 2024-03-11T20:46:22.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:46:22 smithi023 bash[29146]: audit 2024-03-11T20:46:21.066597+0000 mon.a (mon.0) 431 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T20:46:22.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:46:22 smithi023 bash[29146]: audit 2024-03-11T20:46:21.074930+0000 mon.a (mon.0) 432 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T20:46:22.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:46:22 smithi023 bash[29146]: audit 2024-03-11T20:46:21.269582+0000 mon.a (mon.0) 433 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:46:22.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:46:22 smithi023 bash[29146]: audit 2024-03-11T20:46:21.281143+0000 mon.a (mon.0) 434 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:46:22.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:46:22 smithi023 bash[29146]: audit 2024-03-11T20:46:21.287941+0000 mon.a (mon.0) 435 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:46:22.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:46:22 smithi023 bash[29146]: cluster 2024-03-11T20:46:21.377057+0000 mgr.y (mgr.14152) 195 : cluster 0 pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-11T20:46:23.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:46:23 smithi146 bash[23809]: cluster 2024-03-11T20:46:20.225378+0000 osd.3 (osd.3) 1 : cluster 0 purged_snaps scrub starts 2024-03-11T20:46:23.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:46:23 smithi146 bash[23809]: cluster 2024-03-11T20:46:20.225465+0000 osd.3 (osd.3) 2 : cluster 0 purged_snaps scrub ok 2024-03-11T20:46:23.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:46:23 smithi146 bash[23809]: cluster 2024-03-11T20:46:22.079473+0000 mon.a (mon.0) 436 : cluster 1 osd.3 [v2:172.21.15.23:6826/938496814,v1:172.21.15.23:6827/938496814] boot 2024-03-11T20:46:23.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:46:23 smithi146 bash[23809]: cluster 2024-03-11T20:46:22.079603+0000 mon.a (mon.0) 437 : cluster 0 osdmap e28: 4 total, 4 up, 4 in 2024-03-11T20:46:23.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:46:23 smithi146 bash[23809]: audit 2024-03-11T20:46:22.080732+0000 mon.a (mon.0) 438 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T20:46:23.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:46:23 smithi023 bash[21372]: cluster 2024-03-11T20:46:20.225378+0000 osd.3 (osd.3) 1 : cluster 0 purged_snaps scrub starts 2024-03-11T20:46:23.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:46:23 smithi023 bash[21372]: cluster 2024-03-11T20:46:20.225465+0000 osd.3 (osd.3) 2 : cluster 0 purged_snaps scrub ok 2024-03-11T20:46:23.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:46:23 smithi023 bash[21372]: cluster 2024-03-11T20:46:22.079473+0000 mon.a (mon.0) 436 : cluster 1 osd.3 [v2:172.21.15.23:6826/938496814,v1:172.21.15.23:6827/938496814] boot 2024-03-11T20:46:23.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:46:23 smithi023 bash[21372]: cluster 2024-03-11T20:46:22.079603+0000 mon.a (mon.0) 437 : cluster 0 osdmap e28: 4 total, 4 up, 4 in 2024-03-11T20:46:23.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:46:23 smithi023 bash[21372]: audit 2024-03-11T20:46:22.080732+0000 mon.a (mon.0) 438 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T20:46:23.533 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:46:23 smithi023 bash[29146]: cluster 2024-03-11T20:46:20.225378+0000 osd.3 (osd.3) 1 : cluster 0 purged_snaps scrub starts 2024-03-11T20:46:23.533 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:46:23 smithi023 bash[29146]: cluster 2024-03-11T20:46:20.225465+0000 osd.3 (osd.3) 2 : cluster 0 purged_snaps scrub ok 2024-03-11T20:46:23.533 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:46:23 smithi023 bash[29146]: cluster 2024-03-11T20:46:22.079473+0000 mon.a (mon.0) 436 : cluster 1 osd.3 [v2:172.21.15.23:6826/938496814,v1:172.21.15.23:6827/938496814] boot 2024-03-11T20:46:23.534 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:46:23 smithi023 bash[29146]: cluster 2024-03-11T20:46:22.079603+0000 mon.a (mon.0) 437 : cluster 0 osdmap e28: 4 total, 4 up, 4 in 2024-03-11T20:46:23.534 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:46:23 smithi023 bash[29146]: audit 2024-03-11T20:46:22.080732+0000 mon.a (mon.0) 438 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T20:46:24.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:46:24 smithi146 bash[23809]: cluster 2024-03-11T20:46:23.103038+0000 mon.a (mon.0) 439 : cluster 0 osdmap e29: 4 total, 4 up, 4 in 2024-03-11T20:46:24.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:46:24 smithi146 bash[23809]: cluster 2024-03-11T20:46:23.377547+0000 mgr.y (mgr.14152) 196 : cluster 0 pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-11T20:46:24.532 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:46:24 smithi023 bash[21372]: cluster 2024-03-11T20:46:23.103038+0000 mon.a (mon.0) 439 : cluster 0 osdmap e29: 4 total, 4 up, 4 in 2024-03-11T20:46:24.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:46:24 smithi023 bash[21372]: cluster 2024-03-11T20:46:23.377547+0000 mgr.y (mgr.14152) 196 : cluster 0 pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-11T20:46:24.533 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:46:24 smithi023 bash[29146]: cluster 2024-03-11T20:46:23.103038+0000 mon.a (mon.0) 439 : cluster 0 osdmap e29: 4 total, 4 up, 4 in 2024-03-11T20:46:24.533 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:46:24 smithi023 bash[29146]: cluster 2024-03-11T20:46:23.377547+0000 mgr.y (mgr.14152) 196 : cluster 0 pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-11T20:46:26.699 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T20:46:26.719 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:46:26 smithi146 bash[23809]: cluster 2024-03-11T20:46:25.378118+0000 mgr.y (mgr.14152) 197 : cluster 0 pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-11T20:46:26.782 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:46:26 smithi023 bash[29146]: cluster 2024-03-11T20:46:25.378118+0000 mgr.y (mgr.14152) 197 : cluster 0 pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-11T20:46:26.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:46:26 smithi023 bash[21372]: cluster 2024-03-11T20:46:25.378118+0000 mgr.y (mgr.14152) 197 : cluster 0 pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-11T20:46:28.750 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:46:28 smithi023 bash[21372]: cluster 2024-03-11T20:46:27.378525+0000 mgr.y (mgr.14152) 198 : cluster 0 pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-11T20:46:28.751 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:46:28 smithi023 bash[29146]: cluster 2024-03-11T20:46:27.378525+0000 mgr.y (mgr.14152) 198 : cluster 0 pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-11T20:46:28.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:46:28 smithi146 bash[23809]: cluster 2024-03-11T20:46:27.378525+0000 mgr.y (mgr.14152) 198 : cluster 0 pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-11T20:46:29.595 INFO:teuthology.orchestra.run.smithi146.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-11T20:46:29.595 INFO:teuthology.orchestra.run.smithi146.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-11T20:46:29.595 INFO:teuthology.orchestra.run.smithi146.stderr: stderr: 10+0 records in 2024-03-11T20:46:29.595 INFO:teuthology.orchestra.run.smithi146.stderr:10+0 records out 2024-03-11T20:46:29.595 INFO:teuthology.orchestra.run.smithi146.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0165042 s, 635 MB/s 2024-03-11T20:46:29.595 INFO:teuthology.orchestra.run.smithi146.stderr:--> Zapping successful for: 2024-03-11T20:46:30.225 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi146:vg_nvme/lv_4 2024-03-11T20:46:30.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:46:30 smithi146 bash[23809]: cluster 2024-03-11T20:46:29.378899+0000 mgr.y (mgr.14152) 199 : cluster 0 pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-11T20:46:30.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:46:30 smithi023 bash[21372]: cluster 2024-03-11T20:46:29.378899+0000 mgr.y (mgr.14152) 199 : cluster 0 pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-11T20:46:30.783 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:46:30 smithi023 bash[29146]: cluster 2024-03-11T20:46:29.378899+0000 mgr.y (mgr.14152) 199 : cluster 0 pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-11T20:46:32.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:46:31 smithi146 bash[23809]: cephadm 2024-03-11T20:46:30.780308+0000 mgr.y (mgr.14152) 200 : cephadm 1 Detected new or changed devices on smithi023 2024-03-11T20:46:32.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:46:31 smithi146 bash[23809]: audit 2024-03-11T20:46:30.794241+0000 mon.a (mon.0) 440 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:46:32.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:46:31 smithi146 bash[23809]: audit 2024-03-11T20:46:30.808165+0000 mon.a (mon.0) 441 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:46:32.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:46:31 smithi146 bash[23809]: audit 2024-03-11T20:46:30.811078+0000 mon.a (mon.0) 442 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:46:32.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:46:31 smithi146 bash[23809]: audit 2024-03-11T20:46:30.813097+0000 mon.a (mon.0) 443 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:46:32.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:46:31 smithi146 bash[23809]: audit 2024-03-11T20:46:30.814963+0000 mon.a (mon.0) 444 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:46:32.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:46:31 smithi146 bash[23809]: audit 2024-03-11T20:46:30.816759+0000 mon.a (mon.0) 445 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:46:32.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:46:31 smithi146 bash[23809]: cephadm 2024-03-11T20:46:30.818034+0000 mgr.y (mgr.14152) 201 : cephadm 1 Adjusting osd_memory_target on smithi023 to 4060M 2024-03-11T20:46:32.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:46:31 smithi146 bash[23809]: audit 2024-03-11T20:46:30.826474+0000 mon.a (mon.0) 446 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:46:32.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:46:31 smithi146 bash[23809]: audit 2024-03-11T20:46:30.833032+0000 mon.a (mon.0) 447 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:46:32.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:46:31 smithi146 bash[23809]: audit 2024-03-11T20:46:30.834570+0000 mon.a (mon.0) 448 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:46:32.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:46:31 smithi146 bash[23809]: audit 2024-03-11T20:46:30.843957+0000 mon.a (mon.0) 449 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:46:32.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:46:31 smithi023 bash[21372]: cephadm 2024-03-11T20:46:30.780308+0000 mgr.y (mgr.14152) 200 : cephadm 1 Detected new or changed devices on smithi023 2024-03-11T20:46:32.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:46:31 smithi023 bash[21372]: audit 2024-03-11T20:46:30.794241+0000 mon.a (mon.0) 440 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:46:32.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:46:31 smithi023 bash[21372]: audit 2024-03-11T20:46:30.808165+0000 mon.a (mon.0) 441 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:46:32.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:46:31 smithi023 bash[21372]: audit 2024-03-11T20:46:30.811078+0000 mon.a (mon.0) 442 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:46:32.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:46:31 smithi023 bash[21372]: audit 2024-03-11T20:46:30.813097+0000 mon.a (mon.0) 443 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:46:32.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:46:31 smithi023 bash[21372]: audit 2024-03-11T20:46:30.814963+0000 mon.a (mon.0) 444 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:46:32.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:46:31 smithi023 bash[21372]: audit 2024-03-11T20:46:30.816759+0000 mon.a (mon.0) 445 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:46:32.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:46:31 smithi023 bash[21372]: cephadm 2024-03-11T20:46:30.818034+0000 mgr.y (mgr.14152) 201 : cephadm 1 Adjusting osd_memory_target on smithi023 to 4060M 2024-03-11T20:46:32.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:46:31 smithi023 bash[21372]: audit 2024-03-11T20:46:30.826474+0000 mon.a (mon.0) 446 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:46:32.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:46:31 smithi023 bash[21372]: audit 2024-03-11T20:46:30.833032+0000 mon.a (mon.0) 447 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:46:32.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:46:31 smithi023 bash[21372]: audit 2024-03-11T20:46:30.834570+0000 mon.a (mon.0) 448 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:46:32.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:46:31 smithi023 bash[21372]: audit 2024-03-11T20:46:30.843957+0000 mon.a (mon.0) 449 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:46:32.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:46:31 smithi023 bash[29146]: cephadm 2024-03-11T20:46:30.780308+0000 mgr.y (mgr.14152) 200 : cephadm 1 Detected new or changed devices on smithi023 2024-03-11T20:46:32.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:46:31 smithi023 bash[29146]: audit 2024-03-11T20:46:30.794241+0000 mon.a (mon.0) 440 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:46:32.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:46:31 smithi023 bash[29146]: audit 2024-03-11T20:46:30.808165+0000 mon.a (mon.0) 441 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:46:32.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:46:31 smithi023 bash[29146]: audit 2024-03-11T20:46:30.811078+0000 mon.a (mon.0) 442 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:46:32.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:46:31 smithi023 bash[29146]: audit 2024-03-11T20:46:30.813097+0000 mon.a (mon.0) 443 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:46:32.286 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:46:31 smithi023 bash[29146]: audit 2024-03-11T20:46:30.814963+0000 mon.a (mon.0) 444 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:46:32.286 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:46:31 smithi023 bash[29146]: audit 2024-03-11T20:46:30.816759+0000 mon.a (mon.0) 445 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:46:32.286 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:46:31 smithi023 bash[29146]: cephadm 2024-03-11T20:46:30.818034+0000 mgr.y (mgr.14152) 201 : cephadm 1 Adjusting osd_memory_target on smithi023 to 4060M 2024-03-11T20:46:32.286 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:46:31 smithi023 bash[29146]: audit 2024-03-11T20:46:30.826474+0000 mon.a (mon.0) 446 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:46:32.286 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:46:31 smithi023 bash[29146]: audit 2024-03-11T20:46:30.833032+0000 mon.a (mon.0) 447 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:46:32.286 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:46:31 smithi023 bash[29146]: audit 2024-03-11T20:46:30.834570+0000 mon.a (mon.0) 448 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:46:32.286 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:46:31 smithi023 bash[29146]: audit 2024-03-11T20:46:30.843957+0000 mon.a (mon.0) 449 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:46:33.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:46:32 smithi146 bash[23809]: cluster 2024-03-11T20:46:31.379497+0000 mgr.y (mgr.14152) 202 : cluster 0 pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-11T20:46:33.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:46:32 smithi023 bash[21372]: cluster 2024-03-11T20:46:31.379497+0000 mgr.y (mgr.14152) 202 : cluster 0 pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-11T20:46:33.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:46:32 smithi023 bash[29146]: cluster 2024-03-11T20:46:31.379497+0000 mgr.y (mgr.14152) 202 : cluster 0 pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-11T20:46:34.935 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T20:46:35.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:46:34 smithi146 bash[23809]: cluster 2024-03-11T20:46:33.380026+0000 mgr.y (mgr.14152) 203 : cluster 0 pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-11T20:46:35.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:46:34 smithi023 bash[21372]: cluster 2024-03-11T20:46:33.380026+0000 mgr.y (mgr.14152) 203 : cluster 0 pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-11T20:46:35.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:46:34 smithi023 bash[29146]: cluster 2024-03-11T20:46:33.380026+0000 mgr.y (mgr.14152) 203 : cluster 0 pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-11T20:46:37.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:46:36 smithi146 bash[23809]: cluster 2024-03-11T20:46:35.380579+0000 mgr.y (mgr.14152) 204 : cluster 0 pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-11T20:46:37.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:46:36 smithi023 bash[21372]: cluster 2024-03-11T20:46:35.380579+0000 mgr.y (mgr.14152) 204 : cluster 0 pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-11T20:46:37.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:46:36 smithi023 bash[29146]: cluster 2024-03-11T20:46:35.380579+0000 mgr.y (mgr.14152) 204 : cluster 0 pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-11T20:46:38.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:46:37 smithi146 bash[23809]: audit 2024-03-11T20:46:37.266639+0000 mon.a (mon.0) 450 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T20:46:38.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:46:37 smithi146 bash[23809]: audit 2024-03-11T20:46:37.273828+0000 mon.a (mon.0) 451 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T20:46:38.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:46:37 smithi146 bash[23809]: audit 2024-03-11T20:46:37.275665+0000 mon.a (mon.0) 452 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:46:38.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:46:37 smithi023 bash[21372]: audit 2024-03-11T20:46:37.266639+0000 mon.a (mon.0) 450 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T20:46:38.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:46:37 smithi023 bash[21372]: audit 2024-03-11T20:46:37.273828+0000 mon.a (mon.0) 451 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T20:46:38.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:46:37 smithi023 bash[21372]: audit 2024-03-11T20:46:37.275665+0000 mon.a (mon.0) 452 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:46:38.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:46:37 smithi023 bash[29146]: audit 2024-03-11T20:46:37.266639+0000 mon.a (mon.0) 450 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T20:46:38.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:46:37 smithi023 bash[29146]: audit 2024-03-11T20:46:37.273828+0000 mon.a (mon.0) 451 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T20:46:38.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:46:37 smithi023 bash[29146]: audit 2024-03-11T20:46:37.275665+0000 mon.a (mon.0) 452 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:46:39.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:46:38 smithi146 bash[23809]: audit 2024-03-11T20:46:37.259193+0000 mgr.y (mgr.14152) 205 : audit 0 from='client.24184 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi146:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:46:39.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:46:38 smithi146 bash[23809]: cluster 2024-03-11T20:46:37.381098+0000 mgr.y (mgr.14152) 206 : cluster 0 pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-11T20:46:39.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:46:38 smithi023 bash[29146]: audit 2024-03-11T20:46:37.259193+0000 mgr.y (mgr.14152) 205 : audit 0 from='client.24184 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi146:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:46:39.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:46:38 smithi023 bash[29146]: cluster 2024-03-11T20:46:37.381098+0000 mgr.y (mgr.14152) 206 : cluster 0 pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-11T20:46:39.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:46:38 smithi023 bash[21372]: audit 2024-03-11T20:46:37.259193+0000 mgr.y (mgr.14152) 205 : audit 0 from='client.24184 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi146:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:46:39.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:46:38 smithi023 bash[21372]: cluster 2024-03-11T20:46:37.381098+0000 mgr.y (mgr.14152) 206 : cluster 0 pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-11T20:46:41.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:46:40 smithi146 bash[23809]: cluster 2024-03-11T20:46:39.381633+0000 mgr.y (mgr.14152) 207 : cluster 0 pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-11T20:46:41.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:46:40 smithi146 bash[23809]: cluster 2024-03-11T20:46:39.891861+0000 osd.2 (osd.2) 3 : cluster 0 purged_snaps scrub starts 2024-03-11T20:46:41.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:46:40 smithi146 bash[23809]: cluster 2024-03-11T20:46:39.891994+0000 osd.2 (osd.2) 4 : cluster 0 purged_snaps scrub ok 2024-03-11T20:46:41.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:46:40 smithi023 bash[21372]: cluster 2024-03-11T20:46:39.381633+0000 mgr.y (mgr.14152) 207 : cluster 0 pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-11T20:46:41.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:46:40 smithi023 bash[21372]: cluster 2024-03-11T20:46:39.891861+0000 osd.2 (osd.2) 3 : cluster 0 purged_snaps scrub starts 2024-03-11T20:46:41.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:46:40 smithi023 bash[21372]: cluster 2024-03-11T20:46:39.891994+0000 osd.2 (osd.2) 4 : cluster 0 purged_snaps scrub ok 2024-03-11T20:46:41.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:46:40 smithi023 bash[29146]: cluster 2024-03-11T20:46:39.381633+0000 mgr.y (mgr.14152) 207 : cluster 0 pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-11T20:46:41.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:46:40 smithi023 bash[29146]: cluster 2024-03-11T20:46:39.891861+0000 osd.2 (osd.2) 3 : cluster 0 purged_snaps scrub starts 2024-03-11T20:46:41.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:46:40 smithi023 bash[29146]: cluster 2024-03-11T20:46:39.891994+0000 osd.2 (osd.2) 4 : cluster 0 purged_snaps scrub ok 2024-03-11T20:46:42.245 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:46:41 smithi146 bash[23809]: cluster 2024-03-11T20:46:40.885017+0000 mon.a (mon.0) 453 : cluster 0 osdmap e30: 4 total, 4 up, 4 in 2024-03-11T20:46:42.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:46:41 smithi023 bash[21372]: cluster 2024-03-11T20:46:40.885017+0000 mon.a (mon.0) 453 : cluster 0 osdmap e30: 4 total, 4 up, 4 in 2024-03-11T20:46:42.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:46:41 smithi023 bash[29146]: cluster 2024-03-11T20:46:40.885017+0000 mon.a (mon.0) 453 : cluster 0 osdmap e30: 4 total, 4 up, 4 in 2024-03-11T20:46:43.158 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:46:42 smithi146 bash[23809]: cluster 2024-03-11T20:46:41.382171+0000 mgr.y (mgr.14152) 208 : cluster 0 pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-11T20:46:43.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:46:42 smithi023 bash[21372]: cluster 2024-03-11T20:46:41.382171+0000 mgr.y (mgr.14152) 208 : cluster 0 pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-11T20:46:43.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:46:42 smithi023 bash[29146]: cluster 2024-03-11T20:46:41.382171+0000 mgr.y (mgr.14152) 208 : cluster 0 pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-11T20:46:45.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:46:44 smithi146 bash[23809]: cluster 2024-03-11T20:46:43.382726+0000 mgr.y (mgr.14152) 209 : cluster 0 pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-11T20:46:45.282 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:46:44 smithi023 bash[29146]: cluster 2024-03-11T20:46:43.382726+0000 mgr.y (mgr.14152) 209 : cluster 0 pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-11T20:46:45.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:46:44 smithi023 bash[21372]: cluster 2024-03-11T20:46:43.382726+0000 mgr.y (mgr.14152) 209 : cluster 0 pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-11T20:46:46.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:46:45 smithi146 bash[23809]: audit 2024-03-11T20:46:45.436902+0000 mon.a (mon.0) 454 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4dcbc91e-7035-413d-97bc-c9d804eb16d7"}]: dispatch 2024-03-11T20:46:46.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:46:45 smithi146 bash[23809]: audit 2024-03-11T20:46:45.438060+0000 mon.b (mon.1) 9 : audit 1 from='client.? 172.21.15.146:0/3908171123' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4dcbc91e-7035-413d-97bc-c9d804eb16d7"}]: dispatch 2024-03-11T20:46:46.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:46:45 smithi146 bash[23809]: audit 2024-03-11T20:46:45.444559+0000 mon.a (mon.0) 455 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4dcbc91e-7035-413d-97bc-c9d804eb16d7"}]': finished 2024-03-11T20:46:46.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:46:45 smithi146 bash[23809]: cluster 2024-03-11T20:46:45.450600+0000 mon.a (mon.0) 456 : cluster 0 osdmap e31: 5 total, 4 up, 5 in 2024-03-11T20:46:46.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:46:45 smithi146 bash[23809]: audit 2024-03-11T20:46:45.451303+0000 mon.a (mon.0) 457 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T20:46:46.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:46:45 smithi023 bash[29146]: audit 2024-03-11T20:46:45.436902+0000 mon.a (mon.0) 454 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4dcbc91e-7035-413d-97bc-c9d804eb16d7"}]: dispatch 2024-03-11T20:46:46.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:46:45 smithi023 bash[29146]: audit 2024-03-11T20:46:45.438060+0000 mon.b (mon.1) 9 : audit 1 from='client.? 172.21.15.146:0/3908171123' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4dcbc91e-7035-413d-97bc-c9d804eb16d7"}]: dispatch 2024-03-11T20:46:46.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:46:45 smithi023 bash[29146]: audit 2024-03-11T20:46:45.444559+0000 mon.a (mon.0) 455 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4dcbc91e-7035-413d-97bc-c9d804eb16d7"}]': finished 2024-03-11T20:46:46.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:46:45 smithi023 bash[29146]: cluster 2024-03-11T20:46:45.450600+0000 mon.a (mon.0) 456 : cluster 0 osdmap e31: 5 total, 4 up, 5 in 2024-03-11T20:46:46.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:46:45 smithi023 bash[29146]: audit 2024-03-11T20:46:45.451303+0000 mon.a (mon.0) 457 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T20:46:46.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:46:45 smithi023 bash[21372]: audit 2024-03-11T20:46:45.436902+0000 mon.a (mon.0) 454 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4dcbc91e-7035-413d-97bc-c9d804eb16d7"}]: dispatch 2024-03-11T20:46:46.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:46:45 smithi023 bash[21372]: audit 2024-03-11T20:46:45.438060+0000 mon.b (mon.1) 9 : audit 1 from='client.? 172.21.15.146:0/3908171123' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4dcbc91e-7035-413d-97bc-c9d804eb16d7"}]: dispatch 2024-03-11T20:46:46.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:46:45 smithi023 bash[21372]: audit 2024-03-11T20:46:45.444559+0000 mon.a (mon.0) 455 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4dcbc91e-7035-413d-97bc-c9d804eb16d7"}]': finished 2024-03-11T20:46:46.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:46:45 smithi023 bash[21372]: cluster 2024-03-11T20:46:45.450600+0000 mon.a (mon.0) 456 : cluster 0 osdmap e31: 5 total, 4 up, 5 in 2024-03-11T20:46:46.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:46:45 smithi023 bash[21372]: audit 2024-03-11T20:46:45.451303+0000 mon.a (mon.0) 457 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T20:46:47.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:46:46 smithi146 bash[23809]: cluster 2024-03-11T20:46:45.383278+0000 mgr.y (mgr.14152) 210 : cluster 0 pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-11T20:46:47.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:46:46 smithi146 bash[23809]: audit 2024-03-11T20:46:46.188435+0000 mon.b (mon.1) 10 : audit 0 from='client.? 172.21.15.146:0/1559005390' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T20:46:47.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:46:46 smithi023 bash[21372]: cluster 2024-03-11T20:46:45.383278+0000 mgr.y (mgr.14152) 210 : cluster 0 pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-11T20:46:47.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:46:46 smithi023 bash[21372]: audit 2024-03-11T20:46:46.188435+0000 mon.b (mon.1) 10 : audit 0 from='client.? 172.21.15.146:0/1559005390' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T20:46:47.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:46:46 smithi023 bash[29146]: cluster 2024-03-11T20:46:45.383278+0000 mgr.y (mgr.14152) 210 : cluster 0 pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-11T20:46:47.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:46:46 smithi023 bash[29146]: audit 2024-03-11T20:46:46.188435+0000 mon.b (mon.1) 10 : audit 0 from='client.? 172.21.15.146:0/1559005390' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T20:46:48.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:46:47 smithi146 bash[23809]: cluster 2024-03-11T20:46:46.956575+0000 osd.1 (osd.1) 5 : cluster 0 1.0 scrub starts 2024-03-11T20:46:48.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:46:47 smithi146 bash[23809]: cluster 2024-03-11T20:46:46.960649+0000 osd.1 (osd.1) 6 : cluster 0 1.0 scrub ok 2024-03-11T20:46:48.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:46:47 smithi023 bash[21372]: cluster 2024-03-11T20:46:46.956575+0000 osd.1 (osd.1) 5 : cluster 0 1.0 scrub starts 2024-03-11T20:46:48.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:46:47 smithi023 bash[21372]: cluster 2024-03-11T20:46:46.960649+0000 osd.1 (osd.1) 6 : cluster 0 1.0 scrub ok 2024-03-11T20:46:48.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:46:47 smithi023 bash[29146]: cluster 2024-03-11T20:46:46.956575+0000 osd.1 (osd.1) 5 : cluster 0 1.0 scrub starts 2024-03-11T20:46:48.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:46:47 smithi023 bash[29146]: cluster 2024-03-11T20:46:46.960649+0000 osd.1 (osd.1) 6 : cluster 0 1.0 scrub ok 2024-03-11T20:46:49.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:46:48 smithi146 bash[23809]: cluster 2024-03-11T20:46:47.383829+0000 mgr.y (mgr.14152) 211 : cluster 0 pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-11T20:46:49.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:46:48 smithi023 bash[21372]: cluster 2024-03-11T20:46:47.383829+0000 mgr.y (mgr.14152) 211 : cluster 0 pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-11T20:46:49.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:46:48 smithi023 bash[29146]: cluster 2024-03-11T20:46:47.383829+0000 mgr.y (mgr.14152) 211 : cluster 0 pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-11T20:46:51.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:46:50 smithi146 bash[23809]: cluster 2024-03-11T20:46:49.384318+0000 mgr.y (mgr.14152) 212 : cluster 0 pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-11T20:46:51.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:46:50 smithi023 bash[21372]: cluster 2024-03-11T20:46:49.384318+0000 mgr.y (mgr.14152) 212 : cluster 0 pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-11T20:46:51.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:46:50 smithi023 bash[29146]: cluster 2024-03-11T20:46:49.384318+0000 mgr.y (mgr.14152) 212 : cluster 0 pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-11T20:46:53.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:46:52 smithi146 bash[23809]: cluster 2024-03-11T20:46:51.384879+0000 mgr.y (mgr.14152) 213 : cluster 0 pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-11T20:46:53.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:46:52 smithi023 bash[21372]: cluster 2024-03-11T20:46:51.384879+0000 mgr.y (mgr.14152) 213 : cluster 0 pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-11T20:46:53.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:46:52 smithi023 bash[29146]: cluster 2024-03-11T20:46:51.384879+0000 mgr.y (mgr.14152) 213 : cluster 0 pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-11T20:46:55.241 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:46:54 smithi146 bash[23809]: cluster 2024-03-11T20:46:53.385386+0000 mgr.y (mgr.14152) 214 : cluster 0 pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-11T20:46:55.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:46:54 smithi023 bash[21372]: cluster 2024-03-11T20:46:53.385386+0000 mgr.y (mgr.14152) 214 : cluster 0 pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-11T20:46:55.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:46:54 smithi023 bash[29146]: cluster 2024-03-11T20:46:53.385386+0000 mgr.y (mgr.14152) 214 : cluster 0 pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-11T20:46:57.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:46:57 smithi146 bash[23809]: cluster 2024-03-11T20:46:55.385958+0000 mgr.y (mgr.14152) 215 : cluster 0 pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-11T20:46:57.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:46:57 smithi023 bash[21372]: cluster 2024-03-11T20:46:55.385958+0000 mgr.y (mgr.14152) 215 : cluster 0 pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-11T20:46:57.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:46:57 smithi023 bash[29146]: cluster 2024-03-11T20:46:55.385958+0000 mgr.y (mgr.14152) 215 : cluster 0 pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-11T20:46:58.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:46:58 smithi146 bash[23809]: audit 2024-03-11T20:46:57.564246+0000 mon.a (mon.0) 458 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-11T20:46:58.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:46:58 smithi146 bash[23809]: audit 2024-03-11T20:46:57.566112+0000 mon.a (mon.0) 459 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:46:58.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:46:58 smithi023 bash[29146]: audit 2024-03-11T20:46:57.564246+0000 mon.a (mon.0) 458 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-11T20:46:58.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:46:58 smithi023 bash[29146]: audit 2024-03-11T20:46:57.566112+0000 mon.a (mon.0) 459 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:46:58.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:46:58 smithi023 bash[21372]: audit 2024-03-11T20:46:57.564246+0000 mon.a (mon.0) 458 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-11T20:46:58.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:46:58 smithi023 bash[21372]: audit 2024-03-11T20:46:57.566112+0000 mon.a (mon.0) 459 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:46:59.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:46:59 smithi146 bash[23809]: cluster 2024-03-11T20:46:57.386447+0000 mgr.y (mgr.14152) 216 : cluster 0 pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-11T20:46:59.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:46:59 smithi146 bash[23809]: cephadm 2024-03-11T20:46:57.567978+0000 mgr.y (mgr.14152) 217 : cephadm 1 Deploying daemon osd.4 on smithi146 2024-03-11T20:46:59.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:46:59 smithi023 bash[21372]: cluster 2024-03-11T20:46:57.386447+0000 mgr.y (mgr.14152) 216 : cluster 0 pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-11T20:46:59.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:46:59 smithi023 bash[21372]: cephadm 2024-03-11T20:46:57.567978+0000 mgr.y (mgr.14152) 217 : cephadm 1 Deploying daemon osd.4 on smithi146 2024-03-11T20:46:59.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:46:59 smithi023 bash[29146]: cluster 2024-03-11T20:46:57.386447+0000 mgr.y (mgr.14152) 216 : cluster 0 pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-11T20:46:59.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:46:59 smithi023 bash[29146]: cephadm 2024-03-11T20:46:57.567978+0000 mgr.y (mgr.14152) 217 : cephadm 1 Deploying daemon osd.4 on smithi146 2024-03-11T20:47:01.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:47:01 smithi023 bash[21372]: cluster 2024-03-11T20:46:59.386983+0000 mgr.y (mgr.14152) 218 : cluster 0 pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-11T20:47:01.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:47:01 smithi023 bash[29146]: cluster 2024-03-11T20:46:59.386983+0000 mgr.y (mgr.14152) 218 : cluster 0 pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-11T20:47:01.362 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:47:01 smithi146 bash[23809]: cluster 2024-03-11T20:46:59.386983+0000 mgr.y (mgr.14152) 218 : cluster 0 pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-11T20:47:01.706 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Mar 11 20:47:01 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:47:01.706 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Mar 11 20:47:01 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:47:01.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:47:01 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:47:01.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:47:01 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:47:03.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:47:02 smithi146 bash[23809]: cluster 2024-03-11T20:47:01.060717+0000 osd.1 (osd.1) 7 : cluster 0 purged_snaps scrub starts 2024-03-11T20:47:03.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:47:02 smithi146 bash[23809]: cluster 2024-03-11T20:47:01.060814+0000 osd.1 (osd.1) 8 : cluster 0 purged_snaps scrub ok 2024-03-11T20:47:03.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:47:02 smithi146 bash[23809]: cluster 2024-03-11T20:47:01.387667+0000 mgr.y (mgr.14152) 219 : cluster 0 pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-11T20:47:03.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:47:02 smithi146 bash[23809]: audit 2024-03-11T20:47:01.766474+0000 mon.a (mon.0) 460 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:47:03.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:47:02 smithi146 bash[23809]: audit 2024-03-11T20:47:01.767211+0000 mon.a (mon.0) 461 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:47:03.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:47:02 smithi146 bash[23809]: audit 2024-03-11T20:47:01.784154+0000 mon.a (mon.0) 462 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:47:03.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:47:02 smithi146 bash[23809]: cluster 2024-03-11T20:47:02.054887+0000 mon.a (mon.0) 463 : cluster 0 osdmap e32: 5 total, 4 up, 5 in 2024-03-11T20:47:03.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:47:02 smithi146 bash[23809]: audit 2024-03-11T20:47:02.055143+0000 mon.a (mon.0) 464 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T20:47:03.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:47:02 smithi023 bash[21372]: cluster 2024-03-11T20:47:01.060717+0000 osd.1 (osd.1) 7 : cluster 0 purged_snaps scrub starts 2024-03-11T20:47:03.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:47:02 smithi023 bash[21372]: cluster 2024-03-11T20:47:01.060814+0000 osd.1 (osd.1) 8 : cluster 0 purged_snaps scrub ok 2024-03-11T20:47:03.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:47:02 smithi023 bash[21372]: cluster 2024-03-11T20:47:01.387667+0000 mgr.y (mgr.14152) 219 : cluster 0 pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-11T20:47:03.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:47:02 smithi023 bash[21372]: audit 2024-03-11T20:47:01.766474+0000 mon.a (mon.0) 460 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:47:03.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:47:02 smithi023 bash[21372]: audit 2024-03-11T20:47:01.767211+0000 mon.a (mon.0) 461 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:47:03.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:47:02 smithi023 bash[21372]: audit 2024-03-11T20:47:01.784154+0000 mon.a (mon.0) 462 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:47:03.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:47:02 smithi023 bash[21372]: cluster 2024-03-11T20:47:02.054887+0000 mon.a (mon.0) 463 : cluster 0 osdmap e32: 5 total, 4 up, 5 in 2024-03-11T20:47:03.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:47:02 smithi023 bash[21372]: audit 2024-03-11T20:47:02.055143+0000 mon.a (mon.0) 464 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T20:47:03.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:47:02 smithi023 bash[29146]: cluster 2024-03-11T20:47:01.060717+0000 osd.1 (osd.1) 7 : cluster 0 purged_snaps scrub starts 2024-03-11T20:47:03.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:47:02 smithi023 bash[29146]: cluster 2024-03-11T20:47:01.060814+0000 osd.1 (osd.1) 8 : cluster 0 purged_snaps scrub ok 2024-03-11T20:47:03.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:47:02 smithi023 bash[29146]: cluster 2024-03-11T20:47:01.387667+0000 mgr.y (mgr.14152) 219 : cluster 0 pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-11T20:47:03.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:47:02 smithi023 bash[29146]: audit 2024-03-11T20:47:01.766474+0000 mon.a (mon.0) 460 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:47:03.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:47:02 smithi023 bash[29146]: audit 2024-03-11T20:47:01.767211+0000 mon.a (mon.0) 461 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:47:03.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:47:02 smithi023 bash[29146]: audit 2024-03-11T20:47:01.784154+0000 mon.a (mon.0) 462 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:47:03.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:47:02 smithi023 bash[29146]: cluster 2024-03-11T20:47:02.054887+0000 mon.a (mon.0) 463 : cluster 0 osdmap e32: 5 total, 4 up, 5 in 2024-03-11T20:47:03.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:47:02 smithi023 bash[29146]: audit 2024-03-11T20:47:02.055143+0000 mon.a (mon.0) 464 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T20:47:05.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:47:04 smithi023 bash[21372]: cluster 2024-03-11T20:47:03.388320+0000 mgr.y (mgr.14152) 220 : cluster 0 pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-11T20:47:05.033 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:47:04 smithi023 bash[29146]: cluster 2024-03-11T20:47:03.388320+0000 mgr.y (mgr.14152) 220 : cluster 0 pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-11T20:47:05.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:47:04 smithi146 bash[23809]: cluster 2024-03-11T20:47:03.388320+0000 mgr.y (mgr.14152) 220 : cluster 0 pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-11T20:47:07.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:47:06 smithi023 bash[21372]: cluster 2024-03-11T20:47:05.388861+0000 mgr.y (mgr.14152) 221 : cluster 0 pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-11T20:47:07.033 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:47:06 smithi023 bash[29146]: cluster 2024-03-11T20:47:05.388861+0000 mgr.y (mgr.14152) 221 : cluster 0 pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-11T20:47:07.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:47:06 smithi146 bash[23809]: cluster 2024-03-11T20:47:05.388861+0000 mgr.y (mgr.14152) 221 : cluster 0 pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-11T20:47:09.111 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:47:08 smithi146 bash[23809]: cluster 2024-03-11T20:47:07.389489+0000 mgr.y (mgr.14152) 222 : cluster 0 pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-11T20:47:09.111 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:47:08 smithi146 bash[23809]: audit 2024-03-11T20:47:07.877449+0000 mon.a (mon.0) 465 : audit 1 from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-11T20:47:09.111 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:47:08 smithi146 bash[23809]: audit 2024-03-11T20:47:07.878788+0000 mon.b (mon.1) 11 : audit 1 from='osd.4 [v2:172.21.15.146:6800/615746847,v1:172.21.15.146:6801/615746847]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-11T20:47:09.111 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:47:08 smithi146 bash[23809]: audit 2024-03-11T20:47:08.574822+0000 mon.a (mon.0) 466 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:47:09.111 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:47:08 smithi146 bash[23809]: audit 2024-03-11T20:47:08.585057+0000 mon.a (mon.0) 467 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:47:09.111 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:47:08 smithi146 bash[23809]: audit 2024-03-11T20:47:08.587040+0000 mon.a (mon.0) 468 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:47:09.111 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:47:08 smithi146 bash[23809]: audit 2024-03-11T20:47:08.589025+0000 mon.a (mon.0) 469 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:47:09.111 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:47:08 smithi146 bash[23809]: audit 2024-03-11T20:47:08.599521+0000 mon.a (mon.0) 470 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:47:09.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:47:08 smithi023 bash[21372]: cluster 2024-03-11T20:47:07.389489+0000 mgr.y (mgr.14152) 222 : cluster 0 pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-11T20:47:09.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:47:08 smithi023 bash[21372]: audit 2024-03-11T20:47:07.877449+0000 mon.a (mon.0) 465 : audit 1 from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-11T20:47:09.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:47:08 smithi023 bash[21372]: audit 2024-03-11T20:47:07.878788+0000 mon.b (mon.1) 11 : audit 1 from='osd.4 [v2:172.21.15.146:6800/615746847,v1:172.21.15.146:6801/615746847]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-11T20:47:09.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:47:08 smithi023 bash[21372]: audit 2024-03-11T20:47:08.574822+0000 mon.a (mon.0) 466 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:47:09.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:47:08 smithi023 bash[21372]: audit 2024-03-11T20:47:08.585057+0000 mon.a (mon.0) 467 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:47:09.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:47:08 smithi023 bash[21372]: audit 2024-03-11T20:47:08.587040+0000 mon.a (mon.0) 468 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:47:09.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:47:08 smithi023 bash[21372]: audit 2024-03-11T20:47:08.589025+0000 mon.a (mon.0) 469 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:47:09.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:47:08 smithi023 bash[21372]: audit 2024-03-11T20:47:08.599521+0000 mon.a (mon.0) 470 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:47:09.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:47:08 smithi023 bash[29146]: cluster 2024-03-11T20:47:07.389489+0000 mgr.y (mgr.14152) 222 : cluster 0 pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-11T20:47:09.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:47:08 smithi023 bash[29146]: audit 2024-03-11T20:47:07.877449+0000 mon.a (mon.0) 465 : audit 1 from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-11T20:47:09.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:47:08 smithi023 bash[29146]: audit 2024-03-11T20:47:07.878788+0000 mon.b (mon.1) 11 : audit 1 from='osd.4 [v2:172.21.15.146:6800/615746847,v1:172.21.15.146:6801/615746847]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-11T20:47:09.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:47:08 smithi023 bash[29146]: audit 2024-03-11T20:47:08.574822+0000 mon.a (mon.0) 466 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:47:09.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:47:08 smithi023 bash[29146]: audit 2024-03-11T20:47:08.585057+0000 mon.a (mon.0) 467 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:47:09.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:47:08 smithi023 bash[29146]: audit 2024-03-11T20:47:08.587040+0000 mon.a (mon.0) 468 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:47:09.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:47:08 smithi023 bash[29146]: audit 2024-03-11T20:47:08.589025+0000 mon.a (mon.0) 469 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:47:09.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:47:08 smithi023 bash[29146]: audit 2024-03-11T20:47:08.599521+0000 mon.a (mon.0) 470 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:47:10.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:47:09 smithi146 bash[23809]: cluster 2024-03-11T20:47:08.508655+0000 osd.0 (osd.0) 7 : cluster 0 purged_snaps scrub starts 2024-03-11T20:47:10.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:47:09 smithi146 bash[23809]: cluster 2024-03-11T20:47:08.508742+0000 osd.0 (osd.0) 8 : cluster 0 purged_snaps scrub ok 2024-03-11T20:47:10.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:47:09 smithi146 bash[23809]: audit 2024-03-11T20:47:08.797912+0000 mon.a (mon.0) 471 : audit 1 from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-11T20:47:10.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:47:09 smithi146 bash[23809]: cluster 2024-03-11T20:47:08.807186+0000 mon.a (mon.0) 472 : cluster 0 osdmap e33: 5 total, 4 up, 5 in 2024-03-11T20:47:10.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:47:09 smithi146 bash[23809]: audit 2024-03-11T20:47:08.808786+0000 mon.b (mon.1) 12 : audit 1 from='osd.4 [v2:172.21.15.146:6800/615746847,v1:172.21.15.146:6801/615746847]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-03-11T20:47:10.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:47:09 smithi146 bash[23809]: audit 2024-03-11T20:47:08.813093+0000 mon.a (mon.0) 473 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T20:47:10.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:47:09 smithi146 bash[23809]: audit 2024-03-11T20:47:08.813864+0000 mon.a (mon.0) 474 : audit 1 from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-03-11T20:47:10.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:47:09 smithi146 bash[23809]: audit 2024-03-11T20:47:09.800974+0000 mon.a (mon.0) 475 : audit 1 from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]': finished 2024-03-11T20:47:10.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:47:09 smithi146 bash[23809]: cluster 2024-03-11T20:47:09.810868+0000 mon.a (mon.0) 476 : cluster 0 osdmap e34: 5 total, 4 up, 5 in 2024-03-11T20:47:10.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:47:09 smithi146 bash[23809]: audit 2024-03-11T20:47:09.811737+0000 mon.a (mon.0) 477 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T20:47:10.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:47:09 smithi146 bash[23809]: audit 2024-03-11T20:47:09.816646+0000 mon.a (mon.0) 478 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T20:47:10.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:47:09 smithi023 bash[21372]: cluster 2024-03-11T20:47:08.508655+0000 osd.0 (osd.0) 7 : cluster 0 purged_snaps scrub starts 2024-03-11T20:47:10.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:47:09 smithi023 bash[21372]: cluster 2024-03-11T20:47:08.508742+0000 osd.0 (osd.0) 8 : cluster 0 purged_snaps scrub ok 2024-03-11T20:47:10.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:47:09 smithi023 bash[21372]: audit 2024-03-11T20:47:08.797912+0000 mon.a (mon.0) 471 : audit 1 from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-11T20:47:10.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:47:09 smithi023 bash[21372]: cluster 2024-03-11T20:47:08.807186+0000 mon.a (mon.0) 472 : cluster 0 osdmap e33: 5 total, 4 up, 5 in 2024-03-11T20:47:10.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:47:09 smithi023 bash[21372]: audit 2024-03-11T20:47:08.808786+0000 mon.b (mon.1) 12 : audit 1 from='osd.4 [v2:172.21.15.146:6800/615746847,v1:172.21.15.146:6801/615746847]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-03-11T20:47:10.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:47:09 smithi023 bash[21372]: audit 2024-03-11T20:47:08.813093+0000 mon.a (mon.0) 473 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T20:47:10.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:47:09 smithi023 bash[21372]: audit 2024-03-11T20:47:08.813864+0000 mon.a (mon.0) 474 : audit 1 from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-03-11T20:47:10.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:47:09 smithi023 bash[21372]: audit 2024-03-11T20:47:09.800974+0000 mon.a (mon.0) 475 : audit 1 from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]': finished 2024-03-11T20:47:10.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:47:09 smithi023 bash[21372]: cluster 2024-03-11T20:47:09.810868+0000 mon.a (mon.0) 476 : cluster 0 osdmap e34: 5 total, 4 up, 5 in 2024-03-11T20:47:10.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:47:09 smithi023 bash[21372]: audit 2024-03-11T20:47:09.811737+0000 mon.a (mon.0) 477 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T20:47:10.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:47:09 smithi023 bash[21372]: audit 2024-03-11T20:47:09.816646+0000 mon.a (mon.0) 478 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T20:47:10.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:47:09 smithi023 bash[29146]: cluster 2024-03-11T20:47:08.508655+0000 osd.0 (osd.0) 7 : cluster 0 purged_snaps scrub starts 2024-03-11T20:47:10.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:47:09 smithi023 bash[29146]: cluster 2024-03-11T20:47:08.508742+0000 osd.0 (osd.0) 8 : cluster 0 purged_snaps scrub ok 2024-03-11T20:47:10.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:47:09 smithi023 bash[29146]: audit 2024-03-11T20:47:08.797912+0000 mon.a (mon.0) 471 : audit 1 from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-11T20:47:10.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:47:09 smithi023 bash[29146]: cluster 2024-03-11T20:47:08.807186+0000 mon.a (mon.0) 472 : cluster 0 osdmap e33: 5 total, 4 up, 5 in 2024-03-11T20:47:10.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:47:09 smithi023 bash[29146]: audit 2024-03-11T20:47:08.808786+0000 mon.b (mon.1) 12 : audit 1 from='osd.4 [v2:172.21.15.146:6800/615746847,v1:172.21.15.146:6801/615746847]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-03-11T20:47:10.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:47:09 smithi023 bash[29146]: audit 2024-03-11T20:47:08.813093+0000 mon.a (mon.0) 473 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T20:47:10.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:47:09 smithi023 bash[29146]: audit 2024-03-11T20:47:08.813864+0000 mon.a (mon.0) 474 : audit 1 from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-03-11T20:47:10.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:47:09 smithi023 bash[29146]: audit 2024-03-11T20:47:09.800974+0000 mon.a (mon.0) 475 : audit 1 from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]': finished 2024-03-11T20:47:10.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:47:09 smithi023 bash[29146]: cluster 2024-03-11T20:47:09.810868+0000 mon.a (mon.0) 476 : cluster 0 osdmap e34: 5 total, 4 up, 5 in 2024-03-11T20:47:10.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:47:09 smithi023 bash[29146]: audit 2024-03-11T20:47:09.811737+0000 mon.a (mon.0) 477 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T20:47:10.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:47:09 smithi023 bash[29146]: audit 2024-03-11T20:47:09.816646+0000 mon.a (mon.0) 478 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T20:47:11.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:47:10 smithi146 bash[23809]: cluster 2024-03-11T20:47:08.898891+0000 osd.4 (osd.4) 1 : cluster 0 purged_snaps scrub starts 2024-03-11T20:47:11.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:47:10 smithi146 bash[23809]: cluster 2024-03-11T20:47:08.898987+0000 osd.4 (osd.4) 2 : cluster 0 purged_snaps scrub ok 2024-03-11T20:47:11.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:47:10 smithi146 bash[23809]: cluster 2024-03-11T20:47:09.390120+0000 mgr.y (mgr.14152) 223 : cluster 0 pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-11T20:47:11.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:47:10 smithi146 bash[23809]: audit 2024-03-11T20:47:10.339672+0000 mon.a (mon.0) 479 : audit 1 from='osd.4 ' entity='osd.4' 2024-03-11T20:47:11.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:47:10 smithi146 bash[23809]: cluster 2024-03-11T20:47:10.817169+0000 mon.a (mon.0) 480 : cluster 1 osd.4 [v2:172.21.15.146:6800/615746847,v1:172.21.15.146:6801/615746847] boot 2024-03-11T20:47:11.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:47:10 smithi146 bash[23809]: cluster 2024-03-11T20:47:10.817274+0000 mon.a (mon.0) 481 : cluster 0 osdmap e35: 5 total, 5 up, 5 in 2024-03-11T20:47:11.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:47:10 smithi146 bash[23809]: audit 2024-03-11T20:47:10.817652+0000 mon.a (mon.0) 482 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T20:47:11.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:47:10 smithi023 bash[21372]: cluster 2024-03-11T20:47:08.898891+0000 osd.4 (osd.4) 1 : cluster 0 purged_snaps scrub starts 2024-03-11T20:47:11.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:47:10 smithi023 bash[21372]: cluster 2024-03-11T20:47:08.898987+0000 osd.4 (osd.4) 2 : cluster 0 purged_snaps scrub ok 2024-03-11T20:47:11.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:47:10 smithi023 bash[21372]: cluster 2024-03-11T20:47:09.390120+0000 mgr.y (mgr.14152) 223 : cluster 0 pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-11T20:47:11.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:47:10 smithi023 bash[21372]: audit 2024-03-11T20:47:10.339672+0000 mon.a (mon.0) 479 : audit 1 from='osd.4 ' entity='osd.4' 2024-03-11T20:47:11.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:47:10 smithi023 bash[21372]: cluster 2024-03-11T20:47:10.817169+0000 mon.a (mon.0) 480 : cluster 1 osd.4 [v2:172.21.15.146:6800/615746847,v1:172.21.15.146:6801/615746847] boot 2024-03-11T20:47:11.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:47:10 smithi023 bash[21372]: cluster 2024-03-11T20:47:10.817274+0000 mon.a (mon.0) 481 : cluster 0 osdmap e35: 5 total, 5 up, 5 in 2024-03-11T20:47:11.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:47:10 smithi023 bash[21372]: audit 2024-03-11T20:47:10.817652+0000 mon.a (mon.0) 482 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T20:47:11.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:47:10 smithi023 bash[29146]: cluster 2024-03-11T20:47:08.898891+0000 osd.4 (osd.4) 1 : cluster 0 purged_snaps scrub starts 2024-03-11T20:47:11.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:47:10 smithi023 bash[29146]: cluster 2024-03-11T20:47:08.898987+0000 osd.4 (osd.4) 2 : cluster 0 purged_snaps scrub ok 2024-03-11T20:47:11.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:47:10 smithi023 bash[29146]: cluster 2024-03-11T20:47:09.390120+0000 mgr.y (mgr.14152) 223 : cluster 0 pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-11T20:47:11.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:47:10 smithi023 bash[29146]: audit 2024-03-11T20:47:10.339672+0000 mon.a (mon.0) 479 : audit 1 from='osd.4 ' entity='osd.4' 2024-03-11T20:47:11.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:47:10 smithi023 bash[29146]: cluster 2024-03-11T20:47:10.817169+0000 mon.a (mon.0) 480 : cluster 1 osd.4 [v2:172.21.15.146:6800/615746847,v1:172.21.15.146:6801/615746847] boot 2024-03-11T20:47:11.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:47:10 smithi023 bash[29146]: cluster 2024-03-11T20:47:10.817274+0000 mon.a (mon.0) 481 : cluster 0 osdmap e35: 5 total, 5 up, 5 in 2024-03-11T20:47:11.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:47:10 smithi023 bash[29146]: audit 2024-03-11T20:47:10.817652+0000 mon.a (mon.0) 482 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T20:47:11.924 INFO:teuthology.orchestra.run.smithi146.stdout:Created osd(s) 4 on host 'smithi146' 2024-03-11T20:47:12.555 DEBUG:teuthology.orchestra.run.smithi146:osd.4> sudo journalctl -f -n 0 -u ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@osd.4.service 2024-03-11T20:47:12.557 INFO:tasks.cephadm:Deploying osd.5 on smithi146 with /dev/vg_nvme/lv_3... 2024-03-11T20:47:12.557 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-11T20:47:13.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:47:12 smithi146 bash[23809]: cluster 2024-03-11T20:47:11.390728+0000 mgr.y (mgr.14152) 224 : cluster 0 pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-11T20:47:13.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:47:12 smithi146 bash[23809]: cluster 2024-03-11T20:47:11.817190+0000 mon.a (mon.0) 483 : cluster 0 osdmap e36: 5 total, 5 up, 5 in 2024-03-11T20:47:13.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:47:12 smithi146 bash[23809]: audit 2024-03-11T20:47:11.911388+0000 mon.a (mon.0) 484 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:47:13.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:47:12 smithi146 bash[23809]: audit 2024-03-11T20:47:11.920428+0000 mon.a (mon.0) 485 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:47:13.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:47:12 smithi146 bash[23809]: audit 2024-03-11T20:47:11.946558+0000 mon.a (mon.0) 486 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:47:13.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:47:12 smithi023 bash[29146]: cluster 2024-03-11T20:47:11.390728+0000 mgr.y (mgr.14152) 224 : cluster 0 pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-11T20:47:13.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:47:12 smithi023 bash[29146]: cluster 2024-03-11T20:47:11.817190+0000 mon.a (mon.0) 483 : cluster 0 osdmap e36: 5 total, 5 up, 5 in 2024-03-11T20:47:13.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:47:12 smithi023 bash[29146]: audit 2024-03-11T20:47:11.911388+0000 mon.a (mon.0) 484 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:47:13.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:47:12 smithi023 bash[29146]: audit 2024-03-11T20:47:11.920428+0000 mon.a (mon.0) 485 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:47:13.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:47:12 smithi023 bash[29146]: audit 2024-03-11T20:47:11.946558+0000 mon.a (mon.0) 486 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:47:13.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:47:12 smithi023 bash[21372]: cluster 2024-03-11T20:47:11.390728+0000 mgr.y (mgr.14152) 224 : cluster 0 pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-11T20:47:13.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:47:12 smithi023 bash[21372]: cluster 2024-03-11T20:47:11.817190+0000 mon.a (mon.0) 483 : cluster 0 osdmap e36: 5 total, 5 up, 5 in 2024-03-11T20:47:13.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:47:12 smithi023 bash[21372]: audit 2024-03-11T20:47:11.911388+0000 mon.a (mon.0) 484 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:47:13.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:47:12 smithi023 bash[21372]: audit 2024-03-11T20:47:11.920428+0000 mon.a (mon.0) 485 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:47:13.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:47:12 smithi023 bash[21372]: audit 2024-03-11T20:47:11.946558+0000 mon.a (mon.0) 486 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:47:14.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:47:13 smithi146 bash[23809]: cluster 2024-03-11T20:47:12.833933+0000 mon.a (mon.0) 487 : cluster 0 osdmap e37: 5 total, 5 up, 5 in 2024-03-11T20:47:14.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:47:13 smithi023 bash[21372]: cluster 2024-03-11T20:47:12.833933+0000 mon.a (mon.0) 487 : cluster 0 osdmap e37: 5 total, 5 up, 5 in 2024-03-11T20:47:14.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:47:13 smithi023 bash[29146]: cluster 2024-03-11T20:47:12.833933+0000 mon.a (mon.0) 487 : cluster 0 osdmap e37: 5 total, 5 up, 5 in 2024-03-11T20:47:15.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:47:14 smithi146 bash[23809]: cluster 2024-03-11T20:47:13.391296+0000 mgr.y (mgr.14152) 225 : cluster 0 pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T20:47:15.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:47:14 smithi146 bash[23809]: cluster 2024-03-11T20:47:13.856550+0000 mon.a (mon.0) 488 : cluster 0 osdmap e38: 5 total, 5 up, 5 in 2024-03-11T20:47:15.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:47:14 smithi023 bash[21372]: cluster 2024-03-11T20:47:13.391296+0000 mgr.y (mgr.14152) 225 : cluster 0 pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T20:47:15.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:47:14 smithi023 bash[21372]: cluster 2024-03-11T20:47:13.856550+0000 mon.a (mon.0) 488 : cluster 0 osdmap e38: 5 total, 5 up, 5 in 2024-03-11T20:47:15.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:47:14 smithi023 bash[29146]: cluster 2024-03-11T20:47:13.391296+0000 mgr.y (mgr.14152) 225 : cluster 0 pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T20:47:15.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:47:14 smithi023 bash[29146]: cluster 2024-03-11T20:47:13.856550+0000 mon.a (mon.0) 488 : cluster 0 osdmap e38: 5 total, 5 up, 5 in 2024-03-11T20:47:16.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:47:15 smithi146 bash[23809]: cluster 2024-03-11T20:47:14.858142+0000 mon.a (mon.0) 489 : cluster 0 osdmap e39: 5 total, 5 up, 5 in 2024-03-11T20:47:16.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:47:15 smithi023 bash[21372]: cluster 2024-03-11T20:47:14.858142+0000 mon.a (mon.0) 489 : cluster 0 osdmap e39: 5 total, 5 up, 5 in 2024-03-11T20:47:16.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:47:15 smithi023 bash[29146]: cluster 2024-03-11T20:47:14.858142+0000 mon.a (mon.0) 489 : cluster 0 osdmap e39: 5 total, 5 up, 5 in 2024-03-11T20:47:16.902 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T20:47:17.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:47:16 smithi146 bash[23809]: cluster 2024-03-11T20:47:15.391866+0000 mgr.y (mgr.14152) 226 : cluster 0 pgmap v206: 1 pgs: 1 peering; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-11T20:47:17.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:47:16 smithi023 bash[21372]: cluster 2024-03-11T20:47:15.391866+0000 mgr.y (mgr.14152) 226 : cluster 0 pgmap v206: 1 pgs: 1 peering; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-11T20:47:17.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:47:16 smithi023 bash[29146]: cluster 2024-03-11T20:47:15.391866+0000 mgr.y (mgr.14152) 226 : cluster 0 pgmap v206: 1 pgs: 1 peering; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-11T20:47:19.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:47:18 smithi146 bash[23809]: cluster 2024-03-11T20:47:17.392426+0000 mgr.y (mgr.14152) 227 : cluster 0 pgmap v207: 1 pgs: 1 peering; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 103 KiB/s, 0 objects/s recovering 2024-03-11T20:47:19.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:47:18 smithi023 bash[21372]: cluster 2024-03-11T20:47:17.392426+0000 mgr.y (mgr.14152) 227 : cluster 0 pgmap v207: 1 pgs: 1 peering; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 103 KiB/s, 0 objects/s recovering 2024-03-11T20:47:19.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:47:18 smithi023 bash[29146]: cluster 2024-03-11T20:47:17.392426+0000 mgr.y (mgr.14152) 227 : cluster 0 pgmap v207: 1 pgs: 1 peering; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 103 KiB/s, 0 objects/s recovering 2024-03-11T20:47:20.716 INFO:teuthology.orchestra.run.smithi146.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-11T20:47:20.716 INFO:teuthology.orchestra.run.smithi146.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-11T20:47:20.716 INFO:teuthology.orchestra.run.smithi146.stderr: stderr: 10+0 records in 2024-03-11T20:47:20.716 INFO:teuthology.orchestra.run.smithi146.stderr:10+0 records out 2024-03-11T20:47:20.716 INFO:teuthology.orchestra.run.smithi146.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0179813 s, 583 MB/s 2024-03-11T20:47:20.716 INFO:teuthology.orchestra.run.smithi146.stderr:--> Zapping successful for: 2024-03-11T20:47:21.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:47:20 smithi146 bash[23809]: cluster 2024-03-11T20:47:19.393004+0000 mgr.y (mgr.14152) 228 : cluster 0 pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 88 KiB/s, 0 objects/s recovering 2024-03-11T20:47:21.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:47:20 smithi023 bash[21372]: cluster 2024-03-11T20:47:19.393004+0000 mgr.y (mgr.14152) 228 : cluster 0 pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 88 KiB/s, 0 objects/s recovering 2024-03-11T20:47:21.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:47:20 smithi023 bash[29146]: cluster 2024-03-11T20:47:19.393004+0000 mgr.y (mgr.14152) 228 : cluster 0 pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 88 KiB/s, 0 objects/s recovering 2024-03-11T20:47:21.545 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi146:vg_nvme/lv_3 2024-03-11T20:47:23.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:47:23 smithi146 bash[23809]: cluster 2024-03-11T20:47:21.393543+0000 mgr.y (mgr.14152) 229 : cluster 0 pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-11T20:47:23.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:47:23 smithi146 bash[23809]: cephadm 2024-03-11T20:47:22.043634+0000 mgr.y (mgr.14152) 230 : cephadm 1 Detected new or changed devices on smithi146 2024-03-11T20:47:23.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:47:23 smithi146 bash[23809]: audit 2024-03-11T20:47:22.058852+0000 mon.a (mon.0) 490 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:47:23.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:47:23 smithi146 bash[23809]: audit 2024-03-11T20:47:22.068296+0000 mon.a (mon.0) 491 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:47:23.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:47:23 smithi146 bash[23809]: audit 2024-03-11T20:47:22.070643+0000 mon.a (mon.0) 492 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:47:23.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:47:23 smithi146 bash[23809]: audit 2024-03-11T20:47:22.078807+0000 mon.a (mon.0) 493 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:47:23.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:47:23 smithi146 bash[23809]: audit 2024-03-11T20:47:22.083541+0000 mon.a (mon.0) 494 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:47:23.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:47:23 smithi146 bash[23809]: audit 2024-03-11T20:47:22.085445+0000 mon.a (mon.0) 495 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:47:23.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:47:23 smithi146 bash[23809]: audit 2024-03-11T20:47:22.095995+0000 mon.a (mon.0) 496 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:47:23.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:47:23 smithi023 bash[21372]: cluster 2024-03-11T20:47:21.393543+0000 mgr.y (mgr.14152) 229 : cluster 0 pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-11T20:47:23.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:47:23 smithi023 bash[21372]: cephadm 2024-03-11T20:47:22.043634+0000 mgr.y (mgr.14152) 230 : cephadm 1 Detected new or changed devices on smithi146 2024-03-11T20:47:23.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:47:23 smithi023 bash[21372]: audit 2024-03-11T20:47:22.058852+0000 mon.a (mon.0) 490 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:47:23.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:47:23 smithi023 bash[21372]: audit 2024-03-11T20:47:22.068296+0000 mon.a (mon.0) 491 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:47:23.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:47:23 smithi023 bash[21372]: audit 2024-03-11T20:47:22.070643+0000 mon.a (mon.0) 492 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:47:23.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:47:23 smithi023 bash[21372]: audit 2024-03-11T20:47:22.078807+0000 mon.a (mon.0) 493 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:47:23.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:47:23 smithi023 bash[21372]: audit 2024-03-11T20:47:22.083541+0000 mon.a (mon.0) 494 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:47:23.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:47:23 smithi023 bash[21372]: audit 2024-03-11T20:47:22.085445+0000 mon.a (mon.0) 495 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:47:23.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:47:23 smithi023 bash[21372]: audit 2024-03-11T20:47:22.095995+0000 mon.a (mon.0) 496 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:47:23.534 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:47:23 smithi023 bash[29146]: cluster 2024-03-11T20:47:21.393543+0000 mgr.y (mgr.14152) 229 : cluster 0 pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-11T20:47:23.534 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:47:23 smithi023 bash[29146]: cephadm 2024-03-11T20:47:22.043634+0000 mgr.y (mgr.14152) 230 : cephadm 1 Detected new or changed devices on smithi146 2024-03-11T20:47:23.534 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:47:23 smithi023 bash[29146]: audit 2024-03-11T20:47:22.058852+0000 mon.a (mon.0) 490 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:47:23.534 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:47:23 smithi023 bash[29146]: audit 2024-03-11T20:47:22.068296+0000 mon.a (mon.0) 491 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:47:23.534 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:47:23 smithi023 bash[29146]: audit 2024-03-11T20:47:22.070643+0000 mon.a (mon.0) 492 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:47:23.534 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:47:23 smithi023 bash[29146]: audit 2024-03-11T20:47:22.078807+0000 mon.a (mon.0) 493 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:47:23.534 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:47:23 smithi023 bash[29146]: audit 2024-03-11T20:47:22.083541+0000 mon.a (mon.0) 494 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:47:23.534 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:47:23 smithi023 bash[29146]: audit 2024-03-11T20:47:22.085445+0000 mon.a (mon.0) 495 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:47:23.534 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:47:23 smithi023 bash[29146]: audit 2024-03-11T20:47:22.095995+0000 mon.a (mon.0) 496 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:47:24.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:47:24 smithi146 bash[23809]: cephadm 2024-03-11T20:47:22.072061+0000 mgr.y (mgr.14152) 231 : cephadm 1 Adjusting osd_memory_target on smithi146 to 17265M 2024-03-11T20:47:24.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:47:24 smithi023 bash[21372]: cephadm 2024-03-11T20:47:22.072061+0000 mgr.y (mgr.14152) 231 : cephadm 1 Adjusting osd_memory_target on smithi146 to 17265M 2024-03-11T20:47:24.533 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:47:24 smithi023 bash[29146]: cephadm 2024-03-11T20:47:22.072061+0000 mgr.y (mgr.14152) 231 : cephadm 1 Adjusting osd_memory_target on smithi146 to 17265M 2024-03-11T20:47:25.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:47:25 smithi146 bash[23809]: cluster 2024-03-11T20:47:23.394218+0000 mgr.y (mgr.14152) 232 : cluster 0 pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-03-11T20:47:25.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:47:25 smithi023 bash[21372]: cluster 2024-03-11T20:47:23.394218+0000 mgr.y (mgr.14152) 232 : cluster 0 pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-03-11T20:47:25.533 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:47:25 smithi023 bash[29146]: cluster 2024-03-11T20:47:23.394218+0000 mgr.y (mgr.14152) 232 : cluster 0 pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-03-11T20:47:26.258 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T20:47:26.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:47:26 smithi146 bash[23809]: cluster 2024-03-11T20:47:25.394811+0000 mgr.y (mgr.14152) 233 : cluster 0 pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T20:47:26.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:47:26 smithi023 bash[21372]: cluster 2024-03-11T20:47:25.394811+0000 mgr.y (mgr.14152) 233 : cluster 0 pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T20:47:26.533 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:47:26 smithi023 bash[29146]: cluster 2024-03-11T20:47:25.394811+0000 mgr.y (mgr.14152) 233 : cluster 0 pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T20:47:28.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:47:28 smithi146 bash[23809]: cluster 2024-03-11T20:47:27.395303+0000 mgr.y (mgr.14152) 234 : cluster 0 pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T20:47:28.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:47:28 smithi023 bash[21372]: cluster 2024-03-11T20:47:27.395303+0000 mgr.y (mgr.14152) 234 : cluster 0 pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T20:47:28.783 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:47:28 smithi023 bash[29146]: cluster 2024-03-11T20:47:27.395303+0000 mgr.y (mgr.14152) 234 : cluster 0 pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T20:47:29.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:47:29 smithi146 bash[23809]: audit 2024-03-11T20:47:28.871819+0000 mgr.y (mgr.14152) 235 : audit 0 from='client.24211 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi146:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:47:29.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:47:29 smithi146 bash[23809]: audit 2024-03-11T20:47:28.878938+0000 mon.a (mon.0) 497 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T20:47:29.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:47:29 smithi146 bash[23809]: audit 2024-03-11T20:47:28.885960+0000 mon.a (mon.0) 498 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T20:47:29.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:47:29 smithi146 bash[23809]: audit 2024-03-11T20:47:28.887757+0000 mon.a (mon.0) 499 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:47:29.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:47:29 smithi023 bash[21372]: audit 2024-03-11T20:47:28.871819+0000 mgr.y (mgr.14152) 235 : audit 0 from='client.24211 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi146:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:47:29.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:47:29 smithi023 bash[21372]: audit 2024-03-11T20:47:28.878938+0000 mon.a (mon.0) 497 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T20:47:29.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:47:29 smithi023 bash[21372]: audit 2024-03-11T20:47:28.885960+0000 mon.a (mon.0) 498 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T20:47:29.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:47:29 smithi023 bash[21372]: audit 2024-03-11T20:47:28.887757+0000 mon.a (mon.0) 499 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:47:29.783 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:47:29 smithi023 bash[29146]: audit 2024-03-11T20:47:28.871819+0000 mgr.y (mgr.14152) 235 : audit 0 from='client.24211 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi146:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:47:29.783 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:47:29 smithi023 bash[29146]: audit 2024-03-11T20:47:28.878938+0000 mon.a (mon.0) 497 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T20:47:29.783 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:47:29 smithi023 bash[29146]: audit 2024-03-11T20:47:28.885960+0000 mon.a (mon.0) 498 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T20:47:29.784 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:47:29 smithi023 bash[29146]: audit 2024-03-11T20:47:28.887757+0000 mon.a (mon.0) 499 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:47:30.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:47:30 smithi146 bash[23809]: cluster 2024-03-11T20:47:29.395883+0000 mgr.y (mgr.14152) 236 : cluster 0 pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T20:47:30.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:47:30 smithi023 bash[21372]: cluster 2024-03-11T20:47:29.395883+0000 mgr.y (mgr.14152) 236 : cluster 0 pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T20:47:30.783 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:47:30 smithi023 bash[29146]: cluster 2024-03-11T20:47:29.395883+0000 mgr.y (mgr.14152) 236 : cluster 0 pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T20:47:32.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:47:32 smithi146 bash[23809]: cluster 2024-03-11T20:47:31.396442+0000 mgr.y (mgr.14152) 237 : cluster 0 pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T20:47:32.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:47:32 smithi023 bash[21372]: cluster 2024-03-11T20:47:31.396442+0000 mgr.y (mgr.14152) 237 : cluster 0 pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T20:47:32.783 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:47:32 smithi023 bash[29146]: cluster 2024-03-11T20:47:31.396442+0000 mgr.y (mgr.14152) 237 : cluster 0 pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T20:47:34.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:47:34 smithi023 bash[21372]: cluster 2024-03-11T20:47:33.397008+0000 mgr.y (mgr.14152) 238 : cluster 0 pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T20:47:34.783 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:47:34 smithi023 bash[29146]: cluster 2024-03-11T20:47:33.397008+0000 mgr.y (mgr.14152) 238 : cluster 0 pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T20:47:34.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:47:34 smithi146 bash[23809]: cluster 2024-03-11T20:47:33.397008+0000 mgr.y (mgr.14152) 238 : cluster 0 pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T20:47:36.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:47:36 smithi023 bash[21372]: cluster 2024-03-11T20:47:35.397567+0000 mgr.y (mgr.14152) 239 : cluster 0 pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T20:47:36.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:47:36 smithi023 bash[21372]: cluster 2024-03-11T20:47:35.529094+0000 osd.3 (osd.3) 3 : cluster 0 purged_snaps scrub starts 2024-03-11T20:47:36.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:47:36 smithi023 bash[21372]: cluster 2024-03-11T20:47:35.529193+0000 osd.3 (osd.3) 4 : cluster 0 purged_snaps scrub ok 2024-03-11T20:47:36.783 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:47:36 smithi023 bash[29146]: cluster 2024-03-11T20:47:35.397567+0000 mgr.y (mgr.14152) 239 : cluster 0 pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T20:47:36.783 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:47:36 smithi023 bash[29146]: cluster 2024-03-11T20:47:35.529094+0000 osd.3 (osd.3) 3 : cluster 0 purged_snaps scrub starts 2024-03-11T20:47:36.784 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:47:36 smithi023 bash[29146]: cluster 2024-03-11T20:47:35.529193+0000 osd.3 (osd.3) 4 : cluster 0 purged_snaps scrub ok 2024-03-11T20:47:37.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:47:36 smithi146 bash[23809]: cluster 2024-03-11T20:47:35.397567+0000 mgr.y (mgr.14152) 239 : cluster 0 pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T20:47:37.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:47:36 smithi146 bash[23809]: cluster 2024-03-11T20:47:35.529094+0000 osd.3 (osd.3) 3 : cluster 0 purged_snaps scrub starts 2024-03-11T20:47:37.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:47:36 smithi146 bash[23809]: cluster 2024-03-11T20:47:35.529193+0000 osd.3 (osd.3) 4 : cluster 0 purged_snaps scrub ok 2024-03-11T20:47:38.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:47:37 smithi146 bash[23809]: cluster 2024-03-11T20:47:36.550848+0000 mon.a (mon.0) 500 : cluster 0 osdmap e40: 5 total, 5 up, 5 in 2024-03-11T20:47:38.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:47:37 smithi146 bash[23809]: audit 2024-03-11T20:47:36.936113+0000 mon.a (mon.0) 501 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aefcb766-8bae-4ff4-b71e-6691c8ac02f1"}]: dispatch 2024-03-11T20:47:38.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:47:37 smithi146 bash[23809]: audit 2024-03-11T20:47:36.937437+0000 mon.b (mon.1) 13 : audit 1 from='client.? 172.21.15.146:0/3383526964' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aefcb766-8bae-4ff4-b71e-6691c8ac02f1"}]: dispatch 2024-03-11T20:47:38.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:47:37 smithi146 bash[23809]: audit 2024-03-11T20:47:36.942096+0000 mon.a (mon.0) 502 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "aefcb766-8bae-4ff4-b71e-6691c8ac02f1"}]': finished 2024-03-11T20:47:38.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:47:37 smithi146 bash[23809]: cluster 2024-03-11T20:47:36.947830+0000 mon.a (mon.0) 503 : cluster 0 osdmap e41: 6 total, 5 up, 6 in 2024-03-11T20:47:38.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:47:37 smithi146 bash[23809]: audit 2024-03-11T20:47:36.948266+0000 mon.a (mon.0) 504 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T20:47:38.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:47:37 smithi023 bash[21372]: cluster 2024-03-11T20:47:36.550848+0000 mon.a (mon.0) 500 : cluster 0 osdmap e40: 5 total, 5 up, 5 in 2024-03-11T20:47:38.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:47:37 smithi023 bash[21372]: audit 2024-03-11T20:47:36.936113+0000 mon.a (mon.0) 501 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aefcb766-8bae-4ff4-b71e-6691c8ac02f1"}]: dispatch 2024-03-11T20:47:38.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:47:37 smithi023 bash[21372]: audit 2024-03-11T20:47:36.937437+0000 mon.b (mon.1) 13 : audit 1 from='client.? 172.21.15.146:0/3383526964' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aefcb766-8bae-4ff4-b71e-6691c8ac02f1"}]: dispatch 2024-03-11T20:47:38.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:47:37 smithi023 bash[21372]: audit 2024-03-11T20:47:36.942096+0000 mon.a (mon.0) 502 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "aefcb766-8bae-4ff4-b71e-6691c8ac02f1"}]': finished 2024-03-11T20:47:38.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:47:37 smithi023 bash[21372]: cluster 2024-03-11T20:47:36.947830+0000 mon.a (mon.0) 503 : cluster 0 osdmap e41: 6 total, 5 up, 6 in 2024-03-11T20:47:38.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:47:37 smithi023 bash[21372]: audit 2024-03-11T20:47:36.948266+0000 mon.a (mon.0) 504 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T20:47:38.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:47:37 smithi023 bash[29146]: cluster 2024-03-11T20:47:36.550848+0000 mon.a (mon.0) 500 : cluster 0 osdmap e40: 5 total, 5 up, 5 in 2024-03-11T20:47:38.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:47:37 smithi023 bash[29146]: audit 2024-03-11T20:47:36.936113+0000 mon.a (mon.0) 501 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aefcb766-8bae-4ff4-b71e-6691c8ac02f1"}]: dispatch 2024-03-11T20:47:38.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:47:37 smithi023 bash[29146]: audit 2024-03-11T20:47:36.937437+0000 mon.b (mon.1) 13 : audit 1 from='client.? 172.21.15.146:0/3383526964' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aefcb766-8bae-4ff4-b71e-6691c8ac02f1"}]: dispatch 2024-03-11T20:47:38.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:47:37 smithi023 bash[29146]: audit 2024-03-11T20:47:36.942096+0000 mon.a (mon.0) 502 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "aefcb766-8bae-4ff4-b71e-6691c8ac02f1"}]': finished 2024-03-11T20:47:38.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:47:37 smithi023 bash[29146]: cluster 2024-03-11T20:47:36.947830+0000 mon.a (mon.0) 503 : cluster 0 osdmap e41: 6 total, 5 up, 6 in 2024-03-11T20:47:38.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:47:37 smithi023 bash[29146]: audit 2024-03-11T20:47:36.948266+0000 mon.a (mon.0) 504 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T20:47:39.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:47:38 smithi146 bash[23809]: cluster 2024-03-11T20:47:37.398073+0000 mgr.y (mgr.14152) 240 : cluster 0 pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T20:47:39.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:47:38 smithi146 bash[23809]: audit 2024-03-11T20:47:37.691820+0000 mon.b (mon.1) 14 : audit 0 from='client.? 172.21.15.146:0/4097390791' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T20:47:39.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:47:38 smithi023 bash[21372]: cluster 2024-03-11T20:47:37.398073+0000 mgr.y (mgr.14152) 240 : cluster 0 pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T20:47:39.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:47:38 smithi023 bash[21372]: audit 2024-03-11T20:47:37.691820+0000 mon.b (mon.1) 14 : audit 0 from='client.? 172.21.15.146:0/4097390791' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T20:47:39.033 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:47:38 smithi023 bash[29146]: cluster 2024-03-11T20:47:37.398073+0000 mgr.y (mgr.14152) 240 : cluster 0 pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T20:47:39.033 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:47:38 smithi023 bash[29146]: audit 2024-03-11T20:47:37.691820+0000 mon.b (mon.1) 14 : audit 0 from='client.? 172.21.15.146:0/4097390791' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T20:47:41.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:47:40 smithi146 bash[23809]: cluster 2024-03-11T20:47:39.398647+0000 mgr.y (mgr.14152) 241 : cluster 0 pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T20:47:41.033 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:47:40 smithi023 bash[29146]: cluster 2024-03-11T20:47:39.398647+0000 mgr.y (mgr.14152) 241 : cluster 0 pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T20:47:41.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:47:40 smithi023 bash[21372]: cluster 2024-03-11T20:47:39.398647+0000 mgr.y (mgr.14152) 241 : cluster 0 pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T20:47:43.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:47:42 smithi146 bash[23809]: cluster 2024-03-11T20:47:41.399189+0000 mgr.y (mgr.14152) 242 : cluster 0 pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T20:47:43.033 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:47:42 smithi023 bash[29146]: cluster 2024-03-11T20:47:41.399189+0000 mgr.y (mgr.14152) 242 : cluster 0 pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T20:47:43.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:47:42 smithi023 bash[21372]: cluster 2024-03-11T20:47:41.399189+0000 mgr.y (mgr.14152) 242 : cluster 0 pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T20:47:45.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:47:44 smithi146 bash[23809]: cluster 2024-03-11T20:47:43.399797+0000 mgr.y (mgr.14152) 243 : cluster 0 pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T20:47:45.033 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:47:44 smithi023 bash[29146]: cluster 2024-03-11T20:47:43.399797+0000 mgr.y (mgr.14152) 243 : cluster 0 pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T20:47:45.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:47:44 smithi023 bash[21372]: cluster 2024-03-11T20:47:43.399797+0000 mgr.y (mgr.14152) 243 : cluster 0 pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T20:47:46.857 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:47:46 smithi146 bash[23809]: cluster 2024-03-11T20:47:45.400327+0000 mgr.y (mgr.14152) 244 : cluster 0 pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T20:47:47.033 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:47:46 smithi023 bash[29146]: cluster 2024-03-11T20:47:45.400327+0000 mgr.y (mgr.14152) 244 : cluster 0 pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T20:47:47.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:47:46 smithi023 bash[21372]: cluster 2024-03-11T20:47:45.400327+0000 mgr.y (mgr.14152) 244 : cluster 0 pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T20:47:48.871 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:47:48 smithi146 bash[23809]: cluster 2024-03-11T20:47:47.400861+0000 mgr.y (mgr.14152) 245 : cluster 0 pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T20:47:49.033 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:47:48 smithi023 bash[29146]: cluster 2024-03-11T20:47:47.400861+0000 mgr.y (mgr.14152) 245 : cluster 0 pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T20:47:49.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:47:48 smithi023 bash[21372]: cluster 2024-03-11T20:47:47.400861+0000 mgr.y (mgr.14152) 245 : cluster 0 pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T20:47:50.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:47:49 smithi146 bash[23809]: audit 2024-03-11T20:47:49.368754+0000 mon.a (mon.0) 505 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-11T20:47:50.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:47:49 smithi146 bash[23809]: audit 2024-03-11T20:47:49.370480+0000 mon.a (mon.0) 506 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:47:50.033 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:47:49 smithi023 bash[29146]: audit 2024-03-11T20:47:49.368754+0000 mon.a (mon.0) 505 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-11T20:47:50.033 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:47:49 smithi023 bash[29146]: audit 2024-03-11T20:47:49.370480+0000 mon.a (mon.0) 506 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:47:50.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:47:49 smithi023 bash[21372]: audit 2024-03-11T20:47:49.368754+0000 mon.a (mon.0) 505 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-11T20:47:50.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:47:49 smithi023 bash[21372]: audit 2024-03-11T20:47:49.370480+0000 mon.a (mon.0) 506 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:47:50.962 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:47:50 smithi146 bash[23809]: cephadm 2024-03-11T20:47:49.372178+0000 mgr.y (mgr.14152) 246 : cephadm 1 Deploying daemon osd.5 on smithi146 2024-03-11T20:47:50.962 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:47:50 smithi146 bash[23809]: cluster 2024-03-11T20:47:49.401424+0000 mgr.y (mgr.14152) 247 : cluster 0 pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T20:47:51.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:47:50 smithi023 bash[21372]: cephadm 2024-03-11T20:47:49.372178+0000 mgr.y (mgr.14152) 246 : cephadm 1 Deploying daemon osd.5 on smithi146 2024-03-11T20:47:51.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:47:50 smithi023 bash[21372]: cluster 2024-03-11T20:47:49.401424+0000 mgr.y (mgr.14152) 247 : cluster 0 pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T20:47:51.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:47:50 smithi023 bash[29146]: cephadm 2024-03-11T20:47:49.372178+0000 mgr.y (mgr.14152) 246 : cephadm 1 Deploying daemon osd.5 on smithi146 2024-03-11T20:47:51.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:47:50 smithi023 bash[29146]: cluster 2024-03-11T20:47:49.401424+0000 mgr.y (mgr.14152) 247 : cluster 0 pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T20:47:52.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:47:52 smithi146 bash[23809]: cluster 2024-03-11T20:47:51.401959+0000 mgr.y (mgr.14152) 248 : cluster 0 pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T20:47:53.033 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:47:52 smithi023 bash[29146]: cluster 2024-03-11T20:47:51.401959+0000 mgr.y (mgr.14152) 248 : cluster 0 pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T20:47:53.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:47:52 smithi023 bash[21372]: cluster 2024-03-11T20:47:51.401959+0000 mgr.y (mgr.14152) 248 : cluster 0 pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T20:47:53.617 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:47:53 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:47:53.617 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Mar 11 20:47:53 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:47:53.618 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 11 20:47:53 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:47:54.010 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Mar 11 20:47:53 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:47:54.011 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 11 20:47:53 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:47:54.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:47:53 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:47:55.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:47:54 smithi146 bash[23809]: cluster 2024-03-11T20:47:53.402515+0000 mgr.y (mgr.14152) 249 : cluster 0 pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T20:47:55.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:47:54 smithi146 bash[23809]: audit 2024-03-11T20:47:53.918872+0000 mon.a (mon.0) 507 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:47:55.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:47:54 smithi146 bash[23809]: audit 2024-03-11T20:47:53.919829+0000 mon.a (mon.0) 508 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:47:55.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:47:54 smithi146 bash[23809]: audit 2024-03-11T20:47:53.937236+0000 mon.a (mon.0) 509 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:47:55.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:47:54 smithi023 bash[29146]: cluster 2024-03-11T20:47:53.402515+0000 mgr.y (mgr.14152) 249 : cluster 0 pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T20:47:55.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:47:54 smithi023 bash[29146]: audit 2024-03-11T20:47:53.918872+0000 mon.a (mon.0) 507 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:47:55.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:47:54 smithi023 bash[29146]: audit 2024-03-11T20:47:53.919829+0000 mon.a (mon.0) 508 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:47:55.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:47:54 smithi023 bash[29146]: audit 2024-03-11T20:47:53.937236+0000 mon.a (mon.0) 509 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:47:55.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:47:54 smithi023 bash[21372]: cluster 2024-03-11T20:47:53.402515+0000 mgr.y (mgr.14152) 249 : cluster 0 pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T20:47:55.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:47:54 smithi023 bash[21372]: audit 2024-03-11T20:47:53.918872+0000 mon.a (mon.0) 507 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:47:55.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:47:54 smithi023 bash[21372]: audit 2024-03-11T20:47:53.919829+0000 mon.a (mon.0) 508 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:47:55.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:47:54 smithi023 bash[21372]: audit 2024-03-11T20:47:53.937236+0000 mon.a (mon.0) 509 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:47:57.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:47:56 smithi146 bash[23809]: cluster 2024-03-11T20:47:55.403071+0000 mgr.y (mgr.14152) 250 : cluster 0 pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T20:47:57.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:47:56 smithi023 bash[29146]: cluster 2024-03-11T20:47:55.403071+0000 mgr.y (mgr.14152) 250 : cluster 0 pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T20:47:57.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:47:56 smithi023 bash[21372]: cluster 2024-03-11T20:47:55.403071+0000 mgr.y (mgr.14152) 250 : cluster 0 pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T20:47:59.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:47:58 smithi146 bash[23809]: cluster 2024-03-11T20:47:57.403625+0000 mgr.y (mgr.14152) 251 : cluster 0 pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T20:47:59.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:47:58 smithi023 bash[21372]: cluster 2024-03-11T20:47:57.403625+0000 mgr.y (mgr.14152) 251 : cluster 0 pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T20:47:59.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:47:58 smithi023 bash[29146]: cluster 2024-03-11T20:47:57.403625+0000 mgr.y (mgr.14152) 251 : cluster 0 pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T20:48:00.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:00 smithi146 bash[23809]: audit 2024-03-11T20:47:59.263490+0000 mon.a (mon.0) 510 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:48:00.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:00 smithi146 bash[23809]: audit 2024-03-11T20:47:59.273719+0000 mon.a (mon.0) 511 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:48:00.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:00 smithi146 bash[23809]: cluster 2024-03-11T20:47:59.404238+0000 mgr.y (mgr.14152) 252 : cluster 0 pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T20:48:00.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:00 smithi146 bash[23809]: audit 2024-03-11T20:47:59.959358+0000 mon.a (mon.0) 512 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:48:00.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:00 smithi146 bash[23809]: audit 2024-03-11T20:47:59.961476+0000 mon.a (mon.0) 513 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:48:00.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:00 smithi146 bash[23809]: audit 2024-03-11T20:47:59.976706+0000 mon.a (mon.0) 514 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:48:00.533 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:00 smithi023 bash[29146]: audit 2024-03-11T20:47:59.263490+0000 mon.a (mon.0) 510 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:48:00.533 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:00 smithi023 bash[29146]: audit 2024-03-11T20:47:59.273719+0000 mon.a (mon.0) 511 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:48:00.533 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:00 smithi023 bash[29146]: cluster 2024-03-11T20:47:59.404238+0000 mgr.y (mgr.14152) 252 : cluster 0 pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T20:48:00.533 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:00 smithi023 bash[29146]: audit 2024-03-11T20:47:59.959358+0000 mon.a (mon.0) 512 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:48:00.533 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:00 smithi023 bash[29146]: audit 2024-03-11T20:47:59.961476+0000 mon.a (mon.0) 513 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:48:00.533 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:00 smithi023 bash[29146]: audit 2024-03-11T20:47:59.976706+0000 mon.a (mon.0) 514 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:48:00.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:00 smithi023 bash[21372]: audit 2024-03-11T20:47:59.263490+0000 mon.a (mon.0) 510 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:48:00.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:00 smithi023 bash[21372]: audit 2024-03-11T20:47:59.273719+0000 mon.a (mon.0) 511 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:48:00.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:00 smithi023 bash[21372]: cluster 2024-03-11T20:47:59.404238+0000 mgr.y (mgr.14152) 252 : cluster 0 pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T20:48:00.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:00 smithi023 bash[21372]: audit 2024-03-11T20:47:59.959358+0000 mon.a (mon.0) 512 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:48:00.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:00 smithi023 bash[21372]: audit 2024-03-11T20:47:59.961476+0000 mon.a (mon.0) 513 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:48:00.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:00 smithi023 bash[21372]: audit 2024-03-11T20:47:59.976706+0000 mon.a (mon.0) 514 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:48:01.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:01 smithi146 bash[23809]: audit 2024-03-11T20:48:00.621952+0000 mon.a (mon.0) 515 : audit 1 from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-11T20:48:01.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:01 smithi146 bash[23809]: audit 2024-03-11T20:48:00.623887+0000 mon.b (mon.1) 15 : audit 1 from='osd.5 [v2:172.21.15.146:6808/4003230814,v1:172.21.15.146:6809/4003230814]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-11T20:48:01.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:01 smithi023 bash[21372]: audit 2024-03-11T20:48:00.621952+0000 mon.a (mon.0) 515 : audit 1 from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-11T20:48:01.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:01 smithi023 bash[21372]: audit 2024-03-11T20:48:00.623887+0000 mon.b (mon.1) 15 : audit 1 from='osd.5 [v2:172.21.15.146:6808/4003230814,v1:172.21.15.146:6809/4003230814]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-11T20:48:01.783 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:01 smithi023 bash[29146]: audit 2024-03-11T20:48:00.621952+0000 mon.a (mon.0) 515 : audit 1 from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-11T20:48:01.783 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:01 smithi023 bash[29146]: audit 2024-03-11T20:48:00.623887+0000 mon.b (mon.1) 15 : audit 1 from='osd.5 [v2:172.21.15.146:6808/4003230814,v1:172.21.15.146:6809/4003230814]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-11T20:48:02.736 INFO:teuthology.orchestra.run.smithi146.stdout:Created osd(s) 5 on host 'smithi146' 2024-03-11T20:48:02.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:02 smithi146 bash[23809]: audit 2024-03-11T20:48:01.309884+0000 mon.a (mon.0) 516 : audit 1 from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-11T20:48:02.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:02 smithi146 bash[23809]: cluster 2024-03-11T20:48:01.315533+0000 mon.a (mon.0) 517 : cluster 0 osdmap e42: 6 total, 5 up, 6 in 2024-03-11T20:48:02.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:02 smithi146 bash[23809]: audit 2024-03-11T20:48:01.316050+0000 mon.a (mon.0) 518 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T20:48:02.756 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:02 smithi146 bash[23809]: audit 2024-03-11T20:48:01.317205+0000 mon.a (mon.0) 519 : audit 1 from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-03-11T20:48:02.756 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:02 smithi146 bash[23809]: audit 2024-03-11T20:48:01.318391+0000 mon.b (mon.1) 16 : audit 1 from='osd.5 [v2:172.21.15.146:6808/4003230814,v1:172.21.15.146:6809/4003230814]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-03-11T20:48:02.756 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:02 smithi146 bash[23809]: cluster 2024-03-11T20:48:01.404888+0000 mgr.y (mgr.14152) 253 : cluster 0 pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T20:48:02.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:02 smithi023 bash[21372]: audit 2024-03-11T20:48:01.309884+0000 mon.a (mon.0) 516 : audit 1 from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-11T20:48:02.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:02 smithi023 bash[21372]: cluster 2024-03-11T20:48:01.315533+0000 mon.a (mon.0) 517 : cluster 0 osdmap e42: 6 total, 5 up, 6 in 2024-03-11T20:48:02.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:02 smithi023 bash[21372]: audit 2024-03-11T20:48:01.316050+0000 mon.a (mon.0) 518 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T20:48:02.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:02 smithi023 bash[21372]: audit 2024-03-11T20:48:01.317205+0000 mon.a (mon.0) 519 : audit 1 from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-03-11T20:48:02.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:02 smithi023 bash[21372]: audit 2024-03-11T20:48:01.318391+0000 mon.b (mon.1) 16 : audit 1 from='osd.5 [v2:172.21.15.146:6808/4003230814,v1:172.21.15.146:6809/4003230814]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-03-11T20:48:02.784 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:02 smithi023 bash[21372]: cluster 2024-03-11T20:48:01.404888+0000 mgr.y (mgr.14152) 253 : cluster 0 pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T20:48:02.784 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:02 smithi023 bash[29146]: audit 2024-03-11T20:48:01.309884+0000 mon.a (mon.0) 516 : audit 1 from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-11T20:48:02.784 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:02 smithi023 bash[29146]: cluster 2024-03-11T20:48:01.315533+0000 mon.a (mon.0) 517 : cluster 0 osdmap e42: 6 total, 5 up, 6 in 2024-03-11T20:48:02.784 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:02 smithi023 bash[29146]: audit 2024-03-11T20:48:01.316050+0000 mon.a (mon.0) 518 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T20:48:02.784 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:02 smithi023 bash[29146]: audit 2024-03-11T20:48:01.317205+0000 mon.a (mon.0) 519 : audit 1 from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-03-11T20:48:02.784 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:02 smithi023 bash[29146]: audit 2024-03-11T20:48:01.318391+0000 mon.b (mon.1) 16 : audit 1 from='osd.5 [v2:172.21.15.146:6808/4003230814,v1:172.21.15.146:6809/4003230814]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-03-11T20:48:02.784 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:02 smithi023 bash[29146]: cluster 2024-03-11T20:48:01.404888+0000 mgr.y (mgr.14152) 253 : cluster 0 pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T20:48:03.489 DEBUG:teuthology.orchestra.run.smithi146:osd.5> sudo journalctl -f -n 0 -u ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@osd.5.service 2024-03-11T20:48:03.492 INFO:tasks.cephadm:Deploying osd.6 on smithi146 with /dev/vg_nvme/lv_2... 2024-03-11T20:48:03.492 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-11T20:48:03.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:03 smithi146 bash[23809]: audit 2024-03-11T20:48:02.312324+0000 mon.a (mon.0) 520 : audit 1 from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]': finished 2024-03-11T20:48:03.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:03 smithi146 bash[23809]: cluster 2024-03-11T20:48:02.328631+0000 mon.a (mon.0) 521 : cluster 0 osdmap e43: 6 total, 5 up, 6 in 2024-03-11T20:48:03.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:03 smithi146 bash[23809]: audit 2024-03-11T20:48:02.329779+0000 mon.a (mon.0) 522 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T20:48:03.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:03 smithi146 bash[23809]: audit 2024-03-11T20:48:02.715333+0000 mon.a (mon.0) 523 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:48:03.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:03 smithi146 bash[23809]: audit 2024-03-11T20:48:02.719626+0000 mon.a (mon.0) 524 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:48:03.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:03 smithi146 bash[23809]: audit 2024-03-11T20:48:02.726091+0000 mon.a (mon.0) 525 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:48:03.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:03 smithi146 bash[23809]: audit 2024-03-11T20:48:02.838248+0000 mon.a (mon.0) 526 : audit 1 from='osd.5 ' entity='osd.5' 2024-03-11T20:48:03.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:03 smithi146 bash[23809]: cluster 2024-03-11T20:48:03.324784+0000 mon.a (mon.0) 527 : cluster 1 osd.5 [v2:172.21.15.146:6808/4003230814,v1:172.21.15.146:6809/4003230814] boot 2024-03-11T20:48:03.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:03 smithi146 bash[23809]: cluster 2024-03-11T20:48:03.324870+0000 mon.a (mon.0) 528 : cluster 0 osdmap e44: 6 total, 6 up, 6 in 2024-03-11T20:48:03.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:03 smithi146 bash[23809]: audit 2024-03-11T20:48:03.325062+0000 mon.a (mon.0) 529 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T20:48:03.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:03 smithi023 bash[21372]: audit 2024-03-11T20:48:02.312324+0000 mon.a (mon.0) 520 : audit 1 from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]': finished 2024-03-11T20:48:03.784 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:03 smithi023 bash[21372]: cluster 2024-03-11T20:48:02.328631+0000 mon.a (mon.0) 521 : cluster 0 osdmap e43: 6 total, 5 up, 6 in 2024-03-11T20:48:03.784 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:03 smithi023 bash[21372]: audit 2024-03-11T20:48:02.329779+0000 mon.a (mon.0) 522 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T20:48:03.784 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:03 smithi023 bash[21372]: audit 2024-03-11T20:48:02.715333+0000 mon.a (mon.0) 523 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:48:03.784 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:03 smithi023 bash[21372]: audit 2024-03-11T20:48:02.719626+0000 mon.a (mon.0) 524 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:48:03.784 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:03 smithi023 bash[21372]: audit 2024-03-11T20:48:02.726091+0000 mon.a (mon.0) 525 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:48:03.784 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:03 smithi023 bash[21372]: audit 2024-03-11T20:48:02.838248+0000 mon.a (mon.0) 526 : audit 1 from='osd.5 ' entity='osd.5' 2024-03-11T20:48:03.784 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:03 smithi023 bash[21372]: cluster 2024-03-11T20:48:03.324784+0000 mon.a (mon.0) 527 : cluster 1 osd.5 [v2:172.21.15.146:6808/4003230814,v1:172.21.15.146:6809/4003230814] boot 2024-03-11T20:48:03.784 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:03 smithi023 bash[21372]: cluster 2024-03-11T20:48:03.324870+0000 mon.a (mon.0) 528 : cluster 0 osdmap e44: 6 total, 6 up, 6 in 2024-03-11T20:48:03.784 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:03 smithi023 bash[21372]: audit 2024-03-11T20:48:03.325062+0000 mon.a (mon.0) 529 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T20:48:03.784 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:03 smithi023 bash[29146]: audit 2024-03-11T20:48:02.312324+0000 mon.a (mon.0) 520 : audit 1 from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]': finished 2024-03-11T20:48:03.784 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:03 smithi023 bash[29146]: cluster 2024-03-11T20:48:02.328631+0000 mon.a (mon.0) 521 : cluster 0 osdmap e43: 6 total, 5 up, 6 in 2024-03-11T20:48:03.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:03 smithi023 bash[29146]: audit 2024-03-11T20:48:02.329779+0000 mon.a (mon.0) 522 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T20:48:03.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:03 smithi023 bash[29146]: audit 2024-03-11T20:48:02.715333+0000 mon.a (mon.0) 523 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:48:03.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:03 smithi023 bash[29146]: audit 2024-03-11T20:48:02.719626+0000 mon.a (mon.0) 524 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:48:03.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:03 smithi023 bash[29146]: audit 2024-03-11T20:48:02.726091+0000 mon.a (mon.0) 525 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:48:03.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:03 smithi023 bash[29146]: audit 2024-03-11T20:48:02.838248+0000 mon.a (mon.0) 526 : audit 1 from='osd.5 ' entity='osd.5' 2024-03-11T20:48:03.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:03 smithi023 bash[29146]: cluster 2024-03-11T20:48:03.324784+0000 mon.a (mon.0) 527 : cluster 1 osd.5 [v2:172.21.15.146:6808/4003230814,v1:172.21.15.146:6809/4003230814] boot 2024-03-11T20:48:03.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:03 smithi023 bash[29146]: cluster 2024-03-11T20:48:03.324870+0000 mon.a (mon.0) 528 : cluster 0 osdmap e44: 6 total, 6 up, 6 in 2024-03-11T20:48:03.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:03 smithi023 bash[29146]: audit 2024-03-11T20:48:03.325062+0000 mon.a (mon.0) 529 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T20:48:04.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:04 smithi146 bash[23809]: cluster 2024-03-11T20:48:01.575244+0000 osd.5 (osd.5) 1 : cluster 0 purged_snaps scrub starts 2024-03-11T20:48:04.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:04 smithi146 bash[23809]: cluster 2024-03-11T20:48:01.575333+0000 osd.5 (osd.5) 2 : cluster 0 purged_snaps scrub ok 2024-03-11T20:48:04.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:04 smithi146 bash[23809]: cluster 2024-03-11T20:48:03.405406+0000 mgr.y (mgr.14152) 254 : cluster 0 pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T20:48:04.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:04 smithi146 bash[23809]: cluster 2024-03-11T20:48:04.327561+0000 mon.a (mon.0) 530 : cluster 0 osdmap e45: 6 total, 6 up, 6 in 2024-03-11T20:48:04.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:04 smithi023 bash[21372]: cluster 2024-03-11T20:48:01.575244+0000 osd.5 (osd.5) 1 : cluster 0 purged_snaps scrub starts 2024-03-11T20:48:04.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:04 smithi023 bash[21372]: cluster 2024-03-11T20:48:01.575333+0000 osd.5 (osd.5) 2 : cluster 0 purged_snaps scrub ok 2024-03-11T20:48:04.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:04 smithi023 bash[21372]: cluster 2024-03-11T20:48:03.405406+0000 mgr.y (mgr.14152) 254 : cluster 0 pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T20:48:04.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:04 smithi023 bash[21372]: cluster 2024-03-11T20:48:04.327561+0000 mon.a (mon.0) 530 : cluster 0 osdmap e45: 6 total, 6 up, 6 in 2024-03-11T20:48:04.783 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:04 smithi023 bash[29146]: cluster 2024-03-11T20:48:01.575244+0000 osd.5 (osd.5) 1 : cluster 0 purged_snaps scrub starts 2024-03-11T20:48:04.783 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:04 smithi023 bash[29146]: cluster 2024-03-11T20:48:01.575333+0000 osd.5 (osd.5) 2 : cluster 0 purged_snaps scrub ok 2024-03-11T20:48:04.784 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:04 smithi023 bash[29146]: cluster 2024-03-11T20:48:03.405406+0000 mgr.y (mgr.14152) 254 : cluster 0 pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T20:48:04.784 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:04 smithi023 bash[29146]: cluster 2024-03-11T20:48:04.327561+0000 mon.a (mon.0) 530 : cluster 0 osdmap e45: 6 total, 6 up, 6 in 2024-03-11T20:48:06.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:06 smithi146 bash[23809]: cluster 2024-03-11T20:48:05.330172+0000 mon.a (mon.0) 531 : cluster 0 osdmap e46: 6 total, 6 up, 6 in 2024-03-11T20:48:06.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:06 smithi146 bash[23809]: cluster 2024-03-11T20:48:05.406058+0000 mgr.y (mgr.14152) 255 : cluster 0 pgmap v238: 1 pgs: 1 remapped+peering; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-11T20:48:06.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:06 smithi023 bash[21372]: cluster 2024-03-11T20:48:05.330172+0000 mon.a (mon.0) 531 : cluster 0 osdmap e46: 6 total, 6 up, 6 in 2024-03-11T20:48:06.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:06 smithi023 bash[21372]: cluster 2024-03-11T20:48:05.406058+0000 mgr.y (mgr.14152) 255 : cluster 0 pgmap v238: 1 pgs: 1 remapped+peering; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-11T20:48:06.783 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:06 smithi023 bash[29146]: cluster 2024-03-11T20:48:05.330172+0000 mon.a (mon.0) 531 : cluster 0 osdmap e46: 6 total, 6 up, 6 in 2024-03-11T20:48:06.783 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:06 smithi023 bash[29146]: cluster 2024-03-11T20:48:05.406058+0000 mgr.y (mgr.14152) 255 : cluster 0 pgmap v238: 1 pgs: 1 remapped+peering; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-11T20:48:07.725 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T20:48:07.744 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:07 smithi146 bash[23809]: cluster 2024-03-11T20:48:06.323789+0000 mon.a (mon.0) 532 : cluster 3 Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-11T20:48:07.744 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:07 smithi146 bash[23809]: cluster 2024-03-11T20:48:06.353056+0000 mon.a (mon.0) 533 : cluster 0 osdmap e47: 6 total, 6 up, 6 in 2024-03-11T20:48:07.783 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:07 smithi023 bash[29146]: cluster 2024-03-11T20:48:06.323789+0000 mon.a (mon.0) 532 : cluster 3 Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-11T20:48:07.783 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:07 smithi023 bash[29146]: cluster 2024-03-11T20:48:06.353056+0000 mon.a (mon.0) 533 : cluster 0 osdmap e47: 6 total, 6 up, 6 in 2024-03-11T20:48:07.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:07 smithi023 bash[21372]: cluster 2024-03-11T20:48:06.323789+0000 mon.a (mon.0) 532 : cluster 3 Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-11T20:48:07.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:07 smithi023 bash[21372]: cluster 2024-03-11T20:48:06.353056+0000 mon.a (mon.0) 533 : cluster 0 osdmap e47: 6 total, 6 up, 6 in 2024-03-11T20:48:08.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:08 smithi146 bash[23809]: cluster 2024-03-11T20:48:07.376764+0000 mon.a (mon.0) 534 : cluster 0 osdmap e48: 6 total, 6 up, 6 in 2024-03-11T20:48:08.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:08 smithi146 bash[23809]: cluster 2024-03-11T20:48:07.406711+0000 mgr.y (mgr.14152) 256 : cluster 0 pgmap v241: 1 pgs: 1 remapped+peering; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-11T20:48:08.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:08 smithi146 bash[23809]: cluster 2024-03-11T20:48:07.781873+0000 osd.2 (osd.2) 5 : cluster 0 purged_snaps scrub starts 2024-03-11T20:48:08.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:08 smithi146 bash[23809]: cluster 2024-03-11T20:48:07.781964+0000 osd.2 (osd.2) 6 : cluster 0 purged_snaps scrub ok 2024-03-11T20:48:08.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:08 smithi023 bash[21372]: cluster 2024-03-11T20:48:07.376764+0000 mon.a (mon.0) 534 : cluster 0 osdmap e48: 6 total, 6 up, 6 in 2024-03-11T20:48:08.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:08 smithi023 bash[21372]: cluster 2024-03-11T20:48:07.406711+0000 mgr.y (mgr.14152) 256 : cluster 0 pgmap v241: 1 pgs: 1 remapped+peering; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-11T20:48:08.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:08 smithi023 bash[21372]: cluster 2024-03-11T20:48:07.781873+0000 osd.2 (osd.2) 5 : cluster 0 purged_snaps scrub starts 2024-03-11T20:48:08.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:08 smithi023 bash[21372]: cluster 2024-03-11T20:48:07.781964+0000 osd.2 (osd.2) 6 : cluster 0 purged_snaps scrub ok 2024-03-11T20:48:08.783 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:08 smithi023 bash[29146]: cluster 2024-03-11T20:48:07.376764+0000 mon.a (mon.0) 534 : cluster 0 osdmap e48: 6 total, 6 up, 6 in 2024-03-11T20:48:08.784 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:08 smithi023 bash[29146]: cluster 2024-03-11T20:48:07.406711+0000 mgr.y (mgr.14152) 256 : cluster 0 pgmap v241: 1 pgs: 1 remapped+peering; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-11T20:48:08.784 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:08 smithi023 bash[29146]: cluster 2024-03-11T20:48:07.781873+0000 osd.2 (osd.2) 5 : cluster 0 purged_snaps scrub starts 2024-03-11T20:48:08.784 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:08 smithi023 bash[29146]: cluster 2024-03-11T20:48:07.781964+0000 osd.2 (osd.2) 6 : cluster 0 purged_snaps scrub ok 2024-03-11T20:48:09.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:09 smithi146 bash[23809]: cluster 2024-03-11T20:48:08.399387+0000 mon.a (mon.0) 535 : cluster 0 osdmap e49: 6 total, 6 up, 6 in 2024-03-11T20:48:09.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:09 smithi023 bash[21372]: cluster 2024-03-11T20:48:08.399387+0000 mon.a (mon.0) 535 : cluster 0 osdmap e49: 6 total, 6 up, 6 in 2024-03-11T20:48:09.783 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:09 smithi023 bash[29146]: cluster 2024-03-11T20:48:08.399387+0000 mon.a (mon.0) 535 : cluster 0 osdmap e49: 6 total, 6 up, 6 in 2024-03-11T20:48:10.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:10 smithi146 bash[23809]: cluster 2024-03-11T20:48:09.407348+0000 mgr.y (mgr.14152) 257 : cluster 0 pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 141 KiB/s, 0 objects/s recovering 2024-03-11T20:48:10.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:10 smithi023 bash[21372]: cluster 2024-03-11T20:48:09.407348+0000 mgr.y (mgr.14152) 257 : cluster 0 pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 141 KiB/s, 0 objects/s recovering 2024-03-11T20:48:10.783 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:10 smithi023 bash[29146]: cluster 2024-03-11T20:48:09.407348+0000 mgr.y (mgr.14152) 257 : cluster 0 pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 141 KiB/s, 0 objects/s recovering 2024-03-11T20:48:11.669 INFO:teuthology.orchestra.run.smithi146.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-11T20:48:11.669 INFO:teuthology.orchestra.run.smithi146.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-11T20:48:11.669 INFO:teuthology.orchestra.run.smithi146.stderr: stderr: 10+0 records in 2024-03-11T20:48:11.669 INFO:teuthology.orchestra.run.smithi146.stderr:10+0 records out 2024-03-11T20:48:11.669 INFO:teuthology.orchestra.run.smithi146.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0159658 s, 657 MB/s 2024-03-11T20:48:11.669 INFO:teuthology.orchestra.run.smithi146.stderr:--> Zapping successful for: 2024-03-11T20:48:11.709 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:11 smithi146 bash[23809]: cluster 2024-03-11T20:48:10.397517+0000 mon.a (mon.0) 536 : cluster 1 Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-11T20:48:11.709 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:11 smithi146 bash[23809]: cluster 2024-03-11T20:48:10.397614+0000 mon.a (mon.0) 537 : cluster 1 Cluster is now healthy 2024-03-11T20:48:11.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:11 smithi023 bash[21372]: cluster 2024-03-11T20:48:10.397517+0000 mon.a (mon.0) 536 : cluster 1 Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-11T20:48:11.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:11 smithi023 bash[21372]: cluster 2024-03-11T20:48:10.397614+0000 mon.a (mon.0) 537 : cluster 1 Cluster is now healthy 2024-03-11T20:48:11.783 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:11 smithi023 bash[29146]: cluster 2024-03-11T20:48:10.397517+0000 mon.a (mon.0) 536 : cluster 1 Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-11T20:48:11.784 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:11 smithi023 bash[29146]: cluster 2024-03-11T20:48:10.397614+0000 mon.a (mon.0) 537 : cluster 1 Cluster is now healthy 2024-03-11T20:48:12.529 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi146:vg_nvme/lv_2 2024-03-11T20:48:12.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:12 smithi146 bash[23809]: cluster 2024-03-11T20:48:11.408022+0000 mgr.y (mgr.14152) 258 : cluster 0 pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-11T20:48:12.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:12 smithi023 bash[21372]: cluster 2024-03-11T20:48:11.408022+0000 mgr.y (mgr.14152) 258 : cluster 0 pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-11T20:48:12.783 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:12 smithi023 bash[29146]: cluster 2024-03-11T20:48:11.408022+0000 mgr.y (mgr.14152) 258 : cluster 0 pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-11T20:48:14.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:13 smithi146 bash[23809]: cephadm 2024-03-11T20:48:12.924646+0000 mgr.y (mgr.14152) 259 : cephadm 1 Detected new or changed devices on smithi146 2024-03-11T20:48:14.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:13 smithi146 bash[23809]: audit 2024-03-11T20:48:12.937358+0000 mon.a (mon.0) 538 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:48:14.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:13 smithi146 bash[23809]: audit 2024-03-11T20:48:12.947564+0000 mon.a (mon.0) 539 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:48:14.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:13 smithi146 bash[23809]: audit 2024-03-11T20:48:12.950085+0000 mon.a (mon.0) 540 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:48:14.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:13 smithi146 bash[23809]: audit 2024-03-11T20:48:12.951968+0000 mon.a (mon.0) 541 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:48:14.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:13 smithi146 bash[23809]: cephadm 2024-03-11T20:48:12.953292+0000 mgr.y (mgr.14152) 260 : cephadm 1 Adjusting osd_memory_target on smithi146 to 8632M 2024-03-11T20:48:14.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:13 smithi146 bash[23809]: audit 2024-03-11T20:48:12.960181+0000 mon.a (mon.0) 542 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:48:14.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:13 smithi146 bash[23809]: audit 2024-03-11T20:48:12.964519+0000 mon.a (mon.0) 543 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:48:14.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:13 smithi146 bash[23809]: audit 2024-03-11T20:48:12.966419+0000 mon.a (mon.0) 544 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:48:14.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:13 smithi146 bash[23809]: audit 2024-03-11T20:48:12.976703+0000 mon.a (mon.0) 545 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:48:14.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:13 smithi023 bash[21372]: cephadm 2024-03-11T20:48:12.924646+0000 mgr.y (mgr.14152) 259 : cephadm 1 Detected new or changed devices on smithi146 2024-03-11T20:48:14.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:13 smithi023 bash[21372]: audit 2024-03-11T20:48:12.937358+0000 mon.a (mon.0) 538 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:48:14.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:13 smithi023 bash[21372]: audit 2024-03-11T20:48:12.947564+0000 mon.a (mon.0) 539 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:48:14.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:13 smithi023 bash[21372]: audit 2024-03-11T20:48:12.950085+0000 mon.a (mon.0) 540 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:48:14.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:13 smithi023 bash[21372]: audit 2024-03-11T20:48:12.951968+0000 mon.a (mon.0) 541 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:48:14.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:13 smithi023 bash[21372]: cephadm 2024-03-11T20:48:12.953292+0000 mgr.y (mgr.14152) 260 : cephadm 1 Adjusting osd_memory_target on smithi146 to 8632M 2024-03-11T20:48:14.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:13 smithi023 bash[21372]: audit 2024-03-11T20:48:12.960181+0000 mon.a (mon.0) 542 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:48:14.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:13 smithi023 bash[21372]: audit 2024-03-11T20:48:12.964519+0000 mon.a (mon.0) 543 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:48:14.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:13 smithi023 bash[21372]: audit 2024-03-11T20:48:12.966419+0000 mon.a (mon.0) 544 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:48:14.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:13 smithi023 bash[21372]: audit 2024-03-11T20:48:12.976703+0000 mon.a (mon.0) 545 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:48:14.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:13 smithi023 bash[29146]: cephadm 2024-03-11T20:48:12.924646+0000 mgr.y (mgr.14152) 259 : cephadm 1 Detected new or changed devices on smithi146 2024-03-11T20:48:14.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:13 smithi023 bash[29146]: audit 2024-03-11T20:48:12.937358+0000 mon.a (mon.0) 538 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:48:14.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:13 smithi023 bash[29146]: audit 2024-03-11T20:48:12.947564+0000 mon.a (mon.0) 539 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:48:14.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:13 smithi023 bash[29146]: audit 2024-03-11T20:48:12.950085+0000 mon.a (mon.0) 540 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:48:14.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:13 smithi023 bash[29146]: audit 2024-03-11T20:48:12.951968+0000 mon.a (mon.0) 541 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:48:14.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:13 smithi023 bash[29146]: cephadm 2024-03-11T20:48:12.953292+0000 mgr.y (mgr.14152) 260 : cephadm 1 Adjusting osd_memory_target on smithi146 to 8632M 2024-03-11T20:48:14.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:13 smithi023 bash[29146]: audit 2024-03-11T20:48:12.960181+0000 mon.a (mon.0) 542 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:48:14.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:13 smithi023 bash[29146]: audit 2024-03-11T20:48:12.964519+0000 mon.a (mon.0) 543 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:48:14.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:13 smithi023 bash[29146]: audit 2024-03-11T20:48:12.966419+0000 mon.a (mon.0) 544 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:48:14.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:13 smithi023 bash[29146]: audit 2024-03-11T20:48:12.976703+0000 mon.a (mon.0) 545 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:48:15.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:14 smithi146 bash[23809]: cluster 2024-03-11T20:48:13.408650+0000 mgr.y (mgr.14152) 261 : cluster 0 pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-03-11T20:48:15.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:14 smithi023 bash[21372]: cluster 2024-03-11T20:48:13.408650+0000 mgr.y (mgr.14152) 261 : cluster 0 pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-03-11T20:48:15.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:14 smithi023 bash[29146]: cluster 2024-03-11T20:48:13.408650+0000 mgr.y (mgr.14152) 261 : cluster 0 pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-03-11T20:48:17.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:16 smithi146 bash[23809]: cluster 2024-03-11T20:48:15.409264+0000 mgr.y (mgr.14152) 262 : cluster 0 pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-11T20:48:17.269 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T20:48:17.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:16 smithi023 bash[21372]: cluster 2024-03-11T20:48:15.409264+0000 mgr.y (mgr.14152) 262 : cluster 0 pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-11T20:48:17.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:16 smithi023 bash[29146]: cluster 2024-03-11T20:48:15.409264+0000 mgr.y (mgr.14152) 262 : cluster 0 pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-11T20:48:19.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:19 smithi146 bash[23809]: cluster 2024-03-11T20:48:17.409816+0000 mgr.y (mgr.14152) 263 : cluster 0 pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-11T20:48:19.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:18 smithi023 bash[21372]: cluster 2024-03-11T20:48:17.409816+0000 mgr.y (mgr.14152) 263 : cluster 0 pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-11T20:48:19.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:19 smithi023 bash[29146]: cluster 2024-03-11T20:48:17.409816+0000 mgr.y (mgr.14152) 263 : cluster 0 pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-11T20:48:20.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:20 smithi023 bash[21372]: cluster 2024-03-11T20:48:19.281093+0000 osd.0 (osd.0) 9 : cluster 0 purged_snaps scrub starts 2024-03-11T20:48:20.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:20 smithi023 bash[21372]: cluster 2024-03-11T20:48:19.281186+0000 osd.0 (osd.0) 10 : cluster 0 purged_snaps scrub ok 2024-03-11T20:48:20.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:20 smithi023 bash[21372]: cluster 2024-03-11T20:48:19.569640+0000 osd.5 (osd.5) 3 : cluster 0 1.0 scrub starts 2024-03-11T20:48:20.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:20 smithi023 bash[21372]: cluster 2024-03-11T20:48:19.573701+0000 osd.5 (osd.5) 4 : cluster 0 1.0 scrub ok 2024-03-11T20:48:20.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:20 smithi023 bash[21372]: audit 2024-03-11T20:48:19.973158+0000 mon.a (mon.0) 546 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T20:48:20.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:20 smithi023 bash[21372]: audit 2024-03-11T20:48:19.979608+0000 mon.a (mon.0) 547 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T20:48:20.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:20 smithi023 bash[21372]: audit 2024-03-11T20:48:19.981316+0000 mon.a (mon.0) 548 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:48:20.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:20 smithi023 bash[29146]: cluster 2024-03-11T20:48:19.281093+0000 osd.0 (osd.0) 9 : cluster 0 purged_snaps scrub starts 2024-03-11T20:48:20.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:20 smithi023 bash[29146]: cluster 2024-03-11T20:48:19.281186+0000 osd.0 (osd.0) 10 : cluster 0 purged_snaps scrub ok 2024-03-11T20:48:20.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:20 smithi023 bash[29146]: cluster 2024-03-11T20:48:19.569640+0000 osd.5 (osd.5) 3 : cluster 0 1.0 scrub starts 2024-03-11T20:48:20.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:20 smithi023 bash[29146]: cluster 2024-03-11T20:48:19.573701+0000 osd.5 (osd.5) 4 : cluster 0 1.0 scrub ok 2024-03-11T20:48:20.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:20 smithi023 bash[29146]: audit 2024-03-11T20:48:19.973158+0000 mon.a (mon.0) 546 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T20:48:20.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:20 smithi023 bash[29146]: audit 2024-03-11T20:48:19.979608+0000 mon.a (mon.0) 547 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T20:48:20.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:20 smithi023 bash[29146]: audit 2024-03-11T20:48:19.981316+0000 mon.a (mon.0) 548 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:48:20.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:20 smithi146 bash[23809]: cluster 2024-03-11T20:48:19.281093+0000 osd.0 (osd.0) 9 : cluster 0 purged_snaps scrub starts 2024-03-11T20:48:20.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:20 smithi146 bash[23809]: cluster 2024-03-11T20:48:19.281186+0000 osd.0 (osd.0) 10 : cluster 0 purged_snaps scrub ok 2024-03-11T20:48:20.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:20 smithi146 bash[23809]: cluster 2024-03-11T20:48:19.569640+0000 osd.5 (osd.5) 3 : cluster 0 1.0 scrub starts 2024-03-11T20:48:20.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:20 smithi146 bash[23809]: cluster 2024-03-11T20:48:19.573701+0000 osd.5 (osd.5) 4 : cluster 0 1.0 scrub ok 2024-03-11T20:48:20.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:20 smithi146 bash[23809]: audit 2024-03-11T20:48:19.973158+0000 mon.a (mon.0) 546 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T20:48:20.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:20 smithi146 bash[23809]: audit 2024-03-11T20:48:19.979608+0000 mon.a (mon.0) 547 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T20:48:20.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:20 smithi146 bash[23809]: audit 2024-03-11T20:48:19.981316+0000 mon.a (mon.0) 548 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:48:21.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:21 smithi023 bash[29146]: cluster 2024-03-11T20:48:19.410534+0000 mgr.y (mgr.14152) 264 : cluster 0 pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-03-11T20:48:21.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:21 smithi023 bash[29146]: audit 2024-03-11T20:48:19.965865+0000 mgr.y (mgr.14152) 265 : audit 0 from='client.24238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi146:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:48:21.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:21 smithi023 bash[29146]: cluster 2024-03-11T20:48:20.032612+0000 mon.a (mon.0) 549 : cluster 0 osdmap e50: 6 total, 6 up, 6 in 2024-03-11T20:48:21.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:21 smithi023 bash[21372]: cluster 2024-03-11T20:48:19.410534+0000 mgr.y (mgr.14152) 264 : cluster 0 pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-03-11T20:48:21.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:21 smithi023 bash[21372]: audit 2024-03-11T20:48:19.965865+0000 mgr.y (mgr.14152) 265 : audit 0 from='client.24238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi146:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:48:21.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:21 smithi023 bash[21372]: cluster 2024-03-11T20:48:20.032612+0000 mon.a (mon.0) 549 : cluster 0 osdmap e50: 6 total, 6 up, 6 in 2024-03-11T20:48:21.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:21 smithi146 bash[23809]: cluster 2024-03-11T20:48:19.410534+0000 mgr.y (mgr.14152) 264 : cluster 0 pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-03-11T20:48:21.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:21 smithi146 bash[23809]: audit 2024-03-11T20:48:19.965865+0000 mgr.y (mgr.14152) 265 : audit 0 from='client.24238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi146:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:48:21.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:21 smithi146 bash[23809]: cluster 2024-03-11T20:48:20.032612+0000 mon.a (mon.0) 549 : cluster 0 osdmap e50: 6 total, 6 up, 6 in 2024-03-11T20:48:22.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:22 smithi146 bash[23809]: cluster 2024-03-11T20:48:20.242232+0000 osd.1 (osd.1) 9 : cluster 0 purged_snaps scrub starts 2024-03-11T20:48:22.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:22 smithi146 bash[23809]: cluster 2024-03-11T20:48:20.242348+0000 osd.1 (osd.1) 10 : cluster 0 purged_snaps scrub ok 2024-03-11T20:48:22.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:22 smithi146 bash[23809]: cluster 2024-03-11T20:48:21.058126+0000 mon.a (mon.0) 550 : cluster 0 osdmap e51: 6 total, 6 up, 6 in 2024-03-11T20:48:22.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:22 smithi023 bash[21372]: cluster 2024-03-11T20:48:20.242232+0000 osd.1 (osd.1) 9 : cluster 0 purged_snaps scrub starts 2024-03-11T20:48:22.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:22 smithi023 bash[21372]: cluster 2024-03-11T20:48:20.242348+0000 osd.1 (osd.1) 10 : cluster 0 purged_snaps scrub ok 2024-03-11T20:48:22.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:22 smithi023 bash[21372]: cluster 2024-03-11T20:48:21.058126+0000 mon.a (mon.0) 550 : cluster 0 osdmap e51: 6 total, 6 up, 6 in 2024-03-11T20:48:22.534 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:22 smithi023 bash[29146]: cluster 2024-03-11T20:48:20.242232+0000 osd.1 (osd.1) 9 : cluster 0 purged_snaps scrub starts 2024-03-11T20:48:22.534 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:22 smithi023 bash[29146]: cluster 2024-03-11T20:48:20.242348+0000 osd.1 (osd.1) 10 : cluster 0 purged_snaps scrub ok 2024-03-11T20:48:22.534 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:22 smithi023 bash[29146]: cluster 2024-03-11T20:48:21.058126+0000 mon.a (mon.0) 550 : cluster 0 osdmap e51: 6 total, 6 up, 6 in 2024-03-11T20:48:23.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:23 smithi146 bash[23809]: cluster 2024-03-11T20:48:21.411047+0000 mgr.y (mgr.14152) 266 : cluster 0 pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-11T20:48:23.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:23 smithi023 bash[21372]: cluster 2024-03-11T20:48:21.411047+0000 mgr.y (mgr.14152) 266 : cluster 0 pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-11T20:48:23.533 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:23 smithi023 bash[29146]: cluster 2024-03-11T20:48:21.411047+0000 mgr.y (mgr.14152) 266 : cluster 0 pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-11T20:48:25.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:25 smithi146 bash[23809]: cluster 2024-03-11T20:48:23.411648+0000 mgr.y (mgr.14152) 267 : cluster 0 pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-11T20:48:25.533 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:25 smithi023 bash[29146]: cluster 2024-03-11T20:48:23.411648+0000 mgr.y (mgr.14152) 267 : cluster 0 pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-11T20:48:25.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:25 smithi023 bash[21372]: cluster 2024-03-11T20:48:23.411648+0000 mgr.y (mgr.14152) 267 : cluster 0 pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-11T20:48:27.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:27 smithi146 bash[23809]: cluster 2024-03-11T20:48:25.412176+0000 mgr.y (mgr.14152) 268 : cluster 0 pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-11T20:48:27.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:27 smithi023 bash[21372]: cluster 2024-03-11T20:48:25.412176+0000 mgr.y (mgr.14152) 268 : cluster 0 pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-11T20:48:27.533 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:27 smithi023 bash[29146]: cluster 2024-03-11T20:48:25.412176+0000 mgr.y (mgr.14152) 268 : cluster 0 pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-11T20:48:29.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:29 smithi146 bash[23809]: cluster 2024-03-11T20:48:27.412695+0000 mgr.y (mgr.14152) 269 : cluster 0 pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-11T20:48:29.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:29 smithi146 bash[23809]: audit 2024-03-11T20:48:28.224194+0000 mon.a (mon.0) 551 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5cb151e8-1f9b-4617-8a86-adfb06bb9606"}]: dispatch 2024-03-11T20:48:29.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:29 smithi146 bash[23809]: audit 2024-03-11T20:48:28.226188+0000 mon.b (mon.1) 17 : audit 1 from='client.? 172.21.15.146:0/3582115793' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5cb151e8-1f9b-4617-8a86-adfb06bb9606"}]: dispatch 2024-03-11T20:48:29.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:29 smithi146 bash[23809]: audit 2024-03-11T20:48:28.230458+0000 mon.a (mon.0) 552 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5cb151e8-1f9b-4617-8a86-adfb06bb9606"}]': finished 2024-03-11T20:48:29.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:29 smithi146 bash[23809]: cluster 2024-03-11T20:48:28.235988+0000 mon.a (mon.0) 553 : cluster 0 osdmap e52: 7 total, 6 up, 7 in 2024-03-11T20:48:29.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:29 smithi146 bash[23809]: audit 2024-03-11T20:48:28.236834+0000 mon.a (mon.0) 554 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T20:48:29.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:29 smithi146 bash[23809]: audit 2024-03-11T20:48:29.001692+0000 mon.b (mon.1) 18 : audit 0 from='client.? 172.21.15.146:0/1361108519' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T20:48:29.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:29 smithi023 bash[21372]: cluster 2024-03-11T20:48:27.412695+0000 mgr.y (mgr.14152) 269 : cluster 0 pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-11T20:48:29.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:29 smithi023 bash[21372]: audit 2024-03-11T20:48:28.224194+0000 mon.a (mon.0) 551 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5cb151e8-1f9b-4617-8a86-adfb06bb9606"}]: dispatch 2024-03-11T20:48:29.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:29 smithi023 bash[21372]: audit 2024-03-11T20:48:28.226188+0000 mon.b (mon.1) 17 : audit 1 from='client.? 172.21.15.146:0/3582115793' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5cb151e8-1f9b-4617-8a86-adfb06bb9606"}]: dispatch 2024-03-11T20:48:29.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:29 smithi023 bash[21372]: audit 2024-03-11T20:48:28.230458+0000 mon.a (mon.0) 552 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5cb151e8-1f9b-4617-8a86-adfb06bb9606"}]': finished 2024-03-11T20:48:29.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:29 smithi023 bash[21372]: cluster 2024-03-11T20:48:28.235988+0000 mon.a (mon.0) 553 : cluster 0 osdmap e52: 7 total, 6 up, 7 in 2024-03-11T20:48:29.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:29 smithi023 bash[21372]: audit 2024-03-11T20:48:28.236834+0000 mon.a (mon.0) 554 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T20:48:29.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:29 smithi023 bash[21372]: audit 2024-03-11T20:48:29.001692+0000 mon.b (mon.1) 18 : audit 0 from='client.? 172.21.15.146:0/1361108519' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T20:48:29.534 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:29 smithi023 bash[29146]: cluster 2024-03-11T20:48:27.412695+0000 mgr.y (mgr.14152) 269 : cluster 0 pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-11T20:48:29.534 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:29 smithi023 bash[29146]: audit 2024-03-11T20:48:28.224194+0000 mon.a (mon.0) 551 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5cb151e8-1f9b-4617-8a86-adfb06bb9606"}]: dispatch 2024-03-11T20:48:29.534 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:29 smithi023 bash[29146]: audit 2024-03-11T20:48:28.226188+0000 mon.b (mon.1) 17 : audit 1 from='client.? 172.21.15.146:0/3582115793' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5cb151e8-1f9b-4617-8a86-adfb06bb9606"}]: dispatch 2024-03-11T20:48:29.534 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:29 smithi023 bash[29146]: audit 2024-03-11T20:48:28.230458+0000 mon.a (mon.0) 552 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5cb151e8-1f9b-4617-8a86-adfb06bb9606"}]': finished 2024-03-11T20:48:29.534 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:29 smithi023 bash[29146]: cluster 2024-03-11T20:48:28.235988+0000 mon.a (mon.0) 553 : cluster 0 osdmap e52: 7 total, 6 up, 7 in 2024-03-11T20:48:29.534 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:29 smithi023 bash[29146]: audit 2024-03-11T20:48:28.236834+0000 mon.a (mon.0) 554 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T20:48:29.534 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:29 smithi023 bash[29146]: audit 2024-03-11T20:48:29.001692+0000 mon.b (mon.1) 18 : audit 0 from='client.? 172.21.15.146:0/1361108519' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T20:48:31.443 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:31 smithi146 bash[23809]: cluster 2024-03-11T20:48:29.413210+0000 mgr.y (mgr.14152) 270 : cluster 0 pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-11T20:48:31.533 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:31 smithi023 bash[29146]: cluster 2024-03-11T20:48:29.413210+0000 mgr.y (mgr.14152) 270 : cluster 0 pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-11T20:48:31.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:31 smithi023 bash[21372]: cluster 2024-03-11T20:48:29.413210+0000 mgr.y (mgr.14152) 270 : cluster 0 pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-11T20:48:32.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:32 smithi146 bash[23809]: cluster 2024-03-11T20:48:31.413784+0000 mgr.y (mgr.14152) 271 : cluster 0 pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-11T20:48:32.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:32 smithi023 bash[21372]: cluster 2024-03-11T20:48:31.413784+0000 mgr.y (mgr.14152) 271 : cluster 0 pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-11T20:48:32.533 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:32 smithi023 bash[29146]: cluster 2024-03-11T20:48:31.413784+0000 mgr.y (mgr.14152) 271 : cluster 0 pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-11T20:48:34.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:34 smithi146 bash[23809]: cluster 2024-03-11T20:48:33.414317+0000 mgr.y (mgr.14152) 272 : cluster 0 pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-11T20:48:34.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:34 smithi023 bash[21372]: cluster 2024-03-11T20:48:33.414317+0000 mgr.y (mgr.14152) 272 : cluster 0 pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-11T20:48:34.783 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:34 smithi023 bash[29146]: cluster 2024-03-11T20:48:33.414317+0000 mgr.y (mgr.14152) 272 : cluster 0 pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-11T20:48:36.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:36 smithi023 bash[21372]: cluster 2024-03-11T20:48:35.414833+0000 mgr.y (mgr.14152) 273 : cluster 0 pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-11T20:48:36.784 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:36 smithi023 bash[29146]: cluster 2024-03-11T20:48:35.414833+0000 mgr.y (mgr.14152) 273 : cluster 0 pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-11T20:48:37.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:36 smithi146 bash[23809]: cluster 2024-03-11T20:48:35.414833+0000 mgr.y (mgr.14152) 273 : cluster 0 pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-11T20:48:39.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:38 smithi146 bash[23809]: cluster 2024-03-11T20:48:37.360143+0000 osd.4 (osd.4) 3 : cluster 0 purged_snaps scrub starts 2024-03-11T20:48:39.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:38 smithi146 bash[23809]: cluster 2024-03-11T20:48:37.360287+0000 osd.4 (osd.4) 4 : cluster 0 purged_snaps scrub ok 2024-03-11T20:48:39.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:38 smithi146 bash[23809]: cluster 2024-03-11T20:48:37.415369+0000 mgr.y (mgr.14152) 274 : cluster 0 pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-11T20:48:39.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:38 smithi146 bash[23809]: cluster 2024-03-11T20:48:37.557748+0000 mon.a (mon.0) 555 : cluster 0 osdmap e53: 7 total, 6 up, 7 in 2024-03-11T20:48:39.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:38 smithi146 bash[23809]: audit 2024-03-11T20:48:37.558046+0000 mon.a (mon.0) 556 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T20:48:39.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:38 smithi023 bash[21372]: cluster 2024-03-11T20:48:37.360143+0000 osd.4 (osd.4) 3 : cluster 0 purged_snaps scrub starts 2024-03-11T20:48:39.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:38 smithi023 bash[21372]: cluster 2024-03-11T20:48:37.360287+0000 osd.4 (osd.4) 4 : cluster 0 purged_snaps scrub ok 2024-03-11T20:48:39.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:38 smithi023 bash[21372]: cluster 2024-03-11T20:48:37.415369+0000 mgr.y (mgr.14152) 274 : cluster 0 pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-11T20:48:39.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:38 smithi023 bash[21372]: cluster 2024-03-11T20:48:37.557748+0000 mon.a (mon.0) 555 : cluster 0 osdmap e53: 7 total, 6 up, 7 in 2024-03-11T20:48:39.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:38 smithi023 bash[21372]: audit 2024-03-11T20:48:37.558046+0000 mon.a (mon.0) 556 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T20:48:39.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:38 smithi023 bash[29146]: cluster 2024-03-11T20:48:37.360143+0000 osd.4 (osd.4) 3 : cluster 0 purged_snaps scrub starts 2024-03-11T20:48:39.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:38 smithi023 bash[29146]: cluster 2024-03-11T20:48:37.360287+0000 osd.4 (osd.4) 4 : cluster 0 purged_snaps scrub ok 2024-03-11T20:48:39.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:38 smithi023 bash[29146]: cluster 2024-03-11T20:48:37.415369+0000 mgr.y (mgr.14152) 274 : cluster 0 pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-11T20:48:39.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:38 smithi023 bash[29146]: cluster 2024-03-11T20:48:37.557748+0000 mon.a (mon.0) 555 : cluster 0 osdmap e53: 7 total, 6 up, 7 in 2024-03-11T20:48:39.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:38 smithi023 bash[29146]: audit 2024-03-11T20:48:37.558046+0000 mon.a (mon.0) 556 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T20:48:41.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:40 smithi146 bash[23809]: cluster 2024-03-11T20:48:39.415899+0000 mgr.y (mgr.14152) 275 : cluster 0 pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-11T20:48:41.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:40 smithi023 bash[21372]: cluster 2024-03-11T20:48:39.415899+0000 mgr.y (mgr.14152) 275 : cluster 0 pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-11T20:48:41.033 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:40 smithi023 bash[29146]: cluster 2024-03-11T20:48:39.415899+0000 mgr.y (mgr.14152) 275 : cluster 0 pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-11T20:48:41.888 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:41 smithi146 bash[23809]: audit 2024-03-11T20:48:40.705619+0000 mon.a (mon.0) 557 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-11T20:48:41.888 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:41 smithi146 bash[23809]: audit 2024-03-11T20:48:40.707501+0000 mon.a (mon.0) 558 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:48:41.888 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:41 smithi146 bash[23809]: cephadm 2024-03-11T20:48:40.709339+0000 mgr.y (mgr.14152) 276 : cephadm 1 Deploying daemon osd.6 on smithi146 2024-03-11T20:48:42.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:41 smithi023 bash[29146]: audit 2024-03-11T20:48:40.705619+0000 mon.a (mon.0) 557 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-11T20:48:42.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:41 smithi023 bash[29146]: audit 2024-03-11T20:48:40.707501+0000 mon.a (mon.0) 558 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:48:42.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:41 smithi023 bash[29146]: cephadm 2024-03-11T20:48:40.709339+0000 mgr.y (mgr.14152) 276 : cephadm 1 Deploying daemon osd.6 on smithi146 2024-03-11T20:48:42.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:41 smithi023 bash[21372]: audit 2024-03-11T20:48:40.705619+0000 mon.a (mon.0) 557 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-11T20:48:42.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:41 smithi023 bash[21372]: audit 2024-03-11T20:48:40.707501+0000 mon.a (mon.0) 558 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:48:42.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:41 smithi023 bash[21372]: cephadm 2024-03-11T20:48:40.709339+0000 mgr.y (mgr.14152) 276 : cephadm 1 Deploying daemon osd.6 on smithi146 2024-03-11T20:48:42.741 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:42 smithi146 bash[23809]: cluster 2024-03-11T20:48:41.416445+0000 mgr.y (mgr.14152) 277 : cluster 0 pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-11T20:48:43.033 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:42 smithi023 bash[29146]: cluster 2024-03-11T20:48:41.416445+0000 mgr.y (mgr.14152) 277 : cluster 0 pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-11T20:48:43.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:42 smithi023 bash[21372]: cluster 2024-03-11T20:48:41.416445+0000 mgr.y (mgr.14152) 277 : cluster 0 pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-11T20:48:44.829 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:44 smithi146 bash[23809]: cluster 2024-03-11T20:48:43.416985+0000 mgr.y (mgr.14152) 278 : cluster 0 pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-11T20:48:45.033 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:44 smithi023 bash[29146]: cluster 2024-03-11T20:48:43.416985+0000 mgr.y (mgr.14152) 278 : cluster 0 pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-11T20:48:45.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:44 smithi023 bash[21372]: cluster 2024-03-11T20:48:43.416985+0000 mgr.y (mgr.14152) 278 : cluster 0 pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-11T20:48:45.088 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:44 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:48:45.088 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Mar 11 20:48:44 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:48:45.089 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 11 20:48:44 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:48:45.089 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 11 20:48:44 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:48:45.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:45 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:48:45.510 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Mar 11 20:48:45 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:48:45.511 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 11 20:48:45 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:48:45.511 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 11 20:48:45 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:48:46.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:46 smithi146 bash[23809]: audit 2024-03-11T20:48:45.254434+0000 mon.a (mon.0) 559 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:48:46.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:46 smithi146 bash[23809]: audit 2024-03-11T20:48:45.255335+0000 mon.a (mon.0) 560 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:48:46.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:46 smithi146 bash[23809]: audit 2024-03-11T20:48:45.271185+0000 mon.a (mon.0) 561 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:48:46.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:46 smithi146 bash[23809]: cluster 2024-03-11T20:48:45.417556+0000 mgr.y (mgr.14152) 279 : cluster 0 pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-11T20:48:46.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:46 smithi023 bash[21372]: audit 2024-03-11T20:48:45.254434+0000 mon.a (mon.0) 559 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:48:46.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:46 smithi023 bash[21372]: audit 2024-03-11T20:48:45.255335+0000 mon.a (mon.0) 560 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:48:46.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:46 smithi023 bash[21372]: audit 2024-03-11T20:48:45.271185+0000 mon.a (mon.0) 561 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:48:46.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:46 smithi023 bash[21372]: cluster 2024-03-11T20:48:45.417556+0000 mgr.y (mgr.14152) 279 : cluster 0 pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-11T20:48:46.534 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:46 smithi023 bash[29146]: audit 2024-03-11T20:48:45.254434+0000 mon.a (mon.0) 559 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:48:46.534 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:46 smithi023 bash[29146]: audit 2024-03-11T20:48:45.255335+0000 mon.a (mon.0) 560 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:48:46.534 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:46 smithi023 bash[29146]: audit 2024-03-11T20:48:45.271185+0000 mon.a (mon.0) 561 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:48:46.534 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:46 smithi023 bash[29146]: cluster 2024-03-11T20:48:45.417556+0000 mgr.y (mgr.14152) 279 : cluster 0 pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-11T20:48:48.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:48 smithi146 bash[23809]: cluster 2024-03-11T20:48:47.418090+0000 mgr.y (mgr.14152) 280 : cluster 0 pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-11T20:48:48.783 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:48 smithi023 bash[29146]: cluster 2024-03-11T20:48:47.418090+0000 mgr.y (mgr.14152) 280 : cluster 0 pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-11T20:48:48.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:48 smithi023 bash[21372]: cluster 2024-03-11T20:48:47.418090+0000 mgr.y (mgr.14152) 280 : cluster 0 pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-11T20:48:49.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:49 smithi146 bash[23809]: cluster 2024-03-11T20:48:48.511620+0000 osd.3 (osd.3) 5 : cluster 0 purged_snaps scrub starts 2024-03-11T20:48:49.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:49 smithi146 bash[23809]: cluster 2024-03-11T20:48:48.511716+0000 osd.3 (osd.3) 6 : cluster 0 purged_snaps scrub ok 2024-03-11T20:48:49.783 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:49 smithi023 bash[29146]: cluster 2024-03-11T20:48:48.511620+0000 osd.3 (osd.3) 5 : cluster 0 purged_snaps scrub starts 2024-03-11T20:48:49.783 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:49 smithi023 bash[29146]: cluster 2024-03-11T20:48:48.511716+0000 osd.3 (osd.3) 6 : cluster 0 purged_snaps scrub ok 2024-03-11T20:48:49.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:49 smithi023 bash[21372]: cluster 2024-03-11T20:48:48.511620+0000 osd.3 (osd.3) 5 : cluster 0 purged_snaps scrub starts 2024-03-11T20:48:49.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:49 smithi023 bash[21372]: cluster 2024-03-11T20:48:48.511716+0000 osd.3 (osd.3) 6 : cluster 0 purged_snaps scrub ok 2024-03-11T20:48:50.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:50 smithi146 bash[23809]: cluster 2024-03-11T20:48:49.418664+0000 mgr.y (mgr.14152) 281 : cluster 0 pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-11T20:48:50.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:50 smithi146 bash[23809]: cluster 2024-03-11T20:48:49.503782+0000 mon.a (mon.0) 562 : cluster 0 osdmap e54: 7 total, 6 up, 7 in 2024-03-11T20:48:50.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:50 smithi146 bash[23809]: audit 2024-03-11T20:48:49.504030+0000 mon.a (mon.0) 563 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T20:48:50.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:50 smithi023 bash[21372]: cluster 2024-03-11T20:48:49.418664+0000 mgr.y (mgr.14152) 281 : cluster 0 pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-11T20:48:50.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:50 smithi023 bash[21372]: cluster 2024-03-11T20:48:49.503782+0000 mon.a (mon.0) 562 : cluster 0 osdmap e54: 7 total, 6 up, 7 in 2024-03-11T20:48:50.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:50 smithi023 bash[21372]: audit 2024-03-11T20:48:49.504030+0000 mon.a (mon.0) 563 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T20:48:50.783 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:50 smithi023 bash[29146]: cluster 2024-03-11T20:48:49.418664+0000 mgr.y (mgr.14152) 281 : cluster 0 pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-11T20:48:50.784 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:50 smithi023 bash[29146]: cluster 2024-03-11T20:48:49.503782+0000 mon.a (mon.0) 562 : cluster 0 osdmap e54: 7 total, 6 up, 7 in 2024-03-11T20:48:50.784 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:50 smithi023 bash[29146]: audit 2024-03-11T20:48:49.504030+0000 mon.a (mon.0) 563 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T20:48:52.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:52 smithi146 bash[23809]: cluster 2024-03-11T20:48:51.419196+0000 mgr.y (mgr.14152) 282 : cluster 0 pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-11T20:48:52.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:52 smithi146 bash[23809]: audit 2024-03-11T20:48:51.638015+0000 mon.a (mon.0) 564 : audit 1 from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-11T20:48:52.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:52 smithi146 bash[23809]: audit 2024-03-11T20:48:51.640312+0000 mon.b (mon.1) 19 : audit 1 from='osd.6 [v2:172.21.15.146:6816/3334092376,v1:172.21.15.146:6817/3334092376]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-11T20:48:52.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:52 smithi146 bash[23809]: audit 2024-03-11T20:48:52.268612+0000 mon.a (mon.0) 565 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:48:52.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:52 smithi146 bash[23809]: audit 2024-03-11T20:48:52.283003+0000 mon.a (mon.0) 566 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:48:52.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:52 smithi146 bash[23809]: audit 2024-03-11T20:48:52.285242+0000 mon.a (mon.0) 567 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:48:52.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:52 smithi146 bash[23809]: audit 2024-03-11T20:48:52.287123+0000 mon.a (mon.0) 568 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:48:52.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:52 smithi146 bash[23809]: audit 2024-03-11T20:48:52.297480+0000 mon.a (mon.0) 569 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:48:52.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:52 smithi023 bash[21372]: cluster 2024-03-11T20:48:51.419196+0000 mgr.y (mgr.14152) 282 : cluster 0 pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-11T20:48:52.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:52 smithi023 bash[21372]: audit 2024-03-11T20:48:51.638015+0000 mon.a (mon.0) 564 : audit 1 from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-11T20:48:52.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:52 smithi023 bash[21372]: audit 2024-03-11T20:48:51.640312+0000 mon.b (mon.1) 19 : audit 1 from='osd.6 [v2:172.21.15.146:6816/3334092376,v1:172.21.15.146:6817/3334092376]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-11T20:48:52.784 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:52 smithi023 bash[21372]: audit 2024-03-11T20:48:52.268612+0000 mon.a (mon.0) 565 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:48:52.784 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:52 smithi023 bash[21372]: audit 2024-03-11T20:48:52.283003+0000 mon.a (mon.0) 566 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:48:52.784 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:52 smithi023 bash[21372]: audit 2024-03-11T20:48:52.285242+0000 mon.a (mon.0) 567 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:48:52.784 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:52 smithi023 bash[21372]: audit 2024-03-11T20:48:52.287123+0000 mon.a (mon.0) 568 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:48:52.784 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:52 smithi023 bash[21372]: audit 2024-03-11T20:48:52.297480+0000 mon.a (mon.0) 569 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:48:52.784 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:52 smithi023 bash[29146]: cluster 2024-03-11T20:48:51.419196+0000 mgr.y (mgr.14152) 282 : cluster 0 pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-11T20:48:52.784 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:52 smithi023 bash[29146]: audit 2024-03-11T20:48:51.638015+0000 mon.a (mon.0) 564 : audit 1 from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-11T20:48:52.784 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:52 smithi023 bash[29146]: audit 2024-03-11T20:48:51.640312+0000 mon.b (mon.1) 19 : audit 1 from='osd.6 [v2:172.21.15.146:6816/3334092376,v1:172.21.15.146:6817/3334092376]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-11T20:48:52.784 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:52 smithi023 bash[29146]: audit 2024-03-11T20:48:52.268612+0000 mon.a (mon.0) 565 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:48:52.784 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:52 smithi023 bash[29146]: audit 2024-03-11T20:48:52.283003+0000 mon.a (mon.0) 566 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:48:52.784 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:52 smithi023 bash[29146]: audit 2024-03-11T20:48:52.285242+0000 mon.a (mon.0) 567 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:48:52.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:52 smithi023 bash[29146]: audit 2024-03-11T20:48:52.287123+0000 mon.a (mon.0) 568 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:48:52.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:52 smithi023 bash[29146]: audit 2024-03-11T20:48:52.297480+0000 mon.a (mon.0) 569 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:48:54.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:53 smithi146 bash[23809]: audit 2024-03-11T20:48:52.534367+0000 mon.a (mon.0) 570 : audit 1 from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-11T20:48:54.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:53 smithi146 bash[23809]: cluster 2024-03-11T20:48:52.541302+0000 mon.a (mon.0) 571 : cluster 0 osdmap e55: 7 total, 6 up, 7 in 2024-03-11T20:48:54.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:53 smithi146 bash[23809]: audit 2024-03-11T20:48:52.541846+0000 mon.a (mon.0) 572 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T20:48:54.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:53 smithi146 bash[23809]: audit 2024-03-11T20:48:52.543959+0000 mon.a (mon.0) 573 : audit 1 from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-03-11T20:48:54.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:53 smithi146 bash[23809]: audit 2024-03-11T20:48:52.546035+0000 mon.b (mon.1) 20 : audit 1 from='osd.6 [v2:172.21.15.146:6816/3334092376,v1:172.21.15.146:6817/3334092376]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-03-11T20:48:54.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:53 smithi023 bash[21372]: audit 2024-03-11T20:48:52.534367+0000 mon.a (mon.0) 570 : audit 1 from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-11T20:48:54.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:53 smithi023 bash[21372]: cluster 2024-03-11T20:48:52.541302+0000 mon.a (mon.0) 571 : cluster 0 osdmap e55: 7 total, 6 up, 7 in 2024-03-11T20:48:54.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:53 smithi023 bash[21372]: audit 2024-03-11T20:48:52.541846+0000 mon.a (mon.0) 572 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T20:48:54.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:53 smithi023 bash[21372]: audit 2024-03-11T20:48:52.543959+0000 mon.a (mon.0) 573 : audit 1 from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-03-11T20:48:54.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:53 smithi023 bash[21372]: audit 2024-03-11T20:48:52.546035+0000 mon.b (mon.1) 20 : audit 1 from='osd.6 [v2:172.21.15.146:6816/3334092376,v1:172.21.15.146:6817/3334092376]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-03-11T20:48:54.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:53 smithi023 bash[29146]: audit 2024-03-11T20:48:52.534367+0000 mon.a (mon.0) 570 : audit 1 from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-11T20:48:54.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:53 smithi023 bash[29146]: cluster 2024-03-11T20:48:52.541302+0000 mon.a (mon.0) 571 : cluster 0 osdmap e55: 7 total, 6 up, 7 in 2024-03-11T20:48:54.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:53 smithi023 bash[29146]: audit 2024-03-11T20:48:52.541846+0000 mon.a (mon.0) 572 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T20:48:54.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:53 smithi023 bash[29146]: audit 2024-03-11T20:48:52.543959+0000 mon.a (mon.0) 573 : audit 1 from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-03-11T20:48:54.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:53 smithi023 bash[29146]: audit 2024-03-11T20:48:52.546035+0000 mon.b (mon.1) 20 : audit 1 from='osd.6 [v2:172.21.15.146:6816/3334092376,v1:172.21.15.146:6817/3334092376]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-03-11T20:48:54.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:54 smithi146 bash[23809]: cluster 2024-03-11T20:48:53.419867+0000 mgr.y (mgr.14152) 283 : cluster 0 pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-11T20:48:54.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:54 smithi146 bash[23809]: audit 2024-03-11T20:48:53.538477+0000 mon.a (mon.0) 574 : audit 1 from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]': finished 2024-03-11T20:48:54.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:54 smithi146 bash[23809]: cluster 2024-03-11T20:48:53.547901+0000 mon.a (mon.0) 575 : cluster 0 osdmap e56: 7 total, 6 up, 7 in 2024-03-11T20:48:54.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:54 smithi146 bash[23809]: audit 2024-03-11T20:48:53.549192+0000 mon.a (mon.0) 576 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T20:48:54.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:54 smithi146 bash[23809]: audit 2024-03-11T20:48:53.551470+0000 mon.a (mon.0) 577 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T20:48:54.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:54 smithi146 bash[23809]: audit 2024-03-11T20:48:54.080836+0000 mon.a (mon.0) 578 : audit 1 from='osd.6 ' entity='osd.6' 2024-03-11T20:48:54.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:54 smithi146 bash[23809]: cluster 2024-03-11T20:48:54.541467+0000 mon.a (mon.0) 579 : cluster 1 osd.6 [v2:172.21.15.146:6816/3334092376,v1:172.21.15.146:6817/3334092376] boot 2024-03-11T20:48:54.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:54 smithi146 bash[23809]: cluster 2024-03-11T20:48:54.541581+0000 mon.a (mon.0) 580 : cluster 0 osdmap e57: 7 total, 7 up, 7 in 2024-03-11T20:48:54.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:54 smithi146 bash[23809]: audit 2024-03-11T20:48:54.542520+0000 mon.a (mon.0) 581 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T20:48:55.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:54 smithi023 bash[21372]: cluster 2024-03-11T20:48:53.419867+0000 mgr.y (mgr.14152) 283 : cluster 0 pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-11T20:48:55.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:54 smithi023 bash[21372]: audit 2024-03-11T20:48:53.538477+0000 mon.a (mon.0) 574 : audit 1 from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]': finished 2024-03-11T20:48:55.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:54 smithi023 bash[21372]: cluster 2024-03-11T20:48:53.547901+0000 mon.a (mon.0) 575 : cluster 0 osdmap e56: 7 total, 6 up, 7 in 2024-03-11T20:48:55.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:54 smithi023 bash[21372]: audit 2024-03-11T20:48:53.549192+0000 mon.a (mon.0) 576 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T20:48:55.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:54 smithi023 bash[21372]: audit 2024-03-11T20:48:53.551470+0000 mon.a (mon.0) 577 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T20:48:55.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:54 smithi023 bash[21372]: audit 2024-03-11T20:48:54.080836+0000 mon.a (mon.0) 578 : audit 1 from='osd.6 ' entity='osd.6' 2024-03-11T20:48:55.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:54 smithi023 bash[21372]: cluster 2024-03-11T20:48:54.541467+0000 mon.a (mon.0) 579 : cluster 1 osd.6 [v2:172.21.15.146:6816/3334092376,v1:172.21.15.146:6817/3334092376] boot 2024-03-11T20:48:55.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:54 smithi023 bash[21372]: cluster 2024-03-11T20:48:54.541581+0000 mon.a (mon.0) 580 : cluster 0 osdmap e57: 7 total, 7 up, 7 in 2024-03-11T20:48:55.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:54 smithi023 bash[21372]: audit 2024-03-11T20:48:54.542520+0000 mon.a (mon.0) 581 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T20:48:55.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:54 smithi023 bash[29146]: cluster 2024-03-11T20:48:53.419867+0000 mgr.y (mgr.14152) 283 : cluster 0 pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-11T20:48:55.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:54 smithi023 bash[29146]: audit 2024-03-11T20:48:53.538477+0000 mon.a (mon.0) 574 : audit 1 from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]': finished 2024-03-11T20:48:55.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:54 smithi023 bash[29146]: cluster 2024-03-11T20:48:53.547901+0000 mon.a (mon.0) 575 : cluster 0 osdmap e56: 7 total, 6 up, 7 in 2024-03-11T20:48:55.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:54 smithi023 bash[29146]: audit 2024-03-11T20:48:53.549192+0000 mon.a (mon.0) 576 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T20:48:55.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:54 smithi023 bash[29146]: audit 2024-03-11T20:48:53.551470+0000 mon.a (mon.0) 577 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T20:48:55.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:54 smithi023 bash[29146]: audit 2024-03-11T20:48:54.080836+0000 mon.a (mon.0) 578 : audit 1 from='osd.6 ' entity='osd.6' 2024-03-11T20:48:55.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:54 smithi023 bash[29146]: cluster 2024-03-11T20:48:54.541467+0000 mon.a (mon.0) 579 : cluster 1 osd.6 [v2:172.21.15.146:6816/3334092376,v1:172.21.15.146:6817/3334092376] boot 2024-03-11T20:48:55.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:54 smithi023 bash[29146]: cluster 2024-03-11T20:48:54.541581+0000 mon.a (mon.0) 580 : cluster 0 osdmap e57: 7 total, 7 up, 7 in 2024-03-11T20:48:55.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:54 smithi023 bash[29146]: audit 2024-03-11T20:48:54.542520+0000 mon.a (mon.0) 581 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T20:48:55.835 INFO:teuthology.orchestra.run.smithi146.stdout:Created osd(s) 6 on host 'smithi146' 2024-03-11T20:48:55.855 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:55 smithi146 bash[23809]: cluster 2024-03-11T20:48:52.621978+0000 osd.6 (osd.6) 1 : cluster 0 purged_snaps scrub starts 2024-03-11T20:48:55.855 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:55 smithi146 bash[23809]: cluster 2024-03-11T20:48:52.622131+0000 osd.6 (osd.6) 2 : cluster 0 purged_snaps scrub ok 2024-03-11T20:48:55.855 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:55 smithi146 bash[23809]: cluster 2024-03-11T20:48:55.545974+0000 mon.a (mon.0) 582 : cluster 0 osdmap e58: 7 total, 7 up, 7 in 2024-03-11T20:48:56.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:55 smithi023 bash[21372]: cluster 2024-03-11T20:48:52.621978+0000 osd.6 (osd.6) 1 : cluster 0 purged_snaps scrub starts 2024-03-11T20:48:56.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:55 smithi023 bash[21372]: cluster 2024-03-11T20:48:52.622131+0000 osd.6 (osd.6) 2 : cluster 0 purged_snaps scrub ok 2024-03-11T20:48:56.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:55 smithi023 bash[21372]: cluster 2024-03-11T20:48:55.545974+0000 mon.a (mon.0) 582 : cluster 0 osdmap e58: 7 total, 7 up, 7 in 2024-03-11T20:48:56.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:55 smithi023 bash[29146]: cluster 2024-03-11T20:48:52.621978+0000 osd.6 (osd.6) 1 : cluster 0 purged_snaps scrub starts 2024-03-11T20:48:56.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:55 smithi023 bash[29146]: cluster 2024-03-11T20:48:52.622131+0000 osd.6 (osd.6) 2 : cluster 0 purged_snaps scrub ok 2024-03-11T20:48:56.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:55 smithi023 bash[29146]: cluster 2024-03-11T20:48:55.545974+0000 mon.a (mon.0) 582 : cluster 0 osdmap e58: 7 total, 7 up, 7 in 2024-03-11T20:48:56.488 DEBUG:teuthology.orchestra.run.smithi146:osd.6> sudo journalctl -f -n 0 -u ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@osd.6.service 2024-03-11T20:48:56.491 INFO:tasks.cephadm:Deploying osd.7 on smithi146 with /dev/vg_nvme/lv_1... 2024-03-11T20:48:56.491 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-11T20:48:57.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:56 smithi146 bash[23809]: cluster 2024-03-11T20:48:55.420506+0000 mgr.y (mgr.14152) 284 : cluster 0 pgmap v274: 1 pgs: 1 remapped+peering; 577 KiB data, 386 MiB used, 625 GiB / 626 GiB avail 2024-03-11T20:48:57.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:56 smithi146 bash[23809]: audit 2024-03-11T20:48:55.814583+0000 mon.a (mon.0) 583 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:48:57.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:56 smithi146 bash[23809]: audit 2024-03-11T20:48:55.824486+0000 mon.a (mon.0) 584 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:48:57.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:56 smithi146 bash[23809]: audit 2024-03-11T20:48:55.842518+0000 mon.a (mon.0) 585 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:48:57.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:56 smithi146 bash[23809]: cluster 2024-03-11T20:48:56.552326+0000 mon.a (mon.0) 586 : cluster 0 osdmap e59: 7 total, 7 up, 7 in 2024-03-11T20:48:57.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:56 smithi023 bash[21372]: cluster 2024-03-11T20:48:55.420506+0000 mgr.y (mgr.14152) 284 : cluster 0 pgmap v274: 1 pgs: 1 remapped+peering; 577 KiB data, 386 MiB used, 625 GiB / 626 GiB avail 2024-03-11T20:48:57.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:56 smithi023 bash[21372]: audit 2024-03-11T20:48:55.814583+0000 mon.a (mon.0) 583 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:48:57.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:56 smithi023 bash[21372]: audit 2024-03-11T20:48:55.824486+0000 mon.a (mon.0) 584 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:48:57.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:56 smithi023 bash[21372]: audit 2024-03-11T20:48:55.842518+0000 mon.a (mon.0) 585 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:48:57.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:56 smithi023 bash[21372]: cluster 2024-03-11T20:48:56.552326+0000 mon.a (mon.0) 586 : cluster 0 osdmap e59: 7 total, 7 up, 7 in 2024-03-11T20:48:57.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:56 smithi023 bash[29146]: cluster 2024-03-11T20:48:55.420506+0000 mgr.y (mgr.14152) 284 : cluster 0 pgmap v274: 1 pgs: 1 remapped+peering; 577 KiB data, 386 MiB used, 625 GiB / 626 GiB avail 2024-03-11T20:48:57.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:56 smithi023 bash[29146]: audit 2024-03-11T20:48:55.814583+0000 mon.a (mon.0) 583 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:48:57.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:56 smithi023 bash[29146]: audit 2024-03-11T20:48:55.824486+0000 mon.a (mon.0) 584 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:48:57.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:56 smithi023 bash[29146]: audit 2024-03-11T20:48:55.842518+0000 mon.a (mon.0) 585 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:48:57.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:56 smithi023 bash[29146]: cluster 2024-03-11T20:48:56.552326+0000 mon.a (mon.0) 586 : cluster 0 osdmap e59: 7 total, 7 up, 7 in 2024-03-11T20:48:59.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:58 smithi146 bash[23809]: cluster 2024-03-11T20:48:57.421047+0000 mgr.y (mgr.14152) 285 : cluster 0 pgmap v277: 1 pgs: 1 remapped+peering; 577 KiB data, 386 MiB used, 625 GiB / 626 GiB avail 2024-03-11T20:48:59.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:58 smithi146 bash[23809]: cluster 2024-03-11T20:48:57.556109+0000 mon.a (mon.0) 587 : cluster 0 osdmap e60: 7 total, 7 up, 7 in 2024-03-11T20:48:59.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:58 smithi023 bash[21372]: cluster 2024-03-11T20:48:57.421047+0000 mgr.y (mgr.14152) 285 : cluster 0 pgmap v277: 1 pgs: 1 remapped+peering; 577 KiB data, 386 MiB used, 625 GiB / 626 GiB avail 2024-03-11T20:48:59.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:58 smithi023 bash[21372]: cluster 2024-03-11T20:48:57.556109+0000 mon.a (mon.0) 587 : cluster 0 osdmap e60: 7 total, 7 up, 7 in 2024-03-11T20:48:59.033 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:58 smithi023 bash[29146]: cluster 2024-03-11T20:48:57.421047+0000 mgr.y (mgr.14152) 285 : cluster 0 pgmap v277: 1 pgs: 1 remapped+peering; 577 KiB data, 386 MiB used, 625 GiB / 626 GiB avail 2024-03-11T20:48:59.033 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:58 smithi023 bash[29146]: cluster 2024-03-11T20:48:57.556109+0000 mon.a (mon.0) 587 : cluster 0 osdmap e60: 7 total, 7 up, 7 in 2024-03-11T20:49:00.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:48:59 smithi146 bash[23809]: cluster 2024-03-11T20:48:58.562555+0000 mon.a (mon.0) 588 : cluster 0 osdmap e61: 7 total, 7 up, 7 in 2024-03-11T20:49:00.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:48:59 smithi023 bash[21372]: cluster 2024-03-11T20:48:58.562555+0000 mon.a (mon.0) 588 : cluster 0 osdmap e61: 7 total, 7 up, 7 in 2024-03-11T20:49:00.033 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:48:59 smithi023 bash[29146]: cluster 2024-03-11T20:48:58.562555+0000 mon.a (mon.0) 588 : cluster 0 osdmap e61: 7 total, 7 up, 7 in 2024-03-11T20:49:00.790 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T20:49:01.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:00 smithi146 bash[23809]: cluster 2024-03-11T20:48:59.421697+0000 mgr.y (mgr.14152) 286 : cluster 0 pgmap v280: 1 pgs: 1 remapped+peering; 577 KiB data, 385 MiB used, 625 GiB / 626 GiB avail 2024-03-11T20:49:01.033 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:00 smithi023 bash[29146]: cluster 2024-03-11T20:48:59.421697+0000 mgr.y (mgr.14152) 286 : cluster 0 pgmap v280: 1 pgs: 1 remapped+peering; 577 KiB data, 385 MiB used, 625 GiB / 626 GiB avail 2024-03-11T20:49:01.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:00 smithi023 bash[21372]: cluster 2024-03-11T20:48:59.421697+0000 mgr.y (mgr.14152) 286 : cluster 0 pgmap v280: 1 pgs: 1 remapped+peering; 577 KiB data, 385 MiB used, 625 GiB / 626 GiB avail 2024-03-11T20:49:03.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:02 smithi146 bash[23809]: cluster 2024-03-11T20:49:01.422262+0000 mgr.y (mgr.14152) 287 : cluster 0 pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 211 MiB used, 626 GiB / 626 GiB avail; 98 KiB/s, 0 objects/s recovering 2024-03-11T20:49:03.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:02 smithi023 bash[21372]: cluster 2024-03-11T20:49:01.422262+0000 mgr.y (mgr.14152) 287 : cluster 0 pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 211 MiB used, 626 GiB / 626 GiB avail; 98 KiB/s, 0 objects/s recovering 2024-03-11T20:49:03.033 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:02 smithi023 bash[29146]: cluster 2024-03-11T20:49:01.422262+0000 mgr.y (mgr.14152) 287 : cluster 0 pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 211 MiB used, 626 GiB / 626 GiB avail; 98 KiB/s, 0 objects/s recovering 2024-03-11T20:49:05.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:04 smithi146 bash[23809]: cluster 2024-03-11T20:49:03.422826+0000 mgr.y (mgr.14152) 288 : cluster 0 pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 211 MiB used, 626 GiB / 626 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-03-11T20:49:05.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:04 smithi023 bash[21372]: cluster 2024-03-11T20:49:03.422826+0000 mgr.y (mgr.14152) 288 : cluster 0 pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 211 MiB used, 626 GiB / 626 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-03-11T20:49:05.033 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:04 smithi023 bash[29146]: cluster 2024-03-11T20:49:03.422826+0000 mgr.y (mgr.14152) 288 : cluster 0 pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 211 MiB used, 626 GiB / 626 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-03-11T20:49:05.290 INFO:teuthology.orchestra.run.smithi146.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-11T20:49:05.290 INFO:teuthology.orchestra.run.smithi146.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-11T20:49:05.291 INFO:teuthology.orchestra.run.smithi146.stderr: stderr: 10+0 records in 2024-03-11T20:49:05.291 INFO:teuthology.orchestra.run.smithi146.stderr:10+0 records out 2024-03-11T20:49:05.291 INFO:teuthology.orchestra.run.smithi146.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0141637 s, 740 MB/s 2024-03-11T20:49:05.291 INFO:teuthology.orchestra.run.smithi146.stderr:--> Zapping successful for: 2024-03-11T20:49:06.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:05 smithi146 bash[23809]: cluster 2024-03-11T20:49:04.462945+0000 osd.5 (osd.5) 5 : cluster 0 purged_snaps scrub starts 2024-03-11T20:49:06.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:05 smithi146 bash[23809]: cluster 2024-03-11T20:49:04.463082+0000 osd.5 (osd.5) 6 : cluster 0 purged_snaps scrub ok 2024-03-11T20:49:06.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:05 smithi146 bash[23809]: cluster 2024-03-11T20:49:04.604121+0000 mon.a (mon.0) 589 : cluster 0 osdmap e62: 7 total, 7 up, 7 in 2024-03-11T20:49:06.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:05 smithi023 bash[21372]: cluster 2024-03-11T20:49:04.462945+0000 osd.5 (osd.5) 5 : cluster 0 purged_snaps scrub starts 2024-03-11T20:49:06.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:05 smithi023 bash[21372]: cluster 2024-03-11T20:49:04.463082+0000 osd.5 (osd.5) 6 : cluster 0 purged_snaps scrub ok 2024-03-11T20:49:06.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:05 smithi023 bash[21372]: cluster 2024-03-11T20:49:04.604121+0000 mon.a (mon.0) 589 : cluster 0 osdmap e62: 7 total, 7 up, 7 in 2024-03-11T20:49:06.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:05 smithi023 bash[29146]: cluster 2024-03-11T20:49:04.462945+0000 osd.5 (osd.5) 5 : cluster 0 purged_snaps scrub starts 2024-03-11T20:49:06.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:05 smithi023 bash[29146]: cluster 2024-03-11T20:49:04.463082+0000 osd.5 (osd.5) 6 : cluster 0 purged_snaps scrub ok 2024-03-11T20:49:06.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:05 smithi023 bash[29146]: cluster 2024-03-11T20:49:04.604121+0000 mon.a (mon.0) 589 : cluster 0 osdmap e62: 7 total, 7 up, 7 in 2024-03-11T20:49:06.148 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi146:vg_nvme/lv_1 2024-03-11T20:49:07.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:06 smithi146 bash[23809]: cluster 2024-03-11T20:49:05.423441+0000 mgr.y (mgr.14152) 289 : cluster 0 pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 211 MiB used, 626 GiB / 626 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-03-11T20:49:07.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:06 smithi146 bash[23809]: audit 2024-03-11T20:49:06.600515+0000 mon.a (mon.0) 590 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:49:07.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:06 smithi146 bash[23809]: audit 2024-03-11T20:49:06.611303+0000 mon.a (mon.0) 591 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:49:07.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:06 smithi146 bash[23809]: audit 2024-03-11T20:49:06.613839+0000 mon.a (mon.0) 592 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:49:07.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:06 smithi146 bash[23809]: audit 2024-03-11T20:49:06.615812+0000 mon.a (mon.0) 593 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:49:07.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:06 smithi146 bash[23809]: audit 2024-03-11T20:49:06.617473+0000 mon.a (mon.0) 594 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:49:07.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:06 smithi146 bash[23809]: audit 2024-03-11T20:49:06.626031+0000 mon.a (mon.0) 595 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:49:07.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:06 smithi023 bash[21372]: cluster 2024-03-11T20:49:05.423441+0000 mgr.y (mgr.14152) 289 : cluster 0 pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 211 MiB used, 626 GiB / 626 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-03-11T20:49:07.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:06 smithi023 bash[21372]: audit 2024-03-11T20:49:06.600515+0000 mon.a (mon.0) 590 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:49:07.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:06 smithi023 bash[21372]: audit 2024-03-11T20:49:06.611303+0000 mon.a (mon.0) 591 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:49:07.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:06 smithi023 bash[21372]: audit 2024-03-11T20:49:06.613839+0000 mon.a (mon.0) 592 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:49:07.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:06 smithi023 bash[21372]: audit 2024-03-11T20:49:06.615812+0000 mon.a (mon.0) 593 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:49:07.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:06 smithi023 bash[21372]: audit 2024-03-11T20:49:06.617473+0000 mon.a (mon.0) 594 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:49:07.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:06 smithi023 bash[21372]: audit 2024-03-11T20:49:06.626031+0000 mon.a (mon.0) 595 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:49:07.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:06 smithi023 bash[29146]: cluster 2024-03-11T20:49:05.423441+0000 mgr.y (mgr.14152) 289 : cluster 0 pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 211 MiB used, 626 GiB / 626 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-03-11T20:49:07.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:06 smithi023 bash[29146]: audit 2024-03-11T20:49:06.600515+0000 mon.a (mon.0) 590 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:49:07.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:06 smithi023 bash[29146]: audit 2024-03-11T20:49:06.611303+0000 mon.a (mon.0) 591 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:49:07.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:06 smithi023 bash[29146]: audit 2024-03-11T20:49:06.613839+0000 mon.a (mon.0) 592 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:49:07.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:06 smithi023 bash[29146]: audit 2024-03-11T20:49:06.615812+0000 mon.a (mon.0) 593 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:49:07.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:06 smithi023 bash[29146]: audit 2024-03-11T20:49:06.617473+0000 mon.a (mon.0) 594 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:49:07.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:06 smithi023 bash[29146]: audit 2024-03-11T20:49:06.626031+0000 mon.a (mon.0) 595 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:49:08.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:07 smithi146 bash[23809]: cephadm 2024-03-11T20:49:06.590796+0000 mgr.y (mgr.14152) 290 : cephadm 1 Detected new or changed devices on smithi146 2024-03-11T20:49:08.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:07 smithi146 bash[23809]: cephadm 2024-03-11T20:49:06.618871+0000 mgr.y (mgr.14152) 291 : cephadm 1 Adjusting osd_memory_target on smithi146 to 5755M 2024-03-11T20:49:08.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:07 smithi146 bash[23809]: audit 2024-03-11T20:49:06.631016+0000 mon.a (mon.0) 596 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:49:08.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:07 smithi146 bash[23809]: audit 2024-03-11T20:49:06.639177+0000 mon.a (mon.0) 597 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:49:08.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:07 smithi146 bash[23809]: audit 2024-03-11T20:49:06.649744+0000 mon.a (mon.0) 598 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:49:08.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:07 smithi023 bash[21372]: cephadm 2024-03-11T20:49:06.590796+0000 mgr.y (mgr.14152) 290 : cephadm 1 Detected new or changed devices on smithi146 2024-03-11T20:49:08.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:07 smithi023 bash[21372]: cephadm 2024-03-11T20:49:06.618871+0000 mgr.y (mgr.14152) 291 : cephadm 1 Adjusting osd_memory_target on smithi146 to 5755M 2024-03-11T20:49:08.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:07 smithi023 bash[21372]: audit 2024-03-11T20:49:06.631016+0000 mon.a (mon.0) 596 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:49:08.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:07 smithi023 bash[21372]: audit 2024-03-11T20:49:06.639177+0000 mon.a (mon.0) 597 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:49:08.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:07 smithi023 bash[21372]: audit 2024-03-11T20:49:06.649744+0000 mon.a (mon.0) 598 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:49:08.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:07 smithi023 bash[29146]: cephadm 2024-03-11T20:49:06.590796+0000 mgr.y (mgr.14152) 290 : cephadm 1 Detected new or changed devices on smithi146 2024-03-11T20:49:08.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:07 smithi023 bash[29146]: cephadm 2024-03-11T20:49:06.618871+0000 mgr.y (mgr.14152) 291 : cephadm 1 Adjusting osd_memory_target on smithi146 to 5755M 2024-03-11T20:49:08.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:07 smithi023 bash[29146]: audit 2024-03-11T20:49:06.631016+0000 mon.a (mon.0) 596 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:49:08.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:07 smithi023 bash[29146]: audit 2024-03-11T20:49:06.639177+0000 mon.a (mon.0) 597 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:49:08.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:07 smithi023 bash[29146]: audit 2024-03-11T20:49:06.649744+0000 mon.a (mon.0) 598 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:49:09.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:08 smithi146 bash[23809]: cluster 2024-03-11T20:49:07.423987+0000 mgr.y (mgr.14152) 292 : cluster 0 pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 211 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-03-11T20:49:09.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:08 smithi023 bash[21372]: cluster 2024-03-11T20:49:07.423987+0000 mgr.y (mgr.14152) 292 : cluster 0 pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 211 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-03-11T20:49:09.033 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:08 smithi023 bash[29146]: cluster 2024-03-11T20:49:07.423987+0000 mgr.y (mgr.14152) 292 : cluster 0 pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 211 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-03-11T20:49:10.888 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T20:49:10.911 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:10 smithi146 bash[23809]: cluster 2024-03-11T20:49:09.424551+0000 mgr.y (mgr.14152) 293 : cluster 0 pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 211 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-11T20:49:11.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:10 smithi023 bash[21372]: cluster 2024-03-11T20:49:09.424551+0000 mgr.y (mgr.14152) 293 : cluster 0 pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 211 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-11T20:49:11.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:10 smithi023 bash[29146]: cluster 2024-03-11T20:49:09.424551+0000 mgr.y (mgr.14152) 293 : cluster 0 pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 211 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-11T20:49:13.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:12 smithi146 bash[23809]: cluster 2024-03-11T20:49:11.425106+0000 mgr.y (mgr.14152) 294 : cluster 0 pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 211 MiB used, 626 GiB / 626 GiB avail 2024-03-11T20:49:13.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:12 smithi023 bash[21372]: cluster 2024-03-11T20:49:11.425106+0000 mgr.y (mgr.14152) 294 : cluster 0 pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 211 MiB used, 626 GiB / 626 GiB avail 2024-03-11T20:49:13.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:12 smithi023 bash[29146]: cluster 2024-03-11T20:49:11.425106+0000 mgr.y (mgr.14152) 294 : cluster 0 pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 211 MiB used, 626 GiB / 626 GiB avail 2024-03-11T20:49:14.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:13 smithi146 bash[23809]: audit 2024-03-11T20:49:13.628259+0000 mon.a (mon.0) 599 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T20:49:14.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:13 smithi146 bash[23809]: audit 2024-03-11T20:49:13.634907+0000 mon.a (mon.0) 600 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T20:49:14.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:13 smithi146 bash[23809]: audit 2024-03-11T20:49:13.636631+0000 mon.a (mon.0) 601 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:49:14.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:13 smithi023 bash[21372]: audit 2024-03-11T20:49:13.628259+0000 mon.a (mon.0) 599 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T20:49:14.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:13 smithi023 bash[21372]: audit 2024-03-11T20:49:13.634907+0000 mon.a (mon.0) 600 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T20:49:14.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:13 smithi023 bash[21372]: audit 2024-03-11T20:49:13.636631+0000 mon.a (mon.0) 601 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:49:14.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:13 smithi023 bash[29146]: audit 2024-03-11T20:49:13.628259+0000 mon.a (mon.0) 599 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T20:49:14.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:13 smithi023 bash[29146]: audit 2024-03-11T20:49:13.634907+0000 mon.a (mon.0) 600 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T20:49:14.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:13 smithi023 bash[29146]: audit 2024-03-11T20:49:13.636631+0000 mon.a (mon.0) 601 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:49:15.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:14 smithi146 bash[23809]: cluster 2024-03-11T20:49:13.425658+0000 mgr.y (mgr.14152) 295 : cluster 0 pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 211 MiB used, 626 GiB / 626 GiB avail 2024-03-11T20:49:15.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:14 smithi146 bash[23809]: audit 2024-03-11T20:49:13.621076+0000 mgr.y (mgr.14152) 296 : audit 0 from='client.24265 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi146:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:49:15.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:14 smithi023 bash[21372]: cluster 2024-03-11T20:49:13.425658+0000 mgr.y (mgr.14152) 295 : cluster 0 pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 211 MiB used, 626 GiB / 626 GiB avail 2024-03-11T20:49:15.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:14 smithi023 bash[21372]: audit 2024-03-11T20:49:13.621076+0000 mgr.y (mgr.14152) 296 : audit 0 from='client.24265 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi146:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:49:15.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:14 smithi023 bash[29146]: cluster 2024-03-11T20:49:13.425658+0000 mgr.y (mgr.14152) 295 : cluster 0 pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 211 MiB used, 626 GiB / 626 GiB avail 2024-03-11T20:49:15.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:14 smithi023 bash[29146]: audit 2024-03-11T20:49:13.621076+0000 mgr.y (mgr.14152) 296 : audit 0 from='client.24265 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi146:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:49:17.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:16 smithi146 bash[23809]: cluster 2024-03-11T20:49:15.426255+0000 mgr.y (mgr.14152) 297 : cluster 0 pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 211 MiB used, 626 GiB / 626 GiB avail 2024-03-11T20:49:17.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:16 smithi023 bash[21372]: cluster 2024-03-11T20:49:15.426255+0000 mgr.y (mgr.14152) 297 : cluster 0 pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 211 MiB used, 626 GiB / 626 GiB avail 2024-03-11T20:49:17.033 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:16 smithi023 bash[29146]: cluster 2024-03-11T20:49:15.426255+0000 mgr.y (mgr.14152) 297 : cluster 0 pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 211 MiB used, 626 GiB / 626 GiB avail 2024-03-11T20:49:19.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:18 smithi146 bash[23809]: cluster 2024-03-11T20:49:17.426830+0000 mgr.y (mgr.14152) 298 : cluster 0 pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 211 MiB used, 626 GiB / 626 GiB avail 2024-03-11T20:49:19.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:18 smithi023 bash[21372]: cluster 2024-03-11T20:49:17.426830+0000 mgr.y (mgr.14152) 298 : cluster 0 pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 211 MiB used, 626 GiB / 626 GiB avail 2024-03-11T20:49:19.033 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:18 smithi023 bash[29146]: cluster 2024-03-11T20:49:17.426830+0000 mgr.y (mgr.14152) 298 : cluster 0 pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 211 MiB used, 626 GiB / 626 GiB avail 2024-03-11T20:49:21.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:20 smithi146 bash[23809]: cluster 2024-03-11T20:49:19.427423+0000 mgr.y (mgr.14152) 299 : cluster 0 pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 211 MiB used, 626 GiB / 626 GiB avail 2024-03-11T20:49:21.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:20 smithi023 bash[21372]: cluster 2024-03-11T20:49:19.427423+0000 mgr.y (mgr.14152) 299 : cluster 0 pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 211 MiB used, 626 GiB / 626 GiB avail 2024-03-11T20:49:21.033 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:20 smithi023 bash[29146]: cluster 2024-03-11T20:49:19.427423+0000 mgr.y (mgr.14152) 299 : cluster 0 pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 211 MiB used, 626 GiB / 626 GiB avail 2024-03-11T20:49:23.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:22 smithi146 bash[23809]: cluster 2024-03-11T20:49:21.427963+0000 mgr.y (mgr.14152) 300 : cluster 0 pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 211 MiB used, 626 GiB / 626 GiB avail 2024-03-11T20:49:23.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:22 smithi146 bash[23809]: audit 2024-03-11T20:49:21.978178+0000 mon.a (mon.0) 602 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bdbd11a2-fa09-4f82-93a2-6b2d9c750bfd"}]: dispatch 2024-03-11T20:49:23.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:22 smithi146 bash[23809]: audit 2024-03-11T20:49:21.980560+0000 mon.b (mon.1) 21 : audit 1 from='client.? 172.21.15.146:0/2850942623' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bdbd11a2-fa09-4f82-93a2-6b2d9c750bfd"}]: dispatch 2024-03-11T20:49:23.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:22 smithi146 bash[23809]: audit 2024-03-11T20:49:21.990504+0000 mon.a (mon.0) 603 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bdbd11a2-fa09-4f82-93a2-6b2d9c750bfd"}]': finished 2024-03-11T20:49:23.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:22 smithi146 bash[23809]: cluster 2024-03-11T20:49:22.000418+0000 mon.a (mon.0) 604 : cluster 0 osdmap e63: 8 total, 7 up, 8 in 2024-03-11T20:49:23.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:22 smithi146 bash[23809]: audit 2024-03-11T20:49:22.000845+0000 mon.a (mon.0) 605 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T20:49:23.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:22 smithi146 bash[23809]: audit 2024-03-11T20:49:22.680883+0000 mon.b (mon.1) 22 : audit 0 from='client.? 172.21.15.146:0/218426464' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T20:49:23.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:22 smithi023 bash[21372]: cluster 2024-03-11T20:49:21.427963+0000 mgr.y (mgr.14152) 300 : cluster 0 pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 211 MiB used, 626 GiB / 626 GiB avail 2024-03-11T20:49:23.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:22 smithi023 bash[21372]: audit 2024-03-11T20:49:21.978178+0000 mon.a (mon.0) 602 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bdbd11a2-fa09-4f82-93a2-6b2d9c750bfd"}]: dispatch 2024-03-11T20:49:23.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:22 smithi023 bash[21372]: audit 2024-03-11T20:49:21.980560+0000 mon.b (mon.1) 21 : audit 1 from='client.? 172.21.15.146:0/2850942623' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bdbd11a2-fa09-4f82-93a2-6b2d9c750bfd"}]: dispatch 2024-03-11T20:49:23.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:22 smithi023 bash[21372]: audit 2024-03-11T20:49:21.990504+0000 mon.a (mon.0) 603 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bdbd11a2-fa09-4f82-93a2-6b2d9c750bfd"}]': finished 2024-03-11T20:49:23.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:22 smithi023 bash[21372]: cluster 2024-03-11T20:49:22.000418+0000 mon.a (mon.0) 604 : cluster 0 osdmap e63: 8 total, 7 up, 8 in 2024-03-11T20:49:23.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:22 smithi023 bash[21372]: audit 2024-03-11T20:49:22.000845+0000 mon.a (mon.0) 605 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T20:49:23.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:22 smithi023 bash[21372]: audit 2024-03-11T20:49:22.680883+0000 mon.b (mon.1) 22 : audit 0 from='client.? 172.21.15.146:0/218426464' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T20:49:23.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:22 smithi023 bash[29146]: cluster 2024-03-11T20:49:21.427963+0000 mgr.y (mgr.14152) 300 : cluster 0 pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 211 MiB used, 626 GiB / 626 GiB avail 2024-03-11T20:49:23.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:22 smithi023 bash[29146]: audit 2024-03-11T20:49:21.978178+0000 mon.a (mon.0) 602 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bdbd11a2-fa09-4f82-93a2-6b2d9c750bfd"}]: dispatch 2024-03-11T20:49:23.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:22 smithi023 bash[29146]: audit 2024-03-11T20:49:21.980560+0000 mon.b (mon.1) 21 : audit 1 from='client.? 172.21.15.146:0/2850942623' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bdbd11a2-fa09-4f82-93a2-6b2d9c750bfd"}]: dispatch 2024-03-11T20:49:23.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:22 smithi023 bash[29146]: audit 2024-03-11T20:49:21.990504+0000 mon.a (mon.0) 603 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bdbd11a2-fa09-4f82-93a2-6b2d9c750bfd"}]': finished 2024-03-11T20:49:23.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:22 smithi023 bash[29146]: cluster 2024-03-11T20:49:22.000418+0000 mon.a (mon.0) 604 : cluster 0 osdmap e63: 8 total, 7 up, 8 in 2024-03-11T20:49:23.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:22 smithi023 bash[29146]: audit 2024-03-11T20:49:22.000845+0000 mon.a (mon.0) 605 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T20:49:23.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:22 smithi023 bash[29146]: audit 2024-03-11T20:49:22.680883+0000 mon.b (mon.1) 22 : audit 0 from='client.? 172.21.15.146:0/218426464' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T20:49:25.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:24 smithi146 bash[23809]: cluster 2024-03-11T20:49:23.428537+0000 mgr.y (mgr.14152) 301 : cluster 0 pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 211 MiB used, 626 GiB / 626 GiB avail 2024-03-11T20:49:25.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:24 smithi023 bash[21372]: cluster 2024-03-11T20:49:23.428537+0000 mgr.y (mgr.14152) 301 : cluster 0 pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 211 MiB used, 626 GiB / 626 GiB avail 2024-03-11T20:49:25.033 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:24 smithi023 bash[29146]: cluster 2024-03-11T20:49:23.428537+0000 mgr.y (mgr.14152) 301 : cluster 0 pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 211 MiB used, 626 GiB / 626 GiB avail 2024-03-11T20:49:27.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:26 smithi146 bash[23809]: cluster 2024-03-11T20:49:25.429121+0000 mgr.y (mgr.14152) 302 : cluster 0 pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-11T20:49:27.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:26 smithi023 bash[21372]: cluster 2024-03-11T20:49:25.429121+0000 mgr.y (mgr.14152) 302 : cluster 0 pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-11T20:49:27.033 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:26 smithi023 bash[29146]: cluster 2024-03-11T20:49:25.429121+0000 mgr.y (mgr.14152) 302 : cluster 0 pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-11T20:49:28.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:27 smithi023 bash[21372]: cluster 2024-03-11T20:49:26.779061+0000 osd.2 (osd.2) 7 : cluster 0 purged_snaps scrub starts 2024-03-11T20:49:28.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:27 smithi023 bash[21372]: cluster 2024-03-11T20:49:26.779191+0000 osd.2 (osd.2) 8 : cluster 0 purged_snaps scrub ok 2024-03-11T20:49:28.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:27 smithi023 bash[29146]: cluster 2024-03-11T20:49:26.779061+0000 osd.2 (osd.2) 7 : cluster 0 purged_snaps scrub starts 2024-03-11T20:49:28.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:27 smithi023 bash[29146]: cluster 2024-03-11T20:49:26.779191+0000 osd.2 (osd.2) 8 : cluster 0 purged_snaps scrub ok 2024-03-11T20:49:28.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:27 smithi146 bash[23809]: cluster 2024-03-11T20:49:26.779061+0000 osd.2 (osd.2) 7 : cluster 0 purged_snaps scrub starts 2024-03-11T20:49:28.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:27 smithi146 bash[23809]: cluster 2024-03-11T20:49:26.779191+0000 osd.2 (osd.2) 8 : cluster 0 purged_snaps scrub ok 2024-03-11T20:49:29.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:28 smithi023 bash[21372]: cluster 2024-03-11T20:49:27.429694+0000 mgr.y (mgr.14152) 303 : cluster 0 pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-11T20:49:29.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:28 smithi023 bash[21372]: cluster 2024-03-11T20:49:27.789410+0000 mon.a (mon.0) 606 : cluster 0 osdmap e64: 8 total, 7 up, 8 in 2024-03-11T20:49:29.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:28 smithi023 bash[21372]: audit 2024-03-11T20:49:27.789728+0000 mon.a (mon.0) 607 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T20:49:29.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:28 smithi023 bash[29146]: cluster 2024-03-11T20:49:27.429694+0000 mgr.y (mgr.14152) 303 : cluster 0 pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-11T20:49:29.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:28 smithi023 bash[29146]: cluster 2024-03-11T20:49:27.789410+0000 mon.a (mon.0) 606 : cluster 0 osdmap e64: 8 total, 7 up, 8 in 2024-03-11T20:49:29.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:28 smithi023 bash[29146]: audit 2024-03-11T20:49:27.789728+0000 mon.a (mon.0) 607 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T20:49:29.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:28 smithi146 bash[23809]: cluster 2024-03-11T20:49:27.429694+0000 mgr.y (mgr.14152) 303 : cluster 0 pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-11T20:49:29.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:28 smithi146 bash[23809]: cluster 2024-03-11T20:49:27.789410+0000 mon.a (mon.0) 606 : cluster 0 osdmap e64: 8 total, 7 up, 8 in 2024-03-11T20:49:29.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:28 smithi146 bash[23809]: audit 2024-03-11T20:49:27.789728+0000 mon.a (mon.0) 607 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T20:49:31.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:30 smithi146 bash[23809]: cluster 2024-03-11T20:49:29.430248+0000 mgr.y (mgr.14152) 304 : cluster 0 pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-11T20:49:31.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:30 smithi146 bash[23809]: cluster 2024-03-11T20:49:29.450451+0000 osd.1 (osd.1) 11 : cluster 0 purged_snaps scrub starts 2024-03-11T20:49:31.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:30 smithi146 bash[23809]: cluster 2024-03-11T20:49:29.450541+0000 osd.1 (osd.1) 12 : cluster 0 purged_snaps scrub ok 2024-03-11T20:49:31.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:30 smithi146 bash[23809]: cluster 2024-03-11T20:49:29.803268+0000 mon.a (mon.0) 608 : cluster 0 osdmap e65: 8 total, 7 up, 8 in 2024-03-11T20:49:31.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:30 smithi146 bash[23809]: audit 2024-03-11T20:49:29.803534+0000 mon.a (mon.0) 609 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T20:49:31.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:30 smithi023 bash[21372]: cluster 2024-03-11T20:49:29.430248+0000 mgr.y (mgr.14152) 304 : cluster 0 pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-11T20:49:31.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:30 smithi023 bash[21372]: cluster 2024-03-11T20:49:29.450451+0000 osd.1 (osd.1) 11 : cluster 0 purged_snaps scrub starts 2024-03-11T20:49:31.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:30 smithi023 bash[21372]: cluster 2024-03-11T20:49:29.450541+0000 osd.1 (osd.1) 12 : cluster 0 purged_snaps scrub ok 2024-03-11T20:49:31.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:30 smithi023 bash[21372]: cluster 2024-03-11T20:49:29.803268+0000 mon.a (mon.0) 608 : cluster 0 osdmap e65: 8 total, 7 up, 8 in 2024-03-11T20:49:31.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:30 smithi023 bash[21372]: audit 2024-03-11T20:49:29.803534+0000 mon.a (mon.0) 609 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T20:49:31.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:30 smithi023 bash[29146]: cluster 2024-03-11T20:49:29.430248+0000 mgr.y (mgr.14152) 304 : cluster 0 pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-11T20:49:31.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:30 smithi023 bash[29146]: cluster 2024-03-11T20:49:29.450451+0000 osd.1 (osd.1) 11 : cluster 0 purged_snaps scrub starts 2024-03-11T20:49:31.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:30 smithi023 bash[29146]: cluster 2024-03-11T20:49:29.450541+0000 osd.1 (osd.1) 12 : cluster 0 purged_snaps scrub ok 2024-03-11T20:49:31.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:30 smithi023 bash[29146]: cluster 2024-03-11T20:49:29.803268+0000 mon.a (mon.0) 608 : cluster 0 osdmap e65: 8 total, 7 up, 8 in 2024-03-11T20:49:31.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:30 smithi023 bash[29146]: audit 2024-03-11T20:49:29.803534+0000 mon.a (mon.0) 609 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T20:49:33.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:32 smithi146 bash[23809]: cluster 2024-03-11T20:49:31.430950+0000 mgr.y (mgr.14152) 305 : cluster 0 pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-11T20:49:33.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:32 smithi023 bash[21372]: cluster 2024-03-11T20:49:31.430950+0000 mgr.y (mgr.14152) 305 : cluster 0 pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-11T20:49:33.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:32 smithi023 bash[29146]: cluster 2024-03-11T20:49:31.430950+0000 mgr.y (mgr.14152) 305 : cluster 0 pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-11T20:49:35.077 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:34 smithi146 bash[23809]: cluster 2024-03-11T20:49:33.431723+0000 mgr.y (mgr.14152) 306 : cluster 0 pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-11T20:49:35.077 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:34 smithi146 bash[23809]: audit 2024-03-11T20:49:34.678857+0000 mon.a (mon.0) 610 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-11T20:49:35.077 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:34 smithi146 bash[23809]: audit 2024-03-11T20:49:34.680609+0000 mon.a (mon.0) 611 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:49:35.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:34 smithi023 bash[21372]: cluster 2024-03-11T20:49:33.431723+0000 mgr.y (mgr.14152) 306 : cluster 0 pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-11T20:49:35.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:34 smithi023 bash[21372]: audit 2024-03-11T20:49:34.678857+0000 mon.a (mon.0) 610 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-11T20:49:35.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:34 smithi023 bash[21372]: audit 2024-03-11T20:49:34.680609+0000 mon.a (mon.0) 611 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:49:35.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:34 smithi023 bash[29146]: cluster 2024-03-11T20:49:33.431723+0000 mgr.y (mgr.14152) 306 : cluster 0 pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-11T20:49:35.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:34 smithi023 bash[29146]: audit 2024-03-11T20:49:34.678857+0000 mon.a (mon.0) 610 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-11T20:49:35.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:34 smithi023 bash[29146]: audit 2024-03-11T20:49:34.680609+0000 mon.a (mon.0) 611 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:49:36.085 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:35 smithi146 bash[23809]: cephadm 2024-03-11T20:49:34.682286+0000 mgr.y (mgr.14152) 307 : cephadm 1 Deploying daemon osd.7 on smithi146 2024-03-11T20:49:36.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:35 smithi023 bash[21372]: cephadm 2024-03-11T20:49:34.682286+0000 mgr.y (mgr.14152) 307 : cephadm 1 Deploying daemon osd.7 on smithi146 2024-03-11T20:49:36.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:35 smithi023 bash[29146]: cephadm 2024-03-11T20:49:34.682286+0000 mgr.y (mgr.14152) 307 : cephadm 1 Deploying daemon osd.7 on smithi146 2024-03-11T20:49:37.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:36 smithi146 bash[23809]: cluster 2024-03-11T20:49:35.432333+0000 mgr.y (mgr.14152) 308 : cluster 0 pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-11T20:49:37.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:36 smithi023 bash[21372]: cluster 2024-03-11T20:49:35.432333+0000 mgr.y (mgr.14152) 308 : cluster 0 pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-11T20:49:37.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:36 smithi023 bash[29146]: cluster 2024-03-11T20:49:35.432333+0000 mgr.y (mgr.14152) 308 : cluster 0 pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-11T20:49:39.111 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Mar 11 20:49:39 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:49:39.111 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 11 20:49:39 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:49:39.112 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:38 smithi146 bash[23809]: cluster 2024-03-11T20:49:37.432873+0000 mgr.y (mgr.14152) 309 : cluster 0 pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-11T20:49:39.112 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:39 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:49:39.112 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 11 20:49:39 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:49:39.112 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 11 20:49:39 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:49:39.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:38 smithi023 bash[21372]: cluster 2024-03-11T20:49:37.432873+0000 mgr.y (mgr.14152) 309 : cluster 0 pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-11T20:49:39.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:38 smithi023 bash[29146]: cluster 2024-03-11T20:49:37.432873+0000 mgr.y (mgr.14152) 309 : cluster 0 pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-11T20:49:39.386 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Mar 11 20:49:39 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:49:39.386 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 11 20:49:39 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:49:39.387 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 11 20:49:39 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:49:39.387 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 11 20:49:39 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:49:39.387 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:39 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:49:40.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:40 smithi146 bash[23809]: cluster 2024-03-11T20:49:39.433391+0000 mgr.y (mgr.14152) 310 : cluster 0 pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-11T20:49:40.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:40 smithi146 bash[23809]: audit 2024-03-11T20:49:39.445013+0000 mon.a (mon.0) 612 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:49:40.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:40 smithi146 bash[23809]: audit 2024-03-11T20:49:39.457159+0000 mon.a (mon.0) 613 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:49:40.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:40 smithi146 bash[23809]: audit 2024-03-11T20:49:39.477730+0000 mon.a (mon.0) 614 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:49:40.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:40 smithi023 bash[21372]: cluster 2024-03-11T20:49:39.433391+0000 mgr.y (mgr.14152) 310 : cluster 0 pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-11T20:49:40.784 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:40 smithi023 bash[21372]: audit 2024-03-11T20:49:39.445013+0000 mon.a (mon.0) 612 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:49:40.784 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:40 smithi023 bash[21372]: audit 2024-03-11T20:49:39.457159+0000 mon.a (mon.0) 613 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:49:40.784 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:40 smithi023 bash[21372]: audit 2024-03-11T20:49:39.477730+0000 mon.a (mon.0) 614 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:49:40.784 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:40 smithi023 bash[29146]: cluster 2024-03-11T20:49:39.433391+0000 mgr.y (mgr.14152) 310 : cluster 0 pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-11T20:49:40.784 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:40 smithi023 bash[29146]: audit 2024-03-11T20:49:39.445013+0000 mon.a (mon.0) 612 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:49:40.784 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:40 smithi023 bash[29146]: audit 2024-03-11T20:49:39.457159+0000 mon.a (mon.0) 613 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:49:40.784 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:40 smithi023 bash[29146]: audit 2024-03-11T20:49:39.477730+0000 mon.a (mon.0) 614 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:49:42.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:42 smithi146 bash[23809]: cluster 2024-03-11T20:49:41.434009+0000 mgr.y (mgr.14152) 311 : cluster 0 pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-11T20:49:42.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:42 smithi023 bash[21372]: cluster 2024-03-11T20:49:41.434009+0000 mgr.y (mgr.14152) 311 : cluster 0 pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-11T20:49:42.784 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:42 smithi023 bash[29146]: cluster 2024-03-11T20:49:41.434009+0000 mgr.y (mgr.14152) 311 : cluster 0 pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-11T20:49:44.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:44 smithi146 bash[23809]: cluster 2024-03-11T20:49:43.434596+0000 mgr.y (mgr.14152) 312 : cluster 0 pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-11T20:49:44.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:44 smithi023 bash[21372]: cluster 2024-03-11T20:49:43.434596+0000 mgr.y (mgr.14152) 312 : cluster 0 pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-11T20:49:44.784 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:44 smithi023 bash[29146]: cluster 2024-03-11T20:49:43.434596+0000 mgr.y (mgr.14152) 312 : cluster 0 pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-11T20:49:46.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:45 smithi146 bash[23809]: audit 2024-03-11T20:49:44.824089+0000 mon.a (mon.0) 615 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:49:46.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:45 smithi146 bash[23809]: audit 2024-03-11T20:49:44.837269+0000 mon.a (mon.0) 616 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:49:46.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:45 smithi146 bash[23809]: audit 2024-03-11T20:49:45.516767+0000 mon.a (mon.0) 617 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:49:46.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:45 smithi146 bash[23809]: audit 2024-03-11T20:49:45.518736+0000 mon.a (mon.0) 618 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:49:46.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:45 smithi146 bash[23809]: audit 2024-03-11T20:49:45.531835+0000 mon.a (mon.0) 619 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:49:46.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:45 smithi023 bash[21372]: audit 2024-03-11T20:49:44.824089+0000 mon.a (mon.0) 615 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:49:46.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:45 smithi023 bash[21372]: audit 2024-03-11T20:49:44.837269+0000 mon.a (mon.0) 616 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:49:46.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:45 smithi023 bash[21372]: audit 2024-03-11T20:49:45.516767+0000 mon.a (mon.0) 617 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:49:46.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:45 smithi023 bash[21372]: audit 2024-03-11T20:49:45.518736+0000 mon.a (mon.0) 618 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:49:46.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:45 smithi023 bash[21372]: audit 2024-03-11T20:49:45.531835+0000 mon.a (mon.0) 619 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:49:46.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:45 smithi023 bash[29146]: audit 2024-03-11T20:49:44.824089+0000 mon.a (mon.0) 615 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:49:46.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:45 smithi023 bash[29146]: audit 2024-03-11T20:49:44.837269+0000 mon.a (mon.0) 616 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:49:46.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:45 smithi023 bash[29146]: audit 2024-03-11T20:49:45.516767+0000 mon.a (mon.0) 617 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:49:46.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:45 smithi023 bash[29146]: audit 2024-03-11T20:49:45.518736+0000 mon.a (mon.0) 618 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:49:46.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:45 smithi023 bash[29146]: audit 2024-03-11T20:49:45.531835+0000 mon.a (mon.0) 619 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:49:47.170 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:46 smithi146 bash[23809]: cluster 2024-03-11T20:49:45.435261+0000 mgr.y (mgr.14152) 313 : cluster 0 pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-11T20:49:47.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:46 smithi146 bash[23809]: audit 2024-03-11T20:49:46.032166+0000 mon.a (mon.0) 620 : audit 1 from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-11T20:49:47.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:46 smithi146 bash[23809]: audit 2024-03-11T20:49:46.034880+0000 mon.b (mon.1) 23 : audit 1 from='osd.7 [v2:172.21.15.146:6824/1678407923,v1:172.21.15.146:6825/1678407923]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-11T20:49:47.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:46 smithi023 bash[21372]: cluster 2024-03-11T20:49:45.435261+0000 mgr.y (mgr.14152) 313 : cluster 0 pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-11T20:49:47.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:46 smithi023 bash[21372]: audit 2024-03-11T20:49:46.032166+0000 mon.a (mon.0) 620 : audit 1 from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-11T20:49:47.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:46 smithi023 bash[21372]: audit 2024-03-11T20:49:46.034880+0000 mon.b (mon.1) 23 : audit 1 from='osd.7 [v2:172.21.15.146:6824/1678407923,v1:172.21.15.146:6825/1678407923]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-11T20:49:47.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:46 smithi023 bash[29146]: cluster 2024-03-11T20:49:45.435261+0000 mgr.y (mgr.14152) 313 : cluster 0 pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-11T20:49:47.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:46 smithi023 bash[29146]: audit 2024-03-11T20:49:46.032166+0000 mon.a (mon.0) 620 : audit 1 from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-11T20:49:47.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:46 smithi023 bash[29146]: audit 2024-03-11T20:49:46.034880+0000 mon.b (mon.1) 23 : audit 1 from='osd.7 [v2:172.21.15.146:6824/1678407923,v1:172.21.15.146:6825/1678407923]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-11T20:49:48.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:47 smithi146 bash[23809]: cluster 2024-03-11T20:49:46.219315+0000 osd.0 (osd.0) 11 : cluster 0 purged_snaps scrub starts 2024-03-11T20:49:48.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:47 smithi146 bash[23809]: cluster 2024-03-11T20:49:46.219407+0000 osd.0 (osd.0) 12 : cluster 0 purged_snaps scrub ok 2024-03-11T20:49:48.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:47 smithi146 bash[23809]: audit 2024-03-11T20:49:46.881033+0000 mon.a (mon.0) 621 : audit 1 from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-11T20:49:48.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:47 smithi146 bash[23809]: audit 2024-03-11T20:49:46.888565+0000 mon.b (mon.1) 24 : audit 1 from='osd.7 [v2:172.21.15.146:6824/1678407923,v1:172.21.15.146:6825/1678407923]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-03-11T20:49:48.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:47 smithi146 bash[23809]: cluster 2024-03-11T20:49:46.890891+0000 mon.a (mon.0) 622 : cluster 0 osdmap e66: 8 total, 7 up, 8 in 2024-03-11T20:49:48.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:47 smithi146 bash[23809]: audit 2024-03-11T20:49:46.891365+0000 mon.a (mon.0) 623 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T20:49:48.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:47 smithi146 bash[23809]: audit 2024-03-11T20:49:46.892246+0000 mon.a (mon.0) 624 : audit 1 from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-03-11T20:49:48.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:47 smithi023 bash[21372]: cluster 2024-03-11T20:49:46.219315+0000 osd.0 (osd.0) 11 : cluster 0 purged_snaps scrub starts 2024-03-11T20:49:48.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:47 smithi023 bash[21372]: cluster 2024-03-11T20:49:46.219407+0000 osd.0 (osd.0) 12 : cluster 0 purged_snaps scrub ok 2024-03-11T20:49:48.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:47 smithi023 bash[21372]: audit 2024-03-11T20:49:46.881033+0000 mon.a (mon.0) 621 : audit 1 from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-11T20:49:48.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:47 smithi023 bash[21372]: audit 2024-03-11T20:49:46.888565+0000 mon.b (mon.1) 24 : audit 1 from='osd.7 [v2:172.21.15.146:6824/1678407923,v1:172.21.15.146:6825/1678407923]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-03-11T20:49:48.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:47 smithi023 bash[21372]: cluster 2024-03-11T20:49:46.890891+0000 mon.a (mon.0) 622 : cluster 0 osdmap e66: 8 total, 7 up, 8 in 2024-03-11T20:49:48.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:47 smithi023 bash[21372]: audit 2024-03-11T20:49:46.891365+0000 mon.a (mon.0) 623 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T20:49:48.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:47 smithi023 bash[21372]: audit 2024-03-11T20:49:46.892246+0000 mon.a (mon.0) 624 : audit 1 from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-03-11T20:49:48.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:47 smithi023 bash[29146]: cluster 2024-03-11T20:49:46.219315+0000 osd.0 (osd.0) 11 : cluster 0 purged_snaps scrub starts 2024-03-11T20:49:48.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:47 smithi023 bash[29146]: cluster 2024-03-11T20:49:46.219407+0000 osd.0 (osd.0) 12 : cluster 0 purged_snaps scrub ok 2024-03-11T20:49:48.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:47 smithi023 bash[29146]: audit 2024-03-11T20:49:46.881033+0000 mon.a (mon.0) 621 : audit 1 from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-11T20:49:48.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:47 smithi023 bash[29146]: audit 2024-03-11T20:49:46.888565+0000 mon.b (mon.1) 24 : audit 1 from='osd.7 [v2:172.21.15.146:6824/1678407923,v1:172.21.15.146:6825/1678407923]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-03-11T20:49:48.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:47 smithi023 bash[29146]: cluster 2024-03-11T20:49:46.890891+0000 mon.a (mon.0) 622 : cluster 0 osdmap e66: 8 total, 7 up, 8 in 2024-03-11T20:49:48.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:47 smithi023 bash[29146]: audit 2024-03-11T20:49:46.891365+0000 mon.a (mon.0) 623 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T20:49:48.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:47 smithi023 bash[29146]: audit 2024-03-11T20:49:46.892246+0000 mon.a (mon.0) 624 : audit 1 from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-03-11T20:49:48.562 INFO:teuthology.orchestra.run.smithi146.stdout:Created osd(s) 7 on host 'smithi146' 2024-03-11T20:49:49.225 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:48 smithi146 bash[23809]: cluster 2024-03-11T20:49:47.435818+0000 mgr.y (mgr.14152) 314 : cluster 0 pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-11T20:49:49.226 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:48 smithi146 bash[23809]: audit 2024-03-11T20:49:47.903213+0000 mon.a (mon.0) 625 : audit 1 from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]': finished 2024-03-11T20:49:49.226 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:48 smithi146 bash[23809]: cluster 2024-03-11T20:49:47.916633+0000 mon.a (mon.0) 626 : cluster 0 osdmap e67: 8 total, 7 up, 8 in 2024-03-11T20:49:49.226 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:48 smithi146 bash[23809]: audit 2024-03-11T20:49:47.917484+0000 mon.a (mon.0) 627 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T20:49:49.226 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:48 smithi146 bash[23809]: audit 2024-03-11T20:49:47.918776+0000 mon.a (mon.0) 628 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T20:49:49.226 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:48 smithi146 bash[23809]: audit 2024-03-11T20:49:48.478529+0000 mon.a (mon.0) 629 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:49:49.226 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:48 smithi146 bash[23809]: audit 2024-03-11T20:49:48.480382+0000 mon.a (mon.0) 630 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:49:49.226 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:48 smithi146 bash[23809]: audit 2024-03-11T20:49:48.494542+0000 mon.a (mon.0) 631 : audit 1 from='osd.7 ' entity='osd.7' 2024-03-11T20:49:49.226 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:48 smithi146 bash[23809]: audit 2024-03-11T20:49:48.513414+0000 mon.a (mon.0) 632 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:49:49.227 DEBUG:teuthology.orchestra.run.smithi146:osd.7> sudo journalctl -f -n 0 -u ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@osd.7.service 2024-03-11T20:49:49.230 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-03-11T20:49:49.230 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd stat -f json 2024-03-11T20:49:49.242 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:48 smithi023 bash[29146]: cluster 2024-03-11T20:49:47.435818+0000 mgr.y (mgr.14152) 314 : cluster 0 pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-11T20:49:49.242 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:48 smithi023 bash[29146]: audit 2024-03-11T20:49:47.903213+0000 mon.a (mon.0) 625 : audit 1 from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]': finished 2024-03-11T20:49:49.242 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:48 smithi023 bash[29146]: cluster 2024-03-11T20:49:47.916633+0000 mon.a (mon.0) 626 : cluster 0 osdmap e67: 8 total, 7 up, 8 in 2024-03-11T20:49:49.242 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:48 smithi023 bash[29146]: audit 2024-03-11T20:49:47.917484+0000 mon.a (mon.0) 627 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T20:49:49.242 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:48 smithi023 bash[29146]: audit 2024-03-11T20:49:47.918776+0000 mon.a (mon.0) 628 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T20:49:49.242 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:48 smithi023 bash[29146]: audit 2024-03-11T20:49:48.478529+0000 mon.a (mon.0) 629 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:49:49.242 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:48 smithi023 bash[29146]: audit 2024-03-11T20:49:48.480382+0000 mon.a (mon.0) 630 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:49:49.242 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:48 smithi023 bash[29146]: audit 2024-03-11T20:49:48.494542+0000 mon.a (mon.0) 631 : audit 1 from='osd.7 ' entity='osd.7' 2024-03-11T20:49:49.243 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:48 smithi023 bash[29146]: audit 2024-03-11T20:49:48.513414+0000 mon.a (mon.0) 632 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:49:49.243 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:48 smithi023 bash[21372]: cluster 2024-03-11T20:49:47.435818+0000 mgr.y (mgr.14152) 314 : cluster 0 pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-11T20:49:49.243 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:48 smithi023 bash[21372]: audit 2024-03-11T20:49:47.903213+0000 mon.a (mon.0) 625 : audit 1 from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]': finished 2024-03-11T20:49:49.243 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:48 smithi023 bash[21372]: cluster 2024-03-11T20:49:47.916633+0000 mon.a (mon.0) 626 : cluster 0 osdmap e67: 8 total, 7 up, 8 in 2024-03-11T20:49:49.243 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:48 smithi023 bash[21372]: audit 2024-03-11T20:49:47.917484+0000 mon.a (mon.0) 627 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T20:49:49.243 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:48 smithi023 bash[21372]: audit 2024-03-11T20:49:47.918776+0000 mon.a (mon.0) 628 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T20:49:49.243 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:48 smithi023 bash[21372]: audit 2024-03-11T20:49:48.478529+0000 mon.a (mon.0) 629 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:49:49.243 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:48 smithi023 bash[21372]: audit 2024-03-11T20:49:48.480382+0000 mon.a (mon.0) 630 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:49:49.243 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:48 smithi023 bash[21372]: audit 2024-03-11T20:49:48.494542+0000 mon.a (mon.0) 631 : audit 1 from='osd.7 ' entity='osd.7' 2024-03-11T20:49:49.243 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:48 smithi023 bash[21372]: audit 2024-03-11T20:49:48.513414+0000 mon.a (mon.0) 632 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:49:50.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:49 smithi146 bash[23809]: cluster 2024-03-11T20:49:47.081453+0000 osd.7 (osd.7) 1 : cluster 0 purged_snaps scrub starts 2024-03-11T20:49:50.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:49 smithi146 bash[23809]: cluster 2024-03-11T20:49:47.081572+0000 osd.7 (osd.7) 2 : cluster 0 purged_snaps scrub ok 2024-03-11T20:49:50.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:49 smithi146 bash[23809]: cluster 2024-03-11T20:49:48.922074+0000 mon.a (mon.0) 633 : cluster 1 osd.7 [v2:172.21.15.146:6824/1678407923,v1:172.21.15.146:6825/1678407923] boot 2024-03-11T20:49:50.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:49 smithi146 bash[23809]: cluster 2024-03-11T20:49:48.922226+0000 mon.a (mon.0) 634 : cluster 0 osdmap e68: 8 total, 8 up, 8 in 2024-03-11T20:49:50.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:49 smithi146 bash[23809]: audit 2024-03-11T20:49:48.925561+0000 mon.a (mon.0) 635 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T20:49:50.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:49 smithi146 bash[23809]: cluster 2024-03-11T20:49:49.370385+0000 osd.4 (osd.4) 5 : cluster 0 purged_snaps scrub starts 2024-03-11T20:49:50.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:49 smithi146 bash[23809]: cluster 2024-03-11T20:49:49.370486+0000 osd.4 (osd.4) 6 : cluster 0 purged_snaps scrub ok 2024-03-11T20:49:50.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:49 smithi146 bash[23809]: cluster 2024-03-11T20:49:49.559008+0000 mon.a (mon.0) 636 : cluster 0 osdmap e69: 8 total, 8 up, 8 in 2024-03-11T20:49:50.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:49 smithi023 bash[21372]: cluster 2024-03-11T20:49:47.081453+0000 osd.7 (osd.7) 1 : cluster 0 purged_snaps scrub starts 2024-03-11T20:49:50.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:49 smithi023 bash[21372]: cluster 2024-03-11T20:49:47.081572+0000 osd.7 (osd.7) 2 : cluster 0 purged_snaps scrub ok 2024-03-11T20:49:50.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:49 smithi023 bash[21372]: cluster 2024-03-11T20:49:48.922074+0000 mon.a (mon.0) 633 : cluster 1 osd.7 [v2:172.21.15.146:6824/1678407923,v1:172.21.15.146:6825/1678407923] boot 2024-03-11T20:49:50.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:49 smithi023 bash[21372]: cluster 2024-03-11T20:49:48.922226+0000 mon.a (mon.0) 634 : cluster 0 osdmap e68: 8 total, 8 up, 8 in 2024-03-11T20:49:50.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:49 smithi023 bash[21372]: audit 2024-03-11T20:49:48.925561+0000 mon.a (mon.0) 635 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T20:49:50.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:49 smithi023 bash[21372]: cluster 2024-03-11T20:49:49.370385+0000 osd.4 (osd.4) 5 : cluster 0 purged_snaps scrub starts 2024-03-11T20:49:50.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:49 smithi023 bash[21372]: cluster 2024-03-11T20:49:49.370486+0000 osd.4 (osd.4) 6 : cluster 0 purged_snaps scrub ok 2024-03-11T20:49:50.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:49 smithi023 bash[21372]: cluster 2024-03-11T20:49:49.559008+0000 mon.a (mon.0) 636 : cluster 0 osdmap e69: 8 total, 8 up, 8 in 2024-03-11T20:49:50.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:49 smithi023 bash[29146]: cluster 2024-03-11T20:49:47.081453+0000 osd.7 (osd.7) 1 : cluster 0 purged_snaps scrub starts 2024-03-11T20:49:50.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:49 smithi023 bash[29146]: cluster 2024-03-11T20:49:47.081572+0000 osd.7 (osd.7) 2 : cluster 0 purged_snaps scrub ok 2024-03-11T20:49:50.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:49 smithi023 bash[29146]: cluster 2024-03-11T20:49:48.922074+0000 mon.a (mon.0) 633 : cluster 1 osd.7 [v2:172.21.15.146:6824/1678407923,v1:172.21.15.146:6825/1678407923] boot 2024-03-11T20:49:50.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:49 smithi023 bash[29146]: cluster 2024-03-11T20:49:48.922226+0000 mon.a (mon.0) 634 : cluster 0 osdmap e68: 8 total, 8 up, 8 in 2024-03-11T20:49:50.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:49 smithi023 bash[29146]: audit 2024-03-11T20:49:48.925561+0000 mon.a (mon.0) 635 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T20:49:50.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:49 smithi023 bash[29146]: cluster 2024-03-11T20:49:49.370385+0000 osd.4 (osd.4) 5 : cluster 0 purged_snaps scrub starts 2024-03-11T20:49:50.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:49 smithi023 bash[29146]: cluster 2024-03-11T20:49:49.370486+0000 osd.4 (osd.4) 6 : cluster 0 purged_snaps scrub ok 2024-03-11T20:49:50.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:49 smithi023 bash[29146]: cluster 2024-03-11T20:49:49.559008+0000 mon.a (mon.0) 636 : cluster 0 osdmap e69: 8 total, 8 up, 8 in 2024-03-11T20:49:51.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:50 smithi146 bash[23809]: cluster 2024-03-11T20:49:49.436236+0000 mgr.y (mgr.14152) 315 : cluster 0 pgmap v312: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-11T20:49:51.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:50 smithi146 bash[23809]: cluster 2024-03-11T20:49:50.568950+0000 mon.a (mon.0) 637 : cluster 0 osdmap e70: 8 total, 8 up, 8 in 2024-03-11T20:49:51.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:50 smithi023 bash[21372]: cluster 2024-03-11T20:49:49.436236+0000 mgr.y (mgr.14152) 315 : cluster 0 pgmap v312: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-11T20:49:51.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:50 smithi023 bash[21372]: cluster 2024-03-11T20:49:50.568950+0000 mon.a (mon.0) 637 : cluster 0 osdmap e70: 8 total, 8 up, 8 in 2024-03-11T20:49:51.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:50 smithi023 bash[29146]: cluster 2024-03-11T20:49:49.436236+0000 mgr.y (mgr.14152) 315 : cluster 0 pgmap v312: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-11T20:49:51.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:50 smithi023 bash[29146]: cluster 2024-03-11T20:49:50.568950+0000 mon.a (mon.0) 637 : cluster 0 osdmap e70: 8 total, 8 up, 8 in 2024-03-11T20:49:53.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:52 smithi146 bash[23809]: cluster 2024-03-11T20:49:51.436779+0000 mgr.y (mgr.14152) 316 : cluster 0 pgmap v315: 1 pgs: 1 remapped+peering; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:49:53.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:52 smithi146 bash[23809]: cluster 2024-03-11T20:49:51.574446+0000 mon.a (mon.0) 638 : cluster 0 osdmap e71: 8 total, 8 up, 8 in 2024-03-11T20:49:53.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:52 smithi023 bash[21372]: cluster 2024-03-11T20:49:51.436779+0000 mgr.y (mgr.14152) 316 : cluster 0 pgmap v315: 1 pgs: 1 remapped+peering; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:49:53.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:52 smithi023 bash[21372]: cluster 2024-03-11T20:49:51.574446+0000 mon.a (mon.0) 638 : cluster 0 osdmap e71: 8 total, 8 up, 8 in 2024-03-11T20:49:53.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:52 smithi023 bash[29146]: cluster 2024-03-11T20:49:51.436779+0000 mgr.y (mgr.14152) 316 : cluster 0 pgmap v315: 1 pgs: 1 remapped+peering; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:49:53.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:52 smithi023 bash[29146]: cluster 2024-03-11T20:49:51.574446+0000 mon.a (mon.0) 638 : cluster 0 osdmap e71: 8 total, 8 up, 8 in 2024-03-11T20:49:54.002 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:49:54.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:53 smithi146 bash[23809]: cluster 2024-03-11T20:49:52.577039+0000 mon.a (mon.0) 639 : cluster 0 osdmap e72: 8 total, 8 up, 8 in 2024-03-11T20:49:54.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:53 smithi023 bash[21372]: cluster 2024-03-11T20:49:52.577039+0000 mon.a (mon.0) 639 : cluster 0 osdmap e72: 8 total, 8 up, 8 in 2024-03-11T20:49:54.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:53 smithi023 bash[29146]: cluster 2024-03-11T20:49:52.577039+0000 mon.a (mon.0) 639 : cluster 0 osdmap e72: 8 total, 8 up, 8 in 2024-03-11T20:49:54.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:54 smithi146 bash[23809]: cluster 2024-03-11T20:49:52.895776+0000 osd.6 (osd.6) 3 : cluster 0 purged_snaps scrub starts 2024-03-11T20:49:54.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:54 smithi146 bash[23809]: cluster 2024-03-11T20:49:52.895869+0000 osd.6 (osd.6) 4 : cluster 0 purged_snaps scrub ok 2024-03-11T20:49:54.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:54 smithi146 bash[23809]: cluster 2024-03-11T20:49:53.437370+0000 mgr.y (mgr.14152) 317 : cluster 0 pgmap v318: 1 pgs: 1 remapped+peering; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:49:54.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:54 smithi146 bash[23809]: cluster 2024-03-11T20:49:53.597062+0000 mon.a (mon.0) 640 : cluster 0 osdmap e73: 8 total, 8 up, 8 in 2024-03-11T20:49:54.860 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:54 smithi023 bash[21372]: cluster 2024-03-11T20:49:52.895776+0000 osd.6 (osd.6) 3 : cluster 0 purged_snaps scrub starts 2024-03-11T20:49:54.860 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:54 smithi023 bash[21372]: cluster 2024-03-11T20:49:52.895869+0000 osd.6 (osd.6) 4 : cluster 0 purged_snaps scrub ok 2024-03-11T20:49:54.860 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:54 smithi023 bash[21372]: cluster 2024-03-11T20:49:53.437370+0000 mgr.y (mgr.14152) 317 : cluster 0 pgmap v318: 1 pgs: 1 remapped+peering; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:49:54.860 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:54 smithi023 bash[21372]: cluster 2024-03-11T20:49:53.597062+0000 mon.a (mon.0) 640 : cluster 0 osdmap e73: 8 total, 8 up, 8 in 2024-03-11T20:49:54.860 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:54 smithi023 bash[29146]: cluster 2024-03-11T20:49:52.895776+0000 osd.6 (osd.6) 3 : cluster 0 purged_snaps scrub starts 2024-03-11T20:49:54.860 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:54 smithi023 bash[29146]: cluster 2024-03-11T20:49:52.895869+0000 osd.6 (osd.6) 4 : cluster 0 purged_snaps scrub ok 2024-03-11T20:49:54.860 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:54 smithi023 bash[29146]: cluster 2024-03-11T20:49:53.437370+0000 mgr.y (mgr.14152) 317 : cluster 0 pgmap v318: 1 pgs: 1 remapped+peering; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:49:54.860 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:54 smithi023 bash[29146]: cluster 2024-03-11T20:49:53.597062+0000 mon.a (mon.0) 640 : cluster 0 osdmap e73: 8 total, 8 up, 8 in 2024-03-11T20:49:56.950 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-11T20:49:56.964 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:56 smithi023 bash[21372]: cluster 2024-03-11T20:49:55.437967+0000 mgr.y (mgr.14152) 318 : cluster 0 pgmap v320: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 118 KiB/s, 0 objects/s recovering 2024-03-11T20:49:56.964 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:56 smithi023 bash[29146]: cluster 2024-03-11T20:49:55.437967+0000 mgr.y (mgr.14152) 318 : cluster 0 pgmap v320: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 118 KiB/s, 0 objects/s recovering 2024-03-11T20:49:57.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:56 smithi146 bash[23809]: cluster 2024-03-11T20:49:55.437967+0000 mgr.y (mgr.14152) 318 : cluster 0 pgmap v320: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 118 KiB/s, 0 objects/s recovering 2024-03-11T20:49:57.584 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":73,"num_osds":8,"num_up_osds":8,"osd_up_since":1710190188,"num_in_osds":8,"osd_in_since":1710190161,"num_remapped_pgs":0} 2024-03-11T20:49:57.585 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd dump --format=json 2024-03-11T20:49:58.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:57 smithi146 bash[23809]: audit 2024-03-11T20:49:56.954703+0000 mon.c (mon.2) 14 : audit 0 from='client.? 172.21.15.23:0/1763419517' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-11T20:49:58.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:57 smithi023 bash[21372]: audit 2024-03-11T20:49:56.954703+0000 mon.c (mon.2) 14 : audit 0 from='client.? 172.21.15.23:0/1763419517' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-11T20:49:58.033 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:57 smithi023 bash[29146]: audit 2024-03-11T20:49:56.954703+0000 mon.c (mon.2) 14 : audit 0 from='client.? 172.21.15.23:0/1763419517' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-11T20:49:59.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:58 smithi146 bash[23809]: cluster 2024-03-11T20:49:57.438550+0000 mgr.y (mgr.14152) 319 : cluster 0 pgmap v321: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-11T20:49:59.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:58 smithi146 bash[23809]: audit 2024-03-11T20:49:58.134710+0000 mon.a (mon.0) 641 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:49:59.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:58 smithi146 bash[23809]: audit 2024-03-11T20:49:58.147522+0000 mon.a (mon.0) 642 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:49:59.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:58 smithi146 bash[23809]: audit 2024-03-11T20:49:58.150169+0000 mon.a (mon.0) 643 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:49:59.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:58 smithi146 bash[23809]: audit 2024-03-11T20:49:58.152153+0000 mon.a (mon.0) 644 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:49:59.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:58 smithi146 bash[23809]: audit 2024-03-11T20:49:58.153869+0000 mon.a (mon.0) 645 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:49:59.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:58 smithi146 bash[23809]: audit 2024-03-11T20:49:58.155603+0000 mon.a (mon.0) 646 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:49:59.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:58 smithi146 bash[23809]: audit 2024-03-11T20:49:58.166303+0000 mon.a (mon.0) 647 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:49:59.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:58 smithi146 bash[23809]: audit 2024-03-11T20:49:58.170766+0000 mon.a (mon.0) 648 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:49:59.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:58 smithi146 bash[23809]: audit 2024-03-11T20:49:58.172742+0000 mon.a (mon.0) 649 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:49:59.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:58 smithi146 bash[23809]: audit 2024-03-11T20:49:58.182970+0000 mon.a (mon.0) 650 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:49:59.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:58 smithi023 bash[29146]: cluster 2024-03-11T20:49:57.438550+0000 mgr.y (mgr.14152) 319 : cluster 0 pgmap v321: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-11T20:49:59.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:58 smithi023 bash[29146]: audit 2024-03-11T20:49:58.134710+0000 mon.a (mon.0) 641 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:49:59.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:58 smithi023 bash[29146]: audit 2024-03-11T20:49:58.147522+0000 mon.a (mon.0) 642 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:49:59.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:58 smithi023 bash[29146]: audit 2024-03-11T20:49:58.150169+0000 mon.a (mon.0) 643 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:49:59.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:58 smithi023 bash[29146]: audit 2024-03-11T20:49:58.152153+0000 mon.a (mon.0) 644 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:49:59.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:58 smithi023 bash[29146]: audit 2024-03-11T20:49:58.153869+0000 mon.a (mon.0) 645 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:49:59.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:58 smithi023 bash[29146]: audit 2024-03-11T20:49:58.155603+0000 mon.a (mon.0) 646 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:49:59.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:58 smithi023 bash[29146]: audit 2024-03-11T20:49:58.166303+0000 mon.a (mon.0) 647 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:49:59.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:58 smithi023 bash[29146]: audit 2024-03-11T20:49:58.170766+0000 mon.a (mon.0) 648 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:49:59.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:58 smithi023 bash[29146]: audit 2024-03-11T20:49:58.172742+0000 mon.a (mon.0) 649 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:49:59.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:58 smithi023 bash[29146]: audit 2024-03-11T20:49:58.182970+0000 mon.a (mon.0) 650 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:49:59.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:58 smithi023 bash[21372]: cluster 2024-03-11T20:49:57.438550+0000 mgr.y (mgr.14152) 319 : cluster 0 pgmap v321: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-11T20:49:59.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:58 smithi023 bash[21372]: audit 2024-03-11T20:49:58.134710+0000 mon.a (mon.0) 641 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:49:59.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:58 smithi023 bash[21372]: audit 2024-03-11T20:49:58.147522+0000 mon.a (mon.0) 642 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:49:59.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:58 smithi023 bash[21372]: audit 2024-03-11T20:49:58.150169+0000 mon.a (mon.0) 643 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:49:59.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:58 smithi023 bash[21372]: audit 2024-03-11T20:49:58.152153+0000 mon.a (mon.0) 644 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:49:59.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:58 smithi023 bash[21372]: audit 2024-03-11T20:49:58.153869+0000 mon.a (mon.0) 645 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:49:59.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:58 smithi023 bash[21372]: audit 2024-03-11T20:49:58.155603+0000 mon.a (mon.0) 646 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:49:59.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:58 smithi023 bash[21372]: audit 2024-03-11T20:49:58.166303+0000 mon.a (mon.0) 647 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:49:59.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:58 smithi023 bash[21372]: audit 2024-03-11T20:49:58.170766+0000 mon.a (mon.0) 648 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:49:59.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:58 smithi023 bash[21372]: audit 2024-03-11T20:49:58.172742+0000 mon.a (mon.0) 649 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:49:59.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:58 smithi023 bash[21372]: audit 2024-03-11T20:49:58.182970+0000 mon.a (mon.0) 650 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:50:00.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:59 smithi146 bash[23809]: cluster 2024-03-11T20:49:58.061723+0000 osd.7 (osd.7) 3 : cluster 0 1.0 scrub starts 2024-03-11T20:50:00.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:59 smithi146 bash[23809]: cluster 2024-03-11T20:49:58.064789+0000 osd.7 (osd.7) 4 : cluster 0 1.0 scrub ok 2024-03-11T20:50:00.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:59 smithi146 bash[23809]: cephadm 2024-03-11T20:49:58.122648+0000 mgr.y (mgr.14152) 320 : cephadm 1 Detected new or changed devices on smithi146 2024-03-11T20:50:00.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:49:59 smithi146 bash[23809]: cephadm 2024-03-11T20:49:58.156985+0000 mgr.y (mgr.14152) 321 : cephadm 1 Adjusting osd_memory_target on smithi146 to 4316M 2024-03-11T20:50:00.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:59 smithi023 bash[21372]: cluster 2024-03-11T20:49:58.061723+0000 osd.7 (osd.7) 3 : cluster 0 1.0 scrub starts 2024-03-11T20:50:00.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:59 smithi023 bash[21372]: cluster 2024-03-11T20:49:58.064789+0000 osd.7 (osd.7) 4 : cluster 0 1.0 scrub ok 2024-03-11T20:50:00.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:59 smithi023 bash[21372]: cephadm 2024-03-11T20:49:58.122648+0000 mgr.y (mgr.14152) 320 : cephadm 1 Detected new or changed devices on smithi146 2024-03-11T20:50:00.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:49:59 smithi023 bash[21372]: cephadm 2024-03-11T20:49:58.156985+0000 mgr.y (mgr.14152) 321 : cephadm 1 Adjusting osd_memory_target on smithi146 to 4316M 2024-03-11T20:50:00.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:59 smithi023 bash[29146]: cluster 2024-03-11T20:49:58.061723+0000 osd.7 (osd.7) 3 : cluster 0 1.0 scrub starts 2024-03-11T20:50:00.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:59 smithi023 bash[29146]: cluster 2024-03-11T20:49:58.064789+0000 osd.7 (osd.7) 4 : cluster 0 1.0 scrub ok 2024-03-11T20:50:00.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:59 smithi023 bash[29146]: cephadm 2024-03-11T20:49:58.122648+0000 mgr.y (mgr.14152) 320 : cephadm 1 Detected new or changed devices on smithi146 2024-03-11T20:50:00.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:49:59 smithi023 bash[29146]: cephadm 2024-03-11T20:49:58.156985+0000 mgr.y (mgr.14152) 321 : cephadm 1 Adjusting osd_memory_target on smithi146 to 4316M 2024-03-11T20:50:01.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:00 smithi146 bash[23809]: cluster 2024-03-11T20:49:59.439132+0000 mgr.y (mgr.14152) 322 : cluster 0 pgmap v322: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-03-11T20:50:01.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:00 smithi146 bash[23809]: cluster 2024-03-11T20:50:00.000133+0000 mon.a (mon.0) 651 : cluster 1 overall HEALTH_OK 2024-03-11T20:50:01.033 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:00 smithi023 bash[29146]: cluster 2024-03-11T20:49:59.439132+0000 mgr.y (mgr.14152) 322 : cluster 0 pgmap v322: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-03-11T20:50:01.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:00 smithi023 bash[29146]: cluster 2024-03-11T20:50:00.000133+0000 mon.a (mon.0) 651 : cluster 1 overall HEALTH_OK 2024-03-11T20:50:01.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:00 smithi023 bash[21372]: cluster 2024-03-11T20:49:59.439132+0000 mgr.y (mgr.14152) 322 : cluster 0 pgmap v322: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-03-11T20:50:01.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:00 smithi023 bash[21372]: cluster 2024-03-11T20:50:00.000133+0000 mon.a (mon.0) 651 : cluster 1 overall HEALTH_OK 2024-03-11T20:50:02.395 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:50:02.939 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:02 smithi023 bash[21372]: cluster 2024-03-11T20:50:01.439819+0000 mgr.y (mgr.14152) 323 : cluster 0 pgmap v323: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-03-11T20:50:03.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:02 smithi146 bash[23809]: cluster 2024-03-11T20:50:01.439819+0000 mgr.y (mgr.14152) 323 : cluster 0 pgmap v323: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-03-11T20:50:03.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:02 smithi023 bash[29146]: cluster 2024-03-11T20:50:01.439819+0000 mgr.y (mgr.14152) 323 : cluster 0 pgmap v323: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-03-11T20:50:04.944 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:04 smithi023 bash[21372]: cluster 2024-03-11T20:50:03.440398+0000 mgr.y (mgr.14152) 324 : cluster 0 pgmap v324: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-11T20:50:04.945 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:04 smithi023 bash[29146]: cluster 2024-03-11T20:50:03.440398+0000 mgr.y (mgr.14152) 324 : cluster 0 pgmap v324: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-11T20:50:05.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:04 smithi146 bash[23809]: cluster 2024-03-11T20:50:03.440398+0000 mgr.y (mgr.14152) 324 : cluster 0 pgmap v324: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-11T20:50:05.340 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-11T20:50:05.340 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":73,"fsid":"67ba00f8-dfe7-11ee-95c8-87774f69a715","created":"2024-03-11T20:40:13.664802+0000","modified":"2024-03-11T20:49:53.580062+0000","last_up_change":"2024-03-11T20:49:48.903029+0000","last_in_change":"2024-03-11T20:49:21.979501+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T20:45:31.964570+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3df72b4a-8fe4-4265-b4c6-af022e1e23e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":2154627393},{"type":"v1","addr":"172.21.15.23:6803","nonce":2154627393}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":2154627393},{"type":"v1","addr":"172.21.15.23:6805","nonce":2154627393}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":2154627393},{"type":"v1","addr":"172.21.15.23:6809","nonce":2154627393}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":2154627393},{"type":"v1","addr":"172.21.15.23:6807","nonce":2154627393}]},"public_addr":"172.21.15.23:6803/2154627393","cluster_addr":"172.21.15.23:6805/2154627393","heartbeat_back_addr":"172.21.15.23:6809/2154627393","heartbeat_front_addr":"172.21.15.23:6807/2154627393","state":["exists","up"]},{"osd":1,"uuid":"e8fd7500-9081-41a1-8077-a4d0f0e12a2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":2506707705},{"type":"v1","addr":"172.21.15.23:6811","nonce":2506707705}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":2506707705},{"type":"v1","addr":"172.21.15.23:6813","nonce":2506707705}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":2506707705},{"type":"v1","addr":"172.21.15.23:6817","nonce":2506707705}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":2506707705},{"type":"v1","addr":"172.21.15.23:6815","nonce":2506707705}]},"public_addr":"172.21.15.23:6811/2506707705","cluster_addr":"172.21.15.23:6813/2506707705","heartbeat_back_addr":"172.21.15.23:6817/2506707705","heartbeat_front_addr":"172.21.15.23:6815/2506707705","state":["exists","up"]},{"osd":2,"uuid":"dd85d52a-df48-4b03-8612-cecfd82b5987","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":1770748056},{"type":"v1","addr":"172.21.15.23:6819","nonce":1770748056}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":1770748056},{"type":"v1","addr":"172.21.15.23:6821","nonce":1770748056}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":1770748056},{"type":"v1","addr":"172.21.15.23:6825","nonce":1770748056}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":1770748056},{"type":"v1","addr":"172.21.15.23:6823","nonce":1770748056}]},"public_addr":"172.21.15.23:6819/1770748056","cluster_addr":"172.21.15.23:6821/1770748056","heartbeat_back_addr":"172.21.15.23:6825/1770748056","heartbeat_front_addr":"172.21.15.23:6823/1770748056","state":["exists","up"]},{"osd":3,"uuid":"e3021106-3779-4b94-9bbf-a7724544110f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":28,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":938496814},{"type":"v1","addr":"172.21.15.23:6827","nonce":938496814}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":938496814},{"type":"v1","addr":"172.21.15.23:6829","nonce":938496814}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":938496814},{"type":"v1","addr":"172.21.15.23:6833","nonce":938496814}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":938496814},{"type":"v1","addr":"172.21.15.23:6831","nonce":938496814}]},"public_addr":"172.21.15.23:6827/938496814","cluster_addr":"172.21.15.23:6829/938496814","heartbeat_back_addr":"172.21.15.23:6833/938496814","heartbeat_front_addr":"172.21.15.23:6831/938496814","state":["exists","up"]},{"osd":4,"uuid":"4dcbc91e-7035-413d-97bc-c9d804eb16d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":615746847},{"type":"v1","addr":"172.21.15.146:6801","nonce":615746847}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":615746847},{"type":"v1","addr":"172.21.15.146:6803","nonce":615746847}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":615746847},{"type":"v1","addr":"172.21.15.146:6807","nonce":615746847}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":615746847},{"type":"v1","addr":"172.21.15.146:6805","nonce":615746847}]},"public_addr":"172.21.15.146:6801/615746847","cluster_addr":"172.21.15.146:6803/615746847","heartbeat_back_addr":"172.21.15.146:6807/615746847","heartbeat_front_addr":"172.21.15.146:6805/615746847","state":["exists","up"]},{"osd":5,"uuid":"aefcb766-8bae-4ff4-b71e-6691c8ac02f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":47,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":4003230814},{"type":"v1","addr":"172.21.15.146:6809","nonce":4003230814}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":4003230814},{"type":"v1","addr":"172.21.15.146:6811","nonce":4003230814}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":4003230814},{"type":"v1","addr":"172.21.15.146:6815","nonce":4003230814}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":4003230814},{"type":"v1","addr":"172.21.15.146:6813","nonce":4003230814}]},"public_addr":"172.21.15.146:6809/4003230814","cluster_addr":"172.21.15.146:6811/4003230814","heartbeat_back_addr":"172.21.15.146:6815/4003230814","heartbeat_front_addr":"172.21.15.146:6813/4003230814","state":["exists","up"]},{"osd":6,"uuid":"5cb151e8-1f9b-4617-8a86-adfb06bb9606","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":3334092376},{"type":"v1","addr":"172.21.15.146:6817","nonce":3334092376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3334092376},{"type":"v1","addr":"172.21.15.146:6819","nonce":3334092376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":3334092376},{"type":"v1","addr":"172.21.15.146:6823","nonce":3334092376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3334092376},{"type":"v1","addr":"172.21.15.146:6821","nonce":3334092376}]},"public_addr":"172.21.15.146:6817/3334092376","cluster_addr":"172.21.15.146:6819/3334092376","heartbeat_back_addr":"172.21.15.146:6823/3334092376","heartbeat_front_addr":"172.21.15.146:6821/3334092376","state":["exists","up"]},{"osd":7,"uuid":"bdbd11a2-fa09-4f82-93a2-6b2d9c750bfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":1678407923},{"type":"v1","addr":"172.21.15.146:6825","nonce":1678407923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":1678407923},{"type":"v1","addr":"172.21.15.146:6827","nonce":1678407923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":1678407923},{"type":"v1","addr":"172.21.15.146:6831","nonce":1678407923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":1678407923},{"type":"v1","addr":"172.21.15.146:6829","nonce":1678407923}]},"public_addr":"172.21.15.146:6825/1678407923","cluster_addr":"172.21.15.146:6827/1678407923","heartbeat_back_addr":"172.21.15.146:6831/1678407923","heartbeat_front_addr":"172.21.15.146:6829/1678407923","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T20:49:46.219412+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T20:49:29.450545+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T20:49:26.779201+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T20:48:48.511721+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T20:49:49.370492+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T20:49:04.463089+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T20:49:52.895874+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T20:49:47.081578+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:0/2356701899":"2024-03-12T20:41:21.301516+0000","172.21.15.23:0/3833230375":"2024-03-12T20:41:21.301516+0000","172.21.15.23:6801/1057699850":"2024-03-12T20:41:21.301516+0000","172.21.15.23:6800/2813578081":"2024-03-12T20:40:45.361537+0000","172.21.15.23:6800/1057699850":"2024-03-12T20:41:21.301516+0000","172.21.15.23:0/827653812":"2024-03-12T20:41:21.301516+0000","172.21.15.23:0/2004146740":"2024-03-12T20:40:45.361537+0000","172.21.15.23:0/3708753659":"2024-03-12T20:40:45.361537+0000","172.21.15.23:6801/2813578081":"2024-03-12T20:40:45.361537+0000","172.21.15.23:0/1618334186":"2024-03-12T20:40:45.361537+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T20:50:06.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:05 smithi146 bash[23809]: audit 2024-03-11T20:50:05.343883+0000 mon.a (mon.0) 652 : audit 0 from='client.? 172.21.15.23:0/3887399209' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T20:50:06.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:05 smithi023 bash[21372]: audit 2024-03-11T20:50:05.343883+0000 mon.a (mon.0) 652 : audit 0 from='client.? 172.21.15.23:0/3887399209' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T20:50:06.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:05 smithi023 bash[29146]: audit 2024-03-11T20:50:05.343883+0000 mon.a (mon.0) 652 : audit 0 from='client.? 172.21.15.23:0/3887399209' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T20:50:06.038 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-11T20:45:31.964570+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-11T20:50:06.038 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-11T20:50:07.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:06 smithi146 bash[23809]: cluster 2024-03-11T20:50:05.440958+0000 mgr.y (mgr.14152) 325 : cluster 0 pgmap v325: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s, 0 objects/s recovering 2024-03-11T20:50:07.033 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:06 smithi023 bash[29146]: cluster 2024-03-11T20:50:05.440958+0000 mgr.y (mgr.14152) 325 : cluster 0 pgmap v325: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s, 0 objects/s recovering 2024-03-11T20:50:07.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:06 smithi023 bash[21372]: cluster 2024-03-11T20:50:05.440958+0000 mgr.y (mgr.14152) 325 : cluster 0 pgmap v325: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s, 0 objects/s recovering 2024-03-11T20:50:09.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:08 smithi146 bash[23809]: cluster 2024-03-11T20:50:07.441469+0000 mgr.y (mgr.14152) 326 : cluster 0 pgmap v326: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:50:09.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:08 smithi146 bash[23809]: cluster 2024-03-11T20:50:07.654937+0000 osd.3 (osd.3) 7 : cluster 0 purged_snaps scrub starts 2024-03-11T20:50:09.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:08 smithi146 bash[23809]: cluster 2024-03-11T20:50:07.655039+0000 osd.3 (osd.3) 8 : cluster 0 purged_snaps scrub ok 2024-03-11T20:50:09.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:08 smithi146 bash[23809]: cluster 2024-03-11T20:50:07.695027+0000 mon.a (mon.0) 653 : cluster 0 osdmap e74: 8 total, 8 up, 8 in 2024-03-11T20:50:09.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:08 smithi023 bash[21372]: cluster 2024-03-11T20:50:07.441469+0000 mgr.y (mgr.14152) 326 : cluster 0 pgmap v326: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:50:09.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:08 smithi023 bash[21372]: cluster 2024-03-11T20:50:07.654937+0000 osd.3 (osd.3) 7 : cluster 0 purged_snaps scrub starts 2024-03-11T20:50:09.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:08 smithi023 bash[21372]: cluster 2024-03-11T20:50:07.655039+0000 osd.3 (osd.3) 8 : cluster 0 purged_snaps scrub ok 2024-03-11T20:50:09.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:08 smithi023 bash[21372]: cluster 2024-03-11T20:50:07.695027+0000 mon.a (mon.0) 653 : cluster 0 osdmap e74: 8 total, 8 up, 8 in 2024-03-11T20:50:09.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:08 smithi023 bash[29146]: cluster 2024-03-11T20:50:07.441469+0000 mgr.y (mgr.14152) 326 : cluster 0 pgmap v326: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:50:09.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:08 smithi023 bash[29146]: cluster 2024-03-11T20:50:07.654937+0000 osd.3 (osd.3) 7 : cluster 0 purged_snaps scrub starts 2024-03-11T20:50:09.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:08 smithi023 bash[29146]: cluster 2024-03-11T20:50:07.655039+0000 osd.3 (osd.3) 8 : cluster 0 purged_snaps scrub ok 2024-03-11T20:50:09.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:08 smithi023 bash[29146]: cluster 2024-03-11T20:50:07.695027+0000 mon.a (mon.0) 653 : cluster 0 osdmap e74: 8 total, 8 up, 8 in 2024-03-11T20:50:10.805 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:50:11.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:10 smithi146 bash[23809]: cluster 2024-03-11T20:50:09.442063+0000 mgr.y (mgr.14152) 327 : cluster 0 pgmap v328: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:50:11.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:10 smithi146 bash[23809]: cluster 2024-03-11T20:50:09.855450+0000 osd.5 (osd.5) 7 : cluster 0 purged_snaps scrub starts 2024-03-11T20:50:11.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:10 smithi146 bash[23809]: cluster 2024-03-11T20:50:09.855547+0000 osd.5 (osd.5) 8 : cluster 0 purged_snaps scrub ok 2024-03-11T20:50:11.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:10 smithi023 bash[21372]: cluster 2024-03-11T20:50:09.442063+0000 mgr.y (mgr.14152) 327 : cluster 0 pgmap v328: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:50:11.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:10 smithi023 bash[21372]: cluster 2024-03-11T20:50:09.855450+0000 osd.5 (osd.5) 7 : cluster 0 purged_snaps scrub starts 2024-03-11T20:50:11.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:10 smithi023 bash[21372]: cluster 2024-03-11T20:50:09.855547+0000 osd.5 (osd.5) 8 : cluster 0 purged_snaps scrub ok 2024-03-11T20:50:11.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:10 smithi023 bash[29146]: cluster 2024-03-11T20:50:09.442063+0000 mgr.y (mgr.14152) 327 : cluster 0 pgmap v328: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:50:11.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:10 smithi023 bash[29146]: cluster 2024-03-11T20:50:09.855450+0000 osd.5 (osd.5) 7 : cluster 0 purged_snaps scrub starts 2024-03-11T20:50:11.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:10 smithi023 bash[29146]: cluster 2024-03-11T20:50:09.855547+0000 osd.5 (osd.5) 8 : cluster 0 purged_snaps scrub ok 2024-03-11T20:50:11.966 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:11 smithi023 bash[21372]: cluster 2024-03-11T20:50:10.724137+0000 mon.a (mon.0) 654 : cluster 0 osdmap e75: 8 total, 8 up, 8 in 2024-03-11T20:50:11.967 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:11 smithi023 bash[29146]: cluster 2024-03-11T20:50:10.724137+0000 mon.a (mon.0) 654 : cluster 0 osdmap e75: 8 total, 8 up, 8 in 2024-03-11T20:50:12.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:11 smithi146 bash[23809]: cluster 2024-03-11T20:50:10.724137+0000 mon.a (mon.0) 654 : cluster 0 osdmap e75: 8 total, 8 up, 8 in 2024-03-11T20:50:13.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:12 smithi146 bash[23809]: cluster 2024-03-11T20:50:11.442631+0000 mgr.y (mgr.14152) 328 : cluster 0 pgmap v330: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:50:13.033 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:12 smithi023 bash[29146]: cluster 2024-03-11T20:50:11.442631+0000 mgr.y (mgr.14152) 328 : cluster 0 pgmap v330: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:50:13.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:12 smithi023 bash[21372]: cluster 2024-03-11T20:50:11.442631+0000 mgr.y (mgr.14152) 328 : cluster 0 pgmap v330: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:50:13.534 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 1 2024-03-11T20:50:14.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:13 smithi146 bash[23809]: audit 2024-03-11T20:50:13.538567+0000 mon.a (mon.0) 655 : audit 0 from='client.? 172.21.15.23:0/1560925510' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T20:50:14.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:13 smithi023 bash[21372]: audit 2024-03-11T20:50:13.538567+0000 mon.a (mon.0) 655 : audit 0 from='client.? 172.21.15.23:0/1560925510' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T20:50:14.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:13 smithi023 bash[29146]: audit 2024-03-11T20:50:13.538567+0000 mon.a (mon.0) 655 : audit 0 from='client.? 172.21.15.23:0/1560925510' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T20:50:14.158 INFO:tasks.cephadm:Adding ceph.rgw.foo.a on smithi023 2024-03-11T20:50:14.158 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph orch apply rgw foo.a --placement '1;smithi023=foo.a' 2024-03-11T20:50:15.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:14 smithi146 bash[23809]: cluster 2024-03-11T20:50:13.443182+0000 mgr.y (mgr.14152) 329 : cluster 0 pgmap v331: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:50:15.033 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:14 smithi023 bash[29146]: cluster 2024-03-11T20:50:13.443182+0000 mgr.y (mgr.14152) 329 : cluster 0 pgmap v331: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:50:15.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:14 smithi023 bash[21372]: cluster 2024-03-11T20:50:13.443182+0000 mgr.y (mgr.14152) 329 : cluster 0 pgmap v331: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:50:17.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:16 smithi146 bash[23809]: cluster 2024-03-11T20:50:15.443785+0000 mgr.y (mgr.14152) 330 : cluster 0 pgmap v332: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:50:17.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:16 smithi023 bash[29146]: cluster 2024-03-11T20:50:15.443785+0000 mgr.y (mgr.14152) 330 : cluster 0 pgmap v332: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:50:17.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:16 smithi023 bash[21372]: cluster 2024-03-11T20:50:15.443785+0000 mgr.y (mgr.14152) 330 : cluster 0 pgmap v332: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:50:18.906 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T20:50:19.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:18 smithi023 bash[21372]: cluster 2024-03-11T20:50:17.444348+0000 mgr.y (mgr.14152) 331 : cluster 0 pgmap v333: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:50:19.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:18 smithi023 bash[29146]: cluster 2024-03-11T20:50:17.444348+0000 mgr.y (mgr.14152) 331 : cluster 0 pgmap v333: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:50:19.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:18 smithi146 bash[23809]: cluster 2024-03-11T20:50:17.444348+0000 mgr.y (mgr.14152) 331 : cluster 0 pgmap v333: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:50:21.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:20 smithi146 bash[23809]: cluster 2024-03-11T20:50:19.444914+0000 mgr.y (mgr.14152) 332 : cluster 0 pgmap v334: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:50:21.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:20 smithi023 bash[21372]: cluster 2024-03-11T20:50:19.444914+0000 mgr.y (mgr.14152) 332 : cluster 0 pgmap v334: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:50:21.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:20 smithi023 bash[29146]: cluster 2024-03-11T20:50:19.444914+0000 mgr.y (mgr.14152) 332 : cluster 0 pgmap v334: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:50:21.944 INFO:teuthology.orchestra.run.smithi146.stdout:Scheduled rgw.foo.a update... 2024-03-11T20:50:22.689 DEBUG:teuthology.orchestra.run.smithi023:rgw.foo.a> sudo journalctl -f -n 0 -u ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@rgw.foo.a.service 2024-03-11T20:50:22.692 INFO:tasks.cephadm:Adding ceph.iscsi.iscsi.a on smithi146 2024-03-11T20:50:22.692 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd pool create datapool 3 3 replicated 2024-03-11T20:50:22.951 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:22 smithi023 bash[21372]: cluster 2024-03-11T20:50:21.445500+0000 mgr.y (mgr.14152) 333 : cluster 0 pgmap v335: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:50:22.951 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:22 smithi023 bash[21372]: audit 2024-03-11T20:50:21.928060+0000 mgr.y (mgr.14152) 334 : audit 0 from='client.24298 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi023=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:50:22.951 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:22 smithi023 bash[21372]: cephadm 2024-03-11T20:50:21.932118+0000 mgr.y (mgr.14152) 335 : cephadm 1 Saving service rgw.foo.a spec with placement smithi023=foo.a;count:1 2024-03-11T20:50:22.952 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:22 smithi023 bash[21372]: audit 2024-03-11T20:50:21.948193+0000 mon.a (mon.0) 656 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:50:23.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:22 smithi146 bash[23809]: cluster 2024-03-11T20:50:21.445500+0000 mgr.y (mgr.14152) 333 : cluster 0 pgmap v335: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:50:23.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:22 smithi146 bash[23809]: audit 2024-03-11T20:50:21.928060+0000 mgr.y (mgr.14152) 334 : audit 0 from='client.24298 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi023=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:50:23.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:22 smithi146 bash[23809]: cephadm 2024-03-11T20:50:21.932118+0000 mgr.y (mgr.14152) 335 : cephadm 1 Saving service rgw.foo.a spec with placement smithi023=foo.a;count:1 2024-03-11T20:50:23.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:22 smithi146 bash[23809]: audit 2024-03-11T20:50:21.948193+0000 mon.a (mon.0) 656 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:50:23.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:22 smithi146 bash[23809]: audit 2024-03-11T20:50:21.986723+0000 mon.a (mon.0) 657 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:50:23.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:22 smithi146 bash[23809]: audit 2024-03-11T20:50:21.990654+0000 mon.a (mon.0) 658 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:50:23.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:22 smithi146 bash[23809]: audit 2024-03-11T20:50:21.992524+0000 mon.a (mon.0) 659 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:50:23.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:22 smithi146 bash[23809]: audit 2024-03-11T20:50:22.003199+0000 mon.a (mon.0) 660 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:50:23.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:22 smithi146 bash[23809]: audit 2024-03-11T20:50:22.009734+0000 mon.a (mon.0) 661 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-11T20:50:23.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:22 smithi146 bash[23809]: audit 2024-03-11T20:50:22.013252+0000 mon.a (mon.0) 662 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-11T20:50:23.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:22 smithi146 bash[23809]: audit 2024-03-11T20:50:22.028972+0000 mon.a (mon.0) 663 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:50:23.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:22 smithi146 bash[23809]: audit 2024-03-11T20:50:22.053885+0000 mon.a (mon.0) 664 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:50:23.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:22 smithi146 bash[23809]: cephadm 2024-03-11T20:50:22.056155+0000 mgr.y (mgr.14152) 336 : cephadm 1 Deploying daemon rgw.foo.a on smithi023 2024-03-11T20:50:23.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:22 smithi023 bash[21372]: audit 2024-03-11T20:50:21.986723+0000 mon.a (mon.0) 657 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:50:23.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:22 smithi023 bash[21372]: audit 2024-03-11T20:50:21.990654+0000 mon.a (mon.0) 658 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:50:23.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:22 smithi023 bash[21372]: audit 2024-03-11T20:50:21.992524+0000 mon.a (mon.0) 659 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:50:23.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:22 smithi023 bash[21372]: audit 2024-03-11T20:50:22.003199+0000 mon.a (mon.0) 660 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:50:23.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:22 smithi023 bash[21372]: audit 2024-03-11T20:50:22.009734+0000 mon.a (mon.0) 661 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-11T20:50:23.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:22 smithi023 bash[21372]: audit 2024-03-11T20:50:22.013252+0000 mon.a (mon.0) 662 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-11T20:50:23.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:22 smithi023 bash[21372]: audit 2024-03-11T20:50:22.028972+0000 mon.a (mon.0) 663 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:50:23.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:22 smithi023 bash[21372]: audit 2024-03-11T20:50:22.053885+0000 mon.a (mon.0) 664 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:50:23.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:22 smithi023 bash[21372]: cephadm 2024-03-11T20:50:22.056155+0000 mgr.y (mgr.14152) 336 : cephadm 1 Deploying daemon rgw.foo.a on smithi023 2024-03-11T20:50:23.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:22 smithi023 bash[29146]: cluster 2024-03-11T20:50:21.445500+0000 mgr.y (mgr.14152) 333 : cluster 0 pgmap v335: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:50:23.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:22 smithi023 bash[29146]: audit 2024-03-11T20:50:21.928060+0000 mgr.y (mgr.14152) 334 : audit 0 from='client.24298 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi023=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:50:23.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:22 smithi023 bash[29146]: cephadm 2024-03-11T20:50:21.932118+0000 mgr.y (mgr.14152) 335 : cephadm 1 Saving service rgw.foo.a spec with placement smithi023=foo.a;count:1 2024-03-11T20:50:23.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:22 smithi023 bash[29146]: audit 2024-03-11T20:50:21.948193+0000 mon.a (mon.0) 656 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:50:23.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:22 smithi023 bash[29146]: audit 2024-03-11T20:50:21.986723+0000 mon.a (mon.0) 657 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:50:23.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:22 smithi023 bash[29146]: audit 2024-03-11T20:50:21.990654+0000 mon.a (mon.0) 658 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:50:23.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:22 smithi023 bash[29146]: audit 2024-03-11T20:50:21.992524+0000 mon.a (mon.0) 659 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:50:23.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:22 smithi023 bash[29146]: audit 2024-03-11T20:50:22.003199+0000 mon.a (mon.0) 660 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:50:23.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:22 smithi023 bash[29146]: audit 2024-03-11T20:50:22.009734+0000 mon.a (mon.0) 661 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-11T20:50:23.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:22 smithi023 bash[29146]: audit 2024-03-11T20:50:22.013252+0000 mon.a (mon.0) 662 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-11T20:50:23.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:22 smithi023 bash[29146]: audit 2024-03-11T20:50:22.028972+0000 mon.a (mon.0) 663 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:50:23.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:22 smithi023 bash[29146]: audit 2024-03-11T20:50:22.053885+0000 mon.a (mon.0) 664 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:50:23.286 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:22 smithi023 bash[29146]: cephadm 2024-03-11T20:50:22.056155+0000 mgr.y (mgr.14152) 336 : cephadm 1 Deploying daemon rgw.foo.a on smithi023 2024-03-11T20:50:24.645 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:24 smithi023 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:50:24.645 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:50:24 smithi023 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:50:24.646 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:24 smithi023 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:50:24.646 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 11 20:50:24 smithi023 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:50:24.646 INFO:journalctl@ceph.osd.1.smithi023.stdout:Mar 11 20:50:24 smithi023 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:50:24.647 INFO:journalctl@ceph.osd.3.smithi023.stdout:Mar 11 20:50:24 smithi023 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:50:24.647 INFO:journalctl@ceph.osd.2.smithi023.stdout:Mar 11 20:50:24 smithi023 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:50:24.903 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:24 smithi023 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:50:24.904 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:50:24 smithi023 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:50:24.904 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 11 20:50:24 smithi023 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:50:24.904 INFO:journalctl@ceph.osd.1.smithi023.stdout:Mar 11 20:50:24 smithi023 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:50:24.905 INFO:journalctl@ceph.osd.3.smithi023.stdout:Mar 11 20:50:24 smithi023 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:50:24.905 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:24 smithi023 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:50:24.905 INFO:journalctl@ceph.osd.2.smithi023.stdout:Mar 11 20:50:24 smithi023 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:50:24.906 INFO:journalctl@ceph.rgw.foo.a.smithi023.stdout:Mar 11 20:50:24 smithi023 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:50:24.906 INFO:journalctl@ceph.rgw.foo.a.smithi023.stdout:Mar 11 20:50:24 smithi023 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:50:24.906 INFO:journalctl@ceph.rgw.foo.a.smithi023.stdout:Mar 11 20:50:24 smithi023 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:50:24.906 INFO:journalctl@ceph.rgw.foo.a.smithi023.stdout:Mar 11 20:50:24 smithi023 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:50:24.906 INFO:journalctl@ceph.rgw.foo.a.smithi023.stdout:Mar 11 20:50:24 smithi023 systemd[1]: Started Ceph rgw.foo.a for 67ba00f8-dfe7-11ee-95c8-87774f69a715. 2024-03-11T20:50:25.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:24 smithi146 bash[23809]: cluster 2024-03-11T20:50:23.446092+0000 mgr.y (mgr.14152) 337 : cluster 0 pgmap v336: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:50:25.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:24 smithi146 bash[23809]: audit 2024-03-11T20:50:24.960302+0000 mon.a (mon.0) 665 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:50:25.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:24 smithi023 bash[21372]: cluster 2024-03-11T20:50:23.446092+0000 mgr.y (mgr.14152) 337 : cluster 0 pgmap v336: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:50:25.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:24 smithi023 bash[21372]: audit 2024-03-11T20:50:24.960302+0000 mon.a (mon.0) 665 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:50:25.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:24 smithi023 bash[29146]: cluster 2024-03-11T20:50:23.446092+0000 mgr.y (mgr.14152) 337 : cluster 0 pgmap v336: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:50:25.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:24 smithi023 bash[29146]: audit 2024-03-11T20:50:24.960302+0000 mon.a (mon.0) 665 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:50:26.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:25 smithi146 bash[23809]: audit 2024-03-11T20:50:24.970583+0000 mon.a (mon.0) 666 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:50:26.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:25 smithi146 bash[23809]: audit 2024-03-11T20:50:24.980609+0000 mon.a (mon.0) 667 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:50:26.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:25 smithi146 bash[23809]: cephadm 2024-03-11T20:50:24.981319+0000 mgr.y (mgr.14152) 338 : cephadm 1 Saving service rgw.foo.a spec with placement smithi023=foo.a;count:1 2024-03-11T20:50:26.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:25 smithi146 bash[23809]: audit 2024-03-11T20:50:24.984942+0000 mon.a (mon.0) 668 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:50:26.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:25 smithi146 bash[23809]: audit 2024-03-11T20:50:24.989376+0000 mon.a (mon.0) 669 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:50:26.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:25 smithi146 bash[23809]: audit 2024-03-11T20:50:25.037722+0000 mon.a (mon.0) 670 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:50:26.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:25 smithi146 bash[23809]: audit 2024-03-11T20:50:25.546592+0000 mon.a (mon.0) 671 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:50:26.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:25 smithi023 bash[21372]: audit 2024-03-11T20:50:24.970583+0000 mon.a (mon.0) 666 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:50:26.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:25 smithi023 bash[21372]: audit 2024-03-11T20:50:24.980609+0000 mon.a (mon.0) 667 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:50:26.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:25 smithi023 bash[21372]: cephadm 2024-03-11T20:50:24.981319+0000 mgr.y (mgr.14152) 338 : cephadm 1 Saving service rgw.foo.a spec with placement smithi023=foo.a;count:1 2024-03-11T20:50:26.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:25 smithi023 bash[21372]: audit 2024-03-11T20:50:24.984942+0000 mon.a (mon.0) 668 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:50:26.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:25 smithi023 bash[21372]: audit 2024-03-11T20:50:24.989376+0000 mon.a (mon.0) 669 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:50:26.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:25 smithi023 bash[21372]: audit 2024-03-11T20:50:25.037722+0000 mon.a (mon.0) 670 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:50:26.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:25 smithi023 bash[21372]: audit 2024-03-11T20:50:25.546592+0000 mon.a (mon.0) 671 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:50:26.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:25 smithi023 bash[29146]: audit 2024-03-11T20:50:24.970583+0000 mon.a (mon.0) 666 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:50:26.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:25 smithi023 bash[29146]: audit 2024-03-11T20:50:24.980609+0000 mon.a (mon.0) 667 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:50:26.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:25 smithi023 bash[29146]: cephadm 2024-03-11T20:50:24.981319+0000 mgr.y (mgr.14152) 338 : cephadm 1 Saving service rgw.foo.a spec with placement smithi023=foo.a;count:1 2024-03-11T20:50:26.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:25 smithi023 bash[29146]: audit 2024-03-11T20:50:24.984942+0000 mon.a (mon.0) 668 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:50:26.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:25 smithi023 bash[29146]: audit 2024-03-11T20:50:24.989376+0000 mon.a (mon.0) 669 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:50:26.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:25 smithi023 bash[29146]: audit 2024-03-11T20:50:25.037722+0000 mon.a (mon.0) 670 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:50:26.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:25 smithi023 bash[29146]: audit 2024-03-11T20:50:25.546592+0000 mon.a (mon.0) 671 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:50:27.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:27 smithi146 bash[23809]: cluster 2024-03-11T20:50:25.446504+0000 mgr.y (mgr.14152) 339 : cluster 0 pgmap v337: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:50:27.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:26 smithi023 bash[21372]: cluster 2024-03-11T20:50:25.446504+0000 mgr.y (mgr.14152) 339 : cluster 0 pgmap v337: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:50:27.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:27 smithi023 bash[29146]: cluster 2024-03-11T20:50:25.446504+0000 mgr.y (mgr.14152) 339 : cluster 0 pgmap v337: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:50:27.436 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T20:50:28.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:28 smithi023 bash[21372]: cluster 2024-03-11T20:50:27.015307+0000 mon.a (mon.0) 672 : cluster 0 osdmap e76: 8 total, 8 up, 8 in 2024-03-11T20:50:28.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:28 smithi023 bash[21372]: audit 2024-03-11T20:50:27.028537+0000 mon.c (mon.2) 15 : audit 1 from='client.? 172.21.15.23:0/2656031793' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-11T20:50:28.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:28 smithi023 bash[21372]: audit 2024-03-11T20:50:27.029178+0000 mon.a (mon.0) 673 : audit 1 from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-11T20:50:28.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:28 smithi023 bash[29146]: cluster 2024-03-11T20:50:27.015307+0000 mon.a (mon.0) 672 : cluster 0 osdmap e76: 8 total, 8 up, 8 in 2024-03-11T20:50:28.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:28 smithi023 bash[29146]: audit 2024-03-11T20:50:27.028537+0000 mon.c (mon.2) 15 : audit 1 from='client.? 172.21.15.23:0/2656031793' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-11T20:50:28.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:28 smithi023 bash[29146]: audit 2024-03-11T20:50:27.029178+0000 mon.a (mon.0) 673 : audit 1 from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-11T20:50:28.430 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:28 smithi146 bash[23809]: cluster 2024-03-11T20:50:27.015307+0000 mon.a (mon.0) 672 : cluster 0 osdmap e76: 8 total, 8 up, 8 in 2024-03-11T20:50:28.430 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:28 smithi146 bash[23809]: audit 2024-03-11T20:50:27.028537+0000 mon.c (mon.2) 15 : audit 1 from='client.? 172.21.15.23:0/2656031793' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-11T20:50:28.430 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:28 smithi146 bash[23809]: audit 2024-03-11T20:50:27.029178+0000 mon.a (mon.0) 673 : audit 1 from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-11T20:50:29.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:29 smithi023 bash[21372]: cluster 2024-03-11T20:50:27.447134+0000 mgr.y (mgr.14152) 340 : cluster 0 pgmap v339: 33 pgs: 4 creating+peering, 1 active+clean, 28 unknown; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:50:29.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:29 smithi023 bash[21372]: cluster 2024-03-11T20:50:28.008086+0000 mon.a (mon.0) 674 : cluster 3 Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:50:29.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:29 smithi023 bash[21372]: audit 2024-03-11T20:50:28.033296+0000 mon.a (mon.0) 675 : audit 1 from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-11T20:50:29.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:29 smithi023 bash[21372]: cluster 2024-03-11T20:50:28.037784+0000 mon.a (mon.0) 676 : cluster 0 osdmap e77: 8 total, 8 up, 8 in 2024-03-11T20:50:29.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:29 smithi023 bash[29146]: cluster 2024-03-11T20:50:27.447134+0000 mgr.y (mgr.14152) 340 : cluster 0 pgmap v339: 33 pgs: 4 creating+peering, 1 active+clean, 28 unknown; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:50:29.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:29 smithi023 bash[29146]: cluster 2024-03-11T20:50:28.008086+0000 mon.a (mon.0) 674 : cluster 3 Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:50:29.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:29 smithi023 bash[29146]: audit 2024-03-11T20:50:28.033296+0000 mon.a (mon.0) 675 : audit 1 from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-11T20:50:29.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:29 smithi023 bash[29146]: cluster 2024-03-11T20:50:28.037784+0000 mon.a (mon.0) 676 : cluster 0 osdmap e77: 8 total, 8 up, 8 in 2024-03-11T20:50:29.384 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:29 smithi146 bash[23809]: cluster 2024-03-11T20:50:27.447134+0000 mgr.y (mgr.14152) 340 : cluster 0 pgmap v339: 33 pgs: 4 creating+peering, 1 active+clean, 28 unknown; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:50:29.384 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:29 smithi146 bash[23809]: cluster 2024-03-11T20:50:28.008086+0000 mon.a (mon.0) 674 : cluster 3 Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:50:29.384 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:29 smithi146 bash[23809]: audit 2024-03-11T20:50:28.033296+0000 mon.a (mon.0) 675 : audit 1 from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-11T20:50:29.384 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:29 smithi146 bash[23809]: cluster 2024-03-11T20:50:28.037784+0000 mon.a (mon.0) 676 : cluster 0 osdmap e77: 8 total, 8 up, 8 in 2024-03-11T20:50:30.317 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:30 smithi146 bash[23809]: cluster 2024-03-11T20:50:29.058793+0000 mon.a (mon.0) 677 : cluster 0 osdmap e78: 8 total, 8 up, 8 in 2024-03-11T20:50:30.337 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:30 smithi146 bash[23809]: audit 2024-03-11T20:50:29.066962+0000 mon.c (mon.2) 16 : audit 1 from='client.? 172.21.15.23:0/2185274774' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-11T20:50:30.337 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:30 smithi146 bash[23809]: audit 2024-03-11T20:50:29.068833+0000 mon.a (mon.0) 678 : audit 1 from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-11T20:50:30.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:30 smithi023 bash[21372]: cluster 2024-03-11T20:50:29.058793+0000 mon.a (mon.0) 677 : cluster 0 osdmap e78: 8 total, 8 up, 8 in 2024-03-11T20:50:30.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:30 smithi023 bash[21372]: audit 2024-03-11T20:50:29.066962+0000 mon.c (mon.2) 16 : audit 1 from='client.? 172.21.15.23:0/2185274774' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-11T20:50:30.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:30 smithi023 bash[21372]: audit 2024-03-11T20:50:29.068833+0000 mon.a (mon.0) 678 : audit 1 from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-11T20:50:30.534 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:30 smithi023 bash[29146]: cluster 2024-03-11T20:50:29.058793+0000 mon.a (mon.0) 677 : cluster 0 osdmap e78: 8 total, 8 up, 8 in 2024-03-11T20:50:30.534 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:30 smithi023 bash[29146]: audit 2024-03-11T20:50:29.066962+0000 mon.c (mon.2) 16 : audit 1 from='client.? 172.21.15.23:0/2185274774' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-11T20:50:30.534 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:30 smithi023 bash[29146]: audit 2024-03-11T20:50:29.068833+0000 mon.a (mon.0) 678 : audit 1 from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-11T20:50:31.165 INFO:teuthology.orchestra.run.smithi146.stderr:pool 'datapool' created 2024-03-11T20:50:31.396 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:31 smithi146 bash[23809]: cluster 2024-03-11T20:50:29.447898+0000 mgr.y (mgr.14152) 341 : cluster 0 pgmap v342: 65 pgs: 9 creating+peering, 15 active+clean, 41 unknown; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 341 B/s wr, 0 op/s 2024-03-11T20:50:31.397 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:31 smithi146 bash[23809]: audit 2024-03-11T20:50:30.063376+0000 mon.a (mon.0) 679 : audit 1 from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-11T20:50:31.397 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:31 smithi146 bash[23809]: cluster 2024-03-11T20:50:30.076531+0000 mon.a (mon.0) 680 : cluster 0 osdmap e79: 8 total, 8 up, 8 in 2024-03-11T20:50:31.397 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:31 smithi146 bash[23809]: audit 2024-03-11T20:50:30.550544+0000 mon.a (mon.0) 681 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:50:31.397 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:31 smithi146 bash[23809]: audit 2024-03-11T20:50:30.564953+0000 mon.a (mon.0) 682 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:50:31.397 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:31 smithi146 bash[23809]: audit 2024-03-11T20:50:30.566573+0000 mon.a (mon.0) 683 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:50:31.397 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:31 smithi146 bash[23809]: audit 2024-03-11T20:50:30.568319+0000 mon.a (mon.0) 684 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:50:31.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:31 smithi023 bash[21372]: cluster 2024-03-11T20:50:29.447898+0000 mgr.y (mgr.14152) 341 : cluster 0 pgmap v342: 65 pgs: 9 creating+peering, 15 active+clean, 41 unknown; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 341 B/s wr, 0 op/s 2024-03-11T20:50:31.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:31 smithi023 bash[21372]: audit 2024-03-11T20:50:30.063376+0000 mon.a (mon.0) 679 : audit 1 from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-11T20:50:31.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:31 smithi023 bash[21372]: cluster 2024-03-11T20:50:30.076531+0000 mon.a (mon.0) 680 : cluster 0 osdmap e79: 8 total, 8 up, 8 in 2024-03-11T20:50:31.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:31 smithi023 bash[21372]: audit 2024-03-11T20:50:30.550544+0000 mon.a (mon.0) 681 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:50:31.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:31 smithi023 bash[21372]: audit 2024-03-11T20:50:30.564953+0000 mon.a (mon.0) 682 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:50:31.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:31 smithi023 bash[21372]: audit 2024-03-11T20:50:30.566573+0000 mon.a (mon.0) 683 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:50:31.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:31 smithi023 bash[21372]: audit 2024-03-11T20:50:30.568319+0000 mon.a (mon.0) 684 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:50:31.534 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:31 smithi023 bash[29146]: cluster 2024-03-11T20:50:29.447898+0000 mgr.y (mgr.14152) 341 : cluster 0 pgmap v342: 65 pgs: 9 creating+peering, 15 active+clean, 41 unknown; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 341 B/s wr, 0 op/s 2024-03-11T20:50:31.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:31 smithi023 bash[29146]: audit 2024-03-11T20:50:30.063376+0000 mon.a (mon.0) 679 : audit 1 from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-11T20:50:31.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:31 smithi023 bash[29146]: cluster 2024-03-11T20:50:30.076531+0000 mon.a (mon.0) 680 : cluster 0 osdmap e79: 8 total, 8 up, 8 in 2024-03-11T20:50:31.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:31 smithi023 bash[29146]: audit 2024-03-11T20:50:30.550544+0000 mon.a (mon.0) 681 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:50:31.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:31 smithi023 bash[29146]: audit 2024-03-11T20:50:30.564953+0000 mon.a (mon.0) 682 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:50:31.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:31 smithi023 bash[29146]: audit 2024-03-11T20:50:30.566573+0000 mon.a (mon.0) 683 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:50:31.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:31 smithi023 bash[29146]: audit 2024-03-11T20:50:30.568319+0000 mon.a (mon.0) 684 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:50:32.013 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- rbd pool init datapool 2024-03-11T20:50:32.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:32 smithi146 bash[23809]: cephadm 2024-03-11T20:50:30.578944+0000 mgr.y (mgr.14152) 342 : cephadm 1 Checking dashboard <-> RGW credentials 2024-03-11T20:50:32.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:32 smithi146 bash[23809]: audit 2024-03-11T20:50:30.700805+0000 mon.a (mon.0) 685 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-11T20:50:32.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:32 smithi146 bash[23809]: audit 2024-03-11T20:50:30.703846+0000 mon.b (mon.1) 25 : audit 1 from='client.? 172.21.15.146:0/3534100233' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-11T20:50:32.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:32 smithi146 bash[23809]: audit 2024-03-11T20:50:31.151357+0000 mon.a (mon.0) 686 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-03-11T20:50:32.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:32 smithi146 bash[23809]: cluster 2024-03-11T20:50:31.154756+0000 mon.a (mon.0) 687 : cluster 0 osdmap e80: 8 total, 8 up, 8 in 2024-03-11T20:50:32.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:32 smithi146 bash[23809]: audit 2024-03-11T20:50:31.159212+0000 mon.c (mon.2) 17 : audit 1 from='client.? 172.21.15.23:0/2185274774' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-11T20:50:32.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:32 smithi146 bash[23809]: audit 2024-03-11T20:50:31.159873+0000 mon.a (mon.0) 688 : audit 1 from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-11T20:50:32.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:32 smithi146 bash[23809]: cluster 2024-03-11T20:50:31.448748+0000 mgr.y (mgr.14152) 343 : cluster 0 pgmap v345: 100 pgs: 23 creating+peering, 28 active+clean, 49 unknown; 578 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 767 B/s wr, 1 op/s 2024-03-11T20:50:32.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:32 smithi023 bash[21372]: cephadm 2024-03-11T20:50:30.578944+0000 mgr.y (mgr.14152) 342 : cephadm 1 Checking dashboard <-> RGW credentials 2024-03-11T20:50:32.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:32 smithi023 bash[21372]: audit 2024-03-11T20:50:30.700805+0000 mon.a (mon.0) 685 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-11T20:50:32.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:32 smithi023 bash[21372]: audit 2024-03-11T20:50:30.703846+0000 mon.b (mon.1) 25 : audit 1 from='client.? 172.21.15.146:0/3534100233' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-11T20:50:32.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:32 smithi023 bash[21372]: audit 2024-03-11T20:50:31.151357+0000 mon.a (mon.0) 686 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-03-11T20:50:32.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:32 smithi023 bash[21372]: cluster 2024-03-11T20:50:31.154756+0000 mon.a (mon.0) 687 : cluster 0 osdmap e80: 8 total, 8 up, 8 in 2024-03-11T20:50:32.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:32 smithi023 bash[21372]: audit 2024-03-11T20:50:31.159212+0000 mon.c (mon.2) 17 : audit 1 from='client.? 172.21.15.23:0/2185274774' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-11T20:50:32.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:32 smithi023 bash[21372]: audit 2024-03-11T20:50:31.159873+0000 mon.a (mon.0) 688 : audit 1 from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-11T20:50:32.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:32 smithi023 bash[21372]: cluster 2024-03-11T20:50:31.448748+0000 mgr.y (mgr.14152) 343 : cluster 0 pgmap v345: 100 pgs: 23 creating+peering, 28 active+clean, 49 unknown; 578 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 767 B/s wr, 1 op/s 2024-03-11T20:50:32.534 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:32 smithi023 bash[29146]: cephadm 2024-03-11T20:50:30.578944+0000 mgr.y (mgr.14152) 342 : cephadm 1 Checking dashboard <-> RGW credentials 2024-03-11T20:50:32.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:32 smithi023 bash[29146]: audit 2024-03-11T20:50:30.700805+0000 mon.a (mon.0) 685 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-11T20:50:32.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:32 smithi023 bash[29146]: audit 2024-03-11T20:50:30.703846+0000 mon.b (mon.1) 25 : audit 1 from='client.? 172.21.15.146:0/3534100233' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-11T20:50:32.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:32 smithi023 bash[29146]: audit 2024-03-11T20:50:31.151357+0000 mon.a (mon.0) 686 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-03-11T20:50:32.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:32 smithi023 bash[29146]: cluster 2024-03-11T20:50:31.154756+0000 mon.a (mon.0) 687 : cluster 0 osdmap e80: 8 total, 8 up, 8 in 2024-03-11T20:50:32.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:32 smithi023 bash[29146]: audit 2024-03-11T20:50:31.159212+0000 mon.c (mon.2) 17 : audit 1 from='client.? 172.21.15.23:0/2185274774' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-11T20:50:32.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:32 smithi023 bash[29146]: audit 2024-03-11T20:50:31.159873+0000 mon.a (mon.0) 688 : audit 1 from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-11T20:50:32.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:32 smithi023 bash[29146]: cluster 2024-03-11T20:50:31.448748+0000 mgr.y (mgr.14152) 343 : cluster 0 pgmap v345: 100 pgs: 23 creating+peering, 28 active+clean, 49 unknown; 578 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 767 B/s wr, 1 op/s 2024-03-11T20:50:33.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:33 smithi146 bash[23809]: audit 2024-03-11T20:50:32.162406+0000 mon.a (mon.0) 689 : audit 1 from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-11T20:50:33.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:33 smithi146 bash[23809]: cluster 2024-03-11T20:50:32.173357+0000 mon.a (mon.0) 690 : cluster 0 osdmap e81: 8 total, 8 up, 8 in 2024-03-11T20:50:33.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:33 smithi023 bash[21372]: audit 2024-03-11T20:50:32.162406+0000 mon.a (mon.0) 689 : audit 1 from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-11T20:50:33.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:33 smithi023 bash[21372]: cluster 2024-03-11T20:50:32.173357+0000 mon.a (mon.0) 690 : cluster 0 osdmap e81: 8 total, 8 up, 8 in 2024-03-11T20:50:33.534 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:33 smithi023 bash[29146]: audit 2024-03-11T20:50:32.162406+0000 mon.a (mon.0) 689 : audit 1 from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-11T20:50:33.534 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:33 smithi023 bash[29146]: cluster 2024-03-11T20:50:32.173357+0000 mon.a (mon.0) 690 : cluster 0 osdmap e81: 8 total, 8 up, 8 in 2024-03-11T20:50:34.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:34 smithi146 bash[23809]: cluster 2024-03-11T20:50:33.263707+0000 mon.a (mon.0) 691 : cluster 0 osdmap e82: 8 total, 8 up, 8 in 2024-03-11T20:50:34.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:34 smithi146 bash[23809]: audit 2024-03-11T20:50:33.267469+0000 mon.c (mon.2) 18 : audit 1 from='client.? 172.21.15.23:0/2185274774' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-11T20:50:34.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:34 smithi146 bash[23809]: audit 2024-03-11T20:50:33.271357+0000 mon.a (mon.0) 692 : audit 1 from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-11T20:50:34.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:34 smithi146 bash[23809]: audit 2024-03-11T20:50:33.274091+0000 mon.a (mon.0) 693 : audit 1 from='client.? 172.21.15.23:0/4257498094' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-11T20:50:34.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:34 smithi146 bash[23809]: cluster 2024-03-11T20:50:33.449604+0000 mgr.y (mgr.14152) 344 : cluster 0 pgmap v348: 132 pgs: 30 creating+peering, 51 active+clean, 51 unknown; 578 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1023 B/s wr, 4 op/s 2024-03-11T20:50:34.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:34 smithi023 bash[21372]: cluster 2024-03-11T20:50:33.263707+0000 mon.a (mon.0) 691 : cluster 0 osdmap e82: 8 total, 8 up, 8 in 2024-03-11T20:50:34.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:34 smithi023 bash[21372]: audit 2024-03-11T20:50:33.267469+0000 mon.c (mon.2) 18 : audit 1 from='client.? 172.21.15.23:0/2185274774' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-11T20:50:34.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:34 smithi023 bash[21372]: audit 2024-03-11T20:50:33.271357+0000 mon.a (mon.0) 692 : audit 1 from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-11T20:50:34.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:34 smithi023 bash[21372]: audit 2024-03-11T20:50:33.274091+0000 mon.a (mon.0) 693 : audit 1 from='client.? 172.21.15.23:0/4257498094' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-11T20:50:34.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:34 smithi023 bash[21372]: cluster 2024-03-11T20:50:33.449604+0000 mgr.y (mgr.14152) 344 : cluster 0 pgmap v348: 132 pgs: 30 creating+peering, 51 active+clean, 51 unknown; 578 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1023 B/s wr, 4 op/s 2024-03-11T20:50:34.534 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:34 smithi023 bash[29146]: cluster 2024-03-11T20:50:33.263707+0000 mon.a (mon.0) 691 : cluster 0 osdmap e82: 8 total, 8 up, 8 in 2024-03-11T20:50:34.534 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:34 smithi023 bash[29146]: audit 2024-03-11T20:50:33.267469+0000 mon.c (mon.2) 18 : audit 1 from='client.? 172.21.15.23:0/2185274774' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-11T20:50:34.534 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:34 smithi023 bash[29146]: audit 2024-03-11T20:50:33.271357+0000 mon.a (mon.0) 692 : audit 1 from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-11T20:50:34.534 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:34 smithi023 bash[29146]: audit 2024-03-11T20:50:33.274091+0000 mon.a (mon.0) 693 : audit 1 from='client.? 172.21.15.23:0/4257498094' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-11T20:50:34.534 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:34 smithi023 bash[29146]: cluster 2024-03-11T20:50:33.449604+0000 mgr.y (mgr.14152) 344 : cluster 0 pgmap v348: 132 pgs: 30 creating+peering, 51 active+clean, 51 unknown; 578 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1023 B/s wr, 4 op/s 2024-03-11T20:50:35.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:35 smithi023 bash[21372]: audit 2024-03-11T20:50:34.280685+0000 mon.a (mon.0) 694 : audit 1 from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-11T20:50:35.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:35 smithi023 bash[21372]: audit 2024-03-11T20:50:34.280919+0000 mon.a (mon.0) 695 : audit 1 from='client.? 172.21.15.23:0/4257498094' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-11T20:50:35.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:35 smithi023 bash[21372]: cluster 2024-03-11T20:50:34.291541+0000 mon.a (mon.0) 696 : cluster 0 osdmap e83: 8 total, 8 up, 8 in 2024-03-11T20:50:35.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:35 smithi023 bash[21372]: audit 2024-03-11T20:50:34.296759+0000 mon.a (mon.0) 697 : audit 1 from='client.? 172.21.15.23:0/4257498094' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-11T20:50:35.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:35 smithi023 bash[21372]: audit 2024-03-11T20:50:34.317478+0000 mon.c (mon.2) 19 : audit 1 from='client.? 172.21.15.23:0/2185274774' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-11T20:50:35.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:35 smithi023 bash[21372]: audit 2024-03-11T20:50:34.319984+0000 mon.a (mon.0) 698 : audit 1 from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-11T20:50:35.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:35 smithi023 bash[21372]: cluster 2024-03-11T20:50:34.559590+0000 mon.a (mon.0) 699 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:50:35.534 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:35 smithi023 bash[29146]: audit 2024-03-11T20:50:34.280685+0000 mon.a (mon.0) 694 : audit 1 from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-11T20:50:35.534 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:35 smithi023 bash[29146]: audit 2024-03-11T20:50:34.280919+0000 mon.a (mon.0) 695 : audit 1 from='client.? 172.21.15.23:0/4257498094' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-11T20:50:35.534 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:35 smithi023 bash[29146]: cluster 2024-03-11T20:50:34.291541+0000 mon.a (mon.0) 696 : cluster 0 osdmap e83: 8 total, 8 up, 8 in 2024-03-11T20:50:35.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:35 smithi023 bash[29146]: audit 2024-03-11T20:50:34.296759+0000 mon.a (mon.0) 697 : audit 1 from='client.? 172.21.15.23:0/4257498094' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-11T20:50:35.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:35 smithi023 bash[29146]: audit 2024-03-11T20:50:34.317478+0000 mon.c (mon.2) 19 : audit 1 from='client.? 172.21.15.23:0/2185274774' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-11T20:50:35.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:35 smithi023 bash[29146]: audit 2024-03-11T20:50:34.319984+0000 mon.a (mon.0) 698 : audit 1 from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-11T20:50:35.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:35 smithi023 bash[29146]: cluster 2024-03-11T20:50:34.559590+0000 mon.a (mon.0) 699 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:50:35.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:35 smithi146 bash[23809]: audit 2024-03-11T20:50:34.280685+0000 mon.a (mon.0) 694 : audit 1 from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-11T20:50:35.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:35 smithi146 bash[23809]: audit 2024-03-11T20:50:34.280919+0000 mon.a (mon.0) 695 : audit 1 from='client.? 172.21.15.23:0/4257498094' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-11T20:50:35.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:35 smithi146 bash[23809]: cluster 2024-03-11T20:50:34.291541+0000 mon.a (mon.0) 696 : cluster 0 osdmap e83: 8 total, 8 up, 8 in 2024-03-11T20:50:35.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:35 smithi146 bash[23809]: audit 2024-03-11T20:50:34.296759+0000 mon.a (mon.0) 697 : audit 1 from='client.? 172.21.15.23:0/4257498094' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-11T20:50:35.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:35 smithi146 bash[23809]: audit 2024-03-11T20:50:34.317478+0000 mon.c (mon.2) 19 : audit 1 from='client.? 172.21.15.23:0/2185274774' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-11T20:50:35.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:35 smithi146 bash[23809]: audit 2024-03-11T20:50:34.319984+0000 mon.a (mon.0) 698 : audit 1 from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-11T20:50:35.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:35 smithi146 bash[23809]: cluster 2024-03-11T20:50:34.559590+0000 mon.a (mon.0) 699 : cluster 3 Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:50:36.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:36 smithi146 bash[23809]: audit 2024-03-11T20:50:35.303941+0000 mon.a (mon.0) 700 : audit 1 from='client.? 172.21.15.23:0/4257498094' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-11T20:50:36.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:36 smithi146 bash[23809]: audit 2024-03-11T20:50:35.304074+0000 mon.a (mon.0) 701 : audit 1 from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-11T20:50:36.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:36 smithi146 bash[23809]: cluster 2024-03-11T20:50:35.320509+0000 mon.a (mon.0) 702 : cluster 0 osdmap e84: 8 total, 8 up, 8 in 2024-03-11T20:50:36.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:36 smithi146 bash[23809]: cluster 2024-03-11T20:50:35.450365+0000 mgr.y (mgr.14152) 345 : cluster 0 pgmap v351: 132 pgs: 14 creating+peering, 104 active+clean, 14 unknown; 578 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 4.7 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-03-11T20:50:36.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:36 smithi146 bash[23809]: audit 2024-03-11T20:50:35.616489+0000 mon.a (mon.0) 703 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:50:36.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:36 smithi146 bash[23809]: audit 2024-03-11T20:50:35.626501+0000 mon.a (mon.0) 704 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:50:36.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:36 smithi146 bash[23809]: audit 2024-03-11T20:50:35.642299+0000 mon.a (mon.0) 705 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:50:36.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:36 smithi146 bash[23809]: audit 2024-03-11T20:50:35.717033+0000 mon.a (mon.0) 706 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:50:36.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:36 smithi146 bash[23809]: audit 2024-03-11T20:50:35.719101+0000 mon.a (mon.0) 707 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:50:36.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:36 smithi146 bash[23809]: audit 2024-03-11T20:50:35.720075+0000 mon.a (mon.0) 708 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:50:36.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:36 smithi146 bash[23809]: cephadm 2024-03-11T20:50:35.725211+0000 mgr.y (mgr.14152) 346 : cephadm 1 Checking dashboard <-> RGW credentials 2024-03-11T20:50:36.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:36 smithi146 bash[23809]: audit 2024-03-11T20:50:36.019422+0000 mon.a (mon.0) 709 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:50:36.773 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T20:50:36.784 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:36 smithi023 bash[29146]: audit 2024-03-11T20:50:35.303941+0000 mon.a (mon.0) 700 : audit 1 from='client.? 172.21.15.23:0/4257498094' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-11T20:50:36.784 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:36 smithi023 bash[29146]: audit 2024-03-11T20:50:35.304074+0000 mon.a (mon.0) 701 : audit 1 from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-11T20:50:36.784 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:36 smithi023 bash[29146]: cluster 2024-03-11T20:50:35.320509+0000 mon.a (mon.0) 702 : cluster 0 osdmap e84: 8 total, 8 up, 8 in 2024-03-11T20:50:36.784 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:36 smithi023 bash[29146]: cluster 2024-03-11T20:50:35.450365+0000 mgr.y (mgr.14152) 345 : cluster 0 pgmap v351: 132 pgs: 14 creating+peering, 104 active+clean, 14 unknown; 578 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 4.7 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-03-11T20:50:36.784 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:36 smithi023 bash[29146]: audit 2024-03-11T20:50:35.616489+0000 mon.a (mon.0) 703 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:50:36.784 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:36 smithi023 bash[29146]: audit 2024-03-11T20:50:35.626501+0000 mon.a (mon.0) 704 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:50:36.784 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:36 smithi023 bash[29146]: audit 2024-03-11T20:50:35.642299+0000 mon.a (mon.0) 705 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:50:36.784 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:36 smithi023 bash[29146]: audit 2024-03-11T20:50:35.717033+0000 mon.a (mon.0) 706 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:50:36.784 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:36 smithi023 bash[29146]: audit 2024-03-11T20:50:35.719101+0000 mon.a (mon.0) 707 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:50:36.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:36 smithi023 bash[29146]: audit 2024-03-11T20:50:35.720075+0000 mon.a (mon.0) 708 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:50:36.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:36 smithi023 bash[29146]: cephadm 2024-03-11T20:50:35.725211+0000 mgr.y (mgr.14152) 346 : cephadm 1 Checking dashboard <-> RGW credentials 2024-03-11T20:50:36.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:36 smithi023 bash[29146]: audit 2024-03-11T20:50:36.019422+0000 mon.a (mon.0) 709 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:50:36.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:36 smithi023 bash[21372]: audit 2024-03-11T20:50:35.303941+0000 mon.a (mon.0) 700 : audit 1 from='client.? 172.21.15.23:0/4257498094' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-11T20:50:36.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:36 smithi023 bash[21372]: audit 2024-03-11T20:50:35.304074+0000 mon.a (mon.0) 701 : audit 1 from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-11T20:50:36.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:36 smithi023 bash[21372]: cluster 2024-03-11T20:50:35.320509+0000 mon.a (mon.0) 702 : cluster 0 osdmap e84: 8 total, 8 up, 8 in 2024-03-11T20:50:36.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:36 smithi023 bash[21372]: cluster 2024-03-11T20:50:35.450365+0000 mgr.y (mgr.14152) 345 : cluster 0 pgmap v351: 132 pgs: 14 creating+peering, 104 active+clean, 14 unknown; 578 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 4.7 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-03-11T20:50:36.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:36 smithi023 bash[21372]: audit 2024-03-11T20:50:35.616489+0000 mon.a (mon.0) 703 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:50:36.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:36 smithi023 bash[21372]: audit 2024-03-11T20:50:35.626501+0000 mon.a (mon.0) 704 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:50:36.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:36 smithi023 bash[21372]: audit 2024-03-11T20:50:35.642299+0000 mon.a (mon.0) 705 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:50:36.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:36 smithi023 bash[21372]: audit 2024-03-11T20:50:35.717033+0000 mon.a (mon.0) 706 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:50:36.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:36 smithi023 bash[21372]: audit 2024-03-11T20:50:35.719101+0000 mon.a (mon.0) 707 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:50:36.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:36 smithi023 bash[21372]: audit 2024-03-11T20:50:35.720075+0000 mon.a (mon.0) 708 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:50:36.786 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:36 smithi023 bash[21372]: cephadm 2024-03-11T20:50:35.725211+0000 mgr.y (mgr.14152) 346 : cephadm 1 Checking dashboard <-> RGW credentials 2024-03-11T20:50:36.786 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:36 smithi023 bash[21372]: audit 2024-03-11T20:50:36.019422+0000 mon.a (mon.0) 709 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:50:38.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:38 smithi023 bash[21372]: cluster 2024-03-11T20:50:37.451576+0000 mgr.y (mgr.14152) 347 : cluster 0 pgmap v352: 132 pgs: 3 creating+peering, 122 active+clean, 7 unknown; 579 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 1.7 KiB/s wr, 34 op/s 2024-03-11T20:50:38.784 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:38 smithi023 bash[29146]: cluster 2024-03-11T20:50:37.451576+0000 mgr.y (mgr.14152) 347 : cluster 0 pgmap v352: 132 pgs: 3 creating+peering, 122 active+clean, 7 unknown; 579 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 1.7 KiB/s wr, 34 op/s 2024-03-11T20:50:38.816 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:38 smithi146 bash[23809]: cluster 2024-03-11T20:50:37.451576+0000 mgr.y (mgr.14152) 347 : cluster 0 pgmap v352: 132 pgs: 3 creating+peering, 122 active+clean, 7 unknown; 579 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 1.7 KiB/s wr, 34 op/s 2024-03-11T20:50:40.784 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:40 smithi023 bash[21372]: cluster 2024-03-11T20:50:39.452828+0000 mgr.y (mgr.14152) 348 : cluster 0 pgmap v353: 132 pgs: 132 active+clean; 581 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s rd, 4.0 KiB/s wr, 113 op/s 2024-03-11T20:50:40.784 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:40 smithi023 bash[21372]: cluster 2024-03-11T20:50:39.561830+0000 mon.a (mon.0) 710 : cluster 3 Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:50:40.784 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:40 smithi023 bash[21372]: audit 2024-03-11T20:50:39.735277+0000 mon.a (mon.0) 711 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-11T20:50:40.784 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:40 smithi023 bash[21372]: audit 2024-03-11T20:50:39.738233+0000 mon.b (mon.1) 26 : audit 1 from='client.? 172.21.15.146:0/3605892022' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-11T20:50:40.784 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:40 smithi023 bash[29146]: cluster 2024-03-11T20:50:39.452828+0000 mgr.y (mgr.14152) 348 : cluster 0 pgmap v353: 132 pgs: 132 active+clean; 581 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s rd, 4.0 KiB/s wr, 113 op/s 2024-03-11T20:50:40.784 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:40 smithi023 bash[29146]: cluster 2024-03-11T20:50:39.561830+0000 mon.a (mon.0) 710 : cluster 3 Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:50:40.784 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:40 smithi023 bash[29146]: audit 2024-03-11T20:50:39.735277+0000 mon.a (mon.0) 711 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-11T20:50:40.784 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:40 smithi023 bash[29146]: audit 2024-03-11T20:50:39.738233+0000 mon.b (mon.1) 26 : audit 1 from='client.? 172.21.15.146:0/3605892022' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-11T20:50:41.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:40 smithi146 bash[23809]: cluster 2024-03-11T20:50:39.452828+0000 mgr.y (mgr.14152) 348 : cluster 0 pgmap v353: 132 pgs: 132 active+clean; 581 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s rd, 4.0 KiB/s wr, 113 op/s 2024-03-11T20:50:41.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:40 smithi146 bash[23809]: cluster 2024-03-11T20:50:39.561830+0000 mon.a (mon.0) 710 : cluster 3 Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:50:41.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:40 smithi146 bash[23809]: audit 2024-03-11T20:50:39.735277+0000 mon.a (mon.0) 711 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-11T20:50:41.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:40 smithi146 bash[23809]: audit 2024-03-11T20:50:39.738233+0000 mon.b (mon.1) 26 : audit 1 from='client.? 172.21.15.146:0/3605892022' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-11T20:50:42.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:41 smithi146 bash[23809]: audit 2024-03-11T20:50:40.571895+0000 mon.a (mon.0) 712 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-03-11T20:50:42.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:41 smithi146 bash[23809]: cluster 2024-03-11T20:50:40.579045+0000 mon.a (mon.0) 713 : cluster 0 osdmap e85: 8 total, 8 up, 8 in 2024-03-11T20:50:42.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:41 smithi023 bash[21372]: audit 2024-03-11T20:50:40.571895+0000 mon.a (mon.0) 712 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-03-11T20:50:42.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:41 smithi023 bash[21372]: cluster 2024-03-11T20:50:40.579045+0000 mon.a (mon.0) 713 : cluster 0 osdmap e85: 8 total, 8 up, 8 in 2024-03-11T20:50:42.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:41 smithi023 bash[29146]: audit 2024-03-11T20:50:40.571895+0000 mon.a (mon.0) 712 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-03-11T20:50:42.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:41 smithi023 bash[29146]: cluster 2024-03-11T20:50:40.579045+0000 mon.a (mon.0) 713 : cluster 0 osdmap e85: 8 total, 8 up, 8 in 2024-03-11T20:50:42.877 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:42 smithi146 bash[23809]: cluster 2024-03-11T20:50:41.453575+0000 mgr.y (mgr.14152) 349 : cluster 0 pgmap v355: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 4.9 KiB/s wr, 159 op/s 2024-03-11T20:50:42.877 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:42 smithi146 bash[23809]: cluster 2024-03-11T20:50:41.572746+0000 mon.a (mon.0) 714 : cluster 1 Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-11T20:50:42.877 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:42 smithi146 bash[23809]: cluster 2024-03-11T20:50:41.572838+0000 mon.a (mon.0) 715 : cluster 1 Cluster is now healthy 2024-03-11T20:50:42.877 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:42 smithi146 bash[23809]: cluster 2024-03-11T20:50:41.585459+0000 mon.a (mon.0) 716 : cluster 0 osdmap e86: 8 total, 8 up, 8 in 2024-03-11T20:50:43.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:42 smithi023 bash[21372]: cluster 2024-03-11T20:50:41.453575+0000 mgr.y (mgr.14152) 349 : cluster 0 pgmap v355: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 4.9 KiB/s wr, 159 op/s 2024-03-11T20:50:43.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:42 smithi023 bash[21372]: cluster 2024-03-11T20:50:41.572746+0000 mon.a (mon.0) 714 : cluster 1 Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-11T20:50:43.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:42 smithi023 bash[21372]: cluster 2024-03-11T20:50:41.572838+0000 mon.a (mon.0) 715 : cluster 1 Cluster is now healthy 2024-03-11T20:50:43.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:42 smithi023 bash[21372]: cluster 2024-03-11T20:50:41.585459+0000 mon.a (mon.0) 716 : cluster 0 osdmap e86: 8 total, 8 up, 8 in 2024-03-11T20:50:43.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:42 smithi023 bash[29146]: cluster 2024-03-11T20:50:41.453575+0000 mgr.y (mgr.14152) 349 : cluster 0 pgmap v355: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 4.9 KiB/s wr, 159 op/s 2024-03-11T20:50:43.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:42 smithi023 bash[29146]: cluster 2024-03-11T20:50:41.572746+0000 mon.a (mon.0) 714 : cluster 1 Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-11T20:50:43.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:42 smithi023 bash[29146]: cluster 2024-03-11T20:50:41.572838+0000 mon.a (mon.0) 715 : cluster 1 Cluster is now healthy 2024-03-11T20:50:43.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:42 smithi023 bash[29146]: cluster 2024-03-11T20:50:41.585459+0000 mon.a (mon.0) 716 : cluster 0 osdmap e86: 8 total, 8 up, 8 in 2024-03-11T20:50:43.416 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph orch apply iscsi datapool admin admin --trusted_ip_list 172.21.15.146 --placement '1;smithi146=iscsi.a' 2024-03-11T20:50:43.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:43 smithi146 bash[23809]: cluster 2024-03-11T20:50:42.625078+0000 mon.a (mon.0) 717 : cluster 0 osdmap e87: 8 total, 8 up, 8 in 2024-03-11T20:50:44.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:43 smithi023 bash[21372]: cluster 2024-03-11T20:50:42.625078+0000 mon.a (mon.0) 717 : cluster 0 osdmap e87: 8 total, 8 up, 8 in 2024-03-11T20:50:44.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:43 smithi023 bash[29146]: cluster 2024-03-11T20:50:42.625078+0000 mon.a (mon.0) 717 : cluster 0 osdmap e87: 8 total, 8 up, 8 in 2024-03-11T20:50:45.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:44 smithi146 bash[23809]: cluster 2024-03-11T20:50:43.454473+0000 mgr.y (mgr.14152) 350 : cluster 0 pgmap v358: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 4.7 KiB/s wr, 161 op/s 2024-03-11T20:50:45.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:44 smithi023 bash[21372]: cluster 2024-03-11T20:50:43.454473+0000 mgr.y (mgr.14152) 350 : cluster 0 pgmap v358: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 4.7 KiB/s wr, 161 op/s 2024-03-11T20:50:45.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:44 smithi023 bash[29146]: cluster 2024-03-11T20:50:43.454473+0000 mgr.y (mgr.14152) 350 : cluster 0 pgmap v358: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 4.7 KiB/s wr, 161 op/s 2024-03-11T20:50:47.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:46 smithi146 bash[23809]: cluster 2024-03-11T20:50:45.455767+0000 mgr.y (mgr.14152) 351 : cluster 0 pgmap v359: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 2.2 KiB/s wr, 73 op/s 2024-03-11T20:50:47.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:46 smithi023 bash[21372]: cluster 2024-03-11T20:50:45.455767+0000 mgr.y (mgr.14152) 351 : cluster 0 pgmap v359: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 2.2 KiB/s wr, 73 op/s 2024-03-11T20:50:47.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:46 smithi023 bash[29146]: cluster 2024-03-11T20:50:45.455767+0000 mgr.y (mgr.14152) 351 : cluster 0 pgmap v359: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 2.2 KiB/s wr, 73 op/s 2024-03-11T20:50:48.169 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T20:50:49.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:48 smithi146 bash[23809]: cluster 2024-03-11T20:50:47.456689+0000 mgr.y (mgr.14152) 352 : cluster 0 pgmap v360: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 297 B/s wr, 0 op/s 2024-03-11T20:50:49.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:48 smithi023 bash[21372]: cluster 2024-03-11T20:50:47.456689+0000 mgr.y (mgr.14152) 352 : cluster 0 pgmap v360: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 297 B/s wr, 0 op/s 2024-03-11T20:50:49.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:48 smithi023 bash[29146]: cluster 2024-03-11T20:50:47.456689+0000 mgr.y (mgr.14152) 352 : cluster 0 pgmap v360: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 297 B/s wr, 0 op/s 2024-03-11T20:50:49.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:49 smithi146 bash[23809]: cluster 2024-03-11T20:50:48.822019+0000 osd.1 (osd.1) 13 : cluster 0 purged_snaps scrub starts 2024-03-11T20:50:49.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:49 smithi146 bash[23809]: cluster 2024-03-11T20:50:48.822137+0000 osd.1 (osd.1) 14 : cluster 0 purged_snaps scrub ok 2024-03-11T20:50:49.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:49 smithi146 bash[23809]: cluster 2024-03-11T20:50:49.579486+0000 mon.a (mon.0) 718 : cluster 0 osdmap e88: 8 total, 8 up, 8 in 2024-03-11T20:50:50.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:49 smithi023 bash[21372]: cluster 2024-03-11T20:50:48.822019+0000 osd.1 (osd.1) 13 : cluster 0 purged_snaps scrub starts 2024-03-11T20:50:50.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:49 smithi023 bash[21372]: cluster 2024-03-11T20:50:48.822137+0000 osd.1 (osd.1) 14 : cluster 0 purged_snaps scrub ok 2024-03-11T20:50:50.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:49 smithi023 bash[21372]: cluster 2024-03-11T20:50:49.579486+0000 mon.a (mon.0) 718 : cluster 0 osdmap e88: 8 total, 8 up, 8 in 2024-03-11T20:50:50.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:49 smithi023 bash[29146]: cluster 2024-03-11T20:50:48.822019+0000 osd.1 (osd.1) 13 : cluster 0 purged_snaps scrub starts 2024-03-11T20:50:50.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:49 smithi023 bash[29146]: cluster 2024-03-11T20:50:48.822137+0000 osd.1 (osd.1) 14 : cluster 0 purged_snaps scrub ok 2024-03-11T20:50:50.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:49 smithi023 bash[29146]: cluster 2024-03-11T20:50:49.579486+0000 mon.a (mon.0) 718 : cluster 0 osdmap e88: 8 total, 8 up, 8 in 2024-03-11T20:50:51.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:50 smithi146 bash[23809]: cluster 2024-03-11T20:50:49.457769+0000 mgr.y (mgr.14152) 353 : cluster 0 pgmap v361: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-03-11T20:50:51.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:50 smithi023 bash[21372]: cluster 2024-03-11T20:50:49.457769+0000 mgr.y (mgr.14152) 353 : cluster 0 pgmap v361: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-03-11T20:50:51.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:50 smithi023 bash[29146]: cluster 2024-03-11T20:50:49.457769+0000 mgr.y (mgr.14152) 353 : cluster 0 pgmap v361: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-03-11T20:50:51.285 INFO:teuthology.orchestra.run.smithi146.stdout:Scheduled iscsi.datapool update... 2024-03-11T20:50:51.933 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:51 smithi146 bash[23809]: cluster 2024-03-11T20:50:50.472493+0000 osd.7 (osd.7) 5 : cluster 0 purged_snaps scrub starts 2024-03-11T20:50:51.933 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:51 smithi146 bash[23809]: cluster 2024-03-11T20:50:50.472590+0000 osd.7 (osd.7) 6 : cluster 0 purged_snaps scrub ok 2024-03-11T20:50:51.933 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:51 smithi146 bash[23809]: cluster 2024-03-11T20:50:50.690097+0000 mon.a (mon.0) 719 : cluster 0 osdmap e89: 8 total, 8 up, 8 in 2024-03-11T20:50:51.933 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:51 smithi146 bash[23809]: audit 2024-03-11T20:50:51.289074+0000 mon.a (mon.0) 720 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:50:51.933 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:51 smithi146 bash[23809]: audit 2024-03-11T20:50:51.327360+0000 mon.a (mon.0) 721 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:50:51.934 INFO:tasks.cephadm:Distributing iscsi-gateway.cfg... 2024-03-11T20:50:51.935 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-11T20:50:51.935 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-03-11T20:50:51.945 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:51 smithi023 bash[21372]: cluster 2024-03-11T20:50:50.472493+0000 osd.7 (osd.7) 5 : cluster 0 purged_snaps scrub starts 2024-03-11T20:50:51.945 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:51 smithi023 bash[21372]: cluster 2024-03-11T20:50:50.472590+0000 osd.7 (osd.7) 6 : cluster 0 purged_snaps scrub ok 2024-03-11T20:50:51.945 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:51 smithi023 bash[21372]: cluster 2024-03-11T20:50:50.690097+0000 mon.a (mon.0) 719 : cluster 0 osdmap e89: 8 total, 8 up, 8 in 2024-03-11T20:50:51.945 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:51 smithi023 bash[21372]: audit 2024-03-11T20:50:51.289074+0000 mon.a (mon.0) 720 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:50:51.946 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:51 smithi023 bash[21372]: audit 2024-03-11T20:50:51.327360+0000 mon.a (mon.0) 721 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:50:51.946 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:51 smithi023 bash[29146]: cluster 2024-03-11T20:50:50.472493+0000 osd.7 (osd.7) 5 : cluster 0 purged_snaps scrub starts 2024-03-11T20:50:51.946 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:51 smithi023 bash[29146]: cluster 2024-03-11T20:50:50.472590+0000 osd.7 (osd.7) 6 : cluster 0 purged_snaps scrub ok 2024-03-11T20:50:51.946 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:51 smithi023 bash[29146]: cluster 2024-03-11T20:50:50.690097+0000 mon.a (mon.0) 719 : cluster 0 osdmap e89: 8 total, 8 up, 8 in 2024-03-11T20:50:51.946 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:51 smithi023 bash[29146]: audit 2024-03-11T20:50:51.289074+0000 mon.a (mon.0) 720 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:50:51.946 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:51 smithi023 bash[29146]: audit 2024-03-11T20:50:51.327360+0000 mon.a (mon.0) 721 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:50:51.948 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-11T20:50:51.948 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-03-11T20:50:51.960 DEBUG:teuthology.orchestra.run.smithi146:iscsi.iscsi.a> sudo journalctl -f -n 0 -u ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@iscsi.iscsi.a.service 2024-03-11T20:50:52.005 INFO:tasks.cephadm:Adding prometheus.a on smithi146 2024-03-11T20:50:52.006 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph orch apply prometheus '1;smithi146=a' 2024-03-11T20:50:52.764 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:52 smithi146 bash[23809]: audit 2024-03-11T20:50:51.266481+0000 mgr.y (mgr.14152) 354 : audit 0 from='client.24382 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.146", "placement": "1;smithi146=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:50:52.764 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:52 smithi146 bash[23809]: cephadm 2024-03-11T20:50:51.272883+0000 mgr.y (mgr.14152) 355 : cephadm 1 Saving service iscsi.datapool spec with placement smithi146=iscsi.a;count:1 2024-03-11T20:50:52.764 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:52 smithi146 bash[23809]: cluster 2024-03-11T20:50:51.458481+0000 mgr.y (mgr.14152) 356 : cluster 0 pgmap v364: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-03-11T20:50:52.764 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:52 smithi146 bash[23809]: cluster 2024-03-11T20:50:51.776509+0000 osd.2 (osd.2) 9 : cluster 0 purged_snaps scrub starts 2024-03-11T20:50:52.764 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:52 smithi146 bash[23809]: cluster 2024-03-11T20:50:51.776609+0000 osd.2 (osd.2) 10 : cluster 0 purged_snaps scrub ok 2024-03-11T20:50:52.764 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:52 smithi146 bash[23809]: audit 2024-03-11T20:50:52.056408+0000 mon.a (mon.0) 722 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:50:52.764 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:52 smithi146 bash[23809]: audit 2024-03-11T20:50:52.057838+0000 mon.a (mon.0) 723 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:50:52.764 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:52 smithi146 bash[23809]: audit 2024-03-11T20:50:52.069676+0000 mon.a (mon.0) 724 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:50:52.764 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:52 smithi146 bash[23809]: audit 2024-03-11T20:50:52.073511+0000 mon.a (mon.0) 725 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-11T20:50:52.764 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:52 smithi146 bash[23809]: audit 2024-03-11T20:50:52.075121+0000 mon.a (mon.0) 726 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-11T20:50:52.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:52 smithi146 bash[23809]: audit 2024-03-11T20:50:52.089027+0000 mon.a (mon.0) 727 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:50:52.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:52 smithi146 bash[23809]: cephadm 2024-03-11T20:50:52.090362+0000 mgr.y (mgr.14152) 357 : cephadm 1 Deploying daemon iscsi.iscsi.a on smithi146 2024-03-11T20:50:53.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:52 smithi023 bash[29146]: audit 2024-03-11T20:50:51.266481+0000 mgr.y (mgr.14152) 354 : audit 0 from='client.24382 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.146", "placement": "1;smithi146=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:50:53.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:52 smithi023 bash[29146]: cephadm 2024-03-11T20:50:51.272883+0000 mgr.y (mgr.14152) 355 : cephadm 1 Saving service iscsi.datapool spec with placement smithi146=iscsi.a;count:1 2024-03-11T20:50:53.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:52 smithi023 bash[29146]: cluster 2024-03-11T20:50:51.458481+0000 mgr.y (mgr.14152) 356 : cluster 0 pgmap v364: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-03-11T20:50:53.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:52 smithi023 bash[29146]: cluster 2024-03-11T20:50:51.776509+0000 osd.2 (osd.2) 9 : cluster 0 purged_snaps scrub starts 2024-03-11T20:50:53.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:52 smithi023 bash[29146]: cluster 2024-03-11T20:50:51.776609+0000 osd.2 (osd.2) 10 : cluster 0 purged_snaps scrub ok 2024-03-11T20:50:53.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:52 smithi023 bash[29146]: audit 2024-03-11T20:50:52.056408+0000 mon.a (mon.0) 722 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:50:53.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:52 smithi023 bash[29146]: audit 2024-03-11T20:50:52.057838+0000 mon.a (mon.0) 723 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:50:53.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:52 smithi023 bash[29146]: audit 2024-03-11T20:50:52.069676+0000 mon.a (mon.0) 724 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:50:53.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:52 smithi023 bash[29146]: audit 2024-03-11T20:50:52.073511+0000 mon.a (mon.0) 725 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-11T20:50:53.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:52 smithi023 bash[29146]: audit 2024-03-11T20:50:52.075121+0000 mon.a (mon.0) 726 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-11T20:50:53.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:52 smithi023 bash[29146]: audit 2024-03-11T20:50:52.089027+0000 mon.a (mon.0) 727 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:50:53.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:52 smithi023 bash[29146]: cephadm 2024-03-11T20:50:52.090362+0000 mgr.y (mgr.14152) 357 : cephadm 1 Deploying daemon iscsi.iscsi.a on smithi146 2024-03-11T20:50:53.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:52 smithi023 bash[21372]: audit 2024-03-11T20:50:51.266481+0000 mgr.y (mgr.14152) 354 : audit 0 from='client.24382 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.146", "placement": "1;smithi146=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:50:53.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:52 smithi023 bash[21372]: cephadm 2024-03-11T20:50:51.272883+0000 mgr.y (mgr.14152) 355 : cephadm 1 Saving service iscsi.datapool spec with placement smithi146=iscsi.a;count:1 2024-03-11T20:50:53.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:52 smithi023 bash[21372]: cluster 2024-03-11T20:50:51.458481+0000 mgr.y (mgr.14152) 356 : cluster 0 pgmap v364: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-03-11T20:50:53.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:52 smithi023 bash[21372]: cluster 2024-03-11T20:50:51.776509+0000 osd.2 (osd.2) 9 : cluster 0 purged_snaps scrub starts 2024-03-11T20:50:53.036 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:52 smithi023 bash[21372]: cluster 2024-03-11T20:50:51.776609+0000 osd.2 (osd.2) 10 : cluster 0 purged_snaps scrub ok 2024-03-11T20:50:53.036 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:52 smithi023 bash[21372]: audit 2024-03-11T20:50:52.056408+0000 mon.a (mon.0) 722 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:50:53.036 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:52 smithi023 bash[21372]: audit 2024-03-11T20:50:52.057838+0000 mon.a (mon.0) 723 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:50:53.036 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:52 smithi023 bash[21372]: audit 2024-03-11T20:50:52.069676+0000 mon.a (mon.0) 724 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:50:53.036 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:52 smithi023 bash[21372]: audit 2024-03-11T20:50:52.073511+0000 mon.a (mon.0) 725 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-11T20:50:53.036 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:52 smithi023 bash[21372]: audit 2024-03-11T20:50:52.075121+0000 mon.a (mon.0) 726 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-11T20:50:53.036 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:52 smithi023 bash[21372]: audit 2024-03-11T20:50:52.089027+0000 mon.a (mon.0) 727 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:50:53.036 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:52 smithi023 bash[21372]: cephadm 2024-03-11T20:50:52.090362+0000 mgr.y (mgr.14152) 357 : cephadm 1 Deploying daemon iscsi.iscsi.a on smithi146 2024-03-11T20:50:53.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:53 smithi146 bash[23809]: cluster 2024-03-11T20:50:52.708449+0000 mon.a (mon.0) 728 : cluster 0 osdmap e90: 8 total, 8 up, 8 in 2024-03-11T20:50:54.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:53 smithi023 bash[21372]: cluster 2024-03-11T20:50:52.708449+0000 mon.a (mon.0) 728 : cluster 0 osdmap e90: 8 total, 8 up, 8 in 2024-03-11T20:50:54.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:53 smithi023 bash[29146]: cluster 2024-03-11T20:50:52.708449+0000 mon.a (mon.0) 728 : cluster 0 osdmap e90: 8 total, 8 up, 8 in 2024-03-11T20:50:54.760 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Mar 11 20:50:54 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:50:54.760 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 11 20:50:54 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:50:54.761 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 11 20:50:54 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:50:54.761 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 11 20:50:54 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:50:54.761 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 11 20:50:54 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:50:54.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:54 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:50:54.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:54 smithi146 bash[23809]: cluster 2024-03-11T20:50:53.459390+0000 mgr.y (mgr.14152) 358 : cluster 0 pgmap v366: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:50:54.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:54 smithi146 bash[23809]: cluster 2024-03-11T20:50:54.587566+0000 mon.a (mon.0) 729 : cluster 0 osdmap e91: 8 total, 8 up, 8 in 2024-03-11T20:50:55.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:54 smithi023 bash[21372]: cluster 2024-03-11T20:50:53.459390+0000 mgr.y (mgr.14152) 358 : cluster 0 pgmap v366: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:50:55.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:54 smithi023 bash[21372]: cluster 2024-03-11T20:50:54.587566+0000 mon.a (mon.0) 729 : cluster 0 osdmap e91: 8 total, 8 up, 8 in 2024-03-11T20:50:55.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:54 smithi023 bash[29146]: cluster 2024-03-11T20:50:53.459390+0000 mgr.y (mgr.14152) 358 : cluster 0 pgmap v366: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:50:55.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:54 smithi023 bash[29146]: cluster 2024-03-11T20:50:54.587566+0000 mon.a (mon.0) 729 : cluster 0 osdmap e91: 8 total, 8 up, 8 in 2024-03-11T20:50:55.061 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Mar 11 20:50:54 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:50:55.062 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 11 20:50:54 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:50:55.062 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:54 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:50:55.062 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 11 20:50:54 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:50:55.062 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 11 20:50:54 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:50:55.063 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 11 20:50:54 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:50:56.367 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:56 smithi146 bash[23809]: audit 2024-03-11T20:50:55.122625+0000 mon.a (mon.0) 730 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:50:56.368 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:56 smithi146 bash[23809]: audit 2024-03-11T20:50:55.133546+0000 mon.a (mon.0) 731 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:50:56.368 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:56 smithi146 bash[23809]: audit 2024-03-11T20:50:55.170386+0000 mon.a (mon.0) 732 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:50:56.368 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:56 smithi146 bash[23809]: cephadm 2024-03-11T20:50:55.172106+0000 mgr.y (mgr.14152) 359 : cephadm 1 Checking pool "datapool" exists for service iscsi.datapool 2024-03-11T20:50:56.368 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:56 smithi146 bash[23809]: audit 2024-03-11T20:50:55.187646+0000 mon.a (mon.0) 733 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:50:56.368 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:56 smithi146 bash[23809]: audit 2024-03-11T20:50:55.252872+0000 mon.a (mon.0) 734 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:50:56.368 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:56 smithi146 bash[23809]: cluster 2024-03-11T20:50:55.460322+0000 mgr.y (mgr.14152) 360 : cluster 0 pgmap v368: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:50:56.368 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:56 smithi146 bash[23809]: audit 2024-03-11T20:50:55.555989+0000 mon.a (mon.0) 735 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:50:56.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:56 smithi023 bash[21372]: audit 2024-03-11T20:50:55.122625+0000 mon.a (mon.0) 730 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:50:56.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:56 smithi023 bash[21372]: audit 2024-03-11T20:50:55.133546+0000 mon.a (mon.0) 731 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:50:56.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:56 smithi023 bash[21372]: audit 2024-03-11T20:50:55.170386+0000 mon.a (mon.0) 732 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:50:56.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:56 smithi023 bash[21372]: cephadm 2024-03-11T20:50:55.172106+0000 mgr.y (mgr.14152) 359 : cephadm 1 Checking pool "datapool" exists for service iscsi.datapool 2024-03-11T20:50:56.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:56 smithi023 bash[21372]: audit 2024-03-11T20:50:55.187646+0000 mon.a (mon.0) 733 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:50:56.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:56 smithi023 bash[21372]: audit 2024-03-11T20:50:55.252872+0000 mon.a (mon.0) 734 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:50:56.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:56 smithi023 bash[21372]: cluster 2024-03-11T20:50:55.460322+0000 mgr.y (mgr.14152) 360 : cluster 0 pgmap v368: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:50:56.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:56 smithi023 bash[21372]: audit 2024-03-11T20:50:55.555989+0000 mon.a (mon.0) 735 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:50:56.537 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:56 smithi023 bash[29146]: audit 2024-03-11T20:50:55.122625+0000 mon.a (mon.0) 730 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:50:56.537 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:56 smithi023 bash[29146]: audit 2024-03-11T20:50:55.133546+0000 mon.a (mon.0) 731 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:50:56.537 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:56 smithi023 bash[29146]: audit 2024-03-11T20:50:55.170386+0000 mon.a (mon.0) 732 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:50:56.538 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:56 smithi023 bash[29146]: cephadm 2024-03-11T20:50:55.172106+0000 mgr.y (mgr.14152) 359 : cephadm 1 Checking pool "datapool" exists for service iscsi.datapool 2024-03-11T20:50:56.538 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:56 smithi023 bash[29146]: audit 2024-03-11T20:50:55.187646+0000 mon.a (mon.0) 733 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:50:56.538 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:56 smithi023 bash[29146]: audit 2024-03-11T20:50:55.252872+0000 mon.a (mon.0) 734 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:50:56.538 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:56 smithi023 bash[29146]: cluster 2024-03-11T20:50:55.460322+0000 mgr.y (mgr.14152) 360 : cluster 0 pgmap v368: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:50:56.538 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:56 smithi023 bash[29146]: audit 2024-03-11T20:50:55.555989+0000 mon.a (mon.0) 735 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:50:56.791 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T20:50:57.010 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 20:50:56 smithi146 bash[49028]: debug Processing osd blocklist entries for this node 2024-03-11T20:50:57.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:57 smithi146 bash[23809]: audit 2024-03-11T20:50:57.026511+0000 mon.b (mon.1) 27 : audit 0 from='client.? 172.21.15.146:0/545773228' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-11T20:50:57.510 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 20:50:57 smithi146 bash[49028]: debug Reading the configuration object to update local LIO configuration 2024-03-11T20:50:57.511 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 20:50:57 smithi146 bash[49028]: debug Configuration does not have an entry for this host(smithi146.front.sepia.ceph.com) - nothing to define to LIO 2024-03-11T20:50:57.511 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 20:50:57 smithi146 bash[49028]: debug * Running on http://[::]:5000/ (Press CTRL+C to quit) 2024-03-11T20:50:57.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:57 smithi023 bash[21372]: audit 2024-03-11T20:50:57.026511+0000 mon.b (mon.1) 27 : audit 0 from='client.? 172.21.15.146:0/545773228' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-11T20:50:57.534 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:57 smithi023 bash[29146]: audit 2024-03-11T20:50:57.026511+0000 mon.b (mon.1) 27 : audit 0 from='client.? 172.21.15.146:0/545773228' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-11T20:50:58.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:58 smithi146 bash[23809]: cluster 2024-03-11T20:50:57.208677+0000 mon.a (mon.0) 736 : cluster 0 mgrmap e16: y(active, since 9m), standbys: x 2024-03-11T20:50:58.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:50:58 smithi146 bash[23809]: cluster 2024-03-11T20:50:57.461213+0000 mgr.y (mgr.14152) 361 : cluster 0 pgmap v369: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:50:58.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:58 smithi023 bash[21372]: cluster 2024-03-11T20:50:57.208677+0000 mon.a (mon.0) 736 : cluster 0 mgrmap e16: y(active, since 9m), standbys: x 2024-03-11T20:50:58.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:50:58 smithi023 bash[21372]: cluster 2024-03-11T20:50:57.461213+0000 mgr.y (mgr.14152) 361 : cluster 0 pgmap v369: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:50:58.534 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:58 smithi023 bash[29146]: cluster 2024-03-11T20:50:57.208677+0000 mon.a (mon.0) 736 : cluster 0 mgrmap e16: y(active, since 9m), standbys: x 2024-03-11T20:50:58.534 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:50:58 smithi023 bash[29146]: cluster 2024-03-11T20:50:57.461213+0000 mgr.y (mgr.14152) 361 : cluster 0 pgmap v369: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:50:59.963 INFO:teuthology.orchestra.run.smithi146.stdout:Scheduled prometheus update... 2024-03-11T20:51:00.745 DEBUG:teuthology.orchestra.run.smithi146:prometheus.a> sudo journalctl -f -n 0 -u ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@prometheus.a.service 2024-03-11T20:51:00.747 INFO:tasks.cephadm:Adding node-exporter.a on smithi023 2024-03-11T20:51:00.747 INFO:tasks.cephadm:Adding node-exporter.b on smithi146 2024-03-11T20:51:00.747 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph orch apply node-exporter '2;smithi023=a;smithi146=b' 2024-03-11T20:51:01.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:00 smithi146 bash[23809]: cluster 2024-03-11T20:50:59.462248+0000 mgr.y (mgr.14152) 362 : cluster 0 pgmap v370: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 127 B/s wr, 1 op/s 2024-03-11T20:51:01.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:00 smithi146 bash[23809]: cluster 2024-03-11T20:50:59.580637+0000 mon.a (mon.0) 737 : cluster 0 osdmap e92: 8 total, 8 up, 8 in 2024-03-11T20:51:01.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:00 smithi146 bash[23809]: audit 2024-03-11T20:50:59.952868+0000 mgr.y (mgr.14152) 363 : audit 0 from='client.24418 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi146=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:51:01.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:00 smithi146 bash[23809]: cephadm 2024-03-11T20:50:59.956559+0000 mgr.y (mgr.14152) 364 : cephadm 1 Saving service prometheus spec with placement smithi146=a;count:1 2024-03-11T20:51:01.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:00 smithi146 bash[23809]: audit 2024-03-11T20:50:59.966368+0000 mon.a (mon.0) 738 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:51:01.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:00 smithi023 bash[21372]: cluster 2024-03-11T20:50:59.462248+0000 mgr.y (mgr.14152) 362 : cluster 0 pgmap v370: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 127 B/s wr, 1 op/s 2024-03-11T20:51:01.074 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:00 smithi023 bash[21372]: cluster 2024-03-11T20:50:59.580637+0000 mon.a (mon.0) 737 : cluster 0 osdmap e92: 8 total, 8 up, 8 in 2024-03-11T20:51:01.074 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:00 smithi023 bash[21372]: audit 2024-03-11T20:50:59.952868+0000 mgr.y (mgr.14152) 363 : audit 0 from='client.24418 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi146=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:51:01.074 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:00 smithi023 bash[21372]: cephadm 2024-03-11T20:50:59.956559+0000 mgr.y (mgr.14152) 364 : cephadm 1 Saving service prometheus spec with placement smithi146=a;count:1 2024-03-11T20:51:01.074 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:00 smithi023 bash[21372]: audit 2024-03-11T20:50:59.966368+0000 mon.a (mon.0) 738 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:51:01.075 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:00 smithi023 bash[29146]: cluster 2024-03-11T20:50:59.462248+0000 mgr.y (mgr.14152) 362 : cluster 0 pgmap v370: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 127 B/s wr, 1 op/s 2024-03-11T20:51:01.075 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:00 smithi023 bash[29146]: cluster 2024-03-11T20:50:59.580637+0000 mon.a (mon.0) 737 : cluster 0 osdmap e92: 8 total, 8 up, 8 in 2024-03-11T20:51:01.075 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:00 smithi023 bash[29146]: audit 2024-03-11T20:50:59.952868+0000 mgr.y (mgr.14152) 363 : audit 0 from='client.24418 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi146=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:51:01.075 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:00 smithi023 bash[29146]: cephadm 2024-03-11T20:50:59.956559+0000 mgr.y (mgr.14152) 364 : cephadm 1 Saving service prometheus spec with placement smithi146=a;count:1 2024-03-11T20:51:01.075 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:00 smithi023 bash[29146]: audit 2024-03-11T20:50:59.966368+0000 mon.a (mon.0) 738 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:51:02.798 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Mar 11 20:51:02 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:51:03.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:02 smithi146 bash[23809]: cluster 2024-03-11T20:51:01.463245+0000 mgr.y (mgr.14152) 365 : cluster 0 pgmap v372: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 127 B/s wr, 1 op/s 2024-03-11T20:51:03.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:02 smithi146 bash[23809]: audit 2024-03-11T20:51:01.802872+0000 mon.a (mon.0) 739 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:51:03.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:02 smithi146 bash[23809]: audit 2024-03-11T20:51:01.816526+0000 mon.a (mon.0) 740 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:51:03.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:02 smithi146 bash[23809]: audit 2024-03-11T20:51:01.818997+0000 mon.a (mon.0) 741 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:51:03.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:02 smithi146 bash[23809]: audit 2024-03-11T20:51:01.821041+0000 mon.a (mon.0) 742 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:51:03.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:02 smithi146 bash[23809]: audit 2024-03-11T20:51:01.832014+0000 mon.a (mon.0) 743 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:51:03.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:02 smithi023 bash[21372]: cluster 2024-03-11T20:51:01.463245+0000 mgr.y (mgr.14152) 365 : cluster 0 pgmap v372: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 127 B/s wr, 1 op/s 2024-03-11T20:51:03.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:02 smithi023 bash[21372]: audit 2024-03-11T20:51:01.802872+0000 mon.a (mon.0) 739 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:51:03.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:02 smithi023 bash[21372]: audit 2024-03-11T20:51:01.816526+0000 mon.a (mon.0) 740 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:51:03.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:02 smithi023 bash[21372]: audit 2024-03-11T20:51:01.818997+0000 mon.a (mon.0) 741 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:51:03.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:02 smithi023 bash[21372]: audit 2024-03-11T20:51:01.821041+0000 mon.a (mon.0) 742 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:51:03.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:02 smithi023 bash[21372]: audit 2024-03-11T20:51:01.832014+0000 mon.a (mon.0) 743 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:51:03.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:02 smithi023 bash[29146]: cluster 2024-03-11T20:51:01.463245+0000 mgr.y (mgr.14152) 365 : cluster 0 pgmap v372: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 127 B/s wr, 1 op/s 2024-03-11T20:51:03.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:02 smithi023 bash[29146]: audit 2024-03-11T20:51:01.802872+0000 mon.a (mon.0) 739 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:51:03.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:02 smithi023 bash[29146]: audit 2024-03-11T20:51:01.816526+0000 mon.a (mon.0) 740 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:51:03.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:02 smithi023 bash[29146]: audit 2024-03-11T20:51:01.818997+0000 mon.a (mon.0) 741 : audit 0 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:51:03.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:02 smithi023 bash[29146]: audit 2024-03-11T20:51:01.821041+0000 mon.a (mon.0) 742 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:51:03.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:02 smithi023 bash[29146]: audit 2024-03-11T20:51:01.832014+0000 mon.a (mon.0) 743 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:51:04.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:03 smithi146 bash[23809]: cephadm 2024-03-11T20:51:02.153250+0000 mgr.y (mgr.14152) 366 : cephadm 1 Deploying daemon prometheus.a on smithi146 2024-03-11T20:51:04.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:03 smithi146 bash[23809]: cluster 2024-03-11T20:51:02.765484+0000 osd.6 (osd.6) 5 : cluster 0 purged_snaps scrub starts 2024-03-11T20:51:04.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:03 smithi146 bash[23809]: cluster 2024-03-11T20:51:02.765602+0000 osd.6 (osd.6) 6 : cluster 0 purged_snaps scrub ok 2024-03-11T20:51:04.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:03 smithi146 bash[23809]: cluster 2024-03-11T20:51:02.839719+0000 mon.a (mon.0) 744 : cluster 0 osdmap e93: 8 total, 8 up, 8 in 2024-03-11T20:51:04.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:03 smithi023 bash[21372]: cephadm 2024-03-11T20:51:02.153250+0000 mgr.y (mgr.14152) 366 : cephadm 1 Deploying daemon prometheus.a on smithi146 2024-03-11T20:51:04.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:03 smithi023 bash[21372]: cluster 2024-03-11T20:51:02.765484+0000 osd.6 (osd.6) 5 : cluster 0 purged_snaps scrub starts 2024-03-11T20:51:04.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:03 smithi023 bash[21372]: cluster 2024-03-11T20:51:02.765602+0000 osd.6 (osd.6) 6 : cluster 0 purged_snaps scrub ok 2024-03-11T20:51:04.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:03 smithi023 bash[21372]: cluster 2024-03-11T20:51:02.839719+0000 mon.a (mon.0) 744 : cluster 0 osdmap e93: 8 total, 8 up, 8 in 2024-03-11T20:51:04.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:03 smithi023 bash[29146]: cephadm 2024-03-11T20:51:02.153250+0000 mgr.y (mgr.14152) 366 : cephadm 1 Deploying daemon prometheus.a on smithi146 2024-03-11T20:51:04.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:03 smithi023 bash[29146]: cluster 2024-03-11T20:51:02.765484+0000 osd.6 (osd.6) 5 : cluster 0 purged_snaps scrub starts 2024-03-11T20:51:04.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:03 smithi023 bash[29146]: cluster 2024-03-11T20:51:02.765602+0000 osd.6 (osd.6) 6 : cluster 0 purged_snaps scrub ok 2024-03-11T20:51:04.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:03 smithi023 bash[29146]: cluster 2024-03-11T20:51:02.839719+0000 mon.a (mon.0) 744 : cluster 0 osdmap e93: 8 total, 8 up, 8 in 2024-03-11T20:51:05.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:04 smithi146 bash[23809]: cluster 2024-03-11T20:51:03.464286+0000 mgr.y (mgr.14152) 367 : cluster 0 pgmap v374: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 127 B/s wr, 1 op/s 2024-03-11T20:51:05.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:04 smithi146 bash[23809]: cluster 2024-03-11T20:51:04.585166+0000 mon.a (mon.0) 745 : cluster 0 osdmap e94: 8 total, 8 up, 8 in 2024-03-11T20:51:05.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:04 smithi023 bash[21372]: cluster 2024-03-11T20:51:03.464286+0000 mgr.y (mgr.14152) 367 : cluster 0 pgmap v374: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 127 B/s wr, 1 op/s 2024-03-11T20:51:05.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:04 smithi023 bash[21372]: cluster 2024-03-11T20:51:04.585166+0000 mon.a (mon.0) 745 : cluster 0 osdmap e94: 8 total, 8 up, 8 in 2024-03-11T20:51:05.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:04 smithi023 bash[29146]: cluster 2024-03-11T20:51:03.464286+0000 mgr.y (mgr.14152) 367 : cluster 0 pgmap v374: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 127 B/s wr, 1 op/s 2024-03-11T20:51:05.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:04 smithi023 bash[29146]: cluster 2024-03-11T20:51:04.585166+0000 mon.a (mon.0) 745 : cluster 0 osdmap e94: 8 total, 8 up, 8 in 2024-03-11T20:51:05.293 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T20:51:07.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:06 smithi146 bash[23809]: cluster 2024-03-11T20:51:05.465418+0000 mgr.y (mgr.14152) 368 : cluster 0 pgmap v376: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T20:51:07.010 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 20:51:06 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T20:51:07.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:06 smithi023 bash[21372]: cluster 2024-03-11T20:51:05.465418+0000 mgr.y (mgr.14152) 368 : cluster 0 pgmap v376: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T20:51:07.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:06 smithi023 bash[29146]: cluster 2024-03-11T20:51:05.465418+0000 mgr.y (mgr.14152) 368 : cluster 0 pgmap v376: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T20:51:08.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:07 smithi146 bash[23809]: audit 2024-03-11T20:51:06.709997+0000 mgr.y (mgr.14152) 369 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:51:08.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:07 smithi023 bash[21372]: audit 2024-03-11T20:51:06.709997+0000 mgr.y (mgr.14152) 369 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:51:08.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:07 smithi023 bash[29146]: audit 2024-03-11T20:51:06.709997+0000 mgr.y (mgr.14152) 369 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:51:09.130 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:08 smithi146 bash[23809]: cluster 2024-03-11T20:51:07.466468+0000 mgr.y (mgr.14152) 370 : cluster 0 pgmap v377: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 648 B/s rd, 0 op/s 2024-03-11T20:51:09.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:08 smithi023 bash[21372]: cluster 2024-03-11T20:51:07.466468+0000 mgr.y (mgr.14152) 370 : cluster 0 pgmap v377: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 648 B/s rd, 0 op/s 2024-03-11T20:51:09.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:08 smithi023 bash[29146]: cluster 2024-03-11T20:51:07.466468+0000 mgr.y (mgr.14152) 370 : cluster 0 pgmap v377: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 648 B/s rd, 0 op/s 2024-03-11T20:51:09.532 INFO:teuthology.orchestra.run.smithi146.stdout:Scheduled node-exporter update... 2024-03-11T20:51:10.620 DEBUG:teuthology.orchestra.run.smithi023:node-exporter.a> sudo journalctl -f -n 0 -u ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@node-exporter.a.service 2024-03-11T20:51:10.623 DEBUG:teuthology.orchestra.run.smithi146:node-exporter.b> sudo journalctl -f -n 0 -u ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@node-exporter.b.service 2024-03-11T20:51:10.625 INFO:tasks.cephadm:Adding alertmanager.a on smithi023 2024-03-11T20:51:10.626 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph orch apply alertmanager '1;smithi023=a' 2024-03-11T20:51:10.784 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:10 smithi023 bash[21372]: cluster 2024-03-11T20:51:09.467613+0000 mgr.y (mgr.14152) 371 : cluster 0 pgmap v378: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:51:10.784 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:10 smithi023 bash[21372]: audit 2024-03-11T20:51:09.522076+0000 mgr.y (mgr.14152) 372 : audit 0 from='client.24424 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi023=a;smithi146=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:51:10.784 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:10 smithi023 bash[21372]: cephadm 2024-03-11T20:51:09.525768+0000 mgr.y (mgr.14152) 373 : cephadm 1 Saving service node-exporter spec with placement smithi023=a;smithi146=b;count:2 2024-03-11T20:51:10.784 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:10 smithi023 bash[21372]: audit 2024-03-11T20:51:09.535028+0000 mon.a (mon.0) 746 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:51:10.784 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:10 smithi023 bash[29146]: cluster 2024-03-11T20:51:09.467613+0000 mgr.y (mgr.14152) 371 : cluster 0 pgmap v378: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:51:10.784 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:10 smithi023 bash[29146]: audit 2024-03-11T20:51:09.522076+0000 mgr.y (mgr.14152) 372 : audit 0 from='client.24424 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi023=a;smithi146=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:51:10.784 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:10 smithi023 bash[29146]: cephadm 2024-03-11T20:51:09.525768+0000 mgr.y (mgr.14152) 373 : cephadm 1 Saving service node-exporter spec with placement smithi023=a;smithi146=b;count:2 2024-03-11T20:51:10.784 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:10 smithi023 bash[29146]: audit 2024-03-11T20:51:09.535028+0000 mon.a (mon.0) 746 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:51:11.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:10 smithi146 bash[23809]: cluster 2024-03-11T20:51:09.467613+0000 mgr.y (mgr.14152) 371 : cluster 0 pgmap v378: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:51:11.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:10 smithi146 bash[23809]: audit 2024-03-11T20:51:09.522076+0000 mgr.y (mgr.14152) 372 : audit 0 from='client.24424 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi023=a;smithi146=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:51:11.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:10 smithi146 bash[23809]: cephadm 2024-03-11T20:51:09.525768+0000 mgr.y (mgr.14152) 373 : cephadm 1 Saving service node-exporter spec with placement smithi023=a;smithi146=b;count:2 2024-03-11T20:51:11.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:10 smithi146 bash[23809]: audit 2024-03-11T20:51:09.535028+0000 mon.a (mon.0) 746 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:51:13.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:12 smithi146 bash[23809]: cluster 2024-03-11T20:51:11.468620+0000 mgr.y (mgr.14152) 374 : cluster 0 pgmap v379: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:51:13.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:12 smithi023 bash[21372]: cluster 2024-03-11T20:51:11.468620+0000 mgr.y (mgr.14152) 374 : cluster 0 pgmap v379: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:51:13.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:12 smithi023 bash[29146]: cluster 2024-03-11T20:51:11.468620+0000 mgr.y (mgr.14152) 374 : cluster 0 pgmap v379: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:51:15.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:14 smithi146 bash[23809]: cluster 2024-03-11T20:51:13.170039+0000 osd.0 (osd.0) 13 : cluster 0 purged_snaps scrub starts 2024-03-11T20:51:15.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:14 smithi146 bash[23809]: cluster 2024-03-11T20:51:13.170170+0000 osd.0 (osd.0) 14 : cluster 0 purged_snaps scrub ok 2024-03-11T20:51:15.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:14 smithi146 bash[23809]: cluster 2024-03-11T20:51:13.469675+0000 mgr.y (mgr.14152) 375 : cluster 0 pgmap v380: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:51:15.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:14 smithi146 bash[23809]: cluster 2024-03-11T20:51:13.580463+0000 mon.a (mon.0) 747 : cluster 0 osdmap e95: 8 total, 8 up, 8 in 2024-03-11T20:51:15.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:14 smithi023 bash[21372]: cluster 2024-03-11T20:51:13.170039+0000 osd.0 (osd.0) 13 : cluster 0 purged_snaps scrub starts 2024-03-11T20:51:15.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:14 smithi023 bash[21372]: cluster 2024-03-11T20:51:13.170170+0000 osd.0 (osd.0) 14 : cluster 0 purged_snaps scrub ok 2024-03-11T20:51:15.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:14 smithi023 bash[21372]: cluster 2024-03-11T20:51:13.469675+0000 mgr.y (mgr.14152) 375 : cluster 0 pgmap v380: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:51:15.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:14 smithi023 bash[21372]: cluster 2024-03-11T20:51:13.580463+0000 mon.a (mon.0) 747 : cluster 0 osdmap e95: 8 total, 8 up, 8 in 2024-03-11T20:51:15.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:14 smithi023 bash[29146]: cluster 2024-03-11T20:51:13.170039+0000 osd.0 (osd.0) 13 : cluster 0 purged_snaps scrub starts 2024-03-11T20:51:15.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:14 smithi023 bash[29146]: cluster 2024-03-11T20:51:13.170170+0000 osd.0 (osd.0) 14 : cluster 0 purged_snaps scrub ok 2024-03-11T20:51:15.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:14 smithi023 bash[29146]: cluster 2024-03-11T20:51:13.469675+0000 mgr.y (mgr.14152) 375 : cluster 0 pgmap v380: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:51:15.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:14 smithi023 bash[29146]: cluster 2024-03-11T20:51:13.580463+0000 mon.a (mon.0) 747 : cluster 0 osdmap e95: 8 total, 8 up, 8 in 2024-03-11T20:51:15.420 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T20:51:15.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:15 smithi146 bash[23809]: cluster 2024-03-11T20:51:14.309084+0000 osd.4 (osd.4) 7 : cluster 0 purged_snaps scrub starts 2024-03-11T20:51:15.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:15 smithi146 bash[23809]: cluster 2024-03-11T20:51:14.309208+0000 osd.4 (osd.4) 8 : cluster 0 purged_snaps scrub ok 2024-03-11T20:51:15.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:15 smithi146 bash[23809]: cluster 2024-03-11T20:51:14.593971+0000 mon.a (mon.0) 748 : cluster 0 osdmap e96: 8 total, 8 up, 8 in 2024-03-11T20:51:16.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:15 smithi023 bash[21372]: cluster 2024-03-11T20:51:14.309084+0000 osd.4 (osd.4) 7 : cluster 0 purged_snaps scrub starts 2024-03-11T20:51:16.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:15 smithi023 bash[21372]: cluster 2024-03-11T20:51:14.309208+0000 osd.4 (osd.4) 8 : cluster 0 purged_snaps scrub ok 2024-03-11T20:51:16.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:15 smithi023 bash[21372]: cluster 2024-03-11T20:51:14.593971+0000 mon.a (mon.0) 748 : cluster 0 osdmap e96: 8 total, 8 up, 8 in 2024-03-11T20:51:16.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:15 smithi023 bash[29146]: cluster 2024-03-11T20:51:14.309084+0000 osd.4 (osd.4) 7 : cluster 0 purged_snaps scrub starts 2024-03-11T20:51:16.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:15 smithi023 bash[29146]: cluster 2024-03-11T20:51:14.309208+0000 osd.4 (osd.4) 8 : cluster 0 purged_snaps scrub ok 2024-03-11T20:51:16.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:15 smithi023 bash[29146]: cluster 2024-03-11T20:51:14.593971+0000 mon.a (mon.0) 748 : cluster 0 osdmap e96: 8 total, 8 up, 8 in 2024-03-11T20:51:17.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:16 smithi146 bash[23809]: cluster 2024-03-11T20:51:15.470657+0000 mgr.y (mgr.14152) 376 : cluster 0 pgmap v383: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:51:17.010 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 20:51:16 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T20:51:17.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:16 smithi023 bash[21372]: cluster 2024-03-11T20:51:15.470657+0000 mgr.y (mgr.14152) 376 : cluster 0 pgmap v383: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:51:17.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:16 smithi023 bash[29146]: cluster 2024-03-11T20:51:15.470657+0000 mgr.y (mgr.14152) 376 : cluster 0 pgmap v383: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:51:18.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:17 smithi146 bash[23809]: audit 2024-03-11T20:51:16.721251+0000 mgr.y (mgr.14152) 377 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:51:18.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:17 smithi023 bash[21372]: audit 2024-03-11T20:51:16.721251+0000 mgr.y (mgr.14152) 377 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:51:18.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:17 smithi023 bash[29146]: audit 2024-03-11T20:51:16.721251+0000 mgr.y (mgr.14152) 377 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:51:19.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:18 smithi146 bash[23809]: cluster 2024-03-11T20:51:17.471603+0000 mgr.y (mgr.14152) 378 : cluster 0 pgmap v384: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T20:51:19.033 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:18 smithi023 bash[29146]: cluster 2024-03-11T20:51:17.471603+0000 mgr.y (mgr.14152) 378 : cluster 0 pgmap v384: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T20:51:19.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:18 smithi023 bash[21372]: cluster 2024-03-11T20:51:17.471603+0000 mgr.y (mgr.14152) 378 : cluster 0 pgmap v384: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T20:51:21.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:20 smithi146 bash[23809]: cluster 2024-03-11T20:51:19.472667+0000 mgr.y (mgr.14152) 379 : cluster 0 pgmap v385: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:51:21.033 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:20 smithi023 bash[29146]: cluster 2024-03-11T20:51:19.472667+0000 mgr.y (mgr.14152) 379 : cluster 0 pgmap v385: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:51:21.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:20 smithi023 bash[21372]: cluster 2024-03-11T20:51:19.472667+0000 mgr.y (mgr.14152) 379 : cluster 0 pgmap v385: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:51:22.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:21 smithi146 bash[23809]: cluster 2024-03-11T20:51:20.271375+0000 osd.7 (osd.7) 7 : cluster 0 1.0 scrub starts 2024-03-11T20:51:22.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:21 smithi146 bash[23809]: cluster 2024-03-11T20:51:20.274837+0000 osd.7 (osd.7) 8 : cluster 0 1.0 scrub ok 2024-03-11T20:51:22.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:21 smithi146 bash[23809]: audit 2024-03-11T20:51:21.395443+0000 mon.a (mon.0) 749 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 2]}]: dispatch 2024-03-11T20:51:22.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:21 smithi146 bash[23809]: audit 2024-03-11T20:51:21.396256+0000 mon.a (mon.0) 750 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-03-11T20:51:22.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:21 smithi146 bash[23809]: audit 2024-03-11T20:51:21.396987+0000 mon.a (mon.0) 751 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]: dispatch 2024-03-11T20:51:22.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:21 smithi146 bash[23809]: audit 2024-03-11T20:51:21.397680+0000 mon.a (mon.0) 752 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]: dispatch 2024-03-11T20:51:22.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:21 smithi023 bash[21372]: cluster 2024-03-11T20:51:20.271375+0000 osd.7 (osd.7) 7 : cluster 0 1.0 scrub starts 2024-03-11T20:51:22.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:21 smithi023 bash[21372]: cluster 2024-03-11T20:51:20.274837+0000 osd.7 (osd.7) 8 : cluster 0 1.0 scrub ok 2024-03-11T20:51:22.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:21 smithi023 bash[21372]: audit 2024-03-11T20:51:21.395443+0000 mon.a (mon.0) 749 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 2]}]: dispatch 2024-03-11T20:51:22.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:21 smithi023 bash[21372]: audit 2024-03-11T20:51:21.396256+0000 mon.a (mon.0) 750 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-03-11T20:51:22.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:21 smithi023 bash[21372]: audit 2024-03-11T20:51:21.396987+0000 mon.a (mon.0) 751 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]: dispatch 2024-03-11T20:51:22.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:21 smithi023 bash[21372]: audit 2024-03-11T20:51:21.397680+0000 mon.a (mon.0) 752 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]: dispatch 2024-03-11T20:51:22.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:21 smithi023 bash[29146]: cluster 2024-03-11T20:51:20.271375+0000 osd.7 (osd.7) 7 : cluster 0 1.0 scrub starts 2024-03-11T20:51:22.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:21 smithi023 bash[29146]: cluster 2024-03-11T20:51:20.274837+0000 osd.7 (osd.7) 8 : cluster 0 1.0 scrub ok 2024-03-11T20:51:22.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:21 smithi023 bash[29146]: audit 2024-03-11T20:51:21.395443+0000 mon.a (mon.0) 749 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 2]}]: dispatch 2024-03-11T20:51:22.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:21 smithi023 bash[29146]: audit 2024-03-11T20:51:21.396256+0000 mon.a (mon.0) 750 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-03-11T20:51:22.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:21 smithi023 bash[29146]: audit 2024-03-11T20:51:21.396987+0000 mon.a (mon.0) 751 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]: dispatch 2024-03-11T20:51:22.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:21 smithi023 bash[29146]: audit 2024-03-11T20:51:21.397680+0000 mon.a (mon.0) 752 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]: dispatch 2024-03-11T20:51:22.594 INFO:teuthology.orchestra.run.smithi146.stdout:Scheduled alertmanager update... 2024-03-11T20:51:22.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:22 smithi146 bash[23809]: cluster 2024-03-11T20:51:21.473712+0000 mgr.y (mgr.14152) 380 : cluster 0 pgmap v386: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T20:51:22.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:22 smithi146 bash[23809]: audit 2024-03-11T20:51:21.638695+0000 mon.a (mon.0) 753 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 2]}]': finished 2024-03-11T20:51:22.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:22 smithi146 bash[23809]: audit 2024-03-11T20:51:21.638832+0000 mon.a (mon.0) 754 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]': finished 2024-03-11T20:51:22.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:22 smithi146 bash[23809]: audit 2024-03-11T20:51:21.638940+0000 mon.a (mon.0) 755 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]': finished 2024-03-11T20:51:22.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:22 smithi146 bash[23809]: audit 2024-03-11T20:51:21.639111+0000 mon.a (mon.0) 756 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]': finished 2024-03-11T20:51:22.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:22 smithi146 bash[23809]: cluster 2024-03-11T20:51:21.651783+0000 mon.a (mon.0) 757 : cluster 0 osdmap e97: 8 total, 8 up, 8 in 2024-03-11T20:51:22.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:22 smithi146 bash[23809]: audit 2024-03-11T20:51:22.597440+0000 mon.a (mon.0) 758 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:51:23.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:22 smithi023 bash[21372]: cluster 2024-03-11T20:51:21.473712+0000 mgr.y (mgr.14152) 380 : cluster 0 pgmap v386: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T20:51:23.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:22 smithi023 bash[21372]: audit 2024-03-11T20:51:21.638695+0000 mon.a (mon.0) 753 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 2]}]': finished 2024-03-11T20:51:23.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:22 smithi023 bash[21372]: audit 2024-03-11T20:51:21.638832+0000 mon.a (mon.0) 754 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]': finished 2024-03-11T20:51:23.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:22 smithi023 bash[21372]: audit 2024-03-11T20:51:21.638940+0000 mon.a (mon.0) 755 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]': finished 2024-03-11T20:51:23.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:22 smithi023 bash[21372]: audit 2024-03-11T20:51:21.639111+0000 mon.a (mon.0) 756 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]': finished 2024-03-11T20:51:23.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:22 smithi023 bash[21372]: cluster 2024-03-11T20:51:21.651783+0000 mon.a (mon.0) 757 : cluster 0 osdmap e97: 8 total, 8 up, 8 in 2024-03-11T20:51:23.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:22 smithi023 bash[21372]: audit 2024-03-11T20:51:22.597440+0000 mon.a (mon.0) 758 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:51:23.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:22 smithi023 bash[29146]: cluster 2024-03-11T20:51:21.473712+0000 mgr.y (mgr.14152) 380 : cluster 0 pgmap v386: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T20:51:23.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:22 smithi023 bash[29146]: audit 2024-03-11T20:51:21.638695+0000 mon.a (mon.0) 753 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 2]}]': finished 2024-03-11T20:51:23.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:22 smithi023 bash[29146]: audit 2024-03-11T20:51:21.638832+0000 mon.a (mon.0) 754 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]': finished 2024-03-11T20:51:23.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:22 smithi023 bash[29146]: audit 2024-03-11T20:51:21.638940+0000 mon.a (mon.0) 755 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]': finished 2024-03-11T20:51:23.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:22 smithi023 bash[29146]: audit 2024-03-11T20:51:21.639111+0000 mon.a (mon.0) 756 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]': finished 2024-03-11T20:51:23.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:22 smithi023 bash[29146]: cluster 2024-03-11T20:51:21.651783+0000 mon.a (mon.0) 757 : cluster 0 osdmap e97: 8 total, 8 up, 8 in 2024-03-11T20:51:23.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:22 smithi023 bash[29146]: audit 2024-03-11T20:51:22.597440+0000 mon.a (mon.0) 758 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:51:23.749 DEBUG:teuthology.orchestra.run.smithi023:alertmanager.a> sudo journalctl -f -n 0 -u ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@alertmanager.a.service 2024-03-11T20:51:23.752 INFO:tasks.cephadm:Adding grafana.a on smithi146 2024-03-11T20:51:23.752 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph orch apply grafana '1;smithi146=a' 2024-03-11T20:51:23.972 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Mar 11 20:51:23 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:51:23.972 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 11 20:51:23 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:51:23.972 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 11 20:51:23 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:51:23.973 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 11 20:51:23 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:51:23.973 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 11 20:51:23 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:51:23.973 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 20:51:23 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:51:23.974 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:23 smithi146 bash[23809]: audit 2024-03-11T20:51:22.586571+0000 mgr.y (mgr.14152) 381 : audit 0 from='client.24430 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi023=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:51:23.974 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:23 smithi146 bash[23809]: cephadm 2024-03-11T20:51:22.590081+0000 mgr.y (mgr.14152) 382 : cephadm 1 Saving service alertmanager spec with placement smithi023=a;count:1 2024-03-11T20:51:23.974 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:23 smithi146 bash[23809]: cluster 2024-03-11T20:51:22.648473+0000 mon.a (mon.0) 759 : cluster 0 osdmap e98: 8 total, 8 up, 8 in 2024-03-11T20:51:23.974 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:23 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:51:24.010 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Mar 11 20:51:23 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:51:24.010 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Mar 11 20:51:23 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:51:24.010 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Mar 11 20:51:23 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:51:24.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:23 smithi023 bash[21372]: audit 2024-03-11T20:51:22.586571+0000 mgr.y (mgr.14152) 381 : audit 0 from='client.24430 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi023=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:51:24.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:23 smithi023 bash[21372]: cephadm 2024-03-11T20:51:22.590081+0000 mgr.y (mgr.14152) 382 : cephadm 1 Saving service alertmanager spec with placement smithi023=a;count:1 2024-03-11T20:51:24.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:23 smithi023 bash[21372]: cluster 2024-03-11T20:51:22.648473+0000 mon.a (mon.0) 759 : cluster 0 osdmap e98: 8 total, 8 up, 8 in 2024-03-11T20:51:24.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:23 smithi023 bash[29146]: audit 2024-03-11T20:51:22.586571+0000 mgr.y (mgr.14152) 381 : audit 0 from='client.24430 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi023=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:51:24.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:23 smithi023 bash[29146]: cephadm 2024-03-11T20:51:22.590081+0000 mgr.y (mgr.14152) 382 : cephadm 1 Saving service alertmanager spec with placement smithi023=a;count:1 2024-03-11T20:51:24.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:23 smithi023 bash[29146]: cluster 2024-03-11T20:51:22.648473+0000 mon.a (mon.0) 759 : cluster 0 osdmap e98: 8 total, 8 up, 8 in 2024-03-11T20:51:24.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:24 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:51:24.260 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Mar 11 20:51:24 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:51:24.260 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 11 20:51:24 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:51:24.261 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 20:51:24 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:51:24.261 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 11 20:51:24 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:51:24.261 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Mar 11 20:51:24 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:51:24.261 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Mar 11 20:51:24 smithi146 systemd[1]: Started Ceph prometheus.a for 67ba00f8-dfe7-11ee-95c8-87774f69a715. 2024-03-11T20:51:24.262 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 11 20:51:24 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:51:24.262 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 11 20:51:24 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:51:25.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:24 smithi146 bash[23809]: cluster 2024-03-11T20:51:23.474641+0000 mgr.y (mgr.14152) 383 : cluster 0 pgmap v389: 132 pgs: 3 peering, 129 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-03-11T20:51:25.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:24 smithi146 bash[23809]: cluster 2024-03-11T20:51:23.706444+0000 osd.5 (osd.5) 9 : cluster 0 purged_snaps scrub starts 2024-03-11T20:51:25.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:24 smithi146 bash[23809]: cluster 2024-03-11T20:51:23.706537+0000 osd.5 (osd.5) 10 : cluster 0 purged_snaps scrub ok 2024-03-11T20:51:25.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:24 smithi146 bash[23809]: audit 2024-03-11T20:51:24.279744+0000 mon.a (mon.0) 760 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:51:25.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:24 smithi146 bash[23809]: audit 2024-03-11T20:51:24.295571+0000 mon.a (mon.0) 761 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:51:25.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:24 smithi146 bash[23809]: audit 2024-03-11T20:51:24.343032+0000 mon.a (mon.0) 762 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:51:25.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:24 smithi146 bash[23809]: audit 2024-03-11T20:51:24.351798+0000 mon.a (mon.0) 763 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-11T20:51:25.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:24 smithi023 bash[21372]: cluster 2024-03-11T20:51:23.474641+0000 mgr.y (mgr.14152) 383 : cluster 0 pgmap v389: 132 pgs: 3 peering, 129 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-03-11T20:51:25.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:24 smithi023 bash[21372]: cluster 2024-03-11T20:51:23.706444+0000 osd.5 (osd.5) 9 : cluster 0 purged_snaps scrub starts 2024-03-11T20:51:25.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:24 smithi023 bash[21372]: cluster 2024-03-11T20:51:23.706537+0000 osd.5 (osd.5) 10 : cluster 0 purged_snaps scrub ok 2024-03-11T20:51:25.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:24 smithi023 bash[21372]: audit 2024-03-11T20:51:24.279744+0000 mon.a (mon.0) 760 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:51:25.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:24 smithi023 bash[21372]: audit 2024-03-11T20:51:24.295571+0000 mon.a (mon.0) 761 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:51:25.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:24 smithi023 bash[21372]: audit 2024-03-11T20:51:24.343032+0000 mon.a (mon.0) 762 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:51:25.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:24 smithi023 bash[21372]: audit 2024-03-11T20:51:24.351798+0000 mon.a (mon.0) 763 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-11T20:51:25.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:24 smithi023 bash[29146]: cluster 2024-03-11T20:51:23.474641+0000 mgr.y (mgr.14152) 383 : cluster 0 pgmap v389: 132 pgs: 3 peering, 129 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-03-11T20:51:25.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:24 smithi023 bash[29146]: cluster 2024-03-11T20:51:23.706444+0000 osd.5 (osd.5) 9 : cluster 0 purged_snaps scrub starts 2024-03-11T20:51:25.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:24 smithi023 bash[29146]: cluster 2024-03-11T20:51:23.706537+0000 osd.5 (osd.5) 10 : cluster 0 purged_snaps scrub ok 2024-03-11T20:51:25.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:24 smithi023 bash[29146]: audit 2024-03-11T20:51:24.279744+0000 mon.a (mon.0) 760 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:51:25.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:24 smithi023 bash[29146]: audit 2024-03-11T20:51:24.295571+0000 mon.a (mon.0) 761 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:51:25.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:24 smithi023 bash[29146]: audit 2024-03-11T20:51:24.343032+0000 mon.a (mon.0) 762 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' 2024-03-11T20:51:25.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:24 smithi023 bash[29146]: audit 2024-03-11T20:51:24.351798+0000 mon.a (mon.0) 763 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-11T20:51:25.711 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:51:25 smithi023 bash[21627]: ignoring --setuser ceph since I am not root 2024-03-11T20:51:25.711 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:51:25 smithi023 bash[21627]: ignoring --setgroup ceph since I am not root 2024-03-11T20:51:25.711 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:51:25 smithi023 bash[21627]: debug 2024-03-11T20:51:25.559+0000 7f643c07c200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T20:51:25.712 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Mar 11 20:51:25 smithi146 bash[24737]: ignoring --setuser ceph since I am not root 2024-03-11T20:51:25.712 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Mar 11 20:51:25 smithi146 bash[24737]: ignoring --setgroup ceph since I am not root 2024-03-11T20:51:25.712 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Mar 11 20:51:25 smithi146 bash[24737]: debug 2024-03-11T20:51:25.562+0000 7f8449808200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T20:51:25.712 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Mar 11 20:51:25 smithi146 bash[50312]: ts=2024-03-11T20:51:25.426Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-03-11T20:51:25.713 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Mar 11 20:51:25 smithi146 bash[50312]: ts=2024-03-11T20:51:25.426Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-03-11T20:51:25.713 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Mar 11 20:51:25 smithi146 bash[50312]: ts=2024-03-11T20:51:25.426Z caller=main.go:570 level=info host_details="(Linux 5.15.0-100-generic #110-Ubuntu SMP Wed Feb 7 13:27:48 UTC 2024 x86_64 smithi146 (none))" 2024-03-11T20:51:25.713 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Mar 11 20:51:25 smithi146 bash[50312]: ts=2024-03-11T20:51:25.426Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-11T20:51:25.713 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Mar 11 20:51:25 smithi146 bash[50312]: ts=2024-03-11T20:51:25.426Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-11T20:51:25.713 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Mar 11 20:51:25 smithi146 bash[50312]: ts=2024-03-11T20:51:25.427Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-03-11T20:51:25.713 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Mar 11 20:51:25 smithi146 bash[50312]: ts=2024-03-11T20:51:25.428Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-03-11T20:51:25.713 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Mar 11 20:51:25 smithi146 bash[50312]: ts=2024-03-11T20:51:25.429Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-03-11T20:51:25.713 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Mar 11 20:51:25 smithi146 bash[50312]: ts=2024-03-11T20:51:25.429Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-03-11T20:51:25.713 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Mar 11 20:51:25 smithi146 bash[50312]: ts=2024-03-11T20:51:25.433Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-11T20:51:25.713 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Mar 11 20:51:25 smithi146 bash[50312]: ts=2024-03-11T20:51:25.433Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=2.542µs 2024-03-11T20:51:25.713 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Mar 11 20:51:25 smithi146 bash[50312]: ts=2024-03-11T20:51:25.433Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-11T20:51:25.713 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Mar 11 20:51:25 smithi146 bash[50312]: ts=2024-03-11T20:51:25.435Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2024-03-11T20:51:25.713 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Mar 11 20:51:25 smithi146 bash[50312]: ts=2024-03-11T20:51:25.435Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=61.994µs wal_replay_duration=2.030688ms wbl_replay_duration=153ns total_replay_duration=2.204866ms 2024-03-11T20:51:25.713 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Mar 11 20:51:25 smithi146 bash[50312]: ts=2024-03-11T20:51:25.436Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-03-11T20:51:25.714 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Mar 11 20:51:25 smithi146 bash[50312]: ts=2024-03-11T20:51:25.436Z caller=main.go:1029 level=info msg="TSDB started" 2024-03-11T20:51:25.714 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Mar 11 20:51:25 smithi146 bash[50312]: ts=2024-03-11T20:51:25.436Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-11T20:51:25.714 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Mar 11 20:51:25 smithi146 bash[50312]: ts=2024-03-11T20:51:25.438Z caller=refresh.go:80 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.23:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.23:8765: connect: connection refused" 2024-03-11T20:51:25.714 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Mar 11 20:51:25 smithi146 bash[50312]: ts=2024-03-11T20:51:25.438Z caller=refresh.go:80 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.23:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.23:8765: connect: connection refused" 2024-03-11T20:51:25.714 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Mar 11 20:51:25 smithi146 bash[50312]: ts=2024-03-11T20:51:25.438Z caller=refresh.go:80 level=error component="discovery manager scrape" discovery=http config=nvmeof msg="Unable to refresh target groups" err="Get \"http://172.21.15.23:8765/sd/prometheus/sd-config?service=nvmeof\": dial tcp 172.21.15.23:8765: connect: connection refused" 2024-03-11T20:51:25.714 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Mar 11 20:51:25 smithi146 bash[50312]: ts=2024-03-11T20:51:25.467Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=30.795395ms db_storage=1.177µs remote_storage=1.54µs web_handler=534ns query_engine=598ns scrape=366.081µs scrape_sd=142.738µs notify=1.416µs notify_sd=2.631µs rules=29.65297ms tracing=6.597µs 2024-03-11T20:51:25.714 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Mar 11 20:51:25 smithi146 bash[50312]: ts=2024-03-11T20:51:25.467Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-03-11T20:51:25.714 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Mar 11 20:51:25 smithi146 bash[50312]: ts=2024-03-11T20:51:25.467Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-03-11T20:51:26.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:25 smithi146 bash[23809]: cluster 2024-03-11T20:51:24.714750+0000 mon.a (mon.0) 764 : cluster 0 osdmap e99: 8 total, 8 up, 8 in 2024-03-11T20:51:26.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:25 smithi146 bash[23809]: audit 2024-03-11T20:51:25.343211+0000 mon.a (mon.0) 765 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-11T20:51:26.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:25 smithi146 bash[23809]: cluster 2024-03-11T20:51:25.355998+0000 mon.a (mon.0) 766 : cluster 0 mgrmap e17: y(active, since 10m), standbys: x 2024-03-11T20:51:26.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:25 smithi023 bash[21372]: cluster 2024-03-11T20:51:24.714750+0000 mon.a (mon.0) 764 : cluster 0 osdmap e99: 8 total, 8 up, 8 in 2024-03-11T20:51:26.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:25 smithi023 bash[21372]: audit 2024-03-11T20:51:25.343211+0000 mon.a (mon.0) 765 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-11T20:51:26.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:25 smithi023 bash[21372]: cluster 2024-03-11T20:51:25.355998+0000 mon.a (mon.0) 766 : cluster 0 mgrmap e17: y(active, since 10m), standbys: x 2024-03-11T20:51:26.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:25 smithi023 bash[29146]: cluster 2024-03-11T20:51:24.714750+0000 mon.a (mon.0) 764 : cluster 0 osdmap e99: 8 total, 8 up, 8 in 2024-03-11T20:51:26.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:25 smithi023 bash[29146]: audit 2024-03-11T20:51:25.343211+0000 mon.a (mon.0) 765 : audit 1 from='mgr.14152 172.21.15.23:0/1003770480' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-11T20:51:26.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:25 smithi023 bash[29146]: cluster 2024-03-11T20:51:25.355998+0000 mon.a (mon.0) 766 : cluster 0 mgrmap e17: y(active, since 10m), standbys: x 2024-03-11T20:51:27.472 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Mar 11 20:51:27 smithi146 bash[24737]: debug 2024-03-11T20:51:27.130+0000 7f8449808200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T20:51:27.472 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Mar 11 20:51:27 smithi146 bash[24737]: debug 2024-03-11T20:51:27.230+0000 7f8449808200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T20:51:27.534 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:51:27 smithi023 bash[21627]: debug 2024-03-11T20:51:27.187+0000 7f643c07c200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T20:51:27.534 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:51:27 smithi023 bash[21627]: debug 2024-03-11T20:51:27.291+0000 7f643c07c200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T20:51:27.760 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Mar 11 20:51:27 smithi146 bash[24737]: debug 2024-03-11T20:51:27.474+0000 7f8449808200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T20:51:28.034 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:51:27 smithi023 bash[21627]: debug 2024-03-11T20:51:27.547+0000 7f643c07c200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T20:51:28.760 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Mar 11 20:51:28 smithi146 bash[24737]: debug 2024-03-11T20:51:28.282+0000 7f8449808200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T20:51:28.760 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Mar 11 20:51:28 smithi146 bash[24737]: debug 2024-03-11T20:51:28.390+0000 7f8449808200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T20:51:28.760 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Mar 11 20:51:28 smithi146 bash[24737]: debug 2024-03-11T20:51:28.494+0000 7f8449808200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T20:51:28.783 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:51:28 smithi023 bash[21627]: debug 2024-03-11T20:51:28.379+0000 7f643c07c200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T20:51:28.784 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:51:28 smithi023 bash[21627]: debug 2024-03-11T20:51:28.495+0000 7f643c07c200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T20:51:28.784 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:51:28 smithi023 bash[21627]: debug 2024-03-11T20:51:28.603+0000 7f643c07c200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T20:51:29.260 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Mar 11 20:51:28 smithi146 bash[24737]: debug 2024-03-11T20:51:28.854+0000 7f8449808200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-11T20:51:29.260 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Mar 11 20:51:28 smithi146 bash[24737]: debug 2024-03-11T20:51:28.966+0000 7f8449808200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T20:51:29.260 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Mar 11 20:51:29 smithi146 bash[24737]: debug 2024-03-11T20:51:29.070+0000 7f8449808200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T20:51:29.283 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:51:28 smithi023 bash[21627]: debug 2024-03-11T20:51:28.967+0000 7f643c07c200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-11T20:51:29.284 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:51:29 smithi023 bash[21627]: debug 2024-03-11T20:51:29.087+0000 7f643c07c200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T20:51:29.284 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:51:29 smithi023 bash[21627]: debug 2024-03-11T20:51:29.195+0000 7f643c07c200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T20:51:29.519 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T20:51:29.760 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Mar 11 20:51:29 smithi146 bash[24737]: debug 2024-03-11T20:51:29.426+0000 7f8449808200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T20:51:29.760 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Mar 11 20:51:29 smithi146 bash[24737]: debug 2024-03-11T20:51:29.670+0000 7f8449808200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T20:51:30.033 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:51:29 smithi023 bash[21627]: debug 2024-03-11T20:51:29.567+0000 7f643c07c200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T20:51:30.034 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:51:29 smithi023 bash[21627]: debug 2024-03-11T20:51:29.815+0000 7f643c07c200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T20:51:30.260 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Mar 11 20:51:29 smithi146 bash[24737]: debug 2024-03-11T20:51:29.934+0000 7f8449808200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T20:51:30.260 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Mar 11 20:51:30 smithi146 bash[24737]: debug 2024-03-11T20:51:30.046+0000 7f8449808200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T20:51:30.492 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:51:30 smithi023 bash[21627]: debug 2024-03-11T20:51:30.087+0000 7f643c07c200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T20:51:30.492 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:51:30 smithi023 bash[21627]: debug 2024-03-11T20:51:30.207+0000 7f643c07c200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T20:51:30.589 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Mar 11 20:51:30 smithi146 bash[24737]: debug 2024-03-11T20:51:30.322+0000 7f8449808200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T20:51:30.590 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Mar 11 20:51:30 smithi146 bash[24737]: debug 2024-03-11T20:51:30.570+0000 7f8449808200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T20:51:30.750 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:51:30 smithi023 bash[21627]: debug 2024-03-11T20:51:30.495+0000 7f643c07c200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T20:51:30.751 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:30 smithi023 bash[21372]: cluster 2024-03-11T20:51:28.785583+0000 osd.5 (osd.5) 11 : cluster 0 2.16 scrub starts 2024-03-11T20:51:30.751 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:30 smithi023 bash[21372]: cluster 2024-03-11T20:51:28.787500+0000 osd.5 (osd.5) 12 : cluster 0 2.16 scrub ok 2024-03-11T20:51:30.751 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:30 smithi023 bash[21372]: cluster 2024-03-11T20:51:29.592646+0000 mon.a (mon.0) 767 : cluster 0 osdmap e100: 8 total, 8 up, 8 in 2024-03-11T20:51:30.751 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:30 smithi023 bash[29146]: cluster 2024-03-11T20:51:28.785583+0000 osd.5 (osd.5) 11 : cluster 0 2.16 scrub starts 2024-03-11T20:51:30.751 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:30 smithi023 bash[29146]: cluster 2024-03-11T20:51:28.787500+0000 osd.5 (osd.5) 12 : cluster 0 2.16 scrub ok 2024-03-11T20:51:30.752 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:30 smithi023 bash[29146]: cluster 2024-03-11T20:51:29.592646+0000 mon.a (mon.0) 767 : cluster 0 osdmap e100: 8 total, 8 up, 8 in 2024-03-11T20:51:30.938 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:30 smithi146 bash[23809]: cluster 2024-03-11T20:51:28.785583+0000 osd.5 (osd.5) 11 : cluster 0 2.16 scrub starts 2024-03-11T20:51:30.938 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:30 smithi146 bash[23809]: cluster 2024-03-11T20:51:28.787500+0000 osd.5 (osd.5) 12 : cluster 0 2.16 scrub ok 2024-03-11T20:51:30.938 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:30 smithi146 bash[23809]: cluster 2024-03-11T20:51:29.592646+0000 mon.a (mon.0) 767 : cluster 0 osdmap e100: 8 total, 8 up, 8 in 2024-03-11T20:51:31.034 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:51:30 smithi023 bash[21627]: debug 2024-03-11T20:51:30.751+0000 7f643c07c200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T20:51:31.260 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Mar 11 20:51:30 smithi146 bash[24737]: debug 2024-03-11T20:51:30.990+0000 7f8449808200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T20:51:31.260 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Mar 11 20:51:31 smithi146 bash[24737]: debug 2024-03-11T20:51:31.110+0000 7f8449808200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T20:51:31.534 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:51:31 smithi023 bash[21627]: debug 2024-03-11T20:51:31.171+0000 7f643c07c200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T20:51:31.534 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:51:31 smithi023 bash[21627]: debug 2024-03-11T20:51:31.295+0000 7f643c07c200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T20:51:31.696 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Mar 11 20:51:31 smithi146 bash[24737]: debug 2024-03-11T20:51:31.366+0000 7f8449808200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T20:51:32.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:31 smithi146 bash[23809]: cluster 2024-03-11T20:51:30.844345+0000 osd.5 (osd.5) 13 : cluster 0 2.12 scrub starts 2024-03-11T20:51:32.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:31 smithi146 bash[23809]: cluster 2024-03-11T20:51:30.845817+0000 osd.5 (osd.5) 14 : cluster 0 2.12 scrub ok 2024-03-11T20:51:32.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:31 smithi023 bash[29146]: cluster 2024-03-11T20:51:30.844345+0000 osd.5 (osd.5) 13 : cluster 0 2.12 scrub starts 2024-03-11T20:51:32.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:31 smithi023 bash[29146]: cluster 2024-03-11T20:51:30.845817+0000 osd.5 (osd.5) 14 : cluster 0 2.12 scrub ok 2024-03-11T20:51:32.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:31 smithi023 bash[21372]: cluster 2024-03-11T20:51:30.844345+0000 osd.5 (osd.5) 13 : cluster 0 2.12 scrub starts 2024-03-11T20:51:32.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:31 smithi023 bash[21372]: cluster 2024-03-11T20:51:30.845817+0000 osd.5 (osd.5) 14 : cluster 0 2.12 scrub ok 2024-03-11T20:51:32.034 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:51:31 smithi023 bash[21627]: debug 2024-03-11T20:51:31.555+0000 7f643c07c200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T20:51:32.699 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Mar 11 20:51:32 smithi146 bash[24737]: debug 2024-03-11T20:51:32.390+0000 7f8449808200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T20:51:32.699 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Mar 11 20:51:32 smithi146 bash[24737]: debug 2024-03-11T20:51:32.510+0000 7f8449808200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T20:51:33.010 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Mar 11 20:51:32 smithi146 bash[24737]: debug 2024-03-11T20:51:32.802+0000 7f8449808200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T20:51:33.010 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Mar 11 20:51:32 smithi146 bash[24737]: debug 2024-03-11T20:51:32.918+0000 7f8449808200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T20:51:33.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:32 smithi146 bash[23809]: cluster 2024-03-11T20:51:31.186948+0000 osd.7 (osd.7) 9 : cluster 0 3.e deep-scrub starts 2024-03-11T20:51:33.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:32 smithi146 bash[23809]: cluster 2024-03-11T20:51:31.193997+0000 osd.7 (osd.7) 10 : cluster 0 3.e deep-scrub ok 2024-03-11T20:51:33.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:32 smithi146 bash[23809]: cluster 2024-03-11T20:51:31.822435+0000 osd.0 (osd.0) 15 : cluster 0 3.1 deep-scrub starts 2024-03-11T20:51:33.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:32 smithi146 bash[23809]: cluster 2024-03-11T20:51:31.824596+0000 osd.2 (osd.2) 11 : cluster 0 2.1 scrub starts 2024-03-11T20:51:33.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:32 smithi146 bash[23809]: cluster 2024-03-11T20:51:31.827344+0000 osd.2 (osd.2) 12 : cluster 0 2.1 scrub ok 2024-03-11T20:51:33.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:32 smithi146 bash[23809]: cluster 2024-03-11T20:51:31.835288+0000 osd.0 (osd.0) 16 : cluster 0 3.1 deep-scrub ok 2024-03-11T20:51:33.033 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:51:32 smithi023 bash[21627]: debug 2024-03-11T20:51:32.607+0000 7f643c07c200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T20:51:33.034 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:51:32 smithi023 bash[21627]: debug 2024-03-11T20:51:32.731+0000 7f643c07c200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T20:51:33.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:32 smithi023 bash[29146]: cluster 2024-03-11T20:51:31.186948+0000 osd.7 (osd.7) 9 : cluster 0 3.e deep-scrub starts 2024-03-11T20:51:33.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:32 smithi023 bash[29146]: cluster 2024-03-11T20:51:31.193997+0000 osd.7 (osd.7) 10 : cluster 0 3.e deep-scrub ok 2024-03-11T20:51:33.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:32 smithi023 bash[29146]: cluster 2024-03-11T20:51:31.822435+0000 osd.0 (osd.0) 15 : cluster 0 3.1 deep-scrub starts 2024-03-11T20:51:33.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:32 smithi023 bash[29146]: cluster 2024-03-11T20:51:31.824596+0000 osd.2 (osd.2) 11 : cluster 0 2.1 scrub starts 2024-03-11T20:51:33.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:32 smithi023 bash[29146]: cluster 2024-03-11T20:51:31.827344+0000 osd.2 (osd.2) 12 : cluster 0 2.1 scrub ok 2024-03-11T20:51:33.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:32 smithi023 bash[29146]: cluster 2024-03-11T20:51:31.835288+0000 osd.0 (osd.0) 16 : cluster 0 3.1 deep-scrub ok 2024-03-11T20:51:33.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:32 smithi023 bash[21372]: cluster 2024-03-11T20:51:31.186948+0000 osd.7 (osd.7) 9 : cluster 0 3.e deep-scrub starts 2024-03-11T20:51:33.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:32 smithi023 bash[21372]: cluster 2024-03-11T20:51:31.193997+0000 osd.7 (osd.7) 10 : cluster 0 3.e deep-scrub ok 2024-03-11T20:51:33.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:32 smithi023 bash[21372]: cluster 2024-03-11T20:51:31.822435+0000 osd.0 (osd.0) 15 : cluster 0 3.1 deep-scrub starts 2024-03-11T20:51:33.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:32 smithi023 bash[21372]: cluster 2024-03-11T20:51:31.824596+0000 osd.2 (osd.2) 11 : cluster 0 2.1 scrub starts 2024-03-11T20:51:33.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:32 smithi023 bash[21372]: cluster 2024-03-11T20:51:31.827344+0000 osd.2 (osd.2) 12 : cluster 0 2.1 scrub ok 2024-03-11T20:51:33.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:32 smithi023 bash[21372]: cluster 2024-03-11T20:51:31.835288+0000 osd.0 (osd.0) 16 : cluster 0 3.1 deep-scrub ok 2024-03-11T20:51:33.533 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:51:33 smithi023 bash[21627]: debug 2024-03-11T20:51:33.039+0000 7f643c07c200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T20:51:33.534 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:51:33 smithi023 bash[21627]: debug 2024-03-11T20:51:33.159+0000 7f643c07c200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T20:51:33.718 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Mar 11 20:51:33 smithi146 bash[24737]: debug 2024-03-11T20:51:33.366+0000 7f8449808200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T20:51:33.718 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Mar 11 20:51:33 smithi146 bash[24737]: debug 2024-03-11T20:51:33.486+0000 7f8449808200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T20:51:33.718 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Mar 11 20:51:33 smithi146 bash[24737]: [11/Mar/2024:20:51:33] ENGINE Bus STARTING 2024-03-11T20:51:33.718 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Mar 11 20:51:33 smithi146 bash[24737]: CherryPy Checker: 2024-03-11T20:51:33.718 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Mar 11 20:51:33 smithi146 bash[24737]: The Application mounted at '' has an empty config. 2024-03-11T20:51:33.719 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Mar 11 20:51:33 smithi146 bash[24737]: [11/Mar/2024:20:51:33] ENGINE Serving on http://:::9283 2024-03-11T20:51:33.719 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Mar 11 20:51:33 smithi146 bash[24737]: [11/Mar/2024:20:51:33] ENGINE Bus STARTED 2024-03-11T20:51:33.915 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:51:33 smithi023 bash[21627]: debug 2024-03-11T20:51:33.639+0000 7f643c07c200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T20:51:33.916 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:51:33 smithi023 bash[21627]: debug 2024-03-11T20:51:33.767+0000 7f643c07c200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T20:51:33.916 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:33 smithi023 bash[29146]: cluster 2024-03-11T20:51:32.821882+0000 osd.0 (osd.0) 17 : cluster 0 4.b scrub starts 2024-03-11T20:51:33.916 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:33 smithi023 bash[29146]: cluster 2024-03-11T20:51:32.823814+0000 osd.0 (osd.0) 18 : cluster 0 4.b scrub ok 2024-03-11T20:51:33.916 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:33 smithi023 bash[29146]: cluster 2024-03-11T20:51:33.508336+0000 mon.a (mon.0) 768 : cluster 0 Standby manager daemon x restarted 2024-03-11T20:51:33.916 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:33 smithi023 bash[29146]: cluster 2024-03-11T20:51:33.508477+0000 mon.a (mon.0) 769 : cluster 0 Standby manager daemon x started 2024-03-11T20:51:33.916 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:33 smithi023 bash[29146]: audit 2024-03-11T20:51:33.514513+0000 mon.b (mon.1) 28 : audit 0 from='mgr.? 172.21.15.146:0/3108702774' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-11T20:51:33.916 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:33 smithi023 bash[29146]: audit 2024-03-11T20:51:33.515711+0000 mon.b (mon.1) 29 : audit 0 from='mgr.? 172.21.15.146:0/3108702774' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T20:51:33.917 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:33 smithi023 bash[29146]: audit 2024-03-11T20:51:33.519027+0000 mon.b (mon.1) 30 : audit 0 from='mgr.? 172.21.15.146:0/3108702774' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-11T20:51:33.917 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:33 smithi023 bash[29146]: audit 2024-03-11T20:51:33.519474+0000 mon.b (mon.1) 31 : audit 0 from='mgr.? 172.21.15.146:0/3108702774' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T20:51:33.917 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:33 smithi023 bash[21372]: cluster 2024-03-11T20:51:32.821882+0000 osd.0 (osd.0) 17 : cluster 0 4.b scrub starts 2024-03-11T20:51:33.917 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:33 smithi023 bash[21372]: cluster 2024-03-11T20:51:32.823814+0000 osd.0 (osd.0) 18 : cluster 0 4.b scrub ok 2024-03-11T20:51:33.917 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:33 smithi023 bash[21372]: cluster 2024-03-11T20:51:33.508336+0000 mon.a (mon.0) 768 : cluster 0 Standby manager daemon x restarted 2024-03-11T20:51:33.917 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:33 smithi023 bash[21372]: cluster 2024-03-11T20:51:33.508477+0000 mon.a (mon.0) 769 : cluster 0 Standby manager daemon x started 2024-03-11T20:51:33.917 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:33 smithi023 bash[21372]: audit 2024-03-11T20:51:33.514513+0000 mon.b (mon.1) 28 : audit 0 from='mgr.? 172.21.15.146:0/3108702774' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-11T20:51:33.917 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:33 smithi023 bash[21372]: audit 2024-03-11T20:51:33.515711+0000 mon.b (mon.1) 29 : audit 0 from='mgr.? 172.21.15.146:0/3108702774' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T20:51:33.917 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:33 smithi023 bash[21372]: audit 2024-03-11T20:51:33.519027+0000 mon.b (mon.1) 30 : audit 0 from='mgr.? 172.21.15.146:0/3108702774' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-11T20:51:33.917 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:33 smithi023 bash[21372]: audit 2024-03-11T20:51:33.519474+0000 mon.b (mon.1) 31 : audit 0 from='mgr.? 172.21.15.146:0/3108702774' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T20:51:33.975 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:33 smithi146 bash[23809]: cluster 2024-03-11T20:51:32.821882+0000 osd.0 (osd.0) 17 : cluster 0 4.b scrub starts 2024-03-11T20:51:33.975 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:33 smithi146 bash[23809]: cluster 2024-03-11T20:51:32.823814+0000 osd.0 (osd.0) 18 : cluster 0 4.b scrub ok 2024-03-11T20:51:33.975 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:33 smithi146 bash[23809]: cluster 2024-03-11T20:51:33.508336+0000 mon.a (mon.0) 768 : cluster 0 Standby manager daemon x restarted 2024-03-11T20:51:33.975 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:33 smithi146 bash[23809]: cluster 2024-03-11T20:51:33.508477+0000 mon.a (mon.0) 769 : cluster 0 Standby manager daemon x started 2024-03-11T20:51:33.975 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:33 smithi146 bash[23809]: audit 2024-03-11T20:51:33.514513+0000 mon.b (mon.1) 28 : audit 0 from='mgr.? 172.21.15.146:0/3108702774' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-11T20:51:33.975 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:33 smithi146 bash[23809]: audit 2024-03-11T20:51:33.515711+0000 mon.b (mon.1) 29 : audit 0 from='mgr.? 172.21.15.146:0/3108702774' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T20:51:33.975 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:33 smithi146 bash[23809]: audit 2024-03-11T20:51:33.519027+0000 mon.b (mon.1) 30 : audit 0 from='mgr.? 172.21.15.146:0/3108702774' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-11T20:51:33.976 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:33 smithi146 bash[23809]: audit 2024-03-11T20:51:33.519474+0000 mon.b (mon.1) 31 : audit 0 from='mgr.? 172.21.15.146:0/3108702774' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T20:51:34.284 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:51:33 smithi023 bash[21627]: [11/Mar/2024:20:51:33] ENGINE Bus STARTING 2024-03-11T20:51:34.284 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:51:34 smithi023 bash[21627]: CherryPy Checker: 2024-03-11T20:51:34.284 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:51:34 smithi023 bash[21627]: The Application mounted at '' has an empty config. 2024-03-11T20:51:34.284 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:51:34 smithi023 bash[21627]: [11/Mar/2024:20:51:34] ENGINE Serving on http://:::9283 2024-03-11T20:51:34.284 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:51:34 smithi023 bash[21627]: [11/Mar/2024:20:51:34] ENGINE Bus STARTED 2024-03-11T20:51:34.818 INFO:teuthology.orchestra.run.smithi146.stdout:Scheduled grafana update... 2024-03-11T20:51:35.010 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 20:51:34 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T20:51:35.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:34 smithi146 bash[23809]: cluster 2024-03-11T20:51:33.677555+0000 osd.6 (osd.6) 7 : cluster 0 4.5 scrub starts 2024-03-11T20:51:35.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:34 smithi146 bash[23809]: cluster 2024-03-11T20:51:33.678794+0000 osd.6 (osd.6) 8 : cluster 0 4.5 scrub ok 2024-03-11T20:51:35.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:34 smithi146 bash[23809]: cluster 2024-03-11T20:51:33.715851+0000 mon.a (mon.0) 770 : cluster 0 mgrmap e18: y(active, since 10m), standbys: x 2024-03-11T20:51:35.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:34 smithi146 bash[23809]: cluster 2024-03-11T20:51:33.790143+0000 mon.a (mon.0) 771 : cluster 1 Active manager daemon y restarted 2024-03-11T20:51:35.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:34 smithi146 bash[23809]: cluster 2024-03-11T20:51:33.791014+0000 mon.a (mon.0) 772 : cluster 1 Activating manager daemon y 2024-03-11T20:51:35.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:34 smithi146 bash[23809]: cluster 2024-03-11T20:51:33.805877+0000 mon.a (mon.0) 773 : cluster 0 osdmap e101: 8 total, 8 up, 8 in 2024-03-11T20:51:35.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:34 smithi146 bash[23809]: cluster 2024-03-11T20:51:33.806883+0000 mon.a (mon.0) 774 : cluster 0 mgrmap e19: y(active, starting, since 0.0160746s), standbys: x 2024-03-11T20:51:35.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:34 smithi146 bash[23809]: audit 2024-03-11T20:51:33.808402+0000 mon.a (mon.0) 775 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T20:51:35.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:34 smithi146 bash[23809]: audit 2024-03-11T20:51:33.808557+0000 mon.a (mon.0) 776 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T20:51:35.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:34 smithi146 bash[23809]: audit 2024-03-11T20:51:33.808805+0000 mon.a (mon.0) 777 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T20:51:35.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:34 smithi146 bash[23809]: audit 2024-03-11T20:51:33.809953+0000 mon.a (mon.0) 778 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-11T20:51:35.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:34 smithi146 bash[23809]: audit 2024-03-11T20:51:33.810242+0000 mon.a (mon.0) 779 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-11T20:51:35.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:34 smithi146 bash[23809]: audit 2024-03-11T20:51:33.810531+0000 mon.a (mon.0) 780 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T20:51:35.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:34 smithi146 bash[23809]: audit 2024-03-11T20:51:33.810887+0000 mon.a (mon.0) 781 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T20:51:35.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:34 smithi146 bash[23809]: audit 2024-03-11T20:51:33.811079+0000 mon.a (mon.0) 782 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T20:51:35.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:34 smithi146 bash[23809]: audit 2024-03-11T20:51:33.811293+0000 mon.a (mon.0) 783 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T20:51:35.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:34 smithi146 bash[23809]: audit 2024-03-11T20:51:33.811522+0000 mon.a (mon.0) 784 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T20:51:35.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:34 smithi146 bash[23809]: audit 2024-03-11T20:51:33.811773+0000 mon.a (mon.0) 785 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T20:51:35.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:34 smithi146 bash[23809]: audit 2024-03-11T20:51:33.812063+0000 mon.a (mon.0) 786 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T20:51:35.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:34 smithi146 bash[23809]: audit 2024-03-11T20:51:33.812406+0000 mon.a (mon.0) 787 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T20:51:35.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:34 smithi146 bash[23809]: audit 2024-03-11T20:51:33.812825+0000 mon.a (mon.0) 788 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T20:51:35.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:34 smithi146 bash[23809]: audit 2024-03-11T20:51:33.812948+0000 mon.a (mon.0) 789 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T20:51:35.013 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:34 smithi146 bash[23809]: audit 2024-03-11T20:51:33.813427+0000 mon.a (mon.0) 790 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T20:51:35.013 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:34 smithi146 bash[23809]: cluster 2024-03-11T20:51:33.821016+0000 osd.5 (osd.5) 15 : cluster 0 2.2 deep-scrub starts 2024-03-11T20:51:35.013 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:34 smithi146 bash[23809]: cluster 2024-03-11T20:51:33.823678+0000 osd.5 (osd.5) 16 : cluster 0 2.2 deep-scrub ok 2024-03-11T20:51:35.013 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:34 smithi146 bash[23809]: cluster 2024-03-11T20:51:33.841589+0000 mon.a (mon.0) 791 : cluster 1 Manager daemon y is now available 2024-03-11T20:51:35.013 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:34 smithi146 bash[23809]: cluster 2024-03-11T20:51:33.881133+0000 osd.1 (osd.1) 15 : cluster 0 3.0 scrub starts 2024-03-11T20:51:35.013 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:34 smithi146 bash[23809]: cluster 2024-03-11T20:51:33.883816+0000 osd.1 (osd.1) 16 : cluster 0 3.0 scrub ok 2024-03-11T20:51:35.013 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:34 smithi146 bash[23809]: audit 2024-03-11T20:51:33.886712+0000 mon.a (mon.0) 792 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:51:35.013 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:34 smithi146 bash[23809]: audit 2024-03-11T20:51:33.888548+0000 mon.a (mon.0) 793 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:51:35.013 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:34 smithi146 bash[23809]: audit 2024-03-11T20:51:33.916581+0000 mon.a (mon.0) 794 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:51:35.013 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:34 smithi146 bash[23809]: audit 2024-03-11T20:51:33.919773+0000 mon.a (mon.0) 795 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:51:35.013 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:34 smithi146 bash[23809]: cluster 2024-03-11T20:51:33.925733+0000 osd.3 (osd.3) 9 : cluster 0 purged_snaps scrub starts 2024-03-11T20:51:35.013 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:34 smithi146 bash[23809]: cluster 2024-03-11T20:51:33.925793+0000 osd.3 (osd.3) 10 : cluster 0 purged_snaps scrub ok 2024-03-11T20:51:35.013 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:34 smithi146 bash[23809]: audit 2024-03-11T20:51:33.931541+0000 mon.a (mon.0) 796 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T20:51:35.013 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:34 smithi146 bash[23809]: audit 2024-03-11T20:51:33.952039+0000 mon.a (mon.0) 797 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T20:51:35.013 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:34 smithi146 bash[23809]: cluster 2024-03-11T20:51:34.593588+0000 mon.a (mon.0) 798 : cluster 0 osdmap e102: 8 total, 8 up, 8 in 2024-03-11T20:51:35.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:34 smithi023 bash[29146]: cluster 2024-03-11T20:51:33.677555+0000 osd.6 (osd.6) 7 : cluster 0 4.5 scrub starts 2024-03-11T20:51:35.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:34 smithi023 bash[29146]: cluster 2024-03-11T20:51:33.678794+0000 osd.6 (osd.6) 8 : cluster 0 4.5 scrub ok 2024-03-11T20:51:35.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:34 smithi023 bash[29146]: cluster 2024-03-11T20:51:33.715851+0000 mon.a (mon.0) 770 : cluster 0 mgrmap e18: y(active, since 10m), standbys: x 2024-03-11T20:51:35.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:34 smithi023 bash[29146]: cluster 2024-03-11T20:51:33.790143+0000 mon.a (mon.0) 771 : cluster 1 Active manager daemon y restarted 2024-03-11T20:51:35.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:34 smithi023 bash[29146]: cluster 2024-03-11T20:51:33.791014+0000 mon.a (mon.0) 772 : cluster 1 Activating manager daemon y 2024-03-11T20:51:35.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:34 smithi023 bash[29146]: cluster 2024-03-11T20:51:33.805877+0000 mon.a (mon.0) 773 : cluster 0 osdmap e101: 8 total, 8 up, 8 in 2024-03-11T20:51:35.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:34 smithi023 bash[29146]: cluster 2024-03-11T20:51:33.806883+0000 mon.a (mon.0) 774 : cluster 0 mgrmap e19: y(active, starting, since 0.0160746s), standbys: x 2024-03-11T20:51:35.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:34 smithi023 bash[29146]: audit 2024-03-11T20:51:33.808402+0000 mon.a (mon.0) 775 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T20:51:35.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:34 smithi023 bash[29146]: audit 2024-03-11T20:51:33.808557+0000 mon.a (mon.0) 776 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T20:51:35.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:34 smithi023 bash[29146]: audit 2024-03-11T20:51:33.808805+0000 mon.a (mon.0) 777 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T20:51:35.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:34 smithi023 bash[29146]: audit 2024-03-11T20:51:33.809953+0000 mon.a (mon.0) 778 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-11T20:51:35.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:34 smithi023 bash[29146]: audit 2024-03-11T20:51:33.810242+0000 mon.a (mon.0) 779 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-11T20:51:35.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:34 smithi023 bash[29146]: audit 2024-03-11T20:51:33.810531+0000 mon.a (mon.0) 780 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T20:51:35.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:34 smithi023 bash[29146]: audit 2024-03-11T20:51:33.810887+0000 mon.a (mon.0) 781 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T20:51:35.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:34 smithi023 bash[29146]: audit 2024-03-11T20:51:33.811079+0000 mon.a (mon.0) 782 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T20:51:35.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:34 smithi023 bash[29146]: audit 2024-03-11T20:51:33.811293+0000 mon.a (mon.0) 783 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T20:51:35.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:34 smithi023 bash[29146]: audit 2024-03-11T20:51:33.811522+0000 mon.a (mon.0) 784 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T20:51:35.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:34 smithi023 bash[29146]: audit 2024-03-11T20:51:33.811773+0000 mon.a (mon.0) 785 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T20:51:35.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:34 smithi023 bash[29146]: audit 2024-03-11T20:51:33.812063+0000 mon.a (mon.0) 786 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T20:51:35.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:34 smithi023 bash[29146]: audit 2024-03-11T20:51:33.812406+0000 mon.a (mon.0) 787 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T20:51:35.036 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:34 smithi023 bash[29146]: audit 2024-03-11T20:51:33.812825+0000 mon.a (mon.0) 788 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T20:51:35.036 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:34 smithi023 bash[21372]: cluster 2024-03-11T20:51:33.677555+0000 osd.6 (osd.6) 7 : cluster 0 4.5 scrub starts 2024-03-11T20:51:35.036 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:34 smithi023 bash[21372]: cluster 2024-03-11T20:51:33.678794+0000 osd.6 (osd.6) 8 : cluster 0 4.5 scrub ok 2024-03-11T20:51:35.036 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:34 smithi023 bash[21372]: cluster 2024-03-11T20:51:33.715851+0000 mon.a (mon.0) 770 : cluster 0 mgrmap e18: y(active, since 10m), standbys: x 2024-03-11T20:51:35.036 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:34 smithi023 bash[21372]: cluster 2024-03-11T20:51:33.790143+0000 mon.a (mon.0) 771 : cluster 1 Active manager daemon y restarted 2024-03-11T20:51:35.036 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:34 smithi023 bash[21372]: cluster 2024-03-11T20:51:33.791014+0000 mon.a (mon.0) 772 : cluster 1 Activating manager daemon y 2024-03-11T20:51:35.036 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:34 smithi023 bash[21372]: cluster 2024-03-11T20:51:33.805877+0000 mon.a (mon.0) 773 : cluster 0 osdmap e101: 8 total, 8 up, 8 in 2024-03-11T20:51:35.036 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:34 smithi023 bash[21372]: cluster 2024-03-11T20:51:33.806883+0000 mon.a (mon.0) 774 : cluster 0 mgrmap e19: y(active, starting, since 0.0160746s), standbys: x 2024-03-11T20:51:35.036 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:34 smithi023 bash[21372]: audit 2024-03-11T20:51:33.808402+0000 mon.a (mon.0) 775 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T20:51:35.036 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:34 smithi023 bash[21372]: audit 2024-03-11T20:51:33.808557+0000 mon.a (mon.0) 776 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T20:51:35.036 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:34 smithi023 bash[21372]: audit 2024-03-11T20:51:33.808805+0000 mon.a (mon.0) 777 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T20:51:35.036 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:34 smithi023 bash[21372]: audit 2024-03-11T20:51:33.809953+0000 mon.a (mon.0) 778 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-11T20:51:35.037 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:34 smithi023 bash[21372]: audit 2024-03-11T20:51:33.810242+0000 mon.a (mon.0) 779 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-11T20:51:35.037 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:34 smithi023 bash[21372]: audit 2024-03-11T20:51:33.810531+0000 mon.a (mon.0) 780 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T20:51:35.037 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:34 smithi023 bash[21372]: audit 2024-03-11T20:51:33.810887+0000 mon.a (mon.0) 781 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T20:51:35.037 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:34 smithi023 bash[21372]: audit 2024-03-11T20:51:33.811079+0000 mon.a (mon.0) 782 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T20:51:35.037 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:34 smithi023 bash[21372]: audit 2024-03-11T20:51:33.811293+0000 mon.a (mon.0) 783 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T20:51:35.037 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:34 smithi023 bash[21372]: audit 2024-03-11T20:51:33.811522+0000 mon.a (mon.0) 784 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T20:51:35.037 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:34 smithi023 bash[21372]: audit 2024-03-11T20:51:33.811773+0000 mon.a (mon.0) 785 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T20:51:35.037 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:34 smithi023 bash[21372]: audit 2024-03-11T20:51:33.812063+0000 mon.a (mon.0) 786 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T20:51:35.037 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:34 smithi023 bash[21372]: audit 2024-03-11T20:51:33.812406+0000 mon.a (mon.0) 787 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T20:51:35.037 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:34 smithi023 bash[21372]: audit 2024-03-11T20:51:33.812825+0000 mon.a (mon.0) 788 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T20:51:35.037 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:34 smithi023 bash[29146]: audit 2024-03-11T20:51:33.812948+0000 mon.a (mon.0) 789 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T20:51:35.037 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:34 smithi023 bash[29146]: audit 2024-03-11T20:51:33.813427+0000 mon.a (mon.0) 790 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T20:51:35.038 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:34 smithi023 bash[29146]: cluster 2024-03-11T20:51:33.821016+0000 osd.5 (osd.5) 15 : cluster 0 2.2 deep-scrub starts 2024-03-11T20:51:35.038 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:34 smithi023 bash[29146]: cluster 2024-03-11T20:51:33.823678+0000 osd.5 (osd.5) 16 : cluster 0 2.2 deep-scrub ok 2024-03-11T20:51:35.038 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:34 smithi023 bash[29146]: cluster 2024-03-11T20:51:33.841589+0000 mon.a (mon.0) 791 : cluster 1 Manager daemon y is now available 2024-03-11T20:51:35.038 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:34 smithi023 bash[29146]: cluster 2024-03-11T20:51:33.881133+0000 osd.1 (osd.1) 15 : cluster 0 3.0 scrub starts 2024-03-11T20:51:35.038 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:34 smithi023 bash[29146]: cluster 2024-03-11T20:51:33.883816+0000 osd.1 (osd.1) 16 : cluster 0 3.0 scrub ok 2024-03-11T20:51:35.038 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:34 smithi023 bash[29146]: audit 2024-03-11T20:51:33.886712+0000 mon.a (mon.0) 792 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:51:35.038 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:34 smithi023 bash[29146]: audit 2024-03-11T20:51:33.888548+0000 mon.a (mon.0) 793 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:51:35.038 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:34 smithi023 bash[29146]: audit 2024-03-11T20:51:33.916581+0000 mon.a (mon.0) 794 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:51:35.038 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:34 smithi023 bash[29146]: audit 2024-03-11T20:51:33.919773+0000 mon.a (mon.0) 795 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:51:35.038 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:34 smithi023 bash[29146]: cluster 2024-03-11T20:51:33.925733+0000 osd.3 (osd.3) 9 : cluster 0 purged_snaps scrub starts 2024-03-11T20:51:35.038 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:34 smithi023 bash[29146]: cluster 2024-03-11T20:51:33.925793+0000 osd.3 (osd.3) 10 : cluster 0 purged_snaps scrub ok 2024-03-11T20:51:35.038 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:34 smithi023 bash[29146]: audit 2024-03-11T20:51:33.931541+0000 mon.a (mon.0) 796 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T20:51:35.038 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:34 smithi023 bash[29146]: audit 2024-03-11T20:51:33.952039+0000 mon.a (mon.0) 797 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T20:51:35.038 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:34 smithi023 bash[29146]: cluster 2024-03-11T20:51:34.593588+0000 mon.a (mon.0) 798 : cluster 0 osdmap e102: 8 total, 8 up, 8 in 2024-03-11T20:51:35.039 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:34 smithi023 bash[21372]: audit 2024-03-11T20:51:33.812948+0000 mon.a (mon.0) 789 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T20:51:35.039 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:34 smithi023 bash[21372]: audit 2024-03-11T20:51:33.813427+0000 mon.a (mon.0) 790 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T20:51:35.039 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:34 smithi023 bash[21372]: cluster 2024-03-11T20:51:33.821016+0000 osd.5 (osd.5) 15 : cluster 0 2.2 deep-scrub starts 2024-03-11T20:51:35.039 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:34 smithi023 bash[21372]: cluster 2024-03-11T20:51:33.823678+0000 osd.5 (osd.5) 16 : cluster 0 2.2 deep-scrub ok 2024-03-11T20:51:35.039 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:34 smithi023 bash[21372]: cluster 2024-03-11T20:51:33.841589+0000 mon.a (mon.0) 791 : cluster 1 Manager daemon y is now available 2024-03-11T20:51:35.039 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:34 smithi023 bash[21372]: cluster 2024-03-11T20:51:33.881133+0000 osd.1 (osd.1) 15 : cluster 0 3.0 scrub starts 2024-03-11T20:51:35.039 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:34 smithi023 bash[21372]: cluster 2024-03-11T20:51:33.883816+0000 osd.1 (osd.1) 16 : cluster 0 3.0 scrub ok 2024-03-11T20:51:35.039 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:34 smithi023 bash[21372]: audit 2024-03-11T20:51:33.886712+0000 mon.a (mon.0) 792 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:51:35.039 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:34 smithi023 bash[21372]: audit 2024-03-11T20:51:33.888548+0000 mon.a (mon.0) 793 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:51:35.039 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:34 smithi023 bash[21372]: audit 2024-03-11T20:51:33.916581+0000 mon.a (mon.0) 794 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:51:35.039 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:34 smithi023 bash[21372]: audit 2024-03-11T20:51:33.919773+0000 mon.a (mon.0) 795 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:51:35.039 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:34 smithi023 bash[21372]: cluster 2024-03-11T20:51:33.925733+0000 osd.3 (osd.3) 9 : cluster 0 purged_snaps scrub starts 2024-03-11T20:51:35.039 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:34 smithi023 bash[21372]: cluster 2024-03-11T20:51:33.925793+0000 osd.3 (osd.3) 10 : cluster 0 purged_snaps scrub ok 2024-03-11T20:51:35.039 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:34 smithi023 bash[21372]: audit 2024-03-11T20:51:33.931541+0000 mon.a (mon.0) 796 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T20:51:35.039 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:34 smithi023 bash[21372]: audit 2024-03-11T20:51:33.952039+0000 mon.a (mon.0) 797 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T20:51:35.040 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:34 smithi023 bash[21372]: cluster 2024-03-11T20:51:34.593588+0000 mon.a (mon.0) 798 : cluster 0 osdmap e102: 8 total, 8 up, 8 in 2024-03-11T20:51:35.608 DEBUG:teuthology.orchestra.run.smithi146:grafana.a> sudo journalctl -f -n 0 -u ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@grafana.a.service 2024-03-11T20:51:35.611 INFO:tasks.cephadm:Setting up client nodes... 2024-03-11T20:51:35.611 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-11T20:51:36.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:35 smithi146 bash[23809]: audit 2024-03-11T20:51:34.794350+0000 mon.a (mon.0) 799 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:51:36.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:35 smithi146 bash[23809]: cluster 2024-03-11T20:51:34.804053+0000 mon.a (mon.0) 800 : cluster 0 mgrmap e20: y(active, since 1.01324s), standbys: x 2024-03-11T20:51:36.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:35 smithi146 bash[23809]: cephadm 2024-03-11T20:51:34.816212+0000 mgr.y (mgr.24380) 4 : cephadm 1 Saving service grafana spec with placement smithi146=a;count:1 2024-03-11T20:51:36.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:35 smithi146 bash[23809]: audit 2024-03-11T20:51:34.816215+0000 mon.a (mon.0) 801 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:51:36.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:35 smithi146 bash[23809]: audit 2024-03-11T20:51:34.820594+0000 mon.a (mon.0) 802 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:51:36.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:35 smithi146 bash[23809]: cephadm 2024-03-11T20:51:35.067016+0000 mgr.y (mgr.24380) 5 : cephadm 1 [11/Mar/2024:20:51:35] ENGINE Bus STARTING 2024-03-11T20:51:36.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:35 smithi146 bash[23809]: cephadm 2024-03-11T20:51:35.189425+0000 mgr.y (mgr.24380) 6 : cephadm 1 [11/Mar/2024:20:51:35] ENGINE Serving on https://172.21.15.23:7150 2024-03-11T20:51:36.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:35 smithi146 bash[23809]: cephadm 2024-03-11T20:51:35.290719+0000 mgr.y (mgr.24380) 7 : cephadm 1 [11/Mar/2024:20:51:35] ENGINE Serving on http://172.21.15.23:8765 2024-03-11T20:51:36.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:35 smithi146 bash[23809]: cephadm 2024-03-11T20:51:35.290904+0000 mgr.y (mgr.24380) 8 : cephadm 1 [11/Mar/2024:20:51:35] ENGINE Bus STARTED 2024-03-11T20:51:36.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:35 smithi023 bash[21372]: audit 2024-03-11T20:51:34.794350+0000 mon.a (mon.0) 799 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:51:36.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:35 smithi023 bash[21372]: cluster 2024-03-11T20:51:34.804053+0000 mon.a (mon.0) 800 : cluster 0 mgrmap e20: y(active, since 1.01324s), standbys: x 2024-03-11T20:51:36.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:35 smithi023 bash[21372]: cephadm 2024-03-11T20:51:34.816212+0000 mgr.y (mgr.24380) 4 : cephadm 1 Saving service grafana spec with placement smithi146=a;count:1 2024-03-11T20:51:36.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:35 smithi023 bash[21372]: audit 2024-03-11T20:51:34.816215+0000 mon.a (mon.0) 801 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:51:36.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:35 smithi023 bash[21372]: audit 2024-03-11T20:51:34.820594+0000 mon.a (mon.0) 802 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:51:36.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:35 smithi023 bash[21372]: cephadm 2024-03-11T20:51:35.067016+0000 mgr.y (mgr.24380) 5 : cephadm 1 [11/Mar/2024:20:51:35] ENGINE Bus STARTING 2024-03-11T20:51:36.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:35 smithi023 bash[21372]: cephadm 2024-03-11T20:51:35.189425+0000 mgr.y (mgr.24380) 6 : cephadm 1 [11/Mar/2024:20:51:35] ENGINE Serving on https://172.21.15.23:7150 2024-03-11T20:51:36.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:35 smithi023 bash[21372]: cephadm 2024-03-11T20:51:35.290719+0000 mgr.y (mgr.24380) 7 : cephadm 1 [11/Mar/2024:20:51:35] ENGINE Serving on http://172.21.15.23:8765 2024-03-11T20:51:36.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:35 smithi023 bash[21372]: cephadm 2024-03-11T20:51:35.290904+0000 mgr.y (mgr.24380) 8 : cephadm 1 [11/Mar/2024:20:51:35] ENGINE Bus STARTED 2024-03-11T20:51:36.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:35 smithi023 bash[29146]: audit 2024-03-11T20:51:34.794350+0000 mon.a (mon.0) 799 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:51:36.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:35 smithi023 bash[29146]: cluster 2024-03-11T20:51:34.804053+0000 mon.a (mon.0) 800 : cluster 0 mgrmap e20: y(active, since 1.01324s), standbys: x 2024-03-11T20:51:36.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:35 smithi023 bash[29146]: cephadm 2024-03-11T20:51:34.816212+0000 mgr.y (mgr.24380) 4 : cephadm 1 Saving service grafana spec with placement smithi146=a;count:1 2024-03-11T20:51:36.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:35 smithi023 bash[29146]: audit 2024-03-11T20:51:34.816215+0000 mon.a (mon.0) 801 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:51:36.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:35 smithi023 bash[29146]: audit 2024-03-11T20:51:34.820594+0000 mon.a (mon.0) 802 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:51:36.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:35 smithi023 bash[29146]: cephadm 2024-03-11T20:51:35.067016+0000 mgr.y (mgr.24380) 5 : cephadm 1 [11/Mar/2024:20:51:35] ENGINE Bus STARTING 2024-03-11T20:51:36.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:35 smithi023 bash[29146]: cephadm 2024-03-11T20:51:35.189425+0000 mgr.y (mgr.24380) 6 : cephadm 1 [11/Mar/2024:20:51:35] ENGINE Serving on https://172.21.15.23:7150 2024-03-11T20:51:36.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:35 smithi023 bash[29146]: cephadm 2024-03-11T20:51:35.290719+0000 mgr.y (mgr.24380) 7 : cephadm 1 [11/Mar/2024:20:51:35] ENGINE Serving on http://172.21.15.23:8765 2024-03-11T20:51:36.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:35 smithi023 bash[29146]: cephadm 2024-03-11T20:51:35.290904+0000 mgr.y (mgr.24380) 8 : cephadm 1 [11/Mar/2024:20:51:35] ENGINE Bus STARTED 2024-03-11T20:51:37.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:36 smithi146 bash[23809]: cluster 2024-03-11T20:51:35.168951+0000 osd.7 (osd.7) 11 : cluster 0 3.d deep-scrub starts 2024-03-11T20:51:37.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:36 smithi146 bash[23809]: cluster 2024-03-11T20:51:35.187080+0000 osd.7 (osd.7) 12 : cluster 0 3.d deep-scrub ok 2024-03-11T20:51:37.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:36 smithi146 bash[23809]: cluster 2024-03-11T20:51:35.668468+0000 osd.4 (osd.4) 9 : cluster 0 4.c deep-scrub starts 2024-03-11T20:51:37.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:36 smithi146 bash[23809]: cluster 2024-03-11T20:51:35.670143+0000 osd.4 (osd.4) 10 : cluster 0 4.c deep-scrub ok 2024-03-11T20:51:37.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:36 smithi146 bash[23809]: cluster 2024-03-11T20:51:35.795797+0000 osd.2 (osd.2) 13 : cluster 0 6.f scrub starts 2024-03-11T20:51:37.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:36 smithi146 bash[23809]: cluster 2024-03-11T20:51:35.797492+0000 osd.2 (osd.2) 14 : cluster 0 6.f scrub ok 2024-03-11T20:51:37.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:36 smithi146 bash[23809]: cluster 2024-03-11T20:51:35.811779+0000 mgr.y (mgr.24380) 9 : cluster 0 pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:51:37.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:36 smithi146 bash[23809]: cluster 2024-03-11T20:51:35.890309+0000 osd.0 (osd.0) 19 : cluster 0 2.13 scrub starts 2024-03-11T20:51:37.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:36 smithi146 bash[23809]: cluster 2024-03-11T20:51:35.892639+0000 osd.0 (osd.0) 20 : cluster 0 2.13 scrub ok 2024-03-11T20:51:37.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:36 smithi146 bash[23809]: cluster 2024-03-11T20:51:36.813165+0000 mon.a (mon.0) 803 : cluster 0 mgrmap e21: y(active, since 3s), standbys: x 2024-03-11T20:51:37.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:36 smithi023 bash[21372]: cluster 2024-03-11T20:51:35.168951+0000 osd.7 (osd.7) 11 : cluster 0 3.d deep-scrub starts 2024-03-11T20:51:37.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:36 smithi023 bash[21372]: cluster 2024-03-11T20:51:35.187080+0000 osd.7 (osd.7) 12 : cluster 0 3.d deep-scrub ok 2024-03-11T20:51:37.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:36 smithi023 bash[21372]: cluster 2024-03-11T20:51:35.668468+0000 osd.4 (osd.4) 9 : cluster 0 4.c deep-scrub starts 2024-03-11T20:51:37.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:36 smithi023 bash[21372]: cluster 2024-03-11T20:51:35.670143+0000 osd.4 (osd.4) 10 : cluster 0 4.c deep-scrub ok 2024-03-11T20:51:37.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:36 smithi023 bash[21372]: cluster 2024-03-11T20:51:35.795797+0000 osd.2 (osd.2) 13 : cluster 0 6.f scrub starts 2024-03-11T20:51:37.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:36 smithi023 bash[21372]: cluster 2024-03-11T20:51:35.797492+0000 osd.2 (osd.2) 14 : cluster 0 6.f scrub ok 2024-03-11T20:51:37.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:36 smithi023 bash[21372]: cluster 2024-03-11T20:51:35.811779+0000 mgr.y (mgr.24380) 9 : cluster 0 pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:51:37.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:36 smithi023 bash[21372]: cluster 2024-03-11T20:51:35.890309+0000 osd.0 (osd.0) 19 : cluster 0 2.13 scrub starts 2024-03-11T20:51:37.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:36 smithi023 bash[21372]: cluster 2024-03-11T20:51:35.892639+0000 osd.0 (osd.0) 20 : cluster 0 2.13 scrub ok 2024-03-11T20:51:37.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:36 smithi023 bash[21372]: cluster 2024-03-11T20:51:36.813165+0000 mon.a (mon.0) 803 : cluster 0 mgrmap e21: y(active, since 3s), standbys: x 2024-03-11T20:51:37.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:36 smithi023 bash[29146]: cluster 2024-03-11T20:51:35.168951+0000 osd.7 (osd.7) 11 : cluster 0 3.d deep-scrub starts 2024-03-11T20:51:37.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:36 smithi023 bash[29146]: cluster 2024-03-11T20:51:35.187080+0000 osd.7 (osd.7) 12 : cluster 0 3.d deep-scrub ok 2024-03-11T20:51:37.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:36 smithi023 bash[29146]: cluster 2024-03-11T20:51:35.668468+0000 osd.4 (osd.4) 9 : cluster 0 4.c deep-scrub starts 2024-03-11T20:51:37.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:36 smithi023 bash[29146]: cluster 2024-03-11T20:51:35.670143+0000 osd.4 (osd.4) 10 : cluster 0 4.c deep-scrub ok 2024-03-11T20:51:37.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:36 smithi023 bash[29146]: cluster 2024-03-11T20:51:35.795797+0000 osd.2 (osd.2) 13 : cluster 0 6.f scrub starts 2024-03-11T20:51:37.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:36 smithi023 bash[29146]: cluster 2024-03-11T20:51:35.797492+0000 osd.2 (osd.2) 14 : cluster 0 6.f scrub ok 2024-03-11T20:51:37.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:36 smithi023 bash[29146]: cluster 2024-03-11T20:51:35.811779+0000 mgr.y (mgr.24380) 9 : cluster 0 pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:51:37.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:36 smithi023 bash[29146]: cluster 2024-03-11T20:51:35.890309+0000 osd.0 (osd.0) 19 : cluster 0 2.13 scrub starts 2024-03-11T20:51:37.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:36 smithi023 bash[29146]: cluster 2024-03-11T20:51:35.892639+0000 osd.0 (osd.0) 20 : cluster 0 2.13 scrub ok 2024-03-11T20:51:37.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:36 smithi023 bash[29146]: cluster 2024-03-11T20:51:36.813165+0000 mon.a (mon.0) 803 : cluster 0 mgrmap e21: y(active, since 3s), standbys: x 2024-03-11T20:51:38.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:37 smithi146 bash[23809]: cluster 2024-03-11T20:51:36.696774+0000 osd.6 (osd.6) 9 : cluster 0 2.14 scrub starts 2024-03-11T20:51:38.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:37 smithi146 bash[23809]: cluster 2024-03-11T20:51:36.699172+0000 osd.6 (osd.6) 10 : cluster 0 2.14 scrub ok 2024-03-11T20:51:38.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:37 smithi023 bash[21372]: cluster 2024-03-11T20:51:36.696774+0000 osd.6 (osd.6) 9 : cluster 0 2.14 scrub starts 2024-03-11T20:51:38.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:37 smithi023 bash[21372]: cluster 2024-03-11T20:51:36.699172+0000 osd.6 (osd.6) 10 : cluster 0 2.14 scrub ok 2024-03-11T20:51:38.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:37 smithi023 bash[29146]: cluster 2024-03-11T20:51:36.696774+0000 osd.6 (osd.6) 9 : cluster 0 2.14 scrub starts 2024-03-11T20:51:38.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:37 smithi023 bash[29146]: cluster 2024-03-11T20:51:36.699172+0000 osd.6 (osd.6) 10 : cluster 0 2.14 scrub ok 2024-03-11T20:51:39.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:38 smithi146 bash[23809]: cluster 2024-03-11T20:51:37.144176+0000 osd.7 (osd.7) 13 : cluster 0 2.b scrub starts 2024-03-11T20:51:39.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:38 smithi146 bash[23809]: cluster 2024-03-11T20:51:37.146614+0000 osd.7 (osd.7) 14 : cluster 0 2.b scrub ok 2024-03-11T20:51:39.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:38 smithi146 bash[23809]: cluster 2024-03-11T20:51:37.667562+0000 osd.4 (osd.4) 11 : cluster 0 3.14 scrub starts 2024-03-11T20:51:39.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:38 smithi146 bash[23809]: cluster 2024-03-11T20:51:37.671835+0000 osd.4 (osd.4) 12 : cluster 0 3.14 scrub ok 2024-03-11T20:51:39.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:38 smithi146 bash[23809]: cluster 2024-03-11T20:51:37.780864+0000 osd.2 (osd.2) 15 : cluster 0 2.0 scrub starts 2024-03-11T20:51:39.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:38 smithi146 bash[23809]: cluster 2024-03-11T20:51:37.781938+0000 osd.2 (osd.2) 16 : cluster 0 2.0 scrub ok 2024-03-11T20:51:39.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:38 smithi146 bash[23809]: cluster 2024-03-11T20:51:37.812140+0000 mgr.y (mgr.24380) 10 : cluster 0 pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:51:39.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:38 smithi146 bash[23809]: cluster 2024-03-11T20:51:37.860578+0000 osd.5 (osd.5) 17 : cluster 0 2.3 scrub starts 2024-03-11T20:51:39.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:38 smithi146 bash[23809]: cluster 2024-03-11T20:51:37.862993+0000 osd.5 (osd.5) 18 : cluster 0 2.3 scrub ok 2024-03-11T20:51:39.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:38 smithi023 bash[21372]: cluster 2024-03-11T20:51:37.144176+0000 osd.7 (osd.7) 13 : cluster 0 2.b scrub starts 2024-03-11T20:51:39.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:38 smithi023 bash[21372]: cluster 2024-03-11T20:51:37.146614+0000 osd.7 (osd.7) 14 : cluster 0 2.b scrub ok 2024-03-11T20:51:39.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:38 smithi023 bash[21372]: cluster 2024-03-11T20:51:37.667562+0000 osd.4 (osd.4) 11 : cluster 0 3.14 scrub starts 2024-03-11T20:51:39.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:38 smithi023 bash[21372]: cluster 2024-03-11T20:51:37.671835+0000 osd.4 (osd.4) 12 : cluster 0 3.14 scrub ok 2024-03-11T20:51:39.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:38 smithi023 bash[21372]: cluster 2024-03-11T20:51:37.780864+0000 osd.2 (osd.2) 15 : cluster 0 2.0 scrub starts 2024-03-11T20:51:39.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:38 smithi023 bash[21372]: cluster 2024-03-11T20:51:37.781938+0000 osd.2 (osd.2) 16 : cluster 0 2.0 scrub ok 2024-03-11T20:51:39.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:38 smithi023 bash[21372]: cluster 2024-03-11T20:51:37.812140+0000 mgr.y (mgr.24380) 10 : cluster 0 pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:51:39.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:38 smithi023 bash[21372]: cluster 2024-03-11T20:51:37.860578+0000 osd.5 (osd.5) 17 : cluster 0 2.3 scrub starts 2024-03-11T20:51:39.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:38 smithi023 bash[21372]: cluster 2024-03-11T20:51:37.862993+0000 osd.5 (osd.5) 18 : cluster 0 2.3 scrub ok 2024-03-11T20:51:39.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:38 smithi023 bash[29146]: cluster 2024-03-11T20:51:37.144176+0000 osd.7 (osd.7) 13 : cluster 0 2.b scrub starts 2024-03-11T20:51:39.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:38 smithi023 bash[29146]: cluster 2024-03-11T20:51:37.146614+0000 osd.7 (osd.7) 14 : cluster 0 2.b scrub ok 2024-03-11T20:51:39.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:38 smithi023 bash[29146]: cluster 2024-03-11T20:51:37.667562+0000 osd.4 (osd.4) 11 : cluster 0 3.14 scrub starts 2024-03-11T20:51:39.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:38 smithi023 bash[29146]: cluster 2024-03-11T20:51:37.671835+0000 osd.4 (osd.4) 12 : cluster 0 3.14 scrub ok 2024-03-11T20:51:39.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:38 smithi023 bash[29146]: cluster 2024-03-11T20:51:37.780864+0000 osd.2 (osd.2) 15 : cluster 0 2.0 scrub starts 2024-03-11T20:51:39.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:38 smithi023 bash[29146]: cluster 2024-03-11T20:51:37.781938+0000 osd.2 (osd.2) 16 : cluster 0 2.0 scrub ok 2024-03-11T20:51:39.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:38 smithi023 bash[29146]: cluster 2024-03-11T20:51:37.812140+0000 mgr.y (mgr.24380) 10 : cluster 0 pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:51:39.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:38 smithi023 bash[29146]: cluster 2024-03-11T20:51:37.860578+0000 osd.5 (osd.5) 17 : cluster 0 2.3 scrub starts 2024-03-11T20:51:39.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:38 smithi023 bash[29146]: cluster 2024-03-11T20:51:37.862993+0000 osd.5 (osd.5) 18 : cluster 0 2.3 scrub ok 2024-03-11T20:51:39.768 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:51:40.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:39 smithi146 bash[23809]: cluster 2024-03-11T20:51:38.867671+0000 mon.a (mon.0) 804 : cluster 0 mgrmap e22: y(active, since 5s), standbys: x 2024-03-11T20:51:40.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:39 smithi023 bash[21372]: cluster 2024-03-11T20:51:38.867671+0000 mon.a (mon.0) 804 : cluster 0 mgrmap e22: y(active, since 5s), standbys: x 2024-03-11T20:51:40.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:39 smithi023 bash[29146]: cluster 2024-03-11T20:51:38.867671+0000 mon.a (mon.0) 804 : cluster 0 mgrmap e22: y(active, since 5s), standbys: x 2024-03-11T20:51:41.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:40 smithi146 bash[23809]: cluster 2024-03-11T20:51:39.359570+0000 osd.3 (osd.3) 11 : cluster 0 4.17 scrub starts 2024-03-11T20:51:41.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:40 smithi146 bash[23809]: cluster 2024-03-11T20:51:39.361954+0000 osd.3 (osd.3) 12 : cluster 0 4.17 scrub ok 2024-03-11T20:51:41.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:40 smithi146 bash[23809]: cluster 2024-03-11T20:51:39.812695+0000 mgr.y (mgr.24380) 11 : cluster 0 pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 1 KiB/s rd, 1 op/s 2024-03-11T20:51:41.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:40 smithi146 bash[23809]: cluster 2024-03-11T20:51:39.907821+0000 osd.0 (osd.0) 21 : cluster 0 4.6 scrub starts 2024-03-11T20:51:41.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:40 smithi146 bash[23809]: cluster 2024-03-11T20:51:39.909276+0000 osd.0 (osd.0) 22 : cluster 0 4.6 scrub ok 2024-03-11T20:51:41.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:40 smithi146 bash[23809]: audit 2024-03-11T20:51:39.970672+0000 mon.a (mon.0) 805 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:51:41.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:40 smithi146 bash[23809]: audit 2024-03-11T20:51:39.980822+0000 mon.a (mon.0) 806 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:51:41.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:40 smithi146 bash[23809]: audit 2024-03-11T20:51:40.206245+0000 mon.a (mon.0) 807 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:51:41.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:40 smithi146 bash[23809]: audit 2024-03-11T20:51:40.217278+0000 mon.a (mon.0) 808 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:51:41.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:40 smithi023 bash[21372]: cluster 2024-03-11T20:51:39.359570+0000 osd.3 (osd.3) 11 : cluster 0 4.17 scrub starts 2024-03-11T20:51:41.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:40 smithi023 bash[21372]: cluster 2024-03-11T20:51:39.361954+0000 osd.3 (osd.3) 12 : cluster 0 4.17 scrub ok 2024-03-11T20:51:41.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:40 smithi023 bash[21372]: cluster 2024-03-11T20:51:39.812695+0000 mgr.y (mgr.24380) 11 : cluster 0 pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 1 KiB/s rd, 1 op/s 2024-03-11T20:51:41.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:40 smithi023 bash[21372]: cluster 2024-03-11T20:51:39.907821+0000 osd.0 (osd.0) 21 : cluster 0 4.6 scrub starts 2024-03-11T20:51:41.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:40 smithi023 bash[21372]: cluster 2024-03-11T20:51:39.909276+0000 osd.0 (osd.0) 22 : cluster 0 4.6 scrub ok 2024-03-11T20:51:41.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:40 smithi023 bash[21372]: audit 2024-03-11T20:51:39.970672+0000 mon.a (mon.0) 805 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:51:41.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:40 smithi023 bash[21372]: audit 2024-03-11T20:51:39.980822+0000 mon.a (mon.0) 806 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:51:41.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:40 smithi023 bash[21372]: audit 2024-03-11T20:51:40.206245+0000 mon.a (mon.0) 807 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:51:41.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:40 smithi023 bash[21372]: audit 2024-03-11T20:51:40.217278+0000 mon.a (mon.0) 808 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:51:41.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:40 smithi023 bash[29146]: cluster 2024-03-11T20:51:39.359570+0000 osd.3 (osd.3) 11 : cluster 0 4.17 scrub starts 2024-03-11T20:51:41.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:40 smithi023 bash[29146]: cluster 2024-03-11T20:51:39.361954+0000 osd.3 (osd.3) 12 : cluster 0 4.17 scrub ok 2024-03-11T20:51:41.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:40 smithi023 bash[29146]: cluster 2024-03-11T20:51:39.812695+0000 mgr.y (mgr.24380) 11 : cluster 0 pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 1 KiB/s rd, 1 op/s 2024-03-11T20:51:41.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:40 smithi023 bash[29146]: cluster 2024-03-11T20:51:39.907821+0000 osd.0 (osd.0) 21 : cluster 0 4.6 scrub starts 2024-03-11T20:51:41.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:40 smithi023 bash[29146]: cluster 2024-03-11T20:51:39.909276+0000 osd.0 (osd.0) 22 : cluster 0 4.6 scrub ok 2024-03-11T20:51:41.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:40 smithi023 bash[29146]: audit 2024-03-11T20:51:39.970672+0000 mon.a (mon.0) 805 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:51:41.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:40 smithi023 bash[29146]: audit 2024-03-11T20:51:39.980822+0000 mon.a (mon.0) 806 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:51:41.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:40 smithi023 bash[29146]: audit 2024-03-11T20:51:40.206245+0000 mon.a (mon.0) 807 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:51:41.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:40 smithi023 bash[29146]: audit 2024-03-11T20:51:40.217278+0000 mon.a (mon.0) 808 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:51:42.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:42 smithi023 bash[21372]: cluster 2024-03-11T20:51:40.316341+0000 osd.3 (osd.3) 13 : cluster 0 4.14 scrub starts 2024-03-11T20:51:42.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:42 smithi023 bash[21372]: cluster 2024-03-11T20:51:40.319222+0000 osd.3 (osd.3) 14 : cluster 0 4.14 scrub ok 2024-03-11T20:51:42.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:42 smithi023 bash[21372]: cluster 2024-03-11T20:51:40.972755+0000 osd.1 (osd.1) 17 : cluster 0 2.9 deep-scrub starts 2024-03-11T20:51:42.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:42 smithi023 bash[21372]: cluster 2024-03-11T20:51:40.974128+0000 osd.1 (osd.1) 18 : cluster 0 2.9 deep-scrub ok 2024-03-11T20:51:42.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:42 smithi023 bash[21372]: audit 2024-03-11T20:51:41.038580+0000 mon.a (mon.0) 809 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:51:42.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:42 smithi023 bash[21372]: audit 2024-03-11T20:51:41.043965+0000 mon.a (mon.0) 810 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:51:42.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:42 smithi023 bash[21372]: audit 2024-03-11T20:51:41.045310+0000 mon.a (mon.0) 811 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:51:42.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:42 smithi023 bash[21372]: audit 2024-03-11T20:51:41.046060+0000 mon.a (mon.0) 812 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:51:42.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:42 smithi023 bash[21372]: audit 2024-03-11T20:51:41.046715+0000 mon.a (mon.0) 813 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:51:42.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:42 smithi023 bash[21372]: audit 2024-03-11T20:51:41.047333+0000 mon.a (mon.0) 814 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:51:42.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:42 smithi023 bash[21372]: cephadm 2024-03-11T20:51:41.047849+0000 mgr.y (mgr.24380) 12 : cephadm 1 Adjusting osd_memory_target on smithi146 to 3804M 2024-03-11T20:51:42.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:42 smithi023 bash[21372]: audit 2024-03-11T20:51:41.052098+0000 mon.a (mon.0) 815 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:51:42.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:42 smithi023 bash[21372]: audit 2024-03-11T20:51:41.223438+0000 mon.a (mon.0) 816 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:51:42.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:42 smithi023 bash[21372]: audit 2024-03-11T20:51:41.229039+0000 mon.a (mon.0) 817 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:51:42.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:42 smithi023 bash[21372]: audit 2024-03-11T20:51:41.230358+0000 mon.a (mon.0) 818 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:51:42.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:42 smithi023 bash[21372]: audit 2024-03-11T20:51:41.231258+0000 mon.a (mon.0) 819 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:51:42.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:42 smithi023 bash[21372]: audit 2024-03-11T20:51:41.232047+0000 mon.a (mon.0) 820 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:51:42.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:42 smithi023 bash[21372]: audit 2024-03-11T20:51:41.232796+0000 mon.a (mon.0) 821 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:51:42.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:42 smithi023 bash[21372]: cephadm 2024-03-11T20:51:41.233372+0000 mgr.y (mgr.24380) 13 : cephadm 1 Adjusting osd_memory_target on smithi023 to 3804M 2024-03-11T20:51:42.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:42 smithi023 bash[21372]: audit 2024-03-11T20:51:41.237272+0000 mon.a (mon.0) 822 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:51:42.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:42 smithi023 bash[21372]: audit 2024-03-11T20:51:41.239731+0000 mon.a (mon.0) 823 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:51:42.286 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:42 smithi023 bash[29146]: cluster 2024-03-11T20:51:40.316341+0000 osd.3 (osd.3) 13 : cluster 0 4.14 scrub starts 2024-03-11T20:51:42.286 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:42 smithi023 bash[29146]: cluster 2024-03-11T20:51:40.319222+0000 osd.3 (osd.3) 14 : cluster 0 4.14 scrub ok 2024-03-11T20:51:42.286 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:42 smithi023 bash[29146]: cluster 2024-03-11T20:51:40.972755+0000 osd.1 (osd.1) 17 : cluster 0 2.9 deep-scrub starts 2024-03-11T20:51:42.286 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:42 smithi023 bash[29146]: cluster 2024-03-11T20:51:40.974128+0000 osd.1 (osd.1) 18 : cluster 0 2.9 deep-scrub ok 2024-03-11T20:51:42.286 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:42 smithi023 bash[29146]: audit 2024-03-11T20:51:41.038580+0000 mon.a (mon.0) 809 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:51:42.286 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:42 smithi023 bash[29146]: audit 2024-03-11T20:51:41.043965+0000 mon.a (mon.0) 810 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:51:42.286 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:42 smithi023 bash[29146]: audit 2024-03-11T20:51:41.045310+0000 mon.a (mon.0) 811 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:51:42.286 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:42 smithi023 bash[29146]: audit 2024-03-11T20:51:41.046060+0000 mon.a (mon.0) 812 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:51:42.286 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:42 smithi023 bash[29146]: audit 2024-03-11T20:51:41.046715+0000 mon.a (mon.0) 813 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:51:42.286 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:42 smithi023 bash[29146]: audit 2024-03-11T20:51:41.047333+0000 mon.a (mon.0) 814 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:51:42.286 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:42 smithi023 bash[29146]: cephadm 2024-03-11T20:51:41.047849+0000 mgr.y (mgr.24380) 12 : cephadm 1 Adjusting osd_memory_target on smithi146 to 3804M 2024-03-11T20:51:42.286 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:42 smithi023 bash[29146]: audit 2024-03-11T20:51:41.052098+0000 mon.a (mon.0) 815 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:51:42.287 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:42 smithi023 bash[29146]: audit 2024-03-11T20:51:41.223438+0000 mon.a (mon.0) 816 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:51:42.287 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:42 smithi023 bash[29146]: audit 2024-03-11T20:51:41.229039+0000 mon.a (mon.0) 817 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:51:42.287 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:42 smithi023 bash[29146]: audit 2024-03-11T20:51:41.230358+0000 mon.a (mon.0) 818 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:51:42.287 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:42 smithi023 bash[29146]: audit 2024-03-11T20:51:41.231258+0000 mon.a (mon.0) 819 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:51:42.287 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:42 smithi023 bash[29146]: audit 2024-03-11T20:51:41.232047+0000 mon.a (mon.0) 820 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:51:42.287 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:42 smithi023 bash[29146]: audit 2024-03-11T20:51:41.232796+0000 mon.a (mon.0) 821 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:51:42.287 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:42 smithi023 bash[29146]: cephadm 2024-03-11T20:51:41.233372+0000 mgr.y (mgr.24380) 13 : cephadm 1 Adjusting osd_memory_target on smithi023 to 3804M 2024-03-11T20:51:42.287 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:42 smithi023 bash[29146]: audit 2024-03-11T20:51:41.237272+0000 mon.a (mon.0) 822 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:51:42.287 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:42 smithi023 bash[29146]: audit 2024-03-11T20:51:41.239731+0000 mon.a (mon.0) 823 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:51:42.287 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:42 smithi023 bash[21372]: audit 2024-03-11T20:51:41.240488+0000 mon.a (mon.0) 824 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:51:42.287 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:42 smithi023 bash[21372]: cephadm 2024-03-11T20:51:41.241605+0000 mgr.y (mgr.24380) 14 : cephadm 1 Updating smithi023:/etc/ceph/ceph.conf 2024-03-11T20:51:42.288 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:42 smithi023 bash[21372]: cephadm 2024-03-11T20:51:41.241858+0000 mgr.y (mgr.24380) 15 : cephadm 1 Updating smithi146:/etc/ceph/ceph.conf 2024-03-11T20:51:42.288 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:42 smithi023 bash[21372]: cluster 2024-03-11T20:51:41.867452+0000 osd.0 (osd.0) 23 : cluster 0 6.9 scrub starts 2024-03-11T20:51:42.288 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:42 smithi023 bash[21372]: cluster 2024-03-11T20:51:41.869021+0000 osd.0 (osd.0) 24 : cluster 0 6.9 scrub ok 2024-03-11T20:51:42.288 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:42 smithi023 bash[21372]: audit 2024-03-11T20:51:41.960743+0000 mon.a (mon.0) 825 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:51:42.288 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:42 smithi023 bash[21372]: audit 2024-03-11T20:51:41.967148+0000 mon.a (mon.0) 826 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:51:42.288 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:42 smithi023 bash[21372]: audit 2024-03-11T20:51:41.973126+0000 mon.a (mon.0) 827 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:51:42.288 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:42 smithi023 bash[21372]: audit 2024-03-11T20:51:41.978875+0000 mon.a (mon.0) 828 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:51:42.288 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:42 smithi023 bash[21372]: audit 2024-03-11T20:51:41.985100+0000 mon.a (mon.0) 829 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:51:42.288 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:42 smithi023 bash[29146]: audit 2024-03-11T20:51:41.240488+0000 mon.a (mon.0) 824 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:51:42.288 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:42 smithi023 bash[29146]: cephadm 2024-03-11T20:51:41.241605+0000 mgr.y (mgr.24380) 14 : cephadm 1 Updating smithi023:/etc/ceph/ceph.conf 2024-03-11T20:51:42.288 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:42 smithi023 bash[29146]: cephadm 2024-03-11T20:51:41.241858+0000 mgr.y (mgr.24380) 15 : cephadm 1 Updating smithi146:/etc/ceph/ceph.conf 2024-03-11T20:51:42.288 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:42 smithi023 bash[29146]: cluster 2024-03-11T20:51:41.867452+0000 osd.0 (osd.0) 23 : cluster 0 6.9 scrub starts 2024-03-11T20:51:42.288 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:42 smithi023 bash[29146]: cluster 2024-03-11T20:51:41.869021+0000 osd.0 (osd.0) 24 : cluster 0 6.9 scrub ok 2024-03-11T20:51:42.289 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:42 smithi023 bash[29146]: audit 2024-03-11T20:51:41.960743+0000 mon.a (mon.0) 825 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:51:42.289 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:42 smithi023 bash[29146]: audit 2024-03-11T20:51:41.967148+0000 mon.a (mon.0) 826 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:51:42.289 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:42 smithi023 bash[29146]: audit 2024-03-11T20:51:41.973126+0000 mon.a (mon.0) 827 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:51:42.289 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:42 smithi023 bash[29146]: audit 2024-03-11T20:51:41.978875+0000 mon.a (mon.0) 828 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:51:42.289 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:42 smithi023 bash[29146]: audit 2024-03-11T20:51:41.985100+0000 mon.a (mon.0) 829 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:51:42.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:42 smithi146 bash[23809]: cluster 2024-03-11T20:51:40.316341+0000 osd.3 (osd.3) 13 : cluster 0 4.14 scrub starts 2024-03-11T20:51:42.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:42 smithi146 bash[23809]: cluster 2024-03-11T20:51:40.319222+0000 osd.3 (osd.3) 14 : cluster 0 4.14 scrub ok 2024-03-11T20:51:42.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:42 smithi146 bash[23809]: cluster 2024-03-11T20:51:40.972755+0000 osd.1 (osd.1) 17 : cluster 0 2.9 deep-scrub starts 2024-03-11T20:51:42.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:42 smithi146 bash[23809]: cluster 2024-03-11T20:51:40.974128+0000 osd.1 (osd.1) 18 : cluster 0 2.9 deep-scrub ok 2024-03-11T20:51:42.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:42 smithi146 bash[23809]: audit 2024-03-11T20:51:41.038580+0000 mon.a (mon.0) 809 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:51:42.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:42 smithi146 bash[23809]: audit 2024-03-11T20:51:41.043965+0000 mon.a (mon.0) 810 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:51:42.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:42 smithi146 bash[23809]: audit 2024-03-11T20:51:41.045310+0000 mon.a (mon.0) 811 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:51:42.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:42 smithi146 bash[23809]: audit 2024-03-11T20:51:41.046060+0000 mon.a (mon.0) 812 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:51:42.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:42 smithi146 bash[23809]: audit 2024-03-11T20:51:41.046715+0000 mon.a (mon.0) 813 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:51:42.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:42 smithi146 bash[23809]: audit 2024-03-11T20:51:41.047333+0000 mon.a (mon.0) 814 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:51:42.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:42 smithi146 bash[23809]: cephadm 2024-03-11T20:51:41.047849+0000 mgr.y (mgr.24380) 12 : cephadm 1 Adjusting osd_memory_target on smithi146 to 3804M 2024-03-11T20:51:42.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:42 smithi146 bash[23809]: audit 2024-03-11T20:51:41.052098+0000 mon.a (mon.0) 815 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:51:42.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:42 smithi146 bash[23809]: audit 2024-03-11T20:51:41.223438+0000 mon.a (mon.0) 816 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:51:42.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:42 smithi146 bash[23809]: audit 2024-03-11T20:51:41.229039+0000 mon.a (mon.0) 817 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:51:42.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:42 smithi146 bash[23809]: audit 2024-03-11T20:51:41.230358+0000 mon.a (mon.0) 818 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:51:42.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:42 smithi146 bash[23809]: audit 2024-03-11T20:51:41.231258+0000 mon.a (mon.0) 819 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:51:42.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:42 smithi146 bash[23809]: audit 2024-03-11T20:51:41.232047+0000 mon.a (mon.0) 820 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:51:42.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:42 smithi146 bash[23809]: audit 2024-03-11T20:51:41.232796+0000 mon.a (mon.0) 821 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:51:42.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:42 smithi146 bash[23809]: cephadm 2024-03-11T20:51:41.233372+0000 mgr.y (mgr.24380) 13 : cephadm 1 Adjusting osd_memory_target on smithi023 to 3804M 2024-03-11T20:51:42.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:42 smithi146 bash[23809]: audit 2024-03-11T20:51:41.237272+0000 mon.a (mon.0) 822 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:51:42.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:42 smithi146 bash[23809]: audit 2024-03-11T20:51:41.239731+0000 mon.a (mon.0) 823 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:51:42.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:42 smithi146 bash[23809]: audit 2024-03-11T20:51:41.240488+0000 mon.a (mon.0) 824 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:51:42.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:42 smithi146 bash[23809]: cephadm 2024-03-11T20:51:41.241605+0000 mgr.y (mgr.24380) 14 : cephadm 1 Updating smithi023:/etc/ceph/ceph.conf 2024-03-11T20:51:42.513 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:42 smithi146 bash[23809]: cephadm 2024-03-11T20:51:41.241858+0000 mgr.y (mgr.24380) 15 : cephadm 1 Updating smithi146:/etc/ceph/ceph.conf 2024-03-11T20:51:42.513 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:42 smithi146 bash[23809]: cluster 2024-03-11T20:51:41.867452+0000 osd.0 (osd.0) 23 : cluster 0 6.9 scrub starts 2024-03-11T20:51:42.513 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:42 smithi146 bash[23809]: cluster 2024-03-11T20:51:41.869021+0000 osd.0 (osd.0) 24 : cluster 0 6.9 scrub ok 2024-03-11T20:51:42.513 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:42 smithi146 bash[23809]: audit 2024-03-11T20:51:41.960743+0000 mon.a (mon.0) 825 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:51:42.513 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:42 smithi146 bash[23809]: audit 2024-03-11T20:51:41.967148+0000 mon.a (mon.0) 826 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:51:42.513 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:42 smithi146 bash[23809]: audit 2024-03-11T20:51:41.973126+0000 mon.a (mon.0) 827 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:51:42.513 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:42 smithi146 bash[23809]: audit 2024-03-11T20:51:41.978875+0000 mon.a (mon.0) 828 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:51:42.513 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:42 smithi146 bash[23809]: audit 2024-03-11T20:51:41.985100+0000 mon.a (mon.0) 829 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:51:43.044 INFO:teuthology.orchestra.run.smithi023.stdout:[client.0] 2024-03-11T20:51:43.044 INFO:teuthology.orchestra.run.smithi023.stdout: key = AQDfbu9lY9k3AhAAo2QIroUZDvFgFxExkVFYNg== 2024-03-11T20:51:43.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:43 smithi146 bash[23809]: cluster 2024-03-11T20:51:41.339742+0000 osd.3 (osd.3) 15 : cluster 0 4.1d scrub starts 2024-03-11T20:51:43.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:43 smithi146 bash[23809]: cluster 2024-03-11T20:51:41.341462+0000 osd.3 (osd.3) 16 : cluster 0 4.1d scrub ok 2024-03-11T20:51:43.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:43 smithi146 bash[23809]: cephadm 2024-03-11T20:51:41.456543+0000 mgr.y (mgr.24380) 16 : cephadm 1 Updating smithi023:/var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T20:51:43.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:43 smithi146 bash[23809]: cephadm 2024-03-11T20:51:41.456863+0000 mgr.y (mgr.24380) 17 : cephadm 1 Updating smithi146:/var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T20:51:43.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:43 smithi146 bash[23809]: cephadm 2024-03-11T20:51:41.621873+0000 mgr.y (mgr.24380) 18 : cephadm 1 Updating smithi146:/etc/ceph/ceph.client.admin.keyring 2024-03-11T20:51:43.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:43 smithi146 bash[23809]: cephadm 2024-03-11T20:51:41.622051+0000 mgr.y (mgr.24380) 19 : cephadm 1 Updating smithi023:/etc/ceph/ceph.client.admin.keyring 2024-03-11T20:51:43.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:43 smithi146 bash[23809]: cephadm 2024-03-11T20:51:41.787569+0000 mgr.y (mgr.24380) 20 : cephadm 1 Updating smithi023:/var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T20:51:43.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:43 smithi146 bash[23809]: cephadm 2024-03-11T20:51:41.788188+0000 mgr.y (mgr.24380) 21 : cephadm 1 Updating smithi146:/var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T20:51:43.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:43 smithi146 bash[23809]: cluster 2024-03-11T20:51:41.813221+0000 mgr.y (mgr.24380) 22 : cluster 0 pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-11T20:51:43.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:43 smithi146 bash[23809]: cluster 2024-03-11T20:51:41.930405+0000 osd.1 (osd.1) 19 : cluster 0 3.4 scrub starts 2024-03-11T20:51:43.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:43 smithi146 bash[23809]: cluster 2024-03-11T20:51:41.932888+0000 osd.1 (osd.1) 20 : cluster 0 3.4 scrub ok 2024-03-11T20:51:43.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:43 smithi146 bash[23809]: cephadm 2024-03-11T20:51:42.012790+0000 mgr.y (mgr.24380) 23 : cephadm 1 Deploying daemon alertmanager.a on smithi023 2024-03-11T20:51:43.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:43 smithi146 bash[23809]: cluster 2024-03-11T20:51:42.592529+0000 osd.4 (osd.4) 13 : cluster 0 3.3 scrub starts 2024-03-11T20:51:43.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:43 smithi146 bash[23809]: cluster 2024-03-11T20:51:42.596478+0000 osd.4 (osd.4) 14 : cluster 0 3.3 scrub ok 2024-03-11T20:51:43.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:43 smithi146 bash[23809]: audit 2024-03-11T20:51:43.036478+0000 mon.c (mon.2) 20 : audit 1 from='client.? 172.21.15.23:0/4293877918' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T20:51:43.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:43 smithi146 bash[23809]: audit 2024-03-11T20:51:43.036973+0000 mon.a (mon.0) 830 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T20:51:43.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:43 smithi023 bash[21372]: cluster 2024-03-11T20:51:41.339742+0000 osd.3 (osd.3) 15 : cluster 0 4.1d scrub starts 2024-03-11T20:51:43.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:43 smithi023 bash[21372]: cluster 2024-03-11T20:51:41.341462+0000 osd.3 (osd.3) 16 : cluster 0 4.1d scrub ok 2024-03-11T20:51:43.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:43 smithi023 bash[21372]: cephadm 2024-03-11T20:51:41.456543+0000 mgr.y (mgr.24380) 16 : cephadm 1 Updating smithi023:/var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T20:51:43.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:43 smithi023 bash[21372]: cephadm 2024-03-11T20:51:41.456863+0000 mgr.y (mgr.24380) 17 : cephadm 1 Updating smithi146:/var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T20:51:43.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:43 smithi023 bash[21372]: cephadm 2024-03-11T20:51:41.621873+0000 mgr.y (mgr.24380) 18 : cephadm 1 Updating smithi146:/etc/ceph/ceph.client.admin.keyring 2024-03-11T20:51:43.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:43 smithi023 bash[21372]: cephadm 2024-03-11T20:51:41.622051+0000 mgr.y (mgr.24380) 19 : cephadm 1 Updating smithi023:/etc/ceph/ceph.client.admin.keyring 2024-03-11T20:51:43.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:43 smithi023 bash[21372]: cephadm 2024-03-11T20:51:41.787569+0000 mgr.y (mgr.24380) 20 : cephadm 1 Updating smithi023:/var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T20:51:43.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:43 smithi023 bash[21372]: cephadm 2024-03-11T20:51:41.788188+0000 mgr.y (mgr.24380) 21 : cephadm 1 Updating smithi146:/var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T20:51:43.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:43 smithi023 bash[21372]: cluster 2024-03-11T20:51:41.813221+0000 mgr.y (mgr.24380) 22 : cluster 0 pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-11T20:51:43.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:43 smithi023 bash[21372]: cluster 2024-03-11T20:51:41.930405+0000 osd.1 (osd.1) 19 : cluster 0 3.4 scrub starts 2024-03-11T20:51:43.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:43 smithi023 bash[21372]: cluster 2024-03-11T20:51:41.932888+0000 osd.1 (osd.1) 20 : cluster 0 3.4 scrub ok 2024-03-11T20:51:43.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:43 smithi023 bash[21372]: cephadm 2024-03-11T20:51:42.012790+0000 mgr.y (mgr.24380) 23 : cephadm 1 Deploying daemon alertmanager.a on smithi023 2024-03-11T20:51:43.536 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:43 smithi023 bash[21372]: cluster 2024-03-11T20:51:42.592529+0000 osd.4 (osd.4) 13 : cluster 0 3.3 scrub starts 2024-03-11T20:51:43.536 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:43 smithi023 bash[21372]: cluster 2024-03-11T20:51:42.596478+0000 osd.4 (osd.4) 14 : cluster 0 3.3 scrub ok 2024-03-11T20:51:43.536 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:43 smithi023 bash[21372]: audit 2024-03-11T20:51:43.036478+0000 mon.c (mon.2) 20 : audit 1 from='client.? 172.21.15.23:0/4293877918' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T20:51:43.536 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:43 smithi023 bash[21372]: audit 2024-03-11T20:51:43.036973+0000 mon.a (mon.0) 830 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T20:51:43.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:43 smithi023 bash[29146]: cluster 2024-03-11T20:51:41.339742+0000 osd.3 (osd.3) 15 : cluster 0 4.1d scrub starts 2024-03-11T20:51:43.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:43 smithi023 bash[29146]: cluster 2024-03-11T20:51:41.341462+0000 osd.3 (osd.3) 16 : cluster 0 4.1d scrub ok 2024-03-11T20:51:43.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:43 smithi023 bash[29146]: cephadm 2024-03-11T20:51:41.456543+0000 mgr.y (mgr.24380) 16 : cephadm 1 Updating smithi023:/var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T20:51:43.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:43 smithi023 bash[29146]: cephadm 2024-03-11T20:51:41.456863+0000 mgr.y (mgr.24380) 17 : cephadm 1 Updating smithi146:/var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T20:51:43.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:43 smithi023 bash[29146]: cephadm 2024-03-11T20:51:41.621873+0000 mgr.y (mgr.24380) 18 : cephadm 1 Updating smithi146:/etc/ceph/ceph.client.admin.keyring 2024-03-11T20:51:43.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:43 smithi023 bash[29146]: cephadm 2024-03-11T20:51:41.622051+0000 mgr.y (mgr.24380) 19 : cephadm 1 Updating smithi023:/etc/ceph/ceph.client.admin.keyring 2024-03-11T20:51:43.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:43 smithi023 bash[29146]: cephadm 2024-03-11T20:51:41.787569+0000 mgr.y (mgr.24380) 20 : cephadm 1 Updating smithi023:/var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T20:51:43.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:43 smithi023 bash[29146]: cephadm 2024-03-11T20:51:41.788188+0000 mgr.y (mgr.24380) 21 : cephadm 1 Updating smithi146:/var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T20:51:43.537 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:43 smithi023 bash[29146]: cluster 2024-03-11T20:51:41.813221+0000 mgr.y (mgr.24380) 22 : cluster 0 pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-11T20:51:43.537 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:43 smithi023 bash[29146]: cluster 2024-03-11T20:51:41.930405+0000 osd.1 (osd.1) 19 : cluster 0 3.4 scrub starts 2024-03-11T20:51:43.537 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:43 smithi023 bash[29146]: cluster 2024-03-11T20:51:41.932888+0000 osd.1 (osd.1) 20 : cluster 0 3.4 scrub ok 2024-03-11T20:51:43.537 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:43 smithi023 bash[29146]: cephadm 2024-03-11T20:51:42.012790+0000 mgr.y (mgr.24380) 23 : cephadm 1 Deploying daemon alertmanager.a on smithi023 2024-03-11T20:51:43.537 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:43 smithi023 bash[29146]: cluster 2024-03-11T20:51:42.592529+0000 osd.4 (osd.4) 13 : cluster 0 3.3 scrub starts 2024-03-11T20:51:43.537 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:43 smithi023 bash[29146]: cluster 2024-03-11T20:51:42.596478+0000 osd.4 (osd.4) 14 : cluster 0 3.3 scrub ok 2024-03-11T20:51:43.537 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:43 smithi023 bash[29146]: audit 2024-03-11T20:51:43.036478+0000 mon.c (mon.2) 20 : audit 1 from='client.? 172.21.15.23:0/4293877918' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T20:51:43.537 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:43 smithi023 bash[29146]: audit 2024-03-11T20:51:43.036973+0000 mon.a (mon.0) 830 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T20:51:44.036 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-11T20:51:44.036 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-11T20:51:44.036 DEBUG:teuthology.orchestra.run.smithi023:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-11T20:51:44.062 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-11T20:51:44.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:44 smithi146 bash[23809]: cluster 2024-03-11T20:51:42.116460+0000 osd.7 (osd.7) 15 : cluster 0 2.1c scrub starts 2024-03-11T20:51:44.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:44 smithi146 bash[23809]: cluster 2024-03-11T20:51:42.118931+0000 osd.7 (osd.7) 16 : cluster 0 2.1c scrub ok 2024-03-11T20:51:44.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:44 smithi146 bash[23809]: cluster 2024-03-11T20:51:42.298257+0000 osd.3 (osd.3) 17 : cluster 0 3.b scrub starts 2024-03-11T20:51:44.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:44 smithi146 bash[23809]: cluster 2024-03-11T20:51:42.300715+0000 osd.3 (osd.3) 18 : cluster 0 3.b scrub ok 2024-03-11T20:51:44.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:44 smithi146 bash[23809]: cluster 2024-03-11T20:51:42.706508+0000 osd.6 (osd.6) 11 : cluster 0 2.17 scrub starts 2024-03-11T20:51:44.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:44 smithi146 bash[23809]: cluster 2024-03-11T20:51:42.708753+0000 osd.6 (osd.6) 12 : cluster 0 2.17 scrub ok 2024-03-11T20:51:44.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:44 smithi146 bash[23809]: cluster 2024-03-11T20:51:42.809354+0000 osd.2 (osd.2) 17 : cluster 0 4.f scrub starts 2024-03-11T20:51:44.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:44 smithi146 bash[23809]: cluster 2024-03-11T20:51:42.811348+0000 osd.2 (osd.2) 18 : cluster 0 4.f scrub ok 2024-03-11T20:51:44.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:44 smithi146 bash[23809]: audit 2024-03-11T20:51:43.043738+0000 mon.a (mon.0) 831 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-11T20:51:44.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:44 smithi146 bash[23809]: cluster 2024-03-11T20:51:43.633043+0000 osd.4 (osd.4) 15 : cluster 0 4.18 scrub starts 2024-03-11T20:51:44.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:44 smithi146 bash[23809]: cluster 2024-03-11T20:51:43.635497+0000 osd.4 (osd.4) 16 : cluster 0 4.18 scrub ok 2024-03-11T20:51:44.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:44 smithi146 bash[23809]: cluster 2024-03-11T20:51:43.907682+0000 osd.0 (osd.0) 25 : cluster 0 6.18 scrub starts 2024-03-11T20:51:44.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:44 smithi146 bash[23809]: cluster 2024-03-11T20:51:43.910065+0000 osd.0 (osd.0) 26 : cluster 0 6.18 scrub ok 2024-03-11T20:51:44.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:44 smithi023 bash[21372]: cluster 2024-03-11T20:51:42.116460+0000 osd.7 (osd.7) 15 : cluster 0 2.1c scrub starts 2024-03-11T20:51:44.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:44 smithi023 bash[21372]: cluster 2024-03-11T20:51:42.118931+0000 osd.7 (osd.7) 16 : cluster 0 2.1c scrub ok 2024-03-11T20:51:44.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:44 smithi023 bash[21372]: cluster 2024-03-11T20:51:42.298257+0000 osd.3 (osd.3) 17 : cluster 0 3.b scrub starts 2024-03-11T20:51:44.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:44 smithi023 bash[21372]: cluster 2024-03-11T20:51:42.300715+0000 osd.3 (osd.3) 18 : cluster 0 3.b scrub ok 2024-03-11T20:51:44.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:44 smithi023 bash[21372]: cluster 2024-03-11T20:51:42.706508+0000 osd.6 (osd.6) 11 : cluster 0 2.17 scrub starts 2024-03-11T20:51:44.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:44 smithi023 bash[21372]: cluster 2024-03-11T20:51:42.708753+0000 osd.6 (osd.6) 12 : cluster 0 2.17 scrub ok 2024-03-11T20:51:44.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:44 smithi023 bash[21372]: cluster 2024-03-11T20:51:42.809354+0000 osd.2 (osd.2) 17 : cluster 0 4.f scrub starts 2024-03-11T20:51:44.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:44 smithi023 bash[21372]: cluster 2024-03-11T20:51:42.811348+0000 osd.2 (osd.2) 18 : cluster 0 4.f scrub ok 2024-03-11T20:51:44.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:44 smithi023 bash[21372]: audit 2024-03-11T20:51:43.043738+0000 mon.a (mon.0) 831 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-11T20:51:44.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:44 smithi023 bash[21372]: cluster 2024-03-11T20:51:43.633043+0000 osd.4 (osd.4) 15 : cluster 0 4.18 scrub starts 2024-03-11T20:51:44.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:44 smithi023 bash[21372]: cluster 2024-03-11T20:51:43.635497+0000 osd.4 (osd.4) 16 : cluster 0 4.18 scrub ok 2024-03-11T20:51:44.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:44 smithi023 bash[21372]: cluster 2024-03-11T20:51:43.907682+0000 osd.0 (osd.0) 25 : cluster 0 6.18 scrub starts 2024-03-11T20:51:44.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:44 smithi023 bash[21372]: cluster 2024-03-11T20:51:43.910065+0000 osd.0 (osd.0) 26 : cluster 0 6.18 scrub ok 2024-03-11T20:51:44.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:44 smithi023 bash[29146]: cluster 2024-03-11T20:51:42.116460+0000 osd.7 (osd.7) 15 : cluster 0 2.1c scrub starts 2024-03-11T20:51:44.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:44 smithi023 bash[29146]: cluster 2024-03-11T20:51:42.118931+0000 osd.7 (osd.7) 16 : cluster 0 2.1c scrub ok 2024-03-11T20:51:44.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:44 smithi023 bash[29146]: cluster 2024-03-11T20:51:42.298257+0000 osd.3 (osd.3) 17 : cluster 0 3.b scrub starts 2024-03-11T20:51:44.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:44 smithi023 bash[29146]: cluster 2024-03-11T20:51:42.300715+0000 osd.3 (osd.3) 18 : cluster 0 3.b scrub ok 2024-03-11T20:51:44.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:44 smithi023 bash[29146]: cluster 2024-03-11T20:51:42.706508+0000 osd.6 (osd.6) 11 : cluster 0 2.17 scrub starts 2024-03-11T20:51:44.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:44 smithi023 bash[29146]: cluster 2024-03-11T20:51:42.708753+0000 osd.6 (osd.6) 12 : cluster 0 2.17 scrub ok 2024-03-11T20:51:44.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:44 smithi023 bash[29146]: cluster 2024-03-11T20:51:42.809354+0000 osd.2 (osd.2) 17 : cluster 0 4.f scrub starts 2024-03-11T20:51:44.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:44 smithi023 bash[29146]: cluster 2024-03-11T20:51:42.811348+0000 osd.2 (osd.2) 18 : cluster 0 4.f scrub ok 2024-03-11T20:51:44.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:44 smithi023 bash[29146]: audit 2024-03-11T20:51:43.043738+0000 mon.a (mon.0) 831 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-11T20:51:44.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:44 smithi023 bash[29146]: cluster 2024-03-11T20:51:43.633043+0000 osd.4 (osd.4) 15 : cluster 0 4.18 scrub starts 2024-03-11T20:51:44.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:44 smithi023 bash[29146]: cluster 2024-03-11T20:51:43.635497+0000 osd.4 (osd.4) 16 : cluster 0 4.18 scrub ok 2024-03-11T20:51:44.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:44 smithi023 bash[29146]: cluster 2024-03-11T20:51:43.907682+0000 osd.0 (osd.0) 25 : cluster 0 6.18 scrub starts 2024-03-11T20:51:44.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:44 smithi023 bash[29146]: cluster 2024-03-11T20:51:43.910065+0000 osd.0 (osd.0) 26 : cluster 0 6.18 scrub ok 2024-03-11T20:51:45.070 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 20:51:44 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T20:51:45.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:45 smithi146 bash[23809]: cluster 2024-03-11T20:51:43.813829+0000 mgr.y (mgr.24380) 24 : cluster 0 pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-11T20:51:45.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:45 smithi146 bash[23809]: cluster 2024-03-11T20:51:44.590657+0000 osd.4 (osd.4) 17 : cluster 0 6.2 scrub starts 2024-03-11T20:51:45.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:45 smithi146 bash[23809]: cluster 2024-03-11T20:51:44.593244+0000 osd.4 (osd.4) 18 : cluster 0 6.2 scrub ok 2024-03-11T20:51:45.533 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:45 smithi023 bash[29146]: cluster 2024-03-11T20:51:43.813829+0000 mgr.y (mgr.24380) 24 : cluster 0 pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-11T20:51:45.534 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:45 smithi023 bash[29146]: cluster 2024-03-11T20:51:44.590657+0000 osd.4 (osd.4) 17 : cluster 0 6.2 scrub starts 2024-03-11T20:51:45.534 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:45 smithi023 bash[29146]: cluster 2024-03-11T20:51:44.593244+0000 osd.4 (osd.4) 18 : cluster 0 6.2 scrub ok 2024-03-11T20:51:45.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:45 smithi023 bash[21372]: cluster 2024-03-11T20:51:43.813829+0000 mgr.y (mgr.24380) 24 : cluster 0 pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-11T20:51:45.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:45 smithi023 bash[21372]: cluster 2024-03-11T20:51:44.590657+0000 osd.4 (osd.4) 17 : cluster 0 6.2 scrub starts 2024-03-11T20:51:45.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:45 smithi023 bash[21372]: cluster 2024-03-11T20:51:44.593244+0000 osd.4 (osd.4) 18 : cluster 0 6.2 scrub ok 2024-03-11T20:51:46.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:46 smithi146 bash[23809]: audit 2024-03-11T20:51:44.821949+0000 mgr.y (mgr.24380) 25 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:51:46.533 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:46 smithi023 bash[29146]: audit 2024-03-11T20:51:44.821949+0000 mgr.y (mgr.24380) 25 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:51:46.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:46 smithi023 bash[21372]: audit 2024-03-11T20:51:44.821949+0000 mgr.y (mgr.24380) 25 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:51:47.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:47 smithi146 bash[23809]: cluster 2024-03-11T20:51:45.756627+0000 osd.2 (osd.2) 19 : cluster 0 6.14 scrub starts 2024-03-11T20:51:47.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:47 smithi146 bash[23809]: cluster 2024-03-11T20:51:45.759129+0000 osd.2 (osd.2) 20 : cluster 0 6.14 scrub ok 2024-03-11T20:51:47.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:47 smithi146 bash[23809]: cluster 2024-03-11T20:51:45.815299+0000 mgr.y (mgr.24380) 26 : cluster 0 pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-11T20:51:47.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:47 smithi146 bash[23809]: cluster 2024-03-11T20:51:45.877464+0000 osd.5 (osd.5) 19 : cluster 0 3.16 scrub starts 2024-03-11T20:51:47.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:47 smithi146 bash[23809]: cluster 2024-03-11T20:51:45.881253+0000 osd.5 (osd.5) 20 : cluster 0 3.16 scrub ok 2024-03-11T20:51:47.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:47 smithi146 bash[23809]: cluster 2024-03-11T20:51:46.597201+0000 osd.4 (osd.4) 19 : cluster 0 4.13 scrub starts 2024-03-11T20:51:47.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:47 smithi146 bash[23809]: cluster 2024-03-11T20:51:46.599556+0000 osd.4 (osd.4) 20 : cluster 0 4.13 scrub ok 2024-03-11T20:51:47.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:47 smithi023 bash[21372]: cluster 2024-03-11T20:51:45.756627+0000 osd.2 (osd.2) 19 : cluster 0 6.14 scrub starts 2024-03-11T20:51:47.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:47 smithi023 bash[21372]: cluster 2024-03-11T20:51:45.759129+0000 osd.2 (osd.2) 20 : cluster 0 6.14 scrub ok 2024-03-11T20:51:47.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:47 smithi023 bash[21372]: cluster 2024-03-11T20:51:45.815299+0000 mgr.y (mgr.24380) 26 : cluster 0 pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-11T20:51:47.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:47 smithi023 bash[21372]: cluster 2024-03-11T20:51:45.877464+0000 osd.5 (osd.5) 19 : cluster 0 3.16 scrub starts 2024-03-11T20:51:47.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:47 smithi023 bash[21372]: cluster 2024-03-11T20:51:45.881253+0000 osd.5 (osd.5) 20 : cluster 0 3.16 scrub ok 2024-03-11T20:51:47.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:47 smithi023 bash[21372]: cluster 2024-03-11T20:51:46.597201+0000 osd.4 (osd.4) 19 : cluster 0 4.13 scrub starts 2024-03-11T20:51:47.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:47 smithi023 bash[21372]: cluster 2024-03-11T20:51:46.599556+0000 osd.4 (osd.4) 20 : cluster 0 4.13 scrub ok 2024-03-11T20:51:47.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:47 smithi023 bash[29146]: cluster 2024-03-11T20:51:45.756627+0000 osd.2 (osd.2) 19 : cluster 0 6.14 scrub starts 2024-03-11T20:51:47.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:47 smithi023 bash[29146]: cluster 2024-03-11T20:51:45.759129+0000 osd.2 (osd.2) 20 : cluster 0 6.14 scrub ok 2024-03-11T20:51:47.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:47 smithi023 bash[29146]: cluster 2024-03-11T20:51:45.815299+0000 mgr.y (mgr.24380) 26 : cluster 0 pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-11T20:51:47.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:47 smithi023 bash[29146]: cluster 2024-03-11T20:51:45.877464+0000 osd.5 (osd.5) 19 : cluster 0 3.16 scrub starts 2024-03-11T20:51:47.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:47 smithi023 bash[29146]: cluster 2024-03-11T20:51:45.881253+0000 osd.5 (osd.5) 20 : cluster 0 3.16 scrub ok 2024-03-11T20:51:47.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:47 smithi023 bash[29146]: cluster 2024-03-11T20:51:46.597201+0000 osd.4 (osd.4) 19 : cluster 0 4.13 scrub starts 2024-03-11T20:51:47.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:47 smithi023 bash[29146]: cluster 2024-03-11T20:51:46.599556+0000 osd.4 (osd.4) 20 : cluster 0 4.13 scrub ok 2024-03-11T20:51:48.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:48 smithi146 bash[23809]: cluster 2024-03-11T20:51:46.301991+0000 osd.3 (osd.3) 19 : cluster 0 3.1e scrub starts 2024-03-11T20:51:48.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:48 smithi146 bash[23809]: cluster 2024-03-11T20:51:46.306671+0000 osd.3 (osd.3) 20 : cluster 0 3.1e scrub ok 2024-03-11T20:51:48.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:48 smithi146 bash[23809]: cluster 2024-03-11T20:51:47.926545+0000 osd.0 (osd.0) 27 : cluster 0 4.16 scrub starts 2024-03-11T20:51:48.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:48 smithi146 bash[23809]: cluster 2024-03-11T20:51:47.928934+0000 osd.0 (osd.0) 28 : cluster 0 4.16 scrub ok 2024-03-11T20:51:48.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:48 smithi023 bash[21372]: cluster 2024-03-11T20:51:46.301991+0000 osd.3 (osd.3) 19 : cluster 0 3.1e scrub starts 2024-03-11T20:51:48.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:48 smithi023 bash[21372]: cluster 2024-03-11T20:51:46.306671+0000 osd.3 (osd.3) 20 : cluster 0 3.1e scrub ok 2024-03-11T20:51:48.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:48 smithi023 bash[21372]: cluster 2024-03-11T20:51:47.926545+0000 osd.0 (osd.0) 27 : cluster 0 4.16 scrub starts 2024-03-11T20:51:48.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:48 smithi023 bash[21372]: cluster 2024-03-11T20:51:47.928934+0000 osd.0 (osd.0) 28 : cluster 0 4.16 scrub ok 2024-03-11T20:51:48.534 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:48 smithi023 bash[29146]: cluster 2024-03-11T20:51:46.301991+0000 osd.3 (osd.3) 19 : cluster 0 3.1e scrub starts 2024-03-11T20:51:48.534 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:48 smithi023 bash[29146]: cluster 2024-03-11T20:51:46.306671+0000 osd.3 (osd.3) 20 : cluster 0 3.1e scrub ok 2024-03-11T20:51:48.534 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:48 smithi023 bash[29146]: cluster 2024-03-11T20:51:47.926545+0000 osd.0 (osd.0) 27 : cluster 0 4.16 scrub starts 2024-03-11T20:51:48.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:48 smithi023 bash[29146]: cluster 2024-03-11T20:51:47.928934+0000 osd.0 (osd.0) 28 : cluster 0 4.16 scrub ok 2024-03-11T20:51:48.836 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T20:51:49.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:49 smithi146 bash[23809]: cluster 2024-03-11T20:51:47.323657+0000 osd.3 (osd.3) 21 : cluster 0 6.b scrub starts 2024-03-11T20:51:49.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:49 smithi146 bash[23809]: cluster 2024-03-11T20:51:47.325652+0000 osd.3 (osd.3) 22 : cluster 0 6.b scrub ok 2024-03-11T20:51:49.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:49 smithi146 bash[23809]: cluster 2024-03-11T20:51:47.816030+0000 mgr.y (mgr.24380) 27 : cluster 0 pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-11T20:51:49.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:49 smithi146 bash[23809]: cluster 2024-03-11T20:51:48.925321+0000 osd.0 (osd.0) 29 : cluster 0 3.12 scrub starts 2024-03-11T20:51:49.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:49 smithi146 bash[23809]: cluster 2024-03-11T20:51:48.929317+0000 osd.0 (osd.0) 30 : cluster 0 3.12 scrub ok 2024-03-11T20:51:49.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:49 smithi146 bash[23809]: audit 2024-03-11T20:51:48.952670+0000 mon.a (mon.0) 832 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:51:49.356 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:49 smithi023 bash[21372]: cluster 2024-03-11T20:51:47.323657+0000 osd.3 (osd.3) 21 : cluster 0 6.b scrub starts 2024-03-11T20:51:49.357 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:49 smithi023 bash[21372]: cluster 2024-03-11T20:51:47.325652+0000 osd.3 (osd.3) 22 : cluster 0 6.b scrub ok 2024-03-11T20:51:49.357 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:49 smithi023 bash[21372]: cluster 2024-03-11T20:51:47.816030+0000 mgr.y (mgr.24380) 27 : cluster 0 pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-11T20:51:49.357 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:49 smithi023 bash[21372]: cluster 2024-03-11T20:51:48.925321+0000 osd.0 (osd.0) 29 : cluster 0 3.12 scrub starts 2024-03-11T20:51:49.357 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:49 smithi023 bash[21372]: cluster 2024-03-11T20:51:48.929317+0000 osd.0 (osd.0) 30 : cluster 0 3.12 scrub ok 2024-03-11T20:51:49.357 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:49 smithi023 bash[21372]: audit 2024-03-11T20:51:48.952670+0000 mon.a (mon.0) 832 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:51:49.357 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:49 smithi023 bash[29146]: cluster 2024-03-11T20:51:47.323657+0000 osd.3 (osd.3) 21 : cluster 0 6.b scrub starts 2024-03-11T20:51:49.357 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:49 smithi023 bash[29146]: cluster 2024-03-11T20:51:47.325652+0000 osd.3 (osd.3) 22 : cluster 0 6.b scrub ok 2024-03-11T20:51:49.357 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:49 smithi023 bash[29146]: cluster 2024-03-11T20:51:47.816030+0000 mgr.y (mgr.24380) 27 : cluster 0 pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-11T20:51:49.357 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:49 smithi023 bash[29146]: cluster 2024-03-11T20:51:48.925321+0000 osd.0 (osd.0) 29 : cluster 0 3.12 scrub starts 2024-03-11T20:51:49.357 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:49 smithi023 bash[29146]: cluster 2024-03-11T20:51:48.929317+0000 osd.0 (osd.0) 30 : cluster 0 3.12 scrub ok 2024-03-11T20:51:49.358 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:49 smithi023 bash[29146]: audit 2024-03-11T20:51:48.952670+0000 mon.a (mon.0) 832 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:51:50.384 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:50 smithi146 bash[23809]: cluster 2024-03-11T20:51:48.872462+0000 osd.2 (osd.2) 21 : cluster 0 4.1c scrub starts 2024-03-11T20:51:50.384 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:50 smithi146 bash[23809]: cluster 2024-03-11T20:51:48.874849+0000 osd.2 (osd.2) 22 : cluster 0 4.1c scrub ok 2024-03-11T20:51:50.384 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:50 smithi146 bash[23809]: cluster 2024-03-11T20:51:49.158764+0000 osd.7 (osd.7) 17 : cluster 0 6.8 scrub starts 2024-03-11T20:51:50.384 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:50 smithi146 bash[23809]: cluster 2024-03-11T20:51:49.161315+0000 osd.7 (osd.7) 18 : cluster 0 6.8 scrub ok 2024-03-11T20:51:50.385 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:50 smithi146 bash[23809]: cluster 2024-03-11T20:51:49.914532+0000 osd.0 (osd.0) 31 : cluster 0 4.1e scrub starts 2024-03-11T20:51:50.385 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:50 smithi146 bash[23809]: cluster 2024-03-11T20:51:49.916952+0000 osd.0 (osd.0) 32 : cluster 0 4.1e scrub ok 2024-03-11T20:51:50.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:50 smithi023 bash[21372]: cluster 2024-03-11T20:51:48.872462+0000 osd.2 (osd.2) 21 : cluster 0 4.1c scrub starts 2024-03-11T20:51:50.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:50 smithi023 bash[21372]: cluster 2024-03-11T20:51:48.874849+0000 osd.2 (osd.2) 22 : cluster 0 4.1c scrub ok 2024-03-11T20:51:50.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:50 smithi023 bash[21372]: cluster 2024-03-11T20:51:49.158764+0000 osd.7 (osd.7) 17 : cluster 0 6.8 scrub starts 2024-03-11T20:51:50.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:50 smithi023 bash[21372]: cluster 2024-03-11T20:51:49.161315+0000 osd.7 (osd.7) 18 : cluster 0 6.8 scrub ok 2024-03-11T20:51:50.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:50 smithi023 bash[21372]: cluster 2024-03-11T20:51:49.914532+0000 osd.0 (osd.0) 31 : cluster 0 4.1e scrub starts 2024-03-11T20:51:50.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:50 smithi023 bash[21372]: cluster 2024-03-11T20:51:49.916952+0000 osd.0 (osd.0) 32 : cluster 0 4.1e scrub ok 2024-03-11T20:51:50.534 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:50 smithi023 bash[29146]: cluster 2024-03-11T20:51:48.872462+0000 osd.2 (osd.2) 21 : cluster 0 4.1c scrub starts 2024-03-11T20:51:50.534 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:50 smithi023 bash[29146]: cluster 2024-03-11T20:51:48.874849+0000 osd.2 (osd.2) 22 : cluster 0 4.1c scrub ok 2024-03-11T20:51:50.534 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:50 smithi023 bash[29146]: cluster 2024-03-11T20:51:49.158764+0000 osd.7 (osd.7) 17 : cluster 0 6.8 scrub starts 2024-03-11T20:51:50.534 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:50 smithi023 bash[29146]: cluster 2024-03-11T20:51:49.161315+0000 osd.7 (osd.7) 18 : cluster 0 6.8 scrub ok 2024-03-11T20:51:50.534 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:50 smithi023 bash[29146]: cluster 2024-03-11T20:51:49.914532+0000 osd.0 (osd.0) 31 : cluster 0 4.1e scrub starts 2024-03-11T20:51:50.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:50 smithi023 bash[29146]: cluster 2024-03-11T20:51:49.916952+0000 osd.0 (osd.0) 32 : cluster 0 4.1e scrub ok 2024-03-11T20:51:51.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:51 smithi146 bash[23809]: cluster 2024-03-11T20:51:49.817565+0000 mgr.y (mgr.24380) 28 : cluster 0 pgmap v12: 132 pgs: 1 active+clean+scrubbing, 131 active+clean; 582 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-11T20:51:51.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:51 smithi146 bash[23809]: cluster 2024-03-11T20:51:49.819157+0000 osd.5 (osd.5) 21 : cluster 0 6.d scrub starts 2024-03-11T20:51:51.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:51 smithi146 bash[23809]: cluster 2024-03-11T20:51:49.821723+0000 osd.5 (osd.5) 22 : cluster 0 6.d scrub ok 2024-03-11T20:51:51.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:51 smithi146 bash[23809]: cluster 2024-03-11T20:51:50.125029+0000 osd.7 (osd.7) 19 : cluster 0 3.15 scrub starts 2024-03-11T20:51:51.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:51 smithi146 bash[23809]: cluster 2024-03-11T20:51:50.126807+0000 osd.7 (osd.7) 20 : cluster 0 3.15 scrub ok 2024-03-11T20:51:51.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:51 smithi146 bash[23809]: cluster 2024-03-11T20:51:50.566422+0000 osd.4 (osd.4) 21 : cluster 0 3.1a scrub starts 2024-03-11T20:51:51.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:51 smithi146 bash[23809]: cluster 2024-03-11T20:51:50.570643+0000 osd.4 (osd.4) 22 : cluster 0 3.1a scrub ok 2024-03-11T20:51:51.269 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:51 smithi023 bash[21372]: cluster 2024-03-11T20:51:49.817565+0000 mgr.y (mgr.24380) 28 : cluster 0 pgmap v12: 132 pgs: 1 active+clean+scrubbing, 131 active+clean; 582 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-11T20:51:51.269 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:51 smithi023 bash[21372]: cluster 2024-03-11T20:51:49.819157+0000 osd.5 (osd.5) 21 : cluster 0 6.d scrub starts 2024-03-11T20:51:51.269 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:51 smithi023 bash[21372]: cluster 2024-03-11T20:51:49.821723+0000 osd.5 (osd.5) 22 : cluster 0 6.d scrub ok 2024-03-11T20:51:51.269 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:51 smithi023 bash[21372]: cluster 2024-03-11T20:51:50.125029+0000 osd.7 (osd.7) 19 : cluster 0 3.15 scrub starts 2024-03-11T20:51:51.269 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:51 smithi023 bash[21372]: cluster 2024-03-11T20:51:50.126807+0000 osd.7 (osd.7) 20 : cluster 0 3.15 scrub ok 2024-03-11T20:51:51.269 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:51 smithi023 bash[21372]: cluster 2024-03-11T20:51:50.566422+0000 osd.4 (osd.4) 21 : cluster 0 3.1a scrub starts 2024-03-11T20:51:51.269 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:51 smithi023 bash[21372]: cluster 2024-03-11T20:51:50.570643+0000 osd.4 (osd.4) 22 : cluster 0 3.1a scrub ok 2024-03-11T20:51:51.270 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:51 smithi023 bash[29146]: cluster 2024-03-11T20:51:49.817565+0000 mgr.y (mgr.24380) 28 : cluster 0 pgmap v12: 132 pgs: 1 active+clean+scrubbing, 131 active+clean; 582 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-11T20:51:51.270 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:51 smithi023 bash[29146]: cluster 2024-03-11T20:51:49.819157+0000 osd.5 (osd.5) 21 : cluster 0 6.d scrub starts 2024-03-11T20:51:51.270 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:51 smithi023 bash[29146]: cluster 2024-03-11T20:51:49.821723+0000 osd.5 (osd.5) 22 : cluster 0 6.d scrub ok 2024-03-11T20:51:51.270 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:51 smithi023 bash[29146]: cluster 2024-03-11T20:51:50.125029+0000 osd.7 (osd.7) 19 : cluster 0 3.15 scrub starts 2024-03-11T20:51:51.270 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:51 smithi023 bash[29146]: cluster 2024-03-11T20:51:50.126807+0000 osd.7 (osd.7) 20 : cluster 0 3.15 scrub ok 2024-03-11T20:51:51.270 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:51 smithi023 bash[29146]: cluster 2024-03-11T20:51:50.566422+0000 osd.4 (osd.4) 21 : cluster 0 3.1a scrub starts 2024-03-11T20:51:51.270 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:51 smithi023 bash[29146]: cluster 2024-03-11T20:51:50.570643+0000 osd.4 (osd.4) 22 : cluster 0 3.1a scrub ok 2024-03-11T20:51:52.127 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:52 smithi023 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:51:52.127 INFO:journalctl@ceph.osd.3.smithi023.stdout:Mar 11 20:51:52 smithi023 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:51:52.128 INFO:journalctl@ceph.rgw.foo.a.smithi023.stdout:Mar 11 20:51:52 smithi023 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:51:52.128 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:51:52 smithi023 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:51:52.128 INFO:journalctl@ceph.osd.1.smithi023.stdout:Mar 11 20:51:52 smithi023 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:51:52.129 INFO:journalctl@ceph.osd.2.smithi023.stdout:Mar 11 20:51:52 smithi023 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:51:52.129 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 11 20:51:52 smithi023 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:51:52.129 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:52 smithi023 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:51:52.355 INFO:teuthology.orchestra.run.smithi146.stdout:[client.1] 2024-03-11T20:51:52.355 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQDobu9lk2TsFBAA7axXg56KwZxvGg5BsDzy4A== 2024-03-11T20:51:52.410 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:51:52 smithi023 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:51:52.410 INFO:journalctl@ceph.osd.3.smithi023.stdout:Mar 11 20:51:52 smithi023 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:51:52.410 INFO:journalctl@ceph.rgw.foo.a.smithi023.stdout:Mar 11 20:51:52 smithi023 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:51:52.411 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 11 20:51:52 smithi023 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:51:52.411 INFO:journalctl@ceph.osd.1.smithi023.stdout:Mar 11 20:51:52 smithi023 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:51:52.411 INFO:journalctl@ceph.osd.2.smithi023.stdout:Mar 11 20:51:52 smithi023 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:51:52.411 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:52 smithi023 bash[29146]: cluster 2024-03-11T20:51:50.837513+0000 osd.5 (osd.5) 23 : cluster 0 3.1c scrub starts 2024-03-11T20:51:52.412 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:52 smithi023 bash[29146]: cluster 2024-03-11T20:51:50.841891+0000 osd.5 (osd.5) 24 : cluster 0 3.1c scrub ok 2024-03-11T20:51:52.412 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:52 smithi023 bash[29146]: cluster 2024-03-11T20:51:52.045425+0000 osd.1 (osd.1) 21 : cluster 0 3.19 scrub starts 2024-03-11T20:51:52.412 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:52 smithi023 bash[29146]: cluster 2024-03-11T20:51:52.048183+0000 osd.1 (osd.1) 22 : cluster 0 3.19 scrub ok 2024-03-11T20:51:52.412 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:52 smithi023 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:51:52.412 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:52 smithi023 bash[21372]: cluster 2024-03-11T20:51:50.837513+0000 osd.5 (osd.5) 23 : cluster 0 3.1c scrub starts 2024-03-11T20:51:52.412 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:52 smithi023 bash[21372]: cluster 2024-03-11T20:51:50.841891+0000 osd.5 (osd.5) 24 : cluster 0 3.1c scrub ok 2024-03-11T20:51:52.412 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:52 smithi023 bash[21372]: cluster 2024-03-11T20:51:52.045425+0000 osd.1 (osd.1) 21 : cluster 0 3.19 scrub starts 2024-03-11T20:51:52.412 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:52 smithi023 bash[21372]: cluster 2024-03-11T20:51:52.048183+0000 osd.1 (osd.1) 22 : cluster 0 3.19 scrub ok 2024-03-11T20:51:52.412 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:52 smithi023 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:51:52.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:52 smithi146 bash[23809]: cluster 2024-03-11T20:51:50.837513+0000 osd.5 (osd.5) 23 : cluster 0 3.1c scrub starts 2024-03-11T20:51:52.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:52 smithi146 bash[23809]: cluster 2024-03-11T20:51:50.841891+0000 osd.5 (osd.5) 24 : cluster 0 3.1c scrub ok 2024-03-11T20:51:52.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:52 smithi146 bash[23809]: cluster 2024-03-11T20:51:52.045425+0000 osd.1 (osd.1) 21 : cluster 0 3.19 scrub starts 2024-03-11T20:51:52.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:52 smithi146 bash[23809]: cluster 2024-03-11T20:51:52.048183+0000 osd.1 (osd.1) 22 : cluster 0 3.19 scrub ok 2024-03-11T20:51:53.059 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-11T20:51:53.059 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-03-11T20:51:53.059 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-11T20:51:53.080 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-11T20:51:53.080 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-11T20:51:53.081 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph mgr dump --format=json 2024-03-11T20:51:53.372 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:53 smithi023 bash[21372]: cluster 2024-03-11T20:51:51.818193+0000 mgr.y (mgr.24380) 29 : cluster 0 pgmap v13: 132 pgs: 1 active+clean+scrubbing, 131 active+clean; 582 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-11T20:51:53.372 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:53 smithi023 bash[21372]: audit 2024-03-11T20:51:52.350686+0000 mon.a (mon.0) 833 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T20:51:53.372 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:53 smithi023 bash[21372]: audit 2024-03-11T20:51:52.353651+0000 mon.b (mon.1) 32 : audit 1 from='client.? 172.21.15.146:0/628922789' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T20:51:53.372 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:53 smithi023 bash[21372]: audit 2024-03-11T20:51:52.355584+0000 mon.a (mon.0) 834 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-11T20:51:53.372 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:53 smithi023 bash[21372]: audit 2024-03-11T20:51:52.416345+0000 mon.a (mon.0) 835 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:51:53.372 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:53 smithi023 bash[21372]: audit 2024-03-11T20:51:52.421690+0000 mon.a (mon.0) 836 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:51:53.372 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:53 smithi023 bash[21372]: audit 2024-03-11T20:51:52.453380+0000 mon.a (mon.0) 837 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:51:53.372 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:53 smithi023 bash[21372]: audit 2024-03-11T20:51:52.457193+0000 mon.a (mon.0) 838 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:51:53.372 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:53 smithi023 bash[21372]: cluster 2024-03-11T20:51:52.930157+0000 osd.1 (osd.1) 23 : cluster 0 purged_snaps scrub starts 2024-03-11T20:51:53.372 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:53 smithi023 bash[21372]: cluster 2024-03-11T20:51:52.930263+0000 osd.1 (osd.1) 24 : cluster 0 purged_snaps scrub ok 2024-03-11T20:51:53.373 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:53 smithi023 bash[29146]: cluster 2024-03-11T20:51:51.818193+0000 mgr.y (mgr.24380) 29 : cluster 0 pgmap v13: 132 pgs: 1 active+clean+scrubbing, 131 active+clean; 582 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-11T20:51:53.373 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:53 smithi023 bash[29146]: audit 2024-03-11T20:51:52.350686+0000 mon.a (mon.0) 833 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T20:51:53.373 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:53 smithi023 bash[29146]: audit 2024-03-11T20:51:52.353651+0000 mon.b (mon.1) 32 : audit 1 from='client.? 172.21.15.146:0/628922789' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T20:51:53.373 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:53 smithi023 bash[29146]: audit 2024-03-11T20:51:52.355584+0000 mon.a (mon.0) 834 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-11T20:51:53.373 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:53 smithi023 bash[29146]: audit 2024-03-11T20:51:52.416345+0000 mon.a (mon.0) 835 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:51:53.373 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:53 smithi023 bash[29146]: audit 2024-03-11T20:51:52.421690+0000 mon.a (mon.0) 836 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:51:53.373 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:53 smithi023 bash[29146]: audit 2024-03-11T20:51:52.453380+0000 mon.a (mon.0) 837 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:51:53.373 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:53 smithi023 bash[29146]: audit 2024-03-11T20:51:52.457193+0000 mon.a (mon.0) 838 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:51:53.373 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:53 smithi023 bash[29146]: cluster 2024-03-11T20:51:52.930157+0000 osd.1 (osd.1) 23 : cluster 0 purged_snaps scrub starts 2024-03-11T20:51:53.373 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:53 smithi023 bash[29146]: cluster 2024-03-11T20:51:52.930263+0000 osd.1 (osd.1) 24 : cluster 0 purged_snaps scrub ok 2024-03-11T20:51:53.482 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 11 20:51:53 smithi023 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:51:53.483 INFO:journalctl@ceph.osd.1.smithi023.stdout:Mar 11 20:51:53 smithi023 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:51:53.483 INFO:journalctl@ceph.osd.2.smithi023.stdout:Mar 11 20:51:53 smithi023 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:51:53.483 INFO:journalctl@ceph.osd.3.smithi023.stdout:Mar 11 20:51:53 smithi023 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:51:53.484 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:51:53 smithi023 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:51:53.484 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:53 smithi023 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:51:53.484 INFO:journalctl@ceph.rgw.foo.a.smithi023.stdout:Mar 11 20:51:53 smithi023 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:51:53.484 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:53 smithi023 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:51:53.485 INFO:journalctl@ceph.alertmanager.a.smithi023.stdout:Mar 11 20:51:53 smithi023 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:51:53.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:53 smithi146 bash[23809]: cluster 2024-03-11T20:51:51.818193+0000 mgr.y (mgr.24380) 29 : cluster 0 pgmap v13: 132 pgs: 1 active+clean+scrubbing, 131 active+clean; 582 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-11T20:51:53.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:53 smithi146 bash[23809]: audit 2024-03-11T20:51:52.350686+0000 mon.a (mon.0) 833 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T20:51:53.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:53 smithi146 bash[23809]: audit 2024-03-11T20:51:52.353651+0000 mon.b (mon.1) 32 : audit 1 from='client.? 172.21.15.146:0/628922789' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T20:51:53.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:53 smithi146 bash[23809]: audit 2024-03-11T20:51:52.355584+0000 mon.a (mon.0) 834 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-11T20:51:53.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:53 smithi146 bash[23809]: audit 2024-03-11T20:51:52.416345+0000 mon.a (mon.0) 835 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:51:53.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:53 smithi146 bash[23809]: audit 2024-03-11T20:51:52.421690+0000 mon.a (mon.0) 836 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:51:53.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:53 smithi146 bash[23809]: audit 2024-03-11T20:51:52.453380+0000 mon.a (mon.0) 837 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:51:53.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:53 smithi146 bash[23809]: audit 2024-03-11T20:51:52.457193+0000 mon.a (mon.0) 838 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:51:53.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:53 smithi146 bash[23809]: cluster 2024-03-11T20:51:52.930157+0000 osd.1 (osd.1) 23 : cluster 0 purged_snaps scrub starts 2024-03-11T20:51:53.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:53 smithi146 bash[23809]: cluster 2024-03-11T20:51:52.930263+0000 osd.1 (osd.1) 24 : cluster 0 purged_snaps scrub ok 2024-03-11T20:51:53.759 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:51:53 smithi023 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:51:53.759 INFO:journalctl@ceph.osd.1.smithi023.stdout:Mar 11 20:51:53 smithi023 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:51:53.759 INFO:journalctl@ceph.osd.3.smithi023.stdout:Mar 11 20:51:53 smithi023 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:51:53.760 INFO:journalctl@ceph.rgw.foo.a.smithi023.stdout:Mar 11 20:51:53 smithi023 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:51:53.760 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 11 20:51:53 smithi023 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:51:53.760 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 11 20:51:53 smithi023 systemd[1]: Started Ceph node-exporter.a for 67ba00f8-dfe7-11ee-95c8-87774f69a715. 2024-03-11T20:51:53.761 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:53 smithi023 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:51:53.761 INFO:journalctl@ceph.alertmanager.a.smithi023.stdout:Mar 11 20:51:53 smithi023 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:51:53.761 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:53 smithi023 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:51:53.761 INFO:journalctl@ceph.osd.2.smithi023.stdout:Mar 11 20:51:53 smithi023 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:51:54.034 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 11 20:51:53 smithi023 bash[58141]: Unable to find image 'quay.io/prometheus/node-exporter:v1.5.0' locally 2024-03-11T20:51:54.034 INFO:journalctl@ceph.alertmanager.a.smithi023.stdout:Mar 11 20:51:53 smithi023 bash[57975]: ts=2024-03-11T20:51:53.786Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-03-11T20:51:54.034 INFO:journalctl@ceph.alertmanager.a.smithi023.stdout:Mar 11 20:51:53 smithi023 bash[57975]: ts=2024-03-11T20:51:53.786Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-03-11T20:51:54.034 INFO:journalctl@ceph.alertmanager.a.smithi023.stdout:Mar 11 20:51:53 smithi023 bash[57975]: ts=2024-03-11T20:51:53.788Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.23 port=9094 2024-03-11T20:51:54.034 INFO:journalctl@ceph.alertmanager.a.smithi023.stdout:Mar 11 20:51:53 smithi023 bash[57975]: ts=2024-03-11T20:51:53.791Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-03-11T20:51:54.035 INFO:journalctl@ceph.alertmanager.a.smithi023.stdout:Mar 11 20:51:53 smithi023 bash[57975]: ts=2024-03-11T20:51:53.821Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-11T20:51:54.035 INFO:journalctl@ceph.alertmanager.a.smithi023.stdout:Mar 11 20:51:53 smithi023 bash[57975]: ts=2024-03-11T20:51:53.822Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-11T20:51:54.035 INFO:journalctl@ceph.alertmanager.a.smithi023.stdout:Mar 11 20:51:53 smithi023 bash[57975]: ts=2024-03-11T20:51:53.825Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-03-11T20:51:54.035 INFO:journalctl@ceph.alertmanager.a.smithi023.stdout:Mar 11 20:51:53 smithi023 bash[57975]: ts=2024-03-11T20:51:53.825Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-03-11T20:51:54.458 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:54 smithi146 bash[23809]: cluster 2024-03-11T20:51:52.284474+0000 osd.3 (osd.3) 23 : cluster 0 4.11 scrub starts 2024-03-11T20:51:54.458 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:54 smithi146 bash[23809]: cluster 2024-03-11T20:51:52.285875+0000 osd.3 (osd.3) 24 : cluster 0 4.11 scrub ok 2024-03-11T20:51:54.458 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:54 smithi146 bash[23809]: cephadm 2024-03-11T20:51:52.460479+0000 mgr.y (mgr.24380) 30 : cephadm 1 Deploying daemon node-exporter.a on smithi023 2024-03-11T20:51:54.459 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:54 smithi146 bash[23809]: cluster 2024-03-11T20:51:52.744530+0000 osd.6 (osd.6) 13 : cluster 0 5.2 scrub starts 2024-03-11T20:51:54.459 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:54 smithi146 bash[23809]: cluster 2024-03-11T20:51:52.747432+0000 osd.6 (osd.6) 14 : cluster 0 5.2 scrub ok 2024-03-11T20:51:54.459 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:54 smithi146 bash[23809]: cluster 2024-03-11T20:51:52.836604+0000 osd.2 (osd.2) 23 : cluster 0 2.e scrub starts 2024-03-11T20:51:54.459 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:54 smithi146 bash[23809]: cluster 2024-03-11T20:51:52.838917+0000 osd.2 (osd.2) 24 : cluster 0 2.e scrub ok 2024-03-11T20:51:54.459 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:54 smithi146 bash[23809]: cluster 2024-03-11T20:51:53.465368+0000 mon.a (mon.0) 839 : cluster 0 osdmap e103: 8 total, 8 up, 8 in 2024-03-11T20:51:54.459 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:54 smithi146 bash[23809]: cluster 2024-03-11T20:51:53.556957+0000 osd.4 (osd.4) 23 : cluster 0 5.1 scrub starts 2024-03-11T20:51:54.459 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:54 smithi146 bash[23809]: cluster 2024-03-11T20:51:53.560494+0000 osd.4 (osd.4) 24 : cluster 0 5.1 scrub ok 2024-03-11T20:51:54.459 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:54 smithi146 bash[23809]: audit 2024-03-11T20:51:53.765976+0000 mon.a (mon.0) 840 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:51:54.459 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:54 smithi146 bash[23809]: audit 2024-03-11T20:51:53.771077+0000 mon.a (mon.0) 841 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:51:54.459 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:54 smithi146 bash[23809]: audit 2024-03-11T20:51:53.780202+0000 mon.a (mon.0) 842 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:51:54.459 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:54 smithi146 bash[23809]: audit 2024-03-11T20:51:53.924728+0000 mon.a (mon.0) 843 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:51:54.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:54 smithi023 bash[21372]: cluster 2024-03-11T20:51:52.284474+0000 osd.3 (osd.3) 23 : cluster 0 4.11 scrub starts 2024-03-11T20:51:54.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:54 smithi023 bash[21372]: cluster 2024-03-11T20:51:52.285875+0000 osd.3 (osd.3) 24 : cluster 0 4.11 scrub ok 2024-03-11T20:51:54.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:54 smithi023 bash[21372]: cephadm 2024-03-11T20:51:52.460479+0000 mgr.y (mgr.24380) 30 : cephadm 1 Deploying daemon node-exporter.a on smithi023 2024-03-11T20:51:54.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:54 smithi023 bash[21372]: cluster 2024-03-11T20:51:52.744530+0000 osd.6 (osd.6) 13 : cluster 0 5.2 scrub starts 2024-03-11T20:51:54.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:54 smithi023 bash[21372]: cluster 2024-03-11T20:51:52.747432+0000 osd.6 (osd.6) 14 : cluster 0 5.2 scrub ok 2024-03-11T20:51:54.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:54 smithi023 bash[21372]: cluster 2024-03-11T20:51:52.836604+0000 osd.2 (osd.2) 23 : cluster 0 2.e scrub starts 2024-03-11T20:51:54.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:54 smithi023 bash[21372]: cluster 2024-03-11T20:51:52.838917+0000 osd.2 (osd.2) 24 : cluster 0 2.e scrub ok 2024-03-11T20:51:54.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:54 smithi023 bash[21372]: cluster 2024-03-11T20:51:53.465368+0000 mon.a (mon.0) 839 : cluster 0 osdmap e103: 8 total, 8 up, 8 in 2024-03-11T20:51:54.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:54 smithi023 bash[21372]: cluster 2024-03-11T20:51:53.556957+0000 osd.4 (osd.4) 23 : cluster 0 5.1 scrub starts 2024-03-11T20:51:54.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:54 smithi023 bash[21372]: cluster 2024-03-11T20:51:53.560494+0000 osd.4 (osd.4) 24 : cluster 0 5.1 scrub ok 2024-03-11T20:51:54.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:54 smithi023 bash[21372]: audit 2024-03-11T20:51:53.765976+0000 mon.a (mon.0) 840 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:51:54.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:54 smithi023 bash[21372]: audit 2024-03-11T20:51:53.771077+0000 mon.a (mon.0) 841 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:51:54.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:54 smithi023 bash[21372]: audit 2024-03-11T20:51:53.780202+0000 mon.a (mon.0) 842 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:51:54.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:54 smithi023 bash[21372]: audit 2024-03-11T20:51:53.924728+0000 mon.a (mon.0) 843 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:51:54.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:54 smithi023 bash[29146]: cluster 2024-03-11T20:51:52.284474+0000 osd.3 (osd.3) 23 : cluster 0 4.11 scrub starts 2024-03-11T20:51:54.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:54 smithi023 bash[29146]: cluster 2024-03-11T20:51:52.285875+0000 osd.3 (osd.3) 24 : cluster 0 4.11 scrub ok 2024-03-11T20:51:54.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:54 smithi023 bash[29146]: cephadm 2024-03-11T20:51:52.460479+0000 mgr.y (mgr.24380) 30 : cephadm 1 Deploying daemon node-exporter.a on smithi023 2024-03-11T20:51:54.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:54 smithi023 bash[29146]: cluster 2024-03-11T20:51:52.744530+0000 osd.6 (osd.6) 13 : cluster 0 5.2 scrub starts 2024-03-11T20:51:54.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:54 smithi023 bash[29146]: cluster 2024-03-11T20:51:52.747432+0000 osd.6 (osd.6) 14 : cluster 0 5.2 scrub ok 2024-03-11T20:51:54.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:54 smithi023 bash[29146]: cluster 2024-03-11T20:51:52.836604+0000 osd.2 (osd.2) 23 : cluster 0 2.e scrub starts 2024-03-11T20:51:54.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:54 smithi023 bash[29146]: cluster 2024-03-11T20:51:52.838917+0000 osd.2 (osd.2) 24 : cluster 0 2.e scrub ok 2024-03-11T20:51:54.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:54 smithi023 bash[29146]: cluster 2024-03-11T20:51:53.465368+0000 mon.a (mon.0) 839 : cluster 0 osdmap e103: 8 total, 8 up, 8 in 2024-03-11T20:51:54.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:54 smithi023 bash[29146]: cluster 2024-03-11T20:51:53.556957+0000 osd.4 (osd.4) 23 : cluster 0 5.1 scrub starts 2024-03-11T20:51:54.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:54 smithi023 bash[29146]: cluster 2024-03-11T20:51:53.560494+0000 osd.4 (osd.4) 24 : cluster 0 5.1 scrub ok 2024-03-11T20:51:54.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:54 smithi023 bash[29146]: audit 2024-03-11T20:51:53.765976+0000 mon.a (mon.0) 840 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:51:54.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:54 smithi023 bash[29146]: audit 2024-03-11T20:51:53.771077+0000 mon.a (mon.0) 841 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:51:54.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:54 smithi023 bash[29146]: audit 2024-03-11T20:51:53.780202+0000 mon.a (mon.0) 842 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:51:54.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:54 smithi023 bash[29146]: audit 2024-03-11T20:51:53.924728+0000 mon.a (mon.0) 843 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:51:54.709 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 11 20:51:54 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:51:54.709 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 20:51:54 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:51:54.710 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Mar 11 20:51:54 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:51:54.710 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:54 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:51:54.710 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 11 20:51:54 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:51:54.711 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 11 20:51:54 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:51:54.711 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Mar 11 20:51:54 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:51:54.711 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 11 20:51:54 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:51:55.010 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Mar 11 20:51:54 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:51:55.011 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 11 20:51:54 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:51:55.011 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 11 20:51:54 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:51:55.011 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 11 20:51:54 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:51:55.011 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 20:51:54 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:51:55.011 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 20:51:54 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T20:51:55.012 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Mar 11 20:51:54 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:51:55.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:54 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:51:55.012 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 11 20:51:54 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:51:55.012 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Mar 11 20:51:54 smithi146 systemd[1]: Started Ceph node-exporter.b for 67ba00f8-dfe7-11ee-95c8-87774f69a715. 2024-03-11T20:51:55.013 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Mar 11 20:51:54 smithi146 bash[51341]: Unable to find image 'quay.io/prometheus/node-exporter:v1.5.0' locally 2024-03-11T20:51:55.027 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 11 20:51:54 smithi023 bash[58141]: v1.5.0: Pulling from prometheus/node-exporter 2024-03-11T20:51:55.284 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 11 20:51:55 smithi023 bash[58141]: 22b70bddd3ac: Pulling fs layer 2024-03-11T20:51:55.284 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 11 20:51:55 smithi023 bash[58141]: 5c12815fee55: Pulling fs layer 2024-03-11T20:51:55.284 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 11 20:51:55 smithi023 bash[58141]: c0e87333d380: Pulling fs layer 2024-03-11T20:51:55.284 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:51:55 smithi023 bash[21627]: [11/Mar/2024:20:51:55] ENGINE Bus STOPPING 2024-03-11T20:51:55.284 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:51:55 smithi023 bash[21627]: [11/Mar/2024:20:51:55] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-11T20:51:55.285 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:51:55 smithi023 bash[21627]: [11/Mar/2024:20:51:55] ENGINE Bus STOPPED 2024-03-11T20:51:55.285 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:51:55 smithi023 bash[21627]: [11/Mar/2024:20:51:55] ENGINE Bus STARTING 2024-03-11T20:51:55.285 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:51:55 smithi023 bash[21627]: [11/Mar/2024:20:51:55] ENGINE Serving on http://:::9283 2024-03-11T20:51:55.285 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:51:55 smithi023 bash[21627]: [11/Mar/2024:20:51:55] ENGINE Bus STARTED 2024-03-11T20:51:55.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:55 smithi023 bash[21372]: cluster 2024-03-11T20:51:53.288809+0000 osd.3 (osd.3) 25 : cluster 0 4.19 scrub starts 2024-03-11T20:51:55.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:55 smithi023 bash[21372]: cluster 2024-03-11T20:51:53.289734+0000 osd.3 (osd.3) 26 : cluster 0 4.19 scrub ok 2024-03-11T20:51:55.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:55 smithi023 bash[21372]: cephadm 2024-03-11T20:51:53.781150+0000 mgr.y (mgr.24380) 31 : cephadm 1 Deploying daemon node-exporter.b on smithi146 2024-03-11T20:51:55.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:55 smithi023 bash[21372]: cluster 2024-03-11T20:51:53.818558+0000 mgr.y (mgr.24380) 32 : cluster 0 pgmap v15: 132 pgs: 1 active+clean+scrubbing, 131 active+clean; 582 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:51:55.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:55 smithi023 bash[21372]: cluster 2024-03-11T20:51:53.874039+0000 osd.5 (osd.5) 25 : cluster 0 3.1d scrub starts 2024-03-11T20:51:55.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:55 smithi023 bash[21372]: cluster 2024-03-11T20:51:53.881611+0000 osd.5 (osd.5) 26 : cluster 0 3.1d scrub ok 2024-03-11T20:51:55.286 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:55 smithi023 bash[21372]: audit 2024-03-11T20:51:54.889903+0000 mon.a (mon.0) 844 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:51:55.286 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:55 smithi023 bash[21372]: audit 2024-03-11T20:51:54.903885+0000 mon.a (mon.0) 845 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:51:55.286 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:55 smithi023 bash[21372]: audit 2024-03-11T20:51:54.956730+0000 mon.a (mon.0) 846 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:51:55.286 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:55 smithi023 bash[21372]: audit 2024-03-11T20:51:54.965344+0000 mon.a (mon.0) 847 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:51:55.286 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:55 smithi023 bash[21372]: audit 2024-03-11T20:51:54.983092+0000 mon.a (mon.0) 848 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:51:55.286 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:55 smithi023 bash[21372]: cluster 2024-03-11T20:51:55.035398+0000 osd.1 (osd.1) 25 : cluster 0 6.1 scrub starts 2024-03-11T20:51:55.286 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:55 smithi023 bash[21372]: cluster 2024-03-11T20:51:55.037461+0000 osd.1 (osd.1) 26 : cluster 0 6.1 scrub ok 2024-03-11T20:51:55.286 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:55 smithi023 bash[21372]: audit 2024-03-11T20:51:55.058968+0000 mon.a (mon.0) 849 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:51:55.286 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:55 smithi023 bash[21372]: audit 2024-03-11T20:51:55.068942+0000 mon.a (mon.0) 850 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:51:55.286 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:55 smithi023 bash[21372]: audit 2024-03-11T20:51:55.077453+0000 mon.a (mon.0) 851 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-11T20:51:55.286 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:55 smithi023 bash[21372]: audit 2024-03-11T20:51:55.088827+0000 mon.a (mon.0) 852 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:51:55.287 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:55 smithi023 bash[29146]: cluster 2024-03-11T20:51:53.288809+0000 osd.3 (osd.3) 25 : cluster 0 4.19 scrub starts 2024-03-11T20:51:55.287 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:55 smithi023 bash[29146]: cluster 2024-03-11T20:51:53.289734+0000 osd.3 (osd.3) 26 : cluster 0 4.19 scrub ok 2024-03-11T20:51:55.287 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:55 smithi023 bash[29146]: cephadm 2024-03-11T20:51:53.781150+0000 mgr.y (mgr.24380) 31 : cephadm 1 Deploying daemon node-exporter.b on smithi146 2024-03-11T20:51:55.287 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:55 smithi023 bash[29146]: cluster 2024-03-11T20:51:53.818558+0000 mgr.y (mgr.24380) 32 : cluster 0 pgmap v15: 132 pgs: 1 active+clean+scrubbing, 131 active+clean; 582 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:51:55.287 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:55 smithi023 bash[29146]: cluster 2024-03-11T20:51:53.874039+0000 osd.5 (osd.5) 25 : cluster 0 3.1d scrub starts 2024-03-11T20:51:55.287 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:55 smithi023 bash[29146]: cluster 2024-03-11T20:51:53.881611+0000 osd.5 (osd.5) 26 : cluster 0 3.1d scrub ok 2024-03-11T20:51:55.287 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:55 smithi023 bash[29146]: audit 2024-03-11T20:51:54.889903+0000 mon.a (mon.0) 844 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:51:55.287 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:55 smithi023 bash[29146]: audit 2024-03-11T20:51:54.903885+0000 mon.a (mon.0) 845 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:51:55.287 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:55 smithi023 bash[29146]: audit 2024-03-11T20:51:54.956730+0000 mon.a (mon.0) 846 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:51:55.287 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:55 smithi023 bash[29146]: audit 2024-03-11T20:51:54.965344+0000 mon.a (mon.0) 847 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:51:55.288 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:55 smithi023 bash[29146]: audit 2024-03-11T20:51:54.983092+0000 mon.a (mon.0) 848 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:51:55.288 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:55 smithi023 bash[29146]: cluster 2024-03-11T20:51:55.035398+0000 osd.1 (osd.1) 25 : cluster 0 6.1 scrub starts 2024-03-11T20:51:55.288 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:55 smithi023 bash[29146]: cluster 2024-03-11T20:51:55.037461+0000 osd.1 (osd.1) 26 : cluster 0 6.1 scrub ok 2024-03-11T20:51:55.288 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:55 smithi023 bash[29146]: audit 2024-03-11T20:51:55.058968+0000 mon.a (mon.0) 849 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:51:55.288 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:55 smithi023 bash[29146]: audit 2024-03-11T20:51:55.068942+0000 mon.a (mon.0) 850 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:51:55.288 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:55 smithi023 bash[29146]: audit 2024-03-11T20:51:55.077453+0000 mon.a (mon.0) 851 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-11T20:51:55.288 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:55 smithi023 bash[29146]: audit 2024-03-11T20:51:55.088827+0000 mon.a (mon.0) 852 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:51:55.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:55 smithi146 bash[23809]: cluster 2024-03-11T20:51:53.288809+0000 osd.3 (osd.3) 25 : cluster 0 4.19 scrub starts 2024-03-11T20:51:55.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:55 smithi146 bash[23809]: cluster 2024-03-11T20:51:53.289734+0000 osd.3 (osd.3) 26 : cluster 0 4.19 scrub ok 2024-03-11T20:51:55.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:55 smithi146 bash[23809]: cephadm 2024-03-11T20:51:53.781150+0000 mgr.y (mgr.24380) 31 : cephadm 1 Deploying daemon node-exporter.b on smithi146 2024-03-11T20:51:55.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:55 smithi146 bash[23809]: cluster 2024-03-11T20:51:53.818558+0000 mgr.y (mgr.24380) 32 : cluster 0 pgmap v15: 132 pgs: 1 active+clean+scrubbing, 131 active+clean; 582 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:51:55.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:55 smithi146 bash[23809]: cluster 2024-03-11T20:51:53.874039+0000 osd.5 (osd.5) 25 : cluster 0 3.1d scrub starts 2024-03-11T20:51:55.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:55 smithi146 bash[23809]: cluster 2024-03-11T20:51:53.881611+0000 osd.5 (osd.5) 26 : cluster 0 3.1d scrub ok 2024-03-11T20:51:55.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:55 smithi146 bash[23809]: audit 2024-03-11T20:51:54.889903+0000 mon.a (mon.0) 844 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:51:55.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:55 smithi146 bash[23809]: audit 2024-03-11T20:51:54.903885+0000 mon.a (mon.0) 845 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:51:55.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:55 smithi146 bash[23809]: audit 2024-03-11T20:51:54.956730+0000 mon.a (mon.0) 846 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:51:55.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:55 smithi146 bash[23809]: audit 2024-03-11T20:51:54.965344+0000 mon.a (mon.0) 847 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:51:55.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:55 smithi146 bash[23809]: audit 2024-03-11T20:51:54.983092+0000 mon.a (mon.0) 848 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:51:55.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:55 smithi146 bash[23809]: cluster 2024-03-11T20:51:55.035398+0000 osd.1 (osd.1) 25 : cluster 0 6.1 scrub starts 2024-03-11T20:51:55.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:55 smithi146 bash[23809]: cluster 2024-03-11T20:51:55.037461+0000 osd.1 (osd.1) 26 : cluster 0 6.1 scrub ok 2024-03-11T20:51:55.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:55 smithi146 bash[23809]: audit 2024-03-11T20:51:55.058968+0000 mon.a (mon.0) 849 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:51:55.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:55 smithi146 bash[23809]: audit 2024-03-11T20:51:55.068942+0000 mon.a (mon.0) 850 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:51:55.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:55 smithi146 bash[23809]: audit 2024-03-11T20:51:55.077453+0000 mon.a (mon.0) 851 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-11T20:51:55.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:55 smithi146 bash[23809]: audit 2024-03-11T20:51:55.088827+0000 mon.a (mon.0) 852 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:51:55.752 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 11 20:51:55 smithi023 bash[58141]: 5c12815fee55: Verifying Checksum 2024-03-11T20:51:55.752 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 11 20:51:55 smithi023 bash[58141]: 5c12815fee55: Download complete 2024-03-11T20:51:55.752 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 11 20:51:55 smithi023 bash[58141]: 22b70bddd3ac: Verifying Checksum 2024-03-11T20:51:55.752 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 11 20:51:55 smithi023 bash[58141]: 22b70bddd3ac: Download complete 2024-03-11T20:51:55.752 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 11 20:51:55 smithi023 bash[58141]: 22b70bddd3ac: Pull complete 2024-03-11T20:51:55.866 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:51:55 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:51:55.866 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Mar 11 20:51:55 smithi146 bash[51341]: v1.5.0: Pulling from prometheus/node-exporter 2024-03-11T20:51:56.034 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 11 20:51:55 smithi023 bash[58141]: c0e87333d380: Verifying Checksum 2024-03-11T20:51:56.034 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 11 20:51:55 smithi023 bash[58141]: c0e87333d380: Download complete 2024-03-11T20:51:56.034 INFO:journalctl@ceph.alertmanager.a.smithi023.stdout:Mar 11 20:51:55 smithi023 bash[57975]: ts=2024-03-11T20:51:55.791Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000237905s 2024-03-11T20:51:56.362 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 11 20:51:56 smithi023 bash[58141]: 5c12815fee55: Pull complete 2024-03-11T20:51:56.362 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 11 20:51:56 smithi023 bash[58141]: c0e87333d380: Pull complete 2024-03-11T20:51:56.362 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:56 smithi023 bash[21372]: audit 2024-03-11T20:51:54.829873+0000 mgr.y (mgr.24380) 33 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:51:56.362 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:56 smithi023 bash[21372]: cephadm 2024-03-11T20:51:55.014989+0000 mgr.y (mgr.24380) 34 : cephadm 1 Regenerating cephadm self-signed grafana TLS certificates 2024-03-11T20:51:56.362 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:56 smithi023 bash[21372]: audit 2024-03-11T20:51:55.078702+0000 mgr.y (mgr.24380) 35 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-11T20:51:56.362 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:56 smithi023 bash[21372]: cephadm 2024-03-11T20:51:55.100957+0000 mgr.y (mgr.24380) 36 : cephadm 1 Deploying daemon grafana.a on smithi146 2024-03-11T20:51:56.362 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:56 smithi023 bash[21372]: cluster 2024-03-11T20:51:55.161870+0000 osd.7 (osd.7) 21 : cluster 0 6.12 scrub starts 2024-03-11T20:51:56.362 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:56 smithi023 bash[21372]: cluster 2024-03-11T20:51:55.164126+0000 osd.7 (osd.7) 22 : cluster 0 6.12 scrub ok 2024-03-11T20:51:56.363 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:56 smithi023 bash[21372]: cluster 2024-03-11T20:51:55.544322+0000 osd.4 (osd.4) 25 : cluster 0 6.1a deep-scrub starts 2024-03-11T20:51:56.363 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:56 smithi023 bash[21372]: cluster 2024-03-11T20:51:55.545753+0000 osd.4 (osd.4) 26 : cluster 0 6.1a deep-scrub ok 2024-03-11T20:51:56.363 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:56 smithi023 bash[29146]: audit 2024-03-11T20:51:54.829873+0000 mgr.y (mgr.24380) 33 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:51:56.363 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:56 smithi023 bash[29146]: cephadm 2024-03-11T20:51:55.014989+0000 mgr.y (mgr.24380) 34 : cephadm 1 Regenerating cephadm self-signed grafana TLS certificates 2024-03-11T20:51:56.363 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:56 smithi023 bash[29146]: audit 2024-03-11T20:51:55.078702+0000 mgr.y (mgr.24380) 35 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-11T20:51:56.363 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:56 smithi023 bash[29146]: cephadm 2024-03-11T20:51:55.100957+0000 mgr.y (mgr.24380) 36 : cephadm 1 Deploying daemon grafana.a on smithi146 2024-03-11T20:51:56.363 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:56 smithi023 bash[29146]: cluster 2024-03-11T20:51:55.161870+0000 osd.7 (osd.7) 21 : cluster 0 6.12 scrub starts 2024-03-11T20:51:56.363 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:56 smithi023 bash[29146]: cluster 2024-03-11T20:51:55.164126+0000 osd.7 (osd.7) 22 : cluster 0 6.12 scrub ok 2024-03-11T20:51:56.363 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:56 smithi023 bash[29146]: cluster 2024-03-11T20:51:55.544322+0000 osd.4 (osd.4) 25 : cluster 0 6.1a deep-scrub starts 2024-03-11T20:51:56.363 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:56 smithi023 bash[29146]: cluster 2024-03-11T20:51:55.545753+0000 osd.4 (osd.4) 26 : cluster 0 6.1a deep-scrub ok 2024-03-11T20:51:56.510 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Mar 11 20:51:56 smithi146 bash[51341]: 22b70bddd3ac: Pulling fs layer 2024-03-11T20:51:56.510 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Mar 11 20:51:56 smithi146 bash[51341]: 5c12815fee55: Pulling fs layer 2024-03-11T20:51:56.510 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Mar 11 20:51:56 smithi146 bash[51341]: c0e87333d380: Pulling fs layer 2024-03-11T20:51:56.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:56 smithi146 bash[23809]: audit 2024-03-11T20:51:54.829873+0000 mgr.y (mgr.24380) 33 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:51:56.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:56 smithi146 bash[23809]: cephadm 2024-03-11T20:51:55.014989+0000 mgr.y (mgr.24380) 34 : cephadm 1 Regenerating cephadm self-signed grafana TLS certificates 2024-03-11T20:51:56.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:56 smithi146 bash[23809]: audit 2024-03-11T20:51:55.078702+0000 mgr.y (mgr.24380) 35 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-11T20:51:56.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:56 smithi146 bash[23809]: cephadm 2024-03-11T20:51:55.100957+0000 mgr.y (mgr.24380) 36 : cephadm 1 Deploying daemon grafana.a on smithi146 2024-03-11T20:51:56.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:56 smithi146 bash[23809]: cluster 2024-03-11T20:51:55.161870+0000 osd.7 (osd.7) 21 : cluster 0 6.12 scrub starts 2024-03-11T20:51:56.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:56 smithi146 bash[23809]: cluster 2024-03-11T20:51:55.164126+0000 osd.7 (osd.7) 22 : cluster 0 6.12 scrub ok 2024-03-11T20:51:56.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:56 smithi146 bash[23809]: cluster 2024-03-11T20:51:55.544322+0000 osd.4 (osd.4) 25 : cluster 0 6.1a deep-scrub starts 2024-03-11T20:51:56.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:56 smithi146 bash[23809]: cluster 2024-03-11T20:51:55.545753+0000 osd.4 (osd.4) 26 : cluster 0 6.1a deep-scrub ok 2024-03-11T20:51:56.807 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 11 20:51:56 smithi023 bash[58141]: Digest: sha256:39c642b2b337e38c18e80266fb14383754178202f40103646337722a594d984c 2024-03-11T20:51:56.807 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 11 20:51:56 smithi023 bash[58141]: Status: Downloaded newer image for quay.io/prometheus/node-exporter:v1.5.0 2024-03-11T20:51:56.838 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Mar 11 20:51:56 smithi146 bash[51341]: 5c12815fee55: Verifying Checksum 2024-03-11T20:51:56.838 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Mar 11 20:51:56 smithi146 bash[51341]: 5c12815fee55: Download complete 2024-03-11T20:51:56.838 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Mar 11 20:51:56 smithi146 bash[51341]: 22b70bddd3ac: Verifying Checksum 2024-03-11T20:51:56.838 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Mar 11 20:51:56 smithi146 bash[51341]: 22b70bddd3ac: Download complete 2024-03-11T20:51:56.838 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Mar 11 20:51:56 smithi146 bash[51341]: 22b70bddd3ac: Pull complete 2024-03-11T20:51:56.838 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Mar 11 20:51:56 smithi146 bash[51341]: c0e87333d380: Verifying Checksum 2024-03-11T20:51:56.838 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Mar 11 20:51:56 smithi146 bash[51341]: c0e87333d380: Download complete 2024-03-11T20:51:57.510 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Mar 11 20:51:57 smithi146 bash[51341]: 5c12815fee55: Pull complete 2024-03-11T20:51:57.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:57 smithi146 bash[23809]: cluster 2024-03-11T20:51:55.819480+0000 mgr.y (mgr.24380) 37 : cluster 0 pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-03-11T20:51:57.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:57 smithi146 bash[23809]: cluster 2024-03-11T20:51:55.902824+0000 osd.5 (osd.5) 27 : cluster 0 3.c scrub starts 2024-03-11T20:51:57.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:57 smithi146 bash[23809]: cluster 2024-03-11T20:51:55.906638+0000 osd.5 (osd.5) 28 : cluster 0 3.c scrub ok 2024-03-11T20:51:57.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:57 smithi146 bash[23809]: cluster 2024-03-11T20:51:56.471548+0000 osd.7 (osd.7) 23 : cluster 0 purged_snaps scrub starts 2024-03-11T20:51:57.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:57 smithi146 bash[23809]: cluster 2024-03-11T20:51:56.471662+0000 osd.7 (osd.7) 24 : cluster 0 purged_snaps scrub ok 2024-03-11T20:51:57.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:57 smithi023 bash[21372]: cluster 2024-03-11T20:51:55.819480+0000 mgr.y (mgr.24380) 37 : cluster 0 pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-03-11T20:51:57.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:57 smithi023 bash[21372]: cluster 2024-03-11T20:51:55.902824+0000 osd.5 (osd.5) 27 : cluster 0 3.c scrub starts 2024-03-11T20:51:57.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:57 smithi023 bash[21372]: cluster 2024-03-11T20:51:55.906638+0000 osd.5 (osd.5) 28 : cluster 0 3.c scrub ok 2024-03-11T20:51:57.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:57 smithi023 bash[21372]: cluster 2024-03-11T20:51:56.471548+0000 osd.7 (osd.7) 23 : cluster 0 purged_snaps scrub starts 2024-03-11T20:51:57.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:57 smithi023 bash[21372]: cluster 2024-03-11T20:51:56.471662+0000 osd.7 (osd.7) 24 : cluster 0 purged_snaps scrub ok 2024-03-11T20:51:57.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:57 smithi023 bash[29146]: cluster 2024-03-11T20:51:55.819480+0000 mgr.y (mgr.24380) 37 : cluster 0 pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-03-11T20:51:57.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:57 smithi023 bash[29146]: cluster 2024-03-11T20:51:55.902824+0000 osd.5 (osd.5) 27 : cluster 0 3.c scrub starts 2024-03-11T20:51:57.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:57 smithi023 bash[29146]: cluster 2024-03-11T20:51:55.906638+0000 osd.5 (osd.5) 28 : cluster 0 3.c scrub ok 2024-03-11T20:51:57.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:57 smithi023 bash[29146]: cluster 2024-03-11T20:51:56.471548+0000 osd.7 (osd.7) 23 : cluster 0 purged_snaps scrub starts 2024-03-11T20:51:57.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:57 smithi023 bash[29146]: cluster 2024-03-11T20:51:56.471662+0000 osd.7 (osd.7) 24 : cluster 0 purged_snaps scrub ok 2024-03-11T20:51:57.822 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Mar 11 20:51:57 smithi146 bash[51341]: c0e87333d380: Pull complete 2024-03-11T20:51:57.822 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Mar 11 20:51:57 smithi146 bash[51341]: Digest: sha256:39c642b2b337e38c18e80266fb14383754178202f40103646337722a594d984c 2024-03-11T20:51:57.875 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:51:58.181 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 11 20:51:57 smithi023 bash[58141]: ts=2024-03-11T20:51:57.978Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-03-11T20:51:58.181 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 11 20:51:57 smithi023 bash[58141]: ts=2024-03-11T20:51:57.978Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-03-11T20:51:58.181 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 11 20:51:57 smithi023 bash[58141]: ts=2024-03-11T20:51:57.978Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-03-11T20:51:58.182 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 11 20:51:57 smithi023 bash[58141]: ts=2024-03-11T20:51:57.978Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-03-11T20:51:58.182 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 11 20:51:57 smithi023 bash[58141]: ts=2024-03-11T20:51:57.979Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-03-11T20:51:58.182 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 11 20:51:57 smithi023 bash[58141]: ts=2024-03-11T20:51:57.979Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-11T20:51:58.182 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 11 20:51:57 smithi023 bash[58141]: ts=2024-03-11T20:51:57.980Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-03-11T20:51:58.182 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 11 20:51:57 smithi023 bash[58141]: ts=2024-03-11T20:51:57.980Z caller=node_exporter.go:117 level=info collector=arp 2024-03-11T20:51:58.182 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 11 20:51:57 smithi023 bash[58141]: ts=2024-03-11T20:51:57.980Z caller=node_exporter.go:117 level=info collector=bcache 2024-03-11T20:51:58.182 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 11 20:51:57 smithi023 bash[58141]: ts=2024-03-11T20:51:57.980Z caller=node_exporter.go:117 level=info collector=bonding 2024-03-11T20:51:58.182 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 11 20:51:57 smithi023 bash[58141]: ts=2024-03-11T20:51:57.980Z caller=node_exporter.go:117 level=info collector=btrfs 2024-03-11T20:51:58.182 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 11 20:51:57 smithi023 bash[58141]: ts=2024-03-11T20:51:57.980Z caller=node_exporter.go:117 level=info collector=conntrack 2024-03-11T20:51:58.182 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 11 20:51:57 smithi023 bash[58141]: ts=2024-03-11T20:51:57.980Z caller=node_exporter.go:117 level=info collector=cpu 2024-03-11T20:51:58.183 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 11 20:51:57 smithi023 bash[58141]: ts=2024-03-11T20:51:57.980Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-03-11T20:51:58.183 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 11 20:51:57 smithi023 bash[58141]: ts=2024-03-11T20:51:57.980Z caller=node_exporter.go:117 level=info collector=diskstats 2024-03-11T20:51:58.183 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 11 20:51:57 smithi023 bash[58141]: ts=2024-03-11T20:51:57.980Z caller=node_exporter.go:117 level=info collector=dmi 2024-03-11T20:51:58.183 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 11 20:51:57 smithi023 bash[58141]: ts=2024-03-11T20:51:57.980Z caller=node_exporter.go:117 level=info collector=edac 2024-03-11T20:51:58.183 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 11 20:51:57 smithi023 bash[58141]: ts=2024-03-11T20:51:57.980Z caller=node_exporter.go:117 level=info collector=entropy 2024-03-11T20:51:58.183 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 11 20:51:57 smithi023 bash[58141]: ts=2024-03-11T20:51:57.980Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-03-11T20:51:58.183 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 11 20:51:57 smithi023 bash[58141]: ts=2024-03-11T20:51:57.980Z caller=node_exporter.go:117 level=info collector=filefd 2024-03-11T20:51:58.183 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 11 20:51:57 smithi023 bash[58141]: ts=2024-03-11T20:51:57.980Z caller=node_exporter.go:117 level=info collector=filesystem 2024-03-11T20:51:58.183 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 11 20:51:57 smithi023 bash[58141]: ts=2024-03-11T20:51:57.980Z caller=node_exporter.go:117 level=info collector=hwmon 2024-03-11T20:51:58.183 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 11 20:51:57 smithi023 bash[58141]: ts=2024-03-11T20:51:57.980Z caller=node_exporter.go:117 level=info collector=infiniband 2024-03-11T20:51:58.183 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 11 20:51:57 smithi023 bash[58141]: ts=2024-03-11T20:51:57.980Z caller=node_exporter.go:117 level=info collector=ipvs 2024-03-11T20:51:58.183 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 11 20:51:57 smithi023 bash[58141]: ts=2024-03-11T20:51:57.980Z caller=node_exporter.go:117 level=info collector=loadavg 2024-03-11T20:51:58.183 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 11 20:51:57 smithi023 bash[58141]: ts=2024-03-11T20:51:57.980Z caller=node_exporter.go:117 level=info collector=mdadm 2024-03-11T20:51:58.184 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 11 20:51:57 smithi023 bash[58141]: ts=2024-03-11T20:51:57.980Z caller=node_exporter.go:117 level=info collector=meminfo 2024-03-11T20:51:58.184 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 11 20:51:57 smithi023 bash[58141]: ts=2024-03-11T20:51:57.980Z caller=node_exporter.go:117 level=info collector=netclass 2024-03-11T20:51:58.184 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 11 20:51:57 smithi023 bash[58141]: ts=2024-03-11T20:51:57.980Z caller=node_exporter.go:117 level=info collector=netdev 2024-03-11T20:51:58.184 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 11 20:51:57 smithi023 bash[58141]: ts=2024-03-11T20:51:57.980Z caller=node_exporter.go:117 level=info collector=netstat 2024-03-11T20:51:58.184 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 11 20:51:57 smithi023 bash[58141]: ts=2024-03-11T20:51:57.980Z caller=node_exporter.go:117 level=info collector=nfs 2024-03-11T20:51:58.184 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 11 20:51:57 smithi023 bash[58141]: ts=2024-03-11T20:51:57.980Z caller=node_exporter.go:117 level=info collector=nfsd 2024-03-11T20:51:58.184 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 11 20:51:57 smithi023 bash[58141]: ts=2024-03-11T20:51:57.980Z caller=node_exporter.go:117 level=info collector=nvme 2024-03-11T20:51:58.185 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 11 20:51:57 smithi023 bash[58141]: ts=2024-03-11T20:51:57.980Z caller=node_exporter.go:117 level=info collector=os 2024-03-11T20:51:58.185 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 11 20:51:57 smithi023 bash[58141]: ts=2024-03-11T20:51:57.980Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-03-11T20:51:58.185 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 11 20:51:57 smithi023 bash[58141]: ts=2024-03-11T20:51:57.980Z caller=node_exporter.go:117 level=info collector=pressure 2024-03-11T20:51:58.185 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 11 20:51:57 smithi023 bash[58141]: ts=2024-03-11T20:51:57.980Z caller=node_exporter.go:117 level=info collector=rapl 2024-03-11T20:51:58.185 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 11 20:51:57 smithi023 bash[58141]: ts=2024-03-11T20:51:57.980Z caller=node_exporter.go:117 level=info collector=schedstat 2024-03-11T20:51:58.185 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 11 20:51:57 smithi023 bash[58141]: ts=2024-03-11T20:51:57.980Z caller=node_exporter.go:117 level=info collector=selinux 2024-03-11T20:51:58.185 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 11 20:51:57 smithi023 bash[58141]: ts=2024-03-11T20:51:57.980Z caller=node_exporter.go:117 level=info collector=sockstat 2024-03-11T20:51:58.185 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 11 20:51:57 smithi023 bash[58141]: ts=2024-03-11T20:51:57.980Z caller=node_exporter.go:117 level=info collector=softnet 2024-03-11T20:51:58.185 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 11 20:51:57 smithi023 bash[58141]: ts=2024-03-11T20:51:57.980Z caller=node_exporter.go:117 level=info collector=stat 2024-03-11T20:51:58.185 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 11 20:51:57 smithi023 bash[58141]: ts=2024-03-11T20:51:57.980Z caller=node_exporter.go:117 level=info collector=tapestats 2024-03-11T20:51:58.185 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 11 20:51:57 smithi023 bash[58141]: ts=2024-03-11T20:51:57.980Z caller=node_exporter.go:117 level=info collector=textfile 2024-03-11T20:51:58.185 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 11 20:51:57 smithi023 bash[58141]: ts=2024-03-11T20:51:57.980Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-03-11T20:51:58.185 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 11 20:51:57 smithi023 bash[58141]: ts=2024-03-11T20:51:57.980Z caller=node_exporter.go:117 level=info collector=time 2024-03-11T20:51:58.185 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 11 20:51:57 smithi023 bash[58141]: ts=2024-03-11T20:51:57.980Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-03-11T20:51:58.186 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 11 20:51:57 smithi023 bash[58141]: ts=2024-03-11T20:51:57.980Z caller=node_exporter.go:117 level=info collector=uname 2024-03-11T20:51:58.186 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 11 20:51:57 smithi023 bash[58141]: ts=2024-03-11T20:51:57.980Z caller=node_exporter.go:117 level=info collector=vmstat 2024-03-11T20:51:58.186 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 11 20:51:57 smithi023 bash[58141]: ts=2024-03-11T20:51:57.980Z caller=node_exporter.go:117 level=info collector=xfs 2024-03-11T20:51:58.186 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 11 20:51:57 smithi023 bash[58141]: ts=2024-03-11T20:51:57.980Z caller=node_exporter.go:117 level=info collector=zfs 2024-03-11T20:51:58.186 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 11 20:51:57 smithi023 bash[58141]: ts=2024-03-11T20:51:57.980Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-03-11T20:51:58.186 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 11 20:51:57 smithi023 bash[58141]: ts=2024-03-11T20:51:57.980Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-03-11T20:51:58.260 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Mar 11 20:51:57 smithi146 bash[51341]: Status: Downloaded newer image for quay.io/prometheus/node-exporter:v1.5.0 2024-03-11T20:51:58.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:58 smithi023 bash[21372]: cluster 2024-03-11T20:51:56.289524+0000 osd.3 (osd.3) 27 : cluster 0 6.1b scrub starts 2024-03-11T20:51:58.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:58 smithi023 bash[21372]: cluster 2024-03-11T20:51:56.292086+0000 osd.3 (osd.3) 28 : cluster 0 6.1b scrub ok 2024-03-11T20:51:58.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:58 smithi023 bash[21372]: cluster 2024-03-11T20:51:56.891645+0000 osd.5 (osd.5) 29 : cluster 0 6.a scrub starts 2024-03-11T20:51:58.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:58 smithi023 bash[21372]: cluster 2024-03-11T20:51:56.893257+0000 osd.5 (osd.5) 30 : cluster 0 6.a scrub ok 2024-03-11T20:51:58.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:58 smithi023 bash[21372]: cluster 2024-03-11T20:51:57.089572+0000 osd.7 (osd.7) 25 : cluster 0 3.f scrub starts 2024-03-11T20:51:58.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:58 smithi023 bash[21372]: cluster 2024-03-11T20:51:57.094389+0000 osd.7 (osd.7) 26 : cluster 0 3.f scrub ok 2024-03-11T20:51:58.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:58 smithi023 bash[21372]: cluster 2024-03-11T20:51:57.170564+0000 mon.a (mon.0) 853 : cluster 0 osdmap e104: 8 total, 8 up, 8 in 2024-03-11T20:51:58.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:58 smithi023 bash[29146]: cluster 2024-03-11T20:51:56.289524+0000 osd.3 (osd.3) 27 : cluster 0 6.1b scrub starts 2024-03-11T20:51:58.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:58 smithi023 bash[29146]: cluster 2024-03-11T20:51:56.292086+0000 osd.3 (osd.3) 28 : cluster 0 6.1b scrub ok 2024-03-11T20:51:58.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:58 smithi023 bash[29146]: cluster 2024-03-11T20:51:56.891645+0000 osd.5 (osd.5) 29 : cluster 0 6.a scrub starts 2024-03-11T20:51:58.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:58 smithi023 bash[29146]: cluster 2024-03-11T20:51:56.893257+0000 osd.5 (osd.5) 30 : cluster 0 6.a scrub ok 2024-03-11T20:51:58.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:58 smithi023 bash[29146]: cluster 2024-03-11T20:51:57.089572+0000 osd.7 (osd.7) 25 : cluster 0 3.f scrub starts 2024-03-11T20:51:58.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:58 smithi023 bash[29146]: cluster 2024-03-11T20:51:57.094389+0000 osd.7 (osd.7) 26 : cluster 0 3.f scrub ok 2024-03-11T20:51:58.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:58 smithi023 bash[29146]: cluster 2024-03-11T20:51:57.170564+0000 mon.a (mon.0) 853 : cluster 0 osdmap e104: 8 total, 8 up, 8 in 2024-03-11T20:51:58.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:58 smithi146 bash[23809]: cluster 2024-03-11T20:51:56.289524+0000 osd.3 (osd.3) 27 : cluster 0 6.1b scrub starts 2024-03-11T20:51:58.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:58 smithi146 bash[23809]: cluster 2024-03-11T20:51:56.292086+0000 osd.3 (osd.3) 28 : cluster 0 6.1b scrub ok 2024-03-11T20:51:58.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:58 smithi146 bash[23809]: cluster 2024-03-11T20:51:56.891645+0000 osd.5 (osd.5) 29 : cluster 0 6.a scrub starts 2024-03-11T20:51:58.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:58 smithi146 bash[23809]: cluster 2024-03-11T20:51:56.893257+0000 osd.5 (osd.5) 30 : cluster 0 6.a scrub ok 2024-03-11T20:51:58.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:58 smithi146 bash[23809]: cluster 2024-03-11T20:51:57.089572+0000 osd.7 (osd.7) 25 : cluster 0 3.f scrub starts 2024-03-11T20:51:58.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:58 smithi146 bash[23809]: cluster 2024-03-11T20:51:57.094389+0000 osd.7 (osd.7) 26 : cluster 0 3.f scrub ok 2024-03-11T20:51:58.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:58 smithi146 bash[23809]: cluster 2024-03-11T20:51:57.170564+0000 mon.a (mon.0) 853 : cluster 0 osdmap e104: 8 total, 8 up, 8 in 2024-03-11T20:51:59.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:59 smithi023 bash[29146]: cluster 2024-03-11T20:51:57.709798+0000 osd.2 (osd.2) 25 : cluster 0 2.10 scrub starts 2024-03-11T20:51:59.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:59 smithi023 bash[29146]: cluster 2024-03-11T20:51:57.712211+0000 osd.2 (osd.2) 26 : cluster 0 2.10 scrub ok 2024-03-11T20:51:59.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:59 smithi023 bash[29146]: cluster 2024-03-11T20:51:57.820164+0000 mgr.y (mgr.24380) 38 : cluster 0 pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-11T20:51:59.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:51:59 smithi023 bash[29146]: audit 2024-03-11T20:51:58.935419+0000 mon.a (mon.0) 854 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:51:59.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:59 smithi023 bash[21372]: cluster 2024-03-11T20:51:57.709798+0000 osd.2 (osd.2) 25 : cluster 0 2.10 scrub starts 2024-03-11T20:51:59.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:59 smithi023 bash[21372]: cluster 2024-03-11T20:51:57.712211+0000 osd.2 (osd.2) 26 : cluster 0 2.10 scrub ok 2024-03-11T20:51:59.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:59 smithi023 bash[21372]: cluster 2024-03-11T20:51:57.820164+0000 mgr.y (mgr.24380) 38 : cluster 0 pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-11T20:51:59.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:51:59 smithi023 bash[21372]: audit 2024-03-11T20:51:58.935419+0000 mon.a (mon.0) 854 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:51:59.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:59 smithi146 bash[23809]: cluster 2024-03-11T20:51:57.709798+0000 osd.2 (osd.2) 25 : cluster 0 2.10 scrub starts 2024-03-11T20:51:59.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:59 smithi146 bash[23809]: cluster 2024-03-11T20:51:57.712211+0000 osd.2 (osd.2) 26 : cluster 0 2.10 scrub ok 2024-03-11T20:51:59.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:59 smithi146 bash[23809]: cluster 2024-03-11T20:51:57.820164+0000 mgr.y (mgr.24380) 38 : cluster 0 pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-11T20:51:59.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:51:59 smithi146 bash[23809]: audit 2024-03-11T20:51:58.935419+0000 mon.a (mon.0) 854 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:52:00.511 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Mar 11 20:52:00 smithi146 bash[51341]: ts=2024-03-11T20:52:00.124Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-03-11T20:52:00.511 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Mar 11 20:52:00 smithi146 bash[51341]: ts=2024-03-11T20:52:00.125Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-03-11T20:52:00.511 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Mar 11 20:52:00 smithi146 bash[51341]: ts=2024-03-11T20:52:00.125Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-03-11T20:52:00.511 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Mar 11 20:52:00 smithi146 bash[51341]: ts=2024-03-11T20:52:00.125Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-11T20:52:00.511 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Mar 11 20:52:00 smithi146 bash[51341]: ts=2024-03-11T20:52:00.126Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-03-11T20:52:00.511 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Mar 11 20:52:00 smithi146 bash[51341]: ts=2024-03-11T20:52:00.126Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-03-11T20:52:00.511 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Mar 11 20:52:00 smithi146 bash[51341]: ts=2024-03-11T20:52:00.127Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-03-11T20:52:00.512 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Mar 11 20:52:00 smithi146 bash[51341]: ts=2024-03-11T20:52:00.127Z caller=node_exporter.go:117 level=info collector=arp 2024-03-11T20:52:00.512 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Mar 11 20:52:00 smithi146 bash[51341]: ts=2024-03-11T20:52:00.127Z caller=node_exporter.go:117 level=info collector=bcache 2024-03-11T20:52:00.512 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Mar 11 20:52:00 smithi146 bash[51341]: ts=2024-03-11T20:52:00.127Z caller=node_exporter.go:117 level=info collector=bonding 2024-03-11T20:52:00.512 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Mar 11 20:52:00 smithi146 bash[51341]: ts=2024-03-11T20:52:00.127Z caller=node_exporter.go:117 level=info collector=btrfs 2024-03-11T20:52:00.512 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Mar 11 20:52:00 smithi146 bash[51341]: ts=2024-03-11T20:52:00.127Z caller=node_exporter.go:117 level=info collector=conntrack 2024-03-11T20:52:00.512 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Mar 11 20:52:00 smithi146 bash[51341]: ts=2024-03-11T20:52:00.127Z caller=node_exporter.go:117 level=info collector=cpu 2024-03-11T20:52:00.512 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Mar 11 20:52:00 smithi146 bash[51341]: ts=2024-03-11T20:52:00.127Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-03-11T20:52:00.512 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Mar 11 20:52:00 smithi146 bash[51341]: ts=2024-03-11T20:52:00.127Z caller=node_exporter.go:117 level=info collector=diskstats 2024-03-11T20:52:00.512 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Mar 11 20:52:00 smithi146 bash[51341]: ts=2024-03-11T20:52:00.127Z caller=node_exporter.go:117 level=info collector=dmi 2024-03-11T20:52:00.512 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Mar 11 20:52:00 smithi146 bash[51341]: ts=2024-03-11T20:52:00.127Z caller=node_exporter.go:117 level=info collector=edac 2024-03-11T20:52:00.512 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Mar 11 20:52:00 smithi146 bash[51341]: ts=2024-03-11T20:52:00.127Z caller=node_exporter.go:117 level=info collector=entropy 2024-03-11T20:52:00.512 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Mar 11 20:52:00 smithi146 bash[51341]: ts=2024-03-11T20:52:00.127Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-03-11T20:52:00.512 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Mar 11 20:52:00 smithi146 bash[51341]: ts=2024-03-11T20:52:00.127Z caller=node_exporter.go:117 level=info collector=filefd 2024-03-11T20:52:00.512 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Mar 11 20:52:00 smithi146 bash[51341]: ts=2024-03-11T20:52:00.127Z caller=node_exporter.go:117 level=info collector=filesystem 2024-03-11T20:52:00.512 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Mar 11 20:52:00 smithi146 bash[51341]: ts=2024-03-11T20:52:00.127Z caller=node_exporter.go:117 level=info collector=hwmon 2024-03-11T20:52:00.512 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Mar 11 20:52:00 smithi146 bash[51341]: ts=2024-03-11T20:52:00.127Z caller=node_exporter.go:117 level=info collector=infiniband 2024-03-11T20:52:00.513 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Mar 11 20:52:00 smithi146 bash[51341]: ts=2024-03-11T20:52:00.127Z caller=node_exporter.go:117 level=info collector=ipvs 2024-03-11T20:52:00.513 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Mar 11 20:52:00 smithi146 bash[51341]: ts=2024-03-11T20:52:00.127Z caller=node_exporter.go:117 level=info collector=loadavg 2024-03-11T20:52:00.513 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Mar 11 20:52:00 smithi146 bash[51341]: ts=2024-03-11T20:52:00.127Z caller=node_exporter.go:117 level=info collector=mdadm 2024-03-11T20:52:00.513 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Mar 11 20:52:00 smithi146 bash[51341]: ts=2024-03-11T20:52:00.127Z caller=node_exporter.go:117 level=info collector=meminfo 2024-03-11T20:52:00.513 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Mar 11 20:52:00 smithi146 bash[51341]: ts=2024-03-11T20:52:00.127Z caller=node_exporter.go:117 level=info collector=netclass 2024-03-11T20:52:00.513 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Mar 11 20:52:00 smithi146 bash[51341]: ts=2024-03-11T20:52:00.127Z caller=node_exporter.go:117 level=info collector=netdev 2024-03-11T20:52:00.513 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Mar 11 20:52:00 smithi146 bash[51341]: ts=2024-03-11T20:52:00.127Z caller=node_exporter.go:117 level=info collector=netstat 2024-03-11T20:52:00.513 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Mar 11 20:52:00 smithi146 bash[51341]: ts=2024-03-11T20:52:00.127Z caller=node_exporter.go:117 level=info collector=nfs 2024-03-11T20:52:00.513 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Mar 11 20:52:00 smithi146 bash[51341]: ts=2024-03-11T20:52:00.127Z caller=node_exporter.go:117 level=info collector=nfsd 2024-03-11T20:52:00.513 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Mar 11 20:52:00 smithi146 bash[51341]: ts=2024-03-11T20:52:00.127Z caller=node_exporter.go:117 level=info collector=nvme 2024-03-11T20:52:00.513 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Mar 11 20:52:00 smithi146 bash[51341]: ts=2024-03-11T20:52:00.127Z caller=node_exporter.go:117 level=info collector=os 2024-03-11T20:52:00.514 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Mar 11 20:52:00 smithi146 bash[51341]: ts=2024-03-11T20:52:00.127Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-03-11T20:52:00.514 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Mar 11 20:52:00 smithi146 bash[51341]: ts=2024-03-11T20:52:00.127Z caller=node_exporter.go:117 level=info collector=pressure 2024-03-11T20:52:00.514 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Mar 11 20:52:00 smithi146 bash[51341]: ts=2024-03-11T20:52:00.127Z caller=node_exporter.go:117 level=info collector=rapl 2024-03-11T20:52:00.514 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Mar 11 20:52:00 smithi146 bash[51341]: ts=2024-03-11T20:52:00.127Z caller=node_exporter.go:117 level=info collector=schedstat 2024-03-11T20:52:00.514 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Mar 11 20:52:00 smithi146 bash[51341]: ts=2024-03-11T20:52:00.127Z caller=node_exporter.go:117 level=info collector=selinux 2024-03-11T20:52:00.514 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Mar 11 20:52:00 smithi146 bash[51341]: ts=2024-03-11T20:52:00.127Z caller=node_exporter.go:117 level=info collector=sockstat 2024-03-11T20:52:00.514 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Mar 11 20:52:00 smithi146 bash[51341]: ts=2024-03-11T20:52:00.127Z caller=node_exporter.go:117 level=info collector=softnet 2024-03-11T20:52:00.514 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Mar 11 20:52:00 smithi146 bash[51341]: ts=2024-03-11T20:52:00.127Z caller=node_exporter.go:117 level=info collector=stat 2024-03-11T20:52:00.514 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Mar 11 20:52:00 smithi146 bash[51341]: ts=2024-03-11T20:52:00.127Z caller=node_exporter.go:117 level=info collector=tapestats 2024-03-11T20:52:00.514 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Mar 11 20:52:00 smithi146 bash[51341]: ts=2024-03-11T20:52:00.127Z caller=node_exporter.go:117 level=info collector=textfile 2024-03-11T20:52:00.514 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Mar 11 20:52:00 smithi146 bash[51341]: ts=2024-03-11T20:52:00.127Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-03-11T20:52:00.514 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Mar 11 20:52:00 smithi146 bash[51341]: ts=2024-03-11T20:52:00.127Z caller=node_exporter.go:117 level=info collector=time 2024-03-11T20:52:00.514 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Mar 11 20:52:00 smithi146 bash[51341]: ts=2024-03-11T20:52:00.127Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-03-11T20:52:00.514 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Mar 11 20:52:00 smithi146 bash[51341]: ts=2024-03-11T20:52:00.127Z caller=node_exporter.go:117 level=info collector=uname 2024-03-11T20:52:00.515 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Mar 11 20:52:00 smithi146 bash[51341]: ts=2024-03-11T20:52:00.127Z caller=node_exporter.go:117 level=info collector=vmstat 2024-03-11T20:52:00.515 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Mar 11 20:52:00 smithi146 bash[51341]: ts=2024-03-11T20:52:00.127Z caller=node_exporter.go:117 level=info collector=xfs 2024-03-11T20:52:00.515 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Mar 11 20:52:00 smithi146 bash[51341]: ts=2024-03-11T20:52:00.127Z caller=node_exporter.go:117 level=info collector=zfs 2024-03-11T20:52:00.515 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Mar 11 20:52:00 smithi146 bash[51341]: ts=2024-03-11T20:52:00.128Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-03-11T20:52:00.515 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Mar 11 20:52:00 smithi146 bash[51341]: ts=2024-03-11T20:52:00.128Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-03-11T20:52:00.784 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:00 smithi023 bash[21372]: cluster 2024-03-11T20:51:59.128528+0000 osd.1 (osd.1) 27 : cluster 0 4.12 scrub starts 2024-03-11T20:52:00.784 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:00 smithi023 bash[21372]: cluster 2024-03-11T20:51:59.129708+0000 osd.1 (osd.1) 28 : cluster 0 4.12 scrub ok 2024-03-11T20:52:00.784 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:00 smithi023 bash[21372]: cluster 2024-03-11T20:51:59.382053+0000 osd.3 (osd.3) 29 : cluster 0 2.1b scrub starts 2024-03-11T20:52:00.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:00 smithi023 bash[21372]: cluster 2024-03-11T20:51:59.384631+0000 osd.3 (osd.3) 30 : cluster 0 2.1b scrub ok 2024-03-11T20:52:00.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:00 smithi023 bash[21372]: cluster 2024-03-11T20:51:59.525590+0000 osd.4 (osd.4) 27 : cluster 0 3.9 scrub starts 2024-03-11T20:52:00.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:00 smithi023 bash[21372]: cluster 2024-03-11T20:51:59.529043+0000 osd.4 (osd.4) 28 : cluster 0 3.9 scrub ok 2024-03-11T20:52:00.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:00 smithi023 bash[21372]: cluster 2024-03-11T20:51:59.772478+0000 osd.0 (osd.0) 33 : cluster 0 3.17 scrub starts 2024-03-11T20:52:00.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:00 smithi023 bash[21372]: cluster 2024-03-11T20:51:59.774846+0000 osd.0 (osd.0) 34 : cluster 0 3.17 scrub ok 2024-03-11T20:52:00.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:00 smithi023 bash[21372]: cluster 2024-03-11T20:51:59.820971+0000 mgr.y (mgr.24380) 39 : cluster 0 pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T20:52:00.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:00 smithi023 bash[29146]: cluster 2024-03-11T20:51:59.128528+0000 osd.1 (osd.1) 27 : cluster 0 4.12 scrub starts 2024-03-11T20:52:00.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:00 smithi023 bash[29146]: cluster 2024-03-11T20:51:59.129708+0000 osd.1 (osd.1) 28 : cluster 0 4.12 scrub ok 2024-03-11T20:52:00.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:00 smithi023 bash[29146]: cluster 2024-03-11T20:51:59.382053+0000 osd.3 (osd.3) 29 : cluster 0 2.1b scrub starts 2024-03-11T20:52:00.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:00 smithi023 bash[29146]: cluster 2024-03-11T20:51:59.384631+0000 osd.3 (osd.3) 30 : cluster 0 2.1b scrub ok 2024-03-11T20:52:00.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:00 smithi023 bash[29146]: cluster 2024-03-11T20:51:59.525590+0000 osd.4 (osd.4) 27 : cluster 0 3.9 scrub starts 2024-03-11T20:52:00.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:00 smithi023 bash[29146]: cluster 2024-03-11T20:51:59.529043+0000 osd.4 (osd.4) 28 : cluster 0 3.9 scrub ok 2024-03-11T20:52:00.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:00 smithi023 bash[29146]: cluster 2024-03-11T20:51:59.772478+0000 osd.0 (osd.0) 33 : cluster 0 3.17 scrub starts 2024-03-11T20:52:00.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:00 smithi023 bash[29146]: cluster 2024-03-11T20:51:59.774846+0000 osd.0 (osd.0) 34 : cluster 0 3.17 scrub ok 2024-03-11T20:52:00.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:00 smithi023 bash[29146]: cluster 2024-03-11T20:51:59.820971+0000 mgr.y (mgr.24380) 39 : cluster 0 pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T20:52:00.905 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-11T20:52:01.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:00 smithi146 bash[23809]: cluster 2024-03-11T20:51:59.128528+0000 osd.1 (osd.1) 27 : cluster 0 4.12 scrub starts 2024-03-11T20:52:01.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:00 smithi146 bash[23809]: cluster 2024-03-11T20:51:59.129708+0000 osd.1 (osd.1) 28 : cluster 0 4.12 scrub ok 2024-03-11T20:52:01.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:00 smithi146 bash[23809]: cluster 2024-03-11T20:51:59.382053+0000 osd.3 (osd.3) 29 : cluster 0 2.1b scrub starts 2024-03-11T20:52:01.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:00 smithi146 bash[23809]: cluster 2024-03-11T20:51:59.384631+0000 osd.3 (osd.3) 30 : cluster 0 2.1b scrub ok 2024-03-11T20:52:01.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:00 smithi146 bash[23809]: cluster 2024-03-11T20:51:59.525590+0000 osd.4 (osd.4) 27 : cluster 0 3.9 scrub starts 2024-03-11T20:52:01.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:00 smithi146 bash[23809]: cluster 2024-03-11T20:51:59.529043+0000 osd.4 (osd.4) 28 : cluster 0 3.9 scrub ok 2024-03-11T20:52:01.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:00 smithi146 bash[23809]: cluster 2024-03-11T20:51:59.772478+0000 osd.0 (osd.0) 33 : cluster 0 3.17 scrub starts 2024-03-11T20:52:01.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:00 smithi146 bash[23809]: cluster 2024-03-11T20:51:59.774846+0000 osd.0 (osd.0) 34 : cluster 0 3.17 scrub ok 2024-03-11T20:52:01.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:00 smithi146 bash[23809]: cluster 2024-03-11T20:51:59.820971+0000 mgr.y (mgr.24380) 39 : cluster 0 pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T20:52:01.589 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":22,"flags":0,"active_gid":24380,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6800","nonce":1501533945},{"type":"v1","addr":"172.21.15.23:6801","nonce":1501533945}]},"active_addr":"172.21.15.23:6801/1501533945","active_change":"2024-03-11T20:51:33.790784+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24436,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.23:8443/","prometheus":"http://172.21.15.23:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":101,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.23:0","nonce":2319417683}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.23:0","nonce":2089680079}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.23:0","nonce":272691770}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.23:0","nonce":1843845734}]}]} 2024-03-11T20:52:01.594 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-11T20:52:01.594 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-11T20:52:01.594 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd dump --format=json 2024-03-11T20:52:02.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:01 smithi146 bash[23809]: cluster 2024-03-11T20:51:59.677813+0000 osd.6 (osd.6) 15 : cluster 0 2.1a scrub starts 2024-03-11T20:52:02.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:01 smithi146 bash[23809]: cluster 2024-03-11T20:51:59.680614+0000 osd.6 (osd.6) 16 : cluster 0 2.1a scrub ok 2024-03-11T20:52:02.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:01 smithi146 bash[23809]: cluster 2024-03-11T20:52:00.070125+0000 osd.7 (osd.7) 27 : cluster 0 2.5 scrub starts 2024-03-11T20:52:02.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:01 smithi146 bash[23809]: cluster 2024-03-11T20:52:00.071798+0000 osd.7 (osd.7) 28 : cluster 0 2.5 scrub ok 2024-03-11T20:52:02.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:01 smithi146 bash[23809]: cluster 2024-03-11T20:52:00.805791+0000 osd.0 (osd.0) 35 : cluster 0 3.1b scrub starts 2024-03-11T20:52:02.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:01 smithi146 bash[23809]: cluster 2024-03-11T20:52:00.808979+0000 osd.0 (osd.0) 36 : cluster 0 3.1b scrub ok 2024-03-11T20:52:02.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:01 smithi146 bash[23809]: audit 2024-03-11T20:52:00.901760+0000 mon.a (mon.0) 855 : audit 0 from='client.? 172.21.15.23:0/1671003372' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-11T20:52:02.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:01 smithi023 bash[21372]: cluster 2024-03-11T20:51:59.677813+0000 osd.6 (osd.6) 15 : cluster 0 2.1a scrub starts 2024-03-11T20:52:02.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:01 smithi023 bash[21372]: cluster 2024-03-11T20:51:59.680614+0000 osd.6 (osd.6) 16 : cluster 0 2.1a scrub ok 2024-03-11T20:52:02.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:01 smithi023 bash[21372]: cluster 2024-03-11T20:52:00.070125+0000 osd.7 (osd.7) 27 : cluster 0 2.5 scrub starts 2024-03-11T20:52:02.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:01 smithi023 bash[21372]: cluster 2024-03-11T20:52:00.071798+0000 osd.7 (osd.7) 28 : cluster 0 2.5 scrub ok 2024-03-11T20:52:02.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:01 smithi023 bash[21372]: cluster 2024-03-11T20:52:00.805791+0000 osd.0 (osd.0) 35 : cluster 0 3.1b scrub starts 2024-03-11T20:52:02.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:01 smithi023 bash[21372]: cluster 2024-03-11T20:52:00.808979+0000 osd.0 (osd.0) 36 : cluster 0 3.1b scrub ok 2024-03-11T20:52:02.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:01 smithi023 bash[21372]: audit 2024-03-11T20:52:00.901760+0000 mon.a (mon.0) 855 : audit 0 from='client.? 172.21.15.23:0/1671003372' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-11T20:52:02.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:01 smithi023 bash[29146]: cluster 2024-03-11T20:51:59.677813+0000 osd.6 (osd.6) 15 : cluster 0 2.1a scrub starts 2024-03-11T20:52:02.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:01 smithi023 bash[29146]: cluster 2024-03-11T20:51:59.680614+0000 osd.6 (osd.6) 16 : cluster 0 2.1a scrub ok 2024-03-11T20:52:02.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:01 smithi023 bash[29146]: cluster 2024-03-11T20:52:00.070125+0000 osd.7 (osd.7) 27 : cluster 0 2.5 scrub starts 2024-03-11T20:52:02.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:01 smithi023 bash[29146]: cluster 2024-03-11T20:52:00.071798+0000 osd.7 (osd.7) 28 : cluster 0 2.5 scrub ok 2024-03-11T20:52:02.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:01 smithi023 bash[29146]: cluster 2024-03-11T20:52:00.805791+0000 osd.0 (osd.0) 35 : cluster 0 3.1b scrub starts 2024-03-11T20:52:02.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:01 smithi023 bash[29146]: cluster 2024-03-11T20:52:00.808979+0000 osd.0 (osd.0) 36 : cluster 0 3.1b scrub ok 2024-03-11T20:52:02.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:01 smithi023 bash[29146]: audit 2024-03-11T20:52:00.901760+0000 mon.a (mon.0) 855 : audit 0 from='client.? 172.21.15.23:0/1671003372' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-11T20:52:03.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:02 smithi146 bash[23809]: cluster 2024-03-11T20:52:00.916268+0000 osd.5 (osd.5) 31 : cluster 0 6.19 deep-scrub starts 2024-03-11T20:52:03.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:02 smithi146 bash[23809]: cluster 2024-03-11T20:52:00.918441+0000 osd.5 (osd.5) 32 : cluster 0 6.19 deep-scrub ok 2024-03-11T20:52:03.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:02 smithi146 bash[23809]: cluster 2024-03-11T20:52:01.103172+0000 osd.1 (osd.1) 29 : cluster 0 2.d scrub starts 2024-03-11T20:52:03.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:02 smithi146 bash[23809]: cluster 2024-03-11T20:52:01.105481+0000 osd.1 (osd.1) 30 : cluster 0 2.d scrub ok 2024-03-11T20:52:03.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:02 smithi146 bash[23809]: cluster 2024-03-11T20:52:01.509966+0000 osd.4 (osd.4) 29 : cluster 0 4.d scrub starts 2024-03-11T20:52:03.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:02 smithi146 bash[23809]: cluster 2024-03-11T20:52:01.513291+0000 osd.4 (osd.4) 30 : cluster 0 4.d scrub ok 2024-03-11T20:52:03.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:02 smithi146 bash[23809]: cluster 2024-03-11T20:52:01.821569+0000 mgr.y (mgr.24380) 40 : cluster 0 pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T20:52:03.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:02 smithi146 bash[23809]: cluster 2024-03-11T20:52:01.843931+0000 osd.2 (osd.2) 27 : cluster 0 2.c scrub starts 2024-03-11T20:52:03.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:02 smithi146 bash[23809]: cluster 2024-03-11T20:52:01.845541+0000 osd.2 (osd.2) 28 : cluster 0 2.c scrub ok 2024-03-11T20:52:03.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:02 smithi023 bash[21372]: cluster 2024-03-11T20:52:00.916268+0000 osd.5 (osd.5) 31 : cluster 0 6.19 deep-scrub starts 2024-03-11T20:52:03.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:02 smithi023 bash[21372]: cluster 2024-03-11T20:52:00.918441+0000 osd.5 (osd.5) 32 : cluster 0 6.19 deep-scrub ok 2024-03-11T20:52:03.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:02 smithi023 bash[21372]: cluster 2024-03-11T20:52:01.103172+0000 osd.1 (osd.1) 29 : cluster 0 2.d scrub starts 2024-03-11T20:52:03.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:02 smithi023 bash[21372]: cluster 2024-03-11T20:52:01.105481+0000 osd.1 (osd.1) 30 : cluster 0 2.d scrub ok 2024-03-11T20:52:03.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:02 smithi023 bash[21372]: cluster 2024-03-11T20:52:01.509966+0000 osd.4 (osd.4) 29 : cluster 0 4.d scrub starts 2024-03-11T20:52:03.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:02 smithi023 bash[21372]: cluster 2024-03-11T20:52:01.513291+0000 osd.4 (osd.4) 30 : cluster 0 4.d scrub ok 2024-03-11T20:52:03.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:02 smithi023 bash[21372]: cluster 2024-03-11T20:52:01.821569+0000 mgr.y (mgr.24380) 40 : cluster 0 pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T20:52:03.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:02 smithi023 bash[21372]: cluster 2024-03-11T20:52:01.843931+0000 osd.2 (osd.2) 27 : cluster 0 2.c scrub starts 2024-03-11T20:52:03.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:02 smithi023 bash[21372]: cluster 2024-03-11T20:52:01.845541+0000 osd.2 (osd.2) 28 : cluster 0 2.c scrub ok 2024-03-11T20:52:03.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:02 smithi023 bash[29146]: cluster 2024-03-11T20:52:00.916268+0000 osd.5 (osd.5) 31 : cluster 0 6.19 deep-scrub starts 2024-03-11T20:52:03.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:02 smithi023 bash[29146]: cluster 2024-03-11T20:52:00.918441+0000 osd.5 (osd.5) 32 : cluster 0 6.19 deep-scrub ok 2024-03-11T20:52:03.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:02 smithi023 bash[29146]: cluster 2024-03-11T20:52:01.103172+0000 osd.1 (osd.1) 29 : cluster 0 2.d scrub starts 2024-03-11T20:52:03.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:02 smithi023 bash[29146]: cluster 2024-03-11T20:52:01.105481+0000 osd.1 (osd.1) 30 : cluster 0 2.d scrub ok 2024-03-11T20:52:03.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:02 smithi023 bash[29146]: cluster 2024-03-11T20:52:01.509966+0000 osd.4 (osd.4) 29 : cluster 0 4.d scrub starts 2024-03-11T20:52:03.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:02 smithi023 bash[29146]: cluster 2024-03-11T20:52:01.513291+0000 osd.4 (osd.4) 30 : cluster 0 4.d scrub ok 2024-03-11T20:52:03.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:02 smithi023 bash[29146]: cluster 2024-03-11T20:52:01.821569+0000 mgr.y (mgr.24380) 40 : cluster 0 pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T20:52:03.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:02 smithi023 bash[29146]: cluster 2024-03-11T20:52:01.843931+0000 osd.2 (osd.2) 27 : cluster 0 2.c scrub starts 2024-03-11T20:52:03.036 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:02 smithi023 bash[29146]: cluster 2024-03-11T20:52:01.845541+0000 osd.2 (osd.2) 28 : cluster 0 2.c scrub ok 2024-03-11T20:52:04.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:03 smithi146 bash[23809]: cluster 2024-03-11T20:52:01.876633+0000 osd.5 (osd.5) 33 : cluster 0 2.18 deep-scrub starts 2024-03-11T20:52:04.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:03 smithi146 bash[23809]: cluster 2024-03-11T20:52:01.880024+0000 osd.5 (osd.5) 34 : cluster 0 2.18 deep-scrub ok 2024-03-11T20:52:04.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:03 smithi146 bash[23809]: cluster 2024-03-11T20:52:02.139984+0000 osd.1 (osd.1) 31 : cluster 0 6.1d scrub starts 2024-03-11T20:52:04.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:03 smithi146 bash[23809]: cluster 2024-03-11T20:52:02.142351+0000 osd.1 (osd.1) 32 : cluster 0 6.1d scrub ok 2024-03-11T20:52:04.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:03 smithi146 bash[23809]: cluster 2024-03-11T20:52:02.548633+0000 osd.4 (osd.4) 31 : cluster 0 6.17 scrub starts 2024-03-11T20:52:04.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:03 smithi146 bash[23809]: cluster 2024-03-11T20:52:02.551636+0000 osd.4 (osd.4) 32 : cluster 0 6.17 scrub ok 2024-03-11T20:52:04.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:03 smithi146 bash[23809]: cluster 2024-03-11T20:52:02.649994+0000 osd.6 (osd.6) 17 : cluster 0 2.11 scrub starts 2024-03-11T20:52:04.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:03 smithi146 bash[23809]: cluster 2024-03-11T20:52:02.653968+0000 osd.6 (osd.6) 18 : cluster 0 2.11 scrub ok 2024-03-11T20:52:04.034 INFO:journalctl@ceph.alertmanager.a.smithi023.stdout:Mar 11 20:52:03 smithi023 bash[57975]: ts=2024-03-11T20:52:03.794Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.002952557s 2024-03-11T20:52:04.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:03 smithi023 bash[29146]: cluster 2024-03-11T20:52:01.876633+0000 osd.5 (osd.5) 33 : cluster 0 2.18 deep-scrub starts 2024-03-11T20:52:04.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:03 smithi023 bash[29146]: cluster 2024-03-11T20:52:01.880024+0000 osd.5 (osd.5) 34 : cluster 0 2.18 deep-scrub ok 2024-03-11T20:52:04.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:03 smithi023 bash[29146]: cluster 2024-03-11T20:52:02.139984+0000 osd.1 (osd.1) 31 : cluster 0 6.1d scrub starts 2024-03-11T20:52:04.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:03 smithi023 bash[29146]: cluster 2024-03-11T20:52:02.142351+0000 osd.1 (osd.1) 32 : cluster 0 6.1d scrub ok 2024-03-11T20:52:04.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:03 smithi023 bash[29146]: cluster 2024-03-11T20:52:02.548633+0000 osd.4 (osd.4) 31 : cluster 0 6.17 scrub starts 2024-03-11T20:52:04.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:03 smithi023 bash[29146]: cluster 2024-03-11T20:52:02.551636+0000 osd.4 (osd.4) 32 : cluster 0 6.17 scrub ok 2024-03-11T20:52:04.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:03 smithi023 bash[29146]: cluster 2024-03-11T20:52:02.649994+0000 osd.6 (osd.6) 17 : cluster 0 2.11 scrub starts 2024-03-11T20:52:04.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:03 smithi023 bash[29146]: cluster 2024-03-11T20:52:02.653968+0000 osd.6 (osd.6) 18 : cluster 0 2.11 scrub ok 2024-03-11T20:52:04.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:03 smithi023 bash[21372]: cluster 2024-03-11T20:52:01.876633+0000 osd.5 (osd.5) 33 : cluster 0 2.18 deep-scrub starts 2024-03-11T20:52:04.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:03 smithi023 bash[21372]: cluster 2024-03-11T20:52:01.880024+0000 osd.5 (osd.5) 34 : cluster 0 2.18 deep-scrub ok 2024-03-11T20:52:04.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:03 smithi023 bash[21372]: cluster 2024-03-11T20:52:02.139984+0000 osd.1 (osd.1) 31 : cluster 0 6.1d scrub starts 2024-03-11T20:52:04.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:03 smithi023 bash[21372]: cluster 2024-03-11T20:52:02.142351+0000 osd.1 (osd.1) 32 : cluster 0 6.1d scrub ok 2024-03-11T20:52:04.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:03 smithi023 bash[21372]: cluster 2024-03-11T20:52:02.548633+0000 osd.4 (osd.4) 31 : cluster 0 6.17 scrub starts 2024-03-11T20:52:04.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:03 smithi023 bash[21372]: cluster 2024-03-11T20:52:02.551636+0000 osd.4 (osd.4) 32 : cluster 0 6.17 scrub ok 2024-03-11T20:52:04.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:03 smithi023 bash[21372]: cluster 2024-03-11T20:52:02.649994+0000 osd.6 (osd.6) 17 : cluster 0 2.11 scrub starts 2024-03-11T20:52:04.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:03 smithi023 bash[21372]: cluster 2024-03-11T20:52:02.653968+0000 osd.6 (osd.6) 18 : cluster 0 2.11 scrub ok 2024-03-11T20:52:05.010 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 20:52:04 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T20:52:05.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:04 smithi146 bash[23809]: cluster 2024-03-11T20:52:03.079196+0000 osd.7 (osd.7) 29 : cluster 0 6.3 scrub starts 2024-03-11T20:52:05.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:04 smithi146 bash[23809]: cluster 2024-03-11T20:52:03.081981+0000 osd.7 (osd.7) 30 : cluster 0 6.3 scrub ok 2024-03-11T20:52:05.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:04 smithi146 bash[23809]: cluster 2024-03-11T20:52:03.353582+0000 osd.3 (osd.3) 31 : cluster 0 5.0 scrub starts 2024-03-11T20:52:05.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:04 smithi146 bash[23809]: cluster 2024-03-11T20:52:03.356627+0000 osd.3 (osd.3) 32 : cluster 0 5.0 scrub ok 2024-03-11T20:52:05.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:04 smithi146 bash[23809]: cluster 2024-03-11T20:52:03.580999+0000 osd.4 (osd.4) 33 : cluster 0 4.1b scrub starts 2024-03-11T20:52:05.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:04 smithi146 bash[23809]: cluster 2024-03-11T20:52:03.583574+0000 osd.4 (osd.4) 34 : cluster 0 4.1b scrub ok 2024-03-11T20:52:05.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:04 smithi146 bash[23809]: cluster 2024-03-11T20:52:03.822221+0000 mgr.y (mgr.24380) 41 : cluster 0 pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-03-11T20:52:05.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:04 smithi146 bash[23809]: cluster 2024-03-11T20:52:03.916331+0000 osd.0 (osd.0) 37 : cluster 0 6.10 scrub starts 2024-03-11T20:52:05.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:04 smithi146 bash[23809]: cluster 2024-03-11T20:52:03.918713+0000 osd.0 (osd.0) 38 : cluster 0 6.10 scrub ok 2024-03-11T20:52:05.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:04 smithi146 bash[23809]: audit 2024-03-11T20:52:03.983272+0000 mon.a (mon.0) 856 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:52:05.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:04 smithi023 bash[21372]: cluster 2024-03-11T20:52:03.079196+0000 osd.7 (osd.7) 29 : cluster 0 6.3 scrub starts 2024-03-11T20:52:05.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:04 smithi023 bash[21372]: cluster 2024-03-11T20:52:03.081981+0000 osd.7 (osd.7) 30 : cluster 0 6.3 scrub ok 2024-03-11T20:52:05.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:04 smithi023 bash[21372]: cluster 2024-03-11T20:52:03.353582+0000 osd.3 (osd.3) 31 : cluster 0 5.0 scrub starts 2024-03-11T20:52:05.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:04 smithi023 bash[21372]: cluster 2024-03-11T20:52:03.356627+0000 osd.3 (osd.3) 32 : cluster 0 5.0 scrub ok 2024-03-11T20:52:05.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:04 smithi023 bash[21372]: cluster 2024-03-11T20:52:03.580999+0000 osd.4 (osd.4) 33 : cluster 0 4.1b scrub starts 2024-03-11T20:52:05.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:04 smithi023 bash[21372]: cluster 2024-03-11T20:52:03.583574+0000 osd.4 (osd.4) 34 : cluster 0 4.1b scrub ok 2024-03-11T20:52:05.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:04 smithi023 bash[21372]: cluster 2024-03-11T20:52:03.822221+0000 mgr.y (mgr.24380) 41 : cluster 0 pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-03-11T20:52:05.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:04 smithi023 bash[21372]: cluster 2024-03-11T20:52:03.916331+0000 osd.0 (osd.0) 37 : cluster 0 6.10 scrub starts 2024-03-11T20:52:05.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:04 smithi023 bash[21372]: cluster 2024-03-11T20:52:03.918713+0000 osd.0 (osd.0) 38 : cluster 0 6.10 scrub ok 2024-03-11T20:52:05.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:04 smithi023 bash[21372]: audit 2024-03-11T20:52:03.983272+0000 mon.a (mon.0) 856 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:52:05.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:04 smithi023 bash[29146]: cluster 2024-03-11T20:52:03.079196+0000 osd.7 (osd.7) 29 : cluster 0 6.3 scrub starts 2024-03-11T20:52:05.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:04 smithi023 bash[29146]: cluster 2024-03-11T20:52:03.081981+0000 osd.7 (osd.7) 30 : cluster 0 6.3 scrub ok 2024-03-11T20:52:05.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:04 smithi023 bash[29146]: cluster 2024-03-11T20:52:03.353582+0000 osd.3 (osd.3) 31 : cluster 0 5.0 scrub starts 2024-03-11T20:52:05.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:04 smithi023 bash[29146]: cluster 2024-03-11T20:52:03.356627+0000 osd.3 (osd.3) 32 : cluster 0 5.0 scrub ok 2024-03-11T20:52:05.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:04 smithi023 bash[29146]: cluster 2024-03-11T20:52:03.580999+0000 osd.4 (osd.4) 33 : cluster 0 4.1b scrub starts 2024-03-11T20:52:05.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:04 smithi023 bash[29146]: cluster 2024-03-11T20:52:03.583574+0000 osd.4 (osd.4) 34 : cluster 0 4.1b scrub ok 2024-03-11T20:52:05.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:04 smithi023 bash[29146]: cluster 2024-03-11T20:52:03.822221+0000 mgr.y (mgr.24380) 41 : cluster 0 pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-03-11T20:52:05.036 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:04 smithi023 bash[29146]: cluster 2024-03-11T20:52:03.916331+0000 osd.0 (osd.0) 37 : cluster 0 6.10 scrub starts 2024-03-11T20:52:05.036 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:04 smithi023 bash[29146]: cluster 2024-03-11T20:52:03.918713+0000 osd.0 (osd.0) 38 : cluster 0 6.10 scrub ok 2024-03-11T20:52:05.036 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:04 smithi023 bash[29146]: audit 2024-03-11T20:52:03.983272+0000 mon.a (mon.0) 856 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:52:06.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:05 smithi146 bash[23809]: cluster 2024-03-11T20:52:03.930784+0000 osd.5 (osd.5) 35 : cluster 0 3.5 scrub starts 2024-03-11T20:52:06.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:05 smithi146 bash[23809]: cluster 2024-03-11T20:52:03.937337+0000 osd.5 (osd.5) 36 : cluster 0 3.5 scrub ok 2024-03-11T20:52:06.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:05 smithi146 bash[23809]: cluster 2024-03-11T20:52:04.152892+0000 osd.1 (osd.1) 33 : cluster 0 2.15 scrub starts 2024-03-11T20:52:06.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:05 smithi146 bash[23809]: cluster 2024-03-11T20:52:04.155209+0000 osd.1 (osd.1) 34 : cluster 0 2.15 scrub ok 2024-03-11T20:52:06.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:05 smithi146 bash[23809]: cluster 2024-03-11T20:52:04.385049+0000 osd.3 (osd.3) 33 : cluster 0 4.0 scrub starts 2024-03-11T20:52:06.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:05 smithi146 bash[23809]: cluster 2024-03-11T20:52:04.386509+0000 osd.3 (osd.3) 34 : cluster 0 4.0 scrub ok 2024-03-11T20:52:06.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:05 smithi146 bash[23809]: cluster 2024-03-11T20:52:04.646096+0000 osd.6 (osd.6) 19 : cluster 0 3.a scrub starts 2024-03-11T20:52:06.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:05 smithi146 bash[23809]: cluster 2024-03-11T20:52:04.651597+0000 osd.6 (osd.6) 20 : cluster 0 3.a scrub ok 2024-03-11T20:52:06.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:05 smithi146 bash[23809]: audit 2024-03-11T20:52:04.837958+0000 mgr.y (mgr.24380) 42 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:52:06.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:05 smithi146 bash[23809]: cluster 2024-03-11T20:52:04.900185+0000 osd.0 (osd.0) 39 : cluster 0 3.6 scrub starts 2024-03-11T20:52:06.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:05 smithi146 bash[23809]: cluster 2024-03-11T20:52:04.904710+0000 osd.0 (osd.0) 40 : cluster 0 3.6 scrub ok 2024-03-11T20:52:06.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:05 smithi023 bash[21372]: cluster 2024-03-11T20:52:03.930784+0000 osd.5 (osd.5) 35 : cluster 0 3.5 scrub starts 2024-03-11T20:52:06.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:05 smithi023 bash[21372]: cluster 2024-03-11T20:52:03.937337+0000 osd.5 (osd.5) 36 : cluster 0 3.5 scrub ok 2024-03-11T20:52:06.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:05 smithi023 bash[21372]: cluster 2024-03-11T20:52:04.152892+0000 osd.1 (osd.1) 33 : cluster 0 2.15 scrub starts 2024-03-11T20:52:06.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:05 smithi023 bash[21372]: cluster 2024-03-11T20:52:04.155209+0000 osd.1 (osd.1) 34 : cluster 0 2.15 scrub ok 2024-03-11T20:52:06.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:05 smithi023 bash[21372]: cluster 2024-03-11T20:52:04.385049+0000 osd.3 (osd.3) 33 : cluster 0 4.0 scrub starts 2024-03-11T20:52:06.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:05 smithi023 bash[21372]: cluster 2024-03-11T20:52:04.386509+0000 osd.3 (osd.3) 34 : cluster 0 4.0 scrub ok 2024-03-11T20:52:06.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:05 smithi023 bash[21372]: cluster 2024-03-11T20:52:04.646096+0000 osd.6 (osd.6) 19 : cluster 0 3.a scrub starts 2024-03-11T20:52:06.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:05 smithi023 bash[21372]: cluster 2024-03-11T20:52:04.651597+0000 osd.6 (osd.6) 20 : cluster 0 3.a scrub ok 2024-03-11T20:52:06.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:05 smithi023 bash[21372]: audit 2024-03-11T20:52:04.837958+0000 mgr.y (mgr.24380) 42 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:52:06.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:05 smithi023 bash[21372]: cluster 2024-03-11T20:52:04.900185+0000 osd.0 (osd.0) 39 : cluster 0 3.6 scrub starts 2024-03-11T20:52:06.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:05 smithi023 bash[21372]: cluster 2024-03-11T20:52:04.904710+0000 osd.0 (osd.0) 40 : cluster 0 3.6 scrub ok 2024-03-11T20:52:06.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:05 smithi023 bash[29146]: cluster 2024-03-11T20:52:03.930784+0000 osd.5 (osd.5) 35 : cluster 0 3.5 scrub starts 2024-03-11T20:52:06.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:05 smithi023 bash[29146]: cluster 2024-03-11T20:52:03.937337+0000 osd.5 (osd.5) 36 : cluster 0 3.5 scrub ok 2024-03-11T20:52:06.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:05 smithi023 bash[29146]: cluster 2024-03-11T20:52:04.152892+0000 osd.1 (osd.1) 33 : cluster 0 2.15 scrub starts 2024-03-11T20:52:06.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:05 smithi023 bash[29146]: cluster 2024-03-11T20:52:04.155209+0000 osd.1 (osd.1) 34 : cluster 0 2.15 scrub ok 2024-03-11T20:52:06.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:05 smithi023 bash[29146]: cluster 2024-03-11T20:52:04.385049+0000 osd.3 (osd.3) 33 : cluster 0 4.0 scrub starts 2024-03-11T20:52:06.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:05 smithi023 bash[29146]: cluster 2024-03-11T20:52:04.386509+0000 osd.3 (osd.3) 34 : cluster 0 4.0 scrub ok 2024-03-11T20:52:06.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:05 smithi023 bash[29146]: cluster 2024-03-11T20:52:04.646096+0000 osd.6 (osd.6) 19 : cluster 0 3.a scrub starts 2024-03-11T20:52:06.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:05 smithi023 bash[29146]: cluster 2024-03-11T20:52:04.651597+0000 osd.6 (osd.6) 20 : cluster 0 3.a scrub ok 2024-03-11T20:52:06.036 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:05 smithi023 bash[29146]: audit 2024-03-11T20:52:04.837958+0000 mgr.y (mgr.24380) 42 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:52:06.036 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:05 smithi023 bash[29146]: cluster 2024-03-11T20:52:04.900185+0000 osd.0 (osd.0) 39 : cluster 0 3.6 scrub starts 2024-03-11T20:52:06.036 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:05 smithi023 bash[29146]: cluster 2024-03-11T20:52:04.904710+0000 osd.0 (osd.0) 40 : cluster 0 3.6 scrub ok 2024-03-11T20:52:06.388 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:52:06.784 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:06 smithi023 bash[21372]: cluster 2024-03-11T20:52:05.434827+0000 osd.3 (osd.3) 35 : cluster 0 3.18 scrub starts 2024-03-11T20:52:06.784 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:06 smithi023 bash[21372]: cluster 2024-03-11T20:52:05.438647+0000 osd.3 (osd.3) 36 : cluster 0 3.18 scrub ok 2024-03-11T20:52:06.784 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:06 smithi023 bash[21372]: cluster 2024-03-11T20:52:05.509072+0000 osd.4 (osd.4) 35 : cluster 0 2.f scrub starts 2024-03-11T20:52:06.784 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:06 smithi023 bash[21372]: cluster 2024-03-11T20:52:05.511930+0000 osd.4 (osd.4) 36 : cluster 0 2.f scrub ok 2024-03-11T20:52:06.784 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:06 smithi023 bash[21372]: cluster 2024-03-11T20:52:05.823765+0000 mgr.y (mgr.24380) 43 : cluster 0 pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:52:06.784 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:06 smithi023 bash[21372]: cluster 2024-03-11T20:52:06.009716+0000 osd.7 (osd.7) 31 : cluster 0 2.1d scrub starts 2024-03-11T20:52:06.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:06 smithi023 bash[21372]: cluster 2024-03-11T20:52:06.012211+0000 osd.7 (osd.7) 32 : cluster 0 2.1d scrub ok 2024-03-11T20:52:06.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:06 smithi023 bash[29146]: cluster 2024-03-11T20:52:05.434827+0000 osd.3 (osd.3) 35 : cluster 0 3.18 scrub starts 2024-03-11T20:52:06.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:06 smithi023 bash[29146]: cluster 2024-03-11T20:52:05.438647+0000 osd.3 (osd.3) 36 : cluster 0 3.18 scrub ok 2024-03-11T20:52:06.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:06 smithi023 bash[29146]: cluster 2024-03-11T20:52:05.509072+0000 osd.4 (osd.4) 35 : cluster 0 2.f scrub starts 2024-03-11T20:52:06.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:06 smithi023 bash[29146]: cluster 2024-03-11T20:52:05.511930+0000 osd.4 (osd.4) 36 : cluster 0 2.f scrub ok 2024-03-11T20:52:06.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:06 smithi023 bash[29146]: cluster 2024-03-11T20:52:05.823765+0000 mgr.y (mgr.24380) 43 : cluster 0 pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:52:06.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:06 smithi023 bash[29146]: cluster 2024-03-11T20:52:06.009716+0000 osd.7 (osd.7) 31 : cluster 0 2.1d scrub starts 2024-03-11T20:52:06.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:06 smithi023 bash[29146]: cluster 2024-03-11T20:52:06.012211+0000 osd.7 (osd.7) 32 : cluster 0 2.1d scrub ok 2024-03-11T20:52:07.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:06 smithi146 bash[23809]: cluster 2024-03-11T20:52:05.434827+0000 osd.3 (osd.3) 35 : cluster 0 3.18 scrub starts 2024-03-11T20:52:07.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:06 smithi146 bash[23809]: cluster 2024-03-11T20:52:05.438647+0000 osd.3 (osd.3) 36 : cluster 0 3.18 scrub ok 2024-03-11T20:52:07.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:06 smithi146 bash[23809]: cluster 2024-03-11T20:52:05.509072+0000 osd.4 (osd.4) 35 : cluster 0 2.f scrub starts 2024-03-11T20:52:07.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:06 smithi146 bash[23809]: cluster 2024-03-11T20:52:05.511930+0000 osd.4 (osd.4) 36 : cluster 0 2.f scrub ok 2024-03-11T20:52:07.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:06 smithi146 bash[23809]: cluster 2024-03-11T20:52:05.823765+0000 mgr.y (mgr.24380) 43 : cluster 0 pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:52:07.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:06 smithi146 bash[23809]: cluster 2024-03-11T20:52:06.009716+0000 osd.7 (osd.7) 31 : cluster 0 2.1d scrub starts 2024-03-11T20:52:07.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:06 smithi146 bash[23809]: cluster 2024-03-11T20:52:06.012211+0000 osd.7 (osd.7) 32 : cluster 0 2.1d scrub ok 2024-03-11T20:52:08.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:07 smithi146 bash[23809]: cluster 2024-03-11T20:52:05.875236+0000 osd.5 (osd.5) 37 : cluster 0 4.8 scrub starts 2024-03-11T20:52:08.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:07 smithi146 bash[23809]: cluster 2024-03-11T20:52:05.877384+0000 osd.5 (osd.5) 38 : cluster 0 4.8 scrub ok 2024-03-11T20:52:08.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:07 smithi146 bash[23809]: cluster 2024-03-11T20:52:06.870257+0000 osd.0 (osd.0) 41 : cluster 0 4.3 scrub starts 2024-03-11T20:52:08.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:07 smithi146 bash[23809]: cluster 2024-03-11T20:52:06.873079+0000 osd.0 (osd.0) 42 : cluster 0 4.3 scrub ok 2024-03-11T20:52:08.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:07 smithi023 bash[21372]: cluster 2024-03-11T20:52:05.875236+0000 osd.5 (osd.5) 37 : cluster 0 4.8 scrub starts 2024-03-11T20:52:08.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:07 smithi023 bash[21372]: cluster 2024-03-11T20:52:05.877384+0000 osd.5 (osd.5) 38 : cluster 0 4.8 scrub ok 2024-03-11T20:52:08.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:07 smithi023 bash[21372]: cluster 2024-03-11T20:52:06.870257+0000 osd.0 (osd.0) 41 : cluster 0 4.3 scrub starts 2024-03-11T20:52:08.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:07 smithi023 bash[21372]: cluster 2024-03-11T20:52:06.873079+0000 osd.0 (osd.0) 42 : cluster 0 4.3 scrub ok 2024-03-11T20:52:08.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:07 smithi023 bash[29146]: cluster 2024-03-11T20:52:05.875236+0000 osd.5 (osd.5) 37 : cluster 0 4.8 scrub starts 2024-03-11T20:52:08.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:07 smithi023 bash[29146]: cluster 2024-03-11T20:52:05.877384+0000 osd.5 (osd.5) 38 : cluster 0 4.8 scrub ok 2024-03-11T20:52:08.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:07 smithi023 bash[29146]: cluster 2024-03-11T20:52:06.870257+0000 osd.0 (osd.0) 41 : cluster 0 4.3 scrub starts 2024-03-11T20:52:08.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:07 smithi023 bash[29146]: cluster 2024-03-11T20:52:06.873079+0000 osd.0 (osd.0) 42 : cluster 0 4.3 scrub ok 2024-03-11T20:52:08.927 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:08 smithi023 bash[21372]: cluster 2024-03-11T20:52:07.499325+0000 osd.4 (osd.4) 37 : cluster 0 6.1e scrub starts 2024-03-11T20:52:08.927 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:08 smithi023 bash[21372]: cluster 2024-03-11T20:52:07.500753+0000 osd.4 (osd.4) 38 : cluster 0 6.1e scrub ok 2024-03-11T20:52:08.928 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:08 smithi023 bash[21372]: cluster 2024-03-11T20:52:07.824470+0000 mgr.y (mgr.24380) 44 : cluster 0 pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 960 B/s rd, 0 op/s 2024-03-11T20:52:08.928 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:08 smithi023 bash[21372]: cluster 2024-03-11T20:52:07.833686+0000 osd.0 (osd.0) 43 : cluster 0 6.0 scrub starts 2024-03-11T20:52:08.928 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:08 smithi023 bash[21372]: cluster 2024-03-11T20:52:07.836146+0000 osd.0 (osd.0) 44 : cluster 0 6.0 scrub ok 2024-03-11T20:52:08.928 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:08 smithi023 bash[29146]: cluster 2024-03-11T20:52:07.499325+0000 osd.4 (osd.4) 37 : cluster 0 6.1e scrub starts 2024-03-11T20:52:08.928 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:08 smithi023 bash[29146]: cluster 2024-03-11T20:52:07.500753+0000 osd.4 (osd.4) 38 : cluster 0 6.1e scrub ok 2024-03-11T20:52:08.928 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:08 smithi023 bash[29146]: cluster 2024-03-11T20:52:07.824470+0000 mgr.y (mgr.24380) 44 : cluster 0 pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 960 B/s rd, 0 op/s 2024-03-11T20:52:08.928 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:08 smithi023 bash[29146]: cluster 2024-03-11T20:52:07.833686+0000 osd.0 (osd.0) 43 : cluster 0 6.0 scrub starts 2024-03-11T20:52:08.928 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:08 smithi023 bash[29146]: cluster 2024-03-11T20:52:07.836146+0000 osd.0 (osd.0) 44 : cluster 0 6.0 scrub ok 2024-03-11T20:52:09.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:08 smithi146 bash[23809]: cluster 2024-03-11T20:52:07.499325+0000 osd.4 (osd.4) 37 : cluster 0 6.1e scrub starts 2024-03-11T20:52:09.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:08 smithi146 bash[23809]: cluster 2024-03-11T20:52:07.500753+0000 osd.4 (osd.4) 38 : cluster 0 6.1e scrub ok 2024-03-11T20:52:09.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:08 smithi146 bash[23809]: cluster 2024-03-11T20:52:07.824470+0000 mgr.y (mgr.24380) 44 : cluster 0 pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 960 B/s rd, 0 op/s 2024-03-11T20:52:09.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:08 smithi146 bash[23809]: cluster 2024-03-11T20:52:07.833686+0000 osd.0 (osd.0) 43 : cluster 0 6.0 scrub starts 2024-03-11T20:52:09.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:08 smithi146 bash[23809]: cluster 2024-03-11T20:52:07.836146+0000 osd.0 (osd.0) 44 : cluster 0 6.0 scrub ok 2024-03-11T20:52:09.317 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-11T20:52:09.317 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":104,"fsid":"67ba00f8-dfe7-11ee-95c8-87774f69a715","created":"2024-03-11T20:40:13.664802+0000","modified":"2024-03-11T20:51:57.160397+0000","last_up_change":"2024-03-11T20:49:48.903029+0000","last_in_change":"2024-03-11T20:49:21.979501+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T20:45:31.964570+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-03-11T20:50:26.421486+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"default.rgw.log","create_time":"2024-03-11T20:50:28.592615+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.control","create_time":"2024-03-11T20:50:30.536371+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"datapool","create_time":"2024-03-11T20:50:31.135845+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":87,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-03-11T20:50:33.233163+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3df72b4a-8fe4-4265-b4c6-af022e1e23e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":2154627393},{"type":"v1","addr":"172.21.15.23:6803","nonce":2154627393}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":2154627393},{"type":"v1","addr":"172.21.15.23:6805","nonce":2154627393}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":2154627393},{"type":"v1","addr":"172.21.15.23:6809","nonce":2154627393}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":2154627393},{"type":"v1","addr":"172.21.15.23:6807","nonce":2154627393}]},"public_addr":"172.21.15.23:6803/2154627393","cluster_addr":"172.21.15.23:6805/2154627393","heartbeat_back_addr":"172.21.15.23:6809/2154627393","heartbeat_front_addr":"172.21.15.23:6807/2154627393","state":["exists","up"]},{"osd":1,"uuid":"e8fd7500-9081-41a1-8077-a4d0f0e12a2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":2506707705},{"type":"v1","addr":"172.21.15.23:6811","nonce":2506707705}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":2506707705},{"type":"v1","addr":"172.21.15.23:6813","nonce":2506707705}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":2506707705},{"type":"v1","addr":"172.21.15.23:6817","nonce":2506707705}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":2506707705},{"type":"v1","addr":"172.21.15.23:6815","nonce":2506707705}]},"public_addr":"172.21.15.23:6811/2506707705","cluster_addr":"172.21.15.23:6813/2506707705","heartbeat_back_addr":"172.21.15.23:6817/2506707705","heartbeat_front_addr":"172.21.15.23:6815/2506707705","state":["exists","up"]},{"osd":2,"uuid":"dd85d52a-df48-4b03-8612-cecfd82b5987","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":1770748056},{"type":"v1","addr":"172.21.15.23:6819","nonce":1770748056}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":1770748056},{"type":"v1","addr":"172.21.15.23:6821","nonce":1770748056}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":1770748056},{"type":"v1","addr":"172.21.15.23:6825","nonce":1770748056}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":1770748056},{"type":"v1","addr":"172.21.15.23:6823","nonce":1770748056}]},"public_addr":"172.21.15.23:6819/1770748056","cluster_addr":"172.21.15.23:6821/1770748056","heartbeat_back_addr":"172.21.15.23:6825/1770748056","heartbeat_front_addr":"172.21.15.23:6823/1770748056","state":["exists","up"]},{"osd":3,"uuid":"e3021106-3779-4b94-9bbf-a7724544110f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":28,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":938496814},{"type":"v1","addr":"172.21.15.23:6827","nonce":938496814}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":938496814},{"type":"v1","addr":"172.21.15.23:6829","nonce":938496814}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":938496814},{"type":"v1","addr":"172.21.15.23:6833","nonce":938496814}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":938496814},{"type":"v1","addr":"172.21.15.23:6831","nonce":938496814}]},"public_addr":"172.21.15.23:6827/938496814","cluster_addr":"172.21.15.23:6829/938496814","heartbeat_back_addr":"172.21.15.23:6833/938496814","heartbeat_front_addr":"172.21.15.23:6831/938496814","state":["exists","up"]},{"osd":4,"uuid":"4dcbc91e-7035-413d-97bc-c9d804eb16d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":615746847},{"type":"v1","addr":"172.21.15.146:6801","nonce":615746847}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":615746847},{"type":"v1","addr":"172.21.15.146:6803","nonce":615746847}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":615746847},{"type":"v1","addr":"172.21.15.146:6807","nonce":615746847}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":615746847},{"type":"v1","addr":"172.21.15.146:6805","nonce":615746847}]},"public_addr":"172.21.15.146:6801/615746847","cluster_addr":"172.21.15.146:6803/615746847","heartbeat_back_addr":"172.21.15.146:6807/615746847","heartbeat_front_addr":"172.21.15.146:6805/615746847","state":["exists","up"]},{"osd":5,"uuid":"aefcb766-8bae-4ff4-b71e-6691c8ac02f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":4003230814},{"type":"v1","addr":"172.21.15.146:6809","nonce":4003230814}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":4003230814},{"type":"v1","addr":"172.21.15.146:6811","nonce":4003230814}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":4003230814},{"type":"v1","addr":"172.21.15.146:6815","nonce":4003230814}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":4003230814},{"type":"v1","addr":"172.21.15.146:6813","nonce":4003230814}]},"public_addr":"172.21.15.146:6809/4003230814","cluster_addr":"172.21.15.146:6811/4003230814","heartbeat_back_addr":"172.21.15.146:6815/4003230814","heartbeat_front_addr":"172.21.15.146:6813/4003230814","state":["exists","up"]},{"osd":6,"uuid":"5cb151e8-1f9b-4617-8a86-adfb06bb9606","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":3334092376},{"type":"v1","addr":"172.21.15.146:6817","nonce":3334092376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3334092376},{"type":"v1","addr":"172.21.15.146:6819","nonce":3334092376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":3334092376},{"type":"v1","addr":"172.21.15.146:6823","nonce":3334092376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3334092376},{"type":"v1","addr":"172.21.15.146:6821","nonce":3334092376}]},"public_addr":"172.21.15.146:6817/3334092376","cluster_addr":"172.21.15.146:6819/3334092376","heartbeat_back_addr":"172.21.15.146:6823/3334092376","heartbeat_front_addr":"172.21.15.146:6821/3334092376","state":["exists","up"]},{"osd":7,"uuid":"bdbd11a2-fa09-4f82-93a2-6b2d9c750bfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":1678407923},{"type":"v1","addr":"172.21.15.146:6825","nonce":1678407923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":1678407923},{"type":"v1","addr":"172.21.15.146:6827","nonce":1678407923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":1678407923},{"type":"v1","addr":"172.21.15.146:6831","nonce":1678407923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":1678407923},{"type":"v1","addr":"172.21.15.146:6829","nonce":1678407923}]},"public_addr":"172.21.15.146:6825/1678407923","cluster_addr":"172.21.15.146:6827/1678407923","heartbeat_back_addr":"172.21.15.146:6831/1678407923","heartbeat_front_addr":"172.21.15.146:6829/1678407923","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T20:51:13.170176+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T20:51:52.930267+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T20:50:51.776614+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T20:51:33.925796+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T20:51:14.309213+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T20:51:23.706542+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T20:51:02.765608+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T20:51:56.471668+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":7,"to":2}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]},{"pgid":"4.f","mappings":[{"from":1,"to":2}]},{"pgid":"4.14","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:0/1618334186":"2024-03-12T20:40:45.361537+0000","172.21.15.23:6801/2813578081":"2024-03-12T20:40:45.361537+0000","172.21.15.23:0/2869940180":"2024-03-12T20:51:33.790205+0000","172.21.15.23:6800/3037564930":"2024-03-12T20:51:33.790205+0000","172.21.15.23:0/3297567084":"2024-03-12T20:51:33.790205+0000","172.21.15.23:0/3708753659":"2024-03-12T20:40:45.361537+0000","172.21.15.23:0/2004146740":"2024-03-12T20:40:45.361537+0000","172.21.15.23:0/827653812":"2024-03-12T20:41:21.301516+0000","172.21.15.23:6800/1057699850":"2024-03-12T20:41:21.301516+0000","172.21.15.23:6800/2813578081":"2024-03-12T20:40:45.361537+0000","172.21.15.23:6801/1057699850":"2024-03-12T20:41:21.301516+0000","172.21.15.23:0/3833230375":"2024-03-12T20:41:21.301516+0000","172.21.15.23:6801/3037564930":"2024-03-12T20:51:33.790205+0000","172.21.15.23:0/1182809102":"2024-03-12T20:51:33.790205+0000","172.21.15.23:0/2122134187":"2024-03-12T20:51:33.790205+0000","172.21.15.23:0/2356701899":"2024-03-12T20:41:21.301516+0000","172.21.15.23:0/843974100":"2024-03-12T20:51:33.790205+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T20:52:09.959 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:09 smithi023 bash[21372]: cluster 2024-03-11T20:52:09.042365+0000 osd.7 (osd.7) 33 : cluster 0 2.8 deep-scrub starts 2024-03-11T20:52:09.959 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:09 smithi023 bash[21372]: cluster 2024-03-11T20:52:09.043648+0000 osd.7 (osd.7) 34 : cluster 0 2.8 deep-scrub ok 2024-03-11T20:52:09.959 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:09 smithi023 bash[21372]: audit 2024-03-11T20:52:09.317225+0000 mon.c (mon.2) 21 : audit 0 from='client.? 172.21.15.23:0/1731214620' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T20:52:09.959 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:09 smithi023 bash[29146]: cluster 2024-03-11T20:52:09.042365+0000 osd.7 (osd.7) 33 : cluster 0 2.8 deep-scrub starts 2024-03-11T20:52:09.959 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:09 smithi023 bash[29146]: cluster 2024-03-11T20:52:09.043648+0000 osd.7 (osd.7) 34 : cluster 0 2.8 deep-scrub ok 2024-03-11T20:52:09.959 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:09 smithi023 bash[29146]: audit 2024-03-11T20:52:09.317225+0000 mon.c (mon.2) 21 : audit 0 from='client.? 172.21.15.23:0/1731214620' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T20:52:09.960 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-11T20:52:09.961 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd dump --format=json 2024-03-11T20:52:10.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:09 smithi146 bash[23809]: cluster 2024-03-11T20:52:09.042365+0000 osd.7 (osd.7) 33 : cluster 0 2.8 deep-scrub starts 2024-03-11T20:52:10.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:09 smithi146 bash[23809]: cluster 2024-03-11T20:52:09.043648+0000 osd.7 (osd.7) 34 : cluster 0 2.8 deep-scrub ok 2024-03-11T20:52:10.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:09 smithi146 bash[23809]: audit 2024-03-11T20:52:09.317225+0000 mon.c (mon.2) 21 : audit 0 from='client.? 172.21.15.23:0/1731214620' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T20:52:11.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:10 smithi146 bash[23809]: cluster 2024-03-11T20:52:09.167667+0000 osd.1 (osd.1) 35 : cluster 0 6.4 scrub starts 2024-03-11T20:52:11.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:10 smithi146 bash[23809]: cluster 2024-03-11T20:52:09.169718+0000 osd.1 (osd.1) 36 : cluster 0 6.4 scrub ok 2024-03-11T20:52:11.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:10 smithi146 bash[23809]: cluster 2024-03-11T20:52:09.575108+0000 osd.4 (osd.4) 39 : cluster 0 4.1 scrub starts 2024-03-11T20:52:11.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:10 smithi146 bash[23809]: cluster 2024-03-11T20:52:09.577093+0000 osd.4 (osd.4) 40 : cluster 0 4.1 scrub ok 2024-03-11T20:52:11.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:10 smithi146 bash[23809]: cluster 2024-03-11T20:52:09.825875+0000 mgr.y (mgr.24380) 45 : cluster 0 pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:52:11.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:10 smithi146 bash[23809]: cluster 2024-03-11T20:52:09.840497+0000 osd.5 (osd.5) 39 : cluster 0 4.15 scrub starts 2024-03-11T20:52:11.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:10 smithi146 bash[23809]: cluster 2024-03-11T20:52:09.843586+0000 osd.5 (osd.5) 40 : cluster 0 4.15 scrub ok 2024-03-11T20:52:11.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:10 smithi023 bash[21372]: cluster 2024-03-11T20:52:09.167667+0000 osd.1 (osd.1) 35 : cluster 0 6.4 scrub starts 2024-03-11T20:52:11.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:10 smithi023 bash[21372]: cluster 2024-03-11T20:52:09.169718+0000 osd.1 (osd.1) 36 : cluster 0 6.4 scrub ok 2024-03-11T20:52:11.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:10 smithi023 bash[21372]: cluster 2024-03-11T20:52:09.575108+0000 osd.4 (osd.4) 39 : cluster 0 4.1 scrub starts 2024-03-11T20:52:11.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:10 smithi023 bash[21372]: cluster 2024-03-11T20:52:09.577093+0000 osd.4 (osd.4) 40 : cluster 0 4.1 scrub ok 2024-03-11T20:52:11.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:10 smithi023 bash[21372]: cluster 2024-03-11T20:52:09.825875+0000 mgr.y (mgr.24380) 45 : cluster 0 pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:52:11.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:10 smithi023 bash[21372]: cluster 2024-03-11T20:52:09.840497+0000 osd.5 (osd.5) 39 : cluster 0 4.15 scrub starts 2024-03-11T20:52:11.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:10 smithi023 bash[21372]: cluster 2024-03-11T20:52:09.843586+0000 osd.5 (osd.5) 40 : cluster 0 4.15 scrub ok 2024-03-11T20:52:11.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:10 smithi023 bash[29146]: cluster 2024-03-11T20:52:09.167667+0000 osd.1 (osd.1) 35 : cluster 0 6.4 scrub starts 2024-03-11T20:52:11.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:10 smithi023 bash[29146]: cluster 2024-03-11T20:52:09.169718+0000 osd.1 (osd.1) 36 : cluster 0 6.4 scrub ok 2024-03-11T20:52:11.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:10 smithi023 bash[29146]: cluster 2024-03-11T20:52:09.575108+0000 osd.4 (osd.4) 39 : cluster 0 4.1 scrub starts 2024-03-11T20:52:11.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:10 smithi023 bash[29146]: cluster 2024-03-11T20:52:09.577093+0000 osd.4 (osd.4) 40 : cluster 0 4.1 scrub ok 2024-03-11T20:52:11.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:10 smithi023 bash[29146]: cluster 2024-03-11T20:52:09.825875+0000 mgr.y (mgr.24380) 45 : cluster 0 pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:52:11.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:10 smithi023 bash[29146]: cluster 2024-03-11T20:52:09.840497+0000 osd.5 (osd.5) 39 : cluster 0 4.15 scrub starts 2024-03-11T20:52:11.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:10 smithi023 bash[29146]: cluster 2024-03-11T20:52:09.843586+0000 osd.5 (osd.5) 40 : cluster 0 4.15 scrub ok 2024-03-11T20:52:12.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:11 smithi146 bash[23809]: cluster 2024-03-11T20:52:10.156831+0000 osd.1 (osd.1) 37 : cluster 0 4.2 deep-scrub starts 2024-03-11T20:52:12.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:11 smithi146 bash[23809]: cluster 2024-03-11T20:52:10.160197+0000 osd.1 (osd.1) 38 : cluster 0 4.2 deep-scrub ok 2024-03-11T20:52:12.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:11 smithi023 bash[21372]: cluster 2024-03-11T20:52:10.156831+0000 osd.1 (osd.1) 37 : cluster 0 4.2 deep-scrub starts 2024-03-11T20:52:12.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:11 smithi023 bash[21372]: cluster 2024-03-11T20:52:10.160197+0000 osd.1 (osd.1) 38 : cluster 0 4.2 deep-scrub ok 2024-03-11T20:52:12.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:11 smithi023 bash[29146]: cluster 2024-03-11T20:52:10.156831+0000 osd.1 (osd.1) 37 : cluster 0 4.2 deep-scrub starts 2024-03-11T20:52:12.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:11 smithi023 bash[29146]: cluster 2024-03-11T20:52:10.160197+0000 osd.1 (osd.1) 38 : cluster 0 4.2 deep-scrub ok 2024-03-11T20:52:13.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:12 smithi146 bash[23809]: cluster 2024-03-11T20:52:11.199309+0000 osd.1 (osd.1) 39 : cluster 0 2.6 scrub starts 2024-03-11T20:52:13.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:12 smithi146 bash[23809]: cluster 2024-03-11T20:52:11.202541+0000 osd.1 (osd.1) 40 : cluster 0 2.6 scrub ok 2024-03-11T20:52:13.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:12 smithi146 bash[23809]: cluster 2024-03-11T20:52:11.460239+0000 osd.3 (osd.3) 37 : cluster 0 6.13 deep-scrub starts 2024-03-11T20:52:13.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:12 smithi146 bash[23809]: cluster 2024-03-11T20:52:11.462470+0000 osd.3 (osd.3) 38 : cluster 0 6.13 deep-scrub ok 2024-03-11T20:52:13.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:12 smithi146 bash[23809]: cluster 2024-03-11T20:52:11.664492+0000 osd.6 (osd.6) 21 : cluster 0 3.10 scrub starts 2024-03-11T20:52:13.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:12 smithi146 bash[23809]: cluster 2024-03-11T20:52:11.667793+0000 osd.6 (osd.6) 22 : cluster 0 3.10 scrub ok 2024-03-11T20:52:13.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:12 smithi146 bash[23809]: cluster 2024-03-11T20:52:11.826602+0000 mgr.y (mgr.24380) 46 : cluster 0 pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T20:52:13.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:12 smithi146 bash[23809]: cluster 2024-03-11T20:52:11.895862+0000 osd.5 (osd.5) 41 : cluster 0 6.7 scrub starts 2024-03-11T20:52:13.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:12 smithi146 bash[23809]: cluster 2024-03-11T20:52:11.898559+0000 osd.5 (osd.5) 42 : cluster 0 6.7 scrub ok 2024-03-11T20:52:13.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:12 smithi146 bash[23809]: cluster 2024-03-11T20:52:12.016682+0000 osd.7 (osd.7) 35 : cluster 0 3.13 scrub starts 2024-03-11T20:52:13.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:12 smithi146 bash[23809]: cluster 2024-03-11T20:52:12.020459+0000 osd.7 (osd.7) 36 : cluster 0 3.13 scrub ok 2024-03-11T20:52:13.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:12 smithi146 bash[23809]: cluster 2024-03-11T20:52:12.652205+0000 osd.6 (osd.6) 23 : cluster 0 4.1f scrub starts 2024-03-11T20:52:13.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:12 smithi146 bash[23809]: cluster 2024-03-11T20:52:12.655053+0000 osd.6 (osd.6) 24 : cluster 0 4.1f scrub ok 2024-03-11T20:52:13.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:12 smithi023 bash[21372]: cluster 2024-03-11T20:52:11.199309+0000 osd.1 (osd.1) 39 : cluster 0 2.6 scrub starts 2024-03-11T20:52:13.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:12 smithi023 bash[21372]: cluster 2024-03-11T20:52:11.202541+0000 osd.1 (osd.1) 40 : cluster 0 2.6 scrub ok 2024-03-11T20:52:13.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:12 smithi023 bash[21372]: cluster 2024-03-11T20:52:11.460239+0000 osd.3 (osd.3) 37 : cluster 0 6.13 deep-scrub starts 2024-03-11T20:52:13.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:12 smithi023 bash[21372]: cluster 2024-03-11T20:52:11.462470+0000 osd.3 (osd.3) 38 : cluster 0 6.13 deep-scrub ok 2024-03-11T20:52:13.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:12 smithi023 bash[21372]: cluster 2024-03-11T20:52:11.664492+0000 osd.6 (osd.6) 21 : cluster 0 3.10 scrub starts 2024-03-11T20:52:13.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:12 smithi023 bash[21372]: cluster 2024-03-11T20:52:11.667793+0000 osd.6 (osd.6) 22 : cluster 0 3.10 scrub ok 2024-03-11T20:52:13.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:12 smithi023 bash[21372]: cluster 2024-03-11T20:52:11.826602+0000 mgr.y (mgr.24380) 46 : cluster 0 pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T20:52:13.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:12 smithi023 bash[21372]: cluster 2024-03-11T20:52:11.895862+0000 osd.5 (osd.5) 41 : cluster 0 6.7 scrub starts 2024-03-11T20:52:13.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:12 smithi023 bash[21372]: cluster 2024-03-11T20:52:11.898559+0000 osd.5 (osd.5) 42 : cluster 0 6.7 scrub ok 2024-03-11T20:52:13.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:12 smithi023 bash[21372]: cluster 2024-03-11T20:52:12.016682+0000 osd.7 (osd.7) 35 : cluster 0 3.13 scrub starts 2024-03-11T20:52:13.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:12 smithi023 bash[21372]: cluster 2024-03-11T20:52:12.020459+0000 osd.7 (osd.7) 36 : cluster 0 3.13 scrub ok 2024-03-11T20:52:13.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:12 smithi023 bash[21372]: cluster 2024-03-11T20:52:12.652205+0000 osd.6 (osd.6) 23 : cluster 0 4.1f scrub starts 2024-03-11T20:52:13.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:12 smithi023 bash[21372]: cluster 2024-03-11T20:52:12.655053+0000 osd.6 (osd.6) 24 : cluster 0 4.1f scrub ok 2024-03-11T20:52:13.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:12 smithi023 bash[29146]: cluster 2024-03-11T20:52:11.199309+0000 osd.1 (osd.1) 39 : cluster 0 2.6 scrub starts 2024-03-11T20:52:13.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:12 smithi023 bash[29146]: cluster 2024-03-11T20:52:11.202541+0000 osd.1 (osd.1) 40 : cluster 0 2.6 scrub ok 2024-03-11T20:52:13.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:12 smithi023 bash[29146]: cluster 2024-03-11T20:52:11.460239+0000 osd.3 (osd.3) 37 : cluster 0 6.13 deep-scrub starts 2024-03-11T20:52:13.036 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:12 smithi023 bash[29146]: cluster 2024-03-11T20:52:11.462470+0000 osd.3 (osd.3) 38 : cluster 0 6.13 deep-scrub ok 2024-03-11T20:52:13.036 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:12 smithi023 bash[29146]: cluster 2024-03-11T20:52:11.664492+0000 osd.6 (osd.6) 21 : cluster 0 3.10 scrub starts 2024-03-11T20:52:13.036 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:12 smithi023 bash[29146]: cluster 2024-03-11T20:52:11.667793+0000 osd.6 (osd.6) 22 : cluster 0 3.10 scrub ok 2024-03-11T20:52:13.036 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:12 smithi023 bash[29146]: cluster 2024-03-11T20:52:11.826602+0000 mgr.y (mgr.24380) 46 : cluster 0 pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T20:52:13.036 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:12 smithi023 bash[29146]: cluster 2024-03-11T20:52:11.895862+0000 osd.5 (osd.5) 41 : cluster 0 6.7 scrub starts 2024-03-11T20:52:13.036 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:12 smithi023 bash[29146]: cluster 2024-03-11T20:52:11.898559+0000 osd.5 (osd.5) 42 : cluster 0 6.7 scrub ok 2024-03-11T20:52:13.036 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:12 smithi023 bash[29146]: cluster 2024-03-11T20:52:12.016682+0000 osd.7 (osd.7) 35 : cluster 0 3.13 scrub starts 2024-03-11T20:52:13.036 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:12 smithi023 bash[29146]: cluster 2024-03-11T20:52:12.020459+0000 osd.7 (osd.7) 36 : cluster 0 3.13 scrub ok 2024-03-11T20:52:13.036 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:12 smithi023 bash[29146]: cluster 2024-03-11T20:52:12.652205+0000 osd.6 (osd.6) 23 : cluster 0 4.1f scrub starts 2024-03-11T20:52:13.036 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:12 smithi023 bash[29146]: cluster 2024-03-11T20:52:12.655053+0000 osd.6 (osd.6) 24 : cluster 0 4.1f scrub ok 2024-03-11T20:52:14.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:13 smithi146 bash[23809]: cluster 2024-03-11T20:52:12.511858+0000 osd.4 (osd.4) 41 : cluster 0 6.e scrub starts 2024-03-11T20:52:14.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:13 smithi146 bash[23809]: cluster 2024-03-11T20:52:12.513975+0000 osd.4 (osd.4) 42 : cluster 0 6.e scrub ok 2024-03-11T20:52:14.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:13 smithi146 bash[23809]: cluster 2024-03-11T20:52:12.813891+0000 osd.0 (osd.0) 45 : cluster 0 2.1f scrub starts 2024-03-11T20:52:14.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:13 smithi146 bash[23809]: cluster 2024-03-11T20:52:12.816094+0000 osd.0 (osd.0) 46 : cluster 0 2.1f scrub ok 2024-03-11T20:52:14.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:13 smithi023 bash[21372]: cluster 2024-03-11T20:52:12.511858+0000 osd.4 (osd.4) 41 : cluster 0 6.e scrub starts 2024-03-11T20:52:14.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:13 smithi023 bash[21372]: cluster 2024-03-11T20:52:12.513975+0000 osd.4 (osd.4) 42 : cluster 0 6.e scrub ok 2024-03-11T20:52:14.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:13 smithi023 bash[21372]: cluster 2024-03-11T20:52:12.813891+0000 osd.0 (osd.0) 45 : cluster 0 2.1f scrub starts 2024-03-11T20:52:14.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:13 smithi023 bash[21372]: cluster 2024-03-11T20:52:12.816094+0000 osd.0 (osd.0) 46 : cluster 0 2.1f scrub ok 2024-03-11T20:52:14.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:13 smithi023 bash[29146]: cluster 2024-03-11T20:52:12.511858+0000 osd.4 (osd.4) 41 : cluster 0 6.e scrub starts 2024-03-11T20:52:14.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:13 smithi023 bash[29146]: cluster 2024-03-11T20:52:12.513975+0000 osd.4 (osd.4) 42 : cluster 0 6.e scrub ok 2024-03-11T20:52:14.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:13 smithi023 bash[29146]: cluster 2024-03-11T20:52:12.813891+0000 osd.0 (osd.0) 45 : cluster 0 2.1f scrub starts 2024-03-11T20:52:14.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:13 smithi023 bash[29146]: cluster 2024-03-11T20:52:12.816094+0000 osd.0 (osd.0) 46 : cluster 0 2.1f scrub ok 2024-03-11T20:52:14.753 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:52:15.010 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 20:52:14 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T20:52:15.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:14 smithi146 bash[23809]: cluster 2024-03-11T20:52:13.827258+0000 mgr.y (mgr.24380) 47 : cluster 0 pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T20:52:15.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:14 smithi146 bash[23809]: cluster 2024-03-11T20:52:14.610787+0000 osd.6 (osd.6) 25 : cluster 0 2.7 scrub starts 2024-03-11T20:52:15.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:14 smithi146 bash[23809]: cluster 2024-03-11T20:52:14.613697+0000 osd.6 (osd.6) 26 : cluster 0 2.7 scrub ok 2024-03-11T20:52:15.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:14 smithi023 bash[21372]: cluster 2024-03-11T20:52:13.827258+0000 mgr.y (mgr.24380) 47 : cluster 0 pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T20:52:15.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:14 smithi023 bash[21372]: cluster 2024-03-11T20:52:14.610787+0000 osd.6 (osd.6) 25 : cluster 0 2.7 scrub starts 2024-03-11T20:52:15.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:14 smithi023 bash[21372]: cluster 2024-03-11T20:52:14.613697+0000 osd.6 (osd.6) 26 : cluster 0 2.7 scrub ok 2024-03-11T20:52:15.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:14 smithi023 bash[29146]: cluster 2024-03-11T20:52:13.827258+0000 mgr.y (mgr.24380) 47 : cluster 0 pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T20:52:15.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:14 smithi023 bash[29146]: cluster 2024-03-11T20:52:14.610787+0000 osd.6 (osd.6) 25 : cluster 0 2.7 scrub starts 2024-03-11T20:52:15.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:14 smithi023 bash[29146]: cluster 2024-03-11T20:52:14.613697+0000 osd.6 (osd.6) 26 : cluster 0 2.7 scrub ok 2024-03-11T20:52:15.916 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:15 smithi023 bash[21372]: cluster 2024-03-11T20:52:14.173053+0000 osd.1 (osd.1) 41 : cluster 0 2.4 scrub starts 2024-03-11T20:52:15.916 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:15 smithi023 bash[21372]: cluster 2024-03-11T20:52:14.175314+0000 osd.1 (osd.1) 42 : cluster 0 2.4 scrub ok 2024-03-11T20:52:15.916 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:15 smithi023 bash[21372]: cluster 2024-03-11T20:52:14.484895+0000 osd.2 (osd.2) 29 : cluster 0 purged_snaps scrub starts 2024-03-11T20:52:15.916 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:15 smithi023 bash[21372]: cluster 2024-03-11T20:52:14.485009+0000 osd.2 (osd.2) 30 : cluster 0 purged_snaps scrub ok 2024-03-11T20:52:15.916 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:15 smithi023 bash[21372]: cluster 2024-03-11T20:52:14.739376+0000 mon.a (mon.0) 857 : cluster 0 osdmap e105: 8 total, 8 up, 8 in 2024-03-11T20:52:15.916 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:15 smithi023 bash[21372]: audit 2024-03-11T20:52:14.849168+0000 mgr.y (mgr.24380) 48 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:52:15.916 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:15 smithi023 bash[29146]: cluster 2024-03-11T20:52:14.173053+0000 osd.1 (osd.1) 41 : cluster 0 2.4 scrub starts 2024-03-11T20:52:15.917 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:15 smithi023 bash[29146]: cluster 2024-03-11T20:52:14.175314+0000 osd.1 (osd.1) 42 : cluster 0 2.4 scrub ok 2024-03-11T20:52:15.917 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:15 smithi023 bash[29146]: cluster 2024-03-11T20:52:14.484895+0000 osd.2 (osd.2) 29 : cluster 0 purged_snaps scrub starts 2024-03-11T20:52:15.917 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:15 smithi023 bash[29146]: cluster 2024-03-11T20:52:14.485009+0000 osd.2 (osd.2) 30 : cluster 0 purged_snaps scrub ok 2024-03-11T20:52:15.917 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:15 smithi023 bash[29146]: cluster 2024-03-11T20:52:14.739376+0000 mon.a (mon.0) 857 : cluster 0 osdmap e105: 8 total, 8 up, 8 in 2024-03-11T20:52:15.917 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:15 smithi023 bash[29146]: audit 2024-03-11T20:52:14.849168+0000 mgr.y (mgr.24380) 48 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:52:16.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:15 smithi146 bash[23809]: cluster 2024-03-11T20:52:14.173053+0000 osd.1 (osd.1) 41 : cluster 0 2.4 scrub starts 2024-03-11T20:52:16.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:15 smithi146 bash[23809]: cluster 2024-03-11T20:52:14.175314+0000 osd.1 (osd.1) 42 : cluster 0 2.4 scrub ok 2024-03-11T20:52:16.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:15 smithi146 bash[23809]: cluster 2024-03-11T20:52:14.484895+0000 osd.2 (osd.2) 29 : cluster 0 purged_snaps scrub starts 2024-03-11T20:52:16.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:15 smithi146 bash[23809]: cluster 2024-03-11T20:52:14.485009+0000 osd.2 (osd.2) 30 : cluster 0 purged_snaps scrub ok 2024-03-11T20:52:16.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:15 smithi146 bash[23809]: cluster 2024-03-11T20:52:14.739376+0000 mon.a (mon.0) 857 : cluster 0 osdmap e105: 8 total, 8 up, 8 in 2024-03-11T20:52:16.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:15 smithi146 bash[23809]: audit 2024-03-11T20:52:14.849168+0000 mgr.y (mgr.24380) 48 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:52:17.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:16 smithi146 bash[23809]: cluster 2024-03-11T20:52:15.497375+0000 osd.4 (osd.4) 43 : cluster 0 4.e scrub starts 2024-03-11T20:52:17.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:16 smithi146 bash[23809]: cluster 2024-03-11T20:52:15.500557+0000 osd.4 (osd.4) 44 : cluster 0 4.e scrub ok 2024-03-11T20:52:17.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:16 smithi146 bash[23809]: cluster 2024-03-11T20:52:15.508799+0000 osd.3 (osd.3) 39 : cluster 0 6.6 scrub starts 2024-03-11T20:52:17.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:16 smithi146 bash[23809]: cluster 2024-03-11T20:52:15.511418+0000 osd.3 (osd.3) 40 : cluster 0 6.6 scrub ok 2024-03-11T20:52:17.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:16 smithi146 bash[23809]: cluster 2024-03-11T20:52:15.828741+0000 mgr.y (mgr.24380) 49 : cluster 0 pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:52:17.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:16 smithi146 bash[23809]: cluster 2024-03-11T20:52:15.864746+0000 osd.0 (osd.0) 47 : cluster 0 3.1f scrub starts 2024-03-11T20:52:17.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:16 smithi146 bash[23809]: cluster 2024-03-11T20:52:15.868589+0000 osd.0 (osd.0) 48 : cluster 0 3.1f scrub ok 2024-03-11T20:52:17.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:16 smithi023 bash[21372]: cluster 2024-03-11T20:52:15.497375+0000 osd.4 (osd.4) 43 : cluster 0 4.e scrub starts 2024-03-11T20:52:17.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:16 smithi023 bash[21372]: cluster 2024-03-11T20:52:15.500557+0000 osd.4 (osd.4) 44 : cluster 0 4.e scrub ok 2024-03-11T20:52:17.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:16 smithi023 bash[21372]: cluster 2024-03-11T20:52:15.508799+0000 osd.3 (osd.3) 39 : cluster 0 6.6 scrub starts 2024-03-11T20:52:17.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:16 smithi023 bash[21372]: cluster 2024-03-11T20:52:15.511418+0000 osd.3 (osd.3) 40 : cluster 0 6.6 scrub ok 2024-03-11T20:52:17.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:16 smithi023 bash[21372]: cluster 2024-03-11T20:52:15.828741+0000 mgr.y (mgr.24380) 49 : cluster 0 pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:52:17.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:16 smithi023 bash[21372]: cluster 2024-03-11T20:52:15.864746+0000 osd.0 (osd.0) 47 : cluster 0 3.1f scrub starts 2024-03-11T20:52:17.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:16 smithi023 bash[21372]: cluster 2024-03-11T20:52:15.868589+0000 osd.0 (osd.0) 48 : cluster 0 3.1f scrub ok 2024-03-11T20:52:17.036 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:16 smithi023 bash[29146]: cluster 2024-03-11T20:52:15.497375+0000 osd.4 (osd.4) 43 : cluster 0 4.e scrub starts 2024-03-11T20:52:17.036 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:16 smithi023 bash[29146]: cluster 2024-03-11T20:52:15.500557+0000 osd.4 (osd.4) 44 : cluster 0 4.e scrub ok 2024-03-11T20:52:17.036 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:16 smithi023 bash[29146]: cluster 2024-03-11T20:52:15.508799+0000 osd.3 (osd.3) 39 : cluster 0 6.6 scrub starts 2024-03-11T20:52:17.036 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:16 smithi023 bash[29146]: cluster 2024-03-11T20:52:15.511418+0000 osd.3 (osd.3) 40 : cluster 0 6.6 scrub ok 2024-03-11T20:52:17.036 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:16 smithi023 bash[29146]: cluster 2024-03-11T20:52:15.828741+0000 mgr.y (mgr.24380) 49 : cluster 0 pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:52:17.036 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:16 smithi023 bash[29146]: cluster 2024-03-11T20:52:15.864746+0000 osd.0 (osd.0) 47 : cluster 0 3.1f scrub starts 2024-03-11T20:52:17.036 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:16 smithi023 bash[29146]: cluster 2024-03-11T20:52:15.868589+0000 osd.0 (osd.0) 48 : cluster 0 3.1f scrub ok 2024-03-11T20:52:17.494 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-11T20:52:17.494 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":105,"fsid":"67ba00f8-dfe7-11ee-95c8-87774f69a715","created":"2024-03-11T20:40:13.664802+0000","modified":"2024-03-11T20:52:14.712622+0000","last_up_change":"2024-03-11T20:49:48.903029+0000","last_in_change":"2024-03-11T20:49:21.979501+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T20:45:31.964570+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-03-11T20:50:26.421486+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"default.rgw.log","create_time":"2024-03-11T20:50:28.592615+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.control","create_time":"2024-03-11T20:50:30.536371+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"datapool","create_time":"2024-03-11T20:50:31.135845+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":87,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-03-11T20:50:33.233163+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3df72b4a-8fe4-4265-b4c6-af022e1e23e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":2154627393},{"type":"v1","addr":"172.21.15.23:6803","nonce":2154627393}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":2154627393},{"type":"v1","addr":"172.21.15.23:6805","nonce":2154627393}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":2154627393},{"type":"v1","addr":"172.21.15.23:6809","nonce":2154627393}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":2154627393},{"type":"v1","addr":"172.21.15.23:6807","nonce":2154627393}]},"public_addr":"172.21.15.23:6803/2154627393","cluster_addr":"172.21.15.23:6805/2154627393","heartbeat_back_addr":"172.21.15.23:6809/2154627393","heartbeat_front_addr":"172.21.15.23:6807/2154627393","state":["exists","up"]},{"osd":1,"uuid":"e8fd7500-9081-41a1-8077-a4d0f0e12a2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":2506707705},{"type":"v1","addr":"172.21.15.23:6811","nonce":2506707705}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":2506707705},{"type":"v1","addr":"172.21.15.23:6813","nonce":2506707705}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":2506707705},{"type":"v1","addr":"172.21.15.23:6817","nonce":2506707705}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":2506707705},{"type":"v1","addr":"172.21.15.23:6815","nonce":2506707705}]},"public_addr":"172.21.15.23:6811/2506707705","cluster_addr":"172.21.15.23:6813/2506707705","heartbeat_back_addr":"172.21.15.23:6817/2506707705","heartbeat_front_addr":"172.21.15.23:6815/2506707705","state":["exists","up"]},{"osd":2,"uuid":"dd85d52a-df48-4b03-8612-cecfd82b5987","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":1770748056},{"type":"v1","addr":"172.21.15.23:6819","nonce":1770748056}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":1770748056},{"type":"v1","addr":"172.21.15.23:6821","nonce":1770748056}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":1770748056},{"type":"v1","addr":"172.21.15.23:6825","nonce":1770748056}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":1770748056},{"type":"v1","addr":"172.21.15.23:6823","nonce":1770748056}]},"public_addr":"172.21.15.23:6819/1770748056","cluster_addr":"172.21.15.23:6821/1770748056","heartbeat_back_addr":"172.21.15.23:6825/1770748056","heartbeat_front_addr":"172.21.15.23:6823/1770748056","state":["exists","up"]},{"osd":3,"uuid":"e3021106-3779-4b94-9bbf-a7724544110f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":28,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":938496814},{"type":"v1","addr":"172.21.15.23:6827","nonce":938496814}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":938496814},{"type":"v1","addr":"172.21.15.23:6829","nonce":938496814}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":938496814},{"type":"v1","addr":"172.21.15.23:6833","nonce":938496814}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":938496814},{"type":"v1","addr":"172.21.15.23:6831","nonce":938496814}]},"public_addr":"172.21.15.23:6827/938496814","cluster_addr":"172.21.15.23:6829/938496814","heartbeat_back_addr":"172.21.15.23:6833/938496814","heartbeat_front_addr":"172.21.15.23:6831/938496814","state":["exists","up"]},{"osd":4,"uuid":"4dcbc91e-7035-413d-97bc-c9d804eb16d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":615746847},{"type":"v1","addr":"172.21.15.146:6801","nonce":615746847}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":615746847},{"type":"v1","addr":"172.21.15.146:6803","nonce":615746847}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":615746847},{"type":"v1","addr":"172.21.15.146:6807","nonce":615746847}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":615746847},{"type":"v1","addr":"172.21.15.146:6805","nonce":615746847}]},"public_addr":"172.21.15.146:6801/615746847","cluster_addr":"172.21.15.146:6803/615746847","heartbeat_back_addr":"172.21.15.146:6807/615746847","heartbeat_front_addr":"172.21.15.146:6805/615746847","state":["exists","up"]},{"osd":5,"uuid":"aefcb766-8bae-4ff4-b71e-6691c8ac02f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":4003230814},{"type":"v1","addr":"172.21.15.146:6809","nonce":4003230814}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":4003230814},{"type":"v1","addr":"172.21.15.146:6811","nonce":4003230814}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":4003230814},{"type":"v1","addr":"172.21.15.146:6815","nonce":4003230814}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":4003230814},{"type":"v1","addr":"172.21.15.146:6813","nonce":4003230814}]},"public_addr":"172.21.15.146:6809/4003230814","cluster_addr":"172.21.15.146:6811/4003230814","heartbeat_back_addr":"172.21.15.146:6815/4003230814","heartbeat_front_addr":"172.21.15.146:6813/4003230814","state":["exists","up"]},{"osd":6,"uuid":"5cb151e8-1f9b-4617-8a86-adfb06bb9606","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":3334092376},{"type":"v1","addr":"172.21.15.146:6817","nonce":3334092376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3334092376},{"type":"v1","addr":"172.21.15.146:6819","nonce":3334092376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":3334092376},{"type":"v1","addr":"172.21.15.146:6823","nonce":3334092376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3334092376},{"type":"v1","addr":"172.21.15.146:6821","nonce":3334092376}]},"public_addr":"172.21.15.146:6817/3334092376","cluster_addr":"172.21.15.146:6819/3334092376","heartbeat_back_addr":"172.21.15.146:6823/3334092376","heartbeat_front_addr":"172.21.15.146:6821/3334092376","state":["exists","up"]},{"osd":7,"uuid":"bdbd11a2-fa09-4f82-93a2-6b2d9c750bfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":1678407923},{"type":"v1","addr":"172.21.15.146:6825","nonce":1678407923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":1678407923},{"type":"v1","addr":"172.21.15.146:6827","nonce":1678407923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":1678407923},{"type":"v1","addr":"172.21.15.146:6831","nonce":1678407923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":1678407923},{"type":"v1","addr":"172.21.15.146:6829","nonce":1678407923}]},"public_addr":"172.21.15.146:6825/1678407923","cluster_addr":"172.21.15.146:6827/1678407923","heartbeat_back_addr":"172.21.15.146:6831/1678407923","heartbeat_front_addr":"172.21.15.146:6829/1678407923","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T20:51:13.170176+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T20:51:52.930267+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T20:52:14.485014+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T20:51:33.925796+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T20:51:14.309213+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T20:51:23.706542+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T20:51:02.765608+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T20:51:56.471668+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":7,"to":2}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]},{"pgid":"4.f","mappings":[{"from":1,"to":2}]},{"pgid":"4.14","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:0/1618334186":"2024-03-12T20:40:45.361537+0000","172.21.15.23:6801/2813578081":"2024-03-12T20:40:45.361537+0000","172.21.15.23:0/2869940180":"2024-03-12T20:51:33.790205+0000","172.21.15.23:6800/3037564930":"2024-03-12T20:51:33.790205+0000","172.21.15.23:0/3297567084":"2024-03-12T20:51:33.790205+0000","172.21.15.23:0/3708753659":"2024-03-12T20:40:45.361537+0000","172.21.15.23:0/2004146740":"2024-03-12T20:40:45.361537+0000","172.21.15.23:0/827653812":"2024-03-12T20:41:21.301516+0000","172.21.15.23:6800/1057699850":"2024-03-12T20:41:21.301516+0000","172.21.15.23:6800/2813578081":"2024-03-12T20:40:45.361537+0000","172.21.15.23:6801/1057699850":"2024-03-12T20:41:21.301516+0000","172.21.15.23:0/3833230375":"2024-03-12T20:41:21.301516+0000","172.21.15.23:6801/3037564930":"2024-03-12T20:51:33.790205+0000","172.21.15.23:0/1182809102":"2024-03-12T20:51:33.790205+0000","172.21.15.23:0/2122134187":"2024-03-12T20:51:33.790205+0000","172.21.15.23:0/2356701899":"2024-03-12T20:41:21.301516+0000","172.21.15.23:0/843974100":"2024-03-12T20:51:33.790205+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T20:52:18.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:17 smithi146 bash[23809]: cluster 2024-03-11T20:52:16.093919+0000 osd.1 (osd.1) 43 : cluster 0 4.4 scrub starts 2024-03-11T20:52:18.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:17 smithi146 bash[23809]: cluster 2024-03-11T20:52:16.096249+0000 osd.1 (osd.1) 44 : cluster 0 4.4 scrub ok 2024-03-11T20:52:18.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:17 smithi146 bash[23809]: cluster 2024-03-11T20:52:16.844592+0000 osd.0 (osd.0) 49 : cluster 0 6.16 scrub starts 2024-03-11T20:52:18.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:17 smithi146 bash[23809]: cluster 2024-03-11T20:52:16.846232+0000 osd.0 (osd.0) 50 : cluster 0 6.16 scrub ok 2024-03-11T20:52:18.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:17 smithi146 bash[23809]: audit 2024-03-11T20:52:17.495962+0000 mon.c (mon.2) 22 : audit 0 from='client.? 172.21.15.23:0/564074523' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T20:52:18.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:17 smithi023 bash[29146]: cluster 2024-03-11T20:52:16.093919+0000 osd.1 (osd.1) 43 : cluster 0 4.4 scrub starts 2024-03-11T20:52:18.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:17 smithi023 bash[29146]: cluster 2024-03-11T20:52:16.096249+0000 osd.1 (osd.1) 44 : cluster 0 4.4 scrub ok 2024-03-11T20:52:18.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:17 smithi023 bash[29146]: cluster 2024-03-11T20:52:16.844592+0000 osd.0 (osd.0) 49 : cluster 0 6.16 scrub starts 2024-03-11T20:52:18.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:17 smithi023 bash[29146]: cluster 2024-03-11T20:52:16.846232+0000 osd.0 (osd.0) 50 : cluster 0 6.16 scrub ok 2024-03-11T20:52:18.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:17 smithi023 bash[29146]: audit 2024-03-11T20:52:17.495962+0000 mon.c (mon.2) 22 : audit 0 from='client.? 172.21.15.23:0/564074523' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T20:52:18.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:17 smithi023 bash[21372]: cluster 2024-03-11T20:52:16.093919+0000 osd.1 (osd.1) 43 : cluster 0 4.4 scrub starts 2024-03-11T20:52:18.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:17 smithi023 bash[21372]: cluster 2024-03-11T20:52:16.096249+0000 osd.1 (osd.1) 44 : cluster 0 4.4 scrub ok 2024-03-11T20:52:18.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:17 smithi023 bash[21372]: cluster 2024-03-11T20:52:16.844592+0000 osd.0 (osd.0) 49 : cluster 0 6.16 scrub starts 2024-03-11T20:52:18.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:17 smithi023 bash[21372]: cluster 2024-03-11T20:52:16.846232+0000 osd.0 (osd.0) 50 : cluster 0 6.16 scrub ok 2024-03-11T20:52:18.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:17 smithi023 bash[21372]: audit 2024-03-11T20:52:17.495962+0000 mon.c (mon.2) 22 : audit 0 from='client.? 172.21.15.23:0/564074523' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T20:52:18.127 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-11T20:52:18.128 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-11T20:52:18.128 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-11T20:52:18.129 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-11T20:52:18.129 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-11T20:52:18.129 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-11T20:52:18.130 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-03-11T20:52:18.130 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-03-11T20:52:19.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:18 smithi023 bash[21372]: cluster 2024-03-11T20:52:17.829455+0000 mgr.y (mgr.24380) 50 : cluster 0 pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:52:19.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:18 smithi023 bash[21372]: cluster 2024-03-11T20:52:17.881885+0000 osd.6 (osd.6) 27 : cluster 0 purged_snaps scrub starts 2024-03-11T20:52:19.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:18 smithi023 bash[21372]: cluster 2024-03-11T20:52:17.882034+0000 osd.6 (osd.6) 28 : cluster 0 purged_snaps scrub ok 2024-03-11T20:52:19.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:18 smithi023 bash[29146]: cluster 2024-03-11T20:52:17.829455+0000 mgr.y (mgr.24380) 50 : cluster 0 pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:52:19.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:18 smithi023 bash[29146]: cluster 2024-03-11T20:52:17.881885+0000 osd.6 (osd.6) 27 : cluster 0 purged_snaps scrub starts 2024-03-11T20:52:19.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:18 smithi023 bash[29146]: cluster 2024-03-11T20:52:17.882034+0000 osd.6 (osd.6) 28 : cluster 0 purged_snaps scrub ok 2024-03-11T20:52:19.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:18 smithi146 bash[23809]: cluster 2024-03-11T20:52:17.829455+0000 mgr.y (mgr.24380) 50 : cluster 0 pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:52:19.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:18 smithi146 bash[23809]: cluster 2024-03-11T20:52:17.881885+0000 osd.6 (osd.6) 27 : cluster 0 purged_snaps scrub starts 2024-03-11T20:52:19.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:18 smithi146 bash[23809]: cluster 2024-03-11T20:52:17.882034+0000 osd.6 (osd.6) 28 : cluster 0 purged_snaps scrub ok 2024-03-11T20:52:20.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:19 smithi146 bash[23809]: cluster 2024-03-11T20:52:18.093712+0000 osd.7 (osd.7) 37 : cluster 0 6.1c deep-scrub starts 2024-03-11T20:52:20.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:19 smithi146 bash[23809]: cluster 2024-03-11T20:52:18.098899+0000 osd.7 (osd.7) 38 : cluster 0 6.1c deep-scrub ok 2024-03-11T20:52:20.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:19 smithi146 bash[23809]: cluster 2024-03-11T20:52:18.790605+0000 mon.a (mon.0) 858 : cluster 0 osdmap e106: 8 total, 8 up, 8 in 2024-03-11T20:52:20.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:19 smithi146 bash[23809]: audit 2024-03-11T20:52:19.013959+0000 mon.a (mon.0) 859 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:52:20.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:19 smithi146 bash[23809]: cluster 2024-03-11T20:52:19.574155+0000 osd.6 (osd.6) 29 : cluster 0 4.a scrub starts 2024-03-11T20:52:20.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:19 smithi146 bash[23809]: cluster 2024-03-11T20:52:19.576409+0000 osd.6 (osd.6) 30 : cluster 0 4.a scrub ok 2024-03-11T20:52:20.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:19 smithi023 bash[21372]: cluster 2024-03-11T20:52:18.093712+0000 osd.7 (osd.7) 37 : cluster 0 6.1c deep-scrub starts 2024-03-11T20:52:20.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:19 smithi023 bash[21372]: cluster 2024-03-11T20:52:18.098899+0000 osd.7 (osd.7) 38 : cluster 0 6.1c deep-scrub ok 2024-03-11T20:52:20.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:19 smithi023 bash[21372]: cluster 2024-03-11T20:52:18.790605+0000 mon.a (mon.0) 858 : cluster 0 osdmap e106: 8 total, 8 up, 8 in 2024-03-11T20:52:20.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:19 smithi023 bash[21372]: audit 2024-03-11T20:52:19.013959+0000 mon.a (mon.0) 859 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:52:20.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:19 smithi023 bash[21372]: cluster 2024-03-11T20:52:19.574155+0000 osd.6 (osd.6) 29 : cluster 0 4.a scrub starts 2024-03-11T20:52:20.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:19 smithi023 bash[21372]: cluster 2024-03-11T20:52:19.576409+0000 osd.6 (osd.6) 30 : cluster 0 4.a scrub ok 2024-03-11T20:52:20.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:19 smithi023 bash[29146]: cluster 2024-03-11T20:52:18.093712+0000 osd.7 (osd.7) 37 : cluster 0 6.1c deep-scrub starts 2024-03-11T20:52:20.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:19 smithi023 bash[29146]: cluster 2024-03-11T20:52:18.098899+0000 osd.7 (osd.7) 38 : cluster 0 6.1c deep-scrub ok 2024-03-11T20:52:20.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:19 smithi023 bash[29146]: cluster 2024-03-11T20:52:18.790605+0000 mon.a (mon.0) 858 : cluster 0 osdmap e106: 8 total, 8 up, 8 in 2024-03-11T20:52:20.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:19 smithi023 bash[29146]: audit 2024-03-11T20:52:19.013959+0000 mon.a (mon.0) 859 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:52:20.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:19 smithi023 bash[29146]: cluster 2024-03-11T20:52:19.574155+0000 osd.6 (osd.6) 29 : cluster 0 4.a scrub starts 2024-03-11T20:52:20.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:19 smithi023 bash[29146]: cluster 2024-03-11T20:52:19.576409+0000 osd.6 (osd.6) 30 : cluster 0 4.a scrub ok 2024-03-11T20:52:21.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:20 smithi146 bash[23809]: cluster 2024-03-11T20:52:19.445502+0000 osd.3 (osd.3) 41 : cluster 0 3.7 scrub starts 2024-03-11T20:52:21.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:20 smithi146 bash[23809]: cluster 2024-03-11T20:52:19.449947+0000 osd.3 (osd.3) 42 : cluster 0 3.7 scrub ok 2024-03-11T20:52:21.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:20 smithi146 bash[23809]: cluster 2024-03-11T20:52:19.499546+0000 osd.4 (osd.4) 45 : cluster 0 4.1a scrub starts 2024-03-11T20:52:21.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:20 smithi146 bash[23809]: cluster 2024-03-11T20:52:19.501821+0000 osd.4 (osd.4) 46 : cluster 0 4.1a scrub ok 2024-03-11T20:52:21.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:20 smithi146 bash[23809]: cluster 2024-03-11T20:52:19.830926+0000 mgr.y (mgr.24380) 51 : cluster 0 pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:52:21.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:20 smithi023 bash[21372]: cluster 2024-03-11T20:52:19.445502+0000 osd.3 (osd.3) 41 : cluster 0 3.7 scrub starts 2024-03-11T20:52:21.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:20 smithi023 bash[21372]: cluster 2024-03-11T20:52:19.449947+0000 osd.3 (osd.3) 42 : cluster 0 3.7 scrub ok 2024-03-11T20:52:21.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:20 smithi023 bash[21372]: cluster 2024-03-11T20:52:19.499546+0000 osd.4 (osd.4) 45 : cluster 0 4.1a scrub starts 2024-03-11T20:52:21.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:20 smithi023 bash[21372]: cluster 2024-03-11T20:52:19.501821+0000 osd.4 (osd.4) 46 : cluster 0 4.1a scrub ok 2024-03-11T20:52:21.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:20 smithi023 bash[21372]: cluster 2024-03-11T20:52:19.830926+0000 mgr.y (mgr.24380) 51 : cluster 0 pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:52:21.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:20 smithi023 bash[29146]: cluster 2024-03-11T20:52:19.445502+0000 osd.3 (osd.3) 41 : cluster 0 3.7 scrub starts 2024-03-11T20:52:21.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:20 smithi023 bash[29146]: cluster 2024-03-11T20:52:19.449947+0000 osd.3 (osd.3) 42 : cluster 0 3.7 scrub ok 2024-03-11T20:52:21.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:20 smithi023 bash[29146]: cluster 2024-03-11T20:52:19.499546+0000 osd.4 (osd.4) 45 : cluster 0 4.1a scrub starts 2024-03-11T20:52:21.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:20 smithi023 bash[29146]: cluster 2024-03-11T20:52:19.501821+0000 osd.4 (osd.4) 46 : cluster 0 4.1a scrub ok 2024-03-11T20:52:21.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:20 smithi023 bash[29146]: cluster 2024-03-11T20:52:19.830926+0000 mgr.y (mgr.24380) 51 : cluster 0 pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:52:23.036 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:52:23.036 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:52:23.037 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:52:23.037 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:52:23.041 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:52:23.042 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:52:23.042 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:52:23.043 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:52:23.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:22 smithi146 bash[23809]: cluster 2024-03-11T20:52:21.083892+0000 osd.7 (osd.7) 39 : cluster 0 3.11 scrub starts 2024-03-11T20:52:23.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:22 smithi146 bash[23809]: cluster 2024-03-11T20:52:21.088509+0000 osd.7 (osd.7) 40 : cluster 0 3.11 scrub ok 2024-03-11T20:52:23.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:22 smithi146 bash[23809]: cluster 2024-03-11T20:52:21.533479+0000 osd.4 (osd.4) 47 : cluster 0 4.9 deep-scrub starts 2024-03-11T20:52:23.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:22 smithi146 bash[23809]: cluster 2024-03-11T20:52:21.538871+0000 osd.4 (osd.4) 48 : cluster 0 4.9 deep-scrub ok 2024-03-11T20:52:23.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:22 smithi146 bash[23809]: cluster 2024-03-11T20:52:21.831655+0000 mgr.y (mgr.24380) 52 : cluster 0 pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:52:23.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:22 smithi023 bash[21372]: cluster 2024-03-11T20:52:21.083892+0000 osd.7 (osd.7) 39 : cluster 0 3.11 scrub starts 2024-03-11T20:52:23.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:22 smithi023 bash[21372]: cluster 2024-03-11T20:52:21.088509+0000 osd.7 (osd.7) 40 : cluster 0 3.11 scrub ok 2024-03-11T20:52:23.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:22 smithi023 bash[21372]: cluster 2024-03-11T20:52:21.533479+0000 osd.4 (osd.4) 47 : cluster 0 4.9 deep-scrub starts 2024-03-11T20:52:23.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:22 smithi023 bash[21372]: cluster 2024-03-11T20:52:21.538871+0000 osd.4 (osd.4) 48 : cluster 0 4.9 deep-scrub ok 2024-03-11T20:52:23.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:22 smithi023 bash[21372]: cluster 2024-03-11T20:52:21.831655+0000 mgr.y (mgr.24380) 52 : cluster 0 pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:52:23.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:22 smithi023 bash[29146]: cluster 2024-03-11T20:52:21.083892+0000 osd.7 (osd.7) 39 : cluster 0 3.11 scrub starts 2024-03-11T20:52:23.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:22 smithi023 bash[29146]: cluster 2024-03-11T20:52:21.088509+0000 osd.7 (osd.7) 40 : cluster 0 3.11 scrub ok 2024-03-11T20:52:23.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:22 smithi023 bash[29146]: cluster 2024-03-11T20:52:21.533479+0000 osd.4 (osd.4) 47 : cluster 0 4.9 deep-scrub starts 2024-03-11T20:52:23.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:22 smithi023 bash[29146]: cluster 2024-03-11T20:52:21.538871+0000 osd.4 (osd.4) 48 : cluster 0 4.9 deep-scrub ok 2024-03-11T20:52:23.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:22 smithi023 bash[29146]: cluster 2024-03-11T20:52:21.831655+0000 mgr.y (mgr.24380) 52 : cluster 0 pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:52:24.195 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:23 smithi146 bash[23809]: cluster 2024-03-11T20:52:22.072019+0000 osd.7 (osd.7) 41 : cluster 0 6.15 scrub starts 2024-03-11T20:52:24.195 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:23 smithi146 bash[23809]: cluster 2024-03-11T20:52:22.074590+0000 osd.7 (osd.7) 42 : cluster 0 6.15 scrub ok 2024-03-11T20:52:24.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:23 smithi023 bash[21372]: cluster 2024-03-11T20:52:22.072019+0000 osd.7 (osd.7) 41 : cluster 0 6.15 scrub starts 2024-03-11T20:52:24.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:23 smithi023 bash[21372]: cluster 2024-03-11T20:52:22.074590+0000 osd.7 (osd.7) 42 : cluster 0 6.15 scrub ok 2024-03-11T20:52:24.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:23 smithi023 bash[29146]: cluster 2024-03-11T20:52:22.072019+0000 osd.7 (osd.7) 41 : cluster 0 6.15 scrub starts 2024-03-11T20:52:24.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:23 smithi023 bash[29146]: cluster 2024-03-11T20:52:22.074590+0000 osd.7 (osd.7) 42 : cluster 0 6.15 scrub ok 2024-03-11T20:52:25.149 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:24 smithi023 bash[21372]: cluster 2024-03-11T20:52:23.138667+0000 osd.4 (osd.4) 49 : cluster 0 purged_snaps scrub starts 2024-03-11T20:52:25.149 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:24 smithi023 bash[21372]: cluster 2024-03-11T20:52:23.138815+0000 osd.4 (osd.4) 50 : cluster 0 purged_snaps scrub ok 2024-03-11T20:52:25.149 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:24 smithi023 bash[21372]: cluster 2024-03-11T20:52:23.163046+0000 osd.1 (osd.1) 45 : cluster 0 2.a scrub starts 2024-03-11T20:52:25.149 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:24 smithi023 bash[21372]: cluster 2024-03-11T20:52:23.165111+0000 osd.1 (osd.1) 46 : cluster 0 2.a scrub ok 2024-03-11T20:52:25.149 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:24 smithi023 bash[21372]: cluster 2024-03-11T20:52:23.832190+0000 mgr.y (mgr.24380) 53 : cluster 0 pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T20:52:25.149 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:24 smithi023 bash[21372]: cluster 2024-03-11T20:52:23.934752+0000 mon.a (mon.0) 860 : cluster 0 osdmap e107: 8 total, 8 up, 8 in 2024-03-11T20:52:25.149 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:24 smithi023 bash[29146]: cluster 2024-03-11T20:52:23.138667+0000 osd.4 (osd.4) 49 : cluster 0 purged_snaps scrub starts 2024-03-11T20:52:25.149 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:24 smithi023 bash[29146]: cluster 2024-03-11T20:52:23.138815+0000 osd.4 (osd.4) 50 : cluster 0 purged_snaps scrub ok 2024-03-11T20:52:25.150 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:24 smithi023 bash[29146]: cluster 2024-03-11T20:52:23.163046+0000 osd.1 (osd.1) 45 : cluster 0 2.a scrub starts 2024-03-11T20:52:25.150 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:24 smithi023 bash[29146]: cluster 2024-03-11T20:52:23.165111+0000 osd.1 (osd.1) 46 : cluster 0 2.a scrub ok 2024-03-11T20:52:25.150 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:24 smithi023 bash[29146]: cluster 2024-03-11T20:52:23.832190+0000 mgr.y (mgr.24380) 53 : cluster 0 pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T20:52:25.150 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:24 smithi023 bash[29146]: cluster 2024-03-11T20:52:23.934752+0000 mon.a (mon.0) 860 : cluster 0 osdmap e107: 8 total, 8 up, 8 in 2024-03-11T20:52:25.226 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:24 smithi146 bash[23809]: cluster 2024-03-11T20:52:23.138667+0000 osd.4 (osd.4) 49 : cluster 0 purged_snaps scrub starts 2024-03-11T20:52:25.226 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:24 smithi146 bash[23809]: cluster 2024-03-11T20:52:23.138815+0000 osd.4 (osd.4) 50 : cluster 0 purged_snaps scrub ok 2024-03-11T20:52:25.226 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:24 smithi146 bash[23809]: cluster 2024-03-11T20:52:23.163046+0000 osd.1 (osd.1) 45 : cluster 0 2.a scrub starts 2024-03-11T20:52:25.226 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:24 smithi146 bash[23809]: cluster 2024-03-11T20:52:23.165111+0000 osd.1 (osd.1) 46 : cluster 0 2.a scrub ok 2024-03-11T20:52:25.226 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:24 smithi146 bash[23809]: cluster 2024-03-11T20:52:23.832190+0000 mgr.y (mgr.24380) 53 : cluster 0 pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T20:52:25.226 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:24 smithi146 bash[23809]: cluster 2024-03-11T20:52:23.934752+0000 mon.a (mon.0) 860 : cluster 0 osdmap e107: 8 total, 8 up, 8 in 2024-03-11T20:52:25.226 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 20:52:24 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T20:52:25.508 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Mar 11 20:52:25 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:52:25.509 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 11 20:52:25 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:52:25.509 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 11 20:52:25 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:52:25.509 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 20:52:25 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:52:25.510 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Mar 11 20:52:25 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:52:25.510 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:25 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:52:25.510 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:25 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:52:25.510 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:25 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:52:25.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:25 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:52:25.511 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 11 20:52:25 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:52:25.511 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 11 20:52:25 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:52:25.511 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Mar 11 20:52:25 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:52:25.511 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Mar 11 20:52:25 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:52:25.762 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 11 20:52:25 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:52:25.762 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 11 20:52:25 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:52:25.762 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Mar 11 20:52:25 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:52:25.763 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Mar 11 20:52:25 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:52:25.763 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 11 20:52:25 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:52:25.763 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 11 20:52:25 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:52:25.764 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 20:52:25 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:52:25.764 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:25 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:52:25.764 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:25 smithi146 systemd[1]: Started Ceph grafana.a for 67ba00f8-dfe7-11ee-95c8-87774f69a715. 2024-03-11T20:52:25.764 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:25 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T20:52:26.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:25 smithi146 bash[23809]: cluster 2024-03-11T20:52:24.393973+0000 osd.3 (osd.3) 43 : cluster 0 2.19 scrub starts 2024-03-11T20:52:26.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:25 smithi146 bash[23809]: cluster 2024-03-11T20:52:24.396870+0000 osd.3 (osd.3) 44 : cluster 0 2.19 scrub ok 2024-03-11T20:52:26.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:25 smithi146 bash[23809]: audit 2024-03-11T20:52:24.859297+0000 mgr.y (mgr.24380) 54 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:52:26.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:25 smithi146 bash[23809]: audit 2024-03-11T20:52:25.693557+0000 mon.a (mon.0) 861 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:52:26.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:25 smithi146 bash[23809]: audit 2024-03-11T20:52:25.710023+0000 mon.a (mon.0) 862 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:52:26.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:25 smithi146 bash[23809]: audit 2024-03-11T20:52:25.760792+0000 mon.a (mon.0) 863 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:52:26.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:25 smithi146 bash[23809]: audit 2024-03-11T20:52:25.767429+0000 mon.a (mon.0) 864 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:52:26.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:25 smithi146 bash[23809]: audit 2024-03-11T20:52:25.832993+0000 mon.a (mon.0) 865 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:52:26.271 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:25 smithi023 bash[21372]: cluster 2024-03-11T20:52:24.393973+0000 osd.3 (osd.3) 43 : cluster 0 2.19 scrub starts 2024-03-11T20:52:26.271 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:25 smithi023 bash[21372]: cluster 2024-03-11T20:52:24.396870+0000 osd.3 (osd.3) 44 : cluster 0 2.19 scrub ok 2024-03-11T20:52:26.271 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:25 smithi023 bash[21372]: audit 2024-03-11T20:52:24.859297+0000 mgr.y (mgr.24380) 54 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:52:26.271 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:25 smithi023 bash[21372]: audit 2024-03-11T20:52:25.693557+0000 mon.a (mon.0) 861 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:52:26.271 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:25 smithi023 bash[21372]: audit 2024-03-11T20:52:25.710023+0000 mon.a (mon.0) 862 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:52:26.271 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:25 smithi023 bash[21372]: audit 2024-03-11T20:52:25.760792+0000 mon.a (mon.0) 863 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:52:26.271 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:25 smithi023 bash[21372]: audit 2024-03-11T20:52:25.767429+0000 mon.a (mon.0) 864 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:52:26.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:25 smithi023 bash[21372]: audit 2024-03-11T20:52:25.832993+0000 mon.a (mon.0) 865 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:52:26.272 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:25 smithi023 bash[29146]: cluster 2024-03-11T20:52:24.393973+0000 osd.3 (osd.3) 43 : cluster 0 2.19 scrub starts 2024-03-11T20:52:26.272 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:25 smithi023 bash[29146]: cluster 2024-03-11T20:52:24.396870+0000 osd.3 (osd.3) 44 : cluster 0 2.19 scrub ok 2024-03-11T20:52:26.272 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:25 smithi023 bash[29146]: audit 2024-03-11T20:52:24.859297+0000 mgr.y (mgr.24380) 54 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:52:26.272 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:25 smithi023 bash[29146]: audit 2024-03-11T20:52:25.693557+0000 mon.a (mon.0) 861 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:52:26.272 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:25 smithi023 bash[29146]: audit 2024-03-11T20:52:25.710023+0000 mon.a (mon.0) 862 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:52:26.272 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:25 smithi023 bash[29146]: audit 2024-03-11T20:52:25.760792+0000 mon.a (mon.0) 863 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:52:26.272 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:25 smithi023 bash[29146]: audit 2024-03-11T20:52:25.767429+0000 mon.a (mon.0) 864 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:52:26.272 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:25 smithi023 bash[29146]: audit 2024-03-11T20:52:25.832993+0000 mon.a (mon.0) 865 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:52:27.086 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:26 smithi146 bash[23809]: cluster 2024-03-11T20:52:25.112707+0000 osd.7 (osd.7) 43 : cluster 0 6.5 scrub starts 2024-03-11T20:52:27.087 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:26 smithi146 bash[23809]: cluster 2024-03-11T20:52:25.115225+0000 osd.7 (osd.7) 44 : cluster 0 6.5 scrub ok 2024-03-11T20:52:27.087 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:26 smithi146 bash[23809]: cluster 2024-03-11T20:52:25.123564+0000 osd.1 (osd.1) 47 : cluster 0 4.7 scrub starts 2024-03-11T20:52:27.087 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:26 smithi146 bash[23809]: cluster 2024-03-11T20:52:25.125690+0000 osd.1 (osd.1) 48 : cluster 0 4.7 scrub ok 2024-03-11T20:52:27.087 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:26 smithi146 bash[23809]: cluster 2024-03-11T20:52:25.833694+0000 mgr.y (mgr.24380) 55 : cluster 0 pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:52:27.087 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:26 smithi146 bash[23809]: audit 2024-03-11T20:52:26.360393+0000 mon.a (mon.0) 866 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:52:27.087 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:26 smithi146 bash[23809]: audit 2024-03-11T20:52:26.368779+0000 mon.a (mon.0) 867 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:52:27.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:26 smithi023 bash[29146]: cluster 2024-03-11T20:52:25.112707+0000 osd.7 (osd.7) 43 : cluster 0 6.5 scrub starts 2024-03-11T20:52:27.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:26 smithi023 bash[29146]: cluster 2024-03-11T20:52:25.115225+0000 osd.7 (osd.7) 44 : cluster 0 6.5 scrub ok 2024-03-11T20:52:27.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:26 smithi023 bash[29146]: cluster 2024-03-11T20:52:25.123564+0000 osd.1 (osd.1) 47 : cluster 0 4.7 scrub starts 2024-03-11T20:52:27.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:26 smithi023 bash[29146]: cluster 2024-03-11T20:52:25.125690+0000 osd.1 (osd.1) 48 : cluster 0 4.7 scrub ok 2024-03-11T20:52:27.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:26 smithi023 bash[29146]: cluster 2024-03-11T20:52:25.833694+0000 mgr.y (mgr.24380) 55 : cluster 0 pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:52:27.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:26 smithi023 bash[29146]: audit 2024-03-11T20:52:26.360393+0000 mon.a (mon.0) 866 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:52:27.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:26 smithi023 bash[29146]: audit 2024-03-11T20:52:26.368779+0000 mon.a (mon.0) 867 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:52:27.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:26 smithi023 bash[21372]: cluster 2024-03-11T20:52:25.112707+0000 osd.7 (osd.7) 43 : cluster 0 6.5 scrub starts 2024-03-11T20:52:27.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:26 smithi023 bash[21372]: cluster 2024-03-11T20:52:25.115225+0000 osd.7 (osd.7) 44 : cluster 0 6.5 scrub ok 2024-03-11T20:52:27.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:26 smithi023 bash[21372]: cluster 2024-03-11T20:52:25.123564+0000 osd.1 (osd.1) 47 : cluster 0 4.7 scrub starts 2024-03-11T20:52:27.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:26 smithi023 bash[21372]: cluster 2024-03-11T20:52:25.125690+0000 osd.1 (osd.1) 48 : cluster 0 4.7 scrub ok 2024-03-11T20:52:27.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:26 smithi023 bash[21372]: cluster 2024-03-11T20:52:25.833694+0000 mgr.y (mgr.24380) 55 : cluster 0 pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:52:27.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:26 smithi023 bash[21372]: audit 2024-03-11T20:52:26.360393+0000 mon.a (mon.0) 866 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:52:27.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:26 smithi023 bash[21372]: audit 2024-03-11T20:52:26.368779+0000 mon.a (mon.0) 867 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:52:27.348 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:27 smithi146 bash[51661]: logger=settings t=2024-03-11T20:52:27.094323551Z level=info msg="Starting Grafana" version=9.4.12 commit=37ee25ba46 branch=HEAD compiled=2023-05-22T15:58:05Z 2024-03-11T20:52:27.348 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:27 smithi146 bash[51661]: logger=settings t=2024-03-11T20:52:27.094521297Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead." 2024-03-11T20:52:27.348 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:27 smithi146 bash[51661]: logger=settings t=2024-03-11T20:52:27.094542338Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-03-11T20:52:27.348 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:27 smithi146 bash[51661]: logger=settings t=2024-03-11T20:52:27.094550529Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-03-11T20:52:27.348 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:27 smithi146 bash[51661]: logger=settings t=2024-03-11T20:52:27.094557512Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-03-11T20:52:27.348 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:27 smithi146 bash[51661]: logger=settings t=2024-03-11T20:52:27.094564371Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-03-11T20:52:27.348 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:27 smithi146 bash[51661]: logger=settings t=2024-03-11T20:52:27.094571182Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-03-11T20:52:27.348 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:27 smithi146 bash[51661]: logger=settings t=2024-03-11T20:52:27.094577908Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-03-11T20:52:27.349 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:27 smithi146 bash[51661]: logger=settings t=2024-03-11T20:52:27.094584782Z level=info msg="Path Home" path=/usr/share/grafana 2024-03-11T20:52:27.349 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:27 smithi146 bash[51661]: logger=settings t=2024-03-11T20:52:27.094591542Z level=info msg="Path Data" path=/var/lib/grafana 2024-03-11T20:52:27.349 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:27 smithi146 bash[51661]: logger=settings t=2024-03-11T20:52:27.094603896Z level=info msg="Path Logs" path=/var/log/grafana 2024-03-11T20:52:27.349 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:27 smithi146 bash[51661]: logger=settings t=2024-03-11T20:52:27.094621325Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-03-11T20:52:27.349 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:27 smithi146 bash[51661]: logger=settings t=2024-03-11T20:52:27.094634914Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-03-11T20:52:27.349 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:27 smithi146 bash[51661]: logger=settings t=2024-03-11T20:52:27.094652473Z level=info msg="App mode production" 2024-03-11T20:52:27.349 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:27 smithi146 bash[51661]: logger=sqlstore t=2024-03-11T20:52:27.094732753Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-03-11T20:52:27.349 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:27 smithi146 bash[51661]: logger=sqlstore t=2024-03-11T20:52:27.094768091Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-03-11T20:52:27.349 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:27 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:27.0961509Z level=info msg="Starting DB migrations" 2024-03-11T20:52:27.349 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:27 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:27.096275865Z level=info msg="Executing migration" id="create migration_log table" 2024-03-11T20:52:27.349 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:27 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:27.127103419Z level=info msg="Executing migration" id="create user table" 2024-03-11T20:52:27.349 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:27 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:27.152416103Z level=info msg="Executing migration" id="add unique index user.login" 2024-03-11T20:52:27.349 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:27 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:27.16939957Z level=info msg="Executing migration" id="add unique index user.email" 2024-03-11T20:52:27.349 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:27 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:27.202756775Z level=info msg="Executing migration" id="drop index UQE_user_login - v1" 2024-03-11T20:52:27.349 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:27 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:27.219761518Z level=info msg="Executing migration" id="drop index UQE_user_email - v1" 2024-03-11T20:52:27.349 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:27 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:27.253546991Z level=info msg="Executing migration" id="Rename table user to user_v1 - v1" 2024-03-11T20:52:27.350 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:27 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:27.270625652Z level=info msg="Executing migration" id="create user table v2" 2024-03-11T20:52:27.350 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:27 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:27.287451489Z level=info msg="Executing migration" id="create index UQE_user_login - v2" 2024-03-11T20:52:27.350 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:27 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:27.32937496Z level=info msg="Executing migration" id="create index UQE_user_email - v2" 2024-03-11T20:52:27.350 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:27 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:27.354755456Z level=info msg="Executing migration" id="copy data_source v1 to v2" 2024-03-11T20:52:27.615 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:27 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:27.379858963Z level=info msg="Executing migration" id="Drop old table user_v1" 2024-03-11T20:52:27.615 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:27 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:27.421864565Z level=info msg="Executing migration" id="Add column help_flags1 to user table" 2024-03-11T20:52:27.615 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:27 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:27.447098272Z level=info msg="Executing migration" id="Update user table charset" 2024-03-11T20:52:27.616 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:27 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:27.489065933Z level=info msg="Executing migration" id="Add last_seen_at column to user" 2024-03-11T20:52:27.616 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:27 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:27.530924194Z level=info msg="Executing migration" id="Add missing user data" 2024-03-11T20:52:27.616 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:27 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:27.598054786Z level=info msg="Executing migration" id="Add is_disabled column to user" 2024-03-11T20:52:27.616 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:27 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:27.62332854Z level=info msg="Executing migration" id="Add index user.login/user.email" 2024-03-11T20:52:27.868 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:27 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:27.665293109Z level=info msg="Executing migration" id="Add is_service_account column to user" 2024-03-11T20:52:27.868 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:27 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:27.690467062Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2024-03-11T20:52:27.868 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:27 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:27.70751115Z level=info msg="Executing migration" id="create temp user table v1-7" 2024-03-11T20:52:27.868 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:27 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:27.749556609Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v1-7" 2024-03-11T20:52:27.868 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:27 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:27.799928623Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v1-7" 2024-03-11T20:52:27.868 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:27 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:27.825213024Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v1-7" 2024-03-11T20:52:27.868 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:27 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:27.850319652Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v1-7" 2024-03-11T20:52:27.868 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:27 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:27.875710105Z level=info msg="Executing migration" id="Update temp_user table charset" 2024-03-11T20:52:28.120 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:27 smithi146 bash[23809]: cluster 2024-03-11T20:52:26.395641+0000 osd.3 (osd.3) 45 : cluster 0 2.1e scrub starts 2024-03-11T20:52:28.121 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:27 smithi146 bash[23809]: cluster 2024-03-11T20:52:26.396861+0000 osd.3 (osd.3) 46 : cluster 0 2.1e scrub ok 2024-03-11T20:52:28.121 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:27 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:27.900892323Z level=info msg="Executing migration" id="drop index IDX_temp_user_email - v1" 2024-03-11T20:52:28.121 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:27 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:27.91780305Z level=info msg="Executing migration" id="drop index IDX_temp_user_org_id - v1" 2024-03-11T20:52:28.121 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:27 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:27.934765359Z level=info msg="Executing migration" id="drop index IDX_temp_user_code - v1" 2024-03-11T20:52:28.121 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:27 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:27.968475916Z level=info msg="Executing migration" id="drop index IDX_temp_user_status - v1" 2024-03-11T20:52:28.121 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:27 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:27.993725187Z level=info msg="Executing migration" id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-03-11T20:52:28.121 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:28 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:28.019056108Z level=info msg="Executing migration" id="create temp_user v2" 2024-03-11T20:52:28.121 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:28 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:28.060807674Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v2" 2024-03-11T20:52:28.121 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:28 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:28.103189737Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v2" 2024-03-11T20:52:28.121 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:28 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:28.128485627Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v2" 2024-03-11T20:52:28.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:27 smithi023 bash[21372]: cluster 2024-03-11T20:52:26.395641+0000 osd.3 (osd.3) 45 : cluster 0 2.1e scrub starts 2024-03-11T20:52:28.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:27 smithi023 bash[21372]: cluster 2024-03-11T20:52:26.396861+0000 osd.3 (osd.3) 46 : cluster 0 2.1e scrub ok 2024-03-11T20:52:28.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:27 smithi023 bash[29146]: cluster 2024-03-11T20:52:26.395641+0000 osd.3 (osd.3) 45 : cluster 0 2.1e scrub starts 2024-03-11T20:52:28.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:27 smithi023 bash[29146]: cluster 2024-03-11T20:52:26.396861+0000 osd.3 (osd.3) 46 : cluster 0 2.1e scrub ok 2024-03-11T20:52:28.389 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:28 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:28.1536256Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v2" 2024-03-11T20:52:28.389 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:28 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:28.195798441Z level=info msg="Executing migration" id="copy temp_user v1 to v2" 2024-03-11T20:52:28.390 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:28 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:28.220983657Z level=info msg="Executing migration" id="drop temp_user_tmp_qwerty" 2024-03-11T20:52:28.390 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:28 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:28.262930041Z level=info msg="Executing migration" id="Set created for temp users that will otherwise prematurely expire" 2024-03-11T20:52:28.390 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:28 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:28.288177102Z level=info msg="Executing migration" id="create star table" 2024-03-11T20:52:28.390 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:28 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:28.330155787Z level=info msg="Executing migration" id="add unique index star.user_id_dashboard_id" 2024-03-11T20:52:28.390 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:28 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:28.372297362Z level=info msg="Executing migration" id="create org table v1" 2024-03-11T20:52:28.390 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:28 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:28.3974992Z level=info msg="Executing migration" id="create index UQE_org_name - v1" 2024-03-11T20:52:28.675 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:28 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:28.422669023Z level=info msg="Executing migration" id="create org_user table v1" 2024-03-11T20:52:28.675 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:28 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:28.447941321Z level=info msg="Executing migration" id="create index IDX_org_user_org_id - v1" 2024-03-11T20:52:28.675 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:28 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:28.489812822Z level=info msg="Executing migration" id="create index UQE_org_user_org_id_user_id - v1" 2024-03-11T20:52:28.675 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:28 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:28.531887326Z level=info msg="Executing migration" id="create index IDX_org_user_user_id - v1" 2024-03-11T20:52:28.675 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:28 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:28.557116317Z level=info msg="Executing migration" id="Update org table charset" 2024-03-11T20:52:28.675 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:28 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:28.598942776Z level=info msg="Executing migration" id="Update org_user table charset" 2024-03-11T20:52:28.676 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:28 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:28.641072418Z level=info msg="Executing migration" id="Migrate all Read Only Viewers to Viewers" 2024-03-11T20:52:28.676 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:28 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:28.682999809Z level=info msg="Executing migration" id="create dashboard table" 2024-03-11T20:52:28.947 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:28 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:28.708594573Z level=info msg="Executing migration" id="add index dashboard.account_id" 2024-03-11T20:52:28.947 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:28 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:28.733777345Z level=info msg="Executing migration" id="add unique index dashboard_account_id_slug" 2024-03-11T20:52:28.947 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:28 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:28.75906547Z level=info msg="Executing migration" id="create dashboard_tag table" 2024-03-11T20:52:28.947 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:28 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:28.801034507Z level=info msg="Executing migration" id="add unique index dashboard_tag.dasboard_id_term" 2024-03-11T20:52:28.947 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:28 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:28.82616008Z level=info msg="Executing migration" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-03-11T20:52:28.947 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:28 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:28.851448058Z level=info msg="Executing migration" id="Rename table dashboard to dashboard_v1 - v1" 2024-03-11T20:52:28.947 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:28 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:28.876664033Z level=info msg="Executing migration" id="create dashboard v2" 2024-03-11T20:52:28.947 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:28 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:28.901840665Z level=info msg="Executing migration" id="create index IDX_dashboard_org_id - v2" 2024-03-11T20:52:28.947 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:28 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:28.927121576Z level=info msg="Executing migration" id="create index UQE_dashboard_org_id_slug - v2" 2024-03-11T20:52:29.213 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:28 smithi146 bash[23809]: cluster 2024-03-11T20:52:27.834354+0000 mgr.y (mgr.24380) 56 : cluster 0 pgmap v36: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T20:52:29.214 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:28 smithi146 bash[23809]: cluster 2024-03-11T20:52:28.043748+0000 osd.7 (osd.7) 45 : cluster 0 1.0 scrub starts 2024-03-11T20:52:29.214 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:28 smithi146 bash[23809]: cluster 2024-03-11T20:52:28.047562+0000 osd.7 (osd.7) 46 : cluster 0 1.0 scrub ok 2024-03-11T20:52:29.214 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:28 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:28.969082844Z level=info msg="Executing migration" id="copy dashboard v1 to v2" 2024-03-11T20:52:29.214 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:28 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:28.994297093Z level=info msg="Executing migration" id="drop table dashboard_v1" 2024-03-11T20:52:29.214 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:29 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:29.036401381Z level=info msg="Executing migration" id="alter dashboard.data to mediumtext v1" 2024-03-11T20:52:29.214 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:29 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:29.061497892Z level=info msg="Executing migration" id="Add column updated_by in dashboard - v2" 2024-03-11T20:52:29.214 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:29 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:29.103553959Z level=info msg="Executing migration" id="Add column created_by in dashboard - v2" 2024-03-11T20:52:29.214 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:29 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:29.128826581Z level=info msg="Executing migration" id="Add column gnetId in dashboard" 2024-03-11T20:52:29.214 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:29 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:29.154083166Z level=info msg="Executing migration" id="Add index for gnetId in dashboard" 2024-03-11T20:52:29.214 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:29 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:29.19610064Z level=info msg="Executing migration" id="Add column plugin_id in dashboard" 2024-03-11T20:52:29.214 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:29 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:29.221254129Z level=info msg="Executing migration" id="Add index for plugin_id in dashboard" 2024-03-11T20:52:29.261 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:28 smithi023 bash[21372]: cluster 2024-03-11T20:52:27.834354+0000 mgr.y (mgr.24380) 56 : cluster 0 pgmap v36: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T20:52:29.261 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:28 smithi023 bash[21372]: cluster 2024-03-11T20:52:28.043748+0000 osd.7 (osd.7) 45 : cluster 0 1.0 scrub starts 2024-03-11T20:52:29.262 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:28 smithi023 bash[21372]: cluster 2024-03-11T20:52:28.047562+0000 osd.7 (osd.7) 46 : cluster 0 1.0 scrub ok 2024-03-11T20:52:29.262 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:28 smithi023 bash[29146]: cluster 2024-03-11T20:52:27.834354+0000 mgr.y (mgr.24380) 56 : cluster 0 pgmap v36: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T20:52:29.262 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:28 smithi023 bash[29146]: cluster 2024-03-11T20:52:28.043748+0000 osd.7 (osd.7) 45 : cluster 0 1.0 scrub starts 2024-03-11T20:52:29.262 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:28 smithi023 bash[29146]: cluster 2024-03-11T20:52:28.047562+0000 osd.7 (osd.7) 46 : cluster 0 1.0 scrub ok 2024-03-11T20:52:29.499 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:29 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:29.254859853Z level=info msg="Executing migration" id="Add index for dashboard_id in dashboard_tag" 2024-03-11T20:52:29.499 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:29 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:29.280092826Z level=info msg="Executing migration" id="Update dashboard table charset" 2024-03-11T20:52:29.499 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:29 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:29.322017974Z level=info msg="Executing migration" id="Update dashboard_tag table charset" 2024-03-11T20:52:29.499 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:29 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:29.347153167Z level=info msg="Executing migration" id="Add column folder_id in dashboard" 2024-03-11T20:52:29.499 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:29 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:29.372396949Z level=info msg="Executing migration" id="Add column isFolder in dashboard" 2024-03-11T20:52:29.499 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:29 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:29.414388649Z level=info msg="Executing migration" id="Add column has_acl in dashboard" 2024-03-11T20:52:29.499 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:29 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:29.439632123Z level=info msg="Executing migration" id="Add column uid in dashboard" 2024-03-11T20:52:29.500 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:29 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:29.464747279Z level=info msg="Executing migration" id="Update uid column values in dashboard" 2024-03-11T20:52:29.500 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:29 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:29.506795959Z level=info msg="Executing migration" id="Add unique index dashboard_org_id_uid" 2024-03-11T20:52:29.761 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:29 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:29.53197055Z level=info msg="Executing migration" id="Remove unique index org_id_slug" 2024-03-11T20:52:29.761 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:29 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:29.557405505Z level=info msg="Executing migration" id="Update dashboard title length" 2024-03-11T20:52:29.761 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:29 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:29.599356838Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_title_folder_id" 2024-03-11T20:52:29.761 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:29 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:29.641350669Z level=info msg="Executing migration" id="create dashboard_provisioning" 2024-03-11T20:52:29.761 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:29 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:29.675296418Z level=info msg="Executing migration" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-03-11T20:52:29.761 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:29 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:29.692191671Z level=info msg="Executing migration" id="create dashboard_provisioning v2" 2024-03-11T20:52:29.761 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:29 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:29.709190084Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-03-11T20:52:29.761 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:29 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:29.73433919Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-03-11T20:52:29.761 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:29 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:29.75131197Z level=info msg="Executing migration" id="copy dashboard_provisioning v1 to v2" 2024-03-11T20:52:30.037 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:29 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:29.793215909Z level=info msg="Executing migration" id="drop dashboard_provisioning_tmp_qwerty" 2024-03-11T20:52:30.038 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:29 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:29.835192334Z level=info msg="Executing migration" id="Add check_sum column" 2024-03-11T20:52:30.038 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:29 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:29.860516808Z level=info msg="Executing migration" id="Add index for dashboard_title" 2024-03-11T20:52:30.038 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:29 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:29.885802306Z level=info msg="Executing migration" id="delete tags for deleted dashboards" 2024-03-11T20:52:30.038 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:29 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:29.927704911Z level=info msg="Executing migration" id="delete stars for deleted dashboards" 2024-03-11T20:52:30.038 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:29 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:29.953012856Z level=info msg="Executing migration" id="Add index for dashboard_is_folder" 2024-03-11T20:52:30.038 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:29 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:29.978233339Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2024-03-11T20:52:30.038 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:30 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:30.045190799Z level=info msg="Executing migration" id="create data_source table" 2024-03-11T20:52:30.298 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:30 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:30.087187941Z level=info msg="Executing migration" id="add index data_source.account_id" 2024-03-11T20:52:30.298 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:30 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:30.129313162Z level=info msg="Executing migration" id="add unique index data_source.account_id_name" 2024-03-11T20:52:30.298 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:30 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:30.154562461Z level=info msg="Executing migration" id="drop index IDX_data_source_account_id - v1" 2024-03-11T20:52:30.299 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:30 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:30.196504236Z level=info msg="Executing migration" id="drop index UQE_data_source_account_id_name - v1" 2024-03-11T20:52:30.299 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:30 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:30.221707133Z level=info msg="Executing migration" id="Rename table data_source to data_source_v1 - v1" 2024-03-11T20:52:30.299 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:30 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:30.247033891Z level=info msg="Executing migration" id="create data_source table v2" 2024-03-11T20:52:30.299 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:30 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:30.272172862Z level=info msg="Executing migration" id="create index IDX_data_source_org_id - v2" 2024-03-11T20:52:30.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:30 smithi023 bash[21372]: audit 2024-03-11T20:52:29.301867+0000 mon.a (mon.0) 868 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:52:30.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:30 smithi023 bash[21372]: cluster 2024-03-11T20:52:29.371607+0000 osd.3 (osd.3) 47 : cluster 0 4.10 scrub starts 2024-03-11T20:52:30.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:30 smithi023 bash[21372]: cluster 2024-03-11T20:52:29.372890+0000 osd.3 (osd.3) 48 : cluster 0 4.10 scrub ok 2024-03-11T20:52:30.534 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:30 smithi023 bash[29146]: audit 2024-03-11T20:52:29.301867+0000 mon.a (mon.0) 868 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:52:30.534 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:30 smithi023 bash[29146]: cluster 2024-03-11T20:52:29.371607+0000 osd.3 (osd.3) 47 : cluster 0 4.10 scrub starts 2024-03-11T20:52:30.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:30 smithi023 bash[29146]: cluster 2024-03-11T20:52:29.372890+0000 osd.3 (osd.3) 48 : cluster 0 4.10 scrub ok 2024-03-11T20:52:30.550 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:30 smithi146 bash[23809]: audit 2024-03-11T20:52:29.301867+0000 mon.a (mon.0) 868 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:52:30.550 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:30 smithi146 bash[23809]: cluster 2024-03-11T20:52:29.371607+0000 osd.3 (osd.3) 47 : cluster 0 4.10 scrub starts 2024-03-11T20:52:30.550 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:30 smithi146 bash[23809]: cluster 2024-03-11T20:52:29.372890+0000 osd.3 (osd.3) 48 : cluster 0 4.10 scrub ok 2024-03-11T20:52:30.551 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:30 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:30.314222799Z level=info msg="Executing migration" id="create index UQE_data_source_org_id_name - v2" 2024-03-11T20:52:30.551 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:30 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:30.356250787Z level=info msg="Executing migration" id="Drop old table data_source_v1 #2" 2024-03-11T20:52:30.551 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:30 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:30.381506799Z level=info msg="Executing migration" id="Add column with_credentials" 2024-03-11T20:52:30.551 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:30 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:30.406715355Z level=info msg="Executing migration" id="Add secure json data column" 2024-03-11T20:52:30.551 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:30 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:30.431956602Z level=info msg="Executing migration" id="Update data_source table charset" 2024-03-11T20:52:30.551 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:30 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:30.457190165Z level=info msg="Executing migration" id="Update initial version to 1" 2024-03-11T20:52:30.551 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:30 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:30.499202059Z level=info msg="Executing migration" id="Add read_only data column" 2024-03-11T20:52:30.551 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:30 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:30.524385779Z level=info msg="Executing migration" id="Migrate logging ds to loki ds" 2024-03-11T20:52:30.551 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:30 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:30.558067272Z level=info msg="Executing migration" id="Update json_data with nulls" 2024-03-11T20:52:30.828 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:30 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:30.591933979Z level=info msg="Executing migration" id="Add uid column" 2024-03-11T20:52:30.828 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:30 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:30.634048603Z level=info msg="Executing migration" id="Update uid value" 2024-03-11T20:52:30.828 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:30 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:30.692676836Z level=info msg="Executing migration" id="Add unique index datasource_org_id_uid" 2024-03-11T20:52:30.828 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:30 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:30.717885182Z level=info msg="Executing migration" id="add unique index datasource_org_id_is_default" 2024-03-11T20:52:30.828 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:30 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:30.74320568Z level=info msg="Executing migration" id="create api_key table" 2024-03-11T20:52:30.828 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:30 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:30.768474389Z level=info msg="Executing migration" id="add index api_key.account_id" 2024-03-11T20:52:30.828 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:30 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:30.793614623Z level=info msg="Executing migration" id="add index api_key.key" 2024-03-11T20:52:30.828 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:30 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:30.835604621Z level=info msg="Executing migration" id="add index api_key.account_id_name" 2024-03-11T20:52:31.080 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:30 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:30.877489004Z level=info msg="Executing migration" id="drop index IDX_api_key_account_id - v1" 2024-03-11T20:52:31.080 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:30 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:30.902658771Z level=info msg="Executing migration" id="drop index UQE_api_key_key - v1" 2024-03-11T20:52:31.080 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:30 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:30.927899157Z level=info msg="Executing migration" id="drop index UQE_api_key_account_id_name - v1" 2024-03-11T20:52:31.080 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:30 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:30.953276231Z level=info msg="Executing migration" id="Rename table api_key to api_key_v1 - v1" 2024-03-11T20:52:31.080 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:30 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:30.986766763Z level=info msg="Executing migration" id="create api_key table v2" 2024-03-11T20:52:31.080 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:31 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:31.012155468Z level=info msg="Executing migration" id="create index IDX_api_key_org_id - v2" 2024-03-11T20:52:31.080 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:31 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:31.037323792Z level=info msg="Executing migration" id="create index UQE_api_key_key - v2" 2024-03-11T20:52:31.080 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:31 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:31.062457232Z level=info msg="Executing migration" id="create index UQE_api_key_org_id_name - v2" 2024-03-11T20:52:31.080 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:31 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:31.087752019Z level=info msg="Executing migration" id="copy api_key v1 to v2" 2024-03-11T20:52:31.365 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:31 smithi146 bash[23809]: cluster 2024-03-11T20:52:29.835461+0000 mgr.y (mgr.24380) 57 : cluster 0 pgmap v37: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:52:31.366 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:31 smithi146 bash[23809]: cluster 2024-03-11T20:52:30.529025+0000 osd.5 (osd.5) 43 : cluster 0 purged_snaps scrub starts 2024-03-11T20:52:31.366 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:31 smithi146 bash[23809]: cluster 2024-03-11T20:52:30.529136+0000 osd.5 (osd.5) 44 : cluster 0 purged_snaps scrub ok 2024-03-11T20:52:31.366 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:31 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:31.129483038Z level=info msg="Executing migration" id="Drop old table api_key_v1" 2024-03-11T20:52:31.366 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:31 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:31.155079886Z level=info msg="Executing migration" id="Update api_key table charset" 2024-03-11T20:52:31.366 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:31 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:31.180214714Z level=info msg="Executing migration" id="Add expires to api_key table" 2024-03-11T20:52:31.366 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:31 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:31.222151877Z level=info msg="Executing migration" id="Add service account foreign key" 2024-03-11T20:52:31.366 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:31 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:31.247358744Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2024-03-11T20:52:31.366 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:31 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:31.333298987Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2024-03-11T20:52:31.366 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:31 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:31.373450854Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2024-03-11T20:52:31.570 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:31 smithi023 bash[21372]: cluster 2024-03-11T20:52:29.835461+0000 mgr.y (mgr.24380) 57 : cluster 0 pgmap v37: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:52:31.570 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:31 smithi023 bash[21372]: cluster 2024-03-11T20:52:30.529025+0000 osd.5 (osd.5) 43 : cluster 0 purged_snaps scrub starts 2024-03-11T20:52:31.570 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:31 smithi023 bash[21372]: cluster 2024-03-11T20:52:30.529136+0000 osd.5 (osd.5) 44 : cluster 0 purged_snaps scrub ok 2024-03-11T20:52:31.570 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:31 smithi023 bash[29146]: cluster 2024-03-11T20:52:29.835461+0000 mgr.y (mgr.24380) 57 : cluster 0 pgmap v37: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:52:31.571 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:31 smithi023 bash[29146]: cluster 2024-03-11T20:52:30.529025+0000 osd.5 (osd.5) 43 : cluster 0 purged_snaps scrub starts 2024-03-11T20:52:31.571 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:31 smithi023 bash[29146]: cluster 2024-03-11T20:52:30.529136+0000 osd.5 (osd.5) 44 : cluster 0 purged_snaps scrub ok 2024-03-11T20:52:31.644 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:31 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:31.398634137Z level=info msg="Executing migration" id="create dashboard_snapshot table v4" 2024-03-11T20:52:31.644 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:31 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:31.423749813Z level=info msg="Executing migration" id="drop table dashboard_snapshot_v4 #1" 2024-03-11T20:52:31.644 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:31 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:31.465812911Z level=info msg="Executing migration" id="create dashboard_snapshot table v5 #2" 2024-03-11T20:52:31.644 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:31 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:31.50788091Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_key - v5" 2024-03-11T20:52:31.645 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:31 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:31.593815346Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-03-11T20:52:31.645 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:31 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:31.619243062Z level=info msg="Executing migration" id="create index IDX_dashboard_snapshot_user_id - v5" 2024-03-11T20:52:31.645 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:31 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:31.652706549Z level=info msg="Executing migration" id="alter dashboard_snapshot to mediumtext v2" 2024-03-11T20:52:31.898 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:31 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:31.669683552Z level=info msg="Executing migration" id="Update dashboard_snapshot table charset" 2024-03-11T20:52:31.898 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:31 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:31.686557554Z level=info msg="Executing migration" id="Add column external_delete_url to dashboard_snapshots table" 2024-03-11T20:52:31.898 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:31 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:31.703480035Z level=info msg="Executing migration" id="Add encrypted dashboard json column" 2024-03-11T20:52:31.899 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:31 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:31.745469153Z level=info msg="Executing migration" id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-03-11T20:52:31.899 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:31 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:31.77929364Z level=info msg="Executing migration" id="create quota table v1" 2024-03-11T20:52:31.899 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:31 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:31.796208601Z level=info msg="Executing migration" id="create index UQE_quota_org_id_user_id_target - v1" 2024-03-11T20:52:31.899 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:31 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:31.82983243Z level=info msg="Executing migration" id="Update quota table charset" 2024-03-11T20:52:31.899 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:31 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:31.846749335Z level=info msg="Executing migration" id="create plugin_setting table" 2024-03-11T20:52:31.899 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:31 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:31.863613822Z level=info msg="Executing migration" id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-03-11T20:52:31.899 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:31 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:31.880478541Z level=info msg="Executing migration" id="Add column plugin_version to plugin_settings" 2024-03-11T20:52:31.899 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:31 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:31.905833511Z level=info msg="Executing migration" id="Update plugin_setting table charset" 2024-03-11T20:52:32.183 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:31 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:31.931254191Z level=info msg="Executing migration" id="create session table" 2024-03-11T20:52:32.183 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:31 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:31.956264275Z level=info msg="Executing migration" id="Drop old table playlist table" 2024-03-11T20:52:32.184 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:31 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:31.998241152Z level=info msg="Executing migration" id="Drop old table playlist_item table" 2024-03-11T20:52:32.184 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:32 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:32.0403703Z level=info msg="Executing migration" id="create playlist table v2" 2024-03-11T20:52:32.184 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:32 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:32.065639891Z level=info msg="Executing migration" id="create playlist item table v2" 2024-03-11T20:52:32.184 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:32 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:32.107779746Z level=info msg="Executing migration" id="Update playlist table charset" 2024-03-11T20:52:32.184 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:32 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:32.149685184Z level=info msg="Executing migration" id="Update playlist_item table charset" 2024-03-11T20:52:32.184 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:32 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:32.191726532Z level=info msg="Executing migration" id="drop preferences table v2" 2024-03-11T20:52:32.470 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:32 smithi146 bash[23809]: cluster 2024-03-11T20:52:31.336730+0000 mon.a (mon.0) 869 : cluster 0 osdmap e108: 8 total, 8 up, 8 in 2024-03-11T20:52:32.470 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:32 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:32.233750624Z level=info msg="Executing migration" id="drop preferences table v3" 2024-03-11T20:52:32.470 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:32 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:32.259237122Z level=info msg="Executing migration" id="create preferences table v3" 2024-03-11T20:52:32.470 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:32 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:32.301336194Z level=info msg="Executing migration" id="Update preferences table charset" 2024-03-11T20:52:32.470 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:32 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:32.34326674Z level=info msg="Executing migration" id="Add column team_id in preferences" 2024-03-11T20:52:32.471 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:32 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:32.385402017Z level=info msg="Executing migration" id="Update team_id column values in preferences" 2024-03-11T20:52:32.471 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:32 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:32.410974109Z level=info msg="Executing migration" id="Add column week_start in preferences" 2024-03-11T20:52:32.471 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:32 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:32.43618145Z level=info msg="Executing migration" id="Add column preferences.json_data" 2024-03-11T20:52:32.471 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:32 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:32.478157977Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2024-03-11T20:52:32.723 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:32 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:32.520134507Z level=info msg="Executing migration" id="Add preferences index org_id" 2024-03-11T20:52:32.723 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:32 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:32.545477049Z level=info msg="Executing migration" id="Add preferences index user_id" 2024-03-11T20:52:32.723 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:32 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:32.562344814Z level=info msg="Executing migration" id="create alert table v1" 2024-03-11T20:52:32.723 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:32 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:32.579081201Z level=info msg="Executing migration" id="add index alert org_id & id " 2024-03-11T20:52:32.723 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:32 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:32.612812651Z level=info msg="Executing migration" id="add index alert state" 2024-03-11T20:52:32.723 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:32 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:32.646454792Z level=info msg="Executing migration" id="add index alert dashboard_id" 2024-03-11T20:52:32.723 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:32 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:32.688462029Z level=info msg="Executing migration" id="Create alert_rule_tag table v1" 2024-03-11T20:52:32.723 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:32 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:32.730539628Z level=info msg="Executing migration" id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-03-11T20:52:32.784 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:32 smithi023 bash[21372]: cluster 2024-03-11T20:52:31.336730+0000 mon.a (mon.0) 869 : cluster 0 osdmap e108: 8 total, 8 up, 8 in 2024-03-11T20:52:32.784 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:32 smithi023 bash[29146]: cluster 2024-03-11T20:52:31.336730+0000 mon.a (mon.0) 869 : cluster 0 osdmap e108: 8 total, 8 up, 8 in 2024-03-11T20:52:32.992 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:32 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:32.755798161Z level=info msg="Executing migration" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-03-11T20:52:32.992 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:32 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:32.797640531Z level=info msg="Executing migration" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-03-11T20:52:32.992 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:32 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:32.822951163Z level=info msg="Executing migration" id="Create alert_rule_tag table v2" 2024-03-11T20:52:32.992 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:32 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:32.848203867Z level=info msg="Executing migration" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-03-11T20:52:32.992 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:32 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:32.873454786Z level=info msg="Executing migration" id="copy alert_rule_tag v1 to v2" 2024-03-11T20:52:32.992 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:32 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:32.89866528Z level=info msg="Executing migration" id="drop table alert_rule_tag_v1" 2024-03-11T20:52:32.992 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:32 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:32.923963369Z level=info msg="Executing migration" id="create alert_notification table v1" 2024-03-11T20:52:32.992 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:32 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:32.94917669Z level=info msg="Executing migration" id="Add column is_default" 2024-03-11T20:52:32.992 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:32 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:32.974386757Z level=info msg="Executing migration" id="Add column frequency" 2024-03-11T20:52:32.992 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:32 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:32.999595001Z level=info msg="Executing migration" id="Add column send_reminder" 2024-03-11T20:52:33.253 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:33 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:33.024852264Z level=info msg="Executing migration" id="Add column disable_resolve_message" 2024-03-11T20:52:33.253 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:33 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:33.050110578Z level=info msg="Executing migration" id="add index alert_notification org_id & name" 2024-03-11T20:52:33.253 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:33 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:33.075344639Z level=info msg="Executing migration" id="Update alert table charset" 2024-03-11T20:52:33.253 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:33 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:33.10049652Z level=info msg="Executing migration" id="Update alert_notification table charset" 2024-03-11T20:52:33.253 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:33 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:33.142460829Z level=info msg="Executing migration" id="create notification_journal table v1" 2024-03-11T20:52:33.253 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:33 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:33.167787998Z level=info msg="Executing migration" id="add index notification_journal org_id & alert_id & notifier_id" 2024-03-11T20:52:33.253 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:33 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:33.193091835Z level=info msg="Executing migration" id="drop alert_notification_journal" 2024-03-11T20:52:33.253 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:33 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:33.218247673Z level=info msg="Executing migration" id="create alert_notification_state table v1" 2024-03-11T20:52:33.253 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:33 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:33.260238657Z level=info msg="Executing migration" id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-03-11T20:52:33.505 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:33 smithi146 bash[23809]: cluster 2024-03-11T20:52:31.836252+0000 mgr.y (mgr.24380) 58 : cluster 0 pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:52:33.505 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:33 smithi146 bash[23809]: audit 2024-03-11T20:52:32.463695+0000 mon.a (mon.0) 870 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:52:33.505 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:33 smithi146 bash[23809]: audit 2024-03-11T20:52:32.474645+0000 mon.a (mon.0) 871 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:52:33.505 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:33 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:33.285453558Z level=info msg="Executing migration" id="Add for to alert table" 2024-03-11T20:52:33.506 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:33 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:33.311013817Z level=info msg="Executing migration" id="Add column uid in alert_notification" 2024-03-11T20:52:33.506 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:33 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:33.352983972Z level=info msg="Executing migration" id="Update uid column values in alert_notification" 2024-03-11T20:52:33.506 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:33 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:33.395004093Z level=info msg="Executing migration" id="Add unique index alert_notification_org_id_uid" 2024-03-11T20:52:33.506 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:33 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:33.420393198Z level=info msg="Executing migration" id="Remove unique index org_id_name" 2024-03-11T20:52:33.506 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:33 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:33.44553933Z level=info msg="Executing migration" id="Add column secure_settings in alert_notification" 2024-03-11T20:52:33.506 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:33 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:33.470841146Z level=info msg="Executing migration" id="alter alert.settings to mediumtext" 2024-03-11T20:52:33.506 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:33 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:33.512747785Z level=info msg="Executing migration" id="Add non-unique index alert_notification_state_alert_id" 2024-03-11T20:52:33.713 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:33 smithi023 bash[21372]: cluster 2024-03-11T20:52:31.836252+0000 mgr.y (mgr.24380) 58 : cluster 0 pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:52:33.714 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:33 smithi023 bash[21372]: audit 2024-03-11T20:52:32.463695+0000 mon.a (mon.0) 870 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:52:33.714 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:33 smithi023 bash[21372]: audit 2024-03-11T20:52:32.474645+0000 mon.a (mon.0) 871 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:52:33.714 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:33 smithi023 bash[29146]: cluster 2024-03-11T20:52:31.836252+0000 mgr.y (mgr.24380) 58 : cluster 0 pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:52:33.714 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:33 smithi023 bash[29146]: audit 2024-03-11T20:52:32.463695+0000 mon.a (mon.0) 870 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:52:33.714 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:33 smithi023 bash[29146]: audit 2024-03-11T20:52:32.474645+0000 mon.a (mon.0) 871 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:52:33.760 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:33 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:33.537986646Z level=info msg="Executing migration" id="Add non-unique index alert_rule_tag_alert_id" 2024-03-11T20:52:33.761 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:33 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:33.563270846Z level=info msg="Executing migration" id="Drop old annotation table v4" 2024-03-11T20:52:33.761 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:33 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:33.630175432Z level=info msg="Executing migration" id="create annotation table v5" 2024-03-11T20:52:33.761 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:33 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:33.67214708Z level=info msg="Executing migration" id="add index annotation 0 v3" 2024-03-11T20:52:33.761 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:33 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:33.689155977Z level=info msg="Executing migration" id="add index annotation 1 v3" 2024-03-11T20:52:33.761 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:33 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:33.705984849Z level=info msg="Executing migration" id="add index annotation 2 v3" 2024-03-11T20:52:33.761 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:33 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:33.739684854Z level=info msg="Executing migration" id="add index annotation 3 v3" 2024-03-11T20:52:33.809 INFO:teuthology.orchestra.run.smithi023.stdout:188978561079 2024-03-11T20:52:33.810 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-11T20:52:34.026 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:33 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:33.77326404Z level=info msg="Executing migration" id="add index annotation 4 v3" 2024-03-11T20:52:34.026 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:33 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:33.79858237Z level=info msg="Executing migration" id="Update annotation table charset" 2024-03-11T20:52:34.026 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:33 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:33.840529841Z level=info msg="Executing migration" id="Add column region_id to annotation table" 2024-03-11T20:52:34.026 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:33 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:33.857471819Z level=info msg="Executing migration" id="Drop category_id index" 2024-03-11T20:52:34.027 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:33 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:33.874350838Z level=info msg="Executing migration" id="Add column tags to annotation table" 2024-03-11T20:52:34.027 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:33 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:33.908005143Z level=info msg="Executing migration" id="Create annotation_tag table v2" 2024-03-11T20:52:34.027 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:33 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:33.933271503Z level=info msg="Executing migration" id="Add unique index annotation_tag.annotation_id_tag_id" 2024-03-11T20:52:34.027 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:33 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:33.975197042Z level=info msg="Executing migration" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-03-11T20:52:34.027 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:34 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:34.000415206Z level=info msg="Executing migration" id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-03-11T20:52:34.027 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:34 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:34.034006129Z level=info msg="Executing migration" id="Create annotation_tag table v3" 2024-03-11T20:52:34.295 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:34 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:34.059209371Z level=info msg="Executing migration" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-03-11T20:52:34.296 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:34 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:34.084529Z level=info msg="Executing migration" id="copy annotation_tag v2 to v3" 2024-03-11T20:52:34.296 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:34 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:34.126422947Z level=info msg="Executing migration" id="drop table annotation_tag_v2" 2024-03-11T20:52:34.296 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:34 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:34.151684917Z level=info msg="Executing migration" id="Update alert annotations and set TEXT to empty" 2024-03-11T20:52:34.296 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:34 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:34.193692756Z level=info msg="Executing migration" id="Add created time to annotation table" 2024-03-11T20:52:34.296 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:34 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:34.235634305Z level=info msg="Executing migration" id="Add updated time to annotation table" 2024-03-11T20:52:34.296 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:34 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:34.277862893Z level=info msg="Executing migration" id="Add index for created in annotation table" 2024-03-11T20:52:34.296 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:34 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:34.303074075Z level=info msg="Executing migration" id="Add index for updated in annotation table" 2024-03-11T20:52:34.523 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:34 smithi023 bash[29146]: cluster 2024-03-11T20:52:33.340185+0000 osd.3 (osd.3) 49 : cluster 0 6.c scrub starts 2024-03-11T20:52:34.524 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:34 smithi023 bash[29146]: cluster 2024-03-11T20:52:33.342749+0000 osd.3 (osd.3) 50 : cluster 0 6.c scrub ok 2024-03-11T20:52:34.524 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:34 smithi023 bash[29146]: audit 2024-03-11T20:52:34.042790+0000 mon.a (mon.0) 872 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:52:34.524 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:34 smithi023 bash[21372]: cluster 2024-03-11T20:52:33.340185+0000 osd.3 (osd.3) 49 : cluster 0 6.c scrub starts 2024-03-11T20:52:34.524 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:34 smithi023 bash[21372]: cluster 2024-03-11T20:52:33.342749+0000 osd.3 (osd.3) 50 : cluster 0 6.c scrub ok 2024-03-11T20:52:34.524 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:34 smithi023 bash[21372]: audit 2024-03-11T20:52:34.042790+0000 mon.a (mon.0) 872 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:52:34.556 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:34 smithi146 bash[23809]: cluster 2024-03-11T20:52:33.340185+0000 osd.3 (osd.3) 49 : cluster 0 6.c scrub starts 2024-03-11T20:52:34.556 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:34 smithi146 bash[23809]: cluster 2024-03-11T20:52:33.342749+0000 osd.3 (osd.3) 50 : cluster 0 6.c scrub ok 2024-03-11T20:52:34.556 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:34 smithi146 bash[23809]: audit 2024-03-11T20:52:34.042790+0000 mon.a (mon.0) 872 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:52:34.556 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:34 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:34.345156075Z level=info msg="Executing migration" id="Convert existing annotations from seconds to milliseconds" 2024-03-11T20:52:34.557 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:34 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:34.387272145Z level=info msg="Executing migration" id="Add epoch_end column" 2024-03-11T20:52:34.557 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:34 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:34.412619732Z level=info msg="Executing migration" id="Add index for epoch_end" 2024-03-11T20:52:34.557 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:34 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:34.446265911Z level=info msg="Executing migration" id="Make epoch_end the same as epoch" 2024-03-11T20:52:34.557 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:34 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:34.488144164Z level=info msg="Executing migration" id="Move region to single row" 2024-03-11T20:52:34.557 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:34 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:34.513426108Z level=info msg="Executing migration" id="Remove index org_id_epoch from annotation table" 2024-03-11T20:52:34.557 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:34 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:34.538662426Z level=info msg="Executing migration" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-03-11T20:52:34.557 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:34 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:34.563929596Z level=info msg="Executing migration" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-03-11T20:52:34.842 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:34 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:34.589185461Z level=info msg="Executing migration" id="Add index for org_id_epoch_end_epoch on annotation table" 2024-03-11T20:52:34.842 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:34 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:34.631078454Z level=info msg="Executing migration" id="Remove index org_id_epoch_epoch_end from annotation table" 2024-03-11T20:52:34.842 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:34 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:34.673088482Z level=info msg="Executing migration" id="Add index for alert_id on annotation table" 2024-03-11T20:52:34.842 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:34 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:34.714974392Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2024-03-11T20:52:34.842 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:34 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:34.757119945Z level=info msg="Executing migration" id="create test_data table" 2024-03-11T20:52:34.842 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:34 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:34.782371069Z level=info msg="Executing migration" id="create dashboard_version table v1" 2024-03-11T20:52:34.842 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:34 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:34.807553211Z level=info msg="Executing migration" id="add index dashboard_version.dashboard_id" 2024-03-11T20:52:34.843 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:34 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:34.849572718Z level=info msg="Executing migration" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-03-11T20:52:35.094 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 20:52:34 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T20:52:35.094 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:34 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:34.866581556Z level=info msg="Executing migration" id="Set dashboard version to 1 where 0" 2024-03-11T20:52:35.094 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:34 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:34.908448773Z level=info msg="Executing migration" id="save existing dashboard data in dashboard_version table v1" 2024-03-11T20:52:35.094 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:34 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:34.933698747Z level=info msg="Executing migration" id="alter dashboard_version.data to mediumtext v1" 2024-03-11T20:52:35.095 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:34 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:34.958912751Z level=info msg="Executing migration" id="create team table" 2024-03-11T20:52:35.095 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:34 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:34.984323307Z level=info msg="Executing migration" id="add index team.org_id" 2024-03-11T20:52:35.095 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:35 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:35.009500395Z level=info msg="Executing migration" id="add unique index team_org_id_name" 2024-03-11T20:52:35.095 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:35 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:35.03479948Z level=info msg="Executing migration" id="create team member table" 2024-03-11T20:52:35.095 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:35 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:35.060000698Z level=info msg="Executing migration" id="add index team_member.org_id" 2024-03-11T20:52:35.095 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:35 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:35.101945871Z level=info msg="Executing migration" id="add unique index team_member_org_id_team_id_user_id" 2024-03-11T20:52:35.355 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:35 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:35.127217206Z level=info msg="Executing migration" id="add index team_member.team_id" 2024-03-11T20:52:35.355 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:35 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:35.152501074Z level=info msg="Executing migration" id="Add column email to team table" 2024-03-11T20:52:35.355 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:35 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:35.194562259Z level=info msg="Executing migration" id="Add column external to team_member table" 2024-03-11T20:52:35.355 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:35 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:35.219873978Z level=info msg="Executing migration" id="Add column permission to team_member table" 2024-03-11T20:52:35.355 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:35 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:35.24512139Z level=info msg="Executing migration" id="create dashboard acl table" 2024-03-11T20:52:35.355 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:35 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:35.270380812Z level=info msg="Executing migration" id="add index dashboard_acl_dashboard_id" 2024-03-11T20:52:35.356 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:35 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:35.295653426Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_user_id" 2024-03-11T20:52:35.356 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:35 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:35.320892151Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_team_id" 2024-03-11T20:52:35.356 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:35 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:35.362819147Z level=info msg="Executing migration" id="add index dashboard_acl_user_id" 2024-03-11T20:52:35.607 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:35 smithi146 bash[23809]: cluster 2024-03-11T20:52:33.836908+0000 mgr.y (mgr.24380) 59 : cluster 0 pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T20:52:35.607 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:35 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:35.404823827Z level=info msg="Executing migration" id="add index dashboard_acl_team_id" 2024-03-11T20:52:35.607 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:35 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:35.43018087Z level=info msg="Executing migration" id="add index dashboard_acl_org_id_role" 2024-03-11T20:52:35.607 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:35 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:35.455388111Z level=info msg="Executing migration" id="add index dashboard_permission" 2024-03-11T20:52:35.607 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:35 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:35.497268033Z level=info msg="Executing migration" id="save default acl rules in dashboard_acl table" 2024-03-11T20:52:35.608 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:35 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:35.522553056Z level=info msg="Executing migration" id="delete acl rules for deleted dashboards and folders" 2024-03-11T20:52:35.608 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:35 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:35.547718506Z level=info msg="Executing migration" id="create tag table" 2024-03-11T20:52:35.608 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:35 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:35.589579676Z level=info msg="Executing migration" id="add index tag.key_value" 2024-03-11T20:52:35.608 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:35 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:35.615047539Z level=info msg="Executing migration" id="create login attempt table" 2024-03-11T20:52:35.749 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:35 smithi023 bash[21372]: cluster 2024-03-11T20:52:33.836908+0000 mgr.y (mgr.24380) 59 : cluster 0 pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T20:52:35.749 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:35 smithi023 bash[29146]: cluster 2024-03-11T20:52:33.836908+0000 mgr.y (mgr.24380) 59 : cluster 0 pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T20:52:35.780 INFO:teuthology.orchestra.run.smithi023.stdout:34359738475 2024-03-11T20:52:35.781 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-11T20:52:35.867 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:35 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:35.657018658Z level=info msg="Executing migration" id="add index login_attempt.username" 2024-03-11T20:52:35.868 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:35 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:35.682288919Z level=info msg="Executing migration" id="drop index IDX_login_attempt_username - v1" 2024-03-11T20:52:35.868 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:35 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:35.707438288Z level=info msg="Executing migration" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-03-11T20:52:35.868 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:35 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:35.757685321Z level=info msg="Executing migration" id="create login_attempt v2" 2024-03-11T20:52:35.868 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:35 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:35.783047355Z level=info msg="Executing migration" id="create index IDX_login_attempt_username - v2" 2024-03-11T20:52:35.868 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:35 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:35.808301462Z level=info msg="Executing migration" id="copy login_attempt v1 to v2" 2024-03-11T20:52:35.868 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:35 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:35.833513141Z level=info msg="Executing migration" id="drop login_attempt_tmp_qwerty" 2024-03-11T20:52:35.868 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:35 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:35.875428564Z level=info msg="Executing migration" id="create user auth table" 2024-03-11T20:52:36.128 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:35 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:35.900713842Z level=info msg="Executing migration" id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-03-11T20:52:36.129 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:35 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:35.925924508Z level=info msg="Executing migration" id="alter user_auth.auth_id to length 190" 2024-03-11T20:52:36.129 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:35 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:35.951177815Z level=info msg="Executing migration" id="Add OAuth access token to user_auth" 2024-03-11T20:52:36.129 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:35 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:35.97639537Z level=info msg="Executing migration" id="Add OAuth refresh token to user_auth" 2024-03-11T20:52:36.129 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:36 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:36.018351674Z level=info msg="Executing migration" id="Add OAuth token type to user_auth" 2024-03-11T20:52:36.129 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:36 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:36.043655715Z level=info msg="Executing migration" id="Add OAuth expiry to user_auth" 2024-03-11T20:52:36.129 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:36 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:36.085550219Z level=info msg="Executing migration" id="Add index to user_id column in user_auth" 2024-03-11T20:52:36.129 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:36 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:36.136377916Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2024-03-11T20:52:36.406 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:36 smithi146 bash[23809]: audit 2024-03-11T20:52:34.869917+0000 mgr.y (mgr.24380) 60 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:52:36.406 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:36 smithi146 bash[23809]: cluster 2024-03-11T20:52:35.394246+0000 osd.3 (osd.3) 51 : cluster 0 3.2 scrub starts 2024-03-11T20:52:36.406 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:36 smithi146 bash[23809]: cluster 2024-03-11T20:52:35.399008+0000 osd.3 (osd.3) 52 : cluster 0 3.2 scrub ok 2024-03-11T20:52:36.406 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:36 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:36.161511665Z level=info msg="Executing migration" id="create server_lock table" 2024-03-11T20:52:36.406 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:36 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:36.186854976Z level=info msg="Executing migration" id="add index server_lock.operation_uid" 2024-03-11T20:52:36.406 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:36 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:36.228906978Z level=info msg="Executing migration" id="create user auth token table" 2024-03-11T20:52:36.407 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:36 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:36.2541517Z level=info msg="Executing migration" id="add unique index user_auth_token.auth_token" 2024-03-11T20:52:36.407 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:36 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:36.279324513Z level=info msg="Executing migration" id="add unique index user_auth_token.prev_auth_token" 2024-03-11T20:52:36.407 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:36 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:36.304568885Z level=info msg="Executing migration" id="add index user_auth_token.user_id" 2024-03-11T20:52:36.407 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:36 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:36.346522772Z level=info msg="Executing migration" id="Add revoked_at to the user auth token" 2024-03-11T20:52:36.407 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:36 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:36.371754821Z level=info msg="Executing migration" id="create cache_data table" 2024-03-11T20:52:36.407 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:36 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:36.41380395Z level=info msg="Executing migration" id="add unique index cache_data.cache_key" 2024-03-11T20:52:36.627 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:36 smithi023 bash[21372]: audit 2024-03-11T20:52:34.869917+0000 mgr.y (mgr.24380) 60 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:52:36.627 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:36 smithi023 bash[21372]: cluster 2024-03-11T20:52:35.394246+0000 osd.3 (osd.3) 51 : cluster 0 3.2 scrub starts 2024-03-11T20:52:36.627 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:36 smithi023 bash[21372]: cluster 2024-03-11T20:52:35.399008+0000 osd.3 (osd.3) 52 : cluster 0 3.2 scrub ok 2024-03-11T20:52:36.627 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:36 smithi023 bash[29146]: audit 2024-03-11T20:52:34.869917+0000 mgr.y (mgr.24380) 60 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:52:36.627 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:36 smithi023 bash[29146]: cluster 2024-03-11T20:52:35.394246+0000 osd.3 (osd.3) 51 : cluster 0 3.2 scrub starts 2024-03-11T20:52:36.627 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:36 smithi023 bash[29146]: cluster 2024-03-11T20:52:35.399008+0000 osd.3 (osd.3) 52 : cluster 0 3.2 scrub ok 2024-03-11T20:52:36.714 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:36 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:36.439086768Z level=info msg="Executing migration" id="create short_url table v1" 2024-03-11T20:52:36.714 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:36 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:36.481034422Z level=info msg="Executing migration" id="add index short_url.org_id-uid" 2024-03-11T20:52:36.714 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:36 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:36.506219958Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2024-03-11T20:52:36.714 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:36 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:36.531401974Z level=info msg="Executing migration" id="delete alert_definition table" 2024-03-11T20:52:36.714 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:36 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:36.573441684Z level=info msg="Executing migration" id="recreate alert_definition table" 2024-03-11T20:52:36.714 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:36 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:36.61543407Z level=info msg="Executing migration" id="add index in alert_definition on org_id and title columns" 2024-03-11T20:52:36.714 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:36 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:36.72170664Z level=info msg="Executing migration" id="add index in alert_definition on org_id and uid columns" 2024-03-11T20:52:37.010 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:36 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:36.821484804Z level=info msg="Executing migration" id="alter alert_definition table data column to mediumtext in mysql" 2024-03-11T20:52:37.010 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:36 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:36.867462944Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and title columns" 2024-03-11T20:52:37.010 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:36 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:36.966327156Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and uid columns" 2024-03-11T20:52:37.353 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:37 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:37.083676722Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and title columns" 2024-03-11T20:52:37.353 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:37 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:37.10066844Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and uid columns" 2024-03-11T20:52:37.353 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:37 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:37.125866967Z level=info msg="Executing migration" id="Add column paused in alert_definition" 2024-03-11T20:52:37.353 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:37 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:37.151121206Z level=info msg="Executing migration" id="drop alert_definition table" 2024-03-11T20:52:37.353 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:37 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:37.226505534Z level=info msg="Executing migration" id="delete alert_definition_version table" 2024-03-11T20:52:37.354 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:37 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:37.268447052Z level=info msg="Executing migration" id="recreate alert_definition_version table" 2024-03-11T20:52:37.354 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:37 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:37.293758874Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-03-11T20:52:37.354 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:37 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:37.319034102Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-03-11T20:52:37.354 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:37 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:37.360906408Z level=info msg="Executing migration" id="alter alert_definition_version table data column to mediumtext in mysql" 2024-03-11T20:52:37.599 INFO:teuthology.orchestra.run.smithi023.stdout:55834574944 2024-03-11T20:52:37.600 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-11T20:52:37.622 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:37 smithi146 bash[23809]: cluster 2024-03-11T20:52:35.837998+0000 mgr.y (mgr.24380) 61 : cluster 0 pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:52:37.622 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:37 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:37.403183335Z level=info msg="Executing migration" id="drop alert_definition_version table" 2024-03-11T20:52:37.622 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:37 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:37.445253114Z level=info msg="Executing migration" id="create alert_instance table" 2024-03-11T20:52:37.622 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:37 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:37.487259198Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-03-11T20:52:37.623 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:37 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:37.51255035Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, current_state columns" 2024-03-11T20:52:37.623 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:37 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:37.537784792Z level=info msg="Executing migration" id="add column current_state_end to alert_instance" 2024-03-11T20:52:37.623 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:37 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:37.563023338Z level=info msg="Executing migration" id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-03-11T20:52:37.623 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:37 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:37.605051635Z level=info msg="Executing migration" id="remove index def_org_id, current_state on alert_instance" 2024-03-11T20:52:37.623 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:37 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:37.630284323Z level=info msg="Executing migration" id="rename def_org_id to rule_org_id in alert_instance" 2024-03-11T20:52:37.638 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:37 smithi023 bash[21372]: cluster 2024-03-11T20:52:35.837998+0000 mgr.y (mgr.24380) 61 : cluster 0 pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:52:37.639 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:37 smithi023 bash[29146]: cluster 2024-03-11T20:52:35.837998+0000 mgr.y (mgr.24380) 61 : cluster 0 pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:52:37.899 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:37 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:37.688834459Z level=info msg="Executing migration" id="rename def_uid to rule_uid in alert_instance" 2024-03-11T20:52:37.899 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:37 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:37.730783053Z level=info msg="Executing migration" id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-03-11T20:52:37.899 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:37 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:37.772708598Z level=info msg="Executing migration" id="add index rule_org_id, current_state on alert_instance" 2024-03-11T20:52:37.899 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:37 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:37.798017326Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2024-03-11T20:52:37.900 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:37 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:37.839933584Z level=info msg="Executing migration" id="create alert_rule table" 2024-03-11T20:52:37.900 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:37 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:37.865183512Z level=info msg="Executing migration" id="add index in alert_rule on org_id and title columns" 2024-03-11T20:52:37.900 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:37 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:37.907099282Z level=info msg="Executing migration" id="add index in alert_rule on org_id and uid columns" 2024-03-11T20:52:38.168 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:37 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:37.93236102Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-03-11T20:52:38.168 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:37 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:37.95755274Z level=info msg="Executing migration" id="alter alert_rule table data column to mediumtext in mysql" 2024-03-11T20:52:38.168 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:37 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:37.982806545Z level=info msg="Executing migration" id="add column for to alert_rule" 2024-03-11T20:52:38.168 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:38 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:38.008059541Z level=info msg="Executing migration" id="add column annotations to alert_rule" 2024-03-11T20:52:38.168 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:38 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:38.049916858Z level=info msg="Executing migration" id="add column labels to alert_rule" 2024-03-11T20:52:38.169 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:38 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:38.092055187Z level=info msg="Executing migration" id="remove unique index from alert_rule on org_id, title columns" 2024-03-11T20:52:38.169 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:38 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:38.134065129Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-03-11T20:52:38.169 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:38 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:38.176084826Z level=info msg="Executing migration" id="add dashboard_uid column to alert_rule" 2024-03-11T20:52:38.421 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:38 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:38.218085421Z level=info msg="Executing migration" id="add panel_id column to alert_rule" 2024-03-11T20:52:38.421 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:38 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:38.260134285Z level=info msg="Executing migration" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-03-11T20:52:38.421 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:38 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:38.285409434Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2024-03-11T20:52:38.421 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:38 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:38.310604044Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2024-03-11T20:52:38.421 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:38 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:38.3524642Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2024-03-11T20:52:38.421 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:38 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:38.377729833Z level=info msg="Executing migration" id="create alert_rule_version table" 2024-03-11T20:52:38.421 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:38 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:38.428520878Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-03-11T20:52:38.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:38 smithi146 bash[23809]: cluster 2024-03-11T20:52:37.203465+0000 osd.7 (osd.7) 47 : cluster 0 3.e scrub starts 2024-03-11T20:52:38.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:38 smithi146 bash[23809]: cluster 2024-03-11T20:52:37.207648+0000 osd.7 (osd.7) 48 : cluster 0 3.e scrub ok 2024-03-11T20:52:38.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:38 smithi146 bash[23809]: cluster 2024-03-11T20:52:37.427120+0000 osd.3 (osd.3) 53 : cluster 0 6.1f scrub starts 2024-03-11T20:52:38.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:38 smithi146 bash[23809]: cluster 2024-03-11T20:52:37.429719+0000 osd.3 (osd.3) 54 : cluster 0 6.1f scrub ok 2024-03-11T20:52:38.673 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:38 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:38.470465603Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-03-11T20:52:38.673 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:38 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:38.495677113Z level=info msg="Executing migration" id="alter alert_rule_version table data column to mediumtext in mysql" 2024-03-11T20:52:38.673 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:38 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:38.520944199Z level=info msg="Executing migration" id="add column for to alert_rule_version" 2024-03-11T20:52:38.673 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:38 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:38.546146352Z level=info msg="Executing migration" id="add column annotations to alert_rule_version" 2024-03-11T20:52:38.673 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:38 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:38.588071838Z level=info msg="Executing migration" id="add column labels to alert_rule_version" 2024-03-11T20:52:38.673 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:38 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:38.613327281Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2024-03-11T20:52:38.673 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:38 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:38.65526699Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2024-03-11T20:52:38.673 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:38 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:38.680525465Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2024-03-11T20:52:38.784 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:38 smithi023 bash[29146]: cluster 2024-03-11T20:52:37.203465+0000 osd.7 (osd.7) 47 : cluster 0 3.e scrub starts 2024-03-11T20:52:38.784 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:38 smithi023 bash[29146]: cluster 2024-03-11T20:52:37.207648+0000 osd.7 (osd.7) 48 : cluster 0 3.e scrub ok 2024-03-11T20:52:38.784 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:38 smithi023 bash[29146]: cluster 2024-03-11T20:52:37.427120+0000 osd.3 (osd.3) 53 : cluster 0 6.1f scrub starts 2024-03-11T20:52:38.784 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:38 smithi023 bash[29146]: cluster 2024-03-11T20:52:37.429719+0000 osd.3 (osd.3) 54 : cluster 0 6.1f scrub ok 2024-03-11T20:52:38.784 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:52:38 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:20:52:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T20:52:38.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:38 smithi023 bash[21372]: cluster 2024-03-11T20:52:37.203465+0000 osd.7 (osd.7) 47 : cluster 0 3.e scrub starts 2024-03-11T20:52:38.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:38 smithi023 bash[21372]: cluster 2024-03-11T20:52:37.207648+0000 osd.7 (osd.7) 48 : cluster 0 3.e scrub ok 2024-03-11T20:52:38.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:38 smithi023 bash[21372]: cluster 2024-03-11T20:52:37.427120+0000 osd.3 (osd.3) 53 : cluster 0 6.1f scrub starts 2024-03-11T20:52:38.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:38 smithi023 bash[21372]: cluster 2024-03-11T20:52:37.429719+0000 osd.3 (osd.3) 54 : cluster 0 6.1f scrub ok 2024-03-11T20:52:38.824 INFO:teuthology.orchestra.run.smithi023.stdout:150323855427 2024-03-11T20:52:38.824 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-11T20:52:38.933 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:38 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:38.705631617Z level=info msg="Executing migration" id=create_alert_configuration_table 2024-03-11T20:52:38.933 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:38 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:38.730978479Z level=info msg="Executing migration" id="Add column default in alert_configuration" 2024-03-11T20:52:38.933 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:38 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:38.756219758Z level=info msg="Executing migration" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-03-11T20:52:38.933 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:38 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:38.798191906Z level=info msg="Executing migration" id="add column org_id in alert_configuration" 2024-03-11T20:52:38.933 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:38 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:38.823496405Z level=info msg="Executing migration" id="add index in alert_configuration table on org_id column" 2024-03-11T20:52:38.934 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:38 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:38.865392541Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2024-03-11T20:52:38.934 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:38 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:38.890644634Z level=info msg="Executing migration" id=create_ngalert_configuration_table 2024-03-11T20:52:38.934 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:38 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:38.915842597Z level=info msg="Executing migration" id="add index in ngalert_configuration on org_id column" 2024-03-11T20:52:38.934 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:38 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:38.941138417Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2024-03-11T20:52:39.194 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:38 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:38.967413069Z level=info msg="Executing migration" id="create provenance_type table" 2024-03-11T20:52:39.194 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:38 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:38.991669101Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2024-03-11T20:52:39.194 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:39 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:39.016790859Z level=info msg="Executing migration" id="create alert_image table" 2024-03-11T20:52:39.194 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:39 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:39.042097182Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2024-03-11T20:52:39.194 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:39 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:39.067334765Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2024-03-11T20:52:39.194 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:39 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:39.109333847Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2024-03-11T20:52:39.194 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:39 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:39.134619697Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2024-03-11T20:52:39.195 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:39 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:39.159790591Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2024-03-11T20:52:39.195 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:39 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:39.160087867Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2024-03-11T20:52:39.195 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:39 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:39.201709867Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2024-03-11T20:52:39.270 INFO:teuthology.orchestra.run.smithi023.stdout:292057776164 2024-03-11T20:52:39.271 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-03-11T20:52:39.446 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:39 smithi146 bash[23809]: cluster 2024-03-11T20:52:37.838581+0000 mgr.y (mgr.24380) 62 : cluster 0 pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-11T20:52:39.447 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:39 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:39.243694729Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2024-03-11T20:52:39.447 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:39 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:39.268914033Z level=info msg="Executing migration" id="clear migration entry \"remove unified alerting data\"" 2024-03-11T20:52:39.447 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:39 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:39.269373974Z level=info msg="Executing migration" id="move dashboard alerts to unified alerting" 2024-03-11T20:52:39.447 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:39 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:39.269774451Z level=info msg="alerts found to migrate" alerts=0 2024-03-11T20:52:39.447 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:39 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:39.294238668Z level=info msg="Executing migration" id="create library_element table v1" 2024-03-11T20:52:39.447 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:39 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:39.319490359Z level=info msg="Executing migration" id="add index library_element org_id-folder_id-name-kind" 2024-03-11T20:52:39.447 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:39 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:39.361575886Z level=info msg="Executing migration" id="create library_element_connection table v1" 2024-03-11T20:52:39.447 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:39 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:39.386986318Z level=info msg="Executing migration" id="add index library_element_connection element_id-kind-connection_id" 2024-03-11T20:52:39.447 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:39 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:39.429093058Z level=info msg="Executing migration" id="add unique index library_element org_id_uid" 2024-03-11T20:52:39.447 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:39 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:39.454372498Z level=info msg="Executing migration" id="increase max description length to 2048" 2024-03-11T20:52:39.534 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:39 smithi023 bash[29146]: cluster 2024-03-11T20:52:37.838581+0000 mgr.y (mgr.24380) 62 : cluster 0 pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-11T20:52:39.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:39 smithi023 bash[21372]: cluster 2024-03-11T20:52:37.838581+0000 mgr.y (mgr.24380) 62 : cluster 0 pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-11T20:52:39.715 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:39 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:39.47946613Z level=info msg="Executing migration" id="clone move dashboard alerts to unified alerting" 2024-03-11T20:52:39.715 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:39 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:39.504748645Z level=info msg="Executing migration" id="create data_keys table" 2024-03-11T20:52:39.715 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:39 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:39.530021095Z level=info msg="Executing migration" id="create secrets table" 2024-03-11T20:52:39.716 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:39 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:39.572006367Z level=info msg="Executing migration" id="rename data_keys name column to id" 2024-03-11T20:52:39.716 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:39 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:39.605634013Z level=info msg="Executing migration" id="add name column into data_keys" 2024-03-11T20:52:39.716 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:39 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:39.630905806Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2024-03-11T20:52:39.716 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:39 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:39.681218048Z level=info msg="Executing migration" id="rename data_keys name column to label" 2024-03-11T20:52:39.716 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:39 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:39.723139452Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2024-03-11T20:52:39.968 INFO:teuthology.orchestra.run.smithi023.stdout:244813135918 2024-03-11T20:52:39.969 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-03-11T20:52:39.984 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:39 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:39.765096508Z level=info msg="Executing migration" id="create kv_store table v1" 2024-03-11T20:52:39.984 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:39 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:39.807095916Z level=info msg="Executing migration" id="add index kv_store.org_id-namespace-key" 2024-03-11T20:52:39.985 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:39 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:39.849137634Z level=info msg="Executing migration" id="update dashboard_uid and panel_id from existing annotations" 2024-03-11T20:52:39.985 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:39 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:39.866063474Z level=info msg="Executing migration" id="create permission table" 2024-03-11T20:52:39.985 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:39 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:39.899659691Z level=info msg="Executing migration" id="add unique index permission.role_id" 2024-03-11T20:52:39.985 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:39 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:39.916634043Z level=info msg="Executing migration" id="add unique index role_id_action_scope" 2024-03-11T20:52:39.985 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:39 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:39.941826204Z level=info msg="Executing migration" id="create role table" 2024-03-11T20:52:39.985 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:39 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:39.967099033Z level=info msg="Executing migration" id="add column display_name" 2024-03-11T20:52:39.985 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:39 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:39.992302347Z level=info msg="Executing migration" id="add column group_name" 2024-03-11T20:52:40.109 INFO:teuthology.orchestra.run.smithi023.stdout:120259084364 2024-03-11T20:52:40.110 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-11T20:52:40.172 INFO:teuthology.orchestra.run.smithi023.stdout:81604378711 2024-03-11T20:52:40.173 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-11T20:52:40.260 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:40 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:40.017443379Z level=info msg="Executing migration" id="add index role.org_id" 2024-03-11T20:52:40.260 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:40 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:40.042816822Z level=info msg="Executing migration" id="add unique index role_org_id_name" 2024-03-11T20:52:40.261 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:40 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:40.068015666Z level=info msg="Executing migration" id="add index role_org_id_uid" 2024-03-11T20:52:40.261 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:40 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:40.093337488Z level=info msg="Executing migration" id="create team role table" 2024-03-11T20:52:40.261 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:40 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:40.118352839Z level=info msg="Executing migration" id="add index team_role.org_id" 2024-03-11T20:52:40.261 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:40 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:40.14361097Z level=info msg="Executing migration" id="add unique index team_role_org_id_team_id_role_id" 2024-03-11T20:52:40.261 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:40 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:40.185448927Z level=info msg="Executing migration" id="add index team_role.team_id" 2024-03-11T20:52:40.261 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:40 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:40.227628692Z level=info msg="Executing migration" id="create user role table" 2024-03-11T20:52:40.531 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:40 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:40.278282427Z level=info msg="Executing migration" id="add index user_role.org_id" 2024-03-11T20:52:40.531 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:40 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:40.303690271Z level=info msg="Executing migration" id="add unique index user_role_org_id_user_id_role_id" 2024-03-11T20:52:40.531 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:40 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:40.345545876Z level=info msg="Executing migration" id="add index user_role.user_id" 2024-03-11T20:52:40.532 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:40 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:40.370846676Z level=info msg="Executing migration" id="create builtin role table" 2024-03-11T20:52:40.532 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:40 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:40.39611639Z level=info msg="Executing migration" id="add index builtin_role.role_id" 2024-03-11T20:52:40.532 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:40 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:40.43805439Z level=info msg="Executing migration" id="add index builtin_role.name" 2024-03-11T20:52:40.532 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:40 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:40.463371609Z level=info msg="Executing migration" id="Add column org_id to builtin_role table" 2024-03-11T20:52:40.532 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:40 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:40.505343719Z level=info msg="Executing migration" id="add index builtin_role.org_id" 2024-03-11T20:52:40.532 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:40 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:40.522237056Z level=info msg="Executing migration" id="add unique index builtin_role_org_id_role_id_role" 2024-03-11T20:52:40.532 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:40 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:40.539105194Z level=info msg="Executing migration" id="Remove unique index role_org_id_uid" 2024-03-11T20:52:40.784 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:40 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:40.556013647Z level=info msg="Executing migration" id="add unique index role.uid" 2024-03-11T20:52:40.784 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:40 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:40.572907888Z level=info msg="Executing migration" id="create seed assignment table" 2024-03-11T20:52:40.784 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:40 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:40.606553175Z level=info msg="Executing migration" id="add unique index builtin_role_role_name" 2024-03-11T20:52:40.784 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:40 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:40.62355093Z level=info msg="Executing migration" id="add column hidden to role table" 2024-03-11T20:52:40.784 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:40 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:40.657531599Z level=info msg="Executing migration" id="create query_history table v1" 2024-03-11T20:52:40.784 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:40 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:40.699244545Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2024-03-11T20:52:40.784 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:40 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:40.724544282Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2024-03-11T20:52:40.784 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:40 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:40.791419509Z level=info msg="Executing migration" id="teams permissions migration" 2024-03-11T20:52:41.045 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:40 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:40.808392799Z level=info msg="Executing migration" id="dashboard permissions" 2024-03-11T20:52:41.045 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:40 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:40.825336886Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2024-03-11T20:52:41.045 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:40 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:40.858898926Z level=info msg="Executing migration" id="drop managed folder create actions" 2024-03-11T20:52:41.045 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:40 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:40.875851863Z level=info msg="Executing migration" id="alerting notification permissions" 2024-03-11T20:52:41.045 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:40 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:40.909375134Z level=info msg="Executing migration" id="create query_history_star table v1" 2024-03-11T20:52:41.045 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:40 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:40.926527128Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2024-03-11T20:52:41.045 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:40 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:40.951749308Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2024-03-11T20:52:41.045 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:40 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:40.97696032Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2024-03-11T20:52:41.045 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:41 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:41.002210243Z level=info msg="Executing migration" id="create correlation table v1" 2024-03-11T20:52:41.045 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:41 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:41.027470994Z level=info msg="Executing migration" id="add index correlations.uid" 2024-03-11T20:52:41.046 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:41 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:41.052675258Z level=info msg="Executing migration" id="add index correlations.source_uid" 2024-03-11T20:52:41.297 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:41 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:41.07789323Z level=info msg="Executing migration" id="add correlation config column" 2024-03-11T20:52:41.297 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:41 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:41.103225246Z level=info msg="Executing migration" id="create entity_events table" 2024-03-11T20:52:41.297 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:41 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:41.137073767Z level=info msg="Executing migration" id="create dashboard public config v1" 2024-03-11T20:52:41.297 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:41 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:41.179028354Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2024-03-11T20:52:41.297 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:41 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:41.179410873Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2024-03-11T20:52:41.298 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:41 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:41.220970406Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-11T20:52:41.298 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:41 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:41.221449416Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-11T20:52:41.298 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:41 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:41.263051358Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2024-03-11T20:52:41.574 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:41 smithi146 bash[23809]: cluster 2024-03-11T20:52:39.840019+0000 mgr.y (mgr.24380) 63 : cluster 0 pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:52:41.574 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:41 smithi146 bash[23809]: cluster 2024-03-11T20:52:40.429489+0000 osd.3 (osd.3) 55 : cluster 0 3.8 scrub starts 2024-03-11T20:52:41.574 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:41 smithi146 bash[23809]: cluster 2024-03-11T20:52:40.434156+0000 osd.3 (osd.3) 56 : cluster 0 3.8 scrub ok 2024-03-11T20:52:41.575 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:41 smithi146 bash[23809]: audit 2024-03-11T20:52:41.199249+0000 mon.a (mon.0) 873 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:52:41.575 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:41 smithi146 bash[23809]: audit 2024-03-11T20:52:41.211442+0000 mon.a (mon.0) 874 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:52:41.575 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:41 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:41.305022899Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2024-03-11T20:52:41.575 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:41 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:41.346976183Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2024-03-11T20:52:41.575 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:41 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:41.388905274Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-11T20:52:41.575 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:41 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:41.431012416Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2024-03-11T20:52:41.575 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:41 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:41.456207828Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-03-11T20:52:41.575 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:41 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:41.481493911Z level=info msg="Executing migration" id="Drop public config table" 2024-03-11T20:52:41.575 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:41 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:41.581927035Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2024-03-11T20:52:41.784 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:41 smithi023 bash[21372]: cluster 2024-03-11T20:52:39.840019+0000 mgr.y (mgr.24380) 63 : cluster 0 pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:52:41.784 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:41 smithi023 bash[21372]: cluster 2024-03-11T20:52:40.429489+0000 osd.3 (osd.3) 55 : cluster 0 3.8 scrub starts 2024-03-11T20:52:41.784 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:41 smithi023 bash[21372]: cluster 2024-03-11T20:52:40.434156+0000 osd.3 (osd.3) 56 : cluster 0 3.8 scrub ok 2024-03-11T20:52:41.784 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:41 smithi023 bash[21372]: audit 2024-03-11T20:52:41.199249+0000 mon.a (mon.0) 873 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:52:41.784 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:41 smithi023 bash[21372]: audit 2024-03-11T20:52:41.211442+0000 mon.a (mon.0) 874 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:52:41.784 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:41 smithi023 bash[29146]: cluster 2024-03-11T20:52:39.840019+0000 mgr.y (mgr.24380) 63 : cluster 0 pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:52:41.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:41 smithi023 bash[29146]: cluster 2024-03-11T20:52:40.429489+0000 osd.3 (osd.3) 55 : cluster 0 3.8 scrub starts 2024-03-11T20:52:41.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:41 smithi023 bash[29146]: cluster 2024-03-11T20:52:40.434156+0000 osd.3 (osd.3) 56 : cluster 0 3.8 scrub ok 2024-03-11T20:52:41.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:41 smithi023 bash[29146]: audit 2024-03-11T20:52:41.199249+0000 mon.a (mon.0) 873 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:52:41.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:41 smithi023 bash[29146]: audit 2024-03-11T20:52:41.211442+0000 mon.a (mon.0) 874 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:52:41.826 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:41 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:41.607306474Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2024-03-11T20:52:41.827 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:41 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:41.632550069Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-03-11T20:52:41.827 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:41 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:41.674528883Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2024-03-11T20:52:41.827 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:41 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:41.699791007Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2024-03-11T20:52:41.827 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:41 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:41.733372678Z level=info msg="Executing migration" id="add annotations_enabled column" 2024-03-11T20:52:41.827 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:41 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:41.75853551Z level=info msg="Executing migration" id="add time_selection_enabled column" 2024-03-11T20:52:41.827 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:41 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:41.792055901Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2024-03-11T20:52:41.827 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:41 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:41.834120891Z level=info msg="Executing migration" id="add share column" 2024-03-11T20:52:42.088 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:41 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:41.859335082Z level=info msg="Executing migration" id="create default alerting folders" 2024-03-11T20:52:42.088 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:41 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:41.884671072Z level=info msg="Executing migration" id="create file table" 2024-03-11T20:52:42.088 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:41 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:41.926627049Z level=info msg="Executing migration" id="file table idx: path natural pk" 2024-03-11T20:52:42.088 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:41 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:41.95190813Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2024-03-11T20:52:42.088 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:41 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:41.977156381Z level=info msg="Executing migration" id="create file_meta table" 2024-03-11T20:52:42.088 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:41 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:41.994051108Z level=info msg="Executing migration" id="file table idx: path key" 2024-03-11T20:52:42.088 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:42 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:42.010923873Z level=info msg="Executing migration" id="set path collation in file table" 2024-03-11T20:52:42.088 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:42 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:42.027778501Z level=info msg="Executing migration" id="managed permissions migration" 2024-03-11T20:52:42.088 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:42 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:42.044700905Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2024-03-11T20:52:42.088 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:42 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:42.070185404Z level=info msg="Executing migration" id="RBAC action name migrator" 2024-03-11T20:52:42.088 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:42 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:42.095427714Z level=info msg="Executing migration" id="Add UID column to playlist" 2024-03-11T20:52:42.340 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:42 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:42.120700029Z level=info msg="Executing migration" id="Update uid column values in playlist" 2024-03-11T20:52:42.340 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:42 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:42.145949537Z level=info msg="Executing migration" id="Add index for uid in playlist" 2024-03-11T20:52:42.340 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:42 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:42.187849916Z level=info msg="Executing migration" id="update group index for alert rules" 2024-03-11T20:52:42.340 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:42 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:42.213229982Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2024-03-11T20:52:42.340 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:42 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:42.23848189Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2024-03-11T20:52:42.340 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:42 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:42.280397575Z level=info msg="Executing migration" id="add action column to seed_assignment" 2024-03-11T20:52:42.340 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:42 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:42.305548885Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2024-03-11T20:52:42.340 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:42 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:42.347611832Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2024-03-11T20:52:42.600 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:42 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:42.389661882Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2024-03-11T20:52:42.600 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:42 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:42.456746506Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2024-03-11T20:52:42.600 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:42 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:42.498732788Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2024-03-11T20:52:42.600 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:42 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:42.52392562Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2024-03-11T20:52:42.600 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:42 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:42.565918981Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2024-03-11T20:52:42.600 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:42 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:42.607803107Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2024-03-11T20:52:42.601 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:42 smithi146 bash[23809]: audit 2024-03-11T20:52:41.782052+0000 mon.a (mon.0) 875 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:52:42.601 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:42 smithi146 bash[23809]: audit 2024-03-11T20:52:41.783785+0000 mon.a (mon.0) 876 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:52:42.601 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:42 smithi146 bash[23809]: audit 2024-03-11T20:52:41.801085+0000 mon.a (mon.0) 877 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:52:42.724 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:42 smithi023 bash[21372]: audit 2024-03-11T20:52:41.782052+0000 mon.a (mon.0) 875 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:52:42.725 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:42 smithi023 bash[21372]: audit 2024-03-11T20:52:41.783785+0000 mon.a (mon.0) 876 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:52:42.725 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:42 smithi023 bash[21372]: audit 2024-03-11T20:52:41.801085+0000 mon.a (mon.0) 877 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:52:42.725 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:42 smithi023 bash[29146]: audit 2024-03-11T20:52:41.782052+0000 mon.a (mon.0) 875 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:52:42.725 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:42 smithi023 bash[29146]: audit 2024-03-11T20:52:41.783785+0000 mon.a (mon.0) 876 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:52:42.725 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:42 smithi023 bash[29146]: audit 2024-03-11T20:52:41.801085+0000 mon.a (mon.0) 877 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:52:42.980 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:42 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:42.649781791Z level=info msg="Executing migration" id="create folder table" 2024-03-11T20:52:42.980 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:42 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:42.691727353Z level=info msg="Executing migration" id="Add index for parent_uid" 2024-03-11T20:52:42.980 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:42 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:42.733707714Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2024-03-11T20:52:42.980 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:42 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:42.759105254Z level=info msg="Executing migration" id="Update folder title length" 2024-03-11T20:52:42.980 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:42 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:42.784289482Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2024-03-11T20:52:42.980 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:42 smithi146 bash[51661]: logger=migrator t=2024-03-11T20:52:42.826100413Z level=info msg="migrations completed" performed=485 skipped=0 duration=15.729863101s 2024-03-11T20:52:42.980 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:42 smithi146 bash[51661]: logger=sqlstore t=2024-03-11T20:52:42.827566823Z level=info msg="Created default organization" 2024-03-11T20:52:42.980 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:42 smithi146 bash[51661]: logger=plugin.loader t=2024-03-11T20:52:42.987474482Z level=info msg="Plugin registered" pluginID=input 2024-03-11T20:52:43.187 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:52:43.187 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:52:43.187 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:52:43.188 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:52:43.188 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:52:43.260 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:43 smithi146 bash[51661]: logger=plugin.loader t=2024-03-11T20:52:43.017351133Z level=info msg="Plugin registered" pluginID=grafana-piechart-panel 2024-03-11T20:52:43.260 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:43 smithi146 bash[51661]: logger=plugin.loader t=2024-03-11T20:52:43.01737343Z level=info msg="Plugin registered" pluginID=vonage-status-panel 2024-03-11T20:52:43.260 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:43 smithi146 bash[51661]: logger=secrets t=2024-03-11T20:52:43.017412042Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-03-11T20:52:43.261 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:43 smithi146 bash[51661]: logger=query_data t=2024-03-11T20:52:43.019900855Z level=info msg="Query Service initialization" 2024-03-11T20:52:43.261 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:43 smithi146 bash[51661]: logger=live.push_http t=2024-03-11T20:52:43.024407211Z level=info msg="Live Push Gateway initialization" 2024-03-11T20:52:43.687 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:43 smithi023 bash[29146]: cluster 2024-03-11T20:52:41.840711+0000 mgr.y (mgr.24380) 64 : cluster 0 pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 974 B/s rd, 0 op/s 2024-03-11T20:52:43.687 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:43 smithi023 bash[29146]: cephadm 2024-03-11T20:52:41.877804+0000 mgr.y (mgr.24380) 65 : cephadm 1 Reconfiguring alertmanager.a (dependencies changed)... 2024-03-11T20:52:43.687 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:43 smithi023 bash[29146]: cephadm 2024-03-11T20:52:41.886394+0000 mgr.y (mgr.24380) 66 : cephadm 1 Reconfiguring daemon alertmanager.a on smithi023 2024-03-11T20:52:43.687 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:43 smithi023 bash[29146]: cluster 2024-03-11T20:52:42.467391+0000 osd.3 (osd.3) 57 : cluster 0 6.11 scrub starts 2024-03-11T20:52:43.687 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:43 smithi023 bash[29146]: cluster 2024-03-11T20:52:42.470370+0000 osd.3 (osd.3) 58 : cluster 0 6.11 scrub ok 2024-03-11T20:52:43.687 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:43 smithi023 bash[29146]: cluster 2024-03-11T20:52:42.998281+0000 osd.0 (osd.0) 51 : cluster 0 purged_snaps scrub starts 2024-03-11T20:52:43.687 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:43 smithi023 bash[29146]: cluster 2024-03-11T20:52:42.998393+0000 osd.0 (osd.0) 52 : cluster 0 purged_snaps scrub ok 2024-03-11T20:52:43.687 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:43 smithi023 bash[21372]: cluster 2024-03-11T20:52:41.840711+0000 mgr.y (mgr.24380) 64 : cluster 0 pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 974 B/s rd, 0 op/s 2024-03-11T20:52:43.688 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:43 smithi023 bash[21372]: cephadm 2024-03-11T20:52:41.877804+0000 mgr.y (mgr.24380) 65 : cephadm 1 Reconfiguring alertmanager.a (dependencies changed)... 2024-03-11T20:52:43.688 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:43 smithi023 bash[21372]: cephadm 2024-03-11T20:52:41.886394+0000 mgr.y (mgr.24380) 66 : cephadm 1 Reconfiguring daemon alertmanager.a on smithi023 2024-03-11T20:52:43.688 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:43 smithi023 bash[21372]: cluster 2024-03-11T20:52:42.467391+0000 osd.3 (osd.3) 57 : cluster 0 6.11 scrub starts 2024-03-11T20:52:43.688 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:43 smithi023 bash[21372]: cluster 2024-03-11T20:52:42.470370+0000 osd.3 (osd.3) 58 : cluster 0 6.11 scrub ok 2024-03-11T20:52:43.688 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:43 smithi023 bash[21372]: cluster 2024-03-11T20:52:42.998281+0000 osd.0 (osd.0) 51 : cluster 0 purged_snaps scrub starts 2024-03-11T20:52:43.688 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:43 smithi023 bash[21372]: cluster 2024-03-11T20:52:42.998393+0000 osd.0 (osd.0) 52 : cluster 0 purged_snaps scrub ok 2024-03-11T20:52:43.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:43 smithi146 bash[23809]: cluster 2024-03-11T20:52:41.840711+0000 mgr.y (mgr.24380) 64 : cluster 0 pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 974 B/s rd, 0 op/s 2024-03-11T20:52:43.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:43 smithi146 bash[23809]: cephadm 2024-03-11T20:52:41.877804+0000 mgr.y (mgr.24380) 65 : cephadm 1 Reconfiguring alertmanager.a (dependencies changed)... 2024-03-11T20:52:43.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:43 smithi146 bash[23809]: cephadm 2024-03-11T20:52:41.886394+0000 mgr.y (mgr.24380) 66 : cephadm 1 Reconfiguring daemon alertmanager.a on smithi023 2024-03-11T20:52:43.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:43 smithi146 bash[23809]: cluster 2024-03-11T20:52:42.467391+0000 osd.3 (osd.3) 57 : cluster 0 6.11 scrub starts 2024-03-11T20:52:43.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:43 smithi146 bash[23809]: cluster 2024-03-11T20:52:42.470370+0000 osd.3 (osd.3) 58 : cluster 0 6.11 scrub ok 2024-03-11T20:52:43.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:43 smithi146 bash[23809]: cluster 2024-03-11T20:52:42.998281+0000 osd.0 (osd.0) 51 : cluster 0 purged_snaps scrub starts 2024-03-11T20:52:43.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:43 smithi146 bash[23809]: cluster 2024-03-11T20:52:42.998393+0000 osd.0 (osd.0) 52 : cluster 0 purged_snaps scrub ok 2024-03-11T20:52:44.203 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:52:44.203 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:52:44.203 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:52:44.253 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:43 smithi146 bash[51661]: logger=infra.usagestats.collector t=2024-03-11T20:52:43.982536496Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-03-11T20:52:44.253 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:43 smithi146 bash[51661]: logger=provisioning.datasources t=2024-03-11T20:52:43.983626053Z level=info msg="inserting datasource from configuration " name=Dashboard1 uid=P43CA22E17D0F9596 2024-03-11T20:52:44.253 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:44 smithi146 bash[51661]: logger=provisioning.datasources t=2024-03-11T20:52:44.018871565Z level=info msg="inserting datasource from configuration " name=Loki uid=P8E80F9AEF21F6940 2024-03-11T20:52:44.253 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:44 smithi146 bash[51661]: logger=provisioning.alerting t=2024-03-11T20:52:44.052709016Z level=info msg="starting to provision alerting" 2024-03-11T20:52:44.253 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:44 smithi146 bash[51661]: logger=provisioning.alerting t=2024-03-11T20:52:44.052735944Z level=info msg="finished to provision alerting" 2024-03-11T20:52:44.253 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:44 smithi146 bash[51661]: logger=grafanaStorageLogger t=2024-03-11T20:52:44.053137289Z level=info msg="storage starting" 2024-03-11T20:52:44.253 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:44 smithi146 bash[51661]: logger=ngalert.state.manager t=2024-03-11T20:52:44.053311623Z level=info msg="Warming state cache for startup" 2024-03-11T20:52:44.253 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:44 smithi146 bash[51661]: logger=http.server t=2024-03-11T20:52:44.05674512Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-03-11T20:52:44.253 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:44 smithi146 bash[51661]: logger=ngalert.state.manager t=2024-03-11T20:52:44.139926549Z level=info msg="State cache has been initialized" states=0 duration=86.608456ms 2024-03-11T20:52:44.254 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:44 smithi146 bash[51661]: logger=ticker t=2024-03-11T20:52:44.140151389Z level=info msg=starting first_tick=2024-03-11T20:52:50Z 2024-03-11T20:52:44.254 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:44 smithi146 bash[51661]: logger=ngalert.multiorg.alertmanager t=2024-03-11T20:52:44.140077263Z level=info msg="starting MultiOrg Alertmanager" 2024-03-11T20:52:44.254 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:44 smithi146 bash[51661]: logger=sqlstore.transactions t=2024-03-11T20:52:44.14947828Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-03-11T20:52:44.254 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:44 smithi146 bash[51661]: logger=sqlstore.transactions t=2024-03-11T20:52:44.152703891Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-03-11T20:52:44.254 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:44 smithi146 bash[51661]: logger=sqlstore.transactions t=2024-03-11T20:52:44.163548962Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=1 code="database is locked" 2024-03-11T20:52:44.254 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:44 smithi146 bash[51661]: logger=sqlstore.transactions t=2024-03-11T20:52:44.215007461Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-03-11T20:52:44.254 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:44 smithi146 bash[51661]: logger=sqlstore.transactions t=2024-03-11T20:52:44.260402322Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=1 code="database is locked" 2024-03-11T20:52:45.010 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 20:52:44 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T20:52:45.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:44 smithi146 bash[23809]: cluster 2024-03-11T20:52:43.818898+0000 mon.a (mon.0) 878 : cluster 0 osdmap e109: 8 total, 8 up, 8 in 2024-03-11T20:52:45.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:44 smithi146 bash[23809]: cluster 2024-03-11T20:52:43.841397+0000 mgr.y (mgr.24380) 67 : cluster 0 pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:52:45.011 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:44 smithi146 bash[51661]: logger=sqlstore.transactions t=2024-03-11T20:52:44.703846022Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=2 code="database is locked" 2024-03-11T20:52:45.011 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:44 smithi146 bash[51661]: logger=sqlstore.transactions t=2024-03-11T20:52:44.714637811Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=3 code="database is locked" 2024-03-11T20:52:45.011 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 20:52:44 smithi146 bash[51661]: logger=sqlstore.transactions t=2024-03-11T20:52:44.725390093Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=4 code="database is locked" 2024-03-11T20:52:45.210 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:44 smithi023 bash[21372]: cluster 2024-03-11T20:52:43.818898+0000 mon.a (mon.0) 878 : cluster 0 osdmap e109: 8 total, 8 up, 8 in 2024-03-11T20:52:45.210 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:44 smithi023 bash[21372]: cluster 2024-03-11T20:52:43.841397+0000 mgr.y (mgr.24380) 67 : cluster 0 pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:52:45.210 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:44 smithi023 bash[29146]: cluster 2024-03-11T20:52:43.818898+0000 mon.a (mon.0) 878 : cluster 0 osdmap e109: 8 total, 8 up, 8 in 2024-03-11T20:52:45.210 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:44 smithi023 bash[29146]: cluster 2024-03-11T20:52:43.841397+0000 mgr.y (mgr.24380) 67 : cluster 0 pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:52:46.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:45 smithi023 bash[21372]: audit 2024-03-11T20:52:44.877894+0000 mgr.y (mgr.24380) 68 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:52:46.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:45 smithi023 bash[21372]: cluster 2024-03-11T20:52:44.916805+0000 osd.5 (osd.5) 45 : cluster 0 2.3 scrub starts 2024-03-11T20:52:46.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:45 smithi023 bash[21372]: cluster 2024-03-11T20:52:44.918926+0000 osd.5 (osd.5) 46 : cluster 0 2.3 scrub ok 2024-03-11T20:52:46.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:45 smithi023 bash[29146]: audit 2024-03-11T20:52:44.877894+0000 mgr.y (mgr.24380) 68 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:52:46.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:45 smithi023 bash[29146]: cluster 2024-03-11T20:52:44.916805+0000 osd.5 (osd.5) 45 : cluster 0 2.3 scrub starts 2024-03-11T20:52:46.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:45 smithi023 bash[29146]: cluster 2024-03-11T20:52:44.918926+0000 osd.5 (osd.5) 46 : cluster 0 2.3 scrub ok 2024-03-11T20:52:46.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:45 smithi146 bash[23809]: audit 2024-03-11T20:52:44.877894+0000 mgr.y (mgr.24380) 68 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:52:46.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:45 smithi146 bash[23809]: cluster 2024-03-11T20:52:44.916805+0000 osd.5 (osd.5) 45 : cluster 0 2.3 scrub starts 2024-03-11T20:52:46.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:45 smithi146 bash[23809]: cluster 2024-03-11T20:52:44.918926+0000 osd.5 (osd.5) 46 : cluster 0 2.3 scrub ok 2024-03-11T20:52:47.146 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:46 smithi023 bash[21372]: cluster 2024-03-11T20:52:45.842839+0000 mgr.y (mgr.24380) 69 : cluster 0 pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:52:47.147 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:46 smithi023 bash[29146]: cluster 2024-03-11T20:52:45.842839+0000 mgr.y (mgr.24380) 69 : cluster 0 pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:52:47.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:46 smithi146 bash[23809]: cluster 2024-03-11T20:52:45.842839+0000 mgr.y (mgr.24380) 69 : cluster 0 pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:52:47.970 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:47 smithi023 bash[21372]: cluster 2024-03-11T20:52:46.944701+0000 osd.0 (osd.0) 53 : cluster 0 6.9 scrub starts 2024-03-11T20:52:47.971 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:47 smithi023 bash[21372]: cluster 2024-03-11T20:52:46.946554+0000 osd.0 (osd.0) 54 : cluster 0 6.9 scrub ok 2024-03-11T20:52:47.971 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:47 smithi023 bash[29146]: cluster 2024-03-11T20:52:46.944701+0000 osd.0 (osd.0) 53 : cluster 0 6.9 scrub starts 2024-03-11T20:52:47.971 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:47 smithi023 bash[29146]: cluster 2024-03-11T20:52:46.946554+0000 osd.0 (osd.0) 54 : cluster 0 6.9 scrub ok 2024-03-11T20:52:48.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:47 smithi146 bash[23809]: cluster 2024-03-11T20:52:46.944701+0000 osd.0 (osd.0) 53 : cluster 0 6.9 scrub starts 2024-03-11T20:52:48.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:47 smithi146 bash[23809]: cluster 2024-03-11T20:52:46.946554+0000 osd.0 (osd.0) 54 : cluster 0 6.9 scrub ok 2024-03-11T20:52:48.506 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:52:48 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:20:52:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T20:52:49.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:48 smithi146 bash[23809]: cluster 2024-03-11T20:52:47.202575+0000 osd.7 (osd.7) 49 : cluster 0 3.d scrub starts 2024-03-11T20:52:49.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:48 smithi146 bash[23809]: cluster 2024-03-11T20:52:47.207696+0000 osd.7 (osd.7) 50 : cluster 0 3.d scrub ok 2024-03-11T20:52:49.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:48 smithi146 bash[23809]: cluster 2024-03-11T20:52:47.843524+0000 mgr.y (mgr.24380) 70 : cluster 0 pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 716 B/s rd, 0 op/s 2024-03-11T20:52:49.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:48 smithi146 bash[23809]: cluster 2024-03-11T20:52:47.931056+0000 osd.0 (osd.0) 55 : cluster 0 2.13 scrub starts 2024-03-11T20:52:49.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:48 smithi146 bash[23809]: cluster 2024-03-11T20:52:47.933308+0000 osd.0 (osd.0) 56 : cluster 0 2.13 scrub ok 2024-03-11T20:52:49.278 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:48 smithi023 bash[21372]: cluster 2024-03-11T20:52:47.202575+0000 osd.7 (osd.7) 49 : cluster 0 3.d scrub starts 2024-03-11T20:52:49.278 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:48 smithi023 bash[21372]: cluster 2024-03-11T20:52:47.207696+0000 osd.7 (osd.7) 50 : cluster 0 3.d scrub ok 2024-03-11T20:52:49.278 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:48 smithi023 bash[21372]: cluster 2024-03-11T20:52:47.843524+0000 mgr.y (mgr.24380) 70 : cluster 0 pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 716 B/s rd, 0 op/s 2024-03-11T20:52:49.278 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:48 smithi023 bash[21372]: cluster 2024-03-11T20:52:47.931056+0000 osd.0 (osd.0) 55 : cluster 0 2.13 scrub starts 2024-03-11T20:52:49.278 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:48 smithi023 bash[21372]: cluster 2024-03-11T20:52:47.933308+0000 osd.0 (osd.0) 56 : cluster 0 2.13 scrub ok 2024-03-11T20:52:49.278 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:48 smithi023 bash[29146]: cluster 2024-03-11T20:52:47.202575+0000 osd.7 (osd.7) 49 : cluster 0 3.d scrub starts 2024-03-11T20:52:49.278 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:48 smithi023 bash[29146]: cluster 2024-03-11T20:52:47.207696+0000 osd.7 (osd.7) 50 : cluster 0 3.d scrub ok 2024-03-11T20:52:49.278 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:48 smithi023 bash[29146]: cluster 2024-03-11T20:52:47.843524+0000 mgr.y (mgr.24380) 70 : cluster 0 pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 716 B/s rd, 0 op/s 2024-03-11T20:52:49.278 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:48 smithi023 bash[29146]: cluster 2024-03-11T20:52:47.931056+0000 osd.0 (osd.0) 55 : cluster 0 2.13 scrub starts 2024-03-11T20:52:49.279 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:48 smithi023 bash[29146]: cluster 2024-03-11T20:52:47.933308+0000 osd.0 (osd.0) 56 : cluster 0 2.13 scrub ok 2024-03-11T20:52:50.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:49 smithi146 bash[23809]: cluster 2024-03-11T20:52:48.161708+0000 osd.1 (osd.1) 49 : cluster 0 3.4 scrub starts 2024-03-11T20:52:50.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:49 smithi146 bash[23809]: cluster 2024-03-11T20:52:48.166882+0000 osd.1 (osd.1) 50 : cluster 0 3.4 scrub ok 2024-03-11T20:52:50.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:49 smithi146 bash[23809]: audit 2024-03-11T20:52:49.069870+0000 mon.a (mon.0) 879 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:52:50.267 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:49 smithi023 bash[21372]: cluster 2024-03-11T20:52:48.161708+0000 osd.1 (osd.1) 49 : cluster 0 3.4 scrub starts 2024-03-11T20:52:50.267 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:49 smithi023 bash[21372]: cluster 2024-03-11T20:52:48.166882+0000 osd.1 (osd.1) 50 : cluster 0 3.4 scrub ok 2024-03-11T20:52:50.267 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:49 smithi023 bash[21372]: audit 2024-03-11T20:52:49.069870+0000 mon.a (mon.0) 879 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:52:50.267 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:49 smithi023 bash[29146]: cluster 2024-03-11T20:52:48.161708+0000 osd.1 (osd.1) 49 : cluster 0 3.4 scrub starts 2024-03-11T20:52:50.268 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:49 smithi023 bash[29146]: cluster 2024-03-11T20:52:48.166882+0000 osd.1 (osd.1) 50 : cluster 0 3.4 scrub ok 2024-03-11T20:52:50.268 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:49 smithi023 bash[29146]: audit 2024-03-11T20:52:49.069870+0000 mon.a (mon.0) 879 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:52:51.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:50 smithi146 bash[23809]: cluster 2024-03-11T20:52:49.207686+0000 osd.1 (osd.1) 51 : cluster 0 3.0 deep-scrub starts 2024-03-11T20:52:51.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:50 smithi146 bash[23809]: cluster 2024-03-11T20:52:49.216370+0000 osd.1 (osd.1) 52 : cluster 0 3.0 deep-scrub ok 2024-03-11T20:52:51.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:50 smithi146 bash[23809]: cluster 2024-03-11T20:52:49.458851+0000 osd.4 (osd.4) 51 : cluster 0 4.18 scrub starts 2024-03-11T20:52:51.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:50 smithi146 bash[23809]: cluster 2024-03-11T20:52:49.460980+0000 osd.4 (osd.4) 52 : cluster 0 4.18 scrub ok 2024-03-11T20:52:51.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:50 smithi146 bash[23809]: cluster 2024-03-11T20:52:49.582827+0000 osd.3 (osd.3) 59 : cluster 0 purged_snaps scrub starts 2024-03-11T20:52:51.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:50 smithi146 bash[23809]: cluster 2024-03-11T20:52:49.582945+0000 osd.3 (osd.3) 60 : cluster 0 purged_snaps scrub ok 2024-03-11T20:52:51.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:50 smithi146 bash[23809]: cluster 2024-03-11T20:52:49.844958+0000 mgr.y (mgr.24380) 71 : cluster 0 pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:52:51.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:50 smithi146 bash[23809]: cluster 2024-03-11T20:52:49.867040+0000 mon.a (mon.0) 880 : cluster 0 osdmap e110: 8 total, 8 up, 8 in 2024-03-11T20:52:51.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:50 smithi023 bash[21372]: cluster 2024-03-11T20:52:49.207686+0000 osd.1 (osd.1) 51 : cluster 0 3.0 deep-scrub starts 2024-03-11T20:52:51.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:50 smithi023 bash[21372]: cluster 2024-03-11T20:52:49.216370+0000 osd.1 (osd.1) 52 : cluster 0 3.0 deep-scrub ok 2024-03-11T20:52:51.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:50 smithi023 bash[21372]: cluster 2024-03-11T20:52:49.458851+0000 osd.4 (osd.4) 51 : cluster 0 4.18 scrub starts 2024-03-11T20:52:51.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:50 smithi023 bash[21372]: cluster 2024-03-11T20:52:49.460980+0000 osd.4 (osd.4) 52 : cluster 0 4.18 scrub ok 2024-03-11T20:52:51.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:50 smithi023 bash[21372]: cluster 2024-03-11T20:52:49.582827+0000 osd.3 (osd.3) 59 : cluster 0 purged_snaps scrub starts 2024-03-11T20:52:51.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:50 smithi023 bash[21372]: cluster 2024-03-11T20:52:49.582945+0000 osd.3 (osd.3) 60 : cluster 0 purged_snaps scrub ok 2024-03-11T20:52:51.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:50 smithi023 bash[21372]: cluster 2024-03-11T20:52:49.844958+0000 mgr.y (mgr.24380) 71 : cluster 0 pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:52:51.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:50 smithi023 bash[21372]: cluster 2024-03-11T20:52:49.867040+0000 mon.a (mon.0) 880 : cluster 0 osdmap e110: 8 total, 8 up, 8 in 2024-03-11T20:52:51.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:50 smithi023 bash[29146]: cluster 2024-03-11T20:52:49.207686+0000 osd.1 (osd.1) 51 : cluster 0 3.0 deep-scrub starts 2024-03-11T20:52:51.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:50 smithi023 bash[29146]: cluster 2024-03-11T20:52:49.216370+0000 osd.1 (osd.1) 52 : cluster 0 3.0 deep-scrub ok 2024-03-11T20:52:51.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:50 smithi023 bash[29146]: cluster 2024-03-11T20:52:49.458851+0000 osd.4 (osd.4) 51 : cluster 0 4.18 scrub starts 2024-03-11T20:52:51.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:50 smithi023 bash[29146]: cluster 2024-03-11T20:52:49.460980+0000 osd.4 (osd.4) 52 : cluster 0 4.18 scrub ok 2024-03-11T20:52:51.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:50 smithi023 bash[29146]: cluster 2024-03-11T20:52:49.582827+0000 osd.3 (osd.3) 59 : cluster 0 purged_snaps scrub starts 2024-03-11T20:52:51.286 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:50 smithi023 bash[29146]: cluster 2024-03-11T20:52:49.582945+0000 osd.3 (osd.3) 60 : cluster 0 purged_snaps scrub ok 2024-03-11T20:52:51.286 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:50 smithi023 bash[29146]: cluster 2024-03-11T20:52:49.844958+0000 mgr.y (mgr.24380) 71 : cluster 0 pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:52:51.286 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:50 smithi023 bash[29146]: cluster 2024-03-11T20:52:49.867040+0000 mon.a (mon.0) 880 : cluster 0 osdmap e110: 8 total, 8 up, 8 in 2024-03-11T20:52:53.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:52 smithi146 bash[23809]: cluster 2024-03-11T20:52:51.158564+0000 osd.7 (osd.7) 51 : cluster 0 2.b deep-scrub starts 2024-03-11T20:52:53.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:52 smithi146 bash[23809]: cluster 2024-03-11T20:52:51.161266+0000 osd.7 (osd.7) 52 : cluster 0 2.b deep-scrub ok 2024-03-11T20:52:53.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:52 smithi146 bash[23809]: cluster 2024-03-11T20:52:51.432623+0000 osd.4 (osd.4) 53 : cluster 0 3.14 scrub starts 2024-03-11T20:52:53.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:52 smithi146 bash[23809]: cluster 2024-03-11T20:52:51.436741+0000 osd.4 (osd.4) 54 : cluster 0 3.14 scrub ok 2024-03-11T20:52:53.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:52 smithi146 bash[23809]: cluster 2024-03-11T20:52:51.845478+0000 mgr.y (mgr.24380) 72 : cluster 0 pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:52:53.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:52 smithi146 bash[23809]: cluster 2024-03-11T20:52:51.929521+0000 osd.0 (osd.0) 57 : cluster 0 4.b deep-scrub starts 2024-03-11T20:52:53.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:52 smithi146 bash[23809]: cluster 2024-03-11T20:52:51.931290+0000 osd.0 (osd.0) 58 : cluster 0 4.b deep-scrub ok 2024-03-11T20:52:53.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:52 smithi023 bash[21372]: cluster 2024-03-11T20:52:51.158564+0000 osd.7 (osd.7) 51 : cluster 0 2.b deep-scrub starts 2024-03-11T20:52:53.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:52 smithi023 bash[21372]: cluster 2024-03-11T20:52:51.161266+0000 osd.7 (osd.7) 52 : cluster 0 2.b deep-scrub ok 2024-03-11T20:52:53.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:52 smithi023 bash[21372]: cluster 2024-03-11T20:52:51.432623+0000 osd.4 (osd.4) 53 : cluster 0 3.14 scrub starts 2024-03-11T20:52:53.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:52 smithi023 bash[21372]: cluster 2024-03-11T20:52:51.436741+0000 osd.4 (osd.4) 54 : cluster 0 3.14 scrub ok 2024-03-11T20:52:53.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:52 smithi023 bash[21372]: cluster 2024-03-11T20:52:51.845478+0000 mgr.y (mgr.24380) 72 : cluster 0 pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:52:53.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:52 smithi023 bash[21372]: cluster 2024-03-11T20:52:51.929521+0000 osd.0 (osd.0) 57 : cluster 0 4.b deep-scrub starts 2024-03-11T20:52:53.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:52 smithi023 bash[21372]: cluster 2024-03-11T20:52:51.931290+0000 osd.0 (osd.0) 58 : cluster 0 4.b deep-scrub ok 2024-03-11T20:52:53.286 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:52 smithi023 bash[29146]: cluster 2024-03-11T20:52:51.158564+0000 osd.7 (osd.7) 51 : cluster 0 2.b deep-scrub starts 2024-03-11T20:52:53.286 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:52 smithi023 bash[29146]: cluster 2024-03-11T20:52:51.161266+0000 osd.7 (osd.7) 52 : cluster 0 2.b deep-scrub ok 2024-03-11T20:52:53.286 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:52 smithi023 bash[29146]: cluster 2024-03-11T20:52:51.432623+0000 osd.4 (osd.4) 53 : cluster 0 3.14 scrub starts 2024-03-11T20:52:53.286 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:52 smithi023 bash[29146]: cluster 2024-03-11T20:52:51.436741+0000 osd.4 (osd.4) 54 : cluster 0 3.14 scrub ok 2024-03-11T20:52:53.286 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:52 smithi023 bash[29146]: cluster 2024-03-11T20:52:51.845478+0000 mgr.y (mgr.24380) 72 : cluster 0 pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:52:53.287 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:52 smithi023 bash[29146]: cluster 2024-03-11T20:52:51.929521+0000 osd.0 (osd.0) 57 : cluster 0 4.b deep-scrub starts 2024-03-11T20:52:53.287 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:52 smithi023 bash[29146]: cluster 2024-03-11T20:52:51.931290+0000 osd.0 (osd.0) 58 : cluster 0 4.b deep-scrub ok 2024-03-11T20:52:54.436 INFO:teuthology.orchestra.run.smithi023.stdout:150323855430 2024-03-11T20:52:55.260 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 20:52:54 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T20:52:55.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:54 smithi146 bash[23809]: cluster 2024-03-11T20:52:53.448345+0000 osd.3 (osd.3) 61 : cluster 0 6.b deep-scrub starts 2024-03-11T20:52:55.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:54 smithi146 bash[23809]: cluster 2024-03-11T20:52:53.450874+0000 osd.3 (osd.3) 62 : cluster 0 6.b deep-scrub ok 2024-03-11T20:52:55.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:54 smithi146 bash[23809]: cluster 2024-03-11T20:52:53.803821+0000 osd.5 (osd.5) 47 : cluster 0 2.16 scrub starts 2024-03-11T20:52:55.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:54 smithi146 bash[23809]: cluster 2024-03-11T20:52:53.806134+0000 osd.5 (osd.5) 48 : cluster 0 2.16 scrub ok 2024-03-11T20:52:55.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:54 smithi146 bash[23809]: cluster 2024-03-11T20:52:53.846073+0000 mgr.y (mgr.24380) 73 : cluster 0 pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:52:55.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:54 smithi146 bash[23809]: audit 2024-03-11T20:52:54.440259+0000 mon.c (mon.2) 23 : audit 0 from='client.? 172.21.15.23:0/4213876998' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-11T20:52:55.304 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:54 smithi023 bash[29146]: cluster 2024-03-11T20:52:53.448345+0000 osd.3 (osd.3) 61 : cluster 0 6.b deep-scrub starts 2024-03-11T20:52:55.305 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:54 smithi023 bash[29146]: cluster 2024-03-11T20:52:53.450874+0000 osd.3 (osd.3) 62 : cluster 0 6.b deep-scrub ok 2024-03-11T20:52:55.305 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:54 smithi023 bash[29146]: cluster 2024-03-11T20:52:53.803821+0000 osd.5 (osd.5) 47 : cluster 0 2.16 scrub starts 2024-03-11T20:52:55.305 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:54 smithi023 bash[29146]: cluster 2024-03-11T20:52:53.806134+0000 osd.5 (osd.5) 48 : cluster 0 2.16 scrub ok 2024-03-11T20:52:55.305 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:54 smithi023 bash[29146]: cluster 2024-03-11T20:52:53.846073+0000 mgr.y (mgr.24380) 73 : cluster 0 pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:52:55.305 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:54 smithi023 bash[29146]: audit 2024-03-11T20:52:54.440259+0000 mon.c (mon.2) 23 : audit 0 from='client.? 172.21.15.23:0/4213876998' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-11T20:52:55.305 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:54 smithi023 bash[21372]: cluster 2024-03-11T20:52:53.448345+0000 osd.3 (osd.3) 61 : cluster 0 6.b deep-scrub starts 2024-03-11T20:52:55.305 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:54 smithi023 bash[21372]: cluster 2024-03-11T20:52:53.450874+0000 osd.3 (osd.3) 62 : cluster 0 6.b deep-scrub ok 2024-03-11T20:52:55.305 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:54 smithi023 bash[21372]: cluster 2024-03-11T20:52:53.803821+0000 osd.5 (osd.5) 47 : cluster 0 2.16 scrub starts 2024-03-11T20:52:55.305 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:54 smithi023 bash[21372]: cluster 2024-03-11T20:52:53.806134+0000 osd.5 (osd.5) 48 : cluster 0 2.16 scrub ok 2024-03-11T20:52:55.305 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:54 smithi023 bash[21372]: cluster 2024-03-11T20:52:53.846073+0000 mgr.y (mgr.24380) 73 : cluster 0 pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:52:55.305 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:54 smithi023 bash[21372]: audit 2024-03-11T20:52:54.440259+0000 mon.c (mon.2) 23 : audit 0 from='client.? 172.21.15.23:0/4213876998' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-11T20:52:56.035 INFO:journalctl@ceph.alertmanager.a.smithi023.stdout:Mar 11 20:52:55 smithi023 systemd[1]: Stopping Ceph alertmanager.a for 67ba00f8-dfe7-11ee-95c8-87774f69a715... 2024-03-11T20:52:56.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:55 smithi023 bash[21372]: cluster 2024-03-11T20:52:54.773110+0000 osd.6 (osd.6) 31 : cluster 0 2.14 scrub starts 2024-03-11T20:52:56.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:55 smithi023 bash[21372]: cluster 2024-03-11T20:52:54.775289+0000 osd.6 (osd.6) 32 : cluster 0 2.14 scrub ok 2024-03-11T20:52:56.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:55 smithi023 bash[21372]: cluster 2024-03-11T20:52:54.876887+0000 osd.2 (osd.2) 31 : cluster 0 6.f scrub starts 2024-03-11T20:52:56.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:55 smithi023 bash[21372]: cluster 2024-03-11T20:52:54.879282+0000 osd.2 (osd.2) 32 : cluster 0 6.f scrub ok 2024-03-11T20:52:56.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:55 smithi023 bash[21372]: audit 2024-03-11T20:52:54.885837+0000 mgr.y (mgr.24380) 74 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:52:56.036 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:55 smithi023 bash[29146]: cluster 2024-03-11T20:52:54.773110+0000 osd.6 (osd.6) 31 : cluster 0 2.14 scrub starts 2024-03-11T20:52:56.036 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:55 smithi023 bash[29146]: cluster 2024-03-11T20:52:54.775289+0000 osd.6 (osd.6) 32 : cluster 0 2.14 scrub ok 2024-03-11T20:52:56.036 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:55 smithi023 bash[29146]: cluster 2024-03-11T20:52:54.876887+0000 osd.2 (osd.2) 31 : cluster 0 6.f scrub starts 2024-03-11T20:52:56.036 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:55 smithi023 bash[29146]: cluster 2024-03-11T20:52:54.879282+0000 osd.2 (osd.2) 32 : cluster 0 6.f scrub ok 2024-03-11T20:52:56.036 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:55 smithi023 bash[29146]: audit 2024-03-11T20:52:54.885837+0000 mgr.y (mgr.24380) 74 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:52:56.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:55 smithi146 bash[23809]: cluster 2024-03-11T20:52:54.773110+0000 osd.6 (osd.6) 31 : cluster 0 2.14 scrub starts 2024-03-11T20:52:56.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:55 smithi146 bash[23809]: cluster 2024-03-11T20:52:54.775289+0000 osd.6 (osd.6) 32 : cluster 0 2.14 scrub ok 2024-03-11T20:52:56.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:55 smithi146 bash[23809]: cluster 2024-03-11T20:52:54.876887+0000 osd.2 (osd.2) 31 : cluster 0 6.f scrub starts 2024-03-11T20:52:56.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:55 smithi146 bash[23809]: cluster 2024-03-11T20:52:54.879282+0000 osd.2 (osd.2) 32 : cluster 0 6.f scrub ok 2024-03-11T20:52:56.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:55 smithi146 bash[23809]: audit 2024-03-11T20:52:54.885837+0000 mgr.y (mgr.24380) 74 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:52:57.181 INFO:teuthology.orchestra.run.smithi023.stdout:34359738480 2024-03-11T20:52:57.196 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:56 smithi023 bash[21372]: cluster 2024-03-11T20:52:55.847580+0000 mgr.y (mgr.24380) 75 : cluster 0 pgmap v53: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:52:57.197 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:56 smithi023 bash[21372]: cluster 2024-03-11T20:52:55.988742+0000 osd.0 (osd.0) 59 : cluster 0 3.12 scrub starts 2024-03-11T20:52:57.197 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:56 smithi023 bash[21372]: cluster 2024-03-11T20:52:55.992789+0000 osd.0 (osd.0) 60 : cluster 0 3.12 scrub ok 2024-03-11T20:52:57.197 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:56 smithi023 bash[29146]: cluster 2024-03-11T20:52:55.847580+0000 mgr.y (mgr.24380) 75 : cluster 0 pgmap v53: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:52:57.197 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:56 smithi023 bash[29146]: cluster 2024-03-11T20:52:55.988742+0000 osd.0 (osd.0) 59 : cluster 0 3.12 scrub starts 2024-03-11T20:52:57.197 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:56 smithi023 bash[29146]: cluster 2024-03-11T20:52:55.992789+0000 osd.0 (osd.0) 60 : cluster 0 3.12 scrub ok 2024-03-11T20:52:57.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:56 smithi146 bash[23809]: cluster 2024-03-11T20:52:55.847580+0000 mgr.y (mgr.24380) 75 : cluster 0 pgmap v53: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:52:57.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:56 smithi146 bash[23809]: cluster 2024-03-11T20:52:55.988742+0000 osd.0 (osd.0) 59 : cluster 0 3.12 scrub starts 2024-03-11T20:52:57.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:56 smithi146 bash[23809]: cluster 2024-03-11T20:52:55.992789+0000 osd.0 (osd.0) 60 : cluster 0 3.12 scrub ok 2024-03-11T20:52:57.897 INFO:journalctl@ceph.alertmanager.a.smithi023.stdout:Mar 11 20:52:57 smithi023 bash[57975]: ts=2024-03-11T20:52:57.717Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-03-11T20:52:58.257 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:57 smithi023 bash[29146]: cluster 2024-03-11T20:52:56.890186+0000 osd.2 (osd.2) 33 : cluster 0 2.0 scrub starts 2024-03-11T20:52:58.257 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:57 smithi023 bash[29146]: cluster 2024-03-11T20:52:56.891405+0000 osd.2 (osd.2) 34 : cluster 0 2.0 scrub ok 2024-03-11T20:52:58.257 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:57 smithi023 bash[29146]: audit 2024-03-11T20:52:57.185659+0000 mon.a (mon.0) 881 : audit 0 from='client.? 172.21.15.23:0/4172899253' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-11T20:52:58.257 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:57 smithi023 bash[21372]: cluster 2024-03-11T20:52:56.890186+0000 osd.2 (osd.2) 33 : cluster 0 2.0 scrub starts 2024-03-11T20:52:58.257 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:57 smithi023 bash[21372]: cluster 2024-03-11T20:52:56.891405+0000 osd.2 (osd.2) 34 : cluster 0 2.0 scrub ok 2024-03-11T20:52:58.257 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:57 smithi023 bash[21372]: audit 2024-03-11T20:52:57.185659+0000 mon.a (mon.0) 881 : audit 0 from='client.? 172.21.15.23:0/4172899253' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-11T20:52:58.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:57 smithi146 bash[23809]: cluster 2024-03-11T20:52:56.890186+0000 osd.2 (osd.2) 33 : cluster 0 2.0 scrub starts 2024-03-11T20:52:58.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:57 smithi146 bash[23809]: cluster 2024-03-11T20:52:56.891405+0000 osd.2 (osd.2) 34 : cluster 0 2.0 scrub ok 2024-03-11T20:52:58.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:57 smithi146 bash[23809]: audit 2024-03-11T20:52:57.185659+0000 mon.a (mon.0) 881 : audit 0 from='client.? 172.21.15.23:0/4172899253' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-11T20:52:58.534 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:52:58 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:20:52:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T20:52:58.619 INFO:teuthology.orchestra.run.smithi023.stdout:55834574949 2024-03-11T20:52:59.013 INFO:teuthology.orchestra.run.smithi023.stdout:292057776168 2024-03-11T20:52:59.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:58 smithi146 bash[23809]: cluster 2024-03-11T20:52:57.243975+0000 osd.7 (osd.7) 53 : cluster 0 3.15 scrub starts 2024-03-11T20:52:59.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:58 smithi146 bash[23809]: cluster 2024-03-11T20:52:57.247698+0000 osd.7 (osd.7) 54 : cluster 0 3.15 scrub ok 2024-03-11T20:52:59.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:58 smithi146 bash[23809]: cluster 2024-03-11T20:52:57.848042+0000 mgr.y (mgr.24380) 76 : cluster 0 pgmap v54: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:52:59.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:58 smithi146 bash[23809]: cluster 2024-03-11T20:52:57.940803+0000 osd.2 (osd.2) 35 : cluster 0 6.14 scrub starts 2024-03-11T20:52:59.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:58 smithi146 bash[23809]: cluster 2024-03-11T20:52:57.942377+0000 osd.2 (osd.2) 36 : cluster 0 6.14 scrub ok 2024-03-11T20:52:59.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:58 smithi146 bash[23809]: audit 2024-03-11T20:52:58.623510+0000 mon.c (mon.2) 24 : audit 0 from='client.? 172.21.15.23:0/82377312' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-11T20:52:59.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:58 smithi023 bash[21372]: cluster 2024-03-11T20:52:57.243975+0000 osd.7 (osd.7) 53 : cluster 0 3.15 scrub starts 2024-03-11T20:52:59.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:58 smithi023 bash[21372]: cluster 2024-03-11T20:52:57.247698+0000 osd.7 (osd.7) 54 : cluster 0 3.15 scrub ok 2024-03-11T20:52:59.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:58 smithi023 bash[21372]: cluster 2024-03-11T20:52:57.848042+0000 mgr.y (mgr.24380) 76 : cluster 0 pgmap v54: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:52:59.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:58 smithi023 bash[21372]: cluster 2024-03-11T20:52:57.940803+0000 osd.2 (osd.2) 35 : cluster 0 6.14 scrub starts 2024-03-11T20:52:59.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:58 smithi023 bash[21372]: cluster 2024-03-11T20:52:57.942377+0000 osd.2 (osd.2) 36 : cluster 0 6.14 scrub ok 2024-03-11T20:52:59.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:58 smithi023 bash[21372]: audit 2024-03-11T20:52:58.623510+0000 mon.c (mon.2) 24 : audit 0 from='client.? 172.21.15.23:0/82377312' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-11T20:52:59.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:58 smithi023 bash[29146]: cluster 2024-03-11T20:52:57.243975+0000 osd.7 (osd.7) 53 : cluster 0 3.15 scrub starts 2024-03-11T20:52:59.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:58 smithi023 bash[29146]: cluster 2024-03-11T20:52:57.247698+0000 osd.7 (osd.7) 54 : cluster 0 3.15 scrub ok 2024-03-11T20:52:59.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:58 smithi023 bash[29146]: cluster 2024-03-11T20:52:57.848042+0000 mgr.y (mgr.24380) 76 : cluster 0 pgmap v54: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:52:59.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:58 smithi023 bash[29146]: cluster 2024-03-11T20:52:57.940803+0000 osd.2 (osd.2) 35 : cluster 0 6.14 scrub starts 2024-03-11T20:52:59.286 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:58 smithi023 bash[29146]: cluster 2024-03-11T20:52:57.942377+0000 osd.2 (osd.2) 36 : cluster 0 6.14 scrub ok 2024-03-11T20:52:59.286 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:58 smithi023 bash[29146]: audit 2024-03-11T20:52:58.623510+0000 mon.c (mon.2) 24 : audit 0 from='client.? 172.21.15.23:0/82377312' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-11T20:52:59.359 INFO:tasks.cephadm.ceph_manager.ceph:need seq 150323855427 got 150323855430 for osd.4 2024-03-11T20:52:59.360 DEBUG:teuthology.parallel:result is None 2024-03-11T20:53:00.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:59 smithi146 bash[23809]: cluster 2024-03-11T20:52:58.779816+0000 osd.5 (osd.5) 49 : cluster 0 2.2 scrub starts 2024-03-11T20:53:00.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:59 smithi146 bash[23809]: cluster 2024-03-11T20:52:58.781925+0000 osd.5 (osd.5) 50 : cluster 0 2.2 scrub ok 2024-03-11T20:53:00.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:52:59 smithi146 bash[23809]: audit 2024-03-11T20:52:59.017282+0000 mon.a (mon.0) 882 : audit 0 from='client.? 172.21.15.23:0/1555099921' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-11T20:53:00.313 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:59 smithi023 bash[21372]: cluster 2024-03-11T20:52:58.779816+0000 osd.5 (osd.5) 49 : cluster 0 2.2 scrub starts 2024-03-11T20:53:00.313 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:59 smithi023 bash[21372]: cluster 2024-03-11T20:52:58.781925+0000 osd.5 (osd.5) 50 : cluster 0 2.2 scrub ok 2024-03-11T20:53:00.313 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:52:59 smithi023 bash[21372]: audit 2024-03-11T20:52:59.017282+0000 mon.a (mon.0) 882 : audit 0 from='client.? 172.21.15.23:0/1555099921' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-11T20:53:00.313 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:59 smithi023 bash[29146]: cluster 2024-03-11T20:52:58.779816+0000 osd.5 (osd.5) 49 : cluster 0 2.2 scrub starts 2024-03-11T20:53:00.313 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:59 smithi023 bash[29146]: cluster 2024-03-11T20:52:58.781925+0000 osd.5 (osd.5) 50 : cluster 0 2.2 scrub ok 2024-03-11T20:53:00.313 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:52:59 smithi023 bash[29146]: audit 2024-03-11T20:52:59.017282+0000 mon.a (mon.0) 882 : audit 0 from='client.? 172.21.15.23:0/1555099921' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-11T20:53:00.469 INFO:teuthology.orchestra.run.smithi023.stdout:188978561084 2024-03-11T20:53:00.784 INFO:journalctl@ceph.alertmanager.a.smithi023.stdout:Mar 11 20:53:00 smithi023 bash[62056]: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-alertmanager-a 2024-03-11T20:53:01.170 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738475 got 34359738480 for osd.0 2024-03-11T20:53:01.170 DEBUG:teuthology.parallel:result is None 2024-03-11T20:53:01.180 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:00 smithi023 bash[21372]: cluster 2024-03-11T20:52:59.453248+0000 osd.3 (osd.3) 63 : cluster 0 4.1d scrub starts 2024-03-11T20:53:01.180 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:00 smithi023 bash[21372]: cluster 2024-03-11T20:52:59.455667+0000 osd.3 (osd.3) 64 : cluster 0 4.1d scrub ok 2024-03-11T20:53:01.180 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:00 smithi023 bash[21372]: cluster 2024-03-11T20:52:59.798899+0000 osd.5 (osd.5) 51 : cluster 0 2.12 scrub starts 2024-03-11T20:53:01.180 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:00 smithi023 bash[21372]: cluster 2024-03-11T20:52:59.801239+0000 osd.5 (osd.5) 52 : cluster 0 2.12 scrub ok 2024-03-11T20:53:01.181 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:00 smithi023 bash[21372]: cluster 2024-03-11T20:52:59.849365+0000 mgr.y (mgr.24380) 77 : cluster 0 pgmap v55: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:53:01.181 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:00 smithi023 bash[21372]: audit 2024-03-11T20:53:00.474119+0000 mon.a (mon.0) 883 : audit 0 from='client.? 172.21.15.23:0/2168848223' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-11T20:53:01.181 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:00 smithi023 bash[21372]: cluster 2024-03-11T20:53:00.549068+0000 osd.4 (osd.4) 55 : cluster 0 4.c scrub starts 2024-03-11T20:53:01.181 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:00 smithi023 bash[21372]: cluster 2024-03-11T20:53:00.551386+0000 osd.4 (osd.4) 56 : cluster 0 4.c scrub ok 2024-03-11T20:53:01.181 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:00 smithi023 bash[29146]: cluster 2024-03-11T20:52:59.453248+0000 osd.3 (osd.3) 63 : cluster 0 4.1d scrub starts 2024-03-11T20:53:01.181 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:00 smithi023 bash[29146]: cluster 2024-03-11T20:52:59.455667+0000 osd.3 (osd.3) 64 : cluster 0 4.1d scrub ok 2024-03-11T20:53:01.181 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:00 smithi023 bash[29146]: cluster 2024-03-11T20:52:59.798899+0000 osd.5 (osd.5) 51 : cluster 0 2.12 scrub starts 2024-03-11T20:53:01.181 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:00 smithi023 bash[29146]: cluster 2024-03-11T20:52:59.801239+0000 osd.5 (osd.5) 52 : cluster 0 2.12 scrub ok 2024-03-11T20:53:01.181 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:00 smithi023 bash[29146]: cluster 2024-03-11T20:52:59.849365+0000 mgr.y (mgr.24380) 77 : cluster 0 pgmap v55: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:53:01.181 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:00 smithi023 bash[29146]: audit 2024-03-11T20:53:00.474119+0000 mon.a (mon.0) 883 : audit 0 from='client.? 172.21.15.23:0/2168848223' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-11T20:53:01.181 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:00 smithi023 bash[29146]: cluster 2024-03-11T20:53:00.549068+0000 osd.4 (osd.4) 55 : cluster 0 4.c scrub starts 2024-03-11T20:53:01.181 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:00 smithi023 bash[29146]: cluster 2024-03-11T20:53:00.551386+0000 osd.4 (osd.4) 56 : cluster 0 4.c scrub ok 2024-03-11T20:53:01.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:00 smithi146 bash[23809]: cluster 2024-03-11T20:52:59.453248+0000 osd.3 (osd.3) 63 : cluster 0 4.1d scrub starts 2024-03-11T20:53:01.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:00 smithi146 bash[23809]: cluster 2024-03-11T20:52:59.455667+0000 osd.3 (osd.3) 64 : cluster 0 4.1d scrub ok 2024-03-11T20:53:01.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:00 smithi146 bash[23809]: cluster 2024-03-11T20:52:59.798899+0000 osd.5 (osd.5) 51 : cluster 0 2.12 scrub starts 2024-03-11T20:53:01.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:00 smithi146 bash[23809]: cluster 2024-03-11T20:52:59.801239+0000 osd.5 (osd.5) 52 : cluster 0 2.12 scrub ok 2024-03-11T20:53:01.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:00 smithi146 bash[23809]: cluster 2024-03-11T20:52:59.849365+0000 mgr.y (mgr.24380) 77 : cluster 0 pgmap v55: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:53:01.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:00 smithi146 bash[23809]: audit 2024-03-11T20:53:00.474119+0000 mon.a (mon.0) 883 : audit 0 from='client.? 172.21.15.23:0/2168848223' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-11T20:53:01.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:00 smithi146 bash[23809]: cluster 2024-03-11T20:53:00.549068+0000 osd.4 (osd.4) 55 : cluster 0 4.c scrub starts 2024-03-11T20:53:01.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:00 smithi146 bash[23809]: cluster 2024-03-11T20:53:00.551386+0000 osd.4 (osd.4) 56 : cluster 0 4.c scrub ok 2024-03-11T20:53:01.533 INFO:teuthology.orchestra.run.smithi023.stdout:244813135923 2024-03-11T20:53:02.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:01 smithi146 bash[23809]: cluster 2024-03-11T20:53:00.207427+0000 osd.7 (osd.7) 55 : cluster 0 2.1c deep-scrub starts 2024-03-11T20:53:02.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:01 smithi146 bash[23809]: cluster 2024-03-11T20:53:00.209779+0000 osd.7 (osd.7) 56 : cluster 0 2.1c deep-scrub ok 2024-03-11T20:53:02.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:01 smithi146 bash[23809]: cluster 2024-03-11T20:53:00.430858+0000 osd.3 (osd.3) 65 : cluster 0 4.14 deep-scrub starts 2024-03-11T20:53:02.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:01 smithi146 bash[23809]: cluster 2024-03-11T20:53:00.434521+0000 osd.3 (osd.3) 66 : cluster 0 4.14 deep-scrub ok 2024-03-11T20:53:02.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:01 smithi146 bash[23809]: cluster 2024-03-11T20:53:00.893976+0000 osd.2 (osd.2) 37 : cluster 0 2.e scrub starts 2024-03-11T20:53:02.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:01 smithi146 bash[23809]: cluster 2024-03-11T20:53:00.896762+0000 osd.2 (osd.2) 38 : cluster 0 2.e scrub ok 2024-03-11T20:53:02.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:01 smithi146 bash[23809]: audit 2024-03-11T20:53:01.537534+0000 mon.a (mon.0) 884 : audit 0 from='client.? 172.21.15.23:0/4003792180' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-11T20:53:02.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:01 smithi023 bash[29146]: cluster 2024-03-11T20:53:00.207427+0000 osd.7 (osd.7) 55 : cluster 0 2.1c deep-scrub starts 2024-03-11T20:53:02.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:01 smithi023 bash[29146]: cluster 2024-03-11T20:53:00.209779+0000 osd.7 (osd.7) 56 : cluster 0 2.1c deep-scrub ok 2024-03-11T20:53:02.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:01 smithi023 bash[29146]: cluster 2024-03-11T20:53:00.430858+0000 osd.3 (osd.3) 65 : cluster 0 4.14 deep-scrub starts 2024-03-11T20:53:02.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:01 smithi023 bash[29146]: cluster 2024-03-11T20:53:00.434521+0000 osd.3 (osd.3) 66 : cluster 0 4.14 deep-scrub ok 2024-03-11T20:53:02.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:01 smithi023 bash[29146]: cluster 2024-03-11T20:53:00.893976+0000 osd.2 (osd.2) 37 : cluster 0 2.e scrub starts 2024-03-11T20:53:02.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:01 smithi023 bash[29146]: cluster 2024-03-11T20:53:00.896762+0000 osd.2 (osd.2) 38 : cluster 0 2.e scrub ok 2024-03-11T20:53:02.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:01 smithi023 bash[29146]: audit 2024-03-11T20:53:01.537534+0000 mon.a (mon.0) 884 : audit 0 from='client.? 172.21.15.23:0/4003792180' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-11T20:53:02.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:01 smithi023 bash[21372]: cluster 2024-03-11T20:53:00.207427+0000 osd.7 (osd.7) 55 : cluster 0 2.1c deep-scrub starts 2024-03-11T20:53:02.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:01 smithi023 bash[21372]: cluster 2024-03-11T20:53:00.209779+0000 osd.7 (osd.7) 56 : cluster 0 2.1c deep-scrub ok 2024-03-11T20:53:02.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:01 smithi023 bash[21372]: cluster 2024-03-11T20:53:00.430858+0000 osd.3 (osd.3) 65 : cluster 0 4.14 deep-scrub starts 2024-03-11T20:53:02.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:01 smithi023 bash[21372]: cluster 2024-03-11T20:53:00.434521+0000 osd.3 (osd.3) 66 : cluster 0 4.14 deep-scrub ok 2024-03-11T20:53:02.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:01 smithi023 bash[21372]: cluster 2024-03-11T20:53:00.893976+0000 osd.2 (osd.2) 37 : cluster 0 2.e scrub starts 2024-03-11T20:53:02.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:01 smithi023 bash[21372]: cluster 2024-03-11T20:53:00.896762+0000 osd.2 (osd.2) 38 : cluster 0 2.e scrub ok 2024-03-11T20:53:02.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:01 smithi023 bash[21372]: audit 2024-03-11T20:53:01.537534+0000 mon.a (mon.0) 884 : audit 0 from='client.? 172.21.15.23:0/4003792180' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-11T20:53:02.429 INFO:tasks.cephadm.ceph_manager.ceph:need seq 292057776164 got 292057776168 for osd.7 2024-03-11T20:53:02.429 DEBUG:teuthology.parallel:result is None 2024-03-11T20:53:02.444 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574944 got 55834574949 for osd.1 2024-03-11T20:53:02.445 DEBUG:teuthology.parallel:result is None 2024-03-11T20:53:03.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:02 smithi146 bash[23809]: cluster 2024-03-11T20:53:01.850073+0000 mgr.y (mgr.24380) 78 : cluster 0 pgmap v56: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 854 B/s rd, 0 op/s 2024-03-11T20:53:03.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:02 smithi023 bash[21372]: cluster 2024-03-11T20:53:01.850073+0000 mgr.y (mgr.24380) 78 : cluster 0 pgmap v56: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 854 B/s rd, 0 op/s 2024-03-11T20:53:03.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:02 smithi023 bash[29146]: cluster 2024-03-11T20:53:01.850073+0000 mgr.y (mgr.24380) 78 : cluster 0 pgmap v56: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 854 B/s rd, 0 op/s 2024-03-11T20:53:03.284 INFO:journalctl@ceph.alertmanager.a.smithi023.stdout:Mar 11 20:53:02 smithi023 systemd[1]: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-03-11T20:53:03.285 INFO:journalctl@ceph.alertmanager.a.smithi023.stdout:Mar 11 20:53:02 smithi023 systemd[1]: Stopped Ceph alertmanager.a for 67ba00f8-dfe7-11ee-95c8-87774f69a715. 2024-03-11T20:53:03.285 INFO:journalctl@ceph.alertmanager.a.smithi023.stdout:Mar 11 20:53:02 smithi023 systemd[1]: Started Ceph alertmanager.a for 67ba00f8-dfe7-11ee-95c8-87774f69a715. 2024-03-11T20:53:03.928 INFO:teuthology.orchestra.run.smithi023.stdout:81604378716 2024-03-11T20:53:04.046 INFO:tasks.cephadm.ceph_manager.ceph:need seq 188978561079 got 188978561084 for osd.5 2024-03-11T20:53:04.046 DEBUG:teuthology.parallel:result is None 2024-03-11T20:53:04.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:04 smithi023 bash[29146]: cluster 2024-03-11T20:53:02.398069+0000 osd.3 (osd.3) 67 : cluster 0 4.11 scrub starts 2024-03-11T20:53:04.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:04 smithi023 bash[29146]: cluster 2024-03-11T20:53:02.400675+0000 osd.3 (osd.3) 68 : cluster 0 4.11 scrub ok 2024-03-11T20:53:04.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:04 smithi023 bash[29146]: cluster 2024-03-11T20:53:02.897179+0000 osd.2 (osd.2) 39 : cluster 0 2.10 scrub starts 2024-03-11T20:53:04.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:04 smithi023 bash[29146]: cluster 2024-03-11T20:53:02.899439+0000 osd.2 (osd.2) 40 : cluster 0 2.10 scrub ok 2024-03-11T20:53:04.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:04 smithi023 bash[29146]: audit 2024-03-11T20:53:03.012892+0000 mon.a (mon.0) 885 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:53:04.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:04 smithi023 bash[29146]: audit 2024-03-11T20:53:03.019953+0000 mon.a (mon.0) 886 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:53:04.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:04 smithi023 bash[29146]: cephadm 2024-03-11T20:53:03.045420+0000 mgr.y (mgr.24380) 79 : cephadm 1 Reconfiguring prometheus.a (dependencies changed)... 2024-03-11T20:53:04.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:04 smithi023 bash[29146]: cephadm 2024-03-11T20:53:03.306993+0000 mgr.y (mgr.24380) 80 : cephadm 1 Reconfiguring daemon prometheus.a on smithi146 2024-03-11T20:53:04.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:04 smithi023 bash[29146]: audit 2024-03-11T20:53:03.931824+0000 mon.a (mon.0) 887 : audit 0 from='client.? 172.21.15.23:0/40912626' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-11T20:53:04.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:04 smithi023 bash[21372]: cluster 2024-03-11T20:53:02.398069+0000 osd.3 (osd.3) 67 : cluster 0 4.11 scrub starts 2024-03-11T20:53:04.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:04 smithi023 bash[21372]: cluster 2024-03-11T20:53:02.400675+0000 osd.3 (osd.3) 68 : cluster 0 4.11 scrub ok 2024-03-11T20:53:04.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:04 smithi023 bash[21372]: cluster 2024-03-11T20:53:02.897179+0000 osd.2 (osd.2) 39 : cluster 0 2.10 scrub starts 2024-03-11T20:53:04.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:04 smithi023 bash[21372]: cluster 2024-03-11T20:53:02.899439+0000 osd.2 (osd.2) 40 : cluster 0 2.10 scrub ok 2024-03-11T20:53:04.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:04 smithi023 bash[21372]: audit 2024-03-11T20:53:03.012892+0000 mon.a (mon.0) 885 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:53:04.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:04 smithi023 bash[21372]: audit 2024-03-11T20:53:03.019953+0000 mon.a (mon.0) 886 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:53:04.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:04 smithi023 bash[21372]: cephadm 2024-03-11T20:53:03.045420+0000 mgr.y (mgr.24380) 79 : cephadm 1 Reconfiguring prometheus.a (dependencies changed)... 2024-03-11T20:53:04.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:04 smithi023 bash[21372]: cephadm 2024-03-11T20:53:03.306993+0000 mgr.y (mgr.24380) 80 : cephadm 1 Reconfiguring daemon prometheus.a on smithi146 2024-03-11T20:53:04.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:04 smithi023 bash[21372]: audit 2024-03-11T20:53:03.931824+0000 mon.a (mon.0) 887 : audit 0 from='client.? 172.21.15.23:0/40912626' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-11T20:53:04.452 INFO:teuthology.orchestra.run.smithi023.stdout:120259084369 2024-03-11T20:53:04.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:04 smithi146 bash[23809]: cluster 2024-03-11T20:53:02.398069+0000 osd.3 (osd.3) 67 : cluster 0 4.11 scrub starts 2024-03-11T20:53:04.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:04 smithi146 bash[23809]: cluster 2024-03-11T20:53:02.400675+0000 osd.3 (osd.3) 68 : cluster 0 4.11 scrub ok 2024-03-11T20:53:04.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:04 smithi146 bash[23809]: cluster 2024-03-11T20:53:02.897179+0000 osd.2 (osd.2) 39 : cluster 0 2.10 scrub starts 2024-03-11T20:53:04.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:04 smithi146 bash[23809]: cluster 2024-03-11T20:53:02.899439+0000 osd.2 (osd.2) 40 : cluster 0 2.10 scrub ok 2024-03-11T20:53:04.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:04 smithi146 bash[23809]: audit 2024-03-11T20:53:03.012892+0000 mon.a (mon.0) 885 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:53:04.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:04 smithi146 bash[23809]: audit 2024-03-11T20:53:03.019953+0000 mon.a (mon.0) 886 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:53:04.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:04 smithi146 bash[23809]: cephadm 2024-03-11T20:53:03.045420+0000 mgr.y (mgr.24380) 79 : cephadm 1 Reconfiguring prometheus.a (dependencies changed)... 2024-03-11T20:53:04.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:04 smithi146 bash[23809]: cephadm 2024-03-11T20:53:03.306993+0000 mgr.y (mgr.24380) 80 : cephadm 1 Reconfiguring daemon prometheus.a on smithi146 2024-03-11T20:53:04.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:04 smithi146 bash[23809]: audit 2024-03-11T20:53:03.931824+0000 mon.a (mon.0) 887 : audit 0 from='client.? 172.21.15.23:0/40912626' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-11T20:53:04.913 INFO:tasks.cephadm.ceph_manager.ceph:need seq 244813135918 got 244813135923 for osd.6 2024-03-11T20:53:04.913 DEBUG:teuthology.parallel:result is None 2024-03-11T20:53:05.213 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 20:53:04 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T20:53:05.214 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:05 smithi146 bash[23809]: cluster 2024-03-11T20:53:03.106326+0000 osd.0 (osd.0) 61 : cluster 0 3.1 scrub starts 2024-03-11T20:53:05.214 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:05 smithi146 bash[23809]: cluster 2024-03-11T20:53:03.110439+0000 osd.0 (osd.0) 62 : cluster 0 3.1 scrub ok 2024-03-11T20:53:05.214 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:05 smithi146 bash[23809]: cluster 2024-03-11T20:53:03.448172+0000 osd.3 (osd.3) 69 : cluster 0 3.b scrub starts 2024-03-11T20:53:05.214 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:05 smithi146 bash[23809]: cluster 2024-03-11T20:53:03.452514+0000 osd.3 (osd.3) 70 : cluster 0 3.b scrub ok 2024-03-11T20:53:05.214 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:05 smithi146 bash[23809]: cluster 2024-03-11T20:53:03.761634+0000 osd.5 (osd.5) 53 : cluster 0 6.a deep-scrub starts 2024-03-11T20:53:05.214 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:05 smithi146 bash[23809]: cluster 2024-03-11T20:53:03.762878+0000 osd.5 (osd.5) 54 : cluster 0 6.a deep-scrub ok 2024-03-11T20:53:05.214 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:05 smithi146 bash[23809]: cluster 2024-03-11T20:53:03.850687+0000 mgr.y (mgr.24380) 81 : cluster 0 pgmap v57: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T20:53:05.214 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:05 smithi146 bash[23809]: audit 2024-03-11T20:53:04.099694+0000 mon.a (mon.0) 888 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:53:05.214 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:05 smithi146 bash[23809]: cluster 2024-03-11T20:53:04.139950+0000 osd.0 (osd.0) 63 : cluster 0 4.6 scrub starts 2024-03-11T20:53:05.214 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:05 smithi146 bash[23809]: cluster 2024-03-11T20:53:04.140869+0000 osd.0 (osd.0) 64 : cluster 0 4.6 scrub ok 2024-03-11T20:53:05.214 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:05 smithi146 bash[23809]: audit 2024-03-11T20:53:04.456239+0000 mon.a (mon.0) 889 : audit 0 from='client.? 172.21.15.23:0/2646818671' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-11T20:53:05.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:05 smithi023 bash[29146]: cluster 2024-03-11T20:53:03.106326+0000 osd.0 (osd.0) 61 : cluster 0 3.1 scrub starts 2024-03-11T20:53:05.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:05 smithi023 bash[29146]: cluster 2024-03-11T20:53:03.110439+0000 osd.0 (osd.0) 62 : cluster 0 3.1 scrub ok 2024-03-11T20:53:05.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:05 smithi023 bash[29146]: cluster 2024-03-11T20:53:03.448172+0000 osd.3 (osd.3) 69 : cluster 0 3.b scrub starts 2024-03-11T20:53:05.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:05 smithi023 bash[29146]: cluster 2024-03-11T20:53:03.452514+0000 osd.3 (osd.3) 70 : cluster 0 3.b scrub ok 2024-03-11T20:53:05.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:05 smithi023 bash[29146]: cluster 2024-03-11T20:53:03.761634+0000 osd.5 (osd.5) 53 : cluster 0 6.a deep-scrub starts 2024-03-11T20:53:05.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:05 smithi023 bash[29146]: cluster 2024-03-11T20:53:03.762878+0000 osd.5 (osd.5) 54 : cluster 0 6.a deep-scrub ok 2024-03-11T20:53:05.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:05 smithi023 bash[29146]: cluster 2024-03-11T20:53:03.850687+0000 mgr.y (mgr.24380) 81 : cluster 0 pgmap v57: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T20:53:05.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:05 smithi023 bash[29146]: audit 2024-03-11T20:53:04.099694+0000 mon.a (mon.0) 888 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:53:05.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:05 smithi023 bash[29146]: cluster 2024-03-11T20:53:04.139950+0000 osd.0 (osd.0) 63 : cluster 0 4.6 scrub starts 2024-03-11T20:53:05.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:05 smithi023 bash[29146]: cluster 2024-03-11T20:53:04.140869+0000 osd.0 (osd.0) 64 : cluster 0 4.6 scrub ok 2024-03-11T20:53:05.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:05 smithi023 bash[29146]: audit 2024-03-11T20:53:04.456239+0000 mon.a (mon.0) 889 : audit 0 from='client.? 172.21.15.23:0/2646818671' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-11T20:53:05.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:05 smithi023 bash[21372]: cluster 2024-03-11T20:53:03.106326+0000 osd.0 (osd.0) 61 : cluster 0 3.1 scrub starts 2024-03-11T20:53:05.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:05 smithi023 bash[21372]: cluster 2024-03-11T20:53:03.110439+0000 osd.0 (osd.0) 62 : cluster 0 3.1 scrub ok 2024-03-11T20:53:05.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:05 smithi023 bash[21372]: cluster 2024-03-11T20:53:03.448172+0000 osd.3 (osd.3) 69 : cluster 0 3.b scrub starts 2024-03-11T20:53:05.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:05 smithi023 bash[21372]: cluster 2024-03-11T20:53:03.452514+0000 osd.3 (osd.3) 70 : cluster 0 3.b scrub ok 2024-03-11T20:53:05.286 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:05 smithi023 bash[21372]: cluster 2024-03-11T20:53:03.761634+0000 osd.5 (osd.5) 53 : cluster 0 6.a deep-scrub starts 2024-03-11T20:53:05.286 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:05 smithi023 bash[21372]: cluster 2024-03-11T20:53:03.762878+0000 osd.5 (osd.5) 54 : cluster 0 6.a deep-scrub ok 2024-03-11T20:53:05.286 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:05 smithi023 bash[21372]: cluster 2024-03-11T20:53:03.850687+0000 mgr.y (mgr.24380) 81 : cluster 0 pgmap v57: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T20:53:05.286 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:05 smithi023 bash[21372]: audit 2024-03-11T20:53:04.099694+0000 mon.a (mon.0) 888 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:53:05.286 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:05 smithi023 bash[21372]: cluster 2024-03-11T20:53:04.139950+0000 osd.0 (osd.0) 63 : cluster 0 4.6 scrub starts 2024-03-11T20:53:05.286 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:05 smithi023 bash[21372]: cluster 2024-03-11T20:53:04.140869+0000 osd.0 (osd.0) 64 : cluster 0 4.6 scrub ok 2024-03-11T20:53:05.286 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:05 smithi023 bash[21372]: audit 2024-03-11T20:53:04.456239+0000 mon.a (mon.0) 889 : audit 0 from='client.? 172.21.15.23:0/2646818671' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-11T20:53:06.157 INFO:tasks.cephadm.ceph_manager.ceph:need seq 81604378711 got 81604378716 for osd.2 2024-03-11T20:53:06.157 DEBUG:teuthology.parallel:result is None 2024-03-11T20:53:06.236 INFO:tasks.cephadm.ceph_manager.ceph:need seq 120259084364 got 120259084369 for osd.3 2024-03-11T20:53:06.237 DEBUG:teuthology.parallel:result is None 2024-03-11T20:53:06.237 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-11T20:53:06.237 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T20:53:06.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:06 smithi146 bash[23809]: cluster 2024-03-11T20:53:04.318775+0000 osd.7 (osd.7) 57 : cluster 0 6.12 scrub starts 2024-03-11T20:53:06.263 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:06 smithi146 bash[23809]: cluster 2024-03-11T20:53:04.321226+0000 osd.7 (osd.7) 58 : cluster 0 6.12 scrub ok 2024-03-11T20:53:06.263 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:06 smithi146 bash[23809]: cluster 2024-03-11T20:53:04.791226+0000 osd.5 (osd.5) 55 : cluster 0 3.1d deep-scrub starts 2024-03-11T20:53:06.263 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:06 smithi146 bash[23809]: cluster 2024-03-11T20:53:04.802476+0000 osd.5 (osd.5) 56 : cluster 0 3.1d deep-scrub ok 2024-03-11T20:53:06.263 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:06 smithi146 bash[23809]: audit 2024-03-11T20:53:04.893817+0000 mgr.y (mgr.24380) 82 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:53:06.263 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:06 smithi146 bash[23809]: cluster 2024-03-11T20:53:05.098760+0000 osd.0 (osd.0) 65 : cluster 0 4.16 scrub starts 2024-03-11T20:53:06.263 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:06 smithi146 bash[23809]: cluster 2024-03-11T20:53:05.100842+0000 osd.0 (osd.0) 66 : cluster 0 4.16 scrub ok 2024-03-11T20:53:06.324 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:06 smithi023 bash[21372]: cluster 2024-03-11T20:53:04.318775+0000 osd.7 (osd.7) 57 : cluster 0 6.12 scrub starts 2024-03-11T20:53:06.324 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:06 smithi023 bash[21372]: cluster 2024-03-11T20:53:04.321226+0000 osd.7 (osd.7) 58 : cluster 0 6.12 scrub ok 2024-03-11T20:53:06.324 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:06 smithi023 bash[21372]: cluster 2024-03-11T20:53:04.791226+0000 osd.5 (osd.5) 55 : cluster 0 3.1d deep-scrub starts 2024-03-11T20:53:06.324 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:06 smithi023 bash[21372]: cluster 2024-03-11T20:53:04.802476+0000 osd.5 (osd.5) 56 : cluster 0 3.1d deep-scrub ok 2024-03-11T20:53:06.324 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:06 smithi023 bash[21372]: audit 2024-03-11T20:53:04.893817+0000 mgr.y (mgr.24380) 82 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:53:06.324 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:06 smithi023 bash[21372]: cluster 2024-03-11T20:53:05.098760+0000 osd.0 (osd.0) 65 : cluster 0 4.16 scrub starts 2024-03-11T20:53:06.324 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:06 smithi023 bash[21372]: cluster 2024-03-11T20:53:05.100842+0000 osd.0 (osd.0) 66 : cluster 0 4.16 scrub ok 2024-03-11T20:53:06.325 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:06 smithi023 bash[29146]: cluster 2024-03-11T20:53:04.318775+0000 osd.7 (osd.7) 57 : cluster 0 6.12 scrub starts 2024-03-11T20:53:06.325 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:06 smithi023 bash[29146]: cluster 2024-03-11T20:53:04.321226+0000 osd.7 (osd.7) 58 : cluster 0 6.12 scrub ok 2024-03-11T20:53:06.325 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:06 smithi023 bash[29146]: cluster 2024-03-11T20:53:04.791226+0000 osd.5 (osd.5) 55 : cluster 0 3.1d deep-scrub starts 2024-03-11T20:53:06.325 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:06 smithi023 bash[29146]: cluster 2024-03-11T20:53:04.802476+0000 osd.5 (osd.5) 56 : cluster 0 3.1d deep-scrub ok 2024-03-11T20:53:06.325 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:06 smithi023 bash[29146]: audit 2024-03-11T20:53:04.893817+0000 mgr.y (mgr.24380) 82 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:53:06.325 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:06 smithi023 bash[29146]: cluster 2024-03-11T20:53:05.098760+0000 osd.0 (osd.0) 65 : cluster 0 4.16 scrub starts 2024-03-11T20:53:06.325 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:06 smithi023 bash[29146]: cluster 2024-03-11T20:53:05.100842+0000 osd.0 (osd.0) 66 : cluster 0 4.16 scrub ok 2024-03-11T20:53:06.429 INFO:journalctl@ceph.alertmanager.a.smithi023.stdout:Mar 11 20:53:06 smithi023 bash[62486]: ts=2024-03-11T20:53:06.399Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-03-11T20:53:06.429 INFO:journalctl@ceph.alertmanager.a.smithi023.stdout:Mar 11 20:53:06 smithi023 bash[62486]: ts=2024-03-11T20:53:06.399Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-03-11T20:53:06.429 INFO:journalctl@ceph.alertmanager.a.smithi023.stdout:Mar 11 20:53:06 smithi023 bash[62486]: ts=2024-03-11T20:53:06.401Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.23 port=9094 2024-03-11T20:53:06.429 INFO:journalctl@ceph.alertmanager.a.smithi023.stdout:Mar 11 20:53:06 smithi023 bash[62486]: ts=2024-03-11T20:53:06.401Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-03-11T20:53:06.784 INFO:journalctl@ceph.alertmanager.a.smithi023.stdout:Mar 11 20:53:06 smithi023 bash[62486]: ts=2024-03-11T20:53:06.433Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-11T20:53:06.784 INFO:journalctl@ceph.alertmanager.a.smithi023.stdout:Mar 11 20:53:06 smithi023 bash[62486]: ts=2024-03-11T20:53:06.433Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-11T20:53:06.784 INFO:journalctl@ceph.alertmanager.a.smithi023.stdout:Mar 11 20:53:06 smithi023 bash[62486]: ts=2024-03-11T20:53:06.437Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-03-11T20:53:06.784 INFO:journalctl@ceph.alertmanager.a.smithi023.stdout:Mar 11 20:53:06 smithi023 bash[62486]: ts=2024-03-11T20:53:06.437Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-03-11T20:53:07.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:07 smithi146 bash[23809]: cluster 2024-03-11T20:53:05.362123+0000 osd.7 (osd.7) 59 : cluster 0 2.5 scrub starts 2024-03-11T20:53:07.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:07 smithi146 bash[23809]: cluster 2024-03-11T20:53:05.364419+0000 osd.7 (osd.7) 60 : cluster 0 2.5 scrub ok 2024-03-11T20:53:07.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:07 smithi146 bash[23809]: cluster 2024-03-11T20:53:05.507791+0000 osd.3 (osd.3) 71 : cluster 0 6.1b scrub starts 2024-03-11T20:53:07.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:07 smithi146 bash[23809]: cluster 2024-03-11T20:53:05.510050+0000 osd.3 (osd.3) 72 : cluster 0 6.1b scrub ok 2024-03-11T20:53:07.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:07 smithi146 bash[23809]: cluster 2024-03-11T20:53:05.834367+0000 osd.2 (osd.2) 41 : cluster 0 4.f scrub starts 2024-03-11T20:53:07.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:07 smithi146 bash[23809]: cluster 2024-03-11T20:53:05.836642+0000 osd.2 (osd.2) 42 : cluster 0 4.f scrub ok 2024-03-11T20:53:07.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:07 smithi146 bash[23809]: cluster 2024-03-11T20:53:05.852282+0000 mgr.y (mgr.24380) 83 : cluster 0 pgmap v58: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:53:07.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:07 smithi023 bash[21372]: cluster 2024-03-11T20:53:05.362123+0000 osd.7 (osd.7) 59 : cluster 0 2.5 scrub starts 2024-03-11T20:53:07.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:07 smithi023 bash[21372]: cluster 2024-03-11T20:53:05.364419+0000 osd.7 (osd.7) 60 : cluster 0 2.5 scrub ok 2024-03-11T20:53:07.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:07 smithi023 bash[21372]: cluster 2024-03-11T20:53:05.507791+0000 osd.3 (osd.3) 71 : cluster 0 6.1b scrub starts 2024-03-11T20:53:07.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:07 smithi023 bash[21372]: cluster 2024-03-11T20:53:05.510050+0000 osd.3 (osd.3) 72 : cluster 0 6.1b scrub ok 2024-03-11T20:53:07.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:07 smithi023 bash[21372]: cluster 2024-03-11T20:53:05.834367+0000 osd.2 (osd.2) 41 : cluster 0 4.f scrub starts 2024-03-11T20:53:07.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:07 smithi023 bash[21372]: cluster 2024-03-11T20:53:05.836642+0000 osd.2 (osd.2) 42 : cluster 0 4.f scrub ok 2024-03-11T20:53:07.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:07 smithi023 bash[21372]: cluster 2024-03-11T20:53:05.852282+0000 mgr.y (mgr.24380) 83 : cluster 0 pgmap v58: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:53:07.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:07 smithi023 bash[29146]: cluster 2024-03-11T20:53:05.362123+0000 osd.7 (osd.7) 59 : cluster 0 2.5 scrub starts 2024-03-11T20:53:07.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:07 smithi023 bash[29146]: cluster 2024-03-11T20:53:05.364419+0000 osd.7 (osd.7) 60 : cluster 0 2.5 scrub ok 2024-03-11T20:53:07.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:07 smithi023 bash[29146]: cluster 2024-03-11T20:53:05.507791+0000 osd.3 (osd.3) 71 : cluster 0 6.1b scrub starts 2024-03-11T20:53:07.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:07 smithi023 bash[29146]: cluster 2024-03-11T20:53:05.510050+0000 osd.3 (osd.3) 72 : cluster 0 6.1b scrub ok 2024-03-11T20:53:07.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:07 smithi023 bash[29146]: cluster 2024-03-11T20:53:05.834367+0000 osd.2 (osd.2) 41 : cluster 0 4.f scrub starts 2024-03-11T20:53:07.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:07 smithi023 bash[29146]: cluster 2024-03-11T20:53:05.836642+0000 osd.2 (osd.2) 42 : cluster 0 4.f scrub ok 2024-03-11T20:53:07.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:07 smithi023 bash[29146]: cluster 2024-03-11T20:53:05.852282+0000 mgr.y (mgr.24380) 83 : cluster 0 pgmap v58: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:53:07.761 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Mar 11 20:53:07 smithi146 systemd[1]: Stopping Ceph prometheus.a for 67ba00f8-dfe7-11ee-95c8-87774f69a715... 2024-03-11T20:53:07.761 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Mar 11 20:53:07 smithi146 bash[50312]: ts=2024-03-11T20:53:07.631Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-11T20:53:07.761 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Mar 11 20:53:07 smithi146 bash[50312]: ts=2024-03-11T20:53:07.631Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-03-11T20:53:07.761 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Mar 11 20:53:07 smithi146 bash[50312]: ts=2024-03-11T20:53:07.631Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-03-11T20:53:07.761 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Mar 11 20:53:07 smithi146 bash[50312]: ts=2024-03-11T20:53:07.631Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-11T20:53:07.761 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Mar 11 20:53:07 smithi146 bash[50312]: ts=2024-03-11T20:53:07.631Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-03-11T20:53:07.761 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Mar 11 20:53:07 smithi146 bash[50312]: ts=2024-03-11T20:53:07.631Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-03-11T20:53:07.761 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Mar 11 20:53:07 smithi146 bash[50312]: ts=2024-03-11T20:53:07.631Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-03-11T20:53:07.761 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Mar 11 20:53:07 smithi146 bash[50312]: ts=2024-03-11T20:53:07.631Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-03-11T20:53:07.761 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Mar 11 20:53:07 smithi146 bash[50312]: ts=2024-03-11T20:53:07.631Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-03-11T20:53:07.762 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Mar 11 20:53:07 smithi146 bash[50312]: ts=2024-03-11T20:53:07.632Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-03-11T20:53:07.762 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Mar 11 20:53:07 smithi146 bash[50312]: ts=2024-03-11T20:53:07.632Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-03-11T20:53:07.762 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Mar 11 20:53:07 smithi146 bash[50312]: ts=2024-03-11T20:53:07.632Z caller=main.go:1147 level=info msg="See you next time!" 2024-03-11T20:53:08.164 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:08 smithi146 bash[23809]: cluster 2024-03-11T20:53:06.341261+0000 osd.7 (osd.7) 61 : cluster 0 6.3 scrub starts 2024-03-11T20:53:08.164 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:08 smithi146 bash[23809]: cluster 2024-03-11T20:53:06.344376+0000 osd.7 (osd.7) 62 : cluster 0 6.3 scrub ok 2024-03-11T20:53:08.164 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:08 smithi146 bash[23809]: cluster 2024-03-11T20:53:06.532741+0000 osd.3 (osd.3) 73 : cluster 0 4.17 scrub starts 2024-03-11T20:53:08.164 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:08 smithi146 bash[23809]: cluster 2024-03-11T20:53:06.534599+0000 osd.3 (osd.3) 74 : cluster 0 4.17 scrub ok 2024-03-11T20:53:08.164 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:08 smithi146 bash[23809]: cluster 2024-03-11T20:53:06.769964+0000 osd.5 (osd.5) 57 : cluster 0 6.d scrub starts 2024-03-11T20:53:08.164 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:08 smithi146 bash[23809]: cluster 2024-03-11T20:53:06.771616+0000 osd.5 (osd.5) 58 : cluster 0 6.d scrub ok 2024-03-11T20:53:08.164 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:08 smithi146 bash[23809]: cluster 2024-03-11T20:53:06.794822+0000 osd.2 (osd.2) 43 : cluster 0 2.1 scrub starts 2024-03-11T20:53:08.164 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:08 smithi146 bash[23809]: cluster 2024-03-11T20:53:06.797305+0000 osd.2 (osd.2) 44 : cluster 0 2.1 scrub ok 2024-03-11T20:53:08.164 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:08 smithi146 bash[23809]: cluster 2024-03-11T20:53:07.559248+0000 osd.4 (osd.4) 57 : cluster 0 6.1a scrub starts 2024-03-11T20:53:08.165 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:08 smithi146 bash[23809]: cluster 2024-03-11T20:53:07.560725+0000 osd.4 (osd.4) 58 : cluster 0 6.1a scrub ok 2024-03-11T20:53:08.398 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:08 smithi023 bash[21372]: cluster 2024-03-11T20:53:06.341261+0000 osd.7 (osd.7) 61 : cluster 0 6.3 scrub starts 2024-03-11T20:53:08.399 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:08 smithi023 bash[21372]: cluster 2024-03-11T20:53:06.344376+0000 osd.7 (osd.7) 62 : cluster 0 6.3 scrub ok 2024-03-11T20:53:08.399 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:08 smithi023 bash[21372]: cluster 2024-03-11T20:53:06.532741+0000 osd.3 (osd.3) 73 : cluster 0 4.17 scrub starts 2024-03-11T20:53:08.399 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:08 smithi023 bash[21372]: cluster 2024-03-11T20:53:06.534599+0000 osd.3 (osd.3) 74 : cluster 0 4.17 scrub ok 2024-03-11T20:53:08.399 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:08 smithi023 bash[21372]: cluster 2024-03-11T20:53:06.769964+0000 osd.5 (osd.5) 57 : cluster 0 6.d scrub starts 2024-03-11T20:53:08.399 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:08 smithi023 bash[21372]: cluster 2024-03-11T20:53:06.771616+0000 osd.5 (osd.5) 58 : cluster 0 6.d scrub ok 2024-03-11T20:53:08.399 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:08 smithi023 bash[21372]: cluster 2024-03-11T20:53:06.794822+0000 osd.2 (osd.2) 43 : cluster 0 2.1 scrub starts 2024-03-11T20:53:08.399 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:08 smithi023 bash[21372]: cluster 2024-03-11T20:53:06.797305+0000 osd.2 (osd.2) 44 : cluster 0 2.1 scrub ok 2024-03-11T20:53:08.399 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:08 smithi023 bash[21372]: cluster 2024-03-11T20:53:07.559248+0000 osd.4 (osd.4) 57 : cluster 0 6.1a scrub starts 2024-03-11T20:53:08.399 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:08 smithi023 bash[21372]: cluster 2024-03-11T20:53:07.560725+0000 osd.4 (osd.4) 58 : cluster 0 6.1a scrub ok 2024-03-11T20:53:08.399 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:08 smithi023 bash[29146]: cluster 2024-03-11T20:53:06.341261+0000 osd.7 (osd.7) 61 : cluster 0 6.3 scrub starts 2024-03-11T20:53:08.399 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:08 smithi023 bash[29146]: cluster 2024-03-11T20:53:06.344376+0000 osd.7 (osd.7) 62 : cluster 0 6.3 scrub ok 2024-03-11T20:53:08.400 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:08 smithi023 bash[29146]: cluster 2024-03-11T20:53:06.532741+0000 osd.3 (osd.3) 73 : cluster 0 4.17 scrub starts 2024-03-11T20:53:08.400 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:08 smithi023 bash[29146]: cluster 2024-03-11T20:53:06.534599+0000 osd.3 (osd.3) 74 : cluster 0 4.17 scrub ok 2024-03-11T20:53:08.400 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:08 smithi023 bash[29146]: cluster 2024-03-11T20:53:06.769964+0000 osd.5 (osd.5) 57 : cluster 0 6.d scrub starts 2024-03-11T20:53:08.400 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:08 smithi023 bash[29146]: cluster 2024-03-11T20:53:06.771616+0000 osd.5 (osd.5) 58 : cluster 0 6.d scrub ok 2024-03-11T20:53:08.400 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:08 smithi023 bash[29146]: cluster 2024-03-11T20:53:06.794822+0000 osd.2 (osd.2) 43 : cluster 0 2.1 scrub starts 2024-03-11T20:53:08.400 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:08 smithi023 bash[29146]: cluster 2024-03-11T20:53:06.797305+0000 osd.2 (osd.2) 44 : cluster 0 2.1 scrub ok 2024-03-11T20:53:08.400 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:08 smithi023 bash[29146]: cluster 2024-03-11T20:53:07.559248+0000 osd.4 (osd.4) 57 : cluster 0 6.1a scrub starts 2024-03-11T20:53:08.400 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:08 smithi023 bash[29146]: cluster 2024-03-11T20:53:07.560725+0000 osd.4 (osd.4) 58 : cluster 0 6.1a scrub ok 2024-03-11T20:53:08.418 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Mar 11 20:53:08 smithi146 bash[52316]: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-prometheus-a 2024-03-11T20:53:08.418 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Mar 11 20:53:08 smithi146 bash[52379]: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-prometheus-a 2024-03-11T20:53:08.418 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Mar 11 20:53:08 smithi146 systemd[1]: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@prometheus.a.service: Deactivated successfully. 2024-03-11T20:53:08.418 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Mar 11 20:53:08 smithi146 systemd[1]: Stopped Ceph prometheus.a for 67ba00f8-dfe7-11ee-95c8-87774f69a715. 2024-03-11T20:53:08.418 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Mar 11 20:53:08 smithi146 systemd[1]: Started Ceph prometheus.a for 67ba00f8-dfe7-11ee-95c8-87774f69a715. 2024-03-11T20:53:08.652 INFO:journalctl@ceph.alertmanager.a.smithi023.stdout:Mar 11 20:53:08 smithi023 bash[62486]: ts=2024-03-11T20:53:08.402Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000413488s 2024-03-11T20:53:08.652 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:53:08 smithi023 bash[21627]: [11/Mar/2024:20:53:08] ENGINE Bus STOPPING 2024-03-11T20:53:08.652 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:53:08 smithi023 bash[21627]: [11/Mar/2024:20:53:08] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-11T20:53:08.652 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:53:08 smithi023 bash[21627]: [11/Mar/2024:20:53:08] ENGINE Bus STOPPED 2024-03-11T20:53:08.652 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:53:08 smithi023 bash[21627]: [11/Mar/2024:20:53:08] ENGINE Bus STARTING 2024-03-11T20:53:08.652 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:53:08 smithi023 bash[21627]: [11/Mar/2024:20:53:08] ENGINE Serving on http://:::9283 2024-03-11T20:53:08.652 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:53:08 smithi023 bash[21627]: [11/Mar/2024:20:53:08] ENGINE Bus STARTED 2024-03-11T20:53:08.653 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:53:08 smithi023 bash[21627]: [11/Mar/2024:20:53:08] ENGINE Bus STOPPING 2024-03-11T20:53:09.034 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:53:08 smithi023 bash[21627]: [11/Mar/2024:20:53:08] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-11T20:53:09.034 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:53:08 smithi023 bash[21627]: [11/Mar/2024:20:53:08] ENGINE Bus STOPPED 2024-03-11T20:53:09.034 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:53:08 smithi023 bash[21627]: [11/Mar/2024:20:53:08] ENGINE Bus STARTING 2024-03-11T20:53:09.034 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:53:08 smithi023 bash[21627]: [11/Mar/2024:20:53:08] ENGINE Serving on http://:::9283 2024-03-11T20:53:09.034 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:53:08 smithi023 bash[21627]: [11/Mar/2024:20:53:08] ENGINE Bus STARTED 2024-03-11T20:53:09.034 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:53:08 smithi023 bash[21627]: [11/Mar/2024:20:53:08] ENGINE Bus STOPPING 2024-03-11T20:53:09.034 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:53:08 smithi023 bash[21627]: [11/Mar/2024:20:53:08] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-11T20:53:09.034 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:53:08 smithi023 bash[21627]: [11/Mar/2024:20:53:08] ENGINE Bus STOPPED 2024-03-11T20:53:09.034 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:53:08 smithi023 bash[21627]: [11/Mar/2024:20:53:08] ENGINE Bus STARTING 2024-03-11T20:53:09.034 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:53:08 smithi023 bash[21627]: [11/Mar/2024:20:53:08] ENGINE Serving on http://:::9283 2024-03-11T20:53:09.034 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:53:08 smithi023 bash[21627]: [11/Mar/2024:20:53:08] ENGINE Bus STARTED 2024-03-11T20:53:09.397 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:09 smithi146 bash[23809]: cluster 2024-03-11T20:53:07.790208+0000 osd.5 (osd.5) 59 : cluster 0 6.19 deep-scrub starts 2024-03-11T20:53:09.397 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:09 smithi146 bash[23809]: cluster 2024-03-11T20:53:07.792686+0000 osd.5 (osd.5) 60 : cluster 0 6.19 deep-scrub ok 2024-03-11T20:53:09.397 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:09 smithi146 bash[23809]: cluster 2024-03-11T20:53:07.852958+0000 mgr.y (mgr.24380) 84 : cluster 0 pgmap v59: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T20:53:09.397 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:09 smithi146 bash[23809]: cluster 2024-03-11T20:53:07.856943+0000 osd.6 (osd.6) 33 : cluster 0 4.5 deep-scrub starts 2024-03-11T20:53:09.397 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:09 smithi146 bash[23809]: cluster 2024-03-11T20:53:07.859529+0000 osd.6 (osd.6) 34 : cluster 0 4.5 deep-scrub ok 2024-03-11T20:53:09.397 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:09 smithi146 bash[23809]: cluster 2024-03-11T20:53:07.874553+0000 osd.1 (osd.1) 53 : cluster 0 purged_snaps scrub starts 2024-03-11T20:53:09.397 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:09 smithi146 bash[23809]: cluster 2024-03-11T20:53:07.874644+0000 osd.1 (osd.1) 54 : cluster 0 purged_snaps scrub ok 2024-03-11T20:53:09.397 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:09 smithi146 bash[23809]: cluster 2024-03-11T20:53:08.088759+0000 mon.a (mon.0) 890 : cluster 0 osdmap e111: 8 total, 8 up, 8 in 2024-03-11T20:53:09.398 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:09 smithi146 bash[23809]: audit 2024-03-11T20:53:08.429136+0000 mon.a (mon.0) 891 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:53:09.398 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:09 smithi146 bash[23809]: audit 2024-03-11T20:53:08.441716+0000 mon.a (mon.0) 892 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:53:09.398 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:09 smithi146 bash[23809]: audit 2024-03-11T20:53:08.498785+0000 mon.a (mon.0) 893 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-11T20:53:09.398 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:09 smithi146 bash[23809]: audit 2024-03-11T20:53:08.501229+0000 mon.a (mon.0) 894 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi023:9093"}]: dispatch 2024-03-11T20:53:09.398 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:09 smithi146 bash[23809]: audit 2024-03-11T20:53:08.508170+0000 mon.a (mon.0) 895 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:53:09.398 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:09 smithi146 bash[23809]: audit 2024-03-11T20:53:08.523783+0000 mon.a (mon.0) 896 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-11T20:53:09.398 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:09 smithi146 bash[23809]: audit 2024-03-11T20:53:08.531714+0000 mon.a (mon.0) 897 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi146.front.sepia.ceph.com:9095"}]: dispatch 2024-03-11T20:53:09.398 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:09 smithi146 bash[23809]: audit 2024-03-11T20:53:08.541551+0000 mon.a (mon.0) 898 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:53:09.398 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:09 smithi146 bash[23809]: audit 2024-03-11T20:53:08.552817+0000 mon.a (mon.0) 899 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-11T20:53:09.398 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:09 smithi146 bash[23809]: audit 2024-03-11T20:53:08.559738+0000 mon.a (mon.0) 900 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi146.front.sepia.ceph.com:3000"}]: dispatch 2024-03-11T20:53:09.398 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:09 smithi146 bash[23809]: audit 2024-03-11T20:53:08.564757+0000 mon.a (mon.0) 901 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:53:09.398 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:09 smithi146 bash[23809]: audit 2024-03-11T20:53:08.584359+0000 mon.a (mon.0) 902 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:53:09.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:09 smithi023 bash[21372]: cluster 2024-03-11T20:53:07.790208+0000 osd.5 (osd.5) 59 : cluster 0 6.19 deep-scrub starts 2024-03-11T20:53:09.536 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:09 smithi023 bash[21372]: cluster 2024-03-11T20:53:07.792686+0000 osd.5 (osd.5) 60 : cluster 0 6.19 deep-scrub ok 2024-03-11T20:53:09.536 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:09 smithi023 bash[21372]: cluster 2024-03-11T20:53:07.852958+0000 mgr.y (mgr.24380) 84 : cluster 0 pgmap v59: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T20:53:09.536 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:09 smithi023 bash[21372]: cluster 2024-03-11T20:53:07.856943+0000 osd.6 (osd.6) 33 : cluster 0 4.5 deep-scrub starts 2024-03-11T20:53:09.536 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:09 smithi023 bash[21372]: cluster 2024-03-11T20:53:07.859529+0000 osd.6 (osd.6) 34 : cluster 0 4.5 deep-scrub ok 2024-03-11T20:53:09.536 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:09 smithi023 bash[21372]: cluster 2024-03-11T20:53:07.874553+0000 osd.1 (osd.1) 53 : cluster 0 purged_snaps scrub starts 2024-03-11T20:53:09.536 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:09 smithi023 bash[21372]: cluster 2024-03-11T20:53:07.874644+0000 osd.1 (osd.1) 54 : cluster 0 purged_snaps scrub ok 2024-03-11T20:53:09.536 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:09 smithi023 bash[21372]: cluster 2024-03-11T20:53:08.088759+0000 mon.a (mon.0) 890 : cluster 0 osdmap e111: 8 total, 8 up, 8 in 2024-03-11T20:53:09.536 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:09 smithi023 bash[21372]: audit 2024-03-11T20:53:08.429136+0000 mon.a (mon.0) 891 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:53:09.536 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:09 smithi023 bash[21372]: audit 2024-03-11T20:53:08.441716+0000 mon.a (mon.0) 892 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:53:09.536 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:09 smithi023 bash[21372]: audit 2024-03-11T20:53:08.498785+0000 mon.a (mon.0) 893 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-11T20:53:09.536 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:09 smithi023 bash[21372]: audit 2024-03-11T20:53:08.501229+0000 mon.a (mon.0) 894 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi023:9093"}]: dispatch 2024-03-11T20:53:09.536 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:09 smithi023 bash[21372]: audit 2024-03-11T20:53:08.508170+0000 mon.a (mon.0) 895 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:53:09.536 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:09 smithi023 bash[21372]: audit 2024-03-11T20:53:08.523783+0000 mon.a (mon.0) 896 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-11T20:53:09.536 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:09 smithi023 bash[21372]: audit 2024-03-11T20:53:08.531714+0000 mon.a (mon.0) 897 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi146.front.sepia.ceph.com:9095"}]: dispatch 2024-03-11T20:53:09.537 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:09 smithi023 bash[21372]: audit 2024-03-11T20:53:08.541551+0000 mon.a (mon.0) 898 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:53:09.537 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:09 smithi023 bash[21372]: audit 2024-03-11T20:53:08.552817+0000 mon.a (mon.0) 899 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-11T20:53:09.537 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:09 smithi023 bash[21372]: audit 2024-03-11T20:53:08.559738+0000 mon.a (mon.0) 900 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi146.front.sepia.ceph.com:3000"}]: dispatch 2024-03-11T20:53:09.537 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:09 smithi023 bash[21372]: audit 2024-03-11T20:53:08.564757+0000 mon.a (mon.0) 901 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:53:09.537 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:09 smithi023 bash[21372]: audit 2024-03-11T20:53:08.584359+0000 mon.a (mon.0) 902 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:53:09.537 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:09 smithi023 bash[29146]: cluster 2024-03-11T20:53:07.790208+0000 osd.5 (osd.5) 59 : cluster 0 6.19 deep-scrub starts 2024-03-11T20:53:09.537 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:09 smithi023 bash[29146]: cluster 2024-03-11T20:53:07.792686+0000 osd.5 (osd.5) 60 : cluster 0 6.19 deep-scrub ok 2024-03-11T20:53:09.537 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:09 smithi023 bash[29146]: cluster 2024-03-11T20:53:07.852958+0000 mgr.y (mgr.24380) 84 : cluster 0 pgmap v59: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T20:53:09.537 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:09 smithi023 bash[29146]: cluster 2024-03-11T20:53:07.856943+0000 osd.6 (osd.6) 33 : cluster 0 4.5 deep-scrub starts 2024-03-11T20:53:09.537 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:09 smithi023 bash[29146]: cluster 2024-03-11T20:53:07.859529+0000 osd.6 (osd.6) 34 : cluster 0 4.5 deep-scrub ok 2024-03-11T20:53:09.537 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:09 smithi023 bash[29146]: cluster 2024-03-11T20:53:07.874553+0000 osd.1 (osd.1) 53 : cluster 0 purged_snaps scrub starts 2024-03-11T20:53:09.538 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:09 smithi023 bash[29146]: cluster 2024-03-11T20:53:07.874644+0000 osd.1 (osd.1) 54 : cluster 0 purged_snaps scrub ok 2024-03-11T20:53:09.538 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:09 smithi023 bash[29146]: cluster 2024-03-11T20:53:08.088759+0000 mon.a (mon.0) 890 : cluster 0 osdmap e111: 8 total, 8 up, 8 in 2024-03-11T20:53:09.538 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:09 smithi023 bash[29146]: audit 2024-03-11T20:53:08.429136+0000 mon.a (mon.0) 891 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:53:09.538 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:09 smithi023 bash[29146]: audit 2024-03-11T20:53:08.441716+0000 mon.a (mon.0) 892 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:53:09.538 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:09 smithi023 bash[29146]: audit 2024-03-11T20:53:08.498785+0000 mon.a (mon.0) 893 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-11T20:53:09.538 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:09 smithi023 bash[29146]: audit 2024-03-11T20:53:08.501229+0000 mon.a (mon.0) 894 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi023:9093"}]: dispatch 2024-03-11T20:53:09.538 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:09 smithi023 bash[29146]: audit 2024-03-11T20:53:08.508170+0000 mon.a (mon.0) 895 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:53:09.538 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:09 smithi023 bash[29146]: audit 2024-03-11T20:53:08.523783+0000 mon.a (mon.0) 896 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-11T20:53:09.538 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:09 smithi023 bash[29146]: audit 2024-03-11T20:53:08.531714+0000 mon.a (mon.0) 897 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi146.front.sepia.ceph.com:9095"}]: dispatch 2024-03-11T20:53:09.538 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:09 smithi023 bash[29146]: audit 2024-03-11T20:53:08.541551+0000 mon.a (mon.0) 898 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:53:09.538 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:09 smithi023 bash[29146]: audit 2024-03-11T20:53:08.552817+0000 mon.a (mon.0) 899 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-11T20:53:09.538 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:09 smithi023 bash[29146]: audit 2024-03-11T20:53:08.559738+0000 mon.a (mon.0) 900 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi146.front.sepia.ceph.com:3000"}]: dispatch 2024-03-11T20:53:09.538 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:09 smithi023 bash[29146]: audit 2024-03-11T20:53:08.564757+0000 mon.a (mon.0) 901 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:53:09.539 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:09 smithi023 bash[29146]: audit 2024-03-11T20:53:08.584359+0000 mon.a (mon.0) 902 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:53:09.761 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Mar 11 20:53:09 smithi146 bash[52406]: ts=2024-03-11T20:53:09.492Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-03-11T20:53:09.761 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Mar 11 20:53:09 smithi146 bash[52406]: ts=2024-03-11T20:53:09.492Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-03-11T20:53:09.761 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Mar 11 20:53:09 smithi146 bash[52406]: ts=2024-03-11T20:53:09.492Z caller=main.go:570 level=info host_details="(Linux 5.15.0-100-generic #110-Ubuntu SMP Wed Feb 7 13:27:48 UTC 2024 x86_64 smithi146 (none))" 2024-03-11T20:53:09.761 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Mar 11 20:53:09 smithi146 bash[52406]: ts=2024-03-11T20:53:09.492Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-11T20:53:09.761 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Mar 11 20:53:09 smithi146 bash[52406]: ts=2024-03-11T20:53:09.492Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-11T20:53:09.761 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Mar 11 20:53:09 smithi146 bash[52406]: ts=2024-03-11T20:53:09.493Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-03-11T20:53:09.762 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Mar 11 20:53:09 smithi146 bash[52406]: ts=2024-03-11T20:53:09.494Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-03-11T20:53:09.762 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Mar 11 20:53:09 smithi146 bash[52406]: ts=2024-03-11T20:53:09.496Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-03-11T20:53:09.762 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Mar 11 20:53:09 smithi146 bash[52406]: ts=2024-03-11T20:53:09.496Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-03-11T20:53:09.762 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Mar 11 20:53:09 smithi146 bash[52406]: ts=2024-03-11T20:53:09.498Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-11T20:53:09.762 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Mar 11 20:53:09 smithi146 bash[52406]: ts=2024-03-11T20:53:09.498Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.779µs 2024-03-11T20:53:09.762 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Mar 11 20:53:09 smithi146 bash[52406]: ts=2024-03-11T20:53:09.498Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-11T20:53:09.762 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Mar 11 20:53:09 smithi146 bash[52406]: ts=2024-03-11T20:53:09.498Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2024-03-11T20:53:09.762 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Mar 11 20:53:09 smithi146 bash[52406]: ts=2024-03-11T20:53:09.498Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2024-03-11T20:53:09.762 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Mar 11 20:53:09 smithi146 bash[52406]: ts=2024-03-11T20:53:09.498Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=45.961µs wal_replay_duration=655.832µs wbl_replay_duration=129ns total_replay_duration=721.722µs 2024-03-11T20:53:09.762 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Mar 11 20:53:09 smithi146 bash[52406]: ts=2024-03-11T20:53:09.499Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-03-11T20:53:09.762 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Mar 11 20:53:09 smithi146 bash[52406]: ts=2024-03-11T20:53:09.499Z caller=main.go:1029 level=info msg="TSDB started" 2024-03-11T20:53:09.762 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Mar 11 20:53:09 smithi146 bash[52406]: ts=2024-03-11T20:53:09.499Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-11T20:53:09.762 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Mar 11 20:53:09 smithi146 bash[52406]: ts=2024-03-11T20:53:09.524Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=24.92605ms db_storage=992ns remote_storage=1.261µs web_handler=322ns query_engine=658ns scrape=267.209µs scrape_sd=142.449µs notify=28.785µs notify_sd=30.264µs rules=24.073814ms tracing=6.104µs 2024-03-11T20:53:09.762 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Mar 11 20:53:09 smithi146 bash[52406]: ts=2024-03-11T20:53:09.524Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-03-11T20:53:09.763 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Mar 11 20:53:09 smithi146 bash[52406]: ts=2024-03-11T20:53:09.524Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-03-11T20:53:10.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:10 smithi146 bash[23809]: cluster 2024-03-11T20:53:08.167747+0000 osd.1 (osd.1) 55 : cluster 0 6.1 scrub starts 2024-03-11T20:53:10.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:10 smithi146 bash[23809]: cluster 2024-03-11T20:53:08.170023+0000 osd.1 (osd.1) 56 : cluster 0 6.1 scrub ok 2024-03-11T20:53:10.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:10 smithi146 bash[23809]: cluster 2024-03-11T20:53:08.490969+0000 osd.3 (osd.3) 75 : cluster 0 5.0 scrub starts 2024-03-11T20:53:10.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:10 smithi146 bash[23809]: cluster 2024-03-11T20:53:08.493509+0000 osd.3 (osd.3) 76 : cluster 0 5.0 scrub ok 2024-03-11T20:53:10.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:10 smithi146 bash[23809]: audit 2024-03-11T20:53:08.499591+0000 mgr.y (mgr.24380) 85 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-11T20:53:10.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:10 smithi146 bash[23809]: audit 2024-03-11T20:53:08.501831+0000 mgr.y (mgr.24380) 86 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi023:9093"}]: dispatch 2024-03-11T20:53:10.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:10 smithi146 bash[23809]: audit 2024-03-11T20:53:08.524700+0000 mgr.y (mgr.24380) 87 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-11T20:53:10.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:10 smithi146 bash[23809]: audit 2024-03-11T20:53:08.532447+0000 mgr.y (mgr.24380) 88 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi146.front.sepia.ceph.com:9095"}]: dispatch 2024-03-11T20:53:10.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:10 smithi146 bash[23809]: audit 2024-03-11T20:53:08.553291+0000 mgr.y (mgr.24380) 89 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-11T20:53:10.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:10 smithi146 bash[23809]: audit 2024-03-11T20:53:08.560102+0000 mgr.y (mgr.24380) 90 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi146.front.sepia.ceph.com:3000"}]: dispatch 2024-03-11T20:53:10.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:10 smithi146 bash[23809]: cluster 2024-03-11T20:53:08.795756+0000 osd.2 (osd.2) 45 : cluster 0 2.c scrub starts 2024-03-11T20:53:10.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:10 smithi146 bash[23809]: cluster 2024-03-11T20:53:08.797199+0000 osd.2 (osd.2) 46 : cluster 0 2.c scrub ok 2024-03-11T20:53:10.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:10 smithi146 bash[23809]: cluster 2024-03-11T20:53:09.338653+0000 osd.7 (osd.7) 63 : cluster 0 3.f scrub starts 2024-03-11T20:53:10.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:10 smithi146 bash[23809]: cluster 2024-03-11T20:53:09.345373+0000 osd.7 (osd.7) 64 : cluster 0 3.f scrub ok 2024-03-11T20:53:10.534 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:10 smithi023 bash[29146]: cluster 2024-03-11T20:53:08.167747+0000 osd.1 (osd.1) 55 : cluster 0 6.1 scrub starts 2024-03-11T20:53:10.534 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:10 smithi023 bash[29146]: cluster 2024-03-11T20:53:08.170023+0000 osd.1 (osd.1) 56 : cluster 0 6.1 scrub ok 2024-03-11T20:53:10.534 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:10 smithi023 bash[29146]: cluster 2024-03-11T20:53:08.490969+0000 osd.3 (osd.3) 75 : cluster 0 5.0 scrub starts 2024-03-11T20:53:10.534 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:10 smithi023 bash[29146]: cluster 2024-03-11T20:53:08.493509+0000 osd.3 (osd.3) 76 : cluster 0 5.0 scrub ok 2024-03-11T20:53:10.534 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:10 smithi023 bash[29146]: audit 2024-03-11T20:53:08.499591+0000 mgr.y (mgr.24380) 85 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-11T20:53:10.534 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:10 smithi023 bash[29146]: audit 2024-03-11T20:53:08.501831+0000 mgr.y (mgr.24380) 86 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi023:9093"}]: dispatch 2024-03-11T20:53:10.534 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:10 smithi023 bash[29146]: audit 2024-03-11T20:53:08.524700+0000 mgr.y (mgr.24380) 87 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-11T20:53:10.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:10 smithi023 bash[29146]: audit 2024-03-11T20:53:08.532447+0000 mgr.y (mgr.24380) 88 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi146.front.sepia.ceph.com:9095"}]: dispatch 2024-03-11T20:53:10.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:10 smithi023 bash[29146]: audit 2024-03-11T20:53:08.553291+0000 mgr.y (mgr.24380) 89 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-11T20:53:10.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:10 smithi023 bash[29146]: audit 2024-03-11T20:53:08.560102+0000 mgr.y (mgr.24380) 90 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi146.front.sepia.ceph.com:3000"}]: dispatch 2024-03-11T20:53:10.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:10 smithi023 bash[29146]: cluster 2024-03-11T20:53:08.795756+0000 osd.2 (osd.2) 45 : cluster 0 2.c scrub starts 2024-03-11T20:53:10.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:10 smithi023 bash[29146]: cluster 2024-03-11T20:53:08.797199+0000 osd.2 (osd.2) 46 : cluster 0 2.c scrub ok 2024-03-11T20:53:10.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:10 smithi023 bash[29146]: cluster 2024-03-11T20:53:09.338653+0000 osd.7 (osd.7) 63 : cluster 0 3.f scrub starts 2024-03-11T20:53:10.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:10 smithi023 bash[29146]: cluster 2024-03-11T20:53:09.345373+0000 osd.7 (osd.7) 64 : cluster 0 3.f scrub ok 2024-03-11T20:53:10.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:10 smithi023 bash[21372]: cluster 2024-03-11T20:53:08.167747+0000 osd.1 (osd.1) 55 : cluster 0 6.1 scrub starts 2024-03-11T20:53:10.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:10 smithi023 bash[21372]: cluster 2024-03-11T20:53:08.170023+0000 osd.1 (osd.1) 56 : cluster 0 6.1 scrub ok 2024-03-11T20:53:10.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:10 smithi023 bash[21372]: cluster 2024-03-11T20:53:08.490969+0000 osd.3 (osd.3) 75 : cluster 0 5.0 scrub starts 2024-03-11T20:53:10.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:10 smithi023 bash[21372]: cluster 2024-03-11T20:53:08.493509+0000 osd.3 (osd.3) 76 : cluster 0 5.0 scrub ok 2024-03-11T20:53:10.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:10 smithi023 bash[21372]: audit 2024-03-11T20:53:08.499591+0000 mgr.y (mgr.24380) 85 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-11T20:53:10.536 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:10 smithi023 bash[21372]: audit 2024-03-11T20:53:08.501831+0000 mgr.y (mgr.24380) 86 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi023:9093"}]: dispatch 2024-03-11T20:53:10.536 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:10 smithi023 bash[21372]: audit 2024-03-11T20:53:08.524700+0000 mgr.y (mgr.24380) 87 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-11T20:53:10.536 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:10 smithi023 bash[21372]: audit 2024-03-11T20:53:08.532447+0000 mgr.y (mgr.24380) 88 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi146.front.sepia.ceph.com:9095"}]: dispatch 2024-03-11T20:53:10.536 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:10 smithi023 bash[21372]: audit 2024-03-11T20:53:08.553291+0000 mgr.y (mgr.24380) 89 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-11T20:53:10.536 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:10 smithi023 bash[21372]: audit 2024-03-11T20:53:08.560102+0000 mgr.y (mgr.24380) 90 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi146.front.sepia.ceph.com:3000"}]: dispatch 2024-03-11T20:53:10.536 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:10 smithi023 bash[21372]: cluster 2024-03-11T20:53:08.795756+0000 osd.2 (osd.2) 45 : cluster 0 2.c scrub starts 2024-03-11T20:53:10.536 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:10 smithi023 bash[21372]: cluster 2024-03-11T20:53:08.797199+0000 osd.2 (osd.2) 46 : cluster 0 2.c scrub ok 2024-03-11T20:53:10.536 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:10 smithi023 bash[21372]: cluster 2024-03-11T20:53:09.338653+0000 osd.7 (osd.7) 63 : cluster 0 3.f scrub starts 2024-03-11T20:53:10.536 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:10 smithi023 bash[21372]: cluster 2024-03-11T20:53:09.345373+0000 osd.7 (osd.7) 64 : cluster 0 3.f scrub ok 2024-03-11T20:53:11.039 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:53:11.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:11 smithi146 bash[23809]: cluster 2024-03-11T20:53:09.854287+0000 mgr.y (mgr.24380) 91 : cluster 0 pgmap v61: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:53:11.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:11 smithi146 bash[23809]: cluster 2024-03-11T20:53:10.060047+0000 osd.0 (osd.0) 67 : cluster 0 4.1e scrub starts 2024-03-11T20:53:11.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:11 smithi146 bash[23809]: cluster 2024-03-11T20:53:10.062336+0000 osd.0 (osd.0) 68 : cluster 0 4.1e scrub ok 2024-03-11T20:53:11.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:11 smithi146 bash[23809]: cluster 2024-03-11T20:53:10.391917+0000 osd.7 (osd.7) 65 : cluster 0 purged_snaps scrub starts 2024-03-11T20:53:11.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:11 smithi146 bash[23809]: cluster 2024-03-11T20:53:10.392048+0000 osd.7 (osd.7) 66 : cluster 0 purged_snaps scrub ok 2024-03-11T20:53:11.534 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:11 smithi023 bash[29146]: cluster 2024-03-11T20:53:09.854287+0000 mgr.y (mgr.24380) 91 : cluster 0 pgmap v61: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:53:11.534 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:11 smithi023 bash[29146]: cluster 2024-03-11T20:53:10.060047+0000 osd.0 (osd.0) 67 : cluster 0 4.1e scrub starts 2024-03-11T20:53:11.534 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:11 smithi023 bash[29146]: cluster 2024-03-11T20:53:10.062336+0000 osd.0 (osd.0) 68 : cluster 0 4.1e scrub ok 2024-03-11T20:53:11.534 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:11 smithi023 bash[29146]: cluster 2024-03-11T20:53:10.391917+0000 osd.7 (osd.7) 65 : cluster 0 purged_snaps scrub starts 2024-03-11T20:53:11.534 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:11 smithi023 bash[29146]: cluster 2024-03-11T20:53:10.392048+0000 osd.7 (osd.7) 66 : cluster 0 purged_snaps scrub ok 2024-03-11T20:53:11.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:11 smithi023 bash[21372]: cluster 2024-03-11T20:53:09.854287+0000 mgr.y (mgr.24380) 91 : cluster 0 pgmap v61: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:53:11.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:11 smithi023 bash[21372]: cluster 2024-03-11T20:53:10.060047+0000 osd.0 (osd.0) 67 : cluster 0 4.1e scrub starts 2024-03-11T20:53:11.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:11 smithi023 bash[21372]: cluster 2024-03-11T20:53:10.062336+0000 osd.0 (osd.0) 68 : cluster 0 4.1e scrub ok 2024-03-11T20:53:11.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:11 smithi023 bash[21372]: cluster 2024-03-11T20:53:10.391917+0000 osd.7 (osd.7) 65 : cluster 0 purged_snaps scrub starts 2024-03-11T20:53:11.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:11 smithi023 bash[21372]: cluster 2024-03-11T20:53:10.392048+0000 osd.7 (osd.7) 66 : cluster 0 purged_snaps scrub ok 2024-03-11T20:53:12.250 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:12 smithi023 bash[21372]: cluster 2024-03-11T20:53:11.096033+0000 mon.a (mon.0) 903 : cluster 0 osdmap e112: 8 total, 8 up, 8 in 2024-03-11T20:53:12.250 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:12 smithi023 bash[29146]: cluster 2024-03-11T20:53:11.096033+0000 mon.a (mon.0) 903 : cluster 0 osdmap e112: 8 total, 8 up, 8 in 2024-03-11T20:53:12.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:12 smithi146 bash[23809]: cluster 2024-03-11T20:53:11.096033+0000 mon.a (mon.0) 903 : cluster 0 osdmap e112: 8 total, 8 up, 8 in 2024-03-11T20:53:13.398 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:13 smithi023 bash[21372]: cluster 2024-03-11T20:53:11.425161+0000 osd.3 (osd.3) 77 : cluster 0 3.1e scrub starts 2024-03-11T20:53:13.398 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:13 smithi023 bash[21372]: cluster 2024-03-11T20:53:11.429939+0000 osd.3 (osd.3) 78 : cluster 0 3.1e scrub ok 2024-03-11T20:53:13.398 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:13 smithi023 bash[21372]: cluster 2024-03-11T20:53:11.836901+0000 osd.5 (osd.5) 61 : cluster 0 4.15 deep-scrub starts 2024-03-11T20:53:13.398 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:13 smithi023 bash[21372]: cluster 2024-03-11T20:53:11.843919+0000 osd.5 (osd.5) 62 : cluster 0 4.15 deep-scrub ok 2024-03-11T20:53:13.398 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:13 smithi023 bash[21372]: cluster 2024-03-11T20:53:11.854971+0000 mgr.y (mgr.24380) 92 : cluster 0 pgmap v63: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:53:13.398 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:13 smithi023 bash[29146]: cluster 2024-03-11T20:53:11.425161+0000 osd.3 (osd.3) 77 : cluster 0 3.1e scrub starts 2024-03-11T20:53:13.398 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:13 smithi023 bash[29146]: cluster 2024-03-11T20:53:11.429939+0000 osd.3 (osd.3) 78 : cluster 0 3.1e scrub ok 2024-03-11T20:53:13.398 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:13 smithi023 bash[29146]: cluster 2024-03-11T20:53:11.836901+0000 osd.5 (osd.5) 61 : cluster 0 4.15 deep-scrub starts 2024-03-11T20:53:13.398 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:13 smithi023 bash[29146]: cluster 2024-03-11T20:53:11.843919+0000 osd.5 (osd.5) 62 : cluster 0 4.15 deep-scrub ok 2024-03-11T20:53:13.399 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:13 smithi023 bash[29146]: cluster 2024-03-11T20:53:11.854971+0000 mgr.y (mgr.24380) 92 : cluster 0 pgmap v63: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:53:13.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:13 smithi146 bash[23809]: cluster 2024-03-11T20:53:11.425161+0000 osd.3 (osd.3) 77 : cluster 0 3.1e scrub starts 2024-03-11T20:53:13.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:13 smithi146 bash[23809]: cluster 2024-03-11T20:53:11.429939+0000 osd.3 (osd.3) 78 : cluster 0 3.1e scrub ok 2024-03-11T20:53:13.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:13 smithi146 bash[23809]: cluster 2024-03-11T20:53:11.836901+0000 osd.5 (osd.5) 61 : cluster 0 4.15 deep-scrub starts 2024-03-11T20:53:13.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:13 smithi146 bash[23809]: cluster 2024-03-11T20:53:11.843919+0000 osd.5 (osd.5) 62 : cluster 0 4.15 deep-scrub ok 2024-03-11T20:53:13.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:13 smithi146 bash[23809]: cluster 2024-03-11T20:53:11.854971+0000 mgr.y (mgr.24380) 92 : cluster 0 pgmap v63: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:53:13.791 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-11T20:53:13.793 INFO:teuthology.orchestra.run.smithi023.stderr:dumped all 2024-03-11T20:53:14.449 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:14 smithi023 bash[21372]: cluster 2024-03-11T20:53:12.818458+0000 osd.5 (osd.5) 63 : cluster 0 3.c scrub starts 2024-03-11T20:53:14.449 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:14 smithi023 bash[21372]: cluster 2024-03-11T20:53:12.822831+0000 osd.5 (osd.5) 64 : cluster 0 3.c scrub ok 2024-03-11T20:53:14.449 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:14 smithi023 bash[21372]: cluster 2024-03-11T20:53:13.134307+0000 osd.1 (osd.1) 57 : cluster 0 3.19 scrub starts 2024-03-11T20:53:14.449 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:14 smithi023 bash[21372]: cluster 2024-03-11T20:53:13.138903+0000 osd.1 (osd.1) 58 : cluster 0 3.19 scrub ok 2024-03-11T20:53:14.449 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:14 smithi023 bash[21372]: cluster 2024-03-11T20:53:13.555554+0000 osd.4 (osd.4) 59 : cluster 0 3.3 scrub starts 2024-03-11T20:53:14.449 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:14 smithi023 bash[21372]: cluster 2024-03-11T20:53:13.560296+0000 osd.4 (osd.4) 60 : cluster 0 3.3 scrub ok 2024-03-11T20:53:14.449 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:14 smithi023 bash[21372]: audit 2024-03-11T20:53:13.821297+0000 mon.a (mon.0) 904 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:53:14.449 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:14 smithi023 bash[21372]: audit 2024-03-11T20:53:13.828165+0000 mon.a (mon.0) 905 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:53:14.449 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:14 smithi023 bash[29146]: cluster 2024-03-11T20:53:12.818458+0000 osd.5 (osd.5) 63 : cluster 0 3.c scrub starts 2024-03-11T20:53:14.450 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:14 smithi023 bash[29146]: cluster 2024-03-11T20:53:12.822831+0000 osd.5 (osd.5) 64 : cluster 0 3.c scrub ok 2024-03-11T20:53:14.450 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:14 smithi023 bash[29146]: cluster 2024-03-11T20:53:13.134307+0000 osd.1 (osd.1) 57 : cluster 0 3.19 scrub starts 2024-03-11T20:53:14.450 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:14 smithi023 bash[29146]: cluster 2024-03-11T20:53:13.138903+0000 osd.1 (osd.1) 58 : cluster 0 3.19 scrub ok 2024-03-11T20:53:14.450 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:14 smithi023 bash[29146]: cluster 2024-03-11T20:53:13.555554+0000 osd.4 (osd.4) 59 : cluster 0 3.3 scrub starts 2024-03-11T20:53:14.450 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:14 smithi023 bash[29146]: cluster 2024-03-11T20:53:13.560296+0000 osd.4 (osd.4) 60 : cluster 0 3.3 scrub ok 2024-03-11T20:53:14.450 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:14 smithi023 bash[29146]: audit 2024-03-11T20:53:13.821297+0000 mon.a (mon.0) 904 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:53:14.450 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:14 smithi023 bash[29146]: audit 2024-03-11T20:53:13.828165+0000 mon.a (mon.0) 905 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:53:14.451 INFO:teuthology.orchestra.run.smithi023.stdout:{"pg_ready":true,"pg_map":{"version":63,"stamp":"2024-03-11T20:53:11.854573+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596392,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1088,"num_read_kb":934,"num_write":615,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":664,"ondisk_log_size":664,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":253364,"kb_used_data":8092,"kb_used_omap":12,"kb_used_meta":245171,"kb_avail":749707852,"statfs":{"total":767960285184,"available":767700840448,"internally_reserved":0,"allocated":8286208,"data_stored":4990696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":251055666},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.004251"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":84,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.084380+0000","last_change":"2024-03-11T20:53:05.510279+0000","last_active":"2024-03-11T20:53:08.084380+0000","last_peered":"2024-03-11T20:53:08.084380+0000","last_clean":"2024-03-11T20:53:08.084380+0000","last_became_active":"2024-03-11T20:50:34.338390+0000","last_became_peered":"2024-03-11T20:50:34.338390+0000","last_unstale":"2024-03-11T20:53:08.084380+0000","last_undegraded":"2024-03-11T20:53:08.084380+0000","last_fullsized":"2024-03-11T20:53:08.084380+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:05.510060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:53:05.510060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:21.840101+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":80,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.086155+0000","last_change":"2024-03-11T20:51:53.289821+0000","last_active":"2024-03-11T20:53:08.086155+0000","last_peered":"2024-03-11T20:53:08.086155+0000","last_clean":"2024-03-11T20:53:08.086155+0000","last_became_active":"2024-03-11T20:50:32.210194+0000","last_became_peered":"2024-03-11T20:50:32.210194+0000","last_unstale":"2024-03-11T20:53:08.086155+0000","last_undegraded":"2024-03-11T20:53:08.086155+0000","last_fullsized":"2024-03-11T20:53:08.086155+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:53.289737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:51:53.289737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:17.298542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":87,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.118206+0000","last_change":"2024-03-11T20:52:12.816303+0000","last_active":"2024-03-11T20:53:08.118206+0000","last_peered":"2024-03-11T20:53:08.118206+0000","last_clean":"2024-03-11T20:53:08.118206+0000","last_became_active":"2024-03-11T20:50:28.061739+0000","last_became_peered":"2024-03-11T20:50:28.061739+0000","last_unstale":"2024-03-11T20:53:08.118206+0000","last_undegraded":"2024-03-11T20:53:08.118206+0000","last_fullsized":"2024-03-11T20:53:08.118206+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:12.816104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:52:12.816104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:24.698876+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"84'10","reported_seq":99,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.088547+0000","last_change":"2024-03-11T20:51:46.306937+0000","last_active":"2024-03-11T20:53:08.088547+0000","last_peered":"2024-03-11T20:53:08.088547+0000","last_clean":"2024-03-11T20:53:08.088547+0000","last_became_active":"2024-03-11T20:50:30.101721+0000","last_became_peered":"2024-03-11T20:50:30.101721+0000","last_unstale":"2024-03-11T20:53:08.088547+0000","last_undegraded":"2024-03-11T20:53:08.088547+0000","last_fullsized":"2024-03-11T20:53:08.088547+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'10","last_scrub_stamp":"2024-03-11T20:51:46.306683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:51:46.306683+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:08.780340+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":84,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.117767+0000","last_change":"2024-03-11T20:53:07.560827+0000","last_active":"2024-03-11T20:53:08.117767+0000","last_peered":"2024-03-11T20:53:08.117767+0000","last_clean":"2024-03-11T20:53:08.117767+0000","last_became_active":"2024-03-11T20:50:34.318627+0000","last_became_peered":"2024-03-11T20:50:34.318627+0000","last_unstale":"2024-03-11T20:53:08.117767+0000","last_undegraded":"2024-03-11T20:53:08.117767+0000","last_fullsized":"2024-03-11T20:53:08.117767+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:07.560731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:55.545758+0000","last_clean_scrub_stamp":"2024-03-11T20:53:07.560731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:36.783905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":88,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.118068+0000","last_change":"2024-03-11T20:52:49.461204+0000","last_active":"2024-03-11T20:53:08.118068+0000","last_peered":"2024-03-11T20:53:08.118068+0000","last_clean":"2024-03-11T20:53:08.118068+0000","last_became_active":"2024-03-11T20:50:32.197413+0000","last_became_peered":"2024-03-11T20:50:32.197413+0000","last_unstale":"2024-03-11T20:53:08.118068+0000","last_undegraded":"2024-03-11T20:53:08.118068+0000","last_fullsized":"2024-03-11T20:53:08.118068+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:49.460993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:52:49.460993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:04.898406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":88,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.087892+0000","last_change":"2024-03-11T20:52:26.396960+0000","last_active":"2024-03-11T20:53:08.087892+0000","last_peered":"2024-03-11T20:53:08.087892+0000","last_clean":"2024-03-11T20:53:08.087892+0000","last_became_active":"2024-03-11T20:50:28.058274+0000","last_became_peered":"2024-03-11T20:50:28.058274+0000","last_unstale":"2024-03-11T20:53:08.087892+0000","last_undegraded":"2024-03-11T20:53:08.087892+0000","last_fullsized":"2024-03-11T20:53:08.087892+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:26.396865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:52:26.396865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:52.964932+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"84'11","reported_seq":102,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.117872+0000","last_change":"2024-03-11T20:52:15.868867+0000","last_active":"2024-03-11T20:53:08.117872+0000","last_peered":"2024-03-11T20:53:08.117872+0000","last_clean":"2024-03-11T20:53:08.117872+0000","last_became_active":"2024-03-11T20:50:30.092638+0000","last_became_peered":"2024-03-11T20:50:30.092638+0000","last_unstale":"2024-03-11T20:53:08.117872+0000","last_undegraded":"2024-03-11T20:53:08.117872+0000","last_fullsized":"2024-03-11T20:53:08.117872+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T20:52:15.868599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:52:15.868599+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:44.240121+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":84,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.119395+0000","last_change":"2024-03-11T20:53:07.792944+0000","last_active":"2024-03-11T20:53:08.119395+0000","last_peered":"2024-03-11T20:53:08.119395+0000","last_clean":"2024-03-11T20:53:08.119395+0000","last_became_active":"2024-03-11T20:50:34.334495+0000","last_became_peered":"2024-03-11T20:50:34.334495+0000","last_unstale":"2024-03-11T20:53:08.119395+0000","last_undegraded":"2024-03-11T20:53:08.119395+0000","last_fullsized":"2024-03-11T20:53:08.119395+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:07.792697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:53:07.792697+0000","last_clean_scrub_stamp":"2024-03-11T20:53:07.792697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:23.012728+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":80,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.120037+0000","last_change":"2024-03-11T20:52:03.583828+0000","last_active":"2024-03-11T20:53:08.120037+0000","last_peered":"2024-03-11T20:53:08.120037+0000","last_clean":"2024-03-11T20:53:08.120037+0000","last_became_active":"2024-03-11T20:50:32.199561+0000","last_became_peered":"2024-03-11T20:50:32.199561+0000","last_unstale":"2024-03-11T20:53:08.120037+0000","last_undegraded":"2024-03-11T20:53:08.120037+0000","last_fullsized":"2024-03-11T20:53:08.120037+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:03.583594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:52:03.583594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:22.995130+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":88,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.117652+0000","last_change":"2024-03-11T20:52:06.012514+0000","last_active":"2024-03-11T20:53:08.117652+0000","last_peered":"2024-03-11T20:53:08.117652+0000","last_clean":"2024-03-11T20:53:08.117652+0000","last_became_active":"2024-03-11T20:50:28.079225+0000","last_became_peered":"2024-03-11T20:50:28.079225+0000","last_unstale":"2024-03-11T20:53:08.117652+0000","last_undegraded":"2024-03-11T20:53:08.117652+0000","last_fullsized":"2024-03-11T20:53:08.117652+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:06.012226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:52:06.012226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:18.639749+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"84'15","reported_seq":109,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.121105+0000","last_change":"2024-03-11T20:51:50.842138+0000","last_active":"2024-03-11T20:53:08.121105+0000","last_peered":"2024-03-11T20:53:08.121105+0000","last_clean":"2024-03-11T20:53:08.121105+0000","last_became_active":"2024-03-11T20:50:30.105354+0000","last_became_peered":"2024-03-11T20:50:30.105354+0000","last_unstale":"2024-03-11T20:53:08.121105+0000","last_undegraded":"2024-03-11T20:53:08.121105+0000","last_fullsized":"2024-03-11T20:53:08.121105+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15","last_scrub_stamp":"2024-03-11T20:51:50.841902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:51:50.841902+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:16.306055+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":75,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.113471+0000","last_change":"2024-03-11T20:51:43.910363+0000","last_active":"2024-03-11T20:53:08.113471+0000","last_peered":"2024-03-11T20:53:08.113471+0000","last_clean":"2024-03-11T20:53:08.113471+0000","last_became_active":"2024-03-11T20:50:34.333230+0000","last_became_peered":"2024-03-11T20:50:34.333230+0000","last_unstale":"2024-03-11T20:53:08.113471+0000","last_undegraded":"2024-03-11T20:53:08.113471+0000","last_fullsized":"2024-03-11T20:53:08.113471+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:43.910075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:51:43.910075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:12.992446+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":80,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.118847+0000","last_change":"2024-03-11T20:52:19.502063+0000","last_active":"2024-03-11T20:53:08.118847+0000","last_peered":"2024-03-11T20:53:08.118847+0000","last_clean":"2024-03-11T20:53:08.118847+0000","last_became_active":"2024-03-11T20:50:32.200126+0000","last_became_peered":"2024-03-11T20:50:32.200126+0000","last_unstale":"2024-03-11T20:53:08.118847+0000","last_undegraded":"2024-03-11T20:53:08.118847+0000","last_fullsized":"2024-03-11T20:53:08.118847+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:19.501832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:52:19.501832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:26.456710+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":96,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.122345+0000","last_change":"2024-03-11T20:53:00.210016+0000","last_active":"2024-03-11T20:53:08.122345+0000","last_peered":"2024-03-11T20:53:08.122345+0000","last_clean":"2024-03-11T20:53:08.122345+0000","last_became_active":"2024-03-11T20:50:28.068978+0000","last_became_peered":"2024-03-11T20:50:28.068978+0000","last_unstale":"2024-03-11T20:53:08.122345+0000","last_undegraded":"2024-03-11T20:53:08.122345+0000","last_fullsized":"2024-03-11T20:53:08.122345+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:00.209791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:53:00.209791+0000","last_clean_scrub_stamp":"2024-03-11T20:53:00.209791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:01.669184+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"110'17","reported_seq":120,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.120151+0000","last_change":"2024-03-11T20:53:04.802853+0000","last_active":"2024-03-11T20:53:08.120151+0000","last_peered":"2024-03-11T20:53:08.120151+0000","last_clean":"2024-03-11T20:53:08.120151+0000","last_became_active":"2024-03-11T20:50:30.108364+0000","last_became_peered":"2024-03-11T20:50:30.108364+0000","last_unstale":"2024-03-11T20:53:08.120151+0000","last_undegraded":"2024-03-11T20:53:08.120151+0000","last_fullsized":"2024-03-11T20:53:08.120151+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'17","last_scrub_stamp":"2024-03-11T20:53:04.802488+0000","last_deep_scrub":"110'17","last_deep_scrub_stamp":"2024-03-11T20:53:04.802488+0000","last_clean_scrub_stamp":"2024-03-11T20:53:04.802488+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:26.988743+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":76,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.089353+0000","last_change":"2024-03-11T20:52:37.430006+0000","last_active":"2024-03-11T20:53:08.089353+0000","last_peered":"2024-03-11T20:53:08.089353+0000","last_clean":"2024-03-11T20:53:08.089353+0000","last_became_active":"2024-03-11T20:50:34.334558+0000","last_became_peered":"2024-03-11T20:50:34.334558+0000","last_unstale":"2024-03-11T20:53:08.089353+0000","last_undegraded":"2024-03-11T20:53:08.089353+0000","last_fullsized":"2024-03-11T20:53:08.089353+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:37.429729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:52:37.429729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:37.737857+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":88,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.090274+0000","last_change":"2024-03-11T20:52:59.455887+0000","last_active":"2024-03-11T20:53:08.090274+0000","last_peered":"2024-03-11T20:53:08.090274+0000","last_clean":"2024-03-11T20:53:08.090274+0000","last_became_active":"2024-03-11T20:50:32.202533+0000","last_became_peered":"2024-03-11T20:50:32.202533+0000","last_unstale":"2024-03-11T20:53:08.090274+0000","last_undegraded":"2024-03-11T20:53:08.090274+0000","last_fullsized":"2024-03-11T20:53:08.090274+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:59.455677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:52:59.455677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:29.453221+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"77'1","reported_seq":96,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.090889+0000","last_change":"2024-03-11T20:51:59.384746+0000","last_active":"2024-03-11T20:53:08.090889+0000","last_peered":"2024-03-11T20:53:08.090889+0000","last_clean":"2024-03-11T20:53:08.090889+0000","last_became_active":"2024-03-11T20:50:28.068840+0000","last_became_peered":"2024-03-11T20:50:28.068840+0000","last_unstale":"2024-03-11T20:53:08.090889+0000","last_undegraded":"2024-03-11T20:53:08.090889+0000","last_fullsized":"2024-03-11T20:53:08.090889+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-03-11T20:51:59.384636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:51:59.384636+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:26.361222+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"84'9","reported_seq":100,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.118120+0000","last_change":"2024-03-11T20:51:50.570896+0000","last_active":"2024-03-11T20:53:08.118120+0000","last_peered":"2024-03-11T20:53:08.118120+0000","last_clean":"2024-03-11T20:53:08.118120+0000","last_became_active":"2024-03-11T20:50:30.086169+0000","last_became_peered":"2024-03-11T20:50:30.086169+0000","last_unstale":"2024-03-11T20:53:08.118120+0000","last_undegraded":"2024-03-11T20:53:08.118120+0000","last_fullsized":"2024-03-11T20:53:08.118120+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'9","last_scrub_stamp":"2024-03-11T20:51:50.570657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:51:50.570657+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:15.854580+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":76,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.123647+0000","last_change":"2024-03-11T20:52:07.500890+0000","last_active":"2024-03-11T20:53:08.123647+0000","last_peered":"2024-03-11T20:53:08.123647+0000","last_clean":"2024-03-11T20:53:08.123647+0000","last_became_active":"2024-03-11T20:50:34.321714+0000","last_became_peered":"2024-03-11T20:50:34.321714+0000","last_unstale":"2024-03-11T20:53:08.123647+0000","last_undegraded":"2024-03-11T20:53:08.123647+0000","last_fullsized":"2024-03-11T20:53:08.123647+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:07.500759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:52:07.500759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:09.208355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":80,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.116199+0000","last_change":"2024-03-11T20:51:48.875085+0000","last_active":"2024-03-11T20:53:08.116199+0000","last_peered":"2024-03-11T20:53:08.116199+0000","last_clean":"2024-03-11T20:53:08.116199+0000","last_became_active":"2024-03-11T20:50:32.192704+0000","last_became_peered":"2024-03-11T20:50:32.192704+0000","last_unstale":"2024-03-11T20:53:08.116199+0000","last_undegraded":"2024-03-11T20:53:08.116199+0000","last_fullsized":"2024-03-11T20:53:08.116199+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:48.874867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:51:48.874867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:10.854648+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":88,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.086524+0000","last_change":"2024-03-11T20:51:59.680978+0000","last_active":"2024-03-11T20:53:08.086524+0000","last_peered":"2024-03-11T20:53:08.086524+0000","last_clean":"2024-03-11T20:53:08.086524+0000","last_became_active":"2024-03-11T20:50:28.081538+0000","last_became_peered":"2024-03-11T20:50:28.081538+0000","last_unstale":"2024-03-11T20:53:08.086524+0000","last_undegraded":"2024-03-11T20:53:08.086524+0000","last_fullsized":"2024-03-11T20:53:08.086524+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:59.680629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:51:59.680629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:09.502304+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"84'5","reported_seq":93,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.116663+0000","last_change":"2024-03-11T20:52:00.809218+0000","last_active":"2024-03-11T20:53:08.116663+0000","last_peered":"2024-03-11T20:53:08.116663+0000","last_clean":"2024-03-11T20:53:08.116663+0000","last_became_active":"2024-03-11T20:50:30.099866+0000","last_became_peered":"2024-03-11T20:50:30.099866+0000","last_unstale":"2024-03-11T20:53:08.116663+0000","last_undegraded":"2024-03-11T20:53:08.116663+0000","last_fullsized":"2024-03-11T20:53:08.116663+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'5","last_scrub_stamp":"2024-03-11T20:52:00.808991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:52:00.808991+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:29.806692+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":76,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.088702+0000","last_change":"2024-03-11T20:52:02.142598+0000","last_active":"2024-03-11T20:53:08.088702+0000","last_peered":"2024-03-11T20:53:08.088702+0000","last_clean":"2024-03-11T20:53:08.088702+0000","last_became_active":"2024-03-11T20:50:34.307730+0000","last_became_peered":"2024-03-11T20:50:34.307730+0000","last_unstale":"2024-03-11T20:53:08.088702+0000","last_undegraded":"2024-03-11T20:53:08.088702+0000","last_fullsized":"2024-03-11T20:53:08.088702+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:02.142361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:52:02.142361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:09.439702+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1f","version":"84'11","reported_seq":129,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.086882+0000","last_change":"2024-03-11T20:52:12.655265+0000","last_active":"2024-03-11T20:53:08.086882+0000","last_peered":"2024-03-11T20:53:08.086882+0000","last_clean":"2024-03-11T20:53:08.086882+0000","last_became_active":"2024-03-11T20:50:32.200195+0000","last_became_peered":"2024-03-11T20:50:32.200195+0000","last_unstale":"2024-03-11T20:53:08.086882+0000","last_undegraded":"2024-03-11T20:53:08.086882+0000","last_fullsized":"2024-03-11T20:53:08.086882+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T20:52:12.655061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:52:12.655061+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:22.459298+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"77'1","reported_seq":89,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.086293+0000","last_change":"2024-03-11T20:52:24.397205+0000","last_active":"2024-03-11T20:53:08.086293+0000","last_peered":"2024-03-11T20:53:08.086293+0000","last_clean":"2024-03-11T20:53:08.086293+0000","last_became_active":"2024-03-11T20:50:28.068827+0000","last_became_peered":"2024-03-11T20:50:28.068827+0000","last_unstale":"2024-03-11T20:53:08.086293+0000","last_undegraded":"2024-03-11T20:53:08.086293+0000","last_fullsized":"2024-03-11T20:53:08.086293+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-03-11T20:52:24.396881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:52:24.396881+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:53.894381+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"84'9","reported_seq":100,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.085025+0000","last_change":"2024-03-11T20:52:05.438999+0000","last_active":"2024-03-11T20:53:08.085025+0000","last_peered":"2024-03-11T20:53:08.085025+0000","last_clean":"2024-03-11T20:53:08.085025+0000","last_became_active":"2024-03-11T20:50:30.083936+0000","last_became_peered":"2024-03-11T20:50:30.083936+0000","last_unstale":"2024-03-11T20:53:08.085025+0000","last_undegraded":"2024-03-11T20:53:08.085025+0000","last_fullsized":"2024-03-11T20:53:08.085025+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'9","last_scrub_stamp":"2024-03-11T20:52:05.438658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:52:05.438658+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:19.765612+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"105'2","reported_seq":79,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.121685+0000","last_change":"2024-03-11T20:52:18.099224+0000","last_active":"2024-03-11T20:53:08.121685+0000","last_peered":"2024-03-11T20:53:08.121685+0000","last_clean":"2024-03-11T20:53:08.121685+0000","last_became_active":"2024-03-11T20:50:34.327356+0000","last_became_peered":"2024-03-11T20:50:34.327356+0000","last_unstale":"2024-03-11T20:53:08.121685+0000","last_undegraded":"2024-03-11T20:53:08.121685+0000","last_fullsized":"2024-03-11T20:53:08.121685+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'2","last_scrub_stamp":"2024-03-11T20:52:18.098911+0000","last_deep_scrub":"105'2","last_deep_scrub_stamp":"2024-03-11T20:52:18.098911+0000","last_clean_scrub_stamp":"2024-03-11T20:52:18.098911+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:21.277529+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":79,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.117639+0000","last_change":"2024-03-11T20:51:49.917165+0000","last_active":"2024-03-11T20:53:08.117639+0000","last_peered":"2024-03-11T20:53:08.117639+0000","last_clean":"2024-03-11T20:53:08.117639+0000","last_became_active":"2024-03-11T20:50:32.648869+0000","last_became_peered":"2024-03-11T20:50:32.648869+0000","last_unstale":"2024-03-11T20:53:08.117639+0000","last_undegraded":"2024-03-11T20:53:08.117639+0000","last_fullsized":"2024-03-11T20:53:08.117639+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:49.916962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:51:49.916962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:09.543460+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":88,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.118473+0000","last_change":"2024-03-11T20:52:01.880353+0000","last_active":"2024-03-11T20:53:08.118473+0000","last_peered":"2024-03-11T20:53:08.118473+0000","last_clean":"2024-03-11T20:53:08.118473+0000","last_became_active":"2024-03-11T20:50:28.066124+0000","last_became_peered":"2024-03-11T20:50:28.066124+0000","last_unstale":"2024-03-11T20:53:08.118473+0000","last_undegraded":"2024-03-11T20:53:08.118473+0000","last_fullsized":"2024-03-11T20:53:08.118473+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:01.880052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:52:01.880052+0000","last_clean_scrub_stamp":"2024-03-11T20:52:01.880052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:22.449810+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"84'15","reported_seq":109,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.085028+0000","last_change":"2024-03-11T20:51:52.048351+0000","last_active":"2024-03-11T20:53:08.085028+0000","last_peered":"2024-03-11T20:53:08.085028+0000","last_clean":"2024-03-11T20:53:08.085028+0000","last_became_active":"2024-03-11T20:50:30.088446+0000","last_became_peered":"2024-03-11T20:50:30.088446+0000","last_unstale":"2024-03-11T20:53:08.085028+0000","last_undegraded":"2024-03-11T20:53:08.085028+0000","last_fullsized":"2024-03-11T20:53:08.085028+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15","last_scrub_stamp":"2024-03-11T20:51:52.048191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:51:52.048191+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":76,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.091261+0000","last_change":"2024-03-11T20:52:11.462704+0000","last_active":"2024-03-11T20:53:08.091261+0000","last_peered":"2024-03-11T20:53:08.091261+0000","last_clean":"2024-03-11T20:53:08.091261+0000","last_became_active":"2024-03-11T20:50:34.337878+0000","last_became_peered":"2024-03-11T20:50:34.337878+0000","last_unstale":"2024-03-11T20:53:08.091261+0000","last_undegraded":"2024-03-11T20:53:08.091261+0000","last_fullsized":"2024-03-11T20:53:08.091261+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:11.462480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:52:11.462480+0000","last_clean_scrub_stamp":"2024-03-11T20:52:11.462480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:22.958223+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":88,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.087509+0000","last_change":"2024-03-11T20:53:02.400889+0000","last_active":"2024-03-11T20:53:08.087509+0000","last_peered":"2024-03-11T20:53:08.087509+0000","last_clean":"2024-03-11T20:53:08.087509+0000","last_became_active":"2024-03-11T20:50:32.646323+0000","last_became_peered":"2024-03-11T20:50:32.646323+0000","last_unstale":"2024-03-11T20:53:08.087509+0000","last_undegraded":"2024-03-11T20:53:08.087509+0000","last_fullsized":"2024-03-11T20:53:08.087509+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:02.400685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:53:02.400685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:04.399694+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":88,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.087096+0000","last_change":"2024-03-11T20:51:42.709004+0000","last_active":"2024-03-11T20:53:08.087096+0000","last_peered":"2024-03-11T20:53:08.087096+0000","last_clean":"2024-03-11T20:53:08.087096+0000","last_became_active":"2024-03-11T20:50:28.086513+0000","last_became_peered":"2024-03-11T20:50:28.086513+0000","last_unstale":"2024-03-11T20:53:08.087096+0000","last_undegraded":"2024-03-11T20:53:08.087096+0000","last_fullsized":"2024-03-11T20:53:08.087096+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:42.708769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:51:42.708769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:09.223144+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"84'9","reported_seq":100,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.120972+0000","last_change":"2024-03-11T20:51:45.881582+0000","last_active":"2024-03-11T20:53:08.120972+0000","last_peered":"2024-03-11T20:53:08.120972+0000","last_clean":"2024-03-11T20:53:08.120972+0000","last_became_active":"2024-03-11T20:50:30.109869+0000","last_became_peered":"2024-03-11T20:50:30.109869+0000","last_unstale":"2024-03-11T20:53:08.120972+0000","last_undegraded":"2024-03-11T20:53:08.120972+0000","last_fullsized":"2024-03-11T20:53:08.120972+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'9","last_scrub_stamp":"2024-03-11T20:51:45.881270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:51:45.881270+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:13.772431+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":84,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.121233+0000","last_change":"2024-03-11T20:53:04.321530+0000","last_active":"2024-03-11T20:53:08.121233+0000","last_peered":"2024-03-11T20:53:08.121233+0000","last_clean":"2024-03-11T20:53:08.121233+0000","last_became_active":"2024-03-11T20:50:34.321443+0000","last_became_peered":"2024-03-11T20:50:34.321443+0000","last_unstale":"2024-03-11T20:53:08.121233+0000","last_undegraded":"2024-03-11T20:53:08.121233+0000","last_fullsized":"2024-03-11T20:53:08.121233+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:04.321235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:53:04.321235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:23.221472+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":80,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.085132+0000","last_change":"2024-03-11T20:52:29.373009+0000","last_active":"2024-03-11T20:53:08.085132+0000","last_peered":"2024-03-11T20:53:08.085132+0000","last_clean":"2024-03-11T20:53:08.085132+0000","last_became_active":"2024-03-11T20:50:32.211995+0000","last_became_peered":"2024-03-11T20:50:32.211995+0000","last_unstale":"2024-03-11T20:53:08.085132+0000","last_undegraded":"2024-03-11T20:53:08.085132+0000","last_fullsized":"2024-03-11T20:53:08.085132+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:29.372897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:52:29.372897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:32.682617+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":96,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.120687+0000","last_change":"2024-03-11T20:52:53.806361+0000","last_active":"2024-03-11T20:53:08.120687+0000","last_peered":"2024-03-11T20:53:08.120687+0000","last_clean":"2024-03-11T20:53:08.120687+0000","last_became_active":"2024-03-11T20:50:28.075760+0000","last_became_peered":"2024-03-11T20:50:28.075760+0000","last_unstale":"2024-03-11T20:53:08.120687+0000","last_undegraded":"2024-03-11T20:53:08.120687+0000","last_fullsized":"2024-03-11T20:53:08.120687+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:53.806149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:52:53.806149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:15.472221+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"84'6","reported_seq":92,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.117865+0000","last_change":"2024-03-11T20:51:59.774999+0000","last_active":"2024-03-11T20:53:08.117865+0000","last_peered":"2024-03-11T20:53:08.117865+0000","last_clean":"2024-03-11T20:53:08.117865+0000","last_became_active":"2024-03-11T20:50:30.092991+0000","last_became_peered":"2024-03-11T20:50:30.092991+0000","last_unstale":"2024-03-11T20:53:08.117865+0000","last_undegraded":"2024-03-11T20:53:08.117865+0000","last_fullsized":"2024-03-11T20:53:08.117865+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6","last_scrub_stamp":"2024-03-11T20:51:59.774854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:51:59.774854+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:16.885326+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.11","version":"84'1","reported_seq":77,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.085631+0000","last_change":"2024-03-11T20:52:42.470691+0000","last_active":"2024-03-11T20:53:08.085631+0000","last_peered":"2024-03-11T20:53:08.085631+0000","last_clean":"2024-03-11T20:53:08.085631+0000","last_became_active":"2024-03-11T20:50:34.339326+0000","last_became_peered":"2024-03-11T20:50:34.339326+0000","last_unstale":"2024-03-11T20:53:08.085631+0000","last_undegraded":"2024-03-11T20:53:08.085631+0000","last_fullsized":"2024-03-11T20:53:08.085631+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'1","last_scrub_stamp":"2024-03-11T20:52:42.470380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:52:42.470380+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:08.720300+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":80,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.120298+0000","last_change":"2024-03-11T20:51:46.599809+0000","last_active":"2024-03-11T20:53:08.120298+0000","last_peered":"2024-03-11T20:53:08.120298+0000","last_clean":"2024-03-11T20:53:08.120298+0000","last_became_active":"2024-03-11T20:50:32.211379+0000","last_became_peered":"2024-03-11T20:50:32.211379+0000","last_unstale":"2024-03-11T20:53:08.120298+0000","last_undegraded":"2024-03-11T20:53:08.120298+0000","last_fullsized":"2024-03-11T20:53:08.120298+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:46.599566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:51:46.599566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":88,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.089669+0000","last_change":"2024-03-11T20:52:04.155479+0000","last_active":"2024-03-11T20:53:08.089669+0000","last_peered":"2024-03-11T20:53:08.089669+0000","last_clean":"2024-03-11T20:53:08.089669+0000","last_became_active":"2024-03-11T20:50:28.049937+0000","last_became_peered":"2024-03-11T20:50:28.049937+0000","last_unstale":"2024-03-11T20:53:08.089669+0000","last_undegraded":"2024-03-11T20:53:08.089669+0000","last_fullsized":"2024-03-11T20:53:08.089669+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:04.155223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:52:04.155223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"84'10","reported_seq":107,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.119989+0000","last_change":"2024-03-11T20:52:51.437032+0000","last_active":"2024-03-11T20:53:08.119989+0000","last_peered":"2024-03-11T20:53:08.119989+0000","last_clean":"2024-03-11T20:53:08.119989+0000","last_became_active":"2024-03-11T20:50:30.105752+0000","last_became_peered":"2024-03-11T20:50:30.105752+0000","last_unstale":"2024-03-11T20:53:08.119989+0000","last_undegraded":"2024-03-11T20:53:08.119989+0000","last_fullsized":"2024-03-11T20:53:08.119989+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'10","last_scrub_stamp":"2024-03-11T20:52:51.436752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:52:51.436752+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:15.812213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":75,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.113945+0000","last_change":"2024-03-11T20:52:03.919036+0000","last_active":"2024-03-11T20:53:08.113945+0000","last_peered":"2024-03-11T20:53:08.113945+0000","last_clean":"2024-03-11T20:53:08.113945+0000","last_became_active":"2024-03-11T20:50:34.332465+0000","last_became_peered":"2024-03-11T20:50:34.332465+0000","last_unstale":"2024-03-11T20:53:08.113945+0000","last_undegraded":"2024-03-11T20:53:08.113945+0000","last_fullsized":"2024-03-11T20:53:08.113945+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:03.918724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:52:03.918724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:19.041856+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":80,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.086258+0000","last_change":"2024-03-11T20:51:59.129831+0000","last_active":"2024-03-11T20:53:08.086258+0000","last_peered":"2024-03-11T20:53:08.086258+0000","last_clean":"2024-03-11T20:53:08.086258+0000","last_became_active":"2024-03-11T20:50:32.194792+0000","last_became_peered":"2024-03-11T20:50:32.194792+0000","last_unstale":"2024-03-11T20:53:08.086258+0000","last_undegraded":"2024-03-11T20:53:08.086258+0000","last_fullsized":"2024-03-11T20:53:08.086258+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:59.129712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:51:59.129712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:15.832569+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":96,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.086336+0000","last_change":"2024-03-11T20:52:54.775523+0000","last_active":"2024-03-11T20:53:08.086336+0000","last_peered":"2024-03-11T20:53:08.086336+0000","last_clean":"2024-03-11T20:53:08.086336+0000","last_became_active":"2024-03-11T20:50:28.082737+0000","last_became_peered":"2024-03-11T20:50:28.082737+0000","last_unstale":"2024-03-11T20:53:08.086336+0000","last_undegraded":"2024-03-11T20:53:08.086336+0000","last_fullsized":"2024-03-11T20:53:08.086336+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:54.775303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:52:54.775303+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:11.643155+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"84'9","reported_seq":108,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.122979+0000","last_change":"2024-03-11T20:52:57.247973+0000","last_active":"2024-03-11T20:53:08.122979+0000","last_peered":"2024-03-11T20:53:08.122979+0000","last_clean":"2024-03-11T20:53:08.122979+0000","last_became_active":"2024-03-11T20:50:30.085418+0000","last_became_peered":"2024-03-11T20:50:30.085418+0000","last_unstale":"2024-03-11T20:53:08.122979+0000","last_undegraded":"2024-03-11T20:53:08.122979+0000","last_fullsized":"2024-03-11T20:53:08.122979+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'9","last_scrub_stamp":"2024-03-11T20:52:57.247711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:52:57.247711+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:09.030171+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":76,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.124072+0000","last_change":"2024-03-11T20:52:02.551988+0000","last_active":"2024-03-11T20:53:08.124072+0000","last_peered":"2024-03-11T20:53:08.124072+0000","last_clean":"2024-03-11T20:53:08.124072+0000","last_became_active":"2024-03-11T20:50:34.328449+0000","last_became_peered":"2024-03-11T20:50:34.328449+0000","last_unstale":"2024-03-11T20:53:08.124072+0000","last_undegraded":"2024-03-11T20:53:08.124072+0000","last_fullsized":"2024-03-11T20:53:08.124072+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:02.551652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:52:02.551652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.15","version":"84'11","reported_seq":123,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.120470+0000","last_change":"2024-03-11T20:52:09.843826+0000","last_active":"2024-03-11T20:53:08.120470+0000","last_peered":"2024-03-11T20:53:08.120470+0000","last_clean":"2024-03-11T20:53:08.120470+0000","last_became_active":"2024-03-11T20:50:32.650064+0000","last_became_peered":"2024-03-11T20:50:32.650064+0000","last_unstale":"2024-03-11T20:53:08.120470+0000","last_undegraded":"2024-03-11T20:53:08.120470+0000","last_fullsized":"2024-03-11T20:53:08.120470+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T20:52:09.843595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:52:09.843595+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:10.816759+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":95,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.116183+0000","last_change":"2024-03-11T20:52:47.933545+0000","last_active":"2024-03-11T20:53:08.116183+0000","last_peered":"2024-03-11T20:53:08.116183+0000","last_clean":"2024-03-11T20:53:08.116183+0000","last_became_active":"2024-03-11T20:50:28.044321+0000","last_became_peered":"2024-03-11T20:50:28.044321+0000","last_unstale":"2024-03-11T20:53:08.116183+0000","last_undegraded":"2024-03-11T20:53:08.116183+0000","last_fullsized":"2024-03-11T20:53:08.116183+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:47.933320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:52:47.933320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:55.580970+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"84'9","reported_seq":107,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.116643+0000","last_change":"2024-03-11T20:52:55.993159+0000","last_active":"2024-03-11T20:53:08.116643+0000","last_peered":"2024-03-11T20:53:08.116643+0000","last_clean":"2024-03-11T20:53:08.116643+0000","last_became_active":"2024-03-11T20:50:30.099856+0000","last_became_peered":"2024-03-11T20:50:30.099856+0000","last_unstale":"2024-03-11T20:53:08.116643+0000","last_undegraded":"2024-03-11T20:53:08.116643+0000","last_fullsized":"2024-03-11T20:53:08.116643+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'9","last_scrub_stamp":"2024-03-11T20:52:55.992803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:52:55.992803+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:14.843849+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":75,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.117372+0000","last_change":"2024-03-11T20:52:16.846357+0000","last_active":"2024-03-11T20:53:08.117372+0000","last_peered":"2024-03-11T20:53:08.117372+0000","last_clean":"2024-03-11T20:53:08.117372+0000","last_became_active":"2024-03-11T20:50:34.330351+0000","last_became_peered":"2024-03-11T20:50:34.330351+0000","last_unstale":"2024-03-11T20:53:08.117372+0000","last_undegraded":"2024-03-11T20:53:08.117372+0000","last_fullsized":"2024-03-11T20:53:08.117372+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:16.846238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:52:16.846238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:38.667608+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"110'13","reported_seq":142,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.086956+0000","last_change":"2024-03-11T20:53:00.434710+0000","last_active":"2024-03-11T20:53:08.086956+0000","last_peered":"2024-03-11T20:53:08.086956+0000","last_clean":"2024-03-11T20:53:08.086956+0000","last_became_active":"2024-03-11T20:51:22.669419+0000","last_became_peered":"2024-03-11T20:51:22.669419+0000","last_unstale":"2024-03-11T20:53:08.086956+0000","last_undegraded":"2024-03-11T20:53:08.086956+0000","last_fullsized":"2024-03-11T20:53:08.086956+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'13","last_scrub_stamp":"2024-03-11T20:53:00.434532+0000","last_deep_scrub":"110'13","last_deep_scrub_stamp":"2024-03-11T20:53:00.434532+0000","last_clean_scrub_stamp":"2024-03-11T20:53:00.434532+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:25.979437+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":96,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.121578+0000","last_change":"2024-03-11T20:52:59.801478+0000","last_active":"2024-03-11T20:53:08.121578+0000","last_peered":"2024-03-11T20:53:08.121578+0000","last_clean":"2024-03-11T20:53:08.121578+0000","last_became_active":"2024-03-11T20:50:28.070325+0000","last_became_peered":"2024-03-11T20:50:28.070325+0000","last_unstale":"2024-03-11T20:53:08.121578+0000","last_undegraded":"2024-03-11T20:53:08.121578+0000","last_fullsized":"2024-03-11T20:53:08.121578+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:59.801249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:52:59.801249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:29.742165+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"84'11","reported_seq":103,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.122603+0000","last_change":"2024-03-11T20:52:12.020706+0000","last_active":"2024-03-11T20:53:08.122603+0000","last_peered":"2024-03-11T20:53:08.122603+0000","last_clean":"2024-03-11T20:53:08.122603+0000","last_became_active":"2024-03-11T20:50:30.087154+0000","last_became_peered":"2024-03-11T20:50:30.087154+0000","last_unstale":"2024-03-11T20:53:08.122603+0000","last_undegraded":"2024-03-11T20:53:08.122603+0000","last_fullsized":"2024-03-11T20:53:08.122603+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T20:52:12.020466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:52:12.020466+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:29.103467+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":76,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.122306+0000","last_change":"2024-03-11T20:52:22.074863+0000","last_active":"2024-03-11T20:53:08.122306+0000","last_peered":"2024-03-11T20:53:08.122306+0000","last_clean":"2024-03-11T20:53:08.122306+0000","last_became_active":"2024-03-11T20:50:34.318231+0000","last_became_peered":"2024-03-11T20:50:34.318231+0000","last_unstale":"2024-03-11T20:53:08.122306+0000","last_undegraded":"2024-03-11T20:53:08.122306+0000","last_fullsized":"2024-03-11T20:53:08.122306+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:22.074605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:52:22.074605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:39.515449+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":88,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.089629+0000","last_change":"2024-03-11T20:53:06.534811+0000","last_active":"2024-03-11T20:53:08.089629+0000","last_peered":"2024-03-11T20:53:08.089629+0000","last_clean":"2024-03-11T20:53:08.089629+0000","last_became_active":"2024-03-11T20:50:32.651361+0000","last_became_peered":"2024-03-11T20:50:32.651361+0000","last_unstale":"2024-03-11T20:53:08.089629+0000","last_undegraded":"2024-03-11T20:53:08.089629+0000","last_fullsized":"2024-03-11T20:53:08.089629+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:06.534615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:53:06.534615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:15.085973+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"77'1","reported_seq":96,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.085830+0000","last_change":"2024-03-11T20:52:02.654230+0000","last_active":"2024-03-11T20:53:08.085830+0000","last_peered":"2024-03-11T20:53:08.085830+0000","last_clean":"2024-03-11T20:53:08.085830+0000","last_became_active":"2024-03-11T20:50:28.083256+0000","last_became_peered":"2024-03-11T20:50:28.083256+0000","last_unstale":"2024-03-11T20:53:08.085830+0000","last_undegraded":"2024-03-11T20:53:08.085830+0000","last_fullsized":"2024-03-11T20:53:08.085830+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-03-11T20:52:02.653980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:52:02.653980+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:11.645595+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"84'4","reported_seq":90,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.085374+0000","last_change":"2024-03-11T20:52:11.668147+0000","last_active":"2024-03-11T20:53:08.085374+0000","last_peered":"2024-03-11T20:53:08.085374+0000","last_clean":"2024-03-11T20:53:08.085374+0000","last_became_active":"2024-03-11T20:50:30.101572+0000","last_became_peered":"2024-03-11T20:50:30.101572+0000","last_unstale":"2024-03-11T20:53:08.085374+0000","last_undegraded":"2024-03-11T20:53:08.085374+0000","last_fullsized":"2024-03-11T20:53:08.085374+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'4","last_scrub_stamp":"2024-03-11T20:52:11.667810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:52:11.667810+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:40.354519+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":84,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.116453+0000","last_change":"2024-03-11T20:52:57.942490+0000","last_active":"2024-03-11T20:53:08.116453+0000","last_peered":"2024-03-11T20:53:08.116453+0000","last_clean":"2024-03-11T20:53:08.116453+0000","last_became_active":"2024-03-11T20:50:34.320001+0000","last_became_peered":"2024-03-11T20:50:34.320001+0000","last_unstale":"2024-03-11T20:53:08.116453+0000","last_undegraded":"2024-03-11T20:53:08.116453+0000","last_fullsized":"2024-03-11T20:53:08.116453+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:57.942382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:52:57.942382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:03.245684+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":87,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.116351+0000","last_change":"2024-03-11T20:53:05.100946+0000","last_active":"2024-03-11T20:53:08.116351+0000","last_peered":"2024-03-11T20:53:08.116351+0000","last_clean":"2024-03-11T20:53:08.116351+0000","last_became_active":"2024-03-11T20:50:32.646569+0000","last_became_peered":"2024-03-11T20:50:32.646569+0000","last_unstale":"2024-03-11T20:53:08.116351+0000","last_undegraded":"2024-03-11T20:53:08.116351+0000","last_fullsized":"2024-03-11T20:53:08.116351+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:05.100847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:53:05.100847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:07.646846+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":96,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.113904+0000","last_change":"2024-03-11T20:53:02.899718+0000","last_active":"2024-03-11T20:53:08.113904+0000","last_peered":"2024-03-11T20:53:08.113904+0000","last_clean":"2024-03-11T20:53:08.113904+0000","last_became_active":"2024-03-11T20:50:28.044592+0000","last_became_peered":"2024-03-11T20:50:28.044592+0000","last_unstale":"2024-03-11T20:53:08.113904+0000","last_undegraded":"2024-03-11T20:53:08.113904+0000","last_fullsized":"2024-03-11T20:53:08.113904+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:02.899469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:53:02.899469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:21.925538+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"84'11","reported_seq":103,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.121193+0000","last_change":"2024-03-11T20:52:21.089002+0000","last_active":"2024-03-11T20:53:08.121193+0000","last_peered":"2024-03-11T20:53:08.121193+0000","last_clean":"2024-03-11T20:53:08.121193+0000","last_became_active":"2024-03-11T20:50:30.103212+0000","last_became_peered":"2024-03-11T20:50:30.103212+0000","last_unstale":"2024-03-11T20:53:08.121193+0000","last_undegraded":"2024-03-11T20:53:08.121193+0000","last_fullsized":"2024-03-11T20:53:08.121193+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T20:52:21.088529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:52:21.088529+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:32.580581+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":84,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.091213+0000","last_change":"2024-03-11T20:52:53.451122+0000","last_active":"2024-03-11T20:53:08.091213+0000","last_peered":"2024-03-11T20:53:08.091213+0000","last_clean":"2024-03-11T20:53:08.091213+0000","last_became_active":"2024-03-11T20:50:34.336077+0000","last_became_peered":"2024-03-11T20:50:34.336077+0000","last_unstale":"2024-03-11T20:53:08.091213+0000","last_undegraded":"2024-03-11T20:53:08.091213+0000","last_fullsized":"2024-03-11T20:53:08.091213+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:53.450885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:52:53.450885+0000","last_clean_scrub_stamp":"2024-03-11T20:52:53.450885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:10.721360+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"77'2","reported_seq":104,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.123655+0000","last_change":"2024-03-11T20:52:05.512151+0000","last_active":"2024-03-11T20:53:08.123655+0000","last_peered":"2024-03-11T20:53:08.123655+0000","last_clean":"2024-03-11T20:53:08.123655+0000","last_became_active":"2024-03-11T20:50:28.059564+0000","last_became_peered":"2024-03-11T20:50:28.059564+0000","last_unstale":"2024-03-11T20:53:08.123655+0000","last_undegraded":"2024-03-11T20:53:08.123655+0000","last_fullsized":"2024-03-11T20:53:08.123655+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'2","last_scrub_stamp":"2024-03-11T20:52:05.511938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:52:05.511938+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:11.675698+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"100'16","reported_seq":116,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.122928+0000","last_change":"2024-03-11T20:52:37.207901+0000","last_active":"2024-03-11T20:53:08.122928+0000","last_peered":"2024-03-11T20:53:08.122928+0000","last_clean":"2024-03-11T20:53:08.122928+0000","last_became_active":"2024-03-11T20:50:30.083708+0000","last_became_peered":"2024-03-11T20:50:30.083708+0000","last_unstale":"2024-03-11T20:53:08.122928+0000","last_undegraded":"2024-03-11T20:53:08.122928+0000","last_fullsized":"2024-03-11T20:53:08.122928+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'16","last_scrub_stamp":"2024-03-11T20:52:37.207662+0000","last_deep_scrub":"100'16","last_deep_scrub_stamp":"2024-03-11T20:51:31.194005+0000","last_clean_scrub_stamp":"2024-03-11T20:52:37.207662+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:56.687016+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"106'12","reported_seq":124,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.118001+0000","last_change":"2024-03-11T20:52:21.539124+0000","last_active":"2024-03-11T20:53:08.118001+0000","last_peered":"2024-03-11T20:53:08.118001+0000","last_clean":"2024-03-11T20:53:08.118001+0000","last_became_active":"2024-03-11T20:50:32.210620+0000","last_became_peered":"2024-03-11T20:50:32.210620+0000","last_unstale":"2024-03-11T20:53:08.118001+0000","last_undegraded":"2024-03-11T20:53:08.118001+0000","last_fullsized":"2024-03-11T20:53:08.118001+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'12","last_scrub_stamp":"2024-03-11T20:52:21.538885+0000","last_deep_scrub":"106'12","last_deep_scrub_stamp":"2024-03-11T20:52:21.538885+0000","last_clean_scrub_stamp":"2024-03-11T20:52:21.538885+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:37.059443+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":75,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.114575+0000","last_change":"2024-03-11T20:52:07.836375+0000","last_active":"2024-03-11T20:53:08.114575+0000","last_peered":"2024-03-11T20:53:08.114575+0000","last_clean":"2024-03-11T20:53:08.114575+0000","last_became_active":"2024-03-11T20:50:34.328932+0000","last_became_peered":"2024-03-11T20:50:34.328932+0000","last_unstale":"2024-03-11T20:53:08.114575+0000","last_undegraded":"2024-03-11T20:53:08.114575+0000","last_fullsized":"2024-03-11T20:53:08.114575+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:07.836159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:52:07.836159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:23.376900+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":88,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.088495+0000","last_change":"2024-03-11T20:52:14.175560+0000","last_active":"2024-03-11T20:53:08.088495+0000","last_peered":"2024-03-11T20:53:08.088495+0000","last_clean":"2024-03-11T20:53:08.088495+0000","last_became_active":"2024-03-11T20:50:28.057091+0000","last_became_peered":"2024-03-11T20:50:28.057091+0000","last_unstale":"2024-03-11T20:53:08.088495+0000","last_undegraded":"2024-03-11T20:53:08.088495+0000","last_fullsized":"2024-03-11T20:53:08.088495+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:14.175327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:52:14.175327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:37.308976+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"84'16","reported_seq":139,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.119980+0000","last_change":"2024-03-11T20:52:03.937604+0000","last_active":"2024-03-11T20:53:08.119980+0000","last_peered":"2024-03-11T20:53:08.119980+0000","last_clean":"2024-03-11T20:53:08.119980+0000","last_became_active":"2024-03-11T20:50:30.094977+0000","last_became_peered":"2024-03-11T20:50:30.094977+0000","last_unstale":"2024-03-11T20:53:08.119980+0000","last_undegraded":"2024-03-11T20:53:08.119980+0000","last_fullsized":"2024-03-11T20:53:08.119980+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'16","last_scrub_stamp":"2024-03-11T20:52:03.937351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:52:03.937351+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:32.484876+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":80,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.087418+0000","last_change":"2024-03-11T20:52:10.160318+0000","last_active":"2024-03-11T20:53:08.087418+0000","last_peered":"2024-03-11T20:53:08.087418+0000","last_clean":"2024-03-11T20:53:08.087418+0000","last_became_active":"2024-03-11T20:50:32.206394+0000","last_became_peered":"2024-03-11T20:50:32.206394+0000","last_unstale":"2024-03-11T20:53:08.087418+0000","last_undegraded":"2024-03-11T20:53:08.087418+0000","last_fullsized":"2024-03-11T20:53:08.087418+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:10.160202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:52:10.160202+0000","last_clean_scrub_stamp":"2024-03-11T20:52:10.160202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:29.912826+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":76,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.088155+0000","last_change":"2024-03-11T20:52:15.511775+0000","last_active":"2024-03-11T20:53:08.088155+0000","last_peered":"2024-03-11T20:53:08.088155+0000","last_clean":"2024-03-11T20:53:08.088155+0000","last_became_active":"2024-03-11T20:50:34.329253+0000","last_became_peered":"2024-03-11T20:50:34.329253+0000","last_unstale":"2024-03-11T20:53:08.088155+0000","last_undegraded":"2024-03-11T20:53:08.088155+0000","last_fullsized":"2024-03-11T20:53:08.088155+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:15.511437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:52:15.511437+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:27.180999+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":96,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.121819+0000","last_change":"2024-03-11T20:52:58.782154+0000","last_active":"2024-03-11T20:53:08.121819+0000","last_peered":"2024-03-11T20:53:08.121819+0000","last_clean":"2024-03-11T20:53:08.121819+0000","last_became_active":"2024-03-11T20:50:28.075678+0000","last_became_peered":"2024-03-11T20:50:28.075678+0000","last_unstale":"2024-03-11T20:53:08.121819+0000","last_undegraded":"2024-03-11T20:53:08.121819+0000","last_fullsized":"2024-03-11T20:53:08.121819+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:58.781938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:33.823704+0000","last_clean_scrub_stamp":"2024-03-11T20:52:58.781938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:19.252494+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"84'19","reported_seq":120,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.121214+0000","last_change":"2024-03-11T20:51:42.596727+0000","last_active":"2024-03-11T20:53:08.121214+0000","last_peered":"2024-03-11T20:53:08.121214+0000","last_clean":"2024-03-11T20:53:08.121214+0000","last_became_active":"2024-03-11T20:50:30.097543+0000","last_became_peered":"2024-03-11T20:50:30.097543+0000","last_unstale":"2024-03-11T20:53:08.121214+0000","last_undegraded":"2024-03-11T20:53:08.121214+0000","last_fullsized":"2024-03-11T20:53:08.121214+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'19","last_scrub_stamp":"2024-03-11T20:51:42.596491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:51:42.596491+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":80,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.088208+0000","last_change":"2024-03-11T20:52:16.096498+0000","last_active":"2024-03-11T20:53:08.088208+0000","last_peered":"2024-03-11T20:53:08.088208+0000","last_clean":"2024-03-11T20:53:08.088208+0000","last_became_active":"2024-03-11T20:50:32.211494+0000","last_became_peered":"2024-03-11T20:50:32.211494+0000","last_unstale":"2024-03-11T20:53:08.088208+0000","last_undegraded":"2024-03-11T20:53:08.088208+0000","last_fullsized":"2024-03-11T20:53:08.088208+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:16.096259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:52:16.096259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:16.697891+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":76,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.122362+0000","last_change":"2024-03-11T20:52:25.115485+0000","last_active":"2024-03-11T20:53:08.122362+0000","last_peered":"2024-03-11T20:53:08.122362+0000","last_clean":"2024-03-11T20:53:08.122362+0000","last_became_active":"2024-03-11T20:50:34.321038+0000","last_became_peered":"2024-03-11T20:50:34.321038+0000","last_unstale":"2024-03-11T20:53:08.122362+0000","last_undegraded":"2024-03-11T20:53:08.122362+0000","last_fullsized":"2024-03-11T20:53:08.122362+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:25.115238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:52:25.115238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:34.308256+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"109'26","reported_seq":132,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.083946+0000","last_change":"2024-03-11T20:52:49.216560+0000","last_active":"2024-03-11T20:53:08.083946+0000","last_peered":"2024-03-11T20:53:08.083946+0000","last_clean":"2024-03-11T20:53:08.083946+0000","last_became_active":"2024-03-11T20:50:30.098371+0000","last_became_peered":"2024-03-11T20:50:30.098371+0000","last_unstale":"2024-03-11T20:53:08.083946+0000","last_undegraded":"2024-03-11T20:53:08.083946+0000","last_fullsized":"2024-03-11T20:53:08.083946+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'26","last_scrub_stamp":"2024-03-11T20:52:49.216378+0000","last_deep_scrub":"109'26","last_deep_scrub_stamp":"2024-03-11T20:52:49.216378+0000","last_clean_scrub_stamp":"2024-03-11T20:52:49.216378+0000","objects_scrubbed":8,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:49.844091+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":96,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.114120+0000","last_change":"2024-03-11T20:53:06.797551+0000","last_active":"2024-03-11T20:53:08.114120+0000","last_peered":"2024-03-11T20:53:08.114120+0000","last_clean":"2024-03-11T20:53:08.114120+0000","last_became_active":"2024-03-11T20:50:28.045625+0000","last_became_peered":"2024-03-11T20:50:28.045625+0000","last_unstale":"2024-03-11T20:53:08.114120+0000","last_undegraded":"2024-03-11T20:53:08.114120+0000","last_fullsized":"2024-03-11T20:53:08.114120+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:06.797336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:53:06.797336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:24.649727+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":80,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.091708+0000","last_change":"2024-03-11T20:52:25.125883+0000","last_active":"2024-03-11T20:53:08.091708+0000","last_peered":"2024-03-11T20:53:08.091708+0000","last_clean":"2024-03-11T20:53:08.091708+0000","last_became_active":"2024-03-11T20:50:32.205996+0000","last_became_peered":"2024-03-11T20:50:32.205996+0000","last_unstale":"2024-03-11T20:53:08.091708+0000","last_undegraded":"2024-03-11T20:53:08.091708+0000","last_fullsized":"2024-03-11T20:53:08.091708+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:25.125696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:52:25.125696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:52.303574+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":76,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.125337+0000","last_change":"2024-03-11T20:52:11.898901+0000","last_active":"2024-03-11T20:53:08.125337+0000","last_peered":"2024-03-11T20:53:08.125337+0000","last_clean":"2024-03-11T20:53:08.125337+0000","last_became_active":"2024-03-11T20:50:34.336066+0000","last_became_peered":"2024-03-11T20:50:34.336066+0000","last_unstale":"2024-03-11T20:53:08.125337+0000","last_undegraded":"2024-03-11T20:53:08.125337+0000","last_fullsized":"2024-03-11T20:53:08.125337+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:11.898570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:52:11.898570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:29.169074+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"84'10","reported_seq":99,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.084436+0000","last_change":"2024-03-11T20:52:35.399348+0000","last_active":"2024-03-11T20:53:08.084436+0000","last_peered":"2024-03-11T20:53:08.084436+0000","last_clean":"2024-03-11T20:53:08.084436+0000","last_became_active":"2024-03-11T20:50:30.101747+0000","last_became_peered":"2024-03-11T20:50:30.101747+0000","last_unstale":"2024-03-11T20:53:08.084436+0000","last_undegraded":"2024-03-11T20:53:08.084436+0000","last_fullsized":"2024-03-11T20:53:08.084436+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'10","last_scrub_stamp":"2024-03-11T20:52:35.399027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:52:35.399027+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:57.620344+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":96,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.119284+0000","last_change":"2024-03-11T20:52:44.919146+0000","last_active":"2024-03-11T20:53:08.119284+0000","last_peered":"2024-03-11T20:53:08.119284+0000","last_clean":"2024-03-11T20:53:08.119284+0000","last_became_active":"2024-03-11T20:50:28.085503+0000","last_became_peered":"2024-03-11T20:50:28.085503+0000","last_unstale":"2024-03-11T20:53:08.119284+0000","last_undegraded":"2024-03-11T20:53:08.119284+0000","last_fullsized":"2024-03-11T20:53:08.119284+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:44.918937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:52:44.918937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:47.097462+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"102'128","reported_seq":353,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.118763+0000","last_change":"2024-03-11T20:52:28.047817+0000","last_active":"2024-03-11T20:53:08.118763+0000","last_peered":"2024-03-11T20:53:08.118763+0000","last_clean":"2024-03-11T20:53:08.118763+0000","last_became_active":"2024-03-11T20:49:52.584543+0000","last_became_peered":"2024-03-11T20:49:52.584543+0000","last_unstale":"2024-03-11T20:53:08.118763+0000","last_undegraded":"2024-03-11T20:53:08.118763+0000","last_fullsized":"2024-03-11T20:53:08.118763+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"102'128","last_scrub_stamp":"2024-03-11T20:52:28.047573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:45:32.381751+0000","last_clean_scrub_stamp":"2024-03-11T20:52:28.047573+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:54.167386+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":88,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.087326+0000","last_change":"2024-03-11T20:53:07.859791+0000","last_active":"2024-03-11T20:53:08.087326+0000","last_peered":"2024-03-11T20:53:08.087326+0000","last_clean":"2024-03-11T20:53:08.087326+0000","last_became_active":"2024-03-11T20:50:32.652128+0000","last_became_peered":"2024-03-11T20:50:32.652128+0000","last_unstale":"2024-03-11T20:53:08.087326+0000","last_undegraded":"2024-03-11T20:53:08.087326+0000","last_fullsized":"2024-03-11T20:53:08.087326+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:07.859545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:53:07.859545+0000","last_clean_scrub_stamp":"2024-03-11T20:53:07.859545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:20.657302+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":76,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.087928+0000","last_change":"2024-03-11T20:52:09.169969+0000","last_active":"2024-03-11T20:53:08.087928+0000","last_peered":"2024-03-11T20:53:08.087928+0000","last_clean":"2024-03-11T20:53:08.087928+0000","last_became_active":"2024-03-11T20:50:34.326321+0000","last_became_peered":"2024-03-11T20:50:34.326321+0000","last_unstale":"2024-03-11T20:53:08.087928+0000","last_undegraded":"2024-03-11T20:53:08.087928+0000","last_fullsized":"2024-03-11T20:53:08.087928+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:09.169726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:52:09.169726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:11.048514+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"100'21","reported_seq":119,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.115387+0000","last_change":"2024-03-11T20:53:03.110687+0000","last_active":"2024-03-11T20:53:08.115387+0000","last_peered":"2024-03-11T20:53:08.115387+0000","last_clean":"2024-03-11T20:53:08.115387+0000","last_became_active":"2024-03-11T20:50:30.106110+0000","last_became_peered":"2024-03-11T20:50:30.106110+0000","last_unstale":"2024-03-11T20:53:08.115387+0000","last_undegraded":"2024-03-11T20:53:08.115387+0000","last_fullsized":"2024-03-11T20:53:08.115387+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'21","last_scrub_stamp":"2024-03-11T20:53:03.110450+0000","last_deep_scrub":"100'21","last_deep_scrub_stamp":"2024-03-11T20:51:31.835302+0000","last_clean_scrub_stamp":"2024-03-11T20:53:03.110450+0000","objects_scrubbed":7,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:23.836851+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":53,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.113575+0000","last_change":"2024-03-11T20:52:56.891528+0000","last_active":"2024-03-11T20:53:08.113575+0000","last_peered":"2024-03-11T20:53:08.113575+0000","last_clean":"2024-03-11T20:53:08.113575+0000","last_became_active":"2024-03-11T20:51:22.668492+0000","last_became_peered":"2024-03-11T20:51:22.668492+0000","last_unstale":"2024-03-11T20:53:08.113575+0000","last_undegraded":"2024-03-11T20:53:08.113575+0000","last_fullsized":"2024-03-11T20:53:08.113575+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:56.891410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:52:56.891410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:57.500706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":87,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.117478+0000","last_change":"2024-03-11T20:53:04.140960+0000","last_active":"2024-03-11T20:53:08.117478+0000","last_peered":"2024-03-11T20:53:08.117478+0000","last_clean":"2024-03-11T20:53:08.117478+0000","last_became_active":"2024-03-11T20:50:32.195566+0000","last_became_peered":"2024-03-11T20:50:32.195566+0000","last_unstale":"2024-03-11T20:53:08.117478+0000","last_undegraded":"2024-03-11T20:53:08.117478+0000","last_fullsized":"2024-03-11T20:53:08.117478+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:04.140875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:53:04.140875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:29.280686+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":84,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.170188+0000","last_change":"2024-03-11T20:53:08.170244+0000","last_active":"2024-03-11T20:53:08.170188+0000","last_peered":"2024-03-11T20:53:08.170188+0000","last_clean":"2024-03-11T20:53:08.170188+0000","last_became_active":"2024-03-11T20:50:34.306750+0000","last_became_peered":"2024-03-11T20:50:34.306750+0000","last_unstale":"2024-03-11T20:53:08.170188+0000","last_undegraded":"2024-03-11T20:53:08.170188+0000","last_fullsized":"2024-03-11T20:53:08.170188+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:08.170036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:53:08.170036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:35.208803+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"87'2","reported_seq":89,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.086239+0000","last_change":"2024-03-11T20:51:52.747686+0000","last_active":"2024-03-11T20:53:08.086239+0000","last_peered":"2024-03-11T20:53:08.086239+0000","last_clean":"2024-03-11T20:53:08.086239+0000","last_became_active":"2024-03-11T20:50:32.195452+0000","last_became_peered":"2024-03-11T20:50:32.195452+0000","last_unstale":"2024-03-11T20:53:08.086239+0000","last_undegraded":"2024-03-11T20:53:08.086239+0000","last_fullsized":"2024-03-11T20:53:08.086239+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'2","last_scrub_stamp":"2024-03-11T20:51:52.747446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:51:52.747446+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:13.971732+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0024049620000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":96,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.122723+0000","last_change":"2024-03-11T20:53:05.364652+0000","last_active":"2024-03-11T20:53:08.122723+0000","last_peered":"2024-03-11T20:53:08.122723+0000","last_clean":"2024-03-11T20:53:08.122723+0000","last_became_active":"2024-03-11T20:50:28.065245+0000","last_became_peered":"2024-03-11T20:50:28.065245+0000","last_unstale":"2024-03-11T20:53:08.122723+0000","last_undegraded":"2024-03-11T20:53:08.122723+0000","last_fullsized":"2024-03-11T20:53:08.122723+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:05.364429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:53:05.364429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:25.580910+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"84'30","reported_seq":175,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.089112+0000","last_change":"2024-03-11T20:52:48.167124+0000","last_active":"2024-03-11T20:53:08.089112+0000","last_peered":"2024-03-11T20:53:08.089112+0000","last_clean":"2024-03-11T20:53:08.089112+0000","last_became_active":"2024-03-11T20:50:30.095177+0000","last_became_peered":"2024-03-11T20:50:30.095177+0000","last_unstale":"2024-03-11T20:53:08.089112+0000","last_undegraded":"2024-03-11T20:53:08.089112+0000","last_fullsized":"2024-03-11T20:53:08.089112+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'30","last_scrub_stamp":"2024-03-11T20:52:48.166893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:52:48.166893+0000","objects_scrubbed":10,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:05.469139+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":51,"num_read_kb":36,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"84'11","reported_seq":122,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.116961+0000","last_change":"2024-03-11T20:52:06.873338+0000","last_active":"2024-03-11T20:53:08.116961+0000","last_peered":"2024-03-11T20:53:08.116961+0000","last_clean":"2024-03-11T20:53:08.116961+0000","last_became_active":"2024-03-11T20:50:32.649414+0000","last_became_peered":"2024-03-11T20:50:32.649414+0000","last_unstale":"2024-03-11T20:53:08.116961+0000","last_undegraded":"2024-03-11T20:53:08.116961+0000","last_fullsized":"2024-03-11T20:53:08.116961+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T20:52:06.873096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:52:06.873096+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:33.849748+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":76,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.118393+0000","last_change":"2024-03-11T20:51:44.593465+0000","last_active":"2024-03-11T20:53:08.118393+0000","last_peered":"2024-03-11T20:53:08.118393+0000","last_clean":"2024-03-11T20:53:08.118393+0000","last_became_active":"2024-03-11T20:50:34.323647+0000","last_became_peered":"2024-03-11T20:50:34.323647+0000","last_unstale":"2024-03-11T20:53:08.118393+0000","last_undegraded":"2024-03-11T20:53:08.118393+0000","last_fullsized":"2024-03-11T20:53:08.118393+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:44.593255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:51:44.593255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:11.206090+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":87,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.117582+0000","last_change":"2024-03-11T20:51:53.560890+0000","last_active":"2024-03-11T20:53:08.117582+0000","last_peered":"2024-03-11T20:53:08.117582+0000","last_clean":"2024-03-11T20:53:08.117582+0000","last_became_active":"2024-03-11T20:50:32.650102+0000","last_became_peered":"2024-03-11T20:50:32.650102+0000","last_unstale":"2024-03-11T20:53:08.117582+0000","last_undegraded":"2024-03-11T20:53:08.117582+0000","last_fullsized":"2024-03-11T20:53:08.117582+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:53.560509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:51:53.560509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:11.831119+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0022993710000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"77'1","reported_seq":89,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.090235+0000","last_change":"2024-03-11T20:52:11.202801+0000","last_active":"2024-03-11T20:53:08.090235+0000","last_peered":"2024-03-11T20:53:08.090235+0000","last_clean":"2024-03-11T20:53:08.090235+0000","last_became_active":"2024-03-11T20:50:28.065076+0000","last_became_peered":"2024-03-11T20:50:28.065076+0000","last_unstale":"2024-03-11T20:53:08.090235+0000","last_undegraded":"2024-03-11T20:53:08.090235+0000","last_fullsized":"2024-03-11T20:53:08.090235+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-03-11T20:52:11.202551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:52:11.202551+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:15.958895+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"84'13","reported_seq":111,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.087983+0000","last_change":"2024-03-11T20:52:19.450300+0000","last_active":"2024-03-11T20:53:08.087983+0000","last_peered":"2024-03-11T20:53:08.087983+0000","last_clean":"2024-03-11T20:53:08.087983+0000","last_became_active":"2024-03-11T20:50:30.097669+0000","last_became_peered":"2024-03-11T20:50:30.097669+0000","last_unstale":"2024-03-11T20:53:08.087983+0000","last_undegraded":"2024-03-11T20:53:08.087983+0000","last_fullsized":"2024-03-11T20:53:08.087983+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'13","last_scrub_stamp":"2024-03-11T20:52:19.449958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:52:19.449958+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:20.076288+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"84'11","reported_seq":123,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.085641+0000","last_change":"2024-03-11T20:52:04.386639+0000","last_active":"2024-03-11T20:53:08.085641+0000","last_peered":"2024-03-11T20:53:08.085641+0000","last_clean":"2024-03-11T20:53:08.085641+0000","last_became_active":"2024-03-11T20:50:32.647041+0000","last_became_peered":"2024-03-11T20:50:32.647041+0000","last_unstale":"2024-03-11T20:53:08.085641+0000","last_undegraded":"2024-03-11T20:53:08.085641+0000","last_fullsized":"2024-03-11T20:53:08.085641+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T20:52:04.386516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:52:04.386516+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:12.473327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":84,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.121809+0000","last_change":"2024-03-11T20:53:06.344681+0000","last_active":"2024-03-11T20:53:08.121809+0000","last_peered":"2024-03-11T20:53:08.121809+0000","last_clean":"2024-03-11T20:53:08.121809+0000","last_became_active":"2024-03-11T20:50:34.326947+0000","last_became_peered":"2024-03-11T20:50:34.326947+0000","last_unstale":"2024-03-11T20:53:08.121809+0000","last_undegraded":"2024-03-11T20:53:08.121809+0000","last_fullsized":"2024-03-11T20:53:08.121809+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:06.344386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:53:06.344386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:09.051382+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"91'5","reported_seq":236,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.493671+0000","last_change":"2024-03-11T20:53:08.493671+0000","last_active":"2024-03-11T20:53:08.493671+0000","last_peered":"2024-03-11T20:53:08.493671+0000","last_clean":"2024-03-11T20:53:08.493671+0000","last_became_active":"2024-03-11T20:50:32.211118+0000","last_became_peered":"2024-03-11T20:50:32.211118+0000","last_unstale":"2024-03-11T20:53:08.493671+0000","last_undegraded":"2024-03-11T20:53:08.493671+0000","last_fullsized":"2024-03-11T20:53:08.493671+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'5","last_scrub_stamp":"2024-03-11T20:53:08.493516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:53:08.493516+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:17.353002+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0035533370000000002,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":138,"num_read_kb":133,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":88,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.086969+0000","last_change":"2024-03-11T20:52:14.613926+0000","last_active":"2024-03-11T20:53:08.086969+0000","last_peered":"2024-03-11T20:53:08.086969+0000","last_clean":"2024-03-11T20:53:08.086969+0000","last_became_active":"2024-03-11T20:50:28.071707+0000","last_became_peered":"2024-03-11T20:50:28.071707+0000","last_unstale":"2024-03-11T20:53:08.086969+0000","last_undegraded":"2024-03-11T20:53:08.086969+0000","last_fullsized":"2024-03-11T20:53:08.086969+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:14.613707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:52:14.613707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:27.285465+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"84'12","reported_seq":101,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.117746+0000","last_change":"2024-03-11T20:52:04.904973+0000","last_active":"2024-03-11T20:53:08.117746+0000","last_peered":"2024-03-11T20:53:08.117746+0000","last_clean":"2024-03-11T20:53:08.117746+0000","last_became_active":"2024-03-11T20:50:30.096350+0000","last_became_peered":"2024-03-11T20:50:30.096350+0000","last_unstale":"2024-03-11T20:53:08.117746+0000","last_undegraded":"2024-03-11T20:53:08.117746+0000","last_fullsized":"2024-03-11T20:53:08.117746+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'12","last_scrub_stamp":"2024-03-11T20:52:04.904721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:52:04.904721+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:31.218718+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":80,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.118328+0000","last_change":"2024-03-11T20:52:09.577290+0000","last_active":"2024-03-11T20:53:08.118328+0000","last_peered":"2024-03-11T20:53:08.118328+0000","last_clean":"2024-03-11T20:53:08.118328+0000","last_became_active":"2024-03-11T20:50:32.190896+0000","last_became_peered":"2024-03-11T20:50:32.190896+0000","last_unstale":"2024-03-11T20:53:08.118328+0000","last_undegraded":"2024-03-11T20:53:08.118328+0000","last_fullsized":"2024-03-11T20:53:08.118328+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:09.577102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:52:09.577102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:13.346655+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":76,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.086058+0000","last_change":"2024-03-11T20:52:33.343010+0000","last_active":"2024-03-11T20:53:08.086058+0000","last_peered":"2024-03-11T20:53:08.086058+0000","last_clean":"2024-03-11T20:53:08.086058+0000","last_became_active":"2024-03-11T20:50:34.335370+0000","last_became_peered":"2024-03-11T20:50:34.335370+0000","last_unstale":"2024-03-11T20:53:08.086058+0000","last_undegraded":"2024-03-11T20:53:08.086058+0000","last_fullsized":"2024-03-11T20:53:08.086058+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:33.342759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:52:33.342759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:33.629066+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"84'11","reported_seq":123,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.124132+0000","last_change":"2024-03-11T20:52:15.500805+0000","last_active":"2024-03-11T20:53:08.124132+0000","last_peered":"2024-03-11T20:53:08.124132+0000","last_clean":"2024-03-11T20:53:08.124132+0000","last_became_active":"2024-03-11T20:50:32.197412+0000","last_became_peered":"2024-03-11T20:50:32.197412+0000","last_unstale":"2024-03-11T20:53:08.124132+0000","last_undegraded":"2024-03-11T20:53:08.124132+0000","last_fullsized":"2024-03-11T20:53:08.124132+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T20:52:15.500571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:52:15.500571+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:37.062481+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":88,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.119066+0000","last_change":"2024-03-11T20:52:09.043821+0000","last_active":"2024-03-11T20:53:08.119066+0000","last_peered":"2024-03-11T20:53:08.119066+0000","last_clean":"2024-03-11T20:53:08.119066+0000","last_became_active":"2024-03-11T20:50:28.079413+0000","last_became_peered":"2024-03-11T20:50:28.079413+0000","last_unstale":"2024-03-11T20:53:08.119066+0000","last_undegraded":"2024-03-11T20:53:08.119066+0000","last_fullsized":"2024-03-11T20:53:08.119066+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:09.043655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:52:09.043655+0000","last_clean_scrub_stamp":"2024-03-11T20:52:09.043655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:17.358596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"84'12","reported_seq":107,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.118000+0000","last_change":"2024-03-11T20:51:59.529431+0000","last_active":"2024-03-11T20:53:08.118000+0000","last_peered":"2024-03-11T20:53:08.118000+0000","last_clean":"2024-03-11T20:53:08.118000+0000","last_became_active":"2024-03-11T20:50:30.083388+0000","last_became_peered":"2024-03-11T20:50:30.083388+0000","last_unstale":"2024-03-11T20:53:08.118000+0000","last_undegraded":"2024-03-11T20:53:08.118000+0000","last_fullsized":"2024-03-11T20:53:08.118000+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'12","last_scrub_stamp":"2024-03-11T20:51:59.529049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:51:59.529049+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:08.466940+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":84,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.120439+0000","last_change":"2024-03-11T20:53:06.771715+0000","last_active":"2024-03-11T20:53:08.120439+0000","last_peered":"2024-03-11T20:53:08.120439+0000","last_clean":"2024-03-11T20:53:08.120439+0000","last_became_active":"2024-03-11T20:50:34.337717+0000","last_became_peered":"2024-03-11T20:50:34.337717+0000","last_unstale":"2024-03-11T20:53:08.120439+0000","last_undegraded":"2024-03-11T20:53:08.120439+0000","last_fullsized":"2024-03-11T20:53:08.120439+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:06.771620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:53:06.771620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:30.302985+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":53,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.115286+0000","last_change":"2024-03-11T20:53:05.836875+0000","last_active":"2024-03-11T20:53:08.115286+0000","last_peered":"2024-03-11T20:53:08.115286+0000","last_clean":"2024-03-11T20:53:08.115286+0000","last_became_active":"2024-03-11T20:51:22.662693+0000","last_became_peered":"2024-03-11T20:51:22.662693+0000","last_unstale":"2024-03-11T20:53:08.115286+0000","last_undegraded":"2024-03-11T20:53:08.115286+0000","last_fullsized":"2024-03-11T20:53:08.115286+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:05.836655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:53:05.836655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:21.447337+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":88,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.084710+0000","last_change":"2024-03-11T20:51:40.974308+0000","last_active":"2024-03-11T20:53:08.084710+0000","last_peered":"2024-03-11T20:53:08.084710+0000","last_clean":"2024-03-11T20:53:08.084710+0000","last_became_active":"2024-03-11T20:50:28.058790+0000","last_became_peered":"2024-03-11T20:50:28.058790+0000","last_unstale":"2024-03-11T20:53:08.084710+0000","last_undegraded":"2024-03-11T20:53:08.084710+0000","last_fullsized":"2024-03-11T20:53:08.084710+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:40.974147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:40.974147+0000","last_clean_scrub_stamp":"2024-03-11T20:51:40.974147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"84'15","reported_seq":109,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.085187+0000","last_change":"2024-03-11T20:52:40.434414+0000","last_active":"2024-03-11T20:53:08.085187+0000","last_peered":"2024-03-11T20:53:08.085187+0000","last_clean":"2024-03-11T20:53:08.085187+0000","last_became_active":"2024-03-11T20:50:30.096215+0000","last_became_peered":"2024-03-11T20:50:30.096215+0000","last_unstale":"2024-03-11T20:53:08.085187+0000","last_undegraded":"2024-03-11T20:53:08.085187+0000","last_fullsized":"2024-03-11T20:53:08.085187+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15","last_scrub_stamp":"2024-03-11T20:52:40.434169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:52:40.434169+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:45.160041+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":76,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.123840+0000","last_change":"2024-03-11T20:52:12.514160+0000","last_active":"2024-03-11T20:53:08.123840+0000","last_peered":"2024-03-11T20:53:08.123840+0000","last_clean":"2024-03-11T20:53:08.123840+0000","last_became_active":"2024-03-11T20:50:34.331803+0000","last_became_peered":"2024-03-11T20:50:34.331803+0000","last_unstale":"2024-03-11T20:53:08.123840+0000","last_undegraded":"2024-03-11T20:53:08.123840+0000","last_fullsized":"2024-03-11T20:53:08.123840+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:12.513982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:52:12.513982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:31.458459+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":88,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.086548+0000","last_change":"2024-03-11T20:52:23.165274+0000","last_active":"2024-03-11T20:53:08.086548+0000","last_peered":"2024-03-11T20:53:08.086548+0000","last_clean":"2024-03-11T20:53:08.086548+0000","last_became_active":"2024-03-11T20:50:28.057121+0000","last_became_peered":"2024-03-11T20:50:28.057121+0000","last_unstale":"2024-03-11T20:53:08.086548+0000","last_undegraded":"2024-03-11T20:53:08.086548+0000","last_fullsized":"2024-03-11T20:53:08.086548+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:23.165119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:52:23.165119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:34.501490+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"84'9","reported_seq":108,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.091028+0000","last_change":"2024-03-11T20:53:03.452779+0000","last_active":"2024-03-11T20:53:08.091028+0000","last_peered":"2024-03-11T20:53:08.091028+0000","last_clean":"2024-03-11T20:53:08.091028+0000","last_became_active":"2024-03-11T20:50:30.089260+0000","last_became_peered":"2024-03-11T20:50:30.089260+0000","last_unstale":"2024-03-11T20:53:08.091028+0000","last_undegraded":"2024-03-11T20:53:08.091028+0000","last_fullsized":"2024-03-11T20:53:08.091028+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'9","last_scrub_stamp":"2024-03-11T20:53:03.452528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:53:03.452528+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:23.786382+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":88,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.119706+0000","last_change":"2024-03-11T20:53:00.551613+0000","last_active":"2024-03-11T20:53:08.119706+0000","last_peered":"2024-03-11T20:53:08.119706+0000","last_clean":"2024-03-11T20:53:08.119706+0000","last_became_active":"2024-03-11T20:50:32.200001+0000","last_became_peered":"2024-03-11T20:50:32.200001+0000","last_unstale":"2024-03-11T20:53:08.119706+0000","last_undegraded":"2024-03-11T20:53:08.119706+0000","last_fullsized":"2024-03-11T20:53:08.119706+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:00.551395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:35.670149+0000","last_clean_scrub_stamp":"2024-03-11T20:53:00.551395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:08.450926+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":84,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.115624+0000","last_change":"2024-03-11T20:52:54.879520+0000","last_active":"2024-03-11T20:53:08.115624+0000","last_peered":"2024-03-11T20:53:08.115624+0000","last_clean":"2024-03-11T20:53:08.115624+0000","last_became_active":"2024-03-11T20:50:34.326354+0000","last_became_peered":"2024-03-11T20:50:34.326354+0000","last_unstale":"2024-03-11T20:53:08.115624+0000","last_undegraded":"2024-03-11T20:53:08.115624+0000","last_fullsized":"2024-03-11T20:53:08.115624+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:54.879292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:52:54.879292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:00.895826+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"98'12","reported_seq":133,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.125020+0000","last_change":"2024-03-11T20:52:01.513643+0000","last_active":"2024-03-11T20:53:08.125020+0000","last_peered":"2024-03-11T20:53:08.125020+0000","last_clean":"2024-03-11T20:53:08.125020+0000","last_became_active":"2024-03-11T20:51:22.713613+0000","last_became_peered":"2024-03-11T20:51:22.713613+0000","last_unstale":"2024-03-11T20:53:08.125020+0000","last_undegraded":"2024-03-11T20:53:08.125020+0000","last_fullsized":"2024-03-11T20:53:08.125020+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'12","last_scrub_stamp":"2024-03-11T20:52:01.513305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:52:01.513305+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:20.040552+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":96,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.122061+0000","last_change":"2024-03-11T20:52:51.161508+0000","last_active":"2024-03-11T20:53:08.122061+0000","last_peered":"2024-03-11T20:53:08.122061+0000","last_clean":"2024-03-11T20:53:08.122061+0000","last_became_active":"2024-03-11T20:50:28.069267+0000","last_became_peered":"2024-03-11T20:50:28.069267+0000","last_unstale":"2024-03-11T20:53:08.122061+0000","last_undegraded":"2024-03-11T20:53:08.122061+0000","last_fullsized":"2024-03-11T20:53:08.122061+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:51.161281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:52:51.161281+0000","last_clean_scrub_stamp":"2024-03-11T20:52:51.161281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:57.333795+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"84'19","reported_seq":115,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.086784+0000","last_change":"2024-03-11T20:52:04.651969+0000","last_active":"2024-03-11T20:53:08.086784+0000","last_peered":"2024-03-11T20:53:08.086784+0000","last_clean":"2024-03-11T20:53:08.086784+0000","last_became_active":"2024-03-11T20:50:30.103666+0000","last_became_peered":"2024-03-11T20:50:30.103666+0000","last_unstale":"2024-03-11T20:53:08.086784+0000","last_undegraded":"2024-03-11T20:53:08.086784+0000","last_fullsized":"2024-03-11T20:53:08.086784+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'19","last_scrub_stamp":"2024-03-11T20:52:04.651612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:52:04.651612+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":76,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.118464+0000","last_change":"2024-03-11T20:51:49.161619+0000","last_active":"2024-03-11T20:53:08.118464+0000","last_peered":"2024-03-11T20:53:08.118464+0000","last_clean":"2024-03-11T20:53:08.118464+0000","last_became_active":"2024-03-11T20:50:34.327139+0000","last_became_peered":"2024-03-11T20:50:34.327139+0000","last_unstale":"2024-03-11T20:53:08.118464+0000","last_undegraded":"2024-03-11T20:53:08.118464+0000","last_fullsized":"2024-03-11T20:53:08.118464+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:49.161328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:51:49.161328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:15.535434+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":96,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.797266+0000","last_change":"2024-03-11T20:53:08.797289+0000","last_active":"2024-03-11T20:53:08.797266+0000","last_peered":"2024-03-11T20:53:08.797266+0000","last_clean":"2024-03-11T20:53:08.797266+0000","last_became_active":"2024-03-11T20:50:28.049901+0000","last_became_peered":"2024-03-11T20:50:28.049901+0000","last_unstale":"2024-03-11T20:53:08.797266+0000","last_undegraded":"2024-03-11T20:53:08.797266+0000","last_fullsized":"2024-03-11T20:53:08.797266+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:08.797203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:53:08.797203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:12.120056+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"102'25","reported_seq":128,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.122777+0000","last_change":"2024-03-11T20:52:47.207949+0000","last_active":"2024-03-11T20:53:08.122777+0000","last_peered":"2024-03-11T20:53:08.122777+0000","last_clean":"2024-03-11T20:53:08.122777+0000","last_became_active":"2024-03-11T20:50:30.111819+0000","last_became_peered":"2024-03-11T20:50:30.111819+0000","last_unstale":"2024-03-11T20:53:08.122777+0000","last_undegraded":"2024-03-11T20:53:08.122777+0000","last_fullsized":"2024-03-11T20:53:08.122777+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"102'25","last_scrub_stamp":"2024-03-11T20:52:47.207710+0000","last_deep_scrub":"102'25","last_deep_scrub_stamp":"2024-03-11T20:51:35.187094+0000","last_clean_scrub_stamp":"2024-03-11T20:52:47.207710+0000","objects_scrubbed":8,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:55.994836+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":80,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.085954+0000","last_change":"2024-03-11T20:52:19.576642+0000","last_active":"2024-03-11T20:53:08.085954+0000","last_peered":"2024-03-11T20:53:08.085954+0000","last_clean":"2024-03-11T20:53:08.085954+0000","last_became_active":"2024-03-11T20:50:32.649877+0000","last_became_peered":"2024-03-11T20:50:32.649877+0000","last_unstale":"2024-03-11T20:53:08.085954+0000","last_undegraded":"2024-03-11T20:53:08.085954+0000","last_fullsized":"2024-03-11T20:53:08.085954+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:19.576426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:52:19.576426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:20.103831+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":83,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.114771+0000","last_change":"2024-03-11T20:52:46.946707+0000","last_active":"2024-03-11T20:53:08.114771+0000","last_peered":"2024-03-11T20:53:08.114771+0000","last_clean":"2024-03-11T20:53:08.114771+0000","last_became_active":"2024-03-11T20:50:34.319536+0000","last_became_peered":"2024-03-11T20:50:34.319536+0000","last_unstale":"2024-03-11T20:53:08.114771+0000","last_undegraded":"2024-03-11T20:53:08.114771+0000","last_fullsized":"2024-03-11T20:53:08.114771+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:46.946561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:52:46.946561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:52.449594+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":88,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.089363+0000","last_change":"2024-03-11T20:52:01.105719+0000","last_active":"2024-03-11T20:53:08.089363+0000","last_peered":"2024-03-11T20:53:08.089363+0000","last_clean":"2024-03-11T20:53:08.089363+0000","last_became_active":"2024-03-11T20:50:28.043982+0000","last_became_peered":"2024-03-11T20:50:28.043982+0000","last_unstale":"2024-03-11T20:53:08.089363+0000","last_undegraded":"2024-03-11T20:53:08.089363+0000","last_fullsized":"2024-03-11T20:53:08.089363+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:01.105491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:52:01.105491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:29.786816+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"84'10","reported_seq":99,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.120663+0000","last_change":"2024-03-11T20:51:55.906909+0000","last_active":"2024-03-11T20:53:08.120663+0000","last_peered":"2024-03-11T20:53:08.120663+0000","last_clean":"2024-03-11T20:53:08.120663+0000","last_became_active":"2024-03-11T20:50:30.109549+0000","last_became_peered":"2024-03-11T20:50:30.109549+0000","last_unstale":"2024-03-11T20:53:08.120663+0000","last_undegraded":"2024-03-11T20:53:08.120663+0000","last_fullsized":"2024-03-11T20:53:08.120663+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'10","last_scrub_stamp":"2024-03-11T20:51:55.906649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:51:55.906649+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:11.023881+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":87,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.117038+0000","last_change":"2024-03-11T20:52:51.931398+0000","last_active":"2024-03-11T20:53:08.117038+0000","last_peered":"2024-03-11T20:53:08.117038+0000","last_clean":"2024-03-11T20:53:08.117038+0000","last_became_active":"2024-03-11T20:50:32.194119+0000","last_became_peered":"2024-03-11T20:50:32.194119+0000","last_unstale":"2024-03-11T20:53:08.117038+0000","last_undegraded":"2024-03-11T20:53:08.117038+0000","last_fullsized":"2024-03-11T20:53:08.117038+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:51.931295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:52:51.931295+0000","last_clean_scrub_stamp":"2024-03-11T20:52:51.931295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:55.241599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":84,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.122543+0000","last_change":"2024-03-11T20:53:03.762967+0000","last_active":"2024-03-11T20:53:08.122543+0000","last_peered":"2024-03-11T20:53:08.122543+0000","last_clean":"2024-03-11T20:53:08.122543+0000","last_became_active":"2024-03-11T20:50:34.337173+0000","last_became_peered":"2024-03-11T20:50:34.337173+0000","last_unstale":"2024-03-11T20:53:08.122543+0000","last_undegraded":"2024-03-11T20:53:08.122543+0000","last_fullsized":"2024-03-11T20:53:08.122543+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:03.762882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:53:03.762882+0000","last_clean_scrub_stamp":"2024-03-11T20:53:03.762882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:29.931672+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":96,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.114947+0000","last_change":"2024-03-11T20:53:00.897188+0000","last_active":"2024-03-11T20:53:08.114947+0000","last_peered":"2024-03-11T20:53:08.114947+0000","last_clean":"2024-03-11T20:53:08.114947+0000","last_became_active":"2024-03-11T20:50:28.058942+0000","last_became_peered":"2024-03-11T20:50:28.058942+0000","last_unstale":"2024-03-11T20:53:08.114947+0000","last_undegraded":"2024-03-11T20:53:08.114947+0000","last_fullsized":"2024-03-11T20:53:08.114947+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:00.896807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:53:00.896807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:13.585784+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"84'15","reported_seq":117,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:09.345702+0000","last_change":"2024-03-11T20:53:09.345898+0000","last_active":"2024-03-11T20:53:09.345702+0000","last_peered":"2024-03-11T20:53:09.345702+0000","last_clean":"2024-03-11T20:53:09.345702+0000","last_became_active":"2024-03-11T20:50:30.101512+0000","last_became_peered":"2024-03-11T20:50:30.101512+0000","last_unstale":"2024-03-11T20:53:09.345702+0000","last_undegraded":"2024-03-11T20:53:09.345702+0000","last_fullsized":"2024-03-11T20:53:09.345702+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15","last_scrub_stamp":"2024-03-11T20:53:09.345405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:53:09.345405+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:28.384531+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":80,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-11T20:53:08.118216+0000","last_change":"2024-03-11T20:52:05.877747+0000","last_active":"2024-03-11T20:53:08.118216+0000","last_peered":"2024-03-11T20:53:08.118216+0000","last_clean":"2024-03-11T20:53:08.118216+0000","last_became_active":"2024-03-11T20:50:32.652854+0000","last_became_peered":"2024-03-11T20:50:32.652854+0000","last_unstale":"2024-03-11T20:53:08.118216+0000","last_undegraded":"2024-03-11T20:53:08.118216+0000","last_fullsized":"2024-03-11T20:53:08.118216+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:05.877396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:52:05.877396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:32.105270+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":138,"num_read_kb":133,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":701,"num_read_kb":458,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":428,"ondisk_log_size":428,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":68,"seq":292057776171,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32564,"kb_used_data":1384,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712588,"statfs":{"total":95995035648,"available":95961690112,"internally_reserved":0,"allocated":1417216,"data_stored":993106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":57,"seq":244813135925,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32548,"kb_used_data":1368,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712604,"statfs":{"total":95995035648,"available":95961706496,"internally_reserved":0,"allocated":1400832,"data_stored":993116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574952,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31976,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713176,"statfs":{"total":95995035648,"available":95962292224,"internally_reserved":0,"allocated":815104,"data_stored":402994,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738483,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32556,"kb_used_data":1376,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712596,"statfs":{"total":95995035648,"available":95961698304,"internally_reserved":0,"allocated":1409024,"data_stored":992048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378718,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31956,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713196,"statfs":{"total":95995035648,"available":95962312704,"internally_reserved":0,"allocated":798720,"data_stored":401842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":28,"seq":120259084371,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27800,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717352,"statfs":{"total":95995035648,"available":95966568448,"internally_reserved":0,"allocated":802816,"data_stored":402218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":35,"seq":150323855434,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32000,"kb_used_data":820,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713152,"statfs":{"total":95995035648,"available":95962267648,"internally_reserved":0,"allocated":839680,"data_stored":403608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":44,"seq":188978561087,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31964,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713188,"statfs":{"total":95995035648,"available":95962304512,"internally_reserved":0,"allocated":802816,"data_stored":401764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T20:53:14.457 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T20:53:14.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:14 smithi146 bash[23809]: cluster 2024-03-11T20:53:12.818458+0000 osd.5 (osd.5) 63 : cluster 0 3.c scrub starts 2024-03-11T20:53:14.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:14 smithi146 bash[23809]: cluster 2024-03-11T20:53:12.822831+0000 osd.5 (osd.5) 64 : cluster 0 3.c scrub ok 2024-03-11T20:53:14.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:14 smithi146 bash[23809]: cluster 2024-03-11T20:53:13.134307+0000 osd.1 (osd.1) 57 : cluster 0 3.19 scrub starts 2024-03-11T20:53:14.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:14 smithi146 bash[23809]: cluster 2024-03-11T20:53:13.138903+0000 osd.1 (osd.1) 58 : cluster 0 3.19 scrub ok 2024-03-11T20:53:14.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:14 smithi146 bash[23809]: cluster 2024-03-11T20:53:13.555554+0000 osd.4 (osd.4) 59 : cluster 0 3.3 scrub starts 2024-03-11T20:53:14.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:14 smithi146 bash[23809]: cluster 2024-03-11T20:53:13.560296+0000 osd.4 (osd.4) 60 : cluster 0 3.3 scrub ok 2024-03-11T20:53:14.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:14 smithi146 bash[23809]: audit 2024-03-11T20:53:13.821297+0000 mon.a (mon.0) 904 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:53:14.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:14 smithi146 bash[23809]: audit 2024-03-11T20:53:13.828165+0000 mon.a (mon.0) 905 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:53:15.260 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 20:53:14 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T20:53:15.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:15 smithi146 bash[23809]: cluster 2024-03-11T20:53:13.410083+0000 osd.3 (osd.3) 79 : cluster 0 4.0 scrub starts 2024-03-11T20:53:15.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:15 smithi146 bash[23809]: cluster 2024-03-11T20:53:13.413134+0000 osd.3 (osd.3) 80 : cluster 0 4.0 scrub ok 2024-03-11T20:53:15.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:15 smithi146 bash[23809]: audit 2024-03-11T20:53:13.778968+0000 mgr.y (mgr.24380) 93 : audit 0 from='client.24542 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T20:53:15.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:15 smithi146 bash[23809]: cluster 2024-03-11T20:53:13.855360+0000 mgr.y (mgr.24380) 94 : cluster 0 pgmap v64: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T20:53:15.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:15 smithi146 bash[23809]: cluster 2024-03-11T20:53:14.107224+0000 osd.1 (osd.1) 59 : cluster 0 2.15 scrub starts 2024-03-11T20:53:15.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:15 smithi146 bash[23809]: cluster 2024-03-11T20:53:14.109730+0000 osd.1 (osd.1) 60 : cluster 0 2.15 scrub ok 2024-03-11T20:53:15.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:15 smithi146 bash[23809]: cluster 2024-03-11T20:53:14.166036+0000 osd.0 (osd.0) 69 : cluster 0 6.18 scrub starts 2024-03-11T20:53:15.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:15 smithi146 bash[23809]: cluster 2024-03-11T20:53:14.168288+0000 osd.0 (osd.0) 70 : cluster 0 6.18 scrub ok 2024-03-11T20:53:15.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:15 smithi146 bash[23809]: cluster 2024-03-11T20:53:14.599818+0000 osd.4 (osd.4) 61 : cluster 0 6.17 scrub starts 2024-03-11T20:53:15.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:15 smithi146 bash[23809]: cluster 2024-03-11T20:53:14.601433+0000 osd.4 (osd.4) 62 : cluster 0 6.17 scrub ok 2024-03-11T20:53:15.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:15 smithi146 bash[23809]: audit 2024-03-11T20:53:14.680901+0000 mon.a (mon.0) 906 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:53:15.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:15 smithi146 bash[23809]: audit 2024-03-11T20:53:14.686953+0000 mon.a (mon.0) 907 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:53:15.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:15 smithi023 bash[21372]: cluster 2024-03-11T20:53:13.410083+0000 osd.3 (osd.3) 79 : cluster 0 4.0 scrub starts 2024-03-11T20:53:15.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:15 smithi023 bash[21372]: cluster 2024-03-11T20:53:13.413134+0000 osd.3 (osd.3) 80 : cluster 0 4.0 scrub ok 2024-03-11T20:53:15.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:15 smithi023 bash[21372]: audit 2024-03-11T20:53:13.778968+0000 mgr.y (mgr.24380) 93 : audit 0 from='client.24542 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T20:53:15.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:15 smithi023 bash[21372]: cluster 2024-03-11T20:53:13.855360+0000 mgr.y (mgr.24380) 94 : cluster 0 pgmap v64: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T20:53:15.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:15 smithi023 bash[21372]: cluster 2024-03-11T20:53:14.107224+0000 osd.1 (osd.1) 59 : cluster 0 2.15 scrub starts 2024-03-11T20:53:15.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:15 smithi023 bash[21372]: cluster 2024-03-11T20:53:14.109730+0000 osd.1 (osd.1) 60 : cluster 0 2.15 scrub ok 2024-03-11T20:53:15.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:15 smithi023 bash[21372]: cluster 2024-03-11T20:53:14.166036+0000 osd.0 (osd.0) 69 : cluster 0 6.18 scrub starts 2024-03-11T20:53:15.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:15 smithi023 bash[21372]: cluster 2024-03-11T20:53:14.168288+0000 osd.0 (osd.0) 70 : cluster 0 6.18 scrub ok 2024-03-11T20:53:15.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:15 smithi023 bash[21372]: cluster 2024-03-11T20:53:14.599818+0000 osd.4 (osd.4) 61 : cluster 0 6.17 scrub starts 2024-03-11T20:53:15.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:15 smithi023 bash[21372]: cluster 2024-03-11T20:53:14.601433+0000 osd.4 (osd.4) 62 : cluster 0 6.17 scrub ok 2024-03-11T20:53:15.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:15 smithi023 bash[21372]: audit 2024-03-11T20:53:14.680901+0000 mon.a (mon.0) 906 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:53:15.536 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:15 smithi023 bash[21372]: audit 2024-03-11T20:53:14.686953+0000 mon.a (mon.0) 907 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:53:15.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:15 smithi023 bash[29146]: cluster 2024-03-11T20:53:13.410083+0000 osd.3 (osd.3) 79 : cluster 0 4.0 scrub starts 2024-03-11T20:53:15.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:15 smithi023 bash[29146]: cluster 2024-03-11T20:53:13.413134+0000 osd.3 (osd.3) 80 : cluster 0 4.0 scrub ok 2024-03-11T20:53:15.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:15 smithi023 bash[29146]: audit 2024-03-11T20:53:13.778968+0000 mgr.y (mgr.24380) 93 : audit 0 from='client.24542 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T20:53:15.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:15 smithi023 bash[29146]: cluster 2024-03-11T20:53:13.855360+0000 mgr.y (mgr.24380) 94 : cluster 0 pgmap v64: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T20:53:15.537 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:15 smithi023 bash[29146]: cluster 2024-03-11T20:53:14.107224+0000 osd.1 (osd.1) 59 : cluster 0 2.15 scrub starts 2024-03-11T20:53:15.537 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:15 smithi023 bash[29146]: cluster 2024-03-11T20:53:14.109730+0000 osd.1 (osd.1) 60 : cluster 0 2.15 scrub ok 2024-03-11T20:53:15.537 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:15 smithi023 bash[29146]: cluster 2024-03-11T20:53:14.166036+0000 osd.0 (osd.0) 69 : cluster 0 6.18 scrub starts 2024-03-11T20:53:15.537 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:15 smithi023 bash[29146]: cluster 2024-03-11T20:53:14.168288+0000 osd.0 (osd.0) 70 : cluster 0 6.18 scrub ok 2024-03-11T20:53:15.537 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:15 smithi023 bash[29146]: cluster 2024-03-11T20:53:14.599818+0000 osd.4 (osd.4) 61 : cluster 0 6.17 scrub starts 2024-03-11T20:53:15.537 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:15 smithi023 bash[29146]: cluster 2024-03-11T20:53:14.601433+0000 osd.4 (osd.4) 62 : cluster 0 6.17 scrub ok 2024-03-11T20:53:15.538 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:15 smithi023 bash[29146]: audit 2024-03-11T20:53:14.680901+0000 mon.a (mon.0) 906 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:53:15.538 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:15 smithi023 bash[29146]: audit 2024-03-11T20:53:14.686953+0000 mon.a (mon.0) 907 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:53:16.401 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:16 smithi023 bash[21372]: audit 2024-03-11T20:53:14.901870+0000 mgr.y (mgr.24380) 95 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:53:16.401 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:16 smithi023 bash[21372]: audit 2024-03-11T20:53:15.370410+0000 mon.a (mon.0) 908 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:53:16.401 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:16 smithi023 bash[21372]: audit 2024-03-11T20:53:15.372400+0000 mon.a (mon.0) 909 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:53:16.401 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:16 smithi023 bash[21372]: audit 2024-03-11T20:53:15.383275+0000 mon.a (mon.0) 910 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:53:16.402 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:16 smithi023 bash[29146]: audit 2024-03-11T20:53:14.901870+0000 mgr.y (mgr.24380) 95 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:53:16.402 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:16 smithi023 bash[29146]: audit 2024-03-11T20:53:15.370410+0000 mon.a (mon.0) 908 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:53:16.402 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:16 smithi023 bash[29146]: audit 2024-03-11T20:53:15.372400+0000 mon.a (mon.0) 909 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:53:16.402 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:16 smithi023 bash[29146]: audit 2024-03-11T20:53:15.383275+0000 mon.a (mon.0) 910 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:53:16.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:16 smithi146 bash[23809]: audit 2024-03-11T20:53:14.901870+0000 mgr.y (mgr.24380) 95 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:53:16.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:16 smithi146 bash[23809]: audit 2024-03-11T20:53:15.370410+0000 mon.a (mon.0) 908 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:53:16.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:16 smithi146 bash[23809]: audit 2024-03-11T20:53:15.372400+0000 mon.a (mon.0) 909 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:53:16.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:16 smithi146 bash[23809]: audit 2024-03-11T20:53:15.383275+0000 mon.a (mon.0) 910 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:53:16.784 INFO:journalctl@ceph.alertmanager.a.smithi023.stdout:Mar 11 20:53:16 smithi023 bash[62486]: ts=2024-03-11T20:53:16.404Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.002855574s 2024-03-11T20:53:17.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:17 smithi146 bash[23809]: cluster 2024-03-11T20:53:15.796069+0000 osd.2 (osd.2) 47 : cluster 0 4.1c scrub starts 2024-03-11T20:53:17.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:17 smithi146 bash[23809]: cluster 2024-03-11T20:53:15.798427+0000 osd.2 (osd.2) 48 : cluster 0 4.1c scrub ok 2024-03-11T20:53:17.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:17 smithi146 bash[23809]: cluster 2024-03-11T20:53:15.856774+0000 mgr.y (mgr.24380) 96 : cluster 0 pgmap v65: 132 pgs: 132 active+clean; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:53:17.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:17 smithi023 bash[21372]: cluster 2024-03-11T20:53:15.796069+0000 osd.2 (osd.2) 47 : cluster 0 4.1c scrub starts 2024-03-11T20:53:17.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:17 smithi023 bash[21372]: cluster 2024-03-11T20:53:15.798427+0000 osd.2 (osd.2) 48 : cluster 0 4.1c scrub ok 2024-03-11T20:53:17.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:17 smithi023 bash[21372]: cluster 2024-03-11T20:53:15.856774+0000 mgr.y (mgr.24380) 96 : cluster 0 pgmap v65: 132 pgs: 132 active+clean; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:53:17.534 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:17 smithi023 bash[29146]: cluster 2024-03-11T20:53:15.796069+0000 osd.2 (osd.2) 47 : cluster 0 4.1c scrub starts 2024-03-11T20:53:17.534 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:17 smithi023 bash[29146]: cluster 2024-03-11T20:53:15.798427+0000 osd.2 (osd.2) 48 : cluster 0 4.1c scrub ok 2024-03-11T20:53:17.534 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:17 smithi023 bash[29146]: cluster 2024-03-11T20:53:15.856774+0000 mgr.y (mgr.24380) 96 : cluster 0 pgmap v65: 132 pgs: 132 active+clean; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:53:18.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:18 smithi146 bash[23809]: cluster 2024-03-11T20:53:16.956780+0000 osd.6 (osd.6) 35 : cluster 0 3.a scrub starts 2024-03-11T20:53:18.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:18 smithi146 bash[23809]: cluster 2024-03-11T20:53:16.962353+0000 osd.6 (osd.6) 36 : cluster 0 3.a scrub ok 2024-03-11T20:53:18.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:18 smithi146 bash[23809]: cluster 2024-03-11T20:53:17.166476+0000 osd.1 (osd.1) 61 : cluster 0 2.9 scrub starts 2024-03-11T20:53:18.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:18 smithi146 bash[23809]: cluster 2024-03-11T20:53:17.168903+0000 osd.1 (osd.1) 62 : cluster 0 2.9 scrub ok 2024-03-11T20:53:18.534 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:53:18 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:20:53:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T20:53:18.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:18 smithi023 bash[21372]: cluster 2024-03-11T20:53:16.956780+0000 osd.6 (osd.6) 35 : cluster 0 3.a scrub starts 2024-03-11T20:53:18.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:18 smithi023 bash[21372]: cluster 2024-03-11T20:53:16.962353+0000 osd.6 (osd.6) 36 : cluster 0 3.a scrub ok 2024-03-11T20:53:18.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:18 smithi023 bash[21372]: cluster 2024-03-11T20:53:17.166476+0000 osd.1 (osd.1) 61 : cluster 0 2.9 scrub starts 2024-03-11T20:53:18.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:18 smithi023 bash[21372]: cluster 2024-03-11T20:53:17.168903+0000 osd.1 (osd.1) 62 : cluster 0 2.9 scrub ok 2024-03-11T20:53:18.534 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:18 smithi023 bash[29146]: cluster 2024-03-11T20:53:16.956780+0000 osd.6 (osd.6) 35 : cluster 0 3.a scrub starts 2024-03-11T20:53:18.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:18 smithi023 bash[29146]: cluster 2024-03-11T20:53:16.962353+0000 osd.6 (osd.6) 36 : cluster 0 3.a scrub ok 2024-03-11T20:53:18.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:18 smithi023 bash[29146]: cluster 2024-03-11T20:53:17.166476+0000 osd.1 (osd.1) 61 : cluster 0 2.9 scrub starts 2024-03-11T20:53:18.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:18 smithi023 bash[29146]: cluster 2024-03-11T20:53:17.168903+0000 osd.1 (osd.1) 62 : cluster 0 2.9 scrub ok 2024-03-11T20:53:19.259 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:53:19.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:19 smithi146 bash[23809]: cluster 2024-03-11T20:53:17.780380+0000 osd.5 (osd.5) 65 : cluster 0 3.16 scrub starts 2024-03-11T20:53:19.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:19 smithi146 bash[23809]: cluster 2024-03-11T20:53:17.784022+0000 osd.5 (osd.5) 66 : cluster 0 3.16 scrub ok 2024-03-11T20:53:19.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:19 smithi146 bash[23809]: cluster 2024-03-11T20:53:17.857481+0000 mgr.y (mgr.24380) 97 : cluster 0 pgmap v66: 132 pgs: 132 active+clean; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T20:53:19.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:19 smithi146 bash[23809]: cluster 2024-03-11T20:53:18.131965+0000 osd.0 (osd.0) 71 : cluster 0 3.17 scrub starts 2024-03-11T20:53:19.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:19 smithi146 bash[23809]: cluster 2024-03-11T20:53:18.135875+0000 osd.0 (osd.0) 72 : cluster 0 3.17 scrub ok 2024-03-11T20:53:19.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:19 smithi146 bash[23809]: cluster 2024-03-11T20:53:18.328996+0000 osd.7 (osd.7) 67 : cluster 0 6.8 scrub starts 2024-03-11T20:53:19.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:19 smithi146 bash[23809]: cluster 2024-03-11T20:53:18.331606+0000 osd.7 (osd.7) 68 : cluster 0 6.8 scrub ok 2024-03-11T20:53:19.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:19 smithi146 bash[23809]: audit 2024-03-11T20:53:19.113494+0000 mon.a (mon.0) 911 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:53:19.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:19 smithi023 bash[21372]: cluster 2024-03-11T20:53:17.780380+0000 osd.5 (osd.5) 65 : cluster 0 3.16 scrub starts 2024-03-11T20:53:19.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:19 smithi023 bash[21372]: cluster 2024-03-11T20:53:17.784022+0000 osd.5 (osd.5) 66 : cluster 0 3.16 scrub ok 2024-03-11T20:53:19.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:19 smithi023 bash[21372]: cluster 2024-03-11T20:53:17.857481+0000 mgr.y (mgr.24380) 97 : cluster 0 pgmap v66: 132 pgs: 132 active+clean; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T20:53:19.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:19 smithi023 bash[21372]: cluster 2024-03-11T20:53:18.131965+0000 osd.0 (osd.0) 71 : cluster 0 3.17 scrub starts 2024-03-11T20:53:19.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:19 smithi023 bash[21372]: cluster 2024-03-11T20:53:18.135875+0000 osd.0 (osd.0) 72 : cluster 0 3.17 scrub ok 2024-03-11T20:53:19.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:19 smithi023 bash[21372]: cluster 2024-03-11T20:53:18.328996+0000 osd.7 (osd.7) 67 : cluster 0 6.8 scrub starts 2024-03-11T20:53:19.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:19 smithi023 bash[21372]: cluster 2024-03-11T20:53:18.331606+0000 osd.7 (osd.7) 68 : cluster 0 6.8 scrub ok 2024-03-11T20:53:19.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:19 smithi023 bash[21372]: audit 2024-03-11T20:53:19.113494+0000 mon.a (mon.0) 911 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:53:19.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:19 smithi023 bash[29146]: cluster 2024-03-11T20:53:17.780380+0000 osd.5 (osd.5) 65 : cluster 0 3.16 scrub starts 2024-03-11T20:53:19.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:19 smithi023 bash[29146]: cluster 2024-03-11T20:53:17.784022+0000 osd.5 (osd.5) 66 : cluster 0 3.16 scrub ok 2024-03-11T20:53:19.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:19 smithi023 bash[29146]: cluster 2024-03-11T20:53:17.857481+0000 mgr.y (mgr.24380) 97 : cluster 0 pgmap v66: 132 pgs: 132 active+clean; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T20:53:19.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:19 smithi023 bash[29146]: cluster 2024-03-11T20:53:18.131965+0000 osd.0 (osd.0) 71 : cluster 0 3.17 scrub starts 2024-03-11T20:53:19.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:19 smithi023 bash[29146]: cluster 2024-03-11T20:53:18.135875+0000 osd.0 (osd.0) 72 : cluster 0 3.17 scrub ok 2024-03-11T20:53:19.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:19 smithi023 bash[29146]: cluster 2024-03-11T20:53:18.328996+0000 osd.7 (osd.7) 67 : cluster 0 6.8 scrub starts 2024-03-11T20:53:19.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:19 smithi023 bash[29146]: cluster 2024-03-11T20:53:18.331606+0000 osd.7 (osd.7) 68 : cluster 0 6.8 scrub ok 2024-03-11T20:53:19.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:19 smithi023 bash[29146]: audit 2024-03-11T20:53:19.113494+0000 mon.a (mon.0) 911 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:53:20.456 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:20 smithi023 bash[21372]: cluster 2024-03-11T20:53:18.440857+0000 osd.3 (osd.3) 81 : cluster 0 4.19 scrub starts 2024-03-11T20:53:20.456 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:20 smithi023 bash[21372]: cluster 2024-03-11T20:53:18.443202+0000 osd.3 (osd.3) 82 : cluster 0 4.19 scrub ok 2024-03-11T20:53:20.457 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:20 smithi023 bash[21372]: cluster 2024-03-11T20:53:18.913779+0000 osd.6 (osd.6) 37 : cluster 0 2.17 scrub starts 2024-03-11T20:53:20.457 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:20 smithi023 bash[21372]: cluster 2024-03-11T20:53:18.916143+0000 osd.6 (osd.6) 38 : cluster 0 2.17 scrub ok 2024-03-11T20:53:20.457 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:20 smithi023 bash[29146]: cluster 2024-03-11T20:53:18.440857+0000 osd.3 (osd.3) 81 : cluster 0 4.19 scrub starts 2024-03-11T20:53:20.457 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:20 smithi023 bash[29146]: cluster 2024-03-11T20:53:18.443202+0000 osd.3 (osd.3) 82 : cluster 0 4.19 scrub ok 2024-03-11T20:53:20.457 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:20 smithi023 bash[29146]: cluster 2024-03-11T20:53:18.913779+0000 osd.6 (osd.6) 37 : cluster 0 2.17 scrub starts 2024-03-11T20:53:20.457 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:20 smithi023 bash[29146]: cluster 2024-03-11T20:53:18.916143+0000 osd.6 (osd.6) 38 : cluster 0 2.17 scrub ok 2024-03-11T20:53:20.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:20 smithi146 bash[23809]: cluster 2024-03-11T20:53:18.440857+0000 osd.3 (osd.3) 81 : cluster 0 4.19 scrub starts 2024-03-11T20:53:20.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:20 smithi146 bash[23809]: cluster 2024-03-11T20:53:18.443202+0000 osd.3 (osd.3) 82 : cluster 0 4.19 scrub ok 2024-03-11T20:53:20.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:20 smithi146 bash[23809]: cluster 2024-03-11T20:53:18.913779+0000 osd.6 (osd.6) 37 : cluster 0 2.17 scrub starts 2024-03-11T20:53:20.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:20 smithi146 bash[23809]: cluster 2024-03-11T20:53:18.916143+0000 osd.6 (osd.6) 38 : cluster 0 2.17 scrub ok 2024-03-11T20:53:21.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:21 smithi023 bash[21372]: cluster 2024-03-11T20:53:19.858859+0000 mgr.y (mgr.24380) 98 : cluster 0 pgmap v67: 132 pgs: 132 active+clean; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:53:21.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:21 smithi023 bash[29146]: cluster 2024-03-11T20:53:19.858859+0000 mgr.y (mgr.24380) 98 : cluster 0 pgmap v67: 132 pgs: 132 active+clean; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:53:21.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:21 smithi146 bash[23809]: cluster 2024-03-11T20:53:19.858859+0000 mgr.y (mgr.24380) 98 : cluster 0 pgmap v67: 132 pgs: 132 active+clean; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:53:22.058 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-11T20:53:22.064 INFO:teuthology.orchestra.run.smithi023.stderr:dumped all 2024-03-11T20:53:22.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:22 smithi146 bash[23809]: cluster 2024-03-11T20:53:20.943655+0000 osd.6 (osd.6) 39 : cluster 0 2.1a scrub starts 2024-03-11T20:53:22.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:22 smithi146 bash[23809]: cluster 2024-03-11T20:53:20.946217+0000 osd.6 (osd.6) 40 : cluster 0 2.1a scrub ok 2024-03-11T20:53:22.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:22 smithi146 bash[23809]: cluster 2024-03-11T20:53:21.160295+0000 osd.1 (osd.1) 63 : cluster 0 6.1d scrub starts 2024-03-11T20:53:22.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:22 smithi146 bash[23809]: cluster 2024-03-11T20:53:21.162643+0000 osd.1 (osd.1) 64 : cluster 0 6.1d scrub ok 2024-03-11T20:53:22.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:22 smithi146 bash[23809]: cluster 2024-03-11T20:53:21.436771+0000 osd.4 (osd.4) 63 : cluster 0 4.13 scrub starts 2024-03-11T20:53:22.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:22 smithi146 bash[23809]: cluster 2024-03-11T20:53:21.439108+0000 osd.4 (osd.4) 64 : cluster 0 4.13 scrub ok 2024-03-11T20:53:22.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:22 smithi023 bash[21372]: cluster 2024-03-11T20:53:20.943655+0000 osd.6 (osd.6) 39 : cluster 0 2.1a scrub starts 2024-03-11T20:53:22.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:22 smithi023 bash[21372]: cluster 2024-03-11T20:53:20.946217+0000 osd.6 (osd.6) 40 : cluster 0 2.1a scrub ok 2024-03-11T20:53:22.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:22 smithi023 bash[21372]: cluster 2024-03-11T20:53:21.160295+0000 osd.1 (osd.1) 63 : cluster 0 6.1d scrub starts 2024-03-11T20:53:22.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:22 smithi023 bash[21372]: cluster 2024-03-11T20:53:21.162643+0000 osd.1 (osd.1) 64 : cluster 0 6.1d scrub ok 2024-03-11T20:53:22.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:22 smithi023 bash[21372]: cluster 2024-03-11T20:53:21.436771+0000 osd.4 (osd.4) 63 : cluster 0 4.13 scrub starts 2024-03-11T20:53:22.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:22 smithi023 bash[21372]: cluster 2024-03-11T20:53:21.439108+0000 osd.4 (osd.4) 64 : cluster 0 4.13 scrub ok 2024-03-11T20:53:22.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:22 smithi023 bash[29146]: cluster 2024-03-11T20:53:20.943655+0000 osd.6 (osd.6) 39 : cluster 0 2.1a scrub starts 2024-03-11T20:53:22.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:22 smithi023 bash[29146]: cluster 2024-03-11T20:53:20.946217+0000 osd.6 (osd.6) 40 : cluster 0 2.1a scrub ok 2024-03-11T20:53:22.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:22 smithi023 bash[29146]: cluster 2024-03-11T20:53:21.160295+0000 osd.1 (osd.1) 63 : cluster 0 6.1d scrub starts 2024-03-11T20:53:22.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:22 smithi023 bash[29146]: cluster 2024-03-11T20:53:21.162643+0000 osd.1 (osd.1) 64 : cluster 0 6.1d scrub ok 2024-03-11T20:53:22.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:22 smithi023 bash[29146]: cluster 2024-03-11T20:53:21.436771+0000 osd.4 (osd.4) 63 : cluster 0 4.13 scrub starts 2024-03-11T20:53:22.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:22 smithi023 bash[29146]: cluster 2024-03-11T20:53:21.439108+0000 osd.4 (osd.4) 64 : cluster 0 4.13 scrub ok 2024-03-11T20:53:22.810 INFO:teuthology.orchestra.run.smithi023.stdout:{"pg_ready":true,"pg_map":{"version":68,"stamp":"2024-03-11T20:53:21.859050+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596392,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1098,"num_read_kb":944,"num_write":615,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":665,"ondisk_log_size":665,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":253440,"kb_used_data":8188,"kb_used_omap":12,"kb_used_meta":245171,"kb_avail":749707776,"statfs":{"total":767960285184,"available":767700762624,"internally_reserved":0,"allocated":8384512,"data_stored":5052992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":251055666},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.765391"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":86,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:11.425782+0000","last_change":"2024-03-11T20:53:05.510279+0000","last_active":"2024-03-11T20:53:11.425782+0000","last_peered":"2024-03-11T20:53:11.425782+0000","last_clean":"2024-03-11T20:53:11.425782+0000","last_became_active":"2024-03-11T20:50:34.338390+0000","last_became_peered":"2024-03-11T20:50:34.338390+0000","last_unstale":"2024-03-11T20:53:11.425782+0000","last_undegraded":"2024-03-11T20:53:11.425782+0000","last_fullsized":"2024-03-11T20:53:11.425782+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:05.510060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:53:05.510060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:21.840101+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":90,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:18.443361+0000","last_change":"2024-03-11T20:53:18.443416+0000","last_active":"2024-03-11T20:53:18.443361+0000","last_peered":"2024-03-11T20:53:18.443361+0000","last_clean":"2024-03-11T20:53:18.443361+0000","last_became_active":"2024-03-11T20:50:32.210194+0000","last_became_peered":"2024-03-11T20:50:32.210194+0000","last_unstale":"2024-03-11T20:53:18.443361+0000","last_undegraded":"2024-03-11T20:53:18.443361+0000","last_fullsized":"2024-03-11T20:53:18.443361+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:18.443215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:53:18.443215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:23.962891+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":89,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:11.979742+0000","last_change":"2024-03-11T20:52:12.816303+0000","last_active":"2024-03-11T20:53:11.979742+0000","last_peered":"2024-03-11T20:53:11.979742+0000","last_clean":"2024-03-11T20:53:11.979742+0000","last_became_active":"2024-03-11T20:50:28.061739+0000","last_became_peered":"2024-03-11T20:50:28.061739+0000","last_unstale":"2024-03-11T20:53:11.979742+0000","last_undegraded":"2024-03-11T20:53:11.979742+0000","last_fullsized":"2024-03-11T20:53:11.979742+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:12.816104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:52:12.816104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:24.698876+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"84'10","reported_seq":109,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:11.430029+0000","last_change":"2024-03-11T20:53:11.430067+0000","last_active":"2024-03-11T20:53:11.430029+0000","last_peered":"2024-03-11T20:53:11.430029+0000","last_clean":"2024-03-11T20:53:11.430029+0000","last_became_active":"2024-03-11T20:50:30.101721+0000","last_became_peered":"2024-03-11T20:50:30.101721+0000","last_unstale":"2024-03-11T20:53:11.430029+0000","last_undegraded":"2024-03-11T20:53:11.430029+0000","last_fullsized":"2024-03-11T20:53:11.430029+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'10","last_scrub_stamp":"2024-03-11T20:53:11.429944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:53:11.429944+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:29.275642+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":86,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:11.982920+0000","last_change":"2024-03-11T20:53:07.560827+0000","last_active":"2024-03-11T20:53:11.982920+0000","last_peered":"2024-03-11T20:53:11.982920+0000","last_clean":"2024-03-11T20:53:11.982920+0000","last_became_active":"2024-03-11T20:50:34.318627+0000","last_became_peered":"2024-03-11T20:50:34.318627+0000","last_unstale":"2024-03-11T20:53:11.982920+0000","last_undegraded":"2024-03-11T20:53:11.982920+0000","last_fullsized":"2024-03-11T20:53:11.982920+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:07.560731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:55.545758+0000","last_clean_scrub_stamp":"2024-03-11T20:53:07.560731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:36.783905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":90,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:11.982731+0000","last_change":"2024-03-11T20:52:49.461204+0000","last_active":"2024-03-11T20:53:11.982731+0000","last_peered":"2024-03-11T20:53:11.982731+0000","last_clean":"2024-03-11T20:53:11.982731+0000","last_became_active":"2024-03-11T20:50:32.197413+0000","last_became_peered":"2024-03-11T20:50:32.197413+0000","last_unstale":"2024-03-11T20:53:11.982731+0000","last_undegraded":"2024-03-11T20:53:11.982731+0000","last_fullsized":"2024-03-11T20:53:11.982731+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:49.460993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:52:49.460993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:04.898406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":90,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:11.427866+0000","last_change":"2024-03-11T20:52:26.396960+0000","last_active":"2024-03-11T20:53:11.427866+0000","last_peered":"2024-03-11T20:53:11.427866+0000","last_clean":"2024-03-11T20:53:11.427866+0000","last_became_active":"2024-03-11T20:50:28.058274+0000","last_became_peered":"2024-03-11T20:50:28.058274+0000","last_unstale":"2024-03-11T20:53:11.427866+0000","last_undegraded":"2024-03-11T20:53:11.427866+0000","last_fullsized":"2024-03-11T20:53:11.427866+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:26.396865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:52:26.396865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:52.964932+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"84'11","reported_seq":104,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:11.979682+0000","last_change":"2024-03-11T20:52:15.868867+0000","last_active":"2024-03-11T20:53:11.979682+0000","last_peered":"2024-03-11T20:53:11.979682+0000","last_clean":"2024-03-11T20:53:11.979682+0000","last_became_active":"2024-03-11T20:50:30.092638+0000","last_became_peered":"2024-03-11T20:50:30.092638+0000","last_unstale":"2024-03-11T20:53:11.979682+0000","last_undegraded":"2024-03-11T20:53:11.979682+0000","last_fullsized":"2024-03-11T20:53:11.979682+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T20:52:15.868599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:52:15.868599+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:44.240121+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":86,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:11.837750+0000","last_change":"2024-03-11T20:53:07.792944+0000","last_active":"2024-03-11T20:53:11.837750+0000","last_peered":"2024-03-11T20:53:11.837750+0000","last_clean":"2024-03-11T20:53:11.837750+0000","last_became_active":"2024-03-11T20:50:34.334495+0000","last_became_peered":"2024-03-11T20:50:34.334495+0000","last_unstale":"2024-03-11T20:53:11.837750+0000","last_undegraded":"2024-03-11T20:53:11.837750+0000","last_fullsized":"2024-03-11T20:53:11.837750+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:07.792697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:53:07.792697+0000","last_clean_scrub_stamp":"2024-03-11T20:53:07.792697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:23.012728+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":82,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:11.983264+0000","last_change":"2024-03-11T20:52:03.583828+0000","last_active":"2024-03-11T20:53:11.983264+0000","last_peered":"2024-03-11T20:53:11.983264+0000","last_clean":"2024-03-11T20:53:11.983264+0000","last_became_active":"2024-03-11T20:50:32.199561+0000","last_became_peered":"2024-03-11T20:50:32.199561+0000","last_unstale":"2024-03-11T20:53:11.983264+0000","last_undegraded":"2024-03-11T20:53:11.983264+0000","last_fullsized":"2024-03-11T20:53:11.983264+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:03.583594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:52:03.583594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:22.995130+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":90,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:11.101297+0000","last_change":"2024-03-11T20:52:06.012514+0000","last_active":"2024-03-11T20:53:11.101297+0000","last_peered":"2024-03-11T20:53:11.101297+0000","last_clean":"2024-03-11T20:53:11.101297+0000","last_became_active":"2024-03-11T20:50:28.079225+0000","last_became_peered":"2024-03-11T20:50:28.079225+0000","last_unstale":"2024-03-11T20:53:11.101297+0000","last_undegraded":"2024-03-11T20:53:11.101297+0000","last_fullsized":"2024-03-11T20:53:11.101297+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:06.012226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:52:06.012226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:18.639749+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"84'15","reported_seq":111,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:11.839410+0000","last_change":"2024-03-11T20:51:50.842138+0000","last_active":"2024-03-11T20:53:11.839410+0000","last_peered":"2024-03-11T20:53:11.839410+0000","last_clean":"2024-03-11T20:53:11.839410+0000","last_became_active":"2024-03-11T20:50:30.105354+0000","last_became_peered":"2024-03-11T20:50:30.105354+0000","last_unstale":"2024-03-11T20:53:11.839410+0000","last_undegraded":"2024-03-11T20:53:11.839410+0000","last_fullsized":"2024-03-11T20:53:11.839410+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15","last_scrub_stamp":"2024-03-11T20:51:50.841902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:51:50.841902+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:16.306055+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":85,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:14.168464+0000","last_change":"2024-03-11T20:53:14.168525+0000","last_active":"2024-03-11T20:53:14.168464+0000","last_peered":"2024-03-11T20:53:14.168464+0000","last_clean":"2024-03-11T20:53:14.168464+0000","last_became_active":"2024-03-11T20:50:34.333230+0000","last_became_peered":"2024-03-11T20:50:34.333230+0000","last_unstale":"2024-03-11T20:53:14.168464+0000","last_undegraded":"2024-03-11T20:53:14.168464+0000","last_fullsized":"2024-03-11T20:53:14.168464+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:14.168304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:53:14.168304+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:19.290761+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":82,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:11.983569+0000","last_change":"2024-03-11T20:52:19.502063+0000","last_active":"2024-03-11T20:53:11.983569+0000","last_peered":"2024-03-11T20:53:11.983569+0000","last_clean":"2024-03-11T20:53:11.983569+0000","last_became_active":"2024-03-11T20:50:32.200126+0000","last_became_peered":"2024-03-11T20:50:32.200126+0000","last_unstale":"2024-03-11T20:53:11.983569+0000","last_undegraded":"2024-03-11T20:53:11.983569+0000","last_fullsized":"2024-03-11T20:53:11.983569+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:19.501832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:52:19.501832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:26.456710+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":98,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:11.101150+0000","last_change":"2024-03-11T20:53:00.210016+0000","last_active":"2024-03-11T20:53:11.101150+0000","last_peered":"2024-03-11T20:53:11.101150+0000","last_clean":"2024-03-11T20:53:11.101150+0000","last_became_active":"2024-03-11T20:50:28.068978+0000","last_became_peered":"2024-03-11T20:50:28.068978+0000","last_unstale":"2024-03-11T20:53:11.101150+0000","last_undegraded":"2024-03-11T20:53:11.101150+0000","last_fullsized":"2024-03-11T20:53:11.101150+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:00.209791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:53:00.209791+0000","last_clean_scrub_stamp":"2024-03-11T20:53:00.209791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:01.669184+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"110'17","reported_seq":122,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:11.838842+0000","last_change":"2024-03-11T20:53:04.802853+0000","last_active":"2024-03-11T20:53:11.838842+0000","last_peered":"2024-03-11T20:53:11.838842+0000","last_clean":"2024-03-11T20:53:11.838842+0000","last_became_active":"2024-03-11T20:50:30.108364+0000","last_became_peered":"2024-03-11T20:50:30.108364+0000","last_unstale":"2024-03-11T20:53:11.838842+0000","last_undegraded":"2024-03-11T20:53:11.838842+0000","last_fullsized":"2024-03-11T20:53:11.838842+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'17","last_scrub_stamp":"2024-03-11T20:53:04.802488+0000","last_deep_scrub":"110'17","last_deep_scrub_stamp":"2024-03-11T20:53:04.802488+0000","last_clean_scrub_stamp":"2024-03-11T20:53:04.802488+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:26.988743+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":78,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:11.428502+0000","last_change":"2024-03-11T20:52:37.430006+0000","last_active":"2024-03-11T20:53:11.428502+0000","last_peered":"2024-03-11T20:53:11.428502+0000","last_clean":"2024-03-11T20:53:11.428502+0000","last_became_active":"2024-03-11T20:50:34.334558+0000","last_became_peered":"2024-03-11T20:50:34.334558+0000","last_unstale":"2024-03-11T20:53:11.428502+0000","last_undegraded":"2024-03-11T20:53:11.428502+0000","last_fullsized":"2024-03-11T20:53:11.428502+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:37.429729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:52:37.429729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:37.737857+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":90,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:11.427341+0000","last_change":"2024-03-11T20:52:59.455887+0000","last_active":"2024-03-11T20:53:11.427341+0000","last_peered":"2024-03-11T20:53:11.427341+0000","last_clean":"2024-03-11T20:53:11.427341+0000","last_became_active":"2024-03-11T20:50:32.202533+0000","last_became_peered":"2024-03-11T20:50:32.202533+0000","last_unstale":"2024-03-11T20:53:11.427341+0000","last_undegraded":"2024-03-11T20:53:11.427341+0000","last_fullsized":"2024-03-11T20:53:11.427341+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:59.455677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:52:59.455677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:29.453221+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"77'1","reported_seq":98,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:11.426519+0000","last_change":"2024-03-11T20:51:59.384746+0000","last_active":"2024-03-11T20:53:11.426519+0000","last_peered":"2024-03-11T20:53:11.426519+0000","last_clean":"2024-03-11T20:53:11.426519+0000","last_became_active":"2024-03-11T20:50:28.068840+0000","last_became_peered":"2024-03-11T20:50:28.068840+0000","last_unstale":"2024-03-11T20:53:11.426519+0000","last_undegraded":"2024-03-11T20:53:11.426519+0000","last_fullsized":"2024-03-11T20:53:11.426519+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-03-11T20:51:59.384636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:51:59.384636+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:26.361222+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"84'9","reported_seq":102,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:11.983011+0000","last_change":"2024-03-11T20:51:50.570896+0000","last_active":"2024-03-11T20:53:11.983011+0000","last_peered":"2024-03-11T20:53:11.983011+0000","last_clean":"2024-03-11T20:53:11.983011+0000","last_became_active":"2024-03-11T20:50:30.086169+0000","last_became_peered":"2024-03-11T20:50:30.086169+0000","last_unstale":"2024-03-11T20:53:11.983011+0000","last_undegraded":"2024-03-11T20:53:11.983011+0000","last_fullsized":"2024-03-11T20:53:11.983011+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'9","last_scrub_stamp":"2024-03-11T20:51:50.570657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:51:50.570657+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:15.854580+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":78,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:11.984804+0000","last_change":"2024-03-11T20:52:07.500890+0000","last_active":"2024-03-11T20:53:11.984804+0000","last_peered":"2024-03-11T20:53:11.984804+0000","last_clean":"2024-03-11T20:53:11.984804+0000","last_became_active":"2024-03-11T20:50:34.321714+0000","last_became_peered":"2024-03-11T20:50:34.321714+0000","last_unstale":"2024-03-11T20:53:11.984804+0000","last_undegraded":"2024-03-11T20:53:11.984804+0000","last_fullsized":"2024-03-11T20:53:11.984804+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:07.500759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:52:07.500759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":90,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:15.798598+0000","last_change":"2024-03-11T20:53:15.798662+0000","last_active":"2024-03-11T20:53:15.798598+0000","last_peered":"2024-03-11T20:53:15.798598+0000","last_clean":"2024-03-11T20:53:15.798598+0000","last_became_active":"2024-03-11T20:50:32.192704+0000","last_became_peered":"2024-03-11T20:50:32.192704+0000","last_unstale":"2024-03-11T20:53:15.798598+0000","last_undegraded":"2024-03-11T20:53:15.798598+0000","last_fullsized":"2024-03-11T20:53:15.798598+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:15.798438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:53:15.798438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:29.061321+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":90,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:11.433225+0000","last_change":"2024-03-11T20:51:59.680978+0000","last_active":"2024-03-11T20:53:11.433225+0000","last_peered":"2024-03-11T20:53:11.433225+0000","last_clean":"2024-03-11T20:53:11.433225+0000","last_became_active":"2024-03-11T20:50:28.081538+0000","last_became_peered":"2024-03-11T20:50:28.081538+0000","last_unstale":"2024-03-11T20:53:11.433225+0000","last_undegraded":"2024-03-11T20:53:11.433225+0000","last_fullsized":"2024-03-11T20:53:11.433225+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:59.680629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:51:59.680629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"84'5","reported_seq":95,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:11.979194+0000","last_change":"2024-03-11T20:52:00.809218+0000","last_active":"2024-03-11T20:53:11.979194+0000","last_peered":"2024-03-11T20:53:11.979194+0000","last_clean":"2024-03-11T20:53:11.979194+0000","last_became_active":"2024-03-11T20:50:30.099866+0000","last_became_peered":"2024-03-11T20:50:30.099866+0000","last_unstale":"2024-03-11T20:53:11.979194+0000","last_undegraded":"2024-03-11T20:53:11.979194+0000","last_fullsized":"2024-03-11T20:53:11.979194+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'5","last_scrub_stamp":"2024-03-11T20:52:00.808991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:52:00.808991+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:29.806692+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":78,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:11.096517+0000","last_change":"2024-03-11T20:52:02.142598+0000","last_active":"2024-03-11T20:53:11.096517+0000","last_peered":"2024-03-11T20:53:11.096517+0000","last_clean":"2024-03-11T20:53:11.096517+0000","last_became_active":"2024-03-11T20:50:34.307730+0000","last_became_peered":"2024-03-11T20:50:34.307730+0000","last_unstale":"2024-03-11T20:53:11.096517+0000","last_undegraded":"2024-03-11T20:53:11.096517+0000","last_fullsized":"2024-03-11T20:53:11.096517+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:02.142361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:52:02.142361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1f","version":"84'11","reported_seq":133,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:13.268304+0000","last_change":"2024-03-11T20:52:12.655265+0000","last_active":"2024-03-11T20:53:13.268304+0000","last_peered":"2024-03-11T20:53:13.268304+0000","last_clean":"2024-03-11T20:53:13.268304+0000","last_became_active":"2024-03-11T20:50:32.200195+0000","last_became_peered":"2024-03-11T20:50:32.200195+0000","last_unstale":"2024-03-11T20:53:13.268304+0000","last_undegraded":"2024-03-11T20:53:13.268304+0000","last_fullsized":"2024-03-11T20:53:13.268304+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T20:52:12.655061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:52:12.655061+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:22.459298+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"77'1","reported_seq":91,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:11.425805+0000","last_change":"2024-03-11T20:52:24.397205+0000","last_active":"2024-03-11T20:53:11.425805+0000","last_peered":"2024-03-11T20:53:11.425805+0000","last_clean":"2024-03-11T20:53:11.425805+0000","last_became_active":"2024-03-11T20:50:28.068827+0000","last_became_peered":"2024-03-11T20:50:28.068827+0000","last_unstale":"2024-03-11T20:53:11.425805+0000","last_undegraded":"2024-03-11T20:53:11.425805+0000","last_fullsized":"2024-03-11T20:53:11.425805+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-03-11T20:52:24.396881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:52:24.396881+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:53.894381+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"84'9","reported_seq":102,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:11.425488+0000","last_change":"2024-03-11T20:52:05.438999+0000","last_active":"2024-03-11T20:53:11.425488+0000","last_peered":"2024-03-11T20:53:11.425488+0000","last_clean":"2024-03-11T20:53:11.425488+0000","last_became_active":"2024-03-11T20:50:30.083936+0000","last_became_peered":"2024-03-11T20:50:30.083936+0000","last_unstale":"2024-03-11T20:53:11.425488+0000","last_undegraded":"2024-03-11T20:53:11.425488+0000","last_fullsized":"2024-03-11T20:53:11.425488+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'9","last_scrub_stamp":"2024-03-11T20:52:05.438658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:52:05.438658+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:19.765612+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"105'2","reported_seq":81,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:11.100820+0000","last_change":"2024-03-11T20:52:18.099224+0000","last_active":"2024-03-11T20:53:11.100820+0000","last_peered":"2024-03-11T20:53:11.100820+0000","last_clean":"2024-03-11T20:53:11.100820+0000","last_became_active":"2024-03-11T20:50:34.327356+0000","last_became_peered":"2024-03-11T20:50:34.327356+0000","last_unstale":"2024-03-11T20:53:11.100820+0000","last_undegraded":"2024-03-11T20:53:11.100820+0000","last_fullsized":"2024-03-11T20:53:11.100820+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'2","last_scrub_stamp":"2024-03-11T20:52:18.098911+0000","last_deep_scrub":"105'2","last_deep_scrub_stamp":"2024-03-11T20:52:18.098911+0000","last_clean_scrub_stamp":"2024-03-11T20:52:18.098911+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:21.277529+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":89,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:11.979767+0000","last_change":"2024-03-11T20:53:10.062579+0000","last_active":"2024-03-11T20:53:11.979767+0000","last_peered":"2024-03-11T20:53:11.979767+0000","last_clean":"2024-03-11T20:53:11.979767+0000","last_became_active":"2024-03-11T20:50:32.648869+0000","last_became_peered":"2024-03-11T20:50:32.648869+0000","last_unstale":"2024-03-11T20:53:11.979767+0000","last_undegraded":"2024-03-11T20:53:11.979767+0000","last_fullsized":"2024-03-11T20:53:11.979767+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:10.062346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:53:10.062346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:39.641031+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":90,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:11.837581+0000","last_change":"2024-03-11T20:52:01.880353+0000","last_active":"2024-03-11T20:53:11.837581+0000","last_peered":"2024-03-11T20:53:11.837581+0000","last_clean":"2024-03-11T20:53:11.837581+0000","last_became_active":"2024-03-11T20:50:28.066124+0000","last_became_peered":"2024-03-11T20:50:28.066124+0000","last_unstale":"2024-03-11T20:53:11.837581+0000","last_undegraded":"2024-03-11T20:53:11.837581+0000","last_fullsized":"2024-03-11T20:53:11.837581+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:01.880052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:52:01.880052+0000","last_clean_scrub_stamp":"2024-03-11T20:52:01.880052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:22.449810+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"84'15","reported_seq":119,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:13.139120+0000","last_change":"2024-03-11T20:53:13.139250+0000","last_active":"2024-03-11T20:53:13.139120+0000","last_peered":"2024-03-11T20:53:13.139120+0000","last_clean":"2024-03-11T20:53:13.139120+0000","last_became_active":"2024-03-11T20:50:30.088446+0000","last_became_peered":"2024-03-11T20:50:30.088446+0000","last_unstale":"2024-03-11T20:53:13.139120+0000","last_undegraded":"2024-03-11T20:53:13.139120+0000","last_fullsized":"2024-03-11T20:53:13.139120+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15","last_scrub_stamp":"2024-03-11T20:53:13.138917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:53:13.138917+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:28.013588+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":78,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:11.427579+0000","last_change":"2024-03-11T20:52:11.462704+0000","last_active":"2024-03-11T20:53:11.427579+0000","last_peered":"2024-03-11T20:53:11.427579+0000","last_clean":"2024-03-11T20:53:11.427579+0000","last_became_active":"2024-03-11T20:50:34.337878+0000","last_became_peered":"2024-03-11T20:50:34.337878+0000","last_unstale":"2024-03-11T20:53:11.427579+0000","last_undegraded":"2024-03-11T20:53:11.427579+0000","last_fullsized":"2024-03-11T20:53:11.427579+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:11.462480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:52:11.462480+0000","last_clean_scrub_stamp":"2024-03-11T20:52:11.462480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:22.958223+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":90,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:11.426539+0000","last_change":"2024-03-11T20:53:02.400889+0000","last_active":"2024-03-11T20:53:11.426539+0000","last_peered":"2024-03-11T20:53:11.426539+0000","last_clean":"2024-03-11T20:53:11.426539+0000","last_became_active":"2024-03-11T20:50:32.646323+0000","last_became_peered":"2024-03-11T20:50:32.646323+0000","last_unstale":"2024-03-11T20:53:11.426539+0000","last_undegraded":"2024-03-11T20:53:11.426539+0000","last_fullsized":"2024-03-11T20:53:11.426539+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:02.400685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:53:02.400685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:04.399694+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":98,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:18.916303+0000","last_change":"2024-03-11T20:53:18.916363+0000","last_active":"2024-03-11T20:53:18.916303+0000","last_peered":"2024-03-11T20:53:18.916303+0000","last_clean":"2024-03-11T20:53:18.916303+0000","last_became_active":"2024-03-11T20:50:28.086513+0000","last_became_peered":"2024-03-11T20:50:28.086513+0000","last_unstale":"2024-03-11T20:53:18.916303+0000","last_undegraded":"2024-03-11T20:53:18.916303+0000","last_fullsized":"2024-03-11T20:53:18.916303+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:18.916152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:53:18.916152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:20.894545+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"84'9","reported_seq":110,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:17.784185+0000","last_change":"2024-03-11T20:53:17.784263+0000","last_active":"2024-03-11T20:53:17.784185+0000","last_peered":"2024-03-11T20:53:17.784185+0000","last_clean":"2024-03-11T20:53:17.784185+0000","last_became_active":"2024-03-11T20:50:30.109869+0000","last_became_peered":"2024-03-11T20:50:30.109869+0000","last_unstale":"2024-03-11T20:53:17.784185+0000","last_undegraded":"2024-03-11T20:53:17.784185+0000","last_fullsized":"2024-03-11T20:53:17.784185+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'9","last_scrub_stamp":"2024-03-11T20:53:17.784035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:53:17.784035+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:30.540981+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":86,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:11.099835+0000","last_change":"2024-03-11T20:53:04.321530+0000","last_active":"2024-03-11T20:53:11.099835+0000","last_peered":"2024-03-11T20:53:11.099835+0000","last_clean":"2024-03-11T20:53:11.099835+0000","last_became_active":"2024-03-11T20:50:34.321443+0000","last_became_peered":"2024-03-11T20:50:34.321443+0000","last_unstale":"2024-03-11T20:53:11.099835+0000","last_undegraded":"2024-03-11T20:53:11.099835+0000","last_fullsized":"2024-03-11T20:53:11.099835+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:04.321235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:53:04.321235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:23.221472+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":82,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:11.425488+0000","last_change":"2024-03-11T20:52:29.373009+0000","last_active":"2024-03-11T20:53:11.425488+0000","last_peered":"2024-03-11T20:53:11.425488+0000","last_clean":"2024-03-11T20:53:11.425488+0000","last_became_active":"2024-03-11T20:50:32.211995+0000","last_became_peered":"2024-03-11T20:50:32.211995+0000","last_unstale":"2024-03-11T20:53:11.425488+0000","last_undegraded":"2024-03-11T20:53:11.425488+0000","last_fullsized":"2024-03-11T20:53:11.425488+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:29.372897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:52:29.372897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:32.682617+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":98,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:11.838656+0000","last_change":"2024-03-11T20:52:53.806361+0000","last_active":"2024-03-11T20:53:11.838656+0000","last_peered":"2024-03-11T20:53:11.838656+0000","last_clean":"2024-03-11T20:53:11.838656+0000","last_became_active":"2024-03-11T20:50:28.075760+0000","last_became_peered":"2024-03-11T20:50:28.075760+0000","last_unstale":"2024-03-11T20:53:11.838656+0000","last_undegraded":"2024-03-11T20:53:11.838656+0000","last_fullsized":"2024-03-11T20:53:11.838656+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:53.806149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:52:53.806149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:15.472221+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"84'6","reported_seq":102,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:18.136051+0000","last_change":"2024-03-11T20:53:18.136127+0000","last_active":"2024-03-11T20:53:18.136051+0000","last_peered":"2024-03-11T20:53:18.136051+0000","last_clean":"2024-03-11T20:53:18.136051+0000","last_became_active":"2024-03-11T20:50:30.092991+0000","last_became_peered":"2024-03-11T20:50:30.092991+0000","last_unstale":"2024-03-11T20:53:18.136051+0000","last_undegraded":"2024-03-11T20:53:18.136051+0000","last_fullsized":"2024-03-11T20:53:18.136051+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6","last_scrub_stamp":"2024-03-11T20:53:18.135889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:53:18.135889+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:46.013838+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.11","version":"84'1","reported_seq":79,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:11.425228+0000","last_change":"2024-03-11T20:52:42.470691+0000","last_active":"2024-03-11T20:53:11.425228+0000","last_peered":"2024-03-11T20:53:11.425228+0000","last_clean":"2024-03-11T20:53:11.425228+0000","last_became_active":"2024-03-11T20:50:34.339326+0000","last_became_peered":"2024-03-11T20:50:34.339326+0000","last_unstale":"2024-03-11T20:53:11.425228+0000","last_undegraded":"2024-03-11T20:53:11.425228+0000","last_fullsized":"2024-03-11T20:53:11.425228+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'1","last_scrub_stamp":"2024-03-11T20:52:42.470380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:52:42.470380+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:08.720300+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":82,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:11.983277+0000","last_change":"2024-03-11T20:51:46.599809+0000","last_active":"2024-03-11T20:53:11.983277+0000","last_peered":"2024-03-11T20:53:11.983277+0000","last_clean":"2024-03-11T20:53:11.983277+0000","last_became_active":"2024-03-11T20:50:32.211379+0000","last_became_peered":"2024-03-11T20:50:32.211379+0000","last_unstale":"2024-03-11T20:53:11.983277+0000","last_undegraded":"2024-03-11T20:53:11.983277+0000","last_fullsized":"2024-03-11T20:53:11.983277+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:46.599566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:51:46.599566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":98,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:14.109914+0000","last_change":"2024-03-11T20:53:14.109971+0000","last_active":"2024-03-11T20:53:14.109914+0000","last_peered":"2024-03-11T20:53:14.109914+0000","last_clean":"2024-03-11T20:53:14.109914+0000","last_became_active":"2024-03-11T20:50:28.049937+0000","last_became_peered":"2024-03-11T20:50:28.049937+0000","last_unstale":"2024-03-11T20:53:14.109914+0000","last_undegraded":"2024-03-11T20:53:14.109914+0000","last_fullsized":"2024-03-11T20:53:14.109914+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:14.109743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:53:14.109743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:31.769637+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"84'10","reported_seq":109,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:11.983802+0000","last_change":"2024-03-11T20:52:51.437032+0000","last_active":"2024-03-11T20:53:11.983802+0000","last_peered":"2024-03-11T20:53:11.983802+0000","last_clean":"2024-03-11T20:53:11.983802+0000","last_became_active":"2024-03-11T20:50:30.105752+0000","last_became_peered":"2024-03-11T20:50:30.105752+0000","last_unstale":"2024-03-11T20:53:11.983802+0000","last_undegraded":"2024-03-11T20:53:11.983802+0000","last_fullsized":"2024-03-11T20:53:11.983802+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'10","last_scrub_stamp":"2024-03-11T20:52:51.436752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:52:51.436752+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:15.812213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":77,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:11.978704+0000","last_change":"2024-03-11T20:52:03.919036+0000","last_active":"2024-03-11T20:53:11.978704+0000","last_peered":"2024-03-11T20:53:11.978704+0000","last_clean":"2024-03-11T20:53:11.978704+0000","last_became_active":"2024-03-11T20:50:34.332465+0000","last_became_peered":"2024-03-11T20:50:34.332465+0000","last_unstale":"2024-03-11T20:53:11.978704+0000","last_undegraded":"2024-03-11T20:53:11.978704+0000","last_fullsized":"2024-03-11T20:53:11.978704+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:03.918724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:52:03.918724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:19.041856+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":82,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:11.096879+0000","last_change":"2024-03-11T20:51:59.129831+0000","last_active":"2024-03-11T20:53:11.096879+0000","last_peered":"2024-03-11T20:53:11.096879+0000","last_clean":"2024-03-11T20:53:11.096879+0000","last_became_active":"2024-03-11T20:50:32.194792+0000","last_became_peered":"2024-03-11T20:50:32.194792+0000","last_unstale":"2024-03-11T20:53:11.096879+0000","last_undegraded":"2024-03-11T20:53:11.096879+0000","last_fullsized":"2024-03-11T20:53:11.096879+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:59.129712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:51:59.129712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:15.832569+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":98,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:11.433604+0000","last_change":"2024-03-11T20:52:54.775523+0000","last_active":"2024-03-11T20:53:11.433604+0000","last_peered":"2024-03-11T20:53:11.433604+0000","last_clean":"2024-03-11T20:53:11.433604+0000","last_became_active":"2024-03-11T20:50:28.082737+0000","last_became_peered":"2024-03-11T20:50:28.082737+0000","last_unstale":"2024-03-11T20:53:11.433604+0000","last_undegraded":"2024-03-11T20:53:11.433604+0000","last_fullsized":"2024-03-11T20:53:11.433604+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:54.775303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:52:54.775303+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:11.643155+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"84'9","reported_seq":110,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:11.101585+0000","last_change":"2024-03-11T20:52:57.247973+0000","last_active":"2024-03-11T20:53:11.101585+0000","last_peered":"2024-03-11T20:53:11.101585+0000","last_clean":"2024-03-11T20:53:11.101585+0000","last_became_active":"2024-03-11T20:50:30.085418+0000","last_became_peered":"2024-03-11T20:50:30.085418+0000","last_unstale":"2024-03-11T20:53:11.101585+0000","last_undegraded":"2024-03-11T20:53:11.101585+0000","last_fullsized":"2024-03-11T20:53:11.101585+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'9","last_scrub_stamp":"2024-03-11T20:52:57.247711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:52:57.247711+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:09.030171+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":86,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:14.601530+0000","last_change":"2024-03-11T20:53:14.601565+0000","last_active":"2024-03-11T20:53:14.601530+0000","last_peered":"2024-03-11T20:53:14.601530+0000","last_clean":"2024-03-11T20:53:14.601530+0000","last_became_active":"2024-03-11T20:50:34.328449+0000","last_became_peered":"2024-03-11T20:50:34.328449+0000","last_unstale":"2024-03-11T20:53:14.601530+0000","last_undegraded":"2024-03-11T20:53:14.601530+0000","last_fullsized":"2024-03-11T20:53:14.601530+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:14.601439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:53:14.601439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:19.740273+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.15","version":"112'12","reported_seq":136,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:13.268488+0000","last_change":"2024-03-11T20:53:11.844176+0000","last_active":"2024-03-11T20:53:13.268488+0000","last_peered":"2024-03-11T20:53:13.268488+0000","last_clean":"2024-03-11T20:53:13.268488+0000","last_became_active":"2024-03-11T20:50:32.650064+0000","last_became_peered":"2024-03-11T20:50:32.650064+0000","last_unstale":"2024-03-11T20:53:13.268488+0000","last_undegraded":"2024-03-11T20:53:13.268488+0000","last_fullsized":"2024-03-11T20:53:13.268488+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'12","last_scrub_stamp":"2024-03-11T20:53:11.843951+0000","last_deep_scrub":"112'12","last_deep_scrub_stamp":"2024-03-11T20:53:11.843951+0000","last_clean_scrub_stamp":"2024-03-11T20:53:11.843951+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:40.680380+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":97,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:11.979024+0000","last_change":"2024-03-11T20:52:47.933545+0000","last_active":"2024-03-11T20:53:11.979024+0000","last_peered":"2024-03-11T20:53:11.979024+0000","last_clean":"2024-03-11T20:53:11.979024+0000","last_became_active":"2024-03-11T20:50:28.044321+0000","last_became_peered":"2024-03-11T20:50:28.044321+0000","last_unstale":"2024-03-11T20:53:11.979024+0000","last_undegraded":"2024-03-11T20:53:11.979024+0000","last_fullsized":"2024-03-11T20:53:11.979024+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:47.933320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:52:47.933320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:55.580970+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"84'9","reported_seq":109,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:11.979130+0000","last_change":"2024-03-11T20:52:55.993159+0000","last_active":"2024-03-11T20:53:11.979130+0000","last_peered":"2024-03-11T20:53:11.979130+0000","last_clean":"2024-03-11T20:53:11.979130+0000","last_became_active":"2024-03-11T20:50:30.099856+0000","last_became_peered":"2024-03-11T20:50:30.099856+0000","last_unstale":"2024-03-11T20:53:11.979130+0000","last_undegraded":"2024-03-11T20:53:11.979130+0000","last_fullsized":"2024-03-11T20:53:11.979130+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'9","last_scrub_stamp":"2024-03-11T20:52:55.992803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:52:55.992803+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:14.843849+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":77,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:11.979513+0000","last_change":"2024-03-11T20:52:16.846357+0000","last_active":"2024-03-11T20:53:11.979513+0000","last_peered":"2024-03-11T20:53:11.979513+0000","last_clean":"2024-03-11T20:53:11.979513+0000","last_became_active":"2024-03-11T20:50:34.330351+0000","last_became_peered":"2024-03-11T20:50:34.330351+0000","last_unstale":"2024-03-11T20:53:11.979513+0000","last_undegraded":"2024-03-11T20:53:11.979513+0000","last_fullsized":"2024-03-11T20:53:11.979513+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:16.846238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:52:16.846238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:38.667608+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"110'13","reported_seq":146,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:13.264064+0000","last_change":"2024-03-11T20:53:00.434710+0000","last_active":"2024-03-11T20:53:13.264064+0000","last_peered":"2024-03-11T20:53:13.264064+0000","last_clean":"2024-03-11T20:53:13.264064+0000","last_became_active":"2024-03-11T20:51:22.669419+0000","last_became_peered":"2024-03-11T20:51:22.669419+0000","last_unstale":"2024-03-11T20:53:13.264064+0000","last_undegraded":"2024-03-11T20:53:13.264064+0000","last_fullsized":"2024-03-11T20:53:13.264064+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'13","last_scrub_stamp":"2024-03-11T20:53:00.434532+0000","last_deep_scrub":"110'13","last_deep_scrub_stamp":"2024-03-11T20:53:00.434532+0000","last_clean_scrub_stamp":"2024-03-11T20:53:00.434532+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:25.979437+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":98,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:11.839009+0000","last_change":"2024-03-11T20:52:59.801478+0000","last_active":"2024-03-11T20:53:11.839009+0000","last_peered":"2024-03-11T20:53:11.839009+0000","last_clean":"2024-03-11T20:53:11.839009+0000","last_became_active":"2024-03-11T20:50:28.070325+0000","last_became_peered":"2024-03-11T20:50:28.070325+0000","last_unstale":"2024-03-11T20:53:11.839009+0000","last_undegraded":"2024-03-11T20:53:11.839009+0000","last_fullsized":"2024-03-11T20:53:11.839009+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:59.801249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:52:59.801249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:29.742165+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"84'11","reported_seq":105,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:11.100527+0000","last_change":"2024-03-11T20:52:12.020706+0000","last_active":"2024-03-11T20:53:11.100527+0000","last_peered":"2024-03-11T20:53:11.100527+0000","last_clean":"2024-03-11T20:53:11.100527+0000","last_became_active":"2024-03-11T20:50:30.087154+0000","last_became_peered":"2024-03-11T20:50:30.087154+0000","last_unstale":"2024-03-11T20:53:11.100527+0000","last_undegraded":"2024-03-11T20:53:11.100527+0000","last_fullsized":"2024-03-11T20:53:11.100527+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T20:52:12.020466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:52:12.020466+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:29.103467+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":78,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:11.100718+0000","last_change":"2024-03-11T20:52:22.074863+0000","last_active":"2024-03-11T20:53:11.100718+0000","last_peered":"2024-03-11T20:53:11.100718+0000","last_clean":"2024-03-11T20:53:11.100718+0000","last_became_active":"2024-03-11T20:50:34.318231+0000","last_became_peered":"2024-03-11T20:50:34.318231+0000","last_unstale":"2024-03-11T20:53:11.100718+0000","last_undegraded":"2024-03-11T20:53:11.100718+0000","last_fullsized":"2024-03-11T20:53:11.100718+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:22.074605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:52:22.074605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:39.515449+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":90,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:11.428699+0000","last_change":"2024-03-11T20:53:06.534811+0000","last_active":"2024-03-11T20:53:11.428699+0000","last_peered":"2024-03-11T20:53:11.428699+0000","last_clean":"2024-03-11T20:53:11.428699+0000","last_became_active":"2024-03-11T20:50:32.651361+0000","last_became_peered":"2024-03-11T20:50:32.651361+0000","last_unstale":"2024-03-11T20:53:11.428699+0000","last_undegraded":"2024-03-11T20:53:11.428699+0000","last_fullsized":"2024-03-11T20:53:11.428699+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:06.534615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:53:06.534615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:15.085973+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"77'1","reported_seq":98,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:11.432974+0000","last_change":"2024-03-11T20:52:02.654230+0000","last_active":"2024-03-11T20:53:11.432974+0000","last_peered":"2024-03-11T20:53:11.432974+0000","last_clean":"2024-03-11T20:53:11.432974+0000","last_became_active":"2024-03-11T20:50:28.083256+0000","last_became_peered":"2024-03-11T20:50:28.083256+0000","last_unstale":"2024-03-11T20:53:11.432974+0000","last_undegraded":"2024-03-11T20:53:11.432974+0000","last_fullsized":"2024-03-11T20:53:11.432974+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-03-11T20:52:02.653980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:52:02.653980+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:11.645595+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"84'4","reported_seq":92,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:11.432963+0000","last_change":"2024-03-11T20:52:11.668147+0000","last_active":"2024-03-11T20:53:11.432963+0000","last_peered":"2024-03-11T20:53:11.432963+0000","last_clean":"2024-03-11T20:53:11.432963+0000","last_became_active":"2024-03-11T20:50:30.101572+0000","last_became_peered":"2024-03-11T20:50:30.101572+0000","last_unstale":"2024-03-11T20:53:11.432963+0000","last_undegraded":"2024-03-11T20:53:11.432963+0000","last_fullsized":"2024-03-11T20:53:11.432963+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'4","last_scrub_stamp":"2024-03-11T20:52:11.667810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:52:11.667810+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:40.354519+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":86,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:11.097854+0000","last_change":"2024-03-11T20:52:57.942490+0000","last_active":"2024-03-11T20:53:11.097854+0000","last_peered":"2024-03-11T20:53:11.097854+0000","last_clean":"2024-03-11T20:53:11.097854+0000","last_became_active":"2024-03-11T20:50:34.320001+0000","last_became_peered":"2024-03-11T20:50:34.320001+0000","last_unstale":"2024-03-11T20:53:11.097854+0000","last_undegraded":"2024-03-11T20:53:11.097854+0000","last_fullsized":"2024-03-11T20:53:11.097854+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:57.942382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:52:57.942382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:03.245684+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":89,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:11.979512+0000","last_change":"2024-03-11T20:53:05.100946+0000","last_active":"2024-03-11T20:53:11.979512+0000","last_peered":"2024-03-11T20:53:11.979512+0000","last_clean":"2024-03-11T20:53:11.979512+0000","last_became_active":"2024-03-11T20:50:32.646569+0000","last_became_peered":"2024-03-11T20:50:32.646569+0000","last_unstale":"2024-03-11T20:53:11.979512+0000","last_undegraded":"2024-03-11T20:53:11.979512+0000","last_fullsized":"2024-03-11T20:53:11.979512+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:05.100847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:53:05.100847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:07.646846+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":98,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:11.097022+0000","last_change":"2024-03-11T20:53:02.899718+0000","last_active":"2024-03-11T20:53:11.097022+0000","last_peered":"2024-03-11T20:53:11.097022+0000","last_clean":"2024-03-11T20:53:11.097022+0000","last_became_active":"2024-03-11T20:50:28.044592+0000","last_became_peered":"2024-03-11T20:50:28.044592+0000","last_unstale":"2024-03-11T20:53:11.097022+0000","last_undegraded":"2024-03-11T20:53:11.097022+0000","last_fullsized":"2024-03-11T20:53:11.097022+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:02.899469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:53:02.899469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:21.925538+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"84'11","reported_seq":105,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:11.100311+0000","last_change":"2024-03-11T20:52:21.089002+0000","last_active":"2024-03-11T20:53:11.100311+0000","last_peered":"2024-03-11T20:53:11.100311+0000","last_clean":"2024-03-11T20:53:11.100311+0000","last_became_active":"2024-03-11T20:50:30.103212+0000","last_became_peered":"2024-03-11T20:50:30.103212+0000","last_unstale":"2024-03-11T20:53:11.100311+0000","last_undegraded":"2024-03-11T20:53:11.100311+0000","last_fullsized":"2024-03-11T20:53:11.100311+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T20:52:21.088529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:52:21.088529+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:32.580581+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":86,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:11.427255+0000","last_change":"2024-03-11T20:52:53.451122+0000","last_active":"2024-03-11T20:53:11.427255+0000","last_peered":"2024-03-11T20:53:11.427255+0000","last_clean":"2024-03-11T20:53:11.427255+0000","last_became_active":"2024-03-11T20:50:34.336077+0000","last_became_peered":"2024-03-11T20:50:34.336077+0000","last_unstale":"2024-03-11T20:53:11.427255+0000","last_undegraded":"2024-03-11T20:53:11.427255+0000","last_fullsized":"2024-03-11T20:53:11.427255+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:53.450885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:52:53.450885+0000","last_clean_scrub_stamp":"2024-03-11T20:52:53.450885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:10.721360+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"77'2","reported_seq":106,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:11.984748+0000","last_change":"2024-03-11T20:52:05.512151+0000","last_active":"2024-03-11T20:53:11.984748+0000","last_peered":"2024-03-11T20:53:11.984748+0000","last_clean":"2024-03-11T20:53:11.984748+0000","last_became_active":"2024-03-11T20:50:28.059564+0000","last_became_peered":"2024-03-11T20:50:28.059564+0000","last_unstale":"2024-03-11T20:53:11.984748+0000","last_undegraded":"2024-03-11T20:53:11.984748+0000","last_fullsized":"2024-03-11T20:53:11.984748+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'2","last_scrub_stamp":"2024-03-11T20:52:05.511938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:52:05.511938+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"100'16","reported_seq":118,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:11.101252+0000","last_change":"2024-03-11T20:52:37.207901+0000","last_active":"2024-03-11T20:53:11.101252+0000","last_peered":"2024-03-11T20:53:11.101252+0000","last_clean":"2024-03-11T20:53:11.101252+0000","last_became_active":"2024-03-11T20:50:30.083708+0000","last_became_peered":"2024-03-11T20:50:30.083708+0000","last_unstale":"2024-03-11T20:53:11.101252+0000","last_undegraded":"2024-03-11T20:53:11.101252+0000","last_fullsized":"2024-03-11T20:53:11.101252+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'16","last_scrub_stamp":"2024-03-11T20:52:37.207662+0000","last_deep_scrub":"100'16","last_deep_scrub_stamp":"2024-03-11T20:51:31.194005+0000","last_clean_scrub_stamp":"2024-03-11T20:52:37.207662+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:56.687016+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"106'12","reported_seq":128,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:13.268373+0000","last_change":"2024-03-11T20:52:21.539124+0000","last_active":"2024-03-11T20:53:13.268373+0000","last_peered":"2024-03-11T20:53:13.268373+0000","last_clean":"2024-03-11T20:53:13.268373+0000","last_became_active":"2024-03-11T20:50:32.210620+0000","last_became_peered":"2024-03-11T20:50:32.210620+0000","last_unstale":"2024-03-11T20:53:13.268373+0000","last_undegraded":"2024-03-11T20:53:13.268373+0000","last_fullsized":"2024-03-11T20:53:13.268373+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'12","last_scrub_stamp":"2024-03-11T20:52:21.538885+0000","last_deep_scrub":"106'12","last_deep_scrub_stamp":"2024-03-11T20:52:21.538885+0000","last_clean_scrub_stamp":"2024-03-11T20:52:21.538885+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:37.059443+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":77,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:11.978813+0000","last_change":"2024-03-11T20:52:07.836375+0000","last_active":"2024-03-11T20:53:11.978813+0000","last_peered":"2024-03-11T20:53:11.978813+0000","last_clean":"2024-03-11T20:53:11.978813+0000","last_became_active":"2024-03-11T20:50:34.328932+0000","last_became_peered":"2024-03-11T20:50:34.328932+0000","last_unstale":"2024-03-11T20:53:11.978813+0000","last_undegraded":"2024-03-11T20:53:11.978813+0000","last_fullsized":"2024-03-11T20:53:11.978813+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:07.836159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:52:07.836159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:23.376900+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":90,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:11.097619+0000","last_change":"2024-03-11T20:52:14.175560+0000","last_active":"2024-03-11T20:53:11.097619+0000","last_peered":"2024-03-11T20:53:11.097619+0000","last_clean":"2024-03-11T20:53:11.097619+0000","last_became_active":"2024-03-11T20:50:28.057091+0000","last_became_peered":"2024-03-11T20:50:28.057091+0000","last_unstale":"2024-03-11T20:53:11.097619+0000","last_undegraded":"2024-03-11T20:53:11.097619+0000","last_fullsized":"2024-03-11T20:53:11.097619+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:14.175327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:52:14.175327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:37.308976+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"84'16","reported_seq":143,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:13.268457+0000","last_change":"2024-03-11T20:52:03.937604+0000","last_active":"2024-03-11T20:53:13.268457+0000","last_peered":"2024-03-11T20:53:13.268457+0000","last_clean":"2024-03-11T20:53:13.268457+0000","last_became_active":"2024-03-11T20:50:30.094977+0000","last_became_peered":"2024-03-11T20:50:30.094977+0000","last_unstale":"2024-03-11T20:53:13.268457+0000","last_undegraded":"2024-03-11T20:53:13.268457+0000","last_fullsized":"2024-03-11T20:53:13.268457+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'16","last_scrub_stamp":"2024-03-11T20:52:03.937351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:52:03.937351+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:32.484876+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":82,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:11.097217+0000","last_change":"2024-03-11T20:52:10.160318+0000","last_active":"2024-03-11T20:53:11.097217+0000","last_peered":"2024-03-11T20:53:11.097217+0000","last_clean":"2024-03-11T20:53:11.097217+0000","last_became_active":"2024-03-11T20:50:32.206394+0000","last_became_peered":"2024-03-11T20:50:32.206394+0000","last_unstale":"2024-03-11T20:53:11.097217+0000","last_undegraded":"2024-03-11T20:53:11.097217+0000","last_fullsized":"2024-03-11T20:53:11.097217+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:10.160202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:52:10.160202+0000","last_clean_scrub_stamp":"2024-03-11T20:52:10.160202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:29.912826+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":78,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:11.427905+0000","last_change":"2024-03-11T20:52:15.511775+0000","last_active":"2024-03-11T20:53:11.427905+0000","last_peered":"2024-03-11T20:53:11.427905+0000","last_clean":"2024-03-11T20:53:11.427905+0000","last_became_active":"2024-03-11T20:50:34.329253+0000","last_became_peered":"2024-03-11T20:50:34.329253+0000","last_unstale":"2024-03-11T20:53:11.427905+0000","last_undegraded":"2024-03-11T20:53:11.427905+0000","last_fullsized":"2024-03-11T20:53:11.427905+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:15.511437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:52:15.511437+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:27.180999+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":98,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:11.837952+0000","last_change":"2024-03-11T20:52:58.782154+0000","last_active":"2024-03-11T20:53:11.837952+0000","last_peered":"2024-03-11T20:53:11.837952+0000","last_clean":"2024-03-11T20:53:11.837952+0000","last_became_active":"2024-03-11T20:50:28.075678+0000","last_became_peered":"2024-03-11T20:50:28.075678+0000","last_unstale":"2024-03-11T20:53:11.837952+0000","last_undegraded":"2024-03-11T20:53:11.837952+0000","last_fullsized":"2024-03-11T20:53:11.837952+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:58.781938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:33.823704+0000","last_clean_scrub_stamp":"2024-03-11T20:52:58.781938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:19.252494+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"84'19","reported_seq":130,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:13.560470+0000","last_change":"2024-03-11T20:53:13.560575+0000","last_active":"2024-03-11T20:53:13.560470+0000","last_peered":"2024-03-11T20:53:13.560470+0000","last_clean":"2024-03-11T20:53:13.560470+0000","last_became_active":"2024-03-11T20:50:30.097543+0000","last_became_peered":"2024-03-11T20:50:30.097543+0000","last_unstale":"2024-03-11T20:53:13.560470+0000","last_undegraded":"2024-03-11T20:53:13.560470+0000","last_fullsized":"2024-03-11T20:53:13.560470+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'19","last_scrub_stamp":"2024-03-11T20:53:13.560307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:53:13.560307+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:17.749885+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":82,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:11.097526+0000","last_change":"2024-03-11T20:52:16.096498+0000","last_active":"2024-03-11T20:53:11.097526+0000","last_peered":"2024-03-11T20:53:11.097526+0000","last_clean":"2024-03-11T20:53:11.097526+0000","last_became_active":"2024-03-11T20:50:32.211494+0000","last_became_peered":"2024-03-11T20:50:32.211494+0000","last_unstale":"2024-03-11T20:53:11.097526+0000","last_undegraded":"2024-03-11T20:53:11.097526+0000","last_fullsized":"2024-03-11T20:53:11.097526+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:16.096259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:52:16.096259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:16.697891+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":78,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:11.100787+0000","last_change":"2024-03-11T20:52:25.115485+0000","last_active":"2024-03-11T20:53:11.100787+0000","last_peered":"2024-03-11T20:53:11.100787+0000","last_clean":"2024-03-11T20:53:11.100787+0000","last_became_active":"2024-03-11T20:50:34.321038+0000","last_became_peered":"2024-03-11T20:50:34.321038+0000","last_unstale":"2024-03-11T20:53:11.100787+0000","last_undegraded":"2024-03-11T20:53:11.100787+0000","last_fullsized":"2024-03-11T20:53:11.100787+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:25.115238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:52:25.115238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:34.308256+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"109'26","reported_seq":134,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:11.096515+0000","last_change":"2024-03-11T20:52:49.216560+0000","last_active":"2024-03-11T20:53:11.096515+0000","last_peered":"2024-03-11T20:53:11.096515+0000","last_clean":"2024-03-11T20:53:11.096515+0000","last_became_active":"2024-03-11T20:50:30.098371+0000","last_became_peered":"2024-03-11T20:50:30.098371+0000","last_unstale":"2024-03-11T20:53:11.096515+0000","last_undegraded":"2024-03-11T20:53:11.096515+0000","last_fullsized":"2024-03-11T20:53:11.096515+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'26","last_scrub_stamp":"2024-03-11T20:52:49.216378+0000","last_deep_scrub":"109'26","last_deep_scrub_stamp":"2024-03-11T20:52:49.216378+0000","last_clean_scrub_stamp":"2024-03-11T20:52:49.216378+0000","objects_scrubbed":8,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:49.844091+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":98,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:11.097099+0000","last_change":"2024-03-11T20:53:06.797551+0000","last_active":"2024-03-11T20:53:11.097099+0000","last_peered":"2024-03-11T20:53:11.097099+0000","last_clean":"2024-03-11T20:53:11.097099+0000","last_became_active":"2024-03-11T20:50:28.045625+0000","last_became_peered":"2024-03-11T20:50:28.045625+0000","last_unstale":"2024-03-11T20:53:11.097099+0000","last_undegraded":"2024-03-11T20:53:11.097099+0000","last_fullsized":"2024-03-11T20:53:11.097099+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:06.797336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:53:06.797336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:24.649727+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":82,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:11.096743+0000","last_change":"2024-03-11T20:52:25.125883+0000","last_active":"2024-03-11T20:53:11.096743+0000","last_peered":"2024-03-11T20:53:11.096743+0000","last_clean":"2024-03-11T20:53:11.096743+0000","last_became_active":"2024-03-11T20:50:32.205996+0000","last_became_peered":"2024-03-11T20:50:32.205996+0000","last_unstale":"2024-03-11T20:53:11.096743+0000","last_undegraded":"2024-03-11T20:53:11.096743+0000","last_fullsized":"2024-03-11T20:53:11.096743+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:25.125696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:52:25.125696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:52.303574+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":78,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:11.839916+0000","last_change":"2024-03-11T20:52:11.898901+0000","last_active":"2024-03-11T20:53:11.839916+0000","last_peered":"2024-03-11T20:53:11.839916+0000","last_clean":"2024-03-11T20:53:11.839916+0000","last_became_active":"2024-03-11T20:50:34.336066+0000","last_became_peered":"2024-03-11T20:50:34.336066+0000","last_unstale":"2024-03-11T20:53:11.839916+0000","last_undegraded":"2024-03-11T20:53:11.839916+0000","last_fullsized":"2024-03-11T20:53:11.839916+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:11.898570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:52:11.898570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:29.169074+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"84'10","reported_seq":101,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:11.425793+0000","last_change":"2024-03-11T20:52:35.399348+0000","last_active":"2024-03-11T20:53:11.425793+0000","last_peered":"2024-03-11T20:53:11.425793+0000","last_clean":"2024-03-11T20:53:11.425793+0000","last_became_active":"2024-03-11T20:50:30.101747+0000","last_became_peered":"2024-03-11T20:50:30.101747+0000","last_unstale":"2024-03-11T20:53:11.425793+0000","last_undegraded":"2024-03-11T20:53:11.425793+0000","last_fullsized":"2024-03-11T20:53:11.425793+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'10","last_scrub_stamp":"2024-03-11T20:52:35.399027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:52:35.399027+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:57.620344+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":98,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:11.837957+0000","last_change":"2024-03-11T20:52:44.919146+0000","last_active":"2024-03-11T20:53:11.837957+0000","last_peered":"2024-03-11T20:53:11.837957+0000","last_clean":"2024-03-11T20:53:11.837957+0000","last_became_active":"2024-03-11T20:50:28.085503+0000","last_became_peered":"2024-03-11T20:50:28.085503+0000","last_unstale":"2024-03-11T20:53:11.837957+0000","last_undegraded":"2024-03-11T20:53:11.837957+0000","last_fullsized":"2024-03-11T20:53:11.837957+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:44.918937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:52:44.918937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:47.097462+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"102'128","reported_seq":355,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:11.100499+0000","last_change":"2024-03-11T20:52:28.047817+0000","last_active":"2024-03-11T20:53:11.100499+0000","last_peered":"2024-03-11T20:53:11.100499+0000","last_clean":"2024-03-11T20:53:11.100499+0000","last_became_active":"2024-03-11T20:49:52.584543+0000","last_became_peered":"2024-03-11T20:49:52.584543+0000","last_unstale":"2024-03-11T20:53:11.100499+0000","last_undegraded":"2024-03-11T20:53:11.100499+0000","last_fullsized":"2024-03-11T20:53:11.100499+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"102'128","last_scrub_stamp":"2024-03-11T20:52:28.047573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:45:32.381751+0000","last_clean_scrub_stamp":"2024-03-11T20:52:28.047573+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:54.167386+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":90,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:11.433590+0000","last_change":"2024-03-11T20:53:07.859791+0000","last_active":"2024-03-11T20:53:11.433590+0000","last_peered":"2024-03-11T20:53:11.433590+0000","last_clean":"2024-03-11T20:53:11.433590+0000","last_became_active":"2024-03-11T20:50:32.652128+0000","last_became_peered":"2024-03-11T20:50:32.652128+0000","last_unstale":"2024-03-11T20:53:11.433590+0000","last_undegraded":"2024-03-11T20:53:11.433590+0000","last_fullsized":"2024-03-11T20:53:11.433590+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:07.859545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:53:07.859545+0000","last_clean_scrub_stamp":"2024-03-11T20:53:07.859545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:20.657302+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":78,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:11.097420+0000","last_change":"2024-03-11T20:52:09.169969+0000","last_active":"2024-03-11T20:53:11.097420+0000","last_peered":"2024-03-11T20:53:11.097420+0000","last_clean":"2024-03-11T20:53:11.097420+0000","last_became_active":"2024-03-11T20:50:34.326321+0000","last_became_peered":"2024-03-11T20:50:34.326321+0000","last_unstale":"2024-03-11T20:53:11.097420+0000","last_undegraded":"2024-03-11T20:53:11.097420+0000","last_fullsized":"2024-03-11T20:53:11.097420+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:09.169726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:52:09.169726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"100'21","reported_seq":121,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:11.978908+0000","last_change":"2024-03-11T20:53:03.110687+0000","last_active":"2024-03-11T20:53:11.978908+0000","last_peered":"2024-03-11T20:53:11.978908+0000","last_clean":"2024-03-11T20:53:11.978908+0000","last_became_active":"2024-03-11T20:50:30.106110+0000","last_became_peered":"2024-03-11T20:50:30.106110+0000","last_unstale":"2024-03-11T20:53:11.978908+0000","last_undegraded":"2024-03-11T20:53:11.978908+0000","last_fullsized":"2024-03-11T20:53:11.978908+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'21","last_scrub_stamp":"2024-03-11T20:53:03.110450+0000","last_deep_scrub":"100'21","last_deep_scrub_stamp":"2024-03-11T20:51:31.835302+0000","last_clean_scrub_stamp":"2024-03-11T20:53:03.110450+0000","objects_scrubbed":7,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:23.836851+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":55,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:11.096705+0000","last_change":"2024-03-11T20:52:56.891528+0000","last_active":"2024-03-11T20:53:11.096705+0000","last_peered":"2024-03-11T20:53:11.096705+0000","last_clean":"2024-03-11T20:53:11.096705+0000","last_became_active":"2024-03-11T20:51:22.668492+0000","last_became_peered":"2024-03-11T20:51:22.668492+0000","last_unstale":"2024-03-11T20:53:11.096705+0000","last_undegraded":"2024-03-11T20:53:11.096705+0000","last_fullsized":"2024-03-11T20:53:11.096705+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:56.891410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:52:56.891410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:57.500706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":89,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:11.979656+0000","last_change":"2024-03-11T20:53:04.140960+0000","last_active":"2024-03-11T20:53:11.979656+0000","last_peered":"2024-03-11T20:53:11.979656+0000","last_clean":"2024-03-11T20:53:11.979656+0000","last_became_active":"2024-03-11T20:50:32.195566+0000","last_became_peered":"2024-03-11T20:50:32.195566+0000","last_unstale":"2024-03-11T20:53:11.979656+0000","last_undegraded":"2024-03-11T20:53:11.979656+0000","last_fullsized":"2024-03-11T20:53:11.979656+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:04.140875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:53:04.140875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:29.280686+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":86,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:11.098387+0000","last_change":"2024-03-11T20:53:08.170244+0000","last_active":"2024-03-11T20:53:11.098387+0000","last_peered":"2024-03-11T20:53:11.098387+0000","last_clean":"2024-03-11T20:53:11.098387+0000","last_became_active":"2024-03-11T20:50:34.306750+0000","last_became_peered":"2024-03-11T20:50:34.306750+0000","last_unstale":"2024-03-11T20:53:11.098387+0000","last_undegraded":"2024-03-11T20:53:11.098387+0000","last_fullsized":"2024-03-11T20:53:11.098387+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:08.170036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:53:08.170036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:35.208803+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"87'2","reported_seq":91,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:11.433011+0000","last_change":"2024-03-11T20:51:52.747686+0000","last_active":"2024-03-11T20:53:11.433011+0000","last_peered":"2024-03-11T20:53:11.433011+0000","last_clean":"2024-03-11T20:53:11.433011+0000","last_became_active":"2024-03-11T20:50:32.195452+0000","last_became_peered":"2024-03-11T20:50:32.195452+0000","last_unstale":"2024-03-11T20:53:11.433011+0000","last_undegraded":"2024-03-11T20:53:11.433011+0000","last_fullsized":"2024-03-11T20:53:11.433011+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'2","last_scrub_stamp":"2024-03-11T20:51:52.747446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:51:52.747446+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:13.971732+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0024049620000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":98,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:11.101087+0000","last_change":"2024-03-11T20:53:05.364652+0000","last_active":"2024-03-11T20:53:11.101087+0000","last_peered":"2024-03-11T20:53:11.101087+0000","last_clean":"2024-03-11T20:53:11.101087+0000","last_became_active":"2024-03-11T20:50:28.065245+0000","last_became_peered":"2024-03-11T20:50:28.065245+0000","last_unstale":"2024-03-11T20:53:11.101087+0000","last_undegraded":"2024-03-11T20:53:11.101087+0000","last_fullsized":"2024-03-11T20:53:11.101087+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:05.364429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:53:05.364429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:25.580910+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"84'30","reported_seq":179,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:13.264028+0000","last_change":"2024-03-11T20:52:48.167124+0000","last_active":"2024-03-11T20:53:13.264028+0000","last_peered":"2024-03-11T20:53:13.264028+0000","last_clean":"2024-03-11T20:53:13.264028+0000","last_became_active":"2024-03-11T20:50:30.095177+0000","last_became_peered":"2024-03-11T20:50:30.095177+0000","last_unstale":"2024-03-11T20:53:13.264028+0000","last_undegraded":"2024-03-11T20:53:13.264028+0000","last_fullsized":"2024-03-11T20:53:13.264028+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'30","last_scrub_stamp":"2024-03-11T20:52:48.166893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:52:48.166893+0000","objects_scrubbed":10,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:05.469139+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":51,"num_read_kb":36,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"84'11","reported_seq":126,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:13.264041+0000","last_change":"2024-03-11T20:52:06.873338+0000","last_active":"2024-03-11T20:53:13.264041+0000","last_peered":"2024-03-11T20:53:13.264041+0000","last_clean":"2024-03-11T20:53:13.264041+0000","last_became_active":"2024-03-11T20:50:32.649414+0000","last_became_peered":"2024-03-11T20:50:32.649414+0000","last_unstale":"2024-03-11T20:53:13.264041+0000","last_undegraded":"2024-03-11T20:53:13.264041+0000","last_fullsized":"2024-03-11T20:53:13.264041+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T20:52:06.873096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:52:06.873096+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:33.849748+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":78,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:11.983227+0000","last_change":"2024-03-11T20:51:44.593465+0000","last_active":"2024-03-11T20:53:11.983227+0000","last_peered":"2024-03-11T20:53:11.983227+0000","last_clean":"2024-03-11T20:53:11.983227+0000","last_became_active":"2024-03-11T20:50:34.323647+0000","last_became_peered":"2024-03-11T20:50:34.323647+0000","last_unstale":"2024-03-11T20:53:11.983227+0000","last_undegraded":"2024-03-11T20:53:11.983227+0000","last_fullsized":"2024-03-11T20:53:11.983227+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:44.593255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:51:44.593255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":89,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:11.982653+0000","last_change":"2024-03-11T20:51:53.560890+0000","last_active":"2024-03-11T20:53:11.982653+0000","last_peered":"2024-03-11T20:53:11.982653+0000","last_clean":"2024-03-11T20:53:11.982653+0000","last_became_active":"2024-03-11T20:50:32.650102+0000","last_became_peered":"2024-03-11T20:50:32.650102+0000","last_unstale":"2024-03-11T20:53:11.982653+0000","last_undegraded":"2024-03-11T20:53:11.982653+0000","last_fullsized":"2024-03-11T20:53:11.982653+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:53.560509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:51:53.560509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0022993710000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"77'1","reported_seq":91,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:11.096955+0000","last_change":"2024-03-11T20:52:11.202801+0000","last_active":"2024-03-11T20:53:11.096955+0000","last_peered":"2024-03-11T20:53:11.096955+0000","last_clean":"2024-03-11T20:53:11.096955+0000","last_became_active":"2024-03-11T20:50:28.065076+0000","last_became_peered":"2024-03-11T20:50:28.065076+0000","last_unstale":"2024-03-11T20:53:11.096955+0000","last_undegraded":"2024-03-11T20:53:11.096955+0000","last_fullsized":"2024-03-11T20:53:11.096955+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-03-11T20:52:11.202551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:52:11.202551+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:15.958895+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"84'13","reported_seq":113,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:11.428110+0000","last_change":"2024-03-11T20:52:19.450300+0000","last_active":"2024-03-11T20:53:11.428110+0000","last_peered":"2024-03-11T20:53:11.428110+0000","last_clean":"2024-03-11T20:53:11.428110+0000","last_became_active":"2024-03-11T20:50:30.097669+0000","last_became_peered":"2024-03-11T20:50:30.097669+0000","last_unstale":"2024-03-11T20:53:11.428110+0000","last_undegraded":"2024-03-11T20:53:11.428110+0000","last_fullsized":"2024-03-11T20:53:11.428110+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'13","last_scrub_stamp":"2024-03-11T20:52:19.449958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:52:19.449958+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:20.076288+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"84'11","reported_seq":135,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:13.413293+0000","last_change":"2024-03-11T20:53:13.413293+0000","last_active":"2024-03-11T20:53:13.413293+0000","last_peered":"2024-03-11T20:53:13.413293+0000","last_clean":"2024-03-11T20:53:13.413293+0000","last_became_active":"2024-03-11T20:50:32.647041+0000","last_became_peered":"2024-03-11T20:50:32.647041+0000","last_unstale":"2024-03-11T20:53:13.413293+0000","last_undegraded":"2024-03-11T20:53:13.413293+0000","last_fullsized":"2024-03-11T20:53:13.413293+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T20:53:13.413140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:53:13.413140+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:27.508758+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":86,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:11.100207+0000","last_change":"2024-03-11T20:53:06.344681+0000","last_active":"2024-03-11T20:53:11.100207+0000","last_peered":"2024-03-11T20:53:11.100207+0000","last_clean":"2024-03-11T20:53:11.100207+0000","last_became_active":"2024-03-11T20:50:34.326947+0000","last_became_peered":"2024-03-11T20:50:34.326947+0000","last_unstale":"2024-03-11T20:53:11.100207+0000","last_undegraded":"2024-03-11T20:53:11.100207+0000","last_fullsized":"2024-03-11T20:53:11.100207+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:06.344386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:53:06.344386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:09.051382+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"91'5","reported_seq":248,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:17.993526+0000","last_change":"2024-03-11T20:53:08.493671+0000","last_active":"2024-03-11T20:53:17.993526+0000","last_peered":"2024-03-11T20:53:17.993526+0000","last_clean":"2024-03-11T20:53:17.993526+0000","last_became_active":"2024-03-11T20:50:32.211118+0000","last_became_peered":"2024-03-11T20:50:32.211118+0000","last_unstale":"2024-03-11T20:53:17.993526+0000","last_undegraded":"2024-03-11T20:53:17.993526+0000","last_fullsized":"2024-03-11T20:53:17.993526+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'5","last_scrub_stamp":"2024-03-11T20:53:08.493516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:53:08.493516+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:17.353002+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0035533370000000002,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":148,"num_read_kb":143,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":90,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:11.434149+0000","last_change":"2024-03-11T20:52:14.613926+0000","last_active":"2024-03-11T20:53:11.434149+0000","last_peered":"2024-03-11T20:53:11.434149+0000","last_clean":"2024-03-11T20:53:11.434149+0000","last_became_active":"2024-03-11T20:50:28.071707+0000","last_became_peered":"2024-03-11T20:50:28.071707+0000","last_unstale":"2024-03-11T20:53:11.434149+0000","last_undegraded":"2024-03-11T20:53:11.434149+0000","last_fullsized":"2024-03-11T20:53:11.434149+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:14.613707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:52:14.613707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:27.285465+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"84'12","reported_seq":103,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:11.979802+0000","last_change":"2024-03-11T20:52:04.904973+0000","last_active":"2024-03-11T20:53:11.979802+0000","last_peered":"2024-03-11T20:53:11.979802+0000","last_clean":"2024-03-11T20:53:11.979802+0000","last_became_active":"2024-03-11T20:50:30.096350+0000","last_became_peered":"2024-03-11T20:50:30.096350+0000","last_unstale":"2024-03-11T20:53:11.979802+0000","last_undegraded":"2024-03-11T20:53:11.979802+0000","last_fullsized":"2024-03-11T20:53:11.979802+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'12","last_scrub_stamp":"2024-03-11T20:52:04.904721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:52:04.904721+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:31.218718+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":82,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:11.983166+0000","last_change":"2024-03-11T20:52:09.577290+0000","last_active":"2024-03-11T20:53:11.983166+0000","last_peered":"2024-03-11T20:53:11.983166+0000","last_clean":"2024-03-11T20:53:11.983166+0000","last_became_active":"2024-03-11T20:50:32.190896+0000","last_became_peered":"2024-03-11T20:50:32.190896+0000","last_unstale":"2024-03-11T20:53:11.983166+0000","last_undegraded":"2024-03-11T20:53:11.983166+0000","last_fullsized":"2024-03-11T20:53:11.983166+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:09.577102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:52:09.577102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:13.346655+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":78,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:11.426580+0000","last_change":"2024-03-11T20:52:33.343010+0000","last_active":"2024-03-11T20:53:11.426580+0000","last_peered":"2024-03-11T20:53:11.426580+0000","last_clean":"2024-03-11T20:53:11.426580+0000","last_became_active":"2024-03-11T20:50:34.335370+0000","last_became_peered":"2024-03-11T20:50:34.335370+0000","last_unstale":"2024-03-11T20:53:11.426580+0000","last_undegraded":"2024-03-11T20:53:11.426580+0000","last_fullsized":"2024-03-11T20:53:11.426580+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:33.342759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:52:33.342759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:33.629066+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"84'11","reported_seq":127,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:13.268332+0000","last_change":"2024-03-11T20:52:15.500805+0000","last_active":"2024-03-11T20:53:13.268332+0000","last_peered":"2024-03-11T20:53:13.268332+0000","last_clean":"2024-03-11T20:53:13.268332+0000","last_became_active":"2024-03-11T20:50:32.197412+0000","last_became_peered":"2024-03-11T20:50:32.197412+0000","last_unstale":"2024-03-11T20:53:13.268332+0000","last_undegraded":"2024-03-11T20:53:13.268332+0000","last_fullsized":"2024-03-11T20:53:13.268332+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T20:52:15.500571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:52:15.500571+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:37.062481+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":90,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:11.100171+0000","last_change":"2024-03-11T20:52:09.043821+0000","last_active":"2024-03-11T20:53:11.100171+0000","last_peered":"2024-03-11T20:53:11.100171+0000","last_clean":"2024-03-11T20:53:11.100171+0000","last_became_active":"2024-03-11T20:50:28.079413+0000","last_became_peered":"2024-03-11T20:50:28.079413+0000","last_unstale":"2024-03-11T20:53:11.100171+0000","last_undegraded":"2024-03-11T20:53:11.100171+0000","last_fullsized":"2024-03-11T20:53:11.100171+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:09.043655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:52:09.043655+0000","last_clean_scrub_stamp":"2024-03-11T20:52:09.043655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:17.358596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"84'12","reported_seq":109,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:11.982895+0000","last_change":"2024-03-11T20:51:59.529431+0000","last_active":"2024-03-11T20:53:11.982895+0000","last_peered":"2024-03-11T20:53:11.982895+0000","last_clean":"2024-03-11T20:53:11.982895+0000","last_became_active":"2024-03-11T20:50:30.083388+0000","last_became_peered":"2024-03-11T20:50:30.083388+0000","last_unstale":"2024-03-11T20:53:11.982895+0000","last_undegraded":"2024-03-11T20:53:11.982895+0000","last_fullsized":"2024-03-11T20:53:11.982895+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'12","last_scrub_stamp":"2024-03-11T20:51:59.529049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:51:59.529049+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":86,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:11.839134+0000","last_change":"2024-03-11T20:53:06.771715+0000","last_active":"2024-03-11T20:53:11.839134+0000","last_peered":"2024-03-11T20:53:11.839134+0000","last_clean":"2024-03-11T20:53:11.839134+0000","last_became_active":"2024-03-11T20:50:34.337717+0000","last_became_peered":"2024-03-11T20:50:34.337717+0000","last_unstale":"2024-03-11T20:53:11.839134+0000","last_undegraded":"2024-03-11T20:53:11.839134+0000","last_fullsized":"2024-03-11T20:53:11.839134+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:06.771620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:53:06.771620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:30.302985+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":55,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:11.097704+0000","last_change":"2024-03-11T20:53:05.836875+0000","last_active":"2024-03-11T20:53:11.097704+0000","last_peered":"2024-03-11T20:53:11.097704+0000","last_clean":"2024-03-11T20:53:11.097704+0000","last_became_active":"2024-03-11T20:51:22.662693+0000","last_became_peered":"2024-03-11T20:51:22.662693+0000","last_unstale":"2024-03-11T20:53:11.097704+0000","last_undegraded":"2024-03-11T20:53:11.097704+0000","last_fullsized":"2024-03-11T20:53:11.097704+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:05.836655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:53:05.836655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:21.447337+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":98,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:17.169092+0000","last_change":"2024-03-11T20:53:17.169155+0000","last_active":"2024-03-11T20:53:17.169092+0000","last_peered":"2024-03-11T20:53:17.169092+0000","last_clean":"2024-03-11T20:53:17.169092+0000","last_became_active":"2024-03-11T20:50:28.058790+0000","last_became_peered":"2024-03-11T20:50:28.058790+0000","last_unstale":"2024-03-11T20:53:17.169092+0000","last_undegraded":"2024-03-11T20:53:17.169092+0000","last_fullsized":"2024-03-11T20:53:17.169092+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:17.168912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:40.974147+0000","last_clean_scrub_stamp":"2024-03-11T20:53:17.168912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:23.351155+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"84'15","reported_seq":111,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:11.425841+0000","last_change":"2024-03-11T20:52:40.434414+0000","last_active":"2024-03-11T20:53:11.425841+0000","last_peered":"2024-03-11T20:53:11.425841+0000","last_clean":"2024-03-11T20:53:11.425841+0000","last_became_active":"2024-03-11T20:50:30.096215+0000","last_became_peered":"2024-03-11T20:50:30.096215+0000","last_unstale":"2024-03-11T20:53:11.425841+0000","last_undegraded":"2024-03-11T20:53:11.425841+0000","last_fullsized":"2024-03-11T20:53:11.425841+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15","last_scrub_stamp":"2024-03-11T20:52:40.434169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:52:40.434169+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:45.160041+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":78,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:11.985047+0000","last_change":"2024-03-11T20:52:12.514160+0000","last_active":"2024-03-11T20:53:11.985047+0000","last_peered":"2024-03-11T20:53:11.985047+0000","last_clean":"2024-03-11T20:53:11.985047+0000","last_became_active":"2024-03-11T20:50:34.331803+0000","last_became_peered":"2024-03-11T20:50:34.331803+0000","last_unstale":"2024-03-11T20:53:11.985047+0000","last_undegraded":"2024-03-11T20:53:11.985047+0000","last_fullsized":"2024-03-11T20:53:11.985047+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:12.513982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:52:12.513982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:31.458459+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":90,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:11.097005+0000","last_change":"2024-03-11T20:52:23.165274+0000","last_active":"2024-03-11T20:53:11.097005+0000","last_peered":"2024-03-11T20:53:11.097005+0000","last_clean":"2024-03-11T20:53:11.097005+0000","last_became_active":"2024-03-11T20:50:28.057121+0000","last_became_peered":"2024-03-11T20:50:28.057121+0000","last_unstale":"2024-03-11T20:53:11.097005+0000","last_undegraded":"2024-03-11T20:53:11.097005+0000","last_fullsized":"2024-03-11T20:53:11.097005+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:23.165119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:52:23.165119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:34.501490+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"84'9","reported_seq":110,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:11.426861+0000","last_change":"2024-03-11T20:53:03.452779+0000","last_active":"2024-03-11T20:53:11.426861+0000","last_peered":"2024-03-11T20:53:11.426861+0000","last_clean":"2024-03-11T20:53:11.426861+0000","last_became_active":"2024-03-11T20:50:30.089260+0000","last_became_peered":"2024-03-11T20:50:30.089260+0000","last_unstale":"2024-03-11T20:53:11.426861+0000","last_undegraded":"2024-03-11T20:53:11.426861+0000","last_fullsized":"2024-03-11T20:53:11.426861+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'9","last_scrub_stamp":"2024-03-11T20:53:03.452528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:53:03.452528+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:23.786382+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":90,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:11.983741+0000","last_change":"2024-03-11T20:53:00.551613+0000","last_active":"2024-03-11T20:53:11.983741+0000","last_peered":"2024-03-11T20:53:11.983741+0000","last_clean":"2024-03-11T20:53:11.983741+0000","last_became_active":"2024-03-11T20:50:32.200001+0000","last_became_peered":"2024-03-11T20:50:32.200001+0000","last_unstale":"2024-03-11T20:53:11.983741+0000","last_undegraded":"2024-03-11T20:53:11.983741+0000","last_fullsized":"2024-03-11T20:53:11.983741+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:00.551395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:35.670149+0000","last_clean_scrub_stamp":"2024-03-11T20:53:00.551395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:08.450926+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":86,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:11.097835+0000","last_change":"2024-03-11T20:52:54.879520+0000","last_active":"2024-03-11T20:53:11.097835+0000","last_peered":"2024-03-11T20:53:11.097835+0000","last_clean":"2024-03-11T20:53:11.097835+0000","last_became_active":"2024-03-11T20:50:34.326354+0000","last_became_peered":"2024-03-11T20:50:34.326354+0000","last_unstale":"2024-03-11T20:53:11.097835+0000","last_undegraded":"2024-03-11T20:53:11.097835+0000","last_fullsized":"2024-03-11T20:53:11.097835+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:54.879292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:52:54.879292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:00.895826+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"98'12","reported_seq":137,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:13.268442+0000","last_change":"2024-03-11T20:52:01.513643+0000","last_active":"2024-03-11T20:53:13.268442+0000","last_peered":"2024-03-11T20:53:13.268442+0000","last_clean":"2024-03-11T20:53:13.268442+0000","last_became_active":"2024-03-11T20:51:22.713613+0000","last_became_peered":"2024-03-11T20:51:22.713613+0000","last_unstale":"2024-03-11T20:53:13.268442+0000","last_undegraded":"2024-03-11T20:53:13.268442+0000","last_fullsized":"2024-03-11T20:53:13.268442+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'12","last_scrub_stamp":"2024-03-11T20:52:01.513305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:52:01.513305+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:20.040552+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":98,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:11.100269+0000","last_change":"2024-03-11T20:52:51.161508+0000","last_active":"2024-03-11T20:53:11.100269+0000","last_peered":"2024-03-11T20:53:11.100269+0000","last_clean":"2024-03-11T20:53:11.100269+0000","last_became_active":"2024-03-11T20:50:28.069267+0000","last_became_peered":"2024-03-11T20:50:28.069267+0000","last_unstale":"2024-03-11T20:53:11.100269+0000","last_undegraded":"2024-03-11T20:53:11.100269+0000","last_fullsized":"2024-03-11T20:53:11.100269+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:51.161281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:52:51.161281+0000","last_clean_scrub_stamp":"2024-03-11T20:52:51.161281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:57.333795+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"84'19","reported_seq":125,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:16.962524+0000","last_change":"2024-03-11T20:53:16.962607+0000","last_active":"2024-03-11T20:53:16.962524+0000","last_peered":"2024-03-11T20:53:16.962524+0000","last_clean":"2024-03-11T20:53:16.962524+0000","last_became_active":"2024-03-11T20:50:30.103666+0000","last_became_peered":"2024-03-11T20:50:30.103666+0000","last_unstale":"2024-03-11T20:53:16.962524+0000","last_undegraded":"2024-03-11T20:53:16.962524+0000","last_fullsized":"2024-03-11T20:53:16.962524+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'19","last_scrub_stamp":"2024-03-11T20:53:16.962364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:53:16.962364+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:37.903805+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":86,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:18.331775+0000","last_change":"2024-03-11T20:53:18.331836+0000","last_active":"2024-03-11T20:53:18.331775+0000","last_peered":"2024-03-11T20:53:18.331775+0000","last_clean":"2024-03-11T20:53:18.331775+0000","last_became_active":"2024-03-11T20:50:34.327139+0000","last_became_peered":"2024-03-11T20:50:34.327139+0000","last_unstale":"2024-03-11T20:53:18.331775+0000","last_undegraded":"2024-03-11T20:53:18.331775+0000","last_fullsized":"2024-03-11T20:53:18.331775+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:18.331616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:53:18.331616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:43.891860+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":98,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:11.097604+0000","last_change":"2024-03-11T20:53:08.797289+0000","last_active":"2024-03-11T20:53:11.097604+0000","last_peered":"2024-03-11T20:53:11.097604+0000","last_clean":"2024-03-11T20:53:11.097604+0000","last_became_active":"2024-03-11T20:50:28.049901+0000","last_became_peered":"2024-03-11T20:50:28.049901+0000","last_unstale":"2024-03-11T20:53:11.097604+0000","last_undegraded":"2024-03-11T20:53:11.097604+0000","last_fullsized":"2024-03-11T20:53:11.097604+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:08.797203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:53:08.797203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:12.120056+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"102'25","reported_seq":130,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:11.101352+0000","last_change":"2024-03-11T20:52:47.207949+0000","last_active":"2024-03-11T20:53:11.101352+0000","last_peered":"2024-03-11T20:53:11.101352+0000","last_clean":"2024-03-11T20:53:11.101352+0000","last_became_active":"2024-03-11T20:50:30.111819+0000","last_became_peered":"2024-03-11T20:50:30.111819+0000","last_unstale":"2024-03-11T20:53:11.101352+0000","last_undegraded":"2024-03-11T20:53:11.101352+0000","last_fullsized":"2024-03-11T20:53:11.101352+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"102'25","last_scrub_stamp":"2024-03-11T20:52:47.207710+0000","last_deep_scrub":"102'25","last_deep_scrub_stamp":"2024-03-11T20:51:35.187094+0000","last_clean_scrub_stamp":"2024-03-11T20:52:47.207710+0000","objects_scrubbed":8,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:55.994836+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":82,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:11.432904+0000","last_change":"2024-03-11T20:52:19.576642+0000","last_active":"2024-03-11T20:53:11.432904+0000","last_peered":"2024-03-11T20:53:11.432904+0000","last_clean":"2024-03-11T20:53:11.432904+0000","last_became_active":"2024-03-11T20:50:32.649877+0000","last_became_peered":"2024-03-11T20:50:32.649877+0000","last_unstale":"2024-03-11T20:53:11.432904+0000","last_undegraded":"2024-03-11T20:53:11.432904+0000","last_fullsized":"2024-03-11T20:53:11.432904+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:19.576426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:52:19.576426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:20.103831+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":85,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:11.978779+0000","last_change":"2024-03-11T20:52:46.946707+0000","last_active":"2024-03-11T20:53:11.978779+0000","last_peered":"2024-03-11T20:53:11.978779+0000","last_clean":"2024-03-11T20:53:11.978779+0000","last_became_active":"2024-03-11T20:50:34.319536+0000","last_became_peered":"2024-03-11T20:50:34.319536+0000","last_unstale":"2024-03-11T20:53:11.978779+0000","last_undegraded":"2024-03-11T20:53:11.978779+0000","last_fullsized":"2024-03-11T20:53:11.978779+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:46.946561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:52:46.946561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:52.449594+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":90,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:11.096668+0000","last_change":"2024-03-11T20:52:01.105719+0000","last_active":"2024-03-11T20:53:11.096668+0000","last_peered":"2024-03-11T20:53:11.096668+0000","last_clean":"2024-03-11T20:53:11.096668+0000","last_became_active":"2024-03-11T20:50:28.043982+0000","last_became_peered":"2024-03-11T20:50:28.043982+0000","last_unstale":"2024-03-11T20:53:11.096668+0000","last_undegraded":"2024-03-11T20:53:11.096668+0000","last_fullsized":"2024-03-11T20:53:11.096668+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:01.105491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:52:01.105491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:29.786816+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"84'10","reported_seq":109,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:12.823008+0000","last_change":"2024-03-11T20:53:12.823097+0000","last_active":"2024-03-11T20:53:12.823008+0000","last_peered":"2024-03-11T20:53:12.823008+0000","last_clean":"2024-03-11T20:53:12.823008+0000","last_became_active":"2024-03-11T20:50:30.109549+0000","last_became_peered":"2024-03-11T20:50:30.109549+0000","last_unstale":"2024-03-11T20:53:12.823008+0000","last_undegraded":"2024-03-11T20:53:12.823008+0000","last_fullsized":"2024-03-11T20:53:12.823008+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'10","last_scrub_stamp":"2024-03-11T20:53:12.822842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:53:12.822842+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:28.411634+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":89,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:11.979362+0000","last_change":"2024-03-11T20:52:51.931398+0000","last_active":"2024-03-11T20:53:11.979362+0000","last_peered":"2024-03-11T20:53:11.979362+0000","last_clean":"2024-03-11T20:53:11.979362+0000","last_became_active":"2024-03-11T20:50:32.194119+0000","last_became_peered":"2024-03-11T20:50:32.194119+0000","last_unstale":"2024-03-11T20:53:11.979362+0000","last_undegraded":"2024-03-11T20:53:11.979362+0000","last_fullsized":"2024-03-11T20:53:11.979362+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:51.931295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:52:51.931295+0000","last_clean_scrub_stamp":"2024-03-11T20:52:51.931295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:55.241599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":86,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:11.838741+0000","last_change":"2024-03-11T20:53:03.762967+0000","last_active":"2024-03-11T20:53:11.838741+0000","last_peered":"2024-03-11T20:53:11.838741+0000","last_clean":"2024-03-11T20:53:11.838741+0000","last_became_active":"2024-03-11T20:50:34.337173+0000","last_became_peered":"2024-03-11T20:50:34.337173+0000","last_unstale":"2024-03-11T20:53:11.838741+0000","last_undegraded":"2024-03-11T20:53:11.838741+0000","last_fullsized":"2024-03-11T20:53:11.838741+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:03.762882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:53:03.762882+0000","last_clean_scrub_stamp":"2024-03-11T20:53:03.762882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:29.931672+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":98,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:11.097712+0000","last_change":"2024-03-11T20:53:00.897188+0000","last_active":"2024-03-11T20:53:11.097712+0000","last_peered":"2024-03-11T20:53:11.097712+0000","last_clean":"2024-03-11T20:53:11.097712+0000","last_became_active":"2024-03-11T20:50:28.058942+0000","last_became_peered":"2024-03-11T20:50:28.058942+0000","last_unstale":"2024-03-11T20:53:11.097712+0000","last_undegraded":"2024-03-11T20:53:11.097712+0000","last_fullsized":"2024-03-11T20:53:11.097712+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:00.896807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:53:00.896807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:13.585784+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"84'15","reported_seq":119,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:11.101902+0000","last_change":"2024-03-11T20:53:09.345898+0000","last_active":"2024-03-11T20:53:11.101902+0000","last_peered":"2024-03-11T20:53:11.101902+0000","last_clean":"2024-03-11T20:53:11.101902+0000","last_became_active":"2024-03-11T20:50:30.101512+0000","last_became_peered":"2024-03-11T20:50:30.101512+0000","last_unstale":"2024-03-11T20:53:11.101902+0000","last_undegraded":"2024-03-11T20:53:11.101902+0000","last_fullsized":"2024-03-11T20:53:11.101902+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15","last_scrub_stamp":"2024-03-11T20:53:09.345405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:53:09.345405+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:28.384531+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":82,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-11T20:53:11.837417+0000","last_change":"2024-03-11T20:52:05.877747+0000","last_active":"2024-03-11T20:53:11.837417+0000","last_peered":"2024-03-11T20:53:11.837417+0000","last_clean":"2024-03-11T20:53:11.837417+0000","last_became_active":"2024-03-11T20:50:32.652854+0000","last_became_peered":"2024-03-11T20:50:32.652854+0000","last_unstale":"2024-03-11T20:53:11.837417+0000","last_undegraded":"2024-03-11T20:53:11.837417+0000","last_fullsized":"2024-03-11T20:53:11.837417+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:05.877396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:52:05.877396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:32.105270+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":148,"num_read_kb":143,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":93,"ondisk_log_size":93,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":701,"num_read_kb":458,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":428,"ondisk_log_size":428,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":68,"seq":292057776173,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32572,"kb_used_data":1396,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712580,"statfs":{"total":95995035648,"available":95961681920,"internally_reserved":0,"allocated":1429504,"data_stored":1000893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":57,"seq":244813135927,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32556,"kb_used_data":1380,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712596,"statfs":{"total":95995035648,"available":95961698304,"internally_reserved":0,"allocated":1413120,"data_stored":1000903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574954,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31984,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713168,"statfs":{"total":95995035648,"available":95962284032,"internally_reserved":0,"allocated":827392,"data_stored":410781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738485,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32564,"kb_used_data":1388,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712588,"statfs":{"total":95995035648,"available":95961690112,"internally_reserved":0,"allocated":1421312,"data_stored":999835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378720,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31968,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713184,"statfs":{"total":95995035648,"available":95962300416,"internally_reserved":0,"allocated":811008,"data_stored":409629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":28,"seq":120259084373,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27812,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717340,"statfs":{"total":95995035648,"available":95966556160,"internally_reserved":0,"allocated":815104,"data_stored":410005,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":35,"seq":150323855436,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32012,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713140,"statfs":{"total":95995035648,"available":95962255360,"internally_reserved":0,"allocated":851968,"data_stored":411395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":44,"seq":188978561089,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31972,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713180,"statfs":{"total":95995035648,"available":95962296320,"internally_reserved":0,"allocated":815104,"data_stored":409551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T20:53:22.816 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-11T20:53:22.816 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-11T20:53:22.816 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-11T20:53:22.816 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph health --format=json 2024-03-11T20:53:23.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:23 smithi146 bash[23809]: cluster 2024-03-11T20:53:21.859245+0000 mgr.y (mgr.24380) 99 : cluster 0 pgmap v68: 132 pgs: 132 active+clean; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 951 B/s rd, 0 op/s 2024-03-11T20:53:23.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:23 smithi146 bash[23809]: audit 2024-03-11T20:53:22.046131+0000 mgr.y (mgr.24380) 100 : audit 0 from='client.14661 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T20:53:23.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:23 smithi146 bash[23809]: cluster 2024-03-11T20:53:22.054536+0000 osd.0 (osd.0) 73 : cluster 0 6.10 scrub starts 2024-03-11T20:53:23.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:23 smithi146 bash[23809]: cluster 2024-03-11T20:53:22.056800+0000 osd.0 (osd.0) 74 : cluster 0 6.10 scrub ok 2024-03-11T20:53:23.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:23 smithi146 bash[23809]: cluster 2024-03-11T20:53:22.400058+0000 osd.4 (osd.4) 65 : cluster 0 3.9 scrub starts 2024-03-11T20:53:23.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:23 smithi146 bash[23809]: cluster 2024-03-11T20:53:22.404117+0000 osd.4 (osd.4) 66 : cluster 0 3.9 scrub ok 2024-03-11T20:53:23.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:23 smithi023 bash[21372]: cluster 2024-03-11T20:53:21.859245+0000 mgr.y (mgr.24380) 99 : cluster 0 pgmap v68: 132 pgs: 132 active+clean; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 951 B/s rd, 0 op/s 2024-03-11T20:53:23.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:23 smithi023 bash[21372]: audit 2024-03-11T20:53:22.046131+0000 mgr.y (mgr.24380) 100 : audit 0 from='client.14661 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T20:53:23.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:23 smithi023 bash[21372]: cluster 2024-03-11T20:53:22.054536+0000 osd.0 (osd.0) 73 : cluster 0 6.10 scrub starts 2024-03-11T20:53:23.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:23 smithi023 bash[21372]: cluster 2024-03-11T20:53:22.056800+0000 osd.0 (osd.0) 74 : cluster 0 6.10 scrub ok 2024-03-11T20:53:23.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:23 smithi023 bash[21372]: cluster 2024-03-11T20:53:22.400058+0000 osd.4 (osd.4) 65 : cluster 0 3.9 scrub starts 2024-03-11T20:53:23.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:23 smithi023 bash[21372]: cluster 2024-03-11T20:53:22.404117+0000 osd.4 (osd.4) 66 : cluster 0 3.9 scrub ok 2024-03-11T20:53:23.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:23 smithi023 bash[29146]: cluster 2024-03-11T20:53:21.859245+0000 mgr.y (mgr.24380) 99 : cluster 0 pgmap v68: 132 pgs: 132 active+clean; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 951 B/s rd, 0 op/s 2024-03-11T20:53:23.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:23 smithi023 bash[29146]: audit 2024-03-11T20:53:22.046131+0000 mgr.y (mgr.24380) 100 : audit 0 from='client.14661 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T20:53:23.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:23 smithi023 bash[29146]: cluster 2024-03-11T20:53:22.054536+0000 osd.0 (osd.0) 73 : cluster 0 6.10 scrub starts 2024-03-11T20:53:23.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:23 smithi023 bash[29146]: cluster 2024-03-11T20:53:22.056800+0000 osd.0 (osd.0) 74 : cluster 0 6.10 scrub ok 2024-03-11T20:53:23.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:23 smithi023 bash[29146]: cluster 2024-03-11T20:53:22.400058+0000 osd.4 (osd.4) 65 : cluster 0 3.9 scrub starts 2024-03-11T20:53:23.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:23 smithi023 bash[29146]: cluster 2024-03-11T20:53:22.404117+0000 osd.4 (osd.4) 66 : cluster 0 3.9 scrub ok 2024-03-11T20:53:24.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:24 smithi146 bash[23809]: cluster 2024-03-11T20:53:23.176897+0000 osd.1 (osd.1) 65 : cluster 0 6.4 scrub starts 2024-03-11T20:53:24.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:24 smithi146 bash[23809]: cluster 2024-03-11T20:53:23.179495+0000 osd.1 (osd.1) 66 : cluster 0 6.4 scrub ok 2024-03-11T20:53:24.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:24 smithi023 bash[21372]: cluster 2024-03-11T20:53:23.176897+0000 osd.1 (osd.1) 65 : cluster 0 6.4 scrub starts 2024-03-11T20:53:24.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:24 smithi023 bash[21372]: cluster 2024-03-11T20:53:23.179495+0000 osd.1 (osd.1) 66 : cluster 0 6.4 scrub ok 2024-03-11T20:53:24.534 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:24 smithi023 bash[29146]: cluster 2024-03-11T20:53:23.176897+0000 osd.1 (osd.1) 65 : cluster 0 6.4 scrub starts 2024-03-11T20:53:24.534 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:24 smithi023 bash[29146]: cluster 2024-03-11T20:53:23.179495+0000 osd.1 (osd.1) 66 : cluster 0 6.4 scrub ok 2024-03-11T20:53:25.180 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 20:53:24 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T20:53:25.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:25 smithi146 bash[23809]: cluster 2024-03-11T20:53:23.343517+0000 osd.3 (osd.3) 83 : cluster 0 3.18 scrub starts 2024-03-11T20:53:25.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:25 smithi146 bash[23809]: cluster 2024-03-11T20:53:23.347520+0000 osd.3 (osd.3) 84 : cluster 0 3.18 scrub ok 2024-03-11T20:53:25.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:25 smithi146 bash[23809]: cluster 2024-03-11T20:53:23.859884+0000 mgr.y (mgr.24380) 101 : cluster 0 pgmap v69: 132 pgs: 132 active+clean; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T20:53:25.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:25 smithi146 bash[23809]: cluster 2024-03-11T20:53:24.207365+0000 osd.1 (osd.1) 67 : cluster 0 4.12 scrub starts 2024-03-11T20:53:25.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:25 smithi146 bash[23809]: cluster 2024-03-11T20:53:24.209737+0000 osd.1 (osd.1) 68 : cluster 0 4.12 scrub ok 2024-03-11T20:53:25.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:25 smithi146 bash[23809]: cluster 2024-03-11T20:53:24.396551+0000 osd.4 (osd.4) 67 : cluster 0 purged_snaps scrub starts 2024-03-11T20:53:25.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:25 smithi146 bash[23809]: cluster 2024-03-11T20:53:24.396664+0000 osd.4 (osd.4) 68 : cluster 0 purged_snaps scrub ok 2024-03-11T20:53:25.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:25 smithi023 bash[21372]: cluster 2024-03-11T20:53:23.343517+0000 osd.3 (osd.3) 83 : cluster 0 3.18 scrub starts 2024-03-11T20:53:25.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:25 smithi023 bash[21372]: cluster 2024-03-11T20:53:23.347520+0000 osd.3 (osd.3) 84 : cluster 0 3.18 scrub ok 2024-03-11T20:53:25.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:25 smithi023 bash[21372]: cluster 2024-03-11T20:53:23.859884+0000 mgr.y (mgr.24380) 101 : cluster 0 pgmap v69: 132 pgs: 132 active+clean; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T20:53:25.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:25 smithi023 bash[21372]: cluster 2024-03-11T20:53:24.207365+0000 osd.1 (osd.1) 67 : cluster 0 4.12 scrub starts 2024-03-11T20:53:25.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:25 smithi023 bash[21372]: cluster 2024-03-11T20:53:24.209737+0000 osd.1 (osd.1) 68 : cluster 0 4.12 scrub ok 2024-03-11T20:53:25.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:25 smithi023 bash[21372]: cluster 2024-03-11T20:53:24.396551+0000 osd.4 (osd.4) 67 : cluster 0 purged_snaps scrub starts 2024-03-11T20:53:25.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:25 smithi023 bash[21372]: cluster 2024-03-11T20:53:24.396664+0000 osd.4 (osd.4) 68 : cluster 0 purged_snaps scrub ok 2024-03-11T20:53:25.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:25 smithi023 bash[29146]: cluster 2024-03-11T20:53:23.343517+0000 osd.3 (osd.3) 83 : cluster 0 3.18 scrub starts 2024-03-11T20:53:25.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:25 smithi023 bash[29146]: cluster 2024-03-11T20:53:23.347520+0000 osd.3 (osd.3) 84 : cluster 0 3.18 scrub ok 2024-03-11T20:53:25.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:25 smithi023 bash[29146]: cluster 2024-03-11T20:53:23.859884+0000 mgr.y (mgr.24380) 101 : cluster 0 pgmap v69: 132 pgs: 132 active+clean; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T20:53:25.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:25 smithi023 bash[29146]: cluster 2024-03-11T20:53:24.207365+0000 osd.1 (osd.1) 67 : cluster 0 4.12 scrub starts 2024-03-11T20:53:25.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:25 smithi023 bash[29146]: cluster 2024-03-11T20:53:24.209737+0000 osd.1 (osd.1) 68 : cluster 0 4.12 scrub ok 2024-03-11T20:53:25.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:25 smithi023 bash[29146]: cluster 2024-03-11T20:53:24.396551+0000 osd.4 (osd.4) 67 : cluster 0 purged_snaps scrub starts 2024-03-11T20:53:25.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:25 smithi023 bash[29146]: cluster 2024-03-11T20:53:24.396664+0000 osd.4 (osd.4) 68 : cluster 0 purged_snaps scrub ok 2024-03-11T20:53:26.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:26 smithi146 bash[23809]: cluster 2024-03-11T20:53:24.897985+0000 osd.6 (osd.6) 41 : cluster 0 2.11 scrub starts 2024-03-11T20:53:26.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:26 smithi146 bash[23809]: cluster 2024-03-11T20:53:24.900902+0000 osd.6 (osd.6) 42 : cluster 0 2.11 scrub ok 2024-03-11T20:53:26.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:26 smithi146 bash[23809]: audit 2024-03-11T20:53:24.909906+0000 mgr.y (mgr.24380) 102 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:53:26.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:26 smithi146 bash[23809]: cluster 2024-03-11T20:53:25.201686+0000 mon.a (mon.0) 912 : cluster 0 osdmap e113: 8 total, 8 up, 8 in 2024-03-11T20:53:26.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:26 smithi146 bash[23809]: cluster 2024-03-11T20:53:25.468545+0000 osd.6 (osd.6) 43 : cluster 0 purged_snaps scrub starts 2024-03-11T20:53:26.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:26 smithi146 bash[23809]: cluster 2024-03-11T20:53:25.468656+0000 osd.6 (osd.6) 44 : cluster 0 purged_snaps scrub ok 2024-03-11T20:53:26.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:26 smithi023 bash[21372]: cluster 2024-03-11T20:53:24.897985+0000 osd.6 (osd.6) 41 : cluster 0 2.11 scrub starts 2024-03-11T20:53:26.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:26 smithi023 bash[21372]: cluster 2024-03-11T20:53:24.900902+0000 osd.6 (osd.6) 42 : cluster 0 2.11 scrub ok 2024-03-11T20:53:26.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:26 smithi023 bash[21372]: audit 2024-03-11T20:53:24.909906+0000 mgr.y (mgr.24380) 102 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:53:26.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:26 smithi023 bash[21372]: cluster 2024-03-11T20:53:25.201686+0000 mon.a (mon.0) 912 : cluster 0 osdmap e113: 8 total, 8 up, 8 in 2024-03-11T20:53:26.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:26 smithi023 bash[21372]: cluster 2024-03-11T20:53:25.468545+0000 osd.6 (osd.6) 43 : cluster 0 purged_snaps scrub starts 2024-03-11T20:53:26.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:26 smithi023 bash[21372]: cluster 2024-03-11T20:53:25.468656+0000 osd.6 (osd.6) 44 : cluster 0 purged_snaps scrub ok 2024-03-11T20:53:26.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:26 smithi023 bash[29146]: cluster 2024-03-11T20:53:24.897985+0000 osd.6 (osd.6) 41 : cluster 0 2.11 scrub starts 2024-03-11T20:53:26.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:26 smithi023 bash[29146]: cluster 2024-03-11T20:53:24.900902+0000 osd.6 (osd.6) 42 : cluster 0 2.11 scrub ok 2024-03-11T20:53:26.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:26 smithi023 bash[29146]: audit 2024-03-11T20:53:24.909906+0000 mgr.y (mgr.24380) 102 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:53:26.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:26 smithi023 bash[29146]: cluster 2024-03-11T20:53:25.201686+0000 mon.a (mon.0) 912 : cluster 0 osdmap e113: 8 total, 8 up, 8 in 2024-03-11T20:53:26.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:26 smithi023 bash[29146]: cluster 2024-03-11T20:53:25.468545+0000 osd.6 (osd.6) 43 : cluster 0 purged_snaps scrub starts 2024-03-11T20:53:26.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:26 smithi023 bash[29146]: cluster 2024-03-11T20:53:25.468656+0000 osd.6 (osd.6) 44 : cluster 0 purged_snaps scrub ok 2024-03-11T20:53:27.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:27 smithi146 bash[23809]: cluster 2024-03-11T20:53:25.860585+0000 mgr.y (mgr.24380) 103 : cluster 0 pgmap v71: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:53:27.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:27 smithi146 bash[23809]: cluster 2024-03-11T20:53:25.897081+0000 osd.6 (osd.6) 45 : cluster 0 5.2 scrub starts 2024-03-11T20:53:27.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:27 smithi146 bash[23809]: cluster 2024-03-11T20:53:25.900030+0000 osd.6 (osd.6) 46 : cluster 0 5.2 scrub ok 2024-03-11T20:53:27.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:27 smithi146 bash[23809]: cluster 2024-03-11T20:53:26.154852+0000 osd.1 (osd.1) 69 : cluster 0 2.6 scrub starts 2024-03-11T20:53:27.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:27 smithi146 bash[23809]: cluster 2024-03-11T20:53:26.158247+0000 osd.1 (osd.1) 70 : cluster 0 2.6 scrub ok 2024-03-11T20:53:27.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:27 smithi146 bash[23809]: cluster 2024-03-11T20:53:26.207740+0000 mon.a (mon.0) 913 : cluster 0 osdmap e114: 8 total, 8 up, 8 in 2024-03-11T20:53:27.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:27 smithi146 bash[23809]: cluster 2024-03-11T20:53:26.442202+0000 osd.4 (osd.4) 69 : cluster 0 6.1e scrub starts 2024-03-11T20:53:27.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:27 smithi146 bash[23809]: cluster 2024-03-11T20:53:26.444517+0000 osd.4 (osd.4) 70 : cluster 0 6.1e scrub ok 2024-03-11T20:53:27.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:27 smithi023 bash[21372]: cluster 2024-03-11T20:53:25.860585+0000 mgr.y (mgr.24380) 103 : cluster 0 pgmap v71: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:53:27.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:27 smithi023 bash[21372]: cluster 2024-03-11T20:53:25.897081+0000 osd.6 (osd.6) 45 : cluster 0 5.2 scrub starts 2024-03-11T20:53:27.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:27 smithi023 bash[21372]: cluster 2024-03-11T20:53:25.900030+0000 osd.6 (osd.6) 46 : cluster 0 5.2 scrub ok 2024-03-11T20:53:27.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:27 smithi023 bash[21372]: cluster 2024-03-11T20:53:26.154852+0000 osd.1 (osd.1) 69 : cluster 0 2.6 scrub starts 2024-03-11T20:53:27.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:27 smithi023 bash[21372]: cluster 2024-03-11T20:53:26.158247+0000 osd.1 (osd.1) 70 : cluster 0 2.6 scrub ok 2024-03-11T20:53:27.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:27 smithi023 bash[21372]: cluster 2024-03-11T20:53:26.207740+0000 mon.a (mon.0) 913 : cluster 0 osdmap e114: 8 total, 8 up, 8 in 2024-03-11T20:53:27.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:27 smithi023 bash[21372]: cluster 2024-03-11T20:53:26.442202+0000 osd.4 (osd.4) 69 : cluster 0 6.1e scrub starts 2024-03-11T20:53:27.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:27 smithi023 bash[21372]: cluster 2024-03-11T20:53:26.444517+0000 osd.4 (osd.4) 70 : cluster 0 6.1e scrub ok 2024-03-11T20:53:27.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:27 smithi023 bash[29146]: cluster 2024-03-11T20:53:25.860585+0000 mgr.y (mgr.24380) 103 : cluster 0 pgmap v71: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:53:27.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:27 smithi023 bash[29146]: cluster 2024-03-11T20:53:25.897081+0000 osd.6 (osd.6) 45 : cluster 0 5.2 scrub starts 2024-03-11T20:53:27.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:27 smithi023 bash[29146]: cluster 2024-03-11T20:53:25.900030+0000 osd.6 (osd.6) 46 : cluster 0 5.2 scrub ok 2024-03-11T20:53:27.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:27 smithi023 bash[29146]: cluster 2024-03-11T20:53:26.154852+0000 osd.1 (osd.1) 69 : cluster 0 2.6 scrub starts 2024-03-11T20:53:27.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:27 smithi023 bash[29146]: cluster 2024-03-11T20:53:26.158247+0000 osd.1 (osd.1) 70 : cluster 0 2.6 scrub ok 2024-03-11T20:53:27.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:27 smithi023 bash[29146]: cluster 2024-03-11T20:53:26.207740+0000 mon.a (mon.0) 913 : cluster 0 osdmap e114: 8 total, 8 up, 8 in 2024-03-11T20:53:27.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:27 smithi023 bash[29146]: cluster 2024-03-11T20:53:26.442202+0000 osd.4 (osd.4) 69 : cluster 0 6.1e scrub starts 2024-03-11T20:53:27.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:27 smithi023 bash[29146]: cluster 2024-03-11T20:53:26.444517+0000 osd.4 (osd.4) 70 : cluster 0 6.1e scrub ok 2024-03-11T20:53:27.611 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:53:28.492 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:53:28 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:20:53:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T20:53:28.492 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:28 smithi023 bash[21372]: cluster 2024-03-11T20:53:27.048766+0000 osd.0 (osd.0) 75 : cluster 0 6.0 deep-scrub starts 2024-03-11T20:53:28.492 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:28 smithi023 bash[21372]: cluster 2024-03-11T20:53:27.053197+0000 osd.0 (osd.0) 76 : cluster 0 6.0 deep-scrub ok 2024-03-11T20:53:28.492 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:28 smithi023 bash[21372]: cluster 2024-03-11T20:53:27.417111+0000 osd.4 (osd.4) 71 : cluster 0 6.2 scrub starts 2024-03-11T20:53:28.492 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:28 smithi023 bash[21372]: cluster 2024-03-11T20:53:27.419509+0000 osd.4 (osd.4) 72 : cluster 0 6.2 scrub ok 2024-03-11T20:53:28.492 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:28 smithi023 bash[29146]: cluster 2024-03-11T20:53:27.048766+0000 osd.0 (osd.0) 75 : cluster 0 6.0 deep-scrub starts 2024-03-11T20:53:28.493 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:28 smithi023 bash[29146]: cluster 2024-03-11T20:53:27.053197+0000 osd.0 (osd.0) 76 : cluster 0 6.0 deep-scrub ok 2024-03-11T20:53:28.493 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:28 smithi023 bash[29146]: cluster 2024-03-11T20:53:27.417111+0000 osd.4 (osd.4) 71 : cluster 0 6.2 scrub starts 2024-03-11T20:53:28.493 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:28 smithi023 bash[29146]: cluster 2024-03-11T20:53:27.419509+0000 osd.4 (osd.4) 72 : cluster 0 6.2 scrub ok 2024-03-11T20:53:28.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:28 smithi146 bash[23809]: cluster 2024-03-11T20:53:27.048766+0000 osd.0 (osd.0) 75 : cluster 0 6.0 deep-scrub starts 2024-03-11T20:53:28.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:28 smithi146 bash[23809]: cluster 2024-03-11T20:53:27.053197+0000 osd.0 (osd.0) 76 : cluster 0 6.0 deep-scrub ok 2024-03-11T20:53:28.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:28 smithi146 bash[23809]: cluster 2024-03-11T20:53:27.417111+0000 osd.4 (osd.4) 71 : cluster 0 6.2 scrub starts 2024-03-11T20:53:28.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:28 smithi146 bash[23809]: cluster 2024-03-11T20:53:27.419509+0000 osd.4 (osd.4) 72 : cluster 0 6.2 scrub ok 2024-03-11T20:53:29.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:29 smithi146 bash[23809]: cluster 2024-03-11T20:53:27.382885+0000 osd.3 (osd.3) 85 : cluster 0 3.7 scrub starts 2024-03-11T20:53:29.546 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:29 smithi146 bash[23809]: cluster 2024-03-11T20:53:27.386960+0000 osd.3 (osd.3) 86 : cluster 0 3.7 scrub ok 2024-03-11T20:53:29.546 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:29 smithi146 bash[23809]: cluster 2024-03-11T20:53:27.861282+0000 mgr.y (mgr.24380) 104 : cluster 0 pgmap v73: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T20:53:29.547 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:29 smithi146 bash[23809]: cluster 2024-03-11T20:53:28.140094+0000 osd.1 (osd.1) 71 : cluster 0 4.4 scrub starts 2024-03-11T20:53:29.547 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:29 smithi146 bash[23809]: cluster 2024-03-11T20:53:28.142452+0000 osd.1 (osd.1) 72 : cluster 0 4.4 scrub ok 2024-03-11T20:53:29.547 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:29 smithi146 bash[23809]: cluster 2024-03-11T20:53:28.294915+0000 osd.7 (osd.7) 69 : cluster 0 2.8 scrub starts 2024-03-11T20:53:29.547 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:29 smithi146 bash[23809]: cluster 2024-03-11T20:53:28.297446+0000 osd.7 (osd.7) 70 : cluster 0 2.8 scrub ok 2024-03-11T20:53:29.547 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:29 smithi146 bash[23809]: cluster 2024-03-11T20:53:28.408207+0000 osd.4 (osd.4) 73 : cluster 0 2.f scrub starts 2024-03-11T20:53:29.547 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:29 smithi146 bash[23809]: cluster 2024-03-11T20:53:28.411543+0000 osd.4 (osd.4) 74 : cluster 0 2.f scrub ok 2024-03-11T20:53:29.547 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:29 smithi023 bash[29146]: cluster 2024-03-11T20:53:27.382885+0000 osd.3 (osd.3) 85 : cluster 0 3.7 scrub starts 2024-03-11T20:53:29.547 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:29 smithi023 bash[29146]: cluster 2024-03-11T20:53:27.386960+0000 osd.3 (osd.3) 86 : cluster 0 3.7 scrub ok 2024-03-11T20:53:29.548 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:29 smithi023 bash[29146]: cluster 2024-03-11T20:53:27.861282+0000 mgr.y (mgr.24380) 104 : cluster 0 pgmap v73: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T20:53:29.548 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:29 smithi023 bash[29146]: cluster 2024-03-11T20:53:28.140094+0000 osd.1 (osd.1) 71 : cluster 0 4.4 scrub starts 2024-03-11T20:53:29.548 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:29 smithi023 bash[29146]: cluster 2024-03-11T20:53:28.142452+0000 osd.1 (osd.1) 72 : cluster 0 4.4 scrub ok 2024-03-11T20:53:29.548 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:29 smithi023 bash[29146]: cluster 2024-03-11T20:53:28.294915+0000 osd.7 (osd.7) 69 : cluster 0 2.8 scrub starts 2024-03-11T20:53:29.548 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:29 smithi023 bash[29146]: cluster 2024-03-11T20:53:28.297446+0000 osd.7 (osd.7) 70 : cluster 0 2.8 scrub ok 2024-03-11T20:53:29.548 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:29 smithi023 bash[29146]: cluster 2024-03-11T20:53:28.408207+0000 osd.4 (osd.4) 73 : cluster 0 2.f scrub starts 2024-03-11T20:53:29.548 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:29 smithi023 bash[29146]: cluster 2024-03-11T20:53:28.411543+0000 osd.4 (osd.4) 74 : cluster 0 2.f scrub ok 2024-03-11T20:53:29.548 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:29 smithi023 bash[21372]: cluster 2024-03-11T20:53:27.382885+0000 osd.3 (osd.3) 85 : cluster 0 3.7 scrub starts 2024-03-11T20:53:29.548 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:29 smithi023 bash[21372]: cluster 2024-03-11T20:53:27.386960+0000 osd.3 (osd.3) 86 : cluster 0 3.7 scrub ok 2024-03-11T20:53:29.548 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:29 smithi023 bash[21372]: cluster 2024-03-11T20:53:27.861282+0000 mgr.y (mgr.24380) 104 : cluster 0 pgmap v73: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T20:53:29.548 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:29 smithi023 bash[21372]: cluster 2024-03-11T20:53:28.140094+0000 osd.1 (osd.1) 71 : cluster 0 4.4 scrub starts 2024-03-11T20:53:29.548 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:29 smithi023 bash[21372]: cluster 2024-03-11T20:53:28.142452+0000 osd.1 (osd.1) 72 : cluster 0 4.4 scrub ok 2024-03-11T20:53:29.549 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:29 smithi023 bash[21372]: cluster 2024-03-11T20:53:28.294915+0000 osd.7 (osd.7) 69 : cluster 0 2.8 scrub starts 2024-03-11T20:53:29.549 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:29 smithi023 bash[21372]: cluster 2024-03-11T20:53:28.297446+0000 osd.7 (osd.7) 70 : cluster 0 2.8 scrub ok 2024-03-11T20:53:29.549 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:29 smithi023 bash[21372]: cluster 2024-03-11T20:53:28.408207+0000 osd.4 (osd.4) 73 : cluster 0 2.f scrub starts 2024-03-11T20:53:29.549 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:29 smithi023 bash[21372]: cluster 2024-03-11T20:53:28.411543+0000 osd.4 (osd.4) 74 : cluster 0 2.f scrub ok 2024-03-11T20:53:30.470 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-11T20:53:30.470 INFO:teuthology.orchestra.run.smithi023.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-11T20:53:30.485 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:30 smithi023 bash[21372]: cluster 2024-03-11T20:53:28.356478+0000 osd.3 (osd.3) 87 : cluster 0 6.13 scrub starts 2024-03-11T20:53:30.485 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:30 smithi023 bash[21372]: cluster 2024-03-11T20:53:28.358129+0000 osd.3 (osd.3) 88 : cluster 0 6.13 scrub ok 2024-03-11T20:53:30.485 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:30 smithi023 bash[21372]: cluster 2024-03-11T20:53:29.320798+0000 osd.7 (osd.7) 71 : cluster 0 2.1d scrub starts 2024-03-11T20:53:30.485 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:30 smithi023 bash[21372]: cluster 2024-03-11T20:53:29.323263+0000 osd.7 (osd.7) 72 : cluster 0 2.1d scrub ok 2024-03-11T20:53:30.485 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:30 smithi023 bash[21372]: cluster 2024-03-11T20:53:29.962524+0000 osd.0 (osd.0) 77 : cluster 0 2.1f deep-scrub starts 2024-03-11T20:53:30.485 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:30 smithi023 bash[21372]: cluster 2024-03-11T20:53:29.964895+0000 osd.0 (osd.0) 78 : cluster 0 2.1f deep-scrub ok 2024-03-11T20:53:30.485 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:30 smithi023 bash[29146]: cluster 2024-03-11T20:53:28.356478+0000 osd.3 (osd.3) 87 : cluster 0 6.13 scrub starts 2024-03-11T20:53:30.485 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:30 smithi023 bash[29146]: cluster 2024-03-11T20:53:28.358129+0000 osd.3 (osd.3) 88 : cluster 0 6.13 scrub ok 2024-03-11T20:53:30.485 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:30 smithi023 bash[29146]: cluster 2024-03-11T20:53:29.320798+0000 osd.7 (osd.7) 71 : cluster 0 2.1d scrub starts 2024-03-11T20:53:30.485 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:30 smithi023 bash[29146]: cluster 2024-03-11T20:53:29.323263+0000 osd.7 (osd.7) 72 : cluster 0 2.1d scrub ok 2024-03-11T20:53:30.486 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:30 smithi023 bash[29146]: cluster 2024-03-11T20:53:29.962524+0000 osd.0 (osd.0) 77 : cluster 0 2.1f deep-scrub starts 2024-03-11T20:53:30.486 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:30 smithi023 bash[29146]: cluster 2024-03-11T20:53:29.964895+0000 osd.0 (osd.0) 78 : cluster 0 2.1f deep-scrub ok 2024-03-11T20:53:30.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:30 smithi146 bash[23809]: cluster 2024-03-11T20:53:28.356478+0000 osd.3 (osd.3) 87 : cluster 0 6.13 scrub starts 2024-03-11T20:53:30.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:30 smithi146 bash[23809]: cluster 2024-03-11T20:53:28.358129+0000 osd.3 (osd.3) 88 : cluster 0 6.13 scrub ok 2024-03-11T20:53:30.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:30 smithi146 bash[23809]: cluster 2024-03-11T20:53:29.320798+0000 osd.7 (osd.7) 71 : cluster 0 2.1d scrub starts 2024-03-11T20:53:30.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:30 smithi146 bash[23809]: cluster 2024-03-11T20:53:29.323263+0000 osd.7 (osd.7) 72 : cluster 0 2.1d scrub ok 2024-03-11T20:53:30.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:30 smithi146 bash[23809]: cluster 2024-03-11T20:53:29.962524+0000 osd.0 (osd.0) 77 : cluster 0 2.1f deep-scrub starts 2024-03-11T20:53:30.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:30 smithi146 bash[23809]: cluster 2024-03-11T20:53:29.964895+0000 osd.0 (osd.0) 78 : cluster 0 2.1f deep-scrub ok 2024-03-11T20:53:31.138 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-11T20:53:31.138 INFO:tasks.cephadm:Setup complete, yielding 2024-03-11T20:53:31.138 INFO:teuthology.run_tasks:Running task thrashosds... 2024-03-11T20:53:31.149 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-03-11T20:53:31.149 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client.0': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '2b0101eb74d60610d544cb8a473e23a35ac1cc0b'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'cephadm': {'cephadm_mode': 'root'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '2b0101eb74d60610d544cb8a473e23a35ac1cc0b'}}, 'workunit': {'branch': 'main', 'sha1': '1c349451176cc5b4ebfb24b22eaaa754e05cff6c'}} 2024-03-11T20:53:31.150 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-03-11T20:53:31.150 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-03-11T20:53:31.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:31 smithi146 bash[23809]: cluster 2024-03-11T20:53:29.812232+0000 osd.5 (osd.5) 67 : cluster 0 3.1c scrub starts 2024-03-11T20:53:31.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:31 smithi146 bash[23809]: cluster 2024-03-11T20:53:29.817096+0000 osd.5 (osd.5) 68 : cluster 0 3.1c scrub ok 2024-03-11T20:53:31.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:31 smithi146 bash[23809]: cluster 2024-03-11T20:53:29.862702+0000 mgr.y (mgr.24380) 105 : cluster 0 pgmap v74: 132 pgs: 132 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:53:31.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:31 smithi146 bash[23809]: cluster 2024-03-11T20:53:29.931443+0000 osd.6 (osd.6) 47 : cluster 0 4.a scrub starts 2024-03-11T20:53:31.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:31 smithi146 bash[23809]: cluster 2024-03-11T20:53:29.933626+0000 osd.6 (osd.6) 48 : cluster 0 4.a scrub ok 2024-03-11T20:53:31.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:31 smithi146 bash[23809]: cluster 2024-03-11T20:53:30.138038+0000 osd.1 (osd.1) 73 : cluster 0 2.d scrub starts 2024-03-11T20:53:31.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:31 smithi146 bash[23809]: cluster 2024-03-11T20:53:30.139535+0000 osd.1 (osd.1) 74 : cluster 0 2.d scrub ok 2024-03-11T20:53:31.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:31 smithi146 bash[23809]: audit 2024-03-11T20:53:30.474195+0000 mon.c (mon.2) 25 : audit 0 from='client.? 172.21.15.23:0/1401544547' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-11T20:53:31.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:31 smithi023 bash[21372]: cluster 2024-03-11T20:53:29.812232+0000 osd.5 (osd.5) 67 : cluster 0 3.1c scrub starts 2024-03-11T20:53:31.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:31 smithi023 bash[21372]: cluster 2024-03-11T20:53:29.817096+0000 osd.5 (osd.5) 68 : cluster 0 3.1c scrub ok 2024-03-11T20:53:31.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:31 smithi023 bash[21372]: cluster 2024-03-11T20:53:29.862702+0000 mgr.y (mgr.24380) 105 : cluster 0 pgmap v74: 132 pgs: 132 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:53:31.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:31 smithi023 bash[21372]: cluster 2024-03-11T20:53:29.931443+0000 osd.6 (osd.6) 47 : cluster 0 4.a scrub starts 2024-03-11T20:53:31.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:31 smithi023 bash[21372]: cluster 2024-03-11T20:53:29.933626+0000 osd.6 (osd.6) 48 : cluster 0 4.a scrub ok 2024-03-11T20:53:31.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:31 smithi023 bash[21372]: cluster 2024-03-11T20:53:30.138038+0000 osd.1 (osd.1) 73 : cluster 0 2.d scrub starts 2024-03-11T20:53:31.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:31 smithi023 bash[21372]: cluster 2024-03-11T20:53:30.139535+0000 osd.1 (osd.1) 74 : cluster 0 2.d scrub ok 2024-03-11T20:53:31.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:31 smithi023 bash[21372]: audit 2024-03-11T20:53:30.474195+0000 mon.c (mon.2) 25 : audit 0 from='client.? 172.21.15.23:0/1401544547' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-11T20:53:31.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:31 smithi023 bash[29146]: cluster 2024-03-11T20:53:29.812232+0000 osd.5 (osd.5) 67 : cluster 0 3.1c scrub starts 2024-03-11T20:53:31.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:31 smithi023 bash[29146]: cluster 2024-03-11T20:53:29.817096+0000 osd.5 (osd.5) 68 : cluster 0 3.1c scrub ok 2024-03-11T20:53:31.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:31 smithi023 bash[29146]: cluster 2024-03-11T20:53:29.862702+0000 mgr.y (mgr.24380) 105 : cluster 0 pgmap v74: 132 pgs: 132 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:53:31.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:31 smithi023 bash[29146]: cluster 2024-03-11T20:53:29.931443+0000 osd.6 (osd.6) 47 : cluster 0 4.a scrub starts 2024-03-11T20:53:31.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:31 smithi023 bash[29146]: cluster 2024-03-11T20:53:29.933626+0000 osd.6 (osd.6) 48 : cluster 0 4.a scrub ok 2024-03-11T20:53:31.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:31 smithi023 bash[29146]: cluster 2024-03-11T20:53:30.138038+0000 osd.1 (osd.1) 73 : cluster 0 2.d scrub starts 2024-03-11T20:53:31.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:31 smithi023 bash[29146]: cluster 2024-03-11T20:53:30.139535+0000 osd.1 (osd.1) 74 : cluster 0 2.d scrub ok 2024-03-11T20:53:31.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:31 smithi023 bash[29146]: audit 2024-03-11T20:53:30.474195+0000 mon.c (mon.2) 25 : audit 0 from='client.? 172.21.15.23:0/1401544547' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-11T20:53:32.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:32 smithi146 bash[23809]: cluster 2024-03-11T20:53:30.336893+0000 osd.3 (osd.3) 89 : cluster 0 2.1b scrub starts 2024-03-11T20:53:32.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:32 smithi146 bash[23809]: cluster 2024-03-11T20:53:30.340993+0000 osd.3 (osd.3) 90 : cluster 0 2.1b scrub ok 2024-03-11T20:53:32.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:32 smithi146 bash[23809]: cluster 2024-03-11T20:53:30.900835+0000 osd.6 (osd.6) 49 : cluster 0 4.1f deep-scrub starts 2024-03-11T20:53:32.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:32 smithi146 bash[23809]: cluster 2024-03-11T20:53:30.905779+0000 osd.6 (osd.6) 50 : cluster 0 4.1f deep-scrub ok 2024-03-11T20:53:32.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:32 smithi146 bash[23809]: cluster 2024-03-11T20:53:31.926281+0000 osd.0 (osd.0) 79 : cluster 0 3.1b scrub starts 2024-03-11T20:53:32.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:32 smithi146 bash[23809]: cluster 2024-03-11T20:53:31.929486+0000 osd.0 (osd.0) 80 : cluster 0 3.1b scrub ok 2024-03-11T20:53:32.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:32 smithi023 bash[21372]: cluster 2024-03-11T20:53:30.336893+0000 osd.3 (osd.3) 89 : cluster 0 2.1b scrub starts 2024-03-11T20:53:32.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:32 smithi023 bash[21372]: cluster 2024-03-11T20:53:30.340993+0000 osd.3 (osd.3) 90 : cluster 0 2.1b scrub ok 2024-03-11T20:53:32.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:32 smithi023 bash[21372]: cluster 2024-03-11T20:53:30.900835+0000 osd.6 (osd.6) 49 : cluster 0 4.1f deep-scrub starts 2024-03-11T20:53:32.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:32 smithi023 bash[21372]: cluster 2024-03-11T20:53:30.905779+0000 osd.6 (osd.6) 50 : cluster 0 4.1f deep-scrub ok 2024-03-11T20:53:32.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:32 smithi023 bash[21372]: cluster 2024-03-11T20:53:31.926281+0000 osd.0 (osd.0) 79 : cluster 0 3.1b scrub starts 2024-03-11T20:53:32.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:32 smithi023 bash[21372]: cluster 2024-03-11T20:53:31.929486+0000 osd.0 (osd.0) 80 : cluster 0 3.1b scrub ok 2024-03-11T20:53:32.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:32 smithi023 bash[29146]: cluster 2024-03-11T20:53:30.336893+0000 osd.3 (osd.3) 89 : cluster 0 2.1b scrub starts 2024-03-11T20:53:32.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:32 smithi023 bash[29146]: cluster 2024-03-11T20:53:30.340993+0000 osd.3 (osd.3) 90 : cluster 0 2.1b scrub ok 2024-03-11T20:53:32.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:32 smithi023 bash[29146]: cluster 2024-03-11T20:53:30.900835+0000 osd.6 (osd.6) 49 : cluster 0 4.1f deep-scrub starts 2024-03-11T20:53:32.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:32 smithi023 bash[29146]: cluster 2024-03-11T20:53:30.905779+0000 osd.6 (osd.6) 50 : cluster 0 4.1f deep-scrub ok 2024-03-11T20:53:32.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:32 smithi023 bash[29146]: cluster 2024-03-11T20:53:31.926281+0000 osd.0 (osd.0) 79 : cluster 0 3.1b scrub starts 2024-03-11T20:53:32.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:32 smithi023 bash[29146]: cluster 2024-03-11T20:53:31.929486+0000 osd.0 (osd.0) 80 : cluster 0 3.1b scrub ok 2024-03-11T20:53:33.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:33 smithi146 bash[23809]: cluster 2024-03-11T20:53:31.818485+0000 osd.5 (osd.5) 69 : cluster 0 2.18 scrub starts 2024-03-11T20:53:33.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:33 smithi146 bash[23809]: cluster 2024-03-11T20:53:31.821094+0000 osd.5 (osd.5) 70 : cluster 0 2.18 scrub ok 2024-03-11T20:53:33.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:33 smithi146 bash[23809]: cluster 2024-03-11T20:53:31.861326+0000 osd.6 (osd.6) 51 : cluster 0 2.7 scrub starts 2024-03-11T20:53:33.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:33 smithi146 bash[23809]: cluster 2024-03-11T20:53:31.863394+0000 mgr.y (mgr.24380) 106 : cluster 0 pgmap v75: 132 pgs: 132 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:53:33.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:33 smithi146 bash[23809]: cluster 2024-03-11T20:53:31.863732+0000 osd.6 (osd.6) 52 : cluster 0 2.7 scrub ok 2024-03-11T20:53:33.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:33 smithi146 bash[23809]: cluster 2024-03-11T20:53:32.163393+0000 osd.2 (osd.2) 49 : cluster 0 purged_snaps scrub starts 2024-03-11T20:53:33.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:33 smithi146 bash[23809]: cluster 2024-03-11T20:53:32.163517+0000 osd.2 (osd.2) 50 : cluster 0 purged_snaps scrub ok 2024-03-11T20:53:33.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:33 smithi146 bash[23809]: cluster 2024-03-11T20:53:32.282008+0000 mon.a (mon.0) 914 : cluster 0 osdmap e115: 8 total, 8 up, 8 in 2024-03-11T20:53:33.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:33 smithi146 bash[23809]: cluster 2024-03-11T20:53:32.527252+0000 osd.4 (osd.4) 75 : cluster 0 5.1 scrub starts 2024-03-11T20:53:33.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:33 smithi146 bash[23809]: cluster 2024-03-11T20:53:32.529822+0000 osd.4 (osd.4) 76 : cluster 0 5.1 scrub ok 2024-03-11T20:53:33.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:33 smithi146 bash[23809]: cluster 2024-03-11T20:53:32.938372+0000 osd.0 (osd.0) 81 : cluster 0 3.6 scrub starts 2024-03-11T20:53:33.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:33 smithi146 bash[23809]: cluster 2024-03-11T20:53:32.942966+0000 osd.0 (osd.0) 82 : cluster 0 3.6 scrub ok 2024-03-11T20:53:33.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:33 smithi023 bash[21372]: cluster 2024-03-11T20:53:31.818485+0000 osd.5 (osd.5) 69 : cluster 0 2.18 scrub starts 2024-03-11T20:53:33.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:33 smithi023 bash[21372]: cluster 2024-03-11T20:53:31.821094+0000 osd.5 (osd.5) 70 : cluster 0 2.18 scrub ok 2024-03-11T20:53:33.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:33 smithi023 bash[21372]: cluster 2024-03-11T20:53:31.861326+0000 osd.6 (osd.6) 51 : cluster 0 2.7 scrub starts 2024-03-11T20:53:33.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:33 smithi023 bash[21372]: cluster 2024-03-11T20:53:31.863394+0000 mgr.y (mgr.24380) 106 : cluster 0 pgmap v75: 132 pgs: 132 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:53:33.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:33 smithi023 bash[21372]: cluster 2024-03-11T20:53:31.863732+0000 osd.6 (osd.6) 52 : cluster 0 2.7 scrub ok 2024-03-11T20:53:33.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:33 smithi023 bash[21372]: cluster 2024-03-11T20:53:32.163393+0000 osd.2 (osd.2) 49 : cluster 0 purged_snaps scrub starts 2024-03-11T20:53:33.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:33 smithi023 bash[21372]: cluster 2024-03-11T20:53:32.163517+0000 osd.2 (osd.2) 50 : cluster 0 purged_snaps scrub ok 2024-03-11T20:53:33.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:33 smithi023 bash[21372]: cluster 2024-03-11T20:53:32.282008+0000 mon.a (mon.0) 914 : cluster 0 osdmap e115: 8 total, 8 up, 8 in 2024-03-11T20:53:33.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:33 smithi023 bash[21372]: cluster 2024-03-11T20:53:32.527252+0000 osd.4 (osd.4) 75 : cluster 0 5.1 scrub starts 2024-03-11T20:53:33.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:33 smithi023 bash[21372]: cluster 2024-03-11T20:53:32.529822+0000 osd.4 (osd.4) 76 : cluster 0 5.1 scrub ok 2024-03-11T20:53:33.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:33 smithi023 bash[21372]: cluster 2024-03-11T20:53:32.938372+0000 osd.0 (osd.0) 81 : cluster 0 3.6 scrub starts 2024-03-11T20:53:33.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:33 smithi023 bash[21372]: cluster 2024-03-11T20:53:32.942966+0000 osd.0 (osd.0) 82 : cluster 0 3.6 scrub ok 2024-03-11T20:53:33.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:33 smithi023 bash[29146]: cluster 2024-03-11T20:53:31.818485+0000 osd.5 (osd.5) 69 : cluster 0 2.18 scrub starts 2024-03-11T20:53:33.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:33 smithi023 bash[29146]: cluster 2024-03-11T20:53:31.821094+0000 osd.5 (osd.5) 70 : cluster 0 2.18 scrub ok 2024-03-11T20:53:33.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:33 smithi023 bash[29146]: cluster 2024-03-11T20:53:31.861326+0000 osd.6 (osd.6) 51 : cluster 0 2.7 scrub starts 2024-03-11T20:53:33.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:33 smithi023 bash[29146]: cluster 2024-03-11T20:53:31.863394+0000 mgr.y (mgr.24380) 106 : cluster 0 pgmap v75: 132 pgs: 132 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:53:33.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:33 smithi023 bash[29146]: cluster 2024-03-11T20:53:31.863732+0000 osd.6 (osd.6) 52 : cluster 0 2.7 scrub ok 2024-03-11T20:53:33.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:33 smithi023 bash[29146]: cluster 2024-03-11T20:53:32.163393+0000 osd.2 (osd.2) 49 : cluster 0 purged_snaps scrub starts 2024-03-11T20:53:33.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:33 smithi023 bash[29146]: cluster 2024-03-11T20:53:32.163517+0000 osd.2 (osd.2) 50 : cluster 0 purged_snaps scrub ok 2024-03-11T20:53:33.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:33 smithi023 bash[29146]: cluster 2024-03-11T20:53:32.282008+0000 mon.a (mon.0) 914 : cluster 0 osdmap e115: 8 total, 8 up, 8 in 2024-03-11T20:53:33.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:33 smithi023 bash[29146]: cluster 2024-03-11T20:53:32.527252+0000 osd.4 (osd.4) 75 : cluster 0 5.1 scrub starts 2024-03-11T20:53:33.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:33 smithi023 bash[29146]: cluster 2024-03-11T20:53:32.529822+0000 osd.4 (osd.4) 76 : cluster 0 5.1 scrub ok 2024-03-11T20:53:33.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:33 smithi023 bash[29146]: cluster 2024-03-11T20:53:32.938372+0000 osd.0 (osd.0) 81 : cluster 0 3.6 scrub starts 2024-03-11T20:53:33.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:33 smithi023 bash[29146]: cluster 2024-03-11T20:53:32.942966+0000 osd.0 (osd.0) 82 : cluster 0 3.6 scrub ok 2024-03-11T20:53:34.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:34 smithi023 bash[21372]: cluster 2024-03-11T20:53:33.222865+0000 osd.3 (osd.3) 91 : cluster 0 6.6 scrub starts 2024-03-11T20:53:34.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:34 smithi023 bash[21372]: cluster 2024-03-11T20:53:33.225286+0000 osd.3 (osd.3) 92 : cluster 0 6.6 scrub ok 2024-03-11T20:53:34.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:34 smithi023 bash[21372]: cluster 2024-03-11T20:53:33.553462+0000 osd.4 (osd.4) 77 : cluster 0 4.1 scrub starts 2024-03-11T20:53:34.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:34 smithi023 bash[21372]: cluster 2024-03-11T20:53:33.556051+0000 osd.4 (osd.4) 78 : cluster 0 4.1 scrub ok 2024-03-11T20:53:34.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:34 smithi023 bash[21372]: audit 2024-03-11T20:53:34.146172+0000 mon.a (mon.0) 915 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:53:34.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:34 smithi023 bash[29146]: cluster 2024-03-11T20:53:33.222865+0000 osd.3 (osd.3) 91 : cluster 0 6.6 scrub starts 2024-03-11T20:53:34.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:34 smithi023 bash[29146]: cluster 2024-03-11T20:53:33.225286+0000 osd.3 (osd.3) 92 : cluster 0 6.6 scrub ok 2024-03-11T20:53:34.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:34 smithi023 bash[29146]: cluster 2024-03-11T20:53:33.553462+0000 osd.4 (osd.4) 77 : cluster 0 4.1 scrub starts 2024-03-11T20:53:34.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:34 smithi023 bash[29146]: cluster 2024-03-11T20:53:33.556051+0000 osd.4 (osd.4) 78 : cluster 0 4.1 scrub ok 2024-03-11T20:53:34.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:34 smithi023 bash[29146]: audit 2024-03-11T20:53:34.146172+0000 mon.a (mon.0) 915 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:53:34.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:34 smithi146 bash[23809]: cluster 2024-03-11T20:53:33.222865+0000 osd.3 (osd.3) 91 : cluster 0 6.6 scrub starts 2024-03-11T20:53:34.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:34 smithi146 bash[23809]: cluster 2024-03-11T20:53:33.225286+0000 osd.3 (osd.3) 92 : cluster 0 6.6 scrub ok 2024-03-11T20:53:34.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:34 smithi146 bash[23809]: cluster 2024-03-11T20:53:33.553462+0000 osd.4 (osd.4) 77 : cluster 0 4.1 scrub starts 2024-03-11T20:53:34.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:34 smithi146 bash[23809]: cluster 2024-03-11T20:53:33.556051+0000 osd.4 (osd.4) 78 : cluster 0 4.1 scrub ok 2024-03-11T20:53:34.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:34 smithi146 bash[23809]: audit 2024-03-11T20:53:34.146172+0000 mon.a (mon.0) 915 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:53:35.261 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 20:53:34 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T20:53:35.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:35 smithi023 bash[21372]: cluster 2024-03-11T20:53:33.864155+0000 mgr.y (mgr.24380) 107 : cluster 0 pgmap v77: 132 pgs: 132 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T20:53:35.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:35 smithi023 bash[21372]: cluster 2024-03-11T20:53:34.193137+0000 osd.3 (osd.3) 93 : cluster 0 4.10 scrub starts 2024-03-11T20:53:35.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:35 smithi023 bash[21372]: cluster 2024-03-11T20:53:34.195668+0000 osd.3 (osd.3) 94 : cluster 0 4.10 scrub ok 2024-03-11T20:53:35.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:35 smithi023 bash[21372]: cluster 2024-03-11T20:53:34.345951+0000 osd.7 (osd.7) 73 : cluster 0 6.1c scrub starts 2024-03-11T20:53:35.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:35 smithi023 bash[21372]: cluster 2024-03-11T20:53:34.349618+0000 osd.7 (osd.7) 74 : cluster 0 6.1c scrub ok 2024-03-11T20:53:35.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:35 smithi023 bash[21372]: cluster 2024-03-11T20:53:34.542592+0000 osd.4 (osd.4) 79 : cluster 0 3.1a scrub starts 2024-03-11T20:53:35.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:35 smithi023 bash[21372]: cluster 2024-03-11T20:53:34.546574+0000 osd.4 (osd.4) 80 : cluster 0 3.1a scrub ok 2024-03-11T20:53:35.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:35 smithi023 bash[29146]: cluster 2024-03-11T20:53:33.864155+0000 mgr.y (mgr.24380) 107 : cluster 0 pgmap v77: 132 pgs: 132 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T20:53:35.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:35 smithi023 bash[29146]: cluster 2024-03-11T20:53:34.193137+0000 osd.3 (osd.3) 93 : cluster 0 4.10 scrub starts 2024-03-11T20:53:35.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:35 smithi023 bash[29146]: cluster 2024-03-11T20:53:34.195668+0000 osd.3 (osd.3) 94 : cluster 0 4.10 scrub ok 2024-03-11T20:53:35.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:35 smithi023 bash[29146]: cluster 2024-03-11T20:53:34.345951+0000 osd.7 (osd.7) 73 : cluster 0 6.1c scrub starts 2024-03-11T20:53:35.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:35 smithi023 bash[29146]: cluster 2024-03-11T20:53:34.349618+0000 osd.7 (osd.7) 74 : cluster 0 6.1c scrub ok 2024-03-11T20:53:35.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:35 smithi023 bash[29146]: cluster 2024-03-11T20:53:34.542592+0000 osd.4 (osd.4) 79 : cluster 0 3.1a scrub starts 2024-03-11T20:53:35.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:35 smithi023 bash[29146]: cluster 2024-03-11T20:53:34.546574+0000 osd.4 (osd.4) 80 : cluster 0 3.1a scrub ok 2024-03-11T20:53:35.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:35 smithi146 bash[23809]: cluster 2024-03-11T20:53:33.864155+0000 mgr.y (mgr.24380) 107 : cluster 0 pgmap v77: 132 pgs: 132 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T20:53:35.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:35 smithi146 bash[23809]: cluster 2024-03-11T20:53:34.193137+0000 osd.3 (osd.3) 93 : cluster 0 4.10 scrub starts 2024-03-11T20:53:35.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:35 smithi146 bash[23809]: cluster 2024-03-11T20:53:34.195668+0000 osd.3 (osd.3) 94 : cluster 0 4.10 scrub ok 2024-03-11T20:53:35.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:35 smithi146 bash[23809]: cluster 2024-03-11T20:53:34.345951+0000 osd.7 (osd.7) 73 : cluster 0 6.1c scrub starts 2024-03-11T20:53:35.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:35 smithi146 bash[23809]: cluster 2024-03-11T20:53:34.349618+0000 osd.7 (osd.7) 74 : cluster 0 6.1c scrub ok 2024-03-11T20:53:35.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:35 smithi146 bash[23809]: cluster 2024-03-11T20:53:34.542592+0000 osd.4 (osd.4) 79 : cluster 0 3.1a scrub starts 2024-03-11T20:53:35.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:35 smithi146 bash[23809]: cluster 2024-03-11T20:53:34.546574+0000 osd.4 (osd.4) 80 : cluster 0 3.1a scrub ok 2024-03-11T20:53:35.941 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:53:36.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:36 smithi146 bash[23809]: audit 2024-03-11T20:53:34.918431+0000 mgr.y (mgr.24380) 108 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:53:36.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:36 smithi146 bash[23809]: cluster 2024-03-11T20:53:35.136780+0000 osd.1 (osd.1) 75 : cluster 0 4.2 scrub starts 2024-03-11T20:53:36.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:36 smithi146 bash[23809]: cluster 2024-03-11T20:53:35.139335+0000 osd.1 (osd.1) 76 : cluster 0 4.2 scrub ok 2024-03-11T20:53:36.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:36 smithi146 bash[23809]: cluster 2024-03-11T20:53:35.169959+0000 osd.3 (osd.3) 95 : cluster 0 6.c scrub starts 2024-03-11T20:53:36.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:36 smithi146 bash[23809]: cluster 2024-03-11T20:53:35.172282+0000 osd.3 (osd.3) 96 : cluster 0 6.c scrub ok 2024-03-11T20:53:36.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:36 smithi146 bash[23809]: cluster 2024-03-11T20:53:35.958274+0000 osd.0 (osd.0) 83 : cluster 0 4.3 scrub starts 2024-03-11T20:53:36.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:36 smithi146 bash[23809]: cluster 2024-03-11T20:53:35.960565+0000 osd.0 (osd.0) 84 : cluster 0 4.3 scrub ok 2024-03-11T20:53:36.772 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:36 smithi023 bash[29146]: audit 2024-03-11T20:53:34.918431+0000 mgr.y (mgr.24380) 108 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:53:36.772 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:36 smithi023 bash[29146]: cluster 2024-03-11T20:53:35.136780+0000 osd.1 (osd.1) 75 : cluster 0 4.2 scrub starts 2024-03-11T20:53:36.772 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:36 smithi023 bash[29146]: cluster 2024-03-11T20:53:35.139335+0000 osd.1 (osd.1) 76 : cluster 0 4.2 scrub ok 2024-03-11T20:53:36.772 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:36 smithi023 bash[29146]: cluster 2024-03-11T20:53:35.169959+0000 osd.3 (osd.3) 95 : cluster 0 6.c scrub starts 2024-03-11T20:53:36.772 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:36 smithi023 bash[29146]: cluster 2024-03-11T20:53:35.172282+0000 osd.3 (osd.3) 96 : cluster 0 6.c scrub ok 2024-03-11T20:53:36.772 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:36 smithi023 bash[29146]: cluster 2024-03-11T20:53:35.958274+0000 osd.0 (osd.0) 83 : cluster 0 4.3 scrub starts 2024-03-11T20:53:36.772 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:36 smithi023 bash[29146]: cluster 2024-03-11T20:53:35.960565+0000 osd.0 (osd.0) 84 : cluster 0 4.3 scrub ok 2024-03-11T20:53:36.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:36 smithi023 bash[21372]: audit 2024-03-11T20:53:34.918431+0000 mgr.y (mgr.24380) 108 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:53:36.773 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:36 smithi023 bash[21372]: cluster 2024-03-11T20:53:35.136780+0000 osd.1 (osd.1) 75 : cluster 0 4.2 scrub starts 2024-03-11T20:53:36.773 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:36 smithi023 bash[21372]: cluster 2024-03-11T20:53:35.139335+0000 osd.1 (osd.1) 76 : cluster 0 4.2 scrub ok 2024-03-11T20:53:36.773 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:36 smithi023 bash[21372]: cluster 2024-03-11T20:53:35.169959+0000 osd.3 (osd.3) 95 : cluster 0 6.c scrub starts 2024-03-11T20:53:36.773 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:36 smithi023 bash[21372]: cluster 2024-03-11T20:53:35.172282+0000 osd.3 (osd.3) 96 : cluster 0 6.c scrub ok 2024-03-11T20:53:36.773 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:36 smithi023 bash[21372]: cluster 2024-03-11T20:53:35.958274+0000 osd.0 (osd.0) 83 : cluster 0 4.3 scrub starts 2024-03-11T20:53:36.773 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:36 smithi023 bash[21372]: cluster 2024-03-11T20:53:35.960565+0000 osd.0 (osd.0) 84 : cluster 0 4.3 scrub ok 2024-03-11T20:53:37.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:37 smithi023 bash[21372]: cluster 2024-03-11T20:53:35.865661+0000 mgr.y (mgr.24380) 109 : cluster 0 pgmap v78: 132 pgs: 132 active+clean; 582 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T20:53:37.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:37 smithi023 bash[21372]: cluster 2024-03-11T20:53:37.091471+0000 osd.1 (osd.1) 77 : cluster 0 2.a scrub starts 2024-03-11T20:53:37.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:37 smithi023 bash[21372]: cluster 2024-03-11T20:53:37.093953+0000 osd.1 (osd.1) 78 : cluster 0 2.a scrub ok 2024-03-11T20:53:37.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:37 smithi023 bash[29146]: cluster 2024-03-11T20:53:35.865661+0000 mgr.y (mgr.24380) 109 : cluster 0 pgmap v78: 132 pgs: 132 active+clean; 582 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T20:53:37.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:37 smithi023 bash[29146]: cluster 2024-03-11T20:53:37.091471+0000 osd.1 (osd.1) 77 : cluster 0 2.a scrub starts 2024-03-11T20:53:37.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:37 smithi023 bash[29146]: cluster 2024-03-11T20:53:37.093953+0000 osd.1 (osd.1) 78 : cluster 0 2.a scrub ok 2024-03-11T20:53:37.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:37 smithi146 bash[23809]: cluster 2024-03-11T20:53:35.865661+0000 mgr.y (mgr.24380) 109 : cluster 0 pgmap v78: 132 pgs: 132 active+clean; 582 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T20:53:37.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:37 smithi146 bash[23809]: cluster 2024-03-11T20:53:37.091471+0000 osd.1 (osd.1) 77 : cluster 0 2.a scrub starts 2024-03-11T20:53:37.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:37 smithi146 bash[23809]: cluster 2024-03-11T20:53:37.093953+0000 osd.1 (osd.1) 78 : cluster 0 2.a scrub ok 2024-03-11T20:53:38.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:38 smithi146 bash[23809]: cluster 2024-03-11T20:53:37.646388+0000 osd.5 (osd.5) 71 : cluster 0 purged_snaps scrub starts 2024-03-11T20:53:38.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:38 smithi146 bash[23809]: cluster 2024-03-11T20:53:37.646532+0000 osd.5 (osd.5) 72 : cluster 0 purged_snaps scrub ok 2024-03-11T20:53:38.786 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:38 smithi023 bash[21372]: cluster 2024-03-11T20:53:37.646388+0000 osd.5 (osd.5) 71 : cluster 0 purged_snaps scrub starts 2024-03-11T20:53:38.786 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:38 smithi023 bash[21372]: cluster 2024-03-11T20:53:37.646532+0000 osd.5 (osd.5) 72 : cluster 0 purged_snaps scrub ok 2024-03-11T20:53:38.786 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:53:38 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:20:53:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T20:53:38.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:38 smithi023 bash[29146]: cluster 2024-03-11T20:53:37.646388+0000 osd.5 (osd.5) 71 : cluster 0 purged_snaps scrub starts 2024-03-11T20:53:38.787 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:38 smithi023 bash[29146]: cluster 2024-03-11T20:53:37.646532+0000 osd.5 (osd.5) 72 : cluster 0 purged_snaps scrub ok 2024-03-11T20:53:39.284 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:53:38 smithi023 bash[21627]: [11/Mar/2024:20:53:38] ENGINE Bus STOPPING 2024-03-11T20:53:39.284 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:53:38 smithi023 bash[21627]: [11/Mar/2024:20:53:38] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-11T20:53:39.284 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:53:38 smithi023 bash[21627]: [11/Mar/2024:20:53:38] ENGINE Bus STOPPED 2024-03-11T20:53:39.284 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:53:38 smithi023 bash[21627]: [11/Mar/2024:20:53:38] ENGINE Bus STARTING 2024-03-11T20:53:39.284 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:53:38 smithi023 bash[21627]: [11/Mar/2024:20:53:38] ENGINE Serving on http://:::9283 2024-03-11T20:53:39.284 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:53:38 smithi023 bash[21627]: [11/Mar/2024:20:53:38] ENGINE Bus STARTED 2024-03-11T20:53:39.446 INFO:tasks.thrashosds:Beginning thrashosds... 2024-03-11T20:53:39.446 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-11T20:53:39.446 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T20:53:39.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:39 smithi146 bash[23809]: cluster 2024-03-11T20:53:37.866386+0000 mgr.y (mgr.24380) 110 : cluster 0 pgmap v79: 132 pgs: 132 active+clean; 582 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:53:39.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:39 smithi146 bash[23809]: cluster 2024-03-11T20:53:38.115055+0000 osd.1 (osd.1) 79 : cluster 0 2.4 scrub starts 2024-03-11T20:53:39.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:39 smithi146 bash[23809]: cluster 2024-03-11T20:53:38.117104+0000 osd.1 (osd.1) 80 : cluster 0 2.4 scrub ok 2024-03-11T20:53:39.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:39 smithi146 bash[23809]: cluster 2024-03-11T20:53:38.213215+0000 osd.3 (osd.3) 97 : cluster 0 6.1f scrub starts 2024-03-11T20:53:39.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:39 smithi146 bash[23809]: cluster 2024-03-11T20:53:38.215556+0000 osd.3 (osd.3) 98 : cluster 0 6.1f scrub ok 2024-03-11T20:53:39.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:39 smithi146 bash[23809]: cluster 2024-03-11T20:53:38.319467+0000 mon.a (mon.0) 916 : cluster 0 osdmap e116: 8 total, 8 up, 8 in 2024-03-11T20:53:39.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:39 smithi146 bash[23809]: audit 2024-03-11T20:53:38.778240+0000 mon.a (mon.0) 917 : audit 1 from='client.? ' entity='client.admin' 2024-03-11T20:53:39.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:39 smithi146 bash[23809]: audit 2024-03-11T20:53:38.845157+0000 mon.a (mon.0) 918 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:53:39.784 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:39 smithi023 bash[21372]: cluster 2024-03-11T20:53:37.866386+0000 mgr.y (mgr.24380) 110 : cluster 0 pgmap v79: 132 pgs: 132 active+clean; 582 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:53:39.784 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:39 smithi023 bash[21372]: cluster 2024-03-11T20:53:38.115055+0000 osd.1 (osd.1) 79 : cluster 0 2.4 scrub starts 2024-03-11T20:53:39.784 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:39 smithi023 bash[21372]: cluster 2024-03-11T20:53:38.117104+0000 osd.1 (osd.1) 80 : cluster 0 2.4 scrub ok 2024-03-11T20:53:39.784 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:39 smithi023 bash[21372]: cluster 2024-03-11T20:53:38.213215+0000 osd.3 (osd.3) 97 : cluster 0 6.1f scrub starts 2024-03-11T20:53:39.784 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:39 smithi023 bash[21372]: cluster 2024-03-11T20:53:38.215556+0000 osd.3 (osd.3) 98 : cluster 0 6.1f scrub ok 2024-03-11T20:53:39.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:39 smithi023 bash[21372]: cluster 2024-03-11T20:53:38.319467+0000 mon.a (mon.0) 916 : cluster 0 osdmap e116: 8 total, 8 up, 8 in 2024-03-11T20:53:39.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:39 smithi023 bash[21372]: audit 2024-03-11T20:53:38.778240+0000 mon.a (mon.0) 917 : audit 1 from='client.? ' entity='client.admin' 2024-03-11T20:53:39.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:39 smithi023 bash[21372]: audit 2024-03-11T20:53:38.845157+0000 mon.a (mon.0) 918 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:53:39.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:39 smithi023 bash[29146]: cluster 2024-03-11T20:53:37.866386+0000 mgr.y (mgr.24380) 110 : cluster 0 pgmap v79: 132 pgs: 132 active+clean; 582 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:53:39.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:39 smithi023 bash[29146]: cluster 2024-03-11T20:53:38.115055+0000 osd.1 (osd.1) 79 : cluster 0 2.4 scrub starts 2024-03-11T20:53:39.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:39 smithi023 bash[29146]: cluster 2024-03-11T20:53:38.117104+0000 osd.1 (osd.1) 80 : cluster 0 2.4 scrub ok 2024-03-11T20:53:39.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:39 smithi023 bash[29146]: cluster 2024-03-11T20:53:38.213215+0000 osd.3 (osd.3) 97 : cluster 0 6.1f scrub starts 2024-03-11T20:53:39.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:39 smithi023 bash[29146]: cluster 2024-03-11T20:53:38.215556+0000 osd.3 (osd.3) 98 : cluster 0 6.1f scrub ok 2024-03-11T20:53:39.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:39 smithi023 bash[29146]: cluster 2024-03-11T20:53:38.319467+0000 mon.a (mon.0) 916 : cluster 0 osdmap e116: 8 total, 8 up, 8 in 2024-03-11T20:53:39.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:39 smithi023 bash[29146]: audit 2024-03-11T20:53:38.778240+0000 mon.a (mon.0) 917 : audit 1 from='client.? ' entity='client.admin' 2024-03-11T20:53:39.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:39 smithi023 bash[29146]: audit 2024-03-11T20:53:38.845157+0000 mon.a (mon.0) 918 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:53:41.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:40 smithi146 bash[23809]: cluster 2024-03-11T20:53:39.287950+0000 osd.7 (osd.7) 75 : cluster 0 3.13 deep-scrub starts 2024-03-11T20:53:41.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:40 smithi146 bash[23809]: cluster 2024-03-11T20:53:39.292112+0000 osd.7 (osd.7) 76 : cluster 0 3.13 deep-scrub ok 2024-03-11T20:53:41.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:40 smithi146 bash[23809]: cluster 2024-03-11T20:53:39.867819+0000 mgr.y (mgr.24380) 111 : cluster 0 pgmap v81: 132 pgs: 132 active+clean; 582 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:53:41.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:40 smithi023 bash[21372]: cluster 2024-03-11T20:53:39.287950+0000 osd.7 (osd.7) 75 : cluster 0 3.13 deep-scrub starts 2024-03-11T20:53:41.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:40 smithi023 bash[21372]: cluster 2024-03-11T20:53:39.292112+0000 osd.7 (osd.7) 76 : cluster 0 3.13 deep-scrub ok 2024-03-11T20:53:41.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:40 smithi023 bash[21372]: cluster 2024-03-11T20:53:39.867819+0000 mgr.y (mgr.24380) 111 : cluster 0 pgmap v81: 132 pgs: 132 active+clean; 582 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:53:41.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:40 smithi023 bash[29146]: cluster 2024-03-11T20:53:39.287950+0000 osd.7 (osd.7) 75 : cluster 0 3.13 deep-scrub starts 2024-03-11T20:53:41.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:40 smithi023 bash[29146]: cluster 2024-03-11T20:53:39.292112+0000 osd.7 (osd.7) 76 : cluster 0 3.13 deep-scrub ok 2024-03-11T20:53:41.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:40 smithi023 bash[29146]: cluster 2024-03-11T20:53:39.867819+0000 mgr.y (mgr.24380) 111 : cluster 0 pgmap v81: 132 pgs: 132 active+clean; 582 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:53:42.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:41 smithi146 bash[23809]: cluster 2024-03-11T20:53:40.040721+0000 osd.0 (osd.0) 85 : cluster 0 6.16 scrub starts 2024-03-11T20:53:42.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:41 smithi146 bash[23809]: cluster 2024-03-11T20:53:40.043042+0000 osd.0 (osd.0) 86 : cluster 0 6.16 scrub ok 2024-03-11T20:53:42.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:41 smithi023 bash[21372]: cluster 2024-03-11T20:53:40.040721+0000 osd.0 (osd.0) 85 : cluster 0 6.16 scrub starts 2024-03-11T20:53:42.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:41 smithi023 bash[21372]: cluster 2024-03-11T20:53:40.043042+0000 osd.0 (osd.0) 86 : cluster 0 6.16 scrub ok 2024-03-11T20:53:42.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:41 smithi023 bash[29146]: cluster 2024-03-11T20:53:40.040721+0000 osd.0 (osd.0) 85 : cluster 0 6.16 scrub starts 2024-03-11T20:53:42.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:41 smithi023 bash[29146]: cluster 2024-03-11T20:53:40.043042+0000 osd.0 (osd.0) 86 : cluster 0 6.16 scrub ok 2024-03-11T20:53:43.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:42 smithi146 bash[23809]: cluster 2024-03-11T20:53:40.850914+0000 osd.5 (osd.5) 73 : cluster 0 6.7 scrub starts 2024-03-11T20:53:43.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:42 smithi146 bash[23809]: cluster 2024-03-11T20:53:40.853280+0000 osd.5 (osd.5) 74 : cluster 0 6.7 scrub ok 2024-03-11T20:53:43.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:42 smithi146 bash[23809]: cluster 2024-03-11T20:53:41.868358+0000 mgr.y (mgr.24380) 112 : cluster 0 pgmap v82: 132 pgs: 132 active+clean; 582 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T20:53:43.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:42 smithi023 bash[21372]: cluster 2024-03-11T20:53:40.850914+0000 osd.5 (osd.5) 73 : cluster 0 6.7 scrub starts 2024-03-11T20:53:43.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:42 smithi023 bash[21372]: cluster 2024-03-11T20:53:40.853280+0000 osd.5 (osd.5) 74 : cluster 0 6.7 scrub ok 2024-03-11T20:53:43.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:42 smithi023 bash[21372]: cluster 2024-03-11T20:53:41.868358+0000 mgr.y (mgr.24380) 112 : cluster 0 pgmap v82: 132 pgs: 132 active+clean; 582 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T20:53:43.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:42 smithi023 bash[29146]: cluster 2024-03-11T20:53:40.850914+0000 osd.5 (osd.5) 73 : cluster 0 6.7 scrub starts 2024-03-11T20:53:43.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:42 smithi023 bash[29146]: cluster 2024-03-11T20:53:40.853280+0000 osd.5 (osd.5) 74 : cluster 0 6.7 scrub ok 2024-03-11T20:53:43.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:42 smithi023 bash[29146]: cluster 2024-03-11T20:53:41.868358+0000 mgr.y (mgr.24380) 112 : cluster 0 pgmap v82: 132 pgs: 132 active+clean; 582 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T20:53:43.851 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:53:44.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:43 smithi146 bash[23809]: cluster 2024-03-11T20:53:42.294009+0000 osd.7 (osd.7) 77 : cluster 0 3.11 scrub starts 2024-03-11T20:53:44.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:43 smithi146 bash[23809]: cluster 2024-03-11T20:53:42.298297+0000 osd.7 (osd.7) 78 : cluster 0 3.11 scrub ok 2024-03-11T20:53:44.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:43 smithi146 bash[23809]: cluster 2024-03-11T20:53:42.561581+0000 osd.4 (osd.4) 81 : cluster 0 4.d scrub starts 2024-03-11T20:53:44.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:43 smithi146 bash[23809]: cluster 2024-03-11T20:53:42.564771+0000 osd.4 (osd.4) 82 : cluster 0 4.d scrub ok 2024-03-11T20:53:44.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:43 smithi023 bash[29146]: cluster 2024-03-11T20:53:42.294009+0000 osd.7 (osd.7) 77 : cluster 0 3.11 scrub starts 2024-03-11T20:53:44.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:43 smithi023 bash[29146]: cluster 2024-03-11T20:53:42.298297+0000 osd.7 (osd.7) 78 : cluster 0 3.11 scrub ok 2024-03-11T20:53:44.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:43 smithi023 bash[29146]: cluster 2024-03-11T20:53:42.561581+0000 osd.4 (osd.4) 81 : cluster 0 4.d scrub starts 2024-03-11T20:53:44.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:43 smithi023 bash[29146]: cluster 2024-03-11T20:53:42.564771+0000 osd.4 (osd.4) 82 : cluster 0 4.d scrub ok 2024-03-11T20:53:44.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:43 smithi023 bash[21372]: cluster 2024-03-11T20:53:42.294009+0000 osd.7 (osd.7) 77 : cluster 0 3.11 scrub starts 2024-03-11T20:53:44.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:43 smithi023 bash[21372]: cluster 2024-03-11T20:53:42.298297+0000 osd.7 (osd.7) 78 : cluster 0 3.11 scrub ok 2024-03-11T20:53:44.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:43 smithi023 bash[21372]: cluster 2024-03-11T20:53:42.561581+0000 osd.4 (osd.4) 81 : cluster 0 4.d scrub starts 2024-03-11T20:53:44.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:43 smithi023 bash[21372]: cluster 2024-03-11T20:53:42.564771+0000 osd.4 (osd.4) 82 : cluster 0 4.d scrub ok 2024-03-11T20:53:44.926 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:44 smithi146 bash[23809]: cluster 2024-03-11T20:53:42.784426+0000 osd.6 (osd.6) 53 : cluster 0 3.10 scrub starts 2024-03-11T20:53:44.926 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:44 smithi146 bash[23809]: cluster 2024-03-11T20:53:42.788143+0000 osd.6 (osd.6) 54 : cluster 0 3.10 scrub ok 2024-03-11T20:53:44.926 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:44 smithi146 bash[23809]: cluster 2024-03-11T20:53:43.269794+0000 osd.7 (osd.7) 79 : cluster 0 6.5 scrub starts 2024-03-11T20:53:44.926 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:44 smithi146 bash[23809]: cluster 2024-03-11T20:53:43.272057+0000 osd.7 (osd.7) 80 : cluster 0 6.5 scrub ok 2024-03-11T20:53:44.926 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:44 smithi146 bash[23809]: cluster 2024-03-11T20:53:43.868765+0000 mgr.y (mgr.24380) 113 : cluster 0 pgmap v83: 132 pgs: 132 active+clean; 582 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:53:44.926 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:44 smithi146 bash[23809]: audit 2024-03-11T20:53:44.478811+0000 mon.a (mon.0) 919 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:53:44.926 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:44 smithi146 bash[23809]: audit 2024-03-11T20:53:44.483812+0000 mon.a (mon.0) 920 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:53:44.951 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:44 smithi023 bash[21372]: cluster 2024-03-11T20:53:42.784426+0000 osd.6 (osd.6) 53 : cluster 0 3.10 scrub starts 2024-03-11T20:53:44.951 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:44 smithi023 bash[21372]: cluster 2024-03-11T20:53:42.788143+0000 osd.6 (osd.6) 54 : cluster 0 3.10 scrub ok 2024-03-11T20:53:44.951 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:44 smithi023 bash[21372]: cluster 2024-03-11T20:53:43.269794+0000 osd.7 (osd.7) 79 : cluster 0 6.5 scrub starts 2024-03-11T20:53:44.951 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:44 smithi023 bash[21372]: cluster 2024-03-11T20:53:43.272057+0000 osd.7 (osd.7) 80 : cluster 0 6.5 scrub ok 2024-03-11T20:53:44.952 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:44 smithi023 bash[21372]: cluster 2024-03-11T20:53:43.868765+0000 mgr.y (mgr.24380) 113 : cluster 0 pgmap v83: 132 pgs: 132 active+clean; 582 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:53:44.952 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:44 smithi023 bash[21372]: audit 2024-03-11T20:53:44.478811+0000 mon.a (mon.0) 919 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:53:44.952 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:44 smithi023 bash[21372]: audit 2024-03-11T20:53:44.483812+0000 mon.a (mon.0) 920 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:53:44.952 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:44 smithi023 bash[29146]: cluster 2024-03-11T20:53:42.784426+0000 osd.6 (osd.6) 53 : cluster 0 3.10 scrub starts 2024-03-11T20:53:44.952 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:44 smithi023 bash[29146]: cluster 2024-03-11T20:53:42.788143+0000 osd.6 (osd.6) 54 : cluster 0 3.10 scrub ok 2024-03-11T20:53:44.952 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:44 smithi023 bash[29146]: cluster 2024-03-11T20:53:43.269794+0000 osd.7 (osd.7) 79 : cluster 0 6.5 scrub starts 2024-03-11T20:53:44.952 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:44 smithi023 bash[29146]: cluster 2024-03-11T20:53:43.272057+0000 osd.7 (osd.7) 80 : cluster 0 6.5 scrub ok 2024-03-11T20:53:44.952 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:44 smithi023 bash[29146]: cluster 2024-03-11T20:53:43.868765+0000 mgr.y (mgr.24380) 113 : cluster 0 pgmap v83: 132 pgs: 132 active+clean; 582 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:53:44.952 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:44 smithi023 bash[29146]: audit 2024-03-11T20:53:44.478811+0000 mon.a (mon.0) 919 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:53:44.952 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:44 smithi023 bash[29146]: audit 2024-03-11T20:53:44.483812+0000 mon.a (mon.0) 920 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:53:45.260 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 20:53:44 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T20:53:46.121 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:45 smithi023 bash[21372]: cluster 2024-03-11T20:53:44.823128+0000 osd.5 (osd.5) 75 : cluster 0 4.8 scrub starts 2024-03-11T20:53:46.122 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:45 smithi023 bash[21372]: cluster 2024-03-11T20:53:44.824742+0000 osd.5 (osd.5) 76 : cluster 0 4.8 scrub ok 2024-03-11T20:53:46.122 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:45 smithi023 bash[21372]: audit 2024-03-11T20:53:44.833227+0000 mon.a (mon.0) 921 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:53:46.122 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:45 smithi023 bash[21372]: audit 2024-03-11T20:53:44.840547+0000 mon.a (mon.0) 922 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:53:46.122 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:45 smithi023 bash[21372]: audit 2024-03-11T20:53:44.929586+0000 mgr.y (mgr.24380) 114 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:53:46.122 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:45 smithi023 bash[21372]: cluster 2024-03-11T20:53:44.979253+0000 osd.0 (osd.0) 87 : cluster 0 3.1f scrub starts 2024-03-11T20:53:46.122 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:45 smithi023 bash[21372]: cluster 2024-03-11T20:53:44.981379+0000 osd.0 (osd.0) 88 : cluster 0 3.1f scrub ok 2024-03-11T20:53:46.122 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:45 smithi023 bash[21372]: audit 2024-03-11T20:53:45.115860+0000 mon.a (mon.0) 923 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:53:46.122 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:45 smithi023 bash[21372]: audit 2024-03-11T20:53:45.117700+0000 mon.a (mon.0) 924 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:53:46.122 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:45 smithi023 bash[21372]: audit 2024-03-11T20:53:45.127932+0000 mon.a (mon.0) 925 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:53:46.122 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:45 smithi023 bash[29146]: cluster 2024-03-11T20:53:44.823128+0000 osd.5 (osd.5) 75 : cluster 0 4.8 scrub starts 2024-03-11T20:53:46.123 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:45 smithi023 bash[29146]: cluster 2024-03-11T20:53:44.824742+0000 osd.5 (osd.5) 76 : cluster 0 4.8 scrub ok 2024-03-11T20:53:46.123 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:45 smithi023 bash[29146]: audit 2024-03-11T20:53:44.833227+0000 mon.a (mon.0) 921 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:53:46.123 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:45 smithi023 bash[29146]: audit 2024-03-11T20:53:44.840547+0000 mon.a (mon.0) 922 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:53:46.123 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:45 smithi023 bash[29146]: audit 2024-03-11T20:53:44.929586+0000 mgr.y (mgr.24380) 114 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:53:46.123 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:45 smithi023 bash[29146]: cluster 2024-03-11T20:53:44.979253+0000 osd.0 (osd.0) 87 : cluster 0 3.1f scrub starts 2024-03-11T20:53:46.123 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:45 smithi023 bash[29146]: cluster 2024-03-11T20:53:44.981379+0000 osd.0 (osd.0) 88 : cluster 0 3.1f scrub ok 2024-03-11T20:53:46.123 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:45 smithi023 bash[29146]: audit 2024-03-11T20:53:45.115860+0000 mon.a (mon.0) 923 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:53:46.123 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:45 smithi023 bash[29146]: audit 2024-03-11T20:53:45.117700+0000 mon.a (mon.0) 924 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:53:46.123 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:45 smithi023 bash[29146]: audit 2024-03-11T20:53:45.127932+0000 mon.a (mon.0) 925 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:53:46.263 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:45 smithi146 bash[23809]: cluster 2024-03-11T20:53:44.823128+0000 osd.5 (osd.5) 75 : cluster 0 4.8 scrub starts 2024-03-11T20:53:46.264 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:45 smithi146 bash[23809]: cluster 2024-03-11T20:53:44.824742+0000 osd.5 (osd.5) 76 : cluster 0 4.8 scrub ok 2024-03-11T20:53:46.264 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:45 smithi146 bash[23809]: audit 2024-03-11T20:53:44.833227+0000 mon.a (mon.0) 921 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:53:46.264 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:45 smithi146 bash[23809]: audit 2024-03-11T20:53:44.840547+0000 mon.a (mon.0) 922 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:53:46.264 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:45 smithi146 bash[23809]: audit 2024-03-11T20:53:44.929586+0000 mgr.y (mgr.24380) 114 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:53:46.264 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:45 smithi146 bash[23809]: cluster 2024-03-11T20:53:44.979253+0000 osd.0 (osd.0) 87 : cluster 0 3.1f scrub starts 2024-03-11T20:53:46.264 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:45 smithi146 bash[23809]: cluster 2024-03-11T20:53:44.981379+0000 osd.0 (osd.0) 88 : cluster 0 3.1f scrub ok 2024-03-11T20:53:46.264 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:45 smithi146 bash[23809]: audit 2024-03-11T20:53:45.115860+0000 mon.a (mon.0) 923 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:53:46.264 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:45 smithi146 bash[23809]: audit 2024-03-11T20:53:45.117700+0000 mon.a (mon.0) 924 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:53:46.264 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:45 smithi146 bash[23809]: audit 2024-03-11T20:53:45.127932+0000 mon.a (mon.0) 925 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:53:46.554 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-11T20:53:46.558 INFO:teuthology.orchestra.run.smithi023.stderr:dumped all 2024-03-11T20:53:47.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:46 smithi146 bash[23809]: cluster 2024-03-11T20:53:45.840520+0000 osd.5 (osd.5) 77 : cluster 0 3.5 scrub starts 2024-03-11T20:53:47.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:46 smithi146 bash[23809]: cluster 2024-03-11T20:53:45.846051+0000 osd.5 (osd.5) 78 : cluster 0 3.5 scrub ok 2024-03-11T20:53:47.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:46 smithi146 bash[23809]: cluster 2024-03-11T20:53:45.870182+0000 mgr.y (mgr.24380) 115 : cluster 0 pgmap v84: 132 pgs: 132 active+clean; 582 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:53:47.281 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:46 smithi023 bash[29146]: cluster 2024-03-11T20:53:45.840520+0000 osd.5 (osd.5) 77 : cluster 0 3.5 scrub starts 2024-03-11T20:53:47.281 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:46 smithi023 bash[29146]: cluster 2024-03-11T20:53:45.846051+0000 osd.5 (osd.5) 78 : cluster 0 3.5 scrub ok 2024-03-11T20:53:47.281 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:46 smithi023 bash[29146]: cluster 2024-03-11T20:53:45.870182+0000 mgr.y (mgr.24380) 115 : cluster 0 pgmap v84: 132 pgs: 132 active+clean; 582 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:53:47.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:46 smithi023 bash[21372]: cluster 2024-03-11T20:53:45.840520+0000 osd.5 (osd.5) 77 : cluster 0 3.5 scrub starts 2024-03-11T20:53:47.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:46 smithi023 bash[21372]: cluster 2024-03-11T20:53:45.846051+0000 osd.5 (osd.5) 78 : cluster 0 3.5 scrub ok 2024-03-11T20:53:47.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:46 smithi023 bash[21372]: cluster 2024-03-11T20:53:45.870182+0000 mgr.y (mgr.24380) 115 : cluster 0 pgmap v84: 132 pgs: 132 active+clean; 582 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:53:47.283 INFO:teuthology.orchestra.run.smithi023.stdout:{"pg_ready":true,"pg_map":{"version":84,"stamp":"2024-03-11T20:53:45.869006+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596392,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1123,"num_read_kb":969,"num_write":615,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":671,"ondisk_log_size":671,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":261188,"kb_used_data":7612,"kb_used_omap":12,"kb_used_meta":253491,"kb_avail":749700028,"statfs":{"total":767960285184,"available":767692828672,"internally_reserved":0,"allocated":7794688,"data_stored":4889216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":259575346},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.004521"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":94,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.329454+0000","last_change":"2024-03-11T20:53:05.510279+0000","last_active":"2024-03-11T20:53:38.329454+0000","last_peered":"2024-03-11T20:53:38.329454+0000","last_clean":"2024-03-11T20:53:38.329454+0000","last_became_active":"2024-03-11T20:50:34.338390+0000","last_became_peered":"2024-03-11T20:50:34.338390+0000","last_unstale":"2024-03-11T20:53:38.329454+0000","last_undegraded":"2024-03-11T20:53:38.329454+0000","last_fullsized":"2024-03-11T20:53:38.329454+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:05.510060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:53:05.510060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:21.840101+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":98,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.329173+0000","last_change":"2024-03-11T20:53:18.443416+0000","last_active":"2024-03-11T20:53:38.329173+0000","last_peered":"2024-03-11T20:53:38.329173+0000","last_clean":"2024-03-11T20:53:38.329173+0000","last_became_active":"2024-03-11T20:50:32.210194+0000","last_became_peered":"2024-03-11T20:50:32.210194+0000","last_unstale":"2024-03-11T20:53:38.329173+0000","last_undegraded":"2024-03-11T20:53:38.329173+0000","last_fullsized":"2024-03-11T20:53:38.329173+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:18.443215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:53:18.443215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:23.962891+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":105,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.325201+0000","last_change":"2024-03-11T20:53:29.965205+0000","last_active":"2024-03-11T20:53:38.325201+0000","last_peered":"2024-03-11T20:53:38.325201+0000","last_clean":"2024-03-11T20:53:38.325201+0000","last_became_active":"2024-03-11T20:50:28.061739+0000","last_became_peered":"2024-03-11T20:50:28.061739+0000","last_unstale":"2024-03-11T20:53:38.325201+0000","last_undegraded":"2024-03-11T20:53:38.325201+0000","last_fullsized":"2024-03-11T20:53:38.325201+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:29.964907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:53:29.964907+0000","last_clean_scrub_stamp":"2024-03-11T20:53:29.964907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:53.399727+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"84'10","reported_seq":117,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.331551+0000","last_change":"2024-03-11T20:53:11.430067+0000","last_active":"2024-03-11T20:53:38.331551+0000","last_peered":"2024-03-11T20:53:38.331551+0000","last_clean":"2024-03-11T20:53:38.331551+0000","last_became_active":"2024-03-11T20:50:30.101721+0000","last_became_peered":"2024-03-11T20:50:30.101721+0000","last_unstale":"2024-03-11T20:53:38.331551+0000","last_undegraded":"2024-03-11T20:53:38.331551+0000","last_fullsized":"2024-03-11T20:53:38.331551+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'10","last_scrub_stamp":"2024-03-11T20:53:11.429944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:53:11.429944+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:29.275642+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":94,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.322785+0000","last_change":"2024-03-11T20:53:07.560827+0000","last_active":"2024-03-11T20:53:38.322785+0000","last_peered":"2024-03-11T20:53:38.322785+0000","last_clean":"2024-03-11T20:53:38.322785+0000","last_became_active":"2024-03-11T20:50:34.318627+0000","last_became_peered":"2024-03-11T20:50:34.318627+0000","last_unstale":"2024-03-11T20:53:38.322785+0000","last_undegraded":"2024-03-11T20:53:38.322785+0000","last_fullsized":"2024-03-11T20:53:38.322785+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:07.560731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:55.545758+0000","last_clean_scrub_stamp":"2024-03-11T20:53:07.560731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:36.783905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":98,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.322569+0000","last_change":"2024-03-11T20:52:49.461204+0000","last_active":"2024-03-11T20:53:38.322569+0000","last_peered":"2024-03-11T20:53:38.322569+0000","last_clean":"2024-03-11T20:53:38.322569+0000","last_became_active":"2024-03-11T20:50:32.197413+0000","last_became_peered":"2024-03-11T20:50:32.197413+0000","last_unstale":"2024-03-11T20:53:38.322569+0000","last_undegraded":"2024-03-11T20:53:38.322569+0000","last_fullsized":"2024-03-11T20:53:38.322569+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:49.460993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:52:49.460993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:04.898406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":98,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.331025+0000","last_change":"2024-03-11T20:52:26.396960+0000","last_active":"2024-03-11T20:53:38.331025+0000","last_peered":"2024-03-11T20:53:38.331025+0000","last_clean":"2024-03-11T20:53:38.331025+0000","last_became_active":"2024-03-11T20:50:28.058274+0000","last_became_peered":"2024-03-11T20:50:28.058274+0000","last_unstale":"2024-03-11T20:53:38.331025+0000","last_undegraded":"2024-03-11T20:53:38.331025+0000","last_fullsized":"2024-03-11T20:53:38.331025+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:26.396865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:52:26.396865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:52.964932+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"84'11","reported_seq":112,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.325069+0000","last_change":"2024-03-11T20:52:15.868867+0000","last_active":"2024-03-11T20:53:38.325069+0000","last_peered":"2024-03-11T20:53:38.325069+0000","last_clean":"2024-03-11T20:53:38.325069+0000","last_became_active":"2024-03-11T20:50:30.092638+0000","last_became_peered":"2024-03-11T20:50:30.092638+0000","last_unstale":"2024-03-11T20:53:38.325069+0000","last_undegraded":"2024-03-11T20:53:38.325069+0000","last_fullsized":"2024-03-11T20:53:38.325069+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T20:52:15.868599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:52:15.868599+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:44.240121+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":94,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.835084+0000","last_change":"2024-03-11T20:53:07.792944+0000","last_active":"2024-03-11T20:53:38.835084+0000","last_peered":"2024-03-11T20:53:38.835084+0000","last_clean":"2024-03-11T20:53:38.835084+0000","last_became_active":"2024-03-11T20:50:34.334495+0000","last_became_peered":"2024-03-11T20:50:34.334495+0000","last_unstale":"2024-03-11T20:53:38.835084+0000","last_undegraded":"2024-03-11T20:53:38.835084+0000","last_fullsized":"2024-03-11T20:53:38.835084+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:07.792697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:53:07.792697+0000","last_clean_scrub_stamp":"2024-03-11T20:53:07.792697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:23.012728+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":90,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.322978+0000","last_change":"2024-03-11T20:52:03.583828+0000","last_active":"2024-03-11T20:53:38.322978+0000","last_peered":"2024-03-11T20:53:38.322978+0000","last_clean":"2024-03-11T20:53:38.322978+0000","last_became_active":"2024-03-11T20:50:32.199561+0000","last_became_peered":"2024-03-11T20:50:32.199561+0000","last_unstale":"2024-03-11T20:53:38.322978+0000","last_undegraded":"2024-03-11T20:53:38.322978+0000","last_fullsized":"2024-03-11T20:53:38.322978+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:03.583594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:52:03.583594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":106,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.336031+0000","last_change":"2024-03-11T20:53:29.323499+0000","last_active":"2024-03-11T20:53:38.336031+0000","last_peered":"2024-03-11T20:53:38.336031+0000","last_clean":"2024-03-11T20:53:38.336031+0000","last_became_active":"2024-03-11T20:50:28.079225+0000","last_became_peered":"2024-03-11T20:50:28.079225+0000","last_unstale":"2024-03-11T20:53:38.336031+0000","last_undegraded":"2024-03-11T20:53:38.336031+0000","last_fullsized":"2024-03-11T20:53:38.336031+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:29.323273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:53:29.323273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:36.362049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"84'15","reported_seq":127,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.837970+0000","last_change":"2024-03-11T20:53:29.817448+0000","last_active":"2024-03-11T20:53:38.837970+0000","last_peered":"2024-03-11T20:53:38.837970+0000","last_clean":"2024-03-11T20:53:38.837970+0000","last_became_active":"2024-03-11T20:50:30.105354+0000","last_became_peered":"2024-03-11T20:50:30.105354+0000","last_unstale":"2024-03-11T20:53:38.837970+0000","last_undegraded":"2024-03-11T20:53:38.837970+0000","last_fullsized":"2024-03-11T20:53:38.837970+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15","last_scrub_stamp":"2024-03-11T20:53:29.817112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:53:29.817112+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:47.069548+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":93,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.322556+0000","last_change":"2024-03-11T20:53:14.168525+0000","last_active":"2024-03-11T20:53:38.322556+0000","last_peered":"2024-03-11T20:53:38.322556+0000","last_clean":"2024-03-11T20:53:38.322556+0000","last_became_active":"2024-03-11T20:50:34.333230+0000","last_became_peered":"2024-03-11T20:50:34.333230+0000","last_unstale":"2024-03-11T20:53:38.322556+0000","last_undegraded":"2024-03-11T20:53:38.322556+0000","last_fullsized":"2024-03-11T20:53:38.322556+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:14.168304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:53:14.168304+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:19.290761+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":90,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.323746+0000","last_change":"2024-03-11T20:52:19.502063+0000","last_active":"2024-03-11T20:53:38.323746+0000","last_peered":"2024-03-11T20:53:38.323746+0000","last_clean":"2024-03-11T20:53:38.323746+0000","last_became_active":"2024-03-11T20:50:32.200126+0000","last_became_peered":"2024-03-11T20:50:32.200126+0000","last_unstale":"2024-03-11T20:53:38.323746+0000","last_undegraded":"2024-03-11T20:53:38.323746+0000","last_fullsized":"2024-03-11T20:53:38.323746+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:19.501832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:52:19.501832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":106,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.336128+0000","last_change":"2024-03-11T20:53:00.210016+0000","last_active":"2024-03-11T20:53:38.336128+0000","last_peered":"2024-03-11T20:53:38.336128+0000","last_clean":"2024-03-11T20:53:38.336128+0000","last_became_active":"2024-03-11T20:50:28.068978+0000","last_became_peered":"2024-03-11T20:50:28.068978+0000","last_unstale":"2024-03-11T20:53:38.336128+0000","last_undegraded":"2024-03-11T20:53:38.336128+0000","last_fullsized":"2024-03-11T20:53:38.336128+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:00.209791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:53:00.209791+0000","last_clean_scrub_stamp":"2024-03-11T20:53:00.209791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:01.669184+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"110'17","reported_seq":130,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.837592+0000","last_change":"2024-03-11T20:53:04.802853+0000","last_active":"2024-03-11T20:53:38.837592+0000","last_peered":"2024-03-11T20:53:38.837592+0000","last_clean":"2024-03-11T20:53:38.837592+0000","last_became_active":"2024-03-11T20:50:30.108364+0000","last_became_peered":"2024-03-11T20:50:30.108364+0000","last_unstale":"2024-03-11T20:53:38.837592+0000","last_undegraded":"2024-03-11T20:53:38.837592+0000","last_fullsized":"2024-03-11T20:53:38.837592+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'17","last_scrub_stamp":"2024-03-11T20:53:04.802488+0000","last_deep_scrub":"110'17","last_deep_scrub_stamp":"2024-03-11T20:53:04.802488+0000","last_clean_scrub_stamp":"2024-03-11T20:53:04.802488+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:26.988743+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":94,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.331094+0000","last_change":"2024-03-11T20:53:38.215774+0000","last_active":"2024-03-11T20:53:38.331094+0000","last_peered":"2024-03-11T20:53:38.331094+0000","last_clean":"2024-03-11T20:53:38.331094+0000","last_became_active":"2024-03-11T20:50:34.334558+0000","last_became_peered":"2024-03-11T20:50:34.334558+0000","last_unstale":"2024-03-11T20:53:38.331094+0000","last_undegraded":"2024-03-11T20:53:38.331094+0000","last_fullsized":"2024-03-11T20:53:38.331094+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:38.215565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:53:38.215565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:03.143663+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":98,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.330248+0000","last_change":"2024-03-11T20:52:59.455887+0000","last_active":"2024-03-11T20:53:38.330248+0000","last_peered":"2024-03-11T20:53:38.330248+0000","last_clean":"2024-03-11T20:53:38.330248+0000","last_became_active":"2024-03-11T20:50:32.202533+0000","last_became_peered":"2024-03-11T20:50:32.202533+0000","last_unstale":"2024-03-11T20:53:38.330248+0000","last_undegraded":"2024-03-11T20:53:38.330248+0000","last_fullsized":"2024-03-11T20:53:38.330248+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:59.455677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:52:59.455677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:29.453221+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"77'1","reported_seq":114,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.330139+0000","last_change":"2024-03-11T20:53:30.341375+0000","last_active":"2024-03-11T20:53:38.330139+0000","last_peered":"2024-03-11T20:53:38.330139+0000","last_clean":"2024-03-11T20:53:38.330139+0000","last_became_active":"2024-03-11T20:50:28.068840+0000","last_became_peered":"2024-03-11T20:50:28.068840+0000","last_unstale":"2024-03-11T20:53:38.330139+0000","last_undegraded":"2024-03-11T20:53:38.330139+0000","last_fullsized":"2024-03-11T20:53:38.330139+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-03-11T20:53:30.341011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:53:30.341011+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:38.426341+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"84'9","reported_seq":118,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.323124+0000","last_change":"2024-03-11T20:53:34.546830+0000","last_active":"2024-03-11T20:53:38.323124+0000","last_peered":"2024-03-11T20:53:38.323124+0000","last_clean":"2024-03-11T20:53:38.323124+0000","last_became_active":"2024-03-11T20:50:30.086169+0000","last_became_peered":"2024-03-11T20:50:30.086169+0000","last_unstale":"2024-03-11T20:53:38.323124+0000","last_undegraded":"2024-03-11T20:53:38.323124+0000","last_fullsized":"2024-03-11T20:53:38.323124+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'9","last_scrub_stamp":"2024-03-11T20:53:34.546587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:53:34.546587+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:58.297447+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":94,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.324864+0000","last_change":"2024-03-11T20:53:26.444831+0000","last_active":"2024-03-11T20:53:38.324864+0000","last_peered":"2024-03-11T20:53:38.324864+0000","last_clean":"2024-03-11T20:53:38.324864+0000","last_became_active":"2024-03-11T20:50:34.321714+0000","last_became_peered":"2024-03-11T20:50:34.321714+0000","last_unstale":"2024-03-11T20:53:38.324864+0000","last_undegraded":"2024-03-11T20:53:38.324864+0000","last_fullsized":"2024-03-11T20:53:38.324864+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:26.444527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:53:26.444527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:52.656618+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":98,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.882936+0000","last_change":"2024-03-11T20:53:15.798662+0000","last_active":"2024-03-11T20:53:38.882936+0000","last_peered":"2024-03-11T20:53:38.882936+0000","last_clean":"2024-03-11T20:53:38.882936+0000","last_became_active":"2024-03-11T20:50:32.192704+0000","last_became_peered":"2024-03-11T20:50:32.192704+0000","last_unstale":"2024-03-11T20:53:38.882936+0000","last_undegraded":"2024-03-11T20:53:38.882936+0000","last_fullsized":"2024-03-11T20:53:38.882936+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:15.798438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:53:15.798438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:29.061321+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":106,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.858837+0000","last_change":"2024-03-11T20:53:20.946464+0000","last_active":"2024-03-11T20:53:38.858837+0000","last_peered":"2024-03-11T20:53:38.858837+0000","last_clean":"2024-03-11T20:53:38.858837+0000","last_became_active":"2024-03-11T20:50:28.081538+0000","last_became_peered":"2024-03-11T20:50:28.081538+0000","last_unstale":"2024-03-11T20:53:38.858837+0000","last_undegraded":"2024-03-11T20:53:38.858837+0000","last_fullsized":"2024-03-11T20:53:38.858837+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:20.946227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:53:20.946227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:50.298331+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"84'5","reported_seq":111,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.322860+0000","last_change":"2024-03-11T20:53:31.929747+0000","last_active":"2024-03-11T20:53:38.322860+0000","last_peered":"2024-03-11T20:53:38.322860+0000","last_clean":"2024-03-11T20:53:38.322860+0000","last_became_active":"2024-03-11T20:50:30.099866+0000","last_became_peered":"2024-03-11T20:50:30.099866+0000","last_unstale":"2024-03-11T20:53:38.322860+0000","last_undegraded":"2024-03-11T20:53:38.322860+0000","last_fullsized":"2024-03-11T20:53:38.322860+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'5","last_scrub_stamp":"2024-03-11T20:53:31.929501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:53:31.929501+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:36.645523+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":94,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.323016+0000","last_change":"2024-03-11T20:53:21.162869+0000","last_active":"2024-03-11T20:53:38.323016+0000","last_peered":"2024-03-11T20:53:38.323016+0000","last_clean":"2024-03-11T20:53:38.323016+0000","last_became_active":"2024-03-11T20:50:34.307730+0000","last_became_peered":"2024-03-11T20:50:34.307730+0000","last_unstale":"2024-03-11T20:53:38.323016+0000","last_undegraded":"2024-03-11T20:53:38.323016+0000","last_fullsized":"2024-03-11T20:53:38.323016+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:21.162653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:53:21.162653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:34.261060+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1f","version":"114'12","reported_seq":155,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.859761+0000","last_change":"2024-03-11T20:53:30.906029+0000","last_active":"2024-03-11T20:53:38.859761+0000","last_peered":"2024-03-11T20:53:38.859761+0000","last_clean":"2024-03-11T20:53:38.859761+0000","last_became_active":"2024-03-11T20:50:32.200195+0000","last_became_peered":"2024-03-11T20:50:32.200195+0000","last_unstale":"2024-03-11T20:53:38.859761+0000","last_undegraded":"2024-03-11T20:53:38.859761+0000","last_fullsized":"2024-03-11T20:53:38.859761+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'12","last_scrub_stamp":"2024-03-11T20:53:30.905791+0000","last_deep_scrub":"114'12","last_deep_scrub_stamp":"2024-03-11T20:53:30.905791+0000","last_clean_scrub_stamp":"2024-03-11T20:53:30.905791+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:59.478960+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"77'1","reported_seq":99,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.329211+0000","last_change":"2024-03-11T20:52:24.397205+0000","last_active":"2024-03-11T20:53:38.329211+0000","last_peered":"2024-03-11T20:53:38.329211+0000","last_clean":"2024-03-11T20:53:38.329211+0000","last_became_active":"2024-03-11T20:50:28.068827+0000","last_became_peered":"2024-03-11T20:50:28.068827+0000","last_unstale":"2024-03-11T20:53:38.329211+0000","last_undegraded":"2024-03-11T20:53:38.329211+0000","last_fullsized":"2024-03-11T20:53:38.329211+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-03-11T20:52:24.396881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:52:24.396881+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:53.894381+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"84'9","reported_seq":118,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.328853+0000","last_change":"2024-03-11T20:53:23.347800+0000","last_active":"2024-03-11T20:53:38.328853+0000","last_peered":"2024-03-11T20:53:38.328853+0000","last_clean":"2024-03-11T20:53:38.328853+0000","last_became_active":"2024-03-11T20:50:30.083936+0000","last_became_peered":"2024-03-11T20:50:30.083936+0000","last_unstale":"2024-03-11T20:53:38.328853+0000","last_undegraded":"2024-03-11T20:53:38.328853+0000","last_fullsized":"2024-03-11T20:53:38.328853+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'9","last_scrub_stamp":"2024-03-11T20:53:23.347531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:53:23.347531+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:42.009935+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"105'2","reported_seq":97,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.335367+0000","last_change":"2024-03-11T20:53:34.350049+0000","last_active":"2024-03-11T20:53:38.335367+0000","last_peered":"2024-03-11T20:53:38.335367+0000","last_clean":"2024-03-11T20:53:38.335367+0000","last_became_active":"2024-03-11T20:50:34.327356+0000","last_became_peered":"2024-03-11T20:50:34.327356+0000","last_unstale":"2024-03-11T20:53:38.335367+0000","last_undegraded":"2024-03-11T20:53:38.335367+0000","last_fullsized":"2024-03-11T20:53:38.335367+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'2","last_scrub_stamp":"2024-03-11T20:53:34.349639+0000","last_deep_scrub":"105'2","last_deep_scrub_stamp":"2024-03-11T20:52:18.098911+0000","last_clean_scrub_stamp":"2024-03-11T20:53:34.349639+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:49.549358+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":97,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.324468+0000","last_change":"2024-03-11T20:53:10.062579+0000","last_active":"2024-03-11T20:53:38.324468+0000","last_peered":"2024-03-11T20:53:38.324468+0000","last_clean":"2024-03-11T20:53:38.324468+0000","last_became_active":"2024-03-11T20:50:32.648869+0000","last_became_peered":"2024-03-11T20:50:32.648869+0000","last_unstale":"2024-03-11T20:53:38.324468+0000","last_undegraded":"2024-03-11T20:53:38.324468+0000","last_fullsized":"2024-03-11T20:53:38.324468+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:10.062346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:53:10.062346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:39.641031+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":106,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.836132+0000","last_change":"2024-03-11T20:53:31.821391+0000","last_active":"2024-03-11T20:53:38.836132+0000","last_peered":"2024-03-11T20:53:38.836132+0000","last_clean":"2024-03-11T20:53:38.836132+0000","last_became_active":"2024-03-11T20:50:28.066124+0000","last_became_peered":"2024-03-11T20:50:28.066124+0000","last_unstale":"2024-03-11T20:53:38.836132+0000","last_undegraded":"2024-03-11T20:53:38.836132+0000","last_fullsized":"2024-03-11T20:53:38.836132+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:31.821104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:52:01.880052+0000","last_clean_scrub_stamp":"2024-03-11T20:53:31.821104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:35.848818+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"84'15","reported_seq":127,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.320438+0000","last_change":"2024-03-11T20:53:13.139250+0000","last_active":"2024-03-11T20:53:38.320438+0000","last_peered":"2024-03-11T20:53:38.320438+0000","last_clean":"2024-03-11T20:53:38.320438+0000","last_became_active":"2024-03-11T20:50:30.088446+0000","last_became_peered":"2024-03-11T20:50:30.088446+0000","last_unstale":"2024-03-11T20:53:38.320438+0000","last_undegraded":"2024-03-11T20:53:38.320438+0000","last_fullsized":"2024-03-11T20:53:38.320438+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15","last_scrub_stamp":"2024-03-11T20:53:13.138917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:53:13.138917+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:28.013588+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":94,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.331012+0000","last_change":"2024-03-11T20:53:28.358235+0000","last_active":"2024-03-11T20:53:38.331012+0000","last_peered":"2024-03-11T20:53:38.331012+0000","last_clean":"2024-03-11T20:53:38.331012+0000","last_became_active":"2024-03-11T20:50:34.337878+0000","last_became_peered":"2024-03-11T20:50:34.337878+0000","last_unstale":"2024-03-11T20:53:38.331012+0000","last_undegraded":"2024-03-11T20:53:38.331012+0000","last_fullsized":"2024-03-11T20:53:38.331012+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:28.358134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:52:11.462480+0000","last_clean_scrub_stamp":"2024-03-11T20:53:28.358134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:45.511381+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":98,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.329908+0000","last_change":"2024-03-11T20:53:02.400889+0000","last_active":"2024-03-11T20:53:38.329908+0000","last_peered":"2024-03-11T20:53:38.329908+0000","last_clean":"2024-03-11T20:53:38.329908+0000","last_became_active":"2024-03-11T20:50:32.646323+0000","last_became_peered":"2024-03-11T20:50:32.646323+0000","last_unstale":"2024-03-11T20:53:38.329908+0000","last_undegraded":"2024-03-11T20:53:38.329908+0000","last_fullsized":"2024-03-11T20:53:38.329908+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:02.400685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:53:02.400685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:04.399694+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":106,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.859866+0000","last_change":"2024-03-11T20:53:18.916363+0000","last_active":"2024-03-11T20:53:38.859866+0000","last_peered":"2024-03-11T20:53:38.859866+0000","last_clean":"2024-03-11T20:53:38.859866+0000","last_became_active":"2024-03-11T20:50:28.086513+0000","last_became_peered":"2024-03-11T20:50:28.086513+0000","last_unstale":"2024-03-11T20:53:38.859866+0000","last_undegraded":"2024-03-11T20:53:38.859866+0000","last_fullsized":"2024-03-11T20:53:38.859866+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:18.916152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:53:18.916152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:20.894545+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"84'9","reported_seq":118,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.837473+0000","last_change":"2024-03-11T20:53:17.784263+0000","last_active":"2024-03-11T20:53:38.837473+0000","last_peered":"2024-03-11T20:53:38.837473+0000","last_clean":"2024-03-11T20:53:38.837473+0000","last_became_active":"2024-03-11T20:50:30.109869+0000","last_became_peered":"2024-03-11T20:50:30.109869+0000","last_unstale":"2024-03-11T20:53:38.837473+0000","last_undegraded":"2024-03-11T20:53:38.837473+0000","last_fullsized":"2024-03-11T20:53:38.837473+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'9","last_scrub_stamp":"2024-03-11T20:53:17.784035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:53:17.784035+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:30.540981+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":94,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.334015+0000","last_change":"2024-03-11T20:53:04.321530+0000","last_active":"2024-03-11T20:53:38.334015+0000","last_peered":"2024-03-11T20:53:38.334015+0000","last_clean":"2024-03-11T20:53:38.334015+0000","last_became_active":"2024-03-11T20:50:34.321443+0000","last_became_peered":"2024-03-11T20:50:34.321443+0000","last_unstale":"2024-03-11T20:53:38.334015+0000","last_undegraded":"2024-03-11T20:53:38.334015+0000","last_fullsized":"2024-03-11T20:53:38.334015+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:04.321235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:53:04.321235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:23.221472+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":98,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.328829+0000","last_change":"2024-03-11T20:53:34.195962+0000","last_active":"2024-03-11T20:53:38.328829+0000","last_peered":"2024-03-11T20:53:38.328829+0000","last_clean":"2024-03-11T20:53:38.328829+0000","last_became_active":"2024-03-11T20:50:32.211995+0000","last_became_peered":"2024-03-11T20:50:32.211995+0000","last_unstale":"2024-03-11T20:53:38.328829+0000","last_undegraded":"2024-03-11T20:53:38.328829+0000","last_fullsized":"2024-03-11T20:53:38.328829+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:34.195681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:53:34.195681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:47.493709+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":106,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.837452+0000","last_change":"2024-03-11T20:52:53.806361+0000","last_active":"2024-03-11T20:53:38.837452+0000","last_peered":"2024-03-11T20:53:38.837452+0000","last_clean":"2024-03-11T20:53:38.837452+0000","last_became_active":"2024-03-11T20:50:28.075760+0000","last_became_peered":"2024-03-11T20:50:28.075760+0000","last_unstale":"2024-03-11T20:53:38.837452+0000","last_undegraded":"2024-03-11T20:53:38.837452+0000","last_fullsized":"2024-03-11T20:53:38.837452+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:53.806149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:52:53.806149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:15.472221+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"84'6","reported_seq":110,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.324926+0000","last_change":"2024-03-11T20:53:18.136127+0000","last_active":"2024-03-11T20:53:38.324926+0000","last_peered":"2024-03-11T20:53:38.324926+0000","last_clean":"2024-03-11T20:53:38.324926+0000","last_became_active":"2024-03-11T20:50:30.092991+0000","last_became_peered":"2024-03-11T20:50:30.092991+0000","last_unstale":"2024-03-11T20:53:38.324926+0000","last_undegraded":"2024-03-11T20:53:38.324926+0000","last_fullsized":"2024-03-11T20:53:38.324926+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6","last_scrub_stamp":"2024-03-11T20:53:18.135889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:53:18.135889+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:46.013838+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.11","version":"84'1","reported_seq":87,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.328802+0000","last_change":"2024-03-11T20:52:42.470691+0000","last_active":"2024-03-11T20:53:38.328802+0000","last_peered":"2024-03-11T20:53:38.328802+0000","last_clean":"2024-03-11T20:53:38.328802+0000","last_became_active":"2024-03-11T20:50:34.339326+0000","last_became_peered":"2024-03-11T20:50:34.339326+0000","last_unstale":"2024-03-11T20:53:38.328802+0000","last_undegraded":"2024-03-11T20:53:38.328802+0000","last_fullsized":"2024-03-11T20:53:38.328802+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'1","last_scrub_stamp":"2024-03-11T20:52:42.470380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:52:42.470380+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:08.720300+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":98,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.323291+0000","last_change":"2024-03-11T20:53:21.439340+0000","last_active":"2024-03-11T20:53:38.323291+0000","last_peered":"2024-03-11T20:53:38.323291+0000","last_clean":"2024-03-11T20:53:38.323291+0000","last_became_active":"2024-03-11T20:50:32.211379+0000","last_became_peered":"2024-03-11T20:50:32.211379+0000","last_unstale":"2024-03-11T20:53:38.323291+0000","last_undegraded":"2024-03-11T20:53:38.323291+0000","last_fullsized":"2024-03-11T20:53:38.323291+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:21.439121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:53:21.439121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:43.786620+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":106,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.323813+0000","last_change":"2024-03-11T20:53:14.109971+0000","last_active":"2024-03-11T20:53:38.323813+0000","last_peered":"2024-03-11T20:53:38.323813+0000","last_clean":"2024-03-11T20:53:38.323813+0000","last_became_active":"2024-03-11T20:50:28.049937+0000","last_became_peered":"2024-03-11T20:50:28.049937+0000","last_unstale":"2024-03-11T20:53:38.323813+0000","last_undegraded":"2024-03-11T20:53:38.323813+0000","last_fullsized":"2024-03-11T20:53:38.323813+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:14.109743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:53:14.109743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:31.769637+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"84'10","reported_seq":117,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.324559+0000","last_change":"2024-03-11T20:52:51.437032+0000","last_active":"2024-03-11T20:53:38.324559+0000","last_peered":"2024-03-11T20:53:38.324559+0000","last_clean":"2024-03-11T20:53:38.324559+0000","last_became_active":"2024-03-11T20:50:30.105752+0000","last_became_peered":"2024-03-11T20:50:30.105752+0000","last_unstale":"2024-03-11T20:53:38.324559+0000","last_undegraded":"2024-03-11T20:53:38.324559+0000","last_fullsized":"2024-03-11T20:53:38.324559+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'10","last_scrub_stamp":"2024-03-11T20:52:51.436752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:52:51.436752+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:15.812213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":93,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.319309+0000","last_change":"2024-03-11T20:53:22.057073+0000","last_active":"2024-03-11T20:53:38.319309+0000","last_peered":"2024-03-11T20:53:38.319309+0000","last_clean":"2024-03-11T20:53:38.319309+0000","last_became_active":"2024-03-11T20:50:34.332465+0000","last_became_peered":"2024-03-11T20:50:34.332465+0000","last_unstale":"2024-03-11T20:53:38.319309+0000","last_undegraded":"2024-03-11T20:53:38.319309+0000","last_fullsized":"2024-03-11T20:53:38.319309+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:22.056809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:53:22.056809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:42.841855+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":98,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.319395+0000","last_change":"2024-03-11T20:53:24.209982+0000","last_active":"2024-03-11T20:53:38.319395+0000","last_peered":"2024-03-11T20:53:38.319395+0000","last_clean":"2024-03-11T20:53:38.319395+0000","last_became_active":"2024-03-11T20:50:32.194792+0000","last_became_peered":"2024-03-11T20:50:32.194792+0000","last_unstale":"2024-03-11T20:53:38.319395+0000","last_undegraded":"2024-03-11T20:53:38.319395+0000","last_fullsized":"2024-03-11T20:53:38.319395+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:24.209748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:53:24.209748+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:43.454286+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":106,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.859115+0000","last_change":"2024-03-11T20:52:54.775523+0000","last_active":"2024-03-11T20:53:38.859115+0000","last_peered":"2024-03-11T20:53:38.859115+0000","last_clean":"2024-03-11T20:53:38.859115+0000","last_became_active":"2024-03-11T20:50:28.082737+0000","last_became_peered":"2024-03-11T20:50:28.082737+0000","last_unstale":"2024-03-11T20:53:38.859115+0000","last_undegraded":"2024-03-11T20:53:38.859115+0000","last_fullsized":"2024-03-11T20:53:38.859115+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:54.775303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:52:54.775303+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:11.643155+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"84'9","reported_seq":118,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.336298+0000","last_change":"2024-03-11T20:52:57.247973+0000","last_active":"2024-03-11T20:53:38.336298+0000","last_peered":"2024-03-11T20:53:38.336298+0000","last_clean":"2024-03-11T20:53:38.336298+0000","last_became_active":"2024-03-11T20:50:30.085418+0000","last_became_peered":"2024-03-11T20:50:30.085418+0000","last_unstale":"2024-03-11T20:53:38.336298+0000","last_undegraded":"2024-03-11T20:53:38.336298+0000","last_fullsized":"2024-03-11T20:53:38.336298+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'9","last_scrub_stamp":"2024-03-11T20:52:57.247711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:52:57.247711+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:09.030171+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":94,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.324970+0000","last_change":"2024-03-11T20:53:14.601565+0000","last_active":"2024-03-11T20:53:38.324970+0000","last_peered":"2024-03-11T20:53:38.324970+0000","last_clean":"2024-03-11T20:53:38.324970+0000","last_became_active":"2024-03-11T20:50:34.328449+0000","last_became_peered":"2024-03-11T20:50:34.328449+0000","last_unstale":"2024-03-11T20:53:38.324970+0000","last_undegraded":"2024-03-11T20:53:38.324970+0000","last_fullsized":"2024-03-11T20:53:38.324970+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:14.601439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:53:14.601439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:19.740273+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.15","version":"112'12","reported_seq":149,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.837925+0000","last_change":"2024-03-11T20:53:11.844176+0000","last_active":"2024-03-11T20:53:38.837925+0000","last_peered":"2024-03-11T20:53:38.837925+0000","last_clean":"2024-03-11T20:53:38.837925+0000","last_became_active":"2024-03-11T20:50:32.650064+0000","last_became_peered":"2024-03-11T20:50:32.650064+0000","last_unstale":"2024-03-11T20:53:38.837925+0000","last_undegraded":"2024-03-11T20:53:38.837925+0000","last_fullsized":"2024-03-11T20:53:38.837925+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'12","last_scrub_stamp":"2024-03-11T20:53:11.843951+0000","last_deep_scrub":"112'12","last_deep_scrub_stamp":"2024-03-11T20:53:11.843951+0000","last_clean_scrub_stamp":"2024-03-11T20:53:11.843951+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:40.680380+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":105,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.320246+0000","last_change":"2024-03-11T20:52:47.933545+0000","last_active":"2024-03-11T20:53:38.320246+0000","last_peered":"2024-03-11T20:53:38.320246+0000","last_clean":"2024-03-11T20:53:38.320246+0000","last_became_active":"2024-03-11T20:50:28.044321+0000","last_became_peered":"2024-03-11T20:50:28.044321+0000","last_unstale":"2024-03-11T20:53:38.320246+0000","last_undegraded":"2024-03-11T20:53:38.320246+0000","last_fullsized":"2024-03-11T20:53:38.320246+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:47.933320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:52:47.933320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:55.580970+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"84'9","reported_seq":117,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.321404+0000","last_change":"2024-03-11T20:52:55.993159+0000","last_active":"2024-03-11T20:53:38.321404+0000","last_peered":"2024-03-11T20:53:38.321404+0000","last_clean":"2024-03-11T20:53:38.321404+0000","last_became_active":"2024-03-11T20:50:30.099856+0000","last_became_peered":"2024-03-11T20:50:30.099856+0000","last_unstale":"2024-03-11T20:53:38.321404+0000","last_undegraded":"2024-03-11T20:53:38.321404+0000","last_fullsized":"2024-03-11T20:53:38.321404+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'9","last_scrub_stamp":"2024-03-11T20:52:55.992803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:52:55.992803+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:14.843849+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":93,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:40.043228+0000","last_change":"2024-03-11T20:53:40.043289+0000","last_active":"2024-03-11T20:53:40.043228+0000","last_peered":"2024-03-11T20:53:40.043228+0000","last_clean":"2024-03-11T20:53:40.043228+0000","last_became_active":"2024-03-11T20:50:34.330351+0000","last_became_peered":"2024-03-11T20:50:34.330351+0000","last_unstale":"2024-03-11T20:53:40.043228+0000","last_undegraded":"2024-03-11T20:53:40.043228+0000","last_fullsized":"2024-03-11T20:53:40.043228+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:40.043054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:53:40.043054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:56.504086+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"110'13","reported_seq":159,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.331632+0000","last_change":"2024-03-11T20:53:00.434710+0000","last_active":"2024-03-11T20:53:38.331632+0000","last_peered":"2024-03-11T20:53:38.331632+0000","last_clean":"2024-03-11T20:53:38.331632+0000","last_became_active":"2024-03-11T20:51:22.669419+0000","last_became_peered":"2024-03-11T20:51:22.669419+0000","last_unstale":"2024-03-11T20:53:38.331632+0000","last_undegraded":"2024-03-11T20:53:38.331632+0000","last_fullsized":"2024-03-11T20:53:38.331632+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'13","last_scrub_stamp":"2024-03-11T20:53:00.434532+0000","last_deep_scrub":"110'13","last_deep_scrub_stamp":"2024-03-11T20:53:00.434532+0000","last_clean_scrub_stamp":"2024-03-11T20:53:00.434532+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:25.979437+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":106,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.836449+0000","last_change":"2024-03-11T20:52:59.801478+0000","last_active":"2024-03-11T20:53:38.836449+0000","last_peered":"2024-03-11T20:53:38.836449+0000","last_clean":"2024-03-11T20:53:38.836449+0000","last_became_active":"2024-03-11T20:50:28.070325+0000","last_became_peered":"2024-03-11T20:50:28.070325+0000","last_unstale":"2024-03-11T20:53:38.836449+0000","last_undegraded":"2024-03-11T20:53:38.836449+0000","last_fullsized":"2024-03-11T20:53:38.836449+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:59.801249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:52:59.801249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:29.742165+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"116'16","reported_seq":126,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:39.292185+0000","last_change":"2024-03-11T20:53:39.292223+0000","last_active":"2024-03-11T20:53:39.292185+0000","last_peered":"2024-03-11T20:53:39.292185+0000","last_clean":"2024-03-11T20:53:39.292185+0000","last_became_active":"2024-03-11T20:50:30.087154+0000","last_became_peered":"2024-03-11T20:50:30.087154+0000","last_unstale":"2024-03-11T20:53:39.292185+0000","last_undegraded":"2024-03-11T20:53:39.292185+0000","last_fullsized":"2024-03-11T20:53:39.292185+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'16","last_scrub_stamp":"2024-03-11T20:53:39.292118+0000","last_deep_scrub":"116'16","last_deep_scrub_stamp":"2024-03-11T20:53:39.292118+0000","last_clean_scrub_stamp":"2024-03-11T20:53:39.292118+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:06.274297+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":86,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.334196+0000","last_change":"2024-03-11T20:52:22.074863+0000","last_active":"2024-03-11T20:53:38.334196+0000","last_peered":"2024-03-11T20:53:38.334196+0000","last_clean":"2024-03-11T20:53:38.334196+0000","last_became_active":"2024-03-11T20:50:34.318231+0000","last_became_peered":"2024-03-11T20:50:34.318231+0000","last_unstale":"2024-03-11T20:53:38.334196+0000","last_undegraded":"2024-03-11T20:53:38.334196+0000","last_fullsized":"2024-03-11T20:53:38.334196+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:22.074605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:52:22.074605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:39.515449+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":98,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.331323+0000","last_change":"2024-03-11T20:53:06.534811+0000","last_active":"2024-03-11T20:53:38.331323+0000","last_peered":"2024-03-11T20:53:38.331323+0000","last_clean":"2024-03-11T20:53:38.331323+0000","last_became_active":"2024-03-11T20:50:32.651361+0000","last_became_peered":"2024-03-11T20:50:32.651361+0000","last_unstale":"2024-03-11T20:53:38.331323+0000","last_undegraded":"2024-03-11T20:53:38.331323+0000","last_fullsized":"2024-03-11T20:53:38.331323+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:06.534615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:53:06.534615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:15.085973+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"77'1","reported_seq":114,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.858671+0000","last_change":"2024-03-11T20:53:24.901219+0000","last_active":"2024-03-11T20:53:38.858671+0000","last_peered":"2024-03-11T20:53:38.858671+0000","last_clean":"2024-03-11T20:53:38.858671+0000","last_became_active":"2024-03-11T20:50:28.083256+0000","last_became_peered":"2024-03-11T20:50:28.083256+0000","last_unstale":"2024-03-11T20:53:38.858671+0000","last_undegraded":"2024-03-11T20:53:38.858671+0000","last_fullsized":"2024-03-11T20:53:38.858671+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-03-11T20:53:24.900978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:53:24.900978+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:38.729662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"84'4","reported_seq":108,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:42.788315+0000","last_change":"2024-03-11T20:53:42.788395+0000","last_active":"2024-03-11T20:53:42.788315+0000","last_peered":"2024-03-11T20:53:42.788315+0000","last_clean":"2024-03-11T20:53:42.788315+0000","last_became_active":"2024-03-11T20:50:30.101572+0000","last_became_peered":"2024-03-11T20:50:30.101572+0000","last_unstale":"2024-03-11T20:53:42.788315+0000","last_undegraded":"2024-03-11T20:53:42.788315+0000","last_fullsized":"2024-03-11T20:53:42.788315+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'4","last_scrub_stamp":"2024-03-11T20:53:42.788154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:53:42.788154+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:11.351245+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":94,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.882182+0000","last_change":"2024-03-11T20:52:57.942490+0000","last_active":"2024-03-11T20:53:38.882182+0000","last_peered":"2024-03-11T20:53:38.882182+0000","last_clean":"2024-03-11T20:53:38.882182+0000","last_became_active":"2024-03-11T20:50:34.320001+0000","last_became_peered":"2024-03-11T20:50:34.320001+0000","last_unstale":"2024-03-11T20:53:38.882182+0000","last_undegraded":"2024-03-11T20:53:38.882182+0000","last_fullsized":"2024-03-11T20:53:38.882182+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:57.942382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:52:57.942382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:03.245684+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":97,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.323620+0000","last_change":"2024-03-11T20:53:05.100946+0000","last_active":"2024-03-11T20:53:38.323620+0000","last_peered":"2024-03-11T20:53:38.323620+0000","last_clean":"2024-03-11T20:53:38.323620+0000","last_became_active":"2024-03-11T20:50:32.646569+0000","last_became_peered":"2024-03-11T20:50:32.646569+0000","last_unstale":"2024-03-11T20:53:38.323620+0000","last_undegraded":"2024-03-11T20:53:38.323620+0000","last_fullsized":"2024-03-11T20:53:38.323620+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:05.100847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:53:05.100847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:07.646846+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":106,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.882472+0000","last_change":"2024-03-11T20:53:02.899718+0000","last_active":"2024-03-11T20:53:38.882472+0000","last_peered":"2024-03-11T20:53:38.882472+0000","last_clean":"2024-03-11T20:53:38.882472+0000","last_became_active":"2024-03-11T20:50:28.044592+0000","last_became_peered":"2024-03-11T20:50:28.044592+0000","last_unstale":"2024-03-11T20:53:38.882472+0000","last_undegraded":"2024-03-11T20:53:38.882472+0000","last_fullsized":"2024-03-11T20:53:38.882472+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:02.899469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:53:02.899469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:21.925538+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"84'11","reported_seq":121,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:42.298470+0000","last_change":"2024-03-11T20:53:42.298549+0000","last_active":"2024-03-11T20:53:42.298470+0000","last_peered":"2024-03-11T20:53:42.298470+0000","last_clean":"2024-03-11T20:53:42.298470+0000","last_became_active":"2024-03-11T20:50:30.103212+0000","last_became_peered":"2024-03-11T20:50:30.103212+0000","last_unstale":"2024-03-11T20:53:42.298470+0000","last_undegraded":"2024-03-11T20:53:42.298470+0000","last_fullsized":"2024-03-11T20:53:42.298470+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T20:53:42.298307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:53:42.298307+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:10.739880+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":94,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.330751+0000","last_change":"2024-03-11T20:52:53.451122+0000","last_active":"2024-03-11T20:53:38.330751+0000","last_peered":"2024-03-11T20:53:38.330751+0000","last_clean":"2024-03-11T20:53:38.330751+0000","last_became_active":"2024-03-11T20:50:34.336077+0000","last_became_peered":"2024-03-11T20:50:34.336077+0000","last_unstale":"2024-03-11T20:53:38.330751+0000","last_undegraded":"2024-03-11T20:53:38.330751+0000","last_fullsized":"2024-03-11T20:53:38.330751+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:53.450885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:52:53.450885+0000","last_clean_scrub_stamp":"2024-03-11T20:52:53.450885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:10.721360+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"77'2","reported_seq":122,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.324011+0000","last_change":"2024-03-11T20:53:28.411797+0000","last_active":"2024-03-11T20:53:38.324011+0000","last_peered":"2024-03-11T20:53:38.324011+0000","last_clean":"2024-03-11T20:53:38.324011+0000","last_became_active":"2024-03-11T20:50:28.059564+0000","last_became_peered":"2024-03-11T20:50:28.059564+0000","last_unstale":"2024-03-11T20:53:38.324011+0000","last_undegraded":"2024-03-11T20:53:38.324011+0000","last_fullsized":"2024-03-11T20:53:38.324011+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'2","last_scrub_stamp":"2024-03-11T20:53:28.411558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:53:28.411558+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:49.755187+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"100'16","reported_seq":126,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.336558+0000","last_change":"2024-03-11T20:52:37.207901+0000","last_active":"2024-03-11T20:53:38.336558+0000","last_peered":"2024-03-11T20:53:38.336558+0000","last_clean":"2024-03-11T20:53:38.336558+0000","last_became_active":"2024-03-11T20:50:30.083708+0000","last_became_peered":"2024-03-11T20:50:30.083708+0000","last_unstale":"2024-03-11T20:53:38.336558+0000","last_undegraded":"2024-03-11T20:53:38.336558+0000","last_fullsized":"2024-03-11T20:53:38.336558+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'16","last_scrub_stamp":"2024-03-11T20:52:37.207662+0000","last_deep_scrub":"100'16","last_deep_scrub_stamp":"2024-03-11T20:51:31.194005+0000","last_clean_scrub_stamp":"2024-03-11T20:52:37.207662+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:56.687016+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"106'12","reported_seq":141,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.322997+0000","last_change":"2024-03-11T20:52:21.539124+0000","last_active":"2024-03-11T20:53:38.322997+0000","last_peered":"2024-03-11T20:53:38.322997+0000","last_clean":"2024-03-11T20:53:38.322997+0000","last_became_active":"2024-03-11T20:50:32.210620+0000","last_became_peered":"2024-03-11T20:50:32.210620+0000","last_unstale":"2024-03-11T20:53:38.322997+0000","last_undegraded":"2024-03-11T20:53:38.322997+0000","last_fullsized":"2024-03-11T20:53:38.322997+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'12","last_scrub_stamp":"2024-03-11T20:52:21.538885+0000","last_deep_scrub":"106'12","last_deep_scrub_stamp":"2024-03-11T20:52:21.538885+0000","last_clean_scrub_stamp":"2024-03-11T20:52:21.538885+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":93,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.321843+0000","last_change":"2024-03-11T20:53:27.053454+0000","last_active":"2024-03-11T20:53:38.321843+0000","last_peered":"2024-03-11T20:53:38.321843+0000","last_clean":"2024-03-11T20:53:38.321843+0000","last_became_active":"2024-03-11T20:50:34.328932+0000","last_became_peered":"2024-03-11T20:50:34.328932+0000","last_unstale":"2024-03-11T20:53:38.321843+0000","last_undegraded":"2024-03-11T20:53:38.321843+0000","last_fullsized":"2024-03-11T20:53:38.321843+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:27.053208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:53:27.053208+0000","last_clean_scrub_stamp":"2024-03-11T20:53:27.053208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:28.263449+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":106,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.325021+0000","last_change":"2024-03-11T20:53:38.117372+0000","last_active":"2024-03-11T20:53:38.325021+0000","last_peered":"2024-03-11T20:53:38.325021+0000","last_clean":"2024-03-11T20:53:38.325021+0000","last_became_active":"2024-03-11T20:50:28.057091+0000","last_became_peered":"2024-03-11T20:50:28.057091+0000","last_unstale":"2024-03-11T20:53:38.325021+0000","last_undegraded":"2024-03-11T20:53:38.325021+0000","last_fullsized":"2024-03-11T20:53:38.325021+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:38.117113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:53:38.117113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:45.724302+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"84'16","reported_seq":156,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.837394+0000","last_change":"2024-03-11T20:52:03.937604+0000","last_active":"2024-03-11T20:53:38.837394+0000","last_peered":"2024-03-11T20:53:38.837394+0000","last_clean":"2024-03-11T20:53:38.837394+0000","last_became_active":"2024-03-11T20:50:30.094977+0000","last_became_peered":"2024-03-11T20:50:30.094977+0000","last_unstale":"2024-03-11T20:53:38.837394+0000","last_undegraded":"2024-03-11T20:53:38.837394+0000","last_fullsized":"2024-03-11T20:53:38.837394+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'16","last_scrub_stamp":"2024-03-11T20:52:03.937351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:52:03.937351+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":98,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.320378+0000","last_change":"2024-03-11T20:53:35.139583+0000","last_active":"2024-03-11T20:53:38.320378+0000","last_peered":"2024-03-11T20:53:38.320378+0000","last_clean":"2024-03-11T20:53:38.320378+0000","last_became_active":"2024-03-11T20:50:32.206394+0000","last_became_peered":"2024-03-11T20:50:32.206394+0000","last_unstale":"2024-03-11T20:53:38.320378+0000","last_undegraded":"2024-03-11T20:53:38.320378+0000","last_fullsized":"2024-03-11T20:53:38.320378+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:35.139348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:52:10.160202+0000","last_clean_scrub_stamp":"2024-03-11T20:53:35.139348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:00.755689+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":94,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.331134+0000","last_change":"2024-03-11T20:53:33.225496+0000","last_active":"2024-03-11T20:53:38.331134+0000","last_peered":"2024-03-11T20:53:38.331134+0000","last_clean":"2024-03-11T20:53:38.331134+0000","last_became_active":"2024-03-11T20:50:34.329253+0000","last_became_peered":"2024-03-11T20:50:34.329253+0000","last_unstale":"2024-03-11T20:53:38.331134+0000","last_undegraded":"2024-03-11T20:53:38.331134+0000","last_fullsized":"2024-03-11T20:53:38.331134+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:33.225296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:53:33.225296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:48.626406+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":106,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.836515+0000","last_change":"2024-03-11T20:52:58.782154+0000","last_active":"2024-03-11T20:53:38.836515+0000","last_peered":"2024-03-11T20:53:38.836515+0000","last_clean":"2024-03-11T20:53:38.836515+0000","last_became_active":"2024-03-11T20:50:28.075678+0000","last_became_peered":"2024-03-11T20:50:28.075678+0000","last_unstale":"2024-03-11T20:53:38.836515+0000","last_undegraded":"2024-03-11T20:53:38.836515+0000","last_fullsized":"2024-03-11T20:53:38.836515+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:58.781938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:33.823704+0000","last_clean_scrub_stamp":"2024-03-11T20:52:58.781938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:19.252494+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"84'19","reported_seq":138,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.324245+0000","last_change":"2024-03-11T20:53:13.560575+0000","last_active":"2024-03-11T20:53:38.324245+0000","last_peered":"2024-03-11T20:53:38.324245+0000","last_clean":"2024-03-11T20:53:38.324245+0000","last_became_active":"2024-03-11T20:50:30.097543+0000","last_became_peered":"2024-03-11T20:50:30.097543+0000","last_unstale":"2024-03-11T20:53:38.324245+0000","last_undegraded":"2024-03-11T20:53:38.324245+0000","last_fullsized":"2024-03-11T20:53:38.324245+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'19","last_scrub_stamp":"2024-03-11T20:53:13.560307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:53:13.560307+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:17.749885+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":98,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.324785+0000","last_change":"2024-03-11T20:53:28.142708+0000","last_active":"2024-03-11T20:53:38.324785+0000","last_peered":"2024-03-11T20:53:38.324785+0000","last_clean":"2024-03-11T20:53:38.324785+0000","last_became_active":"2024-03-11T20:50:32.211494+0000","last_became_peered":"2024-03-11T20:50:32.211494+0000","last_unstale":"2024-03-11T20:53:38.324785+0000","last_undegraded":"2024-03-11T20:53:38.324785+0000","last_fullsized":"2024-03-11T20:53:38.324785+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:28.142465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:53:28.142465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:50.624591+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":94,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:43.272228+0000","last_change":"2024-03-11T20:53:43.272285+0000","last_active":"2024-03-11T20:53:43.272228+0000","last_peered":"2024-03-11T20:53:43.272228+0000","last_clean":"2024-03-11T20:53:43.272228+0000","last_became_active":"2024-03-11T20:50:34.321038+0000","last_became_peered":"2024-03-11T20:50:34.321038+0000","last_unstale":"2024-03-11T20:53:43.272228+0000","last_undegraded":"2024-03-11T20:53:43.272228+0000","last_fullsized":"2024-03-11T20:53:43.272228+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:43.272073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:53:43.272073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:52.600692+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"109'26","reported_seq":142,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.319194+0000","last_change":"2024-03-11T20:52:49.216560+0000","last_active":"2024-03-11T20:53:38.319194+0000","last_peered":"2024-03-11T20:53:38.319194+0000","last_clean":"2024-03-11T20:53:38.319194+0000","last_became_active":"2024-03-11T20:50:30.098371+0000","last_became_peered":"2024-03-11T20:50:30.098371+0000","last_unstale":"2024-03-11T20:53:38.319194+0000","last_undegraded":"2024-03-11T20:53:38.319194+0000","last_fullsized":"2024-03-11T20:53:38.319194+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'26","last_scrub_stamp":"2024-03-11T20:52:49.216378+0000","last_deep_scrub":"109'26","last_deep_scrub_stamp":"2024-03-11T20:52:49.216378+0000","last_clean_scrub_stamp":"2024-03-11T20:52:49.216378+0000","objects_scrubbed":8,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:49.844091+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":106,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.882816+0000","last_change":"2024-03-11T20:53:06.797551+0000","last_active":"2024-03-11T20:53:38.882816+0000","last_peered":"2024-03-11T20:53:38.882816+0000","last_clean":"2024-03-11T20:53:38.882816+0000","last_became_active":"2024-03-11T20:50:28.045625+0000","last_became_peered":"2024-03-11T20:50:28.045625+0000","last_unstale":"2024-03-11T20:53:38.882816+0000","last_undegraded":"2024-03-11T20:53:38.882816+0000","last_fullsized":"2024-03-11T20:53:38.882816+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:06.797336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:53:06.797336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:24.649727+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":90,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.321468+0000","last_change":"2024-03-11T20:52:25.125883+0000","last_active":"2024-03-11T20:53:38.321468+0000","last_peered":"2024-03-11T20:53:38.321468+0000","last_clean":"2024-03-11T20:53:38.321468+0000","last_became_active":"2024-03-11T20:50:32.205996+0000","last_became_peered":"2024-03-11T20:50:32.205996+0000","last_unstale":"2024-03-11T20:53:38.321468+0000","last_undegraded":"2024-03-11T20:53:38.321468+0000","last_fullsized":"2024-03-11T20:53:38.321468+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:25.125696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:52:25.125696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:52.303574+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":94,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:40.853470+0000","last_change":"2024-03-11T20:53:40.853529+0000","last_active":"2024-03-11T20:53:40.853470+0000","last_peered":"2024-03-11T20:53:40.853470+0000","last_clean":"2024-03-11T20:53:40.853470+0000","last_became_active":"2024-03-11T20:50:34.336066+0000","last_became_peered":"2024-03-11T20:50:34.336066+0000","last_unstale":"2024-03-11T20:53:40.853470+0000","last_undegraded":"2024-03-11T20:53:40.853470+0000","last_fullsized":"2024-03-11T20:53:40.853470+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:40.853290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:53:40.853290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:02.791101+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"84'10","reported_seq":109,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.329193+0000","last_change":"2024-03-11T20:52:35.399348+0000","last_active":"2024-03-11T20:53:38.329193+0000","last_peered":"2024-03-11T20:53:38.329193+0000","last_clean":"2024-03-11T20:53:38.329193+0000","last_became_active":"2024-03-11T20:50:30.101747+0000","last_became_peered":"2024-03-11T20:50:30.101747+0000","last_unstale":"2024-03-11T20:53:38.329193+0000","last_undegraded":"2024-03-11T20:53:38.329193+0000","last_fullsized":"2024-03-11T20:53:38.329193+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'10","last_scrub_stamp":"2024-03-11T20:52:35.399027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:52:35.399027+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:57.620344+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":106,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.836941+0000","last_change":"2024-03-11T20:52:44.919146+0000","last_active":"2024-03-11T20:53:38.836941+0000","last_peered":"2024-03-11T20:53:38.836941+0000","last_clean":"2024-03-11T20:53:38.836941+0000","last_became_active":"2024-03-11T20:50:28.085503+0000","last_became_peered":"2024-03-11T20:50:28.085503+0000","last_unstale":"2024-03-11T20:53:38.836941+0000","last_undegraded":"2024-03-11T20:53:38.836941+0000","last_fullsized":"2024-03-11T20:53:38.836941+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:44.918937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:52:44.918937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:47.097462+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"102'128","reported_seq":363,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.334738+0000","last_change":"2024-03-11T20:52:28.047817+0000","last_active":"2024-03-11T20:53:38.334738+0000","last_peered":"2024-03-11T20:53:38.334738+0000","last_clean":"2024-03-11T20:53:38.334738+0000","last_became_active":"2024-03-11T20:49:52.584543+0000","last_became_peered":"2024-03-11T20:49:52.584543+0000","last_unstale":"2024-03-11T20:53:38.334738+0000","last_undegraded":"2024-03-11T20:53:38.334738+0000","last_fullsized":"2024-03-11T20:53:38.334738+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"102'128","last_scrub_stamp":"2024-03-11T20:52:28.047573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:45:32.381751+0000","last_clean_scrub_stamp":"2024-03-11T20:52:28.047573+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:54.167386+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":98,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.859248+0000","last_change":"2024-03-11T20:53:07.859791+0000","last_active":"2024-03-11T20:53:38.859248+0000","last_peered":"2024-03-11T20:53:38.859248+0000","last_clean":"2024-03-11T20:53:38.859248+0000","last_became_active":"2024-03-11T20:50:32.652128+0000","last_became_peered":"2024-03-11T20:50:32.652128+0000","last_unstale":"2024-03-11T20:53:38.859248+0000","last_undegraded":"2024-03-11T20:53:38.859248+0000","last_fullsized":"2024-03-11T20:53:38.859248+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:07.859545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:53:07.859545+0000","last_clean_scrub_stamp":"2024-03-11T20:53:07.859545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:20.657302+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":94,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.324527+0000","last_change":"2024-03-11T20:53:23.179735+0000","last_active":"2024-03-11T20:53:38.324527+0000","last_peered":"2024-03-11T20:53:38.324527+0000","last_clean":"2024-03-11T20:53:38.324527+0000","last_became_active":"2024-03-11T20:50:34.326321+0000","last_became_peered":"2024-03-11T20:50:34.326321+0000","last_unstale":"2024-03-11T20:53:38.324527+0000","last_undegraded":"2024-03-11T20:53:38.324527+0000","last_fullsized":"2024-03-11T20:53:38.324527+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:23.179506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:53:23.179506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:45.358796+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"100'21","reported_seq":129,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.320405+0000","last_change":"2024-03-11T20:53:03.110687+0000","last_active":"2024-03-11T20:53:38.320405+0000","last_peered":"2024-03-11T20:53:38.320405+0000","last_clean":"2024-03-11T20:53:38.320405+0000","last_became_active":"2024-03-11T20:50:30.106110+0000","last_became_peered":"2024-03-11T20:50:30.106110+0000","last_unstale":"2024-03-11T20:53:38.320405+0000","last_undegraded":"2024-03-11T20:53:38.320405+0000","last_fullsized":"2024-03-11T20:53:38.320405+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'21","last_scrub_stamp":"2024-03-11T20:53:03.110450+0000","last_deep_scrub":"100'21","last_deep_scrub_stamp":"2024-03-11T20:51:31.835302+0000","last_clean_scrub_stamp":"2024-03-11T20:53:03.110450+0000","objects_scrubbed":7,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:23.836851+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":64,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.882136+0000","last_change":"2024-03-11T20:52:56.891528+0000","last_active":"2024-03-11T20:53:38.882136+0000","last_peered":"2024-03-11T20:53:38.882136+0000","last_clean":"2024-03-11T20:53:38.882136+0000","last_became_active":"2024-03-11T20:51:22.668492+0000","last_became_peered":"2024-03-11T20:51:22.668492+0000","last_unstale":"2024-03-11T20:53:38.882136+0000","last_undegraded":"2024-03-11T20:53:38.882136+0000","last_fullsized":"2024-03-11T20:53:38.882136+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:56.891410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:52:56.891410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:57.500706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":97,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.324201+0000","last_change":"2024-03-11T20:53:04.140960+0000","last_active":"2024-03-11T20:53:38.324201+0000","last_peered":"2024-03-11T20:53:38.324201+0000","last_clean":"2024-03-11T20:53:38.324201+0000","last_became_active":"2024-03-11T20:50:32.195566+0000","last_became_peered":"2024-03-11T20:50:32.195566+0000","last_unstale":"2024-03-11T20:53:38.324201+0000","last_undegraded":"2024-03-11T20:53:38.324201+0000","last_fullsized":"2024-03-11T20:53:38.324201+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:04.140875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:53:04.140875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:29.280686+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":94,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.318406+0000","last_change":"2024-03-11T20:53:08.170244+0000","last_active":"2024-03-11T20:53:38.318406+0000","last_peered":"2024-03-11T20:53:38.318406+0000","last_clean":"2024-03-11T20:53:38.318406+0000","last_became_active":"2024-03-11T20:50:34.306750+0000","last_became_peered":"2024-03-11T20:50:34.306750+0000","last_unstale":"2024-03-11T20:53:38.318406+0000","last_undegraded":"2024-03-11T20:53:38.318406+0000","last_fullsized":"2024-03-11T20:53:38.318406+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:08.170036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:53:08.170036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:35.208803+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"87'2","reported_seq":107,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.858674+0000","last_change":"2024-03-11T20:53:25.900353+0000","last_active":"2024-03-11T20:53:38.858674+0000","last_peered":"2024-03-11T20:53:38.858674+0000","last_clean":"2024-03-11T20:53:38.858674+0000","last_became_active":"2024-03-11T20:50:32.195452+0000","last_became_peered":"2024-03-11T20:50:32.195452+0000","last_unstale":"2024-03-11T20:53:38.858674+0000","last_undegraded":"2024-03-11T20:53:38.858674+0000","last_fullsized":"2024-03-11T20:53:38.858674+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'2","last_scrub_stamp":"2024-03-11T20:53:25.900041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:53:25.900041+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:42.706289+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0024049620000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":106,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.336007+0000","last_change":"2024-03-11T20:53:05.364652+0000","last_active":"2024-03-11T20:53:38.336007+0000","last_peered":"2024-03-11T20:53:38.336007+0000","last_clean":"2024-03-11T20:53:38.336007+0000","last_became_active":"2024-03-11T20:50:28.065245+0000","last_became_peered":"2024-03-11T20:50:28.065245+0000","last_unstale":"2024-03-11T20:53:38.336007+0000","last_undegraded":"2024-03-11T20:53:38.336007+0000","last_fullsized":"2024-03-11T20:53:38.336007+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:05.364429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:53:05.364429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:25.580910+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"84'30","reported_seq":192,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.325597+0000","last_change":"2024-03-11T20:52:48.167124+0000","last_active":"2024-03-11T20:53:38.325597+0000","last_peered":"2024-03-11T20:53:38.325597+0000","last_clean":"2024-03-11T20:53:38.325597+0000","last_became_active":"2024-03-11T20:50:30.095177+0000","last_became_peered":"2024-03-11T20:50:30.095177+0000","last_unstale":"2024-03-11T20:53:38.325597+0000","last_undegraded":"2024-03-11T20:53:38.325597+0000","last_fullsized":"2024-03-11T20:53:38.325597+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'30","last_scrub_stamp":"2024-03-11T20:52:48.166893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:52:48.166893+0000","objects_scrubbed":10,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:05.469139+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":51,"num_read_kb":36,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"84'11","reported_seq":147,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.323222+0000","last_change":"2024-03-11T20:53:35.960764+0000","last_active":"2024-03-11T20:53:38.323222+0000","last_peered":"2024-03-11T20:53:38.323222+0000","last_clean":"2024-03-11T20:53:38.323222+0000","last_became_active":"2024-03-11T20:50:32.649414+0000","last_became_peered":"2024-03-11T20:50:32.649414+0000","last_unstale":"2024-03-11T20:53:38.323222+0000","last_undegraded":"2024-03-11T20:53:38.323222+0000","last_fullsized":"2024-03-11T20:53:38.323222+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T20:53:35.960574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:53:35.960574+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:52.893266+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":94,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.323448+0000","last_change":"2024-03-11T20:53:27.419736+0000","last_active":"2024-03-11T20:53:38.323448+0000","last_peered":"2024-03-11T20:53:38.323448+0000","last_clean":"2024-03-11T20:53:38.323448+0000","last_became_active":"2024-03-11T20:50:34.323647+0000","last_became_peered":"2024-03-11T20:50:34.323647+0000","last_unstale":"2024-03-11T20:53:38.323448+0000","last_undegraded":"2024-03-11T20:53:38.323448+0000","last_fullsized":"2024-03-11T20:53:38.323448+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:27.419519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:53:27.419519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:29.550989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":105,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.322319+0000","last_change":"2024-03-11T20:53:32.530057+0000","last_active":"2024-03-11T20:53:38.322319+0000","last_peered":"2024-03-11T20:53:38.322319+0000","last_clean":"2024-03-11T20:53:38.322319+0000","last_became_active":"2024-03-11T20:50:32.650102+0000","last_became_peered":"2024-03-11T20:50:32.650102+0000","last_unstale":"2024-03-11T20:53:38.322319+0000","last_undegraded":"2024-03-11T20:53:38.322319+0000","last_fullsized":"2024-03-11T20:53:38.322319+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:32.529832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:53:32.529832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:48.340417+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.0022993710000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"77'1","reported_seq":107,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.319697+0000","last_change":"2024-03-11T20:53:26.158578+0000","last_active":"2024-03-11T20:53:38.319697+0000","last_peered":"2024-03-11T20:53:38.319697+0000","last_clean":"2024-03-11T20:53:38.319697+0000","last_became_active":"2024-03-11T20:50:28.065076+0000","last_became_peered":"2024-03-11T20:50:28.065076+0000","last_unstale":"2024-03-11T20:53:38.319697+0000","last_undegraded":"2024-03-11T20:53:38.319697+0000","last_fullsized":"2024-03-11T20:53:38.319697+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-03-11T20:53:26.158259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:53:26.158259+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:48.651482+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"84'13","reported_seq":129,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.330774+0000","last_change":"2024-03-11T20:53:27.387229+0000","last_active":"2024-03-11T20:53:38.330774+0000","last_peered":"2024-03-11T20:53:38.330774+0000","last_clean":"2024-03-11T20:53:38.330774+0000","last_became_active":"2024-03-11T20:50:30.097669+0000","last_became_peered":"2024-03-11T20:50:30.097669+0000","last_unstale":"2024-03-11T20:53:38.330774+0000","last_undegraded":"2024-03-11T20:53:38.330774+0000","last_fullsized":"2024-03-11T20:53:38.330774+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'13","last_scrub_stamp":"2024-03-11T20:53:27.386971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:53:27.386971+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:46.046919+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"84'11","reported_seq":148,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.329352+0000","last_change":"2024-03-11T20:53:13.413293+0000","last_active":"2024-03-11T20:53:38.329352+0000","last_peered":"2024-03-11T20:53:38.329352+0000","last_clean":"2024-03-11T20:53:38.329352+0000","last_became_active":"2024-03-11T20:50:32.647041+0000","last_became_peered":"2024-03-11T20:50:32.647041+0000","last_unstale":"2024-03-11T20:53:38.329352+0000","last_undegraded":"2024-03-11T20:53:38.329352+0000","last_fullsized":"2024-03-11T20:53:38.329352+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T20:53:13.413140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:53:13.413140+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:27.508758+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":94,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.334472+0000","last_change":"2024-03-11T20:53:06.344681+0000","last_active":"2024-03-11T20:53:38.334472+0000","last_peered":"2024-03-11T20:53:38.334472+0000","last_clean":"2024-03-11T20:53:38.334472+0000","last_became_active":"2024-03-11T20:50:34.326947+0000","last_became_peered":"2024-03-11T20:50:34.326947+0000","last_unstale":"2024-03-11T20:53:38.334472+0000","last_undegraded":"2024-03-11T20:53:38.334472+0000","last_fullsized":"2024-03-11T20:53:38.334472+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:06.344386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:53:06.344386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:09.051382+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"91'5","reported_seq":281,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:43.048670+0000","last_change":"2024-03-11T20:53:08.493671+0000","last_active":"2024-03-11T20:53:43.048670+0000","last_peered":"2024-03-11T20:53:43.048670+0000","last_clean":"2024-03-11T20:53:43.048670+0000","last_became_active":"2024-03-11T20:50:32.211118+0000","last_became_peered":"2024-03-11T20:50:32.211118+0000","last_unstale":"2024-03-11T20:53:43.048670+0000","last_undegraded":"2024-03-11T20:53:43.048670+0000","last_fullsized":"2024-03-11T20:53:43.048670+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'5","last_scrub_stamp":"2024-03-11T20:53:08.493516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:53:08.493516+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:17.353002+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0035533370000000002,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":173,"num_read_kb":168,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":106,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.859794+0000","last_change":"2024-03-11T20:53:31.863953+0000","last_active":"2024-03-11T20:53:38.859794+0000","last_peered":"2024-03-11T20:53:38.859794+0000","last_clean":"2024-03-11T20:53:38.859794+0000","last_became_active":"2024-03-11T20:50:28.071707+0000","last_became_peered":"2024-03-11T20:50:28.071707+0000","last_unstale":"2024-03-11T20:53:38.859794+0000","last_undegraded":"2024-03-11T20:53:38.859794+0000","last_fullsized":"2024-03-11T20:53:38.859794+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:31.863747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:53:31.863747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:44.440903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"84'12","reported_seq":119,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.324601+0000","last_change":"2024-03-11T20:53:32.943227+0000","last_active":"2024-03-11T20:53:38.324601+0000","last_peered":"2024-03-11T20:53:38.324601+0000","last_clean":"2024-03-11T20:53:38.324601+0000","last_became_active":"2024-03-11T20:50:30.096350+0000","last_became_peered":"2024-03-11T20:50:30.096350+0000","last_unstale":"2024-03-11T20:53:38.324601+0000","last_undegraded":"2024-03-11T20:53:38.324601+0000","last_fullsized":"2024-03-11T20:53:38.324601+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'12","last_scrub_stamp":"2024-03-11T20:53:32.942979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:53:32.942979+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:43.116324+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":98,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.323417+0000","last_change":"2024-03-11T20:53:33.556343+0000","last_active":"2024-03-11T20:53:38.323417+0000","last_peered":"2024-03-11T20:53:38.323417+0000","last_clean":"2024-03-11T20:53:38.323417+0000","last_became_active":"2024-03-11T20:50:32.190896+0000","last_became_peered":"2024-03-11T20:50:32.190896+0000","last_unstale":"2024-03-11T20:53:38.323417+0000","last_undegraded":"2024-03-11T20:53:38.323417+0000","last_fullsized":"2024-03-11T20:53:38.323417+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:33.556062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:53:33.556062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:58.966484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":94,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.330115+0000","last_change":"2024-03-11T20:53:35.172511+0000","last_active":"2024-03-11T20:53:38.330115+0000","last_peered":"2024-03-11T20:53:38.330115+0000","last_clean":"2024-03-11T20:53:38.330115+0000","last_became_active":"2024-03-11T20:50:34.335370+0000","last_became_peered":"2024-03-11T20:50:34.335370+0000","last_unstale":"2024-03-11T20:53:38.330115+0000","last_undegraded":"2024-03-11T20:53:38.330115+0000","last_fullsized":"2024-03-11T20:53:38.330115+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:35.172297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:53:35.172297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:51.240908+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"84'11","reported_seq":140,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.324881+0000","last_change":"2024-03-11T20:52:15.500805+0000","last_active":"2024-03-11T20:53:38.324881+0000","last_peered":"2024-03-11T20:53:38.324881+0000","last_clean":"2024-03-11T20:53:38.324881+0000","last_became_active":"2024-03-11T20:50:32.197412+0000","last_became_peered":"2024-03-11T20:50:32.197412+0000","last_unstale":"2024-03-11T20:53:38.324881+0000","last_undegraded":"2024-03-11T20:53:38.324881+0000","last_fullsized":"2024-03-11T20:53:38.324881+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T20:52:15.500571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:52:15.500571+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":106,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.334812+0000","last_change":"2024-03-11T20:53:28.297690+0000","last_active":"2024-03-11T20:53:38.334812+0000","last_peered":"2024-03-11T20:53:38.334812+0000","last_clean":"2024-03-11T20:53:38.334812+0000","last_became_active":"2024-03-11T20:50:28.079413+0000","last_became_peered":"2024-03-11T20:50:28.079413+0000","last_unstale":"2024-03-11T20:53:38.334812+0000","last_undegraded":"2024-03-11T20:53:38.334812+0000","last_fullsized":"2024-03-11T20:53:38.334812+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:28.297455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:52:09.043655+0000","last_clean_scrub_stamp":"2024-03-11T20:53:28.297455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:57.298447+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"84'12","reported_seq":125,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.323203+0000","last_change":"2024-03-11T20:53:22.404447+0000","last_active":"2024-03-11T20:53:38.323203+0000","last_peered":"2024-03-11T20:53:38.323203+0000","last_clean":"2024-03-11T20:53:38.323203+0000","last_became_active":"2024-03-11T20:50:30.083388+0000","last_became_peered":"2024-03-11T20:50:30.083388+0000","last_unstale":"2024-03-11T20:53:38.323203+0000","last_undegraded":"2024-03-11T20:53:38.323203+0000","last_fullsized":"2024-03-11T20:53:38.323203+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'12","last_scrub_stamp":"2024-03-11T20:53:22.404131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:53:22.404131+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:45.149957+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":94,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.835246+0000","last_change":"2024-03-11T20:53:06.771715+0000","last_active":"2024-03-11T20:53:38.835246+0000","last_peered":"2024-03-11T20:53:38.835246+0000","last_clean":"2024-03-11T20:53:38.835246+0000","last_became_active":"2024-03-11T20:50:34.337717+0000","last_became_peered":"2024-03-11T20:50:34.337717+0000","last_unstale":"2024-03-11T20:53:38.835246+0000","last_undegraded":"2024-03-11T20:53:38.835246+0000","last_fullsized":"2024-03-11T20:53:38.835246+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:06.771620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:53:06.771620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:30.302985+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":64,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.882697+0000","last_change":"2024-03-11T20:53:05.836875+0000","last_active":"2024-03-11T20:53:38.882697+0000","last_peered":"2024-03-11T20:53:38.882697+0000","last_clean":"2024-03-11T20:53:38.882697+0000","last_became_active":"2024-03-11T20:51:22.662693+0000","last_became_peered":"2024-03-11T20:51:22.662693+0000","last_unstale":"2024-03-11T20:53:38.882697+0000","last_undegraded":"2024-03-11T20:53:38.882697+0000","last_fullsized":"2024-03-11T20:53:38.882697+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:05.836655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:53:05.836655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:21.447337+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":106,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.320164+0000","last_change":"2024-03-11T20:53:17.169155+0000","last_active":"2024-03-11T20:53:38.320164+0000","last_peered":"2024-03-11T20:53:38.320164+0000","last_clean":"2024-03-11T20:53:38.320164+0000","last_became_active":"2024-03-11T20:50:28.058790+0000","last_became_peered":"2024-03-11T20:50:28.058790+0000","last_unstale":"2024-03-11T20:53:38.320164+0000","last_undegraded":"2024-03-11T20:53:38.320164+0000","last_fullsized":"2024-03-11T20:53:38.320164+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:17.168912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:40.974147+0000","last_clean_scrub_stamp":"2024-03-11T20:53:17.168912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:23.351155+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"84'15","reported_seq":119,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.329136+0000","last_change":"2024-03-11T20:52:40.434414+0000","last_active":"2024-03-11T20:53:38.329136+0000","last_peered":"2024-03-11T20:53:38.329136+0000","last_clean":"2024-03-11T20:53:38.329136+0000","last_became_active":"2024-03-11T20:50:30.096215+0000","last_became_peered":"2024-03-11T20:50:30.096215+0000","last_unstale":"2024-03-11T20:53:38.329136+0000","last_undegraded":"2024-03-11T20:53:38.329136+0000","last_fullsized":"2024-03-11T20:53:38.329136+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15","last_scrub_stamp":"2024-03-11T20:52:40.434169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:52:40.434169+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:45.160041+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":86,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.325046+0000","last_change":"2024-03-11T20:52:12.514160+0000","last_active":"2024-03-11T20:53:38.325046+0000","last_peered":"2024-03-11T20:53:38.325046+0000","last_clean":"2024-03-11T20:53:38.325046+0000","last_became_active":"2024-03-11T20:50:34.331803+0000","last_became_peered":"2024-03-11T20:50:34.331803+0000","last_unstale":"2024-03-11T20:53:38.325046+0000","last_undegraded":"2024-03-11T20:53:38.325046+0000","last_fullsized":"2024-03-11T20:53:38.325046+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:12.513982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:52:12.513982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":106,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.319742+0000","last_change":"2024-03-11T20:53:37.094181+0000","last_active":"2024-03-11T20:53:38.319742+0000","last_peered":"2024-03-11T20:53:38.319742+0000","last_clean":"2024-03-11T20:53:38.319742+0000","last_became_active":"2024-03-11T20:50:28.057121+0000","last_became_peered":"2024-03-11T20:50:28.057121+0000","last_unstale":"2024-03-11T20:53:38.319742+0000","last_undegraded":"2024-03-11T20:53:38.319742+0000","last_fullsized":"2024-03-11T20:53:38.319742+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:37.093963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:53:37.093963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:51.583205+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"84'9","reported_seq":118,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.330466+0000","last_change":"2024-03-11T20:53:03.452779+0000","last_active":"2024-03-11T20:53:38.330466+0000","last_peered":"2024-03-11T20:53:38.330466+0000","last_clean":"2024-03-11T20:53:38.330466+0000","last_became_active":"2024-03-11T20:50:30.089260+0000","last_became_peered":"2024-03-11T20:50:30.089260+0000","last_unstale":"2024-03-11T20:53:38.330466+0000","last_undegraded":"2024-03-11T20:53:38.330466+0000","last_fullsized":"2024-03-11T20:53:38.330466+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'9","last_scrub_stamp":"2024-03-11T20:53:03.452528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:53:03.452528+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:23.786382+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":98,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.323356+0000","last_change":"2024-03-11T20:53:00.551613+0000","last_active":"2024-03-11T20:53:38.323356+0000","last_peered":"2024-03-11T20:53:38.323356+0000","last_clean":"2024-03-11T20:53:38.323356+0000","last_became_active":"2024-03-11T20:50:32.200001+0000","last_became_peered":"2024-03-11T20:50:32.200001+0000","last_unstale":"2024-03-11T20:53:38.323356+0000","last_undegraded":"2024-03-11T20:53:38.323356+0000","last_fullsized":"2024-03-11T20:53:38.323356+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:00.551395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:35.670149+0000","last_clean_scrub_stamp":"2024-03-11T20:53:00.551395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:08.450926+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":94,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.882367+0000","last_change":"2024-03-11T20:52:54.879520+0000","last_active":"2024-03-11T20:53:38.882367+0000","last_peered":"2024-03-11T20:53:38.882367+0000","last_clean":"2024-03-11T20:53:38.882367+0000","last_became_active":"2024-03-11T20:50:34.326354+0000","last_became_peered":"2024-03-11T20:50:34.326354+0000","last_unstale":"2024-03-11T20:53:38.882367+0000","last_undegraded":"2024-03-11T20:53:38.882367+0000","last_fullsized":"2024-03-11T20:53:38.882367+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:54.879292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:52:54.879292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:00.895826+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"98'12","reported_seq":158,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:42.565055+0000","last_change":"2024-03-11T20:53:42.565055+0000","last_active":"2024-03-11T20:53:42.565055+0000","last_peered":"2024-03-11T20:53:42.565055+0000","last_clean":"2024-03-11T20:53:42.565055+0000","last_became_active":"2024-03-11T20:51:22.713613+0000","last_became_peered":"2024-03-11T20:51:22.713613+0000","last_unstale":"2024-03-11T20:53:42.565055+0000","last_undegraded":"2024-03-11T20:53:42.565055+0000","last_fullsized":"2024-03-11T20:53:42.565055+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'12","last_scrub_stamp":"2024-03-11T20:53:42.564785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:53:42.564785+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:55.923659+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":106,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.335329+0000","last_change":"2024-03-11T20:52:51.161508+0000","last_active":"2024-03-11T20:53:38.335329+0000","last_peered":"2024-03-11T20:53:38.335329+0000","last_clean":"2024-03-11T20:53:38.335329+0000","last_became_active":"2024-03-11T20:50:28.069267+0000","last_became_peered":"2024-03-11T20:50:28.069267+0000","last_unstale":"2024-03-11T20:53:38.335329+0000","last_undegraded":"2024-03-11T20:53:38.335329+0000","last_fullsized":"2024-03-11T20:53:38.335329+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:51.161281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:52:51.161281+0000","last_clean_scrub_stamp":"2024-03-11T20:52:51.161281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:57.333795+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"84'19","reported_seq":133,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.858915+0000","last_change":"2024-03-11T20:53:16.962607+0000","last_active":"2024-03-11T20:53:38.858915+0000","last_peered":"2024-03-11T20:53:38.858915+0000","last_clean":"2024-03-11T20:53:38.858915+0000","last_became_active":"2024-03-11T20:50:30.103666+0000","last_became_peered":"2024-03-11T20:50:30.103666+0000","last_unstale":"2024-03-11T20:53:38.858915+0000","last_undegraded":"2024-03-11T20:53:38.858915+0000","last_fullsized":"2024-03-11T20:53:38.858915+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'19","last_scrub_stamp":"2024-03-11T20:53:16.962364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:53:16.962364+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:37.903805+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":94,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.334361+0000","last_change":"2024-03-11T20:53:18.331836+0000","last_active":"2024-03-11T20:53:38.334361+0000","last_peered":"2024-03-11T20:53:38.334361+0000","last_clean":"2024-03-11T20:53:38.334361+0000","last_became_active":"2024-03-11T20:50:34.327139+0000","last_became_peered":"2024-03-11T20:50:34.327139+0000","last_unstale":"2024-03-11T20:53:38.334361+0000","last_undegraded":"2024-03-11T20:53:38.334361+0000","last_fullsized":"2024-03-11T20:53:38.334361+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:18.331616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:53:18.331616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:43.891860+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":106,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.883223+0000","last_change":"2024-03-11T20:53:08.797289+0000","last_active":"2024-03-11T20:53:38.883223+0000","last_peered":"2024-03-11T20:53:38.883223+0000","last_clean":"2024-03-11T20:53:38.883223+0000","last_became_active":"2024-03-11T20:50:28.049901+0000","last_became_peered":"2024-03-11T20:50:28.049901+0000","last_unstale":"2024-03-11T20:53:38.883223+0000","last_undegraded":"2024-03-11T20:53:38.883223+0000","last_fullsized":"2024-03-11T20:53:38.883223+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:08.797203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:53:08.797203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:12.120056+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"102'25","reported_seq":138,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.336267+0000","last_change":"2024-03-11T20:52:47.207949+0000","last_active":"2024-03-11T20:53:38.336267+0000","last_peered":"2024-03-11T20:53:38.336267+0000","last_clean":"2024-03-11T20:53:38.336267+0000","last_became_active":"2024-03-11T20:50:30.111819+0000","last_became_peered":"2024-03-11T20:50:30.111819+0000","last_unstale":"2024-03-11T20:53:38.336267+0000","last_undegraded":"2024-03-11T20:53:38.336267+0000","last_fullsized":"2024-03-11T20:53:38.336267+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"102'25","last_scrub_stamp":"2024-03-11T20:52:47.207710+0000","last_deep_scrub":"102'25","last_deep_scrub_stamp":"2024-03-11T20:51:35.187094+0000","last_clean_scrub_stamp":"2024-03-11T20:52:47.207710+0000","objects_scrubbed":8,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:55.994836+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":98,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.858569+0000","last_change":"2024-03-11T20:53:29.933875+0000","last_active":"2024-03-11T20:53:38.858569+0000","last_peered":"2024-03-11T20:53:38.858569+0000","last_clean":"2024-03-11T20:53:38.858569+0000","last_became_active":"2024-03-11T20:50:32.649877+0000","last_became_peered":"2024-03-11T20:50:32.649877+0000","last_unstale":"2024-03-11T20:53:38.858569+0000","last_undegraded":"2024-03-11T20:53:38.858569+0000","last_fullsized":"2024-03-11T20:53:38.858569+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:29.933643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:53:29.933643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:38.875482+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":93,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.321680+0000","last_change":"2024-03-11T20:52:46.946707+0000","last_active":"2024-03-11T20:53:38.321680+0000","last_peered":"2024-03-11T20:53:38.321680+0000","last_clean":"2024-03-11T20:53:38.321680+0000","last_became_active":"2024-03-11T20:50:34.319536+0000","last_became_peered":"2024-03-11T20:50:34.319536+0000","last_unstale":"2024-03-11T20:53:38.321680+0000","last_undegraded":"2024-03-11T20:53:38.321680+0000","last_fullsized":"2024-03-11T20:53:38.321680+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:46.946561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:52:46.946561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:52.449594+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":106,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.323588+0000","last_change":"2024-03-11T20:53:30.139626+0000","last_active":"2024-03-11T20:53:38.323588+0000","last_peered":"2024-03-11T20:53:38.323588+0000","last_clean":"2024-03-11T20:53:38.323588+0000","last_became_active":"2024-03-11T20:50:28.043982+0000","last_became_peered":"2024-03-11T20:50:28.043982+0000","last_unstale":"2024-03-11T20:53:38.323588+0000","last_undegraded":"2024-03-11T20:53:38.323588+0000","last_fullsized":"2024-03-11T20:53:38.323588+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:30.139539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:53:30.139539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:31.588939+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"84'10","reported_seq":117,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.837512+0000","last_change":"2024-03-11T20:53:12.823097+0000","last_active":"2024-03-11T20:53:38.837512+0000","last_peered":"2024-03-11T20:53:38.837512+0000","last_clean":"2024-03-11T20:53:38.837512+0000","last_became_active":"2024-03-11T20:50:30.109549+0000","last_became_peered":"2024-03-11T20:50:30.109549+0000","last_unstale":"2024-03-11T20:53:38.837512+0000","last_undegraded":"2024-03-11T20:53:38.837512+0000","last_fullsized":"2024-03-11T20:53:38.837512+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'10","last_scrub_stamp":"2024-03-11T20:53:12.822842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:53:12.822842+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:28.411634+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":97,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.323372+0000","last_change":"2024-03-11T20:52:51.931398+0000","last_active":"2024-03-11T20:53:38.323372+0000","last_peered":"2024-03-11T20:53:38.323372+0000","last_clean":"2024-03-11T20:53:38.323372+0000","last_became_active":"2024-03-11T20:50:32.194119+0000","last_became_peered":"2024-03-11T20:50:32.194119+0000","last_unstale":"2024-03-11T20:53:38.323372+0000","last_undegraded":"2024-03-11T20:53:38.323372+0000","last_fullsized":"2024-03-11T20:53:38.323372+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:51.931295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:52:51.931295+0000","last_clean_scrub_stamp":"2024-03-11T20:52:51.931295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:55.241599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":94,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.835186+0000","last_change":"2024-03-11T20:53:03.762967+0000","last_active":"2024-03-11T20:53:38.835186+0000","last_peered":"2024-03-11T20:53:38.835186+0000","last_clean":"2024-03-11T20:53:38.835186+0000","last_became_active":"2024-03-11T20:50:34.337173+0000","last_became_peered":"2024-03-11T20:50:34.337173+0000","last_unstale":"2024-03-11T20:53:38.835186+0000","last_undegraded":"2024-03-11T20:53:38.835186+0000","last_fullsized":"2024-03-11T20:53:38.835186+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:03.762882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:53:03.762882+0000","last_clean_scrub_stamp":"2024-03-11T20:53:03.762882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:29.931672+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":106,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.883164+0000","last_change":"2024-03-11T20:53:00.897188+0000","last_active":"2024-03-11T20:53:38.883164+0000","last_peered":"2024-03-11T20:53:38.883164+0000","last_clean":"2024-03-11T20:53:38.883164+0000","last_became_active":"2024-03-11T20:50:28.058942+0000","last_became_peered":"2024-03-11T20:50:28.058942+0000","last_unstale":"2024-03-11T20:53:38.883164+0000","last_undegraded":"2024-03-11T20:53:38.883164+0000","last_fullsized":"2024-03-11T20:53:38.883164+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:00.896807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:53:00.896807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:13.585784+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"84'15","reported_seq":127,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.336763+0000","last_change":"2024-03-11T20:53:09.345898+0000","last_active":"2024-03-11T20:53:38.336763+0000","last_peered":"2024-03-11T20:53:38.336763+0000","last_clean":"2024-03-11T20:53:38.336763+0000","last_became_active":"2024-03-11T20:50:30.101512+0000","last_became_peered":"2024-03-11T20:50:30.101512+0000","last_unstale":"2024-03-11T20:53:38.336763+0000","last_undegraded":"2024-03-11T20:53:38.336763+0000","last_fullsized":"2024-03-11T20:53:38.336763+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15","last_scrub_stamp":"2024-03-11T20:53:09.345405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:53:09.345405+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:28.384531+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":98,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:44.824836+0000","last_change":"2024-03-11T20:53:44.824870+0000","last_active":"2024-03-11T20:53:44.824836+0000","last_peered":"2024-03-11T20:53:44.824836+0000","last_clean":"2024-03-11T20:53:44.824836+0000","last_became_active":"2024-03-11T20:50:32.652854+0000","last_became_peered":"2024-03-11T20:50:32.652854+0000","last_unstale":"2024-03-11T20:53:44.824836+0000","last_undegraded":"2024-03-11T20:53:44.824836+0000","last_fullsized":"2024-03-11T20:53:44.824836+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:44.824748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:53:44.824748+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:08.788594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":173,"num_read_kb":168,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94,"ondisk_log_size":94,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":701,"num_read_kb":458,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":433,"ondisk_log_size":433,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":68,"seq":292057776178,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36660,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708492,"statfs":{"total":95995035648,"available":95957495808,"internally_reserved":0,"allocated":1355776,"data_stored":980421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":57,"seq":244813135932,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32484,"kb_used_data":1308,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712668,"statfs":{"total":95995035648,"available":95961772032,"internally_reserved":0,"allocated":1339392,"data_stored":980431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574959,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31916,"kb_used_data":736,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713236,"statfs":{"total":95995035648,"available":95962353664,"internally_reserved":0,"allocated":753664,"data_stored":390309,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738490,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32492,"kb_used_data":1316,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712660,"statfs":{"total":95995035648,"available":95961763840,"internally_reserved":0,"allocated":1347584,"data_stored":979363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378725,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31900,"kb_used_data":720,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713252,"statfs":{"total":95995035648,"available":95962370048,"internally_reserved":0,"allocated":737280,"data_stored":389157,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":28,"seq":120259084378,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27740,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717412,"statfs":{"total":95995035648,"available":95966629888,"internally_reserved":0,"allocated":741376,"data_stored":389533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":35,"seq":150323855441,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36096,"kb_used_data":760,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93709056,"statfs":{"total":95995035648,"available":95958073344,"internally_reserved":0,"allocated":778240,"data_stored":390923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":44,"seq":188978561094,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31900,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713252,"statfs":{"total":95995035648,"available":95962370048,"internally_reserved":0,"allocated":741376,"data_stored":389079,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T20:53:47.293 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T20:53:48.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:47 smithi146 bash[23809]: audit 2024-03-11T20:53:46.520618+0000 mgr.y (mgr.24380) 116 : audit 0 from='client.24566 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T20:53:48.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:47 smithi146 bash[23809]: cluster 2024-03-11T20:53:46.581426+0000 osd.4 (osd.4) 83 : cluster 0 4.1b scrub starts 2024-03-11T20:53:48.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:47 smithi146 bash[23809]: cluster 2024-03-11T20:53:46.583175+0000 osd.4 (osd.4) 84 : cluster 0 4.1b scrub ok 2024-03-11T20:53:48.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:47 smithi023 bash[21372]: audit 2024-03-11T20:53:46.520618+0000 mgr.y (mgr.24380) 116 : audit 0 from='client.24566 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T20:53:48.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:47 smithi023 bash[21372]: cluster 2024-03-11T20:53:46.581426+0000 osd.4 (osd.4) 83 : cluster 0 4.1b scrub starts 2024-03-11T20:53:48.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:47 smithi023 bash[21372]: cluster 2024-03-11T20:53:46.583175+0000 osd.4 (osd.4) 84 : cluster 0 4.1b scrub ok 2024-03-11T20:53:48.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:47 smithi023 bash[29146]: audit 2024-03-11T20:53:46.520618+0000 mgr.y (mgr.24380) 116 : audit 0 from='client.24566 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T20:53:48.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:47 smithi023 bash[29146]: cluster 2024-03-11T20:53:46.581426+0000 osd.4 (osd.4) 83 : cluster 0 4.1b scrub starts 2024-03-11T20:53:48.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:47 smithi023 bash[29146]: cluster 2024-03-11T20:53:46.583175+0000 osd.4 (osd.4) 84 : cluster 0 4.1b scrub ok 2024-03-11T20:53:48.784 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:53:48 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:20:53:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T20:53:49.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:48 smithi146 bash[23809]: cluster 2024-03-11T20:53:47.220992+0000 osd.7 (osd.7) 81 : cluster 0 6.15 deep-scrub starts 2024-03-11T20:53:49.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:48 smithi146 bash[23809]: cluster 2024-03-11T20:53:47.223414+0000 osd.7 (osd.7) 82 : cluster 0 6.15 deep-scrub ok 2024-03-11T20:53:49.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:48 smithi146 bash[23809]: cluster 2024-03-11T20:53:47.354222+0000 osd.3 (osd.3) 99 : cluster 0 3.8 deep-scrub starts 2024-03-11T20:53:49.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:48 smithi146 bash[23809]: cluster 2024-03-11T20:53:47.364697+0000 osd.3 (osd.3) 100 : cluster 0 3.8 deep-scrub ok 2024-03-11T20:53:49.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:48 smithi146 bash[23809]: cluster 2024-03-11T20:53:47.870873+0000 mgr.y (mgr.24380) 117 : cluster 0 pgmap v85: 132 pgs: 132 active+clean; 582 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:53:49.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:48 smithi023 bash[21372]: cluster 2024-03-11T20:53:47.220992+0000 osd.7 (osd.7) 81 : cluster 0 6.15 deep-scrub starts 2024-03-11T20:53:49.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:48 smithi023 bash[21372]: cluster 2024-03-11T20:53:47.223414+0000 osd.7 (osd.7) 82 : cluster 0 6.15 deep-scrub ok 2024-03-11T20:53:49.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:48 smithi023 bash[21372]: cluster 2024-03-11T20:53:47.354222+0000 osd.3 (osd.3) 99 : cluster 0 3.8 deep-scrub starts 2024-03-11T20:53:49.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:48 smithi023 bash[21372]: cluster 2024-03-11T20:53:47.364697+0000 osd.3 (osd.3) 100 : cluster 0 3.8 deep-scrub ok 2024-03-11T20:53:49.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:48 smithi023 bash[21372]: cluster 2024-03-11T20:53:47.870873+0000 mgr.y (mgr.24380) 117 : cluster 0 pgmap v85: 132 pgs: 132 active+clean; 582 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:53:49.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:48 smithi023 bash[29146]: cluster 2024-03-11T20:53:47.220992+0000 osd.7 (osd.7) 81 : cluster 0 6.15 deep-scrub starts 2024-03-11T20:53:49.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:48 smithi023 bash[29146]: cluster 2024-03-11T20:53:47.223414+0000 osd.7 (osd.7) 82 : cluster 0 6.15 deep-scrub ok 2024-03-11T20:53:49.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:48 smithi023 bash[29146]: cluster 2024-03-11T20:53:47.354222+0000 osd.3 (osd.3) 99 : cluster 0 3.8 deep-scrub starts 2024-03-11T20:53:49.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:48 smithi023 bash[29146]: cluster 2024-03-11T20:53:47.364697+0000 osd.3 (osd.3) 100 : cluster 0 3.8 deep-scrub ok 2024-03-11T20:53:49.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:48 smithi023 bash[29146]: cluster 2024-03-11T20:53:47.870873+0000 mgr.y (mgr.24380) 117 : cluster 0 pgmap v85: 132 pgs: 132 active+clean; 582 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:53:50.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:49 smithi146 bash[23809]: audit 2024-03-11T20:53:49.178247+0000 mon.a (mon.0) 926 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:53:50.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:49 smithi023 bash[21372]: audit 2024-03-11T20:53:49.178247+0000 mon.a (mon.0) 926 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:53:50.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:49 smithi023 bash[29146]: audit 2024-03-11T20:53:49.178247+0000 mon.a (mon.0) 926 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:53:51.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:50 smithi146 bash[23809]: cluster 2024-03-11T20:53:49.835001+0000 osd.5 (osd.5) 79 : cluster 0 2.3 deep-scrub starts 2024-03-11T20:53:51.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:50 smithi146 bash[23809]: cluster 2024-03-11T20:53:49.837612+0000 osd.5 (osd.5) 80 : cluster 0 2.3 deep-scrub ok 2024-03-11T20:53:51.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:50 smithi146 bash[23809]: cluster 2024-03-11T20:53:49.872781+0000 mgr.y (mgr.24380) 118 : cluster 0 pgmap v86: 132 pgs: 1 active+clean+scrubbing+deep, 131 active+clean; 582 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-11T20:53:51.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:50 smithi146 bash[23809]: cluster 2024-03-11T20:53:50.024237+0000 osd.1 (osd.1) 81 : cluster 0 3.0 scrub starts 2024-03-11T20:53:51.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:50 smithi146 bash[23809]: cluster 2024-03-11T20:53:50.029170+0000 osd.1 (osd.1) 82 : cluster 0 3.0 scrub ok 2024-03-11T20:53:51.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:50 smithi023 bash[21372]: cluster 2024-03-11T20:53:49.835001+0000 osd.5 (osd.5) 79 : cluster 0 2.3 deep-scrub starts 2024-03-11T20:53:51.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:50 smithi023 bash[21372]: cluster 2024-03-11T20:53:49.837612+0000 osd.5 (osd.5) 80 : cluster 0 2.3 deep-scrub ok 2024-03-11T20:53:51.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:50 smithi023 bash[21372]: cluster 2024-03-11T20:53:49.872781+0000 mgr.y (mgr.24380) 118 : cluster 0 pgmap v86: 132 pgs: 1 active+clean+scrubbing+deep, 131 active+clean; 582 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-11T20:53:51.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:50 smithi023 bash[21372]: cluster 2024-03-11T20:53:50.024237+0000 osd.1 (osd.1) 81 : cluster 0 3.0 scrub starts 2024-03-11T20:53:51.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:50 smithi023 bash[21372]: cluster 2024-03-11T20:53:50.029170+0000 osd.1 (osd.1) 82 : cluster 0 3.0 scrub ok 2024-03-11T20:53:51.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:50 smithi023 bash[29146]: cluster 2024-03-11T20:53:49.835001+0000 osd.5 (osd.5) 79 : cluster 0 2.3 deep-scrub starts 2024-03-11T20:53:51.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:50 smithi023 bash[29146]: cluster 2024-03-11T20:53:49.837612+0000 osd.5 (osd.5) 80 : cluster 0 2.3 deep-scrub ok 2024-03-11T20:53:51.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:50 smithi023 bash[29146]: cluster 2024-03-11T20:53:49.872781+0000 mgr.y (mgr.24380) 118 : cluster 0 pgmap v86: 132 pgs: 1 active+clean+scrubbing+deep, 131 active+clean; 582 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-11T20:53:51.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:50 smithi023 bash[29146]: cluster 2024-03-11T20:53:50.024237+0000 osd.1 (osd.1) 81 : cluster 0 3.0 scrub starts 2024-03-11T20:53:51.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:50 smithi023 bash[29146]: cluster 2024-03-11T20:53:50.029170+0000 osd.1 (osd.1) 82 : cluster 0 3.0 scrub ok 2024-03-11T20:53:52.088 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:53:52.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:51 smithi146 bash[23809]: cluster 2024-03-11T20:53:50.596443+0000 osd.4 (osd.4) 85 : cluster 0 4.1a deep-scrub starts 2024-03-11T20:53:52.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:51 smithi146 bash[23809]: cluster 2024-03-11T20:53:50.599018+0000 osd.4 (osd.4) 86 : cluster 0 4.1a deep-scrub ok 2024-03-11T20:53:52.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:51 smithi023 bash[21372]: cluster 2024-03-11T20:53:50.596443+0000 osd.4 (osd.4) 85 : cluster 0 4.1a deep-scrub starts 2024-03-11T20:53:52.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:51 smithi023 bash[21372]: cluster 2024-03-11T20:53:50.599018+0000 osd.4 (osd.4) 86 : cluster 0 4.1a deep-scrub ok 2024-03-11T20:53:52.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:51 smithi023 bash[29146]: cluster 2024-03-11T20:53:50.596443+0000 osd.4 (osd.4) 85 : cluster 0 4.1a deep-scrub starts 2024-03-11T20:53:52.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:51 smithi023 bash[29146]: cluster 2024-03-11T20:53:50.599018+0000 osd.4 (osd.4) 86 : cluster 0 4.1a deep-scrub ok 2024-03-11T20:53:53.217 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:52 smithi023 bash[29146]: cluster 2024-03-11T20:53:51.873532+0000 mgr.y (mgr.24380) 119 : cluster 0 pgmap v87: 132 pgs: 1 active+clean+scrubbing+deep, 131 active+clean; 582 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T20:53:53.218 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:52 smithi023 bash[21372]: cluster 2024-03-11T20:53:51.873532+0000 mgr.y (mgr.24380) 119 : cluster 0 pgmap v87: 132 pgs: 1 active+clean+scrubbing+deep, 131 active+clean; 582 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T20:53:53.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:52 smithi146 bash[23809]: cluster 2024-03-11T20:53:51.873532+0000 mgr.y (mgr.24380) 119 : cluster 0 pgmap v87: 132 pgs: 1 active+clean+scrubbing+deep, 131 active+clean; 582 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T20:53:54.697 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-11T20:53:54.701 INFO:teuthology.orchestra.run.smithi023.stderr:dumped all 2024-03-11T20:53:55.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:54 smithi023 bash[21372]: cluster 2024-03-11T20:53:53.874331+0000 mgr.y (mgr.24380) 120 : cluster 0 pgmap v88: 132 pgs: 1 active+clean+scrubbing+deep, 131 active+clean; 582 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T20:53:55.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:54 smithi023 bash[29146]: cluster 2024-03-11T20:53:53.874331+0000 mgr.y (mgr.24380) 120 : cluster 0 pgmap v88: 132 pgs: 1 active+clean+scrubbing+deep, 131 active+clean; 582 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T20:53:55.260 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 20:53:54 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T20:53:55.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:54 smithi146 bash[23809]: cluster 2024-03-11T20:53:53.874331+0000 mgr.y (mgr.24380) 120 : cluster 0 pgmap v88: 132 pgs: 1 active+clean+scrubbing+deep, 131 active+clean; 582 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T20:53:55.427 INFO:teuthology.orchestra.run.smithi023.stdout:{"pg_ready":true,"pg_map":{"version":88,"stamp":"2024-03-11T20:53:53.873851+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596392,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1128,"num_read_kb":974,"num_write":615,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":678,"ondisk_log_size":678,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":261180,"kb_used_data":7612,"kb_used_omap":12,"kb_used_meta":253491,"kb_avail":749700036,"statfs":{"total":767960285184,"available":767692836864,"internally_reserved":0,"allocated":7794688,"data_stored":4889216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":259575346},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005798"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":94,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.329454+0000","last_change":"2024-03-11T20:53:05.510279+0000","last_active":"2024-03-11T20:53:38.329454+0000","last_peered":"2024-03-11T20:53:38.329454+0000","last_clean":"2024-03-11T20:53:38.329454+0000","last_became_active":"2024-03-11T20:50:34.338390+0000","last_became_peered":"2024-03-11T20:50:34.338390+0000","last_unstale":"2024-03-11T20:53:38.329454+0000","last_undegraded":"2024-03-11T20:53:38.329454+0000","last_fullsized":"2024-03-11T20:53:38.329454+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:05.510060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:53:05.510060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:21.840101+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":98,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.329173+0000","last_change":"2024-03-11T20:53:18.443416+0000","last_active":"2024-03-11T20:53:38.329173+0000","last_peered":"2024-03-11T20:53:38.329173+0000","last_clean":"2024-03-11T20:53:38.329173+0000","last_became_active":"2024-03-11T20:50:32.210194+0000","last_became_peered":"2024-03-11T20:50:32.210194+0000","last_unstale":"2024-03-11T20:53:38.329173+0000","last_undegraded":"2024-03-11T20:53:38.329173+0000","last_fullsized":"2024-03-11T20:53:38.329173+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:18.443215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:53:18.443215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:23.962891+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":105,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.325201+0000","last_change":"2024-03-11T20:53:29.965205+0000","last_active":"2024-03-11T20:53:38.325201+0000","last_peered":"2024-03-11T20:53:38.325201+0000","last_clean":"2024-03-11T20:53:38.325201+0000","last_became_active":"2024-03-11T20:50:28.061739+0000","last_became_peered":"2024-03-11T20:50:28.061739+0000","last_unstale":"2024-03-11T20:53:38.325201+0000","last_undegraded":"2024-03-11T20:53:38.325201+0000","last_fullsized":"2024-03-11T20:53:38.325201+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:29.964907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:53:29.964907+0000","last_clean_scrub_stamp":"2024-03-11T20:53:29.964907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:53.399727+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"84'10","reported_seq":117,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.331551+0000","last_change":"2024-03-11T20:53:11.430067+0000","last_active":"2024-03-11T20:53:38.331551+0000","last_peered":"2024-03-11T20:53:38.331551+0000","last_clean":"2024-03-11T20:53:38.331551+0000","last_became_active":"2024-03-11T20:50:30.101721+0000","last_became_peered":"2024-03-11T20:50:30.101721+0000","last_unstale":"2024-03-11T20:53:38.331551+0000","last_undegraded":"2024-03-11T20:53:38.331551+0000","last_fullsized":"2024-03-11T20:53:38.331551+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'10","last_scrub_stamp":"2024-03-11T20:53:11.429944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:53:11.429944+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:29.275642+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":94,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.322785+0000","last_change":"2024-03-11T20:53:07.560827+0000","last_active":"2024-03-11T20:53:38.322785+0000","last_peered":"2024-03-11T20:53:38.322785+0000","last_clean":"2024-03-11T20:53:38.322785+0000","last_became_active":"2024-03-11T20:50:34.318627+0000","last_became_peered":"2024-03-11T20:50:34.318627+0000","last_unstale":"2024-03-11T20:53:38.322785+0000","last_undegraded":"2024-03-11T20:53:38.322785+0000","last_fullsized":"2024-03-11T20:53:38.322785+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:07.560731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:55.545758+0000","last_clean_scrub_stamp":"2024-03-11T20:53:07.560731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:36.783905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":98,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.322569+0000","last_change":"2024-03-11T20:52:49.461204+0000","last_active":"2024-03-11T20:53:38.322569+0000","last_peered":"2024-03-11T20:53:38.322569+0000","last_clean":"2024-03-11T20:53:38.322569+0000","last_became_active":"2024-03-11T20:50:32.197413+0000","last_became_peered":"2024-03-11T20:50:32.197413+0000","last_unstale":"2024-03-11T20:53:38.322569+0000","last_undegraded":"2024-03-11T20:53:38.322569+0000","last_fullsized":"2024-03-11T20:53:38.322569+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:49.460993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:52:49.460993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:04.898406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":98,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.331025+0000","last_change":"2024-03-11T20:52:26.396960+0000","last_active":"2024-03-11T20:53:38.331025+0000","last_peered":"2024-03-11T20:53:38.331025+0000","last_clean":"2024-03-11T20:53:38.331025+0000","last_became_active":"2024-03-11T20:50:28.058274+0000","last_became_peered":"2024-03-11T20:50:28.058274+0000","last_unstale":"2024-03-11T20:53:38.331025+0000","last_undegraded":"2024-03-11T20:53:38.331025+0000","last_fullsized":"2024-03-11T20:53:38.331025+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:26.396865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:52:26.396865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:52.964932+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"84'11","reported_seq":120,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:44.981447+0000","last_change":"2024-03-11T20:53:44.981477+0000","last_active":"2024-03-11T20:53:44.981447+0000","last_peered":"2024-03-11T20:53:44.981447+0000","last_clean":"2024-03-11T20:53:44.981447+0000","last_became_active":"2024-03-11T20:50:30.092638+0000","last_became_peered":"2024-03-11T20:50:30.092638+0000","last_unstale":"2024-03-11T20:53:44.981447+0000","last_undegraded":"2024-03-11T20:53:44.981447+0000","last_fullsized":"2024-03-11T20:53:44.981447+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T20:53:44.981383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:53:44.981383+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:49.030024+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":94,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.835084+0000","last_change":"2024-03-11T20:53:07.792944+0000","last_active":"2024-03-11T20:53:38.835084+0000","last_peered":"2024-03-11T20:53:38.835084+0000","last_clean":"2024-03-11T20:53:38.835084+0000","last_became_active":"2024-03-11T20:50:34.334495+0000","last_became_peered":"2024-03-11T20:50:34.334495+0000","last_unstale":"2024-03-11T20:53:38.835084+0000","last_undegraded":"2024-03-11T20:53:38.835084+0000","last_fullsized":"2024-03-11T20:53:38.835084+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:07.792697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:53:07.792697+0000","last_clean_scrub_stamp":"2024-03-11T20:53:07.792697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:23.012728+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":98,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:46.583334+0000","last_change":"2024-03-11T20:53:46.583392+0000","last_active":"2024-03-11T20:53:46.583334+0000","last_peered":"2024-03-11T20:53:46.583334+0000","last_clean":"2024-03-11T20:53:46.583334+0000","last_became_active":"2024-03-11T20:50:32.199561+0000","last_became_peered":"2024-03-11T20:50:32.199561+0000","last_unstale":"2024-03-11T20:53:46.583334+0000","last_undegraded":"2024-03-11T20:53:46.583334+0000","last_fullsized":"2024-03-11T20:53:46.583334+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:46.583185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:53:46.583185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:13.629525+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":106,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.336031+0000","last_change":"2024-03-11T20:53:29.323499+0000","last_active":"2024-03-11T20:53:38.336031+0000","last_peered":"2024-03-11T20:53:38.336031+0000","last_clean":"2024-03-11T20:53:38.336031+0000","last_became_active":"2024-03-11T20:50:28.079225+0000","last_became_peered":"2024-03-11T20:50:28.079225+0000","last_unstale":"2024-03-11T20:53:38.336031+0000","last_undegraded":"2024-03-11T20:53:38.336031+0000","last_fullsized":"2024-03-11T20:53:38.336031+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:29.323273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:53:29.323273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:36.362049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"84'15","reported_seq":127,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.837970+0000","last_change":"2024-03-11T20:53:29.817448+0000","last_active":"2024-03-11T20:53:38.837970+0000","last_peered":"2024-03-11T20:53:38.837970+0000","last_clean":"2024-03-11T20:53:38.837970+0000","last_became_active":"2024-03-11T20:50:30.105354+0000","last_became_peered":"2024-03-11T20:50:30.105354+0000","last_unstale":"2024-03-11T20:53:38.837970+0000","last_undegraded":"2024-03-11T20:53:38.837970+0000","last_fullsized":"2024-03-11T20:53:38.837970+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15","last_scrub_stamp":"2024-03-11T20:53:29.817112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:53:29.817112+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:47.069548+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":93,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.322556+0000","last_change":"2024-03-11T20:53:14.168525+0000","last_active":"2024-03-11T20:53:38.322556+0000","last_peered":"2024-03-11T20:53:38.322556+0000","last_clean":"2024-03-11T20:53:38.322556+0000","last_became_active":"2024-03-11T20:50:34.333230+0000","last_became_peered":"2024-03-11T20:50:34.333230+0000","last_unstale":"2024-03-11T20:53:38.322556+0000","last_undegraded":"2024-03-11T20:53:38.322556+0000","last_fullsized":"2024-03-11T20:53:38.322556+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:14.168304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:53:14.168304+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:19.290761+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":90,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.323746+0000","last_change":"2024-03-11T20:52:19.502063+0000","last_active":"2024-03-11T20:53:38.323746+0000","last_peered":"2024-03-11T20:53:38.323746+0000","last_clean":"2024-03-11T20:53:38.323746+0000","last_became_active":"2024-03-11T20:50:32.200126+0000","last_became_peered":"2024-03-11T20:50:32.200126+0000","last_unstale":"2024-03-11T20:53:38.323746+0000","last_undegraded":"2024-03-11T20:53:38.323746+0000","last_fullsized":"2024-03-11T20:53:38.323746+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:19.501832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:52:19.501832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":106,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.336128+0000","last_change":"2024-03-11T20:53:00.210016+0000","last_active":"2024-03-11T20:53:38.336128+0000","last_peered":"2024-03-11T20:53:38.336128+0000","last_clean":"2024-03-11T20:53:38.336128+0000","last_became_active":"2024-03-11T20:50:28.068978+0000","last_became_peered":"2024-03-11T20:50:28.068978+0000","last_unstale":"2024-03-11T20:53:38.336128+0000","last_undegraded":"2024-03-11T20:53:38.336128+0000","last_fullsized":"2024-03-11T20:53:38.336128+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:00.209791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:53:00.209791+0000","last_clean_scrub_stamp":"2024-03-11T20:53:00.209791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:01.669184+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"110'17","reported_seq":130,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.837592+0000","last_change":"2024-03-11T20:53:04.802853+0000","last_active":"2024-03-11T20:53:38.837592+0000","last_peered":"2024-03-11T20:53:38.837592+0000","last_clean":"2024-03-11T20:53:38.837592+0000","last_became_active":"2024-03-11T20:50:30.108364+0000","last_became_peered":"2024-03-11T20:50:30.108364+0000","last_unstale":"2024-03-11T20:53:38.837592+0000","last_undegraded":"2024-03-11T20:53:38.837592+0000","last_fullsized":"2024-03-11T20:53:38.837592+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'17","last_scrub_stamp":"2024-03-11T20:53:04.802488+0000","last_deep_scrub":"110'17","last_deep_scrub_stamp":"2024-03-11T20:53:04.802488+0000","last_clean_scrub_stamp":"2024-03-11T20:53:04.802488+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:26.988743+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":94,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.331094+0000","last_change":"2024-03-11T20:53:38.215774+0000","last_active":"2024-03-11T20:53:38.331094+0000","last_peered":"2024-03-11T20:53:38.331094+0000","last_clean":"2024-03-11T20:53:38.331094+0000","last_became_active":"2024-03-11T20:50:34.334558+0000","last_became_peered":"2024-03-11T20:50:34.334558+0000","last_unstale":"2024-03-11T20:53:38.331094+0000","last_undegraded":"2024-03-11T20:53:38.331094+0000","last_fullsized":"2024-03-11T20:53:38.331094+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:38.215565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:53:38.215565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:03.143663+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":98,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.330248+0000","last_change":"2024-03-11T20:52:59.455887+0000","last_active":"2024-03-11T20:53:38.330248+0000","last_peered":"2024-03-11T20:53:38.330248+0000","last_clean":"2024-03-11T20:53:38.330248+0000","last_became_active":"2024-03-11T20:50:32.202533+0000","last_became_peered":"2024-03-11T20:50:32.202533+0000","last_unstale":"2024-03-11T20:53:38.330248+0000","last_undegraded":"2024-03-11T20:53:38.330248+0000","last_fullsized":"2024-03-11T20:53:38.330248+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:59.455677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:52:59.455677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:29.453221+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"77'1","reported_seq":114,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.330139+0000","last_change":"2024-03-11T20:53:30.341375+0000","last_active":"2024-03-11T20:53:38.330139+0000","last_peered":"2024-03-11T20:53:38.330139+0000","last_clean":"2024-03-11T20:53:38.330139+0000","last_became_active":"2024-03-11T20:50:28.068840+0000","last_became_peered":"2024-03-11T20:50:28.068840+0000","last_unstale":"2024-03-11T20:53:38.330139+0000","last_undegraded":"2024-03-11T20:53:38.330139+0000","last_fullsized":"2024-03-11T20:53:38.330139+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-03-11T20:53:30.341011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:53:30.341011+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:38.426341+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"84'9","reported_seq":118,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.323124+0000","last_change":"2024-03-11T20:53:34.546830+0000","last_active":"2024-03-11T20:53:38.323124+0000","last_peered":"2024-03-11T20:53:38.323124+0000","last_clean":"2024-03-11T20:53:38.323124+0000","last_became_active":"2024-03-11T20:50:30.086169+0000","last_became_peered":"2024-03-11T20:50:30.086169+0000","last_unstale":"2024-03-11T20:53:38.323124+0000","last_undegraded":"2024-03-11T20:53:38.323124+0000","last_fullsized":"2024-03-11T20:53:38.323124+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'9","last_scrub_stamp":"2024-03-11T20:53:34.546587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:53:34.546587+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:58.297447+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":94,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.324864+0000","last_change":"2024-03-11T20:53:26.444831+0000","last_active":"2024-03-11T20:53:38.324864+0000","last_peered":"2024-03-11T20:53:38.324864+0000","last_clean":"2024-03-11T20:53:38.324864+0000","last_became_active":"2024-03-11T20:50:34.321714+0000","last_became_peered":"2024-03-11T20:50:34.321714+0000","last_unstale":"2024-03-11T20:53:38.324864+0000","last_undegraded":"2024-03-11T20:53:38.324864+0000","last_fullsized":"2024-03-11T20:53:38.324864+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:26.444527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:53:26.444527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:52.656618+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":98,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.882936+0000","last_change":"2024-03-11T20:53:15.798662+0000","last_active":"2024-03-11T20:53:38.882936+0000","last_peered":"2024-03-11T20:53:38.882936+0000","last_clean":"2024-03-11T20:53:38.882936+0000","last_became_active":"2024-03-11T20:50:32.192704+0000","last_became_peered":"2024-03-11T20:50:32.192704+0000","last_unstale":"2024-03-11T20:53:38.882936+0000","last_undegraded":"2024-03-11T20:53:38.882936+0000","last_fullsized":"2024-03-11T20:53:38.882936+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:15.798438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:53:15.798438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:29.061321+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":106,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.858837+0000","last_change":"2024-03-11T20:53:20.946464+0000","last_active":"2024-03-11T20:53:38.858837+0000","last_peered":"2024-03-11T20:53:38.858837+0000","last_clean":"2024-03-11T20:53:38.858837+0000","last_became_active":"2024-03-11T20:50:28.081538+0000","last_became_peered":"2024-03-11T20:50:28.081538+0000","last_unstale":"2024-03-11T20:53:38.858837+0000","last_undegraded":"2024-03-11T20:53:38.858837+0000","last_fullsized":"2024-03-11T20:53:38.858837+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:20.946227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:53:20.946227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:50.298331+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"84'5","reported_seq":111,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.322860+0000","last_change":"2024-03-11T20:53:31.929747+0000","last_active":"2024-03-11T20:53:38.322860+0000","last_peered":"2024-03-11T20:53:38.322860+0000","last_clean":"2024-03-11T20:53:38.322860+0000","last_became_active":"2024-03-11T20:50:30.099866+0000","last_became_peered":"2024-03-11T20:50:30.099866+0000","last_unstale":"2024-03-11T20:53:38.322860+0000","last_undegraded":"2024-03-11T20:53:38.322860+0000","last_fullsized":"2024-03-11T20:53:38.322860+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'5","last_scrub_stamp":"2024-03-11T20:53:31.929501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:53:31.929501+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:36.645523+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":94,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.323016+0000","last_change":"2024-03-11T20:53:21.162869+0000","last_active":"2024-03-11T20:53:38.323016+0000","last_peered":"2024-03-11T20:53:38.323016+0000","last_clean":"2024-03-11T20:53:38.323016+0000","last_became_active":"2024-03-11T20:50:34.307730+0000","last_became_peered":"2024-03-11T20:50:34.307730+0000","last_unstale":"2024-03-11T20:53:38.323016+0000","last_undegraded":"2024-03-11T20:53:38.323016+0000","last_fullsized":"2024-03-11T20:53:38.323016+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:21.162653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:53:21.162653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:34.261060+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1f","version":"114'12","reported_seq":156,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:43.292426+0000","last_change":"2024-03-11T20:53:30.906029+0000","last_active":"2024-03-11T20:53:43.292426+0000","last_peered":"2024-03-11T20:53:43.292426+0000","last_clean":"2024-03-11T20:53:43.292426+0000","last_became_active":"2024-03-11T20:50:32.200195+0000","last_became_peered":"2024-03-11T20:50:32.200195+0000","last_unstale":"2024-03-11T20:53:43.292426+0000","last_undegraded":"2024-03-11T20:53:43.292426+0000","last_fullsized":"2024-03-11T20:53:43.292426+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'12","last_scrub_stamp":"2024-03-11T20:53:30.905791+0000","last_deep_scrub":"114'12","last_deep_scrub_stamp":"2024-03-11T20:53:30.905791+0000","last_clean_scrub_stamp":"2024-03-11T20:53:30.905791+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:59.478960+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"77'1","reported_seq":99,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.329211+0000","last_change":"2024-03-11T20:52:24.397205+0000","last_active":"2024-03-11T20:53:38.329211+0000","last_peered":"2024-03-11T20:53:38.329211+0000","last_clean":"2024-03-11T20:53:38.329211+0000","last_became_active":"2024-03-11T20:50:28.068827+0000","last_became_peered":"2024-03-11T20:50:28.068827+0000","last_unstale":"2024-03-11T20:53:38.329211+0000","last_undegraded":"2024-03-11T20:53:38.329211+0000","last_fullsized":"2024-03-11T20:53:38.329211+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-03-11T20:52:24.396881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:52:24.396881+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:53.894381+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"84'9","reported_seq":118,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.328853+0000","last_change":"2024-03-11T20:53:23.347800+0000","last_active":"2024-03-11T20:53:38.328853+0000","last_peered":"2024-03-11T20:53:38.328853+0000","last_clean":"2024-03-11T20:53:38.328853+0000","last_became_active":"2024-03-11T20:50:30.083936+0000","last_became_peered":"2024-03-11T20:50:30.083936+0000","last_unstale":"2024-03-11T20:53:38.328853+0000","last_undegraded":"2024-03-11T20:53:38.328853+0000","last_fullsized":"2024-03-11T20:53:38.328853+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'9","last_scrub_stamp":"2024-03-11T20:53:23.347531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:53:23.347531+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:42.009935+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"105'2","reported_seq":97,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.335367+0000","last_change":"2024-03-11T20:53:34.350049+0000","last_active":"2024-03-11T20:53:38.335367+0000","last_peered":"2024-03-11T20:53:38.335367+0000","last_clean":"2024-03-11T20:53:38.335367+0000","last_became_active":"2024-03-11T20:50:34.327356+0000","last_became_peered":"2024-03-11T20:50:34.327356+0000","last_unstale":"2024-03-11T20:53:38.335367+0000","last_undegraded":"2024-03-11T20:53:38.335367+0000","last_fullsized":"2024-03-11T20:53:38.335367+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'2","last_scrub_stamp":"2024-03-11T20:53:34.349639+0000","last_deep_scrub":"105'2","last_deep_scrub_stamp":"2024-03-11T20:52:18.098911+0000","last_clean_scrub_stamp":"2024-03-11T20:53:34.349639+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:49.549358+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":97,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.324468+0000","last_change":"2024-03-11T20:53:10.062579+0000","last_active":"2024-03-11T20:53:38.324468+0000","last_peered":"2024-03-11T20:53:38.324468+0000","last_clean":"2024-03-11T20:53:38.324468+0000","last_became_active":"2024-03-11T20:50:32.648869+0000","last_became_peered":"2024-03-11T20:50:32.648869+0000","last_unstale":"2024-03-11T20:53:38.324468+0000","last_undegraded":"2024-03-11T20:53:38.324468+0000","last_fullsized":"2024-03-11T20:53:38.324468+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:10.062346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:53:10.062346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:39.641031+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":106,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.836132+0000","last_change":"2024-03-11T20:53:31.821391+0000","last_active":"2024-03-11T20:53:38.836132+0000","last_peered":"2024-03-11T20:53:38.836132+0000","last_clean":"2024-03-11T20:53:38.836132+0000","last_became_active":"2024-03-11T20:50:28.066124+0000","last_became_peered":"2024-03-11T20:50:28.066124+0000","last_unstale":"2024-03-11T20:53:38.836132+0000","last_undegraded":"2024-03-11T20:53:38.836132+0000","last_fullsized":"2024-03-11T20:53:38.836132+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:31.821104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:52:01.880052+0000","last_clean_scrub_stamp":"2024-03-11T20:53:31.821104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:35.848818+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"84'15","reported_seq":127,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.320438+0000","last_change":"2024-03-11T20:53:13.139250+0000","last_active":"2024-03-11T20:53:38.320438+0000","last_peered":"2024-03-11T20:53:38.320438+0000","last_clean":"2024-03-11T20:53:38.320438+0000","last_became_active":"2024-03-11T20:50:30.088446+0000","last_became_peered":"2024-03-11T20:50:30.088446+0000","last_unstale":"2024-03-11T20:53:38.320438+0000","last_undegraded":"2024-03-11T20:53:38.320438+0000","last_fullsized":"2024-03-11T20:53:38.320438+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15","last_scrub_stamp":"2024-03-11T20:53:13.138917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:53:13.138917+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:28.013588+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":94,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.331012+0000","last_change":"2024-03-11T20:53:28.358235+0000","last_active":"2024-03-11T20:53:38.331012+0000","last_peered":"2024-03-11T20:53:38.331012+0000","last_clean":"2024-03-11T20:53:38.331012+0000","last_became_active":"2024-03-11T20:50:34.337878+0000","last_became_peered":"2024-03-11T20:50:34.337878+0000","last_unstale":"2024-03-11T20:53:38.331012+0000","last_undegraded":"2024-03-11T20:53:38.331012+0000","last_fullsized":"2024-03-11T20:53:38.331012+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:28.358134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:52:11.462480+0000","last_clean_scrub_stamp":"2024-03-11T20:53:28.358134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:45.511381+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":98,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.329908+0000","last_change":"2024-03-11T20:53:02.400889+0000","last_active":"2024-03-11T20:53:38.329908+0000","last_peered":"2024-03-11T20:53:38.329908+0000","last_clean":"2024-03-11T20:53:38.329908+0000","last_became_active":"2024-03-11T20:50:32.646323+0000","last_became_peered":"2024-03-11T20:50:32.646323+0000","last_unstale":"2024-03-11T20:53:38.329908+0000","last_undegraded":"2024-03-11T20:53:38.329908+0000","last_fullsized":"2024-03-11T20:53:38.329908+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:02.400685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:53:02.400685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:04.399694+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":106,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.859866+0000","last_change":"2024-03-11T20:53:18.916363+0000","last_active":"2024-03-11T20:53:38.859866+0000","last_peered":"2024-03-11T20:53:38.859866+0000","last_clean":"2024-03-11T20:53:38.859866+0000","last_became_active":"2024-03-11T20:50:28.086513+0000","last_became_peered":"2024-03-11T20:50:28.086513+0000","last_unstale":"2024-03-11T20:53:38.859866+0000","last_undegraded":"2024-03-11T20:53:38.859866+0000","last_fullsized":"2024-03-11T20:53:38.859866+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:18.916152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:53:18.916152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:20.894545+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"84'9","reported_seq":118,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.837473+0000","last_change":"2024-03-11T20:53:17.784263+0000","last_active":"2024-03-11T20:53:38.837473+0000","last_peered":"2024-03-11T20:53:38.837473+0000","last_clean":"2024-03-11T20:53:38.837473+0000","last_became_active":"2024-03-11T20:50:30.109869+0000","last_became_peered":"2024-03-11T20:50:30.109869+0000","last_unstale":"2024-03-11T20:53:38.837473+0000","last_undegraded":"2024-03-11T20:53:38.837473+0000","last_fullsized":"2024-03-11T20:53:38.837473+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'9","last_scrub_stamp":"2024-03-11T20:53:17.784035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:53:17.784035+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:30.540981+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":94,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.334015+0000","last_change":"2024-03-11T20:53:04.321530+0000","last_active":"2024-03-11T20:53:38.334015+0000","last_peered":"2024-03-11T20:53:38.334015+0000","last_clean":"2024-03-11T20:53:38.334015+0000","last_became_active":"2024-03-11T20:50:34.321443+0000","last_became_peered":"2024-03-11T20:50:34.321443+0000","last_unstale":"2024-03-11T20:53:38.334015+0000","last_undegraded":"2024-03-11T20:53:38.334015+0000","last_fullsized":"2024-03-11T20:53:38.334015+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:04.321235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:53:04.321235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:23.221472+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":98,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.328829+0000","last_change":"2024-03-11T20:53:34.195962+0000","last_active":"2024-03-11T20:53:38.328829+0000","last_peered":"2024-03-11T20:53:38.328829+0000","last_clean":"2024-03-11T20:53:38.328829+0000","last_became_active":"2024-03-11T20:50:32.211995+0000","last_became_peered":"2024-03-11T20:50:32.211995+0000","last_unstale":"2024-03-11T20:53:38.328829+0000","last_undegraded":"2024-03-11T20:53:38.328829+0000","last_fullsized":"2024-03-11T20:53:38.328829+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:34.195681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:53:34.195681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:47.493709+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":106,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.837452+0000","last_change":"2024-03-11T20:52:53.806361+0000","last_active":"2024-03-11T20:53:38.837452+0000","last_peered":"2024-03-11T20:53:38.837452+0000","last_clean":"2024-03-11T20:53:38.837452+0000","last_became_active":"2024-03-11T20:50:28.075760+0000","last_became_peered":"2024-03-11T20:50:28.075760+0000","last_unstale":"2024-03-11T20:53:38.837452+0000","last_undegraded":"2024-03-11T20:53:38.837452+0000","last_fullsized":"2024-03-11T20:53:38.837452+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:53.806149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:52:53.806149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:15.472221+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"84'6","reported_seq":110,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.324926+0000","last_change":"2024-03-11T20:53:18.136127+0000","last_active":"2024-03-11T20:53:38.324926+0000","last_peered":"2024-03-11T20:53:38.324926+0000","last_clean":"2024-03-11T20:53:38.324926+0000","last_became_active":"2024-03-11T20:50:30.092991+0000","last_became_peered":"2024-03-11T20:50:30.092991+0000","last_unstale":"2024-03-11T20:53:38.324926+0000","last_undegraded":"2024-03-11T20:53:38.324926+0000","last_fullsized":"2024-03-11T20:53:38.324926+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6","last_scrub_stamp":"2024-03-11T20:53:18.135889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:53:18.135889+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:46.013838+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.11","version":"84'1","reported_seq":87,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.328802+0000","last_change":"2024-03-11T20:52:42.470691+0000","last_active":"2024-03-11T20:53:38.328802+0000","last_peered":"2024-03-11T20:53:38.328802+0000","last_clean":"2024-03-11T20:53:38.328802+0000","last_became_active":"2024-03-11T20:50:34.339326+0000","last_became_peered":"2024-03-11T20:50:34.339326+0000","last_unstale":"2024-03-11T20:53:38.328802+0000","last_undegraded":"2024-03-11T20:53:38.328802+0000","last_fullsized":"2024-03-11T20:53:38.328802+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'1","last_scrub_stamp":"2024-03-11T20:52:42.470380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:52:42.470380+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:08.720300+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":98,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.323291+0000","last_change":"2024-03-11T20:53:21.439340+0000","last_active":"2024-03-11T20:53:38.323291+0000","last_peered":"2024-03-11T20:53:38.323291+0000","last_clean":"2024-03-11T20:53:38.323291+0000","last_became_active":"2024-03-11T20:50:32.211379+0000","last_became_peered":"2024-03-11T20:50:32.211379+0000","last_unstale":"2024-03-11T20:53:38.323291+0000","last_undegraded":"2024-03-11T20:53:38.323291+0000","last_fullsized":"2024-03-11T20:53:38.323291+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:21.439121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:53:21.439121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:43.786620+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":106,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.323813+0000","last_change":"2024-03-11T20:53:14.109971+0000","last_active":"2024-03-11T20:53:38.323813+0000","last_peered":"2024-03-11T20:53:38.323813+0000","last_clean":"2024-03-11T20:53:38.323813+0000","last_became_active":"2024-03-11T20:50:28.049937+0000","last_became_peered":"2024-03-11T20:50:28.049937+0000","last_unstale":"2024-03-11T20:53:38.323813+0000","last_undegraded":"2024-03-11T20:53:38.323813+0000","last_fullsized":"2024-03-11T20:53:38.323813+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:14.109743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:53:14.109743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:31.769637+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"84'10","reported_seq":117,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.324559+0000","last_change":"2024-03-11T20:52:51.437032+0000","last_active":"2024-03-11T20:53:38.324559+0000","last_peered":"2024-03-11T20:53:38.324559+0000","last_clean":"2024-03-11T20:53:38.324559+0000","last_became_active":"2024-03-11T20:50:30.105752+0000","last_became_peered":"2024-03-11T20:50:30.105752+0000","last_unstale":"2024-03-11T20:53:38.324559+0000","last_undegraded":"2024-03-11T20:53:38.324559+0000","last_fullsized":"2024-03-11T20:53:38.324559+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'10","last_scrub_stamp":"2024-03-11T20:52:51.436752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:52:51.436752+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:15.812213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":93,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.319309+0000","last_change":"2024-03-11T20:53:22.057073+0000","last_active":"2024-03-11T20:53:38.319309+0000","last_peered":"2024-03-11T20:53:38.319309+0000","last_clean":"2024-03-11T20:53:38.319309+0000","last_became_active":"2024-03-11T20:50:34.332465+0000","last_became_peered":"2024-03-11T20:50:34.332465+0000","last_unstale":"2024-03-11T20:53:38.319309+0000","last_undegraded":"2024-03-11T20:53:38.319309+0000","last_fullsized":"2024-03-11T20:53:38.319309+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:22.056809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:53:22.056809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:42.841855+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":98,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.319395+0000","last_change":"2024-03-11T20:53:24.209982+0000","last_active":"2024-03-11T20:53:38.319395+0000","last_peered":"2024-03-11T20:53:38.319395+0000","last_clean":"2024-03-11T20:53:38.319395+0000","last_became_active":"2024-03-11T20:50:32.194792+0000","last_became_peered":"2024-03-11T20:50:32.194792+0000","last_unstale":"2024-03-11T20:53:38.319395+0000","last_undegraded":"2024-03-11T20:53:38.319395+0000","last_fullsized":"2024-03-11T20:53:38.319395+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:24.209748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:53:24.209748+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:43.454286+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":106,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.859115+0000","last_change":"2024-03-11T20:52:54.775523+0000","last_active":"2024-03-11T20:53:38.859115+0000","last_peered":"2024-03-11T20:53:38.859115+0000","last_clean":"2024-03-11T20:53:38.859115+0000","last_became_active":"2024-03-11T20:50:28.082737+0000","last_became_peered":"2024-03-11T20:50:28.082737+0000","last_unstale":"2024-03-11T20:53:38.859115+0000","last_undegraded":"2024-03-11T20:53:38.859115+0000","last_fullsized":"2024-03-11T20:53:38.859115+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:54.775303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:52:54.775303+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:11.643155+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"84'9","reported_seq":118,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.336298+0000","last_change":"2024-03-11T20:52:57.247973+0000","last_active":"2024-03-11T20:53:38.336298+0000","last_peered":"2024-03-11T20:53:38.336298+0000","last_clean":"2024-03-11T20:53:38.336298+0000","last_became_active":"2024-03-11T20:50:30.085418+0000","last_became_peered":"2024-03-11T20:50:30.085418+0000","last_unstale":"2024-03-11T20:53:38.336298+0000","last_undegraded":"2024-03-11T20:53:38.336298+0000","last_fullsized":"2024-03-11T20:53:38.336298+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'9","last_scrub_stamp":"2024-03-11T20:52:57.247711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:52:57.247711+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:09.030171+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":94,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.324970+0000","last_change":"2024-03-11T20:53:14.601565+0000","last_active":"2024-03-11T20:53:38.324970+0000","last_peered":"2024-03-11T20:53:38.324970+0000","last_clean":"2024-03-11T20:53:38.324970+0000","last_became_active":"2024-03-11T20:50:34.328449+0000","last_became_peered":"2024-03-11T20:50:34.328449+0000","last_unstale":"2024-03-11T20:53:38.324970+0000","last_undegraded":"2024-03-11T20:53:38.324970+0000","last_fullsized":"2024-03-11T20:53:38.324970+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:14.601439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:53:14.601439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:19.740273+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.15","version":"112'12","reported_seq":150,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:43.292412+0000","last_change":"2024-03-11T20:53:11.844176+0000","last_active":"2024-03-11T20:53:43.292412+0000","last_peered":"2024-03-11T20:53:43.292412+0000","last_clean":"2024-03-11T20:53:43.292412+0000","last_became_active":"2024-03-11T20:50:32.650064+0000","last_became_peered":"2024-03-11T20:50:32.650064+0000","last_unstale":"2024-03-11T20:53:43.292412+0000","last_undegraded":"2024-03-11T20:53:43.292412+0000","last_fullsized":"2024-03-11T20:53:43.292412+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'12","last_scrub_stamp":"2024-03-11T20:53:11.843951+0000","last_deep_scrub":"112'12","last_deep_scrub_stamp":"2024-03-11T20:53:11.843951+0000","last_clean_scrub_stamp":"2024-03-11T20:53:11.843951+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:40.680380+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":105,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.320246+0000","last_change":"2024-03-11T20:52:47.933545+0000","last_active":"2024-03-11T20:53:38.320246+0000","last_peered":"2024-03-11T20:53:38.320246+0000","last_clean":"2024-03-11T20:53:38.320246+0000","last_became_active":"2024-03-11T20:50:28.044321+0000","last_became_peered":"2024-03-11T20:50:28.044321+0000","last_unstale":"2024-03-11T20:53:38.320246+0000","last_undegraded":"2024-03-11T20:53:38.320246+0000","last_fullsized":"2024-03-11T20:53:38.320246+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:47.933320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:52:47.933320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:55.580970+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"84'9","reported_seq":117,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.321404+0000","last_change":"2024-03-11T20:52:55.993159+0000","last_active":"2024-03-11T20:53:38.321404+0000","last_peered":"2024-03-11T20:53:38.321404+0000","last_clean":"2024-03-11T20:53:38.321404+0000","last_became_active":"2024-03-11T20:50:30.099856+0000","last_became_peered":"2024-03-11T20:50:30.099856+0000","last_unstale":"2024-03-11T20:53:38.321404+0000","last_undegraded":"2024-03-11T20:53:38.321404+0000","last_fullsized":"2024-03-11T20:53:38.321404+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'9","last_scrub_stamp":"2024-03-11T20:52:55.992803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:52:55.992803+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:14.843849+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":93,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:40.043228+0000","last_change":"2024-03-11T20:53:40.043289+0000","last_active":"2024-03-11T20:53:40.043228+0000","last_peered":"2024-03-11T20:53:40.043228+0000","last_clean":"2024-03-11T20:53:40.043228+0000","last_became_active":"2024-03-11T20:50:34.330351+0000","last_became_peered":"2024-03-11T20:50:34.330351+0000","last_unstale":"2024-03-11T20:53:40.043228+0000","last_undegraded":"2024-03-11T20:53:40.043228+0000","last_fullsized":"2024-03-11T20:53:40.043228+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:40.043054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:53:40.043054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:56.504086+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"110'13","reported_seq":160,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:43.288495+0000","last_change":"2024-03-11T20:53:00.434710+0000","last_active":"2024-03-11T20:53:43.288495+0000","last_peered":"2024-03-11T20:53:43.288495+0000","last_clean":"2024-03-11T20:53:43.288495+0000","last_became_active":"2024-03-11T20:51:22.669419+0000","last_became_peered":"2024-03-11T20:51:22.669419+0000","last_unstale":"2024-03-11T20:53:43.288495+0000","last_undegraded":"2024-03-11T20:53:43.288495+0000","last_fullsized":"2024-03-11T20:53:43.288495+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'13","last_scrub_stamp":"2024-03-11T20:53:00.434532+0000","last_deep_scrub":"110'13","last_deep_scrub_stamp":"2024-03-11T20:53:00.434532+0000","last_clean_scrub_stamp":"2024-03-11T20:53:00.434532+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:25.979437+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":106,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.836449+0000","last_change":"2024-03-11T20:52:59.801478+0000","last_active":"2024-03-11T20:53:38.836449+0000","last_peered":"2024-03-11T20:53:38.836449+0000","last_clean":"2024-03-11T20:53:38.836449+0000","last_became_active":"2024-03-11T20:50:28.070325+0000","last_became_peered":"2024-03-11T20:50:28.070325+0000","last_unstale":"2024-03-11T20:53:38.836449+0000","last_undegraded":"2024-03-11T20:53:38.836449+0000","last_fullsized":"2024-03-11T20:53:38.836449+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:59.801249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:52:59.801249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:29.742165+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"116'16","reported_seq":126,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:39.292185+0000","last_change":"2024-03-11T20:53:39.292223+0000","last_active":"2024-03-11T20:53:39.292185+0000","last_peered":"2024-03-11T20:53:39.292185+0000","last_clean":"2024-03-11T20:53:39.292185+0000","last_became_active":"2024-03-11T20:50:30.087154+0000","last_became_peered":"2024-03-11T20:50:30.087154+0000","last_unstale":"2024-03-11T20:53:39.292185+0000","last_undegraded":"2024-03-11T20:53:39.292185+0000","last_fullsized":"2024-03-11T20:53:39.292185+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'16","last_scrub_stamp":"2024-03-11T20:53:39.292118+0000","last_deep_scrub":"116'16","last_deep_scrub_stamp":"2024-03-11T20:53:39.292118+0000","last_clean_scrub_stamp":"2024-03-11T20:53:39.292118+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:06.274297+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":94,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:47.223595+0000","last_change":"2024-03-11T20:53:47.223659+0000","last_active":"2024-03-11T20:53:47.223595+0000","last_peered":"2024-03-11T20:53:47.223595+0000","last_clean":"2024-03-11T20:53:47.223595+0000","last_became_active":"2024-03-11T20:50:34.318231+0000","last_became_peered":"2024-03-11T20:50:34.318231+0000","last_unstale":"2024-03-11T20:53:47.223595+0000","last_undegraded":"2024-03-11T20:53:47.223595+0000","last_fullsized":"2024-03-11T20:53:47.223595+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:47.223425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:53:47.223425+0000","last_clean_scrub_stamp":"2024-03-11T20:53:47.223425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:11.267529+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":98,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.331323+0000","last_change":"2024-03-11T20:53:06.534811+0000","last_active":"2024-03-11T20:53:38.331323+0000","last_peered":"2024-03-11T20:53:38.331323+0000","last_clean":"2024-03-11T20:53:38.331323+0000","last_became_active":"2024-03-11T20:50:32.651361+0000","last_became_peered":"2024-03-11T20:50:32.651361+0000","last_unstale":"2024-03-11T20:53:38.331323+0000","last_undegraded":"2024-03-11T20:53:38.331323+0000","last_fullsized":"2024-03-11T20:53:38.331323+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:06.534615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:53:06.534615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:15.085973+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"77'1","reported_seq":114,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.858671+0000","last_change":"2024-03-11T20:53:24.901219+0000","last_active":"2024-03-11T20:53:38.858671+0000","last_peered":"2024-03-11T20:53:38.858671+0000","last_clean":"2024-03-11T20:53:38.858671+0000","last_became_active":"2024-03-11T20:50:28.083256+0000","last_became_peered":"2024-03-11T20:50:28.083256+0000","last_unstale":"2024-03-11T20:53:38.858671+0000","last_undegraded":"2024-03-11T20:53:38.858671+0000","last_fullsized":"2024-03-11T20:53:38.858671+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-03-11T20:53:24.900978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:53:24.900978+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:38.729662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"84'4","reported_seq":108,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:42.788315+0000","last_change":"2024-03-11T20:53:42.788395+0000","last_active":"2024-03-11T20:53:42.788315+0000","last_peered":"2024-03-11T20:53:42.788315+0000","last_clean":"2024-03-11T20:53:42.788315+0000","last_became_active":"2024-03-11T20:50:30.101572+0000","last_became_peered":"2024-03-11T20:50:30.101572+0000","last_unstale":"2024-03-11T20:53:42.788315+0000","last_undegraded":"2024-03-11T20:53:42.788315+0000","last_fullsized":"2024-03-11T20:53:42.788315+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'4","last_scrub_stamp":"2024-03-11T20:53:42.788154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:53:42.788154+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:11.351245+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":94,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.882182+0000","last_change":"2024-03-11T20:52:57.942490+0000","last_active":"2024-03-11T20:53:38.882182+0000","last_peered":"2024-03-11T20:53:38.882182+0000","last_clean":"2024-03-11T20:53:38.882182+0000","last_became_active":"2024-03-11T20:50:34.320001+0000","last_became_peered":"2024-03-11T20:50:34.320001+0000","last_unstale":"2024-03-11T20:53:38.882182+0000","last_undegraded":"2024-03-11T20:53:38.882182+0000","last_fullsized":"2024-03-11T20:53:38.882182+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:57.942382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:52:57.942382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:03.245684+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":97,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.323620+0000","last_change":"2024-03-11T20:53:05.100946+0000","last_active":"2024-03-11T20:53:38.323620+0000","last_peered":"2024-03-11T20:53:38.323620+0000","last_clean":"2024-03-11T20:53:38.323620+0000","last_became_active":"2024-03-11T20:50:32.646569+0000","last_became_peered":"2024-03-11T20:50:32.646569+0000","last_unstale":"2024-03-11T20:53:38.323620+0000","last_undegraded":"2024-03-11T20:53:38.323620+0000","last_fullsized":"2024-03-11T20:53:38.323620+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:05.100847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:53:05.100847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:07.646846+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":106,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.882472+0000","last_change":"2024-03-11T20:53:02.899718+0000","last_active":"2024-03-11T20:53:38.882472+0000","last_peered":"2024-03-11T20:53:38.882472+0000","last_clean":"2024-03-11T20:53:38.882472+0000","last_became_active":"2024-03-11T20:50:28.044592+0000","last_became_peered":"2024-03-11T20:50:28.044592+0000","last_unstale":"2024-03-11T20:53:38.882472+0000","last_undegraded":"2024-03-11T20:53:38.882472+0000","last_fullsized":"2024-03-11T20:53:38.882472+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:02.899469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:53:02.899469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:21.925538+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"84'11","reported_seq":121,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:42.298470+0000","last_change":"2024-03-11T20:53:42.298549+0000","last_active":"2024-03-11T20:53:42.298470+0000","last_peered":"2024-03-11T20:53:42.298470+0000","last_clean":"2024-03-11T20:53:42.298470+0000","last_became_active":"2024-03-11T20:50:30.103212+0000","last_became_peered":"2024-03-11T20:50:30.103212+0000","last_unstale":"2024-03-11T20:53:42.298470+0000","last_undegraded":"2024-03-11T20:53:42.298470+0000","last_fullsized":"2024-03-11T20:53:42.298470+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T20:53:42.298307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:53:42.298307+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:10.739880+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":94,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.330751+0000","last_change":"2024-03-11T20:52:53.451122+0000","last_active":"2024-03-11T20:53:38.330751+0000","last_peered":"2024-03-11T20:53:38.330751+0000","last_clean":"2024-03-11T20:53:38.330751+0000","last_became_active":"2024-03-11T20:50:34.336077+0000","last_became_peered":"2024-03-11T20:50:34.336077+0000","last_unstale":"2024-03-11T20:53:38.330751+0000","last_undegraded":"2024-03-11T20:53:38.330751+0000","last_fullsized":"2024-03-11T20:53:38.330751+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:53.450885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:52:53.450885+0000","last_clean_scrub_stamp":"2024-03-11T20:52:53.450885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:10.721360+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"77'2","reported_seq":122,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.324011+0000","last_change":"2024-03-11T20:53:28.411797+0000","last_active":"2024-03-11T20:53:38.324011+0000","last_peered":"2024-03-11T20:53:38.324011+0000","last_clean":"2024-03-11T20:53:38.324011+0000","last_became_active":"2024-03-11T20:50:28.059564+0000","last_became_peered":"2024-03-11T20:50:28.059564+0000","last_unstale":"2024-03-11T20:53:38.324011+0000","last_undegraded":"2024-03-11T20:53:38.324011+0000","last_fullsized":"2024-03-11T20:53:38.324011+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'2","last_scrub_stamp":"2024-03-11T20:53:28.411558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:53:28.411558+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:49.755187+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"100'16","reported_seq":126,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.336558+0000","last_change":"2024-03-11T20:52:37.207901+0000","last_active":"2024-03-11T20:53:38.336558+0000","last_peered":"2024-03-11T20:53:38.336558+0000","last_clean":"2024-03-11T20:53:38.336558+0000","last_became_active":"2024-03-11T20:50:30.083708+0000","last_became_peered":"2024-03-11T20:50:30.083708+0000","last_unstale":"2024-03-11T20:53:38.336558+0000","last_undegraded":"2024-03-11T20:53:38.336558+0000","last_fullsized":"2024-03-11T20:53:38.336558+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'16","last_scrub_stamp":"2024-03-11T20:52:37.207662+0000","last_deep_scrub":"100'16","last_deep_scrub_stamp":"2024-03-11T20:51:31.194005+0000","last_clean_scrub_stamp":"2024-03-11T20:52:37.207662+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:56.687016+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"106'12","reported_seq":142,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:43.293064+0000","last_change":"2024-03-11T20:52:21.539124+0000","last_active":"2024-03-11T20:53:43.293064+0000","last_peered":"2024-03-11T20:53:43.293064+0000","last_clean":"2024-03-11T20:53:43.293064+0000","last_became_active":"2024-03-11T20:50:32.210620+0000","last_became_peered":"2024-03-11T20:50:32.210620+0000","last_unstale":"2024-03-11T20:53:43.293064+0000","last_undegraded":"2024-03-11T20:53:43.293064+0000","last_fullsized":"2024-03-11T20:53:43.293064+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'12","last_scrub_stamp":"2024-03-11T20:52:21.538885+0000","last_deep_scrub":"106'12","last_deep_scrub_stamp":"2024-03-11T20:52:21.538885+0000","last_clean_scrub_stamp":"2024-03-11T20:52:21.538885+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":93,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.321843+0000","last_change":"2024-03-11T20:53:27.053454+0000","last_active":"2024-03-11T20:53:38.321843+0000","last_peered":"2024-03-11T20:53:38.321843+0000","last_clean":"2024-03-11T20:53:38.321843+0000","last_became_active":"2024-03-11T20:50:34.328932+0000","last_became_peered":"2024-03-11T20:50:34.328932+0000","last_unstale":"2024-03-11T20:53:38.321843+0000","last_undegraded":"2024-03-11T20:53:38.321843+0000","last_fullsized":"2024-03-11T20:53:38.321843+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:27.053208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:53:27.053208+0000","last_clean_scrub_stamp":"2024-03-11T20:53:27.053208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:28.263449+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":106,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.325021+0000","last_change":"2024-03-11T20:53:38.117372+0000","last_active":"2024-03-11T20:53:38.325021+0000","last_peered":"2024-03-11T20:53:38.325021+0000","last_clean":"2024-03-11T20:53:38.325021+0000","last_became_active":"2024-03-11T20:50:28.057091+0000","last_became_peered":"2024-03-11T20:50:28.057091+0000","last_unstale":"2024-03-11T20:53:38.325021+0000","last_undegraded":"2024-03-11T20:53:38.325021+0000","last_fullsized":"2024-03-11T20:53:38.325021+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:38.117113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:53:38.117113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:45.724302+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"84'16","reported_seq":165,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:45.846335+0000","last_change":"2024-03-11T20:53:45.846335+0000","last_active":"2024-03-11T20:53:45.846335+0000","last_peered":"2024-03-11T20:53:45.846335+0000","last_clean":"2024-03-11T20:53:45.846335+0000","last_became_active":"2024-03-11T20:50:30.094977+0000","last_became_peered":"2024-03-11T20:50:30.094977+0000","last_unstale":"2024-03-11T20:53:45.846335+0000","last_undegraded":"2024-03-11T20:53:45.846335+0000","last_fullsized":"2024-03-11T20:53:45.846335+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'16","last_scrub_stamp":"2024-03-11T20:53:45.846082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:53:45.846082+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:50.615017+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":98,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.320378+0000","last_change":"2024-03-11T20:53:35.139583+0000","last_active":"2024-03-11T20:53:38.320378+0000","last_peered":"2024-03-11T20:53:38.320378+0000","last_clean":"2024-03-11T20:53:38.320378+0000","last_became_active":"2024-03-11T20:50:32.206394+0000","last_became_peered":"2024-03-11T20:50:32.206394+0000","last_unstale":"2024-03-11T20:53:38.320378+0000","last_undegraded":"2024-03-11T20:53:38.320378+0000","last_fullsized":"2024-03-11T20:53:38.320378+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:35.139348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:52:10.160202+0000","last_clean_scrub_stamp":"2024-03-11T20:53:35.139348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:00.755689+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":94,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.331134+0000","last_change":"2024-03-11T20:53:33.225496+0000","last_active":"2024-03-11T20:53:38.331134+0000","last_peered":"2024-03-11T20:53:38.331134+0000","last_clean":"2024-03-11T20:53:38.331134+0000","last_became_active":"2024-03-11T20:50:34.329253+0000","last_became_peered":"2024-03-11T20:50:34.329253+0000","last_unstale":"2024-03-11T20:53:38.331134+0000","last_undegraded":"2024-03-11T20:53:38.331134+0000","last_fullsized":"2024-03-11T20:53:38.331134+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:33.225296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:53:33.225296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:48.626406+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":106,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.836515+0000","last_change":"2024-03-11T20:52:58.782154+0000","last_active":"2024-03-11T20:53:38.836515+0000","last_peered":"2024-03-11T20:53:38.836515+0000","last_clean":"2024-03-11T20:53:38.836515+0000","last_became_active":"2024-03-11T20:50:28.075678+0000","last_became_peered":"2024-03-11T20:50:28.075678+0000","last_unstale":"2024-03-11T20:53:38.836515+0000","last_undegraded":"2024-03-11T20:53:38.836515+0000","last_fullsized":"2024-03-11T20:53:38.836515+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:58.781938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:33.823704+0000","last_clean_scrub_stamp":"2024-03-11T20:52:58.781938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:19.252494+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"84'19","reported_seq":138,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.324245+0000","last_change":"2024-03-11T20:53:13.560575+0000","last_active":"2024-03-11T20:53:38.324245+0000","last_peered":"2024-03-11T20:53:38.324245+0000","last_clean":"2024-03-11T20:53:38.324245+0000","last_became_active":"2024-03-11T20:50:30.097543+0000","last_became_peered":"2024-03-11T20:50:30.097543+0000","last_unstale":"2024-03-11T20:53:38.324245+0000","last_undegraded":"2024-03-11T20:53:38.324245+0000","last_fullsized":"2024-03-11T20:53:38.324245+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'19","last_scrub_stamp":"2024-03-11T20:53:13.560307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:53:13.560307+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:17.749885+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":98,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.324785+0000","last_change":"2024-03-11T20:53:28.142708+0000","last_active":"2024-03-11T20:53:38.324785+0000","last_peered":"2024-03-11T20:53:38.324785+0000","last_clean":"2024-03-11T20:53:38.324785+0000","last_became_active":"2024-03-11T20:50:32.211494+0000","last_became_peered":"2024-03-11T20:50:32.211494+0000","last_unstale":"2024-03-11T20:53:38.324785+0000","last_undegraded":"2024-03-11T20:53:38.324785+0000","last_fullsized":"2024-03-11T20:53:38.324785+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:28.142465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:53:28.142465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:50.624591+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":94,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:43.272228+0000","last_change":"2024-03-11T20:53:43.272285+0000","last_active":"2024-03-11T20:53:43.272228+0000","last_peered":"2024-03-11T20:53:43.272228+0000","last_clean":"2024-03-11T20:53:43.272228+0000","last_became_active":"2024-03-11T20:50:34.321038+0000","last_became_peered":"2024-03-11T20:50:34.321038+0000","last_unstale":"2024-03-11T20:53:43.272228+0000","last_undegraded":"2024-03-11T20:53:43.272228+0000","last_fullsized":"2024-03-11T20:53:43.272228+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:43.272073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:53:43.272073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:52.600692+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"109'26","reported_seq":142,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.319194+0000","last_change":"2024-03-11T20:52:49.216560+0000","last_active":"2024-03-11T20:53:38.319194+0000","last_peered":"2024-03-11T20:53:38.319194+0000","last_clean":"2024-03-11T20:53:38.319194+0000","last_became_active":"2024-03-11T20:50:30.098371+0000","last_became_peered":"2024-03-11T20:50:30.098371+0000","last_unstale":"2024-03-11T20:53:38.319194+0000","last_undegraded":"2024-03-11T20:53:38.319194+0000","last_fullsized":"2024-03-11T20:53:38.319194+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'26","last_scrub_stamp":"2024-03-11T20:52:49.216378+0000","last_deep_scrub":"109'26","last_deep_scrub_stamp":"2024-03-11T20:52:49.216378+0000","last_clean_scrub_stamp":"2024-03-11T20:52:49.216378+0000","objects_scrubbed":8,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:49.844091+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":106,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.882816+0000","last_change":"2024-03-11T20:53:06.797551+0000","last_active":"2024-03-11T20:53:38.882816+0000","last_peered":"2024-03-11T20:53:38.882816+0000","last_clean":"2024-03-11T20:53:38.882816+0000","last_became_active":"2024-03-11T20:50:28.045625+0000","last_became_peered":"2024-03-11T20:50:28.045625+0000","last_unstale":"2024-03-11T20:53:38.882816+0000","last_undegraded":"2024-03-11T20:53:38.882816+0000","last_fullsized":"2024-03-11T20:53:38.882816+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:06.797336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:53:06.797336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:24.649727+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":90,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.321468+0000","last_change":"2024-03-11T20:52:25.125883+0000","last_active":"2024-03-11T20:53:38.321468+0000","last_peered":"2024-03-11T20:53:38.321468+0000","last_clean":"2024-03-11T20:53:38.321468+0000","last_became_active":"2024-03-11T20:50:32.205996+0000","last_became_peered":"2024-03-11T20:50:32.205996+0000","last_unstale":"2024-03-11T20:53:38.321468+0000","last_undegraded":"2024-03-11T20:53:38.321468+0000","last_fullsized":"2024-03-11T20:53:38.321468+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:25.125696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:52:25.125696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:52.303574+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":94,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:40.853470+0000","last_change":"2024-03-11T20:53:40.853529+0000","last_active":"2024-03-11T20:53:40.853470+0000","last_peered":"2024-03-11T20:53:40.853470+0000","last_clean":"2024-03-11T20:53:40.853470+0000","last_became_active":"2024-03-11T20:50:34.336066+0000","last_became_peered":"2024-03-11T20:50:34.336066+0000","last_unstale":"2024-03-11T20:53:40.853470+0000","last_undegraded":"2024-03-11T20:53:40.853470+0000","last_fullsized":"2024-03-11T20:53:40.853470+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:40.853290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:53:40.853290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:02.791101+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"84'10","reported_seq":109,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.329193+0000","last_change":"2024-03-11T20:52:35.399348+0000","last_active":"2024-03-11T20:53:38.329193+0000","last_peered":"2024-03-11T20:53:38.329193+0000","last_clean":"2024-03-11T20:53:38.329193+0000","last_became_active":"2024-03-11T20:50:30.101747+0000","last_became_peered":"2024-03-11T20:50:30.101747+0000","last_unstale":"2024-03-11T20:53:38.329193+0000","last_undegraded":"2024-03-11T20:53:38.329193+0000","last_fullsized":"2024-03-11T20:53:38.329193+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'10","last_scrub_stamp":"2024-03-11T20:52:35.399027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:52:35.399027+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:57.620344+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":110,"reported_epoch":116,"state":"active+clean+scrubbing+deep","last_fresh":"2024-03-11T20:53:49.835145+0000","last_change":"2024-03-11T20:53:49.832311+0000","last_active":"2024-03-11T20:53:49.835145+0000","last_peered":"2024-03-11T20:53:49.835145+0000","last_clean":"2024-03-11T20:53:49.835145+0000","last_became_active":"2024-03-11T20:50:28.085503+0000","last_became_peered":"2024-03-11T20:50:28.085503+0000","last_unstale":"2024-03-11T20:53:49.835145+0000","last_undegraded":"2024-03-11T20:53:49.835145+0000","last_fullsized":"2024-03-11T20:53:49.835145+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:44.918937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:52:44.918937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"deep scrubbing for 0s","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"102'128","reported_seq":363,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.334738+0000","last_change":"2024-03-11T20:52:28.047817+0000","last_active":"2024-03-11T20:53:38.334738+0000","last_peered":"2024-03-11T20:53:38.334738+0000","last_clean":"2024-03-11T20:53:38.334738+0000","last_became_active":"2024-03-11T20:49:52.584543+0000","last_became_peered":"2024-03-11T20:49:52.584543+0000","last_unstale":"2024-03-11T20:53:38.334738+0000","last_undegraded":"2024-03-11T20:53:38.334738+0000","last_fullsized":"2024-03-11T20:53:38.334738+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"102'128","last_scrub_stamp":"2024-03-11T20:52:28.047573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:45:32.381751+0000","last_clean_scrub_stamp":"2024-03-11T20:52:28.047573+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:54.167386+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":98,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.859248+0000","last_change":"2024-03-11T20:53:07.859791+0000","last_active":"2024-03-11T20:53:38.859248+0000","last_peered":"2024-03-11T20:53:38.859248+0000","last_clean":"2024-03-11T20:53:38.859248+0000","last_became_active":"2024-03-11T20:50:32.652128+0000","last_became_peered":"2024-03-11T20:50:32.652128+0000","last_unstale":"2024-03-11T20:53:38.859248+0000","last_undegraded":"2024-03-11T20:53:38.859248+0000","last_fullsized":"2024-03-11T20:53:38.859248+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:07.859545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:53:07.859545+0000","last_clean_scrub_stamp":"2024-03-11T20:53:07.859545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:20.657302+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":94,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.324527+0000","last_change":"2024-03-11T20:53:23.179735+0000","last_active":"2024-03-11T20:53:38.324527+0000","last_peered":"2024-03-11T20:53:38.324527+0000","last_clean":"2024-03-11T20:53:38.324527+0000","last_became_active":"2024-03-11T20:50:34.326321+0000","last_became_peered":"2024-03-11T20:50:34.326321+0000","last_unstale":"2024-03-11T20:53:38.324527+0000","last_undegraded":"2024-03-11T20:53:38.324527+0000","last_fullsized":"2024-03-11T20:53:38.324527+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:23.179506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:53:23.179506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:45.358796+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"100'21","reported_seq":129,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.320405+0000","last_change":"2024-03-11T20:53:03.110687+0000","last_active":"2024-03-11T20:53:38.320405+0000","last_peered":"2024-03-11T20:53:38.320405+0000","last_clean":"2024-03-11T20:53:38.320405+0000","last_became_active":"2024-03-11T20:50:30.106110+0000","last_became_peered":"2024-03-11T20:50:30.106110+0000","last_unstale":"2024-03-11T20:53:38.320405+0000","last_undegraded":"2024-03-11T20:53:38.320405+0000","last_fullsized":"2024-03-11T20:53:38.320405+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'21","last_scrub_stamp":"2024-03-11T20:53:03.110450+0000","last_deep_scrub":"100'21","last_deep_scrub_stamp":"2024-03-11T20:51:31.835302+0000","last_clean_scrub_stamp":"2024-03-11T20:53:03.110450+0000","objects_scrubbed":7,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:23.836851+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":64,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.882136+0000","last_change":"2024-03-11T20:52:56.891528+0000","last_active":"2024-03-11T20:53:38.882136+0000","last_peered":"2024-03-11T20:53:38.882136+0000","last_clean":"2024-03-11T20:53:38.882136+0000","last_became_active":"2024-03-11T20:51:22.668492+0000","last_became_peered":"2024-03-11T20:51:22.668492+0000","last_unstale":"2024-03-11T20:53:38.882136+0000","last_undegraded":"2024-03-11T20:53:38.882136+0000","last_fullsized":"2024-03-11T20:53:38.882136+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:56.891410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:52:56.891410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:57.500706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":97,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.324201+0000","last_change":"2024-03-11T20:53:04.140960+0000","last_active":"2024-03-11T20:53:38.324201+0000","last_peered":"2024-03-11T20:53:38.324201+0000","last_clean":"2024-03-11T20:53:38.324201+0000","last_became_active":"2024-03-11T20:50:32.195566+0000","last_became_peered":"2024-03-11T20:50:32.195566+0000","last_unstale":"2024-03-11T20:53:38.324201+0000","last_undegraded":"2024-03-11T20:53:38.324201+0000","last_fullsized":"2024-03-11T20:53:38.324201+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:04.140875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:53:04.140875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:29.280686+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":94,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.318406+0000","last_change":"2024-03-11T20:53:08.170244+0000","last_active":"2024-03-11T20:53:38.318406+0000","last_peered":"2024-03-11T20:53:38.318406+0000","last_clean":"2024-03-11T20:53:38.318406+0000","last_became_active":"2024-03-11T20:50:34.306750+0000","last_became_peered":"2024-03-11T20:50:34.306750+0000","last_unstale":"2024-03-11T20:53:38.318406+0000","last_undegraded":"2024-03-11T20:53:38.318406+0000","last_fullsized":"2024-03-11T20:53:38.318406+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:08.170036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:53:08.170036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:35.208803+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"87'2","reported_seq":107,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.858674+0000","last_change":"2024-03-11T20:53:25.900353+0000","last_active":"2024-03-11T20:53:38.858674+0000","last_peered":"2024-03-11T20:53:38.858674+0000","last_clean":"2024-03-11T20:53:38.858674+0000","last_became_active":"2024-03-11T20:50:32.195452+0000","last_became_peered":"2024-03-11T20:50:32.195452+0000","last_unstale":"2024-03-11T20:53:38.858674+0000","last_undegraded":"2024-03-11T20:53:38.858674+0000","last_fullsized":"2024-03-11T20:53:38.858674+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'2","last_scrub_stamp":"2024-03-11T20:53:25.900041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:53:25.900041+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:42.706289+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0024049620000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":106,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.336007+0000","last_change":"2024-03-11T20:53:05.364652+0000","last_active":"2024-03-11T20:53:38.336007+0000","last_peered":"2024-03-11T20:53:38.336007+0000","last_clean":"2024-03-11T20:53:38.336007+0000","last_became_active":"2024-03-11T20:50:28.065245+0000","last_became_peered":"2024-03-11T20:50:28.065245+0000","last_unstale":"2024-03-11T20:53:38.336007+0000","last_undegraded":"2024-03-11T20:53:38.336007+0000","last_fullsized":"2024-03-11T20:53:38.336007+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:05.364429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:53:05.364429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:25.580910+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"84'30","reported_seq":193,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:43.288477+0000","last_change":"2024-03-11T20:52:48.167124+0000","last_active":"2024-03-11T20:53:43.288477+0000","last_peered":"2024-03-11T20:53:43.288477+0000","last_clean":"2024-03-11T20:53:43.288477+0000","last_became_active":"2024-03-11T20:50:30.095177+0000","last_became_peered":"2024-03-11T20:50:30.095177+0000","last_unstale":"2024-03-11T20:53:43.288477+0000","last_undegraded":"2024-03-11T20:53:43.288477+0000","last_fullsized":"2024-03-11T20:53:43.288477+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'30","last_scrub_stamp":"2024-03-11T20:52:48.166893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:52:48.166893+0000","objects_scrubbed":10,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:05.469139+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":51,"num_read_kb":36,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"84'11","reported_seq":148,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:43.288466+0000","last_change":"2024-03-11T20:53:35.960764+0000","last_active":"2024-03-11T20:53:43.288466+0000","last_peered":"2024-03-11T20:53:43.288466+0000","last_clean":"2024-03-11T20:53:43.288466+0000","last_became_active":"2024-03-11T20:50:32.649414+0000","last_became_peered":"2024-03-11T20:50:32.649414+0000","last_unstale":"2024-03-11T20:53:43.288466+0000","last_undegraded":"2024-03-11T20:53:43.288466+0000","last_fullsized":"2024-03-11T20:53:43.288466+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T20:53:35.960574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:53:35.960574+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:52.893266+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":94,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.323448+0000","last_change":"2024-03-11T20:53:27.419736+0000","last_active":"2024-03-11T20:53:38.323448+0000","last_peered":"2024-03-11T20:53:38.323448+0000","last_clean":"2024-03-11T20:53:38.323448+0000","last_became_active":"2024-03-11T20:50:34.323647+0000","last_became_peered":"2024-03-11T20:50:34.323647+0000","last_unstale":"2024-03-11T20:53:38.323448+0000","last_undegraded":"2024-03-11T20:53:38.323448+0000","last_fullsized":"2024-03-11T20:53:38.323448+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:27.419519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:53:27.419519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:29.550989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":105,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.322319+0000","last_change":"2024-03-11T20:53:32.530057+0000","last_active":"2024-03-11T20:53:38.322319+0000","last_peered":"2024-03-11T20:53:38.322319+0000","last_clean":"2024-03-11T20:53:38.322319+0000","last_became_active":"2024-03-11T20:50:32.650102+0000","last_became_peered":"2024-03-11T20:50:32.650102+0000","last_unstale":"2024-03-11T20:53:38.322319+0000","last_undegraded":"2024-03-11T20:53:38.322319+0000","last_fullsized":"2024-03-11T20:53:38.322319+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:32.529832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:53:32.529832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:48.340417+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.0022993710000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"77'1","reported_seq":107,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.319697+0000","last_change":"2024-03-11T20:53:26.158578+0000","last_active":"2024-03-11T20:53:38.319697+0000","last_peered":"2024-03-11T20:53:38.319697+0000","last_clean":"2024-03-11T20:53:38.319697+0000","last_became_active":"2024-03-11T20:50:28.065076+0000","last_became_peered":"2024-03-11T20:50:28.065076+0000","last_unstale":"2024-03-11T20:53:38.319697+0000","last_undegraded":"2024-03-11T20:53:38.319697+0000","last_fullsized":"2024-03-11T20:53:38.319697+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-03-11T20:53:26.158259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:53:26.158259+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:48.651482+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"84'13","reported_seq":129,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.330774+0000","last_change":"2024-03-11T20:53:27.387229+0000","last_active":"2024-03-11T20:53:38.330774+0000","last_peered":"2024-03-11T20:53:38.330774+0000","last_clean":"2024-03-11T20:53:38.330774+0000","last_became_active":"2024-03-11T20:50:30.097669+0000","last_became_peered":"2024-03-11T20:50:30.097669+0000","last_unstale":"2024-03-11T20:53:38.330774+0000","last_undegraded":"2024-03-11T20:53:38.330774+0000","last_fullsized":"2024-03-11T20:53:38.330774+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'13","last_scrub_stamp":"2024-03-11T20:53:27.386971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:53:27.386971+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:46.046919+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"84'11","reported_seq":149,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:43.288379+0000","last_change":"2024-03-11T20:53:13.413293+0000","last_active":"2024-03-11T20:53:43.288379+0000","last_peered":"2024-03-11T20:53:43.288379+0000","last_clean":"2024-03-11T20:53:43.288379+0000","last_became_active":"2024-03-11T20:50:32.647041+0000","last_became_peered":"2024-03-11T20:50:32.647041+0000","last_unstale":"2024-03-11T20:53:43.288379+0000","last_undegraded":"2024-03-11T20:53:43.288379+0000","last_fullsized":"2024-03-11T20:53:43.288379+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T20:53:13.413140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:53:13.413140+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:27.508758+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":94,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.334472+0000","last_change":"2024-03-11T20:53:06.344681+0000","last_active":"2024-03-11T20:53:38.334472+0000","last_peered":"2024-03-11T20:53:38.334472+0000","last_clean":"2024-03-11T20:53:38.334472+0000","last_became_active":"2024-03-11T20:50:34.326947+0000","last_became_peered":"2024-03-11T20:50:34.326947+0000","last_unstale":"2024-03-11T20:53:38.334472+0000","last_undegraded":"2024-03-11T20:53:38.334472+0000","last_fullsized":"2024-03-11T20:53:38.334472+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:06.344386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:53:06.344386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:09.051382+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"91'5","reported_seq":286,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:48.057942+0000","last_change":"2024-03-11T20:53:08.493671+0000","last_active":"2024-03-11T20:53:48.057942+0000","last_peered":"2024-03-11T20:53:48.057942+0000","last_clean":"2024-03-11T20:53:48.057942+0000","last_became_active":"2024-03-11T20:50:32.211118+0000","last_became_peered":"2024-03-11T20:50:32.211118+0000","last_unstale":"2024-03-11T20:53:48.057942+0000","last_undegraded":"2024-03-11T20:53:48.057942+0000","last_fullsized":"2024-03-11T20:53:48.057942+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'5","last_scrub_stamp":"2024-03-11T20:53:08.493516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:53:08.493516+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:17.353002+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0035533370000000002,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":178,"num_read_kb":173,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":106,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.859794+0000","last_change":"2024-03-11T20:53:31.863953+0000","last_active":"2024-03-11T20:53:38.859794+0000","last_peered":"2024-03-11T20:53:38.859794+0000","last_clean":"2024-03-11T20:53:38.859794+0000","last_became_active":"2024-03-11T20:50:28.071707+0000","last_became_peered":"2024-03-11T20:50:28.071707+0000","last_unstale":"2024-03-11T20:53:38.859794+0000","last_undegraded":"2024-03-11T20:53:38.859794+0000","last_fullsized":"2024-03-11T20:53:38.859794+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:31.863747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:53:31.863747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:44.440903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"84'12","reported_seq":119,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.324601+0000","last_change":"2024-03-11T20:53:32.943227+0000","last_active":"2024-03-11T20:53:38.324601+0000","last_peered":"2024-03-11T20:53:38.324601+0000","last_clean":"2024-03-11T20:53:38.324601+0000","last_became_active":"2024-03-11T20:50:30.096350+0000","last_became_peered":"2024-03-11T20:50:30.096350+0000","last_unstale":"2024-03-11T20:53:38.324601+0000","last_undegraded":"2024-03-11T20:53:38.324601+0000","last_fullsized":"2024-03-11T20:53:38.324601+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'12","last_scrub_stamp":"2024-03-11T20:53:32.942979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:53:32.942979+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:43.116324+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":98,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.323417+0000","last_change":"2024-03-11T20:53:33.556343+0000","last_active":"2024-03-11T20:53:38.323417+0000","last_peered":"2024-03-11T20:53:38.323417+0000","last_clean":"2024-03-11T20:53:38.323417+0000","last_became_active":"2024-03-11T20:50:32.190896+0000","last_became_peered":"2024-03-11T20:50:32.190896+0000","last_unstale":"2024-03-11T20:53:38.323417+0000","last_undegraded":"2024-03-11T20:53:38.323417+0000","last_fullsized":"2024-03-11T20:53:38.323417+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:33.556062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:53:33.556062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:58.966484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":94,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.330115+0000","last_change":"2024-03-11T20:53:35.172511+0000","last_active":"2024-03-11T20:53:38.330115+0000","last_peered":"2024-03-11T20:53:38.330115+0000","last_clean":"2024-03-11T20:53:38.330115+0000","last_became_active":"2024-03-11T20:50:34.335370+0000","last_became_peered":"2024-03-11T20:50:34.335370+0000","last_unstale":"2024-03-11T20:53:38.330115+0000","last_undegraded":"2024-03-11T20:53:38.330115+0000","last_fullsized":"2024-03-11T20:53:38.330115+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:35.172297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:53:35.172297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:51.240908+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"84'11","reported_seq":141,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:43.292419+0000","last_change":"2024-03-11T20:52:15.500805+0000","last_active":"2024-03-11T20:53:43.292419+0000","last_peered":"2024-03-11T20:53:43.292419+0000","last_clean":"2024-03-11T20:53:43.292419+0000","last_became_active":"2024-03-11T20:50:32.197412+0000","last_became_peered":"2024-03-11T20:50:32.197412+0000","last_unstale":"2024-03-11T20:53:43.292419+0000","last_undegraded":"2024-03-11T20:53:43.292419+0000","last_fullsized":"2024-03-11T20:53:43.292419+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T20:52:15.500571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:52:15.500571+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":106,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.334812+0000","last_change":"2024-03-11T20:53:28.297690+0000","last_active":"2024-03-11T20:53:38.334812+0000","last_peered":"2024-03-11T20:53:38.334812+0000","last_clean":"2024-03-11T20:53:38.334812+0000","last_became_active":"2024-03-11T20:50:28.079413+0000","last_became_peered":"2024-03-11T20:50:28.079413+0000","last_unstale":"2024-03-11T20:53:38.334812+0000","last_undegraded":"2024-03-11T20:53:38.334812+0000","last_fullsized":"2024-03-11T20:53:38.334812+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:28.297455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:52:09.043655+0000","last_clean_scrub_stamp":"2024-03-11T20:53:28.297455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:57.298447+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"84'12","reported_seq":125,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.323203+0000","last_change":"2024-03-11T20:53:22.404447+0000","last_active":"2024-03-11T20:53:38.323203+0000","last_peered":"2024-03-11T20:53:38.323203+0000","last_clean":"2024-03-11T20:53:38.323203+0000","last_became_active":"2024-03-11T20:50:30.083388+0000","last_became_peered":"2024-03-11T20:50:30.083388+0000","last_unstale":"2024-03-11T20:53:38.323203+0000","last_undegraded":"2024-03-11T20:53:38.323203+0000","last_fullsized":"2024-03-11T20:53:38.323203+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'12","last_scrub_stamp":"2024-03-11T20:53:22.404131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:53:22.404131+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:45.149957+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":94,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.835246+0000","last_change":"2024-03-11T20:53:06.771715+0000","last_active":"2024-03-11T20:53:38.835246+0000","last_peered":"2024-03-11T20:53:38.835246+0000","last_clean":"2024-03-11T20:53:38.835246+0000","last_became_active":"2024-03-11T20:50:34.337717+0000","last_became_peered":"2024-03-11T20:50:34.337717+0000","last_unstale":"2024-03-11T20:53:38.835246+0000","last_undegraded":"2024-03-11T20:53:38.835246+0000","last_fullsized":"2024-03-11T20:53:38.835246+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:06.771620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:53:06.771620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:30.302985+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":64,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.882697+0000","last_change":"2024-03-11T20:53:05.836875+0000","last_active":"2024-03-11T20:53:38.882697+0000","last_peered":"2024-03-11T20:53:38.882697+0000","last_clean":"2024-03-11T20:53:38.882697+0000","last_became_active":"2024-03-11T20:51:22.662693+0000","last_became_peered":"2024-03-11T20:51:22.662693+0000","last_unstale":"2024-03-11T20:53:38.882697+0000","last_undegraded":"2024-03-11T20:53:38.882697+0000","last_fullsized":"2024-03-11T20:53:38.882697+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:05.836655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:53:05.836655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:21.447337+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":106,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.320164+0000","last_change":"2024-03-11T20:53:17.169155+0000","last_active":"2024-03-11T20:53:38.320164+0000","last_peered":"2024-03-11T20:53:38.320164+0000","last_clean":"2024-03-11T20:53:38.320164+0000","last_became_active":"2024-03-11T20:50:28.058790+0000","last_became_peered":"2024-03-11T20:50:28.058790+0000","last_unstale":"2024-03-11T20:53:38.320164+0000","last_undegraded":"2024-03-11T20:53:38.320164+0000","last_fullsized":"2024-03-11T20:53:38.320164+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:17.168912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:40.974147+0000","last_clean_scrub_stamp":"2024-03-11T20:53:17.168912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:23.351155+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"116'22","reported_seq":134,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:47.364849+0000","last_change":"2024-03-11T20:53:47.364932+0000","last_active":"2024-03-11T20:53:47.364849+0000","last_peered":"2024-03-11T20:53:47.364849+0000","last_clean":"2024-03-11T20:53:47.364849+0000","last_became_active":"2024-03-11T20:50:30.096215+0000","last_became_peered":"2024-03-11T20:50:30.096215+0000","last_unstale":"2024-03-11T20:53:47.364849+0000","last_undegraded":"2024-03-11T20:53:47.364849+0000","last_fullsized":"2024-03-11T20:53:47.364849+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'22","last_scrub_stamp":"2024-03-11T20:53:47.364709+0000","last_deep_scrub":"116'22","last_deep_scrub_stamp":"2024-03-11T20:53:47.364709+0000","last_clean_scrub_stamp":"2024-03-11T20:53:47.364709+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:10.547868+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":86,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.325046+0000","last_change":"2024-03-11T20:52:12.514160+0000","last_active":"2024-03-11T20:53:38.325046+0000","last_peered":"2024-03-11T20:53:38.325046+0000","last_clean":"2024-03-11T20:53:38.325046+0000","last_became_active":"2024-03-11T20:50:34.331803+0000","last_became_peered":"2024-03-11T20:50:34.331803+0000","last_unstale":"2024-03-11T20:53:38.325046+0000","last_undegraded":"2024-03-11T20:53:38.325046+0000","last_fullsized":"2024-03-11T20:53:38.325046+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:12.513982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:52:12.513982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":106,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.319742+0000","last_change":"2024-03-11T20:53:37.094181+0000","last_active":"2024-03-11T20:53:38.319742+0000","last_peered":"2024-03-11T20:53:38.319742+0000","last_clean":"2024-03-11T20:53:38.319742+0000","last_became_active":"2024-03-11T20:50:28.057121+0000","last_became_peered":"2024-03-11T20:50:28.057121+0000","last_unstale":"2024-03-11T20:53:38.319742+0000","last_undegraded":"2024-03-11T20:53:38.319742+0000","last_fullsized":"2024-03-11T20:53:38.319742+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:37.093963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:53:37.093963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:51.583205+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"84'9","reported_seq":118,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.330466+0000","last_change":"2024-03-11T20:53:03.452779+0000","last_active":"2024-03-11T20:53:38.330466+0000","last_peered":"2024-03-11T20:53:38.330466+0000","last_clean":"2024-03-11T20:53:38.330466+0000","last_became_active":"2024-03-11T20:50:30.089260+0000","last_became_peered":"2024-03-11T20:50:30.089260+0000","last_unstale":"2024-03-11T20:53:38.330466+0000","last_undegraded":"2024-03-11T20:53:38.330466+0000","last_fullsized":"2024-03-11T20:53:38.330466+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'9","last_scrub_stamp":"2024-03-11T20:53:03.452528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:53:03.452528+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:23.786382+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":98,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.323356+0000","last_change":"2024-03-11T20:53:00.551613+0000","last_active":"2024-03-11T20:53:38.323356+0000","last_peered":"2024-03-11T20:53:38.323356+0000","last_clean":"2024-03-11T20:53:38.323356+0000","last_became_active":"2024-03-11T20:50:32.200001+0000","last_became_peered":"2024-03-11T20:50:32.200001+0000","last_unstale":"2024-03-11T20:53:38.323356+0000","last_undegraded":"2024-03-11T20:53:38.323356+0000","last_fullsized":"2024-03-11T20:53:38.323356+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:00.551395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:35.670149+0000","last_clean_scrub_stamp":"2024-03-11T20:53:00.551395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:08.450926+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":94,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.882367+0000","last_change":"2024-03-11T20:52:54.879520+0000","last_active":"2024-03-11T20:53:38.882367+0000","last_peered":"2024-03-11T20:53:38.882367+0000","last_clean":"2024-03-11T20:53:38.882367+0000","last_became_active":"2024-03-11T20:50:34.326354+0000","last_became_peered":"2024-03-11T20:50:34.326354+0000","last_unstale":"2024-03-11T20:53:38.882367+0000","last_undegraded":"2024-03-11T20:53:38.882367+0000","last_fullsized":"2024-03-11T20:53:38.882367+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:54.879292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:52:54.879292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:00.895826+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"98'12","reported_seq":159,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:43.292462+0000","last_change":"2024-03-11T20:53:42.565055+0000","last_active":"2024-03-11T20:53:43.292462+0000","last_peered":"2024-03-11T20:53:43.292462+0000","last_clean":"2024-03-11T20:53:43.292462+0000","last_became_active":"2024-03-11T20:51:22.713613+0000","last_became_peered":"2024-03-11T20:51:22.713613+0000","last_unstale":"2024-03-11T20:53:43.292462+0000","last_undegraded":"2024-03-11T20:53:43.292462+0000","last_fullsized":"2024-03-11T20:53:43.292462+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'12","last_scrub_stamp":"2024-03-11T20:53:42.564785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:53:42.564785+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:55.923659+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":106,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.335329+0000","last_change":"2024-03-11T20:52:51.161508+0000","last_active":"2024-03-11T20:53:38.335329+0000","last_peered":"2024-03-11T20:53:38.335329+0000","last_clean":"2024-03-11T20:53:38.335329+0000","last_became_active":"2024-03-11T20:50:28.069267+0000","last_became_peered":"2024-03-11T20:50:28.069267+0000","last_unstale":"2024-03-11T20:53:38.335329+0000","last_undegraded":"2024-03-11T20:53:38.335329+0000","last_fullsized":"2024-03-11T20:53:38.335329+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:51.161281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:52:51.161281+0000","last_clean_scrub_stamp":"2024-03-11T20:52:51.161281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:57.333795+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"84'19","reported_seq":133,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.858915+0000","last_change":"2024-03-11T20:53:16.962607+0000","last_active":"2024-03-11T20:53:38.858915+0000","last_peered":"2024-03-11T20:53:38.858915+0000","last_clean":"2024-03-11T20:53:38.858915+0000","last_became_active":"2024-03-11T20:50:30.103666+0000","last_became_peered":"2024-03-11T20:50:30.103666+0000","last_unstale":"2024-03-11T20:53:38.858915+0000","last_undegraded":"2024-03-11T20:53:38.858915+0000","last_fullsized":"2024-03-11T20:53:38.858915+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'19","last_scrub_stamp":"2024-03-11T20:53:16.962364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:53:16.962364+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:37.903805+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":94,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.334361+0000","last_change":"2024-03-11T20:53:18.331836+0000","last_active":"2024-03-11T20:53:38.334361+0000","last_peered":"2024-03-11T20:53:38.334361+0000","last_clean":"2024-03-11T20:53:38.334361+0000","last_became_active":"2024-03-11T20:50:34.327139+0000","last_became_peered":"2024-03-11T20:50:34.327139+0000","last_unstale":"2024-03-11T20:53:38.334361+0000","last_undegraded":"2024-03-11T20:53:38.334361+0000","last_fullsized":"2024-03-11T20:53:38.334361+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:18.331616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:53:18.331616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:43.891860+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":106,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.883223+0000","last_change":"2024-03-11T20:53:08.797289+0000","last_active":"2024-03-11T20:53:38.883223+0000","last_peered":"2024-03-11T20:53:38.883223+0000","last_clean":"2024-03-11T20:53:38.883223+0000","last_became_active":"2024-03-11T20:50:28.049901+0000","last_became_peered":"2024-03-11T20:50:28.049901+0000","last_unstale":"2024-03-11T20:53:38.883223+0000","last_undegraded":"2024-03-11T20:53:38.883223+0000","last_fullsized":"2024-03-11T20:53:38.883223+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:08.797203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:53:08.797203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:12.120056+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"102'25","reported_seq":138,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.336267+0000","last_change":"2024-03-11T20:52:47.207949+0000","last_active":"2024-03-11T20:53:38.336267+0000","last_peered":"2024-03-11T20:53:38.336267+0000","last_clean":"2024-03-11T20:53:38.336267+0000","last_became_active":"2024-03-11T20:50:30.111819+0000","last_became_peered":"2024-03-11T20:50:30.111819+0000","last_unstale":"2024-03-11T20:53:38.336267+0000","last_undegraded":"2024-03-11T20:53:38.336267+0000","last_fullsized":"2024-03-11T20:53:38.336267+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"102'25","last_scrub_stamp":"2024-03-11T20:52:47.207710+0000","last_deep_scrub":"102'25","last_deep_scrub_stamp":"2024-03-11T20:51:35.187094+0000","last_clean_scrub_stamp":"2024-03-11T20:52:47.207710+0000","objects_scrubbed":8,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:55.994836+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":98,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.858569+0000","last_change":"2024-03-11T20:53:29.933875+0000","last_active":"2024-03-11T20:53:38.858569+0000","last_peered":"2024-03-11T20:53:38.858569+0000","last_clean":"2024-03-11T20:53:38.858569+0000","last_became_active":"2024-03-11T20:50:32.649877+0000","last_became_peered":"2024-03-11T20:50:32.649877+0000","last_unstale":"2024-03-11T20:53:38.858569+0000","last_undegraded":"2024-03-11T20:53:38.858569+0000","last_fullsized":"2024-03-11T20:53:38.858569+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:29.933643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:53:29.933643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:38.875482+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":93,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.321680+0000","last_change":"2024-03-11T20:52:46.946707+0000","last_active":"2024-03-11T20:53:38.321680+0000","last_peered":"2024-03-11T20:53:38.321680+0000","last_clean":"2024-03-11T20:53:38.321680+0000","last_became_active":"2024-03-11T20:50:34.319536+0000","last_became_peered":"2024-03-11T20:50:34.319536+0000","last_unstale":"2024-03-11T20:53:38.321680+0000","last_undegraded":"2024-03-11T20:53:38.321680+0000","last_fullsized":"2024-03-11T20:53:38.321680+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:46.946561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:52:46.946561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:52.449594+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":106,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.323588+0000","last_change":"2024-03-11T20:53:30.139626+0000","last_active":"2024-03-11T20:53:38.323588+0000","last_peered":"2024-03-11T20:53:38.323588+0000","last_clean":"2024-03-11T20:53:38.323588+0000","last_became_active":"2024-03-11T20:50:28.043982+0000","last_became_peered":"2024-03-11T20:50:28.043982+0000","last_unstale":"2024-03-11T20:53:38.323588+0000","last_undegraded":"2024-03-11T20:53:38.323588+0000","last_fullsized":"2024-03-11T20:53:38.323588+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:30.139539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:53:30.139539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:31.588939+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"84'10","reported_seq":117,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.837512+0000","last_change":"2024-03-11T20:53:12.823097+0000","last_active":"2024-03-11T20:53:38.837512+0000","last_peered":"2024-03-11T20:53:38.837512+0000","last_clean":"2024-03-11T20:53:38.837512+0000","last_became_active":"2024-03-11T20:50:30.109549+0000","last_became_peered":"2024-03-11T20:50:30.109549+0000","last_unstale":"2024-03-11T20:53:38.837512+0000","last_undegraded":"2024-03-11T20:53:38.837512+0000","last_fullsized":"2024-03-11T20:53:38.837512+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'10","last_scrub_stamp":"2024-03-11T20:53:12.822842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:53:12.822842+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:28.411634+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":97,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.323372+0000","last_change":"2024-03-11T20:52:51.931398+0000","last_active":"2024-03-11T20:53:38.323372+0000","last_peered":"2024-03-11T20:53:38.323372+0000","last_clean":"2024-03-11T20:53:38.323372+0000","last_became_active":"2024-03-11T20:50:32.194119+0000","last_became_peered":"2024-03-11T20:50:32.194119+0000","last_unstale":"2024-03-11T20:53:38.323372+0000","last_undegraded":"2024-03-11T20:53:38.323372+0000","last_fullsized":"2024-03-11T20:53:38.323372+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:51.931295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:52:51.931295+0000","last_clean_scrub_stamp":"2024-03-11T20:52:51.931295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:53:55.241599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":94,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.835186+0000","last_change":"2024-03-11T20:53:03.762967+0000","last_active":"2024-03-11T20:53:38.835186+0000","last_peered":"2024-03-11T20:53:38.835186+0000","last_clean":"2024-03-11T20:53:38.835186+0000","last_became_active":"2024-03-11T20:50:34.337173+0000","last_became_peered":"2024-03-11T20:50:34.337173+0000","last_unstale":"2024-03-11T20:53:38.835186+0000","last_undegraded":"2024-03-11T20:53:38.835186+0000","last_fullsized":"2024-03-11T20:53:38.835186+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:03.762882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:53:03.762882+0000","last_clean_scrub_stamp":"2024-03-11T20:53:03.762882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:29.931672+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":106,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.883164+0000","last_change":"2024-03-11T20:53:00.897188+0000","last_active":"2024-03-11T20:53:38.883164+0000","last_peered":"2024-03-11T20:53:38.883164+0000","last_clean":"2024-03-11T20:53:38.883164+0000","last_became_active":"2024-03-11T20:50:28.058942+0000","last_became_peered":"2024-03-11T20:50:28.058942+0000","last_unstale":"2024-03-11T20:53:38.883164+0000","last_undegraded":"2024-03-11T20:53:38.883164+0000","last_fullsized":"2024-03-11T20:53:38.883164+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:00.896807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:53:00.896807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:13.585784+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"84'15","reported_seq":127,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:38.336763+0000","last_change":"2024-03-11T20:53:09.345898+0000","last_active":"2024-03-11T20:53:38.336763+0000","last_peered":"2024-03-11T20:53:38.336763+0000","last_clean":"2024-03-11T20:53:38.336763+0000","last_became_active":"2024-03-11T20:50:30.101512+0000","last_became_peered":"2024-03-11T20:50:30.101512+0000","last_unstale":"2024-03-11T20:53:38.336763+0000","last_undegraded":"2024-03-11T20:53:38.336763+0000","last_fullsized":"2024-03-11T20:53:38.336763+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15","last_scrub_stamp":"2024-03-11T20:53:09.345405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:53:09.345405+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:54:28.384531+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":98,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-11T20:53:44.824836+0000","last_change":"2024-03-11T20:53:44.824870+0000","last_active":"2024-03-11T20:53:44.824836+0000","last_peered":"2024-03-11T20:53:44.824836+0000","last_clean":"2024-03-11T20:53:44.824836+0000","last_became_active":"2024-03-11T20:50:32.652854+0000","last_became_peered":"2024-03-11T20:50:32.652854+0000","last_unstale":"2024-03-11T20:53:44.824836+0000","last_undegraded":"2024-03-11T20:53:44.824836+0000","last_fullsized":"2024-03-11T20:53:44.824836+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:44.824748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:53:44.824748+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:08.788594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":178,"num_read_kb":173,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94,"ondisk_log_size":94,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":701,"num_read_kb":458,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":440,"ondisk_log_size":440,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":68,"seq":292057776179,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36660,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708492,"statfs":{"total":95995035648,"available":95957495808,"internally_reserved":0,"allocated":1355776,"data_stored":980421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":57,"seq":244813135933,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32484,"kb_used_data":1308,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712668,"statfs":{"total":95995035648,"available":95961772032,"internally_reserved":0,"allocated":1339392,"data_stored":980431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574960,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31912,"kb_used_data":736,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713240,"statfs":{"total":95995035648,"available":95962357760,"internally_reserved":0,"allocated":753664,"data_stored":390309,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738491,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32492,"kb_used_data":1316,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712660,"statfs":{"total":95995035648,"available":95961763840,"internally_reserved":0,"allocated":1347584,"data_stored":979363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378726,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31896,"kb_used_data":720,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713256,"statfs":{"total":95995035648,"available":95962374144,"internally_reserved":0,"allocated":737280,"data_stored":389157,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":28,"seq":120259084379,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27740,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717412,"statfs":{"total":95995035648,"available":95966629888,"internally_reserved":0,"allocated":741376,"data_stored":389533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":35,"seq":150323855442,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36096,"kb_used_data":760,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93709056,"statfs":{"total":95995035648,"available":95958073344,"internally_reserved":0,"allocated":778240,"data_stored":390923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":44,"seq":188978561095,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31900,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713252,"statfs":{"total":95995035648,"available":95962370048,"internally_reserved":0,"allocated":741376,"data_stored":389079,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T20:53:55.433 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-11T20:53:55.434 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd dump 2024-03-11T20:53:56.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:55 smithi146 bash[23809]: cluster 2024-03-11T20:53:54.307333+0000 osd.3 (osd.3) 101 : cluster 0 2.1e scrub starts 2024-03-11T20:53:56.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:55 smithi146 bash[23809]: cluster 2024-03-11T20:53:54.309354+0000 osd.3 (osd.3) 102 : cluster 0 2.1e scrub ok 2024-03-11T20:53:56.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:55 smithi146 bash[23809]: audit 2024-03-11T20:53:54.664975+0000 mgr.y (mgr.24380) 121 : audit 0 from='client.24572 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T20:53:56.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:55 smithi146 bash[23809]: audit 2024-03-11T20:53:54.937788+0000 mgr.y (mgr.24380) 122 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:53:56.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:55 smithi023 bash[21372]: cluster 2024-03-11T20:53:54.307333+0000 osd.3 (osd.3) 101 : cluster 0 2.1e scrub starts 2024-03-11T20:53:56.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:55 smithi023 bash[21372]: cluster 2024-03-11T20:53:54.309354+0000 osd.3 (osd.3) 102 : cluster 0 2.1e scrub ok 2024-03-11T20:53:56.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:55 smithi023 bash[21372]: audit 2024-03-11T20:53:54.664975+0000 mgr.y (mgr.24380) 121 : audit 0 from='client.24572 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T20:53:56.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:55 smithi023 bash[21372]: audit 2024-03-11T20:53:54.937788+0000 mgr.y (mgr.24380) 122 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:53:56.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:55 smithi023 bash[29146]: cluster 2024-03-11T20:53:54.307333+0000 osd.3 (osd.3) 101 : cluster 0 2.1e scrub starts 2024-03-11T20:53:56.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:55 smithi023 bash[29146]: cluster 2024-03-11T20:53:54.309354+0000 osd.3 (osd.3) 102 : cluster 0 2.1e scrub ok 2024-03-11T20:53:56.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:55 smithi023 bash[29146]: audit 2024-03-11T20:53:54.664975+0000 mgr.y (mgr.24380) 121 : audit 0 from='client.24572 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T20:53:56.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:55 smithi023 bash[29146]: audit 2024-03-11T20:53:54.937788+0000 mgr.y (mgr.24380) 122 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:53:57.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:56 smithi146 bash[23809]: cluster 2024-03-11T20:53:55.235322+0000 osd.7 (osd.7) 83 : cluster 0 1.0 scrub starts 2024-03-11T20:53:57.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:56 smithi146 bash[23809]: cluster 2024-03-11T20:53:55.238846+0000 osd.7 (osd.7) 84 : cluster 0 1.0 scrub ok 2024-03-11T20:53:57.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:56 smithi146 bash[23809]: cluster 2024-03-11T20:53:55.875853+0000 mgr.y (mgr.24380) 123 : cluster 0 pgmap v89: 132 pgs: 132 active+clean; 582 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:53:57.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:56 smithi146 bash[23809]: cluster 2024-03-11T20:53:55.882519+0000 osd.0 (osd.0) 89 : cluster 0 6.9 scrub starts 2024-03-11T20:53:57.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:56 smithi146 bash[23809]: cluster 2024-03-11T20:53:55.884817+0000 osd.0 (osd.0) 90 : cluster 0 6.9 scrub ok 2024-03-11T20:53:57.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:56 smithi146 bash[23809]: cluster 2024-03-11T20:53:56.022717+0000 osd.1 (osd.1) 83 : cluster 0 4.7 scrub starts 2024-03-11T20:53:57.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:56 smithi146 bash[23809]: cluster 2024-03-11T20:53:56.025175+0000 osd.1 (osd.1) 84 : cluster 0 4.7 scrub ok 2024-03-11T20:53:57.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:56 smithi023 bash[21372]: cluster 2024-03-11T20:53:55.235322+0000 osd.7 (osd.7) 83 : cluster 0 1.0 scrub starts 2024-03-11T20:53:57.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:56 smithi023 bash[21372]: cluster 2024-03-11T20:53:55.238846+0000 osd.7 (osd.7) 84 : cluster 0 1.0 scrub ok 2024-03-11T20:53:57.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:56 smithi023 bash[21372]: cluster 2024-03-11T20:53:55.875853+0000 mgr.y (mgr.24380) 123 : cluster 0 pgmap v89: 132 pgs: 132 active+clean; 582 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:53:57.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:56 smithi023 bash[21372]: cluster 2024-03-11T20:53:55.882519+0000 osd.0 (osd.0) 89 : cluster 0 6.9 scrub starts 2024-03-11T20:53:57.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:56 smithi023 bash[21372]: cluster 2024-03-11T20:53:55.884817+0000 osd.0 (osd.0) 90 : cluster 0 6.9 scrub ok 2024-03-11T20:53:57.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:56 smithi023 bash[21372]: cluster 2024-03-11T20:53:56.022717+0000 osd.1 (osd.1) 83 : cluster 0 4.7 scrub starts 2024-03-11T20:53:57.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:56 smithi023 bash[21372]: cluster 2024-03-11T20:53:56.025175+0000 osd.1 (osd.1) 84 : cluster 0 4.7 scrub ok 2024-03-11T20:53:57.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:56 smithi023 bash[29146]: cluster 2024-03-11T20:53:55.235322+0000 osd.7 (osd.7) 83 : cluster 0 1.0 scrub starts 2024-03-11T20:53:57.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:56 smithi023 bash[29146]: cluster 2024-03-11T20:53:55.238846+0000 osd.7 (osd.7) 84 : cluster 0 1.0 scrub ok 2024-03-11T20:53:57.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:56 smithi023 bash[29146]: cluster 2024-03-11T20:53:55.875853+0000 mgr.y (mgr.24380) 123 : cluster 0 pgmap v89: 132 pgs: 132 active+clean; 582 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:53:57.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:56 smithi023 bash[29146]: cluster 2024-03-11T20:53:55.882519+0000 osd.0 (osd.0) 89 : cluster 0 6.9 scrub starts 2024-03-11T20:53:57.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:56 smithi023 bash[29146]: cluster 2024-03-11T20:53:55.884817+0000 osd.0 (osd.0) 90 : cluster 0 6.9 scrub ok 2024-03-11T20:53:57.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:56 smithi023 bash[29146]: cluster 2024-03-11T20:53:56.022717+0000 osd.1 (osd.1) 83 : cluster 0 4.7 scrub starts 2024-03-11T20:53:57.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:56 smithi023 bash[29146]: cluster 2024-03-11T20:53:56.025175+0000 osd.1 (osd.1) 84 : cluster 0 4.7 scrub ok 2024-03-11T20:53:58.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:57 smithi146 bash[23809]: cluster 2024-03-11T20:53:56.586278+0000 osd.3 (osd.3) 103 : cluster 0 purged_snaps scrub starts 2024-03-11T20:53:58.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:57 smithi146 bash[23809]: cluster 2024-03-11T20:53:56.586421+0000 osd.3 (osd.3) 104 : cluster 0 purged_snaps scrub ok 2024-03-11T20:53:58.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:57 smithi146 bash[23809]: cluster 2024-03-11T20:53:56.963994+0000 mon.a (mon.0) 927 : cluster 0 osdmap e117: 8 total, 8 up, 8 in 2024-03-11T20:53:58.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:57 smithi146 bash[23809]: cluster 2024-03-11T20:53:57.641592+0000 osd.4 (osd.4) 87 : cluster 0 6.e scrub starts 2024-03-11T20:53:58.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:57 smithi146 bash[23809]: cluster 2024-03-11T20:53:57.644027+0000 osd.4 (osd.4) 88 : cluster 0 6.e scrub ok 2024-03-11T20:53:58.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:57 smithi023 bash[21372]: cluster 2024-03-11T20:53:56.586278+0000 osd.3 (osd.3) 103 : cluster 0 purged_snaps scrub starts 2024-03-11T20:53:58.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:57 smithi023 bash[21372]: cluster 2024-03-11T20:53:56.586421+0000 osd.3 (osd.3) 104 : cluster 0 purged_snaps scrub ok 2024-03-11T20:53:58.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:57 smithi023 bash[21372]: cluster 2024-03-11T20:53:56.963994+0000 mon.a (mon.0) 927 : cluster 0 osdmap e117: 8 total, 8 up, 8 in 2024-03-11T20:53:58.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:57 smithi023 bash[21372]: cluster 2024-03-11T20:53:57.641592+0000 osd.4 (osd.4) 87 : cluster 0 6.e scrub starts 2024-03-11T20:53:58.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:57 smithi023 bash[21372]: cluster 2024-03-11T20:53:57.644027+0000 osd.4 (osd.4) 88 : cluster 0 6.e scrub ok 2024-03-11T20:53:58.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:57 smithi023 bash[29146]: cluster 2024-03-11T20:53:56.586278+0000 osd.3 (osd.3) 103 : cluster 0 purged_snaps scrub starts 2024-03-11T20:53:58.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:57 smithi023 bash[29146]: cluster 2024-03-11T20:53:56.586421+0000 osd.3 (osd.3) 104 : cluster 0 purged_snaps scrub ok 2024-03-11T20:53:58.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:57 smithi023 bash[29146]: cluster 2024-03-11T20:53:56.963994+0000 mon.a (mon.0) 927 : cluster 0 osdmap e117: 8 total, 8 up, 8 in 2024-03-11T20:53:58.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:57 smithi023 bash[29146]: cluster 2024-03-11T20:53:57.641592+0000 osd.4 (osd.4) 87 : cluster 0 6.e scrub starts 2024-03-11T20:53:58.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:57 smithi023 bash[29146]: cluster 2024-03-11T20:53:57.644027+0000 osd.4 (osd.4) 88 : cluster 0 6.e scrub ok 2024-03-11T20:53:58.784 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:53:58 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:20:53:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T20:53:59.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:58 smithi146 bash[23809]: cluster 2024-03-11T20:53:57.247580+0000 osd.7 (osd.7) 85 : cluster 0 3.d scrub starts 2024-03-11T20:53:59.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:58 smithi146 bash[23809]: cluster 2024-03-11T20:53:57.252603+0000 osd.7 (osd.7) 86 : cluster 0 3.d scrub ok 2024-03-11T20:53:59.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:58 smithi146 bash[23809]: cluster 2024-03-11T20:53:57.728688+0000 osd.2 (osd.2) 51 : cluster 0 2.0 scrub starts 2024-03-11T20:53:59.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:58 smithi146 bash[23809]: cluster 2024-03-11T20:53:57.731094+0000 osd.2 (osd.2) 52 : cluster 0 2.0 scrub ok 2024-03-11T20:53:59.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:53:58 smithi146 bash[23809]: cluster 2024-03-11T20:53:57.876645+0000 mgr.y (mgr.24380) 124 : cluster 0 pgmap v91: 132 pgs: 132 active+clean; 582 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:53:59.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:58 smithi023 bash[21372]: cluster 2024-03-11T20:53:57.247580+0000 osd.7 (osd.7) 85 : cluster 0 3.d scrub starts 2024-03-11T20:53:59.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:58 smithi023 bash[21372]: cluster 2024-03-11T20:53:57.252603+0000 osd.7 (osd.7) 86 : cluster 0 3.d scrub ok 2024-03-11T20:53:59.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:58 smithi023 bash[21372]: cluster 2024-03-11T20:53:57.728688+0000 osd.2 (osd.2) 51 : cluster 0 2.0 scrub starts 2024-03-11T20:53:59.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:58 smithi023 bash[21372]: cluster 2024-03-11T20:53:57.731094+0000 osd.2 (osd.2) 52 : cluster 0 2.0 scrub ok 2024-03-11T20:53:59.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:53:58 smithi023 bash[21372]: cluster 2024-03-11T20:53:57.876645+0000 mgr.y (mgr.24380) 124 : cluster 0 pgmap v91: 132 pgs: 132 active+clean; 582 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:53:59.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:58 smithi023 bash[29146]: cluster 2024-03-11T20:53:57.247580+0000 osd.7 (osd.7) 85 : cluster 0 3.d scrub starts 2024-03-11T20:53:59.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:58 smithi023 bash[29146]: cluster 2024-03-11T20:53:57.252603+0000 osd.7 (osd.7) 86 : cluster 0 3.d scrub ok 2024-03-11T20:53:59.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:58 smithi023 bash[29146]: cluster 2024-03-11T20:53:57.728688+0000 osd.2 (osd.2) 51 : cluster 0 2.0 scrub starts 2024-03-11T20:53:59.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:58 smithi023 bash[29146]: cluster 2024-03-11T20:53:57.731094+0000 osd.2 (osd.2) 52 : cluster 0 2.0 scrub ok 2024-03-11T20:53:59.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:53:58 smithi023 bash[29146]: cluster 2024-03-11T20:53:57.876645+0000 mgr.y (mgr.24380) 124 : cluster 0 pgmap v91: 132 pgs: 132 active+clean; 582 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:54:00.231 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:54:01.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:00 smithi146 bash[23809]: cluster 2024-03-11T20:53:59.738663+0000 osd.0 (osd.0) 91 : cluster 0 4.b scrub starts 2024-03-11T20:54:01.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:00 smithi146 bash[23809]: cluster 2024-03-11T20:53:59.740659+0000 osd.0 (osd.0) 92 : cluster 0 4.b scrub ok 2024-03-11T20:54:01.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:00 smithi146 bash[23809]: cluster 2024-03-11T20:53:59.878210+0000 mgr.y (mgr.24380) 125 : cluster 0 pgmap v92: 132 pgs: 132 active+clean; 582 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:54:01.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:00 smithi023 bash[21372]: cluster 2024-03-11T20:53:59.738663+0000 osd.0 (osd.0) 91 : cluster 0 4.b scrub starts 2024-03-11T20:54:01.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:00 smithi023 bash[21372]: cluster 2024-03-11T20:53:59.740659+0000 osd.0 (osd.0) 92 : cluster 0 4.b scrub ok 2024-03-11T20:54:01.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:00 smithi023 bash[21372]: cluster 2024-03-11T20:53:59.878210+0000 mgr.y (mgr.24380) 125 : cluster 0 pgmap v92: 132 pgs: 132 active+clean; 582 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:54:01.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:00 smithi023 bash[29146]: cluster 2024-03-11T20:53:59.738663+0000 osd.0 (osd.0) 91 : cluster 0 4.b scrub starts 2024-03-11T20:54:01.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:00 smithi023 bash[29146]: cluster 2024-03-11T20:53:59.740659+0000 osd.0 (osd.0) 92 : cluster 0 4.b scrub ok 2024-03-11T20:54:01.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:00 smithi023 bash[29146]: cluster 2024-03-11T20:53:59.878210+0000 mgr.y (mgr.24380) 125 : cluster 0 pgmap v92: 132 pgs: 132 active+clean; 582 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:54:02.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:01 smithi146 bash[23809]: cluster 2024-03-11T20:54:00.453313+0000 osd.3 (osd.3) 105 : cluster 0 2.19 scrub starts 2024-03-11T20:54:02.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:01 smithi146 bash[23809]: cluster 2024-03-11T20:54:00.456338+0000 osd.3 (osd.3) 106 : cluster 0 2.19 scrub ok 2024-03-11T20:54:02.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:01 smithi023 bash[21372]: cluster 2024-03-11T20:54:00.453313+0000 osd.3 (osd.3) 105 : cluster 0 2.19 scrub starts 2024-03-11T20:54:02.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:01 smithi023 bash[21372]: cluster 2024-03-11T20:54:00.456338+0000 osd.3 (osd.3) 106 : cluster 0 2.19 scrub ok 2024-03-11T20:54:02.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:01 smithi023 bash[29146]: cluster 2024-03-11T20:54:00.453313+0000 osd.3 (osd.3) 105 : cluster 0 2.19 scrub starts 2024-03-11T20:54:02.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:01 smithi023 bash[29146]: cluster 2024-03-11T20:54:00.456338+0000 osd.3 (osd.3) 106 : cluster 0 2.19 scrub ok 2024-03-11T20:54:02.870 INFO:teuthology.orchestra.run.smithi023.stdout:epoch 117 2024-03-11T20:54:02.871 INFO:teuthology.orchestra.run.smithi023.stdout:fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 2024-03-11T20:54:02.871 INFO:teuthology.orchestra.run.smithi023.stdout:created 2024-03-11T20:40:13.664802+0000 2024-03-11T20:54:02.871 INFO:teuthology.orchestra.run.smithi023.stdout:modified 2024-03-11T20:53:56.931245+0000 2024-03-11T20:54:02.871 INFO:teuthology.orchestra.run.smithi023.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-03-11T20:54:02.871 INFO:teuthology.orchestra.run.smithi023.stdout:crush_version 18 2024-03-11T20:54:02.871 INFO:teuthology.orchestra.run.smithi023.stdout:full_ratio 0.95 2024-03-11T20:54:02.871 INFO:teuthology.orchestra.run.smithi023.stdout:backfillfull_ratio 0.9 2024-03-11T20:54:02.871 INFO:teuthology.orchestra.run.smithi023.stdout:nearfull_ratio 0.85 2024-03-11T20:54:02.872 INFO:teuthology.orchestra.run.smithi023.stdout:require_min_compat_client luminous 2024-03-11T20:54:02.872 INFO:teuthology.orchestra.run.smithi023.stdout:min_compat_client luminous 2024-03-11T20:54:02.872 INFO:teuthology.orchestra.run.smithi023.stdout:require_osd_release squid 2024-03-11T20:54:02.872 INFO:teuthology.orchestra.run.smithi023.stdout:stretch_mode_enabled false 2024-03-11T20:54:02.872 INFO:teuthology.orchestra.run.smithi023.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 22 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 7.89 2024-03-11T20:54:02.872 INFO:teuthology.orchestra.run.smithi023.stdout:pool 2 '.rgw.root' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 78 flags hashpspool stripe_width 0 application rgw read_balance_score 1.50 2024-03-11T20:54:02.872 INFO:teuthology.orchestra.run.smithi023.stdout:pool 3 'default.rgw.log' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 80 flags hashpspool stripe_width 0 application rgw read_balance_score 1.50 2024-03-11T20:54:02.872 INFO:teuthology.orchestra.run.smithi023.stdout:pool 4 'default.rgw.control' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 82 flags hashpspool stripe_width 0 application rgw read_balance_score 2.25 2024-03-11T20:54:02.873 INFO:teuthology.orchestra.run.smithi023.stdout:pool 5 'datapool' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 3 pgp_num 3 autoscale_mode off last_change 87 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 2.65 2024-03-11T20:54:02.873 INFO:teuthology.orchestra.run.smithi023.stdout:pool 6 'default.rgw.meta' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 84 flags hashpspool stripe_width 0 pg_autoscale_bias 4 application rgw read_balance_score 1.75 2024-03-11T20:54:02.873 INFO:teuthology.orchestra.run.smithi023.stdout:max_osd 8 2024-03-11T20:54:02.873 INFO:teuthology.orchestra.run.smithi023.stdout:osd.0 up in weight 1 up_from 8 up_thru 82 down_at 0 last_clean_interval [0,0) [v2:172.21.15.23:6802/2154627393,v1:172.21.15.23:6803/2154627393] [v2:172.21.15.23:6804/2154627393,v1:172.21.15.23:6805/2154627393] exists,up 3df72b4a-8fe4-4265-b4c6-af022e1e23e1 2024-03-11T20:54:02.873 INFO:teuthology.orchestra.run.smithi023.stdout:osd.1 up in weight 1 up_from 13 up_thru 82 down_at 0 last_clean_interval [0,0) [v2:172.21.15.23:6810/2506707705,v1:172.21.15.23:6811/2506707705] [v2:172.21.15.23:6812/2506707705,v1:172.21.15.23:6813/2506707705] exists,up e8fd7500-9081-41a1-8077-a4d0f0e12a2c 2024-03-11T20:54:02.873 INFO:teuthology.orchestra.run.smithi023.stdout:osd.2 up in weight 1 up_from 19 up_thru 97 down_at 0 last_clean_interval [0,0) [v2:172.21.15.23:6818/1770748056,v1:172.21.15.23:6819/1770748056] [v2:172.21.15.23:6820/1770748056,v1:172.21.15.23:6821/1770748056] exists,up dd85d52a-df48-4b03-8612-cecfd82b5987 2024-03-11T20:54:02.873 INFO:teuthology.orchestra.run.smithi023.stdout:osd.3 up in weight 1 up_from 28 up_thru 97 down_at 0 last_clean_interval [0,0) [v2:172.21.15.23:6826/938496814,v1:172.21.15.23:6827/938496814] [v2:172.21.15.23:6828/938496814,v1:172.21.15.23:6829/938496814] exists,up e3021106-3779-4b94-9bbf-a7724544110f 2024-03-11T20:54:02.873 INFO:teuthology.orchestra.run.smithi023.stdout:osd.4 up in weight 1 up_from 35 up_thru 97 down_at 0 last_clean_interval [0,0) [v2:172.21.15.146:6800/615746847,v1:172.21.15.146:6801/615746847] [v2:172.21.15.146:6802/615746847,v1:172.21.15.146:6803/615746847] exists,up 4dcbc91e-7035-413d-97bc-c9d804eb16d7 2024-03-11T20:54:02.874 INFO:teuthology.orchestra.run.smithi023.stdout:osd.5 up in weight 1 up_from 44 up_thru 82 down_at 0 last_clean_interval [0,0) [v2:172.21.15.146:6808/4003230814,v1:172.21.15.146:6809/4003230814] [v2:172.21.15.146:6810/4003230814,v1:172.21.15.146:6811/4003230814] exists,up aefcb766-8bae-4ff4-b71e-6691c8ac02f1 2024-03-11T20:54:02.874 INFO:teuthology.orchestra.run.smithi023.stdout:osd.6 up in weight 1 up_from 57 up_thru 80 down_at 0 last_clean_interval [0,0) [v2:172.21.15.146:6816/3334092376,v1:172.21.15.146:6817/3334092376] [v2:172.21.15.146:6818/3334092376,v1:172.21.15.146:6819/3334092376] exists,up 5cb151e8-1f9b-4617-8a86-adfb06bb9606 2024-03-11T20:54:02.874 INFO:teuthology.orchestra.run.smithi023.stdout:osd.7 up in weight 1 up_from 68 up_thru 82 down_at 0 last_clean_interval [0,0) [v2:172.21.15.146:6824/1678407923,v1:172.21.15.146:6825/1678407923] [v2:172.21.15.146:6826/1678407923,v1:172.21.15.146:6827/1678407923] exists,up bdbd11a2-fa09-4f82-93a2-6b2d9c750bfd 2024-03-11T20:54:02.874 INFO:teuthology.orchestra.run.smithi023.stdout:pg_upmap_items 2.0 [7,2] 2024-03-11T20:54:02.874 INFO:teuthology.orchestra.run.smithi023.stdout:pg_upmap_items 4.d [1,5] 2024-03-11T20:54:02.874 INFO:teuthology.orchestra.run.smithi023.stdout:pg_upmap_items 4.f [1,2] 2024-03-11T20:54:02.874 INFO:teuthology.orchestra.run.smithi023.stdout:pg_upmap_items 4.14 [1,2] 2024-03-11T20:54:02.874 INFO:teuthology.orchestra.run.smithi023.stdout:blocklist 172.21.15.23:0/1618334186 expires 2024-03-12T20:40:45.361537+0000 2024-03-11T20:54:02.874 INFO:teuthology.orchestra.run.smithi023.stdout:blocklist 172.21.15.23:6801/2813578081 expires 2024-03-12T20:40:45.361537+0000 2024-03-11T20:54:02.875 INFO:teuthology.orchestra.run.smithi023.stdout:blocklist 172.21.15.23:0/2869940180 expires 2024-03-12T20:51:33.790205+0000 2024-03-11T20:54:02.875 INFO:teuthology.orchestra.run.smithi023.stdout:blocklist 172.21.15.23:6800/3037564930 expires 2024-03-12T20:51:33.790205+0000 2024-03-11T20:54:02.875 INFO:teuthology.orchestra.run.smithi023.stdout:blocklist 172.21.15.23:0/3297567084 expires 2024-03-12T20:51:33.790205+0000 2024-03-11T20:54:02.875 INFO:teuthology.orchestra.run.smithi023.stdout:blocklist 172.21.15.23:0/3708753659 expires 2024-03-12T20:40:45.361537+0000 2024-03-11T20:54:02.875 INFO:teuthology.orchestra.run.smithi023.stdout:blocklist 172.21.15.23:0/2004146740 expires 2024-03-12T20:40:45.361537+0000 2024-03-11T20:54:02.875 INFO:teuthology.orchestra.run.smithi023.stdout:blocklist 172.21.15.23:0/827653812 expires 2024-03-12T20:41:21.301516+0000 2024-03-11T20:54:02.875 INFO:teuthology.orchestra.run.smithi023.stdout:blocklist 172.21.15.23:6800/1057699850 expires 2024-03-12T20:41:21.301516+0000 2024-03-11T20:54:02.875 INFO:teuthology.orchestra.run.smithi023.stdout:blocklist 172.21.15.23:6800/2813578081 expires 2024-03-12T20:40:45.361537+0000 2024-03-11T20:54:02.876 INFO:teuthology.orchestra.run.smithi023.stdout:blocklist 172.21.15.23:6801/1057699850 expires 2024-03-12T20:41:21.301516+0000 2024-03-11T20:54:02.876 INFO:teuthology.orchestra.run.smithi023.stdout:blocklist 172.21.15.23:0/3833230375 expires 2024-03-12T20:41:21.301516+0000 2024-03-11T20:54:02.876 INFO:teuthology.orchestra.run.smithi023.stdout:blocklist 172.21.15.23:6801/3037564930 expires 2024-03-12T20:51:33.790205+0000 2024-03-11T20:54:02.876 INFO:teuthology.orchestra.run.smithi023.stdout:blocklist 172.21.15.23:0/1182809102 expires 2024-03-12T20:51:33.790205+0000 2024-03-11T20:54:02.876 INFO:teuthology.orchestra.run.smithi023.stdout:blocklist 172.21.15.23:0/2122134187 expires 2024-03-12T20:51:33.790205+0000 2024-03-11T20:54:02.876 INFO:teuthology.orchestra.run.smithi023.stdout:blocklist 172.21.15.23:0/2356701899 expires 2024-03-12T20:41:21.301516+0000 2024-03-11T20:54:02.876 INFO:teuthology.orchestra.run.smithi023.stdout:blocklist 172.21.15.23:0/843974100 expires 2024-03-12T20:51:33.790205+0000 2024-03-11T20:54:03.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:02 smithi146 bash[23809]: cluster 2024-03-11T20:54:01.689773+0000 osd.0 (osd.0) 93 : cluster 0 2.13 scrub starts 2024-03-11T20:54:03.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:02 smithi146 bash[23809]: cluster 2024-03-11T20:54:01.692186+0000 osd.0 (osd.0) 94 : cluster 0 2.13 scrub ok 2024-03-11T20:54:03.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:02 smithi146 bash[23809]: cluster 2024-03-11T20:54:01.879059+0000 mgr.y (mgr.24380) 126 : cluster 0 pgmap v93: 132 pgs: 132 active+clean; 582 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:54:03.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:02 smithi146 bash[23809]: cluster 2024-03-11T20:54:02.635999+0000 osd.4 (osd.4) 89 : cluster 0 4.9 scrub starts 2024-03-11T20:54:03.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:02 smithi146 bash[23809]: cluster 2024-03-11T20:54:02.638992+0000 osd.4 (osd.4) 90 : cluster 0 4.9 scrub ok 2024-03-11T20:54:03.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:02 smithi146 bash[23809]: audit 2024-03-11T20:54:02.871894+0000 mon.c (mon.2) 26 : audit 0 from='client.? 172.21.15.23:0/3276451575' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-11T20:54:03.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:02 smithi023 bash[21372]: cluster 2024-03-11T20:54:01.689773+0000 osd.0 (osd.0) 93 : cluster 0 2.13 scrub starts 2024-03-11T20:54:03.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:02 smithi023 bash[21372]: cluster 2024-03-11T20:54:01.692186+0000 osd.0 (osd.0) 94 : cluster 0 2.13 scrub ok 2024-03-11T20:54:03.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:02 smithi023 bash[21372]: cluster 2024-03-11T20:54:01.879059+0000 mgr.y (mgr.24380) 126 : cluster 0 pgmap v93: 132 pgs: 132 active+clean; 582 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:54:03.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:02 smithi023 bash[21372]: cluster 2024-03-11T20:54:02.635999+0000 osd.4 (osd.4) 89 : cluster 0 4.9 scrub starts 2024-03-11T20:54:03.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:02 smithi023 bash[21372]: cluster 2024-03-11T20:54:02.638992+0000 osd.4 (osd.4) 90 : cluster 0 4.9 scrub ok 2024-03-11T20:54:03.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:02 smithi023 bash[21372]: audit 2024-03-11T20:54:02.871894+0000 mon.c (mon.2) 26 : audit 0 from='client.? 172.21.15.23:0/3276451575' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-11T20:54:03.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:02 smithi023 bash[29146]: cluster 2024-03-11T20:54:01.689773+0000 osd.0 (osd.0) 93 : cluster 0 2.13 scrub starts 2024-03-11T20:54:03.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:02 smithi023 bash[29146]: cluster 2024-03-11T20:54:01.692186+0000 osd.0 (osd.0) 94 : cluster 0 2.13 scrub ok 2024-03-11T20:54:03.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:02 smithi023 bash[29146]: cluster 2024-03-11T20:54:01.879059+0000 mgr.y (mgr.24380) 126 : cluster 0 pgmap v93: 132 pgs: 132 active+clean; 582 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:54:03.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:02 smithi023 bash[29146]: cluster 2024-03-11T20:54:02.635999+0000 osd.4 (osd.4) 89 : cluster 0 4.9 scrub starts 2024-03-11T20:54:03.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:02 smithi023 bash[29146]: cluster 2024-03-11T20:54:02.638992+0000 osd.4 (osd.4) 90 : cluster 0 4.9 scrub ok 2024-03-11T20:54:03.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:02 smithi023 bash[29146]: audit 2024-03-11T20:54:02.871894+0000 mon.c (mon.2) 26 : audit 0 from='client.? 172.21.15.23:0/3276451575' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-11T20:54:03.516 INFO:tasks.cephadm.ceph_manager.ceph:['osd.0 up in weight 1 up_from 8 up_thru 82 down_at 0 last_clean_interval [0,0) [v2:172.21.15.23:6802/2154627393,v1:172.21.15.23:6803/2154627393] [v2:172.21.15.23:6804/2154627393,v1:172.21.15.23:6805/2154627393] exists,up 3df72b4a-8fe4-4265-b4c6-af022e1e23e1', 'osd.1 up in weight 1 up_from 13 up_thru 82 down_at 0 last_clean_interval [0,0) [v2:172.21.15.23:6810/2506707705,v1:172.21.15.23:6811/2506707705] [v2:172.21.15.23:6812/2506707705,v1:172.21.15.23:6813/2506707705] exists,up e8fd7500-9081-41a1-8077-a4d0f0e12a2c', 'osd.2 up in weight 1 up_from 19 up_thru 97 down_at 0 last_clean_interval [0,0) [v2:172.21.15.23:6818/1770748056,v1:172.21.15.23:6819/1770748056] [v2:172.21.15.23:6820/1770748056,v1:172.21.15.23:6821/1770748056] exists,up dd85d52a-df48-4b03-8612-cecfd82b5987', 'osd.3 up in weight 1 up_from 28 up_thru 97 down_at 0 last_clean_interval [0,0) [v2:172.21.15.23:6826/938496814,v1:172.21.15.23:6827/938496814] [v2:172.21.15.23:6828/938496814,v1:172.21.15.23:6829/938496814] exists,up e3021106-3779-4b94-9bbf-a7724544110f', 'osd.4 up in weight 1 up_from 35 up_thru 97 down_at 0 last_clean_interval [0,0) [v2:172.21.15.146:6800/615746847,v1:172.21.15.146:6801/615746847] [v2:172.21.15.146:6802/615746847,v1:172.21.15.146:6803/615746847] exists,up 4dcbc91e-7035-413d-97bc-c9d804eb16d7', 'osd.5 up in weight 1 up_from 44 up_thru 82 down_at 0 last_clean_interval [0,0) [v2:172.21.15.146:6808/4003230814,v1:172.21.15.146:6809/4003230814] [v2:172.21.15.146:6810/4003230814,v1:172.21.15.146:6811/4003230814] exists,up aefcb766-8bae-4ff4-b71e-6691c8ac02f1', 'osd.6 up in weight 1 up_from 57 up_thru 80 down_at 0 last_clean_interval [0,0) [v2:172.21.15.146:6816/3334092376,v1:172.21.15.146:6817/3334092376] [v2:172.21.15.146:6818/3334092376,v1:172.21.15.146:6819/3334092376] exists,up 5cb151e8-1f9b-4617-8a86-adfb06bb9606', 'osd.7 up in weight 1 up_from 68 up_thru 82 down_at 0 last_clean_interval [0,0) [v2:172.21.15.146:6824/1678407923,v1:172.21.15.146:6825/1678407923] [v2:172.21.15.146:6826/1678407923,v1:172.21.15.146:6827/1678407923] exists,up bdbd11a2-fa09-4f82-93a2-6b2d9c750bfd'] 2024-03-11T20:54:03.517 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph daemon mon.a config show 2024-03-11T20:54:05.260 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 20:54:04 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T20:54:05.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:05 smithi146 bash[23809]: cluster 2024-03-11T20:54:03.879738+0000 mgr.y (mgr.24380) 127 : cluster 0 pgmap v94: 132 pgs: 132 active+clean; 582 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:54:05.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:05 smithi146 bash[23809]: audit 2024-03-11T20:54:04.209628+0000 mon.a (mon.0) 928 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:54:05.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:05 smithi023 bash[21372]: cluster 2024-03-11T20:54:03.879738+0000 mgr.y (mgr.24380) 127 : cluster 0 pgmap v94: 132 pgs: 132 active+clean; 582 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:54:05.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:05 smithi023 bash[21372]: audit 2024-03-11T20:54:04.209628+0000 mon.a (mon.0) 928 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:54:05.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:05 smithi023 bash[29146]: cluster 2024-03-11T20:54:03.879738+0000 mgr.y (mgr.24380) 127 : cluster 0 pgmap v94: 132 pgs: 132 active+clean; 582 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:54:05.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:05 smithi023 bash[29146]: audit 2024-03-11T20:54:04.209628+0000 mon.a (mon.0) 928 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:54:06.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:06 smithi023 bash[21372]: cluster 2024-03-11T20:54:04.148969+0000 osd.7 (osd.7) 87 : cluster 0 3.e scrub starts 2024-03-11T20:54:06.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:06 smithi023 bash[21372]: cluster 2024-03-11T20:54:04.153279+0000 osd.7 (osd.7) 88 : cluster 0 3.e scrub ok 2024-03-11T20:54:06.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:06 smithi023 bash[21372]: audit 2024-03-11T20:54:04.949064+0000 mgr.y (mgr.24380) 128 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:54:06.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:06 smithi023 bash[29146]: cluster 2024-03-11T20:54:04.148969+0000 osd.7 (osd.7) 87 : cluster 0 3.e scrub starts 2024-03-11T20:54:06.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:06 smithi023 bash[29146]: cluster 2024-03-11T20:54:04.153279+0000 osd.7 (osd.7) 88 : cluster 0 3.e scrub ok 2024-03-11T20:54:06.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:06 smithi023 bash[29146]: audit 2024-03-11T20:54:04.949064+0000 mgr.y (mgr.24380) 128 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:54:06.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:06 smithi146 bash[23809]: cluster 2024-03-11T20:54:04.148969+0000 osd.7 (osd.7) 87 : cluster 0 3.e scrub starts 2024-03-11T20:54:06.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:06 smithi146 bash[23809]: cluster 2024-03-11T20:54:04.153279+0000 osd.7 (osd.7) 88 : cluster 0 3.e scrub ok 2024-03-11T20:54:06.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:06 smithi146 bash[23809]: audit 2024-03-11T20:54:04.949064+0000 mgr.y (mgr.24380) 128 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:54:07.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:07 smithi023 bash[21372]: cluster 2024-03-11T20:54:05.128427+0000 osd.7 (osd.7) 89 : cluster 0 2.b scrub starts 2024-03-11T20:54:07.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:07 smithi023 bash[21372]: cluster 2024-03-11T20:54:05.131026+0000 osd.7 (osd.7) 90 : cluster 0 2.b scrub ok 2024-03-11T20:54:07.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:07 smithi023 bash[21372]: cluster 2024-03-11T20:54:05.493942+0000 osd.3 (osd.3) 107 : cluster 0 3.2 scrub starts 2024-03-11T20:54:07.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:07 smithi023 bash[21372]: cluster 2024-03-11T20:54:05.498595+0000 osd.3 (osd.3) 108 : cluster 0 3.2 scrub ok 2024-03-11T20:54:07.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:07 smithi023 bash[21372]: cluster 2024-03-11T20:54:05.593038+0000 osd.0 (osd.0) 95 : cluster 0 purged_snaps scrub starts 2024-03-11T20:54:07.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:07 smithi023 bash[21372]: cluster 2024-03-11T20:54:05.593130+0000 osd.0 (osd.0) 96 : cluster 0 purged_snaps scrub ok 2024-03-11T20:54:07.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:07 smithi023 bash[21372]: cluster 2024-03-11T20:54:05.778670+0000 osd.2 (osd.2) 53 : cluster 0 6.f scrub starts 2024-03-11T20:54:07.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:07 smithi023 bash[21372]: cluster 2024-03-11T20:54:05.780777+0000 osd.2 (osd.2) 54 : cluster 0 6.f scrub ok 2024-03-11T20:54:07.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:07 smithi023 bash[21372]: cluster 2024-03-11T20:54:05.881179+0000 mgr.y (mgr.24380) 129 : cluster 0 pgmap v95: 132 pgs: 132 active+clean; 582 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:54:07.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:07 smithi023 bash[21372]: cluster 2024-03-11T20:54:06.044657+0000 mon.a (mon.0) 929 : cluster 0 osdmap e118: 8 total, 8 up, 8 in 2024-03-11T20:54:07.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:07 smithi023 bash[29146]: cluster 2024-03-11T20:54:05.128427+0000 osd.7 (osd.7) 89 : cluster 0 2.b scrub starts 2024-03-11T20:54:07.286 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:07 smithi023 bash[29146]: cluster 2024-03-11T20:54:05.131026+0000 osd.7 (osd.7) 90 : cluster 0 2.b scrub ok 2024-03-11T20:54:07.286 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:07 smithi023 bash[29146]: cluster 2024-03-11T20:54:05.493942+0000 osd.3 (osd.3) 107 : cluster 0 3.2 scrub starts 2024-03-11T20:54:07.286 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:07 smithi023 bash[29146]: cluster 2024-03-11T20:54:05.498595+0000 osd.3 (osd.3) 108 : cluster 0 3.2 scrub ok 2024-03-11T20:54:07.286 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:07 smithi023 bash[29146]: cluster 2024-03-11T20:54:05.593038+0000 osd.0 (osd.0) 95 : cluster 0 purged_snaps scrub starts 2024-03-11T20:54:07.286 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:07 smithi023 bash[29146]: cluster 2024-03-11T20:54:05.593130+0000 osd.0 (osd.0) 96 : cluster 0 purged_snaps scrub ok 2024-03-11T20:54:07.286 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:07 smithi023 bash[29146]: cluster 2024-03-11T20:54:05.778670+0000 osd.2 (osd.2) 53 : cluster 0 6.f scrub starts 2024-03-11T20:54:07.286 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:07 smithi023 bash[29146]: cluster 2024-03-11T20:54:05.780777+0000 osd.2 (osd.2) 54 : cluster 0 6.f scrub ok 2024-03-11T20:54:07.286 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:07 smithi023 bash[29146]: cluster 2024-03-11T20:54:05.881179+0000 mgr.y (mgr.24380) 129 : cluster 0 pgmap v95: 132 pgs: 132 active+clean; 582 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:54:07.286 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:07 smithi023 bash[29146]: cluster 2024-03-11T20:54:06.044657+0000 mon.a (mon.0) 929 : cluster 0 osdmap e118: 8 total, 8 up, 8 in 2024-03-11T20:54:07.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:07 smithi146 bash[23809]: cluster 2024-03-11T20:54:05.128427+0000 osd.7 (osd.7) 89 : cluster 0 2.b scrub starts 2024-03-11T20:54:07.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:07 smithi146 bash[23809]: cluster 2024-03-11T20:54:05.131026+0000 osd.7 (osd.7) 90 : cluster 0 2.b scrub ok 2024-03-11T20:54:07.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:07 smithi146 bash[23809]: cluster 2024-03-11T20:54:05.493942+0000 osd.3 (osd.3) 107 : cluster 0 3.2 scrub starts 2024-03-11T20:54:07.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:07 smithi146 bash[23809]: cluster 2024-03-11T20:54:05.498595+0000 osd.3 (osd.3) 108 : cluster 0 3.2 scrub ok 2024-03-11T20:54:07.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:07 smithi146 bash[23809]: cluster 2024-03-11T20:54:05.593038+0000 osd.0 (osd.0) 95 : cluster 0 purged_snaps scrub starts 2024-03-11T20:54:07.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:07 smithi146 bash[23809]: cluster 2024-03-11T20:54:05.593130+0000 osd.0 (osd.0) 96 : cluster 0 purged_snaps scrub ok 2024-03-11T20:54:07.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:07 smithi146 bash[23809]: cluster 2024-03-11T20:54:05.778670+0000 osd.2 (osd.2) 53 : cluster 0 6.f scrub starts 2024-03-11T20:54:07.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:07 smithi146 bash[23809]: cluster 2024-03-11T20:54:05.780777+0000 osd.2 (osd.2) 54 : cluster 0 6.f scrub ok 2024-03-11T20:54:07.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:07 smithi146 bash[23809]: cluster 2024-03-11T20:54:05.881179+0000 mgr.y (mgr.24380) 129 : cluster 0 pgmap v95: 132 pgs: 132 active+clean; 582 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:54:07.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:07 smithi146 bash[23809]: cluster 2024-03-11T20:54:06.044657+0000 mon.a (mon.0) 929 : cluster 0 osdmap e118: 8 total, 8 up, 8 in 2024-03-11T20:54:08.311 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:54:08.332 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:08 smithi023 bash[21372]: cluster 2024-03-11T20:54:06.513535+0000 osd.3 (osd.3) 109 : cluster 0 4.11 deep-scrub starts 2024-03-11T20:54:08.332 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:08 smithi023 bash[21372]: cluster 2024-03-11T20:54:06.516401+0000 osd.3 (osd.3) 110 : cluster 0 4.11 deep-scrub ok 2024-03-11T20:54:08.332 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:08 smithi023 bash[21372]: cluster 2024-03-11T20:54:06.807081+0000 osd.2 (osd.2) 55 : cluster 0 6.14 scrub starts 2024-03-11T20:54:08.332 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:08 smithi023 bash[21372]: cluster 2024-03-11T20:54:06.809725+0000 osd.2 (osd.2) 56 : cluster 0 6.14 scrub ok 2024-03-11T20:54:08.332 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:08 smithi023 bash[29146]: cluster 2024-03-11T20:54:06.513535+0000 osd.3 (osd.3) 109 : cluster 0 4.11 deep-scrub starts 2024-03-11T20:54:08.333 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:08 smithi023 bash[29146]: cluster 2024-03-11T20:54:06.516401+0000 osd.3 (osd.3) 110 : cluster 0 4.11 deep-scrub ok 2024-03-11T20:54:08.333 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:08 smithi023 bash[29146]: cluster 2024-03-11T20:54:06.807081+0000 osd.2 (osd.2) 55 : cluster 0 6.14 scrub starts 2024-03-11T20:54:08.333 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:08 smithi023 bash[29146]: cluster 2024-03-11T20:54:06.809725+0000 osd.2 (osd.2) 56 : cluster 0 6.14 scrub ok 2024-03-11T20:54:08.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:08 smithi146 bash[23809]: cluster 2024-03-11T20:54:06.513535+0000 osd.3 (osd.3) 109 : cluster 0 4.11 deep-scrub starts 2024-03-11T20:54:08.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:08 smithi146 bash[23809]: cluster 2024-03-11T20:54:06.516401+0000 osd.3 (osd.3) 110 : cluster 0 4.11 deep-scrub ok 2024-03-11T20:54:08.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:08 smithi146 bash[23809]: cluster 2024-03-11T20:54:06.807081+0000 osd.2 (osd.2) 55 : cluster 0 6.14 scrub starts 2024-03-11T20:54:08.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:08 smithi146 bash[23809]: cluster 2024-03-11T20:54:06.809725+0000 osd.2 (osd.2) 56 : cluster 0 6.14 scrub ok 2024-03-11T20:54:08.784 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:54:08 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:20:54:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T20:54:09.306 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:09 smithi023 bash[21372]: cluster 2024-03-11T20:54:07.152739+0000 osd.7 (osd.7) 91 : cluster 0 2.1c deep-scrub starts 2024-03-11T20:54:09.306 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:09 smithi023 bash[21372]: cluster 2024-03-11T20:54:07.155116+0000 osd.7 (osd.7) 92 : cluster 0 2.1c deep-scrub ok 2024-03-11T20:54:09.306 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:09 smithi023 bash[21372]: cluster 2024-03-11T20:54:07.825755+0000 osd.1 (osd.1) 85 : cluster 0 3.4 scrub starts 2024-03-11T20:54:09.306 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:09 smithi023 bash[21372]: cluster 2024-03-11T20:54:07.829697+0000 osd.1 (osd.1) 86 : cluster 0 3.4 scrub ok 2024-03-11T20:54:09.306 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:09 smithi023 bash[21372]: cluster 2024-03-11T20:54:07.881871+0000 mgr.y (mgr.24380) 130 : cluster 0 pgmap v97: 132 pgs: 132 active+clean; 582 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:54:09.307 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:09 smithi023 bash[29146]: cluster 2024-03-11T20:54:07.152739+0000 osd.7 (osd.7) 91 : cluster 0 2.1c deep-scrub starts 2024-03-11T20:54:09.307 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:09 smithi023 bash[29146]: cluster 2024-03-11T20:54:07.155116+0000 osd.7 (osd.7) 92 : cluster 0 2.1c deep-scrub ok 2024-03-11T20:54:09.307 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:09 smithi023 bash[29146]: cluster 2024-03-11T20:54:07.825755+0000 osd.1 (osd.1) 85 : cluster 0 3.4 scrub starts 2024-03-11T20:54:09.307 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:09 smithi023 bash[29146]: cluster 2024-03-11T20:54:07.829697+0000 osd.1 (osd.1) 86 : cluster 0 3.4 scrub ok 2024-03-11T20:54:09.307 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:09 smithi023 bash[29146]: cluster 2024-03-11T20:54:07.881871+0000 mgr.y (mgr.24380) 130 : cluster 0 pgmap v97: 132 pgs: 132 active+clean; 582 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:54:09.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:09 smithi146 bash[23809]: cluster 2024-03-11T20:54:07.152739+0000 osd.7 (osd.7) 91 : cluster 0 2.1c deep-scrub starts 2024-03-11T20:54:09.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:09 smithi146 bash[23809]: cluster 2024-03-11T20:54:07.155116+0000 osd.7 (osd.7) 92 : cluster 0 2.1c deep-scrub ok 2024-03-11T20:54:09.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:09 smithi146 bash[23809]: cluster 2024-03-11T20:54:07.825755+0000 osd.1 (osd.1) 85 : cluster 0 3.4 scrub starts 2024-03-11T20:54:09.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:09 smithi146 bash[23809]: cluster 2024-03-11T20:54:07.829697+0000 osd.1 (osd.1) 86 : cluster 0 3.4 scrub ok 2024-03-11T20:54:09.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:09 smithi146 bash[23809]: cluster 2024-03-11T20:54:07.881871+0000 mgr.y (mgr.24380) 130 : cluster 0 pgmap v97: 132 pgs: 132 active+clean; 582 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:54:10.786 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-11T20:54:10.786 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "mon.a", 2024-03-11T20:54:10.786 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster": "ceph", 2024-03-11T20:54:10.786 INFO:teuthology.orchestra.run.smithi023.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-03-11T20:54:10.786 INFO:teuthology.orchestra.run.smithi023.stdout: "admin_socket_mode": "", 2024-03-11T20:54:10.786 INFO:teuthology.orchestra.run.smithi023.stdout: "auth_allow_insecure_global_id_reclaim": "false", 2024-03-11T20:54:10.786 INFO:teuthology.orchestra.run.smithi023.stdout: "auth_client_required": "cephx, none", 2024-03-11T20:54:10.786 INFO:teuthology.orchestra.run.smithi023.stdout: "auth_cluster_required": "cephx", 2024-03-11T20:54:10.786 INFO:teuthology.orchestra.run.smithi023.stdout: "auth_debug": "true", 2024-03-11T20:54:10.786 INFO:teuthology.orchestra.run.smithi023.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-03-11T20:54:10.786 INFO:teuthology.orchestra.run.smithi023.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-03-11T20:54:10.787 INFO:teuthology.orchestra.run.smithi023.stdout: "auth_service_required": "cephx", 2024-03-11T20:54:10.787 INFO:teuthology.orchestra.run.smithi023.stdout: "auth_service_ticket_ttl": "240.000000", 2024-03-11T20:54:10.787 INFO:teuthology.orchestra.run.smithi023.stdout: "auth_supported": "", 2024-03-11T20:54:10.787 INFO:teuthology.orchestra.run.smithi023.stdout: "bdev_aio": "true", 2024-03-11T20:54:10.787 INFO:teuthology.orchestra.run.smithi023.stdout: "bdev_aio_max_queue_depth": "1024", 2024-03-11T20:54:10.787 INFO:teuthology.orchestra.run.smithi023.stdout: "bdev_aio_poll_ms": "250", 2024-03-11T20:54:10.787 INFO:teuthology.orchestra.run.smithi023.stdout: "bdev_aio_reap_max": "16", 2024-03-11T20:54:10.787 INFO:teuthology.orchestra.run.smithi023.stdout: "bdev_async_discard": "false", 2024-03-11T20:54:10.787 INFO:teuthology.orchestra.run.smithi023.stdout: "bdev_block_size": "4096", 2024-03-11T20:54:10.787 INFO:teuthology.orchestra.run.smithi023.stdout: "bdev_debug_aio": "false", 2024-03-11T20:54:10.787 INFO:teuthology.orchestra.run.smithi023.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-03-11T20:54:10.787 INFO:teuthology.orchestra.run.smithi023.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-03-11T20:54:10.787 INFO:teuthology.orchestra.run.smithi023.stdout: "bdev_debug_inflight_ios": "false", 2024-03-11T20:54:10.787 INFO:teuthology.orchestra.run.smithi023.stdout: "bdev_enable_discard": "false", 2024-03-11T20:54:10.788 INFO:teuthology.orchestra.run.smithi023.stdout: "bdev_flock_retry": "3", 2024-03-11T20:54:10.788 INFO:teuthology.orchestra.run.smithi023.stdout: "bdev_flock_retry_interval": "0.100000", 2024-03-11T20:54:10.788 INFO:teuthology.orchestra.run.smithi023.stdout: "bdev_inject_crash": "0", 2024-03-11T20:54:10.788 INFO:teuthology.orchestra.run.smithi023.stdout: "bdev_inject_crash_flush_delay": "2", 2024-03-11T20:54:10.788 INFO:teuthology.orchestra.run.smithi023.stdout: "bdev_ioring": "false", 2024-03-11T20:54:10.788 INFO:teuthology.orchestra.run.smithi023.stdout: "bdev_ioring_hipri": "false", 2024-03-11T20:54:10.788 INFO:teuthology.orchestra.run.smithi023.stdout: "bdev_ioring_sqthread_poll": "false", 2024-03-11T20:54:10.788 INFO:teuthology.orchestra.run.smithi023.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-03-11T20:54:10.788 INFO:teuthology.orchestra.run.smithi023.stdout: "bdev_read_buffer_alignment": "4096", 2024-03-11T20:54:10.788 INFO:teuthology.orchestra.run.smithi023.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-03-11T20:54:10.788 INFO:teuthology.orchestra.run.smithi023.stdout: "bdev_type": "", 2024-03-11T20:54:10.788 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_alloc_size": "1048576", 2024-03-11T20:54:10.788 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_allocator": "hybrid", 2024-03-11T20:54:10.788 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_buffered_io": "true", 2024-03-11T20:54:10.789 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_check_for_zeros": "false", 2024-03-11T20:54:10.789 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_check_volume_selector_often": "false", 2024-03-11T20:54:10.789 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-03-11T20:54:10.789 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_compact_log_sync": "false", 2024-03-11T20:54:10.789 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-03-11T20:54:10.789 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-03-11T20:54:10.789 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_log_compact_min_size": "16777216", 2024-03-11T20:54:10.789 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_log_replay_check_allocations": "true", 2024-03-11T20:54:10.789 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_max_log_runway": "4194304", 2024-03-11T20:54:10.789 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_max_prefetch": "1048576", 2024-03-11T20:54:10.789 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_min_flush_size": "524288", 2024-03-11T20:54:10.789 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_min_log_runway": "1048576", 2024-03-11T20:54:10.789 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_replay_recovery": "false", 2024-03-11T20:54:10.790 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-03-11T20:54:10.790 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_shared_alloc_size": "65536", 2024-03-11T20:54:10.790 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_sync_write": "false", 2024-03-11T20:54:10.790 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-03-11T20:54:10.790 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-03-11T20:54:10.790 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-03-11T20:54:10.790 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_allocation_from_file": "true", 2024-03-11T20:54:10.790 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_allocator": "hybrid", 2024-03-11T20:54:10.790 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-03-11T20:54:10.790 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-03-11T20:54:10.790 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-03-11T20:54:10.790 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-03-11T20:54:10.790 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-03-11T20:54:10.790 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-03-11T20:54:10.791 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_blobid_prealloc": "10240", 2024-03-11T20:54:10.791 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_block_create": "true", 2024-03-11T20:54:10.791 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_block_db_create": "false", 2024-03-11T20:54:10.791 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_block_db_path": "", 2024-03-11T20:54:10.791 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_block_db_size": "0", 2024-03-11T20:54:10.791 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_block_path": "", 2024-03-11T20:54:10.791 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_block_preallocate_file": "false", 2024-03-11T20:54:10.791 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_block_size": "107374182400", 2024-03-11T20:54:10.791 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_block_wal_create": "false", 2024-03-11T20:54:10.791 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_block_wal_path": "", 2024-03-11T20:54:10.791 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_block_wal_size": "100663296", 2024-03-11T20:54:10.791 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_bluefs": "true", 2024-03-11T20:54:10.791 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-03-11T20:54:10.791 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_bluefs_env_mirror": "false", 2024-03-11T20:54:10.791 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-03-11T20:54:10.792 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-03-11T20:54:10.792 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-03-11T20:54:10.792 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-03-11T20:54:10.792 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-03-11T20:54:10.792 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-03-11T20:54:10.792 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_autotune": "true", 2024-03-11T20:54:10.792 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-03-11T20:54:10.792 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-03-11T20:54:10.792 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-03-11T20:54:10.792 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-03-11T20:54:10.792 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_size": "0", 2024-03-11T20:54:10.792 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-03-11T20:54:10.792 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-03-11T20:54:10.793 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-03-11T20:54:10.793 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-03-11T20:54:10.793 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_type": "2q", 2024-03-11T20:54:10.793 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-03-11T20:54:10.793 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_clone_cow": "true", 2024-03-11T20:54:10.793 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_compression_algorithm": "snappy", 2024-03-11T20:54:10.793 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_compression_max_blob_size": "0", 2024-03-11T20:54:10.793 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-03-11T20:54:10.793 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-03-11T20:54:10.793 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_compression_min_blob_size": "0", 2024-03-11T20:54:10.794 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-03-11T20:54:10.794 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-03-11T20:54:10.794 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_compression_mode": "none", 2024-03-11T20:54:10.794 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-03-11T20:54:10.794 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_csum_type": "crc32c", 2024-03-11T20:54:10.794 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_debug_enforce_min_alloc_size": "0", 2024-03-11T20:54:10.794 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_debug_enforce_settings": "default", 2024-03-11T20:54:10.794 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_debug_freelist": "false", 2024-03-11T20:54:10.794 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_debug_fsck_abort": "false", 2024-03-11T20:54:10.794 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-03-11T20:54:10.794 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-03-11T20:54:10.794 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_debug_inject_read_err": "false", 2024-03-11T20:54:10.795 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_debug_legacy_omap": "false", 2024-03-11T20:54:10.795 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-03-11T20:54:10.795 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-03-11T20:54:10.795 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-03-11T20:54:10.795 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-03-11T20:54:10.795 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_debug_prefill": "0.000000", 2024-03-11T20:54:10.795 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-03-11T20:54:10.795 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-03-11T20:54:10.795 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-03-11T20:54:10.795 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_debug_small_allocations": "0", 2024-03-11T20:54:10.796 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-03-11T20:54:10.796 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_default_buffered_read": "true", 2024-03-11T20:54:10.796 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_default_buffered_write": "false", 2024-03-11T20:54:10.796 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_deferred_batch_ops": "0", 2024-03-11T20:54:10.796 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-03-11T20:54:10.796 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-03-11T20:54:10.796 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_elastic_shared_blobs": "true", 2024-03-11T20:54:10.796 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-03-11T20:54:10.796 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-03-11T20:54:10.796 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-03-11T20:54:10.796 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-03-11T20:54:10.796 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-03-11T20:54:10.796 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_fail_eio": "false", 2024-03-11T20:54:10.796 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-03-11T20:54:10.797 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-03-11T20:54:10.797 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-03-11T20:54:10.797 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-03-11T20:54:10.797 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_fsck_on_mkfs": "true", 2024-03-11T20:54:10.797 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-03-11T20:54:10.797 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_fsck_on_mount": "false", 2024-03-11T20:54:10.797 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-03-11T20:54:10.797 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_fsck_on_umount": "false", 2024-03-11T20:54:10.797 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-03-11T20:54:10.797 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-03-11T20:54:10.797 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-03-11T20:54:10.798 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-03-11T20:54:10.798 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-03-11T20:54:10.798 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-03-11T20:54:10.798 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-03-11T20:54:10.798 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-03-11T20:54:10.798 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_ignore_data_csum": "false", 2024-03-11T20:54:10.798 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-03-11T20:54:10.798 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_kvbackend": "rocksdb", 2024-03-11T20:54:10.798 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-03-11T20:54:10.798 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-03-11T20:54:10.798 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_log_op_age": "5.000000", 2024-03-11T20:54:10.798 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_max_alloc_size": "0", 2024-03-11T20:54:10.798 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_max_blob_size": "0", 2024-03-11T20:54:10.798 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-03-11T20:54:10.799 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-03-11T20:54:10.799 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_max_defer_interval": "3.000000", 2024-03-11T20:54:10.799 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_max_deferred_txc": "32", 2024-03-11T20:54:10.799 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_min_alloc_size": "0", 2024-03-11T20:54:10.799 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-03-11T20:54:10.799 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-03-11T20:54:10.799 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_nid_prealloc": "1024", 2024-03-11T20:54:10.799 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_prefer_deferred_size": "0", 2024-03-11T20:54:10.799 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-03-11T20:54:10.799 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-03-11T20:54:10.800 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_qfsck_on_mount": "true", 2024-03-11T20:54:10.800 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_retry_disk_reads": "3", 2024-03-11T20:54:10.800 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_rocksdb_cf": "true", 2024-03-11T20:54:10.800 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-03-11T20:54:10.800 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-03-11T20:54:10.800 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_rocksdb_options_annex": "", 2024-03-11T20:54:10.800 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_spdk_coremask": "0x1", 2024-03-11T20:54:10.800 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_spdk_io_sleep": "5", 2024-03-11T20:54:10.800 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_spdk_max_io_completion": "0", 2024-03-11T20:54:10.800 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_spdk_mem": "512", 2024-03-11T20:54:10.801 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_sync_submit_transaction": "false", 2024-03-11T20:54:10.801 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_throttle_bytes": "67108864", 2024-03-11T20:54:10.801 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_throttle_cost_per_io": "0", 2024-03-11T20:54:10.801 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-03-11T20:54:10.801 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-03-11T20:54:10.801 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-03-11T20:54:10.801 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-03-11T20:54:10.801 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_tracing": "false", 2024-03-11T20:54:10.801 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-03-11T20:54:10.801 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-03-11T20:54:10.801 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_volume_selection_reserved": "0", 2024-03-11T20:54:10.801 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-03-11T20:54:10.802 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-03-11T20:54:10.802 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-03-11T20:54:10.802 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-03-11T20:54:10.802 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-03-11T20:54:10.802 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-03-11T20:54:10.802 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_zero_block_detection": "false", 2024-03-11T20:54:10.802 INFO:teuthology.orchestra.run.smithi023.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-03-11T20:54:10.802 INFO:teuthology.orchestra.run.smithi023.stdout: "cephfs_mirror_action_update_interval": "2", 2024-03-11T20:54:10.802 INFO:teuthology.orchestra.run.smithi023.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-03-11T20:54:10.803 INFO:teuthology.orchestra.run.smithi023.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-03-11T20:54:10.803 INFO:teuthology.orchestra.run.smithi023.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-03-11T20:54:10.803 INFO:teuthology.orchestra.run.smithi023.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-03-11T20:54:10.803 INFO:teuthology.orchestra.run.smithi023.stdout: "cephfs_mirror_mount_timeout": "10", 2024-03-11T20:54:10.803 INFO:teuthology.orchestra.run.smithi023.stdout: "cephfs_mirror_perf_stats_prio": "5", 2024-03-11T20:54:10.803 INFO:teuthology.orchestra.run.smithi023.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-03-11T20:54:10.803 INFO:teuthology.orchestra.run.smithi023.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-03-11T20:54:10.803 INFO:teuthology.orchestra.run.smithi023.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-03-11T20:54:10.803 INFO:teuthology.orchestra.run.smithi023.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-03-11T20:54:10.803 INFO:teuthology.orchestra.run.smithi023.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-03-11T20:54:10.804 INFO:teuthology.orchestra.run.smithi023.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-03-11T20:54:10.804 INFO:teuthology.orchestra.run.smithi023.stdout: "cephx_cluster_require_signatures": "false", 2024-03-11T20:54:10.804 INFO:teuthology.orchestra.run.smithi023.stdout: "cephx_cluster_require_version": "2", 2024-03-11T20:54:10.804 INFO:teuthology.orchestra.run.smithi023.stdout: "cephx_require_signatures": "false", 2024-03-11T20:54:10.804 INFO:teuthology.orchestra.run.smithi023.stdout: "cephx_require_version": "2", 2024-03-11T20:54:10.804 INFO:teuthology.orchestra.run.smithi023.stdout: "cephx_service_require_signatures": "false", 2024-03-11T20:54:10.804 INFO:teuthology.orchestra.run.smithi023.stdout: "cephx_service_require_version": "2", 2024-03-11T20:54:10.804 INFO:teuthology.orchestra.run.smithi023.stdout: "cephx_sign_messages": "true", 2024-03-11T20:54:10.804 INFO:teuthology.orchestra.run.smithi023.stdout: "chdir": "", 2024-03-11T20:54:10.804 INFO:teuthology.orchestra.run.smithi023.stdout: "client_acl_type": "", 2024-03-11T20:54:10.805 INFO:teuthology.orchestra.run.smithi023.stdout: "client_asio_thread_count": "2", 2024-03-11T20:54:10.805 INFO:teuthology.orchestra.run.smithi023.stdout: "client_cache_mid": "0.750000", 2024-03-11T20:54:10.805 INFO:teuthology.orchestra.run.smithi023.stdout: "client_cache_size": "16384", 2024-03-11T20:54:10.805 INFO:teuthology.orchestra.run.smithi023.stdout: "client_caps_release_delay": "5", 2024-03-11T20:54:10.805 INFO:teuthology.orchestra.run.smithi023.stdout: "client_check_pool_perm": "true", 2024-03-11T20:54:10.805 INFO:teuthology.orchestra.run.smithi023.stdout: "client_collect_and_send_global_metrics": "false", 2024-03-11T20:54:10.805 INFO:teuthology.orchestra.run.smithi023.stdout: "client_debug_force_sync_read": "false", 2024-03-11T20:54:10.805 INFO:teuthology.orchestra.run.smithi023.stdout: "client_debug_getattr_caps": "false", 2024-03-11T20:54:10.805 INFO:teuthology.orchestra.run.smithi023.stdout: "client_debug_inject_tick_delay": "0", 2024-03-11T20:54:10.805 INFO:teuthology.orchestra.run.smithi023.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-03-11T20:54:10.805 INFO:teuthology.orchestra.run.smithi023.stdout: "client_die_on_failed_remount": "false", 2024-03-11T20:54:10.805 INFO:teuthology.orchestra.run.smithi023.stdout: "client_dirsize_rbytes": "true", 2024-03-11T20:54:10.805 INFO:teuthology.orchestra.run.smithi023.stdout: "client_force_lazyio": "false", 2024-03-11T20:54:10.806 INFO:teuthology.orchestra.run.smithi023.stdout: "client_fs": "", 2024-03-11T20:54:10.806 INFO:teuthology.orchestra.run.smithi023.stdout: "client_inject_fixed_oldest_tid": "false", 2024-03-11T20:54:10.806 INFO:teuthology.orchestra.run.smithi023.stdout: "client_inject_release_failure": "false", 2024-03-11T20:54:10.806 INFO:teuthology.orchestra.run.smithi023.stdout: "client_max_inline_size": "4096", 2024-03-11T20:54:10.806 INFO:teuthology.orchestra.run.smithi023.stdout: "client_max_retries_on_remount_failure": "5", 2024-03-11T20:54:10.806 INFO:teuthology.orchestra.run.smithi023.stdout: "client_mds_namespace": "", 2024-03-11T20:54:10.806 INFO:teuthology.orchestra.run.smithi023.stdout: "client_metadata": "", 2024-03-11T20:54:10.806 INFO:teuthology.orchestra.run.smithi023.stdout: "client_mount_gid": "-1", 2024-03-11T20:54:10.806 INFO:teuthology.orchestra.run.smithi023.stdout: "client_mount_timeout": "300", 2024-03-11T20:54:10.806 INFO:teuthology.orchestra.run.smithi023.stdout: "client_mount_uid": "-1", 2024-03-11T20:54:10.806 INFO:teuthology.orchestra.run.smithi023.stdout: "client_mountpoint": "/", 2024-03-11T20:54:10.806 INFO:teuthology.orchestra.run.smithi023.stdout: "client_notify_timeout": "10", 2024-03-11T20:54:10.806 INFO:teuthology.orchestra.run.smithi023.stdout: "client_oc": "true", 2024-03-11T20:54:10.807 INFO:teuthology.orchestra.run.smithi023.stdout: "client_oc_max_dirty": "104857600", 2024-03-11T20:54:10.807 INFO:teuthology.orchestra.run.smithi023.stdout: "client_oc_max_dirty_age": "5.000000", 2024-03-11T20:54:10.807 INFO:teuthology.orchestra.run.smithi023.stdout: "client_oc_max_objects": "1000", 2024-03-11T20:54:10.807 INFO:teuthology.orchestra.run.smithi023.stdout: "client_oc_size": "209715200", 2024-03-11T20:54:10.807 INFO:teuthology.orchestra.run.smithi023.stdout: "client_oc_target_dirty": "8388608", 2024-03-11T20:54:10.807 INFO:teuthology.orchestra.run.smithi023.stdout: "client_permissions": "true", 2024-03-11T20:54:10.807 INFO:teuthology.orchestra.run.smithi023.stdout: "client_quota": "true", 2024-03-11T20:54:10.807 INFO:teuthology.orchestra.run.smithi023.stdout: "client_quota_df": "true", 2024-03-11T20:54:10.807 INFO:teuthology.orchestra.run.smithi023.stdout: "client_readahead_max_bytes": "0", 2024-03-11T20:54:10.808 INFO:teuthology.orchestra.run.smithi023.stdout: "client_readahead_max_periods": "4", 2024-03-11T20:54:10.808 INFO:teuthology.orchestra.run.smithi023.stdout: "client_readahead_min": "131072", 2024-03-11T20:54:10.808 INFO:teuthology.orchestra.run.smithi023.stdout: "client_reconnect_stale": "false", 2024-03-11T20:54:10.808 INFO:teuthology.orchestra.run.smithi023.stdout: "client_shutdown_timeout": "30", 2024-03-11T20:54:10.808 INFO:teuthology.orchestra.run.smithi023.stdout: "client_snapdir": ".snap", 2024-03-11T20:54:10.808 INFO:teuthology.orchestra.run.smithi023.stdout: "client_tick_interval": "1", 2024-03-11T20:54:10.808 INFO:teuthology.orchestra.run.smithi023.stdout: "client_trace": "", 2024-03-11T20:54:10.808 INFO:teuthology.orchestra.run.smithi023.stdout: "client_try_dentry_invalidate": "false", 2024-03-11T20:54:10.808 INFO:teuthology.orchestra.run.smithi023.stdout: "client_use_faked_inos": "false", 2024-03-11T20:54:10.808 INFO:teuthology.orchestra.run.smithi023.stdout: "client_use_random_mds": "false", 2024-03-11T20:54:10.808 INFO:teuthology.orchestra.run.smithi023.stdout: "clog_to_graylog": "false", 2024-03-11T20:54:10.808 INFO:teuthology.orchestra.run.smithi023.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-03-11T20:54:10.808 INFO:teuthology.orchestra.run.smithi023.stdout: "clog_to_graylog_port": "12201", 2024-03-11T20:54:10.808 INFO:teuthology.orchestra.run.smithi023.stdout: "clog_to_monitors": "default=true", 2024-03-11T20:54:10.809 INFO:teuthology.orchestra.run.smithi023.stdout: "clog_to_syslog": "false", 2024-03-11T20:54:10.809 INFO:teuthology.orchestra.run.smithi023.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-03-11T20:54:10.809 INFO:teuthology.orchestra.run.smithi023.stdout: "clog_to_syslog_level": "info", 2024-03-11T20:54:10.809 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_addr": "-", 2024-03-11T20:54:10.809 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_network": "", 2024-03-11T20:54:10.809 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_network_interface": "", 2024-03-11T20:54:10.809 INFO:teuthology.orchestra.run.smithi023.stdout: "compressor_zlib_isal": "false", 2024-03-11T20:54:10.809 INFO:teuthology.orchestra.run.smithi023.stdout: "compressor_zlib_level": "5", 2024-03-11T20:54:10.809 INFO:teuthology.orchestra.run.smithi023.stdout: "compressor_zlib_winsize": "-15", 2024-03-11T20:54:10.809 INFO:teuthology.orchestra.run.smithi023.stdout: "compressor_zstd_level": "1", 2024-03-11T20:54:10.809 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379", 2024-03-11T20:54:10.809 INFO:teuthology.orchestra.run.smithi023.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-03-11T20:54:10.809 INFO:teuthology.orchestra.run.smithi023.stdout: "crimson_alien_op_num_threads": "6", 2024-03-11T20:54:10.810 INFO:teuthology.orchestra.run.smithi023.stdout: "crimson_alien_thread_cpu_cores": "", 2024-03-11T20:54:10.810 INFO:teuthology.orchestra.run.smithi023.stdout: "crimson_osd_obc_lru_size": "512", 2024-03-11T20:54:10.810 INFO:teuthology.orchestra.run.smithi023.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-03-11T20:54:10.810 INFO:teuthology.orchestra.run.smithi023.stdout: "crimson_seastar_cpu_cores": "", 2024-03-11T20:54:10.810 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "", 2024-03-11T20:54:10.810 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location_hook": "", 2024-03-11T20:54:10.810 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location_hook_timeout": "10", 2024-03-11T20:54:10.810 INFO:teuthology.orchestra.run.smithi023.stdout: "daemonize": "false", 2024-03-11T20:54:10.810 INFO:teuthology.orchestra.run.smithi023.stdout: "daos_pool": "tank", 2024-03-11T20:54:10.810 INFO:teuthology.orchestra.run.smithi023.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-03-11T20:54:10.810 INFO:teuthology.orchestra.run.smithi023.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-03-11T20:54:10.810 INFO:teuthology.orchestra.run.smithi023.stdout: "dbstore_db_name_prefix": "dbstore", 2024-03-11T20:54:10.810 INFO:teuthology.orchestra.run.smithi023.stdout: "debug_alienstore": "0/5", 2024-03-11T20:54:10.810 INFO:teuthology.orchestra.run.smithi023.stdout: "debug_allow_any_pool_priority": "false", 2024-03-11T20:54:10.811 INFO:teuthology.orchestra.run.smithi023.stdout: "debug_asok": "1/5", 2024-03-11T20:54:10.811 INFO:teuthology.orchestra.run.smithi023.stdout: "debug_asok_assert_abort": "false", 2024-03-11T20:54:10.811 INFO:teuthology.orchestra.run.smithi023.stdout: "debug_asserts_on_shutdown": "true", 2024-03-11T20:54:10.811 INFO:teuthology.orchestra.run.smithi023.stdout: "debug_auth": "1/5", 2024-03-11T20:54:10.811 INFO:teuthology.orchestra.run.smithi023.stdout: "debug_bdev": "1/3", 2024-03-11T20:54:10.811 INFO:teuthology.orchestra.run.smithi023.stdout: "debug_bluefs": "1/5", 2024-03-11T20:54:10.811 INFO:teuthology.orchestra.run.smithi023.stdout: "debug_bluestore": "1/5", 2024-03-11T20:54:10.811 INFO:teuthology.orchestra.run.smithi023.stdout: "debug_buffer": "0/1", 2024-03-11T20:54:10.811 INFO:teuthology.orchestra.run.smithi023.stdout: "debug_ceph_exporter": "1/5", 2024-03-11T20:54:10.811 INFO:teuthology.orchestra.run.smithi023.stdout: "debug_cephfs_mirror": "0/5", 2024-03-11T20:54:10.811 INFO:teuthology.orchestra.run.smithi023.stdout: "debug_cephsqlite": "0/5", 2024-03-11T20:54:10.811 INFO:teuthology.orchestra.run.smithi023.stdout: "debug_client": "0/5", 2024-03-11T20:54:10.811 INFO:teuthology.orchestra.run.smithi023.stdout: "debug_compressor": "1/5", 2024-03-11T20:54:10.811 INFO:teuthology.orchestra.run.smithi023.stdout: "debug_context": "0/1", 2024-03-11T20:54:10.811 INFO:teuthology.orchestra.run.smithi023.stdout: "debug_crimson_interrupt": "0/5", 2024-03-11T20:54:10.812 INFO:teuthology.orchestra.run.smithi023.stdout: "debug_crush": "1/1", 2024-03-11T20:54:10.812 INFO:teuthology.orchestra.run.smithi023.stdout: "debug_crypto": "1/5", 2024-03-11T20:54:10.812 INFO:teuthology.orchestra.run.smithi023.stdout: "debug_cyanstore": "0/5", 2024-03-11T20:54:10.812 INFO:teuthology.orchestra.run.smithi023.stdout: "debug_deliberately_leak_memory": "false", 2024-03-11T20:54:10.812 INFO:teuthology.orchestra.run.smithi023.stdout: "debug_disable_randomized_ping": "false", 2024-03-11T20:54:10.812 INFO:teuthology.orchestra.run.smithi023.stdout: "debug_dpdk": "1/5", 2024-03-11T20:54:10.812 INFO:teuthology.orchestra.run.smithi023.stdout: "debug_eventtrace": "1/5", 2024-03-11T20:54:10.812 INFO:teuthology.orchestra.run.smithi023.stdout: "debug_filer": "0/1", 2024-03-11T20:54:10.812 INFO:teuthology.orchestra.run.smithi023.stdout: "debug_filestore": "1/3", 2024-03-11T20:54:10.812 INFO:teuthology.orchestra.run.smithi023.stdout: "debug_finisher": "1/1", 2024-03-11T20:54:10.812 INFO:teuthology.orchestra.run.smithi023.stdout: "debug_fuse": "1/5", 2024-03-11T20:54:10.812 INFO:teuthology.orchestra.run.smithi023.stdout: "debug_heartbeat_testing_span": "0", 2024-03-11T20:54:10.812 INFO:teuthology.orchestra.run.smithi023.stdout: "debug_heartbeatmap": "1/5", 2024-03-11T20:54:10.812 INFO:teuthology.orchestra.run.smithi023.stdout: "debug_immutable_obj_cache": "0/5", 2024-03-11T20:54:10.813 INFO:teuthology.orchestra.run.smithi023.stdout: "debug_javaclient": "1/5", 2024-03-11T20:54:10.813 INFO:teuthology.orchestra.run.smithi023.stdout: "debug_journal": "1/3", 2024-03-11T20:54:10.813 INFO:teuthology.orchestra.run.smithi023.stdout: "debug_journaler": "0/5", 2024-03-11T20:54:10.813 INFO:teuthology.orchestra.run.smithi023.stdout: "debug_kstore": "1/5", 2024-03-11T20:54:10.813 INFO:teuthology.orchestra.run.smithi023.stdout: "debug_lockdep": "0/1", 2024-03-11T20:54:10.813 INFO:teuthology.orchestra.run.smithi023.stdout: "debug_mclock": "1/5", 2024-03-11T20:54:10.813 INFO:teuthology.orchestra.run.smithi023.stdout: "debug_mds": "1/5", 2024-03-11T20:54:10.813 INFO:teuthology.orchestra.run.smithi023.stdout: "debug_mds_balancer": "1/5", 2024-03-11T20:54:10.813 INFO:teuthology.orchestra.run.smithi023.stdout: "debug_mds_locker": "1/5", 2024-03-11T20:54:10.813 INFO:teuthology.orchestra.run.smithi023.stdout: "debug_mds_log": "1/5", 2024-03-11T20:54:10.813 INFO:teuthology.orchestra.run.smithi023.stdout: "debug_mds_log_expire": "1/5", 2024-03-11T20:54:10.813 INFO:teuthology.orchestra.run.smithi023.stdout: "debug_mds_migrator": "1/5", 2024-03-11T20:54:10.813 INFO:teuthology.orchestra.run.smithi023.stdout: "debug_memstore": "1/5", 2024-03-11T20:54:10.813 INFO:teuthology.orchestra.run.smithi023.stdout: "debug_mgr": "2/5", 2024-03-11T20:54:10.814 INFO:teuthology.orchestra.run.smithi023.stdout: "debug_mgrc": "1/5", 2024-03-11T20:54:10.814 INFO:teuthology.orchestra.run.smithi023.stdout: "debug_mon": "20/20", 2024-03-11T20:54:10.814 INFO:teuthology.orchestra.run.smithi023.stdout: "debug_monc": "0/10", 2024-03-11T20:54:10.814 INFO:teuthology.orchestra.run.smithi023.stdout: "debug_ms": "1/1", 2024-03-11T20:54:10.814 INFO:teuthology.orchestra.run.smithi023.stdout: "debug_none": "0/5", 2024-03-11T20:54:10.814 INFO:teuthology.orchestra.run.smithi023.stdout: "debug_objclass": "0/5", 2024-03-11T20:54:10.814 INFO:teuthology.orchestra.run.smithi023.stdout: "debug_objectcacher": "0/5", 2024-03-11T20:54:10.814 INFO:teuthology.orchestra.run.smithi023.stdout: "debug_objecter": "0/1", 2024-03-11T20:54:10.814 INFO:teuthology.orchestra.run.smithi023.stdout: "debug_optracker": "0/5", 2024-03-11T20:54:10.814 INFO:teuthology.orchestra.run.smithi023.stdout: "debug_osd": "1/5", 2024-03-11T20:54:10.814 INFO:teuthology.orchestra.run.smithi023.stdout: "debug_paxos": "20/20", 2024-03-11T20:54:10.814 INFO:teuthology.orchestra.run.smithi023.stdout: "debug_perfcounter": "1/5", 2024-03-11T20:54:10.814 INFO:teuthology.orchestra.run.smithi023.stdout: "debug_prioritycache": "1/5", 2024-03-11T20:54:10.814 INFO:teuthology.orchestra.run.smithi023.stdout: "debug_rados": "0/5", 2024-03-11T20:54:10.815 INFO:teuthology.orchestra.run.smithi023.stdout: "debug_rbd": "0/5", 2024-03-11T20:54:10.815 INFO:teuthology.orchestra.run.smithi023.stdout: "debug_rbd_mirror": "0/5", 2024-03-11T20:54:10.815 INFO:teuthology.orchestra.run.smithi023.stdout: "debug_rbd_pwl": "0/5", 2024-03-11T20:54:10.815 INFO:teuthology.orchestra.run.smithi023.stdout: "debug_rbd_replay": "0/5", 2024-03-11T20:54:10.815 INFO:teuthology.orchestra.run.smithi023.stdout: "debug_refs": "0/0", 2024-03-11T20:54:10.815 INFO:teuthology.orchestra.run.smithi023.stdout: "debug_reserver": "1/1", 2024-03-11T20:54:10.815 INFO:teuthology.orchestra.run.smithi023.stdout: "debug_rgw": "1/5", 2024-03-11T20:54:10.815 INFO:teuthology.orchestra.run.smithi023.stdout: "debug_rgw_access": "1/5", 2024-03-11T20:54:10.815 INFO:teuthology.orchestra.run.smithi023.stdout: "debug_rgw_datacache": "1/5", 2024-03-11T20:54:10.815 INFO:teuthology.orchestra.run.smithi023.stdout: "debug_rgw_dbstore": "1/5", 2024-03-11T20:54:10.815 INFO:teuthology.orchestra.run.smithi023.stdout: "debug_rgw_flight": "1/5", 2024-03-11T20:54:10.815 INFO:teuthology.orchestra.run.smithi023.stdout: "debug_rgw_lifecycle": "1/5", 2024-03-11T20:54:10.815 INFO:teuthology.orchestra.run.smithi023.stdout: "debug_rgw_sync": "1/5", 2024-03-11T20:54:10.815 INFO:teuthology.orchestra.run.smithi023.stdout: "debug_rocksdb": "4/5", 2024-03-11T20:54:10.816 INFO:teuthology.orchestra.run.smithi023.stdout: "debug_seastore": "0/5", 2024-03-11T20:54:10.816 INFO:teuthology.orchestra.run.smithi023.stdout: "debug_seastore_backref": "0/5", 2024-03-11T20:54:10.816 INFO:teuthology.orchestra.run.smithi023.stdout: "debug_seastore_cache": "0/5", 2024-03-11T20:54:10.816 INFO:teuthology.orchestra.run.smithi023.stdout: "debug_seastore_cleaner": "0/5", 2024-03-11T20:54:10.816 INFO:teuthology.orchestra.run.smithi023.stdout: "debug_seastore_device": "0/5", 2024-03-11T20:54:10.816 INFO:teuthology.orchestra.run.smithi023.stdout: "debug_seastore_epm": "0/5", 2024-03-11T20:54:10.816 INFO:teuthology.orchestra.run.smithi023.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-03-11T20:54:10.816 INFO:teuthology.orchestra.run.smithi023.stdout: "debug_seastore_journal": "0/5", 2024-03-11T20:54:10.816 INFO:teuthology.orchestra.run.smithi023.stdout: "debug_seastore_lba": "0/5", 2024-03-11T20:54:10.816 INFO:teuthology.orchestra.run.smithi023.stdout: "debug_seastore_odata": "0/5", 2024-03-11T20:54:10.816 INFO:teuthology.orchestra.run.smithi023.stdout: "debug_seastore_omap": "0/5", 2024-03-11T20:54:10.816 INFO:teuthology.orchestra.run.smithi023.stdout: "debug_seastore_onode": "0/5", 2024-03-11T20:54:10.816 INFO:teuthology.orchestra.run.smithi023.stdout: "debug_seastore_t": "0/5", 2024-03-11T20:54:10.816 INFO:teuthology.orchestra.run.smithi023.stdout: "debug_seastore_tm": "0/5", 2024-03-11T20:54:10.816 INFO:teuthology.orchestra.run.smithi023.stdout: "debug_striper": "0/1", 2024-03-11T20:54:10.817 INFO:teuthology.orchestra.run.smithi023.stdout: "debug_test": "0/5", 2024-03-11T20:54:10.817 INFO:teuthology.orchestra.run.smithi023.stdout: "debug_throttle": "1/1", 2024-03-11T20:54:10.817 INFO:teuthology.orchestra.run.smithi023.stdout: "debug_timer": "0/1", 2024-03-11T20:54:10.817 INFO:teuthology.orchestra.run.smithi023.stdout: "debug_tp": "0/5", 2024-03-11T20:54:10.817 INFO:teuthology.orchestra.run.smithi023.stdout: "debug_trace": "1/5", 2024-03-11T20:54:10.817 INFO:teuthology.orchestra.run.smithi023.stdout: "defer_client_eviction_on_laggy_osds": "true", 2024-03-11T20:54:10.817 INFO:teuthology.orchestra.run.smithi023.stdout: "device_failure_prediction_mode": "none", 2024-03-11T20:54:10.817 INFO:teuthology.orchestra.run.smithi023.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-03-11T20:54:10.817 INFO:teuthology.orchestra.run.smithi023.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2024-03-11T20:54:10.817 INFO:teuthology.orchestra.run.smithi023.stdout: "err_to_graylog": "false", 2024-03-11T20:54:10.817 INFO:teuthology.orchestra.run.smithi023.stdout: "err_to_journald": "false", 2024-03-11T20:54:10.817 INFO:teuthology.orchestra.run.smithi023.stdout: "err_to_stderr": "true", 2024-03-11T20:54:10.817 INFO:teuthology.orchestra.run.smithi023.stdout: "err_to_syslog": "false", 2024-03-11T20:54:10.817 INFO:teuthology.orchestra.run.smithi023.stdout: "event_tracing": "false", 2024-03-11T20:54:10.817 INFO:teuthology.orchestra.run.smithi023.stdout: "exporter_addr": "0.0.0.0", 2024-03-11T20:54:10.818 INFO:teuthology.orchestra.run.smithi023.stdout: "exporter_http_port": "9926", 2024-03-11T20:54:10.818 INFO:teuthology.orchestra.run.smithi023.stdout: "exporter_prio_limit": "5", 2024-03-11T20:54:10.818 INFO:teuthology.orchestra.run.smithi023.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-03-11T20:54:10.818 INFO:teuthology.orchestra.run.smithi023.stdout: "exporter_sort_metrics": "true", 2024-03-11T20:54:10.818 INFO:teuthology.orchestra.run.smithi023.stdout: "exporter_stats_period": "5", 2024-03-11T20:54:10.818 INFO:teuthology.orchestra.run.smithi023.stdout: "fake_statfs_for_testing": "0", 2024-03-11T20:54:10.818 INFO:teuthology.orchestra.run.smithi023.stdout: "fatal_signal_handlers": "true", 2024-03-11T20:54:10.818 INFO:teuthology.orchestra.run.smithi023.stdout: "filer_max_purge_ops": "10", 2024-03-11T20:54:10.818 INFO:teuthology.orchestra.run.smithi023.stdout: "filer_max_truncate_ops": "128", 2024-03-11T20:54:10.818 INFO:teuthology.orchestra.run.smithi023.stdout: "filestore_apply_finisher_threads": "1", 2024-03-11T20:54:10.818 INFO:teuthology.orchestra.run.smithi023.stdout: "filestore_blackhole": "false", 2024-03-11T20:54:10.818 INFO:teuthology.orchestra.run.smithi023.stdout: "filestore_btrfs_clone_range": "true", 2024-03-11T20:54:10.818 INFO:teuthology.orchestra.run.smithi023.stdout: "filestore_btrfs_snap": "true", 2024-03-11T20:54:10.818 INFO:teuthology.orchestra.run.smithi023.stdout: "filestore_caller_concurrency": "10", 2024-03-11T20:54:10.819 INFO:teuthology.orchestra.run.smithi023.stdout: "filestore_collect_device_partition_information": "true", 2024-03-11T20:54:10.819 INFO:teuthology.orchestra.run.smithi023.stdout: "filestore_commit_timeout": "600.000000", 2024-03-11T20:54:10.819 INFO:teuthology.orchestra.run.smithi023.stdout: "filestore_debug_inject_read_err": "false", 2024-03-11T20:54:10.819 INFO:teuthology.orchestra.run.smithi023.stdout: "filestore_debug_omap_check": "false", 2024-03-11T20:54:10.819 INFO:teuthology.orchestra.run.smithi023.stdout: "filestore_debug_random_read_err": "0.000000", 2024-03-11T20:54:10.819 INFO:teuthology.orchestra.run.smithi023.stdout: "filestore_debug_verify_split": "false", 2024-03-11T20:54:10.819 INFO:teuthology.orchestra.run.smithi023.stdout: "filestore_dump_file": "", 2024-03-11T20:54:10.819 INFO:teuthology.orchestra.run.smithi023.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-03-11T20:54:10.819 INFO:teuthology.orchestra.run.smithi023.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-03-11T20:54:10.819 INFO:teuthology.orchestra.run.smithi023.stdout: "filestore_fadvise": "true", 2024-03-11T20:54:10.819 INFO:teuthology.orchestra.run.smithi023.stdout: "filestore_fail_eio": "true", 2024-03-11T20:54:10.819 INFO:teuthology.orchestra.run.smithi023.stdout: "filestore_fd_cache_shards": "16", 2024-03-11T20:54:10.819 INFO:teuthology.orchestra.run.smithi023.stdout: "filestore_fd_cache_size": "128", 2024-03-11T20:54:10.819 INFO:teuthology.orchestra.run.smithi023.stdout: "filestore_fiemap": "false", 2024-03-11T20:54:10.820 INFO:teuthology.orchestra.run.smithi023.stdout: "filestore_fiemap_threshold": "4096", 2024-03-11T20:54:10.820 INFO:teuthology.orchestra.run.smithi023.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-03-11T20:54:10.820 INFO:teuthology.orchestra.run.smithi023.stdout: "filestore_index_retry_probability": "0.000000", 2024-03-11T20:54:10.820 INFO:teuthology.orchestra.run.smithi023.stdout: "filestore_inject_stall": "0", 2024-03-11T20:54:10.820 INFO:teuthology.orchestra.run.smithi023.stdout: "filestore_journal_parallel": "false", 2024-03-11T20:54:10.820 INFO:teuthology.orchestra.run.smithi023.stdout: "filestore_journal_trailing": "false", 2024-03-11T20:54:10.820 INFO:teuthology.orchestra.run.smithi023.stdout: "filestore_journal_writeahead": "false", 2024-03-11T20:54:10.820 INFO:teuthology.orchestra.run.smithi023.stdout: "filestore_kill_at": "0", 2024-03-11T20:54:10.820 INFO:teuthology.orchestra.run.smithi023.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-03-11T20:54:10.820 INFO:teuthology.orchestra.run.smithi023.stdout: "filestore_max_inline_xattr_size": "0", 2024-03-11T20:54:10.820 INFO:teuthology.orchestra.run.smithi023.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-03-11T20:54:10.820 INFO:teuthology.orchestra.run.smithi023.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-03-11T20:54:10.820 INFO:teuthology.orchestra.run.smithi023.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-03-11T20:54:10.820 INFO:teuthology.orchestra.run.smithi023.stdout: "filestore_max_inline_xattrs": "0", 2024-03-11T20:54:10.821 INFO:teuthology.orchestra.run.smithi023.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-03-11T20:54:10.821 INFO:teuthology.orchestra.run.smithi023.stdout: "filestore_max_inline_xattrs_other": "2", 2024-03-11T20:54:10.821 INFO:teuthology.orchestra.run.smithi023.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-03-11T20:54:10.821 INFO:teuthology.orchestra.run.smithi023.stdout: "filestore_max_sync_interval": "5.000000", 2024-03-11T20:54:10.821 INFO:teuthology.orchestra.run.smithi023.stdout: "filestore_max_xattr_value_size": "0", 2024-03-11T20:54:10.821 INFO:teuthology.orchestra.run.smithi023.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-03-11T20:54:10.821 INFO:teuthology.orchestra.run.smithi023.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-03-11T20:54:10.821 INFO:teuthology.orchestra.run.smithi023.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-03-11T20:54:10.821 INFO:teuthology.orchestra.run.smithi023.stdout: "filestore_merge_threshold": "-10", 2024-03-11T20:54:10.821 INFO:teuthology.orchestra.run.smithi023.stdout: "filestore_min_sync_interval": "0.010000", 2024-03-11T20:54:10.821 INFO:teuthology.orchestra.run.smithi023.stdout: "filestore_odsync_write": "false", 2024-03-11T20:54:10.821 INFO:teuthology.orchestra.run.smithi023.stdout: "filestore_omap_backend": "rocksdb", 2024-03-11T20:54:10.821 INFO:teuthology.orchestra.run.smithi023.stdout: "filestore_omap_backend_path": "", 2024-03-11T20:54:10.821 INFO:teuthology.orchestra.run.smithi023.stdout: "filestore_omap_header_cache_size": "1024", 2024-03-11T20:54:10.822 INFO:teuthology.orchestra.run.smithi023.stdout: "filestore_ondisk_finisher_threads": "1", 2024-03-11T20:54:10.822 INFO:teuthology.orchestra.run.smithi023.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-03-11T20:54:10.822 INFO:teuthology.orchestra.run.smithi023.stdout: "filestore_op_thread_timeout": "60", 2024-03-11T20:54:10.822 INFO:teuthology.orchestra.run.smithi023.stdout: "filestore_op_threads": "2", 2024-03-11T20:54:10.822 INFO:teuthology.orchestra.run.smithi023.stdout: "filestore_punch_hole": "false", 2024-03-11T20:54:10.822 INFO:teuthology.orchestra.run.smithi023.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-03-11T20:54:10.822 INFO:teuthology.orchestra.run.smithi023.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-03-11T20:54:10.822 INFO:teuthology.orchestra.run.smithi023.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-03-11T20:54:10.822 INFO:teuthology.orchestra.run.smithi023.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-03-11T20:54:10.822 INFO:teuthology.orchestra.run.smithi023.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-03-11T20:54:10.822 INFO:teuthology.orchestra.run.smithi023.stdout: "filestore_queue_max_bytes": "104857600", 2024-03-11T20:54:10.822 INFO:teuthology.orchestra.run.smithi023.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-03-11T20:54:10.822 INFO:teuthology.orchestra.run.smithi023.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-03-11T20:54:10.822 INFO:teuthology.orchestra.run.smithi023.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-03-11T20:54:10.822 INFO:teuthology.orchestra.run.smithi023.stdout: "filestore_queue_max_ops": "50", 2024-03-11T20:54:10.823 INFO:teuthology.orchestra.run.smithi023.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-03-11T20:54:10.823 INFO:teuthology.orchestra.run.smithi023.stdout: "filestore_seek_data_hole": "false", 2024-03-11T20:54:10.823 INFO:teuthology.orchestra.run.smithi023.stdout: "filestore_sloppy_crc": "false", 2024-03-11T20:54:10.823 INFO:teuthology.orchestra.run.smithi023.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-03-11T20:54:10.823 INFO:teuthology.orchestra.run.smithi023.stdout: "filestore_splice": "false", 2024-03-11T20:54:10.823 INFO:teuthology.orchestra.run.smithi023.stdout: "filestore_split_multiple": "2", 2024-03-11T20:54:10.823 INFO:teuthology.orchestra.run.smithi023.stdout: "filestore_split_rand_factor": "20", 2024-03-11T20:54:10.823 INFO:teuthology.orchestra.run.smithi023.stdout: "filestore_update_to": "1000", 2024-03-11T20:54:10.823 INFO:teuthology.orchestra.run.smithi023.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-03-11T20:54:10.823 INFO:teuthology.orchestra.run.smithi023.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-03-11T20:54:10.823 INFO:teuthology.orchestra.run.smithi023.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-03-11T20:54:10.823 INFO:teuthology.orchestra.run.smithi023.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-03-11T20:54:10.823 INFO:teuthology.orchestra.run.smithi023.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-03-11T20:54:10.823 INFO:teuthology.orchestra.run.smithi023.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-03-11T20:54:10.824 INFO:teuthology.orchestra.run.smithi023.stdout: "filestore_wbthrottle_enable": "true", 2024-03-11T20:54:10.824 INFO:teuthology.orchestra.run.smithi023.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-03-11T20:54:10.824 INFO:teuthology.orchestra.run.smithi023.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-03-11T20:54:10.824 INFO:teuthology.orchestra.run.smithi023.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-03-11T20:54:10.824 INFO:teuthology.orchestra.run.smithi023.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-03-11T20:54:10.824 INFO:teuthology.orchestra.run.smithi023.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-03-11T20:54:10.824 INFO:teuthology.orchestra.run.smithi023.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-03-11T20:54:10.824 INFO:teuthology.orchestra.run.smithi023.stdout: "filestore_xfs_extsize": "false", 2024-03-11T20:54:10.824 INFO:teuthology.orchestra.run.smithi023.stdout: "filestore_zfs_snap": "false", 2024-03-11T20:54:10.824 INFO:teuthology.orchestra.run.smithi023.stdout: "fio_dir": "/tmp/fio", 2024-03-11T20:54:10.824 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "67ba00f8-dfe7-11ee-95c8-87774f69a715", 2024-03-11T20:54:10.824 INFO:teuthology.orchestra.run.smithi023.stdout: "fuse_allow_other": "true", 2024-03-11T20:54:10.824 INFO:teuthology.orchestra.run.smithi023.stdout: "fuse_atomic_o_trunc": "true", 2024-03-11T20:54:10.824 INFO:teuthology.orchestra.run.smithi023.stdout: "fuse_big_writes": "true", 2024-03-11T20:54:10.825 INFO:teuthology.orchestra.run.smithi023.stdout: "fuse_debug": "false", 2024-03-11T20:54:10.825 INFO:teuthology.orchestra.run.smithi023.stdout: "fuse_default_permissions": "false", 2024-03-11T20:54:10.825 INFO:teuthology.orchestra.run.smithi023.stdout: "fuse_disable_pagecache": "false", 2024-03-11T20:54:10.825 INFO:teuthology.orchestra.run.smithi023.stdout: "fuse_max_write": "0", 2024-03-11T20:54:10.825 INFO:teuthology.orchestra.run.smithi023.stdout: "fuse_multithreaded": "true", 2024-03-11T20:54:10.825 INFO:teuthology.orchestra.run.smithi023.stdout: "fuse_require_active_mds": "true", 2024-03-11T20:54:10.825 INFO:teuthology.orchestra.run.smithi023.stdout: "fuse_set_user_groups": "true", 2024-03-11T20:54:10.825 INFO:teuthology.orchestra.run.smithi023.stdout: "fuse_splice_move": "true", 2024-03-11T20:54:10.825 INFO:teuthology.orchestra.run.smithi023.stdout: "fuse_splice_read": "true", 2024-03-11T20:54:10.825 INFO:teuthology.orchestra.run.smithi023.stdout: "fuse_splice_write": "true", 2024-03-11T20:54:10.825 INFO:teuthology.orchestra.run.smithi023.stdout: "fuse_syncfs_on_mksnap": "true", 2024-03-11T20:54:10.825 INFO:teuthology.orchestra.run.smithi023.stdout: "fuse_use_invalidate_cb": "true", 2024-03-11T20:54:10.825 INFO:teuthology.orchestra.run.smithi023.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-03-11T20:54:10.825 INFO:teuthology.orchestra.run.smithi023.stdout: "gss_target_name": "ceph", 2024-03-11T20:54:10.825 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_file": "", 2024-03-11T20:54:10.826 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_inject_failure": "0", 2024-03-11T20:54:10.826 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_interval": "5", 2024-03-11T20:54:10.826 INFO:teuthology.orchestra.run.smithi023.stdout: "host": "smithi023", 2024-03-11T20:54:10.826 INFO:teuthology.orchestra.run.smithi023.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-03-11T20:54:10.826 INFO:teuthology.orchestra.run.smithi023.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-03-11T20:54:10.826 INFO:teuthology.orchestra.run.smithi023.stdout: "immutable_object_cache_max_size": "1073741824", 2024-03-11T20:54:10.826 INFO:teuthology.orchestra.run.smithi023.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-03-11T20:54:10.826 INFO:teuthology.orchestra.run.smithi023.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-03-11T20:54:10.826 INFO:teuthology.orchestra.run.smithi023.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-03-11T20:54:10.826 INFO:teuthology.orchestra.run.smithi023.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-03-11T20:54:10.827 INFO:teuthology.orchestra.run.smithi023.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-03-11T20:54:10.827 INFO:teuthology.orchestra.run.smithi023.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-03-11T20:54:10.827 INFO:teuthology.orchestra.run.smithi023.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-03-11T20:54:10.827 INFO:teuthology.orchestra.run.smithi023.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-03-11T20:54:10.827 INFO:teuthology.orchestra.run.smithi023.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-03-11T20:54:10.827 INFO:teuthology.orchestra.run.smithi023.stdout: "immutable_object_cache_watermark": "0.900000", 2024-03-11T20:54:10.827 INFO:teuthology.orchestra.run.smithi023.stdout: "inject_early_sigterm": "false", 2024-03-11T20:54:10.827 INFO:teuthology.orchestra.run.smithi023.stdout: "jaeger_agent_port": "6799", 2024-03-11T20:54:10.827 INFO:teuthology.orchestra.run.smithi023.stdout: "jaeger_tracing_enable": "false", 2024-03-11T20:54:10.827 INFO:teuthology.orchestra.run.smithi023.stdout: "journal_aio": "true", 2024-03-11T20:54:10.827 INFO:teuthology.orchestra.run.smithi023.stdout: "journal_align_min_size": "65536", 2024-03-11T20:54:10.827 INFO:teuthology.orchestra.run.smithi023.stdout: "journal_block_align": "true", 2024-03-11T20:54:10.827 INFO:teuthology.orchestra.run.smithi023.stdout: "journal_block_size": "4096", 2024-03-11T20:54:10.828 INFO:teuthology.orchestra.run.smithi023.stdout: "journal_dio": "true", 2024-03-11T20:54:10.828 INFO:teuthology.orchestra.run.smithi023.stdout: "journal_discard": "false", 2024-03-11T20:54:10.828 INFO:teuthology.orchestra.run.smithi023.stdout: "journal_force_aio": "false", 2024-03-11T20:54:10.828 INFO:teuthology.orchestra.run.smithi023.stdout: "journal_ignore_corruption": "false", 2024-03-11T20:54:10.828 INFO:teuthology.orchestra.run.smithi023.stdout: "journal_max_write_bytes": "10485760", 2024-03-11T20:54:10.828 INFO:teuthology.orchestra.run.smithi023.stdout: "journal_max_write_entries": "100", 2024-03-11T20:54:10.828 INFO:teuthology.orchestra.run.smithi023.stdout: "journal_replay_from": "0", 2024-03-11T20:54:10.828 INFO:teuthology.orchestra.run.smithi023.stdout: "journal_throttle_high_multiple": "0.000000", 2024-03-11T20:54:10.828 INFO:teuthology.orchestra.run.smithi023.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-03-11T20:54:10.828 INFO:teuthology.orchestra.run.smithi023.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-03-11T20:54:10.828 INFO:teuthology.orchestra.run.smithi023.stdout: "journal_throttle_max_multiple": "0.000000", 2024-03-11T20:54:10.829 INFO:teuthology.orchestra.run.smithi023.stdout: "journal_write_header_frequency": "0", 2024-03-11T20:54:10.829 INFO:teuthology.orchestra.run.smithi023.stdout: "journal_zero_on_create": "false", 2024-03-11T20:54:10.829 INFO:teuthology.orchestra.run.smithi023.stdout: "journaler_prefetch_periods": "10", 2024-03-11T20:54:10.829 INFO:teuthology.orchestra.run.smithi023.stdout: "journaler_prezero_periods": "5", 2024-03-11T20:54:10.829 INFO:teuthology.orchestra.run.smithi023.stdout: "journaler_write_head_interval": "15", 2024-03-11T20:54:10.829 INFO:teuthology.orchestra.run.smithi023.stdout: "key": "", 2024-03-11T20:54:10.829 INFO:teuthology.orchestra.run.smithi023.stdout: "keyfile": "", 2024-03-11T20:54:10.829 INFO:teuthology.orchestra.run.smithi023.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-03-11T20:54:10.829 INFO:teuthology.orchestra.run.smithi023.stdout: "kstore_backend": "rocksdb", 2024-03-11T20:54:10.829 INFO:teuthology.orchestra.run.smithi023.stdout: "kstore_default_stripe_size": "65536", 2024-03-11T20:54:10.829 INFO:teuthology.orchestra.run.smithi023.stdout: "kstore_fsck_on_mount": "false", 2024-03-11T20:54:10.829 INFO:teuthology.orchestra.run.smithi023.stdout: "kstore_fsck_on_mount_deep": "true", 2024-03-11T20:54:10.829 INFO:teuthology.orchestra.run.smithi023.stdout: "kstore_max_bytes": "67108864", 2024-03-11T20:54:10.830 INFO:teuthology.orchestra.run.smithi023.stdout: "kstore_max_ops": "512", 2024-03-11T20:54:10.830 INFO:teuthology.orchestra.run.smithi023.stdout: "kstore_nid_prealloc": "1024", 2024-03-11T20:54:10.830 INFO:teuthology.orchestra.run.smithi023.stdout: "kstore_onode_map_size": "1024", 2024-03-11T20:54:10.830 INFO:teuthology.orchestra.run.smithi023.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-03-11T20:54:10.830 INFO:teuthology.orchestra.run.smithi023.stdout: "kstore_sync_submit_transaction": "false", 2024-03-11T20:54:10.830 INFO:teuthology.orchestra.run.smithi023.stdout: "kstore_sync_transaction": "false", 2024-03-11T20:54:10.830 INFO:teuthology.orchestra.run.smithi023.stdout: "librados_thread_count": "2", 2024-03-11T20:54:10.830 INFO:teuthology.orchestra.run.smithi023.stdout: "lockdep": "false", 2024-03-11T20:54:10.830 INFO:teuthology.orchestra.run.smithi023.stdout: "lockdep_force_backtrace": "false", 2024-03-11T20:54:10.830 INFO:teuthology.orchestra.run.smithi023.stdout: "log_coarse_timestamps": "true", 2024-03-11T20:54:10.830 INFO:teuthology.orchestra.run.smithi023.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-03-11T20:54:10.830 INFO:teuthology.orchestra.run.smithi023.stdout: "log_flush_on_exit": "false", 2024-03-11T20:54:10.831 INFO:teuthology.orchestra.run.smithi023.stdout: "log_graylog_host": "127.0.0.1", 2024-03-11T20:54:10.831 INFO:teuthology.orchestra.run.smithi023.stdout: "log_graylog_port": "12201", 2024-03-11T20:54:10.831 INFO:teuthology.orchestra.run.smithi023.stdout: "log_max_new": "1000", 2024-03-11T20:54:10.831 INFO:teuthology.orchestra.run.smithi023.stdout: "log_max_recent": "10000", 2024-03-11T20:54:10.831 INFO:teuthology.orchestra.run.smithi023.stdout: "log_stderr_prefix": "debug ", 2024-03-11T20:54:10.831 INFO:teuthology.orchestra.run.smithi023.stdout: "log_stop_at_utilization": "0.970000", 2024-03-11T20:54:10.831 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": "true", 2024-03-11T20:54:10.831 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_graylog": "false", 2024-03-11T20:54:10.831 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_journald": "false", 2024-03-11T20:54:10.831 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_stderr": "false", 2024-03-11T20:54:10.831 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_syslog": "false", 2024-03-11T20:54:10.832 INFO:teuthology.orchestra.run.smithi023.stdout: "max_rotating_auth_attempts": "10", 2024-03-11T20:54:10.832 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-03-11T20:54:10.832 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_action_on_write_error": "1", 2024-03-11T20:54:10.832 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_alternate_name_max": "8192", 2024-03-11T20:54:10.832 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_asio_thread_count": "2", 2024-03-11T20:54:10.832 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_bal_export_pin": "true", 2024-03-11T20:54:10.832 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_bal_fragment_dirs": "true", 2024-03-11T20:54:10.832 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-03-11T20:54:10.832 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_bal_fragment_interval": "5", 2024-03-11T20:54:10.832 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_bal_fragment_size_max": "100000", 2024-03-11T20:54:10.832 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_bal_idle_threshold": "0.000000", 2024-03-11T20:54:10.832 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_bal_interval": "10", 2024-03-11T20:54:10.833 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_bal_max": "-1", 2024-03-11T20:54:10.833 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_bal_max_until": "-1", 2024-03-11T20:54:10.833 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_bal_merge_size": "50", 2024-03-11T20:54:10.833 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_bal_midchunk": "0.300000", 2024-03-11T20:54:10.833 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_bal_min_rebalance": "0.100000", 2024-03-11T20:54:10.833 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_bal_min_start": "0.200000", 2024-03-11T20:54:10.833 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_bal_minchunk": "0.001000", 2024-03-11T20:54:10.833 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_bal_mode": "0", 2024-03-11T20:54:10.833 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_bal_need_max": "1.200000", 2024-03-11T20:54:10.833 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_bal_need_min": "0.800000", 2024-03-11T20:54:10.833 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_bal_overload_epochs": "2", 2024-03-11T20:54:10.833 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-03-11T20:54:10.834 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_bal_sample_interval": "3.000000", 2024-03-11T20:54:10.834 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_bal_split_bits": "3", 2024-03-11T20:54:10.834 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_bal_split_rd": "25000.000000", 2024-03-11T20:54:10.834 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_bal_split_size": "10000", 2024-03-11T20:54:10.834 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_bal_split_wr": "10000.000000", 2024-03-11T20:54:10.834 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_bal_target_decay": "10.000000", 2024-03-11T20:54:10.834 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-03-11T20:54:10.834 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_beacon_grace": "15.000000", 2024-03-11T20:54:10.834 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_beacon_interval": "4.000000", 2024-03-11T20:54:10.834 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_beacon_mon_down_grace": "60", 2024-03-11T20:54:10.835 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_cache_memory_limit": "4294967296", 2024-03-11T20:54:10.835 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_cache_mid": "0.700000", 2024-03-11T20:54:10.835 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_cache_release_free_interval": "10", 2024-03-11T20:54:10.835 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_cache_reservation": "0.050000", 2024-03-11T20:54:10.835 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-03-11T20:54:10.835 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_cache_trim_interval": "1", 2024-03-11T20:54:10.835 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_cache_trim_threshold": "262144", 2024-03-11T20:54:10.835 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-03-11T20:54:10.835 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-03-11T20:54:10.835 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_client_delegate_inos_pct": "50", 2024-03-11T20:54:10.835 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_client_prealloc_inos": "1000", 2024-03-11T20:54:10.835 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-03-11T20:54:10.836 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_connect_bootstrapping": "false", 2024-03-11T20:54:10.836 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_damage_table_max_entries": "10000", 2024-03-11T20:54:10.836 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-03-11T20:54:10.836 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_debug_auth_pins": "false", 2024-03-11T20:54:10.836 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_debug_frag": "false", 2024-03-11T20:54:10.836 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_debug_scatterstat": "false", 2024-03-11T20:54:10.836 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_debug_subtrees": "false", 2024-03-11T20:54:10.836 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_decay_halflife": "5.000000", 2024-03-11T20:54:10.836 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_default_dir_hash": "2", 2024-03-11T20:54:10.836 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_defer_session_stale": "true", 2024-03-11T20:54:10.837 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_deny_all_reconnect": "false", 2024-03-11T20:54:10.837 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_dir_keys_per_op": "16384", 2024-03-11T20:54:10.837 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_dir_max_commit_size": "10", 2024-03-11T20:54:10.837 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_dir_max_entries": "0", 2024-03-11T20:54:10.837 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_dir_prefetch": "true", 2024-03-11T20:54:10.837 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_dirstat_min_interval": "1.000000", 2024-03-11T20:54:10.837 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_dump_cache_after_rejoin": "false", 2024-03-11T20:54:10.837 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_dump_cache_on_map": "false", 2024-03-11T20:54:10.837 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_dump_cache_threshold_file": "0", 2024-03-11T20:54:10.837 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-03-11T20:54:10.837 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_early_reply": "true", 2024-03-11T20:54:10.837 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_enable_op_tracker": "true", 2024-03-11T20:54:10.838 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_enforce_unique_name": "true", 2024-03-11T20:54:10.838 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_export_ephemeral_distributed": "true", 2024-03-11T20:54:10.838 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-03-11T20:54:10.838 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_export_ephemeral_random": "true", 2024-03-11T20:54:10.838 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-03-11T20:54:10.838 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-03-11T20:54:10.838 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_forward_all_requests_to_auth": "false", 2024-03-11T20:54:10.838 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-03-11T20:54:10.838 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-03-11T20:54:10.839 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-03-11T20:54:10.839 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-03-11T20:54:10.839 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_health_cache_threshold": "1.500000", 2024-03-11T20:54:10.839 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_health_summarize_threshold": "10", 2024-03-11T20:54:10.839 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_heartbeat_grace": "15.000000", 2024-03-11T20:54:10.839 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_heartbeat_reset_grace": "1000", 2024-03-11T20:54:10.839 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_inject_health_dummy": "false", 2024-03-11T20:54:10.839 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-03-11T20:54:10.839 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_inject_migrator_session_race": "false", 2024-03-11T20:54:10.839 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-03-11T20:54:10.840 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-03-11T20:54:10.840 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-03-11T20:54:10.840 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_join_fs": "", 2024-03-11T20:54:10.840 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_journal_format": "1", 2024-03-11T20:54:10.840 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_kill_create_at": "0", 2024-03-11T20:54:10.840 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_kill_export_at": "0", 2024-03-11T20:54:10.840 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_kill_import_at": "0", 2024-03-11T20:54:10.840 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_kill_journal_at": "0", 2024-03-11T20:54:10.840 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_kill_journal_expire_at": "0", 2024-03-11T20:54:10.840 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_kill_journal_replay_at": "0", 2024-03-11T20:54:10.841 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_kill_link_at": "0", 2024-03-11T20:54:10.841 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_kill_mdstable_at": "0", 2024-03-11T20:54:10.841 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_kill_openc_at": "0", 2024-03-11T20:54:10.841 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_kill_rename_at": "0", 2024-03-11T20:54:10.841 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_kill_shutdown_at": "0", 2024-03-11T20:54:10.841 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_kill_skip_replaying_inotable": "false", 2024-03-11T20:54:10.841 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_log_event_large_threshold": "524288", 2024-03-11T20:54:10.841 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_log_events_per_segment": "1024", 2024-03-11T20:54:10.841 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_log_major_segment_event_ratio": "12", 2024-03-11T20:54:10.841 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_log_max_events": "-1", 2024-03-11T20:54:10.841 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_log_max_segments": "128", 2024-03-11T20:54:10.841 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_log_pause": "false", 2024-03-11T20:54:10.841 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_log_segment_size": "0", 2024-03-11T20:54:10.842 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_log_skip_corrupt_events": "false", 2024-03-11T20:54:10.842 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_log_skip_unbounded_events": "false", 2024-03-11T20:54:10.842 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_log_trim_decay_rate": "1.000000", 2024-03-11T20:54:10.842 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_log_trim_threshold": "128", 2024-03-11T20:54:10.842 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_log_trim_upkeep_interval": "1000", 2024-03-11T20:54:10.842 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_log_warn_factor": "2.000000", 2024-03-11T20:54:10.842 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_max_caps_per_client": "1048576", 2024-03-11T20:54:10.842 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_max_completed_flushes": "100000", 2024-03-11T20:54:10.842 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_max_completed_requests": "100000", 2024-03-11T20:54:10.842 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_max_export_size": "20971520", 2024-03-11T20:54:10.842 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_max_file_recover": "32", 2024-03-11T20:54:10.842 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_max_purge_files": "64", 2024-03-11T20:54:10.842 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_max_purge_ops": "8192", 2024-03-11T20:54:10.842 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-03-11T20:54:10.843 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-03-11T20:54:10.843 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_max_snaps_per_dir": "100", 2024-03-11T20:54:10.843 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_metrics_update_interval": "2", 2024-03-11T20:54:10.843 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_min_caps_per_client": "100", 2024-03-11T20:54:10.843 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_min_caps_working_set": "10000", 2024-03-11T20:54:10.843 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-03-11T20:54:10.843 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_numa_node": "-1", 2024-03-11T20:54:10.843 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-03-11T20:54:10.843 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_op_complaint_time": "30.000000", 2024-03-11T20:54:10.843 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_op_history_duration": "600", 2024-03-11T20:54:10.843 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_op_history_size": "20", 2024-03-11T20:54:10.843 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_op_history_slow_op_size": "20", 2024-03-11T20:54:10.843 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_op_history_slow_op_threshold": "10", 2024-03-11T20:54:10.843 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_op_log_threshold": "5", 2024-03-11T20:54:10.843 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_ping_grace": "15", 2024-03-11T20:54:10.844 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_ping_interval": "5", 2024-03-11T20:54:10.844 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-03-11T20:54:10.844 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-03-11T20:54:10.844 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_recall_max_caps": "30000", 2024-03-11T20:54:10.844 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-03-11T20:54:10.844 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_recall_max_decay_threshold": "131072", 2024-03-11T20:54:10.844 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-03-11T20:54:10.844 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_recall_warning_threshold": "262144", 2024-03-11T20:54:10.844 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_reconnect_timeout": "45.000000", 2024-03-11T20:54:10.844 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_replay_interval": "1.000000", 2024-03-11T20:54:10.844 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-03-11T20:54:10.844 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-03-11T20:54:10.844 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_root_ino_gid": "0", 2024-03-11T20:54:10.844 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_root_ino_uid": "0", 2024-03-11T20:54:10.844 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-03-11T20:54:10.845 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_session_blocklist_on_evict": "true", 2024-03-11T20:54:10.845 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_session_blocklist_on_timeout": "true", 2024-03-11T20:54:10.845 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-03-11T20:54:10.845 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-03-11T20:54:10.845 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-03-11T20:54:10.845 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-03-11T20:54:10.845 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-03-11T20:54:10.845 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_session_metadata_threshold": "16777216", 2024-03-11T20:54:10.845 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-03-11T20:54:10.845 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_shutdown_check": "0", 2024-03-11T20:54:10.845 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_skip_ino": "0", 2024-03-11T20:54:10.845 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_sleep_rank_change": "0.000000", 2024-03-11T20:54:10.845 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_snap_max_uid": "4294967294", 2024-03-11T20:54:10.845 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_snap_min_uid": "0", 2024-03-11T20:54:10.845 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_snap_rstat": "false", 2024-03-11T20:54:10.845 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_standby_replay_damaged": "false", 2024-03-11T20:54:10.846 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_symlink_recovery": "true", 2024-03-11T20:54:10.846 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_task_status_update_interval": "2.000000", 2024-03-11T20:54:10.846 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_thrash_exports": "0", 2024-03-11T20:54:10.846 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_thrash_fragments": "0", 2024-03-11T20:54:10.846 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_tick_interval": "5.000000", 2024-03-11T20:54:10.846 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_valgrind_exit": "false", 2024-03-11T20:54:10.846 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_verify_backtrace": "1", 2024-03-11T20:54:10.846 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_verify_scatter": "false", 2024-03-11T20:54:10.846 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_wipe_ino_prealloc": "false", 2024-03-11T20:54:10.846 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_wipe_sessions": "false", 2024-03-11T20:54:10.846 INFO:teuthology.orchestra.run.smithi023.stdout: "mempool_debug": "false", 2024-03-11T20:54:10.846 INFO:teuthology.orchestra.run.smithi023.stdout: "memstore_debug_omit_block_device_write": "false", 2024-03-11T20:54:10.846 INFO:teuthology.orchestra.run.smithi023.stdout: "memstore_device_bytes": "1073741824", 2024-03-11T20:54:10.846 INFO:teuthology.orchestra.run.smithi023.stdout: "memstore_page_set": "false", 2024-03-11T20:54:10.847 INFO:teuthology.orchestra.run.smithi023.stdout: "memstore_page_size": "65536", 2024-03-11T20:54:10.847 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr_client_bytes": "134217728", 2024-03-11T20:54:10.847 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr_client_messages": "512", 2024-03-11T20:54:10.847 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-03-11T20:54:10.847 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr_connect_retry_interval": "1.000000", 2024-03-11T20:54:10.847 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-03-11T20:54:10.847 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-03-11T20:54:10.847 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr_disabled_modules": "", 2024-03-11T20:54:10.847 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-03-11T20:54:10.847 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr_max_pg_num_change": "128", 2024-03-11T20:54:10.847 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr_mds_bytes": "134217728", 2024-03-11T20:54:10.847 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr_mds_messages": "128", 2024-03-11T20:54:10.847 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-03-11T20:54:10.848 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr_mon_bytes": "134217728", 2024-03-11T20:54:10.848 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr_mon_messages": "128", 2024-03-11T20:54:10.848 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr_osd_bytes": "536870912", 2024-03-11T20:54:10.848 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr_osd_messages": "8192", 2024-03-11T20:54:10.848 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr_pool": "true", 2024-03-11T20:54:10.848 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr_service_beacon_grace": "60.000000", 2024-03-11T20:54:10.848 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr_standby_modules": "true", 2024-03-11T20:54:10.848 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr_stats_period": "5", 2024-03-11T20:54:10.848 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr_stats_threshold": "5", 2024-03-11T20:54:10.848 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr_tick_period": "2", 2024-03-11T20:54:10.848 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-03-11T20:54:10.848 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_accept_timeout_factor": "2.000000", 2024-03-11T20:54:10.848 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_allow_pool_delete": "true", 2024-03-11T20:54:10.848 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_allow_pool_size_one": "false", 2024-03-11T20:54:10.848 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_auth_validate_all_caps": "true", 2024-03-11T20:54:10.849 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-03-11T20:54:10.849 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-03-11T20:54:10.849 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_client_bytes": "104857600", 2024-03-11T20:54:10.849 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_client_directed_command_retry": "2", 2024-03-11T20:54:10.849 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_client_hunt_interval": "3.000000", 2024-03-11T20:54:10.849 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-03-11T20:54:10.849 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-03-11T20:54:10.849 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-03-11T20:54:10.849 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_client_hunt_parallel": "3", 2024-03-11T20:54:10.849 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_client_log_interval": "1.000000", 2024-03-11T20:54:10.849 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-03-11T20:54:10.849 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_client_ping_interval": "10.000000", 2024-03-11T20:54:10.849 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_client_ping_timeout": "30.000000", 2024-03-11T20:54:10.849 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_client_target_rank": "-1", 2024-03-11T20:54:10.850 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_clock_drift_allowed": "1.000000", 2024-03-11T20:54:10.850 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-03-11T20:54:10.850 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-03-11T20:54:10.850 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_cluster_log_file_level": "debug", 2024-03-11T20:54:10.850 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_cluster_log_to_file": "true", 2024-03-11T20:54:10.850 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_cluster_log_to_graylog": "false", 2024-03-11T20:54:10.850 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-03-11T20:54:10.850 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-03-11T20:54:10.850 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_cluster_log_to_journald": "false", 2024-03-11T20:54:10.850 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_cluster_log_to_stderr": "true", 2024-03-11T20:54:10.850 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-03-11T20:54:10.850 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-03-11T20:54:10.850 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_cluster_log_to_syslog_level": "info", 2024-03-11T20:54:10.850 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_compact_on_bootstrap": "false", 2024-03-11T20:54:10.851 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_compact_on_start": "false", 2024-03-11T20:54:10.851 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_compact_on_trim": "true", 2024-03-11T20:54:10.851 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_con_tracker_persist_interval": "10", 2024-03-11T20:54:10.851 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_con_tracker_score_halflife": "43200", 2024-03-11T20:54:10.851 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_config_key_max_entry_size": "65536", 2024-03-11T20:54:10.851 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_cpu_threads": "4", 2024-03-11T20:54:10.851 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_crush_min_required_version": "hammer", 2024-03-11T20:54:10.851 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_daemon_bytes": "419430400", 2024-03-11T20:54:10.851 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-03-11T20:54:10.851 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_data_avail_crit": "5", 2024-03-11T20:54:10.851 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_data_avail_warn": "5", 2024-03-11T20:54:10.851 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_data_size_warn": "16106127360", 2024-03-11T20:54:10.851 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_debug_block_osdmap_trim": "false", 2024-03-11T20:54:10.851 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-03-11T20:54:10.851 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_debug_dump_json": "false", 2024-03-11T20:54:10.852 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-03-11T20:54:10.852 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_debug_dump_transactions": "false", 2024-03-11T20:54:10.852 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_debug_extra_checks": "false", 2024-03-11T20:54:10.852 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-03-11T20:54:10.852 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-03-11T20:54:10.852 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_debug_no_require_reef": "false", 2024-03-11T20:54:10.852 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_debug_no_require_squid": "false", 2024-03-11T20:54:10.852 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-03-11T20:54:10.852 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_delta_reset_interval": "10.000000", 2024-03-11T20:54:10.852 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_dns_srv_name": "ceph-mon", 2024-03-11T20:54:10.852 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_down_mkfs_grace": "60", 2024-03-11T20:54:10.852 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_election_default_strategy": "1", 2024-03-11T20:54:10.852 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_election_timeout": "5.000000", 2024-03-11T20:54:10.852 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-03-11T20:54:10.853 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_elector_ping_divisor": "2", 2024-03-11T20:54:10.853 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_elector_ping_timeout": "2.000000", 2024-03-11T20:54:10.853 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_enable_op_tracker": "true", 2024-03-11T20:54:10.853 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_fake_pool_delete": "false", 2024-03-11T20:54:10.853 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_force_quorum_join": "false", 2024-03-11T20:54:10.853 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_globalid_prealloc": "10000", 2024-03-11T20:54:10.853 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_health_detail_to_clog": "true", 2024-03-11T20:54:10.853 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_health_log_update_period": "5", 2024-03-11T20:54:10.853 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_health_max_detail": "50", 2024-03-11T20:54:10.853 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_health_to_clog": "true", 2024-03-11T20:54:10.853 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_health_to_clog_interval": "600", 2024-03-11T20:54:10.853 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-03-11T20:54:10.853 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_host": "[v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0]", 2024-03-11T20:54:10.853 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_host_override": "", 2024-03-11T20:54:10.853 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_initial_members": "", 2024-03-11T20:54:10.854 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-03-11T20:54:10.854 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-03-11T20:54:10.854 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-03-11T20:54:10.854 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-03-11T20:54:10.854 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_keyvaluedb": "rocksdb", 2024-03-11T20:54:10.854 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_lease": "5.000000", 2024-03-11T20:54:10.854 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-03-11T20:54:10.854 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-03-11T20:54:10.854 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_log_full_interval": "50", 2024-03-11T20:54:10.854 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_log_max": "10000", 2024-03-11T20:54:10.854 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_log_max_summary": "50", 2024-03-11T20:54:10.854 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_max_log_entries_per_event": "4096", 2024-03-11T20:54:10.854 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_max_log_epochs": "500", 2024-03-11T20:54:10.854 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_max_mdsmap_epochs": "500", 2024-03-11T20:54:10.855 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_max_mgrmap_epochs": "500", 2024-03-11T20:54:10.855 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_max_osd": "10000", 2024-03-11T20:54:10.855 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_max_pg_per_osd": "10000", 2024-03-11T20:54:10.855 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_max_pool_pg_num": "65536", 2024-03-11T20:54:10.855 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-03-11T20:54:10.855 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-03-11T20:54:10.855 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_mds_force_trim_to": "0", 2024-03-11T20:54:10.855 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_mds_skip_sanity": "false", 2024-03-11T20:54:10.855 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_memory_autotune": "true", 2024-03-11T20:54:10.855 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_memory_target": "2147483648", 2024-03-11T20:54:10.855 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_mgr_beacon_grace": "30", 2024-03-11T20:54:10.855 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-03-11T20:54:10.855 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_mgr_digest_period": "5", 2024-03-11T20:54:10.855 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_mgr_inactive_grace": "60", 2024-03-11T20:54:10.855 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_mgr_mkfs_grace": "240", 2024-03-11T20:54:10.856 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-03-11T20:54:10.856 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_min_osdmap_epochs": "50", 2024-03-11T20:54:10.856 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_op_complaint_time": "30", 2024-03-11T20:54:10.856 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_op_history_duration": "600", 2024-03-11T20:54:10.856 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_op_history_size": "20", 2024-03-11T20:54:10.856 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_op_history_slow_op_size": "20", 2024-03-11T20:54:10.856 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_op_history_slow_op_threshold": "10", 2024-03-11T20:54:10.856 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_op_log_threshold": "5", 2024-03-11T20:54:10.856 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-03-11T20:54:10.856 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-03-11T20:54:10.856 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-03-11T20:54:10.856 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_osd_auto_mark_in": "false", 2024-03-11T20:54:10.856 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_osd_auto_mark_new_in": "true", 2024-03-11T20:54:10.856 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-03-11T20:54:10.856 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-03-11T20:54:10.857 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_osd_cache_size": "500", 2024-03-11T20:54:10.857 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_osd_cache_size_min": "134217728", 2024-03-11T20:54:10.857 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_osd_crush_smoke_test": "true", 2024-03-11T20:54:10.857 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_osd_destroyed_out_interval": "600", 2024-03-11T20:54:10.857 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_osd_down_out_interval": "600", 2024-03-11T20:54:10.857 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-03-11T20:54:10.857 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-03-11T20:54:10.857 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_osd_force_trim_to": "0", 2024-03-11T20:54:10.857 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_osd_full_ratio": "0.950000", 2024-03-11T20:54:10.857 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-03-11T20:54:10.857 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_osd_laggy_halflife": "3600", 2024-03-11T20:54:10.857 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_osd_laggy_max_interval": "300", 2024-03-11T20:54:10.857 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_osd_laggy_weight": "0.300000", 2024-03-11T20:54:10.857 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-03-11T20:54:10.857 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_osd_max_creating_pgs": "1024", 2024-03-11T20:54:10.858 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_osd_max_initial_pgs": "1024", 2024-03-11T20:54:10.858 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_osd_min_down_reporters": "2", 2024-03-11T20:54:10.858 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-03-11T20:54:10.858 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-03-11T20:54:10.858 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-03-11T20:54:10.858 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_osd_prime_pg_temp": "true", 2024-03-11T20:54:10.858 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-03-11T20:54:10.858 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-03-11T20:54:10.858 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_osd_report_timeout": "900", 2024-03-11T20:54:10.858 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-03-11T20:54:10.858 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-03-11T20:54:10.858 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_osd_warn_num_repaired": "10", 2024-03-11T20:54:10.858 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_osd_warn_op_age": "32.000000", 2024-03-11T20:54:10.858 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-03-11T20:54:10.858 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_osdmap_full_prune_interval": "2", 2024-03-11T20:54:10.859 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_osdmap_full_prune_min": "15", 2024-03-11T20:54:10.859 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_osdmap_full_prune_txsize": "2", 2024-03-11T20:54:10.859 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-03-11T20:54:10.859 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_pg_stuck_threshold": "60", 2024-03-11T20:54:10.859 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-03-11T20:54:10.859 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_pg_warn_min_objects": "10000", 2024-03-11T20:54:10.859 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_pg_warn_min_per_osd": "0", 2024-03-11T20:54:10.859 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-03-11T20:54:10.859 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_pool_quota_crit_threshold": "0", 2024-03-11T20:54:10.859 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_pool_quota_warn_threshold": "0", 2024-03-11T20:54:10.859 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_probe_timeout": "2.000000", 2024-03-11T20:54:10.859 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_reweight_max_change": "0.050000", 2024-03-11T20:54:10.859 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_reweight_max_osds": "4", 2024-03-11T20:54:10.859 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-03-11T20:54:10.859 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-03-11T20:54:10.860 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-03-11T20:54:10.860 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-03-11T20:54:10.860 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-03-11T20:54:10.860 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_scrub_interval": "86400", 2024-03-11T20:54:10.860 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_scrub_max_keys": "100", 2024-03-11T20:54:10.860 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_scrub_timeout": "300", 2024-03-11T20:54:10.860 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_session_timeout": "300", 2024-03-11T20:54:10.860 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_smart_report_timeout": "5", 2024-03-11T20:54:10.860 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_stat_smooth_intervals": "6", 2024-03-11T20:54:10.860 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-03-11T20:54:10.860 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_stretch_pool_min_size": "2", 2024-03-11T20:54:10.860 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_stretch_pool_size": "4", 2024-03-11T20:54:10.860 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-03-11T20:54:10.860 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_subscribe_interval": "86400.000000", 2024-03-11T20:54:10.860 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_sync_debug": "false", 2024-03-11T20:54:10.860 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_sync_max_payload_keys": "2000", 2024-03-11T20:54:10.861 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_sync_max_payload_size": "1048576", 2024-03-11T20:54:10.861 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_sync_provider_kill_at": "0", 2024-03-11T20:54:10.861 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_sync_requester_kill_at": "0", 2024-03-11T20:54:10.861 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_sync_timeout": "60.000000", 2024-03-11T20:54:10.861 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_target_pg_per_osd": "100", 2024-03-11T20:54:10.861 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_tick_interval": "5", 2024-03-11T20:54:10.861 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_timecheck_interval": "300.000000", 2024-03-11T20:54:10.861 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-03-11T20:54:10.861 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_use_min_delay_socket": "false", 2024-03-11T20:54:10.862 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_warn_older_version_delay": "604800", 2024-03-11T20:54:10.862 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-03-11T20:54:10.862 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-03-11T20:54:10.862 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-03-11T20:54:10.862 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_warn_on_filestore_osds": "true", 2024-03-11T20:54:10.862 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-03-11T20:54:10.862 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-03-11T20:54:10.862 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-03-11T20:54:10.862 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_warn_on_misplaced": "false", 2024-03-11T20:54:10.862 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-03-11T20:54:10.862 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_warn_on_older_version": "true", 2024-03-11T20:54:10.862 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-03-11T20:54:10.862 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_warn_on_pool_no_app": "true", 2024-03-11T20:54:10.863 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_warn_on_pool_no_redundancy": "true", 2024-03-11T20:54:10.863 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-03-11T20:54:10.863 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-03-11T20:54:10.863 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-03-11T20:54:10.863 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_warn_on_too_few_osds": "false", 2024-03-11T20:54:10.863 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-03-11T20:54:10.863 INFO:teuthology.orchestra.run.smithi023.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-03-11T20:54:10.863 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": "", 2024-03-11T20:54:10.863 INFO:teuthology.orchestra.run.smithi023.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-03-11T20:54:10.863 INFO:teuthology.orchestra.run.smithi023.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-03-11T20:54:10.863 INFO:teuthology.orchestra.run.smithi023.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-03-11T20:54:10.863 INFO:teuthology.orchestra.run.smithi023.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-03-11T20:54:10.864 INFO:teuthology.orchestra.run.smithi023.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-03-11T20:54:10.865 INFO:teuthology.orchestra.run.smithi023.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-03-11T20:54:10.865 INFO:teuthology.orchestra.run.smithi023.stdout: "motr_tracing_enabled": "false", 2024-03-11T20:54:10.865 INFO:teuthology.orchestra.run.smithi023.stdout: "ms_async_op_threads": "3", 2024-03-11T20:54:10.866 INFO:teuthology.orchestra.run.smithi023.stdout: "ms_async_rdma_buffer_size": "131072", 2024-03-11T20:54:10.866 INFO:teuthology.orchestra.run.smithi023.stdout: "ms_async_rdma_cm": "false", 2024-03-11T20:54:10.866 INFO:teuthology.orchestra.run.smithi023.stdout: "ms_async_rdma_device_name": "", 2024-03-11T20:54:10.866 INFO:teuthology.orchestra.run.smithi023.stdout: "ms_async_rdma_dscp": "96", 2024-03-11T20:54:10.866 INFO:teuthology.orchestra.run.smithi023.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-03-11T20:54:10.866 INFO:teuthology.orchestra.run.smithi023.stdout: "ms_async_rdma_gid_idx": "0", 2024-03-11T20:54:10.866 INFO:teuthology.orchestra.run.smithi023.stdout: "ms_async_rdma_local_gid": "", 2024-03-11T20:54:10.866 INFO:teuthology.orchestra.run.smithi023.stdout: "ms_async_rdma_polling_us": "1000", 2024-03-11T20:54:10.866 INFO:teuthology.orchestra.run.smithi023.stdout: "ms_async_rdma_port_num": "1", 2024-03-11T20:54:10.866 INFO:teuthology.orchestra.run.smithi023.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-03-11T20:54:10.866 INFO:teuthology.orchestra.run.smithi023.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-03-11T20:54:10.866 INFO:teuthology.orchestra.run.smithi023.stdout: "ms_async_rdma_roce_ver": "1", 2024-03-11T20:54:10.866 INFO:teuthology.orchestra.run.smithi023.stdout: "ms_async_rdma_send_buffers": "1024", 2024-03-11T20:54:10.866 INFO:teuthology.orchestra.run.smithi023.stdout: "ms_async_rdma_sl": "3", 2024-03-11T20:54:10.866 INFO:teuthology.orchestra.run.smithi023.stdout: "ms_async_rdma_support_srq": "true", 2024-03-11T20:54:10.867 INFO:teuthology.orchestra.run.smithi023.stdout: "ms_async_rdma_type": "ib", 2024-03-11T20:54:10.867 INFO:teuthology.orchestra.run.smithi023.stdout: "ms_async_reap_threshold": "5", 2024-03-11T20:54:10.867 INFO:teuthology.orchestra.run.smithi023.stdout: "ms_bind_before_connect": "false", 2024-03-11T20:54:10.867 INFO:teuthology.orchestra.run.smithi023.stdout: "ms_bind_ipv4": "true", 2024-03-11T20:54:10.867 INFO:teuthology.orchestra.run.smithi023.stdout: "ms_bind_ipv6": "false", 2024-03-11T20:54:10.867 INFO:teuthology.orchestra.run.smithi023.stdout: "ms_bind_msgr1": "true", 2024-03-11T20:54:10.867 INFO:teuthology.orchestra.run.smithi023.stdout: "ms_bind_msgr2": "true", 2024-03-11T20:54:10.867 INFO:teuthology.orchestra.run.smithi023.stdout: "ms_bind_port_max": "7568", 2024-03-11T20:54:10.867 INFO:teuthology.orchestra.run.smithi023.stdout: "ms_bind_port_min": "6800", 2024-03-11T20:54:10.868 INFO:teuthology.orchestra.run.smithi023.stdout: "ms_bind_prefer_ipv4": "false", 2024-03-11T20:54:10.868 INFO:teuthology.orchestra.run.smithi023.stdout: "ms_bind_retry_count": "3", 2024-03-11T20:54:10.868 INFO:teuthology.orchestra.run.smithi023.stdout: "ms_bind_retry_delay": "5", 2024-03-11T20:54:10.868 INFO:teuthology.orchestra.run.smithi023.stdout: "ms_blackhole_client": "false", 2024-03-11T20:54:10.868 INFO:teuthology.orchestra.run.smithi023.stdout: "ms_blackhole_mds": "false", 2024-03-11T20:54:10.868 INFO:teuthology.orchestra.run.smithi023.stdout: "ms_blackhole_mgr": "false", 2024-03-11T20:54:10.868 INFO:teuthology.orchestra.run.smithi023.stdout: "ms_blackhole_mon": "false", 2024-03-11T20:54:10.868 INFO:teuthology.orchestra.run.smithi023.stdout: "ms_blackhole_osd": "false", 2024-03-11T20:54:10.868 INFO:teuthology.orchestra.run.smithi023.stdout: "ms_client_mode": "crc secure", 2024-03-11T20:54:10.868 INFO:teuthology.orchestra.run.smithi023.stdout: "ms_cluster_mode": "crc secure", 2024-03-11T20:54:10.868 INFO:teuthology.orchestra.run.smithi023.stdout: "ms_cluster_type": "", 2024-03-11T20:54:10.868 INFO:teuthology.orchestra.run.smithi023.stdout: "ms_compress_secure": "false", 2024-03-11T20:54:10.868 INFO:teuthology.orchestra.run.smithi023.stdout: "ms_connection_idle_timeout": "900", 2024-03-11T20:54:10.869 INFO:teuthology.orchestra.run.smithi023.stdout: "ms_connection_ready_timeout": "10", 2024-03-11T20:54:10.869 INFO:teuthology.orchestra.run.smithi023.stdout: "ms_crc_data": "true", 2024-03-11T20:54:10.869 INFO:teuthology.orchestra.run.smithi023.stdout: "ms_crc_header": "true", 2024-03-11T20:54:10.869 INFO:teuthology.orchestra.run.smithi023.stdout: "ms_die_on_bad_msg": "false", 2024-03-11T20:54:10.869 INFO:teuthology.orchestra.run.smithi023.stdout: "ms_die_on_bug": "true", 2024-03-11T20:54:10.869 INFO:teuthology.orchestra.run.smithi023.stdout: "ms_die_on_old_message": "true", 2024-03-11T20:54:10.869 INFO:teuthology.orchestra.run.smithi023.stdout: "ms_die_on_skipped_message": "false", 2024-03-11T20:54:10.869 INFO:teuthology.orchestra.run.smithi023.stdout: "ms_die_on_unhandled_msg": "false", 2024-03-11T20:54:10.869 INFO:teuthology.orchestra.run.smithi023.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-03-11T20:54:10.869 INFO:teuthology.orchestra.run.smithi023.stdout: "ms_dpdk_coremask": "0xF", 2024-03-11T20:54:10.870 INFO:teuthology.orchestra.run.smithi023.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-03-11T20:54:10.870 INFO:teuthology.orchestra.run.smithi023.stdout: "ms_dpdk_devs_allowlist": "", 2024-03-11T20:54:10.870 INFO:teuthology.orchestra.run.smithi023.stdout: "ms_dpdk_enable_tso": "true", 2024-03-11T20:54:10.870 INFO:teuthology.orchestra.run.smithi023.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-03-11T20:54:10.870 INFO:teuthology.orchestra.run.smithi023.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-03-11T20:54:10.870 INFO:teuthology.orchestra.run.smithi023.stdout: "ms_dpdk_hugepages": "", 2024-03-11T20:54:10.870 INFO:teuthology.orchestra.run.smithi023.stdout: "ms_dpdk_hw_flow_control": "true", 2024-03-11T20:54:10.870 INFO:teuthology.orchestra.run.smithi023.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-03-11T20:54:10.870 INFO:teuthology.orchestra.run.smithi023.stdout: "ms_dpdk_lro": "true", 2024-03-11T20:54:10.870 INFO:teuthology.orchestra.run.smithi023.stdout: "ms_dpdk_memory_channel": "4", 2024-03-11T20:54:10.870 INFO:teuthology.orchestra.run.smithi023.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-03-11T20:54:10.870 INFO:teuthology.orchestra.run.smithi023.stdout: "ms_dpdk_pmd": "", 2024-03-11T20:54:10.871 INFO:teuthology.orchestra.run.smithi023.stdout: "ms_dpdk_port_id": "0", 2024-03-11T20:54:10.871 INFO:teuthology.orchestra.run.smithi023.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-03-11T20:54:10.871 INFO:teuthology.orchestra.run.smithi023.stdout: "ms_dump_corrupt_message_level": "1", 2024-03-11T20:54:10.871 INFO:teuthology.orchestra.run.smithi023.stdout: "ms_dump_on_send": "false", 2024-03-11T20:54:10.871 INFO:teuthology.orchestra.run.smithi023.stdout: "ms_initial_backoff": "0.200000", 2024-03-11T20:54:10.871 INFO:teuthology.orchestra.run.smithi023.stdout: "ms_inject_delay_max": "1.000000", 2024-03-11T20:54:10.871 INFO:teuthology.orchestra.run.smithi023.stdout: "ms_inject_delay_probability": "0.000000", 2024-03-11T20:54:10.871 INFO:teuthology.orchestra.run.smithi023.stdout: "ms_inject_delay_type": "", 2024-03-11T20:54:10.871 INFO:teuthology.orchestra.run.smithi023.stdout: "ms_inject_internal_delays": "0.000000", 2024-03-11T20:54:10.871 INFO:teuthology.orchestra.run.smithi023.stdout: "ms_inject_network_congestion": "0", 2024-03-11T20:54:10.872 INFO:teuthology.orchestra.run.smithi023.stdout: "ms_inject_socket_failures": "0", 2024-03-11T20:54:10.872 INFO:teuthology.orchestra.run.smithi023.stdout: "ms_learn_addr_from_peer": "true", 2024-03-11T20:54:10.872 INFO:teuthology.orchestra.run.smithi023.stdout: "ms_max_accept_failures": "4", 2024-03-11T20:54:10.872 INFO:teuthology.orchestra.run.smithi023.stdout: "ms_max_backoff": "15.000000", 2024-03-11T20:54:10.872 INFO:teuthology.orchestra.run.smithi023.stdout: "ms_mon_client_mode": "secure crc", 2024-03-11T20:54:10.872 INFO:teuthology.orchestra.run.smithi023.stdout: "ms_mon_cluster_mode": "secure crc", 2024-03-11T20:54:10.872 INFO:teuthology.orchestra.run.smithi023.stdout: "ms_mon_service_mode": "secure crc", 2024-03-11T20:54:10.872 INFO:teuthology.orchestra.run.smithi023.stdout: "ms_osd_compress_min_size": "1024", 2024-03-11T20:54:10.872 INFO:teuthology.orchestra.run.smithi023.stdout: "ms_osd_compress_mode": "none", 2024-03-11T20:54:10.873 INFO:teuthology.orchestra.run.smithi023.stdout: "ms_osd_compression_algorithm": "snappy", 2024-03-11T20:54:10.873 INFO:teuthology.orchestra.run.smithi023.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-03-11T20:54:10.873 INFO:teuthology.orchestra.run.smithi023.stdout: "ms_pq_min_cost": "65536", 2024-03-11T20:54:10.873 INFO:teuthology.orchestra.run.smithi023.stdout: "ms_public_type": "", 2024-03-11T20:54:10.873 INFO:teuthology.orchestra.run.smithi023.stdout: "ms_service_mode": "crc secure", 2024-03-11T20:54:10.873 INFO:teuthology.orchestra.run.smithi023.stdout: "ms_tcp_listen_backlog": "512", 2024-03-11T20:54:10.873 INFO:teuthology.orchestra.run.smithi023.stdout: "ms_tcp_nodelay": "true", 2024-03-11T20:54:10.873 INFO:teuthology.orchestra.run.smithi023.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-03-11T20:54:10.873 INFO:teuthology.orchestra.run.smithi023.stdout: "ms_tcp_rcvbuf": "0", 2024-03-11T20:54:10.873 INFO:teuthology.orchestra.run.smithi023.stdout: "ms_type": "async+posix", 2024-03-11T20:54:10.873 INFO:teuthology.orchestra.run.smithi023.stdout: "no_config_file": "false", 2024-03-11T20:54:10.874 INFO:teuthology.orchestra.run.smithi023.stdout: "objecter_completion_locks_per_session": "32", 2024-03-11T20:54:10.874 INFO:teuthology.orchestra.run.smithi023.stdout: "objecter_debug_inject_relock_delay": "false", 2024-03-11T20:54:10.874 INFO:teuthology.orchestra.run.smithi023.stdout: "objecter_inflight_op_bytes": "104857600", 2024-03-11T20:54:10.874 INFO:teuthology.orchestra.run.smithi023.stdout: "objecter_inflight_ops": "1024", 2024-03-11T20:54:10.874 INFO:teuthology.orchestra.run.smithi023.stdout: "objecter_inject_no_watch_ping": "false", 2024-03-11T20:54:10.874 INFO:teuthology.orchestra.run.smithi023.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-03-11T20:54:10.874 INFO:teuthology.orchestra.run.smithi023.stdout: "objecter_tick_interval": "5.000000", 2024-03-11T20:54:10.874 INFO:teuthology.orchestra.run.smithi023.stdout: "objecter_timeout": "10.000000", 2024-03-11T20:54:10.874 INFO:teuthology.orchestra.run.smithi023.stdout: "objectstore_blackhole": "false", 2024-03-11T20:54:10.874 INFO:teuthology.orchestra.run.smithi023.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2024-03-11T20:54:10.875 INFO:teuthology.orchestra.run.smithi023.stdout: "openssl_engine_opts": "", 2024-03-11T20:54:10.875 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_agent_delay_time": "5.000000", 2024-03-11T20:54:10.875 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_agent_hist_halflife": "1000", 2024-03-11T20:54:10.875 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_agent_max_low_ops": "2", 2024-03-11T20:54:10.875 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_agent_max_ops": "4", 2024-03-11T20:54:10.875 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-03-11T20:54:10.875 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_agent_quantize_effort": "0.100000", 2024-03-11T20:54:10.875 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_agent_slop": "0.020000", 2024-03-11T20:54:10.875 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-03-11T20:54:10.875 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_allow_recovery_below_min_size": "true", 2024-03-11T20:54:10.876 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_asio_thread_count": "2", 2024-03-11T20:54:10.876 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_async_recovery_min_cost": "100", 2024-03-11T20:54:10.876 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_auto_mark_unfound_lost": "false", 2024-03-11T20:54:10.876 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_backfill_retry_interval": "30.000000", 2024-03-11T20:54:10.876 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_backfill_scan_max": "512", 2024-03-11T20:54:10.876 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_backfill_scan_min": "64", 2024-03-11T20:54:10.876 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_backoff_on_degraded": "false", 2024-03-11T20:54:10.876 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_backoff_on_peering": "false", 2024-03-11T20:54:10.876 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_backoff_on_unfound": "true", 2024-03-11T20:54:10.876 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_beacon_report_interval": "300", 2024-03-11T20:54:10.876 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_bench_duration": "30", 2024-03-11T20:54:10.876 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-03-11T20:54:10.876 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_bench_max_block_size": "67108864", 2024-03-11T20:54:10.876 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_bench_small_size_max_iops": "100", 2024-03-11T20:54:10.877 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_blkin_trace_all": "false", 2024-03-11T20:54:10.877 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_blocked_scrub_grace_period": "120", 2024-03-11T20:54:10.877 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-03-11T20:54:10.877 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-03-11T20:54:10.877 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-03-11T20:54:10.877 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_check_for_log_corruption": "false", 2024-03-11T20:54:10.877 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-03-11T20:54:10.877 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-03-11T20:54:10.877 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2024-03-11T20:54:10.877 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-03-11T20:54:10.877 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_class_update_on_start": "true", 2024-03-11T20:54:10.877 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_client_message_cap": "256", 2024-03-11T20:54:10.877 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_client_message_size_cap": "524288000", 2024-03-11T20:54:10.877 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_client_op_priority": "63", 2024-03-11T20:54:10.878 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_client_watch_timeout": "30", 2024-03-11T20:54:10.878 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_command_max_records": "256", 2024-03-11T20:54:10.878 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_compact_on_start": "false", 2024-03-11T20:54:10.878 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-03-11T20:54:10.878 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_crush_chooseleaf_type": "1", 2024-03-11T20:54:10.878 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_crush_initial_weight": "-1.000000", 2024-03-11T20:54:10.878 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_crush_update_on_start": "true", 2024-03-11T20:54:10.878 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_crush_update_weight_set": "true", 2024-03-11T20:54:10.878 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-03-11T20:54:10.878 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-03-11T20:54:10.878 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-03-11T20:54:10.878 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_debug_drop_ping_duration": "0", 2024-03-11T20:54:10.878 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-03-11T20:54:10.878 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_debug_feed_pullee": "-1", 2024-03-11T20:54:10.878 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-03-11T20:54:10.879 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-03-11T20:54:10.879 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-03-11T20:54:10.879 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_debug_misdirected_ops": "false", 2024-03-11T20:54:10.879 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_debug_no_acting_change": "false", 2024-03-11T20:54:10.879 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_debug_no_purge_strays": "false", 2024-03-11T20:54:10.879 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_debug_op_order": "false", 2024-03-11T20:54:10.879 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_debug_pg_log_writeout": "false", 2024-03-11T20:54:10.879 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_debug_pretend_recovery_active": "false", 2024-03-11T20:54:10.879 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-03-11T20:54:10.879 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-03-11T20:54:10.879 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_debug_shutdown": "false", 2024-03-11T20:54:10.879 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-03-11T20:54:10.879 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-03-11T20:54:10.879 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_debug_trim_objects": "false", 2024-03-11T20:54:10.880 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_debug_verify_cached_snaps": "false", 2024-03-11T20:54:10.880 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_debug_verify_missing_on_start": "false", 2024-03-11T20:54:10.880 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_debug_verify_snaps": "false", 2024-03-11T20:54:10.880 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-03-11T20:54:10.880 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-03-11T20:54:10.880 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_deep_scrub_keys": "1024", 2024-03-11T20:54:10.880 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-03-11T20:54:10.880 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-03-11T20:54:10.880 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-03-11T20:54:10.880 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_deep_scrub_stride": "524288", 2024-03-11T20:54:10.880 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-03-11T20:54:10.880 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_default_data_pool_replay_window": "45", 2024-03-11T20:54:10.880 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_default_notify_timeout": "30", 2024-03-11T20:54:10.880 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_delete_sleep": "0.000000", 2024-03-11T20:54:10.880 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-03-11T20:54:10.881 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-03-11T20:54:10.881 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-03-11T20:54:10.881 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_discard_disconnected_ops": "true", 2024-03-11T20:54:10.881 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_enable_op_tracker": "true", 2024-03-11T20:54:10.881 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-03-11T20:54:10.881 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_extblkdev_plugins": "vdo", 2024-03-11T20:54:10.881 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-03-11T20:54:10.881 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-03-11T20:54:10.881 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_fast_info": "true", 2024-03-11T20:54:10.881 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_fast_shutdown": "true", 2024-03-11T20:54:10.881 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-03-11T20:54:10.881 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_fast_shutdown_timeout": "15", 2024-03-11T20:54:10.881 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-03-11T20:54:10.881 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-03-11T20:54:10.882 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-03-11T20:54:10.882 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_function_tracing": "false", 2024-03-11T20:54:10.882 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_heartbeat_grace": "20", 2024-03-11T20:54:10.882 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_heartbeat_interval": "6", 2024-03-11T20:54:10.882 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-03-11T20:54:10.882 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_heartbeat_min_peers": "10", 2024-03-11T20:54:10.882 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_heartbeat_min_size": "2000", 2024-03-11T20:54:10.882 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_heartbeat_stale": "600", 2024-03-11T20:54:10.882 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-03-11T20:54:10.882 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_hit_set_max_size": "100000", 2024-03-11T20:54:10.882 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_hit_set_min_size": "1000", 2024-03-11T20:54:10.882 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-03-11T20:54:10.882 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-03-11T20:54:10.882 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-03-11T20:54:10.883 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-03-11T20:54:10.883 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-03-11T20:54:10.883 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_journal_flush_on_shutdown": "true", 2024-03-11T20:54:10.883 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_journal_size": "5120", 2024-03-11T20:54:10.883 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_kill_backfill_at": "0", 2024-03-11T20:54:10.883 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_loop_before_reset_tphandle": "64", 2024-03-11T20:54:10.883 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_map_cache_size": "50", 2024-03-11T20:54:10.883 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_map_dedup": "true", 2024-03-11T20:54:10.883 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_map_message_max": "40", 2024-03-11T20:54:10.883 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_map_message_max_bytes": "10485760", 2024-03-11T20:54:10.883 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_map_share_max_epochs": "40", 2024-03-11T20:54:10.883 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_max_attr_name_len": "100", 2024-03-11T20:54:10.883 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_max_attr_size": "0", 2024-03-11T20:54:10.883 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_max_backfills": "1", 2024-03-11T20:54:10.883 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_max_markdown_count": "5", 2024-03-11T20:54:10.884 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_max_markdown_period": "600", 2024-03-11T20:54:10.884 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_max_object_name_len": "2048", 2024-03-11T20:54:10.884 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_max_object_namespace_len": "256", 2024-03-11T20:54:10.884 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_max_object_size": "134217728", 2024-03-11T20:54:10.884 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-03-11T20:54:10.884 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_max_omap_entries_per_request": "1024", 2024-03-11T20:54:10.884 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_max_pg_blocked_by": "16", 2024-03-11T20:54:10.884 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_max_pg_log_entries": "10000", 2024-03-11T20:54:10.884 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-03-11T20:54:10.884 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_max_pgls": "1024", 2024-03-11T20:54:10.884 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_max_push_cost": "8388608", 2024-03-11T20:54:10.884 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_max_push_objects": "10", 2024-03-11T20:54:10.884 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_max_scrubs": "3", 2024-03-11T20:54:10.884 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-03-11T20:54:10.884 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_max_trimming_pgs": "2", 2024-03-11T20:54:10.885 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_max_write_op_reply_len": "64", 2024-03-11T20:54:10.885 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_max_write_size": "90", 2024-03-11T20:54:10.885 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-03-11T20:54:10.885 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-03-11T20:54:10.885 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-03-11T20:54:10.885 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-03-11T20:54:10.885 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-03-11T20:54:10.885 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-03-11T20:54:10.885 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-03-11T20:54:10.885 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_mclock_override_recovery_settings": "false", 2024-03-11T20:54:10.885 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_mclock_profile": "balanced", 2024-03-11T20:54:10.885 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-03-11T20:54:10.885 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-03-11T20:54:10.885 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-03-11T20:54:10.885 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-03-11T20:54:10.886 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-03-11T20:54:10.886 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-03-11T20:54:10.886 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-03-11T20:54:10.886 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-03-11T20:54:10.886 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-03-11T20:54:10.886 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-03-11T20:54:10.886 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_mclock_skip_benchmark": "false", 2024-03-11T20:54:10.886 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_memory_base": "805306368", 2024-03-11T20:54:10.886 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_memory_cache_min": "134217728", 2024-03-11T20:54:10.886 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-03-11T20:54:10.886 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-03-11T20:54:10.886 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_memory_target": "4294967296", 2024-03-11T20:54:10.886 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_memory_target_autotune": "false", 2024-03-11T20:54:10.886 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-03-11T20:54:10.887 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_min_pg_log_entries": "250", 2024-03-11T20:54:10.887 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_min_recovery_priority": "0", 2024-03-11T20:54:10.887 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_mon_heartbeat_interval": "30", 2024-03-11T20:54:10.887 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-03-11T20:54:10.887 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_mon_report_interval": "5", 2024-03-11T20:54:10.887 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_mon_report_max_in_flight": "2", 2024-03-11T20:54:10.887 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-03-11T20:54:10.887 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_num_cache_shards": "32", 2024-03-11T20:54:10.887 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_num_op_tracker_shard": "32", 2024-03-11T20:54:10.887 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_numa_auto_affinity": "true", 2024-03-11T20:54:10.887 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_numa_node": "-1", 2024-03-11T20:54:10.887 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_numa_prefer_iface": "true", 2024-03-11T20:54:10.887 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-03-11T20:54:10.887 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_objecter_finishers": "1", 2024-03-11T20:54:10.887 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_objectstore": "bluestore", 2024-03-11T20:54:10.888 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_objectstore_fuse": "false", 2024-03-11T20:54:10.888 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_objectstore_tracing": "false", 2024-03-11T20:54:10.888 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_op_complaint_time": "30.000000", 2024-03-11T20:54:10.888 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_op_history_duration": "600", 2024-03-11T20:54:10.888 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_op_history_size": "20", 2024-03-11T20:54:10.888 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_op_history_slow_op_size": "20", 2024-03-11T20:54:10.888 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-03-11T20:54:10.888 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_op_log_threshold": "5", 2024-03-11T20:54:10.888 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_op_num_shards": "0", 2024-03-11T20:54:10.888 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_op_num_shards_hdd": "5", 2024-03-11T20:54:10.888 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_op_num_shards_ssd": "8", 2024-03-11T20:54:10.888 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_op_num_threads_per_shard": "0", 2024-03-11T20:54:10.888 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-03-11T20:54:10.888 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-03-11T20:54:10.888 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-03-11T20:54:10.889 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_op_pq_min_cost": "65536", 2024-03-11T20:54:10.889 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_op_queue": "mclock_scheduler", 2024-03-11T20:54:10.889 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_op_queue_cut_off": "high", 2024-03-11T20:54:10.889 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_op_thread_suicide_timeout": "150", 2024-03-11T20:54:10.889 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_op_thread_timeout": "15", 2024-03-11T20:54:10.889 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_open_classes_on_start": "true", 2024-03-11T20:54:10.889 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_os_flags": "0", 2024-03-11T20:54:10.889 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_peering_op_priority": "255", 2024-03-11T20:54:10.889 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_pg_delete_cost": "1048576", 2024-03-11T20:54:10.889 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_pg_delete_priority": "5", 2024-03-11T20:54:10.889 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-03-11T20:54:10.889 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-03-11T20:54:10.889 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_pg_log_dups_tracked": "3000", 2024-03-11T20:54:10.889 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_pg_log_trim_max": "10000", 2024-03-11T20:54:10.889 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_pg_log_trim_min": "100", 2024-03-11T20:54:10.890 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-03-11T20:54:10.890 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_pg_object_context_cache_count": "64", 2024-03-11T20:54:10.890 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2024-03-11T20:54:10.890 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_pg_stat_report_interval_max_seconds": "5", 2024-03-11T20:54:10.890 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-03-11T20:54:10.890 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-03-11T20:54:10.890 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-03-11T20:54:10.890 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-03-11T20:54:10.890 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-03-11T20:54:10.890 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-03-11T20:54:10.890 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_pool_default_crimson": "false", 2024-03-11T20:54:10.890 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_pool_default_crush_rule": "-1", 2024-03-11T20:54:10.890 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_pool_default_ec_fast_read": "false", 2024-03-11T20:54:10.890 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-03-11T20:54:10.890 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_pool_default_flag_bulk": "false", 2024-03-11T20:54:10.891 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-03-11T20:54:10.891 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_pool_default_flag_nodelete": "false", 2024-03-11T20:54:10.891 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-03-11T20:54:10.891 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-03-11T20:54:10.891 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_pool_default_flags": "0", 2024-03-11T20:54:10.891 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-03-11T20:54:10.891 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_pool_default_min_size": "0", 2024-03-11T20:54:10.891 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-03-11T20:54:10.891 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_pool_default_pg_num": "32", 2024-03-11T20:54:10.891 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_pool_default_pgp_num": "0", 2024-03-11T20:54:10.891 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-03-11T20:54:10.891 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_pool_default_read_ratio": "70", 2024-03-11T20:54:10.891 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_pool_default_size": "3", 2024-03-11T20:54:10.891 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_pool_default_type": "replicated", 2024-03-11T20:54:10.891 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-03-11T20:54:10.892 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_pool_use_gmt_hitset": "true", 2024-03-11T20:54:10.892 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_push_per_object_cost": "1000", 2024-03-11T20:54:10.892 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_read_ec_check_for_errors": "false", 2024-03-11T20:54:10.892 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_recover_clone_overlap": "true", 2024-03-11T20:54:10.892 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_recover_clone_overlap_limit": "10", 2024-03-11T20:54:10.892 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_recovery_cost": "20971520", 2024-03-11T20:54:10.892 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_recovery_delay_start": "0.000000", 2024-03-11T20:54:10.892 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_recovery_max_active": "0", 2024-03-11T20:54:10.892 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_recovery_max_active_hdd": "3", 2024-03-11T20:54:10.892 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_recovery_max_active_ssd": "10", 2024-03-11T20:54:10.892 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_recovery_max_chunk": "8388608", 2024-03-11T20:54:10.892 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-03-11T20:54:10.892 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_recovery_max_single_start": "1", 2024-03-11T20:54:10.892 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_recovery_op_priority": "3", 2024-03-11T20:54:10.893 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_recovery_op_warn_multiple": "16", 2024-03-11T20:54:10.893 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_recovery_priority": "5", 2024-03-11T20:54:10.893 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_recovery_retry_interval": "30.000000", 2024-03-11T20:54:10.893 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_recovery_sleep": "0.000000", 2024-03-11T20:54:10.893 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-03-11T20:54:10.893 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-03-11T20:54:10.893 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-03-11T20:54:10.893 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_repair_during_recovery": "false", 2024-03-11T20:54:10.893 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_requested_scrub_priority": "120", 2024-03-11T20:54:10.893 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-03-11T20:54:10.893 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_rollback_to_cluster_snap": "", 2024-03-11T20:54:10.893 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_scrub_auto_repair": "false", 2024-03-11T20:54:10.893 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-03-11T20:54:10.893 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-03-11T20:54:10.893 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_scrub_begin_hour": "0", 2024-03-11T20:54:10.894 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_scrub_begin_week_day": "0", 2024-03-11T20:54:10.894 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_scrub_chunk_max": "25", 2024-03-11T20:54:10.894 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_scrub_chunk_min": "5", 2024-03-11T20:54:10.894 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_scrub_cost": "52428800", 2024-03-11T20:54:10.894 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_scrub_disable_reservation_queuing": "false", 2024-03-11T20:54:10.894 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_scrub_during_recovery": "false", 2024-03-11T20:54:10.894 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_scrub_end_hour": "0", 2024-03-11T20:54:10.894 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_scrub_end_week_day": "0", 2024-03-11T20:54:10.894 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_scrub_event_cost": "4096", 2024-03-11T20:54:10.894 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-03-11T20:54:10.894 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-03-11T20:54:10.894 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_scrub_invalid_stats": "true", 2024-03-11T20:54:10.894 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_scrub_load_threshold": "0.500000", 2024-03-11T20:54:10.894 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_scrub_max_interval": "604800.000000", 2024-03-11T20:54:10.895 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_scrub_max_preemptions": "5", 2024-03-11T20:54:10.895 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_scrub_min_interval": "86400.000000", 2024-03-11T20:54:10.895 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_scrub_priority": "5", 2024-03-11T20:54:10.895 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_scrub_reservation_timeout": "300000", 2024-03-11T20:54:10.895 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_scrub_sleep": "0.000000", 2024-03-11T20:54:10.895 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_scrub_slow_reservation_response": "30000", 2024-03-11T20:54:10.895 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-03-11T20:54:10.895 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-03-11T20:54:10.895 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_shutdown_pgref_assert": "false", 2024-03-11T20:54:10.895 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_skip_check_past_interval_bounds": "false", 2024-03-11T20:54:10.895 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_skip_data_digest": "false", 2024-03-11T20:54:10.895 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_smart_report_timeout": "5", 2024-03-11T20:54:10.895 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_snap_trim_cost": "1048576", 2024-03-11T20:54:10.895 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_snap_trim_priority": "5", 2024-03-11T20:54:10.896 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_snap_trim_sleep": "0.000000", 2024-03-11T20:54:10.896 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-03-11T20:54:10.896 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-03-11T20:54:10.896 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-03-11T20:54:10.896 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-03-11T20:54:10.896 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_stats_update_period_scrubbing": "15", 2024-03-11T20:54:10.896 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-03-11T20:54:10.896 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_target_transaction_size": "30", 2024-03-11T20:54:10.896 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-03-11T20:54:10.896 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-03-11T20:54:10.896 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-03-11T20:54:10.896 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-03-11T20:54:10.896 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-03-11T20:54:10.896 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-03-11T20:54:10.896 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-03-11T20:54:10.897 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_tier_default_cache_mode": "writeback", 2024-03-11T20:54:10.897 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-03-11T20:54:10.897 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-03-11T20:54:10.897 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_tracing": "false", 2024-03-11T20:54:10.897 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_use_stale_snap": "false", 2024-03-11T20:54:10.897 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-03-11T20:54:10.897 INFO:teuthology.orchestra.run.smithi023.stdout: "osdc_blkin_trace_all": "false", 2024-03-11T20:54:10.897 INFO:teuthology.orchestra.run.smithi023.stdout: "paxos_kill_at": "0", 2024-03-11T20:54:10.897 INFO:teuthology.orchestra.run.smithi023.stdout: "paxos_max_join_drift": "10", 2024-03-11T20:54:10.897 INFO:teuthology.orchestra.run.smithi023.stdout: "paxos_min": "500", 2024-03-11T20:54:10.898 INFO:teuthology.orchestra.run.smithi023.stdout: "paxos_min_wait": "0.050000", 2024-03-11T20:54:10.898 INFO:teuthology.orchestra.run.smithi023.stdout: "paxos_propose_interval": "1.000000", 2024-03-11T20:54:10.898 INFO:teuthology.orchestra.run.smithi023.stdout: "paxos_service_trim_max": "500", 2024-03-11T20:54:10.898 INFO:teuthology.orchestra.run.smithi023.stdout: "paxos_service_trim_max_multiplier": "20", 2024-03-11T20:54:10.898 INFO:teuthology.orchestra.run.smithi023.stdout: "paxos_service_trim_min": "10", 2024-03-11T20:54:10.898 INFO:teuthology.orchestra.run.smithi023.stdout: "paxos_stash_full_interval": "25", 2024-03-11T20:54:10.898 INFO:teuthology.orchestra.run.smithi023.stdout: "paxos_trim_max": "500", 2024-03-11T20:54:10.898 INFO:teuthology.orchestra.run.smithi023.stdout: "paxos_trim_min": "250", 2024-03-11T20:54:10.898 INFO:teuthology.orchestra.run.smithi023.stdout: "perf": "true", 2024-03-11T20:54:10.898 INFO:teuthology.orchestra.run.smithi023.stdout: "pid_file": "", 2024-03-11T20:54:10.898 INFO:teuthology.orchestra.run.smithi023.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-03-11T20:54:10.898 INFO:teuthology.orchestra.run.smithi023.stdout: "plugin_dir": "/usr/lib64/ceph", 2024-03-11T20:54:10.898 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "-", 2024-03-11T20:54:10.899 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrv": "", 2024-03-11T20:54:10.899 INFO:teuthology.orchestra.run.smithi023.stdout: "public_bind_addr": "-", 2024-03-11T20:54:10.899 INFO:teuthology.orchestra.run.smithi023.stdout: "public_network": "", 2024-03-11T20:54:10.899 INFO:teuthology.orchestra.run.smithi023.stdout: "public_network_interface": "", 2024-03-11T20:54:10.899 INFO:teuthology.orchestra.run.smithi023.stdout: "qat_compressor_enabled": "false", 2024-03-11T20:54:10.899 INFO:teuthology.orchestra.run.smithi023.stdout: "qat_compressor_session_max_number": "256", 2024-03-11T20:54:10.899 INFO:teuthology.orchestra.run.smithi023.stdout: "rados_mon_op_timeout": "0", 2024-03-11T20:54:10.899 INFO:teuthology.orchestra.run.smithi023.stdout: "rados_osd_op_timeout": "0", 2024-03-11T20:54:10.899 INFO:teuthology.orchestra.run.smithi023.stdout: "rados_tracing": "false", 2024-03-11T20:54:10.899 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_atime_update_interval": "60", 2024-03-11T20:54:10.899 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-03-11T20:54:10.899 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_balance_parent_reads": "false", 2024-03-11T20:54:10.899 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_balance_snap_reads": "false", 2024-03-11T20:54:10.899 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_blkin_trace_all": "false", 2024-03-11T20:54:10.899 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_blocklist_expire_seconds": "0", 2024-03-11T20:54:10.900 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_blocklist_on_break_lock": "true", 2024-03-11T20:54:10.900 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_cache": "true", 2024-03-11T20:54:10.900 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_cache_block_writes_upfront": "false", 2024-03-11T20:54:10.900 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_cache_max_dirty": "25165824", 2024-03-11T20:54:10.900 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-03-11T20:54:10.900 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_cache_max_dirty_object": "0", 2024-03-11T20:54:10.900 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_cache_policy": "writearound", 2024-03-11T20:54:10.900 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_cache_size": "33554432", 2024-03-11T20:54:10.900 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_cache_target_dirty": "16777216", 2024-03-11T20:54:10.900 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-03-11T20:54:10.900 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_clone_copy_on_read": "false", 2024-03-11T20:54:10.900 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_compression_hint": "none", 2024-03-11T20:54:10.900 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_concurrent_management_ops": "10", 2024-03-11T20:54:10.900 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-03-11T20:54:10.901 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_default_clone_format": "auto", 2024-03-11T20:54:10.901 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_default_data_pool": "", 2024-03-11T20:54:10.901 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_default_features": "61", 2024-03-11T20:54:10.901 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_default_format": "2", 2024-03-11T20:54:10.901 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_default_map_options": "", 2024-03-11T20:54:10.901 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_default_order": "22", 2024-03-11T20:54:10.901 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_default_pool": "rbd", 2024-03-11T20:54:10.901 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-03-11T20:54:10.901 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_default_stripe_count": "0", 2024-03-11T20:54:10.901 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_default_stripe_unit": "0", 2024-03-11T20:54:10.901 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_disable_zero_copy_writes": "true", 2024-03-11T20:54:10.901 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_discard_granularity_bytes": "65536", 2024-03-11T20:54:10.901 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-03-11T20:54:10.901 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_enable_alloc_hint": "true", 2024-03-11T20:54:10.901 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-03-11T20:54:10.902 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_io_scheduler": "simple", 2024-03-11T20:54:10.902 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-03-11T20:54:10.902 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_journal_commit_age": "5.000000", 2024-03-11T20:54:10.902 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-03-11T20:54:10.902 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-03-11T20:54:10.902 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-03-11T20:54:10.902 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-03-11T20:54:10.902 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_journal_object_flush_interval": "0", 2024-03-11T20:54:10.902 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-03-11T20:54:10.902 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-03-11T20:54:10.902 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_journal_order": "24", 2024-03-11T20:54:10.902 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_journal_pool": "", 2024-03-11T20:54:10.902 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_journal_splay_width": "4", 2024-03-11T20:54:10.903 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_localize_parent_reads": "false", 2024-03-11T20:54:10.903 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_localize_snap_reads": "false", 2024-03-11T20:54:10.903 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-03-11T20:54:10.903 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-03-11T20:54:10.903 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-03-11T20:54:10.903 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-03-11T20:54:10.903 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-03-11T20:54:10.903 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-03-11T20:54:10.903 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_mirror_image_policy_type": "simple", 2024-03-11T20:54:10.903 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-03-11T20:54:10.903 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-03-11T20:54:10.904 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-03-11T20:54:10.904 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-03-11T20:54:10.904 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-03-11T20:54:10.904 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-03-11T20:54:10.904 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-03-11T20:54:10.904 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_mirror_memory_autotune": "true", 2024-03-11T20:54:10.904 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_mirror_memory_base": "805306368", 2024-03-11T20:54:10.904 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-03-11T20:54:10.904 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-03-11T20:54:10.904 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-03-11T20:54:10.904 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-03-11T20:54:10.904 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_mirror_memory_target": "4294967296", 2024-03-11T20:54:10.904 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-03-11T20:54:10.904 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-03-11T20:54:10.905 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-03-11T20:54:10.905 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_mirroring_delete_delay": "0", 2024-03-11T20:54:10.905 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-03-11T20:54:10.905 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_mirroring_replay_delay": "0", 2024-03-11T20:54:10.905 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-03-11T20:54:10.905 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-03-11T20:54:10.905 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_move_to_trash_on_remove": "false", 2024-03-11T20:54:10.905 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-03-11T20:54:10.905 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_mtime_update_interval": "60", 2024-03-11T20:54:10.905 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_non_blocking_aio": "true", 2024-03-11T20:54:10.905 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_op_thread_timeout": "60", 2024-03-11T20:54:10.906 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_op_threads": "1", 2024-03-11T20:54:10.906 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_parent_cache_enabled": "false", 2024-03-11T20:54:10.906 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_persistent_cache_mode": "disabled", 2024-03-11T20:54:10.906 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_persistent_cache_path": "/tmp", 2024-03-11T20:54:10.906 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_persistent_cache_size": "1073741824", 2024-03-11T20:54:10.906 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_plugins": "", 2024-03-11T20:54:10.906 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_qos_bps_burst": "0", 2024-03-11T20:54:10.906 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-03-11T20:54:10.906 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_qos_bps_limit": "0", 2024-03-11T20:54:10.906 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_qos_exclude_ops": "0", 2024-03-11T20:54:10.907 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_qos_iops_burst": "0", 2024-03-11T20:54:10.907 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-03-11T20:54:10.907 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_qos_iops_limit": "0", 2024-03-11T20:54:10.907 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_qos_read_bps_burst": "0", 2024-03-11T20:54:10.907 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-03-11T20:54:10.907 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_qos_read_bps_limit": "0", 2024-03-11T20:54:10.907 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_qos_read_iops_burst": "0", 2024-03-11T20:54:10.907 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-03-11T20:54:10.907 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_qos_read_iops_limit": "0", 2024-03-11T20:54:10.907 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_qos_schedule_tick_min": "50", 2024-03-11T20:54:10.907 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_qos_write_bps_burst": "0", 2024-03-11T20:54:10.908 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-03-11T20:54:10.908 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_qos_write_bps_limit": "0", 2024-03-11T20:54:10.908 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_qos_write_iops_burst": "0", 2024-03-11T20:54:10.908 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-03-11T20:54:10.908 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_qos_write_iops_limit": "0", 2024-03-11T20:54:10.908 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_quiesce_notification_attempts": "10", 2024-03-11T20:54:10.908 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_read_from_replica_policy": "default", 2024-03-11T20:54:10.908 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-03-11T20:54:10.908 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_readahead_max_bytes": "524288", 2024-03-11T20:54:10.908 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_readahead_trigger_requests": "10", 2024-03-11T20:54:10.908 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_request_timed_out_seconds": "30", 2024-03-11T20:54:10.908 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_skip_partial_discard": "true", 2024-03-11T20:54:10.908 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-03-11T20:54:10.909 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_tracing": "false", 2024-03-11T20:54:10.909 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_validate_names": "true", 2024-03-11T20:54:10.909 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_validate_pool": "true", 2024-03-11T20:54:10.909 INFO:teuthology.orchestra.run.smithi023.stdout: "restapi_base_url": "", 2024-03-11T20:54:10.909 INFO:teuthology.orchestra.run.smithi023.stdout: "restapi_log_level": "", 2024-03-11T20:54:10.909 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_acl_grants_max_num": "100", 2024-03-11T20:54:10.909 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_admin_entry": "admin", 2024-03-11T20:54:10.909 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-03-11T20:54:10.909 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_backend_store": "rados", 2024-03-11T20:54:10.909 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_barbican_url": "", 2024-03-11T20:54:10.910 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_beast_enable_async": "true", 2024-03-11T20:54:10.910 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_bucket_counters_cache": "false", 2024-03-11T20:54:10.910 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_bucket_counters_cache_size": "10000", 2024-03-11T20:54:10.910 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-03-11T20:54:10.910 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-03-11T20:54:10.910 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_bucket_index_max_aio": "128", 2024-03-11T20:54:10.910 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-03-11T20:54:10.910 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-03-11T20:54:10.910 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_bucket_quota_ttl": "600", 2024-03-11T20:54:10.911 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-03-11T20:54:10.911 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_cache_enabled": "true", 2024-03-11T20:54:10.911 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_cache_expiry_interval": "900", 2024-03-11T20:54:10.911 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_cache_lru_size": "10000", 2024-03-11T20:54:10.911 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_config_store": "rados", 2024-03-11T20:54:10.911 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_content_length_compat": "false", 2024-03-11T20:54:10.911 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_copy_obj_progress": "true", 2024-03-11T20:54:10.911 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-03-11T20:54:10.911 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_cors_rules_max_num": "100", 2024-03-11T20:54:10.911 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_cross_domain_policy": "", 2024-03-11T20:54:10.912 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_crypt_default_encryption_key": "", 2024-03-11T20:54:10.912 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_crypt_kmip_addr": "", 2024-03-11T20:54:10.912 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_crypt_kmip_ca_path": "", 2024-03-11T20:54:10.912 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_crypt_kmip_client_cert": "", 2024-03-11T20:54:10.912 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_crypt_kmip_client_key": "", 2024-03-11T20:54:10.912 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-03-11T20:54:10.912 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_crypt_kmip_password": "", 2024-03-11T20:54:10.912 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-03-11T20:54:10.912 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_crypt_kmip_username": "", 2024-03-11T20:54:10.912 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_crypt_require_ssl": "true", 2024-03-11T20:54:10.913 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-03-11T20:54:10.913 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-03-11T20:54:10.913 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-03-11T20:54:10.913 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-03-11T20:54:10.913 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-03-11T20:54:10.913 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-03-11T20:54:10.913 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-03-11T20:54:10.913 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-03-11T20:54:10.913 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-03-11T20:54:10.913 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-03-11T20:54:10.913 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-03-11T20:54:10.914 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-03-11T20:54:10.914 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-03-11T20:54:10.914 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-03-11T20:54:10.914 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_crypt_suppress_logs": "true", 2024-03-11T20:54:10.914 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_crypt_vault_addr": "", 2024-03-11T20:54:10.914 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_crypt_vault_auth": "token", 2024-03-11T20:54:10.914 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_crypt_vault_namespace": "", 2024-03-11T20:54:10.914 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_crypt_vault_prefix": "", 2024-03-11T20:54:10.914 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-03-11T20:54:10.914 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-03-11T20:54:10.914 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-03-11T20:54:10.914 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-03-11T20:54:10.914 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_crypt_vault_token_file": "", 2024-03-11T20:54:10.915 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-03-11T20:54:10.915 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_curl_buffersize": "524288", 2024-03-11T20:54:10.915 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_curl_low_speed_limit": "1024", 2024-03-11T20:54:10.915 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_curl_low_speed_time": "300", 2024-03-11T20:54:10.915 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_curl_tcp_keepalive": "0", 2024-03-11T20:54:10.915 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-03-11T20:54:10.915 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-03-11T20:54:10.915 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-03-11T20:54:10.915 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-03-11T20:54:10.915 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-03-11T20:54:10.915 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_d3n_l1_fadvise": "4", 2024-03-11T20:54:10.915 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-03-11T20:54:10.915 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-03-11T20:54:10.915 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-03-11T20:54:10.915 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_d4n_host": "127.0.0.1", 2024-03-11T20:54:10.916 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_d4n_port": "6379", 2024-03-11T20:54:10.916 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-03-11T20:54:10.916 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_data_log_changes_size": "1000", 2024-03-11T20:54:10.916 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_data_log_num_shards": "128", 2024-03-11T20:54:10.916 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-03-11T20:54:10.916 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_data_log_window": "30", 2024-03-11T20:54:10.916 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_data_notify_interval_msec": "0", 2024-03-11T20:54:10.916 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_data_sync_poll_interval": "20", 2024-03-11T20:54:10.916 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_data_sync_spawn_window": "20", 2024-03-11T20:54:10.916 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-03-11T20:54:10.916 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-03-11T20:54:10.916 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-03-11T20:54:10.916 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_default_data_log_backing": "fifo", 2024-03-11T20:54:10.916 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-03-11T20:54:10.917 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_default_region_info_oid": "default.region", 2024-03-11T20:54:10.917 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-03-11T20:54:10.917 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-03-11T20:54:10.917 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_defer_to_bucket_acls": "", 2024-03-11T20:54:10.917 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-03-11T20:54:10.917 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-03-11T20:54:10.917 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-03-11T20:54:10.917 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-03-11T20:54:10.917 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-03-11T20:54:10.917 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-03-11T20:54:10.917 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-03-11T20:54:10.917 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-03-11T20:54:10.917 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_dmclock_data_res": "500.000000", 2024-03-11T20:54:10.917 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-03-11T20:54:10.917 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-03-11T20:54:10.918 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-03-11T20:54:10.918 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-03-11T20:54:10.918 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_dns_name": "", 2024-03-11T20:54:10.918 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_dns_s3website_name": "", 2024-03-11T20:54:10.918 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_dynamic_resharding": "true", 2024-03-11T20:54:10.918 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-03-11T20:54:10.918 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_enable_gc_threads": "true", 2024-03-11T20:54:10.918 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_enable_lc_threads": "true", 2024-03-11T20:54:10.918 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_enable_ops_log": "false", 2024-03-11T20:54:10.918 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_enable_quota_threads": "true", 2024-03-11T20:54:10.918 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_enable_static_website": "false", 2024-03-11T20:54:10.918 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_enable_usage_log": "false", 2024-03-11T20:54:10.918 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_enforce_swift_acls": "true", 2024-03-11T20:54:10.918 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_exit_timeout_secs": "120", 2024-03-11T20:54:10.918 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_expose_bucket": "false", 2024-03-11T20:54:10.919 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_extended_http_attrs": "", 2024-03-11T20:54:10.919 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_filter": "none", 2024-03-11T20:54:10.919 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-03-11T20:54:10.919 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_frontends": "beast port=7480", 2024-03-11T20:54:10.919 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_gc_max_concurrent_io": "10", 2024-03-11T20:54:10.919 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_gc_max_deferred": "50", 2024-03-11T20:54:10.919 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-03-11T20:54:10.919 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_gc_max_objs": "32", 2024-03-11T20:54:10.919 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_gc_max_queue_size": "134213632", 2024-03-11T20:54:10.919 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_gc_max_trim_chunk": "16", 2024-03-11T20:54:10.919 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_gc_obj_min_wait": "7200", 2024-03-11T20:54:10.919 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_gc_processor_max_time": "3600", 2024-03-11T20:54:10.919 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_gc_processor_period": "3600", 2024-03-11T20:54:10.919 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-03-11T20:54:10.920 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_get_obj_window_size": "16777216", 2024-03-11T20:54:10.920 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_healthcheck_disabling_path": "", 2024-03-11T20:54:10.920 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_ignore_get_invalid_range": "false", 2024-03-11T20:54:10.920 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_init_timeout": "300", 2024-03-11T20:54:10.920 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-03-11T20:54:10.920 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-03-11T20:54:10.920 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_kafka_connection_idle": "30", 2024-03-11T20:54:10.920 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_kafka_sleep_timeout": "10", 2024-03-11T20:54:10.920 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-03-11T20:54:10.920 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_keystone_accepted_reader_roles": "", 2024-03-11T20:54:10.920 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-03-11T20:54:10.920 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_keystone_admin_domain": "", 2024-03-11T20:54:10.920 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_keystone_admin_password": "", 2024-03-11T20:54:10.920 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_keystone_admin_password_path": "", 2024-03-11T20:54:10.920 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_keystone_admin_project": "", 2024-03-11T20:54:10.921 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_keystone_admin_tenant": "", 2024-03-11T20:54:10.921 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_keystone_admin_token": "", 2024-03-11T20:54:10.921 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_keystone_admin_token_path": "", 2024-03-11T20:54:10.921 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_keystone_admin_user": "", 2024-03-11T20:54:10.921 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_keystone_api_version": "2", 2024-03-11T20:54:10.921 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_keystone_barbican_domain": "", 2024-03-11T20:54:10.921 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_keystone_barbican_password": "", 2024-03-11T20:54:10.921 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_keystone_barbican_project": "", 2024-03-11T20:54:10.921 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_keystone_barbican_tenant": "", 2024-03-11T20:54:10.921 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_keystone_barbican_user": "", 2024-03-11T20:54:10.921 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-03-11T20:54:10.921 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_keystone_implicit_tenants": "false", 2024-03-11T20:54:10.921 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-03-11T20:54:10.921 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_keystone_service_token_enabled": "false", 2024-03-11T20:54:10.922 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_keystone_token_cache_size": "10000", 2024-03-11T20:54:10.922 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_keystone_url": "", 2024-03-11T20:54:10.922 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_keystone_verify_ssl": "true", 2024-03-11T20:54:10.922 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_lc_debug_interval": "-1", 2024-03-11T20:54:10.922 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_lc_lock_max_time": "90", 2024-03-11T20:54:10.922 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_lc_max_objs": "32", 2024-03-11T20:54:10.922 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_lc_max_rules": "1000", 2024-03-11T20:54:10.922 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_lc_max_worker": "3", 2024-03-11T20:54:10.922 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_lc_max_wp_worker": "3", 2024-03-11T20:54:10.922 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_lc_thread_delay": "0", 2024-03-11T20:54:10.922 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-03-11T20:54:10.922 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_ldap_dnattr": "uid", 2024-03-11T20:54:10.922 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-03-11T20:54:10.922 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_ldap_searchfilter": "", 2024-03-11T20:54:10.923 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-03-11T20:54:10.923 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_ldap_uri": "ldaps://", 2024-03-11T20:54:10.923 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-03-11T20:54:10.923 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-03-11T20:54:10.923 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-03-11T20:54:10.923 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_log_http_headers": "", 2024-03-11T20:54:10.923 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_log_nonexistent_bucket": "false", 2024-03-11T20:54:10.923 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-03-11T20:54:10.923 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_log_object_name_utc": "false", 2024-03-11T20:54:10.923 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_lua_max_memory_per_state": "128000", 2024-03-11T20:54:10.923 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2024-03-11T20:54:10.923 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_max_attr_name_len": "0", 2024-03-11T20:54:10.923 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_max_attr_size": "0", 2024-03-11T20:54:10.923 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_max_attrs_num_in_req": "0", 2024-03-11T20:54:10.924 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_max_chunk_size": "4194304", 2024-03-11T20:54:10.924 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_max_concurrent_requests": "1024", 2024-03-11T20:54:10.924 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-03-11T20:54:10.924 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_max_dynamic_shards": "1999", 2024-03-11T20:54:10.924 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_max_listing_results": "1000", 2024-03-11T20:54:10.924 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_max_notify_retries": "10", 2024-03-11T20:54:10.924 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_max_objs_per_shard": "100000", 2024-03-11T20:54:10.924 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_max_put_param_size": "1048576", 2024-03-11T20:54:10.924 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_max_put_size": "5368709120", 2024-03-11T20:54:10.924 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_max_slo_entries": "1000", 2024-03-11T20:54:10.924 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_md_log_max_shards": "64", 2024-03-11T20:54:10.924 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_md_notify_interval_msec": "200", 2024-03-11T20:54:10.924 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_meta_sync_poll_interval": "20", 2024-03-11T20:54:10.924 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_meta_sync_spawn_window": "20", 2024-03-11T20:54:10.925 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-03-11T20:54:10.925 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_mp_lock_max_time": "600", 2024-03-11T20:54:10.925 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-03-11T20:54:10.925 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_multipart_min_part_size": "5242880", 2024-03-11T20:54:10.925 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-03-11T20:54:10.925 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-03-11T20:54:10.925 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_nfs_fhcache_size": "2017", 2024-03-11T20:54:10.925 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-03-11T20:54:10.925 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-03-11T20:54:10.925 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_nfs_lru_lanes": "5", 2024-03-11T20:54:10.925 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_nfs_max_gc": "300", 2024-03-11T20:54:10.925 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-03-11T20:54:10.925 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_nfs_run_gc_threads": "false", 2024-03-11T20:54:10.925 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_nfs_run_lc_threads": "false", 2024-03-11T20:54:10.926 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_nfs_run_quota_threads": "false", 2024-03-11T20:54:10.926 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_nfs_run_sync_thread": "false", 2024-03-11T20:54:10.926 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-03-11T20:54:10.926 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-03-11T20:54:10.926 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_num_async_rados_threads": "32", 2024-03-11T20:54:10.926 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_num_control_oids": "8", 2024-03-11T20:54:10.926 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_numa_node": "-1", 2024-03-11T20:54:10.926 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_obj_stripe_size": "4194304", 2024-03-11T20:54:10.926 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-03-11T20:54:10.926 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_objexp_chunk_size": "100", 2024-03-11T20:54:10.926 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_objexp_gc_interval": "600", 2024-03-11T20:54:10.926 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_objexp_hints_num_shards": "127", 2024-03-11T20:54:10.926 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-03-11T20:54:10.927 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-03-11T20:54:10.927 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_op_thread_timeout": "600", 2024-03-11T20:54:10.927 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_op_tracing": "false", 2024-03-11T20:54:10.927 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_opa_token": "", 2024-03-11T20:54:10.927 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_opa_url": "", 2024-03-11T20:54:10.927 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_opa_verify_ssl": "true", 2024-03-11T20:54:10.927 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-03-11T20:54:10.927 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-03-11T20:54:10.927 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_ops_log_rados": "false", 2024-03-11T20:54:10.927 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_ops_log_socket_path": "", 2024-03-11T20:54:10.927 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-03-11T20:54:10.927 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-03-11T20:54:10.928 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-03-11T20:54:10.928 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_period_push_interval": "2.000000", 2024-03-11T20:54:10.928 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_period_push_interval_max": "30.000000", 2024-03-11T20:54:10.928 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_period_root_pool": ".rgw.root", 2024-03-11T20:54:10.928 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-03-11T20:54:10.928 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2024-03-11T20:54:10.928 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_posix_cache_lanes": "3", 2024-03-11T20:54:10.928 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_posix_cache_lmdb_count": "3", 2024-03-11T20:54:10.928 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_posix_cache_max_buckets": "100", 2024-03-11T20:54:10.928 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_posix_cache_partitions": "3", 2024-03-11T20:54:10.928 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2024-03-11T20:54:10.928 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_print_continue": "true", 2024-03-11T20:54:10.928 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_print_prohibited_content_length": "false", 2024-03-11T20:54:10.928 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-03-11T20:54:10.928 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-03-11T20:54:10.929 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-03-11T20:54:10.929 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-03-11T20:54:10.929 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_rados_tracing": "false", 2024-03-11T20:54:10.929 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_realm": "", 2024-03-11T20:54:10.929 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_realm_id": "", 2024-03-11T20:54:10.929 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-03-11T20:54:10.929 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_region": "", 2024-03-11T20:54:10.929 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_region_root_pool": ".rgw.root", 2024-03-11T20:54:10.929 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_relaxed_region_enforcement": "false", 2024-03-11T20:54:10.929 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-03-11T20:54:10.929 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-03-11T20:54:10.929 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_request_uri": "", 2024-03-11T20:54:10.929 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_reshard_batch_size": "64", 2024-03-11T20:54:10.929 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-03-11T20:54:10.930 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_reshard_max_aio": "128", 2024-03-11T20:54:10.930 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_reshard_num_logs": "16", 2024-03-11T20:54:10.930 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_reshard_thread_interval": "600", 2024-03-11T20:54:10.930 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_resolve_cname": "false", 2024-03-11T20:54:10.930 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_rest_getusage_op_compat": "false", 2024-03-11T20:54:10.930 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_run_sync_thread": "true", 2024-03-11T20:54:10.930 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-03-11T20:54:10.930 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_s3_auth_use_keystone": "false", 2024-03-11T20:54:10.930 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_s3_auth_use_ldap": "false", 2024-03-11T20:54:10.930 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_s3_auth_use_rados": "true", 2024-03-11T20:54:10.930 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_s3_auth_use_sts": "false", 2024-03-11T20:54:10.930 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-03-11T20:54:10.930 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_s3_success_create_obj_status": "0", 2024-03-11T20:54:10.930 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-03-11T20:54:10.931 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_scheduler_type": "throttler", 2024-03-11T20:54:10.931 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_script_uri": "", 2024-03-11T20:54:10.931 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_service_provider_name": "", 2024-03-11T20:54:10.931 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-03-11T20:54:10.931 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_sts_client_id": "", 2024-03-11T20:54:10.931 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_sts_client_secret": "", 2024-03-11T20:54:10.931 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_sts_entry": "sts", 2024-03-11T20:54:10.931 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_sts_key": "sts", 2024-03-11T20:54:10.931 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_sts_max_session_duration": "43200", 2024-03-11T20:54:10.931 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_sts_min_session_duration": "900", 2024-03-11T20:54:10.931 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_sts_token_introspection_url": "", 2024-03-11T20:54:10.931 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_swift_account_in_url": "false", 2024-03-11T20:54:10.931 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_swift_auth_entry": "auth", 2024-03-11T20:54:10.932 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_swift_auth_url": "", 2024-03-11T20:54:10.932 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_swift_custom_header": "", 2024-03-11T20:54:10.932 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_swift_enforce_content_length": "false", 2024-03-11T20:54:10.932 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_swift_need_stats": "true", 2024-03-11T20:54:10.932 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_swift_tenant_name": "", 2024-03-11T20:54:10.932 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_swift_token_expiration": "86400", 2024-03-11T20:54:10.932 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_swift_url": "", 2024-03-11T20:54:10.932 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_swift_url_prefix": "swift", 2024-03-11T20:54:10.932 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_swift_versioning_enabled": "false", 2024-03-11T20:54:10.932 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-03-11T20:54:10.932 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-03-11T20:54:10.932 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_sync_lease_period": "120", 2024-03-11T20:54:10.932 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-03-11T20:54:10.932 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_sync_log_trim_interval": "1200", 2024-03-11T20:54:10.933 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-03-11T20:54:10.933 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-03-11T20:54:10.933 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-03-11T20:54:10.933 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_sync_obj_etag_verify": "false", 2024-03-11T20:54:10.933 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_sync_trace_history_size": "4096", 2024-03-11T20:54:10.933 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-03-11T20:54:10.933 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-03-11T20:54:10.933 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_thread_pool_size": "512", 2024-03-11T20:54:10.933 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_topic_persistency_max_retries": "0", 2024-03-11T20:54:10.933 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_topic_persistency_sleep_duration": "0", 2024-03-11T20:54:10.933 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_topic_persistency_time_to_live": "0", 2024-03-11T20:54:10.933 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_topic_require_publish_policy": "false", 2024-03-11T20:54:10.933 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_torrent_comment": "", 2024-03-11T20:54:10.933 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_torrent_createby": "", 2024-03-11T20:54:10.934 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_torrent_encoding": "", 2024-03-11T20:54:10.934 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_torrent_flag": "false", 2024-03-11T20:54:10.934 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_torrent_max_size": "5368709120", 2024-03-11T20:54:10.934 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_torrent_origin": "", 2024-03-11T20:54:10.934 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_torrent_sha_unit": "524288", 2024-03-11T20:54:10.934 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_torrent_tracker": "", 2024-03-11T20:54:10.934 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_trust_forwarded_https": "false", 2024-03-11T20:54:10.934 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-03-11T20:54:10.934 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_usage_log_tick_interval": "30", 2024-03-11T20:54:10.934 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_usage_max_shards": "32", 2024-03-11T20:54:10.934 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_usage_max_user_shards": "1", 2024-03-11T20:54:10.934 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_use_opa_authz": "false", 2024-03-11T20:54:10.934 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_user_counters_cache": "false", 2024-03-11T20:54:10.935 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_user_counters_cache_size": "10000", 2024-03-11T20:54:10.935 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-03-11T20:54:10.935 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_user_default_quota_max_size": "-1", 2024-03-11T20:54:10.935 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_user_max_buckets": "1000", 2024-03-11T20:54:10.935 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_user_policies_max_num": "100", 2024-03-11T20:54:10.935 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-03-11T20:54:10.935 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-03-11T20:54:10.935 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_user_quota_sync_interval": "86400", 2024-03-11T20:54:10.935 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-03-11T20:54:10.936 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_user_unique_email": "true", 2024-03-11T20:54:10.936 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_verify_ssl": "true", 2024-03-11T20:54:10.936 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_website_routing_rules_max_num": "50", 2024-03-11T20:54:10.936 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_zone": "", 2024-03-11T20:54:10.936 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_zone_id": "", 2024-03-11T20:54:10.936 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-03-11T20:54:10.936 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_zonegroup": "", 2024-03-11T20:54:10.936 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_zonegroup_id": "", 2024-03-11T20:54:10.936 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-03-11T20:54:10.936 INFO:teuthology.orchestra.run.smithi023.stdout: "rgwlc_auto_session_clear": "true", 2024-03-11T20:54:10.936 INFO:teuthology.orchestra.run.smithi023.stdout: "rgwlc_skip_bucket_step": "false", 2024-03-11T20:54:10.936 INFO:teuthology.orchestra.run.smithi023.stdout: "rocksdb_block_size": "4096", 2024-03-11T20:54:10.936 INFO:teuthology.orchestra.run.smithi023.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-03-11T20:54:10.936 INFO:teuthology.orchestra.run.smithi023.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-03-11T20:54:10.937 INFO:teuthology.orchestra.run.smithi023.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-03-11T20:54:10.937 INFO:teuthology.orchestra.run.smithi023.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-03-11T20:54:10.937 INFO:teuthology.orchestra.run.smithi023.stdout: "rocksdb_cache_shard_bits": "4", 2024-03-11T20:54:10.937 INFO:teuthology.orchestra.run.smithi023.stdout: "rocksdb_cache_size": "536870912", 2024-03-11T20:54:10.937 INFO:teuthology.orchestra.run.smithi023.stdout: "rocksdb_cache_type": "binned_lru", 2024-03-11T20:54:10.937 INFO:teuthology.orchestra.run.smithi023.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-03-11T20:54:10.937 INFO:teuthology.orchestra.run.smithi023.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-03-11T20:54:10.937 INFO:teuthology.orchestra.run.smithi023.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-03-11T20:54:10.937 INFO:teuthology.orchestra.run.smithi023.stdout: "rocksdb_collect_compaction_stats": "false", 2024-03-11T20:54:10.938 INFO:teuthology.orchestra.run.smithi023.stdout: "rocksdb_collect_extended_stats": "false", 2024-03-11T20:54:10.938 INFO:teuthology.orchestra.run.smithi023.stdout: "rocksdb_collect_memory_stats": "false", 2024-03-11T20:54:10.938 INFO:teuthology.orchestra.run.smithi023.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-03-11T20:54:10.938 INFO:teuthology.orchestra.run.smithi023.stdout: "rocksdb_index_type": "binary_search", 2024-03-11T20:54:10.938 INFO:teuthology.orchestra.run.smithi023.stdout: "rocksdb_log_to_ceph_log": "true", 2024-03-11T20:54:10.938 INFO:teuthology.orchestra.run.smithi023.stdout: "rocksdb_metadata_block_size": "4096", 2024-03-11T20:54:10.938 INFO:teuthology.orchestra.run.smithi023.stdout: "rocksdb_partition_filters": "false", 2024-03-11T20:54:10.938 INFO:teuthology.orchestra.run.smithi023.stdout: "rocksdb_perf": "false", 2024-03-11T20:54:10.938 INFO:teuthology.orchestra.run.smithi023.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-03-11T20:54:10.938 INFO:teuthology.orchestra.run.smithi023.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-03-11T20:54:10.938 INFO:teuthology.orchestra.run.smithi023.stdout: "rotating_keys_renewal_timeout": "10", 2024-03-11T20:54:10.938 INFO:teuthology.orchestra.run.smithi023.stdout: "run_dir": "/var/run/ceph", 2024-03-11T20:54:10.939 INFO:teuthology.orchestra.run.smithi023.stdout: "seastore_block_create": "true", 2024-03-11T20:54:10.939 INFO:teuthology.orchestra.run.smithi023.stdout: "seastore_cache_lru_size": "67108864", 2024-03-11T20:54:10.939 INFO:teuthology.orchestra.run.smithi023.stdout: "seastore_cbjournal_size": "5368709120", 2024-03-11T20:54:10.939 INFO:teuthology.orchestra.run.smithi023.stdout: "seastore_data_delta_based_overwrite": "0", 2024-03-11T20:54:10.939 INFO:teuthology.orchestra.run.smithi023.stdout: "seastore_default_max_object_size": "16777216", 2024-03-11T20:54:10.939 INFO:teuthology.orchestra.run.smithi023.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-03-11T20:54:10.939 INFO:teuthology.orchestra.run.smithi023.stdout: "seastore_device_size": "53687091200", 2024-03-11T20:54:10.939 INFO:teuthology.orchestra.run.smithi023.stdout: "seastore_journal_batch_capacity": "16", 2024-03-11T20:54:10.939 INFO:teuthology.orchestra.run.smithi023.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-03-11T20:54:10.939 INFO:teuthology.orchestra.run.smithi023.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-03-11T20:54:10.939 INFO:teuthology.orchestra.run.smithi023.stdout: "seastore_journal_iodepth_limit": "5", 2024-03-11T20:54:10.939 INFO:teuthology.orchestra.run.smithi023.stdout: "seastore_main_device_type": "SSD", 2024-03-11T20:54:10.940 INFO:teuthology.orchestra.run.smithi023.stdout: "seastore_max_concurrent_transactions": "8", 2024-03-11T20:54:10.940 INFO:teuthology.orchestra.run.smithi023.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-03-11T20:54:10.940 INFO:teuthology.orchestra.run.smithi023.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-03-11T20:54:10.940 INFO:teuthology.orchestra.run.smithi023.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-03-11T20:54:10.940 INFO:teuthology.orchestra.run.smithi023.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-03-11T20:54:10.940 INFO:teuthology.orchestra.run.smithi023.stdout: "seastore_segment_size": "67108864", 2024-03-11T20:54:10.940 INFO:teuthology.orchestra.run.smithi023.stdout: "set_keepcaps": "false", 2024-03-11T20:54:10.940 INFO:teuthology.orchestra.run.smithi023.stdout: "setgroup": "ceph", 2024-03-11T20:54:10.940 INFO:teuthology.orchestra.run.smithi023.stdout: "setuser": "ceph", 2024-03-11T20:54:10.940 INFO:teuthology.orchestra.run.smithi023.stdout: "setuser_match_path": "", 2024-03-11T20:54:10.940 INFO:teuthology.orchestra.run.smithi023.stdout: "target_max_misplaced_ratio": "0.050000", 2024-03-11T20:54:10.940 INFO:teuthology.orchestra.run.smithi023.stdout: "thp": "false", 2024-03-11T20:54:10.940 INFO:teuthology.orchestra.run.smithi023.stdout: "threadpool_default_timeout": "60", 2024-03-11T20:54:10.941 INFO:teuthology.orchestra.run.smithi023.stdout: "threadpool_empty_queue_max_wait": "2", 2024-03-11T20:54:10.941 INFO:teuthology.orchestra.run.smithi023.stdout: "throttler_perf_counter": "true" 2024-03-11T20:54:10.941 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-11T20:54:10.942 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:10 smithi023 bash[21372]: cluster 2024-03-11T20:54:09.494968+0000 osd.3 (osd.3) 111 : cluster 0 6.11 scrub starts 2024-03-11T20:54:10.942 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:10 smithi023 bash[21372]: cluster 2024-03-11T20:54:09.497953+0000 osd.3 (osd.3) 112 : cluster 0 6.11 scrub ok 2024-03-11T20:54:10.942 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:10 smithi023 bash[21372]: cluster 2024-03-11T20:54:09.535752+0000 osd.0 (osd.0) 97 : cluster 0 4.16 deep-scrub starts 2024-03-11T20:54:10.942 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:10 smithi023 bash[21372]: cluster 2024-03-11T20:54:09.537962+0000 osd.0 (osd.0) 98 : cluster 0 4.16 deep-scrub ok 2024-03-11T20:54:10.942 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:10 smithi023 bash[21372]: cluster 2024-03-11T20:54:09.882736+0000 mgr.y (mgr.24380) 131 : cluster 0 pgmap v98: 132 pgs: 132 active+clean; 582 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:54:10.942 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:10 smithi023 bash[29146]: cluster 2024-03-11T20:54:09.494968+0000 osd.3 (osd.3) 111 : cluster 0 6.11 scrub starts 2024-03-11T20:54:10.942 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:10 smithi023 bash[29146]: cluster 2024-03-11T20:54:09.497953+0000 osd.3 (osd.3) 112 : cluster 0 6.11 scrub ok 2024-03-11T20:54:10.942 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:10 smithi023 bash[29146]: cluster 2024-03-11T20:54:09.535752+0000 osd.0 (osd.0) 97 : cluster 0 4.16 deep-scrub starts 2024-03-11T20:54:10.943 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:10 smithi023 bash[29146]: cluster 2024-03-11T20:54:09.537962+0000 osd.0 (osd.0) 98 : cluster 0 4.16 deep-scrub ok 2024-03-11T20:54:10.943 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:10 smithi023 bash[29146]: cluster 2024-03-11T20:54:09.882736+0000 mgr.y (mgr.24380) 131 : cluster 0 pgmap v98: 132 pgs: 132 active+clean; 582 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:54:11.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:10 smithi146 bash[23809]: cluster 2024-03-11T20:54:09.494968+0000 osd.3 (osd.3) 111 : cluster 0 6.11 scrub starts 2024-03-11T20:54:11.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:10 smithi146 bash[23809]: cluster 2024-03-11T20:54:09.497953+0000 osd.3 (osd.3) 112 : cluster 0 6.11 scrub ok 2024-03-11T20:54:11.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:10 smithi146 bash[23809]: cluster 2024-03-11T20:54:09.535752+0000 osd.0 (osd.0) 97 : cluster 0 4.16 deep-scrub starts 2024-03-11T20:54:11.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:10 smithi146 bash[23809]: cluster 2024-03-11T20:54:09.537962+0000 osd.0 (osd.0) 98 : cluster 0 4.16 deep-scrub ok 2024-03-11T20:54:11.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:10 smithi146 bash[23809]: cluster 2024-03-11T20:54:09.882736+0000 mgr.y (mgr.24380) 131 : cluster 0 pgmap v98: 132 pgs: 132 active+clean; 582 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:54:11.557 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-03-11T20:54:12.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:11 smithi146 bash[23809]: cluster 2024-03-11T20:54:10.659452+0000 osd.4 (osd.4) 91 : cluster 0 4.e scrub starts 2024-03-11T20:54:12.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:11 smithi146 bash[23809]: cluster 2024-03-11T20:54:10.662428+0000 osd.4 (osd.4) 92 : cluster 0 4.e scrub ok 2024-03-11T20:54:12.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:11 smithi023 bash[29146]: cluster 2024-03-11T20:54:10.659452+0000 osd.4 (osd.4) 91 : cluster 0 4.e scrub starts 2024-03-11T20:54:12.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:11 smithi023 bash[29146]: cluster 2024-03-11T20:54:10.662428+0000 osd.4 (osd.4) 92 : cluster 0 4.e scrub ok 2024-03-11T20:54:12.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:11 smithi023 bash[21372]: cluster 2024-03-11T20:54:10.659452+0000 osd.4 (osd.4) 91 : cluster 0 4.e scrub starts 2024-03-11T20:54:12.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:11 smithi023 bash[21372]: cluster 2024-03-11T20:54:10.662428+0000 osd.4 (osd.4) 92 : cluster 0 4.e scrub ok 2024-03-11T20:54:13.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:12 smithi146 bash[23809]: cluster 2024-03-11T20:54:11.883495+0000 mgr.y (mgr.24380) 132 : cluster 0 pgmap v99: 132 pgs: 132 active+clean; 582 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:54:13.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:12 smithi146 bash[23809]: cluster 2024-03-11T20:54:11.951796+0000 osd.1 (osd.1) 87 : cluster 0 purged_snaps scrub starts 2024-03-11T20:54:13.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:12 smithi146 bash[23809]: cluster 2024-03-11T20:54:11.951912+0000 osd.1 (osd.1) 88 : cluster 0 purged_snaps scrub ok 2024-03-11T20:54:13.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:12 smithi023 bash[21372]: cluster 2024-03-11T20:54:11.883495+0000 mgr.y (mgr.24380) 132 : cluster 0 pgmap v99: 132 pgs: 132 active+clean; 582 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:54:13.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:12 smithi023 bash[21372]: cluster 2024-03-11T20:54:11.951796+0000 osd.1 (osd.1) 87 : cluster 0 purged_snaps scrub starts 2024-03-11T20:54:13.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:12 smithi023 bash[21372]: cluster 2024-03-11T20:54:11.951912+0000 osd.1 (osd.1) 88 : cluster 0 purged_snaps scrub ok 2024-03-11T20:54:13.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:12 smithi023 bash[29146]: cluster 2024-03-11T20:54:11.883495+0000 mgr.y (mgr.24380) 132 : cluster 0 pgmap v99: 132 pgs: 132 active+clean; 582 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:54:13.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:12 smithi023 bash[29146]: cluster 2024-03-11T20:54:11.951796+0000 osd.1 (osd.1) 87 : cluster 0 purged_snaps scrub starts 2024-03-11T20:54:13.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:12 smithi023 bash[29146]: cluster 2024-03-11T20:54:11.951912+0000 osd.1 (osd.1) 88 : cluster 0 purged_snaps scrub ok 2024-03-11T20:54:14.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:13 smithi146 bash[23809]: cluster 2024-03-11T20:54:12.447371+0000 osd.3 (osd.3) 113 : cluster 0 6.b scrub starts 2024-03-11T20:54:14.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:13 smithi146 bash[23809]: cluster 2024-03-11T20:54:12.449765+0000 osd.3 (osd.3) 114 : cluster 0 6.b scrub ok 2024-03-11T20:54:14.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:13 smithi146 bash[23809]: cluster 2024-03-11T20:54:12.701649+0000 mon.a (mon.0) 930 : cluster 0 osdmap e119: 8 total, 8 up, 8 in 2024-03-11T20:54:14.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:13 smithi146 bash[23809]: cluster 2024-03-11T20:54:12.790702+0000 osd.2 (osd.2) 57 : cluster 0 2.c scrub starts 2024-03-11T20:54:14.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:13 smithi146 bash[23809]: cluster 2024-03-11T20:54:12.792871+0000 osd.2 (osd.2) 58 : cluster 0 2.c scrub ok 2024-03-11T20:54:14.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:13 smithi023 bash[29146]: cluster 2024-03-11T20:54:12.447371+0000 osd.3 (osd.3) 113 : cluster 0 6.b scrub starts 2024-03-11T20:54:14.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:13 smithi023 bash[29146]: cluster 2024-03-11T20:54:12.449765+0000 osd.3 (osd.3) 114 : cluster 0 6.b scrub ok 2024-03-11T20:54:14.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:13 smithi023 bash[29146]: cluster 2024-03-11T20:54:12.701649+0000 mon.a (mon.0) 930 : cluster 0 osdmap e119: 8 total, 8 up, 8 in 2024-03-11T20:54:14.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:13 smithi023 bash[29146]: cluster 2024-03-11T20:54:12.790702+0000 osd.2 (osd.2) 57 : cluster 0 2.c scrub starts 2024-03-11T20:54:14.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:13 smithi023 bash[29146]: cluster 2024-03-11T20:54:12.792871+0000 osd.2 (osd.2) 58 : cluster 0 2.c scrub ok 2024-03-11T20:54:14.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:13 smithi023 bash[21372]: cluster 2024-03-11T20:54:12.447371+0000 osd.3 (osd.3) 113 : cluster 0 6.b scrub starts 2024-03-11T20:54:14.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:13 smithi023 bash[21372]: cluster 2024-03-11T20:54:12.449765+0000 osd.3 (osd.3) 114 : cluster 0 6.b scrub ok 2024-03-11T20:54:14.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:13 smithi023 bash[21372]: cluster 2024-03-11T20:54:12.701649+0000 mon.a (mon.0) 930 : cluster 0 osdmap e119: 8 total, 8 up, 8 in 2024-03-11T20:54:14.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:13 smithi023 bash[21372]: cluster 2024-03-11T20:54:12.790702+0000 osd.2 (osd.2) 57 : cluster 0 2.c scrub starts 2024-03-11T20:54:14.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:13 smithi023 bash[21372]: cluster 2024-03-11T20:54:12.792871+0000 osd.2 (osd.2) 58 : cluster 0 2.c scrub ok 2024-03-11T20:54:15.010 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 20:54:14 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T20:54:15.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:14 smithi146 bash[23809]: cluster 2024-03-11T20:54:13.657927+0000 osd.4 (osd.4) 93 : cluster 0 4.18 deep-scrub starts 2024-03-11T20:54:15.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:14 smithi146 bash[23809]: cluster 2024-03-11T20:54:13.660314+0000 osd.4 (osd.4) 94 : cluster 0 4.18 deep-scrub ok 2024-03-11T20:54:15.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:14 smithi146 bash[23809]: cluster 2024-03-11T20:54:13.884132+0000 mgr.y (mgr.24380) 133 : cluster 0 pgmap v101: 132 pgs: 132 active+clean; 582 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T20:54:15.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:14 smithi023 bash[21372]: cluster 2024-03-11T20:54:13.657927+0000 osd.4 (osd.4) 93 : cluster 0 4.18 deep-scrub starts 2024-03-11T20:54:15.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:14 smithi023 bash[21372]: cluster 2024-03-11T20:54:13.660314+0000 osd.4 (osd.4) 94 : cluster 0 4.18 deep-scrub ok 2024-03-11T20:54:15.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:14 smithi023 bash[21372]: cluster 2024-03-11T20:54:13.884132+0000 mgr.y (mgr.24380) 133 : cluster 0 pgmap v101: 132 pgs: 132 active+clean; 582 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T20:54:15.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:14 smithi023 bash[29146]: cluster 2024-03-11T20:54:13.657927+0000 osd.4 (osd.4) 93 : cluster 0 4.18 deep-scrub starts 2024-03-11T20:54:15.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:14 smithi023 bash[29146]: cluster 2024-03-11T20:54:13.660314+0000 osd.4 (osd.4) 94 : cluster 0 4.18 deep-scrub ok 2024-03-11T20:54:15.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:14 smithi023 bash[29146]: cluster 2024-03-11T20:54:13.884132+0000 mgr.y (mgr.24380) 133 : cluster 0 pgmap v101: 132 pgs: 132 active+clean; 582 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T20:54:16.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:15 smithi146 bash[23809]: cluster 2024-03-11T20:54:14.163450+0000 osd.7 (osd.7) 93 : cluster 0 3.15 scrub starts 2024-03-11T20:54:16.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:15 smithi146 bash[23809]: cluster 2024-03-11T20:54:14.167294+0000 osd.7 (osd.7) 94 : cluster 0 3.15 scrub ok 2024-03-11T20:54:16.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:15 smithi146 bash[23809]: cluster 2024-03-11T20:54:14.629123+0000 osd.4 (osd.4) 95 : cluster 0 4.c scrub starts 2024-03-11T20:54:16.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:15 smithi146 bash[23809]: cluster 2024-03-11T20:54:14.631651+0000 osd.4 (osd.4) 96 : cluster 0 4.c scrub ok 2024-03-11T20:54:16.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:15 smithi146 bash[23809]: audit 2024-03-11T20:54:14.957753+0000 mgr.y (mgr.24380) 134 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:54:16.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:15 smithi023 bash[21372]: cluster 2024-03-11T20:54:14.163450+0000 osd.7 (osd.7) 93 : cluster 0 3.15 scrub starts 2024-03-11T20:54:16.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:15 smithi023 bash[21372]: cluster 2024-03-11T20:54:14.167294+0000 osd.7 (osd.7) 94 : cluster 0 3.15 scrub ok 2024-03-11T20:54:16.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:15 smithi023 bash[21372]: cluster 2024-03-11T20:54:14.629123+0000 osd.4 (osd.4) 95 : cluster 0 4.c scrub starts 2024-03-11T20:54:16.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:15 smithi023 bash[21372]: cluster 2024-03-11T20:54:14.631651+0000 osd.4 (osd.4) 96 : cluster 0 4.c scrub ok 2024-03-11T20:54:16.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:15 smithi023 bash[21372]: audit 2024-03-11T20:54:14.957753+0000 mgr.y (mgr.24380) 134 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:54:16.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:15 smithi023 bash[29146]: cluster 2024-03-11T20:54:14.163450+0000 osd.7 (osd.7) 93 : cluster 0 3.15 scrub starts 2024-03-11T20:54:16.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:15 smithi023 bash[29146]: cluster 2024-03-11T20:54:14.167294+0000 osd.7 (osd.7) 94 : cluster 0 3.15 scrub ok 2024-03-11T20:54:16.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:15 smithi023 bash[29146]: cluster 2024-03-11T20:54:14.629123+0000 osd.4 (osd.4) 95 : cluster 0 4.c scrub starts 2024-03-11T20:54:16.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:15 smithi023 bash[29146]: cluster 2024-03-11T20:54:14.631651+0000 osd.4 (osd.4) 96 : cluster 0 4.c scrub ok 2024-03-11T20:54:16.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:15 smithi023 bash[29146]: audit 2024-03-11T20:54:14.957753+0000 mgr.y (mgr.24380) 134 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:54:16.357 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:54:17.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:16 smithi146 bash[23809]: cluster 2024-03-11T20:54:15.184511+0000 osd.7 (osd.7) 95 : cluster 0 6.3 scrub starts 2024-03-11T20:54:17.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:16 smithi146 bash[23809]: cluster 2024-03-11T20:54:15.186762+0000 osd.7 (osd.7) 96 : cluster 0 6.3 scrub ok 2024-03-11T20:54:17.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:16 smithi146 bash[23809]: cluster 2024-03-11T20:54:15.434323+0000 osd.0 (osd.0) 99 : cluster 0 3.12 deep-scrub starts 2024-03-11T20:54:17.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:16 smithi146 bash[23809]: cluster 2024-03-11T20:54:15.443434+0000 osd.0 (osd.0) 100 : cluster 0 3.12 deep-scrub ok 2024-03-11T20:54:17.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:16 smithi146 bash[23809]: cluster 2024-03-11T20:54:15.885543+0000 mgr.y (mgr.24380) 135 : cluster 0 pgmap v102: 132 pgs: 132 active+clean; 582 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T20:54:17.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:16 smithi023 bash[21372]: cluster 2024-03-11T20:54:15.184511+0000 osd.7 (osd.7) 95 : cluster 0 6.3 scrub starts 2024-03-11T20:54:17.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:16 smithi023 bash[21372]: cluster 2024-03-11T20:54:15.186762+0000 osd.7 (osd.7) 96 : cluster 0 6.3 scrub ok 2024-03-11T20:54:17.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:16 smithi023 bash[21372]: cluster 2024-03-11T20:54:15.434323+0000 osd.0 (osd.0) 99 : cluster 0 3.12 deep-scrub starts 2024-03-11T20:54:17.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:16 smithi023 bash[21372]: cluster 2024-03-11T20:54:15.443434+0000 osd.0 (osd.0) 100 : cluster 0 3.12 deep-scrub ok 2024-03-11T20:54:17.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:16 smithi023 bash[21372]: cluster 2024-03-11T20:54:15.885543+0000 mgr.y (mgr.24380) 135 : cluster 0 pgmap v102: 132 pgs: 132 active+clean; 582 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T20:54:17.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:16 smithi023 bash[29146]: cluster 2024-03-11T20:54:15.184511+0000 osd.7 (osd.7) 95 : cluster 0 6.3 scrub starts 2024-03-11T20:54:17.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:16 smithi023 bash[29146]: cluster 2024-03-11T20:54:15.186762+0000 osd.7 (osd.7) 96 : cluster 0 6.3 scrub ok 2024-03-11T20:54:17.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:16 smithi023 bash[29146]: cluster 2024-03-11T20:54:15.434323+0000 osd.0 (osd.0) 99 : cluster 0 3.12 deep-scrub starts 2024-03-11T20:54:17.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:16 smithi023 bash[29146]: cluster 2024-03-11T20:54:15.443434+0000 osd.0 (osd.0) 100 : cluster 0 3.12 deep-scrub ok 2024-03-11T20:54:17.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:16 smithi023 bash[29146]: cluster 2024-03-11T20:54:15.885543+0000 mgr.y (mgr.24380) 135 : cluster 0 pgmap v102: 132 pgs: 132 active+clean; 582 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T20:54:18.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:17 smithi146 bash[23809]: cluster 2024-03-11T20:54:16.745219+0000 osd.2 (osd.2) 59 : cluster 0 2.e scrub starts 2024-03-11T20:54:18.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:17 smithi146 bash[23809]: cluster 2024-03-11T20:54:16.747379+0000 osd.2 (osd.2) 60 : cluster 0 2.e scrub ok 2024-03-11T20:54:18.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:17 smithi146 bash[23809]: cluster 2024-03-11T20:54:17.423030+0000 osd.6 (osd.6) 55 : cluster 0 2.14 scrub starts 2024-03-11T20:54:18.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:17 smithi146 bash[23809]: cluster 2024-03-11T20:54:17.425579+0000 osd.6 (osd.6) 56 : cluster 0 2.14 scrub ok 2024-03-11T20:54:18.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:17 smithi023 bash[21372]: cluster 2024-03-11T20:54:16.745219+0000 osd.2 (osd.2) 59 : cluster 0 2.e scrub starts 2024-03-11T20:54:18.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:17 smithi023 bash[21372]: cluster 2024-03-11T20:54:16.747379+0000 osd.2 (osd.2) 60 : cluster 0 2.e scrub ok 2024-03-11T20:54:18.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:17 smithi023 bash[21372]: cluster 2024-03-11T20:54:17.423030+0000 osd.6 (osd.6) 55 : cluster 0 2.14 scrub starts 2024-03-11T20:54:18.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:17 smithi023 bash[21372]: cluster 2024-03-11T20:54:17.425579+0000 osd.6 (osd.6) 56 : cluster 0 2.14 scrub ok 2024-03-11T20:54:18.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:17 smithi023 bash[29146]: cluster 2024-03-11T20:54:16.745219+0000 osd.2 (osd.2) 59 : cluster 0 2.e scrub starts 2024-03-11T20:54:18.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:17 smithi023 bash[29146]: cluster 2024-03-11T20:54:16.747379+0000 osd.2 (osd.2) 60 : cluster 0 2.e scrub ok 2024-03-11T20:54:18.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:17 smithi023 bash[29146]: cluster 2024-03-11T20:54:17.423030+0000 osd.6 (osd.6) 55 : cluster 0 2.14 scrub starts 2024-03-11T20:54:18.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:17 smithi023 bash[29146]: cluster 2024-03-11T20:54:17.425579+0000 osd.6 (osd.6) 56 : cluster 0 2.14 scrub ok 2024-03-11T20:54:18.725 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:54:18 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:20:54:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T20:54:19.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:18 smithi146 bash[23809]: cluster 2024-03-11T20:54:17.886258+0000 mgr.y (mgr.24380) 136 : cluster 0 pgmap v103: 132 pgs: 132 active+clean; 582 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:54:19.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:18 smithi023 bash[29146]: cluster 2024-03-11T20:54:17.886258+0000 mgr.y (mgr.24380) 136 : cluster 0 pgmap v103: 132 pgs: 132 active+clean; 582 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:54:19.026 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:18 smithi023 bash[21372]: cluster 2024-03-11T20:54:17.886258+0000 mgr.y (mgr.24380) 136 : cluster 0 pgmap v103: 132 pgs: 132 active+clean; 582 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:54:19.193 INFO:teuthology.orchestra.run.smithi023.stdout:mon.a: {} 2024-03-11T20:54:19.208 INFO:teuthology.orchestra.run.smithi023.stdout:mon.b: {} 2024-03-11T20:54:19.218 INFO:teuthology.orchestra.run.smithi023.stdout:mon.c: {} 2024-03-11T20:54:19.220 INFO:teuthology.orchestra.run.smithi023.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-11T20:54:19.220 INFO:teuthology.orchestra.run.smithi023.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-11T20:54:19.220 INFO:teuthology.orchestra.run.smithi023.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-11T20:54:19.816 INFO:teuthology.run_tasks:Running task full_sequential... 2024-03-11T20:54:19.849 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-03-11T20:54:19.853 INFO:tasks.radosbench:Beginning radosbench... 2024-03-11T20:54:19.853 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_0 2024-03-11T20:54:19.854 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd pool create unique_pool_0 16 2024-03-11T20:54:19.854 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-03-11T20:54:19.854 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-11T20:54:19.854 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-11T20:54:19.855 INFO:tasks.thrashosds.thrasher:Removing osd 1, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-11T20:54:19.855 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd out 1 2024-03-11T20:54:19.855 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-03-11T20:54:19.855 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:54:19.856 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-03-11T20:54:19.856 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T20:54:19.856 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-03-11T20:54:19.856 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-11T20:54:19.857 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-03-11T20:54:19.857 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd set noscrub 2024-03-11T20:54:19.893 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:54:19.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:19.896 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:54:19.997 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:54:20.008 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:19 smithi146 bash[23809]: audit 2024-03-11T20:54:19.185749+0000 mon.a (mon.0) 931 : audit 0 from='client.? 172.21.15.23:0/2352440293' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T20:54:20.008 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:19 smithi146 bash[23809]: audit 2024-03-11T20:54:19.239675+0000 mon.a (mon.0) 932 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:54:20.024 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:54:20.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:20.026 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:54:20.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:19 smithi023 bash[21372]: audit 2024-03-11T20:54:19.185749+0000 mon.a (mon.0) 931 : audit 0 from='client.? 172.21.15.23:0/2352440293' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T20:54:20.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:19 smithi023 bash[21372]: audit 2024-03-11T20:54:19.239675+0000 mon.a (mon.0) 932 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:54:20.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:19 smithi023 bash[29146]: audit 2024-03-11T20:54:19.185749+0000 mon.a (mon.0) 931 : audit 0 from='client.? 172.21.15.23:0/2352440293' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T20:54:20.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:19 smithi023 bash[29146]: audit 2024-03-11T20:54:19.239675+0000 mon.a (mon.0) 932 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:54:20.127 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:54:20.146 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:54:20.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:20.148 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:54:20.249 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:54:20.276 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:54:20.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:20.278 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:54:20.379 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:54:20.407 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:54:20.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:20.409 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:54:20.509 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:54:20.536 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:54:20.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:20.538 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:54:20.639 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:54:20.665 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:54:20.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:20.667 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:54:20.768 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:54:20.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:20 smithi023 bash[21372]: cluster 2024-03-11T20:54:19.491286+0000 osd.3 (osd.3) 115 : cluster 0 4.17 scrub starts 2024-03-11T20:54:20.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:20 smithi023 bash[21372]: cluster 2024-03-11T20:54:19.493475+0000 osd.3 (osd.3) 116 : cluster 0 4.17 scrub ok 2024-03-11T20:54:20.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:20 smithi023 bash[21372]: cluster 2024-03-11T20:54:19.560821+0000 osd.4 (osd.4) 97 : cluster 0 3.14 scrub starts 2024-03-11T20:54:20.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:20 smithi023 bash[21372]: cluster 2024-03-11T20:54:19.565074+0000 osd.4 (osd.4) 98 : cluster 0 3.14 scrub ok 2024-03-11T20:54:20.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:20 smithi023 bash[21372]: cluster 2024-03-11T20:54:19.887099+0000 mgr.y (mgr.24380) 137 : cluster 0 pgmap v104: 132 pgs: 132 active+clean; 582 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:54:20.780 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:20 smithi023 bash[29146]: cluster 2024-03-11T20:54:19.491286+0000 osd.3 (osd.3) 115 : cluster 0 4.17 scrub starts 2024-03-11T20:54:20.781 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:20 smithi023 bash[29146]: cluster 2024-03-11T20:54:19.493475+0000 osd.3 (osd.3) 116 : cluster 0 4.17 scrub ok 2024-03-11T20:54:20.781 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:20 smithi023 bash[29146]: cluster 2024-03-11T20:54:19.560821+0000 osd.4 (osd.4) 97 : cluster 0 3.14 scrub starts 2024-03-11T20:54:20.781 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:20 smithi023 bash[29146]: cluster 2024-03-11T20:54:19.565074+0000 osd.4 (osd.4) 98 : cluster 0 3.14 scrub ok 2024-03-11T20:54:20.781 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:20 smithi023 bash[29146]: cluster 2024-03-11T20:54:19.887099+0000 mgr.y (mgr.24380) 137 : cluster 0 pgmap v104: 132 pgs: 132 active+clean; 582 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:54:20.794 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:54:20.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:20.796 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:54:20.897 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:54:20.924 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:54:20.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:20.926 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:54:21.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:20 smithi146 bash[23809]: cluster 2024-03-11T20:54:19.491286+0000 osd.3 (osd.3) 115 : cluster 0 4.17 scrub starts 2024-03-11T20:54:21.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:20 smithi146 bash[23809]: cluster 2024-03-11T20:54:19.493475+0000 osd.3 (osd.3) 116 : cluster 0 4.17 scrub ok 2024-03-11T20:54:21.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:20 smithi146 bash[23809]: cluster 2024-03-11T20:54:19.560821+0000 osd.4 (osd.4) 97 : cluster 0 3.14 scrub starts 2024-03-11T20:54:21.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:20 smithi146 bash[23809]: cluster 2024-03-11T20:54:19.565074+0000 osd.4 (osd.4) 98 : cluster 0 3.14 scrub ok 2024-03-11T20:54:21.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:20 smithi146 bash[23809]: cluster 2024-03-11T20:54:19.887099+0000 mgr.y (mgr.24380) 137 : cluster 0 pgmap v104: 132 pgs: 132 active+clean; 582 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:54:21.027 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:54:21.057 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:54:21.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:21.059 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:54:21.160 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:54:21.189 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:54:21.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:21.191 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:54:21.292 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:54:21.319 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:54:21.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:21.321 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:54:21.422 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:54:21.449 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:54:21.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:21.451 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:54:21.552 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:54:21.581 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:54:21.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:21.583 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:54:21.684 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:54:21.708 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:54:21.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:21.710 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:54:21.810 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:54:21.838 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:54:21.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:21.840 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:54:21.941 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:54:21.969 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:54:21.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:21.972 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:54:22.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:21 smithi023 bash[21372]: cluster 2024-03-11T20:54:20.818389+0000 osd.5 (osd.5) 81 : cluster 0 2.16 scrub starts 2024-03-11T20:54:22.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:21 smithi023 bash[21372]: cluster 2024-03-11T20:54:20.820588+0000 osd.5 (osd.5) 82 : cluster 0 2.16 scrub ok 2024-03-11T20:54:22.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:21 smithi023 bash[29146]: cluster 2024-03-11T20:54:20.818389+0000 osd.5 (osd.5) 81 : cluster 0 2.16 scrub starts 2024-03-11T20:54:22.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:21 smithi023 bash[29146]: cluster 2024-03-11T20:54:20.820588+0000 osd.5 (osd.5) 82 : cluster 0 2.16 scrub ok 2024-03-11T20:54:22.072 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:54:22.083 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:21 smithi146 bash[23809]: cluster 2024-03-11T20:54:20.818389+0000 osd.5 (osd.5) 81 : cluster 0 2.16 scrub starts 2024-03-11T20:54:22.084 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:21 smithi146 bash[23809]: cluster 2024-03-11T20:54:20.820588+0000 osd.5 (osd.5) 82 : cluster 0 2.16 scrub ok 2024-03-11T20:54:22.101 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:54:22.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:22.103 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:54:22.204 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:54:22.232 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:54:22.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:22.234 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:54:22.335 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:54:22.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:22.367 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:54:22.367 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:54:22.468 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:54:22.496 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:54:22.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:22.499 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:54:22.599 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:54:22.635 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:54:22.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:22.637 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:54:22.738 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:54:22.765 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:54:22.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:22.767 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:54:22.868 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:54:22.898 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:54:22.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:22.900 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:54:23.001 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:54:23.013 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:22 smithi023 bash[21372]: cluster 2024-03-11T20:54:21.887785+0000 mgr.y (mgr.24380) 138 : cluster 0 pgmap v105: 132 pgs: 132 active+clean; 582 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:54:23.013 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:22 smithi023 bash[21372]: cluster 2024-03-11T20:54:22.446115+0000 osd.6 (osd.6) 57 : cluster 0 4.5 deep-scrub starts 2024-03-11T20:54:23.013 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:22 smithi023 bash[21372]: cluster 2024-03-11T20:54:22.448802+0000 osd.6 (osd.6) 58 : cluster 0 4.5 deep-scrub ok 2024-03-11T20:54:23.013 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:22 smithi023 bash[29146]: cluster 2024-03-11T20:54:21.887785+0000 mgr.y (mgr.24380) 138 : cluster 0 pgmap v105: 132 pgs: 132 active+clean; 582 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:54:23.013 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:22 smithi023 bash[29146]: cluster 2024-03-11T20:54:22.446115+0000 osd.6 (osd.6) 57 : cluster 0 4.5 deep-scrub starts 2024-03-11T20:54:23.013 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:22 smithi023 bash[29146]: cluster 2024-03-11T20:54:22.448802+0000 osd.6 (osd.6) 58 : cluster 0 4.5 deep-scrub ok 2024-03-11T20:54:23.029 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:54:23.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:23.031 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:54:23.132 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:54:23.158 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:54:23.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:23.161 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:54:23.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:22 smithi146 bash[23809]: cluster 2024-03-11T20:54:21.887785+0000 mgr.y (mgr.24380) 138 : cluster 0 pgmap v105: 132 pgs: 132 active+clean; 582 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:54:23.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:22 smithi146 bash[23809]: cluster 2024-03-11T20:54:22.446115+0000 osd.6 (osd.6) 57 : cluster 0 4.5 deep-scrub starts 2024-03-11T20:54:23.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:22 smithi146 bash[23809]: cluster 2024-03-11T20:54:22.448802+0000 osd.6 (osd.6) 58 : cluster 0 4.5 deep-scrub ok 2024-03-11T20:54:23.261 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:54:23.290 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:54:23.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:23.292 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:54:23.393 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:54:23.421 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:54:23.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:23.423 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:54:23.524 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:54:23.548 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:54:23.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:23.551 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:54:23.651 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:54:23.681 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:54:23.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:23.683 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:54:23.784 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:54:23.811 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:54:23.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:23.813 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:54:23.914 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:54:23.926 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:23 smithi023 bash[21372]: cluster 2024-03-11T20:54:22.527056+0000 osd.4 (osd.4) 99 : cluster 0 3.3 scrub starts 2024-03-11T20:54:23.926 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:23 smithi023 bash[21372]: cluster 2024-03-11T20:54:22.531516+0000 osd.4 (osd.4) 100 : cluster 0 3.3 scrub ok 2024-03-11T20:54:23.926 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:23 smithi023 bash[29146]: cluster 2024-03-11T20:54:22.527056+0000 osd.4 (osd.4) 99 : cluster 0 3.3 scrub starts 2024-03-11T20:54:23.926 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:23 smithi023 bash[29146]: cluster 2024-03-11T20:54:22.531516+0000 osd.4 (osd.4) 100 : cluster 0 3.3 scrub ok 2024-03-11T20:54:23.942 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:54:23.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:23.944 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:54:24.045 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:54:24.058 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:23 smithi146 bash[23809]: cluster 2024-03-11T20:54:22.527056+0000 osd.4 (osd.4) 99 : cluster 0 3.3 scrub starts 2024-03-11T20:54:24.058 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:23 smithi146 bash[23809]: cluster 2024-03-11T20:54:22.531516+0000 osd.4 (osd.4) 100 : cluster 0 3.3 scrub ok 2024-03-11T20:54:24.074 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:54:24.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:24.076 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:54:24.177 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:54:24.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:24.207 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:54:24.207 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:54:24.308 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:54:24.335 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:54:24.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:24.338 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:54:24.439 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:54:24.466 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:54:24.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:24.468 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:54:24.569 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:54:24.598 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:54:24.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:24.600 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:54:24.645 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:54:24.649 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:54:24.649 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:54:24.649 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:54:24.649 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:54:24.701 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:54:24.721 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:54:24.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:24.723 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:54:24.826 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:54:24.855 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:54:24.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:24.857 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:54:24.958 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:54:24.986 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:54:24.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:24.988 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:54:25.089 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:54:25.116 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:54:25.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:25.118 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:54:25.140 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:24 smithi023 bash[21372]: cluster 2024-03-11T20:54:23.489997+0000 osd.4 (osd.4) 101 : cluster 0 6.17 scrub starts 2024-03-11T20:54:25.140 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:24 smithi023 bash[21372]: cluster 2024-03-11T20:54:23.491540+0000 osd.4 (osd.4) 102 : cluster 0 6.17 scrub ok 2024-03-11T20:54:25.141 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:24 smithi023 bash[21372]: cluster 2024-03-11T20:54:23.715747+0000 osd.7 (osd.7) 97 : cluster 0 purged_snaps scrub starts 2024-03-11T20:54:25.141 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:24 smithi023 bash[21372]: cluster 2024-03-11T20:54:23.715860+0000 osd.7 (osd.7) 98 : cluster 0 purged_snaps scrub ok 2024-03-11T20:54:25.141 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:24 smithi023 bash[21372]: cluster 2024-03-11T20:54:23.787501+0000 mon.a (mon.0) 933 : cluster 0 osdmap e120: 8 total, 8 up, 8 in 2024-03-11T20:54:25.141 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:24 smithi023 bash[21372]: cluster 2024-03-11T20:54:23.888498+0000 mgr.y (mgr.24380) 139 : cluster 0 pgmap v107: 132 pgs: 132 active+clean; 582 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:54:25.141 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:24 smithi023 bash[21372]: cluster 2024-03-11T20:54:24.028415+0000 osd.7 (osd.7) 99 : cluster 0 6.12 scrub starts 2024-03-11T20:54:25.141 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:24 smithi023 bash[21372]: cluster 2024-03-11T20:54:24.032729+0000 osd.7 (osd.7) 100 : cluster 0 6.12 scrub ok 2024-03-11T20:54:25.141 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:24 smithi023 bash[29146]: cluster 2024-03-11T20:54:23.489997+0000 osd.4 (osd.4) 101 : cluster 0 6.17 scrub starts 2024-03-11T20:54:25.141 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:24 smithi023 bash[29146]: cluster 2024-03-11T20:54:23.491540+0000 osd.4 (osd.4) 102 : cluster 0 6.17 scrub ok 2024-03-11T20:54:25.141 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:24 smithi023 bash[29146]: cluster 2024-03-11T20:54:23.715747+0000 osd.7 (osd.7) 97 : cluster 0 purged_snaps scrub starts 2024-03-11T20:54:25.141 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:24 smithi023 bash[29146]: cluster 2024-03-11T20:54:23.715860+0000 osd.7 (osd.7) 98 : cluster 0 purged_snaps scrub ok 2024-03-11T20:54:25.141 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:24 smithi023 bash[29146]: cluster 2024-03-11T20:54:23.787501+0000 mon.a (mon.0) 933 : cluster 0 osdmap e120: 8 total, 8 up, 8 in 2024-03-11T20:54:25.142 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:24 smithi023 bash[29146]: cluster 2024-03-11T20:54:23.888498+0000 mgr.y (mgr.24380) 139 : cluster 0 pgmap v107: 132 pgs: 132 active+clean; 582 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:54:25.142 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:24 smithi023 bash[29146]: cluster 2024-03-11T20:54:24.028415+0000 osd.7 (osd.7) 99 : cluster 0 6.12 scrub starts 2024-03-11T20:54:25.142 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:24 smithi023 bash[29146]: cluster 2024-03-11T20:54:24.032729+0000 osd.7 (osd.7) 100 : cluster 0 6.12 scrub ok 2024-03-11T20:54:25.219 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:54:25.249 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:54:25.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:25.251 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:54:25.260 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 20:54:24 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T20:54:25.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:24 smithi146 bash[23809]: cluster 2024-03-11T20:54:23.489997+0000 osd.4 (osd.4) 101 : cluster 0 6.17 scrub starts 2024-03-11T20:54:25.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:24 smithi146 bash[23809]: cluster 2024-03-11T20:54:23.491540+0000 osd.4 (osd.4) 102 : cluster 0 6.17 scrub ok 2024-03-11T20:54:25.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:24 smithi146 bash[23809]: cluster 2024-03-11T20:54:23.715747+0000 osd.7 (osd.7) 97 : cluster 0 purged_snaps scrub starts 2024-03-11T20:54:25.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:24 smithi146 bash[23809]: cluster 2024-03-11T20:54:23.715860+0000 osd.7 (osd.7) 98 : cluster 0 purged_snaps scrub ok 2024-03-11T20:54:25.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:24 smithi146 bash[23809]: cluster 2024-03-11T20:54:23.787501+0000 mon.a (mon.0) 933 : cluster 0 osdmap e120: 8 total, 8 up, 8 in 2024-03-11T20:54:25.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:24 smithi146 bash[23809]: cluster 2024-03-11T20:54:23.888498+0000 mgr.y (mgr.24380) 139 : cluster 0 pgmap v107: 132 pgs: 132 active+clean; 582 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:54:25.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:24 smithi146 bash[23809]: cluster 2024-03-11T20:54:24.028415+0000 osd.7 (osd.7) 99 : cluster 0 6.12 scrub starts 2024-03-11T20:54:25.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:24 smithi146 bash[23809]: cluster 2024-03-11T20:54:24.032729+0000 osd.7 (osd.7) 100 : cluster 0 6.12 scrub ok 2024-03-11T20:54:25.352 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:54:25.380 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:54:25.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:25.383 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:54:25.484 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:54:25.512 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:54:25.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:25.514 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:54:25.615 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:54:25.642 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:54:25.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:25.647 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:54:25.748 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:54:25.774 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:54:25.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:25.776 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:54:25.877 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:54:25.888 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:25 smithi146 bash[23809]: cluster 2024-03-11T20:54:24.738765+0000 osd.2 (osd.2) 61 : cluster 0 4.f scrub starts 2024-03-11T20:54:25.889 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:25 smithi146 bash[23809]: cluster 2024-03-11T20:54:24.740144+0000 osd.2 (osd.2) 62 : cluster 0 4.f scrub ok 2024-03-11T20:54:25.889 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:25 smithi146 bash[23809]: audit 2024-03-11T20:54:24.965720+0000 mgr.y (mgr.24380) 140 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:54:25.905 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:54:25.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:25.908 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:54:26.009 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:54:26.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:25 smithi023 bash[29146]: cluster 2024-03-11T20:54:24.738765+0000 osd.2 (osd.2) 61 : cluster 0 4.f scrub starts 2024-03-11T20:54:26.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:25 smithi023 bash[29146]: cluster 2024-03-11T20:54:24.740144+0000 osd.2 (osd.2) 62 : cluster 0 4.f scrub ok 2024-03-11T20:54:26.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:25 smithi023 bash[29146]: audit 2024-03-11T20:54:24.965720+0000 mgr.y (mgr.24380) 140 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:54:26.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:25 smithi023 bash[21372]: cluster 2024-03-11T20:54:24.738765+0000 osd.2 (osd.2) 61 : cluster 0 4.f scrub starts 2024-03-11T20:54:26.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:25 smithi023 bash[21372]: cluster 2024-03-11T20:54:24.740144+0000 osd.2 (osd.2) 62 : cluster 0 4.f scrub ok 2024-03-11T20:54:26.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:25 smithi023 bash[21372]: audit 2024-03-11T20:54:24.965720+0000 mgr.y (mgr.24380) 140 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:54:26.036 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:54:26.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:26.039 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:54:26.140 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:54:26.165 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:54:26.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:26.167 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:54:26.268 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:54:26.297 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:54:26.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:26.299 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:54:26.400 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:54:26.423 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:54:26.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:26.425 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:54:26.526 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:54:26.554 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:54:26.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:26.556 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:54:26.657 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:54:26.686 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:54:26.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:26.689 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:54:26.790 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:54:26.811 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:54:26.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:26.813 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:54:26.914 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:54:26.924 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:26 smithi023 bash[21372]: cluster 2024-03-11T20:54:25.837851+0000 osd.5 (osd.5) 83 : cluster 0 2.2 scrub starts 2024-03-11T20:54:26.924 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:26 smithi023 bash[21372]: cluster 2024-03-11T20:54:25.840271+0000 osd.5 (osd.5) 84 : cluster 0 2.2 scrub ok 2024-03-11T20:54:26.924 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:26 smithi023 bash[21372]: cluster 2024-03-11T20:54:25.889305+0000 mgr.y (mgr.24380) 141 : cluster 0 pgmap v108: 132 pgs: 132 active+clean; 582 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:54:26.924 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:26 smithi023 bash[29146]: cluster 2024-03-11T20:54:25.837851+0000 osd.5 (osd.5) 83 : cluster 0 2.2 scrub starts 2024-03-11T20:54:26.924 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:26 smithi023 bash[29146]: cluster 2024-03-11T20:54:25.840271+0000 osd.5 (osd.5) 84 : cluster 0 2.2 scrub ok 2024-03-11T20:54:26.924 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:26 smithi023 bash[29146]: cluster 2024-03-11T20:54:25.889305+0000 mgr.y (mgr.24380) 141 : cluster 0 pgmap v108: 132 pgs: 132 active+clean; 582 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:54:26.937 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:54:26.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:26.939 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:54:27.040 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:54:27.067 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:54:27.067 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:26 smithi146 bash[23809]: cluster 2024-03-11T20:54:25.837851+0000 osd.5 (osd.5) 83 : cluster 0 2.2 scrub starts 2024-03-11T20:54:27.068 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:26 smithi146 bash[23809]: cluster 2024-03-11T20:54:25.840271+0000 osd.5 (osd.5) 84 : cluster 0 2.2 scrub ok 2024-03-11T20:54:27.068 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:26 smithi146 bash[23809]: cluster 2024-03-11T20:54:25.889305+0000 mgr.y (mgr.24380) 141 : cluster 0 pgmap v108: 132 pgs: 132 active+clean; 582 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:54:27.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:27.069 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:54:27.169 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:54:27.198 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:54:27.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:27.200 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:54:27.301 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:54:27.327 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:54:27.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:27.330 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:54:27.430 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:54:27.459 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:54:27.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:27.461 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:54:27.562 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:54:27.590 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:54:27.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:27.592 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:54:27.693 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:54:27.722 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:54:27.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:27.724 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:54:27.825 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:54:27.851 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:54:27.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:27.853 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:54:27.954 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:54:27.965 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:27 smithi146 bash[23809]: cluster 2024-03-11T20:54:26.464171+0000 osd.3 (osd.3) 117 : cluster 0 5.0 deep-scrub starts 2024-03-11T20:54:27.965 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:27 smithi146 bash[23809]: cluster 2024-03-11T20:54:26.468046+0000 osd.3 (osd.3) 118 : cluster 0 5.0 deep-scrub ok 2024-03-11T20:54:27.965 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:27 smithi146 bash[23809]: cluster 2024-03-11T20:54:26.668191+0000 osd.0 (osd.0) 101 : cluster 0 6.18 scrub starts 2024-03-11T20:54:27.966 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:27 smithi146 bash[23809]: cluster 2024-03-11T20:54:26.670626+0000 osd.0 (osd.0) 102 : cluster 0 6.18 scrub ok 2024-03-11T20:54:27.981 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:54:27.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:27.984 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:54:28.085 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:54:28.111 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:54:28.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:28.113 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:54:28.113 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:27 smithi023 bash[21372]: cluster 2024-03-11T20:54:26.464171+0000 osd.3 (osd.3) 117 : cluster 0 5.0 deep-scrub starts 2024-03-11T20:54:28.113 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:27 smithi023 bash[21372]: cluster 2024-03-11T20:54:26.468046+0000 osd.3 (osd.3) 118 : cluster 0 5.0 deep-scrub ok 2024-03-11T20:54:28.113 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:27 smithi023 bash[21372]: cluster 2024-03-11T20:54:26.668191+0000 osd.0 (osd.0) 101 : cluster 0 6.18 scrub starts 2024-03-11T20:54:28.113 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:27 smithi023 bash[21372]: cluster 2024-03-11T20:54:26.670626+0000 osd.0 (osd.0) 102 : cluster 0 6.18 scrub ok 2024-03-11T20:54:28.114 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:27 smithi023 bash[29146]: cluster 2024-03-11T20:54:26.464171+0000 osd.3 (osd.3) 117 : cluster 0 5.0 deep-scrub starts 2024-03-11T20:54:28.114 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:27 smithi023 bash[29146]: cluster 2024-03-11T20:54:26.468046+0000 osd.3 (osd.3) 118 : cluster 0 5.0 deep-scrub ok 2024-03-11T20:54:28.114 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:27 smithi023 bash[29146]: cluster 2024-03-11T20:54:26.668191+0000 osd.0 (osd.0) 101 : cluster 0 6.18 scrub starts 2024-03-11T20:54:28.114 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:27 smithi023 bash[29146]: cluster 2024-03-11T20:54:26.670626+0000 osd.0 (osd.0) 102 : cluster 0 6.18 scrub ok 2024-03-11T20:54:28.213 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:54:28.234 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:54:28.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:28.237 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:54:28.338 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:54:28.358 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:54:28.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:28.360 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:54:28.461 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:54:28.472 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:54:28 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:20:54:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T20:54:28.489 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:54:28.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:28.491 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:54:28.592 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:54:28.619 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:54:28.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:28.621 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:54:28.722 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:54:28.752 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:54:28.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:28.754 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:54:28.855 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:54:28.866 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:28 smithi146 bash[23809]: cluster 2024-03-11T20:54:27.490059+0000 osd.3 (osd.3) 119 : cluster 0 6.1b scrub starts 2024-03-11T20:54:28.866 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:28 smithi146 bash[23809]: cluster 2024-03-11T20:54:27.491736+0000 osd.3 (osd.3) 120 : cluster 0 6.1b scrub ok 2024-03-11T20:54:28.866 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:28 smithi146 bash[23809]: cluster 2024-03-11T20:54:27.889950+0000 mgr.y (mgr.24380) 142 : cluster 0 pgmap v109: 132 pgs: 132 active+clean; 582 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:54:28.866 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:28 smithi146 bash[23809]: cluster 2024-03-11T20:54:28.582057+0000 osd.6 (osd.6) 59 : cluster 0 2.17 scrub starts 2024-03-11T20:54:28.866 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:28 smithi146 bash[23809]: cluster 2024-03-11T20:54:28.584197+0000 osd.6 (osd.6) 60 : cluster 0 2.17 scrub ok 2024-03-11T20:54:28.882 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:54:28.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:28.884 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:54:28.985 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:54:29.014 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:54:29.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:29.016 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:54:29.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:28 smithi023 bash[29146]: cluster 2024-03-11T20:54:27.490059+0000 osd.3 (osd.3) 119 : cluster 0 6.1b scrub starts 2024-03-11T20:54:29.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:28 smithi023 bash[29146]: cluster 2024-03-11T20:54:27.491736+0000 osd.3 (osd.3) 120 : cluster 0 6.1b scrub ok 2024-03-11T20:54:29.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:28 smithi023 bash[29146]: cluster 2024-03-11T20:54:27.889950+0000 mgr.y (mgr.24380) 142 : cluster 0 pgmap v109: 132 pgs: 132 active+clean; 582 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:54:29.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:28 smithi023 bash[29146]: cluster 2024-03-11T20:54:28.582057+0000 osd.6 (osd.6) 59 : cluster 0 2.17 scrub starts 2024-03-11T20:54:29.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:28 smithi023 bash[29146]: cluster 2024-03-11T20:54:28.584197+0000 osd.6 (osd.6) 60 : cluster 0 2.17 scrub ok 2024-03-11T20:54:29.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:28 smithi023 bash[21372]: cluster 2024-03-11T20:54:27.490059+0000 osd.3 (osd.3) 119 : cluster 0 6.1b scrub starts 2024-03-11T20:54:29.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:28 smithi023 bash[21372]: cluster 2024-03-11T20:54:27.491736+0000 osd.3 (osd.3) 120 : cluster 0 6.1b scrub ok 2024-03-11T20:54:29.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:28 smithi023 bash[21372]: cluster 2024-03-11T20:54:27.889950+0000 mgr.y (mgr.24380) 142 : cluster 0 pgmap v109: 132 pgs: 132 active+clean; 582 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:54:29.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:28 smithi023 bash[21372]: cluster 2024-03-11T20:54:28.582057+0000 osd.6 (osd.6) 59 : cluster 0 2.17 scrub starts 2024-03-11T20:54:29.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:28 smithi023 bash[21372]: cluster 2024-03-11T20:54:28.584197+0000 osd.6 (osd.6) 60 : cluster 0 2.17 scrub ok 2024-03-11T20:54:29.117 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:54:29.143 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:54:29.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:29.146 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:54:29.246 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:54:29.274 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:54:29.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:29.277 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:54:29.377 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:54:29.404 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:54:29.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:29.406 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:54:29.507 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:54:29.537 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:54:29.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:29.540 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:54:29.640 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:54:29.668 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:54:29.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:29.671 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:54:29.771 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:54:29.798 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:54:29.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:29.801 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:54:29.902 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:54:29.913 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:29 smithi023 bash[21372]: cluster 2024-03-11T20:54:28.498622+0000 osd.3 (osd.3) 121 : cluster 0 3.b scrub starts 2024-03-11T20:54:29.913 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:29 smithi023 bash[21372]: cluster 2024-03-11T20:54:28.501481+0000 osd.3 (osd.3) 122 : cluster 0 3.b scrub ok 2024-03-11T20:54:29.913 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:29 smithi023 bash[29146]: cluster 2024-03-11T20:54:28.498622+0000 osd.3 (osd.3) 121 : cluster 0 3.b scrub starts 2024-03-11T20:54:29.913 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:29 smithi023 bash[29146]: cluster 2024-03-11T20:54:28.501481+0000 osd.3 (osd.3) 122 : cluster 0 3.b scrub ok 2024-03-11T20:54:29.926 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:54:29.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:29.929 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:54:30.030 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:54:30.057 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:54:30.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:30.059 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:54:30.161 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:54:30.173 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:29 smithi146 bash[23809]: cluster 2024-03-11T20:54:28.498622+0000 osd.3 (osd.3) 121 : cluster 0 3.b scrub starts 2024-03-11T20:54:30.173 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:29 smithi146 bash[23809]: cluster 2024-03-11T20:54:28.501481+0000 osd.3 (osd.3) 122 : cluster 0 3.b scrub ok 2024-03-11T20:54:30.189 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:54:30.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:30.191 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:54:30.292 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:54:30.320 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:54:30.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:30.322 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:54:30.423 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:54:30.449 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:54:30.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:30.451 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:54:30.551 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:54:30.580 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:54:30.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:30.582 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:54:30.683 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:54:30.711 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:54:30.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:30.714 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:54:30.814 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:54:30.841 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:54:30.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:30.843 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:54:30.944 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:54:30.974 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:54:30.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:30.976 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:54:31.077 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:54:31.090 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:30 smithi023 bash[29146]: cluster 2024-03-11T20:54:29.531526+0000 osd.3 (osd.3) 123 : cluster 0 4.19 scrub starts 2024-03-11T20:54:31.090 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:30 smithi023 bash[29146]: cluster 2024-03-11T20:54:29.532703+0000 osd.3 (osd.3) 124 : cluster 0 4.19 scrub ok 2024-03-11T20:54:31.090 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:30 smithi023 bash[29146]: cluster 2024-03-11T20:54:29.856239+0000 osd.1 (osd.1) 89 : cluster 0 2.9 scrub starts 2024-03-11T20:54:31.090 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:30 smithi023 bash[29146]: cluster 2024-03-11T20:54:29.858247+0000 osd.1 (osd.1) 90 : cluster 0 2.9 scrub ok 2024-03-11T20:54:31.090 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:30 smithi023 bash[29146]: cluster 2024-03-11T20:54:29.891415+0000 mgr.y (mgr.24380) 143 : cluster 0 pgmap v110: 132 pgs: 132 active+clean; 582 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:54:31.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:30 smithi023 bash[21372]: cluster 2024-03-11T20:54:29.531526+0000 osd.3 (osd.3) 123 : cluster 0 4.19 scrub starts 2024-03-11T20:54:31.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:30 smithi023 bash[21372]: cluster 2024-03-11T20:54:29.532703+0000 osd.3 (osd.3) 124 : cluster 0 4.19 scrub ok 2024-03-11T20:54:31.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:30 smithi023 bash[21372]: cluster 2024-03-11T20:54:29.856239+0000 osd.1 (osd.1) 89 : cluster 0 2.9 scrub starts 2024-03-11T20:54:31.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:30 smithi023 bash[21372]: cluster 2024-03-11T20:54:29.858247+0000 osd.1 (osd.1) 90 : cluster 0 2.9 scrub ok 2024-03-11T20:54:31.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:30 smithi023 bash[21372]: cluster 2024-03-11T20:54:29.891415+0000 mgr.y (mgr.24380) 143 : cluster 0 pgmap v110: 132 pgs: 132 active+clean; 582 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:54:31.106 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:54:31.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:31.108 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:54:31.209 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:54:31.221 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:30 smithi146 bash[23809]: cluster 2024-03-11T20:54:29.531526+0000 osd.3 (osd.3) 123 : cluster 0 4.19 scrub starts 2024-03-11T20:54:31.221 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:30 smithi146 bash[23809]: cluster 2024-03-11T20:54:29.532703+0000 osd.3 (osd.3) 124 : cluster 0 4.19 scrub ok 2024-03-11T20:54:31.221 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:30 smithi146 bash[23809]: cluster 2024-03-11T20:54:29.856239+0000 osd.1 (osd.1) 89 : cluster 0 2.9 scrub starts 2024-03-11T20:54:31.221 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:30 smithi146 bash[23809]: cluster 2024-03-11T20:54:29.858247+0000 osd.1 (osd.1) 90 : cluster 0 2.9 scrub ok 2024-03-11T20:54:31.221 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:30 smithi146 bash[23809]: cluster 2024-03-11T20:54:29.891415+0000 mgr.y (mgr.24380) 143 : cluster 0 pgmap v110: 132 pgs: 132 active+clean; 582 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:54:31.237 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:54:31.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:31.240 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:54:31.341 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:54:31.370 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:54:31.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:31.373 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:54:31.473 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:54:31.538 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:54:31.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:31.540 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:54:31.641 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:54:31.668 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:54:31.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:31.670 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:54:31.771 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:54:31.798 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:54:31.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:31.800 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:54:31.901 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:54:31.913 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:31 smithi023 bash[21372]: cluster 2024-03-11T20:54:30.576012+0000 osd.3 (osd.3) 125 : cluster 0 4.14 scrub starts 2024-03-11T20:54:31.913 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:31 smithi023 bash[21372]: cluster 2024-03-11T20:54:30.577952+0000 osd.3 (osd.3) 126 : cluster 0 4.14 scrub ok 2024-03-11T20:54:31.913 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:31 smithi023 bash[21372]: cluster 2024-03-11T20:54:30.675593+0000 osd.0 (osd.0) 103 : cluster 0 3.1 scrub starts 2024-03-11T20:54:31.913 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:31 smithi023 bash[21372]: cluster 2024-03-11T20:54:30.679108+0000 osd.0 (osd.0) 104 : cluster 0 3.1 scrub ok 2024-03-11T20:54:31.913 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:31 smithi023 bash[29146]: cluster 2024-03-11T20:54:30.576012+0000 osd.3 (osd.3) 125 : cluster 0 4.14 scrub starts 2024-03-11T20:54:31.913 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:31 smithi023 bash[29146]: cluster 2024-03-11T20:54:30.577952+0000 osd.3 (osd.3) 126 : cluster 0 4.14 scrub ok 2024-03-11T20:54:31.913 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:31 smithi023 bash[29146]: cluster 2024-03-11T20:54:30.675593+0000 osd.0 (osd.0) 103 : cluster 0 3.1 scrub starts 2024-03-11T20:54:31.913 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:31 smithi023 bash[29146]: cluster 2024-03-11T20:54:30.679108+0000 osd.0 (osd.0) 104 : cluster 0 3.1 scrub ok 2024-03-11T20:54:31.929 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:54:31.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:31.932 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:54:32.032 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:54:32.059 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:54:32.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:32.061 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:54:32.162 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:54:32.191 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:54:32.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:32.194 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:54:32.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:31 smithi146 bash[23809]: cluster 2024-03-11T20:54:30.576012+0000 osd.3 (osd.3) 125 : cluster 0 4.14 scrub starts 2024-03-11T20:54:32.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:31 smithi146 bash[23809]: cluster 2024-03-11T20:54:30.577952+0000 osd.3 (osd.3) 126 : cluster 0 4.14 scrub ok 2024-03-11T20:54:32.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:31 smithi146 bash[23809]: cluster 2024-03-11T20:54:30.675593+0000 osd.0 (osd.0) 103 : cluster 0 3.1 scrub starts 2024-03-11T20:54:32.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:31 smithi146 bash[23809]: cluster 2024-03-11T20:54:30.679108+0000 osd.0 (osd.0) 104 : cluster 0 3.1 scrub ok 2024-03-11T20:54:32.294 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:54:32.316 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:54:32.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:32.318 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:54:32.418 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:54:32.445 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:54:32.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:32.448 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:54:32.548 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:54:32.570 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:54:32.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:32.573 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:54:32.674 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:54:32.700 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:54:32.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:32.702 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:54:32.802 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:54:32.829 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:54:32.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:32.831 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:54:32.932 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:54:32.959 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:54:32.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:32.961 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:54:33.062 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:54:33.084 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:54:33.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:33.086 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:54:33.186 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:54:33.211 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:54:33.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:33.213 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:54:33.313 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:54:33.316 INFO:teuthology.orchestra.run.smithi023.stderr:pool 'unique_pool_0' created 2024-03-11T20:54:33.334 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:33 smithi023 bash[29146]: cluster 2024-03-11T20:54:31.892138+0000 mgr.y (mgr.24380) 144 : cluster 0 pgmap v111: 132 pgs: 132 active+clean; 582 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:54:33.334 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:33 smithi023 bash[29146]: audit 2024-03-11T20:54:32.804851+0000 mon.a (mon.0) 934 : audit 1 from='client.? 172.21.15.23:0/2408002045' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-03-11T20:54:33.335 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:33 smithi023 bash[21372]: cluster 2024-03-11T20:54:31.892138+0000 mgr.y (mgr.24380) 144 : cluster 0 pgmap v111: 132 pgs: 132 active+clean; 582 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:54:33.335 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:33 smithi023 bash[21372]: audit 2024-03-11T20:54:32.804851+0000 mon.a (mon.0) 934 : audit 1 from='client.? 172.21.15.23:0/2408002045' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-03-11T20:54:33.345 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:54:33.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:33.348 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:54:33.449 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:54:33.474 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:54:33.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:33.476 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:54:33.577 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:54:33.588 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:33 smithi146 bash[23809]: cluster 2024-03-11T20:54:31.892138+0000 mgr.y (mgr.24380) 144 : cluster 0 pgmap v111: 132 pgs: 132 active+clean; 582 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:54:33.588 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:33 smithi146 bash[23809]: audit 2024-03-11T20:54:32.804851+0000 mon.a (mon.0) 934 : audit 1 from='client.? 172.21.15.23:0/2408002045' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-03-11T20:54:33.605 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:54:33.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:33.607 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:54:33.708 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:54:33.734 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:54:33.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:33.737 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:54:33.838 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:54:33.866 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:54:33.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:33.868 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:54:33.968 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:54:33.996 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:54:33.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:33.998 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:54:34.099 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:54:34.126 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:54:34.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:34.128 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:54:34.230 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:54:34.257 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:54:34.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:34.259 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:54:34.360 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:54:34.371 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:34 smithi146 bash[23809]: cluster 2024-03-11T20:54:32.670529+0000 osd.0 (osd.0) 105 : cluster 0 6.0 scrub starts 2024-03-11T20:54:34.371 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:34 smithi146 bash[23809]: cluster 2024-03-11T20:54:32.671855+0000 osd.0 (osd.0) 106 : cluster 0 6.0 scrub ok 2024-03-11T20:54:34.371 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:34 smithi146 bash[23809]: audit 2024-03-11T20:54:33.310647+0000 mon.a (mon.0) 935 : audit 1 from='client.? 172.21.15.23:0/2408002045' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-03-11T20:54:34.371 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:34 smithi146 bash[23809]: cluster 2024-03-11T20:54:33.314445+0000 mon.a (mon.0) 936 : cluster 0 osdmap e121: 8 total, 8 up, 8 in 2024-03-11T20:54:34.371 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:34 smithi146 bash[23809]: audit 2024-03-11T20:54:34.254668+0000 mon.a (mon.0) 937 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:54:34.383 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:54:34.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:34.386 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:54:34.486 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:54:34.513 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:54:34.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:34.516 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:54:34.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:34 smithi023 bash[21372]: cluster 2024-03-11T20:54:32.670529+0000 osd.0 (osd.0) 105 : cluster 0 6.0 scrub starts 2024-03-11T20:54:34.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:34 smithi023 bash[21372]: cluster 2024-03-11T20:54:32.671855+0000 osd.0 (osd.0) 106 : cluster 0 6.0 scrub ok 2024-03-11T20:54:34.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:34 smithi023 bash[21372]: audit 2024-03-11T20:54:33.310647+0000 mon.a (mon.0) 935 : audit 1 from='client.? 172.21.15.23:0/2408002045' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-03-11T20:54:34.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:34 smithi023 bash[21372]: cluster 2024-03-11T20:54:33.314445+0000 mon.a (mon.0) 936 : cluster 0 osdmap e121: 8 total, 8 up, 8 in 2024-03-11T20:54:34.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:34 smithi023 bash[21372]: audit 2024-03-11T20:54:34.254668+0000 mon.a (mon.0) 937 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:54:34.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:34 smithi023 bash[29146]: cluster 2024-03-11T20:54:32.670529+0000 osd.0 (osd.0) 105 : cluster 0 6.0 scrub starts 2024-03-11T20:54:34.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:34 smithi023 bash[29146]: cluster 2024-03-11T20:54:32.671855+0000 osd.0 (osd.0) 106 : cluster 0 6.0 scrub ok 2024-03-11T20:54:34.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:34 smithi023 bash[29146]: audit 2024-03-11T20:54:33.310647+0000 mon.a (mon.0) 935 : audit 1 from='client.? 172.21.15.23:0/2408002045' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-03-11T20:54:34.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:34 smithi023 bash[29146]: cluster 2024-03-11T20:54:33.314445+0000 mon.a (mon.0) 936 : cluster 0 osdmap e121: 8 total, 8 up, 8 in 2024-03-11T20:54:34.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:34 smithi023 bash[29146]: audit 2024-03-11T20:54:34.254668+0000 mon.a (mon.0) 937 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:54:34.617 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:54:34.645 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:54:34.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:34.647 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:54:34.748 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:54:34.775 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:54:34.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:34.778 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:54:34.879 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:54:34.906 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:54:34.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:34.908 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:54:35.009 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:54:35.037 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:54:35.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:35.039 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:54:35.140 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:54:35.151 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 20:54:34 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T20:54:35.167 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:54:35.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:35.169 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:54:35.247 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-03-11T20:54:35.269 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:54:35.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:35 smithi023 bash[29146]: cluster 2024-03-11T20:54:33.564756+0000 osd.3 (osd.3) 127 : cluster 0 4.0 scrub starts 2024-03-11T20:54:35.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:35 smithi023 bash[29146]: cluster 2024-03-11T20:54:33.567740+0000 osd.3 (osd.3) 128 : cluster 0 4.0 scrub ok 2024-03-11T20:54:35.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:35 smithi023 bash[29146]: cluster 2024-03-11T20:54:33.809503+0000 osd.2 (osd.2) 63 : cluster 0 2.10 deep-scrub starts 2024-03-11T20:54:35.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:35 smithi023 bash[29146]: cluster 2024-03-11T20:54:33.812267+0000 osd.2 (osd.2) 64 : cluster 0 2.10 deep-scrub ok 2024-03-11T20:54:35.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:35 smithi023 bash[29146]: cluster 2024-03-11T20:54:33.892614+0000 mgr.y (mgr.24380) 145 : cluster 0 pgmap v113: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:54:35.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:35 smithi023 bash[29146]: cluster 2024-03-11T20:54:34.310689+0000 mon.a (mon.0) 938 : cluster 3 Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:54:35.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:35 smithi023 bash[29146]: cluster 2024-03-11T20:54:34.328772+0000 mon.a (mon.0) 939 : cluster 0 osdmap e122: 8 total, 8 up, 8 in 2024-03-11T20:54:35.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:35 smithi023 bash[29146]: cluster 2024-03-11T20:54:34.670533+0000 osd.0 (osd.0) 107 : cluster 0 4.6 scrub starts 2024-03-11T20:54:35.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:35 smithi023 bash[29146]: cluster 2024-03-11T20:54:34.673173+0000 osd.0 (osd.0) 108 : cluster 0 4.6 scrub ok 2024-03-11T20:54:35.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:35 smithi023 bash[21372]: cluster 2024-03-11T20:54:33.564756+0000 osd.3 (osd.3) 127 : cluster 0 4.0 scrub starts 2024-03-11T20:54:35.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:35 smithi023 bash[21372]: cluster 2024-03-11T20:54:33.567740+0000 osd.3 (osd.3) 128 : cluster 0 4.0 scrub ok 2024-03-11T20:54:35.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:35 smithi023 bash[21372]: cluster 2024-03-11T20:54:33.809503+0000 osd.2 (osd.2) 63 : cluster 0 2.10 deep-scrub starts 2024-03-11T20:54:35.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:35 smithi023 bash[21372]: cluster 2024-03-11T20:54:33.812267+0000 osd.2 (osd.2) 64 : cluster 0 2.10 deep-scrub ok 2024-03-11T20:54:35.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:35 smithi023 bash[21372]: cluster 2024-03-11T20:54:33.892614+0000 mgr.y (mgr.24380) 145 : cluster 0 pgmap v113: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:54:35.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:35 smithi023 bash[21372]: cluster 2024-03-11T20:54:34.310689+0000 mon.a (mon.0) 938 : cluster 3 Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:54:35.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:35 smithi023 bash[21372]: cluster 2024-03-11T20:54:34.328772+0000 mon.a (mon.0) 939 : cluster 0 osdmap e122: 8 total, 8 up, 8 in 2024-03-11T20:54:35.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:35 smithi023 bash[21372]: cluster 2024-03-11T20:54:34.670533+0000 osd.0 (osd.0) 107 : cluster 0 4.6 scrub starts 2024-03-11T20:54:35.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:35 smithi023 bash[21372]: cluster 2024-03-11T20:54:34.673173+0000 osd.0 (osd.0) 108 : cluster 0 4.6 scrub ok 2024-03-11T20:54:35.297 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:54:35.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:35.299 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:54:35.400 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:54:35.411 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:35 smithi146 bash[23809]: cluster 2024-03-11T20:54:33.564756+0000 osd.3 (osd.3) 127 : cluster 0 4.0 scrub starts 2024-03-11T20:54:35.412 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:35 smithi146 bash[23809]: cluster 2024-03-11T20:54:33.567740+0000 osd.3 (osd.3) 128 : cluster 0 4.0 scrub ok 2024-03-11T20:54:35.412 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:35 smithi146 bash[23809]: cluster 2024-03-11T20:54:33.809503+0000 osd.2 (osd.2) 63 : cluster 0 2.10 deep-scrub starts 2024-03-11T20:54:35.412 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:35 smithi146 bash[23809]: cluster 2024-03-11T20:54:33.812267+0000 osd.2 (osd.2) 64 : cluster 0 2.10 deep-scrub ok 2024-03-11T20:54:35.412 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:35 smithi146 bash[23809]: cluster 2024-03-11T20:54:33.892614+0000 mgr.y (mgr.24380) 145 : cluster 0 pgmap v113: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:54:35.412 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:35 smithi146 bash[23809]: cluster 2024-03-11T20:54:34.310689+0000 mon.a (mon.0) 938 : cluster 3 Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:54:35.412 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:35 smithi146 bash[23809]: cluster 2024-03-11T20:54:34.328772+0000 mon.a (mon.0) 939 : cluster 0 osdmap e122: 8 total, 8 up, 8 in 2024-03-11T20:54:35.412 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:35 smithi146 bash[23809]: cluster 2024-03-11T20:54:34.670533+0000 osd.0 (osd.0) 107 : cluster 0 4.6 scrub starts 2024-03-11T20:54:35.412 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:35 smithi146 bash[23809]: cluster 2024-03-11T20:54:34.673173+0000 osd.0 (osd.0) 108 : cluster 0 4.6 scrub ok 2024-03-11T20:54:35.427 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:54:35.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:35.429 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:54:35.530 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:54:35.549 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:54:35.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:35.551 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:54:35.652 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:54:35.679 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:54:35.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:35.681 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:54:35.782 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:54:35.810 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:54:35.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:35.811 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:54:35.878 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-11T20:54:35.878 INFO:teuthology.orchestra.run.smithi023.stdout: "ops": [], 2024-03-11T20:54:35.878 INFO:teuthology.orchestra.run.smithi023.stdout: "num_ops": 0 2024-03-11T20:54:35.878 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-11T20:54:35.912 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:54:35.940 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:54:35.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:35.943 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:54:36.044 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:54:36.047 INFO:teuthology.orchestra.run.smithi023.stdout:osd.0: {} 2024-03-11T20:54:36.069 INFO:teuthology.orchestra.run.smithi023.stdout:osd.1: {} 2024-03-11T20:54:36.071 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:54:36.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:36.073 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:54:36.090 INFO:teuthology.orchestra.run.smithi023.stdout:osd.2: {} 2024-03-11T20:54:36.110 INFO:teuthology.orchestra.run.smithi023.stdout:osd.3: {} 2024-03-11T20:54:36.130 INFO:teuthology.orchestra.run.smithi023.stdout:osd.4: {} 2024-03-11T20:54:36.154 INFO:teuthology.orchestra.run.smithi023.stdout:osd.5: {} 2024-03-11T20:54:36.174 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:54:36.177 INFO:teuthology.orchestra.run.smithi023.stdout:osd.6: {} 2024-03-11T20:54:36.194 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:54:36.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:36.196 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:54:36.201 INFO:teuthology.orchestra.run.smithi023.stdout:osd.7: {} 2024-03-11T20:54:36.203 INFO:teuthology.orchestra.run.smithi023.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T20:54:36.203 INFO:teuthology.orchestra.run.smithi023.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T20:54:36.203 INFO:teuthology.orchestra.run.smithi023.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T20:54:36.203 INFO:teuthology.orchestra.run.smithi023.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T20:54:36.203 INFO:teuthology.orchestra.run.smithi023.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T20:54:36.203 INFO:teuthology.orchestra.run.smithi023.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T20:54:36.203 INFO:teuthology.orchestra.run.smithi023.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T20:54:36.203 INFO:teuthology.orchestra.run.smithi023.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T20:54:36.297 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:54:36.308 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:36 smithi146 bash[23809]: cluster 2024-03-11T20:54:34.797158+0000 osd.2 (osd.2) 65 : cluster 0 2.1 scrub starts 2024-03-11T20:54:36.309 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:36 smithi146 bash[23809]: cluster 2024-03-11T20:54:34.798329+0000 osd.2 (osd.2) 66 : cluster 0 2.1 scrub ok 2024-03-11T20:54:36.309 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:36 smithi146 bash[23809]: audit 2024-03-11T20:54:34.976169+0000 mgr.y (mgr.24380) 146 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:54:36.309 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:36 smithi146 bash[23809]: cluster 2024-03-11T20:54:35.326542+0000 mon.a (mon.0) 940 : cluster 0 osdmap e123: 8 total, 8 up, 8 in 2024-03-11T20:54:36.309 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:36 smithi146 bash[23809]: audit 2024-03-11T20:54:35.738356+0000 mon.c (mon.2) 27 : audit 1 from='client.? 172.21.15.23:0/3085882637' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T20:54:36.309 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:36 smithi146 bash[23809]: audit 2024-03-11T20:54:35.738732+0000 mon.a (mon.0) 941 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T20:54:36.309 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:36 smithi146 bash[23809]: cluster 2024-03-11T20:54:35.882233+0000 osd.1 (osd.1) 91 : cluster 0 3.19 scrub starts 2024-03-11T20:54:36.309 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:36 smithi146 bash[23809]: cluster 2024-03-11T20:54:35.884547+0000 osd.1 (osd.1) 92 : cluster 0 3.19 scrub ok 2024-03-11T20:54:36.309 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:36 smithi146 bash[23809]: audit 2024-03-11T20:54:36.030273+0000 mon.c (mon.2) 28 : audit 0 from='client.? 172.21.15.23:0/425452765' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T20:54:36.309 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:36 smithi146 bash[23809]: audit 2024-03-11T20:54:36.134834+0000 mon.a (mon.0) 942 : audit 1 from='client.? 172.21.15.23:0/939868025' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["1"]}]: dispatch 2024-03-11T20:54:36.309 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:36 smithi146 bash[23809]: cluster 2024-03-11T20:54:36.134900+0000 mon.a (mon.0) 943 : cluster 1 Client client.admin marked osd.1 out, while it was still marked up 2024-03-11T20:54:36.325 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:54:36.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:36.328 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:54:36.346 INFO:teuthology.orchestra.run.smithi023.stderr:marked out osd.1. 2024-03-11T20:54:36.351 INFO:teuthology.orchestra.run.smithi023.stderr:noscrub is set 2024-03-11T20:54:36.360 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:36 smithi023 bash[21372]: debug 2024-03-11T20:54:36.129+0000 7f55248dd700 -1 mon.a@0(leader).osd e123 definitely_dead 0 2024-03-11T20:54:36.360 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:36 smithi023 bash[21372]: cluster 2024-03-11T20:54:34.797158+0000 osd.2 (osd.2) 65 : cluster 0 2.1 scrub starts 2024-03-11T20:54:36.360 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:36 smithi023 bash[21372]: cluster 2024-03-11T20:54:34.798329+0000 osd.2 (osd.2) 66 : cluster 0 2.1 scrub ok 2024-03-11T20:54:36.360 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:36 smithi023 bash[21372]: audit 2024-03-11T20:54:34.976169+0000 mgr.y (mgr.24380) 146 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:54:36.360 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:36 smithi023 bash[21372]: cluster 2024-03-11T20:54:35.326542+0000 mon.a (mon.0) 940 : cluster 0 osdmap e123: 8 total, 8 up, 8 in 2024-03-11T20:54:36.360 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:36 smithi023 bash[21372]: audit 2024-03-11T20:54:35.738356+0000 mon.c (mon.2) 27 : audit 1 from='client.? 172.21.15.23:0/3085882637' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T20:54:36.360 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:36 smithi023 bash[21372]: audit 2024-03-11T20:54:35.738732+0000 mon.a (mon.0) 941 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T20:54:36.360 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:36 smithi023 bash[21372]: cluster 2024-03-11T20:54:35.882233+0000 osd.1 (osd.1) 91 : cluster 0 3.19 scrub starts 2024-03-11T20:54:36.360 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:36 smithi023 bash[21372]: cluster 2024-03-11T20:54:35.884547+0000 osd.1 (osd.1) 92 : cluster 0 3.19 scrub ok 2024-03-11T20:54:36.361 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:36 smithi023 bash[21372]: audit 2024-03-11T20:54:36.030273+0000 mon.c (mon.2) 28 : audit 0 from='client.? 172.21.15.23:0/425452765' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T20:54:36.361 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:36 smithi023 bash[21372]: audit 2024-03-11T20:54:36.134834+0000 mon.a (mon.0) 942 : audit 1 from='client.? 172.21.15.23:0/939868025' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["1"]}]: dispatch 2024-03-11T20:54:36.361 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:36 smithi023 bash[21372]: cluster 2024-03-11T20:54:36.134900+0000 mon.a (mon.0) 943 : cluster 1 Client client.admin marked osd.1 out, while it was still marked up 2024-03-11T20:54:36.361 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:36 smithi023 bash[29146]: cluster 2024-03-11T20:54:34.797158+0000 osd.2 (osd.2) 65 : cluster 0 2.1 scrub starts 2024-03-11T20:54:36.361 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:36 smithi023 bash[29146]: cluster 2024-03-11T20:54:34.798329+0000 osd.2 (osd.2) 66 : cluster 0 2.1 scrub ok 2024-03-11T20:54:36.361 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:36 smithi023 bash[29146]: audit 2024-03-11T20:54:34.976169+0000 mgr.y (mgr.24380) 146 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:54:36.361 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:36 smithi023 bash[29146]: cluster 2024-03-11T20:54:35.326542+0000 mon.a (mon.0) 940 : cluster 0 osdmap e123: 8 total, 8 up, 8 in 2024-03-11T20:54:36.361 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:36 smithi023 bash[29146]: audit 2024-03-11T20:54:35.738356+0000 mon.c (mon.2) 27 : audit 1 from='client.? 172.21.15.23:0/3085882637' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T20:54:36.361 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:36 smithi023 bash[29146]: audit 2024-03-11T20:54:35.738732+0000 mon.a (mon.0) 941 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T20:54:36.361 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:36 smithi023 bash[29146]: cluster 2024-03-11T20:54:35.882233+0000 osd.1 (osd.1) 91 : cluster 0 3.19 scrub starts 2024-03-11T20:54:36.361 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:36 smithi023 bash[29146]: cluster 2024-03-11T20:54:35.884547+0000 osd.1 (osd.1) 92 : cluster 0 3.19 scrub ok 2024-03-11T20:54:36.361 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:36 smithi023 bash[29146]: audit 2024-03-11T20:54:36.030273+0000 mon.c (mon.2) 28 : audit 0 from='client.? 172.21.15.23:0/425452765' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T20:54:36.362 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:36 smithi023 bash[29146]: audit 2024-03-11T20:54:36.134834+0000 mon.a (mon.0) 942 : audit 1 from='client.? 172.21.15.23:0/939868025' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["1"]}]: dispatch 2024-03-11T20:54:36.362 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:36 smithi023 bash[29146]: cluster 2024-03-11T20:54:36.134900+0000 mon.a (mon.0) 943 : cluster 1 Client client.admin marked osd.1 out, while it was still marked up 2024-03-11T20:54:36.428 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:54:36.453 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:54:36.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:36.455 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:54:36.556 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:54:36.583 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:54:36.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:36.585 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:54:36.686 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:54:36.713 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:54:36.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:36.715 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:54:36.816 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:54:36.843 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:54:36.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:36.846 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:54:36.947 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:54:36.977 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:54:36.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:36.979 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:54:37.080 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:54:37.109 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:54:37.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:37.111 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:54:37.212 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:54:37.241 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:54:37.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:37.244 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:54:37.251 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-03-11T20:54:37.345 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:54:37.372 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:54:37.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:37.374 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:54:37.400 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:37 smithi023 bash[29146]: cluster 2024-03-11T20:54:35.757930+0000 osd.2 (osd.2) 67 : cluster 0 4.1c scrub starts 2024-03-11T20:54:37.400 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:37 smithi023 bash[29146]: cluster 2024-03-11T20:54:35.759030+0000 osd.2 (osd.2) 68 : cluster 0 4.1c scrub ok 2024-03-11T20:54:37.400 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:37 smithi023 bash[29146]: cluster 2024-03-11T20:54:35.893100+0000 mgr.y (mgr.24380) 147 : cluster 0 pgmap v116: 148 pgs: 148 active+clean; 582 KiB data, 256 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T20:54:37.400 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:37 smithi023 bash[29146]: cluster 2024-03-11T20:54:36.323021+0000 mon.a (mon.0) 944 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T20:54:37.401 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:37 smithi023 bash[29146]: audit 2024-03-11T20:54:36.327110+0000 mon.a (mon.0) 945 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T20:54:37.401 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:37 smithi023 bash[29146]: audit 2024-03-11T20:54:36.327163+0000 mon.a (mon.0) 946 : audit 1 from='client.? 172.21.15.23:0/939868025' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["1"]}]': finished 2024-03-11T20:54:37.401 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:37 smithi023 bash[29146]: cluster 2024-03-11T20:54:36.330364+0000 mon.a (mon.0) 947 : cluster 0 osdmap e124: 8 total, 8 up, 7 in 2024-03-11T20:54:37.401 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:37 smithi023 bash[29146]: cluster 2024-03-11T20:54:36.774313+0000 osd.2 (osd.2) 69 : cluster 0 purged_snaps scrub starts 2024-03-11T20:54:37.401 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:37 smithi023 bash[29146]: cluster 2024-03-11T20:54:36.774420+0000 osd.2 (osd.2) 70 : cluster 0 purged_snaps scrub ok 2024-03-11T20:54:37.401 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:37 smithi023 bash[21372]: cluster 2024-03-11T20:54:35.757930+0000 osd.2 (osd.2) 67 : cluster 0 4.1c scrub starts 2024-03-11T20:54:37.401 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:37 smithi023 bash[21372]: cluster 2024-03-11T20:54:35.759030+0000 osd.2 (osd.2) 68 : cluster 0 4.1c scrub ok 2024-03-11T20:54:37.401 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:37 smithi023 bash[21372]: cluster 2024-03-11T20:54:35.893100+0000 mgr.y (mgr.24380) 147 : cluster 0 pgmap v116: 148 pgs: 148 active+clean; 582 KiB data, 256 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T20:54:37.401 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:37 smithi023 bash[21372]: cluster 2024-03-11T20:54:36.323021+0000 mon.a (mon.0) 944 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T20:54:37.401 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:37 smithi023 bash[21372]: audit 2024-03-11T20:54:36.327110+0000 mon.a (mon.0) 945 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T20:54:37.402 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:37 smithi023 bash[21372]: audit 2024-03-11T20:54:36.327163+0000 mon.a (mon.0) 946 : audit 1 from='client.? 172.21.15.23:0/939868025' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["1"]}]': finished 2024-03-11T20:54:37.402 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:37 smithi023 bash[21372]: cluster 2024-03-11T20:54:36.330364+0000 mon.a (mon.0) 947 : cluster 0 osdmap e124: 8 total, 8 up, 7 in 2024-03-11T20:54:37.402 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:37 smithi023 bash[21372]: cluster 2024-03-11T20:54:36.774313+0000 osd.2 (osd.2) 69 : cluster 0 purged_snaps scrub starts 2024-03-11T20:54:37.402 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:37 smithi023 bash[21372]: cluster 2024-03-11T20:54:36.774420+0000 osd.2 (osd.2) 70 : cluster 0 purged_snaps scrub ok 2024-03-11T20:54:37.476 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:54:37.495 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:54:37.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:37.497 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:54:37.598 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:54:37.610 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:37 smithi146 bash[23809]: cluster 2024-03-11T20:54:35.757930+0000 osd.2 (osd.2) 67 : cluster 0 4.1c scrub starts 2024-03-11T20:54:37.610 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:37 smithi146 bash[23809]: cluster 2024-03-11T20:54:35.759030+0000 osd.2 (osd.2) 68 : cluster 0 4.1c scrub ok 2024-03-11T20:54:37.610 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:37 smithi146 bash[23809]: cluster 2024-03-11T20:54:35.893100+0000 mgr.y (mgr.24380) 147 : cluster 0 pgmap v116: 148 pgs: 148 active+clean; 582 KiB data, 256 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T20:54:37.610 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:37 smithi146 bash[23809]: cluster 2024-03-11T20:54:36.323021+0000 mon.a (mon.0) 944 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T20:54:37.610 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:37 smithi146 bash[23809]: audit 2024-03-11T20:54:36.327110+0000 mon.a (mon.0) 945 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T20:54:37.610 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:37 smithi146 bash[23809]: audit 2024-03-11T20:54:36.327163+0000 mon.a (mon.0) 946 : audit 1 from='client.? 172.21.15.23:0/939868025' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["1"]}]': finished 2024-03-11T20:54:37.610 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:37 smithi146 bash[23809]: cluster 2024-03-11T20:54:36.330364+0000 mon.a (mon.0) 947 : cluster 0 osdmap e124: 8 total, 8 up, 7 in 2024-03-11T20:54:37.610 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:37 smithi146 bash[23809]: cluster 2024-03-11T20:54:36.774313+0000 osd.2 (osd.2) 69 : cluster 0 purged_snaps scrub starts 2024-03-11T20:54:37.610 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:37 smithi146 bash[23809]: cluster 2024-03-11T20:54:36.774420+0000 osd.2 (osd.2) 70 : cluster 0 purged_snaps scrub ok 2024-03-11T20:54:37.626 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:54:37.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:37.628 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:54:37.729 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:54:37.757 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:54:37.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:37.764 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:54:37.865 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:54:37.890 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:54:37.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:37.893 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:54:37.994 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:54:38.022 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:54:38.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:38.024 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:54:38.125 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:54:38.152 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:54:38.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:38.155 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:54:38.256 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:54:38.285 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:54:38.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:38.287 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:54:38.388 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:54:38.404 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:38 smithi023 bash[21372]: cluster 2024-03-11T20:54:37.055688+0000 osd.4 (osd.4) 103 : cluster 0 purged_snaps scrub starts 2024-03-11T20:54:38.405 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:38 smithi023 bash[21372]: cluster 2024-03-11T20:54:37.055858+0000 osd.4 (osd.4) 104 : cluster 0 purged_snaps scrub ok 2024-03-11T20:54:38.405 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:38 smithi023 bash[21372]: cluster 2024-03-11T20:54:37.366955+0000 mon.a (mon.0) 948 : cluster 0 osdmap e125: 8 total, 8 up, 7 in 2024-03-11T20:54:38.413 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:54:38.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:38.415 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:54:38.516 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:54:38.534 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:54:38 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:20:54:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T20:54:38.534 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:38 smithi023 bash[29146]: cluster 2024-03-11T20:54:37.055688+0000 osd.4 (osd.4) 103 : cluster 0 purged_snaps scrub starts 2024-03-11T20:54:38.534 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:38 smithi023 bash[29146]: cluster 2024-03-11T20:54:37.055858+0000 osd.4 (osd.4) 104 : cluster 0 purged_snaps scrub ok 2024-03-11T20:54:38.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:38 smithi023 bash[29146]: cluster 2024-03-11T20:54:37.366955+0000 mon.a (mon.0) 948 : cluster 0 osdmap e125: 8 total, 8 up, 7 in 2024-03-11T20:54:38.542 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:54:38.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:38.544 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:54:38.644 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:54:38.671 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:54:38.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:38.673 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:54:38.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:38 smithi146 bash[23809]: cluster 2024-03-11T20:54:37.055688+0000 osd.4 (osd.4) 103 : cluster 0 purged_snaps scrub starts 2024-03-11T20:54:38.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:38 smithi146 bash[23809]: cluster 2024-03-11T20:54:37.055858+0000 osd.4 (osd.4) 104 : cluster 0 purged_snaps scrub ok 2024-03-11T20:54:38.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:38 smithi146 bash[23809]: cluster 2024-03-11T20:54:37.366955+0000 mon.a (mon.0) 948 : cluster 0 osdmap e125: 8 total, 8 up, 7 in 2024-03-11T20:54:38.774 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:54:38.801 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:54:38.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:38.803 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:54:38.905 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:54:38.931 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:54:38.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:38.933 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:54:39.033 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:54:39.057 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:54:39.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:39.059 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:54:39.160 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:54:39.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:39.191 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:54:39.191 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:54:39.292 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:54:39.319 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:54:39.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:39.321 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:54:39.422 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:54:39.447 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:54:39.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:39.449 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:54:39.550 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:54:39.561 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:39 smithi146 bash[23809]: cluster 2024-03-11T20:54:37.894029+0000 mgr.y (mgr.24380) 148 : cluster 0 pgmap v119: 148 pgs: 148 active+clean; 582 KiB data, 224 MiB used, 626 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:54:39.576 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:54:39.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:39.578 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:54:39.679 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:54:39.704 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:54:39.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:39.707 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:54:39.784 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:39 smithi023 bash[29146]: cluster 2024-03-11T20:54:37.894029+0000 mgr.y (mgr.24380) 148 : cluster 0 pgmap v119: 148 pgs: 148 active+clean; 582 KiB data, 224 MiB used, 626 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:54:39.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:39 smithi023 bash[21372]: cluster 2024-03-11T20:54:37.894029+0000 mgr.y (mgr.24380) 148 : cluster 0 pgmap v119: 148 pgs: 148 active+clean; 582 KiB data, 224 MiB used, 626 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:54:39.808 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:54:39.838 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:54:39.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:39.841 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:54:39.941 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:54:39.970 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:54:39.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:39.972 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:54:40.073 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:54:40.102 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:54:40.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:40.104 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:54:40.205 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:54:40.233 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:54:40.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:40.235 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:54:40.336 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:54:40.363 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:54:40.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:40.365 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:54:40.466 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:54:40.488 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:54:40.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:40.490 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:54:40.591 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:54:40.618 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:54:40.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:40.624 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:54:40.724 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T20:54:40.725 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:54:40.751 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:54:40.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:40.753 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:54:40.854 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:54:40.879 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:54:40.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:40.882 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:54:40.882 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-11T20:54:40.983 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:54:41.010 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:54:41.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:41.012 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:54:41.113 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:54:41.134 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:54:41.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:41.136 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:54:41.237 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:54:41.265 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:54:41.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:41.267 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:54:41.368 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:54:41.379 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:41 smithi023 bash[21372]: cluster 2024-03-11T20:54:39.895685+0000 mgr.y (mgr.24380) 149 : cluster 0 pgmap v120: 148 pgs: 148 active+clean; 582 KiB data, 233 MiB used, 626 GiB / 626 GiB avail; 1.4 KiB/s rd, 1 op/s; 503 B/s, 13 objects/s recovering 2024-03-11T20:54:41.379 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:41 smithi023 bash[29146]: cluster 2024-03-11T20:54:39.895685+0000 mgr.y (mgr.24380) 149 : cluster 0 pgmap v120: 148 pgs: 148 active+clean; 582 KiB data, 233 MiB used, 626 GiB / 626 GiB avail; 1.4 KiB/s rd, 1 op/s; 503 B/s, 13 objects/s recovering 2024-03-11T20:54:41.394 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:54:41.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:41.397 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:54:41.498 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:54:41.520 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:54:41.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:41.523 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:54:41.624 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:54:41.635 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:41 smithi146 bash[23809]: cluster 2024-03-11T20:54:39.895685+0000 mgr.y (mgr.24380) 149 : cluster 0 pgmap v120: 148 pgs: 148 active+clean; 582 KiB data, 233 MiB used, 626 GiB / 626 GiB avail; 1.4 KiB/s rd, 1 op/s; 503 B/s, 13 objects/s recovering 2024-03-11T20:54:41.650 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:54:41.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:41.652 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:54:41.753 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:54:41.790 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:54:41.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:41.792 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:54:41.893 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:54:41.922 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:54:41.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:41.924 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:54:42.025 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:54:42.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:42.056 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:54:42.056 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:54:42.157 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:54:42.183 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:54:42.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:42.185 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:54:42.286 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:54:42.314 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:54:42.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:42.316 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:54:42.417 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:54:42.445 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:54:42.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:42.448 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:54:42.451 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:54:42.549 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:54:42.571 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:54:42.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:42.573 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:54:42.674 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:54:42.701 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:54:42.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:42.704 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:54:42.804 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:54:42.830 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:54:42.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:42.832 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:54:42.933 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:54:42.961 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:54:42.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:42.963 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:54:43.063 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:54:43.086 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:54:43.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:43.088 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:54:43.189 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:54:43.217 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:54:43.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:43.219 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:54:43.320 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:54:43.344 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:54:43.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:43.346 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:54:43.447 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:54:43.468 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:54:43.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:43.471 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:54:43.471 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:54:43.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:43 smithi146 bash[23809]: cluster 2024-03-11T20:54:41.896247+0000 mgr.y (mgr.24380) 150 : cluster 0 pgmap v121: 148 pgs: 148 active+clean; 582 KiB data, 233 MiB used, 626 GiB / 626 GiB avail; 467 B/s rd, 0 op/s; 426 B/s, 11 objects/s recovering 2024-03-11T20:54:43.572 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:54:43.600 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:54:43.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:43.602 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:54:43.703 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:54:43.714 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:43 smithi023 bash[21372]: cluster 2024-03-11T20:54:41.896247+0000 mgr.y (mgr.24380) 150 : cluster 0 pgmap v121: 148 pgs: 148 active+clean; 582 KiB data, 233 MiB used, 626 GiB / 626 GiB avail; 467 B/s rd, 0 op/s; 426 B/s, 11 objects/s recovering 2024-03-11T20:54:43.715 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:43 smithi023 bash[29146]: cluster 2024-03-11T20:54:41.896247+0000 mgr.y (mgr.24380) 150 : cluster 0 pgmap v121: 148 pgs: 148 active+clean; 582 KiB data, 233 MiB used, 626 GiB / 626 GiB avail; 467 B/s rd, 0 op/s; 426 B/s, 11 objects/s recovering 2024-03-11T20:54:43.732 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:54:43.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:43.734 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:54:43.835 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:54:43.864 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:54:43.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:43.866 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:54:43.936 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 6, 7] out_osds: [1] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-11T20:54:43.936 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-11T20:54:43.968 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:54:43.994 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:54:43.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:43.996 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:54:44.097 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:54:44.126 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:54:44.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:44.128 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:54:44.229 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:54:44.256 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:54:44.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:44.259 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:54:44.359 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:54:44.386 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:54:44.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:44.388 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:54:44.489 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:54:44.517 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:54:44.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:44.519 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:54:44.620 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:54:44.649 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:54:44.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:44.651 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:54:44.752 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:54:44.779 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:54:44.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:44.781 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:54:44.882 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:54:44.904 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:54:44.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:44.906 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:54:45.007 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:54:45.018 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 20:54:44 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T20:54:45.034 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:54:45.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:45.036 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:54:45.136 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:54:45.157 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:54:45.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:45.160 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:54:45.260 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:54:45.277 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:54:45.287 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:54:45.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:45.290 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:54:45.391 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:54:45.402 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:45 smithi023 bash[21372]: cluster 2024-03-11T20:54:43.896908+0000 mgr.y (mgr.24380) 151 : cluster 0 pgmap v122: 148 pgs: 148 active+clean; 582 KiB data, 233 MiB used, 626 GiB / 626 GiB avail; 383 B/s rd, 0 op/s; 350 B/s, 9 objects/s recovering 2024-03-11T20:54:45.402 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:45 smithi023 bash[21372]: audit 2024-03-11T20:54:45.221296+0000 mon.a (mon.0) 949 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:54:45.403 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:45 smithi023 bash[29146]: cluster 2024-03-11T20:54:43.896908+0000 mgr.y (mgr.24380) 151 : cluster 0 pgmap v122: 148 pgs: 148 active+clean; 582 KiB data, 233 MiB used, 626 GiB / 626 GiB avail; 383 B/s rd, 0 op/s; 350 B/s, 9 objects/s recovering 2024-03-11T20:54:45.403 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:45 smithi023 bash[29146]: audit 2024-03-11T20:54:45.221296+0000 mon.a (mon.0) 949 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:54:45.416 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:54:45.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:45.419 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:54:45.510 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:54:45.520 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:54:45.540 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:54:45.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:45.542 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:54:45.643 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:54:45.651 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:45 smithi146 bash[23809]: cluster 2024-03-11T20:54:43.896908+0000 mgr.y (mgr.24380) 151 : cluster 0 pgmap v122: 148 pgs: 148 active+clean; 582 KiB data, 233 MiB used, 626 GiB / 626 GiB avail; 383 B/s rd, 0 op/s; 350 B/s, 9 objects/s recovering 2024-03-11T20:54:45.651 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:45 smithi146 bash[23809]: audit 2024-03-11T20:54:45.221296+0000 mon.a (mon.0) 949 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:54:45.662 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:54:45.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:45.664 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:54:45.765 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:54:45.786 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:54:45.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:45.788 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:54:45.889 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:54:45.917 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:54:45.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:45.919 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:54:46.020 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:54:46.038 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:54:46.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:46.040 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:54:46.141 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:54:46.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:46.171 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:54:46.171 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:54:46.272 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:54:46.299 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:54:46.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:46.301 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:54:46.402 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:54:46.431 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:54:46.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:46.434 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:54:46.535 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:54:46.563 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:54:46.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:46.566 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:54:46.666 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:54:46.678 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:46 smithi023 bash[21372]: audit 2024-03-11T20:54:44.987569+0000 mgr.y (mgr.24380) 152 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:54:46.678 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:46 smithi023 bash[21372]: cluster 2024-03-11T20:54:45.478604+0000 osd.5 (osd.5) 85 : cluster 0 purged_snaps scrub starts 2024-03-11T20:54:46.678 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:46 smithi023 bash[21372]: cluster 2024-03-11T20:54:45.478743+0000 osd.5 (osd.5) 86 : cluster 0 purged_snaps scrub ok 2024-03-11T20:54:46.678 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:46 smithi023 bash[21372]: audit 2024-03-11T20:54:45.902814+0000 mon.a (mon.0) 950 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:54:46.678 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:46 smithi023 bash[21372]: audit 2024-03-11T20:54:45.904139+0000 mon.a (mon.0) 951 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:54:46.678 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:46 smithi023 bash[21372]: audit 2024-03-11T20:54:45.914490+0000 mon.a (mon.0) 952 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:54:46.679 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:46 smithi023 bash[29146]: audit 2024-03-11T20:54:44.987569+0000 mgr.y (mgr.24380) 152 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:54:46.679 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:46 smithi023 bash[29146]: cluster 2024-03-11T20:54:45.478604+0000 osd.5 (osd.5) 85 : cluster 0 purged_snaps scrub starts 2024-03-11T20:54:46.679 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:46 smithi023 bash[29146]: cluster 2024-03-11T20:54:45.478743+0000 osd.5 (osd.5) 86 : cluster 0 purged_snaps scrub ok 2024-03-11T20:54:46.679 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:46 smithi023 bash[29146]: audit 2024-03-11T20:54:45.902814+0000 mon.a (mon.0) 950 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:54:46.679 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:46 smithi023 bash[29146]: audit 2024-03-11T20:54:45.904139+0000 mon.a (mon.0) 951 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:54:46.679 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:46 smithi023 bash[29146]: audit 2024-03-11T20:54:45.914490+0000 mon.a (mon.0) 952 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:54:46.691 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:54:46.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:46.694 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:54:46.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:46 smithi146 bash[23809]: audit 2024-03-11T20:54:44.987569+0000 mgr.y (mgr.24380) 152 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:54:46.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:46 smithi146 bash[23809]: cluster 2024-03-11T20:54:45.478604+0000 osd.5 (osd.5) 85 : cluster 0 purged_snaps scrub starts 2024-03-11T20:54:46.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:46 smithi146 bash[23809]: cluster 2024-03-11T20:54:45.478743+0000 osd.5 (osd.5) 86 : cluster 0 purged_snaps scrub ok 2024-03-11T20:54:46.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:46 smithi146 bash[23809]: audit 2024-03-11T20:54:45.902814+0000 mon.a (mon.0) 950 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:54:46.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:46 smithi146 bash[23809]: audit 2024-03-11T20:54:45.904139+0000 mon.a (mon.0) 951 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:54:46.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:46 smithi146 bash[23809]: audit 2024-03-11T20:54:45.914490+0000 mon.a (mon.0) 952 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:54:46.794 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:54:46.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:46.831 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:54:46.831 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:54:46.932 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:54:46.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:46.965 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:54:46.966 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:54:47.070 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:54:47.098 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:54:47.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:47.100 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:54:47.201 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:54:47.229 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:54:47.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:47.232 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:54:47.333 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:54:47.356 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:54:47.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:47.359 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:54:47.459 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:54:47.486 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:54:47.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:47.489 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:54:47.543 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:47 smithi023 bash[21372]: cluster 2024-03-11T20:54:45.898086+0000 mgr.y (mgr.24380) 153 : cluster 0 pgmap v123: 148 pgs: 148 active+clean; 582 KiB data, 233 MiB used, 626 GiB / 626 GiB avail; 856 B/s rd, 0 op/s; 293 B/s, 8 objects/s recovering 2024-03-11T20:54:47.543 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:47 smithi023 bash[21372]: cluster 2024-03-11T20:54:46.429651+0000 mon.a (mon.0) 953 : cluster 0 osdmap e126: 8 total, 8 up, 7 in 2024-03-11T20:54:47.543 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:47 smithi023 bash[29146]: cluster 2024-03-11T20:54:45.898086+0000 mgr.y (mgr.24380) 153 : cluster 0 pgmap v123: 148 pgs: 148 active+clean; 582 KiB data, 233 MiB used, 626 GiB / 626 GiB avail; 856 B/s rd, 0 op/s; 293 B/s, 8 objects/s recovering 2024-03-11T20:54:47.543 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:47 smithi023 bash[29146]: cluster 2024-03-11T20:54:46.429651+0000 mon.a (mon.0) 953 : cluster 0 osdmap e126: 8 total, 8 up, 7 in 2024-03-11T20:54:47.589 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:54:47.617 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:54:47.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:47.619 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:54:47.720 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:54:47.731 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:47 smithi146 bash[23809]: cluster 2024-03-11T20:54:45.898086+0000 mgr.y (mgr.24380) 153 : cluster 0 pgmap v123: 148 pgs: 148 active+clean; 582 KiB data, 233 MiB used, 626 GiB / 626 GiB avail; 856 B/s rd, 0 op/s; 293 B/s, 8 objects/s recovering 2024-03-11T20:54:47.731 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:47 smithi146 bash[23809]: cluster 2024-03-11T20:54:46.429651+0000 mon.a (mon.0) 953 : cluster 0 osdmap e126: 8 total, 8 up, 7 in 2024-03-11T20:54:47.746 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:54:47.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:47.748 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:54:47.849 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:54:47.873 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:54:47.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:47.876 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:54:47.977 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:54:48.005 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:54:48.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:48.007 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:54:48.108 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:54:48.134 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:54:48.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:48.136 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:54:48.237 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:54:48.264 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:54:48.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:48.266 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:54:48.367 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:54:48.393 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:54:48.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:48.396 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:54:48.426 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:54:48 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:20:54:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T20:54:48.444 INFO:teuthology.orchestra.run.smithi023.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-03-11T20:54:48.496 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:54:48.516 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:54:48.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:48.519 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:54:48.620 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:54:48.631 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:48 smithi146 bash[23809]: audit 2024-03-11T20:54:48.026371+0000 mon.a (mon.0) 954 : audit 1 from='client.? 172.21.15.23:0/4118831800' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:48.646 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:54:48.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:48.648 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:54:48.749 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:54:48.760 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:48 smithi023 bash[21372]: audit 2024-03-11T20:54:48.026371+0000 mon.a (mon.0) 954 : audit 1 from='client.? 172.21.15.23:0/4118831800' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:48.761 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:48 smithi023 bash[29146]: audit 2024-03-11T20:54:48.026371+0000 mon.a (mon.0) 954 : audit 1 from='client.? 172.21.15.23:0/4118831800' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:48.776 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:54:48.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:48.778 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:54:48.879 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:54:48.905 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:54:48.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:48.908 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:54:49.009 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:54:49.038 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:54:49.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:49.040 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:54:49.141 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:54:49.160 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:54:49.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:49.162 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:54:49.264 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:54:49.284 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:54:49.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:49.287 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:54:49.388 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:54:49.415 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:54:49.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:49.417 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:54:49.518 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:54:49.527 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:49 smithi023 bash[29146]: cluster 2024-03-11T20:54:47.898469+0000 mgr.y (mgr.24380) 154 : cluster 0 pgmap v125: 148 pgs: 148 active+clean; 582 KiB data, 233 MiB used, 626 GiB / 626 GiB avail; 818 B/s rd, 0 op/s; 280 B/s, 7 objects/s recovering 2024-03-11T20:54:49.527 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:49 smithi023 bash[29146]: audit 2024-03-11T20:54:48.442753+0000 mon.a (mon.0) 955 : audit 1 from='client.? 172.21.15.23:0/4118831800' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:49.527 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:49 smithi023 bash[29146]: cluster 2024-03-11T20:54:48.450768+0000 mon.a (mon.0) 956 : cluster 0 osdmap e127: 8 total, 8 up, 7 in 2024-03-11T20:54:49.527 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:49 smithi023 bash[29146]: audit 2024-03-11T20:54:49.296993+0000 mon.a (mon.0) 957 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:54:49.527 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:49 smithi023 bash[29146]: audit 2024-03-11T20:54:49.298327+0000 mon.a (mon.0) 958 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:54:49.528 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:49 smithi023 bash[21372]: cluster 2024-03-11T20:54:47.898469+0000 mgr.y (mgr.24380) 154 : cluster 0 pgmap v125: 148 pgs: 148 active+clean; 582 KiB data, 233 MiB used, 626 GiB / 626 GiB avail; 818 B/s rd, 0 op/s; 280 B/s, 7 objects/s recovering 2024-03-11T20:54:49.528 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:49 smithi023 bash[21372]: audit 2024-03-11T20:54:48.442753+0000 mon.a (mon.0) 955 : audit 1 from='client.? 172.21.15.23:0/4118831800' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:49.528 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:49 smithi023 bash[21372]: cluster 2024-03-11T20:54:48.450768+0000 mon.a (mon.0) 956 : cluster 0 osdmap e127: 8 total, 8 up, 7 in 2024-03-11T20:54:49.528 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:49 smithi023 bash[21372]: audit 2024-03-11T20:54:49.296993+0000 mon.a (mon.0) 957 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:54:49.528 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:49 smithi023 bash[21372]: audit 2024-03-11T20:54:49.298327+0000 mon.a (mon.0) 958 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:54:49.542 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:54:49.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:49.544 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:54:49.645 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:54:49.659 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:49 smithi146 bash[23809]: cluster 2024-03-11T20:54:47.898469+0000 mgr.y (mgr.24380) 154 : cluster 0 pgmap v125: 148 pgs: 148 active+clean; 582 KiB data, 233 MiB used, 626 GiB / 626 GiB avail; 818 B/s rd, 0 op/s; 280 B/s, 7 objects/s recovering 2024-03-11T20:54:49.659 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:49 smithi146 bash[23809]: audit 2024-03-11T20:54:48.442753+0000 mon.a (mon.0) 955 : audit 1 from='client.? 172.21.15.23:0/4118831800' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:49.659 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:49 smithi146 bash[23809]: cluster 2024-03-11T20:54:48.450768+0000 mon.a (mon.0) 956 : cluster 0 osdmap e127: 8 total, 8 up, 7 in 2024-03-11T20:54:49.659 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:49 smithi146 bash[23809]: audit 2024-03-11T20:54:49.296993+0000 mon.a (mon.0) 957 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:54:49.659 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:49 smithi146 bash[23809]: audit 2024-03-11T20:54:49.298327+0000 mon.a (mon.0) 958 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:54:49.675 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:54:49.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:49.677 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:54:49.778 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:54:49.805 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:54:49.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:49.808 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:54:49.908 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:54:49.936 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:54:49.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:49.939 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:54:50.039 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:54:50.066 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:54:50.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:50.069 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:54:50.169 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:54:50.197 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:54:50.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:50.199 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:54:50.300 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:54:50.329 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:54:50.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:50.332 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:54:50.432 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:54:50.461 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:54:50.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:50.463 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:54:50.564 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:54:50.592 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:54:50.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:50.594 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:54:50.695 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:54:50.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:50 smithi146 bash[23809]: cluster 2024-03-11T20:54:49.657339+0000 mon.a (mon.0) 959 : cluster 0 osdmap e128: 8 total, 8 up, 7 in 2024-03-11T20:54:50.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:50 smithi146 bash[23809]: cluster 2024-03-11T20:54:49.899908+0000 mgr.y (mgr.24380) 155 : cluster 0 pgmap v128: 148 pgs: 148 active+clean; 582 KiB data, 233 MiB used, 626 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-11T20:54:50.723 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:54:50.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:50.725 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:54:50.767 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:50 smithi023 bash[21372]: cluster 2024-03-11T20:54:49.657339+0000 mon.a (mon.0) 959 : cluster 0 osdmap e128: 8 total, 8 up, 7 in 2024-03-11T20:54:50.767 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:50 smithi023 bash[21372]: cluster 2024-03-11T20:54:49.899908+0000 mgr.y (mgr.24380) 155 : cluster 0 pgmap v128: 148 pgs: 148 active+clean; 582 KiB data, 233 MiB used, 626 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-11T20:54:50.826 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:54:50.853 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:54:50.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:50.856 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:54:50.887 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:50 smithi023 bash[29146]: cluster 2024-03-11T20:54:49.657339+0000 mon.a (mon.0) 959 : cluster 0 osdmap e128: 8 total, 8 up, 7 in 2024-03-11T20:54:50.887 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:50 smithi023 bash[29146]: cluster 2024-03-11T20:54:49.899908+0000 mgr.y (mgr.24380) 155 : cluster 0 pgmap v128: 148 pgs: 148 active+clean; 582 KiB data, 233 MiB used, 626 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-11T20:54:50.956 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:54:50.981 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:54:50.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:50.983 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:54:51.084 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:54:51.111 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:54:51.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:51.113 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:54:51.214 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:54:51.240 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:54:51.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:51.243 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:54:51.344 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:54:51.371 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:54:51.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:51.373 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:54:51.471 DEBUG:teuthology.orchestra.run.smithi023:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_0 bench 90 write' 2024-03-11T20:54:51.472 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-03-11T20:54:51.472 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T20:54:51.473 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:54:51.475 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-03-11T20:54:51.475 INFO:teuthology.orchestra.run:waiting for 3000 2024-03-11T20:54:51.501 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:54:51.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:51.504 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:54:51.533 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:hints = 1 2024-03-11T20:54:51.534 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-03-11T20:54:51.534 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:Object prefix: benchmark_data_smithi023_67959 2024-03-11T20:54:51.534 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-11T20:54:51.534 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 0 0 0 0 0 0 - 0 2024-03-11T20:54:51.605 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:54:51.655 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:54:51.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:51.662 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:54:51.763 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:54:51.764 INFO:teuthology.orchestra.run.smithi023.stdout:ERROR: (22) Invalid argument 2024-03-11T20:54:51.764 INFO:teuthology.orchestra.run.smithi023.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T20:54:51.776 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:51 smithi023 bash[21372]: cluster 2024-03-11T20:54:50.665263+0000 mon.a (mon.0) 960 : cluster 1 Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-11T20:54:51.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:51 smithi023 bash[29146]: cluster 2024-03-11T20:54:50.665263+0000 mon.a (mon.0) 960 : cluster 1 Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-11T20:54:51.802 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:54:51.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:51.803 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:54:51.904 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:54:51.916 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:51 smithi146 bash[23809]: cluster 2024-03-11T20:54:50.665263+0000 mon.a (mon.0) 960 : cluster 1 Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-11T20:54:51.934 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:54:51.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:51.937 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:54:52.038 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:54:52.069 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:54:52.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:52.074 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:54:52.175 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:54:52.203 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:54:52.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:52.206 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:54:52.306 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:54:52.333 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:54:52.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:52.336 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:54:52.437 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:54:52.466 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:54:52.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:52.469 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:54:52.534 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 1 15 2731 2716 169.741 169.75 0.0045663 0.00585431 2024-03-11T20:54:52.570 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:54:52.595 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:54:52.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:52.598 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:54:52.698 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:54:52.710 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:52 smithi023 bash[21372]: cluster 2024-03-11T20:54:50.823983+0000 osd.6 (osd.6) 61 : cluster 0 purged_snaps scrub starts 2024-03-11T20:54:52.710 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:52 smithi023 bash[21372]: cluster 2024-03-11T20:54:50.824100+0000 osd.6 (osd.6) 62 : cluster 0 purged_snaps scrub ok 2024-03-11T20:54:52.710 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:52 smithi023 bash[21372]: cluster 2024-03-11T20:54:51.677267+0000 mon.a (mon.0) 961 : cluster 0 osdmap e129: 8 total, 8 up, 7 in 2024-03-11T20:54:52.710 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:52 smithi023 bash[21372]: cluster 2024-03-11T20:54:51.900393+0000 mgr.y (mgr.24380) 156 : cluster 0 pgmap v130: 148 pgs: 148 active+clean; 582 KiB data, 233 MiB used, 626 GiB / 626 GiB avail; 934 B/s rd, 0 op/s 2024-03-11T20:54:52.710 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:52 smithi023 bash[29146]: cluster 2024-03-11T20:54:50.823983+0000 osd.6 (osd.6) 61 : cluster 0 purged_snaps scrub starts 2024-03-11T20:54:52.710 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:52 smithi023 bash[29146]: cluster 2024-03-11T20:54:50.824100+0000 osd.6 (osd.6) 62 : cluster 0 purged_snaps scrub ok 2024-03-11T20:54:52.711 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:52 smithi023 bash[29146]: cluster 2024-03-11T20:54:51.677267+0000 mon.a (mon.0) 961 : cluster 0 osdmap e129: 8 total, 8 up, 7 in 2024-03-11T20:54:52.711 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:52 smithi023 bash[29146]: cluster 2024-03-11T20:54:51.900393+0000 mgr.y (mgr.24380) 156 : cluster 0 pgmap v130: 148 pgs: 148 active+clean; 582 KiB data, 233 MiB used, 626 GiB / 626 GiB avail; 934 B/s rd, 0 op/s 2024-03-11T20:54:52.728 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:54:52.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:52.732 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:54:52.833 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:54:52.843 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:52 smithi146 bash[23809]: cluster 2024-03-11T20:54:50.823983+0000 osd.6 (osd.6) 61 : cluster 0 purged_snaps scrub starts 2024-03-11T20:54:52.843 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:52 smithi146 bash[23809]: cluster 2024-03-11T20:54:50.824100+0000 osd.6 (osd.6) 62 : cluster 0 purged_snaps scrub ok 2024-03-11T20:54:52.843 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:52 smithi146 bash[23809]: cluster 2024-03-11T20:54:51.677267+0000 mon.a (mon.0) 961 : cluster 0 osdmap e129: 8 total, 8 up, 7 in 2024-03-11T20:54:52.843 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:52 smithi146 bash[23809]: cluster 2024-03-11T20:54:51.900393+0000 mgr.y (mgr.24380) 156 : cluster 0 pgmap v130: 148 pgs: 148 active+clean; 582 KiB data, 233 MiB used, 626 GiB / 626 GiB avail; 934 B/s rd, 0 op/s 2024-03-11T20:54:52.858 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:54:52.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:52.861 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:54:52.961 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:54:52.990 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:54:52.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:52.992 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:54:53.092 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:54:53.118 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:54:53.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:53.120 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:54:53.221 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:54:53.247 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:54:53.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:53.249 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:54:53.350 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:54:53.378 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:54:53.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:53.382 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:54:53.482 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:54:53.508 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:54:53.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:53.510 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:54:53.534 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 2 16 5725 5709 178.394 187.062 0.00510668 0.00558353 2024-03-11T20:54:53.610 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:54:53.636 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:54:53.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:53.640 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:54:53.740 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:54:53.765 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:54:53.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:53.767 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:54:53.867 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:54:53.891 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:54:53.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:53.893 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:54:53.994 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:54:54.024 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:54:54.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:54.028 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:54:54.129 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:54:54.163 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:54:54.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:54.165 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:54:54.266 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:54:54.296 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:54:54.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:54.299 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:54:54.400 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:54:54.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:54.428 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:54:54.428 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:54:54.529 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:54:54.534 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 3 15 8554 8539 177.883 176.875 0.0107533 0.00560642 2024-03-11T20:54:54.562 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:54:54.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:54.563 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:54:54.664 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:54:54.691 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:54:54.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:54.694 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:54:54.795 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:54:54.824 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:54:54.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:54.827 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:54:54.928 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:54:54.950 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:54:54.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:54.952 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:54:55.053 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:54:55.078 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:54:55.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:55.080 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:54:55.181 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:54:55.190 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 20:54:55 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T20:54:55.191 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:54 smithi146 bash[23809]: cluster 2024-03-11T20:54:53.900840+0000 mgr.y (mgr.24380) 157 : cluster 0 pgmap v131: 148 pgs: 148 active+clean; 582 KiB data, 233 MiB used, 626 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T20:54:55.191 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:54 smithi146 bash[23809]: cluster 2024-03-11T20:54:54.668569+0000 mon.a (mon.0) 962 : cluster 0 osdmap e130: 8 total, 8 up, 7 in 2024-03-11T20:54:55.206 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:54:55.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:55.209 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:54:55.277 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-03-11T20:54:55.278 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:54 smithi023 bash[21372]: cluster 2024-03-11T20:54:53.900840+0000 mgr.y (mgr.24380) 157 : cluster 0 pgmap v131: 148 pgs: 148 active+clean; 582 KiB data, 233 MiB used, 626 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T20:54:55.278 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:54 smithi023 bash[21372]: cluster 2024-03-11T20:54:54.668569+0000 mon.a (mon.0) 962 : cluster 0 osdmap e130: 8 total, 8 up, 7 in 2024-03-11T20:54:55.278 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:54 smithi023 bash[29146]: cluster 2024-03-11T20:54:53.900840+0000 mgr.y (mgr.24380) 157 : cluster 0 pgmap v131: 148 pgs: 148 active+clean; 582 KiB data, 233 MiB used, 626 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T20:54:55.278 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:54 smithi023 bash[29146]: cluster 2024-03-11T20:54:54.668569+0000 mon.a (mon.0) 962 : cluster 0 osdmap e130: 8 total, 8 up, 7 in 2024-03-11T20:54:55.310 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:54:55.334 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:54:55.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:55.336 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:54:55.437 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:54:55.479 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:54:55.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:55.482 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:54:55.534 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 4 15 11028 11013 172.065 154.625 0.0114559 0.00579464 2024-03-11T20:54:55.582 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:54:55.605 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:54:55.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:55.607 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:54:55.708 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:54:55.732 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:54:55.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:55.734 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:54:55.835 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:54:55.859 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:54:55.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:55.862 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:54:55.867 INFO:teuthology.orchestra.run.smithi023.stdout:osd.0: {} 2024-03-11T20:54:55.901 INFO:teuthology.orchestra.run.smithi023.stdout:osd.1: {} 2024-03-11T20:54:55.945 INFO:teuthology.orchestra.run.smithi023.stdout:osd.2: {} 2024-03-11T20:54:55.963 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:54:55.986 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:54:55.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:55.988 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:54:55.996 INFO:teuthology.orchestra.run.smithi023.stdout:osd.3: {} 2024-03-11T20:54:56.040 INFO:teuthology.orchestra.run.smithi023.stdout:osd.4: {} 2024-03-11T20:54:56.088 INFO:teuthology.orchestra.run.smithi023.stdout:osd.5: {} 2024-03-11T20:54:56.089 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:54:56.122 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:54:56.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:56.125 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:54:56.126 INFO:teuthology.orchestra.run.smithi023.stdout:osd.6: {} 2024-03-11T20:54:56.166 INFO:teuthology.orchestra.run.smithi023.stdout:osd.7: {} 2024-03-11T20:54:56.170 INFO:teuthology.orchestra.run.smithi023.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T20:54:56.170 INFO:teuthology.orchestra.run.smithi023.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T20:54:56.171 INFO:teuthology.orchestra.run.smithi023.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T20:54:56.171 INFO:teuthology.orchestra.run.smithi023.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T20:54:56.171 INFO:teuthology.orchestra.run.smithi023.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T20:54:56.171 INFO:teuthology.orchestra.run.smithi023.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T20:54:56.171 INFO:teuthology.orchestra.run.smithi023.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T20:54:56.171 INFO:teuthology.orchestra.run.smithi023.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T20:54:56.220 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:55 smithi023 bash[21372]: audit 2024-03-11T20:54:54.997557+0000 mgr.y (mgr.24380) 158 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:54:56.221 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:55 smithi023 bash[21372]: audit 2024-03-11T20:54:55.820324+0000 mon.c (mon.2) 29 : audit 0 from='client.? 172.21.15.23:0/1701628457' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T20:54:56.221 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:55 smithi023 bash[29146]: audit 2024-03-11T20:54:54.997557+0000 mgr.y (mgr.24380) 158 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:54:56.221 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:55 smithi023 bash[29146]: audit 2024-03-11T20:54:55.820324+0000 mon.c (mon.2) 29 : audit 0 from='client.? 172.21.15.23:0/1701628457' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T20:54:56.226 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:54:56.236 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:55 smithi146 bash[23809]: audit 2024-03-11T20:54:54.997557+0000 mgr.y (mgr.24380) 158 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:54:56.236 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:55 smithi146 bash[23809]: audit 2024-03-11T20:54:55.820324+0000 mon.c (mon.2) 29 : audit 0 from='client.? 172.21.15.23:0/1701628457' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T20:54:56.251 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:54:56.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:56.254 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:54:56.355 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:54:56.379 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:54:56.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:56.382 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:54:56.483 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:54:56.514 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:54:56.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:56.516 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:54:56.534 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 5 16 13113 13097 163.7 130.25 0.00273658 0.00609716 2024-03-11T20:54:56.617 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:54:56.643 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:54:56.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:56.645 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:54:56.746 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:54:56.775 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:54:56.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:56.779 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:54:56.879 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:54:56.910 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:54:56.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:56.912 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:54:56.983 INFO:teuthology.orchestra.run.smithi023.stderr:nodeep-scrub is set 2024-03-11T20:54:57.013 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:54:57.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:56 smithi023 bash[21372]: cluster 2024-03-11T20:54:55.901632+0000 mgr.y (mgr.24380) 159 : cluster 0 pgmap v133: 148 pgs: 148 active+clean; 584 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 1.6 KiB/s rd, 93 MiB/s wr, 1.49k op/s 2024-03-11T20:54:57.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:56 smithi023 bash[21372]: audit 2024-03-11T20:54:56.247880+0000 mon.c (mon.2) 30 : audit 1 from='client.? 172.21.15.23:0/2030113423' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T20:54:57.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:56 smithi023 bash[21372]: audit 2024-03-11T20:54:56.248970+0000 mon.a (mon.0) 963 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T20:54:57.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:56 smithi023 bash[29146]: cluster 2024-03-11T20:54:55.901632+0000 mgr.y (mgr.24380) 159 : cluster 0 pgmap v133: 148 pgs: 148 active+clean; 584 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 1.6 KiB/s rd, 93 MiB/s wr, 1.49k op/s 2024-03-11T20:54:57.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:56 smithi023 bash[29146]: audit 2024-03-11T20:54:56.247880+0000 mon.c (mon.2) 30 : audit 1 from='client.? 172.21.15.23:0/2030113423' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T20:54:57.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:56 smithi023 bash[29146]: audit 2024-03-11T20:54:56.248970+0000 mon.a (mon.0) 963 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T20:54:57.036 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:54:57.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:57.038 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:54:57.139 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:54:57.178 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:54:57.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:57.182 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:54:57.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:56 smithi146 bash[23809]: cluster 2024-03-11T20:54:55.901632+0000 mgr.y (mgr.24380) 159 : cluster 0 pgmap v133: 148 pgs: 148 active+clean; 584 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 1.6 KiB/s rd, 93 MiB/s wr, 1.49k op/s 2024-03-11T20:54:57.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:56 smithi146 bash[23809]: audit 2024-03-11T20:54:56.247880+0000 mon.c (mon.2) 30 : audit 1 from='client.? 172.21.15.23:0/2030113423' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T20:54:57.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:56 smithi146 bash[23809]: audit 2024-03-11T20:54:56.248970+0000 mon.a (mon.0) 963 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T20:54:57.283 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:54:57.317 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:54:57.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:57.319 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:54:57.420 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:54:57.460 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:54:57.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:57.462 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:54:57.536 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 6 16 15513 15497 161.414 150 0.0050306 0.0061854 2024-03-11T20:54:57.562 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:54:57.598 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:54:57.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:57.601 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:54:57.702 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:54:57.727 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:54:57.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:57.729 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:54:57.830 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:54:57.861 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:54:57.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:57.863 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:54:57.964 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:54:57.988 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:54:57.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:57.991 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:54:58.092 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:54:58.103 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:57 smithi023 bash[21372]: cluster 2024-03-11T20:54:56.969146+0000 mon.a (mon.0) 964 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T20:54:58.103 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:57 smithi023 bash[21372]: audit 2024-03-11T20:54:56.971697+0000 mon.a (mon.0) 965 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T20:54:58.103 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:57 smithi023 bash[21372]: cluster 2024-03-11T20:54:56.978319+0000 mon.a (mon.0) 966 : cluster 0 osdmap e131: 8 total, 8 up, 7 in 2024-03-11T20:54:58.104 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:58 smithi023 bash[29146]: cluster 2024-03-11T20:54:56.969146+0000 mon.a (mon.0) 964 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T20:54:58.105 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:58 smithi023 bash[29146]: audit 2024-03-11T20:54:56.971697+0000 mon.a (mon.0) 965 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T20:54:58.105 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:58 smithi023 bash[29146]: cluster 2024-03-11T20:54:56.978319+0000 mon.a (mon.0) 966 : cluster 0 osdmap e131: 8 total, 8 up, 7 in 2024-03-11T20:54:58.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:58.122 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:54:58.122 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:54:58.224 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:54:58.251 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:54:58.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:58.253 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:54:58.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:58 smithi146 bash[23809]: cluster 2024-03-11T20:54:56.969146+0000 mon.a (mon.0) 964 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T20:54:58.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:58 smithi146 bash[23809]: audit 2024-03-11T20:54:56.971697+0000 mon.a (mon.0) 965 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T20:54:58.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:58 smithi146 bash[23809]: cluster 2024-03-11T20:54:56.978319+0000 mon.a (mon.0) 966 : cluster 0 osdmap e131: 8 total, 8 up, 7 in 2024-03-11T20:54:58.354 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:54:58.365 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:54:58 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:20:54:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T20:54:58.381 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:54:58.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:58.385 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:54:58.486 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:54:58.512 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:54:58.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:58.516 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:54:58.534 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 7 16 17789 17773 158.675 142.25 0.00474325 0.0062942 2024-03-11T20:54:58.617 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:54:58.638 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:54:58.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:58.639 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:54:58.740 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:54:58.765 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:54:58.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:58.768 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:54:58.869 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:54:58.888 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:54:58.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:58.890 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:54:58.990 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:54:59.016 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:54:59.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:59.020 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:54:59.120 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:54:59.139 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:54:59.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:59.141 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:54:59.172 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:54:59 smithi023 bash[21372]: cluster 2024-03-11T20:54:57.902191+0000 mgr.y (mgr.24380) 160 : cluster 0 pgmap v135: 148 pgs: 148 active+clean; 584 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 822 B/s rd, 94 MiB/s wr, 1.50k op/s 2024-03-11T20:54:59.173 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:54:59 smithi023 bash[29146]: cluster 2024-03-11T20:54:57.902191+0000 mgr.y (mgr.24380) 160 : cluster 0 pgmap v135: 148 pgs: 148 active+clean; 584 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 822 B/s rd, 94 MiB/s wr, 1.50k op/s 2024-03-11T20:54:59.242 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:54:59.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:54:59 smithi146 bash[23809]: cluster 2024-03-11T20:54:57.902191+0000 mgr.y (mgr.24380) 160 : cluster 0 pgmap v135: 148 pgs: 148 active+clean; 584 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 822 B/s rd, 94 MiB/s wr, 1.50k op/s 2024-03-11T20:54:59.271 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:54:59.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:59.275 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:54:59.375 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:54:59.401 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:54:59.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:59.404 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:54:59.506 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:54:59.526 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:54:59.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:59.528 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:54:59.534 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 8 16 19224 19208 150.05 89.6875 0.00191341 0.006561 2024-03-11T20:54:59.629 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:54:59.660 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:54:59.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:59.662 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:54:59.763 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:54:59.791 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:54:59.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:59.793 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:54:59.894 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:54:59.921 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:54:59.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:54:59.923 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:55:00.024 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:55:00.062 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:55:00.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:00.067 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:55:00.167 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:55:00.194 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:55:00.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:00.196 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:55:00.297 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:55:00.331 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:55:00.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:00.333 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:55:00.434 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:00.468 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:00.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:00.473 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:55:00.535 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 9 16 21374 21358 148.307 134.375 0.00792284 0.00673394 2024-03-11T20:55:00.574 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:55:00.603 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:55:00.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:00.606 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:55:00.706 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:55:00.731 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:55:00.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:00.734 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:55:00.834 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:00.845 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:00 smithi023 bash[21372]: cluster 2024-03-11T20:54:59.663396+0000 mon.a (mon.0) 967 : cluster 0 osdmap e132: 8 total, 8 up, 7 in 2024-03-11T20:55:00.845 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:00 smithi023 bash[21372]: cluster 2024-03-11T20:54:59.903680+0000 mgr.y (mgr.24380) 161 : cluster 0 pgmap v137: 148 pgs: 148 active+clean; 1.2 GiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 1.7 KiB/s rd, 207 MiB/s wr, 3.31k op/s 2024-03-11T20:55:00.845 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:00 smithi023 bash[29146]: cluster 2024-03-11T20:54:59.663396+0000 mon.a (mon.0) 967 : cluster 0 osdmap e132: 8 total, 8 up, 7 in 2024-03-11T20:55:00.845 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:00 smithi023 bash[29146]: cluster 2024-03-11T20:54:59.903680+0000 mgr.y (mgr.24380) 161 : cluster 0 pgmap v137: 148 pgs: 148 active+clean; 1.2 GiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 1.7 KiB/s rd, 207 MiB/s wr, 3.31k op/s 2024-03-11T20:55:00.863 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:00.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:00.866 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:55:00.967 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:55:00.993 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:55:00.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:00.996 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:55:00.997 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:55:01.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:00 smithi146 bash[23809]: cluster 2024-03-11T20:54:59.663396+0000 mon.a (mon.0) 967 : cluster 0 osdmap e132: 8 total, 8 up, 7 in 2024-03-11T20:55:01.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:00 smithi146 bash[23809]: cluster 2024-03-11T20:54:59.903680+0000 mgr.y (mgr.24380) 161 : cluster 0 pgmap v137: 148 pgs: 148 active+clean; 1.2 GiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 1.7 KiB/s rd, 207 MiB/s wr, 3.31k op/s 2024-03-11T20:55:01.098 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:01.128 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:01.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:01.130 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:55:01.231 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:55:01.256 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:55:01.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:01.261 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:55:01.362 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:01.386 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:01.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:01.393 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:55:01.493 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:55:01.503 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T20:55:01.529 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:55:01.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:01.532 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:55:01.535 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 10 16 23470 23454 146.576 131 0.00526042 0.00681371 2024-03-11T20:55:01.633 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:01.662 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:01.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:01.665 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:55:01.766 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:55:01.822 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:55:01.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:01.827 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:55:01.928 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:55:01.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:01.954 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:55:01.954 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:55:02.055 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:02.082 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:02.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:02.084 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:55:02.177 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd unset noscrub 2024-03-11T20:55:02.184 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:02.210 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:02.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:02.214 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:55:02.315 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:02.341 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:02.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:02.346 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:55:02.446 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:55:02.486 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:55:02.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:02.490 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:55:02.535 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 11 16 25338 25322 143.856 116.75 0.00421688 0.00694285 2024-03-11T20:55:02.591 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:55:02.612 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:55:02.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:02.615 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:55:02.716 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:02.740 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:02.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:02.743 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:55:02.843 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:02.872 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:02.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:02.875 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:55:02.976 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:55:03.003 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:55:03.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:03.005 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:55:03.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:02 smithi146 bash[23809]: cluster 2024-03-11T20:55:01.904128+0000 mgr.y (mgr.24380) 162 : cluster 0 pgmap v138: 148 pgs: 148 active+clean; 1.2 GiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 1.4 KiB/s rd, 172 MiB/s wr, 2.75k op/s 2024-03-11T20:55:03.033 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:55:03.105 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:55:03.137 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:55:03.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:03.139 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:55:03.240 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:03.272 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:03.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:03.275 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:55:03.376 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:03.400 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:03.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:03.402 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:55:03.503 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:55:03.527 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:55:03.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:03.529 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:55:03.535 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 12 16 27028 27012 140.669 105.625 0.00174792 0.00701469 2024-03-11T20:55:03.629 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:03.658 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:03.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:03.660 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:55:03.761 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:03.780 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:03.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:03.782 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:55:03.877 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:03 smithi023 bash[21372]: cluster 2024-03-11T20:55:01.904128+0000 mgr.y (mgr.24380) 162 : cluster 0 pgmap v138: 148 pgs: 148 active+clean; 1.2 GiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 1.4 KiB/s rd, 172 MiB/s wr, 2.75k op/s 2024-03-11T20:55:03.883 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:55:03.914 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:55:03.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:03.917 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:55:04.018 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:04.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:04.049 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:04.049 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:55:04.150 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:04.176 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:04.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:04.179 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:55:04.280 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:55:04.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:03 smithi023 bash[29146]: cluster 2024-03-11T20:55:01.904128+0000 mgr.y (mgr.24380) 162 : cluster 0 pgmap v138: 148 pgs: 148 active+clean; 1.2 GiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 1.4 KiB/s rd, 172 MiB/s wr, 2.75k op/s 2024-03-11T20:55:04.304 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:55:04.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:04.307 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:55:04.408 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:04.436 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:04.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:04.439 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:55:04.536 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 13 16 28655 28639 137.67 101.688 0.00759716 0.00725577 2024-03-11T20:55:04.539 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:04.562 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:04.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:04.565 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:55:04.666 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:55:04.690 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:55:04.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:04.692 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:55:04.793 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:55:04.823 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:55:04.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:04.827 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:55:04.927 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:04.958 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:04.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:04.960 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:55:05.061 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:55:05.089 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:55:05.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:05.092 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:55:05.193 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:05.203 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 20:55:05 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T20:55:05.217 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:05.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:05.219 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:55:05.320 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:55:05.347 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:55:05.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:05.350 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:55:05.451 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:05.462 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:05 smithi023 bash[21372]: cluster 2024-03-11T20:55:03.904576+0000 mgr.y (mgr.24380) 163 : cluster 0 pgmap v139: 148 pgs: 148 active+clean; 1.2 GiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 639 B/s rd, 82 MiB/s wr, 1.32k op/s 2024-03-11T20:55:05.462 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:05 smithi023 bash[21372]: audit 2024-03-11T20:55:04.315652+0000 mon.a (mon.0) 968 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:55:05.462 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:05 smithi023 bash[21372]: audit 2024-03-11T20:55:04.321292+0000 mon.a (mon.0) 969 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:55:05.463 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:05 smithi023 bash[21372]: cluster 2024-03-11T20:55:04.663328+0000 mon.a (mon.0) 970 : cluster 0 osdmap e133: 8 total, 8 up, 7 in 2024-03-11T20:55:05.463 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:05 smithi023 bash[29146]: cluster 2024-03-11T20:55:03.904576+0000 mgr.y (mgr.24380) 163 : cluster 0 pgmap v139: 148 pgs: 148 active+clean; 1.2 GiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 639 B/s rd, 82 MiB/s wr, 1.32k op/s 2024-03-11T20:55:05.463 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:05 smithi023 bash[29146]: audit 2024-03-11T20:55:04.315652+0000 mon.a (mon.0) 968 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:55:05.463 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:05 smithi023 bash[29146]: audit 2024-03-11T20:55:04.321292+0000 mon.a (mon.0) 969 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:55:05.463 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:05 smithi023 bash[29146]: cluster 2024-03-11T20:55:04.663328+0000 mon.a (mon.0) 970 : cluster 0 osdmap e133: 8 total, 8 up, 7 in 2024-03-11T20:55:05.476 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:05.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:05.481 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:55:05.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:05 smithi146 bash[23809]: cluster 2024-03-11T20:55:03.904576+0000 mgr.y (mgr.24380) 163 : cluster 0 pgmap v139: 148 pgs: 148 active+clean; 1.2 GiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 639 B/s rd, 82 MiB/s wr, 1.32k op/s 2024-03-11T20:55:05.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:05 smithi146 bash[23809]: audit 2024-03-11T20:55:04.315652+0000 mon.a (mon.0) 968 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:55:05.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:05 smithi146 bash[23809]: audit 2024-03-11T20:55:04.321292+0000 mon.a (mon.0) 969 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:55:05.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:05 smithi146 bash[23809]: cluster 2024-03-11T20:55:04.663328+0000 mon.a (mon.0) 970 : cluster 0 osdmap e133: 8 total, 8 up, 7 in 2024-03-11T20:55:05.536 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 14 16 30466 30450 135.921 113.188 0.00471346 0.00735014 2024-03-11T20:55:05.582 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:55:05.607 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:55:05.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:05.609 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:55:05.710 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:05.742 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:05.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:05.745 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:55:05.846 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:55:05.870 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:55:05.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:05.872 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:55:05.973 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:55:06.001 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:55:06.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:06.003 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:55:06.104 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:55:06.131 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:55:06.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:06.134 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:55:06.235 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:55:06.260 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:55:06.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:06.264 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:55:06.365 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:55:06.397 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:55:06.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:06.402 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:55:06.503 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:06.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:06 smithi146 bash[23809]: audit 2024-03-11T20:55:05.005561+0000 mgr.y (mgr.24380) 164 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:55:06.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:06 smithi146 bash[23809]: cluster 2024-03-11T20:55:05.304061+0000 osd.3 (osd.3) 129 : cluster 0 purged_snaps scrub starts 2024-03-11T20:55:06.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:06 smithi146 bash[23809]: cluster 2024-03-11T20:55:05.304131+0000 osd.3 (osd.3) 130 : cluster 0 purged_snaps scrub ok 2024-03-11T20:55:06.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:06 smithi146 bash[23809]: cluster 2024-03-11T20:55:05.680880+0000 mon.a (mon.0) 971 : cluster 0 osdmap e134: 8 total, 8 up, 7 in 2024-03-11T20:55:06.514 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:06 smithi023 bash[21372]: audit 2024-03-11T20:55:05.005561+0000 mgr.y (mgr.24380) 164 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:55:06.515 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:06 smithi023 bash[21372]: cluster 2024-03-11T20:55:05.304061+0000 osd.3 (osd.3) 129 : cluster 0 purged_snaps scrub starts 2024-03-11T20:55:06.515 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:06 smithi023 bash[21372]: cluster 2024-03-11T20:55:05.304131+0000 osd.3 (osd.3) 130 : cluster 0 purged_snaps scrub ok 2024-03-11T20:55:06.515 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:06 smithi023 bash[21372]: cluster 2024-03-11T20:55:05.680880+0000 mon.a (mon.0) 971 : cluster 0 osdmap e134: 8 total, 8 up, 7 in 2024-03-11T20:55:06.515 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:06 smithi023 bash[29146]: audit 2024-03-11T20:55:05.005561+0000 mgr.y (mgr.24380) 164 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:55:06.515 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:06 smithi023 bash[29146]: cluster 2024-03-11T20:55:05.304061+0000 osd.3 (osd.3) 129 : cluster 0 purged_snaps scrub starts 2024-03-11T20:55:06.515 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:06 smithi023 bash[29146]: cluster 2024-03-11T20:55:05.304131+0000 osd.3 (osd.3) 130 : cluster 0 purged_snaps scrub ok 2024-03-11T20:55:06.515 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:06 smithi023 bash[29146]: cluster 2024-03-11T20:55:05.680880+0000 mon.a (mon.0) 971 : cluster 0 osdmap e134: 8 total, 8 up, 7 in 2024-03-11T20:55:06.537 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:06.539 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 15 16 32296 32280 134.456 114.375 0.00867139 0.00742979 2024-03-11T20:55:06.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:06.541 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:55:06.641 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:55:06.667 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:55:06.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:06.670 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:55:06.771 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:06.794 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:06.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:06.800 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:55:06.902 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:06.927 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:06.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:06.929 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:55:07.030 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:55:07.059 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:55:07.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:07.061 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:55:07.162 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:07.185 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:07.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:07.188 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:55:07.289 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:55:07.313 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:55:07.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:07.315 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:55:07.416 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:07.441 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:07.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:07.446 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:55:07.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:07 smithi146 bash[23809]: cluster 2024-03-11T20:55:05.904966+0000 mgr.y (mgr.24380) 165 : cluster 0 pgmap v142: 148 pgs: 148 active+clean; 1.8 GiB data, 5.4 GiB used, 620 GiB / 626 GiB avail; 1.6 KiB/s rd, 199 MiB/s wr, 3.18k op/s 2024-03-11T20:55:07.539 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 16 16 34165 34149 133.353 116.812 0.00818426 0.00749273 2024-03-11T20:55:07.547 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:55:07.564 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:55:07.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:07.567 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:55:07.667 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:55:07.690 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:55:07.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:07.692 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:55:07.784 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:07 smithi023 bash[21372]: cluster 2024-03-11T20:55:05.904966+0000 mgr.y (mgr.24380) 165 : cluster 0 pgmap v142: 148 pgs: 148 active+clean; 1.8 GiB data, 5.4 GiB used, 620 GiB / 626 GiB avail; 1.6 KiB/s rd, 199 MiB/s wr, 3.18k op/s 2024-03-11T20:55:07.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:07 smithi023 bash[29146]: cluster 2024-03-11T20:55:05.904966+0000 mgr.y (mgr.24380) 165 : cluster 0 pgmap v142: 148 pgs: 148 active+clean; 1.8 GiB data, 5.4 GiB used, 620 GiB / 626 GiB avail; 1.6 KiB/s rd, 199 MiB/s wr, 3.18k op/s 2024-03-11T20:55:07.792 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:55:07.826 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:55:07.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:07.830 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:55:07.931 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:55:07.958 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:55:07.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:07.961 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:55:08.062 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:55:08.093 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:55:08.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:08.095 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:55:08.195 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:08.222 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:08.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:08.224 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:55:08.325 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:08.369 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:08.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:08.377 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:55:08.478 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:08.504 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:08.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:08.507 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:55:08.534 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:55:08 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:20:55:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T20:55:08.539 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 17 16 35573 35557 130.684 88 0.010025 0.00764411 2024-03-11T20:55:08.608 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:08.635 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:08.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:08.638 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:55:08.739 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:55:08.766 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:55:08.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:08.772 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:55:08.818 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:55:08.819 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:55:08.872 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:08.904 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:08.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:08.908 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:55:09.009 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:55:09.041 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:55:09.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:09.045 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:55:09.146 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:09.163 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:09.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:09.165 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:55:09.267 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:09.301 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:09.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:09.306 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:55:09.407 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:09.416 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:09 smithi146 bash[23809]: cluster 2024-03-11T20:55:07.905419+0000 mgr.y (mgr.24380) 166 : cluster 0 pgmap v143: 148 pgs: 148 active+clean; 1.8 GiB data, 5.4 GiB used, 620 GiB / 626 GiB avail; 639 B/s rd, 73 MiB/s wr, 1.17k op/s 2024-03-11T20:55:09.435 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:09.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:09.438 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:55:09.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:09 smithi023 bash[21372]: cluster 2024-03-11T20:55:07.905419+0000 mgr.y (mgr.24380) 166 : cluster 0 pgmap v143: 148 pgs: 148 active+clean; 1.8 GiB data, 5.4 GiB used, 620 GiB / 626 GiB avail; 639 B/s rd, 73 MiB/s wr, 1.17k op/s 2024-03-11T20:55:09.534 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:09 smithi023 bash[29146]: cluster 2024-03-11T20:55:07.905419+0000 mgr.y (mgr.24380) 166 : cluster 0 pgmap v143: 148 pgs: 148 active+clean; 1.8 GiB data, 5.4 GiB used, 620 GiB / 626 GiB avail; 639 B/s rd, 73 MiB/s wr, 1.17k op/s 2024-03-11T20:55:09.539 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:55:09.539 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 18 16 37140 37124 128.865 97.9375 0.00453854 0.00775279 2024-03-11T20:55:09.561 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:55:09.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:09.564 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:55:09.665 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:09.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:09.695 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:09.696 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:55:09.797 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:09.827 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:09.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:09.830 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:55:09.931 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:55:09.959 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:55:09.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:09.965 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:55:10.066 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:10.086 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:10.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:10.087 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:55:10.188 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:55:10.210 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:55:10.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:10.211 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:55:10.312 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:10.344 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:10.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:10.348 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:55:10.449 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:55:10.478 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:55:10.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:10.479 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:55:10.539 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 19 16 38462 38446 126.431 82.625 0.00441882 0.00790229 2024-03-11T20:55:10.580 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:10.609 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:10.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:10.613 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:55:10.713 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:10.748 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:10.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:10.751 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:55:10.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:10 smithi146 bash[23809]: cluster 2024-03-11T20:55:09.667024+0000 mon.a (mon.0) 972 : cluster 0 osdmap e135: 8 total, 8 up, 7 in 2024-03-11T20:55:10.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:10 smithi146 bash[23809]: cluster 2024-03-11T20:55:09.906152+0000 mgr.y (mgr.24380) 167 : cluster 0 pgmap v145: 148 pgs: 148 active+clean; 2.3 GiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 1.7 KiB/s rd, 185 MiB/s wr, 2.96k op/s 2024-03-11T20:55:10.852 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:10.878 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:10.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:10.882 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:55:10.983 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:11.007 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:11.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:11.009 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:55:11.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:10 smithi023 bash[21372]: cluster 2024-03-11T20:55:09.667024+0000 mon.a (mon.0) 972 : cluster 0 osdmap e135: 8 total, 8 up, 7 in 2024-03-11T20:55:11.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:10 smithi023 bash[21372]: cluster 2024-03-11T20:55:09.906152+0000 mgr.y (mgr.24380) 167 : cluster 0 pgmap v145: 148 pgs: 148 active+clean; 2.3 GiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 1.7 KiB/s rd, 185 MiB/s wr, 2.96k op/s 2024-03-11T20:55:11.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:10 smithi023 bash[29146]: cluster 2024-03-11T20:55:09.667024+0000 mon.a (mon.0) 972 : cluster 0 osdmap e135: 8 total, 8 up, 7 in 2024-03-11T20:55:11.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:10 smithi023 bash[29146]: cluster 2024-03-11T20:55:09.906152+0000 mgr.y (mgr.24380) 167 : cluster 0 pgmap v145: 148 pgs: 148 active+clean; 2.3 GiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 1.7 KiB/s rd, 185 MiB/s wr, 2.96k op/s 2024-03-11T20:55:11.110 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:11.134 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:11.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:11.136 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:55:11.237 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:55:11.271 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:55:11.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:11.274 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:55:11.375 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:11.402 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:11.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:11.405 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:55:11.506 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:55:11.536 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:55:11.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:11.539 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:55:11.539 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:2024-03-11T20:55:11.544379+0000 min lat: 0.0012354 max lat: 0.198697 avg lat: 0.00802892 2024-03-11T20:55:11.540 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-11T20:55:11.540 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 20 16 39845 39829 124.432 86.4375 0.00966258 0.00802892 2024-03-11T20:55:11.639 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:11.668 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:11.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:11.670 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:55:11.771 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:55:11.797 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:55:11.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:11.799 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:55:11.900 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:55:11.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:11.931 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:55:11.931 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:55:12.032 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:55:12.059 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:55:12.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:12.063 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:55:12.164 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:12.187 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:12.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:12.189 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:55:12.290 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:12.317 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:12.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:12.322 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:55:12.423 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:55:12.448 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:55:12.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:12.450 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:55:12.540 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 21 16 41512 41496 123.467 104.188 0.0119717 0.00809287 2024-03-11T20:55:12.551 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:12.577 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:12.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:12.579 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:55:12.680 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:12.711 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:12.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:12.715 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:55:12.818 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:12.847 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:12.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:12.849 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:55:12.950 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:12.975 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:12.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:12.977 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:55:13.078 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:13.087 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:12 smithi146 bash[23809]: cluster 2024-03-11T20:55:11.906608+0000 mgr.y (mgr.24380) 168 : cluster 0 pgmap v146: 148 pgs: 148 active+clean; 2.3 GiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 1.4 KiB/s rd, 153 MiB/s wr, 2.45k op/s 2024-03-11T20:55:13.102 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:13.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:13.105 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:55:13.109 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:12 smithi023 bash[21372]: cluster 2024-03-11T20:55:11.906608+0000 mgr.y (mgr.24380) 168 : cluster 0 pgmap v146: 148 pgs: 148 active+clean; 2.3 GiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 1.4 KiB/s rd, 153 MiB/s wr, 2.45k op/s 2024-03-11T20:55:13.109 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:12 smithi023 bash[29146]: cluster 2024-03-11T20:55:11.906608+0000 mgr.y (mgr.24380) 168 : cluster 0 pgmap v146: 148 pgs: 148 active+clean; 2.3 GiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 1.4 KiB/s rd, 153 MiB/s wr, 2.45k op/s 2024-03-11T20:55:13.206 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:13.235 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:13.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:13.243 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:55:13.344 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:13.354 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-11T20:55:13.354 INFO:teuthology.orchestra.run.smithi023.stdout: "size": 20, 2024-03-11T20:55:13.354 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 600, 2024-03-11T20:55:13.354 INFO:teuthology.orchestra.run.smithi023.stdout: "ops": [ 2024-03-11T20:55:13.354 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.354 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_op(client.24629.0:18658 7.4 7:2797f15b:::benchmark_data_smithi023_67959_object18657:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-03-11T20:55:13.354 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T20:54:59.031650+0000", 2024-03-11T20:55:13.354 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 14.322188497999999, 2024-03-11T20:55:13.354 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.14570545800000001, 2024-03-11T20:55:13.355 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T20:55:13.355 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T20:55:13.355 INFO:teuthology.orchestra.run.smithi023.stdout: "client_info": { 2024-03-11T20:55:13.355 INFO:teuthology.orchestra.run.smithi023.stdout: "client": "client.24629", 2024-03-11T20:55:13.355 INFO:teuthology.orchestra.run.smithi023.stdout: "client_addr": "172.21.15.23:0/959732609", 2024-03-11T20:55:13.355 INFO:teuthology.orchestra.run.smithi023.stdout: "tid": 18658 2024-03-11T20:55:13.355 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.355 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T20:55:13.355 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.355 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T20:55:13.355 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.031650+0000", 2024-03-11T20:55:13.355 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:55:13.355 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.356 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.356 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T20:55:13.356 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.031650+0000", 2024-03-11T20:55:13.356 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:55:13.356 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.356 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.356 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T20:55:13.356 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.031651+0000", 2024-03-11T20:55:13.356 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.265e-06 2024-03-11T20:55:13.356 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.356 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.356 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T20:55:13.356 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.031815+0000", 2024-03-11T20:55:13.356 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00016434199999999999 2024-03-11T20:55:13.357 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.357 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.357 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T20:55:13.357 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.031846+0000", 2024-03-11T20:55:13.357 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 3.0388000000000001e-05 2024-03-11T20:55:13.357 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.357 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.357 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T20:55:13.357 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.032186+0000", 2024-03-11T20:55:13.357 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00033998900000000001 2024-03-11T20:55:13.357 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.357 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.357 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T20:55:13.357 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.032292+0000", 2024-03-11T20:55:13.358 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00010604500000000001 2024-03-11T20:55:13.358 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.358 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.358 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T20:55:13.358 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.032458+0000", 2024-03-11T20:55:13.358 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00016588 2024-03-11T20:55:13.358 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.358 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.358 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for subops from 4,6", 2024-03-11T20:55:13.358 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.032587+0000", 2024-03-11T20:55:13.358 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00012897900000000001 2024-03-11T20:55:13.358 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.358 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.358 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "op_commit", 2024-03-11T20:55:13.359 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.035062+0000", 2024-03-11T20:55:13.359 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.002475779 2024-03-11T20:55:13.359 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.359 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.359 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T20:55:13.359 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.035116+0000", 2024-03-11T20:55:13.359 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 5.3260999999999997e-05 2024-03-11T20:55:13.359 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.359 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.359 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T20:55:13.359 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.177250+0000", 2024-03-11T20:55:13.359 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.14213466599999999 2024-03-11T20:55:13.359 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.360 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.360 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T20:55:13.360 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.177305+0000", 2024-03-11T20:55:13.360 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 5.4591e-05 2024-03-11T20:55:13.360 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.360 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.360 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T20:55:13.360 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.177355+0000", 2024-03-11T20:55:13.360 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 5.0272999999999998e-05 2024-03-11T20:55:13.360 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:55:13.361 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T20:55:13.361 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:55:13.361 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.361 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.361 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_op(client.24629.0:18661 7.9 7:95522e60:::benchmark_data_smithi023_67959_object18660:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-03-11T20:55:13.361 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T20:54:59.032247+0000", 2024-03-11T20:55:13.361 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 14.321591194, 2024-03-11T20:55:13.361 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.14479431300000001, 2024-03-11T20:55:13.361 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T20:55:13.361 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T20:55:13.361 INFO:teuthology.orchestra.run.smithi023.stdout: "client_info": { 2024-03-11T20:55:13.361 INFO:teuthology.orchestra.run.smithi023.stdout: "client": "client.24629", 2024-03-11T20:55:13.361 INFO:teuthology.orchestra.run.smithi023.stdout: "client_addr": "172.21.15.23:0/959732609", 2024-03-11T20:55:13.361 INFO:teuthology.orchestra.run.smithi023.stdout: "tid": 18661 2024-03-11T20:55:13.362 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.362 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T20:55:13.362 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.362 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T20:55:13.362 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.032247+0000", 2024-03-11T20:55:13.362 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:55:13.362 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.362 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.362 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T20:55:13.363 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.032247+0000", 2024-03-11T20:55:13.363 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:55:13.363 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.363 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.363 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T20:55:13.363 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.032248+0000", 2024-03-11T20:55:13.363 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 8.2900000000000002e-07 2024-03-11T20:55:13.363 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.363 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.363 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T20:55:13.363 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.032342+0000", 2024-03-11T20:55:13.363 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 9.3771999999999996e-05 2024-03-11T20:55:13.363 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.363 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.363 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T20:55:13.364 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.032360+0000", 2024-03-11T20:55:13.364 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.8304e-05 2024-03-11T20:55:13.364 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.364 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.364 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T20:55:13.364 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.032369+0000", 2024-03-11T20:55:13.364 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 9.6050000000000004e-06 2024-03-11T20:55:13.364 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.364 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.364 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T20:55:13.364 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.032414+0000", 2024-03-11T20:55:13.364 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 4.4634e-05 2024-03-11T20:55:13.364 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.364 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.364 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T20:55:13.365 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.032493+0000", 2024-03-11T20:55:13.365 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.8465000000000002e-05 2024-03-11T20:55:13.365 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.365 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.365 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for subops from 4,7", 2024-03-11T20:55:13.365 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.032572+0000", 2024-03-11T20:55:13.365 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.9074999999999998e-05 2024-03-11T20:55:13.365 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.365 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.365 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "op_commit", 2024-03-11T20:55:13.365 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.034682+0000", 2024-03-11T20:55:13.365 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0021104930000000002 2024-03-11T20:55:13.365 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.365 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.365 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T20:55:13.366 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.034719+0000", 2024-03-11T20:55:13.366 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 3.6792999999999998e-05 2024-03-11T20:55:13.366 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.366 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.366 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T20:55:13.366 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.035332+0000", 2024-03-11T20:55:13.366 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00061339299999999995 2024-03-11T20:55:13.366 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.366 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.366 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T20:55:13.366 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.035364+0000", 2024-03-11T20:55:13.366 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 3.1751999999999997e-05 2024-03-11T20:55:13.366 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.366 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.366 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T20:55:13.367 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.177041+0000", 2024-03-11T20:55:13.367 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.141677198 2024-03-11T20:55:13.367 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:55:13.367 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T20:55:13.367 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:55:13.367 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.367 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.367 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_op(client.24629.0:18663 7.9 7:9a69d828:::benchmark_data_smithi023_67959_object18662:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-03-11T20:55:13.367 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T20:54:59.032394+0000", 2024-03-11T20:55:13.367 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 14.321444318999999, 2024-03-11T20:55:13.367 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.144930527, 2024-03-11T20:55:13.367 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T20:55:13.367 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T20:55:13.367 INFO:teuthology.orchestra.run.smithi023.stdout: "client_info": { 2024-03-11T20:55:13.367 INFO:teuthology.orchestra.run.smithi023.stdout: "client": "client.24629", 2024-03-11T20:55:13.368 INFO:teuthology.orchestra.run.smithi023.stdout: "client_addr": "172.21.15.23:0/959732609", 2024-03-11T20:55:13.368 INFO:teuthology.orchestra.run.smithi023.stdout: "tid": 18663 2024-03-11T20:55:13.368 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.368 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T20:55:13.368 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.368 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T20:55:13.368 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.032394+0000", 2024-03-11T20:55:13.368 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:55:13.368 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.368 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.368 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T20:55:13.368 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.032394+0000", 2024-03-11T20:55:13.368 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:55:13.368 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.369 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.369 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T20:55:13.369 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.032395+0000", 2024-03-11T20:55:13.369 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.6400000000000002e-07 2024-03-11T20:55:13.369 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.369 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.369 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T20:55:13.369 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.032418+0000", 2024-03-11T20:55:13.369 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.3530999999999999e-05 2024-03-11T20:55:13.369 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.369 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.369 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T20:55:13.369 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.032432+0000", 2024-03-11T20:55:13.369 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.428e-05 2024-03-11T20:55:13.369 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.370 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.370 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T20:55:13.370 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.032439+0000", 2024-03-11T20:55:13.370 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.9369999999999998e-06 2024-03-11T20:55:13.370 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.370 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.370 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T20:55:13.370 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.033463+0000", 2024-03-11T20:55:13.370 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0010237320000000001 2024-03-11T20:55:13.370 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.370 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.370 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T20:55:13.370 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.033571+0000", 2024-03-11T20:55:13.370 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000108381 2024-03-11T20:55:13.371 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.371 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.371 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for subops from 4,7", 2024-03-11T20:55:13.371 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.033698+0000", 2024-03-11T20:55:13.371 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00012673400000000001 2024-03-11T20:55:13.371 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.371 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.371 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "op_commit", 2024-03-11T20:55:13.371 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.034730+0000", 2024-03-11T20:55:13.371 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.001032348 2024-03-11T20:55:13.371 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.371 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.371 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T20:55:13.371 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.177168+0000", 2024-03-11T20:55:13.371 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.142437955 2024-03-11T20:55:13.372 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.372 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.372 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T20:55:13.372 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.177214+0000", 2024-03-11T20:55:13.372 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 4.5373000000000001e-05 2024-03-11T20:55:13.372 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.372 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.372 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T20:55:13.372 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.177276+0000", 2024-03-11T20:55:13.372 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.2339000000000001e-05 2024-03-11T20:55:13.372 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.372 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.372 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T20:55:13.372 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.177324+0000", 2024-03-11T20:55:13.372 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 4.8253e-05 2024-03-11T20:55:13.373 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:55:13.373 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T20:55:13.373 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:55:13.373 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.373 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.373 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_op(client.24629.0:18666 7.f 7:f8de726c:::benchmark_data_smithi023_67959_object18665:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-03-11T20:55:13.373 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T20:54:59.032744+0000", 2024-03-11T20:55:13.373 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 14.321094111000001, 2024-03-11T20:55:13.373 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.147086895, 2024-03-11T20:55:13.373 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T20:55:13.373 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T20:55:13.373 INFO:teuthology.orchestra.run.smithi023.stdout: "client_info": { 2024-03-11T20:55:13.373 INFO:teuthology.orchestra.run.smithi023.stdout: "client": "client.24629", 2024-03-11T20:55:13.373 INFO:teuthology.orchestra.run.smithi023.stdout: "client_addr": "172.21.15.23:0/959732609", 2024-03-11T20:55:13.374 INFO:teuthology.orchestra.run.smithi023.stdout: "tid": 18666 2024-03-11T20:55:13.374 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.374 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T20:55:13.374 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.374 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T20:55:13.374 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.032744+0000", 2024-03-11T20:55:13.374 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:55:13.374 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.374 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.374 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T20:55:13.374 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.032744+0000", 2024-03-11T20:55:13.374 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:55:13.374 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.374 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.375 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T20:55:13.375 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.032745+0000", 2024-03-11T20:55:13.375 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 8.23e-07 2024-03-11T20:55:13.375 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.375 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.375 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T20:55:13.375 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.032771+0000", 2024-03-11T20:55:13.375 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.5868e-05 2024-03-11T20:55:13.375 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.375 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.375 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T20:55:13.375 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.032781+0000", 2024-03-11T20:55:13.375 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.0009e-05 2024-03-11T20:55:13.375 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.375 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.376 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T20:55:13.376 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.032790+0000", 2024-03-11T20:55:13.376 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 9.4420000000000004e-06 2024-03-11T20:55:13.376 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.376 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.376 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T20:55:13.376 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.032839+0000", 2024-03-11T20:55:13.376 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 4.8816000000000002e-05 2024-03-11T20:55:13.376 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.376 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.376 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T20:55:13.376 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.032947+0000", 2024-03-11T20:55:13.376 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000107635 2024-03-11T20:55:13.376 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.377 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.377 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for subops from 3,7", 2024-03-11T20:55:13.377 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.033070+0000", 2024-03-11T20:55:13.377 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000123084 2024-03-11T20:55:13.377 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.377 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.377 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T20:55:13.377 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.178633+0000", 2024-03-11T20:55:13.377 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.14556374899999999 2024-03-11T20:55:13.377 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.377 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.377 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T20:55:13.377 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.178669+0000", 2024-03-11T20:55:13.377 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 3.5272999999999999e-05 2024-03-11T20:55:13.377 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.377 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.378 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "op_commit", 2024-03-11T20:55:13.378 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.179597+0000", 2024-03-11T20:55:13.378 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000928172 2024-03-11T20:55:13.378 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.378 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.378 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T20:55:13.378 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.179736+0000", 2024-03-11T20:55:13.378 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000138959 2024-03-11T20:55:13.378 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.378 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.378 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T20:55:13.378 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.179831+0000", 2024-03-11T20:55:13.378 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 9.5064999999999999e-05 2024-03-11T20:55:13.379 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:55:13.379 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T20:55:13.379 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:55:13.379 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.379 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.379 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_op(client.24629.0:18667 7.f 7:f1532a40:::benchmark_data_smithi023_67959_object18666:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-03-11T20:55:13.379 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T20:54:59.032804+0000", 2024-03-11T20:55:13.379 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 14.321033884, 2024-03-11T20:55:13.379 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.14727010099999999, 2024-03-11T20:55:13.379 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T20:55:13.379 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T20:55:13.379 INFO:teuthology.orchestra.run.smithi023.stdout: "client_info": { 2024-03-11T20:55:13.379 INFO:teuthology.orchestra.run.smithi023.stdout: "client": "client.24629", 2024-03-11T20:55:13.379 INFO:teuthology.orchestra.run.smithi023.stdout: "client_addr": "172.21.15.23:0/959732609", 2024-03-11T20:55:13.379 INFO:teuthology.orchestra.run.smithi023.stdout: "tid": 18667 2024-03-11T20:55:13.379 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.380 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T20:55:13.380 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.380 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T20:55:13.380 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.032804+0000", 2024-03-11T20:55:13.380 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:55:13.380 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.380 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.380 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T20:55:13.380 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.032804+0000", 2024-03-11T20:55:13.380 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:55:13.380 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.380 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.380 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T20:55:13.380 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.032805+0000", 2024-03-11T20:55:13.381 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.1699999999999998e-07 2024-03-11T20:55:13.381 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.381 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.381 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T20:55:13.381 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.032839+0000", 2024-03-11T20:55:13.381 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 3.4477000000000002e-05 2024-03-11T20:55:13.381 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.381 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.381 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T20:55:13.381 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.032848+0000", 2024-03-11T20:55:13.381 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 8.7020000000000007e-06 2024-03-11T20:55:13.381 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.381 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.381 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T20:55:13.382 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.032857+0000", 2024-03-11T20:55:13.382 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 9.0650000000000005e-06 2024-03-11T20:55:13.382 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.382 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.382 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T20:55:13.382 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.034487+0000", 2024-03-11T20:55:13.382 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0016301899999999999 2024-03-11T20:55:13.382 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.382 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.382 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T20:55:13.382 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.034599+0000", 2024-03-11T20:55:13.382 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000111907 2024-03-11T20:55:13.382 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.382 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.383 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for subops from 3,7", 2024-03-11T20:55:13.383 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.034736+0000", 2024-03-11T20:55:13.383 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000136463 2024-03-11T20:55:13.383 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.383 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.383 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T20:55:13.383 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.179568+0000", 2024-03-11T20:55:13.383 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.144832341 2024-03-11T20:55:13.383 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.383 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.383 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T20:55:13.383 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.179885+0000", 2024-03-11T20:55:13.383 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000317168 2024-03-11T20:55:13.384 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.384 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.384 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "op_commit", 2024-03-11T20:55:13.384 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.179957+0000", 2024-03-11T20:55:13.384 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.1818999999999998e-05 2024-03-11T20:55:13.384 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.384 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.384 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T20:55:13.384 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.180024+0000", 2024-03-11T20:55:13.384 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.7399999999999998e-05 2024-03-11T20:55:13.384 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.384 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.384 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T20:55:13.384 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.180074+0000", 2024-03-11T20:55:13.385 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 4.9951999999999999e-05 2024-03-11T20:55:13.385 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:55:13.385 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T20:55:13.385 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:55:13.385 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.385 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.385 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_op(client.24629.0:18668 7.4 7:247a2eb4:::benchmark_data_smithi023_67959_object18667:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-03-11T20:55:13.385 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T20:54:59.032876+0000", 2024-03-11T20:55:13.385 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 14.320962169, 2024-03-11T20:55:13.385 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.144621046, 2024-03-11T20:55:13.385 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T20:55:13.385 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T20:55:13.385 INFO:teuthology.orchestra.run.smithi023.stdout: "client_info": { 2024-03-11T20:55:13.385 INFO:teuthology.orchestra.run.smithi023.stdout: "client": "client.24629", 2024-03-11T20:55:13.386 INFO:teuthology.orchestra.run.smithi023.stdout: "client_addr": "172.21.15.23:0/959732609", 2024-03-11T20:55:13.386 INFO:teuthology.orchestra.run.smithi023.stdout: "tid": 18668 2024-03-11T20:55:13.386 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.386 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T20:55:13.386 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.386 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T20:55:13.386 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.032876+0000", 2024-03-11T20:55:13.386 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:55:13.386 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.386 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.386 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T20:55:13.386 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.032876+0000", 2024-03-11T20:55:13.386 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:55:13.386 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.386 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.387 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T20:55:13.387 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.032877+0000", 2024-03-11T20:55:13.387 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.2900000000000003e-07 2024-03-11T20:55:13.387 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.387 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.387 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T20:55:13.387 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.032910+0000", 2024-03-11T20:55:13.387 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 3.3711000000000003e-05 2024-03-11T20:55:13.387 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.387 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.387 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T20:55:13.387 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.032919+0000", 2024-03-11T20:55:13.387 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 9.1099999999999992e-06 2024-03-11T20:55:13.387 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.388 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.388 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T20:55:13.388 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.032929+0000", 2024-03-11T20:55:13.388 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 9.5950000000000005e-06 2024-03-11T20:55:13.388 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.388 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.388 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T20:55:13.388 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.033487+0000", 2024-03-11T20:55:13.388 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00055788500000000004 2024-03-11T20:55:13.388 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.388 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.388 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T20:55:13.388 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.033592+0000", 2024-03-11T20:55:13.388 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00010522399999999999 2024-03-11T20:55:13.388 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.389 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.389 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for subops from 4,6", 2024-03-11T20:55:13.389 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.033717+0000", 2024-03-11T20:55:13.389 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000125117 2024-03-11T20:55:13.389 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.389 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.389 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "op_commit", 2024-03-11T20:55:13.389 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.035203+0000", 2024-03-11T20:55:13.389 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.001485267 2024-03-11T20:55:13.389 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.389 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.389 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T20:55:13.389 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.177180+0000", 2024-03-11T20:55:13.389 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.14197754200000001 2024-03-11T20:55:13.389 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.390 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.390 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T20:55:13.390 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.177404+0000", 2024-03-11T20:55:13.390 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000224191 2024-03-11T20:55:13.390 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.390 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.390 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T20:55:13.390 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.177462+0000", 2024-03-11T20:55:13.390 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 5.7958000000000003e-05 2024-03-11T20:55:13.390 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.390 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.390 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T20:55:13.390 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.177497+0000", 2024-03-11T20:55:13.390 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 3.4817000000000002e-05 2024-03-11T20:55:13.391 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:55:13.391 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T20:55:13.391 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:55:13.391 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.391 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.391 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_repop(client.24629.0:18657 7.7 e131/121 7:eb0c1180:::benchmark_data_smithi023_67959_object18656:head v 131'1194, mlcod=131'1192)", 2024-03-11T20:55:13.391 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T20:54:59.033028+0000", 2024-03-11T20:55:13.391 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 14.320810019, 2024-03-11T20:55:13.391 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.14688300500000001, 2024-03-11T20:55:13.391 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T20:55:13.391 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T20:55:13.392 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T20:55:13.392 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.392 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T20:55:13.392 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.033028+0000", 2024-03-11T20:55:13.392 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:55:13.392 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.392 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.392 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T20:55:13.392 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.033028+0000", 2024-03-11T20:55:13.392 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:55:13.392 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.393 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.393 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T20:55:13.393 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.033029+0000", 2024-03-11T20:55:13.393 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 5.5000000000000003e-07 2024-03-11T20:55:13.393 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.393 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.393 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T20:55:13.393 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.033166+0000", 2024-03-11T20:55:13.393 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00013716800000000001 2024-03-11T20:55:13.393 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.393 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.393 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T20:55:13.393 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.033176+0000", 2024-03-11T20:55:13.393 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 9.9660000000000008e-06 2024-03-11T20:55:13.393 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.394 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.394 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T20:55:13.394 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.033184+0000", 2024-03-11T20:55:13.394 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 8.1890000000000007e-06 2024-03-11T20:55:13.394 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.394 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.394 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T20:55:13.394 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.033845+0000", 2024-03-11T20:55:13.394 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00066052000000000005 2024-03-11T20:55:13.394 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.394 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.394 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T20:55:13.394 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.033864+0000", 2024-03-11T20:55:13.395 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.9157999999999999e-05 2024-03-11T20:55:13.395 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.395 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.395 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T20:55:13.395 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.179839+0000", 2024-03-11T20:55:13.395 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.145975245 2024-03-11T20:55:13.395 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.395 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.395 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T20:55:13.395 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.179911+0000", 2024-03-11T20:55:13.395 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.2208999999999999e-05 2024-03-11T20:55:13.395 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:55:13.395 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T20:55:13.395 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:55:13.395 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.396 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.396 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_op(client.24629.0:18670 7.f 7:fe58fd98:::benchmark_data_smithi023_67959_object18669:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-03-11T20:55:13.396 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T20:54:59.033351+0000", 2024-03-11T20:55:13.396 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 14.320487211, 2024-03-11T20:55:13.396 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.15182358700000001, 2024-03-11T20:55:13.396 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T20:55:13.396 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T20:55:13.396 INFO:teuthology.orchestra.run.smithi023.stdout: "client_info": { 2024-03-11T20:55:13.396 INFO:teuthology.orchestra.run.smithi023.stdout: "client": "client.24629", 2024-03-11T20:55:13.396 INFO:teuthology.orchestra.run.smithi023.stdout: "client_addr": "172.21.15.23:0/959732609", 2024-03-11T20:55:13.396 INFO:teuthology.orchestra.run.smithi023.stdout: "tid": 18670 2024-03-11T20:55:13.396 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.396 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T20:55:13.396 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.397 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T20:55:13.397 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.033351+0000", 2024-03-11T20:55:13.397 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:55:13.397 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.397 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.397 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T20:55:13.397 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.033351+0000", 2024-03-11T20:55:13.397 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:55:13.397 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.397 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.397 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T20:55:13.397 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.033352+0000", 2024-03-11T20:55:13.397 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.5799999999999998e-07 2024-03-11T20:55:13.397 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.397 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.398 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T20:55:13.398 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.033379+0000", 2024-03-11T20:55:13.398 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.7657e-05 2024-03-11T20:55:13.398 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.398 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.398 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T20:55:13.398 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.033390+0000", 2024-03-11T20:55:13.398 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.0917999999999999e-05 2024-03-11T20:55:13.398 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.398 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.398 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T20:55:13.398 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.033397+0000", 2024-03-11T20:55:13.398 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.6529999999999997e-06 2024-03-11T20:55:13.398 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.398 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.399 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T20:55:13.399 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.177407+0000", 2024-03-11T20:55:13.399 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.144010323 2024-03-11T20:55:13.399 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.399 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.399 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T20:55:13.399 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.177563+0000", 2024-03-11T20:55:13.399 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00015585600000000001 2024-03-11T20:55:13.399 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.399 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.399 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for subops from 3,7", 2024-03-11T20:55:13.399 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.177727+0000", 2024-03-11T20:55:13.399 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00016380999999999999 2024-03-11T20:55:13.399 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.400 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.400 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T20:55:13.400 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.179927+0000", 2024-03-11T20:55:13.400 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00219959 2024-03-11T20:55:13.400 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.400 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.400 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "op_commit", 2024-03-11T20:55:13.400 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.180092+0000", 2024-03-11T20:55:13.400 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000165069 2024-03-11T20:55:13.400 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.400 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.400 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T20:55:13.400 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.185104+0000", 2024-03-11T20:55:13.400 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0050127549999999998 2024-03-11T20:55:13.401 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.401 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.401 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T20:55:13.401 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.185143+0000", 2024-03-11T20:55:13.401 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 3.8658999999999997e-05 2024-03-11T20:55:13.401 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.401 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.401 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T20:55:13.401 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.185175+0000", 2024-03-11T20:55:13.401 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 3.1538999999999997e-05 2024-03-11T20:55:13.401 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:55:13.401 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T20:55:13.401 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:55:13.402 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.402 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.402 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_repop(client.24629.0:18664 7.d e131/124 7:bc835513:::benchmark_data_smithi023_67959_object18663:head v 131'1135, mlcod=131'1131)", 2024-03-11T20:55:13.402 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T20:54:59.033908+0000", 2024-03-11T20:55:13.402 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 14.319930501, 2024-03-11T20:55:13.402 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.143202995, 2024-03-11T20:55:13.402 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T20:55:13.402 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T20:55:13.402 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T20:55:13.402 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.402 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T20:55:13.402 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.033908+0000", 2024-03-11T20:55:13.402 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:55:13.402 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.403 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.403 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T20:55:13.403 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.033908+0000", 2024-03-11T20:55:13.403 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:55:13.403 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.403 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.403 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T20:55:13.403 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.033908+0000", 2024-03-11T20:55:13.403 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.1200000000000002e-07 2024-03-11T20:55:13.403 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.403 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.403 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T20:55:13.403 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.033946+0000", 2024-03-11T20:55:13.403 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 3.7629999999999997e-05 2024-03-11T20:55:13.404 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.404 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.404 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T20:55:13.404 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.033968+0000", 2024-03-11T20:55:13.404 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.1670000000000001e-05 2024-03-11T20:55:13.404 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.404 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.404 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T20:55:13.404 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.033974+0000", 2024-03-11T20:55:13.404 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.2369999999999997e-06 2024-03-11T20:55:13.404 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.404 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.404 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T20:55:13.404 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.034324+0000", 2024-03-11T20:55:13.404 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00035042599999999998 2024-03-11T20:55:13.405 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.405 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.405 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T20:55:13.405 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.034341+0000", 2024-03-11T20:55:13.405 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.7059999999999999e-05 2024-03-11T20:55:13.405 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.405 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.405 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T20:55:13.405 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.035478+0000", 2024-03-11T20:55:13.405 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0011362169999999999 2024-03-11T20:55:13.405 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.405 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.405 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T20:55:13.405 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.177111+0000", 2024-03-11T20:55:13.405 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.14163304299999999 2024-03-11T20:55:13.406 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:55:13.406 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T20:55:13.406 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:55:13.406 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.406 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.406 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_repop(client.24629.0:18665 7.5 e131/121 7:a1e52dbf:::benchmark_data_smithi023_67959_object18664:head v 131'1179, mlcod=131'1177)", 2024-03-11T20:55:13.406 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T20:54:59.034578+0000", 2024-03-11T20:55:13.406 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 14.319259858000001, 2024-03-11T20:55:13.406 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.14281870199999999, 2024-03-11T20:55:13.406 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T20:55:13.406 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T20:55:13.406 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T20:55:13.406 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.406 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T20:55:13.407 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.034578+0000", 2024-03-11T20:55:13.407 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:55:13.407 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.407 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.407 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T20:55:13.407 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.034578+0000", 2024-03-11T20:55:13.407 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:55:13.407 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.407 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.407 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T20:55:13.407 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.034579+0000", 2024-03-11T20:55:13.407 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.9599999999999999e-07 2024-03-11T20:55:13.407 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.407 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.407 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T20:55:13.408 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.034614+0000", 2024-03-11T20:55:13.408 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 3.5221e-05 2024-03-11T20:55:13.408 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.408 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.408 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T20:55:13.408 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.034626+0000", 2024-03-11T20:55:13.408 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.1878999999999999e-05 2024-03-11T20:55:13.408 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.408 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.408 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T20:55:13.408 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.034634+0000", 2024-03-11T20:55:13.408 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.4309999999999997e-06 2024-03-11T20:55:13.408 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.408 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.408 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T20:55:13.409 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.034746+0000", 2024-03-11T20:55:13.409 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000112456 2024-03-11T20:55:13.409 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.409 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.409 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T20:55:13.409 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.034758+0000", 2024-03-11T20:55:13.409 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.2310000000000001e-05 2024-03-11T20:55:13.409 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.409 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.409 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T20:55:13.409 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.177359+0000", 2024-03-11T20:55:13.409 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.14260107999999999 2024-03-11T20:55:13.409 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.409 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.410 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T20:55:13.410 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.177397+0000", 2024-03-11T20:55:13.410 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 3.7629000000000002e-05 2024-03-11T20:55:13.410 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:55:13.410 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T20:55:13.410 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:55:13.410 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.410 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.410 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_op(client.24629.0:18672 7.f 7:f925ca79:::benchmark_data_smithi023_67959_object18671:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-03-11T20:55:13.410 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T20:54:59.035910+0000", 2024-03-11T20:55:13.410 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 14.317928045, 2024-03-11T20:55:13.410 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.15607690699999999, 2024-03-11T20:55:13.410 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T20:55:13.410 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T20:55:13.411 INFO:teuthology.orchestra.run.smithi023.stdout: "client_info": { 2024-03-11T20:55:13.411 INFO:teuthology.orchestra.run.smithi023.stdout: "client": "client.24629", 2024-03-11T20:55:13.411 INFO:teuthology.orchestra.run.smithi023.stdout: "client_addr": "172.21.15.23:0/959732609", 2024-03-11T20:55:13.411 INFO:teuthology.orchestra.run.smithi023.stdout: "tid": 18672 2024-03-11T20:55:13.411 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.411 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T20:55:13.411 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.411 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T20:55:13.411 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.035910+0000", 2024-03-11T20:55:13.411 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:55:13.411 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.411 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.411 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T20:55:13.411 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.035910+0000", 2024-03-11T20:55:13.412 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:55:13.412 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.412 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.412 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T20:55:13.412 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.035912+0000", 2024-03-11T20:55:13.412 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.4279999999999999e-06 2024-03-11T20:55:13.412 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.412 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.412 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T20:55:13.412 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.035930+0000", 2024-03-11T20:55:13.412 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.8442000000000001e-05 2024-03-11T20:55:13.412 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.412 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.412 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T20:55:13.412 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.177342+0000", 2024-03-11T20:55:13.413 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.14141198999999999 2024-03-11T20:55:13.413 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.413 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.413 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T20:55:13.413 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.177351+0000", 2024-03-11T20:55:13.413 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 9.0129999999999999e-06 2024-03-11T20:55:13.413 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.413 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.413 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T20:55:13.413 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.180124+0000", 2024-03-11T20:55:13.413 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.002772526 2024-03-11T20:55:13.413 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.413 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.413 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T20:55:13.413 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.180243+0000", 2024-03-11T20:55:13.414 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00011927899999999999 2024-03-11T20:55:13.414 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.414 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.414 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for subops from 3,7", 2024-03-11T20:55:13.414 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.180538+0000", 2024-03-11T20:55:13.414 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00029486600000000002 2024-03-11T20:55:13.414 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.414 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.414 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "op_commit", 2024-03-11T20:55:13.414 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.182002+0000", 2024-03-11T20:55:13.414 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0014645630000000001 2024-03-11T20:55:13.414 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.414 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.414 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T20:55:13.415 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.182997+0000", 2024-03-11T20:55:13.415 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00099433799999999999 2024-03-11T20:55:13.415 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.415 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.415 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T20:55:13.415 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.191912+0000", 2024-03-11T20:55:13.415 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0089156419999999997 2024-03-11T20:55:13.415 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.415 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.415 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T20:55:13.415 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.191963+0000", 2024-03-11T20:55:13.415 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 5.1202999999999997e-05 2024-03-11T20:55:13.416 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.416 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.416 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T20:55:13.416 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:54:59.191987+0000", 2024-03-11T20:55:13.416 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.3617000000000001e-05 2024-03-11T20:55:13.416 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:55:13.416 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T20:55:13.416 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:55:13.416 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.416 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.416 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_op(client.24629.0:34155 7.9 7:94764dd5:::benchmark_data_smithi023_67959_object34154:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e134)", 2024-03-11T20:55:13.416 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T20:55:07.541459+0000", 2024-03-11T20:55:13.416 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 5.812378979, 2024-03-11T20:55:13.416 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.17030220200000001, 2024-03-11T20:55:13.417 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T20:55:13.417 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T20:55:13.417 INFO:teuthology.orchestra.run.smithi023.stdout: "client_info": { 2024-03-11T20:55:13.417 INFO:teuthology.orchestra.run.smithi023.stdout: "client": "client.24629", 2024-03-11T20:55:13.417 INFO:teuthology.orchestra.run.smithi023.stdout: "client_addr": "172.21.15.23:0/959732609", 2024-03-11T20:55:13.417 INFO:teuthology.orchestra.run.smithi023.stdout: "tid": 34155 2024-03-11T20:55:13.417 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.417 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T20:55:13.417 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.417 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T20:55:13.417 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:07.541459+0000", 2024-03-11T20:55:13.417 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:55:13.417 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.417 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.418 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T20:55:13.418 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:07.541459+0000", 2024-03-11T20:55:13.418 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:55:13.418 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.418 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.418 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T20:55:13.418 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:07.541461+0000", 2024-03-11T20:55:13.418 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.857e-06 2024-03-11T20:55:13.418 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.418 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.418 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T20:55:13.418 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:07.541485+0000", 2024-03-11T20:55:13.418 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.3881999999999999e-05 2024-03-11T20:55:13.418 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.418 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.419 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T20:55:13.419 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:07.541497+0000", 2024-03-11T20:55:13.419 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.2101e-05 2024-03-11T20:55:13.419 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.419 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.419 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T20:55:13.419 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:07.541505+0000", 2024-03-11T20:55:13.419 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.5780000000000001e-06 2024-03-11T20:55:13.419 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.419 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.419 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T20:55:13.419 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:07.541566+0000", 2024-03-11T20:55:13.419 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.1716000000000003e-05 2024-03-11T20:55:13.419 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.419 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.420 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T20:55:13.420 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:07.541690+0000", 2024-03-11T20:55:13.420 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000123232 2024-03-11T20:55:13.420 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.420 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.420 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for subops from 4,7", 2024-03-11T20:55:13.420 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:07.541818+0000", 2024-03-11T20:55:13.420 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00012846400000000001 2024-03-11T20:55:13.420 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.420 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.420 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "op_commit", 2024-03-11T20:55:13.420 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:07.543258+0000", 2024-03-11T20:55:13.420 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.001440076 2024-03-11T20:55:13.420 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.420 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.421 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T20:55:13.421 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:07.543518+0000", 2024-03-11T20:55:13.421 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00026030999999999998 2024-03-11T20:55:13.421 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.421 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.421 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T20:55:13.421 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:07.711304+0000", 2024-03-11T20:55:13.422 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.167785144 2024-03-11T20:55:13.422 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.422 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.422 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T20:55:13.422 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:07.711645+0000", 2024-03-11T20:55:13.422 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00034175399999999999 2024-03-11T20:55:13.422 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.422 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.422 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T20:55:13.422 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:07.711761+0000", 2024-03-11T20:55:13.422 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000116088 2024-03-11T20:55:13.422 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:55:13.422 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T20:55:13.422 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:55:13.423 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.423 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.423 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_op(client.24629.0:34161 7.9 7:931c88c7:::benchmark_data_smithi023_67959_object34160:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e134)", 2024-03-11T20:55:13.423 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T20:55:07.543468+0000", 2024-03-11T20:55:13.423 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 5.8103703250000001, 2024-03-11T20:55:13.423 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.168838078, 2024-03-11T20:55:13.423 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T20:55:13.423 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T20:55:13.423 INFO:teuthology.orchestra.run.smithi023.stdout: "client_info": { 2024-03-11T20:55:13.423 INFO:teuthology.orchestra.run.smithi023.stdout: "client": "client.24629", 2024-03-11T20:55:13.423 INFO:teuthology.orchestra.run.smithi023.stdout: "client_addr": "172.21.15.23:0/959732609", 2024-03-11T20:55:13.423 INFO:teuthology.orchestra.run.smithi023.stdout: "tid": 34161 2024-03-11T20:55:13.423 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.423 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T20:55:13.424 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.424 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T20:55:13.424 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:07.543468+0000", 2024-03-11T20:55:13.424 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:55:13.424 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.424 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.424 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T20:55:13.424 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:07.543468+0000", 2024-03-11T20:55:13.424 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:55:13.424 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.424 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.424 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T20:55:13.424 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:07.543469+0000", 2024-03-11T20:55:13.424 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.1030000000000001e-06 2024-03-11T20:55:13.424 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.425 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.425 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T20:55:13.425 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:07.543490+0000", 2024-03-11T20:55:13.425 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.0628999999999998e-05 2024-03-11T20:55:13.425 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.425 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.425 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T20:55:13.425 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:07.543499+0000", 2024-03-11T20:55:13.425 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 9.8360000000000004e-06 2024-03-11T20:55:13.425 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.425 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.425 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T20:55:13.425 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:07.543507+0000", 2024-03-11T20:55:13.425 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.4760000000000001e-06 2024-03-11T20:55:13.425 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.426 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.426 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T20:55:13.426 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:07.543553+0000", 2024-03-11T20:55:13.426 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 4.6474999999999998e-05 2024-03-11T20:55:13.426 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.426 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.426 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T20:55:13.426 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:07.543685+0000", 2024-03-11T20:55:13.426 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00013197400000000001 2024-03-11T20:55:13.426 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.426 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.426 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for subops from 4,7", 2024-03-11T20:55:13.426 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:07.543833+0000", 2024-03-11T20:55:13.426 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00014808599999999999 2024-03-11T20:55:13.427 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.427 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.427 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T20:55:13.427 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:07.711934+0000", 2024-03-11T20:55:13.427 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.16810059599999999 2024-03-11T20:55:13.427 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.427 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.427 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T20:55:13.427 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:07.712043+0000", 2024-03-11T20:55:13.427 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000108799 2024-03-11T20:55:13.427 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.427 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.427 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "op_commit", 2024-03-11T20:55:13.427 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:07.712096+0000", 2024-03-11T20:55:13.427 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 5.2982999999999997e-05 2024-03-11T20:55:13.428 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.428 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.428 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T20:55:13.428 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:07.712224+0000", 2024-03-11T20:55:13.428 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00012849000000000001 2024-03-11T20:55:13.428 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.428 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.428 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T20:55:13.428 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:07.712306+0000", 2024-03-11T20:55:13.428 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 8.1631000000000005e-05 2024-03-11T20:55:13.428 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:55:13.428 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T20:55:13.428 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:55:13.428 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.428 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.429 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_op(client.24629.0:34162 7.6 7:6d88eb87:::benchmark_data_smithi023_67959_object34161:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e134)", 2024-03-11T20:55:13.429 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T20:55:07.543723+0000", 2024-03-11T20:55:13.429 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 5.8101149979999995, 2024-03-11T20:55:13.429 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.17552870500000001, 2024-03-11T20:55:13.429 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T20:55:13.429 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T20:55:13.429 INFO:teuthology.orchestra.run.smithi023.stdout: "client_info": { 2024-03-11T20:55:13.429 INFO:teuthology.orchestra.run.smithi023.stdout: "client": "client.24629", 2024-03-11T20:55:13.429 INFO:teuthology.orchestra.run.smithi023.stdout: "client_addr": "172.21.15.23:0/959732609", 2024-03-11T20:55:13.429 INFO:teuthology.orchestra.run.smithi023.stdout: "tid": 34162 2024-03-11T20:55:13.429 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.429 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T20:55:13.429 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.429 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T20:55:13.430 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:07.543723+0000", 2024-03-11T20:55:13.430 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:55:13.430 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.430 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.430 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T20:55:13.430 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:07.543723+0000", 2024-03-11T20:55:13.430 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:55:13.430 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.430 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.430 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T20:55:13.430 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:07.543724+0000", 2024-03-11T20:55:13.430 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 8.2099999999999995e-07 2024-03-11T20:55:13.430 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.430 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.431 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T20:55:13.431 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:07.543747+0000", 2024-03-11T20:55:13.431 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.3298999999999998e-05 2024-03-11T20:55:13.431 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.431 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.431 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T20:55:13.431 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:07.543759+0000", 2024-03-11T20:55:13.431 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.1260999999999999e-05 2024-03-11T20:55:13.431 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.431 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.431 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T20:55:13.431 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:07.543767+0000", 2024-03-11T20:55:13.432 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 8.2479999999999996e-06 2024-03-11T20:55:13.432 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.432 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.432 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T20:55:13.432 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:07.543827+0000", 2024-03-11T20:55:13.432 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.0105000000000001e-05 2024-03-11T20:55:13.432 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.432 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.432 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T20:55:13.432 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:07.543937+0000", 2024-03-11T20:55:13.432 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000110424 2024-03-11T20:55:13.432 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.432 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.432 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for subops from 3,7", 2024-03-11T20:55:13.432 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:07.711705+0000", 2024-03-11T20:55:13.433 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.16776743999999999 2024-03-11T20:55:13.433 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.433 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.433 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T20:55:13.433 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:07.715902+0000", 2024-03-11T20:55:13.433 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0041968020000000003 2024-03-11T20:55:13.433 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.433 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.433 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T20:55:13.433 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:07.718924+0000", 2024-03-11T20:55:13.433 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0030222349999999999 2024-03-11T20:55:13.433 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.433 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.433 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "op_commit", 2024-03-11T20:55:13.433 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:07.719108+0000", 2024-03-11T20:55:13.434 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00018451599999999999 2024-03-11T20:55:13.434 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.434 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.434 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T20:55:13.434 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:07.719188+0000", 2024-03-11T20:55:13.434 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.9585999999999993e-05 2024-03-11T20:55:13.434 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.434 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.434 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T20:55:13.434 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:07.719252+0000", 2024-03-11T20:55:13.434 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.3967999999999995e-05 2024-03-11T20:55:13.434 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:55:13.434 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T20:55:13.434 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:55:13.435 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.435 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.435 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_op(client.24629.0:34165 7.f 7:fe05777b:::benchmark_data_smithi023_67959_object34164:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e134)", 2024-03-11T20:55:13.435 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T20:55:07.543982+0000", 2024-03-11T20:55:13.435 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 5.8098566009999999, 2024-03-11T20:55:13.435 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.179513908, 2024-03-11T20:55:13.435 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T20:55:13.435 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T20:55:13.435 INFO:teuthology.orchestra.run.smithi023.stdout: "client_info": { 2024-03-11T20:55:13.435 INFO:teuthology.orchestra.run.smithi023.stdout: "client": "client.24629", 2024-03-11T20:55:13.435 INFO:teuthology.orchestra.run.smithi023.stdout: "client_addr": "172.21.15.23:0/959732609", 2024-03-11T20:55:13.435 INFO:teuthology.orchestra.run.smithi023.stdout: "tid": 34165 2024-03-11T20:55:13.435 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.435 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T20:55:13.435 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.436 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T20:55:13.436 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:07.543982+0000", 2024-03-11T20:55:13.436 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:55:13.436 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.436 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.436 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T20:55:13.436 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:07.543982+0000", 2024-03-11T20:55:13.436 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:55:13.436 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.436 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.436 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T20:55:13.436 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:07.543982+0000", 2024-03-11T20:55:13.436 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 8.1299999999999999e-07 2024-03-11T20:55:13.436 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.437 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.437 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T20:55:13.437 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:07.544007+0000", 2024-03-11T20:55:13.437 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.4740999999999999e-05 2024-03-11T20:55:13.437 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.437 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.437 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T20:55:13.437 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:07.544017+0000", 2024-03-11T20:55:13.437 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 9.4380000000000001e-06 2024-03-11T20:55:13.437 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.437 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.437 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T20:55:13.438 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:07.544024+0000", 2024-03-11T20:55:13.438 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.0679999999999999e-06 2024-03-11T20:55:13.438 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.438 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.438 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T20:55:13.438 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:07.544083+0000", 2024-03-11T20:55:13.438 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 5.8990999999999998e-05 2024-03-11T20:55:13.438 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.438 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.438 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T20:55:13.438 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:07.711436+0000", 2024-03-11T20:55:13.438 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.16735372100000001 2024-03-11T20:55:13.438 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.438 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.439 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for subops from 3,7", 2024-03-11T20:55:13.439 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:07.711767+0000", 2024-03-11T20:55:13.439 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000330854 2024-03-11T20:55:13.439 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.439 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.439 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T20:55:13.439 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:07.720221+0000", 2024-03-11T20:55:13.439 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0084536239999999999 2024-03-11T20:55:13.439 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.439 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.439 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "op_commit", 2024-03-11T20:55:13.439 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:07.720329+0000", 2024-03-11T20:55:13.439 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00010857399999999999 2024-03-11T20:55:13.439 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.439 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.440 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T20:55:13.440 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:07.723349+0000", 2024-03-11T20:55:13.440 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0030192919999999998 2024-03-11T20:55:13.440 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.440 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.440 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T20:55:13.440 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:07.723432+0000", 2024-03-11T20:55:13.440 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 8.3177999999999995e-05 2024-03-11T20:55:13.440 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.440 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.440 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T20:55:13.440 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:07.723495+0000", 2024-03-11T20:55:13.440 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.3614000000000005e-05 2024-03-11T20:55:13.440 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:55:13.441 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T20:55:13.441 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:55:13.441 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.441 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.441 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_repop_reply(client.24629.0:34155 7.9 e134/124 ondisk, result = 0)", 2024-03-11T20:55:13.441 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T20:55:07.546834+0000", 2024-03-11T20:55:13.441 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 5.8070046409999998, 2024-03-11T20:55:13.441 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.16495328100000001, 2024-03-11T20:55:13.441 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T20:55:13.441 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "started", 2024-03-11T20:55:13.441 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T20:55:13.441 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.441 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T20:55:13.441 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:07.546834+0000", 2024-03-11T20:55:13.441 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:55:13.442 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.442 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.442 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T20:55:13.442 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:07.546834+0000", 2024-03-11T20:55:13.442 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:55:13.442 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.442 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.442 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T20:55:13.442 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:07.546834+0000", 2024-03-11T20:55:13.442 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 8.2200000000000003e-07 2024-03-11T20:55:13.442 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.442 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.442 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T20:55:13.442 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:07.546839+0000", 2024-03-11T20:55:13.443 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 4.5260000000000004e-06 2024-03-11T20:55:13.443 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.443 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.443 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T20:55:13.443 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:07.711111+0000", 2024-03-11T20:55:13.443 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.16427253 2024-03-11T20:55:13.443 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.443 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.443 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T20:55:13.443 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:07.711141+0000", 2024-03-11T20:55:13.443 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.9845999999999999e-05 2024-03-11T20:55:13.443 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.443 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.443 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T20:55:13.443 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:07.711269+0000", 2024-03-11T20:55:13.444 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00012764100000000001 2024-03-11T20:55:13.444 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.444 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.444 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T20:55:13.444 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:07.711289+0000", 2024-03-11T20:55:13.444 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.0091000000000001e-05 2024-03-11T20:55:13.444 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.444 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.444 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T20:55:13.444 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:07.711787+0000", 2024-03-11T20:55:13.444 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00049782500000000003 2024-03-11T20:55:13.444 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:55:13.444 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T20:55:13.444 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:55:13.444 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.445 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.445 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_repop_reply(client.24629.0:34161 7.9 e134/124 ondisk, result = 0)", 2024-03-11T20:55:13.445 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T20:55:07.547143+0000", 2024-03-11T20:55:13.445 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 5.8066950779999997, 2024-03-11T20:55:13.445 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.164817939, 2024-03-11T20:55:13.445 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T20:55:13.445 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "started", 2024-03-11T20:55:13.445 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T20:55:13.445 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.445 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T20:55:13.445 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:07.547143+0000", 2024-03-11T20:55:13.445 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:55:13.445 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.445 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.445 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T20:55:13.446 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:07.547143+0000", 2024-03-11T20:55:13.446 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:55:13.446 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.446 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.446 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T20:55:13.446 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:07.547144+0000", 2024-03-11T20:55:13.446 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.6599999999999996e-07 2024-03-11T20:55:13.446 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.446 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.446 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T20:55:13.446 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:07.547147+0000", 2024-03-11T20:55:13.446 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 3.7270000000000001e-06 2024-03-11T20:55:13.446 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.446 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.447 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T20:55:13.447 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:07.711140+0000", 2024-03-11T20:55:13.447 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.16399266500000001 2024-03-11T20:55:13.447 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.447 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.447 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T20:55:13.447 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:07.711167+0000", 2024-03-11T20:55:13.447 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.6630000000000001e-05 2024-03-11T20:55:13.447 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.447 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.447 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T20:55:13.447 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:07.711849+0000", 2024-03-11T20:55:13.447 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00068214100000000004 2024-03-11T20:55:13.447 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.447 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.447 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T20:55:13.448 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:07.711919+0000", 2024-03-11T20:55:13.448 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.0036000000000005e-05 2024-03-11T20:55:13.448 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.448 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.448 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T20:55:13.448 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:07.711961+0000", 2024-03-11T20:55:13.448 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 4.2073999999999997e-05 2024-03-11T20:55:13.448 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:55:13.448 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T20:55:13.448 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:55:13.448 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.448 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.448 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_op(client.24629.0:39254 7.6 7:6ddd3edd:::benchmark_data_smithi023_67959_object39253:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-03-11T20:55:13.448 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T20:55:11.008293+0000", 2024-03-11T20:55:13.449 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 2.3455450849999999, 2024-03-11T20:55:13.449 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.16090918800000001, 2024-03-11T20:55:13.449 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T20:55:13.449 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T20:55:13.449 INFO:teuthology.orchestra.run.smithi023.stdout: "client_info": { 2024-03-11T20:55:13.449 INFO:teuthology.orchestra.run.smithi023.stdout: "client": "client.24629", 2024-03-11T20:55:13.449 INFO:teuthology.orchestra.run.smithi023.stdout: "client_addr": "172.21.15.23:0/959732609", 2024-03-11T20:55:13.449 INFO:teuthology.orchestra.run.smithi023.stdout: "tid": 39254 2024-03-11T20:55:13.449 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.449 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T20:55:13.449 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.449 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T20:55:13.449 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:11.008293+0000", 2024-03-11T20:55:13.449 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:55:13.449 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.450 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.450 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T20:55:13.450 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:11.008293+0000", 2024-03-11T20:55:13.450 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:55:13.450 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.450 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.450 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T20:55:13.450 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:11.008294+0000", 2024-03-11T20:55:13.450 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.9599999999999999e-07 2024-03-11T20:55:13.450 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.450 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.450 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T20:55:13.450 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:11.008314+0000", 2024-03-11T20:55:13.450 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.0186e-05 2024-03-11T20:55:13.451 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.451 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.451 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T20:55:13.451 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:11.008324+0000", 2024-03-11T20:55:13.451 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 9.5400000000000001e-06 2024-03-11T20:55:13.451 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.451 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.451 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T20:55:13.451 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:11.008330+0000", 2024-03-11T20:55:13.451 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.8079999999999999e-06 2024-03-11T20:55:13.451 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.451 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.451 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T20:55:13.451 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:11.008446+0000", 2024-03-11T20:55:13.451 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000115882 2024-03-11T20:55:13.452 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.452 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.452 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T20:55:13.452 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:11.008556+0000", 2024-03-11T20:55:13.452 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000110223 2024-03-11T20:55:13.452 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.452 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.452 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for subops from 3,7", 2024-03-11T20:55:13.452 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:11.008688+0000", 2024-03-11T20:55:13.452 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000131948 2024-03-11T20:55:13.452 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.452 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.452 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T20:55:13.452 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:11.011776+0000", 2024-03-11T20:55:13.452 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0030871850000000001 2024-03-11T20:55:13.453 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.453 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.453 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "op_commit", 2024-03-11T20:55:13.453 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:11.015495+0000", 2024-03-11T20:55:13.453 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.003719352 2024-03-11T20:55:13.453 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.453 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.453 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T20:55:13.453 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:11.168943+0000", 2024-03-11T20:55:13.453 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.15344817399999999 2024-03-11T20:55:13.453 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.453 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.453 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T20:55:13.453 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:11.169124+0000", 2024-03-11T20:55:13.453 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00018100599999999999 2024-03-11T20:55:13.454 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.454 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.454 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T20:55:13.454 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:11.169202+0000", 2024-03-11T20:55:13.454 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.8188000000000004e-05 2024-03-11T20:55:13.454 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:55:13.454 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T20:55:13.454 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:55:13.454 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.454 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.454 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_op(client.24629.0:39257 7.6 7:6a08cc7f:::benchmark_data_smithi023_67959_object39256:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-03-11T20:55:13.454 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T20:55:11.011607+0000", 2024-03-11T20:55:13.454 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 2.342231204, 2024-03-11T20:55:13.454 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.15780493700000001, 2024-03-11T20:55:13.455 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T20:55:13.455 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T20:55:13.455 INFO:teuthology.orchestra.run.smithi023.stdout: "client_info": { 2024-03-11T20:55:13.455 INFO:teuthology.orchestra.run.smithi023.stdout: "client": "client.24629", 2024-03-11T20:55:13.455 INFO:teuthology.orchestra.run.smithi023.stdout: "client_addr": "172.21.15.23:0/959732609", 2024-03-11T20:55:13.455 INFO:teuthology.orchestra.run.smithi023.stdout: "tid": 39257 2024-03-11T20:55:13.455 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.455 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T20:55:13.455 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.455 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T20:55:13.455 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:11.011607+0000", 2024-03-11T20:55:13.455 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:55:13.455 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.455 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.456 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T20:55:13.456 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:11.011607+0000", 2024-03-11T20:55:13.456 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:55:13.456 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.456 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.456 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T20:55:13.456 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:11.011608+0000", 2024-03-11T20:55:13.456 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.85e-07 2024-03-11T20:55:13.456 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.456 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.456 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T20:55:13.456 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:11.011633+0000", 2024-03-11T20:55:13.456 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.4899e-05 2024-03-11T20:55:13.456 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.456 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.457 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T20:55:13.457 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:11.011643+0000", 2024-03-11T20:55:13.457 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.0524e-05 2024-03-11T20:55:13.457 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.457 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.457 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T20:55:13.457 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:11.011650+0000", 2024-03-11T20:55:13.457 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.1169999999999998e-06 2024-03-11T20:55:13.457 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.457 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.457 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T20:55:13.457 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:11.011813+0000", 2024-03-11T20:55:13.457 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0001627 2024-03-11T20:55:13.457 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.457 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.458 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T20:55:13.458 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:11.011957+0000", 2024-03-11T20:55:13.458 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000143716 2024-03-11T20:55:13.458 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.458 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.458 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for subops from 3,7", 2024-03-11T20:55:13.458 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:11.012097+0000", 2024-03-11T20:55:13.458 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000139939 2024-03-11T20:55:13.458 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.458 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.458 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T20:55:13.458 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:11.015536+0000", 2024-03-11T20:55:13.458 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0034397170000000001 2024-03-11T20:55:13.458 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.458 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.459 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "op_commit", 2024-03-11T20:55:13.459 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:11.016705+0000", 2024-03-11T20:55:13.459 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.001168582 2024-03-11T20:55:13.459 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.459 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.459 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T20:55:13.459 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:11.169280+0000", 2024-03-11T20:55:13.459 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.152574767 2024-03-11T20:55:13.459 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.459 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.459 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T20:55:13.459 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:11.169356+0000", 2024-03-11T20:55:13.459 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.6515999999999997e-05 2024-03-11T20:55:13.459 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.459 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.460 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T20:55:13.460 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:11.169412+0000", 2024-03-11T20:55:13.460 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 5.5674999999999998e-05 2024-03-11T20:55:13.460 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:55:13.460 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T20:55:13.460 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:55:13.460 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.460 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.460 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_op(client.24629.0:39264 7.6 7:6c23b54a:::benchmark_data_smithi023_67959_object39263:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-03-11T20:55:13.460 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T20:55:11.018597+0000", 2024-03-11T20:55:13.460 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 2.3352414490000002, 2024-03-11T20:55:13.460 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.157180764, 2024-03-11T20:55:13.460 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T20:55:13.460 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T20:55:13.460 INFO:teuthology.orchestra.run.smithi023.stdout: "client_info": { 2024-03-11T20:55:13.461 INFO:teuthology.orchestra.run.smithi023.stdout: "client": "client.24629", 2024-03-11T20:55:13.461 INFO:teuthology.orchestra.run.smithi023.stdout: "client_addr": "172.21.15.23:0/959732609", 2024-03-11T20:55:13.461 INFO:teuthology.orchestra.run.smithi023.stdout: "tid": 39264 2024-03-11T20:55:13.461 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.462 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T20:55:13.462 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.462 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T20:55:13.462 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:11.018597+0000", 2024-03-11T20:55:13.462 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:55:13.462 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.462 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.462 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T20:55:13.462 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:11.018597+0000", 2024-03-11T20:55:13.462 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:55:13.462 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.462 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.462 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T20:55:13.462 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:11.018598+0000", 2024-03-11T20:55:13.463 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.226e-06 2024-03-11T20:55:13.463 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.463 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.463 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T20:55:13.463 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:11.018616+0000", 2024-03-11T20:55:13.463 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.8301999999999999e-05 2024-03-11T20:55:13.463 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.463 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.463 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T20:55:13.463 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:11.018628+0000", 2024-03-11T20:55:13.463 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.1522000000000001e-05 2024-03-11T20:55:13.463 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.463 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.463 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T20:55:13.463 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:11.018634+0000", 2024-03-11T20:55:13.464 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 5.7470000000000002e-06 2024-03-11T20:55:13.464 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.464 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.464 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T20:55:13.464 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:11.018673+0000", 2024-03-11T20:55:13.464 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 3.9512999999999999e-05 2024-03-11T20:55:13.464 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.464 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.464 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T20:55:13.464 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:11.018754+0000", 2024-03-11T20:55:13.464 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 8.1099000000000004e-05 2024-03-11T20:55:13.464 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.464 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.464 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for subops from 3,7", 2024-03-11T20:55:13.464 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:11.018839+0000", 2024-03-11T20:55:13.465 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 8.5086000000000005e-05 2024-03-11T20:55:13.465 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.465 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.465 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "op_commit", 2024-03-11T20:55:13.465 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:11.020108+0000", 2024-03-11T20:55:13.465 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.001269163 2024-03-11T20:55:13.465 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.465 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.465 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T20:55:13.465 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:11.020224+0000", 2024-03-11T20:55:13.465 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000115245 2024-03-11T20:55:13.465 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.465 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.465 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T20:55:13.465 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:11.175636+0000", 2024-03-11T20:55:13.466 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.15541218200000001 2024-03-11T20:55:13.466 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.466 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.466 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T20:55:13.466 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:11.175734+0000", 2024-03-11T20:55:13.466 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 9.8380000000000003e-05 2024-03-11T20:55:13.466 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:55:13.466 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:55:13.466 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T20:55:13.466 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:11.175777+0000", 2024-03-11T20:55:13.466 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 4.3299e-05 2024-03-11T20:55:13.466 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:55:13.466 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T20:55:13.466 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:55:13.466 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:55:13.467 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T20:55:13.467 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-11T20:55:13.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:13.467 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:13.468 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:55:13.542 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 22 16 42901 42885 121.801 86.8125 0.0210297 0.00820362 2024-03-11T20:55:13.568 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:13.581 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-11T20:55:13.584 INFO:teuthology.orchestra.run.smithi023.stderr:dumped all 2024-03-11T20:55:13.592 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:13.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:13.595 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:55:13.696 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:13.720 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:13.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:13.722 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:55:13.823 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:13.850 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:13.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:13.853 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:55:13.954 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:13.979 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:13.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:13.982 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:55:14.082 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:14.115 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:14.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:14.118 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:55:14.219 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:14.245 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:14.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:14.246 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:55:14.347 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:55:14.370 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:55:14.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:14.373 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:55:14.474 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:14.500 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:14.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:14.503 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:55:14.540 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 23 16 44495 44479 120.837 99.625 0.00623479 0.0082694 2024-03-11T20:55:14.604 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:14.632 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:14.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:14.635 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:55:14.736 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:14.759 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:14.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:14.762 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:55:14.863 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:55:14.887 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:55:14.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:14.890 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:55:14.991 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:15.018 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:15.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:15.020 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:55:15.121 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:55:15.153 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:55:15.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:15.156 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:55:15.259 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:55:15.260 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 20:55:15 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T20:55:15.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:14 smithi146 bash[23809]: audit 2024-03-11T20:55:13.549012+0000 mgr.y (mgr.24380) 169 : audit 0 from='client.24647 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T20:55:15.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:14 smithi146 bash[23809]: cluster 2024-03-11T20:55:13.907038+0000 mgr.y (mgr.24380) 170 : cluster 0 pgmap v147: 148 pgs: 148 active+clean; 2.3 GiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 622 B/s rd, 64 MiB/s wr, 1.02k op/s 2024-03-11T20:55:15.282 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:55:15.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:15.284 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:55:15.308 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:14 smithi023 bash[21372]: audit 2024-03-11T20:55:13.549012+0000 mgr.y (mgr.24380) 169 : audit 0 from='client.24647 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T20:55:15.308 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:14 smithi023 bash[21372]: cluster 2024-03-11T20:55:13.907038+0000 mgr.y (mgr.24380) 170 : cluster 0 pgmap v147: 148 pgs: 148 active+clean; 2.3 GiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 622 B/s rd, 64 MiB/s wr, 1.02k op/s 2024-03-11T20:55:15.309 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:15 smithi023 bash[29146]: audit 2024-03-11T20:55:13.549012+0000 mgr.y (mgr.24380) 169 : audit 0 from='client.24647 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T20:55:15.309 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:15 smithi023 bash[29146]: cluster 2024-03-11T20:55:13.907038+0000 mgr.y (mgr.24380) 170 : cluster 0 pgmap v147: 148 pgs: 148 active+clean; 2.3 GiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 622 B/s rd, 64 MiB/s wr, 1.02k op/s 2024-03-11T20:55:15.384 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:55:15.418 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:55:15.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:15.420 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:55:15.521 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:15.540 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 24 16 45812 45796 119.231 82.3125 0.0157978 0.00837993 2024-03-11T20:55:15.556 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:15.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:15.558 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:55:15.659 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:15.688 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:15.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:15.691 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:55:15.791 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:55:15.815 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:55:15.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:15.818 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:55:15.919 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:15.945 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:15.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:15.949 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:55:16.050 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:55:16.059 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:15 smithi146 bash[23809]: audit 2024-03-11T20:55:15.013255+0000 mgr.y (mgr.24380) 171 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:55:16.079 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:55:16.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:16.082 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:55:16.183 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:55:16.203 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:55:16.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:16.205 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:55:16.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:16 smithi023 bash[21372]: audit 2024-03-11T20:55:15.013255+0000 mgr.y (mgr.24380) 171 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:55:16.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:16 smithi023 bash[29146]: audit 2024-03-11T20:55:15.013255+0000 mgr.y (mgr.24380) 171 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:55:16.306 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:16.330 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:16.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:16.332 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:55:16.432 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:16.466 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:16.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:16.468 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:55:16.540 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 25 16 46809 46793 116.955 62.3125 0.0103254 0.00854423 2024-03-11T20:55:16.569 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:16.587 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:16.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:16.589 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:55:16.690 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:16.724 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:16.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:16.727 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:55:16.828 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:16.847 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:16.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:16.849 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:55:16.950 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:16.980 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:16.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:16.983 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:55:17.084 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:55:17.114 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:55:17.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:17.117 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:55:17.218 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:17.244 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:17.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:17.247 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:55:17.247 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:16 smithi146 bash[23809]: cluster 2024-03-11T20:55:15.908019+0000 mgr.y (mgr.24380) 172 : cluster 0 pgmap v148: 148 pgs: 148 active+clean; 2.7 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 1023 B/s rd, 98 MiB/s wr, 1.58k op/s 2024-03-11T20:55:17.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:17 smithi023 bash[21372]: cluster 2024-03-11T20:55:15.908019+0000 mgr.y (mgr.24380) 172 : cluster 0 pgmap v148: 148 pgs: 148 active+clean; 2.7 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 1023 B/s rd, 98 MiB/s wr, 1.58k op/s 2024-03-11T20:55:17.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:17 smithi023 bash[29146]: cluster 2024-03-11T20:55:15.908019+0000 mgr.y (mgr.24380) 172 : cluster 0 pgmap v148: 148 pgs: 148 active+clean; 2.7 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 1023 B/s rd, 98 MiB/s wr, 1.58k op/s 2024-03-11T20:55:17.348 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:17.376 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:17.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:17.378 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:55:17.479 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:17.506 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:17.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:17.511 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:55:17.540 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 26 16 47958 47942 115.218 71.8125 0.00538603 0.0086729 2024-03-11T20:55:17.612 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:55:17.630 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:55:17.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:17.632 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:55:17.733 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:17.764 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:17.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:17.769 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:55:17.870 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:17.898 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:17.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:17.902 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:55:18.002 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:18.029 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:18.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:18.033 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:55:18.133 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:18.153 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:18.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:18.155 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:55:18.255 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:55:18.280 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:55:18.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:18.282 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:55:18.383 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:18.409 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:18.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:18.413 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:55:18.514 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:18.541 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 27 16 49182 49166 113.783 76.5 0.00829448 0.00878184 2024-03-11T20:55:18.543 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:18.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:18.546 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:55:18.647 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:18.657 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:55:18 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:20:55:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T20:55:18.672 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:18.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:18.674 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:55:18.775 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:18.799 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:18.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:18.801 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:55:18.902 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:55:18.921 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:55:18.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:18.922 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:55:19.020 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-11T20:55:19.023 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:19.047 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:19.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:19.050 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:55:19.150 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:19 smithi023 bash[21372]: cluster 2024-03-11T20:55:17.908448+0000 mgr.y (mgr.24380) 173 : cluster 0 pgmap v149: 148 pgs: 148 active+clean; 2.7 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 1023 B/s rd, 98 MiB/s wr, 1.58k op/s 2024-03-11T20:55:19.150 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:55:19.151 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:19 smithi023 bash[29146]: cluster 2024-03-11T20:55:17.908448+0000 mgr.y (mgr.24380) 173 : cluster 0 pgmap v149: 148 pgs: 148 active+clean; 2.7 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 1023 B/s rd, 98 MiB/s wr, 1.58k op/s 2024-03-11T20:55:19.166 INFO:teuthology.orchestra.run.smithi023.stdout:{"pg_ready":true,"pg_map":{"version":146,"stamp":"2024-03-11T20:55:11.906321+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2466191784,"num_objects":37820,"num_object_clones":0,"num_object_copies":113460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37820,"num_whiteouts":0,"num_read":1206,"num_read_kb":1052,"num_write":38237,"num_write_kb":2410269,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":1773908,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38311,"ondisk_log_size":38311,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":7393344,"kb_used_data":7057096,"kb_used_omap":10,"kb_used_meta":336117,"kb_avail":648822720,"statfs":{"total":671965249536,"available":664394465280,"internally_reserved":0,"allocated":7226466304,"data_stored":7223333157,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11155,"internal_metadata":344183917},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,3,10,3,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1162870784,"num_objects":17744,"num_object_clones":0,"num_object_copies":53232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17744,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":17744,"num_write_kb":1135616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.244552"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":139,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.672022+0000","last_change":"2024-03-11T20:54:27.491847+0000","last_active":"2024-03-11T20:55:09.672022+0000","last_peered":"2024-03-11T20:55:09.672022+0000","last_clean":"2024-03-11T20:55:09.672022+0000","last_became_active":"2024-03-11T20:50:34.338390+0000","last_became_peered":"2024-03-11T20:50:34.338390+0000","last_unstale":"2024-03-11T20:55:09.672022+0000","last_undegraded":"2024-03-11T20:55:09.672022+0000","last_fullsized":"2024-03-11T20:55:09.672022+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:54:27.491741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:54:27.491741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:37.803402+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":143,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.672014+0000","last_change":"2024-03-11T20:54:29.532825+0000","last_active":"2024-03-11T20:55:09.672014+0000","last_peered":"2024-03-11T20:55:09.672014+0000","last_clean":"2024-03-11T20:55:09.672014+0000","last_became_active":"2024-03-11T20:50:32.210194+0000","last_became_peered":"2024-03-11T20:50:32.210194+0000","last_unstale":"2024-03-11T20:55:09.672014+0000","last_undegraded":"2024-03-11T20:55:09.672014+0000","last_fullsized":"2024-03-11T20:55:09.672014+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:54:29.532707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:54:29.532707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:30.113091+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":144,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.677855+0000","last_change":"2024-03-11T20:53:29.965205+0000","last_active":"2024-03-11T20:55:09.677855+0000","last_peered":"2024-03-11T20:55:09.677855+0000","last_clean":"2024-03-11T20:55:09.677855+0000","last_became_active":"2024-03-11T20:50:28.061739+0000","last_became_peered":"2024-03-11T20:50:28.061739+0000","last_unstale":"2024-03-11T20:55:09.677855+0000","last_undegraded":"2024-03-11T20:55:09.677855+0000","last_fullsized":"2024-03-11T20:55:09.677855+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:29.964907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:53:29.964907+0000","last_clean_scrub_stamp":"2024-03-11T20:53:29.964907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:12.947817+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"84'10","reported_seq":155,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.673547+0000","last_change":"2024-03-11T20:53:11.430067+0000","last_active":"2024-03-11T20:55:09.673547+0000","last_peered":"2024-03-11T20:55:09.673547+0000","last_clean":"2024-03-11T20:55:09.673547+0000","last_became_active":"2024-03-11T20:50:30.101721+0000","last_became_peered":"2024-03-11T20:50:30.101721+0000","last_unstale":"2024-03-11T20:55:09.673547+0000","last_undegraded":"2024-03-11T20:55:09.673547+0000","last_fullsized":"2024-03-11T20:55:09.673547+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'10","last_scrub_stamp":"2024-03-11T20:53:11.429944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:53:11.429944+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":138,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.669870+0000","last_change":"2024-03-11T20:54:37.375230+0000","last_active":"2024-03-11T20:55:09.669870+0000","last_peered":"2024-03-11T20:55:09.669870+0000","last_clean":"2024-03-11T20:55:09.669870+0000","last_became_active":"2024-03-11T20:54:37.373963+0000","last_became_peered":"2024-03-11T20:54:37.373963+0000","last_unstale":"2024-03-11T20:55:09.669870+0000","last_undegraded":"2024-03-11T20:55:09.669870+0000","last_fullsized":"2024-03-11T20:55:09.669870+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:07.560731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:55.545758+0000","last_clean_scrub_stamp":"2024-03-11T20:53:07.560731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:12.376407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":150,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.670022+0000","last_change":"2024-03-11T20:54:37.391524+0000","last_active":"2024-03-11T20:55:09.670022+0000","last_peered":"2024-03-11T20:55:09.670022+0000","last_clean":"2024-03-11T20:55:09.670022+0000","last_became_active":"2024-03-11T20:54:37.390971+0000","last_became_peered":"2024-03-11T20:54:37.390971+0000","last_unstale":"2024-03-11T20:55:09.670022+0000","last_undegraded":"2024-03-11T20:55:09.670022+0000","last_fullsized":"2024-03-11T20:55:09.670022+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:54:13.660325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:54:13.660325+0000","last_clean_scrub_stamp":"2024-03-11T20:54:13.660325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:23.914414+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":144,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.672700+0000","last_change":"2024-03-11T20:53:54.309458+0000","last_active":"2024-03-11T20:55:09.672700+0000","last_peered":"2024-03-11T20:55:09.672700+0000","last_clean":"2024-03-11T20:55:09.672700+0000","last_became_active":"2024-03-11T20:50:28.058274+0000","last_became_peered":"2024-03-11T20:50:28.058274+0000","last_unstale":"2024-03-11T20:55:09.672700+0000","last_undegraded":"2024-03-11T20:55:09.672700+0000","last_fullsized":"2024-03-11T20:55:09.672700+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:54.309358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:53:54.309358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"84'11","reported_seq":159,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.677764+0000","last_change":"2024-03-11T20:53:44.981477+0000","last_active":"2024-03-11T20:55:09.677764+0000","last_peered":"2024-03-11T20:55:09.677764+0000","last_clean":"2024-03-11T20:55:09.677764+0000","last_became_active":"2024-03-11T20:50:30.092638+0000","last_became_peered":"2024-03-11T20:50:30.092638+0000","last_unstale":"2024-03-11T20:55:09.677764+0000","last_undegraded":"2024-03-11T20:55:09.677764+0000","last_fullsized":"2024-03-11T20:55:09.677764+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T20:53:44.981383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:53:44.981383+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:12.947728+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":138,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.674413+0000","last_change":"2024-03-11T20:54:37.413070+0000","last_active":"2024-03-11T20:55:09.674413+0000","last_peered":"2024-03-11T20:55:09.674413+0000","last_clean":"2024-03-11T20:55:09.674413+0000","last_became_active":"2024-03-11T20:54:37.412201+0000","last_became_peered":"2024-03-11T20:54:37.412201+0000","last_unstale":"2024-03-11T20:55:09.674413+0000","last_undegraded":"2024-03-11T20:55:09.674413+0000","last_fullsized":"2024-03-11T20:55:09.674413+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:07.792697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:53:07.792697+0000","last_clean_scrub_stamp":"2024-03-11T20:53:07.792697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:11.510960+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":142,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.671914+0000","last_change":"2024-03-11T20:54:37.368680+0000","last_active":"2024-03-11T20:55:09.671914+0000","last_peered":"2024-03-11T20:55:09.671914+0000","last_clean":"2024-03-11T20:55:09.671914+0000","last_became_active":"2024-03-11T20:54:37.366782+0000","last_became_peered":"2024-03-11T20:54:37.366782+0000","last_unstale":"2024-03-11T20:55:09.671914+0000","last_undegraded":"2024-03-11T20:55:09.671914+0000","last_fullsized":"2024-03-11T20:55:09.671914+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:46.583185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:53:46.583185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:12.376587+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":144,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.678172+0000","last_change":"2024-03-11T20:53:29.323499+0000","last_active":"2024-03-11T20:55:09.678172+0000","last_peered":"2024-03-11T20:55:09.678172+0000","last_clean":"2024-03-11T20:55:09.678172+0000","last_became_active":"2024-03-11T20:50:28.079225+0000","last_became_peered":"2024-03-11T20:50:28.079225+0000","last_unstale":"2024-03-11T20:55:09.678172+0000","last_undegraded":"2024-03-11T20:55:09.678172+0000","last_fullsized":"2024-03-11T20:55:09.678172+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:29.323273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:53:29.323273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:12.204699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"84'15","reported_seq":187,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.674887+0000","last_change":"2024-03-11T20:54:37.411027+0000","last_active":"2024-03-11T20:55:09.674887+0000","last_peered":"2024-03-11T20:55:09.674887+0000","last_clean":"2024-03-11T20:55:09.674887+0000","last_became_active":"2024-03-11T20:54:37.392367+0000","last_became_peered":"2024-03-11T20:54:37.392367+0000","last_unstale":"2024-03-11T20:55:09.674887+0000","last_undegraded":"2024-03-11T20:55:09.674887+0000","last_fullsized":"2024-03-11T20:55:09.674887+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15","last_scrub_stamp":"2024-03-11T20:53:29.817112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:53:29.817112+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:11.510929+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":145,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.676671+0000","last_change":"2024-03-11T20:54:37.421437+0000","last_active":"2024-03-11T20:55:09.676671+0000","last_peered":"2024-03-11T20:55:09.676671+0000","last_clean":"2024-03-11T20:55:09.676671+0000","last_became_active":"2024-03-11T20:54:37.421087+0000","last_became_peered":"2024-03-11T20:54:37.421087+0000","last_unstale":"2024-03-11T20:55:09.676671+0000","last_undegraded":"2024-03-11T20:55:09.676671+0000","last_fullsized":"2024-03-11T20:55:09.676671+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:54:26.670647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:54:26.670647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:49.768618+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":136,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.670300+0000","last_change":"2024-03-11T20:53:50.599246+0000","last_active":"2024-03-11T20:55:09.670300+0000","last_peered":"2024-03-11T20:55:09.670300+0000","last_clean":"2024-03-11T20:55:09.670300+0000","last_became_active":"2024-03-11T20:50:32.200126+0000","last_became_peered":"2024-03-11T20:50:32.200126+0000","last_unstale":"2024-03-11T20:55:09.670300+0000","last_undegraded":"2024-03-11T20:55:09.670300+0000","last_fullsized":"2024-03-11T20:55:09.670300+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:50.599029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:53:50.599029+0000","last_clean_scrub_stamp":"2024-03-11T20:53:50.599029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:12.499095+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":152,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.677636+0000","last_change":"2024-03-11T20:54:07.155380+0000","last_active":"2024-03-11T20:55:09.677636+0000","last_peered":"2024-03-11T20:55:09.677636+0000","last_clean":"2024-03-11T20:55:09.677636+0000","last_became_active":"2024-03-11T20:50:28.068978+0000","last_became_peered":"2024-03-11T20:50:28.068978+0000","last_unstale":"2024-03-11T20:55:09.677636+0000","last_undegraded":"2024-03-11T20:55:09.677636+0000","last_fullsized":"2024-03-11T20:55:09.677636+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:54:07.155130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:54:07.155130+0000","last_clean_scrub_stamp":"2024-03-11T20:54:07.155130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:15.457269+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"110'17","reported_seq":168,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.675092+0000","last_change":"2024-03-11T20:53:04.802853+0000","last_active":"2024-03-11T20:55:09.675092+0000","last_peered":"2024-03-11T20:55:09.675092+0000","last_clean":"2024-03-11T20:55:09.675092+0000","last_became_active":"2024-03-11T20:50:30.108364+0000","last_became_peered":"2024-03-11T20:50:30.108364+0000","last_unstale":"2024-03-11T20:55:09.675092+0000","last_undegraded":"2024-03-11T20:55:09.675092+0000","last_fullsized":"2024-03-11T20:55:09.675092+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'17","last_scrub_stamp":"2024-03-11T20:53:04.802488+0000","last_deep_scrub":"110'17","last_deep_scrub_stamp":"2024-03-11T20:53:04.802488+0000","last_clean_scrub_stamp":"2024-03-11T20:53:04.802488+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:11.510866+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":132,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.674633+0000","last_change":"2024-03-11T20:53:38.215774+0000","last_active":"2024-03-11T20:55:09.674633+0000","last_peered":"2024-03-11T20:55:09.674633+0000","last_clean":"2024-03-11T20:55:09.674633+0000","last_became_active":"2024-03-11T20:50:34.334558+0000","last_became_peered":"2024-03-11T20:50:34.334558+0000","last_unstale":"2024-03-11T20:55:09.674633+0000","last_undegraded":"2024-03-11T20:55:09.674633+0000","last_fullsized":"2024-03-11T20:55:09.674633+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:38.215565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:53:38.215565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":141,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.672160+0000","last_change":"2024-03-11T20:54:37.417457+0000","last_active":"2024-03-11T20:55:09.672160+0000","last_peered":"2024-03-11T20:55:09.672160+0000","last_clean":"2024-03-11T20:55:09.672160+0000","last_became_active":"2024-03-11T20:54:37.417106+0000","last_became_peered":"2024-03-11T20:54:37.417106+0000","last_unstale":"2024-03-11T20:55:09.672160+0000","last_undegraded":"2024-03-11T20:55:09.672160+0000","last_fullsized":"2024-03-11T20:55:09.672160+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:59.455677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:52:59.455677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"77'1","reported_seq":151,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.672248+0000","last_change":"2024-03-11T20:53:30.341375+0000","last_active":"2024-03-11T20:55:09.672248+0000","last_peered":"2024-03-11T20:55:09.672248+0000","last_clean":"2024-03-11T20:55:09.672248+0000","last_became_active":"2024-03-11T20:50:28.068840+0000","last_became_peered":"2024-03-11T20:50:28.068840+0000","last_unstale":"2024-03-11T20:55:09.672248+0000","last_undegraded":"2024-03-11T20:55:09.672248+0000","last_fullsized":"2024-03-11T20:55:09.672248+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-03-11T20:53:30.341011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:53:30.341011+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"84'9","reported_seq":172,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.669941+0000","last_change":"2024-03-11T20:54:37.435855+0000","last_active":"2024-03-11T20:55:09.669941+0000","last_peered":"2024-03-11T20:55:09.669941+0000","last_clean":"2024-03-11T20:55:09.669941+0000","last_became_active":"2024-03-11T20:54:37.383262+0000","last_became_peered":"2024-03-11T20:54:37.383262+0000","last_unstale":"2024-03-11T20:55:09.669941+0000","last_undegraded":"2024-03-11T20:55:09.669941+0000","last_fullsized":"2024-03-11T20:55:09.669941+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'9","last_scrub_stamp":"2024-03-11T20:53:34.546587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:53:34.546587+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:12.376461+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":133,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.673271+0000","last_change":"2024-03-11T20:53:26.444831+0000","last_active":"2024-03-11T20:55:09.673271+0000","last_peered":"2024-03-11T20:55:09.673271+0000","last_clean":"2024-03-11T20:55:09.673271+0000","last_became_active":"2024-03-11T20:50:34.321714+0000","last_became_peered":"2024-03-11T20:50:34.321714+0000","last_unstale":"2024-03-11T20:55:09.673271+0000","last_undegraded":"2024-03-11T20:55:09.673271+0000","last_fullsized":"2024-03-11T20:55:09.673271+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:26.444527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:53:26.444527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:12.376624+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":149,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.668250+0000","last_change":"2024-03-11T20:54:37.417550+0000","last_active":"2024-03-11T20:55:09.668250+0000","last_peered":"2024-03-11T20:55:09.668250+0000","last_clean":"2024-03-11T20:55:09.668250+0000","last_became_active":"2024-03-11T20:54:37.417211+0000","last_became_peered":"2024-03-11T20:54:37.417211+0000","last_unstale":"2024-03-11T20:55:09.668250+0000","last_undegraded":"2024-03-11T20:55:09.668250+0000","last_fullsized":"2024-03-11T20:55:09.668250+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:54:35.759035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:54:35.759035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:47.968896+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":144,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.674313+0000","last_change":"2024-03-11T20:53:20.946464+0000","last_active":"2024-03-11T20:55:09.674313+0000","last_peered":"2024-03-11T20:55:09.674313+0000","last_clean":"2024-03-11T20:55:09.674313+0000","last_became_active":"2024-03-11T20:50:28.081538+0000","last_became_peered":"2024-03-11T20:50:28.081538+0000","last_unstale":"2024-03-11T20:55:09.674313+0000","last_undegraded":"2024-03-11T20:55:09.674313+0000","last_fullsized":"2024-03-11T20:55:09.674313+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:20.946227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:53:20.946227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:14.652731+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"84'5","reported_seq":149,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.674793+0000","last_change":"2024-03-11T20:53:31.929747+0000","last_active":"2024-03-11T20:55:09.674793+0000","last_peered":"2024-03-11T20:55:09.674793+0000","last_clean":"2024-03-11T20:55:09.674793+0000","last_became_active":"2024-03-11T20:50:30.099866+0000","last_became_peered":"2024-03-11T20:50:30.099866+0000","last_unstale":"2024-03-11T20:55:09.674793+0000","last_undegraded":"2024-03-11T20:55:09.674793+0000","last_fullsized":"2024-03-11T20:55:09.674793+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'5","last_scrub_stamp":"2024-03-11T20:53:31.929501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:53:31.929501+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:12.947572+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":33,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.674804+0000","last_change":"2024-03-11T20:54:37.413396+0000","last_active":"2024-03-11T20:55:09.674804+0000","last_peered":"2024-03-11T20:55:09.674804+0000","last_clean":"2024-03-11T20:55:09.674804+0000","last_became_active":"2024-03-11T20:54:37.412072+0000","last_became_peered":"2024-03-11T20:54:37.412072+0000","last_unstale":"2024-03-11T20:55:09.674804+0000","last_undegraded":"2024-03-11T20:55:09.674804+0000","last_fullsized":"2024-03-11T20:55:09.674804+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:21.162653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:53:21.162653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:11.510848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"125'13","reported_seq":222,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.675271+0000","last_change":"2024-03-11T20:54:37.438254+0000","last_active":"2024-03-11T20:55:09.675271+0000","last_peered":"2024-03-11T20:55:09.675271+0000","last_clean":"2024-03-11T20:55:09.675271+0000","last_became_active":"2024-03-11T20:54:37.393996+0000","last_became_peered":"2024-03-11T20:54:37.393996+0000","last_unstale":"2024-03-11T20:55:09.675271+0000","last_undegraded":"2024-03-11T20:55:09.675271+0000","last_fullsized":"2024-03-11T20:55:09.675271+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'12","last_scrub_stamp":"2024-03-11T20:53:30.905791+0000","last_deep_scrub":"114'12","last_deep_scrub_stamp":"2024-03-11T20:53:30.905791+0000","last_clean_scrub_stamp":"2024-03-11T20:53:30.905791+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:14.652749+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"77'1","reported_seq":144,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.672134+0000","last_change":"2024-03-11T20:54:00.456606+0000","last_active":"2024-03-11T20:55:09.672134+0000","last_peered":"2024-03-11T20:55:09.672134+0000","last_clean":"2024-03-11T20:55:09.672134+0000","last_became_active":"2024-03-11T20:50:28.068827+0000","last_became_peered":"2024-03-11T20:50:28.068827+0000","last_unstale":"2024-03-11T20:55:09.672134+0000","last_undegraded":"2024-03-11T20:55:09.672134+0000","last_fullsized":"2024-03-11T20:55:09.672134+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-03-11T20:54:00.456354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:54:00.456354+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:15.166863+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"84'9","reported_seq":171,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.671701+0000","last_change":"2024-03-11T20:54:37.444604+0000","last_active":"2024-03-11T20:55:09.671701+0000","last_peered":"2024-03-11T20:55:09.671701+0000","last_clean":"2024-03-11T20:55:09.671701+0000","last_became_active":"2024-03-11T20:54:37.421280+0000","last_became_peered":"2024-03-11T20:54:37.421280+0000","last_unstale":"2024-03-11T20:55:09.671701+0000","last_undegraded":"2024-03-11T20:55:09.671701+0000","last_fullsized":"2024-03-11T20:55:09.671701+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'9","last_scrub_stamp":"2024-03-11T20:53:23.347531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:53:23.347531+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"105'2","reported_seq":135,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.677845+0000","last_change":"2024-03-11T20:53:34.350049+0000","last_active":"2024-03-11T20:55:09.677845+0000","last_peered":"2024-03-11T20:55:09.677845+0000","last_clean":"2024-03-11T20:55:09.677845+0000","last_became_active":"2024-03-11T20:50:34.327356+0000","last_became_peered":"2024-03-11T20:50:34.327356+0000","last_unstale":"2024-03-11T20:55:09.677845+0000","last_undegraded":"2024-03-11T20:55:09.677845+0000","last_fullsized":"2024-03-11T20:55:09.677845+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'2","last_scrub_stamp":"2024-03-11T20:53:34.349639+0000","last_deep_scrub":"105'2","last_deep_scrub_stamp":"2024-03-11T20:52:18.098911+0000","last_clean_scrub_stamp":"2024-03-11T20:53:34.349639+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:12.204798+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":136,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.677005+0000","last_change":"2024-03-11T20:53:10.062579+0000","last_active":"2024-03-11T20:55:09.677005+0000","last_peered":"2024-03-11T20:55:09.677005+0000","last_clean":"2024-03-11T20:55:09.677005+0000","last_became_active":"2024-03-11T20:50:32.648869+0000","last_became_peered":"2024-03-11T20:50:32.648869+0000","last_unstale":"2024-03-11T20:55:09.677005+0000","last_undegraded":"2024-03-11T20:55:09.677005+0000","last_fullsized":"2024-03-11T20:55:09.677005+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:10.062346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:53:10.062346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:12.947636+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":144,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.674967+0000","last_change":"2024-03-11T20:53:31.821391+0000","last_active":"2024-03-11T20:55:09.674967+0000","last_peered":"2024-03-11T20:55:09.674967+0000","last_clean":"2024-03-11T20:55:09.674967+0000","last_became_active":"2024-03-11T20:50:28.066124+0000","last_became_peered":"2024-03-11T20:50:28.066124+0000","last_unstale":"2024-03-11T20:55:09.674967+0000","last_undegraded":"2024-03-11T20:55:09.674967+0000","last_fullsized":"2024-03-11T20:55:09.674967+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:31.821104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:52:01.880052+0000","last_clean_scrub_stamp":"2024-03-11T20:53:31.821104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:11.510944+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"84'15","reported_seq":91,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.675013+0000","last_change":"2024-03-11T20:54:37.442081+0000","last_active":"2024-03-11T20:55:09.675013+0000","last_peered":"2024-03-11T20:55:09.675013+0000","last_clean":"2024-03-11T20:55:09.675013+0000","last_became_active":"2024-03-11T20:54:37.414396+0000","last_became_peered":"2024-03-11T20:54:37.414396+0000","last_unstale":"2024-03-11T20:55:09.675013+0000","last_undegraded":"2024-03-11T20:55:09.675013+0000","last_fullsized":"2024-03-11T20:55:09.675013+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15","last_scrub_stamp":"2024-03-11T20:54:35.884551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:54:35.884551+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:57.346174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":131,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.672803+0000","last_change":"2024-03-11T20:53:28.358235+0000","last_active":"2024-03-11T20:55:09.672803+0000","last_peered":"2024-03-11T20:55:09.672803+0000","last_clean":"2024-03-11T20:55:09.672803+0000","last_became_active":"2024-03-11T20:50:34.337878+0000","last_became_peered":"2024-03-11T20:50:34.337878+0000","last_unstale":"2024-03-11T20:55:09.672803+0000","last_undegraded":"2024-03-11T20:55:09.672803+0000","last_fullsized":"2024-03-11T20:55:09.672803+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:28.358134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:52:11.462480+0000","last_clean_scrub_stamp":"2024-03-11T20:53:28.358134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":143,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.675221+0000","last_change":"2024-03-11T20:54:06.516699+0000","last_active":"2024-03-11T20:55:09.675221+0000","last_peered":"2024-03-11T20:55:09.675221+0000","last_clean":"2024-03-11T20:55:09.675221+0000","last_became_active":"2024-03-11T20:50:32.646323+0000","last_became_peered":"2024-03-11T20:50:32.646323+0000","last_unstale":"2024-03-11T20:55:09.675221+0000","last_undegraded":"2024-03-11T20:55:09.675221+0000","last_fullsized":"2024-03-11T20:55:09.675221+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:54:06.516412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:54:06.516412+0000","last_clean_scrub_stamp":"2024-03-11T20:54:06.516412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:20.180816+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":152,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.675901+0000","last_change":"2024-03-11T20:54:28.584436+0000","last_active":"2024-03-11T20:55:09.675901+0000","last_peered":"2024-03-11T20:55:09.675901+0000","last_clean":"2024-03-11T20:55:09.675901+0000","last_became_active":"2024-03-11T20:50:28.086513+0000","last_became_peered":"2024-03-11T20:50:28.086513+0000","last_unstale":"2024-03-11T20:55:09.675901+0000","last_undegraded":"2024-03-11T20:55:09.675901+0000","last_fullsized":"2024-03-11T20:55:09.675901+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:54:28.584207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:54:28.584207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:32.948712+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"84'9","reported_seq":172,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.675561+0000","last_change":"2024-03-11T20:54:37.433003+0000","last_active":"2024-03-11T20:55:09.675561+0000","last_peered":"2024-03-11T20:55:09.675561+0000","last_clean":"2024-03-11T20:55:09.675561+0000","last_became_active":"2024-03-11T20:54:37.387187+0000","last_became_peered":"2024-03-11T20:54:37.387187+0000","last_unstale":"2024-03-11T20:55:09.675561+0000","last_undegraded":"2024-03-11T20:55:09.675561+0000","last_fullsized":"2024-03-11T20:55:09.675561+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'9","last_scrub_stamp":"2024-03-11T20:53:17.784035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:53:17.784035+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:11.510993+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":140,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.679488+0000","last_change":"2024-03-11T20:54:24.033101+0000","last_active":"2024-03-11T20:55:09.679488+0000","last_peered":"2024-03-11T20:55:09.679488+0000","last_clean":"2024-03-11T20:55:09.679488+0000","last_became_active":"2024-03-11T20:50:34.321443+0000","last_became_peered":"2024-03-11T20:50:34.321443+0000","last_unstale":"2024-03-11T20:55:09.679488+0000","last_undegraded":"2024-03-11T20:55:09.679488+0000","last_fullsized":"2024-03-11T20:55:09.679488+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:54:24.032749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:54:24.032749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:28.673842+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":142,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.671896+0000","last_change":"2024-03-11T20:54:37.415143+0000","last_active":"2024-03-11T20:55:09.671896+0000","last_peered":"2024-03-11T20:55:09.671896+0000","last_clean":"2024-03-11T20:55:09.671896+0000","last_became_active":"2024-03-11T20:54:37.414805+0000","last_became_peered":"2024-03-11T20:54:37.414805+0000","last_unstale":"2024-03-11T20:55:09.671896+0000","last_undegraded":"2024-03-11T20:55:09.671896+0000","last_fullsized":"2024-03-11T20:55:09.671896+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:34.195681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:53:34.195681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":152,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.675550+0000","last_change":"2024-03-11T20:54:20.820822+0000","last_active":"2024-03-11T20:55:09.675550+0000","last_peered":"2024-03-11T20:55:09.675550+0000","last_clean":"2024-03-11T20:55:09.675550+0000","last_became_active":"2024-03-11T20:50:28.075760+0000","last_became_peered":"2024-03-11T20:50:28.075760+0000","last_unstale":"2024-03-11T20:55:09.675550+0000","last_undegraded":"2024-03-11T20:55:09.675550+0000","last_fullsized":"2024-03-11T20:55:09.675550+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:54:20.820602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:54:20.820602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:44.994722+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"84'6","reported_seq":148,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.677669+0000","last_change":"2024-03-11T20:53:18.136127+0000","last_active":"2024-03-11T20:55:09.677669+0000","last_peered":"2024-03-11T20:55:09.677669+0000","last_clean":"2024-03-11T20:55:09.677669+0000","last_became_active":"2024-03-11T20:50:30.092991+0000","last_became_peered":"2024-03-11T20:50:30.092991+0000","last_unstale":"2024-03-11T20:55:09.677669+0000","last_undegraded":"2024-03-11T20:55:09.677669+0000","last_fullsized":"2024-03-11T20:55:09.677669+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6","last_scrub_stamp":"2024-03-11T20:53:18.135889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:53:18.135889+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:12.947697+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.11","version":"84'1","reported_seq":132,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.671661+0000","last_change":"2024-03-11T20:54:09.498197+0000","last_active":"2024-03-11T20:55:09.671661+0000","last_peered":"2024-03-11T20:55:09.671661+0000","last_clean":"2024-03-11T20:55:09.671661+0000","last_became_active":"2024-03-11T20:50:34.339326+0000","last_became_peered":"2024-03-11T20:50:34.339326+0000","last_unstale":"2024-03-11T20:55:09.671661+0000","last_undegraded":"2024-03-11T20:55:09.671661+0000","last_fullsized":"2024-03-11T20:55:09.671661+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'1","last_scrub_stamp":"2024-03-11T20:54:09.497963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:54:09.497963+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:14.349581+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":142,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.672038+0000","last_change":"2024-03-11T20:54:37.390277+0000","last_active":"2024-03-11T20:55:09.672038+0000","last_peered":"2024-03-11T20:55:09.672038+0000","last_clean":"2024-03-11T20:55:09.672038+0000","last_became_active":"2024-03-11T20:54:37.389501+0000","last_became_peered":"2024-03-11T20:54:37.389501+0000","last_unstale":"2024-03-11T20:55:09.672038+0000","last_undegraded":"2024-03-11T20:55:09.672038+0000","last_fullsized":"2024-03-11T20:55:09.672038+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:21.439121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:53:21.439121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:12.376443+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":32,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.675003+0000","last_change":"2024-03-11T20:54:37.392636+0000","last_active":"2024-03-11T20:55:09.675003+0000","last_peered":"2024-03-11T20:55:09.675003+0000","last_clean":"2024-03-11T20:55:09.675003+0000","last_became_active":"2024-03-11T20:54:37.391879+0000","last_became_peered":"2024-03-11T20:54:37.391879+0000","last_unstale":"2024-03-11T20:55:09.675003+0000","last_undegraded":"2024-03-11T20:55:09.675003+0000","last_fullsized":"2024-03-11T20:55:09.675003+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:14.109743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:53:14.109743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:11.510832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"84'10","reported_seq":163,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.672141+0000","last_change":"2024-03-11T20:54:19.565416+0000","last_active":"2024-03-11T20:55:09.672141+0000","last_peered":"2024-03-11T20:55:09.672141+0000","last_clean":"2024-03-11T20:55:09.672141+0000","last_became_active":"2024-03-11T20:50:30.105752+0000","last_became_peered":"2024-03-11T20:50:30.105752+0000","last_unstale":"2024-03-11T20:55:09.672141+0000","last_undegraded":"2024-03-11T20:55:09.672141+0000","last_fullsized":"2024-03-11T20:55:09.672141+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'10","last_scrub_stamp":"2024-03-11T20:54:19.565088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:54:19.565088+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:44.013842+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":137,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.674705+0000","last_change":"2024-03-11T20:54:37.405955+0000","last_active":"2024-03-11T20:55:09.674705+0000","last_peered":"2024-03-11T20:55:09.674705+0000","last_clean":"2024-03-11T20:55:09.674705+0000","last_became_active":"2024-03-11T20:54:37.405532+0000","last_became_peered":"2024-03-11T20:54:37.405532+0000","last_unstale":"2024-03-11T20:55:09.674705+0000","last_undegraded":"2024-03-11T20:55:09.674705+0000","last_fullsized":"2024-03-11T20:55:09.674705+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:22.056809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:53:22.056809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:12.947605+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":32,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.674024+0000","last_change":"2024-03-11T20:54:37.389040+0000","last_active":"2024-03-11T20:55:09.674024+0000","last_peered":"2024-03-11T20:55:09.674024+0000","last_clean":"2024-03-11T20:55:09.674024+0000","last_became_active":"2024-03-11T20:54:37.387438+0000","last_became_peered":"2024-03-11T20:54:37.387438+0000","last_unstale":"2024-03-11T20:55:09.674024+0000","last_undegraded":"2024-03-11T20:55:09.674024+0000","last_fullsized":"2024-03-11T20:55:09.674024+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:24.209748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:53:24.209748+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:14.652651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":153,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.676684+0000","last_change":"2024-03-11T20:54:17.425850+0000","last_active":"2024-03-11T20:55:09.676684+0000","last_peered":"2024-03-11T20:55:09.676684+0000","last_clean":"2024-03-11T20:55:09.676684+0000","last_became_active":"2024-03-11T20:50:28.082737+0000","last_became_peered":"2024-03-11T20:50:28.082737+0000","last_unstale":"2024-03-11T20:55:09.676684+0000","last_undegraded":"2024-03-11T20:55:09.676684+0000","last_fullsized":"2024-03-11T20:55:09.676684+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:54:17.425596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:54:17.425596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:28.371090+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"84'9","reported_seq":164,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.678359+0000","last_change":"2024-03-11T20:54:14.167574+0000","last_active":"2024-03-11T20:55:09.678359+0000","last_peered":"2024-03-11T20:55:09.678359+0000","last_clean":"2024-03-11T20:55:09.678359+0000","last_became_active":"2024-03-11T20:50:30.085418+0000","last_became_peered":"2024-03-11T20:50:30.085418+0000","last_unstale":"2024-03-11T20:55:09.678359+0000","last_undegraded":"2024-03-11T20:55:09.678359+0000","last_fullsized":"2024-03-11T20:55:09.678359+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'9","last_scrub_stamp":"2024-03-11T20:54:14.167307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:54:14.167307+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:16.332610+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":140,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.673237+0000","last_change":"2024-03-11T20:54:23.491662+0000","last_active":"2024-03-11T20:55:09.673237+0000","last_peered":"2024-03-11T20:55:09.673237+0000","last_clean":"2024-03-11T20:55:09.673237+0000","last_became_active":"2024-03-11T20:50:34.328449+0000","last_became_peered":"2024-03-11T20:50:34.328449+0000","last_unstale":"2024-03-11T20:55:09.673237+0000","last_undegraded":"2024-03-11T20:55:09.673237+0000","last_fullsized":"2024-03-11T20:55:09.673237+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:54:23.491545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:54:23.491545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:52.519403+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.15","version":"112'12","reported_seq":204,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.675243+0000","last_change":"2024-03-11T20:53:11.844176+0000","last_active":"2024-03-11T20:55:09.675243+0000","last_peered":"2024-03-11T20:55:09.675243+0000","last_clean":"2024-03-11T20:55:09.675243+0000","last_became_active":"2024-03-11T20:50:32.650064+0000","last_became_peered":"2024-03-11T20:50:32.650064+0000","last_unstale":"2024-03-11T20:55:09.675243+0000","last_undegraded":"2024-03-11T20:55:09.675243+0000","last_fullsized":"2024-03-11T20:55:09.675243+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'12","last_scrub_stamp":"2024-03-11T20:53:11.843951+0000","last_deep_scrub":"112'12","last_deep_scrub_stamp":"2024-03-11T20:53:11.843951+0000","last_clean_scrub_stamp":"2024-03-11T20:53:11.843951+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:11.510977+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":152,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.675152+0000","last_change":"2024-03-11T20:54:01.692416+0000","last_active":"2024-03-11T20:55:09.675152+0000","last_peered":"2024-03-11T20:55:09.675152+0000","last_clean":"2024-03-11T20:55:09.675152+0000","last_became_active":"2024-03-11T20:50:28.044321+0000","last_became_peered":"2024-03-11T20:50:28.044321+0000","last_unstale":"2024-03-11T20:55:09.675152+0000","last_undegraded":"2024-03-11T20:55:09.675152+0000","last_fullsized":"2024-03-11T20:55:09.675152+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:54:01.692199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:54:01.692199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:25.828989+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"119'13","reported_seq":168,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.676425+0000","last_change":"2024-03-11T20:54:15.443720+0000","last_active":"2024-03-11T20:55:09.676425+0000","last_peered":"2024-03-11T20:55:09.676425+0000","last_clean":"2024-03-11T20:55:09.676425+0000","last_became_active":"2024-03-11T20:50:30.099856+0000","last_became_peered":"2024-03-11T20:50:30.099856+0000","last_unstale":"2024-03-11T20:55:09.676425+0000","last_undegraded":"2024-03-11T20:55:09.676425+0000","last_fullsized":"2024-03-11T20:55:09.676425+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'13","last_scrub_stamp":"2024-03-11T20:54:15.443472+0000","last_deep_scrub":"119'13","last_deep_scrub_stamp":"2024-03-11T20:54:15.443472+0000","last_clean_scrub_stamp":"2024-03-11T20:54:15.443472+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:23.025981+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":131,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.677260+0000","last_change":"2024-03-11T20:53:40.043289+0000","last_active":"2024-03-11T20:55:09.677260+0000","last_peered":"2024-03-11T20:55:09.677260+0000","last_clean":"2024-03-11T20:55:09.677260+0000","last_became_active":"2024-03-11T20:50:34.330351+0000","last_became_peered":"2024-03-11T20:50:34.330351+0000","last_unstale":"2024-03-11T20:55:09.677260+0000","last_undegraded":"2024-03-11T20:55:09.677260+0000","last_fullsized":"2024-03-11T20:55:09.677260+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:40.043054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:53:40.043054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:10.910469+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"125'14","reported_seq":228,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.675669+0000","last_change":"2024-03-11T20:54:37.420735+0000","last_active":"2024-03-11T20:55:09.675669+0000","last_peered":"2024-03-11T20:55:09.675669+0000","last_clean":"2024-03-11T20:55:09.675669+0000","last_became_active":"2024-03-11T20:54:37.420346+0000","last_became_peered":"2024-03-11T20:54:37.420346+0000","last_unstale":"2024-03-11T20:55:09.675669+0000","last_undegraded":"2024-03-11T20:55:09.675669+0000","last_fullsized":"2024-03-11T20:55:09.675669+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'13","last_scrub_stamp":"2024-03-11T20:54:30.577957+0000","last_deep_scrub":"110'13","last_deep_scrub_stamp":"2024-03-11T20:53:00.434532+0000","last_clean_scrub_stamp":"2024-03-11T20:54:30.577957+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:46.787183+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":145,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.675917+0000","last_change":"2024-03-11T20:52:59.801478+0000","last_active":"2024-03-11T20:55:09.675917+0000","last_peered":"2024-03-11T20:55:09.675917+0000","last_clean":"2024-03-11T20:55:09.675917+0000","last_became_active":"2024-03-11T20:50:28.070325+0000","last_became_peered":"2024-03-11T20:50:28.070325+0000","last_unstale":"2024-03-11T20:55:09.675917+0000","last_undegraded":"2024-03-11T20:55:09.675917+0000","last_fullsized":"2024-03-11T20:55:09.675917+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:52:59.801249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:52:59.801249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:11.510898+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"116'16","reported_seq":165,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.677426+0000","last_change":"2024-03-11T20:53:39.292223+0000","last_active":"2024-03-11T20:55:09.677426+0000","last_peered":"2024-03-11T20:55:09.677426+0000","last_clean":"2024-03-11T20:55:09.677426+0000","last_became_active":"2024-03-11T20:50:30.087154+0000","last_became_peered":"2024-03-11T20:50:30.087154+0000","last_unstale":"2024-03-11T20:55:09.677426+0000","last_undegraded":"2024-03-11T20:55:09.677426+0000","last_fullsized":"2024-03-11T20:55:09.677426+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'16","last_scrub_stamp":"2024-03-11T20:53:39.292118+0000","last_deep_scrub":"116'16","last_deep_scrub_stamp":"2024-03-11T20:53:39.292118+0000","last_clean_scrub_stamp":"2024-03-11T20:53:39.292118+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:12.204863+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":132,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.677814+0000","last_change":"2024-03-11T20:53:47.223659+0000","last_active":"2024-03-11T20:55:09.677814+0000","last_peered":"2024-03-11T20:55:09.677814+0000","last_clean":"2024-03-11T20:55:09.677814+0000","last_became_active":"2024-03-11T20:50:34.318231+0000","last_became_peered":"2024-03-11T20:50:34.318231+0000","last_unstale":"2024-03-11T20:55:09.677814+0000","last_undegraded":"2024-03-11T20:55:09.677814+0000","last_fullsized":"2024-03-11T20:55:09.677814+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:47.223425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:53:47.223425+0000","last_clean_scrub_stamp":"2024-03-11T20:53:47.223425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:11.267529+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":149,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.674792+0000","last_change":"2024-03-11T20:54:37.420974+0000","last_active":"2024-03-11T20:55:09.674792+0000","last_peered":"2024-03-11T20:55:09.674792+0000","last_clean":"2024-03-11T20:55:09.674792+0000","last_became_active":"2024-03-11T20:54:37.420546+0000","last_became_peered":"2024-03-11T20:54:37.420546+0000","last_unstale":"2024-03-11T20:55:09.674792+0000","last_undegraded":"2024-03-11T20:55:09.674792+0000","last_fullsized":"2024-03-11T20:55:09.674792+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:54:19.493484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:54:19.493484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:23.105602+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"77'1","reported_seq":163,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.674905+0000","last_change":"2024-03-11T20:54:37.403243+0000","last_active":"2024-03-11T20:55:09.674905+0000","last_peered":"2024-03-11T20:55:09.674905+0000","last_clean":"2024-03-11T20:55:09.674905+0000","last_became_active":"2024-03-11T20:54:37.398457+0000","last_became_peered":"2024-03-11T20:54:37.398457+0000","last_unstale":"2024-03-11T20:55:09.674905+0000","last_undegraded":"2024-03-11T20:55:09.674905+0000","last_fullsized":"2024-03-11T20:55:09.674905+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-03-11T20:53:24.900978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:53:24.900978+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:14.652597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":911,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"84'4","reported_seq":147,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.672870+0000","last_change":"2024-03-11T20:53:42.788395+0000","last_active":"2024-03-11T20:55:09.672870+0000","last_peered":"2024-03-11T20:55:09.672870+0000","last_clean":"2024-03-11T20:55:09.672870+0000","last_became_active":"2024-03-11T20:50:30.101572+0000","last_became_peered":"2024-03-11T20:50:30.101572+0000","last_unstale":"2024-03-11T20:55:09.672870+0000","last_undegraded":"2024-03-11T20:55:09.672870+0000","last_fullsized":"2024-03-11T20:55:09.672870+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'4","last_scrub_stamp":"2024-03-11T20:53:42.788154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:53:42.788154+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:11.351245+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":139,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.668339+0000","last_change":"2024-03-11T20:54:06.809952+0000","last_active":"2024-03-11T20:55:09.668339+0000","last_peered":"2024-03-11T20:55:09.668339+0000","last_clean":"2024-03-11T20:55:09.668339+0000","last_became_active":"2024-03-11T20:50:34.320001+0000","last_became_peered":"2024-03-11T20:50:34.320001+0000","last_unstale":"2024-03-11T20:55:09.668339+0000","last_undegraded":"2024-03-11T20:55:09.668339+0000","last_fullsized":"2024-03-11T20:55:09.668339+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:54:06.809735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:54:06.809735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:23.643000+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":143,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.678734+0000","last_change":"2024-03-11T20:54:09.538185+0000","last_active":"2024-03-11T20:55:09.678734+0000","last_peered":"2024-03-11T20:55:09.678734+0000","last_clean":"2024-03-11T20:55:09.678734+0000","last_became_active":"2024-03-11T20:50:32.646569+0000","last_became_peered":"2024-03-11T20:50:32.646569+0000","last_unstale":"2024-03-11T20:55:09.678734+0000","last_undegraded":"2024-03-11T20:55:09.678734+0000","last_fullsized":"2024-03-11T20:55:09.678734+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:54:09.537972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:54:09.537972+0000","last_clean_scrub_stamp":"2024-03-11T20:54:09.537972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:23.286674+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":158,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.669071+0000","last_change":"2024-03-11T20:54:37.419660+0000","last_active":"2024-03-11T20:55:09.669071+0000","last_peered":"2024-03-11T20:55:09.669071+0000","last_clean":"2024-03-11T20:55:09.669071+0000","last_became_active":"2024-03-11T20:54:37.419155+0000","last_became_peered":"2024-03-11T20:54:37.419155+0000","last_unstale":"2024-03-11T20:55:09.669071+0000","last_undegraded":"2024-03-11T20:55:09.669071+0000","last_fullsized":"2024-03-11T20:55:09.669071+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:54:33.812279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:54:33.812279+0000","last_clean_scrub_stamp":"2024-03-11T20:54:33.812279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:49.852237+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"84'11","reported_seq":160,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.677126+0000","last_change":"2024-03-11T20:53:42.298549+0000","last_active":"2024-03-11T20:55:09.677126+0000","last_peered":"2024-03-11T20:55:09.677126+0000","last_clean":"2024-03-11T20:55:09.677126+0000","last_became_active":"2024-03-11T20:50:30.103212+0000","last_became_peered":"2024-03-11T20:50:30.103212+0000","last_unstale":"2024-03-11T20:55:09.677126+0000","last_undegraded":"2024-03-11T20:55:09.677126+0000","last_fullsized":"2024-03-11T20:55:09.677126+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T20:53:42.298307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:53:42.298307+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:10.739880+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"135'2418","reported_seq":2455,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.842300+0000","last_change":"2024-03-11T20:54:34.364188+0000","last_active":"2024-03-11T20:55:09.842300+0000","last_peered":"2024-03-11T20:55:09.842300+0000","last_clean":"2024-03-11T20:55:09.842300+0000","last_became_active":"2024-03-11T20:54:34.363252+0000","last_became_peered":"2024-03-11T20:54:34.363252+0000","last_unstale":"2024-03-11T20:55:09.842300+0000","last_undegraded":"2024-03-11T20:55:09.842300+0000","last_fullsized":"2024-03-11T20:55:09.842300+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:54:33.308151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:54:33.308151+0000","last_clean_scrub_stamp":"2024-03-11T20:54:33.308151+0000","objects_scrubbed":0,"log_size":2418,"log_dups_size":0,"ondisk_log_size":2418,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:43.493121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":158466048,"num_objects":2418,"num_object_clones":0,"num_object_copies":7254,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2418,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2418,"num_write_kb":154752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":146,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.672615+0000","last_change":"2024-03-11T20:54:37.416558+0000","last_active":"2024-03-11T20:55:09.672615+0000","last_peered":"2024-03-11T20:55:09.672615+0000","last_clean":"2024-03-11T20:55:09.672615+0000","last_became_active":"2024-03-11T20:54:37.416205+0000","last_became_peered":"2024-03-11T20:54:37.416205+0000","last_unstale":"2024-03-11T20:55:09.672615+0000","last_undegraded":"2024-03-11T20:55:09.672615+0000","last_fullsized":"2024-03-11T20:55:09.672615+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:54:12.449779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:52:53.450885+0000","last_clean_scrub_stamp":"2024-03-11T20:54:12.449779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:17.158802+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"77'2","reported_seq":161,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.673071+0000","last_change":"2024-03-11T20:53:28.411797+0000","last_active":"2024-03-11T20:55:09.673071+0000","last_peered":"2024-03-11T20:55:09.673071+0000","last_clean":"2024-03-11T20:55:09.673071+0000","last_became_active":"2024-03-11T20:50:28.059564+0000","last_became_peered":"2024-03-11T20:50:28.059564+0000","last_unstale":"2024-03-11T20:55:09.673071+0000","last_undegraded":"2024-03-11T20:55:09.673071+0000","last_fullsized":"2024-03-11T20:55:09.673071+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'2","last_scrub_stamp":"2024-03-11T20:53:28.411558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:53:28.411558+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:12.376555+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"100'16","reported_seq":191,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.678126+0000","last_change":"2024-03-11T20:54:37.435645+0000","last_active":"2024-03-11T20:55:09.678126+0000","last_peered":"2024-03-11T20:55:09.678126+0000","last_clean":"2024-03-11T20:55:09.678126+0000","last_became_active":"2024-03-11T20:54:37.384802+0000","last_became_peered":"2024-03-11T20:54:37.384802+0000","last_unstale":"2024-03-11T20:55:09.678126+0000","last_undegraded":"2024-03-11T20:55:09.678126+0000","last_fullsized":"2024-03-11T20:55:09.678126+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'16","last_scrub_stamp":"2024-03-11T20:54:04.153293+0000","last_deep_scrub":"100'16","last_deep_scrub_stamp":"2024-03-11T20:51:31.194005+0000","last_clean_scrub_stamp":"2024-03-11T20:54:04.153293+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:22.661284+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"125'13","reported_seq":216,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.672016+0000","last_change":"2024-03-11T20:54:37.420330+0000","last_active":"2024-03-11T20:55:09.672016+0000","last_peered":"2024-03-11T20:55:09.672016+0000","last_clean":"2024-03-11T20:55:09.672016+0000","last_became_active":"2024-03-11T20:54:37.393208+0000","last_became_peered":"2024-03-11T20:54:37.393208+0000","last_unstale":"2024-03-11T20:55:09.672016+0000","last_undegraded":"2024-03-11T20:55:09.672016+0000","last_fullsized":"2024-03-11T20:55:09.672016+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'12","last_scrub_stamp":"2024-03-11T20:54:02.639006+0000","last_deep_scrub":"106'12","last_deep_scrub_stamp":"2024-03-11T20:52:21.538885+0000","last_clean_scrub_stamp":"2024-03-11T20:54:02.639006+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:22.332733+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1","version":"135'2312","reported_seq":2350,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.847730+0000","last_change":"2024-03-11T20:54:34.345029+0000","last_active":"2024-03-11T20:55:09.847730+0000","last_peered":"2024-03-11T20:55:09.847730+0000","last_clean":"2024-03-11T20:55:09.847730+0000","last_became_active":"2024-03-11T20:54:34.344527+0000","last_became_peered":"2024-03-11T20:54:34.344527+0000","last_unstale":"2024-03-11T20:55:09.847730+0000","last_undegraded":"2024-03-11T20:55:09.847730+0000","last_fullsized":"2024-03-11T20:55:09.847730+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:54:33.308151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:54:33.308151+0000","last_clean_scrub_stamp":"2024-03-11T20:54:33.308151+0000","objects_scrubbed":0,"log_size":2312,"log_dups_size":0,"ondisk_log_size":2312,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:41.468033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":151519232,"num_objects":2312,"num_object_clones":0,"num_object_copies":6936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2312,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2312,"num_write_kb":147968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":140,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.674812+0000","last_change":"2024-03-11T20:54:32.671952+0000","last_active":"2024-03-11T20:55:09.674812+0000","last_peered":"2024-03-11T20:55:09.674812+0000","last_clean":"2024-03-11T20:55:09.674812+0000","last_became_active":"2024-03-11T20:50:34.328932+0000","last_became_peered":"2024-03-11T20:50:34.328932+0000","last_unstale":"2024-03-11T20:55:09.674812+0000","last_undegraded":"2024-03-11T20:55:09.674812+0000","last_fullsized":"2024-03-11T20:55:09.674812+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:54:32.671859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:53:27.053208+0000","last_clean_scrub_stamp":"2024-03-11T20:54:32.671859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:37.413110+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":32,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.676363+0000","last_change":"2024-03-11T20:54:37.423075+0000","last_active":"2024-03-11T20:55:09.676363+0000","last_peered":"2024-03-11T20:55:09.676363+0000","last_clean":"2024-03-11T20:55:09.676363+0000","last_became_active":"2024-03-11T20:54:37.422724+0000","last_became_peered":"2024-03-11T20:54:37.422724+0000","last_unstale":"2024-03-11T20:55:09.676363+0000","last_undegraded":"2024-03-11T20:55:09.676363+0000","last_fullsized":"2024-03-11T20:55:09.676363+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:38.117113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:53:38.117113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:12.947666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"84'16","reported_seq":220,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.674900+0000","last_change":"2024-03-11T20:53:45.846335+0000","last_active":"2024-03-11T20:55:09.674900+0000","last_peered":"2024-03-11T20:55:09.674900+0000","last_clean":"2024-03-11T20:55:09.674900+0000","last_became_active":"2024-03-11T20:50:30.094977+0000","last_became_peered":"2024-03-11T20:50:30.094977+0000","last_unstale":"2024-03-11T20:55:09.674900+0000","last_undegraded":"2024-03-11T20:55:09.674900+0000","last_fullsized":"2024-03-11T20:55:09.674900+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'16","last_scrub_stamp":"2024-03-11T20:53:45.846082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:53:45.846082+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:11.511025+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":32,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.676164+0000","last_change":"2024-03-11T20:54:37.380408+0000","last_active":"2024-03-11T20:55:09.676164+0000","last_peered":"2024-03-11T20:55:09.676164+0000","last_clean":"2024-03-11T20:55:09.676164+0000","last_became_active":"2024-03-11T20:54:37.378518+0000","last_became_peered":"2024-03-11T20:54:37.378518+0000","last_unstale":"2024-03-11T20:55:09.676164+0000","last_undegraded":"2024-03-11T20:55:09.676164+0000","last_fullsized":"2024-03-11T20:55:09.676164+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:35.139348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:52:10.160202+0000","last_clean_scrub_stamp":"2024-03-11T20:53:35.139348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:11.511055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"135'2404","reported_seq":2441,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.836544+0000","last_change":"2024-03-11T20:54:34.352203+0000","last_active":"2024-03-11T20:55:09.836544+0000","last_peered":"2024-03-11T20:55:09.836544+0000","last_clean":"2024-03-11T20:55:09.836544+0000","last_became_active":"2024-03-11T20:54:34.351729+0000","last_became_peered":"2024-03-11T20:54:34.351729+0000","last_unstale":"2024-03-11T20:55:09.836544+0000","last_undegraded":"2024-03-11T20:55:09.836544+0000","last_fullsized":"2024-03-11T20:55:09.836544+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:54:33.308151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:54:33.308151+0000","last_clean_scrub_stamp":"2024-03-11T20:54:33.308151+0000","objects_scrubbed":0,"log_size":2404,"log_dups_size":0,"ondisk_log_size":2404,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:59.322040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":157548544,"num_objects":2404,"num_object_clones":0,"num_object_copies":7212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2404,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2404,"num_write_kb":153856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":132,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.672804+0000","last_change":"2024-03-11T20:53:33.225496+0000","last_active":"2024-03-11T20:55:09.672804+0000","last_peered":"2024-03-11T20:55:09.672804+0000","last_clean":"2024-03-11T20:55:09.672804+0000","last_became_active":"2024-03-11T20:50:34.329253+0000","last_became_peered":"2024-03-11T20:50:34.329253+0000","last_unstale":"2024-03-11T20:55:09.672804+0000","last_undegraded":"2024-03-11T20:55:09.672804+0000","last_fullsized":"2024-03-11T20:55:09.672804+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:33.225296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:53:33.225296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":159,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.675965+0000","last_change":"2024-03-11T20:54:37.393367+0000","last_active":"2024-03-11T20:55:09.675965+0000","last_peered":"2024-03-11T20:55:09.675965+0000","last_clean":"2024-03-11T20:55:09.675965+0000","last_became_active":"2024-03-11T20:54:37.392376+0000","last_became_peered":"2024-03-11T20:54:37.392376+0000","last_unstale":"2024-03-11T20:55:09.675965+0000","last_undegraded":"2024-03-11T20:55:09.675965+0000","last_fullsized":"2024-03-11T20:55:09.675965+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:54:25.840284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:33.823704+0000","last_clean_scrub_stamp":"2024-03-11T20:54:25.840284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:55.028355+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"84'19","reported_seq":185,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.672274+0000","last_change":"2024-03-11T20:54:22.531772+0000","last_active":"2024-03-11T20:55:09.672274+0000","last_peered":"2024-03-11T20:55:09.672274+0000","last_clean":"2024-03-11T20:55:09.672274+0000","last_became_active":"2024-03-11T20:50:30.097543+0000","last_became_peered":"2024-03-11T20:50:30.097543+0000","last_unstale":"2024-03-11T20:55:09.672274+0000","last_undegraded":"2024-03-11T20:55:09.672274+0000","last_fullsized":"2024-03-11T20:55:09.672274+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'19","last_scrub_stamp":"2024-03-11T20:54:22.531527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:54:22.531527+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:30.339156+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":31,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.668048+0000","last_change":"2024-03-11T20:54:37.421201+0000","last_active":"2024-03-11T20:55:09.668048+0000","last_peered":"2024-03-11T20:55:09.668048+0000","last_clean":"2024-03-11T20:55:09.668048+0000","last_became_active":"2024-03-11T20:54:37.420844+0000","last_became_peered":"2024-03-11T20:54:37.420844+0000","last_unstale":"2024-03-11T20:55:09.668048+0000","last_undegraded":"2024-03-11T20:55:09.668048+0000","last_fullsized":"2024-03-11T20:55:09.668048+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:28.142465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:53:28.142465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.4","version":"135'2376","reported_seq":2420,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.837365+0000","last_change":"2024-03-11T20:54:37.407887+0000","last_active":"2024-03-11T20:55:09.837365+0000","last_peered":"2024-03-11T20:55:09.837365+0000","last_clean":"2024-03-11T20:55:09.837365+0000","last_became_active":"2024-03-11T20:54:37.407526+0000","last_became_peered":"2024-03-11T20:54:37.407526+0000","last_unstale":"2024-03-11T20:55:09.837365+0000","last_undegraded":"2024-03-11T20:55:09.837365+0000","last_fullsized":"2024-03-11T20:55:09.837365+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:54:33.308151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:54:33.308151+0000","last_clean_scrub_stamp":"2024-03-11T20:54:33.308151+0000","objects_scrubbed":0,"log_size":2376,"log_dups_size":0,"ondisk_log_size":2376,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:44.234801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":155713536,"num_objects":2376,"num_object_clones":0,"num_object_copies":7128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2376,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2376,"num_write_kb":152064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":133,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.677913+0000","last_change":"2024-03-11T20:53:43.272285+0000","last_active":"2024-03-11T20:55:09.677913+0000","last_peered":"2024-03-11T20:55:09.677913+0000","last_clean":"2024-03-11T20:55:09.677913+0000","last_became_active":"2024-03-11T20:50:34.321038+0000","last_became_peered":"2024-03-11T20:50:34.321038+0000","last_unstale":"2024-03-11T20:55:09.677913+0000","last_undegraded":"2024-03-11T20:55:09.677913+0000","last_fullsized":"2024-03-11T20:55:09.677913+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:43.272073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:53:43.272073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:12.204828+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"109'26","reported_seq":164,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.668971+0000","last_change":"2024-03-11T20:54:37.449285+0000","last_active":"2024-03-11T20:55:09.668971+0000","last_peered":"2024-03-11T20:55:09.668971+0000","last_clean":"2024-03-11T20:55:09.668971+0000","last_became_active":"2024-03-11T20:54:37.422161+0000","last_became_peered":"2024-03-11T20:54:37.422161+0000","last_unstale":"2024-03-11T20:55:09.668971+0000","last_undegraded":"2024-03-11T20:55:09.668971+0000","last_fullsized":"2024-03-11T20:55:09.668971+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'26","last_scrub_stamp":"2024-03-11T20:53:50.029183+0000","last_deep_scrub":"109'26","last_deep_scrub_stamp":"2024-03-11T20:52:49.216378+0000","last_clean_scrub_stamp":"2024-03-11T20:53:50.029183+0000","objects_scrubbed":8,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:12.506290+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":152,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.667482+0000","last_change":"2024-03-11T20:54:34.798444+0000","last_active":"2024-03-11T20:55:09.667482+0000","last_peered":"2024-03-11T20:55:09.667482+0000","last_clean":"2024-03-11T20:55:09.667482+0000","last_became_active":"2024-03-11T20:50:28.045625+0000","last_became_peered":"2024-03-11T20:50:28.045625+0000","last_unstale":"2024-03-11T20:55:09.667482+0000","last_undegraded":"2024-03-11T20:55:09.667482+0000","last_fullsized":"2024-03-11T20:55:09.667482+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:54:34.798334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:54:34.798334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:39.257344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":32,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.675980+0000","last_change":"2024-03-11T20:54:37.370649+0000","last_active":"2024-03-11T20:55:09.675980+0000","last_peered":"2024-03-11T20:55:09.675980+0000","last_clean":"2024-03-11T20:55:09.675980+0000","last_became_active":"2024-03-11T20:54:37.369102+0000","last_became_peered":"2024-03-11T20:54:37.369102+0000","last_unstale":"2024-03-11T20:55:09.675980+0000","last_undegraded":"2024-03-11T20:55:09.675980+0000","last_fullsized":"2024-03-11T20:55:09.675980+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:56.025185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:53:56.025185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:19.851235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"135'2363","reported_seq":2401,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.821007+0000","last_change":"2024-03-11T20:54:34.353830+0000","last_active":"2024-03-11T20:55:09.821007+0000","last_peered":"2024-03-11T20:55:09.821007+0000","last_clean":"2024-03-11T20:55:09.821007+0000","last_became_active":"2024-03-11T20:54:34.353310+0000","last_became_peered":"2024-03-11T20:54:34.353310+0000","last_unstale":"2024-03-11T20:55:09.821007+0000","last_undegraded":"2024-03-11T20:55:09.821007+0000","last_fullsized":"2024-03-11T20:55:09.821007+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:54:33.308151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:54:33.308151+0000","last_clean_scrub_stamp":"2024-03-11T20:54:33.308151+0000","objects_scrubbed":0,"log_size":2363,"log_dups_size":0,"ondisk_log_size":2363,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:56:03.259832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154861568,"num_objects":2363,"num_object_clones":0,"num_object_copies":7089,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2363,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2363,"num_write_kb":151232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":133,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.676066+0000","last_change":"2024-03-11T20:53:40.853529+0000","last_active":"2024-03-11T20:55:09.676066+0000","last_peered":"2024-03-11T20:55:09.676066+0000","last_clean":"2024-03-11T20:55:09.676066+0000","last_became_active":"2024-03-11T20:50:34.336066+0000","last_became_peered":"2024-03-11T20:50:34.336066+0000","last_unstale":"2024-03-11T20:55:09.676066+0000","last_undegraded":"2024-03-11T20:55:09.676066+0000","last_fullsized":"2024-03-11T20:55:09.676066+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:40.853290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:53:40.853290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"84'10","reported_seq":155,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.695545+0000","last_change":"2024-03-11T20:54:05.498968+0000","last_active":"2024-03-11T20:55:09.695545+0000","last_peered":"2024-03-11T20:55:09.695545+0000","last_clean":"2024-03-11T20:55:09.695545+0000","last_became_active":"2024-03-11T20:50:30.101747+0000","last_became_peered":"2024-03-11T20:50:30.101747+0000","last_unstale":"2024-03-11T20:55:09.695545+0000","last_undegraded":"2024-03-11T20:55:09.695545+0000","last_fullsized":"2024-03-11T20:55:09.695545+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'10","last_scrub_stamp":"2024-03-11T20:54:05.498640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:54:05.498640+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:10.902134+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":153,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.674588+0000","last_change":"2024-03-11T20:53:49.837840+0000","last_active":"2024-03-11T20:55:09.674588+0000","last_peered":"2024-03-11T20:55:09.674588+0000","last_clean":"2024-03-11T20:55:09.674588+0000","last_became_active":"2024-03-11T20:50:28.085503+0000","last_became_peered":"2024-03-11T20:50:28.085503+0000","last_unstale":"2024-03-11T20:55:09.674588+0000","last_undegraded":"2024-03-11T20:55:09.674588+0000","last_fullsized":"2024-03-11T20:55:09.674588+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:49.837622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:53:49.837622+0000","last_clean_scrub_stamp":"2024-03-11T20:53:49.837622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:11.511041+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"102'128","reported_seq":410,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.676957+0000","last_change":"2024-03-11T20:53:55.239105+0000","last_active":"2024-03-11T20:55:09.676957+0000","last_peered":"2024-03-11T20:55:09.676957+0000","last_clean":"2024-03-11T20:55:09.676957+0000","last_became_active":"2024-03-11T20:49:52.584543+0000","last_became_peered":"2024-03-11T20:49:52.584543+0000","last_unstale":"2024-03-11T20:55:09.676957+0000","last_undegraded":"2024-03-11T20:55:09.676957+0000","last_fullsized":"2024-03-11T20:55:09.676957+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"102'128","last_scrub_stamp":"2024-03-11T20:53:55.238857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:45:32.381751+0000","last_clean_scrub_stamp":"2024-03-11T20:53:55.238857+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:13.380918+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":145,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.675896+0000","last_change":"2024-03-11T20:54:22.449157+0000","last_active":"2024-03-11T20:55:09.675896+0000","last_peered":"2024-03-11T20:55:09.675896+0000","last_clean":"2024-03-11T20:55:09.675896+0000","last_became_active":"2024-03-11T20:50:32.652128+0000","last_became_peered":"2024-03-11T20:50:32.652128+0000","last_unstale":"2024-03-11T20:55:09.675896+0000","last_undegraded":"2024-03-11T20:55:09.675896+0000","last_fullsized":"2024-03-11T20:55:09.675896+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:54:22.448812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:54:22.448812+0000","last_clean_scrub_stamp":"2024-03-11T20:54:22.448812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:40.804398+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.5","version":"135'2430","reported_seq":2468,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.836377+0000","last_change":"2024-03-11T20:54:34.350772+0000","last_active":"2024-03-11T20:55:09.836377+0000","last_peered":"2024-03-11T20:55:09.836377+0000","last_clean":"2024-03-11T20:55:09.836377+0000","last_became_active":"2024-03-11T20:54:34.350458+0000","last_became_peered":"2024-03-11T20:54:34.350458+0000","last_unstale":"2024-03-11T20:55:09.836377+0000","last_undegraded":"2024-03-11T20:55:09.836377+0000","last_fullsized":"2024-03-11T20:55:09.836377+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:54:33.308151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:54:33.308151+0000","last_clean_scrub_stamp":"2024-03-11T20:54:33.308151+0000","objects_scrubbed":0,"log_size":2430,"log_dups_size":0,"ondisk_log_size":2430,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:57.800564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159252480,"num_objects":2430,"num_object_clones":0,"num_object_copies":7290,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2430,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2430,"num_write_kb":155520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":33,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.674554+0000","last_change":"2024-03-11T20:54:37.416297+0000","last_active":"2024-03-11T20:55:09.674554+0000","last_peered":"2024-03-11T20:55:09.674554+0000","last_clean":"2024-03-11T20:55:09.674554+0000","last_became_active":"2024-03-11T20:54:37.415773+0000","last_became_peered":"2024-03-11T20:54:37.415773+0000","last_unstale":"2024-03-11T20:55:09.674554+0000","last_undegraded":"2024-03-11T20:55:09.674554+0000","last_fullsized":"2024-03-11T20:55:09.674554+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:23.179506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:53:23.179506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:11.511008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"100'21","reported_seq":176,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.677035+0000","last_change":"2024-03-11T20:54:30.679263+0000","last_active":"2024-03-11T20:55:09.677035+0000","last_peered":"2024-03-11T20:55:09.677035+0000","last_clean":"2024-03-11T20:55:09.677035+0000","last_became_active":"2024-03-11T20:50:30.106110+0000","last_became_peered":"2024-03-11T20:50:30.106110+0000","last_unstale":"2024-03-11T20:55:09.677035+0000","last_undegraded":"2024-03-11T20:55:09.677035+0000","last_fullsized":"2024-03-11T20:55:09.677035+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'21","last_scrub_stamp":"2024-03-11T20:54:30.679114+0000","last_deep_scrub":"100'21","last_deep_scrub_stamp":"2024-03-11T20:51:31.835302+0000","last_clean_scrub_stamp":"2024-03-11T20:54:30.679114+0000","objects_scrubbed":7,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:35.171691+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":115,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.669248+0000","last_change":"2024-03-11T20:54:37.423175+0000","last_active":"2024-03-11T20:55:09.669248+0000","last_peered":"2024-03-11T20:55:09.669248+0000","last_clean":"2024-03-11T20:55:09.669248+0000","last_became_active":"2024-03-11T20:54:37.422381+0000","last_became_peered":"2024-03-11T20:54:37.422381+0000","last_unstale":"2024-03-11T20:55:09.669248+0000","last_undegraded":"2024-03-11T20:55:09.669248+0000","last_fullsized":"2024-03-11T20:55:09.669248+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:57.731108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:53:57.731108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":150,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.677167+0000","last_change":"2024-03-11T20:54:37.421145+0000","last_active":"2024-03-11T20:55:09.677167+0000","last_peered":"2024-03-11T20:55:09.677167+0000","last_clean":"2024-03-11T20:55:09.677167+0000","last_became_active":"2024-03-11T20:54:37.420780+0000","last_became_peered":"2024-03-11T20:54:37.420780+0000","last_unstale":"2024-03-11T20:55:09.677167+0000","last_undegraded":"2024-03-11T20:55:09.677167+0000","last_fullsized":"2024-03-11T20:55:09.677167+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:54:34.673183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:54:34.673183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:43.042259+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.0","version":"135'2249","reported_seq":2293,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.818770+0000","last_change":"2024-03-11T20:54:37.414515+0000","last_active":"2024-03-11T20:55:09.818770+0000","last_peered":"2024-03-11T20:55:09.818770+0000","last_clean":"2024-03-11T20:55:09.818770+0000","last_became_active":"2024-03-11T20:54:37.413777+0000","last_became_peered":"2024-03-11T20:54:37.413777+0000","last_unstale":"2024-03-11T20:55:09.818770+0000","last_undegraded":"2024-03-11T20:55:09.818770+0000","last_fullsized":"2024-03-11T20:55:09.818770+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:54:33.308151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:54:33.308151+0000","last_clean_scrub_stamp":"2024-03-11T20:54:33.308151+0000","objects_scrubbed":0,"log_size":2249,"log_dups_size":0,"ondisk_log_size":2249,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:46.291105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":147390464,"num_objects":2249,"num_object_clones":0,"num_object_copies":6747,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2249,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2249,"num_write_kb":143936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":33,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.675106+0000","last_change":"2024-03-11T20:54:37.430179+0000","last_active":"2024-03-11T20:55:09.675106+0000","last_peered":"2024-03-11T20:55:09.675106+0000","last_clean":"2024-03-11T20:55:09.675106+0000","last_became_active":"2024-03-11T20:54:37.429673+0000","last_became_peered":"2024-03-11T20:54:37.429673+0000","last_unstale":"2024-03-11T20:55:09.675106+0000","last_undegraded":"2024-03-11T20:55:09.675106+0000","last_fullsized":"2024-03-11T20:55:09.675106+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:08.170036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:53:08.170036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:14.652576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"87'2","reported_seq":146,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.674233+0000","last_change":"2024-03-11T20:53:25.900353+0000","last_active":"2024-03-11T20:55:09.674233+0000","last_peered":"2024-03-11T20:55:09.674233+0000","last_clean":"2024-03-11T20:55:09.674233+0000","last_became_active":"2024-03-11T20:50:32.195452+0000","last_became_peered":"2024-03-11T20:50:32.195452+0000","last_unstale":"2024-03-11T20:55:09.674233+0000","last_undegraded":"2024-03-11T20:55:09.674233+0000","last_fullsized":"2024-03-11T20:55:09.674233+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'2","last_scrub_stamp":"2024-03-11T20:53:25.900041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:53:25.900041+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:14.652672+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0024049620000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":145,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.678073+0000","last_change":"2024-03-11T20:53:05.364652+0000","last_active":"2024-03-11T20:55:09.678073+0000","last_peered":"2024-03-11T20:55:09.678073+0000","last_clean":"2024-03-11T20:55:09.678073+0000","last_became_active":"2024-03-11T20:50:28.065245+0000","last_became_peered":"2024-03-11T20:50:28.065245+0000","last_unstale":"2024-03-11T20:55:09.678073+0000","last_undegraded":"2024-03-11T20:55:09.678073+0000","last_fullsized":"2024-03-11T20:55:09.678073+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:05.364429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:53:05.364429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:12.204639+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"125'31","reported_seq":132,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.668147+0000","last_change":"2024-03-11T20:54:37.451852+0000","last_active":"2024-03-11T20:55:09.668147+0000","last_peered":"2024-03-11T20:55:09.668147+0000","last_clean":"2024-03-11T20:55:09.668147+0000","last_became_active":"2024-03-11T20:54:37.417305+0000","last_became_peered":"2024-03-11T20:54:37.417305+0000","last_unstale":"2024-03-11T20:55:09.668147+0000","last_undegraded":"2024-03-11T20:55:09.668147+0000","last_fullsized":"2024-03-11T20:55:09.668147+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'30","last_scrub_stamp":"2024-03-11T20:54:07.829704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:54:07.829704+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:23.864847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":51,"num_read_kb":36,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"84'11","reported_seq":203,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.674888+0000","last_change":"2024-03-11T20:53:35.960764+0000","last_active":"2024-03-11T20:55:09.674888+0000","last_peered":"2024-03-11T20:55:09.674888+0000","last_clean":"2024-03-11T20:55:09.674888+0000","last_became_active":"2024-03-11T20:50:32.649414+0000","last_became_peered":"2024-03-11T20:50:32.649414+0000","last_unstale":"2024-03-11T20:55:09.674888+0000","last_undegraded":"2024-03-11T20:55:09.674888+0000","last_fullsized":"2024-03-11T20:55:09.674888+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T20:53:35.960574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:53:35.960574+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:12.947758+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"135'2408","reported_seq":2445,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.813097+0000","last_change":"2024-03-11T20:54:34.363543+0000","last_active":"2024-03-11T20:55:09.813097+0000","last_peered":"2024-03-11T20:55:09.813097+0000","last_clean":"2024-03-11T20:55:09.813097+0000","last_became_active":"2024-03-11T20:54:34.362699+0000","last_became_peered":"2024-03-11T20:54:34.362699+0000","last_unstale":"2024-03-11T20:55:09.813097+0000","last_undegraded":"2024-03-11T20:55:09.813097+0000","last_fullsized":"2024-03-11T20:55:09.813097+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:54:33.308151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:54:33.308151+0000","last_clean_scrub_stamp":"2024-03-11T20:54:33.308151+0000","objects_scrubbed":0,"log_size":2408,"log_dups_size":0,"ondisk_log_size":2408,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:41.004933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":157810688,"num_objects":2408,"num_object_clones":0,"num_object_copies":7224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2408,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2408,"num_write_kb":154112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":133,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.669995+0000","last_change":"2024-03-11T20:53:27.419736+0000","last_active":"2024-03-11T20:55:09.669995+0000","last_peered":"2024-03-11T20:55:09.669995+0000","last_clean":"2024-03-11T20:55:09.669995+0000","last_became_active":"2024-03-11T20:50:34.323647+0000","last_became_peered":"2024-03-11T20:50:34.323647+0000","last_unstale":"2024-03-11T20:55:09.669995+0000","last_undegraded":"2024-03-11T20:55:09.669995+0000","last_fullsized":"2024-03-11T20:55:09.669995+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:27.419519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:53:27.419519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:12.376426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":144,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.671838+0000","last_change":"2024-03-11T20:53:32.530057+0000","last_active":"2024-03-11T20:55:09.671838+0000","last_peered":"2024-03-11T20:55:09.671838+0000","last_clean":"2024-03-11T20:55:09.671838+0000","last_became_active":"2024-03-11T20:50:32.650102+0000","last_became_peered":"2024-03-11T20:50:32.650102+0000","last_unstale":"2024-03-11T20:55:09.671838+0000","last_undegraded":"2024-03-11T20:55:09.671838+0000","last_fullsized":"2024-03-11T20:55:09.671838+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:32.529832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:53:32.529832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:12.376521+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.0022993710000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"77'1","reported_seq":44,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.676295+0000","last_change":"2024-03-11T20:54:37.442718+0000","last_active":"2024-03-11T20:55:09.676295+0000","last_peered":"2024-03-11T20:55:09.676295+0000","last_clean":"2024-03-11T20:55:09.676295+0000","last_became_active":"2024-03-11T20:54:37.422775+0000","last_became_peered":"2024-03-11T20:54:37.422775+0000","last_unstale":"2024-03-11T20:55:09.676295+0000","last_undegraded":"2024-03-11T20:55:09.676295+0000","last_fullsized":"2024-03-11T20:55:09.676295+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-03-11T20:53:26.158259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:53:26.158259+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:14.652560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.7","version":"84'13","reported_seq":167,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.674376+0000","last_change":"2024-03-11T20:53:27.387229+0000","last_active":"2024-03-11T20:55:09.674376+0000","last_peered":"2024-03-11T20:55:09.674376+0000","last_clean":"2024-03-11T20:55:09.674376+0000","last_became_active":"2024-03-11T20:50:30.097669+0000","last_became_peered":"2024-03-11T20:50:30.097669+0000","last_unstale":"2024-03-11T20:55:09.674376+0000","last_undegraded":"2024-03-11T20:55:09.674376+0000","last_fullsized":"2024-03-11T20:55:09.674376+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'13","last_scrub_stamp":"2024-03-11T20:53:27.386971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:53:27.386971+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"84'11","reported_seq":211,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.672711+0000","last_change":"2024-03-11T20:54:33.567964+0000","last_active":"2024-03-11T20:55:09.672711+0000","last_peered":"2024-03-11T20:55:09.672711+0000","last_clean":"2024-03-11T20:55:09.672711+0000","last_became_active":"2024-03-11T20:50:32.647041+0000","last_became_peered":"2024-03-11T20:50:32.647041+0000","last_unstale":"2024-03-11T20:55:09.672711+0000","last_undegraded":"2024-03-11T20:55:09.672711+0000","last_fullsized":"2024-03-11T20:55:09.672711+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T20:54:33.567751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:54:33.567751+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:39.321568+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"135'2329","reported_seq":2371,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.829878+0000","last_change":"2024-03-11T20:54:37.420912+0000","last_active":"2024-03-11T20:55:09.829878+0000","last_peered":"2024-03-11T20:55:09.829878+0000","last_clean":"2024-03-11T20:55:09.829878+0000","last_became_active":"2024-03-11T20:54:37.420427+0000","last_became_peered":"2024-03-11T20:54:37.420427+0000","last_unstale":"2024-03-11T20:55:09.829878+0000","last_undegraded":"2024-03-11T20:55:09.829878+0000","last_fullsized":"2024-03-11T20:55:09.829878+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:54:33.308151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:54:33.308151+0000","last_clean_scrub_stamp":"2024-03-11T20:54:33.308151+0000","objects_scrubbed":0,"log_size":2329,"log_dups_size":0,"ondisk_log_size":2329,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:56:02.533832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":152633344,"num_objects":2329,"num_object_clones":0,"num_object_copies":6987,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2329,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2329,"num_write_kb":149056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":141,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.677185+0000","last_change":"2024-03-11T20:54:15.186981+0000","last_active":"2024-03-11T20:55:09.677185+0000","last_peered":"2024-03-11T20:55:09.677185+0000","last_clean":"2024-03-11T20:55:09.677185+0000","last_became_active":"2024-03-11T20:50:34.326947+0000","last_became_peered":"2024-03-11T20:50:34.326947+0000","last_unstale":"2024-03-11T20:55:09.677185+0000","last_undegraded":"2024-03-11T20:55:09.677185+0000","last_fullsized":"2024-03-11T20:55:09.677185+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:54:15.186776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:54:15.186776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:42.215397+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"120'6","reported_seq":423,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.673250+0000","last_change":"2024-03-11T20:54:37.437063+0000","last_active":"2024-03-11T20:55:09.673250+0000","last_peered":"2024-03-11T20:55:09.673250+0000","last_clean":"2024-03-11T20:55:09.673250+0000","last_became_active":"2024-03-11T20:54:37.407080+0000","last_became_peered":"2024-03-11T20:54:37.407080+0000","last_unstale":"2024-03-11T20:55:09.673250+0000","last_undegraded":"2024-03-11T20:55:09.673250+0000","last_fullsized":"2024-03-11T20:55:09.673250+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"120'6","last_scrub_stamp":"2024-03-11T20:54:26.468057+0000","last_deep_scrub":"120'6","last_deep_scrub_stamp":"2024-03-11T20:54:26.468057+0000","last_clean_scrub_stamp":"2024-03-11T20:54:26.468057+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:53.852713+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0035533370000000002,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":257,"num_read_kb":252,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":145,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.675361+0000","last_change":"2024-03-11T20:53:31.863953+0000","last_active":"2024-03-11T20:55:09.675361+0000","last_peered":"2024-03-11T20:55:09.675361+0000","last_clean":"2024-03-11T20:55:09.675361+0000","last_became_active":"2024-03-11T20:50:28.071707+0000","last_became_peered":"2024-03-11T20:50:28.071707+0000","last_unstale":"2024-03-11T20:55:09.675361+0000","last_undegraded":"2024-03-11T20:55:09.675361+0000","last_fullsized":"2024-03-11T20:55:09.675361+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:31.863747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:53:31.863747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:14.652695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"84'12","reported_seq":178,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.676632+0000","last_change":"2024-03-11T20:54:37.431467+0000","last_active":"2024-03-11T20:55:09.676632+0000","last_peered":"2024-03-11T20:55:09.676632+0000","last_clean":"2024-03-11T20:55:09.676632+0000","last_became_active":"2024-03-11T20:54:37.410722+0000","last_became_peered":"2024-03-11T20:54:37.410722+0000","last_unstale":"2024-03-11T20:55:09.676632+0000","last_undegraded":"2024-03-11T20:55:09.676632+0000","last_fullsized":"2024-03-11T20:55:09.676632+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'12","last_scrub_stamp":"2024-03-11T20:53:32.942979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:53:32.942979+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:12.947790+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":137,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.672145+0000","last_change":"2024-03-11T20:53:33.556343+0000","last_active":"2024-03-11T20:55:09.672145+0000","last_peered":"2024-03-11T20:55:09.672145+0000","last_clean":"2024-03-11T20:55:09.672145+0000","last_became_active":"2024-03-11T20:50:32.190896+0000","last_became_peered":"2024-03-11T20:50:32.190896+0000","last_unstale":"2024-03-11T20:55:09.672145+0000","last_undegraded":"2024-03-11T20:55:09.672145+0000","last_fullsized":"2024-03-11T20:55:09.672145+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:33.556062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:53:33.556062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:12.376659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"135'2283","reported_seq":2327,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.835359+0000","last_change":"2024-03-11T20:54:37.423506+0000","last_active":"2024-03-11T20:55:09.835359+0000","last_peered":"2024-03-11T20:55:09.835359+0000","last_clean":"2024-03-11T20:55:09.835359+0000","last_became_active":"2024-03-11T20:54:37.422855+0000","last_became_peered":"2024-03-11T20:54:37.422855+0000","last_unstale":"2024-03-11T20:55:09.835359+0000","last_undegraded":"2024-03-11T20:55:09.835359+0000","last_fullsized":"2024-03-11T20:55:09.835359+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:54:33.308151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:54:33.308151+0000","last_clean_scrub_stamp":"2024-03-11T20:54:33.308151+0000","objects_scrubbed":0,"log_size":2283,"log_dups_size":0,"ondisk_log_size":2283,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:43.118700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":149618688,"num_objects":2283,"num_object_clones":0,"num_object_copies":6849,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2283,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2283,"num_write_kb":146112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":132,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.675374+0000","last_change":"2024-03-11T20:53:35.172511+0000","last_active":"2024-03-11T20:55:09.675374+0000","last_peered":"2024-03-11T20:55:09.675374+0000","last_clean":"2024-03-11T20:55:09.675374+0000","last_became_active":"2024-03-11T20:50:34.335370+0000","last_became_peered":"2024-03-11T20:50:34.335370+0000","last_unstale":"2024-03-11T20:55:09.675374+0000","last_undegraded":"2024-03-11T20:55:09.675374+0000","last_fullsized":"2024-03-11T20:55:09.675374+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:35.172297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:53:35.172297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"125'12","reported_seq":215,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.673175+0000","last_change":"2024-03-11T20:54:37.432833+0000","last_active":"2024-03-11T20:55:09.673175+0000","last_peered":"2024-03-11T20:55:09.673175+0000","last_clean":"2024-03-11T20:55:09.673175+0000","last_became_active":"2024-03-11T20:54:37.396206+0000","last_became_peered":"2024-03-11T20:54:37.396206+0000","last_unstale":"2024-03-11T20:55:09.673175+0000","last_undegraded":"2024-03-11T20:55:09.673175+0000","last_fullsized":"2024-03-11T20:55:09.673175+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T20:54:10.662442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:54:10.662442+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:21.618225+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":151,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.677308+0000","last_change":"2024-03-11T20:54:37.388329+0000","last_active":"2024-03-11T20:55:09.677308+0000","last_peered":"2024-03-11T20:55:09.677308+0000","last_clean":"2024-03-11T20:55:09.677308+0000","last_became_active":"2024-03-11T20:54:37.387539+0000","last_became_peered":"2024-03-11T20:54:37.387539+0000","last_unstale":"2024-03-11T20:55:09.677308+0000","last_undegraded":"2024-03-11T20:55:09.677308+0000","last_fullsized":"2024-03-11T20:55:09.677308+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:28.297455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:52:09.043655+0000","last_clean_scrub_stamp":"2024-03-11T20:53:28.297455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:12.204731+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"84'12","reported_seq":164,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.671950+0000","last_change":"2024-03-11T20:53:22.404447+0000","last_active":"2024-03-11T20:55:09.671950+0000","last_peered":"2024-03-11T20:55:09.671950+0000","last_clean":"2024-03-11T20:55:09.671950+0000","last_became_active":"2024-03-11T20:50:30.083388+0000","last_became_peered":"2024-03-11T20:50:30.083388+0000","last_unstale":"2024-03-11T20:55:09.671950+0000","last_undegraded":"2024-03-11T20:55:09.671950+0000","last_fullsized":"2024-03-11T20:55:09.671950+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'12","last_scrub_stamp":"2024-03-11T20:53:22.404131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:53:22.404131+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:12.376489+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"135'2365","reported_seq":2403,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.817389+0000","last_change":"2024-03-11T20:54:34.364423+0000","last_active":"2024-03-11T20:55:09.817389+0000","last_peered":"2024-03-11T20:55:09.817389+0000","last_clean":"2024-03-11T20:55:09.817389+0000","last_became_active":"2024-03-11T20:54:34.364044+0000","last_became_peered":"2024-03-11T20:54:34.364044+0000","last_unstale":"2024-03-11T20:55:09.817389+0000","last_undegraded":"2024-03-11T20:55:09.817389+0000","last_fullsized":"2024-03-11T20:55:09.817389+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:54:33.308151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:54:33.308151+0000","last_clean_scrub_stamp":"2024-03-11T20:54:33.308151+0000","objects_scrubbed":0,"log_size":2365,"log_dups_size":0,"ondisk_log_size":2365,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:57.123562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154992640,"num_objects":2365,"num_object_clones":0,"num_object_copies":7095,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2365,"num_write_kb":151360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":139,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.675353+0000","last_change":"2024-03-11T20:54:37.425287+0000","last_active":"2024-03-11T20:55:09.675353+0000","last_peered":"2024-03-11T20:55:09.675353+0000","last_clean":"2024-03-11T20:55:09.675353+0000","last_became_active":"2024-03-11T20:54:37.424704+0000","last_became_peered":"2024-03-11T20:54:37.424704+0000","last_unstale":"2024-03-11T20:55:09.675353+0000","last_undegraded":"2024-03-11T20:55:09.675353+0000","last_fullsized":"2024-03-11T20:55:09.675353+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:06.771620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:53:06.771620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:11.510815+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":33,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.674462+0000","last_change":"2024-03-11T20:54:37.406783+0000","last_active":"2024-03-11T20:55:09.674462+0000","last_peered":"2024-03-11T20:55:09.674462+0000","last_clean":"2024-03-11T20:55:09.674462+0000","last_became_active":"2024-03-11T20:54:37.406397+0000","last_became_peered":"2024-03-11T20:54:37.406397+0000","last_unstale":"2024-03-11T20:55:09.674462+0000","last_undegraded":"2024-03-11T20:55:09.674462+0000","last_fullsized":"2024-03-11T20:55:09.674462+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:54:24.740149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:54:24.740149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:43.896969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":32,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.677177+0000","last_change":"2024-03-11T20:54:37.412752+0000","last_active":"2024-03-11T20:55:09.677177+0000","last_peered":"2024-03-11T20:55:09.677177+0000","last_clean":"2024-03-11T20:55:09.677177+0000","last_became_active":"2024-03-11T20:54:37.411996+0000","last_became_peered":"2024-03-11T20:54:37.411996+0000","last_unstale":"2024-03-11T20:55:09.677177+0000","last_undegraded":"2024-03-11T20:55:09.677177+0000","last_fullsized":"2024-03-11T20:55:09.677177+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:54:29.858259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:40.974147+0000","last_clean_scrub_stamp":"2024-03-11T20:54:29.858259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:55.095999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"116'22","reported_seq":194,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.672013+0000","last_change":"2024-03-11T20:54:37.436121+0000","last_active":"2024-03-11T20:55:09.672013+0000","last_peered":"2024-03-11T20:55:09.672013+0000","last_clean":"2024-03-11T20:55:09.672013+0000","last_became_active":"2024-03-11T20:54:37.408552+0000","last_became_peered":"2024-03-11T20:54:37.408552+0000","last_unstale":"2024-03-11T20:55:09.672013+0000","last_undegraded":"2024-03-11T20:55:09.672013+0000","last_fullsized":"2024-03-11T20:55:09.672013+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'22","last_scrub_stamp":"2024-03-11T20:53:47.364709+0000","last_deep_scrub":"116'22","last_deep_scrub_stamp":"2024-03-11T20:53:47.364709+0000","last_clean_scrub_stamp":"2024-03-11T20:53:47.364709+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:12.343940+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.f","version":"135'2430","reported_seq":2468,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.803667+0000","last_change":"2024-03-11T20:54:34.354745+0000","last_active":"2024-03-11T20:55:09.803667+0000","last_peered":"2024-03-11T20:55:09.803667+0000","last_clean":"2024-03-11T20:55:09.803667+0000","last_became_active":"2024-03-11T20:54:34.353660+0000","last_became_peered":"2024-03-11T20:54:34.353660+0000","last_unstale":"2024-03-11T20:55:09.803667+0000","last_undegraded":"2024-03-11T20:55:09.803667+0000","last_fullsized":"2024-03-11T20:55:09.803667+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:54:33.308151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:54:33.308151+0000","last_clean_scrub_stamp":"2024-03-11T20:54:33.308151+0000","objects_scrubbed":0,"log_size":2430,"log_dups_size":0,"ondisk_log_size":2430,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:56:00.249873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159252480,"num_objects":2430,"num_object_clones":0,"num_object_copies":7290,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2430,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2430,"num_write_kb":155520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":139,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.673394+0000","last_change":"2024-03-11T20:54:37.382047+0000","last_active":"2024-03-11T20:55:09.673394+0000","last_peered":"2024-03-11T20:55:09.673394+0000","last_clean":"2024-03-11T20:55:09.673394+0000","last_became_active":"2024-03-11T20:54:37.381187+0000","last_became_peered":"2024-03-11T20:54:37.381187+0000","last_unstale":"2024-03-11T20:55:09.673394+0000","last_undegraded":"2024-03-11T20:55:09.673394+0000","last_fullsized":"2024-03-11T20:55:09.673394+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:57.644041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:53:57.644041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:14.403182+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":31,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.695442+0000","last_change":"2024-03-11T20:54:37.408849+0000","last_active":"2024-03-11T20:55:09.695442+0000","last_peered":"2024-03-11T20:55:09.695442+0000","last_clean":"2024-03-11T20:55:09.695442+0000","last_became_active":"2024-03-11T20:54:37.408476+0000","last_became_peered":"2024-03-11T20:54:37.408476+0000","last_unstale":"2024-03-11T20:55:09.695442+0000","last_undegraded":"2024-03-11T20:55:09.695442+0000","last_fullsized":"2024-03-11T20:55:09.695442+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:37.093963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:53:37.093963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"84'9","reported_seq":164,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.672429+0000","last_change":"2024-03-11T20:54:28.501648+0000","last_active":"2024-03-11T20:55:09.672429+0000","last_peered":"2024-03-11T20:55:09.672429+0000","last_clean":"2024-03-11T20:55:09.672429+0000","last_became_active":"2024-03-11T20:50:30.089260+0000","last_became_peered":"2024-03-11T20:50:30.089260+0000","last_unstale":"2024-03-11T20:55:09.672429+0000","last_undegraded":"2024-03-11T20:55:09.672429+0000","last_fullsized":"2024-03-11T20:55:09.672429+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'9","last_scrub_stamp":"2024-03-11T20:54:28.501486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:54:28.501486+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:46.259869+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":145,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.672052+0000","last_change":"2024-03-11T20:54:14.631875+0000","last_active":"2024-03-11T20:55:09.672052+0000","last_peered":"2024-03-11T20:55:09.672052+0000","last_clean":"2024-03-11T20:55:09.672052+0000","last_became_active":"2024-03-11T20:50:32.200001+0000","last_became_peered":"2024-03-11T20:50:32.200001+0000","last_unstale":"2024-03-11T20:55:09.672052+0000","last_undegraded":"2024-03-11T20:55:09.672052+0000","last_fullsized":"2024-03-11T20:55:09.672052+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:54:14.631661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:35.670149+0000","last_clean_scrub_stamp":"2024-03-11T20:54:14.631661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:16.301224+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"135'2386","reported_seq":2423,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.821547+0000","last_change":"2024-03-11T20:54:34.360042+0000","last_active":"2024-03-11T20:55:09.821547+0000","last_peered":"2024-03-11T20:55:09.821547+0000","last_clean":"2024-03-11T20:55:09.821547+0000","last_became_active":"2024-03-11T20:54:34.345273+0000","last_became_peered":"2024-03-11T20:54:34.345273+0000","last_unstale":"2024-03-11T20:55:09.821547+0000","last_undegraded":"2024-03-11T20:55:09.821547+0000","last_fullsized":"2024-03-11T20:55:09.821547+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:54:33.308151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:54:33.308151+0000","last_clean_scrub_stamp":"2024-03-11T20:54:33.308151+0000","objects_scrubbed":0,"log_size":2386,"log_dups_size":0,"ondisk_log_size":2386,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:42.290812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":156368896,"num_objects":2386,"num_object_clones":0,"num_object_copies":7158,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2386,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2386,"num_write_kb":152704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":140,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.668650+0000","last_change":"2024-03-11T20:54:05.781005+0000","last_active":"2024-03-11T20:55:09.668650+0000","last_peered":"2024-03-11T20:55:09.668650+0000","last_clean":"2024-03-11T20:55:09.668650+0000","last_became_active":"2024-03-11T20:50:34.326354+0000","last_became_peered":"2024-03-11T20:50:34.326354+0000","last_unstale":"2024-03-11T20:55:09.668650+0000","last_undegraded":"2024-03-11T20:55:09.668650+0000","last_fullsized":"2024-03-11T20:55:09.668650+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:54:05.780787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:54:05.780787+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:11.328101+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"125'13","reported_seq":225,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.672963+0000","last_change":"2024-03-11T20:54:37.425992+0000","last_active":"2024-03-11T20:55:09.672963+0000","last_peered":"2024-03-11T20:55:09.672963+0000","last_clean":"2024-03-11T20:55:09.672963+0000","last_became_active":"2024-03-11T20:54:37.396038+0000","last_became_peered":"2024-03-11T20:54:37.396038+0000","last_unstale":"2024-03-11T20:55:09.672963+0000","last_undegraded":"2024-03-11T20:55:09.672963+0000","last_fullsized":"2024-03-11T20:55:09.672963+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'12","last_scrub_stamp":"2024-03-11T20:53:42.564785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:53:42.564785+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:12.376643+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":153,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.677283+0000","last_change":"2024-03-11T20:54:05.131290+0000","last_active":"2024-03-11T20:55:09.677283+0000","last_peered":"2024-03-11T20:55:09.677283+0000","last_clean":"2024-03-11T20:55:09.677283+0000","last_became_active":"2024-03-11T20:50:28.069267+0000","last_became_peered":"2024-03-11T20:50:28.069267+0000","last_unstale":"2024-03-11T20:55:09.677283+0000","last_undegraded":"2024-03-11T20:55:09.677283+0000","last_fullsized":"2024-03-11T20:55:09.677283+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:54:05.131036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:52:51.161281+0000","last_clean_scrub_stamp":"2024-03-11T20:54:05.131036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:22.581351+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"84'19","reported_seq":198,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.674386+0000","last_change":"2024-03-11T20:54:37.453384+0000","last_active":"2024-03-11T20:55:09.674386+0000","last_peered":"2024-03-11T20:55:09.674386+0000","last_clean":"2024-03-11T20:55:09.674386+0000","last_became_active":"2024-03-11T20:54:37.423519+0000","last_became_peered":"2024-03-11T20:54:37.423519+0000","last_unstale":"2024-03-11T20:55:09.674386+0000","last_undegraded":"2024-03-11T20:55:09.674386+0000","last_fullsized":"2024-03-11T20:55:09.674386+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'19","last_scrub_stamp":"2024-03-11T20:53:16.962364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:53:16.962364+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:14.652539+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.9","version":"135'2289","reported_seq":2333,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.843410+0000","last_change":"2024-03-11T20:54:37.406919+0000","last_active":"2024-03-11T20:55:09.843410+0000","last_peered":"2024-03-11T20:55:09.843410+0000","last_clean":"2024-03-11T20:55:09.843410+0000","last_became_active":"2024-03-11T20:54:37.406057+0000","last_became_peered":"2024-03-11T20:54:37.406057+0000","last_unstale":"2024-03-11T20:55:09.843410+0000","last_undegraded":"2024-03-11T20:55:09.843410+0000","last_fullsized":"2024-03-11T20:55:09.843410+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:54:33.308151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:54:33.308151+0000","last_clean_scrub_stamp":"2024-03-11T20:54:33.308151+0000","objects_scrubbed":0,"log_size":2289,"log_dups_size":0,"ondisk_log_size":2289,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:35.105874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150011904,"num_objects":2289,"num_object_clones":0,"num_object_copies":6867,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2289,"num_write_kb":146496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":133,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.676836+0000","last_change":"2024-03-11T20:53:18.331836+0000","last_active":"2024-03-11T20:55:09.676836+0000","last_peered":"2024-03-11T20:55:09.676836+0000","last_clean":"2024-03-11T20:55:09.676836+0000","last_became_active":"2024-03-11T20:50:34.327139+0000","last_became_peered":"2024-03-11T20:50:34.327139+0000","last_unstale":"2024-03-11T20:55:09.676836+0000","last_undegraded":"2024-03-11T20:55:09.676836+0000","last_fullsized":"2024-03-11T20:55:09.676836+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:18.331616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:53:18.331616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:12.204766+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":152,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.667954+0000","last_change":"2024-03-11T20:54:12.793141+0000","last_active":"2024-03-11T20:55:09.667954+0000","last_peered":"2024-03-11T20:55:09.667954+0000","last_clean":"2024-03-11T20:55:09.667954+0000","last_became_active":"2024-03-11T20:50:28.049901+0000","last_became_peered":"2024-03-11T20:50:28.049901+0000","last_unstale":"2024-03-11T20:55:09.667954+0000","last_undegraded":"2024-03-11T20:55:09.667954+0000","last_fullsized":"2024-03-11T20:55:09.667954+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:54:12.792891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:54:12.792891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:37.844847+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"102'25","reported_seq":185,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.678266+0000","last_change":"2024-03-11T20:53:57.252853+0000","last_active":"2024-03-11T20:55:09.678266+0000","last_peered":"2024-03-11T20:55:09.678266+0000","last_clean":"2024-03-11T20:55:09.678266+0000","last_became_active":"2024-03-11T20:50:30.111819+0000","last_became_peered":"2024-03-11T20:50:30.111819+0000","last_unstale":"2024-03-11T20:55:09.678266+0000","last_undegraded":"2024-03-11T20:55:09.678266+0000","last_fullsized":"2024-03-11T20:55:09.678266+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"102'25","last_scrub_stamp":"2024-03-11T20:53:57.252617+0000","last_deep_scrub":"102'25","last_deep_scrub_stamp":"2024-03-11T20:51:35.187094+0000","last_clean_scrub_stamp":"2024-03-11T20:53:57.252617+0000","objects_scrubbed":8,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:11.154783+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":143,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.674156+0000","last_change":"2024-03-11T20:54:37.396626+0000","last_active":"2024-03-11T20:55:09.674156+0000","last_peered":"2024-03-11T20:55:09.674156+0000","last_clean":"2024-03-11T20:55:09.674156+0000","last_became_active":"2024-03-11T20:54:37.395311+0000","last_became_peered":"2024-03-11T20:54:37.395311+0000","last_unstale":"2024-03-11T20:55:09.674156+0000","last_undegraded":"2024-03-11T20:55:09.674156+0000","last_fullsized":"2024-03-11T20:55:09.674156+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:29.933643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:53:29.933643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:14.652628+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.8","version":"135'2281","reported_seq":2318,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.837836+0000","last_change":"2024-03-11T20:54:34.343671+0000","last_active":"2024-03-11T20:55:09.837836+0000","last_peered":"2024-03-11T20:55:09.837836+0000","last_clean":"2024-03-11T20:55:09.837836+0000","last_became_active":"2024-03-11T20:54:34.342864+0000","last_became_peered":"2024-03-11T20:54:34.342864+0000","last_unstale":"2024-03-11T20:55:09.837836+0000","last_undegraded":"2024-03-11T20:55:09.837836+0000","last_fullsized":"2024-03-11T20:55:09.837836+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:54:33.308151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:54:33.308151+0000","last_clean_scrub_stamp":"2024-03-11T20:54:33.308151+0000","objects_scrubbed":0,"log_size":2281,"log_dups_size":0,"ondisk_log_size":2281,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:38.895445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":149487616,"num_objects":2281,"num_object_clones":0,"num_object_copies":6843,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2281,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2281,"num_write_kb":145984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":140,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.677260+0000","last_change":"2024-03-11T20:53:55.885120+0000","last_active":"2024-03-11T20:55:09.677260+0000","last_peered":"2024-03-11T20:55:09.677260+0000","last_clean":"2024-03-11T20:55:09.677260+0000","last_became_active":"2024-03-11T20:50:34.319536+0000","last_became_peered":"2024-03-11T20:50:34.319536+0000","last_unstale":"2024-03-11T20:55:09.677260+0000","last_undegraded":"2024-03-11T20:55:09.677260+0000","last_fullsized":"2024-03-11T20:55:09.677260+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:55.884826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:53:55.884826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:10.910507+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":32,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.672636+0000","last_change":"2024-03-11T20:54:37.378788+0000","last_active":"2024-03-11T20:55:09.672636+0000","last_peered":"2024-03-11T20:55:09.672636+0000","last_clean":"2024-03-11T20:55:09.672636+0000","last_became_active":"2024-03-11T20:54:37.377800+0000","last_became_peered":"2024-03-11T20:54:37.377800+0000","last_unstale":"2024-03-11T20:55:09.672636+0000","last_undegraded":"2024-03-11T20:55:09.672636+0000","last_fullsized":"2024-03-11T20:55:09.672636+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:30.139539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:53:30.139539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:12.376675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"84'10","reported_seq":156,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.674719+0000","last_change":"2024-03-11T20:53:12.823097+0000","last_active":"2024-03-11T20:55:09.674719+0000","last_peered":"2024-03-11T20:55:09.674719+0000","last_clean":"2024-03-11T20:55:09.674719+0000","last_became_active":"2024-03-11T20:50:30.109549+0000","last_became_peered":"2024-03-11T20:50:30.109549+0000","last_unstale":"2024-03-11T20:55:09.674719+0000","last_undegraded":"2024-03-11T20:55:09.674719+0000","last_fullsized":"2024-03-11T20:55:09.674719+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'10","last_scrub_stamp":"2024-03-11T20:53:12.822842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:53:12.822842+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:11.510883+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":150,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.675032+0000","last_change":"2024-03-11T20:54:37.409118+0000","last_active":"2024-03-11T20:55:09.675032+0000","last_peered":"2024-03-11T20:55:09.675032+0000","last_clean":"2024-03-11T20:55:09.675032+0000","last_became_active":"2024-03-11T20:54:37.408396+0000","last_became_peered":"2024-03-11T20:54:37.408396+0000","last_unstale":"2024-03-11T20:55:09.675032+0000","last_undegraded":"2024-03-11T20:55:09.675032+0000","last_fullsized":"2024-03-11T20:55:09.675032+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:59.740665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:52:51.931295+0000","last_clean_scrub_stamp":"2024-03-11T20:53:59.740665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:27.517058+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.b","version":"135'2299","reported_seq":2332,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.824345+0000","last_change":"2024-03-11T20:54:37.415822+0000","last_active":"2024-03-11T20:55:09.824345+0000","last_peered":"2024-03-11T20:55:09.824345+0000","last_clean":"2024-03-11T20:55:09.824345+0000","last_became_active":"2024-03-11T20:54:37.414173+0000","last_became_peered":"2024-03-11T20:54:37.414173+0000","last_unstale":"2024-03-11T20:55:09.824345+0000","last_undegraded":"2024-03-11T20:55:09.824345+0000","last_fullsized":"2024-03-11T20:55:09.824345+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:54:33.308151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:54:33.308151+0000","last_clean_scrub_stamp":"2024-03-11T20:54:33.308151+0000","objects_scrubbed":0,"log_size":2299,"log_dups_size":0,"ondisk_log_size":2299,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:39.327822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150667264,"num_objects":2299,"num_object_clones":0,"num_object_copies":6897,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2299,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2299,"num_write_kb":147136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":133,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.676248+0000","last_change":"2024-03-11T20:53:03.762967+0000","last_active":"2024-03-11T20:55:09.676248+0000","last_peered":"2024-03-11T20:55:09.676248+0000","last_clean":"2024-03-11T20:55:09.676248+0000","last_became_active":"2024-03-11T20:50:34.337173+0000","last_became_peered":"2024-03-11T20:50:34.337173+0000","last_unstale":"2024-03-11T20:55:09.676248+0000","last_undegraded":"2024-03-11T20:55:09.676248+0000","last_fullsized":"2024-03-11T20:55:09.676248+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:03.762882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:53:03.762882+0000","last_clean_scrub_stamp":"2024-03-11T20:53:03.762882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:11.510912+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":152,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.670616+0000","last_change":"2024-03-11T20:54:16.747554+0000","last_active":"2024-03-11T20:55:09.670616+0000","last_peered":"2024-03-11T20:55:09.670616+0000","last_clean":"2024-03-11T20:55:09.670616+0000","last_became_active":"2024-03-11T20:50:28.058942+0000","last_became_peered":"2024-03-11T20:50:28.058942+0000","last_unstale":"2024-03-11T20:55:09.670616+0000","last_undegraded":"2024-03-11T20:55:09.670616+0000","last_fullsized":"2024-03-11T20:55:09.670616+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:54:16.747387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:54:16.747387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:30.413193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"84'15","reported_seq":166,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.679120+0000","last_change":"2024-03-11T20:53:09.345898+0000","last_active":"2024-03-11T20:55:09.679120+0000","last_peered":"2024-03-11T20:55:09.679120+0000","last_clean":"2024-03-11T20:55:09.679120+0000","last_became_active":"2024-03-11T20:50:30.101512+0000","last_became_peered":"2024-03-11T20:50:30.101512+0000","last_unstale":"2024-03-11T20:55:09.679120+0000","last_undegraded":"2024-03-11T20:55:09.679120+0000","last_fullsized":"2024-03-11T20:55:09.679120+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15","last_scrub_stamp":"2024-03-11T20:53:09.345405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:53:09.345405+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T20:55:12.204671+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":137,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-11T20:55:09.674861+0000","last_change":"2024-03-11T20:53:44.824870+0000","last_active":"2024-03-11T20:55:09.674861+0000","last_peered":"2024-03-11T20:55:09.674861+0000","last_clean":"2024-03-11T20:55:09.674861+0000","last_became_active":"2024-03-11T20:50:32.652854+0000","last_became_peered":"2024-03-11T20:50:32.652854+0000","last_unstale":"2024-03-11T20:55:09.674861+0000","last_undegraded":"2024-03-11T20:55:09.674861+0000","last_fullsized":"2024-03-11T20:55:09.674861+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:53:44.824748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:53:44.824748+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":2465595392,"num_objects":37622,"num_object_clones":0,"num_object_copies":112866,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37622,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":37622,"num_write_kb":2407808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":7399342080,"data_stored":7399342080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":37622,"ondisk_log_size":37622,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":957,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":5550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":700,"num_read_kb":457,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":71,"num_bytes_recovered":1458,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":475136,"data_stored":12564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":445,"ondisk_log_size":445,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":257,"num_read_kb":252,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":68,"seq":292057776195,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1089676,"kb_used_data":1027972,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":92655476,"statfs":{"total":95995035648,"available":94879207424,"internally_reserved":0,"allocated":1052643328,"data_stored":1052193087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":63175114},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":57,"seq":244813135949,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":927748,"kb_used_data":892412,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":92817404,"statfs":{"total":95995035648,"available":95045021696,"internally_reserved":0,"allocated":913829888,"data_stored":913388069,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":55834574976,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738507,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1228008,"kb_used_data":1170464,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":92517144,"statfs":{"total":95995035648,"available":94737555456,"internally_reserved":0,"allocated":1198555136,"data_stored":1198076352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378742,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1246516,"kb_used_data":1188908,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":92498636,"statfs":{"total":95995035648,"available":94718603264,"internally_reserved":0,"allocated":1217441792,"data_stored":1217014797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":28,"seq":120259084395,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1513152,"kb_used_data":1451448,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":92232000,"statfs":{"total":95995035648,"available":94445568000,"internally_reserved":0,"allocated":1486282752,"data_stored":1485844001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":63175112},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,4],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":35,"seq":150323855458,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":760540,"kb_used_data":733524,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":92984612,"statfs":{"total":95995035648,"available":95216242688,"internally_reserved":0,"allocated":751128576,"data_stored":750662387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":44,"seq":188978561111,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":627704,"kb_used_data":592368,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93117448,"statfs":{"total":95995035648,"available":95352266752,"internally_reserved":0,"allocated":606584832,"data_stored":606154464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1243086848,"data_stored":1243086848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1232273408,"data_stored":1232273408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1545142272,"data_stored":1545142272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":764805120,"data_stored":764805120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":613089280,"data_stored":613089280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":920190976,"data_stored":920190976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1080754176,"data_stored":1080754176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T20:55:19.174 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd pool set .mgr pgp_num 1 2024-03-11T20:55:19.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:19.176 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:55:19.176 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:55:19.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:19 smithi146 bash[23809]: cluster 2024-03-11T20:55:17.908448+0000 mgr.y (mgr.24380) 173 : cluster 0 pgmap v149: 148 pgs: 148 active+clean; 2.7 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 1023 B/s rd, 98 MiB/s wr, 1.58k op/s 2024-03-11T20:55:19.276 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:55:19.321 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:55:19.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:19.325 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:55:19.426 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:19.449 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:19.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:19.452 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:55:19.542 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 28 16 50415 50399 112.472 77.0625 0.013121 0.00888464 2024-03-11T20:55:19.553 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:55:19.577 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:55:19.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:19.582 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:55:19.683 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:55:19.712 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:55:19.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:19.716 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:55:19.817 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:55:19.851 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:55:19.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:19.853 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:55:19.954 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:19.986 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:19.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:19.989 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:55:20.090 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:55:20.137 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:55:20.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:20.140 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:55:20.241 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:20.251 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:20 smithi146 bash[23809]: audit 2024-03-11T20:55:19.343250+0000 mon.a (mon.0) 973 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:55:20.264 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:20.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:20.267 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:55:20.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:20 smithi023 bash[21372]: audit 2024-03-11T20:55:19.343250+0000 mon.a (mon.0) 973 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:55:20.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:20 smithi023 bash[29146]: audit 2024-03-11T20:55:19.343250+0000 mon.a (mon.0) 973 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:55:20.368 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:20.394 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:20.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:20.396 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:55:20.497 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:55:20.525 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:55:20.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:20.527 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:55:20.541 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 29 16 51767 51751 111.507 84.5 0.00978766 0.00896203 2024-03-11T20:55:20.628 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:55:20.646 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:55:20.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:20.648 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:55:20.749 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:20.781 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:20.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:20.786 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:55:20.887 INFO:teuthology.orchestra.run.smithi023.stdout:osd.0: {} 2024-03-11T20:55:20.887 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:20.913 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:20.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:20.916 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:55:20.935 INFO:teuthology.orchestra.run.smithi023.stdout:osd.1: {} 2024-03-11T20:55:20.982 INFO:teuthology.orchestra.run.smithi023.stdout:osd.2: {} 2024-03-11T20:55:21.016 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:55:21.024 INFO:teuthology.orchestra.run.smithi023.stdout:osd.3: {} 2024-03-11T20:55:21.027 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:21 smithi023 bash[21372]: cluster 2024-03-11T20:55:19.909393+0000 mgr.y (mgr.24380) 174 : cluster 0 pgmap v150: 148 pgs: 148 active+clean; 3.1 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 1.5 KiB/s rd, 131 MiB/s wr, 2.10k op/s 2024-03-11T20:55:21.027 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:21 smithi023 bash[21372]: audit 2024-03-11T20:55:20.834915+0000 mon.a (mon.0) 974 : audit 0 from='client.? 172.21.15.23:0/979560704' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T20:55:21.027 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:21 smithi023 bash[29146]: cluster 2024-03-11T20:55:19.909393+0000 mgr.y (mgr.24380) 174 : cluster 0 pgmap v150: 148 pgs: 148 active+clean; 3.1 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 1.5 KiB/s rd, 131 MiB/s wr, 2.10k op/s 2024-03-11T20:55:21.027 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:21 smithi023 bash[29146]: audit 2024-03-11T20:55:20.834915+0000 mon.a (mon.0) 974 : audit 0 from='client.? 172.21.15.23:0/979560704' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T20:55:21.042 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:55:21.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:21.045 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:55:21.074 INFO:teuthology.orchestra.run.smithi023.stdout:osd.4: {} 2024-03-11T20:55:21.114 INFO:teuthology.orchestra.run.smithi023.stdout:osd.5: {} 2024-03-11T20:55:21.146 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:21.152 INFO:teuthology.orchestra.run.smithi023.stdout:osd.6: {} 2024-03-11T20:55:21.182 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:21.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:21.184 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:55:21.191 INFO:teuthology.orchestra.run.smithi023.stdout:osd.7: {} 2024-03-11T20:55:21.196 INFO:teuthology.orchestra.run.smithi023.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T20:55:21.196 INFO:teuthology.orchestra.run.smithi023.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T20:55:21.196 INFO:teuthology.orchestra.run.smithi023.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T20:55:21.196 INFO:teuthology.orchestra.run.smithi023.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T20:55:21.196 INFO:teuthology.orchestra.run.smithi023.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T20:55:21.196 INFO:teuthology.orchestra.run.smithi023.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T20:55:21.196 INFO:teuthology.orchestra.run.smithi023.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T20:55:21.196 INFO:teuthology.orchestra.run.smithi023.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T20:55:21.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:21 smithi146 bash[23809]: cluster 2024-03-11T20:55:19.909393+0000 mgr.y (mgr.24380) 174 : cluster 0 pgmap v150: 148 pgs: 148 active+clean; 3.1 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 1.5 KiB/s rd, 131 MiB/s wr, 2.10k op/s 2024-03-11T20:55:21.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:21 smithi146 bash[23809]: audit 2024-03-11T20:55:20.834915+0000 mon.a (mon.0) 974 : audit 0 from='client.? 172.21.15.23:0/979560704' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T20:55:21.285 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:55:21.317 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:55:21.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:21.321 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:55:21.422 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:55:21.448 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:55:21.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:21.451 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:55:21.541 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 30 16 52968 52952 110.292 75.0625 0.0206661 0.00906001 2024-03-11T20:55:21.551 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:55:21.577 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:55:21.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:21.579 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:55:21.678 INFO:teuthology.orchestra.run.smithi023.stderr:noscrub is unset 2024-03-11T20:55:21.680 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:21.709 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:21.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:21.714 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:55:21.814 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:55:21.840 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:55:21.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:21.846 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:55:21.947 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:55:21.971 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:55:21.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:21.972 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:55:22.074 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:22.102 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:22.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:22.106 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:55:22.207 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:22.219 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:22 smithi146 bash[23809]: cluster 2024-03-11T20:55:20.780850+0000 osd.0 (osd.0) 109 : cluster 0 purged_snaps scrub starts 2024-03-11T20:55:22.219 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:22 smithi146 bash[23809]: cluster 2024-03-11T20:55:20.780921+0000 osd.0 (osd.0) 110 : cluster 0 purged_snaps scrub ok 2024-03-11T20:55:22.219 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:22 smithi146 bash[23809]: audit 2024-03-11T20:55:21.295395+0000 mon.c (mon.2) 31 : audit 1 from='client.? 172.21.15.23:0/1276696761' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T20:55:22.219 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:22 smithi146 bash[23809]: audit 2024-03-11T20:55:21.295999+0000 mon.a (mon.0) 975 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T20:55:22.219 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:22 smithi146 bash[23809]: cluster 2024-03-11T20:55:21.665716+0000 mon.a (mon.0) 976 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T20:55:22.219 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:22 smithi146 bash[23809]: audit 2024-03-11T20:55:21.670854+0000 mon.a (mon.0) 977 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T20:55:22.219 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:22 smithi146 bash[23809]: cluster 2024-03-11T20:55:21.678938+0000 mon.a (mon.0) 978 : cluster 0 osdmap e136: 8 total, 8 up, 7 in 2024-03-11T20:55:22.237 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:22.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:22.239 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:55:22.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:22 smithi023 bash[21372]: cluster 2024-03-11T20:55:20.780850+0000 osd.0 (osd.0) 109 : cluster 0 purged_snaps scrub starts 2024-03-11T20:55:22.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:22 smithi023 bash[21372]: cluster 2024-03-11T20:55:20.780921+0000 osd.0 (osd.0) 110 : cluster 0 purged_snaps scrub ok 2024-03-11T20:55:22.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:22 smithi023 bash[21372]: audit 2024-03-11T20:55:21.295395+0000 mon.c (mon.2) 31 : audit 1 from='client.? 172.21.15.23:0/1276696761' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T20:55:22.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:22 smithi023 bash[21372]: audit 2024-03-11T20:55:21.295999+0000 mon.a (mon.0) 975 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T20:55:22.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:22 smithi023 bash[21372]: cluster 2024-03-11T20:55:21.665716+0000 mon.a (mon.0) 976 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T20:55:22.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:22 smithi023 bash[21372]: audit 2024-03-11T20:55:21.670854+0000 mon.a (mon.0) 977 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T20:55:22.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:22 smithi023 bash[21372]: cluster 2024-03-11T20:55:21.678938+0000 mon.a (mon.0) 978 : cluster 0 osdmap e136: 8 total, 8 up, 7 in 2024-03-11T20:55:22.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:22 smithi023 bash[29146]: cluster 2024-03-11T20:55:20.780850+0000 osd.0 (osd.0) 109 : cluster 0 purged_snaps scrub starts 2024-03-11T20:55:22.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:22 smithi023 bash[29146]: cluster 2024-03-11T20:55:20.780921+0000 osd.0 (osd.0) 110 : cluster 0 purged_snaps scrub ok 2024-03-11T20:55:22.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:22 smithi023 bash[29146]: audit 2024-03-11T20:55:21.295395+0000 mon.c (mon.2) 31 : audit 1 from='client.? 172.21.15.23:0/1276696761' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T20:55:22.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:22 smithi023 bash[29146]: audit 2024-03-11T20:55:21.295999+0000 mon.a (mon.0) 975 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T20:55:22.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:22 smithi023 bash[29146]: cluster 2024-03-11T20:55:21.665716+0000 mon.a (mon.0) 976 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T20:55:22.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:22 smithi023 bash[29146]: audit 2024-03-11T20:55:21.670854+0000 mon.a (mon.0) 977 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T20:55:22.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:22 smithi023 bash[29146]: cluster 2024-03-11T20:55:21.678938+0000 mon.a (mon.0) 978 : cluster 0 osdmap e136: 8 total, 8 up, 7 in 2024-03-11T20:55:22.340 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:22.369 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:22.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:22.372 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:55:22.472 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:22.498 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:22.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:22.502 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:55:22.541 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 31 16 54315 54299 109.45 84.1875 0.00680625 0.00912973 2024-03-11T20:55:22.602 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:55:22.629 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:55:22.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:22.630 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:55:22.731 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:55:22.754 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:55:22.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:22.756 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:55:22.857 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:55:22.892 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:55:22.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:22.897 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:55:22.998 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:23.028 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:23.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:23.031 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:55:23.132 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:55:23.150 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:55:23.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:23.152 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:55:23.253 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:55:23.263 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:23 smithi023 bash[21372]: cluster 2024-03-11T20:55:21.909879+0000 mgr.y (mgr.24380) 175 : cluster 0 pgmap v152: 148 pgs: 148 active+clean; 3.1 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 1023 B/s rd, 82 MiB/s wr, 1.31k op/s 2024-03-11T20:55:23.263 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:23 smithi023 bash[29146]: cluster 2024-03-11T20:55:21.909879+0000 mgr.y (mgr.24380) 175 : cluster 0 pgmap v152: 148 pgs: 148 active+clean; 3.1 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 1023 B/s rd, 82 MiB/s wr, 1.31k op/s 2024-03-11T20:55:23.281 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:55:23.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:23.285 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:55:23.386 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:55:23.420 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:55:23.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:23.423 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:55:23.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:23 smithi146 bash[23809]: cluster 2024-03-11T20:55:21.909879+0000 mgr.y (mgr.24380) 175 : cluster 0 pgmap v152: 148 pgs: 148 active+clean; 3.1 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 1023 B/s rd, 82 MiB/s wr, 1.31k op/s 2024-03-11T20:55:23.524 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:23.541 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 32 16 55543 55527 108.428 76.75 0.0119327 0.00921663 2024-03-11T20:55:23.557 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:23.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:23.559 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:55:23.659 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:55:23.694 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:55:23.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:23.696 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:55:23.797 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:23.821 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:23.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:23.827 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:55:23.928 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:23.959 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:23.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:23.962 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:55:24.063 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:24.092 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:24.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:24.095 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:55:24.196 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:24.228 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:24.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:24.230 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:55:24.330 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:24.362 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:24.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:24.366 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:55:24.467 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:55:24.491 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:55:24.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:24.495 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:55:24.541 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 33 16 56774 56758 107.473 76.9375 0.0175401 0.00929873 2024-03-11T20:55:24.596 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:24.623 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:24.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:24.626 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:55:24.727 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:24.758 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:24.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:24.760 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:55:24.861 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:55:24.890 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:55:24.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:24.896 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:55:24.997 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:25.028 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:25.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:25.031 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:55:25.131 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:55:25.165 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:55:25.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:25.169 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:55:25.269 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:25.279 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 20:55:25 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T20:55:25.283 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:25 smithi146 bash[23809]: cluster 2024-03-11T20:55:23.910276+0000 mgr.y (mgr.24380) 176 : cluster 0 pgmap v153: 148 pgs: 148 active+clean; 3.1 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 1023 B/s rd, 82 MiB/s wr, 1.31k op/s 2024-03-11T20:55:25.283 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:25 smithi146 bash[23809]: cluster 2024-03-11T20:55:24.447064+0000 osd.5 (osd.5) 87 : cluster 0 6.d scrub starts 2024-03-11T20:55:25.283 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:25 smithi146 bash[23809]: cluster 2024-03-11T20:55:24.448627+0000 osd.5 (osd.5) 88 : cluster 0 6.d scrub ok 2024-03-11T20:55:25.283 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:25 smithi146 bash[23809]: cluster 2024-03-11T20:55:24.651212+0000 osd.6 (osd.6) 63 : cluster 0 3.a scrub starts 2024-03-11T20:55:25.284 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:25 smithi146 bash[23809]: cluster 2024-03-11T20:55:24.654345+0000 osd.6 (osd.6) 64 : cluster 0 3.a scrub ok 2024-03-11T20:55:25.284 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:25 smithi146 bash[23809]: cluster 2024-03-11T20:55:24.672775+0000 mon.a (mon.0) 979 : cluster 0 osdmap e137: 8 total, 8 up, 7 in 2024-03-11T20:55:25.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:25 smithi023 bash[21372]: cluster 2024-03-11T20:55:23.910276+0000 mgr.y (mgr.24380) 176 : cluster 0 pgmap v153: 148 pgs: 148 active+clean; 3.1 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 1023 B/s rd, 82 MiB/s wr, 1.31k op/s 2024-03-11T20:55:25.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:25 smithi023 bash[21372]: cluster 2024-03-11T20:55:24.447064+0000 osd.5 (osd.5) 87 : cluster 0 6.d scrub starts 2024-03-11T20:55:25.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:25 smithi023 bash[21372]: cluster 2024-03-11T20:55:24.448627+0000 osd.5 (osd.5) 88 : cluster 0 6.d scrub ok 2024-03-11T20:55:25.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:25 smithi023 bash[21372]: cluster 2024-03-11T20:55:24.651212+0000 osd.6 (osd.6) 63 : cluster 0 3.a scrub starts 2024-03-11T20:55:25.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:25 smithi023 bash[21372]: cluster 2024-03-11T20:55:24.654345+0000 osd.6 (osd.6) 64 : cluster 0 3.a scrub ok 2024-03-11T20:55:25.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:25 smithi023 bash[21372]: cluster 2024-03-11T20:55:24.672775+0000 mon.a (mon.0) 979 : cluster 0 osdmap e137: 8 total, 8 up, 7 in 2024-03-11T20:55:25.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:25 smithi023 bash[29146]: cluster 2024-03-11T20:55:23.910276+0000 mgr.y (mgr.24380) 176 : cluster 0 pgmap v153: 148 pgs: 148 active+clean; 3.1 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 1023 B/s rd, 82 MiB/s wr, 1.31k op/s 2024-03-11T20:55:25.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:25 smithi023 bash[29146]: cluster 2024-03-11T20:55:24.447064+0000 osd.5 (osd.5) 87 : cluster 0 6.d scrub starts 2024-03-11T20:55:25.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:25 smithi023 bash[29146]: cluster 2024-03-11T20:55:24.448627+0000 osd.5 (osd.5) 88 : cluster 0 6.d scrub ok 2024-03-11T20:55:25.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:25 smithi023 bash[29146]: cluster 2024-03-11T20:55:24.651212+0000 osd.6 (osd.6) 63 : cluster 0 3.a scrub starts 2024-03-11T20:55:25.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:25 smithi023 bash[29146]: cluster 2024-03-11T20:55:24.654345+0000 osd.6 (osd.6) 64 : cluster 0 3.a scrub ok 2024-03-11T20:55:25.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:25 smithi023 bash[29146]: cluster 2024-03-11T20:55:24.672775+0000 mon.a (mon.0) 979 : cluster 0 osdmap e137: 8 total, 8 up, 7 in 2024-03-11T20:55:25.297 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:25.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:25.300 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:55:25.400 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:25.449 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:25.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:25.453 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:55:25.541 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 34 16 58135 58119 106.814 85.0625 0.00553029 0.00935617 2024-03-11T20:55:25.555 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:25.589 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:25.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:25.592 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:55:25.693 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:25.708 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T20:55:25.723 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:25.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:25.726 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:55:25.827 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:55:25.849 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:55:25.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:25.852 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:55:25.953 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:25.979 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:25.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:25.984 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:55:26.085 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:26.112 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:26.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:26.117 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:55:26.177 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-11T20:55:26.217 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:26.234 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:26 smithi023 bash[21372]: cluster 2024-03-11T20:55:24.616509+0000 osd.2 (osd.2) 71 : cluster 0 3.0 scrub starts 2024-03-11T20:55:26.234 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:26 smithi023 bash[21372]: cluster 2024-03-11T20:55:24.623285+0000 osd.2 (osd.2) 72 : cluster 0 3.0 scrub ok 2024-03-11T20:55:26.234 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:26 smithi023 bash[21372]: cluster 2024-03-11T20:55:24.713409+0000 osd.3 (osd.3) 131 : cluster 0 4.1d scrub starts 2024-03-11T20:55:26.234 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:26 smithi023 bash[21372]: cluster 2024-03-11T20:55:24.714844+0000 osd.3 (osd.3) 132 : cluster 0 4.1d scrub ok 2024-03-11T20:55:26.234 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:26 smithi023 bash[21372]: cluster 2024-03-11T20:55:24.933860+0000 osd.0 (osd.0) 111 : cluster 0 3.12 scrub starts 2024-03-11T20:55:26.234 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:26 smithi023 bash[21372]: cluster 2024-03-11T20:55:24.941423+0000 osd.0 (osd.0) 112 : cluster 0 3.12 scrub ok 2024-03-11T20:55:26.234 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:26 smithi023 bash[21372]: audit 2024-03-11T20:55:25.025132+0000 mgr.y (mgr.24380) 177 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:55:26.234 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:26 smithi023 bash[21372]: cluster 2024-03-11T20:55:25.486984+0000 osd.5 (osd.5) 89 : cluster 0 2.15 scrub starts 2024-03-11T20:55:26.234 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:26 smithi023 bash[21372]: cluster 2024-03-11T20:55:25.491872+0000 osd.5 (osd.5) 90 : cluster 0 2.15 scrub ok 2024-03-11T20:55:26.235 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:26 smithi023 bash[21372]: cluster 2024-03-11T20:55:25.520483+0000 osd.4 (osd.4) 105 : cluster 0 6.1a scrub starts 2024-03-11T20:55:26.235 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:26 smithi023 bash[21372]: cluster 2024-03-11T20:55:25.521807+0000 osd.4 (osd.4) 106 : cluster 0 6.1a scrub ok 2024-03-11T20:55:26.238 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:26 smithi023 bash[29146]: cluster 2024-03-11T20:55:24.616509+0000 osd.2 (osd.2) 71 : cluster 0 3.0 scrub starts 2024-03-11T20:55:26.238 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:26 smithi023 bash[29146]: cluster 2024-03-11T20:55:24.623285+0000 osd.2 (osd.2) 72 : cluster 0 3.0 scrub ok 2024-03-11T20:55:26.238 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:26 smithi023 bash[29146]: cluster 2024-03-11T20:55:24.713409+0000 osd.3 (osd.3) 131 : cluster 0 4.1d scrub starts 2024-03-11T20:55:26.238 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:26 smithi023 bash[29146]: cluster 2024-03-11T20:55:24.714844+0000 osd.3 (osd.3) 132 : cluster 0 4.1d scrub ok 2024-03-11T20:55:26.238 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:26 smithi023 bash[29146]: cluster 2024-03-11T20:55:24.933860+0000 osd.0 (osd.0) 111 : cluster 0 3.12 scrub starts 2024-03-11T20:55:26.238 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:26 smithi023 bash[29146]: cluster 2024-03-11T20:55:24.941423+0000 osd.0 (osd.0) 112 : cluster 0 3.12 scrub ok 2024-03-11T20:55:26.238 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:26 smithi023 bash[29146]: audit 2024-03-11T20:55:25.025132+0000 mgr.y (mgr.24380) 177 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:55:26.239 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:26 smithi023 bash[29146]: cluster 2024-03-11T20:55:25.486984+0000 osd.5 (osd.5) 89 : cluster 0 2.15 scrub starts 2024-03-11T20:55:26.239 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:26 smithi023 bash[29146]: cluster 2024-03-11T20:55:25.491872+0000 osd.5 (osd.5) 90 : cluster 0 2.15 scrub ok 2024-03-11T20:55:26.239 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:26 smithi023 bash[29146]: cluster 2024-03-11T20:55:25.520483+0000 osd.4 (osd.4) 105 : cluster 0 6.1a scrub starts 2024-03-11T20:55:26.239 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:26 smithi023 bash[29146]: cluster 2024-03-11T20:55:25.521807+0000 osd.4 (osd.4) 106 : cluster 0 6.1a scrub ok 2024-03-11T20:55:26.256 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:26.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:26.261 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:55:26.362 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:55:26.387 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:55:26.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:26.395 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:55:26.496 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:55:26.506 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:26 smithi146 bash[23809]: cluster 2024-03-11T20:55:24.616509+0000 osd.2 (osd.2) 71 : cluster 0 3.0 scrub starts 2024-03-11T20:55:26.506 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:26 smithi146 bash[23809]: cluster 2024-03-11T20:55:24.623285+0000 osd.2 (osd.2) 72 : cluster 0 3.0 scrub ok 2024-03-11T20:55:26.506 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:26 smithi146 bash[23809]: cluster 2024-03-11T20:55:24.713409+0000 osd.3 (osd.3) 131 : cluster 0 4.1d scrub starts 2024-03-11T20:55:26.506 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:26 smithi146 bash[23809]: cluster 2024-03-11T20:55:24.714844+0000 osd.3 (osd.3) 132 : cluster 0 4.1d scrub ok 2024-03-11T20:55:26.506 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:26 smithi146 bash[23809]: cluster 2024-03-11T20:55:24.933860+0000 osd.0 (osd.0) 111 : cluster 0 3.12 scrub starts 2024-03-11T20:55:26.506 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:26 smithi146 bash[23809]: cluster 2024-03-11T20:55:24.941423+0000 osd.0 (osd.0) 112 : cluster 0 3.12 scrub ok 2024-03-11T20:55:26.506 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:26 smithi146 bash[23809]: audit 2024-03-11T20:55:25.025132+0000 mgr.y (mgr.24380) 177 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:55:26.506 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:26 smithi146 bash[23809]: cluster 2024-03-11T20:55:25.486984+0000 osd.5 (osd.5) 89 : cluster 0 2.15 scrub starts 2024-03-11T20:55:26.506 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:26 smithi146 bash[23809]: cluster 2024-03-11T20:55:25.491872+0000 osd.5 (osd.5) 90 : cluster 0 2.15 scrub ok 2024-03-11T20:55:26.506 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:26 smithi146 bash[23809]: cluster 2024-03-11T20:55:25.520483+0000 osd.4 (osd.4) 105 : cluster 0 6.1a scrub starts 2024-03-11T20:55:26.506 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:26 smithi146 bash[23809]: cluster 2024-03-11T20:55:25.521807+0000 osd.4 (osd.4) 106 : cluster 0 6.1a scrub ok 2024-03-11T20:55:26.520 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:55:26.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:26.526 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:55:26.543 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 35 16 59398 59382 106.013 78.9375 0.0114151 0.00942775 2024-03-11T20:55:26.627 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:26.654 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:26.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:26.659 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:55:26.702 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:55:26.705 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:55:26.760 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:55:26.784 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:55:26.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:26.786 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:55:26.887 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:55:26.911 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:55:26.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:26.914 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:55:27.014 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:27.037 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:27.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:27.040 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:55:27.141 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:27.160 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:27.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:27.162 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:55:27.263 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:27.291 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:27.295 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:27 smithi023 bash[21372]: cluster 2024-03-11T20:55:25.691632+0000 osd.3 (osd.3) 133 : cluster 0 3.18 scrub starts 2024-03-11T20:55:27.295 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:27 smithi023 bash[21372]: cluster 2024-03-11T20:55:25.698916+0000 osd.3 (osd.3) 134 : cluster 0 3.18 scrub ok 2024-03-11T20:55:27.295 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:27 smithi023 bash[21372]: cluster 2024-03-11T20:55:25.911268+0000 mgr.y (mgr.24380) 178 : cluster 0 pgmap v155: 148 pgs: 148 active+clean; 3.5 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 95 MiB/s wr, 1.53k op/s 2024-03-11T20:55:27.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:27.296 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:55:27.297 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:27 smithi023 bash[29146]: cluster 2024-03-11T20:55:25.691632+0000 osd.3 (osd.3) 133 : cluster 0 3.18 scrub starts 2024-03-11T20:55:27.297 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:27 smithi023 bash[29146]: cluster 2024-03-11T20:55:25.698916+0000 osd.3 (osd.3) 134 : cluster 0 3.18 scrub ok 2024-03-11T20:55:27.297 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:27 smithi023 bash[29146]: cluster 2024-03-11T20:55:25.911268+0000 mgr.y (mgr.24380) 178 : cluster 0 pgmap v155: 148 pgs: 148 active+clean; 3.5 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 95 MiB/s wr, 1.53k op/s 2024-03-11T20:55:27.397 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:55:27.406 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:27 smithi146 bash[23809]: cluster 2024-03-11T20:55:25.691632+0000 osd.3 (osd.3) 133 : cluster 0 3.18 scrub starts 2024-03-11T20:55:27.407 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:27 smithi146 bash[23809]: cluster 2024-03-11T20:55:25.698916+0000 osd.3 (osd.3) 134 : cluster 0 3.18 scrub ok 2024-03-11T20:55:27.407 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:27 smithi146 bash[23809]: cluster 2024-03-11T20:55:25.911268+0000 mgr.y (mgr.24380) 178 : cluster 0 pgmap v155: 148 pgs: 148 active+clean; 3.5 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 95 MiB/s wr, 1.53k op/s 2024-03-11T20:55:27.427 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:55:27.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:27.429 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:55:27.530 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:55:27.544 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 36 16 60634 60618 105.214 77.25 0.016457 0.00949805 2024-03-11T20:55:27.561 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:55:27.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:27.567 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:55:27.668 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:55:27.702 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:55:27.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:27.705 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:55:27.806 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:55:27.833 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:55:27.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:27.837 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:55:27.937 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:55:27.971 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:55:27.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:27.972 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:55:28.073 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:55:28.105 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:55:28.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:28.106 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:55:28.207 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:55:28.238 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:55:28.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:28.240 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:55:28.341 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:28.355 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:28 smithi146 bash[23809]: cluster 2024-03-11T20:55:26.118833+0000 osd.7 (osd.7) 101 : cluster 0 2.5 scrub starts 2024-03-11T20:55:28.355 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:28 smithi146 bash[23809]: cluster 2024-03-11T20:55:26.137956+0000 osd.7 (osd.7) 102 : cluster 0 2.5 scrub ok 2024-03-11T20:55:28.355 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:28 smithi146 bash[23809]: cluster 2024-03-11T20:55:26.923570+0000 osd.0 (osd.0) 113 : cluster 0 3.1b scrub starts 2024-03-11T20:55:28.355 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:28 smithi146 bash[23809]: cluster 2024-03-11T20:55:26.928166+0000 osd.0 (osd.0) 114 : cluster 0 3.1b scrub ok 2024-03-11T20:55:28.355 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:28 smithi146 bash[23809]: cluster 2024-03-11T20:55:27.439082+0000 osd.5 (osd.5) 91 : cluster 0 6.1d scrub starts 2024-03-11T20:55:28.355 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:28 smithi146 bash[23809]: cluster 2024-03-11T20:55:27.441398+0000 osd.5 (osd.5) 92 : cluster 0 6.1d scrub ok 2024-03-11T20:55:28.356 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:28 smithi146 bash[23809]: cluster 2024-03-11T20:55:27.472195+0000 osd.4 (osd.4) 107 : cluster 0 6.2 scrub starts 2024-03-11T20:55:28.356 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:28 smithi146 bash[23809]: cluster 2024-03-11T20:55:27.548162+0000 osd.4 (osd.4) 108 : cluster 0 6.2 scrub ok 2024-03-11T20:55:28.361 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:28.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:28.363 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:55:28.464 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:55:28.499 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:55:28.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:28.501 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:55:28.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:28 smithi023 bash[21372]: cluster 2024-03-11T20:55:26.118833+0000 osd.7 (osd.7) 101 : cluster 0 2.5 scrub starts 2024-03-11T20:55:28.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:28 smithi023 bash[21372]: cluster 2024-03-11T20:55:26.137956+0000 osd.7 (osd.7) 102 : cluster 0 2.5 scrub ok 2024-03-11T20:55:28.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:28 smithi023 bash[21372]: cluster 2024-03-11T20:55:26.923570+0000 osd.0 (osd.0) 113 : cluster 0 3.1b scrub starts 2024-03-11T20:55:28.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:28 smithi023 bash[21372]: cluster 2024-03-11T20:55:26.928166+0000 osd.0 (osd.0) 114 : cluster 0 3.1b scrub ok 2024-03-11T20:55:28.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:28 smithi023 bash[21372]: cluster 2024-03-11T20:55:27.439082+0000 osd.5 (osd.5) 91 : cluster 0 6.1d scrub starts 2024-03-11T20:55:28.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:28 smithi023 bash[21372]: cluster 2024-03-11T20:55:27.441398+0000 osd.5 (osd.5) 92 : cluster 0 6.1d scrub ok 2024-03-11T20:55:28.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:28 smithi023 bash[21372]: cluster 2024-03-11T20:55:27.472195+0000 osd.4 (osd.4) 107 : cluster 0 6.2 scrub starts 2024-03-11T20:55:28.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:28 smithi023 bash[21372]: cluster 2024-03-11T20:55:27.548162+0000 osd.4 (osd.4) 108 : cluster 0 6.2 scrub ok 2024-03-11T20:55:28.536 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:55:28 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:20:55:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T20:55:28.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:28 smithi023 bash[29146]: cluster 2024-03-11T20:55:26.118833+0000 osd.7 (osd.7) 101 : cluster 0 2.5 scrub starts 2024-03-11T20:55:28.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:28 smithi023 bash[29146]: cluster 2024-03-11T20:55:26.137956+0000 osd.7 (osd.7) 102 : cluster 0 2.5 scrub ok 2024-03-11T20:55:28.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:28 smithi023 bash[29146]: cluster 2024-03-11T20:55:26.923570+0000 osd.0 (osd.0) 113 : cluster 0 3.1b scrub starts 2024-03-11T20:55:28.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:28 smithi023 bash[29146]: cluster 2024-03-11T20:55:26.928166+0000 osd.0 (osd.0) 114 : cluster 0 3.1b scrub ok 2024-03-11T20:55:28.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:28 smithi023 bash[29146]: cluster 2024-03-11T20:55:27.439082+0000 osd.5 (osd.5) 91 : cluster 0 6.1d scrub starts 2024-03-11T20:55:28.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:28 smithi023 bash[29146]: cluster 2024-03-11T20:55:27.441398+0000 osd.5 (osd.5) 92 : cluster 0 6.1d scrub ok 2024-03-11T20:55:28.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:28 smithi023 bash[29146]: cluster 2024-03-11T20:55:27.472195+0000 osd.4 (osd.4) 107 : cluster 0 6.2 scrub starts 2024-03-11T20:55:28.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:28 smithi023 bash[29146]: cluster 2024-03-11T20:55:27.548162+0000 osd.4 (osd.4) 108 : cluster 0 6.2 scrub ok 2024-03-11T20:55:28.543 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 37 16 61707 61691 104.183 67.0625 0.00634251 0.00959207 2024-03-11T20:55:28.602 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:28.626 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:28.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:28.628 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:55:28.728 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:28.762 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:28.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:28.764 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:55:28.865 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:28.887 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:28.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:28.891 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:55:28.992 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:55:29.031 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:55:29.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:29.036 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:55:29.137 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:55:29.147 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:29 smithi146 bash[23809]: cluster 2024-03-11T20:55:27.127640+0000 osd.7 (osd.7) 103 : cluster 0 3.f scrub starts 2024-03-11T20:55:29.147 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:29 smithi146 bash[23809]: cluster 2024-03-11T20:55:27.139573+0000 osd.7 (osd.7) 104 : cluster 0 3.f scrub ok 2024-03-11T20:55:29.147 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:29 smithi146 bash[23809]: cluster 2024-03-11T20:55:27.605073+0000 osd.2 (osd.2) 73 : cluster 0 4.4 scrub starts 2024-03-11T20:55:29.147 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:29 smithi146 bash[23809]: cluster 2024-03-11T20:55:27.609624+0000 osd.2 (osd.2) 74 : cluster 0 4.4 scrub ok 2024-03-11T20:55:29.147 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:29 smithi146 bash[23809]: cluster 2024-03-11T20:55:27.914253+0000 osd.0 (osd.0) 115 : cluster 0 6.10 scrub starts 2024-03-11T20:55:29.147 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:29 smithi146 bash[23809]: cluster 2024-03-11T20:55:27.914359+0000 mgr.y (mgr.24380) 179 : cluster 0 pgmap v156: 148 pgs: 148 active+clean; 3.5 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 639 B/s rd, 50 MiB/s wr, 804 op/s 2024-03-11T20:55:29.147 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:29 smithi146 bash[23809]: cluster 2024-03-11T20:55:27.917105+0000 osd.0 (osd.0) 116 : cluster 0 6.10 scrub ok 2024-03-11T20:55:29.164 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:55:29.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:29.166 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:55:29.267 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:55:29.277 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:29 smithi023 bash[21372]: cluster 2024-03-11T20:55:27.127640+0000 osd.7 (osd.7) 103 : cluster 0 3.f scrub starts 2024-03-11T20:55:29.277 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:29 smithi023 bash[21372]: cluster 2024-03-11T20:55:27.139573+0000 osd.7 (osd.7) 104 : cluster 0 3.f scrub ok 2024-03-11T20:55:29.277 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:29 smithi023 bash[21372]: cluster 2024-03-11T20:55:27.605073+0000 osd.2 (osd.2) 73 : cluster 0 4.4 scrub starts 2024-03-11T20:55:29.277 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:29 smithi023 bash[21372]: cluster 2024-03-11T20:55:27.609624+0000 osd.2 (osd.2) 74 : cluster 0 4.4 scrub ok 2024-03-11T20:55:29.277 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:29 smithi023 bash[21372]: cluster 2024-03-11T20:55:27.914253+0000 osd.0 (osd.0) 115 : cluster 0 6.10 scrub starts 2024-03-11T20:55:29.277 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:29 smithi023 bash[21372]: cluster 2024-03-11T20:55:27.914359+0000 mgr.y (mgr.24380) 179 : cluster 0 pgmap v156: 148 pgs: 148 active+clean; 3.5 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 639 B/s rd, 50 MiB/s wr, 804 op/s 2024-03-11T20:55:29.277 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:29 smithi023 bash[21372]: cluster 2024-03-11T20:55:27.917105+0000 osd.0 (osd.0) 116 : cluster 0 6.10 scrub ok 2024-03-11T20:55:29.277 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:29 smithi023 bash[29146]: cluster 2024-03-11T20:55:27.127640+0000 osd.7 (osd.7) 103 : cluster 0 3.f scrub starts 2024-03-11T20:55:29.277 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:29 smithi023 bash[29146]: cluster 2024-03-11T20:55:27.139573+0000 osd.7 (osd.7) 104 : cluster 0 3.f scrub ok 2024-03-11T20:55:29.278 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:29 smithi023 bash[29146]: cluster 2024-03-11T20:55:27.605073+0000 osd.2 (osd.2) 73 : cluster 0 4.4 scrub starts 2024-03-11T20:55:29.278 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:29 smithi023 bash[29146]: cluster 2024-03-11T20:55:27.609624+0000 osd.2 (osd.2) 74 : cluster 0 4.4 scrub ok 2024-03-11T20:55:29.278 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:29 smithi023 bash[29146]: cluster 2024-03-11T20:55:27.914253+0000 osd.0 (osd.0) 115 : cluster 0 6.10 scrub starts 2024-03-11T20:55:29.278 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:29 smithi023 bash[29146]: cluster 2024-03-11T20:55:27.914359+0000 mgr.y (mgr.24380) 179 : cluster 0 pgmap v156: 148 pgs: 148 active+clean; 3.5 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 639 B/s rd, 50 MiB/s wr, 804 op/s 2024-03-11T20:55:29.278 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:29 smithi023 bash[29146]: cluster 2024-03-11T20:55:27.917105+0000 osd.0 (osd.0) 116 : cluster 0 6.10 scrub ok 2024-03-11T20:55:29.297 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:55:29.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:29.301 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:55:29.402 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:29.432 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:29.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:29.435 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:55:29.536 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:29.543 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 38 16 62983 62967 103.54 79.75 0.0266848 0.00965189 2024-03-11T20:55:29.560 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:29.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:29.563 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:55:29.664 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:29.703 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:29.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:29.706 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:55:29.750 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:55:29.807 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:29.857 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:29.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:29.863 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:55:29.964 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:55:29.996 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:55:29.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:29.997 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:55:30.098 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:55:30.134 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:55:30.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:30.140 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:55:30.240 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:30.275 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:30.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:30.279 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:55:30.379 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:30.409 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:30.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:30.410 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:55:30.511 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:55:30.538 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:55:30.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:30.541 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:55:30.543 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 39 16 64220 64204 102.867 77.3125 0.0196673 0.00971501 2024-03-11T20:55:30.643 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:30.676 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:30.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:30.682 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:55:30.783 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:30.784 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:30 smithi023 bash[29146]: cluster 2024-03-11T20:55:29.654225+0000 osd.3 (osd.3) 135 : cluster 0 3.1e scrub starts 2024-03-11T20:55:30.784 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:30 smithi023 bash[29146]: cluster 2024-03-11T20:55:29.661222+0000 osd.3 (osd.3) 136 : cluster 0 3.1e scrub ok 2024-03-11T20:55:30.784 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:30 smithi023 bash[29146]: cluster 2024-03-11T20:55:29.915042+0000 mgr.y (mgr.24380) 180 : cluster 0 pgmap v157: 148 pgs: 148 active+clean; 3.9 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 95 MiB/s wr, 1.53k op/s 2024-03-11T20:55:30.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:30 smithi023 bash[21372]: cluster 2024-03-11T20:55:29.654225+0000 osd.3 (osd.3) 135 : cluster 0 3.1e scrub starts 2024-03-11T20:55:30.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:30 smithi023 bash[21372]: cluster 2024-03-11T20:55:29.661222+0000 osd.3 (osd.3) 136 : cluster 0 3.1e scrub ok 2024-03-11T20:55:30.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:30 smithi023 bash[21372]: cluster 2024-03-11T20:55:29.915042+0000 mgr.y (mgr.24380) 180 : cluster 0 pgmap v157: 148 pgs: 148 active+clean; 3.9 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 95 MiB/s wr, 1.53k op/s 2024-03-11T20:55:30.808 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:30.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:30.815 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:55:30.915 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:30.943 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:30.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:30.947 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:55:31.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:30 smithi146 bash[23809]: cluster 2024-03-11T20:55:29.654225+0000 osd.3 (osd.3) 135 : cluster 0 3.1e scrub starts 2024-03-11T20:55:31.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:30 smithi146 bash[23809]: cluster 2024-03-11T20:55:29.661222+0000 osd.3 (osd.3) 136 : cluster 0 3.1e scrub ok 2024-03-11T20:55:31.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:30 smithi146 bash[23809]: cluster 2024-03-11T20:55:29.915042+0000 mgr.y (mgr.24380) 180 : cluster 0 pgmap v157: 148 pgs: 148 active+clean; 3.9 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 95 MiB/s wr, 1.53k op/s 2024-03-11T20:55:31.047 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:31.066 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:31.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:31.068 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:55:31.168 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:55:31.202 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:55:31.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:31.204 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:55:31.304 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:55:31.333 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:55:31.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:31.336 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:55:31.437 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:55:31.462 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:55:31.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:31.464 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:55:31.543 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:2024-03-11T20:55:31.547921+0000 min lat: 0.0012354 max lat: 0.333823 avg lat: 0.00979043 2024-03-11T20:55:31.543 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-11T20:55:31.543 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 40 16 65360 65344 102.077 71.25 0.0103697 0.00979043 2024-03-11T20:55:31.565 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:31.590 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:31.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:31.597 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:55:31.698 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:31.724 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:31.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:31.726 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:55:31.827 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:55:31.851 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:55:31.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:31.854 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:55:31.955 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:55:31.963 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:31 smithi146 bash[23809]: cluster 2024-03-11T20:55:30.465291+0000 osd.4 (osd.4) 109 : cluster 0 4.13 scrub starts 2024-03-11T20:55:31.963 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:31 smithi146 bash[23809]: cluster 2024-03-11T20:55:30.470900+0000 osd.4 (osd.4) 110 : cluster 0 4.13 scrub ok 2024-03-11T20:55:31.978 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:55:31.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:31.981 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:55:32.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:31 smithi023 bash[21372]: cluster 2024-03-11T20:55:30.465291+0000 osd.4 (osd.4) 109 : cluster 0 4.13 scrub starts 2024-03-11T20:55:32.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:31 smithi023 bash[21372]: cluster 2024-03-11T20:55:30.470900+0000 osd.4 (osd.4) 110 : cluster 0 4.13 scrub ok 2024-03-11T20:55:32.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:31 smithi023 bash[29146]: cluster 2024-03-11T20:55:30.465291+0000 osd.4 (osd.4) 109 : cluster 0 4.13 scrub starts 2024-03-11T20:55:32.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:31 smithi023 bash[29146]: cluster 2024-03-11T20:55:30.470900+0000 osd.4 (osd.4) 110 : cluster 0 4.13 scrub ok 2024-03-11T20:55:32.082 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:32.112 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:32.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:32.114 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:55:32.214 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:32.240 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:32.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:32.243 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:55:32.343 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:55:32.367 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:55:32.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:32.369 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:55:32.470 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:55:32.495 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:55:32.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:32.497 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:55:32.543 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 41 16 66627 66611 101.519 79.1875 0.00641815 0.00984397 2024-03-11T20:55:32.598 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:32.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:32.631 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:32.631 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:55:32.732 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:55:32.757 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:55:32.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:32.760 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:55:32.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:32 smithi146 bash[23809]: cluster 2024-03-11T20:55:31.541543+0000 osd.2 (osd.2) 75 : cluster 0 2.0 scrub starts 2024-03-11T20:55:32.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:32 smithi146 bash[23809]: cluster 2024-03-11T20:55:31.558997+0000 osd.2 (osd.2) 76 : cluster 0 2.0 scrub ok 2024-03-11T20:55:32.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:32 smithi146 bash[23809]: cluster 2024-03-11T20:55:31.915555+0000 mgr.y (mgr.24380) 181 : cluster 0 pgmap v158: 148 pgs: 148 active+clean; 3.9 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1023 B/s rd, 79 MiB/s wr, 1.26k op/s 2024-03-11T20:55:32.861 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:55:32.883 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:55:32.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:32.885 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:55:32.986 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:32.995 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:32 smithi023 bash[21372]: cluster 2024-03-11T20:55:31.541543+0000 osd.2 (osd.2) 75 : cluster 0 2.0 scrub starts 2024-03-11T20:55:32.995 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:32 smithi023 bash[21372]: cluster 2024-03-11T20:55:31.558997+0000 osd.2 (osd.2) 76 : cluster 0 2.0 scrub ok 2024-03-11T20:55:32.995 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:32 smithi023 bash[21372]: cluster 2024-03-11T20:55:31.915555+0000 mgr.y (mgr.24380) 181 : cluster 0 pgmap v158: 148 pgs: 148 active+clean; 3.9 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1023 B/s rd, 79 MiB/s wr, 1.26k op/s 2024-03-11T20:55:32.996 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:32 smithi023 bash[29146]: cluster 2024-03-11T20:55:31.541543+0000 osd.2 (osd.2) 75 : cluster 0 2.0 scrub starts 2024-03-11T20:55:32.996 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:32 smithi023 bash[29146]: cluster 2024-03-11T20:55:31.558997+0000 osd.2 (osd.2) 76 : cluster 0 2.0 scrub ok 2024-03-11T20:55:32.996 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:32 smithi023 bash[29146]: cluster 2024-03-11T20:55:31.915555+0000 mgr.y (mgr.24380) 181 : cluster 0 pgmap v158: 148 pgs: 148 active+clean; 3.9 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1023 B/s rd, 79 MiB/s wr, 1.26k op/s 2024-03-11T20:55:33.017 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:33.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:33.018 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:55:33.119 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:33.144 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:33.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:33.147 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:55:33.248 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:55:33.281 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:55:33.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:33.285 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:55:33.385 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:55:33.403 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:55:33.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:33.405 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:55:33.506 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:33.653 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:55:33.654 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 42 16 67739 67723 100.756 69.5 0.0115346 0.00991964 2024-03-11T20:55:33.678 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:33.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:33.683 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:55:33.784 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:33.808 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:33.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:33.810 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:55:33.911 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:33.940 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:33 smithi023 bash[29146]: cluster 2024-03-11T20:55:32.582130+0000 osd.3 (osd.3) 137 : cluster 0 4.10 scrub starts 2024-03-11T20:55:33.940 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:33 smithi023 bash[29146]: cluster 2024-03-11T20:55:32.587708+0000 osd.3 (osd.3) 138 : cluster 0 4.10 scrub ok 2024-03-11T20:55:33.940 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:33 smithi023 bash[29146]: cluster 2024-03-11T20:55:33.083625+0000 osd.7 (osd.7) 105 : cluster 0 2.1d scrub starts 2024-03-11T20:55:33.940 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:33 smithi023 bash[29146]: cluster 2024-03-11T20:55:33.455614+0000 osd.5 (osd.5) 93 : cluster 0 3.1d scrub starts 2024-03-11T20:55:33.940 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:33 smithi023 bash[29146]: cluster 2024-03-11T20:55:33.457794+0000 osd.5 (osd.5) 94 : cluster 0 3.1d scrub ok 2024-03-11T20:55:33.941 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:33 smithi023 bash[29146]: cluster 2024-03-11T20:55:33.563030+0000 osd.6 (osd.6) 65 : cluster 0 2.6 scrub starts 2024-03-11T20:55:33.941 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:33 smithi023 bash[29146]: cluster 2024-03-11T20:55:33.564588+0000 osd.6 (osd.6) 66 : cluster 0 2.6 scrub ok 2024-03-11T20:55:33.941 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:33 smithi023 bash[21372]: cluster 2024-03-11T20:55:32.582130+0000 osd.3 (osd.3) 137 : cluster 0 4.10 scrub starts 2024-03-11T20:55:33.942 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:33 smithi023 bash[21372]: cluster 2024-03-11T20:55:32.587708+0000 osd.3 (osd.3) 138 : cluster 0 4.10 scrub ok 2024-03-11T20:55:33.942 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:33 smithi023 bash[21372]: cluster 2024-03-11T20:55:33.083625+0000 osd.7 (osd.7) 105 : cluster 0 2.1d scrub starts 2024-03-11T20:55:33.942 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:33 smithi023 bash[21372]: cluster 2024-03-11T20:55:33.455614+0000 osd.5 (osd.5) 93 : cluster 0 3.1d scrub starts 2024-03-11T20:55:33.942 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:33 smithi023 bash[21372]: cluster 2024-03-11T20:55:33.457794+0000 osd.5 (osd.5) 94 : cluster 0 3.1d scrub ok 2024-03-11T20:55:33.942 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:33 smithi023 bash[21372]: cluster 2024-03-11T20:55:33.563030+0000 osd.6 (osd.6) 65 : cluster 0 2.6 scrub starts 2024-03-11T20:55:33.942 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:33 smithi023 bash[21372]: cluster 2024-03-11T20:55:33.564588+0000 osd.6 (osd.6) 66 : cluster 0 2.6 scrub ok 2024-03-11T20:55:33.942 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:33.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:33.944 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:55:34.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:33 smithi146 bash[23809]: cluster 2024-03-11T20:55:32.582130+0000 osd.3 (osd.3) 137 : cluster 0 4.10 scrub starts 2024-03-11T20:55:34.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:33 smithi146 bash[23809]: cluster 2024-03-11T20:55:32.587708+0000 osd.3 (osd.3) 138 : cluster 0 4.10 scrub ok 2024-03-11T20:55:34.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:33 smithi146 bash[23809]: cluster 2024-03-11T20:55:33.083625+0000 osd.7 (osd.7) 105 : cluster 0 2.1d scrub starts 2024-03-11T20:55:34.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:33 smithi146 bash[23809]: cluster 2024-03-11T20:55:33.455614+0000 osd.5 (osd.5) 93 : cluster 0 3.1d scrub starts 2024-03-11T20:55:34.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:33 smithi146 bash[23809]: cluster 2024-03-11T20:55:33.457794+0000 osd.5 (osd.5) 94 : cluster 0 3.1d scrub ok 2024-03-11T20:55:34.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:33 smithi146 bash[23809]: cluster 2024-03-11T20:55:33.563030+0000 osd.6 (osd.6) 65 : cluster 0 2.6 scrub starts 2024-03-11T20:55:34.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:33 smithi146 bash[23809]: cluster 2024-03-11T20:55:33.564588+0000 osd.6 (osd.6) 66 : cluster 0 2.6 scrub ok 2024-03-11T20:55:34.045 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:55:34.079 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:55:34.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:34.083 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:55:34.184 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:34.212 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:34.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:34.216 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:55:34.317 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:34.334 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:34.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:34.336 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:55:34.437 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:34.463 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:34.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:34.468 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:55:34.543 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 43 15 68680 68665 99.7822 58.875 0.0108054 0.0100156 2024-03-11T20:55:34.569 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:34.597 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:34.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:34.601 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:55:34.702 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:34.722 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:34.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:34.723 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:55:34.824 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:34.849 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:34.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:34.851 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:55:34.952 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:34.962 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:34 smithi023 bash[21372]: cluster 2024-03-11T20:55:33.088193+0000 osd.7 (osd.7) 106 : cluster 0 2.1d scrub ok 2024-03-11T20:55:34.962 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:34 smithi023 bash[21372]: cluster 2024-03-11T20:55:33.579101+0000 osd.2 (osd.2) 77 : cluster 0 6.f scrub starts 2024-03-11T20:55:34.962 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:34 smithi023 bash[21372]: cluster 2024-03-11T20:55:33.589355+0000 osd.2 (osd.2) 78 : cluster 0 6.f scrub ok 2024-03-11T20:55:34.962 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:34 smithi023 bash[21372]: cluster 2024-03-11T20:55:33.841606+0000 osd.0 (osd.0) 117 : cluster 0 4.1e scrub starts 2024-03-11T20:55:34.962 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:34 smithi023 bash[21372]: cluster 2024-03-11T20:55:33.845994+0000 osd.0 (osd.0) 118 : cluster 0 4.1e scrub ok 2024-03-11T20:55:34.962 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:34 smithi023 bash[21372]: cluster 2024-03-11T20:55:33.916079+0000 mgr.y (mgr.24380) 182 : cluster 0 pgmap v159: 148 pgs: 148 active+clean; 3.9 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1023 B/s rd, 79 MiB/s wr, 1.26k op/s 2024-03-11T20:55:34.962 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:34 smithi023 bash[21372]: audit 2024-03-11T20:55:33.935034+0000 mon.a (mon.0) 980 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [3, 2]}]: dispatch 2024-03-11T20:55:34.962 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:34 smithi023 bash[21372]: audit 2024-03-11T20:55:33.935351+0000 mon.a (mon.0) 981 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [3, 5]}]: dispatch 2024-03-11T20:55:34.963 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:34 smithi023 bash[21372]: audit 2024-03-11T20:55:34.367167+0000 mon.a (mon.0) 982 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:55:34.963 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:34 smithi023 bash[29146]: cluster 2024-03-11T20:55:33.088193+0000 osd.7 (osd.7) 106 : cluster 0 2.1d scrub ok 2024-03-11T20:55:34.963 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:34 smithi023 bash[29146]: cluster 2024-03-11T20:55:33.579101+0000 osd.2 (osd.2) 77 : cluster 0 6.f scrub starts 2024-03-11T20:55:34.963 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:34 smithi023 bash[29146]: cluster 2024-03-11T20:55:33.589355+0000 osd.2 (osd.2) 78 : cluster 0 6.f scrub ok 2024-03-11T20:55:34.963 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:34 smithi023 bash[29146]: cluster 2024-03-11T20:55:33.841606+0000 osd.0 (osd.0) 117 : cluster 0 4.1e scrub starts 2024-03-11T20:55:34.963 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:34 smithi023 bash[29146]: cluster 2024-03-11T20:55:33.845994+0000 osd.0 (osd.0) 118 : cluster 0 4.1e scrub ok 2024-03-11T20:55:34.963 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:34 smithi023 bash[29146]: cluster 2024-03-11T20:55:33.916079+0000 mgr.y (mgr.24380) 182 : cluster 0 pgmap v159: 148 pgs: 148 active+clean; 3.9 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1023 B/s rd, 79 MiB/s wr, 1.26k op/s 2024-03-11T20:55:34.963 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:34 smithi023 bash[29146]: audit 2024-03-11T20:55:33.935034+0000 mon.a (mon.0) 980 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [3, 2]}]: dispatch 2024-03-11T20:55:34.963 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:34 smithi023 bash[29146]: audit 2024-03-11T20:55:33.935351+0000 mon.a (mon.0) 981 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [3, 5]}]: dispatch 2024-03-11T20:55:34.963 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:34 smithi023 bash[29146]: audit 2024-03-11T20:55:34.367167+0000 mon.a (mon.0) 982 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:55:34.983 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:34.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:34.985 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:55:35.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:34 smithi146 bash[23809]: cluster 2024-03-11T20:55:33.088193+0000 osd.7 (osd.7) 106 : cluster 0 2.1d scrub ok 2024-03-11T20:55:35.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:34 smithi146 bash[23809]: cluster 2024-03-11T20:55:33.579101+0000 osd.2 (osd.2) 77 : cluster 0 6.f scrub starts 2024-03-11T20:55:35.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:34 smithi146 bash[23809]: cluster 2024-03-11T20:55:33.589355+0000 osd.2 (osd.2) 78 : cluster 0 6.f scrub ok 2024-03-11T20:55:35.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:34 smithi146 bash[23809]: cluster 2024-03-11T20:55:33.841606+0000 osd.0 (osd.0) 117 : cluster 0 4.1e scrub starts 2024-03-11T20:55:35.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:34 smithi146 bash[23809]: cluster 2024-03-11T20:55:33.845994+0000 osd.0 (osd.0) 118 : cluster 0 4.1e scrub ok 2024-03-11T20:55:35.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:34 smithi146 bash[23809]: cluster 2024-03-11T20:55:33.916079+0000 mgr.y (mgr.24380) 182 : cluster 0 pgmap v159: 148 pgs: 148 active+clean; 3.9 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1023 B/s rd, 79 MiB/s wr, 1.26k op/s 2024-03-11T20:55:35.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:34 smithi146 bash[23809]: audit 2024-03-11T20:55:33.935034+0000 mon.a (mon.0) 980 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [3, 2]}]: dispatch 2024-03-11T20:55:35.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:34 smithi146 bash[23809]: audit 2024-03-11T20:55:33.935351+0000 mon.a (mon.0) 981 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [3, 5]}]: dispatch 2024-03-11T20:55:35.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:34 smithi146 bash[23809]: audit 2024-03-11T20:55:34.367167+0000 mon.a (mon.0) 982 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:55:35.086 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:35.095 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 20:55:35 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T20:55:35.119 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:35.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:35.121 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:55:35.222 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:35.240 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:35.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:35.241 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:55:35.342 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:55:35.374 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:55:35.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:35.377 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:55:35.478 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:55:35.497 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:55:35.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:35.499 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:55:35.543 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 44 16 69456 69440 98.6154 48.4375 0.00278257 0.0101129 2024-03-11T20:55:35.600 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:55:35.630 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:55:35.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:35.632 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:55:35.733 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:55:35.762 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:55:35.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:35.765 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:55:35.866 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:35.905 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:35.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:35.911 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:55:36.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:35 smithi146 bash[23809]: audit 2024-03-11T20:55:34.711180+0000 mon.a (mon.0) 983 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [3, 2]}]': finished 2024-03-11T20:55:36.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:35 smithi146 bash[23809]: audit 2024-03-11T20:55:34.711275+0000 mon.a (mon.0) 984 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [3, 5]}]': finished 2024-03-11T20:55:36.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:35 smithi146 bash[23809]: cluster 2024-03-11T20:55:34.715088+0000 mon.a (mon.0) 985 : cluster 0 osdmap e138: 8 total, 8 up, 7 in 2024-03-11T20:55:36.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:35 smithi146 bash[23809]: audit 2024-03-11T20:55:35.033145+0000 mgr.y (mgr.24380) 183 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:55:36.013 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:36.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:35 smithi023 bash[21372]: audit 2024-03-11T20:55:34.711180+0000 mon.a (mon.0) 983 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [3, 2]}]': finished 2024-03-11T20:55:36.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:35 smithi023 bash[21372]: audit 2024-03-11T20:55:34.711275+0000 mon.a (mon.0) 984 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [3, 5]}]': finished 2024-03-11T20:55:36.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:35 smithi023 bash[21372]: cluster 2024-03-11T20:55:34.715088+0000 mon.a (mon.0) 985 : cluster 0 osdmap e138: 8 total, 8 up, 7 in 2024-03-11T20:55:36.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:35 smithi023 bash[21372]: audit 2024-03-11T20:55:35.033145+0000 mgr.y (mgr.24380) 183 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:55:36.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:35 smithi023 bash[29146]: audit 2024-03-11T20:55:34.711180+0000 mon.a (mon.0) 983 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [3, 2]}]': finished 2024-03-11T20:55:36.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:35 smithi023 bash[29146]: audit 2024-03-11T20:55:34.711275+0000 mon.a (mon.0) 984 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [3, 5]}]': finished 2024-03-11T20:55:36.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:35 smithi023 bash[29146]: cluster 2024-03-11T20:55:34.715088+0000 mon.a (mon.0) 985 : cluster 0 osdmap e138: 8 total, 8 up, 7 in 2024-03-11T20:55:36.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:35 smithi023 bash[29146]: audit 2024-03-11T20:55:35.033145+0000 mgr.y (mgr.24380) 183 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:55:36.046 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:36.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:36.048 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:55:36.148 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:36.174 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:36.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:36.177 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:55:36.278 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:36.304 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:36.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:36.308 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:55:36.409 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:55:36.435 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:55:36.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:36.441 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:55:36.542 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:55:36.543 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 45 16 70560 70544 97.9572 69 0.048994 0.0102016 2024-03-11T20:55:36.570 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:55:36.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:36.572 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:55:36.672 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:36.704 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:36.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:36.707 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:55:36.808 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:36.826 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:36.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:36.827 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:55:36.928 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:36.938 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:36 smithi146 bash[23809]: cluster 2024-03-11T20:55:35.457404+0000 osd.4 (osd.4) 111 : cluster 0 3.1a scrub starts 2024-03-11T20:55:36.938 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:36 smithi146 bash[23809]: cluster 2024-03-11T20:55:35.459595+0000 osd.4 (osd.4) 112 : cluster 0 3.1a scrub ok 2024-03-11T20:55:36.938 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:36 smithi146 bash[23809]: cluster 2024-03-11T20:55:35.729142+0000 mon.a (mon.0) 986 : cluster 0 osdmap e139: 8 total, 8 up, 7 in 2024-03-11T20:55:36.938 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:36 smithi146 bash[23809]: cluster 2024-03-11T20:55:35.822721+0000 osd.7 (osd.7) 107 : cluster 0 purged_snaps scrub starts 2024-03-11T20:55:36.938 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:36 smithi146 bash[23809]: cluster 2024-03-11T20:55:35.822801+0000 osd.7 (osd.7) 108 : cluster 0 purged_snaps scrub ok 2024-03-11T20:55:36.938 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:36 smithi146 bash[23809]: cluster 2024-03-11T20:55:35.830928+0000 osd.0 (osd.0) 119 : cluster 0 2.4 scrub starts 2024-03-11T20:55:36.938 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:36 smithi146 bash[23809]: cluster 2024-03-11T20:55:35.833881+0000 osd.0 (osd.0) 120 : cluster 0 2.4 scrub ok 2024-03-11T20:55:36.938 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:36 smithi146 bash[23809]: cluster 2024-03-11T20:55:35.916625+0000 mgr.y (mgr.24380) 184 : cluster 0 pgmap v162: 148 pgs: 2 peering, 146 active+clean; 4.2 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 91 MiB/s wr, 1.46k op/s 2024-03-11T20:55:36.951 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:36.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:36.957 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:55:37.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:36 smithi023 bash[21372]: cluster 2024-03-11T20:55:35.457404+0000 osd.4 (osd.4) 111 : cluster 0 3.1a scrub starts 2024-03-11T20:55:37.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:36 smithi023 bash[21372]: cluster 2024-03-11T20:55:35.459595+0000 osd.4 (osd.4) 112 : cluster 0 3.1a scrub ok 2024-03-11T20:55:37.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:36 smithi023 bash[21372]: cluster 2024-03-11T20:55:35.729142+0000 mon.a (mon.0) 986 : cluster 0 osdmap e139: 8 total, 8 up, 7 in 2024-03-11T20:55:37.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:36 smithi023 bash[21372]: cluster 2024-03-11T20:55:35.822721+0000 osd.7 (osd.7) 107 : cluster 0 purged_snaps scrub starts 2024-03-11T20:55:37.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:36 smithi023 bash[21372]: cluster 2024-03-11T20:55:35.822801+0000 osd.7 (osd.7) 108 : cluster 0 purged_snaps scrub ok 2024-03-11T20:55:37.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:36 smithi023 bash[21372]: cluster 2024-03-11T20:55:35.830928+0000 osd.0 (osd.0) 119 : cluster 0 2.4 scrub starts 2024-03-11T20:55:37.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:36 smithi023 bash[21372]: cluster 2024-03-11T20:55:35.833881+0000 osd.0 (osd.0) 120 : cluster 0 2.4 scrub ok 2024-03-11T20:55:37.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:36 smithi023 bash[21372]: cluster 2024-03-11T20:55:35.916625+0000 mgr.y (mgr.24380) 184 : cluster 0 pgmap v162: 148 pgs: 2 peering, 146 active+clean; 4.2 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 91 MiB/s wr, 1.46k op/s 2024-03-11T20:55:37.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:36 smithi023 bash[29146]: cluster 2024-03-11T20:55:35.457404+0000 osd.4 (osd.4) 111 : cluster 0 3.1a scrub starts 2024-03-11T20:55:37.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:36 smithi023 bash[29146]: cluster 2024-03-11T20:55:35.459595+0000 osd.4 (osd.4) 112 : cluster 0 3.1a scrub ok 2024-03-11T20:55:37.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:36 smithi023 bash[29146]: cluster 2024-03-11T20:55:35.729142+0000 mon.a (mon.0) 986 : cluster 0 osdmap e139: 8 total, 8 up, 7 in 2024-03-11T20:55:37.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:36 smithi023 bash[29146]: cluster 2024-03-11T20:55:35.822721+0000 osd.7 (osd.7) 107 : cluster 0 purged_snaps scrub starts 2024-03-11T20:55:37.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:36 smithi023 bash[29146]: cluster 2024-03-11T20:55:35.822801+0000 osd.7 (osd.7) 108 : cluster 0 purged_snaps scrub ok 2024-03-11T20:55:37.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:36 smithi023 bash[29146]: cluster 2024-03-11T20:55:35.830928+0000 osd.0 (osd.0) 119 : cluster 0 2.4 scrub starts 2024-03-11T20:55:37.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:36 smithi023 bash[29146]: cluster 2024-03-11T20:55:35.833881+0000 osd.0 (osd.0) 120 : cluster 0 2.4 scrub ok 2024-03-11T20:55:37.036 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:36 smithi023 bash[29146]: cluster 2024-03-11T20:55:35.916625+0000 mgr.y (mgr.24380) 184 : cluster 0 pgmap v162: 148 pgs: 2 peering, 146 active+clean; 4.2 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 91 MiB/s wr, 1.46k op/s 2024-03-11T20:55:37.058 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:37.081 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:37.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:37.084 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:55:37.185 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:37.215 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:37.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:37.217 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:55:37.318 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:55:37.346 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:55:37.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:37.348 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:55:37.449 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:37.487 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:37.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:37.490 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:55:37.544 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 46 16 71632 71616 97.2842 67 0.0151798 0.0102723 2024-03-11T20:55:37.591 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:37.621 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:37.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:37.625 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:55:37.726 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:37.749 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:37.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:37.753 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:55:37.756 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:37 smithi023 bash[29146]: cluster 2024-03-11T20:55:36.758631+0000 mon.a (mon.0) 987 : cluster 0 osdmap e140: 8 total, 8 up, 7 in 2024-03-11T20:55:37.756 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:37 smithi023 bash[29146]: cluster 2024-03-11T20:55:37.535310+0000 osd.5 (osd.5) 95 : cluster 0 3.c scrub starts 2024-03-11T20:55:37.756 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:37 smithi023 bash[29146]: cluster 2024-03-11T20:55:37.538962+0000 osd.5 (osd.5) 96 : cluster 0 3.c scrub ok 2024-03-11T20:55:37.855 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:55:37.874 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:55:37.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:37.876 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:55:37.977 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:55:38.001 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:55:38.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:38.003 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:55:38.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:37 smithi146 bash[23809]: cluster 2024-03-11T20:55:36.758631+0000 mon.a (mon.0) 987 : cluster 0 osdmap e140: 8 total, 8 up, 7 in 2024-03-11T20:55:38.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:37 smithi146 bash[23809]: cluster 2024-03-11T20:55:37.535310+0000 osd.5 (osd.5) 95 : cluster 0 3.c scrub starts 2024-03-11T20:55:38.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:37 smithi146 bash[23809]: cluster 2024-03-11T20:55:37.538962+0000 osd.5 (osd.5) 96 : cluster 0 3.c scrub ok 2024-03-11T20:55:38.016 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:37 smithi023 bash[21372]: cluster 2024-03-11T20:55:36.758631+0000 mon.a (mon.0) 987 : cluster 0 osdmap e140: 8 total, 8 up, 7 in 2024-03-11T20:55:38.016 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:37 smithi023 bash[21372]: cluster 2024-03-11T20:55:37.535310+0000 osd.5 (osd.5) 95 : cluster 0 3.c scrub starts 2024-03-11T20:55:38.016 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:37 smithi023 bash[21372]: cluster 2024-03-11T20:55:37.538962+0000 osd.5 (osd.5) 96 : cluster 0 3.c scrub ok 2024-03-11T20:55:38.104 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:38.138 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:38.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:38.143 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:55:38.243 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:55:38.269 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:55:38.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:38.271 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:55:38.372 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:55:38.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:38.402 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:55:38.402 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:55:38.503 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:38.521 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:38.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:38.523 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:55:38.544 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 47 16 72534 72518 96.4138 56.375 0.00327109 0.0103516 2024-03-11T20:55:38.624 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:55:38.652 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:55:38.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:38.654 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:55:38.755 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:55:38.769 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:55:38 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:20:55:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T20:55:38.784 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:55:38.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:38.787 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:55:38.887 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:38.915 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:38.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:38.916 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:55:39.017 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:39.026 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:38 smithi146 bash[23809]: cluster 2024-03-11T20:55:37.629041+0000 osd.2 (osd.2) 79 : cluster 0 6.14 scrub starts 2024-03-11T20:55:39.026 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:38 smithi146 bash[23809]: cluster 2024-03-11T20:55:37.630090+0000 osd.2 (osd.2) 80 : cluster 0 6.14 scrub ok 2024-03-11T20:55:39.026 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:38 smithi146 bash[23809]: cluster 2024-03-11T20:55:37.917225+0000 mgr.y (mgr.24380) 185 : cluster 0 pgmap v164: 148 pgs: 2 peering, 146 active+clean; 4.2 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 853 B/s rd, 57 MiB/s wr, 920 op/s 2024-03-11T20:55:39.026 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:38 smithi146 bash[23809]: cluster 2024-03-11T20:55:38.550240+0000 osd.6 (osd.6) 67 : cluster 0 6.1 scrub starts 2024-03-11T20:55:39.026 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:38 smithi146 bash[23809]: cluster 2024-03-11T20:55:38.551508+0000 osd.6 (osd.6) 68 : cluster 0 6.1 scrub ok 2024-03-11T20:55:39.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:38 smithi023 bash[21372]: cluster 2024-03-11T20:55:37.629041+0000 osd.2 (osd.2) 79 : cluster 0 6.14 scrub starts 2024-03-11T20:55:39.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:38 smithi023 bash[21372]: cluster 2024-03-11T20:55:37.630090+0000 osd.2 (osd.2) 80 : cluster 0 6.14 scrub ok 2024-03-11T20:55:39.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:38 smithi023 bash[21372]: cluster 2024-03-11T20:55:37.917225+0000 mgr.y (mgr.24380) 185 : cluster 0 pgmap v164: 148 pgs: 2 peering, 146 active+clean; 4.2 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 853 B/s rd, 57 MiB/s wr, 920 op/s 2024-03-11T20:55:39.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:38 smithi023 bash[21372]: cluster 2024-03-11T20:55:38.550240+0000 osd.6 (osd.6) 67 : cluster 0 6.1 scrub starts 2024-03-11T20:55:39.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:38 smithi023 bash[21372]: cluster 2024-03-11T20:55:38.551508+0000 osd.6 (osd.6) 68 : cluster 0 6.1 scrub ok 2024-03-11T20:55:39.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:38 smithi023 bash[29146]: cluster 2024-03-11T20:55:37.629041+0000 osd.2 (osd.2) 79 : cluster 0 6.14 scrub starts 2024-03-11T20:55:39.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:38 smithi023 bash[29146]: cluster 2024-03-11T20:55:37.630090+0000 osd.2 (osd.2) 80 : cluster 0 6.14 scrub ok 2024-03-11T20:55:39.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:38 smithi023 bash[29146]: cluster 2024-03-11T20:55:37.917225+0000 mgr.y (mgr.24380) 185 : cluster 0 pgmap v164: 148 pgs: 2 peering, 146 active+clean; 4.2 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 853 B/s rd, 57 MiB/s wr, 920 op/s 2024-03-11T20:55:39.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:38 smithi023 bash[29146]: cluster 2024-03-11T20:55:38.550240+0000 osd.6 (osd.6) 67 : cluster 0 6.1 scrub starts 2024-03-11T20:55:39.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:38 smithi023 bash[29146]: cluster 2024-03-11T20:55:38.551508+0000 osd.6 (osd.6) 68 : cluster 0 6.1 scrub ok 2024-03-11T20:55:39.043 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:39.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:39.045 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:55:39.146 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:39.170 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:39.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:39.174 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:55:39.275 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:55:39.314 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:55:39.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:39.316 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:55:39.417 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:55:39.441 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:55:39.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:39.447 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:55:39.544 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 48 16 73551 73535 95.7293 63.5625 0.0117402 0.0104407 2024-03-11T20:55:39.548 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:39.577 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:39.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:39.581 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:55:39.682 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:55:39.700 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:55:39.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:39.702 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:55:39.802 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:39.839 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:39.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:39.841 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:55:39.942 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:55:39.952 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:39 smithi023 bash[21372]: cluster 2024-03-11T20:55:38.168460+0000 osd.7 (osd.7) 109 : cluster 0 2.8 scrub starts 2024-03-11T20:55:39.952 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:39 smithi023 bash[21372]: cluster 2024-03-11T20:55:38.171002+0000 osd.7 (osd.7) 110 : cluster 0 2.8 scrub ok 2024-03-11T20:55:39.952 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:39 smithi023 bash[21372]: cluster 2024-03-11T20:55:38.444989+0000 osd.4 (osd.4) 113 : cluster 0 3.9 scrub starts 2024-03-11T20:55:39.952 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:39 smithi023 bash[21372]: cluster 2024-03-11T20:55:38.448387+0000 osd.4 (osd.4) 114 : cluster 0 3.9 scrub ok 2024-03-11T20:55:39.952 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:39 smithi023 bash[21372]: cluster 2024-03-11T20:55:38.693799+0000 osd.3 (osd.3) 139 : cluster 0 2.1b scrub starts 2024-03-11T20:55:39.952 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:39 smithi023 bash[21372]: cluster 2024-03-11T20:55:38.701030+0000 osd.3 (osd.3) 140 : cluster 0 2.1b scrub ok 2024-03-11T20:55:39.952 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:39 smithi023 bash[21372]: cluster 2024-03-11T20:55:39.673151+0000 mon.a (mon.0) 988 : cluster 0 osdmap e141: 8 total, 8 up, 7 in 2024-03-11T20:55:39.952 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:39 smithi023 bash[29146]: cluster 2024-03-11T20:55:38.168460+0000 osd.7 (osd.7) 109 : cluster 0 2.8 scrub starts 2024-03-11T20:55:39.952 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:39 smithi023 bash[29146]: cluster 2024-03-11T20:55:38.171002+0000 osd.7 (osd.7) 110 : cluster 0 2.8 scrub ok 2024-03-11T20:55:39.952 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:39 smithi023 bash[29146]: cluster 2024-03-11T20:55:38.444989+0000 osd.4 (osd.4) 113 : cluster 0 3.9 scrub starts 2024-03-11T20:55:39.953 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:39 smithi023 bash[29146]: cluster 2024-03-11T20:55:38.448387+0000 osd.4 (osd.4) 114 : cluster 0 3.9 scrub ok 2024-03-11T20:55:39.953 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:39 smithi023 bash[29146]: cluster 2024-03-11T20:55:38.693799+0000 osd.3 (osd.3) 139 : cluster 0 2.1b scrub starts 2024-03-11T20:55:39.953 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:39 smithi023 bash[29146]: cluster 2024-03-11T20:55:38.701030+0000 osd.3 (osd.3) 140 : cluster 0 2.1b scrub ok 2024-03-11T20:55:39.953 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:39 smithi023 bash[29146]: cluster 2024-03-11T20:55:39.673151+0000 mon.a (mon.0) 988 : cluster 0 osdmap e141: 8 total, 8 up, 7 in 2024-03-11T20:55:39.975 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:55:39.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:39.979 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:55:40.080 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:40.089 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:39 smithi146 bash[23809]: cluster 2024-03-11T20:55:38.168460+0000 osd.7 (osd.7) 109 : cluster 0 2.8 scrub starts 2024-03-11T20:55:40.089 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:39 smithi146 bash[23809]: cluster 2024-03-11T20:55:38.171002+0000 osd.7 (osd.7) 110 : cluster 0 2.8 scrub ok 2024-03-11T20:55:40.089 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:39 smithi146 bash[23809]: cluster 2024-03-11T20:55:38.444989+0000 osd.4 (osd.4) 113 : cluster 0 3.9 scrub starts 2024-03-11T20:55:40.089 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:39 smithi146 bash[23809]: cluster 2024-03-11T20:55:38.448387+0000 osd.4 (osd.4) 114 : cluster 0 3.9 scrub ok 2024-03-11T20:55:40.089 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:39 smithi146 bash[23809]: cluster 2024-03-11T20:55:38.693799+0000 osd.3 (osd.3) 139 : cluster 0 2.1b scrub starts 2024-03-11T20:55:40.089 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:39 smithi146 bash[23809]: cluster 2024-03-11T20:55:38.701030+0000 osd.3 (osd.3) 140 : cluster 0 2.1b scrub ok 2024-03-11T20:55:40.089 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:39 smithi146 bash[23809]: cluster 2024-03-11T20:55:39.673151+0000 mon.a (mon.0) 988 : cluster 0 osdmap e141: 8 total, 8 up, 7 in 2024-03-11T20:55:40.107 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:40.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:40.110 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:55:40.211 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:55:40.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:40.243 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:55:40.244 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:55:40.283 INFO:teuthology.orchestra.run.smithi023.stdout:ERROR: (22) Invalid argument 2024-03-11T20:55:40.283 INFO:teuthology.orchestra.run.smithi023.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T20:55:40.345 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:40.382 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:40.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:40.385 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:55:40.486 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:40.504 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:40.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:40.506 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:55:40.544 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 49 16 74423 74407 94.8879 54.5 0.00323974 0.0105167 2024-03-11T20:55:40.606 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:40.633 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:40.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:40.635 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:55:40.736 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:55:40.762 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:55:40.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:40.764 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:55:40.809 INFO:teuthology.orchestra.run.smithi023.stderr:set pool 1 pgp_num to 1 2024-03-11T20:55:40.865 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:40.891 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:40.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:40.894 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:55:40.995 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:41.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:40 smithi146 bash[23809]: cluster 2024-03-11T20:55:39.510472+0000 osd.4 (osd.4) 115 : cluster 0 5.1 scrub starts 2024-03-11T20:55:41.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:40 smithi146 bash[23809]: cluster 2024-03-11T20:55:39.514483+0000 osd.4 (osd.4) 116 : cluster 0 5.1 scrub ok 2024-03-11T20:55:41.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:40 smithi146 bash[23809]: cluster 2024-03-11T20:55:39.918135+0000 mgr.y (mgr.24380) 186 : cluster 0 pgmap v166: 148 pgs: 148 active+clean; 4.4 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 983 B/s rd, 115 MiB/s wr, 1.83k op/s 2024-03-11T20:55:41.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:40 smithi146 bash[23809]: audit 2024-03-11T20:55:40.766146+0000 mon.c (mon.2) 32 : audit 1 from='client.? 172.21.15.23:0/2160648567' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-03-11T20:55:41.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:40 smithi146 bash[23809]: audit 2024-03-11T20:55:40.767197+0000 mon.a (mon.0) 989 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-03-11T20:55:41.016 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:41.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:41.017 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:55:41.052 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:40 smithi023 bash[21372]: cluster 2024-03-11T20:55:39.510472+0000 osd.4 (osd.4) 115 : cluster 0 5.1 scrub starts 2024-03-11T20:55:41.053 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:40 smithi023 bash[21372]: cluster 2024-03-11T20:55:39.514483+0000 osd.4 (osd.4) 116 : cluster 0 5.1 scrub ok 2024-03-11T20:55:41.053 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:40 smithi023 bash[21372]: cluster 2024-03-11T20:55:39.918135+0000 mgr.y (mgr.24380) 186 : cluster 0 pgmap v166: 148 pgs: 148 active+clean; 4.4 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 983 B/s rd, 115 MiB/s wr, 1.83k op/s 2024-03-11T20:55:41.053 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:40 smithi023 bash[21372]: audit 2024-03-11T20:55:40.766146+0000 mon.c (mon.2) 32 : audit 1 from='client.? 172.21.15.23:0/2160648567' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-03-11T20:55:41.053 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:40 smithi023 bash[21372]: audit 2024-03-11T20:55:40.767197+0000 mon.a (mon.0) 989 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-03-11T20:55:41.053 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:40 smithi023 bash[29146]: cluster 2024-03-11T20:55:39.510472+0000 osd.4 (osd.4) 115 : cluster 0 5.1 scrub starts 2024-03-11T20:55:41.053 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:40 smithi023 bash[29146]: cluster 2024-03-11T20:55:39.514483+0000 osd.4 (osd.4) 116 : cluster 0 5.1 scrub ok 2024-03-11T20:55:41.053 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:40 smithi023 bash[29146]: cluster 2024-03-11T20:55:39.918135+0000 mgr.y (mgr.24380) 186 : cluster 0 pgmap v166: 148 pgs: 148 active+clean; 4.4 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 983 B/s rd, 115 MiB/s wr, 1.83k op/s 2024-03-11T20:55:41.053 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:40 smithi023 bash[29146]: audit 2024-03-11T20:55:40.766146+0000 mon.c (mon.2) 32 : audit 1 from='client.? 172.21.15.23:0/2160648567' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-03-11T20:55:41.053 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:40 smithi023 bash[29146]: audit 2024-03-11T20:55:40.767197+0000 mon.a (mon.0) 989 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-03-11T20:55:41.118 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:41.144 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:41.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:41.146 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:55:41.247 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:41.266 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:41.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:41.268 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:55:41.369 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:55:41.394 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:55:41.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:41.396 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:55:41.497 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:41.519 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:41.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:41.521 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:55:41.544 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 50 16 75080 75064 93.8114 41.0625 0.00517795 0.0106155 2024-03-11T20:55:41.622 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:41.657 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:41.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:41.660 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:55:41.761 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:41.785 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:41.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:41.790 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:55:41.891 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:41.921 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:41.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:41.924 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:55:42.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:41 smithi146 bash[23809]: audit 2024-03-11T20:55:40.786852+0000 mon.a (mon.0) 990 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-03-11T20:55:42.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:41 smithi146 bash[23809]: cluster 2024-03-11T20:55:40.798618+0000 mon.a (mon.0) 991 : cluster 0 osdmap e142: 8 total, 8 up, 7 in 2024-03-11T20:55:42.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:41 smithi146 bash[23809]: cluster 2024-03-11T20:55:40.867912+0000 osd.0 (osd.0) 121 : cluster 0 3.17 scrub starts 2024-03-11T20:55:42.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:41 smithi146 bash[23809]: cluster 2024-03-11T20:55:40.872741+0000 osd.0 (osd.0) 122 : cluster 0 3.17 scrub ok 2024-03-11T20:55:42.025 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:42.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:42.053 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:42.053 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:55:42.154 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:55:42.178 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:55:42.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:42.181 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:55:42.251 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:41 smithi023 bash[21372]: audit 2024-03-11T20:55:40.786852+0000 mon.a (mon.0) 990 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-03-11T20:55:42.251 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:41 smithi023 bash[21372]: cluster 2024-03-11T20:55:40.798618+0000 mon.a (mon.0) 991 : cluster 0 osdmap e142: 8 total, 8 up, 7 in 2024-03-11T20:55:42.251 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:41 smithi023 bash[21372]: cluster 2024-03-11T20:55:40.867912+0000 osd.0 (osd.0) 121 : cluster 0 3.17 scrub starts 2024-03-11T20:55:42.251 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:41 smithi023 bash[21372]: cluster 2024-03-11T20:55:40.872741+0000 osd.0 (osd.0) 122 : cluster 0 3.17 scrub ok 2024-03-11T20:55:42.253 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:41 smithi023 bash[29146]: audit 2024-03-11T20:55:40.786852+0000 mon.a (mon.0) 990 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-03-11T20:55:42.253 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:41 smithi023 bash[29146]: cluster 2024-03-11T20:55:40.798618+0000 mon.a (mon.0) 991 : cluster 0 osdmap e142: 8 total, 8 up, 7 in 2024-03-11T20:55:42.253 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:41 smithi023 bash[29146]: cluster 2024-03-11T20:55:40.867912+0000 osd.0 (osd.0) 121 : cluster 0 3.17 scrub starts 2024-03-11T20:55:42.253 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:41 smithi023 bash[29146]: cluster 2024-03-11T20:55:40.872741+0000 osd.0 (osd.0) 122 : cluster 0 3.17 scrub ok 2024-03-11T20:55:42.282 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:55:42.316 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:55:42.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:42.319 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:55:42.420 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:42.439 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:42.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:42.441 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:55:42.479 INFO:teuthology.orchestra.run.smithi023.stdout:osd.0: {} 2024-03-11T20:55:42.499 INFO:teuthology.orchestra.run.smithi023.stdout:osd.1: {} 2024-03-11T20:55:42.542 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:55:42.545 INFO:teuthology.orchestra.run.smithi023.stdout:osd.2: {} 2024-03-11T20:55:42.545 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 51 16 75838 75822 92.8998 47.375 0.0162937 0.0107588 2024-03-11T20:55:42.592 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:55:42.593 INFO:teuthology.orchestra.run.smithi023.stdout:osd.3: {} 2024-03-11T20:55:42.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:42.594 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:55:42.637 INFO:teuthology.orchestra.run.smithi023.stdout:osd.4: {} 2024-03-11T20:55:42.686 INFO:teuthology.orchestra.run.smithi023.stdout:osd.5: {} 2024-03-11T20:55:42.696 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:42.722 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:42.724 INFO:teuthology.orchestra.run.smithi023.stdout:osd.6: {} 2024-03-11T20:55:42.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:42.726 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:55:42.768 INFO:teuthology.orchestra.run.smithi023.stdout:osd.7: {} 2024-03-11T20:55:42.783 INFO:teuthology.orchestra.run.smithi023.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T20:55:42.783 INFO:teuthology.orchestra.run.smithi023.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T20:55:42.783 INFO:teuthology.orchestra.run.smithi023.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T20:55:42.783 INFO:teuthology.orchestra.run.smithi023.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T20:55:42.783 INFO:teuthology.orchestra.run.smithi023.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T20:55:42.783 INFO:teuthology.orchestra.run.smithi023.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T20:55:42.783 INFO:teuthology.orchestra.run.smithi023.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T20:55:42.783 INFO:teuthology.orchestra.run.smithi023.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T20:55:42.827 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:55:42.836 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:42 smithi023 bash[29146]: cluster 2024-03-11T20:55:41.137027+0000 osd.1 (osd.1) 93 : cluster 0 purged_snaps scrub starts 2024-03-11T20:55:42.836 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:42 smithi023 bash[29146]: cluster 2024-03-11T20:55:41.137093+0000 osd.1 (osd.1) 94 : cluster 0 purged_snaps scrub ok 2024-03-11T20:55:42.836 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:42 smithi023 bash[29146]: cluster 2024-03-11T20:55:41.172863+0000 osd.7 (osd.7) 111 : cluster 0 6.8 scrub starts 2024-03-11T20:55:42.836 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:42 smithi023 bash[29146]: cluster 2024-03-11T20:55:41.179877+0000 osd.7 (osd.7) 112 : cluster 0 6.8 scrub ok 2024-03-11T20:55:42.836 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:42 smithi023 bash[29146]: cluster 2024-03-11T20:55:41.220545+0000 osd.4 (osd.4) 117 : cluster 0 purged_snaps scrub starts 2024-03-11T20:55:42.836 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:42 smithi023 bash[29146]: cluster 2024-03-11T20:55:41.220609+0000 osd.4 (osd.4) 118 : cluster 0 purged_snaps scrub ok 2024-03-11T20:55:42.836 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:42 smithi023 bash[29146]: cluster 2024-03-11T20:55:41.674067+0000 osd.2 (osd.2) 81 : cluster 0 3.4 scrub starts 2024-03-11T20:55:42.836 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:42 smithi023 bash[29146]: cluster 2024-03-11T20:55:41.678842+0000 osd.2 (osd.2) 82 : cluster 0 3.4 scrub ok 2024-03-11T20:55:42.836 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:42 smithi023 bash[29146]: cluster 2024-03-11T20:55:41.790813+0000 mon.a (mon.0) 992 : cluster 0 osdmap e143: 8 total, 8 up, 7 in 2024-03-11T20:55:42.836 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:42 smithi023 bash[29146]: cluster 2024-03-11T20:55:41.918573+0000 mgr.y (mgr.24380) 187 : cluster 0 pgmap v169: 148 pgs: 148 active+clean; 4.5 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 991 B/s rd, 60 MiB/s wr, 958 op/s 2024-03-11T20:55:42.836 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:42 smithi023 bash[29146]: audit 2024-03-11T20:55:42.462539+0000 mon.a (mon.0) 993 : audit 0 from='client.? 172.21.15.23:0/3694946724' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T20:55:42.837 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:42 smithi023 bash[21372]: cluster 2024-03-11T20:55:41.137027+0000 osd.1 (osd.1) 93 : cluster 0 purged_snaps scrub starts 2024-03-11T20:55:42.837 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:42 smithi023 bash[21372]: cluster 2024-03-11T20:55:41.137093+0000 osd.1 (osd.1) 94 : cluster 0 purged_snaps scrub ok 2024-03-11T20:55:42.837 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:42 smithi023 bash[21372]: cluster 2024-03-11T20:55:41.172863+0000 osd.7 (osd.7) 111 : cluster 0 6.8 scrub starts 2024-03-11T20:55:42.837 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:42 smithi023 bash[21372]: cluster 2024-03-11T20:55:41.179877+0000 osd.7 (osd.7) 112 : cluster 0 6.8 scrub ok 2024-03-11T20:55:42.837 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:42 smithi023 bash[21372]: cluster 2024-03-11T20:55:41.220545+0000 osd.4 (osd.4) 117 : cluster 0 purged_snaps scrub starts 2024-03-11T20:55:42.837 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:42 smithi023 bash[21372]: cluster 2024-03-11T20:55:41.220609+0000 osd.4 (osd.4) 118 : cluster 0 purged_snaps scrub ok 2024-03-11T20:55:42.837 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:42 smithi023 bash[21372]: cluster 2024-03-11T20:55:41.674067+0000 osd.2 (osd.2) 81 : cluster 0 3.4 scrub starts 2024-03-11T20:55:42.837 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:42 smithi023 bash[21372]: cluster 2024-03-11T20:55:41.678842+0000 osd.2 (osd.2) 82 : cluster 0 3.4 scrub ok 2024-03-11T20:55:42.837 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:42 smithi023 bash[21372]: cluster 2024-03-11T20:55:41.790813+0000 mon.a (mon.0) 992 : cluster 0 osdmap e143: 8 total, 8 up, 7 in 2024-03-11T20:55:42.837 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:42 smithi023 bash[21372]: cluster 2024-03-11T20:55:41.918573+0000 mgr.y (mgr.24380) 187 : cluster 0 pgmap v169: 148 pgs: 148 active+clean; 4.5 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 991 B/s rd, 60 MiB/s wr, 958 op/s 2024-03-11T20:55:42.837 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:42 smithi023 bash[21372]: audit 2024-03-11T20:55:42.462539+0000 mon.a (mon.0) 993 : audit 0 from='client.? 172.21.15.23:0/3694946724' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T20:55:42.856 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:55:42.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:42.859 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:55:42.960 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:42.969 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:42 smithi146 bash[23809]: cluster 2024-03-11T20:55:41.137027+0000 osd.1 (osd.1) 93 : cluster 0 purged_snaps scrub starts 2024-03-11T20:55:42.969 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:42 smithi146 bash[23809]: cluster 2024-03-11T20:55:41.137093+0000 osd.1 (osd.1) 94 : cluster 0 purged_snaps scrub ok 2024-03-11T20:55:42.969 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:42 smithi146 bash[23809]: cluster 2024-03-11T20:55:41.172863+0000 osd.7 (osd.7) 111 : cluster 0 6.8 scrub starts 2024-03-11T20:55:42.969 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:42 smithi146 bash[23809]: cluster 2024-03-11T20:55:41.179877+0000 osd.7 (osd.7) 112 : cluster 0 6.8 scrub ok 2024-03-11T20:55:42.969 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:42 smithi146 bash[23809]: cluster 2024-03-11T20:55:41.220545+0000 osd.4 (osd.4) 117 : cluster 0 purged_snaps scrub starts 2024-03-11T20:55:42.970 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:42 smithi146 bash[23809]: cluster 2024-03-11T20:55:41.220609+0000 osd.4 (osd.4) 118 : cluster 0 purged_snaps scrub ok 2024-03-11T20:55:42.970 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:42 smithi146 bash[23809]: cluster 2024-03-11T20:55:41.674067+0000 osd.2 (osd.2) 81 : cluster 0 3.4 scrub starts 2024-03-11T20:55:42.970 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:42 smithi146 bash[23809]: cluster 2024-03-11T20:55:41.678842+0000 osd.2 (osd.2) 82 : cluster 0 3.4 scrub ok 2024-03-11T20:55:42.970 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:42 smithi146 bash[23809]: cluster 2024-03-11T20:55:41.790813+0000 mon.a (mon.0) 992 : cluster 0 osdmap e143: 8 total, 8 up, 7 in 2024-03-11T20:55:42.970 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:42 smithi146 bash[23809]: cluster 2024-03-11T20:55:41.918573+0000 mgr.y (mgr.24380) 187 : cluster 0 pgmap v169: 148 pgs: 148 active+clean; 4.5 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 991 B/s rd, 60 MiB/s wr, 958 op/s 2024-03-11T20:55:42.970 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:42 smithi146 bash[23809]: audit 2024-03-11T20:55:42.462539+0000 mon.a (mon.0) 993 : audit 0 from='client.? 172.21.15.23:0/3694946724' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T20:55:42.982 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:42.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:42.984 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:55:43.084 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:55:43.118 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:55:43.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:43.123 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:55:43.223 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:55:43.249 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:55:43.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:43.252 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:55:43.353 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:55:43.389 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:55:43.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:43.393 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:55:43.494 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:43.532 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:43.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:43.535 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:55:43.545 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 52 16 76775 76759 92.2394 58.5625 0.00904911 0.0108357 2024-03-11T20:55:43.636 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:55:43.667 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:55:43.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:43.670 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:55:43.758 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-03-11T20:55:43.770 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:55:43.848 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:55:43.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:43.857 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:55:43.958 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:43.970 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:43 smithi023 bash[21372]: cluster 2024-03-11T20:55:43.516396+0000 osd.5 (osd.5) 97 : cluster 0 2.12 scrub starts 2024-03-11T20:55:43.970 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:43 smithi023 bash[21372]: cluster 2024-03-11T20:55:43.516760+0000 osd.6 (osd.6) 69 : cluster 0 2.11 scrub starts 2024-03-11T20:55:43.970 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:43 smithi023 bash[21372]: cluster 2024-03-11T20:55:43.523898+0000 osd.5 (osd.5) 98 : cluster 0 2.12 scrub ok 2024-03-11T20:55:43.970 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:43 smithi023 bash[21372]: cluster 2024-03-11T20:55:43.524347+0000 osd.6 (osd.6) 70 : cluster 0 2.11 scrub ok 2024-03-11T20:55:43.970 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:43 smithi023 bash[29146]: cluster 2024-03-11T20:55:43.516396+0000 osd.5 (osd.5) 97 : cluster 0 2.12 scrub starts 2024-03-11T20:55:43.970 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:43 smithi023 bash[29146]: cluster 2024-03-11T20:55:43.516760+0000 osd.6 (osd.6) 69 : cluster 0 2.11 scrub starts 2024-03-11T20:55:43.970 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:43 smithi023 bash[29146]: cluster 2024-03-11T20:55:43.523898+0000 osd.5 (osd.5) 98 : cluster 0 2.12 scrub ok 2024-03-11T20:55:43.970 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:43 smithi023 bash[29146]: cluster 2024-03-11T20:55:43.524347+0000 osd.6 (osd.6) 70 : cluster 0 2.11 scrub ok 2024-03-11T20:55:43.994 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:43.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:43.999 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:55:44.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:43 smithi146 bash[23809]: cluster 2024-03-11T20:55:43.516396+0000 osd.5 (osd.5) 97 : cluster 0 2.12 scrub starts 2024-03-11T20:55:44.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:43 smithi146 bash[23809]: cluster 2024-03-11T20:55:43.516760+0000 osd.6 (osd.6) 69 : cluster 0 2.11 scrub starts 2024-03-11T20:55:44.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:43 smithi146 bash[23809]: cluster 2024-03-11T20:55:43.523898+0000 osd.5 (osd.5) 98 : cluster 0 2.12 scrub ok 2024-03-11T20:55:44.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:43 smithi146 bash[23809]: cluster 2024-03-11T20:55:43.524347+0000 osd.6 (osd.6) 70 : cluster 0 2.11 scrub ok 2024-03-11T20:55:44.100 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:55:44.124 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:55:44.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:44.132 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:55:44.233 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:44.260 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:44.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:44.263 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:55:44.364 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:44.394 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:44.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:44.399 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:55:44.501 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:44.528 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:44.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:44.532 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:55:44.545 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 53 16 77816 77800 91.7266 65.0625 0.00912112 0.0108966 2024-03-11T20:55:44.633 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:55:44.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:44.665 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:55:44.665 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:55:44.766 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:55:44.805 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:55:44.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:44.812 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:55:44.913 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:44.937 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:44.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:44.939 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:55:45.040 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:45.071 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:45.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:45.078 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:55:45.080 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:44 smithi023 bash[29146]: cluster 2024-03-11T20:55:43.210768+0000 osd.7 (osd.7) 113 : cluster 0 6.1c scrub starts 2024-03-11T20:55:45.081 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:44 smithi023 bash[29146]: cluster 2024-03-11T20:55:43.212417+0000 osd.7 (osd.7) 114 : cluster 0 6.1c scrub ok 2024-03-11T20:55:45.081 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:44 smithi023 bash[29146]: cluster 2024-03-11T20:55:43.800754+0000 osd.3 (osd.3) 141 : cluster 0 6.13 scrub starts 2024-03-11T20:55:45.081 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:44 smithi023 bash[29146]: cluster 2024-03-11T20:55:43.801632+0000 osd.3 (osd.3) 142 : cluster 0 6.13 scrub ok 2024-03-11T20:55:45.081 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:44 smithi023 bash[29146]: cluster 2024-03-11T20:55:43.829079+0000 osd.0 (osd.0) 123 : cluster 0 3.1f scrub starts 2024-03-11T20:55:45.081 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:44 smithi023 bash[29146]: cluster 2024-03-11T20:55:43.832781+0000 osd.0 (osd.0) 124 : cluster 0 3.1f scrub ok 2024-03-11T20:55:45.081 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:44 smithi023 bash[29146]: cluster 2024-03-11T20:55:43.922005+0000 mgr.y (mgr.24380) 188 : cluster 0 pgmap v170: 148 pgs: 148 active+clean; 4.5 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 852 B/s rd, 51 MiB/s wr, 824 op/s 2024-03-11T20:55:45.081 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:44 smithi023 bash[21372]: cluster 2024-03-11T20:55:43.210768+0000 osd.7 (osd.7) 113 : cluster 0 6.1c scrub starts 2024-03-11T20:55:45.082 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:44 smithi023 bash[21372]: cluster 2024-03-11T20:55:43.212417+0000 osd.7 (osd.7) 114 : cluster 0 6.1c scrub ok 2024-03-11T20:55:45.082 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:44 smithi023 bash[21372]: cluster 2024-03-11T20:55:43.800754+0000 osd.3 (osd.3) 141 : cluster 0 6.13 scrub starts 2024-03-11T20:55:45.082 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:44 smithi023 bash[21372]: cluster 2024-03-11T20:55:43.801632+0000 osd.3 (osd.3) 142 : cluster 0 6.13 scrub ok 2024-03-11T20:55:45.082 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:44 smithi023 bash[21372]: cluster 2024-03-11T20:55:43.829079+0000 osd.0 (osd.0) 123 : cluster 0 3.1f scrub starts 2024-03-11T20:55:45.082 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:44 smithi023 bash[21372]: cluster 2024-03-11T20:55:43.832781+0000 osd.0 (osd.0) 124 : cluster 0 3.1f scrub ok 2024-03-11T20:55:45.082 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:44 smithi023 bash[21372]: cluster 2024-03-11T20:55:43.922005+0000 mgr.y (mgr.24380) 188 : cluster 0 pgmap v170: 148 pgs: 148 active+clean; 4.5 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 852 B/s rd, 51 MiB/s wr, 824 op/s 2024-03-11T20:55:45.180 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:45.209 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:45.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:45.218 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:55:45.260 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 20:55:45 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T20:55:45.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:44 smithi146 bash[23809]: cluster 2024-03-11T20:55:43.210768+0000 osd.7 (osd.7) 113 : cluster 0 6.1c scrub starts 2024-03-11T20:55:45.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:44 smithi146 bash[23809]: cluster 2024-03-11T20:55:43.212417+0000 osd.7 (osd.7) 114 : cluster 0 6.1c scrub ok 2024-03-11T20:55:45.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:44 smithi146 bash[23809]: cluster 2024-03-11T20:55:43.800754+0000 osd.3 (osd.3) 141 : cluster 0 6.13 scrub starts 2024-03-11T20:55:45.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:44 smithi146 bash[23809]: cluster 2024-03-11T20:55:43.801632+0000 osd.3 (osd.3) 142 : cluster 0 6.13 scrub ok 2024-03-11T20:55:45.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:44 smithi146 bash[23809]: cluster 2024-03-11T20:55:43.829079+0000 osd.0 (osd.0) 123 : cluster 0 3.1f scrub starts 2024-03-11T20:55:45.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:44 smithi146 bash[23809]: cluster 2024-03-11T20:55:43.832781+0000 osd.0 (osd.0) 124 : cluster 0 3.1f scrub ok 2024-03-11T20:55:45.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:44 smithi146 bash[23809]: cluster 2024-03-11T20:55:43.922005+0000 mgr.y (mgr.24380) 188 : cluster 0 pgmap v170: 148 pgs: 148 active+clean; 4.5 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 852 B/s rd, 51 MiB/s wr, 824 op/s 2024-03-11T20:55:45.320 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:55:45.351 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:55:45.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:45.352 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:55:45.453 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:55:45.473 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:55:45.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:45.475 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:55:45.545 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 54 16 78797 78781 91.1634 61.3125 0.00688699 0.0109632 2024-03-11T20:55:45.576 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:45.604 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:45.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:45.611 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:55:45.712 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:55:45.749 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:55:45.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:45.751 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:55:45.853 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:55:45.885 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:55:45.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:45.889 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:55:45.990 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:45.999 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:45 smithi146 bash[23809]: cluster 2024-03-11T20:55:44.167341+0000 osd.7 (osd.7) 115 : cluster 0 6.5 scrub starts 2024-03-11T20:55:45.999 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:45 smithi146 bash[23809]: cluster 2024-03-11T20:55:44.171009+0000 osd.7 (osd.7) 116 : cluster 0 6.5 scrub ok 2024-03-11T20:55:46.000 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:45 smithi146 bash[23809]: cluster 2024-03-11T20:55:44.854095+0000 osd.0 (osd.0) 125 : cluster 0 4.3 scrub starts 2024-03-11T20:55:46.000 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:45 smithi146 bash[23809]: cluster 2024-03-11T20:55:44.858746+0000 osd.0 (osd.0) 126 : cluster 0 4.3 scrub ok 2024-03-11T20:55:46.000 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:45 smithi146 bash[23809]: audit 2024-03-11T20:55:45.041124+0000 mgr.y (mgr.24380) 189 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:55:46.000 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:45 smithi146 bash[23809]: cluster 2024-03-11T20:55:45.548529+0000 osd.5 (osd.5) 99 : cluster 0 6.a scrub starts 2024-03-11T20:55:46.000 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:45 smithi146 bash[23809]: cluster 2024-03-11T20:55:45.550782+0000 osd.5 (osd.5) 100 : cluster 0 6.a scrub ok 2024-03-11T20:55:46.019 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:46.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:46.021 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:55:46.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:45 smithi023 bash[29146]: cluster 2024-03-11T20:55:44.167341+0000 osd.7 (osd.7) 115 : cluster 0 6.5 scrub starts 2024-03-11T20:55:46.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:45 smithi023 bash[29146]: cluster 2024-03-11T20:55:44.171009+0000 osd.7 (osd.7) 116 : cluster 0 6.5 scrub ok 2024-03-11T20:55:46.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:45 smithi023 bash[29146]: cluster 2024-03-11T20:55:44.854095+0000 osd.0 (osd.0) 125 : cluster 0 4.3 scrub starts 2024-03-11T20:55:46.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:45 smithi023 bash[29146]: cluster 2024-03-11T20:55:44.858746+0000 osd.0 (osd.0) 126 : cluster 0 4.3 scrub ok 2024-03-11T20:55:46.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:45 smithi023 bash[29146]: audit 2024-03-11T20:55:45.041124+0000 mgr.y (mgr.24380) 189 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:55:46.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:45 smithi023 bash[29146]: cluster 2024-03-11T20:55:45.548529+0000 osd.5 (osd.5) 99 : cluster 0 6.a scrub starts 2024-03-11T20:55:46.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:45 smithi023 bash[29146]: cluster 2024-03-11T20:55:45.550782+0000 osd.5 (osd.5) 100 : cluster 0 6.a scrub ok 2024-03-11T20:55:46.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:45 smithi023 bash[21372]: cluster 2024-03-11T20:55:44.167341+0000 osd.7 (osd.7) 115 : cluster 0 6.5 scrub starts 2024-03-11T20:55:46.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:45 smithi023 bash[21372]: cluster 2024-03-11T20:55:44.171009+0000 osd.7 (osd.7) 116 : cluster 0 6.5 scrub ok 2024-03-11T20:55:46.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:45 smithi023 bash[21372]: cluster 2024-03-11T20:55:44.854095+0000 osd.0 (osd.0) 125 : cluster 0 4.3 scrub starts 2024-03-11T20:55:46.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:45 smithi023 bash[21372]: cluster 2024-03-11T20:55:44.858746+0000 osd.0 (osd.0) 126 : cluster 0 4.3 scrub ok 2024-03-11T20:55:46.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:45 smithi023 bash[21372]: audit 2024-03-11T20:55:45.041124+0000 mgr.y (mgr.24380) 189 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:55:46.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:45 smithi023 bash[21372]: cluster 2024-03-11T20:55:45.548529+0000 osd.5 (osd.5) 99 : cluster 0 6.a scrub starts 2024-03-11T20:55:46.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:45 smithi023 bash[21372]: cluster 2024-03-11T20:55:45.550782+0000 osd.5 (osd.5) 100 : cluster 0 6.a scrub ok 2024-03-11T20:55:46.122 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:55:46.148 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:55:46.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:46.152 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:55:46.253 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:55:46.295 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:55:46.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:46.302 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:55:46.403 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:46.427 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:46.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:46.433 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:55:46.534 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:55:46.545 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 55 16 79804 79788 90.6501 62.9375 0.0123371 0.0110249 2024-03-11T20:55:46.564 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:55:46.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:46.573 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:55:46.673 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:55:46.695 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:55:46.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:46.697 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:55:46.798 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:55:46.820 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:55:46.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:46.823 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:55:46.865 INFO:teuthology.orchestra.run.smithi023.stderr:nodeep-scrub is unset 2024-03-11T20:55:46.924 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:55:46.934 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:46 smithi146 bash[23809]: cluster 2024-03-11T20:55:45.922886+0000 mgr.y (mgr.24380) 190 : cluster 0 pgmap v171: 148 pgs: 148 active+clean; 4.8 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.6 KiB/s rd, 93 MiB/s wr, 1.49k op/s 2024-03-11T20:55:46.934 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:46 smithi146 bash[23809]: audit 2024-03-11T20:55:45.985090+0000 mon.a (mon.0) 994 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:55:46.934 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:46 smithi146 bash[23809]: cluster 2024-03-11T20:55:46.565204+0000 osd.5 (osd.5) 101 : cluster 0 3.1c scrub starts 2024-03-11T20:55:46.934 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:46 smithi146 bash[23809]: cluster 2024-03-11T20:55:46.567405+0000 osd.5 (osd.5) 102 : cluster 0 3.1c scrub ok 2024-03-11T20:55:46.934 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:46 smithi146 bash[23809]: cluster 2024-03-11T20:55:46.567473+0000 osd.6 (osd.6) 71 : cluster 0 4.a scrub starts 2024-03-11T20:55:46.934 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:46 smithi146 bash[23809]: cluster 2024-03-11T20:55:46.569158+0000 osd.6 (osd.6) 72 : cluster 0 4.a scrub ok 2024-03-11T20:55:46.935 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:46 smithi146 bash[23809]: audit 2024-03-11T20:55:46.794438+0000 mon.c (mon.2) 33 : audit 1 from='client.? 172.21.15.23:0/628979423' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T20:55:46.935 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:46 smithi146 bash[23809]: audit 2024-03-11T20:55:46.794961+0000 mon.a (mon.0) 995 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T20:55:46.954 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:55:46.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:46.958 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:55:47.059 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:55:47.084 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:55:47.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:47.088 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:55:47.188 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:55:47.212 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:55:47.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:47.218 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:55:47.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:46 smithi023 bash[21372]: cluster 2024-03-11T20:55:45.922886+0000 mgr.y (mgr.24380) 190 : cluster 0 pgmap v171: 148 pgs: 148 active+clean; 4.8 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.6 KiB/s rd, 93 MiB/s wr, 1.49k op/s 2024-03-11T20:55:47.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:46 smithi023 bash[21372]: audit 2024-03-11T20:55:45.985090+0000 mon.a (mon.0) 994 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:55:47.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:46 smithi023 bash[21372]: cluster 2024-03-11T20:55:46.565204+0000 osd.5 (osd.5) 101 : cluster 0 3.1c scrub starts 2024-03-11T20:55:47.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:46 smithi023 bash[21372]: cluster 2024-03-11T20:55:46.567405+0000 osd.5 (osd.5) 102 : cluster 0 3.1c scrub ok 2024-03-11T20:55:47.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:46 smithi023 bash[21372]: cluster 2024-03-11T20:55:46.567473+0000 osd.6 (osd.6) 71 : cluster 0 4.a scrub starts 2024-03-11T20:55:47.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:46 smithi023 bash[21372]: cluster 2024-03-11T20:55:46.569158+0000 osd.6 (osd.6) 72 : cluster 0 4.a scrub ok 2024-03-11T20:55:47.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:46 smithi023 bash[21372]: audit 2024-03-11T20:55:46.794438+0000 mon.c (mon.2) 33 : audit 1 from='client.? 172.21.15.23:0/628979423' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T20:55:47.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:46 smithi023 bash[21372]: audit 2024-03-11T20:55:46.794961+0000 mon.a (mon.0) 995 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T20:55:47.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:46 smithi023 bash[29146]: cluster 2024-03-11T20:55:45.922886+0000 mgr.y (mgr.24380) 190 : cluster 0 pgmap v171: 148 pgs: 148 active+clean; 4.8 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.6 KiB/s rd, 93 MiB/s wr, 1.49k op/s 2024-03-11T20:55:47.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:46 smithi023 bash[29146]: audit 2024-03-11T20:55:45.985090+0000 mon.a (mon.0) 994 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:55:47.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:46 smithi023 bash[29146]: cluster 2024-03-11T20:55:46.565204+0000 osd.5 (osd.5) 101 : cluster 0 3.1c scrub starts 2024-03-11T20:55:47.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:46 smithi023 bash[29146]: cluster 2024-03-11T20:55:46.567405+0000 osd.5 (osd.5) 102 : cluster 0 3.1c scrub ok 2024-03-11T20:55:47.286 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:46 smithi023 bash[29146]: cluster 2024-03-11T20:55:46.567473+0000 osd.6 (osd.6) 71 : cluster 0 4.a scrub starts 2024-03-11T20:55:47.286 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:46 smithi023 bash[29146]: cluster 2024-03-11T20:55:46.569158+0000 osd.6 (osd.6) 72 : cluster 0 4.a scrub ok 2024-03-11T20:55:47.286 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:46 smithi023 bash[29146]: audit 2024-03-11T20:55:46.794438+0000 mon.c (mon.2) 33 : audit 1 from='client.? 172.21.15.23:0/628979423' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T20:55:47.286 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:46 smithi023 bash[29146]: audit 2024-03-11T20:55:46.794961+0000 mon.a (mon.0) 995 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T20:55:47.319 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:55:47.345 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:55:47.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:47.350 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:55:47.451 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:55:47.479 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:55:47.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:47.481 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:55:47.545 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 56 16 80773 80757 90.1127 60.5625 0.00312088 0.0110807 2024-03-11T20:55:47.582 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:55:47.600 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:55:47.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:47.602 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:55:47.703 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:55:47.729 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:55:47.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:47.731 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:55:47.762 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T20:55:47.832 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:55:47.875 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:55:47.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:47.879 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:55:47.979 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:55:48.006 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:55:48.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:48.009 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:55:48.037 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:47 smithi023 bash[21372]: cluster 2024-03-11T20:55:46.795590+0000 osd.3 (osd.3) 143 : cluster 0 3.7 scrub starts 2024-03-11T20:55:48.037 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:47 smithi023 bash[21372]: cluster 2024-03-11T20:55:46.802016+0000 osd.3 (osd.3) 144 : cluster 0 3.7 scrub ok 2024-03-11T20:55:48.037 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:47 smithi023 bash[21372]: cluster 2024-03-11T20:55:46.833763+0000 mon.a (mon.0) 996 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T20:55:48.037 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:47 smithi023 bash[21372]: cluster 2024-03-11T20:55:46.833786+0000 mon.a (mon.0) 997 : cluster 1 Cluster is now healthy 2024-03-11T20:55:48.037 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:47 smithi023 bash[21372]: audit 2024-03-11T20:55:46.851172+0000 mon.a (mon.0) 998 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T20:55:48.038 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:47 smithi023 bash[21372]: cluster 2024-03-11T20:55:46.861287+0000 mon.a (mon.0) 999 : cluster 0 osdmap e144: 8 total, 8 up, 7 in 2024-03-11T20:55:48.038 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:47 smithi023 bash[21372]: audit 2024-03-11T20:55:46.883309+0000 mon.a (mon.0) 1000 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:55:48.038 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:47 smithi023 bash[21372]: audit 2024-03-11T20:55:46.885082+0000 mon.a (mon.0) 1001 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:55:48.038 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:47 smithi023 bash[21372]: audit 2024-03-11T20:55:46.899392+0000 mon.a (mon.0) 1002 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:55:48.038 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:47 smithi023 bash[29146]: cluster 2024-03-11T20:55:46.795590+0000 osd.3 (osd.3) 143 : cluster 0 3.7 scrub starts 2024-03-11T20:55:48.038 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:47 smithi023 bash[29146]: cluster 2024-03-11T20:55:46.802016+0000 osd.3 (osd.3) 144 : cluster 0 3.7 scrub ok 2024-03-11T20:55:48.038 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:47 smithi023 bash[29146]: cluster 2024-03-11T20:55:46.833763+0000 mon.a (mon.0) 996 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T20:55:48.038 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:47 smithi023 bash[29146]: cluster 2024-03-11T20:55:46.833786+0000 mon.a (mon.0) 997 : cluster 1 Cluster is now healthy 2024-03-11T20:55:48.038 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:47 smithi023 bash[29146]: audit 2024-03-11T20:55:46.851172+0000 mon.a (mon.0) 998 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T20:55:48.038 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:47 smithi023 bash[29146]: cluster 2024-03-11T20:55:46.861287+0000 mon.a (mon.0) 999 : cluster 0 osdmap e144: 8 total, 8 up, 7 in 2024-03-11T20:55:48.038 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:47 smithi023 bash[29146]: audit 2024-03-11T20:55:46.883309+0000 mon.a (mon.0) 1000 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:55:48.039 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:47 smithi023 bash[29146]: audit 2024-03-11T20:55:46.885082+0000 mon.a (mon.0) 1001 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:55:48.039 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:47 smithi023 bash[29146]: audit 2024-03-11T20:55:46.899392+0000 mon.a (mon.0) 1002 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:55:48.110 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:48.162 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:48.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:48.171 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:55:48.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:47 smithi146 bash[23809]: cluster 2024-03-11T20:55:46.795590+0000 osd.3 (osd.3) 143 : cluster 0 3.7 scrub starts 2024-03-11T20:55:48.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:47 smithi146 bash[23809]: cluster 2024-03-11T20:55:46.802016+0000 osd.3 (osd.3) 144 : cluster 0 3.7 scrub ok 2024-03-11T20:55:48.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:47 smithi146 bash[23809]: cluster 2024-03-11T20:55:46.833763+0000 mon.a (mon.0) 996 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T20:55:48.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:47 smithi146 bash[23809]: cluster 2024-03-11T20:55:46.833786+0000 mon.a (mon.0) 997 : cluster 1 Cluster is now healthy 2024-03-11T20:55:48.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:47 smithi146 bash[23809]: audit 2024-03-11T20:55:46.851172+0000 mon.a (mon.0) 998 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T20:55:48.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:47 smithi146 bash[23809]: cluster 2024-03-11T20:55:46.861287+0000 mon.a (mon.0) 999 : cluster 0 osdmap e144: 8 total, 8 up, 7 in 2024-03-11T20:55:48.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:47 smithi146 bash[23809]: audit 2024-03-11T20:55:46.883309+0000 mon.a (mon.0) 1000 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:55:48.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:47 smithi146 bash[23809]: audit 2024-03-11T20:55:46.885082+0000 mon.a (mon.0) 1001 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:55:48.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:47 smithi146 bash[23809]: audit 2024-03-11T20:55:46.899392+0000 mon.a (mon.0) 1002 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:55:48.272 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:48.306 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:48.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:48.310 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:55:48.410 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:48.442 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:48.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:48.443 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:55:48.534 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:55:48 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:20:55:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T20:55:48.543 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:55:48.545 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 57 16 81663 81647 89.5077 55.625 0.0173577 0.0111671 2024-03-11T20:55:48.568 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:55:48.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:48.573 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:55:48.674 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:55:48.704 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:55:48.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:48.706 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:55:48.807 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:48.833 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:48.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:48.840 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:55:48.941 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:48.966 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:48.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:48.969 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:55:49.070 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:49.079 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:48 smithi023 bash[21372]: cluster 2024-03-11T20:55:47.120116+0000 osd.7 (osd.7) 117 : cluster 0 3.d scrub starts 2024-03-11T20:55:49.079 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:48 smithi023 bash[21372]: cluster 2024-03-11T20:55:47.126444+0000 osd.7 (osd.7) 118 : cluster 0 3.d scrub ok 2024-03-11T20:55:49.080 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:48 smithi023 bash[21372]: cluster 2024-03-11T20:55:47.301614+0000 osd.4 (osd.4) 119 : cluster 0 2.f scrub starts 2024-03-11T20:55:49.080 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:48 smithi023 bash[21372]: cluster 2024-03-11T20:55:47.308382+0000 osd.4 (osd.4) 120 : cluster 0 2.f scrub ok 2024-03-11T20:55:49.080 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:48 smithi023 bash[21372]: cluster 2024-03-11T20:55:47.750523+0000 osd.6 (osd.6) 73 : cluster 0 4.12 scrub starts 2024-03-11T20:55:49.080 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:48 smithi023 bash[21372]: cluster 2024-03-11T20:55:47.753912+0000 osd.6 (osd.6) 74 : cluster 0 4.12 scrub ok 2024-03-11T20:55:49.080 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:48 smithi023 bash[21372]: cluster 2024-03-11T20:55:47.757859+0000 osd.3 (osd.3) 145 : cluster 0 6.6 scrub starts 2024-03-11T20:55:49.080 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:48 smithi023 bash[21372]: cluster 2024-03-11T20:55:47.767248+0000 osd.3 (osd.3) 146 : cluster 0 6.6 scrub ok 2024-03-11T20:55:49.080 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:48 smithi023 bash[21372]: cluster 2024-03-11T20:55:47.923354+0000 mgr.y (mgr.24380) 191 : cluster 0 pgmap v173: 148 pgs: 148 active+clean; 4.8 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 717 B/s rd, 38 MiB/s wr, 614 op/s 2024-03-11T20:55:49.080 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:48 smithi023 bash[21372]: cluster 2024-03-11T20:55:48.084502+0000 osd.7 (osd.7) 119 : cluster 0 1.0 scrub starts 2024-03-11T20:55:49.080 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:48 smithi023 bash[21372]: cluster 2024-03-11T20:55:48.086593+0000 osd.7 (osd.7) 120 : cluster 0 1.0 scrub ok 2024-03-11T20:55:49.080 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:48 smithi023 bash[21372]: cluster 2024-03-11T20:55:48.608049+0000 osd.5 (osd.5) 103 : cluster 0 2.18 scrub starts 2024-03-11T20:55:49.081 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:48 smithi023 bash[21372]: cluster 2024-03-11T20:55:48.611711+0000 osd.5 (osd.5) 104 : cluster 0 2.18 scrub ok 2024-03-11T20:55:49.081 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:48 smithi023 bash[21372]: cluster 2024-03-11T20:55:48.719733+0000 osd.2 (osd.2) 83 : cluster 0 2.e scrub starts 2024-03-11T20:55:49.081 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:48 smithi023 bash[21372]: cluster 2024-03-11T20:55:48.721701+0000 osd.2 (osd.2) 84 : cluster 0 2.e scrub ok 2024-03-11T20:55:49.081 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:48 smithi023 bash[29146]: cluster 2024-03-11T20:55:47.120116+0000 osd.7 (osd.7) 117 : cluster 0 3.d scrub starts 2024-03-11T20:55:49.081 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:48 smithi023 bash[29146]: cluster 2024-03-11T20:55:47.126444+0000 osd.7 (osd.7) 118 : cluster 0 3.d scrub ok 2024-03-11T20:55:49.081 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:48 smithi023 bash[29146]: cluster 2024-03-11T20:55:47.301614+0000 osd.4 (osd.4) 119 : cluster 0 2.f scrub starts 2024-03-11T20:55:49.081 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:48 smithi023 bash[29146]: cluster 2024-03-11T20:55:47.308382+0000 osd.4 (osd.4) 120 : cluster 0 2.f scrub ok 2024-03-11T20:55:49.081 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:48 smithi023 bash[29146]: cluster 2024-03-11T20:55:47.750523+0000 osd.6 (osd.6) 73 : cluster 0 4.12 scrub starts 2024-03-11T20:55:49.082 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:48 smithi023 bash[29146]: cluster 2024-03-11T20:55:47.753912+0000 osd.6 (osd.6) 74 : cluster 0 4.12 scrub ok 2024-03-11T20:55:49.082 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:48 smithi023 bash[29146]: cluster 2024-03-11T20:55:47.757859+0000 osd.3 (osd.3) 145 : cluster 0 6.6 scrub starts 2024-03-11T20:55:49.082 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:48 smithi023 bash[29146]: cluster 2024-03-11T20:55:47.767248+0000 osd.3 (osd.3) 146 : cluster 0 6.6 scrub ok 2024-03-11T20:55:49.082 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:48 smithi023 bash[29146]: cluster 2024-03-11T20:55:47.923354+0000 mgr.y (mgr.24380) 191 : cluster 0 pgmap v173: 148 pgs: 148 active+clean; 4.8 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 717 B/s rd, 38 MiB/s wr, 614 op/s 2024-03-11T20:55:49.082 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:48 smithi023 bash[29146]: cluster 2024-03-11T20:55:48.084502+0000 osd.7 (osd.7) 119 : cluster 0 1.0 scrub starts 2024-03-11T20:55:49.082 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:48 smithi023 bash[29146]: cluster 2024-03-11T20:55:48.086593+0000 osd.7 (osd.7) 120 : cluster 0 1.0 scrub ok 2024-03-11T20:55:49.082 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:48 smithi023 bash[29146]: cluster 2024-03-11T20:55:48.608049+0000 osd.5 (osd.5) 103 : cluster 0 2.18 scrub starts 2024-03-11T20:55:49.082 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:48 smithi023 bash[29146]: cluster 2024-03-11T20:55:48.611711+0000 osd.5 (osd.5) 104 : cluster 0 2.18 scrub ok 2024-03-11T20:55:49.082 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:48 smithi023 bash[29146]: cluster 2024-03-11T20:55:48.719733+0000 osd.2 (osd.2) 83 : cluster 0 2.e scrub starts 2024-03-11T20:55:49.082 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:48 smithi023 bash[29146]: cluster 2024-03-11T20:55:48.721701+0000 osd.2 (osd.2) 84 : cluster 0 2.e scrub ok 2024-03-11T20:55:49.097 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:49.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:49.099 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:55:49.200 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:55:49.210 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:48 smithi146 bash[23809]: cluster 2024-03-11T20:55:47.120116+0000 osd.7 (osd.7) 117 : cluster 0 3.d scrub starts 2024-03-11T20:55:49.211 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:48 smithi146 bash[23809]: cluster 2024-03-11T20:55:47.126444+0000 osd.7 (osd.7) 118 : cluster 0 3.d scrub ok 2024-03-11T20:55:49.211 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:48 smithi146 bash[23809]: cluster 2024-03-11T20:55:47.301614+0000 osd.4 (osd.4) 119 : cluster 0 2.f scrub starts 2024-03-11T20:55:49.211 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:48 smithi146 bash[23809]: cluster 2024-03-11T20:55:47.308382+0000 osd.4 (osd.4) 120 : cluster 0 2.f scrub ok 2024-03-11T20:55:49.211 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:48 smithi146 bash[23809]: cluster 2024-03-11T20:55:47.750523+0000 osd.6 (osd.6) 73 : cluster 0 4.12 scrub starts 2024-03-11T20:55:49.211 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:48 smithi146 bash[23809]: cluster 2024-03-11T20:55:47.753912+0000 osd.6 (osd.6) 74 : cluster 0 4.12 scrub ok 2024-03-11T20:55:49.211 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:48 smithi146 bash[23809]: cluster 2024-03-11T20:55:47.757859+0000 osd.3 (osd.3) 145 : cluster 0 6.6 scrub starts 2024-03-11T20:55:49.211 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:48 smithi146 bash[23809]: cluster 2024-03-11T20:55:47.767248+0000 osd.3 (osd.3) 146 : cluster 0 6.6 scrub ok 2024-03-11T20:55:49.211 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:48 smithi146 bash[23809]: cluster 2024-03-11T20:55:47.923354+0000 mgr.y (mgr.24380) 191 : cluster 0 pgmap v173: 148 pgs: 148 active+clean; 4.8 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 717 B/s rd, 38 MiB/s wr, 614 op/s 2024-03-11T20:55:49.211 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:48 smithi146 bash[23809]: cluster 2024-03-11T20:55:48.084502+0000 osd.7 (osd.7) 119 : cluster 0 1.0 scrub starts 2024-03-11T20:55:49.211 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:48 smithi146 bash[23809]: cluster 2024-03-11T20:55:48.086593+0000 osd.7 (osd.7) 120 : cluster 0 1.0 scrub ok 2024-03-11T20:55:49.211 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:48 smithi146 bash[23809]: cluster 2024-03-11T20:55:48.608049+0000 osd.5 (osd.5) 103 : cluster 0 2.18 scrub starts 2024-03-11T20:55:49.211 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:48 smithi146 bash[23809]: cluster 2024-03-11T20:55:48.611711+0000 osd.5 (osd.5) 104 : cluster 0 2.18 scrub ok 2024-03-11T20:55:49.211 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:48 smithi146 bash[23809]: cluster 2024-03-11T20:55:48.719733+0000 osd.2 (osd.2) 83 : cluster 0 2.e scrub starts 2024-03-11T20:55:49.211 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:48 smithi146 bash[23809]: cluster 2024-03-11T20:55:48.721701+0000 osd.2 (osd.2) 84 : cluster 0 2.e scrub ok 2024-03-11T20:55:49.233 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:55:49.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:49.237 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:55:49.338 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:49.368 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:49.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:49.370 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:55:49.436 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 6, 7] out_osds: [1] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-11T20:55:49.436 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-11T20:55:49.436 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-03-11T20:55:49.436 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd dump -f json-pretty 2024-03-11T20:55:49.470 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:49.515 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:49.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:49.519 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:55:49.545 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 58 16 82794 82778 89.1831 70.6875 0.0133145 0.0112073 2024-03-11T20:55:49.620 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:55:49.649 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:55:49.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:49.654 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:55:49.665 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:55:49.755 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:55:49.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:49.796 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:55:49.796 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:55:49.896 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:49.929 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:49.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:49.934 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:55:50.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:49 smithi146 bash[23809]: cluster 2024-03-11T20:55:48.773053+0000 osd.2 (osd.2) 85 : cluster 0 purged_snaps scrub starts 2024-03-11T20:55:50.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:49 smithi146 bash[23809]: cluster 2024-03-11T20:55:48.773117+0000 osd.2 (osd.2) 86 : cluster 0 purged_snaps scrub ok 2024-03-11T20:55:50.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:49 smithi146 bash[23809]: cluster 2024-03-11T20:55:48.884697+0000 mon.a (mon.0) 1003 : cluster 0 osdmap e145: 8 total, 8 up, 7 in 2024-03-11T20:55:50.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:49 smithi146 bash[23809]: audit 2024-03-11T20:55:49.392782+0000 mon.a (mon.0) 1004 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:55:50.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:49 smithi146 bash[23809]: audit 2024-03-11T20:55:49.394661+0000 mon.a (mon.0) 1005 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:55:50.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:49 smithi146 bash[23809]: cluster 2024-03-11T20:55:49.716587+0000 osd.2 (osd.2) 87 : cluster 0 2.c scrub starts 2024-03-11T20:55:50.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:49 smithi146 bash[23809]: cluster 2024-03-11T20:55:49.723656+0000 osd.2 (osd.2) 88 : cluster 0 2.c scrub ok 2024-03-11T20:55:50.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:49 smithi023 bash[21372]: cluster 2024-03-11T20:55:48.773053+0000 osd.2 (osd.2) 85 : cluster 0 purged_snaps scrub starts 2024-03-11T20:55:50.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:49 smithi023 bash[21372]: cluster 2024-03-11T20:55:48.773117+0000 osd.2 (osd.2) 86 : cluster 0 purged_snaps scrub ok 2024-03-11T20:55:50.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:49 smithi023 bash[21372]: cluster 2024-03-11T20:55:48.884697+0000 mon.a (mon.0) 1003 : cluster 0 osdmap e145: 8 total, 8 up, 7 in 2024-03-11T20:55:50.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:49 smithi023 bash[21372]: audit 2024-03-11T20:55:49.392782+0000 mon.a (mon.0) 1004 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:55:50.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:49 smithi023 bash[21372]: audit 2024-03-11T20:55:49.394661+0000 mon.a (mon.0) 1005 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:55:50.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:49 smithi023 bash[21372]: cluster 2024-03-11T20:55:49.716587+0000 osd.2 (osd.2) 87 : cluster 0 2.c scrub starts 2024-03-11T20:55:50.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:49 smithi023 bash[21372]: cluster 2024-03-11T20:55:49.723656+0000 osd.2 (osd.2) 88 : cluster 0 2.c scrub ok 2024-03-11T20:55:50.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:49 smithi023 bash[29146]: cluster 2024-03-11T20:55:48.773053+0000 osd.2 (osd.2) 85 : cluster 0 purged_snaps scrub starts 2024-03-11T20:55:50.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:49 smithi023 bash[29146]: cluster 2024-03-11T20:55:48.773117+0000 osd.2 (osd.2) 86 : cluster 0 purged_snaps scrub ok 2024-03-11T20:55:50.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:49 smithi023 bash[29146]: cluster 2024-03-11T20:55:48.884697+0000 mon.a (mon.0) 1003 : cluster 0 osdmap e145: 8 total, 8 up, 7 in 2024-03-11T20:55:50.036 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:49 smithi023 bash[29146]: audit 2024-03-11T20:55:49.392782+0000 mon.a (mon.0) 1004 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:55:50.036 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:49 smithi023 bash[29146]: audit 2024-03-11T20:55:49.394661+0000 mon.a (mon.0) 1005 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:55:50.036 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:49 smithi023 bash[29146]: cluster 2024-03-11T20:55:49.716587+0000 osd.2 (osd.2) 87 : cluster 0 2.c scrub starts 2024-03-11T20:55:50.036 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:49 smithi023 bash[29146]: cluster 2024-03-11T20:55:49.723656+0000 osd.2 (osd.2) 88 : cluster 0 2.c scrub ok 2024-03-11T20:55:50.036 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:50.060 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:50.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:50.062 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:55:50.163 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:55:50.170 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd set noscrub 2024-03-11T20:55:50.186 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:55:50.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:50.190 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:55:50.291 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:50.312 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:50.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:50.314 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:55:50.415 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:50.442 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:50.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:50.447 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:55:50.546 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 59 16 83877 83861 88.8187 67.6875 0.00527943 0.0112536 2024-03-11T20:55:50.548 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:55:50.574 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:55:50.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:50.577 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:55:50.678 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:50.716 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:50.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:50.719 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:55:50.819 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:50.841 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:50.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:50.843 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:55:50.943 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:50.970 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:50.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:50.975 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:55:51.076 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:55:51.103 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:55:51.106 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:50 smithi023 bash[21372]: cluster 2024-03-11T20:55:49.268388+0000 osd.4 (osd.4) 121 : cluster 0 4.1b scrub starts 2024-03-11T20:55:51.107 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:50 smithi023 bash[21372]: cluster 2024-03-11T20:55:49.273381+0000 osd.4 (osd.4) 122 : cluster 0 4.1b scrub ok 2024-03-11T20:55:51.107 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:50 smithi023 bash[21372]: cluster 2024-03-11T20:55:49.924249+0000 mgr.y (mgr.24380) 192 : cluster 0 pgmap v175: 148 pgs: 148 active+clean; 5.0 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 639 B/s rd, 66 MiB/s wr, 1.06k op/s 2024-03-11T20:55:51.107 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:50 smithi023 bash[21372]: cluster 2024-03-11T20:55:50.616037+0000 osd.5 (osd.5) 105 : cluster 0 6.19 scrub starts 2024-03-11T20:55:51.107 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:50 smithi023 bash[21372]: cluster 2024-03-11T20:55:50.617937+0000 osd.5 (osd.5) 106 : cluster 0 6.19 scrub ok 2024-03-11T20:55:51.107 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:50 smithi023 bash[29146]: cluster 2024-03-11T20:55:49.268388+0000 osd.4 (osd.4) 121 : cluster 0 4.1b scrub starts 2024-03-11T20:55:51.107 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:50 smithi023 bash[29146]: cluster 2024-03-11T20:55:49.273381+0000 osd.4 (osd.4) 122 : cluster 0 4.1b scrub ok 2024-03-11T20:55:51.107 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:50 smithi023 bash[29146]: cluster 2024-03-11T20:55:49.924249+0000 mgr.y (mgr.24380) 192 : cluster 0 pgmap v175: 148 pgs: 148 active+clean; 5.0 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 639 B/s rd, 66 MiB/s wr, 1.06k op/s 2024-03-11T20:55:51.107 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:50 smithi023 bash[29146]: cluster 2024-03-11T20:55:50.616037+0000 osd.5 (osd.5) 105 : cluster 0 6.19 scrub starts 2024-03-11T20:55:51.107 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:50 smithi023 bash[29146]: cluster 2024-03-11T20:55:50.617937+0000 osd.5 (osd.5) 106 : cluster 0 6.19 scrub ok 2024-03-11T20:55:51.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:51.108 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:55:51.209 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:55:51.218 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:50 smithi146 bash[23809]: cluster 2024-03-11T20:55:49.268388+0000 osd.4 (osd.4) 121 : cluster 0 4.1b scrub starts 2024-03-11T20:55:51.218 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:50 smithi146 bash[23809]: cluster 2024-03-11T20:55:49.273381+0000 osd.4 (osd.4) 122 : cluster 0 4.1b scrub ok 2024-03-11T20:55:51.218 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:50 smithi146 bash[23809]: cluster 2024-03-11T20:55:49.924249+0000 mgr.y (mgr.24380) 192 : cluster 0 pgmap v175: 148 pgs: 148 active+clean; 5.0 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 639 B/s rd, 66 MiB/s wr, 1.06k op/s 2024-03-11T20:55:51.218 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:50 smithi146 bash[23809]: cluster 2024-03-11T20:55:50.616037+0000 osd.5 (osd.5) 105 : cluster 0 6.19 scrub starts 2024-03-11T20:55:51.218 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:50 smithi146 bash[23809]: cluster 2024-03-11T20:55:50.617937+0000 osd.5 (osd.5) 106 : cluster 0 6.19 scrub ok 2024-03-11T20:55:51.236 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:55:51.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:51.239 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:55:51.340 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:51.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:51.372 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:51.373 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:55:51.473 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:51.503 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:51.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:51.505 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:55:51.546 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:2024-03-11T20:55:51.550361+0000 min lat: 0.0012354 max lat: 0.370379 avg lat: 0.0113029 2024-03-11T20:55:51.546 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-11T20:55:51.546 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 60 16 84899 84883 88.4029 63.875 0.0790955 0.0113029 2024-03-11T20:55:51.605 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:55:51.639 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:55:51.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:51.645 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:55:51.746 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:51.777 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:51.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:51.781 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:55:51.882 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:55:51.907 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:51 smithi023 bash[21372]: cluster 2024-03-11T20:55:50.775557+0000 osd.2 (osd.2) 89 : cluster 0 7.8 deep-scrub starts 2024-03-11T20:55:51.907 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:51 smithi023 bash[29146]: cluster 2024-03-11T20:55:50.775557+0000 osd.2 (osd.2) 89 : cluster 0 7.8 deep-scrub starts 2024-03-11T20:55:51.908 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:55:51.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:51.911 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:55:52.012 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:55:52.034 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:55:52.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:52.038 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:55:52.139 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:55:52.174 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:55:52.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:52.176 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:55:52.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:51 smithi146 bash[23809]: cluster 2024-03-11T20:55:50.775557+0000 osd.2 (osd.2) 89 : cluster 0 7.8 deep-scrub starts 2024-03-11T20:55:52.277 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:52.304 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:52.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:52.309 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:55:52.409 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:52.440 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:52.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:52.444 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:55:52.545 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:52.546 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 61 16 85833 85817 87.9106 58.375 0.0257677 0.0113636 2024-03-11T20:55:52.564 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:52.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:52.566 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:55:52.667 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:52.685 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:52.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:52.687 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:55:52.789 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:55:52.809 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:55:52.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:52.811 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:55:52.912 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:55:52.936 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:55:52.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:52.941 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:55:53.042 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:55:53.069 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:55:53.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:53.072 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:55:53.173 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:55:53.183 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:52 smithi023 bash[21372]: cluster 2024-03-11T20:55:51.924838+0000 mgr.y (mgr.24380) 193 : cluster 0 pgmap v176: 148 pgs: 148 active+clean; 5.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 73 MiB/s wr, 1.17k op/s 2024-03-11T20:55:53.184 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:52 smithi023 bash[29146]: cluster 2024-03-11T20:55:51.924838+0000 mgr.y (mgr.24380) 193 : cluster 0 pgmap v176: 148 pgs: 148 active+clean; 5.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 73 MiB/s wr, 1.17k op/s 2024-03-11T20:55:53.205 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:55:53.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:53.217 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:55:53.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:52 smithi146 bash[23809]: cluster 2024-03-11T20:55:51.924838+0000 mgr.y (mgr.24380) 193 : cluster 0 pgmap v176: 148 pgs: 148 active+clean; 5.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 73 MiB/s wr, 1.17k op/s 2024-03-11T20:55:53.318 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:53.345 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:53.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:53.349 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:55:53.449 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:55:53.480 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:55:53.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:53.483 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:55:53.546 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 62 16 86461 86445 87.1258 39.25 0.00921201 0.0114711 2024-03-11T20:55:53.584 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:53.616 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:53.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:53.622 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:55:53.678 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:55:53.723 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:53.743 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:55:53.761 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:53.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:53.766 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:55:53.867 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:53.891 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:53.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:53.894 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:55:53.995 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:54.018 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:54.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:54.021 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:55:54.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:53 smithi023 bash[21372]: cluster 2024-03-11T20:55:52.844770+0000 osd.6 (osd.6) 75 : cluster 0 5.2 deep-scrub starts 2024-03-11T20:55:54.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:53 smithi023 bash[21372]: cluster 2024-03-11T20:55:52.848038+0000 osd.6 (osd.6) 76 : cluster 0 5.2 deep-scrub ok 2024-03-11T20:55:54.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:53 smithi023 bash[21372]: cluster 2024-03-11T20:55:52.911999+0000 osd.5 (osd.5) 107 : cluster 0 4.15 scrub starts 2024-03-11T20:55:54.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:53 smithi023 bash[21372]: cluster 2024-03-11T20:55:52.912113+0000 osd.3 (osd.3) 147 : cluster 0 2.a scrub starts 2024-03-11T20:55:54.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:53 smithi023 bash[21372]: cluster 2024-03-11T20:55:52.915207+0000 osd.2 (osd.2) 90 : cluster 0 2.1 scrub starts 2024-03-11T20:55:54.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:53 smithi023 bash[21372]: cluster 2024-03-11T20:55:52.917605+0000 osd.2 (osd.2) 91 : cluster 0 2.1 scrub ok 2024-03-11T20:55:54.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:53 smithi023 bash[21372]: cluster 2024-03-11T20:55:52.917895+0000 osd.3 (osd.3) 148 : cluster 0 2.a scrub ok 2024-03-11T20:55:54.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:53 smithi023 bash[21372]: cluster 2024-03-11T20:55:52.921431+0000 osd.5 (osd.5) 108 : cluster 0 4.15 scrub ok 2024-03-11T20:55:54.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:53 smithi023 bash[21372]: cluster 2024-03-11T20:55:53.052578+0000 osd.7 (osd.7) 121 : cluster 0 3.13 scrub starts 2024-03-11T20:55:54.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:53 smithi023 bash[21372]: cluster 2024-03-11T20:55:53.054930+0000 osd.7 (osd.7) 122 : cluster 0 3.13 scrub ok 2024-03-11T20:55:54.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:53 smithi023 bash[21372]: cluster 2024-03-11T20:55:53.540254+0000 osd.5 (osd.5) 109 : cluster 0 3.16 deep-scrub starts 2024-03-11T20:55:54.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:53 smithi023 bash[21372]: cluster 2024-03-11T20:55:53.560279+0000 osd.5 (osd.5) 110 : cluster 0 3.16 deep-scrub ok 2024-03-11T20:55:54.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:53 smithi023 bash[21372]: cluster 2024-03-11T20:55:53.811652+0000 osd.0 (osd.0) 127 : cluster 0 3.6 scrub starts 2024-03-11T20:55:54.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:53 smithi023 bash[21372]: cluster 2024-03-11T20:55:53.819769+0000 osd.0 (osd.0) 128 : cluster 0 3.6 scrub ok 2024-03-11T20:55:54.036 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:53 smithi023 bash[21372]: cluster 2024-03-11T20:55:53.878962+0000 osd.3 (osd.3) 149 : cluster 0 6.c scrub starts 2024-03-11T20:55:54.036 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:53 smithi023 bash[21372]: cluster 2024-03-11T20:55:53.880342+0000 osd.3 (osd.3) 150 : cluster 0 6.c scrub ok 2024-03-11T20:55:54.036 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:53 smithi023 bash[29146]: cluster 2024-03-11T20:55:52.844770+0000 osd.6 (osd.6) 75 : cluster 0 5.2 deep-scrub starts 2024-03-11T20:55:54.036 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:53 smithi023 bash[29146]: cluster 2024-03-11T20:55:52.848038+0000 osd.6 (osd.6) 76 : cluster 0 5.2 deep-scrub ok 2024-03-11T20:55:54.036 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:53 smithi023 bash[29146]: cluster 2024-03-11T20:55:52.911999+0000 osd.5 (osd.5) 107 : cluster 0 4.15 scrub starts 2024-03-11T20:55:54.036 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:53 smithi023 bash[29146]: cluster 2024-03-11T20:55:52.912113+0000 osd.3 (osd.3) 147 : cluster 0 2.a scrub starts 2024-03-11T20:55:54.036 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:53 smithi023 bash[29146]: cluster 2024-03-11T20:55:52.915207+0000 osd.2 (osd.2) 90 : cluster 0 2.1 scrub starts 2024-03-11T20:55:54.036 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:53 smithi023 bash[29146]: cluster 2024-03-11T20:55:52.917605+0000 osd.2 (osd.2) 91 : cluster 0 2.1 scrub ok 2024-03-11T20:55:54.036 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:53 smithi023 bash[29146]: cluster 2024-03-11T20:55:52.917895+0000 osd.3 (osd.3) 148 : cluster 0 2.a scrub ok 2024-03-11T20:55:54.036 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:53 smithi023 bash[29146]: cluster 2024-03-11T20:55:52.921431+0000 osd.5 (osd.5) 108 : cluster 0 4.15 scrub ok 2024-03-11T20:55:54.036 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:53 smithi023 bash[29146]: cluster 2024-03-11T20:55:53.052578+0000 osd.7 (osd.7) 121 : cluster 0 3.13 scrub starts 2024-03-11T20:55:54.037 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:53 smithi023 bash[29146]: cluster 2024-03-11T20:55:53.054930+0000 osd.7 (osd.7) 122 : cluster 0 3.13 scrub ok 2024-03-11T20:55:54.037 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:53 smithi023 bash[29146]: cluster 2024-03-11T20:55:53.540254+0000 osd.5 (osd.5) 109 : cluster 0 3.16 deep-scrub starts 2024-03-11T20:55:54.037 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:53 smithi023 bash[29146]: cluster 2024-03-11T20:55:53.560279+0000 osd.5 (osd.5) 110 : cluster 0 3.16 deep-scrub ok 2024-03-11T20:55:54.037 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:53 smithi023 bash[29146]: cluster 2024-03-11T20:55:53.811652+0000 osd.0 (osd.0) 127 : cluster 0 3.6 scrub starts 2024-03-11T20:55:54.037 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:53 smithi023 bash[29146]: cluster 2024-03-11T20:55:53.819769+0000 osd.0 (osd.0) 128 : cluster 0 3.6 scrub ok 2024-03-11T20:55:54.037 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:53 smithi023 bash[29146]: cluster 2024-03-11T20:55:53.878962+0000 osd.3 (osd.3) 149 : cluster 0 6.c scrub starts 2024-03-11T20:55:54.037 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:53 smithi023 bash[29146]: cluster 2024-03-11T20:55:53.880342+0000 osd.3 (osd.3) 150 : cluster 0 6.c scrub ok 2024-03-11T20:55:54.122 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:55:54.132 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:53 smithi146 bash[23809]: cluster 2024-03-11T20:55:52.844770+0000 osd.6 (osd.6) 75 : cluster 0 5.2 deep-scrub starts 2024-03-11T20:55:54.133 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:53 smithi146 bash[23809]: cluster 2024-03-11T20:55:52.848038+0000 osd.6 (osd.6) 76 : cluster 0 5.2 deep-scrub ok 2024-03-11T20:55:54.133 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:53 smithi146 bash[23809]: cluster 2024-03-11T20:55:52.911999+0000 osd.5 (osd.5) 107 : cluster 0 4.15 scrub starts 2024-03-11T20:55:54.133 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:53 smithi146 bash[23809]: cluster 2024-03-11T20:55:52.912113+0000 osd.3 (osd.3) 147 : cluster 0 2.a scrub starts 2024-03-11T20:55:54.133 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:53 smithi146 bash[23809]: cluster 2024-03-11T20:55:52.915207+0000 osd.2 (osd.2) 90 : cluster 0 2.1 scrub starts 2024-03-11T20:55:54.133 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:53 smithi146 bash[23809]: cluster 2024-03-11T20:55:52.917605+0000 osd.2 (osd.2) 91 : cluster 0 2.1 scrub ok 2024-03-11T20:55:54.133 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:53 smithi146 bash[23809]: cluster 2024-03-11T20:55:52.917895+0000 osd.3 (osd.3) 148 : cluster 0 2.a scrub ok 2024-03-11T20:55:54.133 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:53 smithi146 bash[23809]: cluster 2024-03-11T20:55:52.921431+0000 osd.5 (osd.5) 108 : cluster 0 4.15 scrub ok 2024-03-11T20:55:54.133 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:53 smithi146 bash[23809]: cluster 2024-03-11T20:55:53.052578+0000 osd.7 (osd.7) 121 : cluster 0 3.13 scrub starts 2024-03-11T20:55:54.133 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:53 smithi146 bash[23809]: cluster 2024-03-11T20:55:53.054930+0000 osd.7 (osd.7) 122 : cluster 0 3.13 scrub ok 2024-03-11T20:55:54.133 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:53 smithi146 bash[23809]: cluster 2024-03-11T20:55:53.540254+0000 osd.5 (osd.5) 109 : cluster 0 3.16 deep-scrub starts 2024-03-11T20:55:54.133 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:53 smithi146 bash[23809]: cluster 2024-03-11T20:55:53.560279+0000 osd.5 (osd.5) 110 : cluster 0 3.16 deep-scrub ok 2024-03-11T20:55:54.133 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:53 smithi146 bash[23809]: cluster 2024-03-11T20:55:53.811652+0000 osd.0 (osd.0) 127 : cluster 0 3.6 scrub starts 2024-03-11T20:55:54.133 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:53 smithi146 bash[23809]: cluster 2024-03-11T20:55:53.819769+0000 osd.0 (osd.0) 128 : cluster 0 3.6 scrub ok 2024-03-11T20:55:54.133 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:53 smithi146 bash[23809]: cluster 2024-03-11T20:55:53.878962+0000 osd.3 (osd.3) 149 : cluster 0 6.c scrub starts 2024-03-11T20:55:54.134 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:53 smithi146 bash[23809]: cluster 2024-03-11T20:55:53.880342+0000 osd.3 (osd.3) 150 : cluster 0 6.c scrub ok 2024-03-11T20:55:54.158 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:55:54.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:54.161 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:55:54.262 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:54.286 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:54.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:54.288 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:55:54.389 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:55:54.411 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:55:54.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:54.413 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:55:54.514 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:55:54.546 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:55:54.546 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 63 16 87368 87352 86.6425 56.6875 0.011259 0.0115322 2024-03-11T20:55:54.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:54.549 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:55:54.650 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:55:54.681 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:55:54.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:54.683 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:55:54.784 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:55:54.817 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:55:54.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:54.819 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:55:54.920 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:54.947 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:54.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:54.950 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:55:55.051 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:55:55.076 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:55:55.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:55.079 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:55:55.180 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:55.207 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:55.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:55.211 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:55:55.222 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:54 smithi023 bash[21372]: cluster 2024-03-11T20:55:53.925981+0000 mgr.y (mgr.24380) 194 : cluster 0 pgmap v177: 148 pgs: 148 active+clean; 5.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 639 B/s rd, 39 MiB/s wr, 621 op/s 2024-03-11T20:55:55.222 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:54 smithi023 bash[29146]: cluster 2024-03-11T20:55:53.925981+0000 mgr.y (mgr.24380) 194 : cluster 0 pgmap v177: 148 pgs: 148 active+clean; 5.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 639 B/s rd, 39 MiB/s wr, 621 op/s 2024-03-11T20:55:55.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:54 smithi146 bash[23809]: cluster 2024-03-11T20:55:53.925981+0000 mgr.y (mgr.24380) 194 : cluster 0 pgmap v177: 148 pgs: 148 active+clean; 5.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 639 B/s rd, 39 MiB/s wr, 621 op/s 2024-03-11T20:55:55.261 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 20:55:55 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T20:55:55.312 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:55.329 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:55.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:55.331 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:55:55.432 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:55.467 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:55.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:55.470 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:55:55.546 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 64 16 88222 88206 86.1227 53.375 0.0124664 0.0116049 2024-03-11T20:55:55.571 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:55.597 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:55.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:55.600 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:55:55.701 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:55:55.725 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:55:55.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:55.729 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:55:55.771 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:55:55.829 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:55:55.847 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:55:55.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:55.849 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:55:55.950 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:55.983 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:55.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:55.987 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:55:56.088 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:55:56.096 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:55 smithi146 bash[23809]: cluster 2024-03-11T20:55:54.223389+0000 osd.4 (osd.4) 123 : cluster 0 6.1e scrub starts 2024-03-11T20:55:56.097 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:55 smithi146 bash[23809]: cluster 2024-03-11T20:55:54.224369+0000 osd.4 (osd.4) 124 : cluster 0 6.1e scrub ok 2024-03-11T20:55:56.097 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:55 smithi146 bash[23809]: cluster 2024-03-11T20:55:54.748811+0000 osd.6 (osd.6) 77 : cluster 0 purged_snaps scrub starts 2024-03-11T20:55:56.097 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:55 smithi146 bash[23809]: cluster 2024-03-11T20:55:54.748894+0000 osd.6 (osd.6) 78 : cluster 0 purged_snaps scrub ok 2024-03-11T20:55:56.097 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:55 smithi146 bash[23809]: cluster 2024-03-11T20:55:55.047420+0000 osd.7 (osd.7) 123 : cluster 0 3.11 scrub starts 2024-03-11T20:55:56.097 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:55 smithi146 bash[23809]: audit 2024-03-11T20:55:55.050405+0000 mgr.y (mgr.24380) 195 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:55:56.097 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:55 smithi146 bash[23809]: cluster 2024-03-11T20:55:55.052054+0000 osd.7 (osd.7) 124 : cluster 0 3.11 scrub ok 2024-03-11T20:55:56.097 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:55 smithi146 bash[23809]: cluster 2024-03-11T20:55:55.686298+0000 mon.a (mon.0) 1006 : cluster 0 osdmap e146: 8 total, 8 up, 7 in 2024-03-11T20:55:56.111 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:55:56.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:56.114 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:55:56.214 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:56.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:56.236 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:56.237 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:55:56.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:55 smithi023 bash[29146]: cluster 2024-03-11T20:55:54.223389+0000 osd.4 (osd.4) 123 : cluster 0 6.1e scrub starts 2024-03-11T20:55:56.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:55 smithi023 bash[29146]: cluster 2024-03-11T20:55:54.224369+0000 osd.4 (osd.4) 124 : cluster 0 6.1e scrub ok 2024-03-11T20:55:56.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:55 smithi023 bash[29146]: cluster 2024-03-11T20:55:54.748811+0000 osd.6 (osd.6) 77 : cluster 0 purged_snaps scrub starts 2024-03-11T20:55:56.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:55 smithi023 bash[29146]: cluster 2024-03-11T20:55:54.748894+0000 osd.6 (osd.6) 78 : cluster 0 purged_snaps scrub ok 2024-03-11T20:55:56.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:55 smithi023 bash[29146]: cluster 2024-03-11T20:55:55.047420+0000 osd.7 (osd.7) 123 : cluster 0 3.11 scrub starts 2024-03-11T20:55:56.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:55 smithi023 bash[29146]: audit 2024-03-11T20:55:55.050405+0000 mgr.y (mgr.24380) 195 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:55:56.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:55 smithi023 bash[29146]: cluster 2024-03-11T20:55:55.052054+0000 osd.7 (osd.7) 124 : cluster 0 3.11 scrub ok 2024-03-11T20:55:56.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:55 smithi023 bash[29146]: cluster 2024-03-11T20:55:55.686298+0000 mon.a (mon.0) 1006 : cluster 0 osdmap e146: 8 total, 8 up, 7 in 2024-03-11T20:55:56.286 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:55 smithi023 bash[21372]: cluster 2024-03-11T20:55:54.223389+0000 osd.4 (osd.4) 123 : cluster 0 6.1e scrub starts 2024-03-11T20:55:56.286 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:55 smithi023 bash[21372]: cluster 2024-03-11T20:55:54.224369+0000 osd.4 (osd.4) 124 : cluster 0 6.1e scrub ok 2024-03-11T20:55:56.286 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:55 smithi023 bash[21372]: cluster 2024-03-11T20:55:54.748811+0000 osd.6 (osd.6) 77 : cluster 0 purged_snaps scrub starts 2024-03-11T20:55:56.286 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:55 smithi023 bash[21372]: cluster 2024-03-11T20:55:54.748894+0000 osd.6 (osd.6) 78 : cluster 0 purged_snaps scrub ok 2024-03-11T20:55:56.286 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:55 smithi023 bash[21372]: cluster 2024-03-11T20:55:55.047420+0000 osd.7 (osd.7) 123 : cluster 0 3.11 scrub starts 2024-03-11T20:55:56.286 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:55 smithi023 bash[21372]: audit 2024-03-11T20:55:55.050405+0000 mgr.y (mgr.24380) 195 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:55:56.286 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:55 smithi023 bash[21372]: cluster 2024-03-11T20:55:55.052054+0000 osd.7 (osd.7) 124 : cluster 0 3.11 scrub ok 2024-03-11T20:55:56.286 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:55 smithi023 bash[21372]: cluster 2024-03-11T20:55:55.686298+0000 mon.a (mon.0) 1006 : cluster 0 osdmap e146: 8 total, 8 up, 7 in 2024-03-11T20:55:56.338 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:56.363 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:56.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:56.365 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:55:56.466 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:55:56.494 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:55:56.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:56.497 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:55:56.546 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 65 16 88956 88940 85.5035 45.875 0.015381 0.0116895 2024-03-11T20:55:56.598 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:56.621 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:56.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:56.623 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:55:56.724 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:55:56.750 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:55:56.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:56.752 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:55:56.853 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:56.879 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:56.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:56.882 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:55:56.982 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:55:57.006 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:55:57.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:57.011 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:55:57.112 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:55:57.133 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:55:57.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:57.135 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:55:57.236 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:57.246 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:56 smithi146 bash[23809]: cluster 2024-03-11T20:55:55.926582+0000 mgr.y (mgr.24380) 196 : cluster 0 pgmap v179: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.4 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 74 MiB/s wr, 1.19k op/s 2024-03-11T20:55:57.246 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:56 smithi146 bash[23809]: cluster 2024-03-11T20:55:55.980417+0000 osd.2 (osd.2) 92 : cluster 0 7.e scrub starts 2024-03-11T20:55:57.246 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:56 smithi146 bash[23809]: cluster 2024-03-11T20:55:56.809383+0000 osd.0 (osd.0) 129 : cluster 0 2.1f scrub starts 2024-03-11T20:55:57.246 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:56 smithi146 bash[23809]: cluster 2024-03-11T20:55:56.810711+0000 osd.0 (osd.0) 130 : cluster 0 2.1f scrub ok 2024-03-11T20:55:57.259 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:57.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:57.264 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:55:57.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:56 smithi023 bash[21372]: cluster 2024-03-11T20:55:55.926582+0000 mgr.y (mgr.24380) 196 : cluster 0 pgmap v179: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.4 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 74 MiB/s wr, 1.19k op/s 2024-03-11T20:55:57.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:56 smithi023 bash[21372]: cluster 2024-03-11T20:55:55.980417+0000 osd.2 (osd.2) 92 : cluster 0 7.e scrub starts 2024-03-11T20:55:57.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:56 smithi023 bash[21372]: cluster 2024-03-11T20:55:56.809383+0000 osd.0 (osd.0) 129 : cluster 0 2.1f scrub starts 2024-03-11T20:55:57.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:56 smithi023 bash[21372]: cluster 2024-03-11T20:55:56.810711+0000 osd.0 (osd.0) 130 : cluster 0 2.1f scrub ok 2024-03-11T20:55:57.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:56 smithi023 bash[29146]: cluster 2024-03-11T20:55:55.926582+0000 mgr.y (mgr.24380) 196 : cluster 0 pgmap v179: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.4 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 74 MiB/s wr, 1.19k op/s 2024-03-11T20:55:57.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:56 smithi023 bash[29146]: cluster 2024-03-11T20:55:55.980417+0000 osd.2 (osd.2) 92 : cluster 0 7.e scrub starts 2024-03-11T20:55:57.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:56 smithi023 bash[29146]: cluster 2024-03-11T20:55:56.809383+0000 osd.0 (osd.0) 129 : cluster 0 2.1f scrub starts 2024-03-11T20:55:57.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:56 smithi023 bash[29146]: cluster 2024-03-11T20:55:56.810711+0000 osd.0 (osd.0) 130 : cluster 0 2.1f scrub ok 2024-03-11T20:55:57.364 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:57.388 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:57.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:57.392 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:55:57.493 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:55:57.516 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:55:57.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:57.518 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:55:57.546 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 66 16 89764 89748 84.9731 50.5 0.00918788 0.0117616 2024-03-11T20:55:57.619 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:57.639 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-11T20:55:57.639 INFO:teuthology.orchestra.run.smithi023.stdout: "ops": [], 2024-03-11T20:55:57.639 INFO:teuthology.orchestra.run.smithi023.stdout: "complaint_time": 30, 2024-03-11T20:55:57.639 INFO:teuthology.orchestra.run.smithi023.stdout: "num_blocked_ops": 0 2024-03-11T20:55:57.639 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-11T20:55:57.657 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:57.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:57.662 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:55:57.762 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:57.801 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:55:57.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:57.804 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:55:57.906 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:57.926 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:57.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:57.930 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:55:58.031 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:55:58.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:57 smithi023 bash[21372]: cluster 2024-03-11T20:55:57.843771+0000 osd.0 (osd.0) 131 : cluster 0 6.16 scrub starts 2024-03-11T20:55:58.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:57 smithi023 bash[21372]: cluster 2024-03-11T20:55:57.849054+0000 osd.0 (osd.0) 132 : cluster 0 6.16 scrub ok 2024-03-11T20:55:58.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:57 smithi023 bash[29146]: cluster 2024-03-11T20:55:57.843771+0000 osd.0 (osd.0) 131 : cluster 0 6.16 scrub starts 2024-03-11T20:55:58.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:57 smithi023 bash[29146]: cluster 2024-03-11T20:55:57.849054+0000 osd.0 (osd.0) 132 : cluster 0 6.16 scrub ok 2024-03-11T20:55:58.055 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:55:58.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:58.063 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:55:58.164 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:58.173 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:57 smithi146 bash[23809]: cluster 2024-03-11T20:55:57.843771+0000 osd.0 (osd.0) 131 : cluster 0 6.16 scrub starts 2024-03-11T20:55:58.174 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:57 smithi146 bash[23809]: cluster 2024-03-11T20:55:57.849054+0000 osd.0 (osd.0) 132 : cluster 0 6.16 scrub ok 2024-03-11T20:55:58.186 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:58.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:58.188 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:55:58.289 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:55:58.320 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:55:58.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:58.325 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:55:58.426 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:58.452 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:58.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:58.456 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:55:58.546 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 67 16 90708 90692 84.5854 59 0.0599753 0.011815 2024-03-11T20:55:58.557 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:55:58.583 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:55:58.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:58.587 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:55:58.688 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:55:58.689 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:55:58 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:20:55:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T20:55:58.717 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:55:58.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:58.720 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:55:58.821 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:58.859 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:58.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:58.862 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:55:58.962 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:55:58.991 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:55:58.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:58.999 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:55:59.100 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:59.118 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:55:59.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:59.120 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:55:59.201 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:58 smithi023 bash[21372]: cluster 2024-03-11T20:55:57.927058+0000 mgr.y (mgr.24380) 197 : cluster 0 pgmap v180: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.4 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 1.1 KiB/s rd, 66 MiB/s wr, 1.05k op/s 2024-03-11T20:55:59.201 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:58 smithi023 bash[21372]: cluster 2024-03-11T20:55:57.972322+0000 osd.2 (osd.2) 93 : cluster 0 7.e scrub ok 2024-03-11T20:55:59.201 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:58 smithi023 bash[21372]: cluster 2024-03-11T20:55:58.013411+0000 osd.7 (osd.7) 125 : cluster 0 6.15 deep-scrub starts 2024-03-11T20:55:59.201 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:58 smithi023 bash[21372]: cluster 2024-03-11T20:55:58.016978+0000 osd.7 (osd.7) 126 : cluster 0 6.15 deep-scrub ok 2024-03-11T20:55:59.201 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:58 smithi023 bash[21372]: cluster 2024-03-11T20:55:58.543098+0000 osd.5 (osd.5) 111 : cluster 0 6.4 scrub starts 2024-03-11T20:55:59.201 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:58 smithi023 bash[21372]: cluster 2024-03-11T20:55:58.547783+0000 osd.5 (osd.5) 112 : cluster 0 6.4 scrub ok 2024-03-11T20:55:59.202 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:58 smithi023 bash[29146]: cluster 2024-03-11T20:55:57.927058+0000 mgr.y (mgr.24380) 197 : cluster 0 pgmap v180: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.4 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 1.1 KiB/s rd, 66 MiB/s wr, 1.05k op/s 2024-03-11T20:55:59.202 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:58 smithi023 bash[29146]: cluster 2024-03-11T20:55:57.972322+0000 osd.2 (osd.2) 93 : cluster 0 7.e scrub ok 2024-03-11T20:55:59.202 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:58 smithi023 bash[29146]: cluster 2024-03-11T20:55:58.013411+0000 osd.7 (osd.7) 125 : cluster 0 6.15 deep-scrub starts 2024-03-11T20:55:59.203 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:58 smithi023 bash[29146]: cluster 2024-03-11T20:55:58.016978+0000 osd.7 (osd.7) 126 : cluster 0 6.15 deep-scrub ok 2024-03-11T20:55:59.203 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:58 smithi023 bash[29146]: cluster 2024-03-11T20:55:58.543098+0000 osd.5 (osd.5) 111 : cluster 0 6.4 scrub starts 2024-03-11T20:55:59.203 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:58 smithi023 bash[29146]: cluster 2024-03-11T20:55:58.547783+0000 osd.5 (osd.5) 112 : cluster 0 6.4 scrub ok 2024-03-11T20:55:59.220 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:59.231 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:58 smithi146 bash[23809]: cluster 2024-03-11T20:55:57.927058+0000 mgr.y (mgr.24380) 197 : cluster 0 pgmap v180: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.4 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 1.1 KiB/s rd, 66 MiB/s wr, 1.05k op/s 2024-03-11T20:55:59.232 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:58 smithi146 bash[23809]: cluster 2024-03-11T20:55:57.972322+0000 osd.2 (osd.2) 93 : cluster 0 7.e scrub ok 2024-03-11T20:55:59.232 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:58 smithi146 bash[23809]: cluster 2024-03-11T20:55:58.013411+0000 osd.7 (osd.7) 125 : cluster 0 6.15 deep-scrub starts 2024-03-11T20:55:59.232 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:58 smithi146 bash[23809]: cluster 2024-03-11T20:55:58.016978+0000 osd.7 (osd.7) 126 : cluster 0 6.15 deep-scrub ok 2024-03-11T20:55:59.232 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:58 smithi146 bash[23809]: cluster 2024-03-11T20:55:58.543098+0000 osd.5 (osd.5) 111 : cluster 0 6.4 scrub starts 2024-03-11T20:55:59.232 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:58 smithi146 bash[23809]: cluster 2024-03-11T20:55:58.547783+0000 osd.5 (osd.5) 112 : cluster 0 6.4 scrub ok 2024-03-11T20:55:59.252 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:55:59.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:59.257 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:55:59.357 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:59.387 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:59.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:59.394 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:55:59.495 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:59.520 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:59.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:59.522 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:55:59.546 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 68 16 91578 91562 84.1411 54.375 0.00969441 0.0118788 2024-03-11T20:55:59.623 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:59.647 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:55:59.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:59.652 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:55:59.753 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:55:59.771 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:55:59.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:59.772 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:55:59.873 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:55:59.903 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:55:59.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:55:59.910 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:56:00.011 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:56:00.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:59 smithi146 bash[23809]: cluster 2024-03-11T20:55:58.884443+0000 osd.0 (osd.0) 133 : cluster 0 6.9 deep-scrub starts 2024-03-11T20:56:00.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:59 smithi146 bash[23809]: cluster 2024-03-11T20:55:58.888883+0000 osd.0 (osd.0) 134 : cluster 0 6.9 deep-scrub ok 2024-03-11T20:56:00.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:59 smithi146 bash[23809]: cluster 2024-03-11T20:55:59.012572+0000 osd.7 (osd.7) 127 : cluster 0 2.1c scrub starts 2024-03-11T20:56:00.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:59 smithi146 bash[23809]: cluster 2024-03-11T20:55:59.021258+0000 osd.7 (osd.7) 128 : cluster 0 2.1c scrub ok 2024-03-11T20:56:00.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:59 smithi146 bash[23809]: cluster 2024-03-11T20:55:59.046128+0000 osd.2 (osd.2) 94 : cluster 0 2.10 scrub starts 2024-03-11T20:56:00.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:59 smithi146 bash[23809]: cluster 2024-03-11T20:55:59.209824+0000 osd.2 (osd.2) 95 : cluster 0 2.10 scrub ok 2024-03-11T20:56:00.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:59 smithi146 bash[23809]: cluster 2024-03-11T20:55:59.884627+0000 osd.3 (osd.3) 151 : cluster 0 6.1f deep-scrub starts 2024-03-11T20:56:00.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:55:59 smithi146 bash[23809]: cluster 2024-03-11T20:55:59.892386+0000 osd.3 (osd.3) 152 : cluster 0 6.1f deep-scrub ok 2024-03-11T20:56:00.028 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-03-11T20:56:00.032 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:56:00.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:00.037 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:56:00.137 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:56:00.163 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:56:00.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:00.166 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:56:00.267 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:00.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:59 smithi023 bash[21372]: cluster 2024-03-11T20:55:58.884443+0000 osd.0 (osd.0) 133 : cluster 0 6.9 deep-scrub starts 2024-03-11T20:56:00.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:59 smithi023 bash[21372]: cluster 2024-03-11T20:55:58.888883+0000 osd.0 (osd.0) 134 : cluster 0 6.9 deep-scrub ok 2024-03-11T20:56:00.286 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:59 smithi023 bash[21372]: cluster 2024-03-11T20:55:59.012572+0000 osd.7 (osd.7) 127 : cluster 0 2.1c scrub starts 2024-03-11T20:56:00.286 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:59 smithi023 bash[21372]: cluster 2024-03-11T20:55:59.021258+0000 osd.7 (osd.7) 128 : cluster 0 2.1c scrub ok 2024-03-11T20:56:00.286 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:59 smithi023 bash[21372]: cluster 2024-03-11T20:55:59.046128+0000 osd.2 (osd.2) 94 : cluster 0 2.10 scrub starts 2024-03-11T20:56:00.286 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:59 smithi023 bash[21372]: cluster 2024-03-11T20:55:59.209824+0000 osd.2 (osd.2) 95 : cluster 0 2.10 scrub ok 2024-03-11T20:56:00.286 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:59 smithi023 bash[21372]: cluster 2024-03-11T20:55:59.884627+0000 osd.3 (osd.3) 151 : cluster 0 6.1f deep-scrub starts 2024-03-11T20:56:00.286 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:55:59 smithi023 bash[21372]: cluster 2024-03-11T20:55:59.892386+0000 osd.3 (osd.3) 152 : cluster 0 6.1f deep-scrub ok 2024-03-11T20:56:00.286 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:59 smithi023 bash[29146]: cluster 2024-03-11T20:55:58.884443+0000 osd.0 (osd.0) 133 : cluster 0 6.9 deep-scrub starts 2024-03-11T20:56:00.286 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:59 smithi023 bash[29146]: cluster 2024-03-11T20:55:58.888883+0000 osd.0 (osd.0) 134 : cluster 0 6.9 deep-scrub ok 2024-03-11T20:56:00.286 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:59 smithi023 bash[29146]: cluster 2024-03-11T20:55:59.012572+0000 osd.7 (osd.7) 127 : cluster 0 2.1c scrub starts 2024-03-11T20:56:00.286 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:59 smithi023 bash[29146]: cluster 2024-03-11T20:55:59.021258+0000 osd.7 (osd.7) 128 : cluster 0 2.1c scrub ok 2024-03-11T20:56:00.286 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:59 smithi023 bash[29146]: cluster 2024-03-11T20:55:59.046128+0000 osd.2 (osd.2) 94 : cluster 0 2.10 scrub starts 2024-03-11T20:56:00.287 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:59 smithi023 bash[29146]: cluster 2024-03-11T20:55:59.209824+0000 osd.2 (osd.2) 95 : cluster 0 2.10 scrub ok 2024-03-11T20:56:00.287 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:59 smithi023 bash[29146]: cluster 2024-03-11T20:55:59.884627+0000 osd.3 (osd.3) 151 : cluster 0 6.1f deep-scrub starts 2024-03-11T20:56:00.287 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:55:59 smithi023 bash[29146]: cluster 2024-03-11T20:55:59.892386+0000 osd.3 (osd.3) 152 : cluster 0 6.1f deep-scrub ok 2024-03-11T20:56:00.297 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:00.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:00.303 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:56:00.404 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:00.432 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:00.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:00.437 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:56:00.538 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:56:00.547 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 69 16 92408 92392 83.6735 51.875 0.0167658 0.0119456 2024-03-11T20:56:00.572 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:56:00.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:00.574 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:56:00.675 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:56:00.703 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:56:00.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:00.705 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:56:00.805 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:00.826 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:00.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:00.829 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:56:00.930 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:56:00.950 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:56:00.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:00.951 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:56:01.054 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:56:01.082 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:56:01.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:01.085 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:56:01.186 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:56:01.195 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:01 smithi023 bash[21372]: cluster 2024-03-11T20:55:59.927897+0000 mgr.y (mgr.24380) 198 : cluster 0 pgmap v181: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.5 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 52 MiB/s wr, 830 op/s 2024-03-11T20:56:01.195 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:01 smithi023 bash[21372]: cluster 2024-03-11T20:56:00.236015+0000 osd.4 (osd.4) 125 : cluster 0 4.d deep-scrub starts 2024-03-11T20:56:01.195 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:01 smithi023 bash[21372]: cluster 2024-03-11T20:56:00.242205+0000 osd.4 (osd.4) 126 : cluster 0 4.d deep-scrub ok 2024-03-11T20:56:01.195 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:01 smithi023 bash[21372]: cluster 2024-03-11T20:56:00.525195+0000 osd.6 (osd.6) 79 : cluster 0 2.7 scrub starts 2024-03-11T20:56:01.195 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:01 smithi023 bash[21372]: cluster 2024-03-11T20:56:00.526096+0000 osd.6 (osd.6) 80 : cluster 0 2.7 scrub ok 2024-03-11T20:56:01.195 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:01 smithi023 bash[29146]: cluster 2024-03-11T20:55:59.927897+0000 mgr.y (mgr.24380) 198 : cluster 0 pgmap v181: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.5 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 52 MiB/s wr, 830 op/s 2024-03-11T20:56:01.196 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:01 smithi023 bash[29146]: cluster 2024-03-11T20:56:00.236015+0000 osd.4 (osd.4) 125 : cluster 0 4.d deep-scrub starts 2024-03-11T20:56:01.196 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:01 smithi023 bash[29146]: cluster 2024-03-11T20:56:00.242205+0000 osd.4 (osd.4) 126 : cluster 0 4.d deep-scrub ok 2024-03-11T20:56:01.196 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:01 smithi023 bash[29146]: cluster 2024-03-11T20:56:00.525195+0000 osd.6 (osd.6) 79 : cluster 0 2.7 scrub starts 2024-03-11T20:56:01.196 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:01 smithi023 bash[29146]: cluster 2024-03-11T20:56:00.526096+0000 osd.6 (osd.6) 80 : cluster 0 2.7 scrub ok 2024-03-11T20:56:01.216 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:56:01.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:01.220 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:56:01.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:00 smithi146 bash[23809]: cluster 2024-03-11T20:55:59.927897+0000 mgr.y (mgr.24380) 198 : cluster 0 pgmap v181: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.5 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 52 MiB/s wr, 830 op/s 2024-03-11T20:56:01.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:00 smithi146 bash[23809]: cluster 2024-03-11T20:56:00.236015+0000 osd.4 (osd.4) 125 : cluster 0 4.d deep-scrub starts 2024-03-11T20:56:01.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:00 smithi146 bash[23809]: cluster 2024-03-11T20:56:00.242205+0000 osd.4 (osd.4) 126 : cluster 0 4.d deep-scrub ok 2024-03-11T20:56:01.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:00 smithi146 bash[23809]: cluster 2024-03-11T20:56:00.525195+0000 osd.6 (osd.6) 79 : cluster 0 2.7 scrub starts 2024-03-11T20:56:01.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:00 smithi146 bash[23809]: cluster 2024-03-11T20:56:00.526096+0000 osd.6 (osd.6) 80 : cluster 0 2.7 scrub ok 2024-03-11T20:56:01.321 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:01.347 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:01.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:01.350 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:56:01.451 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:56:01.477 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:56:01.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:01.481 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:56:01.547 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 70 16 93153 93137 83.1433 46.5625 0.0094854 0.0120216 2024-03-11T20:56:01.582 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:56:01.601 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:56:01.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:01.603 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:56:01.704 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:01.734 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:01.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:01.738 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:56:01.839 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:01.868 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:01.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:01.870 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:56:01.971 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:02.001 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:02.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:02.008 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:56:02.109 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:56:02.144 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:56:02.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:02.149 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:56:02.250 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:02.260 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:02 smithi023 bash[21372]: cluster 2024-03-11T20:56:00.974561+0000 osd.7 (osd.7) 129 : cluster 0 3.15 scrub starts 2024-03-11T20:56:02.260 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:02 smithi023 bash[21372]: cluster 2024-03-11T20:56:00.976017+0000 osd.7 (osd.7) 130 : cluster 0 3.15 scrub ok 2024-03-11T20:56:02.260 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:02 smithi023 bash[21372]: cluster 2024-03-11T20:56:01.515405+0000 osd.5 (osd.5) 113 : cluster 0 3.5 scrub starts 2024-03-11T20:56:02.260 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:02 smithi023 bash[21372]: cluster 2024-03-11T20:56:01.518056+0000 osd.5 (osd.5) 114 : cluster 0 3.5 scrub ok 2024-03-11T20:56:02.260 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:02 smithi023 bash[29146]: cluster 2024-03-11T20:56:00.974561+0000 osd.7 (osd.7) 129 : cluster 0 3.15 scrub starts 2024-03-11T20:56:02.260 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:02 smithi023 bash[29146]: cluster 2024-03-11T20:56:00.976017+0000 osd.7 (osd.7) 130 : cluster 0 3.15 scrub ok 2024-03-11T20:56:02.260 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:02 smithi023 bash[29146]: cluster 2024-03-11T20:56:01.515405+0000 osd.5 (osd.5) 113 : cluster 0 3.5 scrub starts 2024-03-11T20:56:02.260 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:02 smithi023 bash[29146]: cluster 2024-03-11T20:56:01.518056+0000 osd.5 (osd.5) 114 : cluster 0 3.5 scrub ok 2024-03-11T20:56:02.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:01 smithi146 bash[23809]: cluster 2024-03-11T20:56:00.974561+0000 osd.7 (osd.7) 129 : cluster 0 3.15 scrub starts 2024-03-11T20:56:02.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:01 smithi146 bash[23809]: cluster 2024-03-11T20:56:00.976017+0000 osd.7 (osd.7) 130 : cluster 0 3.15 scrub ok 2024-03-11T20:56:02.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:01 smithi146 bash[23809]: cluster 2024-03-11T20:56:01.515405+0000 osd.5 (osd.5) 113 : cluster 0 3.5 scrub starts 2024-03-11T20:56:02.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:01 smithi146 bash[23809]: cluster 2024-03-11T20:56:01.518056+0000 osd.5 (osd.5) 114 : cluster 0 3.5 scrub ok 2024-03-11T20:56:02.274 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:02.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:02.276 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:56:02.377 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:02.437 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:02.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:02.444 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:56:02.544 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:02.547 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 71 16 93938 93922 82.6633 49.0625 0.030792 0.0120848 2024-03-11T20:56:02.566 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:02.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:02.569 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:56:02.670 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:56:02.696 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:56:02.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:02.698 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:56:02.800 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:02.828 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:02.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:02.831 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:56:02.932 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:56:02.962 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:56:02.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:02.971 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:56:03.072 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:56:03.082 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:03 smithi023 bash[29146]: cluster 2024-03-11T20:56:01.928742+0000 mgr.y (mgr.24380) 199 : cluster 0 pgmap v182: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.6 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 54 MiB/s wr, 867 op/s 2024-03-11T20:56:03.082 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:03 smithi023 bash[29146]: cluster 2024-03-11T20:56:01.944894+0000 osd.3 (osd.3) 153 : cluster 0 3.2 scrub starts 2024-03-11T20:56:03.082 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:03 smithi023 bash[29146]: cluster 2024-03-11T20:56:01.961991+0000 osd.3 (osd.3) 154 : cluster 0 3.2 scrub ok 2024-03-11T20:56:03.082 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:03 smithi023 bash[21372]: cluster 2024-03-11T20:56:01.928742+0000 mgr.y (mgr.24380) 199 : cluster 0 pgmap v182: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.6 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 54 MiB/s wr, 867 op/s 2024-03-11T20:56:03.082 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:03 smithi023 bash[21372]: cluster 2024-03-11T20:56:01.944894+0000 osd.3 (osd.3) 153 : cluster 0 3.2 scrub starts 2024-03-11T20:56:03.082 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:03 smithi023 bash[21372]: cluster 2024-03-11T20:56:01.961991+0000 osd.3 (osd.3) 154 : cluster 0 3.2 scrub ok 2024-03-11T20:56:03.099 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:56:03.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:03.102 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:56:03.205 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:03.231 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:03.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:03.234 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:56:03.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:03 smithi146 bash[23809]: cluster 2024-03-11T20:56:01.928742+0000 mgr.y (mgr.24380) 199 : cluster 0 pgmap v182: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.6 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 54 MiB/s wr, 867 op/s 2024-03-11T20:56:03.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:03 smithi146 bash[23809]: cluster 2024-03-11T20:56:01.944894+0000 osd.3 (osd.3) 153 : cluster 0 3.2 scrub starts 2024-03-11T20:56:03.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:03 smithi146 bash[23809]: cluster 2024-03-11T20:56:01.961991+0000 osd.3 (osd.3) 154 : cluster 0 3.2 scrub ok 2024-03-11T20:56:03.335 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:56:03.363 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:56:03.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:03.370 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:56:03.471 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:56:03.495 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:56:03.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:03.498 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:56:03.547 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 72 16 94878 94862 82.3311 58.75 0.0087793 0.0121381 2024-03-11T20:56:03.599 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:03.624 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:03.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:03.630 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:56:03.731 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:56:03.766 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:56:03.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:03.774 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:56:03.875 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:56:03.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:03.908 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:56:03.908 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:56:04.009 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:04.034 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:04.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:04.036 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:56:04.138 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:04.162 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:04.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:04.164 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:56:04.265 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:56:04.275 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:04 smithi023 bash[21372]: cluster 2024-03-11T20:56:03.414253+0000 osd.6 (osd.6) 81 : cluster 0 2.1a scrub starts 2024-03-11T20:56:04.275 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:04 smithi023 bash[21372]: cluster 2024-03-11T20:56:03.416632+0000 osd.6 (osd.6) 82 : cluster 0 2.1a scrub ok 2024-03-11T20:56:04.275 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:04 smithi023 bash[21372]: cluster 2024-03-11T20:56:03.474963+0000 osd.5 (osd.5) 115 : cluster 0 2.3 deep-scrub starts 2024-03-11T20:56:04.275 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:04 smithi023 bash[21372]: cluster 2024-03-11T20:56:03.479175+0000 osd.5 (osd.5) 116 : cluster 0 2.3 deep-scrub ok 2024-03-11T20:56:04.276 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:04 smithi023 bash[29146]: cluster 2024-03-11T20:56:03.414253+0000 osd.6 (osd.6) 81 : cluster 0 2.1a scrub starts 2024-03-11T20:56:04.276 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:04 smithi023 bash[29146]: cluster 2024-03-11T20:56:03.416632+0000 osd.6 (osd.6) 82 : cluster 0 2.1a scrub ok 2024-03-11T20:56:04.276 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:04 smithi023 bash[29146]: cluster 2024-03-11T20:56:03.474963+0000 osd.5 (osd.5) 115 : cluster 0 2.3 deep-scrub starts 2024-03-11T20:56:04.276 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:04 smithi023 bash[29146]: cluster 2024-03-11T20:56:03.479175+0000 osd.5 (osd.5) 116 : cluster 0 2.3 deep-scrub ok 2024-03-11T20:56:04.290 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:56:04.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:04.294 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:56:04.395 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:56:04.437 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:04 smithi146 bash[23809]: cluster 2024-03-11T20:56:03.414253+0000 osd.6 (osd.6) 81 : cluster 0 2.1a scrub starts 2024-03-11T20:56:04.437 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:04 smithi146 bash[23809]: cluster 2024-03-11T20:56:03.416632+0000 osd.6 (osd.6) 82 : cluster 0 2.1a scrub ok 2024-03-11T20:56:04.437 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:04 smithi146 bash[23809]: cluster 2024-03-11T20:56:03.474963+0000 osd.5 (osd.5) 115 : cluster 0 2.3 deep-scrub starts 2024-03-11T20:56:04.437 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:04 smithi146 bash[23809]: cluster 2024-03-11T20:56:03.479175+0000 osd.5 (osd.5) 116 : cluster 0 2.3 deep-scrub ok 2024-03-11T20:56:04.451 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:56:04.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:04.457 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:56:04.547 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 73 16 95873 95857 82.0551 62.1875 0.00821039 0.0121821 2024-03-11T20:56:04.558 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:56:04.585 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:56:04.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:04.589 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:56:04.690 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:04.711 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:04.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:04.713 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:56:04.814 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:56:04.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:04.843 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:56:04.843 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:56:04.844 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:56:04.944 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:56:04.970 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:56:04.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:04.973 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:56:05.073 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:05.102 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:05.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:05.106 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:56:05.207 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:05.216 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 20:56:05 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T20:56:05.217 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:05 smithi146 bash[23809]: cluster 2024-03-11T20:56:03.929192+0000 mgr.y (mgr.24380) 200 : cluster 0 pgmap v183: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.6 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 54 MiB/s wr, 867 op/s 2024-03-11T20:56:05.217 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:05 smithi146 bash[23809]: audit 2024-03-11T20:56:04.414583+0000 mon.a (mon.0) 1007 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:56:05.217 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:05 smithi146 bash[23809]: cluster 2024-03-11T20:56:04.879845+0000 osd.3 (osd.3) 155 : cluster 0 2.1e scrub starts 2024-03-11T20:56:05.217 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:05 smithi146 bash[23809]: cluster 2024-03-11T20:56:04.881590+0000 osd.3 (osd.3) 156 : cluster 0 2.1e scrub ok 2024-03-11T20:56:05.229 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:05.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:05.233 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:56:05.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:05 smithi023 bash[21372]: cluster 2024-03-11T20:56:03.929192+0000 mgr.y (mgr.24380) 200 : cluster 0 pgmap v183: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.6 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 54 MiB/s wr, 867 op/s 2024-03-11T20:56:05.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:05 smithi023 bash[21372]: audit 2024-03-11T20:56:04.414583+0000 mon.a (mon.0) 1007 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:56:05.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:05 smithi023 bash[21372]: cluster 2024-03-11T20:56:04.879845+0000 osd.3 (osd.3) 155 : cluster 0 2.1e scrub starts 2024-03-11T20:56:05.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:05 smithi023 bash[21372]: cluster 2024-03-11T20:56:04.881590+0000 osd.3 (osd.3) 156 : cluster 0 2.1e scrub ok 2024-03-11T20:56:05.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:05 smithi023 bash[29146]: cluster 2024-03-11T20:56:03.929192+0000 mgr.y (mgr.24380) 200 : cluster 0 pgmap v183: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.6 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 54 MiB/s wr, 867 op/s 2024-03-11T20:56:05.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:05 smithi023 bash[29146]: audit 2024-03-11T20:56:04.414583+0000 mon.a (mon.0) 1007 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:56:05.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:05 smithi023 bash[29146]: cluster 2024-03-11T20:56:04.879845+0000 osd.3 (osd.3) 155 : cluster 0 2.1e scrub starts 2024-03-11T20:56:05.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:05 smithi023 bash[29146]: cluster 2024-03-11T20:56:04.881590+0000 osd.3 (osd.3) 156 : cluster 0 2.1e scrub ok 2024-03-11T20:56:05.333 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:05.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:05.369 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:05.370 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:56:05.470 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:56:05.496 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:56:05.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:05.502 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:56:05.547 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 74 16 96596 96580 81.5569 45.1875 0.00798449 0.0122548 2024-03-11T20:56:05.604 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:05.630 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:05.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:05.634 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:56:05.735 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:05.760 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:05.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:05.763 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:56:05.866 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:05.886 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:05.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:05.887 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:56:05.988 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:56:06.013 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:56:06.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:06.018 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:56:06.119 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:06.129 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:06 smithi023 bash[21372]: cluster 2024-03-11T20:56:04.092795+0000 osd.0 (osd.0) 135 : cluster 0 4.16 scrub starts 2024-03-11T20:56:06.130 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:06 smithi023 bash[21372]: cluster 2024-03-11T20:56:04.123665+0000 osd.0 (osd.0) 136 : cluster 0 4.16 scrub ok 2024-03-11T20:56:06.130 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:06 smithi023 bash[21372]: cluster 2024-03-11T20:56:04.967882+0000 osd.2 (osd.2) 96 : cluster 0 4.1c scrub starts 2024-03-11T20:56:06.130 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:06 smithi023 bash[21372]: cluster 2024-03-11T20:56:04.977116+0000 osd.2 (osd.2) 97 : cluster 0 4.1c scrub ok 2024-03-11T20:56:06.130 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:06 smithi023 bash[21372]: audit 2024-03-11T20:56:05.061011+0000 mgr.y (mgr.24380) 201 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:56:06.130 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:06 smithi023 bash[21372]: cluster 2024-03-11T20:56:05.903586+0000 osd.3 (osd.3) 157 : cluster 0 3.8 scrub starts 2024-03-11T20:56:06.130 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:06 smithi023 bash[29146]: cluster 2024-03-11T20:56:04.092795+0000 osd.0 (osd.0) 135 : cluster 0 4.16 scrub starts 2024-03-11T20:56:06.130 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:06 smithi023 bash[29146]: cluster 2024-03-11T20:56:04.123665+0000 osd.0 (osd.0) 136 : cluster 0 4.16 scrub ok 2024-03-11T20:56:06.130 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:06 smithi023 bash[29146]: cluster 2024-03-11T20:56:04.967882+0000 osd.2 (osd.2) 96 : cluster 0 4.1c scrub starts 2024-03-11T20:56:06.130 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:06 smithi023 bash[29146]: cluster 2024-03-11T20:56:04.977116+0000 osd.2 (osd.2) 97 : cluster 0 4.1c scrub ok 2024-03-11T20:56:06.131 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:06 smithi023 bash[29146]: audit 2024-03-11T20:56:05.061011+0000 mgr.y (mgr.24380) 201 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:56:06.131 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:06 smithi023 bash[29146]: cluster 2024-03-11T20:56:05.903586+0000 osd.3 (osd.3) 157 : cluster 0 3.8 scrub starts 2024-03-11T20:56:06.153 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:06.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:06.156 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:56:06.257 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:56:06.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:06 smithi146 bash[23809]: cluster 2024-03-11T20:56:04.092795+0000 osd.0 (osd.0) 135 : cluster 0 4.16 scrub starts 2024-03-11T20:56:06.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:06 smithi146 bash[23809]: cluster 2024-03-11T20:56:04.123665+0000 osd.0 (osd.0) 136 : cluster 0 4.16 scrub ok 2024-03-11T20:56:06.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:06 smithi146 bash[23809]: cluster 2024-03-11T20:56:04.967882+0000 osd.2 (osd.2) 96 : cluster 0 4.1c scrub starts 2024-03-11T20:56:06.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:06 smithi146 bash[23809]: cluster 2024-03-11T20:56:04.977116+0000 osd.2 (osd.2) 97 : cluster 0 4.1c scrub ok 2024-03-11T20:56:06.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:06 smithi146 bash[23809]: audit 2024-03-11T20:56:05.061011+0000 mgr.y (mgr.24380) 201 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:56:06.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:06 smithi146 bash[23809]: cluster 2024-03-11T20:56:05.903586+0000 osd.3 (osd.3) 157 : cluster 0 3.8 scrub starts 2024-03-11T20:56:06.288 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:56:06.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:06.293 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:56:06.394 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:56:06.424 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:56:06.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:06.428 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:56:06.529 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:56:06.547 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 75 16 97534 97518 81.2511 58.625 0.00805217 0.0122975 2024-03-11T20:56:06.554 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:56:06.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:06.557 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:56:06.658 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:06.691 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:06.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:06.698 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:56:06.798 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:06.829 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:06.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:06.835 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:56:06.902 INFO:teuthology.orchestra.run.smithi023.stdout:osd.0: {} 2024-03-11T20:56:06.930 INFO:teuthology.orchestra.run.smithi023.stdout:osd.1: {} 2024-03-11T20:56:06.936 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:56:06.957 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:56:06.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:06.959 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:56:06.984 INFO:teuthology.orchestra.run.smithi023.stdout:osd.2: {} 2024-03-11T20:56:07.060 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:07.071 INFO:teuthology.orchestra.run.smithi023.stdout:osd.3: {} 2024-03-11T20:56:07.080 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:07.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:07.082 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:56:07.111 INFO:teuthology.orchestra.run.smithi023.stdout:osd.4: {} 2024-03-11T20:56:07.145 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:07 smithi023 bash[21372]: cluster 2024-03-11T20:56:05.094872+0000 osd.0 (osd.0) 137 : cluster 0 2.13 scrub starts 2024-03-11T20:56:07.145 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:07 smithi023 bash[21372]: cluster 2024-03-11T20:56:05.096505+0000 osd.0 (osd.0) 138 : cluster 0 2.13 scrub ok 2024-03-11T20:56:07.145 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:07 smithi023 bash[21372]: cluster 2024-03-11T20:56:05.930131+0000 mgr.y (mgr.24380) 202 : cluster 0 pgmap v184: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.9 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 999 B/s rd, 50 MiB/s wr, 805 op/s 2024-03-11T20:56:07.145 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:07 smithi023 bash[21372]: cluster 2024-03-11T20:56:05.941895+0000 osd.3 (osd.3) 158 : cluster 0 3.8 scrub ok 2024-03-11T20:56:07.145 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:07 smithi023 bash[21372]: cluster 2024-03-11T20:56:06.731991+0000 osd.3 (osd.3) 159 : cluster 0 purged_snaps scrub starts 2024-03-11T20:56:07.145 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:07 smithi023 bash[21372]: cluster 2024-03-11T20:56:06.732056+0000 osd.3 (osd.3) 160 : cluster 0 purged_snaps scrub ok 2024-03-11T20:56:07.145 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:07 smithi023 bash[21372]: audit 2024-03-11T20:56:06.855884+0000 mon.c (mon.2) 34 : audit 0 from='client.? 172.21.15.23:0/2536213355' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T20:56:07.145 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:07 smithi023 bash[29146]: cluster 2024-03-11T20:56:05.094872+0000 osd.0 (osd.0) 137 : cluster 0 2.13 scrub starts 2024-03-11T20:56:07.146 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:07 smithi023 bash[29146]: cluster 2024-03-11T20:56:05.096505+0000 osd.0 (osd.0) 138 : cluster 0 2.13 scrub ok 2024-03-11T20:56:07.146 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:07 smithi023 bash[29146]: cluster 2024-03-11T20:56:05.930131+0000 mgr.y (mgr.24380) 202 : cluster 0 pgmap v184: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.9 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 999 B/s rd, 50 MiB/s wr, 805 op/s 2024-03-11T20:56:07.146 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:07 smithi023 bash[29146]: cluster 2024-03-11T20:56:05.941895+0000 osd.3 (osd.3) 158 : cluster 0 3.8 scrub ok 2024-03-11T20:56:07.146 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:07 smithi023 bash[29146]: cluster 2024-03-11T20:56:06.731991+0000 osd.3 (osd.3) 159 : cluster 0 purged_snaps scrub starts 2024-03-11T20:56:07.146 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:07 smithi023 bash[29146]: cluster 2024-03-11T20:56:06.732056+0000 osd.3 (osd.3) 160 : cluster 0 purged_snaps scrub ok 2024-03-11T20:56:07.146 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:07 smithi023 bash[29146]: audit 2024-03-11T20:56:06.855884+0000 mon.c (mon.2) 34 : audit 0 from='client.? 172.21.15.23:0/2536213355' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T20:56:07.156 INFO:teuthology.orchestra.run.smithi023.stdout:osd.5: {} 2024-03-11T20:56:07.182 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:07.202 INFO:teuthology.orchestra.run.smithi023.stdout:osd.6: {} 2024-03-11T20:56:07.213 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:07.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:07.217 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:56:07.247 INFO:teuthology.orchestra.run.smithi023.stdout:osd.7: {} 2024-03-11T20:56:07.257 INFO:teuthology.orchestra.run.smithi023.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T20:56:07.257 INFO:teuthology.orchestra.run.smithi023.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T20:56:07.257 INFO:teuthology.orchestra.run.smithi023.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T20:56:07.257 INFO:teuthology.orchestra.run.smithi023.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T20:56:07.257 INFO:teuthology.orchestra.run.smithi023.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T20:56:07.257 INFO:teuthology.orchestra.run.smithi023.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T20:56:07.257 INFO:teuthology.orchestra.run.smithi023.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T20:56:07.257 INFO:teuthology.orchestra.run.smithi023.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T20:56:07.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:07 smithi146 bash[23809]: cluster 2024-03-11T20:56:05.094872+0000 osd.0 (osd.0) 137 : cluster 0 2.13 scrub starts 2024-03-11T20:56:07.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:07 smithi146 bash[23809]: cluster 2024-03-11T20:56:05.096505+0000 osd.0 (osd.0) 138 : cluster 0 2.13 scrub ok 2024-03-11T20:56:07.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:07 smithi146 bash[23809]: cluster 2024-03-11T20:56:05.930131+0000 mgr.y (mgr.24380) 202 : cluster 0 pgmap v184: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.9 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 999 B/s rd, 50 MiB/s wr, 805 op/s 2024-03-11T20:56:07.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:07 smithi146 bash[23809]: cluster 2024-03-11T20:56:05.941895+0000 osd.3 (osd.3) 158 : cluster 0 3.8 scrub ok 2024-03-11T20:56:07.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:07 smithi146 bash[23809]: cluster 2024-03-11T20:56:06.731991+0000 osd.3 (osd.3) 159 : cluster 0 purged_snaps scrub starts 2024-03-11T20:56:07.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:07 smithi146 bash[23809]: cluster 2024-03-11T20:56:06.732056+0000 osd.3 (osd.3) 160 : cluster 0 purged_snaps scrub ok 2024-03-11T20:56:07.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:07 smithi146 bash[23809]: audit 2024-03-11T20:56:06.855884+0000 mon.c (mon.2) 34 : audit 0 from='client.? 172.21.15.23:0/2536213355' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T20:56:07.279 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-11T20:56:07.279 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-11T20:56:07.279 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 147, 2024-03-11T20:56:07.279 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "67ba00f8-dfe7-11ee-95c8-87774f69a715", 2024-03-11T20:56:07.279 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2024-03-11T20:40:13.664802+0000", 2024-03-11T20:56:07.279 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2024-03-11T20:56:07.026897+0000", 2024-03-11T20:56:07.279 INFO:teuthology.orchestra.run.smithi023.stdout: "last_up_change": "2024-03-11T20:49:48.903029+0000", 2024-03-11T20:56:07.279 INFO:teuthology.orchestra.run.smithi023.stdout: "last_in_change": "2024-03-11T20:54:36.322348+0000", 2024-03-11T20:56:07.279 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-03-11T20:56:07.280 INFO:teuthology.orchestra.run.smithi023.stdout: "flags_num": 5799936, 2024-03-11T20:56:07.280 INFO:teuthology.orchestra.run.smithi023.stdout: "flags_set": [ 2024-03-11T20:56:07.280 INFO:teuthology.orchestra.run.smithi023.stdout: "pglog_hardlimit", 2024-03-11T20:56:07.280 INFO:teuthology.orchestra.run.smithi023.stdout: "purged_snapdirs", 2024-03-11T20:56:07.280 INFO:teuthology.orchestra.run.smithi023.stdout: "recovery_deletes", 2024-03-11T20:56:07.280 INFO:teuthology.orchestra.run.smithi023.stdout: "sortbitwise" 2024-03-11T20:56:07.280 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-11T20:56:07.280 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_version": 18, 2024-03-11T20:56:07.280 INFO:teuthology.orchestra.run.smithi023.stdout: "full_ratio": 0.94999998807907104, 2024-03-11T20:56:07.280 INFO:teuthology.orchestra.run.smithi023.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-03-11T20:56:07.280 INFO:teuthology.orchestra.run.smithi023.stdout: "nearfull_ratio": 0.85000002384185791, 2024-03-11T20:56:07.280 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_snapshot": "", 2024-03-11T20:56:07.280 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_max": 7, 2024-03-11T20:56:07.280 INFO:teuthology.orchestra.run.smithi023.stdout: "max_osd": 8, 2024-03-11T20:56:07.280 INFO:teuthology.orchestra.run.smithi023.stdout: "require_min_compat_client": "luminous", 2024-03-11T20:56:07.281 INFO:teuthology.orchestra.run.smithi023.stdout: "min_compat_client": "luminous", 2024-03-11T20:56:07.281 INFO:teuthology.orchestra.run.smithi023.stdout: "require_osd_release": "squid", 2024-03-11T20:56:07.281 INFO:teuthology.orchestra.run.smithi023.stdout: "allow_crimson": false, 2024-03-11T20:56:07.281 INFO:teuthology.orchestra.run.smithi023.stdout: "pools": [ 2024-03-11T20:56:07.281 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:56:07.281 INFO:teuthology.orchestra.run.smithi023.stdout: "pool": 1, 2024-03-11T20:56:07.281 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_name": ".mgr", 2024-03-11T20:56:07.281 INFO:teuthology.orchestra.run.smithi023.stdout: "create_time": "2024-03-11T20:45:31.964570+0000", 2024-03-11T20:56:07.281 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-11T20:56:07.281 INFO:teuthology.orchestra.run.smithi023.stdout: "flags_names": "hashpspool", 2024-03-11T20:56:07.281 INFO:teuthology.orchestra.run.smithi023.stdout: "type": 1, 2024-03-11T20:56:07.281 INFO:teuthology.orchestra.run.smithi023.stdout: "size": 3, 2024-03-11T20:56:07.281 INFO:teuthology.orchestra.run.smithi023.stdout: "min_size": 2, 2024-03-11T20:56:07.281 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_rule": 0, 2024-03-11T20:56:07.282 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_count": 0, 2024-03-11T20:56:07.282 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_target": 0, 2024-03-11T20:56:07.282 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T20:56:07.282 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T20:56:07.282 INFO:teuthology.orchestra.run.smithi023.stdout: "object_hash": 2, 2024-03-11T20:56:07.282 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_autoscale_mode": "off", 2024-03-11T20:56:07.282 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num": 1, 2024-03-11T20:56:07.282 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_placement_num": 1, 2024-03-11T20:56:07.282 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_placement_num_target": 1, 2024-03-11T20:56:07.282 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num_target": 1, 2024-03-11T20:56:07.282 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num_pending": 1, 2024-03-11T20:56:07.282 INFO:teuthology.orchestra.run.smithi023.stdout: "last_pg_merge_meta": { 2024-03-11T20:56:07.282 INFO:teuthology.orchestra.run.smithi023.stdout: "source_pgid": "0.0", 2024-03-11T20:56:07.282 INFO:teuthology.orchestra.run.smithi023.stdout: "ready_epoch": 0, 2024-03-11T20:56:07.283 INFO:teuthology.orchestra.run.smithi023.stdout: "last_epoch_started": 0, 2024-03-11T20:56:07.283 INFO:teuthology.orchestra.run.smithi023.stdout: "last_epoch_clean": 0, 2024-03-11T20:56:07.283 INFO:teuthology.orchestra.run.smithi023.stdout: "source_version": "0'0", 2024-03-11T20:56:07.283 INFO:teuthology.orchestra.run.smithi023.stdout: "target_version": "0'0" 2024-03-11T20:56:07.283 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:56:07.283 INFO:teuthology.orchestra.run.smithi023.stdout: "last_change": "142", 2024-03-11T20:56:07.283 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend": "0", 2024-03-11T20:56:07.283 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T20:56:07.283 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T20:56:07.283 INFO:teuthology.orchestra.run.smithi023.stdout: "auid": 0, 2024-03-11T20:56:07.283 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_mode": "selfmanaged", 2024-03-11T20:56:07.283 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_seq": 0, 2024-03-11T20:56:07.283 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_epoch": 0, 2024-03-11T20:56:07.283 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_snaps": [], 2024-03-11T20:56:07.283 INFO:teuthology.orchestra.run.smithi023.stdout: "removed_snaps": "[]", 2024-03-11T20:56:07.284 INFO:teuthology.orchestra.run.smithi023.stdout: "quota_max_bytes": 0, 2024-03-11T20:56:07.284 INFO:teuthology.orchestra.run.smithi023.stdout: "quota_max_objects": 0, 2024-03-11T20:56:07.284 INFO:teuthology.orchestra.run.smithi023.stdout: "tiers": [], 2024-03-11T20:56:07.284 INFO:teuthology.orchestra.run.smithi023.stdout: "tier_of": -1, 2024-03-11T20:56:07.284 INFO:teuthology.orchestra.run.smithi023.stdout: "read_tier": -1, 2024-03-11T20:56:07.284 INFO:teuthology.orchestra.run.smithi023.stdout: "write_tier": -1, 2024-03-11T20:56:07.284 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_mode": "none", 2024-03-11T20:56:07.284 INFO:teuthology.orchestra.run.smithi023.stdout: "target_max_bytes": 0, 2024-03-11T20:56:07.284 INFO:teuthology.orchestra.run.smithi023.stdout: "target_max_objects": 0, 2024-03-11T20:56:07.284 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T20:56:07.284 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T20:56:07.284 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T20:56:07.284 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_min_flush_age": 0, 2024-03-11T20:56:07.284 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_min_evict_age": 0, 2024-03-11T20:56:07.285 INFO:teuthology.orchestra.run.smithi023.stdout: "erasure_code_profile": "", 2024-03-11T20:56:07.285 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_params": { 2024-03-11T20:56:07.285 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "none" 2024-03-11T20:56:07.285 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:56:07.285 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_period": 0, 2024-03-11T20:56:07.285 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_count": 0, 2024-03-11T20:56:07.285 INFO:teuthology.orchestra.run.smithi023.stdout: "use_gmt_hitset": true, 2024-03-11T20:56:07.285 INFO:teuthology.orchestra.run.smithi023.stdout: "min_read_recency_for_promote": 0, 2024-03-11T20:56:07.285 INFO:teuthology.orchestra.run.smithi023.stdout: "min_write_recency_for_promote": 0, 2024-03-11T20:56:07.285 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T20:56:07.285 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_search_last_n": 0, 2024-03-11T20:56:07.285 INFO:teuthology.orchestra.run.smithi023.stdout: "grade_table": [], 2024-03-11T20:56:07.285 INFO:teuthology.orchestra.run.smithi023.stdout: "stripe_width": 0, 2024-03-11T20:56:07.285 INFO:teuthology.orchestra.run.smithi023.stdout: "expected_num_objects": 0, 2024-03-11T20:56:07.285 INFO:teuthology.orchestra.run.smithi023.stdout: "fast_read": false, 2024-03-11T20:56:07.286 INFO:teuthology.orchestra.run.smithi023.stdout: "options": { 2024-03-11T20:56:07.286 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num_max": 32, 2024-03-11T20:56:07.286 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num_min": 1 2024-03-11T20:56:07.286 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:56:07.286 INFO:teuthology.orchestra.run.smithi023.stdout: "application_metadata": { 2024-03-11T20:56:07.286 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr": {} 2024-03-11T20:56:07.286 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:56:07.286 INFO:teuthology.orchestra.run.smithi023.stdout: "read_balance": { 2024-03-11T20:56:07.286 INFO:teuthology.orchestra.run.smithi023.stdout: "score_type": "Fair distribution", 2024-03-11T20:56:07.286 INFO:teuthology.orchestra.run.smithi023.stdout: "score_acting": 7.8899998664855957, 2024-03-11T20:56:07.286 INFO:teuthology.orchestra.run.smithi023.stdout: "score_stable": 7.8899998664855957, 2024-03-11T20:56:07.286 INFO:teuthology.orchestra.run.smithi023.stdout: "optimal_score": 0.37999999523162842, 2024-03-11T20:56:07.286 INFO:teuthology.orchestra.run.smithi023.stdout: "raw_score_acting": 3, 2024-03-11T20:56:07.287 INFO:teuthology.orchestra.run.smithi023.stdout: "raw_score_stable": 3, 2024-03-11T20:56:07.287 INFO:teuthology.orchestra.run.smithi023.stdout: "primary_affinity_weighted": 1, 2024-03-11T20:56:07.287 INFO:teuthology.orchestra.run.smithi023.stdout: "average_primary_affinity": 1, 2024-03-11T20:56:07.287 INFO:teuthology.orchestra.run.smithi023.stdout: "average_primary_affinity_weighted": 1 2024-03-11T20:56:07.287 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:56:07.287 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:56:07.287 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:56:07.287 INFO:teuthology.orchestra.run.smithi023.stdout: "pool": 2, 2024-03-11T20:56:07.287 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_name": ".rgw.root", 2024-03-11T20:56:07.287 INFO:teuthology.orchestra.run.smithi023.stdout: "create_time": "2024-03-11T20:50:26.421486+0000", 2024-03-11T20:56:07.287 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-11T20:56:07.287 INFO:teuthology.orchestra.run.smithi023.stdout: "flags_names": "hashpspool", 2024-03-11T20:56:07.287 INFO:teuthology.orchestra.run.smithi023.stdout: "type": 1, 2024-03-11T20:56:07.287 INFO:teuthology.orchestra.run.smithi023.stdout: "size": 3, 2024-03-11T20:56:07.287 INFO:teuthology.orchestra.run.smithi023.stdout: "min_size": 2, 2024-03-11T20:56:07.288 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_rule": 0, 2024-03-11T20:56:07.288 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_count": 0, 2024-03-11T20:56:07.288 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_target": 0, 2024-03-11T20:56:07.288 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T20:56:07.288 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T20:56:07.288 INFO:teuthology.orchestra.run.smithi023.stdout: "object_hash": 2, 2024-03-11T20:56:07.288 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_autoscale_mode": "off", 2024-03-11T20:56:07.288 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num": 32, 2024-03-11T20:56:07.288 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_placement_num": 32, 2024-03-11T20:56:07.288 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_placement_num_target": 32, 2024-03-11T20:56:07.288 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num_target": 32, 2024-03-11T20:56:07.288 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num_pending": 32, 2024-03-11T20:56:07.288 INFO:teuthology.orchestra.run.smithi023.stdout: "last_pg_merge_meta": { 2024-03-11T20:56:07.288 INFO:teuthology.orchestra.run.smithi023.stdout: "source_pgid": "0.0", 2024-03-11T20:56:07.289 INFO:teuthology.orchestra.run.smithi023.stdout: "ready_epoch": 0, 2024-03-11T20:56:07.289 INFO:teuthology.orchestra.run.smithi023.stdout: "last_epoch_started": 0, 2024-03-11T20:56:07.289 INFO:teuthology.orchestra.run.smithi023.stdout: "last_epoch_clean": 0, 2024-03-11T20:56:07.289 INFO:teuthology.orchestra.run.smithi023.stdout: "source_version": "0'0", 2024-03-11T20:56:07.289 INFO:teuthology.orchestra.run.smithi023.stdout: "target_version": "0'0" 2024-03-11T20:56:07.289 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:56:07.289 INFO:teuthology.orchestra.run.smithi023.stdout: "last_change": "78", 2024-03-11T20:56:07.289 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend": "0", 2024-03-11T20:56:07.289 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T20:56:07.289 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T20:56:07.289 INFO:teuthology.orchestra.run.smithi023.stdout: "auid": 0, 2024-03-11T20:56:07.289 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_mode": "selfmanaged", 2024-03-11T20:56:07.289 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_seq": 0, 2024-03-11T20:56:07.289 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_epoch": 0, 2024-03-11T20:56:07.289 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_snaps": [], 2024-03-11T20:56:07.290 INFO:teuthology.orchestra.run.smithi023.stdout: "removed_snaps": "[]", 2024-03-11T20:56:07.290 INFO:teuthology.orchestra.run.smithi023.stdout: "quota_max_bytes": 0, 2024-03-11T20:56:07.290 INFO:teuthology.orchestra.run.smithi023.stdout: "quota_max_objects": 0, 2024-03-11T20:56:07.290 INFO:teuthology.orchestra.run.smithi023.stdout: "tiers": [], 2024-03-11T20:56:07.290 INFO:teuthology.orchestra.run.smithi023.stdout: "tier_of": -1, 2024-03-11T20:56:07.290 INFO:teuthology.orchestra.run.smithi023.stdout: "read_tier": -1, 2024-03-11T20:56:07.290 INFO:teuthology.orchestra.run.smithi023.stdout: "write_tier": -1, 2024-03-11T20:56:07.290 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_mode": "none", 2024-03-11T20:56:07.290 INFO:teuthology.orchestra.run.smithi023.stdout: "target_max_bytes": 0, 2024-03-11T20:56:07.290 INFO:teuthology.orchestra.run.smithi023.stdout: "target_max_objects": 0, 2024-03-11T20:56:07.290 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T20:56:07.290 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T20:56:07.290 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T20:56:07.291 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_min_flush_age": 0, 2024-03-11T20:56:07.291 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_min_evict_age": 0, 2024-03-11T20:56:07.291 INFO:teuthology.orchestra.run.smithi023.stdout: "erasure_code_profile": "", 2024-03-11T20:56:07.291 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_params": { 2024-03-11T20:56:07.291 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "none" 2024-03-11T20:56:07.291 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:56:07.291 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_period": 0, 2024-03-11T20:56:07.291 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_count": 0, 2024-03-11T20:56:07.291 INFO:teuthology.orchestra.run.smithi023.stdout: "use_gmt_hitset": true, 2024-03-11T20:56:07.291 INFO:teuthology.orchestra.run.smithi023.stdout: "min_read_recency_for_promote": 0, 2024-03-11T20:56:07.291 INFO:teuthology.orchestra.run.smithi023.stdout: "min_write_recency_for_promote": 0, 2024-03-11T20:56:07.291 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T20:56:07.291 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_search_last_n": 0, 2024-03-11T20:56:07.292 INFO:teuthology.orchestra.run.smithi023.stdout: "grade_table": [], 2024-03-11T20:56:07.292 INFO:teuthology.orchestra.run.smithi023.stdout: "stripe_width": 0, 2024-03-11T20:56:07.292 INFO:teuthology.orchestra.run.smithi023.stdout: "expected_num_objects": 0, 2024-03-11T20:56:07.292 INFO:teuthology.orchestra.run.smithi023.stdout: "fast_read": false, 2024-03-11T20:56:07.292 INFO:teuthology.orchestra.run.smithi023.stdout: "options": {}, 2024-03-11T20:56:07.292 INFO:teuthology.orchestra.run.smithi023.stdout: "application_metadata": { 2024-03-11T20:56:07.292 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw": {} 2024-03-11T20:56:07.292 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:56:07.292 INFO:teuthology.orchestra.run.smithi023.stdout: "read_balance": { 2024-03-11T20:56:07.292 INFO:teuthology.orchestra.run.smithi023.stdout: "score_type": "Fair distribution", 2024-03-11T20:56:07.292 INFO:teuthology.orchestra.run.smithi023.stdout: "score_acting": 1.4900000095367432, 2024-03-11T20:56:07.292 INFO:teuthology.orchestra.run.smithi023.stdout: "score_stable": 1.4900000095367432, 2024-03-11T20:56:07.293 INFO:teuthology.orchestra.run.smithi023.stdout: "optimal_score": 0.87999999523162842, 2024-03-11T20:56:07.293 INFO:teuthology.orchestra.run.smithi023.stdout: "raw_score_acting": 1.309999942779541, 2024-03-11T20:56:07.293 INFO:teuthology.orchestra.run.smithi023.stdout: "raw_score_stable": 1.309999942779541, 2024-03-11T20:56:07.293 INFO:teuthology.orchestra.run.smithi023.stdout: "primary_affinity_weighted": 1, 2024-03-11T20:56:07.293 INFO:teuthology.orchestra.run.smithi023.stdout: "average_primary_affinity": 1, 2024-03-11T20:56:07.293 INFO:teuthology.orchestra.run.smithi023.stdout: "average_primary_affinity_weighted": 1 2024-03-11T20:56:07.293 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:56:07.293 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:56:07.293 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:56:07.293 INFO:teuthology.orchestra.run.smithi023.stdout: "pool": 3, 2024-03-11T20:56:07.293 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_name": "default.rgw.log", 2024-03-11T20:56:07.293 INFO:teuthology.orchestra.run.smithi023.stdout: "create_time": "2024-03-11T20:50:28.592615+0000", 2024-03-11T20:56:07.293 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-11T20:56:07.293 INFO:teuthology.orchestra.run.smithi023.stdout: "flags_names": "hashpspool", 2024-03-11T20:56:07.293 INFO:teuthology.orchestra.run.smithi023.stdout: "type": 1, 2024-03-11T20:56:07.293 INFO:teuthology.orchestra.run.smithi023.stdout: "size": 3, 2024-03-11T20:56:07.294 INFO:teuthology.orchestra.run.smithi023.stdout: "min_size": 2, 2024-03-11T20:56:07.294 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_rule": 0, 2024-03-11T20:56:07.294 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_count": 0, 2024-03-11T20:56:07.294 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_target": 0, 2024-03-11T20:56:07.294 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T20:56:07.294 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T20:56:07.294 INFO:teuthology.orchestra.run.smithi023.stdout: "object_hash": 2, 2024-03-11T20:56:07.294 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_autoscale_mode": "off", 2024-03-11T20:56:07.294 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num": 32, 2024-03-11T20:56:07.294 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_placement_num": 32, 2024-03-11T20:56:07.294 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_placement_num_target": 32, 2024-03-11T20:56:07.294 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num_target": 32, 2024-03-11T20:56:07.294 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num_pending": 32, 2024-03-11T20:56:07.294 INFO:teuthology.orchestra.run.smithi023.stdout: "last_pg_merge_meta": { 2024-03-11T20:56:07.295 INFO:teuthology.orchestra.run.smithi023.stdout: "source_pgid": "0.0", 2024-03-11T20:56:07.295 INFO:teuthology.orchestra.run.smithi023.stdout: "ready_epoch": 0, 2024-03-11T20:56:07.295 INFO:teuthology.orchestra.run.smithi023.stdout: "last_epoch_started": 0, 2024-03-11T20:56:07.295 INFO:teuthology.orchestra.run.smithi023.stdout: "last_epoch_clean": 0, 2024-03-11T20:56:07.295 INFO:teuthology.orchestra.run.smithi023.stdout: "source_version": "0'0", 2024-03-11T20:56:07.295 INFO:teuthology.orchestra.run.smithi023.stdout: "target_version": "0'0" 2024-03-11T20:56:07.295 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:56:07.295 INFO:teuthology.orchestra.run.smithi023.stdout: "last_change": "80", 2024-03-11T20:56:07.295 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend": "0", 2024-03-11T20:56:07.295 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T20:56:07.295 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T20:56:07.295 INFO:teuthology.orchestra.run.smithi023.stdout: "auid": 0, 2024-03-11T20:56:07.295 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_mode": "selfmanaged", 2024-03-11T20:56:07.295 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_seq": 0, 2024-03-11T20:56:07.296 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_epoch": 0, 2024-03-11T20:56:07.296 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_snaps": [], 2024-03-11T20:56:07.296 INFO:teuthology.orchestra.run.smithi023.stdout: "removed_snaps": "[]", 2024-03-11T20:56:07.296 INFO:teuthology.orchestra.run.smithi023.stdout: "quota_max_bytes": 0, 2024-03-11T20:56:07.296 INFO:teuthology.orchestra.run.smithi023.stdout: "quota_max_objects": 0, 2024-03-11T20:56:07.296 INFO:teuthology.orchestra.run.smithi023.stdout: "tiers": [], 2024-03-11T20:56:07.296 INFO:teuthology.orchestra.run.smithi023.stdout: "tier_of": -1, 2024-03-11T20:56:07.296 INFO:teuthology.orchestra.run.smithi023.stdout: "read_tier": -1, 2024-03-11T20:56:07.296 INFO:teuthology.orchestra.run.smithi023.stdout: "write_tier": -1, 2024-03-11T20:56:07.296 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_mode": "none", 2024-03-11T20:56:07.296 INFO:teuthology.orchestra.run.smithi023.stdout: "target_max_bytes": 0, 2024-03-11T20:56:07.296 INFO:teuthology.orchestra.run.smithi023.stdout: "target_max_objects": 0, 2024-03-11T20:56:07.296 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T20:56:07.296 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T20:56:07.296 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T20:56:07.297 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_min_flush_age": 0, 2024-03-11T20:56:07.297 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_min_evict_age": 0, 2024-03-11T20:56:07.297 INFO:teuthology.orchestra.run.smithi023.stdout: "erasure_code_profile": "", 2024-03-11T20:56:07.297 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_params": { 2024-03-11T20:56:07.297 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "none" 2024-03-11T20:56:07.297 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:56:07.297 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_period": 0, 2024-03-11T20:56:07.297 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_count": 0, 2024-03-11T20:56:07.297 INFO:teuthology.orchestra.run.smithi023.stdout: "use_gmt_hitset": true, 2024-03-11T20:56:07.297 INFO:teuthology.orchestra.run.smithi023.stdout: "min_read_recency_for_promote": 0, 2024-03-11T20:56:07.297 INFO:teuthology.orchestra.run.smithi023.stdout: "min_write_recency_for_promote": 0, 2024-03-11T20:56:07.297 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T20:56:07.297 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_search_last_n": 0, 2024-03-11T20:56:07.297 INFO:teuthology.orchestra.run.smithi023.stdout: "grade_table": [], 2024-03-11T20:56:07.298 INFO:teuthology.orchestra.run.smithi023.stdout: "stripe_width": 0, 2024-03-11T20:56:07.298 INFO:teuthology.orchestra.run.smithi023.stdout: "expected_num_objects": 0, 2024-03-11T20:56:07.298 INFO:teuthology.orchestra.run.smithi023.stdout: "fast_read": false, 2024-03-11T20:56:07.298 INFO:teuthology.orchestra.run.smithi023.stdout: "options": {}, 2024-03-11T20:56:07.298 INFO:teuthology.orchestra.run.smithi023.stdout: "application_metadata": { 2024-03-11T20:56:07.298 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw": {} 2024-03-11T20:56:07.298 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:56:07.298 INFO:teuthology.orchestra.run.smithi023.stdout: "read_balance": { 2024-03-11T20:56:07.298 INFO:teuthology.orchestra.run.smithi023.stdout: "score_type": "Fair distribution", 2024-03-11T20:56:07.298 INFO:teuthology.orchestra.run.smithi023.stdout: "score_acting": 1.7400000095367432, 2024-03-11T20:56:07.298 INFO:teuthology.orchestra.run.smithi023.stdout: "score_stable": 1.7400000095367432, 2024-03-11T20:56:07.298 INFO:teuthology.orchestra.run.smithi023.stdout: "optimal_score": 0.87999999523162842, 2024-03-11T20:56:07.298 INFO:teuthology.orchestra.run.smithi023.stdout: "raw_score_acting": 1.5299999713897705, 2024-03-11T20:56:07.298 INFO:teuthology.orchestra.run.smithi023.stdout: "raw_score_stable": 1.5299999713897705, 2024-03-11T20:56:07.298 INFO:teuthology.orchestra.run.smithi023.stdout: "primary_affinity_weighted": 1, 2024-03-11T20:56:07.299 INFO:teuthology.orchestra.run.smithi023.stdout: "average_primary_affinity": 1, 2024-03-11T20:56:07.299 INFO:teuthology.orchestra.run.smithi023.stdout: "average_primary_affinity_weighted": 1 2024-03-11T20:56:07.299 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:56:07.299 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:56:07.299 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:56:07.299 INFO:teuthology.orchestra.run.smithi023.stdout: "pool": 4, 2024-03-11T20:56:07.299 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_name": "default.rgw.control", 2024-03-11T20:56:07.299 INFO:teuthology.orchestra.run.smithi023.stdout: "create_time": "2024-03-11T20:50:30.536371+0000", 2024-03-11T20:56:07.299 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-11T20:56:07.299 INFO:teuthology.orchestra.run.smithi023.stdout: "flags_names": "hashpspool", 2024-03-11T20:56:07.299 INFO:teuthology.orchestra.run.smithi023.stdout: "type": 1, 2024-03-11T20:56:07.299 INFO:teuthology.orchestra.run.smithi023.stdout: "size": 3, 2024-03-11T20:56:07.299 INFO:teuthology.orchestra.run.smithi023.stdout: "min_size": 2, 2024-03-11T20:56:07.299 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_rule": 0, 2024-03-11T20:56:07.299 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_count": 0, 2024-03-11T20:56:07.299 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_target": 0, 2024-03-11T20:56:07.300 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T20:56:07.300 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T20:56:07.300 INFO:teuthology.orchestra.run.smithi023.stdout: "object_hash": 2, 2024-03-11T20:56:07.300 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_autoscale_mode": "off", 2024-03-11T20:56:07.300 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num": 32, 2024-03-11T20:56:07.300 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_placement_num": 32, 2024-03-11T20:56:07.300 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_placement_num_target": 32, 2024-03-11T20:56:07.300 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num_target": 32, 2024-03-11T20:56:07.300 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num_pending": 32, 2024-03-11T20:56:07.300 INFO:teuthology.orchestra.run.smithi023.stdout: "last_pg_merge_meta": { 2024-03-11T20:56:07.300 INFO:teuthology.orchestra.run.smithi023.stdout: "source_pgid": "0.0", 2024-03-11T20:56:07.300 INFO:teuthology.orchestra.run.smithi023.stdout: "ready_epoch": 0, 2024-03-11T20:56:07.300 INFO:teuthology.orchestra.run.smithi023.stdout: "last_epoch_started": 0, 2024-03-11T20:56:07.300 INFO:teuthology.orchestra.run.smithi023.stdout: "last_epoch_clean": 0, 2024-03-11T20:56:07.300 INFO:teuthology.orchestra.run.smithi023.stdout: "source_version": "0'0", 2024-03-11T20:56:07.301 INFO:teuthology.orchestra.run.smithi023.stdout: "target_version": "0'0" 2024-03-11T20:56:07.301 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:56:07.301 INFO:teuthology.orchestra.run.smithi023.stdout: "last_change": "82", 2024-03-11T20:56:07.301 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend": "0", 2024-03-11T20:56:07.301 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T20:56:07.301 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T20:56:07.301 INFO:teuthology.orchestra.run.smithi023.stdout: "auid": 0, 2024-03-11T20:56:07.301 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_mode": "selfmanaged", 2024-03-11T20:56:07.301 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_seq": 0, 2024-03-11T20:56:07.301 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_epoch": 0, 2024-03-11T20:56:07.301 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_snaps": [], 2024-03-11T20:56:07.301 INFO:teuthology.orchestra.run.smithi023.stdout: "removed_snaps": "[]", 2024-03-11T20:56:07.301 INFO:teuthology.orchestra.run.smithi023.stdout: "quota_max_bytes": 0, 2024-03-11T20:56:07.301 INFO:teuthology.orchestra.run.smithi023.stdout: "quota_max_objects": 0, 2024-03-11T20:56:07.301 INFO:teuthology.orchestra.run.smithi023.stdout: "tiers": [], 2024-03-11T20:56:07.302 INFO:teuthology.orchestra.run.smithi023.stdout: "tier_of": -1, 2024-03-11T20:56:07.302 INFO:teuthology.orchestra.run.smithi023.stdout: "read_tier": -1, 2024-03-11T20:56:07.302 INFO:teuthology.orchestra.run.smithi023.stdout: "write_tier": -1, 2024-03-11T20:56:07.302 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_mode": "none", 2024-03-11T20:56:07.302 INFO:teuthology.orchestra.run.smithi023.stdout: "target_max_bytes": 0, 2024-03-11T20:56:07.302 INFO:teuthology.orchestra.run.smithi023.stdout: "target_max_objects": 0, 2024-03-11T20:56:07.302 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T20:56:07.302 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T20:56:07.302 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T20:56:07.302 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_min_flush_age": 0, 2024-03-11T20:56:07.302 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_min_evict_age": 0, 2024-03-11T20:56:07.302 INFO:teuthology.orchestra.run.smithi023.stdout: "erasure_code_profile": "", 2024-03-11T20:56:07.302 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_params": { 2024-03-11T20:56:07.302 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "none" 2024-03-11T20:56:07.303 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:56:07.303 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_period": 0, 2024-03-11T20:56:07.303 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_count": 0, 2024-03-11T20:56:07.303 INFO:teuthology.orchestra.run.smithi023.stdout: "use_gmt_hitset": true, 2024-03-11T20:56:07.303 INFO:teuthology.orchestra.run.smithi023.stdout: "min_read_recency_for_promote": 0, 2024-03-11T20:56:07.303 INFO:teuthology.orchestra.run.smithi023.stdout: "min_write_recency_for_promote": 0, 2024-03-11T20:56:07.303 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T20:56:07.303 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_search_last_n": 0, 2024-03-11T20:56:07.303 INFO:teuthology.orchestra.run.smithi023.stdout: "grade_table": [], 2024-03-11T20:56:07.303 INFO:teuthology.orchestra.run.smithi023.stdout: "stripe_width": 0, 2024-03-11T20:56:07.303 INFO:teuthology.orchestra.run.smithi023.stdout: "expected_num_objects": 0, 2024-03-11T20:56:07.303 INFO:teuthology.orchestra.run.smithi023.stdout: "fast_read": false, 2024-03-11T20:56:07.303 INFO:teuthology.orchestra.run.smithi023.stdout: "options": {}, 2024-03-11T20:56:07.303 INFO:teuthology.orchestra.run.smithi023.stdout: "application_metadata": { 2024-03-11T20:56:07.303 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw": {} 2024-03-11T20:56:07.303 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:56:07.304 INFO:teuthology.orchestra.run.smithi023.stdout: "read_balance": { 2024-03-11T20:56:07.304 INFO:teuthology.orchestra.run.smithi023.stdout: "score_type": "Fair distribution", 2024-03-11T20:56:07.304 INFO:teuthology.orchestra.run.smithi023.stdout: "score_acting": 2.2400000095367432, 2024-03-11T20:56:07.304 INFO:teuthology.orchestra.run.smithi023.stdout: "score_stable": 2.2400000095367432, 2024-03-11T20:56:07.305 INFO:teuthology.orchestra.run.smithi023.stdout: "optimal_score": 0.87999999523162842, 2024-03-11T20:56:07.305 INFO:teuthology.orchestra.run.smithi023.stdout: "raw_score_acting": 1.9700000286102295, 2024-03-11T20:56:07.305 INFO:teuthology.orchestra.run.smithi023.stdout: "raw_score_stable": 1.9700000286102295, 2024-03-11T20:56:07.305 INFO:teuthology.orchestra.run.smithi023.stdout: "primary_affinity_weighted": 1, 2024-03-11T20:56:07.305 INFO:teuthology.orchestra.run.smithi023.stdout: "average_primary_affinity": 1, 2024-03-11T20:56:07.306 INFO:teuthology.orchestra.run.smithi023.stdout: "average_primary_affinity_weighted": 1 2024-03-11T20:56:07.306 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:56:07.306 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:56:07.306 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:56:07.306 INFO:teuthology.orchestra.run.smithi023.stdout: "pool": 5, 2024-03-11T20:56:07.306 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_name": "datapool", 2024-03-11T20:56:07.306 INFO:teuthology.orchestra.run.smithi023.stdout: "create_time": "2024-03-11T20:50:31.135845+0000", 2024-03-11T20:56:07.306 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 8193, 2024-03-11T20:56:07.306 INFO:teuthology.orchestra.run.smithi023.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-03-11T20:56:07.306 INFO:teuthology.orchestra.run.smithi023.stdout: "type": 1, 2024-03-11T20:56:07.306 INFO:teuthology.orchestra.run.smithi023.stdout: "size": 3, 2024-03-11T20:56:07.306 INFO:teuthology.orchestra.run.smithi023.stdout: "min_size": 2, 2024-03-11T20:56:07.306 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_rule": 0, 2024-03-11T20:56:07.306 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_count": 0, 2024-03-11T20:56:07.306 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_target": 0, 2024-03-11T20:56:07.307 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T20:56:07.307 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T20:56:07.307 INFO:teuthology.orchestra.run.smithi023.stdout: "object_hash": 2, 2024-03-11T20:56:07.307 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_autoscale_mode": "off", 2024-03-11T20:56:07.307 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num": 3, 2024-03-11T20:56:07.307 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_placement_num": 3, 2024-03-11T20:56:07.307 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_placement_num_target": 3, 2024-03-11T20:56:07.307 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num_target": 3, 2024-03-11T20:56:07.307 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num_pending": 3, 2024-03-11T20:56:07.307 INFO:teuthology.orchestra.run.smithi023.stdout: "last_pg_merge_meta": { 2024-03-11T20:56:07.307 INFO:teuthology.orchestra.run.smithi023.stdout: "source_pgid": "0.0", 2024-03-11T20:56:07.307 INFO:teuthology.orchestra.run.smithi023.stdout: "ready_epoch": 0, 2024-03-11T20:56:07.307 INFO:teuthology.orchestra.run.smithi023.stdout: "last_epoch_started": 0, 2024-03-11T20:56:07.307 INFO:teuthology.orchestra.run.smithi023.stdout: "last_epoch_clean": 0, 2024-03-11T20:56:07.307 INFO:teuthology.orchestra.run.smithi023.stdout: "source_version": "0'0", 2024-03-11T20:56:07.308 INFO:teuthology.orchestra.run.smithi023.stdout: "target_version": "0'0" 2024-03-11T20:56:07.308 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:56:07.308 INFO:teuthology.orchestra.run.smithi023.stdout: "last_change": "87", 2024-03-11T20:56:07.308 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend": "0", 2024-03-11T20:56:07.308 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T20:56:07.308 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T20:56:07.308 INFO:teuthology.orchestra.run.smithi023.stdout: "auid": 0, 2024-03-11T20:56:07.308 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_mode": "selfmanaged", 2024-03-11T20:56:07.308 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_seq": 2, 2024-03-11T20:56:07.308 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_epoch": 87, 2024-03-11T20:56:07.308 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_snaps": [], 2024-03-11T20:56:07.308 INFO:teuthology.orchestra.run.smithi023.stdout: "removed_snaps": "[]", 2024-03-11T20:56:07.308 INFO:teuthology.orchestra.run.smithi023.stdout: "quota_max_bytes": 0, 2024-03-11T20:56:07.308 INFO:teuthology.orchestra.run.smithi023.stdout: "quota_max_objects": 0, 2024-03-11T20:56:07.308 INFO:teuthology.orchestra.run.smithi023.stdout: "tiers": [], 2024-03-11T20:56:07.308 INFO:teuthology.orchestra.run.smithi023.stdout: "tier_of": -1, 2024-03-11T20:56:07.309 INFO:teuthology.orchestra.run.smithi023.stdout: "read_tier": -1, 2024-03-11T20:56:07.309 INFO:teuthology.orchestra.run.smithi023.stdout: "write_tier": -1, 2024-03-11T20:56:07.309 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_mode": "none", 2024-03-11T20:56:07.309 INFO:teuthology.orchestra.run.smithi023.stdout: "target_max_bytes": 0, 2024-03-11T20:56:07.309 INFO:teuthology.orchestra.run.smithi023.stdout: "target_max_objects": 0, 2024-03-11T20:56:07.309 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T20:56:07.309 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T20:56:07.309 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T20:56:07.309 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_min_flush_age": 0, 2024-03-11T20:56:07.309 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_min_evict_age": 0, 2024-03-11T20:56:07.309 INFO:teuthology.orchestra.run.smithi023.stdout: "erasure_code_profile": "", 2024-03-11T20:56:07.309 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_params": { 2024-03-11T20:56:07.309 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "none" 2024-03-11T20:56:07.309 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:56:07.310 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_period": 0, 2024-03-11T20:56:07.310 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_count": 0, 2024-03-11T20:56:07.310 INFO:teuthology.orchestra.run.smithi023.stdout: "use_gmt_hitset": true, 2024-03-11T20:56:07.310 INFO:teuthology.orchestra.run.smithi023.stdout: "min_read_recency_for_promote": 0, 2024-03-11T20:56:07.310 INFO:teuthology.orchestra.run.smithi023.stdout: "min_write_recency_for_promote": 0, 2024-03-11T20:56:07.310 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T20:56:07.310 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_search_last_n": 0, 2024-03-11T20:56:07.310 INFO:teuthology.orchestra.run.smithi023.stdout: "grade_table": [], 2024-03-11T20:56:07.310 INFO:teuthology.orchestra.run.smithi023.stdout: "stripe_width": 0, 2024-03-11T20:56:07.310 INFO:teuthology.orchestra.run.smithi023.stdout: "expected_num_objects": 0, 2024-03-11T20:56:07.310 INFO:teuthology.orchestra.run.smithi023.stdout: "fast_read": false, 2024-03-11T20:56:07.310 INFO:teuthology.orchestra.run.smithi023.stdout: "options": {}, 2024-03-11T20:56:07.310 INFO:teuthology.orchestra.run.smithi023.stdout: "application_metadata": { 2024-03-11T20:56:07.310 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd": {} 2024-03-11T20:56:07.311 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:56:07.311 INFO:teuthology.orchestra.run.smithi023.stdout: "read_balance": { 2024-03-11T20:56:07.311 INFO:teuthology.orchestra.run.smithi023.stdout: "score_type": "Fair distribution", 2024-03-11T20:56:07.311 INFO:teuthology.orchestra.run.smithi023.stdout: "score_acting": 2.6700000762939453, 2024-03-11T20:56:07.311 INFO:teuthology.orchestra.run.smithi023.stdout: "score_stable": 2.6700000762939453, 2024-03-11T20:56:07.311 INFO:teuthology.orchestra.run.smithi023.stdout: "optimal_score": 0.75, 2024-03-11T20:56:07.311 INFO:teuthology.orchestra.run.smithi023.stdout: "raw_score_acting": 2, 2024-03-11T20:56:07.311 INFO:teuthology.orchestra.run.smithi023.stdout: "raw_score_stable": 2, 2024-03-11T20:56:07.311 INFO:teuthology.orchestra.run.smithi023.stdout: "primary_affinity_weighted": 1, 2024-03-11T20:56:07.311 INFO:teuthology.orchestra.run.smithi023.stdout: "average_primary_affinity": 1, 2024-03-11T20:56:07.311 INFO:teuthology.orchestra.run.smithi023.stdout: "average_primary_affinity_weighted": 1 2024-03-11T20:56:07.311 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:56:07.311 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:56:07.311 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:56:07.311 INFO:teuthology.orchestra.run.smithi023.stdout: "pool": 6, 2024-03-11T20:56:07.312 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_name": "default.rgw.meta", 2024-03-11T20:56:07.312 INFO:teuthology.orchestra.run.smithi023.stdout: "create_time": "2024-03-11T20:50:33.233163+0000", 2024-03-11T20:56:07.312 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-11T20:56:07.312 INFO:teuthology.orchestra.run.smithi023.stdout: "flags_names": "hashpspool", 2024-03-11T20:56:07.312 INFO:teuthology.orchestra.run.smithi023.stdout: "type": 1, 2024-03-11T20:56:07.312 INFO:teuthology.orchestra.run.smithi023.stdout: "size": 3, 2024-03-11T20:56:07.312 INFO:teuthology.orchestra.run.smithi023.stdout: "min_size": 2, 2024-03-11T20:56:07.312 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_rule": 0, 2024-03-11T20:56:07.312 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_count": 0, 2024-03-11T20:56:07.312 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_target": 0, 2024-03-11T20:56:07.312 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T20:56:07.312 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T20:56:07.312 INFO:teuthology.orchestra.run.smithi023.stdout: "object_hash": 2, 2024-03-11T20:56:07.312 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_autoscale_mode": "off", 2024-03-11T20:56:07.312 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num": 32, 2024-03-11T20:56:07.313 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_placement_num": 32, 2024-03-11T20:56:07.313 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_placement_num_target": 32, 2024-03-11T20:56:07.313 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num_target": 32, 2024-03-11T20:56:07.313 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num_pending": 32, 2024-03-11T20:56:07.313 INFO:teuthology.orchestra.run.smithi023.stdout: "last_pg_merge_meta": { 2024-03-11T20:56:07.313 INFO:teuthology.orchestra.run.smithi023.stdout: "source_pgid": "0.0", 2024-03-11T20:56:07.313 INFO:teuthology.orchestra.run.smithi023.stdout: "ready_epoch": 0, 2024-03-11T20:56:07.313 INFO:teuthology.orchestra.run.smithi023.stdout: "last_epoch_started": 0, 2024-03-11T20:56:07.313 INFO:teuthology.orchestra.run.smithi023.stdout: "last_epoch_clean": 0, 2024-03-11T20:56:07.313 INFO:teuthology.orchestra.run.smithi023.stdout: "source_version": "0'0", 2024-03-11T20:56:07.313 INFO:teuthology.orchestra.run.smithi023.stdout: "target_version": "0'0" 2024-03-11T20:56:07.313 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:56:07.313 INFO:teuthology.orchestra.run.smithi023.stdout: "last_change": "84", 2024-03-11T20:56:07.313 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend": "0", 2024-03-11T20:56:07.313 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T20:56:07.314 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T20:56:07.314 INFO:teuthology.orchestra.run.smithi023.stdout: "auid": 0, 2024-03-11T20:56:07.314 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_mode": "selfmanaged", 2024-03-11T20:56:07.314 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_seq": 0, 2024-03-11T20:56:07.314 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_epoch": 0, 2024-03-11T20:56:07.314 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_snaps": [], 2024-03-11T20:56:07.314 INFO:teuthology.orchestra.run.smithi023.stdout: "removed_snaps": "[]", 2024-03-11T20:56:07.314 INFO:teuthology.orchestra.run.smithi023.stdout: "quota_max_bytes": 0, 2024-03-11T20:56:07.314 INFO:teuthology.orchestra.run.smithi023.stdout: "quota_max_objects": 0, 2024-03-11T20:56:07.314 INFO:teuthology.orchestra.run.smithi023.stdout: "tiers": [], 2024-03-11T20:56:07.314 INFO:teuthology.orchestra.run.smithi023.stdout: "tier_of": -1, 2024-03-11T20:56:07.314 INFO:teuthology.orchestra.run.smithi023.stdout: "read_tier": -1, 2024-03-11T20:56:07.314 INFO:teuthology.orchestra.run.smithi023.stdout: "write_tier": -1, 2024-03-11T20:56:07.314 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_mode": "none", 2024-03-11T20:56:07.315 INFO:teuthology.orchestra.run.smithi023.stdout: "target_max_bytes": 0, 2024-03-11T20:56:07.315 INFO:teuthology.orchestra.run.smithi023.stdout: "target_max_objects": 0, 2024-03-11T20:56:07.315 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T20:56:07.315 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T20:56:07.315 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T20:56:07.315 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_min_flush_age": 0, 2024-03-11T20:56:07.315 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_min_evict_age": 0, 2024-03-11T20:56:07.315 INFO:teuthology.orchestra.run.smithi023.stdout: "erasure_code_profile": "", 2024-03-11T20:56:07.315 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_params": { 2024-03-11T20:56:07.315 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "none" 2024-03-11T20:56:07.315 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:56:07.315 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_period": 0, 2024-03-11T20:56:07.315 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_count": 0, 2024-03-11T20:56:07.315 INFO:teuthology.orchestra.run.smithi023.stdout: "use_gmt_hitset": true, 2024-03-11T20:56:07.316 INFO:teuthology.orchestra.run.smithi023.stdout: "min_read_recency_for_promote": 0, 2024-03-11T20:56:07.316 INFO:teuthology.orchestra.run.smithi023.stdout: "min_write_recency_for_promote": 0, 2024-03-11T20:56:07.316 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T20:56:07.316 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_search_last_n": 0, 2024-03-11T20:56:07.316 INFO:teuthology.orchestra.run.smithi023.stdout: "grade_table": [], 2024-03-11T20:56:07.316 INFO:teuthology.orchestra.run.smithi023.stdout: "stripe_width": 0, 2024-03-11T20:56:07.316 INFO:teuthology.orchestra.run.smithi023.stdout: "expected_num_objects": 0, 2024-03-11T20:56:07.316 INFO:teuthology.orchestra.run.smithi023.stdout: "fast_read": false, 2024-03-11T20:56:07.316 INFO:teuthology.orchestra.run.smithi023.stdout: "options": { 2024-03-11T20:56:07.316 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_autoscale_bias": 4 2024-03-11T20:56:07.316 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:56:07.316 INFO:teuthology.orchestra.run.smithi023.stdout: "application_metadata": { 2024-03-11T20:56:07.316 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw": {} 2024-03-11T20:56:07.316 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:56:07.316 INFO:teuthology.orchestra.run.smithi023.stdout: "read_balance": { 2024-03-11T20:56:07.317 INFO:teuthology.orchestra.run.smithi023.stdout: "score_type": "Fair distribution", 2024-03-11T20:56:07.317 INFO:teuthology.orchestra.run.smithi023.stdout: "score_acting": 1.7400000095367432, 2024-03-11T20:56:07.317 INFO:teuthology.orchestra.run.smithi023.stdout: "score_stable": 1.7400000095367432, 2024-03-11T20:56:07.317 INFO:teuthology.orchestra.run.smithi023.stdout: "optimal_score": 0.87999999523162842, 2024-03-11T20:56:07.317 INFO:teuthology.orchestra.run.smithi023.stdout: "raw_score_acting": 1.5299999713897705, 2024-03-11T20:56:07.317 INFO:teuthology.orchestra.run.smithi023.stdout: "raw_score_stable": 1.5299999713897705, 2024-03-11T20:56:07.317 INFO:teuthology.orchestra.run.smithi023.stdout: "primary_affinity_weighted": 1, 2024-03-11T20:56:07.317 INFO:teuthology.orchestra.run.smithi023.stdout: "average_primary_affinity": 1, 2024-03-11T20:56:07.317 INFO:teuthology.orchestra.run.smithi023.stdout: "average_primary_affinity_weighted": 1 2024-03-11T20:56:07.317 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:56:07.317 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:56:07.317 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:56:07.317 INFO:teuthology.orchestra.run.smithi023.stdout: "pool": 7, 2024-03-11T20:56:07.317 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_name": "unique_pool_0", 2024-03-11T20:56:07.317 INFO:teuthology.orchestra.run.smithi023.stdout: "create_time": "2024-03-11T20:54:33.257999+0000", 2024-03-11T20:56:07.318 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-11T20:56:07.318 INFO:teuthology.orchestra.run.smithi023.stdout: "flags_names": "hashpspool", 2024-03-11T20:56:07.318 INFO:teuthology.orchestra.run.smithi023.stdout: "type": 1, 2024-03-11T20:56:07.318 INFO:teuthology.orchestra.run.smithi023.stdout: "size": 3, 2024-03-11T20:56:07.318 INFO:teuthology.orchestra.run.smithi023.stdout: "min_size": 2, 2024-03-11T20:56:07.318 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_rule": 0, 2024-03-11T20:56:07.318 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_count": 0, 2024-03-11T20:56:07.318 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_target": 0, 2024-03-11T20:56:07.318 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T20:56:07.318 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T20:56:07.318 INFO:teuthology.orchestra.run.smithi023.stdout: "object_hash": 2, 2024-03-11T20:56:07.318 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_autoscale_mode": "off", 2024-03-11T20:56:07.318 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num": 16, 2024-03-11T20:56:07.318 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_placement_num": 16, 2024-03-11T20:56:07.318 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_placement_num_target": 16, 2024-03-11T20:56:07.319 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num_target": 16, 2024-03-11T20:56:07.319 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num_pending": 16, 2024-03-11T20:56:07.319 INFO:teuthology.orchestra.run.smithi023.stdout: "last_pg_merge_meta": { 2024-03-11T20:56:07.319 INFO:teuthology.orchestra.run.smithi023.stdout: "source_pgid": "0.0", 2024-03-11T20:56:07.319 INFO:teuthology.orchestra.run.smithi023.stdout: "ready_epoch": 0, 2024-03-11T20:56:07.319 INFO:teuthology.orchestra.run.smithi023.stdout: "last_epoch_started": 0, 2024-03-11T20:56:07.319 INFO:teuthology.orchestra.run.smithi023.stdout: "last_epoch_clean": 0, 2024-03-11T20:56:07.319 INFO:teuthology.orchestra.run.smithi023.stdout: "source_version": "0'0", 2024-03-11T20:56:07.319 INFO:teuthology.orchestra.run.smithi023.stdout: "target_version": "0'0" 2024-03-11T20:56:07.319 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:56:07.319 INFO:teuthology.orchestra.run.smithi023.stdout: "last_change": "127", 2024-03-11T20:56:07.319 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend": "0", 2024-03-11T20:56:07.319 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T20:56:07.319 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T20:56:07.319 INFO:teuthology.orchestra.run.smithi023.stdout: "auid": 0, 2024-03-11T20:56:07.320 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_mode": "selfmanaged", 2024-03-11T20:56:07.320 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_seq": 0, 2024-03-11T20:56:07.320 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_epoch": 0, 2024-03-11T20:56:07.320 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_snaps": [], 2024-03-11T20:56:07.320 INFO:teuthology.orchestra.run.smithi023.stdout: "removed_snaps": "[]", 2024-03-11T20:56:07.320 INFO:teuthology.orchestra.run.smithi023.stdout: "quota_max_bytes": 0, 2024-03-11T20:56:07.320 INFO:teuthology.orchestra.run.smithi023.stdout: "quota_max_objects": 0, 2024-03-11T20:56:07.320 INFO:teuthology.orchestra.run.smithi023.stdout: "tiers": [], 2024-03-11T20:56:07.320 INFO:teuthology.orchestra.run.smithi023.stdout: "tier_of": -1, 2024-03-11T20:56:07.320 INFO:teuthology.orchestra.run.smithi023.stdout: "read_tier": -1, 2024-03-11T20:56:07.320 INFO:teuthology.orchestra.run.smithi023.stdout: "write_tier": -1, 2024-03-11T20:56:07.320 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_mode": "none", 2024-03-11T20:56:07.320 INFO:teuthology.orchestra.run.smithi023.stdout: "target_max_bytes": 0, 2024-03-11T20:56:07.320 INFO:teuthology.orchestra.run.smithi023.stdout: "target_max_objects": 0, 2024-03-11T20:56:07.320 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T20:56:07.321 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T20:56:07.321 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T20:56:07.321 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_min_flush_age": 0, 2024-03-11T20:56:07.321 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_min_evict_age": 0, 2024-03-11T20:56:07.321 INFO:teuthology.orchestra.run.smithi023.stdout: "erasure_code_profile": "", 2024-03-11T20:56:07.321 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_params": { 2024-03-11T20:56:07.321 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "none" 2024-03-11T20:56:07.321 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:56:07.321 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_period": 0, 2024-03-11T20:56:07.321 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_count": 0, 2024-03-11T20:56:07.321 INFO:teuthology.orchestra.run.smithi023.stdout: "use_gmt_hitset": true, 2024-03-11T20:56:07.321 INFO:teuthology.orchestra.run.smithi023.stdout: "min_read_recency_for_promote": 0, 2024-03-11T20:56:07.321 INFO:teuthology.orchestra.run.smithi023.stdout: "min_write_recency_for_promote": 0, 2024-03-11T20:56:07.321 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T20:56:07.321 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_search_last_n": 0, 2024-03-11T20:56:07.322 INFO:teuthology.orchestra.run.smithi023.stdout: "grade_table": [], 2024-03-11T20:56:07.322 INFO:teuthology.orchestra.run.smithi023.stdout: "stripe_width": 0, 2024-03-11T20:56:07.322 INFO:teuthology.orchestra.run.smithi023.stdout: "expected_num_objects": 0, 2024-03-11T20:56:07.322 INFO:teuthology.orchestra.run.smithi023.stdout: "fast_read": false, 2024-03-11T20:56:07.322 INFO:teuthology.orchestra.run.smithi023.stdout: "options": {}, 2024-03-11T20:56:07.322 INFO:teuthology.orchestra.run.smithi023.stdout: "application_metadata": { 2024-03-11T20:56:07.322 INFO:teuthology.orchestra.run.smithi023.stdout: "rados": {} 2024-03-11T20:56:07.322 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:56:07.322 INFO:teuthology.orchestra.run.smithi023.stdout: "read_balance": { 2024-03-11T20:56:07.322 INFO:teuthology.orchestra.run.smithi023.stdout: "score_type": "Fair distribution", 2024-03-11T20:56:07.322 INFO:teuthology.orchestra.run.smithi023.stdout: "score_acting": 1.9900000095367432, 2024-03-11T20:56:07.322 INFO:teuthology.orchestra.run.smithi023.stdout: "score_stable": 1.9900000095367432, 2024-03-11T20:56:07.322 INFO:teuthology.orchestra.run.smithi023.stdout: "optimal_score": 0.87999999523162842, 2024-03-11T20:56:07.322 INFO:teuthology.orchestra.run.smithi023.stdout: "raw_score_acting": 1.75, 2024-03-11T20:56:07.322 INFO:teuthology.orchestra.run.smithi023.stdout: "raw_score_stable": 1.75, 2024-03-11T20:56:07.323 INFO:teuthology.orchestra.run.smithi023.stdout: "primary_affinity_weighted": 1, 2024-03-11T20:56:07.323 INFO:teuthology.orchestra.run.smithi023.stdout: "average_primary_affinity": 1, 2024-03-11T20:56:07.323 INFO:teuthology.orchestra.run.smithi023.stdout: "average_primary_affinity_weighted": 1 2024-03-11T20:56:07.323 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:56:07.323 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:56:07.323 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-11T20:56:07.323 INFO:teuthology.orchestra.run.smithi023.stdout: "osds": [ 2024-03-11T20:56:07.323 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:56:07.323 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": 0, 2024-03-11T20:56:07.323 INFO:teuthology.orchestra.run.smithi023.stdout: "uuid": "3df72b4a-8fe4-4265-b4c6-af022e1e23e1", 2024-03-11T20:56:07.323 INFO:teuthology.orchestra.run.smithi023.stdout: "up": 1, 2024-03-11T20:56:07.323 INFO:teuthology.orchestra.run.smithi023.stdout: "in": 1, 2024-03-11T20:56:07.323 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 1, 2024-03-11T20:56:07.323 INFO:teuthology.orchestra.run.smithi023.stdout: "primary_affinity": 1, 2024-03-11T20:56:07.323 INFO:teuthology.orchestra.run.smithi023.stdout: "last_clean_begin": 0, 2024-03-11T20:56:07.324 INFO:teuthology.orchestra.run.smithi023.stdout: "last_clean_end": 0, 2024-03-11T20:56:07.324 INFO:teuthology.orchestra.run.smithi023.stdout: "up_from": 8, 2024-03-11T20:56:07.324 INFO:teuthology.orchestra.run.smithi023.stdout: "up_thru": 138, 2024-03-11T20:56:07.324 INFO:teuthology.orchestra.run.smithi023.stdout: "down_at": 0, 2024-03-11T20:56:07.324 INFO:teuthology.orchestra.run.smithi023.stdout: "lost_at": 0, 2024-03-11T20:56:07.324 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2024-03-11T20:56:07.324 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T20:56:07.324 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:56:07.324 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T20:56:07.324 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6802", 2024-03-11T20:56:07.324 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 2154627393 2024-03-11T20:56:07.324 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:56:07.324 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:56:07.324 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T20:56:07.324 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6803", 2024-03-11T20:56:07.325 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 2154627393 2024-03-11T20:56:07.325 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:56:07.325 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T20:56:07.325 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:56:07.325 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_addrs": { 2024-03-11T20:56:07.325 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T20:56:07.325 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:56:07.325 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T20:56:07.325 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6804", 2024-03-11T20:56:07.325 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 2154627393 2024-03-11T20:56:07.325 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:56:07.325 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:56:07.325 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T20:56:07.325 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6805", 2024-03-11T20:56:07.325 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 2154627393 2024-03-11T20:56:07.326 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:56:07.326 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T20:56:07.326 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:56:07.326 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_back_addrs": { 2024-03-11T20:56:07.326 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T20:56:07.326 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:56:07.326 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T20:56:07.326 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6808", 2024-03-11T20:56:07.326 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 2154627393 2024-03-11T20:56:07.326 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:56:07.326 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:56:07.326 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T20:56:07.326 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6809", 2024-03-11T20:56:07.326 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 2154627393 2024-03-11T20:56:07.326 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:56:07.327 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T20:56:07.327 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:56:07.327 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_front_addrs": { 2024-03-11T20:56:07.327 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T20:56:07.327 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:56:07.327 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T20:56:07.327 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6806", 2024-03-11T20:56:07.327 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 2154627393 2024-03-11T20:56:07.327 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:56:07.327 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:56:07.327 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T20:56:07.327 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6807", 2024-03-11T20:56:07.327 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 2154627393 2024-03-11T20:56:07.327 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:56:07.327 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T20:56:07.328 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:56:07.328 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6803/2154627393", 2024-03-11T20:56:07.328 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_addr": "172.21.15.23:6805/2154627393", 2024-03-11T20:56:07.328 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_back_addr": "172.21.15.23:6809/2154627393", 2024-03-11T20:56:07.328 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_front_addr": "172.21.15.23:6807/2154627393", 2024-03-11T20:56:07.328 INFO:teuthology.orchestra.run.smithi023.stdout: "state": [ 2024-03-11T20:56:07.328 INFO:teuthology.orchestra.run.smithi023.stdout: "exists", 2024-03-11T20:56:07.328 INFO:teuthology.orchestra.run.smithi023.stdout: "up" 2024-03-11T20:56:07.328 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T20:56:07.328 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:56:07.328 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:56:07.328 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": 1, 2024-03-11T20:56:07.328 INFO:teuthology.orchestra.run.smithi023.stdout: "uuid": "e8fd7500-9081-41a1-8077-a4d0f0e12a2c", 2024-03-11T20:56:07.328 INFO:teuthology.orchestra.run.smithi023.stdout: "up": 1, 2024-03-11T20:56:07.328 INFO:teuthology.orchestra.run.smithi023.stdout: "in": 0, 2024-03-11T20:56:07.329 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2024-03-11T20:56:07.329 INFO:teuthology.orchestra.run.smithi023.stdout: "primary_affinity": 1, 2024-03-11T20:56:07.329 INFO:teuthology.orchestra.run.smithi023.stdout: "last_clean_begin": 0, 2024-03-11T20:56:07.329 INFO:teuthology.orchestra.run.smithi023.stdout: "last_clean_end": 0, 2024-03-11T20:56:07.329 INFO:teuthology.orchestra.run.smithi023.stdout: "up_from": 13, 2024-03-11T20:56:07.329 INFO:teuthology.orchestra.run.smithi023.stdout: "up_thru": 121, 2024-03-11T20:56:07.329 INFO:teuthology.orchestra.run.smithi023.stdout: "down_at": 0, 2024-03-11T20:56:07.329 INFO:teuthology.orchestra.run.smithi023.stdout: "lost_at": 0, 2024-03-11T20:56:07.329 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2024-03-11T20:56:07.329 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T20:56:07.329 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:56:07.329 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T20:56:07.329 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6810", 2024-03-11T20:56:07.329 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 2506707705 2024-03-11T20:56:07.329 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:56:07.330 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:56:07.330 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T20:56:07.330 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6811", 2024-03-11T20:56:07.330 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 2506707705 2024-03-11T20:56:07.330 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:56:07.330 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T20:56:07.330 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:56:07.330 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_addrs": { 2024-03-11T20:56:07.330 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T20:56:07.330 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:56:07.330 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T20:56:07.330 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6812", 2024-03-11T20:56:07.330 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 2506707705 2024-03-11T20:56:07.330 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:56:07.331 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:56:07.331 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T20:56:07.331 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6813", 2024-03-11T20:56:07.331 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 2506707705 2024-03-11T20:56:07.331 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:56:07.331 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T20:56:07.331 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:56:07.331 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_back_addrs": { 2024-03-11T20:56:07.331 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T20:56:07.331 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:56:07.331 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T20:56:07.331 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6816", 2024-03-11T20:56:07.331 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 2506707705 2024-03-11T20:56:07.331 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:56:07.331 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:56:07.331 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T20:56:07.332 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6817", 2024-03-11T20:56:07.332 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 2506707705 2024-03-11T20:56:07.332 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:56:07.332 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T20:56:07.332 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:56:07.332 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_front_addrs": { 2024-03-11T20:56:07.332 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T20:56:07.332 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:56:07.332 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T20:56:07.332 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6814", 2024-03-11T20:56:07.332 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 2506707705 2024-03-11T20:56:07.332 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:56:07.332 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:56:07.332 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T20:56:07.333 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6815", 2024-03-11T20:56:07.333 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 2506707705 2024-03-11T20:56:07.333 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:56:07.333 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T20:56:07.333 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:56:07.333 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6811/2506707705", 2024-03-11T20:56:07.333 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_addr": "172.21.15.23:6813/2506707705", 2024-03-11T20:56:07.333 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_back_addr": "172.21.15.23:6817/2506707705", 2024-03-11T20:56:07.333 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_front_addr": "172.21.15.23:6815/2506707705", 2024-03-11T20:56:07.333 INFO:teuthology.orchestra.run.smithi023.stdout: "state": [ 2024-03-11T20:56:07.333 INFO:teuthology.orchestra.run.smithi023.stdout: "exists", 2024-03-11T20:56:07.333 INFO:teuthology.orchestra.run.smithi023.stdout: "up" 2024-03-11T20:56:07.333 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T20:56:07.334 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:56:07.334 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:56:07.334 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": 2, 2024-03-11T20:56:07.334 INFO:teuthology.orchestra.run.smithi023.stdout: "uuid": "dd85d52a-df48-4b03-8612-cecfd82b5987", 2024-03-11T20:56:07.334 INFO:teuthology.orchestra.run.smithi023.stdout: "up": 1, 2024-03-11T20:56:07.334 INFO:teuthology.orchestra.run.smithi023.stdout: "in": 1, 2024-03-11T20:56:07.334 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 1, 2024-03-11T20:56:07.334 INFO:teuthology.orchestra.run.smithi023.stdout: "primary_affinity": 1, 2024-03-11T20:56:07.334 INFO:teuthology.orchestra.run.smithi023.stdout: "last_clean_begin": 0, 2024-03-11T20:56:07.334 INFO:teuthology.orchestra.run.smithi023.stdout: "last_clean_end": 0, 2024-03-11T20:56:07.334 INFO:teuthology.orchestra.run.smithi023.stdout: "up_from": 19, 2024-03-11T20:56:07.334 INFO:teuthology.orchestra.run.smithi023.stdout: "up_thru": 138, 2024-03-11T20:56:07.334 INFO:teuthology.orchestra.run.smithi023.stdout: "down_at": 0, 2024-03-11T20:56:07.334 INFO:teuthology.orchestra.run.smithi023.stdout: "lost_at": 0, 2024-03-11T20:56:07.335 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2024-03-11T20:56:07.335 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T20:56:07.335 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:56:07.335 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T20:56:07.335 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6818", 2024-03-11T20:56:07.335 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1770748056 2024-03-11T20:56:07.335 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:56:07.335 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:56:07.335 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T20:56:07.335 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6819", 2024-03-11T20:56:07.335 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1770748056 2024-03-11T20:56:07.335 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:56:07.335 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T20:56:07.335 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:56:07.335 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_addrs": { 2024-03-11T20:56:07.336 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T20:56:07.336 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:56:07.336 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T20:56:07.336 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6820", 2024-03-11T20:56:07.336 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1770748056 2024-03-11T20:56:07.336 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:56:07.336 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:56:07.336 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T20:56:07.336 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6821", 2024-03-11T20:56:07.336 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1770748056 2024-03-11T20:56:07.336 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:56:07.336 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T20:56:07.336 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:56:07.336 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_back_addrs": { 2024-03-11T20:56:07.336 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T20:56:07.337 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:56:07.337 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T20:56:07.337 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6824", 2024-03-11T20:56:07.337 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1770748056 2024-03-11T20:56:07.337 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:56:07.337 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:56:07.337 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T20:56:07.337 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6825", 2024-03-11T20:56:07.337 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1770748056 2024-03-11T20:56:07.337 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:56:07.337 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T20:56:07.337 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:56:07.337 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_front_addrs": { 2024-03-11T20:56:07.337 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T20:56:07.337 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:56:07.338 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T20:56:07.338 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6822", 2024-03-11T20:56:07.338 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1770748056 2024-03-11T20:56:07.338 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:56:07.338 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:56:07.338 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T20:56:07.338 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6823", 2024-03-11T20:56:07.338 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1770748056 2024-03-11T20:56:07.338 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:56:07.338 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T20:56:07.338 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:56:07.338 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6819/1770748056", 2024-03-11T20:56:07.338 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_addr": "172.21.15.23:6821/1770748056", 2024-03-11T20:56:07.338 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_back_addr": "172.21.15.23:6825/1770748056", 2024-03-11T20:56:07.339 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_front_addr": "172.21.15.23:6823/1770748056", 2024-03-11T20:56:07.339 INFO:teuthology.orchestra.run.smithi023.stdout: "state": [ 2024-03-11T20:56:07.339 INFO:teuthology.orchestra.run.smithi023.stdout: "exists", 2024-03-11T20:56:07.339 INFO:teuthology.orchestra.run.smithi023.stdout: "up" 2024-03-11T20:56:07.339 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T20:56:07.339 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:56:07.339 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:56:07.339 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": 3, 2024-03-11T20:56:07.339 INFO:teuthology.orchestra.run.smithi023.stdout: "uuid": "e3021106-3779-4b94-9bbf-a7724544110f", 2024-03-11T20:56:07.339 INFO:teuthology.orchestra.run.smithi023.stdout: "up": 1, 2024-03-11T20:56:07.339 INFO:teuthology.orchestra.run.smithi023.stdout: "in": 1, 2024-03-11T20:56:07.339 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 1, 2024-03-11T20:56:07.339 INFO:teuthology.orchestra.run.smithi023.stdout: "primary_affinity": 1, 2024-03-11T20:56:07.339 INFO:teuthology.orchestra.run.smithi023.stdout: "last_clean_begin": 0, 2024-03-11T20:56:07.339 INFO:teuthology.orchestra.run.smithi023.stdout: "last_clean_end": 0, 2024-03-11T20:56:07.340 INFO:teuthology.orchestra.run.smithi023.stdout: "up_from": 28, 2024-03-11T20:56:07.340 INFO:teuthology.orchestra.run.smithi023.stdout: "up_thru": 124, 2024-03-11T20:56:07.340 INFO:teuthology.orchestra.run.smithi023.stdout: "down_at": 0, 2024-03-11T20:56:07.340 INFO:teuthology.orchestra.run.smithi023.stdout: "lost_at": 0, 2024-03-11T20:56:07.340 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2024-03-11T20:56:07.340 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T20:56:07.340 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:56:07.340 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T20:56:07.340 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6826", 2024-03-11T20:56:07.340 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 938496814 2024-03-11T20:56:07.340 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:56:07.340 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:56:07.340 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T20:56:07.340 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6827", 2024-03-11T20:56:07.341 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 938496814 2024-03-11T20:56:07.341 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:56:07.341 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T20:56:07.341 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:56:07.341 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_addrs": { 2024-03-11T20:56:07.341 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T20:56:07.341 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:56:07.341 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T20:56:07.341 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6828", 2024-03-11T20:56:07.341 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 938496814 2024-03-11T20:56:07.341 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:56:07.341 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:56:07.341 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T20:56:07.341 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6829", 2024-03-11T20:56:07.341 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 938496814 2024-03-11T20:56:07.342 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:56:07.342 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T20:56:07.342 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:56:07.342 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_back_addrs": { 2024-03-11T20:56:07.342 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T20:56:07.342 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:56:07.342 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T20:56:07.342 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6832", 2024-03-11T20:56:07.342 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 938496814 2024-03-11T20:56:07.342 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:56:07.342 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:56:07.342 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T20:56:07.342 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6833", 2024-03-11T20:56:07.342 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 938496814 2024-03-11T20:56:07.343 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:56:07.343 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T20:56:07.343 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:56:07.343 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_front_addrs": { 2024-03-11T20:56:07.343 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T20:56:07.343 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:56:07.343 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T20:56:07.343 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6830", 2024-03-11T20:56:07.343 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 938496814 2024-03-11T20:56:07.343 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:56:07.343 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:56:07.343 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T20:56:07.343 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6831", 2024-03-11T20:56:07.344 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 938496814 2024-03-11T20:56:07.344 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:56:07.344 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T20:56:07.344 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:56:07.344 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6827/938496814", 2024-03-11T20:56:07.344 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_addr": "172.21.15.23:6829/938496814", 2024-03-11T20:56:07.344 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_back_addr": "172.21.15.23:6833/938496814", 2024-03-11T20:56:07.344 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_front_addr": "172.21.15.23:6831/938496814", 2024-03-11T20:56:07.344 INFO:teuthology.orchestra.run.smithi023.stdout: "state": [ 2024-03-11T20:56:07.344 INFO:teuthology.orchestra.run.smithi023.stdout: "exists", 2024-03-11T20:56:07.344 INFO:teuthology.orchestra.run.smithi023.stdout: "up" 2024-03-11T20:56:07.344 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T20:56:07.344 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:56:07.344 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:56:07.344 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": 4, 2024-03-11T20:56:07.345 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:56:07.346 INFO:teuthology.orchestra.run.smithi023.stdout: "uuid": "4dcbc91e-7035-413d-97bc-c9d804eb16d7", 2024-03-11T20:56:07.346 INFO:teuthology.orchestra.run.smithi023.stdout: "up": 1, 2024-03-11T20:56:07.346 INFO:teuthology.orchestra.run.smithi023.stdout: "in": 1, 2024-03-11T20:56:07.346 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 1, 2024-03-11T20:56:07.346 INFO:teuthology.orchestra.run.smithi023.stdout: "primary_affinity": 1, 2024-03-11T20:56:07.346 INFO:teuthology.orchestra.run.smithi023.stdout: "last_clean_begin": 0, 2024-03-11T20:56:07.346 INFO:teuthology.orchestra.run.smithi023.stdout: "last_clean_end": 0, 2024-03-11T20:56:07.346 INFO:teuthology.orchestra.run.smithi023.stdout: "up_from": 35, 2024-03-11T20:56:07.346 INFO:teuthology.orchestra.run.smithi023.stdout: "up_thru": 124, 2024-03-11T20:56:07.346 INFO:teuthology.orchestra.run.smithi023.stdout: "down_at": 0, 2024-03-11T20:56:07.346 INFO:teuthology.orchestra.run.smithi023.stdout: "lost_at": 0, 2024-03-11T20:56:07.346 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2024-03-11T20:56:07.347 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T20:56:07.347 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:56:07.347 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T20:56:07.347 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6800", 2024-03-11T20:56:07.347 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 615746847 2024-03-11T20:56:07.347 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:56:07.347 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:56:07.347 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T20:56:07.347 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6801", 2024-03-11T20:56:07.347 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 615746847 2024-03-11T20:56:07.347 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:56:07.347 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T20:56:07.347 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:56:07.347 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_addrs": { 2024-03-11T20:56:07.348 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T20:56:07.348 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:56:07.348 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T20:56:07.348 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6802", 2024-03-11T20:56:07.348 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 615746847 2024-03-11T20:56:07.348 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:56:07.348 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:56:07.348 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T20:56:07.348 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6803", 2024-03-11T20:56:07.348 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 615746847 2024-03-11T20:56:07.348 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:56:07.348 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T20:56:07.348 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:56:07.348 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_back_addrs": { 2024-03-11T20:56:07.348 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T20:56:07.349 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:56:07.349 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T20:56:07.349 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6806", 2024-03-11T20:56:07.349 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 615746847 2024-03-11T20:56:07.349 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:56:07.349 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:56:07.349 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T20:56:07.349 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6807", 2024-03-11T20:56:07.349 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 615746847 2024-03-11T20:56:07.349 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:56:07.349 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T20:56:07.349 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:56:07.349 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_front_addrs": { 2024-03-11T20:56:07.349 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T20:56:07.349 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:56:07.350 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T20:56:07.350 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6804", 2024-03-11T20:56:07.350 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 615746847 2024-03-11T20:56:07.350 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:56:07.350 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:56:07.350 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T20:56:07.350 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6805", 2024-03-11T20:56:07.350 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 615746847 2024-03-11T20:56:07.350 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:56:07.350 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T20:56:07.350 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:56:07.350 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.146:6801/615746847", 2024-03-11T20:56:07.350 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_addr": "172.21.15.146:6803/615746847", 2024-03-11T20:56:07.350 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_back_addr": "172.21.15.146:6807/615746847", 2024-03-11T20:56:07.350 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_front_addr": "172.21.15.146:6805/615746847", 2024-03-11T20:56:07.351 INFO:teuthology.orchestra.run.smithi023.stdout: "state": [ 2024-03-11T20:56:07.351 INFO:teuthology.orchestra.run.smithi023.stdout: "exists", 2024-03-11T20:56:07.351 INFO:teuthology.orchestra.run.smithi023.stdout: "up" 2024-03-11T20:56:07.351 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T20:56:07.351 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:56:07.351 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:56:07.351 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": 5, 2024-03-11T20:56:07.351 INFO:teuthology.orchestra.run.smithi023.stdout: "uuid": "aefcb766-8bae-4ff4-b71e-6691c8ac02f1", 2024-03-11T20:56:07.351 INFO:teuthology.orchestra.run.smithi023.stdout: "up": 1, 2024-03-11T20:56:07.351 INFO:teuthology.orchestra.run.smithi023.stdout: "in": 1, 2024-03-11T20:56:07.351 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 1, 2024-03-11T20:56:07.351 INFO:teuthology.orchestra.run.smithi023.stdout: "primary_affinity": 1, 2024-03-11T20:56:07.351 INFO:teuthology.orchestra.run.smithi023.stdout: "last_clean_begin": 0, 2024-03-11T20:56:07.351 INFO:teuthology.orchestra.run.smithi023.stdout: "last_clean_end": 0, 2024-03-11T20:56:07.351 INFO:teuthology.orchestra.run.smithi023.stdout: "up_from": 44, 2024-03-11T20:56:07.352 INFO:teuthology.orchestra.run.smithi023.stdout: "up_thru": 124, 2024-03-11T20:56:07.352 INFO:teuthology.orchestra.run.smithi023.stdout: "down_at": 0, 2024-03-11T20:56:07.352 INFO:teuthology.orchestra.run.smithi023.stdout: "lost_at": 0, 2024-03-11T20:56:07.352 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2024-03-11T20:56:07.352 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T20:56:07.352 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:56:07.352 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T20:56:07.352 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6808", 2024-03-11T20:56:07.352 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 4003230814 2024-03-11T20:56:07.352 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:56:07.352 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:56:07.352 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T20:56:07.352 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6809", 2024-03-11T20:56:07.352 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 4003230814 2024-03-11T20:56:07.352 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:56:07.353 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T20:56:07.353 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:56:07.353 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_addrs": { 2024-03-11T20:56:07.353 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T20:56:07.353 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:56:07.353 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T20:56:07.353 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6810", 2024-03-11T20:56:07.353 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 4003230814 2024-03-11T20:56:07.353 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:56:07.353 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:56:07.353 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T20:56:07.353 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6811", 2024-03-11T20:56:07.353 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 4003230814 2024-03-11T20:56:07.353 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:56:07.353 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T20:56:07.354 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:56:07.354 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_back_addrs": { 2024-03-11T20:56:07.354 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T20:56:07.354 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:56:07.354 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T20:56:07.354 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6814", 2024-03-11T20:56:07.354 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 4003230814 2024-03-11T20:56:07.354 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:56:07.354 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:56:07.354 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T20:56:07.354 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6815", 2024-03-11T20:56:07.354 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 4003230814 2024-03-11T20:56:07.354 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:56:07.354 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T20:56:07.355 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:56:07.355 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_front_addrs": { 2024-03-11T20:56:07.355 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T20:56:07.355 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:56:07.355 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T20:56:07.355 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6812", 2024-03-11T20:56:07.355 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 4003230814 2024-03-11T20:56:07.355 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:56:07.355 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:56:07.355 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T20:56:07.355 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6813", 2024-03-11T20:56:07.355 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 4003230814 2024-03-11T20:56:07.355 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:56:07.355 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T20:56:07.355 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:56:07.356 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.146:6809/4003230814", 2024-03-11T20:56:07.356 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_addr": "172.21.15.146:6811/4003230814", 2024-03-11T20:56:07.356 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_back_addr": "172.21.15.146:6815/4003230814", 2024-03-11T20:56:07.356 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_front_addr": "172.21.15.146:6813/4003230814", 2024-03-11T20:56:07.356 INFO:teuthology.orchestra.run.smithi023.stdout: "state": [ 2024-03-11T20:56:07.356 INFO:teuthology.orchestra.run.smithi023.stdout: "exists", 2024-03-11T20:56:07.356 INFO:teuthology.orchestra.run.smithi023.stdout: "up" 2024-03-11T20:56:07.356 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T20:56:07.356 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:56:07.356 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:56:07.356 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": 6, 2024-03-11T20:56:07.356 INFO:teuthology.orchestra.run.smithi023.stdout: "uuid": "5cb151e8-1f9b-4617-8a86-adfb06bb9606", 2024-03-11T20:56:07.356 INFO:teuthology.orchestra.run.smithi023.stdout: "up": 1, 2024-03-11T20:56:07.356 INFO:teuthology.orchestra.run.smithi023.stdout: "in": 1, 2024-03-11T20:56:07.356 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 1, 2024-03-11T20:56:07.357 INFO:teuthology.orchestra.run.smithi023.stdout: "primary_affinity": 1, 2024-03-11T20:56:07.357 INFO:teuthology.orchestra.run.smithi023.stdout: "last_clean_begin": 0, 2024-03-11T20:56:07.357 INFO:teuthology.orchestra.run.smithi023.stdout: "last_clean_end": 0, 2024-03-11T20:56:07.357 INFO:teuthology.orchestra.run.smithi023.stdout: "up_from": 57, 2024-03-11T20:56:07.357 INFO:teuthology.orchestra.run.smithi023.stdout: "up_thru": 124, 2024-03-11T20:56:07.357 INFO:teuthology.orchestra.run.smithi023.stdout: "down_at": 0, 2024-03-11T20:56:07.357 INFO:teuthology.orchestra.run.smithi023.stdout: "lost_at": 0, 2024-03-11T20:56:07.357 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2024-03-11T20:56:07.357 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T20:56:07.357 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:56:07.357 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T20:56:07.357 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6816", 2024-03-11T20:56:07.357 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 3334092376 2024-03-11T20:56:07.357 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:56:07.357 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:56:07.357 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T20:56:07.358 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6817", 2024-03-11T20:56:07.358 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 3334092376 2024-03-11T20:56:07.358 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:56:07.358 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T20:56:07.358 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:56:07.358 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_addrs": { 2024-03-11T20:56:07.358 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T20:56:07.358 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:56:07.358 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T20:56:07.358 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6818", 2024-03-11T20:56:07.358 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 3334092376 2024-03-11T20:56:07.358 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:56:07.358 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:56:07.358 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T20:56:07.359 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6819", 2024-03-11T20:56:07.359 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 3334092376 2024-03-11T20:56:07.359 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:56:07.359 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T20:56:07.359 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:56:07.359 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_back_addrs": { 2024-03-11T20:56:07.359 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T20:56:07.359 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:56:07.359 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T20:56:07.359 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6822", 2024-03-11T20:56:07.359 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 3334092376 2024-03-11T20:56:07.359 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:56:07.359 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:56:07.359 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T20:56:07.359 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6823", 2024-03-11T20:56:07.360 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 3334092376 2024-03-11T20:56:07.360 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:56:07.360 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T20:56:07.360 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:56:07.360 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_front_addrs": { 2024-03-11T20:56:07.360 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T20:56:07.360 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:56:07.360 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T20:56:07.360 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6820", 2024-03-11T20:56:07.360 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 3334092376 2024-03-11T20:56:07.360 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:56:07.360 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:56:07.360 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T20:56:07.360 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6821", 2024-03-11T20:56:07.360 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 3334092376 2024-03-11T20:56:07.360 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:56:07.361 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T20:56:07.361 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:56:07.361 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.146:6817/3334092376", 2024-03-11T20:56:07.361 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_addr": "172.21.15.146:6819/3334092376", 2024-03-11T20:56:07.361 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_back_addr": "172.21.15.146:6823/3334092376", 2024-03-11T20:56:07.361 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_front_addr": "172.21.15.146:6821/3334092376", 2024-03-11T20:56:07.361 INFO:teuthology.orchestra.run.smithi023.stdout: "state": [ 2024-03-11T20:56:07.361 INFO:teuthology.orchestra.run.smithi023.stdout: "exists", 2024-03-11T20:56:07.361 INFO:teuthology.orchestra.run.smithi023.stdout: "up" 2024-03-11T20:56:07.361 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T20:56:07.361 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:56:07.361 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:56:07.361 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": 7, 2024-03-11T20:56:07.361 INFO:teuthology.orchestra.run.smithi023.stdout: "uuid": "bdbd11a2-fa09-4f82-93a2-6b2d9c750bfd", 2024-03-11T20:56:07.361 INFO:teuthology.orchestra.run.smithi023.stdout: "up": 1, 2024-03-11T20:56:07.362 INFO:teuthology.orchestra.run.smithi023.stdout: "in": 1, 2024-03-11T20:56:07.362 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 1, 2024-03-11T20:56:07.362 INFO:teuthology.orchestra.run.smithi023.stdout: "primary_affinity": 1, 2024-03-11T20:56:07.362 INFO:teuthology.orchestra.run.smithi023.stdout: "last_clean_begin": 0, 2024-03-11T20:56:07.362 INFO:teuthology.orchestra.run.smithi023.stdout: "last_clean_end": 0, 2024-03-11T20:56:07.362 INFO:teuthology.orchestra.run.smithi023.stdout: "up_from": 68, 2024-03-11T20:56:07.362 INFO:teuthology.orchestra.run.smithi023.stdout: "up_thru": 124, 2024-03-11T20:56:07.362 INFO:teuthology.orchestra.run.smithi023.stdout: "down_at": 0, 2024-03-11T20:56:07.362 INFO:teuthology.orchestra.run.smithi023.stdout: "lost_at": 0, 2024-03-11T20:56:07.362 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2024-03-11T20:56:07.362 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T20:56:07.362 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:56:07.362 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T20:56:07.362 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6824", 2024-03-11T20:56:07.362 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1678407923 2024-03-11T20:56:07.363 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:56:07.363 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:56:07.363 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T20:56:07.363 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6825", 2024-03-11T20:56:07.363 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1678407923 2024-03-11T20:56:07.363 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:56:07.363 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T20:56:07.363 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:56:07.363 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_addrs": { 2024-03-11T20:56:07.363 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T20:56:07.363 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:56:07.363 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T20:56:07.363 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6826", 2024-03-11T20:56:07.363 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1678407923 2024-03-11T20:56:07.363 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:56:07.364 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:56:07.364 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T20:56:07.364 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6827", 2024-03-11T20:56:07.364 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1678407923 2024-03-11T20:56:07.364 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:56:07.364 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T20:56:07.364 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:56:07.364 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_back_addrs": { 2024-03-11T20:56:07.364 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T20:56:07.364 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:56:07.364 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T20:56:07.364 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6830", 2024-03-11T20:56:07.364 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1678407923 2024-03-11T20:56:07.364 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:56:07.364 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:56:07.364 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T20:56:07.365 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6831", 2024-03-11T20:56:07.365 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1678407923 2024-03-11T20:56:07.365 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:56:07.365 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T20:56:07.365 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:56:07.365 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_front_addrs": { 2024-03-11T20:56:07.365 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T20:56:07.365 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:56:07.365 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T20:56:07.365 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6828", 2024-03-11T20:56:07.365 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1678407923 2024-03-11T20:56:07.365 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:56:07.365 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:56:07.365 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T20:56:07.365 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6829", 2024-03-11T20:56:07.366 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1678407923 2024-03-11T20:56:07.366 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:56:07.366 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T20:56:07.366 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:56:07.366 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.146:6825/1678407923", 2024-03-11T20:56:07.366 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_addr": "172.21.15.146:6827/1678407923", 2024-03-11T20:56:07.366 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_back_addr": "172.21.15.146:6831/1678407923", 2024-03-11T20:56:07.366 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_front_addr": "172.21.15.146:6829/1678407923", 2024-03-11T20:56:07.366 INFO:teuthology.orchestra.run.smithi023.stdout: "state": [ 2024-03-11T20:56:07.366 INFO:teuthology.orchestra.run.smithi023.stdout: "exists", 2024-03-11T20:56:07.366 INFO:teuthology.orchestra.run.smithi023.stdout: "up" 2024-03-11T20:56:07.366 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T20:56:07.366 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:56:07.366 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-11T20:56:07.367 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_xinfo": [ 2024-03-11T20:56:07.367 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:56:07.367 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": 0, 2024-03-11T20:56:07.367 INFO:teuthology.orchestra.run.smithi023.stdout: "down_stamp": "0.000000", 2024-03-11T20:56:07.367 INFO:teuthology.orchestra.run.smithi023.stdout: "laggy_probability": 0, 2024-03-11T20:56:07.367 INFO:teuthology.orchestra.run.smithi023.stdout: "laggy_interval": 0, 2024-03-11T20:56:07.367 INFO:teuthology.orchestra.run.smithi023.stdout: "features": 4540701547738038271, 2024-03-11T20:56:07.367 INFO:teuthology.orchestra.run.smithi023.stdout: "old_weight": 0, 2024-03-11T20:56:07.367 INFO:teuthology.orchestra.run.smithi023.stdout: "last_purged_snaps_scrub": "2024-03-11T20:55:20.780923+0000", 2024-03-11T20:56:07.367 INFO:teuthology.orchestra.run.smithi023.stdout: "dead_epoch": 0 2024-03-11T20:56:07.367 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:56:07.367 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:56:07.367 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": 1, 2024-03-11T20:56:07.367 INFO:teuthology.orchestra.run.smithi023.stdout: "down_stamp": "0.000000", 2024-03-11T20:56:07.368 INFO:teuthology.orchestra.run.smithi023.stdout: "laggy_probability": 0, 2024-03-11T20:56:07.368 INFO:teuthology.orchestra.run.smithi023.stdout: "laggy_interval": 0, 2024-03-11T20:56:07.368 INFO:teuthology.orchestra.run.smithi023.stdout: "features": 4540701547738038271, 2024-03-11T20:56:07.368 INFO:teuthology.orchestra.run.smithi023.stdout: "old_weight": 65536, 2024-03-11T20:56:07.368 INFO:teuthology.orchestra.run.smithi023.stdout: "last_purged_snaps_scrub": "2024-03-11T20:55:41.137095+0000", 2024-03-11T20:56:07.368 INFO:teuthology.orchestra.run.smithi023.stdout: "dead_epoch": 0 2024-03-11T20:56:07.368 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:56:07.368 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:56:07.368 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": 2, 2024-03-11T20:56:07.368 INFO:teuthology.orchestra.run.smithi023.stdout: "down_stamp": "0.000000", 2024-03-11T20:56:07.368 INFO:teuthology.orchestra.run.smithi023.stdout: "laggy_probability": 0, 2024-03-11T20:56:07.368 INFO:teuthology.orchestra.run.smithi023.stdout: "laggy_interval": 0, 2024-03-11T20:56:07.368 INFO:teuthology.orchestra.run.smithi023.stdout: "features": 4540701547738038271, 2024-03-11T20:56:07.368 INFO:teuthology.orchestra.run.smithi023.stdout: "old_weight": 0, 2024-03-11T20:56:07.368 INFO:teuthology.orchestra.run.smithi023.stdout: "last_purged_snaps_scrub": "2024-03-11T20:55:48.773120+0000", 2024-03-11T20:56:07.369 INFO:teuthology.orchestra.run.smithi023.stdout: "dead_epoch": 0 2024-03-11T20:56:07.369 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:56:07.369 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:56:07.369 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": 3, 2024-03-11T20:56:07.369 INFO:teuthology.orchestra.run.smithi023.stdout: "down_stamp": "0.000000", 2024-03-11T20:56:07.369 INFO:teuthology.orchestra.run.smithi023.stdout: "laggy_probability": 0, 2024-03-11T20:56:07.369 INFO:teuthology.orchestra.run.smithi023.stdout: "laggy_interval": 0, 2024-03-11T20:56:07.369 INFO:teuthology.orchestra.run.smithi023.stdout: "features": 4540701547738038271, 2024-03-11T20:56:07.369 INFO:teuthology.orchestra.run.smithi023.stdout: "old_weight": 0, 2024-03-11T20:56:07.369 INFO:teuthology.orchestra.run.smithi023.stdout: "last_purged_snaps_scrub": "2024-03-11T20:56:06.732060+0000", 2024-03-11T20:56:07.369 INFO:teuthology.orchestra.run.smithi023.stdout: "dead_epoch": 0 2024-03-11T20:56:07.369 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:56:07.369 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:56:07.369 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": 4, 2024-03-11T20:56:07.370 INFO:teuthology.orchestra.run.smithi023.stdout: "down_stamp": "0.000000", 2024-03-11T20:56:07.370 INFO:teuthology.orchestra.run.smithi023.stdout: "laggy_probability": 0, 2024-03-11T20:56:07.370 INFO:teuthology.orchestra.run.smithi023.stdout: "laggy_interval": 0, 2024-03-11T20:56:07.370 INFO:teuthology.orchestra.run.smithi023.stdout: "features": 4540701547738038271, 2024-03-11T20:56:07.370 INFO:teuthology.orchestra.run.smithi023.stdout: "old_weight": 0, 2024-03-11T20:56:07.370 INFO:teuthology.orchestra.run.smithi023.stdout: "last_purged_snaps_scrub": "2024-03-11T20:55:41.220612+0000", 2024-03-11T20:56:07.370 INFO:teuthology.orchestra.run.smithi023.stdout: "dead_epoch": 0 2024-03-11T20:56:07.370 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:56:07.370 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:56:07.370 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": 5, 2024-03-11T20:56:07.370 INFO:teuthology.orchestra.run.smithi023.stdout: "down_stamp": "0.000000", 2024-03-11T20:56:07.370 INFO:teuthology.orchestra.run.smithi023.stdout: "laggy_probability": 0, 2024-03-11T20:56:07.370 INFO:teuthology.orchestra.run.smithi023.stdout: "laggy_interval": 0, 2024-03-11T20:56:07.371 INFO:teuthology.orchestra.run.smithi023.stdout: "features": 4540701547738038271, 2024-03-11T20:56:07.371 INFO:teuthology.orchestra.run.smithi023.stdout: "old_weight": 0, 2024-03-11T20:56:07.371 INFO:teuthology.orchestra.run.smithi023.stdout: "last_purged_snaps_scrub": "2024-03-11T20:54:45.478752+0000", 2024-03-11T20:56:07.371 INFO:teuthology.orchestra.run.smithi023.stdout: "dead_epoch": 0 2024-03-11T20:56:07.371 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:56:07.371 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:56:07.371 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": 6, 2024-03-11T20:56:07.371 INFO:teuthology.orchestra.run.smithi023.stdout: "down_stamp": "0.000000", 2024-03-11T20:56:07.371 INFO:teuthology.orchestra.run.smithi023.stdout: "laggy_probability": 0, 2024-03-11T20:56:07.371 INFO:teuthology.orchestra.run.smithi023.stdout: "laggy_interval": 0, 2024-03-11T20:56:07.371 INFO:teuthology.orchestra.run.smithi023.stdout: "features": 4540701547738038271, 2024-03-11T20:56:07.371 INFO:teuthology.orchestra.run.smithi023.stdout: "old_weight": 0, 2024-03-11T20:56:07.371 INFO:teuthology.orchestra.run.smithi023.stdout: "last_purged_snaps_scrub": "2024-03-11T20:55:54.748898+0000", 2024-03-11T20:56:07.371 INFO:teuthology.orchestra.run.smithi023.stdout: "dead_epoch": 0 2024-03-11T20:56:07.371 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:56:07.372 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:56:07.372 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": 7, 2024-03-11T20:56:07.372 INFO:teuthology.orchestra.run.smithi023.stdout: "down_stamp": "0.000000", 2024-03-11T20:56:07.372 INFO:teuthology.orchestra.run.smithi023.stdout: "laggy_probability": 0, 2024-03-11T20:56:07.372 INFO:teuthology.orchestra.run.smithi023.stdout: "laggy_interval": 0, 2024-03-11T20:56:07.372 INFO:teuthology.orchestra.run.smithi023.stdout: "features": 4540701547738038271, 2024-03-11T20:56:07.372 INFO:teuthology.orchestra.run.smithi023.stdout: "old_weight": 0, 2024-03-11T20:56:07.372 INFO:teuthology.orchestra.run.smithi023.stdout: "last_purged_snaps_scrub": "2024-03-11T20:55:35.822804+0000", 2024-03-11T20:56:07.372 INFO:teuthology.orchestra.run.smithi023.stdout: "dead_epoch": 0 2024-03-11T20:56:07.372 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:56:07.372 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-11T20:56:07.372 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_upmap": [], 2024-03-11T20:56:07.372 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_upmap_items": [ 2024-03-11T20:56:07.372 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:56:07.373 INFO:teuthology.orchestra.run.smithi023.stdout: "pgid": "2.0", 2024-03-11T20:56:07.373 INFO:teuthology.orchestra.run.smithi023.stdout: "mappings": [ 2024-03-11T20:56:07.373 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:56:07.373 INFO:teuthology.orchestra.run.smithi023.stdout: "from": 7, 2024-03-11T20:56:07.373 INFO:teuthology.orchestra.run.smithi023.stdout: "to": 2 2024-03-11T20:56:07.373 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:56:07.373 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T20:56:07.373 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:56:07.373 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:56:07.373 INFO:teuthology.orchestra.run.smithi023.stdout: "pgid": "4.16", 2024-03-11T20:56:07.373 INFO:teuthology.orchestra.run.smithi023.stdout: "mappings": [ 2024-03-11T20:56:07.373 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:56:07.373 INFO:teuthology.orchestra.run.smithi023.stdout: "from": 3, 2024-03-11T20:56:07.373 INFO:teuthology.orchestra.run.smithi023.stdout: "to": 2 2024-03-11T20:56:07.374 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:56:07.374 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T20:56:07.374 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:56:07.374 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:56:07.374 INFO:teuthology.orchestra.run.smithi023.stdout: "pgid": "4.1c", 2024-03-11T20:56:07.374 INFO:teuthology.orchestra.run.smithi023.stdout: "mappings": [ 2024-03-11T20:56:07.374 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:56:07.374 INFO:teuthology.orchestra.run.smithi023.stdout: "from": 3, 2024-03-11T20:56:07.374 INFO:teuthology.orchestra.run.smithi023.stdout: "to": 5 2024-03-11T20:56:07.374 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:56:07.374 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T20:56:07.374 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:56:07.374 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-11T20:56:07.375 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_upmap_primaries": [], 2024-03-11T20:56:07.375 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_temp": [], 2024-03-11T20:56:07.375 INFO:teuthology.orchestra.run.smithi023.stdout: "primary_temp": [], 2024-03-11T20:56:07.375 INFO:teuthology.orchestra.run.smithi023.stdout: "blocklist": { 2024-03-11T20:56:07.375 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:0/1618334186": "2024-03-12T20:40:45.361537+0000", 2024-03-11T20:56:07.375 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:6801/2813578081": "2024-03-12T20:40:45.361537+0000", 2024-03-11T20:56:07.375 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:0/2869940180": "2024-03-12T20:51:33.790205+0000", 2024-03-11T20:56:07.375 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:6800/3037564930": "2024-03-12T20:51:33.790205+0000", 2024-03-11T20:56:07.375 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:0/3297567084": "2024-03-12T20:51:33.790205+0000", 2024-03-11T20:56:07.375 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:0/3708753659": "2024-03-12T20:40:45.361537+0000", 2024-03-11T20:56:07.375 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:0/2004146740": "2024-03-12T20:40:45.361537+0000", 2024-03-11T20:56:07.375 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:0/827653812": "2024-03-12T20:41:21.301516+0000", 2024-03-11T20:56:07.375 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:6800/1057699850": "2024-03-12T20:41:21.301516+0000", 2024-03-11T20:56:07.375 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:6800/2813578081": "2024-03-12T20:40:45.361537+0000", 2024-03-11T20:56:07.375 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:6801/1057699850": "2024-03-12T20:41:21.301516+0000", 2024-03-11T20:56:07.375 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:0/3833230375": "2024-03-12T20:41:21.301516+0000", 2024-03-11T20:56:07.376 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:6801/3037564930": "2024-03-12T20:51:33.790205+0000", 2024-03-11T20:56:07.376 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:0/1182809102": "2024-03-12T20:51:33.790205+0000", 2024-03-11T20:56:07.376 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:0/2122134187": "2024-03-12T20:51:33.790205+0000", 2024-03-11T20:56:07.376 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:0/2356701899": "2024-03-12T20:41:21.301516+0000", 2024-03-11T20:56:07.376 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:0/843974100": "2024-03-12T20:51:33.790205+0000" 2024-03-11T20:56:07.376 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:56:07.376 INFO:teuthology.orchestra.run.smithi023.stdout: "range_blocklist": {}, 2024-03-11T20:56:07.376 INFO:teuthology.orchestra.run.smithi023.stdout: "erasure_code_profiles": { 2024-03-11T20:56:07.376 INFO:teuthology.orchestra.run.smithi023.stdout: "default": { 2024-03-11T20:56:07.376 INFO:teuthology.orchestra.run.smithi023.stdout: "crush-failure-domain": "osd", 2024-03-11T20:56:07.376 INFO:teuthology.orchestra.run.smithi023.stdout: "k": "2", 2024-03-11T20:56:07.376 INFO:teuthology.orchestra.run.smithi023.stdout: "m": "1", 2024-03-11T20:56:07.376 INFO:teuthology.orchestra.run.smithi023.stdout: "plugin": "jerasure", 2024-03-11T20:56:07.376 INFO:teuthology.orchestra.run.smithi023.stdout: "technique": "reed_sol_van" 2024-03-11T20:56:07.376 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:56:07.377 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:56:07.377 INFO:teuthology.orchestra.run.smithi023.stdout: "removed_snaps_queue": [], 2024-03-11T20:56:07.377 INFO:teuthology.orchestra.run.smithi023.stdout: "new_removed_snaps": [], 2024-03-11T20:56:07.377 INFO:teuthology.orchestra.run.smithi023.stdout: "new_purged_snaps": [], 2024-03-11T20:56:07.377 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_node_flags": {}, 2024-03-11T20:56:07.377 INFO:teuthology.orchestra.run.smithi023.stdout: "device_class_flags": {}, 2024-03-11T20:56:07.377 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": { 2024-03-11T20:56:07.377 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode_enabled": false, 2024-03-11T20:56:07.377 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_bucket_count": 0, 2024-03-11T20:56:07.377 INFO:teuthology.orchestra.run.smithi023.stdout: "degraded_stretch_mode": 0, 2024-03-11T20:56:07.377 INFO:teuthology.orchestra.run.smithi023.stdout: "recovering_stretch_mode": 0, 2024-03-11T20:56:07.377 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode_bucket": 0 2024-03-11T20:56:07.377 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:56:07.377 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-11T20:56:07.398 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:56:07.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:07.400 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:56:07.501 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:56:07.525 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:56:07.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:07.529 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:56:07.547 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 76 16 98243 98227 80.765 44.3125 0.00349911 0.0123736 2024-03-11T20:56:07.630 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:07.658 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:07.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:07.660 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:56:07.761 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:56:07.789 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:56:07.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:07.792 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:56:07.893 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:07.918 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:07.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:07.922 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:56:08.023 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:56:08.059 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:56:08.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:08.065 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:56:08.074 INFO:teuthology.orchestra.run.smithi023.stderr:noscrub is set 2024-03-11T20:56:08.167 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:08.174 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:08 smithi023 bash[21372]: cluster 2024-03-11T20:56:06.119776+0000 osd.0 (osd.0) 139 : cluster 0 4.b scrub starts 2024-03-11T20:56:08.174 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:08 smithi023 bash[21372]: cluster 2024-03-11T20:56:06.124418+0000 osd.0 (osd.0) 140 : cluster 0 4.b scrub ok 2024-03-11T20:56:08.174 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:08 smithi023 bash[21372]: cluster 2024-03-11T20:56:07.000454+0000 osd.7 (osd.7) 131 : cluster 0 2.b scrub starts 2024-03-11T20:56:08.174 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:08 smithi023 bash[21372]: cluster 2024-03-11T20:56:07.001336+0000 osd.7 (osd.7) 132 : cluster 0 2.b scrub ok 2024-03-11T20:56:08.174 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:08 smithi023 bash[21372]: cluster 2024-03-11T20:56:07.041269+0000 mon.a (mon.0) 1008 : cluster 0 osdmap e147: 8 total, 8 up, 7 in 2024-03-11T20:56:08.175 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:08 smithi023 bash[21372]: audit 2024-03-11T20:56:07.273775+0000 mon.a (mon.0) 1009 : audit 0 from='client.? 172.21.15.23:0/4068550450' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-11T20:56:08.175 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:08 smithi023 bash[21372]: audit 2024-03-11T20:56:07.574006+0000 mon.a (mon.0) 1010 : audit 1 from='client.? 172.21.15.23:0/2599490971' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T20:56:08.175 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:08 smithi023 bash[29146]: cluster 2024-03-11T20:56:06.119776+0000 osd.0 (osd.0) 139 : cluster 0 4.b scrub starts 2024-03-11T20:56:08.175 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:08 smithi023 bash[29146]: cluster 2024-03-11T20:56:06.124418+0000 osd.0 (osd.0) 140 : cluster 0 4.b scrub ok 2024-03-11T20:56:08.175 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:08 smithi023 bash[29146]: cluster 2024-03-11T20:56:07.000454+0000 osd.7 (osd.7) 131 : cluster 0 2.b scrub starts 2024-03-11T20:56:08.175 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:08 smithi023 bash[29146]: cluster 2024-03-11T20:56:07.001336+0000 osd.7 (osd.7) 132 : cluster 0 2.b scrub ok 2024-03-11T20:56:08.175 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:08 smithi023 bash[29146]: cluster 2024-03-11T20:56:07.041269+0000 mon.a (mon.0) 1008 : cluster 0 osdmap e147: 8 total, 8 up, 7 in 2024-03-11T20:56:08.175 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:08 smithi023 bash[29146]: audit 2024-03-11T20:56:07.273775+0000 mon.a (mon.0) 1009 : audit 0 from='client.? 172.21.15.23:0/4068550450' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-11T20:56:08.175 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:08 smithi023 bash[29146]: audit 2024-03-11T20:56:07.574006+0000 mon.a (mon.0) 1010 : audit 1 from='client.? 172.21.15.23:0/2599490971' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T20:56:08.184 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:08.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:08.187 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:56:08.287 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:08.316 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:08.321 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:08 smithi146 bash[23809]: cluster 2024-03-11T20:56:06.119776+0000 osd.0 (osd.0) 139 : cluster 0 4.b scrub starts 2024-03-11T20:56:08.321 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:08 smithi146 bash[23809]: cluster 2024-03-11T20:56:06.124418+0000 osd.0 (osd.0) 140 : cluster 0 4.b scrub ok 2024-03-11T20:56:08.322 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:08 smithi146 bash[23809]: cluster 2024-03-11T20:56:07.000454+0000 osd.7 (osd.7) 131 : cluster 0 2.b scrub starts 2024-03-11T20:56:08.322 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:08 smithi146 bash[23809]: cluster 2024-03-11T20:56:07.001336+0000 osd.7 (osd.7) 132 : cluster 0 2.b scrub ok 2024-03-11T20:56:08.322 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:08 smithi146 bash[23809]: cluster 2024-03-11T20:56:07.041269+0000 mon.a (mon.0) 1008 : cluster 0 osdmap e147: 8 total, 8 up, 7 in 2024-03-11T20:56:08.322 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:08 smithi146 bash[23809]: audit 2024-03-11T20:56:07.273775+0000 mon.a (mon.0) 1009 : audit 0 from='client.? 172.21.15.23:0/4068550450' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-11T20:56:08.322 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:08 smithi146 bash[23809]: audit 2024-03-11T20:56:07.574006+0000 mon.a (mon.0) 1010 : audit 1 from='client.? 172.21.15.23:0/2599490971' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T20:56:08.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:08.323 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:56:08.424 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:56:08.434 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:56:08 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:20:56:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T20:56:08.456 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:56:08.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:08.461 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:56:08.547 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 77 16 99065 99049 80.3833 51.375 0.0647458 0.0124323 2024-03-11T20:56:08.562 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:08.588 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:08.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:08.593 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:56:08.694 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:56:08.719 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:56:08.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:08.721 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:56:08.822 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:56:08.839 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:56:08.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:08.841 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:56:08.942 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:56:08.962 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:56:08.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:08.964 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:56:09.065 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:09.090 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:09.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:09.092 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:56:09.193 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:56:09.220 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:56:09.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:09.223 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:56:09.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:09 smithi146 bash[23809]: cluster 2024-03-11T20:56:07.930484+0000 mgr.y (mgr.24380) 203 : cluster 0 pgmap v186: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.9 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1023 B/s rd, 52 MiB/s wr, 825 op/s 2024-03-11T20:56:09.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:09 smithi146 bash[23809]: cluster 2024-03-11T20:56:08.027980+0000 osd.7 (osd.7) 133 : cluster 0 3.e deep-scrub starts 2024-03-11T20:56:09.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:09 smithi146 bash[23809]: cluster 2024-03-11T20:56:08.036324+0000 osd.7 (osd.7) 134 : cluster 0 3.e deep-scrub ok 2024-03-11T20:56:09.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:09 smithi146 bash[23809]: cluster 2024-03-11T20:56:08.055150+0000 mon.a (mon.0) 1011 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T20:56:09.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:09 smithi146 bash[23809]: audit 2024-03-11T20:56:08.068344+0000 mon.a (mon.0) 1012 : audit 1 from='client.? 172.21.15.23:0/2599490971' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T20:56:09.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:09 smithi146 bash[23809]: cluster 2024-03-11T20:56:08.081028+0000 mon.a (mon.0) 1013 : cluster 0 osdmap e148: 8 total, 8 up, 7 in 2024-03-11T20:56:09.324 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:56:09.348 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:56:09.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:09.358 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:56:09.459 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:56:09.476 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:56:09.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:09.478 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:56:09.547 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 78 16 99686 99670 79.8504 38.8125 0.00288567 0.0125017 2024-03-11T20:56:09.579 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:09.600 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:09.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:09.601 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:56:09.620 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:09 smithi023 bash[21372]: cluster 2024-03-11T20:56:07.930484+0000 mgr.y (mgr.24380) 203 : cluster 0 pgmap v186: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.9 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1023 B/s rd, 52 MiB/s wr, 825 op/s 2024-03-11T20:56:09.620 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:09 smithi023 bash[21372]: cluster 2024-03-11T20:56:08.027980+0000 osd.7 (osd.7) 133 : cluster 0 3.e deep-scrub starts 2024-03-11T20:56:09.620 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:09 smithi023 bash[21372]: cluster 2024-03-11T20:56:08.036324+0000 osd.7 (osd.7) 134 : cluster 0 3.e deep-scrub ok 2024-03-11T20:56:09.620 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:09 smithi023 bash[21372]: cluster 2024-03-11T20:56:08.055150+0000 mon.a (mon.0) 1011 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T20:56:09.620 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:09 smithi023 bash[21372]: audit 2024-03-11T20:56:08.068344+0000 mon.a (mon.0) 1012 : audit 1 from='client.? 172.21.15.23:0/2599490971' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T20:56:09.621 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:09 smithi023 bash[21372]: cluster 2024-03-11T20:56:08.081028+0000 mon.a (mon.0) 1013 : cluster 0 osdmap e148: 8 total, 8 up, 7 in 2024-03-11T20:56:09.621 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:09 smithi023 bash[29146]: cluster 2024-03-11T20:56:07.930484+0000 mgr.y (mgr.24380) 203 : cluster 0 pgmap v186: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.9 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1023 B/s rd, 52 MiB/s wr, 825 op/s 2024-03-11T20:56:09.621 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:09 smithi023 bash[29146]: cluster 2024-03-11T20:56:08.027980+0000 osd.7 (osd.7) 133 : cluster 0 3.e deep-scrub starts 2024-03-11T20:56:09.621 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:09 smithi023 bash[29146]: cluster 2024-03-11T20:56:08.036324+0000 osd.7 (osd.7) 134 : cluster 0 3.e deep-scrub ok 2024-03-11T20:56:09.621 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:09 smithi023 bash[29146]: cluster 2024-03-11T20:56:08.055150+0000 mon.a (mon.0) 1011 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T20:56:09.621 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:09 smithi023 bash[29146]: audit 2024-03-11T20:56:08.068344+0000 mon.a (mon.0) 1012 : audit 1 from='client.? 172.21.15.23:0/2599490971' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T20:56:09.621 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:09 smithi023 bash[29146]: cluster 2024-03-11T20:56:08.081028+0000 mon.a (mon.0) 1013 : cluster 0 osdmap e148: 8 total, 8 up, 7 in 2024-03-11T20:56:09.702 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:09.723 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:09.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:09.724 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:56:09.825 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:09.857 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:09.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:09.859 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:56:09.960 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:09.978 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:09.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:09.980 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:56:10.081 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:10.098 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:10.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:10.100 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:56:10.201 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:56:10.221 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:56:10.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:10.222 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:56:10.323 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:56:10.352 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:56:10.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:10.355 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:56:10.457 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:10.483 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:10.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:10.486 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:56:10.547 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 79 16 100246 100230 79.2826 35 0.00789427 0.0126062 2024-03-11T20:56:10.587 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:56:10.622 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:56:10.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:10.628 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:56:10.729 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:56:10.749 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:56:10.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:10.751 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:56:10.852 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:10.870 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:10.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:10.872 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:56:10.973 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:56:11.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:10 smithi146 bash[23809]: cluster 2024-03-11T20:56:09.931055+0000 mgr.y (mgr.24380) 204 : cluster 0 pgmap v188: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.0 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 639 B/s rd, 46 MiB/s wr, 733 op/s 2024-03-11T20:56:11.015 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:56:11.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:11.018 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:56:11.049 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:10 smithi023 bash[21372]: cluster 2024-03-11T20:56:09.931055+0000 mgr.y (mgr.24380) 204 : cluster 0 pgmap v188: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.0 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 639 B/s rd, 46 MiB/s wr, 733 op/s 2024-03-11T20:56:11.049 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:10 smithi023 bash[29146]: cluster 2024-03-11T20:56:09.931055+0000 mgr.y (mgr.24380) 204 : cluster 0 pgmap v188: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.0 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 639 B/s rd, 46 MiB/s wr, 733 op/s 2024-03-11T20:56:11.119 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:11.145 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:11.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:11.149 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:56:11.250 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:11.280 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:11.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:11.284 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:56:11.385 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:56:11.415 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:56:11.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:11.419 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:56:11.519 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:11.548 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:2024-03-11T20:56:11.552259+0000 min lat: 0.0012354 max lat: 0.439454 avg lat: 0.0126631 2024-03-11T20:56:11.548 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-11T20:56:11.548 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 80 16 101064 101048 78.9306 51.125 0.0281325 0.0126631 2024-03-11T20:56:11.548 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:11.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:11.553 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:56:11.654 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:11.672 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:11.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:11.674 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:56:11.775 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:11.799 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:11.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:11.801 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:56:11.902 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:11.932 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:11.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:11.935 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:56:12.035 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:56:12.063 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:56:12.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:12.067 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:56:12.168 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:12.200 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:12.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:12.205 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:56:12.306 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:56:12.353 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:56:12.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:12.356 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:56:12.457 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:12.499 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:12.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:12.502 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:56:12.548 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 81 16 101814 101798 78.5348 46.875 0.0068674 0.0127276 2024-03-11T20:56:12.567 INFO:tasks.thrashosds.thrasher:j is {'epoch': 147, 'fsid': '67ba00f8-dfe7-11ee-95c8-87774f69a715', 'created': '2024-03-11T20:40:13.664802+0000', 'modified': '2024-03-11T20:56:07.026897+0000', 'last_up_change': '2024-03-11T20:49:48.903029+0000', 'last_in_change': '2024-03-11T20:54:36.322348+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 7, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-11T20:45:31.964570+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '142', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2024-03-11T20:50:26.421486+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '78', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.4900000095367432, 'score_stable': 1.4900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.309999942779541, 'raw_score_stable': 1.309999942779541, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'default.rgw.log', 'create_time': '2024-03-11T20:50:28.592615+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.7400000095367432, 'score_stable': 1.7400000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.5299999713897705, 'raw_score_stable': 1.5299999713897705, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'default.rgw.control', 'create_time': '2024-03-11T20:50:30.536371+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.240000009536743, 'score_stable': 2.240000009536743, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.9700000286102295, 'raw_score_stable': 1.9700000286102295, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'datapool', 'create_time': '2024-03-11T20:50:31.135845+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 87, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6700000762939453, 'score_stable': 2.6700000762939453, 'optimal_score': 0.75, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-03-11T20:50:33.233163+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.7400000095367432, 'score_stable': 1.7400000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.5299999713897705, 'raw_score_stable': 1.5299999713897705, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'unique_pool_0', 'create_time': '2024-03-11T20:54:33.257999+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '127', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': '3df72b4a-8fe4-4265-b4c6-af022e1e23e1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 138, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6802', 'nonce': 2154627393}, {'type': 'v1', 'addr': '172.21.15.23:6803', 'nonce': 2154627393}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6804', 'nonce': 2154627393}, {'type': 'v1', 'addr': '172.21.15.23:6805', 'nonce': 2154627393}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6808', 'nonce': 2154627393}, {'type': 'v1', 'addr': '172.21.15.23:6809', 'nonce': 2154627393}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6806', 'nonce': 2154627393}, {'type': 'v1', 'addr': '172.21.15.23:6807', 'nonce': 2154627393}]}, 'public_addr': '172.21.15.23:6803/2154627393', 'cluster_addr': '172.21.15.23:6805/2154627393', 'heartbeat_back_addr': '172.21.15.23:6809/2154627393', 'heartbeat_front_addr': '172.21.15.23:6807/2154627393', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'e8fd7500-9081-41a1-8077-a4d0f0e12a2c', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 121, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6810', 'nonce': 2506707705}, {'type': 'v1', 'addr': '172.21.15.23:6811', 'nonce': 2506707705}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6812', 'nonce': 2506707705}, {'type': 'v1', 'addr': '172.21.15.23:6813', 'nonce': 2506707705}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6816', 'nonce': 2506707705}, {'type': 'v1', 'addr': '172.21.15.23:6817', 'nonce': 2506707705}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6814', 'nonce': 2506707705}, {'type': 'v1', 'addr': '172.21.15.23:6815', 'nonce': 2506707705}]}, 'public_addr': '172.21.15.23:6811/2506707705', 'cluster_addr': '172.21.15.23:6813/2506707705', 'heartbeat_back_addr': '172.21.15.23:6817/2506707705', 'heartbeat_front_addr': '172.21.15.23:6815/2506707705', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'dd85d52a-df48-4b03-8612-cecfd82b5987', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 19, 'up_thru': 138, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6818', 'nonce': 1770748056}, {'type': 'v1', 'addr': '172.21.15.23:6819', 'nonce': 1770748056}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6820', 'nonce': 1770748056}, {'type': 'v1', 'addr': '172.21.15.23:6821', 'nonce': 1770748056}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6824', 'nonce': 1770748056}, {'type': 'v1', 'addr': '172.21.15.23:6825', 'nonce': 1770748056}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6822', 'nonce': 1770748056}, {'type': 'v1', 'addr': '172.21.15.23:6823', 'nonce': 1770748056}]}, 'public_addr': '172.21.15.23:6819/1770748056', 'cluster_addr': '172.21.15.23:6821/1770748056', 'heartbeat_back_addr': '172.21.15.23:6825/1770748056', 'heartbeat_front_addr': '172.21.15.23:6823/1770748056', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'e3021106-3779-4b94-9bbf-a7724544110f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 28, 'up_thru': 124, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6826', 'nonce': 938496814}, {'type': 'v1', 'addr': '172.21.15.23:6827', 'nonce': 938496814}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6828', 'nonce': 938496814}, {'type': 'v1', 'addr': '172.21.15.23:6829', 'nonce': 938496814}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6832', 'nonce': 938496814}, {'type': 'v1', 'addr': '172.21.15.23:6833', 'nonce': 938496814}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6830', 'nonce': 938496814}, {'type': 'v1', 'addr': '172.21.15.23:6831', 'nonce': 938496814}]}, 'public_addr': '172.21.15.23:6827/938496814', 'cluster_addr': '172.21.15.23:6829/938496814', 'heartbeat_back_addr': '172.21.15.23:6833/938496814', 'heartbeat_front_addr': '172.21.15.23:6831/938496814', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '4dcbc91e-7035-413d-97bc-c9d804eb16d7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 35, 'up_thru': 124, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6800', 'nonce': 615746847}, {'type': 'v1', 'addr': '172.21.15.146:6801', 'nonce': 615746847}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6802', 'nonce': 615746847}, {'type': 'v1', 'addr': '172.21.15.146:6803', 'nonce': 615746847}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6806', 'nonce': 615746847}, {'type': 'v1', 'addr': '172.21.15.146:6807', 'nonce': 615746847}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6804', 'nonce': 615746847}, {'type': 'v1', 'addr': '172.21.15.146:6805', 'nonce': 615746847}]}, 'public_addr': '172.21.15.146:6801/615746847', 'cluster_addr': '172.21.15.146:6803/615746847', 'heartbeat_back_addr': '172.21.15.146:6807/615746847', 'heartbeat_front_addr': '172.21.15.146:6805/615746847', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'aefcb766-8bae-4ff4-b71e-6691c8ac02f1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 44, 'up_thru': 124, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6808', 'nonce': 4003230814}, {'type': 'v1', 'addr': '172.21.15.146:6809', 'nonce': 4003230814}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6810', 'nonce': 4003230814}, {'type': 'v1', 'addr': '172.21.15.146:6811', 'nonce': 4003230814}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6814', 'nonce': 4003230814}, {'type': 'v1', 'addr': '172.21.15.146:6815', 'nonce': 4003230814}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6812', 'nonce': 4003230814}, {'type': 'v1', 'addr': '172.21.15.146:6813', 'nonce': 4003230814}]}, 'public_addr': '172.21.15.146:6809/4003230814', 'cluster_addr': '172.21.15.146:6811/4003230814', 'heartbeat_back_addr': '172.21.15.146:6815/4003230814', 'heartbeat_front_addr': '172.21.15.146:6813/4003230814', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '5cb151e8-1f9b-4617-8a86-adfb06bb9606', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 57, 'up_thru': 124, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6816', 'nonce': 3334092376}, {'type': 'v1', 'addr': '172.21.15.146:6817', 'nonce': 3334092376}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6818', 'nonce': 3334092376}, {'type': 'v1', 'addr': '172.21.15.146:6819', 'nonce': 3334092376}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6822', 'nonce': 3334092376}, {'type': 'v1', 'addr': '172.21.15.146:6823', 'nonce': 3334092376}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6820', 'nonce': 3334092376}, {'type': 'v1', 'addr': '172.21.15.146:6821', 'nonce': 3334092376}]}, 'public_addr': '172.21.15.146:6817/3334092376', 'cluster_addr': '172.21.15.146:6819/3334092376', 'heartbeat_back_addr': '172.21.15.146:6823/3334092376', 'heartbeat_front_addr': '172.21.15.146:6821/3334092376', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'bdbd11a2-fa09-4f82-93a2-6b2d9c750bfd', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 68, 'up_thru': 124, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6824', 'nonce': 1678407923}, {'type': 'v1', 'addr': '172.21.15.146:6825', 'nonce': 1678407923}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6826', 'nonce': 1678407923}, {'type': 'v1', 'addr': '172.21.15.146:6827', 'nonce': 1678407923}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6830', 'nonce': 1678407923}, {'type': 'v1', 'addr': '172.21.15.146:6831', 'nonce': 1678407923}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6828', 'nonce': 1678407923}, {'type': 'v1', 'addr': '172.21.15.146:6829', 'nonce': 1678407923}]}, 'public_addr': '172.21.15.146:6825/1678407923', 'cluster_addr': '172.21.15.146:6827/1678407923', 'heartbeat_back_addr': '172.21.15.146:6831/1678407923', 'heartbeat_front_addr': '172.21.15.146:6829/1678407923', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T20:55:20.780923+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-03-11T20:55:41.137095+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T20:55:48.773120+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T20:56:06.732060+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T20:55:41.220612+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T20:54:45.478752+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T20:55:54.748898+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T20:55:35.822804+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '2.0', 'mappings': [{'from': 7, 'to': 2}]}, {'pgid': '4.16', 'mappings': [{'from': 3, 'to': 2}]}, {'pgid': '4.1c', 'mappings': [{'from': 3, 'to': 5}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.23:0/1618334186': '2024-03-12T20:40:45.361537+0000', '172.21.15.23:6801/2813578081': '2024-03-12T20:40:45.361537+0000', '172.21.15.23:0/2869940180': '2024-03-12T20:51:33.790205+0000', '172.21.15.23:6800/3037564930': '2024-03-12T20:51:33.790205+0000', '172.21.15.23:0/3297567084': '2024-03-12T20:51:33.790205+0000', '172.21.15.23:0/3708753659': '2024-03-12T20:40:45.361537+0000', '172.21.15.23:0/2004146740': '2024-03-12T20:40:45.361537+0000', '172.21.15.23:0/827653812': '2024-03-12T20:41:21.301516+0000', '172.21.15.23:6800/1057699850': '2024-03-12T20:41:21.301516+0000', '172.21.15.23:6800/2813578081': '2024-03-12T20:40:45.361537+0000', '172.21.15.23:6801/1057699850': '2024-03-12T20:41:21.301516+0000', '172.21.15.23:0/3833230375': '2024-03-12T20:41:21.301516+0000', '172.21.15.23:6801/3037564930': '2024-03-12T20:51:33.790205+0000', '172.21.15.23:0/1182809102': '2024-03-12T20:51:33.790205+0000', '172.21.15.23:0/2122134187': '2024-03-12T20:51:33.790205+0000', '172.21.15.23:0/2356701899': '2024-03-12T20:41:21.301516+0000', '172.21.15.23:0/843974100': '2024-03-12T20:51:33.790205+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-03-11T20:56:12.567 INFO:tasks.thrashosds.thrasher:Clearing pg_upmap on 4.1c 2024-03-11T20:56:12.568 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd rm-pg-upmap-items 4.1c 2024-03-11T20:56:12.603 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:56:12.639 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:56:12.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:12.642 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:56:12.743 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:12.772 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:12.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:12.774 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:56:12.875 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:56:12.899 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:56:12.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:12.902 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:56:12.966 INFO:teuthology.orchestra.run.smithi023.stdout:ERROR: (22) Invalid argument 2024-03-11T20:56:12.966 INFO:teuthology.orchestra.run.smithi023.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T20:56:12.983 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:12 smithi023 bash[21372]: cluster 2024-03-11T20:56:11.154611+0000 osd.5 (osd.5) 117 : cluster 0 purged_snaps scrub starts 2024-03-11T20:56:12.983 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:12 smithi023 bash[21372]: cluster 2024-03-11T20:56:11.154689+0000 osd.5 (osd.5) 118 : cluster 0 purged_snaps scrub ok 2024-03-11T20:56:12.983 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:12 smithi023 bash[21372]: cluster 2024-03-11T20:56:11.703032+0000 mon.a (mon.0) 1014 : cluster 0 osdmap e149: 8 total, 8 up, 7 in 2024-03-11T20:56:12.983 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:12 smithi023 bash[21372]: cluster 2024-03-11T20:56:11.931513+0000 mgr.y (mgr.24380) 205 : cluster 0 pgmap v190: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.1 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 853 B/s rd, 41 MiB/s wr, 657 op/s 2024-03-11T20:56:12.983 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:12 smithi023 bash[29146]: cluster 2024-03-11T20:56:11.154611+0000 osd.5 (osd.5) 117 : cluster 0 purged_snaps scrub starts 2024-03-11T20:56:12.983 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:12 smithi023 bash[29146]: cluster 2024-03-11T20:56:11.154689+0000 osd.5 (osd.5) 118 : cluster 0 purged_snaps scrub ok 2024-03-11T20:56:12.984 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:12 smithi023 bash[29146]: cluster 2024-03-11T20:56:11.703032+0000 mon.a (mon.0) 1014 : cluster 0 osdmap e149: 8 total, 8 up, 7 in 2024-03-11T20:56:12.984 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:12 smithi023 bash[29146]: cluster 2024-03-11T20:56:11.931513+0000 mgr.y (mgr.24380) 205 : cluster 0 pgmap v190: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.1 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 853 B/s rd, 41 MiB/s wr, 657 op/s 2024-03-11T20:56:13.002 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:56:13.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:12 smithi146 bash[23809]: cluster 2024-03-11T20:56:11.154611+0000 osd.5 (osd.5) 117 : cluster 0 purged_snaps scrub starts 2024-03-11T20:56:13.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:12 smithi146 bash[23809]: cluster 2024-03-11T20:56:11.154689+0000 osd.5 (osd.5) 118 : cluster 0 purged_snaps scrub ok 2024-03-11T20:56:13.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:12 smithi146 bash[23809]: cluster 2024-03-11T20:56:11.703032+0000 mon.a (mon.0) 1014 : cluster 0 osdmap e149: 8 total, 8 up, 7 in 2024-03-11T20:56:13.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:12 smithi146 bash[23809]: cluster 2024-03-11T20:56:11.931513+0000 mgr.y (mgr.24380) 205 : cluster 0 pgmap v190: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.1 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 853 B/s rd, 41 MiB/s wr, 657 op/s 2024-03-11T20:56:13.028 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:56:13.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:13.031 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:56:13.132 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:56:13.156 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:56:13.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:13.159 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:56:13.260 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:56:13.289 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:56:13.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:13.291 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:56:13.307 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T20:56:13.392 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:56:13.417 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:56:13.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:13.424 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:56:13.525 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:56:13.548 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 82 16 102718 102702 78.266 56.5 0.0158066 0.0127685 2024-03-11T20:56:13.550 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:56:13.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:13.554 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:56:13.655 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:56:13.680 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:56:13.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:13.682 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:56:13.783 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:56:13.806 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:56:13.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:13.808 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:56:13.909 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:13.939 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:13.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:13.942 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:56:14.042 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:56:14.068 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:56:14.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:14.072 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:56:14.173 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:56:14.202 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:56:14.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:14.205 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:56:14.306 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:56:14.329 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:56:14.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:14.332 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:56:14.370 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-11T20:56:14.433 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:56:14.482 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:56:14.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:14.486 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:56:14.546 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-11T20:56:14.548 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 83 16 103422 103406 77.8532 44 0.00618035 0.0128349 2024-03-11T20:56:14.587 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:14.653 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:14.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:14.657 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:56:14.758 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:14.786 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:14.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:14.790 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:56:14.891 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:56:14.918 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:56:14.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:14.923 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:56:15.024 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:56:15.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:15.052 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:56:15.052 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:56:15.153 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:15.175 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:15.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:15.180 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:56:15.260 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 20:56:15 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T20:56:15.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:14 smithi146 bash[23809]: cluster 2024-03-11T20:56:13.932319+0000 mgr.y (mgr.24380) 206 : cluster 0 pgmap v191: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.1 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 742 B/s rd, 36 MiB/s wr, 572 op/s 2024-03-11T20:56:15.281 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:15.286 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:14 smithi023 bash[21372]: cluster 2024-03-11T20:56:13.932319+0000 mgr.y (mgr.24380) 206 : cluster 0 pgmap v191: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.1 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 742 B/s rd, 36 MiB/s wr, 572 op/s 2024-03-11T20:56:15.286 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:14 smithi023 bash[29146]: cluster 2024-03-11T20:56:13.932319+0000 mgr.y (mgr.24380) 206 : cluster 0 pgmap v191: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.1 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 742 B/s rd, 36 MiB/s wr, 572 op/s 2024-03-11T20:56:15.307 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:15.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:15.309 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:56:15.410 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:15.436 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:15.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:15.439 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:56:15.540 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:15.548 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 84 16 104348 104332 77.6153 57.875 0.0146409 0.0128768 2024-03-11T20:56:15.562 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:15.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:15.565 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:56:15.665 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:56:15.691 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:56:15.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:15.693 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:56:15.794 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:56:15.820 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:56:15.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:15.825 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:56:15.926 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:56:15.971 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:56:15.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:15.975 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:56:16.076 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:56:16.085 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:16 smithi146 bash[23809]: audit 2024-03-11T20:56:15.068961+0000 mgr.y (mgr.24380) 207 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:56:16.102 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:56:16.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:16.104 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:56:16.205 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:16.234 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:16.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:16.238 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:56:16.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:16 smithi023 bash[21372]: audit 2024-03-11T20:56:15.068961+0000 mgr.y (mgr.24380) 207 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:56:16.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:16 smithi023 bash[29146]: audit 2024-03-11T20:56:15.068961+0000 mgr.y (mgr.24380) 207 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:56:16.339 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:56:16.362 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:56:16.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:16.366 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:56:16.467 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:56:16.493 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:56:16.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:16.495 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:56:16.548 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 85 16 105346 105330 77.4359 62.375 0.0105566 0.012908 2024-03-11T20:56:16.596 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:56:16.622 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:56:16.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:16.624 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:56:16.725 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:56:16.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:16.757 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:56:16.757 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:56:16.858 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:56:16.882 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:56:16.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:16.885 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:56:16.986 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:17.019 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:17.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:17.026 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:56:17.126 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:17.159 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:17.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:17.162 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:56:17.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:17 smithi146 bash[23809]: cluster 2024-03-11T20:56:15.933301+0000 mgr.y (mgr.24380) 208 : cluster 0 pgmap v192: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.3 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 61 MiB/s wr, 977 op/s 2024-03-11T20:56:17.262 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:17.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:17 smithi023 bash[21372]: cluster 2024-03-11T20:56:15.933301+0000 mgr.y (mgr.24380) 208 : cluster 0 pgmap v192: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.3 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 61 MiB/s wr, 977 op/s 2024-03-11T20:56:17.273 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:17 smithi023 bash[29146]: cluster 2024-03-11T20:56:15.933301+0000 mgr.y (mgr.24380) 208 : cluster 0 pgmap v192: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.3 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 61 MiB/s wr, 977 op/s 2024-03-11T20:56:17.288 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:17.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:17.293 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:56:17.394 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:56:17.427 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:56:17.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:17.429 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:56:17.530 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:17.548 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 86 16 106327 106311 77.2484 61.3125 0.0130816 0.0129396 2024-03-11T20:56:17.556 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:17.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:17.563 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:56:17.664 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:17.698 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:17.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:17.703 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:56:17.804 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:56:17.831 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:56:17.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:17.835 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:56:17.902 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:56:17.903 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:56:17.936 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:17.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:17.971 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:17.971 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:56:18.073 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:56:18.099 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:56:18.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:18.101 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:56:18.202 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:56:18.235 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:56:18.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:18.238 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:56:18.339 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:56:18.369 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:56:18.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:18.375 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:56:18.475 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:56:18.505 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:56:18.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:18.507 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:56:18.548 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 87 16 107306 107290 77.0638 61.1875 0.00527254 0.0129683 2024-03-11T20:56:18.610 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:18.633 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:18.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:18.635 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:56:18.736 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:56:18.769 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:56:18.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:18.772 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:56:18.784 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:56:18 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:20:56:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T20:56:18.873 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:56:18.895 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:56:18.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:18.897 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:56:18.922 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:56:18.923 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:56:18.998 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:56:19.033 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:56:19.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:19.035 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:56:19.136 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:56:19.168 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:56:19.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:19.172 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:56:19.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:19 smithi146 bash[23809]: cluster 2024-03-11T20:56:17.933707+0000 mgr.y (mgr.24380) 209 : cluster 0 pgmap v193: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.3 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 1.0 KiB/s rd, 50 MiB/s wr, 793 op/s 2024-03-11T20:56:19.273 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:19.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:19 smithi023 bash[21372]: cluster 2024-03-11T20:56:17.933707+0000 mgr.y (mgr.24380) 209 : cluster 0 pgmap v193: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.3 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 1.0 KiB/s rd, 50 MiB/s wr, 793 op/s 2024-03-11T20:56:19.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:19 smithi023 bash[29146]: cluster 2024-03-11T20:56:17.933707+0000 mgr.y (mgr.24380) 209 : cluster 0 pgmap v193: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.3 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 1.0 KiB/s rd, 50 MiB/s wr, 793 op/s 2024-03-11T20:56:19.309 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:19.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:19.316 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:56:19.416 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:19.451 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:19.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:19.456 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:56:19.548 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 88 16 108296 108280 76.8911 61.875 0.009374 0.012999 2024-03-11T20:56:19.557 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:56:19.590 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:56:19.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:19.593 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:56:19.694 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:56:19.719 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:56:19.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:19.721 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:56:19.823 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:56:19.856 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:56:19.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:19.858 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:56:19.959 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:19.991 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:19.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:19.995 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:56:20.096 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:20.128 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:20.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:20.131 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:56:20.232 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:56:20.257 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:56:20.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:20.261 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:56:20.361 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:20.386 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:20.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:20.389 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:56:20.490 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:20.516 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:20.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:20.520 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:56:20.549 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 89 16 109260 109244 76.7041 60.25 0.00895545 0.013029 2024-03-11T20:56:20.621 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:56:20.635 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:20 smithi023 bash[21372]: audit 2024-03-11T20:56:19.440005+0000 mon.a (mon.0) 1015 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:56:20.635 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:20 smithi023 bash[21372]: audit 2024-03-11T20:56:19.443813+0000 mon.a (mon.0) 1016 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:56:20.635 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:20 smithi023 bash[29146]: audit 2024-03-11T20:56:19.440005+0000 mon.a (mon.0) 1015 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:56:20.635 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:20 smithi023 bash[29146]: audit 2024-03-11T20:56:19.443813+0000 mon.a (mon.0) 1016 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:56:20.654 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:56:20.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:20.656 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:56:20.757 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:20.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:20 smithi146 bash[23809]: audit 2024-03-11T20:56:19.440005+0000 mon.a (mon.0) 1015 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:56:20.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:20 smithi146 bash[23809]: audit 2024-03-11T20:56:19.443813+0000 mon.a (mon.0) 1016 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:56:20.790 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:20.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:20.793 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:56:20.894 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:20.922 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:20.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:20.924 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:56:21.025 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:56:21.046 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:56:21.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:21.048 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:56:21.149 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:56:21.180 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:56:21.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:21.186 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:56:21.287 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:21.327 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:21.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:21.331 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:56:21.431 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:56:21.463 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:56:21.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:21.470 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:56:21.534 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:21 smithi023 bash[29146]: cluster 2024-03-11T20:56:19.934505+0000 mgr.y (mgr.24380) 210 : cluster 0 pgmap v194: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.5 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1023 B/s rd, 51 MiB/s wr, 821 op/s 2024-03-11T20:56:21.534 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:21 smithi023 bash[29146]: cluster 2024-03-11T20:56:20.931590+0000 osd.0 (osd.0) 141 : cluster 0 purged_snaps scrub starts 2024-03-11T20:56:21.534 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:21 smithi023 bash[29146]: cluster 2024-03-11T20:56:20.931653+0000 osd.0 (osd.0) 142 : cluster 0 purged_snaps scrub ok 2024-03-11T20:56:21.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:21 smithi023 bash[21372]: cluster 2024-03-11T20:56:19.934505+0000 mgr.y (mgr.24380) 210 : cluster 0 pgmap v194: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.5 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1023 B/s rd, 51 MiB/s wr, 821 op/s 2024-03-11T20:56:21.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:21 smithi023 bash[21372]: cluster 2024-03-11T20:56:20.931590+0000 osd.0 (osd.0) 141 : cluster 0 purged_snaps scrub starts 2024-03-11T20:56:21.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:21 smithi023 bash[21372]: cluster 2024-03-11T20:56:20.931653+0000 osd.0 (osd.0) 142 : cluster 0 purged_snaps scrub ok 2024-03-11T20:56:21.548 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 90 1 110238 110237 76.5414 62.0625 0.0180934 0.0130583 2024-03-11T20:56:21.571 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:21.590 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:21.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:21.591 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:56:21.692 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:21.701 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:21 smithi146 bash[23809]: cluster 2024-03-11T20:56:19.934505+0000 mgr.y (mgr.24380) 210 : cluster 0 pgmap v194: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.5 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1023 B/s rd, 51 MiB/s wr, 821 op/s 2024-03-11T20:56:21.701 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:21 smithi146 bash[23809]: cluster 2024-03-11T20:56:20.931590+0000 osd.0 (osd.0) 141 : cluster 0 purged_snaps scrub starts 2024-03-11T20:56:21.701 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:21 smithi146 bash[23809]: cluster 2024-03-11T20:56:20.931653+0000 osd.0 (osd.0) 142 : cluster 0 purged_snaps scrub ok 2024-03-11T20:56:21.716 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:21.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:21.718 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:56:21.819 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:56:21.844 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:56:21.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:21.845 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:56:21.946 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:21.971 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:21.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:21.973 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:56:22.073 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:56:22.092 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:56:22.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:22.094 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:56:22.195 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:22.216 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:22.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:22.218 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:56:22.319 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:56:22.338 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:56:22.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:22.339 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:56:22.440 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:22.464 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:22.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:22.466 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:56:22.549 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:Total time run: 90.0191 2024-03-11T20:56:22.549 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:Total writes made: 110238 2024-03-11T20:56:22.549 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:Write size: 65536 2024-03-11T20:56:22.550 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:Object size: 65536 2024-03-11T20:56:22.550 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:Bandwidth (MB/sec): 76.5379 2024-03-11T20:56:22.550 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:Stddev Bandwidth: 32.0365 2024-03-11T20:56:22.550 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:Max bandwidth (MB/sec): 187.062 2024-03-11T20:56:22.550 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:Min bandwidth (MB/sec): 35 2024-03-11T20:56:22.551 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:Average IOPS: 1224 2024-03-11T20:56:22.551 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:Stddev IOPS: 512.583 2024-03-11T20:56:22.551 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:Max IOPS: 2993 2024-03-11T20:56:22.551 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:Min IOPS: 560 2024-03-11T20:56:22.551 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:Average Latency(s): 0.0130593 2024-03-11T20:56:22.552 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:Stddev Latency(s): 0.0180574 2024-03-11T20:56:22.552 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:Max latency(s): 0.439454 2024-03-11T20:56:22.552 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:Min latency(s): 0.0012354 2024-03-11T20:56:22.556 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:Cleaning up (deleting benchmark objects) 2024-03-11T20:56:22.568 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:22.601 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:22.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:22.603 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:56:22.704 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:56:22.733 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:56:22.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:22.737 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:56:22.838 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:22.862 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:22.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:22.868 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:56:22.969 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:56:22.978 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:22 smithi023 bash[21372]: cluster 2024-03-11T20:56:21.694874+0000 mon.a (mon.0) 1017 : cluster 0 osdmap e150: 8 total, 8 up, 7 in 2024-03-11T20:56:22.979 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:22 smithi023 bash[21372]: cluster 2024-03-11T20:56:21.934854+0000 mgr.y (mgr.24380) 211 : cluster 0 pgmap v196: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.6 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 921 B/s rd, 55 MiB/s wr, 879 op/s 2024-03-11T20:56:22.979 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:22 smithi023 bash[29146]: cluster 2024-03-11T20:56:21.694874+0000 mon.a (mon.0) 1017 : cluster 0 osdmap e150: 8 total, 8 up, 7 in 2024-03-11T20:56:22.979 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:22 smithi023 bash[29146]: cluster 2024-03-11T20:56:21.934854+0000 mgr.y (mgr.24380) 211 : cluster 0 pgmap v196: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.6 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 921 B/s rd, 55 MiB/s wr, 879 op/s 2024-03-11T20:56:23.005 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:56:23.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:23.008 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:56:23.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:22 smithi146 bash[23809]: cluster 2024-03-11T20:56:21.694874+0000 mon.a (mon.0) 1017 : cluster 0 osdmap e150: 8 total, 8 up, 7 in 2024-03-11T20:56:23.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:22 smithi146 bash[23809]: cluster 2024-03-11T20:56:21.934854+0000 mgr.y (mgr.24380) 211 : cluster 0 pgmap v196: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.6 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 921 B/s rd, 55 MiB/s wr, 879 op/s 2024-03-11T20:56:23.109 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:23.137 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:23.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:23.144 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:56:23.245 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:56:23.272 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:56:23.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:23.274 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:56:23.375 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:23.405 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:23.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:23.407 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:56:23.507 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:23.530 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:23.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:23.533 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:56:23.634 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:56:23.663 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:56:23.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:23.672 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:56:23.773 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:56:23.796 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:56:23.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:23.798 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:56:23.899 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:56:23.931 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:56:23.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:23.933 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:56:23.999 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:23 smithi023 bash[21372]: cluster 2024-03-11T20:56:23.782589+0000 osd.2 (osd.2) 98 : cluster 0 7.8 deep-scrub ok 2024-03-11T20:56:23.999 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:23 smithi023 bash[29146]: cluster 2024-03-11T20:56:23.782589+0000 osd.2 (osd.2) 98 : cluster 0 7.8 deep-scrub ok 2024-03-11T20:56:24.033 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:56:24.042 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:23 smithi146 bash[23809]: cluster 2024-03-11T20:56:23.782589+0000 osd.2 (osd.2) 98 : cluster 0 7.8 deep-scrub ok 2024-03-11T20:56:24.057 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:56:24.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:24.061 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:56:24.161 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:56:24.178 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:56:24.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:24.180 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:56:24.281 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:24.302 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:24.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:24.304 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:56:24.405 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:56:24.435 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:56:24.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:24.438 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:56:24.539 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:24.557 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:24.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:24.559 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:56:24.660 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:56:24.684 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:56:24.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:24.686 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:56:24.787 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:56:24.809 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:56:24.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:24.811 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:56:24.912 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:56:24.944 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:56:24.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:24.947 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:56:25.048 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:25.060 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:25 smithi023 bash[21372]: cluster 2024-03-11T20:56:23.935283+0000 mgr.y (mgr.24380) 212 : cluster 0 pgmap v197: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.6 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 921 B/s rd, 55 MiB/s wr, 879 op/s 2024-03-11T20:56:25.061 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:25 smithi023 bash[29146]: cluster 2024-03-11T20:56:23.935283+0000 mgr.y (mgr.24380) 212 : cluster 0 pgmap v197: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.6 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 921 B/s rd, 55 MiB/s wr, 879 op/s 2024-03-11T20:56:25.079 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:25.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:25.083 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:56:25.184 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:25.210 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:25.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:25.214 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:56:25.260 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 20:56:25 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T20:56:25.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:25 smithi146 bash[23809]: cluster 2024-03-11T20:56:23.935283+0000 mgr.y (mgr.24380) 212 : cluster 0 pgmap v197: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.6 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 921 B/s rd, 55 MiB/s wr, 879 op/s 2024-03-11T20:56:25.315 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:25.345 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:25.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:25.347 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:56:25.448 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:25.469 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:25.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:25.470 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:56:25.571 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:56:25.593 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:56:25.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:25.595 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:56:25.696 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:25.718 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:25.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:25.720 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:56:25.821 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:56:25.852 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:56:25.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:25.856 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:56:25.957 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:25.985 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:25.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:25.989 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:56:26.090 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:26.100 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:26 smithi023 bash[21372]: audit 2024-03-11T20:56:25.077649+0000 mgr.y (mgr.24380) 213 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:56:26.101 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:26 smithi023 bash[29146]: audit 2024-03-11T20:56:25.077649+0000 mgr.y (mgr.24380) 213 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:56:26.117 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:26.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:26.125 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:56:26.225 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:56:26.246 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:56:26.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:26.248 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:56:26.349 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:26.358 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:26 smithi146 bash[23809]: audit 2024-03-11T20:56:25.077649+0000 mgr.y (mgr.24380) 213 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:56:26.371 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:26.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:26.374 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:56:26.475 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:26.504 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:26.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:26.510 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:56:26.611 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:26.637 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:26.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:26.641 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:56:26.742 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:26.764 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:26.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:26.765 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:56:26.866 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:26.905 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:26.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:26.908 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:56:27.008 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:56:27.027 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:56:27.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:27.029 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:56:27.130 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:56:27.155 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:56:27.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:27.159 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:56:27.204 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:27 smithi023 bash[21372]: cluster 2024-03-11T20:56:25.936218+0000 mgr.y (mgr.24380) 214 : cluster 0 pgmap v198: 148 pgs: 148 active+clean; 6.6 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 1023 B/s rd, 40 MiB/s wr, 862 op/s 2024-03-11T20:56:27.205 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:27 smithi023 bash[29146]: cluster 2024-03-11T20:56:25.936218+0000 mgr.y (mgr.24380) 214 : cluster 0 pgmap v198: 148 pgs: 148 active+clean; 6.6 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 1023 B/s rd, 40 MiB/s wr, 862 op/s 2024-03-11T20:56:27.260 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:27.279 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:27.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:27.280 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:56:27.381 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:27.411 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:27.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:27.414 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:56:27.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:27 smithi146 bash[23809]: cluster 2024-03-11T20:56:25.936218+0000 mgr.y (mgr.24380) 214 : cluster 0 pgmap v198: 148 pgs: 148 active+clean; 6.6 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 1023 B/s rd, 40 MiB/s wr, 862 op/s 2024-03-11T20:56:27.515 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:27.546 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:27.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:27.549 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:56:27.649 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:27.680 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:27.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:27.683 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:56:27.784 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:27.813 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:27.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:27.817 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:56:27.918 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:27.942 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:27.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:27.944 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:56:28.045 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:28.070 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:28.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:28.075 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:56:28.176 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:28.204 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:28.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:28.207 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:56:28.308 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:28.339 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:28.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:28.341 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:56:28.442 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:56:28.454 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:56:28 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:20:56:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T20:56:28.476 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:56:28.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:28.478 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:56:28.579 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:28.609 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:28.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:28.611 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:56:28.712 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:56:28.747 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:56:28.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:28.750 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:56:28.851 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:56:28.869 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:56:28.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:28.870 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:56:28.971 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:56:28.995 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:56:28.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:28.996 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:56:29.097 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:56:29.109 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:29 smithi146 bash[23809]: cluster 2024-03-11T20:56:27.936694+0000 mgr.y (mgr.24380) 215 : cluster 0 pgmap v199: 148 pgs: 148 active+clean; 6.6 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 1023 B/s rd, 40 MiB/s wr, 862 op/s 2024-03-11T20:56:29.130 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:56:29.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:29.135 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:56:29.235 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:29.257 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:29.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:29.261 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:56:29.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:29 smithi023 bash[21372]: cluster 2024-03-11T20:56:27.936694+0000 mgr.y (mgr.24380) 215 : cluster 0 pgmap v199: 148 pgs: 148 active+clean; 6.6 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 1023 B/s rd, 40 MiB/s wr, 862 op/s 2024-03-11T20:56:29.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:29 smithi023 bash[29146]: cluster 2024-03-11T20:56:27.936694+0000 mgr.y (mgr.24380) 215 : cluster 0 pgmap v199: 148 pgs: 148 active+clean; 6.6 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 1023 B/s rd, 40 MiB/s wr, 862 op/s 2024-03-11T20:56:29.362 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:29.395 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:29.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:29.401 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:56:29.502 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:56:29.522 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:56:29.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:29.525 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:56:29.626 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:56:29.649 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:56:29.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:29.651 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:56:29.752 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:56:29.778 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:56:29.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:29.780 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:56:29.881 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:56:29.909 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:56:29.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:29.915 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:56:30.016 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:30.039 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:30.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:30.041 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:56:30.142 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:56:30.160 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:56:30.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:30.162 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:56:30.263 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:30.288 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:30.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:30.294 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:56:30.395 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:30.442 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:30.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:30.448 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:56:30.548 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:30.569 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:30.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:30.572 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:56:30.673 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:56:30.691 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:56:30.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:30.694 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:56:30.795 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:56:30.819 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:56:30.820 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:30 smithi146 bash[23809]: cluster 2024-03-11T20:56:29.937416+0000 mgr.y (mgr.24380) 216 : cluster 0 pgmap v200: 148 pgs: 148 active+clean; 6.5 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1023 B/s rd, 27 MiB/s wr, 841 op/s 2024-03-11T20:56:30.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:30.821 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:56:30.922 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:56:30.955 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:56:30.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:30.959 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:56:31.060 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:56:31.070 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:30 smithi023 bash[21372]: cluster 2024-03-11T20:56:29.937416+0000 mgr.y (mgr.24380) 216 : cluster 0 pgmap v200: 148 pgs: 148 active+clean; 6.5 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1023 B/s rd, 27 MiB/s wr, 841 op/s 2024-03-11T20:56:31.070 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:30 smithi023 bash[29146]: cluster 2024-03-11T20:56:29.937416+0000 mgr.y (mgr.24380) 216 : cluster 0 pgmap v200: 148 pgs: 148 active+clean; 6.5 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1023 B/s rd, 27 MiB/s wr, 841 op/s 2024-03-11T20:56:31.085 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:56:31.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:31.089 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:56:31.190 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:56:31.214 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:56:31.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:31.220 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:56:31.321 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:56:31.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:31.349 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:56:31.349 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:56:31.450 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:31.474 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:31.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:31.477 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:56:31.578 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:56:31.606 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:56:31.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:31.611 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:56:31.712 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:56:31.739 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:56:31.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:31.741 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:56:31.842 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:56:31.865 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:56:31.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:31.869 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:56:31.970 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:31.993 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:31.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:32.000 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:56:32.101 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:56:32.130 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:56:32.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:32.132 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:56:32.216 INFO:teuthology.orchestra.run.smithi023.stdout:ERROR: (22) Invalid argument 2024-03-11T20:56:32.216 INFO:teuthology.orchestra.run.smithi023.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T20:56:32.233 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:32.256 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:32.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:32.259 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:56:32.360 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:56:32.379 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:56:32.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:32.382 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:56:32.483 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:56:32.505 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:56:32.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:32.509 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:56:32.554 INFO:teuthology.orchestra.run.smithi023.stdout:osd.0: {} 2024-03-11T20:56:32.609 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:32.631 INFO:teuthology.orchestra.run.smithi023.stdout:osd.1: {} 2024-03-11T20:56:32.645 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:32.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:32.649 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:56:32.673 INFO:teuthology.orchestra.run.smithi023.stdout:osd.2: {} 2024-03-11T20:56:32.721 INFO:teuthology.orchestra.run.smithi023.stdout:osd.3: {} 2024-03-11T20:56:32.750 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:56:32.765 INFO:teuthology.orchestra.run.smithi023.stdout:osd.4: {} 2024-03-11T20:56:32.774 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:56:32.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:32.778 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:56:32.810 INFO:teuthology.orchestra.run.smithi023.stdout:osd.5: {} 2024-03-11T20:56:32.850 INFO:teuthology.orchestra.run.smithi023.stdout:osd.6: {} 2024-03-11T20:56:32.878 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:56:32.894 INFO:teuthology.orchestra.run.smithi023.stdout:osd.7: {} 2024-03-11T20:56:32.899 INFO:teuthology.orchestra.run.smithi023.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T20:56:32.899 INFO:teuthology.orchestra.run.smithi023.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T20:56:32.899 INFO:teuthology.orchestra.run.smithi023.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T20:56:32.899 INFO:teuthology.orchestra.run.smithi023.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T20:56:32.899 INFO:teuthology.orchestra.run.smithi023.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T20:56:32.899 INFO:teuthology.orchestra.run.smithi023.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T20:56:32.899 INFO:teuthology.orchestra.run.smithi023.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T20:56:32.900 INFO:teuthology.orchestra.run.smithi023.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T20:56:32.904 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:56:32.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:32.906 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:56:33.006 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:56:33.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:33 smithi146 bash[23809]: cluster 2024-03-11T20:56:31.938107+0000 mgr.y (mgr.24380) 217 : cluster 0 pgmap v201: 148 pgs: 148 active+clean; 6.3 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.1 KiB/s rd, 9.5 MiB/s wr, 795 op/s 2024-03-11T20:56:33.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:33 smithi146 bash[23809]: audit 2024-03-11T20:56:32.507198+0000 mon.c (mon.2) 35 : audit 0 from='client.? 172.21.15.23:0/2402674104' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T20:56:33.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:33 smithi146 bash[23809]: audit 2024-03-11T20:56:32.949566+0000 mon.a (mon.0) 1018 : audit 1 from='client.? 172.21.15.23:0/1984846065' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T20:56:33.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:33 smithi146 bash[23809]: audit 2024-03-11T20:56:32.960019+0000 mon.a (mon.0) 1019 : audit 1 from='client.? 172.21.15.23:0/1769882656' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.1c"}]: dispatch 2024-03-11T20:56:33.022 INFO:teuthology.orchestra.run.smithi023.stderr:clear 4.1c pg_upmap_items mapping 2024-03-11T20:56:33.033 INFO:teuthology.orchestra.run.smithi023.stderr:nodeep-scrub is set 2024-03-11T20:56:33.047 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:56:33.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:33.052 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:56:33.153 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:33.191 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:33.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:33.194 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:56:33.221 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:33 smithi023 bash[21372]: cluster 2024-03-11T20:56:31.938107+0000 mgr.y (mgr.24380) 217 : cluster 0 pgmap v201: 148 pgs: 148 active+clean; 6.3 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.1 KiB/s rd, 9.5 MiB/s wr, 795 op/s 2024-03-11T20:56:33.221 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:33 smithi023 bash[21372]: audit 2024-03-11T20:56:32.507198+0000 mon.c (mon.2) 35 : audit 0 from='client.? 172.21.15.23:0/2402674104' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T20:56:33.221 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:33 smithi023 bash[21372]: audit 2024-03-11T20:56:32.949566+0000 mon.a (mon.0) 1018 : audit 1 from='client.? 172.21.15.23:0/1984846065' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T20:56:33.221 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:33 smithi023 bash[21372]: audit 2024-03-11T20:56:32.960019+0000 mon.a (mon.0) 1019 : audit 1 from='client.? 172.21.15.23:0/1769882656' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.1c"}]: dispatch 2024-03-11T20:56:33.221 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:33 smithi023 bash[29146]: cluster 2024-03-11T20:56:31.938107+0000 mgr.y (mgr.24380) 217 : cluster 0 pgmap v201: 148 pgs: 148 active+clean; 6.3 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.1 KiB/s rd, 9.5 MiB/s wr, 795 op/s 2024-03-11T20:56:33.222 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:33 smithi023 bash[29146]: audit 2024-03-11T20:56:32.507198+0000 mon.c (mon.2) 35 : audit 0 from='client.? 172.21.15.23:0/2402674104' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T20:56:33.222 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:33 smithi023 bash[29146]: audit 2024-03-11T20:56:32.949566+0000 mon.a (mon.0) 1018 : audit 1 from='client.? 172.21.15.23:0/1984846065' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T20:56:33.222 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:33 smithi023 bash[29146]: audit 2024-03-11T20:56:32.960019+0000 mon.a (mon.0) 1019 : audit 1 from='client.? 172.21.15.23:0/1769882656' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.1c"}]: dispatch 2024-03-11T20:56:33.295 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:33.313 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:33.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:33.315 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:56:33.416 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:56:33.445 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:56:33.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:33.454 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:56:33.555 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:33.581 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:33.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:33.586 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:56:33.687 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:33.711 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-03-11T20:56:33.716 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:33.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:33.719 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:56:33.819 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:33.856 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:33.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:33.859 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:56:33.960 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:56:33.982 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:56:33.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:33.984 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:56:34.005 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:34 smithi023 bash[29146]: cluster 2024-03-11T20:56:32.997659+0000 mon.a (mon.0) 1020 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T20:56:34.005 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:34 smithi023 bash[29146]: audit 2024-03-11T20:56:33.017136+0000 mon.a (mon.0) 1021 : audit 1 from='client.? 172.21.15.23:0/1984846065' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T20:56:34.005 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:34 smithi023 bash[29146]: audit 2024-03-11T20:56:33.017313+0000 mon.a (mon.0) 1022 : audit 1 from='client.? 172.21.15.23:0/1769882656' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.1c"}]': finished 2024-03-11T20:56:34.005 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:34 smithi023 bash[29146]: cluster 2024-03-11T20:56:33.028625+0000 mon.a (mon.0) 1023 : cluster 0 osdmap e151: 8 total, 8 up, 7 in 2024-03-11T20:56:34.085 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:56:34.110 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:56:34.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:34.113 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:56:34.214 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:56:34.223 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:34 smithi146 bash[23809]: cluster 2024-03-11T20:56:32.997659+0000 mon.a (mon.0) 1020 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T20:56:34.223 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:34 smithi146 bash[23809]: audit 2024-03-11T20:56:33.017136+0000 mon.a (mon.0) 1021 : audit 1 from='client.? 172.21.15.23:0/1984846065' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T20:56:34.223 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:34 smithi146 bash[23809]: audit 2024-03-11T20:56:33.017313+0000 mon.a (mon.0) 1022 : audit 1 from='client.? 172.21.15.23:0/1769882656' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.1c"}]': finished 2024-03-11T20:56:34.223 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:34 smithi146 bash[23809]: cluster 2024-03-11T20:56:33.028625+0000 mon.a (mon.0) 1023 : cluster 0 osdmap e151: 8 total, 8 up, 7 in 2024-03-11T20:56:34.241 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:56:34.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:34.246 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:56:34.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:34 smithi023 bash[21372]: cluster 2024-03-11T20:56:32.997659+0000 mon.a (mon.0) 1020 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T20:56:34.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:34 smithi023 bash[21372]: audit 2024-03-11T20:56:33.017136+0000 mon.a (mon.0) 1021 : audit 1 from='client.? 172.21.15.23:0/1984846065' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T20:56:34.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:34 smithi023 bash[21372]: audit 2024-03-11T20:56:33.017313+0000 mon.a (mon.0) 1022 : audit 1 from='client.? 172.21.15.23:0/1769882656' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.1c"}]': finished 2024-03-11T20:56:34.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:34 smithi023 bash[21372]: cluster 2024-03-11T20:56:33.028625+0000 mon.a (mon.0) 1023 : cluster 0 osdmap e151: 8 total, 8 up, 7 in 2024-03-11T20:56:34.347 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:34.368 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:34.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:34.369 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:56:34.470 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:56:34.502 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:56:34.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:34.506 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:56:34.607 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:56:34.624 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:56:34.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:34.626 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:56:34.726 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:56:34.758 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:56:34.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:34.761 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:56:34.862 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:34.892 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:34.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:34.894 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:56:34.995 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:35.017 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:35.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:35.019 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:56:35.120 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:56:35.292 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:35 smithi023 bash[29146]: cluster 2024-03-11T20:56:33.938491+0000 mgr.y (mgr.24380) 218 : cluster 0 pgmap v203: 148 pgs: 148 active+clean; 6.3 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.1 KiB/s rd, 9.8 MiB/s wr, 814 op/s 2024-03-11T20:56:35.292 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:35 smithi023 bash[29146]: cluster 2024-03-11T20:56:34.046048+0000 mon.a (mon.0) 1024 : cluster 0 osdmap e152: 8 total, 8 up, 7 in 2024-03-11T20:56:35.292 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:35 smithi023 bash[29146]: audit 2024-03-11T20:56:34.461794+0000 mon.a (mon.0) 1025 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:56:35.292 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:35 smithi023 bash[29146]: audit 2024-03-11T20:56:34.732420+0000 mon.a (mon.0) 1026 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [3, 5]}]: dispatch 2024-03-11T20:56:35.292 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:35 smithi023 bash[21372]: cluster 2024-03-11T20:56:33.938491+0000 mgr.y (mgr.24380) 218 : cluster 0 pgmap v203: 148 pgs: 148 active+clean; 6.3 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.1 KiB/s rd, 9.8 MiB/s wr, 814 op/s 2024-03-11T20:56:35.292 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:35 smithi023 bash[21372]: cluster 2024-03-11T20:56:34.046048+0000 mon.a (mon.0) 1024 : cluster 0 osdmap e152: 8 total, 8 up, 7 in 2024-03-11T20:56:35.292 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:35 smithi023 bash[21372]: audit 2024-03-11T20:56:34.461794+0000 mon.a (mon.0) 1025 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:56:35.292 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:35 smithi023 bash[21372]: audit 2024-03-11T20:56:34.732420+0000 mon.a (mon.0) 1026 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [3, 5]}]: dispatch 2024-03-11T20:56:35.301 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 20:56:35 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T20:56:35.302 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:35 smithi146 bash[23809]: cluster 2024-03-11T20:56:33.938491+0000 mgr.y (mgr.24380) 218 : cluster 0 pgmap v203: 148 pgs: 148 active+clean; 6.3 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.1 KiB/s rd, 9.8 MiB/s wr, 814 op/s 2024-03-11T20:56:35.302 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:35 smithi146 bash[23809]: cluster 2024-03-11T20:56:34.046048+0000 mon.a (mon.0) 1024 : cluster 0 osdmap e152: 8 total, 8 up, 7 in 2024-03-11T20:56:35.302 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:35 smithi146 bash[23809]: audit 2024-03-11T20:56:34.461794+0000 mon.a (mon.0) 1025 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:56:35.302 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:35 smithi146 bash[23809]: audit 2024-03-11T20:56:34.732420+0000 mon.a (mon.0) 1026 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [3, 5]}]: dispatch 2024-03-11T20:56:35.315 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:56:35.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:35.318 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:56:35.419 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:35.443 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:35.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:35.448 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:56:35.549 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:56:35.568 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:56:35.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:35.570 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:56:35.671 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:56:35.698 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:56:35.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:35.700 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:56:35.801 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:56:35.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:35.833 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:56:35.833 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:56:35.934 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:56:35.960 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:56:35.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:35.963 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:56:36.065 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:56:36.091 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:56:36.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:36.095 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:56:36.195 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:36.238 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:36.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:36.244 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:56:36.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:36 smithi146 bash[23809]: audit 2024-03-11T20:56:35.085069+0000 mgr.y (mgr.24380) 219 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:56:36.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:36 smithi146 bash[23809]: audit 2024-03-11T20:56:35.689381+0000 mon.a (mon.0) 1027 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [3, 5]}]': finished 2024-03-11T20:56:36.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:36 smithi146 bash[23809]: cluster 2024-03-11T20:56:35.696837+0000 mon.a (mon.0) 1028 : cluster 0 osdmap e153: 8 total, 8 up, 7 in 2024-03-11T20:56:36.345 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:36.377 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:36.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:36.381 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:56:36.481 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:56:36.503 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:56:36.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:36.506 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:56:36.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:36 smithi023 bash[21372]: audit 2024-03-11T20:56:35.085069+0000 mgr.y (mgr.24380) 219 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:56:36.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:36 smithi023 bash[21372]: audit 2024-03-11T20:56:35.689381+0000 mon.a (mon.0) 1027 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [3, 5]}]': finished 2024-03-11T20:56:36.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:36 smithi023 bash[21372]: cluster 2024-03-11T20:56:35.696837+0000 mon.a (mon.0) 1028 : cluster 0 osdmap e153: 8 total, 8 up, 7 in 2024-03-11T20:56:36.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:36 smithi023 bash[29146]: audit 2024-03-11T20:56:35.085069+0000 mgr.y (mgr.24380) 219 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:56:36.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:36 smithi023 bash[29146]: audit 2024-03-11T20:56:35.689381+0000 mon.a (mon.0) 1027 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [3, 5]}]': finished 2024-03-11T20:56:36.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:36 smithi023 bash[29146]: cluster 2024-03-11T20:56:35.696837+0000 mon.a (mon.0) 1028 : cluster 0 osdmap e153: 8 total, 8 up, 7 in 2024-03-11T20:56:36.607 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:56:36.634 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:56:36.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:36.638 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:56:36.739 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:36.774 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:36.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:36.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:56:36.883 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:56:36.903 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:56:36.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:36.910 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:56:37.011 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:56:37.036 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:56:37.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:37.039 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:56:37.139 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:37.162 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:37.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:37.165 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:56:37.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:37 smithi146 bash[23809]: cluster 2024-03-11T20:56:35.938934+0000 mgr.y (mgr.24380) 220 : cluster 0 pgmap v206: 148 pgs: 148 active+clean; 6.0 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 1.24k op/s 2024-03-11T20:56:37.263 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:37 smithi146 bash[23809]: cluster 2024-03-11T20:56:36.705411+0000 mon.a (mon.0) 1029 : cluster 0 osdmap e154: 8 total, 8 up, 7 in 2024-03-11T20:56:37.265 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:56:37.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:37 smithi023 bash[21372]: cluster 2024-03-11T20:56:35.938934+0000 mgr.y (mgr.24380) 220 : cluster 0 pgmap v206: 148 pgs: 148 active+clean; 6.0 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 1.24k op/s 2024-03-11T20:56:37.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:37 smithi023 bash[21372]: cluster 2024-03-11T20:56:36.705411+0000 mon.a (mon.0) 1029 : cluster 0 osdmap e154: 8 total, 8 up, 7 in 2024-03-11T20:56:37.274 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:37 smithi023 bash[29146]: cluster 2024-03-11T20:56:35.938934+0000 mgr.y (mgr.24380) 220 : cluster 0 pgmap v206: 148 pgs: 148 active+clean; 6.0 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 1.24k op/s 2024-03-11T20:56:37.274 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:37 smithi023 bash[29146]: cluster 2024-03-11T20:56:36.705411+0000 mon.a (mon.0) 1029 : cluster 0 osdmap e154: 8 total, 8 up, 7 in 2024-03-11T20:56:37.295 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:56:37.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:37.296 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:56:37.376 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T20:56:37.397 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:37.451 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:37.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:37.454 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:56:37.555 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:56:37.583 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:56:37.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:37.586 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:56:37.687 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:56:37.722 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:56:37.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:37.732 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:56:37.833 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:37.866 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:37.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:37.870 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:56:37.972 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:56:37.999 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:56:38.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:38.001 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:56:38.102 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:38.131 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:38.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:38.134 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:56:38.235 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:38.243 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd unset noscrub 2024-03-11T20:56:38.256 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:38.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:38.259 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:56:38.360 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:38.388 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:38.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:38.393 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:56:38.494 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:56:38.510 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:56:38 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:20:56:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T20:56:38.510 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:56:38.527 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:56:38.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:38.530 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:56:38.631 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:56:38.665 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:56:38.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:38.669 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:56:38.770 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:38.799 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:38.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:38.802 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:56:38.903 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:38.928 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:38.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:38.931 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:56:39.032 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:39.060 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:39.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:39.062 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:56:39.163 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:39.188 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:39.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:39.190 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:56:39.291 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:56:39.321 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:56:39.323 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:39 smithi146 bash[23809]: cluster 2024-03-11T20:56:37.939405+0000 mgr.y (mgr.24380) 221 : cluster 0 pgmap v208: 148 pgs: 148 active+clean; 6.0 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 991 op/s 2024-03-11T20:56:39.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:39.324 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:56:39.424 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:56:39.436 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:39 smithi023 bash[29146]: cluster 2024-03-11T20:56:37.939405+0000 mgr.y (mgr.24380) 221 : cluster 0 pgmap v208: 148 pgs: 148 active+clean; 6.0 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 991 op/s 2024-03-11T20:56:39.436 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:39 smithi023 bash[21372]: cluster 2024-03-11T20:56:37.939405+0000 mgr.y (mgr.24380) 221 : cluster 0 pgmap v208: 148 pgs: 148 active+clean; 6.0 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 991 op/s 2024-03-11T20:56:39.452 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:56:39.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:39.454 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:56:39.555 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:56:39.580 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:56:39.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:39.582 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:56:39.683 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:56:39.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:39.712 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:56:39.712 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:56:39.813 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:39.837 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:39.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:39.842 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:56:39.943 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:39.976 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:39.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:39.981 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:56:40.082 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:56:40.110 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:56:40.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:40.114 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:56:40.215 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:56:40.246 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:56:40.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:40.249 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:56:40.350 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:56:40.380 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:56:40.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:40.383 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:56:40.484 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:40.515 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:40.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:40.518 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:56:40.619 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:56:40.647 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:56:40.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:40.654 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:56:40.755 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:40.764 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:40 smithi023 bash[21372]: cluster 2024-03-11T20:56:39.940342+0000 mgr.y (mgr.24380) 222 : cluster 0 pgmap v209: 148 pgs: 148 active+clean; 5.9 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.17k op/s 2024-03-11T20:56:40.765 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:40 smithi023 bash[29146]: cluster 2024-03-11T20:56:39.940342+0000 mgr.y (mgr.24380) 222 : cluster 0 pgmap v209: 148 pgs: 148 active+clean; 5.9 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.17k op/s 2024-03-11T20:56:40.785 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:40.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:40.787 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:56:40.888 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:40.912 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:40.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:40.915 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:56:41.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:40 smithi146 bash[23809]: cluster 2024-03-11T20:56:39.940342+0000 mgr.y (mgr.24380) 222 : cluster 0 pgmap v209: 148 pgs: 148 active+clean; 5.9 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.17k op/s 2024-03-11T20:56:41.016 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:41.044 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:41.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:41.050 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:56:41.151 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:56:41.179 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:56:41.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:41.181 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:56:41.221 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 6, 7] out_osds: [1] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-11T20:56:41.221 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-11T20:56:41.221 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-03-11T20:56:41.221 INFO:tasks.thrashosds.thrasher:Setting osd 0 primary_affinity to 0.401270 2024-03-11T20:56:41.221 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd primary-affinity 0 0.40126953695383216 2024-03-11T20:56:41.283 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:56:41.333 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:56:41.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:41.335 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:56:41.436 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:56:41.465 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:56:41.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:41.469 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:56:41.570 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:41.596 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:41.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:41.598 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:56:41.699 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:56:41.734 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:56:41.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:41.738 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:56:41.839 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:56:41.863 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:56:41.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:41.864 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:56:41.965 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:41.994 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:41.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:41.997 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:56:42.098 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:56:42.121 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:56:42.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:42.123 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:56:42.224 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:56:42.249 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:56:42.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:42.253 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:56:42.354 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:56:42.383 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:56:42.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:42.388 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:56:42.489 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:56:42.514 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:56:42.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:42.517 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:56:42.594 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:56:42.618 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:42.642 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:42.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:42.645 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:56:42.745 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:56:42.773 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:56:42.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:42.779 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:56:42.880 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:42.893 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:42 smithi023 bash[21372]: cluster 2024-03-11T20:56:41.144646+0000 osd.7 (osd.7) 135 : cluster 0 purged_snaps scrub starts 2024-03-11T20:56:42.893 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:42 smithi023 bash[21372]: cluster 2024-03-11T20:56:41.144702+0000 osd.7 (osd.7) 136 : cluster 0 purged_snaps scrub ok 2024-03-11T20:56:42.893 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:42 smithi023 bash[21372]: cluster 2024-03-11T20:56:41.708448+0000 mon.a (mon.0) 1030 : cluster 0 osdmap e155: 8 total, 8 up, 7 in 2024-03-11T20:56:42.893 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:42 smithi023 bash[21372]: cluster 2024-03-11T20:56:41.940803+0000 mgr.y (mgr.24380) 223 : cluster 0 pgmap v211: 148 pgs: 148 active+clean; 5.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 655 B/s rd, 0 B/s wr, 820 op/s 2024-03-11T20:56:42.893 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:42 smithi023 bash[29146]: cluster 2024-03-11T20:56:41.144646+0000 osd.7 (osd.7) 135 : cluster 0 purged_snaps scrub starts 2024-03-11T20:56:42.893 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:42 smithi023 bash[29146]: cluster 2024-03-11T20:56:41.144702+0000 osd.7 (osd.7) 136 : cluster 0 purged_snaps scrub ok 2024-03-11T20:56:42.894 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:42 smithi023 bash[29146]: cluster 2024-03-11T20:56:41.708448+0000 mon.a (mon.0) 1030 : cluster 0 osdmap e155: 8 total, 8 up, 7 in 2024-03-11T20:56:42.894 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:42 smithi023 bash[29146]: cluster 2024-03-11T20:56:41.940803+0000 mgr.y (mgr.24380) 223 : cluster 0 pgmap v211: 148 pgs: 148 active+clean; 5.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 655 B/s rd, 0 B/s wr, 820 op/s 2024-03-11T20:56:42.910 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:42.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:42.916 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:56:43.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:42 smithi146 bash[23809]: cluster 2024-03-11T20:56:41.144646+0000 osd.7 (osd.7) 135 : cluster 0 purged_snaps scrub starts 2024-03-11T20:56:43.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:42 smithi146 bash[23809]: cluster 2024-03-11T20:56:41.144702+0000 osd.7 (osd.7) 136 : cluster 0 purged_snaps scrub ok 2024-03-11T20:56:43.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:42 smithi146 bash[23809]: cluster 2024-03-11T20:56:41.708448+0000 mon.a (mon.0) 1030 : cluster 0 osdmap e155: 8 total, 8 up, 7 in 2024-03-11T20:56:43.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:42 smithi146 bash[23809]: cluster 2024-03-11T20:56:41.940803+0000 mgr.y (mgr.24380) 223 : cluster 0 pgmap v211: 148 pgs: 148 active+clean; 5.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 655 B/s rd, 0 B/s wr, 820 op/s 2024-03-11T20:56:43.017 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:56:43.055 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:56:43.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:43.062 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:56:43.162 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:56:43.189 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:56:43.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:43.192 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:56:43.293 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:56:43.315 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:56:43.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:43.319 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:56:43.420 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:56:43.445 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:56:43.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:43.452 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:56:43.552 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:56:43.581 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:56:43.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:43.584 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:56:43.664 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:56:43.685 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:43.725 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:43.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:43.728 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:56:43.829 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:43.858 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:43.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:43.863 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:56:43.965 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:43.995 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:44.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:44.001 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:56:44.102 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:56:44.141 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:56:44.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:44.144 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:56:44.245 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:56:44.274 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:56:44.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:44.276 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:56:44.377 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:44.411 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:44.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:44.414 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:56:44.515 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:44.545 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:44.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:44.548 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:56:44.649 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:44.678 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:44.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:44.680 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:56:44.781 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:56:44.802 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:56:44.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:44.804 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:56:44.905 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:56:44.927 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:56:44.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:44.928 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:56:45.029 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:45.060 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:45.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:45.062 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:56:45.067 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:45 smithi023 bash[21372]: cluster 2024-03-11T20:56:43.941219+0000 mgr.y (mgr.24380) 224 : cluster 0 pgmap v212: 148 pgs: 148 active+clean; 5.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 511 B/s rd, 0 B/s wr, 640 op/s 2024-03-11T20:56:45.089 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:45 smithi146 bash[23809]: cluster 2024-03-11T20:56:43.941219+0000 mgr.y (mgr.24380) 224 : cluster 0 pgmap v212: 148 pgs: 148 active+clean; 5.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 511 B/s rd, 0 B/s wr, 640 op/s 2024-03-11T20:56:45.163 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:45.212 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:45.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:45.217 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:56:45.260 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 20:56:45 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T20:56:45.319 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:56:45.345 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:56:45.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:45.346 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:56:45.447 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:45.460 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:45 smithi023 bash[29146]: cluster 2024-03-11T20:56:43.941219+0000 mgr.y (mgr.24380) 224 : cluster 0 pgmap v212: 148 pgs: 148 active+clean; 5.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 511 B/s rd, 0 B/s wr, 640 op/s 2024-03-11T20:56:45.472 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:45.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:45.475 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:56:45.576 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:56:45.600 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:56:45.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:45.602 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:56:45.703 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:56:45.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:45.734 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:56:45.734 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:56:45.835 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:45.862 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:45.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:45.865 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:56:45.966 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:56:45.995 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:56:45.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:45.998 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:56:46.099 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:56:46.120 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:56:46.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:46.122 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:56:46.223 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:56:46.232 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:46 smithi146 bash[23809]: audit 2024-03-11T20:56:45.092827+0000 mgr.y (mgr.24380) 225 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:56:46.249 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:56:46.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:46.253 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:56:46.279 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:46 smithi023 bash[21372]: audit 2024-03-11T20:56:45.092827+0000 mgr.y (mgr.24380) 225 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:56:46.279 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:46 smithi023 bash[29146]: audit 2024-03-11T20:56:45.092827+0000 mgr.y (mgr.24380) 225 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:56:46.317 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-11T20:56:46.317 INFO:teuthology.orchestra.run.smithi023.stdout: "ops": [], 2024-03-11T20:56:46.317 INFO:teuthology.orchestra.run.smithi023.stdout: "complaint_time": 30, 2024-03-11T20:56:46.317 INFO:teuthology.orchestra.run.smithi023.stdout: "num_blocked_ops": 0 2024-03-11T20:56:46.317 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-11T20:56:46.354 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:56:46.377 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:56:46.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:46.379 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:56:46.480 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:46.510 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:46.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:46.512 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:56:46.613 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:46.640 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:46.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:46.643 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:56:46.744 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:56:46.769 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:56:46.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:46.777 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:56:46.878 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:56:46.904 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:56:46.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:46.910 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:56:47.012 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:56:47.057 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:56:47.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:47.063 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:56:47.164 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:47.218 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:47.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:47.223 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:56:47.308 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:47 smithi023 bash[29146]: cluster 2024-03-11T20:56:45.942018+0000 mgr.y (mgr.24380) 226 : cluster 0 pgmap v213: 148 pgs: 148 active+clean; 5.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 997 B/s rd, 0 B/s wr, 1.15k op/s 2024-03-11T20:56:47.309 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:47 smithi023 bash[29146]: cluster 2024-03-11T20:56:46.026198+0000 osd.1 (osd.1) 95 : cluster 0 purged_snaps scrub starts 2024-03-11T20:56:47.309 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:47 smithi023 bash[29146]: cluster 2024-03-11T20:56:46.026281+0000 osd.1 (osd.1) 96 : cluster 0 purged_snaps scrub ok 2024-03-11T20:56:47.309 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:47 smithi023 bash[29146]: audit 2024-03-11T20:56:46.986823+0000 mon.a (mon.0) 1031 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:56:47.309 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:47 smithi023 bash[21372]: cluster 2024-03-11T20:56:45.942018+0000 mgr.y (mgr.24380) 226 : cluster 0 pgmap v213: 148 pgs: 148 active+clean; 5.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 997 B/s rd, 0 B/s wr, 1.15k op/s 2024-03-11T20:56:47.309 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:47 smithi023 bash[21372]: cluster 2024-03-11T20:56:46.026198+0000 osd.1 (osd.1) 95 : cluster 0 purged_snaps scrub starts 2024-03-11T20:56:47.309 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:47 smithi023 bash[21372]: cluster 2024-03-11T20:56:46.026281+0000 osd.1 (osd.1) 96 : cluster 0 purged_snaps scrub ok 2024-03-11T20:56:47.309 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:47 smithi023 bash[21372]: audit 2024-03-11T20:56:46.986823+0000 mon.a (mon.0) 1031 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:56:47.325 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:56:47.337 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:47 smithi146 bash[23809]: cluster 2024-03-11T20:56:45.942018+0000 mgr.y (mgr.24380) 226 : cluster 0 pgmap v213: 148 pgs: 148 active+clean; 5.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 997 B/s rd, 0 B/s wr, 1.15k op/s 2024-03-11T20:56:47.337 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:47 smithi146 bash[23809]: cluster 2024-03-11T20:56:46.026198+0000 osd.1 (osd.1) 95 : cluster 0 purged_snaps scrub starts 2024-03-11T20:56:47.337 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:47 smithi146 bash[23809]: cluster 2024-03-11T20:56:46.026281+0000 osd.1 (osd.1) 96 : cluster 0 purged_snaps scrub ok 2024-03-11T20:56:47.337 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:47 smithi146 bash[23809]: audit 2024-03-11T20:56:46.986823+0000 mon.a (mon.0) 1031 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:56:47.357 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:56:47.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:47.359 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:56:47.460 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:56:47.478 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:56:47.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:47.480 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:56:47.581 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:47.598 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:47.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:47.600 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:56:47.701 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:47.738 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:47.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:47.741 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:56:47.842 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:56:47.870 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:56:47.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:47.872 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:56:47.973 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:47.996 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:47.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:47.998 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:56:48.100 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:48.108 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:48 smithi023 bash[21372]: cluster 2024-03-11T20:56:47.044843+0000 mon.a (mon.0) 1032 : cluster 0 osdmap e156: 8 total, 8 up, 7 in 2024-03-11T20:56:48.108 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:48 smithi023 bash[21372]: audit 2024-03-11T20:56:47.818734+0000 mon.a (mon.0) 1033 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:56:48.109 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:48 smithi023 bash[21372]: audit 2024-03-11T20:56:47.822467+0000 mon.a (mon.0) 1034 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:56:48.109 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:48 smithi023 bash[21372]: audit 2024-03-11T20:56:47.842042+0000 mon.a (mon.0) 1035 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:56:48.109 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:48 smithi023 bash[29146]: cluster 2024-03-11T20:56:47.044843+0000 mon.a (mon.0) 1032 : cluster 0 osdmap e156: 8 total, 8 up, 7 in 2024-03-11T20:56:48.109 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:48 smithi023 bash[29146]: audit 2024-03-11T20:56:47.818734+0000 mon.a (mon.0) 1033 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:56:48.109 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:48 smithi023 bash[29146]: audit 2024-03-11T20:56:47.822467+0000 mon.a (mon.0) 1034 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:56:48.109 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:48 smithi023 bash[29146]: audit 2024-03-11T20:56:47.842042+0000 mon.a (mon.0) 1035 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:56:48.120 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:48.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:48.122 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:56:48.223 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:56:48.234 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:48 smithi146 bash[23809]: cluster 2024-03-11T20:56:47.044843+0000 mon.a (mon.0) 1032 : cluster 0 osdmap e156: 8 total, 8 up, 7 in 2024-03-11T20:56:48.234 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:48 smithi146 bash[23809]: audit 2024-03-11T20:56:47.818734+0000 mon.a (mon.0) 1033 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:56:48.234 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:48 smithi146 bash[23809]: audit 2024-03-11T20:56:47.822467+0000 mon.a (mon.0) 1034 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:56:48.234 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:48 smithi146 bash[23809]: audit 2024-03-11T20:56:47.842042+0000 mon.a (mon.0) 1035 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:56:48.248 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:56:48.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:48.250 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:56:48.351 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:48.377 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:48.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:48.380 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:56:48.481 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:56:48.502 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:56:48.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:48.503 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:56:48.537 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:56:48 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:20:56:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T20:56:48.605 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:56:48.635 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:56:48.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:48.638 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:56:48.739 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:48.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:48.772 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:48.772 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:56:48.873 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:56:48.907 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:56:48.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:48.910 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:56:49.011 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:56:49.043 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:56:49.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:49.046 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:56:49.146 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:56:49.157 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:49 smithi023 bash[21372]: cluster 2024-03-11T20:56:47.942613+0000 mgr.y (mgr.24380) 227 : cluster 0 pgmap v215: 148 pgs: 148 active+clean; 5.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.07k op/s 2024-03-11T20:56:49.157 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:49 smithi023 bash[29146]: cluster 2024-03-11T20:56:47.942613+0000 mgr.y (mgr.24380) 227 : cluster 0 pgmap v215: 148 pgs: 148 active+clean; 5.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.07k op/s 2024-03-11T20:56:49.177 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:56:49.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:49.179 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:56:49.280 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:56:49.304 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:56:49.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:49.308 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:56:49.409 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:56:49.419 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:49 smithi146 bash[23809]: cluster 2024-03-11T20:56:47.942613+0000 mgr.y (mgr.24380) 227 : cluster 0 pgmap v215: 148 pgs: 148 active+clean; 5.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.07k op/s 2024-03-11T20:56:49.434 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:56:49.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:49.436 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:56:49.536 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:56:49.558 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:56:49.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:49.561 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:56:49.662 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:49.689 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:49.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:49.692 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:56:49.793 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:49.817 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:49.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:49.819 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:56:49.920 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:56:49.940 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:56:49.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:49.942 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:56:49.981 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-03-11T20:56:50.042 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:56:50.062 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:56:50.063 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:50 smithi146 bash[23809]: audit 2024-03-11T20:56:49.480724+0000 mon.a (mon.0) 1036 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:56:50.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:50.064 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:56:50.165 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:56:50.195 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:56:50.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:50.199 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:56:50.300 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:56:50.321 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:50 smithi023 bash[21372]: audit 2024-03-11T20:56:49.480724+0000 mon.a (mon.0) 1036 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:56:50.329 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:56:50.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:50.333 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:56:50.370 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:56:50.434 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:50.451 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:50.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:50.453 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:56:50.534 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:50 smithi023 bash[29146]: audit 2024-03-11T20:56:49.480724+0000 mon.a (mon.0) 1036 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:56:50.554 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:56:50.576 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:56:50.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:50.578 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:56:50.679 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:56:50.723 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:56:50.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:50.727 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:56:50.828 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:50.855 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:50.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:50.859 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:56:50.960 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:56:50.984 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:56:50.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:50.991 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:56:51.092 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:56:51.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:51.126 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:56:51.127 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:56:51.227 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:51.237 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:51 smithi146 bash[23809]: cluster 2024-03-11T20:56:49.943248+0000 mgr.y (mgr.24380) 228 : cluster 0 pgmap v216: 148 pgs: 148 active+clean; 5.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 621 B/s rd, 0 B/s wr, 920 op/s 2024-03-11T20:56:51.237 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:51 smithi146 bash[23809]: cluster 2024-03-11T20:56:50.708770+0000 osd.2 (osd.2) 99 : cluster 0 purged_snaps scrub starts 2024-03-11T20:56:51.237 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:51 smithi146 bash[23809]: cluster 2024-03-11T20:56:50.708837+0000 osd.2 (osd.2) 100 : cluster 0 purged_snaps scrub ok 2024-03-11T20:56:51.249 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:51.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:51.251 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:56:51.352 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:56:51.378 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:56:51.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:51.382 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:56:51.483 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:56:51.506 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:56:51.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:51.508 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:56:51.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:51 smithi023 bash[21372]: cluster 2024-03-11T20:56:49.943248+0000 mgr.y (mgr.24380) 228 : cluster 0 pgmap v216: 148 pgs: 148 active+clean; 5.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 621 B/s rd, 0 B/s wr, 920 op/s 2024-03-11T20:56:51.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:51 smithi023 bash[21372]: cluster 2024-03-11T20:56:50.708770+0000 osd.2 (osd.2) 99 : cluster 0 purged_snaps scrub starts 2024-03-11T20:56:51.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:51 smithi023 bash[21372]: cluster 2024-03-11T20:56:50.708837+0000 osd.2 (osd.2) 100 : cluster 0 purged_snaps scrub ok 2024-03-11T20:56:51.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:51 smithi023 bash[29146]: cluster 2024-03-11T20:56:49.943248+0000 mgr.y (mgr.24380) 228 : cluster 0 pgmap v216: 148 pgs: 148 active+clean; 5.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 621 B/s rd, 0 B/s wr, 920 op/s 2024-03-11T20:56:51.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:51 smithi023 bash[29146]: cluster 2024-03-11T20:56:50.708770+0000 osd.2 (osd.2) 99 : cluster 0 purged_snaps scrub starts 2024-03-11T20:56:51.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:51 smithi023 bash[29146]: cluster 2024-03-11T20:56:50.708837+0000 osd.2 (osd.2) 100 : cluster 0 purged_snaps scrub ok 2024-03-11T20:56:51.609 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:51.636 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:51.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:51.641 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:56:51.741 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:56:51.768 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:56:51.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:51.770 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:56:51.871 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:56:51.891 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:56:51.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:51.893 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:56:51.994 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:52.022 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:52.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:52.025 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:56:52.126 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:56:52.153 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:56:52.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:52.155 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:56:52.256 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:56:52.292 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:56:52.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:52.297 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:56:52.398 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:52.426 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:52.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:52.430 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:56:52.531 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:52.563 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:52.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:52.568 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:56:52.669 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:52.693 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:52.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:52.696 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:56:52.798 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:56:52.801 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:52 smithi023 bash[21372]: cluster 2024-03-11T20:56:51.717505+0000 mon.a (mon.0) 1037 : cluster 0 osdmap e157: 8 total, 8 up, 7 in 2024-03-11T20:56:52.801 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:52 smithi023 bash[21372]: cluster 2024-03-11T20:56:51.945666+0000 mgr.y (mgr.24380) 229 : cluster 0 pgmap v218: 148 pgs: 148 active+clean; 5.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.28k op/s 2024-03-11T20:56:52.802 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:52 smithi023 bash[29146]: cluster 2024-03-11T20:56:51.717505+0000 mon.a (mon.0) 1037 : cluster 0 osdmap e157: 8 total, 8 up, 7 in 2024-03-11T20:56:52.802 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:52 smithi023 bash[29146]: cluster 2024-03-11T20:56:51.945666+0000 mgr.y (mgr.24380) 229 : cluster 0 pgmap v218: 148 pgs: 148 active+clean; 5.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.28k op/s 2024-03-11T20:56:52.825 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:56:52.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:52.832 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:56:52.926 INFO:teuthology.orchestra.run.smithi023.stdout:osd.0: {} 2024-03-11T20:56:52.932 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:52.956 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:52.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:52.959 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:56:52.988 INFO:teuthology.orchestra.run.smithi023.stdout:osd.1: {} 2024-03-11T20:56:53.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:52 smithi146 bash[23809]: cluster 2024-03-11T20:56:51.717505+0000 mon.a (mon.0) 1037 : cluster 0 osdmap e157: 8 total, 8 up, 7 in 2024-03-11T20:56:53.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:52 smithi146 bash[23809]: cluster 2024-03-11T20:56:51.945666+0000 mgr.y (mgr.24380) 229 : cluster 0 pgmap v218: 148 pgs: 148 active+clean; 5.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.28k op/s 2024-03-11T20:56:53.034 INFO:teuthology.orchestra.run.smithi023.stdout:osd.2: {} 2024-03-11T20:56:53.060 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:53.080 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:53.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:53.085 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:56:53.090 INFO:teuthology.orchestra.run.smithi023.stdout:osd.3: {} 2024-03-11T20:56:53.139 INFO:teuthology.orchestra.run.smithi023.stdout:osd.4: {} 2024-03-11T20:56:53.186 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:56:53.188 INFO:teuthology.orchestra.run.smithi023.stdout:osd.5: {} 2024-03-11T20:56:53.207 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:56:53.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:53.209 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:56:53.220 INFO:teuthology.orchestra.run.smithi023.stdout:osd.6: {} 2024-03-11T20:56:53.262 INFO:teuthology.orchestra.run.smithi023.stdout:osd.7: {} 2024-03-11T20:56:53.271 INFO:teuthology.orchestra.run.smithi023.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T20:56:53.271 INFO:teuthology.orchestra.run.smithi023.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T20:56:53.271 INFO:teuthology.orchestra.run.smithi023.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T20:56:53.271 INFO:teuthology.orchestra.run.smithi023.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T20:56:53.271 INFO:teuthology.orchestra.run.smithi023.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T20:56:53.271 INFO:teuthology.orchestra.run.smithi023.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T20:56:53.271 INFO:teuthology.orchestra.run.smithi023.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T20:56:53.271 INFO:teuthology.orchestra.run.smithi023.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T20:56:53.310 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:53.337 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:53.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:53.339 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:56:53.440 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:53.465 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:53.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:53.470 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:56:53.571 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:56:53.594 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:56:53.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:53.599 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:56:53.700 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:56:53.725 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:56:53.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:53.731 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:56:53.741 INFO:teuthology.orchestra.run.smithi023.stderr:noscrub is unset 2024-03-11T20:56:53.832 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:56:53.861 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:56:53.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:53.868 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:56:53.968 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:53.978 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:53 smithi023 bash[21372]: audit 2024-03-11T20:56:52.890283+0000 mon.b (mon.1) 33 : audit 0 from='client.? 172.21.15.23:0/3944150997' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T20:56:53.978 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:53 smithi023 bash[21372]: audit 2024-03-11T20:56:53.562339+0000 mon.a (mon.0) 1038 : audit 1 from='client.? 172.21.15.23:0/1680743124' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T20:56:53.978 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:53 smithi023 bash[29146]: audit 2024-03-11T20:56:52.890283+0000 mon.b (mon.1) 33 : audit 0 from='client.? 172.21.15.23:0/3944150997' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T20:56:53.978 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:53 smithi023 bash[29146]: audit 2024-03-11T20:56:53.562339+0000 mon.a (mon.0) 1038 : audit 1 from='client.? 172.21.15.23:0/1680743124' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T20:56:53.994 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:53.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:53.998 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:56:54.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:53 smithi146 bash[23809]: audit 2024-03-11T20:56:52.890283+0000 mon.b (mon.1) 33 : audit 0 from='client.? 172.21.15.23:0/3944150997' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T20:56:54.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:53 smithi146 bash[23809]: audit 2024-03-11T20:56:53.562339+0000 mon.a (mon.0) 1038 : audit 1 from='client.? 172.21.15.23:0/1680743124' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T20:56:54.099 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:56:54.118 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:56:54.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:54.120 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:56:54.221 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:54.241 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:54.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:54.243 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:56:54.344 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:56:54.369 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:56:54.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:54.371 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:56:54.472 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:56:54.502 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:56:54.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:54.508 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:56:54.609 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:54.643 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:54.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:54.647 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:56:54.747 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:56:54.778 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:56:54.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:54.784 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:56:54.876 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:54 smithi023 bash[21372]: cluster 2024-03-11T20:56:53.726497+0000 mon.a (mon.0) 1039 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T20:56:54.876 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:54 smithi023 bash[21372]: audit 2024-03-11T20:56:53.736464+0000 mon.a (mon.0) 1040 : audit 1 from='client.? 172.21.15.23:0/1680743124' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T20:56:54.876 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:54 smithi023 bash[21372]: cluster 2024-03-11T20:56:53.740523+0000 mon.a (mon.0) 1041 : cluster 0 osdmap e158: 8 total, 8 up, 7 in 2024-03-11T20:56:54.876 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:54 smithi023 bash[21372]: cluster 2024-03-11T20:56:53.909081+0000 osd.6 (osd.6) 83 : cluster 0 2.6 scrub starts 2024-03-11T20:56:54.876 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:54 smithi023 bash[21372]: cluster 2024-03-11T20:56:53.910476+0000 osd.6 (osd.6) 84 : cluster 0 2.6 scrub ok 2024-03-11T20:56:54.876 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:54 smithi023 bash[21372]: cluster 2024-03-11T20:56:53.946125+0000 mgr.y (mgr.24380) 230 : cluster 0 pgmap v220: 148 pgs: 148 active+clean; 5.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 889 B/s rd, 0 B/s wr, 678 op/s 2024-03-11T20:56:54.876 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:54 smithi023 bash[21372]: cluster 2024-03-11T20:56:54.679830+0000 osd.5 (osd.5) 119 : cluster 0 6.1d scrub starts 2024-03-11T20:56:54.876 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:54 smithi023 bash[21372]: cluster 2024-03-11T20:56:54.683287+0000 osd.5 (osd.5) 120 : cluster 0 6.1d scrub ok 2024-03-11T20:56:54.876 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:54 smithi023 bash[29146]: cluster 2024-03-11T20:56:53.726497+0000 mon.a (mon.0) 1039 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T20:56:54.877 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:54 smithi023 bash[29146]: audit 2024-03-11T20:56:53.736464+0000 mon.a (mon.0) 1040 : audit 1 from='client.? 172.21.15.23:0/1680743124' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T20:56:54.877 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:54 smithi023 bash[29146]: cluster 2024-03-11T20:56:53.740523+0000 mon.a (mon.0) 1041 : cluster 0 osdmap e158: 8 total, 8 up, 7 in 2024-03-11T20:56:54.877 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:54 smithi023 bash[29146]: cluster 2024-03-11T20:56:53.909081+0000 osd.6 (osd.6) 83 : cluster 0 2.6 scrub starts 2024-03-11T20:56:54.877 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:54 smithi023 bash[29146]: cluster 2024-03-11T20:56:53.910476+0000 osd.6 (osd.6) 84 : cluster 0 2.6 scrub ok 2024-03-11T20:56:54.877 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:54 smithi023 bash[29146]: cluster 2024-03-11T20:56:53.946125+0000 mgr.y (mgr.24380) 230 : cluster 0 pgmap v220: 148 pgs: 148 active+clean; 5.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 889 B/s rd, 0 B/s wr, 678 op/s 2024-03-11T20:56:54.877 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:54 smithi023 bash[29146]: cluster 2024-03-11T20:56:54.679830+0000 osd.5 (osd.5) 119 : cluster 0 6.1d scrub starts 2024-03-11T20:56:54.877 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:54 smithi023 bash[29146]: cluster 2024-03-11T20:56:54.683287+0000 osd.5 (osd.5) 120 : cluster 0 6.1d scrub ok 2024-03-11T20:56:54.886 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:54.909 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:54.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:54.912 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:56:55.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:54 smithi146 bash[23809]: cluster 2024-03-11T20:56:53.726497+0000 mon.a (mon.0) 1039 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T20:56:55.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:54 smithi146 bash[23809]: audit 2024-03-11T20:56:53.736464+0000 mon.a (mon.0) 1040 : audit 1 from='client.? 172.21.15.23:0/1680743124' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T20:56:55.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:54 smithi146 bash[23809]: cluster 2024-03-11T20:56:53.740523+0000 mon.a (mon.0) 1041 : cluster 0 osdmap e158: 8 total, 8 up, 7 in 2024-03-11T20:56:55.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:54 smithi146 bash[23809]: cluster 2024-03-11T20:56:53.909081+0000 osd.6 (osd.6) 83 : cluster 0 2.6 scrub starts 2024-03-11T20:56:55.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:54 smithi146 bash[23809]: cluster 2024-03-11T20:56:53.910476+0000 osd.6 (osd.6) 84 : cluster 0 2.6 scrub ok 2024-03-11T20:56:55.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:54 smithi146 bash[23809]: cluster 2024-03-11T20:56:53.946125+0000 mgr.y (mgr.24380) 230 : cluster 0 pgmap v220: 148 pgs: 148 active+clean; 5.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 889 B/s rd, 0 B/s wr, 678 op/s 2024-03-11T20:56:55.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:54 smithi146 bash[23809]: cluster 2024-03-11T20:56:54.679830+0000 osd.5 (osd.5) 119 : cluster 0 6.1d scrub starts 2024-03-11T20:56:55.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:54 smithi146 bash[23809]: cluster 2024-03-11T20:56:54.683287+0000 osd.5 (osd.5) 120 : cluster 0 6.1d scrub ok 2024-03-11T20:56:55.012 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:56:55.033 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:56:55.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:55.035 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:56:55.136 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:55.145 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 20:56:55 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T20:56:55.158 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:55.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:55.160 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:56:55.260 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:56:55.288 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:56:55.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:55.291 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:56:55.392 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:56:55.430 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:56:55.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:55.435 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:56:55.536 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:56:55.563 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:56:55.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:55.566 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:56:55.667 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:56:55.689 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:56:55.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:55.691 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:56:55.792 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:55.830 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:55.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:55.833 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:56:55.934 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:56:55.942 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:55 smithi146 bash[23809]: cluster 2024-03-11T20:56:54.031492+0000 osd.4 (osd.4) 127 : cluster 0 4.1 scrub starts 2024-03-11T20:56:55.942 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:55 smithi146 bash[23809]: cluster 2024-03-11T20:56:54.035011+0000 osd.4 (osd.4) 128 : cluster 0 4.1 scrub ok 2024-03-11T20:56:55.942 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:55 smithi146 bash[23809]: cluster 2024-03-11T20:56:54.411520+0000 osd.2 (osd.2) 101 : cluster 0 4.4 scrub starts 2024-03-11T20:56:55.942 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:55 smithi146 bash[23809]: cluster 2024-03-11T20:56:54.415611+0000 osd.2 (osd.2) 102 : cluster 0 4.4 scrub ok 2024-03-11T20:56:55.943 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:55 smithi146 bash[23809]: audit 2024-03-11T20:56:55.100764+0000 mgr.y (mgr.24380) 231 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:56:55.943 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:55 smithi146 bash[23809]: cluster 2024-03-11T20:56:55.629322+0000 osd.5 (osd.5) 121 : cluster 0 6.a scrub starts 2024-03-11T20:56:55.943 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:55 smithi146 bash[23809]: cluster 2024-03-11T20:56:55.630636+0000 osd.5 (osd.5) 122 : cluster 0 6.a scrub ok 2024-03-11T20:56:55.956 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:56:55.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:55.958 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:56:56.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:55 smithi023 bash[21372]: cluster 2024-03-11T20:56:54.031492+0000 osd.4 (osd.4) 127 : cluster 0 4.1 scrub starts 2024-03-11T20:56:56.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:55 smithi023 bash[21372]: cluster 2024-03-11T20:56:54.035011+0000 osd.4 (osd.4) 128 : cluster 0 4.1 scrub ok 2024-03-11T20:56:56.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:55 smithi023 bash[21372]: cluster 2024-03-11T20:56:54.411520+0000 osd.2 (osd.2) 101 : cluster 0 4.4 scrub starts 2024-03-11T20:56:56.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:55 smithi023 bash[21372]: cluster 2024-03-11T20:56:54.415611+0000 osd.2 (osd.2) 102 : cluster 0 4.4 scrub ok 2024-03-11T20:56:56.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:55 smithi023 bash[21372]: audit 2024-03-11T20:56:55.100764+0000 mgr.y (mgr.24380) 231 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:56:56.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:55 smithi023 bash[21372]: cluster 2024-03-11T20:56:55.629322+0000 osd.5 (osd.5) 121 : cluster 0 6.a scrub starts 2024-03-11T20:56:56.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:55 smithi023 bash[21372]: cluster 2024-03-11T20:56:55.630636+0000 osd.5 (osd.5) 122 : cluster 0 6.a scrub ok 2024-03-11T20:56:56.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:55 smithi023 bash[29146]: cluster 2024-03-11T20:56:54.031492+0000 osd.4 (osd.4) 127 : cluster 0 4.1 scrub starts 2024-03-11T20:56:56.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:55 smithi023 bash[29146]: cluster 2024-03-11T20:56:54.035011+0000 osd.4 (osd.4) 128 : cluster 0 4.1 scrub ok 2024-03-11T20:56:56.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:55 smithi023 bash[29146]: cluster 2024-03-11T20:56:54.411520+0000 osd.2 (osd.2) 101 : cluster 0 4.4 scrub starts 2024-03-11T20:56:56.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:55 smithi023 bash[29146]: cluster 2024-03-11T20:56:54.415611+0000 osd.2 (osd.2) 102 : cluster 0 4.4 scrub ok 2024-03-11T20:56:56.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:55 smithi023 bash[29146]: audit 2024-03-11T20:56:55.100764+0000 mgr.y (mgr.24380) 231 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:56:56.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:55 smithi023 bash[29146]: cluster 2024-03-11T20:56:55.629322+0000 osd.5 (osd.5) 121 : cluster 0 6.a scrub starts 2024-03-11T20:56:56.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:55 smithi023 bash[29146]: cluster 2024-03-11T20:56:55.630636+0000 osd.5 (osd.5) 122 : cluster 0 6.a scrub ok 2024-03-11T20:56:56.059 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:56.088 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:56.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:56.090 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:56:56.191 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:56.215 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:56.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:56.217 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:56:56.317 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:56.347 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:56.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:56.349 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:56:56.450 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:56:56.476 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:56:56.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:56.479 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:56:56.580 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:56.605 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:56.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:56.612 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:56:56.713 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:56.737 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:56.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:56.741 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:56:56.842 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:56:56.864 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:56:56.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:56.866 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:56:56.967 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:56:56.978 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:56 smithi023 bash[21372]: cluster 2024-03-11T20:56:55.889041+0000 osd.6 (osd.6) 85 : cluster 0 4.1f scrub starts 2024-03-11T20:56:56.978 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:56 smithi023 bash[21372]: cluster 2024-03-11T20:56:55.894480+0000 osd.6 (osd.6) 86 : cluster 0 4.1f scrub ok 2024-03-11T20:56:56.978 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:56 smithi023 bash[21372]: cluster 2024-03-11T20:56:55.946813+0000 mgr.y (mgr.24380) 232 : cluster 0 pgmap v221: 148 pgs: 148 active+clean; 4.8 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.17k op/s 2024-03-11T20:56:56.978 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:56 smithi023 bash[21372]: cluster 2024-03-11T20:56:56.613233+0000 osd.5 (osd.5) 123 : cluster 0 2.12 scrub starts 2024-03-11T20:56:56.978 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:56 smithi023 bash[21372]: cluster 2024-03-11T20:56:56.614253+0000 osd.5 (osd.5) 124 : cluster 0 2.12 scrub ok 2024-03-11T20:56:56.978 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:56 smithi023 bash[29146]: cluster 2024-03-11T20:56:55.889041+0000 osd.6 (osd.6) 85 : cluster 0 4.1f scrub starts 2024-03-11T20:56:56.979 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:56 smithi023 bash[29146]: cluster 2024-03-11T20:56:55.894480+0000 osd.6 (osd.6) 86 : cluster 0 4.1f scrub ok 2024-03-11T20:56:56.979 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:56 smithi023 bash[29146]: cluster 2024-03-11T20:56:55.946813+0000 mgr.y (mgr.24380) 232 : cluster 0 pgmap v221: 148 pgs: 148 active+clean; 4.8 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.17k op/s 2024-03-11T20:56:56.979 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:56 smithi023 bash[29146]: cluster 2024-03-11T20:56:56.613233+0000 osd.5 (osd.5) 123 : cluster 0 2.12 scrub starts 2024-03-11T20:56:56.979 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:56 smithi023 bash[29146]: cluster 2024-03-11T20:56:56.614253+0000 osd.5 (osd.5) 124 : cluster 0 2.12 scrub ok 2024-03-11T20:56:56.991 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:56:56.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:56.993 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:56:57.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:56 smithi146 bash[23809]: cluster 2024-03-11T20:56:55.889041+0000 osd.6 (osd.6) 85 : cluster 0 4.1f scrub starts 2024-03-11T20:56:57.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:56 smithi146 bash[23809]: cluster 2024-03-11T20:56:55.894480+0000 osd.6 (osd.6) 86 : cluster 0 4.1f scrub ok 2024-03-11T20:56:57.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:56 smithi146 bash[23809]: cluster 2024-03-11T20:56:55.946813+0000 mgr.y (mgr.24380) 232 : cluster 0 pgmap v221: 148 pgs: 148 active+clean; 4.8 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.17k op/s 2024-03-11T20:56:57.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:56 smithi146 bash[23809]: cluster 2024-03-11T20:56:56.613233+0000 osd.5 (osd.5) 123 : cluster 0 2.12 scrub starts 2024-03-11T20:56:57.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:56 smithi146 bash[23809]: cluster 2024-03-11T20:56:56.614253+0000 osd.5 (osd.5) 124 : cluster 0 2.12 scrub ok 2024-03-11T20:56:57.094 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:56:57.118 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:56:57.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:57.121 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:56:57.222 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:57.248 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:57.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:57.253 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:56:57.354 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:56:57.375 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:56:57.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:57.376 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:56:57.477 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:57.506 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:57.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:57.515 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:56:57.616 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:57.643 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:57.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:57.649 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:56:57.749 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:57.798 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:57.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:57.801 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:56:57.902 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:56:57.937 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:56:57.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:57.940 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:56:58.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:57 smithi146 bash[23809]: cluster 2024-03-11T20:56:56.884379+0000 osd.6 (osd.6) 87 : cluster 0 3.10 scrub starts 2024-03-11T20:56:58.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:57 smithi146 bash[23809]: cluster 2024-03-11T20:56:56.886847+0000 osd.6 (osd.6) 88 : cluster 0 3.10 scrub ok 2024-03-11T20:56:58.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:57 smithi023 bash[21372]: cluster 2024-03-11T20:56:56.884379+0000 osd.6 (osd.6) 87 : cluster 0 3.10 scrub starts 2024-03-11T20:56:58.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:57 smithi023 bash[21372]: cluster 2024-03-11T20:56:56.886847+0000 osd.6 (osd.6) 88 : cluster 0 3.10 scrub ok 2024-03-11T20:56:58.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:57 smithi023 bash[29146]: cluster 2024-03-11T20:56:56.884379+0000 osd.6 (osd.6) 87 : cluster 0 3.10 scrub starts 2024-03-11T20:56:58.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:57 smithi023 bash[29146]: cluster 2024-03-11T20:56:56.886847+0000 osd.6 (osd.6) 88 : cluster 0 3.10 scrub ok 2024-03-11T20:56:58.041 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:56:58.063 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:56:58.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:58.064 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:56:58.166 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:58.198 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:56:58.202 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:58.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:58.209 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:56:58.310 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:56:58.344 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:56:58.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:58.346 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:56:58.447 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:58.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:58.483 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:56:58.483 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:56:58.584 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:56:58.615 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:56:58.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:58.620 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:56:58.721 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:56:58.747 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:56:58.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:58.749 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:56:58.777 INFO:teuthology.orchestra.run.smithi023.stderr:set osd.0 primary-affinity to 0.40127 (8262972) 2024-03-11T20:56:58.838 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:56:58 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:20:56:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T20:56:58.850 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:58.868 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:56:58.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:58.869 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:56:58.970 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:58.980 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:58 smithi146 bash[23809]: cluster 2024-03-11T20:56:56.974510+0000 osd.4 (osd.4) 129 : cluster 0 2.d scrub starts 2024-03-11T20:56:58.981 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:58 smithi146 bash[23809]: cluster 2024-03-11T20:56:56.975628+0000 osd.4 (osd.4) 130 : cluster 0 2.d scrub ok 2024-03-11T20:56:58.981 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:58 smithi146 bash[23809]: cluster 2024-03-11T20:56:57.424125+0000 osd.3 (osd.3) 161 : cluster 0 6.11 scrub starts 2024-03-11T20:56:58.981 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:58 smithi146 bash[23809]: cluster 2024-03-11T20:56:57.424722+0000 osd.2 (osd.2) 103 : cluster 0 6.14 scrub starts 2024-03-11T20:56:58.981 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:58 smithi146 bash[23809]: cluster 2024-03-11T20:56:57.427886+0000 osd.2 (osd.2) 104 : cluster 0 6.14 scrub ok 2024-03-11T20:56:58.981 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:58 smithi146 bash[23809]: cluster 2024-03-11T20:56:57.434134+0000 osd.3 (osd.3) 162 : cluster 0 6.11 scrub ok 2024-03-11T20:56:58.981 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:58 smithi146 bash[23809]: cluster 2024-03-11T20:56:57.907832+0000 osd.6 (osd.6) 89 : cluster 0 2.14 scrub starts 2024-03-11T20:56:58.981 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:58 smithi146 bash[23809]: cluster 2024-03-11T20:56:57.909151+0000 osd.6 (osd.6) 90 : cluster 0 2.14 scrub ok 2024-03-11T20:56:58.981 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:58 smithi146 bash[23809]: cluster 2024-03-11T20:56:57.947237+0000 mgr.y (mgr.24380) 233 : cluster 0 pgmap v222: 148 pgs: 148 active+clean; 4.8 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 917 op/s 2024-03-11T20:56:58.981 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:58 smithi146 bash[23809]: cluster 2024-03-11T20:56:58.613363+0000 osd.5 (osd.5) 125 : cluster 0 4.2 scrub starts 2024-03-11T20:56:58.981 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:58 smithi146 bash[23809]: cluster 2024-03-11T20:56:58.614732+0000 osd.5 (osd.5) 126 : cluster 0 4.2 scrub ok 2024-03-11T20:56:58.981 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:58 smithi146 bash[23809]: audit 2024-03-11T20:56:58.684004+0000 mon.a (mon.0) 1042 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.40126953695383216}]: dispatch 2024-03-11T20:56:58.981 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:58 smithi146 bash[23809]: audit 2024-03-11T20:56:58.687846+0000 mon.b (mon.1) 34 : audit 1 from='client.? 172.21.15.23:0/639710456' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.40126953695383216}]: dispatch 2024-03-11T20:56:58.994 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:56:58.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:58.997 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:56:59.098 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:56:59.125 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:56:59.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:59.126 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:56:59.227 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:56:59.254 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:56:59.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:59.256 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:56:59.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:58 smithi023 bash[21372]: cluster 2024-03-11T20:56:56.974510+0000 osd.4 (osd.4) 129 : cluster 0 2.d scrub starts 2024-03-11T20:56:59.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:58 smithi023 bash[21372]: cluster 2024-03-11T20:56:56.975628+0000 osd.4 (osd.4) 130 : cluster 0 2.d scrub ok 2024-03-11T20:56:59.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:58 smithi023 bash[21372]: cluster 2024-03-11T20:56:57.424125+0000 osd.3 (osd.3) 161 : cluster 0 6.11 scrub starts 2024-03-11T20:56:59.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:58 smithi023 bash[21372]: cluster 2024-03-11T20:56:57.424722+0000 osd.2 (osd.2) 103 : cluster 0 6.14 scrub starts 2024-03-11T20:56:59.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:58 smithi023 bash[21372]: cluster 2024-03-11T20:56:57.427886+0000 osd.2 (osd.2) 104 : cluster 0 6.14 scrub ok 2024-03-11T20:56:59.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:58 smithi023 bash[21372]: cluster 2024-03-11T20:56:57.434134+0000 osd.3 (osd.3) 162 : cluster 0 6.11 scrub ok 2024-03-11T20:56:59.286 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:58 smithi023 bash[21372]: cluster 2024-03-11T20:56:57.907832+0000 osd.6 (osd.6) 89 : cluster 0 2.14 scrub starts 2024-03-11T20:56:59.286 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:58 smithi023 bash[21372]: cluster 2024-03-11T20:56:57.909151+0000 osd.6 (osd.6) 90 : cluster 0 2.14 scrub ok 2024-03-11T20:56:59.286 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:58 smithi023 bash[21372]: cluster 2024-03-11T20:56:57.947237+0000 mgr.y (mgr.24380) 233 : cluster 0 pgmap v222: 148 pgs: 148 active+clean; 4.8 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 917 op/s 2024-03-11T20:56:59.286 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:58 smithi023 bash[21372]: cluster 2024-03-11T20:56:58.613363+0000 osd.5 (osd.5) 125 : cluster 0 4.2 scrub starts 2024-03-11T20:56:59.286 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:58 smithi023 bash[21372]: cluster 2024-03-11T20:56:58.614732+0000 osd.5 (osd.5) 126 : cluster 0 4.2 scrub ok 2024-03-11T20:56:59.286 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:58 smithi023 bash[21372]: audit 2024-03-11T20:56:58.684004+0000 mon.a (mon.0) 1042 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.40126953695383216}]: dispatch 2024-03-11T20:56:59.286 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:58 smithi023 bash[21372]: audit 2024-03-11T20:56:58.687846+0000 mon.b (mon.1) 34 : audit 1 from='client.? 172.21.15.23:0/639710456' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.40126953695383216}]: dispatch 2024-03-11T20:56:59.286 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:58 smithi023 bash[29146]: cluster 2024-03-11T20:56:56.974510+0000 osd.4 (osd.4) 129 : cluster 0 2.d scrub starts 2024-03-11T20:56:59.286 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:58 smithi023 bash[29146]: cluster 2024-03-11T20:56:56.975628+0000 osd.4 (osd.4) 130 : cluster 0 2.d scrub ok 2024-03-11T20:56:59.286 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:58 smithi023 bash[29146]: cluster 2024-03-11T20:56:57.424125+0000 osd.3 (osd.3) 161 : cluster 0 6.11 scrub starts 2024-03-11T20:56:59.286 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:58 smithi023 bash[29146]: cluster 2024-03-11T20:56:57.424722+0000 osd.2 (osd.2) 103 : cluster 0 6.14 scrub starts 2024-03-11T20:56:59.287 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:58 smithi023 bash[29146]: cluster 2024-03-11T20:56:57.427886+0000 osd.2 (osd.2) 104 : cluster 0 6.14 scrub ok 2024-03-11T20:56:59.287 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:58 smithi023 bash[29146]: cluster 2024-03-11T20:56:57.434134+0000 osd.3 (osd.3) 162 : cluster 0 6.11 scrub ok 2024-03-11T20:56:59.287 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:58 smithi023 bash[29146]: cluster 2024-03-11T20:56:57.907832+0000 osd.6 (osd.6) 89 : cluster 0 2.14 scrub starts 2024-03-11T20:56:59.287 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:58 smithi023 bash[29146]: cluster 2024-03-11T20:56:57.909151+0000 osd.6 (osd.6) 90 : cluster 0 2.14 scrub ok 2024-03-11T20:56:59.287 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:58 smithi023 bash[29146]: cluster 2024-03-11T20:56:57.947237+0000 mgr.y (mgr.24380) 233 : cluster 0 pgmap v222: 148 pgs: 148 active+clean; 4.8 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 917 op/s 2024-03-11T20:56:59.287 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:58 smithi023 bash[29146]: cluster 2024-03-11T20:56:58.613363+0000 osd.5 (osd.5) 125 : cluster 0 4.2 scrub starts 2024-03-11T20:56:59.287 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:58 smithi023 bash[29146]: cluster 2024-03-11T20:56:58.614732+0000 osd.5 (osd.5) 126 : cluster 0 4.2 scrub ok 2024-03-11T20:56:59.287 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:58 smithi023 bash[29146]: audit 2024-03-11T20:56:58.684004+0000 mon.a (mon.0) 1042 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.40126953695383216}]: dispatch 2024-03-11T20:56:59.287 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:58 smithi023 bash[29146]: audit 2024-03-11T20:56:58.687846+0000 mon.b (mon.1) 34 : audit 1 from='client.? 172.21.15.23:0/639710456' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.40126953695383216}]: dispatch 2024-03-11T20:56:59.357 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:56:59.385 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:56:59.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:59.387 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:56:59.487 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:56:59.513 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:56:59.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:59.515 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:56:59.616 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:56:59.645 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:56:59.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:59.647 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:56:59.719 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-11T20:56:59.723 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T20:56:59.749 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:56:59.796 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:56:59.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:59.802 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:56:59.903 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:56:59.928 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:56:59.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:56:59.932 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:57:00.033 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:57:00.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:59 smithi023 bash[21372]: cluster 2024-03-11T20:56:57.978138+0000 osd.4 (osd.4) 131 : cluster 0 4.1a scrub starts 2024-03-11T20:57:00.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:59 smithi023 bash[21372]: cluster 2024-03-11T20:56:57.989984+0000 osd.4 (osd.4) 132 : cluster 0 4.1a scrub ok 2024-03-11T20:57:00.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:59 smithi023 bash[21372]: cluster 2024-03-11T20:56:58.128129+0000 osd.7 (osd.7) 137 : cluster 0 6.12 scrub starts 2024-03-11T20:57:00.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:59 smithi023 bash[21372]: cluster 2024-03-11T20:56:58.131250+0000 osd.7 (osd.7) 138 : cluster 0 6.12 scrub ok 2024-03-11T20:57:00.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:59 smithi023 bash[21372]: cluster 2024-03-11T20:56:58.450287+0000 osd.2 (osd.2) 105 : cluster 0 3.0 scrub starts 2024-03-11T20:57:00.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:59 smithi023 bash[21372]: cluster 2024-03-11T20:56:58.455121+0000 osd.2 (osd.2) 106 : cluster 0 3.0 scrub ok 2024-03-11T20:57:00.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:59 smithi023 bash[21372]: audit 2024-03-11T20:56:58.770146+0000 mon.a (mon.0) 1043 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.40126953695383216}]': finished 2024-03-11T20:57:00.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:59 smithi023 bash[21372]: cluster 2024-03-11T20:56:58.775766+0000 mon.a (mon.0) 1044 : cluster 0 osdmap e159: 8 total, 8 up, 7 in 2024-03-11T20:57:00.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:59 smithi023 bash[21372]: cluster 2024-03-11T20:56:58.828942+0000 osd.0 (osd.0) 143 : cluster 0 4.6 scrub starts 2024-03-11T20:57:00.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:59 smithi023 bash[21372]: cluster 2024-03-11T20:56:58.833044+0000 osd.0 (osd.0) 144 : cluster 0 4.6 scrub ok 2024-03-11T20:57:00.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:59 smithi023 bash[21372]: cluster 2024-03-11T20:56:58.947691+0000 osd.4 (osd.4) 133 : cluster 0 6.e scrub starts 2024-03-11T20:57:00.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:59 smithi023 bash[21372]: cluster 2024-03-11T20:56:58.949802+0000 osd.4 (osd.4) 134 : cluster 0 6.e scrub ok 2024-03-11T20:57:00.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:59 smithi023 bash[21372]: cluster 2024-03-11T20:56:59.081578+0000 osd.7 (osd.7) 139 : cluster 0 6.3 scrub starts 2024-03-11T20:57:00.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:59 smithi023 bash[21372]: cluster 2024-03-11T20:56:59.083889+0000 osd.7 (osd.7) 140 : cluster 0 6.3 scrub ok 2024-03-11T20:57:00.036 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:59 smithi023 bash[21372]: cluster 2024-03-11T20:56:59.640754+0000 osd.6 (osd.6) 91 : cluster 0 purged_snaps scrub starts 2024-03-11T20:57:00.036 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:56:59 smithi023 bash[21372]: cluster 2024-03-11T20:56:59.640874+0000 osd.6 (osd.6) 92 : cluster 0 purged_snaps scrub ok 2024-03-11T20:57:00.036 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:59 smithi023 bash[29146]: cluster 2024-03-11T20:56:57.978138+0000 osd.4 (osd.4) 131 : cluster 0 4.1a scrub starts 2024-03-11T20:57:00.037 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:59 smithi023 bash[29146]: cluster 2024-03-11T20:56:57.989984+0000 osd.4 (osd.4) 132 : cluster 0 4.1a scrub ok 2024-03-11T20:57:00.037 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:59 smithi023 bash[29146]: cluster 2024-03-11T20:56:58.128129+0000 osd.7 (osd.7) 137 : cluster 0 6.12 scrub starts 2024-03-11T20:57:00.037 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:59 smithi023 bash[29146]: cluster 2024-03-11T20:56:58.131250+0000 osd.7 (osd.7) 138 : cluster 0 6.12 scrub ok 2024-03-11T20:57:00.037 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:59 smithi023 bash[29146]: cluster 2024-03-11T20:56:58.450287+0000 osd.2 (osd.2) 105 : cluster 0 3.0 scrub starts 2024-03-11T20:57:00.037 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:59 smithi023 bash[29146]: cluster 2024-03-11T20:56:58.455121+0000 osd.2 (osd.2) 106 : cluster 0 3.0 scrub ok 2024-03-11T20:57:00.037 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:59 smithi023 bash[29146]: audit 2024-03-11T20:56:58.770146+0000 mon.a (mon.0) 1043 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.40126953695383216}]': finished 2024-03-11T20:57:00.037 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:59 smithi023 bash[29146]: cluster 2024-03-11T20:56:58.775766+0000 mon.a (mon.0) 1044 : cluster 0 osdmap e159: 8 total, 8 up, 7 in 2024-03-11T20:57:00.037 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:59 smithi023 bash[29146]: cluster 2024-03-11T20:56:58.828942+0000 osd.0 (osd.0) 143 : cluster 0 4.6 scrub starts 2024-03-11T20:57:00.037 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:59 smithi023 bash[29146]: cluster 2024-03-11T20:56:58.833044+0000 osd.0 (osd.0) 144 : cluster 0 4.6 scrub ok 2024-03-11T20:57:00.037 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:59 smithi023 bash[29146]: cluster 2024-03-11T20:56:58.947691+0000 osd.4 (osd.4) 133 : cluster 0 6.e scrub starts 2024-03-11T20:57:00.037 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:59 smithi023 bash[29146]: cluster 2024-03-11T20:56:58.949802+0000 osd.4 (osd.4) 134 : cluster 0 6.e scrub ok 2024-03-11T20:57:00.037 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:59 smithi023 bash[29146]: cluster 2024-03-11T20:56:59.081578+0000 osd.7 (osd.7) 139 : cluster 0 6.3 scrub starts 2024-03-11T20:57:00.037 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:59 smithi023 bash[29146]: cluster 2024-03-11T20:56:59.083889+0000 osd.7 (osd.7) 140 : cluster 0 6.3 scrub ok 2024-03-11T20:57:00.038 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:59 smithi023 bash[29146]: cluster 2024-03-11T20:56:59.640754+0000 osd.6 (osd.6) 91 : cluster 0 purged_snaps scrub starts 2024-03-11T20:57:00.038 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:56:59 smithi023 bash[29146]: cluster 2024-03-11T20:56:59.640874+0000 osd.6 (osd.6) 92 : cluster 0 purged_snaps scrub ok 2024-03-11T20:57:00.065 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:57:00.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:00.077 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:57:00.177 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:57:00.205 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:57:00.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:00.207 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:57:00.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:59 smithi146 bash[23809]: cluster 2024-03-11T20:56:57.978138+0000 osd.4 (osd.4) 131 : cluster 0 4.1a scrub starts 2024-03-11T20:57:00.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:59 smithi146 bash[23809]: cluster 2024-03-11T20:56:57.989984+0000 osd.4 (osd.4) 132 : cluster 0 4.1a scrub ok 2024-03-11T20:57:00.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:59 smithi146 bash[23809]: cluster 2024-03-11T20:56:58.128129+0000 osd.7 (osd.7) 137 : cluster 0 6.12 scrub starts 2024-03-11T20:57:00.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:59 smithi146 bash[23809]: cluster 2024-03-11T20:56:58.131250+0000 osd.7 (osd.7) 138 : cluster 0 6.12 scrub ok 2024-03-11T20:57:00.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:59 smithi146 bash[23809]: cluster 2024-03-11T20:56:58.450287+0000 osd.2 (osd.2) 105 : cluster 0 3.0 scrub starts 2024-03-11T20:57:00.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:59 smithi146 bash[23809]: cluster 2024-03-11T20:56:58.455121+0000 osd.2 (osd.2) 106 : cluster 0 3.0 scrub ok 2024-03-11T20:57:00.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:59 smithi146 bash[23809]: audit 2024-03-11T20:56:58.770146+0000 mon.a (mon.0) 1043 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.40126953695383216}]': finished 2024-03-11T20:57:00.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:59 smithi146 bash[23809]: cluster 2024-03-11T20:56:58.775766+0000 mon.a (mon.0) 1044 : cluster 0 osdmap e159: 8 total, 8 up, 7 in 2024-03-11T20:57:00.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:59 smithi146 bash[23809]: cluster 2024-03-11T20:56:58.828942+0000 osd.0 (osd.0) 143 : cluster 0 4.6 scrub starts 2024-03-11T20:57:00.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:59 smithi146 bash[23809]: cluster 2024-03-11T20:56:58.833044+0000 osd.0 (osd.0) 144 : cluster 0 4.6 scrub ok 2024-03-11T20:57:00.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:59 smithi146 bash[23809]: cluster 2024-03-11T20:56:58.947691+0000 osd.4 (osd.4) 133 : cluster 0 6.e scrub starts 2024-03-11T20:57:00.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:59 smithi146 bash[23809]: cluster 2024-03-11T20:56:58.949802+0000 osd.4 (osd.4) 134 : cluster 0 6.e scrub ok 2024-03-11T20:57:00.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:59 smithi146 bash[23809]: cluster 2024-03-11T20:56:59.081578+0000 osd.7 (osd.7) 139 : cluster 0 6.3 scrub starts 2024-03-11T20:57:00.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:59 smithi146 bash[23809]: cluster 2024-03-11T20:56:59.083889+0000 osd.7 (osd.7) 140 : cluster 0 6.3 scrub ok 2024-03-11T20:57:00.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:59 smithi146 bash[23809]: cluster 2024-03-11T20:56:59.640754+0000 osd.6 (osd.6) 91 : cluster 0 purged_snaps scrub starts 2024-03-11T20:57:00.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:56:59 smithi146 bash[23809]: cluster 2024-03-11T20:56:59.640874+0000 osd.6 (osd.6) 92 : cluster 0 purged_snaps scrub ok 2024-03-11T20:57:00.308 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:57:00.331 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:57:00.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:00.333 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:57:00.434 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:00.463 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:00.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:00.467 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:57:00.568 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:57:00.596 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:57:00.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:00.598 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:57:00.699 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:57:00.725 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:57:00.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:00.730 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:57:00.831 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:57:00.852 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:57:00.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:00.857 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:57:00.958 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:00.989 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:00.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:00.993 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:57:01.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:00 smithi146 bash[23809]: cluster 2024-03-11T20:56:59.108772+0000 osd.4 (osd.4) 135 : cluster 0 purged_snaps scrub starts 2024-03-11T20:57:01.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:00 smithi146 bash[23809]: cluster 2024-03-11T20:56:59.108880+0000 osd.4 (osd.4) 136 : cluster 0 purged_snaps scrub ok 2024-03-11T20:57:01.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:00 smithi146 bash[23809]: cluster 2024-03-11T20:56:59.782172+0000 mon.a (mon.0) 1045 : cluster 0 osdmap e160: 8 total, 8 up, 7 in 2024-03-11T20:57:01.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:00 smithi146 bash[23809]: cluster 2024-03-11T20:56:59.814027+0000 osd.0 (osd.0) 145 : cluster 0 7.4 scrub starts 2024-03-11T20:57:01.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:00 smithi146 bash[23809]: cluster 2024-03-11T20:56:59.948010+0000 mgr.y (mgr.24380) 234 : cluster 0 pgmap v225: 148 pgs: 148 active+clean; 4.7 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 659 B/s rd, 0 B/s wr, 1.10k op/s 2024-03-11T20:57:01.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:00 smithi146 bash[23809]: cluster 2024-03-11T20:56:59.959131+0000 osd.6 (osd.6) 93 : cluster 0 2.17 scrub starts 2024-03-11T20:57:01.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:00 smithi146 bash[23809]: cluster 2024-03-11T20:56:59.964885+0000 osd.6 (osd.6) 94 : cluster 0 2.17 scrub ok 2024-03-11T20:57:01.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:00 smithi146 bash[23809]: cluster 2024-03-11T20:57:00.620522+0000 osd.5 (osd.5) 127 : cluster 0 6.7 scrub starts 2024-03-11T20:57:01.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:00 smithi146 bash[23809]: cluster 2024-03-11T20:57:00.632088+0000 osd.5 (osd.5) 128 : cluster 0 6.7 scrub ok 2024-03-11T20:57:01.078 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:00 smithi023 bash[21372]: cluster 2024-03-11T20:56:59.108772+0000 osd.4 (osd.4) 135 : cluster 0 purged_snaps scrub starts 2024-03-11T20:57:01.078 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:00 smithi023 bash[21372]: cluster 2024-03-11T20:56:59.108880+0000 osd.4 (osd.4) 136 : cluster 0 purged_snaps scrub ok 2024-03-11T20:57:01.078 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:00 smithi023 bash[21372]: cluster 2024-03-11T20:56:59.782172+0000 mon.a (mon.0) 1045 : cluster 0 osdmap e160: 8 total, 8 up, 7 in 2024-03-11T20:57:01.078 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:00 smithi023 bash[21372]: cluster 2024-03-11T20:56:59.814027+0000 osd.0 (osd.0) 145 : cluster 0 7.4 scrub starts 2024-03-11T20:57:01.078 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:00 smithi023 bash[21372]: cluster 2024-03-11T20:56:59.948010+0000 mgr.y (mgr.24380) 234 : cluster 0 pgmap v225: 148 pgs: 148 active+clean; 4.7 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 659 B/s rd, 0 B/s wr, 1.10k op/s 2024-03-11T20:57:01.078 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:00 smithi023 bash[21372]: cluster 2024-03-11T20:56:59.959131+0000 osd.6 (osd.6) 93 : cluster 0 2.17 scrub starts 2024-03-11T20:57:01.078 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:00 smithi023 bash[21372]: cluster 2024-03-11T20:56:59.964885+0000 osd.6 (osd.6) 94 : cluster 0 2.17 scrub ok 2024-03-11T20:57:01.078 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:00 smithi023 bash[21372]: cluster 2024-03-11T20:57:00.620522+0000 osd.5 (osd.5) 127 : cluster 0 6.7 scrub starts 2024-03-11T20:57:01.078 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:00 smithi023 bash[21372]: cluster 2024-03-11T20:57:00.632088+0000 osd.5 (osd.5) 128 : cluster 0 6.7 scrub ok 2024-03-11T20:57:01.078 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:00 smithi023 bash[29146]: cluster 2024-03-11T20:56:59.108772+0000 osd.4 (osd.4) 135 : cluster 0 purged_snaps scrub starts 2024-03-11T20:57:01.079 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:00 smithi023 bash[29146]: cluster 2024-03-11T20:56:59.108880+0000 osd.4 (osd.4) 136 : cluster 0 purged_snaps scrub ok 2024-03-11T20:57:01.079 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:00 smithi023 bash[29146]: cluster 2024-03-11T20:56:59.782172+0000 mon.a (mon.0) 1045 : cluster 0 osdmap e160: 8 total, 8 up, 7 in 2024-03-11T20:57:01.079 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:00 smithi023 bash[29146]: cluster 2024-03-11T20:56:59.814027+0000 osd.0 (osd.0) 145 : cluster 0 7.4 scrub starts 2024-03-11T20:57:01.079 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:00 smithi023 bash[29146]: cluster 2024-03-11T20:56:59.948010+0000 mgr.y (mgr.24380) 234 : cluster 0 pgmap v225: 148 pgs: 148 active+clean; 4.7 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 659 B/s rd, 0 B/s wr, 1.10k op/s 2024-03-11T20:57:01.079 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:00 smithi023 bash[29146]: cluster 2024-03-11T20:56:59.959131+0000 osd.6 (osd.6) 93 : cluster 0 2.17 scrub starts 2024-03-11T20:57:01.079 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:00 smithi023 bash[29146]: cluster 2024-03-11T20:56:59.964885+0000 osd.6 (osd.6) 94 : cluster 0 2.17 scrub ok 2024-03-11T20:57:01.079 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:00 smithi023 bash[29146]: cluster 2024-03-11T20:57:00.620522+0000 osd.5 (osd.5) 127 : cluster 0 6.7 scrub starts 2024-03-11T20:57:01.079 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:00 smithi023 bash[29146]: cluster 2024-03-11T20:57:00.632088+0000 osd.5 (osd.5) 128 : cluster 0 6.7 scrub ok 2024-03-11T20:57:01.095 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:57:01.128 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:57:01.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:01.131 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:57:01.232 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:57:01.264 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:57:01.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:01.266 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:57:01.366 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:57:01.404 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:57:01.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:01.407 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:57:01.508 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:57:01.534 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:57:01.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:01.537 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:57:01.637 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:01.663 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:01.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:01.665 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:57:01.766 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:01.783 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:01.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:01.785 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:57:01.886 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:01.897 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:01 smithi146 bash[23809]: cluster 2024-03-11T20:56:59.976953+0000 osd.4 (osd.4) 137 : cluster 0 4.c scrub starts 2024-03-11T20:57:01.897 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:01 smithi146 bash[23809]: cluster 2024-03-11T20:56:59.978043+0000 osd.4 (osd.4) 138 : cluster 0 4.c scrub ok 2024-03-11T20:57:01.897 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:01 smithi146 bash[23809]: cluster 2024-03-11T20:57:01.117381+0000 osd.7 (osd.7) 141 : cluster 0 7.6 scrub starts 2024-03-11T20:57:01.897 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:01 smithi146 bash[23809]: cluster 2024-03-11T20:57:01.308233+0000 osd.0 (osd.0) 146 : cluster 0 7.4 scrub ok 2024-03-11T20:57:01.910 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:01.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:01.911 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:57:02.012 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:02.048 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:02.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:02.055 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:57:02.155 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:02.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:02.189 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:02.189 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:57:02.290 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:57:02.321 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:57:02.324 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:02 smithi023 bash[21372]: cluster 2024-03-11T20:56:59.976953+0000 osd.4 (osd.4) 137 : cluster 0 4.c scrub starts 2024-03-11T20:57:02.325 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:02 smithi023 bash[21372]: cluster 2024-03-11T20:56:59.978043+0000 osd.4 (osd.4) 138 : cluster 0 4.c scrub ok 2024-03-11T20:57:02.325 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:02 smithi023 bash[21372]: cluster 2024-03-11T20:57:01.117381+0000 osd.7 (osd.7) 141 : cluster 0 7.6 scrub starts 2024-03-11T20:57:02.325 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:02 smithi023 bash[21372]: cluster 2024-03-11T20:57:01.308233+0000 osd.0 (osd.0) 146 : cluster 0 7.4 scrub ok 2024-03-11T20:57:02.325 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:02 smithi023 bash[29146]: cluster 2024-03-11T20:56:59.976953+0000 osd.4 (osd.4) 137 : cluster 0 4.c scrub starts 2024-03-11T20:57:02.325 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:02 smithi023 bash[29146]: cluster 2024-03-11T20:56:59.978043+0000 osd.4 (osd.4) 138 : cluster 0 4.c scrub ok 2024-03-11T20:57:02.325 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:02 smithi023 bash[29146]: cluster 2024-03-11T20:57:01.117381+0000 osd.7 (osd.7) 141 : cluster 0 7.6 scrub starts 2024-03-11T20:57:02.325 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:02 smithi023 bash[29146]: cluster 2024-03-11T20:57:01.308233+0000 osd.0 (osd.0) 146 : cluster 0 7.4 scrub ok 2024-03-11T20:57:02.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:02.326 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:57:02.427 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:02.451 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:02.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:02.454 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:57:02.555 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:02.591 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:02.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:02.593 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:57:02.694 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:57:02.729 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:57:02.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:02.732 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:57:02.833 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:57:02.845 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:02 smithi023 bash[21372]: cluster 2024-03-11T20:57:01.487617+0000 osd.2 (osd.2) 107 : cluster 0 6.f scrub starts 2024-03-11T20:57:02.845 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:02 smithi023 bash[21372]: cluster 2024-03-11T20:57:01.498602+0000 osd.2 (osd.2) 108 : cluster 0 6.f scrub ok 2024-03-11T20:57:02.845 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:02 smithi023 bash[21372]: cluster 2024-03-11T20:57:01.883167+0000 osd.0 (osd.0) 147 : cluster 0 6.10 scrub starts 2024-03-11T20:57:02.846 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:02 smithi023 bash[21372]: cluster 2024-03-11T20:57:01.885396+0000 osd.0 (osd.0) 148 : cluster 0 6.10 scrub ok 2024-03-11T20:57:02.846 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:02 smithi023 bash[21372]: cluster 2024-03-11T20:57:01.948975+0000 mgr.y (mgr.24380) 235 : cluster 0 pgmap v226: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 4.6 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.07k op/s 2024-03-11T20:57:02.846 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:02 smithi023 bash[21372]: cluster 2024-03-11T20:57:02.097454+0000 osd.7 (osd.7) 142 : cluster 0 6.18 scrub starts 2024-03-11T20:57:02.846 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:02 smithi023 bash[21372]: cluster 2024-03-11T20:57:02.099137+0000 osd.7 (osd.7) 143 : cluster 0 6.18 scrub ok 2024-03-11T20:57:02.846 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:02 smithi023 bash[21372]: cluster 2024-03-11T20:57:02.408617+0000 osd.3 (osd.3) 163 : cluster 0 2.19 scrub starts 2024-03-11T20:57:02.846 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:02 smithi023 bash[21372]: cluster 2024-03-11T20:57:02.415589+0000 osd.3 (osd.3) 164 : cluster 0 2.19 scrub ok 2024-03-11T20:57:02.846 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:02 smithi023 bash[29146]: cluster 2024-03-11T20:57:01.487617+0000 osd.2 (osd.2) 107 : cluster 0 6.f scrub starts 2024-03-11T20:57:02.846 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:02 smithi023 bash[29146]: cluster 2024-03-11T20:57:01.498602+0000 osd.2 (osd.2) 108 : cluster 0 6.f scrub ok 2024-03-11T20:57:02.846 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:02 smithi023 bash[29146]: cluster 2024-03-11T20:57:01.883167+0000 osd.0 (osd.0) 147 : cluster 0 6.10 scrub starts 2024-03-11T20:57:02.846 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:02 smithi023 bash[29146]: cluster 2024-03-11T20:57:01.885396+0000 osd.0 (osd.0) 148 : cluster 0 6.10 scrub ok 2024-03-11T20:57:02.846 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:02 smithi023 bash[29146]: cluster 2024-03-11T20:57:01.948975+0000 mgr.y (mgr.24380) 235 : cluster 0 pgmap v226: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 4.6 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.07k op/s 2024-03-11T20:57:02.847 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:02 smithi023 bash[29146]: cluster 2024-03-11T20:57:02.097454+0000 osd.7 (osd.7) 142 : cluster 0 6.18 scrub starts 2024-03-11T20:57:02.847 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:02 smithi023 bash[29146]: cluster 2024-03-11T20:57:02.099137+0000 osd.7 (osd.7) 143 : cluster 0 6.18 scrub ok 2024-03-11T20:57:02.847 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:02 smithi023 bash[29146]: cluster 2024-03-11T20:57:02.408617+0000 osd.3 (osd.3) 163 : cluster 0 2.19 scrub starts 2024-03-11T20:57:02.847 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:02 smithi023 bash[29146]: cluster 2024-03-11T20:57:02.415589+0000 osd.3 (osd.3) 164 : cluster 0 2.19 scrub ok 2024-03-11T20:57:02.867 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:57:02.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:02.871 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:57:02.972 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:03.016 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:03.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:03.021 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:57:03.122 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:57:03.150 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:57:03.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:03.152 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:57:03.253 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:03.264 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:02 smithi146 bash[23809]: cluster 2024-03-11T20:57:01.487617+0000 osd.2 (osd.2) 107 : cluster 0 6.f scrub starts 2024-03-11T20:57:03.264 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:02 smithi146 bash[23809]: cluster 2024-03-11T20:57:01.498602+0000 osd.2 (osd.2) 108 : cluster 0 6.f scrub ok 2024-03-11T20:57:03.264 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:02 smithi146 bash[23809]: cluster 2024-03-11T20:57:01.883167+0000 osd.0 (osd.0) 147 : cluster 0 6.10 scrub starts 2024-03-11T20:57:03.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:02 smithi146 bash[23809]: cluster 2024-03-11T20:57:01.885396+0000 osd.0 (osd.0) 148 : cluster 0 6.10 scrub ok 2024-03-11T20:57:03.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:02 smithi146 bash[23809]: cluster 2024-03-11T20:57:01.948975+0000 mgr.y (mgr.24380) 235 : cluster 0 pgmap v226: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 4.6 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.07k op/s 2024-03-11T20:57:03.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:02 smithi146 bash[23809]: cluster 2024-03-11T20:57:02.097454+0000 osd.7 (osd.7) 142 : cluster 0 6.18 scrub starts 2024-03-11T20:57:03.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:02 smithi146 bash[23809]: cluster 2024-03-11T20:57:02.099137+0000 osd.7 (osd.7) 143 : cluster 0 6.18 scrub ok 2024-03-11T20:57:03.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:02 smithi146 bash[23809]: cluster 2024-03-11T20:57:02.408617+0000 osd.3 (osd.3) 163 : cluster 0 2.19 scrub starts 2024-03-11T20:57:03.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:02 smithi146 bash[23809]: cluster 2024-03-11T20:57:02.415589+0000 osd.3 (osd.3) 164 : cluster 0 2.19 scrub ok 2024-03-11T20:57:03.282 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:03.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:03.287 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:57:03.388 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:57:03.416 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:57:03.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:03.419 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:57:03.520 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:03.548 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:03.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:03.552 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:57:03.652 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:03.686 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:03.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:03.689 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:57:03.790 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:57:03.818 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:57:03.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:03.819 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:57:03.920 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:57:03.930 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:03 smithi023 bash[21372]: cluster 2024-03-11T20:57:02.009143+0000 osd.4 (osd.4) 139 : cluster 0 4.e scrub starts 2024-03-11T20:57:03.930 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:03 smithi023 bash[21372]: cluster 2024-03-11T20:57:02.018709+0000 osd.4 (osd.4) 140 : cluster 0 4.e scrub ok 2024-03-11T20:57:03.930 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:03 smithi023 bash[21372]: cluster 2024-03-11T20:57:02.680010+0000 osd.7 (osd.7) 144 : cluster 0 7.6 scrub ok 2024-03-11T20:57:03.930 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:03 smithi023 bash[29146]: cluster 2024-03-11T20:57:02.009143+0000 osd.4 (osd.4) 139 : cluster 0 4.e scrub starts 2024-03-11T20:57:03.930 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:03 smithi023 bash[29146]: cluster 2024-03-11T20:57:02.018709+0000 osd.4 (osd.4) 140 : cluster 0 4.e scrub ok 2024-03-11T20:57:03.930 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:03 smithi023 bash[29146]: cluster 2024-03-11T20:57:02.680010+0000 osd.7 (osd.7) 144 : cluster 0 7.6 scrub ok 2024-03-11T20:57:03.947 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:57:03.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:03.952 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:57:04.052 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:57:04.101 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:57:04.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:04.103 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:57:04.204 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:57:04.237 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:57:04.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:04.241 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:57:04.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:03 smithi146 bash[23809]: cluster 2024-03-11T20:57:02.009143+0000 osd.4 (osd.4) 139 : cluster 0 4.e scrub starts 2024-03-11T20:57:04.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:03 smithi146 bash[23809]: cluster 2024-03-11T20:57:02.018709+0000 osd.4 (osd.4) 140 : cluster 0 4.e scrub ok 2024-03-11T20:57:04.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:03 smithi146 bash[23809]: cluster 2024-03-11T20:57:02.680010+0000 osd.7 (osd.7) 144 : cluster 0 7.6 scrub ok 2024-03-11T20:57:04.274 INFO:teuthology.orchestra.run.smithi023.stdout:ERROR: (22) Invalid argument 2024-03-11T20:57:04.274 INFO:teuthology.orchestra.run.smithi023.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T20:57:04.342 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:57:04.367 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:57:04.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:04.378 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:57:04.479 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:57:04.503 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:57:04.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:04.509 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:57:04.608 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:57:04.608 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:57:04.610 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:57:04.651 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:57:04.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:04.656 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:57:04.757 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:57:04.780 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:57:04.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:04.782 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:57:04.883 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:57:04.905 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:57:04.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:04.907 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:57:05.008 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:57:05.037 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:57:05.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:05.039 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:57:05.106 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:04 smithi146 bash[23809]: cluster 2024-03-11T20:57:03.949434+0000 mgr.y (mgr.24380) 236 : cluster 0 pgmap v227: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 4.6 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 383 B/s rd, 0 B/s wr, 487 op/s 2024-03-11T20:57:05.106 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:04 smithi146 bash[23809]: cluster 2024-03-11T20:57:04.413302+0000 osd.3 (osd.3) 165 : cluster 0 6.b scrub starts 2024-03-11T20:57:05.106 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:04 smithi146 bash[23809]: cluster 2024-03-11T20:57:04.417568+0000 osd.3 (osd.3) 166 : cluster 0 6.b scrub ok 2024-03-11T20:57:05.106 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:04 smithi146 bash[23809]: audit 2024-03-11T20:57:04.499766+0000 mon.a (mon.0) 1046 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:57:05.106 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:04 smithi146 bash[23809]: cluster 2024-03-11T20:57:04.657715+0000 osd.5 (osd.5) 129 : cluster 0 4.8 scrub starts 2024-03-11T20:57:05.106 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:04 smithi146 bash[23809]: cluster 2024-03-11T20:57:04.658873+0000 osd.5 (osd.5) 130 : cluster 0 4.8 scrub ok 2024-03-11T20:57:05.140 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:57:05.161 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:57:05.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:05.163 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:57:05.264 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:05.277 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:05 smithi023 bash[21372]: cluster 2024-03-11T20:57:03.949434+0000 mgr.y (mgr.24380) 236 : cluster 0 pgmap v227: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 4.6 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 383 B/s rd, 0 B/s wr, 487 op/s 2024-03-11T20:57:05.277 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:05 smithi023 bash[21372]: cluster 2024-03-11T20:57:04.413302+0000 osd.3 (osd.3) 165 : cluster 0 6.b scrub starts 2024-03-11T20:57:05.277 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:05 smithi023 bash[21372]: cluster 2024-03-11T20:57:04.417568+0000 osd.3 (osd.3) 166 : cluster 0 6.b scrub ok 2024-03-11T20:57:05.277 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:05 smithi023 bash[21372]: audit 2024-03-11T20:57:04.499766+0000 mon.a (mon.0) 1046 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:57:05.277 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:05 smithi023 bash[21372]: cluster 2024-03-11T20:57:04.657715+0000 osd.5 (osd.5) 129 : cluster 0 4.8 scrub starts 2024-03-11T20:57:05.277 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:05 smithi023 bash[21372]: cluster 2024-03-11T20:57:04.658873+0000 osd.5 (osd.5) 130 : cluster 0 4.8 scrub ok 2024-03-11T20:57:05.278 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:05 smithi023 bash[29146]: cluster 2024-03-11T20:57:03.949434+0000 mgr.y (mgr.24380) 236 : cluster 0 pgmap v227: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 4.6 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 383 B/s rd, 0 B/s wr, 487 op/s 2024-03-11T20:57:05.278 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:05 smithi023 bash[29146]: cluster 2024-03-11T20:57:04.413302+0000 osd.3 (osd.3) 165 : cluster 0 6.b scrub starts 2024-03-11T20:57:05.278 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:05 smithi023 bash[29146]: cluster 2024-03-11T20:57:04.417568+0000 osd.3 (osd.3) 166 : cluster 0 6.b scrub ok 2024-03-11T20:57:05.278 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:05 smithi023 bash[29146]: audit 2024-03-11T20:57:04.499766+0000 mon.a (mon.0) 1046 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:57:05.278 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:05 smithi023 bash[29146]: cluster 2024-03-11T20:57:04.657715+0000 osd.5 (osd.5) 129 : cluster 0 4.8 scrub starts 2024-03-11T20:57:05.278 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:05 smithi023 bash[29146]: cluster 2024-03-11T20:57:04.658873+0000 osd.5 (osd.5) 130 : cluster 0 4.8 scrub ok 2024-03-11T20:57:05.295 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:05.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:05.299 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:57:05.400 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:57:05.424 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:57:05.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:05.430 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:57:05.510 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 20:57:05 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T20:57:05.530 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:57:05.554 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:57:05.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:05.556 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:57:05.657 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:57:05.680 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:57:05.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:05.686 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:57:05.787 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:57:05.793 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 6, 7] out_osds: [1] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-11T20:57:05.793 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-11T20:57:05.793 INFO:tasks.thrashosds.thrasher:Adding osd 1 2024-03-11T20:57:05.793 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd in 1 2024-03-11T20:57:05.810 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:57:05.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:05.814 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:57:05.915 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:05.924 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:05 smithi146 bash[23809]: cluster 2024-03-11T20:57:03.998136+0000 osd.4 (osd.4) 141 : cluster 0 4.9 scrub starts 2024-03-11T20:57:05.925 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:05 smithi146 bash[23809]: cluster 2024-03-11T20:57:04.001432+0000 osd.4 (osd.4) 142 : cluster 0 4.9 scrub ok 2024-03-11T20:57:05.925 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:05 smithi146 bash[23809]: cluster 2024-03-11T20:57:04.852322+0000 osd.0 (osd.0) 149 : cluster 0 3.1f scrub starts 2024-03-11T20:57:05.925 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:05 smithi146 bash[23809]: cluster 2024-03-11T20:57:04.854684+0000 osd.0 (osd.0) 150 : cluster 0 3.1f scrub ok 2024-03-11T20:57:05.925 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:05 smithi146 bash[23809]: audit 2024-03-11T20:57:05.108956+0000 mgr.y (mgr.24380) 237 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:57:05.925 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:05 smithi146 bash[23809]: cluster 2024-03-11T20:57:05.121322+0000 osd.7 (osd.7) 145 : cluster 0 2.9 scrub starts 2024-03-11T20:57:05.943 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:05.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:05.946 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:57:06.047 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:06.073 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:06.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:06.075 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:57:06.176 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:06.206 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:06.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:06.208 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:57:06.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:05 smithi023 bash[21372]: cluster 2024-03-11T20:57:03.998136+0000 osd.4 (osd.4) 141 : cluster 0 4.9 scrub starts 2024-03-11T20:57:06.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:05 smithi023 bash[21372]: cluster 2024-03-11T20:57:04.001432+0000 osd.4 (osd.4) 142 : cluster 0 4.9 scrub ok 2024-03-11T20:57:06.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:05 smithi023 bash[21372]: cluster 2024-03-11T20:57:04.852322+0000 osd.0 (osd.0) 149 : cluster 0 3.1f scrub starts 2024-03-11T20:57:06.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:05 smithi023 bash[21372]: cluster 2024-03-11T20:57:04.854684+0000 osd.0 (osd.0) 150 : cluster 0 3.1f scrub ok 2024-03-11T20:57:06.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:05 smithi023 bash[21372]: audit 2024-03-11T20:57:05.108956+0000 mgr.y (mgr.24380) 237 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:57:06.286 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:05 smithi023 bash[21372]: cluster 2024-03-11T20:57:05.121322+0000 osd.7 (osd.7) 145 : cluster 0 2.9 scrub starts 2024-03-11T20:57:06.286 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:05 smithi023 bash[29146]: cluster 2024-03-11T20:57:03.998136+0000 osd.4 (osd.4) 141 : cluster 0 4.9 scrub starts 2024-03-11T20:57:06.286 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:05 smithi023 bash[29146]: cluster 2024-03-11T20:57:04.001432+0000 osd.4 (osd.4) 142 : cluster 0 4.9 scrub ok 2024-03-11T20:57:06.286 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:05 smithi023 bash[29146]: cluster 2024-03-11T20:57:04.852322+0000 osd.0 (osd.0) 149 : cluster 0 3.1f scrub starts 2024-03-11T20:57:06.286 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:05 smithi023 bash[29146]: cluster 2024-03-11T20:57:04.854684+0000 osd.0 (osd.0) 150 : cluster 0 3.1f scrub ok 2024-03-11T20:57:06.286 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:05 smithi023 bash[29146]: audit 2024-03-11T20:57:05.108956+0000 mgr.y (mgr.24380) 237 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:57:06.286 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:05 smithi023 bash[29146]: cluster 2024-03-11T20:57:05.121322+0000 osd.7 (osd.7) 145 : cluster 0 2.9 scrub starts 2024-03-11T20:57:06.310 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:06.358 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:06.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:06.360 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:57:06.461 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:57:06.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:06.492 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:57:06.492 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:57:06.593 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:06.632 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:06.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:06.634 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:57:06.735 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:57:06.760 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:57:06.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:06.761 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:57:06.810 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-11T20:57:06.862 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:57:06.872 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:06 smithi023 bash[21372]: cluster 2024-03-11T20:57:05.126823+0000 osd.7 (osd.7) 146 : cluster 0 2.9 scrub ok 2024-03-11T20:57:06.872 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:06 smithi023 bash[21372]: cluster 2024-03-11T20:57:05.951702+0000 mgr.y (mgr.24380) 238 : cluster 0 pgmap v228: 148 pgs: 148 active+clean; 4.2 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.14k op/s 2024-03-11T20:57:06.872 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:06 smithi023 bash[21372]: cluster 2024-03-11T20:57:06.587206+0000 osd.5 (osd.5) 131 : cluster 0 4.7 scrub starts 2024-03-11T20:57:06.872 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:06 smithi023 bash[21372]: cluster 2024-03-11T20:57:06.588161+0000 osd.5 (osd.5) 132 : cluster 0 4.7 scrub ok 2024-03-11T20:57:06.872 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:06 smithi023 bash[29146]: cluster 2024-03-11T20:57:05.126823+0000 osd.7 (osd.7) 146 : cluster 0 2.9 scrub ok 2024-03-11T20:57:06.872 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:06 smithi023 bash[29146]: cluster 2024-03-11T20:57:05.951702+0000 mgr.y (mgr.24380) 238 : cluster 0 pgmap v228: 148 pgs: 148 active+clean; 4.2 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.14k op/s 2024-03-11T20:57:06.872 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:06 smithi023 bash[29146]: cluster 2024-03-11T20:57:06.587206+0000 osd.5 (osd.5) 131 : cluster 0 4.7 scrub starts 2024-03-11T20:57:06.872 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:06 smithi023 bash[29146]: cluster 2024-03-11T20:57:06.588161+0000 osd.5 (osd.5) 132 : cluster 0 4.7 scrub ok 2024-03-11T20:57:06.906 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:57:06.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:06.910 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:57:07.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:06 smithi146 bash[23809]: cluster 2024-03-11T20:57:05.126823+0000 osd.7 (osd.7) 146 : cluster 0 2.9 scrub ok 2024-03-11T20:57:07.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:06 smithi146 bash[23809]: cluster 2024-03-11T20:57:05.951702+0000 mgr.y (mgr.24380) 238 : cluster 0 pgmap v228: 148 pgs: 148 active+clean; 4.2 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.14k op/s 2024-03-11T20:57:07.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:06 smithi146 bash[23809]: cluster 2024-03-11T20:57:06.587206+0000 osd.5 (osd.5) 131 : cluster 0 4.7 scrub starts 2024-03-11T20:57:07.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:06 smithi146 bash[23809]: cluster 2024-03-11T20:57:06.588161+0000 osd.5 (osd.5) 132 : cluster 0 4.7 scrub ok 2024-03-11T20:57:07.011 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:57:07.047 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:57:07.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:07.051 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:57:07.152 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:57:07.183 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:57:07.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:07.185 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:57:07.286 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:57:07.310 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:57:07.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:07.315 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:57:07.416 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:57:07.440 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:57:07.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:07.446 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:57:07.547 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:57:07.578 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:57:07.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:07.584 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:57:07.684 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:07.715 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:07.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:07.717 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:57:07.818 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:07.851 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:07.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:07.858 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:57:07.948 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:07 smithi023 bash[21372]: cluster 2024-03-11T20:57:06.016813+0000 osd.4 (osd.4) 143 : cluster 0 4.18 scrub starts 2024-03-11T20:57:07.948 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:07 smithi023 bash[21372]: cluster 2024-03-11T20:57:06.017711+0000 osd.4 (osd.4) 144 : cluster 0 4.18 scrub ok 2024-03-11T20:57:07.948 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:07 smithi023 bash[21372]: cluster 2024-03-11T20:57:06.491068+0000 osd.3 (osd.3) 167 : cluster 0 4.11 scrub starts 2024-03-11T20:57:07.948 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:07 smithi023 bash[21372]: cluster 2024-03-11T20:57:06.500046+0000 osd.3 (osd.3) 168 : cluster 0 4.11 scrub ok 2024-03-11T20:57:07.948 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:07 smithi023 bash[21372]: cluster 2024-03-11T20:57:06.921030+0000 osd.0 (osd.0) 151 : cluster 0 3.1b scrub starts 2024-03-11T20:57:07.949 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:07 smithi023 bash[21372]: cluster 2024-03-11T20:57:06.924036+0000 osd.0 (osd.0) 152 : cluster 0 3.1b scrub ok 2024-03-11T20:57:07.949 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:07 smithi023 bash[21372]: cluster 2024-03-11T20:57:07.092843+0000 osd.7 (osd.7) 147 : cluster 0 7.5 scrub starts 2024-03-11T20:57:07.949 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:07 smithi023 bash[21372]: cluster 2024-03-11T20:57:07.567563+0000 osd.5 (osd.5) 133 : cluster 0 7.b scrub starts 2024-03-11T20:57:07.949 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:07 smithi023 bash[29146]: cluster 2024-03-11T20:57:06.016813+0000 osd.4 (osd.4) 143 : cluster 0 4.18 scrub starts 2024-03-11T20:57:07.949 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:07 smithi023 bash[29146]: cluster 2024-03-11T20:57:06.017711+0000 osd.4 (osd.4) 144 : cluster 0 4.18 scrub ok 2024-03-11T20:57:07.949 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:07 smithi023 bash[29146]: cluster 2024-03-11T20:57:06.491068+0000 osd.3 (osd.3) 167 : cluster 0 4.11 scrub starts 2024-03-11T20:57:07.949 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:07 smithi023 bash[29146]: cluster 2024-03-11T20:57:06.500046+0000 osd.3 (osd.3) 168 : cluster 0 4.11 scrub ok 2024-03-11T20:57:07.949 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:07 smithi023 bash[29146]: cluster 2024-03-11T20:57:06.921030+0000 osd.0 (osd.0) 151 : cluster 0 3.1b scrub starts 2024-03-11T20:57:07.949 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:07 smithi023 bash[29146]: cluster 2024-03-11T20:57:06.924036+0000 osd.0 (osd.0) 152 : cluster 0 3.1b scrub ok 2024-03-11T20:57:07.949 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:07 smithi023 bash[29146]: cluster 2024-03-11T20:57:07.092843+0000 osd.7 (osd.7) 147 : cluster 0 7.5 scrub starts 2024-03-11T20:57:07.949 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:07 smithi023 bash[29146]: cluster 2024-03-11T20:57:07.567563+0000 osd.5 (osd.5) 133 : cluster 0 7.b scrub starts 2024-03-11T20:57:07.959 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:57:07.980 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:57:07.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:07.986 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:57:08.087 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:57:08.097 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:07 smithi146 bash[23809]: cluster 2024-03-11T20:57:06.016813+0000 osd.4 (osd.4) 143 : cluster 0 4.18 scrub starts 2024-03-11T20:57:08.097 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:07 smithi146 bash[23809]: cluster 2024-03-11T20:57:06.017711+0000 osd.4 (osd.4) 144 : cluster 0 4.18 scrub ok 2024-03-11T20:57:08.097 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:07 smithi146 bash[23809]: cluster 2024-03-11T20:57:06.491068+0000 osd.3 (osd.3) 167 : cluster 0 4.11 scrub starts 2024-03-11T20:57:08.097 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:07 smithi146 bash[23809]: cluster 2024-03-11T20:57:06.500046+0000 osd.3 (osd.3) 168 : cluster 0 4.11 scrub ok 2024-03-11T20:57:08.097 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:07 smithi146 bash[23809]: cluster 2024-03-11T20:57:06.921030+0000 osd.0 (osd.0) 151 : cluster 0 3.1b scrub starts 2024-03-11T20:57:08.097 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:07 smithi146 bash[23809]: cluster 2024-03-11T20:57:06.924036+0000 osd.0 (osd.0) 152 : cluster 0 3.1b scrub ok 2024-03-11T20:57:08.097 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:07 smithi146 bash[23809]: cluster 2024-03-11T20:57:07.092843+0000 osd.7 (osd.7) 147 : cluster 0 7.5 scrub starts 2024-03-11T20:57:08.097 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:07 smithi146 bash[23809]: cluster 2024-03-11T20:57:07.567563+0000 osd.5 (osd.5) 133 : cluster 0 7.b scrub starts 2024-03-11T20:57:08.113 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:57:08.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:08.116 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:57:08.217 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:57:08.241 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:57:08.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:08.243 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:57:08.344 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:57:08.370 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:57:08.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:08.372 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:57:08.410 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:57:08 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:20:57:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T20:57:08.473 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:57:08.498 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:57:08.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:08.501 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:57:08.602 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:57:08.626 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:57:08.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:08.630 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:57:08.731 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:08.758 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:08.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:08.765 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:57:08.865 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:57:08.874 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:08 smithi146 bash[23809]: cluster 2024-03-11T20:57:07.072861+0000 osd.4 (osd.4) 145 : cluster 0 3.3 scrub starts 2024-03-11T20:57:08.874 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:08 smithi146 bash[23809]: cluster 2024-03-11T20:57:07.074746+0000 osd.4 (osd.4) 146 : cluster 0 3.3 scrub ok 2024-03-11T20:57:08.874 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:08 smithi146 bash[23809]: cluster 2024-03-11T20:57:07.952153+0000 mgr.y (mgr.24380) 239 : cluster 0 pgmap v229: 148 pgs: 148 active+clean; 4.2 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 892 B/s rd, 0 B/s wr, 997 op/s 2024-03-11T20:57:08.874 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:08 smithi146 bash[23809]: cluster 2024-03-11T20:57:08.539550+0000 osd.5 (osd.5) 134 : cluster 0 2.16 scrub starts 2024-03-11T20:57:08.874 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:08 smithi146 bash[23809]: cluster 2024-03-11T20:57:08.540891+0000 osd.5 (osd.5) 135 : cluster 0 2.16 scrub ok 2024-03-11T20:57:08.898 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:57:08.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:08.901 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:57:09.002 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:57:09.030 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:57:09.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:09.035 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:57:09.135 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:09.169 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:09.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:09.171 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:57:09.272 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:57:09.280 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:08 smithi023 bash[21372]: cluster 2024-03-11T20:57:07.072861+0000 osd.4 (osd.4) 145 : cluster 0 3.3 scrub starts 2024-03-11T20:57:09.280 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:08 smithi023 bash[21372]: cluster 2024-03-11T20:57:07.074746+0000 osd.4 (osd.4) 146 : cluster 0 3.3 scrub ok 2024-03-11T20:57:09.280 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:08 smithi023 bash[21372]: cluster 2024-03-11T20:57:07.952153+0000 mgr.y (mgr.24380) 239 : cluster 0 pgmap v229: 148 pgs: 148 active+clean; 4.2 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 892 B/s rd, 0 B/s wr, 997 op/s 2024-03-11T20:57:09.280 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:08 smithi023 bash[21372]: cluster 2024-03-11T20:57:08.539550+0000 osd.5 (osd.5) 134 : cluster 0 2.16 scrub starts 2024-03-11T20:57:09.280 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:08 smithi023 bash[21372]: cluster 2024-03-11T20:57:08.540891+0000 osd.5 (osd.5) 135 : cluster 0 2.16 scrub ok 2024-03-11T20:57:09.281 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:08 smithi023 bash[29146]: cluster 2024-03-11T20:57:07.072861+0000 osd.4 (osd.4) 145 : cluster 0 3.3 scrub starts 2024-03-11T20:57:09.281 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:08 smithi023 bash[29146]: cluster 2024-03-11T20:57:07.074746+0000 osd.4 (osd.4) 146 : cluster 0 3.3 scrub ok 2024-03-11T20:57:09.281 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:08 smithi023 bash[29146]: cluster 2024-03-11T20:57:07.952153+0000 mgr.y (mgr.24380) 239 : cluster 0 pgmap v229: 148 pgs: 148 active+clean; 4.2 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 892 B/s rd, 0 B/s wr, 997 op/s 2024-03-11T20:57:09.281 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:08 smithi023 bash[29146]: cluster 2024-03-11T20:57:08.539550+0000 osd.5 (osd.5) 134 : cluster 0 2.16 scrub starts 2024-03-11T20:57:09.281 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:08 smithi023 bash[29146]: cluster 2024-03-11T20:57:08.540891+0000 osd.5 (osd.5) 135 : cluster 0 2.16 scrub ok 2024-03-11T20:57:09.291 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:57:09.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:09.293 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:57:09.394 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:57:09.422 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:57:09.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:09.425 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:57:09.526 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:57:09.544 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:57:09.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:09.546 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:57:09.647 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:57:09.676 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:57:09.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:09.678 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:57:09.779 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:09.807 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:09.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:09.810 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:57:09.911 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:57:09.942 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:57:09.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:09.944 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:57:10.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:09 smithi146 bash[23809]: cluster 2024-03-11T20:57:08.077153+0000 osd.4 (osd.4) 147 : cluster 0 3.1 scrub starts 2024-03-11T20:57:10.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:09 smithi146 bash[23809]: cluster 2024-03-11T20:57:08.093858+0000 osd.4 (osd.4) 148 : cluster 0 3.1 scrub ok 2024-03-11T20:57:10.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:09 smithi146 bash[23809]: cluster 2024-03-11T20:57:08.334580+0000 osd.7 (osd.7) 148 : cluster 0 7.5 scrub ok 2024-03-11T20:57:10.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:09 smithi146 bash[23809]: cluster 2024-03-11T20:57:08.719389+0000 osd.5 (osd.5) 136 : cluster 0 7.b scrub ok 2024-03-11T20:57:10.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:09 smithi146 bash[23809]: cluster 2024-03-11T20:57:09.026297+0000 osd.0 (osd.0) 153 : cluster 0 3.6 scrub starts 2024-03-11T20:57:10.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:09 smithi146 bash[23809]: cluster 2024-03-11T20:57:09.048475+0000 osd.0 (osd.0) 154 : cluster 0 3.6 scrub ok 2024-03-11T20:57:10.045 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:10.070 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:10.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:10.074 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:57:10.174 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:57:10.198 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:57:10.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:10.202 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:57:10.256 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:09 smithi023 bash[29146]: cluster 2024-03-11T20:57:08.077153+0000 osd.4 (osd.4) 147 : cluster 0 3.1 scrub starts 2024-03-11T20:57:10.256 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:09 smithi023 bash[29146]: cluster 2024-03-11T20:57:08.093858+0000 osd.4 (osd.4) 148 : cluster 0 3.1 scrub ok 2024-03-11T20:57:10.256 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:09 smithi023 bash[29146]: cluster 2024-03-11T20:57:08.334580+0000 osd.7 (osd.7) 148 : cluster 0 7.5 scrub ok 2024-03-11T20:57:10.256 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:09 smithi023 bash[29146]: cluster 2024-03-11T20:57:08.719389+0000 osd.5 (osd.5) 136 : cluster 0 7.b scrub ok 2024-03-11T20:57:10.256 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:09 smithi023 bash[29146]: cluster 2024-03-11T20:57:09.026297+0000 osd.0 (osd.0) 153 : cluster 0 3.6 scrub starts 2024-03-11T20:57:10.256 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:09 smithi023 bash[29146]: cluster 2024-03-11T20:57:09.048475+0000 osd.0 (osd.0) 154 : cluster 0 3.6 scrub ok 2024-03-11T20:57:10.257 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:09 smithi023 bash[21372]: cluster 2024-03-11T20:57:08.077153+0000 osd.4 (osd.4) 147 : cluster 0 3.1 scrub starts 2024-03-11T20:57:10.257 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:09 smithi023 bash[21372]: cluster 2024-03-11T20:57:08.093858+0000 osd.4 (osd.4) 148 : cluster 0 3.1 scrub ok 2024-03-11T20:57:10.257 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:09 smithi023 bash[21372]: cluster 2024-03-11T20:57:08.334580+0000 osd.7 (osd.7) 148 : cluster 0 7.5 scrub ok 2024-03-11T20:57:10.257 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:09 smithi023 bash[21372]: cluster 2024-03-11T20:57:08.719389+0000 osd.5 (osd.5) 136 : cluster 0 7.b scrub ok 2024-03-11T20:57:10.257 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:09 smithi023 bash[21372]: cluster 2024-03-11T20:57:09.026297+0000 osd.0 (osd.0) 153 : cluster 0 3.6 scrub starts 2024-03-11T20:57:10.257 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:09 smithi023 bash[21372]: cluster 2024-03-11T20:57:09.048475+0000 osd.0 (osd.0) 154 : cluster 0 3.6 scrub ok 2024-03-11T20:57:10.303 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:57:10.329 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:57:10.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:10.331 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:57:10.432 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:57:10.459 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:57:10.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:10.465 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:57:10.566 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:57:10.589 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:57:10.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:10.594 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:57:10.694 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:57:10.725 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:57:10.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:10.729 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:57:10.830 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:57:10.857 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:57:10.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:10.860 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:57:10.960 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:57:10.994 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:57:10.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:10.997 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:57:11.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:10 smithi146 bash[23809]: cluster 2024-03-11T20:57:09.952928+0000 mgr.y (mgr.24380) 240 : cluster 0 pgmap v230: 148 pgs: 148 active+clean; 4.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 805 B/s rd, 0 B/s wr, 1.20k op/s 2024-03-11T20:57:11.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:10 smithi146 bash[23809]: cluster 2024-03-11T20:57:10.426149+0000 osd.3 (osd.3) 169 : cluster 0 4.17 scrub starts 2024-03-11T20:57:11.099 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:57:11.125 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:57:11.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:11.129 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:57:11.230 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:57:11.253 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:10 smithi023 bash[29146]: cluster 2024-03-11T20:57:09.952928+0000 mgr.y (mgr.24380) 240 : cluster 0 pgmap v230: 148 pgs: 148 active+clean; 4.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 805 B/s rd, 0 B/s wr, 1.20k op/s 2024-03-11T20:57:11.253 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:10 smithi023 bash[29146]: cluster 2024-03-11T20:57:10.426149+0000 osd.3 (osd.3) 169 : cluster 0 4.17 scrub starts 2024-03-11T20:57:11.253 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:10 smithi023 bash[21372]: cluster 2024-03-11T20:57:09.952928+0000 mgr.y (mgr.24380) 240 : cluster 0 pgmap v230: 148 pgs: 148 active+clean; 4.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 805 B/s rd, 0 B/s wr, 1.20k op/s 2024-03-11T20:57:11.253 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:10 smithi023 bash[21372]: cluster 2024-03-11T20:57:10.426149+0000 osd.3 (osd.3) 169 : cluster 0 4.17 scrub starts 2024-03-11T20:57:11.256 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:57:11.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:11.258 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:57:11.359 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:57:11.387 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:57:11.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:11.392 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:57:11.493 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:11.519 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:11.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:11.521 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:57:11.622 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:57:11.657 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:57:11.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:11.659 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:57:11.760 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:11.789 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:11.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:11.793 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:57:11.894 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:11.918 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:11.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:11.921 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:57:12.022 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:57:12.060 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:57:12.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:12.066 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:57:12.166 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:57:12.176 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:11 smithi023 bash[21372]: cluster 2024-03-11T20:57:10.433618+0000 osd.3 (osd.3) 170 : cluster 0 4.17 scrub ok 2024-03-11T20:57:12.176 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:11 smithi023 bash[21372]: cluster 2024-03-11T20:57:10.465746+0000 osd.2 (osd.2) 109 : cluster 0 2.0 scrub starts 2024-03-11T20:57:12.176 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:11 smithi023 bash[21372]: cluster 2024-03-11T20:57:10.475844+0000 osd.2 (osd.2) 110 : cluster 0 2.0 scrub ok 2024-03-11T20:57:12.176 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:11 smithi023 bash[21372]: cluster 2024-03-11T20:57:10.854525+0000 osd.6 (osd.6) 95 : cluster 0 4.5 scrub starts 2024-03-11T20:57:12.177 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:11 smithi023 bash[21372]: cluster 2024-03-11T20:57:10.856453+0000 osd.6 (osd.6) 96 : cluster 0 4.5 scrub ok 2024-03-11T20:57:12.177 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:11 smithi023 bash[29146]: cluster 2024-03-11T20:57:10.433618+0000 osd.3 (osd.3) 170 : cluster 0 4.17 scrub ok 2024-03-11T20:57:12.177 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:11 smithi023 bash[29146]: cluster 2024-03-11T20:57:10.465746+0000 osd.2 (osd.2) 109 : cluster 0 2.0 scrub starts 2024-03-11T20:57:12.177 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:11 smithi023 bash[29146]: cluster 2024-03-11T20:57:10.475844+0000 osd.2 (osd.2) 110 : cluster 0 2.0 scrub ok 2024-03-11T20:57:12.177 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:11 smithi023 bash[29146]: cluster 2024-03-11T20:57:10.854525+0000 osd.6 (osd.6) 95 : cluster 0 4.5 scrub starts 2024-03-11T20:57:12.177 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:11 smithi023 bash[29146]: cluster 2024-03-11T20:57:10.856453+0000 osd.6 (osd.6) 96 : cluster 0 4.5 scrub ok 2024-03-11T20:57:12.223 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:57:12.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:12.228 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:57:12.249 INFO:teuthology.orchestra.run.smithi023.stdout:osd.0: {} 2024-03-11T20:57:12.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:11 smithi146 bash[23809]: cluster 2024-03-11T20:57:10.433618+0000 osd.3 (osd.3) 170 : cluster 0 4.17 scrub ok 2024-03-11T20:57:12.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:11 smithi146 bash[23809]: cluster 2024-03-11T20:57:10.465746+0000 osd.2 (osd.2) 109 : cluster 0 2.0 scrub starts 2024-03-11T20:57:12.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:11 smithi146 bash[23809]: cluster 2024-03-11T20:57:10.475844+0000 osd.2 (osd.2) 110 : cluster 0 2.0 scrub ok 2024-03-11T20:57:12.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:11 smithi146 bash[23809]: cluster 2024-03-11T20:57:10.854525+0000 osd.6 (osd.6) 95 : cluster 0 4.5 scrub starts 2024-03-11T20:57:12.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:11 smithi146 bash[23809]: cluster 2024-03-11T20:57:10.856453+0000 osd.6 (osd.6) 96 : cluster 0 4.5 scrub ok 2024-03-11T20:57:12.306 INFO:teuthology.orchestra.run.smithi023.stdout:osd.1: {} 2024-03-11T20:57:12.330 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:57:12.339 INFO:teuthology.orchestra.run.smithi023.stdout:osd.2: {} 2024-03-11T20:57:12.354 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:57:12.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:12.357 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:57:12.377 INFO:teuthology.orchestra.run.smithi023.stdout:osd.3: {} 2024-03-11T20:57:12.417 INFO:teuthology.orchestra.run.smithi023.stdout:osd.4: {} 2024-03-11T20:57:12.450 INFO:teuthology.orchestra.run.smithi023.stdout:osd.5: {} 2024-03-11T20:57:12.457 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:57:12.482 INFO:teuthology.orchestra.run.smithi023.stdout:osd.6: {} 2024-03-11T20:57:12.485 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:57:12.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:12.490 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:57:12.513 INFO:teuthology.orchestra.run.smithi023.stdout:osd.7: {} 2024-03-11T20:57:12.529 INFO:teuthology.orchestra.run.smithi023.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T20:57:12.530 INFO:teuthology.orchestra.run.smithi023.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T20:57:12.530 INFO:teuthology.orchestra.run.smithi023.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T20:57:12.530 INFO:teuthology.orchestra.run.smithi023.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T20:57:12.530 INFO:teuthology.orchestra.run.smithi023.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T20:57:12.530 INFO:teuthology.orchestra.run.smithi023.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T20:57:12.530 INFO:teuthology.orchestra.run.smithi023.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T20:57:12.530 INFO:teuthology.orchestra.run.smithi023.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T20:57:12.591 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:57:12.619 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:57:12.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:12.623 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:57:12.723 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:12.754 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:12.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:12.755 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:57:12.856 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:57:12.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:12.889 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:57:12.889 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:57:12.890 INFO:teuthology.orchestra.run.smithi023.stderr:nodeep-scrub is unset 2024-03-11T20:57:12.989 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:57:13.016 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:57:13.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:13.021 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:57:13.122 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:13.131 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:12 smithi146 bash[23809]: cluster 2024-03-11T20:57:11.066864+0000 osd.4 (osd.4) 149 : cluster 0 7.9 scrub starts 2024-03-11T20:57:13.131 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:12 smithi146 bash[23809]: cluster 2024-03-11T20:57:11.894627+0000 osd.6 (osd.6) 97 : cluster 0 7.d scrub starts 2024-03-11T20:57:13.131 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:12 smithi146 bash[23809]: cluster 2024-03-11T20:57:11.953555+0000 mgr.y (mgr.24380) 241 : cluster 0 pgmap v231: 148 pgs: 148 active+clean; 3.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.04k op/s 2024-03-11T20:57:13.131 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:12 smithi146 bash[23809]: audit 2024-03-11T20:57:12.206151+0000 mon.a (mon.0) 1047 : audit 0 from='client.? 172.21.15.23:0/3063862653' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T20:57:13.131 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:12 smithi146 bash[23809]: audit 2024-03-11T20:57:12.567164+0000 mon.a (mon.0) 1048 : audit 1 from='client.? 172.21.15.23:0/2483351568' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T20:57:13.146 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:13.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:13.149 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:57:13.249 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:13.259 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:12 smithi023 bash[21372]: cluster 2024-03-11T20:57:11.066864+0000 osd.4 (osd.4) 149 : cluster 0 7.9 scrub starts 2024-03-11T20:57:13.259 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:12 smithi023 bash[21372]: cluster 2024-03-11T20:57:11.894627+0000 osd.6 (osd.6) 97 : cluster 0 7.d scrub starts 2024-03-11T20:57:13.259 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:12 smithi023 bash[21372]: cluster 2024-03-11T20:57:11.953555+0000 mgr.y (mgr.24380) 241 : cluster 0 pgmap v231: 148 pgs: 148 active+clean; 3.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.04k op/s 2024-03-11T20:57:13.259 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:12 smithi023 bash[21372]: audit 2024-03-11T20:57:12.206151+0000 mon.a (mon.0) 1047 : audit 0 from='client.? 172.21.15.23:0/3063862653' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T20:57:13.260 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:12 smithi023 bash[21372]: audit 2024-03-11T20:57:12.567164+0000 mon.a (mon.0) 1048 : audit 1 from='client.? 172.21.15.23:0/2483351568' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T20:57:13.260 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:12 smithi023 bash[29146]: cluster 2024-03-11T20:57:11.066864+0000 osd.4 (osd.4) 149 : cluster 0 7.9 scrub starts 2024-03-11T20:57:13.260 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:12 smithi023 bash[29146]: cluster 2024-03-11T20:57:11.894627+0000 osd.6 (osd.6) 97 : cluster 0 7.d scrub starts 2024-03-11T20:57:13.260 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:12 smithi023 bash[29146]: cluster 2024-03-11T20:57:11.953555+0000 mgr.y (mgr.24380) 241 : cluster 0 pgmap v231: 148 pgs: 148 active+clean; 3.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.04k op/s 2024-03-11T20:57:13.260 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:12 smithi023 bash[29146]: audit 2024-03-11T20:57:12.206151+0000 mon.a (mon.0) 1047 : audit 0 from='client.? 172.21.15.23:0/3063862653' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T20:57:13.260 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:12 smithi023 bash[29146]: audit 2024-03-11T20:57:12.567164+0000 mon.a (mon.0) 1048 : audit 1 from='client.? 172.21.15.23:0/2483351568' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T20:57:13.281 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:13.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:13.283 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:57:13.384 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:57:13.412 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:57:13.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:13.417 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:57:13.518 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:57:13.547 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:57:13.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:13.551 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:57:13.652 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:13.681 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:13.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:13.684 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:57:13.785 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:57:13.815 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:57:13.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:13.817 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:57:13.918 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:57:13.927 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:13 smithi146 bash[23809]: cluster 2024-03-11T20:57:12.061112+0000 osd.4 (osd.4) 150 : cluster 0 7.9 scrub ok 2024-03-11T20:57:13.927 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:13 smithi146 bash[23809]: cluster 2024-03-11T20:57:12.876658+0000 mon.a (mon.0) 1049 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T20:57:13.927 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:13 smithi146 bash[23809]: cluster 2024-03-11T20:57:12.876707+0000 mon.a (mon.0) 1050 : cluster 1 Cluster is now healthy 2024-03-11T20:57:13.928 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:13 smithi146 bash[23809]: audit 2024-03-11T20:57:12.883999+0000 mon.a (mon.0) 1051 : audit 1 from='client.? 172.21.15.23:0/2483351568' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T20:57:13.928 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:13 smithi146 bash[23809]: cluster 2024-03-11T20:57:12.890817+0000 mon.a (mon.0) 1052 : cluster 0 osdmap e161: 8 total, 8 up, 7 in 2024-03-11T20:57:13.928 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:13 smithi146 bash[23809]: cluster 2024-03-11T20:57:12.971280+0000 osd.6 (osd.6) 98 : cluster 0 7.d scrub ok 2024-03-11T20:57:13.928 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:13 smithi146 bash[23809]: cluster 2024-03-11T20:57:13.405567+0000 osd.3 (osd.3) 171 : cluster 0 4.19 scrub starts 2024-03-11T20:57:13.928 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:13 smithi146 bash[23809]: cluster 2024-03-11T20:57:13.411595+0000 osd.3 (osd.3) 172 : cluster 0 4.19 scrub ok 2024-03-11T20:57:13.946 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:57:13.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:13.950 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:57:14.051 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:57:14.076 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:57:14.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:14.082 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:57:14.183 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:14.193 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:13 smithi023 bash[21372]: cluster 2024-03-11T20:57:12.061112+0000 osd.4 (osd.4) 150 : cluster 0 7.9 scrub ok 2024-03-11T20:57:14.193 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:13 smithi023 bash[21372]: cluster 2024-03-11T20:57:12.876658+0000 mon.a (mon.0) 1049 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T20:57:14.193 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:13 smithi023 bash[21372]: cluster 2024-03-11T20:57:12.876707+0000 mon.a (mon.0) 1050 : cluster 1 Cluster is now healthy 2024-03-11T20:57:14.194 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:13 smithi023 bash[21372]: audit 2024-03-11T20:57:12.883999+0000 mon.a (mon.0) 1051 : audit 1 from='client.? 172.21.15.23:0/2483351568' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T20:57:14.194 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:13 smithi023 bash[21372]: cluster 2024-03-11T20:57:12.890817+0000 mon.a (mon.0) 1052 : cluster 0 osdmap e161: 8 total, 8 up, 7 in 2024-03-11T20:57:14.194 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:13 smithi023 bash[21372]: cluster 2024-03-11T20:57:12.971280+0000 osd.6 (osd.6) 98 : cluster 0 7.d scrub ok 2024-03-11T20:57:14.194 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:13 smithi023 bash[21372]: cluster 2024-03-11T20:57:13.405567+0000 osd.3 (osd.3) 171 : cluster 0 4.19 scrub starts 2024-03-11T20:57:14.194 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:13 smithi023 bash[21372]: cluster 2024-03-11T20:57:13.411595+0000 osd.3 (osd.3) 172 : cluster 0 4.19 scrub ok 2024-03-11T20:57:14.194 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:13 smithi023 bash[29146]: cluster 2024-03-11T20:57:12.061112+0000 osd.4 (osd.4) 150 : cluster 0 7.9 scrub ok 2024-03-11T20:57:14.194 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:13 smithi023 bash[29146]: cluster 2024-03-11T20:57:12.876658+0000 mon.a (mon.0) 1049 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T20:57:14.195 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:13 smithi023 bash[29146]: cluster 2024-03-11T20:57:12.876707+0000 mon.a (mon.0) 1050 : cluster 1 Cluster is now healthy 2024-03-11T20:57:14.195 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:13 smithi023 bash[29146]: audit 2024-03-11T20:57:12.883999+0000 mon.a (mon.0) 1051 : audit 1 from='client.? 172.21.15.23:0/2483351568' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T20:57:14.195 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:13 smithi023 bash[29146]: cluster 2024-03-11T20:57:12.890817+0000 mon.a (mon.0) 1052 : cluster 0 osdmap e161: 8 total, 8 up, 7 in 2024-03-11T20:57:14.195 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:13 smithi023 bash[29146]: cluster 2024-03-11T20:57:12.971280+0000 osd.6 (osd.6) 98 : cluster 0 7.d scrub ok 2024-03-11T20:57:14.195 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:13 smithi023 bash[29146]: cluster 2024-03-11T20:57:13.405567+0000 osd.3 (osd.3) 171 : cluster 0 4.19 scrub starts 2024-03-11T20:57:14.195 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:13 smithi023 bash[29146]: cluster 2024-03-11T20:57:13.411595+0000 osd.3 (osd.3) 172 : cluster 0 4.19 scrub ok 2024-03-11T20:57:14.214 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:14.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:14.217 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:57:14.317 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:57:14.347 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:57:14.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:14.351 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:57:14.452 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:57:14.481 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:57:14.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:14.483 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:57:14.584 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:57:14.611 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:57:14.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:14.615 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:57:14.716 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:57:14.746 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:57:14.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:14.749 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:57:14.850 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:57:14.872 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:57:14.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:14.874 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:57:14.974 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:57:15.008 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:57:15.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:15.010 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:57:15.111 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:57:15.114 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:14 smithi146 bash[23809]: cluster 2024-03-11T20:57:13.429110+0000 osd.2 (osd.2) 111 : cluster 0 3.4 scrub starts 2024-03-11T20:57:15.114 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:14 smithi146 bash[23809]: cluster 2024-03-11T20:57:13.436627+0000 osd.2 (osd.2) 112 : cluster 0 3.4 scrub ok 2024-03-11T20:57:15.114 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:14 smithi146 bash[23809]: cluster 2024-03-11T20:57:13.851641+0000 osd.6 (osd.6) 99 : cluster 0 7.c deep-scrub starts 2024-03-11T20:57:15.114 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:14 smithi146 bash[23809]: cluster 2024-03-11T20:57:13.953995+0000 mgr.y (mgr.24380) 242 : cluster 0 pgmap v233: 148 pgs: 148 active+clean; 3.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.07k op/s 2024-03-11T20:57:15.114 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:14 smithi146 bash[23809]: cluster 2024-03-11T20:57:14.482912+0000 osd.5 (osd.5) 137 : cluster 0 7.0 deep-scrub starts 2024-03-11T20:57:15.127 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:14 smithi023 bash[21372]: cluster 2024-03-11T20:57:13.429110+0000 osd.2 (osd.2) 111 : cluster 0 3.4 scrub starts 2024-03-11T20:57:15.127 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:14 smithi023 bash[21372]: cluster 2024-03-11T20:57:13.436627+0000 osd.2 (osd.2) 112 : cluster 0 3.4 scrub ok 2024-03-11T20:57:15.127 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:14 smithi023 bash[21372]: cluster 2024-03-11T20:57:13.851641+0000 osd.6 (osd.6) 99 : cluster 0 7.c deep-scrub starts 2024-03-11T20:57:15.127 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:14 smithi023 bash[21372]: cluster 2024-03-11T20:57:13.953995+0000 mgr.y (mgr.24380) 242 : cluster 0 pgmap v233: 148 pgs: 148 active+clean; 3.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.07k op/s 2024-03-11T20:57:15.127 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:14 smithi023 bash[21372]: cluster 2024-03-11T20:57:14.482912+0000 osd.5 (osd.5) 137 : cluster 0 7.0 deep-scrub starts 2024-03-11T20:57:15.128 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:14 smithi023 bash[29146]: cluster 2024-03-11T20:57:13.429110+0000 osd.2 (osd.2) 111 : cluster 0 3.4 scrub starts 2024-03-11T20:57:15.128 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:14 smithi023 bash[29146]: cluster 2024-03-11T20:57:13.436627+0000 osd.2 (osd.2) 112 : cluster 0 3.4 scrub ok 2024-03-11T20:57:15.128 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:14 smithi023 bash[29146]: cluster 2024-03-11T20:57:13.851641+0000 osd.6 (osd.6) 99 : cluster 0 7.c deep-scrub starts 2024-03-11T20:57:15.128 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:14 smithi023 bash[29146]: cluster 2024-03-11T20:57:13.953995+0000 mgr.y (mgr.24380) 242 : cluster 0 pgmap v233: 148 pgs: 148 active+clean; 3.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.07k op/s 2024-03-11T20:57:15.128 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:14 smithi023 bash[29146]: cluster 2024-03-11T20:57:14.482912+0000 osd.5 (osd.5) 137 : cluster 0 7.0 deep-scrub starts 2024-03-11T20:57:15.144 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:57:15.146 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 20:57:15 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T20:57:15.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:15.147 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:57:15.248 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:57:15.285 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:57:15.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:15.289 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:57:15.390 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:57:15.419 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:57:15.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:15.422 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:57:15.523 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:57:15.555 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:57:15.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:15.556 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:57:15.657 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:57:15.681 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:57:15.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:15.684 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:57:15.785 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:57:15.809 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:57:15.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:15.811 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:57:15.890 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:57:15.890 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:57:15.912 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:57:15.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:15 smithi023 bash[21372]: cluster 2024-03-11T20:57:14.072957+0000 osd.4 (osd.4) 151 : cluster 0 7.1 scrub starts 2024-03-11T20:57:15.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:15 smithi023 bash[21372]: cluster 2024-03-11T20:57:14.935896+0000 osd.4 (osd.4) 152 : cluster 0 7.1 scrub ok 2024-03-11T20:57:15.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:15 smithi023 bash[21372]: audit 2024-03-11T20:57:15.116820+0000 mgr.y (mgr.24380) 243 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:57:15.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:15 smithi023 bash[21372]: cluster 2024-03-11T20:57:15.373577+0000 osd.3 (osd.3) 173 : cluster 0 6.1b scrub starts 2024-03-11T20:57:15.921 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:15 smithi023 bash[21372]: cluster 2024-03-11T20:57:15.380213+0000 osd.3 (osd.3) 174 : cluster 0 6.1b scrub ok 2024-03-11T20:57:15.921 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:15 smithi023 bash[21372]: cluster 2024-03-11T20:57:15.570647+0000 osd.5 (osd.5) 138 : cluster 0 2.2 scrub starts 2024-03-11T20:57:15.921 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:15 smithi023 bash[21372]: cluster 2024-03-11T20:57:15.573617+0000 osd.5 (osd.5) 139 : cluster 0 2.2 scrub ok 2024-03-11T20:57:15.921 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:15 smithi023 bash[29146]: cluster 2024-03-11T20:57:14.072957+0000 osd.4 (osd.4) 151 : cluster 0 7.1 scrub starts 2024-03-11T20:57:15.921 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:15 smithi023 bash[29146]: cluster 2024-03-11T20:57:14.935896+0000 osd.4 (osd.4) 152 : cluster 0 7.1 scrub ok 2024-03-11T20:57:15.921 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:15 smithi023 bash[29146]: audit 2024-03-11T20:57:15.116820+0000 mgr.y (mgr.24380) 243 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:57:15.921 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:15 smithi023 bash[29146]: cluster 2024-03-11T20:57:15.373577+0000 osd.3 (osd.3) 173 : cluster 0 6.1b scrub starts 2024-03-11T20:57:15.921 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:15 smithi023 bash[29146]: cluster 2024-03-11T20:57:15.380213+0000 osd.3 (osd.3) 174 : cluster 0 6.1b scrub ok 2024-03-11T20:57:15.921 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:15 smithi023 bash[29146]: cluster 2024-03-11T20:57:15.570647+0000 osd.5 (osd.5) 138 : cluster 0 2.2 scrub starts 2024-03-11T20:57:15.921 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:15 smithi023 bash[29146]: cluster 2024-03-11T20:57:15.573617+0000 osd.5 (osd.5) 139 : cluster 0 2.2 scrub ok 2024-03-11T20:57:15.946 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:57:15.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:15.952 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:57:16.052 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:57:16.061 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:15 smithi146 bash[23809]: cluster 2024-03-11T20:57:14.072957+0000 osd.4 (osd.4) 151 : cluster 0 7.1 scrub starts 2024-03-11T20:57:16.061 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:15 smithi146 bash[23809]: cluster 2024-03-11T20:57:14.935896+0000 osd.4 (osd.4) 152 : cluster 0 7.1 scrub ok 2024-03-11T20:57:16.061 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:15 smithi146 bash[23809]: audit 2024-03-11T20:57:15.116820+0000 mgr.y (mgr.24380) 243 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:57:16.061 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:15 smithi146 bash[23809]: cluster 2024-03-11T20:57:15.373577+0000 osd.3 (osd.3) 173 : cluster 0 6.1b scrub starts 2024-03-11T20:57:16.061 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:15 smithi146 bash[23809]: cluster 2024-03-11T20:57:15.380213+0000 osd.3 (osd.3) 174 : cluster 0 6.1b scrub ok 2024-03-11T20:57:16.062 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:15 smithi146 bash[23809]: cluster 2024-03-11T20:57:15.570647+0000 osd.5 (osd.5) 138 : cluster 0 2.2 scrub starts 2024-03-11T20:57:16.062 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:15 smithi146 bash[23809]: cluster 2024-03-11T20:57:15.573617+0000 osd.5 (osd.5) 139 : cluster 0 2.2 scrub ok 2024-03-11T20:57:16.074 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:57:16.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:16.077 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:57:16.178 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:57:16.201 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:57:16.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:16.205 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:57:16.306 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:16.332 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:16.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:16.336 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:57:16.437 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:57:16.463 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:57:16.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:16.466 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:57:16.567 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:57:16.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:16.600 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:57:16.601 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:57:16.701 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:57:16.727 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:57:16.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:16.728 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:57:16.829 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:57:16.858 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:57:16.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:16.865 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:57:16.966 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:57:16.991 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:57:16.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:16.995 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:57:17.096 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:57:17.129 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:57:17.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:17.131 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:57:17.232 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:57:17.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:16 smithi146 bash[23809]: cluster 2024-03-11T20:57:15.092404+0000 osd.4 (osd.4) 153 : cluster 0 3.14 deep-scrub starts 2024-03-11T20:57:17.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:16 smithi146 bash[23809]: cluster 2024-03-11T20:57:15.125541+0000 osd.4 (osd.4) 154 : cluster 0 3.14 deep-scrub ok 2024-03-11T20:57:17.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:16 smithi146 bash[23809]: cluster 2024-03-11T20:57:15.471695+0000 osd.2 (osd.2) 113 : cluster 0 7.e scrub starts 2024-03-11T20:57:17.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:16 smithi146 bash[23809]: cluster 2024-03-11T20:57:15.954930+0000 mgr.y (mgr.24380) 244 : cluster 0 pgmap v234: 148 pgs: 1 active+clean+scrubbing, 145 active+clean, 2 active+clean+scrubbing+deep; 3.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.06k op/s 2024-03-11T20:57:17.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:16 smithi146 bash[23809]: cluster 2024-03-11T20:57:16.407857+0000 osd.3 (osd.3) 175 : cluster 0 4.0 scrub starts 2024-03-11T20:57:17.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:16 smithi146 bash[23809]: cluster 2024-03-11T20:57:16.412772+0000 osd.3 (osd.3) 176 : cluster 0 4.0 scrub ok 2024-03-11T20:57:17.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:16 smithi146 bash[23809]: cluster 2024-03-11T20:57:16.526149+0000 osd.5 (osd.5) 140 : cluster 0 6.d scrub starts 2024-03-11T20:57:17.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:16 smithi146 bash[23809]: cluster 2024-03-11T20:57:16.530660+0000 osd.5 (osd.5) 141 : cluster 0 6.d scrub ok 2024-03-11T20:57:17.273 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T20:57:17.277 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:57:17.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:17.281 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:57:17.382 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:57:17.391 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:17 smithi023 bash[21372]: cluster 2024-03-11T20:57:15.092404+0000 osd.4 (osd.4) 153 : cluster 0 3.14 deep-scrub starts 2024-03-11T20:57:17.391 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:17 smithi023 bash[21372]: cluster 2024-03-11T20:57:15.125541+0000 osd.4 (osd.4) 154 : cluster 0 3.14 deep-scrub ok 2024-03-11T20:57:17.391 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:17 smithi023 bash[21372]: cluster 2024-03-11T20:57:15.471695+0000 osd.2 (osd.2) 113 : cluster 0 7.e scrub starts 2024-03-11T20:57:17.391 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:17 smithi023 bash[21372]: cluster 2024-03-11T20:57:15.954930+0000 mgr.y (mgr.24380) 244 : cluster 0 pgmap v234: 148 pgs: 1 active+clean+scrubbing, 145 active+clean, 2 active+clean+scrubbing+deep; 3.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.06k op/s 2024-03-11T20:57:17.391 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:17 smithi023 bash[21372]: cluster 2024-03-11T20:57:16.407857+0000 osd.3 (osd.3) 175 : cluster 0 4.0 scrub starts 2024-03-11T20:57:17.391 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:17 smithi023 bash[21372]: cluster 2024-03-11T20:57:16.412772+0000 osd.3 (osd.3) 176 : cluster 0 4.0 scrub ok 2024-03-11T20:57:17.391 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:17 smithi023 bash[21372]: cluster 2024-03-11T20:57:16.526149+0000 osd.5 (osd.5) 140 : cluster 0 6.d scrub starts 2024-03-11T20:57:17.391 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:17 smithi023 bash[21372]: cluster 2024-03-11T20:57:16.530660+0000 osd.5 (osd.5) 141 : cluster 0 6.d scrub ok 2024-03-11T20:57:17.391 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:17 smithi023 bash[29146]: cluster 2024-03-11T20:57:15.092404+0000 osd.4 (osd.4) 153 : cluster 0 3.14 deep-scrub starts 2024-03-11T20:57:17.392 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:17 smithi023 bash[29146]: cluster 2024-03-11T20:57:15.125541+0000 osd.4 (osd.4) 154 : cluster 0 3.14 deep-scrub ok 2024-03-11T20:57:17.392 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:17 smithi023 bash[29146]: cluster 2024-03-11T20:57:15.471695+0000 osd.2 (osd.2) 113 : cluster 0 7.e scrub starts 2024-03-11T20:57:17.392 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:17 smithi023 bash[29146]: cluster 2024-03-11T20:57:15.954930+0000 mgr.y (mgr.24380) 244 : cluster 0 pgmap v234: 148 pgs: 1 active+clean+scrubbing, 145 active+clean, 2 active+clean+scrubbing+deep; 3.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.06k op/s 2024-03-11T20:57:17.392 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:17 smithi023 bash[29146]: cluster 2024-03-11T20:57:16.407857+0000 osd.3 (osd.3) 175 : cluster 0 4.0 scrub starts 2024-03-11T20:57:17.392 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:17 smithi023 bash[29146]: cluster 2024-03-11T20:57:16.412772+0000 osd.3 (osd.3) 176 : cluster 0 4.0 scrub ok 2024-03-11T20:57:17.392 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:17 smithi023 bash[29146]: cluster 2024-03-11T20:57:16.526149+0000 osd.5 (osd.5) 140 : cluster 0 6.d scrub starts 2024-03-11T20:57:17.392 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:17 smithi023 bash[29146]: cluster 2024-03-11T20:57:16.530660+0000 osd.5 (osd.5) 141 : cluster 0 6.d scrub ok 2024-03-11T20:57:17.419 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:57:17.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:17.424 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:57:17.485 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd set noscrub 2024-03-11T20:57:17.524 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:57:17.565 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:57:17.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:17.571 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:57:17.672 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:17.705 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:17.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:17.709 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:57:17.810 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:57:17.834 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:57:17.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:17.837 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:57:17.933 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:17 smithi023 bash[21372]: cluster 2024-03-11T20:57:16.031295+0000 osd.4 (osd.4) 155 : cluster 0 6.17 deep-scrub starts 2024-03-11T20:57:17.933 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:17 smithi023 bash[21372]: cluster 2024-03-11T20:57:16.032290+0000 osd.4 (osd.4) 156 : cluster 0 6.17 deep-scrub ok 2024-03-11T20:57:17.933 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:17 smithi023 bash[21372]: cluster 2024-03-11T20:57:16.443810+0000 osd.2 (osd.2) 114 : cluster 0 4.1c deep-scrub starts 2024-03-11T20:57:17.933 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:17 smithi023 bash[21372]: cluster 2024-03-11T20:57:16.454261+0000 osd.2 (osd.2) 115 : cluster 0 4.1c deep-scrub ok 2024-03-11T20:57:17.933 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:17 smithi023 bash[21372]: cluster 2024-03-11T20:57:16.561903+0000 osd.2 (osd.2) 116 : cluster 0 7.e scrub ok 2024-03-11T20:57:17.933 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:17 smithi023 bash[21372]: cluster 2024-03-11T20:57:16.846274+0000 osd.6 (osd.6) 100 : cluster 0 7.7 scrub starts 2024-03-11T20:57:17.933 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:17 smithi023 bash[21372]: cluster 2024-03-11T20:57:17.486021+0000 osd.5 (osd.5) 142 : cluster 0 2.15 scrub starts 2024-03-11T20:57:17.933 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:17 smithi023 bash[21372]: cluster 2024-03-11T20:57:17.487421+0000 osd.5 (osd.5) 143 : cluster 0 2.15 scrub ok 2024-03-11T20:57:17.934 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:17 smithi023 bash[29146]: cluster 2024-03-11T20:57:16.031295+0000 osd.4 (osd.4) 155 : cluster 0 6.17 deep-scrub starts 2024-03-11T20:57:17.934 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:17 smithi023 bash[29146]: cluster 2024-03-11T20:57:16.032290+0000 osd.4 (osd.4) 156 : cluster 0 6.17 deep-scrub ok 2024-03-11T20:57:17.934 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:17 smithi023 bash[29146]: cluster 2024-03-11T20:57:16.443810+0000 osd.2 (osd.2) 114 : cluster 0 4.1c deep-scrub starts 2024-03-11T20:57:17.934 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:17 smithi023 bash[29146]: cluster 2024-03-11T20:57:16.454261+0000 osd.2 (osd.2) 115 : cluster 0 4.1c deep-scrub ok 2024-03-11T20:57:17.934 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:17 smithi023 bash[29146]: cluster 2024-03-11T20:57:16.561903+0000 osd.2 (osd.2) 116 : cluster 0 7.e scrub ok 2024-03-11T20:57:17.934 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:17 smithi023 bash[29146]: cluster 2024-03-11T20:57:16.846274+0000 osd.6 (osd.6) 100 : cluster 0 7.7 scrub starts 2024-03-11T20:57:17.934 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:17 smithi023 bash[29146]: cluster 2024-03-11T20:57:17.486021+0000 osd.5 (osd.5) 142 : cluster 0 2.15 scrub starts 2024-03-11T20:57:17.934 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:17 smithi023 bash[29146]: cluster 2024-03-11T20:57:17.487421+0000 osd.5 (osd.5) 143 : cluster 0 2.15 scrub ok 2024-03-11T20:57:17.937 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:57:17.961 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:57:17.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:17.964 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:57:18.066 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:57:18.093 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:57:18.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:18.095 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:57:18.196 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:57:18.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:17 smithi146 bash[23809]: cluster 2024-03-11T20:57:16.031295+0000 osd.4 (osd.4) 155 : cluster 0 6.17 deep-scrub starts 2024-03-11T20:57:18.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:17 smithi146 bash[23809]: cluster 2024-03-11T20:57:16.032290+0000 osd.4 (osd.4) 156 : cluster 0 6.17 deep-scrub ok 2024-03-11T20:57:18.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:17 smithi146 bash[23809]: cluster 2024-03-11T20:57:16.443810+0000 osd.2 (osd.2) 114 : cluster 0 4.1c deep-scrub starts 2024-03-11T20:57:18.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:17 smithi146 bash[23809]: cluster 2024-03-11T20:57:16.454261+0000 osd.2 (osd.2) 115 : cluster 0 4.1c deep-scrub ok 2024-03-11T20:57:18.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:17 smithi146 bash[23809]: cluster 2024-03-11T20:57:16.561903+0000 osd.2 (osd.2) 116 : cluster 0 7.e scrub ok 2024-03-11T20:57:18.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:17 smithi146 bash[23809]: cluster 2024-03-11T20:57:16.846274+0000 osd.6 (osd.6) 100 : cluster 0 7.7 scrub starts 2024-03-11T20:57:18.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:17 smithi146 bash[23809]: cluster 2024-03-11T20:57:17.486021+0000 osd.5 (osd.5) 142 : cluster 0 2.15 scrub starts 2024-03-11T20:57:18.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:17 smithi146 bash[23809]: cluster 2024-03-11T20:57:17.487421+0000 osd.5 (osd.5) 143 : cluster 0 2.15 scrub ok 2024-03-11T20:57:18.220 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:57:18.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:18.223 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:57:18.324 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:57:18.354 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:57:18.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:18.357 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:57:18.458 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:57:18.478 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:57:18.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:18.482 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:57:18.534 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:57:18 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:20:57:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T20:57:18.583 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:57:18.609 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:57:18.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:18.611 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:57:18.711 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:57:18.748 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:57:18.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:18.750 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:57:18.851 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:57:18.879 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:57:18.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:18.882 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:57:18.983 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:57:19.002 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:18 smithi023 bash[21372]: cluster 2024-03-11T20:57:17.438756+0000 osd.2 (osd.2) 117 : cluster 0 2.c deep-scrub starts 2024-03-11T20:57:19.003 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:18 smithi023 bash[21372]: cluster 2024-03-11T20:57:17.447184+0000 osd.2 (osd.2) 118 : cluster 0 2.c deep-scrub ok 2024-03-11T20:57:19.003 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:18 smithi023 bash[21372]: cluster 2024-03-11T20:57:17.825992+0000 osd.6 (osd.6) 101 : cluster 0 7.7 scrub ok 2024-03-11T20:57:19.003 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:18 smithi023 bash[21372]: cluster 2024-03-11T20:57:17.955309+0000 mgr.y (mgr.24380) 245 : cluster 0 pgmap v235: 148 pgs: 1 active+clean+scrubbing, 145 active+clean, 2 active+clean+scrubbing+deep; 3.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.06k op/s 2024-03-11T20:57:19.007 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:57:19.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:19.014 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:57:19.115 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:19.124 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:18 smithi146 bash[23809]: cluster 2024-03-11T20:57:17.438756+0000 osd.2 (osd.2) 117 : cluster 0 2.c deep-scrub starts 2024-03-11T20:57:19.124 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:18 smithi146 bash[23809]: cluster 2024-03-11T20:57:17.447184+0000 osd.2 (osd.2) 118 : cluster 0 2.c deep-scrub ok 2024-03-11T20:57:19.124 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:18 smithi146 bash[23809]: cluster 2024-03-11T20:57:17.825992+0000 osd.6 (osd.6) 101 : cluster 0 7.7 scrub ok 2024-03-11T20:57:19.124 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:18 smithi146 bash[23809]: cluster 2024-03-11T20:57:17.955309+0000 mgr.y (mgr.24380) 245 : cluster 0 pgmap v235: 148 pgs: 1 active+clean+scrubbing, 145 active+clean, 2 active+clean+scrubbing+deep; 3.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.06k op/s 2024-03-11T20:57:19.137 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:19.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:19.144 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:57:19.245 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:57:19.273 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:57:19.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:19.277 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:57:19.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:19 smithi023 bash[29146]: cluster 2024-03-11T20:57:17.438756+0000 osd.2 (osd.2) 117 : cluster 0 2.c deep-scrub starts 2024-03-11T20:57:19.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:19 smithi023 bash[29146]: cluster 2024-03-11T20:57:17.447184+0000 osd.2 (osd.2) 118 : cluster 0 2.c deep-scrub ok 2024-03-11T20:57:19.284 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:19 smithi023 bash[29146]: cluster 2024-03-11T20:57:17.825992+0000 osd.6 (osd.6) 101 : cluster 0 7.7 scrub ok 2024-03-11T20:57:19.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:19 smithi023 bash[29146]: cluster 2024-03-11T20:57:17.955309+0000 mgr.y (mgr.24380) 245 : cluster 0 pgmap v235: 148 pgs: 1 active+clean+scrubbing, 145 active+clean, 2 active+clean+scrubbing+deep; 3.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.06k op/s 2024-03-11T20:57:19.377 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:57:19.405 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:57:19.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:19.409 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:57:19.510 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:57:19.550 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:57:19.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:19.552 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:57:19.653 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:57:19.682 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:57:19.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:19.685 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:57:19.786 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:57:19.807 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:57:19.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:19.810 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:57:19.911 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:57:19.934 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:57:19.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:19.937 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:57:20.038 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:57:20.065 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:57:20.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:20.071 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:57:20.172 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:20.202 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:20.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:20.205 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:57:20.306 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:57:20.332 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:57:20.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:20.334 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:57:20.435 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:57:20.462 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:57:20.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:20.469 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:57:20.569 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:57:20.582 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:20 smithi146 bash[23809]: cluster 2024-03-11T20:57:19.061965+0000 osd.4 (osd.4) 157 : cluster 0 6.2 scrub starts 2024-03-11T20:57:20.582 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:20 smithi146 bash[23809]: cluster 2024-03-11T20:57:19.063706+0000 osd.4 (osd.4) 158 : cluster 0 6.2 scrub ok 2024-03-11T20:57:20.582 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:20 smithi146 bash[23809]: audit 2024-03-11T20:57:19.527599+0000 mon.a (mon.0) 1053 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:57:20.582 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:20 smithi146 bash[23809]: audit 2024-03-11T20:57:19.529788+0000 mon.a (mon.0) 1054 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:57:20.582 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:20 smithi146 bash[23809]: cluster 2024-03-11T20:57:19.956045+0000 mgr.y (mgr.24380) 246 : cluster 0 pgmap v236: 148 pgs: 146 active+clean, 2 active+clean+scrubbing+deep; 3.5 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 896 op/s 2024-03-11T20:57:20.582 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:20 smithi146 bash[23809]: cluster 2024-03-11T20:57:19.956210+0000 osd.0 (osd.0) 155 : cluster 0 4.b scrub starts 2024-03-11T20:57:20.582 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:20 smithi146 bash[23809]: cluster 2024-03-11T20:57:19.959830+0000 osd.0 (osd.0) 156 : cluster 0 4.b scrub ok 2024-03-11T20:57:20.598 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:57:20.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:20.601 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:57:20.702 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:20.731 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:20.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:20.734 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:57:20.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:20 smithi023 bash[21372]: cluster 2024-03-11T20:57:19.061965+0000 osd.4 (osd.4) 157 : cluster 0 6.2 scrub starts 2024-03-11T20:57:20.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:20 smithi023 bash[21372]: cluster 2024-03-11T20:57:19.063706+0000 osd.4 (osd.4) 158 : cluster 0 6.2 scrub ok 2024-03-11T20:57:20.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:20 smithi023 bash[21372]: audit 2024-03-11T20:57:19.527599+0000 mon.a (mon.0) 1053 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:57:20.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:20 smithi023 bash[21372]: audit 2024-03-11T20:57:19.529788+0000 mon.a (mon.0) 1054 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:57:20.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:20 smithi023 bash[21372]: cluster 2024-03-11T20:57:19.956045+0000 mgr.y (mgr.24380) 246 : cluster 0 pgmap v236: 148 pgs: 146 active+clean, 2 active+clean+scrubbing+deep; 3.5 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 896 op/s 2024-03-11T20:57:20.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:20 smithi023 bash[21372]: cluster 2024-03-11T20:57:19.956210+0000 osd.0 (osd.0) 155 : cluster 0 4.b scrub starts 2024-03-11T20:57:20.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:20 smithi023 bash[21372]: cluster 2024-03-11T20:57:19.959830+0000 osd.0 (osd.0) 156 : cluster 0 4.b scrub ok 2024-03-11T20:57:20.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:20 smithi023 bash[29146]: cluster 2024-03-11T20:57:19.061965+0000 osd.4 (osd.4) 157 : cluster 0 6.2 scrub starts 2024-03-11T20:57:20.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:20 smithi023 bash[29146]: cluster 2024-03-11T20:57:19.063706+0000 osd.4 (osd.4) 158 : cluster 0 6.2 scrub ok 2024-03-11T20:57:20.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:20 smithi023 bash[29146]: audit 2024-03-11T20:57:19.527599+0000 mon.a (mon.0) 1053 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:57:20.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:20 smithi023 bash[29146]: audit 2024-03-11T20:57:19.529788+0000 mon.a (mon.0) 1054 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:57:20.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:20 smithi023 bash[29146]: cluster 2024-03-11T20:57:19.956045+0000 mgr.y (mgr.24380) 246 : cluster 0 pgmap v236: 148 pgs: 146 active+clean, 2 active+clean+scrubbing+deep; 3.5 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 896 op/s 2024-03-11T20:57:20.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:20 smithi023 bash[29146]: cluster 2024-03-11T20:57:19.956210+0000 osd.0 (osd.0) 155 : cluster 0 4.b scrub starts 2024-03-11T20:57:20.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:20 smithi023 bash[29146]: cluster 2024-03-11T20:57:19.959830+0000 osd.0 (osd.0) 156 : cluster 0 4.b scrub ok 2024-03-11T20:57:20.836 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:57:20.855 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:57:20.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:20.858 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:57:20.959 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:57:20.986 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:57:20.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:20.992 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:57:21.093 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:57:21.117 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:57:21.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:21.120 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:57:21.221 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:21.248 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:21.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:21.252 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:57:21.353 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:21.376 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:21.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:21.378 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:57:21.479 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:57:21.516 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:57:21.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:21.521 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:57:21.623 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:21.650 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:21.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:21.655 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:57:21.756 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:57:21.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:21 smithi146 bash[23809]: cluster 2024-03-11T20:57:20.481317+0000 osd.3 (osd.3) 177 : cluster 0 7.3 scrub starts 2024-03-11T20:57:21.766 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:21 smithi023 bash[21372]: cluster 2024-03-11T20:57:20.481317+0000 osd.3 (osd.3) 177 : cluster 0 7.3 scrub starts 2024-03-11T20:57:21.766 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:21 smithi023 bash[29146]: cluster 2024-03-11T20:57:20.481317+0000 osd.3 (osd.3) 177 : cluster 0 7.3 scrub starts 2024-03-11T20:57:21.787 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:57:21.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:21.789 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:57:21.890 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:21.923 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:21.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:21.926 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:57:22.027 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:57:22.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:22.059 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:57:22.059 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:57:22.160 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:57:22.192 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:57:22.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:22.194 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:57:22.295 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:57:22.316 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:57:22.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:22.319 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:57:22.420 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:22.444 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:22.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:22.450 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:57:22.551 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:57:22.598 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:57:22.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:22.602 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:57:22.634 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-11T20:57:22.634 INFO:teuthology.orchestra.run.smithi023.stdout: "ops": [ 2024-03-11T20:57:22.635 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.635 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_repop(osd.6.0:33190 7.c e161/121)", 2024-03-11T20:57:22.635 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T20:57:22.614194+0000", 2024-03-11T20:57:22.635 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 0.016267862000000001, 2024-03-11T20:57:22.635 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.016293433, 2024-03-11T20:57:22.635 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T20:57:22.635 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "queued for pg", 2024-03-11T20:57:22.635 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T20:57:22.635 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.635 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T20:57:22.635 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.614194+0000", 2024-03-11T20:57:22.635 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:57:22.635 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.635 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.636 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T20:57:22.636 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.614194+0000", 2024-03-11T20:57:22.636 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:57:22.636 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.636 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.636 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T20:57:22.636 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.614195+0000", 2024-03-11T20:57:22.636 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 9.8200000000000008e-07 2024-03-11T20:57:22.636 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.636 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.636 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T20:57:22.636 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.614201+0000", 2024-03-11T20:57:22.636 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 5.801e-06 2024-03-11T20:57:22.636 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.637 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.637 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T20:57:22.637 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.614211+0000", 2024-03-11T20:57:22.637 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.0730999999999999e-05 2024-03-11T20:57:22.637 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.637 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.638 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T20:57:22.638 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.614218+0000", 2024-03-11T20:57:22.638 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.2099999999999996e-06 2024-03-11T20:57:22.638 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:57:22.638 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T20:57:22.638 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:57:22.638 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.638 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.638 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_op(client.24629.0:167139 7.a 7:5522b5e8:::benchmark_data_smithi023_67959_object56898:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e161)", 2024-03-11T20:57:22.638 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T20:57:22.615673+0000", 2024-03-11T20:57:22.639 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 0.014789081000000001, 2024-03-11T20:57:22.639 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.014876074, 2024-03-11T20:57:22.639 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T20:57:22.639 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "waiting for sub ops", 2024-03-11T20:57:22.639 INFO:teuthology.orchestra.run.smithi023.stdout: "client_info": { 2024-03-11T20:57:22.639 INFO:teuthology.orchestra.run.smithi023.stdout: "client": "client.24629", 2024-03-11T20:57:22.639 INFO:teuthology.orchestra.run.smithi023.stdout: "client_addr": "172.21.15.23:0/959732609", 2024-03-11T20:57:22.639 INFO:teuthology.orchestra.run.smithi023.stdout: "tid": 167139 2024-03-11T20:57:22.639 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.639 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T20:57:22.639 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.639 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T20:57:22.639 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.615673+0000", 2024-03-11T20:57:22.639 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:57:22.640 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.640 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.640 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T20:57:22.640 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.615673+0000", 2024-03-11T20:57:22.640 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:57:22.640 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.640 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.640 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T20:57:22.640 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.615674+0000", 2024-03-11T20:57:22.640 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.579e-06 2024-03-11T20:57:22.640 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.640 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.640 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T20:57:22.640 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.615680+0000", 2024-03-11T20:57:22.641 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.032e-06 2024-03-11T20:57:22.641 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.641 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.641 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T20:57:22.641 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.615693+0000", 2024-03-11T20:57:22.641 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.312e-05 2024-03-11T20:57:22.641 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.641 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.641 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T20:57:22.641 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.615702+0000", 2024-03-11T20:57:22.641 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 9.0159999999999997e-06 2024-03-11T20:57:22.641 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.641 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.642 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T20:57:22.642 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.620212+0000", 2024-03-11T20:57:22.642 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0045095969999999997 2024-03-11T20:57:22.642 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.642 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.642 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T20:57:22.642 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.620332+0000", 2024-03-11T20:57:22.642 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000120617 2024-03-11T20:57:22.642 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.642 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.642 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for subops from 2,5", 2024-03-11T20:57:22.642 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.620449+0000", 2024-03-11T20:57:22.642 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000116474 2024-03-11T20:57:22.643 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.643 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.643 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T20:57:22.643 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.626515+0000", 2024-03-11T20:57:22.643 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0060664400000000002 2024-03-11T20:57:22.644 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:57:22.644 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T20:57:22.644 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:57:22.644 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.644 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.644 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_repop(osd.6.0:33192 7.c e161/121)", 2024-03-11T20:57:22.644 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T20:57:22.617972+0000", 2024-03-11T20:57:22.644 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 0.012489418, 2024-03-11T20:57:22.644 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.012665991999999999, 2024-03-11T20:57:22.644 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T20:57:22.644 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "queued for pg", 2024-03-11T20:57:22.645 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T20:57:22.645 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.645 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T20:57:22.645 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.617972+0000", 2024-03-11T20:57:22.645 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:57:22.645 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.645 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.645 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T20:57:22.645 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.617972+0000", 2024-03-11T20:57:22.645 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:57:22.645 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.645 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.645 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T20:57:22.645 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.617973+0000", 2024-03-11T20:57:22.646 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 3.8500000000000002e-07 2024-03-11T20:57:22.646 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.646 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.646 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T20:57:22.646 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.617977+0000", 2024-03-11T20:57:22.646 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 4.6199999999999998e-06 2024-03-11T20:57:22.646 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.646 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.646 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T20:57:22.646 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.617985+0000", 2024-03-11T20:57:22.646 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.4930000000000001e-06 2024-03-11T20:57:22.646 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.646 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.646 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T20:57:22.647 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.617990+0000", 2024-03-11T20:57:22.647 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 5.8069999999999996e-06 2024-03-11T20:57:22.647 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:57:22.647 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T20:57:22.647 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:57:22.647 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.647 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.647 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_repop(client.24629.0:167136 7.6 e161/159 7:61062030:::benchmark_data_smithi023_67959_object56895:head v 161'10386, mlcod=161'10384)", 2024-03-11T20:57:22.647 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T20:57:22.623499+0000", 2024-03-11T20:57:22.647 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 0.0069624930000000002, 2024-03-11T20:57:22.647 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.007200077, 2024-03-11T20:57:22.647 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T20:57:22.647 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "started", 2024-03-11T20:57:22.647 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T20:57:22.648 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.648 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T20:57:22.648 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.623499+0000", 2024-03-11T20:57:22.648 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:57:22.648 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.648 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.648 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T20:57:22.648 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.623499+0000", 2024-03-11T20:57:22.648 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:57:22.648 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.648 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.648 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T20:57:22.648 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.623500+0000", 2024-03-11T20:57:22.649 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.0070000000000001e-06 2024-03-11T20:57:22.649 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.649 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.649 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T20:57:22.649 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.623509+0000", 2024-03-11T20:57:22.649 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 8.8929999999999994e-06 2024-03-11T20:57:22.649 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.649 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.649 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T20:57:22.649 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.623521+0000", 2024-03-11T20:57:22.649 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.2012e-05 2024-03-11T20:57:22.649 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.649 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.649 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T20:57:22.649 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.623530+0000", 2024-03-11T20:57:22.650 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 8.6989999999999992e-06 2024-03-11T20:57:22.650 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.650 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.650 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T20:57:22.650 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.623597+0000", 2024-03-11T20:57:22.650 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.6839000000000002e-05 2024-03-11T20:57:22.650 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.650 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.650 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T20:57:22.650 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.623618+0000", 2024-03-11T20:57:22.650 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.1512999999999999e-05 2024-03-11T20:57:22.650 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:57:22.650 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T20:57:22.650 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:57:22.651 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.651 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.651 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_repop(osd.6.0:33177 7.c e161/121 7:38caf6c8:::benchmark_data_smithi023_67959_object64656:head v 161'12141, mlcod=161'12129)", 2024-03-11T20:57:22.651 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T20:57:22.592336+0000", 2024-03-11T20:57:22.651 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 0.038125222, 2024-03-11T20:57:22.651 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.038435509999999999, 2024-03-11T20:57:22.651 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T20:57:22.651 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "started", 2024-03-11T20:57:22.651 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T20:57:22.651 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.651 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T20:57:22.651 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.592336+0000", 2024-03-11T20:57:22.651 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:57:22.651 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.651 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.652 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T20:57:22.652 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.592336+0000", 2024-03-11T20:57:22.652 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:57:22.652 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.652 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.652 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T20:57:22.652 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.592337+0000", 2024-03-11T20:57:22.652 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.4499999999999996e-07 2024-03-11T20:57:22.652 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.652 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.652 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T20:57:22.653 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.592343+0000", 2024-03-11T20:57:22.653 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 5.6960000000000002e-06 2024-03-11T20:57:22.653 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.653 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.653 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T20:57:22.653 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.592351+0000", 2024-03-11T20:57:22.653 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 8.6519999999999995e-06 2024-03-11T20:57:22.653 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.653 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.653 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T20:57:22.653 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.592358+0000", 2024-03-11T20:57:22.653 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.3690000000000003e-06 2024-03-11T20:57:22.653 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.653 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.654 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T20:57:22.654 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.615914+0000", 2024-03-11T20:57:22.654 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.023556437999999999 2024-03-11T20:57:22.654 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.654 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.654 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T20:57:22.654 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.616115+0000", 2024-03-11T20:57:22.654 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00020116900000000001 2024-03-11T20:57:22.654 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:57:22.654 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T20:57:22.654 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:57:22.654 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.654 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.655 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_repop(osd.6.0:33178 7.c e161/121 7:38cccb42:::benchmark_data_smithi023_67959_object105593:head v 161'12142, mlcod=161'12129)", 2024-03-11T20:57:22.655 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T20:57:22.593886+0000", 2024-03-11T20:57:22.655 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 0.036576055000000003, 2024-03-11T20:57:22.655 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.036956071, 2024-03-11T20:57:22.655 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T20:57:22.655 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "started", 2024-03-11T20:57:22.655 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T20:57:22.655 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.655 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T20:57:22.655 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.593886+0000", 2024-03-11T20:57:22.655 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:57:22.655 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.655 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.655 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T20:57:22.655 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.593886+0000", 2024-03-11T20:57:22.656 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:57:22.656 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.656 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.656 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T20:57:22.656 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.593886+0000", 2024-03-11T20:57:22.656 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 8.5099999999999998e-07 2024-03-11T20:57:22.656 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.656 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.656 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T20:57:22.656 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.593894+0000", 2024-03-11T20:57:22.656 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.8019999999999992e-06 2024-03-11T20:57:22.656 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.656 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.657 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T20:57:22.657 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.593904+0000", 2024-03-11T20:57:22.657 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 9.3109999999999995e-06 2024-03-11T20:57:22.657 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.657 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.657 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T20:57:22.657 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.593911+0000", 2024-03-11T20:57:22.657 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.1690000000000003e-06 2024-03-11T20:57:22.657 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.657 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.657 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T20:57:22.657 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.618886+0000", 2024-03-11T20:57:22.658 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.024975357 2024-03-11T20:57:22.658 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.658 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.658 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T20:57:22.658 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.618904+0000", 2024-03-11T20:57:22.658 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.7940000000000001e-05 2024-03-11T20:57:22.658 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:57:22.658 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T20:57:22.658 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:57:22.658 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.658 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.658 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_repop(client.24629.0:167140 7.d e161/124 7:bdba13f4:::benchmark_data_smithi023_67959_object56899:head v 161'10443, mlcod=161'10440)", 2024-03-11T20:57:22.658 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T20:57:22.625065+0000", 2024-03-11T20:57:22.658 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 0.0053966780000000002, 2024-03-11T20:57:22.659 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0058829779999999996, 2024-03-11T20:57:22.659 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T20:57:22.659 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "started", 2024-03-11T20:57:22.659 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T20:57:22.659 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.659 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T20:57:22.659 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.625065+0000", 2024-03-11T20:57:22.659 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:57:22.659 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.659 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.659 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T20:57:22.659 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.625065+0000", 2024-03-11T20:57:22.659 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:57:22.659 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.659 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.660 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T20:57:22.660 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.625066+0000", 2024-03-11T20:57:22.660 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.9599999999999998e-07 2024-03-11T20:57:22.660 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.660 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.660 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T20:57:22.660 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.625073+0000", 2024-03-11T20:57:22.660 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.2150000000000004e-06 2024-03-11T20:57:22.660 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.660 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.660 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T20:57:22.660 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.625085+0000", 2024-03-11T20:57:22.660 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.1882000000000001e-05 2024-03-11T20:57:22.660 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.661 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.661 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T20:57:22.661 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.625092+0000", 2024-03-11T20:57:22.661 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.9430000000000003e-06 2024-03-11T20:57:22.661 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.661 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.661 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T20:57:22.661 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.625158+0000", 2024-03-11T20:57:22.661 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.6110000000000002e-05 2024-03-11T20:57:22.661 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.661 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.661 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T20:57:22.661 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.625178+0000", 2024-03-11T20:57:22.661 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.0088000000000001e-05 2024-03-11T20:57:22.662 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:57:22.662 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T20:57:22.662 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:57:22.662 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.662 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.662 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_repop(osd.6.0:33194 7.c e161/121)", 2024-03-11T20:57:22.662 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T20:57:22.625111+0000", 2024-03-11T20:57:22.662 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 0.0053506780000000002, 2024-03-11T20:57:22.662 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0059427250000000003, 2024-03-11T20:57:22.662 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T20:57:22.662 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "queued for pg", 2024-03-11T20:57:22.662 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T20:57:22.662 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.662 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T20:57:22.663 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.625111+0000", 2024-03-11T20:57:22.663 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:57:22.663 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.663 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.663 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T20:57:22.663 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.625111+0000", 2024-03-11T20:57:22.663 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:57:22.663 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.663 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.663 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T20:57:22.663 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.625111+0000", 2024-03-11T20:57:22.664 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.7300000000000002e-07 2024-03-11T20:57:22.664 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.664 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.664 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T20:57:22.664 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.625115+0000", 2024-03-11T20:57:22.664 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 4.031e-06 2024-03-11T20:57:22.664 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.665 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.665 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T20:57:22.665 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.625121+0000", 2024-03-11T20:57:22.665 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.1569999999999997e-06 2024-03-11T20:57:22.665 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.665 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.665 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T20:57:22.665 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.625126+0000", 2024-03-11T20:57:22.665 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 4.6619999999999996e-06 2024-03-11T20:57:22.665 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:57:22.665 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T20:57:22.665 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:57:22.665 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.665 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.666 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_repop(osd.6.0:33179 7.c e161/121 7:38cea3b7:::benchmark_data_smithi023_67959_object67088:head v 161'12143, mlcod=161'12129)", 2024-03-11T20:57:22.666 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T20:57:22.595166+0000", 2024-03-11T20:57:22.666 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 0.035295909, 2024-03-11T20:57:22.666 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.035947702999999998, 2024-03-11T20:57:22.666 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T20:57:22.666 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "started", 2024-03-11T20:57:22.666 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T20:57:22.666 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.666 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T20:57:22.666 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.595166+0000", 2024-03-11T20:57:22.666 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:57:22.666 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.666 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.666 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T20:57:22.666 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.595166+0000", 2024-03-11T20:57:22.667 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:57:22.667 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.667 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.667 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T20:57:22.667 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.595166+0000", 2024-03-11T20:57:22.667 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.0999999999999998e-07 2024-03-11T20:57:22.667 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.667 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.667 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T20:57:22.667 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.595172+0000", 2024-03-11T20:57:22.667 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 5.9610000000000002e-06 2024-03-11T20:57:22.667 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.667 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.667 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T20:57:22.667 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.595182+0000", 2024-03-11T20:57:22.668 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 9.5659999999999996e-06 2024-03-11T20:57:22.668 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.668 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.668 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T20:57:22.668 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.595188+0000", 2024-03-11T20:57:22.668 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.6590000000000001e-06 2024-03-11T20:57:22.668 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.668 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.668 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T20:57:22.668 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.621287+0000", 2024-03-11T20:57:22.668 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.026098215000000001 2024-03-11T20:57:22.668 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.668 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.668 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T20:57:22.669 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.621307+0000", 2024-03-11T20:57:22.669 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.9993999999999999e-05 2024-03-11T20:57:22.669 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:57:22.669 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T20:57:22.669 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:57:22.669 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.669 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.669 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_repop(client.24629.0:167141 7.7 e161/121 7:edefe8e6:::benchmark_data_smithi023_67959_object56900:head v 161'10548, mlcod=161'10546)", 2024-03-11T20:57:22.669 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T20:57:22.625133+0000", 2024-03-11T20:57:22.669 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 0.0053282889999999999, 2024-03-11T20:57:22.669 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0059839530000000002, 2024-03-11T20:57:22.669 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T20:57:22.669 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T20:57:22.669 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T20:57:22.669 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.670 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T20:57:22.670 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.625133+0000", 2024-03-11T20:57:22.670 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:57:22.670 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.670 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.670 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T20:57:22.670 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.625133+0000", 2024-03-11T20:57:22.670 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:57:22.670 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.670 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.670 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T20:57:22.670 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.625133+0000", 2024-03-11T20:57:22.670 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.8099999999999999e-07 2024-03-11T20:57:22.670 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.671 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.671 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T20:57:22.671 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.625140+0000", 2024-03-11T20:57:22.671 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.6499999999999999e-06 2024-03-11T20:57:22.671 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.671 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.671 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T20:57:22.671 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.625145+0000", 2024-03-11T20:57:22.671 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 5.3499999999999996e-06 2024-03-11T20:57:22.671 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.671 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.671 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T20:57:22.671 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.625150+0000", 2024-03-11T20:57:22.671 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 4.6929999999999998e-06 2024-03-11T20:57:22.671 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.672 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.672 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T20:57:22.672 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.625235+0000", 2024-03-11T20:57:22.672 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 8.4915999999999998e-05 2024-03-11T20:57:22.672 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.672 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.672 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T20:57:22.673 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.625255+0000", 2024-03-11T20:57:22.673 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.9848999999999999e-05 2024-03-11T20:57:22.673 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.673 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.673 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T20:57:22.673 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.631039+0000", 2024-03-11T20:57:22.673 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.005783921 2024-03-11T20:57:22.673 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.673 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.673 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T20:57:22.673 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.631117+0000", 2024-03-11T20:57:22.673 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.8393000000000007e-05 2024-03-11T20:57:22.673 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:57:22.673 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T20:57:22.674 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:57:22.674 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.674 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.674 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_repop(osd.6.0:33180 7.c e161/121 7:38d28751:::benchmark_data_smithi023_67959_object86194:head v 161'12144, mlcod=161'12129)", 2024-03-11T20:57:22.674 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T20:57:22.596466+0000", 2024-03-11T20:57:22.674 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 0.033995569000000003, 2024-03-11T20:57:22.675 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.034794549000000001, 2024-03-11T20:57:22.675 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T20:57:22.675 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "started", 2024-03-11T20:57:22.675 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T20:57:22.675 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.675 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T20:57:22.675 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.596466+0000", 2024-03-11T20:57:22.675 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:57:22.675 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.675 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.675 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T20:57:22.675 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.596466+0000", 2024-03-11T20:57:22.675 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:57:22.676 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.676 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.676 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T20:57:22.676 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.596467+0000", 2024-03-11T20:57:22.676 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.9800000000000003e-07 2024-03-11T20:57:22.676 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.676 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.676 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T20:57:22.676 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.596474+0000", 2024-03-11T20:57:22.676 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.7730000000000002e-06 2024-03-11T20:57:22.676 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.676 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.676 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T20:57:22.677 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.596484+0000", 2024-03-11T20:57:22.677 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.0387000000000001e-05 2024-03-11T20:57:22.677 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.677 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.677 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T20:57:22.677 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.596491+0000", 2024-03-11T20:57:22.677 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.7170000000000002e-06 2024-03-11T20:57:22.677 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.677 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.677 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T20:57:22.677 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.626107+0000", 2024-03-11T20:57:22.677 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.029615932000000001 2024-03-11T20:57:22.677 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.677 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.678 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T20:57:22.678 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.626124+0000", 2024-03-11T20:57:22.678 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.7761000000000002e-05 2024-03-11T20:57:22.678 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:57:22.678 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T20:57:22.678 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:57:22.678 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.678 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.678 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_repop(client.24629.0:167122 7.c e161/121)", 2024-03-11T20:57:22.678 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T20:57:22.625164+0000", 2024-03-11T20:57:22.678 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 0.0052979389999999998, 2024-03-11T20:57:22.678 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0061570319999999998, 2024-03-11T20:57:22.678 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T20:57:22.678 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "queued for pg", 2024-03-11T20:57:22.679 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T20:57:22.679 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.679 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T20:57:22.679 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.625164+0000", 2024-03-11T20:57:22.679 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:57:22.679 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.679 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.679 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T20:57:22.679 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.625164+0000", 2024-03-11T20:57:22.679 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:57:22.679 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.679 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.679 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T20:57:22.679 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.625164+0000", 2024-03-11T20:57:22.680 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.9600000000000001e-07 2024-03-11T20:57:22.680 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.680 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.680 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T20:57:22.680 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.625167+0000", 2024-03-11T20:57:22.680 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.9040000000000001e-06 2024-03-11T20:57:22.680 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.680 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.680 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T20:57:22.680 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.625172+0000", 2024-03-11T20:57:22.680 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 5.5559999999999998e-06 2024-03-11T20:57:22.680 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.681 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.681 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T20:57:22.681 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.625177+0000", 2024-03-11T20:57:22.681 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 4.194e-06 2024-03-11T20:57:22.681 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:57:22.681 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T20:57:22.681 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:57:22.681 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.681 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.681 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_repop(osd.6.0:33181 7.c e161/121 7:38d294a7:::benchmark_data_smithi023_67959_object68114:head v 161'12145, mlcod=161'12129)", 2024-03-11T20:57:22.681 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T20:57:22.597935+0000", 2024-03-11T20:57:22.681 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 0.032526866000000001, 2024-03-11T20:57:22.681 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.033443541, 2024-03-11T20:57:22.681 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T20:57:22.682 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "started", 2024-03-11T20:57:22.682 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T20:57:22.682 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.682 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T20:57:22.682 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.597935+0000", 2024-03-11T20:57:22.682 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:57:22.682 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.682 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.682 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T20:57:22.682 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.597935+0000", 2024-03-11T20:57:22.682 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:57:22.682 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.682 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.682 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T20:57:22.683 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.597936+0000", 2024-03-11T20:57:22.683 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 8.3600000000000002e-07 2024-03-11T20:57:22.683 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.683 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.683 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T20:57:22.683 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.597943+0000", 2024-03-11T20:57:22.683 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.0940000000000002e-06 2024-03-11T20:57:22.683 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.683 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.683 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T20:57:22.683 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.597953+0000", 2024-03-11T20:57:22.683 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.0710000000000001e-05 2024-03-11T20:57:22.683 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.683 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.683 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T20:57:22.684 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.597960+0000", 2024-03-11T20:57:22.684 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.8739999999999997e-06 2024-03-11T20:57:22.684 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.684 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.684 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T20:57:22.684 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.628535+0000", 2024-03-11T20:57:22.684 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.030574667999999999 2024-03-11T20:57:22.684 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.684 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.684 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T20:57:22.684 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.628557+0000", 2024-03-11T20:57:22.684 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.1895e-05 2024-03-11T20:57:22.684 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:57:22.684 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T20:57:22.684 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:57:22.685 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.685 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.685 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_repop(client.24629.0:167123 7.c e161/121)", 2024-03-11T20:57:22.685 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T20:57:22.627502+0000", 2024-03-11T20:57:22.685 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 0.0029594920000000002, 2024-03-11T20:57:22.685 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0039415459999999998, 2024-03-11T20:57:22.685 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T20:57:22.685 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "queued for pg", 2024-03-11T20:57:22.685 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T20:57:22.685 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.685 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T20:57:22.685 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.627502+0000", 2024-03-11T20:57:22.685 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:57:22.685 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.686 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.686 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T20:57:22.686 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.627502+0000", 2024-03-11T20:57:22.686 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:57:22.686 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.686 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.686 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T20:57:22.686 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.627503+0000", 2024-03-11T20:57:22.686 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 5.9500000000000002e-07 2024-03-11T20:57:22.686 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.686 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.686 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T20:57:22.686 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.627508+0000", 2024-03-11T20:57:22.687 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 4.9060000000000001e-06 2024-03-11T20:57:22.687 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.687 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.687 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T20:57:22.687 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.627518+0000", 2024-03-11T20:57:22.687 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.0154e-05 2024-03-11T20:57:22.687 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.687 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.687 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T20:57:22.687 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.627524+0000", 2024-03-11T20:57:22.687 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.2709999999999998e-06 2024-03-11T20:57:22.687 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:57:22.687 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T20:57:22.688 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:57:22.688 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.688 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.688 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_op(client.24629.0:167144 7.2 7.d19242d2 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e161)", 2024-03-11T20:57:22.688 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T20:57:22.630098+0000", 2024-03-11T20:57:22.688 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 0.00036328199999999998, 2024-03-11T20:57:22.688 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0014579059999999999, 2024-03-11T20:57:22.688 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T20:57:22.688 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "queued for pg", 2024-03-11T20:57:22.688 INFO:teuthology.orchestra.run.smithi023.stdout: "client_info": { 2024-03-11T20:57:22.688 INFO:teuthology.orchestra.run.smithi023.stdout: "client": "client.24629", 2024-03-11T20:57:22.688 INFO:teuthology.orchestra.run.smithi023.stdout: "client_addr": "172.21.15.23:0/959732609", 2024-03-11T20:57:22.688 INFO:teuthology.orchestra.run.smithi023.stdout: "tid": 167144 2024-03-11T20:57:22.689 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.689 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T20:57:22.689 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.689 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T20:57:22.689 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.630098+0000", 2024-03-11T20:57:22.689 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:57:22.689 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.689 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.689 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T20:57:22.689 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.630098+0000", 2024-03-11T20:57:22.689 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:57:22.689 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.689 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.689 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T20:57:22.689 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.630100+0000", 2024-03-11T20:57:22.690 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.488e-06 2024-03-11T20:57:22.690 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.690 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.690 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T20:57:22.690 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.630107+0000", 2024-03-11T20:57:22.690 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.6190000000000002e-06 2024-03-11T20:57:22.690 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.690 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.690 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T20:57:22.690 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.630124+0000", 2024-03-11T20:57:22.690 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.6087999999999999e-05 2024-03-11T20:57:22.690 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.690 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.690 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T20:57:22.691 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.630132+0000", 2024-03-11T20:57:22.691 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 8.8720000000000008e-06 2024-03-11T20:57:22.691 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:57:22.691 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T20:57:22.691 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:57:22.691 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.691 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.691 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_repop(osd.6.0:33182 7.c e161/121)", 2024-03-11T20:57:22.691 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T20:57:22.599434+0000", 2024-03-11T20:57:22.691 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 0.031028014999999999, 2024-03-11T20:57:22.691 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.032185680000000001, 2024-03-11T20:57:22.691 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T20:57:22.691 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "queued for pg", 2024-03-11T20:57:22.692 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T20:57:22.692 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.692 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T20:57:22.692 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.599434+0000", 2024-03-11T20:57:22.692 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:57:22.692 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.692 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.692 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T20:57:22.692 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.599434+0000", 2024-03-11T20:57:22.692 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:57:22.693 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.693 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.693 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T20:57:22.693 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.599434+0000", 2024-03-11T20:57:22.693 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.5300000000000003e-07 2024-03-11T20:57:22.693 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.693 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.693 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T20:57:22.693 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.599441+0000", 2024-03-11T20:57:22.693 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.6490000000000002e-06 2024-03-11T20:57:22.693 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.693 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.693 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T20:57:22.694 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.599465+0000", 2024-03-11T20:57:22.694 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.3938999999999999e-05 2024-03-11T20:57:22.694 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.694 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.694 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T20:57:22.694 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.599472+0000", 2024-03-11T20:57:22.694 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.2060000000000001e-06 2024-03-11T20:57:22.694 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:57:22.694 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T20:57:22.694 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:57:22.694 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.694 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.694 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_repop(client.24629.0:167133 7.c e161/121)", 2024-03-11T20:57:22.694 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T20:57:22.630464+0000", 2024-03-11T20:57:22.695 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 4294967295.9999976, 2024-03-11T20:57:22.695 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.001210359, 2024-03-11T20:57:22.695 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T20:57:22.695 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "queued for pg", 2024-03-11T20:57:22.695 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T20:57:22.695 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.695 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T20:57:22.695 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.630464+0000", 2024-03-11T20:57:22.695 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:57:22.695 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.695 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.695 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T20:57:22.695 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.630464+0000", 2024-03-11T20:57:22.695 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:57:22.696 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.696 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.696 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T20:57:22.696 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.630465+0000", 2024-03-11T20:57:22.696 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 8.1200000000000002e-07 2024-03-11T20:57:22.696 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.696 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.696 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T20:57:22.696 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.630471+0000", 2024-03-11T20:57:22.696 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.1600000000000003e-06 2024-03-11T20:57:22.696 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.696 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.696 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T20:57:22.696 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.630481+0000", 2024-03-11T20:57:22.697 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 9.8260000000000005e-06 2024-03-11T20:57:22.697 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.697 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.697 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T20:57:22.697 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.630488+0000", 2024-03-11T20:57:22.697 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.306e-06 2024-03-11T20:57:22.697 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:57:22.697 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T20:57:22.697 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:57:22.697 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.697 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.697 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_op(client.24629.0:167145 7.2 7.4ddfa1a2 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e161)", 2024-03-11T20:57:22.697 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T20:57:22.630996+0000", 2024-03-11T20:57:22.697 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 4294967295.9994655, 2024-03-11T20:57:22.698 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00075912099999999997, 2024-03-11T20:57:22.698 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T20:57:22.698 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "queued for pg", 2024-03-11T20:57:22.698 INFO:teuthology.orchestra.run.smithi023.stdout: "client_info": { 2024-03-11T20:57:22.698 INFO:teuthology.orchestra.run.smithi023.stdout: "client": "client.24629", 2024-03-11T20:57:22.698 INFO:teuthology.orchestra.run.smithi023.stdout: "client_addr": "172.21.15.23:0/959732609", 2024-03-11T20:57:22.698 INFO:teuthology.orchestra.run.smithi023.stdout: "tid": 167145 2024-03-11T20:57:22.698 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.698 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T20:57:22.698 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.698 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T20:57:22.698 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.630996+0000", 2024-03-11T20:57:22.698 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:57:22.698 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.698 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.699 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T20:57:22.699 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.630996+0000", 2024-03-11T20:57:22.699 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:57:22.699 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.699 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.699 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T20:57:22.699 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.630997+0000", 2024-03-11T20:57:22.699 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 9.4600000000000003e-07 2024-03-11T20:57:22.699 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.699 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.699 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T20:57:22.699 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.631051+0000", 2024-03-11T20:57:22.699 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 5.3517999999999998e-05 2024-03-11T20:57:22.699 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.699 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.700 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T20:57:22.700 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.631064+0000", 2024-03-11T20:57:22.700 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.3713000000000001e-05 2024-03-11T20:57:22.700 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.700 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.700 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T20:57:22.700 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.631074+0000", 2024-03-11T20:57:22.700 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.0169999999999999e-05 2024-03-11T20:57:22.700 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:57:22.700 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T20:57:22.700 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:57:22.700 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.700 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.700 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_repop(client.24629.0:167146 7.7 e161/121 7:e80f6085:::benchmark_data_smithi023_67959_object56905:head v 161'10549, mlcod=161'10546)", 2024-03-11T20:57:22.700 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T20:57:22.631434+0000", 2024-03-11T20:57:22.701 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 4294967295.9990273, 2024-03-11T20:57:22.701 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000394332, 2024-03-11T20:57:22.701 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T20:57:22.701 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "started", 2024-03-11T20:57:22.701 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T20:57:22.701 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.701 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T20:57:22.701 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.631434+0000", 2024-03-11T20:57:22.701 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:57:22.701 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.701 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.701 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T20:57:22.701 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.631434+0000", 2024-03-11T20:57:22.701 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:57:22.701 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.702 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.702 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T20:57:22.702 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.631435+0000", 2024-03-11T20:57:22.702 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.8800000000000002e-07 2024-03-11T20:57:22.702 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.702 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.702 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T20:57:22.702 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.631441+0000", 2024-03-11T20:57:22.702 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.0680000000000002e-06 2024-03-11T20:57:22.702 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.702 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.702 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T20:57:22.702 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.631476+0000", 2024-03-11T20:57:22.702 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 3.4421000000000001e-05 2024-03-11T20:57:22.703 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.703 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.703 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T20:57:22.703 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.631483+0000", 2024-03-11T20:57:22.703 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.7400000000000004e-06 2024-03-11T20:57:22.703 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.703 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.703 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T20:57:22.703 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.631705+0000", 2024-03-11T20:57:22.703 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00022126399999999999 2024-03-11T20:57:22.703 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.703 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.703 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T20:57:22.703 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.631731+0000", 2024-03-11T20:57:22.703 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.5955000000000001e-05 2024-03-11T20:57:22.704 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:57:22.704 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T20:57:22.704 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:57:22.704 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.704 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.704 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_repop(osd.6.0:33183 7.c e161/121)", 2024-03-11T20:57:22.704 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T20:57:22.600820+0000", 2024-03-11T20:57:22.704 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 0.029641574, 2024-03-11T20:57:22.704 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.031076939000000001, 2024-03-11T20:57:22.704 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T20:57:22.704 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "queued for pg", 2024-03-11T20:57:22.704 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T20:57:22.704 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.704 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T20:57:22.705 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.600820+0000", 2024-03-11T20:57:22.705 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:57:22.705 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.705 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.705 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T20:57:22.705 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.600820+0000", 2024-03-11T20:57:22.705 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:57:22.705 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.705 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.705 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T20:57:22.705 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.600821+0000", 2024-03-11T20:57:22.705 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.7700000000000004e-07 2024-03-11T20:57:22.705 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.706 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.706 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T20:57:22.706 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.600827+0000", 2024-03-11T20:57:22.706 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.708e-06 2024-03-11T20:57:22.706 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.706 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.706 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T20:57:22.706 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.600837+0000", 2024-03-11T20:57:22.706 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 9.6369999999999994e-06 2024-03-11T20:57:22.706 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.706 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.706 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T20:57:22.706 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.600843+0000", 2024-03-11T20:57:22.706 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.0859999999999998e-06 2024-03-11T20:57:22.707 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:57:22.707 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T20:57:22.707 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:57:22.707 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.707 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.707 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_repop(osd.6.0:33184 7.c e161/121)", 2024-03-11T20:57:22.707 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T20:57:22.602455+0000", 2024-03-11T20:57:22.707 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 0.028006697000000001, 2024-03-11T20:57:22.707 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.02948866, 2024-03-11T20:57:22.707 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T20:57:22.707 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "queued for pg", 2024-03-11T20:57:22.707 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T20:57:22.707 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.707 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T20:57:22.707 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.602455+0000", 2024-03-11T20:57:22.708 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:57:22.708 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.708 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.708 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T20:57:22.708 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.602455+0000", 2024-03-11T20:57:22.708 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:57:22.708 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.708 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.708 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T20:57:22.708 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.602456+0000", 2024-03-11T20:57:22.708 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.4799999999999998e-07 2024-03-11T20:57:22.708 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.708 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.708 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T20:57:22.709 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.602461+0000", 2024-03-11T20:57:22.709 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 5.7300000000000002e-06 2024-03-11T20:57:22.709 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.709 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.709 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T20:57:22.709 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.602471+0000", 2024-03-11T20:57:22.709 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.0023e-05 2024-03-11T20:57:22.709 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.709 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.709 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T20:57:22.709 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.602478+0000", 2024-03-11T20:57:22.709 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.7469999999999999e-06 2024-03-11T20:57:22.709 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:57:22.709 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T20:57:22.709 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:57:22.710 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.710 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.710 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_repop(osd.6.0:33185 7.c e161/121)", 2024-03-11T20:57:22.710 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T20:57:22.606147+0000", 2024-03-11T20:57:22.710 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 0.024315013, 2024-03-11T20:57:22.710 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.025846043999999999, 2024-03-11T20:57:22.710 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T20:57:22.710 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "queued for pg", 2024-03-11T20:57:22.710 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T20:57:22.710 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.710 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T20:57:22.710 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.606147+0000", 2024-03-11T20:57:22.710 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:57:22.710 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.711 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.711 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T20:57:22.711 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.606147+0000", 2024-03-11T20:57:22.711 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:57:22.711 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.711 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.711 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T20:57:22.711 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.606148+0000", 2024-03-11T20:57:22.711 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 9.9800000000000002e-07 2024-03-11T20:57:22.711 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.711 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.711 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T20:57:22.711 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.606157+0000", 2024-03-11T20:57:22.711 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 8.969e-06 2024-03-11T20:57:22.712 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.712 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.712 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T20:57:22.712 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.607772+0000", 2024-03-11T20:57:22.712 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0016155169999999999 2024-03-11T20:57:22.712 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.712 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.712 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T20:57:22.712 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.607781+0000", 2024-03-11T20:57:22.712 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 8.9469999999999992e-06 2024-03-11T20:57:22.712 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:57:22.712 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T20:57:22.712 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:57:22.712 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.712 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.713 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_repop(osd.6.0:33186 7.c e161/121)", 2024-03-11T20:57:22.713 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T20:57:22.607813+0000", 2024-03-11T20:57:22.713 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 0.022648636, 2024-03-11T20:57:22.713 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.024225103000000001, 2024-03-11T20:57:22.713 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T20:57:22.713 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "queued for pg", 2024-03-11T20:57:22.713 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T20:57:22.713 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.713 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T20:57:22.713 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.607813+0000", 2024-03-11T20:57:22.713 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:57:22.713 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.713 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.713 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T20:57:22.713 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.607813+0000", 2024-03-11T20:57:22.714 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:57:22.714 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.714 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.714 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T20:57:22.714 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.607814+0000", 2024-03-11T20:57:22.714 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.5300000000000004e-07 2024-03-11T20:57:22.714 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.714 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.714 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T20:57:22.714 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.607821+0000", 2024-03-11T20:57:22.714 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.311e-06 2024-03-11T20:57:22.714 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.714 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.715 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T20:57:22.715 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.607831+0000", 2024-03-11T20:57:22.715 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 9.9890000000000004e-06 2024-03-11T20:57:22.715 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.715 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.715 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T20:57:22.715 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.607836+0000", 2024-03-11T20:57:22.715 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 5.485e-06 2024-03-11T20:57:22.715 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:57:22.715 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T20:57:22.715 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:57:22.715 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.715 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.715 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_repop(osd.6.0:33187 7.c e161/121)", 2024-03-11T20:57:22.715 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T20:57:22.608791+0000", 2024-03-11T20:57:22.716 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 0.021670575000000001, 2024-03-11T20:57:22.716 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.023292763000000001, 2024-03-11T20:57:22.716 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T20:57:22.716 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "queued for pg", 2024-03-11T20:57:22.716 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T20:57:22.716 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.716 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T20:57:22.716 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.608791+0000", 2024-03-11T20:57:22.716 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:57:22.716 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.716 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.716 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T20:57:22.716 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.608791+0000", 2024-03-11T20:57:22.716 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:57:22.716 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.717 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.717 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T20:57:22.717 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.608792+0000", 2024-03-11T20:57:22.717 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.68e-07 2024-03-11T20:57:22.717 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.717 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.717 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T20:57:22.717 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.608798+0000", 2024-03-11T20:57:22.717 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.6290000000000004e-06 2024-03-11T20:57:22.717 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.717 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.717 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T20:57:22.717 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.608808+0000", 2024-03-11T20:57:22.717 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 9.9709999999999999e-06 2024-03-11T20:57:22.717 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.718 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.718 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T20:57:22.718 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.608815+0000", 2024-03-11T20:57:22.718 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.5719999999999999e-06 2024-03-11T20:57:22.718 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:57:22.718 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T20:57:22.718 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:57:22.718 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.718 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.718 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_repop(osd.6.0:33188 7.c e161/121)", 2024-03-11T20:57:22.718 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T20:57:22.610381+0000", 2024-03-11T20:57:22.718 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 0.020080858, 2024-03-11T20:57:22.718 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.021758006, 2024-03-11T20:57:22.718 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T20:57:22.719 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "queued for pg", 2024-03-11T20:57:22.719 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T20:57:22.719 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.719 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T20:57:22.719 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.610381+0000", 2024-03-11T20:57:22.719 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:57:22.719 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.719 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.719 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T20:57:22.719 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.610381+0000", 2024-03-11T20:57:22.719 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:57:22.719 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.719 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.719 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T20:57:22.720 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.610382+0000", 2024-03-11T20:57:22.720 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 9.16e-07 2024-03-11T20:57:22.720 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.720 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.720 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T20:57:22.720 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.610390+0000", 2024-03-11T20:57:22.720 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.9699999999999999e-06 2024-03-11T20:57:22.720 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.720 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.720 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T20:57:22.720 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.610401+0000", 2024-03-11T20:57:22.720 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.1684e-05 2024-03-11T20:57:22.720 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.720 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.721 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T20:57:22.721 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.610409+0000", 2024-03-11T20:57:22.721 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.6079999999999998e-06 2024-03-11T20:57:22.721 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:57:22.721 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T20:57:22.721 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:57:22.721 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.721 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.721 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_repop(osd.6.0:33189 7.c e161/121)", 2024-03-11T20:57:22.721 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T20:57:22.612029+0000", 2024-03-11T20:57:22.721 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 0.018432720999999999, 2024-03-11T20:57:22.721 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.020155129000000001, 2024-03-11T20:57:22.721 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T20:57:22.721 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "queued for pg", 2024-03-11T20:57:22.721 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T20:57:22.722 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.722 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T20:57:22.722 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.612029+0000", 2024-03-11T20:57:22.722 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:57:22.722 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.722 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.722 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T20:57:22.722 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.612029+0000", 2024-03-11T20:57:22.722 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:57:22.722 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.722 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.722 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T20:57:22.722 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.612030+0000", 2024-03-11T20:57:22.722 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.2399999999999997e-07 2024-03-11T20:57:22.723 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.723 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.723 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T20:57:22.723 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.612036+0000", 2024-03-11T20:57:22.723 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.4559999999999997e-06 2024-03-11T20:57:22.723 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.723 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.723 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T20:57:22.723 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.612045+0000", 2024-03-11T20:57:22.723 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 9.4339999999999998e-06 2024-03-11T20:57:22.723 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:57:22.723 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:57:22.723 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T20:57:22.723 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:22.612053+0000", 2024-03-11T20:57:22.723 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.0330000000000003e-06 2024-03-11T20:57:22.724 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:57:22.724 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T20:57:22.724 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:57:22.724 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:57:22.724 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-11T20:57:22.724 INFO:teuthology.orchestra.run.smithi023.stdout: "num_ops": 26 2024-03-11T20:57:22.724 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-11T20:57:22.724 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:57:22.725 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:22 smithi023 bash[21372]: cluster 2024-03-11T20:57:20.792672+0000 osd.6 (osd.6) 102 : cluster 0 3.a deep-scrub starts 2024-03-11T20:57:22.725 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:22 smithi023 bash[21372]: cluster 2024-03-11T20:57:20.816269+0000 osd.6 (osd.6) 103 : cluster 0 3.a deep-scrub ok 2024-03-11T20:57:22.725 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:22 smithi023 bash[21372]: cluster 2024-03-11T20:57:21.506460+0000 osd.3 (osd.3) 178 : cluster 0 6.0 deep-scrub starts 2024-03-11T20:57:22.725 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:22 smithi023 bash[21372]: cluster 2024-03-11T20:57:21.562851+0000 osd.3 (osd.3) 179 : cluster 0 7.3 scrub ok 2024-03-11T20:57:22.725 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:22 smithi023 bash[21372]: cluster 2024-03-11T20:57:21.568395+0000 osd.3 (osd.3) 180 : cluster 0 6.0 deep-scrub ok 2024-03-11T20:57:22.725 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:22 smithi023 bash[21372]: cluster 2024-03-11T20:57:21.956681+0000 mgr.y (mgr.24380) 247 : cluster 0 pgmap v237: 148 pgs: 146 active+clean, 2 active+clean+scrubbing+deep; 3.4 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 889 op/s 2024-03-11T20:57:22.726 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:22 smithi023 bash[29146]: cluster 2024-03-11T20:57:20.792672+0000 osd.6 (osd.6) 102 : cluster 0 3.a deep-scrub starts 2024-03-11T20:57:22.726 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:22 smithi023 bash[29146]: cluster 2024-03-11T20:57:20.816269+0000 osd.6 (osd.6) 103 : cluster 0 3.a deep-scrub ok 2024-03-11T20:57:22.726 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:22 smithi023 bash[29146]: cluster 2024-03-11T20:57:21.506460+0000 osd.3 (osd.3) 178 : cluster 0 6.0 deep-scrub starts 2024-03-11T20:57:22.726 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:22 smithi023 bash[29146]: cluster 2024-03-11T20:57:21.562851+0000 osd.3 (osd.3) 179 : cluster 0 7.3 scrub ok 2024-03-11T20:57:22.726 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:22 smithi023 bash[29146]: cluster 2024-03-11T20:57:21.568395+0000 osd.3 (osd.3) 180 : cluster 0 6.0 deep-scrub ok 2024-03-11T20:57:22.726 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:22 smithi023 bash[29146]: cluster 2024-03-11T20:57:21.956681+0000 mgr.y (mgr.24380) 247 : cluster 0 pgmap v237: 148 pgs: 146 active+clean, 2 active+clean+scrubbing+deep; 3.4 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 889 op/s 2024-03-11T20:57:22.759 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:57:22.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:22 smithi146 bash[23809]: cluster 2024-03-11T20:57:20.792672+0000 osd.6 (osd.6) 102 : cluster 0 3.a deep-scrub starts 2024-03-11T20:57:22.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:22 smithi146 bash[23809]: cluster 2024-03-11T20:57:20.816269+0000 osd.6 (osd.6) 103 : cluster 0 3.a deep-scrub ok 2024-03-11T20:57:22.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:22 smithi146 bash[23809]: cluster 2024-03-11T20:57:21.506460+0000 osd.3 (osd.3) 178 : cluster 0 6.0 deep-scrub starts 2024-03-11T20:57:22.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:22 smithi146 bash[23809]: cluster 2024-03-11T20:57:21.562851+0000 osd.3 (osd.3) 179 : cluster 0 7.3 scrub ok 2024-03-11T20:57:22.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:22 smithi146 bash[23809]: cluster 2024-03-11T20:57:21.568395+0000 osd.3 (osd.3) 180 : cluster 0 6.0 deep-scrub ok 2024-03-11T20:57:22.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:22 smithi146 bash[23809]: cluster 2024-03-11T20:57:21.956681+0000 mgr.y (mgr.24380) 247 : cluster 0 pgmap v237: 148 pgs: 146 active+clean, 2 active+clean+scrubbing+deep; 3.4 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 889 op/s 2024-03-11T20:57:22.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:22.766 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:57:22.866 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:57:22.895 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:57:22.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:22.899 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:57:23.000 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:23.022 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:23.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:23.024 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:57:23.125 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:23.135 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:23 smithi023 bash[21372]: debug 2024-03-11T20:57:23.080+0000 7f55248dd700 -1 mon.a@0(leader).osd e161 definitely_dead 0 2024-03-11T20:57:23.158 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:23.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:23.161 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:57:23.262 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:23.284 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:23.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:23.286 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:57:23.387 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:57:23.405 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:57:23.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:23.407 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:57:23.507 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:57:23.535 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:57:23.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:23.537 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:57:23.577 INFO:teuthology.orchestra.run.smithi023.stderr:marked in osd.1. 2024-03-11T20:57:23.639 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:57:23.662 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:57:23.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:23.664 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:57:23.765 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:23.783 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:23.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:23.785 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:57:23.811 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:23 smithi023 bash[21372]: cluster 2024-03-11T20:57:22.450715+0000 osd.3 (osd.3) 181 : cluster 0 7.a scrub starts 2024-03-11T20:57:23.811 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:23 smithi023 bash[21372]: cluster 2024-03-11T20:57:22.545652+0000 osd.5 (osd.5) 144 : cluster 0 3.17 scrub starts 2024-03-11T20:57:23.811 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:23 smithi023 bash[21372]: cluster 2024-03-11T20:57:22.555112+0000 osd.5 (osd.5) 145 : cluster 0 3.17 scrub ok 2024-03-11T20:57:23.811 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:23 smithi023 bash[21372]: audit 2024-03-11T20:57:23.085628+0000 mon.a (mon.0) 1055 : audit 0 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["1"]}]: dispatch 2024-03-11T20:57:23.811 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:23 smithi023 bash[21372]: audit 2024-03-11T20:57:23.089532+0000 mon.b (mon.1) 35 : audit 0 from='client.? 172.21.15.23:0/2108660560' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["1"]}]: dispatch 2024-03-11T20:57:23.811 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:23 smithi023 bash[29146]: cluster 2024-03-11T20:57:22.450715+0000 osd.3 (osd.3) 181 : cluster 0 7.a scrub starts 2024-03-11T20:57:23.812 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:23 smithi023 bash[29146]: cluster 2024-03-11T20:57:22.545652+0000 osd.5 (osd.5) 144 : cluster 0 3.17 scrub starts 2024-03-11T20:57:23.812 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:23 smithi023 bash[29146]: cluster 2024-03-11T20:57:22.555112+0000 osd.5 (osd.5) 145 : cluster 0 3.17 scrub ok 2024-03-11T20:57:23.812 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:23 smithi023 bash[29146]: audit 2024-03-11T20:57:23.085628+0000 mon.a (mon.0) 1055 : audit 0 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["1"]}]: dispatch 2024-03-11T20:57:23.812 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:23 smithi023 bash[29146]: audit 2024-03-11T20:57:23.089532+0000 mon.b (mon.1) 35 : audit 0 from='client.? 172.21.15.23:0/2108660560' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["1"]}]: dispatch 2024-03-11T20:57:23.885 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:57:23.905 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:57:23.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:23.907 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:57:24.008 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:57:24.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:23 smithi146 bash[23809]: cluster 2024-03-11T20:57:22.450715+0000 osd.3 (osd.3) 181 : cluster 0 7.a scrub starts 2024-03-11T20:57:24.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:23 smithi146 bash[23809]: cluster 2024-03-11T20:57:22.545652+0000 osd.5 (osd.5) 144 : cluster 0 3.17 scrub starts 2024-03-11T20:57:24.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:23 smithi146 bash[23809]: cluster 2024-03-11T20:57:22.555112+0000 osd.5 (osd.5) 145 : cluster 0 3.17 scrub ok 2024-03-11T20:57:24.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:23 smithi146 bash[23809]: audit 2024-03-11T20:57:23.085628+0000 mon.a (mon.0) 1055 : audit 0 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["1"]}]: dispatch 2024-03-11T20:57:24.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:23 smithi146 bash[23809]: audit 2024-03-11T20:57:23.089532+0000 mon.b (mon.1) 35 : audit 0 from='client.? 172.21.15.23:0/2108660560' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["1"]}]: dispatch 2024-03-11T20:57:24.028 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:57:24.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:24.029 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:57:24.130 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:57:24.150 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:57:24.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:24.152 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:57:24.253 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:57:24.255 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-03-11T20:57:24.273 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:57:24.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:24.275 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:57:24.375 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:57:24.399 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:57:24.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:24.401 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:57:24.502 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:24.523 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:24.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:24.525 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:57:24.626 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:57:24.647 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:57:24.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:24.649 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:57:24.750 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:57:24.769 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:57:24.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:24.771 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:57:24.871 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:57:24.880 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:24 smithi023 bash[29146]: cluster 2024-03-11T20:57:23.033223+0000 osd.4 (osd.4) 159 : cluster 0 4.13 scrub starts 2024-03-11T20:57:24.880 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:24 smithi023 bash[29146]: cluster 2024-03-11T20:57:23.036247+0000 osd.4 (osd.4) 160 : cluster 0 4.13 scrub ok 2024-03-11T20:57:24.880 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:24 smithi023 bash[29146]: audit 2024-03-11T20:57:23.566733+0000 mon.a (mon.0) 1056 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["1"]}]': finished 2024-03-11T20:57:24.880 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:24 smithi023 bash[29146]: cluster 2024-03-11T20:57:23.570986+0000 mon.a (mon.0) 1057 : cluster 0 osdmap e162: 8 total, 8 up, 8 in 2024-03-11T20:57:24.881 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:24 smithi023 bash[29146]: cluster 2024-03-11T20:57:23.600536+0000 osd.5 (osd.5) 146 : cluster 0 3.1d scrub starts 2024-03-11T20:57:24.881 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:24 smithi023 bash[29146]: cluster 2024-03-11T20:57:23.603956+0000 osd.5 (osd.5) 147 : cluster 0 3.1d scrub ok 2024-03-11T20:57:24.881 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:24 smithi023 bash[29146]: cluster 2024-03-11T20:57:23.821842+0000 osd.3 (osd.3) 182 : cluster 0 7.a scrub ok 2024-03-11T20:57:24.881 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:24 smithi023 bash[29146]: cluster 2024-03-11T20:57:23.957050+0000 mgr.y (mgr.24380) 248 : cluster 0 pgmap v239: 148 pgs: 146 active+clean, 2 active+clean+scrubbing+deep; 3.4 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 889 op/s 2024-03-11T20:57:24.881 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:24 smithi023 bash[21372]: cluster 2024-03-11T20:57:23.033223+0000 osd.4 (osd.4) 159 : cluster 0 4.13 scrub starts 2024-03-11T20:57:24.881 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:24 smithi023 bash[21372]: cluster 2024-03-11T20:57:23.036247+0000 osd.4 (osd.4) 160 : cluster 0 4.13 scrub ok 2024-03-11T20:57:24.881 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:24 smithi023 bash[21372]: audit 2024-03-11T20:57:23.566733+0000 mon.a (mon.0) 1056 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["1"]}]': finished 2024-03-11T20:57:24.881 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:24 smithi023 bash[21372]: cluster 2024-03-11T20:57:23.570986+0000 mon.a (mon.0) 1057 : cluster 0 osdmap e162: 8 total, 8 up, 8 in 2024-03-11T20:57:24.881 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:24 smithi023 bash[21372]: cluster 2024-03-11T20:57:23.600536+0000 osd.5 (osd.5) 146 : cluster 0 3.1d scrub starts 2024-03-11T20:57:24.881 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:24 smithi023 bash[21372]: cluster 2024-03-11T20:57:23.603956+0000 osd.5 (osd.5) 147 : cluster 0 3.1d scrub ok 2024-03-11T20:57:24.881 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:24 smithi023 bash[21372]: cluster 2024-03-11T20:57:23.821842+0000 osd.3 (osd.3) 182 : cluster 0 7.a scrub ok 2024-03-11T20:57:24.881 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:24 smithi023 bash[21372]: cluster 2024-03-11T20:57:23.957050+0000 mgr.y (mgr.24380) 248 : cluster 0 pgmap v239: 148 pgs: 146 active+clean, 2 active+clean+scrubbing+deep; 3.4 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 889 op/s 2024-03-11T20:57:24.890 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:57:24.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:24.892 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:57:24.966 INFO:tasks.thrashosds.thrasher:Added osd 1 2024-03-11T20:57:24.993 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:57:25.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:24 smithi146 bash[23809]: cluster 2024-03-11T20:57:23.033223+0000 osd.4 (osd.4) 159 : cluster 0 4.13 scrub starts 2024-03-11T20:57:25.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:24 smithi146 bash[23809]: cluster 2024-03-11T20:57:23.036247+0000 osd.4 (osd.4) 160 : cluster 0 4.13 scrub ok 2024-03-11T20:57:25.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:24 smithi146 bash[23809]: audit 2024-03-11T20:57:23.566733+0000 mon.a (mon.0) 1056 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["1"]}]': finished 2024-03-11T20:57:25.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:24 smithi146 bash[23809]: cluster 2024-03-11T20:57:23.570986+0000 mon.a (mon.0) 1057 : cluster 0 osdmap e162: 8 total, 8 up, 8 in 2024-03-11T20:57:25.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:24 smithi146 bash[23809]: cluster 2024-03-11T20:57:23.600536+0000 osd.5 (osd.5) 146 : cluster 0 3.1d scrub starts 2024-03-11T20:57:25.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:24 smithi146 bash[23809]: cluster 2024-03-11T20:57:23.603956+0000 osd.5 (osd.5) 147 : cluster 0 3.1d scrub ok 2024-03-11T20:57:25.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:24 smithi146 bash[23809]: cluster 2024-03-11T20:57:23.821842+0000 osd.3 (osd.3) 182 : cluster 0 7.a scrub ok 2024-03-11T20:57:25.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:24 smithi146 bash[23809]: cluster 2024-03-11T20:57:23.957050+0000 mgr.y (mgr.24380) 248 : cluster 0 pgmap v239: 148 pgs: 146 active+clean, 2 active+clean+scrubbing+deep; 3.4 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 889 op/s 2024-03-11T20:57:25.011 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:57:25.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:25.013 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:57:25.113 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:57:25.137 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:57:25.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:25.139 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:57:25.240 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:57:25.267 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:57:25.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:25.269 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:57:25.370 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:57:25.394 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:57:25.395 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:57:25.397 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:57:25.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:25.401 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:57:25.502 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:25.510 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 20:57:25 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T20:57:25.530 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:25.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:25.533 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:57:25.634 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:25.654 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:25.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:25.656 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:57:25.757 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:57:25.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:25 smithi023 bash[21372]: cluster 2024-03-11T20:57:24.022616+0000 osd.4 (osd.4) 161 : cluster 0 3.9 scrub starts 2024-03-11T20:57:25.769 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:25 smithi023 bash[21372]: cluster 2024-03-11T20:57:24.024984+0000 osd.4 (osd.4) 162 : cluster 0 3.9 scrub ok 2024-03-11T20:57:25.769 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:25 smithi023 bash[21372]: cluster 2024-03-11T20:57:24.252150+0000 osd.7 (osd.7) 149 : cluster 0 2.5 scrub starts 2024-03-11T20:57:25.769 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:25 smithi023 bash[21372]: cluster 2024-03-11T20:57:24.253499+0000 osd.7 (osd.7) 150 : cluster 0 2.5 scrub ok 2024-03-11T20:57:25.769 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:25 smithi023 bash[21372]: cluster 2024-03-11T20:57:24.599633+0000 mon.a (mon.0) 1058 : cluster 0 osdmap e163: 8 total, 8 up, 8 in 2024-03-11T20:57:25.769 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:25 smithi023 bash[21372]: cluster 2024-03-11T20:57:24.953701+0000 osd.1 (osd.1) 97 : cluster 0 3.19 scrub starts 2024-03-11T20:57:25.769 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:25 smithi023 bash[21372]: cluster 2024-03-11T20:57:24.955531+0000 osd.1 (osd.1) 98 : cluster 0 3.19 scrub ok 2024-03-11T20:57:25.769 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:25 smithi023 bash[21372]: audit 2024-03-11T20:57:25.125091+0000 mgr.y (mgr.24380) 249 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:57:25.769 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:25 smithi023 bash[29146]: cluster 2024-03-11T20:57:24.022616+0000 osd.4 (osd.4) 161 : cluster 0 3.9 scrub starts 2024-03-11T20:57:25.769 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:25 smithi023 bash[29146]: cluster 2024-03-11T20:57:24.024984+0000 osd.4 (osd.4) 162 : cluster 0 3.9 scrub ok 2024-03-11T20:57:25.769 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:25 smithi023 bash[29146]: cluster 2024-03-11T20:57:24.252150+0000 osd.7 (osd.7) 149 : cluster 0 2.5 scrub starts 2024-03-11T20:57:25.769 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:25 smithi023 bash[29146]: cluster 2024-03-11T20:57:24.253499+0000 osd.7 (osd.7) 150 : cluster 0 2.5 scrub ok 2024-03-11T20:57:25.769 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:25 smithi023 bash[29146]: cluster 2024-03-11T20:57:24.599633+0000 mon.a (mon.0) 1058 : cluster 0 osdmap e163: 8 total, 8 up, 8 in 2024-03-11T20:57:25.770 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:25 smithi023 bash[29146]: cluster 2024-03-11T20:57:24.953701+0000 osd.1 (osd.1) 97 : cluster 0 3.19 scrub starts 2024-03-11T20:57:25.770 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:25 smithi023 bash[29146]: cluster 2024-03-11T20:57:24.955531+0000 osd.1 (osd.1) 98 : cluster 0 3.19 scrub ok 2024-03-11T20:57:25.770 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:25 smithi023 bash[29146]: audit 2024-03-11T20:57:25.125091+0000 mgr.y (mgr.24380) 249 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:57:25.796 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:57:25.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:25.802 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:57:25.903 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:57:25.912 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:25 smithi146 bash[23809]: cluster 2024-03-11T20:57:24.022616+0000 osd.4 (osd.4) 161 : cluster 0 3.9 scrub starts 2024-03-11T20:57:25.913 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:25 smithi146 bash[23809]: cluster 2024-03-11T20:57:24.024984+0000 osd.4 (osd.4) 162 : cluster 0 3.9 scrub ok 2024-03-11T20:57:25.913 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:25 smithi146 bash[23809]: cluster 2024-03-11T20:57:24.252150+0000 osd.7 (osd.7) 149 : cluster 0 2.5 scrub starts 2024-03-11T20:57:25.913 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:25 smithi146 bash[23809]: cluster 2024-03-11T20:57:24.253499+0000 osd.7 (osd.7) 150 : cluster 0 2.5 scrub ok 2024-03-11T20:57:25.913 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:25 smithi146 bash[23809]: cluster 2024-03-11T20:57:24.599633+0000 mon.a (mon.0) 1058 : cluster 0 osdmap e163: 8 total, 8 up, 8 in 2024-03-11T20:57:25.913 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:25 smithi146 bash[23809]: cluster 2024-03-11T20:57:24.953701+0000 osd.1 (osd.1) 97 : cluster 0 3.19 scrub starts 2024-03-11T20:57:25.913 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:25 smithi146 bash[23809]: cluster 2024-03-11T20:57:24.955531+0000 osd.1 (osd.1) 98 : cluster 0 3.19 scrub ok 2024-03-11T20:57:25.913 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:25 smithi146 bash[23809]: audit 2024-03-11T20:57:25.125091+0000 mgr.y (mgr.24380) 249 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:57:25.926 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:57:25.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:25.932 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:57:26.033 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:26.065 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:26.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:26.072 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:57:26.173 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:57:26.203 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:57:26.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:26.207 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:57:26.308 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:57:26.351 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:57:26.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:26.355 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:57:26.455 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:26.502 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:26.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:26.506 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:57:26.606 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:57:26.632 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:57:26.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:26.634 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:57:26.735 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:26.745 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:26 smithi023 bash[21372]: cluster 2024-03-11T20:57:25.492037+0000 osd.2 (osd.2) 119 : cluster 0 2.1 scrub starts 2024-03-11T20:57:26.745 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:26 smithi023 bash[21372]: cluster 2024-03-11T20:57:25.493142+0000 osd.2 (osd.2) 120 : cluster 0 2.1 scrub ok 2024-03-11T20:57:26.745 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:26 smithi023 bash[21372]: cluster 2024-03-11T20:57:25.607429+0000 mon.a (mon.0) 1059 : cluster 0 osdmap e164: 8 total, 8 up, 8 in 2024-03-11T20:57:26.746 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:26 smithi023 bash[21372]: cluster 2024-03-11T20:57:25.615616+0000 osd.2 (osd.2) 121 : cluster 0 7.b starting backfill to osd.1 from (0'0,0'0] MAX to 161'10233 2024-03-11T20:57:26.746 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:26 smithi023 bash[21372]: cluster 2024-03-11T20:57:25.615844+0000 osd.0 (osd.0) 157 : cluster 0 7.9 starting backfill to osd.1 from (0'0,0'0] MAX to 161'10289 2024-03-11T20:57:26.746 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:26 smithi023 bash[21372]: cluster 2024-03-11T20:57:25.616266+0000 osd.5 (osd.5) 148 : cluster 0 7.0 starting backfill to osd.1 from (0'0,0'0] MAX to 161'11837 2024-03-11T20:57:26.746 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:26 smithi023 bash[21372]: cluster 2024-03-11T20:57:25.616338+0000 osd.6 (osd.6) 104 : cluster 0 7.d starting backfill to osd.1 from (0'0,0'0] MAX to 161'10463 2024-03-11T20:57:26.746 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:26 smithi023 bash[21372]: cluster 2024-03-11T20:57:25.617499+0000 osd.0 (osd.0) 158 : cluster 0 7.4 starting backfill to osd.1 from (0'0,0'0] MAX to 161'10475 2024-03-11T20:57:26.746 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:26 smithi023 bash[21372]: cluster 2024-03-11T20:57:25.619347+0000 osd.3 (osd.3) 183 : cluster 0 7.2 starting backfill to osd.1 from (0'0,0'0] MAX to 161'10331 2024-03-11T20:57:26.746 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:26 smithi023 bash[21372]: cluster 2024-03-11T20:57:25.957537+0000 mgr.y (mgr.24380) 250 : cluster 0 pgmap v242: 148 pgs: 1 active+clean+scrubbing+deep, 6 remapped+peering, 141 active+clean; 3.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 511 B/s rd, 0 B/s wr, 798 op/s; 363 B/s, 9 objects/s recovering 2024-03-11T20:57:26.746 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:26 smithi023 bash[29146]: cluster 2024-03-11T20:57:25.492037+0000 osd.2 (osd.2) 119 : cluster 0 2.1 scrub starts 2024-03-11T20:57:26.746 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:26 smithi023 bash[29146]: cluster 2024-03-11T20:57:25.493142+0000 osd.2 (osd.2) 120 : cluster 0 2.1 scrub ok 2024-03-11T20:57:26.747 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:26 smithi023 bash[29146]: cluster 2024-03-11T20:57:25.607429+0000 mon.a (mon.0) 1059 : cluster 0 osdmap e164: 8 total, 8 up, 8 in 2024-03-11T20:57:26.747 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:26 smithi023 bash[29146]: cluster 2024-03-11T20:57:25.615616+0000 osd.2 (osd.2) 121 : cluster 0 7.b starting backfill to osd.1 from (0'0,0'0] MAX to 161'10233 2024-03-11T20:57:26.747 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:26 smithi023 bash[29146]: cluster 2024-03-11T20:57:25.615844+0000 osd.0 (osd.0) 157 : cluster 0 7.9 starting backfill to osd.1 from (0'0,0'0] MAX to 161'10289 2024-03-11T20:57:26.747 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:26 smithi023 bash[29146]: cluster 2024-03-11T20:57:25.616266+0000 osd.5 (osd.5) 148 : cluster 0 7.0 starting backfill to osd.1 from (0'0,0'0] MAX to 161'11837 2024-03-11T20:57:26.747 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:26 smithi023 bash[29146]: cluster 2024-03-11T20:57:25.616338+0000 osd.6 (osd.6) 104 : cluster 0 7.d starting backfill to osd.1 from (0'0,0'0] MAX to 161'10463 2024-03-11T20:57:26.747 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:26 smithi023 bash[29146]: cluster 2024-03-11T20:57:25.617499+0000 osd.0 (osd.0) 158 : cluster 0 7.4 starting backfill to osd.1 from (0'0,0'0] MAX to 161'10475 2024-03-11T20:57:26.747 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:26 smithi023 bash[29146]: cluster 2024-03-11T20:57:25.619347+0000 osd.3 (osd.3) 183 : cluster 0 7.2 starting backfill to osd.1 from (0'0,0'0] MAX to 161'10331 2024-03-11T20:57:26.747 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:26 smithi023 bash[29146]: cluster 2024-03-11T20:57:25.957537+0000 mgr.y (mgr.24380) 250 : cluster 0 pgmap v242: 148 pgs: 1 active+clean+scrubbing+deep, 6 remapped+peering, 141 active+clean; 3.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 511 B/s rd, 0 B/s wr, 798 op/s; 363 B/s, 9 objects/s recovering 2024-03-11T20:57:26.761 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:26.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:26.765 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:57:26.866 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:26.893 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:26.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:26.895 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:57:26.996 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:57:27.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:26 smithi146 bash[23809]: cluster 2024-03-11T20:57:25.492037+0000 osd.2 (osd.2) 119 : cluster 0 2.1 scrub starts 2024-03-11T20:57:27.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:26 smithi146 bash[23809]: cluster 2024-03-11T20:57:25.493142+0000 osd.2 (osd.2) 120 : cluster 0 2.1 scrub ok 2024-03-11T20:57:27.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:26 smithi146 bash[23809]: cluster 2024-03-11T20:57:25.607429+0000 mon.a (mon.0) 1059 : cluster 0 osdmap e164: 8 total, 8 up, 8 in 2024-03-11T20:57:27.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:26 smithi146 bash[23809]: cluster 2024-03-11T20:57:25.615616+0000 osd.2 (osd.2) 121 : cluster 0 7.b starting backfill to osd.1 from (0'0,0'0] MAX to 161'10233 2024-03-11T20:57:27.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:26 smithi146 bash[23809]: cluster 2024-03-11T20:57:25.615844+0000 osd.0 (osd.0) 157 : cluster 0 7.9 starting backfill to osd.1 from (0'0,0'0] MAX to 161'10289 2024-03-11T20:57:27.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:26 smithi146 bash[23809]: cluster 2024-03-11T20:57:25.616266+0000 osd.5 (osd.5) 148 : cluster 0 7.0 starting backfill to osd.1 from (0'0,0'0] MAX to 161'11837 2024-03-11T20:57:27.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:26 smithi146 bash[23809]: cluster 2024-03-11T20:57:25.616338+0000 osd.6 (osd.6) 104 : cluster 0 7.d starting backfill to osd.1 from (0'0,0'0] MAX to 161'10463 2024-03-11T20:57:27.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:26 smithi146 bash[23809]: cluster 2024-03-11T20:57:25.617499+0000 osd.0 (osd.0) 158 : cluster 0 7.4 starting backfill to osd.1 from (0'0,0'0] MAX to 161'10475 2024-03-11T20:57:27.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:26 smithi146 bash[23809]: cluster 2024-03-11T20:57:25.619347+0000 osd.3 (osd.3) 183 : cluster 0 7.2 starting backfill to osd.1 from (0'0,0'0] MAX to 161'10331 2024-03-11T20:57:27.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:26 smithi146 bash[23809]: cluster 2024-03-11T20:57:25.957537+0000 mgr.y (mgr.24380) 250 : cluster 0 pgmap v242: 148 pgs: 1 active+clean+scrubbing+deep, 6 remapped+peering, 141 active+clean; 3.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 511 B/s rd, 0 B/s wr, 798 op/s; 363 B/s, 9 objects/s recovering 2024-03-11T20:57:27.020 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:57:27.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:27.025 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:57:27.126 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:27.156 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:27.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:27.162 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:57:27.263 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:57:27.298 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:57:27.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:27.301 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:57:27.402 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:57:27.424 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:57:27.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:27.426 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:57:27.527 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:57:27.558 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:57:27.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:27.562 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:57:27.663 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:57:27.695 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:57:27.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:27.700 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:57:27.801 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:27.834 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:27.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:27.836 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:57:27.937 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:57:27.959 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:57:27.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:27.963 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:57:28.064 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:57:28.090 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:57:28.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:28.092 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:57:28.193 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:57:28.226 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:57:28.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:28.231 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:57:28.334 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:57:28.365 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:57:28.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:28.370 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:57:28.471 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:57:28.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:28.502 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:57:28.502 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:57:28.534 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:57:28 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:20:57:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T20:57:28.603 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:57:28.625 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:57:28.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:28.627 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:57:28.728 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:57:28.754 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:57:28.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:28.756 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:57:28.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:28 smithi146 bash[23809]: cluster 2024-03-11T20:57:27.287662+0000 osd.5 (osd.5) 149 : cluster 0 purged_snaps scrub starts 2024-03-11T20:57:28.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:28 smithi146 bash[23809]: cluster 2024-03-11T20:57:27.287726+0000 osd.5 (osd.5) 150 : cluster 0 purged_snaps scrub ok 2024-03-11T20:57:28.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:28 smithi146 bash[23809]: cluster 2024-03-11T20:57:27.642182+0000 mon.a (mon.0) 1060 : cluster 0 osdmap e165: 8 total, 8 up, 8 in 2024-03-11T20:57:28.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:28 smithi146 bash[23809]: cluster 2024-03-11T20:57:27.958024+0000 mgr.y (mgr.24380) 251 : cluster 0 pgmap v244: 148 pgs: 1 active+clean+scrubbing+deep, 6 remapped+peering, 141 active+clean; 3.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 550 op/s; 496 B/s, 13 objects/s recovering 2024-03-11T20:57:28.857 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:57:28.893 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:57:28.897 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:28 smithi023 bash[21372]: cluster 2024-03-11T20:57:27.287662+0000 osd.5 (osd.5) 149 : cluster 0 purged_snaps scrub starts 2024-03-11T20:57:28.897 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:28 smithi023 bash[21372]: cluster 2024-03-11T20:57:27.287726+0000 osd.5 (osd.5) 150 : cluster 0 purged_snaps scrub ok 2024-03-11T20:57:28.897 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:28 smithi023 bash[21372]: cluster 2024-03-11T20:57:27.642182+0000 mon.a (mon.0) 1060 : cluster 0 osdmap e165: 8 total, 8 up, 8 in 2024-03-11T20:57:28.898 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:28 smithi023 bash[21372]: cluster 2024-03-11T20:57:27.958024+0000 mgr.y (mgr.24380) 251 : cluster 0 pgmap v244: 148 pgs: 1 active+clean+scrubbing+deep, 6 remapped+peering, 141 active+clean; 3.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 550 op/s; 496 B/s, 13 objects/s recovering 2024-03-11T20:57:28.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:28.898 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:57:28.899 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:28 smithi023 bash[29146]: cluster 2024-03-11T20:57:27.287662+0000 osd.5 (osd.5) 149 : cluster 0 purged_snaps scrub starts 2024-03-11T20:57:28.899 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:28 smithi023 bash[29146]: cluster 2024-03-11T20:57:27.287726+0000 osd.5 (osd.5) 150 : cluster 0 purged_snaps scrub ok 2024-03-11T20:57:28.899 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:28 smithi023 bash[29146]: cluster 2024-03-11T20:57:27.642182+0000 mon.a (mon.0) 1060 : cluster 0 osdmap e165: 8 total, 8 up, 8 in 2024-03-11T20:57:28.899 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:28 smithi023 bash[29146]: cluster 2024-03-11T20:57:27.958024+0000 mgr.y (mgr.24380) 251 : cluster 0 pgmap v244: 148 pgs: 1 active+clean+scrubbing+deep, 6 remapped+peering, 141 active+clean; 3.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 550 op/s; 496 B/s, 13 objects/s recovering 2024-03-11T20:57:29.000 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:29.024 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:29.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:29.026 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:57:29.127 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:57:29.155 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:57:29.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:29.162 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:57:29.263 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:57:29.294 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:57:29.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:29.296 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:57:29.397 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:57:29.431 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:57:29.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:29.437 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:57:29.538 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:57:29.560 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:57:29.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:29.562 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:57:29.662 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:57:29.695 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:57:29.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:29.698 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:57:29.799 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:57:29.837 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:57:29.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:29.844 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:57:29.944 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:29.952 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:29 smithi023 bash[21372]: cluster 2024-03-11T20:57:28.246563+0000 osd.7 (osd.7) 151 : cluster 0 4.1e deep-scrub starts 2024-03-11T20:57:29.953 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:29 smithi023 bash[21372]: cluster 2024-03-11T20:57:28.248145+0000 osd.7 (osd.7) 152 : cluster 0 4.1e deep-scrub ok 2024-03-11T20:57:29.953 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:29 smithi023 bash[21372]: cluster 2024-03-11T20:57:28.599164+0000 osd.6 (osd.6) 105 : cluster 0 7.c deep-scrub ok 2024-03-11T20:57:29.953 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:29 smithi023 bash[29146]: cluster 2024-03-11T20:57:28.246563+0000 osd.7 (osd.7) 151 : cluster 0 4.1e deep-scrub starts 2024-03-11T20:57:29.953 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:29 smithi023 bash[29146]: cluster 2024-03-11T20:57:28.248145+0000 osd.7 (osd.7) 152 : cluster 0 4.1e deep-scrub ok 2024-03-11T20:57:29.953 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:29 smithi023 bash[29146]: cluster 2024-03-11T20:57:28.599164+0000 osd.6 (osd.6) 105 : cluster 0 7.c deep-scrub ok 2024-03-11T20:57:29.967 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 6, 7, 1] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-11T20:57:29.968 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-11T20:57:29.968 INFO:tasks.thrashosds.thrasher:inject_pause on osd.1 2024-03-11T20:57:29.968 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-03-11T20:57:29.968 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-03-11T20:57:29.968 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph daemon osd.1 config set filestore_inject_stall 3 2024-03-11T20:57:29.980 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:29.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:29.983 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:57:30.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:29 smithi146 bash[23809]: cluster 2024-03-11T20:57:28.246563+0000 osd.7 (osd.7) 151 : cluster 0 4.1e deep-scrub starts 2024-03-11T20:57:30.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:29 smithi146 bash[23809]: cluster 2024-03-11T20:57:28.248145+0000 osd.7 (osd.7) 152 : cluster 0 4.1e deep-scrub ok 2024-03-11T20:57:30.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:29 smithi146 bash[23809]: cluster 2024-03-11T20:57:28.599164+0000 osd.6 (osd.6) 105 : cluster 0 7.c deep-scrub ok 2024-03-11T20:57:30.084 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:57:30.156 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:57:30.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:30.162 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:57:30.263 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:57:30.297 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:57:30.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:30.302 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:57:30.403 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:57:30.430 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:57:30.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:30.432 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:57:30.533 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:30.559 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:30.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:30.562 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:57:30.663 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:57:30.688 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:57:30.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:30.689 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:57:30.790 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:57:30.802 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:30 smithi146 bash[23809]: cluster 2024-03-11T20:57:28.837616+0000 osd.6 (osd.6) 106 : cluster 0 2.7 scrub starts 2024-03-11T20:57:30.803 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:30 smithi146 bash[23809]: cluster 2024-03-11T20:57:28.838648+0000 osd.6 (osd.6) 107 : cluster 0 2.7 scrub ok 2024-03-11T20:57:30.803 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:30 smithi146 bash[23809]: cluster 2024-03-11T20:57:29.266993+0000 osd.7 (osd.7) 153 : cluster 0 3.f scrub starts 2024-03-11T20:57:30.803 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:30 smithi146 bash[23809]: cluster 2024-03-11T20:57:29.272938+0000 osd.7 (osd.7) 154 : cluster 0 3.f scrub ok 2024-03-11T20:57:30.803 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:30 smithi146 bash[23809]: cluster 2024-03-11T20:57:29.958769+0000 mgr.y (mgr.24380) 252 : cluster 0 pgmap v245: 148 pgs: 1 active+remapped+backfill_wait, 1 active+clean+scrubbing+deep, 5 remapped+peering, 141 active+clean; 3.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 540 op/s; 3156/156654 objects misplaced (2.015%); 363 B/s, 11 objects/s recovering 2024-03-11T20:57:30.815 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:57:30.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:30.817 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:57:30.918 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:57:30.937 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:57:30.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:30.942 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:57:31.006 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:30 smithi023 bash[21372]: cluster 2024-03-11T20:57:28.837616+0000 osd.6 (osd.6) 106 : cluster 0 2.7 scrub starts 2024-03-11T20:57:31.006 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:30 smithi023 bash[21372]: cluster 2024-03-11T20:57:28.838648+0000 osd.6 (osd.6) 107 : cluster 0 2.7 scrub ok 2024-03-11T20:57:31.006 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:30 smithi023 bash[21372]: cluster 2024-03-11T20:57:29.266993+0000 osd.7 (osd.7) 153 : cluster 0 3.f scrub starts 2024-03-11T20:57:31.007 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:30 smithi023 bash[21372]: cluster 2024-03-11T20:57:29.272938+0000 osd.7 (osd.7) 154 : cluster 0 3.f scrub ok 2024-03-11T20:57:31.007 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:30 smithi023 bash[21372]: cluster 2024-03-11T20:57:29.958769+0000 mgr.y (mgr.24380) 252 : cluster 0 pgmap v245: 148 pgs: 1 active+remapped+backfill_wait, 1 active+clean+scrubbing+deep, 5 remapped+peering, 141 active+clean; 3.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 540 op/s; 3156/156654 objects misplaced (2.015%); 363 B/s, 11 objects/s recovering 2024-03-11T20:57:31.007 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:30 smithi023 bash[29146]: cluster 2024-03-11T20:57:28.837616+0000 osd.6 (osd.6) 106 : cluster 0 2.7 scrub starts 2024-03-11T20:57:31.007 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:30 smithi023 bash[29146]: cluster 2024-03-11T20:57:28.838648+0000 osd.6 (osd.6) 107 : cluster 0 2.7 scrub ok 2024-03-11T20:57:31.007 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:30 smithi023 bash[29146]: cluster 2024-03-11T20:57:29.266993+0000 osd.7 (osd.7) 153 : cluster 0 3.f scrub starts 2024-03-11T20:57:31.007 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:30 smithi023 bash[29146]: cluster 2024-03-11T20:57:29.272938+0000 osd.7 (osd.7) 154 : cluster 0 3.f scrub ok 2024-03-11T20:57:31.007 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:30 smithi023 bash[29146]: cluster 2024-03-11T20:57:29.958769+0000 mgr.y (mgr.24380) 252 : cluster 0 pgmap v245: 148 pgs: 1 active+remapped+backfill_wait, 1 active+clean+scrubbing+deep, 5 remapped+peering, 141 active+clean; 3.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 540 op/s; 3156/156654 objects misplaced (2.015%); 363 B/s, 11 objects/s recovering 2024-03-11T20:57:31.043 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:31.063 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:31.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:31.066 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:57:31.167 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:31.210 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:31.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:31.212 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:57:31.313 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:57:31.337 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:57:31.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:31.344 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:57:31.426 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:57:31.445 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:31.467 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:31.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:31.469 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:57:31.570 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:31.598 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:31.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:31.600 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:57:31.701 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:31.732 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:31.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:31.738 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:57:31.839 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:57:31.872 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:57:31.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:31.875 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:57:31.976 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:31.985 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:31 smithi146 bash[23809]: cluster 2024-03-11T20:57:30.120339+0000 osd.4 (osd.4) 163 : cluster 0 6.1a scrub starts 2024-03-11T20:57:31.985 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:31 smithi146 bash[23809]: cluster 2024-03-11T20:57:30.121557+0000 osd.4 (osd.4) 164 : cluster 0 6.1a scrub ok 2024-03-11T20:57:31.985 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:31 smithi146 bash[23809]: cluster 2024-03-11T20:57:30.873126+0000 osd.3 (osd.3) 184 : cluster 0 purged_snaps scrub starts 2024-03-11T20:57:31.985 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:31 smithi146 bash[23809]: cluster 2024-03-11T20:57:30.873189+0000 osd.3 (osd.3) 185 : cluster 0 purged_snaps scrub ok 2024-03-11T20:57:31.998 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:31.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:32.000 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:57:32.004 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:31 smithi023 bash[21372]: cluster 2024-03-11T20:57:30.120339+0000 osd.4 (osd.4) 163 : cluster 0 6.1a scrub starts 2024-03-11T20:57:32.004 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:31 smithi023 bash[21372]: cluster 2024-03-11T20:57:30.121557+0000 osd.4 (osd.4) 164 : cluster 0 6.1a scrub ok 2024-03-11T20:57:32.004 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:31 smithi023 bash[21372]: cluster 2024-03-11T20:57:30.873126+0000 osd.3 (osd.3) 184 : cluster 0 purged_snaps scrub starts 2024-03-11T20:57:32.004 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:31 smithi023 bash[21372]: cluster 2024-03-11T20:57:30.873189+0000 osd.3 (osd.3) 185 : cluster 0 purged_snaps scrub ok 2024-03-11T20:57:32.004 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:31 smithi023 bash[29146]: cluster 2024-03-11T20:57:30.120339+0000 osd.4 (osd.4) 163 : cluster 0 6.1a scrub starts 2024-03-11T20:57:32.004 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:31 smithi023 bash[29146]: cluster 2024-03-11T20:57:30.121557+0000 osd.4 (osd.4) 164 : cluster 0 6.1a scrub ok 2024-03-11T20:57:32.005 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:31 smithi023 bash[29146]: cluster 2024-03-11T20:57:30.873126+0000 osd.3 (osd.3) 184 : cluster 0 purged_snaps scrub starts 2024-03-11T20:57:32.005 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:31 smithi023 bash[29146]: cluster 2024-03-11T20:57:30.873189+0000 osd.3 (osd.3) 185 : cluster 0 purged_snaps scrub ok 2024-03-11T20:57:32.100 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:57:32.128 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:57:32.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:32.131 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:57:32.231 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:57:32.259 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:57:32.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:32.262 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:57:32.364 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:57:32.391 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:57:32.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:32.393 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:57:32.494 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:32.517 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:32.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:32.519 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:57:32.620 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:57:32.655 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:57:32.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:32.660 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:57:32.762 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:57:32.783 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:57:32.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:32.785 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:57:32.885 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:32.914 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:32.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:32.918 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:57:33.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:32 smithi146 bash[23809]: cluster 2024-03-11T20:57:31.734571+0000 mon.a (mon.0) 1061 : cluster 0 osdmap e166: 8 total, 8 up, 8 in 2024-03-11T20:57:33.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:32 smithi146 bash[23809]: cluster 2024-03-11T20:57:31.959186+0000 mgr.y (mgr.24380) 253 : cluster 0 pgmap v247: 148 pgs: 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait, 142 active+clean; 3.0 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 805 B/s rd, 0 B/s wr, 536 op/s; 17248/148920 objects misplaced (11.582%); 12 MiB/s, 198 objects/s recovering 2024-03-11T20:57:33.019 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:33.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:32 smithi023 bash[21372]: cluster 2024-03-11T20:57:31.734571+0000 mon.a (mon.0) 1061 : cluster 0 osdmap e166: 8 total, 8 up, 8 in 2024-03-11T20:57:33.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:32 smithi023 bash[21372]: cluster 2024-03-11T20:57:31.959186+0000 mgr.y (mgr.24380) 253 : cluster 0 pgmap v247: 148 pgs: 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait, 142 active+clean; 3.0 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 805 B/s rd, 0 B/s wr, 536 op/s; 17248/148920 objects misplaced (11.582%); 12 MiB/s, 198 objects/s recovering 2024-03-11T20:57:33.030 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:32 smithi023 bash[29146]: cluster 2024-03-11T20:57:31.734571+0000 mon.a (mon.0) 1061 : cluster 0 osdmap e166: 8 total, 8 up, 8 in 2024-03-11T20:57:33.030 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:32 smithi023 bash[29146]: cluster 2024-03-11T20:57:31.959186+0000 mgr.y (mgr.24380) 253 : cluster 0 pgmap v247: 148 pgs: 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait, 142 active+clean; 3.0 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 805 B/s rd, 0 B/s wr, 536 op/s; 17248/148920 objects misplaced (11.582%); 12 MiB/s, 198 objects/s recovering 2024-03-11T20:57:33.049 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:33.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:33.055 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:57:33.156 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:57:33.181 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:57:33.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:33.185 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:57:33.285 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:33.311 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:33.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:33.315 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:57:33.416 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:33.451 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:33.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:33.453 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:57:33.554 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:33.573 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:33.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:33.575 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:57:33.676 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:57:33.707 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:57:33.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:33.710 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:57:33.811 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:33.844 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:33.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:33.846 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:57:33.946 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:57:33.976 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:57:33.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:33.982 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:57:34.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:33 smithi146 bash[23809]: cluster 2024-03-11T20:57:32.123228+0000 osd.4 (osd.4) 165 : cluster 0 3.1a deep-scrub starts 2024-03-11T20:57:34.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:33 smithi146 bash[23809]: cluster 2024-03-11T20:57:32.156237+0000 osd.4 (osd.4) 166 : cluster 0 3.1a deep-scrub ok 2024-03-11T20:57:34.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:33 smithi146 bash[23809]: cluster 2024-03-11T20:57:32.750661+0000 mon.a (mon.0) 1062 : cluster 3 Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-11T20:57:34.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:33 smithi023 bash[21372]: cluster 2024-03-11T20:57:32.123228+0000 osd.4 (osd.4) 165 : cluster 0 3.1a deep-scrub starts 2024-03-11T20:57:34.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:33 smithi023 bash[21372]: cluster 2024-03-11T20:57:32.156237+0000 osd.4 (osd.4) 166 : cluster 0 3.1a deep-scrub ok 2024-03-11T20:57:34.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:33 smithi023 bash[21372]: cluster 2024-03-11T20:57:32.750661+0000 mon.a (mon.0) 1062 : cluster 3 Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-11T20:57:34.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:33 smithi023 bash[29146]: cluster 2024-03-11T20:57:32.123228+0000 osd.4 (osd.4) 165 : cluster 0 3.1a deep-scrub starts 2024-03-11T20:57:34.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:33 smithi023 bash[29146]: cluster 2024-03-11T20:57:32.156237+0000 osd.4 (osd.4) 166 : cluster 0 3.1a deep-scrub ok 2024-03-11T20:57:34.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:33 smithi023 bash[29146]: cluster 2024-03-11T20:57:32.750661+0000 mon.a (mon.0) 1062 : cluster 3 Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-11T20:57:34.082 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:57:34.109 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:57:34.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:34.111 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:57:34.212 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:34.236 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:34.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:34.241 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:57:34.341 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:57:34.365 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:57:34.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:34.370 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:57:34.471 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:57:34.495 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:57:34.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:34.497 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:57:34.598 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:34.619 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:34.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:34.621 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:57:34.721 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:57:34.745 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:57:34.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:34.747 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:57:34.832 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:57:34.848 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:57:34.889 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:57:34.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:34.894 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:57:34.995 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:57:35.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:34 smithi146 bash[23809]: cluster 2024-03-11T20:57:33.142626+0000 osd.4 (osd.4) 167 : cluster 0 5.1 scrub starts 2024-03-11T20:57:35.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:34 smithi146 bash[23809]: cluster 2024-03-11T20:57:33.143841+0000 osd.4 (osd.4) 168 : cluster 0 5.1 scrub ok 2024-03-11T20:57:35.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:34 smithi146 bash[23809]: cluster 2024-03-11T20:57:33.280123+0000 osd.7 (osd.7) 155 : cluster 0 2.1d scrub starts 2024-03-11T20:57:35.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:34 smithi146 bash[23809]: cluster 2024-03-11T20:57:33.281295+0000 osd.7 (osd.7) 156 : cluster 0 2.1d scrub ok 2024-03-11T20:57:35.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:34 smithi146 bash[23809]: cluster 2024-03-11T20:57:33.959681+0000 mgr.y (mgr.24380) 254 : cluster 0 pgmap v248: 148 pgs: 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait, 142 active+clean; 3.0 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 426 op/s; 17248/148920 objects misplaced (11.582%); 9.8 MiB/s, 157 objects/s recovering 2024-03-11T20:57:35.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:34 smithi146 bash[23809]: audit 2024-03-11T20:57:34.555525+0000 mon.a (mon.0) 1063 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:57:35.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:34 smithi146 bash[23809]: audit 2024-03-11T20:57:34.556271+0000 mon.a (mon.0) 1064 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:57:35.012 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:57:35.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:35.014 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:57:35.115 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:57:35.145 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:57:35.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:35.147 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:57:35.248 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:57:35.262 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:34 smithi023 bash[21372]: cluster 2024-03-11T20:57:33.142626+0000 osd.4 (osd.4) 167 : cluster 0 5.1 scrub starts 2024-03-11T20:57:35.263 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:34 smithi023 bash[21372]: cluster 2024-03-11T20:57:33.143841+0000 osd.4 (osd.4) 168 : cluster 0 5.1 scrub ok 2024-03-11T20:57:35.263 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:34 smithi023 bash[21372]: cluster 2024-03-11T20:57:33.280123+0000 osd.7 (osd.7) 155 : cluster 0 2.1d scrub starts 2024-03-11T20:57:35.263 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:34 smithi023 bash[21372]: cluster 2024-03-11T20:57:33.281295+0000 osd.7 (osd.7) 156 : cluster 0 2.1d scrub ok 2024-03-11T20:57:35.263 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:34 smithi023 bash[21372]: cluster 2024-03-11T20:57:33.959681+0000 mgr.y (mgr.24380) 254 : cluster 0 pgmap v248: 148 pgs: 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait, 142 active+clean; 3.0 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 426 op/s; 17248/148920 objects misplaced (11.582%); 9.8 MiB/s, 157 objects/s recovering 2024-03-11T20:57:35.263 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:34 smithi023 bash[21372]: audit 2024-03-11T20:57:34.555525+0000 mon.a (mon.0) 1063 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:57:35.263 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:34 smithi023 bash[21372]: audit 2024-03-11T20:57:34.556271+0000 mon.a (mon.0) 1064 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:57:35.263 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:34 smithi023 bash[29146]: cluster 2024-03-11T20:57:33.142626+0000 osd.4 (osd.4) 167 : cluster 0 5.1 scrub starts 2024-03-11T20:57:35.263 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:34 smithi023 bash[29146]: cluster 2024-03-11T20:57:33.143841+0000 osd.4 (osd.4) 168 : cluster 0 5.1 scrub ok 2024-03-11T20:57:35.263 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:34 smithi023 bash[29146]: cluster 2024-03-11T20:57:33.280123+0000 osd.7 (osd.7) 155 : cluster 0 2.1d scrub starts 2024-03-11T20:57:35.263 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:34 smithi023 bash[29146]: cluster 2024-03-11T20:57:33.281295+0000 osd.7 (osd.7) 156 : cluster 0 2.1d scrub ok 2024-03-11T20:57:35.264 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:34 smithi023 bash[29146]: cluster 2024-03-11T20:57:33.959681+0000 mgr.y (mgr.24380) 254 : cluster 0 pgmap v248: 148 pgs: 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait, 142 active+clean; 3.0 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 426 op/s; 17248/148920 objects misplaced (11.582%); 9.8 MiB/s, 157 objects/s recovering 2024-03-11T20:57:35.264 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:34 smithi023 bash[29146]: audit 2024-03-11T20:57:34.555525+0000 mon.a (mon.0) 1063 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:57:35.264 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:34 smithi023 bash[29146]: audit 2024-03-11T20:57:34.556271+0000 mon.a (mon.0) 1064 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:57:35.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:35.298 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:57:35.298 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:57:35.399 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:57:35.408 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 20:57:35 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T20:57:35.426 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:57:35.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:35.431 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:57:35.532 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:57:35.553 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:57:35.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:35.555 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:57:35.656 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:57:35.677 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:57:35.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:35.679 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:57:35.780 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:35.816 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:35.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:35.818 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:57:35.919 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:57:35.929 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:35 smithi023 bash[21372]: cluster 2024-03-11T20:57:34.865382+0000 osd.6 (osd.6) 108 : cluster 0 4.a scrub starts 2024-03-11T20:57:35.929 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:35 smithi023 bash[21372]: cluster 2024-03-11T20:57:34.873783+0000 osd.6 (osd.6) 109 : cluster 0 4.a scrub ok 2024-03-11T20:57:35.929 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:35 smithi023 bash[21372]: audit 2024-03-11T20:57:35.133131+0000 mgr.y (mgr.24380) 255 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:57:35.930 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:35 smithi023 bash[29146]: cluster 2024-03-11T20:57:34.865382+0000 osd.6 (osd.6) 108 : cluster 0 4.a scrub starts 2024-03-11T20:57:35.930 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:35 smithi023 bash[29146]: cluster 2024-03-11T20:57:34.873783+0000 osd.6 (osd.6) 109 : cluster 0 4.a scrub ok 2024-03-11T20:57:35.930 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:35 smithi023 bash[29146]: audit 2024-03-11T20:57:35.133131+0000 mgr.y (mgr.24380) 255 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:57:35.952 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:57:35.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:35.955 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:57:36.056 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:57:36.065 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:35 smithi146 bash[23809]: cluster 2024-03-11T20:57:34.865382+0000 osd.6 (osd.6) 108 : cluster 0 4.a scrub starts 2024-03-11T20:57:36.065 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:35 smithi146 bash[23809]: cluster 2024-03-11T20:57:34.873783+0000 osd.6 (osd.6) 109 : cluster 0 4.a scrub ok 2024-03-11T20:57:36.066 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:35 smithi146 bash[23809]: audit 2024-03-11T20:57:35.133131+0000 mgr.y (mgr.24380) 255 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:57:36.077 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:57:36.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:36.084 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:57:36.185 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:57:36.216 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:57:36.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:36.223 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:57:36.324 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:57:36.354 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:57:36.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:36.360 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:57:36.461 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:57:36.483 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:57:36.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:36.489 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:57:36.589 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:57:36.611 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:57:36.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:36.615 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:57:36.716 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:57:36.747 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:57:36.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:36.752 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:57:36.853 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:36.907 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:36.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:36.911 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:57:37.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:36 smithi146 bash[23809]: cluster 2024-03-11T20:57:35.960634+0000 mgr.y (mgr.24380) 256 : cluster 0 pgmap v249: 148 pgs: 3 active+remapped+backfilling, 1 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 142 active+clean; 2.8 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 832 op/s; 14726/138396 objects misplaced (10.640%); 20 MiB/s, 315 objects/s recovering 2024-03-11T20:57:37.012 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:57:37.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:36 smithi023 bash[21372]: cluster 2024-03-11T20:57:35.960634+0000 mgr.y (mgr.24380) 256 : cluster 0 pgmap v249: 148 pgs: 3 active+remapped+backfilling, 1 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 142 active+clean; 2.8 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 832 op/s; 14726/138396 objects misplaced (10.640%); 20 MiB/s, 315 objects/s recovering 2024-03-11T20:57:37.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:36 smithi023 bash[29146]: cluster 2024-03-11T20:57:35.960634+0000 mgr.y (mgr.24380) 256 : cluster 0 pgmap v249: 148 pgs: 3 active+remapped+backfilling, 1 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 142 active+clean; 2.8 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 832 op/s; 14726/138396 objects misplaced (10.640%); 20 MiB/s, 315 objects/s recovering 2024-03-11T20:57:37.036 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:57:37.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:37.043 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:57:37.144 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:57:37.167 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:57:37.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:37.169 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:57:37.270 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:57:37.293 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:57:37.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:37.297 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:57:37.398 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:57:37.425 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:57:37.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:37.427 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:57:37.528 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:57:37.553 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:57:37.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:37.556 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:57:37.657 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:57:37.681 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:57:37.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:37.685 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:57:37.786 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:57:37.807 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:57:37.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:37.811 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:57:37.899 INFO:teuthology.orchestra.run.smithi023.stdout:osd.0: {} 2024-03-11T20:57:37.912 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:57:37.934 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:57:37.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:37.938 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:57:37.947 INFO:teuthology.orchestra.run.smithi023.stdout:osd.1: {} 2024-03-11T20:57:37.988 INFO:teuthology.orchestra.run.smithi023.stdout:osd.2: {} 2024-03-11T20:57:38.029 INFO:teuthology.orchestra.run.smithi023.stdout:osd.3: {} 2024-03-11T20:57:38.038 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:57:38.067 INFO:teuthology.orchestra.run.smithi023.stdout:osd.4: {} 2024-03-11T20:57:38.068 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:57:38.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:38.072 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:57:38.106 INFO:teuthology.orchestra.run.smithi023.stdout:osd.5: {} 2024-03-11T20:57:38.146 INFO:teuthology.orchestra.run.smithi023.stdout:osd.6: {} 2024-03-11T20:57:38.173 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:38.181 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:37 smithi146 bash[23809]: audit 2024-03-11T20:57:37.866130+0000 mon.c (mon.2) 36 : audit 0 from='client.? 172.21.15.23:0/3715710326' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T20:57:38.186 INFO:teuthology.orchestra.run.smithi023.stdout:osd.7: {} 2024-03-11T20:57:38.195 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:37 smithi023 bash[21372]: audit 2024-03-11T20:57:37.866130+0000 mon.c (mon.2) 36 : audit 0 from='client.? 172.21.15.23:0/3715710326' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T20:57:38.196 INFO:teuthology.orchestra.run.smithi023.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T20:57:38.196 INFO:teuthology.orchestra.run.smithi023.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T20:57:38.196 INFO:teuthology.orchestra.run.smithi023.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T20:57:38.196 INFO:teuthology.orchestra.run.smithi023.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T20:57:38.196 INFO:teuthology.orchestra.run.smithi023.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T20:57:38.196 INFO:teuthology.orchestra.run.smithi023.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T20:57:38.196 INFO:teuthology.orchestra.run.smithi023.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T20:57:38.196 INFO:teuthology.orchestra.run.smithi023.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T20:57:38.198 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:38.199 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:37 smithi023 bash[29146]: audit 2024-03-11T20:57:37.866130+0000 mon.c (mon.2) 36 : audit 0 from='client.? 172.21.15.23:0/3715710326' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T20:57:38.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:38.200 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:57:38.300 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:57:38.347 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:57:38.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:38.351 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:57:38.451 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:57:38.461 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:57:38 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:20:57:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T20:57:38.471 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:57:38.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:38.473 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:57:38.574 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:57:38.602 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:57:38.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:38.604 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:57:38.705 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:57:38.743 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:57:38.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:38.746 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:57:38.847 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:38.870 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:38.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:38.874 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:57:38.975 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:38.996 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:38.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:38.998 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:57:39.099 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:57:39.108 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:38 smithi146 bash[23809]: cluster 2024-03-11T20:57:37.156968+0000 osd.4 (osd.4) 169 : cluster 0 2.f scrub starts 2024-03-11T20:57:39.108 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:38 smithi146 bash[23809]: cluster 2024-03-11T20:57:37.161069+0000 osd.4 (osd.4) 170 : cluster 0 2.f scrub ok 2024-03-11T20:57:39.108 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:38 smithi146 bash[23809]: cluster 2024-03-11T20:57:37.961142+0000 mgr.y (mgr.24380) 257 : cluster 0 pgmap v250: 148 pgs: 3 active+remapped+backfilling, 1 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 142 active+clean; 2.8 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 692 op/s; 14726/138396 objects misplaced (10.640%); 16 MiB/s, 262 objects/s recovering 2024-03-11T20:57:39.127 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:57:39.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:39.131 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:57:39.231 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:57:39.254 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:57:39.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:39.256 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:57:39.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:38 smithi023 bash[21372]: cluster 2024-03-11T20:57:37.156968+0000 osd.4 (osd.4) 169 : cluster 0 2.f scrub starts 2024-03-11T20:57:39.284 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:38 smithi023 bash[21372]: cluster 2024-03-11T20:57:37.161069+0000 osd.4 (osd.4) 170 : cluster 0 2.f scrub ok 2024-03-11T20:57:39.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:38 smithi023 bash[21372]: cluster 2024-03-11T20:57:37.961142+0000 mgr.y (mgr.24380) 257 : cluster 0 pgmap v250: 148 pgs: 3 active+remapped+backfilling, 1 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 142 active+clean; 2.8 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 692 op/s; 14726/138396 objects misplaced (10.640%); 16 MiB/s, 262 objects/s recovering 2024-03-11T20:57:39.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:38 smithi023 bash[29146]: cluster 2024-03-11T20:57:37.156968+0000 osd.4 (osd.4) 169 : cluster 0 2.f scrub starts 2024-03-11T20:57:39.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:38 smithi023 bash[29146]: cluster 2024-03-11T20:57:37.161069+0000 osd.4 (osd.4) 170 : cluster 0 2.f scrub ok 2024-03-11T20:57:39.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:38 smithi023 bash[29146]: cluster 2024-03-11T20:57:37.961142+0000 mgr.y (mgr.24380) 257 : cluster 0 pgmap v250: 148 pgs: 3 active+remapped+backfilling, 1 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 142 active+clean; 2.8 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 692 op/s; 14726/138396 objects misplaced (10.640%); 16 MiB/s, 262 objects/s recovering 2024-03-11T20:57:39.357 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:57:39.398 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:57:39.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:39.401 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:57:39.502 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:57:39.531 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:57:39.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:39.535 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:57:39.636 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:39.659 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:39.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:39.661 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:57:39.762 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:57:39.788 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:57:39.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:39.789 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:57:39.890 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:39.907 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:39.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:39.909 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:57:39.939 INFO:teuthology.orchestra.run.smithi023.stderr:noscrub is set 2024-03-11T20:57:40.011 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:57:40.035 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:57:40.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:40.037 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:57:40.138 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:40.165 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:40.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:40.168 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:57:40.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:39 smithi146 bash[23809]: audit 2024-03-11T20:57:39.055243+0000 mon.c (mon.2) 37 : audit 1 from='client.? 172.21.15.23:0/436109217' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T20:57:40.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:39 smithi146 bash[23809]: audit 2024-03-11T20:57:39.056263+0000 mon.a (mon.0) 1065 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T20:57:40.269 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:57:40.279 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:40 smithi023 bash[21372]: audit 2024-03-11T20:57:39.055243+0000 mon.c (mon.2) 37 : audit 1 from='client.? 172.21.15.23:0/436109217' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T20:57:40.279 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:40 smithi023 bash[21372]: audit 2024-03-11T20:57:39.056263+0000 mon.a (mon.0) 1065 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T20:57:40.279 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:40 smithi023 bash[29146]: audit 2024-03-11T20:57:39.055243+0000 mon.c (mon.2) 37 : audit 1 from='client.? 172.21.15.23:0/436109217' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T20:57:40.279 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:40 smithi023 bash[29146]: audit 2024-03-11T20:57:39.056263+0000 mon.a (mon.0) 1065 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T20:57:40.298 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:57:40.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:40.301 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:57:40.402 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:57:40.441 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:57:40.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:40.445 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:57:40.546 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:40.581 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:40.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:40.584 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:57:40.685 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:40.707 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:40.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:40.709 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:57:40.810 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:40.845 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:40.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:40.848 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:57:40.949 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:40.972 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:40.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:40.976 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:57:41.077 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:57:41.111 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:57:41.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:41.116 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:57:41.216 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:41.225 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:40 smithi023 bash[21372]: cluster 2024-03-11T20:57:39.865135+0000 osd.6 (osd.6) 110 : cluster 0 2.11 deep-scrub starts 2024-03-11T20:57:41.225 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:40 smithi023 bash[21372]: cluster 2024-03-11T20:57:39.868771+0000 osd.6 (osd.6) 111 : cluster 0 2.11 deep-scrub ok 2024-03-11T20:57:41.226 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:40 smithi023 bash[21372]: cluster 2024-03-11T20:57:39.932040+0000 mon.a (mon.0) 1066 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T20:57:41.226 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:40 smithi023 bash[21372]: audit 2024-03-11T20:57:39.933656+0000 mon.a (mon.0) 1067 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T20:57:41.226 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:40 smithi023 bash[21372]: cluster 2024-03-11T20:57:39.940148+0000 mon.a (mon.0) 1068 : cluster 0 osdmap e167: 8 total, 8 up, 8 in 2024-03-11T20:57:41.226 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:40 smithi023 bash[21372]: cluster 2024-03-11T20:57:39.961527+0000 mgr.y (mgr.24380) 258 : cluster 0 pgmap v252: 148 pgs: 3 active+remapped+backfilling, 1 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 142 active+clean; 2.7 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 622 B/s rd, 0 B/s wr, 542 op/s; 14483/135543 objects misplaced (10.685%); 10 MiB/s, 165 objects/s recovering 2024-03-11T20:57:41.226 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:40 smithi023 bash[29146]: cluster 2024-03-11T20:57:39.865135+0000 osd.6 (osd.6) 110 : cluster 0 2.11 deep-scrub starts 2024-03-11T20:57:41.226 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:40 smithi023 bash[29146]: cluster 2024-03-11T20:57:39.868771+0000 osd.6 (osd.6) 111 : cluster 0 2.11 deep-scrub ok 2024-03-11T20:57:41.226 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:40 smithi023 bash[29146]: cluster 2024-03-11T20:57:39.932040+0000 mon.a (mon.0) 1066 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T20:57:41.226 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:40 smithi023 bash[29146]: audit 2024-03-11T20:57:39.933656+0000 mon.a (mon.0) 1067 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T20:57:41.226 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:40 smithi023 bash[29146]: cluster 2024-03-11T20:57:39.940148+0000 mon.a (mon.0) 1068 : cluster 0 osdmap e167: 8 total, 8 up, 8 in 2024-03-11T20:57:41.226 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:40 smithi023 bash[29146]: cluster 2024-03-11T20:57:39.961527+0000 mgr.y (mgr.24380) 258 : cluster 0 pgmap v252: 148 pgs: 3 active+remapped+backfilling, 1 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 142 active+clean; 2.7 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 622 B/s rd, 0 B/s wr, 542 op/s; 14483/135543 objects misplaced (10.685%); 10 MiB/s, 165 objects/s recovering 2024-03-11T20:57:41.257 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:41.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:40 smithi146 bash[23809]: cluster 2024-03-11T20:57:39.865135+0000 osd.6 (osd.6) 110 : cluster 0 2.11 deep-scrub starts 2024-03-11T20:57:41.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:40 smithi146 bash[23809]: cluster 2024-03-11T20:57:39.868771+0000 osd.6 (osd.6) 111 : cluster 0 2.11 deep-scrub ok 2024-03-11T20:57:41.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:40 smithi146 bash[23809]: cluster 2024-03-11T20:57:39.932040+0000 mon.a (mon.0) 1066 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T20:57:41.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:40 smithi146 bash[23809]: audit 2024-03-11T20:57:39.933656+0000 mon.a (mon.0) 1067 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T20:57:41.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:40 smithi146 bash[23809]: cluster 2024-03-11T20:57:39.940148+0000 mon.a (mon.0) 1068 : cluster 0 osdmap e167: 8 total, 8 up, 8 in 2024-03-11T20:57:41.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:40 smithi146 bash[23809]: cluster 2024-03-11T20:57:39.961527+0000 mgr.y (mgr.24380) 258 : cluster 0 pgmap v252: 148 pgs: 3 active+remapped+backfilling, 1 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 142 active+clean; 2.7 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 622 B/s rd, 0 B/s wr, 542 op/s; 14483/135543 objects misplaced (10.685%); 10 MiB/s, 165 objects/s recovering 2024-03-11T20:57:41.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:41.263 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:57:41.364 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:57:41.404 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:57:41.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:41.410 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:57:41.510 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:57:41.533 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:57:41.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:41.538 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:57:41.639 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:57:41.665 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:57:41.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:41.668 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:57:41.769 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:41.791 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:41.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:41.794 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:57:41.894 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:57:41.918 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:57:41.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:41.920 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:57:42.021 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:42.040 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:42.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:42.042 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:57:42.143 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:57:42.181 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:57:42.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:42.188 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:57:42.289 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:42.316 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:42.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:42.322 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:57:42.423 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:57:42.441 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:57:42.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:42.443 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:57:42.544 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:57:42.566 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:57:42.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:42.567 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:57:42.669 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:57:42.692 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:57:42.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:42.694 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:57:42.795 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:57:42.831 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:57:42.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:42.834 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:57:42.935 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:42.968 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:42.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:42.974 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:57:43.077 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:57:43.107 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:57:43.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:43.110 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:57:43.211 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:57:43.221 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:43 smithi023 bash[21372]: cluster 2024-03-11T20:57:41.962138+0000 mgr.y (mgr.24380) 259 : cluster 0 pgmap v253: 148 pgs: 3 active+remapped+backfilling, 1 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 142 active+clean; 2.6 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 730 op/s; 12250/127023 objects misplaced (9.644%); 17 MiB/s, 268 objects/s recovering 2024-03-11T20:57:43.221 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:43 smithi023 bash[29146]: cluster 2024-03-11T20:57:41.962138+0000 mgr.y (mgr.24380) 259 : cluster 0 pgmap v253: 148 pgs: 3 active+remapped+backfilling, 1 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 142 active+clean; 2.6 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 730 op/s; 12250/127023 objects misplaced (9.644%); 17 MiB/s, 268 objects/s recovering 2024-03-11T20:57:43.248 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:57:43.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:43.257 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:57:43.358 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:57:43.380 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:57:43.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:43.382 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:57:43.483 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:43.491 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:43 smithi146 bash[23809]: cluster 2024-03-11T20:57:41.962138+0000 mgr.y (mgr.24380) 259 : cluster 0 pgmap v253: 148 pgs: 3 active+remapped+backfilling, 1 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 142 active+clean; 2.6 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 730 op/s; 12250/127023 objects misplaced (9.644%); 17 MiB/s, 268 objects/s recovering 2024-03-11T20:57:43.512 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:43.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:43.516 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:57:43.617 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:43.640 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:43.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:43.642 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:57:43.742 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:57:43.786 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:57:43.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:43.788 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:57:43.889 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:57:43.916 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:57:43.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:43.918 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:57:44.019 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:57:44.043 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:57:44.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:44.046 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:57:44.146 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:57:44.176 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:57:44.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:44.184 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:57:44.285 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:44.322 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:44.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:44.325 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:57:44.426 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:57:44.445 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:57:44.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:44.447 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:57:44.547 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:44.574 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:44.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:44.576 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:57:44.677 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:44.701 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:44.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:44.707 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:57:44.807 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:57:44.835 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:57:44.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:44.839 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:57:44.939 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:57:44.972 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:57:44.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:44.977 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:57:45.078 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:57:45.107 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:57:45.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:45.111 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:57:45.212 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:57:45.241 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:57:45.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:45.243 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:57:45.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:45 smithi023 bash[21372]: cluster 2024-03-11T20:57:43.962748+0000 mgr.y (mgr.24380) 260 : cluster 0 pgmap v254: 148 pgs: 3 active+remapped+backfilling, 1 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 142 active+clean; 2.6 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 730 op/s; 12250/127023 objects misplaced (9.644%); 17 MiB/s, 268 objects/s recovering 2024-03-11T20:57:45.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:45 smithi023 bash[29146]: cluster 2024-03-11T20:57:43.962748+0000 mgr.y (mgr.24380) 260 : cluster 0 pgmap v254: 148 pgs: 3 active+remapped+backfilling, 1 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 142 active+clean; 2.6 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 730 op/s; 12250/127023 objects misplaced (9.644%); 17 MiB/s, 268 objects/s recovering 2024-03-11T20:57:45.344 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:57:45.380 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:57:45.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:45.382 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:57:45.483 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:57:45.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:45 smithi146 bash[23809]: cluster 2024-03-11T20:57:43.962748+0000 mgr.y (mgr.24380) 260 : cluster 0 pgmap v254: 148 pgs: 3 active+remapped+backfilling, 1 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 142 active+clean; 2.6 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 730 op/s; 12250/127023 objects misplaced (9.644%); 17 MiB/s, 268 objects/s recovering 2024-03-11T20:57:45.511 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 20:57:45 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T20:57:45.516 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:57:45.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:45.517 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:57:45.525 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T20:57:45.617 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:57:45.669 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:57:45.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:45.671 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:57:45.772 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:57:45.812 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:57:45.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:45.815 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:57:45.916 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:45.935 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:45.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:45.937 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:57:46.038 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:57:46.060 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:57:46.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:46.062 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:57:46.163 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:57:46.189 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:57:46.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:46.191 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:57:46.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:46 smithi023 bash[21372]: audit 2024-03-11T20:57:45.140995+0000 mgr.y (mgr.24380) 261 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:57:46.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:46 smithi023 bash[29146]: audit 2024-03-11T20:57:45.140995+0000 mgr.y (mgr.24380) 261 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:57:46.291 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:57:46.324 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:57:46.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:46.334 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:57:46.435 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:46.475 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:46.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:46.477 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:57:46.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:46 smithi146 bash[23809]: audit 2024-03-11T20:57:45.140995+0000 mgr.y (mgr.24380) 261 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:57:46.578 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:46.603 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:46.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:46.607 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:57:46.708 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:57:46.738 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:57:46.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:46.740 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:57:46.841 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:46.874 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:46.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:46.877 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:57:46.978 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:57:47.022 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:57:47.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:47.028 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:57:47.129 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:57:47.138 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:47 smithi023 bash[21372]: cluster 2024-03-11T20:57:45.963645+0000 mgr.y (mgr.24380) 262 : cluster 0 pgmap v255: 148 pgs: 3 active+remapped+backfilling, 1 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 142 active+clean; 2.4 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 704 op/s; 10097/117273 objects misplaced (8.610%); 16 MiB/s, 259 objects/s recovering 2024-03-11T20:57:47.139 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:47 smithi023 bash[29146]: cluster 2024-03-11T20:57:45.963645+0000 mgr.y (mgr.24380) 262 : cluster 0 pgmap v255: 148 pgs: 3 active+remapped+backfilling, 1 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 142 active+clean; 2.4 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 704 op/s; 10097/117273 objects misplaced (8.610%); 16 MiB/s, 259 objects/s recovering 2024-03-11T20:57:47.162 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:57:47.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:47.164 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:57:47.265 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:57:47.287 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:57:47.290 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:47 smithi146 bash[23809]: cluster 2024-03-11T20:57:45.963645+0000 mgr.y (mgr.24380) 262 : cluster 0 pgmap v255: 148 pgs: 3 active+remapped+backfilling, 1 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 142 active+clean; 2.4 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 704 op/s; 10097/117273 objects misplaced (8.610%); 16 MiB/s, 259 objects/s recovering 2024-03-11T20:57:47.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:47.292 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:57:47.393 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:57:47.411 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:57:47.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:47.413 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:57:47.514 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:57:47.536 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:57:47.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:47.539 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:57:47.640 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:57:47.675 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:57:47.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:47.679 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:57:47.780 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:57:47.808 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:57:47.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:47.810 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:57:47.911 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:57:47.936 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:57:47.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:47.940 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:57:48.041 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:48.065 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:48.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:48.067 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:57:48.168 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:57:48.178 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:48 smithi146 bash[23809]: audit 2024-03-11T20:57:47.898453+0000 mon.a (mon.0) 1069 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:57:48.191 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:57:48.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:48.192 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:57:48.293 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:48.338 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:48.343 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:48 smithi023 bash[21372]: audit 2024-03-11T20:57:47.898453+0000 mon.a (mon.0) 1069 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:57:48.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:48.345 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:57:48.345 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:48 smithi023 bash[29146]: audit 2024-03-11T20:57:47.898453+0000 mon.a (mon.0) 1069 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:57:48.446 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:48.461 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-11T20:57:48.489 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:48.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:48.493 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:57:48.594 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:48.616 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:48.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:48.618 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:57:48.628 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:57:48 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:20:57:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T20:57:48.688 INFO:teuthology.orchestra.run.smithi023.stdout:ERROR: (22) Invalid argument 2024-03-11T20:57:48.689 INFO:teuthology.orchestra.run.smithi023.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T20:57:48.719 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:57:48.765 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:57:48.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:48.771 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:57:48.872 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:48.904 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-11T20:57:48.904 INFO:teuthology.orchestra.run.smithi023.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-03-11T20:57:48.904 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-11T20:57:48.924 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:48.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:48.929 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:57:49.030 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:57:49.055 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:57:49.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:49.057 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:57:49.158 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:49.193 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:49.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:49.197 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:57:49.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:49 smithi023 bash[21372]: cluster 2024-03-11T20:57:47.967199+0000 mgr.y (mgr.24380) 263 : cluster 0 pgmap v256: 148 pgs: 3 active+remapped+backfilling, 1 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 142 active+clean; 2.4 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 704 op/s; 10097/117273 objects misplaced (8.610%); 16 MiB/s, 259 objects/s recovering 2024-03-11T20:57:49.286 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:49 smithi023 bash[21372]: audit 2024-03-11T20:57:48.896005+0000 mon.a (mon.0) 1070 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:57:49.286 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:49 smithi023 bash[21372]: audit 2024-03-11T20:57:48.897398+0000 mon.a (mon.0) 1071 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:57:49.286 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:49 smithi023 bash[21372]: audit 2024-03-11T20:57:48.919659+0000 mon.a (mon.0) 1072 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:57:49.287 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:49 smithi023 bash[29146]: cluster 2024-03-11T20:57:47.967199+0000 mgr.y (mgr.24380) 263 : cluster 0 pgmap v256: 148 pgs: 3 active+remapped+backfilling, 1 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 142 active+clean; 2.4 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 704 op/s; 10097/117273 objects misplaced (8.610%); 16 MiB/s, 259 objects/s recovering 2024-03-11T20:57:49.287 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:49 smithi023 bash[29146]: audit 2024-03-11T20:57:48.896005+0000 mon.a (mon.0) 1070 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:57:49.287 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:49 smithi023 bash[29146]: audit 2024-03-11T20:57:48.897398+0000 mon.a (mon.0) 1071 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:57:49.287 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:49 smithi023 bash[29146]: audit 2024-03-11T20:57:48.919659+0000 mon.a (mon.0) 1072 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:57:49.298 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:57:49.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:49.332 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:57:49.332 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:57:49.433 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:49.472 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:49.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:49.479 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:57:49.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:49 smithi146 bash[23809]: cluster 2024-03-11T20:57:47.967199+0000 mgr.y (mgr.24380) 263 : cluster 0 pgmap v256: 148 pgs: 3 active+remapped+backfilling, 1 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 142 active+clean; 2.4 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 704 op/s; 10097/117273 objects misplaced (8.610%); 16 MiB/s, 259 objects/s recovering 2024-03-11T20:57:49.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:49 smithi146 bash[23809]: audit 2024-03-11T20:57:48.896005+0000 mon.a (mon.0) 1070 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:57:49.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:49 smithi146 bash[23809]: audit 2024-03-11T20:57:48.897398+0000 mon.a (mon.0) 1071 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:57:49.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:49 smithi146 bash[23809]: audit 2024-03-11T20:57:48.919659+0000 mon.a (mon.0) 1072 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:57:49.579 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:49.615 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:49.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:49.618 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:57:49.719 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:57:49.747 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:57:49.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:49.749 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:57:49.850 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:57:49.873 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:57:49.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:49.877 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:57:49.978 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:57:49.998 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:57:49.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:50.000 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:57:50.101 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:50.124 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:50.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:50.126 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:57:50.227 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:57:50.252 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:57:50.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:50.255 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:57:50.356 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:57:50.381 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:57:50.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:50.384 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:57:50.485 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:57:50.507 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:57:50.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:50.511 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:57:50.613 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:50.635 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:50.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:50.637 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:57:50.738 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:50.747 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:50 smithi146 bash[23809]: audit 2024-03-11T20:57:49.583053+0000 mon.a (mon.0) 1073 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:57:50.748 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:50 smithi146 bash[23809]: audit 2024-03-11T20:57:49.584685+0000 mon.a (mon.0) 1074 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:57:50.748 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:50 smithi146 bash[23809]: cluster 2024-03-11T20:57:49.967980+0000 mgr.y (mgr.24380) 264 : cluster 0 pgmap v257: 148 pgs: 3 active+remapped+backfilling, 1 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 142 active+clean; 2.3 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1020 B/s rd, 0 B/s wr, 692 op/s; 9896/114741 objects misplaced (8.625%); 16 MiB/s, 258 objects/s recovering 2024-03-11T20:57:50.767 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:50.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:50.769 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:57:50.870 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:57:50.894 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:57:50.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:50.896 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:57:50.996 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:57:51.019 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:57:51.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:51.022 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:57:51.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:50 smithi023 bash[21372]: audit 2024-03-11T20:57:49.583053+0000 mon.a (mon.0) 1073 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:57:51.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:50 smithi023 bash[21372]: audit 2024-03-11T20:57:49.584685+0000 mon.a (mon.0) 1074 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:57:51.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:50 smithi023 bash[21372]: cluster 2024-03-11T20:57:49.967980+0000 mgr.y (mgr.24380) 264 : cluster 0 pgmap v257: 148 pgs: 3 active+remapped+backfilling, 1 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 142 active+clean; 2.3 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1020 B/s rd, 0 B/s wr, 692 op/s; 9896/114741 objects misplaced (8.625%); 16 MiB/s, 258 objects/s recovering 2024-03-11T20:57:51.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:50 smithi023 bash[29146]: audit 2024-03-11T20:57:49.583053+0000 mon.a (mon.0) 1073 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:57:51.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:50 smithi023 bash[29146]: audit 2024-03-11T20:57:49.584685+0000 mon.a (mon.0) 1074 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:57:51.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:50 smithi023 bash[29146]: cluster 2024-03-11T20:57:49.967980+0000 mgr.y (mgr.24380) 264 : cluster 0 pgmap v257: 148 pgs: 3 active+remapped+backfilling, 1 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 142 active+clean; 2.3 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1020 B/s rd, 0 B/s wr, 692 op/s; 9896/114741 objects misplaced (8.625%); 16 MiB/s, 258 objects/s recovering 2024-03-11T20:57:51.123 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:51.145 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:51.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:51.148 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:57:51.249 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:57:51.285 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T20:57:51.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:51.289 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T20:57:51.390 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:57:51.412 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:57:51.414 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-03-11T20:57:51.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:51.415 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:57:51.516 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:57:51.541 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:57:51.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:51.543 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:57:51.644 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:57:51.651 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:51 smithi146 bash[23809]: cluster 2024-03-11T20:57:49.902429+0000 osd.0 (osd.0) 159 : cluster 0 purged_snaps scrub starts 2024-03-11T20:57:51.651 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:51 smithi146 bash[23809]: cluster 2024-03-11T20:57:49.902606+0000 osd.0 (osd.0) 160 : cluster 0 purged_snaps scrub ok 2024-03-11T20:57:51.651 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:51 smithi146 bash[23809]: cluster 2024-03-11T20:57:50.720682+0000 mon.a (mon.0) 1075 : cluster 0 osdmap e168: 8 total, 8 up, 8 in 2024-03-11T20:57:51.668 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:57:51.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:51.670 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:57:51.771 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:57:51.794 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:57:51.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:51.798 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:57:51.899 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:51.910 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:51 smithi023 bash[21372]: cluster 2024-03-11T20:57:49.902429+0000 osd.0 (osd.0) 159 : cluster 0 purged_snaps scrub starts 2024-03-11T20:57:51.910 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:51 smithi023 bash[21372]: cluster 2024-03-11T20:57:49.902606+0000 osd.0 (osd.0) 160 : cluster 0 purged_snaps scrub ok 2024-03-11T20:57:51.910 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:51 smithi023 bash[21372]: cluster 2024-03-11T20:57:50.720682+0000 mon.a (mon.0) 1075 : cluster 0 osdmap e168: 8 total, 8 up, 8 in 2024-03-11T20:57:51.910 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:51 smithi023 bash[29146]: cluster 2024-03-11T20:57:49.902429+0000 osd.0 (osd.0) 159 : cluster 0 purged_snaps scrub starts 2024-03-11T20:57:51.911 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:51 smithi023 bash[29146]: cluster 2024-03-11T20:57:49.902606+0000 osd.0 (osd.0) 160 : cluster 0 purged_snaps scrub ok 2024-03-11T20:57:51.911 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:51 smithi023 bash[29146]: cluster 2024-03-11T20:57:50.720682+0000 mon.a (mon.0) 1075 : cluster 0 osdmap e168: 8 total, 8 up, 8 in 2024-03-11T20:57:51.927 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:51.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:51.931 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:57:52.032 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:57:52.054 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:57:52.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:52.060 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:57:52.161 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:57:52.184 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:57:52.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:52.188 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:57:52.288 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:57:52.318 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:57:52.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:52.321 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:57:52.422 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:52.447 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:52.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:52.449 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:57:52.550 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:52.590 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:52.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:52.593 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:57:52.694 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:57:52.717 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:57:52.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:52.719 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:57:52.820 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:57:52.846 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:57:52.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:52.850 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:57:52.911 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:57:52.935 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:52 smithi023 bash[21372]: cluster 2024-03-11T20:57:51.174793+0000 osd.1 (osd.1) 99 : cluster 0 purged_snaps scrub starts 2024-03-11T20:57:52.936 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:52 smithi023 bash[21372]: cluster 2024-03-11T20:57:51.174871+0000 osd.1 (osd.1) 100 : cluster 0 purged_snaps scrub ok 2024-03-11T20:57:52.936 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:52 smithi023 bash[21372]: cluster 2024-03-11T20:57:51.732229+0000 mon.a (mon.0) 1076 : cluster 0 osdmap e169: 8 total, 8 up, 8 in 2024-03-11T20:57:52.936 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:52 smithi023 bash[21372]: cluster 2024-03-11T20:57:51.968527+0000 mgr.y (mgr.24380) 265 : cluster 0 pgmap v260: 148 pgs: 3 active+remapped+backfilling, 1 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 142 active+clean; 2.2 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 839 op/s; 7990/106899 objects misplaced (7.474%); 20 MiB/s, 316 objects/s recovering 2024-03-11T20:57:52.936 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:52 smithi023 bash[29146]: cluster 2024-03-11T20:57:51.174793+0000 osd.1 (osd.1) 99 : cluster 0 purged_snaps scrub starts 2024-03-11T20:57:52.936 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:52 smithi023 bash[29146]: cluster 2024-03-11T20:57:51.174871+0000 osd.1 (osd.1) 100 : cluster 0 purged_snaps scrub ok 2024-03-11T20:57:52.936 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:52 smithi023 bash[29146]: cluster 2024-03-11T20:57:51.732229+0000 mon.a (mon.0) 1076 : cluster 0 osdmap e169: 8 total, 8 up, 8 in 2024-03-11T20:57:52.936 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:52 smithi023 bash[29146]: cluster 2024-03-11T20:57:51.968527+0000 mgr.y (mgr.24380) 265 : cluster 0 pgmap v260: 148 pgs: 3 active+remapped+backfilling, 1 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 142 active+clean; 2.2 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 839 op/s; 7990/106899 objects misplaced (7.474%); 20 MiB/s, 316 objects/s recovering 2024-03-11T20:57:52.951 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:57:52.974 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:57:52.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:52.976 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:57:53.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:52 smithi146 bash[23809]: cluster 2024-03-11T20:57:51.174793+0000 osd.1 (osd.1) 99 : cluster 0 purged_snaps scrub starts 2024-03-11T20:57:53.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:52 smithi146 bash[23809]: cluster 2024-03-11T20:57:51.174871+0000 osd.1 (osd.1) 100 : cluster 0 purged_snaps scrub ok 2024-03-11T20:57:53.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:52 smithi146 bash[23809]: cluster 2024-03-11T20:57:51.732229+0000 mon.a (mon.0) 1076 : cluster 0 osdmap e169: 8 total, 8 up, 8 in 2024-03-11T20:57:53.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:52 smithi146 bash[23809]: cluster 2024-03-11T20:57:51.968527+0000 mgr.y (mgr.24380) 265 : cluster 0 pgmap v260: 148 pgs: 3 active+remapped+backfilling, 1 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 142 active+clean; 2.2 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 839 op/s; 7990/106899 objects misplaced (7.474%); 20 MiB/s, 316 objects/s recovering 2024-03-11T20:57:53.078 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:53.101 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:53.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:53.103 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:57:53.204 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:57:53.223 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:57:53.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:53.225 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:57:53.326 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:53.348 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:53.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:53.350 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:57:53.451 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:57:53.478 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:57:53.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:53.480 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:57:53.581 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:57:53.601 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:57:53.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:53.603 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:57:53.704 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:57:53.731 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:57:53.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:53.733 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:57:53.834 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:57:53.843 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:53 smithi023 bash[21372]: cluster 2024-03-11T20:57:52.741027+0000 mon.a (mon.0) 1077 : cluster 0 osdmap e170: 8 total, 8 up, 8 in 2024-03-11T20:57:53.843 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:53 smithi023 bash[29146]: cluster 2024-03-11T20:57:52.741027+0000 mon.a (mon.0) 1077 : cluster 0 osdmap e170: 8 total, 8 up, 8 in 2024-03-11T20:57:53.856 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:57:53.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:53.859 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:57:53.959 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:57:53.968 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:53 smithi146 bash[23809]: cluster 2024-03-11T20:57:52.741027+0000 mon.a (mon.0) 1077 : cluster 0 osdmap e170: 8 total, 8 up, 8 in 2024-03-11T20:57:53.979 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:57:53.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:53.981 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:57:54.082 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:57:54.103 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:57:54.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:54.104 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:57:54.205 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:57:54.228 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:57:54.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:54.229 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:57:54.330 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:57:54.353 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:57:54.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:54.355 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:57:54.456 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:57:54.480 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:57:54.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:54.482 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:57:54.583 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:54.608 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:54.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:54.611 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:57:54.712 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:57:54.738 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:57:54.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:54.740 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:57:54.840 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:57:54.868 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:57:54.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:54.870 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:57:54.944 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:57:54.970 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:57:54.994 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:57:54.996 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:54 smithi146 bash[23809]: cluster 2024-03-11T20:57:53.739562+0000 mon.a (mon.0) 1078 : cluster 0 osdmap e171: 8 total, 8 up, 8 in 2024-03-11T20:57:54.996 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:54 smithi146 bash[23809]: cluster 2024-03-11T20:57:53.968963+0000 mgr.y (mgr.24380) 266 : cluster 0 pgmap v263: 148 pgs: 3 active+remapped+backfilling, 1 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 142 active+clean; 2.2 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 654 op/s; 7990/106899 objects misplaced (7.474%); 20 MiB/s, 316 objects/s recovering 2024-03-11T20:57:54.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:54.997 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:57:55.036 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:54 smithi023 bash[21372]: cluster 2024-03-11T20:57:53.739562+0000 mon.a (mon.0) 1078 : cluster 0 osdmap e171: 8 total, 8 up, 8 in 2024-03-11T20:57:55.036 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:54 smithi023 bash[21372]: cluster 2024-03-11T20:57:53.968963+0000 mgr.y (mgr.24380) 266 : cluster 0 pgmap v263: 148 pgs: 3 active+remapped+backfilling, 1 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 142 active+clean; 2.2 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 654 op/s; 7990/106899 objects misplaced (7.474%); 20 MiB/s, 316 objects/s recovering 2024-03-11T20:57:55.036 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:54 smithi023 bash[29146]: cluster 2024-03-11T20:57:53.739562+0000 mon.a (mon.0) 1078 : cluster 0 osdmap e171: 8 total, 8 up, 8 in 2024-03-11T20:57:55.036 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:54 smithi023 bash[29146]: cluster 2024-03-11T20:57:53.968963+0000 mgr.y (mgr.24380) 266 : cluster 0 pgmap v263: 148 pgs: 3 active+remapped+backfilling, 1 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 142 active+clean; 2.2 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 654 op/s; 7990/106899 objects misplaced (7.474%); 20 MiB/s, 316 objects/s recovering 2024-03-11T20:57:55.098 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:57:55.129 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:57:55.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:55.132 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:57:55.232 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:57:55.260 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:57:55.260 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 20:57:55 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T20:57:55.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:55.262 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:57:55.363 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:57:55.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:55.416 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:57:55.416 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:57:55.517 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:57:55.542 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:57:55.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:55.548 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:57:55.649 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:55.684 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:55.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:55.687 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:57:55.788 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:57:55.836 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:57:55.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:55.842 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:57:55.943 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:55.957 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:55 smithi146 bash[23809]: cluster 2024-03-11T20:57:54.753111+0000 mon.a (mon.0) 1079 : cluster 0 osdmap e172: 8 total, 8 up, 8 in 2024-03-11T20:57:55.957 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:55 smithi146 bash[23809]: cluster 2024-03-11T20:57:54.993732+0000 osd.2 (osd.2) 122 : cluster 0 purged_snaps scrub starts 2024-03-11T20:57:55.957 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:55 smithi146 bash[23809]: cluster 2024-03-11T20:57:54.993798+0000 osd.2 (osd.2) 123 : cluster 0 purged_snaps scrub ok 2024-03-11T20:57:55.957 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:55 smithi146 bash[23809]: audit 2024-03-11T20:57:55.149102+0000 mgr.y (mgr.24380) 267 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:57:55.959 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:57:55.967 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:55.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:55.970 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:57:55.999 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:55 smithi023 bash[21372]: cluster 2024-03-11T20:57:54.753111+0000 mon.a (mon.0) 1079 : cluster 0 osdmap e172: 8 total, 8 up, 8 in 2024-03-11T20:57:55.999 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:55 smithi023 bash[21372]: cluster 2024-03-11T20:57:54.993732+0000 osd.2 (osd.2) 122 : cluster 0 purged_snaps scrub starts 2024-03-11T20:57:55.999 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:55 smithi023 bash[21372]: cluster 2024-03-11T20:57:54.993798+0000 osd.2 (osd.2) 123 : cluster 0 purged_snaps scrub ok 2024-03-11T20:57:55.999 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:55 smithi023 bash[21372]: audit 2024-03-11T20:57:55.149102+0000 mgr.y (mgr.24380) 267 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:57:55.999 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:55 smithi023 bash[29146]: cluster 2024-03-11T20:57:54.753111+0000 mon.a (mon.0) 1079 : cluster 0 osdmap e172: 8 total, 8 up, 8 in 2024-03-11T20:57:56.000 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:55 smithi023 bash[29146]: cluster 2024-03-11T20:57:54.993732+0000 osd.2 (osd.2) 122 : cluster 0 purged_snaps scrub starts 2024-03-11T20:57:56.000 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:55 smithi023 bash[29146]: cluster 2024-03-11T20:57:54.993798+0000 osd.2 (osd.2) 123 : cluster 0 purged_snaps scrub ok 2024-03-11T20:57:56.000 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:55 smithi023 bash[29146]: audit 2024-03-11T20:57:55.149102+0000 mgr.y (mgr.24380) 267 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:57:56.070 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:56.103 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:56.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:56.105 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:57:56.206 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:57:56.228 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:57:56.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:56.230 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:57:56.331 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:57:56.359 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:57:56.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:56.361 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:57:56.462 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:57:56.482 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:57:56.482 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 6, 7, 1] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-11T20:57:56.482 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-11T20:57:56.482 INFO:tasks.thrashosds.thrasher:Killing osd 1, live_osds are [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-11T20:57:56.482 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-11T20:57:56.483 DEBUG:teuthology.orchestra.run.smithi023:> sudo systemctl stop ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@osd.1 2024-03-11T20:57:56.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:56.484 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:57:56.585 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:57:56.595 INFO:journalctl@ceph.osd.1.smithi023.stdout:Mar 11 20:57:56 smithi023 systemd[1]: Stopping Ceph osd.1 for 67ba00f8-dfe7-11ee-95c8-87774f69a715... 2024-03-11T20:57:56.609 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:57:56.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:56.612 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:57:56.713 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:57:56.738 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:57:56.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:56.744 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:57:56.845 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:57:56.872 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:57:56.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:56.876 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:57:56.977 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:56.986 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:56 smithi146 bash[23809]: cluster 2024-03-11T20:57:55.790922+0000 mon.a (mon.0) 1080 : cluster 0 osdmap e173: 8 total, 8 up, 8 in 2024-03-11T20:57:56.986 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:56 smithi146 bash[23809]: cluster 2024-03-11T20:57:55.969435+0000 mgr.y (mgr.24380) 268 : cluster 0 pgmap v266: 148 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 145 active+clean; 2.0 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 669 op/s; 5260/98883 objects misplaced (5.319%); 34 MiB/s, 546 objects/s recovering 2024-03-11T20:57:57.001 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:57.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:57.003 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:57:57.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:56 smithi023 bash[21372]: cluster 2024-03-11T20:57:55.790922+0000 mon.a (mon.0) 1080 : cluster 0 osdmap e173: 8 total, 8 up, 8 in 2024-03-11T20:57:57.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:56 smithi023 bash[21372]: cluster 2024-03-11T20:57:55.969435+0000 mgr.y (mgr.24380) 268 : cluster 0 pgmap v266: 148 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 145 active+clean; 2.0 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 669 op/s; 5260/98883 objects misplaced (5.319%); 34 MiB/s, 546 objects/s recovering 2024-03-11T20:57:57.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:56 smithi023 bash[29146]: cluster 2024-03-11T20:57:55.790922+0000 mon.a (mon.0) 1080 : cluster 0 osdmap e173: 8 total, 8 up, 8 in 2024-03-11T20:57:57.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:56 smithi023 bash[29146]: cluster 2024-03-11T20:57:55.969435+0000 mgr.y (mgr.24380) 268 : cluster 0 pgmap v266: 148 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 145 active+clean; 2.0 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 669 op/s; 5260/98883 objects misplaced (5.319%); 34 MiB/s, 546 objects/s recovering 2024-03-11T20:57:57.104 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:57:57.125 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:57:57.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:57.129 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:57:57.230 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:57:57.257 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:57:57.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:57.260 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:57:57.361 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:57.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:57.411 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:57.412 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:57:57.512 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:57:57.544 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:57:57.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:57.548 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:57:57.649 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:57.667 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:57.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:57.669 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:57:57.770 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:57:57.792 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:57:57.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:57.794 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:57:57.895 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:57.926 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:57.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:57.931 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:57:58.032 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:57:58.054 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:57:58.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:58.057 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:57:58.158 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:57:58.168 INFO:journalctl@ceph.osd.1.smithi023.stdout:Mar 11 20:57:58 smithi023 bash[37667]: debug 2024-03-11T20:57:58.048+0000 7f32fb1a5700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-03-11T20:57:58.169 INFO:journalctl@ceph.osd.1.smithi023.stdout:Mar 11 20:57:58 smithi023 bash[37667]: debug 2024-03-11T20:57:58.048+0000 7f32fb1a5700 -1 osd.1 173 *** Got signal Terminated *** 2024-03-11T20:57:58.169 INFO:journalctl@ceph.osd.1.smithi023.stdout:Mar 11 20:57:58 smithi023 bash[37667]: debug 2024-03-11T20:57:58.048+0000 7f32fb1a5700 -1 osd.1 173 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T20:57:58.181 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:57:58.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:58.182 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:57:58.283 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:58.306 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:58.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:58.309 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:57:58.409 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:57:58.435 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:57:58.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:58.439 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:57:58.535 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:57:58 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:20:57:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T20:57:58.539 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:58.570 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:57:58.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:58.572 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:57:58.673 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:58.703 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:58.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:58.707 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:57:58.808 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:57:58.830 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:57:58.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:58.832 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:57:58.933 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:57:58.960 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:57:58.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:58.966 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:57:59.067 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:57:59.100 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:57:59.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:59.104 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:57:59.205 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:57:59.214 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:59 smithi146 bash[23809]: cluster 2024-03-11T20:57:57.971790+0000 mgr.y (mgr.24380) 269 : cluster 0 pgmap v267: 148 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 145 active+clean; 2.0 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 977 B/s rd, 0 B/s wr, 510 op/s; 5260/98883 objects misplaced (5.319%); 26 MiB/s, 417 objects/s recovering 2024-03-11T20:57:59.214 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:57:59 smithi146 bash[23809]: cluster 2024-03-11T20:57:58.054708+0000 mon.a (mon.0) 1081 : cluster 1 osd.1 marked itself down and dead 2024-03-11T20:57:59.227 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:57:59.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:59.229 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:57:59.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:59 smithi023 bash[21372]: cluster 2024-03-11T20:57:57.971790+0000 mgr.y (mgr.24380) 269 : cluster 0 pgmap v267: 148 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 145 active+clean; 2.0 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 977 B/s rd, 0 B/s wr, 510 op/s; 5260/98883 objects misplaced (5.319%); 26 MiB/s, 417 objects/s recovering 2024-03-11T20:57:59.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:57:59 smithi023 bash[21372]: cluster 2024-03-11T20:57:58.054708+0000 mon.a (mon.0) 1081 : cluster 1 osd.1 marked itself down and dead 2024-03-11T20:57:59.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:59 smithi023 bash[29146]: cluster 2024-03-11T20:57:57.971790+0000 mgr.y (mgr.24380) 269 : cluster 0 pgmap v267: 148 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 145 active+clean; 2.0 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 977 B/s rd, 0 B/s wr, 510 op/s; 5260/98883 objects misplaced (5.319%); 26 MiB/s, 417 objects/s recovering 2024-03-11T20:57:59.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:57:59 smithi023 bash[29146]: cluster 2024-03-11T20:57:58.054708+0000 mon.a (mon.0) 1081 : cluster 1 osd.1 marked itself down and dead 2024-03-11T20:57:59.330 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:57:59.354 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:57:59.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:59.356 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:57:59.457 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:57:59.484 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:57:59.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:59.486 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:57:59.587 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:57:59.617 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:57:59.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:59.619 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:57:59.720 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:57:59.756 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:57:59.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:59.760 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:57:59.861 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:59.897 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:57:59.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:57:59.899 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:58:00.000 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:00.021 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:00.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:00.023 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:58:00.124 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:58:00.134 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:00 smithi023 bash[29146]: cluster 2024-03-11T20:57:59.030173+0000 mon.a (mon.0) 1082 : cluster 3 Health check failed: 1 osds down (OSD_DOWN) 2024-03-11T20:58:00.134 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:00 smithi023 bash[29146]: cluster 2024-03-11T20:57:59.071753+0000 mon.a (mon.0) 1083 : cluster 0 osdmap e174: 8 total, 7 up, 8 in 2024-03-11T20:58:00.135 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:00 smithi023 bash[29146]: cluster 2024-03-11T20:57:59.708944+0000 mon.a (mon.0) 1084 : cluster 0 osdmap e175: 8 total, 7 up, 8 in 2024-03-11T20:58:00.135 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:00 smithi023 bash[21372]: cluster 2024-03-11T20:57:59.030173+0000 mon.a (mon.0) 1082 : cluster 3 Health check failed: 1 osds down (OSD_DOWN) 2024-03-11T20:58:00.135 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:00 smithi023 bash[21372]: cluster 2024-03-11T20:57:59.071753+0000 mon.a (mon.0) 1083 : cluster 0 osdmap e174: 8 total, 7 up, 8 in 2024-03-11T20:58:00.135 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:00 smithi023 bash[21372]: cluster 2024-03-11T20:57:59.708944+0000 mon.a (mon.0) 1084 : cluster 0 osdmap e175: 8 total, 7 up, 8 in 2024-03-11T20:58:00.149 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:58:00.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:00.150 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:58:00.251 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:00.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:00 smithi146 bash[23809]: cluster 2024-03-11T20:57:59.030173+0000 mon.a (mon.0) 1082 : cluster 3 Health check failed: 1 osds down (OSD_DOWN) 2024-03-11T20:58:00.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:00 smithi146 bash[23809]: cluster 2024-03-11T20:57:59.071753+0000 mon.a (mon.0) 1083 : cluster 0 osdmap e174: 8 total, 7 up, 8 in 2024-03-11T20:58:00.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:00 smithi146 bash[23809]: cluster 2024-03-11T20:57:59.708944+0000 mon.a (mon.0) 1084 : cluster 0 osdmap e175: 8 total, 7 up, 8 in 2024-03-11T20:58:00.275 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:00.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:00.278 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:58:00.378 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:00.404 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:00.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:00.406 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:58:00.507 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:00.534 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:00.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:00.536 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:58:00.637 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:00.665 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:00.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:00.667 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:58:00.768 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:00.793 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:00.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:00.795 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:58:00.896 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:00.916 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:00.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:00.919 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:58:01.019 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:01.068 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:01.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:01.075 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:58:01.176 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:01.210 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:01.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:01.213 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:58:01.216 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:01 smithi023 bash[21372]: cluster 2024-03-11T20:57:59.972726+0000 mgr.y (mgr.24380) 270 : cluster 0 pgmap v270: 148 pgs: 1 remapped+peering, 109 active+clean, 19 active+undersized, 1 active+remapped+backfill_toofull, 9 stale+active+clean, 8 active+undersized+degraded, 1 active+clean+remapped; 1.9 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 980 B/s rd, 0 B/s wr, 737 op/s; 1841/95367 objects degraded (1.930%); 4004/95367 objects misplaced (4.199%); 36 MiB/s, 574 objects/s recovering 2024-03-11T20:58:01.216 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:01 smithi023 bash[21372]: cluster 2024-03-11T20:58:00.704920+0000 mon.a (mon.0) 1085 : cluster 3 Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-11T20:58:01.216 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:01 smithi023 bash[21372]: cluster 2024-03-11T20:58:00.705006+0000 mon.a (mon.0) 1086 : cluster 3 Health check failed: Degraded data redundancy: 1841/95367 objects degraded (1.930%), 8 pgs degraded (PG_DEGRADED) 2024-03-11T20:58:01.217 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:01 smithi023 bash[21372]: cluster 2024-03-11T20:58:00.720711+0000 mon.a (mon.0) 1087 : cluster 0 osdmap e176: 8 total, 7 up, 8 in 2024-03-11T20:58:01.217 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:01 smithi023 bash[29146]: cluster 2024-03-11T20:57:59.972726+0000 mgr.y (mgr.24380) 270 : cluster 0 pgmap v270: 148 pgs: 1 remapped+peering, 109 active+clean, 19 active+undersized, 1 active+remapped+backfill_toofull, 9 stale+active+clean, 8 active+undersized+degraded, 1 active+clean+remapped; 1.9 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 980 B/s rd, 0 B/s wr, 737 op/s; 1841/95367 objects degraded (1.930%); 4004/95367 objects misplaced (4.199%); 36 MiB/s, 574 objects/s recovering 2024-03-11T20:58:01.217 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:01 smithi023 bash[29146]: cluster 2024-03-11T20:58:00.704920+0000 mon.a (mon.0) 1085 : cluster 3 Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-11T20:58:01.217 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:01 smithi023 bash[29146]: cluster 2024-03-11T20:58:00.705006+0000 mon.a (mon.0) 1086 : cluster 3 Health check failed: Degraded data redundancy: 1841/95367 objects degraded (1.930%), 8 pgs degraded (PG_DEGRADED) 2024-03-11T20:58:01.217 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:01 smithi023 bash[29146]: cluster 2024-03-11T20:58:00.720711+0000 mon.a (mon.0) 1087 : cluster 0 osdmap e176: 8 total, 7 up, 8 in 2024-03-11T20:58:01.314 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:01.324 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:01 smithi146 bash[23809]: cluster 2024-03-11T20:57:59.972726+0000 mgr.y (mgr.24380) 270 : cluster 0 pgmap v270: 148 pgs: 1 remapped+peering, 109 active+clean, 19 active+undersized, 1 active+remapped+backfill_toofull, 9 stale+active+clean, 8 active+undersized+degraded, 1 active+clean+remapped; 1.9 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 980 B/s rd, 0 B/s wr, 737 op/s; 1841/95367 objects degraded (1.930%); 4004/95367 objects misplaced (4.199%); 36 MiB/s, 574 objects/s recovering 2024-03-11T20:58:01.324 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:01 smithi146 bash[23809]: cluster 2024-03-11T20:58:00.704920+0000 mon.a (mon.0) 1085 : cluster 3 Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-11T20:58:01.324 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:01 smithi146 bash[23809]: cluster 2024-03-11T20:58:00.705006+0000 mon.a (mon.0) 1086 : cluster 3 Health check failed: Degraded data redundancy: 1841/95367 objects degraded (1.930%), 8 pgs degraded (PG_DEGRADED) 2024-03-11T20:58:01.325 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:01 smithi146 bash[23809]: cluster 2024-03-11T20:58:00.720711+0000 mon.a (mon.0) 1087 : cluster 0 osdmap e176: 8 total, 7 up, 8 in 2024-03-11T20:58:01.339 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:01.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:01.343 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:58:01.444 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:01.474 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:01.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:01.478 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:58:01.579 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:01.605 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:01.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:01.607 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:58:01.708 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:01.729 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:01.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:01.731 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:58:01.832 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:58:01.866 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:58:01.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:01.868 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:58:01.969 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:01.988 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:01.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:01.989 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:58:02.090 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:02.100 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:02 smithi023 bash[21372]: cluster 2024-03-11T20:58:02.042077+0000 mon.a (mon.0) 1088 : cluster 1 Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-11T20:58:02.101 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:02 smithi023 bash[29146]: cluster 2024-03-11T20:58:02.042077+0000 mon.a (mon.0) 1088 : cluster 1 Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-11T20:58:02.117 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:02.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:02.119 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:58:02.220 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:02.251 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:02.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:02.255 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:58:02.356 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:02.381 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:02.385 INFO:journalctl@ceph.osd.1.smithi023.stdout:Mar 11 20:58:02 smithi023 bash[80327]: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd-1 2024-03-11T20:58:02.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:02.386 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:58:02.487 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:58:02.497 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:02 smithi146 bash[23809]: cluster 2024-03-11T20:58:02.042077+0000 mon.a (mon.0) 1088 : cluster 1 Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-11T20:58:02.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:02.523 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:58:02.524 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:58:02.625 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:02.667 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:02.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:02.673 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:58:02.773 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:58:02.804 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:58:02.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:02.810 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:58:02.911 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:02.935 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:02.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:02.937 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:58:03.038 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:03.059 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:03.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:03.061 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:58:03.162 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:03.175 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:03 smithi146 bash[23809]: cluster 2024-03-11T20:58:01.973320+0000 mgr.y (mgr.24380) 271 : cluster 0 pgmap v272: 148 pgs: 1 remapped+peering, 89 active+clean, 34 active+undersized, 22 active+undersized+degraded, 2 active+clean+remapped; 1.8 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 341 B/s rd, 0 B/s wr, 517 op/s; 5592/89577 objects degraded (6.243%); 3791/89577 objects misplaced (4.232%); 12 MiB/s, 197 objects/s recovering 2024-03-11T20:58:03.175 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:03 smithi146 bash[23809]: cluster 2024-03-11T20:58:02.564881+0000 osd.6 (osd.6) 112 : cluster 0 purged_snaps scrub starts 2024-03-11T20:58:03.175 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:03 smithi146 bash[23809]: cluster 2024-03-11T20:58:02.564959+0000 osd.6 (osd.6) 113 : cluster 0 purged_snaps scrub ok 2024-03-11T20:58:03.189 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:03.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:03.192 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:58:03.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:03 smithi023 bash[21372]: cluster 2024-03-11T20:58:01.973320+0000 mgr.y (mgr.24380) 271 : cluster 0 pgmap v272: 148 pgs: 1 remapped+peering, 89 active+clean, 34 active+undersized, 22 active+undersized+degraded, 2 active+clean+remapped; 1.8 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 341 B/s rd, 0 B/s wr, 517 op/s; 5592/89577 objects degraded (6.243%); 3791/89577 objects misplaced (4.232%); 12 MiB/s, 197 objects/s recovering 2024-03-11T20:58:03.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:03 smithi023 bash[21372]: cluster 2024-03-11T20:58:02.564881+0000 osd.6 (osd.6) 112 : cluster 0 purged_snaps scrub starts 2024-03-11T20:58:03.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:03 smithi023 bash[21372]: cluster 2024-03-11T20:58:02.564959+0000 osd.6 (osd.6) 113 : cluster 0 purged_snaps scrub ok 2024-03-11T20:58:03.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:03 smithi023 bash[29146]: cluster 2024-03-11T20:58:01.973320+0000 mgr.y (mgr.24380) 271 : cluster 0 pgmap v272: 148 pgs: 1 remapped+peering, 89 active+clean, 34 active+undersized, 22 active+undersized+degraded, 2 active+clean+remapped; 1.8 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 341 B/s rd, 0 B/s wr, 517 op/s; 5592/89577 objects degraded (6.243%); 3791/89577 objects misplaced (4.232%); 12 MiB/s, 197 objects/s recovering 2024-03-11T20:58:03.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:03 smithi023 bash[29146]: cluster 2024-03-11T20:58:02.564881+0000 osd.6 (osd.6) 112 : cluster 0 purged_snaps scrub starts 2024-03-11T20:58:03.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:03 smithi023 bash[29146]: cluster 2024-03-11T20:58:02.564959+0000 osd.6 (osd.6) 113 : cluster 0 purged_snaps scrub ok 2024-03-11T20:58:03.293 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:03.324 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:03.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:03.328 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:58:03.429 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:03.454 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:03.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:03.456 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:58:03.557 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:03.582 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:03.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:03.584 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:58:03.637 INFO:journalctl@ceph.osd.1.smithi023.stdout:Mar 11 20:58:03 smithi023 bash[80793]: Error response from daemon: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd-1 2024-03-11T20:58:03.685 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:03.728 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:03.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:03.732 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:58:03.833 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:03.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:03.860 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:03.861 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:58:03.961 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:58:03.985 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:58:03.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:03.987 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:58:04.087 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:58:04.111 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:58:04.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:04.113 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:58:04.214 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:58:04.246 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:58:04.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:04.249 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:58:04.350 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:04.376 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:04.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:04.379 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:58:04.480 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:58:04.488 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:04 smithi023 bash[21372]: cluster 2024-03-11T20:58:02.524992+0000 osd.7 (osd.7) 157 : cluster 0 purged_snaps scrub starts 2024-03-11T20:58:04.488 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:04 smithi023 bash[21372]: cluster 2024-03-11T20:58:02.525082+0000 osd.7 (osd.7) 158 : cluster 0 purged_snaps scrub ok 2024-03-11T20:58:04.488 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:04 smithi023 bash[21372]: cluster 2024-03-11T20:58:03.105113+0000 mon.a (mon.0) 1089 : cluster 0 osdmap e177: 8 total, 7 up, 8 in 2024-03-11T20:58:04.489 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:04 smithi023 bash[29146]: cluster 2024-03-11T20:58:02.524992+0000 osd.7 (osd.7) 157 : cluster 0 purged_snaps scrub starts 2024-03-11T20:58:04.489 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:04 smithi023 bash[29146]: cluster 2024-03-11T20:58:02.525082+0000 osd.7 (osd.7) 158 : cluster 0 purged_snaps scrub ok 2024-03-11T20:58:04.489 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:04 smithi023 bash[29146]: cluster 2024-03-11T20:58:03.105113+0000 mon.a (mon.0) 1089 : cluster 0 osdmap e177: 8 total, 7 up, 8 in 2024-03-11T20:58:04.513 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:58:04.514 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:04 smithi146 bash[23809]: cluster 2024-03-11T20:58:02.524992+0000 osd.7 (osd.7) 157 : cluster 0 purged_snaps scrub starts 2024-03-11T20:58:04.514 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:04 smithi146 bash[23809]: cluster 2024-03-11T20:58:02.525082+0000 osd.7 (osd.7) 158 : cluster 0 purged_snaps scrub ok 2024-03-11T20:58:04.514 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:04 smithi146 bash[23809]: cluster 2024-03-11T20:58:03.105113+0000 mon.a (mon.0) 1089 : cluster 0 osdmap e177: 8 total, 7 up, 8 in 2024-03-11T20:58:04.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:04.517 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:58:04.617 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:04.644 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:04.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:04.648 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:58:04.749 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:04.775 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:04.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:04.777 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:58:04.877 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:04.909 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:04.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:04.912 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:58:05.013 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:05.038 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:05.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:05.041 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:58:05.141 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:05.142 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:05 smithi023 bash[21372]: cluster 2024-03-11T20:58:03.973931+0000 mgr.y (mgr.24380) 272 : cluster 0 pgmap v274: 148 pgs: 1 remapped+peering, 89 active+clean, 34 active+undersized, 22 active+undersized+degraded, 2 active+clean+remapped; 1.8 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 417 B/s rd, 0 B/s wr, 632 op/s; 5592/89577 objects degraded (6.243%); 3791/89577 objects misplaced (4.232%); 15 MiB/s, 241 objects/s recovering 2024-03-11T20:58:05.143 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:05 smithi023 bash[21372]: audit 2024-03-11T20:58:04.611430+0000 mon.a (mon.0) 1090 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:58:05.143 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:05 smithi023 bash[21372]: audit 2024-03-11T20:58:04.613454+0000 mon.a (mon.0) 1091 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:58:05.143 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:05 smithi023 bash[29146]: cluster 2024-03-11T20:58:03.973931+0000 mgr.y (mgr.24380) 272 : cluster 0 pgmap v274: 148 pgs: 1 remapped+peering, 89 active+clean, 34 active+undersized, 22 active+undersized+degraded, 2 active+clean+remapped; 1.8 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 417 B/s rd, 0 B/s wr, 632 op/s; 5592/89577 objects degraded (6.243%); 3791/89577 objects misplaced (4.232%); 15 MiB/s, 241 objects/s recovering 2024-03-11T20:58:05.143 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:05 smithi023 bash[29146]: audit 2024-03-11T20:58:04.611430+0000 mon.a (mon.0) 1090 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:58:05.143 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:05 smithi023 bash[29146]: audit 2024-03-11T20:58:04.613454+0000 mon.a (mon.0) 1091 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:58:05.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:05.171 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:05.171 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:58:05.272 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:05.301 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:05.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:05.308 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:58:05.409 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:58:05.438 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:58:05.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:05.442 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:58:05.510 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 20:58:05 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T20:58:05.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:05 smithi146 bash[23809]: cluster 2024-03-11T20:58:03.973931+0000 mgr.y (mgr.24380) 272 : cluster 0 pgmap v274: 148 pgs: 1 remapped+peering, 89 active+clean, 34 active+undersized, 22 active+undersized+degraded, 2 active+clean+remapped; 1.8 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 417 B/s rd, 0 B/s wr, 632 op/s; 5592/89577 objects degraded (6.243%); 3791/89577 objects misplaced (4.232%); 15 MiB/s, 241 objects/s recovering 2024-03-11T20:58:05.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:05 smithi146 bash[23809]: audit 2024-03-11T20:58:04.611430+0000 mon.a (mon.0) 1090 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:58:05.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:05 smithi146 bash[23809]: audit 2024-03-11T20:58:04.613454+0000 mon.a (mon.0) 1091 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:58:05.542 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:58:05.570 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:58:05.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:05.573 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:58:05.674 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:05.703 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:05.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:05.705 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:58:05.806 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:58:05.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:05.835 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:58:05.835 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:58:05.936 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:58:05.965 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:58:05.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:05.969 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:58:06.070 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:06.092 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:06.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:06.095 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:58:06.196 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:58:06.226 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:58:06.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:06.228 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:58:06.329 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:58:06.358 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:58:06.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:06.361 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:58:06.462 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:58:06.480 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:58:06.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:06.482 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:58:06.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:06 smithi146 bash[23809]: audit 2024-03-11T20:58:05.157173+0000 mgr.y (mgr.24380) 273 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:58:06.547 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:06 smithi023 bash[29146]: audit 2024-03-11T20:58:05.157173+0000 mgr.y (mgr.24380) 273 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:58:06.548 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:06 smithi023 bash[21372]: audit 2024-03-11T20:58:05.157173+0000 mgr.y (mgr.24380) 273 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:58:06.583 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:58:06.615 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:58:06.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:06.617 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:58:06.718 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:06.752 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:06.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:06.754 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:58:06.855 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:58:06.873 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:58:06.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:06.875 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:58:06.976 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:07.011 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:07.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:07.014 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:58:07.115 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:07.140 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:07.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:07.142 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:58:07.243 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:07.251 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:07 smithi146 bash[23809]: cluster 2024-03-11T20:58:05.974783+0000 mgr.y (mgr.24380) 274 : cluster 0 pgmap v275: 148 pgs: 89 active+clean, 34 active+undersized, 22 active+undersized+degraded, 3 active+clean+remapped; 1.6 GiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.07k op/s; 4926/78780 objects degraded (6.253%); 4947/78780 objects misplaced (6.280%); 12 MiB/s, 189 objects/s recovering 2024-03-11T20:58:07.251 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:07 smithi146 bash[23809]: cluster 2024-03-11T20:58:06.138696+0000 mon.a (mon.0) 1092 : cluster 3 Health check update: Degraded data redundancy: 4926/78780 objects degraded (6.253%), 22 pgs degraded (PG_DEGRADED) 2024-03-11T20:58:07.271 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:07.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:07.273 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:58:07.374 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:07.406 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:07.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:07.408 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:58:07.509 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:58:07.529 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:58:07.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:07.530 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:58:07.534 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:07 smithi023 bash[29146]: cluster 2024-03-11T20:58:05.974783+0000 mgr.y (mgr.24380) 274 : cluster 0 pgmap v275: 148 pgs: 89 active+clean, 34 active+undersized, 22 active+undersized+degraded, 3 active+clean+remapped; 1.6 GiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.07k op/s; 4926/78780 objects degraded (6.253%); 4947/78780 objects misplaced (6.280%); 12 MiB/s, 189 objects/s recovering 2024-03-11T20:58:07.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:07 smithi023 bash[29146]: cluster 2024-03-11T20:58:06.138696+0000 mon.a (mon.0) 1092 : cluster 3 Health check update: Degraded data redundancy: 4926/78780 objects degraded (6.253%), 22 pgs degraded (PG_DEGRADED) 2024-03-11T20:58:07.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:07 smithi023 bash[21372]: cluster 2024-03-11T20:58:05.974783+0000 mgr.y (mgr.24380) 274 : cluster 0 pgmap v275: 148 pgs: 89 active+clean, 34 active+undersized, 22 active+undersized+degraded, 3 active+clean+remapped; 1.6 GiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.07k op/s; 4926/78780 objects degraded (6.253%); 4947/78780 objects misplaced (6.280%); 12 MiB/s, 189 objects/s recovering 2024-03-11T20:58:07.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:07 smithi023 bash[21372]: cluster 2024-03-11T20:58:06.138696+0000 mon.a (mon.0) 1092 : cluster 3 Health check update: Degraded data redundancy: 4926/78780 objects degraded (6.253%), 22 pgs degraded (PG_DEGRADED) 2024-03-11T20:58:07.631 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:07.659 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:07.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:07.664 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:58:07.765 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:07.790 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:07.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:07.792 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:58:07.892 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:58:07.921 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:58:07.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:07.926 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:58:08.026 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:58:08.049 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:58:08.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:08.054 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:58:08.154 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:08.177 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:08.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:08.182 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:58:08.283 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:08.313 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:08.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:08.317 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:58:08.410 INFO:teuthology.orchestra.run.smithi023.stdout:osd.0: {} 2024-03-11T20:58:08.419 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:08.420 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:58:08 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:20:58:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T20:58:08.464 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:08.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:08.466 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:58:08.478 INFO:teuthology.orchestra.run.smithi023.stdout:osd.2: {} 2024-03-11T20:58:08.543 INFO:teuthology.orchestra.run.smithi023.stdout:osd.3: {} 2024-03-11T20:58:08.568 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:08.576 INFO:teuthology.orchestra.run.smithi023.stdout:osd.4: {} 2024-03-11T20:58:08.588 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:08.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:08.594 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:58:08.617 INFO:teuthology.orchestra.run.smithi023.stdout:osd.5: {} 2024-03-11T20:58:08.652 INFO:teuthology.orchestra.run.smithi023.stdout:osd.6: {} 2024-03-11T20:58:08.695 INFO:teuthology.orchestra.run.smithi023.stdout:osd.7: {} 2024-03-11T20:58:08.695 INFO:teuthology.orchestra.run.smithi023.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T20:58:08.695 INFO:teuthology.orchestra.run.smithi023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-11T20:58:08.696 INFO:teuthology.orchestra.run.smithi023.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T20:58:08.696 INFO:teuthology.orchestra.run.smithi023.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T20:58:08.696 INFO:teuthology.orchestra.run.smithi023.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T20:58:08.696 INFO:teuthology.orchestra.run.smithi023.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T20:58:08.696 INFO:teuthology.orchestra.run.smithi023.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T20:58:08.696 INFO:teuthology.orchestra.run.smithi023.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T20:58:08.696 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:08.721 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:08.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:08.728 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:58:08.829 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:08.848 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:08.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:08.850 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:58:08.951 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:08.980 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-11T20:58:08.981 INFO:teuthology.orchestra.run.smithi023.stdout: "size": 20, 2024-03-11T20:58:08.981 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 600, 2024-03-11T20:58:08.981 INFO:teuthology.orchestra.run.smithi023.stdout: "ops": [ 2024-03-11T20:58:08.981 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:08.981 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_op(client.24281.0:48 4.0 4:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-03-11T20:58:08.981 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T20:50:32.202140+0000", 2024-03-11T20:58:08.981 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 456.775370377, 2024-03-11T20:58:08.981 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.45329894999999998, 2024-03-11T20:58:08.981 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T20:58:08.981 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T20:58:08.981 INFO:teuthology.orchestra.run.smithi023.stdout: "client_info": { 2024-03-11T20:58:08.981 INFO:teuthology.orchestra.run.smithi023.stdout: "client": "client.24281", 2024-03-11T20:58:08.981 INFO:teuthology.orchestra.run.smithi023.stdout: "client_addr": "172.21.15.23:0/2185274774", 2024-03-11T20:58:08.982 INFO:teuthology.orchestra.run.smithi023.stdout: "tid": 48 2024-03-11T20:58:08.982 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:08.982 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T20:58:08.982 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:08.982 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T20:58:08.982 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:50:32.202140+0000", 2024-03-11T20:58:08.982 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:58:08.982 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:08.982 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:08.982 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T20:58:08.982 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:50:32.202140+0000", 2024-03-11T20:58:08.982 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:58:08.982 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:08.983 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:08.983 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T20:58:08.983 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:50:32.202143+0000", 2024-03-11T20:58:08.983 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.7269999999999999e-06 2024-03-11T20:58:08.983 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:08.983 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:08.983 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T20:58:08.983 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:50:32.202153+0000", 2024-03-11T20:58:08.983 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.0516000000000001e-05 2024-03-11T20:58:08.983 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:08.983 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:08.983 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T20:58:08.983 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:50:32.202185+0000", 2024-03-11T20:58:08.983 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 3.1359999999999998e-05 2024-03-11T20:58:08.984 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:08.984 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:08.984 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T20:58:08.984 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:50:32.202208+0000", 2024-03-11T20:58:08.984 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.3640000000000001e-05 2024-03-11T20:58:08.984 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:08.984 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:08.984 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T20:58:08.984 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:50:32.211797+0000", 2024-03-11T20:58:08.984 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0095880270000000007 2024-03-11T20:58:08.984 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:08.984 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:08.984 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for peered", 2024-03-11T20:58:08.985 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:50:32.211808+0000", 2024-03-11T20:58:08.985 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.1389000000000001e-05 2024-03-11T20:58:08.985 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:08.985 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:08.985 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T20:58:08.985 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:50:32.647871+0000", 2024-03-11T20:58:08.985 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.43606277199999999 2024-03-11T20:58:08.985 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:08.985 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:08.985 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T20:58:08.985 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:50:32.648166+0000", 2024-03-11T20:58:08.985 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00029488799999999999 2024-03-11T20:58:08.985 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:08.985 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:08.986 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for subops from 0,7", 2024-03-11T20:58:08.986 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:50:32.648560+0000", 2024-03-11T20:58:08.986 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00039436400000000002 2024-03-11T20:58:08.986 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:08.986 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:08.986 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "op_commit", 2024-03-11T20:58:08.986 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:50:32.654165+0000", 2024-03-11T20:58:08.986 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0056054269999999996 2024-03-11T20:58:08.986 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:08.986 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:08.986 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T20:58:08.986 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:50:32.654987+0000", 2024-03-11T20:58:08.986 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000822074 2024-03-11T20:58:08.987 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:08.987 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:08.987 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T20:58:08.987 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:50:32.655247+0000", 2024-03-11T20:58:08.987 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00025975500000000002 2024-03-11T20:58:08.987 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:08.987 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:08.987 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T20:58:08.987 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:50:32.655346+0000", 2024-03-11T20:58:08.987 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 9.8641999999999995e-05 2024-03-11T20:58:08.987 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:08.987 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:08.987 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T20:58:08.988 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:50:32.655439+0000", 2024-03-11T20:58:08.988 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 9.3368999999999993e-05 2024-03-11T20:58:08.988 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:58:08.988 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T20:58:08.988 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:58:08.988 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:08.988 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:08.988 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_op(client.14412.0:46 4.0 4:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-03-11T20:58:08.988 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T20:50:32.204310+0000", 2024-03-11T20:58:08.988 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 456.77320065100002, 2024-03-11T20:58:08.988 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.45164960399999998, 2024-03-11T20:58:08.988 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T20:58:08.988 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T20:58:08.988 INFO:teuthology.orchestra.run.smithi023.stdout: "client_info": { 2024-03-11T20:58:08.989 INFO:teuthology.orchestra.run.smithi023.stdout: "client": "client.14412", 2024-03-11T20:58:08.989 INFO:teuthology.orchestra.run.smithi023.stdout: "client_addr": "172.21.15.23:0/4257498094", 2024-03-11T20:58:08.989 INFO:teuthology.orchestra.run.smithi023.stdout: "tid": 46 2024-03-11T20:58:08.989 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:08.989 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T20:58:08.989 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:08.989 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T20:58:08.989 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:50:32.204310+0000", 2024-03-11T20:58:08.989 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:58:08.989 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:08.989 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:08.989 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T20:58:08.989 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:50:32.204310+0000", 2024-03-11T20:58:08.989 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:58:08.989 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:08.990 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:08.990 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T20:58:08.990 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:50:32.204311+0000", 2024-03-11T20:58:08.990 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.3319999999999999e-06 2024-03-11T20:58:08.990 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:08.990 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:08.990 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T20:58:08.990 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:50:32.204316+0000", 2024-03-11T20:58:08.990 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 4.8899999999999998e-06 2024-03-11T20:58:08.990 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:08.990 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:08.990 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T20:58:08.990 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:50:32.204329+0000", 2024-03-11T20:58:08.991 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.2955e-05 2024-03-11T20:58:08.991 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:08.991 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:08.991 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T20:58:08.991 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:50:32.204341+0000", 2024-03-11T20:58:08.991 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.2101e-05 2024-03-11T20:58:08.991 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:08.991 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:08.991 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T20:58:08.991 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:50:32.211831+0000", 2024-03-11T20:58:08.991 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0074898370000000001 2024-03-11T20:58:08.991 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:08.991 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:08.991 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for peered", 2024-03-11T20:58:08.991 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:50:32.211837+0000", 2024-03-11T20:58:08.992 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 5.9580000000000004e-06 2024-03-11T20:58:08.992 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:08.992 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:08.992 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T20:58:08.992 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:50:32.651094+0000", 2024-03-11T20:58:08.992 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.43925671900000002 2024-03-11T20:58:08.992 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:08.992 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:08.992 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T20:58:08.992 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:50:32.651327+0000", 2024-03-11T20:58:08.992 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00023367200000000001 2024-03-11T20:58:08.992 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:08.992 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:08.992 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for subops from 0,7", 2024-03-11T20:58:08.993 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:50:32.651708+0000", 2024-03-11T20:58:08.993 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00038037100000000001 2024-03-11T20:58:08.993 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:08.993 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:08.993 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "op_commit", 2024-03-11T20:58:08.993 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:50:32.654331+0000", 2024-03-11T20:58:08.993 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.002623584 2024-03-11T20:58:08.993 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:08.993 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:08.993 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T20:58:08.993 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:50:32.655564+0000", 2024-03-11T20:58:08.993 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.001232421 2024-03-11T20:58:08.993 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:08.993 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:08.993 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T20:58:08.994 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:50:32.655690+0000", 2024-03-11T20:58:08.994 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00012616600000000001 2024-03-11T20:58:08.994 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:08.994 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:08.994 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T20:58:08.994 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:50:32.655883+0000", 2024-03-11T20:58:08.994 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000193289 2024-03-11T20:58:08.994 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:08.994 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:08.994 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T20:58:08.994 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:50:32.655960+0000", 2024-03-11T20:58:08.994 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.6309000000000004e-05 2024-03-11T20:58:08.994 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:58:08.994 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T20:58:08.995 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:58:08.995 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:08.995 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:08.995 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_op(client.24629.0:46652 7.a 7:531a34cf:::benchmark_data_smithi023_67959_object46651:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-03-11T20:58:08.995 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T20:55:16.107764+0000", 2024-03-11T20:58:08.995 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 172.86974640299999, 2024-03-11T20:58:08.995 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.33282781900000002, 2024-03-11T20:58:08.995 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T20:58:08.995 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T20:58:08.995 INFO:teuthology.orchestra.run.smithi023.stdout: "client_info": { 2024-03-11T20:58:08.995 INFO:teuthology.orchestra.run.smithi023.stdout: "client": "client.24629", 2024-03-11T20:58:08.995 INFO:teuthology.orchestra.run.smithi023.stdout: "client_addr": "172.21.15.23:0/959732609", 2024-03-11T20:58:08.995 INFO:teuthology.orchestra.run.smithi023.stdout: "tid": 46652 2024-03-11T20:58:08.995 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:08.995 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T20:58:08.996 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:08.996 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T20:58:08.996 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:16.107764+0000", 2024-03-11T20:58:08.996 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:58:08.996 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:08.996 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:08.996 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T20:58:08.996 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:16.107764+0000", 2024-03-11T20:58:08.996 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:58:08.996 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:08.996 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:08.996 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T20:58:08.996 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:16.107766+0000", 2024-03-11T20:58:08.996 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.342e-06 2024-03-11T20:58:08.997 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:08.997 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:08.997 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T20:58:08.997 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:16.107788+0000", 2024-03-11T20:58:08.997 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.2385e-05 2024-03-11T20:58:08.997 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:08.997 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:08.997 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T20:58:08.997 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:16.107801+0000", 2024-03-11T20:58:08.997 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.272e-05 2024-03-11T20:58:08.997 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:08.997 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:08.997 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T20:58:08.997 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:16.107806+0000", 2024-03-11T20:58:08.997 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 5.6189999999999999e-06 2024-03-11T20:58:08.998 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:08.998 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:08.998 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T20:58:08.998 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:16.108843+0000", 2024-03-11T20:58:08.998 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0010366520000000001 2024-03-11T20:58:08.998 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:08.998 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:08.998 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T20:58:08.998 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:16.108980+0000", 2024-03-11T20:58:08.998 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00013694299999999999 2024-03-11T20:58:08.998 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:08.998 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:08.998 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for subops from 2,5", 2024-03-11T20:58:08.999 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:16.109135+0000", 2024-03-11T20:58:08.999 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00015526900000000001 2024-03-11T20:58:08.999 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:08.999 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:08.999 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "op_commit", 2024-03-11T20:58:08.999 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:16.111103+0000", 2024-03-11T20:58:08.999 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0019676049999999999 2024-03-11T20:58:08.999 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:08.999 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:08.999 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T20:58:08.999 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:16.111281+0000", 2024-03-11T20:58:08.999 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00017801500000000001 2024-03-11T20:58:08.999 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:08.999 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:08.999 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T20:58:09.000 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:16.439290+0000", 2024-03-11T20:58:09.000 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.32800967199999997 2024-03-11T20:58:09.000 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.000 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.000 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T20:58:09.000 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:16.440031+0000", 2024-03-11T20:58:09.000 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00074045000000000003 2024-03-11T20:58:09.000 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.000 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.000 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T20:58:09.000 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:16.440592+0000", 2024-03-11T20:58:09.000 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00056114700000000001 2024-03-11T20:58:09.000 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:58:09.000 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T20:58:09.000 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:58:09.001 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.001 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.001 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_op(client.24629.0:46661 7.a 7:547e7bcf:::benchmark_data_smithi023_67959_object46660:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-03-11T20:58:09.001 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T20:55:16.129528+0000", 2024-03-11T20:58:09.001 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 172.84798225399999, 2024-03-11T20:58:09.001 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.32329965199999999, 2024-03-11T20:58:09.001 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T20:58:09.001 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T20:58:09.001 INFO:teuthology.orchestra.run.smithi023.stdout: "client_info": { 2024-03-11T20:58:09.001 INFO:teuthology.orchestra.run.smithi023.stdout: "client": "client.24629", 2024-03-11T20:58:09.001 INFO:teuthology.orchestra.run.smithi023.stdout: "client_addr": "172.21.15.23:0/959732609", 2024-03-11T20:58:09.001 INFO:teuthology.orchestra.run.smithi023.stdout: "tid": 46661 2024-03-11T20:58:09.001 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.001 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T20:58:09.002 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.002 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T20:58:09.002 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:16.129528+0000", 2024-03-11T20:58:09.002 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:58:09.002 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.002 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.002 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T20:58:09.002 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:16.129528+0000", 2024-03-11T20:58:09.002 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:58:09.002 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.002 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.002 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T20:58:09.002 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:16.129530+0000", 2024-03-11T20:58:09.003 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.649e-06 2024-03-11T20:58:09.003 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.003 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.003 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T20:58:09.003 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:16.129552+0000", 2024-03-11T20:58:09.003 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.2084999999999999e-05 2024-03-11T20:58:09.003 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.003 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.003 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T20:58:09.003 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:16.129570+0000", 2024-03-11T20:58:09.003 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.7529000000000001e-05 2024-03-11T20:58:09.003 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.003 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.003 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T20:58:09.003 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:16.129579+0000", 2024-03-11T20:58:09.004 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 9.1419999999999999e-06 2024-03-11T20:58:09.004 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.004 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.004 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T20:58:09.004 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:16.129645+0000", 2024-03-11T20:58:09.004 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.6005000000000002e-05 2024-03-11T20:58:09.004 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.004 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.004 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T20:58:09.004 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:16.129792+0000", 2024-03-11T20:58:09.004 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000147653 2024-03-11T20:58:09.004 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.004 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.004 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for subops from 2,5", 2024-03-11T20:58:09.005 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:16.129964+0000", 2024-03-11T20:58:09.005 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00017118499999999999 2024-03-11T20:58:09.005 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.005 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.005 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T20:58:09.005 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:16.132228+0000", 2024-03-11T20:58:09.005 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0022645550000000001 2024-03-11T20:58:09.005 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.005 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.005 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "op_commit", 2024-03-11T20:58:09.005 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:16.132306+0000", 2024-03-11T20:58:09.005 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.8065000000000006e-05 2024-03-11T20:58:09.005 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.005 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.005 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T20:58:09.006 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:16.452085+0000", 2024-03-11T20:58:09.006 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.31977834900000002 2024-03-11T20:58:09.006 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.006 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.006 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T20:58:09.006 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:16.452625+0000", 2024-03-11T20:58:09.006 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00054029100000000004 2024-03-11T20:58:09.006 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.006 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.006 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T20:58:09.006 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:16.452828+0000", 2024-03-11T20:58:09.006 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00020314399999999999 2024-03-11T20:58:09.006 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:58:09.007 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T20:58:09.007 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:58:09.007 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.007 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.007 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_op(client.24629.0:46664 7.2 7:4d47a772:::benchmark_data_smithi023_67959_object46663:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-03-11T20:58:09.007 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T20:55:16.132343+0000", 2024-03-11T20:58:09.007 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 172.84516775, 2024-03-11T20:58:09.007 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.32274592200000002, 2024-03-11T20:58:09.007 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T20:58:09.007 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T20:58:09.007 INFO:teuthology.orchestra.run.smithi023.stdout: "client_info": { 2024-03-11T20:58:09.007 INFO:teuthology.orchestra.run.smithi023.stdout: "client": "client.24629", 2024-03-11T20:58:09.007 INFO:teuthology.orchestra.run.smithi023.stdout: "client_addr": "172.21.15.23:0/959732609", 2024-03-11T20:58:09.007 INFO:teuthology.orchestra.run.smithi023.stdout: "tid": 46664 2024-03-11T20:58:09.008 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.008 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T20:58:09.008 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.008 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T20:58:09.008 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:16.132343+0000", 2024-03-11T20:58:09.008 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:58:09.008 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.008 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.008 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T20:58:09.008 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:16.132343+0000", 2024-03-11T20:58:09.008 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:58:09.008 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.008 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.008 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T20:58:09.009 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:16.132344+0000", 2024-03-11T20:58:09.009 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.1540000000000001e-06 2024-03-11T20:58:09.009 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.009 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.009 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T20:58:09.009 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:16.132362+0000", 2024-03-11T20:58:09.009 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.8283000000000001e-05 2024-03-11T20:58:09.009 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.009 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.009 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T20:58:09.009 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:16.132372+0000", 2024-03-11T20:58:09.009 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.005e-05 2024-03-11T20:58:09.009 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.009 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.010 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T20:58:09.010 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:16.132378+0000", 2024-03-11T20:58:09.010 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 5.767e-06 2024-03-11T20:58:09.010 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.010 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.010 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T20:58:09.010 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:16.132430+0000", 2024-03-11T20:58:09.010 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 5.1511999999999997e-05 2024-03-11T20:58:09.010 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.010 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.010 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T20:58:09.010 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:16.132560+0000", 2024-03-11T20:58:09.010 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00013080700000000001 2024-03-11T20:58:09.011 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.011 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.011 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for subops from 2,7", 2024-03-11T20:58:09.011 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:16.132708+0000", 2024-03-11T20:58:09.011 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00014710799999999999 2024-03-11T20:58:09.011 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.011 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.011 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "op_commit", 2024-03-11T20:58:09.011 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:16.135939+0000", 2024-03-11T20:58:09.011 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0032312809999999999 2024-03-11T20:58:09.011 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.011 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.011 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T20:58:09.011 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:16.135984+0000", 2024-03-11T20:58:09.012 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 4.5466000000000001e-05 2024-03-11T20:58:09.012 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.012 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.012 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T20:58:09.012 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:16.454869+0000", 2024-03-11T20:58:09.012 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.31888439200000002 2024-03-11T20:58:09.012 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.012 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.012 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T20:58:09.012 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:16.455029+0000", 2024-03-11T20:58:09.012 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00016065799999999999 2024-03-11T20:58:09.012 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.012 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.012 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T20:58:09.013 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:16.455089+0000", 2024-03-11T20:58:09.013 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 5.9444000000000002e-05 2024-03-11T20:58:09.013 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:58:09.013 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T20:58:09.013 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:58:09.013 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.013 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.013 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_op(client.24629.0:85826 7.3 7:c712866b:::benchmark_data_smithi023_67959_object85825:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-03-11T20:58:09.013 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T20:55:52.509201+0000", 2024-03-11T20:58:09.013 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 136.46830934799999, 2024-03-11T20:58:09.013 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.41218749300000002, 2024-03-11T20:58:09.013 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T20:58:09.013 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T20:58:09.013 INFO:teuthology.orchestra.run.smithi023.stdout: "client_info": { 2024-03-11T20:58:09.014 INFO:teuthology.orchestra.run.smithi023.stdout: "client": "client.24629", 2024-03-11T20:58:09.014 INFO:teuthology.orchestra.run.smithi023.stdout: "client_addr": "172.21.15.23:0/959732609", 2024-03-11T20:58:09.014 INFO:teuthology.orchestra.run.smithi023.stdout: "tid": 85826 2024-03-11T20:58:09.014 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.014 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T20:58:09.014 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.014 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T20:58:09.014 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.509201+0000", 2024-03-11T20:58:09.014 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:58:09.014 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.014 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.014 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T20:58:09.014 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.509201+0000", 2024-03-11T20:58:09.014 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:58:09.014 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.015 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.015 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T20:58:09.015 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.509202+0000", 2024-03-11T20:58:09.015 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.1039999999999999e-06 2024-03-11T20:58:09.015 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.015 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.015 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T20:58:09.015 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.509219+0000", 2024-03-11T20:58:09.015 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.6192999999999999e-05 2024-03-11T20:58:09.015 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.015 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.015 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T20:58:09.015 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.509228+0000", 2024-03-11T20:58:09.015 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 9.2459999999999992e-06 2024-03-11T20:58:09.015 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.016 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.016 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T20:58:09.016 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.509233+0000", 2024-03-11T20:58:09.016 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 5.4040000000000003e-06 2024-03-11T20:58:09.016 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.016 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.016 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T20:58:09.016 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.509286+0000", 2024-03-11T20:58:09.016 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 5.2339000000000002e-05 2024-03-11T20:58:09.016 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.016 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.016 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T20:58:09.016 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.509407+0000", 2024-03-11T20:58:09.016 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00012103 2024-03-11T20:58:09.017 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.017 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.017 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for subops from 2,4", 2024-03-11T20:58:09.017 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.905827+0000", 2024-03-11T20:58:09.017 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.396419931 2024-03-11T20:58:09.017 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.017 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.017 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T20:58:09.017 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.920003+0000", 2024-03-11T20:58:09.017 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.014176393000000001 2024-03-11T20:58:09.017 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.017 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.017 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "op_commit", 2024-03-11T20:58:09.017 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.920186+0000", 2024-03-11T20:58:09.017 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00018330500000000001 2024-03-11T20:58:09.018 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.018 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.018 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T20:58:09.018 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.921229+0000", 2024-03-11T20:58:09.018 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0010428049999999999 2024-03-11T20:58:09.018 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.018 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.018 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T20:58:09.018 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.921332+0000", 2024-03-11T20:58:09.018 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00010282599999999999 2024-03-11T20:58:09.018 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.018 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.018 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T20:58:09.018 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.921389+0000", 2024-03-11T20:58:09.019 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 5.6916999999999997e-05 2024-03-11T20:58:09.019 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:58:09.019 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T20:58:09.019 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:58:09.019 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.019 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.019 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_repop(client.24629.0:85823 7.c e145/121 7:3f3c9d65:::benchmark_data_smithi023_67959_object85822:head v 145'5263, mlcod=145'5261)", 2024-03-11T20:58:09.019 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T20:55:52.509289+0000", 2024-03-11T20:58:09.019 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 136.46822130199999, 2024-03-11T20:58:09.019 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.39653492800000001, 2024-03-11T20:58:09.019 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T20:58:09.019 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T20:58:09.019 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T20:58:09.019 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.019 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T20:58:09.020 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.509289+0000", 2024-03-11T20:58:09.020 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:58:09.020 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.020 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.020 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T20:58:09.020 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.509289+0000", 2024-03-11T20:58:09.020 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:58:09.020 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.020 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.020 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T20:58:09.020 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.509290+0000", 2024-03-11T20:58:09.020 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.9999999999999996e-07 2024-03-11T20:58:09.020 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.020 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.020 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T20:58:09.021 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.509334+0000", 2024-03-11T20:58:09.021 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 4.35e-05 2024-03-11T20:58:09.021 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.021 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.021 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T20:58:09.021 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.509342+0000", 2024-03-11T20:58:09.021 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.9799999999999998e-06 2024-03-11T20:58:09.021 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.021 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.021 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T20:58:09.021 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.509346+0000", 2024-03-11T20:58:09.021 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 4.5800000000000002e-06 2024-03-11T20:58:09.021 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.021 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.022 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T20:58:09.022 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.509389+0000", 2024-03-11T20:58:09.022 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 4.2475999999999998e-05 2024-03-11T20:58:09.022 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.022 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.022 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T20:58:09.022 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.509402+0000", 2024-03-11T20:58:09.022 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.3222e-05 2024-03-11T20:58:09.022 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.022 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.022 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T20:58:09.022 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.905756+0000", 2024-03-11T20:58:09.022 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.39635374699999998 2024-03-11T20:58:09.022 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.023 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.023 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T20:58:09.023 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.905824+0000", 2024-03-11T20:58:09.023 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.8622999999999997e-05 2024-03-11T20:58:09.023 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:58:09.023 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T20:58:09.023 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:58:09.023 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.023 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.023 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_repop(client.24629.0:85827 7.0 e145/124 7:0ef91f2b:::benchmark_data_smithi023_67959_object85826:head v 145'5268, mlcod=145'5266)", 2024-03-11T20:58:09.023 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T20:55:52.513894+0000", 2024-03-11T20:58:09.023 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 136.46361665800001, 2024-03-11T20:58:09.023 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.40481390499999997, 2024-03-11T20:58:09.023 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T20:58:09.024 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T20:58:09.024 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T20:58:09.024 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.024 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T20:58:09.024 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.513894+0000", 2024-03-11T20:58:09.024 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:58:09.024 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.024 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.024 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T20:58:09.024 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.513894+0000", 2024-03-11T20:58:09.024 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:58:09.024 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.024 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.024 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T20:58:09.024 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.513895+0000", 2024-03-11T20:58:09.025 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.049e-06 2024-03-11T20:58:09.025 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.025 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.025 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T20:58:09.025 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.513945+0000", 2024-03-11T20:58:09.025 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 5.0083999999999997e-05 2024-03-11T20:58:09.025 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.025 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.025 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T20:58:09.025 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.905665+0000", 2024-03-11T20:58:09.025 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.39171999499999999 2024-03-11T20:58:09.025 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.025 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.025 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T20:58:09.026 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.905689+0000", 2024-03-11T20:58:09.026 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.3745999999999999e-05 2024-03-11T20:58:09.026 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.026 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.026 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T20:58:09.026 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.905838+0000", 2024-03-11T20:58:09.026 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00014893099999999999 2024-03-11T20:58:09.026 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.026 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.026 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T20:58:09.026 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.905881+0000", 2024-03-11T20:58:09.026 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 4.2766999999999999e-05 2024-03-11T20:58:09.026 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.026 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.027 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T20:58:09.027 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.918659+0000", 2024-03-11T20:58:09.027 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.012777973 2024-03-11T20:58:09.027 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.027 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.027 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T20:58:09.027 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.918708+0000", 2024-03-11T20:58:09.027 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 4.9360000000000002e-05 2024-03-11T20:58:09.027 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:58:09.027 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T20:58:09.027 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:58:09.027 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.027 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.027 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_op(client.24629.0:85831 7.2 7:4fbf6237:::benchmark_data_smithi023_67959_object85830:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-03-11T20:58:09.028 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T20:55:52.527890+0000", 2024-03-11T20:58:09.028 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 136.449620356, 2024-03-11T20:58:09.028 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.39378672599999998, 2024-03-11T20:58:09.028 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T20:58:09.028 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T20:58:09.028 INFO:teuthology.orchestra.run.smithi023.stdout: "client_info": { 2024-03-11T20:58:09.028 INFO:teuthology.orchestra.run.smithi023.stdout: "client": "client.24629", 2024-03-11T20:58:09.028 INFO:teuthology.orchestra.run.smithi023.stdout: "client_addr": "172.21.15.23:0/959732609", 2024-03-11T20:58:09.028 INFO:teuthology.orchestra.run.smithi023.stdout: "tid": 85831 2024-03-11T20:58:09.028 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.028 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T20:58:09.028 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.028 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T20:58:09.028 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.527890+0000", 2024-03-11T20:58:09.029 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:58:09.029 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.029 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.029 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T20:58:09.029 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.527890+0000", 2024-03-11T20:58:09.029 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:58:09.029 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.029 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.029 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T20:58:09.029 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.527892+0000", 2024-03-11T20:58:09.029 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.3570000000000001e-06 2024-03-11T20:58:09.029 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.029 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.029 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T20:58:09.029 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.527913+0000", 2024-03-11T20:58:09.030 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.1311e-05 2024-03-11T20:58:09.030 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.030 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.030 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T20:58:09.030 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.905662+0000", 2024-03-11T20:58:09.030 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.37774891700000002 2024-03-11T20:58:09.030 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.030 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.030 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T20:58:09.030 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.905691+0000", 2024-03-11T20:58:09.030 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.9515000000000001e-05 2024-03-11T20:58:09.030 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.030 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.030 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T20:58:09.031 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.905885+0000", 2024-03-11T20:58:09.031 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00019332400000000001 2024-03-11T20:58:09.031 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.031 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.031 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T20:58:09.031 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.906083+0000", 2024-03-11T20:58:09.031 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000198572 2024-03-11T20:58:09.031 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.031 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.031 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for subops from 2,7", 2024-03-11T20:58:09.031 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.906303+0000", 2024-03-11T20:58:09.031 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00021959 2024-03-11T20:58:09.031 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.031 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.031 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T20:58:09.032 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.912014+0000", 2024-03-11T20:58:09.032 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0057112049999999996 2024-03-11T20:58:09.032 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.032 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.032 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "op_commit", 2024-03-11T20:58:09.032 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.918631+0000", 2024-03-11T20:58:09.032 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0066167470000000001 2024-03-11T20:58:09.032 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.032 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.032 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T20:58:09.032 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.921223+0000", 2024-03-11T20:58:09.032 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0025917789999999998 2024-03-11T20:58:09.032 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.032 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.032 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T20:58:09.032 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.921614+0000", 2024-03-11T20:58:09.033 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00039175700000000001 2024-03-11T20:58:09.033 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.033 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.033 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T20:58:09.033 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.921677+0000", 2024-03-11T20:58:09.033 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.2651999999999996e-05 2024-03-11T20:58:09.033 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:58:09.033 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T20:58:09.033 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:58:09.033 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.033 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.033 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_repop(client.24629.0:85833 7.d e145/124 7:bfb2a6d1:::benchmark_data_smithi023_67959_object85832:head v 145'5379, mlcod=145'5377)", 2024-03-11T20:58:09.033 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T20:55:52.528715+0000", 2024-03-11T20:58:09.033 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 136.44879531000001, 2024-03-11T20:58:09.034 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.39186654100000001, 2024-03-11T20:58:09.034 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T20:58:09.034 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T20:58:09.034 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T20:58:09.034 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.034 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T20:58:09.034 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.528715+0000", 2024-03-11T20:58:09.034 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:58:09.034 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.034 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.034 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T20:58:09.034 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.528715+0000", 2024-03-11T20:58:09.034 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:58:09.034 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.035 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.035 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T20:58:09.035 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.528716+0000", 2024-03-11T20:58:09.035 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 8.1399999999999996e-07 2024-03-11T20:58:09.035 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.035 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.035 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T20:58:09.035 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.528760+0000", 2024-03-11T20:58:09.035 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 4.3809e-05 2024-03-11T20:58:09.035 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.035 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.035 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T20:58:09.035 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.905687+0000", 2024-03-11T20:58:09.035 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.37692657099999999 2024-03-11T20:58:09.036 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.036 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.036 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T20:58:09.036 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.906752+0000", 2024-03-11T20:58:09.036 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0010655759999999999 2024-03-11T20:58:09.036 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.036 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.036 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T20:58:09.036 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.907542+0000", 2024-03-11T20:58:09.036 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00078997799999999999 2024-03-11T20:58:09.036 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.036 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.036 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T20:58:09.036 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.907566+0000", 2024-03-11T20:58:09.037 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.4051999999999999e-05 2024-03-11T20:58:09.037 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.037 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.037 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T20:58:09.037 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.920464+0000", 2024-03-11T20:58:09.037 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.012897764000000001 2024-03-11T20:58:09.037 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.037 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.037 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T20:58:09.037 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.920582+0000", 2024-03-11T20:58:09.037 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000117977 2024-03-11T20:58:09.037 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:58:09.037 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T20:58:09.037 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:58:09.038 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.038 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.038 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_op(client.24629.0:126332 7.a 7:5f290de1:::benchmark_data_smithi023_67959_object16091:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-03-11T20:58:09.038 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T20:56:39.175300+0000", 2024-03-11T20:58:09.038 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 89.802210627999997, 2024-03-11T20:58:09.038 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.44305815700000001, 2024-03-11T20:58:09.038 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T20:58:09.038 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T20:58:09.038 INFO:teuthology.orchestra.run.smithi023.stdout: "client_info": { 2024-03-11T20:58:09.038 INFO:teuthology.orchestra.run.smithi023.stdout: "client": "client.24629", 2024-03-11T20:58:09.038 INFO:teuthology.orchestra.run.smithi023.stdout: "client_addr": "172.21.15.23:0/959732609", 2024-03-11T20:58:09.038 INFO:teuthology.orchestra.run.smithi023.stdout: "tid": 126332 2024-03-11T20:58:09.038 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.038 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T20:58:09.039 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.039 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T20:58:09.039 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.175300+0000", 2024-03-11T20:58:09.039 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:58:09.039 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.039 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.039 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T20:58:09.039 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.175300+0000", 2024-03-11T20:58:09.039 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:58:09.039 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.039 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.039 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T20:58:09.039 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.175300+0000", 2024-03-11T20:58:09.039 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 3.1e-07 2024-03-11T20:58:09.040 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.040 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.040 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T20:58:09.040 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.175302+0000", 2024-03-11T20:58:09.040 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.0420000000000001e-06 2024-03-11T20:58:09.040 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.040 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.040 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T20:58:09.040 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.175307+0000", 2024-03-11T20:58:09.040 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 4.3889999999999998e-06 2024-03-11T20:58:09.040 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.040 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.040 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T20:58:09.040 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.175310+0000", 2024-03-11T20:58:09.040 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 3.2440000000000002e-06 2024-03-11T20:58:09.041 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.041 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.041 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T20:58:09.041 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.175362+0000", 2024-03-11T20:58:09.041 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 5.2441999999999997e-05 2024-03-11T20:58:09.041 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.041 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.041 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T20:58:09.041 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.175462+0000", 2024-03-11T20:58:09.041 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 9.9955000000000001e-05 2024-03-11T20:58:09.041 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.041 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.041 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for subops from 2,5", 2024-03-11T20:58:09.041 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.175532+0000", 2024-03-11T20:58:09.042 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.9468000000000006e-05 2024-03-11T20:58:09.042 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.042 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.042 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "op_commit", 2024-03-11T20:58:09.042 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.178041+0000", 2024-03-11T20:58:09.042 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0025092420000000001 2024-03-11T20:58:09.042 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.042 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.042 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T20:58:09.042 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.184252+0000", 2024-03-11T20:58:09.042 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0062105429999999998 2024-03-11T20:58:09.042 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.042 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.042 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T20:58:09.042 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.617826+0000", 2024-03-11T20:58:09.043 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.43357458500000001 2024-03-11T20:58:09.043 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.043 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.043 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T20:58:09.043 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.618275+0000", 2024-03-11T20:58:09.043 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00044847099999999998 2024-03-11T20:58:09.043 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.043 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.043 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T20:58:09.043 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.618358+0000", 2024-03-11T20:58:09.043 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 8.3466000000000004e-05 2024-03-11T20:58:09.043 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:58:09.043 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T20:58:09.043 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:58:09.043 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.044 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.044 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_op(client.24629.0:126334 7.2 7:40a0664e:::benchmark_data_smithi023_67959_object16093:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-03-11T20:58:09.044 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T20:56:39.175323+0000", 2024-03-11T20:58:09.044 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 89.802187262000004, 2024-03-11T20:58:09.044 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.44290166199999997, 2024-03-11T20:58:09.044 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T20:58:09.044 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T20:58:09.044 INFO:teuthology.orchestra.run.smithi023.stdout: "client_info": { 2024-03-11T20:58:09.044 INFO:teuthology.orchestra.run.smithi023.stdout: "client": "client.24629", 2024-03-11T20:58:09.044 INFO:teuthology.orchestra.run.smithi023.stdout: "client_addr": "172.21.15.23:0/959732609", 2024-03-11T20:58:09.044 INFO:teuthology.orchestra.run.smithi023.stdout: "tid": 126334 2024-03-11T20:58:09.044 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.044 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T20:58:09.044 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.045 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T20:58:09.045 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.175323+0000", 2024-03-11T20:58:09.045 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:58:09.045 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.045 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.045 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T20:58:09.045 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.175323+0000", 2024-03-11T20:58:09.045 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:58:09.045 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.045 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.045 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T20:58:09.045 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.175324+0000", 2024-03-11T20:58:09.045 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.7700000000000001e-07 2024-03-11T20:58:09.045 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.046 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.046 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T20:58:09.046 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.175325+0000", 2024-03-11T20:58:09.046 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.84e-06 2024-03-11T20:58:09.046 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.046 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.046 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T20:58:09.046 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.175329+0000", 2024-03-11T20:58:09.046 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 3.5410000000000001e-06 2024-03-11T20:58:09.046 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.046 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.046 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T20:58:09.046 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.175332+0000", 2024-03-11T20:58:09.046 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 3.1070000000000001e-06 2024-03-11T20:58:09.046 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.047 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.047 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T20:58:09.047 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.176183+0000", 2024-03-11T20:58:09.047 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00085044600000000004 2024-03-11T20:58:09.047 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.047 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.047 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T20:58:09.047 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.176286+0000", 2024-03-11T20:58:09.047 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000103442 2024-03-11T20:58:09.047 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.047 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.047 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for subops from 2,7", 2024-03-11T20:58:09.047 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.176394+0000", 2024-03-11T20:58:09.047 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000108007 2024-03-11T20:58:09.048 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.048 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.048 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "op_commit", 2024-03-11T20:58:09.048 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.178567+0000", 2024-03-11T20:58:09.048 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0021733519999999999 2024-03-11T20:58:09.048 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.048 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.048 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T20:58:09.048 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.179871+0000", 2024-03-11T20:58:09.048 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.001303207 2024-03-11T20:58:09.048 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.048 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.048 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T20:58:09.049 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.618044+0000", 2024-03-11T20:58:09.049 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.43817315699999998 2024-03-11T20:58:09.049 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.049 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.049 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T20:58:09.049 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.618171+0000", 2024-03-11T20:58:09.049 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00012737 2024-03-11T20:58:09.049 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.049 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.049 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T20:58:09.049 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.618225+0000", 2024-03-11T20:58:09.049 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 5.3915999999999997e-05 2024-03-11T20:58:09.049 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:58:09.049 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T20:58:09.050 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:58:09.050 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.050 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.050 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_op(client.24629.0:126340 7.2 7:41534ba6:::benchmark_data_smithi023_67959_object16099:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-03-11T20:58:09.050 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T20:56:39.179515+0000", 2024-03-11T20:58:09.050 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 89.797995549999996, 2024-03-11T20:58:09.050 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.43929723700000001, 2024-03-11T20:58:09.050 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T20:58:09.050 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T20:58:09.050 INFO:teuthology.orchestra.run.smithi023.stdout: "client_info": { 2024-03-11T20:58:09.050 INFO:teuthology.orchestra.run.smithi023.stdout: "client": "client.24629", 2024-03-11T20:58:09.050 INFO:teuthology.orchestra.run.smithi023.stdout: "client_addr": "172.21.15.23:0/959732609", 2024-03-11T20:58:09.050 INFO:teuthology.orchestra.run.smithi023.stdout: "tid": 126340 2024-03-11T20:58:09.050 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.051 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T20:58:09.051 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.051 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T20:58:09.051 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.179515+0000", 2024-03-11T20:58:09.051 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:58:09.051 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.051 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.051 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T20:58:09.051 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.179515+0000", 2024-03-11T20:58:09.051 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:58:09.051 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.051 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.051 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T20:58:09.051 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.179516+0000", 2024-03-11T20:58:09.052 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 4.5400000000000002e-07 2024-03-11T20:58:09.052 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.052 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.052 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T20:58:09.052 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.179519+0000", 2024-03-11T20:58:09.052 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 3.258e-06 2024-03-11T20:58:09.052 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.052 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.052 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T20:58:09.052 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.179527+0000", 2024-03-11T20:58:09.052 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.7800000000000001e-06 2024-03-11T20:58:09.052 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.052 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.052 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T20:58:09.052 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.179533+0000", 2024-03-11T20:58:09.053 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.0469999999999999e-06 2024-03-11T20:58:09.053 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.053 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.053 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T20:58:09.053 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.179914+0000", 2024-03-11T20:58:09.053 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00038138700000000001 2024-03-11T20:58:09.053 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.053 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.053 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T20:58:09.053 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.179987+0000", 2024-03-11T20:58:09.053 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.3115999999999996e-05 2024-03-11T20:58:09.053 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.053 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.053 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for subops from 2,7", 2024-03-11T20:58:09.054 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.182805+0000", 2024-03-11T20:58:09.054 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0028174749999999998 2024-03-11T20:58:09.054 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.054 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.054 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "op_commit", 2024-03-11T20:58:09.054 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.185080+0000", 2024-03-11T20:58:09.054 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0022755219999999999 2024-03-11T20:58:09.054 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.054 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.054 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T20:58:09.054 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.188917+0000", 2024-03-11T20:58:09.054 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0038373470000000001 2024-03-11T20:58:09.054 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.054 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.055 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T20:58:09.055 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.618453+0000", 2024-03-11T20:58:09.055 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.429535746 2024-03-11T20:58:09.055 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.055 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.055 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T20:58:09.055 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.618636+0000", 2024-03-11T20:58:09.055 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00018320000000000001 2024-03-11T20:58:09.055 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.055 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.055 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T20:58:09.055 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.618812+0000", 2024-03-11T20:58:09.055 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00017590499999999999 2024-03-11T20:58:09.055 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:58:09.056 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T20:58:09.056 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:58:09.056 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.056 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.056 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_op(client.24629.0:126341 7.a 7:5cab8f84:::benchmark_data_smithi023_67959_object16100:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-03-11T20:58:09.056 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T20:56:39.179749+0000", 2024-03-11T20:58:09.056 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 89.797761652000005, 2024-03-11T20:58:09.056 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.43930538099999999, 2024-03-11T20:58:09.056 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T20:58:09.056 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T20:58:09.056 INFO:teuthology.orchestra.run.smithi023.stdout: "client_info": { 2024-03-11T20:58:09.056 INFO:teuthology.orchestra.run.smithi023.stdout: "client": "client.24629", 2024-03-11T20:58:09.056 INFO:teuthology.orchestra.run.smithi023.stdout: "client_addr": "172.21.15.23:0/959732609", 2024-03-11T20:58:09.056 INFO:teuthology.orchestra.run.smithi023.stdout: "tid": 126341 2024-03-11T20:58:09.056 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.057 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T20:58:09.057 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.057 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T20:58:09.057 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.179749+0000", 2024-03-11T20:58:09.057 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:58:09.057 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.057 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.057 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T20:58:09.057 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.179749+0000", 2024-03-11T20:58:09.057 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:58:09.057 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.057 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.057 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T20:58:09.057 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.179750+0000", 2024-03-11T20:58:09.058 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.3200000000000005e-07 2024-03-11T20:58:09.058 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.058 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.058 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T20:58:09.058 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.179753+0000", 2024-03-11T20:58:09.058 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 3.6899999999999998e-06 2024-03-11T20:58:09.058 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.058 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.058 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T20:58:09.058 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.179762+0000", 2024-03-11T20:58:09.058 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 9.098e-06 2024-03-11T20:58:09.058 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.058 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.058 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T20:58:09.059 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.179769+0000", 2024-03-11T20:58:09.059 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.4110000000000001e-06 2024-03-11T20:58:09.059 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.059 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.059 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T20:58:09.059 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.179925+0000", 2024-03-11T20:58:09.059 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00015657900000000001 2024-03-11T20:58:09.059 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.059 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.059 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T20:58:09.059 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.180029+0000", 2024-03-11T20:58:09.059 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000104063 2024-03-11T20:58:09.059 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.059 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.060 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for subops from 2,5", 2024-03-11T20:58:09.060 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.180140+0000", 2024-03-11T20:58:09.060 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00011100899999999999 2024-03-11T20:58:09.060 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.060 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.060 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T20:58:09.061 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.184863+0000", 2024-03-11T20:58:09.061 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0047221269999999996 2024-03-11T20:58:09.061 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.061 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.061 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "op_commit", 2024-03-11T20:58:09.061 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.184874+0000", 2024-03-11T20:58:09.061 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.149e-05 2024-03-11T20:58:09.061 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.061 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.062 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T20:58:09.062 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.618759+0000", 2024-03-11T20:58:09.062 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.433884724 2024-03-11T20:58:09.062 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.062 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.062 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T20:58:09.062 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.618923+0000", 2024-03-11T20:58:09.062 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000164186 2024-03-11T20:58:09.062 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.062 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.062 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T20:58:09.062 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.619054+0000", 2024-03-11T20:58:09.062 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00013137199999999999 2024-03-11T20:58:09.062 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:58:09.063 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T20:58:09.063 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:58:09.063 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.063 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.063 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_op(client.24629.0:126362 7.3 7:cbd5d24b:::benchmark_data_smithi023_67959_object16121:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-03-11T20:58:09.063 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T20:56:39.200356+0000", 2024-03-11T20:58:09.063 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 89.777154479000004, 2024-03-11T20:58:09.063 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.44075034099999999, 2024-03-11T20:58:09.063 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T20:58:09.063 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T20:58:09.063 INFO:teuthology.orchestra.run.smithi023.stdout: "client_info": { 2024-03-11T20:58:09.063 INFO:teuthology.orchestra.run.smithi023.stdout: "client": "client.24629", 2024-03-11T20:58:09.063 INFO:teuthology.orchestra.run.smithi023.stdout: "client_addr": "172.21.15.23:0/959732609", 2024-03-11T20:58:09.064 INFO:teuthology.orchestra.run.smithi023.stdout: "tid": 126362 2024-03-11T20:58:09.064 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.064 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T20:58:09.064 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.064 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T20:58:09.064 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.200356+0000", 2024-03-11T20:58:09.064 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:58:09.064 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.064 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.064 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T20:58:09.064 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.200356+0000", 2024-03-11T20:58:09.064 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:58:09.064 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.064 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.065 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T20:58:09.065 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.200357+0000", 2024-03-11T20:58:09.065 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.2979999999999999e-06 2024-03-11T20:58:09.065 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.065 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.065 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T20:58:09.065 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.200363+0000", 2024-03-11T20:58:09.065 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.054e-06 2024-03-11T20:58:09.065 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.065 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.065 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T20:58:09.065 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.332575+0000", 2024-03-11T20:58:09.065 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.13221116899999999 2024-03-11T20:58:09.065 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.065 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.066 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T20:58:09.066 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.332608+0000", 2024-03-11T20:58:09.066 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 3.3596999999999997e-05 2024-03-11T20:58:09.066 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.066 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.066 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T20:58:09.066 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.332770+0000", 2024-03-11T20:58:09.066 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00016126899999999999 2024-03-11T20:58:09.066 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.066 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.066 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T20:58:09.066 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.333108+0000", 2024-03-11T20:58:09.066 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000338387 2024-03-11T20:58:09.066 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.067 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.067 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for subops from 2,4", 2024-03-11T20:58:09.067 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.333417+0000", 2024-03-11T20:58:09.067 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00030934400000000003 2024-03-11T20:58:09.067 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.067 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.067 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T20:58:09.067 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.341605+0000", 2024-03-11T20:58:09.067 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0081881609999999994 2024-03-11T20:58:09.067 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.067 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.067 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "op_commit", 2024-03-11T20:58:09.067 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.341632+0000", 2024-03-11T20:58:09.067 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.6917999999999999e-05 2024-03-11T20:58:09.067 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.068 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.068 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T20:58:09.068 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.640968+0000", 2024-03-11T20:58:09.068 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.29933543400000001 2024-03-11T20:58:09.068 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.068 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.068 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T20:58:09.068 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.641046+0000", 2024-03-11T20:58:09.068 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.8484000000000003e-05 2024-03-11T20:58:09.068 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.068 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.068 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T20:58:09.068 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.641106+0000", 2024-03-11T20:58:09.068 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.0226000000000002e-05 2024-03-11T20:58:09.069 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:58:09.069 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T20:58:09.069 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:58:09.069 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.069 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.069 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_op(client.24406.0:385 5.0 5:353ec8f9:::gateway.conf:head [getxattr epoch (1) in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e161)", 2024-03-11T20:58:09.069 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T20:57:23.587375+0000", 2024-03-11T20:58:09.069 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 45.390135690999998, 2024-03-11T20:58:09.069 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.0496242819999999, 2024-03-11T20:58:09.069 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T20:58:09.069 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "started", 2024-03-11T20:58:09.069 INFO:teuthology.orchestra.run.smithi023.stdout: "client_info": { 2024-03-11T20:58:09.069 INFO:teuthology.orchestra.run.smithi023.stdout: "client": "client.24406", 2024-03-11T20:58:09.070 INFO:teuthology.orchestra.run.smithi023.stdout: "client_addr": "172.21.15.146:0/3388716841", 2024-03-11T20:58:09.070 INFO:teuthology.orchestra.run.smithi023.stdout: "tid": 385 2024-03-11T20:58:09.070 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.070 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T20:58:09.070 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.070 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T20:58:09.070 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:23.587375+0000", 2024-03-11T20:58:09.070 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:58:09.070 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.070 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.070 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T20:58:09.070 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:23.587375+0000", 2024-03-11T20:58:09.070 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:58:09.070 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.070 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.071 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T20:58:09.071 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:23.587377+0000", 2024-03-11T20:58:09.071 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.6729999999999999e-06 2024-03-11T20:58:09.071 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.071 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.071 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T20:58:09.071 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:23.587382+0000", 2024-03-11T20:58:09.071 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 5.3580000000000002e-06 2024-03-11T20:58:09.071 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.071 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.071 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T20:58:09.071 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:23.587393+0000", 2024-03-11T20:58:09.071 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.0808000000000001e-05 2024-03-11T20:58:09.071 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.072 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.072 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T20:58:09.072 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:23.587403+0000", 2024-03-11T20:58:09.072 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 9.9059999999999997e-06 2024-03-11T20:58:09.072 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.072 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.072 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T20:58:09.072 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:23.614080+0000", 2024-03-11T20:58:09.072 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.026676998 2024-03-11T20:58:09.072 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.072 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.072 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for peered", 2024-03-11T20:58:09.072 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:23.614090+0000", 2024-03-11T20:58:09.072 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.0742e-05 2024-03-11T20:58:09.072 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.073 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.073 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T20:58:09.073 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:24.636718+0000", 2024-03-11T20:58:09.073 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.0226275650000001 2024-03-11T20:58:09.073 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.073 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.073 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T20:58:09.073 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:24.636854+0000", 2024-03-11T20:58:09.073 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00013643800000000001 2024-03-11T20:58:09.073 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.073 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.073 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T20:58:09.073 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:24.636999+0000", 2024-03-11T20:58:09.073 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00014479400000000001 2024-03-11T20:58:09.073 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:58:09.073 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T20:58:09.074 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:58:09.074 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.074 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.074 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_op(client.24281.0:1739 4.14 4:2b04a3e9:::notify.6:head [watch reconnect cookie 94110942021888 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e162)", 2024-03-11T20:58:09.074 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T20:57:23.591846+0000", 2024-03-11T20:58:09.074 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 45.385664921, 2024-03-11T20:58:09.074 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.075694433, 2024-03-11T20:58:09.074 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T20:58:09.074 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "started", 2024-03-11T20:58:09.074 INFO:teuthology.orchestra.run.smithi023.stdout: "client_info": { 2024-03-11T20:58:09.074 INFO:teuthology.orchestra.run.smithi023.stdout: "client": "client.24281", 2024-03-11T20:58:09.074 INFO:teuthology.orchestra.run.smithi023.stdout: "client_addr": "172.21.15.23:0/2185274774", 2024-03-11T20:58:09.074 INFO:teuthology.orchestra.run.smithi023.stdout: "tid": 1739 2024-03-11T20:58:09.074 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.074 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T20:58:09.075 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.075 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T20:58:09.075 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:23.591846+0000", 2024-03-11T20:58:09.075 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:58:09.075 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.075 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.075 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T20:58:09.075 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:23.591846+0000", 2024-03-11T20:58:09.075 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:58:09.075 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.075 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.075 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T20:58:09.075 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:23.591847+0000", 2024-03-11T20:58:09.075 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.0759999999999999e-06 2024-03-11T20:58:09.075 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.076 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.076 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T20:58:09.076 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:23.591854+0000", 2024-03-11T20:58:09.076 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.9539999999999998e-06 2024-03-11T20:58:09.076 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.076 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.076 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T20:58:09.076 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:23.591862+0000", 2024-03-11T20:58:09.076 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.9859999999999994e-06 2024-03-11T20:58:09.076 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.076 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.076 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T20:58:09.076 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:23.591868+0000", 2024-03-11T20:58:09.076 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.4019999999999999e-06 2024-03-11T20:58:09.076 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.077 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.077 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T20:58:09.077 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:23.612635+0000", 2024-03-11T20:58:09.077 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.020766413000000001 2024-03-11T20:58:09.077 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.077 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.077 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for peered", 2024-03-11T20:58:09.077 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:23.612646+0000", 2024-03-11T20:58:09.077 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.1812e-05 2024-03-11T20:58:09.077 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.077 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.077 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T20:58:09.077 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:24.639873+0000", 2024-03-11T20:58:09.077 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.0272262089999999 2024-03-11T20:58:09.077 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.078 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.078 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for degraded object", 2024-03-11T20:58:09.078 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:24.640210+0000", 2024-03-11T20:58:09.078 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00033792300000000001 2024-03-11T20:58:09.078 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.078 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.078 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T20:58:09.078 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:24.655501+0000", 2024-03-11T20:58:09.078 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.015290870999999999 2024-03-11T20:58:09.078 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.078 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.078 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T20:58:09.078 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:24.655589+0000", 2024-03-11T20:58:09.078 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 8.7645000000000005e-05 2024-03-11T20:58:09.078 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.079 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.079 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T20:58:09.079 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:24.667540+0000", 2024-03-11T20:58:09.079 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.011951142 2024-03-11T20:58:09.079 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:58:09.079 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T20:58:09.079 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:58:09.079 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.079 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.079 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_op(client.24629.0:167632 7.2 7:4fc417d5:::benchmark_data_smithi023_67959_object57391:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e162)", 2024-03-11T20:58:09.079 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T20:57:23.610529+0000", 2024-03-11T20:58:09.079 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 45.366981868000003, 2024-03-11T20:58:09.079 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.1886766679999998, 2024-03-11T20:58:09.079 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T20:58:09.079 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T20:58:09.079 INFO:teuthology.orchestra.run.smithi023.stdout: "client_info": { 2024-03-11T20:58:09.079 INFO:teuthology.orchestra.run.smithi023.stdout: "client": "client.24629", 2024-03-11T20:58:09.080 INFO:teuthology.orchestra.run.smithi023.stdout: "client_addr": "172.21.15.23:0/959732609", 2024-03-11T20:58:09.080 INFO:teuthology.orchestra.run.smithi023.stdout: "tid": 167632 2024-03-11T20:58:09.080 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.080 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T20:58:09.080 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.080 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T20:58:09.080 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:23.610529+0000", 2024-03-11T20:58:09.080 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:58:09.080 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.080 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.080 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T20:58:09.080 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:23.610529+0000", 2024-03-11T20:58:09.080 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:58:09.080 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.080 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.080 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T20:58:09.081 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:23.610530+0000", 2024-03-11T20:58:09.081 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.13e-06 2024-03-11T20:58:09.081 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.081 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.081 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T20:58:09.081 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:23.610534+0000", 2024-03-11T20:58:09.081 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 4.4190000000000004e-06 2024-03-11T20:58:09.081 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.081 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.081 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T20:58:09.081 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:23.610543+0000", 2024-03-11T20:58:09.081 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 8.7199999999999995e-06 2024-03-11T20:58:09.081 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.081 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.081 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T20:58:09.081 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:23.610551+0000", 2024-03-11T20:58:09.081 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.4889999999999998e-06 2024-03-11T20:58:09.082 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.082 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.082 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T20:58:09.082 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:23.614155+0000", 2024-03-11T20:58:09.082 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0036047639999999999 2024-03-11T20:58:09.082 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.082 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.082 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for peered", 2024-03-11T20:58:09.082 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:23.614164+0000", 2024-03-11T20:58:09.082 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 8.8289999999999997e-06 2024-03-11T20:58:09.082 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.082 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.082 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T20:58:09.082 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:24.614636+0000", 2024-03-11T20:58:09.082 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.0004722509999999 2024-03-11T20:58:09.082 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.083 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.083 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for peered", 2024-03-11T20:58:09.083 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:24.614650+0000", 2024-03-11T20:58:09.083 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.4071e-05 2024-03-11T20:58:09.083 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.083 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.083 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T20:58:09.083 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:25.790645+0000", 2024-03-11T20:58:09.083 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.1759943470000001 2024-03-11T20:58:09.083 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.083 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.083 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T20:58:09.083 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:25.790814+0000", 2024-03-11T20:58:09.084 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00016891200000000001 2024-03-11T20:58:09.084 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.084 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.084 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for subops from 1,2,7", 2024-03-11T20:58:09.084 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:25.790969+0000", 2024-03-11T20:58:09.084 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00015503399999999999 2024-03-11T20:58:09.084 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.084 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.084 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T20:58:09.084 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:25.798529+0000", 2024-03-11T20:58:09.084 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0075605819999999997 2024-03-11T20:58:09.084 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.084 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.084 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T20:58:09.084 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:25.798597+0000", 2024-03-11T20:58:09.085 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.7442999999999998e-05 2024-03-11T20:58:09.085 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.085 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.085 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T20:58:09.085 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:25.798770+0000", 2024-03-11T20:58:09.085 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00017372700000000001 2024-03-11T20:58:09.085 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.085 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.085 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "op_commit", 2024-03-11T20:58:09.085 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:25.799032+0000", 2024-03-11T20:58:09.085 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00026188399999999999 2024-03-11T20:58:09.085 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.085 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.086 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T20:58:09.086 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:25.799087+0000", 2024-03-11T20:58:09.086 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 5.4491999999999999e-05 2024-03-11T20:58:09.086 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.086 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.086 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T20:58:09.086 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:25.799205+0000", 2024-03-11T20:58:09.086 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00011857399999999999 2024-03-11T20:58:09.086 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:58:09.086 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T20:58:09.086 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:58:09.086 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.086 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.086 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_op(client.24629.0:167635 7.2 7:49edfd98:::benchmark_data_smithi023_67959_object57394:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e162)", 2024-03-11T20:58:09.086 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T20:57:23.617769+0000", 2024-03-11T20:58:09.086 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 45.359741257000003, 2024-03-11T20:58:09.087 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.1816013320000001, 2024-03-11T20:58:09.087 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T20:58:09.087 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T20:58:09.087 INFO:teuthology.orchestra.run.smithi023.stdout: "client_info": { 2024-03-11T20:58:09.087 INFO:teuthology.orchestra.run.smithi023.stdout: "client": "client.24629", 2024-03-11T20:58:09.087 INFO:teuthology.orchestra.run.smithi023.stdout: "client_addr": "172.21.15.23:0/959732609", 2024-03-11T20:58:09.087 INFO:teuthology.orchestra.run.smithi023.stdout: "tid": 167635 2024-03-11T20:58:09.087 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.087 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T20:58:09.087 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.087 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T20:58:09.087 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:23.617769+0000", 2024-03-11T20:58:09.087 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:58:09.087 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.087 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.087 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T20:58:09.087 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:23.617769+0000", 2024-03-11T20:58:09.088 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:58:09.088 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.088 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.088 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T20:58:09.088 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:23.617771+0000", 2024-03-11T20:58:09.088 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.5099999999999999e-06 2024-03-11T20:58:09.088 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.088 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.088 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T20:58:09.088 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:23.617777+0000", 2024-03-11T20:58:09.088 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 5.857e-06 2024-03-11T20:58:09.088 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.088 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.089 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T20:58:09.089 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:23.617789+0000", 2024-03-11T20:58:09.089 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.1902000000000001e-05 2024-03-11T20:58:09.089 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.089 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.089 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T20:58:09.089 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:23.617798+0000", 2024-03-11T20:58:09.089 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 9.7350000000000008e-06 2024-03-11T20:58:09.089 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.089 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.089 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T20:58:09.089 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:23.619568+0000", 2024-03-11T20:58:09.089 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.001769685 2024-03-11T20:58:09.089 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.089 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.089 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for peered", 2024-03-11T20:58:09.090 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:23.619581+0000", 2024-03-11T20:58:09.090 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.2926e-05 2024-03-11T20:58:09.090 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.090 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.090 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T20:58:09.090 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:24.615894+0000", 2024-03-11T20:58:09.090 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.99631322099999997 2024-03-11T20:58:09.090 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.090 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.090 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for peered", 2024-03-11T20:58:09.090 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:24.615904+0000", 2024-03-11T20:58:09.090 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 9.5820000000000008e-06 2024-03-11T20:58:09.090 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.090 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.090 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T20:58:09.090 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:25.793735+0000", 2024-03-11T20:58:09.091 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.1778313039999999 2024-03-11T20:58:09.091 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.091 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.091 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T20:58:09.091 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:25.793823+0000", 2024-03-11T20:58:09.091 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 8.8228999999999998e-05 2024-03-11T20:58:09.091 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.091 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.091 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for subops from 1,2,7", 2024-03-11T20:58:09.091 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:25.793917+0000", 2024-03-11T20:58:09.091 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 9.3671999999999994e-05 2024-03-11T20:58:09.091 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.091 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.091 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T20:58:09.091 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:25.798797+0000", 2024-03-11T20:58:09.091 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.004880138 2024-03-11T20:58:09.091 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.092 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.092 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T20:58:09.092 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:25.798819+0000", 2024-03-11T20:58:09.092 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.2348e-05 2024-03-11T20:58:09.092 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.092 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.092 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T20:58:09.092 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:25.798843+0000", 2024-03-11T20:58:09.092 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.3646e-05 2024-03-11T20:58:09.092 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.092 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.092 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "op_commit", 2024-03-11T20:58:09.092 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:25.799214+0000", 2024-03-11T20:58:09.092 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000370944 2024-03-11T20:58:09.092 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.093 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.093 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T20:58:09.093 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:25.799262+0000", 2024-03-11T20:58:09.093 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 4.8000999999999997e-05 2024-03-11T20:58:09.093 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.093 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.093 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T20:58:09.093 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:25.799371+0000", 2024-03-11T20:58:09.093 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000108632 2024-03-11T20:58:09.093 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:58:09.093 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T20:58:09.093 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:58:09.093 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.093 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.093 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_op(client.24629.0:167638 7.2 7:451714c6:::benchmark_data_smithi023_67959_object57397:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e162)", 2024-03-11T20:58:09.094 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T20:57:23.620223+0000", 2024-03-11T20:58:09.094 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 45.357287315999997, 2024-03-11T20:58:09.094 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.1799264410000001, 2024-03-11T20:58:09.094 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T20:58:09.094 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T20:58:09.094 INFO:teuthology.orchestra.run.smithi023.stdout: "client_info": { 2024-03-11T20:58:09.094 INFO:teuthology.orchestra.run.smithi023.stdout: "client": "client.24629", 2024-03-11T20:58:09.094 INFO:teuthology.orchestra.run.smithi023.stdout: "client_addr": "172.21.15.23:0/959732609", 2024-03-11T20:58:09.094 INFO:teuthology.orchestra.run.smithi023.stdout: "tid": 167638 2024-03-11T20:58:09.094 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.094 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T20:58:09.094 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.094 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T20:58:09.095 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:23.620223+0000", 2024-03-11T20:58:09.095 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:58:09.095 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.095 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.095 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T20:58:09.095 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:23.620223+0000", 2024-03-11T20:58:09.095 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T20:58:09.095 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.095 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.095 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T20:58:09.095 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:23.620224+0000", 2024-03-11T20:58:09.095 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 5.3000000000000001e-07 2024-03-11T20:58:09.095 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.095 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.095 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T20:58:09.096 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:23.620228+0000", 2024-03-11T20:58:09.096 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 3.7730000000000001e-06 2024-03-11T20:58:09.096 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.096 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.096 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T20:58:09.096 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:23.620236+0000", 2024-03-11T20:58:09.096 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 8.3559999999999993e-06 2024-03-11T20:58:09.096 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.096 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.096 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T20:58:09.096 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:23.620242+0000", 2024-03-11T20:58:09.096 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.4200000000000004e-06 2024-03-11T20:58:09.096 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.096 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.097 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T20:58:09.097 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:23.630585+0000", 2024-03-11T20:58:09.097 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.010342934 2024-03-11T20:58:09.097 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.097 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.097 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for peered", 2024-03-11T20:58:09.097 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:23.630597+0000", 2024-03-11T20:58:09.097 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.1525000000000001e-05 2024-03-11T20:58:09.097 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.097 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.097 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T20:58:09.097 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:24.615935+0000", 2024-03-11T20:58:09.097 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.98533783900000005 2024-03-11T20:58:09.097 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.098 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.098 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for peered", 2024-03-11T20:58:09.098 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:24.615944+0000", 2024-03-11T20:58:09.098 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 9.0429999999999996e-06 2024-03-11T20:58:09.098 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.098 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.098 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T20:58:09.098 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:25.795926+0000", 2024-03-11T20:58:09.098 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.179982026 2024-03-11T20:58:09.098 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.098 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.098 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T20:58:09.099 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:25.796028+0000", 2024-03-11T20:58:09.099 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000102172 2024-03-11T20:58:09.099 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.099 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.099 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for subops from 1,2,7", 2024-03-11T20:58:09.099 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:25.796195+0000", 2024-03-11T20:58:09.099 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00016758100000000001 2024-03-11T20:58:09.099 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.099 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.099 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T20:58:09.099 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:25.799016+0000", 2024-03-11T20:58:09.099 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0028206220000000001 2024-03-11T20:58:09.099 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.099 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.100 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T20:58:09.100 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:25.799411+0000", 2024-03-11T20:58:09.100 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00039464399999999998 2024-03-11T20:58:09.100 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.100 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.100 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T20:58:09.100 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:25.799468+0000", 2024-03-11T20:58:09.100 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 5.6802999999999998e-05 2024-03-11T20:58:09.100 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.100 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.100 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "op_commit", 2024-03-11T20:58:09.100 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:25.799966+0000", 2024-03-11T20:58:09.100 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00049837600000000003 2024-03-11T20:58:09.100 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.100 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.101 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T20:58:09.101 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:25.800025+0000", 2024-03-11T20:58:09.101 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 5.9462000000000001e-05 2024-03-11T20:58:09.101 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T20:58:09.101 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T20:58:09.101 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T20:58:09.101 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:25.800150+0000", 2024-03-11T20:58:09.101 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00012433499999999999 2024-03-11T20:58:09.101 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:58:09.101 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T20:58:09.101 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:58:09.101 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T20:58:09.101 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T20:58:09.101 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-11T20:58:09.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:09.103 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:09.103 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:58:09.204 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:09.228 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:09.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:09.231 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:58:09.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:09 smithi023 bash[21372]: cluster 2024-03-11T20:58:07.975388+0000 mgr.y (mgr.24380) 275 : cluster 0 pgmap v276: 148 pgs: 89 active+clean, 34 active+undersized, 22 active+undersized+degraded, 3 active+clean+remapped; 1.6 GiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 895 B/s rd, 0 B/s wr, 691 op/s; 4926/78780 objects degraded (6.253%); 4947/78780 objects misplaced (6.280%); 3.0 MiB/s, 46 objects/s recovering 2024-03-11T20:58:09.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:09 smithi023 bash[21372]: audit 2024-03-11T20:58:08.363109+0000 mon.a (mon.0) 1093 : audit 0 from='client.? 172.21.15.23:0/3180579313' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T20:58:09.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:09 smithi023 bash[29146]: cluster 2024-03-11T20:58:07.975388+0000 mgr.y (mgr.24380) 275 : cluster 0 pgmap v276: 148 pgs: 89 active+clean, 34 active+undersized, 22 active+undersized+degraded, 3 active+clean+remapped; 1.6 GiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 895 B/s rd, 0 B/s wr, 691 op/s; 4926/78780 objects degraded (6.253%); 4947/78780 objects misplaced (6.280%); 3.0 MiB/s, 46 objects/s recovering 2024-03-11T20:58:09.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:09 smithi023 bash[29146]: audit 2024-03-11T20:58:08.363109+0000 mon.a (mon.0) 1093 : audit 0 from='client.? 172.21.15.23:0/3180579313' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T20:58:09.332 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:09.363 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:09.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:09.370 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:58:09.471 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:58:09.494 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:58:09.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:09.498 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:58:09.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:09 smithi146 bash[23809]: cluster 2024-03-11T20:58:07.975388+0000 mgr.y (mgr.24380) 275 : cluster 0 pgmap v276: 148 pgs: 89 active+clean, 34 active+undersized, 22 active+undersized+degraded, 3 active+clean+remapped; 1.6 GiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 895 B/s rd, 0 B/s wr, 691 op/s; 4926/78780 objects degraded (6.253%); 4947/78780 objects misplaced (6.280%); 3.0 MiB/s, 46 objects/s recovering 2024-03-11T20:58:09.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:09 smithi146 bash[23809]: audit 2024-03-11T20:58:08.363109+0000 mon.a (mon.0) 1093 : audit 0 from='client.? 172.21.15.23:0/3180579313' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T20:58:09.599 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:09.623 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:09.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:09.626 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:58:09.726 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:09.746 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:09.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:09.751 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:58:09.851 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:58:09.882 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:58:09.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:09.886 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:58:09.987 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:10.017 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:10.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:10.019 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:58:10.119 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:10.146 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:10.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:10.153 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:58:10.176 INFO:teuthology.orchestra.run.smithi023.stderr:nodeep-scrub is set 2024-03-11T20:58:10.254 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:10.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:10 smithi146 bash[23809]: audit 2024-03-11T20:58:09.414247+0000 mon.a (mon.0) 1094 : audit 1 from='client.? 172.21.15.23:0/1736725348' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T20:58:10.275 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:10.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:10.277 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:58:10.378 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:58:10.401 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:58:10.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:10.403 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:58:10.504 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:10.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:10 smithi023 bash[21372]: audit 2024-03-11T20:58:09.414247+0000 mon.a (mon.0) 1094 : audit 1 from='client.? 172.21.15.23:0/1736725348' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T20:58:10.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:10 smithi023 bash[29146]: audit 2024-03-11T20:58:09.414247+0000 mon.a (mon.0) 1094 : audit 1 from='client.? 172.21.15.23:0/1736725348' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T20:58:10.537 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:10.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:10.539 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:58:10.640 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:10.663 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:10.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:10.666 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:58:10.767 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:10.790 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:10.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:10.792 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:58:10.893 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:10.929 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:10.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:10.933 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:58:11.034 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:11.054 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:11.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:11.056 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:58:11.156 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:11.187 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:11.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:11.189 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:58:11.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:11 smithi023 bash[21372]: cluster 2024-03-11T20:58:09.976284+0000 mgr.y (mgr.24380) 276 : cluster 0 pgmap v277: 148 pgs: 89 active+clean, 34 active+undersized, 22 active+undersized+degraded, 3 active+clean+remapped; 1.5 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 552 B/s rd, 0 B/s wr, 581 op/s; 4631/73425 objects degraded (6.307%); 4342/73425 objects misplaced (5.914%); 0 B/s, 0 objects/s recovering 2024-03-11T20:58:11.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:11 smithi023 bash[21372]: cluster 2024-03-11T20:58:10.164475+0000 mon.a (mon.0) 1095 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T20:58:11.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:11 smithi023 bash[21372]: audit 2024-03-11T20:58:10.176380+0000 mon.a (mon.0) 1096 : audit 1 from='client.? 172.21.15.23:0/1736725348' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T20:58:11.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:11 smithi023 bash[21372]: cluster 2024-03-11T20:58:10.181942+0000 mon.a (mon.0) 1097 : cluster 0 osdmap e178: 8 total, 7 up, 8 in 2024-03-11T20:58:11.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:11 smithi023 bash[29146]: cluster 2024-03-11T20:58:09.976284+0000 mgr.y (mgr.24380) 276 : cluster 0 pgmap v277: 148 pgs: 89 active+clean, 34 active+undersized, 22 active+undersized+degraded, 3 active+clean+remapped; 1.5 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 552 B/s rd, 0 B/s wr, 581 op/s; 4631/73425 objects degraded (6.307%); 4342/73425 objects misplaced (5.914%); 0 B/s, 0 objects/s recovering 2024-03-11T20:58:11.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:11 smithi023 bash[29146]: cluster 2024-03-11T20:58:10.164475+0000 mon.a (mon.0) 1095 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T20:58:11.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:11 smithi023 bash[29146]: audit 2024-03-11T20:58:10.176380+0000 mon.a (mon.0) 1096 : audit 1 from='client.? 172.21.15.23:0/1736725348' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T20:58:11.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:11 smithi023 bash[29146]: cluster 2024-03-11T20:58:10.181942+0000 mon.a (mon.0) 1097 : cluster 0 osdmap e178: 8 total, 7 up, 8 in 2024-03-11T20:58:11.290 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:11.317 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:11.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:11.319 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:58:11.420 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:58:11.455 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-11T20:58:11.456 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-11T20:58:11.470 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:58:11.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:11.472 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:58:11.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:11 smithi146 bash[23809]: cluster 2024-03-11T20:58:09.976284+0000 mgr.y (mgr.24380) 276 : cluster 0 pgmap v277: 148 pgs: 89 active+clean, 34 active+undersized, 22 active+undersized+degraded, 3 active+clean+remapped; 1.5 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 552 B/s rd, 0 B/s wr, 581 op/s; 4631/73425 objects degraded (6.307%); 4342/73425 objects misplaced (5.914%); 0 B/s, 0 objects/s recovering 2024-03-11T20:58:11.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:11 smithi146 bash[23809]: cluster 2024-03-11T20:58:10.164475+0000 mon.a (mon.0) 1095 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T20:58:11.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:11 smithi146 bash[23809]: audit 2024-03-11T20:58:10.176380+0000 mon.a (mon.0) 1096 : audit 1 from='client.? 172.21.15.23:0/1736725348' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T20:58:11.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:11 smithi146 bash[23809]: cluster 2024-03-11T20:58:10.181942+0000 mon.a (mon.0) 1097 : cluster 0 osdmap e178: 8 total, 7 up, 8 in 2024-03-11T20:58:11.573 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:58:11.598 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:58:11.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:11.600 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:58:11.701 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:11.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:11.737 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:11.737 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:58:11.838 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:11.872 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:11.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:11.877 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:58:11.978 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:12.002 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:12.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:12.003 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:58:12.104 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:58:12.129 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:58:12.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:12.133 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:58:12.234 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:58:12.259 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:58:12.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:12 smithi146 bash[23809]: cluster 2024-03-11T20:58:12.167118+0000 mon.a (mon.0) 1098 : cluster 3 Health check update: Degraded data redundancy: 4052/64935 objects degraded (6.240%), 22 pgs degraded (PG_DEGRADED) 2024-03-11T20:58:12.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:12.261 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:58:12.362 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:12.369 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:12 smithi023 bash[21372]: cluster 2024-03-11T20:58:12.167118+0000 mon.a (mon.0) 1098 : cluster 3 Health check update: Degraded data redundancy: 4052/64935 objects degraded (6.240%), 22 pgs degraded (PG_DEGRADED) 2024-03-11T20:58:12.370 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:12 smithi023 bash[29146]: cluster 2024-03-11T20:58:12.167118+0000 mon.a (mon.0) 1098 : cluster 3 Health check update: Degraded data redundancy: 4052/64935 objects degraded (6.240%), 22 pgs degraded (PG_DEGRADED) 2024-03-11T20:58:12.382 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:12.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:12.383 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:58:12.484 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:12.514 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:12.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:12.520 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:58:12.620 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:12.643 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:12.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:12.647 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:58:12.726 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-11T20:58:12.748 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:12.786 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:12.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:12.790 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:58:12.846 DEBUG:teuthology.orchestra.run.smithi023:> sudo pkill -f 'journalctl -f -n 0 -u ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@osd.1.service' 2024-03-11T20:58:12.890 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:58:12.894 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T20:58:12.895 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-11T20:58:12.895 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.1 2024-03-11T20:58:12.895 INFO:tasks.thrashosds.thrasher:remote for osd 1 is ubuntu@smithi023.front.sepia.ceph.com 2024-03-11T20:58:12.895 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -n osd.1 --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-1 --journal-path /var/lib/ceph/osd/ceph-1/journal --op list-pgs 2024-03-11T20:58:12.927 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:58:12.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:12.930 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:58:13.032 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:13.053 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:13.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:13.061 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:58:13.149 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/osd.1/config 2024-03-11T20:58:13.162 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:13.188 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:13.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:13.193 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:58:13.295 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:13.305 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:13 smithi146 bash[23809]: cluster 2024-03-11T20:58:11.977091+0000 mgr.y (mgr.24380) 277 : cluster 0 pgmap v279: 148 pgs: 89 active+clean, 34 active+undersized, 22 active+undersized+degraded, 3 active+clean+remapped; 1.3 GiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 926 op/s; 4052/64935 objects degraded (6.240%); 4051/64935 objects misplaced (6.239%); 0 B/s, 0 objects/s recovering 2024-03-11T20:58:13.323 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:13.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:13.327 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:58:13.427 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:13.437 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:13 smithi023 bash[21372]: cluster 2024-03-11T20:58:11.977091+0000 mgr.y (mgr.24380) 277 : cluster 0 pgmap v279: 148 pgs: 89 active+clean, 34 active+undersized, 22 active+undersized+degraded, 3 active+clean+remapped; 1.3 GiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 926 op/s; 4052/64935 objects degraded (6.240%); 4051/64935 objects misplaced (6.239%); 0 B/s, 0 objects/s recovering 2024-03-11T20:58:13.438 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:13 smithi023 bash[29146]: cluster 2024-03-11T20:58:11.977091+0000 mgr.y (mgr.24380) 277 : cluster 0 pgmap v279: 148 pgs: 89 active+clean, 34 active+undersized, 22 active+undersized+degraded, 3 active+clean+remapped; 1.3 GiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 926 op/s; 4052/64935 objects degraded (6.240%); 4051/64935 objects misplaced (6.239%); 0 B/s, 0 objects/s recovering 2024-03-11T20:58:13.456 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T20:58:13.464 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:13.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:13.465 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:58:13.565 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:13.591 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:13.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:13.595 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:58:13.695 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:13.717 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:13.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:13.723 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:58:13.823 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:58:13.848 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:58:13.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:13.851 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:58:13.951 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:13.983 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:13.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:13.987 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:58:14.087 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:14.115 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:14.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:14.120 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:58:14.220 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:58:14.241 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:58:14.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:14.247 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:58:14.347 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:14.368 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:14.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:14.370 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:58:14.471 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:14.494 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:14.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:14.497 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:58:14.597 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:14.622 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:14.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:14.626 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:58:14.727 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:58:14.752 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:58:14.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:14.759 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:58:14.859 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:14.886 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:14.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:14.890 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:58:14.951 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd unset noscrub 2024-03-11T20:58:14.991 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:15.018 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:15.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:15.024 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:58:15.125 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:58:15.148 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:58:15.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:15.150 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:58:15.251 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:58:15.274 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:58:15.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:15.279 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:58:15.380 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:58:15.409 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:58:15.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:15.413 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:58:15.511 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 20:58:15 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T20:58:15.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:15 smithi146 bash[23809]: cluster 2024-03-11T20:58:13.977683+0000 mgr.y (mgr.24380) 278 : cluster 0 pgmap v280: 148 pgs: 89 active+clean, 34 active+undersized, 22 active+undersized+degraded, 3 active+clean+remapped; 1.3 GiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 822 op/s; 4052/64935 objects degraded (6.240%); 4051/64935 objects misplaced (6.239%); 0 B/s, 0 objects/s recovering 2024-03-11T20:58:15.514 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:15.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:15 smithi023 bash[21372]: cluster 2024-03-11T20:58:13.977683+0000 mgr.y (mgr.24380) 278 : cluster 0 pgmap v280: 148 pgs: 89 active+clean, 34 active+undersized, 22 active+undersized+degraded, 3 active+clean+remapped; 1.3 GiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 822 op/s; 4052/64935 objects degraded (6.240%); 4051/64935 objects misplaced (6.239%); 0 B/s, 0 objects/s recovering 2024-03-11T20:58:15.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:15 smithi023 bash[29146]: cluster 2024-03-11T20:58:13.977683+0000 mgr.y (mgr.24380) 278 : cluster 0 pgmap v280: 148 pgs: 89 active+clean, 34 active+undersized, 22 active+undersized+degraded, 3 active+clean+remapped; 1.3 GiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 822 op/s; 4052/64935 objects degraded (6.240%); 4051/64935 objects misplaced (6.239%); 0 B/s, 0 objects/s recovering 2024-03-11T20:58:15.539 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:15.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:15.543 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:58:15.644 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:58:15.675 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:58:15.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:15.681 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:58:15.738 INFO:teuthology.orchestra.run.smithi023.stderr:Creating an OSD daemon form without an OSD FSID value 2024-03-11T20:58:15.782 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:15.803 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:15.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:15.804 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:58:15.905 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:15.934 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:15.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:15.942 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:58:16.043 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:16.069 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:16.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:16.071 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:58:16.172 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:16.205 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:16.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:16.210 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:58:16.311 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:16.339 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:16.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:16.345 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:58:16.446 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:16.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:16.480 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:16.481 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:58:16.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:16 smithi146 bash[23809]: audit 2024-03-11T20:58:15.167621+0000 mgr.y (mgr.24380) 279 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:58:16.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:16 smithi023 bash[21372]: audit 2024-03-11T20:58:15.167621+0000 mgr.y (mgr.24380) 279 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:58:16.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:16 smithi023 bash[29146]: audit 2024-03-11T20:58:15.167621+0000 mgr.y (mgr.24380) 279 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:58:16.581 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:16.605 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:16.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:16.607 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:58:16.707 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:58:16.737 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:58:16.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:16.746 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:58:16.846 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:16.877 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:16.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:16.879 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:58:16.980 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:17.006 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:17.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:17.009 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:58:17.110 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:17.138 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:17.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:17.140 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:58:17.241 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:17.264 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:17.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:17.270 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:58:17.371 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:17.383 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:17 smithi023 bash[21372]: cluster 2024-03-11T20:58:15.978755+0000 mgr.y (mgr.24380) 280 : cluster 0 pgmap v281: 148 pgs: 89 active+clean, 34 active+undersized, 22 active+undersized+degraded, 3 active+clean+remapped; 1.0 GiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 939 op/s; 3136/50628 objects degraded (6.194%); 3202/50628 objects misplaced (6.325%) 2024-03-11T20:58:17.384 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:17 smithi023 bash[29146]: cluster 2024-03-11T20:58:15.978755+0000 mgr.y (mgr.24380) 280 : cluster 0 pgmap v281: 148 pgs: 89 active+clean, 34 active+undersized, 22 active+undersized+degraded, 3 active+clean+remapped; 1.0 GiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 939 op/s; 3136/50628 objects degraded (6.194%); 3202/50628 objects misplaced (6.325%) 2024-03-11T20:58:17.400 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:17.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:17.402 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:58:17.503 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:58:17.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:17 smithi146 bash[23809]: cluster 2024-03-11T20:58:15.978755+0000 mgr.y (mgr.24380) 280 : cluster 0 pgmap v281: 148 pgs: 89 active+clean, 34 active+undersized, 22 active+undersized+degraded, 3 active+clean+remapped; 1.0 GiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 939 op/s; 3136/50628 objects degraded (6.194%); 3202/50628 objects misplaced (6.325%) 2024-03-11T20:58:17.521 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T20:58:17.528 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:58:17.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:17.532 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:58:17.632 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:17.655 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:17.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:17.657 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:58:17.757 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:58:17.780 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:58:17.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:17.782 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:58:17.883 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:17.908 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:17.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:17.913 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:58:18.014 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:18.037 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:18.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:18.040 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:58:18.141 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:18.190 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:18.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:18.192 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:58:18.293 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:18.315 INFO:teuthology.orchestra.run.smithi023.stdout:7.b 2024-03-11T20:58:18.315 INFO:teuthology.orchestra.run.smithi023.stdout:6.e 2024-03-11T20:58:18.315 INFO:teuthology.orchestra.run.smithi023.stdout:6.b 2024-03-11T20:58:18.315 INFO:teuthology.orchestra.run.smithi023.stdout:6.4 2024-03-11T20:58:18.315 INFO:teuthology.orchestra.run.smithi023.stdout:6.1 2024-03-11T20:58:18.315 INFO:teuthology.orchestra.run.smithi023.stdout:6.18 2024-03-11T20:58:18.315 INFO:teuthology.orchestra.run.smithi023.stdout:6.10 2024-03-11T20:58:18.315 INFO:teuthology.orchestra.run.smithi023.stdout:4.f 2024-03-11T20:58:18.315 INFO:teuthology.orchestra.run.smithi023.stdout:4.e 2024-03-11T20:58:18.315 INFO:teuthology.orchestra.run.smithi023.stdout:4.d 2024-03-11T20:58:18.316 INFO:teuthology.orchestra.run.smithi023.stdout:4.9 2024-03-11T20:58:18.316 INFO:teuthology.orchestra.run.smithi023.stdout:3.1a 2024-03-11T20:58:18.316 INFO:teuthology.orchestra.run.smithi023.stdout:3.19 2024-03-11T20:58:18.316 INFO:teuthology.orchestra.run.smithi023.stdout:3.1c 2024-03-11T20:58:18.316 INFO:teuthology.orchestra.run.smithi023.stdout:3.4 2024-03-11T20:58:18.316 INFO:teuthology.orchestra.run.smithi023.stdout:3.18 2024-03-11T20:58:18.316 INFO:teuthology.orchestra.run.smithi023.stdout:2.9 2024-03-11T20:58:18.316 INFO:teuthology.orchestra.run.smithi023.stdout:2.d 2024-03-11T20:58:18.316 INFO:teuthology.orchestra.run.smithi023.stdout:7.9 2024-03-11T20:58:18.316 INFO:teuthology.orchestra.run.smithi023.stdout:3.8 2024-03-11T20:58:18.316 INFO:teuthology.orchestra.run.smithi023.stdout:2.8 2024-03-11T20:58:18.316 INFO:teuthology.orchestra.run.smithi023.stdout:2.0 2024-03-11T20:58:18.316 INFO:teuthology.orchestra.run.smithi023.stdout:3.a 2024-03-11T20:58:18.316 INFO:teuthology.orchestra.run.smithi023.stdout:4.1c 2024-03-11T20:58:18.316 INFO:teuthology.orchestra.run.smithi023.stdout:7.2 2024-03-11T20:58:18.317 INFO:teuthology.orchestra.run.smithi023.stdout:2.6 2024-03-11T20:58:18.317 INFO:teuthology.orchestra.run.smithi023.stdout:2.10 2024-03-11T20:58:18.317 INFO:teuthology.orchestra.run.smithi023.stdout:6.19 2024-03-11T20:58:18.317 INFO:teuthology.orchestra.run.smithi023.stdout:4.18 2024-03-11T20:58:18.317 INFO:teuthology.orchestra.run.smithi023.stdout:7.0 2024-03-11T20:58:18.317 INFO:teuthology.orchestra.run.smithi023.stdout:4.7 2024-03-11T20:58:18.317 INFO:teuthology.orchestra.run.smithi023.stdout:3.0 2024-03-11T20:58:18.317 INFO:teuthology.orchestra.run.smithi023.stdout:4.b 2024-03-11T20:58:18.317 INFO:teuthology.orchestra.run.smithi023.stdout:2.11 2024-03-11T20:58:18.317 INFO:teuthology.orchestra.run.smithi023.stdout:7.4 2024-03-11T20:58:18.317 INFO:teuthology.orchestra.run.smithi023.stdout:4.10 2024-03-11T20:58:18.317 INFO:teuthology.orchestra.run.smithi023.stdout:6.d 2024-03-11T20:58:18.317 INFO:teuthology.orchestra.run.smithi023.stdout:2.15 2024-03-11T20:58:18.317 INFO:teuthology.orchestra.run.smithi023.stdout:4.a 2024-03-11T20:58:18.317 INFO:teuthology.orchestra.run.smithi023.stdout:2.a 2024-03-11T20:58:18.318 INFO:teuthology.orchestra.run.smithi023.stdout:3.16 2024-03-11T20:58:18.318 INFO:teuthology.orchestra.run.smithi023.stdout:2.4 2024-03-11T20:58:18.318 INFO:teuthology.orchestra.run.smithi023.stdout:3.e 2024-03-11T20:58:18.318 INFO:teuthology.orchestra.run.smithi023.stdout:6.1a 2024-03-11T20:58:18.318 INFO:teuthology.orchestra.run.smithi023.stdout:3.6 2024-03-11T20:58:18.318 INFO:teuthology.orchestra.run.smithi023.stdout:4.12 2024-03-11T20:58:18.318 INFO:teuthology.orchestra.run.smithi023.stdout:4.2 2024-03-11T20:58:18.318 INFO:teuthology.orchestra.run.smithi023.stdout:4.13 2024-03-11T20:58:18.318 INFO:teuthology.orchestra.run.smithi023.stdout:7.d 2024-03-11T20:58:18.318 INFO:teuthology.orchestra.run.smithi023.stdout:6.1d 2024-03-11T20:58:18.318 INFO:teuthology.orchestra.run.smithi023.stdout:4.17 2024-03-11T20:58:18.318 INFO:teuthology.orchestra.run.smithi023.stdout:4.14 2024-03-11T20:58:18.318 INFO:teuthology.orchestra.run.smithi023.stdout:4.1b 2024-03-11T20:58:18.318 INFO:teuthology.orchestra.run.smithi023.stdout:4.1d 2024-03-11T20:58:18.318 INFO:teuthology.orchestra.run.smithi023.stdout:5.0 2024-03-11T20:58:18.319 INFO:teuthology.orchestra.run.smithi023.stdout:4.1f 2024-03-11T20:58:18.319 INFO:teuthology.orchestra.run.smithi023.stdout:4.4 2024-03-11T20:58:18.319 INFO:teuthology.orchestra.run.smithi023.stdout:2.2 2024-03-11T20:58:18.319 INFO:teuthology.orchestra.run.smithi023.stdout:4.6 2024-03-11T20:58:18.320 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:18.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:18.327 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:58:18.428 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:18.451 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:18.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:18.454 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:58:18.535 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:58:18 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:20:58:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T20:58:18.554 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:58:18.582 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:58:18.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:18.586 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:58:18.686 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:18.713 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:18.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:18.718 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:58:18.818 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:18.840 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:18.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:18.842 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:58:18.944 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:18.968 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:18.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:18.973 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:58:19.075 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:19.103 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:19.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:19.108 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:58:19.209 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:19.236 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:19.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:19.238 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:58:19.295 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:58:19.313 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:58:19.339 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:19.362 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:19.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:19.364 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:58:19.465 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:19.474 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:19 smithi146 bash[23809]: cluster 2024-03-11T20:58:17.979266+0000 mgr.y (mgr.24380) 281 : cluster 0 pgmap v282: 148 pgs: 89 active+clean, 34 active+undersized, 22 active+undersized+degraded, 3 active+clean+remapped; 1.0 GiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 939 op/s; 3136/50628 objects degraded (6.194%); 3202/50628 objects misplaced (6.325%) 2024-03-11T20:58:19.492 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:19.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:19.496 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:58:19.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:19 smithi023 bash[21372]: cluster 2024-03-11T20:58:17.979266+0000 mgr.y (mgr.24380) 281 : cluster 0 pgmap v282: 148 pgs: 89 active+clean, 34 active+undersized, 22 active+undersized+degraded, 3 active+clean+remapped; 1.0 GiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 939 op/s; 3136/50628 objects degraded (6.194%); 3202/50628 objects misplaced (6.325%) 2024-03-11T20:58:19.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:19 smithi023 bash[29146]: cluster 2024-03-11T20:58:17.979266+0000 mgr.y (mgr.24380) 281 : cluster 0 pgmap v282: 148 pgs: 89 active+clean, 34 active+undersized, 22 active+undersized+degraded, 3 active+clean+remapped; 1.0 GiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 939 op/s; 3136/50628 objects degraded (6.194%); 3202/50628 objects misplaced (6.325%) 2024-03-11T20:58:19.597 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:58:19.625 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:58:19.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:19.627 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:58:19.728 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:19.753 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:19.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:19.757 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:58:19.858 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:19.886 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:19.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:19.888 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:58:19.989 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:58:20.015 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:58:20.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:20.023 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:58:20.124 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:20.146 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:20.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:20.149 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:58:20.250 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:58:20.271 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:58:20.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:20.273 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:58:20.374 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:58:20.393 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:58:20.394 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:20 smithi146 bash[23809]: audit 2024-03-11T20:58:19.635036+0000 mon.a (mon.0) 1099 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:58:20.394 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:20 smithi146 bash[23809]: cluster 2024-03-11T20:58:19.705459+0000 mon.a (mon.0) 1100 : cluster 3 Health check update: Degraded data redundancy: 3136/50628 objects degraded (6.194%), 22 pgs degraded (PG_DEGRADED) 2024-03-11T20:58:20.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:20.395 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:58:20.495 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:20.520 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:20.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:20.522 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:58:20.623 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:20.650 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:20.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:20.655 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:58:20.755 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:58:20.777 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:58:20.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:20.779 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:58:20.784 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:20 smithi023 bash[29146]: audit 2024-03-11T20:58:19.635036+0000 mon.a (mon.0) 1099 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:58:20.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:20 smithi023 bash[29146]: cluster 2024-03-11T20:58:19.705459+0000 mon.a (mon.0) 1100 : cluster 3 Health check update: Degraded data redundancy: 3136/50628 objects degraded (6.194%), 22 pgs degraded (PG_DEGRADED) 2024-03-11T20:58:20.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:20 smithi023 bash[21372]: audit 2024-03-11T20:58:19.635036+0000 mon.a (mon.0) 1099 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:58:20.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:20 smithi023 bash[21372]: cluster 2024-03-11T20:58:19.705459+0000 mon.a (mon.0) 1100 : cluster 3 Health check update: Degraded data redundancy: 3136/50628 objects degraded (6.194%), 22 pgs degraded (PG_DEGRADED) 2024-03-11T20:58:20.880 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:20.907 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:20.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:20.909 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:58:21.010 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:21.040 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:21.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:21.045 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:58:21.145 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:21.173 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:21.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:21.177 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:58:21.278 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:21.302 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:21.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:21.304 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:58:21.405 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:58:21.432 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:58:21.434 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:21 smithi146 bash[23809]: cluster 2024-03-11T20:58:19.982102+0000 mgr.y (mgr.24380) 282 : cluster 0 pgmap v283: 148 pgs: 89 active+clean, 34 active+undersized, 22 active+undersized+degraded, 3 active+clean+remapped; 915 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 964 op/s; 2809/44511 objects degraded (6.311%); 2532/44511 objects misplaced (5.688%) 2024-03-11T20:58:21.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:21.435 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:58:21.534 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:21 smithi023 bash[29146]: cluster 2024-03-11T20:58:19.982102+0000 mgr.y (mgr.24380) 282 : cluster 0 pgmap v283: 148 pgs: 89 active+clean, 34 active+undersized, 22 active+undersized+degraded, 3 active+clean+remapped; 915 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 964 op/s; 2809/44511 objects degraded (6.311%); 2532/44511 objects misplaced (5.688%) 2024-03-11T20:58:21.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:21 smithi023 bash[21372]: cluster 2024-03-11T20:58:19.982102+0000 mgr.y (mgr.24380) 282 : cluster 0 pgmap v283: 148 pgs: 89 active+clean, 34 active+undersized, 22 active+undersized+degraded, 3 active+clean+remapped; 915 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 964 op/s; 2809/44511 objects degraded (6.311%); 2532/44511 objects misplaced (5.688%) 2024-03-11T20:58:21.536 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:21.562 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:21.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:21.564 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:58:21.664 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:21.695 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:21.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:21.698 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:58:21.763 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -n osd.1 --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-1 --journal-path /var/lib/ceph/osd/ceph-1/journal --op export --pgid 3.19 --file /var/log/ceph/exp.3.19.1 2024-03-11T20:58:21.799 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:21.854 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:21.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:21.864 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:58:21.965 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:21.972 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/osd.1/config 2024-03-11T20:58:22.007 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:22.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:22.019 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:58:22.120 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:22.140 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:22.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:22.142 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:58:22.242 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:22.271 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:22.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:22.275 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:58:22.376 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:22.408 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:22.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:22.413 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:58:22.513 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:22.545 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:22.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:22.548 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:58:22.648 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:22.678 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:22.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:22.680 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:58:22.781 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:22.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:22.813 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:22.814 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:58:22.914 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:22.957 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:22.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:22.959 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:58:23.059 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:23.102 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:23.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:23.107 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:58:23.208 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:23.225 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:23.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:23.227 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:58:23.328 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:58:23.345 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:23 smithi146 bash[23809]: cluster 2024-03-11T20:58:21.982956+0000 mgr.y (mgr.24380) 283 : cluster 0 pgmap v284: 148 pgs: 89 active+clean, 34 active+undersized, 22 active+undersized+degraded, 3 active+clean+remapped; 710 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.02k op/s; 2186/34638 objects degraded (6.311%); 2170/34638 objects misplaced (6.265%) 2024-03-11T20:58:23.358 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:58:23.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:23.361 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:58:23.462 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:23.480 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:23.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:23.482 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:58:23.583 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:58:23.596 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:23 smithi023 bash[21372]: cluster 2024-03-11T20:58:21.982956+0000 mgr.y (mgr.24380) 283 : cluster 0 pgmap v284: 148 pgs: 89 active+clean, 34 active+undersized, 22 active+undersized+degraded, 3 active+clean+remapped; 710 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.02k op/s; 2186/34638 objects degraded (6.311%); 2170/34638 objects misplaced (6.265%) 2024-03-11T20:58:23.596 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:23 smithi023 bash[29146]: cluster 2024-03-11T20:58:21.982956+0000 mgr.y (mgr.24380) 283 : cluster 0 pgmap v284: 148 pgs: 89 active+clean, 34 active+undersized, 22 active+undersized+degraded, 3 active+clean+remapped; 710 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.02k op/s; 2186/34638 objects degraded (6.311%); 2170/34638 objects misplaced (6.265%) 2024-03-11T20:58:23.612 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:58:23.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:23.616 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:58:23.717 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:23.746 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:23.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:23.751 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:58:23.851 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:23.876 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:23.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:23.879 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:58:23.979 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:24.010 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:24.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:24.012 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:58:24.113 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:24.138 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:24.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:24.142 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:58:24.243 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:58:24.269 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:58:24.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:24.271 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:58:24.372 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:58:24.402 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:58:24.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:24.405 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:58:24.506 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:24.531 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:24.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:24.533 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:58:24.633 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:24.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:24.658 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:24.659 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:58:24.759 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:24.782 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:24.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:24.786 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:58:24.887 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:24.913 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:24.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:24.916 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:58:25.017 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:25.044 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:25.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:25.046 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:58:25.147 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:25.179 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:25.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:25.181 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:58:25.282 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:25.301 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:25.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:25.303 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:58:25.404 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:25.438 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:25.441 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:25 smithi146 bash[23809]: cluster 2024-03-11T20:58:23.983435+0000 mgr.y (mgr.24380) 284 : cluster 0 pgmap v285: 148 pgs: 89 active+clean, 34 active+undersized, 22 active+undersized+degraded, 3 active+clean+remapped; 710 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 841 op/s; 2186/34638 objects degraded (6.311%); 2170/34638 objects misplaced (6.265%) 2024-03-11T20:58:25.441 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:25 smithi146 bash[23809]: cluster 2024-03-11T20:58:24.706243+0000 mon.a (mon.0) 1101 : cluster 3 Health check update: Degraded data redundancy: 2186/34638 objects degraded (6.311%), 22 pgs degraded (PG_DEGRADED) 2024-03-11T20:58:25.441 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 20:58:25 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T20:58:25.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:25.442 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:58:25.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:25 smithi023 bash[21372]: cluster 2024-03-11T20:58:23.983435+0000 mgr.y (mgr.24380) 284 : cluster 0 pgmap v285: 148 pgs: 89 active+clean, 34 active+undersized, 22 active+undersized+degraded, 3 active+clean+remapped; 710 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 841 op/s; 2186/34638 objects degraded (6.311%); 2170/34638 objects misplaced (6.265%) 2024-03-11T20:58:25.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:25 smithi023 bash[21372]: cluster 2024-03-11T20:58:24.706243+0000 mon.a (mon.0) 1101 : cluster 3 Health check update: Degraded data redundancy: 2186/34638 objects degraded (6.311%), 22 pgs degraded (PG_DEGRADED) 2024-03-11T20:58:25.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:25 smithi023 bash[29146]: cluster 2024-03-11T20:58:23.983435+0000 mgr.y (mgr.24380) 284 : cluster 0 pgmap v285: 148 pgs: 89 active+clean, 34 active+undersized, 22 active+undersized+degraded, 3 active+clean+remapped; 710 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 841 op/s; 2186/34638 objects degraded (6.311%); 2170/34638 objects misplaced (6.265%) 2024-03-11T20:58:25.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:25 smithi023 bash[29146]: cluster 2024-03-11T20:58:24.706243+0000 mon.a (mon.0) 1101 : cluster 3 Health check update: Degraded data redundancy: 2186/34638 objects degraded (6.311%), 22 pgs degraded (PG_DEGRADED) 2024-03-11T20:58:25.542 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:25.572 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:25.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:25.575 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:58:25.675 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:25.704 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:25.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:25.707 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:58:25.808 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:58:25.826 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:58:25.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:25.827 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:58:25.928 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:25.952 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:25.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:25.954 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:58:26.055 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:26.081 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:26.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:26.086 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:58:26.141 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-03-11T20:58:26.141 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-03-11T20:58:26.141 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T20:58:26.141 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.24629.0:214649 7.b 7:d5eb4547:::benchmark_data_smithi023_67959_object104408:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-03-11T20:58:26.141 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T20:58:26.133944+0000", 2024-03-11T20:58:26.141 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.014055748, 2024-03-11T20:58:26.141 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014091223, 2024-03-11T20:58:26.141 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T20:58:26.141 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "waiting for sub ops", 2024-03-11T20:58:26.141 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-03-11T20:58:26.141 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.24629", 2024-03-11T20:58:26.142 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.23:0/959732609", 2024-03-11T20:58:26.142 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 214649 2024-03-11T20:58:26.142 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T20:58:26.142 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T20:58:26.142 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T20:58:26.142 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T20:58:26.142 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.133944+0000", 2024-03-11T20:58:26.142 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T20:58:26.142 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T20:58:26.142 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T20:58:26.142 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T20:58:26.142 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.133944+0000", 2024-03-11T20:58:26.143 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T20:58:26.143 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T20:58:26.143 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T20:58:26.143 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T20:58:26.143 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.133945+0000", 2024-03-11T20:58:26.143 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5650000000000001e-06 2024-03-11T20:58:26.143 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T20:58:26.143 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T20:58:26.143 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T20:58:26.143 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.133952+0000", 2024-03-11T20:58:26.143 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6220000000000003e-06 2024-03-11T20:58:26.143 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T20:58:26.143 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T20:58:26.143 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T20:58:26.144 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.133969+0000", 2024-03-11T20:58:26.144 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7261999999999998e-05 2024-03-11T20:58:26.144 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T20:58:26.144 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T20:58:26.144 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T20:58:26.144 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.133978+0000", 2024-03-11T20:58:26.144 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.1859999999999998e-06 2024-03-11T20:58:26.144 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T20:58:26.144 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T20:58:26.144 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T20:58:26.144 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.134041+0000", 2024-03-11T20:58:26.144 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3120999999999994e-05 2024-03-11T20:58:26.144 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T20:58:26.145 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T20:58:26.145 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T20:58:26.145 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.134171+0000", 2024-03-11T20:58:26.145 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012984300000000001 2024-03-11T20:58:26.145 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T20:58:26.145 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T20:58:26.145 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 6", 2024-03-11T20:58:26.145 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.134298+0000", 2024-03-11T20:58:26.145 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000126802 2024-03-11T20:58:26.145 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T20:58:26.145 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T20:58:26.145 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-03-11T20:58:26.145 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.140337+0000", 2024-03-11T20:58:26.145 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0060387440000000004 2024-03-11T20:58:26.145 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T20:58:26.146 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T20:58:26.146 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T20:58:26.146 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T20:58:26.146 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T20:58:26.146 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.24629.0:214659 7.b 7:d8a44537:::benchmark_data_smithi023_67959_object104418:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-03-11T20:58:26.146 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T20:58:26.145407+0000", 2024-03-11T20:58:26.146 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.0025924300000000002, 2024-03-11T20:58:26.146 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0027158590000000002, 2024-03-11T20:58:26.146 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T20:58:26.147 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "waiting for sub ops", 2024-03-11T20:58:26.147 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-03-11T20:58:26.147 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.24629", 2024-03-11T20:58:26.147 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.23:0/959732609", 2024-03-11T20:58:26.147 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 214659 2024-03-11T20:58:26.147 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T20:58:26.147 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T20:58:26.147 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T20:58:26.147 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T20:58:26.147 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.145407+0000", 2024-03-11T20:58:26.147 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T20:58:26.147 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T20:58:26.147 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T20:58:26.147 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T20:58:26.147 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.145407+0000", 2024-03-11T20:58:26.148 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T20:58:26.148 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T20:58:26.148 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T20:58:26.148 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T20:58:26.148 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.145408+0000", 2024-03-11T20:58:26.148 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5009999999999999e-06 2024-03-11T20:58:26.148 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T20:58:26.148 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T20:58:26.148 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T20:58:26.148 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.145412+0000", 2024-03-11T20:58:26.148 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8840000000000004e-06 2024-03-11T20:58:26.149 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T20:58:26.149 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T20:58:26.149 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T20:58:26.149 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.145424+0000", 2024-03-11T20:58:26.149 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1378e-05 2024-03-11T20:58:26.149 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T20:58:26.149 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T20:58:26.149 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T20:58:26.149 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.145430+0000", 2024-03-11T20:58:26.149 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4470000000000003e-06 2024-03-11T20:58:26.149 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T20:58:26.149 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T20:58:26.149 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T20:58:26.149 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.145480+0000", 2024-03-11T20:58:26.149 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9873999999999997e-05 2024-03-11T20:58:26.150 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T20:58:26.150 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T20:58:26.150 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T20:58:26.150 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.145569+0000", 2024-03-11T20:58:26.150 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8487999999999997e-05 2024-03-11T20:58:26.150 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T20:58:26.150 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T20:58:26.150 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 6", 2024-03-11T20:58:26.150 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.145649+0000", 2024-03-11T20:58:26.150 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.0023000000000003e-05 2024-03-11T20:58:26.150 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T20:58:26.150 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T20:58:26.150 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T20:58:26.151 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T20:58:26.151 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-03-11T20:58:26.151 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 2 2024-03-11T20:58:26.151 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-03-11T20:58:26.187 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:26.216 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:26.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:26.219 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:58:26.320 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:26.343 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:26.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:26.345 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:58:26.446 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:26.454 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:26 smithi146 bash[23809]: audit 2024-03-11T20:58:25.177298+0000 mgr.y (mgr.24380) 285 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:58:26.475 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:26.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:26.476 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:58:26.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:26 smithi023 bash[21372]: audit 2024-03-11T20:58:25.177298+0000 mgr.y (mgr.24380) 285 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:58:26.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:26 smithi023 bash[29146]: audit 2024-03-11T20:58:25.177298+0000 mgr.y (mgr.24380) 285 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:58:26.577 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:26.594 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:26.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:26.603 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:58:26.705 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:58:26.726 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:58:26.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:26.728 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:58:26.829 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:26.856 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:26.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:26.858 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:58:26.959 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:26.990 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:26.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:26.992 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:58:27.093 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:27.114 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:27.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:27.116 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:58:27.171 INFO:teuthology.orchestra.run.smithi023.stderr:Creating an OSD daemon form without an OSD FSID value 2024-03-11T20:58:27.217 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:27.251 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:27.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:27.255 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:58:27.286 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-03-11T20:58:27.356 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:27.401 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:27.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:27.405 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:58:27.506 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:27.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:27 smithi146 bash[23809]: cluster 2024-03-11T20:58:25.984538+0000 mgr.y (mgr.24380) 286 : cluster 0 pgmap v286: 148 pgs: 89 active+clean, 34 active+undersized, 22 active+undersized+degraded, 3 active+clean+remapped; 418 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.23k op/s; 1291/20616 objects degraded (6.262%); 1297/20616 objects misplaced (6.291%) 2024-03-11T20:58:27.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:27 smithi146 bash[23809]: cluster 2024-03-11T20:58:26.575624+0000 osd.4 (osd.4) 171 : cluster 0 purged_snaps scrub starts 2024-03-11T20:58:27.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:27 smithi146 bash[23809]: cluster 2024-03-11T20:58:26.575675+0000 osd.4 (osd.4) 172 : cluster 0 purged_snaps scrub ok 2024-03-11T20:58:27.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:27.526 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:27.526 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:58:27.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:27 smithi023 bash[21372]: cluster 2024-03-11T20:58:25.984538+0000 mgr.y (mgr.24380) 286 : cluster 0 pgmap v286: 148 pgs: 89 active+clean, 34 active+undersized, 22 active+undersized+degraded, 3 active+clean+remapped; 418 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.23k op/s; 1291/20616 objects degraded (6.262%); 1297/20616 objects misplaced (6.291%) 2024-03-11T20:58:27.536 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:27 smithi023 bash[21372]: cluster 2024-03-11T20:58:26.575624+0000 osd.4 (osd.4) 171 : cluster 0 purged_snaps scrub starts 2024-03-11T20:58:27.536 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:27 smithi023 bash[21372]: cluster 2024-03-11T20:58:26.575675+0000 osd.4 (osd.4) 172 : cluster 0 purged_snaps scrub ok 2024-03-11T20:58:27.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:27 smithi023 bash[29146]: cluster 2024-03-11T20:58:25.984538+0000 mgr.y (mgr.24380) 286 : cluster 0 pgmap v286: 148 pgs: 89 active+clean, 34 active+undersized, 22 active+undersized+degraded, 3 active+clean+remapped; 418 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.23k op/s; 1291/20616 objects degraded (6.262%); 1297/20616 objects misplaced (6.291%) 2024-03-11T20:58:27.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:27 smithi023 bash[29146]: cluster 2024-03-11T20:58:26.575624+0000 osd.4 (osd.4) 171 : cluster 0 purged_snaps scrub starts 2024-03-11T20:58:27.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:27 smithi023 bash[29146]: cluster 2024-03-11T20:58:26.575675+0000 osd.4 (osd.4) 172 : cluster 0 purged_snaps scrub ok 2024-03-11T20:58:27.628 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:27.658 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:27.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:27.661 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:58:27.762 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:27.786 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:27.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:27.791 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:58:27.892 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:27.915 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:27.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:27.917 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:58:28.018 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:58:28.043 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:58:28.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:28.046 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:58:28.147 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:28.182 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:28.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:28.184 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:58:28.285 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:28.348 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:28.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:28.351 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:58:28.452 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:28.480 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:28.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:28.484 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:58:28.534 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:58:28 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:20:58:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T20:58:28.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:28 smithi023 bash[21372]: cluster 2024-03-11T20:58:27.319592+0000 mon.a (mon.0) 1102 : cluster 0 osdmap e179: 8 total, 7 up, 8 in 2024-03-11T20:58:28.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:28 smithi023 bash[29146]: cluster 2024-03-11T20:58:27.319592+0000 mon.a (mon.0) 1102 : cluster 0 osdmap e179: 8 total, 7 up, 8 in 2024-03-11T20:58:28.585 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:28.594 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:28 smithi146 bash[23809]: cluster 2024-03-11T20:58:27.319592+0000 mon.a (mon.0) 1102 : cluster 0 osdmap e179: 8 total, 7 up, 8 in 2024-03-11T20:58:28.611 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:28.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:28.617 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:58:28.717 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:28.757 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:28.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:28.759 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:58:28.860 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:28.893 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:28.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:28.898 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:58:28.999 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:29.021 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:29.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:29.023 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:58:29.124 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:29.150 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:29.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:29.152 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:58:29.253 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:29.276 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:29.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:29.279 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:58:29.380 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:29.390 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:29 smithi023 bash[21372]: cluster 2024-03-11T20:58:27.985132+0000 mgr.y (mgr.24380) 287 : cluster 0 pgmap v288: 148 pgs: 89 active+clean, 34 active+undersized, 22 active+undersized+degraded, 3 active+clean+remapped; 418 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 921 B/s rd, 0 B/s wr, 1k op/s; 1291/20616 objects degraded (6.262%); 1297/20616 objects misplaced (6.291%) 2024-03-11T20:58:29.390 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:29 smithi023 bash[29146]: cluster 2024-03-11T20:58:27.985132+0000 mgr.y (mgr.24380) 287 : cluster 0 pgmap v288: 148 pgs: 89 active+clean, 34 active+undersized, 22 active+undersized+degraded, 3 active+clean+remapped; 418 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 921 B/s rd, 0 B/s wr, 1k op/s; 1291/20616 objects degraded (6.262%); 1297/20616 objects misplaced (6.291%) 2024-03-11T20:58:29.412 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:29.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:29.418 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:58:29.519 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:29.528 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:29 smithi146 bash[23809]: cluster 2024-03-11T20:58:27.985132+0000 mgr.y (mgr.24380) 287 : cluster 0 pgmap v288: 148 pgs: 89 active+clean, 34 active+undersized, 22 active+undersized+degraded, 3 active+clean+remapped; 418 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 921 B/s rd, 0 B/s wr, 1k op/s; 1291/20616 objects degraded (6.262%); 1297/20616 objects misplaced (6.291%) 2024-03-11T20:58:29.541 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:29.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:29.545 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:58:29.578 INFO:teuthology.orchestra.run.smithi023.stderr:Exporting 3.19 info 3.19( v 84'15 (0'0,84'15] local-lis/les=162/163 n=7 ec=78/78 lis/c=162/162 les/c/f=163/163/0 sis=162) 2024-03-11T20:58:29.579 INFO:teuthology.orchestra.run.smithi023.stderr:Read #3:98c4c580:::obj_delete_at_hint.0000000100:head# 2024-03-11T20:58:29.579 INFO:teuthology.orchestra.run.smithi023.stderr:Read #3:9a6dd8c0:gc::gc.0:head# 2024-03-11T20:58:29.580 INFO:teuthology.orchestra.run.smithi023.stderr:Read #3:9c5cb460:::obj_delete_at_hint.0000000093:head# 2024-03-11T20:58:29.580 INFO:teuthology.orchestra.run.smithi023.stderr:Read #3:9ed7f5b7:::obj_delete_at_hint.0000000004:head# 2024-03-11T20:58:29.580 INFO:teuthology.orchestra.run.smithi023.stderr:Read #3:9edc23b1:::obj_delete_at_hint.0000000063:head# 2024-03-11T20:58:29.580 INFO:teuthology.orchestra.run.smithi023.stderr:Read #3:9f21944d:::obj_delete_at_hint.0000000122:head# 2024-03-11T20:58:29.580 INFO:teuthology.orchestra.run.smithi023.stderr:Read #3:9fecf831:::obj_delete_at_hint.0000000057:head# 2024-03-11T20:58:29.580 INFO:teuthology.orchestra.run.smithi023.stderr:Export successful 2024-03-11T20:58:29.646 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:29.683 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:29.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:29.690 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:58:29.790 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:29.813 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:29.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:29.815 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:58:29.876 INFO:teuthology.orchestra.run.smithi023.stdout:osd.0: {} 2024-03-11T20:58:29.916 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:58:29.929 INFO:teuthology.orchestra.run.smithi023.stdout:osd.2: {} 2024-03-11T20:58:29.950 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:58:29.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:29.955 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:58:29.976 INFO:teuthology.orchestra.run.smithi023.stdout:osd.3: {} 2024-03-11T20:58:30.020 INFO:teuthology.orchestra.run.smithi023.stdout:osd.4: {} 2024-03-11T20:58:30.056 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:58:30.063 INFO:teuthology.orchestra.run.smithi023.stdout:osd.5: {} 2024-03-11T20:58:30.080 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:58:30.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:30.082 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:58:30.110 INFO:teuthology.orchestra.run.smithi023.stdout:osd.6: {} 2024-03-11T20:58:30.156 INFO:teuthology.orchestra.run.smithi023.stdout:osd.7: {} 2024-03-11T20:58:30.161 INFO:teuthology.orchestra.run.smithi023.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T20:58:30.161 INFO:teuthology.orchestra.run.smithi023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-11T20:58:30.161 INFO:teuthology.orchestra.run.smithi023.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T20:58:30.161 INFO:teuthology.orchestra.run.smithi023.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T20:58:30.162 INFO:teuthology.orchestra.run.smithi023.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T20:58:30.162 INFO:teuthology.orchestra.run.smithi023.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T20:58:30.162 INFO:teuthology.orchestra.run.smithi023.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T20:58:30.162 INFO:teuthology.orchestra.run.smithi023.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T20:58:30.183 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:58:30.215 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:58:30.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:30.218 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:58:30.319 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:30.351 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:30.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:30.354 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:58:30.455 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:58:30.465 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:30 smithi023 bash[21372]: cluster 2024-03-11T20:58:29.707521+0000 mon.a (mon.0) 1103 : cluster 3 Health check update: Degraded data redundancy: 1291/20616 objects degraded (6.262%), 22 pgs degraded (PG_DEGRADED) 2024-03-11T20:58:30.465 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:30 smithi023 bash[21372]: audit 2024-03-11T20:58:29.823745+0000 mon.c (mon.2) 38 : audit 0 from='client.? 172.21.15.23:0/1969574533' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T20:58:30.465 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:30 smithi023 bash[21372]: audit 2024-03-11T20:58:30.035518+0000 mon.a (mon.0) 1104 : audit 1 from='client.? 172.21.15.23:0/2622382931' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T20:58:30.466 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:30 smithi023 bash[29146]: cluster 2024-03-11T20:58:29.707521+0000 mon.a (mon.0) 1103 : cluster 3 Health check update: Degraded data redundancy: 1291/20616 objects degraded (6.262%), 22 pgs degraded (PG_DEGRADED) 2024-03-11T20:58:30.466 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:30 smithi023 bash[29146]: audit 2024-03-11T20:58:29.823745+0000 mon.c (mon.2) 38 : audit 0 from='client.? 172.21.15.23:0/1969574533' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T20:58:30.466 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:30 smithi023 bash[29146]: audit 2024-03-11T20:58:30.035518+0000 mon.a (mon.0) 1104 : audit 1 from='client.? 172.21.15.23:0/2622382931' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T20:58:30.480 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:58:30.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:30.485 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:58:30.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:30 smithi146 bash[23809]: cluster 2024-03-11T20:58:29.707521+0000 mon.a (mon.0) 1103 : cluster 3 Health check update: Degraded data redundancy: 1291/20616 objects degraded (6.262%), 22 pgs degraded (PG_DEGRADED) 2024-03-11T20:58:30.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:30 smithi146 bash[23809]: audit 2024-03-11T20:58:29.823745+0000 mon.c (mon.2) 38 : audit 0 from='client.? 172.21.15.23:0/1969574533' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T20:58:30.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:30 smithi146 bash[23809]: audit 2024-03-11T20:58:30.035518+0000 mon.a (mon.0) 1104 : audit 1 from='client.? 172.21.15.23:0/2622382931' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T20:58:30.586 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:58:30.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:30.620 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:58:30.620 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:58:30.721 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:30.724 INFO:teuthology.orchestra.run.smithi023.stderr:noscrub is unset 2024-03-11T20:58:30.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:30.755 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:30.755 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:58:30.856 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:30.878 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:30.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:30.880 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:58:30.981 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:58:31.014 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:58:31.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:31.018 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:58:31.119 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:31.147 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:31.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:31.152 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:58:31.253 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:58:31.286 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:58:31.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:31.289 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:58:31.390 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:31.400 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:31 smithi146 bash[23809]: cluster 2024-03-11T20:58:29.985980+0000 mgr.y (mgr.24380) 288 : cluster 0 pgmap v289: 148 pgs: 89 active+clean, 34 active+undersized, 22 active+undersized+degraded, 3 active+clean+remapped; 313 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 921 B/s rd, 0 B/s wr, 963 op/s; 1014/15609 objects degraded (6.496%); 726/15609 objects misplaced (4.651%) 2024-03-11T20:58:31.400 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:31 smithi146 bash[23809]: cluster 2024-03-11T20:58:30.711298+0000 mon.a (mon.0) 1105 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T20:58:31.400 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:31 smithi146 bash[23809]: audit 2024-03-11T20:58:30.719461+0000 mon.a (mon.0) 1106 : audit 1 from='client.? 172.21.15.23:0/2622382931' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T20:58:31.400 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:31 smithi146 bash[23809]: cluster 2024-03-11T20:58:30.725343+0000 mon.a (mon.0) 1107 : cluster 0 osdmap e180: 8 total, 7 up, 8 in 2024-03-11T20:58:31.401 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:31 smithi146 bash[23809]: cluster 2024-03-11T20:58:30.849134+0000 osd.2 (osd.2) 124 : cluster 0 2.c scrub starts 2024-03-11T20:58:31.401 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:31 smithi146 bash[23809]: cluster 2024-03-11T20:58:30.854878+0000 osd.2 (osd.2) 125 : cluster 0 2.c scrub ok 2024-03-11T20:58:31.416 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:31.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:31.422 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:58:31.522 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:31.552 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:31.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:31.557 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:58:31.658 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:31.683 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:31.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:31.687 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:58:31.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:31 smithi023 bash[21372]: cluster 2024-03-11T20:58:29.985980+0000 mgr.y (mgr.24380) 288 : cluster 0 pgmap v289: 148 pgs: 89 active+clean, 34 active+undersized, 22 active+undersized+degraded, 3 active+clean+remapped; 313 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 921 B/s rd, 0 B/s wr, 963 op/s; 1014/15609 objects degraded (6.496%); 726/15609 objects misplaced (4.651%) 2024-03-11T20:58:31.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:31 smithi023 bash[21372]: cluster 2024-03-11T20:58:30.711298+0000 mon.a (mon.0) 1105 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T20:58:31.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:31 smithi023 bash[21372]: audit 2024-03-11T20:58:30.719461+0000 mon.a (mon.0) 1106 : audit 1 from='client.? 172.21.15.23:0/2622382931' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T20:58:31.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:31 smithi023 bash[21372]: cluster 2024-03-11T20:58:30.725343+0000 mon.a (mon.0) 1107 : cluster 0 osdmap e180: 8 total, 7 up, 8 in 2024-03-11T20:58:31.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:31 smithi023 bash[21372]: cluster 2024-03-11T20:58:30.849134+0000 osd.2 (osd.2) 124 : cluster 0 2.c scrub starts 2024-03-11T20:58:31.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:31 smithi023 bash[21372]: cluster 2024-03-11T20:58:30.854878+0000 osd.2 (osd.2) 125 : cluster 0 2.c scrub ok 2024-03-11T20:58:31.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:31 smithi023 bash[29146]: cluster 2024-03-11T20:58:29.985980+0000 mgr.y (mgr.24380) 288 : cluster 0 pgmap v289: 148 pgs: 89 active+clean, 34 active+undersized, 22 active+undersized+degraded, 3 active+clean+remapped; 313 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 921 B/s rd, 0 B/s wr, 963 op/s; 1014/15609 objects degraded (6.496%); 726/15609 objects misplaced (4.651%) 2024-03-11T20:58:31.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:31 smithi023 bash[29146]: cluster 2024-03-11T20:58:30.711298+0000 mon.a (mon.0) 1105 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T20:58:31.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:31 smithi023 bash[29146]: audit 2024-03-11T20:58:30.719461+0000 mon.a (mon.0) 1106 : audit 1 from='client.? 172.21.15.23:0/2622382931' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T20:58:31.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:31 smithi023 bash[29146]: cluster 2024-03-11T20:58:30.725343+0000 mon.a (mon.0) 1107 : cluster 0 osdmap e180: 8 total, 7 up, 8 in 2024-03-11T20:58:31.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:31 smithi023 bash[29146]: cluster 2024-03-11T20:58:30.849134+0000 osd.2 (osd.2) 124 : cluster 0 2.c scrub starts 2024-03-11T20:58:31.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:31 smithi023 bash[29146]: cluster 2024-03-11T20:58:30.854878+0000 osd.2 (osd.2) 125 : cluster 0 2.c scrub ok 2024-03-11T20:58:31.788 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:31.811 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:31.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:31.815 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:58:31.915 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:31.951 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:31.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:31.953 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:58:32.054 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:32.056 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:Removed 110238 objects 2024-03-11T20:58:32.056 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:Clean up completed and total clean up time :129.5 2024-03-11T20:58:32.071 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:32.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:32.073 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:58:32.117 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T20:58:32.174 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:58:32.192 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:58:32.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:32.194 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:58:32.295 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:58:32.320 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:58:32.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:32.322 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:58:32.422 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:32.448 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:32.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:32.450 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:58:32.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:32 smithi146 bash[23809]: cluster 2024-03-11T20:58:31.816762+0000 osd.2 (osd.2) 126 : cluster 0 2.e scrub starts 2024-03-11T20:58:32.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:32 smithi146 bash[23809]: cluster 2024-03-11T20:58:31.839982+0000 osd.2 (osd.2) 127 : cluster 0 2.e scrub ok 2024-03-11T20:58:32.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:32 smithi146 bash[23809]: cluster 2024-03-11T20:58:31.904983+0000 osd.6 (osd.6) 114 : cluster 0 2.1a scrub starts 2024-03-11T20:58:32.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:32 smithi146 bash[23809]: cluster 2024-03-11T20:58:31.909941+0000 osd.6 (osd.6) 115 : cluster 0 2.1a scrub ok 2024-03-11T20:58:32.551 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:32.577 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:32.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:32.579 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:58:32.680 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:58:32.706 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:58:32.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:32.708 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:58:32.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:32 smithi023 bash[21372]: cluster 2024-03-11T20:58:31.816762+0000 osd.2 (osd.2) 126 : cluster 0 2.e scrub starts 2024-03-11T20:58:32.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:32 smithi023 bash[21372]: cluster 2024-03-11T20:58:31.839982+0000 osd.2 (osd.2) 127 : cluster 0 2.e scrub ok 2024-03-11T20:58:32.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:32 smithi023 bash[21372]: cluster 2024-03-11T20:58:31.904983+0000 osd.6 (osd.6) 114 : cluster 0 2.1a scrub starts 2024-03-11T20:58:32.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:32 smithi023 bash[21372]: cluster 2024-03-11T20:58:31.909941+0000 osd.6 (osd.6) 115 : cluster 0 2.1a scrub ok 2024-03-11T20:58:32.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:32 smithi023 bash[29146]: cluster 2024-03-11T20:58:31.816762+0000 osd.2 (osd.2) 126 : cluster 0 2.e scrub starts 2024-03-11T20:58:32.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:32 smithi023 bash[29146]: cluster 2024-03-11T20:58:31.839982+0000 osd.2 (osd.2) 127 : cluster 0 2.e scrub ok 2024-03-11T20:58:32.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:32 smithi023 bash[29146]: cluster 2024-03-11T20:58:31.904983+0000 osd.6 (osd.6) 114 : cluster 0 2.1a scrub starts 2024-03-11T20:58:32.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:32 smithi023 bash[29146]: cluster 2024-03-11T20:58:31.909941+0000 osd.6 (osd.6) 115 : cluster 0 2.1a scrub ok 2024-03-11T20:58:32.809 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:32.838 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:32.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:32.840 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:58:32.941 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:58:32.967 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:58:32.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:32.970 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:58:33.071 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:58:33.101 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:58:33.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:33.103 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:58:33.108 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -n osd.1 --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-1 --journal-path /var/lib/ceph/osd/ceph-1/journal --force --op remove --pgid 3.19 2024-03-11T20:58:33.140 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-11T20:58:33.141 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-11T20:58:33.204 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:58:33.254 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:58:33.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:33.256 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:58:33.341 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/osd.1/config 2024-03-11T20:58:33.357 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:33.384 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:33.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:33.387 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:58:33.487 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:33.516 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:33.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:33.518 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:58:33.619 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:58:33.630 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:33 smithi146 bash[23809]: cluster 2024-03-11T20:58:31.940197+0000 osd.0 (osd.0) 161 : cluster 0 3.1b scrub starts 2024-03-11T20:58:33.630 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:33 smithi146 bash[23809]: cluster 2024-03-11T20:58:31.943094+0000 osd.0 (osd.0) 162 : cluster 0 3.1b scrub ok 2024-03-11T20:58:33.630 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:33 smithi146 bash[23809]: cluster 2024-03-11T20:58:31.987223+0000 mgr.y (mgr.24380) 289 : cluster 0 pgmap v291: 148 pgs: 89 active+clean, 34 active+undersized, 22 active+undersized+degraded, 3 active+clean+remapped; 131 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.16k op/s; 430/6849 objects degraded (6.278%); 446/6849 objects misplaced (6.512%) 2024-03-11T20:58:33.630 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:33 smithi146 bash[23809]: cluster 2024-03-11T20:58:32.856858+0000 osd.2 (osd.2) 128 : cluster 0 4.16 scrub starts 2024-03-11T20:58:33.631 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:33 smithi146 bash[23809]: cluster 2024-03-11T20:58:32.858460+0000 osd.2 (osd.2) 129 : cluster 0 4.16 scrub ok 2024-03-11T20:58:33.645 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:58:33.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:33.647 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:58:33.748 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:33.760 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:33 smithi023 bash[21372]: cluster 2024-03-11T20:58:31.940197+0000 osd.0 (osd.0) 161 : cluster 0 3.1b scrub starts 2024-03-11T20:58:33.760 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:33 smithi023 bash[21372]: cluster 2024-03-11T20:58:31.943094+0000 osd.0 (osd.0) 162 : cluster 0 3.1b scrub ok 2024-03-11T20:58:33.761 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:33 smithi023 bash[21372]: cluster 2024-03-11T20:58:31.987223+0000 mgr.y (mgr.24380) 289 : cluster 0 pgmap v291: 148 pgs: 89 active+clean, 34 active+undersized, 22 active+undersized+degraded, 3 active+clean+remapped; 131 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.16k op/s; 430/6849 objects degraded (6.278%); 446/6849 objects misplaced (6.512%) 2024-03-11T20:58:33.761 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:33 smithi023 bash[21372]: cluster 2024-03-11T20:58:32.856858+0000 osd.2 (osd.2) 128 : cluster 0 4.16 scrub starts 2024-03-11T20:58:33.761 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:33 smithi023 bash[21372]: cluster 2024-03-11T20:58:32.858460+0000 osd.2 (osd.2) 129 : cluster 0 4.16 scrub ok 2024-03-11T20:58:33.761 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:33 smithi023 bash[29146]: cluster 2024-03-11T20:58:31.940197+0000 osd.0 (osd.0) 161 : cluster 0 3.1b scrub starts 2024-03-11T20:58:33.761 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:33 smithi023 bash[29146]: cluster 2024-03-11T20:58:31.943094+0000 osd.0 (osd.0) 162 : cluster 0 3.1b scrub ok 2024-03-11T20:58:33.761 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:33 smithi023 bash[29146]: cluster 2024-03-11T20:58:31.987223+0000 mgr.y (mgr.24380) 289 : cluster 0 pgmap v291: 148 pgs: 89 active+clean, 34 active+undersized, 22 active+undersized+degraded, 3 active+clean+remapped; 131 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.16k op/s; 430/6849 objects degraded (6.278%); 446/6849 objects misplaced (6.512%) 2024-03-11T20:58:33.761 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:33 smithi023 bash[29146]: cluster 2024-03-11T20:58:32.856858+0000 osd.2 (osd.2) 128 : cluster 0 4.16 scrub starts 2024-03-11T20:58:33.761 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:33 smithi023 bash[29146]: cluster 2024-03-11T20:58:32.858460+0000 osd.2 (osd.2) 129 : cluster 0 4.16 scrub ok 2024-03-11T20:58:33.777 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:33.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:33.779 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:58:33.880 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:58:33.907 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:58:33.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:33.909 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:58:34.010 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:58:34.033 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:58:34.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:34.036 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:58:34.136 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:34.165 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:34.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:34.167 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:58:34.268 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:34.294 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:34.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:34.296 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:58:34.397 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:34.406 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:34 smithi023 bash[21372]: cluster 2024-03-11T20:58:33.126823+0000 osd.4 (osd.4) 173 : cluster 0 6.1e scrub starts 2024-03-11T20:58:34.406 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:34 smithi023 bash[21372]: cluster 2024-03-11T20:58:33.135740+0000 osd.4 (osd.4) 174 : cluster 0 6.1e scrub ok 2024-03-11T20:58:34.406 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:34 smithi023 bash[21372]: cluster 2024-03-11T20:58:33.547778+0000 osd.3 (osd.3) 186 : cluster 0 6.6 scrub starts 2024-03-11T20:58:34.406 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:34 smithi023 bash[21372]: cluster 2024-03-11T20:58:33.553186+0000 osd.3 (osd.3) 187 : cluster 0 6.6 scrub ok 2024-03-11T20:58:34.407 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:34 smithi023 bash[21372]: cluster 2024-03-11T20:58:33.848547+0000 osd.6 (osd.6) 116 : cluster 0 5.2 scrub starts 2024-03-11T20:58:34.407 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:34 smithi023 bash[21372]: cluster 2024-03-11T20:58:33.851976+0000 osd.6 (osd.6) 117 : cluster 0 5.2 scrub ok 2024-03-11T20:58:34.407 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:34 smithi023 bash[21372]: cluster 2024-03-11T20:58:34.103179+0000 osd.7 (osd.7) 159 : cluster 0 6.5 scrub starts 2024-03-11T20:58:34.407 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:34 smithi023 bash[21372]: cluster 2024-03-11T20:58:34.121200+0000 osd.7 (osd.7) 160 : cluster 0 6.5 scrub ok 2024-03-11T20:58:34.407 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:34 smithi023 bash[29146]: cluster 2024-03-11T20:58:33.126823+0000 osd.4 (osd.4) 173 : cluster 0 6.1e scrub starts 2024-03-11T20:58:34.407 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:34 smithi023 bash[29146]: cluster 2024-03-11T20:58:33.135740+0000 osd.4 (osd.4) 174 : cluster 0 6.1e scrub ok 2024-03-11T20:58:34.407 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:34 smithi023 bash[29146]: cluster 2024-03-11T20:58:33.547778+0000 osd.3 (osd.3) 186 : cluster 0 6.6 scrub starts 2024-03-11T20:58:34.407 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:34 smithi023 bash[29146]: cluster 2024-03-11T20:58:33.553186+0000 osd.3 (osd.3) 187 : cluster 0 6.6 scrub ok 2024-03-11T20:58:34.407 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:34 smithi023 bash[29146]: cluster 2024-03-11T20:58:33.848547+0000 osd.6 (osd.6) 116 : cluster 0 5.2 scrub starts 2024-03-11T20:58:34.407 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:34 smithi023 bash[29146]: cluster 2024-03-11T20:58:33.851976+0000 osd.6 (osd.6) 117 : cluster 0 5.2 scrub ok 2024-03-11T20:58:34.407 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:34 smithi023 bash[29146]: cluster 2024-03-11T20:58:34.103179+0000 osd.7 (osd.7) 159 : cluster 0 6.5 scrub starts 2024-03-11T20:58:34.408 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:34 smithi023 bash[29146]: cluster 2024-03-11T20:58:34.121200+0000 osd.7 (osd.7) 160 : cluster 0 6.5 scrub ok 2024-03-11T20:58:34.420 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:34.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:34.423 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:58:34.524 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:34.545 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:34.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:34.548 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:58:34.648 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:34.660 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:34 smithi146 bash[23809]: cluster 2024-03-11T20:58:33.126823+0000 osd.4 (osd.4) 173 : cluster 0 6.1e scrub starts 2024-03-11T20:58:34.660 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:34 smithi146 bash[23809]: cluster 2024-03-11T20:58:33.135740+0000 osd.4 (osd.4) 174 : cluster 0 6.1e scrub ok 2024-03-11T20:58:34.660 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:34 smithi146 bash[23809]: cluster 2024-03-11T20:58:33.547778+0000 osd.3 (osd.3) 186 : cluster 0 6.6 scrub starts 2024-03-11T20:58:34.660 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:34 smithi146 bash[23809]: cluster 2024-03-11T20:58:33.553186+0000 osd.3 (osd.3) 187 : cluster 0 6.6 scrub ok 2024-03-11T20:58:34.660 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:34 smithi146 bash[23809]: cluster 2024-03-11T20:58:33.848547+0000 osd.6 (osd.6) 116 : cluster 0 5.2 scrub starts 2024-03-11T20:58:34.660 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:34 smithi146 bash[23809]: cluster 2024-03-11T20:58:33.851976+0000 osd.6 (osd.6) 117 : cluster 0 5.2 scrub ok 2024-03-11T20:58:34.660 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:34 smithi146 bash[23809]: cluster 2024-03-11T20:58:34.103179+0000 osd.7 (osd.7) 159 : cluster 0 6.5 scrub starts 2024-03-11T20:58:34.660 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:34 smithi146 bash[23809]: cluster 2024-03-11T20:58:34.121200+0000 osd.7 (osd.7) 160 : cluster 0 6.5 scrub ok 2024-03-11T20:58:34.672 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:34.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:34.675 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:58:34.775 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:58:34.803 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:58:34.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:34.805 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:58:34.906 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:34.927 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:34.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:34.929 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:58:35.030 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:35.051 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:35.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:35.053 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:58:35.141 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-11T20:58:35.142 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T20:58:35.154 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:58:35.178 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:58:35.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:35.181 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:58:35.281 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:35.300 INFO:teuthology.orchestra.run.smithi023.stderr:Creating an OSD daemon form without an OSD FSID value 2024-03-11T20:58:35.306 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:35.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:35.309 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:58:35.409 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:35.421 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 20:58:35 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T20:58:35.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:35 smithi146 bash[23809]: cluster 2024-03-11T20:58:33.988170+0000 mgr.y (mgr.24380) 290 : cluster 0 pgmap v292: 148 pgs: 89 active+clean, 34 active+undersized, 22 active+undersized+degraded, 3 active+clean+remapped; 131 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 573 op/s; 430/6849 objects degraded (6.278%); 446/6849 objects misplaced (6.512%) 2024-03-11T20:58:35.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:35 smithi146 bash[23809]: cluster 2024-03-11T20:58:34.502080+0000 osd.3 (osd.3) 188 : cluster 0 3.1e scrub starts 2024-03-11T20:58:35.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:35 smithi146 bash[23809]: cluster 2024-03-11T20:58:34.504569+0000 osd.3 (osd.3) 189 : cluster 0 3.1e scrub ok 2024-03-11T20:58:35.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:35 smithi146 bash[23809]: audit 2024-03-11T20:58:34.653999+0000 mon.a (mon.0) 1108 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:58:35.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:35 smithi146 bash[23809]: cluster 2024-03-11T20:58:34.708942+0000 mon.a (mon.0) 1109 : cluster 3 Health check update: Degraded data redundancy: 430/6849 objects degraded (6.278%), 22 pgs degraded (PG_DEGRADED) 2024-03-11T20:58:35.435 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:35.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:35.438 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:58:35.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:35 smithi023 bash[21372]: cluster 2024-03-11T20:58:33.988170+0000 mgr.y (mgr.24380) 290 : cluster 0 pgmap v292: 148 pgs: 89 active+clean, 34 active+undersized, 22 active+undersized+degraded, 3 active+clean+remapped; 131 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 573 op/s; 430/6849 objects degraded (6.278%); 446/6849 objects misplaced (6.512%) 2024-03-11T20:58:35.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:35 smithi023 bash[21372]: cluster 2024-03-11T20:58:34.502080+0000 osd.3 (osd.3) 188 : cluster 0 3.1e scrub starts 2024-03-11T20:58:35.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:35 smithi023 bash[21372]: cluster 2024-03-11T20:58:34.504569+0000 osd.3 (osd.3) 189 : cluster 0 3.1e scrub ok 2024-03-11T20:58:35.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:35 smithi023 bash[21372]: audit 2024-03-11T20:58:34.653999+0000 mon.a (mon.0) 1108 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:58:35.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:35 smithi023 bash[21372]: cluster 2024-03-11T20:58:34.708942+0000 mon.a (mon.0) 1109 : cluster 3 Health check update: Degraded data redundancy: 430/6849 objects degraded (6.278%), 22 pgs degraded (PG_DEGRADED) 2024-03-11T20:58:35.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:35 smithi023 bash[29146]: cluster 2024-03-11T20:58:33.988170+0000 mgr.y (mgr.24380) 290 : cluster 0 pgmap v292: 148 pgs: 89 active+clean, 34 active+undersized, 22 active+undersized+degraded, 3 active+clean+remapped; 131 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 573 op/s; 430/6849 objects degraded (6.278%); 446/6849 objects misplaced (6.512%) 2024-03-11T20:58:35.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:35 smithi023 bash[29146]: cluster 2024-03-11T20:58:34.502080+0000 osd.3 (osd.3) 188 : cluster 0 3.1e scrub starts 2024-03-11T20:58:35.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:35 smithi023 bash[29146]: cluster 2024-03-11T20:58:34.504569+0000 osd.3 (osd.3) 189 : cluster 0 3.1e scrub ok 2024-03-11T20:58:35.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:35 smithi023 bash[29146]: audit 2024-03-11T20:58:34.653999+0000 mon.a (mon.0) 1108 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:58:35.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:35 smithi023 bash[29146]: cluster 2024-03-11T20:58:34.708942+0000 mon.a (mon.0) 1109 : cluster 3 Health check update: Degraded data redundancy: 430/6849 objects degraded (6.278%), 22 pgs degraded (PG_DEGRADED) 2024-03-11T20:58:35.538 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:35.565 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:35.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:35.567 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:58:35.668 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:58:35.694 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:58:35.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:35.696 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:58:35.797 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:35.821 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:35.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:35.824 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:58:35.924 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:35.945 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:35.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:35.947 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:58:36.048 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:36.071 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:36.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:36.074 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:58:36.174 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:36.202 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:36.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:36.204 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:58:36.305 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:58:36.334 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:58:36.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:36.336 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:58:36.437 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:58:36.459 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:58:36.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:36.462 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:58:36.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:36 smithi146 bash[23809]: cluster 2024-03-11T20:58:35.087486+0000 osd.5 (osd.5) 151 : cluster 0 7.0 scrub starts 2024-03-11T20:58:36.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:36 smithi146 bash[23809]: cluster 2024-03-11T20:58:35.121975+0000 osd.5 (osd.5) 152 : cluster 0 7.0 scrub ok 2024-03-11T20:58:36.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:36 smithi146 bash[23809]: cluster 2024-03-11T20:58:35.152185+0000 osd.4 (osd.4) 175 : cluster 0 2.13 scrub starts 2024-03-11T20:58:36.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:36 smithi146 bash[23809]: cluster 2024-03-11T20:58:35.154168+0000 osd.4 (osd.4) 176 : cluster 0 2.13 scrub ok 2024-03-11T20:58:36.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:36 smithi146 bash[23809]: audit 2024-03-11T20:58:35.185747+0000 mgr.y (mgr.24380) 291 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:58:36.563 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:36.590 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:36.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:36.593 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:58:36.694 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:36.722 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:36.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:36.725 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:58:36.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:36 smithi023 bash[21372]: cluster 2024-03-11T20:58:35.087486+0000 osd.5 (osd.5) 151 : cluster 0 7.0 scrub starts 2024-03-11T20:58:36.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:36 smithi023 bash[21372]: cluster 2024-03-11T20:58:35.121975+0000 osd.5 (osd.5) 152 : cluster 0 7.0 scrub ok 2024-03-11T20:58:36.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:36 smithi023 bash[21372]: cluster 2024-03-11T20:58:35.152185+0000 osd.4 (osd.4) 175 : cluster 0 2.13 scrub starts 2024-03-11T20:58:36.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:36 smithi023 bash[21372]: cluster 2024-03-11T20:58:35.154168+0000 osd.4 (osd.4) 176 : cluster 0 2.13 scrub ok 2024-03-11T20:58:36.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:36 smithi023 bash[21372]: audit 2024-03-11T20:58:35.185747+0000 mgr.y (mgr.24380) 291 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:58:36.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:36 smithi023 bash[29146]: cluster 2024-03-11T20:58:35.087486+0000 osd.5 (osd.5) 151 : cluster 0 7.0 scrub starts 2024-03-11T20:58:36.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:36 smithi023 bash[29146]: cluster 2024-03-11T20:58:35.121975+0000 osd.5 (osd.5) 152 : cluster 0 7.0 scrub ok 2024-03-11T20:58:36.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:36 smithi023 bash[29146]: cluster 2024-03-11T20:58:35.152185+0000 osd.4 (osd.4) 175 : cluster 0 2.13 scrub starts 2024-03-11T20:58:36.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:36 smithi023 bash[29146]: cluster 2024-03-11T20:58:35.154168+0000 osd.4 (osd.4) 176 : cluster 0 2.13 scrub ok 2024-03-11T20:58:36.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:36 smithi023 bash[29146]: audit 2024-03-11T20:58:35.185747+0000 mgr.y (mgr.24380) 291 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:58:36.820 INFO:teuthology.orchestra.run.smithi023.stdout: marking collection for removal 2024-03-11T20:58:36.820 INFO:teuthology.orchestra.run.smithi023.stdout:setting '_remove' omap key 2024-03-11T20:58:36.820 INFO:teuthology.orchestra.run.smithi023.stdout:finish_remove_pgs 3.19_head removing 3.19 2024-03-11T20:58:36.821 INFO:teuthology.orchestra.run.smithi023.stdout:Remove successful 2024-03-11T20:58:36.825 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:36.846 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:36.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:36.849 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:58:36.950 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:36.978 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:36.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:36.980 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:58:37.081 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:37.107 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:37.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:37.109 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:58:37.210 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:37.230 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:37.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:37.232 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:58:37.312 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-03-11T20:58:37.312 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T20:58:37.315 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_0 2024-03-11T20:58:37.315 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2024-03-11T20:58:37.334 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:37.354 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:37.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:37.356 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:58:37.457 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:37.479 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:37.480 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:37 smithi146 bash[23809]: cluster 2024-03-11T20:58:35.869161+0000 osd.0 (osd.0) 163 : cluster 0 3.1f scrub starts 2024-03-11T20:58:37.480 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:37 smithi146 bash[23809]: cluster 2024-03-11T20:58:35.873006+0000 osd.0 (osd.0) 164 : cluster 0 3.1f scrub ok 2024-03-11T20:58:37.480 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:37 smithi146 bash[23809]: cluster 2024-03-11T20:58:35.989991+0000 mgr.y (mgr.24380) 292 : cluster 0 pgmap v293: 148 pgs: 89 active+clean, 37 active+undersized, 19 active+undersized+degraded, 3 active+clean+remapped; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 771 op/s; 79/594 objects degraded (13.300%) 2024-03-11T20:58:37.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:37.481 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:58:37.577 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:37 smithi023 bash[21372]: cluster 2024-03-11T20:58:35.869161+0000 osd.0 (osd.0) 163 : cluster 0 3.1f scrub starts 2024-03-11T20:58:37.577 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:37 smithi023 bash[21372]: cluster 2024-03-11T20:58:35.873006+0000 osd.0 (osd.0) 164 : cluster 0 3.1f scrub ok 2024-03-11T20:58:37.577 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:37 smithi023 bash[21372]: cluster 2024-03-11T20:58:35.989991+0000 mgr.y (mgr.24380) 292 : cluster 0 pgmap v293: 148 pgs: 89 active+clean, 37 active+undersized, 19 active+undersized+degraded, 3 active+clean+remapped; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 771 op/s; 79/594 objects degraded (13.300%) 2024-03-11T20:58:37.577 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:37 smithi023 bash[29146]: cluster 2024-03-11T20:58:35.869161+0000 osd.0 (osd.0) 163 : cluster 0 3.1f scrub starts 2024-03-11T20:58:37.577 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:37 smithi023 bash[29146]: cluster 2024-03-11T20:58:35.873006+0000 osd.0 (osd.0) 164 : cluster 0 3.1f scrub ok 2024-03-11T20:58:37.578 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:37 smithi023 bash[29146]: cluster 2024-03-11T20:58:35.989991+0000 mgr.y (mgr.24380) 292 : cluster 0 pgmap v293: 148 pgs: 89 active+clean, 37 active+undersized, 19 active+undersized+degraded, 3 active+clean+remapped; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 771 op/s; 79/594 objects degraded (13.300%) 2024-03-11T20:58:37.582 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:37.610 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:37.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:37.612 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:58:37.713 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:37.741 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:37.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:37.744 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:58:37.845 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:37.874 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:37.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:37.876 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:58:37.977 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:38.005 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:38.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:38.007 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:58:38.108 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:38.135 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:38.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:38.138 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:58:38.161 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-03-11T20:58:38.213 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -n osd.1 --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-1 --journal-path /var/lib/ceph/osd/ceph-1/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.3.19.1 2024-03-11T20:58:38.239 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:38.269 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:38.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:38.271 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:58:38.373 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:38.389 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/osd.1/config 2024-03-11T20:58:38.392 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:38.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:38.394 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:58:38.495 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:58:38.521 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:58:38.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:38.524 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:58:38.624 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:38.636 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:38 smithi146 bash[23809]: cluster 2024-03-11T20:58:37.072406+0000 osd.5 (osd.5) 153 : cluster 0 3.c scrub starts 2024-03-11T20:58:38.636 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:38 smithi146 bash[23809]: cluster 2024-03-11T20:58:37.076643+0000 osd.5 (osd.5) 154 : cluster 0 3.c scrub ok 2024-03-11T20:58:38.636 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:38 smithi146 bash[23809]: cluster 2024-03-11T20:58:37.169060+0000 osd.4 (osd.4) 177 : cluster 0 4.1 scrub starts 2024-03-11T20:58:38.636 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:38 smithi146 bash[23809]: cluster 2024-03-11T20:58:37.170240+0000 osd.4 (osd.4) 178 : cluster 0 4.1 scrub ok 2024-03-11T20:58:38.636 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:38 smithi146 bash[23809]: cluster 2024-03-11T20:58:37.271200+0000 osd.5 (osd.5) 155 : cluster 0 purged_snaps scrub starts 2024-03-11T20:58:38.636 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:38 smithi146 bash[23809]: cluster 2024-03-11T20:58:37.271268+0000 osd.5 (osd.5) 156 : cluster 0 purged_snaps scrub ok 2024-03-11T20:58:38.636 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:38 smithi146 bash[23809]: cluster 2024-03-11T20:58:37.427248+0000 mon.a (mon.0) 1110 : cluster 0 osdmap e181: 8 total, 7 up, 8 in 2024-03-11T20:58:38.636 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:38 smithi146 bash[23809]: cluster 2024-03-11T20:58:37.834289+0000 osd.2 (osd.2) 130 : cluster 0 7.8 scrub starts 2024-03-11T20:58:38.636 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:38 smithi146 bash[23809]: cluster 2024-03-11T20:58:37.875627+0000 osd.2 (osd.2) 131 : cluster 0 7.8 scrub ok 2024-03-11T20:58:38.636 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:38 smithi146 bash[23809]: cluster 2024-03-11T20:58:38.134205+0000 osd.7 (osd.7) 161 : cluster 0 6.1c scrub starts 2024-03-11T20:58:38.637 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:38 smithi146 bash[23809]: cluster 2024-03-11T20:58:38.137117+0000 osd.7 (osd.7) 162 : cluster 0 6.1c scrub ok 2024-03-11T20:58:38.651 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:38.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:38.653 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:58:38.754 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:38.783 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:38.784 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:58:38 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:20:58:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T20:58:38.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:38 smithi023 bash[21372]: cluster 2024-03-11T20:58:37.072406+0000 osd.5 (osd.5) 153 : cluster 0 3.c scrub starts 2024-03-11T20:58:38.786 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:38 smithi023 bash[21372]: cluster 2024-03-11T20:58:37.076643+0000 osd.5 (osd.5) 154 : cluster 0 3.c scrub ok 2024-03-11T20:58:38.786 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:38 smithi023 bash[21372]: cluster 2024-03-11T20:58:37.169060+0000 osd.4 (osd.4) 177 : cluster 0 4.1 scrub starts 2024-03-11T20:58:38.786 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:38 smithi023 bash[21372]: cluster 2024-03-11T20:58:37.170240+0000 osd.4 (osd.4) 178 : cluster 0 4.1 scrub ok 2024-03-11T20:58:38.786 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:38 smithi023 bash[21372]: cluster 2024-03-11T20:58:37.271200+0000 osd.5 (osd.5) 155 : cluster 0 purged_snaps scrub starts 2024-03-11T20:58:38.786 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:38 smithi023 bash[21372]: cluster 2024-03-11T20:58:37.271268+0000 osd.5 (osd.5) 156 : cluster 0 purged_snaps scrub ok 2024-03-11T20:58:38.786 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:38 smithi023 bash[21372]: cluster 2024-03-11T20:58:37.427248+0000 mon.a (mon.0) 1110 : cluster 0 osdmap e181: 8 total, 7 up, 8 in 2024-03-11T20:58:38.786 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:38 smithi023 bash[21372]: cluster 2024-03-11T20:58:37.834289+0000 osd.2 (osd.2) 130 : cluster 0 7.8 scrub starts 2024-03-11T20:58:38.786 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:38 smithi023 bash[21372]: cluster 2024-03-11T20:58:37.875627+0000 osd.2 (osd.2) 131 : cluster 0 7.8 scrub ok 2024-03-11T20:58:38.786 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:38 smithi023 bash[21372]: cluster 2024-03-11T20:58:38.134205+0000 osd.7 (osd.7) 161 : cluster 0 6.1c scrub starts 2024-03-11T20:58:38.786 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:38 smithi023 bash[21372]: cluster 2024-03-11T20:58:38.137117+0000 osd.7 (osd.7) 162 : cluster 0 6.1c scrub ok 2024-03-11T20:58:38.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:38.787 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:38 smithi023 bash[29146]: cluster 2024-03-11T20:58:37.072406+0000 osd.5 (osd.5) 153 : cluster 0 3.c scrub starts 2024-03-11T20:58:38.787 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:38 smithi023 bash[29146]: cluster 2024-03-11T20:58:37.076643+0000 osd.5 (osd.5) 154 : cluster 0 3.c scrub ok 2024-03-11T20:58:38.787 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:38 smithi023 bash[29146]: cluster 2024-03-11T20:58:37.169060+0000 osd.4 (osd.4) 177 : cluster 0 4.1 scrub starts 2024-03-11T20:58:38.787 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:38 smithi023 bash[29146]: cluster 2024-03-11T20:58:37.170240+0000 osd.4 (osd.4) 178 : cluster 0 4.1 scrub ok 2024-03-11T20:58:38.787 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:38 smithi023 bash[29146]: cluster 2024-03-11T20:58:37.271200+0000 osd.5 (osd.5) 155 : cluster 0 purged_snaps scrub starts 2024-03-11T20:58:38.787 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:38 smithi023 bash[29146]: cluster 2024-03-11T20:58:37.271268+0000 osd.5 (osd.5) 156 : cluster 0 purged_snaps scrub ok 2024-03-11T20:58:38.787 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:38 smithi023 bash[29146]: cluster 2024-03-11T20:58:37.427248+0000 mon.a (mon.0) 1110 : cluster 0 osdmap e181: 8 total, 7 up, 8 in 2024-03-11T20:58:38.787 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:38 smithi023 bash[29146]: cluster 2024-03-11T20:58:37.834289+0000 osd.2 (osd.2) 130 : cluster 0 7.8 scrub starts 2024-03-11T20:58:38.787 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:38 smithi023 bash[29146]: cluster 2024-03-11T20:58:37.875627+0000 osd.2 (osd.2) 131 : cluster 0 7.8 scrub ok 2024-03-11T20:58:38.788 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:38 smithi023 bash[29146]: cluster 2024-03-11T20:58:38.134205+0000 osd.7 (osd.7) 161 : cluster 0 6.1c scrub starts 2024-03-11T20:58:38.788 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:38 smithi023 bash[29146]: cluster 2024-03-11T20:58:38.137117+0000 osd.7 (osd.7) 162 : cluster 0 6.1c scrub ok 2024-03-11T20:58:38.788 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:58:38.889 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:38.916 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:38.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:38.918 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:58:39.019 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:39.047 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:39.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:39.049 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:58:39.150 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:58:39.178 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:58:39.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:39.181 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:58:39.282 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:39.311 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:39.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:39.313 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:58:39.414 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:58:39.442 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:58:39.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:39.444 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:58:39.545 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:39.549 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:39 smithi023 bash[21372]: cluster 2024-03-11T20:58:37.990857+0000 mgr.y (mgr.24380) 293 : cluster 0 pgmap v295: 148 pgs: 89 active+clean, 37 active+undersized, 19 active+undersized+degraded, 3 active+clean+remapped; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 626 op/s; 79/594 objects degraded (13.300%) 2024-03-11T20:58:39.549 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:39 smithi023 bash[21372]: cluster 2024-03-11T20:58:38.877831+0000 osd.6 (osd.6) 118 : cluster 0 2.14 scrub starts 2024-03-11T20:58:39.549 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:39 smithi023 bash[21372]: cluster 2024-03-11T20:58:38.880260+0000 osd.6 (osd.6) 119 : cluster 0 2.14 scrub ok 2024-03-11T20:58:39.549 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:39 smithi023 bash[29146]: cluster 2024-03-11T20:58:37.990857+0000 mgr.y (mgr.24380) 293 : cluster 0 pgmap v295: 148 pgs: 89 active+clean, 37 active+undersized, 19 active+undersized+degraded, 3 active+clean+remapped; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 626 op/s; 79/594 objects degraded (13.300%) 2024-03-11T20:58:39.550 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:39 smithi023 bash[29146]: cluster 2024-03-11T20:58:38.877831+0000 osd.6 (osd.6) 118 : cluster 0 2.14 scrub starts 2024-03-11T20:58:39.550 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:39 smithi023 bash[29146]: cluster 2024-03-11T20:58:38.880260+0000 osd.6 (osd.6) 119 : cluster 0 2.14 scrub ok 2024-03-11T20:58:39.567 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:39.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:39.569 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:58:39.670 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:39.698 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:39.699 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:39 smithi146 bash[23809]: cluster 2024-03-11T20:58:37.990857+0000 mgr.y (mgr.24380) 293 : cluster 0 pgmap v295: 148 pgs: 89 active+clean, 37 active+undersized, 19 active+undersized+degraded, 3 active+clean+remapped; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 626 op/s; 79/594 objects degraded (13.300%) 2024-03-11T20:58:39.699 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:39 smithi146 bash[23809]: cluster 2024-03-11T20:58:38.877831+0000 osd.6 (osd.6) 118 : cluster 0 2.14 scrub starts 2024-03-11T20:58:39.699 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:39 smithi146 bash[23809]: cluster 2024-03-11T20:58:38.880260+0000 osd.6 (osd.6) 119 : cluster 0 2.14 scrub ok 2024-03-11T20:58:39.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:39.700 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:58:39.802 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:39.829 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:39.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:39.831 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:58:39.932 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:58:39.953 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:58:39.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:39.955 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:58:40.056 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:58:40.082 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:58:40.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:40.085 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:58:40.185 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:40.213 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:40.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:40.215 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:58:40.285 INFO:teuthology.orchestra.run.smithi023.stderr:Creating an OSD daemon form without an OSD FSID value 2024-03-11T20:58:40.315 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:58:40.341 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:58:40.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:40.344 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:58:40.444 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:40.468 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:40.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:40.470 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:58:40.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:40 smithi023 bash[29146]: cluster 2024-03-11T20:58:39.165028+0000 osd.4 (osd.4) 179 : cluster 0 4.c scrub starts 2024-03-11T20:58:40.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:40 smithi023 bash[29146]: cluster 2024-03-11T20:58:39.167628+0000 osd.4 (osd.4) 180 : cluster 0 4.c scrub ok 2024-03-11T20:58:40.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:40 smithi023 bash[29146]: cluster 2024-03-11T20:58:39.710900+0000 mon.a (mon.0) 1111 : cluster 3 Health check update: Degraded data redundancy: 79/594 objects degraded (13.300%), 19 pgs degraded (PG_DEGRADED) 2024-03-11T20:58:40.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:40 smithi023 bash[29146]: cluster 2024-03-11T20:58:39.826415+0000 osd.2 (osd.2) 132 : cluster 0 6.f scrub starts 2024-03-11T20:58:40.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:40 smithi023 bash[29146]: cluster 2024-03-11T20:58:39.864127+0000 osd.2 (osd.2) 133 : cluster 0 6.f scrub ok 2024-03-11T20:58:40.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:40 smithi023 bash[21372]: cluster 2024-03-11T20:58:39.165028+0000 osd.4 (osd.4) 179 : cluster 0 4.c scrub starts 2024-03-11T20:58:40.536 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:40 smithi023 bash[21372]: cluster 2024-03-11T20:58:39.167628+0000 osd.4 (osd.4) 180 : cluster 0 4.c scrub ok 2024-03-11T20:58:40.536 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:40 smithi023 bash[21372]: cluster 2024-03-11T20:58:39.710900+0000 mon.a (mon.0) 1111 : cluster 3 Health check update: Degraded data redundancy: 79/594 objects degraded (13.300%), 19 pgs degraded (PG_DEGRADED) 2024-03-11T20:58:40.536 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:40 smithi023 bash[21372]: cluster 2024-03-11T20:58:39.826415+0000 osd.2 (osd.2) 132 : cluster 0 6.f scrub starts 2024-03-11T20:58:40.536 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:40 smithi023 bash[21372]: cluster 2024-03-11T20:58:39.864127+0000 osd.2 (osd.2) 133 : cluster 0 6.f scrub ok 2024-03-11T20:58:40.571 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:40.598 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:40.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:40.601 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:58:40.702 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:40.714 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:40 smithi146 bash[23809]: cluster 2024-03-11T20:58:39.165028+0000 osd.4 (osd.4) 179 : cluster 0 4.c scrub starts 2024-03-11T20:58:40.714 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:40 smithi146 bash[23809]: cluster 2024-03-11T20:58:39.167628+0000 osd.4 (osd.4) 180 : cluster 0 4.c scrub ok 2024-03-11T20:58:40.714 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:40 smithi146 bash[23809]: cluster 2024-03-11T20:58:39.710900+0000 mon.a (mon.0) 1111 : cluster 3 Health check update: Degraded data redundancy: 79/594 objects degraded (13.300%), 19 pgs degraded (PG_DEGRADED) 2024-03-11T20:58:40.714 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:40 smithi146 bash[23809]: cluster 2024-03-11T20:58:39.826415+0000 osd.2 (osd.2) 132 : cluster 0 6.f scrub starts 2024-03-11T20:58:40.714 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:40 smithi146 bash[23809]: cluster 2024-03-11T20:58:39.864127+0000 osd.2 (osd.2) 133 : cluster 0 6.f scrub ok 2024-03-11T20:58:40.730 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:40.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:40.732 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:58:40.833 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:40.862 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:40.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:40.864 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:58:40.943 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:58:40.944 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:58:40.965 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:40.986 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:40.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:40.989 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:58:41.089 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:41.116 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:41.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:41.119 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:58:41.219 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:41.246 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:41.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:41.248 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:58:41.349 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:58:41.376 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:58:41.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:41.378 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:58:41.479 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:41.489 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:41 smithi146 bash[23809]: cluster 2024-03-11T20:58:39.991929+0000 mgr.y (mgr.24380) 294 : cluster 0 pgmap v296: 148 pgs: 89 active+clean, 37 active+undersized, 19 active+undersized+degraded, 3 active+clean+remapped; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 552 B/s rd, 0 B/s wr, 225 op/s; 79/594 objects degraded (13.300%) 2024-03-11T20:58:41.489 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:41 smithi146 bash[23809]: cluster 2024-03-11T20:58:40.207445+0000 osd.7 (osd.7) 163 : cluster 0 3.12 scrub starts 2024-03-11T20:58:41.489 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:41 smithi146 bash[23809]: cluster 2024-03-11T20:58:40.211319+0000 osd.7 (osd.7) 164 : cluster 0 3.12 scrub ok 2024-03-11T20:58:41.489 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:41 smithi146 bash[23809]: cluster 2024-03-11T20:58:40.517318+0000 osd.3 (osd.3) 190 : cluster 0 7.f scrub starts 2024-03-11T20:58:41.489 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:41 smithi146 bash[23809]: cluster 2024-03-11T20:58:40.529715+0000 osd.3 (osd.3) 191 : cluster 0 7.f scrub ok 2024-03-11T20:58:41.502 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:41.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:41.504 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:58:41.605 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:41.630 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:41.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:41.633 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:58:41.734 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:41.745 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:41 smithi023 bash[21372]: cluster 2024-03-11T20:58:39.991929+0000 mgr.y (mgr.24380) 294 : cluster 0 pgmap v296: 148 pgs: 89 active+clean, 37 active+undersized, 19 active+undersized+degraded, 3 active+clean+remapped; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 552 B/s rd, 0 B/s wr, 225 op/s; 79/594 objects degraded (13.300%) 2024-03-11T20:58:41.745 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:41 smithi023 bash[21372]: cluster 2024-03-11T20:58:40.207445+0000 osd.7 (osd.7) 163 : cluster 0 3.12 scrub starts 2024-03-11T20:58:41.745 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:41 smithi023 bash[21372]: cluster 2024-03-11T20:58:40.211319+0000 osd.7 (osd.7) 164 : cluster 0 3.12 scrub ok 2024-03-11T20:58:41.746 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:41 smithi023 bash[21372]: cluster 2024-03-11T20:58:40.517318+0000 osd.3 (osd.3) 190 : cluster 0 7.f scrub starts 2024-03-11T20:58:41.746 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:41 smithi023 bash[21372]: cluster 2024-03-11T20:58:40.529715+0000 osd.3 (osd.3) 191 : cluster 0 7.f scrub ok 2024-03-11T20:58:41.746 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:41 smithi023 bash[29146]: cluster 2024-03-11T20:58:39.991929+0000 mgr.y (mgr.24380) 294 : cluster 0 pgmap v296: 148 pgs: 89 active+clean, 37 active+undersized, 19 active+undersized+degraded, 3 active+clean+remapped; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 552 B/s rd, 0 B/s wr, 225 op/s; 79/594 objects degraded (13.300%) 2024-03-11T20:58:41.746 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:41 smithi023 bash[29146]: cluster 2024-03-11T20:58:40.207445+0000 osd.7 (osd.7) 163 : cluster 0 3.12 scrub starts 2024-03-11T20:58:41.746 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:41 smithi023 bash[29146]: cluster 2024-03-11T20:58:40.211319+0000 osd.7 (osd.7) 164 : cluster 0 3.12 scrub ok 2024-03-11T20:58:41.746 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:41 smithi023 bash[29146]: cluster 2024-03-11T20:58:40.517318+0000 osd.3 (osd.3) 190 : cluster 0 7.f scrub starts 2024-03-11T20:58:41.746 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:41 smithi023 bash[29146]: cluster 2024-03-11T20:58:40.529715+0000 osd.3 (osd.3) 191 : cluster 0 7.f scrub ok 2024-03-11T20:58:41.761 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:41.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:41.764 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:58:41.865 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:41.892 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:41.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:41.894 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:58:41.968 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:58:41.995 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:42.022 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:42.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:42.024 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:58:42.125 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:42.150 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:42.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:42.152 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:58:42.253 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:42.281 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:42.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:42.283 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:58:42.384 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:42.410 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:42.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:42.413 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:58:42.513 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:42.525 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:42 smithi023 bash[21372]: cluster 2024-03-11T20:58:41.002545+0000 osd.5 (osd.5) 157 : cluster 0 4.3 scrub starts 2024-03-11T20:58:42.525 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:42 smithi023 bash[21372]: cluster 2024-03-11T20:58:41.004195+0000 osd.5 (osd.5) 158 : cluster 0 4.3 scrub ok 2024-03-11T20:58:42.526 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:42 smithi023 bash[21372]: cluster 2024-03-11T20:58:41.479624+0000 osd.3 (osd.3) 192 : cluster 0 3.b scrub starts 2024-03-11T20:58:42.526 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:42 smithi023 bash[21372]: cluster 2024-03-11T20:58:41.483063+0000 osd.3 (osd.3) 193 : cluster 0 3.b scrub ok 2024-03-11T20:58:42.526 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:42 smithi023 bash[21372]: cluster 2024-03-11T20:58:41.900337+0000 osd.6 (osd.6) 120 : cluster 0 2.17 scrub starts 2024-03-11T20:58:42.526 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:42 smithi023 bash[21372]: cluster 2024-03-11T20:58:41.901986+0000 osd.6 (osd.6) 121 : cluster 0 2.17 scrub ok 2024-03-11T20:58:42.526 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:42 smithi023 bash[29146]: cluster 2024-03-11T20:58:41.002545+0000 osd.5 (osd.5) 157 : cluster 0 4.3 scrub starts 2024-03-11T20:58:42.526 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:42 smithi023 bash[29146]: cluster 2024-03-11T20:58:41.004195+0000 osd.5 (osd.5) 158 : cluster 0 4.3 scrub ok 2024-03-11T20:58:42.526 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:42 smithi023 bash[29146]: cluster 2024-03-11T20:58:41.479624+0000 osd.3 (osd.3) 192 : cluster 0 3.b scrub starts 2024-03-11T20:58:42.526 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:42 smithi023 bash[29146]: cluster 2024-03-11T20:58:41.483063+0000 osd.3 (osd.3) 193 : cluster 0 3.b scrub ok 2024-03-11T20:58:42.526 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:42 smithi023 bash[29146]: cluster 2024-03-11T20:58:41.900337+0000 osd.6 (osd.6) 120 : cluster 0 2.17 scrub starts 2024-03-11T20:58:42.526 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:42 smithi023 bash[29146]: cluster 2024-03-11T20:58:41.901986+0000 osd.6 (osd.6) 121 : cluster 0 2.17 scrub ok 2024-03-11T20:58:42.541 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:42.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:42.543 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:58:42.644 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:58:42.674 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:58:42.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:42.676 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:58:42.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:42 smithi146 bash[23809]: cluster 2024-03-11T20:58:41.002545+0000 osd.5 (osd.5) 157 : cluster 0 4.3 scrub starts 2024-03-11T20:58:42.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:42 smithi146 bash[23809]: cluster 2024-03-11T20:58:41.004195+0000 osd.5 (osd.5) 158 : cluster 0 4.3 scrub ok 2024-03-11T20:58:42.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:42 smithi146 bash[23809]: cluster 2024-03-11T20:58:41.479624+0000 osd.3 (osd.3) 192 : cluster 0 3.b scrub starts 2024-03-11T20:58:42.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:42 smithi146 bash[23809]: cluster 2024-03-11T20:58:41.483063+0000 osd.3 (osd.3) 193 : cluster 0 3.b scrub ok 2024-03-11T20:58:42.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:42 smithi146 bash[23809]: cluster 2024-03-11T20:58:41.900337+0000 osd.6 (osd.6) 120 : cluster 0 2.17 scrub starts 2024-03-11T20:58:42.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:42 smithi146 bash[23809]: cluster 2024-03-11T20:58:41.901986+0000 osd.6 (osd.6) 121 : cluster 0 2.17 scrub ok 2024-03-11T20:58:42.777 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:42.803 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:42.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:42.806 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:58:42.906 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:58:42.934 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:58:42.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:42.937 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:58:42.966 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T20:58:43.037 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:58:43.057 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:58:43.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:43.059 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:58:43.160 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:43.187 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:43.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:43.189 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:58:43.290 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:43.318 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:43.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:43.320 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:58:43.341 INFO:teuthology.orchestra.run.smithi023.stdout:get_pg_num_history pg_num_history pg_num_history(e173 pg_nums {1={20=1},2={76=32},3={78=32},4={80=32},5={80=3},6={82=32},7={121=16}} deleted_pools ) 2024-03-11T20:58:43.341 INFO:teuthology.orchestra.run.smithi023.stdout:Importing pgid 3.19 2024-03-11T20:58:43.341 INFO:teuthology.orchestra.run.smithi023.stdout:Write #3:98c4c580:::obj_delete_at_hint.0000000100:head# 2024-03-11T20:58:43.341 INFO:teuthology.orchestra.run.smithi023.stdout:snapset 0=[]:{} 2024-03-11T20:58:43.341 INFO:teuthology.orchestra.run.smithi023.stdout:Write #3:9a6dd8c0:gc::gc.0:head# 2024-03-11T20:58:43.342 INFO:teuthology.orchestra.run.smithi023.stdout:snapset 0=[]:{} 2024-03-11T20:58:43.342 INFO:teuthology.orchestra.run.smithi023.stdout:Write #3:9c5cb460:::obj_delete_at_hint.0000000093:head# 2024-03-11T20:58:43.342 INFO:teuthology.orchestra.run.smithi023.stdout:snapset 0=[]:{} 2024-03-11T20:58:43.342 INFO:teuthology.orchestra.run.smithi023.stdout:Write #3:9ed7f5b7:::obj_delete_at_hint.0000000004:head# 2024-03-11T20:58:43.343 INFO:teuthology.orchestra.run.smithi023.stdout:snapset 0=[]:{} 2024-03-11T20:58:43.343 INFO:teuthology.orchestra.run.smithi023.stdout:Write #3:9edc23b1:::obj_delete_at_hint.0000000063:head# 2024-03-11T20:58:43.343 INFO:teuthology.orchestra.run.smithi023.stdout:snapset 0=[]:{} 2024-03-11T20:58:43.343 INFO:teuthology.orchestra.run.smithi023.stdout:Write #3:9f21944d:::obj_delete_at_hint.0000000122:head# 2024-03-11T20:58:43.343 INFO:teuthology.orchestra.run.smithi023.stdout:snapset 0=[]:{} 2024-03-11T20:58:43.344 INFO:teuthology.orchestra.run.smithi023.stdout:Write #3:9fecf831:::obj_delete_at_hint.0000000057:head# 2024-03-11T20:58:43.344 INFO:teuthology.orchestra.run.smithi023.stdout:snapset 0=[]:{} 2024-03-11T20:58:43.344 INFO:teuthology.orchestra.run.smithi023.stdout:write_pg epoch 163 info 3.19( v 84'15 (0'0,84'15] local-lis/les=162/163 n=7 ec=78/78 lis/c=162/162 les/c/f=163/163/0 sis=162) 2024-03-11T20:58:43.344 INFO:teuthology.orchestra.run.smithi023.stdout:Import successful 2024-03-11T20:58:43.421 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:43.451 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:43.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:43.454 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:58:43.554 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:43.583 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:43.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:43.585 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:58:43.686 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:58:43.714 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:58:43.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:43.716 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:58:43.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:43 smithi146 bash[23809]: cluster 2024-03-11T20:58:41.992712+0000 mgr.y (mgr.24380) 295 : cluster 0 pgmap v297: 148 pgs: 89 active+clean, 37 active+undersized, 19 active+undersized+degraded, 3 active+clean+remapped; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 209 op/s; 79/594 objects degraded (13.300%) 2024-03-11T20:58:43.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:43 smithi146 bash[23809]: cluster 2024-03-11T20:58:42.001075+0000 osd.5 (osd.5) 159 : cluster 0 4.15 scrub starts 2024-03-11T20:58:43.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:43 smithi146 bash[23809]: cluster 2024-03-11T20:58:42.003695+0000 osd.5 (osd.5) 160 : cluster 0 4.15 scrub ok 2024-03-11T20:58:43.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:43 smithi023 bash[29146]: cluster 2024-03-11T20:58:41.992712+0000 mgr.y (mgr.24380) 295 : cluster 0 pgmap v297: 148 pgs: 89 active+clean, 37 active+undersized, 19 active+undersized+degraded, 3 active+clean+remapped; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 209 op/s; 79/594 objects degraded (13.300%) 2024-03-11T20:58:43.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:43 smithi023 bash[29146]: cluster 2024-03-11T20:58:42.001075+0000 osd.5 (osd.5) 159 : cluster 0 4.15 scrub starts 2024-03-11T20:58:43.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:43 smithi023 bash[29146]: cluster 2024-03-11T20:58:42.003695+0000 osd.5 (osd.5) 160 : cluster 0 4.15 scrub ok 2024-03-11T20:58:43.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:43 smithi023 bash[21372]: cluster 2024-03-11T20:58:41.992712+0000 mgr.y (mgr.24380) 295 : cluster 0 pgmap v297: 148 pgs: 89 active+clean, 37 active+undersized, 19 active+undersized+degraded, 3 active+clean+remapped; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 209 op/s; 79/594 objects degraded (13.300%) 2024-03-11T20:58:43.786 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:43 smithi023 bash[21372]: cluster 2024-03-11T20:58:42.001075+0000 osd.5 (osd.5) 159 : cluster 0 4.15 scrub starts 2024-03-11T20:58:43.786 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:43 smithi023 bash[21372]: cluster 2024-03-11T20:58:42.003695+0000 osd.5 (osd.5) 160 : cluster 0 4.15 scrub ok 2024-03-11T20:58:43.817 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:43.842 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:43.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:43.845 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:58:43.946 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:43.972 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:43.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:43.975 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:58:44.075 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:44.098 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:44.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:44.100 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:58:44.201 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:58:44.222 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:58:44.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:44.224 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:58:44.325 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:44.347 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:44.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:44.349 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:58:44.449 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:44.474 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:44.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:44.476 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:58:44.577 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:58:44.600 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:58:44.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:44.602 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:58:44.703 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:44.715 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:44 smithi023 bash[21372]: cluster 2024-03-11T20:58:43.045490+0000 osd.5 (osd.5) 161 : cluster 0 2.18 scrub starts 2024-03-11T20:58:44.715 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:44 smithi023 bash[21372]: cluster 2024-03-11T20:58:43.047013+0000 osd.5 (osd.5) 162 : cluster 0 2.18 scrub ok 2024-03-11T20:58:44.715 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:44 smithi023 bash[21372]: cluster 2024-03-11T20:58:43.131445+0000 osd.4 (osd.4) 181 : cluster 0 4.1a scrub starts 2024-03-11T20:58:44.715 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:44 smithi023 bash[21372]: cluster 2024-03-11T20:58:43.133738+0000 osd.4 (osd.4) 182 : cluster 0 4.1a scrub ok 2024-03-11T20:58:44.716 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:44 smithi023 bash[21372]: cluster 2024-03-11T20:58:43.886762+0000 osd.6 (osd.6) 122 : cluster 0 4.5 scrub starts 2024-03-11T20:58:44.716 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:44 smithi023 bash[21372]: cluster 2024-03-11T20:58:43.888169+0000 osd.6 (osd.6) 123 : cluster 0 4.5 scrub ok 2024-03-11T20:58:44.716 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:44 smithi023 bash[29146]: cluster 2024-03-11T20:58:43.045490+0000 osd.5 (osd.5) 161 : cluster 0 2.18 scrub starts 2024-03-11T20:58:44.716 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:44 smithi023 bash[29146]: cluster 2024-03-11T20:58:43.047013+0000 osd.5 (osd.5) 162 : cluster 0 2.18 scrub ok 2024-03-11T20:58:44.717 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:44 smithi023 bash[29146]: cluster 2024-03-11T20:58:43.131445+0000 osd.4 (osd.4) 181 : cluster 0 4.1a scrub starts 2024-03-11T20:58:44.717 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:44 smithi023 bash[29146]: cluster 2024-03-11T20:58:43.133738+0000 osd.4 (osd.4) 182 : cluster 0 4.1a scrub ok 2024-03-11T20:58:44.717 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:44 smithi023 bash[29146]: cluster 2024-03-11T20:58:43.886762+0000 osd.6 (osd.6) 122 : cluster 0 4.5 scrub starts 2024-03-11T20:58:44.717 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:44 smithi023 bash[29146]: cluster 2024-03-11T20:58:43.888169+0000 osd.6 (osd.6) 123 : cluster 0 4.5 scrub ok 2024-03-11T20:58:44.730 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:44.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:44.733 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:58:44.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:44 smithi146 bash[23809]: cluster 2024-03-11T20:58:43.045490+0000 osd.5 (osd.5) 161 : cluster 0 2.18 scrub starts 2024-03-11T20:58:44.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:44 smithi146 bash[23809]: cluster 2024-03-11T20:58:43.047013+0000 osd.5 (osd.5) 162 : cluster 0 2.18 scrub ok 2024-03-11T20:58:44.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:44 smithi146 bash[23809]: cluster 2024-03-11T20:58:43.131445+0000 osd.4 (osd.4) 181 : cluster 0 4.1a scrub starts 2024-03-11T20:58:44.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:44 smithi146 bash[23809]: cluster 2024-03-11T20:58:43.133738+0000 osd.4 (osd.4) 182 : cluster 0 4.1a scrub ok 2024-03-11T20:58:44.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:44 smithi146 bash[23809]: cluster 2024-03-11T20:58:43.886762+0000 osd.6 (osd.6) 122 : cluster 0 4.5 scrub starts 2024-03-11T20:58:44.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:44 smithi146 bash[23809]: cluster 2024-03-11T20:58:43.888169+0000 osd.6 (osd.6) 123 : cluster 0 4.5 scrub ok 2024-03-11T20:58:44.834 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:44.861 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:44.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:44.863 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:58:44.964 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:44.986 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:44.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:44.988 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:58:45.089 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:45.118 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:45.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:45.121 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:58:45.221 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:45.247 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:45.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:45.249 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:58:45.350 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:45.375 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:45.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:45.377 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:58:45.478 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:45.500 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 20:58:45 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T20:58:45.507 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:45.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:45.510 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:58:45.611 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:45.637 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:45.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:45.640 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:58:45.740 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:45.751 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:45 smithi146 bash[23809]: cluster 2024-03-11T20:58:43.993460+0000 mgr.y (mgr.24380) 296 : cluster 0 pgmap v298: 148 pgs: 89 active+clean, 37 active+undersized, 19 active+undersized+degraded, 3 active+clean+remapped; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 209 op/s; 79/594 objects degraded (13.300%) 2024-03-11T20:58:45.752 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:45 smithi146 bash[23809]: cluster 2024-03-11T20:58:44.014753+0000 osd.5 (osd.5) 163 : cluster 0 2.3 scrub starts 2024-03-11T20:58:45.752 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:45 smithi146 bash[23809]: cluster 2024-03-11T20:58:44.016253+0000 osd.5 (osd.5) 164 : cluster 0 2.3 scrub ok 2024-03-11T20:58:45.752 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:45 smithi146 bash[23809]: cluster 2024-03-11T20:58:44.144430+0000 osd.4 (osd.4) 183 : cluster 0 3.1 scrub starts 2024-03-11T20:58:45.752 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:45 smithi146 bash[23809]: cluster 2024-03-11T20:58:44.148348+0000 osd.4 (osd.4) 184 : cluster 0 3.1 scrub ok 2024-03-11T20:58:45.752 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:45 smithi146 bash[23809]: cluster 2024-03-11T20:58:44.478141+0000 osd.3 (osd.3) 194 : cluster 0 6.13 scrub starts 2024-03-11T20:58:45.752 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:45 smithi146 bash[23809]: cluster 2024-03-11T20:58:44.496950+0000 osd.3 (osd.3) 195 : cluster 0 6.13 scrub ok 2024-03-11T20:58:45.767 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:45.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:45.770 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:58:45.809 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:45 smithi023 bash[21372]: cluster 2024-03-11T20:58:43.993460+0000 mgr.y (mgr.24380) 296 : cluster 0 pgmap v298: 148 pgs: 89 active+clean, 37 active+undersized, 19 active+undersized+degraded, 3 active+clean+remapped; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 209 op/s; 79/594 objects degraded (13.300%) 2024-03-11T20:58:45.809 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:45 smithi023 bash[21372]: cluster 2024-03-11T20:58:44.014753+0000 osd.5 (osd.5) 163 : cluster 0 2.3 scrub starts 2024-03-11T20:58:45.809 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:45 smithi023 bash[21372]: cluster 2024-03-11T20:58:44.016253+0000 osd.5 (osd.5) 164 : cluster 0 2.3 scrub ok 2024-03-11T20:58:45.809 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:45 smithi023 bash[21372]: cluster 2024-03-11T20:58:44.144430+0000 osd.4 (osd.4) 183 : cluster 0 3.1 scrub starts 2024-03-11T20:58:45.809 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:45 smithi023 bash[21372]: cluster 2024-03-11T20:58:44.148348+0000 osd.4 (osd.4) 184 : cluster 0 3.1 scrub ok 2024-03-11T20:58:45.809 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:45 smithi023 bash[21372]: cluster 2024-03-11T20:58:44.478141+0000 osd.3 (osd.3) 194 : cluster 0 6.13 scrub starts 2024-03-11T20:58:45.809 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:45 smithi023 bash[21372]: cluster 2024-03-11T20:58:44.496950+0000 osd.3 (osd.3) 195 : cluster 0 6.13 scrub ok 2024-03-11T20:58:45.810 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:45 smithi023 bash[29146]: cluster 2024-03-11T20:58:43.993460+0000 mgr.y (mgr.24380) 296 : cluster 0 pgmap v298: 148 pgs: 89 active+clean, 37 active+undersized, 19 active+undersized+degraded, 3 active+clean+remapped; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 209 op/s; 79/594 objects degraded (13.300%) 2024-03-11T20:58:45.810 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:45 smithi023 bash[29146]: cluster 2024-03-11T20:58:44.014753+0000 osd.5 (osd.5) 163 : cluster 0 2.3 scrub starts 2024-03-11T20:58:45.810 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:45 smithi023 bash[29146]: cluster 2024-03-11T20:58:44.016253+0000 osd.5 (osd.5) 164 : cluster 0 2.3 scrub ok 2024-03-11T20:58:45.810 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:45 smithi023 bash[29146]: cluster 2024-03-11T20:58:44.144430+0000 osd.4 (osd.4) 183 : cluster 0 3.1 scrub starts 2024-03-11T20:58:45.810 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:45 smithi023 bash[29146]: cluster 2024-03-11T20:58:44.148348+0000 osd.4 (osd.4) 184 : cluster 0 3.1 scrub ok 2024-03-11T20:58:45.810 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:45 smithi023 bash[29146]: cluster 2024-03-11T20:58:44.478141+0000 osd.3 (osd.3) 194 : cluster 0 6.13 scrub starts 2024-03-11T20:58:45.810 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:45 smithi023 bash[29146]: cluster 2024-03-11T20:58:44.496950+0000 osd.3 (osd.3) 195 : cluster 0 6.13 scrub ok 2024-03-11T20:58:45.871 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:45.898 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:45.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:45.900 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:58:46.001 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:58:46.030 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:58:46.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:46.032 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:58:46.134 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:46.159 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:46.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:46.161 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:58:46.262 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:46.291 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:46.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:46.293 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:58:46.394 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:46.416 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:46.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:46.420 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:58:46.520 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:46.540 DEBUG:teuthology.orchestra.run.smithi023:> sudo rm -f /var/log/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/exp.3.19.1 2024-03-11T20:58:46.545 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-03-11T20:58:46.546 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T20:58:46.547 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:46.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:46.550 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:58:46.558 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:46 smithi146 bash[23809]: cluster 2024-03-11T20:58:45.103385+0000 osd.4 (osd.4) 185 : cluster 0 7.9 scrub starts 2024-03-11T20:58:46.558 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:46 smithi146 bash[23809]: cluster 2024-03-11T20:58:45.126008+0000 osd.4 (osd.4) 186 : cluster 0 7.9 scrub ok 2024-03-11T20:58:46.558 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:46 smithi146 bash[23809]: audit 2024-03-11T20:58:45.196780+0000 mgr.y (mgr.24380) 297 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:58:46.558 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:46 smithi146 bash[23809]: cluster 2024-03-11T20:58:45.484354+0000 osd.3 (osd.3) 196 : cluster 0 6.1f scrub starts 2024-03-11T20:58:46.559 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:46 smithi146 bash[23809]: cluster 2024-03-11T20:58:45.526709+0000 osd.3 (osd.3) 197 : cluster 0 6.1f scrub ok 2024-03-11T20:58:46.651 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:46.677 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:46.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:46.680 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:58:46.781 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:58:46.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:46 smithi023 bash[21372]: cluster 2024-03-11T20:58:45.103385+0000 osd.4 (osd.4) 185 : cluster 0 7.9 scrub starts 2024-03-11T20:58:46.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:46 smithi023 bash[21372]: cluster 2024-03-11T20:58:45.126008+0000 osd.4 (osd.4) 186 : cluster 0 7.9 scrub ok 2024-03-11T20:58:46.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:46 smithi023 bash[21372]: audit 2024-03-11T20:58:45.196780+0000 mgr.y (mgr.24380) 297 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:58:46.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:46 smithi023 bash[21372]: cluster 2024-03-11T20:58:45.484354+0000 osd.3 (osd.3) 196 : cluster 0 6.1f scrub starts 2024-03-11T20:58:46.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:46 smithi023 bash[21372]: cluster 2024-03-11T20:58:45.526709+0000 osd.3 (osd.3) 197 : cluster 0 6.1f scrub ok 2024-03-11T20:58:46.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:46 smithi023 bash[29146]: cluster 2024-03-11T20:58:45.103385+0000 osd.4 (osd.4) 185 : cluster 0 7.9 scrub starts 2024-03-11T20:58:46.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:46 smithi023 bash[29146]: cluster 2024-03-11T20:58:45.126008+0000 osd.4 (osd.4) 186 : cluster 0 7.9 scrub ok 2024-03-11T20:58:46.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:46 smithi023 bash[29146]: audit 2024-03-11T20:58:45.196780+0000 mgr.y (mgr.24380) 297 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:58:46.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:46 smithi023 bash[29146]: cluster 2024-03-11T20:58:45.484354+0000 osd.3 (osd.3) 196 : cluster 0 6.1f scrub starts 2024-03-11T20:58:46.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:46 smithi023 bash[29146]: cluster 2024-03-11T20:58:45.526709+0000 osd.3 (osd.3) 197 : cluster 0 6.1f scrub ok 2024-03-11T20:58:46.807 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:58:46.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:46.810 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:58:46.910 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:46.938 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:46.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:46.940 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:58:47.041 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:47.068 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:47.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:47.070 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:58:47.171 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:47.199 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:47.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:47.201 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:58:47.302 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:47.330 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:47.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:47.332 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:58:47.433 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:47.458 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:47.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:47.460 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:58:47.470 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-11T20:58:47.562 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:47.590 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:47.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:47.592 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:58:47.693 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:47.701 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:47 smithi146 bash[23809]: cluster 2024-03-11T20:58:45.995379+0000 mgr.y (mgr.24380) 298 : cluster 0 pgmap v299: 148 pgs: 89 active+clean, 37 active+undersized, 19 active+undersized+degraded, 3 active+clean+remapped; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 79/594 objects degraded (13.300%) 2024-03-11T20:58:47.701 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:47 smithi146 bash[23809]: cluster 2024-03-11T20:58:46.016374+0000 osd.5 (osd.5) 165 : cluster 0 3.5 scrub starts 2024-03-11T20:58:47.701 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:47 smithi146 bash[23809]: cluster 2024-03-11T20:58:46.022933+0000 osd.5 (osd.5) 166 : cluster 0 3.5 scrub ok 2024-03-11T20:58:47.701 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:47 smithi146 bash[23809]: cluster 2024-03-11T20:58:46.532312+0000 osd.3 (osd.3) 198 : cluster 0 2.1b scrub starts 2024-03-11T20:58:47.701 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:47 smithi146 bash[23809]: cluster 2024-03-11T20:58:46.534981+0000 osd.3 (osd.3) 199 : cluster 0 2.1b scrub ok 2024-03-11T20:58:47.701 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:47 smithi146 bash[23809]: cluster 2024-03-11T20:58:46.876733+0000 osd.2 (osd.2) 134 : cluster 0 7.e scrub starts 2024-03-11T20:58:47.702 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:47 smithi146 bash[23809]: cluster 2024-03-11T20:58:46.903438+0000 osd.2 (osd.2) 135 : cluster 0 7.e scrub ok 2024-03-11T20:58:47.715 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:47.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:47.718 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:58:47.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:47 smithi023 bash[21372]: cluster 2024-03-11T20:58:45.995379+0000 mgr.y (mgr.24380) 298 : cluster 0 pgmap v299: 148 pgs: 89 active+clean, 37 active+undersized, 19 active+undersized+degraded, 3 active+clean+remapped; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 79/594 objects degraded (13.300%) 2024-03-11T20:58:47.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:47 smithi023 bash[21372]: cluster 2024-03-11T20:58:46.016374+0000 osd.5 (osd.5) 165 : cluster 0 3.5 scrub starts 2024-03-11T20:58:47.786 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:47 smithi023 bash[21372]: cluster 2024-03-11T20:58:46.022933+0000 osd.5 (osd.5) 166 : cluster 0 3.5 scrub ok 2024-03-11T20:58:47.786 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:47 smithi023 bash[21372]: cluster 2024-03-11T20:58:46.532312+0000 osd.3 (osd.3) 198 : cluster 0 2.1b scrub starts 2024-03-11T20:58:47.786 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:47 smithi023 bash[21372]: cluster 2024-03-11T20:58:46.534981+0000 osd.3 (osd.3) 199 : cluster 0 2.1b scrub ok 2024-03-11T20:58:47.786 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:47 smithi023 bash[21372]: cluster 2024-03-11T20:58:46.876733+0000 osd.2 (osd.2) 134 : cluster 0 7.e scrub starts 2024-03-11T20:58:47.786 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:47 smithi023 bash[21372]: cluster 2024-03-11T20:58:46.903438+0000 osd.2 (osd.2) 135 : cluster 0 7.e scrub ok 2024-03-11T20:58:47.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:47 smithi023 bash[29146]: cluster 2024-03-11T20:58:45.995379+0000 mgr.y (mgr.24380) 298 : cluster 0 pgmap v299: 148 pgs: 89 active+clean, 37 active+undersized, 19 active+undersized+degraded, 3 active+clean+remapped; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 79/594 objects degraded (13.300%) 2024-03-11T20:58:47.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:47 smithi023 bash[29146]: cluster 2024-03-11T20:58:46.016374+0000 osd.5 (osd.5) 165 : cluster 0 3.5 scrub starts 2024-03-11T20:58:47.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:47 smithi023 bash[29146]: cluster 2024-03-11T20:58:46.022933+0000 osd.5 (osd.5) 166 : cluster 0 3.5 scrub ok 2024-03-11T20:58:47.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:47 smithi023 bash[29146]: cluster 2024-03-11T20:58:46.532312+0000 osd.3 (osd.3) 198 : cluster 0 2.1b scrub starts 2024-03-11T20:58:47.787 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:47 smithi023 bash[29146]: cluster 2024-03-11T20:58:46.534981+0000 osd.3 (osd.3) 199 : cluster 0 2.1b scrub ok 2024-03-11T20:58:47.787 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:47 smithi023 bash[29146]: cluster 2024-03-11T20:58:46.876733+0000 osd.2 (osd.2) 134 : cluster 0 7.e scrub starts 2024-03-11T20:58:47.787 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:47 smithi023 bash[29146]: cluster 2024-03-11T20:58:46.903438+0000 osd.2 (osd.2) 135 : cluster 0 7.e scrub ok 2024-03-11T20:58:47.818 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:58:47.845 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:58:47.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:47.847 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:58:47.948 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:47.974 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:47.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:47.976 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:58:48.077 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:48.108 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:48.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:48.111 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:58:48.212 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:48.235 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:48.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:48.238 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:58:48.339 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:48.366 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:48.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:48.369 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:58:48.470 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:58:48.498 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:58:48.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:48.500 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:58:48.601 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:48.612 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:48 smithi146 bash[23809]: cluster 2024-03-11T20:58:47.866699+0000 osd.2 (osd.2) 136 : cluster 0 6.14 scrub starts 2024-03-11T20:58:48.613 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:48 smithi146 bash[23809]: cluster 2024-03-11T20:58:47.869121+0000 osd.2 (osd.2) 137 : cluster 0 6.14 scrub ok 2024-03-11T20:58:48.613 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:48 smithi146 bash[23809]: cluster 2024-03-11T20:58:47.996369+0000 mgr.y (mgr.24380) 299 : cluster 0 pgmap v300: 148 pgs: 89 active+clean, 37 active+undersized, 19 active+undersized+degraded, 3 active+clean+remapped; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 968 B/s rd, 0 op/s; 79/594 objects degraded (13.300%) 2024-03-11T20:58:48.629 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:48.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:48.632 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:58:48.733 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:48.744 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:48 smithi023 bash[21372]: cluster 2024-03-11T20:58:47.866699+0000 osd.2 (osd.2) 136 : cluster 0 6.14 scrub starts 2024-03-11T20:58:48.744 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:48 smithi023 bash[21372]: cluster 2024-03-11T20:58:47.869121+0000 osd.2 (osd.2) 137 : cluster 0 6.14 scrub ok 2024-03-11T20:58:48.744 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:48 smithi023 bash[21372]: cluster 2024-03-11T20:58:47.996369+0000 mgr.y (mgr.24380) 299 : cluster 0 pgmap v300: 148 pgs: 89 active+clean, 37 active+undersized, 19 active+undersized+degraded, 3 active+clean+remapped; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 968 B/s rd, 0 op/s; 79/594 objects degraded (13.300%) 2024-03-11T20:58:48.745 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:58:48 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:20:58:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T20:58:48.745 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:48 smithi023 bash[29146]: cluster 2024-03-11T20:58:47.866699+0000 osd.2 (osd.2) 136 : cluster 0 6.14 scrub starts 2024-03-11T20:58:48.745 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:48 smithi023 bash[29146]: cluster 2024-03-11T20:58:47.869121+0000 osd.2 (osd.2) 137 : cluster 0 6.14 scrub ok 2024-03-11T20:58:48.745 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:48 smithi023 bash[29146]: cluster 2024-03-11T20:58:47.996369+0000 mgr.y (mgr.24380) 299 : cluster 0 pgmap v300: 148 pgs: 89 active+clean, 37 active+undersized, 19 active+undersized+degraded, 3 active+clean+remapped; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 968 B/s rd, 0 op/s; 79/594 objects degraded (13.300%) 2024-03-11T20:58:48.760 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:48.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:48.763 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:58:48.866 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:58:48.894 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:58:48.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:48.896 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:58:48.997 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:49.024 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:49.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:49.027 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:58:49.128 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:49.153 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:49.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:49.156 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:58:49.256 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:49.283 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:49.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:49.286 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:58:49.387 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:58:49.406 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:58:49.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:49.408 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:58:49.509 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:49.532 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:49.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:49.534 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:58:49.635 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:58:49.662 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:58:49.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:49.664 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:58:49.765 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:49.788 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:49.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:49.790 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:58:49.891 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:49.904 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:49 smithi023 bash[29146]: cluster 2024-03-11T20:58:47.978302+0000 osd.5 (osd.5) 167 : cluster 0 6.7 scrub starts 2024-03-11T20:58:49.904 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:49 smithi023 bash[29146]: cluster 2024-03-11T20:58:48.005481+0000 osd.5 (osd.5) 168 : cluster 0 6.7 scrub ok 2024-03-11T20:58:49.904 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:49 smithi023 bash[29146]: audit 2024-03-11T20:58:48.990530+0000 mon.a (mon.0) 1112 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:58:49.904 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:49 smithi023 bash[21372]: cluster 2024-03-11T20:58:47.978302+0000 osd.5 (osd.5) 167 : cluster 0 6.7 scrub starts 2024-03-11T20:58:49.904 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:49 smithi023 bash[21372]: cluster 2024-03-11T20:58:48.005481+0000 osd.5 (osd.5) 168 : cluster 0 6.7 scrub ok 2024-03-11T20:58:49.904 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:49 smithi023 bash[21372]: audit 2024-03-11T20:58:48.990530+0000 mon.a (mon.0) 1112 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:58:49.921 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:49.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:49.924 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:58:50.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:49 smithi146 bash[23809]: cluster 2024-03-11T20:58:47.978302+0000 osd.5 (osd.5) 167 : cluster 0 6.7 scrub starts 2024-03-11T20:58:50.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:49 smithi146 bash[23809]: cluster 2024-03-11T20:58:48.005481+0000 osd.5 (osd.5) 168 : cluster 0 6.7 scrub ok 2024-03-11T20:58:50.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:49 smithi146 bash[23809]: audit 2024-03-11T20:58:48.990530+0000 mon.a (mon.0) 1112 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:58:50.025 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:50.051 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:50.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:50.054 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:58:50.115 INFO:teuthology.orchestra.run.smithi023.stdout:osd.0: {} 2024-03-11T20:58:50.136 INFO:teuthology.orchestra.run.smithi023.stdout:osd.2: {} 2024-03-11T20:58:50.154 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:58:50.157 INFO:teuthology.orchestra.run.smithi023.stdout:osd.3: {} 2024-03-11T20:58:50.177 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:58:50.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:50.180 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:58:50.189 INFO:teuthology.orchestra.run.smithi023.stdout:osd.4: {} 2024-03-11T20:58:50.215 INFO:teuthology.orchestra.run.smithi023.stdout:osd.5: {} 2024-03-11T20:58:50.243 INFO:teuthology.orchestra.run.smithi023.stdout:osd.6: {} 2024-03-11T20:58:50.263 INFO:teuthology.orchestra.run.smithi023.stdout:osd.7: {} 2024-03-11T20:58:50.263 INFO:teuthology.orchestra.run.smithi023.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T20:58:50.263 INFO:teuthology.orchestra.run.smithi023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-11T20:58:50.264 INFO:teuthology.orchestra.run.smithi023.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T20:58:50.264 INFO:teuthology.orchestra.run.smithi023.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T20:58:50.264 INFO:teuthology.orchestra.run.smithi023.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T20:58:50.264 INFO:teuthology.orchestra.run.smithi023.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T20:58:50.264 INFO:teuthology.orchestra.run.smithi023.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T20:58:50.264 INFO:teuthology.orchestra.run.smithi023.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T20:58:50.281 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:50.302 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:50.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:50.304 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:58:50.405 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:50.431 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:50.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:50.433 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:58:50.534 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:50.559 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:50.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:50.561 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:58:50.662 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:50.690 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:50.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:50.693 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:58:50.793 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:58:50.801 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:50 smithi023 bash[21372]: audit 2024-03-11T20:58:49.673467+0000 mon.a (mon.0) 1113 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:58:50.802 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:50 smithi023 bash[21372]: audit 2024-03-11T20:58:49.675349+0000 mon.a (mon.0) 1114 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:58:50.802 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:50 smithi023 bash[21372]: audit 2024-03-11T20:58:49.677659+0000 mon.a (mon.0) 1115 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:58:50.802 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:50 smithi023 bash[21372]: audit 2024-03-11T20:58:49.697712+0000 mon.a (mon.0) 1116 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:58:50.802 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:50 smithi023 bash[21372]: cluster 2024-03-11T20:58:49.997269+0000 mgr.y (mgr.24380) 300 : cluster 0 pgmap v301: 148 pgs: 89 active+clean, 37 active+undersized, 19 active+undersized+degraded, 3 active+clean+remapped; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 79/594 objects degraded (13.300%) 2024-03-11T20:58:50.802 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:50 smithi023 bash[21372]: audit 2024-03-11T20:58:50.095301+0000 mon.c (mon.2) 39 : audit 0 from='client.? 172.21.15.23:0/3107661336' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T20:58:50.802 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:50 smithi023 bash[29146]: audit 2024-03-11T20:58:49.673467+0000 mon.a (mon.0) 1113 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:58:50.802 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:50 smithi023 bash[29146]: audit 2024-03-11T20:58:49.675349+0000 mon.a (mon.0) 1114 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:58:50.802 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:50 smithi023 bash[29146]: audit 2024-03-11T20:58:49.677659+0000 mon.a (mon.0) 1115 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:58:50.802 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:50 smithi023 bash[29146]: audit 2024-03-11T20:58:49.697712+0000 mon.a (mon.0) 1116 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:58:50.802 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:50 smithi023 bash[29146]: cluster 2024-03-11T20:58:49.997269+0000 mgr.y (mgr.24380) 300 : cluster 0 pgmap v301: 148 pgs: 89 active+clean, 37 active+undersized, 19 active+undersized+degraded, 3 active+clean+remapped; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 79/594 objects degraded (13.300%) 2024-03-11T20:58:50.802 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:50 smithi023 bash[29146]: audit 2024-03-11T20:58:50.095301+0000 mon.c (mon.2) 39 : audit 0 from='client.? 172.21.15.23:0/3107661336' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T20:58:50.812 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:58:50.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:50.815 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:58:50.916 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:58:50.940 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:58:50.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:50.943 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:58:51.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:50 smithi146 bash[23809]: audit 2024-03-11T20:58:49.673467+0000 mon.a (mon.0) 1113 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:58:51.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:50 smithi146 bash[23809]: audit 2024-03-11T20:58:49.675349+0000 mon.a (mon.0) 1114 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:58:51.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:50 smithi146 bash[23809]: audit 2024-03-11T20:58:49.677659+0000 mon.a (mon.0) 1115 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:58:51.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:50 smithi146 bash[23809]: audit 2024-03-11T20:58:49.697712+0000 mon.a (mon.0) 1116 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:58:51.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:50 smithi146 bash[23809]: cluster 2024-03-11T20:58:49.997269+0000 mgr.y (mgr.24380) 300 : cluster 0 pgmap v301: 148 pgs: 89 active+clean, 37 active+undersized, 19 active+undersized+degraded, 3 active+clean+remapped; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 79/594 objects degraded (13.300%) 2024-03-11T20:58:51.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:50 smithi146 bash[23809]: audit 2024-03-11T20:58:50.095301+0000 mon.c (mon.2) 39 : audit 0 from='client.? 172.21.15.23:0/3107661336' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T20:58:51.043 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:51.070 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:51.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:51.072 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:58:51.173 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:51.200 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T20:58:51.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:51.202 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T20:58:51.303 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:51.329 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:51.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:51.331 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:58:51.432 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:51.456 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:51.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:51.458 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:58:51.504 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-11T20:58:51.504 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-11T20:58:51.558 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 6, 7, 1] out_osds: [] dead_osds: [1] live_osds: [0, 2, 3, 4, 5, 6, 7] 2024-03-11T20:58:51.558 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-11T20:58:51.558 INFO:tasks.thrashosds.thrasher:Killing osd 4, live_osds are [0, 2, 3, 4, 5, 6, 7] 2024-03-11T20:58:51.558 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-03-11T20:58:51.558 DEBUG:teuthology.orchestra.run.smithi146:> sudo systemctl stop ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@osd.4 2024-03-11T20:58:51.559 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:51.570 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:51 smithi146 bash[23809]: audit 2024-03-11T20:58:50.655106+0000 mon.a (mon.0) 1117 : audit 1 from='client.? 172.21.15.23:0/1502532390' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T20:58:51.570 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:51 smithi146 bash[23809]: cluster 2024-03-11T20:58:50.922261+0000 osd.2 (osd.2) 138 : cluster 0 2.1 scrub starts 2024-03-11T20:58:51.570 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:51 smithi146 bash[23809]: cluster 2024-03-11T20:58:50.924621+0000 osd.2 (osd.2) 139 : cluster 0 2.1 scrub ok 2024-03-11T20:58:51.570 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:51 smithi146 bash[23809]: audit 2024-03-11T20:58:51.006628+0000 mon.c (mon.2) 40 : audit 1 from='client.? 172.21.15.23:0/1016538621' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T20:58:51.571 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:51 smithi146 bash[23809]: audit 2024-03-11T20:58:51.007022+0000 mon.a (mon.0) 1118 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T20:58:51.579 INFO:teuthology.orchestra.run.smithi023.stderr:nodeep-scrub is unset 2024-03-11T20:58:51.588 INFO:teuthology.orchestra.run.smithi023.stderr:pool 'unique_pool_0' removed 2024-03-11T20:58:51.595 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:51.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:51.600 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:58:51.701 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:51.719 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:51.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:51.721 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:58:51.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:51 smithi023 bash[29146]: audit 2024-03-11T20:58:50.655106+0000 mon.a (mon.0) 1117 : audit 1 from='client.? 172.21.15.23:0/1502532390' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T20:58:51.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:51 smithi023 bash[29146]: cluster 2024-03-11T20:58:50.922261+0000 osd.2 (osd.2) 138 : cluster 0 2.1 scrub starts 2024-03-11T20:58:51.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:51 smithi023 bash[29146]: cluster 2024-03-11T20:58:50.924621+0000 osd.2 (osd.2) 139 : cluster 0 2.1 scrub ok 2024-03-11T20:58:51.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:51 smithi023 bash[29146]: audit 2024-03-11T20:58:51.006628+0000 mon.c (mon.2) 40 : audit 1 from='client.? 172.21.15.23:0/1016538621' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T20:58:51.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:51 smithi023 bash[29146]: audit 2024-03-11T20:58:51.007022+0000 mon.a (mon.0) 1118 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T20:58:51.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:51 smithi023 bash[21372]: audit 2024-03-11T20:58:50.655106+0000 mon.a (mon.0) 1117 : audit 1 from='client.? 172.21.15.23:0/1502532390' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T20:58:51.786 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:51 smithi023 bash[21372]: cluster 2024-03-11T20:58:50.922261+0000 osd.2 (osd.2) 138 : cluster 0 2.1 scrub starts 2024-03-11T20:58:51.786 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:51 smithi023 bash[21372]: cluster 2024-03-11T20:58:50.924621+0000 osd.2 (osd.2) 139 : cluster 0 2.1 scrub ok 2024-03-11T20:58:51.786 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:51 smithi023 bash[21372]: audit 2024-03-11T20:58:51.006628+0000 mon.c (mon.2) 40 : audit 1 from='client.? 172.21.15.23:0/1016538621' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T20:58:51.786 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:51 smithi023 bash[21372]: audit 2024-03-11T20:58:51.007022+0000 mon.a (mon.0) 1118 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T20:58:51.822 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:51.843 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:51.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:51.845 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:58:51.946 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:51.958 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 11 20:58:51 smithi146 systemd[1]: Stopping Ceph osd.4 for 67ba00f8-dfe7-11ee-95c8-87774f69a715... 2024-03-11T20:58:51.958 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 11 20:58:51 smithi146 bash[27362]: debug 2024-03-11T20:58:51.699+0000 7fb2d3d5b700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-03-11T20:58:51.958 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 11 20:58:51 smithi146 bash[27362]: debug 2024-03-11T20:58:51.699+0000 7fb2d3d5b700 -1 osd.4 182 *** Got signal Terminated *** 2024-03-11T20:58:51.958 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 11 20:58:51 smithi146 bash[27362]: debug 2024-03-11T20:58:51.699+0000 7fb2d3d5b700 -1 osd.4 182 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T20:58:52.000 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:52.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:52.004 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:58:52.105 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:58:52.131 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:58:52.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:52.133 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:58:52.234 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:52.262 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:52.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:52.264 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:58:52.291 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T20:58:52.366 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:52.388 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:52.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:52.391 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:58:52.491 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:52.521 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:52.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:52.524 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:58:52.625 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:52.652 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:52.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:52.654 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:58:52.755 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:52.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:52 smithi146 bash[23809]: cluster 2024-03-11T20:58:51.298170+0000 osd.7 (osd.7) 165 : cluster 0 3.13 scrub starts 2024-03-11T20:58:52.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:52 smithi146 bash[23809]: cluster 2024-03-11T20:58:51.302613+0000 osd.7 (osd.7) 166 : cluster 0 3.13 scrub ok 2024-03-11T20:58:52.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:52 smithi146 bash[23809]: cluster 2024-03-11T20:58:51.543751+0000 osd.3 (osd.3) 200 : cluster 0 6.c scrub starts 2024-03-11T20:58:52.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:52 smithi146 bash[23809]: cluster 2024-03-11T20:58:51.544250+0000 mon.a (mon.0) 1119 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T20:58:52.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:52 smithi146 bash[23809]: cluster 2024-03-11T20:58:51.546404+0000 osd.3 (osd.3) 201 : cluster 0 6.c scrub ok 2024-03-11T20:58:52.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:52 smithi146 bash[23809]: audit 2024-03-11T20:58:51.558675+0000 mon.a (mon.0) 1120 : audit 1 from='client.? 172.21.15.23:0/1502532390' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T20:58:52.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:52 smithi146 bash[23809]: audit 2024-03-11T20:58:51.558830+0000 mon.a (mon.0) 1121 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-03-11T20:58:52.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:52 smithi146 bash[23809]: cluster 2024-03-11T20:58:51.565270+0000 mon.a (mon.0) 1122 : cluster 0 osdmap e182: 8 total, 7 up, 8 in 2024-03-11T20:58:52.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:52 smithi146 bash[23809]: cluster 2024-03-11T20:58:51.697900+0000 mon.a (mon.0) 1123 : cluster 1 osd.4 marked itself down and dead 2024-03-11T20:58:52.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:52 smithi146 bash[23809]: cluster 2024-03-11T20:58:51.794125+0000 osd.6 (osd.6) 124 : cluster 0 3.10 scrub starts 2024-03-11T20:58:52.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:52 smithi146 bash[23809]: cluster 2024-03-11T20:58:51.796541+0000 osd.6 (osd.6) 125 : cluster 0 3.10 scrub ok 2024-03-11T20:58:52.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:52 smithi146 bash[23809]: cluster 2024-03-11T20:58:51.997705+0000 mgr.y (mgr.24380) 301 : cluster 0 pgmap v303: 132 pgs: 79 active+clean, 34 active+undersized, 19 active+undersized+degraded; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 921 B/s rd, 0 op/s; 79/594 objects degraded (13.300%) 2024-03-11T20:58:52.766 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:52 smithi023 bash[21372]: cluster 2024-03-11T20:58:51.298170+0000 osd.7 (osd.7) 165 : cluster 0 3.13 scrub starts 2024-03-11T20:58:52.766 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:52 smithi023 bash[21372]: cluster 2024-03-11T20:58:51.302613+0000 osd.7 (osd.7) 166 : cluster 0 3.13 scrub ok 2024-03-11T20:58:52.766 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:52 smithi023 bash[21372]: cluster 2024-03-11T20:58:51.543751+0000 osd.3 (osd.3) 200 : cluster 0 6.c scrub starts 2024-03-11T20:58:52.766 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:52 smithi023 bash[21372]: cluster 2024-03-11T20:58:51.544250+0000 mon.a (mon.0) 1119 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T20:58:52.766 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:52 smithi023 bash[21372]: cluster 2024-03-11T20:58:51.546404+0000 osd.3 (osd.3) 201 : cluster 0 6.c scrub ok 2024-03-11T20:58:52.767 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:52 smithi023 bash[21372]: audit 2024-03-11T20:58:51.558675+0000 mon.a (mon.0) 1120 : audit 1 from='client.? 172.21.15.23:0/1502532390' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T20:58:52.767 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:52 smithi023 bash[21372]: audit 2024-03-11T20:58:51.558830+0000 mon.a (mon.0) 1121 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-03-11T20:58:52.767 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:52 smithi023 bash[21372]: cluster 2024-03-11T20:58:51.565270+0000 mon.a (mon.0) 1122 : cluster 0 osdmap e182: 8 total, 7 up, 8 in 2024-03-11T20:58:52.767 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:52 smithi023 bash[21372]: cluster 2024-03-11T20:58:51.697900+0000 mon.a (mon.0) 1123 : cluster 1 osd.4 marked itself down and dead 2024-03-11T20:58:52.767 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:52 smithi023 bash[21372]: cluster 2024-03-11T20:58:51.794125+0000 osd.6 (osd.6) 124 : cluster 0 3.10 scrub starts 2024-03-11T20:58:52.767 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:52 smithi023 bash[21372]: cluster 2024-03-11T20:58:51.796541+0000 osd.6 (osd.6) 125 : cluster 0 3.10 scrub ok 2024-03-11T20:58:52.767 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:52 smithi023 bash[21372]: cluster 2024-03-11T20:58:51.997705+0000 mgr.y (mgr.24380) 301 : cluster 0 pgmap v303: 132 pgs: 79 active+clean, 34 active+undersized, 19 active+undersized+degraded; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 921 B/s rd, 0 op/s; 79/594 objects degraded (13.300%) 2024-03-11T20:58:52.767 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:52 smithi023 bash[29146]: cluster 2024-03-11T20:58:51.298170+0000 osd.7 (osd.7) 165 : cluster 0 3.13 scrub starts 2024-03-11T20:58:52.767 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:52 smithi023 bash[29146]: cluster 2024-03-11T20:58:51.302613+0000 osd.7 (osd.7) 166 : cluster 0 3.13 scrub ok 2024-03-11T20:58:52.767 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:52 smithi023 bash[29146]: cluster 2024-03-11T20:58:51.543751+0000 osd.3 (osd.3) 200 : cluster 0 6.c scrub starts 2024-03-11T20:58:52.767 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:52 smithi023 bash[29146]: cluster 2024-03-11T20:58:51.544250+0000 mon.a (mon.0) 1119 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T20:58:52.767 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:52 smithi023 bash[29146]: cluster 2024-03-11T20:58:51.546404+0000 osd.3 (osd.3) 201 : cluster 0 6.c scrub ok 2024-03-11T20:58:52.767 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:52 smithi023 bash[29146]: audit 2024-03-11T20:58:51.558675+0000 mon.a (mon.0) 1120 : audit 1 from='client.? 172.21.15.23:0/1502532390' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T20:58:52.768 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:52 smithi023 bash[29146]: audit 2024-03-11T20:58:51.558830+0000 mon.a (mon.0) 1121 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-03-11T20:58:52.768 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:52 smithi023 bash[29146]: cluster 2024-03-11T20:58:51.565270+0000 mon.a (mon.0) 1122 : cluster 0 osdmap e182: 8 total, 7 up, 8 in 2024-03-11T20:58:52.768 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:52 smithi023 bash[29146]: cluster 2024-03-11T20:58:51.697900+0000 mon.a (mon.0) 1123 : cluster 1 osd.4 marked itself down and dead 2024-03-11T20:58:52.768 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:52 smithi023 bash[29146]: cluster 2024-03-11T20:58:51.794125+0000 osd.6 (osd.6) 124 : cluster 0 3.10 scrub starts 2024-03-11T20:58:52.768 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:52 smithi023 bash[29146]: cluster 2024-03-11T20:58:51.796541+0000 osd.6 (osd.6) 125 : cluster 0 3.10 scrub ok 2024-03-11T20:58:52.768 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:52 smithi023 bash[29146]: cluster 2024-03-11T20:58:51.997705+0000 mgr.y (mgr.24380) 301 : cluster 0 pgmap v303: 132 pgs: 79 active+clean, 34 active+undersized, 19 active+undersized+degraded; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 921 B/s rd, 0 op/s; 79/594 objects degraded (13.300%) 2024-03-11T20:58:52.779 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:52.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:52.782 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:58:52.883 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:52.909 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:52.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:52.911 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:58:52.983 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-03-11T20:58:52.984 INFO:tasks.radosbench:Beginning radosbench... 2024-03-11T20:58:52.984 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_1 2024-03-11T20:58:52.984 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd pool create unique_pool_1 16 2024-03-11T20:58:53.013 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:58:53.037 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:58:53.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:53.039 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:58:53.140 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:53.167 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:53.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:53.169 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:58:53.270 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:53.292 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:53.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:53.295 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:58:53.396 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:53.423 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:53.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:53.425 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:58:53.505 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T20:58:53.525 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:58:53.552 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:58:53.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:53.555 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:58:53.656 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:53.679 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:53.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:53.681 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:58:53.782 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:53.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:53 smithi023 bash[29146]: cluster 2024-03-11T20:58:51.943530+0000 osd.0 (osd.0) 165 : cluster 0 purged_snaps scrub starts 2024-03-11T20:58:53.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:53 smithi023 bash[29146]: cluster 2024-03-11T20:58:51.943597+0000 osd.0 (osd.0) 166 : cluster 0 purged_snaps scrub ok 2024-03-11T20:58:53.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:53 smithi023 bash[29146]: cluster 2024-03-11T20:58:52.032248+0000 osd.4 (osd.4) 187 : cluster 0 3.3 deep-scrub starts 2024-03-11T20:58:53.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:53 smithi023 bash[29146]: cluster 2024-03-11T20:58:52.058846+0000 osd.4 (osd.4) 188 : cluster 0 3.3 deep-scrub ok 2024-03-11T20:58:53.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:53 smithi023 bash[29146]: cluster 2024-03-11T20:58:52.348101+0000 osd.7 (osd.7) 167 : cluster 0 6.8 scrub starts 2024-03-11T20:58:53.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:53 smithi023 bash[29146]: cluster 2024-03-11T20:58:52.349707+0000 osd.7 (osd.7) 168 : cluster 0 6.8 scrub ok 2024-03-11T20:58:53.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:53 smithi023 bash[29146]: cluster 2024-03-11T20:58:52.559902+0000 mon.a (mon.0) 1124 : cluster 3 Health check update: 2 osds down (OSD_DOWN) 2024-03-11T20:58:53.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:53 smithi023 bash[29146]: cluster 2024-03-11T20:58:52.576363+0000 mon.a (mon.0) 1125 : cluster 0 osdmap e183: 8 total, 6 up, 8 in 2024-03-11T20:58:53.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:53 smithi023 bash[29146]: cluster 2024-03-11T20:58:52.827202+0000 osd.6 (osd.6) 126 : cluster 0 2.7 scrub starts 2024-03-11T20:58:53.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:53 smithi023 bash[29146]: cluster 2024-03-11T20:58:52.829559+0000 osd.6 (osd.6) 127 : cluster 0 2.7 scrub ok 2024-03-11T20:58:53.786 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:53 smithi023 bash[21372]: cluster 2024-03-11T20:58:51.943530+0000 osd.0 (osd.0) 165 : cluster 0 purged_snaps scrub starts 2024-03-11T20:58:53.786 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:53 smithi023 bash[21372]: cluster 2024-03-11T20:58:51.943597+0000 osd.0 (osd.0) 166 : cluster 0 purged_snaps scrub ok 2024-03-11T20:58:53.786 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:53 smithi023 bash[21372]: cluster 2024-03-11T20:58:52.032248+0000 osd.4 (osd.4) 187 : cluster 0 3.3 deep-scrub starts 2024-03-11T20:58:53.786 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:53 smithi023 bash[21372]: cluster 2024-03-11T20:58:52.058846+0000 osd.4 (osd.4) 188 : cluster 0 3.3 deep-scrub ok 2024-03-11T20:58:53.787 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:53 smithi023 bash[21372]: cluster 2024-03-11T20:58:52.348101+0000 osd.7 (osd.7) 167 : cluster 0 6.8 scrub starts 2024-03-11T20:58:53.787 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:53 smithi023 bash[21372]: cluster 2024-03-11T20:58:52.349707+0000 osd.7 (osd.7) 168 : cluster 0 6.8 scrub ok 2024-03-11T20:58:53.787 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:53 smithi023 bash[21372]: cluster 2024-03-11T20:58:52.559902+0000 mon.a (mon.0) 1124 : cluster 3 Health check update: 2 osds down (OSD_DOWN) 2024-03-11T20:58:53.787 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:53 smithi023 bash[21372]: cluster 2024-03-11T20:58:52.576363+0000 mon.a (mon.0) 1125 : cluster 0 osdmap e183: 8 total, 6 up, 8 in 2024-03-11T20:58:53.787 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:53 smithi023 bash[21372]: cluster 2024-03-11T20:58:52.827202+0000 osd.6 (osd.6) 126 : cluster 0 2.7 scrub starts 2024-03-11T20:58:53.787 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:53 smithi023 bash[21372]: cluster 2024-03-11T20:58:52.829559+0000 osd.6 (osd.6) 127 : cluster 0 2.7 scrub ok 2024-03-11T20:58:53.805 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:53.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:53.807 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:58:53.908 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:53.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:53 smithi146 bash[23809]: cluster 2024-03-11T20:58:51.943530+0000 osd.0 (osd.0) 165 : cluster 0 purged_snaps scrub starts 2024-03-11T20:58:53.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:53 smithi146 bash[23809]: cluster 2024-03-11T20:58:51.943597+0000 osd.0 (osd.0) 166 : cluster 0 purged_snaps scrub ok 2024-03-11T20:58:53.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:53 smithi146 bash[23809]: cluster 2024-03-11T20:58:52.032248+0000 osd.4 (osd.4) 187 : cluster 0 3.3 deep-scrub starts 2024-03-11T20:58:53.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:53 smithi146 bash[23809]: cluster 2024-03-11T20:58:52.058846+0000 osd.4 (osd.4) 188 : cluster 0 3.3 deep-scrub ok 2024-03-11T20:58:53.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:53 smithi146 bash[23809]: cluster 2024-03-11T20:58:52.348101+0000 osd.7 (osd.7) 167 : cluster 0 6.8 scrub starts 2024-03-11T20:58:53.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:53 smithi146 bash[23809]: cluster 2024-03-11T20:58:52.349707+0000 osd.7 (osd.7) 168 : cluster 0 6.8 scrub ok 2024-03-11T20:58:53.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:53 smithi146 bash[23809]: cluster 2024-03-11T20:58:52.559902+0000 mon.a (mon.0) 1124 : cluster 3 Health check update: 2 osds down (OSD_DOWN) 2024-03-11T20:58:53.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:53 smithi146 bash[23809]: cluster 2024-03-11T20:58:52.576363+0000 mon.a (mon.0) 1125 : cluster 0 osdmap e183: 8 total, 6 up, 8 in 2024-03-11T20:58:53.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:53 smithi146 bash[23809]: cluster 2024-03-11T20:58:52.827202+0000 osd.6 (osd.6) 126 : cluster 0 2.7 scrub starts 2024-03-11T20:58:53.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:53 smithi146 bash[23809]: cluster 2024-03-11T20:58:52.829559+0000 osd.6 (osd.6) 127 : cluster 0 2.7 scrub ok 2024-03-11T20:58:53.934 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:53.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:53.936 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:58:54.037 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:54.066 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:54.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:54.068 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:58:54.169 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:54.197 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:54.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:54.199 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:58:54.300 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:58:54.327 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:58:54.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:54.329 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:58:54.430 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:54.452 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:54.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:54.454 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:58:54.555 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:54.583 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:54.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:54.586 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:58:54.614 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 11 20:58:54 smithi146 bash[63559]: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd-4 2024-03-11T20:58:54.614 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:54 smithi146 bash[23809]: cluster 2024-03-11T20:58:52.990972+0000 osd.5 (osd.5) 169 : cluster 0 2.12 scrub starts 2024-03-11T20:58:54.614 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:54 smithi146 bash[23809]: cluster 2024-03-11T20:58:52.992279+0000 osd.5 (osd.5) 170 : cluster 0 2.12 scrub ok 2024-03-11T20:58:54.614 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:54 smithi146 bash[23809]: cluster 2024-03-11T20:58:53.356990+0000 osd.7 (osd.7) 169 : cluster 0 6.9 scrub starts 2024-03-11T20:58:54.615 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:54 smithi146 bash[23809]: cluster 2024-03-11T20:58:53.387723+0000 osd.7 (osd.7) 170 : cluster 0 6.9 scrub ok 2024-03-11T20:58:54.615 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:54 smithi146 bash[23809]: cluster 2024-03-11T20:58:53.576728+0000 mon.a (mon.0) 1126 : cluster 0 osdmap e184: 8 total, 6 up, 8 in 2024-03-11T20:58:54.615 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:54 smithi146 bash[23809]: cluster 2024-03-11T20:58:53.998623+0000 mgr.y (mgr.24380) 302 : cluster 0 pgmap v306: 132 pgs: 4 stale+active+undersized+degraded, 13 stale+active+clean, 6 stale+active+undersized, 66 active+clean, 28 active+undersized, 15 active+undersized+degraded; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 79/594 objects degraded (13.300%) 2024-03-11T20:58:54.687 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:54.695 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 11 20:58:54 smithi146 bash[63770]: Error response from daemon: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd-4 2024-03-11T20:58:54.707 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:54.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:54.709 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:58:54.810 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:54.822 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:54 smithi023 bash[29146]: cluster 2024-03-11T20:58:52.990972+0000 osd.5 (osd.5) 169 : cluster 0 2.12 scrub starts 2024-03-11T20:58:54.822 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:54 smithi023 bash[29146]: cluster 2024-03-11T20:58:52.992279+0000 osd.5 (osd.5) 170 : cluster 0 2.12 scrub ok 2024-03-11T20:58:54.822 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:54 smithi023 bash[29146]: cluster 2024-03-11T20:58:53.356990+0000 osd.7 (osd.7) 169 : cluster 0 6.9 scrub starts 2024-03-11T20:58:54.822 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:54 smithi023 bash[29146]: cluster 2024-03-11T20:58:53.387723+0000 osd.7 (osd.7) 170 : cluster 0 6.9 scrub ok 2024-03-11T20:58:54.822 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:54 smithi023 bash[29146]: cluster 2024-03-11T20:58:53.576728+0000 mon.a (mon.0) 1126 : cluster 0 osdmap e184: 8 total, 6 up, 8 in 2024-03-11T20:58:54.822 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:54 smithi023 bash[29146]: cluster 2024-03-11T20:58:53.998623+0000 mgr.y (mgr.24380) 302 : cluster 0 pgmap v306: 132 pgs: 4 stale+active+undersized+degraded, 13 stale+active+clean, 6 stale+active+undersized, 66 active+clean, 28 active+undersized, 15 active+undersized+degraded; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 79/594 objects degraded (13.300%) 2024-03-11T20:58:54.822 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:54 smithi023 bash[21372]: cluster 2024-03-11T20:58:52.990972+0000 osd.5 (osd.5) 169 : cluster 0 2.12 scrub starts 2024-03-11T20:58:54.823 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:54 smithi023 bash[21372]: cluster 2024-03-11T20:58:52.992279+0000 osd.5 (osd.5) 170 : cluster 0 2.12 scrub ok 2024-03-11T20:58:54.823 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:54 smithi023 bash[21372]: cluster 2024-03-11T20:58:53.356990+0000 osd.7 (osd.7) 169 : cluster 0 6.9 scrub starts 2024-03-11T20:58:54.823 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:54 smithi023 bash[21372]: cluster 2024-03-11T20:58:53.387723+0000 osd.7 (osd.7) 170 : cluster 0 6.9 scrub ok 2024-03-11T20:58:54.823 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:54 smithi023 bash[21372]: cluster 2024-03-11T20:58:53.576728+0000 mon.a (mon.0) 1126 : cluster 0 osdmap e184: 8 total, 6 up, 8 in 2024-03-11T20:58:54.823 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:54 smithi023 bash[21372]: cluster 2024-03-11T20:58:53.998623+0000 mgr.y (mgr.24380) 302 : cluster 0 pgmap v306: 132 pgs: 4 stale+active+undersized+degraded, 13 stale+active+clean, 6 stale+active+undersized, 66 active+clean, 28 active+undersized, 15 active+undersized+degraded; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 79/594 objects degraded (13.300%) 2024-03-11T20:58:54.837 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:54.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:54.840 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:58:54.941 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:54.963 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:54.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:54.966 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:58:55.005 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd set noscrub 2024-03-11T20:58:55.066 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:55.091 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:55.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:55.093 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:58:55.194 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:55.222 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:55.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:55.224 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:58:55.325 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:55.353 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:55.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:55.355 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:58:55.456 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:55.485 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:55.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:55.487 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:58:55.511 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 20:58:55 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T20:58:55.588 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:55.617 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:55.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:55.619 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:58:55.720 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:55.748 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:55.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:55.750 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:58:55.851 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:58:55.878 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:58:55.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:55.880 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:58:55.981 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:58:56.001 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:55 smithi146 bash[23809]: cluster 2024-03-11T20:58:54.378802+0000 osd.7 (osd.7) 171 : cluster 0 3.d scrub starts 2024-03-11T20:58:56.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:55 smithi146 bash[23809]: cluster 2024-03-11T20:58:54.381246+0000 osd.7 (osd.7) 172 : cluster 0 3.d scrub ok 2024-03-11T20:58:56.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:55 smithi146 bash[23809]: audit 2024-03-11T20:58:55.205598+0000 mgr.y (mgr.24380) 303 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:58:56.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:55 smithi146 bash[23809]: cluster 2024-03-11T20:58:55.538743+0000 osd.3 (osd.3) 202 : cluster 0 3.7 scrub starts 2024-03-11T20:58:56.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:55 smithi146 bash[23809]: cluster 2024-03-11T20:58:55.540845+0000 osd.3 (osd.3) 203 : cluster 0 3.7 scrub ok 2024-03-11T20:58:56.008 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:58:56.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:56.010 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:58:56.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:55 smithi023 bash[21372]: cluster 2024-03-11T20:58:54.378802+0000 osd.7 (osd.7) 171 : cluster 0 3.d scrub starts 2024-03-11T20:58:56.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:55 smithi023 bash[21372]: cluster 2024-03-11T20:58:54.381246+0000 osd.7 (osd.7) 172 : cluster 0 3.d scrub ok 2024-03-11T20:58:56.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:55 smithi023 bash[21372]: audit 2024-03-11T20:58:55.205598+0000 mgr.y (mgr.24380) 303 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:58:56.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:55 smithi023 bash[21372]: cluster 2024-03-11T20:58:55.538743+0000 osd.3 (osd.3) 202 : cluster 0 3.7 scrub starts 2024-03-11T20:58:56.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:55 smithi023 bash[21372]: cluster 2024-03-11T20:58:55.540845+0000 osd.3 (osd.3) 203 : cluster 0 3.7 scrub ok 2024-03-11T20:58:56.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:55 smithi023 bash[29146]: cluster 2024-03-11T20:58:54.378802+0000 osd.7 (osd.7) 171 : cluster 0 3.d scrub starts 2024-03-11T20:58:56.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:55 smithi023 bash[29146]: cluster 2024-03-11T20:58:54.381246+0000 osd.7 (osd.7) 172 : cluster 0 3.d scrub ok 2024-03-11T20:58:56.036 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:55 smithi023 bash[29146]: audit 2024-03-11T20:58:55.205598+0000 mgr.y (mgr.24380) 303 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:58:56.036 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:55 smithi023 bash[29146]: cluster 2024-03-11T20:58:55.538743+0000 osd.3 (osd.3) 202 : cluster 0 3.7 scrub starts 2024-03-11T20:58:56.036 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:55 smithi023 bash[29146]: cluster 2024-03-11T20:58:55.540845+0000 osd.3 (osd.3) 203 : cluster 0 3.7 scrub ok 2024-03-11T20:58:56.111 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:58:56.139 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:58:56.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:56.141 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:58:56.242 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:56.270 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:56.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:56.272 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:58:56.373 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:56.405 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:56.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:56.408 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:58:56.508 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:56.535 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:56.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:56.538 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:58:56.639 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:58:56.666 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:58:56.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:56.669 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:58:56.770 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:58:56.795 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:58:56.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:56.798 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:58:56.899 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:56.910 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:56 smithi023 bash[21372]: cluster 2024-03-11T20:58:55.387910+0000 osd.7 (osd.7) 173 : cluster 0 6.16 scrub starts 2024-03-11T20:58:56.910 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:56 smithi023 bash[21372]: cluster 2024-03-11T20:58:55.398284+0000 osd.7 (osd.7) 174 : cluster 0 6.16 scrub ok 2024-03-11T20:58:56.910 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:56 smithi023 bash[21372]: cluster 2024-03-11T20:58:56.000527+0000 mgr.y (mgr.24380) 304 : cluster 0 pgmap v307: 132 pgs: 47 active+clean, 11 undersized+peered, 43 active+undersized, 12 undersized+degraded+peered, 19 active+undersized+degraded; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 182/594 objects degraded (30.640%) 2024-03-11T20:58:56.910 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:56 smithi023 bash[29146]: cluster 2024-03-11T20:58:55.387910+0000 osd.7 (osd.7) 173 : cluster 0 6.16 scrub starts 2024-03-11T20:58:56.911 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:56 smithi023 bash[29146]: cluster 2024-03-11T20:58:55.398284+0000 osd.7 (osd.7) 174 : cluster 0 6.16 scrub ok 2024-03-11T20:58:56.911 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:56 smithi023 bash[29146]: cluster 2024-03-11T20:58:56.000527+0000 mgr.y (mgr.24380) 304 : cluster 0 pgmap v307: 132 pgs: 47 active+clean, 11 undersized+peered, 43 active+undersized, 12 undersized+degraded+peered, 19 active+undersized+degraded; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 182/594 objects degraded (30.640%) 2024-03-11T20:58:56.927 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:56.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:56.929 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:58:57.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:56 smithi146 bash[23809]: cluster 2024-03-11T20:58:55.387910+0000 osd.7 (osd.7) 173 : cluster 0 6.16 scrub starts 2024-03-11T20:58:57.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:56 smithi146 bash[23809]: cluster 2024-03-11T20:58:55.398284+0000 osd.7 (osd.7) 174 : cluster 0 6.16 scrub ok 2024-03-11T20:58:57.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:56 smithi146 bash[23809]: cluster 2024-03-11T20:58:56.000527+0000 mgr.y (mgr.24380) 304 : cluster 0 pgmap v307: 132 pgs: 47 active+clean, 11 undersized+peered, 43 active+undersized, 12 undersized+degraded+peered, 19 active+undersized+degraded; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 182/594 objects degraded (30.640%) 2024-03-11T20:58:57.030 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:57.052 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:57.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:57.054 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:58:57.155 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:57.181 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:57.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:57.184 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:58:57.284 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:57.304 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:57.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:57.306 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:58:57.407 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:58:57.430 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:58:57.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:57.432 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:58:57.533 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:57.554 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:57.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:57.556 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:58:57.657 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:57.685 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:57.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:57.688 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:58:57.750 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:58:57.751 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:58:57.789 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:58:57.803 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:57 smithi146 bash[23809]: cluster 2024-03-11T20:58:55.975533+0000 osd.5 (osd.5) 171 : cluster 0 2.16 deep-scrub starts 2024-03-11T20:58:57.803 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:57 smithi146 bash[23809]: cluster 2024-03-11T20:58:55.977284+0000 osd.5 (osd.5) 172 : cluster 0 2.16 deep-scrub ok 2024-03-11T20:58:57.803 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:57 smithi146 bash[23809]: cluster 2024-03-11T20:58:56.404091+0000 osd.7 (osd.7) 175 : cluster 0 1.0 deep-scrub starts 2024-03-11T20:58:57.803 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:57 smithi146 bash[23809]: cluster 2024-03-11T20:58:56.415255+0000 osd.7 (osd.7) 176 : cluster 0 1.0 deep-scrub ok 2024-03-11T20:58:57.804 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:57 smithi146 bash[23809]: cluster 2024-03-11T20:58:56.726480+0000 mon.a (mon.0) 1127 : cluster 3 Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-03-11T20:58:57.804 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:57 smithi146 bash[23809]: cluster 2024-03-11T20:58:56.726548+0000 mon.a (mon.0) 1128 : cluster 3 Health check update: Degraded data redundancy: 182/594 objects degraded (30.640%), 31 pgs degraded (PG_DEGRADED) 2024-03-11T20:58:57.809 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:58:57.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:57.816 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:58:57.916 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:57.942 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:57.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:57.945 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:58:58.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:57 smithi023 bash[21372]: cluster 2024-03-11T20:58:55.975533+0000 osd.5 (osd.5) 171 : cluster 0 2.16 deep-scrub starts 2024-03-11T20:58:58.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:57 smithi023 bash[21372]: cluster 2024-03-11T20:58:55.977284+0000 osd.5 (osd.5) 172 : cluster 0 2.16 deep-scrub ok 2024-03-11T20:58:58.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:57 smithi023 bash[21372]: cluster 2024-03-11T20:58:56.404091+0000 osd.7 (osd.7) 175 : cluster 0 1.0 deep-scrub starts 2024-03-11T20:58:58.036 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:57 smithi023 bash[21372]: cluster 2024-03-11T20:58:56.415255+0000 osd.7 (osd.7) 176 : cluster 0 1.0 deep-scrub ok 2024-03-11T20:58:58.036 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:57 smithi023 bash[21372]: cluster 2024-03-11T20:58:56.726480+0000 mon.a (mon.0) 1127 : cluster 3 Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-03-11T20:58:58.036 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:57 smithi023 bash[21372]: cluster 2024-03-11T20:58:56.726548+0000 mon.a (mon.0) 1128 : cluster 3 Health check update: Degraded data redundancy: 182/594 objects degraded (30.640%), 31 pgs degraded (PG_DEGRADED) 2024-03-11T20:58:58.036 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:57 smithi023 bash[29146]: cluster 2024-03-11T20:58:55.975533+0000 osd.5 (osd.5) 171 : cluster 0 2.16 deep-scrub starts 2024-03-11T20:58:58.036 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:57 smithi023 bash[29146]: cluster 2024-03-11T20:58:55.977284+0000 osd.5 (osd.5) 172 : cluster 0 2.16 deep-scrub ok 2024-03-11T20:58:58.036 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:57 smithi023 bash[29146]: cluster 2024-03-11T20:58:56.404091+0000 osd.7 (osd.7) 175 : cluster 0 1.0 deep-scrub starts 2024-03-11T20:58:58.036 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:57 smithi023 bash[29146]: cluster 2024-03-11T20:58:56.415255+0000 osd.7 (osd.7) 176 : cluster 0 1.0 deep-scrub ok 2024-03-11T20:58:58.036 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:57 smithi023 bash[29146]: cluster 2024-03-11T20:58:56.726480+0000 mon.a (mon.0) 1127 : cluster 3 Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-03-11T20:58:58.036 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:57 smithi023 bash[29146]: cluster 2024-03-11T20:58:56.726548+0000 mon.a (mon.0) 1128 : cluster 3 Health check update: Degraded data redundancy: 182/594 objects degraded (30.640%), 31 pgs degraded (PG_DEGRADED) 2024-03-11T20:58:58.046 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:58.074 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:58.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:58.076 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:58:58.177 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:58.204 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:58.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:58.206 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:58:58.307 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:58.337 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:58.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:58.340 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:58:58.441 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:58.467 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:58.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:58.469 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:58:58.570 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:58.582 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:58:58 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:20:58:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T20:58:58.596 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:58.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:58.599 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:58:58.699 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:58.726 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:58.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:58.728 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:58:58.829 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:58.843 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:58 smithi023 bash[21372]: cluster 2024-03-11T20:58:57.409864+0000 osd.7 (osd.7) 177 : cluster 0 2.1c scrub starts 2024-03-11T20:58:58.843 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:58 smithi023 bash[21372]: cluster 2024-03-11T20:58:57.412098+0000 osd.7 (osd.7) 178 : cluster 0 2.1c scrub ok 2024-03-11T20:58:58.844 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:58:58 smithi023 bash[21372]: cluster 2024-03-11T20:58:58.001615+0000 mgr.y (mgr.24380) 305 : cluster 0 pgmap v308: 132 pgs: 47 active+clean, 11 undersized+peered, 43 active+undersized, 12 undersized+degraded+peered, 19 active+undersized+degraded; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 182/594 objects degraded (30.640%) 2024-03-11T20:58:58.844 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:58 smithi023 bash[29146]: cluster 2024-03-11T20:58:57.409864+0000 osd.7 (osd.7) 177 : cluster 0 2.1c scrub starts 2024-03-11T20:58:58.844 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:58 smithi023 bash[29146]: cluster 2024-03-11T20:58:57.412098+0000 osd.7 (osd.7) 178 : cluster 0 2.1c scrub ok 2024-03-11T20:58:58.844 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:58:58 smithi023 bash[29146]: cluster 2024-03-11T20:58:58.001615+0000 mgr.y (mgr.24380) 305 : cluster 0 pgmap v308: 132 pgs: 47 active+clean, 11 undersized+peered, 43 active+undersized, 12 undersized+degraded+peered, 19 active+undersized+degraded; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 182/594 objects degraded (30.640%) 2024-03-11T20:58:58.861 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:58.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:58.864 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:58:58.964 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:58.976 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:58 smithi146 bash[23809]: cluster 2024-03-11T20:58:57.409864+0000 osd.7 (osd.7) 177 : cluster 0 2.1c scrub starts 2024-03-11T20:58:59.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:58 smithi146 bash[23809]: cluster 2024-03-11T20:58:57.412098+0000 osd.7 (osd.7) 178 : cluster 0 2.1c scrub ok 2024-03-11T20:58:59.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:58:58 smithi146 bash[23809]: cluster 2024-03-11T20:58:58.001615+0000 mgr.y (mgr.24380) 305 : cluster 0 pgmap v308: 132 pgs: 47 active+clean, 11 undersized+peered, 43 active+undersized, 12 undersized+degraded+peered, 19 active+undersized+degraded; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 182/594 objects degraded (30.640%) 2024-03-11T20:58:59.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:59.005 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:59.006 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:58:59.106 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:59.133 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:59.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:59.135 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:58:59.236 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:59.265 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:58:59.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:59.268 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:58:59.369 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:59.394 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:58:59.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:59.396 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:58:59.435 DEBUG:teuthology.orchestra.run.smithi146:> sudo pkill -f 'journalctl -f -n 0 -u ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@osd.4.service' 2024-03-11T20:58:59.497 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:59.512 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T20:58:59.512 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-03-11T20:58:59.512 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.4 2024-03-11T20:58:59.512 INFO:tasks.thrashosds.thrasher:remote for osd 4 is ubuntu@smithi146.front.sepia.ceph.com 2024-03-11T20:58:59.513 INFO:tasks.thrashosds.thrasher:remote for exp osd 1 is ubuntu@smithi023.front.sepia.ceph.com 2024-03-11T20:58:59.513 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -n osd.1 --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-1 --journal-path /var/lib/ceph/osd/ceph-1/journal --op list-pgs 2024-03-11T20:58:59.571 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:58:59.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:59.573 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:58:59.645 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-03-11T20:58:59.645 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-03-11T20:58:59.645 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-03-11T20:58:59.645 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-03-11T20:58:59.675 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:59.689 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/osd.1/config 2024-03-11T20:58:59.699 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:58:59.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:59.701 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:58:59.786 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:58:59.803 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:58:59.828 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:58:59.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:59.831 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:58:59.931 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:58:59.959 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:58:59.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:58:59.962 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:59:00.063 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:00.091 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:00.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:00.094 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:59:00.195 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:00.221 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:00.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:00.223 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:59:00.324 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:00.349 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:00.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:00.351 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:59:00.452 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:00.482 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:00.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:00.484 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:59:00.585 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:00.588 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-03-11T20:59:00.609 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:00.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:00.611 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:59:00.712 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:00.742 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:00.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:00.744 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:59:00.845 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:00.874 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:00.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:00.876 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:59:00.977 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:01.005 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:01.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:01.007 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:59:01.108 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:01.135 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:01.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:01.138 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:59:01.238 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:01.265 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:01.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:01.267 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:59:01.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:01 smithi023 bash[21372]: cluster 2024-03-11T20:58:59.968958+0000 osd.5 (osd.5) 173 : cluster 0 6.a scrub starts 2024-03-11T20:59:01.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:01 smithi023 bash[21372]: cluster 2024-03-11T20:58:59.981764+0000 osd.5 (osd.5) 174 : cluster 0 6.a scrub ok 2024-03-11T20:59:01.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:01 smithi023 bash[21372]: cluster 2024-03-11T20:59:00.003587+0000 mgr.y (mgr.24380) 306 : cluster 0 pgmap v309: 132 pgs: 47 active+clean, 11 undersized+peered, 43 active+undersized, 12 undersized+degraded+peered, 19 active+undersized+degraded; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 182/594 objects degraded (30.640%) 2024-03-11T20:59:01.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:01 smithi023 bash[29146]: cluster 2024-03-11T20:58:59.968958+0000 osd.5 (osd.5) 173 : cluster 0 6.a scrub starts 2024-03-11T20:59:01.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:01 smithi023 bash[29146]: cluster 2024-03-11T20:58:59.981764+0000 osd.5 (osd.5) 174 : cluster 0 6.a scrub ok 2024-03-11T20:59:01.286 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:01 smithi023 bash[29146]: cluster 2024-03-11T20:59:00.003587+0000 mgr.y (mgr.24380) 306 : cluster 0 pgmap v309: 132 pgs: 47 active+clean, 11 undersized+peered, 43 active+undersized, 12 undersized+degraded+peered, 19 active+undersized+degraded; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 182/594 objects degraded (30.640%) 2024-03-11T20:59:01.368 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:01.398 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:01.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:01.400 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:59:01.501 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:01.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:01 smithi146 bash[23809]: cluster 2024-03-11T20:58:59.968958+0000 osd.5 (osd.5) 173 : cluster 0 6.a scrub starts 2024-03-11T20:59:01.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:01 smithi146 bash[23809]: cluster 2024-03-11T20:58:59.981764+0000 osd.5 (osd.5) 174 : cluster 0 6.a scrub ok 2024-03-11T20:59:01.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:01 smithi146 bash[23809]: cluster 2024-03-11T20:59:00.003587+0000 mgr.y (mgr.24380) 306 : cluster 0 pgmap v309: 132 pgs: 47 active+clean, 11 undersized+peered, 43 active+undersized, 12 undersized+degraded+peered, 19 active+undersized+degraded; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 182/594 objects degraded (30.640%) 2024-03-11T20:59:01.528 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:01.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:01.531 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:59:01.631 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:01.661 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:01.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:01.664 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:59:01.765 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:01.792 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:01.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:01.794 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:59:01.895 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:01.923 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:01.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:01.925 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:59:02.026 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:02.052 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:02.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:02.055 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:59:02.156 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:02.185 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:02.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:02.187 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:59:02.187 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:02 smithi023 bash[29146]: cluster 2024-03-11T20:59:00.423669+0000 osd.7 (osd.7) 179 : cluster 0 6.3 scrub starts 2024-03-11T20:59:02.187 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:02 smithi023 bash[29146]: cluster 2024-03-11T20:59:00.468300+0000 osd.7 (osd.7) 180 : cluster 0 6.3 scrub ok 2024-03-11T20:59:02.187 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:02 smithi023 bash[29146]: cluster 2024-03-11T20:59:00.943847+0000 osd.5 (osd.5) 175 : cluster 0 4.8 deep-scrub starts 2024-03-11T20:59:02.187 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:02 smithi023 bash[29146]: cluster 2024-03-11T20:59:00.946083+0000 osd.5 (osd.5) 176 : cluster 0 4.8 deep-scrub ok 2024-03-11T20:59:02.187 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:02 smithi023 bash[29146]: cluster 2024-03-11T20:59:01.600298+0000 osd.3 (osd.3) 204 : cluster 0 purged_snaps scrub starts 2024-03-11T20:59:02.188 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:02 smithi023 bash[29146]: cluster 2024-03-11T20:59:01.600398+0000 osd.3 (osd.3) 205 : cluster 0 purged_snaps scrub ok 2024-03-11T20:59:02.188 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:02 smithi023 bash[21372]: cluster 2024-03-11T20:59:00.423669+0000 osd.7 (osd.7) 179 : cluster 0 6.3 scrub starts 2024-03-11T20:59:02.188 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:02 smithi023 bash[21372]: cluster 2024-03-11T20:59:00.468300+0000 osd.7 (osd.7) 180 : cluster 0 6.3 scrub ok 2024-03-11T20:59:02.188 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:02 smithi023 bash[21372]: cluster 2024-03-11T20:59:00.943847+0000 osd.5 (osd.5) 175 : cluster 0 4.8 deep-scrub starts 2024-03-11T20:59:02.188 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:02 smithi023 bash[21372]: cluster 2024-03-11T20:59:00.946083+0000 osd.5 (osd.5) 176 : cluster 0 4.8 deep-scrub ok 2024-03-11T20:59:02.188 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:02 smithi023 bash[21372]: cluster 2024-03-11T20:59:01.600298+0000 osd.3 (osd.3) 204 : cluster 0 purged_snaps scrub starts 2024-03-11T20:59:02.188 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:02 smithi023 bash[21372]: cluster 2024-03-11T20:59:01.600398+0000 osd.3 (osd.3) 205 : cluster 0 purged_snaps scrub ok 2024-03-11T20:59:02.288 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:02.317 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:02.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:02.322 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:59:02.423 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:02.451 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:02.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:02.453 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:59:02.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:02 smithi146 bash[23809]: cluster 2024-03-11T20:59:00.423669+0000 osd.7 (osd.7) 179 : cluster 0 6.3 scrub starts 2024-03-11T20:59:02.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:02 smithi146 bash[23809]: cluster 2024-03-11T20:59:00.468300+0000 osd.7 (osd.7) 180 : cluster 0 6.3 scrub ok 2024-03-11T20:59:02.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:02 smithi146 bash[23809]: cluster 2024-03-11T20:59:00.943847+0000 osd.5 (osd.5) 175 : cluster 0 4.8 deep-scrub starts 2024-03-11T20:59:02.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:02 smithi146 bash[23809]: cluster 2024-03-11T20:59:00.946083+0000 osd.5 (osd.5) 176 : cluster 0 4.8 deep-scrub ok 2024-03-11T20:59:02.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:02 smithi146 bash[23809]: cluster 2024-03-11T20:59:01.600298+0000 osd.3 (osd.3) 204 : cluster 0 purged_snaps scrub starts 2024-03-11T20:59:02.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:02 smithi146 bash[23809]: cluster 2024-03-11T20:59:01.600398+0000 osd.3 (osd.3) 205 : cluster 0 purged_snaps scrub ok 2024-03-11T20:59:02.553 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:02.578 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:02.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:02.580 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:59:02.681 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:02.710 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:02.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:02.712 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:59:02.813 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:02.837 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:02.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:02.840 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:59:02.941 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:02.968 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:02.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:02.970 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:59:03.071 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:03.099 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:03.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:03.101 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:59:03.202 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:03.224 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:03.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:03.226 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:59:03.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:03 smithi146 bash[23809]: cluster 2024-03-11T20:59:01.394134+0000 osd.7 (osd.7) 181 : cluster 0 2.1d scrub starts 2024-03-11T20:59:03.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:03 smithi146 bash[23809]: cluster 2024-03-11T20:59:01.396365+0000 osd.7 (osd.7) 182 : cluster 0 2.1d scrub ok 2024-03-11T20:59:03.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:03 smithi146 bash[23809]: cluster 2024-03-11T20:59:02.005431+0000 mgr.y (mgr.24380) 307 : cluster 0 pgmap v310: 132 pgs: 47 active+clean, 11 undersized+peered, 43 active+undersized, 12 undersized+degraded+peered, 19 active+undersized+degraded; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 182/594 objects degraded (30.640%) 2024-03-11T20:59:03.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:03 smithi146 bash[23809]: cluster 2024-03-11T20:59:02.077758+0000 mon.a (mon.0) 1129 : cluster 0 osdmap e185: 8 total, 6 up, 8 in 2024-03-11T20:59:03.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:03 smithi146 bash[23809]: cluster 2024-03-11T20:59:02.532955+0000 osd.3 (osd.3) 206 : cluster 0 3.2 scrub starts 2024-03-11T20:59:03.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:03 smithi146 bash[23809]: cluster 2024-03-11T20:59:02.536976+0000 osd.3 (osd.3) 207 : cluster 0 3.2 scrub ok 2024-03-11T20:59:03.327 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:03.340 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:03 smithi023 bash[29146]: cluster 2024-03-11T20:59:01.394134+0000 osd.7 (osd.7) 181 : cluster 0 2.1d scrub starts 2024-03-11T20:59:03.340 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:03 smithi023 bash[29146]: cluster 2024-03-11T20:59:01.396365+0000 osd.7 (osd.7) 182 : cluster 0 2.1d scrub ok 2024-03-11T20:59:03.340 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:03 smithi023 bash[29146]: cluster 2024-03-11T20:59:02.005431+0000 mgr.y (mgr.24380) 307 : cluster 0 pgmap v310: 132 pgs: 47 active+clean, 11 undersized+peered, 43 active+undersized, 12 undersized+degraded+peered, 19 active+undersized+degraded; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 182/594 objects degraded (30.640%) 2024-03-11T20:59:03.340 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:03 smithi023 bash[29146]: cluster 2024-03-11T20:59:02.077758+0000 mon.a (mon.0) 1129 : cluster 0 osdmap e185: 8 total, 6 up, 8 in 2024-03-11T20:59:03.340 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:03 smithi023 bash[29146]: cluster 2024-03-11T20:59:02.532955+0000 osd.3 (osd.3) 206 : cluster 0 3.2 scrub starts 2024-03-11T20:59:03.341 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:03 smithi023 bash[29146]: cluster 2024-03-11T20:59:02.536976+0000 osd.3 (osd.3) 207 : cluster 0 3.2 scrub ok 2024-03-11T20:59:03.341 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:03 smithi023 bash[21372]: cluster 2024-03-11T20:59:01.394134+0000 osd.7 (osd.7) 181 : cluster 0 2.1d scrub starts 2024-03-11T20:59:03.341 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:03 smithi023 bash[21372]: cluster 2024-03-11T20:59:01.396365+0000 osd.7 (osd.7) 182 : cluster 0 2.1d scrub ok 2024-03-11T20:59:03.341 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:03 smithi023 bash[21372]: cluster 2024-03-11T20:59:02.005431+0000 mgr.y (mgr.24380) 307 : cluster 0 pgmap v310: 132 pgs: 47 active+clean, 11 undersized+peered, 43 active+undersized, 12 undersized+degraded+peered, 19 active+undersized+degraded; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 182/594 objects degraded (30.640%) 2024-03-11T20:59:03.341 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:03 smithi023 bash[21372]: cluster 2024-03-11T20:59:02.077758+0000 mon.a (mon.0) 1129 : cluster 0 osdmap e185: 8 total, 6 up, 8 in 2024-03-11T20:59:03.342 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:03 smithi023 bash[21372]: cluster 2024-03-11T20:59:02.532955+0000 osd.3 (osd.3) 206 : cluster 0 3.2 scrub starts 2024-03-11T20:59:03.342 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:03 smithi023 bash[21372]: cluster 2024-03-11T20:59:02.536976+0000 osd.3 (osd.3) 207 : cluster 0 3.2 scrub ok 2024-03-11T20:59:03.357 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:03.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:03.360 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:59:03.461 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:03.490 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:03.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:03.492 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:59:03.593 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:03.618 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:03.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:03.620 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:59:03.721 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:03.750 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:03.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:03.752 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:59:03.853 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:03.882 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:03.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:03.885 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:59:03.986 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:04.014 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:04.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:04.016 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:59:04.117 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:04.120 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:04 smithi023 bash[21372]: cluster 2024-03-11T20:59:02.892789+0000 osd.2 (osd.2) 140 : cluster 0 purged_snaps scrub starts 2024-03-11T20:59:04.120 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:04 smithi023 bash[21372]: cluster 2024-03-11T20:59:02.892912+0000 osd.2 (osd.2) 141 : cluster 0 purged_snaps scrub ok 2024-03-11T20:59:04.121 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:04 smithi023 bash[21372]: cluster 2024-03-11T20:59:03.092297+0000 mon.a (mon.0) 1130 : cluster 0 osdmap e186: 8 total, 6 up, 8 in 2024-03-11T20:59:04.121 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:04 smithi023 bash[21372]: cluster 2024-03-11T20:59:03.482364+0000 osd.3 (osd.3) 208 : cluster 0 2.1e scrub starts 2024-03-11T20:59:04.121 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:04 smithi023 bash[21372]: cluster 2024-03-11T20:59:03.485960+0000 osd.3 (osd.3) 209 : cluster 0 2.1e scrub ok 2024-03-11T20:59:04.143 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:04.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:04.145 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:59:04.246 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:04.275 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:04.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:04.277 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:59:04.378 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:04.389 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:04 smithi023 bash[29146]: cluster 2024-03-11T20:59:02.892789+0000 osd.2 (osd.2) 140 : cluster 0 purged_snaps scrub starts 2024-03-11T20:59:04.389 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:04 smithi023 bash[29146]: cluster 2024-03-11T20:59:02.892912+0000 osd.2 (osd.2) 141 : cluster 0 purged_snaps scrub ok 2024-03-11T20:59:04.389 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:04 smithi023 bash[29146]: cluster 2024-03-11T20:59:03.092297+0000 mon.a (mon.0) 1130 : cluster 0 osdmap e186: 8 total, 6 up, 8 in 2024-03-11T20:59:04.389 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:04 smithi023 bash[29146]: cluster 2024-03-11T20:59:03.482364+0000 osd.3 (osd.3) 208 : cluster 0 2.1e scrub starts 2024-03-11T20:59:04.390 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:04 smithi023 bash[29146]: cluster 2024-03-11T20:59:03.485960+0000 osd.3 (osd.3) 209 : cluster 0 2.1e scrub ok 2024-03-11T20:59:04.406 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:04.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:04.409 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:59:04.510 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:04.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:04 smithi146 bash[23809]: cluster 2024-03-11T20:59:02.892789+0000 osd.2 (osd.2) 140 : cluster 0 purged_snaps scrub starts 2024-03-11T20:59:04.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:04 smithi146 bash[23809]: cluster 2024-03-11T20:59:02.892912+0000 osd.2 (osd.2) 141 : cluster 0 purged_snaps scrub ok 2024-03-11T20:59:04.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:04 smithi146 bash[23809]: cluster 2024-03-11T20:59:03.092297+0000 mon.a (mon.0) 1130 : cluster 0 osdmap e186: 8 total, 6 up, 8 in 2024-03-11T20:59:04.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:04 smithi146 bash[23809]: cluster 2024-03-11T20:59:03.482364+0000 osd.3 (osd.3) 208 : cluster 0 2.1e scrub starts 2024-03-11T20:59:04.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:04 smithi146 bash[23809]: cluster 2024-03-11T20:59:03.485960+0000 osd.3 (osd.3) 209 : cluster 0 2.1e scrub ok 2024-03-11T20:59:04.537 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:04.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:04.539 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:59:04.640 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:04.669 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:04.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:04.671 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:59:04.773 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:04.802 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:04.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:04.804 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:59:04.905 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:04.931 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:04.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:04.933 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:59:05.034 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:05.062 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:05.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:05.064 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:59:05.165 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:05.194 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:05.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:05.197 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:59:05.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:05 smithi146 bash[23809]: cluster 2024-03-11T20:59:04.006757+0000 mgr.y (mgr.24380) 308 : cluster 0 pgmap v313: 132 pgs: 47 active+clean, 11 undersized+peered, 43 active+undersized, 12 undersized+degraded+peered, 19 active+undersized+degraded; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 182/594 objects degraded (30.640%) 2024-03-11T20:59:05.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:05 smithi146 bash[23809]: cluster 2024-03-11T20:59:04.518022+0000 osd.3 (osd.3) 210 : cluster 0 6.11 deep-scrub starts 2024-03-11T20:59:05.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:05 smithi146 bash[23809]: cluster 2024-03-11T20:59:04.534837+0000 osd.3 (osd.3) 211 : cluster 0 6.11 deep-scrub ok 2024-03-11T20:59:05.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:05 smithi146 bash[23809]: cluster 2024-03-11T20:59:04.709413+0000 osd.6 (osd.6) 128 : cluster 0 purged_snaps scrub starts 2024-03-11T20:59:05.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:05 smithi146 bash[23809]: cluster 2024-03-11T20:59:04.709534+0000 osd.6 (osd.6) 129 : cluster 0 purged_snaps scrub ok 2024-03-11T20:59:05.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:05 smithi146 bash[23809]: audit 2024-03-11T20:59:04.719604+0000 mon.a (mon.0) 1131 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:59:05.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:05 smithi146 bash[23809]: cluster 2024-03-11T20:59:04.720075+0000 mon.a (mon.0) 1132 : cluster 3 Health check update: Reduced data availability: 6 pgs inactive (PG_AVAILABILITY) 2024-03-11T20:59:05.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:05 smithi146 bash[23809]: cluster 2024-03-11T20:59:04.720183+0000 mon.a (mon.0) 1133 : cluster 3 Health check update: Degraded data redundancy: 182/594 objects degraded (30.640%), 31 pgs degraded, 30 pgs undersized (PG_DEGRADED) 2024-03-11T20:59:05.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:05 smithi146 bash[23809]: audit 2024-03-11T20:59:04.721414+0000 mon.a (mon.0) 1134 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:59:05.298 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:05.326 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:05.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:05.328 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:59:05.367 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T20:59:05.384 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:05 smithi023 bash[21372]: cluster 2024-03-11T20:59:04.006757+0000 mgr.y (mgr.24380) 308 : cluster 0 pgmap v313: 132 pgs: 47 active+clean, 11 undersized+peered, 43 active+undersized, 12 undersized+degraded+peered, 19 active+undersized+degraded; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 182/594 objects degraded (30.640%) 2024-03-11T20:59:05.384 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:05 smithi023 bash[21372]: cluster 2024-03-11T20:59:04.518022+0000 osd.3 (osd.3) 210 : cluster 0 6.11 deep-scrub starts 2024-03-11T20:59:05.384 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:05 smithi023 bash[21372]: cluster 2024-03-11T20:59:04.534837+0000 osd.3 (osd.3) 211 : cluster 0 6.11 deep-scrub ok 2024-03-11T20:59:05.384 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:05 smithi023 bash[21372]: cluster 2024-03-11T20:59:04.709413+0000 osd.6 (osd.6) 128 : cluster 0 purged_snaps scrub starts 2024-03-11T20:59:05.384 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:05 smithi023 bash[21372]: cluster 2024-03-11T20:59:04.709534+0000 osd.6 (osd.6) 129 : cluster 0 purged_snaps scrub ok 2024-03-11T20:59:05.384 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:05 smithi023 bash[21372]: audit 2024-03-11T20:59:04.719604+0000 mon.a (mon.0) 1131 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:59:05.384 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:05 smithi023 bash[21372]: cluster 2024-03-11T20:59:04.720075+0000 mon.a (mon.0) 1132 : cluster 3 Health check update: Reduced data availability: 6 pgs inactive (PG_AVAILABILITY) 2024-03-11T20:59:05.384 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:05 smithi023 bash[21372]: cluster 2024-03-11T20:59:04.720183+0000 mon.a (mon.0) 1133 : cluster 3 Health check update: Degraded data redundancy: 182/594 objects degraded (30.640%), 31 pgs degraded, 30 pgs undersized (PG_DEGRADED) 2024-03-11T20:59:05.384 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:05 smithi023 bash[21372]: audit 2024-03-11T20:59:04.721414+0000 mon.a (mon.0) 1134 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:59:05.385 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:05 smithi023 bash[29146]: cluster 2024-03-11T20:59:04.006757+0000 mgr.y (mgr.24380) 308 : cluster 0 pgmap v313: 132 pgs: 47 active+clean, 11 undersized+peered, 43 active+undersized, 12 undersized+degraded+peered, 19 active+undersized+degraded; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 182/594 objects degraded (30.640%) 2024-03-11T20:59:05.385 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:05 smithi023 bash[29146]: cluster 2024-03-11T20:59:04.518022+0000 osd.3 (osd.3) 210 : cluster 0 6.11 deep-scrub starts 2024-03-11T20:59:05.385 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:05 smithi023 bash[29146]: cluster 2024-03-11T20:59:04.534837+0000 osd.3 (osd.3) 211 : cluster 0 6.11 deep-scrub ok 2024-03-11T20:59:05.385 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:05 smithi023 bash[29146]: cluster 2024-03-11T20:59:04.709413+0000 osd.6 (osd.6) 128 : cluster 0 purged_snaps scrub starts 2024-03-11T20:59:05.386 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:05 smithi023 bash[29146]: cluster 2024-03-11T20:59:04.709534+0000 osd.6 (osd.6) 129 : cluster 0 purged_snaps scrub ok 2024-03-11T20:59:05.386 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:05 smithi023 bash[29146]: audit 2024-03-11T20:59:04.719604+0000 mon.a (mon.0) 1131 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:59:05.386 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:05 smithi023 bash[29146]: cluster 2024-03-11T20:59:04.720075+0000 mon.a (mon.0) 1132 : cluster 3 Health check update: Reduced data availability: 6 pgs inactive (PG_AVAILABILITY) 2024-03-11T20:59:05.386 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:05 smithi023 bash[29146]: cluster 2024-03-11T20:59:04.720183+0000 mon.a (mon.0) 1133 : cluster 3 Health check update: Degraded data redundancy: 182/594 objects degraded (30.640%), 31 pgs degraded, 30 pgs undersized (PG_DEGRADED) 2024-03-11T20:59:05.386 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:05 smithi023 bash[29146]: audit 2024-03-11T20:59:04.721414+0000 mon.a (mon.0) 1134 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:59:05.388 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 20:59:05 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T20:59:05.430 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:05.451 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:05.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:05.454 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:59:05.554 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:05.580 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:05.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:05.582 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:59:05.683 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:05.710 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:05.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:05.712 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:59:05.813 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:05.842 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:05.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:05.844 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:59:05.945 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:05.968 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:05.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:05.971 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:59:06.071 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:06.101 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:06.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:06.103 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:59:06.204 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:06.233 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:06.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:06.235 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:59:06.258 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:06 smithi023 bash[29146]: cluster 2024-03-11T20:59:05.112619+0000 mon.a (mon.0) 1135 : cluster 0 osdmap e187: 8 total, 6 up, 8 in 2024-03-11T20:59:06.258 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:06 smithi023 bash[29146]: audit 2024-03-11T20:59:05.207067+0000 mgr.y (mgr.24380) 309 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:59:06.258 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:06 smithi023 bash[29146]: cluster 2024-03-11T20:59:05.479189+0000 osd.3 (osd.3) 212 : cluster 0 4.0 scrub starts 2024-03-11T20:59:06.258 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:06 smithi023 bash[29146]: cluster 2024-03-11T20:59:05.483587+0000 osd.3 (osd.3) 213 : cluster 0 4.0 scrub ok 2024-03-11T20:59:06.258 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:06 smithi023 bash[21372]: cluster 2024-03-11T20:59:05.112619+0000 mon.a (mon.0) 1135 : cluster 0 osdmap e187: 8 total, 6 up, 8 in 2024-03-11T20:59:06.259 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:06 smithi023 bash[21372]: audit 2024-03-11T20:59:05.207067+0000 mgr.y (mgr.24380) 309 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:59:06.259 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:06 smithi023 bash[21372]: cluster 2024-03-11T20:59:05.479189+0000 osd.3 (osd.3) 212 : cluster 0 4.0 scrub starts 2024-03-11T20:59:06.259 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:06 smithi023 bash[21372]: cluster 2024-03-11T20:59:05.483587+0000 osd.3 (osd.3) 213 : cluster 0 4.0 scrub ok 2024-03-11T20:59:06.336 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:06.363 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:06.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:06.365 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:59:06.466 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:06.491 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:06.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:06.494 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:59:06.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:06 smithi146 bash[23809]: cluster 2024-03-11T20:59:05.112619+0000 mon.a (mon.0) 1135 : cluster 0 osdmap e187: 8 total, 6 up, 8 in 2024-03-11T20:59:06.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:06 smithi146 bash[23809]: audit 2024-03-11T20:59:05.207067+0000 mgr.y (mgr.24380) 309 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:59:06.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:06 smithi146 bash[23809]: cluster 2024-03-11T20:59:05.479189+0000 osd.3 (osd.3) 212 : cluster 0 4.0 scrub starts 2024-03-11T20:59:06.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:06 smithi146 bash[23809]: cluster 2024-03-11T20:59:05.483587+0000 osd.3 (osd.3) 213 : cluster 0 4.0 scrub ok 2024-03-11T20:59:06.595 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:06.625 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:06.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:06.628 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:59:06.729 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:06.753 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:06.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:06.755 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:59:06.857 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:06.885 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:06.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:06.888 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:59:06.989 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:07.018 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:07.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:07.021 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:59:07.122 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:07.149 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:07.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:07.151 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:59:07.252 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:07.281 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:07.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:07.284 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:59:07.385 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:07.398 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:07 smithi023 bash[29146]: cluster 2024-03-11T20:59:05.662410+0000 osd.7 (osd.7) 183 : cluster 0 purged_snaps scrub starts 2024-03-11T20:59:07.398 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:07 smithi023 bash[29146]: cluster 2024-03-11T20:59:05.662512+0000 osd.7 (osd.7) 184 : cluster 0 purged_snaps scrub ok 2024-03-11T20:59:07.398 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:07 smithi023 bash[29146]: cluster 2024-03-11T20:59:06.007845+0000 mgr.y (mgr.24380) 310 : cluster 0 pgmap v315: 132 pgs: 47 active+clean, 11 undersized+peered, 43 active+undersized, 12 undersized+degraded+peered, 19 active+undersized+degraded; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 182/594 objects degraded (30.640%) 2024-03-11T20:59:07.398 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:07 smithi023 bash[29146]: cluster 2024-03-11T20:59:06.144508+0000 mon.a (mon.0) 1136 : cluster 0 osdmap e188: 8 total, 6 up, 8 in 2024-03-11T20:59:07.398 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:07 smithi023 bash[29146]: cluster 2024-03-11T20:59:06.454491+0000 osd.3 (osd.3) 214 : cluster 0 2.19 scrub starts 2024-03-11T20:59:07.398 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:07 smithi023 bash[29146]: cluster 2024-03-11T20:59:06.457546+0000 osd.3 (osd.3) 215 : cluster 0 2.19 scrub ok 2024-03-11T20:59:07.399 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:07 smithi023 bash[21372]: cluster 2024-03-11T20:59:05.662410+0000 osd.7 (osd.7) 183 : cluster 0 purged_snaps scrub starts 2024-03-11T20:59:07.399 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:07 smithi023 bash[21372]: cluster 2024-03-11T20:59:05.662512+0000 osd.7 (osd.7) 184 : cluster 0 purged_snaps scrub ok 2024-03-11T20:59:07.399 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:07 smithi023 bash[21372]: cluster 2024-03-11T20:59:06.007845+0000 mgr.y (mgr.24380) 310 : cluster 0 pgmap v315: 132 pgs: 47 active+clean, 11 undersized+peered, 43 active+undersized, 12 undersized+degraded+peered, 19 active+undersized+degraded; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 182/594 objects degraded (30.640%) 2024-03-11T20:59:07.399 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:07 smithi023 bash[21372]: cluster 2024-03-11T20:59:06.144508+0000 mon.a (mon.0) 1136 : cluster 0 osdmap e188: 8 total, 6 up, 8 in 2024-03-11T20:59:07.399 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:07 smithi023 bash[21372]: cluster 2024-03-11T20:59:06.454491+0000 osd.3 (osd.3) 214 : cluster 0 2.19 scrub starts 2024-03-11T20:59:07.399 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:07 smithi023 bash[21372]: cluster 2024-03-11T20:59:06.457546+0000 osd.3 (osd.3) 215 : cluster 0 2.19 scrub ok 2024-03-11T20:59:07.411 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:07.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:07.414 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:59:07.478 INFO:teuthology.orchestra.run.smithi023.stderr:Creating an OSD daemon form without an OSD FSID value 2024-03-11T20:59:07.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:07 smithi146 bash[23809]: cluster 2024-03-11T20:59:05.662410+0000 osd.7 (osd.7) 183 : cluster 0 purged_snaps scrub starts 2024-03-11T20:59:07.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:07 smithi146 bash[23809]: cluster 2024-03-11T20:59:05.662512+0000 osd.7 (osd.7) 184 : cluster 0 purged_snaps scrub ok 2024-03-11T20:59:07.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:07 smithi146 bash[23809]: cluster 2024-03-11T20:59:06.007845+0000 mgr.y (mgr.24380) 310 : cluster 0 pgmap v315: 132 pgs: 47 active+clean, 11 undersized+peered, 43 active+undersized, 12 undersized+degraded+peered, 19 active+undersized+degraded; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 182/594 objects degraded (30.640%) 2024-03-11T20:59:07.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:07 smithi146 bash[23809]: cluster 2024-03-11T20:59:06.144508+0000 mon.a (mon.0) 1136 : cluster 0 osdmap e188: 8 total, 6 up, 8 in 2024-03-11T20:59:07.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:07 smithi146 bash[23809]: cluster 2024-03-11T20:59:06.454491+0000 osd.3 (osd.3) 214 : cluster 0 2.19 scrub starts 2024-03-11T20:59:07.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:07 smithi146 bash[23809]: cluster 2024-03-11T20:59:06.457546+0000 osd.3 (osd.3) 215 : cluster 0 2.19 scrub ok 2024-03-11T20:59:07.516 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:07.536 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:07.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:07.538 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:59:07.639 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:07.668 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:07.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:07.671 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:59:07.771 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:07.801 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:07.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:07.804 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:59:07.905 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:07.931 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:07.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:07.933 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:59:08.034 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:08.063 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:08.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:08.065 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:59:08.166 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:08.192 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:08.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:08.194 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:59:08.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:08 smithi146 bash[23809]: cluster 2024-03-11T20:59:06.974372+0000 osd.5 (osd.5) 177 : cluster 0 3.17 scrub starts 2024-03-11T20:59:08.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:08 smithi146 bash[23809]: cluster 2024-03-11T20:59:06.978035+0000 osd.5 (osd.5) 178 : cluster 0 3.17 scrub ok 2024-03-11T20:59:08.295 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:08.323 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:08.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:08.325 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:59:08.346 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:08 smithi023 bash[21372]: cluster 2024-03-11T20:59:06.974372+0000 osd.5 (osd.5) 177 : cluster 0 3.17 scrub starts 2024-03-11T20:59:08.346 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:08 smithi023 bash[21372]: cluster 2024-03-11T20:59:06.978035+0000 osd.5 (osd.5) 178 : cluster 0 3.17 scrub ok 2024-03-11T20:59:08.346 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:08 smithi023 bash[29146]: cluster 2024-03-11T20:59:06.974372+0000 osd.5 (osd.5) 177 : cluster 0 3.17 scrub starts 2024-03-11T20:59:08.347 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:08 smithi023 bash[29146]: cluster 2024-03-11T20:59:06.978035+0000 osd.5 (osd.5) 178 : cluster 0 3.17 scrub ok 2024-03-11T20:59:08.426 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:08.453 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:08.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:08.455 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:59:08.535 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:59:08 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:20:59:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T20:59:08.556 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:08.578 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:08.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:08.580 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:59:08.681 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:08.690 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-03-11T20:59:08.690 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-03-11T20:59:08.690 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 30, 2024-03-11T20:59:08.690 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-03-11T20:59:08.690 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-03-11T20:59:08.710 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:08.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:08.713 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:59:08.814 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:08.841 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:08.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:08.844 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:59:08.944 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:08.969 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:08.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:08.972 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:59:09.072 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:09.099 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:09.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:09.102 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:59:09.203 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:09.215 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:09 smithi023 bash[21372]: cluster 2024-03-11T20:59:08.009348+0000 mgr.y (mgr.24380) 311 : cluster 0 pgmap v317: 132 pgs: 47 active+clean, 11 undersized+peered, 43 active+undersized, 12 undersized+degraded+peered, 19 active+undersized+degraded; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 182/594 objects degraded (30.640%) 2024-03-11T20:59:09.215 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:09 smithi023 bash[29146]: cluster 2024-03-11T20:59:08.009348+0000 mgr.y (mgr.24380) 311 : cluster 0 pgmap v317: 132 pgs: 47 active+clean, 11 undersized+peered, 43 active+undersized, 12 undersized+degraded+peered, 19 active+undersized+degraded; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 182/594 objects degraded (30.640%) 2024-03-11T20:59:09.232 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:09.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:09.235 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:59:09.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:09 smithi146 bash[23809]: cluster 2024-03-11T20:59:08.009348+0000 mgr.y (mgr.24380) 311 : cluster 0 pgmap v317: 132 pgs: 47 active+clean, 11 undersized+peered, 43 active+undersized, 12 undersized+degraded+peered, 19 active+undersized+degraded; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 182/594 objects degraded (30.640%) 2024-03-11T20:59:09.337 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:09.356 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:09.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:09.358 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:59:09.459 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:09.479 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:09.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:09.481 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:59:09.582 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:09.608 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:09.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:09.612 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:59:09.612 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-03-11T20:59:09.675 INFO:teuthology.orchestra.run.smithi023.stdout:osd.0: {} 2024-03-11T20:59:09.713 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:09.732 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:09.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:09.735 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:59:09.835 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:09.854 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:09.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:09.856 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:59:09.957 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:09.980 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:09.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:09.982 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:59:10.083 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:10.111 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:10.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:10.113 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:59:10.142 INFO:teuthology.orchestra.run.smithi023.stdout:osd.2: {} 2024-03-11T20:59:10.172 INFO:teuthology.orchestra.run.smithi023.stdout:osd.3: {} 2024-03-11T20:59:10.178 INFO:teuthology.orchestra.run.smithi023.stderr:pool 'unique_pool_1' created 2024-03-11T20:59:10.205 INFO:teuthology.orchestra.run.smithi023.stdout:osd.5: {} 2024-03-11T20:59:10.213 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:10.227 INFO:teuthology.orchestra.run.smithi023.stdout:osd.6: {} 2024-03-11T20:59:10.238 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:10.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:10.240 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:59:10.248 INFO:teuthology.orchestra.run.smithi023.stdout:osd.7: {} 2024-03-11T20:59:10.248 INFO:teuthology.orchestra.run.smithi023.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T20:59:10.248 INFO:teuthology.orchestra.run.smithi023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-11T20:59:10.249 INFO:teuthology.orchestra.run.smithi023.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T20:59:10.249 INFO:teuthology.orchestra.run.smithi023.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T20:59:10.249 INFO:teuthology.orchestra.run.smithi023.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-03-11T20:59:10.249 INFO:teuthology.orchestra.run.smithi023.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T20:59:10.249 INFO:teuthology.orchestra.run.smithi023.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T20:59:10.249 INFO:teuthology.orchestra.run.smithi023.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T20:59:10.341 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:10.359 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:10.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:10.361 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:59:10.463 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:10.474 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:10 smithi023 bash[29146]: audit 2024-03-11T20:59:09.656270+0000 mon.a (mon.0) 1137 : audit 0 from='client.? 172.21.15.23:0/2253637678' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T20:59:10.474 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:10 smithi023 bash[29146]: audit 2024-03-11T20:59:09.676626+0000 mon.a (mon.0) 1138 : audit 1 from='client.? 172.21.15.23:0/3702163692' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-03-11T20:59:10.474 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:10 smithi023 bash[21372]: audit 2024-03-11T20:59:09.656270+0000 mon.a (mon.0) 1137 : audit 0 from='client.? 172.21.15.23:0/2253637678' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T20:59:10.474 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:10 smithi023 bash[21372]: audit 2024-03-11T20:59:09.676626+0000 mon.a (mon.0) 1138 : audit 1 from='client.? 172.21.15.23:0/3702163692' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-03-11T20:59:10.485 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:10.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:10.487 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:59:10.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:10 smithi146 bash[23809]: audit 2024-03-11T20:59:09.656270+0000 mon.a (mon.0) 1137 : audit 0 from='client.? 172.21.15.23:0/2253637678' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T20:59:10.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:10 smithi146 bash[23809]: audit 2024-03-11T20:59:09.676626+0000 mon.a (mon.0) 1138 : audit 1 from='client.? 172.21.15.23:0/3702163692' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-03-11T20:59:10.588 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:10.613 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:10.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:10.615 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:59:10.715 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:10.743 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:10.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:10.745 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:59:10.846 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:10.874 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:10.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:10.877 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:59:10.978 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:11.006 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:11.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:11.009 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:59:11.110 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:11.137 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:11.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:11.140 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:59:11.240 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:11.269 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:11.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:11.272 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:59:11.373 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:11.399 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:11.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:11.401 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:59:11.450 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:11 smithi023 bash[29146]: cluster 2024-03-11T20:59:10.010313+0000 mgr.y (mgr.24380) 312 : cluster 0 pgmap v318: 132 pgs: 47 active+clean, 11 undersized+peered, 43 active+undersized, 12 undersized+degraded+peered, 19 active+undersized+degraded; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 182/594 objects degraded (30.640%) 2024-03-11T20:59:11.450 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:11 smithi023 bash[29146]: audit 2024-03-11T20:59:10.171218+0000 mon.a (mon.0) 1139 : audit 1 from='client.? 172.21.15.23:0/3702163692' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-03-11T20:59:11.451 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:11 smithi023 bash[29146]: cluster 2024-03-11T20:59:10.178598+0000 mon.a (mon.0) 1140 : cluster 0 osdmap e189: 8 total, 6 up, 8 in 2024-03-11T20:59:11.451 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:11 smithi023 bash[29146]: cluster 2024-03-11T20:59:10.440759+0000 osd.3 (osd.3) 216 : cluster 0 6.0 scrub starts 2024-03-11T20:59:11.451 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:11 smithi023 bash[29146]: cluster 2024-03-11T20:59:10.448838+0000 osd.3 (osd.3) 217 : cluster 0 6.0 scrub ok 2024-03-11T20:59:11.451 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:11 smithi023 bash[21372]: cluster 2024-03-11T20:59:10.010313+0000 mgr.y (mgr.24380) 312 : cluster 0 pgmap v318: 132 pgs: 47 active+clean, 11 undersized+peered, 43 active+undersized, 12 undersized+degraded+peered, 19 active+undersized+degraded; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 182/594 objects degraded (30.640%) 2024-03-11T20:59:11.451 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:11 smithi023 bash[21372]: audit 2024-03-11T20:59:10.171218+0000 mon.a (mon.0) 1139 : audit 1 from='client.? 172.21.15.23:0/3702163692' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-03-11T20:59:11.451 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:11 smithi023 bash[21372]: cluster 2024-03-11T20:59:10.178598+0000 mon.a (mon.0) 1140 : cluster 0 osdmap e189: 8 total, 6 up, 8 in 2024-03-11T20:59:11.451 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:11 smithi023 bash[21372]: cluster 2024-03-11T20:59:10.440759+0000 osd.3 (osd.3) 216 : cluster 0 6.0 scrub starts 2024-03-11T20:59:11.451 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:11 smithi023 bash[21372]: cluster 2024-03-11T20:59:10.448838+0000 osd.3 (osd.3) 217 : cluster 0 6.0 scrub ok 2024-03-11T20:59:11.502 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:11.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:11 smithi146 bash[23809]: cluster 2024-03-11T20:59:10.010313+0000 mgr.y (mgr.24380) 312 : cluster 0 pgmap v318: 132 pgs: 47 active+clean, 11 undersized+peered, 43 active+undersized, 12 undersized+degraded+peered, 19 active+undersized+degraded; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 182/594 objects degraded (30.640%) 2024-03-11T20:59:11.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:11 smithi146 bash[23809]: audit 2024-03-11T20:59:10.171218+0000 mon.a (mon.0) 1139 : audit 1 from='client.? 172.21.15.23:0/3702163692' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-03-11T20:59:11.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:11 smithi146 bash[23809]: cluster 2024-03-11T20:59:10.178598+0000 mon.a (mon.0) 1140 : cluster 0 osdmap e189: 8 total, 6 up, 8 in 2024-03-11T20:59:11.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:11 smithi146 bash[23809]: cluster 2024-03-11T20:59:10.440759+0000 osd.3 (osd.3) 216 : cluster 0 6.0 scrub starts 2024-03-11T20:59:11.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:11 smithi146 bash[23809]: cluster 2024-03-11T20:59:10.448838+0000 osd.3 (osd.3) 217 : cluster 0 6.0 scrub ok 2024-03-11T20:59:11.530 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:11.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:11.533 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:59:11.634 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:11.662 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:11.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:11.664 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:59:11.764 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:11.792 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:11.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:11.795 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:59:11.896 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:11.925 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:11.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:11.928 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:59:12.028 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:12.057 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:12.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:12.059 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:59:12.160 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:12.190 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:12.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:12.193 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:59:12.293 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:12.321 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:12.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:12.324 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:59:12.424 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:12.429 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:12 smithi023 bash[21372]: cluster 2024-03-11T20:59:11.186957+0000 mon.a (mon.0) 1141 : cluster 0 osdmap e190: 8 total, 6 up, 8 in 2024-03-11T20:59:12.430 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:12 smithi023 bash[29146]: cluster 2024-03-11T20:59:11.186957+0000 mon.a (mon.0) 1141 : cluster 0 osdmap e190: 8 total, 6 up, 8 in 2024-03-11T20:59:12.448 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:12.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:12.450 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:59:12.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:12 smithi146 bash[23809]: cluster 2024-03-11T20:59:11.186957+0000 mon.a (mon.0) 1141 : cluster 0 osdmap e190: 8 total, 6 up, 8 in 2024-03-11T20:59:12.551 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:12.578 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:12.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:12.580 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:59:12.681 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:12.710 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:12.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:12.712 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:59:12.802 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-11T20:59:12.802 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-11T20:59:12.814 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:12.822 INFO:teuthology.orchestra.run.smithi023.stdout:7.b 2024-03-11T20:59:12.822 INFO:teuthology.orchestra.run.smithi023.stdout:6.e 2024-03-11T20:59:12.823 INFO:teuthology.orchestra.run.smithi023.stdout:6.b 2024-03-11T20:59:12.823 INFO:teuthology.orchestra.run.smithi023.stdout:6.4 2024-03-11T20:59:12.823 INFO:teuthology.orchestra.run.smithi023.stdout:6.1 2024-03-11T20:59:12.823 INFO:teuthology.orchestra.run.smithi023.stdout:6.18 2024-03-11T20:59:12.823 INFO:teuthology.orchestra.run.smithi023.stdout:6.10 2024-03-11T20:59:12.823 INFO:teuthology.orchestra.run.smithi023.stdout:4.f 2024-03-11T20:59:12.823 INFO:teuthology.orchestra.run.smithi023.stdout:4.e 2024-03-11T20:59:12.823 INFO:teuthology.orchestra.run.smithi023.stdout:4.d 2024-03-11T20:59:12.823 INFO:teuthology.orchestra.run.smithi023.stdout:4.9 2024-03-11T20:59:12.823 INFO:teuthology.orchestra.run.smithi023.stdout:3.1a 2024-03-11T20:59:12.823 INFO:teuthology.orchestra.run.smithi023.stdout:3.19 2024-03-11T20:59:12.823 INFO:teuthology.orchestra.run.smithi023.stdout:3.1c 2024-03-11T20:59:12.823 INFO:teuthology.orchestra.run.smithi023.stdout:3.4 2024-03-11T20:59:12.823 INFO:teuthology.orchestra.run.smithi023.stdout:3.18 2024-03-11T20:59:12.823 INFO:teuthology.orchestra.run.smithi023.stdout:2.9 2024-03-11T20:59:12.824 INFO:teuthology.orchestra.run.smithi023.stdout:2.d 2024-03-11T20:59:12.824 INFO:teuthology.orchestra.run.smithi023.stdout:7.9 2024-03-11T20:59:12.824 INFO:teuthology.orchestra.run.smithi023.stdout:3.8 2024-03-11T20:59:12.824 INFO:teuthology.orchestra.run.smithi023.stdout:2.8 2024-03-11T20:59:12.824 INFO:teuthology.orchestra.run.smithi023.stdout:2.0 2024-03-11T20:59:12.824 INFO:teuthology.orchestra.run.smithi023.stdout:3.a 2024-03-11T20:59:12.824 INFO:teuthology.orchestra.run.smithi023.stdout:4.1c 2024-03-11T20:59:12.824 INFO:teuthology.orchestra.run.smithi023.stdout:7.2 2024-03-11T20:59:12.824 INFO:teuthology.orchestra.run.smithi023.stdout:2.6 2024-03-11T20:59:12.824 INFO:teuthology.orchestra.run.smithi023.stdout:2.10 2024-03-11T20:59:12.824 INFO:teuthology.orchestra.run.smithi023.stdout:6.19 2024-03-11T20:59:12.824 INFO:teuthology.orchestra.run.smithi023.stdout:4.18 2024-03-11T20:59:12.824 INFO:teuthology.orchestra.run.smithi023.stdout:7.0 2024-03-11T20:59:12.824 INFO:teuthology.orchestra.run.smithi023.stdout:4.7 2024-03-11T20:59:12.824 INFO:teuthology.orchestra.run.smithi023.stdout:3.0 2024-03-11T20:59:12.825 INFO:teuthology.orchestra.run.smithi023.stdout:4.b 2024-03-11T20:59:12.825 INFO:teuthology.orchestra.run.smithi023.stdout:2.11 2024-03-11T20:59:12.825 INFO:teuthology.orchestra.run.smithi023.stdout:7.4 2024-03-11T20:59:12.825 INFO:teuthology.orchestra.run.smithi023.stdout:4.10 2024-03-11T20:59:12.825 INFO:teuthology.orchestra.run.smithi023.stdout:6.d 2024-03-11T20:59:12.825 INFO:teuthology.orchestra.run.smithi023.stdout:2.15 2024-03-11T20:59:12.825 INFO:teuthology.orchestra.run.smithi023.stdout:4.a 2024-03-11T20:59:12.825 INFO:teuthology.orchestra.run.smithi023.stdout:2.a 2024-03-11T20:59:12.825 INFO:teuthology.orchestra.run.smithi023.stdout:3.16 2024-03-11T20:59:12.825 INFO:teuthology.orchestra.run.smithi023.stdout:2.4 2024-03-11T20:59:12.825 INFO:teuthology.orchestra.run.smithi023.stdout:3.e 2024-03-11T20:59:12.825 INFO:teuthology.orchestra.run.smithi023.stdout:6.1a 2024-03-11T20:59:12.825 INFO:teuthology.orchestra.run.smithi023.stdout:3.6 2024-03-11T20:59:12.825 INFO:teuthology.orchestra.run.smithi023.stdout:4.12 2024-03-11T20:59:12.826 INFO:teuthology.orchestra.run.smithi023.stdout:4.2 2024-03-11T20:59:12.826 INFO:teuthology.orchestra.run.smithi023.stdout:4.13 2024-03-11T20:59:12.826 INFO:teuthology.orchestra.run.smithi023.stdout:7.d 2024-03-11T20:59:12.826 INFO:teuthology.orchestra.run.smithi023.stdout:6.1d 2024-03-11T20:59:12.826 INFO:teuthology.orchestra.run.smithi023.stdout:4.17 2024-03-11T20:59:12.826 INFO:teuthology.orchestra.run.smithi023.stdout:4.14 2024-03-11T20:59:12.826 INFO:teuthology.orchestra.run.smithi023.stdout:4.1b 2024-03-11T20:59:12.826 INFO:teuthology.orchestra.run.smithi023.stdout:4.1d 2024-03-11T20:59:12.826 INFO:teuthology.orchestra.run.smithi023.stdout:5.0 2024-03-11T20:59:12.826 INFO:teuthology.orchestra.run.smithi023.stdout:4.1f 2024-03-11T20:59:12.826 INFO:teuthology.orchestra.run.smithi023.stdout:4.4 2024-03-11T20:59:12.826 INFO:teuthology.orchestra.run.smithi023.stdout:2.2 2024-03-11T20:59:12.826 INFO:teuthology.orchestra.run.smithi023.stdout:4.6 2024-03-11T20:59:12.835 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd pool application enable unique_pool_1 rados --yes-i-really-mean-it || true 2024-03-11T20:59:12.842 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:12.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:12.845 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:59:12.946 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:12.973 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:12.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:12.975 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:59:13.076 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:13.103 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:13.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:13.105 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:59:13.206 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:13.225 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:13.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:13.227 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:59:13.328 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:13.339 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:13 smithi146 bash[23809]: cluster 2024-03-11T20:59:12.011109+0000 mgr.y (mgr.24380) 313 : cluster 0 pgmap v321: 148 pgs: 8 creating+peering, 8 unknown, 47 active+clean, 11 undersized+peered, 43 active+undersized, 12 undersized+degraded+peered, 19 active+undersized+degraded; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 182/594 objects degraded (30.640%) 2024-03-11T20:59:13.339 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:13 smithi146 bash[23809]: cluster 2024-03-11T20:59:12.191372+0000 mon.a (mon.0) 1142 : cluster 3 Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:59:13.339 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:13 smithi146 bash[23809]: cluster 2024-03-11T20:59:12.203708+0000 mon.a (mon.0) 1143 : cluster 0 osdmap e191: 8 total, 6 up, 8 in 2024-03-11T20:59:13.355 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:13.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:13.357 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:59:13.458 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:13.466 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:13 smithi023 bash[29146]: cluster 2024-03-11T20:59:12.011109+0000 mgr.y (mgr.24380) 313 : cluster 0 pgmap v321: 148 pgs: 8 creating+peering, 8 unknown, 47 active+clean, 11 undersized+peered, 43 active+undersized, 12 undersized+degraded+peered, 19 active+undersized+degraded; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 182/594 objects degraded (30.640%) 2024-03-11T20:59:13.467 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:13 smithi023 bash[29146]: cluster 2024-03-11T20:59:12.191372+0000 mon.a (mon.0) 1142 : cluster 3 Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:59:13.467 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:13 smithi023 bash[29146]: cluster 2024-03-11T20:59:12.203708+0000 mon.a (mon.0) 1143 : cluster 0 osdmap e191: 8 total, 6 up, 8 in 2024-03-11T20:59:13.467 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:13 smithi023 bash[21372]: cluster 2024-03-11T20:59:12.011109+0000 mgr.y (mgr.24380) 313 : cluster 0 pgmap v321: 148 pgs: 8 creating+peering, 8 unknown, 47 active+clean, 11 undersized+peered, 43 active+undersized, 12 undersized+degraded+peered, 19 active+undersized+degraded; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 182/594 objects degraded (30.640%) 2024-03-11T20:59:13.467 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:13 smithi023 bash[21372]: cluster 2024-03-11T20:59:12.191372+0000 mon.a (mon.0) 1142 : cluster 3 Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:59:13.467 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:13 smithi023 bash[21372]: cluster 2024-03-11T20:59:12.203708+0000 mon.a (mon.0) 1143 : cluster 0 osdmap e191: 8 total, 6 up, 8 in 2024-03-11T20:59:13.481 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:13.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:13.483 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:59:13.584 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:13.612 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:13.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:13.615 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:59:13.716 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:13.745 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:13.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:13.748 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:59:13.849 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:13.876 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:13.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:13.878 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:59:13.978 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:14.005 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:14.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:14.008 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:59:14.108 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:14.135 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:14.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:14.138 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:59:14.219 INFO:teuthology.orchestra.run.smithi023.stderr:noscrub is set 2024-03-11T20:59:14.238 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:14.246 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:14 smithi023 bash[29146]: audit 2024-03-11T20:59:13.218041+0000 mon.c (mon.2) 41 : audit 1 from='client.? 172.21.15.23:0/406041358' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T20:59:14.247 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:14 smithi023 bash[29146]: audit 2024-03-11T20:59:13.218542+0000 mon.a (mon.0) 1144 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T20:59:14.247 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:14 smithi023 bash[29146]: cluster 2024-03-11T20:59:13.514822+0000 osd.3 (osd.3) 218 : cluster 0 4.11 scrub starts 2024-03-11T20:59:14.247 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:14 smithi023 bash[29146]: cluster 2024-03-11T20:59:13.516315+0000 osd.3 (osd.3) 219 : cluster 0 4.11 scrub ok 2024-03-11T20:59:14.247 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:14 smithi023 bash[21372]: audit 2024-03-11T20:59:13.218041+0000 mon.c (mon.2) 41 : audit 1 from='client.? 172.21.15.23:0/406041358' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T20:59:14.247 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:14 smithi023 bash[21372]: audit 2024-03-11T20:59:13.218542+0000 mon.a (mon.0) 1144 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T20:59:14.247 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:14 smithi023 bash[21372]: cluster 2024-03-11T20:59:13.514822+0000 osd.3 (osd.3) 218 : cluster 0 4.11 scrub starts 2024-03-11T20:59:14.247 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:14 smithi023 bash[21372]: cluster 2024-03-11T20:59:13.516315+0000 osd.3 (osd.3) 219 : cluster 0 4.11 scrub ok 2024-03-11T20:59:14.261 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:14.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:14.264 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:59:14.269 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -n osd.1 --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-1 --journal-path /var/lib/ceph/osd/ceph-1/journal --op export --pgid 6.1a --file /var/log/ceph/exp.6.1a.1 2024-03-11T20:59:14.365 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:14.376 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:14 smithi146 bash[23809]: audit 2024-03-11T20:59:13.218041+0000 mon.c (mon.2) 41 : audit 1 from='client.? 172.21.15.23:0/406041358' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T20:59:14.376 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:14 smithi146 bash[23809]: audit 2024-03-11T20:59:13.218542+0000 mon.a (mon.0) 1144 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T20:59:14.377 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:14 smithi146 bash[23809]: cluster 2024-03-11T20:59:13.514822+0000 osd.3 (osd.3) 218 : cluster 0 4.11 scrub starts 2024-03-11T20:59:14.377 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:14 smithi146 bash[23809]: cluster 2024-03-11T20:59:13.516315+0000 osd.3 (osd.3) 219 : cluster 0 4.11 scrub ok 2024-03-11T20:59:14.391 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:14.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:14.393 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:59:14.405 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T20:59:14.474 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/osd.1/config 2024-03-11T20:59:14.494 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:14.521 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:14.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:14.523 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:59:14.624 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:14.653 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:14.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:14.656 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:59:14.757 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:14.783 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:14.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:14.785 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:59:14.803 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T20:59:14.886 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:14.914 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:14.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:14.916 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:59:15.017 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:15.045 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:15.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:15.047 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:59:15.148 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:15.174 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:15.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:15.176 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:59:15.277 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:15.305 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:15.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:15.307 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:59:15.409 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:15.435 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:15.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:15.437 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:59:15.511 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 20:59:15 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T20:59:15.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:15 smithi146 bash[23809]: cluster 2024-03-11T20:59:14.011890+0000 mgr.y (mgr.24380) 314 : cluster 0 pgmap v323: 148 pgs: 8 creating+peering, 8 unknown, 47 active+clean, 11 undersized+peered, 43 active+undersized, 12 undersized+degraded+peered, 19 active+undersized+degraded; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 182/594 objects degraded (30.640%) 2024-03-11T20:59:15.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:15 smithi146 bash[23809]: cluster 2024-03-11T20:59:14.197208+0000 mon.a (mon.0) 1145 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T20:59:15.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:15 smithi146 bash[23809]: audit 2024-03-11T20:59:14.209830+0000 mon.a (mon.0) 1146 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T20:59:15.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:15 smithi146 bash[23809]: cluster 2024-03-11T20:59:14.217897+0000 mon.a (mon.0) 1147 : cluster 0 osdmap e192: 8 total, 6 up, 8 in 2024-03-11T20:59:15.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:15 smithi146 bash[23809]: cluster 2024-03-11T20:59:14.602630+0000 osd.7 (osd.7) 185 : cluster 0 4.1e deep-scrub starts 2024-03-11T20:59:15.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:15 smithi146 bash[23809]: cluster 2024-03-11T20:59:14.605671+0000 osd.7 (osd.7) 186 : cluster 0 4.1e deep-scrub ok 2024-03-11T20:59:15.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:15 smithi023 bash[21372]: cluster 2024-03-11T20:59:14.011890+0000 mgr.y (mgr.24380) 314 : cluster 0 pgmap v323: 148 pgs: 8 creating+peering, 8 unknown, 47 active+clean, 11 undersized+peered, 43 active+undersized, 12 undersized+degraded+peered, 19 active+undersized+degraded; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 182/594 objects degraded (30.640%) 2024-03-11T20:59:15.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:15 smithi023 bash[21372]: cluster 2024-03-11T20:59:14.197208+0000 mon.a (mon.0) 1145 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T20:59:15.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:15 smithi023 bash[21372]: audit 2024-03-11T20:59:14.209830+0000 mon.a (mon.0) 1146 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T20:59:15.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:15 smithi023 bash[21372]: cluster 2024-03-11T20:59:14.217897+0000 mon.a (mon.0) 1147 : cluster 0 osdmap e192: 8 total, 6 up, 8 in 2024-03-11T20:59:15.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:15 smithi023 bash[21372]: cluster 2024-03-11T20:59:14.602630+0000 osd.7 (osd.7) 185 : cluster 0 4.1e deep-scrub starts 2024-03-11T20:59:15.536 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:15 smithi023 bash[21372]: cluster 2024-03-11T20:59:14.605671+0000 osd.7 (osd.7) 186 : cluster 0 4.1e deep-scrub ok 2024-03-11T20:59:15.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:15 smithi023 bash[29146]: cluster 2024-03-11T20:59:14.011890+0000 mgr.y (mgr.24380) 314 : cluster 0 pgmap v323: 148 pgs: 8 creating+peering, 8 unknown, 47 active+clean, 11 undersized+peered, 43 active+undersized, 12 undersized+degraded+peered, 19 active+undersized+degraded; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 182/594 objects degraded (30.640%) 2024-03-11T20:59:15.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:15 smithi023 bash[29146]: cluster 2024-03-11T20:59:14.197208+0000 mon.a (mon.0) 1145 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T20:59:15.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:15 smithi023 bash[29146]: audit 2024-03-11T20:59:14.209830+0000 mon.a (mon.0) 1146 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T20:59:15.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:15 smithi023 bash[29146]: cluster 2024-03-11T20:59:14.217897+0000 mon.a (mon.0) 1147 : cluster 0 osdmap e192: 8 total, 6 up, 8 in 2024-03-11T20:59:15.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:15 smithi023 bash[29146]: cluster 2024-03-11T20:59:14.602630+0000 osd.7 (osd.7) 185 : cluster 0 4.1e deep-scrub starts 2024-03-11T20:59:15.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:15 smithi023 bash[29146]: cluster 2024-03-11T20:59:14.605671+0000 osd.7 (osd.7) 186 : cluster 0 4.1e deep-scrub ok 2024-03-11T20:59:15.537 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:15.564 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:15.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:15.566 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:59:15.667 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:15.697 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:15.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:15.699 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:59:15.800 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:15.822 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:15.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:15.824 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:59:15.925 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:15.947 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:15.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:15.949 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:59:16.049 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:16.070 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:16.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:16.072 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:59:16.173 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:16.195 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:16.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:16.197 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:59:16.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:16 smithi023 bash[21372]: audit 2024-03-11T20:59:15.217747+0000 mgr.y (mgr.24380) 315 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:59:16.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:16 smithi023 bash[29146]: audit 2024-03-11T20:59:15.217747+0000 mgr.y (mgr.24380) 315 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:59:16.298 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:16.324 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:16.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:16.327 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:59:16.428 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:16.457 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:16.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:16.460 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:59:16.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:16 smithi146 bash[23809]: audit 2024-03-11T20:59:15.217747+0000 mgr.y (mgr.24380) 315 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:59:16.561 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:16.587 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:16.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:16.590 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:59:16.690 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:16.713 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:16.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:16.715 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:59:16.816 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:16.843 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:16.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:16.846 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:59:16.946 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:16.974 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:16.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:16.977 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:59:16.994 INFO:teuthology.orchestra.run.smithi023.stderr:Creating an OSD daemon form without an OSD FSID value 2024-03-11T20:59:17.079 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:17.105 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:17.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:17.107 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:59:17.208 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:17.234 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:17.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:17.236 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:59:17.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:17 smithi023 bash[21372]: cluster 2024-03-11T20:59:16.013194+0000 mgr.y (mgr.24380) 316 : cluster 0 pgmap v325: 148 pgs: 52 active+clean, 12 undersized+peered, 53 active+undersized, 12 undersized+degraded+peered, 19 active+undersized+degraded; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 182/594 objects degraded (30.640%) 2024-03-11T20:59:17.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:17 smithi023 bash[29146]: cluster 2024-03-11T20:59:16.013194+0000 mgr.y (mgr.24380) 316 : cluster 0 pgmap v325: 148 pgs: 52 active+clean, 12 undersized+peered, 53 active+undersized, 12 undersized+degraded+peered, 19 active+undersized+degraded; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 182/594 objects degraded (30.640%) 2024-03-11T20:59:17.337 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:17.366 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:17.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:17.368 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:59:17.469 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:17.480 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:17 smithi146 bash[23809]: cluster 2024-03-11T20:59:16.013194+0000 mgr.y (mgr.24380) 316 : cluster 0 pgmap v325: 148 pgs: 52 active+clean, 12 undersized+peered, 53 active+undersized, 12 undersized+degraded+peered, 19 active+undersized+degraded; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 182/594 objects degraded (30.640%) 2024-03-11T20:59:17.495 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:17.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:17.497 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:59:17.598 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:17.626 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:17.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:17.628 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:59:17.635 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-11T20:59:17.729 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:17.756 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:17.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:17.759 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:59:17.798 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:59:17.818 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-03-11T20:59:17.818 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T20:59:17.860 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:17.881 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:17.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:17.883 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:59:17.986 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:18.012 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:18.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:18.015 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:59:18.116 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:18.142 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:18.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:18.145 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:59:18.166 INFO:teuthology.orchestra.run.smithi023.stderr:Exporting 6.1a info 6.1a( empty local-lis/les=162/163 n=0 ec=82/82 lis/c=162/162 les/c/f=163/163/0 sis=162) 2024-03-11T20:59:18.167 INFO:teuthology.orchestra.run.smithi023.stderr:Export successful 2024-03-11T20:59:18.246 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:18.270 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:18.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:18.272 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:59:18.373 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:18.384 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:59:18 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:20:59:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T20:59:18.401 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:18.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:18.404 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:59:18.504 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:18.530 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:18.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:18.532 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:59:18.632 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:18.645 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-11T20:59:18.658 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:18.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:18.661 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:59:18.762 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:18.790 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:18.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:18.793 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:59:18.893 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:18.913 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:18.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:18.916 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:59:19.017 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:19.042 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:19.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:19.044 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:59:19.145 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:19.170 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:19.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:19.173 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:59:19.273 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:19.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:19 smithi023 bash[21372]: cluster 2024-03-11T20:59:18.014149+0000 mgr.y (mgr.24380) 317 : cluster 0 pgmap v326: 148 pgs: 52 active+clean, 12 undersized+peered, 53 active+undersized, 12 undersized+degraded+peered, 19 active+undersized+degraded; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 182/594 objects degraded (30.640%) 2024-03-11T20:59:19.285 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:19 smithi146 bash[23809]: cluster 2024-03-11T20:59:18.014149+0000 mgr.y (mgr.24380) 317 : cluster 0 pgmap v326: 148 pgs: 52 active+clean, 12 undersized+peered, 53 active+undersized, 12 undersized+degraded+peered, 19 active+undersized+degraded; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 182/594 objects degraded (30.640%) 2024-03-11T20:59:19.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:19 smithi023 bash[29146]: cluster 2024-03-11T20:59:18.014149+0000 mgr.y (mgr.24380) 317 : cluster 0 pgmap v326: 148 pgs: 52 active+clean, 12 undersized+peered, 53 active+undersized, 12 undersized+degraded+peered, 19 active+undersized+degraded; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 182/594 objects degraded (30.640%) 2024-03-11T20:59:19.301 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:19.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:19.303 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:59:19.404 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:19.429 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:19.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:19.432 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:59:19.532 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:19.559 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:19.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:19.561 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:59:19.662 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:19.689 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:19.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:19.692 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:59:19.793 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:19.819 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:19.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:19.821 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:59:19.921 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:19.948 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:19.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:19.950 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:59:19.989 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -n osd.1 --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-1 --journal-path /var/lib/ceph/osd/ceph-1/journal --force --op remove --pgid 6.1a 2024-03-11T20:59:20.051 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:20.076 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:20.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:20.079 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:59:20.163 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/osd.1/config 2024-03-11T20:59:20.180 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:20.199 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:20.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:20.201 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:59:20.302 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:20.329 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:20.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:20.331 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:59:20.432 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:20.461 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:20.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:20.464 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:59:20.565 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:20.594 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:20.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:20.596 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:59:20.697 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:20.722 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:20.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:20.724 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:59:20.825 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:20.852 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:20.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:20.855 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:59:20.956 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:20.984 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:20.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:20.987 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:59:21.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:20 smithi146 bash[23809]: audit 2024-03-11T20:59:19.745818+0000 mon.a (mon.0) 1148 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:59:21.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:20 smithi146 bash[23809]: audit 2024-03-11T20:59:19.748316+0000 mon.a (mon.0) 1149 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:59:21.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:20 smithi146 bash[23809]: cluster 2024-03-11T20:59:20.015989+0000 mgr.y (mgr.24380) 318 : cluster 0 pgmap v327: 148 pgs: 52 active+clean, 12 undersized+peered, 53 active+undersized, 12 undersized+degraded+peered, 19 active+undersized+degraded; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 182/594 objects degraded (30.640%) 2024-03-11T20:59:21.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:20 smithi023 bash[29146]: audit 2024-03-11T20:59:19.745818+0000 mon.a (mon.0) 1148 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:59:21.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:20 smithi023 bash[29146]: audit 2024-03-11T20:59:19.748316+0000 mon.a (mon.0) 1149 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:59:21.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:20 smithi023 bash[29146]: cluster 2024-03-11T20:59:20.015989+0000 mgr.y (mgr.24380) 318 : cluster 0 pgmap v327: 148 pgs: 52 active+clean, 12 undersized+peered, 53 active+undersized, 12 undersized+degraded+peered, 19 active+undersized+degraded; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 182/594 objects degraded (30.640%) 2024-03-11T20:59:21.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:20 smithi023 bash[21372]: audit 2024-03-11T20:59:19.745818+0000 mon.a (mon.0) 1148 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:59:21.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:20 smithi023 bash[21372]: audit 2024-03-11T20:59:19.748316+0000 mon.a (mon.0) 1149 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:59:21.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:20 smithi023 bash[21372]: cluster 2024-03-11T20:59:20.015989+0000 mgr.y (mgr.24380) 318 : cluster 0 pgmap v327: 148 pgs: 52 active+clean, 12 undersized+peered, 53 active+undersized, 12 undersized+degraded+peered, 19 active+undersized+degraded; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 182/594 objects degraded (30.640%) 2024-03-11T20:59:21.088 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:21.110 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:21.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:21.112 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:59:21.213 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:21.243 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:21.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:21.245 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:59:21.345 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:21.374 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:21.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:21.377 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:59:21.478 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:21.506 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:21.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:21.509 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:59:21.541 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:59:21.609 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:21.638 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:21.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:21.640 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:59:21.740 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:21.770 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:21.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:21.772 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:59:21.873 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:21.898 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:21.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:21.901 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:59:22.002 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:22.029 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:22.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:22.032 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:59:22.132 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:22.160 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:22.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:22.163 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:59:22.264 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:22.290 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:22.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:22.293 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:59:22.393 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:22.420 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:22.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:22.422 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:59:22.523 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:22.549 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:22.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:22.551 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:59:22.652 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:22.680 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:22.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:22.683 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:59:22.783 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:22.811 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:22.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:22.814 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:59:22.914 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:22.940 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:22.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:22.942 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:59:23.043 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:23.069 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:23.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:23.071 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:59:23.172 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:23.181 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:23 smithi023 bash[21372]: cluster 2024-03-11T20:59:22.017023+0000 mgr.y (mgr.24380) 319 : cluster 0 pgmap v328: 148 pgs: 52 active+clean, 12 undersized+peered, 53 active+undersized, 12 undersized+degraded+peered, 19 active+undersized+degraded; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 182/594 objects degraded (30.640%) 2024-03-11T20:59:23.181 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:23 smithi023 bash[29146]: cluster 2024-03-11T20:59:22.017023+0000 mgr.y (mgr.24380) 319 : cluster 0 pgmap v328: 148 pgs: 52 active+clean, 12 undersized+peered, 53 active+undersized, 12 undersized+degraded+peered, 19 active+undersized+degraded; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 182/594 objects degraded (30.640%) 2024-03-11T20:59:23.194 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:23.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:23.196 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:59:23.297 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:23.323 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:23.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:23.326 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:59:23.427 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:23.451 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:23.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:23.453 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:59:23.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:23 smithi146 bash[23809]: cluster 2024-03-11T20:59:22.017023+0000 mgr.y (mgr.24380) 319 : cluster 0 pgmap v328: 148 pgs: 52 active+clean, 12 undersized+peered, 53 active+undersized, 12 undersized+degraded+peered, 19 active+undersized+degraded; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 182/594 objects degraded (30.640%) 2024-03-11T20:59:23.514 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:59:23.514 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:59:23.555 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:23.578 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:23.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:23.580 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:59:23.681 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:23.708 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:23.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:23.710 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:59:23.811 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:23.839 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:23.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:23.841 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:59:23.942 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:23.970 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:23.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:23.972 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:59:24.073 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:24.100 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:24.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:24.102 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:59:24.203 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:24.230 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:24.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:24.232 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:59:24.333 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:24.359 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:24.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:24.361 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:59:24.462 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:24.490 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:24.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:24.492 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:59:24.593 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:24.619 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:24.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:24.621 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:59:24.722 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:24.750 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:24.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:24.752 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:59:24.853 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:24.881 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:24.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:24.883 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:59:24.983 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:25.010 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:25.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:25.012 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:59:25.103 INFO:teuthology.orchestra.run.smithi023.stderr:enabled application 'rados' on pool 'unique_pool_1' 2024-03-11T20:59:25.113 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:25.144 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:25.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:25.147 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:59:25.248 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:25.275 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:25.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:25.277 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:59:25.378 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:25.388 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 20:59:25 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T20:59:25.389 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:25 smithi146 bash[23809]: cluster 2024-03-11T20:59:24.018491+0000 mgr.y (mgr.24380) 320 : cluster 0 pgmap v329: 148 pgs: 52 active+clean, 12 undersized+peered, 53 active+undersized, 12 undersized+degraded+peered, 19 active+undersized+degraded; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 182/594 objects degraded (30.640%) 2024-03-11T20:59:25.389 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:25 smithi146 bash[23809]: audit 2024-03-11T20:59:24.166831+0000 mon.c (mon.2) 42 : audit 1 from='client.? 172.21.15.23:0/1950470583' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:59:25.389 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:25 smithi146 bash[23809]: audit 2024-03-11T20:59:24.167180+0000 mon.a (mon.0) 1150 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:59:25.404 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:25.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:25.406 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:59:25.507 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:25.535 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:25.539 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:25 smithi023 bash[21372]: cluster 2024-03-11T20:59:24.018491+0000 mgr.y (mgr.24380) 320 : cluster 0 pgmap v329: 148 pgs: 52 active+clean, 12 undersized+peered, 53 active+undersized, 12 undersized+degraded+peered, 19 active+undersized+degraded; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 182/594 objects degraded (30.640%) 2024-03-11T20:59:25.539 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:25 smithi023 bash[21372]: audit 2024-03-11T20:59:24.166831+0000 mon.c (mon.2) 42 : audit 1 from='client.? 172.21.15.23:0/1950470583' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:59:25.539 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:25 smithi023 bash[21372]: audit 2024-03-11T20:59:24.167180+0000 mon.a (mon.0) 1150 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:59:25.540 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:25 smithi023 bash[29146]: cluster 2024-03-11T20:59:24.018491+0000 mgr.y (mgr.24380) 320 : cluster 0 pgmap v329: 148 pgs: 52 active+clean, 12 undersized+peered, 53 active+undersized, 12 undersized+degraded+peered, 19 active+undersized+degraded; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 182/594 objects degraded (30.640%) 2024-03-11T20:59:25.540 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:25 smithi023 bash[29146]: audit 2024-03-11T20:59:24.166831+0000 mon.c (mon.2) 42 : audit 1 from='client.? 172.21.15.23:0/1950470583' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:59:25.540 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:25 smithi023 bash[29146]: audit 2024-03-11T20:59:24.167180+0000 mon.a (mon.0) 1150 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:59:25.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:25.541 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:59:25.642 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:25.670 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:25.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:25.672 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:59:25.773 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:25.803 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:25.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:25.805 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:59:25.894 INFO:teuthology.orchestra.run.smithi023.stderr:Creating an OSD daemon form without an OSD FSID value 2024-03-11T20:59:25.906 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:25.927 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:25.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:25.929 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:59:26.030 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:26.059 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:26.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:26.061 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:59:26.162 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:26.189 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:26.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:26.192 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:59:26.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:26 smithi023 bash[21372]: audit 2024-03-11T20:59:25.091587+0000 mon.a (mon.0) 1151 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-11T20:59:26.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:26 smithi023 bash[21372]: cluster 2024-03-11T20:59:25.097025+0000 mon.a (mon.0) 1152 : cluster 0 osdmap e193: 8 total, 6 up, 8 in 2024-03-11T20:59:26.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:26 smithi023 bash[21372]: audit 2024-03-11T20:59:25.229253+0000 mgr.y (mgr.24380) 321 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:59:26.286 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:26 smithi023 bash[29146]: audit 2024-03-11T20:59:25.091587+0000 mon.a (mon.0) 1151 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-11T20:59:26.286 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:26 smithi023 bash[29146]: cluster 2024-03-11T20:59:25.097025+0000 mon.a (mon.0) 1152 : cluster 0 osdmap e193: 8 total, 6 up, 8 in 2024-03-11T20:59:26.286 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:26 smithi023 bash[29146]: audit 2024-03-11T20:59:25.229253+0000 mgr.y (mgr.24380) 321 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:59:26.293 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:26.321 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:26.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:26.323 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:59:26.424 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:26.454 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:26.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:26.456 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:59:26.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:26 smithi146 bash[23809]: audit 2024-03-11T20:59:25.091587+0000 mon.a (mon.0) 1151 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-11T20:59:26.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:26 smithi146 bash[23809]: cluster 2024-03-11T20:59:25.097025+0000 mon.a (mon.0) 1152 : cluster 0 osdmap e193: 8 total, 6 up, 8 in 2024-03-11T20:59:26.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:26 smithi146 bash[23809]: audit 2024-03-11T20:59:25.229253+0000 mgr.y (mgr.24380) 321 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:59:26.557 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:26.583 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:26.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:26.585 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:59:26.685 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:26.712 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:26.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:26.714 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:59:26.815 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:26.843 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:26.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:26.845 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:59:26.946 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:26.974 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:26.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:26.976 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:59:27.077 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:27.104 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:27.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:27.106 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:59:27.207 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:27.231 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:27.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:27.234 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:59:27.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:27 smithi146 bash[23809]: cluster 2024-03-11T20:59:26.019716+0000 mgr.y (mgr.24380) 322 : cluster 0 pgmap v331: 148 pgs: 52 active+clean, 12 undersized+peered, 53 active+undersized, 12 undersized+degraded+peered, 19 active+undersized+degraded; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 182/594 objects degraded (30.640%) 2024-03-11T20:59:27.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:27 smithi146 bash[23809]: cluster 2024-03-11T20:59:26.122583+0000 mon.a (mon.0) 1153 : cluster 1 Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-11T20:59:27.335 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:27.361 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:27.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:27.364 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:59:27.396 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:27 smithi023 bash[21372]: cluster 2024-03-11T20:59:26.019716+0000 mgr.y (mgr.24380) 322 : cluster 0 pgmap v331: 148 pgs: 52 active+clean, 12 undersized+peered, 53 active+undersized, 12 undersized+degraded+peered, 19 active+undersized+degraded; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 182/594 objects degraded (30.640%) 2024-03-11T20:59:27.396 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:27 smithi023 bash[21372]: cluster 2024-03-11T20:59:26.122583+0000 mon.a (mon.0) 1153 : cluster 1 Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-11T20:59:27.396 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:27 smithi023 bash[29146]: cluster 2024-03-11T20:59:26.019716+0000 mgr.y (mgr.24380) 322 : cluster 0 pgmap v331: 148 pgs: 52 active+clean, 12 undersized+peered, 53 active+undersized, 12 undersized+degraded+peered, 19 active+undersized+degraded; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 182/594 objects degraded (30.640%) 2024-03-11T20:59:27.396 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:27 smithi023 bash[29146]: cluster 2024-03-11T20:59:26.122583+0000 mon.a (mon.0) 1153 : cluster 1 Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-11T20:59:27.465 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:27.494 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:27.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:27.496 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:59:27.597 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:27.624 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:27.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:27.626 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:59:27.727 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:27.755 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:27.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:27.758 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:59:27.859 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:27.887 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:27.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:27.889 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:59:27.990 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:28.017 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:28.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:28.019 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:59:28.120 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:28.149 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:28.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:28.151 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:59:28.252 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:28.278 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:28.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:28.280 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:59:28.380 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:28.407 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:28.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:28.409 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:59:28.510 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:28.521 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:59:28 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:20:59:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T20:59:28.537 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:28.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:28.539 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:59:28.640 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:28.666 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:28.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:28.668 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:59:28.769 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:28.794 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:28.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:28.796 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:59:28.897 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:28.923 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:28.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:28.925 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:59:29.026 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:29.055 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:29.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:29.057 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:59:29.158 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:29.186 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:29.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:29.189 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:59:29.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:29 smithi146 bash[23809]: cluster 2024-03-11T20:59:28.020531+0000 mgr.y (mgr.24380) 323 : cluster 0 pgmap v332: 148 pgs: 52 active+clean, 12 undersized+peered, 53 active+undersized, 12 undersized+degraded+peered, 19 active+undersized+degraded; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 182/594 objects degraded (30.640%) 2024-03-11T20:59:29.289 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:29.318 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:29.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:29.321 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:59:29.422 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:29.432 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:29 smithi023 bash[21372]: cluster 2024-03-11T20:59:28.020531+0000 mgr.y (mgr.24380) 323 : cluster 0 pgmap v332: 148 pgs: 52 active+clean, 12 undersized+peered, 53 active+undersized, 12 undersized+degraded+peered, 19 active+undersized+degraded; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 182/594 objects degraded (30.640%) 2024-03-11T20:59:29.432 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:29 smithi023 bash[29146]: cluster 2024-03-11T20:59:28.020531+0000 mgr.y (mgr.24380) 323 : cluster 0 pgmap v332: 148 pgs: 52 active+clean, 12 undersized+peered, 53 active+undersized, 12 undersized+degraded+peered, 19 active+undersized+degraded; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 182/594 objects degraded (30.640%) 2024-03-11T20:59:29.446 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:29.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:29.449 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:59:29.550 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:29.578 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:29.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:29.580 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:59:29.681 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:29.708 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:29.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:29.744 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:59:29.844 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:29.867 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:29.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:29.869 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:59:29.970 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:29.998 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:30.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:30.000 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:59:30.101 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:30.125 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:30.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:30.128 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:59:30.229 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:30.253 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:30.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:30.256 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:59:30.352 DEBUG:teuthology.orchestra.run.smithi023:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_1 bench 90 write' 2024-03-11T20:59:30.354 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-03-11T20:59:30.355 INFO:teuthology.orchestra.run:waiting for 3000 2024-03-11T20:59:30.357 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:30.383 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:30.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:30.385 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:59:30.404 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:hints = 1 2024-03-11T20:59:30.404 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-03-11T20:59:30.404 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:Object prefix: benchmark_data_smithi023_89854 2024-03-11T20:59:30.404 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-11T20:59:30.404 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 0 0 0 0 0 0 - 0 2024-03-11T20:59:30.486 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:30.515 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:30.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:30.517 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:59:30.521 INFO:teuthology.orchestra.run.smithi023.stdout: marking collection for removal 2024-03-11T20:59:30.521 INFO:teuthology.orchestra.run.smithi023.stdout:setting '_remove' omap key 2024-03-11T20:59:30.522 INFO:teuthology.orchestra.run.smithi023.stdout:finish_remove_pgs 6.1a_head removing 6.1a 2024-03-11T20:59:30.522 INFO:teuthology.orchestra.run.smithi023.stdout:Remove successful 2024-03-11T20:59:30.618 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:30.646 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:30.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:30.648 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:59:30.749 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:30.774 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:30.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:30.778 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:59:30.833 INFO:teuthology.orchestra.run.smithi023.stdout:osd.0: {} 2024-03-11T20:59:30.854 INFO:teuthology.orchestra.run.smithi023.stdout:osd.2: {} 2024-03-11T20:59:30.873 INFO:teuthology.orchestra.run.smithi023.stdout:osd.3: {} 2024-03-11T20:59:30.879 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:30.897 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:30.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:30.899 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:59:30.900 INFO:teuthology.orchestra.run.smithi023.stdout:osd.5: {} 2024-03-11T20:59:30.924 INFO:teuthology.orchestra.run.smithi023.stdout:osd.6: {} 2024-03-11T20:59:30.949 INFO:teuthology.orchestra.run.smithi023.stdout:osd.7: {} 2024-03-11T20:59:30.949 INFO:teuthology.orchestra.run.smithi023.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T20:59:30.949 INFO:teuthology.orchestra.run.smithi023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-11T20:59:30.949 INFO:teuthology.orchestra.run.smithi023.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T20:59:30.949 INFO:teuthology.orchestra.run.smithi023.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T20:59:30.949 INFO:teuthology.orchestra.run.smithi023.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-03-11T20:59:30.950 INFO:teuthology.orchestra.run.smithi023.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T20:59:30.950 INFO:teuthology.orchestra.run.smithi023.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T20:59:30.950 INFO:teuthology.orchestra.run.smithi023.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T20:59:30.999 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:31.026 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:31.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:31.028 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:59:31.129 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:31.137 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:31 smithi023 bash[21372]: cluster 2024-03-11T20:59:30.021590+0000 mgr.y (mgr.24380) 324 : cluster 0 pgmap v333: 148 pgs: 52 active+clean, 12 undersized+peered, 53 active+undersized, 12 undersized+degraded+peered, 19 active+undersized+degraded; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 182/594 objects degraded (30.640%) 2024-03-11T20:59:31.137 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:31 smithi023 bash[21372]: audit 2024-03-11T20:59:30.815069+0000 mon.a (mon.0) 1154 : audit 0 from='client.? 172.21.15.23:0/1635813434' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T20:59:31.137 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:31 smithi023 bash[29146]: cluster 2024-03-11T20:59:30.021590+0000 mgr.y (mgr.24380) 324 : cluster 0 pgmap v333: 148 pgs: 52 active+clean, 12 undersized+peered, 53 active+undersized, 12 undersized+degraded+peered, 19 active+undersized+degraded; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 182/594 objects degraded (30.640%) 2024-03-11T20:59:31.137 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:31 smithi023 bash[29146]: audit 2024-03-11T20:59:30.815069+0000 mon.a (mon.0) 1154 : audit 0 from='client.? 172.21.15.23:0/1635813434' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T20:59:31.147 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:31.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:31.149 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:59:31.250 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:31.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:31 smithi146 bash[23809]: cluster 2024-03-11T20:59:30.021590+0000 mgr.y (mgr.24380) 324 : cluster 0 pgmap v333: 148 pgs: 52 active+clean, 12 undersized+peered, 53 active+undersized, 12 undersized+degraded+peered, 19 active+undersized+degraded; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 182/594 objects degraded (30.640%) 2024-03-11T20:59:31.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:31 smithi146 bash[23809]: audit 2024-03-11T20:59:30.815069+0000 mon.a (mon.0) 1154 : audit 0 from='client.? 172.21.15.23:0/1635813434' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T20:59:31.279 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:31.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:31.281 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:59:31.382 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:31.404 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 1 16 212 196 12.2485 12.25 0.00167193 0.00475516 2024-03-11T20:59:31.410 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:31.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:31.412 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:59:31.513 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:31.538 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:31.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:31.540 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:59:31.641 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:31.666 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:31.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:31.668 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:59:31.769 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:31.797 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:31.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:31.801 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:59:31.902 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:31.928 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:31.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:31.931 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:59:32.031 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:32.059 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:32.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:32.062 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:59:32.162 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:32.191 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:32.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:32.193 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:59:32.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:32 smithi023 bash[21372]: cluster 2024-03-11T20:59:32.125551+0000 mon.a (mon.0) 1155 : cluster 3 Health check update: Degraded data redundancy: 222/714 objects degraded (31.092%), 35 pgs degraded, 30 pgs undersized (PG_DEGRADED) 2024-03-11T20:59:32.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:32 smithi023 bash[29146]: cluster 2024-03-11T20:59:32.125551+0000 mon.a (mon.0) 1155 : cluster 3 Health check update: Degraded data redundancy: 222/714 objects degraded (31.092%), 35 pgs degraded, 30 pgs undersized (PG_DEGRADED) 2024-03-11T20:59:32.295 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:32.318 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:32.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:32.320 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:59:32.405 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 2 16 212 196 6.12407 0 - 0.00475516 2024-03-11T20:59:32.421 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:32.448 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:32.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:32.451 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:59:32.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:32 smithi146 bash[23809]: cluster 2024-03-11T20:59:32.125551+0000 mon.a (mon.0) 1155 : cluster 3 Health check update: Degraded data redundancy: 222/714 objects degraded (31.092%), 35 pgs degraded, 30 pgs undersized (PG_DEGRADED) 2024-03-11T20:59:32.552 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:32.577 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:32.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:32.580 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:59:32.681 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:32.710 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:32.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:32.713 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:59:32.813 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:32.840 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:32.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:32.843 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:59:32.943 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:32.971 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:32.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:32.973 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:59:33.074 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:33.103 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:33.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:33.105 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:59:33.206 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:33.215 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:33 smithi023 bash[21372]: cluster 2024-03-11T20:59:32.023672+0000 mgr.y (mgr.24380) 325 : cluster 0 pgmap v334: 148 pgs: 52 active+clean, 12 undersized+peered, 49 active+undersized, 12 undersized+degraded+peered, 23 active+undersized+degraded; 3.1 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 256 KiB/s wr, 3 op/s; 222/714 objects degraded (31.092%) 2024-03-11T20:59:33.215 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:33 smithi023 bash[29146]: cluster 2024-03-11T20:59:32.023672+0000 mgr.y (mgr.24380) 325 : cluster 0 pgmap v334: 148 pgs: 52 active+clean, 12 undersized+peered, 49 active+undersized, 12 undersized+degraded+peered, 23 active+undersized+degraded; 3.1 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 256 KiB/s wr, 3 op/s; 222/714 objects degraded (31.092%) 2024-03-11T20:59:33.230 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:33.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:33.233 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:59:33.333 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:33.360 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:33.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:33.362 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:59:33.405 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 3 16 212 196 4.08271 0 - 0.00475516 2024-03-11T20:59:33.463 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:33.474 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:33 smithi146 bash[23809]: cluster 2024-03-11T20:59:32.023672+0000 mgr.y (mgr.24380) 325 : cluster 0 pgmap v334: 148 pgs: 52 active+clean, 12 undersized+peered, 49 active+undersized, 12 undersized+degraded+peered, 23 active+undersized+degraded; 3.1 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 256 KiB/s wr, 3 op/s; 222/714 objects degraded (31.092%) 2024-03-11T20:59:33.491 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:33.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:33.493 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:59:33.594 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:33.621 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:33.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:33.623 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:59:33.724 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:33.751 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:33.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:33.753 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:59:33.854 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:33.881 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:33.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:33.883 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:59:33.968 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-11T20:59:33.969 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-11T20:59:33.972 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -n osd.4 --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-4 --journal-path /var/lib/ceph/osd/ceph-4/journal --op list-pgs 2024-03-11T20:59:33.984 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:34.043 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:34.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:34.045 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:59:34.143 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/osd.4/config 2024-03-11T20:59:34.146 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:34.166 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:34.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:34.168 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:59:34.269 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:34.296 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:34.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:34.298 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:59:34.399 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:34.405 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 4 16 212 196 3.06204 0 - 0.00475516 2024-03-11T20:59:34.427 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:34.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:34.429 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:59:34.530 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:34.558 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:34.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:34.561 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:59:34.661 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:34.681 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:34.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:34.684 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:59:34.724 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-11T20:59:34.724 INFO:teuthology.orchestra.run.smithi023.stdout: "ops": [], 2024-03-11T20:59:34.724 INFO:teuthology.orchestra.run.smithi023.stdout: "num_ops": 0 2024-03-11T20:59:34.724 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-11T20:59:34.785 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:34.812 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:34.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:34.814 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:59:34.915 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:34.938 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:34.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:34.940 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:59:35.041 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:35.068 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:35.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:35.070 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:59:35.171 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:35.199 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:35.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:35.201 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:59:35.302 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:35.311 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 20:59:35 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T20:59:35.311 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:35 smithi146 bash[23809]: cluster 2024-03-11T20:59:34.024717+0000 mgr.y (mgr.24380) 326 : cluster 0 pgmap v335: 148 pgs: 52 active+clean, 12 undersized+peered, 49 active+undersized, 12 undersized+degraded+peered, 23 active+undersized+degraded; 3.1 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 256 KiB/s wr, 3 op/s; 222/714 objects degraded (31.092%) 2024-03-11T20:59:35.311 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:35 smithi146 bash[23809]: audit 2024-03-11T20:59:34.784664+0000 mon.a (mon.0) 1156 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:59:35.311 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:35 smithi146 bash[23809]: audit 2024-03-11T20:59:34.785824+0000 mon.a (mon.0) 1157 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:59:35.311 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:35 smithi146 bash[23809]: audit 2024-03-11T20:59:34.998810+0000 mon.c (mon.2) 43 : audit 1 from='client.? 172.21.15.23:0/1239193098' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T20:59:35.312 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:35 smithi146 bash[23809]: audit 2024-03-11T20:59:34.999261+0000 mon.a (mon.0) 1158 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T20:59:35.325 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:35.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:35.327 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:59:35.405 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 5 16 212 196 2.44964 0 - 0.00475516 2024-03-11T20:59:35.429 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:35.449 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:35.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:35.452 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:59:35.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:35 smithi023 bash[21372]: cluster 2024-03-11T20:59:34.024717+0000 mgr.y (mgr.24380) 326 : cluster 0 pgmap v335: 148 pgs: 52 active+clean, 12 undersized+peered, 49 active+undersized, 12 undersized+degraded+peered, 23 active+undersized+degraded; 3.1 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 256 KiB/s wr, 3 op/s; 222/714 objects degraded (31.092%) 2024-03-11T20:59:35.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:35 smithi023 bash[21372]: audit 2024-03-11T20:59:34.784664+0000 mon.a (mon.0) 1156 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:59:35.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:35 smithi023 bash[21372]: audit 2024-03-11T20:59:34.785824+0000 mon.a (mon.0) 1157 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:59:35.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:35 smithi023 bash[21372]: audit 2024-03-11T20:59:34.998810+0000 mon.c (mon.2) 43 : audit 1 from='client.? 172.21.15.23:0/1239193098' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T20:59:35.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:35 smithi023 bash[21372]: audit 2024-03-11T20:59:34.999261+0000 mon.a (mon.0) 1158 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T20:59:35.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:35 smithi023 bash[29146]: cluster 2024-03-11T20:59:34.024717+0000 mgr.y (mgr.24380) 326 : cluster 0 pgmap v335: 148 pgs: 52 active+clean, 12 undersized+peered, 49 active+undersized, 12 undersized+degraded+peered, 23 active+undersized+degraded; 3.1 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 256 KiB/s wr, 3 op/s; 222/714 objects degraded (31.092%) 2024-03-11T20:59:35.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:35 smithi023 bash[29146]: audit 2024-03-11T20:59:34.784664+0000 mon.a (mon.0) 1156 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:59:35.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:35 smithi023 bash[29146]: audit 2024-03-11T20:59:34.785824+0000 mon.a (mon.0) 1157 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:59:35.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:35 smithi023 bash[29146]: audit 2024-03-11T20:59:34.998810+0000 mon.c (mon.2) 43 : audit 1 from='client.? 172.21.15.23:0/1239193098' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T20:59:35.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:35 smithi023 bash[29146]: audit 2024-03-11T20:59:34.999261+0000 mon.a (mon.0) 1158 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T20:59:35.552 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:35.582 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:35.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:35.585 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:59:35.602 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-03-11T20:59:35.686 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:35.712 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:35.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:35.715 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:59:35.798 INFO:teuthology.orchestra.run.smithi023.stderr:nodeep-scrub is set 2024-03-11T20:59:35.815 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:35.843 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:35.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:35.846 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:59:35.947 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:35.970 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T20:59:35.976 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:35.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:35.979 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:59:36.079 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:36.108 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:36.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:36.109 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:59:36.210 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:36.217 INFO:teuthology.orchestra.run.smithi146.stderr:Creating an OSD daemon form without an OSD FSID value 2024-03-11T20:59:36.230 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:36.232 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:36 smithi023 bash[21372]: audit 2024-03-11T20:59:35.240642+0000 mgr.y (mgr.24380) 327 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:59:36.232 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:36 smithi023 bash[21372]: cluster 2024-03-11T20:59:35.782893+0000 mon.a (mon.0) 1159 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T20:59:36.232 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:36 smithi023 bash[21372]: audit 2024-03-11T20:59:35.785987+0000 mon.a (mon.0) 1160 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T20:59:36.232 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:36 smithi023 bash[21372]: cluster 2024-03-11T20:59:35.790121+0000 mon.a (mon.0) 1161 : cluster 0 osdmap e194: 8 total, 6 up, 8 in 2024-03-11T20:59:36.232 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:36 smithi023 bash[29146]: audit 2024-03-11T20:59:35.240642+0000 mgr.y (mgr.24380) 327 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:59:36.232 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:36 smithi023 bash[29146]: cluster 2024-03-11T20:59:35.782893+0000 mon.a (mon.0) 1159 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T20:59:36.232 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:36 smithi023 bash[29146]: audit 2024-03-11T20:59:35.785987+0000 mon.a (mon.0) 1160 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T20:59:36.232 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:36 smithi023 bash[29146]: cluster 2024-03-11T20:59:35.790121+0000 mon.a (mon.0) 1161 : cluster 0 osdmap e194: 8 total, 6 up, 8 in 2024-03-11T20:59:36.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:36.233 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:59:36.235 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:36 smithi146 bash[23809]: audit 2024-03-11T20:59:35.240642+0000 mgr.y (mgr.24380) 327 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:59:36.236 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:36 smithi146 bash[23809]: cluster 2024-03-11T20:59:35.782893+0000 mon.a (mon.0) 1159 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T20:59:36.236 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:36 smithi146 bash[23809]: audit 2024-03-11T20:59:35.785987+0000 mon.a (mon.0) 1160 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T20:59:36.236 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:36 smithi146 bash[23809]: cluster 2024-03-11T20:59:35.790121+0000 mon.a (mon.0) 1161 : cluster 0 osdmap e194: 8 total, 6 up, 8 in 2024-03-11T20:59:36.334 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:36.361 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:36.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:36.363 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:59:36.405 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 6 16 212 196 2.04136 0 - 0.00475516 2024-03-11T20:59:36.464 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:36.493 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:36.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:36.495 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:59:36.596 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:36.622 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:36.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:36.624 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:59:36.725 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:36.752 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:36.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:36.755 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:59:36.856 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:36.886 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:36.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:36.889 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:59:36.990 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:37.016 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:37.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:37.018 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:59:37.121 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:37.148 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:37.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:37.151 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:59:37.252 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:37.255 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:37 smithi146 bash[23809]: cluster 2024-03-11T20:59:36.025589+0000 mgr.y (mgr.24380) 328 : cluster 0 pgmap v337: 148 pgs: 52 active+clean, 12 undersized+peered, 43 active+undersized, 12 undersized+degraded+peered, 29 active+undersized+degraded; 13 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 1.2 MiB/s wr, 19 op/s; 304/1182 objects degraded (25.719%) 2024-03-11T20:59:37.274 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:37.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:37.276 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:59:37.377 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:37.389 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:37 smithi023 bash[21372]: cluster 2024-03-11T20:59:36.025589+0000 mgr.y (mgr.24380) 328 : cluster 0 pgmap v337: 148 pgs: 52 active+clean, 12 undersized+peered, 43 active+undersized, 12 undersized+degraded+peered, 29 active+undersized+degraded; 13 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 1.2 MiB/s wr, 19 op/s; 304/1182 objects degraded (25.719%) 2024-03-11T20:59:37.389 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:37 smithi023 bash[29146]: cluster 2024-03-11T20:59:36.025589+0000 mgr.y (mgr.24380) 328 : cluster 0 pgmap v337: 148 pgs: 52 active+clean, 12 undersized+peered, 43 active+undersized, 12 undersized+degraded+peered, 29 active+undersized+degraded; 13 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 1.2 MiB/s wr, 19 op/s; 304/1182 objects degraded (25.719%) 2024-03-11T20:59:37.405 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:37.405 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 7 16 212 196 1.74975 0 - 0.00475516 2024-03-11T20:59:37.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:37.409 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:59:37.510 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:37.533 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:37.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:37.535 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:59:37.636 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:37.664 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:37.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:37.667 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:59:37.768 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:37.797 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:37.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:37.800 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:59:37.901 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:37.926 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:37.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:37.929 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:59:38.029 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:38.056 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:38.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:38.059 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:59:38.160 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:38.185 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:38.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:38.187 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:59:38.245 INFO:teuthology.orchestra.run.smithi146.stdout:6.e 2024-03-11T20:59:38.245 INFO:teuthology.orchestra.run.smithi146.stdout:6.7 2024-03-11T20:59:38.245 INFO:teuthology.orchestra.run.smithi146.stdout:6.6 2024-03-11T20:59:38.245 INFO:teuthology.orchestra.run.smithi146.stdout:6.f 2024-03-11T20:59:38.245 INFO:teuthology.orchestra.run.smithi146.stdout:6.2 2024-03-11T20:59:38.245 INFO:teuthology.orchestra.run.smithi146.stdout:6.1e 2024-03-11T20:59:38.245 INFO:teuthology.orchestra.run.smithi146.stdout:6.15 2024-03-11T20:59:38.245 INFO:teuthology.orchestra.run.smithi146.stdout:6.12 2024-03-11T20:59:38.246 INFO:teuthology.orchestra.run.smithi146.stdout:5.1 2024-03-11T20:59:38.246 INFO:teuthology.orchestra.run.smithi146.stdout:4.f 2024-03-11T20:59:38.246 INFO:teuthology.orchestra.run.smithi146.stdout:4.d 2024-03-11T20:59:38.246 INFO:teuthology.orchestra.run.smithi146.stdout:4.c 2024-03-11T20:59:38.246 INFO:teuthology.orchestra.run.smithi146.stdout:2.b 2024-03-11T20:59:38.246 INFO:teuthology.orchestra.run.smithi146.stdout:3.13 2024-03-11T20:59:38.246 INFO:teuthology.orchestra.run.smithi146.stdout:4.1b 2024-03-11T20:59:38.246 INFO:teuthology.orchestra.run.smithi146.stdout:3.11 2024-03-11T20:59:38.246 INFO:teuthology.orchestra.run.smithi146.stdout:4.e 2024-03-11T20:59:38.246 INFO:teuthology.orchestra.run.smithi146.stdout:3.1 2024-03-11T20:59:38.246 INFO:teuthology.orchestra.run.smithi146.stdout:2.6 2024-03-11T20:59:38.246 INFO:teuthology.orchestra.run.smithi146.stdout:3.f 2024-03-11T20:59:38.246 INFO:teuthology.orchestra.run.smithi146.stdout:3.15 2024-03-11T20:59:38.246 INFO:teuthology.orchestra.run.smithi146.stdout:6.17 2024-03-11T20:59:38.247 INFO:teuthology.orchestra.run.smithi146.stdout:4.5 2024-03-11T20:59:38.247 INFO:teuthology.orchestra.run.smithi146.stdout:6.1d 2024-03-11T20:59:38.247 INFO:teuthology.orchestra.run.smithi146.stdout:3.14 2024-03-11T20:59:38.247 INFO:teuthology.orchestra.run.smithi146.stdout:4.1d 2024-03-11T20:59:38.247 INFO:teuthology.orchestra.run.smithi146.stdout:2.13 2024-03-11T20:59:38.247 INFO:teuthology.orchestra.run.smithi146.stdout:2.f 2024-03-11T20:59:38.247 INFO:teuthology.orchestra.run.smithi146.stdout:2.d 2024-03-11T20:59:38.247 INFO:teuthology.orchestra.run.smithi146.stdout:2.1f 2024-03-11T20:59:38.247 INFO:teuthology.orchestra.run.smithi146.stdout:3.1a 2024-03-11T20:59:38.247 INFO:teuthology.orchestra.run.smithi146.stdout:6.1a 2024-03-11T20:59:38.247 INFO:teuthology.orchestra.run.smithi146.stdout:3.1b 2024-03-11T20:59:38.247 INFO:teuthology.orchestra.run.smithi146.stdout:2.18 2024-03-11T20:59:38.247 INFO:teuthology.orchestra.run.smithi146.stdout:3.6 2024-03-11T20:59:38.247 INFO:teuthology.orchestra.run.smithi146.stdout:2.5 2024-03-11T20:59:38.247 INFO:teuthology.orchestra.run.smithi146.stdout:3.b 2024-03-11T20:59:38.248 INFO:teuthology.orchestra.run.smithi146.stdout:3.1d 2024-03-11T20:59:38.248 INFO:teuthology.orchestra.run.smithi146.stdout:5.0 2024-03-11T20:59:38.248 INFO:teuthology.orchestra.run.smithi146.stdout:2.1a 2024-03-11T20:59:38.248 INFO:teuthology.orchestra.run.smithi146.stdout:3.3 2024-03-11T20:59:38.248 INFO:teuthology.orchestra.run.smithi146.stdout:4.b 2024-03-11T20:59:38.248 INFO:teuthology.orchestra.run.smithi146.stdout:4.1 2024-03-11T20:59:38.248 INFO:teuthology.orchestra.run.smithi146.stdout:2.11 2024-03-11T20:59:38.248 INFO:teuthology.orchestra.run.smithi146.stdout:3.9 2024-03-11T20:59:38.248 INFO:teuthology.orchestra.run.smithi146.stdout:3.a 2024-03-11T20:59:38.248 INFO:teuthology.orchestra.run.smithi146.stdout:3.e 2024-03-11T20:59:38.248 INFO:teuthology.orchestra.run.smithi146.stdout:4.13 2024-03-11T20:59:38.248 INFO:teuthology.orchestra.run.smithi146.stdout:6.14 2024-03-11T20:59:38.248 INFO:teuthology.orchestra.run.smithi146.stdout:4.18 2024-03-11T20:59:38.248 INFO:teuthology.orchestra.run.smithi146.stdout:3.19 2024-03-11T20:59:38.248 INFO:teuthology.orchestra.run.smithi146.stdout:3.1c 2024-03-11T20:59:38.249 INFO:teuthology.orchestra.run.smithi146.stdout:4.1a 2024-03-11T20:59:38.249 INFO:teuthology.orchestra.run.smithi146.stdout:4.2 2024-03-11T20:59:38.249 INFO:teuthology.orchestra.run.smithi146.stdout:4.9 2024-03-11T20:59:38.288 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:38.317 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:38.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:38.320 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:59:38.405 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 8 16 212 196 1.53104 0 - 0.00475516 2024-03-11T20:59:38.421 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:38.446 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:38.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:38.448 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:59:38.549 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:38.560 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:59:38 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:20:59:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T20:59:38.578 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:38.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:38.581 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:59:38.603 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd unset noscrub 2024-03-11T20:59:38.681 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:38.707 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:38.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:38.710 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:59:38.810 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:38.830 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:38.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:38.832 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:59:38.933 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:38.959 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:38.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:38.961 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:59:39.062 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:39.091 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:39.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:39.093 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:59:39.193 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:39.221 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:39.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:39.224 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:59:39.325 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:39.353 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:39.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:39.356 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:59:39.406 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 9 16 212 196 1.36092 0 - 0.00475516 2024-03-11T20:59:39.457 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:39.468 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:39 smithi146 bash[23809]: cluster 2024-03-11T20:59:38.027081+0000 mgr.y (mgr.24380) 329 : cluster 0 pgmap v338: 148 pgs: 52 active+clean, 12 undersized+peered, 43 active+undersized, 12 undersized+degraded+peered, 29 active+undersized+degraded; 13 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 1.2 MiB/s wr, 19 op/s; 304/1182 objects degraded (25.719%) 2024-03-11T20:59:39.484 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:39.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:39.486 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:59:39.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:39 smithi023 bash[29146]: cluster 2024-03-11T20:59:38.027081+0000 mgr.y (mgr.24380) 329 : cluster 0 pgmap v338: 148 pgs: 52 active+clean, 12 undersized+peered, 43 active+undersized, 12 undersized+degraded+peered, 29 active+undersized+degraded; 13 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 1.2 MiB/s wr, 19 op/s; 304/1182 objects degraded (25.719%) 2024-03-11T20:59:39.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:39 smithi023 bash[21372]: cluster 2024-03-11T20:59:38.027081+0000 mgr.y (mgr.24380) 329 : cluster 0 pgmap v338: 148 pgs: 52 active+clean, 12 undersized+peered, 43 active+undersized, 12 undersized+degraded+peered, 29 active+undersized+degraded; 13 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 1.2 MiB/s wr, 19 op/s; 304/1182 objects degraded (25.719%) 2024-03-11T20:59:39.587 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:39.615 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:39.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:39.618 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:59:39.663 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -n osd.1 --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-1 --journal-path /var/lib/ceph/osd/ceph-1/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.6.1a.1 2024-03-11T20:59:39.719 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:39.747 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:39.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:39.749 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:59:39.830 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/osd.1/config 2024-03-11T20:59:39.849 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:39.870 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:39.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:39.872 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:59:39.973 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:40.000 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:40.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:40.002 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:59:40.103 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:40.131 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:40.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:40.134 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:59:40.235 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:40.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:40 smithi146 bash[23809]: cluster 2024-03-11T20:59:40.122930+0000 mon.a (mon.0) 1162 : cluster 3 Health check update: Degraded data redundancy: 304/1182 objects degraded (25.719%), 41 pgs degraded, 30 pgs undersized (PG_DEGRADED) 2024-03-11T20:59:40.262 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:40.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:40.264 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:59:40.365 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:40.394 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:40.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:40.396 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:59:40.406 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 10 16 212 196 1.22484 0 - 0.00475516 2024-03-11T20:59:40.407 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:59:40.409 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:59:40.497 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:40.524 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:40.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:40.526 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:59:40.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:40 smithi023 bash[21372]: cluster 2024-03-11T20:59:40.122930+0000 mon.a (mon.0) 1162 : cluster 3 Health check update: Degraded data redundancy: 304/1182 objects degraded (25.719%), 41 pgs degraded, 30 pgs undersized (PG_DEGRADED) 2024-03-11T20:59:40.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:40 smithi023 bash[29146]: cluster 2024-03-11T20:59:40.122930+0000 mon.a (mon.0) 1162 : cluster 3 Health check update: Degraded data redundancy: 304/1182 objects degraded (25.719%), 41 pgs degraded, 30 pgs undersized (PG_DEGRADED) 2024-03-11T20:59:40.627 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:40.655 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:40.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:40.657 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:59:40.758 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:40.786 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:40.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:40.789 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:59:40.889 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:40.921 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:40.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:40.923 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:59:41.024 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:41.051 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:41.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:41.053 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:59:41.154 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:41.182 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:41.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:41.184 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:59:41.285 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:41.314 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:41.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:41.317 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:59:41.406 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 11 16 212 196 1.11349 0 - 0.00475516 2024-03-11T20:59:41.418 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:41.440 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:41.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:41.443 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:59:41.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:41 smithi146 bash[23809]: cluster 2024-03-11T20:59:40.029090+0000 mgr.y (mgr.24380) 330 : cluster 0 pgmap v339: 148 pgs: 52 active+clean, 12 undersized+peered, 43 active+undersized, 12 undersized+degraded+peered, 29 active+undersized+degraded; 13 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 1.2 MiB/s wr, 19 op/s; 304/1182 objects degraded (25.719%) 2024-03-11T20:59:41.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:41 smithi023 bash[21372]: cluster 2024-03-11T20:59:40.029090+0000 mgr.y (mgr.24380) 330 : cluster 0 pgmap v339: 148 pgs: 52 active+clean, 12 undersized+peered, 43 active+undersized, 12 undersized+degraded+peered, 29 active+undersized+degraded; 13 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 1.2 MiB/s wr, 19 op/s; 304/1182 objects degraded (25.719%) 2024-03-11T20:59:41.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:41 smithi023 bash[29146]: cluster 2024-03-11T20:59:40.029090+0000 mgr.y (mgr.24380) 330 : cluster 0 pgmap v339: 148 pgs: 52 active+clean, 12 undersized+peered, 43 active+undersized, 12 undersized+degraded+peered, 29 active+undersized+degraded; 13 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 1.2 MiB/s wr, 19 op/s; 304/1182 objects degraded (25.719%) 2024-03-11T20:59:41.544 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:41.570 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:41.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:41.573 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:59:41.674 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:41.699 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:41.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:41.702 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:59:41.803 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:41.830 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:41.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:41.833 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:59:41.933 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:41.960 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:41.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:41.962 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:59:42.063 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:42.090 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:42.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:42.092 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:59:42.193 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:42.215 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:42.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:42.218 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:59:42.318 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:42.346 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:42.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:42.349 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:59:42.406 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 12 16 212 196 1.0207 0 - 0.00475516 2024-03-11T20:59:42.444 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:59:42.450 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:42.471 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:42.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:42.473 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:59:42.574 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:42.601 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:42.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:42.604 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:59:42.705 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:42.726 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:42.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:42.728 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:59:42.828 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:42.851 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:42.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:42.853 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:59:42.954 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:42.981 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:42.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:42.984 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:59:43.084 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:43.107 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:43.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:43.109 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:59:43.209 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:43.237 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:43.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:43.239 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:59:43.286 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:43 smithi023 bash[29146]: cluster 2024-03-11T20:59:42.030110+0000 mgr.y (mgr.24380) 331 : cluster 0 pgmap v340: 148 pgs: 52 active+clean, 12 undersized+peered, 43 active+undersized, 12 undersized+degraded+peered, 29 active+undersized+degraded; 13 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 998 KiB/s wr, 15 op/s; 304/1182 objects degraded (25.719%) 2024-03-11T20:59:43.286 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:43 smithi023 bash[21372]: cluster 2024-03-11T20:59:42.030110+0000 mgr.y (mgr.24380) 331 : cluster 0 pgmap v340: 148 pgs: 52 active+clean, 12 undersized+peered, 43 active+undersized, 12 undersized+degraded+peered, 29 active+undersized+degraded; 13 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 998 KiB/s wr, 15 op/s; 304/1182 objects degraded (25.719%) 2024-03-11T20:59:43.340 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:43.367 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:43.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:43.369 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:59:43.406 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 13 16 212 196 0.942185 0 - 0.00475516 2024-03-11T20:59:43.471 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:43.498 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:43.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:43.501 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:59:43.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:43 smithi146 bash[23809]: cluster 2024-03-11T20:59:42.030110+0000 mgr.y (mgr.24380) 331 : cluster 0 pgmap v340: 148 pgs: 52 active+clean, 12 undersized+peered, 43 active+undersized, 12 undersized+degraded+peered, 29 active+undersized+degraded; 13 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 998 KiB/s wr, 15 op/s; 304/1182 objects degraded (25.719%) 2024-03-11T20:59:43.602 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:43.629 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:43.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:43.631 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:59:43.732 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:43.761 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:43.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:43.764 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:59:43.864 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:43.891 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:43.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:43.893 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:59:43.994 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:44.022 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:44.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:44.024 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:59:44.125 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:44.152 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:44.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:44.154 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:59:44.255 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:44.283 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:44.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:44.285 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:59:44.386 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:44.406 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 14 16 212 196 0.874887 0 - 0.00475516 2024-03-11T20:59:44.414 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:44.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:44.416 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:59:44.517 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:44.544 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:44.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:44.546 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:59:44.647 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:44.671 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:44.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:44.674 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:59:44.713 INFO:teuthology.orchestra.run.smithi023.stderr:Creating an OSD daemon form without an OSD FSID value 2024-03-11T20:59:44.775 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:44.802 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:44.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:44.805 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:59:44.905 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:44.931 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:44.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:44.933 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:59:45.034 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:45.062 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:45.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:45.065 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:59:45.166 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:45.192 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:45.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:45.195 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:59:45.295 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:45.306 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 20:59:45 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T20:59:45.307 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:45 smithi146 bash[23809]: cluster 2024-03-11T20:59:44.031618+0000 mgr.y (mgr.24380) 332 : cluster 0 pgmap v341: 148 pgs: 52 active+clean, 12 undersized+peered, 43 active+undersized, 12 undersized+degraded+peered, 29 active+undersized+degraded; 13 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 998 KiB/s wr, 15 op/s; 304/1182 objects degraded (25.719%) 2024-03-11T20:59:45.321 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:45.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:45.323 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:59:45.406 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 15 16 212 196 0.81656 0 - 0.00475516 2024-03-11T20:59:45.424 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:45.452 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:45.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:45.454 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:59:45.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:45 smithi023 bash[21372]: cluster 2024-03-11T20:59:44.031618+0000 mgr.y (mgr.24380) 332 : cluster 0 pgmap v341: 148 pgs: 52 active+clean, 12 undersized+peered, 43 active+undersized, 12 undersized+degraded+peered, 29 active+undersized+degraded; 13 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 998 KiB/s wr, 15 op/s; 304/1182 objects degraded (25.719%) 2024-03-11T20:59:45.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:45 smithi023 bash[29146]: cluster 2024-03-11T20:59:44.031618+0000 mgr.y (mgr.24380) 332 : cluster 0 pgmap v341: 148 pgs: 52 active+clean, 12 undersized+peered, 43 active+undersized, 12 undersized+degraded+peered, 29 active+undersized+degraded; 13 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 998 KiB/s wr, 15 op/s; 304/1182 objects degraded (25.719%) 2024-03-11T20:59:45.554 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:45.582 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:45.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:45.585 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:59:45.685 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:45.710 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:45.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:45.713 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:59:45.813 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:45.842 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:45.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:45.844 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:59:45.945 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:45.971 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:45.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:45.974 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:59:46.075 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:46.103 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:46.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:46.106 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:59:46.206 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:46.234 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:46.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:46.237 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:59:46.338 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:46.349 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:46 smithi023 bash[21372]: audit 2024-03-11T20:59:45.250011+0000 mgr.y (mgr.24380) 333 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:59:46.349 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:46 smithi023 bash[29146]: audit 2024-03-11T20:59:45.250011+0000 mgr.y (mgr.24380) 333 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:59:46.366 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:46.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:46.368 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:59:46.406 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 16 16 212 196 0.765526 0 - 0.00475516 2024-03-11T20:59:46.469 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:46.494 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:46.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:46.496 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:59:46.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:46 smithi146 bash[23809]: audit 2024-03-11T20:59:45.250011+0000 mgr.y (mgr.24380) 333 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:59:46.597 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:46.622 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:46.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:46.625 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:59:46.725 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:46.754 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:46.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:46.756 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:59:46.857 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:46.884 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:46.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:46.886 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:59:46.987 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:47.014 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:47.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:47.016 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:59:47.117 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:47.145 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:47.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:47.148 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:59:47.249 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:47.271 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:47.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:47.273 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:59:47.374 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:47.399 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:47.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:47.402 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:59:47.406 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 17 16 212 196 0.720496 0 - 0.00475516 2024-03-11T20:59:47.503 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:47.527 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:47.529 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:47 smithi146 bash[23809]: cluster 2024-03-11T20:59:46.033963+0000 mgr.y (mgr.24380) 334 : cluster 0 pgmap v342: 148 pgs: 52 active+clean, 12 undersized+peered, 43 active+undersized, 12 undersized+degraded+peered, 29 active+undersized+degraded; 13 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 304/1182 objects degraded (25.719%) 2024-03-11T20:59:47.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:47.530 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:59:47.564 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:47 smithi023 bash[29146]: cluster 2024-03-11T20:59:46.033963+0000 mgr.y (mgr.24380) 334 : cluster 0 pgmap v342: 148 pgs: 52 active+clean, 12 undersized+peered, 43 active+undersized, 12 undersized+degraded+peered, 29 active+undersized+degraded; 13 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 304/1182 objects degraded (25.719%) 2024-03-11T20:59:47.631 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:47.658 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:47.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:47.661 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:59:47.761 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:47.788 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:47.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:47.790 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:59:47.891 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:47.902 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:47 smithi023 bash[21372]: cluster 2024-03-11T20:59:46.033963+0000 mgr.y (mgr.24380) 334 : cluster 0 pgmap v342: 148 pgs: 52 active+clean, 12 undersized+peered, 43 active+undersized, 12 undersized+degraded+peered, 29 active+undersized+degraded; 13 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 304/1182 objects degraded (25.719%) 2024-03-11T20:59:47.917 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:47.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:47.919 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:59:48.020 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:48.050 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:48.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:48.052 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:59:48.153 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:48.181 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:48.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:48.184 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:59:48.284 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:48.308 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:48.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:48.310 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:59:48.407 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 18 16 212 196 0.680466 0 - 0.00475516 2024-03-11T20:59:48.411 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:48.439 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:48.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:48.442 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:59:48.535 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:59:48 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:20:59:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T20:59:48.542 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:48.570 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:48.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:48.573 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:59:48.674 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:48.698 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:48.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:48.701 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:59:48.802 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:48.829 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:48.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:48.831 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:59:48.932 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:48.960 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:48.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:48.962 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:59:49.063 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:49.089 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:49.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:49.092 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:59:49.193 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:49.219 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:49.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:49.222 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:59:49.323 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:49.351 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:49.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:49.353 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:59:49.407 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 19 16 212 196 0.644652 0 - 0.00475516 2024-03-11T20:59:49.455 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:49.466 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:49 smithi146 bash[23809]: cluster 2024-03-11T20:59:48.035540+0000 mgr.y (mgr.24380) 335 : cluster 0 pgmap v343: 148 pgs: 52 active+clean, 12 undersized+peered, 43 active+undersized, 12 undersized+degraded+peered, 29 active+undersized+degraded; 13 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 304/1182 objects degraded (25.719%) 2024-03-11T20:59:49.483 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:49.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:49.485 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:59:49.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:49 smithi023 bash[21372]: cluster 2024-03-11T20:59:48.035540+0000 mgr.y (mgr.24380) 335 : cluster 0 pgmap v343: 148 pgs: 52 active+clean, 12 undersized+peered, 43 active+undersized, 12 undersized+degraded+peered, 29 active+undersized+degraded; 13 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 304/1182 objects degraded (25.719%) 2024-03-11T20:59:49.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:49 smithi023 bash[29146]: cluster 2024-03-11T20:59:48.035540+0000 mgr.y (mgr.24380) 335 : cluster 0 pgmap v343: 148 pgs: 52 active+clean, 12 undersized+peered, 43 active+undersized, 12 undersized+degraded+peered, 29 active+undersized+degraded; 13 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 304/1182 objects degraded (25.719%) 2024-03-11T20:59:49.586 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:49.613 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:49.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:49.616 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:59:49.717 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:49.741 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:49.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:49.744 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:59:49.844 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:49.871 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:49.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:49.873 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:59:49.951 INFO:teuthology.orchestra.run.smithi023.stdout:get_pg_num_history pg_num_history pg_num_history(e173 pg_nums {1={20=1},2={76=32},3={78=32},4={80=32},5={80=3},6={82=32},7={121=16}} deleted_pools ) 2024-03-11T20:59:49.951 INFO:teuthology.orchestra.run.smithi023.stdout:Importing pgid 6.1a 2024-03-11T20:59:49.951 INFO:teuthology.orchestra.run.smithi023.stdout:write_pg epoch 165 info 6.1a( empty local-lis/les=162/163 n=0 ec=82/82 lis/c=162/162 les/c/f=163/163/0 sis=162) 2024-03-11T20:59:49.952 INFO:teuthology.orchestra.run.smithi023.stdout:Import successful 2024-03-11T20:59:49.974 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:49.993 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:49.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:49.996 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:59:50.096 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:50.121 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:50.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:50.124 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:59:50.225 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:50.246 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:50.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:50.248 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:59:50.349 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:50.370 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:50.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:50.372 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:59:50.407 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:2024-03-11T20:59:50.410902+0000 min lat: 0.00147232 max lat: 0.0158892 avg lat: 0.00475516 2024-03-11T20:59:50.407 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-11T20:59:50.407 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 20 16 212 196 0.61242 0 - 0.00475516 2024-03-11T20:59:50.473 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:50.494 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:50.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:50.496 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:59:50.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:50 smithi146 bash[23809]: audit 2024-03-11T20:59:49.788155+0000 mon.a (mon.0) 1163 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:59:50.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:50 smithi146 bash[23809]: audit 2024-03-11T20:59:49.799615+0000 mon.a (mon.0) 1164 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:59:50.597 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:50.617 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:50.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:50.619 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:59:50.653 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-11T20:59:50.653 INFO:teuthology.orchestra.run.smithi023.stdout: "ops": [], 2024-03-11T20:59:50.653 INFO:teuthology.orchestra.run.smithi023.stdout: "complaint_time": 30, 2024-03-11T20:59:50.653 INFO:teuthology.orchestra.run.smithi023.stdout: "num_blocked_ops": 0 2024-03-11T20:59:50.653 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-11T20:59:50.700 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:50 smithi023 bash[21372]: audit 2024-03-11T20:59:49.788155+0000 mon.a (mon.0) 1163 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:59:50.700 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:50 smithi023 bash[21372]: audit 2024-03-11T20:59:49.799615+0000 mon.a (mon.0) 1164 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:59:50.700 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:50 smithi023 bash[29146]: audit 2024-03-11T20:59:49.788155+0000 mon.a (mon.0) 1163 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:59:50.701 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:50 smithi023 bash[29146]: audit 2024-03-11T20:59:49.799615+0000 mon.a (mon.0) 1164 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:59:50.720 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:50.747 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:50.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:50.748 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:59:50.849 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:50.874 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:50.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:50.876 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:59:50.932 INFO:teuthology.orchestra.run.smithi023.stdout:osd.0: {} 2024-03-11T20:59:50.953 INFO:teuthology.orchestra.run.smithi023.stdout:osd.2: {} 2024-03-11T20:59:50.973 INFO:teuthology.orchestra.run.smithi023.stdout:osd.3: {} 2024-03-11T20:59:50.977 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:50.996 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:50.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:50.998 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:59:50.999 INFO:teuthology.orchestra.run.smithi023.stdout:osd.5: {} 2024-03-11T20:59:51.022 INFO:teuthology.orchestra.run.smithi023.stdout:osd.6: {} 2024-03-11T20:59:51.044 INFO:teuthology.orchestra.run.smithi023.stdout:osd.7: {} 2024-03-11T20:59:51.044 INFO:teuthology.orchestra.run.smithi023.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T20:59:51.044 INFO:teuthology.orchestra.run.smithi023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-11T20:59:51.044 INFO:teuthology.orchestra.run.smithi023.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T20:59:51.044 INFO:teuthology.orchestra.run.smithi023.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T20:59:51.045 INFO:teuthology.orchestra.run.smithi023.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-03-11T20:59:51.045 INFO:teuthology.orchestra.run.smithi023.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T20:59:51.045 INFO:teuthology.orchestra.run.smithi023.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T20:59:51.045 INFO:teuthology.orchestra.run.smithi023.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T20:59:51.099 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:51.120 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:51.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:51.123 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:59:51.223 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:51.251 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:51.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:51.253 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:59:51.354 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:51.381 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:51.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:51.383 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:59:51.407 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 21 16 212 196 0.583257 0 - 0.00475516 2024-03-11T20:59:51.484 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:51.495 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:51 smithi146 bash[23809]: cluster 2024-03-11T20:59:50.036512+0000 mgr.y (mgr.24380) 336 : cluster 0 pgmap v344: 148 pgs: 52 active+clean, 12 undersized+peered, 43 active+undersized, 12 undersized+degraded+peered, 29 active+undersized+degraded; 13 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 304/1182 objects degraded (25.719%) 2024-03-11T20:59:51.495 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:51 smithi146 bash[23809]: audit 2024-03-11T20:59:50.466987+0000 mon.a (mon.0) 1165 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:59:51.495 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:51 smithi146 bash[23809]: audit 2024-03-11T20:59:50.468760+0000 mon.a (mon.0) 1166 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:59:51.496 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:51 smithi146 bash[23809]: audit 2024-03-11T20:59:50.477695+0000 mon.a (mon.0) 1167 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:59:51.496 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:51 smithi146 bash[23809]: audit 2024-03-11T20:59:50.915312+0000 mon.a (mon.0) 1168 : audit 0 from='client.? 172.21.15.23:0/1275987913' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T20:59:51.510 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:51.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:51.512 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:59:51.536 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:51 smithi023 bash[21372]: cluster 2024-03-11T20:59:50.036512+0000 mgr.y (mgr.24380) 336 : cluster 0 pgmap v344: 148 pgs: 52 active+clean, 12 undersized+peered, 43 active+undersized, 12 undersized+degraded+peered, 29 active+undersized+degraded; 13 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 304/1182 objects degraded (25.719%) 2024-03-11T20:59:51.536 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:51 smithi023 bash[21372]: audit 2024-03-11T20:59:50.466987+0000 mon.a (mon.0) 1165 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:59:51.536 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:51 smithi023 bash[21372]: audit 2024-03-11T20:59:50.468760+0000 mon.a (mon.0) 1166 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:59:51.536 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:51 smithi023 bash[21372]: audit 2024-03-11T20:59:50.477695+0000 mon.a (mon.0) 1167 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:59:51.536 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:51 smithi023 bash[21372]: audit 2024-03-11T20:59:50.915312+0000 mon.a (mon.0) 1168 : audit 0 from='client.? 172.21.15.23:0/1275987913' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T20:59:51.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:51 smithi023 bash[29146]: cluster 2024-03-11T20:59:50.036512+0000 mgr.y (mgr.24380) 336 : cluster 0 pgmap v344: 148 pgs: 52 active+clean, 12 undersized+peered, 43 active+undersized, 12 undersized+degraded+peered, 29 active+undersized+degraded; 13 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 304/1182 objects degraded (25.719%) 2024-03-11T20:59:51.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:51 smithi023 bash[29146]: audit 2024-03-11T20:59:50.466987+0000 mon.a (mon.0) 1165 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:59:51.537 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:51 smithi023 bash[29146]: audit 2024-03-11T20:59:50.468760+0000 mon.a (mon.0) 1166 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:59:51.537 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:51 smithi023 bash[29146]: audit 2024-03-11T20:59:50.477695+0000 mon.a (mon.0) 1167 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T20:59:51.537 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:51 smithi023 bash[29146]: audit 2024-03-11T20:59:50.915312+0000 mon.a (mon.0) 1168 : audit 0 from='client.? 172.21.15.23:0/1275987913' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T20:59:51.612 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:51.642 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:51.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:51.645 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:59:51.746 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:51.774 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:51.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:51.776 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:59:51.877 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:51.904 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:51.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:51.906 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:59:52.007 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:52.035 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:52.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:52.037 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:59:52.138 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:52.166 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:52.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:52.169 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:59:52.270 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:52.296 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:52.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:52.298 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:59:52.399 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:52.407 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 22 16 212 196 0.556745 0 - 0.00475516 2024-03-11T20:59:52.426 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:52.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:52.429 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:59:52.530 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:52.556 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:52.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:52.559 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:59:52.660 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:52.689 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:52.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:52.691 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:59:52.792 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:52.821 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:52.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:52.823 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:59:52.924 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:52.950 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:52.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:52.953 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:59:52.972 DEBUG:teuthology.orchestra.run.smithi023:> sudo rm -f /var/log/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/exp.6.1a.1 2024-03-11T20:59:53.053 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:53.081 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:53.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:53.084 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:59:53.185 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:53.211 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:53.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:53.213 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:59:53.314 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:53.343 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:53.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:53.345 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:59:53.396 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-03-11T20:59:53.407 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 23 16 212 196 0.53254 0 - 0.00475516 2024-03-11T20:59:53.446 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:53.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:53 smithi146 bash[23809]: cluster 2024-03-11T20:59:52.037791+0000 mgr.y (mgr.24380) 337 : cluster 0 pgmap v345: 148 pgs: 52 active+clean, 12 undersized+peered, 43 active+undersized, 12 undersized+degraded+peered, 29 active+undersized+degraded; 13 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 304/1182 objects degraded (25.719%) 2024-03-11T20:59:53.473 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:53.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:53.475 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:59:53.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:53 smithi023 bash[21372]: cluster 2024-03-11T20:59:52.037791+0000 mgr.y (mgr.24380) 337 : cluster 0 pgmap v345: 148 pgs: 52 active+clean, 12 undersized+peered, 43 active+undersized, 12 undersized+degraded+peered, 29 active+undersized+degraded; 13 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 304/1182 objects degraded (25.719%) 2024-03-11T20:59:53.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:53 smithi023 bash[29146]: cluster 2024-03-11T20:59:52.037791+0000 mgr.y (mgr.24380) 337 : cluster 0 pgmap v345: 148 pgs: 52 active+clean, 12 undersized+peered, 43 active+undersized, 12 undersized+degraded+peered, 29 active+undersized+degraded; 13 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 304/1182 objects degraded (25.719%) 2024-03-11T20:59:53.576 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:53.596 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:53.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:53.598 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:59:53.699 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:53.726 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:53.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:53.729 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:59:53.830 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:53.859 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:53.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:53.861 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:59:53.962 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:53.990 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:53.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:53.993 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:59:54.094 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:54.120 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:54.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:54.123 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:59:54.128 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-11T20:59:54.129 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-11T20:59:54.224 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:54.247 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:54.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:54.249 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:59:54.350 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:54.369 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:54.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:54.372 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:59:54.407 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 24 16 212 196 0.510351 0 - 0.00475516 2024-03-11T20:59:54.473 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:54.484 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:54 smithi146 bash[23809]: cluster 2024-03-11T20:59:54.297410+0000 mon.a (mon.0) 1169 : cluster 3 Health check update: Reduced data availability: 23 pgs inactive (PG_AVAILABILITY) 2024-03-11T20:59:54.484 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:54 smithi146 bash[23809]: cluster 2024-03-11T20:59:54.297439+0000 mon.a (mon.0) 1170 : cluster 3 Health check update: Degraded data redundancy: 304/1182 objects degraded (25.719%), 41 pgs degraded, 85 pgs undersized (PG_DEGRADED) 2024-03-11T20:59:54.500 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:54.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:54.503 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:59:54.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:54 smithi023 bash[29146]: cluster 2024-03-11T20:59:54.297410+0000 mon.a (mon.0) 1169 : cluster 3 Health check update: Reduced data availability: 23 pgs inactive (PG_AVAILABILITY) 2024-03-11T20:59:54.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:54 smithi023 bash[29146]: cluster 2024-03-11T20:59:54.297439+0000 mon.a (mon.0) 1170 : cluster 3 Health check update: Degraded data redundancy: 304/1182 objects degraded (25.719%), 41 pgs degraded, 85 pgs undersized (PG_DEGRADED) 2024-03-11T20:59:54.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:54 smithi023 bash[21372]: cluster 2024-03-11T20:59:54.297410+0000 mon.a (mon.0) 1169 : cluster 3 Health check update: Reduced data availability: 23 pgs inactive (PG_AVAILABILITY) 2024-03-11T20:59:54.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:54 smithi023 bash[21372]: cluster 2024-03-11T20:59:54.297439+0000 mon.a (mon.0) 1170 : cluster 3 Health check update: Degraded data redundancy: 304/1182 objects degraded (25.719%), 41 pgs degraded, 85 pgs undersized (PG_DEGRADED) 2024-03-11T20:59:54.603 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:54.626 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:54.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:54.628 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:59:54.729 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:54.758 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:54.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:54.760 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:59:54.861 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:54.890 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:54.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:54.893 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:59:54.993 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:55.020 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:55.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:55.023 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:59:55.124 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:55.152 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:55.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:55.155 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:59:55.255 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:55.261 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 20:59:55 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T20:59:55.279 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:55.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:55.282 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:59:55.378 INFO:teuthology.orchestra.run.smithi023.stderr:noscrub is unset 2024-03-11T20:59:55.382 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:55.406 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:55.407 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 25 16 212 196 0.489938 0 - 0.00475516 2024-03-11T20:59:55.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:55.409 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:59:55.510 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:55.513 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:55 smithi146 bash[23809]: cluster 2024-03-11T20:59:54.039540+0000 mgr.y (mgr.24380) 338 : cluster 0 pgmap v346: 148 pgs: 52 active+clean, 12 undersized+peered, 43 active+undersized, 12 undersized+degraded+peered, 29 active+undersized+degraded; 13 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 304/1182 objects degraded (25.719%) 2024-03-11T20:59:55.513 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:55 smithi146 bash[23809]: audit 2024-03-11T20:59:54.496707+0000 mon.c (mon.2) 44 : audit 1 from='client.? 172.21.15.23:0/3406875887' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T20:59:55.513 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:55 smithi146 bash[23809]: audit 2024-03-11T20:59:54.497215+0000 mon.a (mon.0) 1171 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T20:59:55.534 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:55.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:55.536 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:59:55.637 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:55.663 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:55.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:55.666 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:59:55.766 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:55.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:55 smithi023 bash[21372]: cluster 2024-03-11T20:59:54.039540+0000 mgr.y (mgr.24380) 338 : cluster 0 pgmap v346: 148 pgs: 52 active+clean, 12 undersized+peered, 43 active+undersized, 12 undersized+degraded+peered, 29 active+undersized+degraded; 13 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 304/1182 objects degraded (25.719%) 2024-03-11T20:59:55.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:55 smithi023 bash[21372]: audit 2024-03-11T20:59:54.496707+0000 mon.c (mon.2) 44 : audit 1 from='client.? 172.21.15.23:0/3406875887' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T20:59:55.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:55 smithi023 bash[21372]: audit 2024-03-11T20:59:54.497215+0000 mon.a (mon.0) 1171 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T20:59:55.779 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:55 smithi023 bash[29146]: cluster 2024-03-11T20:59:54.039540+0000 mgr.y (mgr.24380) 338 : cluster 0 pgmap v346: 148 pgs: 52 active+clean, 12 undersized+peered, 43 active+undersized, 12 undersized+degraded+peered, 29 active+undersized+degraded; 13 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 304/1182 objects degraded (25.719%) 2024-03-11T20:59:55.779 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:55 smithi023 bash[29146]: audit 2024-03-11T20:59:54.496707+0000 mon.c (mon.2) 44 : audit 1 from='client.? 172.21.15.23:0/3406875887' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T20:59:55.779 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:55 smithi023 bash[29146]: audit 2024-03-11T20:59:54.497215+0000 mon.a (mon.0) 1171 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T20:59:55.794 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:55.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:55.797 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:59:55.898 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:55.926 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:55.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:55.929 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:59:56.030 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:56.059 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:56.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:56.061 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:59:56.130 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T20:59:56.163 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:56.201 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:56.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:56.204 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:59:56.305 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:56.331 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:56.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:56.334 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:59:56.408 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 26 16 212 196 0.471094 0 - 0.00475516 2024-03-11T20:59:56.434 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:56.462 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:56.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:56.464 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:59:56.564 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:56.591 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:56.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:56.594 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:59:56.695 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:56.706 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:56 smithi023 bash[29146]: audit 2024-03-11T20:59:55.252346+0000 mgr.y (mgr.24380) 339 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:59:56.706 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:56 smithi023 bash[29146]: cluster 2024-03-11T20:59:55.350804+0000 mon.a (mon.0) 1172 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T20:59:56.706 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:56 smithi023 bash[29146]: audit 2024-03-11T20:59:55.366865+0000 mon.a (mon.0) 1173 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T20:59:56.706 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:56 smithi023 bash[29146]: cluster 2024-03-11T20:59:55.377996+0000 mon.a (mon.0) 1174 : cluster 0 osdmap e195: 8 total, 6 up, 8 in 2024-03-11T20:59:56.706 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:56 smithi023 bash[29146]: cluster 2024-03-11T20:59:56.260969+0000 osd.2 (osd.2) 142 : cluster 0 2.c scrub starts 2024-03-11T20:59:56.706 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:56 smithi023 bash[29146]: cluster 2024-03-11T20:59:56.262739+0000 osd.2 (osd.2) 143 : cluster 0 2.c scrub ok 2024-03-11T20:59:56.707 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:56 smithi023 bash[21372]: audit 2024-03-11T20:59:55.252346+0000 mgr.y (mgr.24380) 339 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:59:56.707 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:56 smithi023 bash[21372]: cluster 2024-03-11T20:59:55.350804+0000 mon.a (mon.0) 1172 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T20:59:56.707 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:56 smithi023 bash[21372]: audit 2024-03-11T20:59:55.366865+0000 mon.a (mon.0) 1173 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T20:59:56.707 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:56 smithi023 bash[21372]: cluster 2024-03-11T20:59:55.377996+0000 mon.a (mon.0) 1174 : cluster 0 osdmap e195: 8 total, 6 up, 8 in 2024-03-11T20:59:56.707 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:56 smithi023 bash[21372]: cluster 2024-03-11T20:59:56.260969+0000 osd.2 (osd.2) 142 : cluster 0 2.c scrub starts 2024-03-11T20:59:56.707 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:56 smithi023 bash[21372]: cluster 2024-03-11T20:59:56.262739+0000 osd.2 (osd.2) 143 : cluster 0 2.c scrub ok 2024-03-11T20:59:56.721 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:56.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:56.723 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:59:56.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:56 smithi146 bash[23809]: audit 2024-03-11T20:59:55.252346+0000 mgr.y (mgr.24380) 339 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:59:56.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:56 smithi146 bash[23809]: cluster 2024-03-11T20:59:55.350804+0000 mon.a (mon.0) 1172 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T20:59:56.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:56 smithi146 bash[23809]: audit 2024-03-11T20:59:55.366865+0000 mon.a (mon.0) 1173 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T20:59:56.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:56 smithi146 bash[23809]: cluster 2024-03-11T20:59:55.377996+0000 mon.a (mon.0) 1174 : cluster 0 osdmap e195: 8 total, 6 up, 8 in 2024-03-11T20:59:56.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:56 smithi146 bash[23809]: cluster 2024-03-11T20:59:56.260969+0000 osd.2 (osd.2) 142 : cluster 0 2.c scrub starts 2024-03-11T20:59:56.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:56 smithi146 bash[23809]: cluster 2024-03-11T20:59:56.262739+0000 osd.2 (osd.2) 143 : cluster 0 2.c scrub ok 2024-03-11T20:59:56.824 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:56.850 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:56.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:56.853 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:59:56.953 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:56.984 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:56.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:56.986 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:59:57.087 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:57.116 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:57.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:57.118 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:59:57.219 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:57.246 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:57.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:57.249 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:59:57.350 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:57.378 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:57.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:57.380 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:59:57.408 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 27 16 212 196 0.453647 0 - 0.00475516 2024-03-11T20:59:57.481 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:57.493 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:57 smithi023 bash[21372]: cluster 2024-03-11T20:59:56.041468+0000 mgr.y (mgr.24380) 340 : cluster 0 pgmap v348: 148 pgs: 52 active+clean, 12 undersized+peered, 43 active+undersized, 12 undersized+degraded+peered, 29 active+undersized+degraded; 13 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 304/1182 objects degraded (25.719%) 2024-03-11T20:59:57.493 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:57 smithi023 bash[29146]: cluster 2024-03-11T20:59:56.041468+0000 mgr.y (mgr.24380) 340 : cluster 0 pgmap v348: 148 pgs: 52 active+clean, 12 undersized+peered, 43 active+undersized, 12 undersized+degraded+peered, 29 active+undersized+degraded; 13 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 304/1182 objects degraded (25.719%) 2024-03-11T20:59:57.507 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:57.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:57.509 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:59:57.610 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:57.638 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:57.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:57.641 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:59:57.742 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:57.753 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:57 smithi146 bash[23809]: cluster 2024-03-11T20:59:56.041468+0000 mgr.y (mgr.24380) 340 : cluster 0 pgmap v348: 148 pgs: 52 active+clean, 12 undersized+peered, 43 active+undersized, 12 undersized+degraded+peered, 29 active+undersized+degraded; 13 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 304/1182 objects degraded (25.719%) 2024-03-11T20:59:57.770 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:57.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:57.773 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:59:57.873 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:57.900 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T20:59:57.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:57.902 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T20:59:57.988 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 6, 7, 1] out_osds: [] dead_osds: [1, 4] live_osds: [0, 2, 3, 5, 6, 7] 2024-03-11T20:59:57.988 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-11T20:59:57.988 INFO:tasks.thrashosds.thrasher:Shrinking pool unique_pool_1 2024-03-11T20:59:57.989 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool unique_pool_1 by 10 min 8 2024-03-11T20:59:57.989 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T20:59:58.003 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:58.030 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:58.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:58.033 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:59:58.133 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:58.160 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:58.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:58.163 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:59:58.263 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:58.275 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-11T20:59:58.286 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:58.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:58.288 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:59:58.389 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:58.408 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 28 16 212 196 0.437445 0 - 0.00475516 2024-03-11T20:59:58.414 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:58.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:58.416 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:59:58.517 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:58.527 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:58 smithi023 bash[21372]: cluster 2024-03-11T20:59:57.182988+0000 osd.0 (osd.0) 167 : cluster 0 3.1f scrub starts 2024-03-11T20:59:58.527 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:58 smithi023 bash[21372]: cluster 2024-03-11T20:59:57.186747+0000 osd.0 (osd.0) 168 : cluster 0 3.1f scrub ok 2024-03-11T20:59:58.527 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 20:59:58 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:20:59:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T20:59:58.527 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:58 smithi023 bash[29146]: cluster 2024-03-11T20:59:57.182988+0000 osd.0 (osd.0) 167 : cluster 0 3.1f scrub starts 2024-03-11T20:59:58.527 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:58 smithi023 bash[29146]: cluster 2024-03-11T20:59:57.186747+0000 osd.0 (osd.0) 168 : cluster 0 3.1f scrub ok 2024-03-11T20:59:58.541 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:58.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:58.543 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:59:58.644 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:58.655 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:58 smithi146 bash[23809]: cluster 2024-03-11T20:59:57.182988+0000 osd.0 (osd.0) 167 : cluster 0 3.1f scrub starts 2024-03-11T20:59:58.656 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:58 smithi146 bash[23809]: cluster 2024-03-11T20:59:57.186747+0000 osd.0 (osd.0) 168 : cluster 0 3.1f scrub ok 2024-03-11T20:59:58.671 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T20:59:58.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:58.673 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T20:59:58.770 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:59:58.774 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:58.799 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T20:59:58.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:58.802 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T20:59:58.903 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:58.930 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:58.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:58.932 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:59:59.033 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:59.062 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T20:59:59.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:59.064 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T20:59:59.165 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:59.192 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:59.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:59.194 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:59:59.295 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:59.322 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:59.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:59.324 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:59:59.408 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 29 16 212 196 0.422362 0 - 0.00475516 2024-03-11T20:59:59.425 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:59.454 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:59.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:59.456 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:59:59.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 20:59:59 smithi146 bash[23809]: cluster 2024-03-11T20:59:58.043364+0000 mgr.y (mgr.24380) 341 : cluster 0 pgmap v349: 148 pgs: 52 active+clean, 12 undersized+peered, 43 active+undersized, 12 undersized+degraded+peered, 29 active+undersized+degraded; 13 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 304/1182 objects degraded (25.719%) 2024-03-11T20:59:59.557 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:59.584 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:59.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:59.587 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:59:59.688 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:59.714 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T20:59:59.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:59.717 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T20:59:59.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 20:59:59 smithi023 bash[21372]: cluster 2024-03-11T20:59:58.043364+0000 mgr.y (mgr.24380) 341 : cluster 0 pgmap v349: 148 pgs: 52 active+clean, 12 undersized+peered, 43 active+undersized, 12 undersized+degraded+peered, 29 active+undersized+degraded; 13 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 304/1182 objects degraded (25.719%) 2024-03-11T20:59:59.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 20:59:59 smithi023 bash[29146]: cluster 2024-03-11T20:59:58.043364+0000 mgr.y (mgr.24380) 341 : cluster 0 pgmap v349: 148 pgs: 52 active+clean, 12 undersized+peered, 43 active+undersized, 12 undersized+degraded+peered, 29 active+undersized+degraded; 13 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 304/1182 objects degraded (25.719%) 2024-03-11T20:59:59.818 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:59.838 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:59.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:59.841 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T20:59:59.941 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:59.963 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T20:59:59.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:59:59.965 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:00:00.066 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:00:00.089 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:00:00.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:00.091 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:00:00.192 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:00:00.218 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:00:00.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:00.220 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:00:00.321 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:00.347 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:00.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:00.350 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:00:00.408 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 30 16 212 196 0.408282 0 - 0.00475516 2024-03-11T21:00:00.451 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:00.479 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:00.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:00.481 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:00:00.582 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:00.611 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:00.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:00.613 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:00:00.714 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:00.725 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[21372]: cluster 2024-03-11T21:00:00.000218+0000 mon.a (mon.0) 1175 : cluster 3 Health detail: HEALTH_WARN nodeep-scrub flag(s) set; 2 osds down; Reduced data availability: 23 pgs inactive; Degraded data redundancy: 304/1182 objects degraded (25.719%), 41 pgs degraded, 85 pgs undersized 2024-03-11T21:00:00.725 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[21372]: cluster 2024-03-11T21:00:00.000279+0000 mon.a (mon.0) 1176 : cluster 3 [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-03-11T21:00:00.725 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[21372]: cluster 2024-03-11T21:00:00.000299+0000 mon.a (mon.0) 1177 : cluster 3 [WRN] OSD_DOWN: 2 osds down 2024-03-11T21:00:00.725 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[21372]: cluster 2024-03-11T21:00:00.000348+0000 mon.a (mon.0) 1178 : cluster 3 osd.1 (root=default,host=smithi023) is down 2024-03-11T21:00:00.725 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[21372]: cluster 2024-03-11T21:00:00.000369+0000 mon.a (mon.0) 1179 : cluster 3 osd.4 (root=default,host=smithi146) is down 2024-03-11T21:00:00.725 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[21372]: cluster 2024-03-11T21:00:00.000393+0000 mon.a (mon.0) 1180 : cluster 3 [WRN] PG_AVAILABILITY: Reduced data availability: 23 pgs inactive 2024-03-11T21:00:00.726 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[21372]: cluster 2024-03-11T21:00:00.000441+0000 mon.a (mon.0) 1181 : cluster 3 pg 2.6 is stuck inactive for 65s, current state undersized+degraded+peered, last acting [6] 2024-03-11T21:00:00.726 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[21372]: cluster 2024-03-11T21:00:00.000466+0000 mon.a (mon.0) 1182 : cluster 3 pg 2.d is stuck inactive for 9m, current state undersized+peered, last acting [3] 2024-03-11T21:00:00.726 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[21372]: cluster 2024-03-11T21:00:00.000494+0000 mon.a (mon.0) 1183 : cluster 3 pg 2.11 is stuck inactive for 65s, current state undersized+degraded+peered, last acting [6] 2024-03-11T21:00:00.726 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[21372]: cluster 2024-03-11T21:00:00.000502+0000 mon.a (mon.0) 1184 : cluster 3 pg 3.6 is stuck inactive for 65s, current state undersized+degraded+peered, last acting [0] 2024-03-11T21:00:00.726 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[21372]: cluster 2024-03-11T21:00:00.000511+0000 mon.a (mon.0) 1185 : cluster 3 pg 3.a is stuck inactive for 65s, current state undersized+degraded+peered, last acting [6] 2024-03-11T21:00:00.726 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[21372]: cluster 2024-03-11T21:00:00.000522+0000 mon.a (mon.0) 1186 : cluster 3 pg 3.e is stuck inactive for 65s, current state undersized+degraded+peered, last acting [7] 2024-03-11T21:00:00.726 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[21372]: cluster 2024-03-11T21:00:00.000531+0000 mon.a (mon.0) 1187 : cluster 3 pg 3.19 is stuck inactive for 65s, current state undersized+degraded+peered, last acting [3] 2024-03-11T21:00:00.726 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[21372]: cluster 2024-03-11T21:00:00.000541+0000 mon.a (mon.0) 1188 : cluster 3 pg 3.1a is stuck inactive for 65s, current state undersized+degraded+peered, last acting [2] 2024-03-11T21:00:00.726 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[21372]: cluster 2024-03-11T21:00:00.000552+0000 mon.a (mon.0) 1189 : cluster 3 pg 3.1c is stuck inactive for 65s, current state undersized+degraded+peered, last acting [5] 2024-03-11T21:00:00.726 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[21372]: cluster 2024-03-11T21:00:00.000563+0000 mon.a (mon.0) 1190 : cluster 3 pg 4.2 is stuck inactive for 65s, current state undersized+peered, last acting [5] 2024-03-11T21:00:00.726 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[21372]: cluster 2024-03-11T21:00:00.000575+0000 mon.a (mon.0) 1191 : cluster 3 pg 4.9 is stuck inactive for 65s, current state undersized+degraded+peered, last acting [3] 2024-03-11T21:00:00.726 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[21372]: cluster 2024-03-11T21:00:00.000585+0000 mon.a (mon.0) 1192 : cluster 3 pg 4.b is stuck inactive for 65s, current state undersized+peered, last acting [0] 2024-03-11T21:00:00.727 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[21372]: cluster 2024-03-11T21:00:00.000593+0000 mon.a (mon.0) 1193 : cluster 3 pg 4.d is stuck inactive for 65s, current state undersized+degraded+peered, last acting [2] 2024-03-11T21:00:00.727 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[21372]: cluster 2024-03-11T21:00:00.000608+0000 mon.a (mon.0) 1194 : cluster 3 pg 4.e is stuck inactive for 65s, current state undersized+degraded+peered, last acting [6] 2024-03-11T21:00:00.727 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[21372]: cluster 2024-03-11T21:00:00.000617+0000 mon.a (mon.0) 1195 : cluster 3 pg 4.f is stuck inactive for 65s, current state undersized+peered, last acting [3] 2024-03-11T21:00:00.727 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[21372]: cluster 2024-03-11T21:00:00.000629+0000 mon.a (mon.0) 1196 : cluster 3 pg 4.13 is stuck inactive for 118s, current state undersized+peered, last acting [6] 2024-03-11T21:00:00.727 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[21372]: cluster 2024-03-11T21:00:00.000639+0000 mon.a (mon.0) 1197 : cluster 3 pg 4.18 is stuck inactive for 118s, current state undersized+peered, last acting [6] 2024-03-11T21:00:00.727 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[21372]: cluster 2024-03-11T21:00:00.000660+0000 mon.a (mon.0) 1198 : cluster 3 pg 4.1b is stuck inactive for 118s, current state undersized+peered, last acting [3] 2024-03-11T21:00:00.727 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[21372]: cluster 2024-03-11T21:00:00.000671+0000 mon.a (mon.0) 1199 : cluster 3 pg 4.1d is stuck inactive for 65s, current state undersized+peered, last acting [5] 2024-03-11T21:00:00.727 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[21372]: cluster 2024-03-11T21:00:00.000681+0000 mon.a (mon.0) 1200 : cluster 3 pg 5.0 is stuck inactive for 65s, current state undersized+degraded+peered, last acting [3] 2024-03-11T21:00:00.727 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[21372]: cluster 2024-03-11T21:00:00.000690+0000 mon.a (mon.0) 1201 : cluster 3 pg 6.e is stuck inactive for 118s, current state undersized+peered, last acting [2] 2024-03-11T21:00:00.727 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[21372]: cluster 2024-03-11T21:00:00.000701+0000 mon.a (mon.0) 1202 : cluster 3 pg 6.1a is stuck inactive for 118s, current state undersized+peered, last acting [5] 2024-03-11T21:00:00.727 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[21372]: cluster 2024-03-11T21:00:00.000710+0000 mon.a (mon.0) 1203 : cluster 3 pg 6.1d is stuck inactive for 65s, current state undersized+peered, last acting [5] 2024-03-11T21:00:00.727 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[21372]: cluster 2024-03-11T21:00:00.000721+0000 mon.a (mon.0) 1204 : cluster 3 [WRN] PG_DEGRADED: Degraded data redundancy: 304/1182 objects degraded (25.719%), 41 pgs degraded, 85 pgs undersized 2024-03-11T21:00:00.727 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[21372]: cluster 2024-03-11T21:00:00.000731+0000 mon.a (mon.0) 1205 : cluster 3 pg 2.2 is stuck undersized for 118s, current state active+undersized, last acting [5,6] 2024-03-11T21:00:00.728 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[21372]: cluster 2024-03-11T21:00:00.000741+0000 mon.a (mon.0) 1206 : cluster 3 pg 2.4 is stuck undersized for 118s, current state active+undersized, last acting [7,0] 2024-03-11T21:00:00.728 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[21372]: cluster 2024-03-11T21:00:00.000751+0000 mon.a (mon.0) 1207 : cluster 3 pg 2.f is stuck undersized for 64s, current state active+undersized+degraded, last acting [7,0] 2024-03-11T21:00:00.728 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[21372]: cluster 2024-03-11T21:00:00.000761+0000 mon.a (mon.0) 1208 : cluster 3 pg 2.10 is stuck undersized for 118s, current state active+undersized, last acting [2,0] 2024-03-11T21:00:00.728 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[21372]: cluster 2024-03-11T21:00:00.000775+0000 mon.a (mon.0) 1209 : cluster 3 pg 2.11 is stuck undersized for 64s, current state undersized+degraded+peered, last acting [6] 2024-03-11T21:00:00.728 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[21372]: cluster 2024-03-11T21:00:00.000786+0000 mon.a (mon.0) 1210 : cluster 3 pg 2.13 is stuck undersized for 64s, current state active+undersized, last acting [3,0] 2024-03-11T21:00:00.728 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[21372]: cluster 2024-03-11T21:00:00.000794+0000 mon.a (mon.0) 1211 : cluster 3 pg 2.15 is stuck undersized for 118s, current state active+undersized, last acting [5,0] 2024-03-11T21:00:00.728 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[21372]: cluster 2024-03-11T21:00:00.000802+0000 mon.a (mon.0) 1212 : cluster 3 pg 2.18 is stuck undersized for 64s, current state active+undersized, last acting [5,7] 2024-03-11T21:00:00.728 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[21372]: cluster 2024-03-11T21:00:00.000813+0000 mon.a (mon.0) 1213 : cluster 3 pg 2.1a is stuck undersized for 64s, current state active+undersized, last acting [6,7] 2024-03-11T21:00:00.728 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[21372]: cluster 2024-03-11T21:00:00.000823+0000 mon.a (mon.0) 1214 : cluster 3 pg 2.1f is stuck undersized for 64s, current state active+undersized, last acting [7,0] 2024-03-11T21:00:00.728 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[29146]: cluster 2024-03-11T21:00:00.000218+0000 mon.a (mon.0) 1175 : cluster 3 Health detail: HEALTH_WARN nodeep-scrub flag(s) set; 2 osds down; Reduced data availability: 23 pgs inactive; Degraded data redundancy: 304/1182 objects degraded (25.719%), 41 pgs degraded, 85 pgs undersized 2024-03-11T21:00:00.728 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[29146]: cluster 2024-03-11T21:00:00.000279+0000 mon.a (mon.0) 1176 : cluster 3 [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-03-11T21:00:00.729 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[29146]: cluster 2024-03-11T21:00:00.000299+0000 mon.a (mon.0) 1177 : cluster 3 [WRN] OSD_DOWN: 2 osds down 2024-03-11T21:00:00.729 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[29146]: cluster 2024-03-11T21:00:00.000348+0000 mon.a (mon.0) 1178 : cluster 3 osd.1 (root=default,host=smithi023) is down 2024-03-11T21:00:00.729 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[29146]: cluster 2024-03-11T21:00:00.000369+0000 mon.a (mon.0) 1179 : cluster 3 osd.4 (root=default,host=smithi146) is down 2024-03-11T21:00:00.729 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[29146]: cluster 2024-03-11T21:00:00.000393+0000 mon.a (mon.0) 1180 : cluster 3 [WRN] PG_AVAILABILITY: Reduced data availability: 23 pgs inactive 2024-03-11T21:00:00.729 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[29146]: cluster 2024-03-11T21:00:00.000441+0000 mon.a (mon.0) 1181 : cluster 3 pg 2.6 is stuck inactive for 65s, current state undersized+degraded+peered, last acting [6] 2024-03-11T21:00:00.729 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[29146]: cluster 2024-03-11T21:00:00.000466+0000 mon.a (mon.0) 1182 : cluster 3 pg 2.d is stuck inactive for 9m, current state undersized+peered, last acting [3] 2024-03-11T21:00:00.729 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[29146]: cluster 2024-03-11T21:00:00.000494+0000 mon.a (mon.0) 1183 : cluster 3 pg 2.11 is stuck inactive for 65s, current state undersized+degraded+peered, last acting [6] 2024-03-11T21:00:00.729 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[29146]: cluster 2024-03-11T21:00:00.000502+0000 mon.a (mon.0) 1184 : cluster 3 pg 3.6 is stuck inactive for 65s, current state undersized+degraded+peered, last acting [0] 2024-03-11T21:00:00.729 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[29146]: cluster 2024-03-11T21:00:00.000511+0000 mon.a (mon.0) 1185 : cluster 3 pg 3.a is stuck inactive for 65s, current state undersized+degraded+peered, last acting [6] 2024-03-11T21:00:00.729 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[29146]: cluster 2024-03-11T21:00:00.000522+0000 mon.a (mon.0) 1186 : cluster 3 pg 3.e is stuck inactive for 65s, current state undersized+degraded+peered, last acting [7] 2024-03-11T21:00:00.729 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[29146]: cluster 2024-03-11T21:00:00.000531+0000 mon.a (mon.0) 1187 : cluster 3 pg 3.19 is stuck inactive for 65s, current state undersized+degraded+peered, last acting [3] 2024-03-11T21:00:00.729 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[29146]: cluster 2024-03-11T21:00:00.000541+0000 mon.a (mon.0) 1188 : cluster 3 pg 3.1a is stuck inactive for 65s, current state undersized+degraded+peered, last acting [2] 2024-03-11T21:00:00.729 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[29146]: cluster 2024-03-11T21:00:00.000552+0000 mon.a (mon.0) 1189 : cluster 3 pg 3.1c is stuck inactive for 65s, current state undersized+degraded+peered, last acting [5] 2024-03-11T21:00:00.729 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[29146]: cluster 2024-03-11T21:00:00.000563+0000 mon.a (mon.0) 1190 : cluster 3 pg 4.2 is stuck inactive for 65s, current state undersized+peered, last acting [5] 2024-03-11T21:00:00.730 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[29146]: cluster 2024-03-11T21:00:00.000575+0000 mon.a (mon.0) 1191 : cluster 3 pg 4.9 is stuck inactive for 65s, current state undersized+degraded+peered, last acting [3] 2024-03-11T21:00:00.730 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[29146]: cluster 2024-03-11T21:00:00.000585+0000 mon.a (mon.0) 1192 : cluster 3 pg 4.b is stuck inactive for 65s, current state undersized+peered, last acting [0] 2024-03-11T21:00:00.730 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[29146]: cluster 2024-03-11T21:00:00.000593+0000 mon.a (mon.0) 1193 : cluster 3 pg 4.d is stuck inactive for 65s, current state undersized+degraded+peered, last acting [2] 2024-03-11T21:00:00.730 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[29146]: cluster 2024-03-11T21:00:00.000608+0000 mon.a (mon.0) 1194 : cluster 3 pg 4.e is stuck inactive for 65s, current state undersized+degraded+peered, last acting [6] 2024-03-11T21:00:00.730 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[21372]: cluster 2024-03-11T21:00:00.000833+0000 mon.a (mon.0) 1215 : cluster 3 pg 3.3 is stuck undersized for 64s, current state active+undersized+degraded, last acting [6,0] 2024-03-11T21:00:00.730 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[21372]: cluster 2024-03-11T21:00:00.000842+0000 mon.a (mon.0) 1216 : cluster 3 pg 3.e is stuck undersized for 64s, current state undersized+degraded+peered, last acting [7] 2024-03-11T21:00:00.730 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[21372]: cluster 2024-03-11T21:00:00.000852+0000 mon.a (mon.0) 1217 : cluster 3 pg 3.11 is stuck undersized for 64s, current state active+undersized+degraded, last acting [7,6] 2024-03-11T21:00:00.730 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[21372]: cluster 2024-03-11T21:00:00.000864+0000 mon.a (mon.0) 1218 : cluster 3 pg 3.13 is stuck undersized for 64s, current state active+undersized+degraded, last acting [7,2] 2024-03-11T21:00:00.730 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[21372]: cluster 2024-03-11T21:00:00.000875+0000 mon.a (mon.0) 1219 : cluster 3 pg 3.14 is stuck undersized for 64s, current state active+undersized+degraded, last acting [7,6] 2024-03-11T21:00:00.730 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[21372]: cluster 2024-03-11T21:00:00.000884+0000 mon.a (mon.0) 1220 : cluster 3 pg 3.15 is stuck undersized for 64s, current state active+undersized+degraded, last acting [7,3] 2024-03-11T21:00:00.730 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[21372]: cluster 2024-03-11T21:00:00.000892+0000 mon.a (mon.0) 1221 : cluster 3 pg 3.16 is stuck undersized for 118s, current state active+undersized+degraded, last acting [5,7] 2024-03-11T21:00:00.731 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[21372]: cluster 2024-03-11T21:00:00.000902+0000 mon.a (mon.0) 1222 : cluster 3 pg 3.18 is stuck undersized for 118s, current state active+undersized+degraded, last acting [3,0] 2024-03-11T21:00:00.731 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[21372]: cluster 2024-03-11T21:00:00.000914+0000 mon.a (mon.0) 1223 : cluster 3 pg 3.19 is stuck undersized for 64s, current state undersized+degraded+peered, last acting [3] 2024-03-11T21:00:00.731 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[21372]: cluster 2024-03-11T21:00:00.000933+0000 mon.a (mon.0) 1224 : cluster 3 pg 3.1a is stuck undersized for 64s, current state undersized+degraded+peered, last acting [2] 2024-03-11T21:00:00.731 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[21372]: cluster 2024-03-11T21:00:00.000950+0000 mon.a (mon.0) 1225 : cluster 3 pg 3.1b is stuck undersized for 64s, current state active+undersized+degraded, last acting [0,7] 2024-03-11T21:00:00.731 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[21372]: cluster 2024-03-11T21:00:00.000960+0000 mon.a (mon.0) 1226 : cluster 3 pg 3.1c is stuck undersized for 64s, current state undersized+degraded+peered, last acting [5] 2024-03-11T21:00:00.731 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[21372]: cluster 2024-03-11T21:00:00.000974+0000 mon.a (mon.0) 1227 : cluster 3 pg 3.1d is stuck undersized for 64s, current state active+undersized+degraded, last acting [5,6] 2024-03-11T21:00:00.731 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[21372]: cluster 2024-03-11T21:00:00.000983+0000 mon.a (mon.0) 1228 : cluster 3 pg 4.2 is stuck undersized for 64s, current state undersized+peered, last acting [5] 2024-03-11T21:00:00.731 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[21372]: cluster 2024-03-11T21:00:00.000993+0000 mon.a (mon.0) 1229 : cluster 3 pg 4.4 is stuck undersized for 118s, current state active+undersized, last acting [2,3] 2024-03-11T21:00:00.731 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[21372]: cluster 2024-03-11T21:00:00.001003+0000 mon.a (mon.0) 1230 : cluster 3 pg 4.9 is stuck undersized for 64s, current state undersized+degraded+peered, last acting [3] 2024-03-11T21:00:00.731 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[21372]: cluster 2024-03-11T21:00:00.001013+0000 mon.a (mon.0) 1231 : cluster 3 pg 4.10 is stuck undersized for 118s, current state active+undersized, last acting [3,6] 2024-03-11T21:00:00.731 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[21372]: cluster 2024-03-11T21:00:00.001023+0000 mon.a (mon.0) 1232 : cluster 3 pg 4.12 is stuck undersized for 118s, current state active+undersized, last acting [6,2] 2024-03-11T21:00:00.731 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[21372]: cluster 2024-03-11T21:00:00.001034+0000 mon.a (mon.0) 1233 : cluster 3 pg 4.13 is stuck undersized for 64s, current state undersized+peered, last acting [6] 2024-03-11T21:00:00.731 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[21372]: cluster 2024-03-11T21:00:00.001045+0000 mon.a (mon.0) 1234 : cluster 3 pg 4.14 is stuck undersized for 118s, current state active+undersized+degraded, last acting [3,7] 2024-03-11T21:00:00.732 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[29146]: cluster 2024-03-11T21:00:00.000617+0000 mon.a (mon.0) 1195 : cluster 3 pg 4.f is stuck inactive for 65s, current state undersized+peered, last acting [3] 2024-03-11T21:00:00.732 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[29146]: cluster 2024-03-11T21:00:00.000629+0000 mon.a (mon.0) 1196 : cluster 3 pg 4.13 is stuck inactive for 118s, current state undersized+peered, last acting [6] 2024-03-11T21:00:00.732 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[29146]: cluster 2024-03-11T21:00:00.000639+0000 mon.a (mon.0) 1197 : cluster 3 pg 4.18 is stuck inactive for 118s, current state undersized+peered, last acting [6] 2024-03-11T21:00:00.732 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[29146]: cluster 2024-03-11T21:00:00.000660+0000 mon.a (mon.0) 1198 : cluster 3 pg 4.1b is stuck inactive for 118s, current state undersized+peered, last acting [3] 2024-03-11T21:00:00.732 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[29146]: cluster 2024-03-11T21:00:00.000671+0000 mon.a (mon.0) 1199 : cluster 3 pg 4.1d is stuck inactive for 65s, current state undersized+peered, last acting [5] 2024-03-11T21:00:00.732 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[29146]: cluster 2024-03-11T21:00:00.000681+0000 mon.a (mon.0) 1200 : cluster 3 pg 5.0 is stuck inactive for 65s, current state undersized+degraded+peered, last acting [3] 2024-03-11T21:00:00.732 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[29146]: cluster 2024-03-11T21:00:00.000690+0000 mon.a (mon.0) 1201 : cluster 3 pg 6.e is stuck inactive for 118s, current state undersized+peered, last acting [2] 2024-03-11T21:00:00.732 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[29146]: cluster 2024-03-11T21:00:00.000701+0000 mon.a (mon.0) 1202 : cluster 3 pg 6.1a is stuck inactive for 118s, current state undersized+peered, last acting [5] 2024-03-11T21:00:00.732 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[29146]: cluster 2024-03-11T21:00:00.000710+0000 mon.a (mon.0) 1203 : cluster 3 pg 6.1d is stuck inactive for 65s, current state undersized+peered, last acting [5] 2024-03-11T21:00:00.732 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[29146]: cluster 2024-03-11T21:00:00.000721+0000 mon.a (mon.0) 1204 : cluster 3 [WRN] PG_DEGRADED: Degraded data redundancy: 304/1182 objects degraded (25.719%), 41 pgs degraded, 85 pgs undersized 2024-03-11T21:00:00.732 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[29146]: cluster 2024-03-11T21:00:00.000731+0000 mon.a (mon.0) 1205 : cluster 3 pg 2.2 is stuck undersized for 118s, current state active+undersized, last acting [5,6] 2024-03-11T21:00:00.732 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[29146]: cluster 2024-03-11T21:00:00.000741+0000 mon.a (mon.0) 1206 : cluster 3 pg 2.4 is stuck undersized for 118s, current state active+undersized, last acting [7,0] 2024-03-11T21:00:00.733 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[29146]: cluster 2024-03-11T21:00:00.000751+0000 mon.a (mon.0) 1207 : cluster 3 pg 2.f is stuck undersized for 64s, current state active+undersized+degraded, last acting [7,0] 2024-03-11T21:00:00.733 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[29146]: cluster 2024-03-11T21:00:00.000761+0000 mon.a (mon.0) 1208 : cluster 3 pg 2.10 is stuck undersized for 118s, current state active+undersized, last acting [2,0] 2024-03-11T21:00:00.733 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[29146]: cluster 2024-03-11T21:00:00.000775+0000 mon.a (mon.0) 1209 : cluster 3 pg 2.11 is stuck undersized for 64s, current state undersized+degraded+peered, last acting [6] 2024-03-11T21:00:00.733 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[29146]: cluster 2024-03-11T21:00:00.000786+0000 mon.a (mon.0) 1210 : cluster 3 pg 2.13 is stuck undersized for 64s, current state active+undersized, last acting [3,0] 2024-03-11T21:00:00.733 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[29146]: cluster 2024-03-11T21:00:00.000794+0000 mon.a (mon.0) 1211 : cluster 3 pg 2.15 is stuck undersized for 118s, current state active+undersized, last acting [5,0] 2024-03-11T21:00:00.733 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[29146]: cluster 2024-03-11T21:00:00.000802+0000 mon.a (mon.0) 1212 : cluster 3 pg 2.18 is stuck undersized for 64s, current state active+undersized, last acting [5,7] 2024-03-11T21:00:00.733 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[29146]: cluster 2024-03-11T21:00:00.000813+0000 mon.a (mon.0) 1213 : cluster 3 pg 2.1a is stuck undersized for 64s, current state active+undersized, last acting [6,7] 2024-03-11T21:00:00.733 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[29146]: cluster 2024-03-11T21:00:00.000823+0000 mon.a (mon.0) 1214 : cluster 3 pg 2.1f is stuck undersized for 64s, current state active+undersized, last acting [7,0] 2024-03-11T21:00:00.733 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[21372]: cluster 2024-03-11T21:00:00.001057+0000 mon.a (mon.0) 1235 : cluster 3 pg 4.17 is stuck undersized for 118s, current state active+undersized, last acting [3,7] 2024-03-11T21:00:00.733 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[21372]: cluster 2024-03-11T21:00:00.001067+0000 mon.a (mon.0) 1236 : cluster 3 pg 4.18 is stuck undersized for 64s, current state undersized+peered, last acting [6] 2024-03-11T21:00:00.733 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[21372]: cluster 2024-03-11T21:00:00.001077+0000 mon.a (mon.0) 1237 : cluster 3 pg 4.1a is stuck undersized for 64s, current state active+undersized, last acting [3,0] 2024-03-11T21:00:00.734 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[21372]: cluster 2024-03-11T21:00:00.001087+0000 mon.a (mon.0) 1238 : cluster 3 pg 4.1b is stuck undersized for 64s, current state undersized+peered, last acting [3] 2024-03-11T21:00:00.734 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[21372]: cluster 2024-03-11T21:00:00.001099+0000 mon.a (mon.0) 1239 : cluster 3 pg 4.1c is stuck undersized for 118s, current state active+undersized, last acting [2,3] 2024-03-11T21:00:00.734 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[21372]: cluster 2024-03-11T21:00:00.001109+0000 mon.a (mon.0) 1240 : cluster 3 pg 4.1d is stuck undersized for 64s, current state undersized+peered, last acting [5] 2024-03-11T21:00:00.734 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[21372]: cluster 2024-03-11T21:00:00.001118+0000 mon.a (mon.0) 1241 : cluster 3 pg 4.1f is stuck undersized for 118s, current state active+undersized+degraded, last acting [6,5] 2024-03-11T21:00:00.734 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[21372]: cluster 2024-03-11T21:00:00.001128+0000 mon.a (mon.0) 1242 : cluster 3 pg 6.6 is stuck undersized for 64s, current state active+undersized, last acting [3,7] 2024-03-11T21:00:00.734 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[21372]: cluster 2024-03-11T21:00:00.001139+0000 mon.a (mon.0) 1243 : cluster 3 pg 6.b is stuck undersized for 118s, current state active+undersized, last acting [3,7] 2024-03-11T21:00:00.734 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[21372]: cluster 2024-03-11T21:00:00.001149+0000 mon.a (mon.0) 1244 : cluster 3 pg 6.10 is stuck undersized for 118s, current state active+undersized, last acting [0,5] 2024-03-11T21:00:00.734 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[21372]: cluster 2024-03-11T21:00:00.001159+0000 mon.a (mon.0) 1245 : cluster 3 pg 6.12 is stuck undersized for 64s, current state active+undersized, last acting [7,2] 2024-03-11T21:00:00.734 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[21372]: cluster 2024-03-11T21:00:00.001169+0000 mon.a (mon.0) 1246 : cluster 3 pg 6.14 is stuck undersized for 64s, current state active+undersized, last acting [2,7] 2024-03-11T21:00:00.734 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[21372]: cluster 2024-03-11T21:00:00.001177+0000 mon.a (mon.0) 1247 : cluster 3 pg 6.15 is stuck undersized for 64s, current state active+undersized, last acting [7,6] 2024-03-11T21:00:00.734 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[21372]: cluster 2024-03-11T21:00:00.001188+0000 mon.a (mon.0) 1248 : cluster 3 pg 6.17 is stuck undersized for 64s, current state active+undersized, last acting [2,5] 2024-03-11T21:00:00.734 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[21372]: cluster 2024-03-11T21:00:00.001197+0000 mon.a (mon.0) 1249 : cluster 3 pg 6.18 is stuck undersized for 118s, current state active+undersized, last acting [7,0] 2024-03-11T21:00:00.734 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[21372]: cluster 2024-03-11T21:00:00.001206+0000 mon.a (mon.0) 1250 : cluster 3 pg 6.19 is stuck undersized for 118s, current state active+undersized, last acting [5,3] 2024-03-11T21:00:00.734 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[21372]: cluster 2024-03-11T21:00:00.001218+0000 mon.a (mon.0) 1251 : cluster 3 pg 6.1a is stuck undersized for 64s, current state undersized+peered, last acting [5] 2024-03-11T21:00:00.735 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[21372]: cluster 2024-03-11T21:00:00.001228+0000 mon.a (mon.0) 1252 : cluster 3 pg 6.1d is stuck undersized for 64s, current state undersized+peered, last acting [5] 2024-03-11T21:00:00.735 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[21372]: cluster 2024-03-11T21:00:00.001237+0000 mon.a (mon.0) 1253 : cluster 3 pg 6.1e is stuck undersized for 64s, current state active+undersized, last acting [6,5] 2024-03-11T21:00:00.735 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[21372]: cluster 2024-03-11T21:00:00.001249+0000 mon.a (mon.0) 1254 : cluster 3 pg 8.5 is active+undersized+degraded, acting [6,7] 2024-03-11T21:00:00.735 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[21372]: cluster 2024-03-11T21:00:00.001260+0000 mon.a (mon.0) 1255 : cluster 3 pg 8.e is active+undersized+degraded, acting [0,2] 2024-03-11T21:00:00.735 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[29146]: cluster 2024-03-11T21:00:00.000833+0000 mon.a (mon.0) 1215 : cluster 3 pg 3.3 is stuck undersized for 64s, current state active+undersized+degraded, last acting [6,0] 2024-03-11T21:00:00.735 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[29146]: cluster 2024-03-11T21:00:00.000842+0000 mon.a (mon.0) 1216 : cluster 3 pg 3.e is stuck undersized for 64s, current state undersized+degraded+peered, last acting [7] 2024-03-11T21:00:00.735 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[29146]: cluster 2024-03-11T21:00:00.000852+0000 mon.a (mon.0) 1217 : cluster 3 pg 3.11 is stuck undersized for 64s, current state active+undersized+degraded, last acting [7,6] 2024-03-11T21:00:00.735 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[29146]: cluster 2024-03-11T21:00:00.000864+0000 mon.a (mon.0) 1218 : cluster 3 pg 3.13 is stuck undersized for 64s, current state active+undersized+degraded, last acting [7,2] 2024-03-11T21:00:00.735 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[29146]: cluster 2024-03-11T21:00:00.000875+0000 mon.a (mon.0) 1219 : cluster 3 pg 3.14 is stuck undersized for 64s, current state active+undersized+degraded, last acting [7,6] 2024-03-11T21:00:00.735 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[29146]: cluster 2024-03-11T21:00:00.000884+0000 mon.a (mon.0) 1220 : cluster 3 pg 3.15 is stuck undersized for 64s, current state active+undersized+degraded, last acting [7,3] 2024-03-11T21:00:00.736 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[29146]: cluster 2024-03-11T21:00:00.000892+0000 mon.a (mon.0) 1221 : cluster 3 pg 3.16 is stuck undersized for 118s, current state active+undersized+degraded, last acting [5,7] 2024-03-11T21:00:00.736 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[29146]: cluster 2024-03-11T21:00:00.000902+0000 mon.a (mon.0) 1222 : cluster 3 pg 3.18 is stuck undersized for 118s, current state active+undersized+degraded, last acting [3,0] 2024-03-11T21:00:00.736 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[29146]: cluster 2024-03-11T21:00:00.000914+0000 mon.a (mon.0) 1223 : cluster 3 pg 3.19 is stuck undersized for 64s, current state undersized+degraded+peered, last acting [3] 2024-03-11T21:00:00.736 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[29146]: cluster 2024-03-11T21:00:00.000933+0000 mon.a (mon.0) 1224 : cluster 3 pg 3.1a is stuck undersized for 64s, current state undersized+degraded+peered, last acting [2] 2024-03-11T21:00:00.736 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[29146]: cluster 2024-03-11T21:00:00.000950+0000 mon.a (mon.0) 1225 : cluster 3 pg 3.1b is stuck undersized for 64s, current state active+undersized+degraded, last acting [0,7] 2024-03-11T21:00:00.736 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[29146]: cluster 2024-03-11T21:00:00.000960+0000 mon.a (mon.0) 1226 : cluster 3 pg 3.1c is stuck undersized for 64s, current state undersized+degraded+peered, last acting [5] 2024-03-11T21:00:00.736 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[29146]: cluster 2024-03-11T21:00:00.000974+0000 mon.a (mon.0) 1227 : cluster 3 pg 3.1d is stuck undersized for 64s, current state active+undersized+degraded, last acting [5,6] 2024-03-11T21:00:00.736 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[29146]: cluster 2024-03-11T21:00:00.000983+0000 mon.a (mon.0) 1228 : cluster 3 pg 4.2 is stuck undersized for 64s, current state undersized+peered, last acting [5] 2024-03-11T21:00:00.736 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[29146]: cluster 2024-03-11T21:00:00.000993+0000 mon.a (mon.0) 1229 : cluster 3 pg 4.4 is stuck undersized for 118s, current state active+undersized, last acting [2,3] 2024-03-11T21:00:00.736 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[29146]: cluster 2024-03-11T21:00:00.001003+0000 mon.a (mon.0) 1230 : cluster 3 pg 4.9 is stuck undersized for 64s, current state undersized+degraded+peered, last acting [3] 2024-03-11T21:00:00.736 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[29146]: cluster 2024-03-11T21:00:00.001013+0000 mon.a (mon.0) 1231 : cluster 3 pg 4.10 is stuck undersized for 118s, current state active+undersized, last acting [3,6] 2024-03-11T21:00:00.736 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[29146]: cluster 2024-03-11T21:00:00.001023+0000 mon.a (mon.0) 1232 : cluster 3 pg 4.12 is stuck undersized for 118s, current state active+undersized, last acting [6,2] 2024-03-11T21:00:00.736 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[29146]: cluster 2024-03-11T21:00:00.001034+0000 mon.a (mon.0) 1233 : cluster 3 pg 4.13 is stuck undersized for 64s, current state undersized+peered, last acting [6] 2024-03-11T21:00:00.736 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[29146]: cluster 2024-03-11T21:00:00.001045+0000 mon.a (mon.0) 1234 : cluster 3 pg 4.14 is stuck undersized for 118s, current state active+undersized+degraded, last acting [3,7] 2024-03-11T21:00:00.737 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[29146]: cluster 2024-03-11T21:00:00.001057+0000 mon.a (mon.0) 1235 : cluster 3 pg 4.17 is stuck undersized for 118s, current state active+undersized, last acting [3,7] 2024-03-11T21:00:00.737 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[29146]: cluster 2024-03-11T21:00:00.001067+0000 mon.a (mon.0) 1236 : cluster 3 pg 4.18 is stuck undersized for 64s, current state undersized+peered, last acting [6] 2024-03-11T21:00:00.737 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[29146]: cluster 2024-03-11T21:00:00.001077+0000 mon.a (mon.0) 1237 : cluster 3 pg 4.1a is stuck undersized for 64s, current state active+undersized, last acting [3,0] 2024-03-11T21:00:00.737 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[29146]: cluster 2024-03-11T21:00:00.001087+0000 mon.a (mon.0) 1238 : cluster 3 pg 4.1b is stuck undersized for 64s, current state undersized+peered, last acting [3] 2024-03-11T21:00:00.737 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[29146]: cluster 2024-03-11T21:00:00.001099+0000 mon.a (mon.0) 1239 : cluster 3 pg 4.1c is stuck undersized for 118s, current state active+undersized, last acting [2,3] 2024-03-11T21:00:00.737 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[29146]: cluster 2024-03-11T21:00:00.001109+0000 mon.a (mon.0) 1240 : cluster 3 pg 4.1d is stuck undersized for 64s, current state undersized+peered, last acting [5] 2024-03-11T21:00:00.737 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[29146]: cluster 2024-03-11T21:00:00.001118+0000 mon.a (mon.0) 1241 : cluster 3 pg 4.1f is stuck undersized for 118s, current state active+undersized+degraded, last acting [6,5] 2024-03-11T21:00:00.737 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[29146]: cluster 2024-03-11T21:00:00.001128+0000 mon.a (mon.0) 1242 : cluster 3 pg 6.6 is stuck undersized for 64s, current state active+undersized, last acting [3,7] 2024-03-11T21:00:00.737 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[29146]: cluster 2024-03-11T21:00:00.001139+0000 mon.a (mon.0) 1243 : cluster 3 pg 6.b is stuck undersized for 118s, current state active+undersized, last acting [3,7] 2024-03-11T21:00:00.737 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[29146]: cluster 2024-03-11T21:00:00.001149+0000 mon.a (mon.0) 1244 : cluster 3 pg 6.10 is stuck undersized for 118s, current state active+undersized, last acting [0,5] 2024-03-11T21:00:00.737 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[29146]: cluster 2024-03-11T21:00:00.001159+0000 mon.a (mon.0) 1245 : cluster 3 pg 6.12 is stuck undersized for 64s, current state active+undersized, last acting [7,2] 2024-03-11T21:00:00.737 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[29146]: cluster 2024-03-11T21:00:00.001169+0000 mon.a (mon.0) 1246 : cluster 3 pg 6.14 is stuck undersized for 64s, current state active+undersized, last acting [2,7] 2024-03-11T21:00:00.737 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[29146]: cluster 2024-03-11T21:00:00.001177+0000 mon.a (mon.0) 1247 : cluster 3 pg 6.15 is stuck undersized for 64s, current state active+undersized, last acting [7,6] 2024-03-11T21:00:00.737 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[29146]: cluster 2024-03-11T21:00:00.001188+0000 mon.a (mon.0) 1248 : cluster 3 pg 6.17 is stuck undersized for 64s, current state active+undersized, last acting [2,5] 2024-03-11T21:00:00.737 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[29146]: cluster 2024-03-11T21:00:00.001197+0000 mon.a (mon.0) 1249 : cluster 3 pg 6.18 is stuck undersized for 118s, current state active+undersized, last acting [7,0] 2024-03-11T21:00:00.738 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[29146]: cluster 2024-03-11T21:00:00.001206+0000 mon.a (mon.0) 1250 : cluster 3 pg 6.19 is stuck undersized for 118s, current state active+undersized, last acting [5,3] 2024-03-11T21:00:00.738 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[29146]: cluster 2024-03-11T21:00:00.001218+0000 mon.a (mon.0) 1251 : cluster 3 pg 6.1a is stuck undersized for 64s, current state undersized+peered, last acting [5] 2024-03-11T21:00:00.738 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[29146]: cluster 2024-03-11T21:00:00.001228+0000 mon.a (mon.0) 1252 : cluster 3 pg 6.1d is stuck undersized for 64s, current state undersized+peered, last acting [5] 2024-03-11T21:00:00.738 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[29146]: cluster 2024-03-11T21:00:00.001237+0000 mon.a (mon.0) 1253 : cluster 3 pg 6.1e is stuck undersized for 64s, current state active+undersized, last acting [6,5] 2024-03-11T21:00:00.738 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[29146]: cluster 2024-03-11T21:00:00.001249+0000 mon.a (mon.0) 1254 : cluster 3 pg 8.5 is active+undersized+degraded, acting [6,7] 2024-03-11T21:00:00.738 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:00 smithi023 bash[29146]: cluster 2024-03-11T21:00:00.001260+0000 mon.a (mon.0) 1255 : cluster 3 pg 8.e is active+undersized+degraded, acting [0,2] 2024-03-11T21:00:00.738 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:00.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:00.741 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:00:00.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:00 smithi146 bash[23809]: cluster 2024-03-11T21:00:00.000218+0000 mon.a (mon.0) 1175 : cluster 3 Health detail: HEALTH_WARN nodeep-scrub flag(s) set; 2 osds down; Reduced data availability: 23 pgs inactive; Degraded data redundancy: 304/1182 objects degraded (25.719%), 41 pgs degraded, 85 pgs undersized 2024-03-11T21:00:00.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:00 smithi146 bash[23809]: cluster 2024-03-11T21:00:00.000279+0000 mon.a (mon.0) 1176 : cluster 3 [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-03-11T21:00:00.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:00 smithi146 bash[23809]: cluster 2024-03-11T21:00:00.000299+0000 mon.a (mon.0) 1177 : cluster 3 [WRN] OSD_DOWN: 2 osds down 2024-03-11T21:00:00.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:00 smithi146 bash[23809]: cluster 2024-03-11T21:00:00.000348+0000 mon.a (mon.0) 1178 : cluster 3 osd.1 (root=default,host=smithi023) is down 2024-03-11T21:00:00.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:00 smithi146 bash[23809]: cluster 2024-03-11T21:00:00.000369+0000 mon.a (mon.0) 1179 : cluster 3 osd.4 (root=default,host=smithi146) is down 2024-03-11T21:00:00.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:00 smithi146 bash[23809]: cluster 2024-03-11T21:00:00.000393+0000 mon.a (mon.0) 1180 : cluster 3 [WRN] PG_AVAILABILITY: Reduced data availability: 23 pgs inactive 2024-03-11T21:00:00.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:00 smithi146 bash[23809]: cluster 2024-03-11T21:00:00.000441+0000 mon.a (mon.0) 1181 : cluster 3 pg 2.6 is stuck inactive for 65s, current state undersized+degraded+peered, last acting [6] 2024-03-11T21:00:00.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:00 smithi146 bash[23809]: cluster 2024-03-11T21:00:00.000466+0000 mon.a (mon.0) 1182 : cluster 3 pg 2.d is stuck inactive for 9m, current state undersized+peered, last acting [3] 2024-03-11T21:00:00.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:00 smithi146 bash[23809]: cluster 2024-03-11T21:00:00.000494+0000 mon.a (mon.0) 1183 : cluster 3 pg 2.11 is stuck inactive for 65s, current state undersized+degraded+peered, last acting [6] 2024-03-11T21:00:00.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:00 smithi146 bash[23809]: cluster 2024-03-11T21:00:00.000502+0000 mon.a (mon.0) 1184 : cluster 3 pg 3.6 is stuck inactive for 65s, current state undersized+degraded+peered, last acting [0] 2024-03-11T21:00:00.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:00 smithi146 bash[23809]: cluster 2024-03-11T21:00:00.000511+0000 mon.a (mon.0) 1185 : cluster 3 pg 3.a is stuck inactive for 65s, current state undersized+degraded+peered, last acting [6] 2024-03-11T21:00:00.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:00 smithi146 bash[23809]: cluster 2024-03-11T21:00:00.000522+0000 mon.a (mon.0) 1186 : cluster 3 pg 3.e is stuck inactive for 65s, current state undersized+degraded+peered, last acting [7] 2024-03-11T21:00:00.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:00 smithi146 bash[23809]: cluster 2024-03-11T21:00:00.000531+0000 mon.a (mon.0) 1187 : cluster 3 pg 3.19 is stuck inactive for 65s, current state undersized+degraded+peered, last acting [3] 2024-03-11T21:00:00.764 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:00 smithi146 bash[23809]: cluster 2024-03-11T21:00:00.000541+0000 mon.a (mon.0) 1188 : cluster 3 pg 3.1a is stuck inactive for 65s, current state undersized+degraded+peered, last acting [2] 2024-03-11T21:00:00.764 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:00 smithi146 bash[23809]: cluster 2024-03-11T21:00:00.000552+0000 mon.a (mon.0) 1189 : cluster 3 pg 3.1c is stuck inactive for 65s, current state undersized+degraded+peered, last acting [5] 2024-03-11T21:00:00.764 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:00 smithi146 bash[23809]: cluster 2024-03-11T21:00:00.000563+0000 mon.a (mon.0) 1190 : cluster 3 pg 4.2 is stuck inactive for 65s, current state undersized+peered, last acting [5] 2024-03-11T21:00:00.764 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:00 smithi146 bash[23809]: cluster 2024-03-11T21:00:00.000575+0000 mon.a (mon.0) 1191 : cluster 3 pg 4.9 is stuck inactive for 65s, current state undersized+degraded+peered, last acting [3] 2024-03-11T21:00:00.764 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:00 smithi146 bash[23809]: cluster 2024-03-11T21:00:00.000585+0000 mon.a (mon.0) 1192 : cluster 3 pg 4.b is stuck inactive for 65s, current state undersized+peered, last acting [0] 2024-03-11T21:00:00.764 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:00 smithi146 bash[23809]: cluster 2024-03-11T21:00:00.000593+0000 mon.a (mon.0) 1193 : cluster 3 pg 4.d is stuck inactive for 65s, current state undersized+degraded+peered, last acting [2] 2024-03-11T21:00:00.764 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:00 smithi146 bash[23809]: cluster 2024-03-11T21:00:00.000608+0000 mon.a (mon.0) 1194 : cluster 3 pg 4.e is stuck inactive for 65s, current state undersized+degraded+peered, last acting [6] 2024-03-11T21:00:00.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:00 smithi146 bash[23809]: cluster 2024-03-11T21:00:00.000617+0000 mon.a (mon.0) 1195 : cluster 3 pg 4.f is stuck inactive for 65s, current state undersized+peered, last acting [3] 2024-03-11T21:00:00.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:00 smithi146 bash[23809]: cluster 2024-03-11T21:00:00.000629+0000 mon.a (mon.0) 1196 : cluster 3 pg 4.13 is stuck inactive for 118s, current state undersized+peered, last acting [6] 2024-03-11T21:00:00.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:00 smithi146 bash[23809]: cluster 2024-03-11T21:00:00.000639+0000 mon.a (mon.0) 1197 : cluster 3 pg 4.18 is stuck inactive for 118s, current state undersized+peered, last acting [6] 2024-03-11T21:00:00.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:00 smithi146 bash[23809]: cluster 2024-03-11T21:00:00.000660+0000 mon.a (mon.0) 1198 : cluster 3 pg 4.1b is stuck inactive for 118s, current state undersized+peered, last acting [3] 2024-03-11T21:00:00.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:00 smithi146 bash[23809]: cluster 2024-03-11T21:00:00.000671+0000 mon.a (mon.0) 1199 : cluster 3 pg 4.1d is stuck inactive for 65s, current state undersized+peered, last acting [5] 2024-03-11T21:00:00.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:00 smithi146 bash[23809]: cluster 2024-03-11T21:00:00.000681+0000 mon.a (mon.0) 1200 : cluster 3 pg 5.0 is stuck inactive for 65s, current state undersized+degraded+peered, last acting [3] 2024-03-11T21:00:00.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:00 smithi146 bash[23809]: cluster 2024-03-11T21:00:00.000690+0000 mon.a (mon.0) 1201 : cluster 3 pg 6.e is stuck inactive for 118s, current state undersized+peered, last acting [2] 2024-03-11T21:00:00.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:00 smithi146 bash[23809]: cluster 2024-03-11T21:00:00.000701+0000 mon.a (mon.0) 1202 : cluster 3 pg 6.1a is stuck inactive for 118s, current state undersized+peered, last acting [5] 2024-03-11T21:00:00.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:00 smithi146 bash[23809]: cluster 2024-03-11T21:00:00.000710+0000 mon.a (mon.0) 1203 : cluster 3 pg 6.1d is stuck inactive for 65s, current state undersized+peered, last acting [5] 2024-03-11T21:00:00.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:00 smithi146 bash[23809]: cluster 2024-03-11T21:00:00.000721+0000 mon.a (mon.0) 1204 : cluster 3 [WRN] PG_DEGRADED: Degraded data redundancy: 304/1182 objects degraded (25.719%), 41 pgs degraded, 85 pgs undersized 2024-03-11T21:00:00.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:00 smithi146 bash[23809]: cluster 2024-03-11T21:00:00.000731+0000 mon.a (mon.0) 1205 : cluster 3 pg 2.2 is stuck undersized for 118s, current state active+undersized, last acting [5,6] 2024-03-11T21:00:00.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:00 smithi146 bash[23809]: cluster 2024-03-11T21:00:00.000741+0000 mon.a (mon.0) 1206 : cluster 3 pg 2.4 is stuck undersized for 118s, current state active+undersized, last acting [7,0] 2024-03-11T21:00:00.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:00 smithi146 bash[23809]: cluster 2024-03-11T21:00:00.000751+0000 mon.a (mon.0) 1207 : cluster 3 pg 2.f is stuck undersized for 64s, current state active+undersized+degraded, last acting [7,0] 2024-03-11T21:00:00.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:00 smithi146 bash[23809]: cluster 2024-03-11T21:00:00.000761+0000 mon.a (mon.0) 1208 : cluster 3 pg 2.10 is stuck undersized for 118s, current state active+undersized, last acting [2,0] 2024-03-11T21:00:00.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:00 smithi146 bash[23809]: cluster 2024-03-11T21:00:00.000775+0000 mon.a (mon.0) 1209 : cluster 3 pg 2.11 is stuck undersized for 64s, current state undersized+degraded+peered, last acting [6] 2024-03-11T21:00:00.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:00 smithi146 bash[23809]: cluster 2024-03-11T21:00:00.000786+0000 mon.a (mon.0) 1210 : cluster 3 pg 2.13 is stuck undersized for 64s, current state active+undersized, last acting [3,0] 2024-03-11T21:00:00.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:00 smithi146 bash[23809]: cluster 2024-03-11T21:00:00.000794+0000 mon.a (mon.0) 1211 : cluster 3 pg 2.15 is stuck undersized for 118s, current state active+undersized, last acting [5,0] 2024-03-11T21:00:00.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:00 smithi146 bash[23809]: cluster 2024-03-11T21:00:00.000802+0000 mon.a (mon.0) 1212 : cluster 3 pg 2.18 is stuck undersized for 64s, current state active+undersized, last acting [5,7] 2024-03-11T21:00:00.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:00 smithi146 bash[23809]: cluster 2024-03-11T21:00:00.000813+0000 mon.a (mon.0) 1213 : cluster 3 pg 2.1a is stuck undersized for 64s, current state active+undersized, last acting [6,7] 2024-03-11T21:00:00.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:00 smithi146 bash[23809]: cluster 2024-03-11T21:00:00.000823+0000 mon.a (mon.0) 1214 : cluster 3 pg 2.1f is stuck undersized for 64s, current state active+undersized, last acting [7,0] 2024-03-11T21:00:00.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:00 smithi146 bash[23809]: cluster 2024-03-11T21:00:00.000833+0000 mon.a (mon.0) 1215 : cluster 3 pg 3.3 is stuck undersized for 64s, current state active+undersized+degraded, last acting [6,0] 2024-03-11T21:00:00.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:00 smithi146 bash[23809]: cluster 2024-03-11T21:00:00.000842+0000 mon.a (mon.0) 1216 : cluster 3 pg 3.e is stuck undersized for 64s, current state undersized+degraded+peered, last acting [7] 2024-03-11T21:00:00.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:00 smithi146 bash[23809]: cluster 2024-03-11T21:00:00.000852+0000 mon.a (mon.0) 1217 : cluster 3 pg 3.11 is stuck undersized for 64s, current state active+undersized+degraded, last acting [7,6] 2024-03-11T21:00:00.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:00 smithi146 bash[23809]: cluster 2024-03-11T21:00:00.000864+0000 mon.a (mon.0) 1218 : cluster 3 pg 3.13 is stuck undersized for 64s, current state active+undersized+degraded, last acting [7,2] 2024-03-11T21:00:00.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:00 smithi146 bash[23809]: cluster 2024-03-11T21:00:00.000875+0000 mon.a (mon.0) 1219 : cluster 3 pg 3.14 is stuck undersized for 64s, current state active+undersized+degraded, last acting [7,6] 2024-03-11T21:00:00.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:00 smithi146 bash[23809]: cluster 2024-03-11T21:00:00.000884+0000 mon.a (mon.0) 1220 : cluster 3 pg 3.15 is stuck undersized for 64s, current state active+undersized+degraded, last acting [7,3] 2024-03-11T21:00:00.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:00 smithi146 bash[23809]: cluster 2024-03-11T21:00:00.000892+0000 mon.a (mon.0) 1221 : cluster 3 pg 3.16 is stuck undersized for 118s, current state active+undersized+degraded, last acting [5,7] 2024-03-11T21:00:00.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:00 smithi146 bash[23809]: cluster 2024-03-11T21:00:00.000902+0000 mon.a (mon.0) 1222 : cluster 3 pg 3.18 is stuck undersized for 118s, current state active+undersized+degraded, last acting [3,0] 2024-03-11T21:00:00.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:00 smithi146 bash[23809]: cluster 2024-03-11T21:00:00.000914+0000 mon.a (mon.0) 1223 : cluster 3 pg 3.19 is stuck undersized for 64s, current state undersized+degraded+peered, last acting [3] 2024-03-11T21:00:00.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:00 smithi146 bash[23809]: cluster 2024-03-11T21:00:00.000933+0000 mon.a (mon.0) 1224 : cluster 3 pg 3.1a is stuck undersized for 64s, current state undersized+degraded+peered, last acting [2] 2024-03-11T21:00:00.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:00 smithi146 bash[23809]: cluster 2024-03-11T21:00:00.000950+0000 mon.a (mon.0) 1225 : cluster 3 pg 3.1b is stuck undersized for 64s, current state active+undersized+degraded, last acting [0,7] 2024-03-11T21:00:00.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:00 smithi146 bash[23809]: cluster 2024-03-11T21:00:00.000960+0000 mon.a (mon.0) 1226 : cluster 3 pg 3.1c is stuck undersized for 64s, current state undersized+degraded+peered, last acting [5] 2024-03-11T21:00:00.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:00 smithi146 bash[23809]: cluster 2024-03-11T21:00:00.000974+0000 mon.a (mon.0) 1227 : cluster 3 pg 3.1d is stuck undersized for 64s, current state active+undersized+degraded, last acting [5,6] 2024-03-11T21:00:00.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:00 smithi146 bash[23809]: cluster 2024-03-11T21:00:00.000983+0000 mon.a (mon.0) 1228 : cluster 3 pg 4.2 is stuck undersized for 64s, current state undersized+peered, last acting [5] 2024-03-11T21:00:00.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:00 smithi146 bash[23809]: cluster 2024-03-11T21:00:00.000993+0000 mon.a (mon.0) 1229 : cluster 3 pg 4.4 is stuck undersized for 118s, current state active+undersized, last acting [2,3] 2024-03-11T21:00:00.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:00 smithi146 bash[23809]: cluster 2024-03-11T21:00:00.001003+0000 mon.a (mon.0) 1230 : cluster 3 pg 4.9 is stuck undersized for 64s, current state undersized+degraded+peered, last acting [3] 2024-03-11T21:00:00.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:00 smithi146 bash[23809]: cluster 2024-03-11T21:00:00.001013+0000 mon.a (mon.0) 1231 : cluster 3 pg 4.10 is stuck undersized for 118s, current state active+undersized, last acting [3,6] 2024-03-11T21:00:00.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:00 smithi146 bash[23809]: cluster 2024-03-11T21:00:00.001023+0000 mon.a (mon.0) 1232 : cluster 3 pg 4.12 is stuck undersized for 118s, current state active+undersized, last acting [6,2] 2024-03-11T21:00:00.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:00 smithi146 bash[23809]: cluster 2024-03-11T21:00:00.001034+0000 mon.a (mon.0) 1233 : cluster 3 pg 4.13 is stuck undersized for 64s, current state undersized+peered, last acting [6] 2024-03-11T21:00:00.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:00 smithi146 bash[23809]: cluster 2024-03-11T21:00:00.001045+0000 mon.a (mon.0) 1234 : cluster 3 pg 4.14 is stuck undersized for 118s, current state active+undersized+degraded, last acting [3,7] 2024-03-11T21:00:00.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:00 smithi146 bash[23809]: cluster 2024-03-11T21:00:00.001057+0000 mon.a (mon.0) 1235 : cluster 3 pg 4.17 is stuck undersized for 118s, current state active+undersized, last acting [3,7] 2024-03-11T21:00:00.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:00 smithi146 bash[23809]: cluster 2024-03-11T21:00:00.001067+0000 mon.a (mon.0) 1236 : cluster 3 pg 4.18 is stuck undersized for 64s, current state undersized+peered, last acting [6] 2024-03-11T21:00:00.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:00 smithi146 bash[23809]: cluster 2024-03-11T21:00:00.001077+0000 mon.a (mon.0) 1237 : cluster 3 pg 4.1a is stuck undersized for 64s, current state active+undersized, last acting [3,0] 2024-03-11T21:00:00.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:00 smithi146 bash[23809]: cluster 2024-03-11T21:00:00.001087+0000 mon.a (mon.0) 1238 : cluster 3 pg 4.1b is stuck undersized for 64s, current state undersized+peered, last acting [3] 2024-03-11T21:00:00.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:00 smithi146 bash[23809]: cluster 2024-03-11T21:00:00.001099+0000 mon.a (mon.0) 1239 : cluster 3 pg 4.1c is stuck undersized for 118s, current state active+undersized, last acting [2,3] 2024-03-11T21:00:00.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:00 smithi146 bash[23809]: cluster 2024-03-11T21:00:00.001109+0000 mon.a (mon.0) 1240 : cluster 3 pg 4.1d is stuck undersized for 64s, current state undersized+peered, last acting [5] 2024-03-11T21:00:00.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:00 smithi146 bash[23809]: cluster 2024-03-11T21:00:00.001118+0000 mon.a (mon.0) 1241 : cluster 3 pg 4.1f is stuck undersized for 118s, current state active+undersized+degraded, last acting [6,5] 2024-03-11T21:00:00.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:00 smithi146 bash[23809]: cluster 2024-03-11T21:00:00.001128+0000 mon.a (mon.0) 1242 : cluster 3 pg 6.6 is stuck undersized for 64s, current state active+undersized, last acting [3,7] 2024-03-11T21:00:00.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:00 smithi146 bash[23809]: cluster 2024-03-11T21:00:00.001139+0000 mon.a (mon.0) 1243 : cluster 3 pg 6.b is stuck undersized for 118s, current state active+undersized, last acting [3,7] 2024-03-11T21:00:00.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:00 smithi146 bash[23809]: cluster 2024-03-11T21:00:00.001149+0000 mon.a (mon.0) 1244 : cluster 3 pg 6.10 is stuck undersized for 118s, current state active+undersized, last acting [0,5] 2024-03-11T21:00:00.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:00 smithi146 bash[23809]: cluster 2024-03-11T21:00:00.001159+0000 mon.a (mon.0) 1245 : cluster 3 pg 6.12 is stuck undersized for 64s, current state active+undersized, last acting [7,2] 2024-03-11T21:00:00.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:00 smithi146 bash[23809]: cluster 2024-03-11T21:00:00.001169+0000 mon.a (mon.0) 1246 : cluster 3 pg 6.14 is stuck undersized for 64s, current state active+undersized, last acting [2,7] 2024-03-11T21:00:00.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:00 smithi146 bash[23809]: cluster 2024-03-11T21:00:00.001177+0000 mon.a (mon.0) 1247 : cluster 3 pg 6.15 is stuck undersized for 64s, current state active+undersized, last acting [7,6] 2024-03-11T21:00:00.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:00 smithi146 bash[23809]: cluster 2024-03-11T21:00:00.001188+0000 mon.a (mon.0) 1248 : cluster 3 pg 6.17 is stuck undersized for 64s, current state active+undersized, last acting [2,5] 2024-03-11T21:00:00.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:00 smithi146 bash[23809]: cluster 2024-03-11T21:00:00.001197+0000 mon.a (mon.0) 1249 : cluster 3 pg 6.18 is stuck undersized for 118s, current state active+undersized, last acting [7,0] 2024-03-11T21:00:00.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:00 smithi146 bash[23809]: cluster 2024-03-11T21:00:00.001206+0000 mon.a (mon.0) 1250 : cluster 3 pg 6.19 is stuck undersized for 118s, current state active+undersized, last acting [5,3] 2024-03-11T21:00:00.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:00 smithi146 bash[23809]: cluster 2024-03-11T21:00:00.001218+0000 mon.a (mon.0) 1251 : cluster 3 pg 6.1a is stuck undersized for 64s, current state undersized+peered, last acting [5] 2024-03-11T21:00:00.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:00 smithi146 bash[23809]: cluster 2024-03-11T21:00:00.001228+0000 mon.a (mon.0) 1252 : cluster 3 pg 6.1d is stuck undersized for 64s, current state undersized+peered, last acting [5] 2024-03-11T21:00:00.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:00 smithi146 bash[23809]: cluster 2024-03-11T21:00:00.001237+0000 mon.a (mon.0) 1253 : cluster 3 pg 6.1e is stuck undersized for 64s, current state active+undersized, last acting [6,5] 2024-03-11T21:00:00.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:00 smithi146 bash[23809]: cluster 2024-03-11T21:00:00.001249+0000 mon.a (mon.0) 1254 : cluster 3 pg 8.5 is active+undersized+degraded, acting [6,7] 2024-03-11T21:00:00.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:00 smithi146 bash[23809]: cluster 2024-03-11T21:00:00.001260+0000 mon.a (mon.0) 1255 : cluster 3 pg 8.e is active+undersized+degraded, acting [0,2] 2024-03-11T21:00:00.842 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:00.870 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:00.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:00.872 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:00:00.973 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:01.002 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:01.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:01.004 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:00:01.105 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:00:01.132 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:00:01.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:01.134 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:00:01.195 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:00:01.235 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:01.257 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:01.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:01.259 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:00:01.360 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:01.388 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:01.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:01.390 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:00:01.408 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 31 16 212 196 0.395112 0 - 0.00475516 2024-03-11T21:00:01.491 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:01.518 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:01.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:01.520 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:00:01.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:01 smithi023 bash[21372]: cluster 2024-03-11T21:00:00.044872+0000 mgr.y (mgr.24380) 342 : cluster 0 pgmap v350: 148 pgs: 52 active+clean, 12 undersized+peered, 43 active+undersized, 12 undersized+degraded+peered, 29 active+undersized+degraded; 13 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 304/1182 objects degraded (25.719%) 2024-03-11T21:00:01.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:01 smithi023 bash[29146]: cluster 2024-03-11T21:00:00.044872+0000 mgr.y (mgr.24380) 342 : cluster 0 pgmap v350: 148 pgs: 52 active+clean, 12 undersized+peered, 43 active+undersized, 12 undersized+degraded+peered, 29 active+undersized+degraded; 13 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 304/1182 objects degraded (25.719%) 2024-03-11T21:00:01.621 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:01.632 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:01 smithi146 bash[23809]: cluster 2024-03-11T21:00:00.044872+0000 mgr.y (mgr.24380) 342 : cluster 0 pgmap v350: 148 pgs: 52 active+clean, 12 undersized+peered, 43 active+undersized, 12 undersized+degraded+peered, 29 active+undersized+degraded; 13 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 304/1182 objects degraded (25.719%) 2024-03-11T21:00:01.648 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:01.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:01.650 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:00:01.751 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:01.779 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:01.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:01.782 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:00:01.820 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:00:01.883 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:01.910 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:01.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:01.912 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:00:02.013 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:02.041 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:02.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:02.043 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:00:02.144 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:02.170 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:02.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:02.172 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:00:02.273 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:00:02.298 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:00:02.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:02.301 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:00:02.401 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:02.408 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 32 16 212 196 0.382765 0 - 0.00475516 2024-03-11T21:00:02.428 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:02.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:02.431 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:00:02.532 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:02.559 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:02.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:02.562 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:00:02.663 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:02.674 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:02 smithi146 bash[23809]: cluster 2024-03-11T21:00:01.664965+0000 osd.6 (osd.6) 130 : cluster 0 5.2 scrub starts 2024-03-11T21:00:02.674 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:02 smithi146 bash[23809]: cluster 2024-03-11T21:00:01.677881+0000 osd.6 (osd.6) 131 : cluster 0 5.2 scrub ok 2024-03-11T21:00:02.674 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:02 smithi146 bash[23809]: cluster 2024-03-11T21:00:02.262340+0000 osd.2 (osd.2) 144 : cluster 0 2.e scrub starts 2024-03-11T21:00:02.674 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:02 smithi146 bash[23809]: cluster 2024-03-11T21:00:02.264184+0000 osd.2 (osd.2) 145 : cluster 0 2.e scrub ok 2024-03-11T21:00:02.690 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:02.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:02.693 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:00:02.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:02 smithi023 bash[21372]: cluster 2024-03-11T21:00:01.664965+0000 osd.6 (osd.6) 130 : cluster 0 5.2 scrub starts 2024-03-11T21:00:02.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:02 smithi023 bash[21372]: cluster 2024-03-11T21:00:01.677881+0000 osd.6 (osd.6) 131 : cluster 0 5.2 scrub ok 2024-03-11T21:00:02.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:02 smithi023 bash[21372]: cluster 2024-03-11T21:00:02.262340+0000 osd.2 (osd.2) 144 : cluster 0 2.e scrub starts 2024-03-11T21:00:02.786 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:02 smithi023 bash[21372]: cluster 2024-03-11T21:00:02.264184+0000 osd.2 (osd.2) 145 : cluster 0 2.e scrub ok 2024-03-11T21:00:02.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:02 smithi023 bash[29146]: cluster 2024-03-11T21:00:01.664965+0000 osd.6 (osd.6) 130 : cluster 0 5.2 scrub starts 2024-03-11T21:00:02.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:02 smithi023 bash[29146]: cluster 2024-03-11T21:00:01.677881+0000 osd.6 (osd.6) 131 : cluster 0 5.2 scrub ok 2024-03-11T21:00:02.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:02 smithi023 bash[29146]: cluster 2024-03-11T21:00:02.262340+0000 osd.2 (osd.2) 144 : cluster 0 2.e scrub starts 2024-03-11T21:00:02.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:02 smithi023 bash[29146]: cluster 2024-03-11T21:00:02.264184+0000 osd.2 (osd.2) 145 : cluster 0 2.e scrub ok 2024-03-11T21:00:02.793 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:02.823 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:02.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:02.825 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:00:02.833 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:00:02.926 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:02.952 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:02.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:02.954 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:00:03.055 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:03.082 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:03.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:03.085 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:00:03.186 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:03.215 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:03.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:03.218 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:00:03.322 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:03.349 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:03.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:03.352 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:00:03.408 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 33 16 212 196 0.371166 0 - 0.00475516 2024-03-11T21:00:03.453 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:03.478 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:03.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:03.480 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:00:03.581 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:03.610 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:03.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:03.612 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:00:03.705 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:03 smithi023 bash[29146]: cluster 2024-03-11T21:00:02.047201+0000 mgr.y (mgr.24380) 343 : cluster 0 pgmap v351: 148 pgs: 52 active+clean, 12 undersized+peered, 43 active+undersized, 12 undersized+degraded+peered, 29 active+undersized+degraded; 13 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 304/1182 objects degraded (25.719%) 2024-03-11T21:00:03.705 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:03 smithi023 bash[29146]: cluster 2024-03-11T21:00:02.198801+0000 osd.5 (osd.5) 179 : cluster 0 purged_snaps scrub starts 2024-03-11T21:00:03.705 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:03 smithi023 bash[29146]: cluster 2024-03-11T21:00:02.198899+0000 osd.5 (osd.5) 180 : cluster 0 purged_snaps scrub ok 2024-03-11T21:00:03.705 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:03 smithi023 bash[29146]: cluster 2024-03-11T21:00:02.440466+0000 mon.a (mon.0) 1256 : cluster 0 osdmap e196: 8 total, 6 up, 8 in 2024-03-11T21:00:03.705 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:03 smithi023 bash[21372]: cluster 2024-03-11T21:00:02.047201+0000 mgr.y (mgr.24380) 343 : cluster 0 pgmap v351: 148 pgs: 52 active+clean, 12 undersized+peered, 43 active+undersized, 12 undersized+degraded+peered, 29 active+undersized+degraded; 13 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 304/1182 objects degraded (25.719%) 2024-03-11T21:00:03.706 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:03 smithi023 bash[21372]: cluster 2024-03-11T21:00:02.198801+0000 osd.5 (osd.5) 179 : cluster 0 purged_snaps scrub starts 2024-03-11T21:00:03.706 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:03 smithi023 bash[21372]: cluster 2024-03-11T21:00:02.198899+0000 osd.5 (osd.5) 180 : cluster 0 purged_snaps scrub ok 2024-03-11T21:00:03.706 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:03 smithi023 bash[21372]: cluster 2024-03-11T21:00:02.440466+0000 mon.a (mon.0) 1256 : cluster 0 osdmap e196: 8 total, 6 up, 8 in 2024-03-11T21:00:03.713 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:03.724 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:03 smithi146 bash[23809]: cluster 2024-03-11T21:00:02.047201+0000 mgr.y (mgr.24380) 343 : cluster 0 pgmap v351: 148 pgs: 52 active+clean, 12 undersized+peered, 43 active+undersized, 12 undersized+degraded+peered, 29 active+undersized+degraded; 13 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 304/1182 objects degraded (25.719%) 2024-03-11T21:00:03.724 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:03 smithi146 bash[23809]: cluster 2024-03-11T21:00:02.198801+0000 osd.5 (osd.5) 179 : cluster 0 purged_snaps scrub starts 2024-03-11T21:00:03.724 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:03 smithi146 bash[23809]: cluster 2024-03-11T21:00:02.198899+0000 osd.5 (osd.5) 180 : cluster 0 purged_snaps scrub ok 2024-03-11T21:00:03.724 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:03 smithi146 bash[23809]: cluster 2024-03-11T21:00:02.440466+0000 mon.a (mon.0) 1256 : cluster 0 osdmap e196: 8 total, 6 up, 8 in 2024-03-11T21:00:03.739 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:03.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:03.741 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:00:03.841 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:03.866 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:03.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:03.869 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:00:03.890 INFO:teuthology.orchestra.run.smithi023.stdout:ERROR: (22) Invalid argument 2024-03-11T21:00:03.891 INFO:teuthology.orchestra.run.smithi023.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:00:03.970 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:03.989 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:03.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:03.991 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:00:04.092 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:04.113 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:04.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:04.115 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:00:04.216 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:04.244 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:04.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:04.246 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:00:04.349 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:04.370 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:04.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:04.373 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:00:04.409 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 34 16 212 196 0.360249 0 - 0.00475516 2024-03-11T21:00:04.473 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:04.484 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:04 smithi146 bash[23809]: cluster 2024-03-11T21:00:02.761301+0000 osd.3 (osd.3) 220 : cluster 0 4.19 scrub starts 2024-03-11T21:00:04.485 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:04 smithi146 bash[23809]: cluster 2024-03-11T21:00:02.764181+0000 osd.3 (osd.3) 221 : cluster 0 4.19 scrub ok 2024-03-11T21:00:04.485 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:04 smithi146 bash[23809]: cluster 2024-03-11T21:00:04.300254+0000 osd.2 (osd.2) 146 : cluster 0 4.16 scrub starts 2024-03-11T21:00:04.485 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:04 smithi146 bash[23809]: cluster 2024-03-11T21:00:04.302531+0000 osd.2 (osd.2) 147 : cluster 0 4.16 scrub ok 2024-03-11T21:00:04.500 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:04.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:04.503 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:00:04.604 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:04.631 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:04.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:04.634 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:00:04.735 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:04.760 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:04.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:04.762 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:00:04.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:04 smithi023 bash[21372]: cluster 2024-03-11T21:00:02.761301+0000 osd.3 (osd.3) 220 : cluster 0 4.19 scrub starts 2024-03-11T21:00:04.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:04 smithi023 bash[21372]: cluster 2024-03-11T21:00:02.764181+0000 osd.3 (osd.3) 221 : cluster 0 4.19 scrub ok 2024-03-11T21:00:04.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:04 smithi023 bash[21372]: cluster 2024-03-11T21:00:04.300254+0000 osd.2 (osd.2) 146 : cluster 0 4.16 scrub starts 2024-03-11T21:00:04.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:04 smithi023 bash[21372]: cluster 2024-03-11T21:00:04.302531+0000 osd.2 (osd.2) 147 : cluster 0 4.16 scrub ok 2024-03-11T21:00:04.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:04 smithi023 bash[29146]: cluster 2024-03-11T21:00:02.761301+0000 osd.3 (osd.3) 220 : cluster 0 4.19 scrub starts 2024-03-11T21:00:04.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:04 smithi023 bash[29146]: cluster 2024-03-11T21:00:02.764181+0000 osd.3 (osd.3) 221 : cluster 0 4.19 scrub ok 2024-03-11T21:00:04.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:04 smithi023 bash[29146]: cluster 2024-03-11T21:00:04.300254+0000 osd.2 (osd.2) 146 : cluster 0 4.16 scrub starts 2024-03-11T21:00:04.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:04 smithi023 bash[29146]: cluster 2024-03-11T21:00:04.302531+0000 osd.2 (osd.2) 147 : cluster 0 4.16 scrub ok 2024-03-11T21:00:04.863 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:04.886 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:04.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:04.888 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:00:04.989 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:00:05.014 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:00:05.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:05.017 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:00:05.117 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:05.146 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:05.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:05.149 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:00:05.249 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:05.277 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:05.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:05.279 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:00:05.380 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:05.409 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:05.409 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 35 16 212 196 0.349957 0 - 0.00475516 2024-03-11T21:00:05.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:05.411 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:00:05.457 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:00:05 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:00:05.512 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:05.513 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:05 smithi146 bash[23809]: cluster 2024-03-11T21:00:04.048130+0000 mgr.y (mgr.24380) 344 : cluster 0 pgmap v353: 148 pgs: 52 active+clean, 12 undersized+peered, 43 active+undersized, 12 undersized+degraded+peered, 29 active+undersized+degraded; 13 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 304/1182 objects degraded (25.719%) 2024-03-11T21:00:05.513 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:05 smithi146 bash[23809]: cluster 2024-03-11T21:00:04.762775+0000 osd.7 (osd.7) 187 : cluster 0 6.5 scrub starts 2024-03-11T21:00:05.513 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:05 smithi146 bash[23809]: cluster 2024-03-11T21:00:04.787214+0000 osd.7 (osd.7) 188 : cluster 0 6.5 scrub ok 2024-03-11T21:00:05.513 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:05 smithi146 bash[23809]: audit 2024-03-11T21:00:04.816666+0000 mon.a (mon.0) 1257 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:00:05.540 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:05.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:05.543 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:00:05.644 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:05.656 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:05 smithi023 bash[29146]: cluster 2024-03-11T21:00:04.048130+0000 mgr.y (mgr.24380) 344 : cluster 0 pgmap v353: 148 pgs: 52 active+clean, 12 undersized+peered, 43 active+undersized, 12 undersized+degraded+peered, 29 active+undersized+degraded; 13 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 304/1182 objects degraded (25.719%) 2024-03-11T21:00:05.656 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:05 smithi023 bash[29146]: cluster 2024-03-11T21:00:04.762775+0000 osd.7 (osd.7) 187 : cluster 0 6.5 scrub starts 2024-03-11T21:00:05.656 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:05 smithi023 bash[29146]: cluster 2024-03-11T21:00:04.787214+0000 osd.7 (osd.7) 188 : cluster 0 6.5 scrub ok 2024-03-11T21:00:05.656 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:05 smithi023 bash[29146]: audit 2024-03-11T21:00:04.816666+0000 mon.a (mon.0) 1257 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:00:05.656 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:05 smithi023 bash[21372]: cluster 2024-03-11T21:00:04.048130+0000 mgr.y (mgr.24380) 344 : cluster 0 pgmap v353: 148 pgs: 52 active+clean, 12 undersized+peered, 43 active+undersized, 12 undersized+degraded+peered, 29 active+undersized+degraded; 13 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 304/1182 objects degraded (25.719%) 2024-03-11T21:00:05.656 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:05 smithi023 bash[21372]: cluster 2024-03-11T21:00:04.762775+0000 osd.7 (osd.7) 187 : cluster 0 6.5 scrub starts 2024-03-11T21:00:05.656 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:05 smithi023 bash[21372]: cluster 2024-03-11T21:00:04.787214+0000 osd.7 (osd.7) 188 : cluster 0 6.5 scrub ok 2024-03-11T21:00:05.656 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:05 smithi023 bash[21372]: audit 2024-03-11T21:00:04.816666+0000 mon.a (mon.0) 1257 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:00:05.674 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:05.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:05.676 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:00:05.777 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:05.807 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:05.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:05.809 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:00:05.910 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:05.939 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:05.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:05.942 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:00:06.042 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:06.070 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:06.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:06.072 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:00:06.173 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:06.200 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:06.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:06.202 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:00:06.302 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:06.331 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:06.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:06.334 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:00:06.409 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 36 16 212 196 0.340236 0 - 0.00475516 2024-03-11T21:00:06.434 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:06.463 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:06.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:06.465 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:00:06.566 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:06.595 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:06.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:06.597 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:00:06.698 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:06.710 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:06 smithi023 bash[21372]: cluster 2024-03-11T21:00:04.668071+0000 osd.3 (osd.3) 222 : cluster 0 purged_snaps scrub starts 2024-03-11T21:00:06.710 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:06 smithi023 bash[21372]: cluster 2024-03-11T21:00:04.668184+0000 osd.3 (osd.3) 223 : cluster 0 purged_snaps scrub ok 2024-03-11T21:00:06.710 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:06 smithi023 bash[21372]: audit 2024-03-11T21:00:05.258303+0000 mgr.y (mgr.24380) 345 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:00:06.710 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:06 smithi023 bash[21372]: cluster 2024-03-11T21:00:05.475219+0000 mon.a (mon.0) 1258 : cluster 0 osdmap e197: 8 total, 6 up, 8 in 2024-03-11T21:00:06.710 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:06 smithi023 bash[21372]: cluster 2024-03-11T21:00:05.627524+0000 osd.2 (osd.2) 148 : cluster 0 purged_snaps scrub starts 2024-03-11T21:00:06.710 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:06 smithi023 bash[21372]: cluster 2024-03-11T21:00:05.627638+0000 osd.2 (osd.2) 149 : cluster 0 purged_snaps scrub ok 2024-03-11T21:00:06.710 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:06 smithi023 bash[21372]: cluster 2024-03-11T21:00:05.721422+0000 osd.7 (osd.7) 189 : cluster 0 6.1c scrub starts 2024-03-11T21:00:06.710 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:06 smithi023 bash[21372]: cluster 2024-03-11T21:00:05.726632+0000 osd.7 (osd.7) 190 : cluster 0 6.1c scrub ok 2024-03-11T21:00:06.710 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:06 smithi023 bash[29146]: cluster 2024-03-11T21:00:04.668071+0000 osd.3 (osd.3) 222 : cluster 0 purged_snaps scrub starts 2024-03-11T21:00:06.710 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:06 smithi023 bash[29146]: cluster 2024-03-11T21:00:04.668184+0000 osd.3 (osd.3) 223 : cluster 0 purged_snaps scrub ok 2024-03-11T21:00:06.710 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:06 smithi023 bash[29146]: audit 2024-03-11T21:00:05.258303+0000 mgr.y (mgr.24380) 345 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:00:06.711 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:06 smithi023 bash[29146]: cluster 2024-03-11T21:00:05.475219+0000 mon.a (mon.0) 1258 : cluster 0 osdmap e197: 8 total, 6 up, 8 in 2024-03-11T21:00:06.711 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:06 smithi023 bash[29146]: cluster 2024-03-11T21:00:05.627524+0000 osd.2 (osd.2) 148 : cluster 0 purged_snaps scrub starts 2024-03-11T21:00:06.711 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:06 smithi023 bash[29146]: cluster 2024-03-11T21:00:05.627638+0000 osd.2 (osd.2) 149 : cluster 0 purged_snaps scrub ok 2024-03-11T21:00:06.711 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:06 smithi023 bash[29146]: cluster 2024-03-11T21:00:05.721422+0000 osd.7 (osd.7) 189 : cluster 0 6.1c scrub starts 2024-03-11T21:00:06.711 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:06 smithi023 bash[29146]: cluster 2024-03-11T21:00:05.726632+0000 osd.7 (osd.7) 190 : cluster 0 6.1c scrub ok 2024-03-11T21:00:06.725 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:06.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:06.728 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:00:06.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:06 smithi146 bash[23809]: cluster 2024-03-11T21:00:04.668071+0000 osd.3 (osd.3) 222 : cluster 0 purged_snaps scrub starts 2024-03-11T21:00:06.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:06 smithi146 bash[23809]: cluster 2024-03-11T21:00:04.668184+0000 osd.3 (osd.3) 223 : cluster 0 purged_snaps scrub ok 2024-03-11T21:00:06.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:06 smithi146 bash[23809]: audit 2024-03-11T21:00:05.258303+0000 mgr.y (mgr.24380) 345 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:00:06.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:06 smithi146 bash[23809]: cluster 2024-03-11T21:00:05.475219+0000 mon.a (mon.0) 1258 : cluster 0 osdmap e197: 8 total, 6 up, 8 in 2024-03-11T21:00:06.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:06 smithi146 bash[23809]: cluster 2024-03-11T21:00:05.627524+0000 osd.2 (osd.2) 148 : cluster 0 purged_snaps scrub starts 2024-03-11T21:00:06.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:06 smithi146 bash[23809]: cluster 2024-03-11T21:00:05.627638+0000 osd.2 (osd.2) 149 : cluster 0 purged_snaps scrub ok 2024-03-11T21:00:06.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:06 smithi146 bash[23809]: cluster 2024-03-11T21:00:05.721422+0000 osd.7 (osd.7) 189 : cluster 0 6.1c scrub starts 2024-03-11T21:00:06.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:06 smithi146 bash[23809]: cluster 2024-03-11T21:00:05.726632+0000 osd.7 (osd.7) 190 : cluster 0 6.1c scrub ok 2024-03-11T21:00:06.828 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:06.854 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:06.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:06.856 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:00:06.957 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:06.983 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:06.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:06.986 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:00:07.087 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:00:07.113 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:00:07.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:07.115 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:00:07.216 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:00:07.244 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:00:07.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:07.246 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:00:07.347 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:07.376 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:07.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:07.382 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:00:07.383 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-11T21:00:07.409 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 37 16 212 196 0.33104 0 - 0.00475516 2024-03-11T21:00:07.483 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:07.504 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:07.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:07.506 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:00:07.607 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:07.615 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:07 smithi023 bash[21372]: cluster 2024-03-11T21:00:05.967231+0000 osd.5 (osd.5) 181 : cluster 0 3.5 scrub starts 2024-03-11T21:00:07.615 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:07 smithi023 bash[21372]: cluster 2024-03-11T21:00:05.971572+0000 osd.5 (osd.5) 182 : cluster 0 3.5 scrub ok 2024-03-11T21:00:07.615 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:07 smithi023 bash[21372]: cluster 2024-03-11T21:00:06.050415+0000 mgr.y (mgr.24380) 346 : cluster 0 pgmap v355: 148 pgs: 52 active+clean, 12 undersized+peered, 43 active+undersized, 12 undersized+degraded+peered, 29 active+undersized+degraded; 13 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 304/1182 objects degraded (25.719%) 2024-03-11T21:00:07.615 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:07 smithi023 bash[21372]: cluster 2024-03-11T21:00:06.488593+0000 mon.a (mon.0) 1259 : cluster 0 osdmap e198: 8 total, 6 up, 8 in 2024-03-11T21:00:07.616 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:07 smithi023 bash[29146]: cluster 2024-03-11T21:00:05.967231+0000 osd.5 (osd.5) 181 : cluster 0 3.5 scrub starts 2024-03-11T21:00:07.616 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:07 smithi023 bash[29146]: cluster 2024-03-11T21:00:05.971572+0000 osd.5 (osd.5) 182 : cluster 0 3.5 scrub ok 2024-03-11T21:00:07.616 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:07 smithi023 bash[29146]: cluster 2024-03-11T21:00:06.050415+0000 mgr.y (mgr.24380) 346 : cluster 0 pgmap v355: 148 pgs: 52 active+clean, 12 undersized+peered, 43 active+undersized, 12 undersized+degraded+peered, 29 active+undersized+degraded; 13 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 304/1182 objects degraded (25.719%) 2024-03-11T21:00:07.616 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:07 smithi023 bash[29146]: cluster 2024-03-11T21:00:06.488593+0000 mon.a (mon.0) 1259 : cluster 0 osdmap e198: 8 total, 6 up, 8 in 2024-03-11T21:00:07.626 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:07.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:07.628 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:00:07.729 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:07.754 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:07.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:07.757 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:00:07.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:07 smithi146 bash[23809]: cluster 2024-03-11T21:00:05.967231+0000 osd.5 (osd.5) 181 : cluster 0 3.5 scrub starts 2024-03-11T21:00:07.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:07 smithi146 bash[23809]: cluster 2024-03-11T21:00:05.971572+0000 osd.5 (osd.5) 182 : cluster 0 3.5 scrub ok 2024-03-11T21:00:07.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:07 smithi146 bash[23809]: cluster 2024-03-11T21:00:06.050415+0000 mgr.y (mgr.24380) 346 : cluster 0 pgmap v355: 148 pgs: 52 active+clean, 12 undersized+peered, 43 active+undersized, 12 undersized+degraded+peered, 29 active+undersized+degraded; 13 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 304/1182 objects degraded (25.719%) 2024-03-11T21:00:07.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:07 smithi146 bash[23809]: cluster 2024-03-11T21:00:06.488593+0000 mon.a (mon.0) 1259 : cluster 0 osdmap e198: 8 total, 6 up, 8 in 2024-03-11T21:00:07.857 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:07.884 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:07.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:07.887 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:00:07.987 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:08.015 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:08.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:08.017 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:00:08.118 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:08.142 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:08.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:08.144 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:00:08.245 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:08.271 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:08.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:08.273 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:00:08.374 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:08.400 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:08.402 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:00:08 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:00:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:00:08.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:08.403 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:00:08.409 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 38 16 212 196 0.322329 0 - 0.00475516 2024-03-11T21:00:08.503 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:08.515 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:08 smithi023 bash[21372]: cluster 2024-03-11T21:00:07.002962+0000 osd.5 (osd.5) 183 : cluster 0 3.c scrub starts 2024-03-11T21:00:08.515 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:08 smithi023 bash[21372]: cluster 2024-03-11T21:00:07.005367+0000 osd.5 (osd.5) 184 : cluster 0 3.c scrub ok 2024-03-11T21:00:08.515 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:08 smithi023 bash[21372]: cluster 2024-03-11T21:00:07.780790+0000 osd.7 (osd.7) 191 : cluster 0 6.16 scrub starts 2024-03-11T21:00:08.515 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:08 smithi023 bash[21372]: cluster 2024-03-11T21:00:07.790627+0000 osd.7 (osd.7) 192 : cluster 0 6.16 scrub ok 2024-03-11T21:00:08.515 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:08 smithi023 bash[29146]: cluster 2024-03-11T21:00:07.002962+0000 osd.5 (osd.5) 183 : cluster 0 3.c scrub starts 2024-03-11T21:00:08.515 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:08 smithi023 bash[29146]: cluster 2024-03-11T21:00:07.005367+0000 osd.5 (osd.5) 184 : cluster 0 3.c scrub ok 2024-03-11T21:00:08.516 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:08 smithi023 bash[29146]: cluster 2024-03-11T21:00:07.780790+0000 osd.7 (osd.7) 191 : cluster 0 6.16 scrub starts 2024-03-11T21:00:08.516 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:08 smithi023 bash[29146]: cluster 2024-03-11T21:00:07.790627+0000 osd.7 (osd.7) 192 : cluster 0 6.16 scrub ok 2024-03-11T21:00:08.531 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:08.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:08.534 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:00:08.635 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:08.663 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:08.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:08.665 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:00:08.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:08 smithi146 bash[23809]: cluster 2024-03-11T21:00:07.002962+0000 osd.5 (osd.5) 183 : cluster 0 3.c scrub starts 2024-03-11T21:00:08.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:08 smithi146 bash[23809]: cluster 2024-03-11T21:00:07.005367+0000 osd.5 (osd.5) 184 : cluster 0 3.c scrub ok 2024-03-11T21:00:08.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:08 smithi146 bash[23809]: cluster 2024-03-11T21:00:07.780790+0000 osd.7 (osd.7) 191 : cluster 0 6.16 scrub starts 2024-03-11T21:00:08.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:08 smithi146 bash[23809]: cluster 2024-03-11T21:00:07.790627+0000 osd.7 (osd.7) 192 : cluster 0 6.16 scrub ok 2024-03-11T21:00:08.766 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:08.790 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:08.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:08.792 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:00:08.893 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:08.921 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:08.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:08.923 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:00:09.023 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:09.051 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:09.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:09.053 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:00:09.154 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:09.182 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:09.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:09.185 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:00:09.285 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:09.306 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:09.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:09.308 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:00:09.409 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:09.410 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 39 16 212 196 0.314064 0 - 0.00475516 2024-03-11T21:00:09.432 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:09.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:09.434 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:00:09.535 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:09.544 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:09 smithi023 bash[21372]: cluster 2024-03-11T21:00:08.052348+0000 mgr.y (mgr.24380) 347 : cluster 0 pgmap v357: 148 pgs: 52 active+clean, 12 undersized+peered, 43 active+undersized, 12 undersized+degraded+peered, 29 active+undersized+degraded; 13 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 304/1182 objects degraded (25.719%) 2024-03-11T21:00:09.545 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:09 smithi023 bash[29146]: cluster 2024-03-11T21:00:08.052348+0000 mgr.y (mgr.24380) 347 : cluster 0 pgmap v357: 148 pgs: 52 active+clean, 12 undersized+peered, 43 active+undersized, 12 undersized+degraded+peered, 29 active+undersized+degraded; 13 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 304/1182 objects degraded (25.719%) 2024-03-11T21:00:09.562 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:09.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:09.565 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:00:09.665 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:09.687 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:09.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:09.689 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:00:09.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:09 smithi146 bash[23809]: cluster 2024-03-11T21:00:08.052348+0000 mgr.y (mgr.24380) 347 : cluster 0 pgmap v357: 148 pgs: 52 active+clean, 12 undersized+peered, 43 active+undersized, 12 undersized+degraded+peered, 29 active+undersized+degraded; 13 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 304/1182 objects degraded (25.719%) 2024-03-11T21:00:09.772 INFO:teuthology.orchestra.run.smithi023.stdout:osd.0: {} 2024-03-11T21:00:09.790 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:00:09.794 INFO:teuthology.orchestra.run.smithi023.stdout:osd.2: {} 2024-03-11T21:00:09.815 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:00:09.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:09.817 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:00:09.824 INFO:teuthology.orchestra.run.smithi023.stdout:osd.3: {} 2024-03-11T21:00:09.850 INFO:teuthology.orchestra.run.smithi023.stdout:osd.5: {} 2024-03-11T21:00:09.874 INFO:teuthology.orchestra.run.smithi023.stdout:osd.6: {} 2024-03-11T21:00:09.899 INFO:teuthology.orchestra.run.smithi023.stdout:osd.7: {} 2024-03-11T21:00:09.899 INFO:teuthology.orchestra.run.smithi023.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:00:09.899 INFO:teuthology.orchestra.run.smithi023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-11T21:00:09.899 INFO:teuthology.orchestra.run.smithi023.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:00:09.899 INFO:teuthology.orchestra.run.smithi023.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:00:09.899 INFO:teuthology.orchestra.run.smithi023.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-03-11T21:00:09.900 INFO:teuthology.orchestra.run.smithi023.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:00:09.900 INFO:teuthology.orchestra.run.smithi023.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:00:09.900 INFO:teuthology.orchestra.run.smithi023.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:00:09.916 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-11T21:00:09.920 INFO:teuthology.orchestra.run.smithi023.stderr:dumped all 2024-03-11T21:00:09.920 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:09.948 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:09.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:09.950 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:00:10.051 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:10.081 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:10.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:10.085 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:00:10.186 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:10.214 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:10.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:10.217 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:00:10.317 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:10.345 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:10.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:10.347 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:00:10.409 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:2024-03-11T21:00:10.413171+0000 min lat: 0.00147232 max lat: 0.0158892 avg lat: 0.00475516 2024-03-11T21:00:10.410 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-11T21:00:10.410 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 40 16 212 196 0.306213 0 - 0.00475516 2024-03-11T21:00:10.448 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:10.474 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:10.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:10.476 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:00:10.523 INFO:teuthology.orchestra.run.smithi023.stderr:nodeep-scrub is unset 2024-03-11T21:00:10.577 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:10.587 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:10 smithi146 bash[23809]: cluster 2024-03-11T21:00:08.749443+0000 osd.3 (osd.3) 224 : cluster 0 6.1b scrub starts 2024-03-11T21:00:10.587 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:10 smithi146 bash[23809]: cluster 2024-03-11T21:00:08.811746+0000 osd.3 (osd.3) 225 : cluster 0 6.1b scrub ok 2024-03-11T21:00:10.587 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:10 smithi146 bash[23809]: audit 2024-03-11T21:00:09.573626+0000 mon.c (mon.2) 45 : audit 1 from='client.? 172.21.15.23:0/2071176360' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:00:10.587 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:10 smithi146 bash[23809]: audit 2024-03-11T21:00:09.573973+0000 mon.a (mon.0) 1260 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:00:10.587 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:10 smithi146 bash[23809]: cluster 2024-03-11T21:00:09.670845+0000 osd.6 (osd.6) 132 : cluster 0 2.14 scrub starts 2024-03-11T21:00:10.587 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:10 smithi146 bash[23809]: cluster 2024-03-11T21:00:09.673082+0000 osd.6 (osd.6) 133 : cluster 0 2.14 scrub ok 2024-03-11T21:00:10.587 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:10 smithi146 bash[23809]: audit 2024-03-11T21:00:09.754078+0000 mon.c (mon.2) 46 : audit 0 from='client.? 172.21.15.23:0/3396403550' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:00:10.587 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:10 smithi146 bash[23809]: cluster 2024-03-11T21:00:10.321873+0000 osd.2 (osd.2) 150 : cluster 0 2.1 scrub starts 2024-03-11T21:00:10.587 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:10 smithi146 bash[23809]: cluster 2024-03-11T21:00:10.324336+0000 osd.2 (osd.2) 151 : cluster 0 2.1 scrub ok 2024-03-11T21:00:10.601 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:10.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:10.604 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:00:10.705 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:10.730 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:10 smithi023 bash[29146]: cluster 2024-03-11T21:00:08.749443+0000 osd.3 (osd.3) 224 : cluster 0 6.1b scrub starts 2024-03-11T21:00:10.730 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:10 smithi023 bash[29146]: cluster 2024-03-11T21:00:08.811746+0000 osd.3 (osd.3) 225 : cluster 0 6.1b scrub ok 2024-03-11T21:00:10.730 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:10 smithi023 bash[29146]: audit 2024-03-11T21:00:09.573626+0000 mon.c (mon.2) 45 : audit 1 from='client.? 172.21.15.23:0/2071176360' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:00:10.730 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:10 smithi023 bash[29146]: audit 2024-03-11T21:00:09.573973+0000 mon.a (mon.0) 1260 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:00:10.730 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:10 smithi023 bash[29146]: cluster 2024-03-11T21:00:09.670845+0000 osd.6 (osd.6) 132 : cluster 0 2.14 scrub starts 2024-03-11T21:00:10.730 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:10 smithi023 bash[29146]: cluster 2024-03-11T21:00:09.673082+0000 osd.6 (osd.6) 133 : cluster 0 2.14 scrub ok 2024-03-11T21:00:10.730 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:10 smithi023 bash[29146]: audit 2024-03-11T21:00:09.754078+0000 mon.c (mon.2) 46 : audit 0 from='client.? 172.21.15.23:0/3396403550' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:00:10.731 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:10 smithi023 bash[29146]: cluster 2024-03-11T21:00:10.321873+0000 osd.2 (osd.2) 150 : cluster 0 2.1 scrub starts 2024-03-11T21:00:10.731 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:10 smithi023 bash[29146]: cluster 2024-03-11T21:00:10.324336+0000 osd.2 (osd.2) 151 : cluster 0 2.1 scrub ok 2024-03-11T21:00:10.731 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:10 smithi023 bash[21372]: cluster 2024-03-11T21:00:08.749443+0000 osd.3 (osd.3) 224 : cluster 0 6.1b scrub starts 2024-03-11T21:00:10.731 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:10 smithi023 bash[21372]: cluster 2024-03-11T21:00:08.811746+0000 osd.3 (osd.3) 225 : cluster 0 6.1b scrub ok 2024-03-11T21:00:10.731 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:10 smithi023 bash[21372]: audit 2024-03-11T21:00:09.573626+0000 mon.c (mon.2) 45 : audit 1 from='client.? 172.21.15.23:0/2071176360' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:00:10.731 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:10 smithi023 bash[21372]: audit 2024-03-11T21:00:09.573973+0000 mon.a (mon.0) 1260 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:00:10.731 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:10 smithi023 bash[21372]: cluster 2024-03-11T21:00:09.670845+0000 osd.6 (osd.6) 132 : cluster 0 2.14 scrub starts 2024-03-11T21:00:10.731 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:10 smithi023 bash[21372]: cluster 2024-03-11T21:00:09.673082+0000 osd.6 (osd.6) 133 : cluster 0 2.14 scrub ok 2024-03-11T21:00:10.731 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:10 smithi023 bash[21372]: audit 2024-03-11T21:00:09.754078+0000 mon.c (mon.2) 46 : audit 0 from='client.? 172.21.15.23:0/3396403550' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:00:10.731 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:10 smithi023 bash[21372]: cluster 2024-03-11T21:00:10.321873+0000 osd.2 (osd.2) 150 : cluster 0 2.1 scrub starts 2024-03-11T21:00:10.731 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:10 smithi023 bash[21372]: cluster 2024-03-11T21:00:10.324336+0000 osd.2 (osd.2) 151 : cluster 0 2.1 scrub ok 2024-03-11T21:00:10.747 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:10.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:10.751 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:00:10.851 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:10.882 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:10.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:10.884 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:00:10.985 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:11.012 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:11.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:11.014 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:00:11.115 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:11.145 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:11.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:11.147 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:00:11.248 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:11.276 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:11.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:11.279 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:00:11.380 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:11.409 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:11.409 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 41 16 212 196 0.298744 0 - 0.00475516 2024-03-11T21:00:11.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:11.411 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:00:11.512 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:11.541 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:11.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:11.543 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:00:11.644 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:11.675 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:11.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:11.677 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:00:11.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:11 smithi146 bash[23809]: cluster 2024-03-11T21:00:09.747710+0000 osd.3 (osd.3) 226 : cluster 0 3.1e scrub starts 2024-03-11T21:00:11.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:11 smithi146 bash[23809]: cluster 2024-03-11T21:00:09.750451+0000 osd.3 (osd.3) 227 : cluster 0 3.1e scrub ok 2024-03-11T21:00:11.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:11 smithi146 bash[23809]: audit 2024-03-11T21:00:09.901110+0000 mgr.y (mgr.24380) 348 : audit 0 from='client.24848 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:00:11.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:11 smithi146 bash[23809]: cluster 2024-03-11T21:00:10.054264+0000 mgr.y (mgr.24380) 349 : cluster 0 pgmap v358: 148 pgs: 52 active+clean, 12 undersized+peered, 43 active+undersized, 12 undersized+degraded+peered, 29 active+undersized+degraded; 13 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 304/1182 objects degraded (25.719%) 2024-03-11T21:00:11.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:11 smithi146 bash[23809]: cluster 2024-03-11T21:00:10.501771+0000 mon.a (mon.0) 1261 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:00:11.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:11 smithi146 bash[23809]: audit 2024-03-11T21:00:10.510510+0000 mon.a (mon.0) 1262 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:00:11.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:11 smithi146 bash[23809]: cluster 2024-03-11T21:00:10.521793+0000 mon.a (mon.0) 1263 : cluster 0 osdmap e199: 8 total, 6 up, 8 in 2024-03-11T21:00:11.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:11 smithi146 bash[23809]: cluster 2024-03-11T21:00:10.624834+0000 osd.6 (osd.6) 134 : cluster 0 2.17 scrub starts 2024-03-11T21:00:11.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:11 smithi146 bash[23809]: cluster 2024-03-11T21:00:10.626546+0000 osd.6 (osd.6) 135 : cluster 0 2.17 scrub ok 2024-03-11T21:00:11.778 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:11.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:11 smithi023 bash[21372]: cluster 2024-03-11T21:00:09.747710+0000 osd.3 (osd.3) 226 : cluster 0 3.1e scrub starts 2024-03-11T21:00:11.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:11 smithi023 bash[21372]: cluster 2024-03-11T21:00:09.750451+0000 osd.3 (osd.3) 227 : cluster 0 3.1e scrub ok 2024-03-11T21:00:11.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:11 smithi023 bash[21372]: audit 2024-03-11T21:00:09.901110+0000 mgr.y (mgr.24380) 348 : audit 0 from='client.24848 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:00:11.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:11 smithi023 bash[21372]: cluster 2024-03-11T21:00:10.054264+0000 mgr.y (mgr.24380) 349 : cluster 0 pgmap v358: 148 pgs: 52 active+clean, 12 undersized+peered, 43 active+undersized, 12 undersized+degraded+peered, 29 active+undersized+degraded; 13 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 304/1182 objects degraded (25.719%) 2024-03-11T21:00:11.786 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:11 smithi023 bash[21372]: cluster 2024-03-11T21:00:10.501771+0000 mon.a (mon.0) 1261 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:00:11.786 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:11 smithi023 bash[21372]: audit 2024-03-11T21:00:10.510510+0000 mon.a (mon.0) 1262 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:00:11.786 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:11 smithi023 bash[21372]: cluster 2024-03-11T21:00:10.521793+0000 mon.a (mon.0) 1263 : cluster 0 osdmap e199: 8 total, 6 up, 8 in 2024-03-11T21:00:11.786 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:11 smithi023 bash[21372]: cluster 2024-03-11T21:00:10.624834+0000 osd.6 (osd.6) 134 : cluster 0 2.17 scrub starts 2024-03-11T21:00:11.786 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:11 smithi023 bash[21372]: cluster 2024-03-11T21:00:10.626546+0000 osd.6 (osd.6) 135 : cluster 0 2.17 scrub ok 2024-03-11T21:00:11.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:11 smithi023 bash[29146]: cluster 2024-03-11T21:00:09.747710+0000 osd.3 (osd.3) 226 : cluster 0 3.1e scrub starts 2024-03-11T21:00:11.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:11 smithi023 bash[29146]: cluster 2024-03-11T21:00:09.750451+0000 osd.3 (osd.3) 227 : cluster 0 3.1e scrub ok 2024-03-11T21:00:11.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:11 smithi023 bash[29146]: audit 2024-03-11T21:00:09.901110+0000 mgr.y (mgr.24380) 348 : audit 0 from='client.24848 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:00:11.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:11 smithi023 bash[29146]: cluster 2024-03-11T21:00:10.054264+0000 mgr.y (mgr.24380) 349 : cluster 0 pgmap v358: 148 pgs: 52 active+clean, 12 undersized+peered, 43 active+undersized, 12 undersized+degraded+peered, 29 active+undersized+degraded; 13 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 304/1182 objects degraded (25.719%) 2024-03-11T21:00:11.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:11 smithi023 bash[29146]: cluster 2024-03-11T21:00:10.501771+0000 mon.a (mon.0) 1261 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:00:11.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:11 smithi023 bash[29146]: audit 2024-03-11T21:00:10.510510+0000 mon.a (mon.0) 1262 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:00:11.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:11 smithi023 bash[29146]: cluster 2024-03-11T21:00:10.521793+0000 mon.a (mon.0) 1263 : cluster 0 osdmap e199: 8 total, 6 up, 8 in 2024-03-11T21:00:11.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:11 smithi023 bash[29146]: cluster 2024-03-11T21:00:10.624834+0000 osd.6 (osd.6) 134 : cluster 0 2.17 scrub starts 2024-03-11T21:00:11.787 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:11 smithi023 bash[29146]: cluster 2024-03-11T21:00:10.626546+0000 osd.6 (osd.6) 135 : cluster 0 2.17 scrub ok 2024-03-11T21:00:11.805 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:11.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:11.808 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:00:11.909 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:11.935 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:11.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:11.938 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:00:12.038 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:12.061 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-11T21:00:12.061 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-11T21:00:12.068 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:12.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:12.071 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:00:12.172 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:12.199 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:12.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:12.201 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:00:12.239 INFO:teuthology.orchestra.run.smithi023.stdout:{"pg_ready":true,"pg_map":{"version":357,"stamp":"2024-03-11T21:00:08.050716+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":13441448,"num_objects":394,"num_object_clones":0,"num_object_copies":1182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":394,"num_whiteouts":0,"num_read":1399,"num_read_kb":1245,"num_write":811,"num_write_kb":15005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":134,"num_bytes_recovered":1775978,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":934,"ondisk_log_size":934,"up":324,"acting":324,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":443,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":1543280,"kb_used_data":472012,"kb_used_omap":12,"kb_used_meta":1071155,"kb_avail":748417936,"statfs":{"total":767960285184,"available":766379966464,"internally_reserved":0,"allocated":483340288,"data_stored":480825853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":1096863282},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.005564"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":271,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-11T21:00:02.442940+0000","last_change":"2024-03-11T20:57:15.380339+0000","last_active":"2024-03-11T21:00:02.442940+0000","last_peered":"2024-03-11T21:00:02.442940+0000","last_clean":"2024-03-11T21:00:02.442940+0000","last_became_active":"2024-03-11T20:50:34.338390+0000","last_became_peered":"2024-03-11T20:50:34.338390+0000","last_unstale":"2024-03-11T21:00:02.442940+0000","last_undegraded":"2024-03-11T21:00:02.442940+0000","last_fullsized":"2024-03-11T21:00:02.442940+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:57:15.380219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:57:15.380219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":283,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-11T21:00:02.764353+0000","last_change":"2024-03-11T21:00:02.764408+0000","last_active":"2024-03-11T21:00:02.764353+0000","last_peered":"2024-03-11T21:00:02.764353+0000","last_clean":"2024-03-11T21:00:02.764353+0000","last_became_active":"2024-03-11T20:50:32.210194+0000","last_became_peered":"2024-03-11T20:50:32.210194+0000","last_unstale":"2024-03-11T21:00:02.764353+0000","last_undegraded":"2024-03-11T21:00:02.764353+0000","last_fullsized":"2024-03-11T21:00:02.764353+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:00:02.764190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:00:02.764190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:01:30.226802+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":88,"reported_epoch":196,"state":"active+undersized","last_fresh":"2024-03-11T21:00:02.810478+0000","last_change":"2024-03-11T20:58:53.599590+0000","last_active":"2024-03-11T21:00:02.810478+0000","last_peered":"2024-03-11T21:00:02.810478+0000","last_clean":"2024-03-11T20:58:51.927983+0000","last_became_active":"2024-03-11T20:58:53.599590+0000","last_became_peered":"2024-03-11T20:58:53.599590+0000","last_unstale":"2024-03-11T21:00:02.810478+0000","last_undegraded":"2024-03-11T21:00:02.810478+0000","last_fullsized":"2024-03-11T20:58:53.585737+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:55:56.810715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:53:29.964907+0000","last_clean_scrub_stamp":"2024-03-11T20:55:56.810715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1e","version":"84'10","reported_seq":295,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-11T21:00:02.447603+0000","last_change":"2024-03-11T20:58:34.504697+0000","last_active":"2024-03-11T21:00:02.447603+0000","last_peered":"2024-03-11T21:00:02.447603+0000","last_clean":"2024-03-11T21:00:02.447603+0000","last_became_active":"2024-03-11T20:50:30.101721+0000","last_became_peered":"2024-03-11T20:50:30.101721+0000","last_unstale":"2024-03-11T21:00:02.447603+0000","last_undegraded":"2024-03-11T21:00:02.447603+0000","last_fullsized":"2024-03-11T21:00:02.447603+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'10","last_scrub_stamp":"2024-03-11T20:58:34.504575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:58:34.504575+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":270,"reported_epoch":196,"state":"undersized+peered","last_fresh":"2024-03-11T21:00:02.438715+0000","last_change":"2024-03-11T20:58:53.582468+0000","last_active":"2024-03-11T20:57:59.063074+0000","last_peered":"2024-03-11T21:00:02.438715+0000","last_clean":"2024-03-11T20:57:55.801316+0000","last_became_active":"2024-03-11T20:57:24.633041+0000","last_became_peered":"2024-03-11T20:58:53.582468+0000","last_unstale":"2024-03-11T21:00:02.438715+0000","last_undegraded":"2024-03-11T21:00:02.438715+0000","last_fullsized":"2024-03-11T20:58:53.580279+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:57:30.121565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:55.545758+0000","last_clean_scrub_stamp":"2024-03-11T20:57:30.121565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":274,"reported_epoch":196,"state":"undersized+peered","last_fresh":"2024-03-11T21:00:02.710527+0000","last_change":"2024-03-11T20:58:53.587921+0000","last_active":"2024-03-11T20:57:59.062731+0000","last_peered":"2024-03-11T21:00:02.710527+0000","last_clean":"2024-03-11T20:57:55.801107+0000","last_became_active":"2024-03-11T20:57:24.633333+0000","last_became_peered":"2024-03-11T20:58:53.587921+0000","last_unstale":"2024-03-11T21:00:02.710527+0000","last_undegraded":"2024-03-11T21:00:02.710527+0000","last_fullsized":"2024-03-11T20:58:53.586178+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:57:06.017716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:54:13.660325+0000","last_clean_scrub_stamp":"2024-03-11T20:57:06.017716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:00:06.663064+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":284,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-11T21:00:02.447022+0000","last_change":"2024-03-11T20:59:03.486143+0000","last_active":"2024-03-11T21:00:02.447022+0000","last_peered":"2024-03-11T21:00:02.447022+0000","last_clean":"2024-03-11T21:00:02.447022+0000","last_became_active":"2024-03-11T20:50:28.058274+0000","last_became_peered":"2024-03-11T20:50:28.058274+0000","last_unstale":"2024-03-11T21:00:02.447022+0000","last_undegraded":"2024-03-11T21:00:02.447022+0000","last_fullsized":"2024-03-11T21:00:02.447022+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:59:03.485966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:59:03.485966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:00:16.712769+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"84'11","reported_seq":315,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-11T21:00:02.756755+0000","last_change":"2024-03-11T20:59:57.186992+0000","last_active":"2024-03-11T21:00:02.756755+0000","last_peered":"2024-03-11T21:00:02.756755+0000","last_clean":"2024-03-11T21:00:02.756755+0000","last_became_active":"2024-03-11T20:50:30.092638+0000","last_became_peered":"2024-03-11T20:50:30.092638+0000","last_unstale":"2024-03-11T21:00:02.756755+0000","last_undegraded":"2024-03-11T21:00:02.756755+0000","last_fullsized":"2024-03-11T21:00:02.756755+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T20:59:57.186758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:59:57.186758+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:00:59.444862+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":279,"reported_epoch":196,"state":"active+undersized","last_fresh":"2024-03-11T21:00:02.438570+0000","last_change":"2024-03-11T20:57:59.726857+0000","last_active":"2024-03-11T21:00:02.438570+0000","last_peered":"2024-03-11T21:00:02.438570+0000","last_clean":"2024-03-11T20:57:55.794158+0000","last_became_active":"2024-03-11T20:57:59.726857+0000","last_became_peered":"2024-03-11T20:57:59.726857+0000","last_unstale":"2024-03-11T21:00:02.438570+0000","last_undegraded":"2024-03-11T21:00:02.438570+0000","last_fullsized":"2024-03-11T20:57:59.714207+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:55:50.617945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:53:07.792697+0000","last_clean_scrub_stamp":"2024-03-11T20:55:50.617945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":266,"reported_epoch":196,"state":"undersized+peered","last_fresh":"2024-03-11T21:00:02.443743+0000","last_change":"2024-03-11T20:58:53.598159+0000","last_active":"2024-03-11T20:57:59.067431+0000","last_peered":"2024-03-11T21:00:02.443743+0000","last_clean":"2024-03-11T20:57:55.801676+0000","last_became_active":"2024-03-11T20:57:24.635672+0000","last_became_peered":"2024-03-11T20:58:53.598159+0000","last_unstale":"2024-03-11T21:00:02.443743+0000","last_undegraded":"2024-03-11T21:00:02.443743+0000","last_fullsized":"2024-03-11T20:58:53.596503+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:55:49.273387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:55:49.273387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":292,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-11T21:00:02.810025+0000","last_change":"2024-03-11T20:59:01.396599+0000","last_active":"2024-03-11T21:00:02.810025+0000","last_peered":"2024-03-11T21:00:02.810025+0000","last_clean":"2024-03-11T21:00:02.810025+0000","last_became_active":"2024-03-11T20:50:28.079225+0000","last_became_peered":"2024-03-11T20:50:28.079225+0000","last_unstale":"2024-03-11T21:00:02.810025+0000","last_undegraded":"2024-03-11T21:00:02.810025+0000","last_fullsized":"2024-03-11T21:00:02.810025+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:59:01.396377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:59:01.396377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:00:29.634132+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"84'15","reported_seq":346,"reported_epoch":196,"state":"undersized+degraded+peered","last_fresh":"2024-03-11T21:00:02.439584+0000","last_change":"2024-03-11T20:58:53.585952+0000","last_active":"2024-03-11T20:58:52.580762+0000","last_peered":"2024-03-11T21:00:02.439584+0000","last_clean":"2024-03-11T20:57:55.794659+0000","last_became_active":"2024-03-11T20:57:59.738907+0000","last_became_peered":"2024-03-11T20:58:53.585952+0000","last_unstale":"2024-03-11T21:00:02.439584+0000","last_undegraded":"2024-03-11T20:58:53.582798+0000","last_fullsized":"2024-03-11T20:58:53.581428+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15","last_scrub_stamp":"2024-03-11T20:55:46.567413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:55:46.567413+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[{"shards":"5","objects":7}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":188,"reported_epoch":196,"state":"active+undersized","last_fresh":"2024-03-11T21:00:02.807403+0000","last_change":"2024-03-11T20:57:59.735731+0000","last_active":"2024-03-11T21:00:02.807403+0000","last_peered":"2024-03-11T21:00:02.807403+0000","last_clean":"2024-03-11T20:54:35.933718+0000","last_became_active":"2024-03-11T20:57:59.735731+0000","last_became_peered":"2024-03-11T20:57:59.735731+0000","last_unstale":"2024-03-11T21:00:02.807403+0000","last_undegraded":"2024-03-11T21:00:02.807403+0000","last_fullsized":"2024-03-11T20:57:59.719866+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:57:02.099146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:57:02.099146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":34,"reported_epoch":196,"state":"active+undersized","last_fresh":"2024-03-11T21:00:02.442146+0000","last_change":"2024-03-11T20:58:53.594996+0000","last_active":"2024-03-11T21:00:02.442146+0000","last_peered":"2024-03-11T21:00:02.442146+0000","last_clean":"2024-03-11T20:50:31.136392+0000","last_became_active":"2024-03-11T20:58:53.594996+0000","last_became_peered":"2024-03-11T20:58:53.594996+0000","last_unstale":"2024-03-11T21:00:02.442146+0000","last_undegraded":"2024-03-11T21:00:02.442146+0000","last_fullsized":"2024-03-11T20:58:53.585744+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:58:43.133748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:53:50.599029+0000","last_clean_scrub_stamp":"2024-03-11T20:58:43.133748+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:00:12.752932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":292,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-11T21:00:02.809075+0000","last_change":"2024-03-11T20:58:57.412315+0000","last_active":"2024-03-11T21:00:02.809075+0000","last_peered":"2024-03-11T21:00:02.809075+0000","last_clean":"2024-03-11T21:00:02.809075+0000","last_became_active":"2024-03-11T20:50:28.068978+0000","last_became_peered":"2024-03-11T20:50:28.068978+0000","last_unstale":"2024-03-11T21:00:02.809075+0000","last_undegraded":"2024-03-11T21:00:02.809075+0000","last_fullsized":"2024-03-11T21:00:02.809075+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:58:57.412109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:54:07.155130+0000","last_clean_scrub_stamp":"2024-03-11T20:58:57.412109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:00:09.172688+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"110'17","reported_seq":311,"reported_epoch":196,"state":"active+undersized+degraded","last_fresh":"2024-03-11T21:00:02.440072+0000","last_change":"2024-03-11T20:58:53.593232+0000","last_active":"2024-03-11T21:00:02.440072+0000","last_peered":"2024-03-11T21:00:02.440072+0000","last_clean":"2024-03-11T20:58:51.945230+0000","last_became_active":"2024-03-11T20:58:53.593232+0000","last_became_peered":"2024-03-11T20:58:53.593232+0000","last_unstale":"2024-03-11T21:00:02.440072+0000","last_undegraded":"2024-03-11T20:58:53.581860+0000","last_fullsized":"2024-03-11T20:58:53.581721+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'17","last_scrub_stamp":"2024-03-11T20:57:23.603960+0000","last_deep_scrub":"110'17","last_deep_scrub_stamp":"2024-03-11T20:53:04.802488+0000","last_clean_scrub_stamp":"2024-03-11T20:57:23.603960+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[{"shards":"5,6","objects":5}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":272,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-11T21:00:02.448372+0000","last_change":"2024-03-11T20:58:45.526833+0000","last_active":"2024-03-11T21:00:02.448372+0000","last_peered":"2024-03-11T21:00:02.448372+0000","last_clean":"2024-03-11T21:00:02.448372+0000","last_became_active":"2024-03-11T20:50:34.334558+0000","last_became_peered":"2024-03-11T20:50:34.334558+0000","last_unstale":"2024-03-11T21:00:02.448372+0000","last_undegraded":"2024-03-11T21:00:02.448372+0000","last_fullsized":"2024-03-11T21:00:02.448372+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:58:45.526714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:55:59.892394+0000","last_clean_scrub_stamp":"2024-03-11T20:58:45.526714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:00:06.045582+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":107,"reported_epoch":196,"state":"undersized+peered","last_fresh":"2024-03-11T21:00:02.439170+0000","last_change":"2024-03-11T20:58:53.586752+0000","last_active":"2024-03-11T20:58:52.579894+0000","last_peered":"2024-03-11T21:00:02.439170+0000","last_clean":"2024-03-11T20:57:55.794215+0000","last_became_active":"2024-03-11T20:57:59.739375+0000","last_became_peered":"2024-03-11T20:58:53.586752+0000","last_unstale":"2024-03-11T21:00:02.439170+0000","last_undegraded":"2024-03-11T21:00:02.439170+0000","last_fullsized":"2024-03-11T20:58:53.581438+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:55:24.714851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:55:24.714851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"77'1","reported_seq":291,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-11T21:00:02.444304+0000","last_change":"2024-03-11T20:58:46.535191+0000","last_active":"2024-03-11T21:00:02.444304+0000","last_peered":"2024-03-11T21:00:02.444304+0000","last_clean":"2024-03-11T21:00:02.444304+0000","last_became_active":"2024-03-11T20:50:28.068840+0000","last_became_peered":"2024-03-11T20:50:28.068840+0000","last_unstale":"2024-03-11T21:00:02.444304+0000","last_undegraded":"2024-03-11T21:00:02.444304+0000","last_fullsized":"2024-03-11T21:00:02.444304+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-03-11T20:58:46.534991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:58:46.534991+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"166'13","reported_seq":293,"reported_epoch":196,"state":"undersized+degraded+peered","last_fresh":"2024-03-11T21:00:02.760025+0000","last_change":"2024-03-11T20:58:53.582284+0000","last_active":"2024-03-11T20:58:52.587401+0000","last_peered":"2024-03-11T21:00:02.760025+0000","last_clean":"2024-03-11T20:57:32.147284+0000","last_became_active":"2024-03-11T20:57:24.633197+0000","last_became_peered":"2024-03-11T20:58:53.582284+0000","last_unstale":"2024-03-11T21:00:02.760025+0000","last_undegraded":"2024-03-11T20:58:53.578654+0000","last_fullsized":"2024-03-11T20:58:53.578559+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'13","last_scrub_stamp":"2024-03-11T20:57:32.156261+0000","last_deep_scrub":"166'13","last_deep_scrub_stamp":"2024-03-11T20:57:32.156261+0000","last_clean_scrub_stamp":"2024-03-11T20:57:32.156261+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:00:07.260326+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":32,"reported_epoch":196,"state":"active+undersized","last_fresh":"2024-03-11T21:00:02.711562+0000","last_change":"2024-03-11T20:58:53.592130+0000","last_active":"2024-03-11T21:00:02.711562+0000","last_peered":"2024-03-11T21:00:02.711562+0000","last_clean":"2024-03-11T20:50:33.234063+0000","last_became_active":"2024-03-11T20:58:53.592130+0000","last_became_peered":"2024-03-11T20:58:53.592130+0000","last_unstale":"2024-03-11T21:00:02.711562+0000","last_undegraded":"2024-03-11T21:00:02.711562+0000","last_fullsized":"2024-03-11T20:58:53.584231+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:58:33.135750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:58:33.135750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":309,"reported_epoch":196,"state":"active+undersized","last_fresh":"2024-03-11T21:00:02.761490+0000","last_change":"2024-03-11T20:57:59.717388+0000","last_active":"2024-03-11T21:00:02.761490+0000","last_peered":"2024-03-11T21:00:02.761490+0000","last_clean":"2024-03-11T20:57:55.796750+0000","last_became_active":"2024-03-11T20:57:59.717388+0000","last_became_peered":"2024-03-11T20:57:59.717388+0000","last_unstale":"2024-03-11T21:00:02.761490+0000","last_undegraded":"2024-03-11T21:00:02.761490+0000","last_fullsized":"2024-03-11T20:57:59.710320+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:57:16.454268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:57:16.454268+0000","last_clean_scrub_stamp":"2024-03-11T20:57:16.454268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:00:07.260257+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":286,"reported_epoch":196,"state":"active+undersized","last_fresh":"2024-03-11T21:00:02.711045+0000","last_change":"2024-03-11T20:58:53.589870+0000","last_active":"2024-03-11T21:00:02.711045+0000","last_peered":"2024-03-11T21:00:02.711045+0000","last_clean":"2024-03-11T20:58:51.925374+0000","last_became_active":"2024-03-11T20:58:53.589870+0000","last_became_peered":"2024-03-11T20:58:53.589870+0000","last_unstale":"2024-03-11T21:00:02.711045+0000","last_undegraded":"2024-03-11T21:00:02.711045+0000","last_fullsized":"2024-03-11T20:58:53.584802+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:58:31.909945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:58:31.909945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6","7"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"84'5","reported_seq":300,"reported_epoch":196,"state":"active+undersized+degraded","last_fresh":"2024-03-11T21:00:02.755379+0000","last_change":"2024-03-11T20:58:53.593916+0000","last_active":"2024-03-11T21:00:02.755379+0000","last_peered":"2024-03-11T21:00:02.755379+0000","last_clean":"2024-03-11T20:58:51.872040+0000","last_became_active":"2024-03-11T20:58:53.593916+0000","last_became_peered":"2024-03-11T20:58:53.593916+0000","last_unstale":"2024-03-11T21:00:02.755379+0000","last_undegraded":"2024-03-11T20:58:53.588720+0000","last_fullsized":"2024-03-11T20:58:53.588479+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'5","last_scrub_stamp":"2024-03-11T20:58:31.943100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:58:31.943100+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[{"shards":"0,7","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":154,"reported_epoch":196,"state":"undersized+peered","last_fresh":"2024-03-11T21:00:02.439460+0000","last_change":"2024-03-11T20:58:53.586646+0000","last_active":"2024-03-11T20:58:52.582468+0000","last_peered":"2024-03-11T21:00:02.439460+0000","last_clean":"2024-03-11T20:50:33.234063+0000","last_became_active":"2024-03-11T20:57:59.737607+0000","last_became_peered":"2024-03-11T20:58:53.586646+0000","last_unstale":"2024-03-11T21:00:02.439460+0000","last_undegraded":"2024-03-11T21:00:02.439460+0000","last_fullsized":"2024-03-11T20:58:53.580643+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:56:54.683291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:56:54.683291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"175'15","reported_seq":427,"reported_epoch":196,"state":"active+undersized+degraded","last_fresh":"2024-03-11T21:00:02.711928+0000","last_change":"2024-03-11T20:57:59.758963+0000","last_active":"2024-03-11T21:00:02.711928+0000","last_peered":"2024-03-11T21:00:02.711928+0000","last_clean":"2024-03-11T20:57:58.496376+0000","last_became_active":"2024-03-11T20:57:59.758963+0000","last_became_peered":"2024-03-11T20:57:59.758963+0000","last_unstale":"2024-03-11T21:00:02.711928+0000","last_undegraded":"2024-03-11T20:57:59.729812+0000","last_fullsized":"2024-03-11T20:57:59.729635+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"125'13","last_scrub_stamp":"2024-03-11T20:56:55.894486+0000","last_deep_scrub":"114'12","last_deep_scrub_stamp":"2024-03-11T20:53:30.905791+0000","last_clean_scrub_stamp":"2024-03-11T20:56:55.894486+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:00:06.663078+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[{"shards":"5,6","objects":1}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"77'1","reported_seq":284,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-11T21:00:02.442783+0000","last_change":"2024-03-11T20:59:06.457866+0000","last_active":"2024-03-11T21:00:02.442783+0000","last_peered":"2024-03-11T21:00:02.442783+0000","last_clean":"2024-03-11T21:00:02.442783+0000","last_became_active":"2024-03-11T20:50:28.068827+0000","last_became_peered":"2024-03-11T20:50:28.068827+0000","last_unstale":"2024-03-11T21:00:02.442783+0000","last_undegraded":"2024-03-11T21:00:02.442783+0000","last_fullsized":"2024-03-11T21:00:02.442783+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-03-11T20:59:06.457561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:59:06.457561+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:00:34.056405+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"84'9","reported_seq":322,"reported_epoch":196,"state":"active+undersized+degraded","last_fresh":"2024-03-11T21:00:02.442414+0000","last_change":"2024-03-11T20:57:59.718773+0000","last_active":"2024-03-11T21:00:02.442414+0000","last_peered":"2024-03-11T21:00:02.442414+0000","last_clean":"2024-03-11T20:57:55.787262+0000","last_became_active":"2024-03-11T20:57:59.718773+0000","last_became_peered":"2024-03-11T20:57:59.718773+0000","last_unstale":"2024-03-11T21:00:02.442414+0000","last_undegraded":"2024-03-11T20:57:59.713607+0000","last_fullsized":"2024-03-11T20:57:59.713489+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'9","last_scrub_stamp":"2024-03-11T20:55:25.698923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:55:25.698923+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":4}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"105'2","reported_seq":275,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-11T21:00:02.809896+0000","last_change":"2024-03-11T20:58:38.137230+0000","last_active":"2024-03-11T21:00:02.809896+0000","last_peered":"2024-03-11T21:00:02.809896+0000","last_clean":"2024-03-11T21:00:02.809896+0000","last_became_active":"2024-03-11T20:50:34.327356+0000","last_became_peered":"2024-03-11T20:50:34.327356+0000","last_unstale":"2024-03-11T21:00:02.809896+0000","last_undegraded":"2024-03-11T21:00:02.809896+0000","last_fullsized":"2024-03-11T21:00:02.809896+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'2","last_scrub_stamp":"2024-03-11T20:58:38.137122+0000","last_deep_scrub":"105'2","last_deep_scrub_stamp":"2024-03-11T20:52:18.098911+0000","last_clean_scrub_stamp":"2024-03-11T20:58:38.137122+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":101,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-11T21:00:02.810356+0000","last_change":"2024-03-11T20:59:14.605895+0000","last_active":"2024-03-11T21:00:02.810356+0000","last_peered":"2024-03-11T21:00:02.810356+0000","last_clean":"2024-03-11T21:00:02.810356+0000","last_became_active":"2024-03-11T20:56:59.804264+0000","last_became_peered":"2024-03-11T20:56:59.804264+0000","last_unstale":"2024-03-11T21:00:02.810356+0000","last_undegraded":"2024-03-11T21:00:02.810356+0000","last_fullsized":"2024-03-11T21:00:02.810356+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:59:14.605681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:59:14.605681+0000","last_clean_scrub_stamp":"2024-03-11T20:59:14.605681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:00:32.782703+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":287,"reported_epoch":196,"state":"active+undersized","last_fresh":"2024-03-11T21:00:02.438567+0000","last_change":"2024-03-11T20:58:53.591066+0000","last_active":"2024-03-11T21:00:02.438567+0000","last_peered":"2024-03-11T21:00:02.438567+0000","last_clean":"2024-03-11T20:58:51.930721+0000","last_became_active":"2024-03-11T20:58:53.591066+0000","last_became_peered":"2024-03-11T20:58:53.591066+0000","last_unstale":"2024-03-11T21:00:02.438567+0000","last_undegraded":"2024-03-11T21:00:02.438567+0000","last_fullsized":"2024-03-11T20:58:53.580542+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:58:43.047023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:52:01.880052+0000","last_clean_scrub_stamp":"2024-03-11T20:58:43.047023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:00:04.439062+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"84'15","reported_seq":196,"reported_epoch":196,"state":"undersized+degraded+peered","last_fresh":"2024-03-11T21:00:02.443650+0000","last_change":"2024-03-11T20:58:53.598699+0000","last_active":"2024-03-11T20:58:52.578768+0000","last_peered":"2024-03-11T21:00:02.443650+0000","last_clean":"2024-03-11T20:57:12.894415+0000","last_became_active":"2024-03-11T20:57:59.734932+0000","last_became_peered":"2024-03-11T20:58:53.598699+0000","last_unstale":"2024-03-11T21:00:02.443650+0000","last_undegraded":"2024-03-11T20:58:53.591931+0000","last_fullsized":"2024-03-11T20:58:53.591802+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15","last_scrub_stamp":"2024-03-11T20:57:24.955535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:57:24.955535+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":7}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":272,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-11T21:00:02.445772+0000","last_change":"2024-03-11T20:58:44.497146+0000","last_active":"2024-03-11T21:00:02.445772+0000","last_peered":"2024-03-11T21:00:02.445772+0000","last_clean":"2024-03-11T21:00:02.445772+0000","last_became_active":"2024-03-11T20:50:34.337878+0000","last_became_peered":"2024-03-11T20:50:34.337878+0000","last_unstale":"2024-03-11T21:00:02.445772+0000","last_undegraded":"2024-03-11T21:00:02.445772+0000","last_fullsized":"2024-03-11T21:00:02.445772+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:58:44.496960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:52:11.462480+0000","last_clean_scrub_stamp":"2024-03-11T20:58:44.496960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":284,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-11T21:00:02.444322+0000","last_change":"2024-03-11T20:59:13.516522+0000","last_active":"2024-03-11T21:00:02.444322+0000","last_peered":"2024-03-11T21:00:02.444322+0000","last_clean":"2024-03-11T21:00:02.444322+0000","last_became_active":"2024-03-11T20:50:32.646323+0000","last_became_peered":"2024-03-11T20:50:32.646323+0000","last_unstale":"2024-03-11T21:00:02.444322+0000","last_undegraded":"2024-03-11T21:00:02.444322+0000","last_fullsized":"2024-03-11T21:00:02.444322+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:59:13.516326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:54:06.516412+0000","last_clean_scrub_stamp":"2024-03-11T20:59:13.516326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:00:43.418463+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":291,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-11T21:00:02.711918+0000","last_change":"2024-03-11T20:58:41.902142+0000","last_active":"2024-03-11T21:00:02.711918+0000","last_peered":"2024-03-11T21:00:02.711918+0000","last_clean":"2024-03-11T21:00:02.711918+0000","last_became_active":"2024-03-11T20:50:28.086513+0000","last_became_peered":"2024-03-11T20:50:28.086513+0000","last_unstale":"2024-03-11T21:00:02.711918+0000","last_undegraded":"2024-03-11T21:00:02.711918+0000","last_fullsized":"2024-03-11T21:00:02.711918+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:58:41.901996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:58:41.901996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:00:09.529254+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"145'13","reported_seq":327,"reported_epoch":196,"state":"active+undersized+degraded","last_fresh":"2024-03-11T21:00:02.439797+0000","last_change":"2024-03-11T20:57:59.740777+0000","last_active":"2024-03-11T21:00:02.439797+0000","last_peered":"2024-03-11T21:00:02.439797+0000","last_clean":"2024-03-11T20:57:55.794621+0000","last_became_active":"2024-03-11T20:57:59.740777+0000","last_became_peered":"2024-03-11T20:57:59.740777+0000","last_unstale":"2024-03-11T21:00:02.439797+0000","last_undegraded":"2024-03-11T20:57:59.720497+0000","last_fullsized":"2024-03-11T20:57:59.720181+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'13","last_scrub_stamp":"2024-03-11T20:55:53.560287+0000","last_deep_scrub":"145'13","last_deep_scrub_stamp":"2024-03-11T20:55:53.560287+0000","last_clean_scrub_stamp":"2024-03-11T20:55:53.560287+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[{"shards":"5,7","objects":4}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":276,"reported_epoch":196,"state":"active+undersized","last_fresh":"2024-03-11T21:00:02.808727+0000","last_change":"2024-03-11T20:58:53.594002+0000","last_active":"2024-03-11T21:00:02.808727+0000","last_peered":"2024-03-11T21:00:02.808727+0000","last_clean":"2024-03-11T20:58:51.928599+0000","last_became_active":"2024-03-11T20:58:53.594002+0000","last_became_peered":"2024-03-11T20:58:53.594002+0000","last_unstale":"2024-03-11T21:00:02.808727+0000","last_undegraded":"2024-03-11T21:00:02.808727+0000","last_fullsized":"2024-03-11T20:58:53.582688+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:56:58.131257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:56:58.131257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":283,"reported_epoch":196,"state":"active+undersized","last_fresh":"2024-03-11T21:00:02.442443+0000","last_change":"2024-03-11T20:57:59.741955+0000","last_active":"2024-03-11T21:00:02.442443+0000","last_peered":"2024-03-11T21:00:02.442443+0000","last_clean":"2024-03-11T20:57:55.787350+0000","last_became_active":"2024-03-11T20:57:59.741955+0000","last_became_peered":"2024-03-11T20:57:59.741955+0000","last_unstale":"2024-03-11T21:00:02.442443+0000","last_undegraded":"2024-03-11T21:00:02.442443+0000","last_fullsized":"2024-03-11T20:57:59.713703+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:55:32.587714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:55:32.587714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":292,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-11T21:00:02.439666+0000","last_change":"2024-03-11T20:58:55.977401+0000","last_active":"2024-03-11T21:00:02.439666+0000","last_peered":"2024-03-11T21:00:02.439666+0000","last_clean":"2024-03-11T21:00:02.439666+0000","last_became_active":"2024-03-11T20:50:28.075760+0000","last_became_peered":"2024-03-11T20:50:28.075760+0000","last_unstale":"2024-03-11T21:00:02.439666+0000","last_undegraded":"2024-03-11T21:00:02.439666+0000","last_fullsized":"2024-03-11T21:00:02.439666+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:58:55.977289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:58:55.977289+0000","last_clean_scrub_stamp":"2024-03-11T20:58:55.977289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:00:18.994146+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"84'6","reported_seq":129,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-11T21:00:02.439968+0000","last_change":"2024-03-11T20:59:06.978298+0000","last_active":"2024-03-11T21:00:02.439968+0000","last_peered":"2024-03-11T21:00:02.439968+0000","last_clean":"2024-03-11T21:00:02.439968+0000","last_became_active":"2024-03-11T20:56:59.794581+0000","last_became_peered":"2024-03-11T20:56:59.794581+0000","last_unstale":"2024-03-11T21:00:02.439968+0000","last_undegraded":"2024-03-11T21:00:02.439968+0000","last_fullsized":"2024-03-11T21:00:02.439968+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6","last_scrub_stamp":"2024-03-11T20:59:06.978049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:59:06.978049+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:00:07.408178+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":8,"num_read_kb":5,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"186'2","reported_seq":274,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-11T21:00:02.441867+0000","last_change":"2024-03-11T20:59:04.535089+0000","last_active":"2024-03-11T21:00:02.441867+0000","last_peered":"2024-03-11T21:00:02.441867+0000","last_clean":"2024-03-11T21:00:02.441867+0000","last_became_active":"2024-03-11T20:50:34.339326+0000","last_became_peered":"2024-03-11T20:50:34.339326+0000","last_unstale":"2024-03-11T21:00:02.441867+0000","last_undegraded":"2024-03-11T21:00:02.441867+0000","last_fullsized":"2024-03-11T21:00:02.441867+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"186'2","last_scrub_stamp":"2024-03-11T20:59:04.534850+0000","last_deep_scrub":"186'2","last_deep_scrub_stamp":"2024-03-11T20:59:04.534850+0000","last_clean_scrub_stamp":"2024-03-11T20:59:04.534850+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:00:13.223167+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":274,"reported_epoch":196,"state":"undersized+peered","last_fresh":"2024-03-11T21:00:02.711056+0000","last_change":"2024-03-11T20:58:53.588633+0000","last_active":"2024-03-11T20:57:59.071472+0000","last_peered":"2024-03-11T21:00:02.711056+0000","last_clean":"2024-03-11T20:57:55.801738+0000","last_became_active":"2024-03-11T20:57:24.633452+0000","last_became_peered":"2024-03-11T20:58:53.588633+0000","last_unstale":"2024-03-11T21:00:02.711056+0000","last_undegraded":"2024-03-11T21:00:02.711056+0000","last_fullsized":"2024-03-11T20:58:53.587575+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:57:23.036253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:57:23.036253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:00:06.663121+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":151,"reported_epoch":196,"state":"active+undersized","last_fresh":"2024-03-11T21:00:02.439821+0000","last_change":"2024-03-11T20:57:59.737265+0000","last_active":"2024-03-11T21:00:02.439821+0000","last_peered":"2024-03-11T21:00:02.439821+0000","last_clean":"2024-03-11T20:50:26.989698+0000","last_became_active":"2024-03-11T20:57:59.737265+0000","last_became_peered":"2024-03-11T20:57:59.737265+0000","last_unstale":"2024-03-11T21:00:02.439821+0000","last_undegraded":"2024-03-11T21:00:02.439821+0000","last_fullsized":"2024-03-11T20:57:59.716555+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:57:17.487427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:57:17.487427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":["5","0"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"161'15","reported_seq":252,"reported_epoch":196,"state":"active+undersized+degraded","last_fresh":"2024-03-11T21:00:02.809374+0000","last_change":"2024-03-11T20:58:53.591564+0000","last_active":"2024-03-11T21:00:02.809374+0000","last_peered":"2024-03-11T21:00:02.809374+0000","last_clean":"2024-03-11T20:57:15.103541+0000","last_became_active":"2024-03-11T20:58:53.591564+0000","last_became_peered":"2024-03-11T20:58:53.591564+0000","last_unstale":"2024-03-11T21:00:02.809374+0000","last_undegraded":"2024-03-11T20:58:53.585835+0000","last_fullsized":"2024-03-11T20:58:53.585704+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"161'15","last_scrub_stamp":"2024-03-11T20:57:15.125548+0000","last_deep_scrub":"161'15","last_deep_scrub_stamp":"2024-03-11T20:57:15.125548+0000","last_clean_scrub_stamp":"2024-03-11T20:57:15.125548+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[{"shards":"6,7","objects":5}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":287,"reported_epoch":196,"state":"active+undersized","last_fresh":"2024-03-11T21:00:02.754339+0000","last_change":"2024-03-11T20:57:59.790697+0000","last_active":"2024-03-11T21:00:02.754339+0000","last_peered":"2024-03-11T21:00:02.754339+0000","last_clean":"2024-03-11T20:57:55.791075+0000","last_became_active":"2024-03-11T20:57:59.790697+0000","last_became_peered":"2024-03-11T20:57:59.790697+0000","last_unstale":"2024-03-11T21:00:02.754339+0000","last_undegraded":"2024-03-11T21:00:02.754339+0000","last_fullsized":"2024-03-11T20:57:59.714976+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:57:01.885405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:57:01.885405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":142,"reported_epoch":196,"state":"active+undersized","last_fresh":"2024-03-11T21:00:02.710585+0000","last_change":"2024-03-11T20:57:59.753917+0000","last_active":"2024-03-11T21:00:02.710585+0000","last_peered":"2024-03-11T21:00:02.710585+0000","last_clean":"2024-03-11T20:50:31.136392+0000","last_became_active":"2024-03-11T20:57:59.753917+0000","last_became_peered":"2024-03-11T20:57:59.753917+0000","last_unstale":"2024-03-11T21:00:02.710585+0000","last_undegraded":"2024-03-11T21:00:02.710585+0000","last_fullsized":"2024-03-11T20:57:59.731364+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:55:47.753927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:55:47.753927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:00:06.663017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":292,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-11T21:00:02.711116+0000","last_change":"2024-03-11T20:58:38.880487+0000","last_active":"2024-03-11T21:00:02.711116+0000","last_peered":"2024-03-11T21:00:02.711116+0000","last_clean":"2024-03-11T21:00:02.711116+0000","last_became_active":"2024-03-11T20:50:28.082737+0000","last_became_peered":"2024-03-11T20:50:28.082737+0000","last_unstale":"2024-03-11T21:00:02.711116+0000","last_undegraded":"2024-03-11T21:00:02.711116+0000","last_fullsized":"2024-03-11T21:00:02.711116+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:58:38.880270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:58:38.880270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"84'9","reported_seq":299,"reported_epoch":196,"state":"active+undersized+degraded","last_fresh":"2024-03-11T21:00:02.810698+0000","last_change":"2024-03-11T20:58:53.592219+0000","last_active":"2024-03-11T21:00:02.810698+0000","last_peered":"2024-03-11T21:00:02.810698+0000","last_clean":"2024-03-11T20:58:51.941332+0000","last_became_active":"2024-03-11T20:58:53.592219+0000","last_became_peered":"2024-03-11T20:58:53.592219+0000","last_unstale":"2024-03-11T21:00:02.810698+0000","last_undegraded":"2024-03-11T20:58:53.587772+0000","last_fullsized":"2024-03-11T20:58:53.587640+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'9","last_scrub_stamp":"2024-03-11T20:56:00.976022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:56:00.976022+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[{"shards":"3,7","objects":4}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":32,"reported_epoch":196,"state":"active+undersized","last_fresh":"2024-03-11T21:00:02.763492+0000","last_change":"2024-03-11T20:58:53.591765+0000","last_active":"2024-03-11T21:00:02.763492+0000","last_peered":"2024-03-11T21:00:02.763492+0000","last_clean":"2024-03-11T20:50:33.234063+0000","last_became_active":"2024-03-11T20:58:53.591765+0000","last_became_peered":"2024-03-11T20:58:53.591765+0000","last_unstale":"2024-03-11T21:00:02.763492+0000","last_undegraded":"2024-03-11T21:00:02.763492+0000","last_fullsized":"2024-03-11T20:58:53.578669+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:57:16.032302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:57:16.032302+0000","last_clean_scrub_stamp":"2024-03-11T20:57:16.032302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:00:07.260236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"112'12","reported_seq":403,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-11T21:00:02.440284+0000","last_change":"2024-03-11T20:58:42.004007+0000","last_active":"2024-03-11T21:00:02.440284+0000","last_peered":"2024-03-11T21:00:02.440284+0000","last_clean":"2024-03-11T21:00:02.440284+0000","last_became_active":"2024-03-11T20:50:32.650064+0000","last_became_peered":"2024-03-11T20:50:32.650064+0000","last_unstale":"2024-03-11T21:00:02.440284+0000","last_undegraded":"2024-03-11T21:00:02.440284+0000","last_fullsized":"2024-03-11T21:00:02.440284+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'12","last_scrub_stamp":"2024-03-11T20:58:42.003708+0000","last_deep_scrub":"112'12","last_deep_scrub_stamp":"2024-03-11T20:53:11.843951+0000","last_clean_scrub_stamp":"2024-03-11T20:58:42.003708+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:00:05.627377+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":35,"reported_epoch":196,"state":"active+undersized","last_fresh":"2024-03-11T21:00:02.443251+0000","last_change":"2024-03-11T20:58:53.595194+0000","last_active":"2024-03-11T21:00:02.443251+0000","last_peered":"2024-03-11T21:00:02.443251+0000","last_clean":"2024-03-11T20:50:26.989698+0000","last_became_active":"2024-03-11T20:58:53.595194+0000","last_became_peered":"2024-03-11T20:58:53.595194+0000","last_unstale":"2024-03-11T21:00:02.443251+0000","last_undegraded":"2024-03-11T21:00:02.443251+0000","last_fullsized":"2024-03-11T20:58:53.586209+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:58:35.154174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:58:35.154174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"119'13","reported_seq":219,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-11T21:00:02.808103+0000","last_change":"2024-03-11T20:58:40.211566+0000","last_active":"2024-03-11T21:00:02.808103+0000","last_peered":"2024-03-11T21:00:02.808103+0000","last_clean":"2024-03-11T21:00:02.808103+0000","last_became_active":"2024-03-11T20:56:59.796618+0000","last_became_peered":"2024-03-11T20:56:59.796618+0000","last_unstale":"2024-03-11T21:00:02.808103+0000","last_undegraded":"2024-03-11T21:00:02.808103+0000","last_fullsized":"2024-03-11T21:00:02.808103+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'13","last_scrub_stamp":"2024-03-11T20:58:40.211330+0000","last_deep_scrub":"119'13","last_deep_scrub_stamp":"2024-03-11T20:54:15.443472+0000","last_clean_scrub_stamp":"2024-03-11T20:58:40.211330+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:00:08.263130+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":93,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-11T21:00:02.809897+0000","last_change":"2024-03-11T20:58:55.398504+0000","last_active":"2024-03-11T21:00:02.809897+0000","last_peered":"2024-03-11T21:00:02.809897+0000","last_clean":"2024-03-11T21:00:02.809897+0000","last_became_active":"2024-03-11T20:56:59.804346+0000","last_became_peered":"2024-03-11T20:56:59.804346+0000","last_unstale":"2024-03-11T21:00:02.809897+0000","last_undegraded":"2024-03-11T21:00:02.809897+0000","last_fullsized":"2024-03-11T21:00:02.809897+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:58:55.398295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:58:55.398295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"175'16","reported_seq":427,"reported_epoch":196,"state":"active+undersized+degraded","last_fresh":"2024-03-11T21:00:02.445824+0000","last_change":"2024-03-11T20:57:59.735460+0000","last_active":"2024-03-11T21:00:02.445824+0000","last_peered":"2024-03-11T21:00:02.445824+0000","last_clean":"2024-03-11T20:57:58.534403+0000","last_became_active":"2024-03-11T20:57:59.735460+0000","last_became_peered":"2024-03-11T20:57:59.735460+0000","last_unstale":"2024-03-11T21:00:02.445824+0000","last_undegraded":"2024-03-11T20:57:59.716258+0000","last_fullsized":"2024-03-11T20:57:59.715113+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'13","last_scrub_stamp":"2024-03-11T20:54:30.577957+0000","last_deep_scrub":"110'13","last_deep_scrub_stamp":"2024-03-11T20:53:00.434532+0000","last_clean_scrub_stamp":"2024-03-11T20:54:30.577957+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:00:07.750986+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"3,7","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":293,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-11T21:00:02.439218+0000","last_change":"2024-03-11T20:58:52.992412+0000","last_active":"2024-03-11T21:00:02.439218+0000","last_peered":"2024-03-11T21:00:02.439218+0000","last_clean":"2024-03-11T21:00:02.439218+0000","last_became_active":"2024-03-11T20:50:28.070325+0000","last_became_peered":"2024-03-11T20:50:28.070325+0000","last_unstale":"2024-03-11T21:00:02.439218+0000","last_undegraded":"2024-03-11T21:00:02.439218+0000","last_fullsized":"2024-03-11T21:00:02.439218+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:58:52.992284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:58:52.992284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:00:19.616539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"116'16","reported_seq":308,"reported_epoch":196,"state":"active+undersized+degraded","last_fresh":"2024-03-11T21:00:02.808552+0000","last_change":"2024-03-11T20:58:53.596372+0000","last_active":"2024-03-11T21:00:02.808552+0000","last_peered":"2024-03-11T21:00:02.808552+0000","last_clean":"2024-03-11T20:58:51.924513+0000","last_became_active":"2024-03-11T20:58:53.596372+0000","last_became_peered":"2024-03-11T20:58:53.596372+0000","last_unstale":"2024-03-11T21:00:02.808552+0000","last_undegraded":"2024-03-11T20:58:53.582906+0000","last_fullsized":"2024-03-11T20:58:53.582771+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'16","last_scrub_stamp":"2024-03-11T20:58:51.302626+0000","last_deep_scrub":"116'16","last_deep_scrub_stamp":"2024-03-11T20:53:39.292118+0000","last_clean_scrub_stamp":"2024-03-11T20:58:51.302626+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[{"shards":"2,7","objects":5}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":267,"reported_epoch":196,"state":"active+undersized","last_fresh":"2024-03-11T21:00:02.809279+0000","last_change":"2024-03-11T20:58:53.589146+0000","last_active":"2024-03-11T21:00:02.809279+0000","last_peered":"2024-03-11T21:00:02.809279+0000","last_clean":"2024-03-11T20:58:51.929605+0000","last_became_active":"2024-03-11T20:58:53.589146+0000","last_became_peered":"2024-03-11T20:58:53.589146+0000","last_unstale":"2024-03-11T21:00:02.809279+0000","last_undegraded":"2024-03-11T21:00:02.809279+0000","last_fullsized":"2024-03-11T20:58:53.584052+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:55:58.016988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:55:58.016988+0000","last_clean_scrub_stamp":"2024-03-11T20:55:58.016988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":290,"reported_epoch":196,"state":"active+undersized","last_fresh":"2024-03-11T21:00:02.448438+0000","last_change":"2024-03-11T20:57:59.731763+0000","last_active":"2024-03-11T21:00:02.448438+0000","last_peered":"2024-03-11T21:00:02.448438+0000","last_clean":"2024-03-11T20:57:55.788208+0000","last_became_active":"2024-03-11T20:57:59.731763+0000","last_became_peered":"2024-03-11T20:57:59.731763+0000","last_unstale":"2024-03-11T21:00:02.448438+0000","last_undegraded":"2024-03-11T21:00:02.448438+0000","last_fullsized":"2024-03-11T20:57:59.716767+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:57:10.433625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:57:10.433625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"166'2","reported_seq":318,"reported_epoch":196,"state":"undersized+degraded+peered","last_fresh":"2024-03-11T21:00:02.710459+0000","last_change":"2024-03-11T20:58:53.587728+0000","last_active":"2024-03-11T20:58:52.589543+0000","last_peered":"2024-03-11T21:00:02.710459+0000","last_clean":"2024-03-11T20:57:55.795676+0000","last_became_active":"2024-03-11T20:57:59.752730+0000","last_became_peered":"2024-03-11T20:58:53.587728+0000","last_unstale":"2024-03-11T21:00:02.710459+0000","last_undegraded":"2024-03-11T20:58:53.585428+0000","last_fullsized":"2024-03-11T20:58:53.585342+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'2","last_scrub_stamp":"2024-03-11T20:57:39.868776+0000","last_deep_scrub":"166'2","last_deep_scrub_stamp":"2024-03-11T20:57:39.868776+0000","last_clean_scrub_stamp":"2024-03-11T20:57:39.868776+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:00:06.663146+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1822,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[{"shards":"6","objects":1}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"84'4","reported_seq":286,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-11T21:00:02.710334+0000","last_change":"2024-03-11T20:58:51.796714+0000","last_active":"2024-03-11T21:00:02.710334+0000","last_peered":"2024-03-11T21:00:02.710334+0000","last_clean":"2024-03-11T21:00:02.710334+0000","last_became_active":"2024-03-11T20:50:30.101572+0000","last_became_peered":"2024-03-11T20:50:30.101572+0000","last_unstale":"2024-03-11T21:00:02.710334+0000","last_undegraded":"2024-03-11T21:00:02.710334+0000","last_fullsized":"2024-03-11T21:00:02.710334+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'4","last_scrub_stamp":"2024-03-11T20:58:51.796549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:58:51.796549+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:00:14.027916+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":290,"reported_epoch":196,"state":"active+undersized","last_fresh":"2024-03-11T21:00:02.761583+0000","last_change":"2024-03-11T20:58:53.592993+0000","last_active":"2024-03-11T21:00:02.761583+0000","last_peered":"2024-03-11T21:00:02.761583+0000","last_clean":"2024-03-11T20:58:51.852265+0000","last_became_active":"2024-03-11T20:58:53.592993+0000","last_became_peered":"2024-03-11T20:58:53.592993+0000","last_unstale":"2024-03-11T21:00:02.761583+0000","last_undegraded":"2024-03-11T21:00:02.761583+0000","last_fullsized":"2024-03-11T20:58:53.581588+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:58:47.869140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:58:47.869140+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":249,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-11T21:00:04.302716+0000","last_change":"2024-03-11T21:00:04.302784+0000","last_active":"2024-03-11T21:00:04.302716+0000","last_peered":"2024-03-11T21:00:04.302716+0000","last_clean":"2024-03-11T21:00:04.302716+0000","last_became_active":"2024-03-11T20:56:59.802356+0000","last_became_peered":"2024-03-11T20:56:59.802356+0000","last_unstale":"2024-03-11T21:00:04.302716+0000","last_undegraded":"2024-03-11T21:00:04.302716+0000","last_fullsized":"2024-03-11T21:00:04.302716+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:00:04.302541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:54:09.537972+0000","last_clean_scrub_stamp":"2024-03-11T21:00:04.302541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:01:17.834886+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":298,"reported_epoch":196,"state":"active+undersized","last_fresh":"2024-03-11T21:00:02.758953+0000","last_change":"2024-03-11T20:57:59.717891+0000","last_active":"2024-03-11T21:00:02.758953+0000","last_peered":"2024-03-11T21:00:02.758953+0000","last_clean":"2024-03-11T20:57:55.802282+0000","last_became_active":"2024-03-11T20:57:59.717891+0000","last_became_peered":"2024-03-11T20:57:59.717891+0000","last_unstale":"2024-03-11T21:00:02.758953+0000","last_undegraded":"2024-03-11T21:00:02.758953+0000","last_fullsized":"2024-03-11T20:57:59.710473+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:55:59.209833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:54:33.812279+0000","last_clean_scrub_stamp":"2024-03-11T20:55:59.209833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:00:07.260129+0000","scrub_duration":211,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":["2","0"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"84'11","reported_seq":295,"reported_epoch":196,"state":"active+undersized+degraded","last_fresh":"2024-03-11T21:00:02.807665+0000","last_change":"2024-03-11T20:58:53.585460+0000","last_active":"2024-03-11T21:00:02.807665+0000","last_peered":"2024-03-11T21:00:02.807665+0000","last_clean":"2024-03-11T20:58:51.927617+0000","last_became_active":"2024-03-11T20:58:53.585460+0000","last_became_peered":"2024-03-11T20:58:53.585460+0000","last_unstale":"2024-03-11T21:00:02.807665+0000","last_undegraded":"2024-03-11T20:58:53.582771+0000","last_fullsized":"2024-03-11T20:58:53.582629+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T20:55:55.052289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:55:55.052289+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[{"shards":"6,7","objects":5}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.5","version":"193'10","reported_seq":31,"reported_epoch":196,"state":"active+undersized+degraded","last_fresh":"2024-03-11T21:00:02.711496+0000","last_change":"2024-03-11T20:59:30.424537+0000","last_active":"2024-03-11T21:00:02.711496+0000","last_peered":"2024-03-11T21:00:02.711496+0000","last_clean":"2024-03-11T20:59:10.160042+0000","last_became_active":"2024-03-11T20:59:11.273538+0000","last_became_peered":"2024-03-11T20:59:11.273538+0000","last_unstale":"2024-03-11T21:00:02.711496+0000","last_undegraded":"2024-03-11T20:59:30.421595+0000","last_fullsized":"2024-03-11T20:59:11.195616+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:59:10.160042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:59:10.160042+0000","last_clean_scrub_stamp":"2024-03-11T20:59:10.160042+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:00:27.221504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":655360,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6","7"],"object_location_counts":[{"shards":"6,7","objects":10}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":287,"reported_epoch":196,"state":"active+undersized","last_fresh":"2024-03-11T21:00:02.445247+0000","last_change":"2024-03-11T20:57:59.730659+0000","last_active":"2024-03-11T21:00:02.445247+0000","last_peered":"2024-03-11T21:00:02.445247+0000","last_clean":"2024-03-11T20:57:55.788032+0000","last_became_active":"2024-03-11T20:57:59.730659+0000","last_became_peered":"2024-03-11T20:57:59.730659+0000","last_unstale":"2024-03-11T21:00:02.445247+0000","last_undegraded":"2024-03-11T21:00:02.445247+0000","last_fullsized":"2024-03-11T20:57:59.716761+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:57:04.417575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:52:53.450885+0000","last_clean_scrub_stamp":"2024-03-11T20:57:04.417575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"77'2","reported_seq":42,"reported_epoch":196,"state":"active+undersized+degraded","last_fresh":"2024-03-11T21:00:02.810366+0000","last_change":"2024-03-11T20:58:53.599311+0000","last_active":"2024-03-11T21:00:02.810366+0000","last_peered":"2024-03-11T21:00:02.810366+0000","last_clean":"2024-03-11T20:50:28.106660+0000","last_became_active":"2024-03-11T20:58:53.599311+0000","last_became_peered":"2024-03-11T20:58:53.599311+0000","last_unstale":"2024-03-11T21:00:02.810366+0000","last_undegraded":"2024-03-11T20:58:53.585399+0000","last_fullsized":"2024-03-11T20:58:53.585261+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'2","last_scrub_stamp":"2024-03-11T20:57:37.161076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:57:37.161076+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[{"shards":"0,7","objects":2}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"100'16","reported_seq":346,"reported_epoch":196,"state":"undersized+degraded+peered","last_fresh":"2024-03-11T21:00:02.810000+0000","last_change":"2024-03-11T20:58:53.589265+0000","last_active":"2024-03-11T20:58:52.597025+0000","last_peered":"2024-03-11T21:00:02.810000+0000","last_clean":"2024-03-11T20:57:55.790479+0000","last_became_active":"2024-03-11T20:57:59.732221+0000","last_became_peered":"2024-03-11T20:58:53.589265+0000","last_unstale":"2024-03-11T21:00:02.810000+0000","last_undegraded":"2024-03-11T20:58:53.588518+0000","last_fullsized":"2024-03-11T20:58:53.588425+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'16","last_scrub_stamp":"2024-03-11T20:56:08.036331+0000","last_deep_scrub":"100'16","last_deep_scrub_stamp":"2024-03-11T20:56:08.036331+0000","last_clean_scrub_stamp":"2024-03-11T20:56:08.036331+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[{"shards":"7","objects":5}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"175'15","reported_seq":129,"reported_epoch":196,"state":"undersized+degraded+peered","last_fresh":"2024-03-11T21:00:02.442092+0000","last_change":"2024-03-11T20:58:53.597111+0000","last_active":"2024-03-11T20:58:52.574944+0000","last_peered":"2024-03-11T21:00:02.442092+0000","last_clean":"2024-03-11T20:52:21.536060+0000","last_became_active":"2024-03-11T20:50:32.210620+0000","last_became_peered":"2024-03-11T20:58:53.597111+0000","last_unstale":"2024-03-11T21:00:02.442092+0000","last_undegraded":"2024-03-11T20:58:53.590101+0000","last_fullsized":"2024-03-11T20:58:53.588423+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"125'13","last_scrub_stamp":"2024-03-11T20:57:04.001441+0000","last_deep_scrub":"106'12","last_deep_scrub_stamp":"2024-03-11T20:52:21.538885+0000","last_clean_scrub_stamp":"2024-03-11T20:57:04.001441+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.e","version":"193'19","reported_seq":40,"reported_epoch":196,"state":"active+undersized+degraded","last_fresh":"2024-03-11T21:00:02.756166+0000","last_change":"2024-03-11T20:59:30.415128+0000","last_active":"2024-03-11T21:00:02.756166+0000","last_peered":"2024-03-11T21:00:02.756166+0000","last_clean":"2024-03-11T20:59:10.160042+0000","last_became_active":"2024-03-11T20:59:11.199909+0000","last_became_peered":"2024-03-11T20:59:11.199909+0000","last_unstale":"2024-03-11T21:00:02.756166+0000","last_undegraded":"2024-03-11T20:59:30.414380+0000","last_fullsized":"2024-03-11T20:59:11.184753+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:59:10.160042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:59:10.160042+0000","last_clean_scrub_stamp":"2024-03-11T20:59:10.160042+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:00:29.973684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1245184,"num_objects":19,"num_object_clones":0,"num_object_copies":57,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19,"num_write_kb":1216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":19}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":101,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-11T21:00:02.441948+0000","last_change":"2024-03-11T20:59:10.448936+0000","last_active":"2024-03-11T21:00:02.441948+0000","last_peered":"2024-03-11T21:00:02.441948+0000","last_clean":"2024-03-11T21:00:02.441948+0000","last_became_active":"2024-03-11T20:56:59.795191+0000","last_became_peered":"2024-03-11T20:56:59.795191+0000","last_unstale":"2024-03-11T21:00:02.441948+0000","last_undegraded":"2024-03-11T21:00:02.441948+0000","last_fullsized":"2024-03-11T21:00:02.441948+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:59:10.448842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:57:21.568400+0000","last_clean_scrub_stamp":"2024-03-11T20:59:10.448842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:00:16.157240+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":66,"reported_epoch":196,"state":"active+undersized","last_fresh":"2024-03-11T21:00:02.809272+0000","last_change":"2024-03-11T20:57:59.763312+0000","last_active":"2024-03-11T21:00:02.809272+0000","last_peered":"2024-03-11T21:00:02.809272+0000","last_clean":"2024-03-11T20:50:26.989698+0000","last_became_active":"2024-03-11T20:57:59.763312+0000","last_became_peered":"2024-03-11T20:57:59.763312+0000","last_unstale":"2024-03-11T21:00:02.809272+0000","last_undegraded":"2024-03-11T21:00:02.809272+0000","last_fullsized":"2024-03-11T20:57:59.724269+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:55:35.833890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:55:35.833890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.5","version":"84'16","reported_seq":419,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-11T21:00:02.439755+0000","last_change":"2024-03-11T20:58:46.023238+0000","last_active":"2024-03-11T21:00:02.439755+0000","last_peered":"2024-03-11T21:00:02.439755+0000","last_clean":"2024-03-11T21:00:02.439755+0000","last_became_active":"2024-03-11T20:50:30.094977+0000","last_became_peered":"2024-03-11T20:50:30.094977+0000","last_unstale":"2024-03-11T21:00:02.439755+0000","last_undegraded":"2024-03-11T21:00:02.439755+0000","last_fullsized":"2024-03-11T21:00:02.439755+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'16","last_scrub_stamp":"2024-03-11T20:58:46.022946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:58:46.022946+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":145,"reported_epoch":196,"state":"undersized+peered","last_fresh":"2024-03-11T21:00:02.439536+0000","last_change":"2024-03-11T20:58:53.584094+0000","last_active":"2024-03-11T20:58:52.583901+0000","last_peered":"2024-03-11T21:00:02.439536+0000","last_clean":"2024-03-11T20:50:31.136392+0000","last_became_active":"2024-03-11T20:57:59.741726+0000","last_became_peered":"2024-03-11T20:58:53.584094+0000","last_unstale":"2024-03-11T21:00:02.439536+0000","last_undegraded":"2024-03-11T21:00:02.439536+0000","last_fullsized":"2024-03-11T20:58:53.581253+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:56:58.614738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:52:10.160202+0000","last_clean_scrub_stamp":"2024-03-11T20:56:58.614738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.8","version":"193'10","reported_seq":34,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-11T21:00:02.438374+0000","last_change":"2024-03-11T20:59:11.277357+0000","last_active":"2024-03-11T21:00:02.438374+0000","last_peered":"2024-03-11T21:00:02.438374+0000","last_clean":"2024-03-11T21:00:02.438374+0000","last_became_active":"2024-03-11T20:59:11.276347+0000","last_became_peered":"2024-03-11T20:59:11.276347+0000","last_unstale":"2024-03-11T21:00:02.438374+0000","last_undegraded":"2024-03-11T21:00:02.438374+0000","last_fullsized":"2024-03-11T21:00:02.438374+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:59:10.160042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:59:10.160042+0000","last_clean_scrub_stamp":"2024-03-11T20:59:10.160042+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:00:23.044560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":655360,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":275,"reported_epoch":196,"state":"active+undersized","last_fresh":"2024-03-11T21:00:02.447172+0000","last_change":"2024-03-11T20:58:53.590457+0000","last_active":"2024-03-11T21:00:02.447172+0000","last_peered":"2024-03-11T21:00:02.447172+0000","last_clean":"2024-03-11T20:58:51.566132+0000","last_became_active":"2024-03-11T20:58:53.590457+0000","last_became_peered":"2024-03-11T20:58:53.590457+0000","last_unstale":"2024-03-11T21:00:02.447172+0000","last_undegraded":"2024-03-11T21:00:02.447172+0000","last_fullsized":"2024-03-11T20:58:53.585672+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:58:33.553195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:58:33.553195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":300,"reported_epoch":196,"state":"active+undersized","last_fresh":"2024-03-11T21:00:02.439340+0000","last_change":"2024-03-11T20:57:59.754777+0000","last_active":"2024-03-11T21:00:02.439340+0000","last_peered":"2024-03-11T21:00:02.439340+0000","last_clean":"2024-03-11T20:57:55.794128+0000","last_became_active":"2024-03-11T20:57:59.754777+0000","last_became_peered":"2024-03-11T20:57:59.754777+0000","last_unstale":"2024-03-11T21:00:02.439340+0000","last_undegraded":"2024-03-11T21:00:02.439340+0000","last_fullsized":"2024-03-11T20:57:59.715656+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:57:15.573623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:33.823704+0000","last_clean_scrub_stamp":"2024-03-11T20:57:15.573623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"182'27","reported_seq":324,"reported_epoch":196,"state":"active+undersized+degraded","last_fresh":"2024-03-11T21:00:02.710939+0000","last_change":"2024-03-11T20:58:53.598409+0000","last_active":"2024-03-11T21:00:02.710939+0000","last_peered":"2024-03-11T21:00:02.710939+0000","last_clean":"2024-03-11T20:58:52.035167+0000","last_became_active":"2024-03-11T20:58:53.598409+0000","last_became_peered":"2024-03-11T20:58:53.598409+0000","last_unstale":"2024-03-11T21:00:02.710939+0000","last_undegraded":"2024-03-11T20:58:53.587213+0000","last_fullsized":"2024-03-11T20:58:53.587053+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'27","last_scrub_stamp":"2024-03-11T20:58:52.058851+0000","last_deep_scrub":"182'27","last_deep_scrub_stamp":"2024-03-11T20:58:52.058851+0000","last_clean_scrub_stamp":"2024-03-11T20:58:52.058851+0000","objects_scrubbed":8,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:00:08.030013+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":["6","0"],"object_location_counts":[{"shards":"0,6","objects":8}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":149,"reported_epoch":196,"state":"active+undersized","last_fresh":"2024-03-11T21:00:02.760980+0000","last_change":"2024-03-11T20:57:59.715918+0000","last_active":"2024-03-11T21:00:02.760980+0000","last_peered":"2024-03-11T21:00:02.760980+0000","last_clean":"2024-03-11T20:57:12.890206+0000","last_became_active":"2024-03-11T20:57:59.715918+0000","last_became_peered":"2024-03-11T20:57:59.715918+0000","last_unstale":"2024-03-11T21:00:02.760980+0000","last_undegraded":"2024-03-11T21:00:02.760980+0000","last_fullsized":"2024-03-11T20:57:59.709808+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:56:54.415623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:56:54.415623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:00:07.260170+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":18,"reported_epoch":196,"state":"undersized+peered","last_fresh":"2024-03-11T21:00:02.442427+0000","last_change":"2024-03-11T20:59:11.195389+0000","last_active":"2024-03-11T20:59:10.160042+0000","last_peered":"2024-03-11T21:00:02.442427+0000","last_clean":"2024-03-11T20:59:10.160042+0000","last_became_active":"0.000000","last_became_peered":"2024-03-11T20:59:11.195389+0000","last_unstale":"2024-03-11T21:00:02.442427+0000","last_undegraded":"2024-03-11T21:00:02.442427+0000","last_fullsized":"2024-03-11T20:59:11.191704+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:59:10.160042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:59:10.160042+0000","last_clean_scrub_stamp":"2024-03-11T20:59:10.160042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:00:26.710802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":281,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-11T21:00:04.787308+0000","last_change":"2024-03-11T21:00:04.787337+0000","last_active":"2024-03-11T21:00:04.787308+0000","last_peered":"2024-03-11T21:00:04.787308+0000","last_clean":"2024-03-11T21:00:04.787308+0000","last_became_active":"2024-03-11T20:50:34.321038+0000","last_became_peered":"2024-03-11T20:50:34.321038+0000","last_unstale":"2024-03-11T21:00:04.787308+0000","last_undegraded":"2024-03-11T21:00:04.787308+0000","last_fullsized":"2024-03-11T21:00:04.787308+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:00:04.787220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:00:04.787220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:01:30.613261+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"109'26","reported_seq":282,"reported_epoch":196,"state":"active+undersized+degraded","last_fresh":"2024-03-11T21:00:02.758909+0000","last_change":"2024-03-11T20:57:59.729455+0000","last_active":"2024-03-11T21:00:02.758909+0000","last_peered":"2024-03-11T21:00:02.758909+0000","last_clean":"2024-03-11T20:57:12.888615+0000","last_became_active":"2024-03-11T20:57:59.729455+0000","last_became_peered":"2024-03-11T20:57:59.729455+0000","last_unstale":"2024-03-11T21:00:02.758909+0000","last_undegraded":"2024-03-11T20:57:59.709717+0000","last_fullsized":"2024-03-11T20:57:59.709549+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'26","last_scrub_stamp":"2024-03-11T20:56:58.455129+0000","last_deep_scrub":"109'26","last_deep_scrub_stamp":"2024-03-11T20:52:49.216378+0000","last_clean_scrub_stamp":"2024-03-11T20:56:58.455129+0000","objects_scrubbed":8,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:00:07.260192+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":["2","6"],"object_location_counts":[{"shards":"2,6","objects":8}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":299,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-11T21:00:02.759736+0000","last_change":"2024-03-11T20:58:50.924866+0000","last_active":"2024-03-11T21:00:02.759736+0000","last_peered":"2024-03-11T21:00:02.759736+0000","last_clean":"2024-03-11T21:00:02.759736+0000","last_became_active":"2024-03-11T20:50:28.045625+0000","last_became_peered":"2024-03-11T20:50:28.045625+0000","last_unstale":"2024-03-11T21:00:02.759736+0000","last_undegraded":"2024-03-11T21:00:02.759736+0000","last_fullsized":"2024-03-11T21:00:02.759736+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:58:50.924633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:58:50.924633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:00:09.989631+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":143,"reported_epoch":196,"state":"active+undersized","last_fresh":"2024-03-11T21:00:02.439983+0000","last_change":"2024-03-11T20:57:59.736501+0000","last_active":"2024-03-11T21:00:02.439983+0000","last_peered":"2024-03-11T21:00:02.439983+0000","last_clean":"2024-03-11T20:50:31.136392+0000","last_became_active":"2024-03-11T20:57:59.736501+0000","last_became_peered":"2024-03-11T20:57:59.736501+0000","last_unstale":"2024-03-11T21:00:02.439983+0000","last_undegraded":"2024-03-11T21:00:02.439983+0000","last_fullsized":"2024-03-11T20:57:59.717171+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:57:06.588166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:57:06.588166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":["5","0"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"193'19","reported_seq":43,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-11T21:00:02.710632+0000","last_change":"2024-03-11T20:59:11.206916+0000","last_active":"2024-03-11T21:00:02.710632+0000","last_peered":"2024-03-11T21:00:02.710632+0000","last_clean":"2024-03-11T21:00:02.710632+0000","last_became_active":"2024-03-11T20:59:11.205958+0000","last_became_peered":"2024-03-11T20:59:11.205958+0000","last_unstale":"2024-03-11T21:00:02.710632+0000","last_undegraded":"2024-03-11T21:00:02.710632+0000","last_fullsized":"2024-03-11T21:00:02.710632+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:59:10.160042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:59:10.160042+0000","last_clean_scrub_stamp":"2024-03-11T20:59:10.160042+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:00:17.242680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1245184,"num_objects":19,"num_object_clones":0,"num_object_copies":57,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19,"num_write_kb":1216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":276,"reported_epoch":196,"state":"active+undersized","last_fresh":"2024-03-11T21:00:02.440115+0000","last_change":"2024-03-11T20:58:53.593205+0000","last_active":"2024-03-11T21:00:02.440115+0000","last_peered":"2024-03-11T21:00:02.440115+0000","last_clean":"2024-03-11T20:58:51.948758+0000","last_became_active":"2024-03-11T20:58:53.593205+0000","last_became_peered":"2024-03-11T20:58:53.593205+0000","last_unstale":"2024-03-11T21:00:02.440115+0000","last_undegraded":"2024-03-11T21:00:02.440115+0000","last_fullsized":"2024-03-11T20:58:53.581807+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:58:48.005486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:58:48.005486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:00:15.056114+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"84'10","reported_seq":295,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-11T21:00:02.442769+0000","last_change":"2024-03-11T20:59:02.537240+0000","last_active":"2024-03-11T21:00:02.442769+0000","last_peered":"2024-03-11T21:00:02.442769+0000","last_clean":"2024-03-11T21:00:02.442769+0000","last_became_active":"2024-03-11T20:50:30.101747+0000","last_became_peered":"2024-03-11T20:50:30.101747+0000","last_unstale":"2024-03-11T21:00:02.442769+0000","last_undegraded":"2024-03-11T21:00:02.442769+0000","last_fullsized":"2024-03-11T21:00:02.442769+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'10","last_scrub_stamp":"2024-03-11T20:59:02.536990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:59:02.536990+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:00:11.459389+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":294,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-11T21:00:02.439132+0000","last_change":"2024-03-11T20:58:44.016367+0000","last_active":"2024-03-11T21:00:02.439132+0000","last_peered":"2024-03-11T21:00:02.439132+0000","last_clean":"2024-03-11T21:00:02.439132+0000","last_became_active":"2024-03-11T20:50:28.085503+0000","last_became_peered":"2024-03-11T20:50:28.085503+0000","last_unstale":"2024-03-11T21:00:02.439132+0000","last_undegraded":"2024-03-11T21:00:02.439132+0000","last_fullsized":"2024-03-11T21:00:02.439132+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:58:44.016257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:56:03.479180+0000","last_clean_scrub_stamp":"2024-03-11T20:58:44.016257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:00:11.376795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"184'130","reported_seq":554,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-11T21:00:02.807989+0000","last_change":"2024-03-11T20:58:56.415519+0000","last_active":"2024-03-11T21:00:02.807989+0000","last_peered":"2024-03-11T21:00:02.807989+0000","last_clean":"2024-03-11T21:00:02.807989+0000","last_became_active":"2024-03-11T20:49:52.584543+0000","last_became_peered":"2024-03-11T20:49:52.584543+0000","last_unstale":"2024-03-11T21:00:02.807989+0000","last_undegraded":"2024-03-11T21:00:02.807989+0000","last_fullsized":"2024-03-11T21:00:02.807989+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"184'130","last_scrub_stamp":"2024-03-11T20:58:56.415267+0000","last_deep_scrub":"184'130","last_deep_scrub_stamp":"2024-03-11T20:58:56.415267+0000","last_clean_scrub_stamp":"2024-03-11T20:58:56.415267+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":287,"reported_epoch":196,"state":"active+undersized","last_fresh":"2024-03-11T21:00:02.711721+0000","last_change":"2024-03-11T20:58:53.591630+0000","last_active":"2024-03-11T21:00:02.711721+0000","last_peered":"2024-03-11T21:00:02.711721+0000","last_clean":"2024-03-11T20:58:51.933033+0000","last_became_active":"2024-03-11T20:58:53.591630+0000","last_became_peered":"2024-03-11T20:58:53.591630+0000","last_unstale":"2024-03-11T21:00:02.711721+0000","last_undegraded":"2024-03-11T21:00:02.711721+0000","last_fullsized":"2024-03-11T20:58:53.587631+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:58:43.888174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:54:22.448812+0000","last_clean_scrub_stamp":"2024-03-11T20:58:43.888174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6","7"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.a","version":"193'11","reported_seq":32,"reported_epoch":196,"state":"active+undersized+degraded","last_fresh":"2024-03-11T21:00:02.759300+0000","last_change":"2024-03-11T20:59:30.420680+0000","last_active":"2024-03-11T21:00:02.759300+0000","last_peered":"2024-03-11T21:00:02.759300+0000","last_clean":"2024-03-11T20:59:10.160042+0000","last_became_active":"2024-03-11T20:59:11.201378+0000","last_became_peered":"2024-03-11T20:59:11.201378+0000","last_unstale":"2024-03-11T21:00:02.759300+0000","last_undegraded":"2024-03-11T20:59:30.418338+0000","last_fullsized":"2024-03-11T20:59:11.193426+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:59:10.160042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:59:10.160042+0000","last_clean_scrub_stamp":"2024-03-11T20:59:10.160042+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:00:12.865615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":720896,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11,"num_write_kb":704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[{"shards":"2,5","objects":11}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":144,"reported_epoch":196,"state":"active+undersized","last_fresh":"2024-03-11T21:00:02.439178+0000","last_change":"2024-03-11T20:57:59.738257+0000","last_active":"2024-03-11T21:00:02.439178+0000","last_peered":"2024-03-11T21:00:02.439178+0000","last_clean":"2024-03-11T20:50:33.234063+0000","last_became_active":"2024-03-11T20:57:59.738257+0000","last_became_peered":"2024-03-11T20:57:59.738257+0000","last_unstale":"2024-03-11T21:00:02.439178+0000","last_undegraded":"2024-03-11T21:00:02.439178+0000","last_fullsized":"2024-03-11T20:57:59.715080+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:55:58.547789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:55:58.547789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"100'21","reported_seq":110,"reported_epoch":196,"state":"active+undersized+degraded","last_fresh":"2024-03-11T21:00:02.443641+0000","last_change":"2024-03-11T20:58:53.598297+0000","last_active":"2024-03-11T21:00:02.443641+0000","last_peered":"2024-03-11T21:00:02.443641+0000","last_clean":"2024-03-11T20:51:31.825760+0000","last_became_active":"2024-03-11T20:58:53.598297+0000","last_became_peered":"2024-03-11T20:58:53.598297+0000","last_unstale":"2024-03-11T21:00:02.443641+0000","last_undegraded":"2024-03-11T20:58:53.592867+0000","last_fullsized":"2024-03-11T20:58:53.592200+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'21","last_scrub_stamp":"2024-03-11T20:58:44.148359+0000","last_deep_scrub":"100'21","last_deep_scrub_stamp":"2024-03-11T20:51:31.835302+0000","last_clean_scrub_stamp":"2024-03-11T20:58:44.148359+0000","objects_scrubbed":7,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:00:11.206612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":7}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":264,"reported_epoch":196,"state":"active+undersized","last_fresh":"2024-03-11T21:00:02.759695+0000","last_change":"2024-03-11T20:57:59.717535+0000","last_active":"2024-03-11T21:00:02.759695+0000","last_peered":"2024-03-11T21:00:02.759695+0000","last_clean":"2024-03-11T20:57:55.802531+0000","last_became_active":"2024-03-11T20:57:59.717535+0000","last_became_peered":"2024-03-11T20:57:59.717535+0000","last_unstale":"2024-03-11T21:00:02.759695+0000","last_undegraded":"2024-03-11T21:00:02.759695+0000","last_fullsized":"2024-03-11T20:57:59.710157+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:57:10.475850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:57:10.475850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:00:07.260278+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":["2","0"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":291,"reported_epoch":196,"state":"active+undersized","last_fresh":"2024-03-11T21:00:02.757097+0000","last_change":"2024-03-11T20:57:59.716585+0000","last_active":"2024-03-11T21:00:02.757097+0000","last_peered":"2024-03-11T21:00:02.757097+0000","last_clean":"2024-03-11T20:57:55.792941+0000","last_became_active":"2024-03-11T20:57:59.716585+0000","last_became_peered":"2024-03-11T20:57:59.716585+0000","last_unstale":"2024-03-11T21:00:02.757097+0000","last_undegraded":"2024-03-11T21:00:02.757097+0000","last_fullsized":"2024-03-11T20:57:59.711634+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:56:58.833050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:56:58.833050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.f","version":"193'11","reported_seq":32,"reported_epoch":196,"state":"active+undersized+degraded","last_fresh":"2024-03-11T21:00:02.447137+0000","last_change":"2024-03-11T20:59:30.416345+0000","last_active":"2024-03-11T21:00:02.447137+0000","last_peered":"2024-03-11T21:00:02.447137+0000","last_clean":"2024-03-11T20:59:10.160042+0000","last_became_active":"2024-03-11T20:59:11.199244+0000","last_became_peered":"2024-03-11T20:59:11.199244+0000","last_unstale":"2024-03-11T21:00:02.447137+0000","last_undegraded":"2024-03-11T20:59:30.411720+0000","last_fullsized":"2024-03-11T20:59:11.191431+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:59:10.160042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:59:10.160042+0000","last_clean_scrub_stamp":"2024-03-11T20:59:10.160042+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:00:31.977299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":720896,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11,"num_write_kb":704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":11}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":143,"reported_epoch":196,"state":"active+undersized","last_fresh":"2024-03-11T21:00:02.710893+0000","last_change":"2024-03-11T20:57:59.753190+0000","last_active":"2024-03-11T21:00:02.710893+0000","last_peered":"2024-03-11T21:00:02.710893+0000","last_clean":"2024-03-11T20:50:33.234063+0000","last_became_active":"2024-03-11T20:57:59.753190+0000","last_became_peered":"2024-03-11T20:57:59.753190+0000","last_unstale":"2024-03-11T21:00:02.710893+0000","last_undegraded":"2024-03-11T21:00:02.710893+0000","last_fullsized":"2024-03-11T20:57:59.727788+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:55:38.551514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:55:38.551514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:00:06.663036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"145'3","reported_seq":294,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-11T21:00:02.710927+0000","last_change":"2024-03-11T21:00:01.678041+0000","last_active":"2024-03-11T21:00:02.710927+0000","last_peered":"2024-03-11T21:00:02.710927+0000","last_clean":"2024-03-11T21:00:02.710927+0000","last_became_active":"2024-03-11T20:50:32.195452+0000","last_became_peered":"2024-03-11T20:50:32.195452+0000","last_unstale":"2024-03-11T21:00:02.710927+0000","last_undegraded":"2024-03-11T21:00:02.710927+0000","last_fullsized":"2024-03-11T21:00:02.710927+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'3","last_scrub_stamp":"2024-03-11T21:00:01.677888+0000","last_deep_scrub":"145'3","last_deep_scrub_stamp":"2024-03-11T20:55:52.848043+0000","last_clean_scrub_stamp":"2024-03-11T21:00:01.677888+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:01:16.699096+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0.0024049620000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":288,"reported_epoch":196,"state":"active+undersized","last_fresh":"2024-03-11T21:00:02.810017+0000","last_change":"2024-03-11T20:58:53.599442+0000","last_active":"2024-03-11T21:00:02.810017+0000","last_peered":"2024-03-11T21:00:02.810017+0000","last_clean":"2024-03-11T20:58:51.930220+0000","last_became_active":"2024-03-11T20:58:53.599442+0000","last_became_peered":"2024-03-11T20:58:53.599442+0000","last_unstale":"2024-03-11T21:00:02.810017+0000","last_undegraded":"2024-03-11T21:00:02.810017+0000","last_fullsized":"2024-03-11T20:58:53.587065+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:57:24.253503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:57:24.253503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"175'33","reported_seq":303,"reported_epoch":196,"state":"active+undersized+degraded","last_fresh":"2024-03-11T21:00:02.761394+0000","last_change":"2024-03-11T20:57:59.737341+0000","last_active":"2024-03-11T21:00:02.761394+0000","last_peered":"2024-03-11T21:00:02.761394+0000","last_clean":"2024-03-11T20:57:23.466211+0000","last_became_active":"2024-03-11T20:57:59.737341+0000","last_became_peered":"2024-03-11T20:57:59.737341+0000","last_unstale":"2024-03-11T21:00:02.761394+0000","last_undegraded":"2024-03-11T20:57:59.710058+0000","last_fullsized":"2024-03-11T20:57:59.709904+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"125'31","last_scrub_stamp":"2024-03-11T20:57:13.436634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:57:13.436634+0000","objects_scrubbed":10,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:00:07.260216+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":51,"num_read_kb":36,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[{"shards":"2,5","objects":10}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"160'12","reported_seq":157,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-11T21:00:02.438783+0000","last_change":"2024-03-11T20:58:41.004369+0000","last_active":"2024-03-11T21:00:02.438783+0000","last_peered":"2024-03-11T21:00:02.438783+0000","last_clean":"2024-03-11T21:00:02.438783+0000","last_became_active":"2024-03-11T20:56:59.797805+0000","last_became_peered":"2024-03-11T20:56:59.797805+0000","last_unstale":"2024-03-11T21:00:02.438783+0000","last_undegraded":"2024-03-11T21:00:02.438783+0000","last_fullsized":"2024-03-11T21:00:02.438783+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"160'12","last_scrub_stamp":"2024-03-11T20:58:41.004207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:58:41.004207+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:00:06.513804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.c","version":"193'14","reported_seq":38,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-11T21:00:02.755227+0000","last_change":"2024-03-11T20:59:11.269951+0000","last_active":"2024-03-11T21:00:02.755227+0000","last_peered":"2024-03-11T21:00:02.755227+0000","last_clean":"2024-03-11T21:00:02.755227+0000","last_became_active":"2024-03-11T20:59:11.269504+0000","last_became_peered":"2024-03-11T20:59:11.269504+0000","last_unstale":"2024-03-11T21:00:02.755227+0000","last_undegraded":"2024-03-11T21:00:02.755227+0000","last_fullsized":"2024-03-11T21:00:02.755227+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:59:10.160042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:59:10.160042+0000","last_clean_scrub_stamp":"2024-03-11T20:59:10.160042+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:00:10.488340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":917504,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":33,"reported_epoch":196,"state":"active+undersized","last_fresh":"2024-03-11T21:00:02.443168+0000","last_change":"2024-03-11T20:58:53.585355+0000","last_active":"2024-03-11T21:00:02.443168+0000","last_peered":"2024-03-11T21:00:02.443168+0000","last_clean":"2024-03-11T20:50:33.234063+0000","last_became_active":"2024-03-11T20:58:53.585355+0000","last_became_peered":"2024-03-11T20:58:53.585355+0000","last_unstale":"2024-03-11T21:00:02.443168+0000","last_undegraded":"2024-03-11T21:00:02.443168+0000","last_fullsized":"2024-03-11T20:58:53.583067+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:57:19.063710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:57:19.063710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":34,"reported_epoch":196,"state":"active+undersized","last_fresh":"2024-03-11T21:00:02.444226+0000","last_change":"2024-03-11T20:58:53.593600+0000","last_active":"2024-03-11T21:00:02.444226+0000","last_peered":"2024-03-11T21:00:02.444226+0000","last_clean":"2024-03-11T20:50:31.136392+0000","last_became_active":"2024-03-11T20:58:53.593600+0000","last_became_peered":"2024-03-11T20:58:53.593600+0000","last_unstale":"2024-03-11T21:00:02.444226+0000","last_undegraded":"2024-03-11T21:00:02.444226+0000","last_fullsized":"2024-03-11T20:58:53.581805+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:57:33.143849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:57:33.143849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"77'1","reported_seq":164,"reported_epoch":196,"state":"undersized+degraded+peered","last_fresh":"2024-03-11T21:00:02.711824+0000","last_change":"2024-03-11T20:58:53.590680+0000","last_active":"2024-03-11T20:58:52.591515+0000","last_peered":"2024-03-11T21:00:02.711824+0000","last_clean":"2024-03-11T20:57:12.899948+0000","last_became_active":"2024-03-11T20:57:59.748735+0000","last_became_peered":"2024-03-11T20:58:53.590680+0000","last_unstale":"2024-03-11T21:00:02.711824+0000","last_undegraded":"2024-03-11T20:58:53.588240+0000","last_fullsized":"2024-03-11T20:58:53.588140+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-03-11T20:56:53.910483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:56:53.910483+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:00:06.663052+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[{"shards":"6","objects":1}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.7","version":"84'13","reported_seq":307,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-11T21:00:02.448165+0000","last_change":"2024-03-11T20:58:55.540966+0000","last_active":"2024-03-11T21:00:02.448165+0000","last_peered":"2024-03-11T21:00:02.448165+0000","last_clean":"2024-03-11T21:00:02.448165+0000","last_became_active":"2024-03-11T20:50:30.097669+0000","last_became_peered":"2024-03-11T20:50:30.097669+0000","last_unstale":"2024-03-11T21:00:02.448165+0000","last_undegraded":"2024-03-11T21:00:02.448165+0000","last_fullsized":"2024-03-11T21:00:02.448165+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'13","last_scrub_stamp":"2024-03-11T20:58:55.540851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:58:55.540851+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:00:17.241120+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"84'11","reported_seq":410,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-11T21:00:02.443783+0000","last_change":"2024-03-11T20:59:05.483698+0000","last_active":"2024-03-11T21:00:02.443783+0000","last_peered":"2024-03-11T21:00:02.443783+0000","last_clean":"2024-03-11T21:00:02.443783+0000","last_became_active":"2024-03-11T20:50:32.647041+0000","last_became_peered":"2024-03-11T20:50:32.647041+0000","last_unstale":"2024-03-11T21:00:02.443783+0000","last_undegraded":"2024-03-11T21:00:02.443783+0000","last_fullsized":"2024-03-11T21:00:02.443783+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T20:59:05.483591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:59:05.483591+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:00:15.750214+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"193'18","reported_seq":42,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-11T21:00:02.438954+0000","last_change":"2024-03-11T20:59:11.277342+0000","last_active":"2024-03-11T21:00:02.438954+0000","last_peered":"2024-03-11T21:00:02.438954+0000","last_clean":"2024-03-11T21:00:02.438954+0000","last_became_active":"2024-03-11T20:59:11.276385+0000","last_became_peered":"2024-03-11T20:59:11.276385+0000","last_unstale":"2024-03-11T21:00:02.438954+0000","last_undegraded":"2024-03-11T21:00:02.438954+0000","last_fullsized":"2024-03-11T21:00:02.438954+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:59:10.160042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:59:10.160042+0000","last_clean_scrub_stamp":"2024-03-11T20:59:10.160042+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:00:20.095188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1179648,"num_objects":18,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18,"num_write_kb":1152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":281,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-11T21:00:02.808376+0000","last_change":"2024-03-11T20:59:00.468582+0000","last_active":"2024-03-11T21:00:02.808376+0000","last_peered":"2024-03-11T21:00:02.808376+0000","last_clean":"2024-03-11T21:00:02.808376+0000","last_became_active":"2024-03-11T20:50:34.326947+0000","last_became_peered":"2024-03-11T20:50:34.326947+0000","last_unstale":"2024-03-11T21:00:02.808376+0000","last_undegraded":"2024-03-11T21:00:02.808376+0000","last_fullsized":"2024-03-11T21:00:02.808376+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:59:00.468313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:59:00.468313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:00:25.985159+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"120'6","reported_seq":781,"reported_epoch":196,"state":"undersized+degraded+peered","last_fresh":"2024-03-11T21:00:02.443929+0000","last_change":"2024-03-11T20:58:53.595424+0000","last_active":"2024-03-11T20:58:52.574858+0000","last_peered":"2024-03-11T21:00:02.443929+0000","last_clean":"2024-03-11T20:57:58.880549+0000","last_became_active":"2024-03-11T20:57:59.735839+0000","last_became_peered":"2024-03-11T20:58:53.595424+0000","last_unstale":"2024-03-11T21:00:02.443929+0000","last_undegraded":"2024-03-11T20:58:53.589473+0000","last_fullsized":"2024-03-11T20:58:53.589249+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"120'6","last_scrub_stamp":"2024-03-11T20:54:26.468057+0000","last_deep_scrub":"120'6","last_deep_scrub_stamp":"2024-03-11T20:54:26.468057+0000","last_clean_scrub_stamp":"2024-03-11T20:54:26.468057+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0035533370000000002,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":466,"num_read_kb":461,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":292,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-11T21:00:02.711784+0000","last_change":"2024-03-11T20:58:52.829793+0000","last_active":"2024-03-11T21:00:02.711784+0000","last_peered":"2024-03-11T21:00:02.711784+0000","last_clean":"2024-03-11T21:00:02.711784+0000","last_became_active":"2024-03-11T20:50:28.071707+0000","last_became_peered":"2024-03-11T20:50:28.071707+0000","last_unstale":"2024-03-11T21:00:02.711784+0000","last_undegraded":"2024-03-11T21:00:02.711784+0000","last_fullsized":"2024-03-11T21:00:02.711784+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:58:52.829571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:58:52.829571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:00:10.323632+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"84'12","reported_seq":343,"reported_epoch":196,"state":"undersized+degraded+peered","last_fresh":"2024-03-11T21:00:02.756184+0000","last_change":"2024-03-11T20:58:53.596563+0000","last_active":"2024-03-11T20:58:52.586086+0000","last_peered":"2024-03-11T21:00:02.756184+0000","last_clean":"2024-03-11T20:57:55.792678+0000","last_became_active":"2024-03-11T20:57:59.727498+0000","last_became_peered":"2024-03-11T20:58:53.596563+0000","last_unstale":"2024-03-11T21:00:02.756184+0000","last_undegraded":"2024-03-11T20:58:53.595805+0000","last_fullsized":"2024-03-11T20:58:53.595718+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'12","last_scrub_stamp":"2024-03-11T20:57:09.048483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:57:09.048483+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0","objects":6}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":34,"reported_epoch":196,"state":"active+undersized","last_fresh":"2024-03-11T21:00:02.438532+0000","last_change":"2024-03-11T20:58:53.589857+0000","last_active":"2024-03-11T21:00:02.438532+0000","last_peered":"2024-03-11T21:00:02.438532+0000","last_clean":"2024-03-11T20:50:31.136392+0000","last_became_active":"2024-03-11T20:58:53.589857+0000","last_became_peered":"2024-03-11T20:58:53.589857+0000","last_unstale":"2024-03-11T21:00:02.438532+0000","last_undegraded":"2024-03-11T21:00:02.438532+0000","last_fullsized":"2024-03-11T20:58:53.579984+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:58:37.170245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:58:37.170245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.2","version":"193'15","reported_seq":36,"reported_epoch":196,"state":"active+undersized+degraded","last_fresh":"2024-03-11T21:00:02.438939+0000","last_change":"2024-03-11T20:59:30.434730+0000","last_active":"2024-03-11T21:00:02.438939+0000","last_peered":"2024-03-11T21:00:02.438939+0000","last_clean":"2024-03-11T20:59:10.160042+0000","last_became_active":"2024-03-11T20:59:11.191338+0000","last_became_peered":"2024-03-11T20:59:11.191338+0000","last_unstale":"2024-03-11T21:00:02.438939+0000","last_undegraded":"2024-03-11T20:59:30.421107+0000","last_fullsized":"2024-03-11T20:59:11.185500+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:59:10.160042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:59:10.160042+0000","last_clean_scrub_stamp":"2024-03-11T20:59:10.160042+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:00:24.625001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":983040,"num_objects":15,"num_object_clones":0,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":["5","0"],"object_location_counts":[{"shards":"0,5","objects":15}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":272,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-11T21:00:02.444337+0000","last_change":"2024-03-11T20:58:51.546704+0000","last_active":"2024-03-11T21:00:02.444337+0000","last_peered":"2024-03-11T21:00:02.444337+0000","last_clean":"2024-03-11T21:00:02.444337+0000","last_became_active":"2024-03-11T20:50:34.335370+0000","last_became_peered":"2024-03-11T20:50:34.335370+0000","last_unstale":"2024-03-11T21:00:02.444337+0000","last_undegraded":"2024-03-11T21:00:02.444337+0000","last_fullsized":"2024-03-11T21:00:02.444337+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:58:51.546414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:58:51.546414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:00:17.852826+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"175'14","reported_seq":58,"reported_epoch":196,"state":"undersized+degraded+peered","last_fresh":"2024-03-11T21:00:02.711575+0000","last_change":"2024-03-11T20:58:53.590530+0000","last_active":"2024-03-11T20:58:52.590839+0000","last_peered":"2024-03-11T21:00:02.711575+0000","last_clean":"2024-03-11T20:50:35.972400+0000","last_became_active":"2024-03-11T20:50:32.197412+0000","last_became_peered":"2024-03-11T20:58:53.590530+0000","last_unstale":"2024-03-11T21:00:02.711575+0000","last_undegraded":"2024-03-11T20:58:53.588139+0000","last_fullsized":"2024-03-11T20:58:53.588047+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"125'12","last_scrub_stamp":"2024-03-11T20:57:02.018732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:57:02.018732+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:00:06.663094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[{"shards":"6","objects":1}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":292,"reported_epoch":196,"state":"active+undersized","last_fresh":"2024-03-11T21:00:02.808094+0000","last_change":"2024-03-11T20:57:59.737501+0000","last_active":"2024-03-11T21:00:02.808094+0000","last_peered":"2024-03-11T21:00:02.808094+0000","last_clean":"2024-03-11T20:57:55.788556+0000","last_became_active":"2024-03-11T20:57:59.737501+0000","last_became_peered":"2024-03-11T20:57:59.737501+0000","last_unstale":"2024-03-11T21:00:02.808094+0000","last_undegraded":"2024-03-11T21:00:02.808094+0000","last_fullsized":"2024-03-11T20:57:59.726522+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:55:38.171010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:52:09.043655+0000","last_clean_scrub_stamp":"2024-03-11T20:55:38.171010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"84'12","reported_seq":74,"reported_epoch":196,"state":"active+undersized+degraded","last_fresh":"2024-03-11T21:00:02.759141+0000","last_change":"2024-03-11T20:58:53.588329+0000","last_active":"2024-03-11T21:00:02.759141+0000","last_peered":"2024-03-11T21:00:02.759141+0000","last_clean":"2024-03-11T20:50:35.601398+0000","last_became_active":"2024-03-11T20:58:53.588329+0000","last_became_peered":"2024-03-11T20:58:53.588329+0000","last_unstale":"2024-03-11T21:00:02.759141+0000","last_undegraded":"2024-03-11T20:58:53.578137+0000","last_fullsized":"2024-03-11T20:58:53.577779+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'12","last_scrub_stamp":"2024-03-11T20:57:24.024992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:57:24.024992+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:00:07.260302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":15,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[{"shards":"2,7","objects":5}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.3","version":"193'7","reported_seq":28,"reported_epoch":196,"state":"active+undersized+degraded","last_fresh":"2024-03-11T21:00:02.438616+0000","last_change":"2024-03-11T20:59:30.430086+0000","last_active":"2024-03-11T21:00:02.438616+0000","last_peered":"2024-03-11T21:00:02.438616+0000","last_clean":"2024-03-11T20:59:10.160042+0000","last_became_active":"2024-03-11T20:59:11.200224+0000","last_became_peered":"2024-03-11T20:59:11.200224+0000","last_unstale":"2024-03-11T21:00:02.438616+0000","last_undegraded":"2024-03-11T20:59:30.428043+0000","last_fullsized":"2024-03-11T20:59:11.185519+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:59:10.160042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:59:10.160042+0000","last_clean_scrub_stamp":"2024-03-11T20:59:10.160042+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:00:15.420584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":458752,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[{"shards":"5,6","objects":7}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":288,"reported_epoch":196,"state":"active+undersized","last_fresh":"2024-03-11T21:00:02.440302+0000","last_change":"2024-03-11T20:57:59.737574+0000","last_active":"2024-03-11T21:00:02.440302+0000","last_peered":"2024-03-11T21:00:02.440302+0000","last_clean":"2024-03-11T20:57:55.794722+0000","last_became_active":"2024-03-11T20:57:59.737574+0000","last_became_peered":"2024-03-11T20:57:59.737574+0000","last_unstale":"2024-03-11T21:00:02.440302+0000","last_undegraded":"2024-03-11T21:00:02.440302+0000","last_fullsized":"2024-03-11T20:57:59.718244+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:57:16.530664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:57:16.530664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":["5","0"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":138,"reported_epoch":196,"state":"undersized+peered","last_fresh":"2024-03-11T21:00:02.448172+0000","last_change":"2024-03-11T20:58:53.599698+0000","last_active":"2024-03-11T20:58:52.577935+0000","last_peered":"2024-03-11T21:00:02.448172+0000","last_clean":"2024-03-11T20:57:12.896690+0000","last_became_active":"2024-03-11T20:57:59.736138+0000","last_became_peered":"2024-03-11T20:58:53.599698+0000","last_unstale":"2024-03-11T21:00:02.448172+0000","last_undegraded":"2024-03-11T21:00:02.448172+0000","last_fullsized":"2024-03-11T20:58:53.598441+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:54:24.740149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:54:24.740149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":143,"reported_epoch":196,"state":"active+undersized","last_fresh":"2024-03-11T21:00:02.807645+0000","last_change":"2024-03-11T20:57:59.733273+0000","last_active":"2024-03-11T21:00:02.807645+0000","last_peered":"2024-03-11T21:00:02.807645+0000","last_clean":"2024-03-11T20:50:26.989698+0000","last_became_active":"2024-03-11T20:57:59.733273+0000","last_became_peered":"2024-03-11T20:57:59.733273+0000","last_unstale":"2024-03-11T21:00:02.807645+0000","last_undegraded":"2024-03-11T21:00:02.807645+0000","last_fullsized":"2024-03-11T20:57:59.721747+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:57:05.126829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:40.974147+0000","last_clean_scrub_stamp":"2024-03-11T20:57:05.126829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"116'22","reported_seq":351,"reported_epoch":196,"state":"active+undersized+degraded","last_fresh":"2024-03-11T21:00:02.443010+0000","last_change":"2024-03-11T20:57:59.738597+0000","last_active":"2024-03-11T21:00:02.443010+0000","last_peered":"2024-03-11T21:00:02.443010+0000","last_clean":"2024-03-11T20:57:55.787481+0000","last_became_active":"2024-03-11T20:57:59.738597+0000","last_became_peered":"2024-03-11T20:57:59.738597+0000","last_unstale":"2024-03-11T21:00:02.443010+0000","last_undegraded":"2024-03-11T20:57:59.714036+0000","last_fullsized":"2024-03-11T20:57:59.713914+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'22","last_scrub_stamp":"2024-03-11T20:56:05.941902+0000","last_deep_scrub":"116'22","last_deep_scrub_stamp":"2024-03-11T20:53:47.364709+0000","last_clean_scrub_stamp":"2024-03-11T20:56:05.941902+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"3,7","objects":7}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"193'16","reported_seq":37,"reported_epoch":196,"state":"active+undersized+degraded","last_fresh":"2024-03-11T21:00:02.441722+0000","last_change":"2024-03-11T20:59:30.424616+0000","last_active":"2024-03-11T21:00:02.441722+0000","last_peered":"2024-03-11T21:00:02.441722+0000","last_clean":"2024-03-11T20:59:10.160042+0000","last_became_active":"2024-03-11T20:59:11.200127+0000","last_became_peered":"2024-03-11T20:59:11.200127+0000","last_unstale":"2024-03-11T21:00:02.441722+0000","last_undegraded":"2024-03-11T20:59:30.418324+0000","last_fullsized":"2024-03-11T20:59:11.192059+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:59:10.160042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:59:10.160042+0000","last_clean_scrub_stamp":"2024-03-11T20:59:10.160042+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:00:28.517716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1048576,"num_objects":16,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":1024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":["3","5"],"object_location_counts":[{"shards":"3,5","objects":16}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":262,"reported_epoch":196,"state":"undersized+peered","last_fresh":"2024-03-11T21:00:02.762835+0000","last_change":"2024-03-11T20:58:53.581282+0000","last_active":"2024-03-11T20:57:59.073089+0000","last_peered":"2024-03-11T21:00:02.762835+0000","last_clean":"2024-03-11T20:57:55.802440+0000","last_became_active":"2024-03-11T20:57:24.644848+0000","last_became_peered":"2024-03-11T20:58:53.581282+0000","last_unstale":"2024-03-11T21:00:02.762835+0000","last_undegraded":"2024-03-11T21:00:02.762835+0000","last_fullsized":"2024-03-11T20:58:53.580102+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:56:58.949812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:56:58.949812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:00:07.260150+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":142,"reported_epoch":196,"state":"active+undersized","last_fresh":"2024-03-11T21:00:02.442307+0000","last_change":"2024-03-11T20:57:59.746584+0000","last_active":"2024-03-11T21:00:02.442307+0000","last_peered":"2024-03-11T21:00:02.442307+0000","last_clean":"2024-03-11T20:57:12.904804+0000","last_became_active":"2024-03-11T20:57:59.746584+0000","last_became_peered":"2024-03-11T20:57:59.746584+0000","last_unstale":"2024-03-11T21:00:02.442307+0000","last_undegraded":"2024-03-11T21:00:02.442307+0000","last_fullsized":"2024-03-11T20:57:59.715329+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:55:52.917902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:55:52.917902+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"84'9","reported_seq":299,"reported_epoch":196,"state":"active+undersized+degraded","last_fresh":"2024-03-11T21:00:02.444635+0000","last_change":"2024-03-11T20:58:53.597489+0000","last_active":"2024-03-11T21:00:02.444635+0000","last_peered":"2024-03-11T21:00:02.444635+0000","last_clean":"2024-03-11T20:58:51.564262+0000","last_became_active":"2024-03-11T20:58:53.597489+0000","last_became_peered":"2024-03-11T20:58:53.597489+0000","last_unstale":"2024-03-11T21:00:02.444635+0000","last_undegraded":"2024-03-11T20:58:53.591121+0000","last_fullsized":"2024-03-11T20:58:53.590940+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'9","last_scrub_stamp":"2024-03-11T20:58:41.483078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:58:41.483078+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:00:03.678974+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":4}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":34,"reported_epoch":196,"state":"active+undersized","last_fresh":"2024-03-11T21:00:02.444961+0000","last_change":"2024-03-11T20:58:53.587230+0000","last_active":"2024-03-11T21:00:02.444961+0000","last_peered":"2024-03-11T21:00:02.444961+0000","last_clean":"2024-03-11T20:50:31.136392+0000","last_became_active":"2024-03-11T20:58:53.587230+0000","last_became_peered":"2024-03-11T20:58:53.587230+0000","last_unstale":"2024-03-11T21:00:02.444961+0000","last_undegraded":"2024-03-11T21:00:02.444961+0000","last_fullsized":"2024-03-11T20:58:53.584283+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:58:39.167639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:35.670149+0000","last_clean_scrub_stamp":"2024-03-11T20:58:39.167639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:00:02.484932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1","version":"193'13","reported_seq":34,"reported_epoch":196,"state":"active+undersized+degraded","last_fresh":"2024-03-11T21:00:02.759127+0000","last_change":"2024-03-11T20:59:30.433034+0000","last_active":"2024-03-11T21:00:02.759127+0000","last_peered":"2024-03-11T21:00:02.759127+0000","last_clean":"2024-03-11T20:59:10.160042+0000","last_became_active":"2024-03-11T20:59:11.204005+0000","last_became_peered":"2024-03-11T20:59:11.204005+0000","last_unstale":"2024-03-11T21:00:02.759127+0000","last_undegraded":"2024-03-11T20:59:30.412505+0000","last_fullsized":"2024-03-11T20:59:11.197631+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:59:10.160042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:59:10.160042+0000","last_clean_scrub_stamp":"2024-03-11T20:59:10.160042+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:00:26.442703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":851968,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":13}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":290,"reported_epoch":196,"state":"active+undersized","last_fresh":"2024-03-11T21:00:02.762844+0000","last_change":"2024-03-11T20:58:53.591621+0000","last_active":"2024-03-11T21:00:02.762844+0000","last_peered":"2024-03-11T21:00:02.762844+0000","last_clean":"2024-03-11T20:58:51.851842+0000","last_became_active":"2024-03-11T20:58:53.591621+0000","last_became_peered":"2024-03-11T20:58:53.591621+0000","last_unstale":"2024-03-11T21:00:02.762844+0000","last_undegraded":"2024-03-11T21:00:02.762844+0000","last_fullsized":"2024-03-11T20:58:53.580673+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:58:39.864138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:58:39.864138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:00:03.266285+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"175'16","reported_seq":291,"reported_epoch":196,"state":"undersized+degraded+peered","last_fresh":"2024-03-11T21:00:02.761068+0000","last_change":"2024-03-11T20:58:53.581738+0000","last_active":"2024-03-11T20:58:52.582520+0000","last_peered":"2024-03-11T21:00:02.761068+0000","last_clean":"2024-03-11T20:56:00.239268+0000","last_became_active":"2024-03-11T20:54:37.396038+0000","last_became_peered":"2024-03-11T20:58:53.581738+0000","last_unstale":"2024-03-11T21:00:02.761068+0000","last_undegraded":"2024-03-11T20:58:53.580088+0000","last_fullsized":"2024-03-11T20:58:53.579848+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"146'14","last_scrub_stamp":"2024-03-11T20:56:00.242211+0000","last_deep_scrub":"146'14","last_deep_scrub_stamp":"2024-03-11T20:56:00.242211+0000","last_clean_scrub_stamp":"2024-03-11T20:56:00.242211+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:00:07.260101+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":288,"reported_epoch":196,"state":"active+undersized","last_fresh":"2024-03-11T21:00:02.808435+0000","last_change":"2024-03-11T20:58:53.590409+0000","last_active":"2024-03-11T21:00:02.808435+0000","last_peered":"2024-03-11T21:00:02.808435+0000","last_clean":"2024-03-11T20:58:51.924247+0000","last_became_active":"2024-03-11T20:58:53.590409+0000","last_became_peered":"2024-03-11T20:58:53.590409+0000","last_unstale":"2024-03-11T21:00:02.808435+0000","last_undegraded":"2024-03-11T21:00:02.808435+0000","last_fullsized":"2024-03-11T20:58:53.582516+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:56:07.001340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:52:51.161281+0000","last_clean_scrub_stamp":"2024-03-11T20:56:07.001340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"161'28","reported_seq":377,"reported_epoch":196,"state":"undersized+degraded+peered","last_fresh":"2024-03-11T21:00:02.711122+0000","last_change":"2024-03-11T20:58:53.587708+0000","last_active":"2024-03-11T20:58:52.589917+0000","last_peered":"2024-03-11T21:00:02.711122+0000","last_clean":"2024-03-11T20:57:55.796044+0000","last_became_active":"2024-03-11T20:57:59.752335+0000","last_became_peered":"2024-03-11T20:58:53.587708+0000","last_unstale":"2024-03-11T21:00:02.711122+0000","last_undegraded":"2024-03-11T20:58:53.587097+0000","last_fullsized":"2024-03-11T20:58:53.587005+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"161'28","last_scrub_stamp":"2024-03-11T20:57:20.816278+0000","last_deep_scrub":"161'28","last_deep_scrub_stamp":"2024-03-11T20:57:20.816278+0000","last_clean_scrub_stamp":"2024-03-11T20:57:20.816278+0000","objects_scrubbed":9,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:00:06.663108+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[{"shards":"6","objects":9}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.6","version":"193'13","reported_seq":37,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-11T21:00:02.711419+0000","last_change":"2024-03-11T20:59:11.272863+0000","last_active":"2024-03-11T21:00:02.711419+0000","last_peered":"2024-03-11T21:00:02.711419+0000","last_clean":"2024-03-11T21:00:02.711419+0000","last_became_active":"2024-03-11T20:59:11.271837+0000","last_became_peered":"2024-03-11T20:59:11.271837+0000","last_unstale":"2024-03-11T21:00:02.711419+0000","last_undegraded":"2024-03-11T21:00:02.711419+0000","last_fullsized":"2024-03-11T21:00:02.711419+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:59:10.160042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:59:10.160042+0000","last_clean_scrub_stamp":"2024-03-11T20:59:10.160042+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:00:26.193809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":851968,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,7],"acting":[6,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":273,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-11T21:00:02.807606+0000","last_change":"2024-03-11T20:58:52.349838+0000","last_active":"2024-03-11T21:00:02.807606+0000","last_peered":"2024-03-11T21:00:02.807606+0000","last_clean":"2024-03-11T21:00:02.807606+0000","last_became_active":"2024-03-11T20:50:34.327139+0000","last_became_peered":"2024-03-11T20:50:34.327139+0000","last_unstale":"2024-03-11T21:00:02.807606+0000","last_undegraded":"2024-03-11T21:00:02.807606+0000","last_fullsized":"2024-03-11T21:00:02.807606+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:58:52.349711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T20:58:52.349711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:00:08.312325+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":307,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-11T21:00:02.760916+0000","last_change":"2024-03-11T20:59:56.262920+0000","last_active":"2024-03-11T21:00:02.760916+0000","last_peered":"2024-03-11T21:00:02.760916+0000","last_clean":"2024-03-11T21:00:02.760916+0000","last_became_active":"2024-03-11T20:50:28.049901+0000","last_became_peered":"2024-03-11T20:50:28.049901+0000","last_unstale":"2024-03-11T21:00:02.760916+0000","last_undegraded":"2024-03-11T21:00:02.760916+0000","last_fullsized":"2024-03-11T21:00:02.760916+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:59:56.262748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:57:17.447191+0000","last_clean_scrub_stamp":"2024-03-11T20:59:56.262748+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:01:17.960088+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"102'25","reported_seq":325,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-11T21:00:02.810355+0000","last_change":"2024-03-11T20:58:54.381401+0000","last_active":"2024-03-11T21:00:02.810355+0000","last_peered":"2024-03-11T21:00:02.810355+0000","last_clean":"2024-03-11T21:00:02.810355+0000","last_became_active":"2024-03-11T20:50:30.111819+0000","last_became_peered":"2024-03-11T20:50:30.111819+0000","last_unstale":"2024-03-11T21:00:02.810355+0000","last_undegraded":"2024-03-11T21:00:02.810355+0000","last_fullsized":"2024-03-11T21:00:02.810355+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"102'25","last_scrub_stamp":"2024-03-11T20:58:54.381253+0000","last_deep_scrub":"102'25","last_deep_scrub_stamp":"2024-03-11T20:51:35.187094+0000","last_clean_scrub_stamp":"2024-03-11T20:58:54.381253+0000","objects_scrubbed":8,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:00:22.595449+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":291,"reported_epoch":196,"state":"active+undersized","last_fresh":"2024-03-11T21:00:02.710763+0000","last_change":"2024-03-11T20:57:59.746890+0000","last_active":"2024-03-11T21:00:02.710763+0000","last_peered":"2024-03-11T21:00:02.710763+0000","last_clean":"2024-03-11T20:57:55.795942+0000","last_became_active":"2024-03-11T20:57:59.746890+0000","last_became_peered":"2024-03-11T20:57:59.746890+0000","last_unstale":"2024-03-11T21:00:02.710763+0000","last_undegraded":"2024-03-11T21:00:02.710763+0000","last_fullsized":"2024-03-11T20:57:59.727745+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:57:34.873790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T20:57:34.873790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:00:06.663132+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6","7"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.7","version":"193'11","reported_seq":32,"reported_epoch":196,"state":"active+undersized+degraded","last_fresh":"2024-03-11T21:00:02.447642+0000","last_change":"2024-03-11T20:59:30.432658+0000","last_active":"2024-03-11T21:00:02.447642+0000","last_peered":"2024-03-11T21:00:02.447642+0000","last_clean":"2024-03-11T20:59:10.160042+0000","last_became_active":"2024-03-11T20:59:11.199437+0000","last_became_peered":"2024-03-11T20:59:11.199437+0000","last_unstale":"2024-03-11T21:00:02.447642+0000","last_undegraded":"2024-03-11T20:59:30.424153+0000","last_fullsized":"2024-03-11T20:59:11.191355+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:59:10.160042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:59:10.160042+0000","last_clean_scrub_stamp":"2024-03-11T20:59:10.160042+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:00:20.728305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":720896,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11,"num_write_kb":704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[{"shards":"3,6","objects":11}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":93,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-11T21:00:02.808519+0000","last_change":"2024-03-11T20:58:53.387954+0000","last_active":"2024-03-11T21:00:02.808519+0000","last_peered":"2024-03-11T21:00:02.808519+0000","last_clean":"2024-03-11T21:00:02.808519+0000","last_became_active":"2024-03-11T20:56:59.801112+0000","last_became_peered":"2024-03-11T20:56:59.801112+0000","last_unstale":"2024-03-11T21:00:02.808519+0000","last_undegraded":"2024-03-11T21:00:02.808519+0000","last_fullsized":"2024-03-11T21:00:02.808519+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:58:53.387734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:55:58.888890+0000","last_clean_scrub_stamp":"2024-03-11T20:58:53.387734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:00:06.670032+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":126,"reported_epoch":196,"state":"undersized+peered","last_fresh":"2024-03-11T21:00:02.446116+0000","last_change":"2024-03-11T20:58:53.598707+0000","last_active":"2024-03-11T20:50:26.989698+0000","last_peered":"2024-03-11T21:00:02.446116+0000","last_clean":"2024-03-11T20:50:26.989698+0000","last_became_active":"0.000000","last_became_peered":"2024-03-11T20:58:53.598707+0000","last_unstale":"2024-03-11T21:00:02.446116+0000","last_undegraded":"2024-03-11T21:00:02.446116+0000","last_fullsized":"2024-03-11T20:58:53.597572+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:56:56.975636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:56:56.975636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"84'10","reported_seq":296,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-11T21:00:02.439434+0000","last_change":"2024-03-11T20:58:37.077086+0000","last_active":"2024-03-11T21:00:02.439434+0000","last_peered":"2024-03-11T21:00:02.439434+0000","last_clean":"2024-03-11T21:00:02.439434+0000","last_became_active":"2024-03-11T20:50:30.109549+0000","last_became_peered":"2024-03-11T20:50:30.109549+0000","last_unstale":"2024-03-11T21:00:02.439434+0000","last_undegraded":"2024-03-11T21:00:02.439434+0000","last_fullsized":"2024-03-11T21:00:02.439434+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'10","last_scrub_stamp":"2024-03-11T20:58:37.076670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:58:37.076670+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":301,"reported_epoch":196,"state":"undersized+peered","last_fresh":"2024-03-11T21:00:02.756303+0000","last_change":"2024-03-11T20:58:53.594203+0000","last_active":"2024-03-11T20:58:52.585150+0000","last_peered":"2024-03-11T21:00:02.756303+0000","last_clean":"2024-03-11T20:57:55.792410+0000","last_became_active":"2024-03-11T20:57:59.720016+0000","last_became_peered":"2024-03-11T20:58:53.594203+0000","last_unstale":"2024-03-11T21:00:02.756303+0000","last_undegraded":"2024-03-11T21:00:02.756303+0000","last_fullsized":"2024-03-11T20:58:53.590551+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:57:19.959837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:52:51.931295+0000","last_clean_scrub_stamp":"2024-03-11T20:57:19.959837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0],"avail_no_missing":["0"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"193'9","reported_seq":30,"reported_epoch":196,"state":"active+undersized+degraded","last_fresh":"2024-03-11T21:00:02.438767+0000","last_change":"2024-03-11T20:59:30.431356+0000","last_active":"2024-03-11T21:00:02.438767+0000","last_peered":"2024-03-11T21:00:02.438767+0000","last_clean":"2024-03-11T20:59:10.160042+0000","last_became_active":"2024-03-11T20:59:11.202617+0000","last_became_peered":"2024-03-11T20:59:11.202617+0000","last_unstale":"2024-03-11T21:00:02.438767+0000","last_undegraded":"2024-03-11T20:59:30.425345+0000","last_fullsized":"2024-03-11T20:59:11.185582+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:59:10.160042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:59:10.160042+0000","last_clean_scrub_stamp":"2024-03-11T20:59:10.160042+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:00:11.430213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":589824,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[{"shards":"2,5","objects":9}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":281,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-11T21:00:02.439567+0000","last_change":"2024-03-11T20:58:59.981867+0000","last_active":"2024-03-11T21:00:02.439567+0000","last_peered":"2024-03-11T21:00:02.439567+0000","last_clean":"2024-03-11T21:00:02.439567+0000","last_became_active":"2024-03-11T20:50:34.337173+0000","last_became_peered":"2024-03-11T20:50:34.337173+0000","last_unstale":"2024-03-11T21:00:02.439567+0000","last_undegraded":"2024-03-11T21:00:02.439567+0000","last_fullsized":"2024-03-11T21:00:02.439567+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:58:59.981770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:53:03.762882+0000","last_clean_scrub_stamp":"2024-03-11T20:58:59.981770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:00:03.690834+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":299,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-11T21:00:02.762425+0000","last_change":"2024-03-11T21:00:02.264421+0000","last_active":"2024-03-11T21:00:02.762425+0000","last_peered":"2024-03-11T21:00:02.762425+0000","last_clean":"2024-03-11T21:00:02.762425+0000","last_became_active":"2024-03-11T20:50:28.058942+0000","last_became_peered":"2024-03-11T20:50:28.058942+0000","last_unstale":"2024-03-11T21:00:02.762425+0000","last_undegraded":"2024-03-11T21:00:02.762425+0000","last_fullsized":"2024-03-11T21:00:02.762425+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:00:02.264195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:00:02.264195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:01:11.312052+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"84'15","reported_seq":309,"reported_epoch":196,"state":"active+undersized+degraded","last_fresh":"2024-03-11T21:00:02.811277+0000","last_change":"2024-03-11T20:58:53.599333+0000","last_active":"2024-03-11T21:00:02.811277+0000","last_peered":"2024-03-11T21:00:02.811277+0000","last_clean":"2024-03-11T20:58:51.932635+0000","last_became_active":"2024-03-11T20:58:53.599333+0000","last_became_peered":"2024-03-11T20:58:53.599333+0000","last_unstale":"2024-03-11T21:00:02.811277+0000","last_undegraded":"2024-03-11T20:58:53.586222+0000","last_fullsized":"2024-03-11T20:58:53.586097+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15","last_scrub_stamp":"2024-03-11T20:57:29.272947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T20:57:29.272947+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[{"shards":"0,7","objects":7}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":277,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-11T21:00:02.438537+0000","last_change":"2024-03-11T20:59:00.946310+0000","last_active":"2024-03-11T21:00:02.438537+0000","last_peered":"2024-03-11T21:00:02.438537+0000","last_clean":"2024-03-11T21:00:02.438537+0000","last_became_active":"2024-03-11T20:50:32.652854+0000","last_became_peered":"2024-03-11T20:50:32.652854+0000","last_unstale":"2024-03-11T21:00:02.438537+0000","last_undegraded":"2024-03-11T21:00:02.438537+0000","last_fullsized":"2024-03-11T21:00:02.438537+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:59:00.946094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:59:00.946094+0000","last_clean_scrub_stamp":"2024-03-11T20:59:00.946094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:00:08.449497+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":12845056,"num_objects":196,"num_object_clones":0,"num_object_copies":588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":122,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":196,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":196,"num_write_kb":12544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":30539776,"data_stored":30539776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":196,"ondisk_log_size":196,"up":36,"acting":36,"num_store_stats":6},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":20,"num_read_kb":20,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1868,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":75,"acting":75,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":166,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":117,"num_bytes_recovered":2228,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":477,"ondisk_log_size":477,"up":68,"acting":68,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":62,"acting":62,"num_store_stats":8},{"poolid":5,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":466,"num_read_kb":461,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":6,"acting":6,"num_store_stats":7},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":74,"acting":74,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":68,"seq":292057776254,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":153324,"kb_used_data":5536,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":93591828,"statfs":{"total":95995035648,"available":95838031872,"internally_reserved":0,"allocated":5668864,"data_stored":5338789,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":151321034},"hb_peers":[0,2,3,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":57,"seq":244813136008,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":148828,"kb_used_data":5200,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":93596324,"statfs":{"total":95995035648,"available":95842635776,"internally_reserved":0,"allocated":5324800,"data_stored":5011119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":147061191},"hb_peers":[0,2,3,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575009,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":524492,"kb_used_data":435000,"kb_used_omap":1,"kb_used_meta":89470,"kb_avail":93220660,"statfs":{"total":95995035648,"available":95457955840,"internally_reserved":0,"allocated":445440000,"data_stored":445130863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":91617733},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738566,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":150948,"kb_used_data":7320,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":93594204,"statfs":{"total":95995035648,"available":95840464896,"internally_reserved":0,"allocated":7495680,"data_stored":7172739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":147061189},"hb_peers":[2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378801,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":145424,"kb_used_data":5956,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":93599728,"statfs":{"total":95995035648,"available":95846121472,"internally_reserved":0,"allocated":6098944,"data_stored":5796101,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":142801349},"hb_peers":[0,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":28,"seq":120259084454,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":206676,"kb_used_data":4744,"kb_used_omap":1,"kb_used_meta":201918,"kb_avail":93538476,"statfs":{"total":95995035648,"available":95783399424,"internally_reserved":0,"allocated":4857856,"data_stored":4551293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":206764488},"hb_peers":[0,2,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":35,"seq":150323855502,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":108288,"kb_used_data":760,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93636864,"statfs":{"total":95995035648,"available":95884148736,"internally_reserved":0,"allocated":778240,"data_stored":456062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":44,"seq":188978561170,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":105300,"kb_used_data":7496,"kb_used_omap":1,"kb_used_meta":97790,"kb_avail":93639852,"statfs":{"total":95995035648,"available":95887208448,"internally_reserved":0,"allocated":7675904,"data_stored":7368887,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":100137413},"hb_peers":[0,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":6094848,"data_stored":6094848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":5308416,"data_stored":5308416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4063232,"data_stored":4063232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":6881280,"data_stored":6881280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":3932160,"data_stored":3932160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4259840,"data_stored":4259840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T21:00:12.246 INFO:tasks.cephadm.ceph_manager.ceph:would drop below min_pgs, proj 6, currently 16 2024-03-11T21:00:12.302 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:12.331 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:12.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:12.334 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:00:12.410 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 42 16 212 196 0.291631 0 - 0.00475516 2024-03-11T21:00:12.434 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:12.463 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:12.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:12.465 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:00:12.566 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:12.592 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:12.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:12.594 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:00:12.695 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:12.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:12 smithi146 bash[23809]: cluster 2024-03-11T21:00:10.710166+0000 osd.3 (osd.3) 228 : cluster 0 2.1b scrub starts 2024-03-11T21:00:12.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:12 smithi146 bash[23809]: cluster 2024-03-11T21:00:10.712801+0000 osd.3 (osd.3) 229 : cluster 0 2.1b scrub ok 2024-03-11T21:00:12.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:12 smithi146 bash[23809]: cluster 2024-03-11T21:00:11.294824+0000 osd.0 (osd.0) 169 : cluster 0 purged_snaps scrub starts 2024-03-11T21:00:12.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:12 smithi146 bash[23809]: cluster 2024-03-11T21:00:11.294912+0000 osd.0 (osd.0) 170 : cluster 0 purged_snaps scrub ok 2024-03-11T21:00:12.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:12 smithi146 bash[23809]: cluster 2024-03-11T21:00:11.542799+0000 mon.a (mon.0) 1264 : cluster 0 osdmap e200: 8 total, 6 up, 8 in 2024-03-11T21:00:12.707 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:12 smithi146 bash[23809]: cluster 2024-03-11T21:00:11.652471+0000 osd.6 (osd.6) 136 : cluster 0 2.7 scrub starts 2024-03-11T21:00:12.707 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:12 smithi146 bash[23809]: cluster 2024-03-11T21:00:11.657463+0000 osd.6 (osd.6) 137 : cluster 0 2.7 scrub ok 2024-03-11T21:00:12.707 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:12 smithi146 bash[23809]: cluster 2024-03-11T21:00:11.811562+0000 osd.7 (osd.7) 193 : cluster 0 1.0 scrub starts 2024-03-11T21:00:12.707 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:12 smithi146 bash[23809]: cluster 2024-03-11T21:00:11.815708+0000 osd.7 (osd.7) 194 : cluster 0 1.0 scrub ok 2024-03-11T21:00:12.707 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:12 smithi146 bash[23809]: cluster 2024-03-11T21:00:11.938525+0000 osd.6 (osd.6) 138 : cluster 0 purged_snaps scrub starts 2024-03-11T21:00:12.707 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:12 smithi146 bash[23809]: cluster 2024-03-11T21:00:11.938614+0000 osd.6 (osd.6) 139 : cluster 0 purged_snaps scrub ok 2024-03-11T21:00:12.707 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:12 smithi146 bash[23809]: cluster 2024-03-11T21:00:12.058033+0000 mgr.y (mgr.24380) 350 : cluster 0 pgmap v361: 148 pgs: 52 active+clean, 12 undersized+peered, 43 active+undersized, 12 undersized+degraded+peered, 29 active+undersized+degraded; 13 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 304/1182 objects degraded (25.719%) 2024-03-11T21:00:12.723 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:12.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:12.725 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:00:12.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:12 smithi023 bash[21372]: cluster 2024-03-11T21:00:10.710166+0000 osd.3 (osd.3) 228 : cluster 0 2.1b scrub starts 2024-03-11T21:00:12.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:12 smithi023 bash[21372]: cluster 2024-03-11T21:00:10.712801+0000 osd.3 (osd.3) 229 : cluster 0 2.1b scrub ok 2024-03-11T21:00:12.786 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:12 smithi023 bash[21372]: cluster 2024-03-11T21:00:11.294824+0000 osd.0 (osd.0) 169 : cluster 0 purged_snaps scrub starts 2024-03-11T21:00:12.786 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:12 smithi023 bash[21372]: cluster 2024-03-11T21:00:11.294912+0000 osd.0 (osd.0) 170 : cluster 0 purged_snaps scrub ok 2024-03-11T21:00:12.786 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:12 smithi023 bash[21372]: cluster 2024-03-11T21:00:11.542799+0000 mon.a (mon.0) 1264 : cluster 0 osdmap e200: 8 total, 6 up, 8 in 2024-03-11T21:00:12.786 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:12 smithi023 bash[21372]: cluster 2024-03-11T21:00:11.652471+0000 osd.6 (osd.6) 136 : cluster 0 2.7 scrub starts 2024-03-11T21:00:12.786 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:12 smithi023 bash[21372]: cluster 2024-03-11T21:00:11.657463+0000 osd.6 (osd.6) 137 : cluster 0 2.7 scrub ok 2024-03-11T21:00:12.786 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:12 smithi023 bash[21372]: cluster 2024-03-11T21:00:11.811562+0000 osd.7 (osd.7) 193 : cluster 0 1.0 scrub starts 2024-03-11T21:00:12.786 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:12 smithi023 bash[21372]: cluster 2024-03-11T21:00:11.815708+0000 osd.7 (osd.7) 194 : cluster 0 1.0 scrub ok 2024-03-11T21:00:12.786 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:12 smithi023 bash[21372]: cluster 2024-03-11T21:00:11.938525+0000 osd.6 (osd.6) 138 : cluster 0 purged_snaps scrub starts 2024-03-11T21:00:12.786 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:12 smithi023 bash[21372]: cluster 2024-03-11T21:00:11.938614+0000 osd.6 (osd.6) 139 : cluster 0 purged_snaps scrub ok 2024-03-11T21:00:12.786 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:12 smithi023 bash[21372]: cluster 2024-03-11T21:00:12.058033+0000 mgr.y (mgr.24380) 350 : cluster 0 pgmap v361: 148 pgs: 52 active+clean, 12 undersized+peered, 43 active+undersized, 12 undersized+degraded+peered, 29 active+undersized+degraded; 13 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 304/1182 objects degraded (25.719%) 2024-03-11T21:00:12.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:12 smithi023 bash[29146]: cluster 2024-03-11T21:00:10.710166+0000 osd.3 (osd.3) 228 : cluster 0 2.1b scrub starts 2024-03-11T21:00:12.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:12 smithi023 bash[29146]: cluster 2024-03-11T21:00:10.712801+0000 osd.3 (osd.3) 229 : cluster 0 2.1b scrub ok 2024-03-11T21:00:12.787 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:12 smithi023 bash[29146]: cluster 2024-03-11T21:00:11.294824+0000 osd.0 (osd.0) 169 : cluster 0 purged_snaps scrub starts 2024-03-11T21:00:12.787 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:12 smithi023 bash[29146]: cluster 2024-03-11T21:00:11.294912+0000 osd.0 (osd.0) 170 : cluster 0 purged_snaps scrub ok 2024-03-11T21:00:12.787 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:12 smithi023 bash[29146]: cluster 2024-03-11T21:00:11.542799+0000 mon.a (mon.0) 1264 : cluster 0 osdmap e200: 8 total, 6 up, 8 in 2024-03-11T21:00:12.787 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:12 smithi023 bash[29146]: cluster 2024-03-11T21:00:11.652471+0000 osd.6 (osd.6) 136 : cluster 0 2.7 scrub starts 2024-03-11T21:00:12.787 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:12 smithi023 bash[29146]: cluster 2024-03-11T21:00:11.657463+0000 osd.6 (osd.6) 137 : cluster 0 2.7 scrub ok 2024-03-11T21:00:12.787 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:12 smithi023 bash[29146]: cluster 2024-03-11T21:00:11.811562+0000 osd.7 (osd.7) 193 : cluster 0 1.0 scrub starts 2024-03-11T21:00:12.787 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:12 smithi023 bash[29146]: cluster 2024-03-11T21:00:11.815708+0000 osd.7 (osd.7) 194 : cluster 0 1.0 scrub ok 2024-03-11T21:00:12.787 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:12 smithi023 bash[29146]: cluster 2024-03-11T21:00:11.938525+0000 osd.6 (osd.6) 138 : cluster 0 purged_snaps scrub starts 2024-03-11T21:00:12.787 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:12 smithi023 bash[29146]: cluster 2024-03-11T21:00:11.938614+0000 osd.6 (osd.6) 139 : cluster 0 purged_snaps scrub ok 2024-03-11T21:00:12.787 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:12 smithi023 bash[29146]: cluster 2024-03-11T21:00:12.058033+0000 mgr.y (mgr.24380) 350 : cluster 0 pgmap v361: 148 pgs: 52 active+clean, 12 undersized+peered, 43 active+undersized, 12 undersized+degraded+peered, 29 active+undersized+degraded; 13 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 304/1182 objects degraded (25.719%) 2024-03-11T21:00:12.826 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:12.852 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:12.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:12.854 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:00:12.955 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:12.982 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:12.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:12.984 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:00:13.085 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:13.112 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:13.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:13.115 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:00:13.216 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:13.245 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:13.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:13.247 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:00:13.348 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:13.375 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:13.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:13.378 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:00:13.410 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 43 16 212 196 0.284849 0 - 0.00475516 2024-03-11T21:00:13.478 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:13.507 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:13.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:13.509 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:00:13.610 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:13.636 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:13.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:13.639 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:00:13.739 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:13.765 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:13.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:13.768 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:00:13.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:13 smithi023 bash[21372]: cluster 2024-03-11T21:00:12.533920+0000 mon.a (mon.0) 1265 : cluster 3 Health check update: Reduced data availability: 24 pgs inactive (PG_AVAILABILITY) 2024-03-11T21:00:13.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:13 smithi023 bash[21372]: cluster 2024-03-11T21:00:12.533978+0000 mon.a (mon.0) 1266 : cluster 3 Health check update: Degraded data redundancy: 304/1182 objects degraded (25.719%), 41 pgs degraded, 96 pgs undersized (PG_DEGRADED) 2024-03-11T21:00:13.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:13 smithi023 bash[21372]: cluster 2024-03-11T21:00:12.579316+0000 mon.a (mon.0) 1267 : cluster 0 osdmap e201: 8 total, 6 up, 8 in 2024-03-11T21:00:13.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:13 smithi023 bash[29146]: cluster 2024-03-11T21:00:12.533920+0000 mon.a (mon.0) 1265 : cluster 3 Health check update: Reduced data availability: 24 pgs inactive (PG_AVAILABILITY) 2024-03-11T21:00:13.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:13 smithi023 bash[29146]: cluster 2024-03-11T21:00:12.533978+0000 mon.a (mon.0) 1266 : cluster 3 Health check update: Degraded data redundancy: 304/1182 objects degraded (25.719%), 41 pgs degraded, 96 pgs undersized (PG_DEGRADED) 2024-03-11T21:00:13.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:13 smithi023 bash[29146]: cluster 2024-03-11T21:00:12.579316+0000 mon.a (mon.0) 1267 : cluster 0 osdmap e201: 8 total, 6 up, 8 in 2024-03-11T21:00:13.869 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:13.880 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:13 smithi146 bash[23809]: cluster 2024-03-11T21:00:12.533920+0000 mon.a (mon.0) 1265 : cluster 3 Health check update: Reduced data availability: 24 pgs inactive (PG_AVAILABILITY) 2024-03-11T21:00:13.880 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:13 smithi146 bash[23809]: cluster 2024-03-11T21:00:12.533978+0000 mon.a (mon.0) 1266 : cluster 3 Health check update: Degraded data redundancy: 304/1182 objects degraded (25.719%), 41 pgs degraded, 96 pgs undersized (PG_DEGRADED) 2024-03-11T21:00:13.880 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:13 smithi146 bash[23809]: cluster 2024-03-11T21:00:12.579316+0000 mon.a (mon.0) 1267 : cluster 0 osdmap e201: 8 total, 6 up, 8 in 2024-03-11T21:00:13.895 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:13.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:13.897 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:00:13.998 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:14.025 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:14.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:14.027 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:00:14.062 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T21:00:14.128 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:14.155 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:14.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:14.158 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:00:14.259 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:14.284 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:14.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:14.287 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:00:14.387 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:14.410 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 44 16 212 196 0.278375 0 - 0.00475516 2024-03-11T21:00:14.418 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:14.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:14.420 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:00:14.451 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd set noscrub 2024-03-11T21:00:14.521 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:14.547 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:14.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:14.549 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:00:14.650 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:14.658 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:14 smithi023 bash[29146]: cluster 2024-03-11T21:00:12.704372+0000 osd.3 (osd.3) 230 : cluster 0 6.13 scrub starts 2024-03-11T21:00:14.658 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:14 smithi023 bash[29146]: cluster 2024-03-11T21:00:12.754612+0000 osd.3 (osd.3) 231 : cluster 0 6.13 scrub ok 2024-03-11T21:00:14.658 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:14 smithi023 bash[29146]: cluster 2024-03-11T21:00:14.059036+0000 mgr.y (mgr.24380) 351 : cluster 0 pgmap v363: 148 pgs: 52 active+clean, 12 undersized+peered, 43 active+undersized, 12 undersized+degraded+peered, 29 active+undersized+degraded; 13 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 304/1182 objects degraded (25.719%) 2024-03-11T21:00:14.658 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:14 smithi023 bash[21372]: cluster 2024-03-11T21:00:12.704372+0000 osd.3 (osd.3) 230 : cluster 0 6.13 scrub starts 2024-03-11T21:00:14.658 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:14 smithi023 bash[21372]: cluster 2024-03-11T21:00:12.754612+0000 osd.3 (osd.3) 231 : cluster 0 6.13 scrub ok 2024-03-11T21:00:14.658 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:14 smithi023 bash[21372]: cluster 2024-03-11T21:00:14.059036+0000 mgr.y (mgr.24380) 351 : cluster 0 pgmap v363: 148 pgs: 52 active+clean, 12 undersized+peered, 43 active+undersized, 12 undersized+degraded+peered, 29 active+undersized+degraded; 13 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 304/1182 objects degraded (25.719%) 2024-03-11T21:00:14.668 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:14.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:14.670 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:00:14.771 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:14.797 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:14.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:14.799 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:00:14.900 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:14.911 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:14 smithi146 bash[23809]: cluster 2024-03-11T21:00:12.704372+0000 osd.3 (osd.3) 230 : cluster 0 6.13 scrub starts 2024-03-11T21:00:14.911 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:14 smithi146 bash[23809]: cluster 2024-03-11T21:00:12.754612+0000 osd.3 (osd.3) 231 : cluster 0 6.13 scrub ok 2024-03-11T21:00:14.911 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:14 smithi146 bash[23809]: cluster 2024-03-11T21:00:14.059036+0000 mgr.y (mgr.24380) 351 : cluster 0 pgmap v363: 148 pgs: 52 active+clean, 12 undersized+peered, 43 active+undersized, 12 undersized+degraded+peered, 29 active+undersized+degraded; 13 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 304/1182 objects degraded (25.719%) 2024-03-11T21:00:14.925 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:00:14.927 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:14.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:14.929 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:00:15.030 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:15.058 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:15.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:15.061 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:00:15.161 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:15.188 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:15.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:15.191 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:00:15.291 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:15.315 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:15.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:15.317 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:00:15.410 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 45 16 212 196 0.272189 0 - 0.00475516 2024-03-11T21:00:15.419 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:15.448 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:15.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:15.451 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:00:15.512 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:00:15 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:00:15.551 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:15.579 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:15.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:15.582 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:00:15.683 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:15.711 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:15.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:15.713 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:00:15.814 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:15.826 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:15 smithi023 bash[21372]: cluster 2024-03-11T21:00:13.689975+0000 osd.3 (osd.3) 232 : cluster 0 6.1f scrub starts 2024-03-11T21:00:15.826 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:15 smithi023 bash[21372]: cluster 2024-03-11T21:00:13.778524+0000 osd.3 (osd.3) 233 : cluster 0 6.1f scrub ok 2024-03-11T21:00:15.826 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:15 smithi023 bash[21372]: cluster 2024-03-11T21:00:14.017036+0000 osd.5 (osd.5) 185 : cluster 0 6.a deep-scrub starts 2024-03-11T21:00:15.826 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:15 smithi023 bash[21372]: cluster 2024-03-11T21:00:14.030588+0000 osd.5 (osd.5) 186 : cluster 0 6.a deep-scrub ok 2024-03-11T21:00:15.826 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:15 smithi023 bash[21372]: cluster 2024-03-11T21:00:14.732742+0000 osd.6 (osd.6) 140 : cluster 0 3.10 scrub starts 2024-03-11T21:00:15.826 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:15 smithi023 bash[21372]: cluster 2024-03-11T21:00:14.736044+0000 osd.6 (osd.6) 141 : cluster 0 3.10 scrub ok 2024-03-11T21:00:15.826 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:15 smithi023 bash[21372]: audit 2024-03-11T21:00:15.266499+0000 mgr.y (mgr.24380) 352 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:00:15.826 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:15 smithi023 bash[29146]: cluster 2024-03-11T21:00:13.689975+0000 osd.3 (osd.3) 232 : cluster 0 6.1f scrub starts 2024-03-11T21:00:15.826 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:15 smithi023 bash[29146]: cluster 2024-03-11T21:00:13.778524+0000 osd.3 (osd.3) 233 : cluster 0 6.1f scrub ok 2024-03-11T21:00:15.826 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:15 smithi023 bash[29146]: cluster 2024-03-11T21:00:14.017036+0000 osd.5 (osd.5) 185 : cluster 0 6.a deep-scrub starts 2024-03-11T21:00:15.827 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:15 smithi023 bash[29146]: cluster 2024-03-11T21:00:14.030588+0000 osd.5 (osd.5) 186 : cluster 0 6.a deep-scrub ok 2024-03-11T21:00:15.827 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:15 smithi023 bash[29146]: cluster 2024-03-11T21:00:14.732742+0000 osd.6 (osd.6) 140 : cluster 0 3.10 scrub starts 2024-03-11T21:00:15.827 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:15 smithi023 bash[29146]: cluster 2024-03-11T21:00:14.736044+0000 osd.6 (osd.6) 141 : cluster 0 3.10 scrub ok 2024-03-11T21:00:15.827 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:15 smithi023 bash[29146]: audit 2024-03-11T21:00:15.266499+0000 mgr.y (mgr.24380) 352 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:00:15.840 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:15.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:15.843 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:00:15.944 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:15.969 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:15.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:15.971 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:00:16.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:15 smithi146 bash[23809]: cluster 2024-03-11T21:00:13.689975+0000 osd.3 (osd.3) 232 : cluster 0 6.1f scrub starts 2024-03-11T21:00:16.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:15 smithi146 bash[23809]: cluster 2024-03-11T21:00:13.778524+0000 osd.3 (osd.3) 233 : cluster 0 6.1f scrub ok 2024-03-11T21:00:16.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:15 smithi146 bash[23809]: cluster 2024-03-11T21:00:14.017036+0000 osd.5 (osd.5) 185 : cluster 0 6.a deep-scrub starts 2024-03-11T21:00:16.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:15 smithi146 bash[23809]: cluster 2024-03-11T21:00:14.030588+0000 osd.5 (osd.5) 186 : cluster 0 6.a deep-scrub ok 2024-03-11T21:00:16.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:15 smithi146 bash[23809]: cluster 2024-03-11T21:00:14.732742+0000 osd.6 (osd.6) 140 : cluster 0 3.10 scrub starts 2024-03-11T21:00:16.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:15 smithi146 bash[23809]: cluster 2024-03-11T21:00:14.736044+0000 osd.6 (osd.6) 141 : cluster 0 3.10 scrub ok 2024-03-11T21:00:16.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:15 smithi146 bash[23809]: audit 2024-03-11T21:00:15.266499+0000 mgr.y (mgr.24380) 352 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:00:16.072 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:16.100 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:16.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:16.102 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:00:16.203 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:16.231 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:16.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:16.233 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:00:16.334 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:00:16.363 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:00:16.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:16.365 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:00:16.410 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 46 16 212 196 0.266271 0 - 0.00475516 2024-03-11T21:00:16.466 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:16.492 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:16.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:16.494 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:00:16.595 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:16.623 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:16.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:16.626 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:00:16.726 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:00:16.755 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:00:16.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:16.757 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:00:16.858 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:16.870 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:16 smithi023 bash[29146]: cluster 2024-03-11T21:00:14.733733+0000 osd.3 (osd.3) 234 : cluster 0 3.2 scrub starts 2024-03-11T21:00:16.870 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:16 smithi023 bash[29146]: cluster 2024-03-11T21:00:14.736257+0000 osd.3 (osd.3) 235 : cluster 0 3.2 scrub ok 2024-03-11T21:00:16.870 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:16 smithi023 bash[29146]: cluster 2024-03-11T21:00:14.969113+0000 osd.5 (osd.5) 187 : cluster 0 4.15 scrub starts 2024-03-11T21:00:16.870 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:16 smithi023 bash[29146]: cluster 2024-03-11T21:00:14.970581+0000 osd.5 (osd.5) 188 : cluster 0 4.15 scrub ok 2024-03-11T21:00:16.870 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:16 smithi023 bash[29146]: cluster 2024-03-11T21:00:15.387686+0000 osd.7 (osd.7) 195 : cluster 0 purged_snaps scrub starts 2024-03-11T21:00:16.870 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:16 smithi023 bash[29146]: cluster 2024-03-11T21:00:15.387815+0000 osd.7 (osd.7) 196 : cluster 0 purged_snaps scrub ok 2024-03-11T21:00:16.870 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:16 smithi023 bash[29146]: cluster 2024-03-11T21:00:16.060407+0000 mgr.y (mgr.24380) 353 : cluster 0 pgmap v364: 148 pgs: 52 active+clean, 12 undersized+peered, 43 active+undersized, 12 undersized+degraded+peered, 29 active+undersized+degraded; 13 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 304/1182 objects degraded (25.719%) 2024-03-11T21:00:16.870 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:16 smithi023 bash[29146]: cluster 2024-03-11T21:00:16.159280+0000 mon.a (mon.0) 1268 : cluster 0 osdmap e202: 8 total, 6 up, 8 in 2024-03-11T21:00:16.870 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:16 smithi023 bash[21372]: cluster 2024-03-11T21:00:14.733733+0000 osd.3 (osd.3) 234 : cluster 0 3.2 scrub starts 2024-03-11T21:00:16.871 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:16 smithi023 bash[21372]: cluster 2024-03-11T21:00:14.736257+0000 osd.3 (osd.3) 235 : cluster 0 3.2 scrub ok 2024-03-11T21:00:16.871 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:16 smithi023 bash[21372]: cluster 2024-03-11T21:00:14.969113+0000 osd.5 (osd.5) 187 : cluster 0 4.15 scrub starts 2024-03-11T21:00:16.871 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:16 smithi023 bash[21372]: cluster 2024-03-11T21:00:14.970581+0000 osd.5 (osd.5) 188 : cluster 0 4.15 scrub ok 2024-03-11T21:00:16.871 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:16 smithi023 bash[21372]: cluster 2024-03-11T21:00:15.387686+0000 osd.7 (osd.7) 195 : cluster 0 purged_snaps scrub starts 2024-03-11T21:00:16.871 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:16 smithi023 bash[21372]: cluster 2024-03-11T21:00:15.387815+0000 osd.7 (osd.7) 196 : cluster 0 purged_snaps scrub ok 2024-03-11T21:00:16.871 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:16 smithi023 bash[21372]: cluster 2024-03-11T21:00:16.060407+0000 mgr.y (mgr.24380) 353 : cluster 0 pgmap v364: 148 pgs: 52 active+clean, 12 undersized+peered, 43 active+undersized, 12 undersized+degraded+peered, 29 active+undersized+degraded; 13 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 304/1182 objects degraded (25.719%) 2024-03-11T21:00:16.871 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:16 smithi023 bash[21372]: cluster 2024-03-11T21:00:16.159280+0000 mon.a (mon.0) 1268 : cluster 0 osdmap e202: 8 total, 6 up, 8 in 2024-03-11T21:00:16.886 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:16.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:16.889 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:00:16.990 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:17.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:16 smithi146 bash[23809]: cluster 2024-03-11T21:00:14.733733+0000 osd.3 (osd.3) 234 : cluster 0 3.2 scrub starts 2024-03-11T21:00:17.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:16 smithi146 bash[23809]: cluster 2024-03-11T21:00:14.736257+0000 osd.3 (osd.3) 235 : cluster 0 3.2 scrub ok 2024-03-11T21:00:17.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:16 smithi146 bash[23809]: cluster 2024-03-11T21:00:14.969113+0000 osd.5 (osd.5) 187 : cluster 0 4.15 scrub starts 2024-03-11T21:00:17.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:16 smithi146 bash[23809]: cluster 2024-03-11T21:00:14.970581+0000 osd.5 (osd.5) 188 : cluster 0 4.15 scrub ok 2024-03-11T21:00:17.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:16 smithi146 bash[23809]: cluster 2024-03-11T21:00:15.387686+0000 osd.7 (osd.7) 195 : cluster 0 purged_snaps scrub starts 2024-03-11T21:00:17.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:16 smithi146 bash[23809]: cluster 2024-03-11T21:00:15.387815+0000 osd.7 (osd.7) 196 : cluster 0 purged_snaps scrub ok 2024-03-11T21:00:17.013 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:16 smithi146 bash[23809]: cluster 2024-03-11T21:00:16.060407+0000 mgr.y (mgr.24380) 353 : cluster 0 pgmap v364: 148 pgs: 52 active+clean, 12 undersized+peered, 43 active+undersized, 12 undersized+degraded+peered, 29 active+undersized+degraded; 13 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 304/1182 objects degraded (25.719%) 2024-03-11T21:00:17.013 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:16 smithi146 bash[23809]: cluster 2024-03-11T21:00:16.159280+0000 mon.a (mon.0) 1268 : cluster 0 osdmap e202: 8 total, 6 up, 8 in 2024-03-11T21:00:17.016 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:17.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:17.019 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:00:17.119 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:17.148 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:17.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:17.150 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:00:17.247 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 6, 7, 1] out_osds: [] dead_osds: [1, 4] live_osds: [0, 2, 3, 5, 6, 7] 2024-03-11T21:00:17.247 INFO:tasks.thrashosds.thrasher:Reviving osd 4 2024-03-11T21:00:17.247 INFO:tasks.cephadm.osd.4:Restarting osd.4 (starting--it wasn't running)... 2024-03-11T21:00:17.247 DEBUG:teuthology.orchestra.run.smithi146:osd.4> sudo journalctl -f -n 0 -u ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@osd.4.service 2024-03-11T21:00:17.250 DEBUG:teuthology.orchestra.run.smithi146:> sudo systemctl start ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@osd.4 2024-03-11T21:00:17.251 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:17.316 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-11T21:00:17.321 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:17.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:17.324 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:00:17.410 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 47 16 212 196 0.260606 0 - 0.00475516 2024-03-11T21:00:17.425 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:17.433 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 11 21:00:17 smithi146 systemd[1]: Started Ceph osd.4 for 67ba00f8-dfe7-11ee-95c8-87774f69a715. 2024-03-11T21:00:17.445 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:17.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:17.447 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:00:17.548 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:17.568 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:17.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:17.570 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:00:17.670 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:00:17.701 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:00:17.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:17.703 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:00:17.804 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:17.831 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:17.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:17.834 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:00:17.934 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:17.961 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:17.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:17.963 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:00:17.981 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:17 smithi023 bash[21372]: cluster 2024-03-11T21:00:16.989608+0000 osd.5 (osd.5) 189 : cluster 0 4.3 scrub starts 2024-03-11T21:00:17.981 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:17 smithi023 bash[21372]: cluster 2024-03-11T21:00:16.992480+0000 osd.5 (osd.5) 190 : cluster 0 4.3 scrub ok 2024-03-11T21:00:17.982 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:17 smithi023 bash[29146]: cluster 2024-03-11T21:00:16.989608+0000 osd.5 (osd.5) 189 : cluster 0 4.3 scrub starts 2024-03-11T21:00:17.982 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:17 smithi023 bash[29146]: cluster 2024-03-11T21:00:16.992480+0000 osd.5 (osd.5) 190 : cluster 0 4.3 scrub ok 2024-03-11T21:00:17.987 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:00:18.064 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:18.088 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:18.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:18.091 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:00:18.191 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:18.192 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-11T21:00:18.192 INFO:teuthology.orchestra.run.smithi023.stdout: "ops": [], 2024-03-11T21:00:18.192 INFO:teuthology.orchestra.run.smithi023.stdout: "num_ops": 0 2024-03-11T21:00:18.192 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-11T21:00:18.211 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:18.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:18.213 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:00:18.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:17 smithi146 bash[23809]: cluster 2024-03-11T21:00:16.989608+0000 osd.5 (osd.5) 189 : cluster 0 4.3 scrub starts 2024-03-11T21:00:18.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:17 smithi146 bash[23809]: cluster 2024-03-11T21:00:16.992480+0000 osd.5 (osd.5) 190 : cluster 0 4.3 scrub ok 2024-03-11T21:00:18.314 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:18.336 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:18.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:18.338 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:00:18.410 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 48 16 212 196 0.255177 0 - 0.00475516 2024-03-11T21:00:18.440 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:18.468 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:18.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:18.469 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:00:18.536 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:00:18 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:00:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:00:18.570 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:18.595 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:18.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:18.598 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:00:18.698 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:18.721 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:18.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:18.724 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:00:18.825 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:18.852 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:18.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:18.855 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:00:18.956 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:18.983 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:18.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:18.985 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:00:19.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:18 smithi146 bash[23809]: cluster 2024-03-11T21:00:17.670505+0000 osd.6 (osd.6) 142 : cluster 0 8.9 scrub starts 2024-03-11T21:00:19.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:18 smithi146 bash[23809]: cluster 2024-03-11T21:00:17.677666+0000 osd.6 (osd.6) 143 : cluster 0 8.9 scrub ok 2024-03-11T21:00:19.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:18 smithi146 bash[23809]: cluster 2024-03-11T21:00:17.729300+0000 osd.3 (osd.3) 236 : cluster 0 6.11 scrub starts 2024-03-11T21:00:19.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:18 smithi146 bash[23809]: cluster 2024-03-11T21:00:17.739689+0000 osd.3 (osd.3) 237 : cluster 0 6.11 scrub ok 2024-03-11T21:00:19.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:18 smithi146 bash[23809]: cluster 2024-03-11T21:00:18.061280+0000 mgr.y (mgr.24380) 354 : cluster 0 pgmap v366: 148 pgs: 52 active+clean, 12 undersized+peered, 43 active+undersized, 12 undersized+degraded+peered, 29 active+undersized+degraded; 13 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 304/1182 objects degraded (25.719%) 2024-03-11T21:00:19.086 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:19.096 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:18 smithi023 bash[21372]: cluster 2024-03-11T21:00:17.670505+0000 osd.6 (osd.6) 142 : cluster 0 8.9 scrub starts 2024-03-11T21:00:19.096 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:18 smithi023 bash[21372]: cluster 2024-03-11T21:00:17.677666+0000 osd.6 (osd.6) 143 : cluster 0 8.9 scrub ok 2024-03-11T21:00:19.096 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:18 smithi023 bash[21372]: cluster 2024-03-11T21:00:17.729300+0000 osd.3 (osd.3) 236 : cluster 0 6.11 scrub starts 2024-03-11T21:00:19.096 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:18 smithi023 bash[21372]: cluster 2024-03-11T21:00:17.739689+0000 osd.3 (osd.3) 237 : cluster 0 6.11 scrub ok 2024-03-11T21:00:19.096 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:18 smithi023 bash[21372]: cluster 2024-03-11T21:00:18.061280+0000 mgr.y (mgr.24380) 354 : cluster 0 pgmap v366: 148 pgs: 52 active+clean, 12 undersized+peered, 43 active+undersized, 12 undersized+degraded+peered, 29 active+undersized+degraded; 13 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 304/1182 objects degraded (25.719%) 2024-03-11T21:00:19.097 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:18 smithi023 bash[29146]: cluster 2024-03-11T21:00:17.670505+0000 osd.6 (osd.6) 142 : cluster 0 8.9 scrub starts 2024-03-11T21:00:19.097 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:18 smithi023 bash[29146]: cluster 2024-03-11T21:00:17.677666+0000 osd.6 (osd.6) 143 : cluster 0 8.9 scrub ok 2024-03-11T21:00:19.097 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:18 smithi023 bash[29146]: cluster 2024-03-11T21:00:17.729300+0000 osd.3 (osd.3) 236 : cluster 0 6.11 scrub starts 2024-03-11T21:00:19.097 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:18 smithi023 bash[29146]: cluster 2024-03-11T21:00:17.739689+0000 osd.3 (osd.3) 237 : cluster 0 6.11 scrub ok 2024-03-11T21:00:19.097 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:18 smithi023 bash[29146]: cluster 2024-03-11T21:00:18.061280+0000 mgr.y (mgr.24380) 354 : cluster 0 pgmap v366: 148 pgs: 52 active+clean, 12 undersized+peered, 43 active+undersized, 12 undersized+degraded+peered, 29 active+undersized+degraded; 13 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 304/1182 objects degraded (25.719%) 2024-03-11T21:00:19.113 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:19.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:19.116 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:00:19.217 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:19.240 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:19.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:19.243 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:00:19.344 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:19.362 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:19.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:19.365 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:00:19.410 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 49 16 212 196 0.249969 0 - 0.00475516 2024-03-11T21:00:19.466 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:19.493 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:19.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:19.496 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:00:19.504 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 11 21:00:19 smithi146 bash[67924]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-11T21:00:19.504 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 11 21:00:19 smithi146 bash[67924]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-03-11T21:00:19.504 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 11 21:00:19 smithi146 bash[67924]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-03-11T21:00:19.504 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 11 21:00:19 smithi146 bash[67924]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-03-11T21:00:19.504 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 11 21:00:19 smithi146 bash[67924]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-4/block 2024-03-11T21:00:19.505 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 11 21:00:19 smithi146 bash[67924]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-11T21:00:19.505 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 11 21:00:19 smithi146 bash[67924]: --> ceph-volume raw activate successful for osd ID: 4 2024-03-11T21:00:19.597 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:19.624 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:19.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:19.627 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:00:19.727 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:19.755 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:19.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:19.757 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:00:19.858 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:19.884 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:19.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:19.887 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:00:19.987 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:20.014 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:20.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:20.017 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:00:20.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:19 smithi023 bash[21372]: cluster 2024-03-11T21:00:18.899222+0000 osd.7 (osd.7) 197 : cluster 0 6.9 scrub starts 2024-03-11T21:00:20.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:19 smithi023 bash[21372]: cluster 2024-03-11T21:00:18.927538+0000 osd.7 (osd.7) 198 : cluster 0 6.9 scrub ok 2024-03-11T21:00:20.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:19 smithi023 bash[21372]: cluster 2024-03-11T21:00:19.047842+0000 osd.0 (osd.0) 171 : cluster 0 8.c scrub starts 2024-03-11T21:00:20.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:19 smithi023 bash[21372]: cluster 2024-03-11T21:00:19.050968+0000 osd.0 (osd.0) 172 : cluster 0 8.c scrub ok 2024-03-11T21:00:20.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:19 smithi023 bash[21372]: audit 2024-03-11T21:00:19.838420+0000 mon.a (mon.0) 1269 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:00:20.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:19 smithi023 bash[21372]: audit 2024-03-11T21:00:19.840128+0000 mon.a (mon.0) 1270 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:00:20.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:19 smithi023 bash[29146]: cluster 2024-03-11T21:00:18.899222+0000 osd.7 (osd.7) 197 : cluster 0 6.9 scrub starts 2024-03-11T21:00:20.036 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:19 smithi023 bash[29146]: cluster 2024-03-11T21:00:18.927538+0000 osd.7 (osd.7) 198 : cluster 0 6.9 scrub ok 2024-03-11T21:00:20.036 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:19 smithi023 bash[29146]: cluster 2024-03-11T21:00:19.047842+0000 osd.0 (osd.0) 171 : cluster 0 8.c scrub starts 2024-03-11T21:00:20.036 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:19 smithi023 bash[29146]: cluster 2024-03-11T21:00:19.050968+0000 osd.0 (osd.0) 172 : cluster 0 8.c scrub ok 2024-03-11T21:00:20.036 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:19 smithi023 bash[29146]: audit 2024-03-11T21:00:19.838420+0000 mon.a (mon.0) 1269 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:00:20.036 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:19 smithi023 bash[29146]: audit 2024-03-11T21:00:19.840128+0000 mon.a (mon.0) 1270 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:00:20.062 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-03-11T21:00:20.117 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:20.134 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:19 smithi146 bash[23809]: cluster 2024-03-11T21:00:18.899222+0000 osd.7 (osd.7) 197 : cluster 0 6.9 scrub starts 2024-03-11T21:00:20.134 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:19 smithi146 bash[23809]: cluster 2024-03-11T21:00:18.927538+0000 osd.7 (osd.7) 198 : cluster 0 6.9 scrub ok 2024-03-11T21:00:20.135 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:19 smithi146 bash[23809]: cluster 2024-03-11T21:00:19.047842+0000 osd.0 (osd.0) 171 : cluster 0 8.c scrub starts 2024-03-11T21:00:20.135 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:19 smithi146 bash[23809]: cluster 2024-03-11T21:00:19.050968+0000 osd.0 (osd.0) 172 : cluster 0 8.c scrub ok 2024-03-11T21:00:20.135 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:19 smithi146 bash[23809]: audit 2024-03-11T21:00:19.838420+0000 mon.a (mon.0) 1269 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:00:20.135 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:19 smithi146 bash[23809]: audit 2024-03-11T21:00:19.840128+0000 mon.a (mon.0) 1270 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:00:20.138 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:20.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:20.140 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:00:20.241 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:20.265 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:20.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:20.268 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:00:20.369 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:00:20.398 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:00:20.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:20.400 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:00:20.410 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 50 16 212 196 0.24497 0 - 0.00475516 2024-03-11T21:00:20.501 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:20.517 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:00:20.527 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:20.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:20.530 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:00:20.630 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:20.652 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:20.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:20.654 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:00:20.755 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:20.778 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:20.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:20.781 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:00:20.881 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:20.906 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:20.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:20.908 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:00:21.008 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:21.038 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:21.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:21.040 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:00:21.141 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:00:21.151 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:20 smithi023 bash[21372]: cluster 2024-03-11T21:00:20.004480+0000 osd.5 (osd.5) 191 : cluster 0 3.17 scrub starts 2024-03-11T21:00:21.151 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:20 smithi023 bash[21372]: cluster 2024-03-11T21:00:20.008506+0000 osd.5 (osd.5) 192 : cluster 0 3.17 scrub ok 2024-03-11T21:00:21.151 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:20 smithi023 bash[21372]: cluster 2024-03-11T21:00:20.062367+0000 mgr.y (mgr.24380) 355 : cluster 0 pgmap v367: 148 pgs: 52 active+clean, 12 undersized+peered, 43 active+undersized, 12 undersized+degraded+peered, 29 active+undersized+degraded; 13 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 304/1182 objects degraded (25.719%) 2024-03-11T21:00:21.152 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:20 smithi023 bash[29146]: cluster 2024-03-11T21:00:20.004480+0000 osd.5 (osd.5) 191 : cluster 0 3.17 scrub starts 2024-03-11T21:00:21.152 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:20 smithi023 bash[29146]: cluster 2024-03-11T21:00:20.008506+0000 osd.5 (osd.5) 192 : cluster 0 3.17 scrub ok 2024-03-11T21:00:21.152 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:20 smithi023 bash[29146]: cluster 2024-03-11T21:00:20.062367+0000 mgr.y (mgr.24380) 355 : cluster 0 pgmap v367: 148 pgs: 52 active+clean, 12 undersized+peered, 43 active+undersized, 12 undersized+degraded+peered, 29 active+undersized+degraded; 13 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 304/1182 objects degraded (25.719%) 2024-03-11T21:00:21.169 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:00:21.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:21.171 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:00:21.188 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:20 smithi146 bash[23809]: cluster 2024-03-11T21:00:20.004480+0000 osd.5 (osd.5) 191 : cluster 0 3.17 scrub starts 2024-03-11T21:00:21.188 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:20 smithi146 bash[23809]: cluster 2024-03-11T21:00:20.008506+0000 osd.5 (osd.5) 192 : cluster 0 3.17 scrub ok 2024-03-11T21:00:21.188 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:20 smithi146 bash[23809]: cluster 2024-03-11T21:00:20.062367+0000 mgr.y (mgr.24380) 355 : cluster 0 pgmap v367: 148 pgs: 52 active+clean, 12 undersized+peered, 43 active+undersized, 12 undersized+degraded+peered, 29 active+undersized+degraded; 13 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 304/1182 objects degraded (25.719%) 2024-03-11T21:00:21.272 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:21.301 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:21.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:21.303 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:00:21.404 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:21.411 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 51 16 212 196 0.240166 0 - 0.00475516 2024-03-11T21:00:21.433 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:21.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:21.435 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:00:21.512 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 11 21:00:21 smithi146 bash[68242]: debug 2024-03-11T21:00:21.277+0000 7f6db4deb700 1 -- 172.21.15.146:0/3939899927 <== mon.1 v2:172.21.15.146:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55bceda97680 con 0x55bcedad2800 2024-03-11T21:00:21.536 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:21.564 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:21.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:21.567 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:00:21.668 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:21.696 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:21.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:21.699 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:00:21.800 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:21.824 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:21.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:21.827 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:00:21.928 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:00:21.957 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:00:21.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:21.960 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:00:22.060 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:22.086 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:22.087 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 11 21:00:21 smithi146 bash[68242]: debug 2024-03-11T21:00:21.829+0000 7f6dbdf837c0 -1 Falling back to public interface 2024-03-11T21:00:22.087 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:21 smithi146 bash[23809]: cluster 2024-03-11T21:00:20.766428+0000 osd.3 (osd.3) 238 : cluster 0 4.0 scrub starts 2024-03-11T21:00:22.087 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:21 smithi146 bash[23809]: cluster 2024-03-11T21:00:20.771530+0000 osd.3 (osd.3) 239 : cluster 0 4.0 scrub ok 2024-03-11T21:00:22.088 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:21 smithi146 bash[23809]: cluster 2024-03-11T21:00:20.823000+0000 osd.7 (osd.7) 199 : cluster 0 3.12 scrub starts 2024-03-11T21:00:22.088 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:21 smithi146 bash[23809]: cluster 2024-03-11T21:00:20.825558+0000 osd.7 (osd.7) 200 : cluster 0 3.12 scrub ok 2024-03-11T21:00:22.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:22.088 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:00:22.189 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:22.202 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:21 smithi023 bash[21372]: cluster 2024-03-11T21:00:20.766428+0000 osd.3 (osd.3) 238 : cluster 0 4.0 scrub starts 2024-03-11T21:00:22.202 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:21 smithi023 bash[21372]: cluster 2024-03-11T21:00:20.771530+0000 osd.3 (osd.3) 239 : cluster 0 4.0 scrub ok 2024-03-11T21:00:22.202 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:21 smithi023 bash[21372]: cluster 2024-03-11T21:00:20.823000+0000 osd.7 (osd.7) 199 : cluster 0 3.12 scrub starts 2024-03-11T21:00:22.202 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:21 smithi023 bash[21372]: cluster 2024-03-11T21:00:20.825558+0000 osd.7 (osd.7) 200 : cluster 0 3.12 scrub ok 2024-03-11T21:00:22.202 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:21 smithi023 bash[29146]: cluster 2024-03-11T21:00:20.766428+0000 osd.3 (osd.3) 238 : cluster 0 4.0 scrub starts 2024-03-11T21:00:22.202 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:21 smithi023 bash[29146]: cluster 2024-03-11T21:00:20.771530+0000 osd.3 (osd.3) 239 : cluster 0 4.0 scrub ok 2024-03-11T21:00:22.202 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:21 smithi023 bash[29146]: cluster 2024-03-11T21:00:20.823000+0000 osd.7 (osd.7) 199 : cluster 0 3.12 scrub starts 2024-03-11T21:00:22.202 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:21 smithi023 bash[29146]: cluster 2024-03-11T21:00:20.825558+0000 osd.7 (osd.7) 200 : cluster 0 3.12 scrub ok 2024-03-11T21:00:22.218 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:22.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:22.220 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:00:22.321 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:22.345 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:22.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:22.347 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:00:22.411 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 52 16 212 196 0.235547 0 - 0.00475516 2024-03-11T21:00:22.448 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:22.475 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:22.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:22.478 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:00:22.579 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:22.604 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:22.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:22.606 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:00:22.707 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:22.735 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:22.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:22.737 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:00:22.838 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:22.861 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:22.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:22.863 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:00:22.964 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:22.975 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:22 smithi146 bash[23809]: cluster 2024-03-11T21:00:21.967518+0000 osd.5 (osd.5) 193 : cluster 0 4.8 scrub starts 2024-03-11T21:00:22.975 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:22 smithi146 bash[23809]: cluster 2024-03-11T21:00:21.969975+0000 osd.5 (osd.5) 194 : cluster 0 4.8 scrub ok 2024-03-11T21:00:22.975 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:22 smithi146 bash[23809]: cluster 2024-03-11T21:00:22.064855+0000 mgr.y (mgr.24380) 356 : cluster 0 pgmap v368: 148 pgs: 52 active+clean, 12 undersized+peered, 43 active+undersized, 12 undersized+degraded+peered, 29 active+undersized+degraded; 13 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 304/1182 objects degraded (25.719%) 2024-03-11T21:00:22.990 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:22.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:22.993 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:00:23.093 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:23.116 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:23.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:23.118 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:00:23.153 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:22 smithi023 bash[21372]: cluster 2024-03-11T21:00:21.967518+0000 osd.5 (osd.5) 193 : cluster 0 4.8 scrub starts 2024-03-11T21:00:23.153 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:22 smithi023 bash[21372]: cluster 2024-03-11T21:00:21.969975+0000 osd.5 (osd.5) 194 : cluster 0 4.8 scrub ok 2024-03-11T21:00:23.153 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:22 smithi023 bash[21372]: cluster 2024-03-11T21:00:22.064855+0000 mgr.y (mgr.24380) 356 : cluster 0 pgmap v368: 148 pgs: 52 active+clean, 12 undersized+peered, 43 active+undersized, 12 undersized+degraded+peered, 29 active+undersized+degraded; 13 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 304/1182 objects degraded (25.719%) 2024-03-11T21:00:23.153 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:22 smithi023 bash[29146]: cluster 2024-03-11T21:00:21.967518+0000 osd.5 (osd.5) 193 : cluster 0 4.8 scrub starts 2024-03-11T21:00:23.153 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:22 smithi023 bash[29146]: cluster 2024-03-11T21:00:21.969975+0000 osd.5 (osd.5) 194 : cluster 0 4.8 scrub ok 2024-03-11T21:00:23.153 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:22 smithi023 bash[29146]: cluster 2024-03-11T21:00:22.064855+0000 mgr.y (mgr.24380) 356 : cluster 0 pgmap v368: 148 pgs: 52 active+clean, 12 undersized+peered, 43 active+undersized, 12 undersized+degraded+peered, 29 active+undersized+degraded; 13 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 304/1182 objects degraded (25.719%) 2024-03-11T21:00:23.219 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:23.245 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:23.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:23.247 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:00:23.348 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:23.372 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:23.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:23.374 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:00:23.411 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 53 16 212 196 0.231103 0 - 0.00475516 2024-03-11T21:00:23.465 INFO:teuthology.orchestra.run.smithi023.stdout:osd.0: {} 2024-03-11T21:00:23.476 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:00:23.486 INFO:teuthology.orchestra.run.smithi023.stdout:osd.2: {} 2024-03-11T21:00:23.495 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:00:23.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:23.497 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:00:23.508 INFO:teuthology.orchestra.run.smithi023.stdout:osd.3: {} 2024-03-11T21:00:23.537 INFO:teuthology.orchestra.run.smithi023.stdout:osd.5: {} 2024-03-11T21:00:23.562 INFO:teuthology.orchestra.run.smithi023.stdout:osd.6: {} 2024-03-11T21:00:23.588 INFO:teuthology.orchestra.run.smithi023.stdout:osd.7: {} 2024-03-11T21:00:23.589 INFO:teuthology.orchestra.run.smithi023.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:00:23.589 INFO:teuthology.orchestra.run.smithi023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-11T21:00:23.589 INFO:teuthology.orchestra.run.smithi023.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:00:23.589 INFO:teuthology.orchestra.run.smithi023.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:00:23.589 INFO:teuthology.orchestra.run.smithi023.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-03-11T21:00:23.589 INFO:teuthology.orchestra.run.smithi023.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:00:23.589 INFO:teuthology.orchestra.run.smithi023.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:00:23.589 INFO:teuthology.orchestra.run.smithi023.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:00:23.598 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:23.618 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:23.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:23.620 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:00:23.634 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:00:23.721 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:23.750 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:23.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:23.753 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:00:23.854 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:00:23.883 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:00:23.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:23.885 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:00:23.986 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:24.011 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:24.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:23 smithi146 bash[23809]: cluster 2024-03-11T21:00:22.851252+0000 osd.7 (osd.7) 201 : cluster 0 6.8 scrub starts 2024-03-11T21:00:24.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:23 smithi146 bash[23809]: cluster 2024-03-11T21:00:22.853670+0000 osd.7 (osd.7) 202 : cluster 0 6.8 scrub ok 2024-03-11T21:00:24.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:23 smithi146 bash[23809]: cluster 2024-03-11T21:00:23.000856+0000 osd.5 (osd.5) 195 : cluster 0 2.3 scrub starts 2024-03-11T21:00:24.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:23 smithi146 bash[23809]: cluster 2024-03-11T21:00:23.002461+0000 osd.5 (osd.5) 196 : cluster 0 2.3 scrub ok 2024-03-11T21:00:24.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:23 smithi146 bash[23809]: audit 2024-03-11T21:00:23.446220+0000 mon.c (mon.2) 47 : audit 0 from='client.? 172.21.15.23:0/4007355859' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:00:24.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:24.014 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:00:24.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:23 smithi023 bash[21372]: cluster 2024-03-11T21:00:22.851252+0000 osd.7 (osd.7) 201 : cluster 0 6.8 scrub starts 2024-03-11T21:00:24.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:23 smithi023 bash[21372]: cluster 2024-03-11T21:00:22.853670+0000 osd.7 (osd.7) 202 : cluster 0 6.8 scrub ok 2024-03-11T21:00:24.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:23 smithi023 bash[21372]: cluster 2024-03-11T21:00:23.000856+0000 osd.5 (osd.5) 195 : cluster 0 2.3 scrub starts 2024-03-11T21:00:24.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:23 smithi023 bash[21372]: cluster 2024-03-11T21:00:23.002461+0000 osd.5 (osd.5) 196 : cluster 0 2.3 scrub ok 2024-03-11T21:00:24.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:23 smithi023 bash[21372]: audit 2024-03-11T21:00:23.446220+0000 mon.c (mon.2) 47 : audit 0 from='client.? 172.21.15.23:0/4007355859' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:00:24.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:23 smithi023 bash[29146]: cluster 2024-03-11T21:00:22.851252+0000 osd.7 (osd.7) 201 : cluster 0 6.8 scrub starts 2024-03-11T21:00:24.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:23 smithi023 bash[29146]: cluster 2024-03-11T21:00:22.853670+0000 osd.7 (osd.7) 202 : cluster 0 6.8 scrub ok 2024-03-11T21:00:24.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:23 smithi023 bash[29146]: cluster 2024-03-11T21:00:23.000856+0000 osd.5 (osd.5) 195 : cluster 0 2.3 scrub starts 2024-03-11T21:00:24.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:23 smithi023 bash[29146]: cluster 2024-03-11T21:00:23.002461+0000 osd.5 (osd.5) 196 : cluster 0 2.3 scrub ok 2024-03-11T21:00:24.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:23 smithi023 bash[29146]: audit 2024-03-11T21:00:23.446220+0000 mon.c (mon.2) 47 : audit 0 from='client.? 172.21.15.23:0/4007355859' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:00:24.115 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:24.142 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:24.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:24.144 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:00:24.245 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:24.274 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:24.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:24.276 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:00:24.377 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:24.403 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:24.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:24.405 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:00:24.411 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 54 16 212 196 0.226823 0 - 0.00475516 2024-03-11T21:00:24.505 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:24.532 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:24.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:24.535 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:00:24.635 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:24.656 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:24.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:24.658 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:00:24.759 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:24.790 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:24.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:24.792 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:00:24.893 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:24.902 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:24 smithi146 bash[23809]: cluster 2024-03-11T21:00:23.801882+0000 osd.7 (osd.7) 203 : cluster 0 2.1c scrub starts 2024-03-11T21:00:24.902 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:24 smithi146 bash[23809]: cluster 2024-03-11T21:00:23.804274+0000 osd.7 (osd.7) 204 : cluster 0 2.1c scrub ok 2024-03-11T21:00:24.902 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:24 smithi146 bash[23809]: cluster 2024-03-11T21:00:24.009248+0000 osd.5 (osd.5) 197 : cluster 0 2.16 scrub starts 2024-03-11T21:00:24.902 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:24 smithi146 bash[23809]: cluster 2024-03-11T21:00:24.010768+0000 osd.5 (osd.5) 198 : cluster 0 2.16 scrub ok 2024-03-11T21:00:24.902 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:24 smithi146 bash[23809]: cluster 2024-03-11T21:00:24.065876+0000 mgr.y (mgr.24380) 357 : cluster 0 pgmap v369: 148 pgs: 52 active+clean, 12 undersized+peered, 43 active+undersized, 12 undersized+degraded+peered, 29 active+undersized+degraded; 13 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 304/1182 objects degraded (25.719%) 2024-03-11T21:00:24.915 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:24.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:24.917 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:00:25.010 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:24 smithi023 bash[21372]: cluster 2024-03-11T21:00:23.801882+0000 osd.7 (osd.7) 203 : cluster 0 2.1c scrub starts 2024-03-11T21:00:25.010 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:24 smithi023 bash[21372]: cluster 2024-03-11T21:00:23.804274+0000 osd.7 (osd.7) 204 : cluster 0 2.1c scrub ok 2024-03-11T21:00:25.010 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:24 smithi023 bash[21372]: cluster 2024-03-11T21:00:24.009248+0000 osd.5 (osd.5) 197 : cluster 0 2.16 scrub starts 2024-03-11T21:00:25.010 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:24 smithi023 bash[21372]: cluster 2024-03-11T21:00:24.010768+0000 osd.5 (osd.5) 198 : cluster 0 2.16 scrub ok 2024-03-11T21:00:25.011 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:24 smithi023 bash[21372]: cluster 2024-03-11T21:00:24.065876+0000 mgr.y (mgr.24380) 357 : cluster 0 pgmap v369: 148 pgs: 52 active+clean, 12 undersized+peered, 43 active+undersized, 12 undersized+degraded+peered, 29 active+undersized+degraded; 13 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 304/1182 objects degraded (25.719%) 2024-03-11T21:00:25.018 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:25.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:25 smithi023 bash[29146]: cluster 2024-03-11T21:00:23.801882+0000 osd.7 (osd.7) 203 : cluster 0 2.1c scrub starts 2024-03-11T21:00:25.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:25 smithi023 bash[29146]: cluster 2024-03-11T21:00:23.804274+0000 osd.7 (osd.7) 204 : cluster 0 2.1c scrub ok 2024-03-11T21:00:25.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:25 smithi023 bash[29146]: cluster 2024-03-11T21:00:24.009248+0000 osd.5 (osd.5) 197 : cluster 0 2.16 scrub starts 2024-03-11T21:00:25.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:25 smithi023 bash[29146]: cluster 2024-03-11T21:00:24.010768+0000 osd.5 (osd.5) 198 : cluster 0 2.16 scrub ok 2024-03-11T21:00:25.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:25 smithi023 bash[29146]: cluster 2024-03-11T21:00:24.065876+0000 mgr.y (mgr.24380) 357 : cluster 0 pgmap v369: 148 pgs: 52 active+clean, 12 undersized+peered, 43 active+undersized, 12 undersized+degraded+peered, 29 active+undersized+degraded; 13 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 304/1182 objects degraded (25.719%) 2024-03-11T21:00:25.037 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:25.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:25.039 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:00:25.140 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:25.143 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 11 21:00:24 smithi146 bash[68242]: debug 2024-03-11T21:00:24.981+0000 7f6dbdf837c0 -1 osd.4 182 log_to_monitors true 2024-03-11T21:00:25.166 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:25.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:25.168 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:00:25.269 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:25.296 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:25.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:25.298 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:00:25.399 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:25.410 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:00:25 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:00:25.411 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 55 16 212 196 0.222699 0 - 0.00475516 2024-03-11T21:00:25.427 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:25.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:25.430 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:00:25.531 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:25.557 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:25.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:25.559 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:00:25.660 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:25.681 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-11T21:00:25.681 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-11T21:00:25.687 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:25.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:25.690 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:00:25.725 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:00:25.791 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:25.818 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:25.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:25.821 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:00:25.921 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:25.936 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:25 smithi146 bash[23809]: cluster 2024-03-11T21:00:24.782008+0000 osd.3 (osd.3) 240 : cluster 0 6.0 scrub starts 2024-03-11T21:00:25.936 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:25 smithi146 bash[23809]: cluster 2024-03-11T21:00:24.793088+0000 osd.3 (osd.3) 241 : cluster 0 6.0 scrub ok 2024-03-11T21:00:25.936 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:25 smithi146 bash[23809]: audit 2024-03-11T21:00:24.986659+0000 mon.a (mon.0) 1271 : audit 1 from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-11T21:00:25.936 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:25 smithi146 bash[23809]: audit 2024-03-11T21:00:24.989094+0000 mon.b (mon.1) 36 : audit 1 from='osd.4 [v2:172.21.15.146:6800/3858359434,v1:172.21.15.146:6801/3858359434]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-11T21:00:25.936 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:25 smithi146 bash[23809]: audit 2024-03-11T21:00:25.277901+0000 mgr.y (mgr.24380) 358 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:00:25.945 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:25.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:25.948 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:00:26.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:25 smithi023 bash[21372]: cluster 2024-03-11T21:00:24.782008+0000 osd.3 (osd.3) 240 : cluster 0 6.0 scrub starts 2024-03-11T21:00:26.067 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:25 smithi023 bash[21372]: cluster 2024-03-11T21:00:24.793088+0000 osd.3 (osd.3) 241 : cluster 0 6.0 scrub ok 2024-03-11T21:00:26.067 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:25 smithi023 bash[21372]: audit 2024-03-11T21:00:24.986659+0000 mon.a (mon.0) 1271 : audit 1 from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-11T21:00:26.067 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:26 smithi023 bash[21372]: audit 2024-03-11T21:00:24.989094+0000 mon.b (mon.1) 36 : audit 1 from='osd.4 [v2:172.21.15.146:6800/3858359434,v1:172.21.15.146:6801/3858359434]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-11T21:00:26.068 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:26.093 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:26.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:26.095 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:00:26.195 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:26.225 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:26.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:26.228 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:00:26.248 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 11 21:00:25 smithi146 bash[68242]: debug 2024-03-11T21:00:25.961+0000 7f6daf3db700 -1 osd.4 182 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T21:00:26.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:26 smithi023 bash[21372]: audit 2024-03-11T21:00:25.277901+0000 mgr.y (mgr.24380) 358 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:00:26.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:26 smithi023 bash[29146]: cluster 2024-03-11T21:00:24.782008+0000 osd.3 (osd.3) 240 : cluster 0 6.0 scrub starts 2024-03-11T21:00:26.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:26 smithi023 bash[29146]: cluster 2024-03-11T21:00:24.793088+0000 osd.3 (osd.3) 241 : cluster 0 6.0 scrub ok 2024-03-11T21:00:26.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:26 smithi023 bash[29146]: audit 2024-03-11T21:00:24.986659+0000 mon.a (mon.0) 1271 : audit 1 from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-11T21:00:26.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:26 smithi023 bash[29146]: audit 2024-03-11T21:00:24.989094+0000 mon.b (mon.1) 36 : audit 1 from='osd.4 [v2:172.21.15.146:6800/3858359434,v1:172.21.15.146:6801/3858359434]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-11T21:00:26.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:26 smithi023 bash[29146]: audit 2024-03-11T21:00:25.277901+0000 mgr.y (mgr.24380) 358 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:00:26.329 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:26.356 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:26.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:26.358 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:00:26.411 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 56 16 212 196 0.218722 0 - 0.00475516 2024-03-11T21:00:26.459 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:26.486 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:26.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:26.488 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:00:26.589 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:00:26.618 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:00:26.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:26.621 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:00:26.722 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:26.749 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:26.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:26.751 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:00:26.851 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:26.880 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:26.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:26.884 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:00:26.984 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:27.006 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:27.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:27.009 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:00:27.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:26 smithi146 bash[23809]: cluster 2024-03-11T21:00:25.878986+0000 osd.7 (osd.7) 205 : cluster 0 3.d scrub starts 2024-03-11T21:00:27.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:26 smithi146 bash[23809]: cluster 2024-03-11T21:00:25.881792+0000 osd.7 (osd.7) 206 : cluster 0 3.d scrub ok 2024-03-11T21:00:27.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:26 smithi146 bash[23809]: audit 2024-03-11T21:00:25.910868+0000 mon.a (mon.0) 1272 : audit 1 from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-11T21:00:27.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:26 smithi146 bash[23809]: audit 2024-03-11T21:00:25.917628+0000 mon.b (mon.1) 37 : audit 1 from='osd.4 [v2:172.21.15.146:6800/3858359434,v1:172.21.15.146:6801/3858359434]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-03-11T21:00:27.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:26 smithi146 bash[23809]: cluster 2024-03-11T21:00:25.917915+0000 mon.a (mon.0) 1273 : cluster 0 osdmap e203: 8 total, 6 up, 8 in 2024-03-11T21:00:27.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:26 smithi146 bash[23809]: audit 2024-03-11T21:00:25.919318+0000 mon.a (mon.0) 1274 : audit 1 from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-03-11T21:00:27.013 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:26 smithi146 bash[23809]: cluster 2024-03-11T21:00:25.942160+0000 osd.5 (osd.5) 199 : cluster 0 2.12 scrub starts 2024-03-11T21:00:27.013 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:26 smithi146 bash[23809]: cluster 2024-03-11T21:00:25.944942+0000 osd.5 (osd.5) 200 : cluster 0 2.12 scrub ok 2024-03-11T21:00:27.013 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:26 smithi146 bash[23809]: cluster 2024-03-11T21:00:26.067831+0000 mgr.y (mgr.24380) 359 : cluster 0 pgmap v371: 148 pgs: 52 active+clean, 12 undersized+peered, 43 active+undersized, 12 undersized+degraded+peered, 29 active+undersized+degraded; 13 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 304/1182 objects degraded (25.719%) 2024-03-11T21:00:27.110 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:27.136 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:27.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:27.139 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:00:27.208 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:26 smithi023 bash[29146]: cluster 2024-03-11T21:00:25.878986+0000 osd.7 (osd.7) 205 : cluster 0 3.d scrub starts 2024-03-11T21:00:27.208 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:26 smithi023 bash[29146]: cluster 2024-03-11T21:00:25.881792+0000 osd.7 (osd.7) 206 : cluster 0 3.d scrub ok 2024-03-11T21:00:27.208 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:26 smithi023 bash[29146]: audit 2024-03-11T21:00:25.910868+0000 mon.a (mon.0) 1272 : audit 1 from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-11T21:00:27.208 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:26 smithi023 bash[29146]: audit 2024-03-11T21:00:25.917628+0000 mon.b (mon.1) 37 : audit 1 from='osd.4 [v2:172.21.15.146:6800/3858359434,v1:172.21.15.146:6801/3858359434]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-03-11T21:00:27.208 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:26 smithi023 bash[29146]: cluster 2024-03-11T21:00:25.917915+0000 mon.a (mon.0) 1273 : cluster 0 osdmap e203: 8 total, 6 up, 8 in 2024-03-11T21:00:27.208 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:26 smithi023 bash[29146]: audit 2024-03-11T21:00:25.919318+0000 mon.a (mon.0) 1274 : audit 1 from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-03-11T21:00:27.208 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:26 smithi023 bash[29146]: cluster 2024-03-11T21:00:25.942160+0000 osd.5 (osd.5) 199 : cluster 0 2.12 scrub starts 2024-03-11T21:00:27.208 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:26 smithi023 bash[29146]: cluster 2024-03-11T21:00:25.944942+0000 osd.5 (osd.5) 200 : cluster 0 2.12 scrub ok 2024-03-11T21:00:27.208 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:26 smithi023 bash[29146]: cluster 2024-03-11T21:00:26.067831+0000 mgr.y (mgr.24380) 359 : cluster 0 pgmap v371: 148 pgs: 52 active+clean, 12 undersized+peered, 43 active+undersized, 12 undersized+degraded+peered, 29 active+undersized+degraded; 13 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 304/1182 objects degraded (25.719%) 2024-03-11T21:00:27.209 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:26 smithi023 bash[21372]: cluster 2024-03-11T21:00:25.878986+0000 osd.7 (osd.7) 205 : cluster 0 3.d scrub starts 2024-03-11T21:00:27.209 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:26 smithi023 bash[21372]: cluster 2024-03-11T21:00:25.881792+0000 osd.7 (osd.7) 206 : cluster 0 3.d scrub ok 2024-03-11T21:00:27.209 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:26 smithi023 bash[21372]: audit 2024-03-11T21:00:25.910868+0000 mon.a (mon.0) 1272 : audit 1 from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-11T21:00:27.209 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:26 smithi023 bash[21372]: audit 2024-03-11T21:00:25.917628+0000 mon.b (mon.1) 37 : audit 1 from='osd.4 [v2:172.21.15.146:6800/3858359434,v1:172.21.15.146:6801/3858359434]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-03-11T21:00:27.209 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:26 smithi023 bash[21372]: cluster 2024-03-11T21:00:25.917915+0000 mon.a (mon.0) 1273 : cluster 0 osdmap e203: 8 total, 6 up, 8 in 2024-03-11T21:00:27.209 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:26 smithi023 bash[21372]: audit 2024-03-11T21:00:25.919318+0000 mon.a (mon.0) 1274 : audit 1 from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-03-11T21:00:27.209 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:26 smithi023 bash[21372]: cluster 2024-03-11T21:00:25.942160+0000 osd.5 (osd.5) 199 : cluster 0 2.12 scrub starts 2024-03-11T21:00:27.209 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:26 smithi023 bash[21372]: cluster 2024-03-11T21:00:25.944942+0000 osd.5 (osd.5) 200 : cluster 0 2.12 scrub ok 2024-03-11T21:00:27.209 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:26 smithi023 bash[21372]: cluster 2024-03-11T21:00:26.067831+0000 mgr.y (mgr.24380) 359 : cluster 0 pgmap v371: 148 pgs: 52 active+clean, 12 undersized+peered, 43 active+undersized, 12 undersized+degraded+peered, 29 active+undersized+degraded; 13 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 304/1182 objects degraded (25.719%) 2024-03-11T21:00:27.240 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:00:27.259 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:00:27.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:27.262 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:00:27.363 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:27.384 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:27.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:27.386 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:00:27.409 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-03-11T21:00:27.409 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-03-11T21:00:27.410 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:00:27.410 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.24281.0:2123 4.d 4.a204812d (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-03-11T21:00:27.410 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T21:00:26.952063+0000", 2024-03-11T21:00:27.410 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.46388601800000001, 2024-03-11T21:00:27.410 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.46391412999999998, 2024-03-11T21:00:27.410 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:00:27.410 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "waiting for peered", 2024-03-11T21:00:27.410 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-03-11T21:00:27.410 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.24281", 2024-03-11T21:00:27.410 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.23:0/2185274774", 2024-03-11T21:00:27.410 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2123 2024-03-11T21:00:27.410 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:00:27.411 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:00:27.411 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:00:27.411 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:00:27.411 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:00:26.952063+0000", 2024-03-11T21:00:27.411 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:00:27.411 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:00:27.411 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:00:27.411 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:00:27.411 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:00:26.952063+0000", 2024-03-11T21:00:27.411 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:00:27.411 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:00:27.411 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:00:27.412 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:00:27.412 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:00:26.952065+0000", 2024-03-11T21:00:27.412 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.216e-06 2024-03-11T21:00:27.412 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:00:27.412 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:00:27.412 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:00:27.412 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:00:26.952074+0000", 2024-03-11T21:00:27.412 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4460000000000001e-06 2024-03-11T21:00:27.412 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:00:27.412 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:00:27.412 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:00:27.412 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:00:26.952125+0000", 2024-03-11T21:00:27.412 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1100999999999997e-05 2024-03-11T21:00:27.412 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:00:27.413 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:00:27.413 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:00:27.413 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:00:26.952145+0000", 2024-03-11T21:00:27.413 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0072999999999999e-05 2024-03-11T21:00:27.413 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:00:27.413 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:00:27.413 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:00:27.413 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:00:26.973658+0000", 2024-03-11T21:00:27.413 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.021513365 2024-03-11T21:00:27.413 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:00:27.413 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:00:27.413 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-03-11T21:00:27.413 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:00:26.973666+0000", 2024-03-11T21:00:27.414 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2459999999999997e-06 2024-03-11T21:00:27.414 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:00:27.414 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:00:27.414 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:00:27.414 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:00:27.414 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:00:27.414 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.24281.0:2124 4.9 4.84ada7c9 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-03-11T21:00:27.414 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T21:00:26.952169+0000", 2024-03-11T21:00:27.414 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.463779936, 2024-03-11T21:00:27.414 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.46385810399999999, 2024-03-11T21:00:27.414 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:00:27.414 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "waiting for peered", 2024-03-11T21:00:27.414 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-03-11T21:00:27.415 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.24281", 2024-03-11T21:00:27.415 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.23:0/2185274774", 2024-03-11T21:00:27.415 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2124 2024-03-11T21:00:27.415 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:00:27.415 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:00:27.415 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:00:27.415 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:00:27.415 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:00:26.952169+0000", 2024-03-11T21:00:27.415 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:00:27.415 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:00:27.415 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:00:27.415 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:00:27.415 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:00:26.952169+0000", 2024-03-11T21:00:27.416 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:00:27.416 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:00:27.416 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:00:27.416 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:00:27.416 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:00:26.952170+0000", 2024-03-11T21:00:27.416 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.128e-06 2024-03-11T21:00:27.416 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:00:27.416 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:00:27.416 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:00:27.416 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:00:26.952177+0000", 2024-03-11T21:00:27.416 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.866e-06 2024-03-11T21:00:27.416 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:00:27.417 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:00:27.417 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:00:27.417 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:00:26.952191+0000", 2024-03-11T21:00:27.417 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3855e-05 2024-03-11T21:00:27.417 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:00:27.417 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:00:27.417 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:00:27.417 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:00:26.952218+0000", 2024-03-11T21:00:27.417 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7304000000000002e-05 2024-03-11T21:00:27.417 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:00:27.417 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:00:27.417 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:00:27.417 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:00:26.973759+0000", 2024-03-11T21:00:27.417 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.021540423 2024-03-11T21:00:27.417 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:00:27.417 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:00:27.418 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-03-11T21:00:27.418 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:00:26.973774+0000", 2024-03-11T21:00:27.418 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5639000000000001e-05 2024-03-11T21:00:27.418 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:00:27.418 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:00:27.418 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:00:27.418 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:00:27.418 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:00:27.418 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.24281.0:2125 4.e 4.f8c99aee (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-03-11T21:00:27.418 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T21:00:26.952234+0000", 2024-03-11T21:00:27.418 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.46371538400000001, 2024-03-11T21:00:27.418 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.46383464800000002, 2024-03-11T21:00:27.418 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:00:27.418 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "waiting for peered", 2024-03-11T21:00:27.418 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-03-11T21:00:27.419 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.24281", 2024-03-11T21:00:27.419 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.23:0/2185274774", 2024-03-11T21:00:27.419 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2125 2024-03-11T21:00:27.419 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:00:27.419 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:00:27.419 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:00:27.419 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:00:27.419 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:00:26.952234+0000", 2024-03-11T21:00:27.419 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:00:27.419 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:00:27.419 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:00:27.419 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:00:27.419 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:00:26.952234+0000", 2024-03-11T21:00:27.419 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:00:27.419 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:00:27.420 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:00:27.420 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:00:27.420 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:00:26.952235+0000", 2024-03-11T21:00:27.420 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8400000000000003e-07 2024-03-11T21:00:27.420 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:00:27.420 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:00:27.420 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:00:27.420 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:00:26.952239+0000", 2024-03-11T21:00:27.420 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6319999999999999e-06 2024-03-11T21:00:27.420 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:00:27.420 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:00:27.420 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:00:27.420 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:00:26.952252+0000", 2024-03-11T21:00:27.420 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3183999999999999e-05 2024-03-11T21:00:27.420 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:00:27.421 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:00:27.421 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:00:27.421 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:00:26.952263+0000", 2024-03-11T21:00:27.421 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0566e-05 2024-03-11T21:00:27.421 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:00:27.421 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:00:27.421 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:00:27.421 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:00:26.971482+0000", 2024-03-11T21:00:27.421 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.019219127999999999 2024-03-11T21:00:27.421 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:00:27.421 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:00:27.421 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-03-11T21:00:27.421 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:00:26.971512+0000", 2024-03-11T21:00:27.421 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.027e-05 2024-03-11T21:00:27.421 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:00:27.422 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:00:27.422 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:00:27.422 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:00:27.422 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-03-11T21:00:27.422 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 3 2024-03-11T21:00:27.422 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-03-11T21:00:27.422 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 57 16 212 196 0.214885 0 - 0.00475516 2024-03-11T21:00:27.488 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:27.512 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:27.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:27.514 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:00:27.614 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:27.641 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:27.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:27.644 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:00:27.682 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T21:00:27.745 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:27.771 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:27.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:27.774 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:00:27.875 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:27.903 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:27.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:27.905 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:00:27.949 INFO:teuthology.orchestra.run.smithi023.stderr:noscrub is set 2024-03-11T21:00:28.006 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:28.037 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:28.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:28.039 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:00:28.140 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:28.149 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:27 smithi023 bash[21372]: cluster 2024-03-11T21:00:26.883949+0000 osd.7 (osd.7) 207 : cluster 0 6.3 scrub starts 2024-03-11T21:00:28.149 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:27 smithi023 bash[21372]: cluster 2024-03-11T21:00:26.912404+0000 mon.a (mon.0) 1275 : cluster 3 Health check update: 1 osds down (OSD_DOWN) 2024-03-11T21:00:28.149 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:27 smithi023 bash[21372]: cluster 2024-03-11T21:00:26.925757+0000 osd.7 (osd.7) 208 : cluster 0 6.3 scrub ok 2024-03-11T21:00:28.149 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:27 smithi023 bash[21372]: cluster 2024-03-11T21:00:26.935582+0000 mon.a (mon.0) 1276 : cluster 1 osd.4 [v2:172.21.15.146:6800/3858359434,v1:172.21.15.146:6801/3858359434] boot 2024-03-11T21:00:28.149 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:27 smithi023 bash[21372]: cluster 2024-03-11T21:00:26.937931+0000 mon.a (mon.0) 1277 : cluster 0 osdmap e204: 8 total, 7 up, 8 in 2024-03-11T21:00:28.149 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:27 smithi023 bash[21372]: audit 2024-03-11T21:00:26.938538+0000 mon.a (mon.0) 1278 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T21:00:28.149 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:27 smithi023 bash[21372]: audit 2024-03-11T21:00:27.402628+0000 mon.c (mon.2) 48 : audit 1 from='client.? 172.21.15.23:0/3712054980' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:00:28.150 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:27 smithi023 bash[21372]: audit 2024-03-11T21:00:27.403319+0000 mon.a (mon.0) 1279 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:00:28.150 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:27 smithi023 bash[29146]: cluster 2024-03-11T21:00:26.883949+0000 osd.7 (osd.7) 207 : cluster 0 6.3 scrub starts 2024-03-11T21:00:28.150 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:27 smithi023 bash[29146]: cluster 2024-03-11T21:00:26.912404+0000 mon.a (mon.0) 1275 : cluster 3 Health check update: 1 osds down (OSD_DOWN) 2024-03-11T21:00:28.150 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:27 smithi023 bash[29146]: cluster 2024-03-11T21:00:26.925757+0000 osd.7 (osd.7) 208 : cluster 0 6.3 scrub ok 2024-03-11T21:00:28.150 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:27 smithi023 bash[29146]: cluster 2024-03-11T21:00:26.935582+0000 mon.a (mon.0) 1276 : cluster 1 osd.4 [v2:172.21.15.146:6800/3858359434,v1:172.21.15.146:6801/3858359434] boot 2024-03-11T21:00:28.150 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:27 smithi023 bash[29146]: cluster 2024-03-11T21:00:26.937931+0000 mon.a (mon.0) 1277 : cluster 0 osdmap e204: 8 total, 7 up, 8 in 2024-03-11T21:00:28.150 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:27 smithi023 bash[29146]: audit 2024-03-11T21:00:26.938538+0000 mon.a (mon.0) 1278 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T21:00:28.150 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:27 smithi023 bash[29146]: audit 2024-03-11T21:00:27.402628+0000 mon.c (mon.2) 48 : audit 1 from='client.? 172.21.15.23:0/3712054980' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:00:28.150 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:27 smithi023 bash[29146]: audit 2024-03-11T21:00:27.403319+0000 mon.a (mon.0) 1279 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:00:28.166 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:28.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:28.170 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:00:28.232 INFO:tasks.thrashosds.thrasher:Reviving osd 1 2024-03-11T21:00:28.232 INFO:tasks.cephadm.osd.1:Restarting osd.1 (starting--it wasn't running)... 2024-03-11T21:00:28.232 DEBUG:teuthology.orchestra.run.smithi023:osd.1> sudo journalctl -f -n 0 -u ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@osd.1.service 2024-03-11T21:00:28.233 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:27 smithi146 bash[23809]: cluster 2024-03-11T21:00:26.883949+0000 osd.7 (osd.7) 207 : cluster 0 6.3 scrub starts 2024-03-11T21:00:28.233 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:27 smithi146 bash[23809]: cluster 2024-03-11T21:00:26.912404+0000 mon.a (mon.0) 1275 : cluster 3 Health check update: 1 osds down (OSD_DOWN) 2024-03-11T21:00:28.233 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:27 smithi146 bash[23809]: cluster 2024-03-11T21:00:26.925757+0000 osd.7 (osd.7) 208 : cluster 0 6.3 scrub ok 2024-03-11T21:00:28.233 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:27 smithi146 bash[23809]: cluster 2024-03-11T21:00:26.935582+0000 mon.a (mon.0) 1276 : cluster 1 osd.4 [v2:172.21.15.146:6800/3858359434,v1:172.21.15.146:6801/3858359434] boot 2024-03-11T21:00:28.233 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:27 smithi146 bash[23809]: cluster 2024-03-11T21:00:26.937931+0000 mon.a (mon.0) 1277 : cluster 0 osdmap e204: 8 total, 7 up, 8 in 2024-03-11T21:00:28.233 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:27 smithi146 bash[23809]: audit 2024-03-11T21:00:26.938538+0000 mon.a (mon.0) 1278 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T21:00:28.233 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:27 smithi146 bash[23809]: audit 2024-03-11T21:00:27.402628+0000 mon.c (mon.2) 48 : audit 1 from='client.? 172.21.15.23:0/3712054980' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:00:28.233 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:27 smithi146 bash[23809]: audit 2024-03-11T21:00:27.403319+0000 mon.a (mon.0) 1279 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:00:28.236 DEBUG:teuthology.orchestra.run.smithi023:> sudo systemctl start ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@osd.1 2024-03-11T21:00:28.271 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:28.296 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-11T21:00:28.297 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:28.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:28.298 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:00:28.398 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:00:28.407 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:00:28 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:00:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:00:28.407 INFO:journalctl@ceph.osd.1.smithi023.stdout:Mar 11 21:00:28 smithi023 systemd[1]: Started Ceph osd.1 for 67ba00f8-dfe7-11ee-95c8-87774f69a715. 2024-03-11T21:00:28.411 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 58 16 1526 1510 1.62695 1.44079 0.00468873 0.614607 2024-03-11T21:00:28.432 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:00:28.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:28.435 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:00:28.536 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:28.560 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:28.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:28.562 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:00:28.663 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:28.688 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:28.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:28.691 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:00:28.791 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:00:28.822 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:00:28.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:28.824 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:00:28.925 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:00:28.957 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:00:28.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:28.960 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:00:29.061 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:00:29.071 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:28 smithi146 bash[23809]: cluster 2024-03-11T21:00:26.032008+0000 osd.4 (osd.4) 1 : cluster 0 purged_snaps scrub starts 2024-03-11T21:00:29.071 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:28 smithi146 bash[23809]: cluster 2024-03-11T21:00:26.032167+0000 osd.4 (osd.4) 2 : cluster 0 purged_snaps scrub ok 2024-03-11T21:00:29.071 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:28 smithi146 bash[23809]: cluster 2024-03-11T21:00:27.534063+0000 osd.6 (osd.6) 144 : cluster 0 8.6 scrub starts 2024-03-11T21:00:29.071 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:28 smithi146 bash[23809]: cluster 2024-03-11T21:00:27.537032+0000 osd.6 (osd.6) 145 : cluster 0 8.6 scrub ok 2024-03-11T21:00:29.071 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:28 smithi146 bash[23809]: cluster 2024-03-11T21:00:27.923711+0000 mon.a (mon.0) 1280 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:00:29.071 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:28 smithi146 bash[23809]: audit 2024-03-11T21:00:27.929586+0000 mon.a (mon.0) 1281 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:00:29.071 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:28 smithi146 bash[23809]: cluster 2024-03-11T21:00:27.951860+0000 mon.a (mon.0) 1282 : cluster 0 osdmap e205: 8 total, 7 up, 8 in 2024-03-11T21:00:29.072 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:28 smithi146 bash[23809]: cluster 2024-03-11T21:00:28.068912+0000 mgr.y (mgr.24380) 360 : cluster 0 pgmap v374: 148 pgs: 52 active+clean, 12 undersized+peered, 43 active+undersized, 12 undersized+degraded+peered, 29 active+undersized+degraded; 13 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 304/1182 objects degraded (25.719%) 2024-03-11T21:00:29.091 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:00:29.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:29.095 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:00:29.196 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:29.222 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:29.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:29.225 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:00:29.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:28 smithi023 bash[21372]: cluster 2024-03-11T21:00:26.032008+0000 osd.4 (osd.4) 1 : cluster 0 purged_snaps scrub starts 2024-03-11T21:00:29.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:28 smithi023 bash[21372]: cluster 2024-03-11T21:00:26.032167+0000 osd.4 (osd.4) 2 : cluster 0 purged_snaps scrub ok 2024-03-11T21:00:29.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:28 smithi023 bash[21372]: cluster 2024-03-11T21:00:27.534063+0000 osd.6 (osd.6) 144 : cluster 0 8.6 scrub starts 2024-03-11T21:00:29.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:28 smithi023 bash[21372]: cluster 2024-03-11T21:00:27.537032+0000 osd.6 (osd.6) 145 : cluster 0 8.6 scrub ok 2024-03-11T21:00:29.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:28 smithi023 bash[21372]: cluster 2024-03-11T21:00:27.923711+0000 mon.a (mon.0) 1280 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:00:29.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:28 smithi023 bash[21372]: audit 2024-03-11T21:00:27.929586+0000 mon.a (mon.0) 1281 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:00:29.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:28 smithi023 bash[21372]: cluster 2024-03-11T21:00:27.951860+0000 mon.a (mon.0) 1282 : cluster 0 osdmap e205: 8 total, 7 up, 8 in 2024-03-11T21:00:29.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:28 smithi023 bash[21372]: cluster 2024-03-11T21:00:28.068912+0000 mgr.y (mgr.24380) 360 : cluster 0 pgmap v374: 148 pgs: 52 active+clean, 12 undersized+peered, 43 active+undersized, 12 undersized+degraded+peered, 29 active+undersized+degraded; 13 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 304/1182 objects degraded (25.719%) 2024-03-11T21:00:29.286 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:28 smithi023 bash[29146]: cluster 2024-03-11T21:00:26.032008+0000 osd.4 (osd.4) 1 : cluster 0 purged_snaps scrub starts 2024-03-11T21:00:29.286 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:28 smithi023 bash[29146]: cluster 2024-03-11T21:00:26.032167+0000 osd.4 (osd.4) 2 : cluster 0 purged_snaps scrub ok 2024-03-11T21:00:29.286 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:28 smithi023 bash[29146]: cluster 2024-03-11T21:00:27.534063+0000 osd.6 (osd.6) 144 : cluster 0 8.6 scrub starts 2024-03-11T21:00:29.286 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:28 smithi023 bash[29146]: cluster 2024-03-11T21:00:27.537032+0000 osd.6 (osd.6) 145 : cluster 0 8.6 scrub ok 2024-03-11T21:00:29.286 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:28 smithi023 bash[29146]: cluster 2024-03-11T21:00:27.923711+0000 mon.a (mon.0) 1280 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:00:29.286 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:28 smithi023 bash[29146]: audit 2024-03-11T21:00:27.929586+0000 mon.a (mon.0) 1281 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:00:29.286 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:28 smithi023 bash[29146]: cluster 2024-03-11T21:00:27.951860+0000 mon.a (mon.0) 1282 : cluster 0 osdmap e205: 8 total, 7 up, 8 in 2024-03-11T21:00:29.286 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:28 smithi023 bash[29146]: cluster 2024-03-11T21:00:28.068912+0000 mgr.y (mgr.24380) 360 : cluster 0 pgmap v374: 148 pgs: 52 active+clean, 12 undersized+peered, 43 active+undersized, 12 undersized+degraded+peered, 29 active+undersized+degraded; 13 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 304/1182 objects degraded (25.719%) 2024-03-11T21:00:29.326 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:29.351 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:29.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:29.355 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:00:29.412 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 59 15 4765 4750 5.03115 202.5 0.00522115 0.198747 2024-03-11T21:00:29.456 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:29.480 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:29.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:29.485 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:00:29.585 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:29.611 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:29.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:29.614 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:00:29.714 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:29.743 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:29.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:29.746 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:00:29.847 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:00:29.876 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:00:29.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:29.879 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:00:29.980 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:30.004 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:30.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:30.006 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:00:30.107 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:30.133 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:30.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:30.138 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:00:30.239 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:30.248 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:30 smithi146 bash[23809]: cluster 2024-03-11T21:00:30.122432+0000 mon.a (mon.0) 1283 : cluster 3 Health check update: Reduced data availability: 16 pgs inactive (PG_AVAILABILITY) 2024-03-11T21:00:30.249 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:30 smithi146 bash[23809]: cluster 2024-03-11T21:00:30.122471+0000 mon.a (mon.0) 1284 : cluster 3 Health check update: Degraded data redundancy: 573/9153 objects degraded (6.260%), 34 pgs degraded, 72 pgs undersized (PG_DEGRADED) 2024-03-11T21:00:30.249 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:30 smithi146 bash[23809]: cluster 2024-03-11T21:00:30.145878+0000 mon.a (mon.0) 1285 : cluster 0 osdmap e206: 8 total, 7 up, 8 in 2024-03-11T21:00:30.264 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:30.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:30.267 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:00:30.368 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:00:30.390 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:00:30.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:30.394 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:00:30.412 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:2024-03-11T21:00:30.415805+0000 min lat: 0.00112042 max lat: 57.5681 avg lat: 0.125267 2024-03-11T21:00:30.412 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-11T21:00:30.412 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 60 15 7679 7664 7.98233 182.125 0.00345425 0.125267 2024-03-11T21:00:30.466 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:30 smithi023 bash[21372]: cluster 2024-03-11T21:00:30.122432+0000 mon.a (mon.0) 1283 : cluster 3 Health check update: Reduced data availability: 16 pgs inactive (PG_AVAILABILITY) 2024-03-11T21:00:30.466 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:30 smithi023 bash[21372]: cluster 2024-03-11T21:00:30.122471+0000 mon.a (mon.0) 1284 : cluster 3 Health check update: Degraded data redundancy: 573/9153 objects degraded (6.260%), 34 pgs degraded, 72 pgs undersized (PG_DEGRADED) 2024-03-11T21:00:30.466 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:30 smithi023 bash[21372]: cluster 2024-03-11T21:00:30.145878+0000 mon.a (mon.0) 1285 : cluster 0 osdmap e206: 8 total, 7 up, 8 in 2024-03-11T21:00:30.466 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:30 smithi023 bash[29146]: cluster 2024-03-11T21:00:30.122432+0000 mon.a (mon.0) 1283 : cluster 3 Health check update: Reduced data availability: 16 pgs inactive (PG_AVAILABILITY) 2024-03-11T21:00:30.467 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:30 smithi023 bash[29146]: cluster 2024-03-11T21:00:30.122471+0000 mon.a (mon.0) 1284 : cluster 3 Health check update: Degraded data redundancy: 573/9153 objects degraded (6.260%), 34 pgs degraded, 72 pgs undersized (PG_DEGRADED) 2024-03-11T21:00:30.467 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:30 smithi023 bash[29146]: cluster 2024-03-11T21:00:30.145878+0000 mon.a (mon.0) 1285 : cluster 0 osdmap e206: 8 total, 7 up, 8 in 2024-03-11T21:00:30.495 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:00:30.521 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:00:30.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:30.523 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:00:30.623 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:30.656 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:30.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:30.659 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:00:30.759 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:00:30.785 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:00:30.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:30.788 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:00:30.889 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:30.907 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:30.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:30.908 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:00:31.009 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:31.041 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:31.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:31.044 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:00:31.144 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:31.173 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:31.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:31.175 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:00:31.222 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:31 smithi023 bash[21372]: cluster 2024-03-11T21:00:30.070721+0000 mgr.y (mgr.24380) 361 : cluster 0 pgmap v375: 148 pgs: 68 active+clean, 9 undersized+peered, 37 active+undersized, 7 undersized+degraded+peered, 27 active+undersized+degraded; 179 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 28 MiB/s wr, 442 op/s; 573/9153 objects degraded (6.260%); 203 KiB/s, 3 objects/s recovering 2024-03-11T21:00:31.276 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:31.286 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:31 smithi146 bash[23809]: cluster 2024-03-11T21:00:30.070721+0000 mgr.y (mgr.24380) 361 : cluster 0 pgmap v375: 148 pgs: 68 active+clean, 9 undersized+peered, 37 active+undersized, 7 undersized+degraded+peered, 27 active+undersized+degraded; 179 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 28 MiB/s wr, 442 op/s; 573/9153 objects degraded (6.260%); 203 KiB/s, 3 objects/s recovering 2024-03-11T21:00:31.306 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:31.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:31.309 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:00:31.410 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:31.412 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 61 16 10367 10351 10.6042 167.938 0.00238203 0.0942936 2024-03-11T21:00:31.438 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:31.439 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:31 smithi023 bash[29146]: cluster 2024-03-11T21:00:30.070721+0000 mgr.y (mgr.24380) 361 : cluster 0 pgmap v375: 148 pgs: 68 active+clean, 9 undersized+peered, 37 active+undersized, 7 undersized+degraded+peered, 27 active+undersized+degraded; 179 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 28 MiB/s wr, 442 op/s; 573/9153 objects degraded (6.260%); 203 KiB/s, 3 objects/s recovering 2024-03-11T21:00:31.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:31.441 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:00:31.542 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:31.569 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:31.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:31.571 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:00:31.672 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:31.703 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:31.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:31.706 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:00:31.806 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:31.841 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:31.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:31.844 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:00:31.945 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:31.973 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:31.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:31.976 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:00:32.077 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:32.105 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:32.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:32.108 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:00:32.208 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:32.235 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:32.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:32.238 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:00:32.339 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:32.370 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:32.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:32.373 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:00:32.412 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 62 16 13261 13245 13.3502 180.875 0.00479803 0.0748985 2024-03-11T21:00:32.473 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:32.484 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:32 smithi023 bash[21372]: cluster 2024-03-11T21:00:32.176215+0000 mon.a (mon.0) 1286 : cluster 1 Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 16 pgs inactive) 2024-03-11T21:00:32.484 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:32 smithi023 bash[29146]: cluster 2024-03-11T21:00:32.176215+0000 mon.a (mon.0) 1286 : cluster 1 Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 16 pgs inactive) 2024-03-11T21:00:32.502 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:32.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:32.504 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:00:32.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:32 smithi146 bash[23809]: cluster 2024-03-11T21:00:32.176215+0000 mon.a (mon.0) 1286 : cluster 1 Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 16 pgs inactive) 2024-03-11T21:00:32.605 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:32.634 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:32.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:32.636 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:00:32.737 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:00:32.764 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:00:32.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:32.766 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:00:32.867 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:32.897 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:32.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:32.900 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:00:33.000 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:33.028 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:33.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:33.030 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:00:33.131 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:33.158 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:33.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:33.161 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:00:33.261 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:33.299 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:33.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:33.302 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:00:33.398 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-11T21:00:33.403 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:00:33.409 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:33 smithi023 bash[21372]: cluster 2024-03-11T21:00:32.071659+0000 mgr.y (mgr.24380) 362 : cluster 0 pgmap v377: 148 pgs: 88 active+clean, 34 active+undersized, 26 active+undersized+degraded; 449 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 73 MiB/s wr, 1.16k op/s; 3287/22128 objects degraded (14.854%); 565 KiB/s, 9 objects/s recovering 2024-03-11T21:00:33.410 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:33 smithi023 bash[29146]: cluster 2024-03-11T21:00:32.071659+0000 mgr.y (mgr.24380) 362 : cluster 0 pgmap v377: 148 pgs: 88 active+clean, 34 active+undersized, 26 active+undersized+degraded; 449 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 73 MiB/s wr, 1.16k op/s; 3287/22128 objects degraded (14.854%); 565 KiB/s, 9 objects/s recovering 2024-03-11T21:00:33.415 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 63 16 15891 15875 15.7471 164.375 0.00362635 0.0634948 2024-03-11T21:00:33.421 INFO:teuthology.orchestra.run.smithi023.stdout:ERROR: (22) Invalid argument 2024-03-11T21:00:33.421 INFO:teuthology.orchestra.run.smithi023.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:00:33.439 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:00:33.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:33.443 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:00:33.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:33 smithi146 bash[23809]: cluster 2024-03-11T21:00:32.071659+0000 mgr.y (mgr.24380) 362 : cluster 0 pgmap v377: 148 pgs: 88 active+clean, 34 active+undersized, 26 active+undersized+degraded; 449 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 73 MiB/s wr, 1.16k op/s; 3287/22128 objects degraded (14.854%); 565 KiB/s, 9 objects/s recovering 2024-03-11T21:00:33.543 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:00:33.574 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:00:33.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:33.577 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:00:33.677 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:33.711 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:33.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:33.714 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:00:33.815 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:33.840 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:33.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:33.843 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:00:33.944 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:00:33.968 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:00:33.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:33.971 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:00:34.072 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:34.095 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:34.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:34.098 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:00:34.198 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:34.225 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:34.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:34.228 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:00:34.285 INFO:journalctl@ceph.osd.1.smithi023.stdout:Mar 11 21:00:33 smithi023 bash[95309]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-11T21:00:34.285 INFO:journalctl@ceph.osd.1.smithi023.stdout:Mar 11 21:00:33 smithi023 bash[95309]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-03-11T21:00:34.285 INFO:journalctl@ceph.osd.1.smithi023.stdout:Mar 11 21:00:33 smithi023 bash[95309]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-03-11T21:00:34.285 INFO:journalctl@ceph.osd.1.smithi023.stdout:Mar 11 21:00:33 smithi023 bash[95309]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-03-11T21:00:34.285 INFO:journalctl@ceph.osd.1.smithi023.stdout:Mar 11 21:00:33 smithi023 bash[95309]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-03-11T21:00:34.285 INFO:journalctl@ceph.osd.1.smithi023.stdout:Mar 11 21:00:33 smithi023 bash[95309]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-11T21:00:34.285 INFO:journalctl@ceph.osd.1.smithi023.stdout:Mar 11 21:00:33 smithi023 bash[95309]: --> ceph-volume raw activate successful for osd ID: 1 2024-03-11T21:00:34.329 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:00:34.357 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:00:34.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:34.359 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:00:34.412 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 64 16 18362 18346 17.9138 154.438 0.00605848 0.0558152 2024-03-11T21:00:34.460 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:34.485 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:34.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:34.488 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:00:34.588 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:00:34.618 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:00:34.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:34.620 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:00:34.721 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:34.746 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:34.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:34.749 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:00:34.850 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:34.874 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:34.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:34.877 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:00:34.977 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:00:35.004 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:00:35.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:35.008 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:00:35.108 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:35.135 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:35.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:35.138 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:00:35.238 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:00:35.265 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:00:35.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:35.268 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:00:35.369 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:35.394 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:35.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:35.397 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:00:35.412 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 65 16 20978 20962 20.1533 163.5 0.00700199 0.0496143 2024-03-11T21:00:35.498 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:35.508 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:35 smithi146 bash[23809]: cluster 2024-03-11T21:00:34.072865+0000 mgr.y (mgr.24380) 363 : cluster 0 pgmap v378: 148 pgs: 88 active+clean, 34 active+undersized, 26 active+undersized+degraded; 449 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 429 B/s rd, 61 MiB/s wr, 977 op/s; 3287/22128 objects degraded (14.854%); 475 KiB/s, 8 objects/s recovering 2024-03-11T21:00:35.508 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:35 smithi146 bash[23809]: audit 2024-03-11T21:00:34.861405+0000 mon.a (mon.0) 1287 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:00:35.508 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:35 smithi146 bash[23809]: audit 2024-03-11T21:00:34.862685+0000 mon.a (mon.0) 1288 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:00:35.508 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:35 smithi146 bash[23809]: cluster 2024-03-11T21:00:35.141475+0000 mon.a (mon.0) 1289 : cluster 3 Health check update: Degraded data redundancy: 3287/22128 objects degraded (14.854%), 26 pgs degraded, 36 pgs undersized (PG_DEGRADED) 2024-03-11T21:00:35.508 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:35 smithi146 bash[23809]: cluster 2024-03-11T21:00:35.148486+0000 mon.a (mon.0) 1290 : cluster 0 osdmap e207: 8 total, 7 up, 8 in 2024-03-11T21:00:35.508 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:00:35 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:00:35.525 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:35.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:35.527 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:00:35.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:35 smithi023 bash[21372]: cluster 2024-03-11T21:00:34.072865+0000 mgr.y (mgr.24380) 363 : cluster 0 pgmap v378: 148 pgs: 88 active+clean, 34 active+undersized, 26 active+undersized+degraded; 449 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 429 B/s rd, 61 MiB/s wr, 977 op/s; 3287/22128 objects degraded (14.854%); 475 KiB/s, 8 objects/s recovering 2024-03-11T21:00:35.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:35 smithi023 bash[21372]: audit 2024-03-11T21:00:34.861405+0000 mon.a (mon.0) 1287 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:00:35.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:35 smithi023 bash[21372]: audit 2024-03-11T21:00:34.862685+0000 mon.a (mon.0) 1288 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:00:35.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:35 smithi023 bash[21372]: cluster 2024-03-11T21:00:35.141475+0000 mon.a (mon.0) 1289 : cluster 3 Health check update: Degraded data redundancy: 3287/22128 objects degraded (14.854%), 26 pgs degraded, 36 pgs undersized (PG_DEGRADED) 2024-03-11T21:00:35.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:35 smithi023 bash[21372]: cluster 2024-03-11T21:00:35.148486+0000 mon.a (mon.0) 1290 : cluster 0 osdmap e207: 8 total, 7 up, 8 in 2024-03-11T21:00:35.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:35 smithi023 bash[29146]: cluster 2024-03-11T21:00:34.072865+0000 mgr.y (mgr.24380) 363 : cluster 0 pgmap v378: 148 pgs: 88 active+clean, 34 active+undersized, 26 active+undersized+degraded; 449 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 429 B/s rd, 61 MiB/s wr, 977 op/s; 3287/22128 objects degraded (14.854%); 475 KiB/s, 8 objects/s recovering 2024-03-11T21:00:35.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:35 smithi023 bash[29146]: audit 2024-03-11T21:00:34.861405+0000 mon.a (mon.0) 1287 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:00:35.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:35 smithi023 bash[29146]: audit 2024-03-11T21:00:34.862685+0000 mon.a (mon.0) 1288 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:00:35.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:35 smithi023 bash[29146]: cluster 2024-03-11T21:00:35.141475+0000 mon.a (mon.0) 1289 : cluster 3 Health check update: Degraded data redundancy: 3287/22128 objects degraded (14.854%), 26 pgs degraded, 36 pgs undersized (PG_DEGRADED) 2024-03-11T21:00:35.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:35 smithi023 bash[29146]: cluster 2024-03-11T21:00:35.148486+0000 mon.a (mon.0) 1290 : cluster 0 osdmap e207: 8 total, 7 up, 8 in 2024-03-11T21:00:35.628 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:00:35.655 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:00:35.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:35.658 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:00:35.758 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:35.788 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:35.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:35.793 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:00:35.894 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:35.913 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:35.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:35.915 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:00:36.016 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:36.045 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:36.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:36.049 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:00:36.149 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:00:36.191 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:00:36.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:36.194 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:00:36.295 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:36.324 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:36.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:36.326 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:00:36.412 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 66 16 22918 22902 21.6849 121.25 0.0100252 0.0461094 2024-03-11T21:00:36.427 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:00:36.454 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:00:36.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:36.457 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:00:36.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:36 smithi146 bash[23809]: audit 2024-03-11T21:00:35.286313+0000 mgr.y (mgr.24380) 364 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:00:36.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:36 smithi023 bash[21372]: audit 2024-03-11T21:00:35.286313+0000 mgr.y (mgr.24380) 364 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:00:36.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:36 smithi023 bash[29146]: audit 2024-03-11T21:00:35.286313+0000 mgr.y (mgr.24380) 364 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:00:36.558 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:00:36.585 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:00:36.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:36.588 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:00:36.689 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:36.718 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:36.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:36.722 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:00:36.823 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:36.855 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:36.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:36.859 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:00:36.960 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:36.988 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:36.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:36.990 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:00:37.091 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:37.135 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:37.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:37.140 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:00:37.197 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-03-11T21:00:37.241 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:00:37.266 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:00:37.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:37.270 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:00:37.370 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:37.399 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:37.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:37.403 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:00:37.412 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 67 16 24836 24820 23.1502 119.875 0.0103086 0.0431902 2024-03-11T21:00:37.505 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:37.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:37 smithi146 bash[23809]: cluster 2024-03-11T21:00:36.074281+0000 mgr.y (mgr.24380) 365 : cluster 0 pgmap v380: 148 pgs: 88 active+clean, 34 active+undersized, 26 active+undersized+degraded; 1.3 GiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 3.6 KiB/s rd, 159 MiB/s wr, 2.56k op/s; 9130/62412 objects degraded (14.629%); 424 KiB/s, 7 objects/s recovering 2024-03-11T21:00:37.532 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:37.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:37.534 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:00:37.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:37 smithi023 bash[21372]: cluster 2024-03-11T21:00:36.074281+0000 mgr.y (mgr.24380) 365 : cluster 0 pgmap v380: 148 pgs: 88 active+clean, 34 active+undersized, 26 active+undersized+degraded; 1.3 GiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 3.6 KiB/s rd, 159 MiB/s wr, 2.56k op/s; 9130/62412 objects degraded (14.629%); 424 KiB/s, 7 objects/s recovering 2024-03-11T21:00:37.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:37 smithi023 bash[29146]: cluster 2024-03-11T21:00:36.074281+0000 mgr.y (mgr.24380) 365 : cluster 0 pgmap v380: 148 pgs: 88 active+clean, 34 active+undersized, 26 active+undersized+degraded; 1.3 GiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 3.6 KiB/s rd, 159 MiB/s wr, 2.56k op/s; 9130/62412 objects degraded (14.629%); 424 KiB/s, 7 objects/s recovering 2024-03-11T21:00:37.635 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:37.661 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:37.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:37.665 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:00:37.766 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:37.791 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:37.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:37.794 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:00:37.841 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:00:37.842 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:00:37.895 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:37.913 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:37.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:37.918 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:00:38.018 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:38.040 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:38.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:38.042 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:00:38.143 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:38.180 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:38.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:38.183 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:00:38.284 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:38.312 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:38.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:38.316 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:00:38.412 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 68 16 26673 26657 24.498 114.812 0.00472428 0.0408145 2024-03-11T21:00:38.417 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:00:38.426 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:00:38 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:00:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:00:38.443 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:00:38.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:38.446 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:00:38.547 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:38.565 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:38.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:38.567 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:00:38.668 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:38.690 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:38.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:38.695 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:00:38.795 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:00:38.826 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:00:38.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:38.829 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:00:38.930 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:00:38.954 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:00:38.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:38.956 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:00:39.057 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:39.074 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:39.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:39.076 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:00:39.177 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:39.203 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:39.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:39.209 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:00:39.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:39 smithi023 bash[21372]: cluster 2024-03-11T21:00:38.075708+0000 mgr.y (mgr.24380) 366 : cluster 0 pgmap v381: 148 pgs: 88 active+clean, 34 active+undersized, 26 active+undersized+degraded; 1.3 GiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 3.6 KiB/s rd, 139 MiB/s wr, 2.22k op/s; 9130/62412 objects degraded (14.629%); 272 KiB/s, 4 objects/s recovering 2024-03-11T21:00:39.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:39 smithi023 bash[29146]: cluster 2024-03-11T21:00:38.075708+0000 mgr.y (mgr.24380) 366 : cluster 0 pgmap v381: 148 pgs: 88 active+clean, 34 active+undersized, 26 active+undersized+degraded; 1.3 GiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 3.6 KiB/s rd, 139 MiB/s wr, 2.22k op/s; 9130/62412 objects degraded (14.629%); 272 KiB/s, 4 objects/s recovering 2024-03-11T21:00:39.309 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:39.337 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:39.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:39.341 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:00:39.412 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 69 16 28286 28270 25.6038 100.812 0.0050575 0.0390508 2024-03-11T21:00:39.443 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:00:39.463 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:00:39.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:39.465 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:00:39.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:39 smithi146 bash[23809]: cluster 2024-03-11T21:00:38.075708+0000 mgr.y (mgr.24380) 366 : cluster 0 pgmap v381: 148 pgs: 88 active+clean, 34 active+undersized, 26 active+undersized+degraded; 1.3 GiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 3.6 KiB/s rd, 139 MiB/s wr, 2.22k op/s; 9130/62412 objects degraded (14.629%); 272 KiB/s, 4 objects/s recovering 2024-03-11T21:00:39.566 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:39.591 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:39.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:39.594 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:00:39.695 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:00:39.722 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:00:39.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:39.724 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:00:39.825 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:39.852 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:39.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:39.854 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:00:39.955 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:00:39.975 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:00:39.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:39.977 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:00:40.077 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:00:40.112 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:00:40.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:40.115 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:00:40.216 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:40.246 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:40.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:40.248 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:00:40.349 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:40.375 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:40.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:40.381 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:00:40.413 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 70 15 30071 30056 26.8325 111.625 0.00600235 0.037262 2024-03-11T21:00:40.481 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:40.507 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:40.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:40.510 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:00:40.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:40 smithi023 bash[21372]: cluster 2024-03-11T21:00:40.143308+0000 mon.a (mon.0) 1291 : cluster 3 Health check update: Degraded data redundancy: 9130/62412 objects degraded (14.629%), 26 pgs degraded, 36 pgs undersized (PG_DEGRADED) 2024-03-11T21:00:40.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:40 smithi023 bash[21372]: cluster 2024-03-11T21:00:40.153529+0000 mon.a (mon.0) 1292 : cluster 0 osdmap e208: 8 total, 7 up, 8 in 2024-03-11T21:00:40.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:40 smithi023 bash[29146]: cluster 2024-03-11T21:00:40.143308+0000 mon.a (mon.0) 1291 : cluster 3 Health check update: Degraded data redundancy: 9130/62412 objects degraded (14.629%), 26 pgs degraded, 36 pgs undersized (PG_DEGRADED) 2024-03-11T21:00:40.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:40 smithi023 bash[29146]: cluster 2024-03-11T21:00:40.153529+0000 mon.a (mon.0) 1292 : cluster 0 osdmap e208: 8 total, 7 up, 8 in 2024-03-11T21:00:40.611 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:00:40.632 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:00:40.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:40.635 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:00:40.736 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:00:40.745 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:40 smithi146 bash[23809]: cluster 2024-03-11T21:00:40.143308+0000 mon.a (mon.0) 1291 : cluster 3 Health check update: Degraded data redundancy: 9130/62412 objects degraded (14.629%), 26 pgs degraded, 36 pgs undersized (PG_DEGRADED) 2024-03-11T21:00:40.745 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:40 smithi146 bash[23809]: cluster 2024-03-11T21:00:40.153529+0000 mon.a (mon.0) 1292 : cluster 0 osdmap e208: 8 total, 7 up, 8 in 2024-03-11T21:00:40.761 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:00:40.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:40.764 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:00:40.864 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:00:40.890 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:00:40.892 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:00:40.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:40.893 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:00:40.994 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:41.022 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:41.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:41.025 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:00:41.125 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:41.151 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:41.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:41.154 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:00:41.169 INFO:journalctl@ceph.osd.1.smithi023.stdout:Mar 11 21:00:41 smithi023 bash[95887]: debug 2024-03-11T21:00:41.006+0000 7fb877c08700 1 -- 172.21.15.23:0/1135364853 <== mon.2 v2:172.21.15.23:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x5587f33fbba0 con 0x5587f343c800 2024-03-11T21:00:41.169 INFO:journalctl@ceph.osd.1.smithi023.stdout:Mar 11 21:00:41 smithi023 bash[95887]: debug 2024-03-11T21:00:41.006+0000 7fb877c08700 1 -- 172.21.15.23:0/1135364853 <== mon.2 v2:172.21.15.23:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5587f3401680 con 0x5587f343c800 2024-03-11T21:00:41.255 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:41.275 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:41.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:41.277 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:00:41.377 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:00:41.424 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 71 16 31991 31975 28.1436 119.938 0.0061132 0.0355268 2024-03-11T21:00:41.450 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:00:41.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:41.452 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:00:41.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:41 smithi023 bash[21372]: cluster 2024-03-11T21:00:40.085539+0000 mgr.y (mgr.24380) 367 : cluster 0 pgmap v382: 148 pgs: 88 active+clean, 34 active+undersized, 26 active+undersized+degraded; 1.5 GiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 23 KiB/s rd, 138 MiB/s wr, 2.24k op/s; 9752/74805 objects degraded (13.037%); 219 KiB/s, 3 objects/s recovering 2024-03-11T21:00:41.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:41 smithi023 bash[29146]: cluster 2024-03-11T21:00:40.085539+0000 mgr.y (mgr.24380) 367 : cluster 0 pgmap v382: 148 pgs: 88 active+clean, 34 active+undersized, 26 active+undersized+degraded; 1.5 GiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 23 KiB/s rd, 138 MiB/s wr, 2.24k op/s; 9752/74805 objects degraded (13.037%); 219 KiB/s, 3 objects/s recovering 2024-03-11T21:00:41.553 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:41.561 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:41 smithi146 bash[23809]: cluster 2024-03-11T21:00:40.085539+0000 mgr.y (mgr.24380) 367 : cluster 0 pgmap v382: 148 pgs: 88 active+clean, 34 active+undersized, 26 active+undersized+degraded; 1.5 GiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 23 KiB/s rd, 138 MiB/s wr, 2.24k op/s; 9752/74805 objects degraded (13.037%); 219 KiB/s, 3 objects/s recovering 2024-03-11T21:00:41.580 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:41.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:41.582 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:00:41.683 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:00:41.722 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:00:41.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:41.725 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:00:41.826 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:00:41.836 INFO:journalctl@ceph.osd.1.smithi023.stdout:Mar 11 21:00:41 smithi023 bash[95887]: debug 2024-03-11T21:00:41.558+0000 7fb880da07c0 -1 Falling back to public interface 2024-03-11T21:00:41.855 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:00:41.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:41.858 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:00:41.910 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:00:41.959 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:00:41.991 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:00:41.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:41.994 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:00:42.094 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:42.121 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:42.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:42.125 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:00:42.226 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:00:42.250 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:00:42.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:42.252 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:00:42.353 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:42.379 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:42.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:42.382 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:00:42.413 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 72 16 34103 34087 29.5859 132 0.00348941 0.0337945 2024-03-11T21:00:42.483 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:42.504 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:42.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:42.508 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:00:42.608 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:42.631 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:42.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:42.634 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:00:42.735 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:42.752 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:42.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:42.754 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:00:42.855 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:42.881 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:42.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:42.884 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:00:42.985 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:43.009 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:43.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:43.012 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:00:43.112 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:00:43.138 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:00:43.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:43.140 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:00:43.241 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:43.265 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:43.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:43.271 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:00:43.372 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:00:43.395 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:00:43.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:43.397 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:00:43.413 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 73 16 35792 35776 30.6265 105.562 0.00237258 0.0325895 2024-03-11T21:00:43.497 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:00:43.506 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:43 smithi146 bash[23809]: cluster 2024-03-11T21:00:42.086691+0000 mgr.y (mgr.24380) 368 : cluster 0 pgmap v384: 148 pgs: 88 active+clean, 34 active+undersized, 26 active+undersized+degraded; 1.8 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 55 KiB/s rd, 177 MiB/s wr, 2.92k op/s; 13206/90204 objects degraded (14.640%) 2024-03-11T21:00:43.524 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:00:43.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:43.526 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:00:43.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:43 smithi023 bash[21372]: cluster 2024-03-11T21:00:42.086691+0000 mgr.y (mgr.24380) 368 : cluster 0 pgmap v384: 148 pgs: 88 active+clean, 34 active+undersized, 26 active+undersized+degraded; 1.8 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 55 KiB/s rd, 177 MiB/s wr, 2.92k op/s; 13206/90204 objects degraded (14.640%) 2024-03-11T21:00:43.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:43 smithi023 bash[29146]: cluster 2024-03-11T21:00:42.086691+0000 mgr.y (mgr.24380) 368 : cluster 0 pgmap v384: 148 pgs: 88 active+clean, 34 active+undersized, 26 active+undersized+degraded; 1.8 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 55 KiB/s rd, 177 MiB/s wr, 2.92k op/s; 13206/90204 objects degraded (14.640%) 2024-03-11T21:00:43.627 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:43.651 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:43.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:43.654 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:00:43.755 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:00:43.785 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:00:43.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:43.788 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:00:43.888 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:43.920 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:43.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:43.924 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:00:44.024 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:44.051 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:44.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:44.057 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:00:44.158 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:44.184 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:44.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:44.189 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:00:44.289 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:44.307 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:44.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:44.309 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:00:44.410 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:44.413 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 74 16 37303 37287 31.4887 94.4375 0.00198906 0.0316892 2024-03-11T21:00:44.440 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:44.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:44.443 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:00:44.544 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:00:44.570 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:00:44.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:44.573 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:00:44.674 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:00:44.702 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:00:44.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:44.705 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:00:44.806 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:44.823 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:44.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:44.825 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:00:44.926 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:44.949 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:44.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:44.951 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:00:45.052 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:45.081 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:45.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:45.087 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:00:45.188 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:45.214 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:45.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:45.216 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:00:45.317 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:45.355 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:45.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:45.359 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:00:45.413 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 75 16 38496 38480 32.0629 74.5625 0.00650142 0.031183 2024-03-11T21:00:45.459 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:00:45.486 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:00:45.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:45.491 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:00:45.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:45 smithi023 bash[21372]: cluster 2024-03-11T21:00:44.087539+0000 mgr.y (mgr.24380) 369 : cluster 0 pgmap v385: 148 pgs: 88 active+clean, 34 active+undersized, 26 active+undersized+degraded; 1.8 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 49 KiB/s rd, 117 MiB/s wr, 1.95k op/s; 13206/90204 objects degraded (14.640%) 2024-03-11T21:00:45.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:45 smithi023 bash[21372]: cluster 2024-03-11T21:00:45.145319+0000 mon.a (mon.0) 1293 : cluster 3 Health check update: Degraded data redundancy: 13206/90204 objects degraded (14.640%), 26 pgs degraded, 36 pgs undersized (PG_DEGRADED) 2024-03-11T21:00:45.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:45 smithi023 bash[21372]: cluster 2024-03-11T21:00:45.149486+0000 mon.a (mon.0) 1294 : cluster 0 osdmap e209: 8 total, 7 up, 8 in 2024-03-11T21:00:45.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:45 smithi023 bash[29146]: cluster 2024-03-11T21:00:44.087539+0000 mgr.y (mgr.24380) 369 : cluster 0 pgmap v385: 148 pgs: 88 active+clean, 34 active+undersized, 26 active+undersized+degraded; 1.8 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 49 KiB/s rd, 117 MiB/s wr, 1.95k op/s; 13206/90204 objects degraded (14.640%) 2024-03-11T21:00:45.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:45 smithi023 bash[29146]: cluster 2024-03-11T21:00:45.145319+0000 mon.a (mon.0) 1293 : cluster 3 Health check update: Degraded data redundancy: 13206/90204 objects degraded (14.640%), 26 pgs degraded, 36 pgs undersized (PG_DEGRADED) 2024-03-11T21:00:45.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:45 smithi023 bash[29146]: cluster 2024-03-11T21:00:45.149486+0000 mon.a (mon.0) 1294 : cluster 0 osdmap e209: 8 total, 7 up, 8 in 2024-03-11T21:00:45.591 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:45.619 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:45.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:45.623 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:00:45.724 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:45.755 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:45.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:45.757 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:00:45.762 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:00:45 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:00:45.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:45 smithi146 bash[23809]: cluster 2024-03-11T21:00:44.087539+0000 mgr.y (mgr.24380) 369 : cluster 0 pgmap v385: 148 pgs: 88 active+clean, 34 active+undersized, 26 active+undersized+degraded; 1.8 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 49 KiB/s rd, 117 MiB/s wr, 1.95k op/s; 13206/90204 objects degraded (14.640%) 2024-03-11T21:00:45.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:45 smithi146 bash[23809]: cluster 2024-03-11T21:00:45.145319+0000 mon.a (mon.0) 1293 : cluster 3 Health check update: Degraded data redundancy: 13206/90204 objects degraded (14.640%), 26 pgs degraded, 36 pgs undersized (PG_DEGRADED) 2024-03-11T21:00:45.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:45 smithi146 bash[23809]: cluster 2024-03-11T21:00:45.149486+0000 mon.a (mon.0) 1294 : cluster 0 osdmap e209: 8 total, 7 up, 8 in 2024-03-11T21:00:45.858 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:45.868 INFO:journalctl@ceph.osd.1.smithi023.stdout:Mar 11 21:00:45 smithi023 bash[95887]: debug 2024-03-11T21:00:45.786+0000 7fb880da07c0 -1 osd.1 173 log_to_monitors true 2024-03-11T21:00:45.883 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:45.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:45.885 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:00:45.986 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:46.010 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:46.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:46.012 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:00:46.113 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:46.137 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:46.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:46.139 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:00:46.240 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:46.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:46.285 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:46.286 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:00:46.386 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:00:46.413 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 76 16 39691 39675 32.6236 74.6875 0.0149758 0.0306474 2024-03-11T21:00:46.414 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:00:46.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:46.417 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:00:46.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:46 smithi146 bash[23809]: audit 2024-03-11T21:00:45.295654+0000 mgr.y (mgr.24380) 370 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:00:46.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:46 smithi146 bash[23809]: audit 2024-03-11T21:00:45.799972+0000 mon.a (mon.0) 1295 : audit 1 from='osd.1 [v2:172.21.15.23:6810/56585354,v1:172.21.15.23:6811/56585354]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-11T21:00:46.518 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:46.535 INFO:journalctl@ceph.osd.1.smithi023.stdout:Mar 11 21:00:46 smithi023 bash[95887]: debug 2024-03-11T21:00:46.406+0000 7fb8721f8700 -1 osd.1 173 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T21:00:46.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:46 smithi023 bash[21372]: audit 2024-03-11T21:00:45.295654+0000 mgr.y (mgr.24380) 370 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:00:46.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:46 smithi023 bash[21372]: audit 2024-03-11T21:00:45.799972+0000 mon.a (mon.0) 1295 : audit 1 from='osd.1 [v2:172.21.15.23:6810/56585354,v1:172.21.15.23:6811/56585354]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-11T21:00:46.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:46 smithi023 bash[29146]: audit 2024-03-11T21:00:45.295654+0000 mgr.y (mgr.24380) 370 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:00:46.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:46 smithi023 bash[29146]: audit 2024-03-11T21:00:45.799972+0000 mon.a (mon.0) 1295 : audit 1 from='osd.1 [v2:172.21.15.23:6810/56585354,v1:172.21.15.23:6811/56585354]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-11T21:00:46.549 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:46.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:46.552 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:00:46.653 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:46.674 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:46.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:46.676 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:00:46.777 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:00:46.820 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:00:46.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:46.824 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:00:46.924 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:46.952 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:46.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:46.955 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:00:47.056 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:47.074 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:47.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:47.076 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:00:47.177 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:00:47.198 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:00:47.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:47.200 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:00:47.301 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:47.326 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:47.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:47.328 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:00:47.413 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 77 16 40810 40794 33.1081 69.9375 0.00324489 0.0301224 2024-03-11T21:00:47.429 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:47.455 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:47.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:47.458 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:00:47.462 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:47 smithi023 bash[21372]: cluster 2024-03-11T21:00:46.088627+0000 mgr.y (mgr.24380) 371 : cluster 0 pgmap v387: 148 pgs: 88 active+clean, 34 active+undersized, 26 active+undersized+degraded; 2.3 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 52 KiB/s rd, 139 MiB/s wr, 2.31k op/s; 16928/115998 objects degraded (14.593%) 2024-03-11T21:00:47.462 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:47 smithi023 bash[21372]: audit 2024-03-11T21:00:46.324894+0000 mon.a (mon.0) 1296 : audit 1 from='osd.1 [v2:172.21.15.23:6810/56585354,v1:172.21.15.23:6811/56585354]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-11T21:00:47.462 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:47 smithi023 bash[21372]: cluster 2024-03-11T21:00:46.329332+0000 mon.a (mon.0) 1297 : cluster 0 osdmap e210: 8 total, 7 up, 8 in 2024-03-11T21:00:47.462 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:47 smithi023 bash[21372]: audit 2024-03-11T21:00:46.329966+0000 mon.a (mon.0) 1298 : audit 1 from='osd.1 [v2:172.21.15.23:6810/56585354,v1:172.21.15.23:6811/56585354]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2024-03-11T21:00:47.463 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:47 smithi023 bash[29146]: cluster 2024-03-11T21:00:46.088627+0000 mgr.y (mgr.24380) 371 : cluster 0 pgmap v387: 148 pgs: 88 active+clean, 34 active+undersized, 26 active+undersized+degraded; 2.3 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 52 KiB/s rd, 139 MiB/s wr, 2.31k op/s; 16928/115998 objects degraded (14.593%) 2024-03-11T21:00:47.463 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:47 smithi023 bash[29146]: audit 2024-03-11T21:00:46.324894+0000 mon.a (mon.0) 1296 : audit 1 from='osd.1 [v2:172.21.15.23:6810/56585354,v1:172.21.15.23:6811/56585354]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-11T21:00:47.463 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:47 smithi023 bash[29146]: cluster 2024-03-11T21:00:46.329332+0000 mon.a (mon.0) 1297 : cluster 0 osdmap e210: 8 total, 7 up, 8 in 2024-03-11T21:00:47.463 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:47 smithi023 bash[29146]: audit 2024-03-11T21:00:46.329966+0000 mon.a (mon.0) 1298 : audit 1 from='osd.1 [v2:172.21.15.23:6810/56585354,v1:172.21.15.23:6811/56585354]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2024-03-11T21:00:47.559 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:47.570 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:47 smithi146 bash[23809]: cluster 2024-03-11T21:00:46.088627+0000 mgr.y (mgr.24380) 371 : cluster 0 pgmap v387: 148 pgs: 88 active+clean, 34 active+undersized, 26 active+undersized+degraded; 2.3 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 52 KiB/s rd, 139 MiB/s wr, 2.31k op/s; 16928/115998 objects degraded (14.593%) 2024-03-11T21:00:47.570 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:47 smithi146 bash[23809]: audit 2024-03-11T21:00:46.324894+0000 mon.a (mon.0) 1296 : audit 1 from='osd.1 [v2:172.21.15.23:6810/56585354,v1:172.21.15.23:6811/56585354]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-11T21:00:47.570 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:47 smithi146 bash[23809]: cluster 2024-03-11T21:00:46.329332+0000 mon.a (mon.0) 1297 : cluster 0 osdmap e210: 8 total, 7 up, 8 in 2024-03-11T21:00:47.570 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:47 smithi146 bash[23809]: audit 2024-03-11T21:00:46.329966+0000 mon.a (mon.0) 1298 : audit 1 from='osd.1 [v2:172.21.15.23:6810/56585354,v1:172.21.15.23:6811/56585354]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2024-03-11T21:00:47.586 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:47.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:47.588 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:00:47.689 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:47.716 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:47.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:47.718 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:00:47.819 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:47.845 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:47.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:47.848 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:00:47.949 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:00:47.976 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:00:47.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:47.978 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:00:48.079 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:48.107 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:48.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:48.109 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:00:48.210 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:48.236 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:48.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:48.238 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:00:48.339 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:48.374 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:48.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:48.376 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:00:48.413 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 78 16 40810 40794 32.6837 0 - 0.0301224 2024-03-11T21:00:48.477 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:00:48.501 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:00:48.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:48.503 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:00:48.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:48 smithi023 bash[21372]: cluster 2024-03-11T21:00:47.325772+0000 mon.a (mon.0) 1299 : cluster 1 Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-11T21:00:48.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:48 smithi023 bash[21372]: cluster 2024-03-11T21:00:47.340055+0000 mon.a (mon.0) 1300 : cluster 1 osd.1 [v2:172.21.15.23:6810/56585354,v1:172.21.15.23:6811/56585354] boot 2024-03-11T21:00:48.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:48 smithi023 bash[21372]: cluster 2024-03-11T21:00:47.340206+0000 mon.a (mon.0) 1301 : cluster 0 osdmap e211: 8 total, 8 up, 8 in 2024-03-11T21:00:48.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:48 smithi023 bash[21372]: audit 2024-03-11T21:00:47.341888+0000 mon.a (mon.0) 1302 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:00:48.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:48 smithi023 bash[21372]: cluster 2024-03-11T21:00:48.340538+0000 mon.a (mon.0) 1303 : cluster 0 osdmap e212: 8 total, 8 up, 8 in 2024-03-11T21:00:48.535 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:00:48 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:00:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:00:48.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:48 smithi023 bash[29146]: cluster 2024-03-11T21:00:47.325772+0000 mon.a (mon.0) 1299 : cluster 1 Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-11T21:00:48.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:48 smithi023 bash[29146]: cluster 2024-03-11T21:00:47.340055+0000 mon.a (mon.0) 1300 : cluster 1 osd.1 [v2:172.21.15.23:6810/56585354,v1:172.21.15.23:6811/56585354] boot 2024-03-11T21:00:48.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:48 smithi023 bash[29146]: cluster 2024-03-11T21:00:47.340206+0000 mon.a (mon.0) 1301 : cluster 0 osdmap e211: 8 total, 8 up, 8 in 2024-03-11T21:00:48.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:48 smithi023 bash[29146]: audit 2024-03-11T21:00:47.341888+0000 mon.a (mon.0) 1302 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:00:48.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:48 smithi023 bash[29146]: cluster 2024-03-11T21:00:48.340538+0000 mon.a (mon.0) 1303 : cluster 0 osdmap e212: 8 total, 8 up, 8 in 2024-03-11T21:00:48.604 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:48.634 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:48.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:48.636 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:00:48.737 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:00:48.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:48 smithi146 bash[23809]: cluster 2024-03-11T21:00:47.325772+0000 mon.a (mon.0) 1299 : cluster 1 Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-11T21:00:48.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:48 smithi146 bash[23809]: cluster 2024-03-11T21:00:47.340055+0000 mon.a (mon.0) 1300 : cluster 1 osd.1 [v2:172.21.15.23:6810/56585354,v1:172.21.15.23:6811/56585354] boot 2024-03-11T21:00:48.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:48 smithi146 bash[23809]: cluster 2024-03-11T21:00:47.340206+0000 mon.a (mon.0) 1301 : cluster 0 osdmap e211: 8 total, 8 up, 8 in 2024-03-11T21:00:48.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:48 smithi146 bash[23809]: audit 2024-03-11T21:00:47.341888+0000 mon.a (mon.0) 1302 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:00:48.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:48 smithi146 bash[23809]: cluster 2024-03-11T21:00:48.340538+0000 mon.a (mon.0) 1303 : cluster 0 osdmap e212: 8 total, 8 up, 8 in 2024-03-11T21:00:48.765 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:00:48.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:48.767 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:00:48.868 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:00:48.897 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:00:48.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:48.899 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:00:49.000 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:49.027 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:49.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:49.029 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:00:49.130 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:49.157 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:49.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:49.158 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:00:49.259 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:49.286 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:49.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:49.288 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:00:49.389 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:49.400 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:49 smithi146 bash[23809]: cluster 2024-03-11T21:00:46.801424+0000 osd.1 (osd.1) 1 : cluster 0 purged_snaps scrub starts 2024-03-11T21:00:49.400 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:49 smithi146 bash[23809]: cluster 2024-03-11T21:00:46.801584+0000 osd.1 (osd.1) 2 : cluster 0 purged_snaps scrub ok 2024-03-11T21:00:49.400 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:49 smithi146 bash[23809]: cluster 2024-03-11T21:00:48.090062+0000 mgr.y (mgr.24380) 372 : cluster 0 pgmap v390: 148 pgs: 88 active+clean, 34 active+undersized, 26 active+undersized+degraded; 2.3 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 852 B/s rd, 90 MiB/s wr, 1.43k op/s; 16928/115998 objects degraded (14.593%) 2024-03-11T21:00:49.400 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:49 smithi146 bash[23809]: cluster 2024-03-11T21:00:49.341211+0000 mon.a (mon.0) 1304 : cluster 0 osdmap e213: 8 total, 8 up, 8 in 2024-03-11T21:00:49.413 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 79 16 40810 40794 32.27 0 - 0.0301224 2024-03-11T21:00:49.415 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:49.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:49.417 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:00:49.518 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:49.549 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:49.552 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:49 smithi023 bash[21372]: cluster 2024-03-11T21:00:46.801424+0000 osd.1 (osd.1) 1 : cluster 0 purged_snaps scrub starts 2024-03-11T21:00:49.552 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:49 smithi023 bash[21372]: cluster 2024-03-11T21:00:46.801584+0000 osd.1 (osd.1) 2 : cluster 0 purged_snaps scrub ok 2024-03-11T21:00:49.552 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:49 smithi023 bash[21372]: cluster 2024-03-11T21:00:48.090062+0000 mgr.y (mgr.24380) 372 : cluster 0 pgmap v390: 148 pgs: 88 active+clean, 34 active+undersized, 26 active+undersized+degraded; 2.3 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 852 B/s rd, 90 MiB/s wr, 1.43k op/s; 16928/115998 objects degraded (14.593%) 2024-03-11T21:00:49.552 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:49 smithi023 bash[21372]: cluster 2024-03-11T21:00:49.341211+0000 mon.a (mon.0) 1304 : cluster 0 osdmap e213: 8 total, 8 up, 8 in 2024-03-11T21:00:49.552 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:49 smithi023 bash[29146]: cluster 2024-03-11T21:00:46.801424+0000 osd.1 (osd.1) 1 : cluster 0 purged_snaps scrub starts 2024-03-11T21:00:49.553 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:49 smithi023 bash[29146]: cluster 2024-03-11T21:00:46.801584+0000 osd.1 (osd.1) 2 : cluster 0 purged_snaps scrub ok 2024-03-11T21:00:49.553 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:49 smithi023 bash[29146]: cluster 2024-03-11T21:00:48.090062+0000 mgr.y (mgr.24380) 372 : cluster 0 pgmap v390: 148 pgs: 88 active+clean, 34 active+undersized, 26 active+undersized+degraded; 2.3 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 852 B/s rd, 90 MiB/s wr, 1.43k op/s; 16928/115998 objects degraded (14.593%) 2024-03-11T21:00:49.553 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:49 smithi023 bash[29146]: cluster 2024-03-11T21:00:49.341211+0000 mon.a (mon.0) 1304 : cluster 0 osdmap e213: 8 total, 8 up, 8 in 2024-03-11T21:00:49.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:49.554 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:00:49.655 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:49.682 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:49.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:49.684 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:00:49.785 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:49.810 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:49.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:49.813 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:00:49.914 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:49.939 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:49.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:49.942 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:00:50.043 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:00:50.066 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:00:50.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:50.068 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:00:50.169 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:00:50.193 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:00:50.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:50.196 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:00:50.297 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:00:50.322 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:00:50.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:50.325 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:00:50.414 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:2024-03-11T21:00:50.417628+0000 min lat: 0.00112042 max lat: 57.5681 avg lat: 0.0307193 2024-03-11T21:00:50.414 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-11T21:00:50.414 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 80 16 41680 41664 32.5462 18.125 0.0379108 0.0307193 2024-03-11T21:00:50.425 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:50.445 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:50.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:50.447 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:00:50.548 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:50.570 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:50.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:50.573 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:00:50.673 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:50.700 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:50.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:50.704 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:00:50.805 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:00:50.825 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:00:50.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:50.828 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:00:50.929 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:50.955 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:50.958 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:50 smithi146 bash[23809]: audit 2024-03-11T21:00:49.889043+0000 mon.a (mon.0) 1305 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:00:50.958 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:50 smithi146 bash[23809]: audit 2024-03-11T21:00:49.896484+0000 mon.a (mon.0) 1306 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:00:50.959 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:50 smithi146 bash[23809]: cluster 2024-03-11T21:00:50.091160+0000 mgr.y (mgr.24380) 373 : cluster 0 pgmap v393: 148 pgs: 100 active+clean, 28 active+undersized, 1 active+recovery_wait+undersized+degraded+remapped, 19 active+undersized+degraded; 2.4 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 21 MiB/s wr, 328 op/s; 17070/119946 objects degraded (14.231%); 16 KiB/s, 5 objects/s recovering 2024-03-11T21:00:50.959 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:50 smithi146 bash[23809]: cluster 2024-03-11T21:00:50.146454+0000 mon.a (mon.0) 1307 : cluster 3 Health check update: Degraded data redundancy: 16928/115998 objects degraded (14.593%), 26 pgs degraded, 36 pgs undersized (PG_DEGRADED) 2024-03-11T21:00:50.959 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:50 smithi146 bash[23809]: audit 2024-03-11T21:00:50.538947+0000 mon.a (mon.0) 1308 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:00:50.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:50.960 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:00:51.015 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:50 smithi023 bash[21372]: audit 2024-03-11T21:00:49.889043+0000 mon.a (mon.0) 1305 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:00:51.015 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:50 smithi023 bash[21372]: audit 2024-03-11T21:00:49.896484+0000 mon.a (mon.0) 1306 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:00:51.015 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:50 smithi023 bash[21372]: cluster 2024-03-11T21:00:50.091160+0000 mgr.y (mgr.24380) 373 : cluster 0 pgmap v393: 148 pgs: 100 active+clean, 28 active+undersized, 1 active+recovery_wait+undersized+degraded+remapped, 19 active+undersized+degraded; 2.4 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 21 MiB/s wr, 328 op/s; 17070/119946 objects degraded (14.231%); 16 KiB/s, 5 objects/s recovering 2024-03-11T21:00:51.015 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:50 smithi023 bash[21372]: cluster 2024-03-11T21:00:50.146454+0000 mon.a (mon.0) 1307 : cluster 3 Health check update: Degraded data redundancy: 16928/115998 objects degraded (14.593%), 26 pgs degraded, 36 pgs undersized (PG_DEGRADED) 2024-03-11T21:00:51.015 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:50 smithi023 bash[21372]: audit 2024-03-11T21:00:50.538947+0000 mon.a (mon.0) 1308 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:00:51.015 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:50 smithi023 bash[29146]: audit 2024-03-11T21:00:49.889043+0000 mon.a (mon.0) 1305 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:00:51.015 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:50 smithi023 bash[29146]: audit 2024-03-11T21:00:49.896484+0000 mon.a (mon.0) 1306 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:00:51.016 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:50 smithi023 bash[29146]: cluster 2024-03-11T21:00:50.091160+0000 mgr.y (mgr.24380) 373 : cluster 0 pgmap v393: 148 pgs: 100 active+clean, 28 active+undersized, 1 active+recovery_wait+undersized+degraded+remapped, 19 active+undersized+degraded; 2.4 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 21 MiB/s wr, 328 op/s; 17070/119946 objects degraded (14.231%); 16 KiB/s, 5 objects/s recovering 2024-03-11T21:00:51.016 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:50 smithi023 bash[29146]: cluster 2024-03-11T21:00:50.146454+0000 mon.a (mon.0) 1307 : cluster 3 Health check update: Degraded data redundancy: 16928/115998 objects degraded (14.593%), 26 pgs degraded, 36 pgs undersized (PG_DEGRADED) 2024-03-11T21:00:51.016 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:50 smithi023 bash[29146]: audit 2024-03-11T21:00:50.538947+0000 mon.a (mon.0) 1308 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:00:51.060 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:51.090 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:51.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:51.093 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:00:51.194 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:51.261 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:51.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:51.265 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:00:51.366 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:51.386 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:51.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:51.391 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:00:51.414 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 81 16 42569 42553 32.8303 55.5625 0.0188377 0.0304515 2024-03-11T21:00:51.491 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:00:51.534 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:00:51.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:51.538 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:00:51.639 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:51.692 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:51.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:51.697 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:00:51.798 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:00:51.819 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:00:51.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:51.825 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:00:51.926 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:51.946 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:51.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:51.948 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:00:52.049 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:52.059 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:51 smithi146 bash[23809]: audit 2024-03-11T21:00:51.477475+0000 mon.a (mon.0) 1309 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:00:52.059 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:51 smithi146 bash[23809]: audit 2024-03-11T21:00:51.478908+0000 mon.a (mon.0) 1310 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:00:52.059 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:51 smithi146 bash[23809]: audit 2024-03-11T21:00:51.487916+0000 mon.a (mon.0) 1311 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:00:52.076 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:52.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:52.078 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:00:52.178 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:52.206 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:52.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:52.210 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:00:52.311 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:52.321 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:52 smithi023 bash[21372]: audit 2024-03-11T21:00:51.477475+0000 mon.a (mon.0) 1309 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:00:52.321 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:52 smithi023 bash[21372]: audit 2024-03-11T21:00:51.478908+0000 mon.a (mon.0) 1310 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:00:52.321 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:52 smithi023 bash[21372]: audit 2024-03-11T21:00:51.487916+0000 mon.a (mon.0) 1311 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:00:52.321 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:52 smithi023 bash[29146]: audit 2024-03-11T21:00:51.477475+0000 mon.a (mon.0) 1309 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:00:52.321 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:52 smithi023 bash[29146]: audit 2024-03-11T21:00:51.478908+0000 mon.a (mon.0) 1310 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:00:52.321 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:52 smithi023 bash[29146]: audit 2024-03-11T21:00:51.487916+0000 mon.a (mon.0) 1311 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:00:52.343 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:52.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:52.346 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:00:52.414 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 82 16 43341 43325 33.0183 48.25 0.0135632 0.0302787 2024-03-11T21:00:52.447 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:52.474 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:52.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:52.477 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:00:52.577 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:52.598 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:52.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:52.601 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:00:52.702 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:52.734 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:52.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:52.736 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:00:52.837 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:52.858 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:52.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:52.861 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:00:52.962 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:52.989 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:52.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:52.992 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:00:53.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:52 smithi023 bash[21372]: cluster 2024-03-11T21:00:52.092154+0000 mgr.y (mgr.24380) 374 : cluster 0 pgmap v394: 148 pgs: 141 active+clean, 3 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped; 2.5 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 710 B/s rd, 33 MiB/s wr, 535 op/s; 17742/125247 objects degraded (14.166%); 1.1 MiB/s, 27 objects/s recovering 2024-03-11T21:00:53.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:52 smithi023 bash[29146]: cluster 2024-03-11T21:00:52.092154+0000 mgr.y (mgr.24380) 374 : cluster 0 pgmap v394: 148 pgs: 141 active+clean, 3 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped; 2.5 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 710 B/s rd, 33 MiB/s wr, 535 op/s; 17742/125247 objects degraded (14.166%); 1.1 MiB/s, 27 objects/s recovering 2024-03-11T21:00:53.092 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:53.101 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:52 smithi146 bash[23809]: cluster 2024-03-11T21:00:52.092154+0000 mgr.y (mgr.24380) 374 : cluster 0 pgmap v394: 148 pgs: 141 active+clean, 3 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped; 2.5 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 710 B/s rd, 33 MiB/s wr, 535 op/s; 17742/125247 objects degraded (14.166%); 1.1 MiB/s, 27 objects/s recovering 2024-03-11T21:00:53.117 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:53.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:53.120 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:00:53.221 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:53.239 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:53.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:53.241 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:00:53.341 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:53.365 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:53.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:53.367 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:00:53.414 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 83 16 44199 44183 33.2665 53.625 0.00792609 0.0300509 2024-03-11T21:00:53.468 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:00:53.507 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:00:53.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:53.515 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:00:53.615 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:53.653 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:53.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:53.659 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:00:53.760 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:53.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:53.790 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:53.790 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:00:53.891 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:00:53.926 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:00:53.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:53.929 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:00:54.030 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:54.098 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:54.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:54.104 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:00:54.205 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:00:54.225 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:00:54.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:54.228 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:00:54.328 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:54.355 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:54.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:54.358 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:00:54.414 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 84 16 45123 45107 33.5579 57.75 0.00694415 0.0297899 2024-03-11T21:00:54.459 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:54.486 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:54.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:54.490 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:00:54.591 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:54.616 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:54.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:54.620 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:00:54.721 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:54.746 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:54.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:54.748 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:00:54.849 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:54.869 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:54.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:54.871 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:00:54.972 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:55.002 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:55.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:55.005 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:00:55.106 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:00:55.125 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:00:55.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:55.127 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:00:55.228 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:00:55.240 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:55 smithi023 bash[21372]: cluster 2024-03-11T21:00:54.092631+0000 mgr.y (mgr.24380) 375 : cluster 0 pgmap v395: 148 pgs: 141 active+clean, 3 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped; 2.5 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 605 B/s rd, 29 MiB/s wr, 456 op/s; 17742/125247 objects degraded (14.166%); 994 KiB/s, 23 objects/s recovering 2024-03-11T21:00:55.240 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:55 smithi023 bash[29146]: cluster 2024-03-11T21:00:54.092631+0000 mgr.y (mgr.24380) 375 : cluster 0 pgmap v395: 148 pgs: 141 active+clean, 3 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped; 2.5 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 605 B/s rd, 29 MiB/s wr, 456 op/s; 17742/125247 objects degraded (14.166%); 994 KiB/s, 23 objects/s recovering 2024-03-11T21:00:55.266 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:00:55.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:55.270 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:00:55.371 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:00:55.379 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:00:55 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:00:55.379 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:55 smithi146 bash[23809]: cluster 2024-03-11T21:00:54.092631+0000 mgr.y (mgr.24380) 375 : cluster 0 pgmap v395: 148 pgs: 141 active+clean, 3 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped; 2.5 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 605 B/s rd, 29 MiB/s wr, 456 op/s; 17742/125247 objects degraded (14.166%); 994 KiB/s, 23 objects/s recovering 2024-03-11T21:00:55.391 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:00:55.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:55.393 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:00:55.414 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 85 16 46083 46067 33.8689 60 0.00316301 0.0295073 2024-03-11T21:00:55.494 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:55.523 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:55.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:55.526 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:00:55.627 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:00:55.651 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:00:55.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:55.653 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:00:55.754 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:00:55.779 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:00:55.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:55.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:00:55.882 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:55.904 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:55.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:55.907 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:00:56.007 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:56.030 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:56.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:56.032 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:00:56.084 INFO:teuthology.orchestra.run.smithi023.stdout:osd.0: {} 2024-03-11T21:00:56.132 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:00:56.135 INFO:teuthology.orchestra.run.smithi023.stdout:osd.1: {} 2024-03-11T21:00:56.172 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:00:56.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:56.176 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:00:56.179 INFO:teuthology.orchestra.run.smithi023.stdout:osd.2: {} 2024-03-11T21:00:56.220 INFO:teuthology.orchestra.run.smithi023.stdout:osd.3: {} 2024-03-11T21:00:56.252 INFO:teuthology.orchestra.run.smithi023.stdout:osd.4: {} 2024-03-11T21:00:56.276 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:56.287 INFO:teuthology.orchestra.run.smithi023.stdout:osd.5: {} 2024-03-11T21:00:56.302 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:56.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:56.306 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:00:56.327 INFO:teuthology.orchestra.run.smithi023.stdout:osd.6: {} 2024-03-11T21:00:56.372 INFO:teuthology.orchestra.run.smithi023.stdout:osd.7: {} 2024-03-11T21:00:56.386 INFO:teuthology.orchestra.run.smithi023.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:00:56.386 INFO:teuthology.orchestra.run.smithi023.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:00:56.386 INFO:teuthology.orchestra.run.smithi023.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:00:56.386 INFO:teuthology.orchestra.run.smithi023.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:00:56.387 INFO:teuthology.orchestra.run.smithi023.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:00:56.387 INFO:teuthology.orchestra.run.smithi023.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:00:56.387 INFO:teuthology.orchestra.run.smithi023.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:00:56.387 INFO:teuthology.orchestra.run.smithi023.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:00:56.401 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:56 smithi023 bash[21372]: cluster 2024-03-11T21:00:55.147426+0000 mon.a (mon.0) 1312 : cluster 3 Health check update: Degraded data redundancy: 17742/125247 objects degraded (14.166%), 7 pgs degraded (PG_DEGRADED) 2024-03-11T21:00:56.401 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:56 smithi023 bash[21372]: audit 2024-03-11T21:00:55.307416+0000 mgr.y (mgr.24380) 376 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:00:56.401 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:56 smithi023 bash[21372]: audit 2024-03-11T21:00:55.915821+0000 mon.c (mon.2) 49 : audit 0 from='client.? 172.21.15.23:0/3110379031' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:00:56.401 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:56 smithi023 bash[29146]: cluster 2024-03-11T21:00:55.147426+0000 mon.a (mon.0) 1312 : cluster 3 Health check update: Degraded data redundancy: 17742/125247 objects degraded (14.166%), 7 pgs degraded (PG_DEGRADED) 2024-03-11T21:00:56.402 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:56 smithi023 bash[29146]: audit 2024-03-11T21:00:55.307416+0000 mgr.y (mgr.24380) 376 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:00:56.402 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:56 smithi023 bash[29146]: audit 2024-03-11T21:00:55.915821+0000 mon.c (mon.2) 49 : audit 0 from='client.? 172.21.15.23:0/3110379031' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:00:56.406 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:00:56.414 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 86 16 46886 46870 34.0586 50.1875 0.0253317 0.0293539 2024-03-11T21:00:56.443 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:00:56.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:56.446 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:00:56.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:56 smithi146 bash[23809]: cluster 2024-03-11T21:00:55.147426+0000 mon.a (mon.0) 1312 : cluster 3 Health check update: Degraded data redundancy: 17742/125247 objects degraded (14.166%), 7 pgs degraded (PG_DEGRADED) 2024-03-11T21:00:56.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:56 smithi146 bash[23809]: audit 2024-03-11T21:00:55.307416+0000 mgr.y (mgr.24380) 376 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:00:56.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:56 smithi146 bash[23809]: audit 2024-03-11T21:00:55.915821+0000 mon.c (mon.2) 49 : audit 0 from='client.? 172.21.15.23:0/3110379031' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:00:56.547 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:56.580 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:56.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:56.584 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:00:56.684 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:56.702 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:56.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:56.704 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:00:56.805 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:56.839 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:56.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:56.841 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:00:56.942 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:56.970 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:56.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:56.974 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:00:57.074 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:00:57.097 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:00:57.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:57.099 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:00:57.200 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:57.209 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:57 smithi146 bash[23809]: cluster 2024-03-11T21:00:56.093587+0000 mgr.y (mgr.24380) 377 : cluster 0 pgmap v396: 148 pgs: 141 active+clean, 3 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped; 2.8 GiB data, 8.7 GiB used, 706 GiB / 715 GiB avail; 1.1 KiB/s rd, 59 MiB/s wr, 941 op/s; 17236/138588 objects degraded (12.437%); 4.8 MiB/s, 83 objects/s recovering 2024-03-11T21:00:57.223 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:57.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:57.225 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:00:57.326 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:00:57.348 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:00:57.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:57.352 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:00:57.414 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 87 16 47681 47665 34.2382 49.6875 0.00393962 0.0292001 2024-03-11T21:00:57.453 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:57.478 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:57.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:57.482 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:00:57.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:57 smithi023 bash[29146]: cluster 2024-03-11T21:00:56.093587+0000 mgr.y (mgr.24380) 377 : cluster 0 pgmap v396: 148 pgs: 141 active+clean, 3 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped; 2.8 GiB data, 8.7 GiB used, 706 GiB / 715 GiB avail; 1.1 KiB/s rd, 59 MiB/s wr, 941 op/s; 17236/138588 objects degraded (12.437%); 4.8 MiB/s, 83 objects/s recovering 2024-03-11T21:00:57.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:57 smithi023 bash[21372]: cluster 2024-03-11T21:00:56.093587+0000 mgr.y (mgr.24380) 377 : cluster 0 pgmap v396: 148 pgs: 141 active+clean, 3 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped; 2.8 GiB data, 8.7 GiB used, 706 GiB / 715 GiB avail; 1.1 KiB/s rd, 59 MiB/s wr, 941 op/s; 17236/138588 objects degraded (12.437%); 4.8 MiB/s, 83 objects/s recovering 2024-03-11T21:00:57.582 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:00:57.612 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:00:57.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:57.616 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:00:57.716 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:57.740 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:57.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:57.742 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:00:57.842 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:57.864 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:57.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:57.866 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:00:57.967 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:57.988 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:57.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:57.990 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:00:58.091 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:58.118 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:58.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:58.121 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:00:58.222 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:58.245 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:58.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:58.251 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:00:58.300 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-11T21:00:58.300 INFO:teuthology.orchestra.run.smithi023.stdout: "ops": [ 2024-03-11T21:00:58.300 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:00:58.300 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_repop(client.14910.0:48500 8.f e213/212 8:f8d7442c:::benchmark_data_smithi023_89854_object48499:head v 213'3109, mlcod=0'0)", 2024-03-11T21:00:58.300 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T21:00:58.299487+0000", 2024-03-11T21:00:58.301 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 0.0033987819999999999, 2024-03-11T21:00:58.301 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00343794, 2024-03-11T21:00:58.301 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:00:58.301 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:00:58.301 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:00:58.301 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:00:58.301 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:00:58.301 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:00:58.299487+0000", 2024-03-11T21:00:58.301 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:00:58.301 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:00:58.301 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:00:58.301 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:00:58.301 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:00:58.299487+0000", 2024-03-11T21:00:58.301 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:00:58.302 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:00:58.302 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:00:58.302 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:00:58.302 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:00:58.299488+0000", 2024-03-11T21:00:58.302 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 8.1299999999999999e-07 2024-03-11T21:00:58.302 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:00:58.302 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:00:58.302 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:00:58.302 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:00:58.299514+0000", 2024-03-11T21:00:58.302 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.6271999999999999e-05 2024-03-11T21:00:58.302 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:00:58.302 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:00:58.302 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:00:58.302 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:00:58.299528+0000", 2024-03-11T21:00:58.302 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.3382e-05 2024-03-11T21:00:58.302 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:00:58.303 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:00:58.303 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:00:58.303 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:00:58.299535+0000", 2024-03-11T21:00:58.303 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.2089999999999999e-06 2024-03-11T21:00:58.303 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:00:58.303 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:00:58.303 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:00:58.303 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:00:58.299713+0000", 2024-03-11T21:00:58.303 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00017809799999999999 2024-03-11T21:00:58.303 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:00:58.303 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:00:58.303 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:00:58.303 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:00:58.299736+0000", 2024-03-11T21:00:58.303 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.2867999999999999e-05 2024-03-11T21:00:58.303 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:00:58.303 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:00:58.304 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T21:00:58.304 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:00:58.302903+0000", 2024-03-11T21:00:58.304 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0031673339999999999 2024-03-11T21:00:58.304 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:00:58.304 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:00:58.304 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:00:58.304 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:00:58.304 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:00:58.304 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_repop(client.14910.0:48501 8.1 e213/212 8:86f92144:::benchmark_data_smithi023_89854_object48500:head v 213'3110, mlcod=193'1)", 2024-03-11T21:00:58.304 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T21:00:58.299577+0000", 2024-03-11T21:00:58.304 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 0.0033092159999999998, 2024-03-11T21:00:58.304 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0034329970000000001, 2024-03-11T21:00:58.304 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:00:58.304 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:00:58.305 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:00:58.305 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:00:58.305 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:00:58.305 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:00:58.299577+0000", 2024-03-11T21:00:58.305 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:00:58.305 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:00:58.305 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:00:58.305 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:00:58.305 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:00:58.299577+0000", 2024-03-11T21:00:58.305 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:00:58.305 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:00:58.305 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:00:58.305 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:00:58.305 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:00:58.299577+0000", 2024-03-11T21:00:58.305 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 4.6699999999999999e-07 2024-03-11T21:00:58.306 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:00:58.306 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:00:58.306 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:00:58.306 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:00:58.299600+0000", 2024-03-11T21:00:58.306 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.2425999999999998e-05 2024-03-11T21:00:58.306 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:00:58.306 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:00:58.306 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:00:58.306 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:00:58.299607+0000", 2024-03-11T21:00:58.306 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.644e-06 2024-03-11T21:00:58.306 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:00:58.306 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:00:58.306 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:00:58.307 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:00:58.299612+0000", 2024-03-11T21:00:58.307 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 5.0599999999999998e-06 2024-03-11T21:00:58.307 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:00:58.307 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:00:58.307 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:00:58.307 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:00:58.299663+0000", 2024-03-11T21:00:58.307 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 5.0695999999999997e-05 2024-03-11T21:00:58.307 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:00:58.307 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:00:58.307 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:00:58.307 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:00:58.299686+0000", 2024-03-11T21:00:58.307 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.3019000000000001e-05 2024-03-11T21:00:58.307 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:00:58.307 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:00:58.307 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T21:00:58.307 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:00:58.302848+0000", 2024-03-11T21:00:58.308 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0031620039999999999 2024-03-11T21:00:58.308 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:00:58.308 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:00:58.308 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:00:58.308 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:00:58.308 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-11T21:00:58.308 INFO:teuthology.orchestra.run.smithi023.stdout: "num_ops": 2 2024-03-11T21:00:58.308 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-11T21:00:58.351 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:58.370 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:58.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:58.373 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:00:58.414 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 88 16 48614 48598 34.5117 58.3125 0.00386307 0.0289685 2024-03-11T21:00:58.474 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:58.491 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:58.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:58.493 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:00:58.594 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:58.619 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:58.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:58.621 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:00:58.647 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:00:58 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:00:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:00:58.722 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:58.755 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:58.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:58.758 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:00:58.859 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:58.883 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:58.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:58.888 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:00:58.989 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:59.019 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:59.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:59.023 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:00:59.124 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:59.143 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:00:59.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:59.145 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:00:59.246 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:59.272 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:00:59.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:59.274 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:00:59.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:00:59 smithi023 bash[21372]: cluster 2024-03-11T21:00:58.094089+0000 mgr.y (mgr.24380) 378 : cluster 0 pgmap v397: 148 pgs: 141 active+clean, 3 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped; 2.8 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 944 B/s rd, 48 MiB/s wr, 772 op/s; 17236/138588 objects degraded (12.437%); 3.9 MiB/s, 68 objects/s recovering 2024-03-11T21:00:59.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:00:59 smithi023 bash[29146]: cluster 2024-03-11T21:00:58.094089+0000 mgr.y (mgr.24380) 378 : cluster 0 pgmap v397: 148 pgs: 141 active+clean, 3 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped; 2.8 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 944 B/s rd, 48 MiB/s wr, 772 op/s; 17236/138588 objects degraded (12.437%); 3.9 MiB/s, 68 objects/s recovering 2024-03-11T21:00:59.374 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:59.382 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:00:59 smithi146 bash[23809]: cluster 2024-03-11T21:00:58.094089+0000 mgr.y (mgr.24380) 378 : cluster 0 pgmap v397: 148 pgs: 141 active+clean, 3 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped; 2.8 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 944 B/s rd, 48 MiB/s wr, 772 op/s; 17236/138588 objects degraded (12.437%); 3.9 MiB/s, 68 objects/s recovering 2024-03-11T21:00:59.394 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:59.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:59.396 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:00:59.415 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 89 16 49258 49242 34.5761 40.25 0.00681152 0.0289119 2024-03-11T21:00:59.497 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:59.528 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:00:59.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:59.533 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:00:59.634 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:59.659 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:00:59.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:59.662 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:00:59.763 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:59.799 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:00:59.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:59.804 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:00:59.905 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:00:59.926 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:00:59.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:00:59.928 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:01:00.029 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:01:00.047 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:01:00.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:00.049 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:01:00.150 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:00.171 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:00.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:00.174 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:01:00.275 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:01:00.306 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:01:00.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:00.312 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:01:00.413 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:01:00.415 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 90 9 50082 50073 34.769 51.9375 0.00969625 0.0287543 2024-03-11T21:01:00.436 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:01:00.439 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:00 smithi023 bash[21372]: cluster 2024-03-11T21:01:00.148375+0000 mon.a (mon.0) 1313 : cluster 3 Health check update: Degraded data redundancy: 17236/138588 objects degraded (12.437%), 7 pgs degraded (PG_DEGRADED) 2024-03-11T21:01:00.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:00.440 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:01:00.440 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:00 smithi023 bash[29146]: cluster 2024-03-11T21:01:00.148375+0000 mon.a (mon.0) 1313 : cluster 3 Health check update: Degraded data redundancy: 17236/138588 objects degraded (12.437%), 7 pgs degraded (PG_DEGRADED) 2024-03-11T21:01:00.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:00 smithi146 bash[23809]: cluster 2024-03-11T21:01:00.148375+0000 mon.a (mon.0) 1313 : cluster 3 Health check update: Degraded data redundancy: 17236/138588 objects degraded (12.437%), 7 pgs degraded (PG_DEGRADED) 2024-03-11T21:01:00.541 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:01:00.562 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:01:00.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:00.565 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:01:00.666 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:00.693 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:00.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:00.695 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:01:00.795 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:01:00.821 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:01:00.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:00.823 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:01:00.924 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:00.951 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:00.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:00.953 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:01:01.053 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:01.078 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:01.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:01.080 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:01:01.181 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:01.206 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:01.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:01.208 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:01:01.309 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:01:01.334 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:01:01.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:01.337 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:01:01.415 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:Total time run: 90.0226 2024-03-11T21:01:01.415 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:Total writes made: 50082 2024-03-11T21:01:01.415 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:Write size: 65536 2024-03-11T21:01:01.415 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:Object size: 65536 2024-03-11T21:01:01.415 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:Bandwidth (MB/sec): 34.7704 2024-03-11T21:01:01.415 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:Stddev Bandwidth: 55.3885 2024-03-11T21:01:01.415 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:Max bandwidth (MB/sec): 202.5 2024-03-11T21:01:01.415 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:Min bandwidth (MB/sec): 0 2024-03-11T21:01:01.415 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:Average IOPS: 556 2024-03-11T21:01:01.415 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:Stddev IOPS: 886.217 2024-03-11T21:01:01.416 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:Max IOPS: 3240 2024-03-11T21:01:01.416 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:Min IOPS: 0 2024-03-11T21:01:01.416 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:Average Latency(s): 0.0287531 2024-03-11T21:01:01.416 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:Stddev Latency(s): 1.02871 2024-03-11T21:01:01.416 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:Max latency(s): 57.5681 2024-03-11T21:01:01.416 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:Min latency(s): 0.00112042 2024-03-11T21:01:01.424 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:Cleaning up (deleting benchmark objects) 2024-03-11T21:01:01.438 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:01:01.452 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:01 smithi146 bash[23809]: cluster 2024-03-11T21:01:00.094937+0000 mgr.y (mgr.24380) 379 : cluster 0 pgmap v398: 148 pgs: 141 active+clean, 3 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped; 2.9 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 856 B/s rd, 54 MiB/s wr, 865 op/s; 17236/143892 objects degraded (11.978%); 3.6 MiB/s, 62 objects/s recovering 2024-03-11T21:01:01.471 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:01:01.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:01.474 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:01:01.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:01 smithi023 bash[21372]: cluster 2024-03-11T21:01:00.094937+0000 mgr.y (mgr.24380) 379 : cluster 0 pgmap v398: 148 pgs: 141 active+clean, 3 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped; 2.9 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 856 B/s rd, 54 MiB/s wr, 865 op/s; 17236/143892 objects degraded (11.978%); 3.6 MiB/s, 62 objects/s recovering 2024-03-11T21:01:01.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:01 smithi023 bash[29146]: cluster 2024-03-11T21:01:00.094937+0000 mgr.y (mgr.24380) 379 : cluster 0 pgmap v398: 148 pgs: 141 active+clean, 3 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped; 2.9 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 856 B/s rd, 54 MiB/s wr, 865 op/s; 17236/143892 objects degraded (11.978%); 3.6 MiB/s, 62 objects/s recovering 2024-03-11T21:01:01.574 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:01:01.602 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:01:01.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:01.607 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:01:01.708 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:01:01.734 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:01:01.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:01.738 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:01:01.838 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:01:01.861 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:01:01.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:01.989 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:01:02.090 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:01:02.117 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:01:02.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:02.122 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:01:02.223 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:01:02.264 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:01:02.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:02.270 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:01:02.371 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:02.390 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:02.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:02.391 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:01:02.492 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:02.514 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:02.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:02.516 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:01:02.617 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:01:02.641 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:01:02.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:02.644 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:01:02.745 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:01:02.786 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:01:02.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:02.791 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:01:02.892 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:01:02.913 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:01:02.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:02.915 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:01:03.016 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:01:03.041 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:01:03.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:03.045 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:01:03.146 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:03.173 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:03.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:03.177 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:01:03.278 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:01:03.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:03 smithi023 bash[21372]: cluster 2024-03-11T21:01:02.095973+0000 mgr.y (mgr.24380) 380 : cluster 0 pgmap v399: 148 pgs: 141 active+clean, 3 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped; 3.0 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 53 MiB/s wr, 845 op/s; 16671/150360 objects degraded (11.087%); 6.1 MiB/s, 100 objects/s recovering 2024-03-11T21:01:03.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:03 smithi023 bash[29146]: cluster 2024-03-11T21:01:02.095973+0000 mgr.y (mgr.24380) 380 : cluster 0 pgmap v399: 148 pgs: 141 active+clean, 3 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped; 3.0 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 53 MiB/s wr, 845 op/s; 16671/150360 objects degraded (11.087%); 6.1 MiB/s, 100 objects/s recovering 2024-03-11T21:01:03.304 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:01:03.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:03.312 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:01:03.413 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:01:03.443 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:01:03.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:03.446 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:01:03.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:03 smithi146 bash[23809]: cluster 2024-03-11T21:01:02.095973+0000 mgr.y (mgr.24380) 380 : cluster 0 pgmap v399: 148 pgs: 141 active+clean, 3 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped; 3.0 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 53 MiB/s wr, 845 op/s; 16671/150360 objects degraded (11.087%); 6.1 MiB/s, 100 objects/s recovering 2024-03-11T21:01:03.546 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:01:03.578 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:01:03.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:03.580 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:01:03.681 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:03.705 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:03.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:03.708 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:01:03.776 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd reweight 0 1 2024-03-11T21:01:03.808 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:01:03.859 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:01:03.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:03.863 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:01:03.964 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:01:04.015 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:01:04.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:04.024 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:01:04.124 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:01:04.151 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:01:04.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:04.155 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:01:04.254 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-11T21:01:04.254 INFO:teuthology.orchestra.run.smithi023.stdout: "size": 20, 2024-03-11T21:01:04.255 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 600, 2024-03-11T21:01:04.255 INFO:teuthology.orchestra.run.smithi023.stdout: "ops": [ 2024-03-11T21:01:04.255 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.255 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_op(client.24629.0:85807 7.e 7:7e755064:::benchmark_data_smithi023_67959_object85806:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-03-11T21:01:04.255 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T20:55:52.494442+0000", 2024-03-11T21:01:04.255 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 311.75198101199999, 2024-03-11T21:01:04.255 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.347726015, 2024-03-11T21:01:04.255 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:01:04.255 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:01:04.255 INFO:teuthology.orchestra.run.smithi023.stdout: "client_info": { 2024-03-11T21:01:04.255 INFO:teuthology.orchestra.run.smithi023.stdout: "client": "client.24629", 2024-03-11T21:01:04.255 INFO:teuthology.orchestra.run.smithi023.stdout: "client_addr": "172.21.15.23:0/959732609", 2024-03-11T21:01:04.255 INFO:teuthology.orchestra.run.smithi023.stdout: "tid": 85807 2024-03-11T21:01:04.255 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.256 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:01:04.256 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.256 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:01:04.256 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.494442+0000", 2024-03-11T21:01:04.256 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:01:04.256 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.256 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.256 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:01:04.256 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.494442+0000", 2024-03-11T21:01:04.256 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:01:04.256 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.256 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.256 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:01:04.256 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.494444+0000", 2024-03-11T21:01:04.257 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.4670000000000001e-06 2024-03-11T21:01:04.257 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.257 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.257 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:01:04.257 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.494466+0000", 2024-03-11T21:01:04.257 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.1770999999999999e-05 2024-03-11T21:01:04.257 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.257 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.257 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:01:04.257 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.494480+0000", 2024-03-11T21:01:04.257 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.4110999999999999e-05 2024-03-11T21:01:04.257 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.257 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.257 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:01:04.257 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.494488+0000", 2024-03-11T21:01:04.258 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.9100000000000005e-06 2024-03-11T21:01:04.258 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.258 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.258 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:01:04.258 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.494551+0000", 2024-03-11T21:01:04.258 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.3643e-05 2024-03-11T21:01:04.258 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.258 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.258 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:01:04.258 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.494767+0000", 2024-03-11T21:01:04.258 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00021525299999999999 2024-03-11T21:01:04.258 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.258 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.258 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for subops from 0,5", 2024-03-11T21:01:04.259 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.494957+0000", 2024-03-11T21:01:04.259 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000190163 2024-03-11T21:01:04.259 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.259 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.259 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "op_commit", 2024-03-11T21:01:04.259 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.500345+0000", 2024-03-11T21:01:04.259 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0053883250000000002 2024-03-11T21:01:04.259 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.259 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.259 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:01:04.259 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.500382+0000", 2024-03-11T21:01:04.259 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 3.6504000000000002e-05 2024-03-11T21:01:04.259 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.259 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.260 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:01:04.260 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.842019+0000", 2024-03-11T21:01:04.260 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.34163723899999998 2024-03-11T21:01:04.260 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.260 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.260 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T21:01:04.260 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.842101+0000", 2024-03-11T21:01:04.260 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 8.2674000000000002e-05 2024-03-11T21:01:04.260 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.260 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.260 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:01:04.260 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.842168+0000", 2024-03-11T21:01:04.260 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.6954999999999998e-05 2024-03-11T21:01:04.260 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:01:04.261 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:01:04.261 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:01:04.261 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.261 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.261 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_op(client.24629.0:85810 7.e 7:72a2aec7:::benchmark_data_smithi023_67959_object85809:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-03-11T21:01:04.261 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T20:55:52.497222+0000", 2024-03-11T21:01:04.261 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 311.74920113500002, 2024-03-11T21:01:04.261 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.35518546899999998, 2024-03-11T21:01:04.261 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:01:04.261 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:01:04.261 INFO:teuthology.orchestra.run.smithi023.stdout: "client_info": { 2024-03-11T21:01:04.261 INFO:teuthology.orchestra.run.smithi023.stdout: "client": "client.24629", 2024-03-11T21:01:04.262 INFO:teuthology.orchestra.run.smithi023.stdout: "client_addr": "172.21.15.23:0/959732609", 2024-03-11T21:01:04.262 INFO:teuthology.orchestra.run.smithi023.stdout: "tid": 85810 2024-03-11T21:01:04.262 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.262 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:01:04.262 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.262 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:01:04.262 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.497222+0000", 2024-03-11T21:01:04.262 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:01:04.262 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.262 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.262 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:01:04.262 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.497222+0000", 2024-03-11T21:01:04.262 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:01:04.262 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.263 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.263 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:01:04.263 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.497224+0000", 2024-03-11T21:01:04.263 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.3489999999999999e-06 2024-03-11T21:01:04.263 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.263 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.263 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:01:04.263 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.497253+0000", 2024-03-11T21:01:04.263 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.9770000000000001e-05 2024-03-11T21:01:04.263 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.263 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.263 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:01:04.263 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.497278+0000", 2024-03-11T21:01:04.263 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.4719000000000002e-05 2024-03-11T21:01:04.264 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.264 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.264 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:01:04.264 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.497288+0000", 2024-03-11T21:01:04.264 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 9.8050000000000001e-06 2024-03-11T21:01:04.264 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.264 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.264 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:01:04.264 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.497349+0000", 2024-03-11T21:01:04.264 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.0773999999999998e-05 2024-03-11T21:01:04.264 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.264 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.264 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:01:04.264 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.497477+0000", 2024-03-11T21:01:04.265 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00012798099999999999 2024-03-11T21:01:04.265 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.265 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.265 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for subops from 0,5", 2024-03-11T21:01:04.265 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.497640+0000", 2024-03-11T21:01:04.265 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000163783 2024-03-11T21:01:04.265 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.265 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.265 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "op_commit", 2024-03-11T21:01:04.265 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.500476+0000", 2024-03-11T21:01:04.265 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0028359560000000002 2024-03-11T21:01:04.265 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.265 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.265 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:01:04.265 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.503234+0000", 2024-03-11T21:01:04.265 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0027580080000000002 2024-03-11T21:01:04.266 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.266 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.266 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:01:04.266 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.852263+0000", 2024-03-11T21:01:04.266 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.349028438 2024-03-11T21:01:04.266 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.266 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.266 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T21:01:04.266 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.852335+0000", 2024-03-11T21:01:04.266 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.2323999999999994e-05 2024-03-11T21:01:04.266 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.266 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.266 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:01:04.266 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.852408+0000", 2024-03-11T21:01:04.267 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.2562000000000001e-05 2024-03-11T21:01:04.267 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:01:04.267 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:01:04.267 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:01:04.267 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.267 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.267 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_op(client.24629.0:95865 7.8 7:16200b46:::benchmark_data_smithi023_67959_object95864:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-03-11T21:01:04.267 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T20:56:04.548271+0000", 2024-03-11T21:01:04.267 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 299.69815269700001, 2024-03-11T21:01:04.267 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.34331800899999998, 2024-03-11T21:01:04.267 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:01:04.267 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:01:04.267 INFO:teuthology.orchestra.run.smithi023.stdout: "client_info": { 2024-03-11T21:01:04.267 INFO:teuthology.orchestra.run.smithi023.stdout: "client": "client.24629", 2024-03-11T21:01:04.268 INFO:teuthology.orchestra.run.smithi023.stdout: "client_addr": "172.21.15.23:0/959732609", 2024-03-11T21:01:04.268 INFO:teuthology.orchestra.run.smithi023.stdout: "tid": 95865 2024-03-11T21:01:04.268 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.268 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:01:04.268 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.268 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:01:04.268 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:04.548271+0000", 2024-03-11T21:01:04.268 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:01:04.268 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.268 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.268 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:01:04.268 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:04.548271+0000", 2024-03-11T21:01:04.268 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:01:04.268 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.268 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.268 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:01:04.269 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:04.548272+0000", 2024-03-11T21:01:04.269 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.4219999999999999e-06 2024-03-11T21:01:04.269 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.269 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.269 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:01:04.269 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:04.548296+0000", 2024-03-11T21:01:04.269 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.4107000000000001e-05 2024-03-11T21:01:04.269 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.269 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.269 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:01:04.269 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:04.548309+0000", 2024-03-11T21:01:04.269 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.2361000000000001e-05 2024-03-11T21:01:04.269 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.269 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.270 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:01:04.270 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:04.548317+0000", 2024-03-11T21:01:04.270 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 8.0690000000000002e-06 2024-03-11T21:01:04.270 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.270 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.270 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:01:04.270 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:04.552681+0000", 2024-03-11T21:01:04.270 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0043647160000000003 2024-03-11T21:01:04.270 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.270 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.270 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:01:04.270 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:04.552804+0000", 2024-03-11T21:01:04.270 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00012268500000000001 2024-03-11T21:01:04.270 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.271 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.271 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for subops from 4,7", 2024-03-11T21:01:04.271 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:04.552960+0000", 2024-03-11T21:01:04.271 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00015603700000000001 2024-03-11T21:01:04.271 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.271 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.271 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "op_commit", 2024-03-11T21:01:04.271 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:04.559246+0000", 2024-03-11T21:01:04.271 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.006285639 2024-03-11T21:01:04.271 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.271 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.271 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:01:04.271 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:04.559501+0000", 2024-03-11T21:01:04.271 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000255226 2024-03-11T21:01:04.271 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.271 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.272 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:01:04.272 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:04.891500+0000", 2024-03-11T21:01:04.272 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.331999127 2024-03-11T21:01:04.272 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.272 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.272 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T21:01:04.272 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:04.891542+0000", 2024-03-11T21:01:04.272 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 4.2009999999999999e-05 2024-03-11T21:01:04.272 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.272 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.272 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:01:04.272 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:04.891589+0000", 2024-03-11T21:01:04.272 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 4.6610000000000003e-05 2024-03-11T21:01:04.272 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:01:04.272 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:01:04.272 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:01:04.273 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.273 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.273 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_op(client.24629.0:95869 7.8 7:108786e6:::benchmark_data_smithi023_67959_object95868:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-03-11T21:01:04.273 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T20:56:04.549145+0000", 2024-03-11T21:01:04.273 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 299.69727889900003, 2024-03-11T21:01:04.273 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.34263138300000001, 2024-03-11T21:01:04.273 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:01:04.273 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:01:04.273 INFO:teuthology.orchestra.run.smithi023.stdout: "client_info": { 2024-03-11T21:01:04.273 INFO:teuthology.orchestra.run.smithi023.stdout: "client": "client.24629", 2024-03-11T21:01:04.273 INFO:teuthology.orchestra.run.smithi023.stdout: "client_addr": "172.21.15.23:0/959732609", 2024-03-11T21:01:04.273 INFO:teuthology.orchestra.run.smithi023.stdout: "tid": 95869 2024-03-11T21:01:04.273 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.273 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:01:04.274 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.274 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:01:04.274 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:04.549145+0000", 2024-03-11T21:01:04.274 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:01:04.274 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.274 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.274 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:01:04.274 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:04.549145+0000", 2024-03-11T21:01:04.274 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:01:04.274 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.274 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.274 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:01:04.274 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:04.549145+0000", 2024-03-11T21:01:04.275 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 8.7899999999999997e-07 2024-03-11T21:01:04.275 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.275 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.275 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:01:04.275 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:04.549166+0000", 2024-03-11T21:01:04.275 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.0806999999999999e-05 2024-03-11T21:01:04.275 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.275 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.275 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:01:04.275 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:04.549181+0000", 2024-03-11T21:01:04.275 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.4583000000000001e-05 2024-03-11T21:01:04.275 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.275 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.276 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:01:04.276 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:04.549188+0000", 2024-03-11T21:01:04.276 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.216e-06 2024-03-11T21:01:04.276 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.276 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.276 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:01:04.276 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:04.555791+0000", 2024-03-11T21:01:04.276 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.006602821 2024-03-11T21:01:04.276 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.276 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.276 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:01:04.276 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:04.555931+0000", 2024-03-11T21:01:04.276 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00014017500000000001 2024-03-11T21:01:04.276 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.277 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.277 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for subops from 4,7", 2024-03-11T21:01:04.277 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:04.556098+0000", 2024-03-11T21:01:04.277 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000166909 2024-03-11T21:01:04.277 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.277 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.277 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:01:04.277 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:04.559541+0000", 2024-03-11T21:01:04.277 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0034432989999999999 2024-03-11T21:01:04.277 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.277 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.277 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:01:04.277 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:04.891650+0000", 2024-03-11T21:01:04.278 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.33210917899999998 2024-03-11T21:01:04.278 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.278 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.278 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "op_commit", 2024-03-11T21:01:04.278 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:04.891664+0000", 2024-03-11T21:01:04.278 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.3687e-05 2024-03-11T21:01:04.278 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.278 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.278 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T21:01:04.278 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:04.891721+0000", 2024-03-11T21:01:04.278 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 5.6409999999999997e-05 2024-03-11T21:01:04.278 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.278 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.279 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:01:04.279 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:04.891776+0000", 2024-03-11T21:01:04.279 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 5.5417999999999997e-05 2024-03-11T21:01:04.279 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:01:04.279 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:01:04.279 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:01:04.279 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.279 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.279 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_repop(client.24629.0:95879 7.0 e146/124 7:094c45ce:::benchmark_data_smithi023_67959_object95878:head v 146'5914, mlcod=146'5912)", 2024-03-11T21:01:04.279 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T20:56:04.560512+0000", 2024-03-11T21:01:04.279 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 299.68591124199997, 2024-03-11T21:01:04.279 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.33832996500000001, 2024-03-11T21:01:04.279 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:01:04.279 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:01:04.280 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:01:04.280 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.280 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:01:04.280 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:04.560512+0000", 2024-03-11T21:01:04.280 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:01:04.280 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.280 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.280 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:01:04.280 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:04.560512+0000", 2024-03-11T21:01:04.280 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:01:04.280 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.280 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.280 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:01:04.280 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:04.560513+0000", 2024-03-11T21:01:04.281 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 3.0499999999999999e-07 2024-03-11T21:01:04.281 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.281 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.281 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:01:04.281 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:04.560594+0000", 2024-03-11T21:01:04.281 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 8.1656999999999994e-05 2024-03-11T21:01:04.281 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.281 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.281 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:01:04.281 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:04.560600+0000", 2024-03-11T21:01:04.281 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 5.6119999999999998e-06 2024-03-11T21:01:04.281 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.281 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.281 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:01:04.282 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:04.560603+0000", 2024-03-11T21:01:04.282 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 3.6969999999999999e-06 2024-03-11T21:01:04.282 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.282 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.282 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:01:04.282 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:04.891627+0000", 2024-03-11T21:01:04.282 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.33102393000000002 2024-03-11T21:01:04.282 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.282 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.282 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:01:04.282 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:04.891650+0000", 2024-03-11T21:01:04.282 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.2147999999999999e-05 2024-03-11T21:01:04.282 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.282 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.283 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T21:01:04.283 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:04.898788+0000", 2024-03-11T21:01:04.283 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0071382360000000001 2024-03-11T21:01:04.283 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.283 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.283 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:01:04.283 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:04.898842+0000", 2024-03-11T21:01:04.283 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 5.4379999999999998e-05 2024-03-11T21:01:04.283 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:01:04.283 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:01:04.283 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:01:04.283 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.283 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.283 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_op(client.24629.0:95895 7.8 7:19f97576:::benchmark_data_smithi023_67959_object95894:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-03-11T21:01:04.284 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T20:56:04.575259+0000", 2024-03-11T21:01:04.284 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 299.67116411000001, 2024-03-11T21:01:04.284 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.34994346500000001, 2024-03-11T21:01:04.284 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:01:04.284 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:01:04.284 INFO:teuthology.orchestra.run.smithi023.stdout: "client_info": { 2024-03-11T21:01:04.284 INFO:teuthology.orchestra.run.smithi023.stdout: "client": "client.24629", 2024-03-11T21:01:04.284 INFO:teuthology.orchestra.run.smithi023.stdout: "client_addr": "172.21.15.23:0/959732609", 2024-03-11T21:01:04.284 INFO:teuthology.orchestra.run.smithi023.stdout: "tid": 95895 2024-03-11T21:01:04.284 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.284 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:01:04.284 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.284 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:01:04.284 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:04.575259+0000", 2024-03-11T21:01:04.284 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:01:04.285 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.285 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.285 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:01:04.285 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:04.575259+0000", 2024-03-11T21:01:04.285 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:01:04.285 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.285 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.285 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:01:04.285 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:04.575261+0000", 2024-03-11T21:01:04.285 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.477e-06 2024-03-11T21:01:04.285 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.285 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.285 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:01:04.285 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:04.575303+0000", 2024-03-11T21:01:04.285 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 4.1841000000000001e-05 2024-03-11T21:01:04.286 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.286 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.286 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:01:04.286 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:04.575314+0000", 2024-03-11T21:01:04.286 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.1616e-05 2024-03-11T21:01:04.286 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.286 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.286 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:01:04.286 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:04.575322+0000", 2024-03-11T21:01:04.286 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.5950000000000001e-06 2024-03-11T21:01:04.286 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.286 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.286 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:01:04.286 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:04.912606+0000", 2024-03-11T21:01:04.287 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.33728387799999998 2024-03-11T21:01:04.287 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.287 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.287 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:01:04.287 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:04.912765+0000", 2024-03-11T21:01:04.287 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00015936800000000001 2024-03-11T21:01:04.287 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.287 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.287 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for subops from 4,7", 2024-03-11T21:01:04.287 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:04.912928+0000", 2024-03-11T21:01:04.287 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00016306900000000001 2024-03-11T21:01:04.287 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.287 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.287 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:01:04.287 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:04.921764+0000", 2024-03-11T21:01:04.288 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0088356700000000003 2024-03-11T21:01:04.288 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.288 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.288 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "op_commit", 2024-03-11T21:01:04.288 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:04.924776+0000", 2024-03-11T21:01:04.288 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0030116850000000001 2024-03-11T21:01:04.288 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.288 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.288 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:01:04.288 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:04.924993+0000", 2024-03-11T21:01:04.288 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000217047 2024-03-11T21:01:04.288 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.288 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.288 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T21:01:04.288 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:04.925130+0000", 2024-03-11T21:01:04.289 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00013763299999999999 2024-03-11T21:01:04.289 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.289 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.289 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:01:04.289 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:04.925203+0000", 2024-03-11T21:01:04.289 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.2585999999999999e-05 2024-03-11T21:01:04.289 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:01:04.289 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:01:04.289 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:01:04.289 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.289 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.289 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_op(client.24629.0:95903 7.8 7:1d4be64b:::benchmark_data_smithi023_67959_object95902:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-03-11T21:01:04.289 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T20:56:04.582398+0000", 2024-03-11T21:01:04.289 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 299.66402587300001, 2024-03-11T21:01:04.289 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.34652825799999998, 2024-03-11T21:01:04.290 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:01:04.290 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:01:04.290 INFO:teuthology.orchestra.run.smithi023.stdout: "client_info": { 2024-03-11T21:01:04.290 INFO:teuthology.orchestra.run.smithi023.stdout: "client": "client.24629", 2024-03-11T21:01:04.290 INFO:teuthology.orchestra.run.smithi023.stdout: "client_addr": "172.21.15.23:0/959732609", 2024-03-11T21:01:04.290 INFO:teuthology.orchestra.run.smithi023.stdout: "tid": 95903 2024-03-11T21:01:04.290 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.290 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:01:04.290 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.290 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:01:04.290 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:04.582398+0000", 2024-03-11T21:01:04.290 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:01:04.290 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.290 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.290 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:01:04.291 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:04.582398+0000", 2024-03-11T21:01:04.291 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:01:04.291 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.291 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.291 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:01:04.291 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:04.582399+0000", 2024-03-11T21:01:04.291 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.285e-06 2024-03-11T21:01:04.291 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.291 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.291 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:01:04.291 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:04.582422+0000", 2024-03-11T21:01:04.291 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.2790999999999999e-05 2024-03-11T21:01:04.291 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.291 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.291 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:01:04.292 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:04.582432+0000", 2024-03-11T21:01:04.292 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.0737000000000001e-05 2024-03-11T21:01:04.292 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.292 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.292 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:01:04.292 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:04.582440+0000", 2024-03-11T21:01:04.292 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.5569999999999998e-06 2024-03-11T21:01:04.292 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.292 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.292 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:01:04.292 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:04.915918+0000", 2024-03-11T21:01:04.292 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.33347801900000001 2024-03-11T21:01:04.292 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.292 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.292 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:01:04.293 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:04.916078+0000", 2024-03-11T21:01:04.293 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00016017900000000001 2024-03-11T21:01:04.293 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.293 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.293 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for subops from 4,7", 2024-03-11T21:01:04.293 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:04.916267+0000", 2024-03-11T21:01:04.293 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000188519 2024-03-11T21:01:04.293 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.293 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.293 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:01:04.293 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:04.924861+0000", 2024-03-11T21:01:04.293 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0085944769999999997 2024-03-11T21:01:04.293 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.293 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.293 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "op_commit", 2024-03-11T21:01:04.294 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:04.924946+0000", 2024-03-11T21:01:04.294 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 8.4858000000000006e-05 2024-03-11T21:01:04.294 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.294 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.294 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:01:04.294 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:04.928787+0000", 2024-03-11T21:01:04.294 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0038412910000000001 2024-03-11T21:01:04.294 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.294 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.294 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T21:01:04.294 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:04.928853+0000", 2024-03-11T21:01:04.294 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.6217000000000007e-05 2024-03-11T21:01:04.294 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.294 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.295 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:01:04.295 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:04.928926+0000", 2024-03-11T21:01:04.295 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.2328000000000003e-05 2024-03-11T21:01:04.295 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:01:04.295 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:01:04.295 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:01:04.295 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.295 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.295 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_op(client.24629.0:95904 7.8 7:1f808330:::benchmark_data_smithi023_67959_object95903:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-03-11T21:01:04.295 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T20:56:04.584562+0000", 2024-03-11T21:01:04.295 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 299.66186109699998, 2024-03-11T21:01:04.295 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.34516982000000002, 2024-03-11T21:01:04.295 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:01:04.295 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:01:04.296 INFO:teuthology.orchestra.run.smithi023.stdout: "client_info": { 2024-03-11T21:01:04.296 INFO:teuthology.orchestra.run.smithi023.stdout: "client": "client.24629", 2024-03-11T21:01:04.296 INFO:teuthology.orchestra.run.smithi023.stdout: "client_addr": "172.21.15.23:0/959732609", 2024-03-11T21:01:04.296 INFO:teuthology.orchestra.run.smithi023.stdout: "tid": 95904 2024-03-11T21:01:04.296 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.296 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:01:04.296 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.296 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:01:04.296 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:04.584562+0000", 2024-03-11T21:01:04.296 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:01:04.296 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.296 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.296 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:01:04.296 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:04.584562+0000", 2024-03-11T21:01:04.296 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:01:04.297 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.297 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.297 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:01:04.297 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:04.584563+0000", 2024-03-11T21:01:04.297 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.111e-06 2024-03-11T21:01:04.297 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.297 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.297 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:01:04.297 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:04.584586+0000", 2024-03-11T21:01:04.297 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.2623e-05 2024-03-11T21:01:04.297 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.297 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.297 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:01:04.297 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:04.584596+0000", 2024-03-11T21:01:04.297 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 9.499e-06 2024-03-11T21:01:04.297 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.298 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.298 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:01:04.298 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:04.584601+0000", 2024-03-11T21:01:04.298 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 5.0969999999999996e-06 2024-03-11T21:01:04.298 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.298 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.298 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:01:04.298 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:04.918690+0000", 2024-03-11T21:01:04.298 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.334089413 2024-03-11T21:01:04.298 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.298 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.298 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:01:04.298 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:04.918815+0000", 2024-03-11T21:01:04.298 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00012475100000000001 2024-03-11T21:01:04.299 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.299 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.299 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for subops from 4,7", 2024-03-11T21:01:04.299 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:04.918965+0000", 2024-03-11T21:01:04.299 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00015017700000000001 2024-03-11T21:01:04.299 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.299 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.299 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:01:04.299 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:04.924926+0000", 2024-03-11T21:01:04.299 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0059611849999999999 2024-03-11T21:01:04.299 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.299 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.299 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "op_commit", 2024-03-11T21:01:04.299 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:04.928938+0000", 2024-03-11T21:01:04.300 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0040120629999999997 2024-03-11T21:01:04.300 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.300 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.300 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:01:04.300 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:04.929608+0000", 2024-03-11T21:01:04.300 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00066941500000000003 2024-03-11T21:01:04.300 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.300 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.300 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T21:01:04.300 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:04.929669+0000", 2024-03-11T21:01:04.300 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.1606999999999995e-05 2024-03-11T21:01:04.300 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.300 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.300 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:01:04.301 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:04.929732+0000", 2024-03-11T21:01:04.301 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.2879000000000006e-05 2024-03-11T21:01:04.301 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:01:04.301 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:01:04.301 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:01:04.301 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.301 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.301 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_repop(client.24629.0:126332 7.a e154/121 7:5f290de1:::benchmark_data_smithi023_67959_object16091:head v 154'8070, mlcod=154'8068)", 2024-03-11T21:01:04.301 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T20:56:39.175601+0000", 2024-03-11T21:01:04.301 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 265.07082221399997, 2024-03-11T21:01:04.301 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.44168193900000002, 2024-03-11T21:01:04.301 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:01:04.301 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:01:04.301 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:01:04.301 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.302 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:01:04.302 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.175601+0000", 2024-03-11T21:01:04.302 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:01:04.302 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.302 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.302 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:01:04.302 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.175601+0000", 2024-03-11T21:01:04.302 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:01:04.302 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.302 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.302 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:01:04.302 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.175602+0000", 2024-03-11T21:01:04.302 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.6000000000000003e-07 2024-03-11T21:01:04.303 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.303 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.303 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:01:04.303 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.175608+0000", 2024-03-11T21:01:04.303 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 5.7590000000000003e-06 2024-03-11T21:01:04.303 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.303 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.303 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:01:04.303 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.175617+0000", 2024-03-11T21:01:04.303 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 8.9039999999999998e-06 2024-03-11T21:01:04.303 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.303 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.303 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:01:04.303 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.175623+0000", 2024-03-11T21:01:04.304 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.083e-06 2024-03-11T21:01:04.304 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.304 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.304 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:01:04.304 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.177021+0000", 2024-03-11T21:01:04.304 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0013985619999999999 2024-03-11T21:01:04.304 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.304 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.304 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:01:04.304 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.177040+0000", 2024-03-11T21:01:04.304 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.8564999999999999e-05 2024-03-11T21:01:04.304 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.304 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.304 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T21:01:04.305 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.617193+0000", 2024-03-11T21:01:04.305 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.440153036 2024-03-11T21:01:04.305 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.305 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.305 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:01:04.305 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.617283+0000", 2024-03-11T21:01:04.305 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 9.0370000000000004e-05 2024-03-11T21:01:04.305 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:01:04.305 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:01:04.305 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:01:04.305 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.305 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.305 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_repop(client.24629.0:126334 7.2 e154/124 7:40a0664e:::benchmark_data_smithi023_67959_object16093:head v 154'7812, mlcod=154'7810)", 2024-03-11T21:01:04.305 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T20:56:39.178229+0000", 2024-03-11T21:01:04.306 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 265.06819414099999, 2024-03-11T21:01:04.306 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.43915848200000002, 2024-03-11T21:01:04.306 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:01:04.306 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:01:04.306 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:01:04.306 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.306 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:01:04.306 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.178229+0000", 2024-03-11T21:01:04.306 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:01:04.306 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.306 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.306 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:01:04.306 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.178229+0000", 2024-03-11T21:01:04.306 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:01:04.307 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.307 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.307 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:01:04.307 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.178230+0000", 2024-03-11T21:01:04.307 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.3600000000000003e-07 2024-03-11T21:01:04.307 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.307 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.307 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:01:04.307 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.178235+0000", 2024-03-11T21:01:04.307 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 5.2700000000000004e-06 2024-03-11T21:01:04.307 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.307 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.307 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:01:04.307 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.178243+0000", 2024-03-11T21:01:04.307 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.6529999999999994e-06 2024-03-11T21:01:04.308 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.308 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.308 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:01:04.308 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.178248+0000", 2024-03-11T21:01:04.308 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 4.7690000000000004e-06 2024-03-11T21:01:04.308 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.308 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.308 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:01:04.308 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.178575+0000", 2024-03-11T21:01:04.308 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00032776700000000002 2024-03-11T21:01:04.308 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.308 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.308 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:01:04.308 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.178594+0000", 2024-03-11T21:01:04.308 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.8249000000000001e-05 2024-03-11T21:01:04.309 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.309 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.309 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T21:01:04.309 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.617294+0000", 2024-03-11T21:01:04.309 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.43870066499999999 2024-03-11T21:01:04.309 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.309 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.309 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:01:04.309 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.617388+0000", 2024-03-11T21:01:04.309 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 9.3373000000000002e-05 2024-03-11T21:01:04.309 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:01:04.309 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:01:04.309 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:01:04.309 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.310 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.310 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_op(client.24629.0:126339 7.e 7:7a2bd3c2:::benchmark_data_smithi023_67959_object16098:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-03-11T21:01:04.310 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T20:56:39.178686+0000", 2024-03-11T21:01:04.310 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 265.06773762300003, 2024-03-11T21:01:04.310 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.44792807800000001, 2024-03-11T21:01:04.310 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:01:04.310 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:01:04.310 INFO:teuthology.orchestra.run.smithi023.stdout: "client_info": { 2024-03-11T21:01:04.310 INFO:teuthology.orchestra.run.smithi023.stdout: "client": "client.24629", 2024-03-11T21:01:04.310 INFO:teuthology.orchestra.run.smithi023.stdout: "client_addr": "172.21.15.23:0/959732609", 2024-03-11T21:01:04.310 INFO:teuthology.orchestra.run.smithi023.stdout: "tid": 126339 2024-03-11T21:01:04.310 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.310 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:01:04.310 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.311 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:01:04.311 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.178686+0000", 2024-03-11T21:01:04.311 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:01:04.311 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.311 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.311 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:01:04.311 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.178686+0000", 2024-03-11T21:01:04.311 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:01:04.311 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.311 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.311 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:01:04.311 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.178687+0000", 2024-03-11T21:01:04.311 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.119e-06 2024-03-11T21:01:04.311 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.311 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.312 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:01:04.312 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.178691+0000", 2024-03-11T21:01:04.312 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 3.5769999999999998e-06 2024-03-11T21:01:04.312 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.312 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.312 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:01:04.312 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.178698+0000", 2024-03-11T21:01:04.312 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.9820000000000008e-06 2024-03-11T21:01:04.312 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.312 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.312 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:01:04.312 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.178705+0000", 2024-03-11T21:01:04.312 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.2269999999999998e-06 2024-03-11T21:01:04.312 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.312 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.313 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:01:04.313 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.180899+0000", 2024-03-11T21:01:04.313 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.002193951 2024-03-11T21:01:04.313 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.313 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.313 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:01:04.313 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.184356+0000", 2024-03-11T21:01:04.313 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0034575750000000001 2024-03-11T21:01:04.313 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.313 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.313 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for subops from 0,5", 2024-03-11T21:01:04.313 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.184465+0000", 2024-03-11T21:01:04.313 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000109131 2024-03-11T21:01:04.313 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.314 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.314 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "op_commit", 2024-03-11T21:01:04.314 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.188467+0000", 2024-03-11T21:01:04.314 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0040014509999999996 2024-03-11T21:01:04.314 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.314 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.314 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:01:04.314 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.626225+0000", 2024-03-11T21:01:04.314 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.43775783400000001 2024-03-11T21:01:04.314 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.314 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.314 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:01:04.314 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.626313+0000", 2024-03-11T21:01:04.314 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 8.8484000000000002e-05 2024-03-11T21:01:04.315 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.315 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.315 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T21:01:04.315 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.626513+0000", 2024-03-11T21:01:04.315 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00020033599999999999 2024-03-11T21:01:04.315 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.315 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.315 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:01:04.315 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.626614+0000", 2024-03-11T21:01:04.315 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000100411 2024-03-11T21:01:04.315 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:01:04.315 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:01:04.315 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:01:04.315 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.316 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.316 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_repop(client.24629.0:126341 7.a e154/121 7:5cab8f84:::benchmark_data_smithi023_67959_object16100:head v 154'8071, mlcod=154'8068)", 2024-03-11T21:01:04.316 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T20:56:39.180228+0000", 2024-03-11T21:01:04.316 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 265.06619583399998, 2024-03-11T21:01:04.316 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.43754316799999998, 2024-03-11T21:01:04.316 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:01:04.316 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:01:04.316 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:01:04.316 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.316 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:01:04.316 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.180228+0000", 2024-03-11T21:01:04.316 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:01:04.316 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.316 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.316 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:01:04.317 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.180228+0000", 2024-03-11T21:01:04.317 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:01:04.317 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.317 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.317 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:01:04.317 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.180228+0000", 2024-03-11T21:01:04.317 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 8.1299999999999999e-07 2024-03-11T21:01:04.317 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.317 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.317 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:01:04.317 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.180234+0000", 2024-03-11T21:01:04.317 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 5.6350000000000003e-06 2024-03-11T21:01:04.317 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.317 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.317 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:01:04.317 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.180245+0000", 2024-03-11T21:01:04.318 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.0695e-05 2024-03-11T21:01:04.318 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.318 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.318 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:01:04.318 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.180252+0000", 2024-03-11T21:01:04.318 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.7560000000000002e-06 2024-03-11T21:01:04.318 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.318 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.318 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:01:04.318 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.184772+0000", 2024-03-11T21:01:04.318 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0045209339999999999 2024-03-11T21:01:04.318 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.318 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.319 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:01:04.319 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.184792+0000", 2024-03-11T21:01:04.319 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.9369999999999999e-05 2024-03-11T21:01:04.319 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.319 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.319 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T21:01:04.319 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.617394+0000", 2024-03-11T21:01:04.319 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.43260256499999999 2024-03-11T21:01:04.319 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.319 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.319 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:01:04.319 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.617771+0000", 2024-03-11T21:01:04.319 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00037639999999999999 2024-03-11T21:01:04.319 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:01:04.319 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:01:04.320 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:01:04.320 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.320 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.320 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_op(client.24629.0:126348 7.8 7:110a9e0b:::benchmark_data_smithi023_67959_object16107:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-03-11T21:01:04.320 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T20:56:39.180909+0000", 2024-03-11T21:01:04.320 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 265.06551456599999, 2024-03-11T21:01:04.320 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.43655640800000001, 2024-03-11T21:01:04.320 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:01:04.320 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:01:04.320 INFO:teuthology.orchestra.run.smithi023.stdout: "client_info": { 2024-03-11T21:01:04.320 INFO:teuthology.orchestra.run.smithi023.stdout: "client": "client.24629", 2024-03-11T21:01:04.320 INFO:teuthology.orchestra.run.smithi023.stdout: "client_addr": "172.21.15.23:0/959732609", 2024-03-11T21:01:04.320 INFO:teuthology.orchestra.run.smithi023.stdout: "tid": 126348 2024-03-11T21:01:04.321 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.321 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:01:04.321 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.321 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:01:04.321 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.180909+0000", 2024-03-11T21:01:04.321 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:01:04.321 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.321 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.321 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:01:04.321 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.180909+0000", 2024-03-11T21:01:04.321 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:01:04.321 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.321 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.321 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:01:04.322 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.180910+0000", 2024-03-11T21:01:04.322 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.0860000000000001e-06 2024-03-11T21:01:04.322 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.322 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.322 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:01:04.322 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.180915+0000", 2024-03-11T21:01:04.322 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 4.7090000000000001e-06 2024-03-11T21:01:04.322 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.322 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.322 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:01:04.322 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.180922+0000", 2024-03-11T21:01:04.322 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.6869999999999994e-06 2024-03-11T21:01:04.322 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.322 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.323 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:01:04.323 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.180929+0000", 2024-03-11T21:01:04.323 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.7619999999999998e-06 2024-03-11T21:01:04.323 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.323 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.323 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:01:04.323 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.180978+0000", 2024-03-11T21:01:04.323 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 4.9029000000000001e-05 2024-03-11T21:01:04.323 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.323 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.323 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:01:04.323 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.181084+0000", 2024-03-11T21:01:04.323 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000106051 2024-03-11T21:01:04.324 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.324 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.324 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for subops from 4,7", 2024-03-11T21:01:04.324 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.181189+0000", 2024-03-11T21:01:04.324 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00010433900000000001 2024-03-11T21:01:04.324 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.324 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.324 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:01:04.324 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.185961+0000", 2024-03-11T21:01:04.324 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0047725099999999998 2024-03-11T21:01:04.324 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.324 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.324 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "op_commit", 2024-03-11T21:01:04.324 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.187679+0000", 2024-03-11T21:01:04.325 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.001717695 2024-03-11T21:01:04.325 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.325 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.325 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:01:04.325 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.617314+0000", 2024-03-11T21:01:04.325 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.42963545400000003 2024-03-11T21:01:04.325 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.325 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.325 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T21:01:04.325 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.617405+0000", 2024-03-11T21:01:04.325 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 9.0555000000000003e-05 2024-03-11T21:01:04.325 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.325 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.325 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:01:04.325 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.617465+0000", 2024-03-11T21:01:04.326 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.0531e-05 2024-03-11T21:01:04.326 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:01:04.326 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:01:04.326 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:01:04.326 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.326 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.326 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_op(client.24629.0:126349 7.e 7:70671e9b:::benchmark_data_smithi023_67959_object16108:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-03-11T21:01:04.326 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T20:56:39.184548+0000", 2024-03-11T21:01:04.326 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 265.06187568899998, 2024-03-11T21:01:04.326 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.44365520200000003, 2024-03-11T21:01:04.326 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:01:04.326 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:01:04.326 INFO:teuthology.orchestra.run.smithi023.stdout: "client_info": { 2024-03-11T21:01:04.327 INFO:teuthology.orchestra.run.smithi023.stdout: "client": "client.24629", 2024-03-11T21:01:04.327 INFO:teuthology.orchestra.run.smithi023.stdout: "client_addr": "172.21.15.23:0/959732609", 2024-03-11T21:01:04.327 INFO:teuthology.orchestra.run.smithi023.stdout: "tid": 126349 2024-03-11T21:01:04.327 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.327 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:01:04.327 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.327 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:01:04.327 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.184548+0000", 2024-03-11T21:01:04.327 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:01:04.327 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.327 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.327 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:01:04.327 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.184548+0000", 2024-03-11T21:01:04.327 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:01:04.327 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.327 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.327 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:01:04.328 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.184549+0000", 2024-03-11T21:01:04.328 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.1650000000000001e-06 2024-03-11T21:01:04.328 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.328 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.328 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:01:04.328 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.184554+0000", 2024-03-11T21:01:04.328 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 4.6140000000000002e-06 2024-03-11T21:01:04.328 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.328 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.328 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:01:04.328 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.184563+0000", 2024-03-11T21:01:04.328 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 9.7089999999999997e-06 2024-03-11T21:01:04.328 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.328 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.329 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:01:04.329 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.184571+0000", 2024-03-11T21:01:04.329 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.4359999999999996e-06 2024-03-11T21:01:04.329 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.329 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.329 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:01:04.329 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.188491+0000", 2024-03-11T21:01:04.329 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0039207859999999999 2024-03-11T21:01:04.329 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.329 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.329 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:01:04.329 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.188622+0000", 2024-03-11T21:01:04.329 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00013099800000000001 2024-03-11T21:01:04.329 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.329 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.330 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for subops from 0,5", 2024-03-11T21:01:04.330 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.617537+0000", 2024-03-11T21:01:04.330 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.42891444699999998 2024-03-11T21:01:04.330 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.330 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.330 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:01:04.330 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.626711+0000", 2024-03-11T21:01:04.330 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.009174204 2024-03-11T21:01:04.330 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.330 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.330 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "op_commit", 2024-03-11T21:01:04.330 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.626738+0000", 2024-03-11T21:01:04.330 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.7132e-05 2024-03-11T21:01:04.330 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.330 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.331 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:01:04.331 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.626811+0000", 2024-03-11T21:01:04.331 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.2512e-05 2024-03-11T21:01:04.331 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.331 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.331 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T21:01:04.331 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.626917+0000", 2024-03-11T21:01:04.331 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00010669400000000001 2024-03-11T21:01:04.331 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.331 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.331 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:01:04.331 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.628203+0000", 2024-03-11T21:01:04.331 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0012855049999999999 2024-03-11T21:01:04.331 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:01:04.331 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:01:04.331 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:01:04.332 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.332 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.332 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_repop(client.24629.0:126340 7.2 e154/124 7:41534ba6:::benchmark_data_smithi023_67959_object16099:head v 154'7813, mlcod=154'7810)", 2024-03-11T21:01:04.332 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T20:56:39.184790+0000", 2024-03-11T21:01:04.332 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 265.06163392100001, 2024-03-11T21:01:04.332 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.43306609800000001, 2024-03-11T21:01:04.332 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:01:04.332 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:01:04.332 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:01:04.332 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.332 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:01:04.332 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.184790+0000", 2024-03-11T21:01:04.332 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:01:04.332 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.332 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.333 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:01:04.333 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.184790+0000", 2024-03-11T21:01:04.333 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:01:04.333 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.333 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.333 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:01:04.333 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.184790+0000", 2024-03-11T21:01:04.333 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.2500000000000005e-07 2024-03-11T21:01:04.333 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.333 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.333 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:01:04.333 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.184807+0000", 2024-03-11T21:01:04.333 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.7019000000000001e-05 2024-03-11T21:01:04.333 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.334 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.334 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:01:04.334 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.184817+0000", 2024-03-11T21:01:04.334 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 9.2730000000000008e-06 2024-03-11T21:01:04.334 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.334 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.334 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:01:04.334 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.184823+0000", 2024-03-11T21:01:04.334 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.4320000000000004e-06 2024-03-11T21:01:04.334 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.334 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.334 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:01:04.334 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.187366+0000", 2024-03-11T21:01:04.334 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0025428109999999999 2024-03-11T21:01:04.335 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.335 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.335 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:01:04.335 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.187385+0000", 2024-03-11T21:01:04.335 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.9547000000000001e-05 2024-03-11T21:01:04.335 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.335 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.335 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T21:01:04.335 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.617777+0000", 2024-03-11T21:01:04.335 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.43039132200000002 2024-03-11T21:01:04.335 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.335 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.335 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:01:04.335 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.617856+0000", 2024-03-11T21:01:04.335 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.8968999999999996e-05 2024-03-11T21:01:04.336 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:01:04.336 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:01:04.336 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:01:04.336 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.336 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.336 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_repop_reply(client.24629.0:126348 7.8 e154/121 ondisk, result = 0)", 2024-03-11T21:01:04.336 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T20:56:39.188761+0000", 2024-03-11T21:01:04.336 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 265.05766263300001, 2024-03-11T21:01:04.336 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.42871197799999999, 2024-03-11T21:01:04.336 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:01:04.336 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "started", 2024-03-11T21:01:04.336 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:01:04.336 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.336 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:01:04.336 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.188761+0000", 2024-03-11T21:01:04.337 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:01:04.337 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.337 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.337 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:01:04.337 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.188761+0000", 2024-03-11T21:01:04.337 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:01:04.337 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.337 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.337 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:01:04.337 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.188762+0000", 2024-03-11T21:01:04.337 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 8.2099999999999995e-07 2024-03-11T21:01:04.337 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.337 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.337 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:01:04.337 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.188768+0000", 2024-03-11T21:01:04.338 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 5.9349999999999999e-06 2024-03-11T21:01:04.338 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.338 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.338 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:01:04.338 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.617200+0000", 2024-03-11T21:01:04.338 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.42843230599999998 2024-03-11T21:01:04.338 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.338 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.338 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:01:04.338 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.617215+0000", 2024-03-11T21:01:04.338 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.5486999999999999e-05 2024-03-11T21:01:04.338 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.338 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.338 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:01:04.339 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.617293+0000", 2024-03-11T21:01:04.339 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.7547999999999997e-05 2024-03-11T21:01:04.339 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.339 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.339 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:01:04.339 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.617304+0000", 2024-03-11T21:01:04.339 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.1472e-05 2024-03-11T21:01:04.339 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.339 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.339 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:01:04.339 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.617473+0000", 2024-03-11T21:01:04.339 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00016840900000000001 2024-03-11T21:01:04.339 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:01:04.339 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:01:04.339 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:01:04.340 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.340 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.340 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_repop_reply(client.24629.0:126339 7.e e154/121 ondisk, result = 0)", 2024-03-11T21:01:04.340 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T20:56:39.190247+0000", 2024-03-11T21:01:04.340 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 265.05617680500001, 2024-03-11T21:01:04.340 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.43599560900000001, 2024-03-11T21:01:04.340 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:01:04.340 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "started", 2024-03-11T21:01:04.340 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:01:04.340 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.340 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:01:04.340 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.190247+0000", 2024-03-11T21:01:04.340 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:01:04.340 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.340 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.341 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:01:04.341 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.190247+0000", 2024-03-11T21:01:04.341 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:01:04.341 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.341 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.341 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:01:04.341 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.190247+0000", 2024-03-11T21:01:04.341 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 5.7599999999999997e-07 2024-03-11T21:01:04.341 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.341 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.341 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:01:04.341 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.190251+0000", 2024-03-11T21:01:04.341 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 3.3589999999999999e-06 2024-03-11T21:01:04.341 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.341 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.342 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:01:04.342 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.617193+0000", 2024-03-11T21:01:04.342 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.42694225200000002 2024-03-11T21:01:04.342 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.342 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.342 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:01:04.342 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.617225+0000", 2024-03-11T21:01:04.342 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 3.1940000000000003e-05 2024-03-11T21:01:04.342 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.342 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.342 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:01:04.342 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.626191+0000", 2024-03-11T21:01:04.342 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0089666639999999992 2024-03-11T21:01:04.342 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.343 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.343 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:01:04.343 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.626211+0000", 2024-03-11T21:01:04.343 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.9658000000000001e-05 2024-03-11T21:01:04.343 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.343 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.343 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:01:04.343 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.626242+0000", 2024-03-11T21:01:04.343 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 3.116e-05 2024-03-11T21:01:04.343 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:01:04.343 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:01:04.343 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:01:04.343 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.343 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.344 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_op(client.24629.0:135255 7.e 7:75e725ce:::benchmark_data_smithi023_67959_object25014:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-03-11T21:01:04.344 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T20:56:47.860631+0000", 2024-03-11T21:01:04.344 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 256.38579241600002, 2024-03-11T21:01:04.344 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.40904401499999998, 2024-03-11T21:01:04.344 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:01:04.344 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:01:04.344 INFO:teuthology.orchestra.run.smithi023.stdout: "client_info": { 2024-03-11T21:01:04.344 INFO:teuthology.orchestra.run.smithi023.stdout: "client": "client.24629", 2024-03-11T21:01:04.344 INFO:teuthology.orchestra.run.smithi023.stdout: "client_addr": "172.21.15.23:0/959732609", 2024-03-11T21:01:04.344 INFO:teuthology.orchestra.run.smithi023.stdout: "tid": 135255 2024-03-11T21:01:04.344 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.344 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:01:04.344 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.344 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:01:04.344 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:47.860631+0000", 2024-03-11T21:01:04.345 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:01:04.345 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.345 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.345 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:01:04.345 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:47.860631+0000", 2024-03-11T21:01:04.345 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:01:04.345 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.345 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.345 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:01:04.345 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:47.860633+0000", 2024-03-11T21:01:04.345 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.7859999999999999e-06 2024-03-11T21:01:04.345 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.345 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.345 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:01:04.345 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:47.860639+0000", 2024-03-11T21:01:04.346 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 5.6930000000000004e-06 2024-03-11T21:01:04.346 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.346 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.346 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:01:04.346 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:47.860650+0000", 2024-03-11T21:01:04.346 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.1321000000000001e-05 2024-03-11T21:01:04.346 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.346 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.346 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:01:04.346 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:47.860657+0000", 2024-03-11T21:01:04.346 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.6680000000000001e-06 2024-03-11T21:01:04.346 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.346 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.346 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:01:04.347 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:47.860747+0000", 2024-03-11T21:01:04.347 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 8.9596000000000001e-05 2024-03-11T21:01:04.347 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.347 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.347 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:01:04.347 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:47.860918+0000", 2024-03-11T21:01:04.347 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000170784 2024-03-11T21:01:04.347 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.347 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.347 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for subops from 0,5", 2024-03-11T21:01:04.347 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:47.861045+0000", 2024-03-11T21:01:04.347 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000127284 2024-03-11T21:01:04.347 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.347 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.347 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:01:04.348 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:47.866388+0000", 2024-03-11T21:01:04.348 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0053429239999999998 2024-03-11T21:01:04.348 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.348 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.348 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "op_commit", 2024-03-11T21:01:04.348 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:47.866473+0000", 2024-03-11T21:01:04.348 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 8.4703000000000005e-05 2024-03-11T21:01:04.348 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.348 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.348 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:01:04.348 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:48.269388+0000", 2024-03-11T21:01:04.348 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.40291516700000002 2024-03-11T21:01:04.348 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.348 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.348 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T21:01:04.349 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:48.269568+0000", 2024-03-11T21:01:04.349 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00018036699999999999 2024-03-11T21:01:04.349 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.349 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.349 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:01:04.349 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:48.269675+0000", 2024-03-11T21:01:04.349 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000106722 2024-03-11T21:01:04.349 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:01:04.349 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:01:04.349 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:01:04.349 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.349 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.349 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_op(client.24629.0:167641 7.b 7:d3c034eb:::benchmark_data_smithi023_67959_object57400:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e163)", 2024-03-11T21:01:04.349 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T20:57:24.596835+0000", 2024-03-11T21:01:04.349 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 219.64958872400001, 2024-03-11T21:01:04.349 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.241247719, 2024-03-11T21:01:04.350 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:01:04.350 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:01:04.350 INFO:teuthology.orchestra.run.smithi023.stdout: "client_info": { 2024-03-11T21:01:04.350 INFO:teuthology.orchestra.run.smithi023.stdout: "client": "client.24629", 2024-03-11T21:01:04.350 INFO:teuthology.orchestra.run.smithi023.stdout: "client_addr": "172.21.15.23:0/959732609", 2024-03-11T21:01:04.350 INFO:teuthology.orchestra.run.smithi023.stdout: "tid": 167641 2024-03-11T21:01:04.350 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.350 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:01:04.350 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.350 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:01:04.350 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:24.596835+0000", 2024-03-11T21:01:04.350 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:01:04.350 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.351 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.351 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:01:04.351 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:24.596835+0000", 2024-03-11T21:01:04.351 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:01:04.351 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.351 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.351 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:01:04.351 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:24.596838+0000", 2024-03-11T21:01:04.351 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.8100000000000002e-06 2024-03-11T21:01:04.351 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.351 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.351 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:01:04.351 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:24.596852+0000", 2024-03-11T21:01:04.351 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.416e-05 2024-03-11T21:01:04.351 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.351 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.352 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:01:04.352 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:24.596885+0000", 2024-03-11T21:01:04.352 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 3.3683000000000003e-05 2024-03-11T21:01:04.352 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.352 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.352 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:01:04.352 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:24.596909+0000", 2024-03-11T21:01:04.352 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.3753000000000001e-05 2024-03-11T21:01:04.352 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.352 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.352 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:01:04.352 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:24.597090+0000", 2024-03-11T21:01:04.352 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00018072499999999999 2024-03-11T21:01:04.352 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.353 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.353 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "op must wait for map", 2024-03-11T21:01:04.353 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:24.597111+0000", 2024-03-11T21:01:04.353 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.1094000000000002e-05 2024-03-11T21:01:04.353 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.353 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.353 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:01:04.353 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:24.604658+0000", 2024-03-11T21:01:04.353 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0075470959999999997 2024-03-11T21:01:04.353 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.353 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.353 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for peered", 2024-03-11T21:01:04.353 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:24.604665+0000", 2024-03-11T21:01:04.353 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.5899999999999996e-06 2024-03-11T21:01:04.354 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.354 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.354 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:01:04.354 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:25.814645+0000", 2024-03-11T21:01:04.354 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.2099805589999999 2024-03-11T21:01:04.354 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.354 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.354 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:01:04.354 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:25.816689+0000", 2024-03-11T21:01:04.354 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0020433679999999998 2024-03-11T21:01:04.354 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.354 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.354 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for subops from 1,5,6", 2024-03-11T21:01:04.354 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:25.817341+0000", 2024-03-11T21:01:04.354 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00065267199999999997 2024-03-11T21:01:04.355 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.355 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.355 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:01:04.355 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:25.825857+0000", 2024-03-11T21:01:04.355 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0085154789999999994 2024-03-11T21:01:04.355 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.355 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.355 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:01:04.355 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:25.837835+0000", 2024-03-11T21:01:04.355 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.011978595 2024-03-11T21:01:04.355 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.355 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.355 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "op_commit", 2024-03-11T21:01:04.355 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:25.837852+0000", 2024-03-11T21:01:04.356 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.6821999999999999e-05 2024-03-11T21:01:04.356 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.356 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.356 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:01:04.356 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:25.837893+0000", 2024-03-11T21:01:04.356 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 4.0439999999999999e-05 2024-03-11T21:01:04.356 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.356 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.356 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T21:01:04.356 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:25.837959+0000", 2024-03-11T21:01:04.356 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.6190000000000002e-05 2024-03-11T21:01:04.356 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.356 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.357 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:01:04.357 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:25.838082+0000", 2024-03-11T21:01:04.357 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000123683 2024-03-11T21:01:04.357 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:01:04.357 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:01:04.357 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:01:04.357 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.357 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.357 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_op(client.24281.0:1928 4.d 4.a204812d (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-03-11T21:01:04.357 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T20:58:53.171214+0000", 2024-03-11T21:01:04.357 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 131.07520985400001, 2024-03-11T21:01:04.357 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.41157037000000002, 2024-03-11T21:01:04.357 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:01:04.357 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "reached pg", 2024-03-11T21:01:04.357 INFO:teuthology.orchestra.run.smithi023.stdout: "client_info": { 2024-03-11T21:01:04.358 INFO:teuthology.orchestra.run.smithi023.stdout: "client": "client.24281", 2024-03-11T21:01:04.358 INFO:teuthology.orchestra.run.smithi023.stdout: "client_addr": "172.21.15.23:0/2185274774", 2024-03-11T21:01:04.358 INFO:teuthology.orchestra.run.smithi023.stdout: "tid": 1928 2024-03-11T21:01:04.358 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.358 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:01:04.358 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.358 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:01:04.358 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:58:53.171214+0000", 2024-03-11T21:01:04.358 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:01:04.358 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.358 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.358 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:01:04.358 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:58:53.171214+0000", 2024-03-11T21:01:04.359 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:01:04.359 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.359 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.359 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:01:04.359 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:58:53.171216+0000", 2024-03-11T21:01:04.359 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.0980000000000001e-06 2024-03-11T21:01:04.359 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.359 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.359 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:01:04.359 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:58:53.171224+0000", 2024-03-11T21:01:04.359 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 8.4689999999999997e-06 2024-03-11T21:01:04.359 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.359 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.359 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:01:04.359 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:58:53.171248+0000", 2024-03-11T21:01:04.360 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.4068999999999999e-05 2024-03-11T21:01:04.360 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.360 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.360 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:01:04.360 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:58:53.171262+0000", 2024-03-11T21:01:04.360 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.3465000000000001e-05 2024-03-11T21:01:04.360 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.360 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.360 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:01:04.360 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:58:53.171353+0000", 2024-03-11T21:01:04.360 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 9.0943e-05 2024-03-11T21:01:04.360 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.360 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.360 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for peered", 2024-03-11T21:01:04.361 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:58:53.171361+0000", 2024-03-11T21:01:04.361 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 8.5459999999999992e-06 2024-03-11T21:01:04.361 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.361 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.361 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:01:04.361 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:58:53.582528+0000", 2024-03-11T21:01:04.361 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.41116678099999998 2024-03-11T21:01:04.361 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:04.361 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:04.361 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:01:04.361 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:58:53.582784+0000", 2024-03-11T21:01:04.361 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00025599900000000003 2024-03-11T21:01:04.361 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:01:04.361 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:01:04.361 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:01:04.362 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:01:04.362 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:01:04.362 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-11T21:01:04.362 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:04.383 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:04.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:04.385 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:01:04.486 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:04.505 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:04.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:04.507 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:01:04.607 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:01:04.631 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:01:04.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:04.634 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:01:04.734 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:04.785 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:04.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:04.792 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:01:04.893 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:04.919 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:04.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:04.922 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:01:05.022 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:05.055 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:05.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:05.057 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:01:05.158 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:05.178 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:05.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:05.180 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:01:05.236 INFO:teuthology.orchestra.run.smithi023.stderr:nodeep-scrub is set 2024-03-11T21:01:05.281 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:01:05.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:05 smithi023 bash[21372]: cluster 2024-03-11T21:01:04.096537+0000 mgr.y (mgr.24380) 381 : cluster 0 pgmap v400: 148 pgs: 141 active+clean, 3 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped; 3.0 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 852 B/s rd, 44 MiB/s wr, 698 op/s; 16671/150360 objects degraded (11.087%); 5.6 MiB/s, 89 objects/s recovering 2024-03-11T21:01:05.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:05 smithi023 bash[21372]: audit 2024-03-11T21:01:04.257977+0000 mon.c (mon.2) 50 : audit 1 from='client.? 172.21.15.23:0/130160731' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:01:05.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:05 smithi023 bash[21372]: audit 2024-03-11T21:01:04.259016+0000 mon.a (mon.0) 1314 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:01:05.286 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:05 smithi023 bash[21372]: audit 2024-03-11T21:01:04.915900+0000 mon.a (mon.0) 1315 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:01:05.286 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:05 smithi023 bash[21372]: cluster 2024-03-11T21:01:05.149869+0000 mon.a (mon.0) 1316 : cluster 3 Health check update: Degraded data redundancy: 16671/150360 objects degraded (11.087%), 7 pgs degraded (PG_DEGRADED) 2024-03-11T21:01:05.286 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:05 smithi023 bash[29146]: cluster 2024-03-11T21:01:04.096537+0000 mgr.y (mgr.24380) 381 : cluster 0 pgmap v400: 148 pgs: 141 active+clean, 3 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped; 3.0 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 852 B/s rd, 44 MiB/s wr, 698 op/s; 16671/150360 objects degraded (11.087%); 5.6 MiB/s, 89 objects/s recovering 2024-03-11T21:01:05.286 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:05 smithi023 bash[29146]: audit 2024-03-11T21:01:04.257977+0000 mon.c (mon.2) 50 : audit 1 from='client.? 172.21.15.23:0/130160731' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:01:05.286 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:05 smithi023 bash[29146]: audit 2024-03-11T21:01:04.259016+0000 mon.a (mon.0) 1314 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:01:05.286 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:05 smithi023 bash[29146]: audit 2024-03-11T21:01:04.915900+0000 mon.a (mon.0) 1315 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:01:05.286 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:05 smithi023 bash[29146]: cluster 2024-03-11T21:01:05.149869+0000 mon.a (mon.0) 1316 : cluster 3 Health check update: Degraded data redundancy: 16671/150360 objects degraded (11.087%), 7 pgs degraded (PG_DEGRADED) 2024-03-11T21:01:05.317 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:01:05.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:05.323 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:01:05.424 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:01:05.445 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:01:05.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:05.447 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:01:05.512 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:01:05 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:01:05.513 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:05 smithi146 bash[23809]: cluster 2024-03-11T21:01:04.096537+0000 mgr.y (mgr.24380) 381 : cluster 0 pgmap v400: 148 pgs: 141 active+clean, 3 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped; 3.0 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 852 B/s rd, 44 MiB/s wr, 698 op/s; 16671/150360 objects degraded (11.087%); 5.6 MiB/s, 89 objects/s recovering 2024-03-11T21:01:05.513 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:05 smithi146 bash[23809]: audit 2024-03-11T21:01:04.257977+0000 mon.c (mon.2) 50 : audit 1 from='client.? 172.21.15.23:0/130160731' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:01:05.513 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:05 smithi146 bash[23809]: audit 2024-03-11T21:01:04.259016+0000 mon.a (mon.0) 1314 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:01:05.513 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:05 smithi146 bash[23809]: audit 2024-03-11T21:01:04.915900+0000 mon.a (mon.0) 1315 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:01:05.513 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:05 smithi146 bash[23809]: cluster 2024-03-11T21:01:05.149869+0000 mon.a (mon.0) 1316 : cluster 3 Health check update: Degraded data redundancy: 16671/150360 objects degraded (11.087%), 7 pgs degraded (PG_DEGRADED) 2024-03-11T21:01:05.547 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:05.575 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:05.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:05.577 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:01:05.678 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:01:05.700 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:01:05.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:05.702 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:01:05.744 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T21:01:05.803 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:01:05.830 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:01:05.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:05.832 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:01:05.933 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:01:05.955 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:01:05.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:05.959 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:01:06.060 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:06.086 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:06.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:06.096 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:01:06.196 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:06.218 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:06.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:06.220 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:01:06.321 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:01:06.331 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:06 smithi023 bash[21372]: cluster 2024-03-11T21:01:05.204543+0000 mon.a (mon.0) 1317 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:01:06.331 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:06 smithi023 bash[21372]: audit 2024-03-11T21:01:05.220155+0000 mon.a (mon.0) 1318 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:01:06.331 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:06 smithi023 bash[21372]: cluster 2024-03-11T21:01:05.229171+0000 mon.a (mon.0) 1319 : cluster 0 osdmap e214: 8 total, 8 up, 8 in 2024-03-11T21:01:06.331 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:06 smithi023 bash[21372]: audit 2024-03-11T21:01:05.316717+0000 mgr.y (mgr.24380) 382 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:01:06.331 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:06 smithi023 bash[29146]: cluster 2024-03-11T21:01:05.204543+0000 mon.a (mon.0) 1317 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:01:06.332 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:06 smithi023 bash[29146]: audit 2024-03-11T21:01:05.220155+0000 mon.a (mon.0) 1318 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:01:06.332 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:06 smithi023 bash[29146]: cluster 2024-03-11T21:01:05.229171+0000 mon.a (mon.0) 1319 : cluster 0 osdmap e214: 8 total, 8 up, 8 in 2024-03-11T21:01:06.332 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:06 smithi023 bash[29146]: audit 2024-03-11T21:01:05.316717+0000 mgr.y (mgr.24380) 382 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:01:06.359 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:01:06.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:06.366 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:01:06.467 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:01:06.494 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:01:06.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:06.499 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:01:06.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:06 smithi146 bash[23809]: cluster 2024-03-11T21:01:05.204543+0000 mon.a (mon.0) 1317 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:01:06.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:06 smithi146 bash[23809]: audit 2024-03-11T21:01:05.220155+0000 mon.a (mon.0) 1318 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:01:06.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:06 smithi146 bash[23809]: cluster 2024-03-11T21:01:05.229171+0000 mon.a (mon.0) 1319 : cluster 0 osdmap e214: 8 total, 8 up, 8 in 2024-03-11T21:01:06.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:06 smithi146 bash[23809]: audit 2024-03-11T21:01:05.316717+0000 mgr.y (mgr.24380) 382 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:01:06.600 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:01:06.622 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:01:06.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:06.625 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:01:06.726 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:06.753 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:06.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:06.756 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:01:06.857 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:01:06.884 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:01:06.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:06.886 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:01:06.987 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:07.014 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:07.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:07.017 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:01:07.118 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:01:07.150 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:01:07.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:07.153 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:01:07.254 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:07.278 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:07.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:07.280 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:01:07.342 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-11T21:01:07.381 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:01:07.399 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:07 smithi023 bash[21372]: cluster 2024-03-11T21:01:06.097227+0000 mgr.y (mgr.24380) 383 : cluster 0 pgmap v402: 148 pgs: 141 active+clean, 3 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped; 2.9 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 1.1 KiB/s rd, 26 MiB/s wr, 709 op/s; 16000/141846 objects degraded (11.280%); 7.7 MiB/s, 123 objects/s recovering 2024-03-11T21:01:07.399 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:07 smithi023 bash[29146]: cluster 2024-03-11T21:01:06.097227+0000 mgr.y (mgr.24380) 383 : cluster 0 pgmap v402: 148 pgs: 141 active+clean, 3 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped; 2.9 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 1.1 KiB/s rd, 26 MiB/s wr, 709 op/s; 16000/141846 objects degraded (11.280%); 7.7 MiB/s, 123 objects/s recovering 2024-03-11T21:01:07.429 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:01:07.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:07.434 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:01:07.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:07 smithi146 bash[23809]: cluster 2024-03-11T21:01:06.097227+0000 mgr.y (mgr.24380) 383 : cluster 0 pgmap v402: 148 pgs: 141 active+clean, 3 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped; 2.9 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 1.1 KiB/s rd, 26 MiB/s wr, 709 op/s; 16000/141846 objects degraded (11.280%); 7.7 MiB/s, 123 objects/s recovering 2024-03-11T21:01:07.535 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:01:07.555 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:01:07.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:07.558 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:01:07.659 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:07.678 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:07.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:07.682 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:01:07.783 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:07.810 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:07.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:07.814 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:01:07.915 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:01:07.952 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:01:07.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:07.956 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:01:08.057 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:08.077 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:08.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:08.079 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:01:08.180 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:08.211 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:08.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:08.215 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:01:08.316 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:01:08.346 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:01:08.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:08.350 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:01:08.451 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:08.471 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:08.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:08.473 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:01:08.535 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:01:08 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:01:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:01:08.574 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:08.597 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:08.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:08.599 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:01:08.700 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:01:08.732 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:01:08.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:08.737 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:01:08.838 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:08.858 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:08.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:08.860 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:01:08.961 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:01:08.983 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:01:08.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:08.985 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:01:09.086 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:09.120 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:09.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:09.122 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:01:09.223 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:01:09.256 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:01:09.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:09.262 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:01:09.363 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:01:09.374 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:09 smithi023 bash[21372]: cluster 2024-03-11T21:01:08.097817+0000 mgr.y (mgr.24380) 384 : cluster 0 pgmap v403: 148 pgs: 141 active+clean, 3 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped; 2.9 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 1.1 KiB/s rd, 26 MiB/s wr, 709 op/s; 16000/141846 objects degraded (11.280%); 7.7 MiB/s, 123 objects/s recovering 2024-03-11T21:01:09.374 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:09 smithi023 bash[21372]: cluster 2024-03-11T21:01:08.498627+0000 osd.5 (osd.5) 201 : cluster 0 purged_snaps scrub starts 2024-03-11T21:01:09.374 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:09 smithi023 bash[21372]: cluster 2024-03-11T21:01:08.498689+0000 osd.5 (osd.5) 202 : cluster 0 purged_snaps scrub ok 2024-03-11T21:01:09.374 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:09 smithi023 bash[29146]: cluster 2024-03-11T21:01:08.097817+0000 mgr.y (mgr.24380) 384 : cluster 0 pgmap v403: 148 pgs: 141 active+clean, 3 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped; 2.9 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 1.1 KiB/s rd, 26 MiB/s wr, 709 op/s; 16000/141846 objects degraded (11.280%); 7.7 MiB/s, 123 objects/s recovering 2024-03-11T21:01:09.374 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:09 smithi023 bash[29146]: cluster 2024-03-11T21:01:08.498627+0000 osd.5 (osd.5) 201 : cluster 0 purged_snaps scrub starts 2024-03-11T21:01:09.374 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:09 smithi023 bash[29146]: cluster 2024-03-11T21:01:08.498689+0000 osd.5 (osd.5) 202 : cluster 0 purged_snaps scrub ok 2024-03-11T21:01:09.410 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:01:09.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:09.416 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:01:09.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:09 smithi146 bash[23809]: cluster 2024-03-11T21:01:08.097817+0000 mgr.y (mgr.24380) 384 : cluster 0 pgmap v403: 148 pgs: 141 active+clean, 3 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped; 2.9 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 1.1 KiB/s rd, 26 MiB/s wr, 709 op/s; 16000/141846 objects degraded (11.280%); 7.7 MiB/s, 123 objects/s recovering 2024-03-11T21:01:09.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:09 smithi146 bash[23809]: cluster 2024-03-11T21:01:08.498627+0000 osd.5 (osd.5) 201 : cluster 0 purged_snaps scrub starts 2024-03-11T21:01:09.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:09 smithi146 bash[23809]: cluster 2024-03-11T21:01:08.498689+0000 osd.5 (osd.5) 202 : cluster 0 purged_snaps scrub ok 2024-03-11T21:01:09.517 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:01:09.556 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:01:09.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:09.561 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:01:09.662 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:01:09.686 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:01:09.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:09.687 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:01:09.788 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:01:09.818 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:01:09.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:09.821 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:01:09.922 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:01:09.944 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:01:09.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:09.946 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:01:10.046 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:01:10.073 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:01:10.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:10.077 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:01:10.178 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:10.201 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:10.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:10.203 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:01:10.304 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:10.326 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:10.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:10.329 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:01:10.430 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd unset noscrub 2024-03-11T21:01:10.430 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:01:10.440 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:10 smithi023 bash[21372]: cluster 2024-03-11T21:01:09.241364+0000 mon.a (mon.0) 1320 : cluster 0 osdmap e215: 8 total, 8 up, 8 in 2024-03-11T21:01:10.440 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:10 smithi023 bash[21372]: cluster 2024-03-11T21:01:10.150736+0000 mon.a (mon.0) 1321 : cluster 3 Health check update: Degraded data redundancy: 16000/141846 objects degraded (11.280%), 7 pgs degraded (PG_DEGRADED) 2024-03-11T21:01:10.442 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:10 smithi023 bash[29146]: cluster 2024-03-11T21:01:09.241364+0000 mon.a (mon.0) 1320 : cluster 0 osdmap e215: 8 total, 8 up, 8 in 2024-03-11T21:01:10.442 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:10 smithi023 bash[29146]: cluster 2024-03-11T21:01:10.150736+0000 mon.a (mon.0) 1321 : cluster 3 Health check update: Degraded data redundancy: 16000/141846 objects degraded (11.280%), 7 pgs degraded (PG_DEGRADED) 2024-03-11T21:01:10.458 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:01:10.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:10.463 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:01:10.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:10 smithi146 bash[23809]: cluster 2024-03-11T21:01:09.241364+0000 mon.a (mon.0) 1320 : cluster 0 osdmap e215: 8 total, 8 up, 8 in 2024-03-11T21:01:10.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:10 smithi146 bash[23809]: cluster 2024-03-11T21:01:10.150736+0000 mon.a (mon.0) 1321 : cluster 3 Health check update: Degraded data redundancy: 16000/141846 objects degraded (11.280%), 7 pgs degraded (PG_DEGRADED) 2024-03-11T21:01:10.564 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:01:10.615 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:01:10.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:10.619 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:01:10.720 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:01:10.761 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:01:10.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:10.765 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:01:10.866 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:01:10.886 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:01:10.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:10.888 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:01:10.988 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:11.010 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:11.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:11.011 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:01:11.112 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:01:11.143 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:01:11.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:11.146 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:01:11.247 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:11.270 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:11.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:11.272 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:01:11.373 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:11.382 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:11 smithi023 bash[21372]: cluster 2024-03-11T21:01:10.098549+0000 mgr.y (mgr.24380) 385 : cluster 0 pgmap v405: 148 pgs: 141 active+clean, 3 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped; 2.8 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 767 B/s rd, 1.2 MiB/s wr, 634 op/s; 16000/136110 objects degraded (11.755%); 5.2 MiB/s, 83 objects/s recovering 2024-03-11T21:01:11.382 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:11 smithi023 bash[29146]: cluster 2024-03-11T21:01:10.098549+0000 mgr.y (mgr.24380) 385 : cluster 0 pgmap v405: 148 pgs: 141 active+clean, 3 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped; 2.8 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 767 B/s rd, 1.2 MiB/s wr, 634 op/s; 16000/136110 objects degraded (11.755%); 5.2 MiB/s, 83 objects/s recovering 2024-03-11T21:01:11.400 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:11.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:11.408 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:01:11.508 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:01:11.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:11 smithi146 bash[23809]: cluster 2024-03-11T21:01:10.098549+0000 mgr.y (mgr.24380) 385 : cluster 0 pgmap v405: 148 pgs: 141 active+clean, 3 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped; 2.8 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 767 B/s rd, 1.2 MiB/s wr, 634 op/s; 16000/136110 objects degraded (11.755%); 5.2 MiB/s, 83 objects/s recovering 2024-03-11T21:01:11.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:11.542 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:01:11.542 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:01:11.643 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:01:11.678 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:01:11.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:11.682 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:01:11.783 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:11.814 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:11.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:11.818 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:01:11.919 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:01:11.941 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:01:11.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:11.943 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:01:11.946 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:01:12.044 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:01:12.063 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:01:12.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:12.066 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:01:12.167 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:01:12.201 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:01:12.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:12.206 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:01:12.306 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:01:12.331 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:01:12.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:12.333 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:01:12.433 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:12.455 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:12.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:12.457 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:01:12.558 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:12.580 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:12.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:12.585 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:01:12.686 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:12.707 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:12.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:12.711 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:01:12.812 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:01:12.840 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:01:12.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:12.844 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:01:12.945 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:01:12.972 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:01:12.972 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:01:12.978 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:01:12.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:12.988 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:01:13.089 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:01:13.106 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:01:13.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:13.108 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:01:13.209 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:13.253 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:13.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:13.259 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:01:13.359 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:13.383 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:13.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:13.385 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:01:13.486 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:01:13.514 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:01:13.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:13.515 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:01:13.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:13 smithi023 bash[21372]: cluster 2024-03-11T21:01:11.808281+0000 osd.3 (osd.3) 242 : cluster 0 purged_snaps scrub starts 2024-03-11T21:01:13.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:13 smithi023 bash[21372]: cluster 2024-03-11T21:01:11.808327+0000 osd.3 (osd.3) 243 : cluster 0 purged_snaps scrub ok 2024-03-11T21:01:13.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:13 smithi023 bash[21372]: cluster 2024-03-11T21:01:11.943372+0000 osd.2 (osd.2) 152 : cluster 0 purged_snaps scrub starts 2024-03-11T21:01:13.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:13 smithi023 bash[21372]: cluster 2024-03-11T21:01:11.943422+0000 osd.2 (osd.2) 153 : cluster 0 purged_snaps scrub ok 2024-03-11T21:01:13.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:13 smithi023 bash[21372]: cluster 2024-03-11T21:01:12.099392+0000 mgr.y (mgr.24380) 386 : cluster 0 pgmap v406: 148 pgs: 141 active+clean, 3 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped; 2.6 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 MiB/s wr, 957 op/s; 15364/128379 objects degraded (11.968%); 10 MiB/s, 163 objects/s recovering 2024-03-11T21:01:13.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:13 smithi023 bash[21372]: cluster 2024-03-11T21:01:12.294260+0000 mon.a (mon.0) 1322 : cluster 0 osdmap e216: 8 total, 8 up, 8 in 2024-03-11T21:01:13.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:13 smithi023 bash[29146]: cluster 2024-03-11T21:01:11.808281+0000 osd.3 (osd.3) 242 : cluster 0 purged_snaps scrub starts 2024-03-11T21:01:13.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:13 smithi023 bash[29146]: cluster 2024-03-11T21:01:11.808327+0000 osd.3 (osd.3) 243 : cluster 0 purged_snaps scrub ok 2024-03-11T21:01:13.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:13 smithi023 bash[29146]: cluster 2024-03-11T21:01:11.943372+0000 osd.2 (osd.2) 152 : cluster 0 purged_snaps scrub starts 2024-03-11T21:01:13.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:13 smithi023 bash[29146]: cluster 2024-03-11T21:01:11.943422+0000 osd.2 (osd.2) 153 : cluster 0 purged_snaps scrub ok 2024-03-11T21:01:13.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:13 smithi023 bash[29146]: cluster 2024-03-11T21:01:12.099392+0000 mgr.y (mgr.24380) 386 : cluster 0 pgmap v406: 148 pgs: 141 active+clean, 3 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped; 2.6 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 MiB/s wr, 957 op/s; 15364/128379 objects degraded (11.968%); 10 MiB/s, 163 objects/s recovering 2024-03-11T21:01:13.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:13 smithi023 bash[29146]: cluster 2024-03-11T21:01:12.294260+0000 mon.a (mon.0) 1322 : cluster 0 osdmap e216: 8 total, 8 up, 8 in 2024-03-11T21:01:13.616 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:01:13.626 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:13 smithi146 bash[23809]: cluster 2024-03-11T21:01:11.808281+0000 osd.3 (osd.3) 242 : cluster 0 purged_snaps scrub starts 2024-03-11T21:01:13.626 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:13 smithi146 bash[23809]: cluster 2024-03-11T21:01:11.808327+0000 osd.3 (osd.3) 243 : cluster 0 purged_snaps scrub ok 2024-03-11T21:01:13.626 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:13 smithi146 bash[23809]: cluster 2024-03-11T21:01:11.943372+0000 osd.2 (osd.2) 152 : cluster 0 purged_snaps scrub starts 2024-03-11T21:01:13.626 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:13 smithi146 bash[23809]: cluster 2024-03-11T21:01:11.943422+0000 osd.2 (osd.2) 153 : cluster 0 purged_snaps scrub ok 2024-03-11T21:01:13.626 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:13 smithi146 bash[23809]: cluster 2024-03-11T21:01:12.099392+0000 mgr.y (mgr.24380) 386 : cluster 0 pgmap v406: 148 pgs: 141 active+clean, 3 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped; 2.6 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 MiB/s wr, 957 op/s; 15364/128379 objects degraded (11.968%); 10 MiB/s, 163 objects/s recovering 2024-03-11T21:01:13.627 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:13 smithi146 bash[23809]: cluster 2024-03-11T21:01:12.294260+0000 mon.a (mon.0) 1322 : cluster 0 osdmap e216: 8 total, 8 up, 8 in 2024-03-11T21:01:13.637 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:01:13.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:13.639 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:01:13.740 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:01:13.774 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:01:13.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:13.778 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:01:13.878 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:13.901 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:13.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:13.904 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:01:14.005 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:01:14.034 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:01:14.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:14.037 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:01:14.138 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:01:14.163 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:01:14.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:14.166 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:01:14.267 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:01:14.299 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:01:14.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:14.302 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:01:14.402 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:01:14.432 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:01:14.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:14.435 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:01:14.536 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:01:14.566 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:01:14.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:14.571 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:01:14.672 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:01:14.719 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:01:14.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:14.723 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:01:14.824 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:14.850 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:14.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:14.853 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:01:14.954 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:14.979 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:14.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:14.982 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:01:15.011 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:01:15.082 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:15.113 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:15.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:15.117 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:01:15.217 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:01:15.238 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:01:15.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:15.240 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:01:15.341 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:01:15.366 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:01:15.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:15.368 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:01:15.469 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:01:15.479 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:15 smithi023 bash[21372]: cluster 2024-03-11T21:01:14.100301+0000 mgr.y (mgr.24380) 387 : cluster 0 pgmap v408: 148 pgs: 141 active+clean, 3 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped; 2.6 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 511 B/s rd, 0 B/s wr, 561 op/s; 15364/128379 objects degraded (11.968%); 5.0 MiB/s, 79 objects/s recovering 2024-03-11T21:01:15.480 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:15 smithi023 bash[21372]: cluster 2024-03-11T21:01:15.151387+0000 mon.a (mon.0) 1323 : cluster 3 Health check update: Degraded data redundancy: 15364/128379 objects degraded (11.968%), 7 pgs degraded (PG_DEGRADED) 2024-03-11T21:01:15.480 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:15 smithi023 bash[21372]: cluster 2024-03-11T21:01:15.155922+0000 mon.a (mon.0) 1324 : cluster 0 osdmap e217: 8 total, 8 up, 8 in 2024-03-11T21:01:15.480 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:15 smithi023 bash[29146]: cluster 2024-03-11T21:01:14.100301+0000 mgr.y (mgr.24380) 387 : cluster 0 pgmap v408: 148 pgs: 141 active+clean, 3 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped; 2.6 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 511 B/s rd, 0 B/s wr, 561 op/s; 15364/128379 objects degraded (11.968%); 5.0 MiB/s, 79 objects/s recovering 2024-03-11T21:01:15.480 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:15 smithi023 bash[29146]: cluster 2024-03-11T21:01:15.151387+0000 mon.a (mon.0) 1323 : cluster 3 Health check update: Degraded data redundancy: 15364/128379 objects degraded (11.968%), 7 pgs degraded (PG_DEGRADED) 2024-03-11T21:01:15.480 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:15 smithi023 bash[29146]: cluster 2024-03-11T21:01:15.155922+0000 mon.a (mon.0) 1324 : cluster 0 osdmap e217: 8 total, 8 up, 8 in 2024-03-11T21:01:15.494 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:01:15.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:15.496 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:01:15.597 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:01:15.605 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:01:15 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:01:15.606 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:15 smithi146 bash[23809]: cluster 2024-03-11T21:01:14.100301+0000 mgr.y (mgr.24380) 387 : cluster 0 pgmap v408: 148 pgs: 141 active+clean, 3 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped; 2.6 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 511 B/s rd, 0 B/s wr, 561 op/s; 15364/128379 objects degraded (11.968%); 5.0 MiB/s, 79 objects/s recovering 2024-03-11T21:01:15.606 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:15 smithi146 bash[23809]: cluster 2024-03-11T21:01:15.151387+0000 mon.a (mon.0) 1323 : cluster 3 Health check update: Degraded data redundancy: 15364/128379 objects degraded (11.968%), 7 pgs degraded (PG_DEGRADED) 2024-03-11T21:01:15.606 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:15 smithi146 bash[23809]: cluster 2024-03-11T21:01:15.155922+0000 mon.a (mon.0) 1324 : cluster 0 osdmap e217: 8 total, 8 up, 8 in 2024-03-11T21:01:15.618 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:01:15.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:15.620 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:01:15.721 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:15.755 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:15.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:15.763 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:01:15.863 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:15.884 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:15.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:15.888 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:01:15.989 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:01:16.017 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:01:16.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:16.019 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:01:16.120 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:16.144 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:16.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:16.147 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:01:16.248 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:01:16.278 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:01:16.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:16.281 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:01:16.381 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:01:16.408 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:01:16.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:16.412 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:01:16.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:16 smithi146 bash[23809]: audit 2024-03-11T21:01:15.326669+0000 mgr.y (mgr.24380) 388 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:01:16.513 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:01:16.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:16 smithi023 bash[21372]: audit 2024-03-11T21:01:15.326669+0000 mgr.y (mgr.24380) 388 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:01:16.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:16 smithi023 bash[29146]: audit 2024-03-11T21:01:15.326669+0000 mgr.y (mgr.24380) 388 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:01:16.537 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:01:16.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:16.540 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:01:16.640 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:16.667 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:16.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:16.670 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:01:16.771 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:01:16.817 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:01:16.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:16.823 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:01:16.923 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:16.951 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:16.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:16.954 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:01:17.055 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:01:17.081 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:01:17.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:17.083 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:01:17.184 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:01:17.204 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:01:17.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:17.206 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:01:17.307 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:17.326 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:17.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:17.328 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:01:17.429 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:01:17.459 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:01:17.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:17.462 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:01:17.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:17 smithi146 bash[23809]: cluster 2024-03-11T21:01:16.101227+0000 mgr.y (mgr.24380) 389 : cluster 0 pgmap v410: 148 pgs: 141 active+clean, 3 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped; 2.3 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.31k op/s; 14656/114807 objects degraded (12.766%); 12 MiB/s, 195 objects/s recovering 2024-03-11T21:01:17.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:17 smithi146 bash[23809]: cluster 2024-03-11T21:01:16.190277+0000 osd.6 (osd.6) 146 : cluster 0 purged_snaps scrub starts 2024-03-11T21:01:17.513 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:17 smithi146 bash[23809]: cluster 2024-03-11T21:01:16.190343+0000 osd.6 (osd.6) 147 : cluster 0 purged_snaps scrub ok 2024-03-11T21:01:17.513 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:17 smithi146 bash[23809]: cluster 2024-03-11T21:01:16.328673+0000 mon.a (mon.0) 1325 : cluster 0 osdmap e218: 8 total, 8 up, 8 in 2024-03-11T21:01:17.562 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:17.585 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:17.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:17.587 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:01:17.688 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:17.699 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:17 smithi023 bash[21372]: cluster 2024-03-11T21:01:16.101227+0000 mgr.y (mgr.24380) 389 : cluster 0 pgmap v410: 148 pgs: 141 active+clean, 3 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped; 2.3 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.31k op/s; 14656/114807 objects degraded (12.766%); 12 MiB/s, 195 objects/s recovering 2024-03-11T21:01:17.700 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:17 smithi023 bash[21372]: cluster 2024-03-11T21:01:16.190277+0000 osd.6 (osd.6) 146 : cluster 0 purged_snaps scrub starts 2024-03-11T21:01:17.700 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:17 smithi023 bash[21372]: cluster 2024-03-11T21:01:16.190343+0000 osd.6 (osd.6) 147 : cluster 0 purged_snaps scrub ok 2024-03-11T21:01:17.700 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:17 smithi023 bash[21372]: cluster 2024-03-11T21:01:16.328673+0000 mon.a (mon.0) 1325 : cluster 0 osdmap e218: 8 total, 8 up, 8 in 2024-03-11T21:01:17.700 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:17 smithi023 bash[29146]: cluster 2024-03-11T21:01:16.101227+0000 mgr.y (mgr.24380) 389 : cluster 0 pgmap v410: 148 pgs: 141 active+clean, 3 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped; 2.3 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.31k op/s; 14656/114807 objects degraded (12.766%); 12 MiB/s, 195 objects/s recovering 2024-03-11T21:01:17.700 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:17 smithi023 bash[29146]: cluster 2024-03-11T21:01:16.190277+0000 osd.6 (osd.6) 146 : cluster 0 purged_snaps scrub starts 2024-03-11T21:01:17.700 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:17 smithi023 bash[29146]: cluster 2024-03-11T21:01:16.190343+0000 osd.6 (osd.6) 147 : cluster 0 purged_snaps scrub ok 2024-03-11T21:01:17.700 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:17 smithi023 bash[29146]: cluster 2024-03-11T21:01:16.328673+0000 mon.a (mon.0) 1325 : cluster 0 osdmap e218: 8 total, 8 up, 8 in 2024-03-11T21:01:17.719 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:17.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:17.721 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:01:17.822 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:17.845 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:17.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:17.847 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:01:17.947 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:01:17.977 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:01:17.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:17.980 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:01:18.081 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:18.108 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:18.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:18.110 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:01:18.211 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:01:18.240 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:01:18.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:18.243 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:01:18.344 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:01:18.378 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:01:18.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:18.382 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:01:18.483 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:01:18.492 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:01:18 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:01:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:01:18.511 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:01:18.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:18.514 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:01:18.615 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:01:18.646 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:01:18.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:18.649 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:01:18.750 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:18.776 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:18.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:18.780 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:01:18.881 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:18.901 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:18.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:18.903 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:01:19.003 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:01:19.027 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:01:19.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:19.032 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:01:19.132 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:01:19.167 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:01:19.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:19.169 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:01:19.270 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:19.294 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:19.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:19.296 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:01:19.397 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:01:19.413 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:19 smithi023 bash[21372]: cluster 2024-03-11T21:01:18.101840+0000 mgr.y (mgr.24380) 390 : cluster 0 pgmap v412: 148 pgs: 141 active+clean, 3 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped; 2.3 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 754 op/s; 14656/114807 objects degraded (12.766%); 7.4 MiB/s, 117 objects/s recovering 2024-03-11T21:01:19.414 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:19 smithi023 bash[29146]: cluster 2024-03-11T21:01:18.101840+0000 mgr.y (mgr.24380) 390 : cluster 0 pgmap v412: 148 pgs: 141 active+clean, 3 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped; 2.3 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 754 op/s; 14656/114807 objects degraded (12.766%); 7.4 MiB/s, 117 objects/s recovering 2024-03-11T21:01:19.423 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:01:19.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:19.426 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:01:19.527 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:19.535 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:19 smithi146 bash[23809]: cluster 2024-03-11T21:01:18.101840+0000 mgr.y (mgr.24380) 390 : cluster 0 pgmap v412: 148 pgs: 141 active+clean, 3 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped; 2.3 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 754 op/s; 14656/114807 objects degraded (12.766%); 7.4 MiB/s, 117 objects/s recovering 2024-03-11T21:01:19.549 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:19.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:19.551 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:01:19.652 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:19.676 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:19.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:19.678 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:01:19.779 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:01:19.797 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:01:19.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:19.799 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:01:19.900 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:01:19.924 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:01:19.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:19.928 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:01:20.029 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:01:20.061 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:01:20.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:20.063 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:01:20.164 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:01:20.184 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:01:20.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:20.186 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:01:20.287 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:20.315 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:20.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:20.324 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:01:20.425 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:20.457 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:20.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:20.462 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:01:20.513 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:20 smithi146 bash[23809]: audit 2024-03-11T21:01:19.942829+0000 mon.a (mon.0) 1326 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:01:20.513 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:20 smithi146 bash[23809]: cluster 2024-03-11T21:01:20.152671+0000 mon.a (mon.0) 1327 : cluster 3 Health check update: Degraded data redundancy: 14656/114807 objects degraded (12.766%), 7 pgs degraded (PG_DEGRADED) 2024-03-11T21:01:20.513 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:20 smithi146 bash[23809]: cluster 2024-03-11T21:01:20.161472+0000 mon.a (mon.0) 1328 : cluster 0 osdmap e219: 8 total, 8 up, 8 in 2024-03-11T21:01:20.563 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:20.588 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:20.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:20.590 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:01:20.656 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:20 smithi023 bash[29146]: audit 2024-03-11T21:01:19.942829+0000 mon.a (mon.0) 1326 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:01:20.656 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:20 smithi023 bash[29146]: cluster 2024-03-11T21:01:20.152671+0000 mon.a (mon.0) 1327 : cluster 3 Health check update: Degraded data redundancy: 14656/114807 objects degraded (12.766%), 7 pgs degraded (PG_DEGRADED) 2024-03-11T21:01:20.656 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:20 smithi023 bash[29146]: cluster 2024-03-11T21:01:20.161472+0000 mon.a (mon.0) 1328 : cluster 0 osdmap e219: 8 total, 8 up, 8 in 2024-03-11T21:01:20.656 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:20 smithi023 bash[21372]: audit 2024-03-11T21:01:19.942829+0000 mon.a (mon.0) 1326 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:01:20.656 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:20 smithi023 bash[21372]: cluster 2024-03-11T21:01:20.152671+0000 mon.a (mon.0) 1327 : cluster 3 Health check update: Degraded data redundancy: 14656/114807 objects degraded (12.766%), 7 pgs degraded (PG_DEGRADED) 2024-03-11T21:01:20.657 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:20 smithi023 bash[21372]: cluster 2024-03-11T21:01:20.161472+0000 mon.a (mon.0) 1328 : cluster 0 osdmap e219: 8 total, 8 up, 8 in 2024-03-11T21:01:20.690 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:01:20.724 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:01:20.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:20.730 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:01:20.830 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:01:20.862 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:01:20.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:20.868 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:01:20.968 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:20.991 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:20.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:20.993 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:01:21.094 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:01:21.119 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:01:21.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:21.122 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:01:21.223 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:21.247 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:21.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:21.249 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:01:21.350 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:01:21.393 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:01:21.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:21.396 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:01:21.497 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:21.505 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:21 smithi146 bash[23809]: cluster 2024-03-11T21:01:20.102568+0000 mgr.y (mgr.24380) 391 : cluster 0 pgmap v413: 148 pgs: 141 active+clean, 3 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped; 2.2 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 655 B/s rd, 0 B/s wr, 820 op/s; 14656/109161 objects degraded (13.426%); 5.7 MiB/s, 90 objects/s recovering 2024-03-11T21:01:21.516 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:21.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:21.519 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:01:21.620 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:01:21.630 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:21 smithi023 bash[21372]: cluster 2024-03-11T21:01:20.102568+0000 mgr.y (mgr.24380) 391 : cluster 0 pgmap v413: 148 pgs: 141 active+clean, 3 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped; 2.2 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 655 B/s rd, 0 B/s wr, 820 op/s; 14656/109161 objects degraded (13.426%); 5.7 MiB/s, 90 objects/s recovering 2024-03-11T21:01:21.630 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:21 smithi023 bash[29146]: cluster 2024-03-11T21:01:20.102568+0000 mgr.y (mgr.24380) 391 : cluster 0 pgmap v413: 148 pgs: 141 active+clean, 3 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped; 2.2 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 655 B/s rd, 0 B/s wr, 820 op/s; 14656/109161 objects degraded (13.426%); 5.7 MiB/s, 90 objects/s recovering 2024-03-11T21:01:21.650 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:01:21.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:21.652 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:01:21.753 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:01:21.786 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:01:21.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:21.790 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:01:21.890 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:01:21.921 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:01:21.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:21.925 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:01:22.026 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:22.047 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:22.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:22.049 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:01:22.150 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:01:22.179 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:01:22.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:22.180 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:01:22.281 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:22.305 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:22.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:22.306 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:01:22.407 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:01:22.454 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:01:22.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:22.458 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:01:22.559 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:01:22.583 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:01:22.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:22.587 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:01:22.687 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:01:22.718 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:01:22.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:22.723 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:01:22.824 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:01:22.847 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:01:22.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:22.850 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:01:22.951 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:22.975 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:22.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:22.977 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:01:23.078 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:01:23.104 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:01:23.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:23.111 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:01:23.212 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:23.235 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:23.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:23.238 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:01:23.338 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:23.361 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:23.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:23.365 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:01:23.453 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:23 smithi023 bash[21372]: cluster 2024-03-11T21:01:22.103426+0000 mgr.y (mgr.24380) 392 : cluster 0 pgmap v415: 148 pgs: 141 active+clean, 3 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped; 2.1 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 993 op/s; 13941/101484 objects degraded (13.737%); 13 MiB/s, 204 objects/s recovering 2024-03-11T21:01:23.454 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:23 smithi023 bash[29146]: cluster 2024-03-11T21:01:22.103426+0000 mgr.y (mgr.24380) 392 : cluster 0 pgmap v415: 148 pgs: 141 active+clean, 3 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped; 2.1 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 993 op/s; 13941/101484 objects degraded (13.737%); 13 MiB/s, 204 objects/s recovering 2024-03-11T21:01:23.465 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:23.505 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:23.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:23.509 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:01:23.610 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:01:23.621 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:23 smithi146 bash[23809]: cluster 2024-03-11T21:01:22.103426+0000 mgr.y (mgr.24380) 392 : cluster 0 pgmap v415: 148 pgs: 141 active+clean, 3 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped; 2.1 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 993 op/s; 13941/101484 objects degraded (13.737%); 13 MiB/s, 204 objects/s recovering 2024-03-11T21:01:23.631 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:01:23.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:23.633 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:01:23.734 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:01:23.760 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:01:23.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:23.764 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:01:23.864 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:01:23.899 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:01:23.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:23.902 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:01:24.003 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:24.026 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:24.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:24.028 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:01:24.128 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:01:24.161 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:01:24.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:24.164 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:01:24.264 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:24.300 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:24.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:24.304 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:01:24.405 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:01:24.435 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:01:24.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:24.438 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:01:24.538 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:01:24.573 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:01:24.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:24.579 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:01:24.680 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:01:24.703 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:01:24.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:24.708 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:01:24.809 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:01:24.846 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:01:24.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:24.850 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:01:24.951 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:01:24.975 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:01:24.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:24.977 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:01:25.077 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:01:25.099 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:01:25.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:25.101 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:01:25.202 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:25.226 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:25.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:25.231 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:01:25.331 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:01:25.354 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:01:25.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:25.359 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:01:25.460 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:01:25.470 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:25 smithi023 bash[21372]: cluster 2024-03-11T21:01:24.104169+0000 mgr.y (mgr.24380) 393 : cluster 0 pgmap v416: 148 pgs: 141 active+clean, 3 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped; 2.1 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 555 op/s; 13941/101484 objects degraded (13.737%); 5.6 MiB/s, 89 objects/s recovering 2024-03-11T21:01:25.470 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:25 smithi023 bash[21372]: cluster 2024-03-11T21:01:25.154818+0000 mon.a (mon.0) 1329 : cluster 3 Health check update: Degraded data redundancy: 13941/101484 objects degraded (13.737%), 7 pgs degraded (PG_DEGRADED) 2024-03-11T21:01:25.471 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:25 smithi023 bash[29146]: cluster 2024-03-11T21:01:24.104169+0000 mgr.y (mgr.24380) 393 : cluster 0 pgmap v416: 148 pgs: 141 active+clean, 3 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped; 2.1 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 555 op/s; 13941/101484 objects degraded (13.737%); 5.6 MiB/s, 89 objects/s recovering 2024-03-11T21:01:25.471 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:25 smithi023 bash[29146]: cluster 2024-03-11T21:01:25.154818+0000 mon.a (mon.0) 1329 : cluster 3 Health check update: Degraded data redundancy: 13941/101484 objects degraded (13.737%), 7 pgs degraded (PG_DEGRADED) 2024-03-11T21:01:25.491 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:01:25.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:25.495 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:01:25.512 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:01:25 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:01:25.513 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:25 smithi146 bash[23809]: cluster 2024-03-11T21:01:24.104169+0000 mgr.y (mgr.24380) 393 : cluster 0 pgmap v416: 148 pgs: 141 active+clean, 3 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped; 2.1 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 555 op/s; 13941/101484 objects degraded (13.737%); 5.6 MiB/s, 89 objects/s recovering 2024-03-11T21:01:25.513 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:25 smithi146 bash[23809]: cluster 2024-03-11T21:01:25.154818+0000 mon.a (mon.0) 1329 : cluster 3 Health check update: Degraded data redundancy: 13941/101484 objects degraded (13.737%), 7 pgs degraded (PG_DEGRADED) 2024-03-11T21:01:25.596 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:01:25.617 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:01:25.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:25.621 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:01:25.721 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:25.739 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:25.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:25.740 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:01:25.841 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:01:25.878 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:01:25.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:25.883 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:01:25.984 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:01:26.033 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:01:26.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:26.036 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:01:26.137 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:26.182 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:26.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:26.187 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:01:26.287 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:26.318 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:26.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:26.323 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:01:26.424 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:26.457 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:26.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:26.468 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:01:26.569 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:01:26.591 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:01:26.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:26.593 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:01:26.693 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:01:26.724 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:01:26.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:26.728 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:01:26.733 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:26 smithi023 bash[21372]: audit 2024-03-11T21:01:25.334625+0000 mgr.y (mgr.24380) 394 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:01:26.733 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:26 smithi023 bash[29146]: audit 2024-03-11T21:01:25.334625+0000 mgr.y (mgr.24380) 394 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:01:26.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:26 smithi146 bash[23809]: audit 2024-03-11T21:01:25.334625+0000 mgr.y (mgr.24380) 394 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:01:26.829 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:01:26.860 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:01:26.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:26.863 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:01:26.964 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:26.995 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:26.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:26.999 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:01:27.100 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:27.133 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:27.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:27.138 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:01:27.239 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:27.257 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:27.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:27.259 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:01:27.360 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:27.389 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:27.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:27.391 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:01:27.491 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:27.505 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:27 smithi023 bash[21372]: cluster 2024-03-11T21:01:26.105453+0000 mgr.y (mgr.24380) 395 : cluster 0 pgmap v417: 148 pgs: 141 active+clean, 3 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped; 1.8 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 845 op/s; 13289/90030 objects degraded (14.761%); 8.7 MiB/s, 139 objects/s recovering 2024-03-11T21:01:27.505 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:27 smithi023 bash[29146]: cluster 2024-03-11T21:01:26.105453+0000 mgr.y (mgr.24380) 395 : cluster 0 pgmap v417: 148 pgs: 141 active+clean, 3 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped; 1.8 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 845 op/s; 13289/90030 objects degraded (14.761%); 8.7 MiB/s, 139 objects/s recovering 2024-03-11T21:01:27.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:27 smithi146 bash[23809]: cluster 2024-03-11T21:01:26.105453+0000 mgr.y (mgr.24380) 395 : cluster 0 pgmap v417: 148 pgs: 141 active+clean, 3 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped; 1.8 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 845 op/s; 13289/90030 objects degraded (14.761%); 8.7 MiB/s, 139 objects/s recovering 2024-03-11T21:01:27.523 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:27.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:27.527 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:01:27.628 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:01:27.668 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:01:27.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:27.671 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:01:27.772 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:27.801 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:27.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:27.806 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:01:27.907 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:01:27.927 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:01:27.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:27.929 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:01:28.029 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:28.056 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:28.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:28.059 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:01:28.160 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:01:28.186 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:01:28.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:28.191 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:01:28.292 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:28.365 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:28.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:28.369 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:01:28.470 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:01:28.494 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:01:28.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:28.497 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:01:28.598 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:01:28.622 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:01:28.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:28.624 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:01:28.652 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:01:28 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:01:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:01:28.724 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:28.753 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:28.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:28.757 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:01:28.858 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:01:28.880 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:01:28.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:28.886 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:01:28.987 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:01:29.014 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:01:29.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:29.017 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:01:29.118 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:01:29.142 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:01:29.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:29.144 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:01:29.245 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:01:29.278 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:01:29.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:29.283 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:01:29.384 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:29.417 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:29.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:29.420 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:01:29.520 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:29.541 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:29.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:29.543 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:01:29.585 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:29 smithi023 bash[21372]: cluster 2024-03-11T21:01:28.106085+0000 mgr.y (mgr.24380) 396 : cluster 0 pgmap v418: 148 pgs: 141 active+clean, 3 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped; 1.8 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 826 op/s; 13289/90030 objects degraded (14.761%); 8.5 MiB/s, 136 objects/s recovering 2024-03-11T21:01:29.585 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:29 smithi023 bash[21372]: cluster 2024-03-11T21:01:28.217176+0000 osd.7 (osd.7) 209 : cluster 0 purged_snaps scrub starts 2024-03-11T21:01:29.585 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:29 smithi023 bash[21372]: cluster 2024-03-11T21:01:28.217241+0000 osd.7 (osd.7) 210 : cluster 0 purged_snaps scrub ok 2024-03-11T21:01:29.585 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:29 smithi023 bash[21372]: cluster 2024-03-11T21:01:28.400414+0000 mon.a (mon.0) 1330 : cluster 0 osdmap e220: 8 total, 8 up, 8 in 2024-03-11T21:01:29.585 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:29 smithi023 bash[29146]: cluster 2024-03-11T21:01:28.106085+0000 mgr.y (mgr.24380) 396 : cluster 0 pgmap v418: 148 pgs: 141 active+clean, 3 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped; 1.8 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 826 op/s; 13289/90030 objects degraded (14.761%); 8.5 MiB/s, 136 objects/s recovering 2024-03-11T21:01:29.585 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:29 smithi023 bash[29146]: cluster 2024-03-11T21:01:28.217176+0000 osd.7 (osd.7) 209 : cluster 0 purged_snaps scrub starts 2024-03-11T21:01:29.586 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:29 smithi023 bash[29146]: cluster 2024-03-11T21:01:28.217241+0000 osd.7 (osd.7) 210 : cluster 0 purged_snaps scrub ok 2024-03-11T21:01:29.586 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:29 smithi023 bash[29146]: cluster 2024-03-11T21:01:28.400414+0000 mon.a (mon.0) 1330 : cluster 0 osdmap e220: 8 total, 8 up, 8 in 2024-03-11T21:01:29.646 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:29.656 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:29 smithi146 bash[23809]: cluster 2024-03-11T21:01:28.106085+0000 mgr.y (mgr.24380) 396 : cluster 0 pgmap v418: 148 pgs: 141 active+clean, 3 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped; 1.8 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 826 op/s; 13289/90030 objects degraded (14.761%); 8.5 MiB/s, 136 objects/s recovering 2024-03-11T21:01:29.657 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:29 smithi146 bash[23809]: cluster 2024-03-11T21:01:28.217176+0000 osd.7 (osd.7) 209 : cluster 0 purged_snaps scrub starts 2024-03-11T21:01:29.657 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:29 smithi146 bash[23809]: cluster 2024-03-11T21:01:28.217241+0000 osd.7 (osd.7) 210 : cluster 0 purged_snaps scrub ok 2024-03-11T21:01:29.657 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:29 smithi146 bash[23809]: cluster 2024-03-11T21:01:28.400414+0000 mon.a (mon.0) 1330 : cluster 0 osdmap e220: 8 total, 8 up, 8 in 2024-03-11T21:01:29.673 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:29.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:29.677 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:01:29.777 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:29.800 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:29.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:29.802 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:01:29.903 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:29.926 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:29.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:29.928 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:01:30.029 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:01:30.054 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:01:30.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:30.056 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:01:30.157 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:01:30.181 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:01:30.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:30.183 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:01:30.284 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:01:30.312 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:01:30.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:30.315 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:01:30.416 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:30.438 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:30.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:30.440 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:01:30.541 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:30.564 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:30.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:30.567 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:01:30.668 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:01:30.678 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:30 smithi023 bash[21372]: cluster 2024-03-11T21:01:30.155421+0000 mon.a (mon.0) 1331 : cluster 3 Health check update: Degraded data redundancy: 13289/90030 objects degraded (14.761%), 7 pgs degraded (PG_DEGRADED) 2024-03-11T21:01:30.678 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:30 smithi023 bash[21372]: cluster 2024-03-11T21:01:30.169283+0000 mon.a (mon.0) 1332 : cluster 0 osdmap e221: 8 total, 8 up, 8 in 2024-03-11T21:01:30.678 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:30 smithi023 bash[29146]: cluster 2024-03-11T21:01:30.155421+0000 mon.a (mon.0) 1331 : cluster 3 Health check update: Degraded data redundancy: 13289/90030 objects degraded (14.761%), 7 pgs degraded (PG_DEGRADED) 2024-03-11T21:01:30.679 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:30 smithi023 bash[29146]: cluster 2024-03-11T21:01:30.169283+0000 mon.a (mon.0) 1332 : cluster 0 osdmap e221: 8 total, 8 up, 8 in 2024-03-11T21:01:30.702 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:01:30.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:30.705 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:01:30.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:30 smithi146 bash[23809]: cluster 2024-03-11T21:01:30.155421+0000 mon.a (mon.0) 1331 : cluster 3 Health check update: Degraded data redundancy: 13289/90030 objects degraded (14.761%), 7 pgs degraded (PG_DEGRADED) 2024-03-11T21:01:30.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:30 smithi146 bash[23809]: cluster 2024-03-11T21:01:30.169283+0000 mon.a (mon.0) 1332 : cluster 0 osdmap e221: 8 total, 8 up, 8 in 2024-03-11T21:01:30.805 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:01:30.824 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:01:30.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:30.826 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:01:30.927 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:30.950 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:30.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:30.953 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:01:31.053 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:31.077 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:31.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:31.081 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:01:31.182 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:01:31.215 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:01:31.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:31.218 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:01:31.319 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:01:31.350 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:01:31.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:31.355 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:01:31.455 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:01:31.475 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:01:31.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:31.478 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:01:31.579 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:01:31.587 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:31 smithi023 bash[21372]: cluster 2024-03-11T21:01:30.106860+0000 mgr.y (mgr.24380) 397 : cluster 0 pgmap v420: 148 pgs: 141 active+clean, 3 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped; 1.7 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 817 op/s; 13289/84813 objects degraded (15.669%); 8.6 MiB/s, 137 objects/s recovering 2024-03-11T21:01:31.587 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:31 smithi023 bash[29146]: cluster 2024-03-11T21:01:30.106860+0000 mgr.y (mgr.24380) 397 : cluster 0 pgmap v420: 148 pgs: 141 active+clean, 3 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped; 1.7 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 817 op/s; 13289/84813 objects degraded (15.669%); 8.6 MiB/s, 137 objects/s recovering 2024-03-11T21:01:31.607 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:01:31.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:31.610 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:01:31.711 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:31.734 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:31.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:31.738 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:01:31.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:31 smithi146 bash[23809]: cluster 2024-03-11T21:01:30.106860+0000 mgr.y (mgr.24380) 397 : cluster 0 pgmap v420: 148 pgs: 141 active+clean, 3 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped; 1.7 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 817 op/s; 13289/84813 objects degraded (15.669%); 8.6 MiB/s, 137 objects/s recovering 2024-03-11T21:01:31.839 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:01:31.861 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:01:31.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:31.864 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:01:31.964 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:01:31.989 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:01:31.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:31.991 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:01:32.092 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:32.114 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:32.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:32.117 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:01:32.218 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:32.245 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:32.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:32.247 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:01:32.348 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:32.376 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:32.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:32.378 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:01:32.478 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:01:32.513 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:32 smithi146 bash[23809]: cluster 2024-03-11T21:01:31.417907+0000 mon.a (mon.0) 1333 : cluster 0 osdmap e222: 8 total, 8 up, 8 in 2024-03-11T21:01:32.520 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:01:32.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:32.523 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:01:32.624 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:01:32.656 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:01:32.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:32.659 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:01:32.760 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:01:32.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:32 smithi023 bash[21372]: cluster 2024-03-11T21:01:31.417907+0000 mon.a (mon.0) 1333 : cluster 0 osdmap e222: 8 total, 8 up, 8 in 2024-03-11T21:01:32.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:32 smithi023 bash[29146]: cluster 2024-03-11T21:01:31.417907+0000 mon.a (mon.0) 1333 : cluster 0 osdmap e222: 8 total, 8 up, 8 in 2024-03-11T21:01:32.785 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:01:32.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:32.788 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:01:32.888 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:01:32.927 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:01:32.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:32.932 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:01:33.033 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:33.053 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:33.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:33.055 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:01:33.155 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:33.177 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:33.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:33.182 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:01:33.283 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:01:33.337 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:01:33.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:33.339 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:01:33.440 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:01:33.468 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:01:33.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:33.473 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:01:33.574 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:01:33.583 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:33 smithi023 bash[21372]: cluster 2024-03-11T21:01:32.107491+0000 mgr.y (mgr.24380) 398 : cluster 0 pgmap v423: 148 pgs: 1 creating+peering, 141 active+clean, 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped; 1.6 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 670 op/s; 12417/77967 objects degraded (15.926%); 9.1 MiB/s, 145 objects/s recovering 2024-03-11T21:01:33.583 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:33 smithi023 bash[21372]: cluster 2024-03-11T21:01:32.443252+0000 mon.a (mon.0) 1334 : cluster 0 osdmap e223: 8 total, 8 up, 8 in 2024-03-11T21:01:33.583 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:33 smithi023 bash[29146]: cluster 2024-03-11T21:01:32.107491+0000 mgr.y (mgr.24380) 398 : cluster 0 pgmap v423: 148 pgs: 1 creating+peering, 141 active+clean, 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped; 1.6 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 670 op/s; 12417/77967 objects degraded (15.926%); 9.1 MiB/s, 145 objects/s recovering 2024-03-11T21:01:33.583 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:33 smithi023 bash[29146]: cluster 2024-03-11T21:01:32.443252+0000 mon.a (mon.0) 1334 : cluster 0 osdmap e223: 8 total, 8 up, 8 in 2024-03-11T21:01:33.595 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:01:33.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:33.603 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:01:33.704 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:33.713 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:33 smithi146 bash[23809]: cluster 2024-03-11T21:01:32.107491+0000 mgr.y (mgr.24380) 398 : cluster 0 pgmap v423: 148 pgs: 1 creating+peering, 141 active+clean, 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped; 1.6 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 670 op/s; 12417/77967 objects degraded (15.926%); 9.1 MiB/s, 145 objects/s recovering 2024-03-11T21:01:33.713 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:33 smithi146 bash[23809]: cluster 2024-03-11T21:01:32.443252+0000 mon.a (mon.0) 1334 : cluster 0 osdmap e223: 8 total, 8 up, 8 in 2024-03-11T21:01:33.737 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:33.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:33.739 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:01:33.840 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:01:33.860 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:01:33.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:33.862 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:01:33.963 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:33.994 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:33.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:33.999 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:01:34.099 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:01:34.125 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:01:34.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:34.127 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:01:34.228 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:01:34.259 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:01:34.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:34.261 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:01:34.362 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:01:34.397 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:01:34.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:34.402 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:01:34.503 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:01:34.521 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:01:34.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:34.524 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:01:34.625 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:01:34.634 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-11T21:01:34.634 INFO:teuthology.orchestra.run.smithi023.stdout: "ops": [ 2024-03-11T21:01:34.634 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:34.635 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_op(client.14910.0:77205 8.f 8:fce77074:::benchmark_data_smithi023_89854_object27120:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e223)", 2024-03-11T21:01:34.635 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T21:01:34.607736+0000", 2024-03-11T21:01:34.635 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 0.025314940000000001, 2024-03-11T21:01:34.635 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.025351500999999999, 2024-03-11T21:01:34.635 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:01:34.635 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "waiting for sub ops", 2024-03-11T21:01:34.635 INFO:teuthology.orchestra.run.smithi023.stdout: "client_info": { 2024-03-11T21:01:34.635 INFO:teuthology.orchestra.run.smithi023.stdout: "client": "client.14910", 2024-03-11T21:01:34.635 INFO:teuthology.orchestra.run.smithi023.stdout: "client_addr": "172.21.15.23:0/2457905657", 2024-03-11T21:01:34.635 INFO:teuthology.orchestra.run.smithi023.stdout: "tid": 77205 2024-03-11T21:01:34.635 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:34.635 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:01:34.635 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:34.636 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:01:34.636 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:01:34.607736+0000", 2024-03-11T21:01:34.636 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:01:34.636 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:34.636 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:34.636 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:01:34.636 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:01:34.607736+0000", 2024-03-11T21:01:34.636 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:01:34.636 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:34.636 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:34.636 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:01:34.636 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:01:34.607738+0000", 2024-03-11T21:01:34.636 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.7630000000000001e-06 2024-03-11T21:01:34.637 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:34.637 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:34.637 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:01:34.637 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:01:34.607742+0000", 2024-03-11T21:01:34.637 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 4.0470000000000004e-06 2024-03-11T21:01:34.637 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:34.637 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:34.637 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:01:34.637 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:01:34.607752+0000", 2024-03-11T21:01:34.637 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.0233e-05 2024-03-11T21:01:34.637 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:34.637 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:34.637 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:01:34.638 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:01:34.607759+0000", 2024-03-11T21:01:34.638 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.3690000000000001e-06 2024-03-11T21:01:34.638 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:34.638 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:34.638 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:01:34.638 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:01:34.608203+0000", 2024-03-11T21:01:34.638 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00044307100000000001 2024-03-11T21:01:34.638 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:34.638 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:34.638 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:01:34.638 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:01:34.608363+0000", 2024-03-11T21:01:34.638 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00016080399999999999 2024-03-11T21:01:34.638 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:34.638 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:34.639 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for subops from 1,2", 2024-03-11T21:01:34.639 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:01:34.608532+0000", 2024-03-11T21:01:34.639 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00016836800000000001 2024-03-11T21:01:34.639 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:34.639 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:34.639 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "op_commit", 2024-03-11T21:01:34.639 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:01:34.611733+0000", 2024-03-11T21:01:34.639 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0032008259999999999 2024-03-11T21:01:34.639 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:34.639 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:34.639 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:01:34.639 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:01:34.627418+0000", 2024-03-11T21:01:34.639 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.015685049999999999 2024-03-11T21:01:34.639 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:01:34.639 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:01:34.640 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:01:34.640 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:34.640 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:34.640 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_op(client.14910.0:77212 8.f 8:f3a321e4:::benchmark_data_smithi023_89854_object27127:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e223)", 2024-03-11T21:01:34.640 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T21:01:34.617728+0000", 2024-03-11T21:01:34.640 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 0.015323438999999999, 2024-03-11T21:01:34.640 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.015461598999999999, 2024-03-11T21:01:34.640 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:01:34.640 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "waiting for sub ops", 2024-03-11T21:01:34.640 INFO:teuthology.orchestra.run.smithi023.stdout: "client_info": { 2024-03-11T21:01:34.640 INFO:teuthology.orchestra.run.smithi023.stdout: "client": "client.14910", 2024-03-11T21:01:34.640 INFO:teuthology.orchestra.run.smithi023.stdout: "client_addr": "172.21.15.23:0/2457905657", 2024-03-11T21:01:34.640 INFO:teuthology.orchestra.run.smithi023.stdout: "tid": 77212 2024-03-11T21:01:34.641 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:34.641 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:01:34.641 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:34.641 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:01:34.641 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:01:34.617728+0000", 2024-03-11T21:01:34.641 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:01:34.641 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:34.641 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:34.641 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:01:34.641 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:01:34.617728+0000", 2024-03-11T21:01:34.641 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:01:34.641 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:34.641 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:34.641 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:01:34.641 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:01:34.617730+0000", 2024-03-11T21:01:34.641 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.508e-06 2024-03-11T21:01:34.642 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:34.642 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:34.642 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:01:34.642 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:01:34.617741+0000", 2024-03-11T21:01:34.642 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.0617e-05 2024-03-11T21:01:34.642 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:34.642 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:34.642 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:01:34.642 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:01:34.617760+0000", 2024-03-11T21:01:34.642 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.9236999999999999e-05 2024-03-11T21:01:34.642 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:34.642 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:34.642 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:01:34.642 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:01:34.617771+0000", 2024-03-11T21:01:34.643 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.0995999999999999e-05 2024-03-11T21:01:34.643 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:34.643 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:34.643 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:01:34.643 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:01:34.617841+0000", 2024-03-11T21:01:34.643 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.0038999999999998e-05 2024-03-11T21:01:34.643 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:34.643 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:34.643 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:01:34.643 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:01:34.618025+0000", 2024-03-11T21:01:34.643 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00018375600000000001 2024-03-11T21:01:34.643 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:34.643 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:34.643 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for subops from 1,2", 2024-03-11T21:01:34.643 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:01:34.619647+0000", 2024-03-11T21:01:34.643 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0016220239999999999 2024-03-11T21:01:34.644 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:01:34.644 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:01:34.644 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:01:34.644 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:34.644 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:34.644 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_op(client.14910.0:77215 8.b 8:d8616767:::benchmark_data_smithi023_89854_object27130:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e223)", 2024-03-11T21:01:34.644 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T21:01:34.618691+0000", 2024-03-11T21:01:34.644 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 0.014360176000000001, 2024-03-11T21:01:34.644 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.014578343000000001, 2024-03-11T21:01:34.644 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:01:34.644 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "waiting for sub ops", 2024-03-11T21:01:34.644 INFO:teuthology.orchestra.run.smithi023.stdout: "client_info": { 2024-03-11T21:01:34.644 INFO:teuthology.orchestra.run.smithi023.stdout: "client": "client.14910", 2024-03-11T21:01:34.644 INFO:teuthology.orchestra.run.smithi023.stdout: "client_addr": "172.21.15.23:0/2457905657", 2024-03-11T21:01:34.645 INFO:teuthology.orchestra.run.smithi023.stdout: "tid": 77215 2024-03-11T21:01:34.645 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:34.645 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:01:34.645 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:34.645 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:01:34.645 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:01:34.618691+0000", 2024-03-11T21:01:34.645 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:01:34.645 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:34.645 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:34.645 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:01:34.645 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:01:34.618691+0000", 2024-03-11T21:01:34.645 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:01:34.645 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:34.645 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:34.645 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:01:34.645 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:01:34.618692+0000", 2024-03-11T21:01:34.646 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 9.3600000000000002e-07 2024-03-11T21:01:34.646 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:34.646 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:34.646 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:01:34.646 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:01:34.618697+0000", 2024-03-11T21:01:34.646 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 5.079e-06 2024-03-11T21:01:34.646 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:34.646 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:34.646 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:01:34.646 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:01:34.618797+0000", 2024-03-11T21:01:34.646 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000100474 2024-03-11T21:01:34.646 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:34.646 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:34.646 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:01:34.647 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:01:34.618805+0000", 2024-03-11T21:01:34.647 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 8.0490000000000004e-06 2024-03-11T21:01:34.647 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:34.647 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:34.647 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:01:34.647 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:01:34.619248+0000", 2024-03-11T21:01:34.647 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00044314499999999999 2024-03-11T21:01:34.647 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:34.647 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:34.647 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:01:34.647 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:01:34.619388+0000", 2024-03-11T21:01:34.647 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00013987200000000001 2024-03-11T21:01:34.647 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:34.647 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:34.647 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for subops from 1,4", 2024-03-11T21:01:34.648 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:01:34.619536+0000", 2024-03-11T21:01:34.648 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00014786000000000001 2024-03-11T21:01:34.648 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:34.648 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:34.648 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:01:34.648 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:01:34.625249+0000", 2024-03-11T21:01:34.648 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0057125969999999998 2024-03-11T21:01:34.648 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:34.648 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:34.648 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "op_commit", 2024-03-11T21:01:34.648 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:01:34.626359+0000", 2024-03-11T21:01:34.648 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0011102740000000001 2024-03-11T21:01:34.648 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:01:34.648 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:01:34.648 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:01:34.649 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:34.649 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:34.649 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_op(client.14910.0:77218 8.7 8:ed2306a0:::benchmark_data_smithi023_89854_object27133:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e223)", 2024-03-11T21:01:34.649 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T21:01:34.619193+0000", 2024-03-11T21:01:34.649 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 0.013857477, 2024-03-11T21:01:34.649 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.014167083, 2024-03-11T21:01:34.649 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:01:34.649 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "waiting for sub ops", 2024-03-11T21:01:34.649 INFO:teuthology.orchestra.run.smithi023.stdout: "client_info": { 2024-03-11T21:01:34.649 INFO:teuthology.orchestra.run.smithi023.stdout: "client": "client.14910", 2024-03-11T21:01:34.649 INFO:teuthology.orchestra.run.smithi023.stdout: "client_addr": "172.21.15.23:0/2457905657", 2024-03-11T21:01:34.649 INFO:teuthology.orchestra.run.smithi023.stdout: "tid": 77218 2024-03-11T21:01:34.649 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:34.649 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:01:34.649 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:34.650 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:01:34.650 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:01:34.619193+0000", 2024-03-11T21:01:34.650 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:01:34.650 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:34.650 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:34.650 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:01:34.650 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:01:34.619193+0000", 2024-03-11T21:01:34.650 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:01:34.650 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:34.650 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:34.650 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:01:34.650 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:01:34.619194+0000", 2024-03-11T21:01:34.650 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 8.1900000000000001e-07 2024-03-11T21:01:34.650 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:34.651 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:34.651 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:01:34.651 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:01:34.619199+0000", 2024-03-11T21:01:34.651 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 4.9980000000000003e-06 2024-03-11T21:01:34.651 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:34.651 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:34.651 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:01:34.651 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:01:34.619214+0000", 2024-03-11T21:01:34.651 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.4402e-05 2024-03-11T21:01:34.651 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:34.651 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:34.651 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:01:34.651 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:01:34.619225+0000", 2024-03-11T21:01:34.651 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.1483e-05 2024-03-11T21:01:34.651 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:34.651 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:34.652 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:01:34.652 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:01:34.627378+0000", 2024-03-11T21:01:34.652 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0081524749999999993 2024-03-11T21:01:34.652 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:34.652 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:34.652 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:01:34.652 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:01:34.627506+0000", 2024-03-11T21:01:34.652 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000128386 2024-03-11T21:01:34.652 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:34.652 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:34.652 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for subops from 1,6", 2024-03-11T21:01:34.652 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:01:34.627646+0000", 2024-03-11T21:01:34.652 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00014014600000000001 2024-03-11T21:01:34.652 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:01:34.652 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:01:34.653 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:01:34.653 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:34.653 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:34.653 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_repop(client.14910.0:77208 8.8 e223/189 8:13017905:::benchmark_data_smithi023_89854_object27123:head v 223'4750, mlcod=223'4748)", 2024-03-11T21:01:34.653 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T21:01:34.619487+0000", 2024-03-11T21:01:34.653 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 0.013563649000000001, 2024-03-11T21:01:34.653 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.013954819, 2024-03-11T21:01:34.653 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:01:34.653 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "started", 2024-03-11T21:01:34.653 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:01:34.653 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:34.653 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:01:34.653 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:01:34.619487+0000", 2024-03-11T21:01:34.653 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:01:34.653 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:34.654 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:34.654 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:01:34.654 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:01:34.619487+0000", 2024-03-11T21:01:34.654 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:01:34.654 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:34.654 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:34.654 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:01:34.654 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:01:34.619488+0000", 2024-03-11T21:01:34.654 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.7700000000000004e-07 2024-03-11T21:01:34.654 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:34.654 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:34.654 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:01:34.654 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:01:34.619495+0000", 2024-03-11T21:01:34.654 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.2540000000000003e-06 2024-03-11T21:01:34.655 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:34.655 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:34.655 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:01:34.655 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:01:34.619505+0000", 2024-03-11T21:01:34.655 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 9.5699999999999999e-06 2024-03-11T21:01:34.655 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:34.655 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:34.655 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:01:34.655 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:01:34.619511+0000", 2024-03-11T21:01:34.655 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.3110000000000002e-06 2024-03-11T21:01:34.655 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:34.655 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:34.655 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:01:34.655 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:01:34.619698+0000", 2024-03-11T21:01:34.656 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000186474 2024-03-11T21:01:34.656 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:34.656 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:34.656 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:01:34.656 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:01:34.622964+0000", 2024-03-11T21:01:34.656 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0032669029999999998 2024-03-11T21:01:34.656 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:01:34.656 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:01:34.656 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:01:34.656 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:34.656 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:34.656 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_repop(client.14910.0:77213 8.8 e223/189 8:1439e5e1:::benchmark_data_smithi023_89854_object27128:head v 223'4751, mlcod=223'4748)", 2024-03-11T21:01:34.656 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T21:01:34.619528+0000", 2024-03-11T21:01:34.656 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 0.013523019000000001, 2024-03-11T21:01:34.657 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.013984327, 2024-03-11T21:01:34.657 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:01:34.657 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "started", 2024-03-11T21:01:34.657 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:01:34.657 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:34.657 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:01:34.657 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:01:34.619528+0000", 2024-03-11T21:01:34.657 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:01:34.657 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:34.657 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:34.657 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:01:34.657 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:01:34.619528+0000", 2024-03-11T21:01:34.657 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:01:34.657 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:34.657 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:34.657 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:01:34.658 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:01:34.619528+0000", 2024-03-11T21:01:34.658 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.3900000000000001e-07 2024-03-11T21:01:34.658 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:34.658 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:34.658 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:01:34.658 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:01:34.619532+0000", 2024-03-11T21:01:34.658 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 4.1529999999999999e-06 2024-03-11T21:01:34.658 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:34.658 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:34.658 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:01:34.658 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:01:34.619538+0000", 2024-03-11T21:01:34.658 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 5.8459999999999996e-06 2024-03-11T21:01:34.658 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:34.658 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:34.659 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:01:34.659 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:01:34.619543+0000", 2024-03-11T21:01:34.659 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 4.4159999999999997e-06 2024-03-11T21:01:34.659 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:34.659 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:34.659 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:01:34.659 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:01:34.625326+0000", 2024-03-11T21:01:34.659 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0057831139999999998 2024-03-11T21:01:34.659 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:34.659 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:34.659 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:01:34.659 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:01:34.625346+0000", 2024-03-11T21:01:34.659 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.0727999999999999e-05 2024-03-11T21:01:34.659 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:01:34.660 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:01:34.660 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:01:34.660 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:34.660 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:34.660 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_repop(client.14910.0:77211 8.c e223/189 8:3f23b783:::benchmark_data_smithi023_89854_object27126:head v 223'4773, mlcod=223'4771)", 2024-03-11T21:01:34.660 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T21:01:34.619599+0000", 2024-03-11T21:01:34.660 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 0.013452171000000001, 2024-03-11T21:01:34.660 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.013984144, 2024-03-11T21:01:34.660 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:01:34.660 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "started", 2024-03-11T21:01:34.660 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:01:34.660 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:34.660 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:01:34.660 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:01:34.619599+0000", 2024-03-11T21:01:34.660 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:01:34.661 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:34.661 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:34.661 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:01:34.661 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:01:34.619599+0000", 2024-03-11T21:01:34.661 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:01:34.661 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:34.661 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:34.661 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:01:34.661 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:01:34.619599+0000", 2024-03-11T21:01:34.661 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.8000000000000002e-07 2024-03-11T21:01:34.661 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:34.661 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:34.661 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:01:34.661 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:01:34.619603+0000", 2024-03-11T21:01:34.662 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 4.0339999999999998e-06 2024-03-11T21:01:34.662 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:34.662 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:34.662 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:01:34.662 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:01:34.619609+0000", 2024-03-11T21:01:34.662 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.3169999999999998e-06 2024-03-11T21:01:34.662 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:34.662 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:34.662 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:01:34.662 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:01:34.619614+0000", 2024-03-11T21:01:34.662 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 4.5129999999999998e-06 2024-03-11T21:01:34.662 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:34.662 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:34.662 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:01:34.663 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:01:34.619732+0000", 2024-03-11T21:01:34.663 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00011777000000000001 2024-03-11T21:01:34.663 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:34.663 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:34.663 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:01:34.663 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:01:34.619752+0000", 2024-03-11T21:01:34.663 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.9924000000000001e-05 2024-03-11T21:01:34.663 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:01:34.663 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:01:34.663 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:01:34.663 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:34.663 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:34.663 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_repop(client.14910.0:77217 8.c e223/189)", 2024-03-11T21:01:34.663 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T21:01:34.622843+0000", 2024-03-11T21:01:34.663 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 0.010207816999999999, 2024-03-11T21:01:34.664 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.010806811, 2024-03-11T21:01:34.664 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:01:34.664 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "queued for pg", 2024-03-11T21:01:34.664 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:01:34.664 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:34.664 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:01:34.664 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:01:34.622843+0000", 2024-03-11T21:01:34.664 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:01:34.664 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:34.664 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:34.664 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:01:34.664 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:01:34.622843+0000", 2024-03-11T21:01:34.664 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:01:34.664 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:34.664 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:34.665 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:01:34.665 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:01:34.622844+0000", 2024-03-11T21:01:34.665 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.3099999999999997e-07 2024-03-11T21:01:34.665 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:34.665 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:34.665 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:01:34.665 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:01:34.622851+0000", 2024-03-11T21:01:34.665 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.1169999999999998e-06 2024-03-11T21:01:34.665 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:34.665 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:34.665 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:01:34.665 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:01:34.622861+0000", 2024-03-11T21:01:34.665 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 9.9089999999999995e-06 2024-03-11T21:01:34.665 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:34.665 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:34.666 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:01:34.666 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:01:34.622867+0000", 2024-03-11T21:01:34.666 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.5810000000000002e-06 2024-03-11T21:01:34.666 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:01:34.666 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:01:34.666 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:01:34.666 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:34.666 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:34.666 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_repop_reply(client.14910.0:77218 8.7 e223/212)", 2024-03-11T21:01:34.666 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T21:01:34.631597+0000", 2024-03-11T21:01:34.666 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 0.001453569, 2024-03-11T21:01:34.666 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0021051199999999998, 2024-03-11T21:01:34.666 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:01:34.666 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "queued for pg", 2024-03-11T21:01:34.666 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:01:34.667 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:34.667 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:01:34.667 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:01:34.631597+0000", 2024-03-11T21:01:34.667 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:01:34.667 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:34.667 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:34.667 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:01:34.667 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:01:34.631597+0000", 2024-03-11T21:01:34.667 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:01:34.667 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:34.667 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:34.667 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:01:34.667 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:01:34.631599+0000", 2024-03-11T21:01:34.667 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.198e-06 2024-03-11T21:01:34.667 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:34.668 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:34.668 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:01:34.668 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:01:34.631607+0000", 2024-03-11T21:01:34.668 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 8.5080000000000005e-06 2024-03-11T21:01:34.668 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:34.668 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:34.668 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:01:34.668 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:01:34.631629+0000", 2024-03-11T21:01:34.668 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.1484e-05 2024-03-11T21:01:34.668 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:01:34.668 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:01:34.668 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:01:34.668 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:01:34.631639+0000", 2024-03-11T21:01:34.668 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.0104e-05 2024-03-11T21:01:34.668 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:01:34.669 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:01:34.669 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:01:34.669 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:01:34.669 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-11T21:01:34.669 INFO:teuthology.orchestra.run.smithi023.stdout: "num_ops": 9 2024-03-11T21:01:34.669 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-11T21:01:34.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:34.670 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:01:34.670 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:01:34.771 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:01:34.792 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:01:34.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:34.794 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:01:34.895 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:34.910 INFO:teuthology.orchestra.run.smithi023.stdout:osd.0: {} 2024-03-11T21:01:34.913 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:34.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:34.916 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:01:34.944 INFO:teuthology.orchestra.run.smithi023.stdout:osd.1: {} 2024-03-11T21:01:34.985 INFO:teuthology.orchestra.run.smithi023.stdout:osd.2: {} 2024-03-11T21:01:35.017 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:35.047 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:35.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:35.049 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:01:35.074 INFO:teuthology.orchestra.run.smithi023.stdout:osd.3: {} 2024-03-11T21:01:35.126 INFO:teuthology.orchestra.run.smithi023.stdout:osd.4: {} 2024-03-11T21:01:35.150 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:01:35.184 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:01:35.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:35.188 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:01:35.191 INFO:teuthology.orchestra.run.smithi023.stdout:osd.5: {} 2024-03-11T21:01:35.237 INFO:teuthology.orchestra.run.smithi023.stdout:osd.6: {} 2024-03-11T21:01:35.278 INFO:teuthology.orchestra.run.smithi023.stdout:osd.7: {} 2024-03-11T21:01:35.285 INFO:teuthology.orchestra.run.smithi023.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:01:35.286 INFO:teuthology.orchestra.run.smithi023.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:01:35.286 INFO:teuthology.orchestra.run.smithi023.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:01:35.286 INFO:teuthology.orchestra.run.smithi023.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:01:35.286 INFO:teuthology.orchestra.run.smithi023.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:01:35.286 INFO:teuthology.orchestra.run.smithi023.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:01:35.286 INFO:teuthology.orchestra.run.smithi023.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:01:35.286 INFO:teuthology.orchestra.run.smithi023.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:01:35.288 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:01:35.307 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:01:35.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:35.310 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:01:35.411 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:35.428 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:35.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:35.430 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:01:35.450 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:01:35 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:01:35.531 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:35.554 INFO:teuthology.orchestra.run.smithi023.stderr:reweighted osd.0 to 1 (10000) 2024-03-11T21:01:35.559 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:35 smithi023 bash[29146]: cluster 2024-03-11T21:01:34.108133+0000 mgr.y (mgr.24380) 399 : cluster 0 pgmap v425: 148 pgs: 1 creating+peering, 141 active+clean, 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped; 1.6 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 896 B/s rd, 0 B/s wr, 705 op/s; 12417/77967 objects degraded (15.926%); 9.5 MiB/s, 152 objects/s recovering 2024-03-11T21:01:35.559 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:35 smithi023 bash[29146]: audit 2024-03-11T21:01:34.885606+0000 mon.c (mon.2) 51 : audit 0 from='client.? 172.21.15.23:0/1746266173' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:01:35.559 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:35 smithi023 bash[29146]: audit 2024-03-11T21:01:34.963602+0000 mon.a (mon.0) 1335 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:01:35.559 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:35 smithi023 bash[29146]: audit 2024-03-11T21:01:35.112472+0000 mon.a (mon.0) 1336 : audit 1 from='client.? 172.21.15.23:0/775001445' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-11T21:01:35.559 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:35 smithi023 bash[29146]: cluster 2024-03-11T21:01:35.159486+0000 mon.a (mon.0) 1337 : cluster 3 Health check update: Degraded data redundancy: 12417/77967 objects degraded (15.926%), 6 pgs degraded (PG_DEGRADED) 2024-03-11T21:01:35.559 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:35 smithi023 bash[29146]: cluster 2024-03-11T21:01:35.231505+0000 osd.0 (osd.0) 173 : cluster 0 purged_snaps scrub starts 2024-03-11T21:01:35.559 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:35 smithi023 bash[29146]: cluster 2024-03-11T21:01:35.231576+0000 osd.0 (osd.0) 174 : cluster 0 purged_snaps scrub ok 2024-03-11T21:01:35.562 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:35.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:35.564 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:01:35.665 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:01:35.691 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:01:35.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:35.694 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:01:35.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:35 smithi146 bash[23809]: cluster 2024-03-11T21:01:34.108133+0000 mgr.y (mgr.24380) 399 : cluster 0 pgmap v425: 148 pgs: 1 creating+peering, 141 active+clean, 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped; 1.6 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 896 B/s rd, 0 B/s wr, 705 op/s; 12417/77967 objects degraded (15.926%); 9.5 MiB/s, 152 objects/s recovering 2024-03-11T21:01:35.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:35 smithi146 bash[23809]: audit 2024-03-11T21:01:34.885606+0000 mon.c (mon.2) 51 : audit 0 from='client.? 172.21.15.23:0/1746266173' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:01:35.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:35 smithi146 bash[23809]: audit 2024-03-11T21:01:34.963602+0000 mon.a (mon.0) 1335 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:01:35.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:35 smithi146 bash[23809]: audit 2024-03-11T21:01:35.112472+0000 mon.a (mon.0) 1336 : audit 1 from='client.? 172.21.15.23:0/775001445' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-11T21:01:35.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:35 smithi146 bash[23809]: cluster 2024-03-11T21:01:35.159486+0000 mon.a (mon.0) 1337 : cluster 3 Health check update: Degraded data redundancy: 12417/77967 objects degraded (15.926%), 6 pgs degraded (PG_DEGRADED) 2024-03-11T21:01:35.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:35 smithi146 bash[23809]: cluster 2024-03-11T21:01:35.231505+0000 osd.0 (osd.0) 173 : cluster 0 purged_snaps scrub starts 2024-03-11T21:01:35.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:35 smithi146 bash[23809]: cluster 2024-03-11T21:01:35.231576+0000 osd.0 (osd.0) 174 : cluster 0 purged_snaps scrub ok 2024-03-11T21:01:35.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:35 smithi023 bash[21372]: cluster 2024-03-11T21:01:34.108133+0000 mgr.y (mgr.24380) 399 : cluster 0 pgmap v425: 148 pgs: 1 creating+peering, 141 active+clean, 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped; 1.6 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 896 B/s rd, 0 B/s wr, 705 op/s; 12417/77967 objects degraded (15.926%); 9.5 MiB/s, 152 objects/s recovering 2024-03-11T21:01:35.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:35 smithi023 bash[21372]: audit 2024-03-11T21:01:34.885606+0000 mon.c (mon.2) 51 : audit 0 from='client.? 172.21.15.23:0/1746266173' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:01:35.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:35 smithi023 bash[21372]: audit 2024-03-11T21:01:34.963602+0000 mon.a (mon.0) 1335 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:01:35.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:35 smithi023 bash[21372]: audit 2024-03-11T21:01:35.112472+0000 mon.a (mon.0) 1336 : audit 1 from='client.? 172.21.15.23:0/775001445' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-11T21:01:35.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:35 smithi023 bash[21372]: cluster 2024-03-11T21:01:35.159486+0000 mon.a (mon.0) 1337 : cluster 3 Health check update: Degraded data redundancy: 12417/77967 objects degraded (15.926%), 6 pgs degraded (PG_DEGRADED) 2024-03-11T21:01:35.786 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:35 smithi023 bash[21372]: cluster 2024-03-11T21:01:35.231505+0000 osd.0 (osd.0) 173 : cluster 0 purged_snaps scrub starts 2024-03-11T21:01:35.786 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:35 smithi023 bash[21372]: cluster 2024-03-11T21:01:35.231576+0000 osd.0 (osd.0) 174 : cluster 0 purged_snaps scrub ok 2024-03-11T21:01:35.795 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:35.820 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:35.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:35.822 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:01:35.923 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:01:35.953 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:01:35.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:35.958 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:01:36.059 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:01:36.090 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:01:36.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:36.093 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:01:36.193 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:36.217 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:36.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:36.221 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:01:36.322 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:36.354 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:36.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:36.357 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:01:36.458 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:01:36.503 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:01:36.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:36.508 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:01:36.574 INFO:teuthology.orchestra.run.smithi023.stderr:noscrub is unset 2024-03-11T21:01:36.609 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:01:36.619 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:36 smithi023 bash[21372]: audit 2024-03-11T21:01:35.343134+0000 mgr.y (mgr.24380) 400 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:01:36.619 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:36 smithi023 bash[21372]: audit 2024-03-11T21:01:35.536163+0000 mon.c (mon.2) 52 : audit 1 from='client.? 172.21.15.23:0/217273811' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:01:36.619 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:36 smithi023 bash[21372]: audit 2024-03-11T21:01:35.541519+0000 mon.a (mon.0) 1338 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:01:36.619 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:36 smithi023 bash[21372]: audit 2024-03-11T21:01:35.543491+0000 mon.a (mon.0) 1339 : audit 1 from='client.? 172.21.15.23:0/775001445' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-11T21:01:36.619 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:36 smithi023 bash[21372]: cluster 2024-03-11T21:01:35.549993+0000 mon.a (mon.0) 1340 : cluster 0 osdmap e224: 8 total, 8 up, 8 in 2024-03-11T21:01:36.619 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:36 smithi023 bash[29146]: audit 2024-03-11T21:01:35.343134+0000 mgr.y (mgr.24380) 400 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:01:36.619 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:36 smithi023 bash[29146]: audit 2024-03-11T21:01:35.536163+0000 mon.c (mon.2) 52 : audit 1 from='client.? 172.21.15.23:0/217273811' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:01:36.619 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:36 smithi023 bash[29146]: audit 2024-03-11T21:01:35.541519+0000 mon.a (mon.0) 1338 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:01:36.620 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:36 smithi023 bash[29146]: audit 2024-03-11T21:01:35.543491+0000 mon.a (mon.0) 1339 : audit 1 from='client.? 172.21.15.23:0/775001445' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-11T21:01:36.620 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:36 smithi023 bash[29146]: cluster 2024-03-11T21:01:35.549993+0000 mon.a (mon.0) 1340 : cluster 0 osdmap e224: 8 total, 8 up, 8 in 2024-03-11T21:01:36.648 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:01:36.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:36.653 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:01:36.754 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:36.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:36 smithi146 bash[23809]: audit 2024-03-11T21:01:35.343134+0000 mgr.y (mgr.24380) 400 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:01:36.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:36 smithi146 bash[23809]: audit 2024-03-11T21:01:35.536163+0000 mon.c (mon.2) 52 : audit 1 from='client.? 172.21.15.23:0/217273811' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:01:36.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:36 smithi146 bash[23809]: audit 2024-03-11T21:01:35.541519+0000 mon.a (mon.0) 1338 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:01:36.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:36 smithi146 bash[23809]: audit 2024-03-11T21:01:35.543491+0000 mon.a (mon.0) 1339 : audit 1 from='client.? 172.21.15.23:0/775001445' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-11T21:01:36.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:36 smithi146 bash[23809]: cluster 2024-03-11T21:01:35.549993+0000 mon.a (mon.0) 1340 : cluster 0 osdmap e224: 8 total, 8 up, 8 in 2024-03-11T21:01:36.771 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:36.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:36.773 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:01:36.874 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:01:36.898 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:01:36.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:36.900 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:01:37.001 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:01:37.027 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:01:37.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:37.031 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:01:37.131 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:01:37.163 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:01:37.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:37.166 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:01:37.266 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:37.285 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:37.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:37.287 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:01:37.387 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:01:37.412 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:01:37.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:37.415 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:01:37.516 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:01:37.536 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:01:37.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:37.540 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:01:37.641 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:01:37.664 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:01:37.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:37.668 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:01:37.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:37 smithi146 bash[23809]: cluster 2024-03-11T21:01:36.108898+0000 mgr.y (mgr.24380) 401 : cluster 0 pgmap v427: 148 pgs: 1 creating+peering, 141 active+clean, 3 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped; 1.4 GiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 922 op/s; 11559/68397 objects degraded (16.900%); 18 MiB/s, 291 objects/s recovering 2024-03-11T21:01:37.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:37 smithi146 bash[23809]: cluster 2024-03-11T21:01:36.544369+0000 mon.a (mon.0) 1341 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:01:37.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:37 smithi146 bash[23809]: audit 2024-03-11T21:01:36.557434+0000 mon.a (mon.0) 1342 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:01:37.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:37 smithi146 bash[23809]: cluster 2024-03-11T21:01:36.562789+0000 mon.a (mon.0) 1343 : cluster 0 osdmap e225: 8 total, 8 up, 8 in 2024-03-11T21:01:37.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:37 smithi146 bash[23809]: cluster 2024-03-11T21:01:37.018941+0000 osd.4 (osd.4) 3 : cluster 0 purged_snaps scrub starts 2024-03-11T21:01:37.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:37 smithi146 bash[23809]: cluster 2024-03-11T21:01:37.019041+0000 osd.4 (osd.4) 4 : cluster 0 purged_snaps scrub ok 2024-03-11T21:01:37.769 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:37.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:37 smithi023 bash[21372]: cluster 2024-03-11T21:01:36.108898+0000 mgr.y (mgr.24380) 401 : cluster 0 pgmap v427: 148 pgs: 1 creating+peering, 141 active+clean, 3 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped; 1.4 GiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 922 op/s; 11559/68397 objects degraded (16.900%); 18 MiB/s, 291 objects/s recovering 2024-03-11T21:01:37.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:37 smithi023 bash[21372]: cluster 2024-03-11T21:01:36.544369+0000 mon.a (mon.0) 1341 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:01:37.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:37 smithi023 bash[21372]: audit 2024-03-11T21:01:36.557434+0000 mon.a (mon.0) 1342 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:01:37.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:37 smithi023 bash[21372]: cluster 2024-03-11T21:01:36.562789+0000 mon.a (mon.0) 1343 : cluster 0 osdmap e225: 8 total, 8 up, 8 in 2024-03-11T21:01:37.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:37 smithi023 bash[21372]: cluster 2024-03-11T21:01:37.018941+0000 osd.4 (osd.4) 3 : cluster 0 purged_snaps scrub starts 2024-03-11T21:01:37.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:37 smithi023 bash[21372]: cluster 2024-03-11T21:01:37.019041+0000 osd.4 (osd.4) 4 : cluster 0 purged_snaps scrub ok 2024-03-11T21:01:37.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:37 smithi023 bash[29146]: cluster 2024-03-11T21:01:36.108898+0000 mgr.y (mgr.24380) 401 : cluster 0 pgmap v427: 148 pgs: 1 creating+peering, 141 active+clean, 3 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped; 1.4 GiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 922 op/s; 11559/68397 objects degraded (16.900%); 18 MiB/s, 291 objects/s recovering 2024-03-11T21:01:37.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:37 smithi023 bash[29146]: cluster 2024-03-11T21:01:36.544369+0000 mon.a (mon.0) 1341 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:01:37.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:37 smithi023 bash[29146]: audit 2024-03-11T21:01:36.557434+0000 mon.a (mon.0) 1342 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:01:37.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:37 smithi023 bash[29146]: cluster 2024-03-11T21:01:36.562789+0000 mon.a (mon.0) 1343 : cluster 0 osdmap e225: 8 total, 8 up, 8 in 2024-03-11T21:01:37.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:37 smithi023 bash[29146]: cluster 2024-03-11T21:01:37.018941+0000 osd.4 (osd.4) 3 : cluster 0 purged_snaps scrub starts 2024-03-11T21:01:37.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:37 smithi023 bash[29146]: cluster 2024-03-11T21:01:37.019041+0000 osd.4 (osd.4) 4 : cluster 0 purged_snaps scrub ok 2024-03-11T21:01:37.790 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:37.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:37.794 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:01:37.894 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:37.912 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:37.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:37.914 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:01:38.014 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:38.043 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:38.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:38.048 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:01:38.149 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:38.175 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:38.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:38.179 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:01:38.280 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:38.311 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:38.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:38.316 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:01:38.416 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:01:38.438 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:01:38.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:38.440 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:01:38.541 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:38.562 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:38.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:38.564 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:01:38.617 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:38 smithi023 bash[21372]: cluster 2024-03-11T21:01:37.565490+0000 mon.a (mon.0) 1344 : cluster 0 osdmap e226: 8 total, 8 up, 8 in 2024-03-11T21:01:38.617 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:38 smithi023 bash[21372]: cluster 2024-03-11T21:01:38.109409+0000 mgr.y (mgr.24380) 402 : cluster 0 pgmap v430: 148 pgs: 142 active+clean, 3 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped; 1.4 GiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 722 B/s rd, 0 B/s wr, 606 op/s; 11559/67671 objects degraded (17.081%); 9.5 MiB/s, 151 objects/s recovering 2024-03-11T21:01:38.617 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:01:38 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:01:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:01:38.618 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:38 smithi023 bash[29146]: cluster 2024-03-11T21:01:37.565490+0000 mon.a (mon.0) 1344 : cluster 0 osdmap e226: 8 total, 8 up, 8 in 2024-03-11T21:01:38.618 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:38 smithi023 bash[29146]: cluster 2024-03-11T21:01:38.109409+0000 mgr.y (mgr.24380) 402 : cluster 0 pgmap v430: 148 pgs: 142 active+clean, 3 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped; 1.4 GiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 722 B/s rd, 0 B/s wr, 606 op/s; 11559/67671 objects degraded (17.081%); 9.5 MiB/s, 151 objects/s recovering 2024-03-11T21:01:38.619 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-03-11T21:01:38.665 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:38.676 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:38 smithi146 bash[23809]: cluster 2024-03-11T21:01:37.565490+0000 mon.a (mon.0) 1344 : cluster 0 osdmap e226: 8 total, 8 up, 8 in 2024-03-11T21:01:38.676 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:38 smithi146 bash[23809]: cluster 2024-03-11T21:01:38.109409+0000 mgr.y (mgr.24380) 402 : cluster 0 pgmap v430: 148 pgs: 142 active+clean, 3 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped; 1.4 GiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 722 B/s rd, 0 B/s wr, 606 op/s; 11559/67671 objects degraded (17.081%); 9.5 MiB/s, 151 objects/s recovering 2024-03-11T21:01:38.691 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:38.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:38.695 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:01:38.795 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:38.820 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:38.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:38.822 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:01:38.922 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:38.954 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:38.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:38.957 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:01:39.057 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:39.084 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:39.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:39.088 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:01:39.189 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:01:39.220 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:01:39.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:39.222 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:01:39.322 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:39.354 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:39.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:39.359 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:01:39.460 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:39.480 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:39.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:39.482 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:01:39.583 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:39.605 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:39.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:39.607 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:01:39.708 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:01:39.718 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:39 smithi023 bash[21372]: cluster 2024-03-11T21:01:38.370863+0000 osd.6 (osd.6) 148 : cluster 0 4.1f scrub starts 2024-03-11T21:01:39.718 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:39 smithi023 bash[21372]: cluster 2024-03-11T21:01:38.377494+0000 osd.6 (osd.6) 149 : cluster 0 4.1f scrub ok 2024-03-11T21:01:39.719 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:39 smithi023 bash[21372]: cluster 2024-03-11T21:01:38.493584+0000 osd.7 (osd.7) 211 : cluster 0 2.8 scrub starts 2024-03-11T21:01:39.719 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:39 smithi023 bash[21372]: cluster 2024-03-11T21:01:38.494524+0000 osd.7 (osd.7) 212 : cluster 0 2.8 scrub ok 2024-03-11T21:01:39.719 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:39 smithi023 bash[29146]: cluster 2024-03-11T21:01:38.370863+0000 osd.6 (osd.6) 148 : cluster 0 4.1f scrub starts 2024-03-11T21:01:39.719 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:39 smithi023 bash[29146]: cluster 2024-03-11T21:01:38.377494+0000 osd.6 (osd.6) 149 : cluster 0 4.1f scrub ok 2024-03-11T21:01:39.719 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:39 smithi023 bash[29146]: cluster 2024-03-11T21:01:38.493584+0000 osd.7 (osd.7) 211 : cluster 0 2.8 scrub starts 2024-03-11T21:01:39.719 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:39 smithi023 bash[29146]: cluster 2024-03-11T21:01:38.494524+0000 osd.7 (osd.7) 212 : cluster 0 2.8 scrub ok 2024-03-11T21:01:39.736 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:01:39.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:39.741 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:01:39.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:39 smithi146 bash[23809]: cluster 2024-03-11T21:01:38.370863+0000 osd.6 (osd.6) 148 : cluster 0 4.1f scrub starts 2024-03-11T21:01:39.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:39 smithi146 bash[23809]: cluster 2024-03-11T21:01:38.377494+0000 osd.6 (osd.6) 149 : cluster 0 4.1f scrub ok 2024-03-11T21:01:39.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:39 smithi146 bash[23809]: cluster 2024-03-11T21:01:38.493584+0000 osd.7 (osd.7) 211 : cluster 0 2.8 scrub starts 2024-03-11T21:01:39.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:39 smithi146 bash[23809]: cluster 2024-03-11T21:01:38.494524+0000 osd.7 (osd.7) 212 : cluster 0 2.8 scrub ok 2024-03-11T21:01:39.842 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:01:39.865 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:01:39.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:39.869 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:01:39.970 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:01:39.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:39.999 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:01:39.999 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:01:40.100 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:01:40.123 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:01:40.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:40.128 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:01:40.229 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:40.251 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:40.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:40.257 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:01:40.357 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:01:40.390 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:01:40.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:40.393 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:01:40.494 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:01:40.516 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:01:40.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:40.520 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:01:40.621 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:01:40.656 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:01:40.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:40.660 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:01:40.761 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:40.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:40 smithi146 bash[23809]: cluster 2024-03-11T21:01:40.110463+0000 mgr.y (mgr.24380) 403 : cluster 0 pgmap v431: 148 pgs: 142 active+clean, 3 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped; 1.3 GiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 682 B/s rd, 0 B/s wr, 856 op/s; 11559/62553 objects degraded (18.479%); 8.9 MiB/s, 142 objects/s recovering 2024-03-11T21:01:40.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:40 smithi146 bash[23809]: cluster 2024-03-11T21:01:40.161473+0000 mon.a (mon.0) 1345 : cluster 3 Health check update: Degraded data redundancy: 11559/67671 objects degraded (17.081%), 6 pgs degraded (PG_DEGRADED) 2024-03-11T21:01:40.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:40 smithi146 bash[23809]: cluster 2024-03-11T21:01:40.178021+0000 mon.a (mon.0) 1346 : cluster 0 osdmap e227: 8 total, 8 up, 8 in 2024-03-11T21:01:40.790 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:40.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:40.792 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:01:40.893 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:40.913 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:40.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:40.915 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:01:41.015 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:01:41.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:40 smithi023 bash[21372]: cluster 2024-03-11T21:01:40.110463+0000 mgr.y (mgr.24380) 403 : cluster 0 pgmap v431: 148 pgs: 142 active+clean, 3 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped; 1.3 GiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 682 B/s rd, 0 B/s wr, 856 op/s; 11559/62553 objects degraded (18.479%); 8.9 MiB/s, 142 objects/s recovering 2024-03-11T21:01:41.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:40 smithi023 bash[21372]: cluster 2024-03-11T21:01:40.161473+0000 mon.a (mon.0) 1345 : cluster 3 Health check update: Degraded data redundancy: 11559/67671 objects degraded (17.081%), 6 pgs degraded (PG_DEGRADED) 2024-03-11T21:01:41.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:40 smithi023 bash[21372]: cluster 2024-03-11T21:01:40.178021+0000 mon.a (mon.0) 1346 : cluster 0 osdmap e227: 8 total, 8 up, 8 in 2024-03-11T21:01:41.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:40 smithi023 bash[29146]: cluster 2024-03-11T21:01:40.110463+0000 mgr.y (mgr.24380) 403 : cluster 0 pgmap v431: 148 pgs: 142 active+clean, 3 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped; 1.3 GiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 682 B/s rd, 0 B/s wr, 856 op/s; 11559/62553 objects degraded (18.479%); 8.9 MiB/s, 142 objects/s recovering 2024-03-11T21:01:41.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:40 smithi023 bash[29146]: cluster 2024-03-11T21:01:40.161473+0000 mon.a (mon.0) 1345 : cluster 3 Health check update: Degraded data redundancy: 11559/67671 objects degraded (17.081%), 6 pgs degraded (PG_DEGRADED) 2024-03-11T21:01:41.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:40 smithi023 bash[29146]: cluster 2024-03-11T21:01:40.178021+0000 mon.a (mon.0) 1346 : cluster 0 osdmap e227: 8 total, 8 up, 8 in 2024-03-11T21:01:41.056 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:01:41.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:41.060 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:01:41.161 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:41.188 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:41.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:41.196 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:01:41.272 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd reweight 2 1 2024-03-11T21:01:41.296 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:41.322 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:41.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:41.325 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:01:41.426 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:01:41.480 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:01:41.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:41.482 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:01:41.583 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:41.603 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:41.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:41.605 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:01:41.705 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:41.728 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:41.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:41.733 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:01:41.834 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:41.842 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:41 smithi146 bash[23809]: cluster 2024-03-11T21:01:40.560027+0000 osd.7 (osd.7) 213 : cluster 0 3.e scrub starts 2024-03-11T21:01:41.842 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:41 smithi146 bash[23809]: cluster 2024-03-11T21:01:40.561733+0000 osd.7 (osd.7) 214 : cluster 0 3.e scrub ok 2024-03-11T21:01:41.854 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:41.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:41.862 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:01:41.963 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:41.974 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:41 smithi023 bash[21372]: cluster 2024-03-11T21:01:40.560027+0000 osd.7 (osd.7) 213 : cluster 0 3.e scrub starts 2024-03-11T21:01:41.974 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:41 smithi023 bash[21372]: cluster 2024-03-11T21:01:40.561733+0000 osd.7 (osd.7) 214 : cluster 0 3.e scrub ok 2024-03-11T21:01:41.975 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:41 smithi023 bash[29146]: cluster 2024-03-11T21:01:40.560027+0000 osd.7 (osd.7) 213 : cluster 0 3.e scrub starts 2024-03-11T21:01:41.975 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:41 smithi023 bash[29146]: cluster 2024-03-11T21:01:40.561733+0000 osd.7 (osd.7) 214 : cluster 0 3.e scrub ok 2024-03-11T21:01:41.996 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:42.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:42.000 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:01:42.101 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:01:42.123 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:01:42.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:42.128 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:01:42.229 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:42.264 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:42.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:42.269 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:01:42.370 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:01:42.400 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:01:42.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:42.405 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:01:42.506 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:01:42.542 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:01:42.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:42.548 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:01:42.649 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:01:42.675 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:01:42.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:42.680 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:01:42.781 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:01:42.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:42 smithi023 bash[21372]: cluster 2024-03-11T21:01:42.111321+0000 mgr.y (mgr.24380) 404 : cluster 0 pgmap v433: 148 pgs: 142 active+clean, 3 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped; 1.1 GiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 710 op/s; 10879/55620 objects degraded (19.560%); 7.1 MiB/s, 113 objects/s recovering 2024-03-11T21:01:42.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:42 smithi023 bash[29146]: cluster 2024-03-11T21:01:42.111321+0000 mgr.y (mgr.24380) 404 : cluster 0 pgmap v433: 148 pgs: 142 active+clean, 3 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped; 1.1 GiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 710 op/s; 10879/55620 objects degraded (19.560%); 7.1 MiB/s, 113 objects/s recovering 2024-03-11T21:01:42.808 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:01:42.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:42.813 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:01:42.913 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:01:42.962 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:01:42.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:42.965 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:01:43.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:42 smithi146 bash[23809]: cluster 2024-03-11T21:01:42.111321+0000 mgr.y (mgr.24380) 404 : cluster 0 pgmap v433: 148 pgs: 142 active+clean, 3 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped; 1.1 GiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 710 op/s; 10879/55620 objects degraded (19.560%); 7.1 MiB/s, 113 objects/s recovering 2024-03-11T21:01:43.066 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:01:43.087 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:01:43.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:43.093 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:01:43.194 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:43.219 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:43.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:43.224 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:01:43.247 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T21:01:43.291 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-11T21:01:43.325 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:43.350 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:43.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:43.352 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:01:43.371 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:01:43.453 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:01:43.506 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:01:43.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:43.510 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:01:43.610 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:01:43.643 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:01:43.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:43.648 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:01:43.749 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:01:43.758 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:43 smithi023 bash[21372]: cluster 2024-03-11T21:01:42.349183+0000 osd.6 (osd.6) 150 : cluster 0 4.a scrub starts 2024-03-11T21:01:43.758 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:43 smithi023 bash[21372]: cluster 2024-03-11T21:01:42.350305+0000 osd.6 (osd.6) 151 : cluster 0 4.a scrub ok 2024-03-11T21:01:43.759 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:43 smithi023 bash[29146]: cluster 2024-03-11T21:01:42.349183+0000 osd.6 (osd.6) 150 : cluster 0 4.a scrub starts 2024-03-11T21:01:43.759 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:43 smithi023 bash[29146]: cluster 2024-03-11T21:01:42.350305+0000 osd.6 (osd.6) 151 : cluster 0 4.a scrub ok 2024-03-11T21:01:43.774 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:01:43.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:43.780 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:01:43.881 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:01:43.903 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:01:43.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:43.906 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:01:44.007 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:01:44.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:43 smithi146 bash[23809]: cluster 2024-03-11T21:01:42.349183+0000 osd.6 (osd.6) 150 : cluster 0 4.a scrub starts 2024-03-11T21:01:44.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:43 smithi146 bash[23809]: cluster 2024-03-11T21:01:42.350305+0000 osd.6 (osd.6) 151 : cluster 0 4.a scrub ok 2024-03-11T21:01:44.031 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:01:44.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:44.035 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:01:44.136 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:44.158 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:44.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:44.160 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:01:44.261 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:44.286 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:44.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:44.292 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:01:44.393 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:44.414 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:44.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:44.416 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:01:44.517 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:44.538 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:44.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:44.539 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:01:44.640 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:01:44.671 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:01:44.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:44.678 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:01:44.779 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:01:44.822 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:01:44.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:44.824 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:01:44.925 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:01:44.933 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:44 smithi146 bash[23809]: cluster 2024-03-11T21:01:44.111881+0000 mgr.y (mgr.24380) 405 : cluster 0 pgmap v434: 148 pgs: 142 active+clean, 3 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped; 1.1 GiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 677 B/s rd, 0 B/s wr, 564 op/s; 10879/55620 objects degraded (19.560%); 5.6 MiB/s, 90 objects/s recovering 2024-03-11T21:01:44.947 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:01:44.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:44.951 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:01:45.021 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:44 smithi023 bash[21372]: cluster 2024-03-11T21:01:44.111881+0000 mgr.y (mgr.24380) 405 : cluster 0 pgmap v434: 148 pgs: 142 active+clean, 3 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped; 1.1 GiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 677 B/s rd, 0 B/s wr, 564 op/s; 10879/55620 objects degraded (19.560%); 5.6 MiB/s, 90 objects/s recovering 2024-03-11T21:01:45.021 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:44 smithi023 bash[29146]: cluster 2024-03-11T21:01:44.111881+0000 mgr.y (mgr.24380) 405 : cluster 0 pgmap v434: 148 pgs: 142 active+clean, 3 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped; 1.1 GiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 677 B/s rd, 0 B/s wr, 564 op/s; 10879/55620 objects degraded (19.560%); 5.6 MiB/s, 90 objects/s recovering 2024-03-11T21:01:45.052 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:45.074 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:45.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:45.079 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:01:45.180 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:01:45.201 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:01:45.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:45.205 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:01:45.306 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:01:45.332 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:01:45.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:45.338 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:01:45.409 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:01:45.440 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:45.472 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:45.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:45.483 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:01:45.512 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:01:45 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:01:45.584 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:45.606 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:45.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:45.608 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:01:45.709 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:01:45.741 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:01:45.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:45.745 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:01:45.845 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:01:45.855 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:45 smithi146 bash[23809]: cluster 2024-03-11T21:01:44.353620+0000 osd.6 (osd.6) 152 : cluster 0 3.a scrub starts 2024-03-11T21:01:45.855 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:45 smithi146 bash[23809]: cluster 2024-03-11T21:01:44.358278+0000 osd.6 (osd.6) 153 : cluster 0 3.a scrub ok 2024-03-11T21:01:45.855 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:45 smithi146 bash[23809]: cluster 2024-03-11T21:01:45.162834+0000 mon.a (mon.0) 1347 : cluster 3 Health check update: Degraded data redundancy: 10879/55620 objects degraded (19.560%), 6 pgs degraded (PG_DEGRADED) 2024-03-11T21:01:45.855 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:45 smithi146 bash[23809]: audit 2024-03-11T21:01:45.350796+0000 mgr.y (mgr.24380) 406 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:01:45.868 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:01:45.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:45.870 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:01:45.971 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:45.996 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:45.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:45.998 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:01:46.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:45 smithi023 bash[21372]: cluster 2024-03-11T21:01:44.353620+0000 osd.6 (osd.6) 152 : cluster 0 3.a scrub starts 2024-03-11T21:01:46.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:45 smithi023 bash[21372]: cluster 2024-03-11T21:01:44.358278+0000 osd.6 (osd.6) 153 : cluster 0 3.a scrub ok 2024-03-11T21:01:46.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:45 smithi023 bash[21372]: cluster 2024-03-11T21:01:45.162834+0000 mon.a (mon.0) 1347 : cluster 3 Health check update: Degraded data redundancy: 10879/55620 objects degraded (19.560%), 6 pgs degraded (PG_DEGRADED) 2024-03-11T21:01:46.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:45 smithi023 bash[21372]: audit 2024-03-11T21:01:45.350796+0000 mgr.y (mgr.24380) 406 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:01:46.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:45 smithi023 bash[29146]: cluster 2024-03-11T21:01:44.353620+0000 osd.6 (osd.6) 152 : cluster 0 3.a scrub starts 2024-03-11T21:01:46.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:45 smithi023 bash[29146]: cluster 2024-03-11T21:01:44.358278+0000 osd.6 (osd.6) 153 : cluster 0 3.a scrub ok 2024-03-11T21:01:46.036 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:45 smithi023 bash[29146]: cluster 2024-03-11T21:01:45.162834+0000 mon.a (mon.0) 1347 : cluster 3 Health check update: Degraded data redundancy: 10879/55620 objects degraded (19.560%), 6 pgs degraded (PG_DEGRADED) 2024-03-11T21:01:46.036 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:45 smithi023 bash[29146]: audit 2024-03-11T21:01:45.350796+0000 mgr.y (mgr.24380) 406 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:01:46.098 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:01:46.119 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:01:46.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:46.121 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:01:46.222 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:46.255 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:46.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:46.258 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:01:46.358 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:46.382 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:46.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:46.386 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:01:46.487 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:46.511 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:46.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:46.513 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:01:46.614 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:46.640 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:46.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:46.645 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:01:46.745 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:01:46.767 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:01:46.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:46.770 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:01:46.871 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:46.878 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:46 smithi146 bash[23809]: cluster 2024-03-11T21:01:46.112733+0000 mgr.y (mgr.24380) 407 : cluster 0 pgmap v435: 148 pgs: 142 active+clean, 1 active+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped; 893 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 946 op/s; 10037/43431 objects degraded (23.110%); 11 MiB/s, 178 objects/s recovering 2024-03-11T21:01:46.890 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:46.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:46.892 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:01:46.993 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:01:47.001 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:46 smithi023 bash[21372]: cluster 2024-03-11T21:01:46.112733+0000 mgr.y (mgr.24380) 407 : cluster 0 pgmap v435: 148 pgs: 142 active+clean, 1 active+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped; 893 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 946 op/s; 10037/43431 objects degraded (23.110%); 11 MiB/s, 178 objects/s recovering 2024-03-11T21:01:47.002 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:46 smithi023 bash[29146]: cluster 2024-03-11T21:01:46.112733+0000 mgr.y (mgr.24380) 407 : cluster 0 pgmap v435: 148 pgs: 142 active+clean, 1 active+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped; 893 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 946 op/s; 10037/43431 objects degraded (23.110%); 11 MiB/s, 178 objects/s recovering 2024-03-11T21:01:47.013 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:01:47.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:47.016 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:01:47.117 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:01:47.144 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:01:47.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:47.147 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:01:47.248 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:01:47.276 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:01:47.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:47.278 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:01:47.379 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:47.404 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:47.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:47.407 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:01:47.433 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:01:47.433 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:01:47.508 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:01:47.536 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:01:47.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:47.539 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:01:47.639 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:01:47.672 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:01:47.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:47.675 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:01:47.776 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:01:47.784 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:47 smithi023 bash[21372]: cluster 2024-03-11T21:01:46.660651+0000 mon.a (mon.0) 1348 : cluster 0 osdmap e228: 8 total, 8 up, 8 in 2024-03-11T21:01:47.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:47 smithi023 bash[29146]: cluster 2024-03-11T21:01:46.660651+0000 mon.a (mon.0) 1348 : cluster 0 osdmap e228: 8 total, 8 up, 8 in 2024-03-11T21:01:47.797 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:01:47.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:47.799 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:01:47.900 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:47.928 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:47.929 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:47 smithi146 bash[23809]: cluster 2024-03-11T21:01:46.660651+0000 mon.a (mon.0) 1348 : cluster 0 osdmap e228: 8 total, 8 up, 8 in 2024-03-11T21:01:47.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:47.931 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:01:48.032 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:01:48.054 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:01:48.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:48.057 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:01:48.157 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:48.175 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:48.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:48.177 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:01:48.278 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:01:48.300 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:01:48.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:48.302 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:01:48.403 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:01:48.424 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:01:48.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:48.427 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:01:48.527 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:48.559 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:48.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:48.564 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:01:48.658 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:01:48 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:01:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:01:48.664 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:48.693 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:48.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:48.695 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:01:48.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:48 smithi023 bash[21372]: cluster 2024-03-11T21:01:47.343139+0000 osd.6 (osd.6) 154 : cluster 0 2.11 scrub starts 2024-03-11T21:01:48.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:48 smithi023 bash[21372]: cluster 2024-03-11T21:01:47.346112+0000 osd.6 (osd.6) 155 : cluster 0 2.11 scrub ok 2024-03-11T21:01:48.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:48 smithi023 bash[21372]: cluster 2024-03-11T21:01:47.661385+0000 mon.a (mon.0) 1349 : cluster 0 osdmap e229: 8 total, 8 up, 8 in 2024-03-11T21:01:48.786 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:48 smithi023 bash[21372]: cluster 2024-03-11T21:01:48.113143+0000 mgr.y (mgr.24380) 408 : cluster 0 pgmap v438: 148 pgs: 142 active+clean, 1 active+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped; 877 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 837 op/s; 10037/42642 objects degraded (23.538%); 12 MiB/s, 191 objects/s recovering 2024-03-11T21:01:48.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:48 smithi023 bash[29146]: cluster 2024-03-11T21:01:47.343139+0000 osd.6 (osd.6) 154 : cluster 0 2.11 scrub starts 2024-03-11T21:01:48.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:48 smithi023 bash[29146]: cluster 2024-03-11T21:01:47.346112+0000 osd.6 (osd.6) 155 : cluster 0 2.11 scrub ok 2024-03-11T21:01:48.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:48 smithi023 bash[29146]: cluster 2024-03-11T21:01:47.661385+0000 mon.a (mon.0) 1349 : cluster 0 osdmap e229: 8 total, 8 up, 8 in 2024-03-11T21:01:48.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:48 smithi023 bash[29146]: cluster 2024-03-11T21:01:48.113143+0000 mgr.y (mgr.24380) 408 : cluster 0 pgmap v438: 148 pgs: 142 active+clean, 1 active+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped; 877 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 837 op/s; 10037/42642 objects degraded (23.538%); 12 MiB/s, 191 objects/s recovering 2024-03-11T21:01:48.796 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:01:48.822 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:01:48.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:48.826 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:01:48.927 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:48.936 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:48 smithi146 bash[23809]: cluster 2024-03-11T21:01:47.343139+0000 osd.6 (osd.6) 154 : cluster 0 2.11 scrub starts 2024-03-11T21:01:48.936 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:48 smithi146 bash[23809]: cluster 2024-03-11T21:01:47.346112+0000 osd.6 (osd.6) 155 : cluster 0 2.11 scrub ok 2024-03-11T21:01:48.936 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:48 smithi146 bash[23809]: cluster 2024-03-11T21:01:47.661385+0000 mon.a (mon.0) 1349 : cluster 0 osdmap e229: 8 total, 8 up, 8 in 2024-03-11T21:01:48.936 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:48 smithi146 bash[23809]: cluster 2024-03-11T21:01:48.113143+0000 mgr.y (mgr.24380) 408 : cluster 0 pgmap v438: 148 pgs: 142 active+clean, 1 active+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped; 877 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 837 op/s; 10037/42642 objects degraded (23.538%); 12 MiB/s, 191 objects/s recovering 2024-03-11T21:01:48.954 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:48.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:48.957 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:01:49.057 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:01:49.081 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:01:49.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:49.083 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:01:49.184 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:49.209 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:49.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:49.211 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:01:49.312 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:49.334 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:49.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:49.336 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:01:49.437 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:01:49.465 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:01:49.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:49.467 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:01:49.567 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:01:49.588 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:01:49.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:49.591 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:01:49.692 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:01:49.703 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:49 smithi146 bash[23809]: cluster 2024-03-11T21:01:48.340533+0000 osd.6 (osd.6) 156 : cluster 0 2.1a scrub starts 2024-03-11T21:01:49.703 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:49 smithi146 bash[23809]: cluster 2024-03-11T21:01:48.345201+0000 osd.6 (osd.6) 157 : cluster 0 2.1a scrub ok 2024-03-11T21:01:49.703 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:49 smithi146 bash[23809]: cluster 2024-03-11T21:01:48.496580+0000 osd.4 (osd.4) 5 : cluster 0 4.1b scrub starts 2024-03-11T21:01:49.703 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:49 smithi146 bash[23809]: cluster 2024-03-11T21:01:48.498458+0000 osd.4 (osd.4) 6 : cluster 0 4.1b scrub ok 2024-03-11T21:01:49.703 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:49 smithi146 bash[23809]: cluster 2024-03-11T21:01:48.687383+0000 mon.a (mon.0) 1350 : cluster 0 osdmap e230: 8 total, 8 up, 8 in 2024-03-11T21:01:49.719 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:01:49.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:49.721 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:01:49.822 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:01:49.849 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:01:49.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:49.854 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:01:49.954 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:01:49.965 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:49 smithi023 bash[21372]: cluster 2024-03-11T21:01:48.340533+0000 osd.6 (osd.6) 156 : cluster 0 2.1a scrub starts 2024-03-11T21:01:49.965 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:49 smithi023 bash[21372]: cluster 2024-03-11T21:01:48.345201+0000 osd.6 (osd.6) 157 : cluster 0 2.1a scrub ok 2024-03-11T21:01:49.965 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:49 smithi023 bash[21372]: cluster 2024-03-11T21:01:48.496580+0000 osd.4 (osd.4) 5 : cluster 0 4.1b scrub starts 2024-03-11T21:01:49.965 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:49 smithi023 bash[21372]: cluster 2024-03-11T21:01:48.498458+0000 osd.4 (osd.4) 6 : cluster 0 4.1b scrub ok 2024-03-11T21:01:49.965 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:49 smithi023 bash[21372]: cluster 2024-03-11T21:01:48.687383+0000 mon.a (mon.0) 1350 : cluster 0 osdmap e230: 8 total, 8 up, 8 in 2024-03-11T21:01:49.965 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:49 smithi023 bash[29146]: cluster 2024-03-11T21:01:48.340533+0000 osd.6 (osd.6) 156 : cluster 0 2.1a scrub starts 2024-03-11T21:01:49.966 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:49 smithi023 bash[29146]: cluster 2024-03-11T21:01:48.345201+0000 osd.6 (osd.6) 157 : cluster 0 2.1a scrub ok 2024-03-11T21:01:49.966 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:49 smithi023 bash[29146]: cluster 2024-03-11T21:01:48.496580+0000 osd.4 (osd.4) 5 : cluster 0 4.1b scrub starts 2024-03-11T21:01:49.966 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:49 smithi023 bash[29146]: cluster 2024-03-11T21:01:48.498458+0000 osd.4 (osd.4) 6 : cluster 0 4.1b scrub ok 2024-03-11T21:01:49.966 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:49 smithi023 bash[29146]: cluster 2024-03-11T21:01:48.687383+0000 mon.a (mon.0) 1350 : cluster 0 osdmap e230: 8 total, 8 up, 8 in 2024-03-11T21:01:49.990 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:01:49.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:49.993 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:01:50.094 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:01:50.127 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:01:50.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:50.131 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:01:50.232 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:50.262 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:50.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:50.267 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:01:50.368 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:50.402 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:50.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:50.405 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:01:50.506 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:01:50.528 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:01:50.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:50.533 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:01:50.634 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:50.662 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:50.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:50.665 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:01:50.765 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:50.801 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:50.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:50.804 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:01:50.904 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:01:50.922 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:01:50.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:50.923 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:01:50.977 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:50 smithi023 bash[21372]: cluster 2024-03-11T21:01:49.460212+0000 osd.4 (osd.4) 7 : cluster 0 4.d scrub starts 2024-03-11T21:01:50.977 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:50 smithi023 bash[21372]: cluster 2024-03-11T21:01:49.461799+0000 osd.4 (osd.4) 8 : cluster 0 4.d scrub ok 2024-03-11T21:01:50.977 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:50 smithi023 bash[21372]: cluster 2024-03-11T21:01:49.478691+0000 osd.7 (osd.7) 215 : cluster 0 6.15 scrub starts 2024-03-11T21:01:50.977 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:50 smithi023 bash[21372]: cluster 2024-03-11T21:01:49.493979+0000 osd.7 (osd.7) 216 : cluster 0 6.15 scrub ok 2024-03-11T21:01:50.977 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:50 smithi023 bash[21372]: cluster 2024-03-11T21:01:49.684907+0000 mon.a (mon.0) 1351 : cluster 0 osdmap e231: 8 total, 8 up, 8 in 2024-03-11T21:01:50.977 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:50 smithi023 bash[21372]: audit 2024-03-11T21:01:49.986195+0000 mon.a (mon.0) 1352 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:01:50.977 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:50 smithi023 bash[21372]: cluster 2024-03-11T21:01:50.115188+0000 mgr.y (mgr.24380) 409 : cluster 0 pgmap v441: 148 pgs: 142 active+clean, 1 active+undersized+remapped, 4 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped; 829 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 256 op/s; 10034/40353 objects degraded (24.866%); 48 KiB/s, 0 objects/s recovering 2024-03-11T21:01:50.978 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:50 smithi023 bash[21372]: cluster 2024-03-11T21:01:50.163972+0000 mon.a (mon.0) 1353 : cluster 3 Health check update: Degraded data redundancy: 10037/42642 objects degraded (23.538%), 5 pgs degraded (PG_DEGRADED) 2024-03-11T21:01:50.978 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:50 smithi023 bash[29146]: cluster 2024-03-11T21:01:49.460212+0000 osd.4 (osd.4) 7 : cluster 0 4.d scrub starts 2024-03-11T21:01:50.978 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:50 smithi023 bash[29146]: cluster 2024-03-11T21:01:49.461799+0000 osd.4 (osd.4) 8 : cluster 0 4.d scrub ok 2024-03-11T21:01:50.978 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:50 smithi023 bash[29146]: cluster 2024-03-11T21:01:49.478691+0000 osd.7 (osd.7) 215 : cluster 0 6.15 scrub starts 2024-03-11T21:01:50.978 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:50 smithi023 bash[29146]: cluster 2024-03-11T21:01:49.493979+0000 osd.7 (osd.7) 216 : cluster 0 6.15 scrub ok 2024-03-11T21:01:50.978 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:50 smithi023 bash[29146]: cluster 2024-03-11T21:01:49.684907+0000 mon.a (mon.0) 1351 : cluster 0 osdmap e231: 8 total, 8 up, 8 in 2024-03-11T21:01:50.978 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:50 smithi023 bash[29146]: audit 2024-03-11T21:01:49.986195+0000 mon.a (mon.0) 1352 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:01:50.978 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:50 smithi023 bash[29146]: cluster 2024-03-11T21:01:50.115188+0000 mgr.y (mgr.24380) 409 : cluster 0 pgmap v441: 148 pgs: 142 active+clean, 1 active+undersized+remapped, 4 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped; 829 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 256 op/s; 10034/40353 objects degraded (24.866%); 48 KiB/s, 0 objects/s recovering 2024-03-11T21:01:50.978 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:50 smithi023 bash[29146]: cluster 2024-03-11T21:01:50.163972+0000 mon.a (mon.0) 1353 : cluster 3 Health check update: Degraded data redundancy: 10037/42642 objects degraded (23.538%), 5 pgs degraded (PG_DEGRADED) 2024-03-11T21:01:51.013 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:50 smithi146 bash[23809]: cluster 2024-03-11T21:01:49.460212+0000 osd.4 (osd.4) 7 : cluster 0 4.d scrub starts 2024-03-11T21:01:51.013 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:50 smithi146 bash[23809]: cluster 2024-03-11T21:01:49.461799+0000 osd.4 (osd.4) 8 : cluster 0 4.d scrub ok 2024-03-11T21:01:51.013 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:50 smithi146 bash[23809]: cluster 2024-03-11T21:01:49.478691+0000 osd.7 (osd.7) 215 : cluster 0 6.15 scrub starts 2024-03-11T21:01:51.013 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:50 smithi146 bash[23809]: cluster 2024-03-11T21:01:49.493979+0000 osd.7 (osd.7) 216 : cluster 0 6.15 scrub ok 2024-03-11T21:01:51.013 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:50 smithi146 bash[23809]: cluster 2024-03-11T21:01:49.684907+0000 mon.a (mon.0) 1351 : cluster 0 osdmap e231: 8 total, 8 up, 8 in 2024-03-11T21:01:51.013 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:50 smithi146 bash[23809]: audit 2024-03-11T21:01:49.986195+0000 mon.a (mon.0) 1352 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:01:51.014 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:50 smithi146 bash[23809]: cluster 2024-03-11T21:01:50.115188+0000 mgr.y (mgr.24380) 409 : cluster 0 pgmap v441: 148 pgs: 142 active+clean, 1 active+undersized+remapped, 4 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped; 829 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 256 op/s; 10034/40353 objects degraded (24.866%); 48 KiB/s, 0 objects/s recovering 2024-03-11T21:01:51.014 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:50 smithi146 bash[23809]: cluster 2024-03-11T21:01:50.163972+0000 mon.a (mon.0) 1353 : cluster 3 Health check update: Degraded data redundancy: 10037/42642 objects degraded (23.538%), 5 pgs degraded (PG_DEGRADED) 2024-03-11T21:01:51.024 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:01:51.056 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:01:51.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:51.062 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:01:51.163 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:01:51.190 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:01:51.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:51.192 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:01:51.292 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:01:51.313 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:01:51.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:51.318 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:01:51.418 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:51.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:51.449 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:51.450 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:01:51.551 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:01:51.573 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:01:51.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:51.578 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:01:51.678 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:01:51.702 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:01:51.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:51.708 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:01:51.809 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:01:51.856 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:01:51.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:51.859 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:01:51.960 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:01:51.970 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:51 smithi023 bash[21372]: cluster 2024-03-11T21:01:50.469515+0000 osd.4 (osd.4) 9 : cluster 0 4.e scrub starts 2024-03-11T21:01:51.970 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:51 smithi023 bash[21372]: cluster 2024-03-11T21:01:50.475429+0000 osd.4 (osd.4) 10 : cluster 0 4.e scrub ok 2024-03-11T21:01:51.970 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:51 smithi023 bash[21372]: audit 2024-03-11T21:01:51.576977+0000 mon.a (mon.0) 1354 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:01:51.970 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:51 smithi023 bash[29146]: cluster 2024-03-11T21:01:50.469515+0000 osd.4 (osd.4) 9 : cluster 0 4.e scrub starts 2024-03-11T21:01:51.970 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:51 smithi023 bash[29146]: cluster 2024-03-11T21:01:50.475429+0000 osd.4 (osd.4) 10 : cluster 0 4.e scrub ok 2024-03-11T21:01:51.970 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:51 smithi023 bash[29146]: audit 2024-03-11T21:01:51.576977+0000 mon.a (mon.0) 1354 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:01:51.993 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:01:51.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:51.999 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:01:52.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:51 smithi146 bash[23809]: cluster 2024-03-11T21:01:50.469515+0000 osd.4 (osd.4) 9 : cluster 0 4.e scrub starts 2024-03-11T21:01:52.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:51 smithi146 bash[23809]: cluster 2024-03-11T21:01:50.475429+0000 osd.4 (osd.4) 10 : cluster 0 4.e scrub ok 2024-03-11T21:01:52.013 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:51 smithi146 bash[23809]: audit 2024-03-11T21:01:51.576977+0000 mon.a (mon.0) 1354 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:01:52.100 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:52.129 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:52.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:52.131 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:01:52.232 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:52.251 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:52.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:52.253 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:01:52.354 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:01:52.377 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:01:52.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:52.380 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:01:52.481 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:01:52.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:52.511 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:01:52.511 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:01:52.612 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:01:52.640 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:01:52.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:52.641 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:01:52.742 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:01:52.766 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:01:52.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:52.771 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:01:52.872 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:01:52.887 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:52 smithi023 bash[21372]: cluster 2024-03-11T21:01:51.503187+0000 osd.7 (osd.7) 217 : cluster 0 2.1f scrub starts 2024-03-11T21:01:52.887 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:52 smithi023 bash[21372]: cluster 2024-03-11T21:01:51.504285+0000 osd.7 (osd.7) 218 : cluster 0 2.1f scrub ok 2024-03-11T21:01:52.887 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:52 smithi023 bash[21372]: cluster 2024-03-11T21:01:52.116398+0000 mgr.y (mgr.24380) 410 : cluster 0 pgmap v442: 148 pgs: 143 active+clean, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped; 758 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 937 B/s rd, 0 B/s wr, 396 op/s; 8642/36945 objects degraded (23.392%); 10 MiB/s, 255 objects/s recovering 2024-03-11T21:01:52.887 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:52 smithi023 bash[21372]: audit 2024-03-11T21:01:52.275520+0000 mon.a (mon.0) 1355 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:01:52.887 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:52 smithi023 bash[21372]: audit 2024-03-11T21:01:52.287012+0000 mon.a (mon.0) 1356 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:01:52.887 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:52 smithi023 bash[21372]: audit 2024-03-11T21:01:52.306563+0000 mon.a (mon.0) 1357 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:01:52.887 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:52 smithi023 bash[21372]: audit 2024-03-11T21:01:52.309717+0000 mon.a (mon.0) 1358 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:01:52.888 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:52 smithi023 bash[21372]: audit 2024-03-11T21:01:52.311065+0000 mon.a (mon.0) 1359 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:01:52.888 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:52 smithi023 bash[21372]: audit 2024-03-11T21:01:52.312265+0000 mon.a (mon.0) 1360 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:01:52.888 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:52 smithi023 bash[21372]: cephadm 2024-03-11T21:01:52.313144+0000 mgr.y (mgr.24380) 411 : cephadm 1 Adjusting osd_memory_target on smithi146 to 3292M 2024-03-11T21:01:52.888 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:52 smithi023 bash[21372]: audit 2024-03-11T21:01:52.321479+0000 mon.a (mon.0) 1361 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:01:52.888 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:52 smithi023 bash[29146]: cluster 2024-03-11T21:01:51.503187+0000 osd.7 (osd.7) 217 : cluster 0 2.1f scrub starts 2024-03-11T21:01:52.888 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:52 smithi023 bash[29146]: cluster 2024-03-11T21:01:51.504285+0000 osd.7 (osd.7) 218 : cluster 0 2.1f scrub ok 2024-03-11T21:01:52.888 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:52 smithi023 bash[29146]: cluster 2024-03-11T21:01:52.116398+0000 mgr.y (mgr.24380) 410 : cluster 0 pgmap v442: 148 pgs: 143 active+clean, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped; 758 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 937 B/s rd, 0 B/s wr, 396 op/s; 8642/36945 objects degraded (23.392%); 10 MiB/s, 255 objects/s recovering 2024-03-11T21:01:52.888 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:52 smithi023 bash[29146]: audit 2024-03-11T21:01:52.275520+0000 mon.a (mon.0) 1355 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:01:52.889 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:52 smithi023 bash[29146]: audit 2024-03-11T21:01:52.287012+0000 mon.a (mon.0) 1356 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:01:52.889 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:52 smithi023 bash[29146]: audit 2024-03-11T21:01:52.306563+0000 mon.a (mon.0) 1357 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:01:52.889 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:52 smithi023 bash[29146]: audit 2024-03-11T21:01:52.309717+0000 mon.a (mon.0) 1358 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:01:52.889 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:52 smithi023 bash[29146]: audit 2024-03-11T21:01:52.311065+0000 mon.a (mon.0) 1359 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:01:52.889 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:52 smithi023 bash[29146]: audit 2024-03-11T21:01:52.312265+0000 mon.a (mon.0) 1360 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:01:52.889 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:52 smithi023 bash[29146]: cephadm 2024-03-11T21:01:52.313144+0000 mgr.y (mgr.24380) 411 : cephadm 1 Adjusting osd_memory_target on smithi146 to 3292M 2024-03-11T21:01:52.889 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:52 smithi023 bash[29146]: audit 2024-03-11T21:01:52.321479+0000 mon.a (mon.0) 1361 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:01:52.905 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:01:52.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:52.909 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:01:53.010 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:53.014 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:52 smithi146 bash[23809]: cluster 2024-03-11T21:01:51.503187+0000 osd.7 (osd.7) 217 : cluster 0 2.1f scrub starts 2024-03-11T21:01:53.014 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:52 smithi146 bash[23809]: cluster 2024-03-11T21:01:51.504285+0000 osd.7 (osd.7) 218 : cluster 0 2.1f scrub ok 2024-03-11T21:01:53.014 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:52 smithi146 bash[23809]: cluster 2024-03-11T21:01:52.116398+0000 mgr.y (mgr.24380) 410 : cluster 0 pgmap v442: 148 pgs: 143 active+clean, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped; 758 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 937 B/s rd, 0 B/s wr, 396 op/s; 8642/36945 objects degraded (23.392%); 10 MiB/s, 255 objects/s recovering 2024-03-11T21:01:53.014 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:52 smithi146 bash[23809]: audit 2024-03-11T21:01:52.275520+0000 mon.a (mon.0) 1355 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:01:53.014 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:52 smithi146 bash[23809]: audit 2024-03-11T21:01:52.287012+0000 mon.a (mon.0) 1356 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:01:53.014 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:52 smithi146 bash[23809]: audit 2024-03-11T21:01:52.306563+0000 mon.a (mon.0) 1357 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:01:53.014 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:52 smithi146 bash[23809]: audit 2024-03-11T21:01:52.309717+0000 mon.a (mon.0) 1358 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:01:53.014 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:52 smithi146 bash[23809]: audit 2024-03-11T21:01:52.311065+0000 mon.a (mon.0) 1359 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:01:53.014 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:52 smithi146 bash[23809]: audit 2024-03-11T21:01:52.312265+0000 mon.a (mon.0) 1360 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:01:53.014 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:52 smithi146 bash[23809]: cephadm 2024-03-11T21:01:52.313144+0000 mgr.y (mgr.24380) 411 : cephadm 1 Adjusting osd_memory_target on smithi146 to 3292M 2024-03-11T21:01:53.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:52 smithi146 bash[23809]: audit 2024-03-11T21:01:52.321479+0000 mon.a (mon.0) 1361 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:01:53.037 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:53.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:53.039 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:01:53.141 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:53.198 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:53.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:53.200 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:01:53.301 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:01:53.325 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:01:53.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:53.327 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:01:53.428 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:53.461 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:53.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:53.464 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:01:53.565 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:01:53.593 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:01:53.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:53.599 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:01:53.700 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:01:53.723 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:01:53.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:53.727 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:01:53.828 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:01:53.849 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:01:53.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:53.852 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:01:53.953 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:01:53.962 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:53 smithi146 bash[23809]: cluster 2024-03-11T21:01:52.501296+0000 osd.4 (osd.4) 11 : cluster 0 6.2 scrub starts 2024-03-11T21:01:53.962 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:53 smithi146 bash[23809]: cluster 2024-03-11T21:01:52.505290+0000 osd.4 (osd.4) 12 : cluster 0 6.2 scrub ok 2024-03-11T21:01:53.962 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:53 smithi146 bash[23809]: audit 2024-03-11T21:01:53.069193+0000 mon.a (mon.0) 1362 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:01:53.962 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:53 smithi146 bash[23809]: audit 2024-03-11T21:01:53.070427+0000 mon.a (mon.0) 1363 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:01:53.962 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:53 smithi146 bash[23809]: audit 2024-03-11T21:01:53.071696+0000 mon.a (mon.0) 1364 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:01:53.962 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:53 smithi146 bash[23809]: audit 2024-03-11T21:01:53.072653+0000 mon.a (mon.0) 1365 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:01:53.962 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:53 smithi146 bash[23809]: cephadm 2024-03-11T21:01:53.073354+0000 mgr.y (mgr.24380) 412 : cephadm 1 Adjusting osd_memory_target on smithi023 to 3292M 2024-03-11T21:01:53.962 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:53 smithi146 bash[23809]: audit 2024-03-11T21:01:53.079864+0000 mon.a (mon.0) 1366 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:01:53.962 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:53 smithi146 bash[23809]: audit 2024-03-11T21:01:53.087130+0000 mon.a (mon.0) 1367 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:01:53.962 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:53 smithi146 bash[23809]: audit 2024-03-11T21:01:53.088266+0000 mon.a (mon.0) 1368 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:01:53.962 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:53 smithi146 bash[23809]: audit 2024-03-11T21:01:53.099784+0000 mon.a (mon.0) 1369 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:01:53.979 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:01:53.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:53.981 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:01:54.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:53 smithi023 bash[21372]: cluster 2024-03-11T21:01:52.501296+0000 osd.4 (osd.4) 11 : cluster 0 6.2 scrub starts 2024-03-11T21:01:54.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:53 smithi023 bash[21372]: cluster 2024-03-11T21:01:52.505290+0000 osd.4 (osd.4) 12 : cluster 0 6.2 scrub ok 2024-03-11T21:01:54.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:53 smithi023 bash[21372]: audit 2024-03-11T21:01:53.069193+0000 mon.a (mon.0) 1362 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:01:54.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:53 smithi023 bash[21372]: audit 2024-03-11T21:01:53.070427+0000 mon.a (mon.0) 1363 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:01:54.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:53 smithi023 bash[21372]: audit 2024-03-11T21:01:53.071696+0000 mon.a (mon.0) 1364 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:01:54.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:53 smithi023 bash[21372]: audit 2024-03-11T21:01:53.072653+0000 mon.a (mon.0) 1365 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:01:54.036 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:53 smithi023 bash[21372]: cephadm 2024-03-11T21:01:53.073354+0000 mgr.y (mgr.24380) 412 : cephadm 1 Adjusting osd_memory_target on smithi023 to 3292M 2024-03-11T21:01:54.036 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:53 smithi023 bash[21372]: audit 2024-03-11T21:01:53.079864+0000 mon.a (mon.0) 1366 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:01:54.036 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:53 smithi023 bash[21372]: audit 2024-03-11T21:01:53.087130+0000 mon.a (mon.0) 1367 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:01:54.036 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:53 smithi023 bash[21372]: audit 2024-03-11T21:01:53.088266+0000 mon.a (mon.0) 1368 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:01:54.036 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:53 smithi023 bash[21372]: audit 2024-03-11T21:01:53.099784+0000 mon.a (mon.0) 1369 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:01:54.036 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:53 smithi023 bash[29146]: cluster 2024-03-11T21:01:52.501296+0000 osd.4 (osd.4) 11 : cluster 0 6.2 scrub starts 2024-03-11T21:01:54.036 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:53 smithi023 bash[29146]: cluster 2024-03-11T21:01:52.505290+0000 osd.4 (osd.4) 12 : cluster 0 6.2 scrub ok 2024-03-11T21:01:54.036 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:53 smithi023 bash[29146]: audit 2024-03-11T21:01:53.069193+0000 mon.a (mon.0) 1362 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:01:54.036 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:53 smithi023 bash[29146]: audit 2024-03-11T21:01:53.070427+0000 mon.a (mon.0) 1363 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:01:54.036 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:53 smithi023 bash[29146]: audit 2024-03-11T21:01:53.071696+0000 mon.a (mon.0) 1364 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:01:54.036 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:53 smithi023 bash[29146]: audit 2024-03-11T21:01:53.072653+0000 mon.a (mon.0) 1365 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:01:54.037 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:53 smithi023 bash[29146]: cephadm 2024-03-11T21:01:53.073354+0000 mgr.y (mgr.24380) 412 : cephadm 1 Adjusting osd_memory_target on smithi023 to 3292M 2024-03-11T21:01:54.037 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:53 smithi023 bash[29146]: audit 2024-03-11T21:01:53.079864+0000 mon.a (mon.0) 1366 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:01:54.037 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:53 smithi023 bash[29146]: audit 2024-03-11T21:01:53.087130+0000 mon.a (mon.0) 1367 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:01:54.037 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:53 smithi023 bash[29146]: audit 2024-03-11T21:01:53.088266+0000 mon.a (mon.0) 1368 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:01:54.037 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:53 smithi023 bash[29146]: audit 2024-03-11T21:01:53.099784+0000 mon.a (mon.0) 1369 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:01:54.082 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:01:54.117 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:01:54.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:54.121 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:01:54.221 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:54.259 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:54.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:54.262 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:01:54.363 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:01:54.392 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:01:54.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:54.394 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:01:54.495 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:01:54.523 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:01:54.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:54.526 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:01:54.626 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:01:54.655 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:01:54.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:54.658 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:01:54.759 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:54.782 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:54.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:54.786 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:01:54.887 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:01:54.900 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:54 smithi023 bash[29146]: cluster 2024-03-11T21:01:54.116995+0000 mgr.y (mgr.24380) 413 : cluster 0 pgmap v443: 148 pgs: 143 active+clean, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped; 758 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 792 B/s rd, 0 B/s wr, 294 op/s; 8642/36945 objects degraded (23.392%); 8.8 MiB/s, 216 objects/s recovering 2024-03-11T21:01:54.900 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:54 smithi023 bash[21372]: cluster 2024-03-11T21:01:54.116995+0000 mgr.y (mgr.24380) 413 : cluster 0 pgmap v443: 148 pgs: 143 active+clean, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped; 758 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 792 B/s rd, 0 B/s wr, 294 op/s; 8642/36945 objects degraded (23.392%); 8.8 MiB/s, 216 objects/s recovering 2024-03-11T21:01:54.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:54.927 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:01:54.927 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:01:55.013 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:54 smithi146 bash[23809]: cluster 2024-03-11T21:01:54.116995+0000 mgr.y (mgr.24380) 413 : cluster 0 pgmap v443: 148 pgs: 143 active+clean, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped; 758 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 792 B/s rd, 0 B/s wr, 294 op/s; 8642/36945 objects degraded (23.392%); 8.8 MiB/s, 216 objects/s recovering 2024-03-11T21:01:55.028 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:01:55.062 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:01:55.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:55.065 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:01:55.166 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:01:55.206 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:01:55.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:55.210 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:01:55.311 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:01:55.345 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:01:55.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:55.349 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:01:55.449 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:55.475 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:55.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:55.477 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:01:55.578 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:01:55.609 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:01:55.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:55.613 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:01:55.714 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:55.722 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:01:55 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:01:55.736 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:55.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:55.743 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:01:55.844 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:55.858 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:55 smithi023 bash[21372]: cluster 2024-03-11T21:01:54.533139+0000 osd.4 (osd.4) 13 : cluster 0 6.e scrub starts 2024-03-11T21:01:55.858 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:55 smithi023 bash[21372]: cluster 2024-03-11T21:01:54.542606+0000 osd.4 (osd.4) 14 : cluster 0 6.e scrub ok 2024-03-11T21:01:55.858 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:55 smithi023 bash[21372]: cluster 2024-03-11T21:01:54.548779+0000 osd.7 (osd.7) 219 : cluster 0 3.11 scrub starts 2024-03-11T21:01:55.858 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:55 smithi023 bash[21372]: cluster 2024-03-11T21:01:54.550720+0000 osd.7 (osd.7) 220 : cluster 0 3.11 scrub ok 2024-03-11T21:01:55.858 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:55 smithi023 bash[21372]: cluster 2024-03-11T21:01:55.164937+0000 mon.a (mon.0) 1370 : cluster 3 Health check update: Degraded data redundancy: 8642/36945 objects degraded (23.392%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-03-11T21:01:55.858 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:55 smithi023 bash[21372]: audit 2024-03-11T21:01:55.358912+0000 mgr.y (mgr.24380) 414 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:01:55.858 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:55 smithi023 bash[29146]: cluster 2024-03-11T21:01:54.533139+0000 osd.4 (osd.4) 13 : cluster 0 6.e scrub starts 2024-03-11T21:01:55.858 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:55 smithi023 bash[29146]: cluster 2024-03-11T21:01:54.542606+0000 osd.4 (osd.4) 14 : cluster 0 6.e scrub ok 2024-03-11T21:01:55.859 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:55 smithi023 bash[29146]: cluster 2024-03-11T21:01:54.548779+0000 osd.7 (osd.7) 219 : cluster 0 3.11 scrub starts 2024-03-11T21:01:55.859 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:55 smithi023 bash[29146]: cluster 2024-03-11T21:01:54.550720+0000 osd.7 (osd.7) 220 : cluster 0 3.11 scrub ok 2024-03-11T21:01:55.859 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:55 smithi023 bash[29146]: cluster 2024-03-11T21:01:55.164937+0000 mon.a (mon.0) 1370 : cluster 3 Health check update: Degraded data redundancy: 8642/36945 objects degraded (23.392%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-03-11T21:01:55.859 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:55 smithi023 bash[29146]: audit 2024-03-11T21:01:55.358912+0000 mgr.y (mgr.24380) 414 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:01:55.879 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:55.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:55.881 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:01:55.982 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:01:55.990 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:55 smithi146 bash[23809]: cluster 2024-03-11T21:01:54.533139+0000 osd.4 (osd.4) 13 : cluster 0 6.e scrub starts 2024-03-11T21:01:55.990 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:55 smithi146 bash[23809]: cluster 2024-03-11T21:01:54.542606+0000 osd.4 (osd.4) 14 : cluster 0 6.e scrub ok 2024-03-11T21:01:55.990 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:55 smithi146 bash[23809]: cluster 2024-03-11T21:01:54.548779+0000 osd.7 (osd.7) 219 : cluster 0 3.11 scrub starts 2024-03-11T21:01:55.991 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:55 smithi146 bash[23809]: cluster 2024-03-11T21:01:54.550720+0000 osd.7 (osd.7) 220 : cluster 0 3.11 scrub ok 2024-03-11T21:01:55.991 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:55 smithi146 bash[23809]: cluster 2024-03-11T21:01:55.164937+0000 mon.a (mon.0) 1370 : cluster 3 Health check update: Degraded data redundancy: 8642/36945 objects degraded (23.392%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-03-11T21:01:55.991 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:55 smithi146 bash[23809]: audit 2024-03-11T21:01:55.358912+0000 mgr.y (mgr.24380) 414 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:01:56.001 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:01:56.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:56.003 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:01:56.104 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:01:56.125 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:01:56.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:56.127 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:01:56.228 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:01:56.257 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:01:56.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:56.261 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:01:56.361 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:56.379 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:56.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:56.381 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:01:56.482 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:01:56.512 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:01:56.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:56.515 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:01:56.616 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:56.637 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:56.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:56.639 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:01:56.739 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:56.766 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:56.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:56.769 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:01:56.870 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:56.877 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:56 smithi146 bash[23809]: cluster 2024-03-11T21:01:55.564085+0000 osd.4 (osd.4) 15 : cluster 0 4.9 scrub starts 2024-03-11T21:01:56.877 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:56 smithi146 bash[23809]: cluster 2024-03-11T21:01:55.567055+0000 osd.4 (osd.4) 16 : cluster 0 4.9 scrub ok 2024-03-11T21:01:56.877 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:56 smithi146 bash[23809]: cluster 2024-03-11T21:01:56.117822+0000 mgr.y (mgr.24380) 415 : cluster 0 pgmap v444: 148 pgs: 143 active+clean, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped; 528 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 697 op/s; 8150/25923 objects degraded (31.439%); 9.1 MiB/s, 235 objects/s recovering 2024-03-11T21:01:56.881 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:56 smithi023 bash[21372]: cluster 2024-03-11T21:01:55.564085+0000 osd.4 (osd.4) 15 : cluster 0 4.9 scrub starts 2024-03-11T21:01:56.881 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:56 smithi023 bash[21372]: cluster 2024-03-11T21:01:55.567055+0000 osd.4 (osd.4) 16 : cluster 0 4.9 scrub ok 2024-03-11T21:01:56.881 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:56 smithi023 bash[21372]: cluster 2024-03-11T21:01:56.117822+0000 mgr.y (mgr.24380) 415 : cluster 0 pgmap v444: 148 pgs: 143 active+clean, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped; 528 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 697 op/s; 8150/25923 objects degraded (31.439%); 9.1 MiB/s, 235 objects/s recovering 2024-03-11T21:01:56.881 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:56 smithi023 bash[29146]: cluster 2024-03-11T21:01:55.564085+0000 osd.4 (osd.4) 15 : cluster 0 4.9 scrub starts 2024-03-11T21:01:56.881 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:56 smithi023 bash[29146]: cluster 2024-03-11T21:01:55.567055+0000 osd.4 (osd.4) 16 : cluster 0 4.9 scrub ok 2024-03-11T21:01:56.881 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:56 smithi023 bash[29146]: cluster 2024-03-11T21:01:56.117822+0000 mgr.y (mgr.24380) 415 : cluster 0 pgmap v444: 148 pgs: 143 active+clean, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped; 528 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 697 op/s; 8150/25923 objects degraded (31.439%); 9.1 MiB/s, 235 objects/s recovering 2024-03-11T21:01:56.887 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:56.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:56.888 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:01:56.989 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:57.007 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:57.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:57.009 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:01:57.109 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:01:57.134 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:01:57.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:57.136 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:01:57.236 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:01:57.270 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:01:57.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:57.276 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:01:57.377 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:01:57.405 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:01:57.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:57.408 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:01:57.509 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:01:57.540 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:01:57.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:57.544 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:01:57.645 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:01:57.668 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:01:57.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:57.670 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:01:57.771 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:01:57.793 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:01:57.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:57.799 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:01:57.899 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:57.922 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:57.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:57.924 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:01:58.013 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:57 smithi146 bash[23809]: cluster 2024-03-11T21:01:56.561151+0000 osd.4 (osd.4) 17 : cluster 0 4.18 scrub starts 2024-03-11T21:01:58.013 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:57 smithi146 bash[23809]: cluster 2024-03-11T21:01:56.564489+0000 osd.4 (osd.4) 18 : cluster 0 4.18 scrub ok 2024-03-11T21:01:58.013 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:57 smithi146 bash[23809]: cluster 2024-03-11T21:01:56.629631+0000 osd.7 (osd.7) 221 : cluster 0 3.15 scrub starts 2024-03-11T21:01:58.013 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:57 smithi146 bash[23809]: cluster 2024-03-11T21:01:56.635849+0000 osd.7 (osd.7) 222 : cluster 0 3.15 scrub ok 2024-03-11T21:01:58.025 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:58.059 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:58.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:58.065 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:01:58.166 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:01:58.201 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:01:58.205 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:57 smithi023 bash[21372]: cluster 2024-03-11T21:01:56.561151+0000 osd.4 (osd.4) 17 : cluster 0 4.18 scrub starts 2024-03-11T21:01:58.205 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:57 smithi023 bash[21372]: cluster 2024-03-11T21:01:56.564489+0000 osd.4 (osd.4) 18 : cluster 0 4.18 scrub ok 2024-03-11T21:01:58.205 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:57 smithi023 bash[21372]: cluster 2024-03-11T21:01:56.629631+0000 osd.7 (osd.7) 221 : cluster 0 3.15 scrub starts 2024-03-11T21:01:58.205 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:57 smithi023 bash[21372]: cluster 2024-03-11T21:01:56.635849+0000 osd.7 (osd.7) 222 : cluster 0 3.15 scrub ok 2024-03-11T21:01:58.205 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:57 smithi023 bash[29146]: cluster 2024-03-11T21:01:56.561151+0000 osd.4 (osd.4) 17 : cluster 0 4.18 scrub starts 2024-03-11T21:01:58.206 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:57 smithi023 bash[29146]: cluster 2024-03-11T21:01:56.564489+0000 osd.4 (osd.4) 18 : cluster 0 4.18 scrub ok 2024-03-11T21:01:58.206 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:57 smithi023 bash[29146]: cluster 2024-03-11T21:01:56.629631+0000 osd.7 (osd.7) 221 : cluster 0 3.15 scrub starts 2024-03-11T21:01:58.206 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:57 smithi023 bash[29146]: cluster 2024-03-11T21:01:56.635849+0000 osd.7 (osd.7) 222 : cluster 0 3.15 scrub ok 2024-03-11T21:01:58.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:58.207 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:01:58.308 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:58.338 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:58.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:58.341 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:01:58.442 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:58.471 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:58.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:58.473 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:01:58.535 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:01:58 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:01:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:01:58.574 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:01:58.599 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:01:58.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:58.603 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:01:58.704 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:01:58.726 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:01:58.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:58.728 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:01:58.828 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:01:58.852 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:01:58.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:58.853 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:01:58.954 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:58.984 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:58.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:58.990 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:01:59.013 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:58 smithi146 bash[23809]: cluster 2024-03-11T21:01:57.605976+0000 osd.7 (osd.7) 223 : cluster 0 2.b scrub starts 2024-03-11T21:01:59.013 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:58 smithi146 bash[23809]: cluster 2024-03-11T21:01:57.613237+0000 osd.7 (osd.7) 224 : cluster 0 2.b scrub ok 2024-03-11T21:01:59.013 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:58 smithi146 bash[23809]: cluster 2024-03-11T21:01:58.118582+0000 mgr.y (mgr.24380) 416 : cluster 0 pgmap v445: 148 pgs: 143 active+clean, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped; 514 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 615 op/s; 8150/25254 objects degraded (32.272%); 7.7 MiB/s, 200 objects/s recovering 2024-03-11T21:01:59.091 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:59.118 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:01:59.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:59.120 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:01:59.215 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:58 smithi023 bash[21372]: cluster 2024-03-11T21:01:57.605976+0000 osd.7 (osd.7) 223 : cluster 0 2.b scrub starts 2024-03-11T21:01:59.215 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:58 smithi023 bash[21372]: cluster 2024-03-11T21:01:57.613237+0000 osd.7 (osd.7) 224 : cluster 0 2.b scrub ok 2024-03-11T21:01:59.215 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:58 smithi023 bash[21372]: cluster 2024-03-11T21:01:58.118582+0000 mgr.y (mgr.24380) 416 : cluster 0 pgmap v445: 148 pgs: 143 active+clean, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped; 514 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 615 op/s; 8150/25254 objects degraded (32.272%); 7.7 MiB/s, 200 objects/s recovering 2024-03-11T21:01:59.216 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:58 smithi023 bash[29146]: cluster 2024-03-11T21:01:57.605976+0000 osd.7 (osd.7) 223 : cluster 0 2.b scrub starts 2024-03-11T21:01:59.216 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:58 smithi023 bash[29146]: cluster 2024-03-11T21:01:57.613237+0000 osd.7 (osd.7) 224 : cluster 0 2.b scrub ok 2024-03-11T21:01:59.216 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:58 smithi023 bash[29146]: cluster 2024-03-11T21:01:58.118582+0000 mgr.y (mgr.24380) 416 : cluster 0 pgmap v445: 148 pgs: 143 active+clean, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped; 514 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 615 op/s; 8150/25254 objects degraded (32.272%); 7.7 MiB/s, 200 objects/s recovering 2024-03-11T21:01:59.221 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:59.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:59.252 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:59.252 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:01:59.353 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:01:59.383 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:01:59.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:59.386 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:01:59.487 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:01:59.516 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:01:59.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:59.520 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:01:59.621 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:01:59.647 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:01:59.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:59.649 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:01:59.749 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:59.775 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:01:59.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:59.777 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:01:59.877 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:59.917 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:01:59.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:01:59.921 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:02:00.013 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:59 smithi146 bash[23809]: cluster 2024-03-11T21:01:58.551188+0000 osd.4 (osd.4) 19 : cluster 0 3.1a scrub starts 2024-03-11T21:02:00.013 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:01:59 smithi146 bash[23809]: cluster 2024-03-11T21:01:58.553985+0000 osd.4 (osd.4) 20 : cluster 0 3.1a scrub ok 2024-03-11T21:02:00.021 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:02:00.051 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:02:00.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:00.053 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:02:00.097 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:59 smithi023 bash[21372]: cluster 2024-03-11T21:01:58.551188+0000 osd.4 (osd.4) 19 : cluster 0 3.1a scrub starts 2024-03-11T21:02:00.097 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:01:59 smithi023 bash[21372]: cluster 2024-03-11T21:01:58.553985+0000 osd.4 (osd.4) 20 : cluster 0 3.1a scrub ok 2024-03-11T21:02:00.097 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:59 smithi023 bash[29146]: cluster 2024-03-11T21:01:58.551188+0000 osd.4 (osd.4) 19 : cluster 0 3.1a scrub starts 2024-03-11T21:02:00.097 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:01:59 smithi023 bash[29146]: cluster 2024-03-11T21:01:58.553985+0000 osd.4 (osd.4) 20 : cluster 0 3.1a scrub ok 2024-03-11T21:02:00.154 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:02:00.175 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:02:00.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:00.178 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:02:00.279 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:00.308 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:00.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:00.313 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:02:00.414 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:00.440 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:00.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:00.442 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:02:00.543 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:02:00.566 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:02:00.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:00.570 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:02:00.671 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:00.700 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:00.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:00.705 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:02:00.806 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:02:00.829 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:02:00.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:00.831 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:02:00.932 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:02:00.942 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:00 smithi146 bash[23809]: cluster 2024-03-11T21:02:00.119301+0000 mgr.y (mgr.24380) 417 : cluster 0 pgmap v446: 148 pgs: 143 active+clean, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped; 414 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 981 B/s rd, 0 B/s wr, 710 op/s; 7897/20436 objects degraded (38.643%); 7.0 MiB/s, 205 objects/s recovering 2024-03-11T21:02:00.942 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:00 smithi146 bash[23809]: cluster 2024-03-11T21:02:00.166348+0000 mon.a (mon.0) 1371 : cluster 3 Health check update: Degraded data redundancy: 8150/25254 objects degraded (32.272%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-03-11T21:02:00.967 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:02:00.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:00.972 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:02:01.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:00 smithi023 bash[21372]: cluster 2024-03-11T21:02:00.119301+0000 mgr.y (mgr.24380) 417 : cluster 0 pgmap v446: 148 pgs: 143 active+clean, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped; 414 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 981 B/s rd, 0 B/s wr, 710 op/s; 7897/20436 objects degraded (38.643%); 7.0 MiB/s, 205 objects/s recovering 2024-03-11T21:02:01.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:00 smithi023 bash[21372]: cluster 2024-03-11T21:02:00.166348+0000 mon.a (mon.0) 1371 : cluster 3 Health check update: Degraded data redundancy: 8150/25254 objects degraded (32.272%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-03-11T21:02:01.036 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:00 smithi023 bash[29146]: cluster 2024-03-11T21:02:00.119301+0000 mgr.y (mgr.24380) 417 : cluster 0 pgmap v446: 148 pgs: 143 active+clean, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped; 414 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 981 B/s rd, 0 B/s wr, 710 op/s; 7897/20436 objects degraded (38.643%); 7.0 MiB/s, 205 objects/s recovering 2024-03-11T21:02:01.036 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:00 smithi023 bash[29146]: cluster 2024-03-11T21:02:00.166348+0000 mon.a (mon.0) 1371 : cluster 3 Health check update: Degraded data redundancy: 8150/25254 objects degraded (32.272%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-03-11T21:02:01.073 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:02:01.096 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:02:01.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:01.101 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:02:01.202 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:02:01.224 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:02:01.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:01.227 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:02:01.328 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:02:01.355 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:02:01.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:01.358 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:02:01.459 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:01.480 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:01.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:01.485 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:02:01.586 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:01.612 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:01.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:01.615 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:02:01.716 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:01.735 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:01.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:01.736 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:02:01.837 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:01.866 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:01.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:01.873 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:02:01.974 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:02:01.993 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:02:01.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:01.995 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:02:02.095 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:02.126 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:02.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:02.129 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:02:02.230 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:02.240 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:01 smithi023 bash[21372]: cluster 2024-03-11T21:02:00.507647+0000 osd.4 (osd.4) 21 : cluster 0 3.14 scrub starts 2024-03-11T21:02:02.240 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:01 smithi023 bash[21372]: cluster 2024-03-11T21:02:00.509751+0000 osd.4 (osd.4) 22 : cluster 0 3.14 scrub ok 2024-03-11T21:02:02.241 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:01 smithi023 bash[29146]: cluster 2024-03-11T21:02:00.507647+0000 osd.4 (osd.4) 21 : cluster 0 3.14 scrub starts 2024-03-11T21:02:02.241 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:01 smithi023 bash[29146]: cluster 2024-03-11T21:02:00.509751+0000 osd.4 (osd.4) 22 : cluster 0 3.14 scrub ok 2024-03-11T21:02:02.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:01 smithi146 bash[23809]: cluster 2024-03-11T21:02:00.507647+0000 osd.4 (osd.4) 21 : cluster 0 3.14 scrub starts 2024-03-11T21:02:02.263 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:01 smithi146 bash[23809]: cluster 2024-03-11T21:02:00.509751+0000 osd.4 (osd.4) 22 : cluster 0 3.14 scrub ok 2024-03-11T21:02:02.275 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:02.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:02.280 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:02:02.380 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:02.406 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:02.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:02.412 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:02:02.512 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:02:02.544 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:02:02.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:02.549 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:02:02.650 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:02.681 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:02.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:02.683 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:02:02.785 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:02.835 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:02.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:02.841 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:02:02.942 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:02:02.964 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:02:02.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:02.969 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:02:03.007 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:02 smithi023 bash[21372]: cluster 2024-03-11T21:02:01.522517+0000 osd.7 (osd.7) 225 : cluster 0 6.12 scrub starts 2024-03-11T21:02:03.007 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:02 smithi023 bash[21372]: cluster 2024-03-11T21:02:01.526354+0000 osd.7 (osd.7) 226 : cluster 0 6.12 scrub ok 2024-03-11T21:02:03.007 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:02 smithi023 bash[21372]: cluster 2024-03-11T21:02:02.120171+0000 mgr.y (mgr.24380) 418 : cluster 0 pgmap v447: 148 pgs: 143 active+clean, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped; 289 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 720 op/s; 7640/14442 objects degraded (52.901%); 6.0 MiB/s, 199 objects/s recovering 2024-03-11T21:02:03.007 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:02 smithi023 bash[29146]: cluster 2024-03-11T21:02:01.522517+0000 osd.7 (osd.7) 225 : cluster 0 6.12 scrub starts 2024-03-11T21:02:03.007 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:02 smithi023 bash[29146]: cluster 2024-03-11T21:02:01.526354+0000 osd.7 (osd.7) 226 : cluster 0 6.12 scrub ok 2024-03-11T21:02:03.007 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:02 smithi023 bash[29146]: cluster 2024-03-11T21:02:02.120171+0000 mgr.y (mgr.24380) 418 : cluster 0 pgmap v447: 148 pgs: 143 active+clean, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped; 289 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 720 op/s; 7640/14442 objects degraded (52.901%); 6.0 MiB/s, 199 objects/s recovering 2024-03-11T21:02:03.070 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:03.093 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:03.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:03.095 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:02:03.196 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:03.227 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:03.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:03.233 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:02:03.252 INFO:teuthology.orchestra.run.smithi023.stdout:ERROR: (22) Invalid argument 2024-03-11T21:02:03.252 INFO:teuthology.orchestra.run.smithi023.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:02:03.263 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:02 smithi146 bash[23809]: cluster 2024-03-11T21:02:01.522517+0000 osd.7 (osd.7) 225 : cluster 0 6.12 scrub starts 2024-03-11T21:02:03.263 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:02 smithi146 bash[23809]: cluster 2024-03-11T21:02:01.526354+0000 osd.7 (osd.7) 226 : cluster 0 6.12 scrub ok 2024-03-11T21:02:03.263 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:02 smithi146 bash[23809]: cluster 2024-03-11T21:02:02.120171+0000 mgr.y (mgr.24380) 418 : cluster 0 pgmap v447: 148 pgs: 143 active+clean, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped; 289 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 720 op/s; 7640/14442 objects degraded (52.901%); 6.0 MiB/s, 199 objects/s recovering 2024-03-11T21:02:03.334 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:02:03.365 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:02:03.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:03.366 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:02:03.467 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:03.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:03.496 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:03.496 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:02:03.597 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:03.635 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:03.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:03.640 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:02:03.741 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:03.779 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:03.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:03.785 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:02:03.886 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:02:03.916 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:02:03.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:03.920 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:02:04.021 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:04.032 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:03 smithi023 bash[29146]: cluster 2024-03-11T21:02:02.499625+0000 osd.7 (osd.7) 227 : cluster 0 3.f scrub starts 2024-03-11T21:02:04.032 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:03 smithi023 bash[29146]: cluster 2024-03-11T21:02:02.505542+0000 osd.7 (osd.7) 228 : cluster 0 3.f scrub ok 2024-03-11T21:02:04.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:03 smithi023 bash[21372]: cluster 2024-03-11T21:02:02.499625+0000 osd.7 (osd.7) 227 : cluster 0 3.f scrub starts 2024-03-11T21:02:04.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:03 smithi023 bash[21372]: cluster 2024-03-11T21:02:02.505542+0000 osd.7 (osd.7) 228 : cluster 0 3.f scrub ok 2024-03-11T21:02:04.054 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:04.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:04.057 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:02:04.158 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:02:04.176 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:02:04.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:04.178 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:02:04.263 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:03 smithi146 bash[23809]: cluster 2024-03-11T21:02:02.499625+0000 osd.7 (osd.7) 227 : cluster 0 3.f scrub starts 2024-03-11T21:02:04.263 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:03 smithi146 bash[23809]: cluster 2024-03-11T21:02:02.505542+0000 osd.7 (osd.7) 228 : cluster 0 3.f scrub ok 2024-03-11T21:02:04.278 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:04.308 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:04.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:04.310 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:02:04.411 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:02:04.439 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:02:04.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:04.444 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:02:04.544 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:02:04.581 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:02:04.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:04.587 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:02:04.687 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:04.722 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:04.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:04.728 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:02:04.828 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:02:04.860 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:02:04.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:04.864 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:02:04.964 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:04.995 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:04.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:04.998 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:02:05.099 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:02:05.108 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:04 smithi023 bash[21372]: cluster 2024-03-11T21:02:03.448352+0000 osd.4 (osd.4) 23 : cluster 0 6.17 scrub starts 2024-03-11T21:02:05.108 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:04 smithi023 bash[21372]: cluster 2024-03-11T21:02:03.517218+0000 osd.4 (osd.4) 24 : cluster 0 6.17 scrub ok 2024-03-11T21:02:05.108 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:04 smithi023 bash[21372]: cluster 2024-03-11T21:02:04.120639+0000 mgr.y (mgr.24380) 419 : cluster 0 pgmap v448: 148 pgs: 143 active+clean, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped; 289 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 625 op/s; 7640/14442 objects degraded (52.901%); 1.3 MiB/s, 83 objects/s recovering 2024-03-11T21:02:05.108 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:04 smithi023 bash[29146]: cluster 2024-03-11T21:02:03.448352+0000 osd.4 (osd.4) 23 : cluster 0 6.17 scrub starts 2024-03-11T21:02:05.108 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:04 smithi023 bash[29146]: cluster 2024-03-11T21:02:03.517218+0000 osd.4 (osd.4) 24 : cluster 0 6.17 scrub ok 2024-03-11T21:02:05.108 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:04 smithi023 bash[29146]: cluster 2024-03-11T21:02:04.120639+0000 mgr.y (mgr.24380) 419 : cluster 0 pgmap v448: 148 pgs: 143 active+clean, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped; 289 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 625 op/s; 7640/14442 objects degraded (52.901%); 1.3 MiB/s, 83 objects/s recovering 2024-03-11T21:02:05.124 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:02:05.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:05.128 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:02:05.229 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:02:05.238 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:04 smithi146 bash[23809]: cluster 2024-03-11T21:02:03.448352+0000 osd.4 (osd.4) 23 : cluster 0 6.17 scrub starts 2024-03-11T21:02:05.238 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:04 smithi146 bash[23809]: cluster 2024-03-11T21:02:03.517218+0000 osd.4 (osd.4) 24 : cluster 0 6.17 scrub ok 2024-03-11T21:02:05.238 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:04 smithi146 bash[23809]: cluster 2024-03-11T21:02:04.120639+0000 mgr.y (mgr.24380) 419 : cluster 0 pgmap v448: 148 pgs: 143 active+clean, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped; 289 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 625 op/s; 7640/14442 objects degraded (52.901%); 1.3 MiB/s, 83 objects/s recovering 2024-03-11T21:02:05.250 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:02:05.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:05.255 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:02:05.356 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:02:05.384 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:02:05.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:05.387 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:02:05.488 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:02:05.496 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:02:05 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:02:05.509 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:02:05.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:05.513 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:02:05.614 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:02:05.645 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:02:05.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:05.648 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:02:05.749 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:05.778 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:05.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:05.782 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:02:05.883 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:05.884 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:Removed 50082 objects 2024-03-11T21:02:05.884 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:Clean up completed and total clean up time :64.4599 2024-03-11T21:02:05.904 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:05.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:05.906 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:02:06.007 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:02:06.015 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:05 smithi023 bash[21372]: cluster 2024-03-11T21:02:04.515616+0000 osd.7 (osd.7) 229 : cluster 0 2.5 scrub starts 2024-03-11T21:02:06.015 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:05 smithi023 bash[21372]: cluster 2024-03-11T21:02:04.522739+0000 osd.7 (osd.7) 230 : cluster 0 2.5 scrub ok 2024-03-11T21:02:06.015 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:05 smithi023 bash[21372]: audit 2024-03-11T21:02:05.006708+0000 mon.a (mon.0) 1372 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:02:06.015 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:05 smithi023 bash[21372]: cluster 2024-03-11T21:02:05.167333+0000 mon.a (mon.0) 1373 : cluster 3 Health check update: Degraded data redundancy: 7640/14442 objects degraded (52.901%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-03-11T21:02:06.015 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:05 smithi023 bash[21372]: audit 2024-03-11T21:02:05.367110+0000 mgr.y (mgr.24380) 420 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:02:06.016 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:05 smithi023 bash[29146]: cluster 2024-03-11T21:02:04.515616+0000 osd.7 (osd.7) 229 : cluster 0 2.5 scrub starts 2024-03-11T21:02:06.016 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:05 smithi023 bash[29146]: cluster 2024-03-11T21:02:04.522739+0000 osd.7 (osd.7) 230 : cluster 0 2.5 scrub ok 2024-03-11T21:02:06.016 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:05 smithi023 bash[29146]: audit 2024-03-11T21:02:05.006708+0000 mon.a (mon.0) 1372 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:02:06.016 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:05 smithi023 bash[29146]: cluster 2024-03-11T21:02:05.167333+0000 mon.a (mon.0) 1373 : cluster 3 Health check update: Degraded data redundancy: 7640/14442 objects degraded (52.901%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-03-11T21:02:06.016 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:05 smithi023 bash[29146]: audit 2024-03-11T21:02:05.367110+0000 mgr.y (mgr.24380) 420 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:02:06.027 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:02:06.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:06.030 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:02:06.131 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:06.150 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:06.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:06.153 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:02:06.254 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:06.263 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:05 smithi146 bash[23809]: cluster 2024-03-11T21:02:04.515616+0000 osd.7 (osd.7) 229 : cluster 0 2.5 scrub starts 2024-03-11T21:02:06.263 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:05 smithi146 bash[23809]: cluster 2024-03-11T21:02:04.522739+0000 osd.7 (osd.7) 230 : cluster 0 2.5 scrub ok 2024-03-11T21:02:06.263 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:05 smithi146 bash[23809]: audit 2024-03-11T21:02:05.006708+0000 mon.a (mon.0) 1372 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:02:06.263 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:05 smithi146 bash[23809]: cluster 2024-03-11T21:02:05.167333+0000 mon.a (mon.0) 1373 : cluster 3 Health check update: Degraded data redundancy: 7640/14442 objects degraded (52.901%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-03-11T21:02:06.263 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:05 smithi146 bash[23809]: audit 2024-03-11T21:02:05.367110+0000 mgr.y (mgr.24380) 420 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:02:06.281 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:06.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:06.283 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:02:06.384 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:06.406 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:06.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:06.408 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:02:06.508 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:02:06.535 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:02:06.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:06.537 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:02:06.638 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:06.657 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:06.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:06.659 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:02:06.760 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:06.780 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:06.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:06.782 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:02:06.883 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:06.902 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:06.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:06.904 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:02:07.005 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:02:07.030 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:02:07.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:07.032 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:02:07.133 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:07.158 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:07.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:07.161 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:02:07.184 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-03-11T21:02:07.262 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:07.263 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:06 smithi146 bash[23809]: cluster 2024-03-11T21:02:05.526742+0000 osd.7 (osd.7) 231 : cluster 0 3.13 scrub starts 2024-03-11T21:02:07.263 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:06 smithi146 bash[23809]: cluster 2024-03-11T21:02:05.530622+0000 osd.7 (osd.7) 232 : cluster 0 3.13 scrub ok 2024-03-11T21:02:07.263 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:06 smithi146 bash[23809]: cluster 2024-03-11T21:02:06.121414+0000 mgr.y (mgr.24380) 421 : cluster 0 pgmap v449: 148 pgs: 143 active+clean, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped; 34 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 965 op/s; 7084/2220 objects degraded (319.099%); 1.3 MiB/s, 129 objects/s recovering 2024-03-11T21:02:07.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:07 smithi023 bash[21372]: cluster 2024-03-11T21:02:05.526742+0000 osd.7 (osd.7) 231 : cluster 0 3.13 scrub starts 2024-03-11T21:02:07.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:07 smithi023 bash[21372]: cluster 2024-03-11T21:02:05.530622+0000 osd.7 (osd.7) 232 : cluster 0 3.13 scrub ok 2024-03-11T21:02:07.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:07 smithi023 bash[21372]: cluster 2024-03-11T21:02:06.121414+0000 mgr.y (mgr.24380) 421 : cluster 0 pgmap v449: 148 pgs: 143 active+clean, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped; 34 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 965 op/s; 7084/2220 objects degraded (319.099%); 1.3 MiB/s, 129 objects/s recovering 2024-03-11T21:02:07.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:07 smithi023 bash[29146]: cluster 2024-03-11T21:02:05.526742+0000 osd.7 (osd.7) 231 : cluster 0 3.13 scrub starts 2024-03-11T21:02:07.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:07 smithi023 bash[29146]: cluster 2024-03-11T21:02:05.530622+0000 osd.7 (osd.7) 232 : cluster 0 3.13 scrub ok 2024-03-11T21:02:07.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:07 smithi023 bash[29146]: cluster 2024-03-11T21:02:06.121414+0000 mgr.y (mgr.24380) 421 : cluster 0 pgmap v449: 148 pgs: 143 active+clean, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped; 34 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 965 op/s; 7084/2220 objects degraded (319.099%); 1.3 MiB/s, 129 objects/s recovering 2024-03-11T21:02:07.295 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:07.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:07.297 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:02:07.398 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:02:07.424 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:02:07.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:07.426 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:02:07.527 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:07.552 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:07.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:07.554 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:02:07.655 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:02:07.683 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:02:07.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:07.685 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:02:07.786 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:02:07.814 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:02:07.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:07.816 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:02:07.917 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:07.937 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:07.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:07.939 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:02:08.040 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:02:08.059 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:02:08.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:08.061 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:02:08.162 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:08.189 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:08.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:08.191 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:02:08.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:08 smithi023 bash[21372]: cluster 2024-03-11T21:02:07.522400+0000 osd.4 (osd.4) 25 : cluster 0 4.13 scrub starts 2024-03-11T21:02:08.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:08 smithi023 bash[21372]: cluster 2024-03-11T21:02:07.523939+0000 osd.4 (osd.4) 26 : cluster 0 4.13 scrub ok 2024-03-11T21:02:08.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:08 smithi023 bash[29146]: cluster 2024-03-11T21:02:07.522400+0000 osd.4 (osd.4) 25 : cluster 0 4.13 scrub starts 2024-03-11T21:02:08.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:08 smithi023 bash[29146]: cluster 2024-03-11T21:02:07.523939+0000 osd.4 (osd.4) 26 : cluster 0 4.13 scrub ok 2024-03-11T21:02:08.292 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:02:08.319 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:02:08.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:08.322 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:02:08.422 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:08.433 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:02:08 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:02:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:02:08.435 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:08 smithi146 bash[23809]: cluster 2024-03-11T21:02:07.522400+0000 osd.4 (osd.4) 25 : cluster 0 4.13 scrub starts 2024-03-11T21:02:08.435 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:08 smithi146 bash[23809]: cluster 2024-03-11T21:02:07.523939+0000 osd.4 (osd.4) 26 : cluster 0 4.13 scrub ok 2024-03-11T21:02:08.445 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:08.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:08.447 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:02:08.548 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:08.568 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:08.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:08.570 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:02:08.671 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:08.692 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:08.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:08.694 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:02:08.795 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:02:08.817 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:02:08.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:08.819 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:02:08.920 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:08.942 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:08.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:08.945 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:02:09.046 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:09.070 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:09.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:09.072 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:02:09.101 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_1 2024-03-11T21:02:09.101 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd pool rm unique_pool_1 unique_pool_1 --yes-i-really-really-mean-it 2024-03-11T21:02:09.172 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:02:09.199 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:02:09.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:09.200 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:02:09.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:09 smithi023 bash[21372]: cluster 2024-03-11T21:02:08.122024+0000 mgr.y (mgr.24380) 422 : cluster 0 pgmap v450: 148 pgs: 143 active+clean, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped; 21 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 677 op/s; 7084/1572 objects degraded (450.636%); 0 B/s, 88 objects/s recovering 2024-03-11T21:02:09.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:09 smithi023 bash[29146]: cluster 2024-03-11T21:02:08.122024+0000 mgr.y (mgr.24380) 422 : cluster 0 pgmap v450: 148 pgs: 143 active+clean, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped; 21 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 677 op/s; 7084/1572 objects degraded (450.636%); 0 B/s, 88 objects/s recovering 2024-03-11T21:02:09.301 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:02:09.327 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:02:09.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:09.329 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:02:09.429 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:02:09.441 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:09 smithi146 bash[23809]: cluster 2024-03-11T21:02:08.122024+0000 mgr.y (mgr.24380) 422 : cluster 0 pgmap v450: 148 pgs: 143 active+clean, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped; 21 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 677 op/s; 7084/1572 objects degraded (450.636%); 0 B/s, 88 objects/s recovering 2024-03-11T21:02:09.455 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:02:09.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:09.457 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:02:09.529 INFO:teuthology.orchestra.run.smithi023.stdout:osd.0: {} 2024-03-11T21:02:09.558 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:09.570 INFO:teuthology.orchestra.run.smithi023.stdout:osd.1: {} 2024-03-11T21:02:09.586 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:09.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:09.590 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:02:09.609 INFO:teuthology.orchestra.run.smithi023.stdout:osd.2: {} 2024-03-11T21:02:09.641 INFO:teuthology.orchestra.run.smithi023.stdout:osd.3: {} 2024-03-11T21:02:09.678 INFO:teuthology.orchestra.run.smithi023.stdout:osd.4: {} 2024-03-11T21:02:09.691 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:09.707 INFO:teuthology.orchestra.run.smithi023.stdout:osd.5: {} 2024-03-11T21:02:09.711 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:09.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:09.714 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:02:09.734 INFO:teuthology.orchestra.run.smithi023.stdout:osd.6: {} 2024-03-11T21:02:09.762 INFO:teuthology.orchestra.run.smithi023.stdout:osd.7: {} 2024-03-11T21:02:09.764 INFO:teuthology.orchestra.run.smithi023.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:02:09.764 INFO:teuthology.orchestra.run.smithi023.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:02:09.764 INFO:teuthology.orchestra.run.smithi023.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:02:09.764 INFO:teuthology.orchestra.run.smithi023.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:02:09.764 INFO:teuthology.orchestra.run.smithi023.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:02:09.764 INFO:teuthology.orchestra.run.smithi023.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:02:09.764 INFO:teuthology.orchestra.run.smithi023.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:02:09.764 INFO:teuthology.orchestra.run.smithi023.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:02:09.815 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:02:09.839 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:02:09.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:09.841 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:02:09.941 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:02:09.965 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:02:09.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:09.967 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:02:10.068 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:10.089 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:10.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:10.091 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:02:10.126 INFO:teuthology.orchestra.run.smithi023.stderr:nodeep-scrub is unset 2024-03-11T21:02:10.140 INFO:teuthology.orchestra.run.smithi023.stderr:reweighted osd.2 to 1 (10000) 2024-03-11T21:02:10.192 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:10.213 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:10.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:10.217 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:02:10.317 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:10.344 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:10.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:10.346 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:02:10.447 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:02:10.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:10 smithi146 bash[23809]: audit 2024-03-11T21:02:09.335861+0000 mon.c (mon.2) 53 : audit 1 from='client.? 172.21.15.23:0/417883298' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-11T21:02:10.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:10 smithi146 bash[23809]: audit 2024-03-11T21:02:09.336315+0000 mon.a (mon.0) 1374 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-11T21:02:10.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:10 smithi146 bash[23809]: audit 2024-03-11T21:02:09.490650+0000 mon.a (mon.0) 1375 : audit 0 from='client.? 172.21.15.23:0/3676237438' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:02:10.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:10 smithi146 bash[23809]: audit 2024-03-11T21:02:09.948454+0000 mon.a (mon.0) 1376 : audit 1 from='client.? 172.21.15.23:0/1444879367' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:02:10.472 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:02:10.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:10.474 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:02:10.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:10 smithi023 bash[21372]: audit 2024-03-11T21:02:09.335861+0000 mon.c (mon.2) 53 : audit 1 from='client.? 172.21.15.23:0/417883298' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-11T21:02:10.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:10 smithi023 bash[21372]: audit 2024-03-11T21:02:09.336315+0000 mon.a (mon.0) 1374 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-11T21:02:10.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:10 smithi023 bash[21372]: audit 2024-03-11T21:02:09.490650+0000 mon.a (mon.0) 1375 : audit 0 from='client.? 172.21.15.23:0/3676237438' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:02:10.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:10 smithi023 bash[21372]: audit 2024-03-11T21:02:09.948454+0000 mon.a (mon.0) 1376 : audit 1 from='client.? 172.21.15.23:0/1444879367' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:02:10.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:10 smithi023 bash[29146]: audit 2024-03-11T21:02:09.335861+0000 mon.c (mon.2) 53 : audit 1 from='client.? 172.21.15.23:0/417883298' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-11T21:02:10.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:10 smithi023 bash[29146]: audit 2024-03-11T21:02:09.336315+0000 mon.a (mon.0) 1374 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-11T21:02:10.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:10 smithi023 bash[29146]: audit 2024-03-11T21:02:09.490650+0000 mon.a (mon.0) 1375 : audit 0 from='client.? 172.21.15.23:0/3676237438' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:02:10.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:10 smithi023 bash[29146]: audit 2024-03-11T21:02:09.948454+0000 mon.a (mon.0) 1376 : audit 1 from='client.? 172.21.15.23:0/1444879367' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:02:10.574 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:10.597 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:10.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:10.599 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:02:10.700 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:10.725 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:10.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:10.728 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:02:10.829 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:10.850 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:10.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:10.851 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:02:10.952 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:10.973 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:10.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:10.975 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:02:11.076 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:11.096 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:11.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:11.099 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:02:11.200 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:11.220 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:11.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:11.222 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:02:11.325 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:11.348 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:11.349 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:11 smithi023 bash[21372]: cluster 2024-03-11T21:02:09.660064+0000 osd.7 (osd.7) 233 : cluster 0 2.1d scrub starts 2024-03-11T21:02:11.349 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:11 smithi023 bash[21372]: cluster 2024-03-11T21:02:09.662420+0000 osd.7 (osd.7) 234 : cluster 0 2.1d scrub ok 2024-03-11T21:02:11.350 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:11 smithi023 bash[21372]: cluster 2024-03-11T21:02:10.123093+0000 mon.a (mon.0) 1377 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:02:11.350 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:11 smithi023 bash[21372]: cluster 2024-03-11T21:02:10.123340+0000 mgr.y (mgr.24380) 423 : cluster 0 pgmap v451: 148 pgs: 143 active+clean, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped; 1.1 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 684 op/s; 6446/618 objects degraded (1043.042%); 0 B/s, 141 objects/s recovering 2024-03-11T21:02:11.350 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:11 smithi023 bash[21372]: audit 2024-03-11T21:02:10.126003+0000 mon.a (mon.0) 1378 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-11T21:02:11.350 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:11 smithi023 bash[21372]: audit 2024-03-11T21:02:10.126206+0000 mon.a (mon.0) 1379 : audit 1 from='client.? 172.21.15.23:0/1444879367' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:02:11.350 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:11 smithi023 bash[21372]: cluster 2024-03-11T21:02:10.133419+0000 mon.a (mon.0) 1380 : cluster 0 osdmap e232: 8 total, 8 up, 8 in 2024-03-11T21:02:11.350 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:11 smithi023 bash[21372]: cluster 2024-03-11T21:02:10.168651+0000 mon.a (mon.0) 1381 : cluster 3 Health check update: Degraded data redundancy: 7084/1572 objects degraded (450.636%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-03-11T21:02:11.350 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:11 smithi023 bash[21372]: cluster 2024-03-11T21:02:10.174294+0000 mon.a (mon.0) 1382 : cluster 0 osdmap e233: 8 total, 8 up, 8 in 2024-03-11T21:02:11.350 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:11 smithi023 bash[21372]: cluster 2024-03-11T21:02:10.601759+0000 osd.4 (osd.4) 27 : cluster 0 3.9 scrub starts 2024-03-11T21:02:11.351 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:11 smithi023 bash[21372]: cluster 2024-03-11T21:02:10.604322+0000 osd.4 (osd.4) 28 : cluster 0 3.9 scrub ok 2024-03-11T21:02:11.351 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:11 smithi023 bash[29146]: cluster 2024-03-11T21:02:09.660064+0000 osd.7 (osd.7) 233 : cluster 0 2.1d scrub starts 2024-03-11T21:02:11.351 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:11 smithi023 bash[29146]: cluster 2024-03-11T21:02:09.662420+0000 osd.7 (osd.7) 234 : cluster 0 2.1d scrub ok 2024-03-11T21:02:11.351 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:11 smithi023 bash[29146]: cluster 2024-03-11T21:02:10.123093+0000 mon.a (mon.0) 1377 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:02:11.351 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:11 smithi023 bash[29146]: cluster 2024-03-11T21:02:10.123340+0000 mgr.y (mgr.24380) 423 : cluster 0 pgmap v451: 148 pgs: 143 active+clean, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped; 1.1 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 684 op/s; 6446/618 objects degraded (1043.042%); 0 B/s, 141 objects/s recovering 2024-03-11T21:02:11.351 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:11 smithi023 bash[29146]: audit 2024-03-11T21:02:10.126003+0000 mon.a (mon.0) 1378 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-11T21:02:11.351 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:11 smithi023 bash[29146]: audit 2024-03-11T21:02:10.126206+0000 mon.a (mon.0) 1379 : audit 1 from='client.? 172.21.15.23:0/1444879367' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:02:11.351 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:11 smithi023 bash[29146]: cluster 2024-03-11T21:02:10.133419+0000 mon.a (mon.0) 1380 : cluster 0 osdmap e232: 8 total, 8 up, 8 in 2024-03-11T21:02:11.351 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:11 smithi023 bash[29146]: cluster 2024-03-11T21:02:10.168651+0000 mon.a (mon.0) 1381 : cluster 3 Health check update: Degraded data redundancy: 7084/1572 objects degraded (450.636%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-03-11T21:02:11.351 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:11 smithi023 bash[29146]: cluster 2024-03-11T21:02:10.174294+0000 mon.a (mon.0) 1382 : cluster 0 osdmap e233: 8 total, 8 up, 8 in 2024-03-11T21:02:11.352 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:11 smithi023 bash[29146]: cluster 2024-03-11T21:02:10.601759+0000 osd.4 (osd.4) 27 : cluster 0 3.9 scrub starts 2024-03-11T21:02:11.352 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:11 smithi023 bash[29146]: cluster 2024-03-11T21:02:10.604322+0000 osd.4 (osd.4) 28 : cluster 0 3.9 scrub ok 2024-03-11T21:02:11.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:11.352 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:02:11.453 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:02:11.475 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:02:11.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:11.477 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:02:11.513 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:11 smithi146 bash[23809]: cluster 2024-03-11T21:02:09.660064+0000 osd.7 (osd.7) 233 : cluster 0 2.1d scrub starts 2024-03-11T21:02:11.514 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:11 smithi146 bash[23809]: cluster 2024-03-11T21:02:09.662420+0000 osd.7 (osd.7) 234 : cluster 0 2.1d scrub ok 2024-03-11T21:02:11.514 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:11 smithi146 bash[23809]: cluster 2024-03-11T21:02:10.123093+0000 mon.a (mon.0) 1377 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:02:11.514 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:11 smithi146 bash[23809]: cluster 2024-03-11T21:02:10.123340+0000 mgr.y (mgr.24380) 423 : cluster 0 pgmap v451: 148 pgs: 143 active+clean, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped; 1.1 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 684 op/s; 6446/618 objects degraded (1043.042%); 0 B/s, 141 objects/s recovering 2024-03-11T21:02:11.514 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:11 smithi146 bash[23809]: audit 2024-03-11T21:02:10.126003+0000 mon.a (mon.0) 1378 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-11T21:02:11.514 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:11 smithi146 bash[23809]: audit 2024-03-11T21:02:10.126206+0000 mon.a (mon.0) 1379 : audit 1 from='client.? 172.21.15.23:0/1444879367' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:02:11.514 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:11 smithi146 bash[23809]: cluster 2024-03-11T21:02:10.133419+0000 mon.a (mon.0) 1380 : cluster 0 osdmap e232: 8 total, 8 up, 8 in 2024-03-11T21:02:11.514 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:11 smithi146 bash[23809]: cluster 2024-03-11T21:02:10.168651+0000 mon.a (mon.0) 1381 : cluster 3 Health check update: Degraded data redundancy: 7084/1572 objects degraded (450.636%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-03-11T21:02:11.514 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:11 smithi146 bash[23809]: cluster 2024-03-11T21:02:10.174294+0000 mon.a (mon.0) 1382 : cluster 0 osdmap e233: 8 total, 8 up, 8 in 2024-03-11T21:02:11.514 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:11 smithi146 bash[23809]: cluster 2024-03-11T21:02:10.601759+0000 osd.4 (osd.4) 27 : cluster 0 3.9 scrub starts 2024-03-11T21:02:11.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:11 smithi146 bash[23809]: cluster 2024-03-11T21:02:10.604322+0000 osd.4 (osd.4) 28 : cluster 0 3.9 scrub ok 2024-03-11T21:02:11.578 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:02:11.605 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:02:11.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:11.607 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:02:11.707 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:11.729 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:11.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:11.733 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:02:11.835 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:11.855 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:11.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:11.858 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:02:11.959 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:11.980 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:11.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:11.982 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:02:12.083 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:12.110 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:12.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:12.113 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:02:12.213 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:12.233 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:12.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:12.236 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:02:12.336 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:02:12.363 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:02:12.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:12.365 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:02:12.466 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:02:12.488 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:02:12.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:12.490 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:02:12.591 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:12.617 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:12.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:12.619 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:02:12.719 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:12.739 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:12.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:12.741 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:02:12.842 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:02:12.867 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:02:12.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:12.869 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:02:12.970 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:02:12.997 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:02:12.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:12.999 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:02:13.100 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:13.128 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:13.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:13.130 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:02:13.231 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:02:13.257 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:02:13.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:13.259 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:02:13.360 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:13.382 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:13.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:13.384 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:02:13.485 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:02:13.497 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:13 smithi146 bash[23809]: cluster 2024-03-11T21:02:12.124219+0000 mgr.y (mgr.24380) 424 : cluster 0 pgmap v454: 148 pgs: 143 active+clean, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 578 op/s; 4763/594 objects degraded (801.852%); 0 B/s, 359 objects/s recovering 2024-03-11T21:02:13.511 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:02:13.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:13.514 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:02:13.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:13 smithi023 bash[21372]: cluster 2024-03-11T21:02:12.124219+0000 mgr.y (mgr.24380) 424 : cluster 0 pgmap v454: 148 pgs: 143 active+clean, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 578 op/s; 4763/594 objects degraded (801.852%); 0 B/s, 359 objects/s recovering 2024-03-11T21:02:13.535 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:13 smithi023 bash[29146]: cluster 2024-03-11T21:02:12.124219+0000 mgr.y (mgr.24380) 424 : cluster 0 pgmap v454: 148 pgs: 143 active+clean, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 578 op/s; 4763/594 objects degraded (801.852%); 0 B/s, 359 objects/s recovering 2024-03-11T21:02:13.616 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:02:13.640 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:02:13.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:13.643 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:02:13.744 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:02:13.772 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:02:13.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:13.775 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:02:13.876 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:13.895 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:13.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:13.897 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:02:13.998 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:14.025 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:14.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:14.027 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:02:14.128 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:14.157 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:14.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:14.159 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:02:14.260 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:14.263 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:14 smithi146 bash[23809]: cluster 2024-03-11T21:02:13.157314+0000 mon.a (mon.0) 1383 : cluster 0 osdmap e234: 8 total, 8 up, 8 in 2024-03-11T21:02:14.263 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:14 smithi146 bash[23809]: cluster 2024-03-11T21:02:14.166702+0000 mon.a (mon.0) 1384 : cluster 0 osdmap e235: 8 total, 8 up, 8 in 2024-03-11T21:02:14.283 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:14.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:14.285 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:02:14.386 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:14.408 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:14.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:14.410 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:02:14.511 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:02:14.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:14 smithi023 bash[21372]: cluster 2024-03-11T21:02:13.157314+0000 mon.a (mon.0) 1383 : cluster 0 osdmap e234: 8 total, 8 up, 8 in 2024-03-11T21:02:14.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:14 smithi023 bash[21372]: cluster 2024-03-11T21:02:14.166702+0000 mon.a (mon.0) 1384 : cluster 0 osdmap e235: 8 total, 8 up, 8 in 2024-03-11T21:02:14.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:14 smithi023 bash[29146]: cluster 2024-03-11T21:02:13.157314+0000 mon.a (mon.0) 1383 : cluster 0 osdmap e234: 8 total, 8 up, 8 in 2024-03-11T21:02:14.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:14 smithi023 bash[29146]: cluster 2024-03-11T21:02:14.166702+0000 mon.a (mon.0) 1384 : cluster 0 osdmap e235: 8 total, 8 up, 8 in 2024-03-11T21:02:14.537 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:02:14.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:14.539 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:02:14.640 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:14.659 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:14.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:14.661 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:02:14.761 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:14.782 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:14.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:14.786 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:02:14.887 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:02:14.913 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:02:14.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:14.915 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:02:15.015 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:15.035 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:15.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:15.038 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:02:15.139 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:15.159 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:15.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:15.161 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:02:15.262 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:15.284 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:15.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:15.286 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:02:15.387 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:02:15.416 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:02:15.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:15.419 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:02:15.458 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:15 smithi023 bash[21372]: cluster 2024-03-11T21:02:14.124813+0000 mgr.y (mgr.24380) 425 : cluster 0 pgmap v456: 148 pgs: 143 active+clean, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 55 op/s; 4763/594 objects degraded (801.852%); 0 B/s, 386 objects/s recovering 2024-03-11T21:02:15.458 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:15 smithi023 bash[21372]: cluster 2024-03-11T21:02:14.216109+0000 osd.1 (osd.1) 3 : cluster 0 purged_snaps scrub starts 2024-03-11T21:02:15.458 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:15 smithi023 bash[21372]: cluster 2024-03-11T21:02:14.216213+0000 osd.1 (osd.1) 4 : cluster 0 purged_snaps scrub ok 2024-03-11T21:02:15.459 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:15 smithi023 bash[21372]: cluster 2024-03-11T21:02:14.765288+0000 osd.7 (osd.7) 235 : cluster 0 4.1e scrub starts 2024-03-11T21:02:15.459 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:15 smithi023 bash[21372]: cluster 2024-03-11T21:02:14.767238+0000 osd.7 (osd.7) 236 : cluster 0 4.1e scrub ok 2024-03-11T21:02:15.459 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:15 smithi023 bash[21372]: cluster 2024-03-11T21:02:15.170180+0000 mon.a (mon.0) 1385 : cluster 3 Health check update: Degraded data redundancy: 4763/594 objects degraded (801.852%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-03-11T21:02:15.459 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:15 smithi023 bash[29146]: cluster 2024-03-11T21:02:14.124813+0000 mgr.y (mgr.24380) 425 : cluster 0 pgmap v456: 148 pgs: 143 active+clean, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 55 op/s; 4763/594 objects degraded (801.852%); 0 B/s, 386 objects/s recovering 2024-03-11T21:02:15.459 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:15 smithi023 bash[29146]: cluster 2024-03-11T21:02:14.216109+0000 osd.1 (osd.1) 3 : cluster 0 purged_snaps scrub starts 2024-03-11T21:02:15.459 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:15 smithi023 bash[29146]: cluster 2024-03-11T21:02:14.216213+0000 osd.1 (osd.1) 4 : cluster 0 purged_snaps scrub ok 2024-03-11T21:02:15.459 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:15 smithi023 bash[29146]: cluster 2024-03-11T21:02:14.765288+0000 osd.7 (osd.7) 235 : cluster 0 4.1e scrub starts 2024-03-11T21:02:15.459 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:15 smithi023 bash[29146]: cluster 2024-03-11T21:02:14.767238+0000 osd.7 (osd.7) 236 : cluster 0 4.1e scrub ok 2024-03-11T21:02:15.459 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:15 smithi023 bash[29146]: cluster 2024-03-11T21:02:15.170180+0000 mon.a (mon.0) 1385 : cluster 3 Health check update: Degraded data redundancy: 4763/594 objects degraded (801.852%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-03-11T21:02:15.475 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd reweight 3 1 2024-03-11T21:02:15.513 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:02:15 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:02:15.514 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:15 smithi146 bash[23809]: cluster 2024-03-11T21:02:14.124813+0000 mgr.y (mgr.24380) 425 : cluster 0 pgmap v456: 148 pgs: 143 active+clean, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 55 op/s; 4763/594 objects degraded (801.852%); 0 B/s, 386 objects/s recovering 2024-03-11T21:02:15.514 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:15 smithi146 bash[23809]: cluster 2024-03-11T21:02:14.216109+0000 osd.1 (osd.1) 3 : cluster 0 purged_snaps scrub starts 2024-03-11T21:02:15.514 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:15 smithi146 bash[23809]: cluster 2024-03-11T21:02:14.216213+0000 osd.1 (osd.1) 4 : cluster 0 purged_snaps scrub ok 2024-03-11T21:02:15.514 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:15 smithi146 bash[23809]: cluster 2024-03-11T21:02:14.765288+0000 osd.7 (osd.7) 235 : cluster 0 4.1e scrub starts 2024-03-11T21:02:15.514 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:15 smithi146 bash[23809]: cluster 2024-03-11T21:02:14.767238+0000 osd.7 (osd.7) 236 : cluster 0 4.1e scrub ok 2024-03-11T21:02:15.514 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:15 smithi146 bash[23809]: cluster 2024-03-11T21:02:15.170180+0000 mon.a (mon.0) 1385 : cluster 3 Health check update: Degraded data redundancy: 4763/594 objects degraded (801.852%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-03-11T21:02:15.519 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:15.548 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:15.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:15.550 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:02:15.651 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:15.677 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:15.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:15.680 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:02:15.780 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:02:15.800 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:02:15.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:15.802 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:02:15.903 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:15.922 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:15.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:15.924 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:02:16.026 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:16.051 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:16.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:16.053 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:02:16.154 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:02:16.181 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:02:16.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:16.184 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:02:16.285 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:02:16.296 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:16 smithi146 bash[23809]: cluster 2024-03-11T21:02:15.216669+0000 mon.a (mon.0) 1386 : cluster 0 osdmap e236: 8 total, 8 up, 8 in 2024-03-11T21:02:16.296 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:16 smithi146 bash[23809]: audit 2024-03-11T21:02:15.379148+0000 mgr.y (mgr.24380) 426 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:02:16.296 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:16 smithi146 bash[23809]: cluster 2024-03-11T21:02:15.786705+0000 osd.7 (osd.7) 237 : cluster 0 1.0 scrub starts 2024-03-11T21:02:16.296 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:16 smithi146 bash[23809]: cluster 2024-03-11T21:02:15.791891+0000 osd.7 (osd.7) 238 : cluster 0 1.0 scrub ok 2024-03-11T21:02:16.311 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:02:16.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:16.313 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:02:16.413 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:02:16.440 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:02:16.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:16.442 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:02:16.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:16 smithi023 bash[21372]: cluster 2024-03-11T21:02:15.216669+0000 mon.a (mon.0) 1386 : cluster 0 osdmap e236: 8 total, 8 up, 8 in 2024-03-11T21:02:16.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:16 smithi023 bash[21372]: audit 2024-03-11T21:02:15.379148+0000 mgr.y (mgr.24380) 426 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:02:16.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:16 smithi023 bash[21372]: cluster 2024-03-11T21:02:15.786705+0000 osd.7 (osd.7) 237 : cluster 0 1.0 scrub starts 2024-03-11T21:02:16.536 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:16 smithi023 bash[21372]: cluster 2024-03-11T21:02:15.791891+0000 osd.7 (osd.7) 238 : cluster 0 1.0 scrub ok 2024-03-11T21:02:16.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:16 smithi023 bash[29146]: cluster 2024-03-11T21:02:15.216669+0000 mon.a (mon.0) 1386 : cluster 0 osdmap e236: 8 total, 8 up, 8 in 2024-03-11T21:02:16.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:16 smithi023 bash[29146]: audit 2024-03-11T21:02:15.379148+0000 mgr.y (mgr.24380) 426 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:02:16.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:16 smithi023 bash[29146]: cluster 2024-03-11T21:02:15.786705+0000 osd.7 (osd.7) 237 : cluster 0 1.0 scrub starts 2024-03-11T21:02:16.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:16 smithi023 bash[29146]: cluster 2024-03-11T21:02:15.791891+0000 osd.7 (osd.7) 238 : cluster 0 1.0 scrub ok 2024-03-11T21:02:16.543 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:16.562 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:16.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:16.564 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:02:16.664 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:02:16.691 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:02:16.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:16.694 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:02:16.795 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:16.815 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:16.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:16.817 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:02:16.918 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:02:16.945 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:02:16.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:16.947 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:02:17.048 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:17.068 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:17.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:17.071 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:02:17.171 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:17.199 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:17.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:17.201 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:02:17.302 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:17.328 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:17.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:17.330 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:02:17.431 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:17.442 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:17 smithi023 bash[21372]: cluster 2024-03-11T21:02:16.125431+0000 mgr.y (mgr.24380) 427 : cluster 0 pgmap v459: 148 pgs: 144 active+clean, 1 active+undersized+remapped, 3 active+recovering+undersized+degraded+remapped; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 3 op/s; 2012/594 objects degraded (338.721%); 0 B/s, 744 objects/s recovering 2024-03-11T21:02:17.443 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:17 smithi023 bash[21372]: cluster 2024-03-11T21:02:16.228503+0000 mon.a (mon.0) 1387 : cluster 0 osdmap e237: 8 total, 8 up, 8 in 2024-03-11T21:02:17.443 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:17 smithi023 bash[21372]: cluster 2024-03-11T21:02:16.397269+0000 osd.6 (osd.6) 158 : cluster 0 4.5 scrub starts 2024-03-11T21:02:17.443 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:17 smithi023 bash[21372]: cluster 2024-03-11T21:02:16.398867+0000 osd.6 (osd.6) 159 : cluster 0 4.5 scrub ok 2024-03-11T21:02:17.443 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:17 smithi023 bash[21372]: cluster 2024-03-11T21:02:16.831474+0000 osd.7 (osd.7) 239 : cluster 0 6.1c deep-scrub starts 2024-03-11T21:02:17.443 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:17 smithi023 bash[21372]: cluster 2024-03-11T21:02:16.834773+0000 osd.7 (osd.7) 240 : cluster 0 6.1c deep-scrub ok 2024-03-11T21:02:17.443 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:17 smithi023 bash[29146]: cluster 2024-03-11T21:02:16.125431+0000 mgr.y (mgr.24380) 427 : cluster 0 pgmap v459: 148 pgs: 144 active+clean, 1 active+undersized+remapped, 3 active+recovering+undersized+degraded+remapped; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 3 op/s; 2012/594 objects degraded (338.721%); 0 B/s, 744 objects/s recovering 2024-03-11T21:02:17.443 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:17 smithi023 bash[29146]: cluster 2024-03-11T21:02:16.228503+0000 mon.a (mon.0) 1387 : cluster 0 osdmap e237: 8 total, 8 up, 8 in 2024-03-11T21:02:17.443 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:17 smithi023 bash[29146]: cluster 2024-03-11T21:02:16.397269+0000 osd.6 (osd.6) 158 : cluster 0 4.5 scrub starts 2024-03-11T21:02:17.443 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:17 smithi023 bash[29146]: cluster 2024-03-11T21:02:16.398867+0000 osd.6 (osd.6) 159 : cluster 0 4.5 scrub ok 2024-03-11T21:02:17.443 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:17 smithi023 bash[29146]: cluster 2024-03-11T21:02:16.831474+0000 osd.7 (osd.7) 239 : cluster 0 6.1c deep-scrub starts 2024-03-11T21:02:17.443 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:17 smithi023 bash[29146]: cluster 2024-03-11T21:02:16.834773+0000 osd.7 (osd.7) 240 : cluster 0 6.1c deep-scrub ok 2024-03-11T21:02:17.459 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:17.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:17.461 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:02:17.462 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T21:02:17.480 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd set noscrub 2024-03-11T21:02:17.513 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:17 smithi146 bash[23809]: cluster 2024-03-11T21:02:16.125431+0000 mgr.y (mgr.24380) 427 : cluster 0 pgmap v459: 148 pgs: 144 active+clean, 1 active+undersized+remapped, 3 active+recovering+undersized+degraded+remapped; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 3 op/s; 2012/594 objects degraded (338.721%); 0 B/s, 744 objects/s recovering 2024-03-11T21:02:17.513 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:17 smithi146 bash[23809]: cluster 2024-03-11T21:02:16.228503+0000 mon.a (mon.0) 1387 : cluster 0 osdmap e237: 8 total, 8 up, 8 in 2024-03-11T21:02:17.513 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:17 smithi146 bash[23809]: cluster 2024-03-11T21:02:16.397269+0000 osd.6 (osd.6) 158 : cluster 0 4.5 scrub starts 2024-03-11T21:02:17.513 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:17 smithi146 bash[23809]: cluster 2024-03-11T21:02:16.398867+0000 osd.6 (osd.6) 159 : cluster 0 4.5 scrub ok 2024-03-11T21:02:17.513 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:17 smithi146 bash[23809]: cluster 2024-03-11T21:02:16.831474+0000 osd.7 (osd.7) 239 : cluster 0 6.1c deep-scrub starts 2024-03-11T21:02:17.514 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:17 smithi146 bash[23809]: cluster 2024-03-11T21:02:16.834773+0000 osd.7 (osd.7) 240 : cluster 0 6.1c deep-scrub ok 2024-03-11T21:02:17.562 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:17.584 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:17.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:17.587 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:02:17.687 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:02:17.715 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:02:17.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:17.717 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:02:17.818 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:02:17.842 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:02:17.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:17.844 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:02:17.945 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:17.970 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:02:17.971 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:02:17.972 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:17.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:17.975 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:02:18.076 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:02:18.105 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:02:18.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:18.107 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:02:18.208 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:18.236 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:18.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:18.238 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:02:18.340 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:18.368 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:18.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:18.371 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:02:18.471 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:18.501 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:18.503 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:02:18 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:02:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:02:18.503 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:18 smithi023 bash[21372]: cluster 2024-03-11T21:02:17.147730+0000 osd.3 (osd.3) 244 : cluster 0 purged_snaps scrub starts 2024-03-11T21:02:18.503 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:18 smithi023 bash[21372]: cluster 2024-03-11T21:02:17.147837+0000 osd.3 (osd.3) 245 : cluster 0 purged_snaps scrub ok 2024-03-11T21:02:18.503 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:18 smithi023 bash[21372]: cluster 2024-03-11T21:02:17.241105+0000 mon.a (mon.0) 1388 : cluster 0 osdmap e238: 8 total, 8 up, 8 in 2024-03-11T21:02:18.503 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:18 smithi023 bash[21372]: cluster 2024-03-11T21:02:17.664275+0000 osd.5 (osd.5) 203 : cluster 0 4.1d scrub starts 2024-03-11T21:02:18.503 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:18 smithi023 bash[21372]: cluster 2024-03-11T21:02:17.669261+0000 osd.5 (osd.5) 204 : cluster 0 4.1d scrub ok 2024-03-11T21:02:18.504 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:18 smithi023 bash[29146]: cluster 2024-03-11T21:02:17.147730+0000 osd.3 (osd.3) 244 : cluster 0 purged_snaps scrub starts 2024-03-11T21:02:18.504 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:18 smithi023 bash[29146]: cluster 2024-03-11T21:02:17.147837+0000 osd.3 (osd.3) 245 : cluster 0 purged_snaps scrub ok 2024-03-11T21:02:18.504 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:18 smithi023 bash[29146]: cluster 2024-03-11T21:02:17.241105+0000 mon.a (mon.0) 1388 : cluster 0 osdmap e238: 8 total, 8 up, 8 in 2024-03-11T21:02:18.504 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:18 smithi023 bash[29146]: cluster 2024-03-11T21:02:17.664275+0000 osd.5 (osd.5) 203 : cluster 0 4.1d scrub starts 2024-03-11T21:02:18.504 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:18 smithi023 bash[29146]: cluster 2024-03-11T21:02:17.669261+0000 osd.5 (osd.5) 204 : cluster 0 4.1d scrub ok 2024-03-11T21:02:18.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:18.505 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:02:18.513 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:18 smithi146 bash[23809]: cluster 2024-03-11T21:02:17.147730+0000 osd.3 (osd.3) 244 : cluster 0 purged_snaps scrub starts 2024-03-11T21:02:18.513 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:18 smithi146 bash[23809]: cluster 2024-03-11T21:02:17.147837+0000 osd.3 (osd.3) 245 : cluster 0 purged_snaps scrub ok 2024-03-11T21:02:18.513 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:18 smithi146 bash[23809]: cluster 2024-03-11T21:02:17.241105+0000 mon.a (mon.0) 1388 : cluster 0 osdmap e238: 8 total, 8 up, 8 in 2024-03-11T21:02:18.513 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:18 smithi146 bash[23809]: cluster 2024-03-11T21:02:17.664275+0000 osd.5 (osd.5) 203 : cluster 0 4.1d scrub starts 2024-03-11T21:02:18.514 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:18 smithi146 bash[23809]: cluster 2024-03-11T21:02:17.669261+0000 osd.5 (osd.5) 204 : cluster 0 4.1d scrub ok 2024-03-11T21:02:18.605 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:02:18.631 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:02:18.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:18.633 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:02:18.734 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:18.763 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:18.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:18.765 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:02:18.866 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:18.894 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:18.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:18.896 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:02:18.997 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:02:19.024 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:02:19.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:19.026 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:02:19.127 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:19.155 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:19.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:19.157 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:02:19.258 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:19.292 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:19.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:19.295 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:02:19.395 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:19.408 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:19 smithi023 bash[21372]: cluster 2024-03-11T21:02:18.126218+0000 mgr.y (mgr.24380) 428 : cluster 0 pgmap v462: 148 pgs: 1 creating+peering, 145 active+clean, 1 active+undersized+remapped, 1 active+recovering+undersized+degraded+remapped; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 811/594 objects degraded (136.532%); 0 B/s, 987 objects/s recovering 2024-03-11T21:02:19.409 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:19 smithi023 bash[21372]: cluster 2024-03-11T21:02:18.150490+0000 osd.1 (osd.1) 5 : cluster 0 4.f scrub starts 2024-03-11T21:02:19.409 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:19 smithi023 bash[21372]: cluster 2024-03-11T21:02:18.152899+0000 osd.1 (osd.1) 6 : cluster 0 4.f scrub ok 2024-03-11T21:02:19.409 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:19 smithi023 bash[21372]: cluster 2024-03-11T21:02:18.270086+0000 mon.a (mon.0) 1389 : cluster 0 osdmap e239: 8 total, 8 up, 8 in 2024-03-11T21:02:19.409 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:19 smithi023 bash[21372]: cluster 2024-03-11T21:02:18.394781+0000 osd.6 (osd.6) 160 : cluster 0 8.5 scrub starts 2024-03-11T21:02:19.409 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:19 smithi023 bash[21372]: cluster 2024-03-11T21:02:18.410095+0000 osd.6 (osd.6) 161 : cluster 0 8.5 scrub ok 2024-03-11T21:02:19.409 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:19 smithi023 bash[21372]: cluster 2024-03-11T21:02:18.561926+0000 osd.6 (osd.6) 162 : cluster 0 purged_snaps scrub starts 2024-03-11T21:02:19.409 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:19 smithi023 bash[21372]: cluster 2024-03-11T21:02:18.562040+0000 osd.6 (osd.6) 163 : cluster 0 purged_snaps scrub ok 2024-03-11T21:02:19.409 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:19 smithi023 bash[29146]: cluster 2024-03-11T21:02:18.126218+0000 mgr.y (mgr.24380) 428 : cluster 0 pgmap v462: 148 pgs: 1 creating+peering, 145 active+clean, 1 active+undersized+remapped, 1 active+recovering+undersized+degraded+remapped; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 811/594 objects degraded (136.532%); 0 B/s, 987 objects/s recovering 2024-03-11T21:02:19.409 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:19 smithi023 bash[29146]: cluster 2024-03-11T21:02:18.150490+0000 osd.1 (osd.1) 5 : cluster 0 4.f scrub starts 2024-03-11T21:02:19.409 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:19 smithi023 bash[29146]: cluster 2024-03-11T21:02:18.152899+0000 osd.1 (osd.1) 6 : cluster 0 4.f scrub ok 2024-03-11T21:02:19.409 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:19 smithi023 bash[29146]: cluster 2024-03-11T21:02:18.270086+0000 mon.a (mon.0) 1389 : cluster 0 osdmap e239: 8 total, 8 up, 8 in 2024-03-11T21:02:19.409 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:19 smithi023 bash[29146]: cluster 2024-03-11T21:02:18.394781+0000 osd.6 (osd.6) 160 : cluster 0 8.5 scrub starts 2024-03-11T21:02:19.410 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:19 smithi023 bash[29146]: cluster 2024-03-11T21:02:18.410095+0000 osd.6 (osd.6) 161 : cluster 0 8.5 scrub ok 2024-03-11T21:02:19.410 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:19 smithi023 bash[29146]: cluster 2024-03-11T21:02:18.561926+0000 osd.6 (osd.6) 162 : cluster 0 purged_snaps scrub starts 2024-03-11T21:02:19.410 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:19 smithi023 bash[29146]: cluster 2024-03-11T21:02:18.562040+0000 osd.6 (osd.6) 163 : cluster 0 purged_snaps scrub ok 2024-03-11T21:02:19.424 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:19.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:19.426 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:02:19.527 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:02:19.541 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:19 smithi146 bash[23809]: cluster 2024-03-11T21:02:18.126218+0000 mgr.y (mgr.24380) 428 : cluster 0 pgmap v462: 148 pgs: 1 creating+peering, 145 active+clean, 1 active+undersized+remapped, 1 active+recovering+undersized+degraded+remapped; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 811/594 objects degraded (136.532%); 0 B/s, 987 objects/s recovering 2024-03-11T21:02:19.541 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:19 smithi146 bash[23809]: cluster 2024-03-11T21:02:18.150490+0000 osd.1 (osd.1) 5 : cluster 0 4.f scrub starts 2024-03-11T21:02:19.541 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:19 smithi146 bash[23809]: cluster 2024-03-11T21:02:18.152899+0000 osd.1 (osd.1) 6 : cluster 0 4.f scrub ok 2024-03-11T21:02:19.541 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:19 smithi146 bash[23809]: cluster 2024-03-11T21:02:18.270086+0000 mon.a (mon.0) 1389 : cluster 0 osdmap e239: 8 total, 8 up, 8 in 2024-03-11T21:02:19.541 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:19 smithi146 bash[23809]: cluster 2024-03-11T21:02:18.394781+0000 osd.6 (osd.6) 160 : cluster 0 8.5 scrub starts 2024-03-11T21:02:19.542 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:19 smithi146 bash[23809]: cluster 2024-03-11T21:02:18.410095+0000 osd.6 (osd.6) 161 : cluster 0 8.5 scrub ok 2024-03-11T21:02:19.542 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:19 smithi146 bash[23809]: cluster 2024-03-11T21:02:18.561926+0000 osd.6 (osd.6) 162 : cluster 0 purged_snaps scrub starts 2024-03-11T21:02:19.542 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:19 smithi146 bash[23809]: cluster 2024-03-11T21:02:18.562040+0000 osd.6 (osd.6) 163 : cluster 0 purged_snaps scrub ok 2024-03-11T21:02:19.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:19.561 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:02:19.561 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:02:19.662 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:02:19.688 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:02:19.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:19.691 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:02:19.792 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:19.823 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:19.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:19.825 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:02:19.926 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:19.955 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:19.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:19.958 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:02:20.014 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:02:20.059 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:20.079 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:20.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:20.081 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:02:20.182 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:20.208 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:20.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:20.210 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:02:20.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:20 smithi023 bash[21372]: cluster 2024-03-11T21:02:18.680479+0000 osd.2 (osd.2) 154 : cluster 0 2.10 deep-scrub starts 2024-03-11T21:02:20.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:20 smithi023 bash[21372]: cluster 2024-03-11T21:02:18.682936+0000 osd.2 (osd.2) 155 : cluster 0 2.10 deep-scrub ok 2024-03-11T21:02:20.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:20 smithi023 bash[21372]: cluster 2024-03-11T21:02:19.272610+0000 mon.a (mon.0) 1390 : cluster 0 osdmap e240: 8 total, 8 up, 8 in 2024-03-11T21:02:20.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:20 smithi023 bash[21372]: cluster 2024-03-11T21:02:19.377196+0000 osd.6 (osd.6) 164 : cluster 0 2.17 scrub starts 2024-03-11T21:02:20.286 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:20 smithi023 bash[21372]: cluster 2024-03-11T21:02:19.379520+0000 osd.6 (osd.6) 165 : cluster 0 2.17 scrub ok 2024-03-11T21:02:20.286 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:20 smithi023 bash[21372]: cluster 2024-03-11T21:02:19.473573+0000 osd.3 (osd.3) 246 : cluster 0 5.0 scrub starts 2024-03-11T21:02:20.286 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:20 smithi023 bash[21372]: cluster 2024-03-11T21:02:19.475887+0000 osd.3 (osd.3) 247 : cluster 0 5.0 scrub ok 2024-03-11T21:02:20.286 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:20 smithi023 bash[21372]: audit 2024-03-11T21:02:20.034388+0000 mon.a (mon.0) 1391 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:02:20.286 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:20 smithi023 bash[21372]: audit 2024-03-11T21:02:20.036352+0000 mon.a (mon.0) 1392 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:02:20.286 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:20 smithi023 bash[21372]: cluster 2024-03-11T21:02:20.172209+0000 mon.a (mon.0) 1393 : cluster 3 Health check update: Degraded data redundancy: 811/594 objects degraded (136.532%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-11T21:02:20.286 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:20 smithi023 bash[21372]: cluster 2024-03-11T21:02:20.180955+0000 mon.a (mon.0) 1394 : cluster 0 osdmap e241: 8 total, 8 up, 8 in 2024-03-11T21:02:20.286 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:20 smithi023 bash[29146]: cluster 2024-03-11T21:02:18.680479+0000 osd.2 (osd.2) 154 : cluster 0 2.10 deep-scrub starts 2024-03-11T21:02:20.286 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:20 smithi023 bash[29146]: cluster 2024-03-11T21:02:18.682936+0000 osd.2 (osd.2) 155 : cluster 0 2.10 deep-scrub ok 2024-03-11T21:02:20.286 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:20 smithi023 bash[29146]: cluster 2024-03-11T21:02:19.272610+0000 mon.a (mon.0) 1390 : cluster 0 osdmap e240: 8 total, 8 up, 8 in 2024-03-11T21:02:20.286 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:20 smithi023 bash[29146]: cluster 2024-03-11T21:02:19.377196+0000 osd.6 (osd.6) 164 : cluster 0 2.17 scrub starts 2024-03-11T21:02:20.287 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:20 smithi023 bash[29146]: cluster 2024-03-11T21:02:19.379520+0000 osd.6 (osd.6) 165 : cluster 0 2.17 scrub ok 2024-03-11T21:02:20.287 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:20 smithi023 bash[29146]: cluster 2024-03-11T21:02:19.473573+0000 osd.3 (osd.3) 246 : cluster 0 5.0 scrub starts 2024-03-11T21:02:20.287 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:20 smithi023 bash[29146]: cluster 2024-03-11T21:02:19.475887+0000 osd.3 (osd.3) 247 : cluster 0 5.0 scrub ok 2024-03-11T21:02:20.287 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:20 smithi023 bash[29146]: audit 2024-03-11T21:02:20.034388+0000 mon.a (mon.0) 1391 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:02:20.287 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:20 smithi023 bash[29146]: audit 2024-03-11T21:02:20.036352+0000 mon.a (mon.0) 1392 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:02:20.287 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:20 smithi023 bash[29146]: cluster 2024-03-11T21:02:20.172209+0000 mon.a (mon.0) 1393 : cluster 3 Health check update: Degraded data redundancy: 811/594 objects degraded (136.532%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-11T21:02:20.287 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:20 smithi023 bash[29146]: cluster 2024-03-11T21:02:20.180955+0000 mon.a (mon.0) 1394 : cluster 0 osdmap e241: 8 total, 8 up, 8 in 2024-03-11T21:02:20.311 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:20.338 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:20.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:20.339 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:02:20.440 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:20.470 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:20.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:20.473 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:02:20.573 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:20.601 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:20.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:20.603 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:02:20.704 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:20.733 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:20.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:20.735 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:02:20.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:20 smithi146 bash[23809]: cluster 2024-03-11T21:02:18.680479+0000 osd.2 (osd.2) 154 : cluster 0 2.10 deep-scrub starts 2024-03-11T21:02:20.764 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:20 smithi146 bash[23809]: cluster 2024-03-11T21:02:18.682936+0000 osd.2 (osd.2) 155 : cluster 0 2.10 deep-scrub ok 2024-03-11T21:02:20.764 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:20 smithi146 bash[23809]: cluster 2024-03-11T21:02:19.272610+0000 mon.a (mon.0) 1390 : cluster 0 osdmap e240: 8 total, 8 up, 8 in 2024-03-11T21:02:20.764 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:20 smithi146 bash[23809]: cluster 2024-03-11T21:02:19.377196+0000 osd.6 (osd.6) 164 : cluster 0 2.17 scrub starts 2024-03-11T21:02:20.764 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:20 smithi146 bash[23809]: cluster 2024-03-11T21:02:19.379520+0000 osd.6 (osd.6) 165 : cluster 0 2.17 scrub ok 2024-03-11T21:02:20.764 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:20 smithi146 bash[23809]: cluster 2024-03-11T21:02:19.473573+0000 osd.3 (osd.3) 246 : cluster 0 5.0 scrub starts 2024-03-11T21:02:20.764 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:20 smithi146 bash[23809]: cluster 2024-03-11T21:02:19.475887+0000 osd.3 (osd.3) 247 : cluster 0 5.0 scrub ok 2024-03-11T21:02:20.764 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:20 smithi146 bash[23809]: audit 2024-03-11T21:02:20.034388+0000 mon.a (mon.0) 1391 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:02:20.764 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:20 smithi146 bash[23809]: audit 2024-03-11T21:02:20.036352+0000 mon.a (mon.0) 1392 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:02:20.764 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:20 smithi146 bash[23809]: cluster 2024-03-11T21:02:20.172209+0000 mon.a (mon.0) 1393 : cluster 3 Health check update: Degraded data redundancy: 811/594 objects degraded (136.532%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-11T21:02:20.764 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:20 smithi146 bash[23809]: cluster 2024-03-11T21:02:20.180955+0000 mon.a (mon.0) 1394 : cluster 0 osdmap e241: 8 total, 8 up, 8 in 2024-03-11T21:02:20.837 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:02:20.865 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:02:20.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:20.868 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:02:20.969 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:20.999 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:21.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:21.002 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:02:21.102 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:21.126 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:21.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:21.128 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:02:21.229 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:21.258 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:21.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:21.260 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:02:21.361 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:21.385 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:21.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:21.387 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:02:21.488 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:02:21.510 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:21 smithi023 bash[29146]: cluster 2024-03-11T21:02:19.923605+0000 osd.0 (osd.0) 175 : cluster 0 6.10 scrub starts 2024-03-11T21:02:21.510 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:21 smithi023 bash[29146]: cluster 2024-03-11T21:02:19.948352+0000 osd.0 (osd.0) 176 : cluster 0 6.10 scrub ok 2024-03-11T21:02:21.510 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:21 smithi023 bash[29146]: cluster 2024-03-11T21:02:20.095319+0000 osd.1 (osd.1) 7 : cluster 0 2.4 deep-scrub starts 2024-03-11T21:02:21.511 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:21 smithi023 bash[29146]: cluster 2024-03-11T21:02:20.096869+0000 osd.1 (osd.1) 8 : cluster 0 2.4 deep-scrub ok 2024-03-11T21:02:21.511 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:21 smithi023 bash[29146]: cluster 2024-03-11T21:02:20.127386+0000 mgr.y (mgr.24380) 429 : cluster 0 pgmap v465: 148 pgs: 1 creating+peering, 145 active+clean, 1 active+undersized+remapped, 1 active+recovering+undersized+degraded+remapped; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 811/594 objects degraded (136.532%); 0 B/s, 300 objects/s recovering 2024-03-11T21:02:21.511 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:21 smithi023 bash[29146]: cluster 2024-03-11T21:02:20.422610+0000 osd.6 (osd.6) 166 : cluster 0 5.2 scrub starts 2024-03-11T21:02:21.511 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:21 smithi023 bash[29146]: cluster 2024-03-11T21:02:20.438534+0000 osd.6 (osd.6) 167 : cluster 0 5.2 scrub ok 2024-03-11T21:02:21.511 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:21 smithi023 bash[21372]: cluster 2024-03-11T21:02:19.923605+0000 osd.0 (osd.0) 175 : cluster 0 6.10 scrub starts 2024-03-11T21:02:21.511 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:21 smithi023 bash[21372]: cluster 2024-03-11T21:02:19.948352+0000 osd.0 (osd.0) 176 : cluster 0 6.10 scrub ok 2024-03-11T21:02:21.511 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:21 smithi023 bash[21372]: cluster 2024-03-11T21:02:20.095319+0000 osd.1 (osd.1) 7 : cluster 0 2.4 deep-scrub starts 2024-03-11T21:02:21.511 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:21 smithi023 bash[21372]: cluster 2024-03-11T21:02:20.096869+0000 osd.1 (osd.1) 8 : cluster 0 2.4 deep-scrub ok 2024-03-11T21:02:21.511 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:21 smithi023 bash[21372]: cluster 2024-03-11T21:02:20.127386+0000 mgr.y (mgr.24380) 429 : cluster 0 pgmap v465: 148 pgs: 1 creating+peering, 145 active+clean, 1 active+undersized+remapped, 1 active+recovering+undersized+degraded+remapped; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 811/594 objects degraded (136.532%); 0 B/s, 300 objects/s recovering 2024-03-11T21:02:21.511 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:21 smithi023 bash[21372]: cluster 2024-03-11T21:02:20.422610+0000 osd.6 (osd.6) 166 : cluster 0 5.2 scrub starts 2024-03-11T21:02:21.512 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:21 smithi023 bash[21372]: cluster 2024-03-11T21:02:20.438534+0000 osd.6 (osd.6) 167 : cluster 0 5.2 scrub ok 2024-03-11T21:02:21.516 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:02:21.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:21.518 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:02:21.618 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:21.641 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:21.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:21.643 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:02:21.743 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:02:21.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:21 smithi146 bash[23809]: cluster 2024-03-11T21:02:19.923605+0000 osd.0 (osd.0) 175 : cluster 0 6.10 scrub starts 2024-03-11T21:02:21.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:21 smithi146 bash[23809]: cluster 2024-03-11T21:02:19.948352+0000 osd.0 (osd.0) 176 : cluster 0 6.10 scrub ok 2024-03-11T21:02:21.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:21 smithi146 bash[23809]: cluster 2024-03-11T21:02:20.095319+0000 osd.1 (osd.1) 7 : cluster 0 2.4 deep-scrub starts 2024-03-11T21:02:21.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:21 smithi146 bash[23809]: cluster 2024-03-11T21:02:20.096869+0000 osd.1 (osd.1) 8 : cluster 0 2.4 deep-scrub ok 2024-03-11T21:02:21.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:21 smithi146 bash[23809]: cluster 2024-03-11T21:02:20.127386+0000 mgr.y (mgr.24380) 429 : cluster 0 pgmap v465: 148 pgs: 1 creating+peering, 145 active+clean, 1 active+undersized+remapped, 1 active+recovering+undersized+degraded+remapped; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 811/594 objects degraded (136.532%); 0 B/s, 300 objects/s recovering 2024-03-11T21:02:21.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:21 smithi146 bash[23809]: cluster 2024-03-11T21:02:20.422610+0000 osd.6 (osd.6) 166 : cluster 0 5.2 scrub starts 2024-03-11T21:02:21.756 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:21 smithi146 bash[23809]: cluster 2024-03-11T21:02:20.438534+0000 osd.6 (osd.6) 167 : cluster 0 5.2 scrub ok 2024-03-11T21:02:21.771 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:02:21.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:21.774 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:02:21.875 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:21.895 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:21.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:21.897 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:02:21.998 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:02:22.025 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:02:22.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:22.027 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:02:22.041 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:02:22.042 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:02:22.127 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:22.152 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:22.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:22.154 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:02:22.255 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:02:22.280 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:02:22.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:22.282 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:02:22.383 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:22.410 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:22.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:22.413 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:02:22.514 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:02:22.543 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:02:22.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:22.545 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:02:22.619 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:22 smithi023 bash[29146]: cluster 2024-03-11T21:02:21.664964+0000 osd.5 (osd.5) 205 : cluster 0 3.16 scrub starts 2024-03-11T21:02:22.619 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:22 smithi023 bash[29146]: cluster 2024-03-11T21:02:21.668808+0000 osd.5 (osd.5) 206 : cluster 0 3.16 scrub ok 2024-03-11T21:02:22.619 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:22 smithi023 bash[21372]: cluster 2024-03-11T21:02:21.664964+0000 osd.5 (osd.5) 205 : cluster 0 3.16 scrub starts 2024-03-11T21:02:22.619 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:22 smithi023 bash[21372]: cluster 2024-03-11T21:02:21.668808+0000 osd.5 (osd.5) 206 : cluster 0 3.16 scrub ok 2024-03-11T21:02:22.646 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:22.668 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:22.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:22.670 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:02:22.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:22 smithi146 bash[23809]: cluster 2024-03-11T21:02:21.664964+0000 osd.5 (osd.5) 205 : cluster 0 3.16 scrub starts 2024-03-11T21:02:22.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:22 smithi146 bash[23809]: cluster 2024-03-11T21:02:21.668808+0000 osd.5 (osd.5) 206 : cluster 0 3.16 scrub ok 2024-03-11T21:02:22.771 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:22.799 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:22.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:22.801 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:02:22.902 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:22.929 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:22.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:22.931 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:02:23.032 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:23.057 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:23.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:23.059 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:02:23.160 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:23.188 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:23.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:23.190 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:02:23.291 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:23.318 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:23.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:23.320 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:02:23.421 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:02:23.432 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:23 smithi146 bash[23809]: cluster 2024-03-11T21:02:21.905977+0000 osd.0 (osd.0) 177 : cluster 0 3.6 scrub starts 2024-03-11T21:02:23.432 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:23 smithi146 bash[23809]: cluster 2024-03-11T21:02:21.908632+0000 osd.0 (osd.0) 178 : cluster 0 3.6 scrub ok 2024-03-11T21:02:23.432 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:23 smithi146 bash[23809]: cluster 2024-03-11T21:02:22.112780+0000 osd.1 (osd.1) 9 : cluster 0 6.1 scrub starts 2024-03-11T21:02:23.432 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:23 smithi146 bash[23809]: cluster 2024-03-11T21:02:22.126259+0000 osd.1 (osd.1) 10 : cluster 0 6.1 scrub ok 2024-03-11T21:02:23.432 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:23 smithi146 bash[23809]: cluster 2024-03-11T21:02:22.128128+0000 mgr.y (mgr.24380) 430 : cluster 0 pgmap v467: 148 pgs: 148 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 0 B/s, 165 objects/s recovering 2024-03-11T21:02:23.432 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:23 smithi146 bash[23809]: cluster 2024-03-11T21:02:22.284538+0000 mon.a (mon.0) 1395 : cluster 1 Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 811/594 objects degraded (136.532%), 1 pg degraded, 2 pgs undersized) 2024-03-11T21:02:23.432 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:23 smithi146 bash[23809]: cluster 2024-03-11T21:02:22.284588+0000 mon.a (mon.0) 1396 : cluster 1 Cluster is now healthy 2024-03-11T21:02:23.433 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:23 smithi146 bash[23809]: cluster 2024-03-11T21:02:22.615830+0000 osd.4 (osd.4) 29 : cluster 0 5.1 deep-scrub starts 2024-03-11T21:02:23.433 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:23 smithi146 bash[23809]: cluster 2024-03-11T21:02:22.617989+0000 osd.4 (osd.4) 30 : cluster 0 5.1 deep-scrub ok 2024-03-11T21:02:23.449 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:02:23.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:23.451 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:02:23.552 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:23.578 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:23.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:23.580 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:02:23.681 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:23.692 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:23 smithi023 bash[29146]: cluster 2024-03-11T21:02:21.905977+0000 osd.0 (osd.0) 177 : cluster 0 3.6 scrub starts 2024-03-11T21:02:23.692 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:23 smithi023 bash[29146]: cluster 2024-03-11T21:02:21.908632+0000 osd.0 (osd.0) 178 : cluster 0 3.6 scrub ok 2024-03-11T21:02:23.692 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:23 smithi023 bash[29146]: cluster 2024-03-11T21:02:22.112780+0000 osd.1 (osd.1) 9 : cluster 0 6.1 scrub starts 2024-03-11T21:02:23.692 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:23 smithi023 bash[29146]: cluster 2024-03-11T21:02:22.126259+0000 osd.1 (osd.1) 10 : cluster 0 6.1 scrub ok 2024-03-11T21:02:23.692 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:23 smithi023 bash[29146]: cluster 2024-03-11T21:02:22.128128+0000 mgr.y (mgr.24380) 430 : cluster 0 pgmap v467: 148 pgs: 148 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 0 B/s, 165 objects/s recovering 2024-03-11T21:02:23.693 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:23 smithi023 bash[29146]: cluster 2024-03-11T21:02:22.284538+0000 mon.a (mon.0) 1395 : cluster 1 Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 811/594 objects degraded (136.532%), 1 pg degraded, 2 pgs undersized) 2024-03-11T21:02:23.693 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:23 smithi023 bash[29146]: cluster 2024-03-11T21:02:22.284588+0000 mon.a (mon.0) 1396 : cluster 1 Cluster is now healthy 2024-03-11T21:02:23.693 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:23 smithi023 bash[29146]: cluster 2024-03-11T21:02:22.615830+0000 osd.4 (osd.4) 29 : cluster 0 5.1 deep-scrub starts 2024-03-11T21:02:23.693 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:23 smithi023 bash[29146]: cluster 2024-03-11T21:02:22.617989+0000 osd.4 (osd.4) 30 : cluster 0 5.1 deep-scrub ok 2024-03-11T21:02:23.693 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:23 smithi023 bash[21372]: cluster 2024-03-11T21:02:21.905977+0000 osd.0 (osd.0) 177 : cluster 0 3.6 scrub starts 2024-03-11T21:02:23.693 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:23 smithi023 bash[21372]: cluster 2024-03-11T21:02:21.908632+0000 osd.0 (osd.0) 178 : cluster 0 3.6 scrub ok 2024-03-11T21:02:23.693 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:23 smithi023 bash[21372]: cluster 2024-03-11T21:02:22.112780+0000 osd.1 (osd.1) 9 : cluster 0 6.1 scrub starts 2024-03-11T21:02:23.693 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:23 smithi023 bash[21372]: cluster 2024-03-11T21:02:22.126259+0000 osd.1 (osd.1) 10 : cluster 0 6.1 scrub ok 2024-03-11T21:02:23.693 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:23 smithi023 bash[21372]: cluster 2024-03-11T21:02:22.128128+0000 mgr.y (mgr.24380) 430 : cluster 0 pgmap v467: 148 pgs: 148 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 0 B/s, 165 objects/s recovering 2024-03-11T21:02:23.693 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:23 smithi023 bash[21372]: cluster 2024-03-11T21:02:22.284538+0000 mon.a (mon.0) 1395 : cluster 1 Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 811/594 objects degraded (136.532%), 1 pg degraded, 2 pgs undersized) 2024-03-11T21:02:23.693 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:23 smithi023 bash[21372]: cluster 2024-03-11T21:02:22.284588+0000 mon.a (mon.0) 1396 : cluster 1 Cluster is now healthy 2024-03-11T21:02:23.693 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:23 smithi023 bash[21372]: cluster 2024-03-11T21:02:22.615830+0000 osd.4 (osd.4) 29 : cluster 0 5.1 deep-scrub starts 2024-03-11T21:02:23.693 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:23 smithi023 bash[21372]: cluster 2024-03-11T21:02:22.617989+0000 osd.4 (osd.4) 30 : cluster 0 5.1 deep-scrub ok 2024-03-11T21:02:23.710 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:23.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:23.713 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:02:23.813 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:23.839 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:23.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:23.841 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:02:23.942 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:23.968 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:23.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:23.971 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:02:24.072 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:24.111 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:24.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:24.114 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:02:24.215 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:24.241 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:24.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:24.244 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:02:24.345 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:02:24.374 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:02:24.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:24.376 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:02:24.477 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:24.506 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:24.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:24.508 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:02:24.513 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:24 smithi146 bash[23809]: cluster 2024-03-11T21:02:23.135327+0000 osd.1 (osd.1) 11 : cluster 0 4.12 deep-scrub starts 2024-03-11T21:02:24.513 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:24 smithi146 bash[23809]: cluster 2024-03-11T21:02:23.137821+0000 osd.1 (osd.1) 12 : cluster 0 4.12 deep-scrub ok 2024-03-11T21:02:24.513 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:24 smithi146 bash[23809]: cluster 2024-03-11T21:02:23.415636+0000 osd.6 (osd.6) 168 : cluster 0 2.7 deep-scrub starts 2024-03-11T21:02:24.513 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:24 smithi146 bash[23809]: cluster 2024-03-11T21:02:23.417278+0000 osd.6 (osd.6) 169 : cluster 0 2.7 deep-scrub ok 2024-03-11T21:02:24.513 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:24 smithi146 bash[23809]: cluster 2024-03-11T21:02:23.743436+0000 osd.5 (osd.5) 207 : cluster 0 6.19 scrub starts 2024-03-11T21:02:24.514 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:24 smithi146 bash[23809]: cluster 2024-03-11T21:02:23.768643+0000 osd.5 (osd.5) 208 : cluster 0 6.19 scrub ok 2024-03-11T21:02:24.609 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:02:24.636 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:02:24.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:24.639 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:02:24.736 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:24 smithi023 bash[21372]: cluster 2024-03-11T21:02:23.135327+0000 osd.1 (osd.1) 11 : cluster 0 4.12 deep-scrub starts 2024-03-11T21:02:24.736 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:24 smithi023 bash[21372]: cluster 2024-03-11T21:02:23.137821+0000 osd.1 (osd.1) 12 : cluster 0 4.12 deep-scrub ok 2024-03-11T21:02:24.736 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:24 smithi023 bash[21372]: cluster 2024-03-11T21:02:23.415636+0000 osd.6 (osd.6) 168 : cluster 0 2.7 deep-scrub starts 2024-03-11T21:02:24.737 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:24 smithi023 bash[21372]: cluster 2024-03-11T21:02:23.417278+0000 osd.6 (osd.6) 169 : cluster 0 2.7 deep-scrub ok 2024-03-11T21:02:24.737 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:24 smithi023 bash[21372]: cluster 2024-03-11T21:02:23.743436+0000 osd.5 (osd.5) 207 : cluster 0 6.19 scrub starts 2024-03-11T21:02:24.737 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:24 smithi023 bash[21372]: cluster 2024-03-11T21:02:23.768643+0000 osd.5 (osd.5) 208 : cluster 0 6.19 scrub ok 2024-03-11T21:02:24.737 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:24 smithi023 bash[29146]: cluster 2024-03-11T21:02:23.135327+0000 osd.1 (osd.1) 11 : cluster 0 4.12 deep-scrub starts 2024-03-11T21:02:24.737 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:24 smithi023 bash[29146]: cluster 2024-03-11T21:02:23.137821+0000 osd.1 (osd.1) 12 : cluster 0 4.12 deep-scrub ok 2024-03-11T21:02:24.737 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:24 smithi023 bash[29146]: cluster 2024-03-11T21:02:23.415636+0000 osd.6 (osd.6) 168 : cluster 0 2.7 deep-scrub starts 2024-03-11T21:02:24.737 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:24 smithi023 bash[29146]: cluster 2024-03-11T21:02:23.417278+0000 osd.6 (osd.6) 169 : cluster 0 2.7 deep-scrub ok 2024-03-11T21:02:24.737 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:24 smithi023 bash[29146]: cluster 2024-03-11T21:02:23.743436+0000 osd.5 (osd.5) 207 : cluster 0 6.19 scrub starts 2024-03-11T21:02:24.737 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:24 smithi023 bash[29146]: cluster 2024-03-11T21:02:23.768643+0000 osd.5 (osd.5) 208 : cluster 0 6.19 scrub ok 2024-03-11T21:02:24.740 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:02:24.767 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:02:24.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:24.770 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:02:24.871 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:24.899 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:24.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:24.901 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:02:25.002 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:25.028 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:25.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:25.031 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:02:25.131 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:25.157 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:25.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:25.160 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:02:25.261 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:02:25.289 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:02:25.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:25.292 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:02:25.392 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:02:25.419 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:02:25.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:25.421 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:02:25.522 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:25.548 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:25.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:25.551 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:02:25.651 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:25.679 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:25.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:25.681 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:02:25.763 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:02:25 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:02:25.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:25 smithi146 bash[23809]: cluster 2024-03-11T21:02:24.128537+0000 mgr.y (mgr.24380) 431 : cluster 0 pgmap v468: 148 pgs: 148 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 0 B/s, 135 objects/s recovering 2024-03-11T21:02:25.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:25 smithi146 bash[23809]: cluster 2024-03-11T21:02:24.618628+0000 osd.4 (osd.4) 31 : cluster 0 2.f scrub starts 2024-03-11T21:02:25.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:25 smithi146 bash[23809]: cluster 2024-03-11T21:02:24.621947+0000 osd.4 (osd.4) 32 : cluster 0 2.f scrub ok 2024-03-11T21:02:25.764 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:25 smithi146 bash[23809]: cluster 2024-03-11T21:02:24.701683+0000 osd.5 (osd.5) 209 : cluster 0 3.1c scrub starts 2024-03-11T21:02:25.764 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:25 smithi146 bash[23809]: cluster 2024-03-11T21:02:24.705523+0000 osd.5 (osd.5) 210 : cluster 0 3.1c scrub ok 2024-03-11T21:02:25.782 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:02:25.808 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:02:25.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:25.810 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:02:25.814 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:25 smithi023 bash[21372]: cluster 2024-03-11T21:02:24.128537+0000 mgr.y (mgr.24380) 431 : cluster 0 pgmap v468: 148 pgs: 148 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 0 B/s, 135 objects/s recovering 2024-03-11T21:02:25.814 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:25 smithi023 bash[21372]: cluster 2024-03-11T21:02:24.618628+0000 osd.4 (osd.4) 31 : cluster 0 2.f scrub starts 2024-03-11T21:02:25.815 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:25 smithi023 bash[21372]: cluster 2024-03-11T21:02:24.621947+0000 osd.4 (osd.4) 32 : cluster 0 2.f scrub ok 2024-03-11T21:02:25.815 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:25 smithi023 bash[21372]: cluster 2024-03-11T21:02:24.701683+0000 osd.5 (osd.5) 209 : cluster 0 3.1c scrub starts 2024-03-11T21:02:25.815 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:25 smithi023 bash[21372]: cluster 2024-03-11T21:02:24.705523+0000 osd.5 (osd.5) 210 : cluster 0 3.1c scrub ok 2024-03-11T21:02:25.815 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:25 smithi023 bash[29146]: cluster 2024-03-11T21:02:24.128537+0000 mgr.y (mgr.24380) 431 : cluster 0 pgmap v468: 148 pgs: 148 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 0 B/s, 135 objects/s recovering 2024-03-11T21:02:25.815 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:25 smithi023 bash[29146]: cluster 2024-03-11T21:02:24.618628+0000 osd.4 (osd.4) 31 : cluster 0 2.f scrub starts 2024-03-11T21:02:25.815 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:25 smithi023 bash[29146]: cluster 2024-03-11T21:02:24.621947+0000 osd.4 (osd.4) 32 : cluster 0 2.f scrub ok 2024-03-11T21:02:25.815 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:25 smithi023 bash[29146]: cluster 2024-03-11T21:02:24.701683+0000 osd.5 (osd.5) 209 : cluster 0 3.1c scrub starts 2024-03-11T21:02:25.815 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:25 smithi023 bash[29146]: cluster 2024-03-11T21:02:24.705523+0000 osd.5 (osd.5) 210 : cluster 0 3.1c scrub ok 2024-03-11T21:02:25.911 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:25.939 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:25.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:25.941 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:02:26.042 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:26.069 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:26.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:26.071 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:02:26.172 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:26.199 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:26.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:26.202 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:02:26.303 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:26.331 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:26.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:26.333 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:02:26.434 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:26.460 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:26.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:26.462 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:02:26.536 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:26 smithi023 bash[21372]: cluster 2024-03-11T21:02:24.664098+0000 osd.2 (osd.2) 156 : cluster 0 4.1c deep-scrub starts 2024-03-11T21:02:26.536 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:26 smithi023 bash[21372]: cluster 2024-03-11T21:02:24.666318+0000 osd.2 (osd.2) 157 : cluster 0 4.1c deep-scrub ok 2024-03-11T21:02:26.536 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:26 smithi023 bash[21372]: cluster 2024-03-11T21:02:25.180095+0000 osd.1 (osd.1) 13 : cluster 0 6.4 scrub starts 2024-03-11T21:02:26.536 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:26 smithi023 bash[21372]: cluster 2024-03-11T21:02:25.181911+0000 osd.1 (osd.1) 14 : cluster 0 6.4 scrub ok 2024-03-11T21:02:26.536 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:26 smithi023 bash[21372]: audit 2024-03-11T21:02:25.389934+0000 mgr.y (mgr.24380) 432 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:02:26.536 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:26 smithi023 bash[21372]: cluster 2024-03-11T21:02:25.426169+0000 osd.6 (osd.6) 170 : cluster 0 8.9 deep-scrub starts 2024-03-11T21:02:26.536 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:26 smithi023 bash[21372]: cluster 2024-03-11T21:02:25.436059+0000 osd.6 (osd.6) 171 : cluster 0 8.9 deep-scrub ok 2024-03-11T21:02:26.536 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:26 smithi023 bash[21372]: cluster 2024-03-11T21:02:25.530648+0000 osd.3 (osd.3) 248 : cluster 0 4.14 scrub starts 2024-03-11T21:02:26.536 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:26 smithi023 bash[21372]: cluster 2024-03-11T21:02:25.533655+0000 osd.3 (osd.3) 249 : cluster 0 4.14 scrub ok 2024-03-11T21:02:26.537 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:26 smithi023 bash[29146]: cluster 2024-03-11T21:02:24.664098+0000 osd.2 (osd.2) 156 : cluster 0 4.1c deep-scrub starts 2024-03-11T21:02:26.537 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:26 smithi023 bash[29146]: cluster 2024-03-11T21:02:24.666318+0000 osd.2 (osd.2) 157 : cluster 0 4.1c deep-scrub ok 2024-03-11T21:02:26.537 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:26 smithi023 bash[29146]: cluster 2024-03-11T21:02:25.180095+0000 osd.1 (osd.1) 13 : cluster 0 6.4 scrub starts 2024-03-11T21:02:26.537 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:26 smithi023 bash[29146]: cluster 2024-03-11T21:02:25.181911+0000 osd.1 (osd.1) 14 : cluster 0 6.4 scrub ok 2024-03-11T21:02:26.537 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:26 smithi023 bash[29146]: audit 2024-03-11T21:02:25.389934+0000 mgr.y (mgr.24380) 432 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:02:26.537 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:26 smithi023 bash[29146]: cluster 2024-03-11T21:02:25.426169+0000 osd.6 (osd.6) 170 : cluster 0 8.9 deep-scrub starts 2024-03-11T21:02:26.537 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:26 smithi023 bash[29146]: cluster 2024-03-11T21:02:25.436059+0000 osd.6 (osd.6) 171 : cluster 0 8.9 deep-scrub ok 2024-03-11T21:02:26.537 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:26 smithi023 bash[29146]: cluster 2024-03-11T21:02:25.530648+0000 osd.3 (osd.3) 248 : cluster 0 4.14 scrub starts 2024-03-11T21:02:26.537 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:26 smithi023 bash[29146]: cluster 2024-03-11T21:02:25.533655+0000 osd.3 (osd.3) 249 : cluster 0 4.14 scrub ok 2024-03-11T21:02:26.563 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:26.591 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:26.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:26.593 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:02:26.694 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:02:26.715 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:02:26.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:26.717 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:02:26.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:26 smithi146 bash[23809]: cluster 2024-03-11T21:02:24.664098+0000 osd.2 (osd.2) 156 : cluster 0 4.1c deep-scrub starts 2024-03-11T21:02:26.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:26 smithi146 bash[23809]: cluster 2024-03-11T21:02:24.666318+0000 osd.2 (osd.2) 157 : cluster 0 4.1c deep-scrub ok 2024-03-11T21:02:26.764 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:26 smithi146 bash[23809]: cluster 2024-03-11T21:02:25.180095+0000 osd.1 (osd.1) 13 : cluster 0 6.4 scrub starts 2024-03-11T21:02:26.764 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:26 smithi146 bash[23809]: cluster 2024-03-11T21:02:25.181911+0000 osd.1 (osd.1) 14 : cluster 0 6.4 scrub ok 2024-03-11T21:02:26.764 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:26 smithi146 bash[23809]: audit 2024-03-11T21:02:25.389934+0000 mgr.y (mgr.24380) 432 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:02:26.764 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:26 smithi146 bash[23809]: cluster 2024-03-11T21:02:25.426169+0000 osd.6 (osd.6) 170 : cluster 0 8.9 deep-scrub starts 2024-03-11T21:02:26.764 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:26 smithi146 bash[23809]: cluster 2024-03-11T21:02:25.436059+0000 osd.6 (osd.6) 171 : cluster 0 8.9 deep-scrub ok 2024-03-11T21:02:26.764 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:26 smithi146 bash[23809]: cluster 2024-03-11T21:02:25.530648+0000 osd.3 (osd.3) 248 : cluster 0 4.14 scrub starts 2024-03-11T21:02:26.764 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:26 smithi146 bash[23809]: cluster 2024-03-11T21:02:25.533655+0000 osd.3 (osd.3) 249 : cluster 0 4.14 scrub ok 2024-03-11T21:02:26.818 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:02:26.846 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:02:26.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:26.848 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:02:26.949 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:02:26.975 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:02:26.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:26.977 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:02:27.078 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:27.106 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:27.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:27.109 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:02:27.209 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:02:27.236 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:02:27.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:27.238 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:02:27.339 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:27.351 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:27 smithi023 bash[21372]: cluster 2024-03-11T21:02:25.854491+0000 osd.5 (osd.5) 211 : cluster 0 purged_snaps scrub starts 2024-03-11T21:02:27.351 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:27 smithi023 bash[21372]: cluster 2024-03-11T21:02:25.854623+0000 osd.5 (osd.5) 212 : cluster 0 purged_snaps scrub ok 2024-03-11T21:02:27.351 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:27 smithi023 bash[21372]: cluster 2024-03-11T21:02:26.129463+0000 mgr.y (mgr.24380) 433 : cluster 0 pgmap v469: 148 pgs: 148 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 0 B/s, 103 objects/s recovering 2024-03-11T21:02:27.351 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:27 smithi023 bash[21372]: cluster 2024-03-11T21:02:26.331204+0000 mon.a (mon.0) 1397 : cluster 0 osdmap e242: 8 total, 8 up, 8 in 2024-03-11T21:02:27.351 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:27 smithi023 bash[21372]: cluster 2024-03-11T21:02:26.394929+0000 osd.6 (osd.6) 172 : cluster 0 8.6 scrub starts 2024-03-11T21:02:27.351 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:27 smithi023 bash[21372]: cluster 2024-03-11T21:02:26.408784+0000 osd.6 (osd.6) 173 : cluster 0 8.6 scrub ok 2024-03-11T21:02:27.351 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:27 smithi023 bash[29146]: cluster 2024-03-11T21:02:25.854491+0000 osd.5 (osd.5) 211 : cluster 0 purged_snaps scrub starts 2024-03-11T21:02:27.351 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:27 smithi023 bash[29146]: cluster 2024-03-11T21:02:25.854623+0000 osd.5 (osd.5) 212 : cluster 0 purged_snaps scrub ok 2024-03-11T21:02:27.351 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:27 smithi023 bash[29146]: cluster 2024-03-11T21:02:26.129463+0000 mgr.y (mgr.24380) 433 : cluster 0 pgmap v469: 148 pgs: 148 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 0 B/s, 103 objects/s recovering 2024-03-11T21:02:27.352 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:27 smithi023 bash[29146]: cluster 2024-03-11T21:02:26.331204+0000 mon.a (mon.0) 1397 : cluster 0 osdmap e242: 8 total, 8 up, 8 in 2024-03-11T21:02:27.352 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:27 smithi023 bash[29146]: cluster 2024-03-11T21:02:26.394929+0000 osd.6 (osd.6) 172 : cluster 0 8.6 scrub starts 2024-03-11T21:02:27.352 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:27 smithi023 bash[29146]: cluster 2024-03-11T21:02:26.408784+0000 osd.6 (osd.6) 173 : cluster 0 8.6 scrub ok 2024-03-11T21:02:27.366 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:27.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:27.368 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:02:27.469 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:27.498 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:27.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:27.500 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:02:27.513 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:27 smithi146 bash[23809]: cluster 2024-03-11T21:02:25.854491+0000 osd.5 (osd.5) 211 : cluster 0 purged_snaps scrub starts 2024-03-11T21:02:27.513 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:27 smithi146 bash[23809]: cluster 2024-03-11T21:02:25.854623+0000 osd.5 (osd.5) 212 : cluster 0 purged_snaps scrub ok 2024-03-11T21:02:27.513 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:27 smithi146 bash[23809]: cluster 2024-03-11T21:02:26.129463+0000 mgr.y (mgr.24380) 433 : cluster 0 pgmap v469: 148 pgs: 148 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 0 B/s, 103 objects/s recovering 2024-03-11T21:02:27.513 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:27 smithi146 bash[23809]: cluster 2024-03-11T21:02:26.331204+0000 mon.a (mon.0) 1397 : cluster 0 osdmap e242: 8 total, 8 up, 8 in 2024-03-11T21:02:27.513 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:27 smithi146 bash[23809]: cluster 2024-03-11T21:02:26.394929+0000 osd.6 (osd.6) 172 : cluster 0 8.6 scrub starts 2024-03-11T21:02:27.514 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:27 smithi146 bash[23809]: cluster 2024-03-11T21:02:26.408784+0000 osd.6 (osd.6) 173 : cluster 0 8.6 scrub ok 2024-03-11T21:02:27.601 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:27.628 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:27.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:27.631 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:02:27.732 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:27.760 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:27.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:27.762 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:02:27.863 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:02:27.889 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:02:27.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:27.891 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:02:27.992 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:02:28.019 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:02:28.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:28.021 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:02:28.122 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:02:28.149 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:02:28.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:28.151 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:02:28.251 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:02:28.276 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:02:28.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:28.278 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:02:28.379 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:28.390 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:28 smithi146 bash[23809]: cluster 2024-03-11T21:02:26.833321+0000 osd.0 (osd.0) 179 : cluster 0 4.6 deep-scrub starts 2024-03-11T21:02:28.390 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:28 smithi146 bash[23809]: cluster 2024-03-11T21:02:26.839771+0000 osd.0 (osd.0) 180 : cluster 0 4.6 deep-scrub ok 2024-03-11T21:02:28.390 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:28 smithi146 bash[23809]: cluster 2024-03-11T21:02:27.727361+0000 osd.5 (osd.5) 213 : cluster 0 6.d deep-scrub starts 2024-03-11T21:02:28.390 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:28 smithi146 bash[23809]: cluster 2024-03-11T21:02:27.775523+0000 osd.5 (osd.5) 214 : cluster 0 6.d deep-scrub ok 2024-03-11T21:02:28.405 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:28.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:28.407 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:02:28.508 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:02:28.536 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:02:28.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:28.538 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:02:28.639 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:28.650 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:02:28 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:02:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:02:28.651 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:28 smithi023 bash[29146]: cluster 2024-03-11T21:02:26.833321+0000 osd.0 (osd.0) 179 : cluster 0 4.6 deep-scrub starts 2024-03-11T21:02:28.651 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:28 smithi023 bash[29146]: cluster 2024-03-11T21:02:26.839771+0000 osd.0 (osd.0) 180 : cluster 0 4.6 deep-scrub ok 2024-03-11T21:02:28.651 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:28 smithi023 bash[29146]: cluster 2024-03-11T21:02:27.727361+0000 osd.5 (osd.5) 213 : cluster 0 6.d deep-scrub starts 2024-03-11T21:02:28.651 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:28 smithi023 bash[29146]: cluster 2024-03-11T21:02:27.775523+0000 osd.5 (osd.5) 214 : cluster 0 6.d deep-scrub ok 2024-03-11T21:02:28.651 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:28 smithi023 bash[21372]: cluster 2024-03-11T21:02:26.833321+0000 osd.0 (osd.0) 179 : cluster 0 4.6 deep-scrub starts 2024-03-11T21:02:28.651 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:28 smithi023 bash[21372]: cluster 2024-03-11T21:02:26.839771+0000 osd.0 (osd.0) 180 : cluster 0 4.6 deep-scrub ok 2024-03-11T21:02:28.651 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:28 smithi023 bash[21372]: cluster 2024-03-11T21:02:27.727361+0000 osd.5 (osd.5) 213 : cluster 0 6.d deep-scrub starts 2024-03-11T21:02:28.651 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:28 smithi023 bash[21372]: cluster 2024-03-11T21:02:27.775523+0000 osd.5 (osd.5) 214 : cluster 0 6.d deep-scrub ok 2024-03-11T21:02:28.667 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:28.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:28.669 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:02:28.770 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:02:28.797 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:02:28.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:28.799 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:02:28.900 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:02:28.926 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:02:28.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:28.929 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:02:29.029 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:29.054 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:29.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:29.056 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:02:29.157 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:29.185 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:29.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:29.188 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:02:29.288 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:02:29.314 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:02:29.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:29.317 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:02:29.417 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:29.436 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:29.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:29.439 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:02:29.513 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:29 smithi146 bash[23809]: cluster 2024-03-11T21:02:28.129967+0000 mgr.y (mgr.24380) 434 : cluster 0 pgmap v471: 148 pgs: 148 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 101 objects/s recovering 2024-03-11T21:02:29.513 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:29 smithi146 bash[23809]: cluster 2024-03-11T21:02:28.403343+0000 osd.6 (osd.6) 174 : cluster 0 2.14 deep-scrub starts 2024-03-11T21:02:29.514 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:29 smithi146 bash[23809]: cluster 2024-03-11T21:02:28.404991+0000 osd.6 (osd.6) 175 : cluster 0 2.14 deep-scrub ok 2024-03-11T21:02:29.514 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:29 smithi146 bash[23809]: cluster 2024-03-11T21:02:28.621651+0000 osd.2 (osd.2) 158 : cluster 0 purged_snaps scrub starts 2024-03-11T21:02:29.514 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:29 smithi146 bash[23809]: cluster 2024-03-11T21:02:28.621762+0000 osd.2 (osd.2) 159 : cluster 0 purged_snaps scrub ok 2024-03-11T21:02:29.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:29 smithi023 bash[21372]: cluster 2024-03-11T21:02:28.129967+0000 mgr.y (mgr.24380) 434 : cluster 0 pgmap v471: 148 pgs: 148 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 101 objects/s recovering 2024-03-11T21:02:29.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:29 smithi023 bash[21372]: cluster 2024-03-11T21:02:28.403343+0000 osd.6 (osd.6) 174 : cluster 0 2.14 deep-scrub starts 2024-03-11T21:02:29.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:29 smithi023 bash[21372]: cluster 2024-03-11T21:02:28.404991+0000 osd.6 (osd.6) 175 : cluster 0 2.14 deep-scrub ok 2024-03-11T21:02:29.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:29 smithi023 bash[21372]: cluster 2024-03-11T21:02:28.621651+0000 osd.2 (osd.2) 158 : cluster 0 purged_snaps scrub starts 2024-03-11T21:02:29.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:29 smithi023 bash[21372]: cluster 2024-03-11T21:02:28.621762+0000 osd.2 (osd.2) 159 : cluster 0 purged_snaps scrub ok 2024-03-11T21:02:29.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:29 smithi023 bash[29146]: cluster 2024-03-11T21:02:28.129967+0000 mgr.y (mgr.24380) 434 : cluster 0 pgmap v471: 148 pgs: 148 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 101 objects/s recovering 2024-03-11T21:02:29.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:29 smithi023 bash[29146]: cluster 2024-03-11T21:02:28.403343+0000 osd.6 (osd.6) 174 : cluster 0 2.14 deep-scrub starts 2024-03-11T21:02:29.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:29 smithi023 bash[29146]: cluster 2024-03-11T21:02:28.404991+0000 osd.6 (osd.6) 175 : cluster 0 2.14 deep-scrub ok 2024-03-11T21:02:29.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:29 smithi023 bash[29146]: cluster 2024-03-11T21:02:28.621651+0000 osd.2 (osd.2) 158 : cluster 0 purged_snaps scrub starts 2024-03-11T21:02:29.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:29 smithi023 bash[29146]: cluster 2024-03-11T21:02:28.621762+0000 osd.2 (osd.2) 159 : cluster 0 purged_snaps scrub ok 2024-03-11T21:02:29.539 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:02:29.560 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:02:29.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:29.562 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:02:29.662 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:02:29.689 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:02:29.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:29.692 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:02:29.792 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:02:29.821 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:02:29.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:29.823 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:02:29.924 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:29.952 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:29.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:29.954 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:02:30.055 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:02:30.081 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:02:30.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:30.083 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:02:30.184 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:30.215 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:30.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:30.218 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:02:30.318 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:02:30.342 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:02:30.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:30.344 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:02:30.395 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-11T21:02:30.395 INFO:teuthology.orchestra.run.smithi023.stdout: "size": 20, 2024-03-11T21:02:30.395 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 600, 2024-03-11T21:02:30.395 INFO:teuthology.orchestra.run.smithi023.stdout: "ops": [ 2024-03-11T21:02:30.396 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.396 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_op(client.24629.0:46652 7.a 7:531a34cf:::benchmark_data_smithi023_67959_object46651:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-03-11T21:02:30.396 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T20:55:16.107764+0000", 2024-03-11T21:02:30.396 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 434.28814399499998, 2024-03-11T21:02:30.396 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.33282781900000002, 2024-03-11T21:02:30.396 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:02:30.396 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:02:30.396 INFO:teuthology.orchestra.run.smithi023.stdout: "client_info": { 2024-03-11T21:02:30.396 INFO:teuthology.orchestra.run.smithi023.stdout: "client": "client.24629", 2024-03-11T21:02:30.396 INFO:teuthology.orchestra.run.smithi023.stdout: "client_addr": "172.21.15.23:0/959732609", 2024-03-11T21:02:30.396 INFO:teuthology.orchestra.run.smithi023.stdout: "tid": 46652 2024-03-11T21:02:30.396 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.396 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:02:30.396 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.397 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:02:30.397 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:16.107764+0000", 2024-03-11T21:02:30.397 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:02:30.397 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.397 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.397 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:02:30.397 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:16.107764+0000", 2024-03-11T21:02:30.397 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:02:30.397 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.397 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.397 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:02:30.397 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:16.107766+0000", 2024-03-11T21:02:30.397 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.342e-06 2024-03-11T21:02:30.397 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.397 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.397 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:02:30.398 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:16.107788+0000", 2024-03-11T21:02:30.398 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.2385e-05 2024-03-11T21:02:30.398 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.398 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.398 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:02:30.398 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:16.107801+0000", 2024-03-11T21:02:30.398 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.272e-05 2024-03-11T21:02:30.398 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.398 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.398 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:02:30.398 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:16.107806+0000", 2024-03-11T21:02:30.398 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 5.6189999999999999e-06 2024-03-11T21:02:30.399 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.399 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.399 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:02:30.399 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:16.108843+0000", 2024-03-11T21:02:30.399 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0010366520000000001 2024-03-11T21:02:30.399 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.399 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.399 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:02:30.399 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:16.108980+0000", 2024-03-11T21:02:30.399 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00013694299999999999 2024-03-11T21:02:30.399 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.399 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.399 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for subops from 2,5", 2024-03-11T21:02:30.399 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:16.109135+0000", 2024-03-11T21:02:30.399 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00015526900000000001 2024-03-11T21:02:30.400 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.400 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.400 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "op_commit", 2024-03-11T21:02:30.400 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:16.111103+0000", 2024-03-11T21:02:30.400 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0019676049999999999 2024-03-11T21:02:30.400 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.400 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.400 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:02:30.400 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:16.111281+0000", 2024-03-11T21:02:30.400 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00017801500000000001 2024-03-11T21:02:30.400 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.400 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.400 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:02:30.400 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:16.439290+0000", 2024-03-11T21:02:30.400 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.32800967199999997 2024-03-11T21:02:30.400 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.400 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.401 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T21:02:30.401 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:16.440031+0000", 2024-03-11T21:02:30.401 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00074045000000000003 2024-03-11T21:02:30.401 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.401 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.401 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:02:30.401 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:16.440592+0000", 2024-03-11T21:02:30.401 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00056114700000000001 2024-03-11T21:02:30.401 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:02:30.401 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:02:30.401 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:02:30.401 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.401 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.401 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_op(client.24629.0:85826 7.3 7:c712866b:::benchmark_data_smithi023_67959_object85825:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-03-11T21:02:30.401 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T20:55:52.509201+0000", 2024-03-11T21:02:30.402 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 397.88670694000001, 2024-03-11T21:02:30.402 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.41218749300000002, 2024-03-11T21:02:30.402 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:02:30.402 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:02:30.402 INFO:teuthology.orchestra.run.smithi023.stdout: "client_info": { 2024-03-11T21:02:30.402 INFO:teuthology.orchestra.run.smithi023.stdout: "client": "client.24629", 2024-03-11T21:02:30.402 INFO:teuthology.orchestra.run.smithi023.stdout: "client_addr": "172.21.15.23:0/959732609", 2024-03-11T21:02:30.402 INFO:teuthology.orchestra.run.smithi023.stdout: "tid": 85826 2024-03-11T21:02:30.402 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.402 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:02:30.402 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.402 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:02:30.402 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.509201+0000", 2024-03-11T21:02:30.402 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:02:30.403 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.403 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.403 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:02:30.403 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.509201+0000", 2024-03-11T21:02:30.403 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:02:30.403 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.403 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.403 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:02:30.403 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.509202+0000", 2024-03-11T21:02:30.403 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.1039999999999999e-06 2024-03-11T21:02:30.403 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.403 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.403 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:02:30.403 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.509219+0000", 2024-03-11T21:02:30.403 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.6192999999999999e-05 2024-03-11T21:02:30.403 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.404 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.404 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:02:30.404 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.509228+0000", 2024-03-11T21:02:30.404 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 9.2459999999999992e-06 2024-03-11T21:02:30.404 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.404 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.404 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:02:30.404 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.509233+0000", 2024-03-11T21:02:30.404 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 5.4040000000000003e-06 2024-03-11T21:02:30.404 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.404 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.404 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:02:30.404 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.509286+0000", 2024-03-11T21:02:30.404 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 5.2339000000000002e-05 2024-03-11T21:02:30.404 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.405 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.405 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:02:30.405 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.509407+0000", 2024-03-11T21:02:30.405 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00012103 2024-03-11T21:02:30.405 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.405 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.405 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for subops from 2,4", 2024-03-11T21:02:30.405 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.905827+0000", 2024-03-11T21:02:30.405 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.396419931 2024-03-11T21:02:30.405 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.405 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.405 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:02:30.405 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.920003+0000", 2024-03-11T21:02:30.405 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.014176393000000001 2024-03-11T21:02:30.405 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.405 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.405 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "op_commit", 2024-03-11T21:02:30.406 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.920186+0000", 2024-03-11T21:02:30.406 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00018330500000000001 2024-03-11T21:02:30.406 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.406 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.406 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:02:30.406 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.921229+0000", 2024-03-11T21:02:30.406 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0010428049999999999 2024-03-11T21:02:30.406 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.406 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.406 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T21:02:30.406 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.921332+0000", 2024-03-11T21:02:30.406 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00010282599999999999 2024-03-11T21:02:30.406 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.406 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.407 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:02:30.407 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.921389+0000", 2024-03-11T21:02:30.407 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 5.6916999999999997e-05 2024-03-11T21:02:30.407 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:02:30.407 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:02:30.407 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:02:30.407 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.407 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.407 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_repop(client.24629.0:85823 7.c e145/121 7:3f3c9d65:::benchmark_data_smithi023_67959_object85822:head v 145'5263, mlcod=145'5261)", 2024-03-11T21:02:30.407 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T20:55:52.509289+0000", 2024-03-11T21:02:30.407 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 397.88661889399998, 2024-03-11T21:02:30.407 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.39653492800000001, 2024-03-11T21:02:30.407 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:02:30.407 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:02:30.407 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:02:30.407 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.408 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:02:30.408 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.509289+0000", 2024-03-11T21:02:30.408 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:02:30.408 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.408 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.408 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:02:30.408 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.509289+0000", 2024-03-11T21:02:30.408 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:02:30.408 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.408 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.408 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:02:30.408 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.509290+0000", 2024-03-11T21:02:30.408 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.9999999999999996e-07 2024-03-11T21:02:30.408 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.408 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.408 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:02:30.409 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.509334+0000", 2024-03-11T21:02:30.409 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 4.35e-05 2024-03-11T21:02:30.409 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.409 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.409 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:02:30.409 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.509342+0000", 2024-03-11T21:02:30.409 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.9799999999999998e-06 2024-03-11T21:02:30.409 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.409 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.409 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:02:30.409 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.509346+0000", 2024-03-11T21:02:30.409 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 4.5800000000000002e-06 2024-03-11T21:02:30.409 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.409 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.409 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:02:30.409 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.509389+0000", 2024-03-11T21:02:30.410 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 4.2475999999999998e-05 2024-03-11T21:02:30.410 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.410 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.410 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:02:30.410 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.509402+0000", 2024-03-11T21:02:30.410 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.3222e-05 2024-03-11T21:02:30.410 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.410 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.410 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T21:02:30.410 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.905756+0000", 2024-03-11T21:02:30.410 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.39635374699999998 2024-03-11T21:02:30.410 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.410 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.410 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:02:30.410 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.905824+0000", 2024-03-11T21:02:30.410 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.8622999999999997e-05 2024-03-11T21:02:30.411 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:02:30.411 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:02:30.411 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:02:30.411 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.411 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.411 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_repop(client.24629.0:85827 7.0 e145/124 7:0ef91f2b:::benchmark_data_smithi023_67959_object85826:head v 145'5268, mlcod=145'5266)", 2024-03-11T21:02:30.411 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T20:55:52.513894+0000", 2024-03-11T21:02:30.411 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 397.88201425, 2024-03-11T21:02:30.411 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.40481390499999997, 2024-03-11T21:02:30.411 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:02:30.411 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:02:30.411 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:02:30.411 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.411 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:02:30.411 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.513894+0000", 2024-03-11T21:02:30.412 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:02:30.412 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.412 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.412 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:02:30.412 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.513894+0000", 2024-03-11T21:02:30.412 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:02:30.412 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.412 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.412 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:02:30.412 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.513895+0000", 2024-03-11T21:02:30.412 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.049e-06 2024-03-11T21:02:30.412 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.412 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.412 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:02:30.412 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.513945+0000", 2024-03-11T21:02:30.413 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 5.0083999999999997e-05 2024-03-11T21:02:30.413 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.413 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.413 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:02:30.413 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.905665+0000", 2024-03-11T21:02:30.413 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.39171999499999999 2024-03-11T21:02:30.413 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.413 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.413 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:02:30.413 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.905689+0000", 2024-03-11T21:02:30.413 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.3745999999999999e-05 2024-03-11T21:02:30.413 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.413 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.413 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:02:30.413 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.905838+0000", 2024-03-11T21:02:30.413 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00014893099999999999 2024-03-11T21:02:30.414 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.414 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.414 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:02:30.414 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.905881+0000", 2024-03-11T21:02:30.414 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 4.2766999999999999e-05 2024-03-11T21:02:30.414 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.414 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.414 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T21:02:30.414 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.918659+0000", 2024-03-11T21:02:30.414 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.012777973 2024-03-11T21:02:30.414 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.414 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.414 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:02:30.415 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.918708+0000", 2024-03-11T21:02:30.415 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 4.9360000000000002e-05 2024-03-11T21:02:30.415 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:02:30.415 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:02:30.415 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:02:30.415 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.415 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.415 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_op(client.24629.0:85831 7.2 7:4fbf6237:::benchmark_data_smithi023_67959_object85830:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-03-11T21:02:30.415 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T20:55:52.527890+0000", 2024-03-11T21:02:30.415 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 397.86801794799999, 2024-03-11T21:02:30.415 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.39378672599999998, 2024-03-11T21:02:30.415 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:02:30.415 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:02:30.415 INFO:teuthology.orchestra.run.smithi023.stdout: "client_info": { 2024-03-11T21:02:30.415 INFO:teuthology.orchestra.run.smithi023.stdout: "client": "client.24629", 2024-03-11T21:02:30.416 INFO:teuthology.orchestra.run.smithi023.stdout: "client_addr": "172.21.15.23:0/959732609", 2024-03-11T21:02:30.416 INFO:teuthology.orchestra.run.smithi023.stdout: "tid": 85831 2024-03-11T21:02:30.416 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.416 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:02:30.416 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.416 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:02:30.416 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.527890+0000", 2024-03-11T21:02:30.416 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:02:30.416 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.416 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.416 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:02:30.416 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.527890+0000", 2024-03-11T21:02:30.416 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:02:30.416 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.416 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.417 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:02:30.417 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.527892+0000", 2024-03-11T21:02:30.417 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.3570000000000001e-06 2024-03-11T21:02:30.417 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.417 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.417 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:02:30.417 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.527913+0000", 2024-03-11T21:02:30.417 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.1311e-05 2024-03-11T21:02:30.417 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.417 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.417 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:02:30.417 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.905662+0000", 2024-03-11T21:02:30.417 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.37774891700000002 2024-03-11T21:02:30.417 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.417 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.417 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:02:30.418 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.905691+0000", 2024-03-11T21:02:30.418 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.9515000000000001e-05 2024-03-11T21:02:30.418 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.418 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.418 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:02:30.418 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.905885+0000", 2024-03-11T21:02:30.418 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00019332400000000001 2024-03-11T21:02:30.418 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.418 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.418 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:02:30.418 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.906083+0000", 2024-03-11T21:02:30.418 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000198572 2024-03-11T21:02:30.418 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.418 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.418 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for subops from 2,7", 2024-03-11T21:02:30.419 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.906303+0000", 2024-03-11T21:02:30.419 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00021959 2024-03-11T21:02:30.419 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.419 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.419 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:02:30.419 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.912014+0000", 2024-03-11T21:02:30.419 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0057112049999999996 2024-03-11T21:02:30.419 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.419 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.419 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "op_commit", 2024-03-11T21:02:30.419 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.918631+0000", 2024-03-11T21:02:30.419 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0066167470000000001 2024-03-11T21:02:30.419 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.419 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.419 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:02:30.419 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.921223+0000", 2024-03-11T21:02:30.420 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0025917789999999998 2024-03-11T21:02:30.420 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.420 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.420 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T21:02:30.420 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.921614+0000", 2024-03-11T21:02:30.420 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00039175700000000001 2024-03-11T21:02:30.420 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.420 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.420 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:02:30.420 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.921677+0000", 2024-03-11T21:02:30.420 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.2651999999999996e-05 2024-03-11T21:02:30.420 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:02:30.420 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:02:30.421 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:02:30.421 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.421 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.421 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_repop(client.24629.0:85833 7.d e145/124 7:bfb2a6d1:::benchmark_data_smithi023_67959_object85832:head v 145'5379, mlcod=145'5377)", 2024-03-11T21:02:30.421 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T20:55:52.528715+0000", 2024-03-11T21:02:30.421 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 397.867192902, 2024-03-11T21:02:30.421 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.39186654100000001, 2024-03-11T21:02:30.421 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:02:30.421 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:02:30.421 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:02:30.421 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.421 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:02:30.421 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.528715+0000", 2024-03-11T21:02:30.421 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:02:30.421 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.421 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.422 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:02:30.422 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.528715+0000", 2024-03-11T21:02:30.422 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:02:30.422 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.422 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.422 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:02:30.422 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.528716+0000", 2024-03-11T21:02:30.422 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 8.1399999999999996e-07 2024-03-11T21:02:30.422 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.422 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.422 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:02:30.422 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.528760+0000", 2024-03-11T21:02:30.422 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 4.3809e-05 2024-03-11T21:02:30.422 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.423 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.423 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:02:30.423 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.905687+0000", 2024-03-11T21:02:30.423 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.37692657099999999 2024-03-11T21:02:30.423 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.423 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.423 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:02:30.423 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.906752+0000", 2024-03-11T21:02:30.423 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0010655759999999999 2024-03-11T21:02:30.423 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.423 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.423 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:02:30.423 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.907542+0000", 2024-03-11T21:02:30.423 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00078997799999999999 2024-03-11T21:02:30.423 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.423 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.424 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:02:30.424 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.907566+0000", 2024-03-11T21:02:30.424 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.4051999999999999e-05 2024-03-11T21:02:30.424 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.424 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.424 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T21:02:30.424 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.920464+0000", 2024-03-11T21:02:30.424 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.012897764000000001 2024-03-11T21:02:30.424 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.424 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.424 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:02:30.424 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:55:52.920582+0000", 2024-03-11T21:02:30.424 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000117977 2024-03-11T21:02:30.424 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:02:30.424 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:02:30.425 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:02:30.425 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.425 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.425 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_op(client.24629.0:126332 7.a 7:5f290de1:::benchmark_data_smithi023_67959_object16091:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-03-11T21:02:30.425 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T20:56:39.175300+0000", 2024-03-11T21:02:30.425 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 351.22060821999997, 2024-03-11T21:02:30.425 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.44305815700000001, 2024-03-11T21:02:30.425 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:02:30.425 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:02:30.425 INFO:teuthology.orchestra.run.smithi023.stdout: "client_info": { 2024-03-11T21:02:30.425 INFO:teuthology.orchestra.run.smithi023.stdout: "client": "client.24629", 2024-03-11T21:02:30.425 INFO:teuthology.orchestra.run.smithi023.stdout: "client_addr": "172.21.15.23:0/959732609", 2024-03-11T21:02:30.425 INFO:teuthology.orchestra.run.smithi023.stdout: "tid": 126332 2024-03-11T21:02:30.425 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.425 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:02:30.426 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.426 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:02:30.426 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.175300+0000", 2024-03-11T21:02:30.426 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:02:30.426 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.426 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.426 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:02:30.426 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.175300+0000", 2024-03-11T21:02:30.426 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:02:30.426 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.426 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.426 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:02:30.426 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.175300+0000", 2024-03-11T21:02:30.427 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 3.1e-07 2024-03-11T21:02:30.427 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.427 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.427 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:02:30.427 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.175302+0000", 2024-03-11T21:02:30.427 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.0420000000000001e-06 2024-03-11T21:02:30.427 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.427 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.427 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:02:30.427 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.175307+0000", 2024-03-11T21:02:30.427 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 4.3889999999999998e-06 2024-03-11T21:02:30.427 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.427 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.427 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:02:30.427 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.175310+0000", 2024-03-11T21:02:30.428 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 3.2440000000000002e-06 2024-03-11T21:02:30.428 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.428 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.428 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:02:30.428 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.175362+0000", 2024-03-11T21:02:30.428 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 5.2441999999999997e-05 2024-03-11T21:02:30.428 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.428 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.428 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:02:30.428 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.175462+0000", 2024-03-11T21:02:30.428 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 9.9955000000000001e-05 2024-03-11T21:02:30.428 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.428 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.428 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for subops from 2,5", 2024-03-11T21:02:30.428 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.175532+0000", 2024-03-11T21:02:30.428 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.9468000000000006e-05 2024-03-11T21:02:30.428 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.429 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.429 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "op_commit", 2024-03-11T21:02:30.429 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.178041+0000", 2024-03-11T21:02:30.429 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0025092420000000001 2024-03-11T21:02:30.429 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.429 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.429 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:02:30.429 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.184252+0000", 2024-03-11T21:02:30.429 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0062105429999999998 2024-03-11T21:02:30.429 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.429 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.429 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:02:30.429 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.617826+0000", 2024-03-11T21:02:30.429 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.43357458500000001 2024-03-11T21:02:30.429 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.429 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.429 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T21:02:30.430 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.618275+0000", 2024-03-11T21:02:30.430 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00044847099999999998 2024-03-11T21:02:30.430 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.430 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.430 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:02:30.430 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.618358+0000", 2024-03-11T21:02:30.430 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 8.3466000000000004e-05 2024-03-11T21:02:30.430 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:02:30.430 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:02:30.430 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:02:30.430 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.430 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.430 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_op(client.24629.0:126334 7.2 7:40a0664e:::benchmark_data_smithi023_67959_object16093:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-03-11T21:02:30.430 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T20:56:39.175323+0000", 2024-03-11T21:02:30.430 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 351.22058485399998, 2024-03-11T21:02:30.431 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.44290166199999997, 2024-03-11T21:02:30.431 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:02:30.431 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:02:30.431 INFO:teuthology.orchestra.run.smithi023.stdout: "client_info": { 2024-03-11T21:02:30.431 INFO:teuthology.orchestra.run.smithi023.stdout: "client": "client.24629", 2024-03-11T21:02:30.431 INFO:teuthology.orchestra.run.smithi023.stdout: "client_addr": "172.21.15.23:0/959732609", 2024-03-11T21:02:30.431 INFO:teuthology.orchestra.run.smithi023.stdout: "tid": 126334 2024-03-11T21:02:30.431 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.431 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:02:30.431 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.431 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:02:30.431 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.175323+0000", 2024-03-11T21:02:30.431 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:02:30.431 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.431 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.432 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:02:30.432 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.175323+0000", 2024-03-11T21:02:30.432 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:02:30.432 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.432 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.432 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:02:30.432 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.175324+0000", 2024-03-11T21:02:30.432 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.7700000000000001e-07 2024-03-11T21:02:30.432 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.432 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.432 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:02:30.432 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.175325+0000", 2024-03-11T21:02:30.432 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.84e-06 2024-03-11T21:02:30.432 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.432 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.433 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:02:30.433 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.175329+0000", 2024-03-11T21:02:30.433 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 3.5410000000000001e-06 2024-03-11T21:02:30.433 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.433 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.433 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:02:30.433 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.175332+0000", 2024-03-11T21:02:30.433 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 3.1070000000000001e-06 2024-03-11T21:02:30.433 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.433 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.433 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:02:30.433 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.176183+0000", 2024-03-11T21:02:30.433 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00085044600000000004 2024-03-11T21:02:30.433 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.433 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.434 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:02:30.434 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.176286+0000", 2024-03-11T21:02:30.434 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000103442 2024-03-11T21:02:30.434 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.434 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.434 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for subops from 2,7", 2024-03-11T21:02:30.434 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.176394+0000", 2024-03-11T21:02:30.434 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000108007 2024-03-11T21:02:30.434 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.434 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.434 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "op_commit", 2024-03-11T21:02:30.434 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.178567+0000", 2024-03-11T21:02:30.434 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0021733519999999999 2024-03-11T21:02:30.434 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.435 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.435 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:02:30.435 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.179871+0000", 2024-03-11T21:02:30.435 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.001303207 2024-03-11T21:02:30.435 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.435 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.435 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:02:30.435 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.618044+0000", 2024-03-11T21:02:30.435 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.43817315699999998 2024-03-11T21:02:30.435 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.435 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.435 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T21:02:30.435 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.618171+0000", 2024-03-11T21:02:30.435 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00012737 2024-03-11T21:02:30.436 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.436 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.436 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:02:30.436 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.618225+0000", 2024-03-11T21:02:30.436 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 5.3915999999999997e-05 2024-03-11T21:02:30.436 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:02:30.436 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:02:30.436 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:02:30.436 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.436 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.436 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_op(client.24629.0:126340 7.2 7:41534ba6:::benchmark_data_smithi023_67959_object16099:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-03-11T21:02:30.436 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T20:56:39.179515+0000", 2024-03-11T21:02:30.436 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 351.21639314200002, 2024-03-11T21:02:30.436 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.43929723700000001, 2024-03-11T21:02:30.436 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:02:30.437 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:02:30.437 INFO:teuthology.orchestra.run.smithi023.stdout: "client_info": { 2024-03-11T21:02:30.437 INFO:teuthology.orchestra.run.smithi023.stdout: "client": "client.24629", 2024-03-11T21:02:30.437 INFO:teuthology.orchestra.run.smithi023.stdout: "client_addr": "172.21.15.23:0/959732609", 2024-03-11T21:02:30.437 INFO:teuthology.orchestra.run.smithi023.stdout: "tid": 126340 2024-03-11T21:02:30.437 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.437 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:02:30.437 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.437 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:02:30.437 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.179515+0000", 2024-03-11T21:02:30.437 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:02:30.437 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.437 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.437 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:02:30.437 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.179515+0000", 2024-03-11T21:02:30.438 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:02:30.438 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.438 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.438 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:02:30.438 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.179516+0000", 2024-03-11T21:02:30.438 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 4.5400000000000002e-07 2024-03-11T21:02:30.438 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.438 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.438 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:02:30.438 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.179519+0000", 2024-03-11T21:02:30.438 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 3.258e-06 2024-03-11T21:02:30.438 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.438 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.438 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:02:30.438 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.179527+0000", 2024-03-11T21:02:30.438 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.7800000000000001e-06 2024-03-11T21:02:30.439 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.439 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.439 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:02:30.439 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.179533+0000", 2024-03-11T21:02:30.439 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.0469999999999999e-06 2024-03-11T21:02:30.439 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.439 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.439 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:02:30.439 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.179914+0000", 2024-03-11T21:02:30.439 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00038138700000000001 2024-03-11T21:02:30.439 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.439 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.439 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:02:30.439 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.179987+0000", 2024-03-11T21:02:30.439 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.3115999999999996e-05 2024-03-11T21:02:30.439 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.439 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.440 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for subops from 2,7", 2024-03-11T21:02:30.440 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.182805+0000", 2024-03-11T21:02:30.440 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0028174749999999998 2024-03-11T21:02:30.440 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.440 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.440 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "op_commit", 2024-03-11T21:02:30.440 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.185080+0000", 2024-03-11T21:02:30.440 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0022755219999999999 2024-03-11T21:02:30.440 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.440 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.440 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:02:30.440 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.188917+0000", 2024-03-11T21:02:30.440 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0038373470000000001 2024-03-11T21:02:30.440 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.440 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.440 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:02:30.440 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.618453+0000", 2024-03-11T21:02:30.441 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.429535746 2024-03-11T21:02:30.441 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.441 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.441 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T21:02:30.441 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.618636+0000", 2024-03-11T21:02:30.441 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00018320000000000001 2024-03-11T21:02:30.441 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.441 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.441 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:02:30.441 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.618812+0000", 2024-03-11T21:02:30.441 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00017590499999999999 2024-03-11T21:02:30.441 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:02:30.441 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:02:30.441 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:02:30.441 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.441 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.441 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_op(client.24629.0:126341 7.a 7:5cab8f84:::benchmark_data_smithi023_67959_object16100:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-03-11T21:02:30.442 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T20:56:39.179749+0000", 2024-03-11T21:02:30.442 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 351.21615924399998, 2024-03-11T21:02:30.442 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.43930538099999999, 2024-03-11T21:02:30.442 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:02:30.442 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:02:30.442 INFO:teuthology.orchestra.run.smithi023.stdout: "client_info": { 2024-03-11T21:02:30.442 INFO:teuthology.orchestra.run.smithi023.stdout: "client": "client.24629", 2024-03-11T21:02:30.442 INFO:teuthology.orchestra.run.smithi023.stdout: "client_addr": "172.21.15.23:0/959732609", 2024-03-11T21:02:30.442 INFO:teuthology.orchestra.run.smithi023.stdout: "tid": 126341 2024-03-11T21:02:30.442 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.442 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:02:30.442 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.442 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:02:30.442 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.179749+0000", 2024-03-11T21:02:30.443 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:02:30.443 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.443 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.443 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:02:30.443 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.179749+0000", 2024-03-11T21:02:30.443 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:02:30.443 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.443 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.443 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:02:30.443 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.179750+0000", 2024-03-11T21:02:30.443 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.3200000000000005e-07 2024-03-11T21:02:30.443 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.443 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.443 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:02:30.443 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.179753+0000", 2024-03-11T21:02:30.444 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 3.6899999999999998e-06 2024-03-11T21:02:30.444 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.444 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.444 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:02:30.444 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.179762+0000", 2024-03-11T21:02:30.444 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 9.098e-06 2024-03-11T21:02:30.444 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.444 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.444 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:02:30.444 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.179769+0000", 2024-03-11T21:02:30.444 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.4110000000000001e-06 2024-03-11T21:02:30.444 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.444 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.444 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:02:30.444 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.179925+0000", 2024-03-11T21:02:30.445 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00015657900000000001 2024-03-11T21:02:30.445 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.445 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.445 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:02:30.445 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.180029+0000", 2024-03-11T21:02:30.445 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000104063 2024-03-11T21:02:30.445 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.445 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.445 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for subops from 2,5", 2024-03-11T21:02:30.445 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.180140+0000", 2024-03-11T21:02:30.445 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00011100899999999999 2024-03-11T21:02:30.445 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.445 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.446 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:02:30.446 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.184863+0000", 2024-03-11T21:02:30.446 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0047221269999999996 2024-03-11T21:02:30.446 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.446 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.446 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "op_commit", 2024-03-11T21:02:30.446 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.184874+0000", 2024-03-11T21:02:30.446 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.149e-05 2024-03-11T21:02:30.446 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.446 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.446 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:02:30.446 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.618759+0000", 2024-03-11T21:02:30.446 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.433884724 2024-03-11T21:02:30.446 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.447 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.447 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T21:02:30.447 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.618923+0000", 2024-03-11T21:02:30.447 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000164186 2024-03-11T21:02:30.447 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.447 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.447 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:02:30.447 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.619054+0000", 2024-03-11T21:02:30.447 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00013137199999999999 2024-03-11T21:02:30.447 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:02:30.447 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:02:30.447 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:02:30.447 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.447 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.447 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_op(client.24629.0:126362 7.3 7:cbd5d24b:::benchmark_data_smithi023_67959_object16121:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-03-11T21:02:30.448 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T20:56:39.200356+0000", 2024-03-11T21:02:30.448 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 351.19555207100001, 2024-03-11T21:02:30.448 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.44075034099999999, 2024-03-11T21:02:30.448 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:02:30.448 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:02:30.448 INFO:teuthology.orchestra.run.smithi023.stdout: "client_info": { 2024-03-11T21:02:30.448 INFO:teuthology.orchestra.run.smithi023.stdout: "client": "client.24629", 2024-03-11T21:02:30.448 INFO:teuthology.orchestra.run.smithi023.stdout: "client_addr": "172.21.15.23:0/959732609", 2024-03-11T21:02:30.448 INFO:teuthology.orchestra.run.smithi023.stdout: "tid": 126362 2024-03-11T21:02:30.448 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.448 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:02:30.448 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.448 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:02:30.448 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.200356+0000", 2024-03-11T21:02:30.448 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:02:30.448 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.449 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.449 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:02:30.449 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.200356+0000", 2024-03-11T21:02:30.449 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:02:30.449 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.449 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.449 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:02:30.449 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.200357+0000", 2024-03-11T21:02:30.449 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.2979999999999999e-06 2024-03-11T21:02:30.449 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.449 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.449 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:02:30.449 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.200363+0000", 2024-03-11T21:02:30.449 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.054e-06 2024-03-11T21:02:30.449 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.450 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.450 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:02:30.450 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.332575+0000", 2024-03-11T21:02:30.450 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.13221116899999999 2024-03-11T21:02:30.450 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.450 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.450 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:02:30.450 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.332608+0000", 2024-03-11T21:02:30.450 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 3.3596999999999997e-05 2024-03-11T21:02:30.450 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.450 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.450 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:02:30.450 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.332770+0000", 2024-03-11T21:02:30.450 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00016126899999999999 2024-03-11T21:02:30.451 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.451 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.451 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:02:30.451 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.333108+0000", 2024-03-11T21:02:30.451 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000338387 2024-03-11T21:02:30.451 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.451 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.451 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for subops from 2,4", 2024-03-11T21:02:30.451 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.333417+0000", 2024-03-11T21:02:30.451 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00030934400000000003 2024-03-11T21:02:30.451 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.451 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.451 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:02:30.451 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.341605+0000", 2024-03-11T21:02:30.451 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0081881609999999994 2024-03-11T21:02:30.451 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.452 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.452 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "op_commit", 2024-03-11T21:02:30.452 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.341632+0000", 2024-03-11T21:02:30.452 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.6917999999999999e-05 2024-03-11T21:02:30.452 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.452 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.452 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:02:30.452 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.640968+0000", 2024-03-11T21:02:30.452 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.29933543400000001 2024-03-11T21:02:30.452 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.452 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.452 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T21:02:30.452 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.641046+0000", 2024-03-11T21:02:30.452 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.8484000000000003e-05 2024-03-11T21:02:30.452 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.453 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.453 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:02:30.453 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:56:39.641106+0000", 2024-03-11T21:02:30.453 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.0226000000000002e-05 2024-03-11T21:02:30.453 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:02:30.453 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:02:30.453 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:02:30.453 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.453 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.453 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_op(client.24406.0:385 5.0 5:353ec8f9:::gateway.conf:head [getxattr epoch (1) in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e161)", 2024-03-11T21:02:30.453 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T20:57:23.587375+0000", 2024-03-11T21:02:30.453 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 306.80853328299997, 2024-03-11T21:02:30.453 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.0496242819999999, 2024-03-11T21:02:30.453 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:02:30.453 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "started", 2024-03-11T21:02:30.454 INFO:teuthology.orchestra.run.smithi023.stdout: "client_info": { 2024-03-11T21:02:30.454 INFO:teuthology.orchestra.run.smithi023.stdout: "client": "client.24406", 2024-03-11T21:02:30.454 INFO:teuthology.orchestra.run.smithi023.stdout: "client_addr": "172.21.15.146:0/3388716841", 2024-03-11T21:02:30.454 INFO:teuthology.orchestra.run.smithi023.stdout: "tid": 385 2024-03-11T21:02:30.454 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.454 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:02:30.454 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.454 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:02:30.454 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:23.587375+0000", 2024-03-11T21:02:30.454 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:02:30.454 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.454 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.454 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:02:30.454 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:23.587375+0000", 2024-03-11T21:02:30.455 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:02:30.455 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.455 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.455 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:02:30.455 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:23.587377+0000", 2024-03-11T21:02:30.455 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.6729999999999999e-06 2024-03-11T21:02:30.455 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.455 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.455 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:02:30.455 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:23.587382+0000", 2024-03-11T21:02:30.455 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 5.3580000000000002e-06 2024-03-11T21:02:30.455 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.455 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.455 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:02:30.455 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:23.587393+0000", 2024-03-11T21:02:30.456 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.0808000000000001e-05 2024-03-11T21:02:30.456 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.456 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.456 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:02:30.456 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:23.587403+0000", 2024-03-11T21:02:30.456 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 9.9059999999999997e-06 2024-03-11T21:02:30.456 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.456 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.456 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:02:30.456 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:23.614080+0000", 2024-03-11T21:02:30.456 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.026676998 2024-03-11T21:02:30.456 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.456 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.456 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for peered", 2024-03-11T21:02:30.456 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:23.614090+0000", 2024-03-11T21:02:30.457 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.0742e-05 2024-03-11T21:02:30.457 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.457 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.457 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:02:30.457 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:24.636718+0000", 2024-03-11T21:02:30.457 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.0226275650000001 2024-03-11T21:02:30.457 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.457 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.457 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:02:30.457 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:24.636854+0000", 2024-03-11T21:02:30.457 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00013643800000000001 2024-03-11T21:02:30.457 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.457 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.457 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:02:30.457 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:24.636999+0000", 2024-03-11T21:02:30.457 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00014479400000000001 2024-03-11T21:02:30.458 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:02:30.458 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:02:30.458 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:02:30.458 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.458 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.458 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_op(client.24281.0:1739 4.14 4:2b04a3e9:::notify.6:head [watch reconnect cookie 94110942021888 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e162)", 2024-03-11T21:02:30.458 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T20:57:23.591846+0000", 2024-03-11T21:02:30.458 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 306.80406251300002, 2024-03-11T21:02:30.458 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.075694433, 2024-03-11T21:02:30.458 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:02:30.458 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "started", 2024-03-11T21:02:30.458 INFO:teuthology.orchestra.run.smithi023.stdout: "client_info": { 2024-03-11T21:02:30.458 INFO:teuthology.orchestra.run.smithi023.stdout: "client": "client.24281", 2024-03-11T21:02:30.458 INFO:teuthology.orchestra.run.smithi023.stdout: "client_addr": "172.21.15.23:0/2185274774", 2024-03-11T21:02:30.458 INFO:teuthology.orchestra.run.smithi023.stdout: "tid": 1739 2024-03-11T21:02:30.459 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.459 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:02:30.459 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.459 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:02:30.459 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:23.591846+0000", 2024-03-11T21:02:30.459 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:02:30.459 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.459 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.459 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:02:30.459 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:23.591846+0000", 2024-03-11T21:02:30.459 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:02:30.459 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.459 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.459 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:02:30.459 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:23.591847+0000", 2024-03-11T21:02:30.459 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.0759999999999999e-06 2024-03-11T21:02:30.460 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.460 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.460 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:02:30.460 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:23.591854+0000", 2024-03-11T21:02:30.460 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.9539999999999998e-06 2024-03-11T21:02:30.460 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.460 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.460 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:02:30.460 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:23.591862+0000", 2024-03-11T21:02:30.460 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.9859999999999994e-06 2024-03-11T21:02:30.460 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.460 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.460 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:02:30.460 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:23.591868+0000", 2024-03-11T21:02:30.460 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.4019999999999999e-06 2024-03-11T21:02:30.461 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.461 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.461 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:02:30.461 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:23.612635+0000", 2024-03-11T21:02:30.461 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.020766413000000001 2024-03-11T21:02:30.461 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.461 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.461 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for peered", 2024-03-11T21:02:30.461 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:23.612646+0000", 2024-03-11T21:02:30.461 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.1812e-05 2024-03-11T21:02:30.461 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.461 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.461 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:02:30.461 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:24.639873+0000", 2024-03-11T21:02:30.461 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.0272262089999999 2024-03-11T21:02:30.462 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.462 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.462 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for degraded object", 2024-03-11T21:02:30.462 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:24.640210+0000", 2024-03-11T21:02:30.462 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00033792300000000001 2024-03-11T21:02:30.462 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.462 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.462 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:02:30.462 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:24.655501+0000", 2024-03-11T21:02:30.462 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.015290870999999999 2024-03-11T21:02:30.462 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.462 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.462 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:02:30.462 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:24.655589+0000", 2024-03-11T21:02:30.462 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 8.7645000000000005e-05 2024-03-11T21:02:30.463 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.463 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.463 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:02:30.463 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:24.667540+0000", 2024-03-11T21:02:30.463 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.011951142 2024-03-11T21:02:30.463 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:02:30.463 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:02:30.463 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:02:30.463 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.463 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.463 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_op(client.24629.0:167632 7.2 7:4fc417d5:::benchmark_data_smithi023_67959_object57391:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e162)", 2024-03-11T21:02:30.463 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T20:57:23.610529+0000", 2024-03-11T21:02:30.463 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 306.78537946, 2024-03-11T21:02:30.463 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.1886766679999998, 2024-03-11T21:02:30.464 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:02:30.464 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:02:30.464 INFO:teuthology.orchestra.run.smithi023.stdout: "client_info": { 2024-03-11T21:02:30.464 INFO:teuthology.orchestra.run.smithi023.stdout: "client": "client.24629", 2024-03-11T21:02:30.464 INFO:teuthology.orchestra.run.smithi023.stdout: "client_addr": "172.21.15.23:0/959732609", 2024-03-11T21:02:30.464 INFO:teuthology.orchestra.run.smithi023.stdout: "tid": 167632 2024-03-11T21:02:30.464 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.464 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:02:30.464 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.464 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:02:30.464 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:23.610529+0000", 2024-03-11T21:02:30.464 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:02:30.464 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.464 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.464 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:02:30.465 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:23.610529+0000", 2024-03-11T21:02:30.465 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:02:30.465 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.465 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.465 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:02:30.465 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:23.610530+0000", 2024-03-11T21:02:30.465 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.13e-06 2024-03-11T21:02:30.465 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.465 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.465 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:02:30.465 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:23.610534+0000", 2024-03-11T21:02:30.465 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 4.4190000000000004e-06 2024-03-11T21:02:30.465 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.465 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.466 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:02:30.466 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:23.610543+0000", 2024-03-11T21:02:30.466 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 8.7199999999999995e-06 2024-03-11T21:02:30.466 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.466 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.466 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:02:30.466 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:23.610551+0000", 2024-03-11T21:02:30.466 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.4889999999999998e-06 2024-03-11T21:02:30.466 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.466 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.466 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:02:30.466 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:23.614155+0000", 2024-03-11T21:02:30.466 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0036047639999999999 2024-03-11T21:02:30.466 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.466 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.467 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for peered", 2024-03-11T21:02:30.467 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:23.614164+0000", 2024-03-11T21:02:30.467 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 8.8289999999999997e-06 2024-03-11T21:02:30.467 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.467 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.467 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:02:30.467 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:24.614636+0000", 2024-03-11T21:02:30.467 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.0004722509999999 2024-03-11T21:02:30.467 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.467 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.467 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for peered", 2024-03-11T21:02:30.467 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:24.614650+0000", 2024-03-11T21:02:30.467 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.4071e-05 2024-03-11T21:02:30.467 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.467 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.467 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:02:30.468 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:25.790645+0000", 2024-03-11T21:02:30.468 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.1759943470000001 2024-03-11T21:02:30.468 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.468 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.468 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:02:30.468 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:25.790814+0000", 2024-03-11T21:02:30.468 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00016891200000000001 2024-03-11T21:02:30.468 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.468 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.468 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for subops from 1,2,7", 2024-03-11T21:02:30.468 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:25.790969+0000", 2024-03-11T21:02:30.468 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00015503399999999999 2024-03-11T21:02:30.468 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.468 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.468 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:02:30.468 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:25.798529+0000", 2024-03-11T21:02:30.469 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0075605819999999997 2024-03-11T21:02:30.469 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.469 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.469 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:02:30.469 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:25.798597+0000", 2024-03-11T21:02:30.469 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.7442999999999998e-05 2024-03-11T21:02:30.469 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.469 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.469 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:02:30.469 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:25.798770+0000", 2024-03-11T21:02:30.469 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00017372700000000001 2024-03-11T21:02:30.469 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.469 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.469 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "op_commit", 2024-03-11T21:02:30.469 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:25.799032+0000", 2024-03-11T21:02:30.469 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00026188399999999999 2024-03-11T21:02:30.469 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.470 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.470 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T21:02:30.470 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:25.799087+0000", 2024-03-11T21:02:30.470 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 5.4491999999999999e-05 2024-03-11T21:02:30.470 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.470 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.470 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:02:30.470 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:25.799205+0000", 2024-03-11T21:02:30.470 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00011857399999999999 2024-03-11T21:02:30.470 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:02:30.470 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:02:30.470 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:02:30.470 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.470 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.470 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_op(client.24629.0:167635 7.2 7:49edfd98:::benchmark_data_smithi023_67959_object57394:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e162)", 2024-03-11T21:02:30.471 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T20:57:23.617769+0000", 2024-03-11T21:02:30.471 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 306.77813884900002, 2024-03-11T21:02:30.471 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.1816013320000001, 2024-03-11T21:02:30.471 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:02:30.471 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:02:30.471 INFO:teuthology.orchestra.run.smithi023.stdout: "client_info": { 2024-03-11T21:02:30.471 INFO:teuthology.orchestra.run.smithi023.stdout: "client": "client.24629", 2024-03-11T21:02:30.471 INFO:teuthology.orchestra.run.smithi023.stdout: "client_addr": "172.21.15.23:0/959732609", 2024-03-11T21:02:30.471 INFO:teuthology.orchestra.run.smithi023.stdout: "tid": 167635 2024-03-11T21:02:30.471 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.471 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:02:30.471 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.471 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:02:30.471 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:23.617769+0000", 2024-03-11T21:02:30.471 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:02:30.471 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.472 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.472 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:02:30.472 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:23.617769+0000", 2024-03-11T21:02:30.472 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:02:30.472 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.472 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.472 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:02:30.472 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:23.617771+0000", 2024-03-11T21:02:30.472 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.5099999999999999e-06 2024-03-11T21:02:30.472 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.472 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.472 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:02:30.472 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:23.617777+0000", 2024-03-11T21:02:30.472 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 5.857e-06 2024-03-11T21:02:30.472 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.473 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.473 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:02:30.473 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:23.617789+0000", 2024-03-11T21:02:30.473 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.1902000000000001e-05 2024-03-11T21:02:30.473 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.473 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.473 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:02:30.473 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:23.617798+0000", 2024-03-11T21:02:30.473 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 9.7350000000000008e-06 2024-03-11T21:02:30.473 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.473 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.473 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:02:30.473 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:23.619568+0000", 2024-03-11T21:02:30.473 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.001769685 2024-03-11T21:02:30.473 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.473 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.474 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for peered", 2024-03-11T21:02:30.474 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:23.619581+0000", 2024-03-11T21:02:30.474 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.2926e-05 2024-03-11T21:02:30.474 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.474 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.474 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:02:30.474 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:24.615894+0000", 2024-03-11T21:02:30.474 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.99631322099999997 2024-03-11T21:02:30.474 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.474 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.474 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for peered", 2024-03-11T21:02:30.474 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:24.615904+0000", 2024-03-11T21:02:30.474 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 9.5820000000000008e-06 2024-03-11T21:02:30.474 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.474 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.475 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:02:30.475 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:25.793735+0000", 2024-03-11T21:02:30.475 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.1778313039999999 2024-03-11T21:02:30.475 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.475 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.475 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:02:30.475 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:25.793823+0000", 2024-03-11T21:02:30.475 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 8.8228999999999998e-05 2024-03-11T21:02:30.475 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.475 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.475 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for subops from 1,2,7", 2024-03-11T21:02:30.475 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:25.793917+0000", 2024-03-11T21:02:30.475 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 9.3671999999999994e-05 2024-03-11T21:02:30.475 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.475 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.476 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:02:30.476 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:25.798797+0000", 2024-03-11T21:02:30.476 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.004880138 2024-03-11T21:02:30.476 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.476 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.476 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:02:30.476 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:25.798819+0000", 2024-03-11T21:02:30.476 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.2348e-05 2024-03-11T21:02:30.476 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.476 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.476 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:02:30.476 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:25.798843+0000", 2024-03-11T21:02:30.476 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.3646e-05 2024-03-11T21:02:30.476 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.476 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.477 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "op_commit", 2024-03-11T21:02:30.477 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:25.799214+0000", 2024-03-11T21:02:30.477 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000370944 2024-03-11T21:02:30.477 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.477 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.477 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T21:02:30.477 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:25.799262+0000", 2024-03-11T21:02:30.477 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 4.8000999999999997e-05 2024-03-11T21:02:30.477 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.477 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.477 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:02:30.477 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:25.799371+0000", 2024-03-11T21:02:30.477 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000108632 2024-03-11T21:02:30.477 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:02:30.477 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:02:30.478 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:02:30.478 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.478 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.478 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_op(client.24629.0:167638 7.2 7:451714c6:::benchmark_data_smithi023_67959_object57397:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e162)", 2024-03-11T21:02:30.478 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T20:57:23.620223+0000", 2024-03-11T21:02:30.478 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 306.77568490800002, 2024-03-11T21:02:30.478 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.1799264410000001, 2024-03-11T21:02:30.478 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:02:30.478 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:02:30.478 INFO:teuthology.orchestra.run.smithi023.stdout: "client_info": { 2024-03-11T21:02:30.478 INFO:teuthology.orchestra.run.smithi023.stdout: "client": "client.24629", 2024-03-11T21:02:30.478 INFO:teuthology.orchestra.run.smithi023.stdout: "client_addr": "172.21.15.23:0/959732609", 2024-03-11T21:02:30.478 INFO:teuthology.orchestra.run.smithi023.stdout: "tid": 167638 2024-03-11T21:02:30.478 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.479 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:02:30.479 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.479 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:02:30.479 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:23.620223+0000", 2024-03-11T21:02:30.479 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:02:30.479 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.479 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.479 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:02:30.479 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:23.620223+0000", 2024-03-11T21:02:30.479 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:02:30.479 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.479 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.479 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:02:30.479 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:23.620224+0000", 2024-03-11T21:02:30.479 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 5.3000000000000001e-07 2024-03-11T21:02:30.480 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.480 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.480 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:02:30.480 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:23.620228+0000", 2024-03-11T21:02:30.480 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 3.7730000000000001e-06 2024-03-11T21:02:30.480 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.480 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.480 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:02:30.480 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:23.620236+0000", 2024-03-11T21:02:30.480 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 8.3559999999999993e-06 2024-03-11T21:02:30.480 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.480 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.480 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:02:30.480 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:23.620242+0000", 2024-03-11T21:02:30.480 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.4200000000000004e-06 2024-03-11T21:02:30.481 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.481 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.481 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:02:30.481 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:23.630585+0000", 2024-03-11T21:02:30.481 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.010342934 2024-03-11T21:02:30.481 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.481 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.481 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for peered", 2024-03-11T21:02:30.481 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:23.630597+0000", 2024-03-11T21:02:30.481 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.1525000000000001e-05 2024-03-11T21:02:30.481 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.481 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.481 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:02:30.481 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:24.615935+0000", 2024-03-11T21:02:30.481 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.98533783900000005 2024-03-11T21:02:30.481 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.482 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.482 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for peered", 2024-03-11T21:02:30.482 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:24.615944+0000", 2024-03-11T21:02:30.482 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 9.0429999999999996e-06 2024-03-11T21:02:30.482 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.482 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.482 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:02:30.482 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:25.795926+0000", 2024-03-11T21:02:30.482 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.179982026 2024-03-11T21:02:30.482 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.482 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.482 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:02:30.482 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:25.796028+0000", 2024-03-11T21:02:30.482 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000102172 2024-03-11T21:02:30.482 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.483 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.483 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for subops from 1,2,7", 2024-03-11T21:02:30.483 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:25.796195+0000", 2024-03-11T21:02:30.483 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00016758100000000001 2024-03-11T21:02:30.483 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.483 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.483 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:02:30.483 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:25.799016+0000", 2024-03-11T21:02:30.483 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0028206220000000001 2024-03-11T21:02:30.483 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.483 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.483 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:02:30.483 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:25.799411+0000", 2024-03-11T21:02:30.483 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00039464399999999998 2024-03-11T21:02:30.483 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.484 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.484 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:02:30.484 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:25.799468+0000", 2024-03-11T21:02:30.484 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 5.6802999999999998e-05 2024-03-11T21:02:30.484 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.484 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.484 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "op_commit", 2024-03-11T21:02:30.484 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:25.799966+0000", 2024-03-11T21:02:30.484 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00049837600000000003 2024-03-11T21:02:30.484 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.484 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.484 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T21:02:30.484 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:25.800025+0000", 2024-03-11T21:02:30.484 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 5.9462000000000001e-05 2024-03-11T21:02:30.485 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.485 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.485 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:02:30.485 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:57:25.800150+0000", 2024-03-11T21:02:30.485 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00012433499999999999 2024-03-11T21:02:30.485 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:02:30.485 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:02:30.485 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:02:30.485 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.485 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.485 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_op(client.24406.0:472 5.0 5.9f137cac (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e182)", 2024-03-11T21:02:30.485 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T20:58:52.609078+0000", 2024-03-11T21:02:30.485 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 217.78683052900001, 2024-03-11T21:02:30.485 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.98797840100000001, 2024-03-11T21:02:30.485 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:02:30.486 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "reached pg", 2024-03-11T21:02:30.486 INFO:teuthology.orchestra.run.smithi023.stdout: "client_info": { 2024-03-11T21:02:30.486 INFO:teuthology.orchestra.run.smithi023.stdout: "client": "client.24406", 2024-03-11T21:02:30.486 INFO:teuthology.orchestra.run.smithi023.stdout: "client_addr": "172.21.15.146:0/3388716841", 2024-03-11T21:02:30.486 INFO:teuthology.orchestra.run.smithi023.stdout: "tid": 472 2024-03-11T21:02:30.486 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.486 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:02:30.486 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.486 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:02:30.486 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:58:52.609078+0000", 2024-03-11T21:02:30.486 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:02:30.486 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.486 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.486 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:02:30.487 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:58:52.609078+0000", 2024-03-11T21:02:30.487 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:02:30.487 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.487 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.487 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:02:30.487 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:58:52.609080+0000", 2024-03-11T21:02:30.487 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.5560000000000002e-06 2024-03-11T21:02:30.487 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.487 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.487 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:02:30.487 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:58:52.609090+0000", 2024-03-11T21:02:30.487 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 9.8560000000000002e-06 2024-03-11T21:02:30.487 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.487 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.487 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:02:30.488 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:58:52.609112+0000", 2024-03-11T21:02:30.488 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.2104e-05 2024-03-11T21:02:30.488 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.488 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.488 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:02:30.488 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:58:52.609124+0000", 2024-03-11T21:02:30.488 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.2074e-05 2024-03-11T21:02:30.488 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.488 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.488 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:02:30.488 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:58:52.609232+0000", 2024-03-11T21:02:30.488 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000107283 2024-03-11T21:02:30.488 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.488 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.488 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for peered", 2024-03-11T21:02:30.488 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:58:52.609244+0000", 2024-03-11T21:02:30.489 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.2743e-05 2024-03-11T21:02:30.489 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.489 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.489 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:02:30.489 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:58:53.597012+0000", 2024-03-11T21:02:30.489 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.98776751900000004 2024-03-11T21:02:30.489 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.489 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.489 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:02:30.489 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:58:53.597056+0000", 2024-03-11T21:02:30.489 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 4.4265999999999999e-05 2024-03-11T21:02:30.489 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:02:30.489 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:02:30.489 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:02:30.489 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.490 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.490 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_op(client.24281.0:1929 4.9 4.84ada7c9 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-03-11T21:02:30.490 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T20:58:53.171211+0000", 2024-03-11T21:02:30.490 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 217.224696787, 2024-03-11T21:02:30.490 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.42685853400000001, 2024-03-11T21:02:30.490 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:02:30.490 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "reached pg", 2024-03-11T21:02:30.490 INFO:teuthology.orchestra.run.smithi023.stdout: "client_info": { 2024-03-11T21:02:30.490 INFO:teuthology.orchestra.run.smithi023.stdout: "client": "client.24281", 2024-03-11T21:02:30.490 INFO:teuthology.orchestra.run.smithi023.stdout: "client_addr": "172.21.15.23:0/2185274774", 2024-03-11T21:02:30.490 INFO:teuthology.orchestra.run.smithi023.stdout: "tid": 1929 2024-03-11T21:02:30.490 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.490 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:02:30.490 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.490 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:02:30.491 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:58:53.171211+0000", 2024-03-11T21:02:30.491 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:02:30.491 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.491 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.491 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:02:30.491 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:58:53.171211+0000", 2024-03-11T21:02:30.491 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:02:30.491 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.491 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.491 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:02:30.491 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:58:53.171214+0000", 2024-03-11T21:02:30.491 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.165e-06 2024-03-11T21:02:30.491 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.491 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.491 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:02:30.491 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:58:53.171224+0000", 2024-03-11T21:02:30.492 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.0025e-05 2024-03-11T21:02:30.492 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.492 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.492 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:02:30.492 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:58:53.171246+0000", 2024-03-11T21:02:30.492 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.2421000000000001e-05 2024-03-11T21:02:30.492 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.492 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.492 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:02:30.492 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:58:53.171259+0000", 2024-03-11T21:02:30.492 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.2556e-05 2024-03-11T21:02:30.492 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.492 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.492 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:02:30.492 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:58:53.171350+0000", 2024-03-11T21:02:30.493 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 9.1397999999999995e-05 2024-03-11T21:02:30.493 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.493 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.493 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for peered", 2024-03-11T21:02:30.493 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:58:53.171360+0000", 2024-03-11T21:02:30.493 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 9.7079999999999992e-06 2024-03-11T21:02:30.493 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.493 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.493 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:02:30.493 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:58:53.598024+0000", 2024-03-11T21:02:30.493 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.42666479699999998 2024-03-11T21:02:30.493 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.493 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.493 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:02:30.493 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:58:53.598070+0000", 2024-03-11T21:02:30.494 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 4.5463999999999997e-05 2024-03-11T21:02:30.494 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:02:30.494 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:02:30.494 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:02:30.494 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.494 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.494 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_op(client.14910.0:61961 8.f 8:fe04a4b3:::benchmark_data_smithi023_89854_object11876:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e216)", 2024-03-11T21:02:30.494 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T21:01:15.084531+0000", 2024-03-11T21:02:30.494 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 75.311377375000006, 2024-03-11T21:02:30.494 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.34057533099999998, 2024-03-11T21:02:30.494 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:02:30.494 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:02:30.494 INFO:teuthology.orchestra.run.smithi023.stdout: "client_info": { 2024-03-11T21:02:30.494 INFO:teuthology.orchestra.run.smithi023.stdout: "client": "client.14910", 2024-03-11T21:02:30.495 INFO:teuthology.orchestra.run.smithi023.stdout: "client_addr": "172.21.15.23:0/2457905657", 2024-03-11T21:02:30.495 INFO:teuthology.orchestra.run.smithi023.stdout: "tid": 61961 2024-03-11T21:02:30.495 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.495 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:02:30.495 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.495 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:02:30.495 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:01:15.084531+0000", 2024-03-11T21:02:30.495 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:02:30.495 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.495 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.495 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:02:30.495 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:01:15.084531+0000", 2024-03-11T21:02:30.495 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:02:30.495 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.495 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.495 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:02:30.496 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:01:15.084532+0000", 2024-03-11T21:02:30.496 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.1170000000000001e-06 2024-03-11T21:02:30.496 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.496 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.496 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:02:30.496 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:01:15.084538+0000", 2024-03-11T21:02:30.496 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 5.5979999999999996e-06 2024-03-11T21:02:30.496 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.496 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.496 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:02:30.496 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:01:15.084547+0000", 2024-03-11T21:02:30.496 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 9.7359999999999996e-06 2024-03-11T21:02:30.496 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.496 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.497 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:02:30.497 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:01:15.084554+0000", 2024-03-11T21:02:30.497 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.8379999999999996e-06 2024-03-11T21:02:30.497 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.497 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.497 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:02:30.497 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:01:15.087184+0000", 2024-03-11T21:02:30.497 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0026294030000000002 2024-03-11T21:02:30.497 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.497 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.497 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:02:30.497 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:01:15.087325+0000", 2024-03-11T21:02:30.497 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000141197 2024-03-11T21:02:30.497 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.497 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.497 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for subops from 1,2", 2024-03-11T21:02:30.498 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:01:15.087480+0000", 2024-03-11T21:02:30.498 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00015535599999999999 2024-03-11T21:02:30.498 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.498 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.498 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "op_commit", 2024-03-11T21:02:30.498 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:01:15.094784+0000", 2024-03-11T21:02:30.498 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0073041249999999999 2024-03-11T21:02:30.498 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.498 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.498 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:02:30.498 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:01:15.100254+0000", 2024-03-11T21:02:30.498 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0054694169999999999 2024-03-11T21:02:30.498 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.498 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.499 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:02:30.499 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:01:15.419039+0000", 2024-03-11T21:02:30.499 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.31878537899999998 2024-03-11T21:02:30.499 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.499 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.499 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T21:02:30.499 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:01:15.419128+0000", 2024-03-11T21:02:30.499 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 8.9111999999999997e-05 2024-03-11T21:02:30.499 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.499 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.499 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:02:30.499 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:01:15.425106+0000", 2024-03-11T21:02:30.499 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0059780529999999997 2024-03-11T21:02:30.499 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:02:30.499 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:02:30.499 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:02:30.500 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.500 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.500 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_op(client.14910.0:62002 8.f 8:f8c24c1a:::benchmark_data_smithi023_89854_object11917:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e216)", 2024-03-11T21:02:30.500 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T21:01:15.120008+0000", 2024-03-11T21:02:30.500 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 75.275900254000007, 2024-03-11T21:02:30.500 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.307583889, 2024-03-11T21:02:30.500 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:02:30.500 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:02:30.500 INFO:teuthology.orchestra.run.smithi023.stdout: "client_info": { 2024-03-11T21:02:30.500 INFO:teuthology.orchestra.run.smithi023.stdout: "client": "client.14910", 2024-03-11T21:02:30.500 INFO:teuthology.orchestra.run.smithi023.stdout: "client_addr": "172.21.15.23:0/2457905657", 2024-03-11T21:02:30.500 INFO:teuthology.orchestra.run.smithi023.stdout: "tid": 62002 2024-03-11T21:02:30.500 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.500 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:02:30.500 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.500 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:02:30.500 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:01:15.120008+0000", 2024-03-11T21:02:30.501 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:02:30.501 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.501 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.501 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:02:30.501 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:01:15.120008+0000", 2024-03-11T21:02:30.501 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:02:30.501 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.501 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.501 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:02:30.501 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:01:15.120009+0000", 2024-03-11T21:02:30.501 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.53e-06 2024-03-11T21:02:30.501 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.501 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.501 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:02:30.501 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:01:15.120015+0000", 2024-03-11T21:02:30.502 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 5.412e-06 2024-03-11T21:02:30.502 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.502 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.502 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:02:30.502 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:01:15.120028+0000", 2024-03-11T21:02:30.502 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.2894999999999999e-05 2024-03-11T21:02:30.502 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.502 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.502 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:02:30.502 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:01:15.120035+0000", 2024-03-11T21:02:30.502 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.3930000000000002e-06 2024-03-11T21:02:30.502 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.502 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.502 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:02:30.503 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:01:15.121250+0000", 2024-03-11T21:02:30.503 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0012151149999999999 2024-03-11T21:02:30.503 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.503 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.503 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:02:30.503 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:01:15.121411+0000", 2024-03-11T21:02:30.503 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00016028200000000001 2024-03-11T21:02:30.503 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.503 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.503 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for subops from 1,2", 2024-03-11T21:02:30.503 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:01:15.121556+0000", 2024-03-11T21:02:30.503 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00014550000000000001 2024-03-11T21:02:30.503 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.503 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.503 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "op_commit", 2024-03-11T21:02:30.504 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:01:15.126049+0000", 2024-03-11T21:02:30.504 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0044925349999999998 2024-03-11T21:02:30.504 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.504 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.504 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:02:30.504 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:01:15.129122+0000", 2024-03-11T21:02:30.504 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0030730530000000001 2024-03-11T21:02:30.504 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.504 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.504 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:02:30.504 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:01:15.425421+0000", 2024-03-11T21:02:30.504 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.296299637 2024-03-11T21:02:30.504 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.504 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.504 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T21:02:30.504 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:01:15.425563+0000", 2024-03-11T21:02:30.505 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000141462 2024-03-11T21:02:30.505 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:02:30.505 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:02:30.505 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:02:30.505 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:01:15.427592+0000", 2024-03-11T21:02:30.505 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.002029075 2024-03-11T21:02:30.505 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:02:30.505 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:02:30.505 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:02:30.505 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:02:30.505 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:02:30.505 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-11T21:02:30.505 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:02:30.507 INFO:teuthology.orchestra.run.smithi023.stderr:pool 'unique_pool_1' removed 2024-03-11T21:02:30.507 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:30 smithi023 bash[21372]: cluster 2024-03-11T21:02:29.249193+0000 osd.1 (osd.1) 15 : cluster 0 2.a scrub starts 2024-03-11T21:02:30.507 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:30 smithi023 bash[21372]: cluster 2024-03-11T21:02:29.250778+0000 osd.1 (osd.1) 16 : cluster 0 2.a scrub ok 2024-03-11T21:02:30.507 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:30 smithi023 bash[21372]: cluster 2024-03-11T21:02:29.358384+0000 mon.a (mon.0) 1398 : cluster 0 osdmap e243: 8 total, 8 up, 8 in 2024-03-11T21:02:30.507 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:30 smithi023 bash[21372]: audit 2024-03-11T21:02:29.625138+0000 mon.c (mon.2) 54 : audit 1 from='client.? 172.21.15.23:0/3511706091' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T21:02:30.507 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:30 smithi023 bash[21372]: audit 2024-03-11T21:02:29.625576+0000 mon.a (mon.0) 1399 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T21:02:30.507 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:30 smithi023 bash[21372]: cluster 2024-03-11T21:02:29.802179+0000 osd.5 (osd.5) 215 : cluster 0 2.2 scrub starts 2024-03-11T21:02:30.509 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:30 smithi023 bash[21372]: cluster 2024-03-11T21:02:29.807027+0000 osd.5 (osd.5) 216 : cluster 0 2.2 scrub ok 2024-03-11T21:02:30.510 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:30 smithi023 bash[29146]: cluster 2024-03-11T21:02:29.249193+0000 osd.1 (osd.1) 15 : cluster 0 2.a scrub starts 2024-03-11T21:02:30.510 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:30 smithi023 bash[29146]: cluster 2024-03-11T21:02:29.250778+0000 osd.1 (osd.1) 16 : cluster 0 2.a scrub ok 2024-03-11T21:02:30.510 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:30 smithi023 bash[29146]: cluster 2024-03-11T21:02:29.358384+0000 mon.a (mon.0) 1398 : cluster 0 osdmap e243: 8 total, 8 up, 8 in 2024-03-11T21:02:30.510 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:30 smithi023 bash[29146]: audit 2024-03-11T21:02:29.625138+0000 mon.c (mon.2) 54 : audit 1 from='client.? 172.21.15.23:0/3511706091' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T21:02:30.510 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:30 smithi023 bash[29146]: audit 2024-03-11T21:02:29.625576+0000 mon.a (mon.0) 1399 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T21:02:30.510 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:30 smithi023 bash[29146]: cluster 2024-03-11T21:02:29.802179+0000 osd.5 (osd.5) 215 : cluster 0 2.2 scrub starts 2024-03-11T21:02:30.510 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:30 smithi023 bash[29146]: cluster 2024-03-11T21:02:29.807027+0000 osd.5 (osd.5) 216 : cluster 0 2.2 scrub ok 2024-03-11T21:02:30.532 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:02:30.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:30.535 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:02:30.636 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:30.654 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:30.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:30.656 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:02:30.756 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:30.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:30 smithi146 bash[23809]: cluster 2024-03-11T21:02:29.249193+0000 osd.1 (osd.1) 15 : cluster 0 2.a scrub starts 2024-03-11T21:02:30.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:30 smithi146 bash[23809]: cluster 2024-03-11T21:02:29.250778+0000 osd.1 (osd.1) 16 : cluster 0 2.a scrub ok 2024-03-11T21:02:30.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:30 smithi146 bash[23809]: cluster 2024-03-11T21:02:29.358384+0000 mon.a (mon.0) 1398 : cluster 0 osdmap e243: 8 total, 8 up, 8 in 2024-03-11T21:02:30.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:30 smithi146 bash[23809]: audit 2024-03-11T21:02:29.625138+0000 mon.c (mon.2) 54 : audit 1 from='client.? 172.21.15.23:0/3511706091' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T21:02:30.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:30 smithi146 bash[23809]: audit 2024-03-11T21:02:29.625576+0000 mon.a (mon.0) 1399 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T21:02:30.764 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:30 smithi146 bash[23809]: cluster 2024-03-11T21:02:29.802179+0000 osd.5 (osd.5) 215 : cluster 0 2.2 scrub starts 2024-03-11T21:02:30.764 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:30 smithi146 bash[23809]: cluster 2024-03-11T21:02:29.807027+0000 osd.5 (osd.5) 216 : cluster 0 2.2 scrub ok 2024-03-11T21:02:30.784 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:30.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:30.787 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:02:30.887 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:02:30.908 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:02:30.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:30.910 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:02:31.011 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:31.038 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:31.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:31.040 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:02:31.141 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:31.168 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:31.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:31.170 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:02:31.270 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:02:31.296 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:02:31.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:31.299 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:02:31.399 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:31.429 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:31.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:31.431 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:02:31.437 INFO:teuthology.orchestra.run.smithi023.stderr:reweighted osd.3 to 1 (10000) 2024-03-11T21:02:31.533 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:31.556 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:31.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:31.558 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:02:31.659 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:02:31.670 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:31 smithi146 bash[23809]: cluster 2024-03-11T21:02:30.131035+0000 mgr.y (mgr.24380) 435 : cluster 0 pgmap v473: 148 pgs: 148 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:02:31.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:31 smithi146 bash[23809]: cluster 2024-03-11T21:02:30.286578+0000 osd.1 (osd.1) 17 : cluster 0 4.2 scrub starts 2024-03-11T21:02:31.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:31 smithi146 bash[23809]: cluster 2024-03-11T21:02:30.287954+0000 osd.1 (osd.1) 18 : cluster 0 4.2 scrub ok 2024-03-11T21:02:31.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:31 smithi146 bash[23809]: audit 2024-03-11T21:02:30.405277+0000 mon.a (mon.0) 1400 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-03-11T21:02:31.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:31 smithi146 bash[23809]: cluster 2024-03-11T21:02:30.408826+0000 mon.a (mon.0) 1401 : cluster 0 osdmap e244: 8 total, 8 up, 8 in 2024-03-11T21:02:31.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:31 smithi146 bash[23809]: cluster 2024-03-11T21:02:30.475607+0000 osd.4 (osd.4) 33 : cluster 0 6.1a scrub starts 2024-03-11T21:02:31.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:31 smithi146 bash[23809]: cluster 2024-03-11T21:02:30.520838+0000 osd.4 (osd.4) 34 : cluster 0 6.1a scrub ok 2024-03-11T21:02:31.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:31 smithi146 bash[23809]: cluster 2024-03-11T21:02:30.556878+0000 osd.3 (osd.3) 250 : cluster 0 3.18 scrub starts 2024-03-11T21:02:31.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:31 smithi146 bash[23809]: cluster 2024-03-11T21:02:30.560284+0000 osd.3 (osd.3) 251 : cluster 0 3.18 scrub ok 2024-03-11T21:02:31.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:31 smithi146 bash[23809]: audit 2024-03-11T21:02:30.760156+0000 mon.c (mon.2) 55 : audit 1 from='client.? 172.21.15.23:0/1058819224' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-11T21:02:31.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:31 smithi146 bash[23809]: audit 2024-03-11T21:02:30.760883+0000 mon.a (mon.0) 1402 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-11T21:02:31.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:31 smithi146 bash[23809]: cluster 2024-03-11T21:02:30.941639+0000 osd.7 (osd.7) 241 : cluster 0 3.12 scrub starts 2024-03-11T21:02:31.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:31 smithi146 bash[23809]: cluster 2024-03-11T21:02:30.944320+0000 osd.7 (osd.7) 242 : cluster 0 3.12 scrub ok 2024-03-11T21:02:31.685 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:02:31.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:31.687 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:02:31.786 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:31 smithi023 bash[21372]: cluster 2024-03-11T21:02:30.131035+0000 mgr.y (mgr.24380) 435 : cluster 0 pgmap v473: 148 pgs: 148 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:02:31.786 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:31 smithi023 bash[21372]: cluster 2024-03-11T21:02:30.286578+0000 osd.1 (osd.1) 17 : cluster 0 4.2 scrub starts 2024-03-11T21:02:31.786 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:31 smithi023 bash[21372]: cluster 2024-03-11T21:02:30.287954+0000 osd.1 (osd.1) 18 : cluster 0 4.2 scrub ok 2024-03-11T21:02:31.786 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:31 smithi023 bash[21372]: audit 2024-03-11T21:02:30.405277+0000 mon.a (mon.0) 1400 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-03-11T21:02:31.786 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:31 smithi023 bash[21372]: cluster 2024-03-11T21:02:30.408826+0000 mon.a (mon.0) 1401 : cluster 0 osdmap e244: 8 total, 8 up, 8 in 2024-03-11T21:02:31.786 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:31 smithi023 bash[21372]: cluster 2024-03-11T21:02:30.475607+0000 osd.4 (osd.4) 33 : cluster 0 6.1a scrub starts 2024-03-11T21:02:31.786 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:31 smithi023 bash[21372]: cluster 2024-03-11T21:02:30.520838+0000 osd.4 (osd.4) 34 : cluster 0 6.1a scrub ok 2024-03-11T21:02:31.786 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:31 smithi023 bash[21372]: cluster 2024-03-11T21:02:30.556878+0000 osd.3 (osd.3) 250 : cluster 0 3.18 scrub starts 2024-03-11T21:02:31.786 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:31 smithi023 bash[21372]: cluster 2024-03-11T21:02:30.560284+0000 osd.3 (osd.3) 251 : cluster 0 3.18 scrub ok 2024-03-11T21:02:31.786 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:31 smithi023 bash[21372]: audit 2024-03-11T21:02:30.760156+0000 mon.c (mon.2) 55 : audit 1 from='client.? 172.21.15.23:0/1058819224' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-11T21:02:31.787 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:31 smithi023 bash[21372]: audit 2024-03-11T21:02:30.760883+0000 mon.a (mon.0) 1402 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-11T21:02:31.787 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:31 smithi023 bash[21372]: cluster 2024-03-11T21:02:30.941639+0000 osd.7 (osd.7) 241 : cluster 0 3.12 scrub starts 2024-03-11T21:02:31.787 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:31 smithi023 bash[21372]: cluster 2024-03-11T21:02:30.944320+0000 osd.7 (osd.7) 242 : cluster 0 3.12 scrub ok 2024-03-11T21:02:31.787 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:31 smithi023 bash[29146]: cluster 2024-03-11T21:02:30.131035+0000 mgr.y (mgr.24380) 435 : cluster 0 pgmap v473: 148 pgs: 148 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:02:31.787 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:31 smithi023 bash[29146]: cluster 2024-03-11T21:02:30.286578+0000 osd.1 (osd.1) 17 : cluster 0 4.2 scrub starts 2024-03-11T21:02:31.787 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:31 smithi023 bash[29146]: cluster 2024-03-11T21:02:30.287954+0000 osd.1 (osd.1) 18 : cluster 0 4.2 scrub ok 2024-03-11T21:02:31.787 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:31 smithi023 bash[29146]: audit 2024-03-11T21:02:30.405277+0000 mon.a (mon.0) 1400 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-03-11T21:02:31.787 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:31 smithi023 bash[29146]: cluster 2024-03-11T21:02:30.408826+0000 mon.a (mon.0) 1401 : cluster 0 osdmap e244: 8 total, 8 up, 8 in 2024-03-11T21:02:31.787 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:31 smithi023 bash[29146]: cluster 2024-03-11T21:02:30.475607+0000 osd.4 (osd.4) 33 : cluster 0 6.1a scrub starts 2024-03-11T21:02:31.787 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:31 smithi023 bash[29146]: cluster 2024-03-11T21:02:30.520838+0000 osd.4 (osd.4) 34 : cluster 0 6.1a scrub ok 2024-03-11T21:02:31.787 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:31 smithi023 bash[29146]: cluster 2024-03-11T21:02:30.556878+0000 osd.3 (osd.3) 250 : cluster 0 3.18 scrub starts 2024-03-11T21:02:31.787 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:31 smithi023 bash[29146]: cluster 2024-03-11T21:02:30.560284+0000 osd.3 (osd.3) 251 : cluster 0 3.18 scrub ok 2024-03-11T21:02:31.788 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:31 smithi023 bash[29146]: audit 2024-03-11T21:02:30.760156+0000 mon.c (mon.2) 55 : audit 1 from='client.? 172.21.15.23:0/1058819224' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-11T21:02:31.788 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:31 smithi023 bash[29146]: audit 2024-03-11T21:02:30.760883+0000 mon.a (mon.0) 1402 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-11T21:02:31.788 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:31 smithi023 bash[29146]: cluster 2024-03-11T21:02:30.941639+0000 osd.7 (osd.7) 241 : cluster 0 3.12 scrub starts 2024-03-11T21:02:31.788 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:31 smithi023 bash[29146]: cluster 2024-03-11T21:02:30.944320+0000 osd.7 (osd.7) 242 : cluster 0 3.12 scrub ok 2024-03-11T21:02:31.788 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:02:31.817 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:02:31.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:31.820 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:02:31.920 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:02:31.950 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:02:31.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:31.951 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:02:32.052 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:32.079 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:32.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:32.082 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:02:32.183 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:02:32.210 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:02:32.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:32.212 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:02:32.312 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:02:32.336 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:02:32.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:32.338 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:02:32.439 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:32.466 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:32.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:32.468 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:02:32.569 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:32.594 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:32.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:32.597 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:02:32.698 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:02:32.709 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:32 smithi146 bash[23809]: cluster 2024-03-11T21:02:31.359319+0000 osd.6 (osd.6) 176 : cluster 0 3.10 scrub starts 2024-03-11T21:02:32.709 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:32 smithi146 bash[23809]: cluster 2024-03-11T21:02:31.364210+0000 osd.6 (osd.6) 177 : cluster 0 3.10 scrub ok 2024-03-11T21:02:32.709 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:32 smithi146 bash[23809]: audit 2024-03-11T21:02:31.428162+0000 mon.a (mon.0) 1403 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-11T21:02:32.709 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:32 smithi146 bash[23809]: cluster 2024-03-11T21:02:31.435826+0000 mon.a (mon.0) 1404 : cluster 0 osdmap e245: 8 total, 8 up, 8 in 2024-03-11T21:02:32.709 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:32 smithi146 bash[23809]: cluster 2024-03-11T21:02:31.469852+0000 osd.4 (osd.4) 35 : cluster 0 4.c scrub starts 2024-03-11T21:02:32.709 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:32 smithi146 bash[23809]: cluster 2024-03-11T21:02:31.471507+0000 osd.4 (osd.4) 36 : cluster 0 4.c scrub ok 2024-03-11T21:02:32.709 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:32 smithi146 bash[23809]: cluster 2024-03-11T21:02:31.547252+0000 osd.3 (osd.3) 252 : cluster 0 4.10 scrub starts 2024-03-11T21:02:32.709 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:32 smithi146 bash[23809]: cluster 2024-03-11T21:02:31.549026+0000 osd.3 (osd.3) 253 : cluster 0 4.10 scrub ok 2024-03-11T21:02:32.710 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:32 smithi146 bash[23809]: cluster 2024-03-11T21:02:31.895663+0000 osd.7 (osd.7) 243 : cluster 0 2.1c scrub starts 2024-03-11T21:02:32.710 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:32 smithi146 bash[23809]: cluster 2024-03-11T21:02:31.898090+0000 osd.7 (osd.7) 244 : cluster 0 2.1c scrub ok 2024-03-11T21:02:32.725 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:02:32.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:32.727 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:02:32.786 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:32 smithi023 bash[21372]: cluster 2024-03-11T21:02:31.359319+0000 osd.6 (osd.6) 176 : cluster 0 3.10 scrub starts 2024-03-11T21:02:32.786 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:32 smithi023 bash[21372]: cluster 2024-03-11T21:02:31.364210+0000 osd.6 (osd.6) 177 : cluster 0 3.10 scrub ok 2024-03-11T21:02:32.786 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:32 smithi023 bash[21372]: audit 2024-03-11T21:02:31.428162+0000 mon.a (mon.0) 1403 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-11T21:02:32.786 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:32 smithi023 bash[21372]: cluster 2024-03-11T21:02:31.435826+0000 mon.a (mon.0) 1404 : cluster 0 osdmap e245: 8 total, 8 up, 8 in 2024-03-11T21:02:32.786 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:32 smithi023 bash[21372]: cluster 2024-03-11T21:02:31.469852+0000 osd.4 (osd.4) 35 : cluster 0 4.c scrub starts 2024-03-11T21:02:32.786 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:32 smithi023 bash[21372]: cluster 2024-03-11T21:02:31.471507+0000 osd.4 (osd.4) 36 : cluster 0 4.c scrub ok 2024-03-11T21:02:32.786 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:32 smithi023 bash[21372]: cluster 2024-03-11T21:02:31.547252+0000 osd.3 (osd.3) 252 : cluster 0 4.10 scrub starts 2024-03-11T21:02:32.786 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:32 smithi023 bash[21372]: cluster 2024-03-11T21:02:31.549026+0000 osd.3 (osd.3) 253 : cluster 0 4.10 scrub ok 2024-03-11T21:02:32.786 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:32 smithi023 bash[21372]: cluster 2024-03-11T21:02:31.895663+0000 osd.7 (osd.7) 243 : cluster 0 2.1c scrub starts 2024-03-11T21:02:32.786 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:32 smithi023 bash[21372]: cluster 2024-03-11T21:02:31.898090+0000 osd.7 (osd.7) 244 : cluster 0 2.1c scrub ok 2024-03-11T21:02:32.787 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:32 smithi023 bash[29146]: cluster 2024-03-11T21:02:31.359319+0000 osd.6 (osd.6) 176 : cluster 0 3.10 scrub starts 2024-03-11T21:02:32.787 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:32 smithi023 bash[29146]: cluster 2024-03-11T21:02:31.364210+0000 osd.6 (osd.6) 177 : cluster 0 3.10 scrub ok 2024-03-11T21:02:32.787 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:32 smithi023 bash[29146]: audit 2024-03-11T21:02:31.428162+0000 mon.a (mon.0) 1403 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-11T21:02:32.787 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:32 smithi023 bash[29146]: cluster 2024-03-11T21:02:31.435826+0000 mon.a (mon.0) 1404 : cluster 0 osdmap e245: 8 total, 8 up, 8 in 2024-03-11T21:02:32.787 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:32 smithi023 bash[29146]: cluster 2024-03-11T21:02:31.469852+0000 osd.4 (osd.4) 35 : cluster 0 4.c scrub starts 2024-03-11T21:02:32.787 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:32 smithi023 bash[29146]: cluster 2024-03-11T21:02:31.471507+0000 osd.4 (osd.4) 36 : cluster 0 4.c scrub ok 2024-03-11T21:02:32.787 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:32 smithi023 bash[29146]: cluster 2024-03-11T21:02:31.547252+0000 osd.3 (osd.3) 252 : cluster 0 4.10 scrub starts 2024-03-11T21:02:32.787 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:32 smithi023 bash[29146]: cluster 2024-03-11T21:02:31.549026+0000 osd.3 (osd.3) 253 : cluster 0 4.10 scrub ok 2024-03-11T21:02:32.787 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:32 smithi023 bash[29146]: cluster 2024-03-11T21:02:31.895663+0000 osd.7 (osd.7) 243 : cluster 0 2.1c scrub starts 2024-03-11T21:02:32.787 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:32 smithi023 bash[29146]: cluster 2024-03-11T21:02:31.898090+0000 osd.7 (osd.7) 244 : cluster 0 2.1c scrub ok 2024-03-11T21:02:32.829 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:02:32.857 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:02:32.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:32.859 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:02:32.959 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:32.984 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:32.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:32.986 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:02:33.087 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:33.116 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:33.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:33.119 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:02:33.219 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:02:33.247 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:02:33.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:33.250 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:02:33.351 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:33.378 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:33.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:33.381 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:02:33.434 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd reweight 4 1 2024-03-11T21:02:33.437 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-03-11T21:02:33.437 INFO:tasks.radosbench:Beginning radosbench... 2024-03-11T21:02:33.438 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_2 2024-03-11T21:02:33.438 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd pool create unique_pool_2 16 2024-03-11T21:02:33.481 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:33.501 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:33.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:33.503 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:02:33.514 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:33 smithi146 bash[23809]: cluster 2024-03-11T21:02:31.689805+0000 osd.2 (osd.2) 160 : cluster 0 2.0 scrub starts 2024-03-11T21:02:33.514 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:33 smithi146 bash[23809]: cluster 2024-03-11T21:02:31.693831+0000 osd.2 (osd.2) 161 : cluster 0 2.0 scrub ok 2024-03-11T21:02:33.514 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:33 smithi146 bash[23809]: cluster 2024-03-11T21:02:31.853215+0000 osd.0 (osd.0) 181 : cluster 0 4.b scrub starts 2024-03-11T21:02:33.514 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:33 smithi146 bash[23809]: cluster 2024-03-11T21:02:31.854695+0000 osd.0 (osd.0) 182 : cluster 0 4.b scrub ok 2024-03-11T21:02:33.514 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:33 smithi146 bash[23809]: cluster 2024-03-11T21:02:32.131631+0000 mgr.y (mgr.24380) 436 : cluster 0 pgmap v476: 132 pgs: 132 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 705 B/s rd, 0 op/s 2024-03-11T21:02:33.514 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:33 smithi146 bash[23809]: cluster 2024-03-11T21:02:32.300519+0000 osd.1 (osd.1) 19 : cluster 0 6.1d scrub starts 2024-03-11T21:02:33.514 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:33 smithi146 bash[23809]: cluster 2024-03-11T21:02:32.330156+0000 osd.1 (osd.1) 20 : cluster 0 6.1d scrub ok 2024-03-11T21:02:33.514 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:33 smithi146 bash[23809]: cluster 2024-03-11T21:02:32.426498+0000 osd.4 (osd.4) 37 : cluster 0 3.1 scrub starts 2024-03-11T21:02:33.514 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:33 smithi146 bash[23809]: cluster 2024-03-11T21:02:32.429520+0000 osd.4 (osd.4) 38 : cluster 0 3.1 scrub ok 2024-03-11T21:02:33.514 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:33 smithi146 bash[23809]: cluster 2024-03-11T21:02:32.918893+0000 osd.7 (osd.7) 245 : cluster 0 6.5 deep-scrub starts 2024-03-11T21:02:33.514 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:33 smithi146 bash[23809]: cluster 2024-03-11T21:02:32.951098+0000 osd.7 (osd.7) 246 : cluster 0 6.5 deep-scrub ok 2024-03-11T21:02:33.547 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-11T21:02:33.604 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:02:33.628 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:02:33.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:33.630 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:02:33.730 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:02:33.749 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:02:33.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:33.751 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:02:33.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:33 smithi023 bash[29146]: cluster 2024-03-11T21:02:31.689805+0000 osd.2 (osd.2) 160 : cluster 0 2.0 scrub starts 2024-03-11T21:02:33.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:33 smithi023 bash[29146]: cluster 2024-03-11T21:02:31.693831+0000 osd.2 (osd.2) 161 : cluster 0 2.0 scrub ok 2024-03-11T21:02:33.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:33 smithi023 bash[29146]: cluster 2024-03-11T21:02:31.853215+0000 osd.0 (osd.0) 181 : cluster 0 4.b scrub starts 2024-03-11T21:02:33.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:33 smithi023 bash[29146]: cluster 2024-03-11T21:02:31.854695+0000 osd.0 (osd.0) 182 : cluster 0 4.b scrub ok 2024-03-11T21:02:33.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:33 smithi023 bash[29146]: cluster 2024-03-11T21:02:32.131631+0000 mgr.y (mgr.24380) 436 : cluster 0 pgmap v476: 132 pgs: 132 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 705 B/s rd, 0 op/s 2024-03-11T21:02:33.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:33 smithi023 bash[29146]: cluster 2024-03-11T21:02:32.300519+0000 osd.1 (osd.1) 19 : cluster 0 6.1d scrub starts 2024-03-11T21:02:33.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:33 smithi023 bash[29146]: cluster 2024-03-11T21:02:32.330156+0000 osd.1 (osd.1) 20 : cluster 0 6.1d scrub ok 2024-03-11T21:02:33.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:33 smithi023 bash[29146]: cluster 2024-03-11T21:02:32.426498+0000 osd.4 (osd.4) 37 : cluster 0 3.1 scrub starts 2024-03-11T21:02:33.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:33 smithi023 bash[29146]: cluster 2024-03-11T21:02:32.429520+0000 osd.4 (osd.4) 38 : cluster 0 3.1 scrub ok 2024-03-11T21:02:33.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:33 smithi023 bash[29146]: cluster 2024-03-11T21:02:32.918893+0000 osd.7 (osd.7) 245 : cluster 0 6.5 deep-scrub starts 2024-03-11T21:02:33.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:33 smithi023 bash[29146]: cluster 2024-03-11T21:02:32.951098+0000 osd.7 (osd.7) 246 : cluster 0 6.5 deep-scrub ok 2024-03-11T21:02:33.786 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:33 smithi023 bash[21372]: cluster 2024-03-11T21:02:31.689805+0000 osd.2 (osd.2) 160 : cluster 0 2.0 scrub starts 2024-03-11T21:02:33.786 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:33 smithi023 bash[21372]: cluster 2024-03-11T21:02:31.693831+0000 osd.2 (osd.2) 161 : cluster 0 2.0 scrub ok 2024-03-11T21:02:33.786 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:33 smithi023 bash[21372]: cluster 2024-03-11T21:02:31.853215+0000 osd.0 (osd.0) 181 : cluster 0 4.b scrub starts 2024-03-11T21:02:33.790 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:33 smithi023 bash[21372]: cluster 2024-03-11T21:02:31.854695+0000 osd.0 (osd.0) 182 : cluster 0 4.b scrub ok 2024-03-11T21:02:33.790 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:33 smithi023 bash[21372]: cluster 2024-03-11T21:02:32.131631+0000 mgr.y (mgr.24380) 436 : cluster 0 pgmap v476: 132 pgs: 132 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 705 B/s rd, 0 op/s 2024-03-11T21:02:33.790 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:33 smithi023 bash[21372]: cluster 2024-03-11T21:02:32.300519+0000 osd.1 (osd.1) 19 : cluster 0 6.1d scrub starts 2024-03-11T21:02:33.790 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:33 smithi023 bash[21372]: cluster 2024-03-11T21:02:32.330156+0000 osd.1 (osd.1) 20 : cluster 0 6.1d scrub ok 2024-03-11T21:02:33.790 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:33 smithi023 bash[21372]: cluster 2024-03-11T21:02:32.426498+0000 osd.4 (osd.4) 37 : cluster 0 3.1 scrub starts 2024-03-11T21:02:33.790 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:33 smithi023 bash[21372]: cluster 2024-03-11T21:02:32.429520+0000 osd.4 (osd.4) 38 : cluster 0 3.1 scrub ok 2024-03-11T21:02:33.790 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:33 smithi023 bash[21372]: cluster 2024-03-11T21:02:32.918893+0000 osd.7 (osd.7) 245 : cluster 0 6.5 deep-scrub starts 2024-03-11T21:02:33.790 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:33 smithi023 bash[21372]: cluster 2024-03-11T21:02:32.951098+0000 osd.7 (osd.7) 246 : cluster 0 6.5 deep-scrub ok 2024-03-11T21:02:33.852 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:02:33.877 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:02:33.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:33.879 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:02:33.980 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:02:34.006 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:02:34.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:34.008 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:02:34.083 INFO:teuthology.orchestra.run.smithi023.stdout:osd.0: {} 2024-03-11T21:02:34.104 INFO:teuthology.orchestra.run.smithi023.stdout:osd.1: {} 2024-03-11T21:02:34.109 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:02:34.125 INFO:teuthology.orchestra.run.smithi023.stdout:osd.2: {} 2024-03-11T21:02:34.127 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:02:34.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:34.130 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:02:34.146 INFO:teuthology.orchestra.run.smithi023.stdout:osd.3: {} 2024-03-11T21:02:34.170 INFO:teuthology.orchestra.run.smithi023.stdout:osd.4: {} 2024-03-11T21:02:34.194 INFO:teuthology.orchestra.run.smithi023.stdout:osd.5: {} 2024-03-11T21:02:34.231 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:02:34.238 INFO:teuthology.orchestra.run.smithi023.stdout:osd.6: {} 2024-03-11T21:02:34.257 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:02:34.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:34.259 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:02:34.260 INFO:teuthology.orchestra.run.smithi023.stdout:osd.7: {} 2024-03-11T21:02:34.262 INFO:teuthology.orchestra.run.smithi023.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:02:34.262 INFO:teuthology.orchestra.run.smithi023.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:02:34.262 INFO:teuthology.orchestra.run.smithi023.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:02:34.262 INFO:teuthology.orchestra.run.smithi023.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:02:34.262 INFO:teuthology.orchestra.run.smithi023.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:02:34.262 INFO:teuthology.orchestra.run.smithi023.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:02:34.262 INFO:teuthology.orchestra.run.smithi023.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:02:34.262 INFO:teuthology.orchestra.run.smithi023.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:02:34.360 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:34.386 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:34.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:34.389 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:02:34.478 INFO:teuthology.orchestra.run.smithi023.stderr:noscrub is set 2024-03-11T21:02:34.490 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:34.509 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:34.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:34.511 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:02:34.513 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:34 smithi146 bash[23809]: cluster 2024-03-11T21:02:32.814440+0000 osd.0 (osd.0) 183 : cluster 0 3.1b scrub starts 2024-03-11T21:02:34.514 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:34 smithi146 bash[23809]: cluster 2024-03-11T21:02:32.818083+0000 osd.0 (osd.0) 184 : cluster 0 3.1b scrub ok 2024-03-11T21:02:34.514 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:34 smithi146 bash[23809]: cluster 2024-03-11T21:02:32.855877+0000 osd.5 (osd.5) 217 : cluster 0 3.1d scrub starts 2024-03-11T21:02:34.514 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:34 smithi146 bash[23809]: cluster 2024-03-11T21:02:32.859599+0000 osd.5 (osd.5) 218 : cluster 0 3.1d scrub ok 2024-03-11T21:02:34.514 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:34 smithi146 bash[23809]: cluster 2024-03-11T21:02:33.316620+0000 osd.1 (osd.1) 21 : cluster 0 4.4 deep-scrub starts 2024-03-11T21:02:34.514 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:34 smithi146 bash[23809]: cluster 2024-03-11T21:02:33.319210+0000 osd.1 (osd.1) 22 : cluster 0 4.4 deep-scrub ok 2024-03-11T21:02:34.514 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:34 smithi146 bash[23809]: cluster 2024-03-11T21:02:33.384529+0000 osd.4 (osd.4) 39 : cluster 0 6.1e deep-scrub starts 2024-03-11T21:02:34.514 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:34 smithi146 bash[23809]: cluster 2024-03-11T21:02:33.399499+0000 osd.4 (osd.4) 40 : cluster 0 6.1e deep-scrub ok 2024-03-11T21:02:34.514 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:34 smithi146 bash[23809]: audit 2024-03-11T21:02:34.063287+0000 mon.c (mon.2) 56 : audit 0 from='client.? 172.21.15.23:0/2964980874' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:02:34.514 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:34 smithi146 bash[23809]: audit 2024-03-11T21:02:34.369022+0000 mon.c (mon.2) 57 : audit 1 from='client.? 172.21.15.23:0/2181036695' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:02:34.514 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:34 smithi146 bash[23809]: audit 2024-03-11T21:02:34.369426+0000 mon.a (mon.0) 1405 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:02:34.525 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:34 smithi023 bash[21372]: cluster 2024-03-11T21:02:32.814440+0000 osd.0 (osd.0) 183 : cluster 0 3.1b scrub starts 2024-03-11T21:02:34.525 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:34 smithi023 bash[21372]: cluster 2024-03-11T21:02:32.818083+0000 osd.0 (osd.0) 184 : cluster 0 3.1b scrub ok 2024-03-11T21:02:34.525 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:34 smithi023 bash[21372]: cluster 2024-03-11T21:02:32.855877+0000 osd.5 (osd.5) 217 : cluster 0 3.1d scrub starts 2024-03-11T21:02:34.526 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:34 smithi023 bash[21372]: cluster 2024-03-11T21:02:32.859599+0000 osd.5 (osd.5) 218 : cluster 0 3.1d scrub ok 2024-03-11T21:02:34.526 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:34 smithi023 bash[21372]: cluster 2024-03-11T21:02:33.316620+0000 osd.1 (osd.1) 21 : cluster 0 4.4 deep-scrub starts 2024-03-11T21:02:34.526 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:34 smithi023 bash[21372]: cluster 2024-03-11T21:02:33.319210+0000 osd.1 (osd.1) 22 : cluster 0 4.4 deep-scrub ok 2024-03-11T21:02:34.526 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:34 smithi023 bash[21372]: cluster 2024-03-11T21:02:33.384529+0000 osd.4 (osd.4) 39 : cluster 0 6.1e deep-scrub starts 2024-03-11T21:02:34.526 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:34 smithi023 bash[21372]: cluster 2024-03-11T21:02:33.399499+0000 osd.4 (osd.4) 40 : cluster 0 6.1e deep-scrub ok 2024-03-11T21:02:34.526 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:34 smithi023 bash[21372]: audit 2024-03-11T21:02:34.063287+0000 mon.c (mon.2) 56 : audit 0 from='client.? 172.21.15.23:0/2964980874' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:02:34.526 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:34 smithi023 bash[21372]: audit 2024-03-11T21:02:34.369022+0000 mon.c (mon.2) 57 : audit 1 from='client.? 172.21.15.23:0/2181036695' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:02:34.526 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:34 smithi023 bash[21372]: audit 2024-03-11T21:02:34.369426+0000 mon.a (mon.0) 1405 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:02:34.526 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:34 smithi023 bash[29146]: cluster 2024-03-11T21:02:32.814440+0000 osd.0 (osd.0) 183 : cluster 0 3.1b scrub starts 2024-03-11T21:02:34.527 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:34 smithi023 bash[29146]: cluster 2024-03-11T21:02:32.818083+0000 osd.0 (osd.0) 184 : cluster 0 3.1b scrub ok 2024-03-11T21:02:34.527 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:34 smithi023 bash[29146]: cluster 2024-03-11T21:02:32.855877+0000 osd.5 (osd.5) 217 : cluster 0 3.1d scrub starts 2024-03-11T21:02:34.527 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:34 smithi023 bash[29146]: cluster 2024-03-11T21:02:32.859599+0000 osd.5 (osd.5) 218 : cluster 0 3.1d scrub ok 2024-03-11T21:02:34.527 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:34 smithi023 bash[29146]: cluster 2024-03-11T21:02:33.316620+0000 osd.1 (osd.1) 21 : cluster 0 4.4 deep-scrub starts 2024-03-11T21:02:34.527 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:34 smithi023 bash[29146]: cluster 2024-03-11T21:02:33.319210+0000 osd.1 (osd.1) 22 : cluster 0 4.4 deep-scrub ok 2024-03-11T21:02:34.527 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:34 smithi023 bash[29146]: cluster 2024-03-11T21:02:33.384529+0000 osd.4 (osd.4) 39 : cluster 0 6.1e deep-scrub starts 2024-03-11T21:02:34.527 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:34 smithi023 bash[29146]: cluster 2024-03-11T21:02:33.399499+0000 osd.4 (osd.4) 40 : cluster 0 6.1e deep-scrub ok 2024-03-11T21:02:34.527 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:34 smithi023 bash[29146]: audit 2024-03-11T21:02:34.063287+0000 mon.c (mon.2) 56 : audit 0 from='client.? 172.21.15.23:0/2964980874' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:02:34.527 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:34 smithi023 bash[29146]: audit 2024-03-11T21:02:34.369022+0000 mon.c (mon.2) 57 : audit 1 from='client.? 172.21.15.23:0/2181036695' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:02:34.527 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:34 smithi023 bash[29146]: audit 2024-03-11T21:02:34.369426+0000 mon.a (mon.0) 1405 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:02:34.612 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:34.632 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:34.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:34.635 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:02:34.736 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:02:34.765 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:02:34.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:34.767 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:02:34.868 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:34.897 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:34.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:34.899 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:02:35.000 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:02:35.027 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:02:35.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:35.029 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:02:35.130 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:35.156 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:35.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:35.158 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:02:35.259 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:35.287 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:35.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:35.290 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:02:35.391 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:35.417 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:35.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:35.423 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:02:35.524 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:02:35.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:35 smithi023 bash[21372]: cluster 2024-03-11T21:02:33.693238+0000 osd.2 (osd.2) 162 : cluster 0 6.f scrub starts 2024-03-11T21:02:35.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:35 smithi023 bash[21372]: cluster 2024-03-11T21:02:33.767903+0000 osd.2 (osd.2) 163 : cluster 0 6.f scrub ok 2024-03-11T21:02:35.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:35 smithi023 bash[21372]: cluster 2024-03-11T21:02:34.132056+0000 mgr.y (mgr.24380) 437 : cluster 0 pgmap v477: 132 pgs: 132 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-11T21:02:35.536 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:35 smithi023 bash[21372]: cluster 2024-03-11T21:02:34.455501+0000 mon.a (mon.0) 1406 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:02:35.536 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:35 smithi023 bash[21372]: audit 2024-03-11T21:02:34.461826+0000 mon.a (mon.0) 1407 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:02:35.536 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:35 smithi023 bash[21372]: cluster 2024-03-11T21:02:34.480633+0000 mon.a (mon.0) 1408 : cluster 0 osdmap e246: 8 total, 8 up, 8 in 2024-03-11T21:02:35.536 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:35 smithi023 bash[21372]: audit 2024-03-11T21:02:35.060984+0000 mon.a (mon.0) 1409 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:02:35.536 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:35 smithi023 bash[21372]: audit 2024-03-11T21:02:35.063355+0000 mon.a (mon.0) 1410 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:02:35.536 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:35 smithi023 bash[21372]: cluster 2024-03-11T21:02:35.191546+0000 mon.a (mon.0) 1411 : cluster 0 osdmap e247: 8 total, 8 up, 8 in 2024-03-11T21:02:35.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:35 smithi023 bash[29146]: cluster 2024-03-11T21:02:33.693238+0000 osd.2 (osd.2) 162 : cluster 0 6.f scrub starts 2024-03-11T21:02:35.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:35 smithi023 bash[29146]: cluster 2024-03-11T21:02:33.767903+0000 osd.2 (osd.2) 163 : cluster 0 6.f scrub ok 2024-03-11T21:02:35.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:35 smithi023 bash[29146]: cluster 2024-03-11T21:02:34.132056+0000 mgr.y (mgr.24380) 437 : cluster 0 pgmap v477: 132 pgs: 132 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-11T21:02:35.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:35 smithi023 bash[29146]: cluster 2024-03-11T21:02:34.455501+0000 mon.a (mon.0) 1406 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:02:35.537 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:35 smithi023 bash[29146]: audit 2024-03-11T21:02:34.461826+0000 mon.a (mon.0) 1407 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:02:35.537 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:35 smithi023 bash[29146]: cluster 2024-03-11T21:02:34.480633+0000 mon.a (mon.0) 1408 : cluster 0 osdmap e246: 8 total, 8 up, 8 in 2024-03-11T21:02:35.537 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:35 smithi023 bash[29146]: audit 2024-03-11T21:02:35.060984+0000 mon.a (mon.0) 1409 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:02:35.537 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:35 smithi023 bash[29146]: audit 2024-03-11T21:02:35.063355+0000 mon.a (mon.0) 1410 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:02:35.537 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:35 smithi023 bash[29146]: cluster 2024-03-11T21:02:35.191546+0000 mon.a (mon.0) 1411 : cluster 0 osdmap e247: 8 total, 8 up, 8 in 2024-03-11T21:02:35.552 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:02:35.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:35.555 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:02:35.655 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:02:35.667 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:02:35 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:02:35.667 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:35 smithi146 bash[23809]: cluster 2024-03-11T21:02:33.693238+0000 osd.2 (osd.2) 162 : cluster 0 6.f scrub starts 2024-03-11T21:02:35.667 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:35 smithi146 bash[23809]: cluster 2024-03-11T21:02:33.767903+0000 osd.2 (osd.2) 163 : cluster 0 6.f scrub ok 2024-03-11T21:02:35.667 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:35 smithi146 bash[23809]: cluster 2024-03-11T21:02:34.132056+0000 mgr.y (mgr.24380) 437 : cluster 0 pgmap v477: 132 pgs: 132 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-11T21:02:35.667 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:35 smithi146 bash[23809]: cluster 2024-03-11T21:02:34.455501+0000 mon.a (mon.0) 1406 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:02:35.667 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:35 smithi146 bash[23809]: audit 2024-03-11T21:02:34.461826+0000 mon.a (mon.0) 1407 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:02:35.667 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:35 smithi146 bash[23809]: cluster 2024-03-11T21:02:34.480633+0000 mon.a (mon.0) 1408 : cluster 0 osdmap e246: 8 total, 8 up, 8 in 2024-03-11T21:02:35.668 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:35 smithi146 bash[23809]: audit 2024-03-11T21:02:35.060984+0000 mon.a (mon.0) 1409 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:02:35.668 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:35 smithi146 bash[23809]: audit 2024-03-11T21:02:35.063355+0000 mon.a (mon.0) 1410 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:02:35.668 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:35 smithi146 bash[23809]: cluster 2024-03-11T21:02:35.191546+0000 mon.a (mon.0) 1411 : cluster 0 osdmap e247: 8 total, 8 up, 8 in 2024-03-11T21:02:35.683 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:02:35.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:35.685 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:02:35.786 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:02:35.817 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:02:35.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:35.820 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:02:35.921 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:35.950 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:35.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:35.953 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:02:36.055 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:02:36.081 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:02:36.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:36.083 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:02:36.184 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:36.210 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:36.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:36.212 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:02:36.313 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:02:36.342 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:02:36.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:36.345 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:02:36.445 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:02:36.473 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:02:36.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:36.475 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:02:36.576 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:36.605 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:36.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:36.608 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:02:36.709 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:02:36.740 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:02:36.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:36.742 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:02:36.764 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:36 smithi146 bash[23809]: cluster 2024-03-11T21:02:34.744140+0000 osd.2 (osd.2) 164 : cluster 0 6.14 scrub starts 2024-03-11T21:02:36.764 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:36 smithi146 bash[23809]: audit 2024-03-11T21:02:35.399655+0000 mgr.y (mgr.24380) 438 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:02:36.764 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:36 smithi146 bash[23809]: audit 2024-03-11T21:02:35.838347+0000 mon.a (mon.0) 1412 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]: dispatch 2024-03-11T21:02:36.764 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:36 smithi146 bash[23809]: audit 2024-03-11T21:02:35.838704+0000 mon.a (mon.0) 1413 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-03-11T21:02:36.764 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:36 smithi146 bash[23809]: audit 2024-03-11T21:02:35.839015+0000 mon.a (mon.0) 1414 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 2]}]: dispatch 2024-03-11T21:02:36.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:36 smithi023 bash[29146]: cluster 2024-03-11T21:02:34.744140+0000 osd.2 (osd.2) 164 : cluster 0 6.14 scrub starts 2024-03-11T21:02:36.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:36 smithi023 bash[29146]: audit 2024-03-11T21:02:35.399655+0000 mgr.y (mgr.24380) 438 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:02:36.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:36 smithi023 bash[29146]: audit 2024-03-11T21:02:35.838347+0000 mon.a (mon.0) 1412 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]: dispatch 2024-03-11T21:02:36.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:36 smithi023 bash[29146]: audit 2024-03-11T21:02:35.838704+0000 mon.a (mon.0) 1413 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-03-11T21:02:36.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:36 smithi023 bash[29146]: audit 2024-03-11T21:02:35.839015+0000 mon.a (mon.0) 1414 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 2]}]: dispatch 2024-03-11T21:02:36.786 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:36 smithi023 bash[21372]: cluster 2024-03-11T21:02:34.744140+0000 osd.2 (osd.2) 164 : cluster 0 6.14 scrub starts 2024-03-11T21:02:36.786 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:36 smithi023 bash[21372]: audit 2024-03-11T21:02:35.399655+0000 mgr.y (mgr.24380) 438 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:02:36.786 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:36 smithi023 bash[21372]: audit 2024-03-11T21:02:35.838347+0000 mon.a (mon.0) 1412 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]: dispatch 2024-03-11T21:02:36.786 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:36 smithi023 bash[21372]: audit 2024-03-11T21:02:35.838704+0000 mon.a (mon.0) 1413 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-03-11T21:02:36.786 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:36 smithi023 bash[21372]: audit 2024-03-11T21:02:35.839015+0000 mon.a (mon.0) 1414 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 2]}]: dispatch 2024-03-11T21:02:36.844 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:36.871 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:36.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:36.874 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:02:36.974 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:37.003 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:37.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:37.006 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:02:37.107 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:02:37.135 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:02:37.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:37.137 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:02:37.238 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:37.264 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:37.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:37.267 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:02:37.368 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:37.400 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:37.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:37.403 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:02:37.504 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:37.533 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:37.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:37.535 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:02:37.636 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:37.662 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:37.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:37.665 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:02:37.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:37 smithi146 bash[23809]: cluster 2024-03-11T21:02:36.133238+0000 mgr.y (mgr.24380) 439 : cluster 0 pgmap v480: 132 pgs: 132 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-11T21:02:37.764 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:37 smithi146 bash[23809]: audit 2024-03-11T21:02:36.491102+0000 mon.a (mon.0) 1415 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]': finished 2024-03-11T21:02:37.764 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:37 smithi146 bash[23809]: audit 2024-03-11T21:02:36.491280+0000 mon.a (mon.0) 1416 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]': finished 2024-03-11T21:02:37.764 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:37 smithi146 bash[23809]: audit 2024-03-11T21:02:36.491516+0000 mon.a (mon.0) 1417 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 2]}]': finished 2024-03-11T21:02:37.764 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:37 smithi146 bash[23809]: cluster 2024-03-11T21:02:36.500003+0000 mon.a (mon.0) 1418 : cluster 0 osdmap e248: 8 total, 8 up, 8 in 2024-03-11T21:02:37.765 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:02:37.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:37 smithi023 bash[21372]: cluster 2024-03-11T21:02:36.133238+0000 mgr.y (mgr.24380) 439 : cluster 0 pgmap v480: 132 pgs: 132 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-11T21:02:37.786 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:37 smithi023 bash[21372]: audit 2024-03-11T21:02:36.491102+0000 mon.a (mon.0) 1415 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]': finished 2024-03-11T21:02:37.786 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:37 smithi023 bash[21372]: audit 2024-03-11T21:02:36.491280+0000 mon.a (mon.0) 1416 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]': finished 2024-03-11T21:02:37.786 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:37 smithi023 bash[21372]: audit 2024-03-11T21:02:36.491516+0000 mon.a (mon.0) 1417 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 2]}]': finished 2024-03-11T21:02:37.786 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:37 smithi023 bash[21372]: cluster 2024-03-11T21:02:36.500003+0000 mon.a (mon.0) 1418 : cluster 0 osdmap e248: 8 total, 8 up, 8 in 2024-03-11T21:02:37.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:37 smithi023 bash[29146]: cluster 2024-03-11T21:02:36.133238+0000 mgr.y (mgr.24380) 439 : cluster 0 pgmap v480: 132 pgs: 132 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-11T21:02:37.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:37 smithi023 bash[29146]: audit 2024-03-11T21:02:36.491102+0000 mon.a (mon.0) 1415 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]': finished 2024-03-11T21:02:37.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:37 smithi023 bash[29146]: audit 2024-03-11T21:02:36.491280+0000 mon.a (mon.0) 1416 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]': finished 2024-03-11T21:02:37.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:37 smithi023 bash[29146]: audit 2024-03-11T21:02:36.491516+0000 mon.a (mon.0) 1417 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 2]}]': finished 2024-03-11T21:02:37.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:37 smithi023 bash[29146]: cluster 2024-03-11T21:02:36.500003+0000 mon.a (mon.0) 1418 : cluster 0 osdmap e248: 8 total, 8 up, 8 in 2024-03-11T21:02:37.792 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:02:37.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:37.794 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:02:37.895 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:37.922 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:37.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:37.925 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:02:37.996 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T21:02:38.025 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:02:38.053 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:02:38.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:38.055 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:02:38.156 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:02:38.181 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:02:38.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:38.183 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:02:38.284 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:38.311 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:38.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:38.313 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:02:38.333 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:02:38.414 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:38.414 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-11T21:02:38.440 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:38.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:38.443 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:02:38.543 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:02:38.569 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:02:38.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:38.571 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:02:38.672 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:02:38.686 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:38 smithi146 bash[23809]: cluster 2024-03-11T21:02:37.504081+0000 mon.a (mon.0) 1419 : cluster 0 osdmap e249: 8 total, 8 up, 8 in 2024-03-11T21:02:38.700 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:02:38.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:38.703 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:02:38.785 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:02:38 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:02:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:02:38.786 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:38 smithi023 bash[21372]: cluster 2024-03-11T21:02:37.504081+0000 mon.a (mon.0) 1419 : cluster 0 osdmap e249: 8 total, 8 up, 8 in 2024-03-11T21:02:38.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:38 smithi023 bash[29146]: cluster 2024-03-11T21:02:37.504081+0000 mon.a (mon.0) 1419 : cluster 0 osdmap e249: 8 total, 8 up, 8 in 2024-03-11T21:02:38.803 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:38.833 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:38.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:38.835 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:02:38.936 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:38.966 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:38.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:38.968 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:02:39.069 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:39.098 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:39.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:39.101 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:02:39.202 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:02:39.227 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:02:39.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:39.229 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:02:39.330 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:39.358 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:39.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:39.360 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:02:39.461 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:02:39.490 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:02:39.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:39.492 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:02:39.593 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:39.606 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:39 smithi023 bash[21372]: cluster 2024-03-11T21:02:38.133914+0000 mgr.y (mgr.24380) 440 : cluster 0 pgmap v483: 132 pgs: 132 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:02:39.606 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:39 smithi023 bash[29146]: cluster 2024-03-11T21:02:38.133914+0000 mgr.y (mgr.24380) 440 : cluster 0 pgmap v483: 132 pgs: 132 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:02:39.619 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:39.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:39.622 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:02:39.723 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:39.748 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:39.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:39.750 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:02:39.851 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:02:39.862 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:39 smithi146 bash[23809]: cluster 2024-03-11T21:02:38.133914+0000 mgr.y (mgr.24380) 440 : cluster 0 pgmap v483: 132 pgs: 132 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:02:39.877 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:02:39.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:39.879 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:02:39.949 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:02:39.950 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:02:39.980 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:40.008 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:40.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:40.011 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:02:40.111 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:02:40.137 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:02:40.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:40.139 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:02:40.240 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:02:40.262 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:02:40.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:40.264 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:02:40.364 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:40.392 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:40.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:40.394 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:02:40.495 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:02:40.521 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:02:40.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:40.526 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:02:40.627 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:02:40.651 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:02:40.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:40.653 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:02:40.754 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:40.783 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:40.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:40.786 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:02:40.887 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:02:40.909 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:02:40.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:40.911 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:02:41.012 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:02:41.034 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:02:41.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:41.037 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:02:41.138 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:02:41.163 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:02:41.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:41.166 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:02:41.266 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:41.291 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:41.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:41.293 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:02:41.394 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:02:41.421 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:02:41.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:41.424 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:02:41.524 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:41.547 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:41.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:41.549 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:02:41.650 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:41.679 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:41.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:41.681 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:02:41.764 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:41 smithi146 bash[23809]: cluster 2024-03-11T21:02:40.134577+0000 mgr.y (mgr.24380) 441 : cluster 0 pgmap v484: 132 pgs: 132 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 904 B/s rd, 0 op/s 2024-03-11T21:02:41.782 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:02:41.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:41 smithi023 bash[29146]: cluster 2024-03-11T21:02:40.134577+0000 mgr.y (mgr.24380) 441 : cluster 0 pgmap v484: 132 pgs: 132 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 904 B/s rd, 0 op/s 2024-03-11T21:02:41.786 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:41 smithi023 bash[21372]: cluster 2024-03-11T21:02:40.134577+0000 mgr.y (mgr.24380) 441 : cluster 0 pgmap v484: 132 pgs: 132 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 904 B/s rd, 0 op/s 2024-03-11T21:02:41.811 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:02:41.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:41.813 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:02:41.914 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:02:41.933 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:02:41.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:41.935 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:02:41.983 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-03-11T21:02:41.984 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:02:42.036 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:42.066 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:42.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:42.069 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:02:42.170 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:02:42.198 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:02:42.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:42.200 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:02:42.301 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:02:42.328 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:02:42.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:42.331 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:02:42.431 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:02:42.458 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:02:42.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:42.461 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:02:42.561 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:42.587 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:42.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:42.590 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:02:42.690 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:02:42.720 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:02:42.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:42.722 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:02:42.823 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:02:42.826 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-03-11T21:02:42.892 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:02:42.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:42.894 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:02:42.995 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:43.022 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:43.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:43.024 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:02:43.125 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:02:43.150 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:02:43.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:43.153 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:02:43.254 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:43.282 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:43.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:43.284 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:02:43.363 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:02:43.364 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:02:43.385 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:02:43.413 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:02:43.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:43.416 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:02:43.517 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:43.544 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:43.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:43.547 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:02:43.648 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:43.659 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:43 smithi023 bash[21372]: cluster 2024-03-11T21:02:42.135804+0000 mgr.y (mgr.24380) 442 : cluster 0 pgmap v485: 132 pgs: 132 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-03-11T21:02:43.659 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:43 smithi023 bash[29146]: cluster 2024-03-11T21:02:42.135804+0000 mgr.y (mgr.24380) 442 : cluster 0 pgmap v485: 132 pgs: 132 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-03-11T21:02:43.675 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:43.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:43.677 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:02:43.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:43 smithi146 bash[23809]: cluster 2024-03-11T21:02:42.135804+0000 mgr.y (mgr.24380) 442 : cluster 0 pgmap v485: 132 pgs: 132 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-03-11T21:02:43.778 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:02:43.804 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:02:43.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:43.807 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:02:43.908 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:02:43.934 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:02:43.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:43.936 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:02:44.037 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:02:44.064 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:02:44.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:44.067 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:02:44.167 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:02:44.189 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:02:44.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:44.192 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:02:44.293 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:44.303 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 21:02:44 smithi146 bash[51661]: logger=cleanup t=2024-03-11T21:02:44.137600335Z level=info msg="Completed cleanup jobs" duration=83.758141ms 2024-03-11T21:02:44.321 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:44.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:44.323 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:02:44.424 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:44.449 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:44.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:44.452 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:02:44.552 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:44.582 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:44.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:44.584 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:02:44.685 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:02:44.712 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:02:44.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:44.715 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:02:44.815 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:44.841 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:44.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:44.843 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:02:44.944 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:44.972 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:44.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:44.974 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:02:45.075 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:02:45.097 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:02:45.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:45.099 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:02:45.200 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:45.225 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:45.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:45.228 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:02:45.328 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:45.363 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:45.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:45.366 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:02:45.466 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:02:45.492 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:02:45.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:45.495 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:02:45.566 INFO:teuthology.orchestra.run.smithi023.stderr:reweighted osd.4 to 1 (10000) 2024-03-11T21:02:45.578 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:45 smithi023 bash[21372]: cluster 2024-03-11T21:02:44.136461+0000 mgr.y (mgr.24380) 443 : cluster 0 pgmap v486: 132 pgs: 132 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-03-11T21:02:45.579 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:45 smithi023 bash[21372]: audit 2024-03-11T21:02:45.463910+0000 mon.c (mon.2) 58 : audit 1 from='client.? 172.21.15.23:0/3177839742' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-11T21:02:45.579 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:45 smithi023 bash[21372]: audit 2024-03-11T21:02:45.464295+0000 mon.a (mon.0) 1420 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-11T21:02:45.579 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:45 smithi023 bash[29146]: cluster 2024-03-11T21:02:44.136461+0000 mgr.y (mgr.24380) 443 : cluster 0 pgmap v486: 132 pgs: 132 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-03-11T21:02:45.579 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:45 smithi023 bash[29146]: audit 2024-03-11T21:02:45.463910+0000 mon.c (mon.2) 58 : audit 1 from='client.? 172.21.15.23:0/3177839742' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-11T21:02:45.579 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:45 smithi023 bash[29146]: audit 2024-03-11T21:02:45.464295+0000 mon.a (mon.0) 1420 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-11T21:02:45.596 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:02:45.604 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:02:45 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:02:45.605 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:45 smithi146 bash[23809]: cluster 2024-03-11T21:02:44.136461+0000 mgr.y (mgr.24380) 443 : cluster 0 pgmap v486: 132 pgs: 132 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-03-11T21:02:45.605 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:45 smithi146 bash[23809]: audit 2024-03-11T21:02:45.463910+0000 mon.c (mon.2) 58 : audit 1 from='client.? 172.21.15.23:0/3177839742' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-11T21:02:45.605 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:45 smithi146 bash[23809]: audit 2024-03-11T21:02:45.464295+0000 mon.a (mon.0) 1420 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-11T21:02:45.618 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:02:45.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:45.620 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:02:45.721 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:45.741 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:45.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:45.744 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:02:45.844 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:45.870 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:45.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:45.872 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:02:45.973 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:02:46.001 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:02:46.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:46.003 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:02:46.104 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:02:46.133 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:02:46.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:46.135 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:02:46.236 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:46.262 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:46.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:46.264 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:02:46.364 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:46.394 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:46.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:46.396 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:02:46.497 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:46.525 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:46.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:46.528 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:02:46.600 INFO:teuthology.orchestra.run.smithi023.stderr:pool 'unique_pool_2' created 2024-03-11T21:02:46.628 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:46.636 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:46 smithi023 bash[21372]: audit 2024-03-11T21:02:45.407328+0000 mgr.y (mgr.24380) 444 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:02:46.637 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:46 smithi023 bash[21372]: audit 2024-03-11T21:02:45.562197+0000 mon.a (mon.0) 1421 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-11T21:02:46.637 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:46 smithi023 bash[21372]: cluster 2024-03-11T21:02:45.565769+0000 mon.a (mon.0) 1422 : cluster 0 osdmap e250: 8 total, 8 up, 8 in 2024-03-11T21:02:46.637 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:46 smithi023 bash[21372]: audit 2024-03-11T21:02:45.729947+0000 mon.a (mon.0) 1423 : audit 1 from='client.? 172.21.15.23:0/1551635890' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-03-11T21:02:46.637 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:46 smithi023 bash[21372]: cluster 2024-03-11T21:02:46.137170+0000 mgr.y (mgr.24380) 445 : cluster 0 pgmap v488: 132 pgs: 132 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-03-11T21:02:46.637 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:46 smithi023 bash[29146]: audit 2024-03-11T21:02:45.407328+0000 mgr.y (mgr.24380) 444 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:02:46.637 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:46 smithi023 bash[29146]: audit 2024-03-11T21:02:45.562197+0000 mon.a (mon.0) 1421 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-11T21:02:46.637 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:46 smithi023 bash[29146]: cluster 2024-03-11T21:02:45.565769+0000 mon.a (mon.0) 1422 : cluster 0 osdmap e250: 8 total, 8 up, 8 in 2024-03-11T21:02:46.637 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:46 smithi023 bash[29146]: audit 2024-03-11T21:02:45.729947+0000 mon.a (mon.0) 1423 : audit 1 from='client.? 172.21.15.23:0/1551635890' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-03-11T21:02:46.637 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:46 smithi023 bash[29146]: cluster 2024-03-11T21:02:46.137170+0000 mgr.y (mgr.24380) 445 : cluster 0 pgmap v488: 132 pgs: 132 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-03-11T21:02:46.648 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:46.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:46.650 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:02:46.751 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:46.777 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:46.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:46.779 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:02:46.880 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:02:46.891 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:46 smithi146 bash[23809]: audit 2024-03-11T21:02:45.407328+0000 mgr.y (mgr.24380) 444 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:02:46.891 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:46 smithi146 bash[23809]: audit 2024-03-11T21:02:45.562197+0000 mon.a (mon.0) 1421 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-11T21:02:46.891 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:46 smithi146 bash[23809]: cluster 2024-03-11T21:02:45.565769+0000 mon.a (mon.0) 1422 : cluster 0 osdmap e250: 8 total, 8 up, 8 in 2024-03-11T21:02:46.891 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:46 smithi146 bash[23809]: audit 2024-03-11T21:02:45.729947+0000 mon.a (mon.0) 1423 : audit 1 from='client.? 172.21.15.23:0/1551635890' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-03-11T21:02:46.891 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:46 smithi146 bash[23809]: cluster 2024-03-11T21:02:46.137170+0000 mgr.y (mgr.24380) 445 : cluster 0 pgmap v488: 132 pgs: 132 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-03-11T21:02:46.908 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:02:46.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:46.910 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:02:47.011 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:02:47.036 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:02:47.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:47.038 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:02:47.139 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:47.164 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:47.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:47.166 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:02:47.267 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:02:47.293 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:02:47.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:47.295 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:02:47.396 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:02:47.423 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:02:47.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:47.425 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:02:47.526 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:02:47.553 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:02:47.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:47.555 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:02:47.642 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:02:47.656 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:02:47.660 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:47 smithi146 bash[23809]: cluster 2024-03-11T21:02:46.346856+0000 osd.7 (osd.7) 247 : cluster 0 purged_snaps scrub starts 2024-03-11T21:02:47.660 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:47 smithi146 bash[23809]: cluster 2024-03-11T21:02:46.346969+0000 osd.7 (osd.7) 248 : cluster 0 purged_snaps scrub ok 2024-03-11T21:02:47.661 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:47 smithi146 bash[23809]: audit 2024-03-11T21:02:46.586758+0000 mon.a (mon.0) 1424 : audit 1 from='client.? 172.21.15.23:0/1551635890' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-03-11T21:02:47.661 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:47 smithi146 bash[23809]: cluster 2024-03-11T21:02:46.595792+0000 mon.a (mon.0) 1425 : cluster 0 osdmap e251: 8 total, 8 up, 8 in 2024-03-11T21:02:47.677 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:02:47.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:47.679 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:02:47.780 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:47.808 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:47.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:47.809 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:02:47.910 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:02:47.922 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:47 smithi023 bash[21372]: cluster 2024-03-11T21:02:46.346856+0000 osd.7 (osd.7) 247 : cluster 0 purged_snaps scrub starts 2024-03-11T21:02:47.922 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:47 smithi023 bash[21372]: cluster 2024-03-11T21:02:46.346969+0000 osd.7 (osd.7) 248 : cluster 0 purged_snaps scrub ok 2024-03-11T21:02:47.922 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:47 smithi023 bash[21372]: audit 2024-03-11T21:02:46.586758+0000 mon.a (mon.0) 1424 : audit 1 from='client.? 172.21.15.23:0/1551635890' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-03-11T21:02:47.922 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:47 smithi023 bash[21372]: cluster 2024-03-11T21:02:46.595792+0000 mon.a (mon.0) 1425 : cluster 0 osdmap e251: 8 total, 8 up, 8 in 2024-03-11T21:02:47.923 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:47 smithi023 bash[29146]: cluster 2024-03-11T21:02:46.346856+0000 osd.7 (osd.7) 247 : cluster 0 purged_snaps scrub starts 2024-03-11T21:02:47.923 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:47 smithi023 bash[29146]: cluster 2024-03-11T21:02:46.346969+0000 osd.7 (osd.7) 248 : cluster 0 purged_snaps scrub ok 2024-03-11T21:02:47.923 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:47 smithi023 bash[29146]: audit 2024-03-11T21:02:46.586758+0000 mon.a (mon.0) 1424 : audit 1 from='client.? 172.21.15.23:0/1551635890' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-03-11T21:02:47.923 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:47 smithi023 bash[29146]: cluster 2024-03-11T21:02:46.595792+0000 mon.a (mon.0) 1425 : cluster 0 osdmap e251: 8 total, 8 up, 8 in 2024-03-11T21:02:47.938 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:02:47.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:47.940 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:02:48.038 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd reweight 5 1 2024-03-11T21:02:48.041 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:48.067 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:48.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:48.070 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:02:48.170 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:02:48.196 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:02:48.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:48.199 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:02:48.299 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:02:48.326 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:02:48.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:48.328 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:02:48.349 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd pool application enable unique_pool_2 rados --yes-i-really-mean-it || true 2024-03-11T21:02:48.429 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:48.449 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:48.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:48.451 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:02:48.552 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:48.571 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:48.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:48.574 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:02:48.638 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:02:48 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:02:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:02:48.674 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:48.700 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:48.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:48.702 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:02:48.803 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:48.832 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:48.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:48.835 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:02:48.929 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:48 smithi146 bash[23809]: cluster 2024-03-11T21:02:47.604627+0000 mon.a (mon.0) 1426 : cluster 0 osdmap e252: 8 total, 8 up, 8 in 2024-03-11T21:02:48.929 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:48 smithi146 bash[23809]: cluster 2024-03-11T21:02:48.137900+0000 mgr.y (mgr.24380) 446 : cluster 0 pgmap v491: 148 pgs: 2 creating+peering, 14 unknown, 132 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:02:48.935 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:02:48.946 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:48 smithi023 bash[21372]: cluster 2024-03-11T21:02:47.604627+0000 mon.a (mon.0) 1426 : cluster 0 osdmap e252: 8 total, 8 up, 8 in 2024-03-11T21:02:48.946 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:48 smithi023 bash[21372]: cluster 2024-03-11T21:02:48.137900+0000 mgr.y (mgr.24380) 446 : cluster 0 pgmap v491: 148 pgs: 2 creating+peering, 14 unknown, 132 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:02:48.947 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:48 smithi023 bash[29146]: cluster 2024-03-11T21:02:47.604627+0000 mon.a (mon.0) 1426 : cluster 0 osdmap e252: 8 total, 8 up, 8 in 2024-03-11T21:02:48.947 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:48 smithi023 bash[29146]: cluster 2024-03-11T21:02:48.137900+0000 mgr.y (mgr.24380) 446 : cluster 0 pgmap v491: 148 pgs: 2 creating+peering, 14 unknown, 132 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:02:48.963 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:02:48.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:48.965 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:02:49.066 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:02:49.088 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:02:49.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:49.090 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:02:49.191 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:02:49.219 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:02:49.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:49.222 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:02:49.323 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:02:49.349 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:02:49.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:49.351 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:02:49.452 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:49.482 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:49.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:49.484 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:02:49.585 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:02:49.611 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:02:49.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:49.613 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:02:49.714 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:02:49.728 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:49 smithi023 bash[29146]: cluster 2024-03-11T21:02:48.615177+0000 mon.a (mon.0) 1427 : cluster 3 Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:02:49.728 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:49 smithi023 bash[29146]: cluster 2024-03-11T21:02:48.628348+0000 mon.a (mon.0) 1428 : cluster 0 osdmap e253: 8 total, 8 up, 8 in 2024-03-11T21:02:49.728 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:49 smithi023 bash[21372]: cluster 2024-03-11T21:02:48.615177+0000 mon.a (mon.0) 1427 : cluster 3 Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:02:49.728 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:49 smithi023 bash[21372]: cluster 2024-03-11T21:02:48.628348+0000 mon.a (mon.0) 1428 : cluster 0 osdmap e253: 8 total, 8 up, 8 in 2024-03-11T21:02:49.744 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:02:49.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:49.747 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:02:49.848 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:02:49.861 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:49 smithi146 bash[23809]: cluster 2024-03-11T21:02:48.615177+0000 mon.a (mon.0) 1427 : cluster 3 Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:02:49.861 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:49 smithi146 bash[23809]: cluster 2024-03-11T21:02:48.628348+0000 mon.a (mon.0) 1428 : cluster 0 osdmap e253: 8 total, 8 up, 8 in 2024-03-11T21:02:49.877 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:02:49.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:49.879 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:02:49.980 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:50.014 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:50.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:50.017 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:02:50.118 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:50.152 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:50.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:50.154 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:02:50.255 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:50.278 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:50.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:50.281 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:02:50.382 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:02:50.398 INFO:teuthology.orchestra.run.smithi023.stdout:osd.0: {} 2024-03-11T21:02:50.414 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:02:50.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:50.416 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:02:50.419 INFO:teuthology.orchestra.run.smithi023.stdout:osd.1: {} 2024-03-11T21:02:50.440 INFO:teuthology.orchestra.run.smithi023.stdout:osd.2: {} 2024-03-11T21:02:50.462 INFO:teuthology.orchestra.run.smithi023.stdout:osd.3: {} 2024-03-11T21:02:50.486 INFO:teuthology.orchestra.run.smithi023.stdout:osd.4: {} 2024-03-11T21:02:50.511 INFO:teuthology.orchestra.run.smithi023.stdout:osd.5: {} 2024-03-11T21:02:50.516 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:02:50.536 INFO:teuthology.orchestra.run.smithi023.stdout:osd.6: {} 2024-03-11T21:02:50.546 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:02:50.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:50.549 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:02:50.558 INFO:teuthology.orchestra.run.smithi023.stdout:osd.7: {} 2024-03-11T21:02:50.559 INFO:teuthology.orchestra.run.smithi023.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:02:50.559 INFO:teuthology.orchestra.run.smithi023.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:02:50.559 INFO:teuthology.orchestra.run.smithi023.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:02:50.559 INFO:teuthology.orchestra.run.smithi023.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:02:50.559 INFO:teuthology.orchestra.run.smithi023.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:02:50.559 INFO:teuthology.orchestra.run.smithi023.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:02:50.560 INFO:teuthology.orchestra.run.smithi023.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:02:50.560 INFO:teuthology.orchestra.run.smithi023.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:02:50.650 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:50.673 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:50.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:50.675 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:02:50.776 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:50.805 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:50.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:50.808 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:02:50.908 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:50.935 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:50.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:50.938 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:02:51.010 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-03-11T21:02:51.010 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-03-11T21:02:51.011 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 30, 2024-03-11T21:02:51.011 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-03-11T21:02:51.011 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-03-11T21:02:51.039 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:02:51.062 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:02:51.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:51.064 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:02:51.130 INFO:teuthology.orchestra.run.smithi023.stderr:nodeep-scrub is set 2024-03-11T21:02:51.165 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:51.189 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:51.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:51.192 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:02:51.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:51 smithi023 bash[21372]: audit 2024-03-11T21:02:50.105789+0000 mon.a (mon.0) 1429 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:02:51.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:51 smithi023 bash[21372]: audit 2024-03-11T21:02:50.108115+0000 mon.a (mon.0) 1430 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:02:51.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:51 smithi023 bash[21372]: cluster 2024-03-11T21:02:50.138817+0000 mgr.y (mgr.24380) 447 : cluster 0 pgmap v493: 148 pgs: 2 creating+peering, 4 unknown, 142 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T21:02:51.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:51 smithi023 bash[21372]: audit 2024-03-11T21:02:50.379290+0000 mon.a (mon.0) 1431 : audit 0 from='client.? 172.21.15.23:0/2107374877' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:02:51.286 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:51 smithi023 bash[21372]: audit 2024-03-11T21:02:50.585757+0000 mon.a (mon.0) 1432 : audit 1 from='client.? 172.21.15.23:0/1484907570' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:02:51.286 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:51 smithi023 bash[29146]: audit 2024-03-11T21:02:50.105789+0000 mon.a (mon.0) 1429 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:02:51.286 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:51 smithi023 bash[29146]: audit 2024-03-11T21:02:50.108115+0000 mon.a (mon.0) 1430 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:02:51.286 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:51 smithi023 bash[29146]: cluster 2024-03-11T21:02:50.138817+0000 mgr.y (mgr.24380) 447 : cluster 0 pgmap v493: 148 pgs: 2 creating+peering, 4 unknown, 142 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T21:02:51.286 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:51 smithi023 bash[29146]: audit 2024-03-11T21:02:50.379290+0000 mon.a (mon.0) 1431 : audit 0 from='client.? 172.21.15.23:0/2107374877' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:02:51.286 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:51 smithi023 bash[29146]: audit 2024-03-11T21:02:50.585757+0000 mon.a (mon.0) 1432 : audit 1 from='client.? 172.21.15.23:0/1484907570' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:02:51.292 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:02:51.321 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:02:51.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:51.323 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:02:51.424 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:02:51.435 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:51 smithi146 bash[23809]: audit 2024-03-11T21:02:50.105789+0000 mon.a (mon.0) 1429 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:02:51.435 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:51 smithi146 bash[23809]: audit 2024-03-11T21:02:50.108115+0000 mon.a (mon.0) 1430 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:02:51.435 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:51 smithi146 bash[23809]: cluster 2024-03-11T21:02:50.138817+0000 mgr.y (mgr.24380) 447 : cluster 0 pgmap v493: 148 pgs: 2 creating+peering, 4 unknown, 142 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T21:02:51.435 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:51 smithi146 bash[23809]: audit 2024-03-11T21:02:50.379290+0000 mon.a (mon.0) 1431 : audit 0 from='client.? 172.21.15.23:0/2107374877' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:02:51.435 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:51 smithi146 bash[23809]: audit 2024-03-11T21:02:50.585757+0000 mon.a (mon.0) 1432 : audit 1 from='client.? 172.21.15.23:0/1484907570' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:02:51.451 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:02:51.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:51.454 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:02:51.554 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:02:51.581 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:02:51.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:51.584 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:02:51.685 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:51.713 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:51.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:51.716 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:02:51.817 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:51.846 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:51.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:51.848 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:02:51.876 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-03-11T21:02:51.949 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:51.975 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:51.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:51.978 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:02:52.078 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:52.106 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:52.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:52.109 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:02:52.210 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:02:52.221 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:52 smithi146 bash[23809]: cluster 2024-03-11T21:02:51.102281+0000 mon.a (mon.0) 1433 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:02:52.222 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:52 smithi146 bash[23809]: audit 2024-03-11T21:02:51.124286+0000 mon.a (mon.0) 1434 : audit 1 from='client.? 172.21.15.23:0/1484907570' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:02:52.222 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:52 smithi146 bash[23809]: cluster 2024-03-11T21:02:51.130394+0000 mon.a (mon.0) 1435 : cluster 0 osdmap e254: 8 total, 8 up, 8 in 2024-03-11T21:02:52.237 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:02:52.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:52.239 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:02:52.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:52 smithi023 bash[21372]: cluster 2024-03-11T21:02:51.102281+0000 mon.a (mon.0) 1433 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:02:52.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:52 smithi023 bash[21372]: audit 2024-03-11T21:02:51.124286+0000 mon.a (mon.0) 1434 : audit 1 from='client.? 172.21.15.23:0/1484907570' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:02:52.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:52 smithi023 bash[21372]: cluster 2024-03-11T21:02:51.130394+0000 mon.a (mon.0) 1435 : cluster 0 osdmap e254: 8 total, 8 up, 8 in 2024-03-11T21:02:52.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:52 smithi023 bash[29146]: cluster 2024-03-11T21:02:51.102281+0000 mon.a (mon.0) 1433 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:02:52.285 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:52 smithi023 bash[29146]: audit 2024-03-11T21:02:51.124286+0000 mon.a (mon.0) 1434 : audit 1 from='client.? 172.21.15.23:0/1484907570' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:02:52.286 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:52 smithi023 bash[29146]: cluster 2024-03-11T21:02:51.130394+0000 mon.a (mon.0) 1435 : cluster 0 osdmap e254: 8 total, 8 up, 8 in 2024-03-11T21:02:52.340 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:52.369 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:52.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:52.372 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:02:52.472 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:02:52.500 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:02:52.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:52.503 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:02:52.604 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:52.632 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:52.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:52.635 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:02:52.736 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:52.763 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:52.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:52.766 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:02:52.866 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:02:52.892 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:02:52.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:52.894 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:02:52.995 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:02:53.022 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:02:53.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:53.024 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:02:53.125 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:02:53.152 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:02:53.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:53.154 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:02:53.255 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:02:53.283 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:02:53.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:53.286 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:02:53.386 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:53.397 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:53 smithi023 bash[21372]: cluster 2024-03-11T21:02:52.139386+0000 mgr.y (mgr.24380) 448 : cluster 0 pgmap v495: 148 pgs: 148 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 922 B/s rd, 0 op/s 2024-03-11T21:02:53.397 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:53 smithi023 bash[29146]: cluster 2024-03-11T21:02:52.139386+0000 mgr.y (mgr.24380) 448 : cluster 0 pgmap v495: 148 pgs: 148 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 922 B/s rd, 0 op/s 2024-03-11T21:02:53.411 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:53.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:53.413 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:02:53.416 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T21:02:53.513 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:53 smithi146 bash[23809]: cluster 2024-03-11T21:02:52.139386+0000 mgr.y (mgr.24380) 448 : cluster 0 pgmap v495: 148 pgs: 148 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 922 B/s rd, 0 op/s 2024-03-11T21:02:53.514 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:53.531 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:02:53.531 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:02:53.540 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:53.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:53.542 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:02:53.643 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:53.665 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:53.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:53.667 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:02:53.768 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:02:53.790 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:02:53.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:53.792 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:02:53.893 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:02:53.920 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:02:53.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:53.922 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:02:54.023 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:54.035 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd unset noscrub 2024-03-11T21:02:54.053 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:54.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:54.056 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:02:54.156 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:54.166 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:54 smithi023 bash[21372]: audit 2024-03-11T21:02:53.175680+0000 mon.a (mon.0) 1436 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:02:54.166 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:54 smithi023 bash[21372]: audit 2024-03-11T21:02:53.702938+0000 mon.a (mon.0) 1437 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:02:54.166 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:54 smithi023 bash[21372]: audit 2024-03-11T21:02:53.711739+0000 mon.a (mon.0) 1438 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:02:54.167 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:54 smithi023 bash[29146]: audit 2024-03-11T21:02:53.175680+0000 mon.a (mon.0) 1436 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:02:54.167 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:54 smithi023 bash[29146]: audit 2024-03-11T21:02:53.702938+0000 mon.a (mon.0) 1437 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:02:54.167 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:54 smithi023 bash[29146]: audit 2024-03-11T21:02:53.711739+0000 mon.a (mon.0) 1438 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:02:54.178 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:54.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:54.181 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:02:54.282 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:02:54.308 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:02:54.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:54.310 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:02:54.411 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:54.434 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:54.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:54.436 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:02:54.513 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:54 smithi146 bash[23809]: audit 2024-03-11T21:02:53.175680+0000 mon.a (mon.0) 1436 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:02:54.514 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:54 smithi146 bash[23809]: audit 2024-03-11T21:02:53.702938+0000 mon.a (mon.0) 1437 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:02:54.514 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:54 smithi146 bash[23809]: audit 2024-03-11T21:02:53.711739+0000 mon.a (mon.0) 1438 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:02:54.537 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:54.558 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:54.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:54.561 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:02:54.661 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:02:54.684 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:02:54.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:54.686 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:02:54.786 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:02:54.813 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:02:54.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:54.816 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:02:54.916 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:02:54.944 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:02:54.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:54.946 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:02:55.047 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:55.083 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:55.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:55.085 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:02:55.186 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:55.212 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:55.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:55.214 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:02:55.264 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:55 smithi146 bash[23809]: cluster 2024-03-11T21:02:53.549648+0000 osd.0 (osd.0) 185 : cluster 0 purged_snaps scrub starts 2024-03-11T21:02:55.264 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:55 smithi146 bash[23809]: cluster 2024-03-11T21:02:53.549743+0000 osd.0 (osd.0) 186 : cluster 0 purged_snaps scrub ok 2024-03-11T21:02:55.264 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:55 smithi146 bash[23809]: cluster 2024-03-11T21:02:54.140013+0000 mgr.y (mgr.24380) 449 : cluster 0 pgmap v496: 148 pgs: 148 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 782 B/s rd, 0 op/s 2024-03-11T21:02:55.264 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:55 smithi146 bash[23809]: cluster 2024-03-11T21:02:54.160833+0000 mon.a (mon.0) 1439 : cluster 0 osdmap e255: 8 total, 8 up, 8 in 2024-03-11T21:02:55.264 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:55 smithi146 bash[23809]: audit 2024-03-11T21:02:54.287044+0000 mon.a (mon.0) 1440 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:02:55.264 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:55 smithi146 bash[23809]: audit 2024-03-11T21:02:54.288005+0000 mon.a (mon.0) 1441 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:02:55.264 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:55 smithi146 bash[23809]: audit 2024-03-11T21:02:54.295616+0000 mon.a (mon.0) 1442 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:02:55.316 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:55.327 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:55 smithi023 bash[21372]: cluster 2024-03-11T21:02:53.549648+0000 osd.0 (osd.0) 185 : cluster 0 purged_snaps scrub starts 2024-03-11T21:02:55.327 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:55 smithi023 bash[21372]: cluster 2024-03-11T21:02:53.549743+0000 osd.0 (osd.0) 186 : cluster 0 purged_snaps scrub ok 2024-03-11T21:02:55.327 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:55 smithi023 bash[21372]: cluster 2024-03-11T21:02:54.140013+0000 mgr.y (mgr.24380) 449 : cluster 0 pgmap v496: 148 pgs: 148 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 782 B/s rd, 0 op/s 2024-03-11T21:02:55.328 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:55 smithi023 bash[21372]: cluster 2024-03-11T21:02:54.160833+0000 mon.a (mon.0) 1439 : cluster 0 osdmap e255: 8 total, 8 up, 8 in 2024-03-11T21:02:55.328 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:55 smithi023 bash[21372]: audit 2024-03-11T21:02:54.287044+0000 mon.a (mon.0) 1440 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:02:55.328 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:55 smithi023 bash[21372]: audit 2024-03-11T21:02:54.288005+0000 mon.a (mon.0) 1441 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:02:55.328 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:55 smithi023 bash[21372]: audit 2024-03-11T21:02:54.295616+0000 mon.a (mon.0) 1442 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:02:55.328 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:55 smithi023 bash[29146]: cluster 2024-03-11T21:02:53.549648+0000 osd.0 (osd.0) 185 : cluster 0 purged_snaps scrub starts 2024-03-11T21:02:55.328 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:55 smithi023 bash[29146]: cluster 2024-03-11T21:02:53.549743+0000 osd.0 (osd.0) 186 : cluster 0 purged_snaps scrub ok 2024-03-11T21:02:55.328 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:55 smithi023 bash[29146]: cluster 2024-03-11T21:02:54.140013+0000 mgr.y (mgr.24380) 449 : cluster 0 pgmap v496: 148 pgs: 148 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 782 B/s rd, 0 op/s 2024-03-11T21:02:55.328 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:55 smithi023 bash[29146]: cluster 2024-03-11T21:02:54.160833+0000 mon.a (mon.0) 1439 : cluster 0 osdmap e255: 8 total, 8 up, 8 in 2024-03-11T21:02:55.328 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:55 smithi023 bash[29146]: audit 2024-03-11T21:02:54.287044+0000 mon.a (mon.0) 1440 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:02:55.328 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:55 smithi023 bash[29146]: audit 2024-03-11T21:02:54.288005+0000 mon.a (mon.0) 1441 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:02:55.328 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:55 smithi023 bash[29146]: audit 2024-03-11T21:02:54.295616+0000 mon.a (mon.0) 1442 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:02:55.344 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:55.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:55.347 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:02:55.449 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:55.478 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:55.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:55.481 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:02:55.582 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:02:55.609 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:02:55.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:55.611 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:02:55.712 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:02:55.722 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:02:55 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:02:55.738 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:02:55.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:55.740 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:02:55.841 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:02:55.870 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:02:55.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:55.872 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:02:55.973 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:02:56.001 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:02:56.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:56.003 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:02:56.104 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:02:56.132 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:02:56.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:56.136 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:02:56.236 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:02:56.247 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:56 smithi146 bash[23809]: audit 2024-03-11T21:02:55.417673+0000 mgr.y (mgr.24380) 450 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:02:56.263 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:02:56.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:56.266 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:02:56.367 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:02:56.386 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:02:56.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:56.389 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:02:56.489 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:56.500 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:56 smithi023 bash[21372]: audit 2024-03-11T21:02:55.417673+0000 mgr.y (mgr.24380) 450 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:02:56.501 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:56 smithi023 bash[29146]: audit 2024-03-11T21:02:55.417673+0000 mgr.y (mgr.24380) 450 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:02:56.514 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:56.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:56.516 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:02:56.617 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:02:56.644 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:02:56.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:56.647 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:02:56.647 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:02:56.748 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:56.778 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:56.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:56.780 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:02:56.881 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:02:56.909 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:02:56.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:56.912 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:02:57.013 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:57.041 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:57.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:57.043 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:02:57.144 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:57.173 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:57.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:57.176 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:02:57.276 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:57.306 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:57.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:57.308 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:02:57.409 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:57.421 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:57 smithi023 bash[21372]: cluster 2024-03-11T21:02:56.141556+0000 mgr.y (mgr.24380) 451 : cluster 0 pgmap v498: 148 pgs: 148 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-11T21:02:57.421 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:57 smithi023 bash[29146]: cluster 2024-03-11T21:02:56.141556+0000 mgr.y (mgr.24380) 451 : cluster 0 pgmap v498: 148 pgs: 148 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-11T21:02:57.439 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:57.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:57.441 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:02:57.513 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:57 smithi146 bash[23809]: cluster 2024-03-11T21:02:56.141556+0000 mgr.y (mgr.24380) 451 : cluster 0 pgmap v498: 148 pgs: 148 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-11T21:02:57.542 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:02:57.569 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:02:57.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:57.571 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:02:57.672 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:57.700 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:57.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:57.702 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:02:57.802 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:57.828 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:02:57.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:57.831 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:02:57.932 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:02:57.960 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:02:57.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:57.962 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:02:58.063 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:58.091 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:58.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:58.094 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:02:58.195 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:58.221 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:02:58.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:58.224 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:02:58.324 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:58.354 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:58.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:58.356 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:02:58.457 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:58.458 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:02:58 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:02:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:02:58.482 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:58.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:58.484 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:02:58.585 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:02:58.612 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:02:58.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:58.615 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:02:58.631 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:02:58.631 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:02:58.716 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:58.744 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:02:58.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:58.746 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:02:58.847 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:02:58.869 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:02:58.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:58.871 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:02:58.972 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:58.997 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:58.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:58.999 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:02:59.100 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:02:59.126 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:02:59.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:59.128 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:02:59.229 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:59.255 INFO:teuthology.orchestra.run.smithi023.stderr:enabled application 'rados' on pool 'unique_pool_2' 2024-03-11T21:02:59.261 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:59.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:59.264 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:02:59.264 INFO:teuthology.orchestra.run.smithi023.stderr:reweighted osd.5 to 1 (10000) 2024-03-11T21:02:59.365 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:02:59.393 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:02:59.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:59.395 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:02:59.496 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:02:59.507 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:59 smithi023 bash[21372]: cluster 2024-03-11T21:02:58.142142+0000 mgr.y (mgr.24380) 452 : cluster 0 pgmap v499: 148 pgs: 148 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:02:59.507 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:59 smithi023 bash[21372]: audit 2024-03-11T21:02:58.848552+0000 mon.a (mon.0) 1443 : audit 1 from='client.? 172.21.15.23:0/26154841' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:02:59.507 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:59 smithi023 bash[21372]: audit 2024-03-11T21:02:58.878984+0000 mon.a (mon.0) 1444 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-11T21:02:59.507 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:02:59 smithi023 bash[21372]: audit 2024-03-11T21:02:58.880188+0000 mon.b (mon.1) 38 : audit 1 from='client.? 172.21.15.23:0/251633764' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-11T21:02:59.507 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:59 smithi023 bash[29146]: cluster 2024-03-11T21:02:58.142142+0000 mgr.y (mgr.24380) 452 : cluster 0 pgmap v499: 148 pgs: 148 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:02:59.507 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:59 smithi023 bash[29146]: audit 2024-03-11T21:02:58.848552+0000 mon.a (mon.0) 1443 : audit 1 from='client.? 172.21.15.23:0/26154841' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:02:59.507 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:59 smithi023 bash[29146]: audit 2024-03-11T21:02:58.878984+0000 mon.a (mon.0) 1444 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-11T21:02:59.507 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:02:59 smithi023 bash[29146]: audit 2024-03-11T21:02:58.880188+0000 mon.b (mon.1) 38 : audit 1 from='client.? 172.21.15.23:0/251633764' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-11T21:02:59.513 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:59 smithi146 bash[23809]: cluster 2024-03-11T21:02:58.142142+0000 mgr.y (mgr.24380) 452 : cluster 0 pgmap v499: 148 pgs: 148 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:02:59.513 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:59 smithi146 bash[23809]: audit 2024-03-11T21:02:58.848552+0000 mon.a (mon.0) 1443 : audit 1 from='client.? 172.21.15.23:0/26154841' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:02:59.513 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:59 smithi146 bash[23809]: audit 2024-03-11T21:02:58.878984+0000 mon.a (mon.0) 1444 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-11T21:02:59.513 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:02:59 smithi146 bash[23809]: audit 2024-03-11T21:02:58.880188+0000 mon.b (mon.1) 38 : audit 1 from='client.? 172.21.15.23:0/251633764' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-11T21:02:59.522 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:02:59.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:59.525 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:02:59.625 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:59.650 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:02:59.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:59.652 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:02:59.753 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:02:59.788 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:02:59.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:59.791 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:02:59.891 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:02:59.912 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:02:59.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:02:59.914 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:03:00.015 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:00.032 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-03-11T21:03:00.032 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-03-11T21:03:00.032 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-03-11T21:03:00.032 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-03-11T21:03:00.032 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.032 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.24629.0:46654 7.0 7:07c3f001:::benchmark_data_smithi023_67959_object46653:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-03-11T21:03:00.032 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T20:55:16.113658+0000", 2024-03-11T21:03:00.032 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 463.92023464699997, 2024-03-11T21:03:00.032 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.333284262, 2024-03-11T21:03:00.032 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:03:00.032 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:03:00.032 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-03-11T21:03:00.032 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.24629", 2024-03-11T21:03:00.033 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.23:0/959732609", 2024-03-11T21:03:00.033 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 46654 2024-03-11T21:03:00.033 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.033 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:03:00.033 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.033 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:03:00.033 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:16.113658+0000", 2024-03-11T21:03:00.033 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:03:00.033 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.033 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.033 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:03:00.033 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:16.113658+0000", 2024-03-11T21:03:00.033 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:03:00.033 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.033 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.033 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:03:00.034 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:16.113659+0000", 2024-03-11T21:03:00.034 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2699999999999998e-07 2024-03-11T21:03:00.034 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.034 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.034 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:03:00.034 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:16.113765+0000", 2024-03-11T21:03:00.034 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000106488 2024-03-11T21:03:00.034 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.034 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.034 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:03:00.034 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:16.113781+0000", 2024-03-11T21:03:00.034 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6098000000000001e-05 2024-03-11T21:03:00.034 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.035 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.035 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:03:00.035 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:16.113791+0000", 2024-03-11T21:03:00.035 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0104e-05 2024-03-11T21:03:00.035 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.035 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.035 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:03:00.035 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:16.113855+0000", 2024-03-11T21:03:00.035 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3571999999999994e-05 2024-03-11T21:03:00.035 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.035 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.035 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:03:00.035 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:16.113998+0000", 2024-03-11T21:03:00.035 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014309500000000001 2024-03-11T21:03:00.036 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.036 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.036 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2,3", 2024-03-11T21:03:00.036 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:16.114150+0000", 2024-03-11T21:03:00.036 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015200700000000001 2024-03-11T21:03:00.036 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.036 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.036 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-03-11T21:03:00.036 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:16.115508+0000", 2024-03-11T21:03:00.036 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0013582430000000001 2024-03-11T21:03:00.036 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.036 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.036 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:03:00.036 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:16.116688+0000", 2024-03-11T21:03:00.037 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001179418 2024-03-11T21:03:00.037 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.037 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.037 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:03:00.037 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:16.446640+0000", 2024-03-11T21:03:00.037 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.32995237100000002 2024-03-11T21:03:00.037 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.037 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.037 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:03:00.037 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:16.446815+0000", 2024-03-11T21:03:00.037 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000174809 2024-03-11T21:03:00.037 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.037 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.037 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:03:00.037 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:16.446942+0000", 2024-03-11T21:03:00.038 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012713000000000001 2024-03-11T21:03:00.038 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:03:00.038 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:03:00.038 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:03:00.038 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.038 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.038 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.24629.0:46667 7.0 7:0b2df771:::benchmark_data_smithi023_67959_object46666:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-03-11T21:03:00.038 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T20:55:16.138335+0000", 2024-03-11T21:03:00.038 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 463.89555782100001, 2024-03-11T21:03:00.038 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.31666233199999999, 2024-03-11T21:03:00.038 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:03:00.038 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:03:00.038 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-03-11T21:03:00.038 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.24629", 2024-03-11T21:03:00.039 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.23:0/959732609", 2024-03-11T21:03:00.039 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 46667 2024-03-11T21:03:00.039 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.039 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:03:00.039 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.039 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:03:00.039 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:16.138335+0000", 2024-03-11T21:03:00.039 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:03:00.039 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.039 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.039 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:03:00.039 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:16.138335+0000", 2024-03-11T21:03:00.039 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:03:00.039 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.040 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.040 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:03:00.040 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:16.138336+0000", 2024-03-11T21:03:00.040 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.629e-06 2024-03-11T21:03:00.040 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.040 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.040 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:03:00.040 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:16.138381+0000", 2024-03-11T21:03:00.040 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4839000000000002e-05 2024-03-11T21:03:00.040 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.040 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.040 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:03:00.040 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:16.138398+0000", 2024-03-11T21:03:00.040 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6495999999999999e-05 2024-03-11T21:03:00.040 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.040 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.040 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:03:00.041 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:16.138407+0000", 2024-03-11T21:03:00.041 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7729999999999994e-06 2024-03-11T21:03:00.041 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.041 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.041 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:03:00.041 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:16.138476+0000", 2024-03-11T21:03:00.041 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9021000000000002e-05 2024-03-11T21:03:00.041 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.041 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.041 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:03:00.041 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:16.138628+0000", 2024-03-11T21:03:00.041 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000151605 2024-03-11T21:03:00.041 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.041 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.041 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2,3", 2024-03-11T21:03:00.042 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:16.138814+0000", 2024-03-11T21:03:00.042 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018646099999999999 2024-03-11T21:03:00.042 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.042 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.042 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-03-11T21:03:00.042 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:16.141200+0000", 2024-03-11T21:03:00.042 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0023856120000000001 2024-03-11T21:03:00.042 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.042 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.042 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:03:00.042 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:16.141447+0000", 2024-03-11T21:03:00.042 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000246753 2024-03-11T21:03:00.042 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.042 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.043 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:03:00.043 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:16.454588+0000", 2024-03-11T21:03:00.043 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.31314149699999999 2024-03-11T21:03:00.043 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.043 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.043 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:03:00.043 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:16.454826+0000", 2024-03-11T21:03:00.043 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000238204 2024-03-11T21:03:00.043 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.043 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.043 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:03:00.043 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:16.454997+0000", 2024-03-11T21:03:00.043 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017044200000000001 2024-03-11T21:03:00.043 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:03:00.043 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:03:00.043 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:03:00.043 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.044 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.044 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.24629.0:46670 7.0 7:0f30e774:::benchmark_data_smithi023_67959_object46669:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-03-11T21:03:00.044 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T20:55:16.138766+0000", 2024-03-11T21:03:00.044 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 463.89512671699998, 2024-03-11T21:03:00.044 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.31670262599999999, 2024-03-11T21:03:00.044 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:03:00.044 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:03:00.044 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-03-11T21:03:00.044 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.24629", 2024-03-11T21:03:00.044 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.23:0/959732609", 2024-03-11T21:03:00.044 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 46670 2024-03-11T21:03:00.044 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.044 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:03:00.044 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.045 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:03:00.045 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:16.138766+0000", 2024-03-11T21:03:00.045 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:03:00.045 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.045 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.045 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:03:00.045 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:16.138766+0000", 2024-03-11T21:03:00.045 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:03:00.045 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.045 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.045 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:03:00.045 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:16.138767+0000", 2024-03-11T21:03:00.045 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.029e-06 2024-03-11T21:03:00.045 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.046 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.046 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:03:00.046 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:16.138854+0000", 2024-03-11T21:03:00.046 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.7108999999999996e-05 2024-03-11T21:03:00.046 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.046 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.046 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:03:00.046 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:16.138866+0000", 2024-03-11T21:03:00.046 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2488e-05 2024-03-11T21:03:00.046 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.046 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.046 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:03:00.046 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:16.138876+0000", 2024-03-11T21:03:00.046 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2639999999999997e-06 2024-03-11T21:03:00.047 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.047 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.047 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:03:00.047 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:16.140036+0000", 2024-03-11T21:03:00.047 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0011606730000000001 2024-03-11T21:03:00.047 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.047 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.047 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:03:00.047 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:16.140174+0000", 2024-03-11T21:03:00.047 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013812899999999999 2024-03-11T21:03:00.047 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.047 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.047 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2,3", 2024-03-11T21:03:00.047 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:16.140313+0000", 2024-03-11T21:03:00.047 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013817500000000001 2024-03-11T21:03:00.047 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.048 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.048 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-03-11T21:03:00.048 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:16.141326+0000", 2024-03-11T21:03:00.048 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001013485 2024-03-11T21:03:00.048 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.048 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.048 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:03:00.048 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:16.141590+0000", 2024-03-11T21:03:00.048 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000264219 2024-03-11T21:03:00.048 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.048 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.048 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:03:00.048 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:16.455151+0000", 2024-03-11T21:03:00.048 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.31356043099999997 2024-03-11T21:03:00.048 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.049 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.049 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:03:00.049 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:16.455332+0000", 2024-03-11T21:03:00.049 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018160799999999999 2024-03-11T21:03:00.049 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.049 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.049 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:03:00.049 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:16.455468+0000", 2024-03-11T21:03:00.049 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013601600000000001 2024-03-11T21:03:00.049 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:03:00.049 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:03:00.049 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:03:00.049 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.049 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.049 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.24629.0:85827 7.0 7:0ef91f2b:::benchmark_data_smithi023_67959_object85826:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-03-11T21:03:00.049 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T20:55:52.517577+0000", 2024-03-11T21:03:00.050 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 427.51631513699999, 2024-03-11T21:03:00.050 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.40793880999999999, 2024-03-11T21:03:00.050 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:03:00.050 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:03:00.050 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-03-11T21:03:00.050 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.24629", 2024-03-11T21:03:00.050 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.23:0/959732609", 2024-03-11T21:03:00.050 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 85827 2024-03-11T21:03:00.050 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.050 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:03:00.050 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.050 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:03:00.050 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:52.517577+0000", 2024-03-11T21:03:00.051 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:03:00.051 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.051 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.051 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:03:00.051 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:52.517577+0000", 2024-03-11T21:03:00.051 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:03:00.051 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.051 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.051 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:03:00.051 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:52.517579+0000", 2024-03-11T21:03:00.051 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.333e-06 2024-03-11T21:03:00.051 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.051 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.051 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:03:00.052 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:52.517643+0000", 2024-03-11T21:03:00.052 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4332999999999995e-05 2024-03-11T21:03:00.052 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.052 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.052 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:03:00.052 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:52.517656+0000", 2024-03-11T21:03:00.052 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3159000000000001e-05 2024-03-11T21:03:00.052 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.052 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.052 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:03:00.052 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:52.517663+0000", 2024-03-11T21:03:00.052 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9750000000000001e-06 2024-03-11T21:03:00.052 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.052 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.052 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:03:00.053 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:52.517716+0000", 2024-03-11T21:03:00.053 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2787999999999997e-05 2024-03-11T21:03:00.053 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.053 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.053 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:03:00.053 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:52.517812+0000", 2024-03-11T21:03:00.053 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6088999999999994e-05 2024-03-11T21:03:00.053 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.053 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.053 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2,3", 2024-03-11T21:03:00.053 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:52.517927+0000", 2024-03-11T21:03:00.053 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011499500000000001 2024-03-11T21:03:00.053 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.053 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.053 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-03-11T21:03:00.054 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:52.521443+0000", 2024-03-11T21:03:00.054 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0035161369999999999 2024-03-11T21:03:00.054 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.054 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.054 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:03:00.054 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:52.531765+0000", 2024-03-11T21:03:00.054 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010321774000000001 2024-03-11T21:03:00.054 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.054 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.054 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:03:00.054 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:52.925411+0000", 2024-03-11T21:03:00.054 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.39364595200000002 2024-03-11T21:03:00.054 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.054 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.054 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:03:00.055 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:52.925471+0000", 2024-03-11T21:03:00.055 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9932000000000001e-05 2024-03-11T21:03:00.055 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.055 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.055 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:03:00.055 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:52.925516+0000", 2024-03-11T21:03:00.055 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5343000000000003e-05 2024-03-11T21:03:00.055 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:03:00.055 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:03:00.055 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:03:00.055 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.055 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.055 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.24629.0:85830 7.0 7:0fc395f2:::benchmark_data_smithi023_67959_object85829:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-03-11T21:03:00.055 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T20:55:52.517771+0000", 2024-03-11T21:03:00.055 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 427.51612171300002, 2024-03-11T21:03:00.056 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.40786048400000002, 2024-03-11T21:03:00.056 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:03:00.056 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:03:00.056 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-03-11T21:03:00.056 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.24629", 2024-03-11T21:03:00.056 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.23:0/959732609", 2024-03-11T21:03:00.056 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 85830 2024-03-11T21:03:00.056 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.056 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:03:00.056 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.056 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:03:00.056 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:52.517771+0000", 2024-03-11T21:03:00.056 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:03:00.056 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.056 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.056 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:03:00.057 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:52.517771+0000", 2024-03-11T21:03:00.057 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:03:00.057 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.057 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.057 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:03:00.057 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:52.517772+0000", 2024-03-11T21:03:00.057 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1699999999999997e-07 2024-03-11T21:03:00.057 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.057 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.057 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:03:00.057 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:52.517839+0000", 2024-03-11T21:03:00.057 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7427000000000003e-05 2024-03-11T21:03:00.057 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.057 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.057 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:03:00.058 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:52.517850+0000", 2024-03-11T21:03:00.058 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0705e-05 2024-03-11T21:03:00.058 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.058 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.058 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:03:00.058 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:52.517857+0000", 2024-03-11T21:03:00.058 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.413e-06 2024-03-11T21:03:00.058 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.058 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.058 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:03:00.058 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:52.519799+0000", 2024-03-11T21:03:00.058 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001941651 2024-03-11T21:03:00.058 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.058 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.059 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:03:00.059 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:52.519924+0000", 2024-03-11T21:03:00.059 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012546700000000001 2024-03-11T21:03:00.059 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.059 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.059 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2,3", 2024-03-11T21:03:00.059 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:52.520099+0000", 2024-03-11T21:03:00.059 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017457200000000001 2024-03-11T21:03:00.059 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.059 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.059 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-03-11T21:03:00.059 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:52.521570+0000", 2024-03-11T21:03:00.059 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0014713269999999999 2024-03-11T21:03:00.059 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.059 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.059 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:03:00.059 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:52.532428+0000", 2024-03-11T21:03:00.060 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010858273999999999 2024-03-11T21:03:00.060 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.060 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.060 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:03:00.060 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:52.925557+0000", 2024-03-11T21:03:00.060 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.393128376 2024-03-11T21:03:00.060 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.060 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.060 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:03:00.060 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:52.925591+0000", 2024-03-11T21:03:00.060 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4306e-05 2024-03-11T21:03:00.060 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.060 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.060 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:03:00.060 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:52.925631+0000", 2024-03-11T21:03:00.060 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0148999999999998e-05 2024-03-11T21:03:00.060 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:03:00.061 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:03:00.061 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:03:00.061 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.061 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.061 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.24629.0:95879 7.0 7:094c45ce:::benchmark_data_smithi023_67959_object95878:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-03-11T21:03:00.061 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T20:56:04.564166+0000", 2024-03-11T21:03:00.061 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 415.46972655299999, 2024-03-11T21:03:00.061 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.342752265, 2024-03-11T21:03:00.061 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:03:00.061 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:03:00.061 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-03-11T21:03:00.061 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.24629", 2024-03-11T21:03:00.061 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.23:0/959732609", 2024-03-11T21:03:00.061 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 95879 2024-03-11T21:03:00.061 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.061 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:03:00.061 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.062 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:03:00.062 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:56:04.564166+0000", 2024-03-11T21:03:00.062 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:03:00.062 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.062 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.062 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:03:00.062 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:56:04.564166+0000", 2024-03-11T21:03:00.062 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:03:00.062 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.062 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.062 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:03:00.062 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:56:04.564167+0000", 2024-03-11T21:03:00.063 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2410000000000001e-06 2024-03-11T21:03:00.063 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.063 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.063 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:03:00.063 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:56:04.564193+0000", 2024-03-11T21:03:00.063 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5938000000000001e-05 2024-03-11T21:03:00.063 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.063 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.063 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:03:00.063 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:56:04.564202+0000", 2024-03-11T21:03:00.063 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0119999999999994e-06 2024-03-11T21:03:00.063 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.063 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.064 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:03:00.064 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:56:04.564208+0000", 2024-03-11T21:03:00.064 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3600000000000001e-06 2024-03-11T21:03:00.064 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.064 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.064 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:03:00.064 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:56:04.564391+0000", 2024-03-11T21:03:00.064 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018228799999999999 2024-03-11T21:03:00.064 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.064 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.064 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:03:00.064 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:56:04.564495+0000", 2024-03-11T21:03:00.064 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000103935 2024-03-11T21:03:00.064 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.064 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.064 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2,3", 2024-03-11T21:03:00.065 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:56:04.564594+0000", 2024-03-11T21:03:00.065 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.9049000000000007e-05 2024-03-11T21:03:00.065 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.065 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.065 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-03-11T21:03:00.065 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:56:04.567572+0000", 2024-03-11T21:03:00.065 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0029785549999999999 2024-03-11T21:03:00.065 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.065 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.065 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:03:00.065 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:56:04.575594+0000", 2024-03-11T21:03:00.066 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.008021963 2024-03-11T21:03:00.066 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.066 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.066 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:03:00.066 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:56:04.906778+0000", 2024-03-11T21:03:00.066 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.33118418399999999 2024-03-11T21:03:00.066 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.066 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.066 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:03:00.066 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:56:04.906859+0000", 2024-03-11T21:03:00.066 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.0403999999999998e-05 2024-03-11T21:03:00.066 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.066 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.066 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:03:00.067 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:56:04.906918+0000", 2024-03-11T21:03:00.067 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9336000000000002e-05 2024-03-11T21:03:00.067 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:03:00.067 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:03:00.067 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:03:00.067 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.067 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.067 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.24629.0:95888 7.0 7:012e4262:::benchmark_data_smithi023_67959_object95887:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-03-11T21:03:00.067 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T20:56:04.572736+0000", 2024-03-11T21:03:00.067 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 415.46115668499999, 2024-03-11T21:03:00.067 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.33436669099999999, 2024-03-11T21:03:00.067 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:03:00.067 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:03:00.068 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-03-11T21:03:00.068 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.24629", 2024-03-11T21:03:00.068 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.23:0/959732609", 2024-03-11T21:03:00.068 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 95888 2024-03-11T21:03:00.068 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.068 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:03:00.068 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.068 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:03:00.068 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:56:04.572736+0000", 2024-03-11T21:03:00.068 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:03:00.068 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.068 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.068 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:03:00.068 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:56:04.572736+0000", 2024-03-11T21:03:00.069 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:03:00.069 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.069 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.069 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:03:00.069 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:56:04.572738+0000", 2024-03-11T21:03:00.069 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5840000000000002e-06 2024-03-11T21:03:00.069 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.069 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.069 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:03:00.069 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:56:04.572782+0000", 2024-03-11T21:03:00.069 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3389000000000001e-05 2024-03-11T21:03:00.069 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.070 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.070 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:03:00.070 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:56:04.572802+0000", 2024-03-11T21:03:00.070 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0191e-05 2024-03-11T21:03:00.070 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.070 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.070 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:03:00.070 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:56:04.572813+0000", 2024-03-11T21:03:00.070 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0665e-05 2024-03-11T21:03:00.070 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.070 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.070 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:03:00.071 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:56:04.572889+0000", 2024-03-11T21:03:00.071 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6823000000000006e-05 2024-03-11T21:03:00.071 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.071 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.071 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:03:00.071 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:56:04.573068+0000", 2024-03-11T21:03:00.071 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017872 2024-03-11T21:03:00.071 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.071 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.071 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2,3", 2024-03-11T21:03:00.071 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:56:04.573266+0000", 2024-03-11T21:03:00.072 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019760399999999999 2024-03-11T21:03:00.072 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.072 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.072 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-03-11T21:03:00.072 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:56:04.576141+0000", 2024-03-11T21:03:00.072 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002875612 2024-03-11T21:03:00.072 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.072 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.072 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:03:00.072 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:56:04.576563+0000", 2024-03-11T21:03:00.072 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00042171 2024-03-11T21:03:00.072 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.072 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.073 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:03:00.073 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:56:04.906982+0000", 2024-03-11T21:03:00.073 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.33041941600000002 2024-03-11T21:03:00.073 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.073 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.073 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:03:00.073 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:56:04.907051+0000", 2024-03-11T21:03:00.073 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.902e-05 2024-03-11T21:03:00.073 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.073 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.073 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:03:00.073 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:56:04.907102+0000", 2024-03-11T21:03:00.073 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0957e-05 2024-03-11T21:03:00.073 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:03:00.074 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:03:00.074 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:03:00.074 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.074 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.074 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.24629.0:126357 7.0 7:0bd767fa:::benchmark_data_smithi023_67959_object16116:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-03-11T21:03:00.074 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T20:56:39.199264+0000", 2024-03-11T21:03:00.074 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 380.83462835500001, 2024-03-11T21:03:00.074 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.43055240299999997, 2024-03-11T21:03:00.074 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:03:00.074 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:03:00.074 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-03-11T21:03:00.074 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.24629", 2024-03-11T21:03:00.074 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.23:0/959732609", 2024-03-11T21:03:00.074 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 126357 2024-03-11T21:03:00.074 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.075 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:03:00.075 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.075 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:03:00.075 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:56:39.199264+0000", 2024-03-11T21:03:00.075 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:03:00.075 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.075 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.075 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:03:00.075 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:56:39.199264+0000", 2024-03-11T21:03:00.075 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:03:00.075 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.075 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.075 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:03:00.075 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:56:39.199265+0000", 2024-03-11T21:03:00.075 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3880000000000001e-06 2024-03-11T21:03:00.075 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.076 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.076 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:03:00.076 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:56:39.199273+0000", 2024-03-11T21:03:00.076 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.277e-06 2024-03-11T21:03:00.076 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.076 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.076 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:03:00.076 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:56:39.199288+0000", 2024-03-11T21:03:00.076 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5512999999999999e-05 2024-03-11T21:03:00.076 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.076 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.076 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:03:00.076 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:56:39.199296+0000", 2024-03-11T21:03:00.076 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9889999999999992e-06 2024-03-11T21:03:00.076 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.076 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.077 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:03:00.077 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:56:39.199344+0000", 2024-03-11T21:03:00.077 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8094999999999999e-05 2024-03-11T21:03:00.077 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.077 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.077 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:03:00.077 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:56:39.199430+0000", 2024-03-11T21:03:00.077 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5997999999999999e-05 2024-03-11T21:03:00.077 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.077 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.077 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2,3", 2024-03-11T21:03:00.077 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:56:39.199541+0000", 2024-03-11T21:03:00.077 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000110623 2024-03-11T21:03:00.077 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.077 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.077 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-03-11T21:03:00.077 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:56:39.201791+0000", 2024-03-11T21:03:00.078 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0022500799999999998 2024-03-11T21:03:00.078 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.078 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.078 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:03:00.078 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:56:39.342120+0000", 2024-03-11T21:03:00.078 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.14032847200000001 2024-03-11T21:03:00.078 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.078 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.078 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:03:00.078 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:56:39.629697+0000", 2024-03-11T21:03:00.078 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.28757769900000002 2024-03-11T21:03:00.078 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.078 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.078 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:03:00.078 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:56:39.629764+0000", 2024-03-11T21:03:00.078 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6425000000000002e-05 2024-03-11T21:03:00.079 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.079 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.079 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:03:00.079 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:56:39.629816+0000", 2024-03-11T21:03:00.079 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2843999999999998e-05 2024-03-11T21:03:00.079 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:03:00.079 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:03:00.079 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:03:00.079 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.079 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.079 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.24629.0:167604 7.0 7:0f7d4629:::benchmark_data_smithi023_67959_object57363:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e161)", 2024-03-11T21:03:00.079 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T20:57:23.529629+0000", 2024-03-11T21:03:00.079 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 336.50426367199998, 2024-03-11T21:03:00.079 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3054977980000002, 2024-03-11T21:03:00.079 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:03:00.079 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:03:00.079 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-03-11T21:03:00.079 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.24629", 2024-03-11T21:03:00.080 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.23:0/959732609", 2024-03-11T21:03:00.080 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 167604 2024-03-11T21:03:00.080 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.080 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:03:00.080 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.080 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:03:00.080 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:23.529629+0000", 2024-03-11T21:03:00.080 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:03:00.080 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.080 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.080 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:03:00.080 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:23.529629+0000", 2024-03-11T21:03:00.080 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:03:00.080 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.080 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.080 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:03:00.081 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:23.529630+0000", 2024-03-11T21:03:00.081 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6130000000000001e-06 2024-03-11T21:03:00.081 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.081 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.081 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:03:00.081 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:23.529636+0000", 2024-03-11T21:03:00.081 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5389999999999998e-06 2024-03-11T21:03:00.081 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.081 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.081 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:03:00.082 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:23.529651+0000", 2024-03-11T21:03:00.082 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4854e-05 2024-03-11T21:03:00.082 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.082 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.082 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:03:00.082 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:23.529659+0000", 2024-03-11T21:03:00.082 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.3529999999999995e-06 2024-03-11T21:03:00.082 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.082 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.082 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:03:00.082 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:23.601207+0000", 2024-03-11T21:03:00.082 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.071547883000000007 2024-03-11T21:03:00.083 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.083 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.083 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-03-11T21:03:00.083 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:23.601218+0000", 2024-03-11T21:03:00.083 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0553e-05 2024-03-11T21:03:00.083 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.083 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.083 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:03:00.083 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:24.599793+0000", 2024-03-11T21:03:00.083 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.99857510100000002 2024-03-11T21:03:00.083 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.083 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.083 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-03-11T21:03:00.083 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:24.599807+0000", 2024-03-11T21:03:00.084 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4768e-05 2024-03-11T21:03:00.084 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.084 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.084 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:03:00.084 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:25.810108+0000", 2024-03-11T21:03:00.084 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.210300172 2024-03-11T21:03:00.084 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.084 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.084 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:03:00.084 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:25.810297+0000", 2024-03-11T21:03:00.084 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018955099999999999 2024-03-11T21:03:00.084 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.084 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.084 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1,2,3", 2024-03-11T21:03:00.084 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:25.810480+0000", 2024-03-11T21:03:00.085 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018308899999999999 2024-03-11T21:03:00.085 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.085 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.085 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:03:00.085 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:25.813663+0000", 2024-03-11T21:03:00.085 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0031825989999999999 2024-03-11T21:03:00.085 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.085 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.085 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-03-11T21:03:00.085 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:25.813678+0000", 2024-03-11T21:03:00.085 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5492999999999999e-05 2024-03-11T21:03:00.085 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.085 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.085 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:03:00.085 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:25.834308+0000", 2024-03-11T21:03:00.085 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.020630134000000001 2024-03-11T21:03:00.086 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.086 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.086 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:03:00.086 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:25.834849+0000", 2024-03-11T21:03:00.086 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00054052600000000005 2024-03-11T21:03:00.086 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.086 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.086 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:03:00.086 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:25.834938+0000", 2024-03-11T21:03:00.086 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8900000000000006e-05 2024-03-11T21:03:00.086 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.086 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.086 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:03:00.086 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:25.835127+0000", 2024-03-11T21:03:00.086 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018866999999999999 2024-03-11T21:03:00.087 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:03:00.087 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:03:00.087 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:03:00.087 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.087 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.087 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.24629.0:167609 7.0 7:09bc9f38:::benchmark_data_smithi023_67959_object57368:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e161)", 2024-03-11T21:03:00.087 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T20:57:23.541209+0000", 2024-03-11T21:03:00.087 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 336.49268326700002, 2024-03-11T21:03:00.087 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.314609328, 2024-03-11T21:03:00.087 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:03:00.087 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:03:00.087 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-03-11T21:03:00.087 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.24629", 2024-03-11T21:03:00.087 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.23:0/959732609", 2024-03-11T21:03:00.087 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 167609 2024-03-11T21:03:00.087 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.088 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:03:00.088 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.088 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:03:00.088 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:23.541209+0000", 2024-03-11T21:03:00.088 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:03:00.088 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.088 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.088 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:03:00.088 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:23.541209+0000", 2024-03-11T21:03:00.088 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:03:00.088 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.088 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.088 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:03:00.088 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:23.541211+0000", 2024-03-11T21:03:00.088 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.454e-06 2024-03-11T21:03:00.088 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.089 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.089 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:03:00.089 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:23.541214+0000", 2024-03-11T21:03:00.089 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.45e-06 2024-03-11T21:03:00.089 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.089 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.089 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:03:00.089 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:23.541224+0000", 2024-03-11T21:03:00.089 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7550000000000006e-06 2024-03-11T21:03:00.089 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.089 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.089 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:03:00.089 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:23.541237+0000", 2024-03-11T21:03:00.089 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2965e-05 2024-03-11T21:03:00.090 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.090 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.090 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:03:00.090 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:23.601239+0000", 2024-03-11T21:03:00.090 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.060001815999999999 2024-03-11T21:03:00.090 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.090 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.090 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-03-11T21:03:00.090 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:23.601243+0000", 2024-03-11T21:03:00.090 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1320000000000004e-06 2024-03-11T21:03:00.090 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.090 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.090 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:03:00.090 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:24.599840+0000", 2024-03-11T21:03:00.091 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.99859703099999997 2024-03-11T21:03:00.091 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.091 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.091 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-03-11T21:03:00.091 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:24.599850+0000", 2024-03-11T21:03:00.091 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8500000000000006e-06 2024-03-11T21:03:00.091 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.091 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.091 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:03:00.091 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:25.813717+0000", 2024-03-11T21:03:00.091 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.213867203 2024-03-11T21:03:00.091 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.091 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.092 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:03:00.092 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:25.813832+0000", 2024-03-11T21:03:00.092 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000115339 2024-03-11T21:03:00.092 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.092 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.092 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1,2,3", 2024-03-11T21:03:00.092 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:25.813942+0000", 2024-03-11T21:03:00.092 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000109685 2024-03-11T21:03:00.092 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.092 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.092 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-03-11T21:03:00.092 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:25.823164+0000", 2024-03-11T21:03:00.092 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0092219810000000006 2024-03-11T21:03:00.092 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.092 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.093 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:03:00.093 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:25.823281+0000", 2024-03-11T21:03:00.093 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000117352 2024-03-11T21:03:00.093 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.093 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.093 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:03:00.093 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:25.834443+0000", 2024-03-11T21:03:00.093 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.011161470999999999 2024-03-11T21:03:00.093 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.093 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.093 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:03:00.093 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:25.855667+0000", 2024-03-11T21:03:00.093 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.021224410999999999 2024-03-11T21:03:00.093 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.093 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.094 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:03:00.094 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:25.855711+0000", 2024-03-11T21:03:00.094 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4206999999999999e-05 2024-03-11T21:03:00.094 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.094 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.094 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:03:00.094 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:25.855818+0000", 2024-03-11T21:03:00.094 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010722600000000001 2024-03-11T21:03:00.094 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:03:00.094 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:03:00.094 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:03:00.094 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.094 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.094 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.24629.0:167622 7.0 7:05f8291b:::benchmark_data_smithi023_67959_object57381:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e161)", 2024-03-11T21:03:00.095 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T20:57:23.569870+0000", 2024-03-11T21:03:00.095 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 336.46402289299999, 2024-03-11T21:03:00.095 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.286238676, 2024-03-11T21:03:00.095 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:03:00.095 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:03:00.095 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-03-11T21:03:00.095 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.24629", 2024-03-11T21:03:00.095 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.23:0/959732609", 2024-03-11T21:03:00.095 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 167622 2024-03-11T21:03:00.095 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.095 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:03:00.095 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.095 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:03:00.095 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:23.569870+0000", 2024-03-11T21:03:00.095 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:03:00.095 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.096 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.096 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:03:00.096 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:23.569870+0000", 2024-03-11T21:03:00.096 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:03:00.096 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.096 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.096 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:03:00.096 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:23.569871+0000", 2024-03-11T21:03:00.096 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7710000000000001e-06 2024-03-11T21:03:00.096 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.096 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.096 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:03:00.096 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:23.569876+0000", 2024-03-11T21:03:00.096 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4340000000000002e-06 2024-03-11T21:03:00.096 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.097 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.097 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:03:00.097 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:23.569888+0000", 2024-03-11T21:03:00.097 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1882000000000001e-05 2024-03-11T21:03:00.097 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.097 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.097 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:03:00.097 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:23.569895+0000", 2024-03-11T21:03:00.097 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2420000000000003e-06 2024-03-11T21:03:00.097 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.097 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.097 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:03:00.097 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:23.601258+0000", 2024-03-11T21:03:00.097 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.031362637999999998 2024-03-11T21:03:00.097 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.098 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.098 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-03-11T21:03:00.098 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:23.601261+0000", 2024-03-11T21:03:00.098 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.941e-06 2024-03-11T21:03:00.098 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.098 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.098 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:03:00.098 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:24.600448+0000", 2024-03-11T21:03:00.098 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.99918611300000004 2024-03-11T21:03:00.098 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.098 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.098 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-03-11T21:03:00.098 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:24.600460+0000", 2024-03-11T21:03:00.098 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2815e-05 2024-03-11T21:03:00.098 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.099 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.099 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:03:00.099 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:25.815552+0000", 2024-03-11T21:03:00.099 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.215091892 2024-03-11T21:03:00.099 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.099 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.099 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:03:00.099 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:25.815841+0000", 2024-03-11T21:03:00.099 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00028887500000000002 2024-03-11T21:03:00.099 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.099 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.099 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1,2,3", 2024-03-11T21:03:00.099 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:25.816144+0000", 2024-03-11T21:03:00.099 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000303259 2024-03-11T21:03:00.100 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.100 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.100 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:03:00.100 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:25.823406+0000", 2024-03-11T21:03:00.100 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0072613139999999996 2024-03-11T21:03:00.100 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.100 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.100 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-03-11T21:03:00.100 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:25.823442+0000", 2024-03-11T21:03:00.100 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.6322999999999998e-05 2024-03-11T21:03:00.100 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.100 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.100 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:03:00.100 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:25.852482+0000", 2024-03-11T21:03:00.100 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029039644999999999 2024-03-11T21:03:00.101 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.101 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.101 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:03:00.101 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:25.855880+0000", 2024-03-11T21:03:00.101 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0033987470000000001 2024-03-11T21:03:00.101 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.101 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.101 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:03:00.101 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:25.855951+0000", 2024-03-11T21:03:00.101 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0845000000000004e-05 2024-03-11T21:03:00.101 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.101 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.101 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:03:00.101 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:25.856108+0000", 2024-03-11T21:03:00.101 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015694 2024-03-11T21:03:00.102 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:03:00.102 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:03:00.102 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:03:00.102 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.102 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.102 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.24629.0:167642 7.0 7:0a9e6d97:::benchmark_data_smithi023_67959_object57401:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e162)", 2024-03-11T21:03:00.102 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T20:57:23.631163+0000", 2024-03-11T21:03:00.102 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 336.40272911300002, 2024-03-11T21:03:00.102 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2252225609999998, 2024-03-11T21:03:00.102 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:03:00.102 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:03:00.102 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-03-11T21:03:00.102 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.24629", 2024-03-11T21:03:00.102 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.23:0/959732609", 2024-03-11T21:03:00.103 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 167642 2024-03-11T21:03:00.103 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.103 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:03:00.103 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.103 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:03:00.103 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:23.631163+0000", 2024-03-11T21:03:00.103 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:03:00.103 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.103 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.103 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:03:00.103 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:23.631163+0000", 2024-03-11T21:03:00.103 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:03:00.103 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.103 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.103 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:03:00.103 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:23.631165+0000", 2024-03-11T21:03:00.104 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3149999999999999e-06 2024-03-11T21:03:00.104 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.104 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.104 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:03:00.104 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:23.631169+0000", 2024-03-11T21:03:00.104 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2969999999999997e-06 2024-03-11T21:03:00.104 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.104 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.104 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:03:00.104 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:23.631180+0000", 2024-03-11T21:03:00.104 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1311999999999999e-05 2024-03-11T21:03:00.104 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.104 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.104 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:03:00.104 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:23.631186+0000", 2024-03-11T21:03:00.105 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7320000000000003e-06 2024-03-11T21:03:00.105 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.105 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.105 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:03:00.105 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:23.631240+0000", 2024-03-11T21:03:00.105 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4354000000000002e-05 2024-03-11T21:03:00.105 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.105 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.105 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-03-11T21:03:00.105 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:23.631248+0000", 2024-03-11T21:03:00.105 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.0839999999999992e-06 2024-03-11T21:03:00.105 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.105 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.105 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:03:00.105 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:24.600510+0000", 2024-03-11T21:03:00.106 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.969261661 2024-03-11T21:03:00.106 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.106 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.106 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-03-11T21:03:00.106 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:24.600521+0000", 2024-03-11T21:03:00.106 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1371e-05 2024-03-11T21:03:00.106 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.106 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.106 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:03:00.106 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:25.819895+0000", 2024-03-11T21:03:00.106 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.219373464 2024-03-11T21:03:00.106 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.106 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.106 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:03:00.106 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:25.820091+0000", 2024-03-11T21:03:00.106 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019623199999999999 2024-03-11T21:03:00.107 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.107 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.107 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1,2,3", 2024-03-11T21:03:00.107 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:25.820289+0000", 2024-03-11T21:03:00.107 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019784199999999999 2024-03-11T21:03:00.107 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.107 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.107 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-03-11T21:03:00.107 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:25.823457+0000", 2024-03-11T21:03:00.107 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0031684389999999999 2024-03-11T21:03:00.107 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.107 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.107 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:03:00.107 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:25.830674+0000", 2024-03-11T21:03:00.107 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0072164300000000002 2024-03-11T21:03:00.108 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.108 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.108 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:03:00.108 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:25.853054+0000", 2024-03-11T21:03:00.108 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022380161999999999 2024-03-11T21:03:00.108 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.108 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.108 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:03:00.108 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:25.856160+0000", 2024-03-11T21:03:00.108 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0031063010000000001 2024-03-11T21:03:00.108 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.108 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.108 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:03:00.108 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:25.856231+0000", 2024-03-11T21:03:00.108 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0643999999999997e-05 2024-03-11T21:03:00.108 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.109 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.109 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:03:00.109 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:25.856386+0000", 2024-03-11T21:03:00.109 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000154921 2024-03-11T21:03:00.109 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:03:00.109 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:03:00.109 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:03:00.109 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.109 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.109 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.24629.0:167643 7.0 7:0e3489be:::benchmark_data_smithi023_67959_object57402:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e162)", 2024-03-11T21:03:00.109 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T20:57:23.635970+0000", 2024-03-11T21:03:00.109 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 336.397922322, 2024-03-11T21:03:00.109 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.220801539, 2024-03-11T21:03:00.109 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:03:00.109 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:03:00.109 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-03-11T21:03:00.110 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.24629", 2024-03-11T21:03:00.110 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.23:0/959732609", 2024-03-11T21:03:00.110 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 167643 2024-03-11T21:03:00.110 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.110 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:03:00.110 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.110 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:03:00.110 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:23.635970+0000", 2024-03-11T21:03:00.110 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:03:00.110 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.110 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.110 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:03:00.110 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:23.635970+0000", 2024-03-11T21:03:00.110 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:03:00.111 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.111 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.111 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:03:00.111 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:23.635971+0000", 2024-03-11T21:03:00.111 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5099999999999999e-07 2024-03-11T21:03:00.111 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.111 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.111 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:03:00.111 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:23.635977+0000", 2024-03-11T21:03:00.111 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3899999999999998e-06 2024-03-11T21:03:00.111 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.111 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.111 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:03:00.111 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:23.635988+0000", 2024-03-11T21:03:00.111 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0995000000000001e-05 2024-03-11T21:03:00.111 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.112 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.112 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:03:00.112 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:23.635994+0000", 2024-03-11T21:03:00.112 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6200000000000004e-06 2024-03-11T21:03:00.112 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.112 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.112 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:03:00.112 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:23.636036+0000", 2024-03-11T21:03:00.112 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1941999999999999e-05 2024-03-11T21:03:00.112 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.112 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.112 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-03-11T21:03:00.112 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:23.636042+0000", 2024-03-11T21:03:00.112 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4529999999999999e-06 2024-03-11T21:03:00.112 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.113 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.113 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:03:00.113 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:24.600555+0000", 2024-03-11T21:03:00.113 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.96451318100000005 2024-03-11T21:03:00.113 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.113 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.113 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-03-11T21:03:00.113 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:24.600568+0000", 2024-03-11T21:03:00.113 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2323999999999999e-05 2024-03-11T21:03:00.113 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.113 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.113 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:03:00.113 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:25.823524+0000", 2024-03-11T21:03:00.113 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2229559720000001 2024-03-11T21:03:00.113 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.113 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.114 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:03:00.114 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:25.823752+0000", 2024-03-11T21:03:00.114 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000227915 2024-03-11T21:03:00.114 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.114 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.114 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1,2,3", 2024-03-11T21:03:00.114 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:25.824022+0000", 2024-03-11T21:03:00.114 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00027049799999999998 2024-03-11T21:03:00.114 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.114 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.114 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-03-11T21:03:00.114 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:25.834225+0000", 2024-03-11T21:03:00.114 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010202736 2024-03-11T21:03:00.115 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.115 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.115 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:03:00.115 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:25.837571+0000", 2024-03-11T21:03:00.115 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0033459269999999998 2024-03-11T21:03:00.115 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.115 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.115 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:03:00.115 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:25.853120+0000", 2024-03-11T21:03:00.115 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.015548856 2024-03-11T21:03:00.115 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.115 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.115 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:03:00.115 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:25.856475+0000", 2024-03-11T21:03:00.116 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003355439 2024-03-11T21:03:00.116 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.116 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.116 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:03:00.116 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:25.856565+0000", 2024-03-11T21:03:00.116 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.9674999999999998e-05 2024-03-11T21:03:00.116 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.116 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.116 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:03:00.116 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:25.856772+0000", 2024-03-11T21:03:00.116 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00020686500000000001 2024-03-11T21:03:00.116 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:03:00.116 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:03:00.116 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:03:00.117 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.117 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.117 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.24629.0:167644 7.0 7:08ffba5e:::benchmark_data_smithi023_67959_object57403:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e162)", 2024-03-11T21:03:00.117 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T20:57:23.637332+0000", 2024-03-11T21:03:00.117 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 336.39656056500002, 2024-03-11T21:03:00.117 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2197482179999999, 2024-03-11T21:03:00.117 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:03:00.117 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:03:00.117 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-03-11T21:03:00.117 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.24629", 2024-03-11T21:03:00.117 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.23:0/959732609", 2024-03-11T21:03:00.117 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 167644 2024-03-11T21:03:00.117 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.117 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:03:00.117 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.118 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:03:00.118 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:23.637332+0000", 2024-03-11T21:03:00.118 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:03:00.118 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.118 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.118 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:03:00.118 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:23.637332+0000", 2024-03-11T21:03:00.118 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:03:00.118 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.118 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.118 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:03:00.118 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:23.637333+0000", 2024-03-11T21:03:00.118 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.6700000000000002e-07 2024-03-11T21:03:00.118 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.118 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.119 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:03:00.119 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:23.637337+0000", 2024-03-11T21:03:00.119 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.746e-06 2024-03-11T21:03:00.119 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.119 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.119 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:03:00.119 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:23.637348+0000", 2024-03-11T21:03:00.119 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0321e-05 2024-03-11T21:03:00.119 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.119 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.119 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:03:00.119 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:23.637363+0000", 2024-03-11T21:03:00.119 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5486999999999999e-05 2024-03-11T21:03:00.119 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.119 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.119 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:03:00.120 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:23.637425+0000", 2024-03-11T21:03:00.120 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2157999999999997e-05 2024-03-11T21:03:00.120 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.120 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.120 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-03-11T21:03:00.120 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:23.637437+0000", 2024-03-11T21:03:00.120 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1367e-05 2024-03-11T21:03:00.120 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.120 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.120 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:03:00.120 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:24.600600+0000", 2024-03-11T21:03:00.120 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.96316310400000005 2024-03-11T21:03:00.120 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.120 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.120 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-03-11T21:03:00.120 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:24.600611+0000", 2024-03-11T21:03:00.121 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0681e-05 2024-03-11T21:03:00.121 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.121 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.121 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:03:00.121 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:25.827040+0000", 2024-03-11T21:03:00.121 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2264295650000001 2024-03-11T21:03:00.121 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.121 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.121 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:03:00.121 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:25.827221+0000", 2024-03-11T21:03:00.121 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000180411 2024-03-11T21:03:00.121 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.121 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.121 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1,2,3", 2024-03-11T21:03:00.121 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:25.827400+0000", 2024-03-11T21:03:00.121 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017900500000000001 2024-03-11T21:03:00.122 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.122 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.122 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-03-11T21:03:00.122 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:25.834466+0000", 2024-03-11T21:03:00.122 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0070664530000000003 2024-03-11T21:03:00.122 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.122 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.122 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:03:00.122 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:25.839479+0000", 2024-03-11T21:03:00.122 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0050124760000000001 2024-03-11T21:03:00.122 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.122 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.122 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:03:00.122 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:25.854563+0000", 2024-03-11T21:03:00.123 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.015084228 2024-03-11T21:03:00.123 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.123 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.123 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:03:00.123 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:25.856844+0000", 2024-03-11T21:03:00.123 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0022808529999999998 2024-03-11T21:03:00.123 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.123 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.123 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:03:00.123 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:25.856913+0000", 2024-03-11T21:03:00.123 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9406000000000006e-05 2024-03-11T21:03:00.123 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.123 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.123 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:03:00.123 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:25.857080+0000", 2024-03-11T21:03:00.123 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016709 2024-03-11T21:03:00.124 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:03:00.124 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:03:00.124 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:03:00.124 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.124 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.124 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.24629.0:215069 7.b 7:d81f01f8:::benchmark_data_smithi023_67959_object104828:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-03-11T21:03:00.124 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T20:58:26.636113+0000", 2024-03-11T21:03:00.124 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 273.39777937700001, 2024-03-11T21:03:00.124 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.34817976899999997, 2024-03-11T21:03:00.124 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:03:00.124 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:03:00.124 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-03-11T21:03:00.124 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.24629", 2024-03-11T21:03:00.124 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.23:0/959732609", 2024-03-11T21:03:00.124 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 215069 2024-03-11T21:03:00.125 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.125 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:03:00.125 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.125 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:03:00.125 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.636113+0000", 2024-03-11T21:03:00.125 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:03:00.125 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.125 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.125 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:03:00.125 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.636113+0000", 2024-03-11T21:03:00.125 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:03:00.125 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.125 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.125 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:03:00.125 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.636114+0000", 2024-03-11T21:03:00.125 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8200000000000008e-07 2024-03-11T21:03:00.126 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.126 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.126 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:03:00.126 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.636118+0000", 2024-03-11T21:03:00.126 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0969999999999999e-06 2024-03-11T21:03:00.126 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.126 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.126 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:03:00.126 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.636129+0000", 2024-03-11T21:03:00.126 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0458999999999999e-05 2024-03-11T21:03:00.126 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.126 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.126 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:03:00.126 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.636134+0000", 2024-03-11T21:03:00.126 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5539999999999997e-06 2024-03-11T21:03:00.127 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.127 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.127 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:03:00.127 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.636180+0000", 2024-03-11T21:03:00.127 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5655999999999998e-05 2024-03-11T21:03:00.127 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.127 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.127 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:03:00.127 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.636262+0000", 2024-03-11T21:03:00.127 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.2688999999999994e-05 2024-03-11T21:03:00.127 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.127 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.127 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 6", 2024-03-11T21:03:00.127 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.636341+0000", 2024-03-11T21:03:00.128 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.8424999999999995e-05 2024-03-11T21:03:00.128 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.128 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.128 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-03-11T21:03:00.128 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.638666+0000", 2024-03-11T21:03:00.128 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0023247620000000002 2024-03-11T21:03:00.128 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.128 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.128 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:03:00.128 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.983968+0000", 2024-03-11T21:03:00.128 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.34530226400000003 2024-03-11T21:03:00.128 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.128 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.128 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:03:00.128 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.984087+0000", 2024-03-11T21:03:00.128 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011921600000000001 2024-03-11T21:03:00.129 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.129 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.129 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:03:00.129 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.984293+0000", 2024-03-11T21:03:00.129 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000205665 2024-03-11T21:03:00.129 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:03:00.129 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:03:00.129 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:03:00.129 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.129 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.129 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.24629.0:215076 7.b 7:dc9de58c:::benchmark_data_smithi023_67959_object104835:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-03-11T21:03:00.129 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T20:58:26.642581+0000", 2024-03-11T21:03:00.129 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 273.39131170000002, 2024-03-11T21:03:00.129 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.34360015500000002, 2024-03-11T21:03:00.129 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:03:00.129 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:03:00.130 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-03-11T21:03:00.130 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.24629", 2024-03-11T21:03:00.130 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.23:0/959732609", 2024-03-11T21:03:00.130 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 215076 2024-03-11T21:03:00.130 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.130 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:03:00.130 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.130 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:03:00.130 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.642581+0000", 2024-03-11T21:03:00.130 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:03:00.130 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.130 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.130 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:03:00.130 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.642581+0000", 2024-03-11T21:03:00.130 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:03:00.130 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.131 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.131 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:03:00.131 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.642582+0000", 2024-03-11T21:03:00.131 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0380000000000001e-06 2024-03-11T21:03:00.131 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.131 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.131 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:03:00.131 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.642586+0000", 2024-03-11T21:03:00.131 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8530000000000002e-06 2024-03-11T21:03:00.131 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.131 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.131 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:03:00.131 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.642597+0000", 2024-03-11T21:03:00.131 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1342000000000001e-05 2024-03-11T21:03:00.131 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.132 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.132 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:03:00.132 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.642603+0000", 2024-03-11T21:03:00.132 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8549999999999998e-06 2024-03-11T21:03:00.132 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.132 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.132 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:03:00.132 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.642640+0000", 2024-03-11T21:03:00.132 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7209999999999998e-05 2024-03-11T21:03:00.132 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.132 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.132 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:03:00.132 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.642722+0000", 2024-03-11T21:03:00.132 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.2056000000000002e-05 2024-03-11T21:03:00.132 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.133 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.133 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 6", 2024-03-11T21:03:00.133 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.642796+0000", 2024-03-11T21:03:00.133 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4099999999999999e-05 2024-03-11T21:03:00.133 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.133 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.133 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-03-11T21:03:00.133 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.645716+0000", 2024-03-11T21:03:00.133 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0029195839999999998 2024-03-11T21:03:00.133 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.133 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.133 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:03:00.133 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.985972+0000", 2024-03-11T21:03:00.133 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.34025664999999999 2024-03-11T21:03:00.133 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.133 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.134 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:03:00.134 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.986046+0000", 2024-03-11T21:03:00.134 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3802999999999998e-05 2024-03-11T21:03:00.134 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.134 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.134 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:03:00.134 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.986181+0000", 2024-03-11T21:03:00.134 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000134664 2024-03-11T21:03:00.134 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:03:00.134 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:03:00.134 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:03:00.134 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.134 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.134 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.14910.0:61979 8.d 8:bc0ce104:::benchmark_data_smithi023_89854_object11894:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e216)", 2024-03-11T21:03:00.134 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T21:01:15.101693+0000", 2024-03-11T21:03:00.135 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 104.93219927, 2024-03-11T21:03:00.135 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.33422888699999997, 2024-03-11T21:03:00.135 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:03:00.135 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:03:00.135 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-03-11T21:03:00.135 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.14910", 2024-03-11T21:03:00.135 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.23:0/2457905657", 2024-03-11T21:03:00.135 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 61979 2024-03-11T21:03:00.135 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.135 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:03:00.135 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.135 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:03:00.135 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:15.101693+0000", 2024-03-11T21:03:00.135 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:03:00.135 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.135 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.136 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:03:00.136 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:15.101693+0000", 2024-03-11T21:03:00.136 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:03:00.136 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.136 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.136 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:03:00.136 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:15.101695+0000", 2024-03-11T21:03:00.136 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8279999999999999e-06 2024-03-11T21:03:00.136 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.136 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.136 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:03:00.136 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:15.101701+0000", 2024-03-11T21:03:00.136 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8490000000000002e-06 2024-03-11T21:03:00.136 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.136 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.137 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:03:00.137 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:15.101717+0000", 2024-03-11T21:03:00.137 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6234000000000001e-05 2024-03-11T21:03:00.137 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.137 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.137 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:03:00.137 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:15.101725+0000", 2024-03-11T21:03:00.137 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.515e-06 2024-03-11T21:03:00.137 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.137 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.137 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:03:00.137 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:15.101770+0000", 2024-03-11T21:03:00.137 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5596999999999997e-05 2024-03-11T21:03:00.137 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.137 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.137 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:03:00.138 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:15.101849+0000", 2024-03-11T21:03:00.138 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9301000000000005e-05 2024-03-11T21:03:00.138 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.138 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.138 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2,7", 2024-03-11T21:03:00.138 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:15.102002+0000", 2024-03-11T21:03:00.138 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015273700000000001 2024-03-11T21:03:00.138 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.138 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.138 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-03-11T21:03:00.138 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:15.109184+0000", 2024-03-11T21:03:00.138 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0071817779999999998 2024-03-11T21:03:00.138 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.138 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.139 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:03:00.139 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:15.114401+0000", 2024-03-11T21:03:00.139 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0052167369999999999 2024-03-11T21:03:00.139 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.139 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.139 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:03:00.139 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:15.435756+0000", 2024-03-11T21:03:00.139 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.32135482100000001 2024-03-11T21:03:00.139 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.139 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.139 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:03:00.139 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:15.435850+0000", 2024-03-11T21:03:00.139 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4217999999999998e-05 2024-03-11T21:03:00.139 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.139 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.139 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:03:00.140 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:15.435922+0000", 2024-03-11T21:03:00.140 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2272000000000002e-05 2024-03-11T21:03:00.140 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:03:00.140 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:03:00.140 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:03:00.140 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.140 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.140 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.14910.0:61984 8.d 8:ba47ad3b:::benchmark_data_smithi023_89854_object11899:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e216)", 2024-03-11T21:03:00.140 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T21:01:15.103325+0000", 2024-03-11T21:03:00.140 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 104.930567076, 2024-03-11T21:03:00.140 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.33279991199999998, 2024-03-11T21:03:00.140 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:03:00.140 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:03:00.140 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-03-11T21:03:00.140 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.14910", 2024-03-11T21:03:00.141 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.23:0/2457905657", 2024-03-11T21:03:00.141 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 61984 2024-03-11T21:03:00.141 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.141 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:03:00.141 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.141 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:03:00.141 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:15.103325+0000", 2024-03-11T21:03:00.141 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:03:00.141 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.141 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.141 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:03:00.141 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:15.103325+0000", 2024-03-11T21:03:00.141 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:03:00.142 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.142 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.142 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:03:00.142 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:15.103327+0000", 2024-03-11T21:03:00.142 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3379999999999999e-06 2024-03-11T21:03:00.142 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.142 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.142 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:03:00.142 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:15.103332+0000", 2024-03-11T21:03:00.142 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7690000000000002e-06 2024-03-11T21:03:00.142 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.142 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.143 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:03:00.143 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:15.103344+0000", 2024-03-11T21:03:00.143 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1073000000000001e-05 2024-03-11T21:03:00.143 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.143 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.143 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:03:00.143 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:15.103349+0000", 2024-03-11T21:03:00.143 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5369999999999997e-06 2024-03-11T21:03:00.143 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.143 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.143 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:03:00.143 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:15.105933+0000", 2024-03-11T21:03:00.143 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0025837159999999998 2024-03-11T21:03:00.143 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.143 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.143 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:03:00.144 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:15.106048+0000", 2024-03-11T21:03:00.144 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000114975 2024-03-11T21:03:00.144 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.144 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.144 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2,7", 2024-03-11T21:03:00.144 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:15.106157+0000", 2024-03-11T21:03:00.144 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010948 2024-03-11T21:03:00.144 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.144 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.144 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-03-11T21:03:00.145 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:15.109397+0000", 2024-03-11T21:03:00.145 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003240181 2024-03-11T21:03:00.145 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.145 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.145 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:03:00.145 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:15.119890+0000", 2024-03-11T21:03:00.145 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010492761 2024-03-11T21:03:00.145 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.145 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.145 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:03:00.145 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:15.435986+0000", 2024-03-11T21:03:00.145 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.31609562699999999 2024-03-11T21:03:00.145 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.145 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.145 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:03:00.145 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:15.436068+0000", 2024-03-11T21:03:00.146 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.2133999999999997e-05 2024-03-11T21:03:00.146 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.146 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.146 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:03:00.146 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:15.436125+0000", 2024-03-11T21:03:00.146 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7321000000000002e-05 2024-03-11T21:03:00.146 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:03:00.146 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:03:00.146 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:03:00.146 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.146 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.147 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.14910.0:61991 8.d 8:b40d9eed:::benchmark_data_smithi023_89854_object11906:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e216)", 2024-03-11T21:03:00.147 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T21:01:15.112031+0000", 2024-03-11T21:03:00.147 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 104.921861269, 2024-03-11T21:03:00.147 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.32454274100000002, 2024-03-11T21:03:00.147 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:03:00.147 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:03:00.147 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-03-11T21:03:00.147 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.14910", 2024-03-11T21:03:00.147 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.23:0/2457905657", 2024-03-11T21:03:00.147 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 61991 2024-03-11T21:03:00.147 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.147 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:03:00.147 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.148 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:03:00.148 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:15.112031+0000", 2024-03-11T21:03:00.148 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:03:00.148 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.148 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.148 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:03:00.148 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:15.112031+0000", 2024-03-11T21:03:00.148 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:03:00.148 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.148 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.148 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:03:00.148 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:15.112032+0000", 2024-03-11T21:03:00.149 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2500000000000001e-06 2024-03-11T21:03:00.149 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.149 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.149 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:03:00.149 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:15.112038+0000", 2024-03-11T21:03:00.149 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4049999999999999e-06 2024-03-11T21:03:00.149 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.149 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.149 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:03:00.149 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:15.112050+0000", 2024-03-11T21:03:00.149 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1975e-05 2024-03-11T21:03:00.149 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.150 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.150 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:03:00.150 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:15.112057+0000", 2024-03-11T21:03:00.150 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5580000000000003e-06 2024-03-11T21:03:00.150 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.150 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.150 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:03:00.150 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:15.112119+0000", 2024-03-11T21:03:00.150 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1384999999999996e-05 2024-03-11T21:03:00.150 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.151 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.151 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:03:00.151 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:15.112223+0000", 2024-03-11T21:03:00.151 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000103893 2024-03-11T21:03:00.151 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.151 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.151 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2,7", 2024-03-11T21:03:00.151 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:15.112318+0000", 2024-03-11T21:03:00.151 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5048000000000002e-05 2024-03-11T21:03:00.151 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.151 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.151 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-03-11T21:03:00.151 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:15.114440+0000", 2024-03-11T21:03:00.152 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0021220459999999998 2024-03-11T21:03:00.152 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.152 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.152 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:03:00.152 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:15.119939+0000", 2024-03-11T21:03:00.152 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0054994639999999999 2024-03-11T21:03:00.152 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.152 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.152 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:03:00.152 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:15.436414+0000", 2024-03-11T21:03:00.152 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.31647461999999998 2024-03-11T21:03:00.153 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.153 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.153 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:03:00.153 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:15.436498+0000", 2024-03-11T21:03:00.153 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4394999999999994e-05 2024-03-11T21:03:00.153 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.153 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.153 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:03:00.153 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:15.436574+0000", 2024-03-11T21:03:00.153 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5702000000000001e-05 2024-03-11T21:03:00.154 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:03:00.154 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:03:00.154 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:03:00.154 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.154 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.154 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.14910.0:61996 8.d 8:baa1259f:::benchmark_data_smithi023_89854_object11911:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e216)", 2024-03-11T21:03:00.154 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T21:01:15.114800+0000", 2024-03-11T21:03:00.154 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 104.919092466, 2024-03-11T21:03:00.154 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.32193712600000002, 2024-03-11T21:03:00.155 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:03:00.155 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:03:00.155 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-03-11T21:03:00.155 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.14910", 2024-03-11T21:03:00.155 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.23:0/2457905657", 2024-03-11T21:03:00.155 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 61996 2024-03-11T21:03:00.155 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.155 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:03:00.155 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.155 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:03:00.155 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:15.114800+0000", 2024-03-11T21:03:00.156 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:03:00.156 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.156 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.156 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:03:00.156 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:15.114800+0000", 2024-03-11T21:03:00.156 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:03:00.156 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.156 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.156 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:03:00.156 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:15.114801+0000", 2024-03-11T21:03:00.156 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.22e-06 2024-03-11T21:03:00.156 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.156 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.156 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:03:00.156 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:15.114806+0000", 2024-03-11T21:03:00.157 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9269999999999996e-06 2024-03-11T21:03:00.157 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.157 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.157 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:03:00.157 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:15.114823+0000", 2024-03-11T21:03:00.157 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6582000000000001e-05 2024-03-11T21:03:00.157 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.157 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.157 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:03:00.157 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:15.114831+0000", 2024-03-11T21:03:00.157 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.371e-06 2024-03-11T21:03:00.157 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.157 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.157 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:03:00.157 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:15.117258+0000", 2024-03-11T21:03:00.158 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0024268860000000001 2024-03-11T21:03:00.158 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.158 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.158 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:03:00.158 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:15.117400+0000", 2024-03-11T21:03:00.158 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000141901 2024-03-11T21:03:00.158 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.158 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.158 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2,7", 2024-03-11T21:03:00.158 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:15.117533+0000", 2024-03-11T21:03:00.158 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013266600000000001 2024-03-11T21:03:00.158 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.158 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.158 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-03-11T21:03:00.159 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:15.120238+0000", 2024-03-11T21:03:00.159 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0027053839999999999 2024-03-11T21:03:00.159 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.159 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.159 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:03:00.159 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:15.122162+0000", 2024-03-11T21:03:00.159 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0019236839999999999 2024-03-11T21:03:00.159 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.159 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.159 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:03:00.159 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:15.436634+0000", 2024-03-11T21:03:00.159 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.31447211400000002 2024-03-11T21:03:00.159 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.160 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.160 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:03:00.160 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:15.436694+0000", 2024-03-11T21:03:00.160 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0022000000000002e-05 2024-03-11T21:03:00.160 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:00.160 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:00.160 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:03:00.160 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:15.436737+0000", 2024-03-11T21:03:00.160 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3368999999999998e-05 2024-03-11T21:03:00.160 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:03:00.160 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:03:00.160 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:03:00.160 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:03:00.160 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:03:00.160 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-03-11T21:03:00.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:00.162 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:00.162 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:03:00.263 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:00.290 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:00.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:00.292 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:03:00.393 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:03:00.421 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:03:00.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:00.424 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:03:00.513 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:00 smithi146 bash[23809]: audit 2024-03-11T21:02:59.249007+0000 mon.a (mon.0) 1445 : audit 1 from='client.? 172.21.15.23:0/26154841' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-11T21:03:00.514 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:00 smithi146 bash[23809]: audit 2024-03-11T21:02:59.249249+0000 mon.a (mon.0) 1446 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-11T21:03:00.514 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:00 smithi146 bash[23809]: cluster 2024-03-11T21:02:59.257433+0000 mon.a (mon.0) 1447 : cluster 0 osdmap e256: 8 total, 8 up, 8 in 2024-03-11T21:03:00.514 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:00 smithi146 bash[23809]: cluster 2024-03-11T21:03:00.203763+0000 mon.a (mon.0) 1448 : cluster 0 osdmap e257: 8 total, 8 up, 8 in 2024-03-11T21:03:00.524 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:03:00.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:00 smithi023 bash[21372]: audit 2024-03-11T21:02:59.249007+0000 mon.a (mon.0) 1445 : audit 1 from='client.? 172.21.15.23:0/26154841' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-11T21:03:00.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:00 smithi023 bash[21372]: audit 2024-03-11T21:02:59.249249+0000 mon.a (mon.0) 1446 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-11T21:03:00.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:00 smithi023 bash[21372]: cluster 2024-03-11T21:02:59.257433+0000 mon.a (mon.0) 1447 : cluster 0 osdmap e256: 8 total, 8 up, 8 in 2024-03-11T21:03:00.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:00 smithi023 bash[21372]: cluster 2024-03-11T21:03:00.203763+0000 mon.a (mon.0) 1448 : cluster 0 osdmap e257: 8 total, 8 up, 8 in 2024-03-11T21:03:00.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:00 smithi023 bash[29146]: audit 2024-03-11T21:02:59.249007+0000 mon.a (mon.0) 1445 : audit 1 from='client.? 172.21.15.23:0/26154841' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-11T21:03:00.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:00 smithi023 bash[29146]: audit 2024-03-11T21:02:59.249249+0000 mon.a (mon.0) 1446 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-11T21:03:00.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:00 smithi023 bash[29146]: cluster 2024-03-11T21:02:59.257433+0000 mon.a (mon.0) 1447 : cluster 0 osdmap e256: 8 total, 8 up, 8 in 2024-03-11T21:03:00.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:00 smithi023 bash[29146]: cluster 2024-03-11T21:03:00.203763+0000 mon.a (mon.0) 1448 : cluster 0 osdmap e257: 8 total, 8 up, 8 in 2024-03-11T21:03:00.552 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:03:00.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:00.554 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:03:00.655 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:00.681 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:00.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:00.684 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:03:00.784 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:03:00.810 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:03:00.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:00.813 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:03:00.862 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-11T21:03:00.914 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:00.937 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:00.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:00.939 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:03:01.040 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:01.060 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:01.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:01.062 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:03:01.163 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:03:01.191 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:03:01.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:01.193 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:03:01.294 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:03:01.324 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:03:01.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:01.327 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:03:01.352 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd reweight 6 1 2024-03-11T21:03:01.428 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:03:01.447 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:01 smithi023 bash[21372]: cluster 2024-03-11T21:03:00.142883+0000 mgr.y (mgr.24380) 453 : cluster 0 pgmap v501: 148 pgs: 148 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:03:01.447 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:01 smithi023 bash[21372]: cluster 2024-03-11T21:03:00.265501+0000 mon.a (mon.0) 1449 : cluster 1 Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-11T21:03:01.447 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:01 smithi023 bash[29146]: cluster 2024-03-11T21:03:00.142883+0000 mgr.y (mgr.24380) 453 : cluster 0 pgmap v501: 148 pgs: 148 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:03:01.447 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:01 smithi023 bash[29146]: cluster 2024-03-11T21:03:00.265501+0000 mon.a (mon.0) 1449 : cluster 1 Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-11T21:03:01.459 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:03:01.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:01.461 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:03:01.513 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:01 smithi146 bash[23809]: cluster 2024-03-11T21:03:00.142883+0000 mgr.y (mgr.24380) 453 : cluster 0 pgmap v501: 148 pgs: 148 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:03:01.513 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:01 smithi146 bash[23809]: cluster 2024-03-11T21:03:00.265501+0000 mon.a (mon.0) 1449 : cluster 1 Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-11T21:03:01.562 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:01.587 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:01.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:01.590 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:03:01.691 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:03:01.711 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:03:01.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:01.713 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:03:01.813 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:03:01.840 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:03:01.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:01.842 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:03:01.943 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:03:01.971 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:03:01.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:01.973 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:03:02.074 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:03:02.101 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:03:02.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:02.103 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:03:02.204 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:02.231 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:02.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:02.234 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:03:02.334 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:02.361 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:02.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:02.363 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:03:02.399 DEBUG:teuthology.orchestra.run.smithi023:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_2 bench 90 write' 2024-03-11T21:03:02.401 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-03-11T21:03:02.401 INFO:teuthology.orchestra.run:waiting for 3000 2024-03-11T21:03:02.452 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:hints = 1 2024-03-11T21:03:02.452 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-03-11T21:03:02.452 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:Object prefix: benchmark_data_smithi023_106206 2024-03-11T21:03:02.452 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-11T21:03:02.452 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 0 0 0 0 0 0 - 0 2024-03-11T21:03:02.464 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:03:02.496 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:03:02.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:02.499 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:03:02.600 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:03:02.626 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:03:02.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:02.629 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:03:02.730 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:02.759 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:02.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:02.762 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:03:02.862 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:03:02.900 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:03:02.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:02.903 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:03:03.004 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:03:03.030 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:03:03.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:03.034 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:03:03.134 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:03.162 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:03.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:03.165 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:03:03.266 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:03:03.296 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:03:03.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:03.299 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:03:03.400 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:03.428 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:03.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:03.431 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:03:03.453 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 1 16 2884 2868 179.24 179.25 0.00338329 0.0055512 2024-03-11T21:03:03.532 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:03.543 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:03 smithi146 bash[23809]: cluster 2024-03-11T21:03:02.144128+0000 mgr.y (mgr.24380) 454 : cluster 0 pgmap v503: 148 pgs: 148 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-11T21:03:03.568 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:03.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:03.572 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:03:03.673 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:03:03.683 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:03 smithi023 bash[21372]: cluster 2024-03-11T21:03:02.144128+0000 mgr.y (mgr.24380) 454 : cluster 0 pgmap v503: 148 pgs: 148 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-11T21:03:03.683 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:03 smithi023 bash[29146]: cluster 2024-03-11T21:03:02.144128+0000 mgr.y (mgr.24380) 454 : cluster 0 pgmap v503: 148 pgs: 148 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-11T21:03:03.697 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:03:03.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:03.702 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:03:03.803 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:03:03.831 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:03:03.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:03.834 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:03:03.934 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:03.958 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:03.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:03.960 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:03:04.061 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:03:04.094 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:03:04.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:04.097 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:03:04.198 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:03:04.215 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:03:04.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:04.217 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:03:04.318 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:04.339 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:04.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:04.342 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:03:04.442 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:03:04.453 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 2 15 5576 5561 173.77 168.312 0.00499992 0.00574345 2024-03-11T21:03:04.473 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:03:04.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:04.476 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:03:04.577 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:03:04.605 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:03:04.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:04.608 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:03:04.709 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:03:04.737 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:03:04.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:04.740 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:03:04.840 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:04.864 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:04.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:04.867 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:03:04.968 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:03:04.994 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:03:04.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:04.997 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:03:05.098 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:05.131 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:05.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:05.134 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:03:05.235 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:05.263 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:05.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:05.265 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:03:05.366 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:03:05.393 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:03:05.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:05.396 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:03:05.453 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 3 16 8206 8190 170.613 164.312 0.011251 0.00585006 2024-03-11T21:03:05.497 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:03:05.513 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:03:05 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:03:05.514 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:05 smithi146 bash[23809]: cluster 2024-03-11T21:03:04.144605+0000 mgr.y (mgr.24380) 455 : cluster 0 pgmap v504: 148 pgs: 148 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:03:05.514 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:05 smithi146 bash[23809]: audit 2024-03-11T21:03:05.144386+0000 mon.a (mon.0) 1450 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:03:05.514 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:05 smithi146 bash[23809]: audit 2024-03-11T21:03:05.146692+0000 mon.a (mon.0) 1451 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:03:05.524 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:03:05.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:05.529 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:03:05.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:05 smithi023 bash[21372]: cluster 2024-03-11T21:03:04.144605+0000 mgr.y (mgr.24380) 455 : cluster 0 pgmap v504: 148 pgs: 148 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:03:05.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:05 smithi023 bash[21372]: audit 2024-03-11T21:03:05.144386+0000 mon.a (mon.0) 1450 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:03:05.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:05 smithi023 bash[21372]: audit 2024-03-11T21:03:05.146692+0000 mon.a (mon.0) 1451 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:03:05.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:05 smithi023 bash[29146]: cluster 2024-03-11T21:03:04.144605+0000 mgr.y (mgr.24380) 455 : cluster 0 pgmap v504: 148 pgs: 148 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:03:05.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:05 smithi023 bash[29146]: audit 2024-03-11T21:03:05.144386+0000 mon.a (mon.0) 1450 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:03:05.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:05 smithi023 bash[29146]: audit 2024-03-11T21:03:05.146692+0000 mon.a (mon.0) 1451 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:03:05.595 INFO:teuthology.orchestra.run.smithi023.stdout:osd.0: {} 2024-03-11T21:03:05.630 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:03:05.633 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:03:05.639 INFO:teuthology.orchestra.run.smithi023.stdout:osd.1: {} 2024-03-11T21:03:05.661 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:03:05.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:05.664 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:03:05.679 INFO:teuthology.orchestra.run.smithi023.stdout:osd.2: {} 2024-03-11T21:03:05.731 INFO:teuthology.orchestra.run.smithi023.stdout:osd.3: {} 2024-03-11T21:03:05.765 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:05.787 INFO:teuthology.orchestra.run.smithi023.stdout:osd.4: {} 2024-03-11T21:03:05.793 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:05.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:05.795 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:03:05.896 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:03:05.917 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:03:05.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:05.919 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:03:05.976 INFO:teuthology.orchestra.run.smithi023.stdout:osd.5: {} 2024-03-11T21:03:06.020 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:03:06.036 INFO:teuthology.orchestra.run.smithi023.stdout:osd.6: {} 2024-03-11T21:03:06.049 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:03:06.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:06.053 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:03:06.079 INFO:teuthology.orchestra.run.smithi023.stdout:osd.7: {} 2024-03-11T21:03:06.085 INFO:teuthology.orchestra.run.smithi023.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:03:06.085 INFO:teuthology.orchestra.run.smithi023.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:03:06.085 INFO:teuthology.orchestra.run.smithi023.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:03:06.086 INFO:teuthology.orchestra.run.smithi023.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:03:06.086 INFO:teuthology.orchestra.run.smithi023.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:03:06.086 INFO:teuthology.orchestra.run.smithi023.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:03:06.086 INFO:teuthology.orchestra.run.smithi023.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:03:06.086 INFO:teuthology.orchestra.run.smithi023.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:03:06.154 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:03:06.178 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:03:06.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:06.181 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:03:06.282 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:06.314 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:06.315 INFO:teuthology.orchestra.run.smithi023.stderr:noscrub is unset 2024-03-11T21:03:06.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:06.317 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:03:06.357 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:06 smithi023 bash[21372]: audit 2024-03-11T21:03:05.427516+0000 mgr.y (mgr.24380) 456 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:03:06.357 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:06 smithi023 bash[21372]: audit 2024-03-11T21:03:05.543532+0000 mon.a (mon.0) 1452 : audit 0 from='client.? 172.21.15.23:0/2093234337' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:03:06.357 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:06 smithi023 bash[21372]: audit 2024-03-11T21:03:05.888744+0000 mon.a (mon.0) 1453 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:03:06.357 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:06 smithi023 bash[21372]: audit 2024-03-11T21:03:05.889772+0000 mon.b (mon.1) 39 : audit 1 from='client.? 172.21.15.23:0/2712462013' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:03:06.357 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:06 smithi023 bash[29146]: audit 2024-03-11T21:03:05.427516+0000 mgr.y (mgr.24380) 456 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:03:06.357 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:06 smithi023 bash[29146]: audit 2024-03-11T21:03:05.543532+0000 mon.a (mon.0) 1452 : audit 0 from='client.? 172.21.15.23:0/2093234337' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:03:06.357 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:06 smithi023 bash[29146]: audit 2024-03-11T21:03:05.888744+0000 mon.a (mon.0) 1453 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:03:06.357 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:06 smithi023 bash[29146]: audit 2024-03-11T21:03:05.889772+0000 mon.b (mon.1) 39 : audit 1 from='client.? 172.21.15.23:0/2712462013' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:03:06.418 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:06.447 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:06.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:06.450 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:03:06.453 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 4 16 10210 10194 159.269 125.25 0.00697371 0.00626617 2024-03-11T21:03:06.551 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:06.580 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:06.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:06.582 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:03:06.683 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:03:06.692 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:06 smithi146 bash[23809]: audit 2024-03-11T21:03:05.427516+0000 mgr.y (mgr.24380) 456 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:03:06.692 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:06 smithi146 bash[23809]: audit 2024-03-11T21:03:05.543532+0000 mon.a (mon.0) 1452 : audit 0 from='client.? 172.21.15.23:0/2093234337' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:03:06.692 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:06 smithi146 bash[23809]: audit 2024-03-11T21:03:05.888744+0000 mon.a (mon.0) 1453 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:03:06.692 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:06 smithi146 bash[23809]: audit 2024-03-11T21:03:05.889772+0000 mon.b (mon.1) 39 : audit 1 from='client.? 172.21.15.23:0/2712462013' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:03:06.705 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:03:06.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:06.708 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:03:06.809 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:03:06.832 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:03:06.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:06.835 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:03:06.936 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:06.965 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:06.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:06.967 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:03:07.068 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:07.094 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:07.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:07.096 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:03:07.197 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:03:07.230 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:03:07.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:07.234 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:03:07.334 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:07.358 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:07 smithi023 bash[21372]: cluster 2024-03-11T21:03:06.146336+0000 mgr.y (mgr.24380) 457 : cluster 0 pgmap v505: 148 pgs: 148 active+clean; 421 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 52 MiB/s wr, 840 op/s 2024-03-11T21:03:07.358 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:07 smithi023 bash[21372]: cluster 2024-03-11T21:03:06.295808+0000 mon.a (mon.0) 1454 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:03:07.359 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:07 smithi023 bash[21372]: audit 2024-03-11T21:03:06.306637+0000 mon.a (mon.0) 1455 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:03:07.359 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:07 smithi023 bash[21372]: cluster 2024-03-11T21:03:06.318655+0000 mon.a (mon.0) 1456 : cluster 0 osdmap e258: 8 total, 8 up, 8 in 2024-03-11T21:03:07.359 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:07 smithi023 bash[21372]: cluster 2024-03-11T21:03:06.355608+0000 osd.1 (osd.1) 23 : cluster 0 3.0 scrub starts 2024-03-11T21:03:07.359 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:07 smithi023 bash[21372]: cluster 2024-03-11T21:03:06.361187+0000 osd.1 (osd.1) 24 : cluster 0 3.0 scrub ok 2024-03-11T21:03:07.359 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:07 smithi023 bash[21372]: cluster 2024-03-11T21:03:06.544525+0000 osd.6 (osd.6) 178 : cluster 0 3.a scrub starts 2024-03-11T21:03:07.359 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:07 smithi023 bash[21372]: cluster 2024-03-11T21:03:06.547246+0000 osd.6 (osd.6) 179 : cluster 0 3.a scrub ok 2024-03-11T21:03:07.359 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:07 smithi023 bash[21372]: cluster 2024-03-11T21:03:06.936279+0000 osd.4 (osd.4) 41 : cluster 0 purged_snaps scrub starts 2024-03-11T21:03:07.359 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:07 smithi023 bash[21372]: cluster 2024-03-11T21:03:06.936350+0000 osd.4 (osd.4) 42 : cluster 0 purged_snaps scrub ok 2024-03-11T21:03:07.362 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:07.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:07.365 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:03:07.453 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 5 16 12785 12769 159.6 160.938 0.00375699 0.00625658 2024-03-11T21:03:07.465 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:03:07.493 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:03:07.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:07.496 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:03:07.597 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:07.605 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:07 smithi146 bash[23809]: cluster 2024-03-11T21:03:06.146336+0000 mgr.y (mgr.24380) 457 : cluster 0 pgmap v505: 148 pgs: 148 active+clean; 421 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 52 MiB/s wr, 840 op/s 2024-03-11T21:03:07.605 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:07 smithi146 bash[23809]: cluster 2024-03-11T21:03:06.295808+0000 mon.a (mon.0) 1454 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:03:07.605 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:07 smithi146 bash[23809]: audit 2024-03-11T21:03:06.306637+0000 mon.a (mon.0) 1455 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:03:07.606 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:07 smithi146 bash[23809]: cluster 2024-03-11T21:03:06.318655+0000 mon.a (mon.0) 1456 : cluster 0 osdmap e258: 8 total, 8 up, 8 in 2024-03-11T21:03:07.606 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:07 smithi146 bash[23809]: cluster 2024-03-11T21:03:06.355608+0000 osd.1 (osd.1) 23 : cluster 0 3.0 scrub starts 2024-03-11T21:03:07.606 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:07 smithi146 bash[23809]: cluster 2024-03-11T21:03:06.361187+0000 osd.1 (osd.1) 24 : cluster 0 3.0 scrub ok 2024-03-11T21:03:07.606 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:07 smithi146 bash[23809]: cluster 2024-03-11T21:03:06.544525+0000 osd.6 (osd.6) 178 : cluster 0 3.a scrub starts 2024-03-11T21:03:07.606 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:07 smithi146 bash[23809]: cluster 2024-03-11T21:03:06.547246+0000 osd.6 (osd.6) 179 : cluster 0 3.a scrub ok 2024-03-11T21:03:07.606 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:07 smithi146 bash[23809]: cluster 2024-03-11T21:03:06.936279+0000 osd.4 (osd.4) 41 : cluster 0 purged_snaps scrub starts 2024-03-11T21:03:07.606 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:07 smithi146 bash[23809]: cluster 2024-03-11T21:03:06.936350+0000 osd.4 (osd.4) 42 : cluster 0 purged_snaps scrub ok 2024-03-11T21:03:07.616 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:07.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:07.618 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:03:07.719 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:07.753 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:07.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:07.755 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:03:07.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:07 smithi023 bash[29146]: cluster 2024-03-11T21:03:06.146336+0000 mgr.y (mgr.24380) 457 : cluster 0 pgmap v505: 148 pgs: 148 active+clean; 421 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 52 MiB/s wr, 840 op/s 2024-03-11T21:03:07.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:07 smithi023 bash[29146]: cluster 2024-03-11T21:03:06.295808+0000 mon.a (mon.0) 1454 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:03:07.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:07 smithi023 bash[29146]: audit 2024-03-11T21:03:06.306637+0000 mon.a (mon.0) 1455 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:03:07.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:07 smithi023 bash[29146]: cluster 2024-03-11T21:03:06.318655+0000 mon.a (mon.0) 1456 : cluster 0 osdmap e258: 8 total, 8 up, 8 in 2024-03-11T21:03:07.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:07 smithi023 bash[29146]: cluster 2024-03-11T21:03:06.355608+0000 osd.1 (osd.1) 23 : cluster 0 3.0 scrub starts 2024-03-11T21:03:07.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:07 smithi023 bash[29146]: cluster 2024-03-11T21:03:06.361187+0000 osd.1 (osd.1) 24 : cluster 0 3.0 scrub ok 2024-03-11T21:03:07.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:07 smithi023 bash[29146]: cluster 2024-03-11T21:03:06.544525+0000 osd.6 (osd.6) 178 : cluster 0 3.a scrub starts 2024-03-11T21:03:07.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:07 smithi023 bash[29146]: cluster 2024-03-11T21:03:06.547246+0000 osd.6 (osd.6) 179 : cluster 0 3.a scrub ok 2024-03-11T21:03:07.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:07 smithi023 bash[29146]: cluster 2024-03-11T21:03:06.936279+0000 osd.4 (osd.4) 41 : cluster 0 purged_snaps scrub starts 2024-03-11T21:03:07.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:07 smithi023 bash[29146]: cluster 2024-03-11T21:03:06.936350+0000 osd.4 (osd.4) 42 : cluster 0 purged_snaps scrub ok 2024-03-11T21:03:07.857 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:03:07.880 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:03:07.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:07.883 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:03:07.984 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:03:08.006 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:03:08.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:08.007 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:03:08.108 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:08.133 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:08.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:08.135 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:03:08.236 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:08.268 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:08.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:08.271 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:03:08.371 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:03:08.384 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:03:08 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:03:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:03:08.384 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:08 smithi023 bash[21372]: cluster 2024-03-11T21:03:07.310770+0000 osd.1 (osd.1) 25 : cluster 0 2.6 scrub starts 2024-03-11T21:03:08.384 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:08 smithi023 bash[21372]: cluster 2024-03-11T21:03:07.312448+0000 osd.1 (osd.1) 26 : cluster 0 2.6 scrub ok 2024-03-11T21:03:08.384 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:08 smithi023 bash[21372]: cluster 2024-03-11T21:03:07.318809+0000 mon.a (mon.0) 1457 : cluster 0 osdmap e259: 8 total, 8 up, 8 in 2024-03-11T21:03:08.385 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:08 smithi023 bash[29146]: cluster 2024-03-11T21:03:07.310770+0000 osd.1 (osd.1) 25 : cluster 0 2.6 scrub starts 2024-03-11T21:03:08.385 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:08 smithi023 bash[29146]: cluster 2024-03-11T21:03:07.312448+0000 osd.1 (osd.1) 26 : cluster 0 2.6 scrub ok 2024-03-11T21:03:08.385 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:08 smithi023 bash[29146]: cluster 2024-03-11T21:03:07.318809+0000 mon.a (mon.0) 1457 : cluster 0 osdmap e259: 8 total, 8 up, 8 in 2024-03-11T21:03:08.400 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:03:08.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:08.402 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:03:08.453 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 6 16 13769 13753 143.249 61.5 0.00174806 0.00690329 2024-03-11T21:03:08.503 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:03:08.513 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:08 smithi146 bash[23809]: cluster 2024-03-11T21:03:07.310770+0000 osd.1 (osd.1) 25 : cluster 0 2.6 scrub starts 2024-03-11T21:03:08.513 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:08 smithi146 bash[23809]: cluster 2024-03-11T21:03:07.312448+0000 osd.1 (osd.1) 26 : cluster 0 2.6 scrub ok 2024-03-11T21:03:08.513 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:08 smithi146 bash[23809]: cluster 2024-03-11T21:03:07.318809+0000 mon.a (mon.0) 1457 : cluster 0 osdmap e259: 8 total, 8 up, 8 in 2024-03-11T21:03:08.538 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:03:08.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:08.541 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:03:08.642 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:08.668 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:08.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:08.671 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:03:08.772 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:08.797 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:08.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:08.799 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:03:08.900 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:03:08.930 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:03:08.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:08.933 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:03:09.033 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:09.057 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:09.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:09.060 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:03:09.161 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:09.185 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:09.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:09.188 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:03:09.289 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:03:09.313 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:03:09.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:09.316 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:03:09.416 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:09.425 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:09 smithi023 bash[21372]: cluster 2024-03-11T21:03:08.146997+0000 mgr.y (mgr.24380) 458 : cluster 0 pgmap v508: 148 pgs: 148 active+clean; 507 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.3 KiB/s rd, 64 MiB/s wr, 1.02k op/s 2024-03-11T21:03:09.425 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:09 smithi023 bash[21372]: cluster 2024-03-11T21:03:09.072132+0000 osd.7 (osd.7) 249 : cluster 0 6.3 scrub starts 2024-03-11T21:03:09.425 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:09 smithi023 bash[21372]: cluster 2024-03-11T21:03:09.147917+0000 osd.7 (osd.7) 250 : cluster 0 6.3 scrub ok 2024-03-11T21:03:09.425 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:09 smithi023 bash[29146]: cluster 2024-03-11T21:03:08.146997+0000 mgr.y (mgr.24380) 458 : cluster 0 pgmap v508: 148 pgs: 148 active+clean; 507 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.3 KiB/s rd, 64 MiB/s wr, 1.02k op/s 2024-03-11T21:03:09.425 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:09 smithi023 bash[29146]: cluster 2024-03-11T21:03:09.072132+0000 osd.7 (osd.7) 249 : cluster 0 6.3 scrub starts 2024-03-11T21:03:09.426 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:09 smithi023 bash[29146]: cluster 2024-03-11T21:03:09.147917+0000 osd.7 (osd.7) 250 : cluster 0 6.3 scrub ok 2024-03-11T21:03:09.445 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:09.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:09.448 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:03:09.453 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 7 16 15993 15977 142.64 139 0.00187559 0.00699295 2024-03-11T21:03:09.549 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:03:09.578 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:03:09.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:09.581 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:03:09.682 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:03:09.708 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:03:09.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:09.713 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:03:09.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:09 smithi146 bash[23809]: cluster 2024-03-11T21:03:08.146997+0000 mgr.y (mgr.24380) 458 : cluster 0 pgmap v508: 148 pgs: 148 active+clean; 507 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.3 KiB/s rd, 64 MiB/s wr, 1.02k op/s 2024-03-11T21:03:09.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:09 smithi146 bash[23809]: cluster 2024-03-11T21:03:09.072132+0000 osd.7 (osd.7) 249 : cluster 0 6.3 scrub starts 2024-03-11T21:03:09.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:09 smithi146 bash[23809]: cluster 2024-03-11T21:03:09.147917+0000 osd.7 (osd.7) 250 : cluster 0 6.3 scrub ok 2024-03-11T21:03:09.796 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:03:09.814 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:09.838 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:09.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:09.840 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:03:09.941 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:03:09.966 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:03:09.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:09.968 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:03:10.069 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:03:10.087 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:03:10.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:10.089 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:03:10.190 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:03:10.217 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:03:10.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:10.219 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:03:10.319 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:10.353 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:10.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:10.357 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:03:10.453 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 8 16 18182 18166 141.91 136.812 0.00529846 0.0070414 2024-03-11T21:03:10.458 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:10.498 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:10.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:10.500 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:03:10.601 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:03:10.626 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:03:10.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:10.629 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:03:10.730 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:03:10.762 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:03:10.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:10.764 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:03:10.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:10 smithi023 bash[21372]: cluster 2024-03-11T21:03:09.527371+0000 osd.6 (osd.6) 180 : cluster 0 4.1f scrub starts 2024-03-11T21:03:10.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:10 smithi023 bash[21372]: cluster 2024-03-11T21:03:09.534357+0000 osd.6 (osd.6) 181 : cluster 0 4.1f scrub ok 2024-03-11T21:03:10.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:10 smithi023 bash[21372]: cluster 2024-03-11T21:03:09.743450+0000 osd.5 (osd.5) 219 : cluster 0 2.18 scrub starts 2024-03-11T21:03:10.786 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:10 smithi023 bash[21372]: cluster 2024-03-11T21:03:09.746782+0000 osd.5 (osd.5) 220 : cluster 0 2.18 scrub ok 2024-03-11T21:03:10.786 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:10 smithi023 bash[21372]: cluster 2024-03-11T21:03:10.065867+0000 osd.7 (osd.7) 251 : cluster 0 6.16 scrub starts 2024-03-11T21:03:10.786 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:10 smithi023 bash[21372]: cluster 2024-03-11T21:03:10.124064+0000 osd.7 (osd.7) 252 : cluster 0 6.16 scrub ok 2024-03-11T21:03:10.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:10 smithi023 bash[29146]: cluster 2024-03-11T21:03:09.527371+0000 osd.6 (osd.6) 180 : cluster 0 4.1f scrub starts 2024-03-11T21:03:10.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:10 smithi023 bash[29146]: cluster 2024-03-11T21:03:09.534357+0000 osd.6 (osd.6) 181 : cluster 0 4.1f scrub ok 2024-03-11T21:03:10.787 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:10 smithi023 bash[29146]: cluster 2024-03-11T21:03:09.743450+0000 osd.5 (osd.5) 219 : cluster 0 2.18 scrub starts 2024-03-11T21:03:10.787 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:10 smithi023 bash[29146]: cluster 2024-03-11T21:03:09.746782+0000 osd.5 (osd.5) 220 : cluster 0 2.18 scrub ok 2024-03-11T21:03:10.787 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:10 smithi023 bash[29146]: cluster 2024-03-11T21:03:10.065867+0000 osd.7 (osd.7) 251 : cluster 0 6.16 scrub starts 2024-03-11T21:03:10.787 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:10 smithi023 bash[29146]: cluster 2024-03-11T21:03:10.124064+0000 osd.7 (osd.7) 252 : cluster 0 6.16 scrub ok 2024-03-11T21:03:10.865 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:10.896 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:10.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:10.899 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:03:11.000 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:11.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:10 smithi146 bash[23809]: cluster 2024-03-11T21:03:09.527371+0000 osd.6 (osd.6) 180 : cluster 0 4.1f scrub starts 2024-03-11T21:03:11.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:10 smithi146 bash[23809]: cluster 2024-03-11T21:03:09.534357+0000 osd.6 (osd.6) 181 : cluster 0 4.1f scrub ok 2024-03-11T21:03:11.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:10 smithi146 bash[23809]: cluster 2024-03-11T21:03:09.743450+0000 osd.5 (osd.5) 219 : cluster 0 2.18 scrub starts 2024-03-11T21:03:11.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:10 smithi146 bash[23809]: cluster 2024-03-11T21:03:09.746782+0000 osd.5 (osd.5) 220 : cluster 0 2.18 scrub ok 2024-03-11T21:03:11.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:10 smithi146 bash[23809]: cluster 2024-03-11T21:03:10.065867+0000 osd.7 (osd.7) 251 : cluster 0 6.16 scrub starts 2024-03-11T21:03:11.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:10 smithi146 bash[23809]: cluster 2024-03-11T21:03:10.124064+0000 osd.7 (osd.7) 252 : cluster 0 6.16 scrub ok 2024-03-11T21:03:11.029 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:11.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:11.032 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:03:11.133 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:03:11.157 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:03:11.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:11.160 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:03:11.261 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:11.285 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:11.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:11.287 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:03:11.388 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:11.399 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:11 smithi023 bash[21372]: cluster 2024-03-11T21:03:09.698332+0000 osd.2 (osd.2) 165 : cluster 0 6.14 scrub starts 2024-03-11T21:03:11.399 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:11 smithi023 bash[21372]: cluster 2024-03-11T21:03:09.699960+0000 osd.2 (osd.2) 166 : cluster 0 6.14 scrub ok 2024-03-11T21:03:11.399 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:11 smithi023 bash[21372]: cluster 2024-03-11T21:03:10.147806+0000 mgr.y (mgr.24380) 459 : cluster 0 pgmap v509: 148 pgs: 148 active+clean; 902 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 113 MiB/s wr, 1.80k op/s 2024-03-11T21:03:11.399 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:11 smithi023 bash[21372]: cluster 2024-03-11T21:03:10.522073+0000 osd.6 (osd.6) 182 : cluster 0 4.a scrub starts 2024-03-11T21:03:11.399 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:11 smithi023 bash[21372]: cluster 2024-03-11T21:03:10.523788+0000 osd.6 (osd.6) 183 : cluster 0 4.a scrub ok 2024-03-11T21:03:11.399 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:11 smithi023 bash[21372]: cluster 2024-03-11T21:03:11.041311+0000 osd.7 (osd.7) 253 : cluster 0 6.9 scrub starts 2024-03-11T21:03:11.399 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:11 smithi023 bash[21372]: cluster 2024-03-11T21:03:11.086657+0000 osd.7 (osd.7) 254 : cluster 0 6.9 scrub ok 2024-03-11T21:03:11.400 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:11 smithi023 bash[29146]: cluster 2024-03-11T21:03:09.698332+0000 osd.2 (osd.2) 165 : cluster 0 6.14 scrub starts 2024-03-11T21:03:11.400 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:11 smithi023 bash[29146]: cluster 2024-03-11T21:03:09.699960+0000 osd.2 (osd.2) 166 : cluster 0 6.14 scrub ok 2024-03-11T21:03:11.400 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:11 smithi023 bash[29146]: cluster 2024-03-11T21:03:10.147806+0000 mgr.y (mgr.24380) 459 : cluster 0 pgmap v509: 148 pgs: 148 active+clean; 902 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 113 MiB/s wr, 1.80k op/s 2024-03-11T21:03:11.400 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:11 smithi023 bash[29146]: cluster 2024-03-11T21:03:10.522073+0000 osd.6 (osd.6) 182 : cluster 0 4.a scrub starts 2024-03-11T21:03:11.400 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:11 smithi023 bash[29146]: cluster 2024-03-11T21:03:10.523788+0000 osd.6 (osd.6) 183 : cluster 0 4.a scrub ok 2024-03-11T21:03:11.400 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:11 smithi023 bash[29146]: cluster 2024-03-11T21:03:11.041311+0000 osd.7 (osd.7) 253 : cluster 0 6.9 scrub starts 2024-03-11T21:03:11.400 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:11 smithi023 bash[29146]: cluster 2024-03-11T21:03:11.086657+0000 osd.7 (osd.7) 254 : cluster 0 6.9 scrub ok 2024-03-11T21:03:11.413 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:11.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:11.415 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:03:11.427 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T21:03:11.443 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-11T21:03:11.453 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 9 16 20426 20410 141.724 140.25 0.00360965 0.00704997 2024-03-11T21:03:11.516 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:11.526 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:11 smithi146 bash[23809]: cluster 2024-03-11T21:03:09.698332+0000 osd.2 (osd.2) 165 : cluster 0 6.14 scrub starts 2024-03-11T21:03:11.526 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:11 smithi146 bash[23809]: cluster 2024-03-11T21:03:09.699960+0000 osd.2 (osd.2) 166 : cluster 0 6.14 scrub ok 2024-03-11T21:03:11.526 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:11 smithi146 bash[23809]: cluster 2024-03-11T21:03:10.147806+0000 mgr.y (mgr.24380) 459 : cluster 0 pgmap v509: 148 pgs: 148 active+clean; 902 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 113 MiB/s wr, 1.80k op/s 2024-03-11T21:03:11.526 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:11 smithi146 bash[23809]: cluster 2024-03-11T21:03:10.522073+0000 osd.6 (osd.6) 182 : cluster 0 4.a scrub starts 2024-03-11T21:03:11.526 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:11 smithi146 bash[23809]: cluster 2024-03-11T21:03:10.523788+0000 osd.6 (osd.6) 183 : cluster 0 4.a scrub ok 2024-03-11T21:03:11.526 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:11 smithi146 bash[23809]: cluster 2024-03-11T21:03:11.041311+0000 osd.7 (osd.7) 253 : cluster 0 6.9 scrub starts 2024-03-11T21:03:11.526 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:11 smithi146 bash[23809]: cluster 2024-03-11T21:03:11.086657+0000 osd.7 (osd.7) 254 : cluster 0 6.9 scrub ok 2024-03-11T21:03:11.541 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:11.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:11.547 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:03:11.647 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:03:11.672 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:03:11.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:11.675 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:03:11.775 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:03:11.801 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:03:11.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:11.804 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:03:11.904 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:03:11.937 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:03:11.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:11.939 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:03:12.040 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:12.062 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:12.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:12.065 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:03:12.166 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:12.197 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:12.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:12.199 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:03:12.300 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:12.327 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:12.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:12.331 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:03:12.432 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:12.443 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:12 smithi023 bash[21372]: cluster 2024-03-11T21:03:11.304815+0000 osd.4 (osd.4) 43 : cluster 0 2.13 scrub starts 2024-03-11T21:03:12.444 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:12 smithi023 bash[21372]: cluster 2024-03-11T21:03:11.307325+0000 osd.4 (osd.4) 44 : cluster 0 2.13 scrub ok 2024-03-11T21:03:12.444 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:12 smithi023 bash[21372]: cluster 2024-03-11T21:03:11.329483+0000 osd.1 (osd.1) 27 : cluster 0 4.7 scrub starts 2024-03-11T21:03:12.444 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:12 smithi023 bash[21372]: cluster 2024-03-11T21:03:11.331879+0000 osd.1 (osd.1) 28 : cluster 0 4.7 scrub ok 2024-03-11T21:03:12.444 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:12 smithi023 bash[21372]: cluster 2024-03-11T21:03:11.540595+0000 osd.6 (osd.6) 184 : cluster 0 2.1a scrub starts 2024-03-11T21:03:12.444 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:12 smithi023 bash[21372]: cluster 2024-03-11T21:03:11.542657+0000 osd.6 (osd.6) 185 : cluster 0 2.1a scrub ok 2024-03-11T21:03:12.444 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:12 smithi023 bash[21372]: cluster 2024-03-11T21:03:11.721804+0000 osd.5 (osd.5) 221 : cluster 0 6.7 scrub starts 2024-03-11T21:03:12.444 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:12 smithi023 bash[21372]: cluster 2024-03-11T21:03:11.800710+0000 osd.5 (osd.5) 222 : cluster 0 6.7 scrub ok 2024-03-11T21:03:12.444 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:12 smithi023 bash[29146]: cluster 2024-03-11T21:03:11.304815+0000 osd.4 (osd.4) 43 : cluster 0 2.13 scrub starts 2024-03-11T21:03:12.444 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:12 smithi023 bash[29146]: cluster 2024-03-11T21:03:11.307325+0000 osd.4 (osd.4) 44 : cluster 0 2.13 scrub ok 2024-03-11T21:03:12.444 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:12 smithi023 bash[29146]: cluster 2024-03-11T21:03:11.329483+0000 osd.1 (osd.1) 27 : cluster 0 4.7 scrub starts 2024-03-11T21:03:12.445 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:12 smithi023 bash[29146]: cluster 2024-03-11T21:03:11.331879+0000 osd.1 (osd.1) 28 : cluster 0 4.7 scrub ok 2024-03-11T21:03:12.445 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:12 smithi023 bash[29146]: cluster 2024-03-11T21:03:11.540595+0000 osd.6 (osd.6) 184 : cluster 0 2.1a scrub starts 2024-03-11T21:03:12.445 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:12 smithi023 bash[29146]: cluster 2024-03-11T21:03:11.542657+0000 osd.6 (osd.6) 185 : cluster 0 2.1a scrub ok 2024-03-11T21:03:12.445 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:12 smithi023 bash[29146]: cluster 2024-03-11T21:03:11.721804+0000 osd.5 (osd.5) 221 : cluster 0 6.7 scrub starts 2024-03-11T21:03:12.445 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:12 smithi023 bash[29146]: cluster 2024-03-11T21:03:11.800710+0000 osd.5 (osd.5) 222 : cluster 0 6.7 scrub ok 2024-03-11T21:03:12.454 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 10 16 22350 22334 139.564 120.25 0.00653175 0.00715738 2024-03-11T21:03:12.461 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:12.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:12.464 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:03:12.565 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:03:12.588 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:03:12.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:12.591 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:03:12.692 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:03:12.702 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:12 smithi146 bash[23809]: cluster 2024-03-11T21:03:11.304815+0000 osd.4 (osd.4) 43 : cluster 0 2.13 scrub starts 2024-03-11T21:03:12.702 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:12 smithi146 bash[23809]: cluster 2024-03-11T21:03:11.307325+0000 osd.4 (osd.4) 44 : cluster 0 2.13 scrub ok 2024-03-11T21:03:12.702 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:12 smithi146 bash[23809]: cluster 2024-03-11T21:03:11.329483+0000 osd.1 (osd.1) 27 : cluster 0 4.7 scrub starts 2024-03-11T21:03:12.702 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:12 smithi146 bash[23809]: cluster 2024-03-11T21:03:11.331879+0000 osd.1 (osd.1) 28 : cluster 0 4.7 scrub ok 2024-03-11T21:03:12.702 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:12 smithi146 bash[23809]: cluster 2024-03-11T21:03:11.540595+0000 osd.6 (osd.6) 184 : cluster 0 2.1a scrub starts 2024-03-11T21:03:12.703 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:12 smithi146 bash[23809]: cluster 2024-03-11T21:03:11.542657+0000 osd.6 (osd.6) 185 : cluster 0 2.1a scrub ok 2024-03-11T21:03:12.703 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:12 smithi146 bash[23809]: cluster 2024-03-11T21:03:11.721804+0000 osd.5 (osd.5) 221 : cluster 0 6.7 scrub starts 2024-03-11T21:03:12.703 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:12 smithi146 bash[23809]: cluster 2024-03-11T21:03:11.800710+0000 osd.5 (osd.5) 222 : cluster 0 6.7 scrub ok 2024-03-11T21:03:12.718 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:03:12.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:12.720 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:03:12.821 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:03:12.850 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:03:12.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:12.852 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:03:12.953 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:03:12.972 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:03:12.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:12.974 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:03:13.075 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:03:13.099 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:03:13.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:13.102 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:03:13.203 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:03:13.225 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:03:13.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:13.227 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:03:13.327 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:13.345 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:13.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:13.347 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:03:13.448 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:03:13.454 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 11 16 23573 23557 133.825 76.4375 0.00179198 0.00736782 2024-03-11T21:03:13.472 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:03:13.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:13.474 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:03:13.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:13 smithi023 bash[21372]: cluster 2024-03-11T21:03:12.149137+0000 mgr.y (mgr.24380) 460 : cluster 0 pgmap v510: 148 pgs: 148 active+clean; 1.1 GiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 143 MiB/s wr, 2.29k op/s 2024-03-11T21:03:13.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:13 smithi023 bash[21372]: cluster 2024-03-11T21:03:12.256561+0000 osd.4 (osd.4) 45 : cluster 0 4.1 scrub starts 2024-03-11T21:03:13.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:13 smithi023 bash[21372]: cluster 2024-03-11T21:03:12.258836+0000 osd.4 (osd.4) 46 : cluster 0 4.1 scrub ok 2024-03-11T21:03:13.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:13 smithi023 bash[21372]: cluster 2024-03-11T21:03:12.321221+0000 osd.1 (osd.1) 29 : cluster 0 3.4 scrub starts 2024-03-11T21:03:13.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:13 smithi023 bash[21372]: cluster 2024-03-11T21:03:12.324426+0000 osd.1 (osd.1) 30 : cluster 0 3.4 scrub ok 2024-03-11T21:03:13.536 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:13 smithi023 bash[21372]: cluster 2024-03-11T21:03:12.533211+0000 osd.6 (osd.6) 186 : cluster 0 2.11 scrub starts 2024-03-11T21:03:13.536 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:13 smithi023 bash[21372]: cluster 2024-03-11T21:03:12.535587+0000 osd.6 (osd.6) 187 : cluster 0 2.11 scrub ok 2024-03-11T21:03:13.536 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:13 smithi023 bash[21372]: cluster 2024-03-11T21:03:12.758538+0000 osd.5 (osd.5) 223 : cluster 0 6.a scrub starts 2024-03-11T21:03:13.536 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:13 smithi023 bash[21372]: cluster 2024-03-11T21:03:12.778711+0000 osd.5 (osd.5) 224 : cluster 0 6.a scrub ok 2024-03-11T21:03:13.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:13 smithi023 bash[29146]: cluster 2024-03-11T21:03:12.149137+0000 mgr.y (mgr.24380) 460 : cluster 0 pgmap v510: 148 pgs: 148 active+clean; 1.1 GiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 143 MiB/s wr, 2.29k op/s 2024-03-11T21:03:13.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:13 smithi023 bash[29146]: cluster 2024-03-11T21:03:12.256561+0000 osd.4 (osd.4) 45 : cluster 0 4.1 scrub starts 2024-03-11T21:03:13.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:13 smithi023 bash[29146]: cluster 2024-03-11T21:03:12.258836+0000 osd.4 (osd.4) 46 : cluster 0 4.1 scrub ok 2024-03-11T21:03:13.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:13 smithi023 bash[29146]: cluster 2024-03-11T21:03:12.321221+0000 osd.1 (osd.1) 29 : cluster 0 3.4 scrub starts 2024-03-11T21:03:13.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:13 smithi023 bash[29146]: cluster 2024-03-11T21:03:12.324426+0000 osd.1 (osd.1) 30 : cluster 0 3.4 scrub ok 2024-03-11T21:03:13.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:13 smithi023 bash[29146]: cluster 2024-03-11T21:03:12.533211+0000 osd.6 (osd.6) 186 : cluster 0 2.11 scrub starts 2024-03-11T21:03:13.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:13 smithi023 bash[29146]: cluster 2024-03-11T21:03:12.535587+0000 osd.6 (osd.6) 187 : cluster 0 2.11 scrub ok 2024-03-11T21:03:13.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:13 smithi023 bash[29146]: cluster 2024-03-11T21:03:12.758538+0000 osd.5 (osd.5) 223 : cluster 0 6.a scrub starts 2024-03-11T21:03:13.537 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:13 smithi023 bash[29146]: cluster 2024-03-11T21:03:12.778711+0000 osd.5 (osd.5) 224 : cluster 0 6.a scrub ok 2024-03-11T21:03:13.574 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:13.606 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:13.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:13.608 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:03:13.708 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:03:13.743 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:03:13.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:13.746 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:03:13.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:13 smithi146 bash[23809]: cluster 2024-03-11T21:03:12.149137+0000 mgr.y (mgr.24380) 460 : cluster 0 pgmap v510: 148 pgs: 148 active+clean; 1.1 GiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 143 MiB/s wr, 2.29k op/s 2024-03-11T21:03:13.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:13 smithi146 bash[23809]: cluster 2024-03-11T21:03:12.256561+0000 osd.4 (osd.4) 45 : cluster 0 4.1 scrub starts 2024-03-11T21:03:13.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:13 smithi146 bash[23809]: cluster 2024-03-11T21:03:12.258836+0000 osd.4 (osd.4) 46 : cluster 0 4.1 scrub ok 2024-03-11T21:03:13.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:13 smithi146 bash[23809]: cluster 2024-03-11T21:03:12.321221+0000 osd.1 (osd.1) 29 : cluster 0 3.4 scrub starts 2024-03-11T21:03:13.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:13 smithi146 bash[23809]: cluster 2024-03-11T21:03:12.324426+0000 osd.1 (osd.1) 30 : cluster 0 3.4 scrub ok 2024-03-11T21:03:13.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:13 smithi146 bash[23809]: cluster 2024-03-11T21:03:12.533211+0000 osd.6 (osd.6) 186 : cluster 0 2.11 scrub starts 2024-03-11T21:03:13.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:13 smithi146 bash[23809]: cluster 2024-03-11T21:03:12.535587+0000 osd.6 (osd.6) 187 : cluster 0 2.11 scrub ok 2024-03-11T21:03:13.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:13 smithi146 bash[23809]: cluster 2024-03-11T21:03:12.758538+0000 osd.5 (osd.5) 223 : cluster 0 6.a scrub starts 2024-03-11T21:03:13.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:13 smithi146 bash[23809]: cluster 2024-03-11T21:03:12.778711+0000 osd.5 (osd.5) 224 : cluster 0 6.a scrub ok 2024-03-11T21:03:13.847 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:03:13.876 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:03:13.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:13.880 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:03:13.980 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:03:14.010 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:03:14.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:14.016 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:03:14.117 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:03:14.150 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:03:14.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:14.153 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:03:14.254 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:03:14.277 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:03:14.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:14.279 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:03:14.380 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:03:14.401 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:03:14.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:14.403 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:03:14.454 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 12 16 25199 25183 131.141 101.625 0.0398092 0.00761946 2024-03-11T21:03:14.503 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:03:14.533 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:03:14.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:14.536 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:03:14.537 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:14 smithi146 bash[23809]: cluster 2024-03-11T21:03:12.700221+0000 osd.2 (osd.2) 167 : cluster 0 2.e scrub starts 2024-03-11T21:03:14.537 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:14 smithi146 bash[23809]: cluster 2024-03-11T21:03:12.702458+0000 osd.2 (osd.2) 168 : cluster 0 2.e scrub ok 2024-03-11T21:03:14.537 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:14 smithi146 bash[23809]: cluster 2024-03-11T21:03:12.755528+0000 osd.0 (osd.0) 187 : cluster 0 3.1f scrub starts 2024-03-11T21:03:14.537 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:14 smithi146 bash[23809]: cluster 2024-03-11T21:03:12.772190+0000 osd.0 (osd.0) 188 : cluster 0 3.1f scrub ok 2024-03-11T21:03:14.537 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:14 smithi146 bash[23809]: cluster 2024-03-11T21:03:13.514286+0000 osd.3 (osd.3) 254 : cluster 0 3.8 scrub starts 2024-03-11T21:03:14.537 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:14 smithi146 bash[23809]: cluster 2024-03-11T21:03:13.518541+0000 osd.3 (osd.3) 255 : cluster 0 3.8 scrub ok 2024-03-11T21:03:14.537 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:14 smithi146 bash[23809]: cluster 2024-03-11T21:03:14.076957+0000 osd.7 (osd.7) 255 : cluster 0 3.d scrub starts 2024-03-11T21:03:14.537 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:14 smithi146 bash[23809]: cluster 2024-03-11T21:03:14.082567+0000 osd.7 (osd.7) 256 : cluster 0 3.d scrub ok 2024-03-11T21:03:14.637 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:14.663 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:14.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:14.665 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:03:14.766 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:14.784 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:14 smithi023 bash[21372]: cluster 2024-03-11T21:03:12.700221+0000 osd.2 (osd.2) 167 : cluster 0 2.e scrub starts 2024-03-11T21:03:14.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:14 smithi023 bash[21372]: cluster 2024-03-11T21:03:12.702458+0000 osd.2 (osd.2) 168 : cluster 0 2.e scrub ok 2024-03-11T21:03:14.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:14 smithi023 bash[21372]: cluster 2024-03-11T21:03:12.755528+0000 osd.0 (osd.0) 187 : cluster 0 3.1f scrub starts 2024-03-11T21:03:14.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:14 smithi023 bash[21372]: cluster 2024-03-11T21:03:12.772190+0000 osd.0 (osd.0) 188 : cluster 0 3.1f scrub ok 2024-03-11T21:03:14.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:14 smithi023 bash[21372]: cluster 2024-03-11T21:03:13.514286+0000 osd.3 (osd.3) 254 : cluster 0 3.8 scrub starts 2024-03-11T21:03:14.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:14 smithi023 bash[21372]: cluster 2024-03-11T21:03:13.518541+0000 osd.3 (osd.3) 255 : cluster 0 3.8 scrub ok 2024-03-11T21:03:14.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:14 smithi023 bash[21372]: cluster 2024-03-11T21:03:14.076957+0000 osd.7 (osd.7) 255 : cluster 0 3.d scrub starts 2024-03-11T21:03:14.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:14 smithi023 bash[21372]: cluster 2024-03-11T21:03:14.082567+0000 osd.7 (osd.7) 256 : cluster 0 3.d scrub ok 2024-03-11T21:03:14.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:14 smithi023 bash[29146]: cluster 2024-03-11T21:03:12.700221+0000 osd.2 (osd.2) 167 : cluster 0 2.e scrub starts 2024-03-11T21:03:14.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:14 smithi023 bash[29146]: cluster 2024-03-11T21:03:12.702458+0000 osd.2 (osd.2) 168 : cluster 0 2.e scrub ok 2024-03-11T21:03:14.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:14 smithi023 bash[29146]: cluster 2024-03-11T21:03:12.755528+0000 osd.0 (osd.0) 187 : cluster 0 3.1f scrub starts 2024-03-11T21:03:14.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:14 smithi023 bash[29146]: cluster 2024-03-11T21:03:12.772190+0000 osd.0 (osd.0) 188 : cluster 0 3.1f scrub ok 2024-03-11T21:03:14.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:14 smithi023 bash[29146]: cluster 2024-03-11T21:03:13.514286+0000 osd.3 (osd.3) 254 : cluster 0 3.8 scrub starts 2024-03-11T21:03:14.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:14 smithi023 bash[29146]: cluster 2024-03-11T21:03:13.518541+0000 osd.3 (osd.3) 255 : cluster 0 3.8 scrub ok 2024-03-11T21:03:14.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:14 smithi023 bash[29146]: cluster 2024-03-11T21:03:14.076957+0000 osd.7 (osd.7) 255 : cluster 0 3.d scrub starts 2024-03-11T21:03:14.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:14 smithi023 bash[29146]: cluster 2024-03-11T21:03:14.082567+0000 osd.7 (osd.7) 256 : cluster 0 3.d scrub ok 2024-03-11T21:03:14.790 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:14.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:14.792 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:03:14.893 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:03:14.916 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:03:14.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:14.920 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:03:15.020 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:15.044 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:15.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:15.047 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:03:15.148 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:03:15.172 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:03:15.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:15.175 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:03:15.276 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:03:15.300 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:03:15.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:15.303 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:03:15.404 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:15.428 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:15.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:15.430 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:03:15.454 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 13 16 27126 27110 130.317 120.438 0.0041277 0.00766752 2024-03-11T21:03:15.531 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:15.557 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:15.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:15.559 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:03:15.660 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:15.672 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:15 smithi023 bash[21372]: cluster 2024-03-11T21:03:13.726868+0000 osd.2 (osd.2) 169 : cluster 0 4.16 scrub starts 2024-03-11T21:03:15.673 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:15 smithi023 bash[21372]: cluster 2024-03-11T21:03:13.730924+0000 osd.2 (osd.2) 170 : cluster 0 4.16 scrub ok 2024-03-11T21:03:15.673 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:15 smithi023 bash[21372]: cluster 2024-03-11T21:03:14.149574+0000 mgr.y (mgr.24380) 461 : cluster 0 pgmap v511: 148 pgs: 148 active+clean; 1.1 GiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 90 MiB/s wr, 1.45k op/s 2024-03-11T21:03:15.673 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:15 smithi023 bash[21372]: cluster 2024-03-11T21:03:14.562906+0000 osd.3 (osd.3) 256 : cluster 0 6.b scrub starts 2024-03-11T21:03:15.673 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:15 smithi023 bash[21372]: cluster 2024-03-11T21:03:14.642351+0000 osd.3 (osd.3) 257 : cluster 0 6.b scrub ok 2024-03-11T21:03:15.678 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:15 smithi023 bash[29146]: cluster 2024-03-11T21:03:13.726868+0000 osd.2 (osd.2) 169 : cluster 0 4.16 scrub starts 2024-03-11T21:03:15.678 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:15 smithi023 bash[29146]: cluster 2024-03-11T21:03:13.730924+0000 osd.2 (osd.2) 170 : cluster 0 4.16 scrub ok 2024-03-11T21:03:15.678 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:15 smithi023 bash[29146]: cluster 2024-03-11T21:03:14.149574+0000 mgr.y (mgr.24380) 461 : cluster 0 pgmap v511: 148 pgs: 148 active+clean; 1.1 GiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 90 MiB/s wr, 1.45k op/s 2024-03-11T21:03:15.678 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:15 smithi023 bash[29146]: cluster 2024-03-11T21:03:14.562906+0000 osd.3 (osd.3) 256 : cluster 0 6.b scrub starts 2024-03-11T21:03:15.679 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:15 smithi023 bash[29146]: cluster 2024-03-11T21:03:14.642351+0000 osd.3 (osd.3) 257 : cluster 0 6.b scrub ok 2024-03-11T21:03:15.700 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:15.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:15.706 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:03:15.763 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:03:15 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:03:15.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:15 smithi146 bash[23809]: cluster 2024-03-11T21:03:13.726868+0000 osd.2 (osd.2) 169 : cluster 0 4.16 scrub starts 2024-03-11T21:03:15.764 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:15 smithi146 bash[23809]: cluster 2024-03-11T21:03:13.730924+0000 osd.2 (osd.2) 170 : cluster 0 4.16 scrub ok 2024-03-11T21:03:15.764 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:15 smithi146 bash[23809]: cluster 2024-03-11T21:03:14.149574+0000 mgr.y (mgr.24380) 461 : cluster 0 pgmap v511: 148 pgs: 148 active+clean; 1.1 GiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 90 MiB/s wr, 1.45k op/s 2024-03-11T21:03:15.764 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:15 smithi146 bash[23809]: cluster 2024-03-11T21:03:14.562906+0000 osd.3 (osd.3) 256 : cluster 0 6.b scrub starts 2024-03-11T21:03:15.764 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:15 smithi146 bash[23809]: cluster 2024-03-11T21:03:14.642351+0000 osd.3 (osd.3) 257 : cluster 0 6.b scrub ok 2024-03-11T21:03:15.807 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:15.830 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:15.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:15.832 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:03:15.933 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:03:15.975 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:03:15.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:15.981 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:03:16.082 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:16.110 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:16.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:16.113 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:03:16.214 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:16.244 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:16.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:16.248 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:03:16.349 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:03:16.387 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:03:16.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:16.389 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:03:16.410 INFO:teuthology.orchestra.run.smithi023.stderr:reweighted osd.6 to 1 (10000) 2024-03-11T21:03:16.454 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 14 16 28930 28914 129.061 112.75 0.00432171 0.00774166 2024-03-11T21:03:16.490 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:16.517 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:16.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:16.520 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:03:16.621 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:16.646 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:16.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:16.649 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:03:16.750 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:03:16.761 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:16 smithi023 bash[21372]: audit 2024-03-11T21:03:15.435589+0000 mgr.y (mgr.24380) 462 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:03:16.761 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:16 smithi023 bash[21372]: cluster 2024-03-11T21:03:15.615926+0000 osd.3 (osd.3) 258 : cluster 0 4.17 scrub starts 2024-03-11T21:03:16.761 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:16 smithi023 bash[21372]: cluster 2024-03-11T21:03:15.618328+0000 osd.3 (osd.3) 259 : cluster 0 4.17 scrub ok 2024-03-11T21:03:16.761 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:16 smithi023 bash[21372]: cluster 2024-03-11T21:03:15.727097+0000 osd.5 (osd.5) 225 : cluster 0 3.5 scrub starts 2024-03-11T21:03:16.761 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:16 smithi023 bash[21372]: cluster 2024-03-11T21:03:15.734877+0000 osd.5 (osd.5) 226 : cluster 0 3.5 scrub ok 2024-03-11T21:03:16.761 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:16 smithi023 bash[21372]: audit 2024-03-11T21:03:16.137035+0000 mon.c (mon.2) 59 : audit 1 from='client.? 172.21.15.23:0/1221255509' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-11T21:03:16.762 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:16 smithi023 bash[21372]: audit 2024-03-11T21:03:16.137909+0000 mon.a (mon.0) 1458 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-11T21:03:16.762 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:16 smithi023 bash[29146]: audit 2024-03-11T21:03:15.435589+0000 mgr.y (mgr.24380) 462 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:03:16.762 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:16 smithi023 bash[29146]: cluster 2024-03-11T21:03:15.615926+0000 osd.3 (osd.3) 258 : cluster 0 4.17 scrub starts 2024-03-11T21:03:16.762 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:16 smithi023 bash[29146]: cluster 2024-03-11T21:03:15.618328+0000 osd.3 (osd.3) 259 : cluster 0 4.17 scrub ok 2024-03-11T21:03:16.762 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:16 smithi023 bash[29146]: cluster 2024-03-11T21:03:15.727097+0000 osd.5 (osd.5) 225 : cluster 0 3.5 scrub starts 2024-03-11T21:03:16.762 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:16 smithi023 bash[29146]: cluster 2024-03-11T21:03:15.734877+0000 osd.5 (osd.5) 226 : cluster 0 3.5 scrub ok 2024-03-11T21:03:16.762 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:16 smithi023 bash[29146]: audit 2024-03-11T21:03:16.137035+0000 mon.c (mon.2) 59 : audit 1 from='client.? 172.21.15.23:0/1221255509' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-11T21:03:16.762 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:16 smithi023 bash[29146]: audit 2024-03-11T21:03:16.137909+0000 mon.a (mon.0) 1458 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-11T21:03:16.781 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:03:16.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:16.786 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:03:16.886 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:03:16.915 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:03:16.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:16.918 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:03:17.014 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:16 smithi146 bash[23809]: audit 2024-03-11T21:03:15.435589+0000 mgr.y (mgr.24380) 462 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:03:17.014 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:16 smithi146 bash[23809]: cluster 2024-03-11T21:03:15.615926+0000 osd.3 (osd.3) 258 : cluster 0 4.17 scrub starts 2024-03-11T21:03:17.014 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:16 smithi146 bash[23809]: cluster 2024-03-11T21:03:15.618328+0000 osd.3 (osd.3) 259 : cluster 0 4.17 scrub ok 2024-03-11T21:03:17.014 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:16 smithi146 bash[23809]: cluster 2024-03-11T21:03:15.727097+0000 osd.5 (osd.5) 225 : cluster 0 3.5 scrub starts 2024-03-11T21:03:17.014 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:16 smithi146 bash[23809]: cluster 2024-03-11T21:03:15.734877+0000 osd.5 (osd.5) 226 : cluster 0 3.5 scrub ok 2024-03-11T21:03:17.014 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:16 smithi146 bash[23809]: audit 2024-03-11T21:03:16.137035+0000 mon.c (mon.2) 59 : audit 1 from='client.? 172.21.15.23:0/1221255509' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-11T21:03:17.014 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:16 smithi146 bash[23809]: audit 2024-03-11T21:03:16.137909+0000 mon.a (mon.0) 1458 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-11T21:03:17.019 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:03:17.045 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:03:17.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:17.047 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:03:17.148 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:03:17.178 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:03:17.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:17.181 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:03:17.282 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:03:17.314 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:03:17.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:17.317 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:03:17.417 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:03:17.445 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:03:17.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:17.447 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:03:17.454 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 15 16 30833 30817 128.386 118.938 0.00506077 0.00778227 2024-03-11T21:03:17.548 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:03:17.559 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-03-11T21:03:17.559 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:03:17.559 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:17 smithi146 bash[23809]: cluster 2024-03-11T21:03:15.681669+0000 osd.2 (osd.2) 171 : cluster 0 2.c scrub starts 2024-03-11T21:03:17.559 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:17 smithi146 bash[23809]: cluster 2024-03-11T21:03:15.683069+0000 osd.2 (osd.2) 172 : cluster 0 2.c scrub ok 2024-03-11T21:03:17.560 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:17 smithi146 bash[23809]: cluster 2024-03-11T21:03:16.150431+0000 mgr.y (mgr.24380) 463 : cluster 0 pgmap v512: 148 pgs: 148 active+clean; 1.6 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 123 MiB/s wr, 1.97k op/s 2024-03-11T21:03:17.560 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:17 smithi146 bash[23809]: cluster 2024-03-11T21:03:16.355079+0000 osd.4 (osd.4) 47 : cluster 0 4.1a scrub starts 2024-03-11T21:03:17.560 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:17 smithi146 bash[23809]: cluster 2024-03-11T21:03:16.356476+0000 osd.4 (osd.4) 48 : cluster 0 4.1a scrub ok 2024-03-11T21:03:17.560 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:17 smithi146 bash[23809]: audit 2024-03-11T21:03:16.395527+0000 mon.a (mon.0) 1459 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-11T21:03:17.560 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:17 smithi146 bash[23809]: cluster 2024-03-11T21:03:16.400014+0000 mon.a (mon.0) 1460 : cluster 0 osdmap e260: 8 total, 8 up, 8 in 2024-03-11T21:03:17.560 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:17 smithi146 bash[23809]: cluster 2024-03-11T21:03:17.000183+0000 osd.7 (osd.7) 257 : cluster 0 6.8 scrub starts 2024-03-11T21:03:17.560 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:17 smithi146 bash[23809]: cluster 2024-03-11T21:03:17.003970+0000 osd.7 (osd.7) 258 : cluster 0 6.8 scrub ok 2024-03-11T21:03:17.577 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:03:17.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:17.580 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:03:17.681 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:03:17.714 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:03:17.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:17.720 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:03:17.784 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:17 smithi023 bash[21372]: cluster 2024-03-11T21:03:15.681669+0000 osd.2 (osd.2) 171 : cluster 0 2.c scrub starts 2024-03-11T21:03:17.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:17 smithi023 bash[21372]: cluster 2024-03-11T21:03:15.683069+0000 osd.2 (osd.2) 172 : cluster 0 2.c scrub ok 2024-03-11T21:03:17.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:17 smithi023 bash[21372]: cluster 2024-03-11T21:03:16.150431+0000 mgr.y (mgr.24380) 463 : cluster 0 pgmap v512: 148 pgs: 148 active+clean; 1.6 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 123 MiB/s wr, 1.97k op/s 2024-03-11T21:03:17.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:17 smithi023 bash[21372]: cluster 2024-03-11T21:03:16.355079+0000 osd.4 (osd.4) 47 : cluster 0 4.1a scrub starts 2024-03-11T21:03:17.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:17 smithi023 bash[21372]: cluster 2024-03-11T21:03:16.356476+0000 osd.4 (osd.4) 48 : cluster 0 4.1a scrub ok 2024-03-11T21:03:17.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:17 smithi023 bash[21372]: audit 2024-03-11T21:03:16.395527+0000 mon.a (mon.0) 1459 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-11T21:03:17.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:17 smithi023 bash[21372]: cluster 2024-03-11T21:03:16.400014+0000 mon.a (mon.0) 1460 : cluster 0 osdmap e260: 8 total, 8 up, 8 in 2024-03-11T21:03:17.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:17 smithi023 bash[21372]: cluster 2024-03-11T21:03:17.000183+0000 osd.7 (osd.7) 257 : cluster 0 6.8 scrub starts 2024-03-11T21:03:17.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:17 smithi023 bash[21372]: cluster 2024-03-11T21:03:17.003970+0000 osd.7 (osd.7) 258 : cluster 0 6.8 scrub ok 2024-03-11T21:03:17.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:17 smithi023 bash[29146]: cluster 2024-03-11T21:03:15.681669+0000 osd.2 (osd.2) 171 : cluster 0 2.c scrub starts 2024-03-11T21:03:17.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:17 smithi023 bash[29146]: cluster 2024-03-11T21:03:15.683069+0000 osd.2 (osd.2) 172 : cluster 0 2.c scrub ok 2024-03-11T21:03:17.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:17 smithi023 bash[29146]: cluster 2024-03-11T21:03:16.150431+0000 mgr.y (mgr.24380) 463 : cluster 0 pgmap v512: 148 pgs: 148 active+clean; 1.6 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 123 MiB/s wr, 1.97k op/s 2024-03-11T21:03:17.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:17 smithi023 bash[29146]: cluster 2024-03-11T21:03:16.355079+0000 osd.4 (osd.4) 47 : cluster 0 4.1a scrub starts 2024-03-11T21:03:17.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:17 smithi023 bash[29146]: cluster 2024-03-11T21:03:16.356476+0000 osd.4 (osd.4) 48 : cluster 0 4.1a scrub ok 2024-03-11T21:03:17.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:17 smithi023 bash[29146]: audit 2024-03-11T21:03:16.395527+0000 mon.a (mon.0) 1459 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-11T21:03:17.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:17 smithi023 bash[29146]: cluster 2024-03-11T21:03:16.400014+0000 mon.a (mon.0) 1460 : cluster 0 osdmap e260: 8 total, 8 up, 8 in 2024-03-11T21:03:17.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:17 smithi023 bash[29146]: cluster 2024-03-11T21:03:17.000183+0000 osd.7 (osd.7) 257 : cluster 0 6.8 scrub starts 2024-03-11T21:03:17.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:17 smithi023 bash[29146]: cluster 2024-03-11T21:03:17.003970+0000 osd.7 (osd.7) 258 : cluster 0 6.8 scrub ok 2024-03-11T21:03:17.821 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:17.850 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:17.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:17.853 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:03:17.954 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:03:17.979 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:03:17.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:17.986 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:03:18.086 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:18.111 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:18.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:18.116 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:03:18.217 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:03:18.245 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:03:18.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:18.249 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:03:18.257 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd reweight 7 1 2024-03-11T21:03:18.350 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:03:18.376 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:03:18.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:18.378 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:03:18.454 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 16 16 32644 32628 127.435 113.188 0.00535395 0.00783985 2024-03-11T21:03:18.479 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:03:18.488 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:03:18 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:03:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:03:18.489 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:18 smithi023 bash[21372]: cluster 2024-03-11T21:03:17.296523+0000 osd.1 (osd.1) 31 : cluster 0 2.d scrub starts 2024-03-11T21:03:18.489 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:18 smithi023 bash[21372]: cluster 2024-03-11T21:03:17.298798+0000 osd.1 (osd.1) 32 : cluster 0 2.d scrub ok 2024-03-11T21:03:18.489 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:18 smithi023 bash[21372]: cluster 2024-03-11T21:03:17.505378+0000 osd.6 (osd.6) 188 : cluster 0 4.5 scrub starts 2024-03-11T21:03:18.489 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:18 smithi023 bash[21372]: cluster 2024-03-11T21:03:17.506575+0000 osd.6 (osd.6) 189 : cluster 0 4.5 scrub ok 2024-03-11T21:03:18.489 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:18 smithi023 bash[29146]: cluster 2024-03-11T21:03:17.296523+0000 osd.1 (osd.1) 31 : cluster 0 2.d scrub starts 2024-03-11T21:03:18.489 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:18 smithi023 bash[29146]: cluster 2024-03-11T21:03:17.298798+0000 osd.1 (osd.1) 32 : cluster 0 2.d scrub ok 2024-03-11T21:03:18.489 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:18 smithi023 bash[29146]: cluster 2024-03-11T21:03:17.505378+0000 osd.6 (osd.6) 188 : cluster 0 4.5 scrub starts 2024-03-11T21:03:18.490 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:18 smithi023 bash[29146]: cluster 2024-03-11T21:03:17.506575+0000 osd.6 (osd.6) 189 : cluster 0 4.5 scrub ok 2024-03-11T21:03:18.507 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:03:18.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:18.513 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:03:18.614 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:03:18.624 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:18 smithi146 bash[23809]: cluster 2024-03-11T21:03:17.296523+0000 osd.1 (osd.1) 31 : cluster 0 2.d scrub starts 2024-03-11T21:03:18.624 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:18 smithi146 bash[23809]: cluster 2024-03-11T21:03:17.298798+0000 osd.1 (osd.1) 32 : cluster 0 2.d scrub ok 2024-03-11T21:03:18.625 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:18 smithi146 bash[23809]: cluster 2024-03-11T21:03:17.505378+0000 osd.6 (osd.6) 188 : cluster 0 4.5 scrub starts 2024-03-11T21:03:18.625 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:18 smithi146 bash[23809]: cluster 2024-03-11T21:03:17.506575+0000 osd.6 (osd.6) 189 : cluster 0 4.5 scrub ok 2024-03-11T21:03:18.642 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:03:18.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:18.644 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:03:18.655 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:03:18.656 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:03:18.746 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:03:18.775 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:03:18.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:18.778 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:03:18.879 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:18.904 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:18.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:18.908 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:03:19.009 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:03:19.033 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:03:19.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:19.035 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:03:19.136 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:19.159 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:19.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:19.161 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:03:19.262 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:03:19.289 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:03:19.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:19.291 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:03:19.392 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:03:19.420 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:03:19.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:19.425 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:03:19.454 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 17 16 33881 33865 124.487 77.3125 0.0128101 0.0080274 2024-03-11T21:03:19.526 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:19.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:19 smithi023 bash[21372]: cluster 2024-03-11T21:03:18.150980+0000 mgr.y (mgr.24380) 464 : cluster 0 pgmap v514: 148 pgs: 148 active+clean; 1.7 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 119 MiB/s wr, 1.91k op/s 2024-03-11T21:03:19.536 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:19 smithi023 bash[29146]: cluster 2024-03-11T21:03:18.150980+0000 mgr.y (mgr.24380) 464 : cluster 0 pgmap v514: 148 pgs: 148 active+clean; 1.7 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 119 MiB/s wr, 1.91k op/s 2024-03-11T21:03:19.559 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:19.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:19.566 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:03:19.667 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:19.696 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:19.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:19.699 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:03:19.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:19 smithi146 bash[23809]: cluster 2024-03-11T21:03:18.150980+0000 mgr.y (mgr.24380) 464 : cluster 0 pgmap v514: 148 pgs: 148 active+clean; 1.7 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 119 MiB/s wr, 1.91k op/s 2024-03-11T21:03:19.800 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:19.823 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:19.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:19.827 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:03:19.927 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:03:19.945 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:03:19.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:19.947 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:03:20.048 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:03:20.075 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:03:20.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:20.079 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:03:20.180 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:20.209 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:20.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:20.219 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:03:20.229 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-03-11T21:03:20.320 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:03:20.346 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:03:20.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:20.348 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:03:20.449 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:20.454 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 18 16 35270 35254 122.393 86.8125 0.0106731 0.00816385 2024-03-11T21:03:20.458 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:20 smithi023 bash[21372]: cluster 2024-03-11T21:03:20.064446+0000 osd.7 (osd.7) 259 : cluster 0 2.8 scrub starts 2024-03-11T21:03:20.458 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:20 smithi023 bash[21372]: cluster 2024-03-11T21:03:20.066574+0000 osd.7 (osd.7) 260 : cluster 0 2.8 scrub ok 2024-03-11T21:03:20.459 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:20 smithi023 bash[21372]: audit 2024-03-11T21:03:20.174209+0000 mon.a (mon.0) 1461 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:03:20.459 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:20 smithi023 bash[21372]: cluster 2024-03-11T21:03:20.203626+0000 mon.a (mon.0) 1462 : cluster 0 osdmap e261: 8 total, 8 up, 8 in 2024-03-11T21:03:20.459 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:20 smithi023 bash[29146]: cluster 2024-03-11T21:03:20.064446+0000 osd.7 (osd.7) 259 : cluster 0 2.8 scrub starts 2024-03-11T21:03:20.459 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:20 smithi023 bash[29146]: cluster 2024-03-11T21:03:20.066574+0000 osd.7 (osd.7) 260 : cluster 0 2.8 scrub ok 2024-03-11T21:03:20.459 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:20 smithi023 bash[29146]: audit 2024-03-11T21:03:20.174209+0000 mon.a (mon.0) 1461 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:03:20.459 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:20 smithi023 bash[29146]: cluster 2024-03-11T21:03:20.203626+0000 mon.a (mon.0) 1462 : cluster 0 osdmap e261: 8 total, 8 up, 8 in 2024-03-11T21:03:20.474 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:20.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:20.476 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:03:20.513 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:20 smithi146 bash[23809]: cluster 2024-03-11T21:03:20.064446+0000 osd.7 (osd.7) 259 : cluster 0 2.8 scrub starts 2024-03-11T21:03:20.513 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:20 smithi146 bash[23809]: cluster 2024-03-11T21:03:20.066574+0000 osd.7 (osd.7) 260 : cluster 0 2.8 scrub ok 2024-03-11T21:03:20.513 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:20 smithi146 bash[23809]: audit 2024-03-11T21:03:20.174209+0000 mon.a (mon.0) 1461 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:03:20.513 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:20 smithi146 bash[23809]: cluster 2024-03-11T21:03:20.203626+0000 mon.a (mon.0) 1462 : cluster 0 osdmap e261: 8 total, 8 up, 8 in 2024-03-11T21:03:20.577 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:03:20.606 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:03:20.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:20.609 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:03:20.710 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:03:20.733 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:03:20.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:20.735 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:03:20.836 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:03:20.861 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:03:20.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:20.867 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:03:20.968 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:20.993 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:20.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:20.996 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:03:21.097 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:03:21.126 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:03:21.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:21.130 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:03:21.231 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:03:21.251 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:03:21.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:21.253 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:03:21.354 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:21.384 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:21.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:21.386 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:03:21.455 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 19 16 36617 36601 120.382 84.1875 0.00845278 0.00830175 2024-03-11T21:03:21.487 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:03:21.498 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:21 smithi023 bash[21372]: cluster 2024-03-11T21:03:19.685499+0000 osd.2 (osd.2) 173 : cluster 0 2.1 scrub starts 2024-03-11T21:03:21.499 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:21 smithi023 bash[21372]: cluster 2024-03-11T21:03:19.688068+0000 osd.2 (osd.2) 174 : cluster 0 2.1 scrub ok 2024-03-11T21:03:21.499 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:21 smithi023 bash[21372]: cluster 2024-03-11T21:03:20.151685+0000 mgr.y (mgr.24380) 465 : cluster 0 pgmap v515: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 2.0 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 112 MiB/s wr, 1.79k op/s 2024-03-11T21:03:21.499 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:21 smithi023 bash[21372]: cluster 2024-03-11T21:03:20.311005+0000 osd.1 (osd.1) 33 : cluster 0 6.18 scrub starts 2024-03-11T21:03:21.499 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:21 smithi023 bash[21372]: cluster 2024-03-11T21:03:20.329906+0000 osd.1 (osd.1) 34 : cluster 0 6.18 scrub ok 2024-03-11T21:03:21.499 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:21 smithi023 bash[29146]: cluster 2024-03-11T21:03:19.685499+0000 osd.2 (osd.2) 173 : cluster 0 2.1 scrub starts 2024-03-11T21:03:21.499 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:21 smithi023 bash[29146]: cluster 2024-03-11T21:03:19.688068+0000 osd.2 (osd.2) 174 : cluster 0 2.1 scrub ok 2024-03-11T21:03:21.499 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:21 smithi023 bash[29146]: cluster 2024-03-11T21:03:20.151685+0000 mgr.y (mgr.24380) 465 : cluster 0 pgmap v515: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 2.0 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 112 MiB/s wr, 1.79k op/s 2024-03-11T21:03:21.500 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:21 smithi023 bash[29146]: cluster 2024-03-11T21:03:20.311005+0000 osd.1 (osd.1) 33 : cluster 0 6.18 scrub starts 2024-03-11T21:03:21.500 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:21 smithi023 bash[29146]: cluster 2024-03-11T21:03:20.329906+0000 osd.1 (osd.1) 34 : cluster 0 6.18 scrub ok 2024-03-11T21:03:21.513 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:03:21.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:21.517 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:03:21.618 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:21.625 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:21 smithi146 bash[23809]: cluster 2024-03-11T21:03:19.685499+0000 osd.2 (osd.2) 173 : cluster 0 2.1 scrub starts 2024-03-11T21:03:21.625 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:21 smithi146 bash[23809]: cluster 2024-03-11T21:03:19.688068+0000 osd.2 (osd.2) 174 : cluster 0 2.1 scrub ok 2024-03-11T21:03:21.625 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:21 smithi146 bash[23809]: cluster 2024-03-11T21:03:20.151685+0000 mgr.y (mgr.24380) 465 : cluster 0 pgmap v515: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 2.0 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 112 MiB/s wr, 1.79k op/s 2024-03-11T21:03:21.626 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:21 smithi146 bash[23809]: cluster 2024-03-11T21:03:20.311005+0000 osd.1 (osd.1) 33 : cluster 0 6.18 scrub starts 2024-03-11T21:03:21.626 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:21 smithi146 bash[23809]: cluster 2024-03-11T21:03:20.329906+0000 osd.1 (osd.1) 34 : cluster 0 6.18 scrub ok 2024-03-11T21:03:21.637 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:21.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:21.639 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:03:21.739 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:21.760 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:21.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:21.763 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:03:21.863 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:03:21.894 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:03:21.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:21.896 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:03:21.997 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:22.029 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:22.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:22.031 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:03:22.132 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:22.158 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:22.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:22.160 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:03:22.261 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:03:22.290 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:03:22.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:22.293 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:03:22.394 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:03:22.419 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:03:22.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:22.421 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:03:22.455 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:2024-03-11T21:03:22.459220+0000 min lat: 0.0011821 max lat: 0.402819 avg lat: 0.0084046 2024-03-11T21:03:22.455 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-11T21:03:22.455 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 20 16 38071 38055 118.906 90.875 0.00640018 0.0084046 2024-03-11T21:03:22.522 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:22.549 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:22.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:22.553 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:03:22.654 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:03:22.682 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:03:22.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:22.684 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:03:22.785 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:03:22.811 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:03:22.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:22.813 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:03:22.914 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:22.942 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:22.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:22.944 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:03:23.045 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:03:23.072 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:03:23.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:23.076 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:03:23.177 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:03:23.202 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:03:23.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:23.205 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:03:23.306 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:03:23.334 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:03:23.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:23.340 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:03:23.441 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:23.454 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 21 16 39750 39734 118.241 104.938 0.007784 0.00845145 2024-03-11T21:03:23.469 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:23.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:23.471 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:03:23.572 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:03:23.597 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:03:23.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:23.599 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:03:23.700 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:23.727 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:23.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:23.730 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:03:23.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:23 smithi146 bash[23809]: cluster 2024-03-11T21:03:21.875237+0000 osd.5 (osd.5) 227 : cluster 0 3.17 scrub starts 2024-03-11T21:03:23.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:23 smithi146 bash[23809]: cluster 2024-03-11T21:03:21.879675+0000 osd.5 (osd.5) 228 : cluster 0 3.17 scrub ok 2024-03-11T21:03:23.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:23 smithi146 bash[23809]: cluster 2024-03-11T21:03:22.152888+0000 mgr.y (mgr.24380) 466 : cluster 0 pgmap v517: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 2.2 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 133 MiB/s wr, 2.13k op/s 2024-03-11T21:03:23.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:23 smithi146 bash[23809]: cluster 2024-03-11T21:03:22.315996+0000 osd.4 (osd.4) 49 : cluster 0 3.3 scrub starts 2024-03-11T21:03:23.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:23 smithi146 bash[23809]: cluster 2024-03-11T21:03:22.320636+0000 osd.4 (osd.4) 50 : cluster 0 3.3 scrub ok 2024-03-11T21:03:23.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:23 smithi146 bash[23809]: cluster 2024-03-11T21:03:22.616967+0000 osd.2 (osd.2) 175 : cluster 0 2.10 scrub starts 2024-03-11T21:03:23.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:23 smithi146 bash[23809]: cluster 2024-03-11T21:03:22.623469+0000 osd.2 (osd.2) 176 : cluster 0 2.10 scrub ok 2024-03-11T21:03:23.831 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:23.860 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:23.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:23.865 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:03:23.865 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:23 smithi023 bash[21372]: cluster 2024-03-11T21:03:21.875237+0000 osd.5 (osd.5) 227 : cluster 0 3.17 scrub starts 2024-03-11T21:03:23.865 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:23 smithi023 bash[21372]: cluster 2024-03-11T21:03:21.879675+0000 osd.5 (osd.5) 228 : cluster 0 3.17 scrub ok 2024-03-11T21:03:23.865 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:23 smithi023 bash[21372]: cluster 2024-03-11T21:03:22.152888+0000 mgr.y (mgr.24380) 466 : cluster 0 pgmap v517: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 2.2 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 133 MiB/s wr, 2.13k op/s 2024-03-11T21:03:23.865 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:23 smithi023 bash[21372]: cluster 2024-03-11T21:03:22.315996+0000 osd.4 (osd.4) 49 : cluster 0 3.3 scrub starts 2024-03-11T21:03:23.865 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:23 smithi023 bash[21372]: cluster 2024-03-11T21:03:22.320636+0000 osd.4 (osd.4) 50 : cluster 0 3.3 scrub ok 2024-03-11T21:03:23.865 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:23 smithi023 bash[21372]: cluster 2024-03-11T21:03:22.616967+0000 osd.2 (osd.2) 175 : cluster 0 2.10 scrub starts 2024-03-11T21:03:23.865 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:23 smithi023 bash[21372]: cluster 2024-03-11T21:03:22.623469+0000 osd.2 (osd.2) 176 : cluster 0 2.10 scrub ok 2024-03-11T21:03:23.866 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:23 smithi023 bash[29146]: cluster 2024-03-11T21:03:21.875237+0000 osd.5 (osd.5) 227 : cluster 0 3.17 scrub starts 2024-03-11T21:03:23.866 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:23 smithi023 bash[29146]: cluster 2024-03-11T21:03:21.879675+0000 osd.5 (osd.5) 228 : cluster 0 3.17 scrub ok 2024-03-11T21:03:23.866 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:23 smithi023 bash[29146]: cluster 2024-03-11T21:03:22.152888+0000 mgr.y (mgr.24380) 466 : cluster 0 pgmap v517: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 2.2 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 133 MiB/s wr, 2.13k op/s 2024-03-11T21:03:23.866 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:23 smithi023 bash[29146]: cluster 2024-03-11T21:03:22.315996+0000 osd.4 (osd.4) 49 : cluster 0 3.3 scrub starts 2024-03-11T21:03:23.866 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:23 smithi023 bash[29146]: cluster 2024-03-11T21:03:22.320636+0000 osd.4 (osd.4) 50 : cluster 0 3.3 scrub ok 2024-03-11T21:03:23.866 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:23 smithi023 bash[29146]: cluster 2024-03-11T21:03:22.616967+0000 osd.2 (osd.2) 175 : cluster 0 2.10 scrub starts 2024-03-11T21:03:23.866 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:23 smithi023 bash[29146]: cluster 2024-03-11T21:03:22.623469+0000 osd.2 (osd.2) 176 : cluster 0 2.10 scrub ok 2024-03-11T21:03:23.966 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:03:23.997 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:03:23.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:24.000 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:03:24.101 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:03:24.126 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:03:24.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:24.130 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:03:24.231 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:24.256 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:24.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:24.260 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:03:24.361 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:03:24.388 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:03:24.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:24.394 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:03:24.455 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 22 16 41339 41323 117.38 99.3125 0.00400248 0.00851346 2024-03-11T21:03:24.495 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:03:24.525 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:03:24.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:24.527 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:03:24.628 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:03:24.638 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:24 smithi023 bash[29146]: cluster 2024-03-11T21:03:23.345376+0000 osd.4 (osd.4) 51 : cluster 0 4.d scrub starts 2024-03-11T21:03:24.638 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:24 smithi023 bash[29146]: cluster 2024-03-11T21:03:23.347226+0000 osd.4 (osd.4) 52 : cluster 0 4.d scrub ok 2024-03-11T21:03:24.639 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:24 smithi023 bash[29146]: cluster 2024-03-11T21:03:23.361935+0000 osd.1 (osd.1) 35 : cluster 0 2.15 scrub starts 2024-03-11T21:03:24.639 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:24 smithi023 bash[29146]: cluster 2024-03-11T21:03:23.365618+0000 osd.1 (osd.1) 36 : cluster 0 2.15 scrub ok 2024-03-11T21:03:24.639 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:24 smithi023 bash[29146]: cluster 2024-03-11T21:03:23.585072+0000 osd.6 (osd.6) 190 : cluster 0 2.17 scrub starts 2024-03-11T21:03:24.639 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:24 smithi023 bash[29146]: cluster 2024-03-11T21:03:23.586536+0000 osd.6 (osd.6) 191 : cluster 0 2.17 scrub ok 2024-03-11T21:03:24.639 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:24 smithi023 bash[29146]: cluster 2024-03-11T21:03:23.936047+0000 osd.7 (osd.7) 261 : cluster 0 3.e scrub starts 2024-03-11T21:03:24.639 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:24 smithi023 bash[29146]: cluster 2024-03-11T21:03:23.940961+0000 osd.7 (osd.7) 262 : cluster 0 3.e scrub ok 2024-03-11T21:03:24.640 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:24 smithi023 bash[21372]: cluster 2024-03-11T21:03:23.345376+0000 osd.4 (osd.4) 51 : cluster 0 4.d scrub starts 2024-03-11T21:03:24.640 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:24 smithi023 bash[21372]: cluster 2024-03-11T21:03:23.347226+0000 osd.4 (osd.4) 52 : cluster 0 4.d scrub ok 2024-03-11T21:03:24.640 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:24 smithi023 bash[21372]: cluster 2024-03-11T21:03:23.361935+0000 osd.1 (osd.1) 35 : cluster 0 2.15 scrub starts 2024-03-11T21:03:24.640 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:24 smithi023 bash[21372]: cluster 2024-03-11T21:03:23.365618+0000 osd.1 (osd.1) 36 : cluster 0 2.15 scrub ok 2024-03-11T21:03:24.640 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:24 smithi023 bash[21372]: cluster 2024-03-11T21:03:23.585072+0000 osd.6 (osd.6) 190 : cluster 0 2.17 scrub starts 2024-03-11T21:03:24.640 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:24 smithi023 bash[21372]: cluster 2024-03-11T21:03:23.586536+0000 osd.6 (osd.6) 191 : cluster 0 2.17 scrub ok 2024-03-11T21:03:24.640 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:24 smithi023 bash[21372]: cluster 2024-03-11T21:03:23.936047+0000 osd.7 (osd.7) 261 : cluster 0 3.e scrub starts 2024-03-11T21:03:24.640 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:24 smithi023 bash[21372]: cluster 2024-03-11T21:03:23.940961+0000 osd.7 (osd.7) 262 : cluster 0 3.e scrub ok 2024-03-11T21:03:24.653 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:03:24.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:24.656 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:03:24.757 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:24.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:24 smithi146 bash[23809]: cluster 2024-03-11T21:03:23.345376+0000 osd.4 (osd.4) 51 : cluster 0 4.d scrub starts 2024-03-11T21:03:24.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:24 smithi146 bash[23809]: cluster 2024-03-11T21:03:23.347226+0000 osd.4 (osd.4) 52 : cluster 0 4.d scrub ok 2024-03-11T21:03:24.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:24 smithi146 bash[23809]: cluster 2024-03-11T21:03:23.361935+0000 osd.1 (osd.1) 35 : cluster 0 2.15 scrub starts 2024-03-11T21:03:24.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:24 smithi146 bash[23809]: cluster 2024-03-11T21:03:23.365618+0000 osd.1 (osd.1) 36 : cluster 0 2.15 scrub ok 2024-03-11T21:03:24.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:24 smithi146 bash[23809]: cluster 2024-03-11T21:03:23.585072+0000 osd.6 (osd.6) 190 : cluster 0 2.17 scrub starts 2024-03-11T21:03:24.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:24 smithi146 bash[23809]: cluster 2024-03-11T21:03:23.586536+0000 osd.6 (osd.6) 191 : cluster 0 2.17 scrub ok 2024-03-11T21:03:24.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:24 smithi146 bash[23809]: cluster 2024-03-11T21:03:23.936047+0000 osd.7 (osd.7) 261 : cluster 0 3.e scrub starts 2024-03-11T21:03:24.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:24 smithi146 bash[23809]: cluster 2024-03-11T21:03:23.940961+0000 osd.7 (osd.7) 262 : cluster 0 3.e scrub ok 2024-03-11T21:03:24.782 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:24.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:24.789 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:03:24.890 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:03:24.905 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:03:24.913 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:03:24.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:24.915 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:03:25.015 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:25.041 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:03:25.048 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:25.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:25.053 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:03:25.154 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:03:25.180 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:03:25.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:25.183 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:03:25.284 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:25.316 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:25.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:25.319 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:03:25.420 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:03:25.437 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:03:25.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:25.439 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:03:25.455 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 23 16 42729 42713 116.053 86.875 0.0477135 0.00860922 2024-03-11T21:03:25.542 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:03:25.574 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:03:25.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:25.577 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:03:25.678 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:25.702 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:25.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:25.709 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:03:25.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:25 smithi146 bash[23809]: cluster 2024-03-11T21:03:24.153276+0000 mgr.y (mgr.24380) 467 : cluster 0 pgmap v518: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 2.2 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 72 MiB/s wr, 1.16k op/s 2024-03-11T21:03:25.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:25 smithi146 bash[23809]: cluster 2024-03-11T21:03:24.363594+0000 osd.4 (osd.4) 53 : cluster 0 4.e scrub starts 2024-03-11T21:03:25.764 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:25 smithi146 bash[23809]: cluster 2024-03-11T21:03:24.365343+0000 osd.4 (osd.4) 54 : cluster 0 4.e scrub ok 2024-03-11T21:03:25.764 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:25 smithi146 bash[23809]: cluster 2024-03-11T21:03:24.919000+0000 osd.7 (osd.7) 263 : cluster 0 6.12 scrub starts 2024-03-11T21:03:25.764 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:25 smithi146 bash[23809]: cluster 2024-03-11T21:03:24.925429+0000 osd.7 (osd.7) 264 : cluster 0 6.12 scrub ok 2024-03-11T21:03:25.764 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:03:25 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:03:25.784 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:25 smithi023 bash[21372]: cluster 2024-03-11T21:03:24.153276+0000 mgr.y (mgr.24380) 467 : cluster 0 pgmap v518: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 2.2 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 72 MiB/s wr, 1.16k op/s 2024-03-11T21:03:25.784 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:25 smithi023 bash[21372]: cluster 2024-03-11T21:03:24.363594+0000 osd.4 (osd.4) 53 : cluster 0 4.e scrub starts 2024-03-11T21:03:25.784 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:25 smithi023 bash[21372]: cluster 2024-03-11T21:03:24.365343+0000 osd.4 (osd.4) 54 : cluster 0 4.e scrub ok 2024-03-11T21:03:25.784 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:25 smithi023 bash[21372]: cluster 2024-03-11T21:03:24.919000+0000 osd.7 (osd.7) 263 : cluster 0 6.12 scrub starts 2024-03-11T21:03:25.784 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:25 smithi023 bash[21372]: cluster 2024-03-11T21:03:24.925429+0000 osd.7 (osd.7) 264 : cluster 0 6.12 scrub ok 2024-03-11T21:03:25.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:25 smithi023 bash[29146]: cluster 2024-03-11T21:03:24.153276+0000 mgr.y (mgr.24380) 467 : cluster 0 pgmap v518: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 2.2 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 72 MiB/s wr, 1.16k op/s 2024-03-11T21:03:25.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:25 smithi023 bash[29146]: cluster 2024-03-11T21:03:24.363594+0000 osd.4 (osd.4) 53 : cluster 0 4.e scrub starts 2024-03-11T21:03:25.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:25 smithi023 bash[29146]: cluster 2024-03-11T21:03:24.365343+0000 osd.4 (osd.4) 54 : cluster 0 4.e scrub ok 2024-03-11T21:03:25.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:25 smithi023 bash[29146]: cluster 2024-03-11T21:03:24.919000+0000 osd.7 (osd.7) 263 : cluster 0 6.12 scrub starts 2024-03-11T21:03:25.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:25 smithi023 bash[29146]: cluster 2024-03-11T21:03:24.925429+0000 osd.7 (osd.7) 264 : cluster 0 6.12 scrub ok 2024-03-11T21:03:25.809 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:25.835 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:25.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:25.837 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:03:25.938 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:03:25.965 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:03:25.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:25.968 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:03:26.068 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:03:26.094 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:03:26.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:26.097 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:03:26.197 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:03:26.229 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:03:26.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:26.233 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:03:26.334 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:03:26.366 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:03:26.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:26.369 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:03:26.455 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 24 16 44351 44335 115.442 101.375 0.0121495 0.0086574 2024-03-11T21:03:26.470 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:03:26.502 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:03:26.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:26.506 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:03:26.606 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:26.615 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:26 smithi146 bash[23809]: audit 2024-03-11T21:03:25.443618+0000 mgr.y (mgr.24380) 468 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:03:26.635 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:26.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:26.637 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:03:26.661 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:26 smithi023 bash[29146]: audit 2024-03-11T21:03:25.443618+0000 mgr.y (mgr.24380) 468 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:03:26.662 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:26 smithi023 bash[21372]: audit 2024-03-11T21:03:25.443618+0000 mgr.y (mgr.24380) 468 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:03:26.737 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:03:26.761 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:03:26.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:26.763 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:03:26.864 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:26.886 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:26.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:26.889 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:03:26.962 INFO:teuthology.orchestra.run.smithi023.stdout:osd.0: {} 2024-03-11T21:03:26.990 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:03:27.005 INFO:teuthology.orchestra.run.smithi023.stdout:osd.1: {} 2024-03-11T21:03:27.019 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:03:27.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:27.021 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:03:27.041 INFO:teuthology.orchestra.run.smithi023.stdout:osd.2: {} 2024-03-11T21:03:27.076 INFO:teuthology.orchestra.run.smithi023.stdout:osd.3: {} 2024-03-11T21:03:27.122 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:27.127 INFO:teuthology.orchestra.run.smithi023.stdout:osd.4: {} 2024-03-11T21:03:27.146 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:27.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:27.149 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:03:27.172 INFO:teuthology.orchestra.run.smithi023.stdout:osd.5: {} 2024-03-11T21:03:27.212 INFO:teuthology.orchestra.run.smithi023.stdout:osd.6: {} 2024-03-11T21:03:27.249 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:27.254 INFO:teuthology.orchestra.run.smithi023.stdout:osd.7: {} 2024-03-11T21:03:27.264 INFO:teuthology.orchestra.run.smithi023.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:03:27.264 INFO:teuthology.orchestra.run.smithi023.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:03:27.264 INFO:teuthology.orchestra.run.smithi023.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:03:27.264 INFO:teuthology.orchestra.run.smithi023.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:03:27.264 INFO:teuthology.orchestra.run.smithi023.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:03:27.264 INFO:teuthology.orchestra.run.smithi023.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:03:27.264 INFO:teuthology.orchestra.run.smithi023.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:03:27.264 INFO:teuthology.orchestra.run.smithi023.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:03:27.278 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:27.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:27.286 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:03:27.387 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:27.411 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:27.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:27.414 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:03:27.455 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 25 16 45683 45667 114.154 83.25 0.0192002 0.00875424 2024-03-11T21:03:27.496 INFO:teuthology.orchestra.run.smithi023.stderr:nodeep-scrub is unset 2024-03-11T21:03:27.514 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:03:27.531 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:03:27.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:27.533 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:03:27.634 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:03:27.653 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:03:27.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:27.655 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:03:27.756 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:03:27.779 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:03:27.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:27.781 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:03:27.882 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:27.910 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:27.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:27.912 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:03:28.013 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:03:28.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:27 smithi023 bash[21372]: cluster 2024-03-11T21:03:26.154171+0000 mgr.y (mgr.24380) 469 : cluster 0 pgmap v519: 148 pgs: 148 active+clean; 2.6 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 1.0 KiB/s rd, 102 MiB/s wr, 1.62k op/s 2024-03-11T21:03:28.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:27 smithi023 bash[21372]: cluster 2024-03-11T21:03:26.405072+0000 osd.4 (osd.4) 55 : cluster 0 3.14 scrub starts 2024-03-11T21:03:28.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:27 smithi023 bash[21372]: cluster 2024-03-11T21:03:26.408824+0000 osd.4 (osd.4) 56 : cluster 0 3.14 scrub ok 2024-03-11T21:03:28.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:27 smithi023 bash[21372]: cluster 2024-03-11T21:03:26.410379+0000 osd.1 (osd.1) 37 : cluster 0 2.9 scrub starts 2024-03-11T21:03:28.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:27 smithi023 bash[21372]: cluster 2024-03-11T21:03:26.422636+0000 osd.1 (osd.1) 38 : cluster 0 2.9 scrub ok 2024-03-11T21:03:28.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:27 smithi023 bash[21372]: cluster 2024-03-11T21:03:26.507033+0000 osd.6 (osd.6) 192 : cluster 0 2.7 scrub starts 2024-03-11T21:03:28.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:27 smithi023 bash[21372]: cluster 2024-03-11T21:03:26.508211+0000 osd.6 (osd.6) 193 : cluster 0 2.7 scrub ok 2024-03-11T21:03:28.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:27 smithi023 bash[21372]: audit 2024-03-11T21:03:26.912514+0000 mon.c (mon.2) 60 : audit 0 from='client.? 172.21.15.23:0/764344356' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:03:28.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:27 smithi023 bash[21372]: audit 2024-03-11T21:03:27.102203+0000 mon.c (mon.2) 61 : audit 1 from='client.? 172.21.15.23:0/3231230168' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:03:28.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:27 smithi023 bash[21372]: audit 2024-03-11T21:03:27.102786+0000 mon.a (mon.0) 1463 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:03:28.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:27 smithi023 bash[29146]: cluster 2024-03-11T21:03:26.154171+0000 mgr.y (mgr.24380) 469 : cluster 0 pgmap v519: 148 pgs: 148 active+clean; 2.6 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 1.0 KiB/s rd, 102 MiB/s wr, 1.62k op/s 2024-03-11T21:03:28.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:27 smithi023 bash[29146]: cluster 2024-03-11T21:03:26.405072+0000 osd.4 (osd.4) 55 : cluster 0 3.14 scrub starts 2024-03-11T21:03:28.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:27 smithi023 bash[29146]: cluster 2024-03-11T21:03:26.408824+0000 osd.4 (osd.4) 56 : cluster 0 3.14 scrub ok 2024-03-11T21:03:28.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:27 smithi023 bash[29146]: cluster 2024-03-11T21:03:26.410379+0000 osd.1 (osd.1) 37 : cluster 0 2.9 scrub starts 2024-03-11T21:03:28.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:27 smithi023 bash[29146]: cluster 2024-03-11T21:03:26.422636+0000 osd.1 (osd.1) 38 : cluster 0 2.9 scrub ok 2024-03-11T21:03:28.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:27 smithi023 bash[29146]: cluster 2024-03-11T21:03:26.507033+0000 osd.6 (osd.6) 192 : cluster 0 2.7 scrub starts 2024-03-11T21:03:28.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:27 smithi023 bash[29146]: cluster 2024-03-11T21:03:26.508211+0000 osd.6 (osd.6) 193 : cluster 0 2.7 scrub ok 2024-03-11T21:03:28.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:27 smithi023 bash[29146]: audit 2024-03-11T21:03:26.912514+0000 mon.c (mon.2) 60 : audit 0 from='client.? 172.21.15.23:0/764344356' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:03:28.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:27 smithi023 bash[29146]: audit 2024-03-11T21:03:27.102203+0000 mon.c (mon.2) 61 : audit 1 from='client.? 172.21.15.23:0/3231230168' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:03:28.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:27 smithi023 bash[29146]: audit 2024-03-11T21:03:27.102786+0000 mon.a (mon.0) 1463 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:03:28.039 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:03:28.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:28.042 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:03:28.085 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:27 smithi146 bash[23809]: cluster 2024-03-11T21:03:26.154171+0000 mgr.y (mgr.24380) 469 : cluster 0 pgmap v519: 148 pgs: 148 active+clean; 2.6 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 1.0 KiB/s rd, 102 MiB/s wr, 1.62k op/s 2024-03-11T21:03:28.085 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:27 smithi146 bash[23809]: cluster 2024-03-11T21:03:26.405072+0000 osd.4 (osd.4) 55 : cluster 0 3.14 scrub starts 2024-03-11T21:03:28.085 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:27 smithi146 bash[23809]: cluster 2024-03-11T21:03:26.408824+0000 osd.4 (osd.4) 56 : cluster 0 3.14 scrub ok 2024-03-11T21:03:28.085 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:27 smithi146 bash[23809]: cluster 2024-03-11T21:03:26.410379+0000 osd.1 (osd.1) 37 : cluster 0 2.9 scrub starts 2024-03-11T21:03:28.085 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:27 smithi146 bash[23809]: cluster 2024-03-11T21:03:26.422636+0000 osd.1 (osd.1) 38 : cluster 0 2.9 scrub ok 2024-03-11T21:03:28.085 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:27 smithi146 bash[23809]: cluster 2024-03-11T21:03:26.507033+0000 osd.6 (osd.6) 192 : cluster 0 2.7 scrub starts 2024-03-11T21:03:28.085 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:27 smithi146 bash[23809]: cluster 2024-03-11T21:03:26.508211+0000 osd.6 (osd.6) 193 : cluster 0 2.7 scrub ok 2024-03-11T21:03:28.085 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:27 smithi146 bash[23809]: audit 2024-03-11T21:03:26.912514+0000 mon.c (mon.2) 60 : audit 0 from='client.? 172.21.15.23:0/764344356' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:03:28.085 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:27 smithi146 bash[23809]: audit 2024-03-11T21:03:27.102203+0000 mon.c (mon.2) 61 : audit 1 from='client.? 172.21.15.23:0/3231230168' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:03:28.085 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:27 smithi146 bash[23809]: audit 2024-03-11T21:03:27.102786+0000 mon.a (mon.0) 1463 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:03:28.142 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:03:28.174 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:03:28.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:28.177 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:03:28.278 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:03:28.303 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:03:28.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:28.307 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:03:28.408 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:28.437 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:28.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:28.439 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:03:28.455 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 26 16 46333 46317 111.326 40.625 0.00941531 0.00897653 2024-03-11T21:03:28.540 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:03:28.575 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:03:28.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:28.578 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:03:28.679 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:03:28.704 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:03:28.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:28.708 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:03:28.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:28 smithi146 bash[23809]: cluster 2024-03-11T21:03:27.403239+0000 osd.1 (osd.1) 39 : cluster 0 3.19 scrub starts 2024-03-11T21:03:28.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:28 smithi146 bash[23809]: cluster 2024-03-11T21:03:27.406566+0000 osd.1 (osd.1) 40 : cluster 0 3.19 scrub ok 2024-03-11T21:03:28.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:28 smithi146 bash[23809]: cluster 2024-03-11T21:03:27.474500+0000 mon.a (mon.0) 1464 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:03:28.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:28 smithi146 bash[23809]: cluster 2024-03-11T21:03:27.474543+0000 mon.a (mon.0) 1465 : cluster 1 Cluster is now healthy 2024-03-11T21:03:28.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:28 smithi146 bash[23809]: audit 2024-03-11T21:03:27.480674+0000 mon.a (mon.0) 1466 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:03:28.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:28 smithi146 bash[23809]: cluster 2024-03-11T21:03:27.490593+0000 mon.a (mon.0) 1467 : cluster 0 osdmap e262: 8 total, 8 up, 8 in 2024-03-11T21:03:28.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:28 smithi146 bash[23809]: cluster 2024-03-11T21:03:28.094471+0000 osd.7 (osd.7) 265 : cluster 0 3.11 deep-scrub starts 2024-03-11T21:03:28.764 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:28 smithi146 bash[23809]: cluster 2024-03-11T21:03:28.128707+0000 osd.7 (osd.7) 266 : cluster 0 3.11 deep-scrub ok 2024-03-11T21:03:28.784 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:03:28 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:03:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:03:28.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:28 smithi023 bash[21372]: cluster 2024-03-11T21:03:27.403239+0000 osd.1 (osd.1) 39 : cluster 0 3.19 scrub starts 2024-03-11T21:03:28.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:28 smithi023 bash[21372]: cluster 2024-03-11T21:03:27.406566+0000 osd.1 (osd.1) 40 : cluster 0 3.19 scrub ok 2024-03-11T21:03:28.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:28 smithi023 bash[21372]: cluster 2024-03-11T21:03:27.474500+0000 mon.a (mon.0) 1464 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:03:28.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:28 smithi023 bash[21372]: cluster 2024-03-11T21:03:27.474543+0000 mon.a (mon.0) 1465 : cluster 1 Cluster is now healthy 2024-03-11T21:03:28.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:28 smithi023 bash[21372]: audit 2024-03-11T21:03:27.480674+0000 mon.a (mon.0) 1466 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:03:28.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:28 smithi023 bash[21372]: cluster 2024-03-11T21:03:27.490593+0000 mon.a (mon.0) 1467 : cluster 0 osdmap e262: 8 total, 8 up, 8 in 2024-03-11T21:03:28.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:28 smithi023 bash[21372]: cluster 2024-03-11T21:03:28.094471+0000 osd.7 (osd.7) 265 : cluster 0 3.11 deep-scrub starts 2024-03-11T21:03:28.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:28 smithi023 bash[21372]: cluster 2024-03-11T21:03:28.128707+0000 osd.7 (osd.7) 266 : cluster 0 3.11 deep-scrub ok 2024-03-11T21:03:28.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:28 smithi023 bash[29146]: cluster 2024-03-11T21:03:27.403239+0000 osd.1 (osd.1) 39 : cluster 0 3.19 scrub starts 2024-03-11T21:03:28.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:28 smithi023 bash[29146]: cluster 2024-03-11T21:03:27.406566+0000 osd.1 (osd.1) 40 : cluster 0 3.19 scrub ok 2024-03-11T21:03:28.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:28 smithi023 bash[29146]: cluster 2024-03-11T21:03:27.474500+0000 mon.a (mon.0) 1464 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:03:28.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:28 smithi023 bash[29146]: cluster 2024-03-11T21:03:27.474543+0000 mon.a (mon.0) 1465 : cluster 1 Cluster is now healthy 2024-03-11T21:03:28.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:28 smithi023 bash[29146]: audit 2024-03-11T21:03:27.480674+0000 mon.a (mon.0) 1466 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:03:28.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:28 smithi023 bash[29146]: cluster 2024-03-11T21:03:27.490593+0000 mon.a (mon.0) 1467 : cluster 0 osdmap e262: 8 total, 8 up, 8 in 2024-03-11T21:03:28.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:28 smithi023 bash[29146]: cluster 2024-03-11T21:03:28.094471+0000 osd.7 (osd.7) 265 : cluster 0 3.11 deep-scrub starts 2024-03-11T21:03:28.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:28 smithi023 bash[29146]: cluster 2024-03-11T21:03:28.128707+0000 osd.7 (osd.7) 266 : cluster 0 3.11 deep-scrub ok 2024-03-11T21:03:28.809 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:03:28.836 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:03:28.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:28.840 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:03:28.940 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:28.963 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:28.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:28.965 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:03:29.065 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:03:29.099 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:03:29.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:29.103 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:03:29.203 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:03:29.228 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:03:29.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:29.229 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:03:29.330 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:03:29.354 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:03:29.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:29.358 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:03:29.455 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 27 15 47476 47461 109.85 71.5 0.00690044 0.00909811 2024-03-11T21:03:29.459 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:03:29.489 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:03:29.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:29.492 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:03:29.593 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:29.603 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:29 smithi023 bash[21372]: cluster 2024-03-11T21:03:27.584489+0000 osd.0 (osd.0) 189 : cluster 0 6.10 scrub starts 2024-03-11T21:03:29.603 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:29 smithi023 bash[21372]: cluster 2024-03-11T21:03:27.615349+0000 osd.0 (osd.0) 190 : cluster 0 6.10 scrub ok 2024-03-11T21:03:29.603 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:29 smithi023 bash[21372]: cluster 2024-03-11T21:03:28.154810+0000 mgr.y (mgr.24380) 470 : cluster 0 pgmap v521: 148 pgs: 148 active+clean; 2.6 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 83 MiB/s wr, 1.32k op/s 2024-03-11T21:03:29.603 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:29 smithi023 bash[21372]: cluster 2024-03-11T21:03:28.451440+0000 osd.1 (osd.1) 41 : cluster 0 6.1 scrub starts 2024-03-11T21:03:29.603 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:29 smithi023 bash[21372]: cluster 2024-03-11T21:03:28.495270+0000 osd.1 (osd.1) 42 : cluster 0 6.1 scrub ok 2024-03-11T21:03:29.603 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:29 smithi023 bash[21372]: cluster 2024-03-11T21:03:28.651462+0000 osd.3 (osd.3) 260 : cluster 0 6.6 scrub starts 2024-03-11T21:03:29.603 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:29 smithi023 bash[21372]: cluster 2024-03-11T21:03:28.677269+0000 osd.3 (osd.3) 261 : cluster 0 6.6 scrub ok 2024-03-11T21:03:29.604 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:29 smithi023 bash[29146]: cluster 2024-03-11T21:03:27.584489+0000 osd.0 (osd.0) 189 : cluster 0 6.10 scrub starts 2024-03-11T21:03:29.604 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:29 smithi023 bash[29146]: cluster 2024-03-11T21:03:27.615349+0000 osd.0 (osd.0) 190 : cluster 0 6.10 scrub ok 2024-03-11T21:03:29.604 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:29 smithi023 bash[29146]: cluster 2024-03-11T21:03:28.154810+0000 mgr.y (mgr.24380) 470 : cluster 0 pgmap v521: 148 pgs: 148 active+clean; 2.6 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 83 MiB/s wr, 1.32k op/s 2024-03-11T21:03:29.604 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:29 smithi023 bash[29146]: cluster 2024-03-11T21:03:28.451440+0000 osd.1 (osd.1) 41 : cluster 0 6.1 scrub starts 2024-03-11T21:03:29.604 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:29 smithi023 bash[29146]: cluster 2024-03-11T21:03:28.495270+0000 osd.1 (osd.1) 42 : cluster 0 6.1 scrub ok 2024-03-11T21:03:29.604 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:29 smithi023 bash[29146]: cluster 2024-03-11T21:03:28.651462+0000 osd.3 (osd.3) 260 : cluster 0 6.6 scrub starts 2024-03-11T21:03:29.605 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:29 smithi023 bash[29146]: cluster 2024-03-11T21:03:28.677269+0000 osd.3 (osd.3) 261 : cluster 0 6.6 scrub ok 2024-03-11T21:03:29.622 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:29.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:29.626 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:03:29.727 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:03:29.749 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:03:29.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:29.752 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:03:29.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:29 smithi146 bash[23809]: cluster 2024-03-11T21:03:27.584489+0000 osd.0 (osd.0) 189 : cluster 0 6.10 scrub starts 2024-03-11T21:03:29.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:29 smithi146 bash[23809]: cluster 2024-03-11T21:03:27.615349+0000 osd.0 (osd.0) 190 : cluster 0 6.10 scrub ok 2024-03-11T21:03:29.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:29 smithi146 bash[23809]: cluster 2024-03-11T21:03:28.154810+0000 mgr.y (mgr.24380) 470 : cluster 0 pgmap v521: 148 pgs: 148 active+clean; 2.6 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 83 MiB/s wr, 1.32k op/s 2024-03-11T21:03:29.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:29 smithi146 bash[23809]: cluster 2024-03-11T21:03:28.451440+0000 osd.1 (osd.1) 41 : cluster 0 6.1 scrub starts 2024-03-11T21:03:29.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:29 smithi146 bash[23809]: cluster 2024-03-11T21:03:28.495270+0000 osd.1 (osd.1) 42 : cluster 0 6.1 scrub ok 2024-03-11T21:03:29.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:29 smithi146 bash[23809]: cluster 2024-03-11T21:03:28.651462+0000 osd.3 (osd.3) 260 : cluster 0 6.6 scrub starts 2024-03-11T21:03:29.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:29 smithi146 bash[23809]: cluster 2024-03-11T21:03:28.677269+0000 osd.3 (osd.3) 261 : cluster 0 6.6 scrub ok 2024-03-11T21:03:29.852 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:29.877 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:29.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:29.879 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:03:29.979 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:30.002 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:30.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:30.007 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:03:30.108 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:30.137 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:30.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:30.142 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:03:30.243 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:30.271 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:30.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:30.277 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:03:30.378 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:03:30.405 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:03:30.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:30.410 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:03:30.455 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 28 16 48981 48965 109.284 94 0.0164774 0.00914506 2024-03-11T21:03:30.510 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:03:30.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:30.543 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:03:30.543 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:03:30.644 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:03:30.654 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:30 smithi146 bash[23809]: cluster 2024-03-11T21:03:29.059158+0000 osd.5 (osd.5) 229 : cluster 0 2.12 scrub starts 2024-03-11T21:03:30.654 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:30 smithi146 bash[23809]: cluster 2024-03-11T21:03:29.061401+0000 osd.5 (osd.5) 230 : cluster 0 2.12 scrub ok 2024-03-11T21:03:30.654 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:30 smithi146 bash[23809]: cluster 2024-03-11T21:03:29.452000+0000 osd.1 (osd.1) 43 : cluster 0 4.f scrub starts 2024-03-11T21:03:30.654 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:30 smithi146 bash[23809]: cluster 2024-03-11T21:03:29.453945+0000 osd.1 (osd.1) 44 : cluster 0 4.f scrub ok 2024-03-11T21:03:30.654 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:30 smithi146 bash[23809]: cluster 2024-03-11T21:03:29.510964+0000 osd.2 (osd.2) 177 : cluster 0 4.1c scrub starts 2024-03-11T21:03:30.654 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:30 smithi146 bash[23809]: cluster 2024-03-11T21:03:29.516112+0000 osd.2 (osd.2) 178 : cluster 0 4.1c scrub ok 2024-03-11T21:03:30.654 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:30 smithi146 bash[23809]: cluster 2024-03-11T21:03:30.201923+0000 mon.a (mon.0) 1468 : cluster 0 osdmap e263: 8 total, 8 up, 8 in 2024-03-11T21:03:30.667 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:03:30.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:30.671 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:03:30.771 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:30.784 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:30 smithi023 bash[21372]: cluster 2024-03-11T21:03:29.059158+0000 osd.5 (osd.5) 229 : cluster 0 2.12 scrub starts 2024-03-11T21:03:30.784 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:30 smithi023 bash[21372]: cluster 2024-03-11T21:03:29.061401+0000 osd.5 (osd.5) 230 : cluster 0 2.12 scrub ok 2024-03-11T21:03:30.784 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:30 smithi023 bash[21372]: cluster 2024-03-11T21:03:29.452000+0000 osd.1 (osd.1) 43 : cluster 0 4.f scrub starts 2024-03-11T21:03:30.784 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:30 smithi023 bash[21372]: cluster 2024-03-11T21:03:29.453945+0000 osd.1 (osd.1) 44 : cluster 0 4.f scrub ok 2024-03-11T21:03:30.784 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:30 smithi023 bash[21372]: cluster 2024-03-11T21:03:29.510964+0000 osd.2 (osd.2) 177 : cluster 0 4.1c scrub starts 2024-03-11T21:03:30.784 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:30 smithi023 bash[21372]: cluster 2024-03-11T21:03:29.516112+0000 osd.2 (osd.2) 178 : cluster 0 4.1c scrub ok 2024-03-11T21:03:30.784 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:30 smithi023 bash[21372]: cluster 2024-03-11T21:03:30.201923+0000 mon.a (mon.0) 1468 : cluster 0 osdmap e263: 8 total, 8 up, 8 in 2024-03-11T21:03:30.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:30 smithi023 bash[29146]: cluster 2024-03-11T21:03:29.059158+0000 osd.5 (osd.5) 229 : cluster 0 2.12 scrub starts 2024-03-11T21:03:30.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:30 smithi023 bash[29146]: cluster 2024-03-11T21:03:29.061401+0000 osd.5 (osd.5) 230 : cluster 0 2.12 scrub ok 2024-03-11T21:03:30.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:30 smithi023 bash[29146]: cluster 2024-03-11T21:03:29.452000+0000 osd.1 (osd.1) 43 : cluster 0 4.f scrub starts 2024-03-11T21:03:30.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:30 smithi023 bash[29146]: cluster 2024-03-11T21:03:29.453945+0000 osd.1 (osd.1) 44 : cluster 0 4.f scrub ok 2024-03-11T21:03:30.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:30 smithi023 bash[29146]: cluster 2024-03-11T21:03:29.510964+0000 osd.2 (osd.2) 177 : cluster 0 4.1c scrub starts 2024-03-11T21:03:30.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:30 smithi023 bash[29146]: cluster 2024-03-11T21:03:29.516112+0000 osd.2 (osd.2) 178 : cluster 0 4.1c scrub ok 2024-03-11T21:03:30.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:30 smithi023 bash[29146]: cluster 2024-03-11T21:03:30.201923+0000 mon.a (mon.0) 1468 : cluster 0 osdmap e263: 8 total, 8 up, 8 in 2024-03-11T21:03:30.800 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:30.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:30.802 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:03:30.903 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:30.930 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:30.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:30.933 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:03:31.034 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:03:31.058 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:03:31.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:31.062 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:03:31.163 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:03:31.189 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:03:31.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:31.193 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:03:31.294 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:31.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:31.327 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:31.328 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:03:31.428 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:31.453 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:31.455 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 29 15 50319 50304 108.401 83.6875 0.00754617 0.0092195 2024-03-11T21:03:31.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:31.457 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:03:31.557 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:03:31.585 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:03:31.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:31.589 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:03:31.690 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:03:31.710 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:03:31.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:31.712 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:03:31.713 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:31 smithi023 bash[29146]: cluster 2024-03-11T21:03:30.155564+0000 mgr.y (mgr.24380) 471 : cluster 0 pgmap v522: 148 pgs: 148 active+clean; 2.9 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.0 KiB/s rd, 90 MiB/s wr, 1.45k op/s 2024-03-11T21:03:31.714 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:31 smithi023 bash[29146]: cluster 2024-03-11T21:03:30.631744+0000 osd.5 (osd.5) 231 : cluster 0 purged_snaps scrub starts 2024-03-11T21:03:31.714 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:31 smithi023 bash[29146]: cluster 2024-03-11T21:03:30.631822+0000 osd.5 (osd.5) 232 : cluster 0 purged_snaps scrub ok 2024-03-11T21:03:31.714 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:31 smithi023 bash[21372]: cluster 2024-03-11T21:03:30.155564+0000 mgr.y (mgr.24380) 471 : cluster 0 pgmap v522: 148 pgs: 148 active+clean; 2.9 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.0 KiB/s rd, 90 MiB/s wr, 1.45k op/s 2024-03-11T21:03:31.714 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:31 smithi023 bash[21372]: cluster 2024-03-11T21:03:30.631744+0000 osd.5 (osd.5) 231 : cluster 0 purged_snaps scrub starts 2024-03-11T21:03:31.714 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:31 smithi023 bash[21372]: cluster 2024-03-11T21:03:30.631822+0000 osd.5 (osd.5) 232 : cluster 0 purged_snaps scrub ok 2024-03-11T21:03:31.813 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:03:31.833 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:03:31.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:31.836 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:03:31.936 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:31.964 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:31.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:31.969 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:03:32.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:31 smithi146 bash[23809]: cluster 2024-03-11T21:03:30.155564+0000 mgr.y (mgr.24380) 471 : cluster 0 pgmap v522: 148 pgs: 148 active+clean; 2.9 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.0 KiB/s rd, 90 MiB/s wr, 1.45k op/s 2024-03-11T21:03:32.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:31 smithi146 bash[23809]: cluster 2024-03-11T21:03:30.631744+0000 osd.5 (osd.5) 231 : cluster 0 purged_snaps scrub starts 2024-03-11T21:03:32.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:31 smithi146 bash[23809]: cluster 2024-03-11T21:03:30.631822+0000 osd.5 (osd.5) 232 : cluster 0 purged_snaps scrub ok 2024-03-11T21:03:32.070 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:03:32.094 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:03:32.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:32.096 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:03:32.197 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:32.235 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:32.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:32.239 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:03:32.340 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:32.365 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:32.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:32.368 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:03:32.455 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 30 16 51371 51355 106.977 65.6875 0.0169747 0.00934218 2024-03-11T21:03:32.469 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:32.503 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:32.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:32.509 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:03:32.566 INFO:teuthology.orchestra.run.smithi023.stderr:reweighted osd.7 to 1 (10000) 2024-03-11T21:03:32.610 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:03:32.638 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:03:32.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:32.640 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:03:32.741 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:03:32.753 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:32 smithi146 bash[23809]: cluster 2024-03-11T21:03:31.346627+0000 osd.4 (osd.4) 57 : cluster 0 4.18 scrub starts 2024-03-11T21:03:32.753 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:32 smithi146 bash[23809]: cluster 2024-03-11T21:03:31.347837+0000 osd.4 (osd.4) 58 : cluster 0 4.18 scrub ok 2024-03-11T21:03:32.753 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:32 smithi146 bash[23809]: cluster 2024-03-11T21:03:31.541756+0000 mon.a (mon.0) 1469 : cluster 0 osdmap e264: 8 total, 8 up, 8 in 2024-03-11T21:03:32.753 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:32 smithi146 bash[23809]: audit 2024-03-11T21:03:32.178547+0000 mon.c (mon.2) 62 : audit 1 from='client.? 172.21.15.23:0/1016295733' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-11T21:03:32.754 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:32 smithi146 bash[23809]: audit 2024-03-11T21:03:32.178998+0000 mon.a (mon.0) 1470 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-11T21:03:32.772 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:03:32.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:32.775 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:03:32.876 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:03:32.903 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:03:32.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:32.905 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:03:33.006 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:33.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:32 smithi023 bash[21372]: cluster 2024-03-11T21:03:31.346627+0000 osd.4 (osd.4) 57 : cluster 0 4.18 scrub starts 2024-03-11T21:03:33.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:32 smithi023 bash[21372]: cluster 2024-03-11T21:03:31.347837+0000 osd.4 (osd.4) 58 : cluster 0 4.18 scrub ok 2024-03-11T21:03:33.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:32 smithi023 bash[21372]: cluster 2024-03-11T21:03:31.541756+0000 mon.a (mon.0) 1469 : cluster 0 osdmap e264: 8 total, 8 up, 8 in 2024-03-11T21:03:33.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:32 smithi023 bash[21372]: audit 2024-03-11T21:03:32.178547+0000 mon.c (mon.2) 62 : audit 1 from='client.? 172.21.15.23:0/1016295733' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-11T21:03:33.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:32 smithi023 bash[21372]: audit 2024-03-11T21:03:32.178998+0000 mon.a (mon.0) 1470 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-11T21:03:33.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:32 smithi023 bash[29146]: cluster 2024-03-11T21:03:31.346627+0000 osd.4 (osd.4) 57 : cluster 0 4.18 scrub starts 2024-03-11T21:03:33.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:32 smithi023 bash[29146]: cluster 2024-03-11T21:03:31.347837+0000 osd.4 (osd.4) 58 : cluster 0 4.18 scrub ok 2024-03-11T21:03:33.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:32 smithi023 bash[29146]: cluster 2024-03-11T21:03:31.541756+0000 mon.a (mon.0) 1469 : cluster 0 osdmap e264: 8 total, 8 up, 8 in 2024-03-11T21:03:33.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:32 smithi023 bash[29146]: audit 2024-03-11T21:03:32.178547+0000 mon.c (mon.2) 62 : audit 1 from='client.? 172.21.15.23:0/1016295733' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-11T21:03:33.036 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:32 smithi023 bash[29146]: audit 2024-03-11T21:03:32.178998+0000 mon.a (mon.0) 1470 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-11T21:03:33.038 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:33.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:33.041 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:03:33.141 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:03:33.173 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:03:33.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:33.175 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:03:33.276 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:03:33.300 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:03:33.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:33.305 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:03:33.342 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T21:03:33.406 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:03:33.429 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:03:33.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:33.431 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:03:33.455 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 31 16 52482 52466 105.766 69.4375 0.00321606 0.0094006 2024-03-11T21:03:33.461 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd set noscrub 2024-03-11T21:03:33.532 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:03:33.549 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:03:33.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:33.551 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:03:33.652 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:03:33.683 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:03:33.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:33.690 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:03:33.784 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:33 smithi023 bash[21372]: cluster 2024-03-11T21:03:32.156776+0000 mgr.y (mgr.24380) 472 : cluster 0 pgmap v525: 148 pgs: 148 active+clean; 3.0 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 852 B/s rd, 73 MiB/s wr, 1.17k op/s 2024-03-11T21:03:33.784 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:33 smithi023 bash[21372]: audit 2024-03-11T21:03:32.549985+0000 mon.a (mon.0) 1471 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-11T21:03:33.784 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:33 smithi023 bash[21372]: cluster 2024-03-11T21:03:32.555114+0000 mon.a (mon.0) 1472 : cluster 0 osdmap e265: 8 total, 8 up, 8 in 2024-03-11T21:03:33.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:33 smithi023 bash[29146]: cluster 2024-03-11T21:03:32.156776+0000 mgr.y (mgr.24380) 472 : cluster 0 pgmap v525: 148 pgs: 148 active+clean; 3.0 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 852 B/s rd, 73 MiB/s wr, 1.17k op/s 2024-03-11T21:03:33.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:33 smithi023 bash[29146]: audit 2024-03-11T21:03:32.549985+0000 mon.a (mon.0) 1471 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-11T21:03:33.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:33 smithi023 bash[29146]: cluster 2024-03-11T21:03:32.555114+0000 mon.a (mon.0) 1472 : cluster 0 osdmap e265: 8 total, 8 up, 8 in 2024-03-11T21:03:33.790 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:03:33.814 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:03:33.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:33.816 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:03:33.917 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:03:33.928 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:33 smithi146 bash[23809]: cluster 2024-03-11T21:03:32.156776+0000 mgr.y (mgr.24380) 472 : cluster 0 pgmap v525: 148 pgs: 148 active+clean; 3.0 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 852 B/s rd, 73 MiB/s wr, 1.17k op/s 2024-03-11T21:03:33.928 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:33 smithi146 bash[23809]: audit 2024-03-11T21:03:32.549985+0000 mon.a (mon.0) 1471 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-11T21:03:33.928 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:33 smithi146 bash[23809]: cluster 2024-03-11T21:03:32.555114+0000 mon.a (mon.0) 1472 : cluster 0 osdmap e265: 8 total, 8 up, 8 in 2024-03-11T21:03:33.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:33.946 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:03:33.946 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:03:33.993 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd reweight 1 1 2024-03-11T21:03:34.047 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:34.070 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:34.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:34.072 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:03:34.173 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:03:34.211 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:03:34.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:34.220 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:03:34.321 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:03:34.345 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:03:34.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:34.354 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:03:34.454 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:34.455 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 32 16 53635 53619 104.713 72.0625 0.0132901 0.0095443 2024-03-11T21:03:34.496 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:34.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:34.500 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:03:34.599 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-03-11T21:03:34.599 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-03-11T21:03:34.599 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 30, 2024-03-11T21:03:34.599 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-03-11T21:03:34.599 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-03-11T21:03:34.601 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:03:34.616 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:34 smithi146 bash[23809]: cluster 2024-03-11T21:03:32.880425+0000 osd.2 (osd.2) 179 : cluster 0 purged_snaps scrub starts 2024-03-11T21:03:34.616 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:34 smithi146 bash[23809]: cluster 2024-03-11T21:03:32.880481+0000 osd.2 (osd.2) 180 : cluster 0 purged_snaps scrub ok 2024-03-11T21:03:34.616 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:34 smithi146 bash[23809]: cluster 2024-03-11T21:03:33.255817+0000 osd.4 (osd.4) 59 : cluster 0 6.e scrub starts 2024-03-11T21:03:34.616 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:34 smithi146 bash[23809]: cluster 2024-03-11T21:03:33.263474+0000 osd.4 (osd.4) 60 : cluster 0 6.e scrub ok 2024-03-11T21:03:34.616 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:34 smithi146 bash[23809]: cluster 2024-03-11T21:03:33.584646+0000 mon.a (mon.0) 1473 : cluster 0 osdmap e266: 8 total, 8 up, 8 in 2024-03-11T21:03:34.616 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:34 smithi146 bash[23809]: cluster 2024-03-11T21:03:34.157204+0000 mgr.y (mgr.24380) 473 : cluster 0 pgmap v528: 148 pgs: 148 active+clean; 3.0 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 35 MiB/s wr, 556 op/s 2024-03-11T21:03:34.627 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:03:34.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:34.633 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:03:34.734 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:03:34.770 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:03:34.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:34.773 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:03:34.874 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:34.899 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:34.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:34.906 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:03:35.007 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:03:35.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:34 smithi023 bash[21372]: cluster 2024-03-11T21:03:32.880425+0000 osd.2 (osd.2) 179 : cluster 0 purged_snaps scrub starts 2024-03-11T21:03:35.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:34 smithi023 bash[21372]: cluster 2024-03-11T21:03:32.880481+0000 osd.2 (osd.2) 180 : cluster 0 purged_snaps scrub ok 2024-03-11T21:03:35.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:34 smithi023 bash[21372]: cluster 2024-03-11T21:03:33.255817+0000 osd.4 (osd.4) 59 : cluster 0 6.e scrub starts 2024-03-11T21:03:35.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:34 smithi023 bash[21372]: cluster 2024-03-11T21:03:33.263474+0000 osd.4 (osd.4) 60 : cluster 0 6.e scrub ok 2024-03-11T21:03:35.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:34 smithi023 bash[21372]: cluster 2024-03-11T21:03:33.584646+0000 mon.a (mon.0) 1473 : cluster 0 osdmap e266: 8 total, 8 up, 8 in 2024-03-11T21:03:35.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:34 smithi023 bash[21372]: cluster 2024-03-11T21:03:34.157204+0000 mgr.y (mgr.24380) 473 : cluster 0 pgmap v528: 148 pgs: 148 active+clean; 3.0 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 35 MiB/s wr, 556 op/s 2024-03-11T21:03:35.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:34 smithi023 bash[29146]: cluster 2024-03-11T21:03:32.880425+0000 osd.2 (osd.2) 179 : cluster 0 purged_snaps scrub starts 2024-03-11T21:03:35.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:34 smithi023 bash[29146]: cluster 2024-03-11T21:03:32.880481+0000 osd.2 (osd.2) 180 : cluster 0 purged_snaps scrub ok 2024-03-11T21:03:35.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:34 smithi023 bash[29146]: cluster 2024-03-11T21:03:33.255817+0000 osd.4 (osd.4) 59 : cluster 0 6.e scrub starts 2024-03-11T21:03:35.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:34 smithi023 bash[29146]: cluster 2024-03-11T21:03:33.263474+0000 osd.4 (osd.4) 60 : cluster 0 6.e scrub ok 2024-03-11T21:03:35.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:34 smithi023 bash[29146]: cluster 2024-03-11T21:03:33.584646+0000 mon.a (mon.0) 1473 : cluster 0 osdmap e266: 8 total, 8 up, 8 in 2024-03-11T21:03:35.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:34 smithi023 bash[29146]: cluster 2024-03-11T21:03:34.157204+0000 mgr.y (mgr.24380) 473 : cluster 0 pgmap v528: 148 pgs: 148 active+clean; 3.0 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 35 MiB/s wr, 556 op/s 2024-03-11T21:03:35.035 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:03:35.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:35.042 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:03:35.143 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:03:35.170 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:03:35.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:35.177 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:03:35.278 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:35.301 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:35.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:35.303 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:03:35.404 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:03:35.436 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:03:35.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:35.438 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:03:35.455 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 33 16 55002 54986 104.129 85.4375 0.0159732 0.0095979 2024-03-11T21:03:35.539 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:03:35.572 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:03:35.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:35.576 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:03:35.595 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:03:35 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:03:35.677 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:35.688 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:35 smithi023 bash[21372]: cluster 2024-03-11T21:03:33.833513+0000 osd.5 (osd.5) 233 : cluster 0 2.16 scrub starts 2024-03-11T21:03:35.688 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:35 smithi023 bash[21372]: cluster 2024-03-11T21:03:33.835834+0000 osd.5 (osd.5) 234 : cluster 0 2.16 scrub ok 2024-03-11T21:03:35.688 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:35 smithi023 bash[21372]: cluster 2024-03-11T21:03:35.214846+0000 mon.a (mon.0) 1474 : cluster 0 osdmap e267: 8 total, 8 up, 8 in 2024-03-11T21:03:35.688 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:35 smithi023 bash[21372]: audit 2024-03-11T21:03:35.224845+0000 mon.a (mon.0) 1475 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:03:35.688 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:35 smithi023 bash[21372]: audit 2024-03-11T21:03:35.227651+0000 mon.a (mon.0) 1476 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:03:35.688 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:35 smithi023 bash[21372]: audit 2024-03-11T21:03:35.452831+0000 mgr.y (mgr.24380) 474 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:03:35.689 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:35 smithi023 bash[29146]: cluster 2024-03-11T21:03:33.833513+0000 osd.5 (osd.5) 233 : cluster 0 2.16 scrub starts 2024-03-11T21:03:35.689 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:35 smithi023 bash[29146]: cluster 2024-03-11T21:03:33.835834+0000 osd.5 (osd.5) 234 : cluster 0 2.16 scrub ok 2024-03-11T21:03:35.689 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:35 smithi023 bash[29146]: cluster 2024-03-11T21:03:35.214846+0000 mon.a (mon.0) 1474 : cluster 0 osdmap e267: 8 total, 8 up, 8 in 2024-03-11T21:03:35.689 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:35 smithi023 bash[29146]: audit 2024-03-11T21:03:35.224845+0000 mon.a (mon.0) 1475 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:03:35.689 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:35 smithi023 bash[29146]: audit 2024-03-11T21:03:35.227651+0000 mon.a (mon.0) 1476 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:03:35.689 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:35 smithi023 bash[29146]: audit 2024-03-11T21:03:35.452831+0000 mgr.y (mgr.24380) 474 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:03:35.707 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:35.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:35.710 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:03:35.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:35 smithi146 bash[23809]: cluster 2024-03-11T21:03:33.833513+0000 osd.5 (osd.5) 233 : cluster 0 2.16 scrub starts 2024-03-11T21:03:35.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:35 smithi146 bash[23809]: cluster 2024-03-11T21:03:33.835834+0000 osd.5 (osd.5) 234 : cluster 0 2.16 scrub ok 2024-03-11T21:03:35.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:35 smithi146 bash[23809]: cluster 2024-03-11T21:03:35.214846+0000 mon.a (mon.0) 1474 : cluster 0 osdmap e267: 8 total, 8 up, 8 in 2024-03-11T21:03:35.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:35 smithi146 bash[23809]: audit 2024-03-11T21:03:35.224845+0000 mon.a (mon.0) 1475 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:03:35.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:35 smithi146 bash[23809]: audit 2024-03-11T21:03:35.227651+0000 mon.a (mon.0) 1476 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:03:35.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:35 smithi146 bash[23809]: audit 2024-03-11T21:03:35.452831+0000 mgr.y (mgr.24380) 474 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:03:35.811 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:35.842 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:35.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:35.844 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:03:35.945 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:03:35.978 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:03:35.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:35.982 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:03:36.082 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:36.107 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:36.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:36.109 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:03:36.210 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:03:36.230 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-03-11T21:03:36.243 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:03:36.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:36.247 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:03:36.347 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:03:36.386 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:03:36.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:36.389 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:03:36.455 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 34 16 56335 56319 103.516 83.3125 0.0232881 0.00965496 2024-03-11T21:03:36.490 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:03:36.514 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:03:36.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:36.516 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:03:36.617 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:03:36.654 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:03:36.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:36.661 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:03:36.762 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:03:36.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:36 smithi146 bash[23809]: cluster 2024-03-11T21:03:36.157807+0000 mgr.y (mgr.24380) 475 : cluster 0 pgmap v530: 148 pgs: 148 active+clean; 3.3 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1.1 KiB/s rd, 82 MiB/s wr, 1.31k op/s 2024-03-11T21:03:36.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:36 smithi146 bash[23809]: cluster 2024-03-11T21:03:36.226770+0000 osd.4 (osd.4) 61 : cluster 0 4.1b deep-scrub starts 2024-03-11T21:03:36.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:36 smithi146 bash[23809]: cluster 2024-03-11T21:03:36.230638+0000 osd.4 (osd.4) 62 : cluster 0 4.1b deep-scrub ok 2024-03-11T21:03:36.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:36 smithi023 bash[21372]: cluster 2024-03-11T21:03:36.157807+0000 mgr.y (mgr.24380) 475 : cluster 0 pgmap v530: 148 pgs: 148 active+clean; 3.3 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1.1 KiB/s rd, 82 MiB/s wr, 1.31k op/s 2024-03-11T21:03:36.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:36 smithi023 bash[21372]: cluster 2024-03-11T21:03:36.226770+0000 osd.4 (osd.4) 61 : cluster 0 4.1b deep-scrub starts 2024-03-11T21:03:36.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:36 smithi023 bash[21372]: cluster 2024-03-11T21:03:36.230638+0000 osd.4 (osd.4) 62 : cluster 0 4.1b deep-scrub ok 2024-03-11T21:03:36.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:36 smithi023 bash[29146]: cluster 2024-03-11T21:03:36.157807+0000 mgr.y (mgr.24380) 475 : cluster 0 pgmap v530: 148 pgs: 148 active+clean; 3.3 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1.1 KiB/s rd, 82 MiB/s wr, 1.31k op/s 2024-03-11T21:03:36.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:36 smithi023 bash[29146]: cluster 2024-03-11T21:03:36.226770+0000 osd.4 (osd.4) 61 : cluster 0 4.1b deep-scrub starts 2024-03-11T21:03:36.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:36 smithi023 bash[29146]: cluster 2024-03-11T21:03:36.230638+0000 osd.4 (osd.4) 62 : cluster 0 4.1b deep-scrub ok 2024-03-11T21:03:36.794 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:03:36.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:36.797 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:03:36.898 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:03:36.925 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:03:36.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:36.930 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:03:37.030 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:03:37.062 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:03:37.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:37.069 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:03:37.170 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:03:37.210 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:03:37.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:37.214 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:03:37.315 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:03:37.345 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:03:37.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:37.350 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:03:37.450 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:03:37.455 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 35 16 57724 57708 103.039 86.8125 0.00469947 0.00970023 2024-03-11T21:03:37.479 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:03:37.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:37.481 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:03:37.582 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:37.612 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:37.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:37.615 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:03:37.716 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:03:37.746 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:03:37.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:37.751 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:03:37.852 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:03:37.863 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:37 smithi146 bash[23809]: cluster 2024-03-11T21:03:36.407174+0000 osd.6 (osd.6) 194 : cluster 0 2.14 scrub starts 2024-03-11T21:03:37.863 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:37 smithi146 bash[23809]: cluster 2024-03-11T21:03:36.408935+0000 osd.6 (osd.6) 195 : cluster 0 2.14 scrub ok 2024-03-11T21:03:37.863 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:37 smithi146 bash[23809]: cluster 2024-03-11T21:03:36.978874+0000 osd.7 (osd.7) 267 : cluster 0 2.1d scrub starts 2024-03-11T21:03:37.863 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:37 smithi146 bash[23809]: cluster 2024-03-11T21:03:36.980822+0000 osd.7 (osd.7) 268 : cluster 0 2.1d scrub ok 2024-03-11T21:03:37.880 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:03:37.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:37.883 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:03:37.984 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:03:37.993 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:37 smithi023 bash[21372]: cluster 2024-03-11T21:03:36.407174+0000 osd.6 (osd.6) 194 : cluster 0 2.14 scrub starts 2024-03-11T21:03:37.993 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:37 smithi023 bash[21372]: cluster 2024-03-11T21:03:36.408935+0000 osd.6 (osd.6) 195 : cluster 0 2.14 scrub ok 2024-03-11T21:03:37.994 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:37 smithi023 bash[21372]: cluster 2024-03-11T21:03:36.978874+0000 osd.7 (osd.7) 267 : cluster 0 2.1d scrub starts 2024-03-11T21:03:37.994 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:37 smithi023 bash[21372]: cluster 2024-03-11T21:03:36.980822+0000 osd.7 (osd.7) 268 : cluster 0 2.1d scrub ok 2024-03-11T21:03:37.994 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:37 smithi023 bash[29146]: cluster 2024-03-11T21:03:36.407174+0000 osd.6 (osd.6) 194 : cluster 0 2.14 scrub starts 2024-03-11T21:03:37.994 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:37 smithi023 bash[29146]: cluster 2024-03-11T21:03:36.408935+0000 osd.6 (osd.6) 195 : cluster 0 2.14 scrub ok 2024-03-11T21:03:37.994 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:37 smithi023 bash[29146]: cluster 2024-03-11T21:03:36.978874+0000 osd.7 (osd.7) 267 : cluster 0 2.1d scrub starts 2024-03-11T21:03:37.994 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:37 smithi023 bash[29146]: cluster 2024-03-11T21:03:36.980822+0000 osd.7 (osd.7) 268 : cluster 0 2.1d scrub ok 2024-03-11T21:03:38.009 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:03:38.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:38.012 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:03:38.105 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:03:38.106 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:03:38.106 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:03:38.114 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:03:38.139 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:03:38.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:38.143 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:03:38.244 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:03:38.277 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:03:38.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:38.283 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:03:38.383 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:03:38.392 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:03:38 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:03:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:03:38.410 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:03:38.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:38.412 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:03:38.456 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 36 16 59069 59053 102.511 84.0625 0.0127547 0.00974991 2024-03-11T21:03:38.513 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:38.543 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:38.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:38.546 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:03:38.647 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:38.675 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:38.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:38.678 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:03:38.778 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:03:38.815 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:03:38.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:38.816 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:03:38.916 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:03:38.943 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:03:38.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:38.947 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:03:38.988 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:38 smithi023 bash[21372]: cluster 2024-03-11T21:03:37.590096+0000 osd.1 (osd.1) 45 : cluster 0 2.a deep-scrub starts 2024-03-11T21:03:38.988 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:38 smithi023 bash[21372]: cluster 2024-03-11T21:03:37.601221+0000 osd.1 (osd.1) 46 : cluster 0 2.a deep-scrub ok 2024-03-11T21:03:38.989 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:38 smithi023 bash[21372]: cluster 2024-03-11T21:03:38.158335+0000 mgr.y (mgr.24380) 476 : cluster 0 pgmap v531: 148 pgs: 148 active+clean; 3.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 853 B/s rd, 64 MiB/s wr, 1.02k op/s 2024-03-11T21:03:38.989 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:38 smithi023 bash[29146]: cluster 2024-03-11T21:03:37.590096+0000 osd.1 (osd.1) 45 : cluster 0 2.a deep-scrub starts 2024-03-11T21:03:38.989 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:38 smithi023 bash[29146]: cluster 2024-03-11T21:03:37.601221+0000 osd.1 (osd.1) 46 : cluster 0 2.a deep-scrub ok 2024-03-11T21:03:38.989 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:38 smithi023 bash[29146]: cluster 2024-03-11T21:03:38.158335+0000 mgr.y (mgr.24380) 476 : cluster 0 pgmap v531: 148 pgs: 148 active+clean; 3.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 853 B/s rd, 64 MiB/s wr, 1.02k op/s 2024-03-11T21:03:39.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:38 smithi146 bash[23809]: cluster 2024-03-11T21:03:37.590096+0000 osd.1 (osd.1) 45 : cluster 0 2.a deep-scrub starts 2024-03-11T21:03:39.013 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:38 smithi146 bash[23809]: cluster 2024-03-11T21:03:37.601221+0000 osd.1 (osd.1) 46 : cluster 0 2.a deep-scrub ok 2024-03-11T21:03:39.013 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:38 smithi146 bash[23809]: cluster 2024-03-11T21:03:38.158335+0000 mgr.y (mgr.24380) 476 : cluster 0 pgmap v531: 148 pgs: 148 active+clean; 3.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 853 B/s rd, 64 MiB/s wr, 1.02k op/s 2024-03-11T21:03:39.048 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:03:39.084 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:03:39.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:39.087 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:03:39.188 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:03:39.226 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:03:39.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:39.229 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:03:39.329 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:39.355 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:39.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:39.359 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:03:39.456 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 37 16 60360 60344 101.921 80.6875 0.00930172 0.00980676 2024-03-11T21:03:39.460 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:03:39.486 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:03:39.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:39.488 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:03:39.589 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:39.619 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:39.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:39.624 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:03:39.725 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:39.754 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:39.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:39.759 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:03:39.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:39 smithi146 bash[23809]: cluster 2024-03-11T21:03:38.253062+0000 osd.4 (osd.4) 63 : cluster 0 4.9 deep-scrub starts 2024-03-11T21:03:39.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:39 smithi146 bash[23809]: cluster 2024-03-11T21:03:38.258838+0000 osd.4 (osd.4) 64 : cluster 0 4.9 deep-scrub ok 2024-03-11T21:03:39.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:39 smithi146 bash[23809]: cluster 2024-03-11T21:03:38.967186+0000 osd.7 (osd.7) 269 : cluster 0 2.1f scrub starts 2024-03-11T21:03:39.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:39 smithi146 bash[23809]: cluster 2024-03-11T21:03:38.968073+0000 osd.7 (osd.7) 270 : cluster 0 2.1f scrub ok 2024-03-11T21:03:39.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:39 smithi146 bash[23809]: cluster 2024-03-11T21:03:39.232105+0000 osd.4 (osd.4) 65 : cluster 0 3.1a deep-scrub starts 2024-03-11T21:03:39.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:39 smithi146 bash[23809]: cluster 2024-03-11T21:03:39.240081+0000 osd.4 (osd.4) 66 : cluster 0 3.1a deep-scrub ok 2024-03-11T21:03:39.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:39 smithi146 bash[23809]: cluster 2024-03-11T21:03:39.411510+0000 osd.0 (osd.0) 191 : cluster 0 4.b scrub starts 2024-03-11T21:03:39.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:39 smithi146 bash[23809]: cluster 2024-03-11T21:03:39.412988+0000 osd.0 (osd.0) 192 : cluster 0 4.b scrub ok 2024-03-11T21:03:39.859 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:39.884 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:39.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:39.885 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:03:39.986 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:03:40.005 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:03:40.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:40.007 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:03:40.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:39 smithi023 bash[21372]: cluster 2024-03-11T21:03:38.253062+0000 osd.4 (osd.4) 63 : cluster 0 4.9 deep-scrub starts 2024-03-11T21:03:40.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:39 smithi023 bash[21372]: cluster 2024-03-11T21:03:38.258838+0000 osd.4 (osd.4) 64 : cluster 0 4.9 deep-scrub ok 2024-03-11T21:03:40.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:39 smithi023 bash[21372]: cluster 2024-03-11T21:03:38.967186+0000 osd.7 (osd.7) 269 : cluster 0 2.1f scrub starts 2024-03-11T21:03:40.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:39 smithi023 bash[21372]: cluster 2024-03-11T21:03:38.968073+0000 osd.7 (osd.7) 270 : cluster 0 2.1f scrub ok 2024-03-11T21:03:40.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:39 smithi023 bash[21372]: cluster 2024-03-11T21:03:39.232105+0000 osd.4 (osd.4) 65 : cluster 0 3.1a deep-scrub starts 2024-03-11T21:03:40.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:39 smithi023 bash[21372]: cluster 2024-03-11T21:03:39.240081+0000 osd.4 (osd.4) 66 : cluster 0 3.1a deep-scrub ok 2024-03-11T21:03:40.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:39 smithi023 bash[21372]: cluster 2024-03-11T21:03:39.411510+0000 osd.0 (osd.0) 191 : cluster 0 4.b scrub starts 2024-03-11T21:03:40.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:39 smithi023 bash[21372]: cluster 2024-03-11T21:03:39.412988+0000 osd.0 (osd.0) 192 : cluster 0 4.b scrub ok 2024-03-11T21:03:40.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:39 smithi023 bash[29146]: cluster 2024-03-11T21:03:38.253062+0000 osd.4 (osd.4) 63 : cluster 0 4.9 deep-scrub starts 2024-03-11T21:03:40.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:39 smithi023 bash[29146]: cluster 2024-03-11T21:03:38.258838+0000 osd.4 (osd.4) 64 : cluster 0 4.9 deep-scrub ok 2024-03-11T21:03:40.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:39 smithi023 bash[29146]: cluster 2024-03-11T21:03:38.967186+0000 osd.7 (osd.7) 269 : cluster 0 2.1f scrub starts 2024-03-11T21:03:40.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:39 smithi023 bash[29146]: cluster 2024-03-11T21:03:38.968073+0000 osd.7 (osd.7) 270 : cluster 0 2.1f scrub ok 2024-03-11T21:03:40.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:39 smithi023 bash[29146]: cluster 2024-03-11T21:03:39.232105+0000 osd.4 (osd.4) 65 : cluster 0 3.1a deep-scrub starts 2024-03-11T21:03:40.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:39 smithi023 bash[29146]: cluster 2024-03-11T21:03:39.240081+0000 osd.4 (osd.4) 66 : cluster 0 3.1a deep-scrub ok 2024-03-11T21:03:40.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:39 smithi023 bash[29146]: cluster 2024-03-11T21:03:39.411510+0000 osd.0 (osd.0) 191 : cluster 0 4.b scrub starts 2024-03-11T21:03:40.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:39 smithi023 bash[29146]: cluster 2024-03-11T21:03:39.412988+0000 osd.0 (osd.0) 192 : cluster 0 4.b scrub ok 2024-03-11T21:03:40.108 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:03:40.138 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:03:40.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:40.142 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:03:40.242 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:40.264 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:40.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:40.266 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:03:40.366 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:40.393 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:40.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:40.395 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:03:40.456 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 38 16 61110 61094 100.473 46.875 0.0136067 0.00994627 2024-03-11T21:03:40.497 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:40.523 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:40.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:40.527 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:03:40.627 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:40.649 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:40.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:40.655 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:03:40.756 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:40.786 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:40.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:40.789 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:03:40.890 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:03:40.916 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:03:40.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:40 smithi023 bash[21372]: cluster 2024-03-11T21:03:39.525850+0000 osd.6 (osd.6) 196 : cluster 0 purged_snaps scrub starts 2024-03-11T21:03:40.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:40 smithi023 bash[21372]: cluster 2024-03-11T21:03:39.525938+0000 osd.6 (osd.6) 197 : cluster 0 purged_snaps scrub ok 2024-03-11T21:03:40.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:40 smithi023 bash[21372]: cluster 2024-03-11T21:03:39.657464+0000 osd.3 (osd.3) 262 : cluster 0 3.b deep-scrub starts 2024-03-11T21:03:40.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:40 smithi023 bash[21372]: cluster 2024-03-11T21:03:39.659358+0000 mon.a (mon.0) 1477 : cluster 0 osdmap e268: 8 total, 8 up, 8 in 2024-03-11T21:03:40.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:40 smithi023 bash[21372]: cluster 2024-03-11T21:03:39.673308+0000 osd.3 (osd.3) 263 : cluster 0 3.b deep-scrub ok 2024-03-11T21:03:40.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:40 smithi023 bash[21372]: cluster 2024-03-11T21:03:39.966412+0000 osd.7 (osd.7) 271 : cluster 0 6.15 scrub starts 2024-03-11T21:03:40.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:40 smithi023 bash[21372]: cluster 2024-03-11T21:03:39.998244+0000 osd.7 (osd.7) 272 : cluster 0 6.15 scrub ok 2024-03-11T21:03:40.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:40 smithi023 bash[21372]: cluster 2024-03-11T21:03:40.158925+0000 mgr.y (mgr.24380) 477 : cluster 0 pgmap v533: 148 pgs: 148 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 778 B/s rd, 97 MiB/s wr, 1.55k op/s 2024-03-11T21:03:40.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:40 smithi023 bash[21372]: cluster 2024-03-11T21:03:40.204716+0000 mon.a (mon.0) 1478 : cluster 0 osdmap e269: 8 total, 8 up, 8 in 2024-03-11T21:03:40.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:40.920 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:03:40.921 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:40 smithi023 bash[29146]: cluster 2024-03-11T21:03:39.525850+0000 osd.6 (osd.6) 196 : cluster 0 purged_snaps scrub starts 2024-03-11T21:03:40.921 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:40 smithi023 bash[29146]: cluster 2024-03-11T21:03:39.525938+0000 osd.6 (osd.6) 197 : cluster 0 purged_snaps scrub ok 2024-03-11T21:03:40.921 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:40 smithi023 bash[29146]: cluster 2024-03-11T21:03:39.657464+0000 osd.3 (osd.3) 262 : cluster 0 3.b deep-scrub starts 2024-03-11T21:03:40.921 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:40 smithi023 bash[29146]: cluster 2024-03-11T21:03:39.659358+0000 mon.a (mon.0) 1477 : cluster 0 osdmap e268: 8 total, 8 up, 8 in 2024-03-11T21:03:40.921 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:40 smithi023 bash[29146]: cluster 2024-03-11T21:03:39.673308+0000 osd.3 (osd.3) 263 : cluster 0 3.b deep-scrub ok 2024-03-11T21:03:40.921 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:40 smithi023 bash[29146]: cluster 2024-03-11T21:03:39.966412+0000 osd.7 (osd.7) 271 : cluster 0 6.15 scrub starts 2024-03-11T21:03:40.921 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:40 smithi023 bash[29146]: cluster 2024-03-11T21:03:39.998244+0000 osd.7 (osd.7) 272 : cluster 0 6.15 scrub ok 2024-03-11T21:03:40.921 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:40 smithi023 bash[29146]: cluster 2024-03-11T21:03:40.158925+0000 mgr.y (mgr.24380) 477 : cluster 0 pgmap v533: 148 pgs: 148 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 778 B/s rd, 97 MiB/s wr, 1.55k op/s 2024-03-11T21:03:40.921 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:40 smithi023 bash[29146]: cluster 2024-03-11T21:03:40.204716+0000 mon.a (mon.0) 1478 : cluster 0 osdmap e269: 8 total, 8 up, 8 in 2024-03-11T21:03:41.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:40 smithi146 bash[23809]: cluster 2024-03-11T21:03:39.525850+0000 osd.6 (osd.6) 196 : cluster 0 purged_snaps scrub starts 2024-03-11T21:03:41.013 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:40 smithi146 bash[23809]: cluster 2024-03-11T21:03:39.525938+0000 osd.6 (osd.6) 197 : cluster 0 purged_snaps scrub ok 2024-03-11T21:03:41.013 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:40 smithi146 bash[23809]: cluster 2024-03-11T21:03:39.657464+0000 osd.3 (osd.3) 262 : cluster 0 3.b deep-scrub starts 2024-03-11T21:03:41.013 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:40 smithi146 bash[23809]: cluster 2024-03-11T21:03:39.659358+0000 mon.a (mon.0) 1477 : cluster 0 osdmap e268: 8 total, 8 up, 8 in 2024-03-11T21:03:41.013 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:40 smithi146 bash[23809]: cluster 2024-03-11T21:03:39.673308+0000 osd.3 (osd.3) 263 : cluster 0 3.b deep-scrub ok 2024-03-11T21:03:41.013 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:40 smithi146 bash[23809]: cluster 2024-03-11T21:03:39.966412+0000 osd.7 (osd.7) 271 : cluster 0 6.15 scrub starts 2024-03-11T21:03:41.013 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:40 smithi146 bash[23809]: cluster 2024-03-11T21:03:39.998244+0000 osd.7 (osd.7) 272 : cluster 0 6.15 scrub ok 2024-03-11T21:03:41.013 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:40 smithi146 bash[23809]: cluster 2024-03-11T21:03:40.158925+0000 mgr.y (mgr.24380) 477 : cluster 0 pgmap v533: 148 pgs: 148 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 778 B/s rd, 97 MiB/s wr, 1.55k op/s 2024-03-11T21:03:41.013 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:40 smithi146 bash[23809]: cluster 2024-03-11T21:03:40.204716+0000 mon.a (mon.0) 1478 : cluster 0 osdmap e269: 8 total, 8 up, 8 in 2024-03-11T21:03:41.022 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:03:41.046 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:03:41.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:41.048 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:03:41.080 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:03:41.150 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:03:41.171 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:03:41.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:41.173 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:03:41.273 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:03:41.302 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:03:41.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:41.307 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:03:41.408 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:41.442 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:41.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:41.446 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:03:41.458 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 39 16 62198 62182 99.6399 68 0.01762 0.01003 2024-03-11T21:03:41.546 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:03:41.588 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:03:41.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:41.592 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:03:41.692 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:03:41.703 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:41 smithi023 bash[21372]: cluster 2024-03-11T21:03:40.410069+0000 osd.1 (osd.1) 47 : cluster 0 purged_snaps scrub starts 2024-03-11T21:03:41.703 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:41 smithi023 bash[21372]: cluster 2024-03-11T21:03:40.410220+0000 osd.1 (osd.1) 48 : cluster 0 purged_snaps scrub ok 2024-03-11T21:03:41.703 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:41 smithi023 bash[21372]: cluster 2024-03-11T21:03:40.609120+0000 osd.1 (osd.1) 49 : cluster 0 2.4 scrub starts 2024-03-11T21:03:41.703 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:41 smithi023 bash[21372]: cluster 2024-03-11T21:03:40.610802+0000 osd.1 (osd.1) 50 : cluster 0 2.4 scrub ok 2024-03-11T21:03:41.703 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:41 smithi023 bash[21372]: cluster 2024-03-11T21:03:40.669438+0000 osd.3 (osd.3) 264 : cluster 0 2.1e scrub starts 2024-03-11T21:03:41.703 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:41 smithi023 bash[21372]: cluster 2024-03-11T21:03:40.675520+0000 osd.3 (osd.3) 265 : cluster 0 2.1e scrub ok 2024-03-11T21:03:41.703 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:41 smithi023 bash[21372]: cluster 2024-03-11T21:03:41.211763+0000 mon.a (mon.0) 1479 : cluster 0 osdmap e270: 8 total, 8 up, 8 in 2024-03-11T21:03:41.704 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:41 smithi023 bash[29146]: cluster 2024-03-11T21:03:40.410069+0000 osd.1 (osd.1) 47 : cluster 0 purged_snaps scrub starts 2024-03-11T21:03:41.704 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:41 smithi023 bash[29146]: cluster 2024-03-11T21:03:40.410220+0000 osd.1 (osd.1) 48 : cluster 0 purged_snaps scrub ok 2024-03-11T21:03:41.704 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:41 smithi023 bash[29146]: cluster 2024-03-11T21:03:40.609120+0000 osd.1 (osd.1) 49 : cluster 0 2.4 scrub starts 2024-03-11T21:03:41.704 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:41 smithi023 bash[29146]: cluster 2024-03-11T21:03:40.610802+0000 osd.1 (osd.1) 50 : cluster 0 2.4 scrub ok 2024-03-11T21:03:41.704 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:41 smithi023 bash[29146]: cluster 2024-03-11T21:03:40.669438+0000 osd.3 (osd.3) 264 : cluster 0 2.1e scrub starts 2024-03-11T21:03:41.704 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:41 smithi023 bash[29146]: cluster 2024-03-11T21:03:40.675520+0000 osd.3 (osd.3) 265 : cluster 0 2.1e scrub ok 2024-03-11T21:03:41.704 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:41 smithi023 bash[29146]: cluster 2024-03-11T21:03:41.211763+0000 mon.a (mon.0) 1479 : cluster 0 osdmap e270: 8 total, 8 up, 8 in 2024-03-11T21:03:41.724 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:03:41.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:41.728 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:03:41.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:41 smithi146 bash[23809]: cluster 2024-03-11T21:03:40.410069+0000 osd.1 (osd.1) 47 : cluster 0 purged_snaps scrub starts 2024-03-11T21:03:41.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:41 smithi146 bash[23809]: cluster 2024-03-11T21:03:40.410220+0000 osd.1 (osd.1) 48 : cluster 0 purged_snaps scrub ok 2024-03-11T21:03:41.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:41 smithi146 bash[23809]: cluster 2024-03-11T21:03:40.609120+0000 osd.1 (osd.1) 49 : cluster 0 2.4 scrub starts 2024-03-11T21:03:41.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:41 smithi146 bash[23809]: cluster 2024-03-11T21:03:40.610802+0000 osd.1 (osd.1) 50 : cluster 0 2.4 scrub ok 2024-03-11T21:03:41.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:41 smithi146 bash[23809]: cluster 2024-03-11T21:03:40.669438+0000 osd.3 (osd.3) 264 : cluster 0 2.1e scrub starts 2024-03-11T21:03:41.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:41 smithi146 bash[23809]: cluster 2024-03-11T21:03:40.675520+0000 osd.3 (osd.3) 265 : cluster 0 2.1e scrub ok 2024-03-11T21:03:41.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:41 smithi146 bash[23809]: cluster 2024-03-11T21:03:41.211763+0000 mon.a (mon.0) 1479 : cluster 0 osdmap e270: 8 total, 8 up, 8 in 2024-03-11T21:03:41.828 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:41.853 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:41.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:41.857 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:03:41.958 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:41.987 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:41.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:41.990 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:03:42.091 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:03:42.117 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:03:42.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:42.121 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:03:42.221 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:03:42.249 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:03:42.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:42.252 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:03:42.353 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:42.376 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:42.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:42.380 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:03:42.456 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:2024-03-11T21:03:42.460918+0000 min lat: 0.0011821 max lat: 0.509897 avg lat: 0.0100754 2024-03-11T21:03:42.456 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-11T21:03:42.456 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 40 16 63508 63492 99.1956 81.875 0.00718988 0.0100754 2024-03-11T21:03:42.481 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:03:42.505 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:03:42.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:42.511 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:03:42.611 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:03:42.642 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:03:42.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:42.645 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:03:42.745 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:03:42.754 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:42 smithi023 bash[21372]: cluster 2024-03-11T21:03:40.866114+0000 osd.5 (osd.5) 235 : cluster 0 4.3 scrub starts 2024-03-11T21:03:42.754 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:42 smithi023 bash[21372]: cluster 2024-03-11T21:03:40.870708+0000 osd.5 (osd.5) 236 : cluster 0 4.3 scrub ok 2024-03-11T21:03:42.754 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:42 smithi023 bash[21372]: cluster 2024-03-11T21:03:41.965691+0000 osd.7 (osd.7) 273 : cluster 0 6.1c deep-scrub starts 2024-03-11T21:03:42.754 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:42 smithi023 bash[21372]: cluster 2024-03-11T21:03:41.971908+0000 osd.7 (osd.7) 274 : cluster 0 6.1c deep-scrub ok 2024-03-11T21:03:42.754 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:42 smithi023 bash[21372]: cluster 2024-03-11T21:03:42.153677+0000 osd.3 (osd.3) 266 : cluster 0 purged_snaps scrub starts 2024-03-11T21:03:42.754 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:42 smithi023 bash[21372]: cluster 2024-03-11T21:03:42.153749+0000 osd.3 (osd.3) 267 : cluster 0 purged_snaps scrub ok 2024-03-11T21:03:42.754 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:42 smithi023 bash[21372]: cluster 2024-03-11T21:03:42.159493+0000 mgr.y (mgr.24380) 478 : cluster 0 pgmap v536: 148 pgs: 148 active+clean; 3.8 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 75 MiB/s wr, 1.19k op/s 2024-03-11T21:03:42.754 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:42 smithi023 bash[29146]: cluster 2024-03-11T21:03:40.866114+0000 osd.5 (osd.5) 235 : cluster 0 4.3 scrub starts 2024-03-11T21:03:42.754 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:42 smithi023 bash[29146]: cluster 2024-03-11T21:03:40.870708+0000 osd.5 (osd.5) 236 : cluster 0 4.3 scrub ok 2024-03-11T21:03:42.755 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:42 smithi023 bash[29146]: cluster 2024-03-11T21:03:41.965691+0000 osd.7 (osd.7) 273 : cluster 0 6.1c deep-scrub starts 2024-03-11T21:03:42.755 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:42 smithi023 bash[29146]: cluster 2024-03-11T21:03:41.971908+0000 osd.7 (osd.7) 274 : cluster 0 6.1c deep-scrub ok 2024-03-11T21:03:42.755 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:42 smithi023 bash[29146]: cluster 2024-03-11T21:03:42.153677+0000 osd.3 (osd.3) 266 : cluster 0 purged_snaps scrub starts 2024-03-11T21:03:42.755 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:42 smithi023 bash[29146]: cluster 2024-03-11T21:03:42.153749+0000 osd.3 (osd.3) 267 : cluster 0 purged_snaps scrub ok 2024-03-11T21:03:42.755 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:42 smithi023 bash[29146]: cluster 2024-03-11T21:03:42.159493+0000 mgr.y (mgr.24380) 478 : cluster 0 pgmap v536: 148 pgs: 148 active+clean; 3.8 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 75 MiB/s wr, 1.19k op/s 2024-03-11T21:03:42.764 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:03:42.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:42.766 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:03:42.867 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:42.890 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:42.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:42.891 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:03:42.992 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:43.000 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:42 smithi146 bash[23809]: cluster 2024-03-11T21:03:40.866114+0000 osd.5 (osd.5) 235 : cluster 0 4.3 scrub starts 2024-03-11T21:03:43.000 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:42 smithi146 bash[23809]: cluster 2024-03-11T21:03:40.870708+0000 osd.5 (osd.5) 236 : cluster 0 4.3 scrub ok 2024-03-11T21:03:43.001 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:42 smithi146 bash[23809]: cluster 2024-03-11T21:03:41.965691+0000 osd.7 (osd.7) 273 : cluster 0 6.1c deep-scrub starts 2024-03-11T21:03:43.001 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:42 smithi146 bash[23809]: cluster 2024-03-11T21:03:41.971908+0000 osd.7 (osd.7) 274 : cluster 0 6.1c deep-scrub ok 2024-03-11T21:03:43.001 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:42 smithi146 bash[23809]: cluster 2024-03-11T21:03:42.153677+0000 osd.3 (osd.3) 266 : cluster 0 purged_snaps scrub starts 2024-03-11T21:03:43.001 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:42 smithi146 bash[23809]: cluster 2024-03-11T21:03:42.153749+0000 osd.3 (osd.3) 267 : cluster 0 purged_snaps scrub ok 2024-03-11T21:03:43.001 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:42 smithi146 bash[23809]: cluster 2024-03-11T21:03:42.159493+0000 mgr.y (mgr.24380) 478 : cluster 0 pgmap v536: 148 pgs: 148 active+clean; 3.8 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 75 MiB/s wr, 1.19k op/s 2024-03-11T21:03:43.011 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:43.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:43.012 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:03:43.113 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:03:43.139 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:03:43.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:43.141 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:03:43.242 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:43.274 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:43.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:43.278 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:03:43.379 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:03:43.402 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:03:43.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:43.407 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:03:43.456 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 41 16 64272 64256 97.9407 47.75 0.0126112 0.0102052 2024-03-11T21:03:43.508 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:03:43.543 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:03:43.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:43.546 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:03:43.647 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:03:43.681 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:03:43.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:43.683 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:03:43.700 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:43 smithi023 bash[21372]: cluster 2024-03-11T21:03:41.902466+0000 osd.5 (osd.5) 237 : cluster 0 3.c scrub starts 2024-03-11T21:03:43.700 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:43 smithi023 bash[21372]: cluster 2024-03-11T21:03:41.904598+0000 osd.5 (osd.5) 238 : cluster 0 3.c scrub ok 2024-03-11T21:03:43.700 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:43 smithi023 bash[21372]: cluster 2024-03-11T21:03:42.565309+0000 osd.1 (osd.1) 51 : cluster 0 6.1d scrub starts 2024-03-11T21:03:43.700 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:43 smithi023 bash[21372]: cluster 2024-03-11T21:03:42.593237+0000 osd.1 (osd.1) 52 : cluster 0 6.1d scrub ok 2024-03-11T21:03:43.700 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:43 smithi023 bash[21372]: cluster 2024-03-11T21:03:42.675109+0000 mon.a (mon.0) 1480 : cluster 0 osdmap e271: 8 total, 8 up, 8 in 2024-03-11T21:03:43.700 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:43 smithi023 bash[21372]: cluster 2024-03-11T21:03:42.675967+0000 osd.3 (osd.3) 268 : cluster 0 3.7 scrub starts 2024-03-11T21:03:43.700 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:43 smithi023 bash[21372]: cluster 2024-03-11T21:03:42.690404+0000 osd.3 (osd.3) 269 : cluster 0 3.7 scrub ok 2024-03-11T21:03:43.701 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:43 smithi023 bash[29146]: cluster 2024-03-11T21:03:41.902466+0000 osd.5 (osd.5) 237 : cluster 0 3.c scrub starts 2024-03-11T21:03:43.701 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:43 smithi023 bash[29146]: cluster 2024-03-11T21:03:41.904598+0000 osd.5 (osd.5) 238 : cluster 0 3.c scrub ok 2024-03-11T21:03:43.701 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:43 smithi023 bash[29146]: cluster 2024-03-11T21:03:42.565309+0000 osd.1 (osd.1) 51 : cluster 0 6.1d scrub starts 2024-03-11T21:03:43.701 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:43 smithi023 bash[29146]: cluster 2024-03-11T21:03:42.593237+0000 osd.1 (osd.1) 52 : cluster 0 6.1d scrub ok 2024-03-11T21:03:43.701 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:43 smithi023 bash[29146]: cluster 2024-03-11T21:03:42.675109+0000 mon.a (mon.0) 1480 : cluster 0 osdmap e271: 8 total, 8 up, 8 in 2024-03-11T21:03:43.701 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:43 smithi023 bash[29146]: cluster 2024-03-11T21:03:42.675967+0000 osd.3 (osd.3) 268 : cluster 0 3.7 scrub starts 2024-03-11T21:03:43.701 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:43 smithi023 bash[29146]: cluster 2024-03-11T21:03:42.690404+0000 osd.3 (osd.3) 269 : cluster 0 3.7 scrub ok 2024-03-11T21:03:43.784 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:03:43.825 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:03:43.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:43.829 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:03:43.930 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:03:43.956 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:03:43.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:43.958 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:03:44.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:43 smithi146 bash[23809]: cluster 2024-03-11T21:03:41.902466+0000 osd.5 (osd.5) 237 : cluster 0 3.c scrub starts 2024-03-11T21:03:44.013 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:43 smithi146 bash[23809]: cluster 2024-03-11T21:03:41.904598+0000 osd.5 (osd.5) 238 : cluster 0 3.c scrub ok 2024-03-11T21:03:44.013 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:43 smithi146 bash[23809]: cluster 2024-03-11T21:03:42.565309+0000 osd.1 (osd.1) 51 : cluster 0 6.1d scrub starts 2024-03-11T21:03:44.013 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:43 smithi146 bash[23809]: cluster 2024-03-11T21:03:42.593237+0000 osd.1 (osd.1) 52 : cluster 0 6.1d scrub ok 2024-03-11T21:03:44.013 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:43 smithi146 bash[23809]: cluster 2024-03-11T21:03:42.675109+0000 mon.a (mon.0) 1480 : cluster 0 osdmap e271: 8 total, 8 up, 8 in 2024-03-11T21:03:44.013 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:43 smithi146 bash[23809]: cluster 2024-03-11T21:03:42.675967+0000 osd.3 (osd.3) 268 : cluster 0 3.7 scrub starts 2024-03-11T21:03:44.013 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:43 smithi146 bash[23809]: cluster 2024-03-11T21:03:42.690404+0000 osd.3 (osd.3) 269 : cluster 0 3.7 scrub ok 2024-03-11T21:03:44.059 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:03:44.101 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:03:44.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:44.106 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:03:44.207 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:03:44.246 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:03:44.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:44.249 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:03:44.350 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:44.377 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:44.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:44.381 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:03:44.456 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 42 15 65468 65453 97.3899 74.8125 0.00531634 0.0102629 2024-03-11T21:03:44.481 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:44.519 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:44.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:44.522 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:03:44.623 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:03:44.648 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:03:44.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:44.654 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:03:44.755 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:03:44.778 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:03:44.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:44.781 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:03:44.881 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:44.891 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:44 smithi023 bash[21372]: cluster 2024-03-11T21:03:43.318422+0000 osd.4 (osd.4) 67 : cluster 0 6.2 deep-scrub starts 2024-03-11T21:03:44.891 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:44 smithi023 bash[21372]: cluster 2024-03-11T21:03:43.345252+0000 osd.4 (osd.4) 68 : cluster 0 6.2 deep-scrub ok 2024-03-11T21:03:44.891 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:44 smithi023 bash[21372]: cluster 2024-03-11T21:03:43.601611+0000 osd.1 (osd.1) 53 : cluster 0 4.12 scrub starts 2024-03-11T21:03:44.891 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:44 smithi023 bash[21372]: cluster 2024-03-11T21:03:43.605789+0000 osd.1 (osd.1) 54 : cluster 0 4.12 scrub ok 2024-03-11T21:03:44.891 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:44 smithi023 bash[21372]: cluster 2024-03-11T21:03:43.702056+0000 osd.3 (osd.3) 270 : cluster 0 6.c scrub starts 2024-03-11T21:03:44.891 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:44 smithi023 bash[21372]: cluster 2024-03-11T21:03:43.715411+0000 osd.3 (osd.3) 271 : cluster 0 6.c scrub ok 2024-03-11T21:03:44.891 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:44 smithi023 bash[21372]: cluster 2024-03-11T21:03:44.160644+0000 mgr.y (mgr.24380) 479 : cluster 0 pgmap v538: 148 pgs: 148 active+clean; 3.8 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.1 KiB/s rd, 89 MiB/s wr, 1.42k op/s 2024-03-11T21:03:44.891 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:44 smithi023 bash[21372]: cluster 2024-03-11T21:03:44.264897+0000 osd.4 (osd.4) 69 : cluster 0 6.17 scrub starts 2024-03-11T21:03:44.891 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:44 smithi023 bash[21372]: cluster 2024-03-11T21:03:44.289249+0000 osd.6 (osd.6) 198 : cluster 0 5.2 scrub starts 2024-03-11T21:03:44.892 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:44 smithi023 bash[29146]: cluster 2024-03-11T21:03:43.318422+0000 osd.4 (osd.4) 67 : cluster 0 6.2 deep-scrub starts 2024-03-11T21:03:44.892 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:44 smithi023 bash[29146]: cluster 2024-03-11T21:03:43.345252+0000 osd.4 (osd.4) 68 : cluster 0 6.2 deep-scrub ok 2024-03-11T21:03:44.892 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:44 smithi023 bash[29146]: cluster 2024-03-11T21:03:43.601611+0000 osd.1 (osd.1) 53 : cluster 0 4.12 scrub starts 2024-03-11T21:03:44.892 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:44 smithi023 bash[29146]: cluster 2024-03-11T21:03:43.605789+0000 osd.1 (osd.1) 54 : cluster 0 4.12 scrub ok 2024-03-11T21:03:44.892 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:44 smithi023 bash[29146]: cluster 2024-03-11T21:03:43.702056+0000 osd.3 (osd.3) 270 : cluster 0 6.c scrub starts 2024-03-11T21:03:44.892 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:44 smithi023 bash[29146]: cluster 2024-03-11T21:03:43.715411+0000 osd.3 (osd.3) 271 : cluster 0 6.c scrub ok 2024-03-11T21:03:44.892 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:44 smithi023 bash[29146]: cluster 2024-03-11T21:03:44.160644+0000 mgr.y (mgr.24380) 479 : cluster 0 pgmap v538: 148 pgs: 148 active+clean; 3.8 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.1 KiB/s rd, 89 MiB/s wr, 1.42k op/s 2024-03-11T21:03:44.892 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:44 smithi023 bash[29146]: cluster 2024-03-11T21:03:44.264897+0000 osd.4 (osd.4) 69 : cluster 0 6.17 scrub starts 2024-03-11T21:03:44.892 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:44 smithi023 bash[29146]: cluster 2024-03-11T21:03:44.289249+0000 osd.6 (osd.6) 198 : cluster 0 5.2 scrub starts 2024-03-11T21:03:44.909 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:44.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:44.913 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:03:45.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:44 smithi146 bash[23809]: cluster 2024-03-11T21:03:43.318422+0000 osd.4 (osd.4) 67 : cluster 0 6.2 deep-scrub starts 2024-03-11T21:03:45.013 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:44 smithi146 bash[23809]: cluster 2024-03-11T21:03:43.345252+0000 osd.4 (osd.4) 68 : cluster 0 6.2 deep-scrub ok 2024-03-11T21:03:45.013 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:44 smithi146 bash[23809]: cluster 2024-03-11T21:03:43.601611+0000 osd.1 (osd.1) 53 : cluster 0 4.12 scrub starts 2024-03-11T21:03:45.013 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:44 smithi146 bash[23809]: cluster 2024-03-11T21:03:43.605789+0000 osd.1 (osd.1) 54 : cluster 0 4.12 scrub ok 2024-03-11T21:03:45.013 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:44 smithi146 bash[23809]: cluster 2024-03-11T21:03:43.702056+0000 osd.3 (osd.3) 270 : cluster 0 6.c scrub starts 2024-03-11T21:03:45.013 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:44 smithi146 bash[23809]: cluster 2024-03-11T21:03:43.715411+0000 osd.3 (osd.3) 271 : cluster 0 6.c scrub ok 2024-03-11T21:03:45.013 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:44 smithi146 bash[23809]: cluster 2024-03-11T21:03:44.160644+0000 mgr.y (mgr.24380) 479 : cluster 0 pgmap v538: 148 pgs: 148 active+clean; 3.8 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.1 KiB/s rd, 89 MiB/s wr, 1.42k op/s 2024-03-11T21:03:45.013 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:44 smithi146 bash[23809]: cluster 2024-03-11T21:03:44.264897+0000 osd.4 (osd.4) 69 : cluster 0 6.17 scrub starts 2024-03-11T21:03:45.013 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:44 smithi146 bash[23809]: cluster 2024-03-11T21:03:44.289249+0000 osd.6 (osd.6) 198 : cluster 0 5.2 scrub starts 2024-03-11T21:03:45.013 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:03:45.039 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:03:45.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:45.042 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:03:45.143 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:03:45.168 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:03:45.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:45.171 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:03:45.272 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:45.300 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:45.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:45.304 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:03:45.405 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:45.433 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:45.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:45.435 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:03:45.456 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 43 16 66542 66526 96.6844 67.0625 0.0036787 0.0103326 2024-03-11T21:03:45.536 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:45.548 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:03:45 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:03:45.562 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:45.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:45.564 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:03:45.665 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:03:45.704 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:03:45.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:45.708 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:03:45.809 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:03:45.818 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:45 smithi146 bash[23809]: cluster 2024-03-11T21:03:43.928911+0000 osd.5 (osd.5) 239 : cluster 0 4.15 scrub starts 2024-03-11T21:03:45.818 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:45 smithi146 bash[23809]: cluster 2024-03-11T21:03:43.933544+0000 osd.5 (osd.5) 240 : cluster 0 4.15 scrub ok 2024-03-11T21:03:45.818 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:45 smithi146 bash[23809]: cluster 2024-03-11T21:03:44.305707+0000 osd.6 (osd.6) 199 : cluster 0 5.2 scrub ok 2024-03-11T21:03:45.818 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:45 smithi146 bash[23809]: cluster 2024-03-11T21:03:44.322160+0000 osd.4 (osd.4) 70 : cluster 0 6.17 scrub ok 2024-03-11T21:03:45.818 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:45 smithi146 bash[23809]: cluster 2024-03-11T21:03:44.640074+0000 osd.1 (osd.1) 55 : cluster 0 4.4 scrub starts 2024-03-11T21:03:45.819 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:45 smithi146 bash[23809]: cluster 2024-03-11T21:03:44.649306+0000 osd.1 (osd.1) 56 : cluster 0 4.4 scrub ok 2024-03-11T21:03:45.819 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:45 smithi146 bash[23809]: cluster 2024-03-11T21:03:45.269670+0000 osd.4 (osd.4) 71 : cluster 0 4.13 scrub starts 2024-03-11T21:03:45.819 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:45 smithi146 bash[23809]: cluster 2024-03-11T21:03:45.273525+0000 osd.4 (osd.4) 72 : cluster 0 4.13 scrub ok 2024-03-11T21:03:45.819 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:45 smithi146 bash[23809]: cluster 2024-03-11T21:03:45.417528+0000 osd.0 (osd.0) 193 : cluster 0 4.6 scrub starts 2024-03-11T21:03:45.819 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:45 smithi146 bash[23809]: cluster 2024-03-11T21:03:45.421438+0000 osd.0 (osd.0) 194 : cluster 0 4.6 scrub ok 2024-03-11T21:03:45.819 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:45 smithi146 bash[23809]: audit 2024-03-11T21:03:45.459851+0000 mgr.y (mgr.24380) 480 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:03:45.829 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:03:45.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:45.831 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:03:45.932 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:03:45.955 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:03:45.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:45.957 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:03:46.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:45 smithi023 bash[29146]: cluster 2024-03-11T21:03:43.928911+0000 osd.5 (osd.5) 239 : cluster 0 4.15 scrub starts 2024-03-11T21:03:46.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:45 smithi023 bash[29146]: cluster 2024-03-11T21:03:43.933544+0000 osd.5 (osd.5) 240 : cluster 0 4.15 scrub ok 2024-03-11T21:03:46.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:45 smithi023 bash[29146]: cluster 2024-03-11T21:03:44.305707+0000 osd.6 (osd.6) 199 : cluster 0 5.2 scrub ok 2024-03-11T21:03:46.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:45 smithi023 bash[29146]: cluster 2024-03-11T21:03:44.322160+0000 osd.4 (osd.4) 70 : cluster 0 6.17 scrub ok 2024-03-11T21:03:46.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:45 smithi023 bash[29146]: cluster 2024-03-11T21:03:44.640074+0000 osd.1 (osd.1) 55 : cluster 0 4.4 scrub starts 2024-03-11T21:03:46.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:45 smithi023 bash[29146]: cluster 2024-03-11T21:03:44.649306+0000 osd.1 (osd.1) 56 : cluster 0 4.4 scrub ok 2024-03-11T21:03:46.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:45 smithi023 bash[29146]: cluster 2024-03-11T21:03:45.269670+0000 osd.4 (osd.4) 71 : cluster 0 4.13 scrub starts 2024-03-11T21:03:46.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:45 smithi023 bash[29146]: cluster 2024-03-11T21:03:45.273525+0000 osd.4 (osd.4) 72 : cluster 0 4.13 scrub ok 2024-03-11T21:03:46.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:45 smithi023 bash[29146]: cluster 2024-03-11T21:03:45.417528+0000 osd.0 (osd.0) 193 : cluster 0 4.6 scrub starts 2024-03-11T21:03:46.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:45 smithi023 bash[29146]: cluster 2024-03-11T21:03:45.421438+0000 osd.0 (osd.0) 194 : cluster 0 4.6 scrub ok 2024-03-11T21:03:46.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:45 smithi023 bash[29146]: audit 2024-03-11T21:03:45.459851+0000 mgr.y (mgr.24380) 480 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:03:46.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:45 smithi023 bash[21372]: cluster 2024-03-11T21:03:43.928911+0000 osd.5 (osd.5) 239 : cluster 0 4.15 scrub starts 2024-03-11T21:03:46.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:45 smithi023 bash[21372]: cluster 2024-03-11T21:03:43.933544+0000 osd.5 (osd.5) 240 : cluster 0 4.15 scrub ok 2024-03-11T21:03:46.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:45 smithi023 bash[21372]: cluster 2024-03-11T21:03:44.305707+0000 osd.6 (osd.6) 199 : cluster 0 5.2 scrub ok 2024-03-11T21:03:46.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:45 smithi023 bash[21372]: cluster 2024-03-11T21:03:44.322160+0000 osd.4 (osd.4) 70 : cluster 0 6.17 scrub ok 2024-03-11T21:03:46.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:45 smithi023 bash[21372]: cluster 2024-03-11T21:03:44.640074+0000 osd.1 (osd.1) 55 : cluster 0 4.4 scrub starts 2024-03-11T21:03:46.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:45 smithi023 bash[21372]: cluster 2024-03-11T21:03:44.649306+0000 osd.1 (osd.1) 56 : cluster 0 4.4 scrub ok 2024-03-11T21:03:46.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:45 smithi023 bash[21372]: cluster 2024-03-11T21:03:45.269670+0000 osd.4 (osd.4) 71 : cluster 0 4.13 scrub starts 2024-03-11T21:03:46.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:45 smithi023 bash[21372]: cluster 2024-03-11T21:03:45.273525+0000 osd.4 (osd.4) 72 : cluster 0 4.13 scrub ok 2024-03-11T21:03:46.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:45 smithi023 bash[21372]: cluster 2024-03-11T21:03:45.417528+0000 osd.0 (osd.0) 193 : cluster 0 4.6 scrub starts 2024-03-11T21:03:46.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:45 smithi023 bash[21372]: cluster 2024-03-11T21:03:45.421438+0000 osd.0 (osd.0) 194 : cluster 0 4.6 scrub ok 2024-03-11T21:03:46.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:45 smithi023 bash[21372]: audit 2024-03-11T21:03:45.459851+0000 mgr.y (mgr.24380) 480 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:03:46.058 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:03:46.083 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:03:46.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:46.085 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:03:46.186 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:03:46.216 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:03:46.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:46.219 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:03:46.319 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:46.343 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:46.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:46.346 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:03:46.447 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:03:46.456 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 44 16 67149 67133 95.3493 37.9375 0.00814166 0.0104821 2024-03-11T21:03:46.474 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:03:46.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:46.479 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:03:46.580 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:46.607 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:46.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:46.613 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:03:46.714 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:03:46.723 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:46 smithi023 bash[21372]: cluster 2024-03-11T21:03:44.893313+0000 osd.5 (osd.5) 241 : cluster 0 4.8 scrub starts 2024-03-11T21:03:46.724 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:46 smithi023 bash[21372]: cluster 2024-03-11T21:03:44.899069+0000 osd.5 (osd.5) 242 : cluster 0 4.8 scrub ok 2024-03-11T21:03:46.724 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:46 smithi023 bash[21372]: cluster 2024-03-11T21:03:45.973524+0000 osd.7 (osd.7) 275 : cluster 0 3.13 scrub starts 2024-03-11T21:03:46.724 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:46 smithi023 bash[21372]: cluster 2024-03-11T21:03:45.977193+0000 osd.7 (osd.7) 276 : cluster 0 3.13 scrub ok 2024-03-11T21:03:46.724 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:46 smithi023 bash[21372]: cluster 2024-03-11T21:03:46.161972+0000 mgr.y (mgr.24380) 481 : cluster 0 pgmap v539: 148 pgs: 148 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.7 KiB/s rd, 70 MiB/s wr, 1.12k op/s 2024-03-11T21:03:46.724 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:46 smithi023 bash[29146]: cluster 2024-03-11T21:03:44.893313+0000 osd.5 (osd.5) 241 : cluster 0 4.8 scrub starts 2024-03-11T21:03:46.724 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:46 smithi023 bash[29146]: cluster 2024-03-11T21:03:44.899069+0000 osd.5 (osd.5) 242 : cluster 0 4.8 scrub ok 2024-03-11T21:03:46.724 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:46 smithi023 bash[29146]: cluster 2024-03-11T21:03:45.973524+0000 osd.7 (osd.7) 275 : cluster 0 3.13 scrub starts 2024-03-11T21:03:46.724 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:46 smithi023 bash[29146]: cluster 2024-03-11T21:03:45.977193+0000 osd.7 (osd.7) 276 : cluster 0 3.13 scrub ok 2024-03-11T21:03:46.724 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:46 smithi023 bash[29146]: cluster 2024-03-11T21:03:46.161972+0000 mgr.y (mgr.24380) 481 : cluster 0 pgmap v539: 148 pgs: 148 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.7 KiB/s rd, 70 MiB/s wr, 1.12k op/s 2024-03-11T21:03:46.740 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:03:46.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:46.743 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:03:46.844 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:46.858 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:46 smithi146 bash[23809]: cluster 2024-03-11T21:03:44.893313+0000 osd.5 (osd.5) 241 : cluster 0 4.8 scrub starts 2024-03-11T21:03:46.858 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:46 smithi146 bash[23809]: cluster 2024-03-11T21:03:44.899069+0000 osd.5 (osd.5) 242 : cluster 0 4.8 scrub ok 2024-03-11T21:03:46.858 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:46 smithi146 bash[23809]: cluster 2024-03-11T21:03:45.973524+0000 osd.7 (osd.7) 275 : cluster 0 3.13 scrub starts 2024-03-11T21:03:46.859 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:46 smithi146 bash[23809]: cluster 2024-03-11T21:03:45.977193+0000 osd.7 (osd.7) 276 : cluster 0 3.13 scrub ok 2024-03-11T21:03:46.859 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:46 smithi146 bash[23809]: cluster 2024-03-11T21:03:46.161972+0000 mgr.y (mgr.24380) 481 : cluster 0 pgmap v539: 148 pgs: 148 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.7 KiB/s rd, 70 MiB/s wr, 1.12k op/s 2024-03-11T21:03:46.883 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:46.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:46.885 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:03:46.986 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:03:47.012 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:03:47.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:47.016 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:03:47.117 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:03:47.136 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:03:47.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:47.138 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:03:47.239 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:47.268 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:47.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:47.272 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:03:47.373 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:03:47.406 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:03:47.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:47.408 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:03:47.457 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 45 16 68202 68186 94.6927 65.8125 0.00609765 0.0105549 2024-03-11T21:03:47.510 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:47.540 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:47.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:47.541 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:03:47.642 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:03:47.664 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:03:47.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:47.666 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:03:47.767 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:03:47.798 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:03:47.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:47.804 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:03:47.905 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:47.915 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:47 smithi146 bash[23809]: cluster 2024-03-11T21:03:47.376306+0000 osd.0 (osd.0) 195 : cluster 0 3.6 scrub starts 2024-03-11T21:03:47.916 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:47 smithi146 bash[23809]: cluster 2024-03-11T21:03:47.381052+0000 osd.0 (osd.0) 196 : cluster 0 3.6 scrub ok 2024-03-11T21:03:47.934 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:47.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:47.937 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:03:48.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:47 smithi023 bash[21372]: cluster 2024-03-11T21:03:47.376306+0000 osd.0 (osd.0) 195 : cluster 0 3.6 scrub starts 2024-03-11T21:03:48.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:47 smithi023 bash[21372]: cluster 2024-03-11T21:03:47.381052+0000 osd.0 (osd.0) 196 : cluster 0 3.6 scrub ok 2024-03-11T21:03:48.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:47 smithi023 bash[29146]: cluster 2024-03-11T21:03:47.376306+0000 osd.0 (osd.0) 195 : cluster 0 3.6 scrub starts 2024-03-11T21:03:48.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:47 smithi023 bash[29146]: cluster 2024-03-11T21:03:47.381052+0000 osd.0 (osd.0) 196 : cluster 0 3.6 scrub ok 2024-03-11T21:03:48.039 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:48.062 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:48.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:48.067 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:03:48.168 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:48.205 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:48.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:48.209 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:03:48.309 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:48.333 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:48.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:48.336 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:03:48.437 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:48.456 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 46 16 69262 69246 94.0744 66.25 0.010409 0.0106241 2024-03-11T21:03:48.464 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:48.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:48.468 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:03:48.500 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-03-11T21:03:48.500 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-03-11T21:03:48.500 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-03-11T21:03:48.500 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-03-11T21:03:48.500 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.501 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.24629.0:85823 7.c 7:3f3c9d65:::benchmark_data_smithi023_67959_object85822:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-03-11T21:03:48.501 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T20:55:52.512989+0000", 2024-03-11T21:03:48.501 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 475.984693338, 2024-03-11T21:03:48.501 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.39884113199999999, 2024-03-11T21:03:48.501 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:03:48.501 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:03:48.501 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-03-11T21:03:48.501 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.24629", 2024-03-11T21:03:48.501 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.23:0/959732609", 2024-03-11T21:03:48.502 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 85823 2024-03-11T21:03:48.502 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.502 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:03:48.502 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.502 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:03:48.502 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:52.512989+0000", 2024-03-11T21:03:48.502 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:03:48.502 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.503 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.503 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:03:48.503 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:52.512989+0000", 2024-03-11T21:03:48.503 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:03:48.503 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.503 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.503 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:03:48.503 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:52.512992+0000", 2024-03-11T21:03:48.503 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0640000000000001e-06 2024-03-11T21:03:48.503 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.504 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.504 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:03:48.504 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:52.513027+0000", 2024-03-11T21:03:48.504 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5432000000000002e-05 2024-03-11T21:03:48.504 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.504 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.504 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:03:48.504 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:52.513042+0000", 2024-03-11T21:03:48.505 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4769e-05 2024-03-11T21:03:48.505 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.505 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.505 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:03:48.505 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:52.513052+0000", 2024-03-11T21:03:48.505 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0365e-05 2024-03-11T21:03:48.505 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.505 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.505 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:03:48.505 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:52.513107+0000", 2024-03-11T21:03:48.505 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5300000000000002e-05 2024-03-11T21:03:48.505 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.505 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.506 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:03:48.506 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:52.513232+0000", 2024-03-11T21:03:48.506 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012486299999999999 2024-03-11T21:03:48.506 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.506 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.506 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3,7", 2024-03-11T21:03:48.506 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:52.513380+0000", 2024-03-11T21:03:48.506 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000147603 2024-03-11T21:03:48.506 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.506 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.506 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:03:48.506 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:52.516465+0000", 2024-03-11T21:03:48.506 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030850920000000002 2024-03-11T21:03:48.506 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.507 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.507 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-03-11T21:03:48.507 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:52.517077+0000", 2024-03-11T21:03:48.507 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00061202800000000003 2024-03-11T21:03:48.507 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.507 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.507 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:03:48.507 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:52.911632+0000", 2024-03-11T21:03:48.507 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.39455478100000002 2024-03-11T21:03:48.507 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.507 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.507 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:03:48.507 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:52.911778+0000", 2024-03-11T21:03:48.507 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000146129 2024-03-11T21:03:48.507 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.508 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.508 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:03:48.508 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:52.911831+0000", 2024-03-11T21:03:48.508 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2706e-05 2024-03-11T21:03:48.508 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:03:48.508 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:03:48.508 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:03:48.508 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.508 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.508 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.24629.0:85833 7.d 7:bfb2a6d1:::benchmark_data_smithi023_67959_object85832:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-03-11T21:03:48.508 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T20:55:52.532467+0000", 2024-03-11T21:03:48.508 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 475.965215328, 2024-03-11T21:03:48.508 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.39260756299999999, 2024-03-11T21:03:48.508 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:03:48.509 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:03:48.509 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-03-11T21:03:48.509 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.24629", 2024-03-11T21:03:48.509 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.23:0/959732609", 2024-03-11T21:03:48.509 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 85833 2024-03-11T21:03:48.509 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.509 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:03:48.509 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.509 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:03:48.509 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:52.532467+0000", 2024-03-11T21:03:48.509 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:03:48.509 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.509 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.509 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:03:48.509 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:52.532467+0000", 2024-03-11T21:03:48.509 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:03:48.510 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.510 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.510 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:03:48.510 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:52.532469+0000", 2024-03-11T21:03:48.510 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.812e-06 2024-03-11T21:03:48.510 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.510 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.510 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:03:48.510 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:52.532512+0000", 2024-03-11T21:03:48.510 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2347e-05 2024-03-11T21:03:48.510 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.510 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.510 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:03:48.510 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:52.532527+0000", 2024-03-11T21:03:48.510 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5026e-05 2024-03-11T21:03:48.511 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.511 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.511 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:03:48.511 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:52.532535+0000", 2024-03-11T21:03:48.511 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.0160000000000008e-06 2024-03-11T21:03:48.511 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.511 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.511 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:03:48.511 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:52.532590+0000", 2024-03-11T21:03:48.511 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5034000000000002e-05 2024-03-11T21:03:48.511 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.511 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.511 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:03:48.511 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:52.532683+0000", 2024-03-11T21:03:48.511 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.3218000000000001e-05 2024-03-11T21:03:48.512 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.512 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.512 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0,3", 2024-03-11T21:03:48.512 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:52.532778+0000", 2024-03-11T21:03:48.512 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5073000000000003e-05 2024-03-11T21:03:48.512 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.512 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.512 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-03-11T21:03:48.512 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:52.534649+0000", 2024-03-11T21:03:48.512 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0018705530000000001 2024-03-11T21:03:48.512 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.512 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.512 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:03:48.512 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:52.855883+0000", 2024-03-11T21:03:48.513 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.32123408799999997 2024-03-11T21:03:48.513 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.513 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.513 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:03:48.513 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:52.924958+0000", 2024-03-11T21:03:48.513 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.069075757000000002 2024-03-11T21:03:48.513 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.513 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.513 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:03:48.513 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:52.925023+0000", 2024-03-11T21:03:48.513 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4893000000000004e-05 2024-03-11T21:03:48.513 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.513 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.513 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:03:48.513 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:52.925075+0000", 2024-03-11T21:03:48.513 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1746000000000002e-05 2024-03-11T21:03:48.514 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:03:48.514 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:03:48.514 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:03:48.514 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.514 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.514 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.24281.0:1743 4.1f 4:f95f44c2:::notify.0:head [watch reconnect cookie 94110942026496 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e162)", 2024-03-11T21:03:48.514 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T20:57:23.576419+0000", 2024-03-11T21:03:48.514 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 384.92126377, 2024-03-11T21:03:48.514 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0849938729999999, 2024-03-11T21:03:48.514 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:03:48.514 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-03-11T21:03:48.514 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-03-11T21:03:48.514 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.24281", 2024-03-11T21:03:48.514 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.23:0/2185274774", 2024-03-11T21:03:48.515 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1743 2024-03-11T21:03:48.515 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.515 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:03:48.515 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.515 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:03:48.515 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:23.576419+0000", 2024-03-11T21:03:48.515 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:03:48.515 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.515 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.516 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:03:48.516 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:23.576419+0000", 2024-03-11T21:03:48.516 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:03:48.516 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.516 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.516 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:03:48.516 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:23.576421+0000", 2024-03-11T21:03:48.516 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5829999999999999e-06 2024-03-11T21:03:48.516 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.516 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.516 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:03:48.516 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:23.576426+0000", 2024-03-11T21:03:48.516 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8470000000000001e-06 2024-03-11T21:03:48.516 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.517 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.517 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:03:48.517 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:23.576444+0000", 2024-03-11T21:03:48.517 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8009999999999999e-05 2024-03-11T21:03:48.517 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.517 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.517 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:03:48.517 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:23.576453+0000", 2024-03-11T21:03:48.517 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.9239999999999996e-06 2024-03-11T21:03:48.517 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.517 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.517 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:03:48.517 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:23.576517+0000", 2024-03-11T21:03:48.517 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3705000000000001e-05 2024-03-11T21:03:48.517 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.518 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.518 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op must wait for map", 2024-03-11T21:03:48.518 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:23.576523+0000", 2024-03-11T21:03:48.518 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.93e-06 2024-03-11T21:03:48.518 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.518 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.518 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:03:48.518 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:23.596486+0000", 2024-03-11T21:03:48.518 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.019962608 2024-03-11T21:03:48.518 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.518 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.518 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-03-11T21:03:48.518 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:23.596497+0000", 2024-03-11T21:03:48.518 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.133e-05 2024-03-11T21:03:48.519 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.519 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.519 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:03:48.519 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:24.638107+0000", 2024-03-11T21:03:48.519 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0416098309999999 2024-03-11T21:03:48.519 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.519 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.519 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for degraded object", 2024-03-11T21:03:48.519 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:24.638317+0000", 2024-03-11T21:03:48.519 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000209924 2024-03-11T21:03:48.519 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.519 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.519 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:03:48.519 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:24.659754+0000", 2024-03-11T21:03:48.519 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.021436874000000002 2024-03-11T21:03:48.520 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.520 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.520 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:03:48.520 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:24.659836+0000", 2024-03-11T21:03:48.520 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.2045000000000005e-05 2024-03-11T21:03:48.520 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.520 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.520 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:03:48.520 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:24.661413+0000", 2024-03-11T21:03:48.520 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0015772620000000001 2024-03-11T21:03:48.521 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:03:48.521 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:03:48.521 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:03:48.521 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.521 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.521 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.24629.0:167630 7.d 7:b965cb52:::benchmark_data_smithi023_67959_object57389:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e162)", 2024-03-11T21:03:48.521 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T20:57:23.602745+0000", 2024-03-11T21:03:48.521 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 384.89493773700002, 2024-03-11T21:03:48.521 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1738579420000002, 2024-03-11T21:03:48.521 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:03:48.522 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:03:48.522 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-03-11T21:03:48.522 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.24629", 2024-03-11T21:03:48.522 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.23:0/959732609", 2024-03-11T21:03:48.522 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 167630 2024-03-11T21:03:48.522 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.522 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:03:48.522 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.523 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:03:48.523 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:23.602745+0000", 2024-03-11T21:03:48.523 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:03:48.523 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.523 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.523 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:03:48.523 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:23.602745+0000", 2024-03-11T21:03:48.523 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:03:48.523 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.524 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.524 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:03:48.524 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:23.602747+0000", 2024-03-11T21:03:48.524 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.767e-06 2024-03-11T21:03:48.524 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.524 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.524 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:03:48.524 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:23.602753+0000", 2024-03-11T21:03:48.524 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7649999999999999e-06 2024-03-11T21:03:48.525 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.525 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.525 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:03:48.525 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:23.602766+0000", 2024-03-11T21:03:48.525 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3685e-05 2024-03-11T21:03:48.525 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.525 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.525 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:03:48.526 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:23.602775+0000", 2024-03-11T21:03:48.526 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.3899999999999993e-06 2024-03-11T21:03:48.526 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.526 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.526 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:03:48.526 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:23.602929+0000", 2024-03-11T21:03:48.526 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015482899999999999 2024-03-11T21:03:48.526 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.526 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.526 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-03-11T21:03:48.526 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:23.602945+0000", 2024-03-11T21:03:48.526 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5787999999999999e-05 2024-03-11T21:03:48.526 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.527 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.527 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:03:48.527 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:24.610278+0000", 2024-03-11T21:03:48.527 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.007332702 2024-03-11T21:03:48.527 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.527 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.527 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-03-11T21:03:48.527 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:24.610291+0000", 2024-03-11T21:03:48.527 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2982999999999999e-05 2024-03-11T21:03:48.527 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.527 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.527 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:03:48.527 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:25.762736+0000", 2024-03-11T21:03:48.527 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.152444553 2024-03-11T21:03:48.528 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.528 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.528 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:03:48.528 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:25.762840+0000", 2024-03-11T21:03:48.528 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000104047 2024-03-11T21:03:48.528 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.528 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.528 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0,1,3", 2024-03-11T21:03:48.528 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:25.762923+0000", 2024-03-11T21:03:48.528 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.3388999999999997e-05 2024-03-11T21:03:48.528 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.528 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.528 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-03-11T21:03:48.528 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:25.765148+0000", 2024-03-11T21:03:48.528 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0022245949999999998 2024-03-11T21:03:48.529 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.529 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.529 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:03:48.529 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:25.766395+0000", 2024-03-11T21:03:48.529 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001247623 2024-03-11T21:03:48.529 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.529 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.529 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:03:48.529 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:25.767246+0000", 2024-03-11T21:03:48.529 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00085092999999999996 2024-03-11T21:03:48.529 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.529 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.529 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:03:48.529 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:25.775982+0000", 2024-03-11T21:03:48.529 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0087362280000000004 2024-03-11T21:03:48.530 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.530 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.530 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:03:48.530 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:25.776181+0000", 2024-03-11T21:03:48.530 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000198875 2024-03-11T21:03:48.530 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.530 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.530 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:03:48.530 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:57:25.776603+0000", 2024-03-11T21:03:48.530 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00042179300000000001 2024-03-11T21:03:48.530 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:03:48.530 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:03:48.530 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:03:48.530 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.530 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.531 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.24629.0:215020 7.d 7:b71b05f8:::benchmark_data_smithi023_67959_object104779:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-03-11T21:03:48.531 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T20:58:26.561811+0000", 2024-03-11T21:03:48.531 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 321.93587179299999, 2024-03-11T21:03:48.531 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.39884447000000001, 2024-03-11T21:03:48.531 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:03:48.531 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:03:48.531 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-03-11T21:03:48.531 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.24629", 2024-03-11T21:03:48.531 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.23:0/959732609", 2024-03-11T21:03:48.531 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 215020 2024-03-11T21:03:48.531 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.531 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:03:48.531 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.531 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:03:48.531 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.561811+0000", 2024-03-11T21:03:48.532 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:03:48.532 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.532 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.532 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:03:48.532 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.561811+0000", 2024-03-11T21:03:48.532 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:03:48.532 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.532 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.532 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:03:48.532 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.561812+0000", 2024-03-11T21:03:48.532 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5799999999999998e-07 2024-03-11T21:03:48.532 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.532 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.532 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:03:48.532 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.561816+0000", 2024-03-11T21:03:48.533 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1509999999999997e-06 2024-03-11T21:03:48.533 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.533 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.533 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:03:48.533 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.602313+0000", 2024-03-11T21:03:48.533 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040497035000000001 2024-03-11T21:03:48.533 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.533 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.533 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:03:48.533 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.602330+0000", 2024-03-11T21:03:48.533 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7122999999999999e-05 2024-03-11T21:03:48.533 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.533 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.533 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:03:48.533 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.602629+0000", 2024-03-11T21:03:48.534 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00029900100000000002 2024-03-11T21:03:48.534 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.534 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.534 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:03:48.534 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.602787+0000", 2024-03-11T21:03:48.534 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015814799999999999 2024-03-11T21:03:48.534 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.534 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.534 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0,3", 2024-03-11T21:03:48.534 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.602934+0000", 2024-03-11T21:03:48.534 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000146377 2024-03-11T21:03:48.534 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.534 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.534 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-03-11T21:03:48.534 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.612203+0000", 2024-03-11T21:03:48.535 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0092688010000000001 2024-03-11T21:03:48.535 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.535 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.535 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:03:48.535 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.959803+0000", 2024-03-11T21:03:48.535 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.34760072800000003 2024-03-11T21:03:48.535 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.535 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.535 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:03:48.535 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.960317+0000", 2024-03-11T21:03:48.535 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00051367199999999996 2024-03-11T21:03:48.535 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.535 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.535 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:03:48.535 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.960428+0000", 2024-03-11T21:03:48.536 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000110668 2024-03-11T21:03:48.536 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.536 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.536 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:03:48.536 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.960655+0000", 2024-03-11T21:03:48.536 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000227808 2024-03-11T21:03:48.536 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:03:48.536 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:03:48.536 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:03:48.536 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.536 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.536 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.24629.0:215021 7.d 7:bf38f9c8:::benchmark_data_smithi023_67959_object104780:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-03-11T21:03:48.536 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T20:58:26.602373+0000", 2024-03-11T21:03:48.536 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 321.89530957800002, 2024-03-11T21:03:48.537 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.358911119, 2024-03-11T21:03:48.537 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:03:48.537 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:03:48.537 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-03-11T21:03:48.537 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.24629", 2024-03-11T21:03:48.537 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.23:0/959732609", 2024-03-11T21:03:48.537 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 215021 2024-03-11T21:03:48.537 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.537 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:03:48.537 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.537 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:03:48.537 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.602373+0000", 2024-03-11T21:03:48.537 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:03:48.538 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.538 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.538 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:03:48.538 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.602373+0000", 2024-03-11T21:03:48.538 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:03:48.538 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.538 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.538 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:03:48.538 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.602375+0000", 2024-03-11T21:03:48.538 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3880000000000001e-06 2024-03-11T21:03:48.538 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.538 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.538 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:03:48.538 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.602382+0000", 2024-03-11T21:03:48.539 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2819999999999999e-06 2024-03-11T21:03:48.539 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.539 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.539 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:03:48.539 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.602393+0000", 2024-03-11T21:03:48.539 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0958000000000001e-05 2024-03-11T21:03:48.539 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.539 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.539 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:03:48.539 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.602399+0000", 2024-03-11T21:03:48.539 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6189999999999996e-06 2024-03-11T21:03:48.539 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.539 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.539 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:03:48.539 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.606491+0000", 2024-03-11T21:03:48.540 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.004091239 2024-03-11T21:03:48.540 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.540 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.540 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:03:48.540 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.606597+0000", 2024-03-11T21:03:48.540 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010635400000000001 2024-03-11T21:03:48.540 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.540 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.540 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0,3", 2024-03-11T21:03:48.540 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.606706+0000", 2024-03-11T21:03:48.540 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000109411 2024-03-11T21:03:48.540 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.540 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.540 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-03-11T21:03:48.540 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.612208+0000", 2024-03-11T21:03:48.540 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.005501227 2024-03-11T21:03:48.541 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.541 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.541 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:03:48.541 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.959994+0000", 2024-03-11T21:03:48.541 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.34778618100000003 2024-03-11T21:03:48.541 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.541 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.541 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:03:48.541 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.960802+0000", 2024-03-11T21:03:48.542 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00080840299999999996 2024-03-11T21:03:48.542 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.542 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.542 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:03:48.542 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.961002+0000", 2024-03-11T21:03:48.542 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019993300000000001 2024-03-11T21:03:48.542 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.542 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.543 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:03:48.543 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.961284+0000", 2024-03-11T21:03:48.543 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00028212400000000002 2024-03-11T21:03:48.543 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:03:48.543 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:03:48.543 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:03:48.543 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.543 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.543 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.24629.0:215024 7.d 7:b6931494:::benchmark_data_smithi023_67959_object104783:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-03-11T21:03:48.543 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T20:58:26.602408+0000", 2024-03-11T21:03:48.543 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 321.89527452700003, 2024-03-11T21:03:48.543 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.359623796, 2024-03-11T21:03:48.543 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:03:48.543 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:03:48.544 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-03-11T21:03:48.544 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.24629", 2024-03-11T21:03:48.544 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.23:0/959732609", 2024-03-11T21:03:48.544 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 215024 2024-03-11T21:03:48.544 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.544 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:03:48.544 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.544 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:03:48.544 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.602408+0000", 2024-03-11T21:03:48.544 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:03:48.544 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.544 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.544 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:03:48.544 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.602408+0000", 2024-03-11T21:03:48.545 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:03:48.545 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.545 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.545 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:03:48.545 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.602409+0000", 2024-03-11T21:03:48.545 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0299999999999999e-07 2024-03-11T21:03:48.545 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.545 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.545 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:03:48.545 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.602411+0000", 2024-03-11T21:03:48.545 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5100000000000001e-06 2024-03-11T21:03:48.545 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.545 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.545 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:03:48.545 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.602417+0000", 2024-03-11T21:03:48.546 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2049999999999999e-06 2024-03-11T21:03:48.546 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.546 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.546 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:03:48.546 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.602423+0000", 2024-03-11T21:03:48.546 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3870000000000003e-06 2024-03-11T21:03:48.546 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.546 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.546 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:03:48.546 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.609281+0000", 2024-03-11T21:03:48.546 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0068580799999999999 2024-03-11T21:03:48.546 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.546 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.546 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:03:48.547 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.609378+0000", 2024-03-11T21:03:48.547 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6768000000000006e-05 2024-03-11T21:03:48.547 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.547 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.547 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0,3", 2024-03-11T21:03:48.547 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.609473+0000", 2024-03-11T21:03:48.547 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5156999999999997e-05 2024-03-11T21:03:48.547 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.547 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.547 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-03-11T21:03:48.547 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.614603+0000", 2024-03-11T21:03:48.547 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0051302029999999998 2024-03-11T21:03:48.547 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.547 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.547 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:03:48.548 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.960199+0000", 2024-03-11T21:03:48.548 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.345596181 2024-03-11T21:03:48.548 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.548 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.548 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:03:48.548 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.961574+0000", 2024-03-11T21:03:48.548 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001374482 2024-03-11T21:03:48.548 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.548 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.548 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:03:48.548 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.961739+0000", 2024-03-11T21:03:48.548 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016488599999999999 2024-03-11T21:03:48.548 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.548 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.549 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:03:48.549 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.962032+0000", 2024-03-11T21:03:48.549 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00029343400000000002 2024-03-11T21:03:48.549 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:03:48.549 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:03:48.549 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:03:48.549 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.549 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.549 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.24629.0:215025 7.d 7:bf2fa026:::benchmark_data_smithi023_67959_object104784:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-03-11T21:03:48.549 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T20:58:26.602429+0000", 2024-03-11T21:03:48.549 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 321.89525333199998, 2024-03-11T21:03:48.549 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.36044601399999998, 2024-03-11T21:03:48.549 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:03:48.549 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:03:48.550 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-03-11T21:03:48.550 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.24629", 2024-03-11T21:03:48.550 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.23:0/959732609", 2024-03-11T21:03:48.550 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 215025 2024-03-11T21:03:48.550 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.550 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:03:48.550 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.550 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:03:48.550 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.602429+0000", 2024-03-11T21:03:48.550 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:03:48.550 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.550 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.550 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:03:48.550 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.602429+0000", 2024-03-11T21:03:48.551 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:03:48.551 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.551 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.551 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:03:48.551 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.602430+0000", 2024-03-11T21:03:48.551 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.01e-07 2024-03-11T21:03:48.551 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.551 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.551 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:03:48.551 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.602433+0000", 2024-03-11T21:03:48.551 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9299999999999999e-06 2024-03-11T21:03:48.551 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.551 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.551 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:03:48.552 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.602439+0000", 2024-03-11T21:03:48.552 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1279999999999996e-06 2024-03-11T21:03:48.552 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.552 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.552 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:03:48.552 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.602444+0000", 2024-03-11T21:03:48.552 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.259e-06 2024-03-11T21:03:48.552 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.552 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.552 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:03:48.552 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.612231+0000", 2024-03-11T21:03:48.552 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0097864609999999998 2024-03-11T21:03:48.552 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.552 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.552 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:03:48.553 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.612305+0000", 2024-03-11T21:03:48.553 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4077000000000002e-05 2024-03-11T21:03:48.553 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.553 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.553 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0,3", 2024-03-11T21:03:48.553 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.612374+0000", 2024-03-11T21:03:48.553 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9622999999999994e-05 2024-03-11T21:03:48.553 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.553 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.553 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-03-11T21:03:48.553 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.614713+0000", 2024-03-11T21:03:48.553 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0023391010000000001 2024-03-11T21:03:48.553 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.553 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.554 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:03:48.554 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.961428+0000", 2024-03-11T21:03:48.554 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.34671495000000002 2024-03-11T21:03:48.554 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.554 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.554 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:03:48.554 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.962181+0000", 2024-03-11T21:03:48.554 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00075288799999999995 2024-03-11T21:03:48.554 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.554 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.554 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:03:48.554 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.962588+0000", 2024-03-11T21:03:48.554 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00040662800000000002 2024-03-11T21:03:48.555 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.555 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.555 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:03:48.555 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.962875+0000", 2024-03-11T21:03:48.555 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00028756799999999998 2024-03-11T21:03:48.555 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:03:48.555 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:03:48.555 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:03:48.555 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.555 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.555 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.24629.0:215032 7.7 7:e538ec21:::benchmark_data_smithi023_67959_object104791:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-03-11T21:03:48.555 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T20:58:26.602451+0000", 2024-03-11T21:03:48.555 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 321.895232104, 2024-03-11T21:03:48.555 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.35791859199999998, 2024-03-11T21:03:48.556 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:03:48.556 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:03:48.556 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-03-11T21:03:48.556 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.24629", 2024-03-11T21:03:48.556 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.23:0/959732609", 2024-03-11T21:03:48.556 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 215032 2024-03-11T21:03:48.556 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.556 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:03:48.556 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.556 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:03:48.556 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.602451+0000", 2024-03-11T21:03:48.556 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:03:48.556 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.557 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.557 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:03:48.557 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.602451+0000", 2024-03-11T21:03:48.557 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:03:48.557 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.557 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.557 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:03:48.557 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.602451+0000", 2024-03-11T21:03:48.557 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5400000000000001e-07 2024-03-11T21:03:48.557 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.557 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.557 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:03:48.557 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.602454+0000", 2024-03-11T21:03:48.557 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4509999999999999e-06 2024-03-11T21:03:48.558 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.558 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.558 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:03:48.558 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.602460+0000", 2024-03-11T21:03:48.558 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0639999999999999e-06 2024-03-11T21:03:48.558 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.558 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.558 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:03:48.558 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.602465+0000", 2024-03-11T21:03:48.558 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.485e-06 2024-03-11T21:03:48.558 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.558 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.558 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:03:48.559 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.602951+0000", 2024-03-11T21:03:48.559 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00048624999999999997 2024-03-11T21:03:48.559 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.559 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.559 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:03:48.559 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.603020+0000", 2024-03-11T21:03:48.559 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8573999999999998e-05 2024-03-11T21:03:48.559 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.559 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.559 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0,3", 2024-03-11T21:03:48.559 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.603085+0000", 2024-03-11T21:03:48.559 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4653000000000006e-05 2024-03-11T21:03:48.559 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.559 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.560 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-03-11T21:03:48.560 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.617633+0000", 2024-03-11T21:03:48.560 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014547988 2024-03-11T21:03:48.560 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.560 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.560 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:03:48.560 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.617676+0000", 2024-03-11T21:03:48.560 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3139999999999997e-05 2024-03-11T21:03:48.560 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.560 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.560 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:03:48.560 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.960099+0000", 2024-03-11T21:03:48.560 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.34242339799999999 2024-03-11T21:03:48.560 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.560 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.561 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:03:48.561 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.960247+0000", 2024-03-11T21:03:48.561 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014738300000000001 2024-03-11T21:03:48.561 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.561 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.561 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:03:48.561 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.960369+0000", 2024-03-11T21:03:48.561 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000122652 2024-03-11T21:03:48.561 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:03:48.561 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:03:48.561 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:03:48.561 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.561 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.561 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.24629.0:215033 7.7 7:e351a56b:::benchmark_data_smithi023_67959_object104792:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-03-11T21:03:48.562 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T20:58:26.602472+0000", 2024-03-11T21:03:48.562 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 321.89521040900001, 2024-03-11T21:03:48.562 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.35858107700000003, 2024-03-11T21:03:48.562 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:03:48.562 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:03:48.562 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-03-11T21:03:48.562 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.24629", 2024-03-11T21:03:48.562 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.23:0/959732609", 2024-03-11T21:03:48.562 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 215033 2024-03-11T21:03:48.562 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.562 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:03:48.562 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.562 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:03:48.562 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.602472+0000", 2024-03-11T21:03:48.563 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:03:48.563 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.563 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.563 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:03:48.563 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.602472+0000", 2024-03-11T21:03:48.563 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:03:48.563 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.563 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.563 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:03:48.563 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.602473+0000", 2024-03-11T21:03:48.563 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.6399999999999998e-07 2024-03-11T21:03:48.563 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.563 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.563 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:03:48.564 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.602475+0000", 2024-03-11T21:03:48.564 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3410000000000001e-06 2024-03-11T21:03:48.564 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.564 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.564 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:03:48.564 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.602481+0000", 2024-03-11T21:03:48.564 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9499999999999998e-06 2024-03-11T21:03:48.564 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.564 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.564 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:03:48.564 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.602488+0000", 2024-03-11T21:03:48.564 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6030000000000001e-06 2024-03-11T21:03:48.564 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.564 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.564 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:03:48.565 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.606022+0000", 2024-03-11T21:03:48.565 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0035344030000000002 2024-03-11T21:03:48.565 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.565 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.565 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:03:48.565 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.606138+0000", 2024-03-11T21:03:48.565 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011636199999999999 2024-03-11T21:03:48.565 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.565 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.565 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0,3", 2024-03-11T21:03:48.565 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.606257+0000", 2024-03-11T21:03:48.565 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000118363 2024-03-11T21:03:48.565 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.565 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.566 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-03-11T21:03:48.566 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.617640+0000", 2024-03-11T21:03:48.566 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.011383702000000001 2024-03-11T21:03:48.566 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.566 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.566 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:03:48.566 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.619349+0000", 2024-03-11T21:03:48.566 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001708525 2024-03-11T21:03:48.566 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.566 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.566 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:03:48.566 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.960799+0000", 2024-03-11T21:03:48.566 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.34145019100000001 2024-03-11T21:03:48.566 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.567 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.567 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:03:48.567 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.960983+0000", 2024-03-11T21:03:48.567 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000183776 2024-03-11T21:03:48.567 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.567 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.567 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:03:48.567 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.961053+0000", 2024-03-11T21:03:48.567 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0496999999999999e-05 2024-03-11T21:03:48.567 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:03:48.567 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:03:48.567 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:03:48.567 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.567 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.568 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.24629.0:215038 7.7 7:ee77d68c:::benchmark_data_smithi023_67959_object104797:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-03-11T21:03:48.568 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T20:58:26.602494+0000", 2024-03-11T21:03:48.568 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 321.89518849299998, 2024-03-11T21:03:48.568 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.35876947399999998, 2024-03-11T21:03:48.568 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:03:48.568 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:03:48.568 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-03-11T21:03:48.568 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.24629", 2024-03-11T21:03:48.568 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.23:0/959732609", 2024-03-11T21:03:48.568 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 215038 2024-03-11T21:03:48.568 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.568 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:03:48.568 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.568 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:03:48.568 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.602494+0000", 2024-03-11T21:03:48.569 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:03:48.569 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.569 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.569 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:03:48.569 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.602494+0000", 2024-03-11T21:03:48.569 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:03:48.569 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.569 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.569 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:03:48.569 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.602495+0000", 2024-03-11T21:03:48.569 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.6199999999999999e-07 2024-03-11T21:03:48.569 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.569 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.569 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:03:48.570 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.602497+0000", 2024-03-11T21:03:48.570 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6060000000000001e-06 2024-03-11T21:03:48.570 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.570 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.570 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:03:48.570 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.602503+0000", 2024-03-11T21:03:48.570 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8850000000000004e-06 2024-03-11T21:03:48.570 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.570 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.570 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:03:48.570 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.602508+0000", 2024-03-11T21:03:48.570 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0649999999999998e-06 2024-03-11T21:03:48.570 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.570 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.571 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:03:48.571 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.610450+0000", 2024-03-11T21:03:48.571 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.007941989 2024-03-11T21:03:48.571 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.571 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.571 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:03:48.571 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.610562+0000", 2024-03-11T21:03:48.571 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011149 2024-03-11T21:03:48.571 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.571 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.571 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0,3", 2024-03-11T21:03:48.571 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.610696+0000", 2024-03-11T21:03:48.571 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013445800000000001 2024-03-11T21:03:48.571 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.571 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.572 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-03-11T21:03:48.572 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.617647+0000", 2024-03-11T21:03:48.572 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0069511369999999996 2024-03-11T21:03:48.572 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.572 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.572 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:03:48.572 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.619377+0000", 2024-03-11T21:03:48.572 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001729242 2024-03-11T21:03:48.572 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.572 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.572 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:03:48.572 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.961131+0000", 2024-03-11T21:03:48.572 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.34175470699999999 2024-03-11T21:03:48.572 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.572 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.573 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:03:48.573 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.961208+0000", 2024-03-11T21:03:48.573 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6357000000000001e-05 2024-03-11T21:03:48.573 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.573 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.573 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:03:48.573 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.961264+0000", 2024-03-11T21:03:48.573 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6175999999999999e-05 2024-03-11T21:03:48.573 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:03:48.573 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:03:48.573 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:03:48.573 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.573 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.573 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.24629.0:215039 7.7 7:e4676876:::benchmark_data_smithi023_67959_object104798:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-03-11T21:03:48.574 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T20:58:26.602780+0000", 2024-03-11T21:03:48.574 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 321.89490253999998, 2024-03-11T21:03:48.574 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.35866579100000001, 2024-03-11T21:03:48.574 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:03:48.574 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:03:48.574 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-03-11T21:03:48.574 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.24629", 2024-03-11T21:03:48.574 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.23:0/959732609", 2024-03-11T21:03:48.574 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 215039 2024-03-11T21:03:48.574 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.574 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:03:48.574 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.574 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:03:48.574 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.602780+0000", 2024-03-11T21:03:48.575 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:03:48.575 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.575 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.575 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:03:48.575 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.602780+0000", 2024-03-11T21:03:48.575 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:03:48.575 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.575 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.575 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:03:48.575 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.602781+0000", 2024-03-11T21:03:48.575 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.09e-07 2024-03-11T21:03:48.575 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.575 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.575 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:03:48.575 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.602786+0000", 2024-03-11T21:03:48.576 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.69e-06 2024-03-11T21:03:48.576 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.576 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.576 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:03:48.576 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.602794+0000", 2024-03-11T21:03:48.576 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.0450000000000001e-06 2024-03-11T21:03:48.576 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.576 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.576 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:03:48.576 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.602800+0000", 2024-03-11T21:03:48.576 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1959999999999996e-06 2024-03-11T21:03:48.576 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.576 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.576 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:03:48.577 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.613681+0000", 2024-03-11T21:03:48.577 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010880578 2024-03-11T21:03:48.577 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.577 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.577 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:03:48.577 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.613781+0000", 2024-03-11T21:03:48.577 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000100484 2024-03-11T21:03:48.577 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.577 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.577 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0,3", 2024-03-11T21:03:48.577 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.613889+0000", 2024-03-11T21:03:48.577 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000107618 2024-03-11T21:03:48.577 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.577 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.577 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-03-11T21:03:48.578 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.617653+0000", 2024-03-11T21:03:48.578 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0037645769999999999 2024-03-11T21:03:48.578 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.578 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.578 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:03:48.578 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.960524+0000", 2024-03-11T21:03:48.578 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.34287065 2024-03-11T21:03:48.578 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.578 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.578 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:03:48.578 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.961331+0000", 2024-03-11T21:03:48.578 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000807014 2024-03-11T21:03:48.578 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.578 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.579 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:03:48.579 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.961395+0000", 2024-03-11T21:03:48.579 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3666999999999999e-05 2024-03-11T21:03:48.579 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.579 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.579 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:03:48.579 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.961446+0000", 2024-03-11T21:03:48.579 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1363000000000003e-05 2024-03-11T21:03:48.579 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:03:48.579 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:03:48.579 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:03:48.579 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.579 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.579 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.24629.0:215046 7.7 7:e0298817:::benchmark_data_smithi023_67959_object104805:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-03-11T21:03:48.580 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T20:58:26.609466+0000", 2024-03-11T21:03:48.580 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 321.88821676499998, 2024-03-11T21:03:48.580 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.35223008700000003, 2024-03-11T21:03:48.580 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:03:48.580 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:03:48.580 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-03-11T21:03:48.580 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.24629", 2024-03-11T21:03:48.580 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.23:0/959732609", 2024-03-11T21:03:48.580 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 215046 2024-03-11T21:03:48.580 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.580 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:03:48.580 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.580 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:03:48.580 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.609466+0000", 2024-03-11T21:03:48.580 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:03:48.581 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.581 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.581 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:03:48.581 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.609466+0000", 2024-03-11T21:03:48.581 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:03:48.581 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.581 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.581 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:03:48.581 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.609467+0000", 2024-03-11T21:03:48.581 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4610000000000001e-06 2024-03-11T21:03:48.581 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.581 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.581 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:03:48.581 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.609475+0000", 2024-03-11T21:03:48.581 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.187e-06 2024-03-11T21:03:48.582 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.582 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.582 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:03:48.582 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.609488+0000", 2024-03-11T21:03:48.582 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2974e-05 2024-03-11T21:03:48.582 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.582 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.582 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:03:48.582 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.609496+0000", 2024-03-11T21:03:48.582 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1859999999999992e-06 2024-03-11T21:03:48.582 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.582 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.582 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:03:48.582 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.616121+0000", 2024-03-11T21:03:48.583 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.006625377 2024-03-11T21:03:48.583 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.583 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.583 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:03:48.583 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.616187+0000", 2024-03-11T21:03:48.583 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5827999999999994e-05 2024-03-11T21:03:48.583 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.583 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.583 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0,3", 2024-03-11T21:03:48.583 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.616251+0000", 2024-03-11T21:03:48.583 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3737000000000003e-05 2024-03-11T21:03:48.583 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.583 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.583 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-03-11T21:03:48.583 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.617742+0000", 2024-03-11T21:03:48.584 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001490856 2024-03-11T21:03:48.584 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.584 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.584 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:03:48.584 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.960670+0000", 2024-03-11T21:03:48.584 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.34292808000000002 2024-03-11T21:03:48.584 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.584 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.584 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:03:48.584 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.961505+0000", 2024-03-11T21:03:48.584 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00083546200000000003 2024-03-11T21:03:48.584 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.584 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.584 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:03:48.585 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.961639+0000", 2024-03-11T21:03:48.585 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013389700000000001 2024-03-11T21:03:48.585 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.585 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.585 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:03:48.585 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.961696+0000", 2024-03-11T21:03:48.585 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7042e-05 2024-03-11T21:03:48.585 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:03:48.585 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:03:48.585 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:03:48.585 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.585 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.585 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop_reply(client.24629.0:215020 7.d e178/174 ondisk, result = 0)", 2024-03-11T21:03:48.585 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T20:58:26.621080+0000", 2024-03-11T21:03:48.585 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 321.87660240500003, 2024-03-11T21:03:48.586 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.338785106, 2024-03-11T21:03:48.586 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:03:48.586 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-03-11T21:03:48.586 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:03:48.586 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.586 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:03:48.586 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.621080+0000", 2024-03-11T21:03:48.586 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:03:48.586 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.586 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.586 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:03:48.586 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.621080+0000", 2024-03-11T21:03:48.586 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:03:48.587 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.587 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.587 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:03:48.587 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.621081+0000", 2024-03-11T21:03:48.587 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3099999999999998e-07 2024-03-11T21:03:48.587 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.587 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.587 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:03:48.587 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.621085+0000", 2024-03-11T21:03:48.587 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.641e-06 2024-03-11T21:03:48.587 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.587 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.587 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:03:48.587 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.621093+0000", 2024-03-11T21:03:48.588 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9580000000000007e-06 2024-03-11T21:03:48.588 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.588 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.588 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:03:48.588 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.621097+0000", 2024-03-11T21:03:48.588 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5539999999999999e-06 2024-03-11T21:03:48.588 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.588 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.588 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:03:48.588 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.621137+0000", 2024-03-11T21:03:48.588 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0002000000000001e-05 2024-03-11T21:03:48.588 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.588 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.588 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:03:48.589 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.621142+0000", 2024-03-11T21:03:48.589 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4040000000000003e-06 2024-03-11T21:03:48.589 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.589 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.589 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:03:48.589 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.959865+0000", 2024-03-11T21:03:48.589 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.33872301599999999 2024-03-11T21:03:48.589 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:03:48.589 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:03:48.589 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:03:48.589 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.589 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.589 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop_reply(client.24629.0:215021 7.d e178/174 ondisk, result = 0)", 2024-03-11T21:03:48.589 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T20:58:26.621121+0000", 2024-03-11T21:03:48.589 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 321.87656221600002, 2024-03-11T21:03:48.590 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.33891284999999999, 2024-03-11T21:03:48.590 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:03:48.590 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-03-11T21:03:48.590 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:03:48.590 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.590 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:03:48.590 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.621121+0000", 2024-03-11T21:03:48.590 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:03:48.590 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.590 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.590 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:03:48.590 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.621121+0000", 2024-03-11T21:03:48.590 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:03:48.591 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.591 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.591 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:03:48.591 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.621121+0000", 2024-03-11T21:03:48.591 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6899999999999999e-07 2024-03-11T21:03:48.591 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.591 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.591 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:03:48.591 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.621123+0000", 2024-03-11T21:03:48.591 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1320000000000001e-06 2024-03-11T21:03:48.591 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.591 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.591 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:03:48.591 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.621133+0000", 2024-03-11T21:03:48.592 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0421000000000001e-05 2024-03-11T21:03:48.592 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.592 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.592 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:03:48.592 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.621138+0000", 2024-03-11T21:03:48.592 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5079999999999999e-06 2024-03-11T21:03:48.592 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.592 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.592 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:03:48.592 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.959947+0000", 2024-03-11T21:03:48.592 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.338808832 2024-03-11T21:03:48.592 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.592 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.592 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:03:48.592 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.959974+0000", 2024-03-11T21:03:48.593 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7234999999999999e-05 2024-03-11T21:03:48.593 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.593 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.593 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:03:48.593 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.960033+0000", 2024-03-11T21:03:48.593 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9453000000000002e-05 2024-03-11T21:03:48.593 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:03:48.593 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:03:48.593 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:03:48.593 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.593 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.593 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop_reply(client.24629.0:215032 7.7 e178/121 ondisk, result = 0)", 2024-03-11T21:03:48.593 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T20:58:26.621583+0000", 2024-03-11T21:03:48.593 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 321.876099807, 2024-03-11T21:03:48.593 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.33880317700000001, 2024-03-11T21:03:48.594 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:03:48.594 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-03-11T21:03:48.594 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:03:48.594 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.594 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:03:48.594 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.621583+0000", 2024-03-11T21:03:48.594 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:03:48.594 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.594 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.594 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:03:48.594 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.621583+0000", 2024-03-11T21:03:48.594 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:03:48.594 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.594 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.595 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:03:48.595 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.621584+0000", 2024-03-11T21:03:48.595 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2600000000000002e-07 2024-03-11T21:03:48.595 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.595 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.595 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:03:48.595 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.621587+0000", 2024-03-11T21:03:48.595 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8469999999999998e-06 2024-03-11T21:03:48.595 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.595 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.595 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:03:48.595 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.959815+0000", 2024-03-11T21:03:48.595 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.33822709899999998 2024-03-11T21:03:48.595 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.595 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.596 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:03:48.596 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.959838+0000", 2024-03-11T21:03:48.596 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3076000000000001e-05 2024-03-11T21:03:48.596 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.596 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.596 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:03:48.596 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.960063+0000", 2024-03-11T21:03:48.596 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000225672 2024-03-11T21:03:48.596 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.596 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.596 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:03:48.596 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.960083+0000", 2024-03-11T21:03:48.596 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9769999999999999e-05 2024-03-11T21:03:48.596 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.596 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.597 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:03:48.597 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.960386+0000", 2024-03-11T21:03:48.597 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00030308700000000001 2024-03-11T21:03:48.597 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:03:48.597 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:03:48.597 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:03:48.597 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.597 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.597 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.24629.0:215064 7.c 7:344db2c8:::benchmark_data_smithi023_67959_object104823:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-03-11T21:03:48.597 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T20:58:26.632650+0000", 2024-03-11T21:03:48.597 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 321.865033192, 2024-03-11T21:03:48.597 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.34945065800000003, 2024-03-11T21:03:48.597 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:03:48.597 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:03:48.598 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-03-11T21:03:48.598 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.24629", 2024-03-11T21:03:48.598 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.23:0/959732609", 2024-03-11T21:03:48.598 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 215064 2024-03-11T21:03:48.598 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.598 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:03:48.598 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.598 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:03:48.598 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.632650+0000", 2024-03-11T21:03:48.598 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:03:48.598 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.598 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.598 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:03:48.598 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.632650+0000", 2024-03-11T21:03:48.599 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:03:48.599 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.599 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.599 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:03:48.599 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.632651+0000", 2024-03-11T21:03:48.599 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4309999999999999e-06 2024-03-11T21:03:48.599 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.599 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.599 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:03:48.599 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.632657+0000", 2024-03-11T21:03:48.599 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1759999999999998e-06 2024-03-11T21:03:48.599 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.599 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.599 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:03:48.599 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.959804+0000", 2024-03-11T21:03:48.600 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.32714649699999998 2024-03-11T21:03:48.600 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.600 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.600 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:03:48.600 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.959840+0000", 2024-03-11T21:03:48.600 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.6532000000000002e-05 2024-03-11T21:03:48.600 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.600 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.600 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:03:48.600 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.960056+0000", 2024-03-11T21:03:48.600 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00021541600000000001 2024-03-11T21:03:48.600 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.600 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.600 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:03:48.601 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.960394+0000", 2024-03-11T21:03:48.601 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00033841299999999997 2024-03-11T21:03:48.601 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.601 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.601 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3,7", 2024-03-11T21:03:48.601 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.960669+0000", 2024-03-11T21:03:48.601 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00027524599999999999 2024-03-11T21:03:48.601 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.601 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.601 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-03-11T21:03:48.601 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.975084+0000", 2024-03-11T21:03:48.601 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014414938 2024-03-11T21:03:48.601 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.602 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.602 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:03:48.602 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.975205+0000", 2024-03-11T21:03:48.602 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000120701 2024-03-11T21:03:48.602 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.602 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.602 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:03:48.602 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.981867+0000", 2024-03-11T21:03:48.602 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0066616640000000003 2024-03-11T21:03:48.602 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.602 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.602 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:03:48.602 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.982035+0000", 2024-03-11T21:03:48.602 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016854199999999999 2024-03-11T21:03:48.603 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.603 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.603 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:03:48.603 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:26.982100+0000", 2024-03-11T21:03:48.603 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5102000000000001e-05 2024-03-11T21:03:48.603 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:03:48.603 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:03:48.603 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:03:48.603 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.603 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.603 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.24281.0:1927 4.e 4.f8c99aee (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-03-11T21:03:48.603 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T20:58:53.175134+0000", 2024-03-11T21:03:48.603 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 295.322548756, 2024-03-11T21:03:48.603 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.41594428500000002, 2024-03-11T21:03:48.604 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:03:48.604 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "reached pg", 2024-03-11T21:03:48.604 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-03-11T21:03:48.604 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.24281", 2024-03-11T21:03:48.604 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.23:0/2185274774", 2024-03-11T21:03:48.604 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1927 2024-03-11T21:03:48.604 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.604 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:03:48.604 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.604 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:03:48.604 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:53.175134+0000", 2024-03-11T21:03:48.604 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:03:48.604 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.604 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.604 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:03:48.605 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:53.175134+0000", 2024-03-11T21:03:48.605 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:03:48.605 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.605 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.605 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:03:48.605 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:53.175138+0000", 2024-03-11T21:03:48.605 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5360000000000001e-06 2024-03-11T21:03:48.605 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.605 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.605 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:03:48.606 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:53.175154+0000", 2024-03-11T21:03:48.606 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6565000000000001e-05 2024-03-11T21:03:48.606 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.606 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.606 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:03:48.606 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:53.175188+0000", 2024-03-11T21:03:48.606 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.434e-05 2024-03-11T21:03:48.606 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.606 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.606 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:03:48.606 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:53.175204+0000", 2024-03-11T21:03:48.606 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5882000000000001e-05 2024-03-11T21:03:48.606 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.606 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.607 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:03:48.607 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:53.175317+0000", 2024-03-11T21:03:48.607 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000112821 2024-03-11T21:03:48.607 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.607 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.607 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-03-11T21:03:48.607 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:53.175331+0000", 2024-03-11T21:03:48.607 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4022999999999999e-05 2024-03-11T21:03:48.607 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.607 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.607 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:03:48.607 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:53.591044+0000", 2024-03-11T21:03:48.607 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.41571322900000002 2024-03-11T21:03:48.607 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.607 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.608 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:03:48.608 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:58:53.591078+0000", 2024-03-11T21:03:48.608 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3889e-05 2024-03-11T21:03:48.608 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:03:48.608 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:03:48.608 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:03:48.608 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.608 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.608 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.15081.0:63953 9.0 e271/251 9:00745ff8:::benchmark_data_smithi023_106206_object63952:head v 271'4037, mlcod=271'4035)", 2024-03-11T21:03:48.608 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T21:03:42.839252+0000", 2024-03-11T21:03:48.608 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 5.6584305959999996, 2024-03-11T21:03:48.608 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.41234332099999998, 2024-03-11T21:03:48.608 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:03:48.608 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:03:48.608 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:03:48.609 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.609 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:03:48.609 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:42.839252+0000", 2024-03-11T21:03:48.609 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:03:48.609 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.609 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.609 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:03:48.609 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:42.839252+0000", 2024-03-11T21:03:48.609 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:03:48.609 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.609 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.609 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:03:48.609 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:42.839253+0000", 2024-03-11T21:03:48.609 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2500000000000004e-07 2024-03-11T21:03:48.609 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.610 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.610 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:03:48.610 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:42.839387+0000", 2024-03-11T21:03:48.610 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013366800000000001 2024-03-11T21:03:48.610 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.610 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.610 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:03:48.610 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:43.237771+0000", 2024-03-11T21:03:48.610 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.39838409299999999 2024-03-11T21:03:48.610 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.610 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.610 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:03:48.610 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:43.237804+0000", 2024-03-11T21:03:48.610 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3522000000000002e-05 2024-03-11T21:03:48.610 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.611 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.611 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:03:48.611 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:43.238046+0000", 2024-03-11T21:03:48.611 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00024209399999999999 2024-03-11T21:03:48.611 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.611 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.611 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:03:48.611 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:43.238105+0000", 2024-03-11T21:03:48.611 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8962999999999998e-05 2024-03-11T21:03:48.611 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.611 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.611 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:03:48.611 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:43.251390+0000", 2024-03-11T21:03:48.611 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.013284569 2024-03-11T21:03:48.612 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.612 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.612 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:03:48.612 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:43.251596+0000", 2024-03-11T21:03:48.612 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00020548699999999999 2024-03-11T21:03:48.612 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:03:48.612 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:03:48.612 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:03:48.612 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.612 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.612 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.15081.0:63967 9.1 e271/251 9:880b69fc:::benchmark_data_smithi023_106206_object63966:head v 271'4067, mlcod=271'4065)", 2024-03-11T21:03:48.612 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T21:03:42.847563+0000", 2024-03-11T21:03:48.612 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 5.6501193750000001, 2024-03-11T21:03:48.612 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.405977492, 2024-03-11T21:03:48.612 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:03:48.613 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:03:48.613 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:03:48.613 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.613 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:03:48.613 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:42.847563+0000", 2024-03-11T21:03:48.613 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:03:48.613 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.613 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.613 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:03:48.613 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:42.847563+0000", 2024-03-11T21:03:48.613 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:03:48.613 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.613 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.613 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:03:48.613 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:42.847564+0000", 2024-03-11T21:03:48.614 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5700000000000001e-07 2024-03-11T21:03:48.614 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.614 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.614 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:03:48.614 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:42.847618+0000", 2024-03-11T21:03:48.614 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3488e-05 2024-03-11T21:03:48.614 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.614 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.614 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:03:48.614 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:43.237767+0000", 2024-03-11T21:03:48.614 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.390149307 2024-03-11T21:03:48.614 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.614 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.614 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:03:48.615 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:43.237806+0000", 2024-03-11T21:03:48.615 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9022e-05 2024-03-11T21:03:48.615 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.615 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.615 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:03:48.615 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:43.237988+0000", 2024-03-11T21:03:48.615 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000181779 2024-03-11T21:03:48.615 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.615 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.615 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:03:48.615 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:43.238050+0000", 2024-03-11T21:03:48.615 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2131999999999994e-05 2024-03-11T21:03:48.615 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.615 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.615 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:03:48.616 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:43.253360+0000", 2024-03-11T21:03:48.616 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.015309924000000001 2024-03-11T21:03:48.616 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:03:48.616 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:03:48.616 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:03:48.616 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:43.253541+0000", 2024-03-11T21:03:48.616 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018098300000000001 2024-03-11T21:03:48.616 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:03:48.616 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:03:48.616 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:03:48.616 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:03:48.616 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:03:48.616 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-03-11T21:03:48.617 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:48.617 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:03:48 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:03:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:03:48.637 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:48.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:48.639 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:03:48.740 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:03:48.762 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:03:48.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:48.764 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:03:48.779 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:48 smithi146 bash[23809]: cluster 2024-03-11T21:03:47.208645+0000 osd.5 (osd.5) 243 : cluster 0 2.3 scrub starts 2024-03-11T21:03:48.779 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:48 smithi146 bash[23809]: cluster 2024-03-11T21:03:47.212714+0000 osd.5 (osd.5) 244 : cluster 0 2.3 scrub ok 2024-03-11T21:03:48.780 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:48 smithi146 bash[23809]: cluster 2024-03-11T21:03:47.986704+0000 osd.7 (osd.7) 277 : cluster 0 3.15 scrub starts 2024-03-11T21:03:48.780 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:48 smithi146 bash[23809]: cluster 2024-03-11T21:03:47.989403+0000 osd.7 (osd.7) 278 : cluster 0 3.15 scrub ok 2024-03-11T21:03:48.780 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:48 smithi146 bash[23809]: cluster 2024-03-11T21:03:48.162456+0000 mgr.y (mgr.24380) 482 : cluster 0 pgmap v540: 148 pgs: 148 active+clean; 4.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.3 KiB/s rd, 58 MiB/s wr, 924 op/s 2024-03-11T21:03:48.865 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:48.899 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:48.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:48.903 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:03:49.004 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:49.014 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:48 smithi023 bash[21372]: cluster 2024-03-11T21:03:47.208645+0000 osd.5 (osd.5) 243 : cluster 0 2.3 scrub starts 2024-03-11T21:03:49.014 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:48 smithi023 bash[21372]: cluster 2024-03-11T21:03:47.212714+0000 osd.5 (osd.5) 244 : cluster 0 2.3 scrub ok 2024-03-11T21:03:49.015 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:48 smithi023 bash[21372]: cluster 2024-03-11T21:03:47.986704+0000 osd.7 (osd.7) 277 : cluster 0 3.15 scrub starts 2024-03-11T21:03:49.015 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:48 smithi023 bash[21372]: cluster 2024-03-11T21:03:47.989403+0000 osd.7 (osd.7) 278 : cluster 0 3.15 scrub ok 2024-03-11T21:03:49.015 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:48 smithi023 bash[21372]: cluster 2024-03-11T21:03:48.162456+0000 mgr.y (mgr.24380) 482 : cluster 0 pgmap v540: 148 pgs: 148 active+clean; 4.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.3 KiB/s rd, 58 MiB/s wr, 924 op/s 2024-03-11T21:03:49.015 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:48 smithi023 bash[29146]: cluster 2024-03-11T21:03:47.208645+0000 osd.5 (osd.5) 243 : cluster 0 2.3 scrub starts 2024-03-11T21:03:49.015 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:48 smithi023 bash[29146]: cluster 2024-03-11T21:03:47.212714+0000 osd.5 (osd.5) 244 : cluster 0 2.3 scrub ok 2024-03-11T21:03:49.015 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:48 smithi023 bash[29146]: cluster 2024-03-11T21:03:47.986704+0000 osd.7 (osd.7) 277 : cluster 0 3.15 scrub starts 2024-03-11T21:03:49.015 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:48 smithi023 bash[29146]: cluster 2024-03-11T21:03:47.989403+0000 osd.7 (osd.7) 278 : cluster 0 3.15 scrub ok 2024-03-11T21:03:49.015 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:48 smithi023 bash[29146]: cluster 2024-03-11T21:03:48.162456+0000 mgr.y (mgr.24380) 482 : cluster 0 pgmap v540: 148 pgs: 148 active+clean; 4.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.3 KiB/s rd, 58 MiB/s wr, 924 op/s 2024-03-11T21:03:49.032 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:49.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:49.034 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:03:49.135 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:49.164 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:49.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:49.167 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:03:49.267 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:49.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:49.296 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:49.296 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:03:49.397 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:49.428 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:49.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:49.432 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:03:49.456 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 47 16 70064 70048 93.1392 50.125 0.0142983 0.0107314 2024-03-11T21:03:49.532 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:03:49.561 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:03:49.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:49.565 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:03:49.666 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:49.684 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:49.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:49.686 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:03:49.787 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:03:49.797 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-11T21:03:49.815 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:03:49.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:49.819 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:03:49.919 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:49.932 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:49 smithi146 bash[23809]: cluster 2024-03-11T21:03:48.230944+0000 osd.5 (osd.5) 245 : cluster 0 4.1d scrub starts 2024-03-11T21:03:49.932 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:49 smithi146 bash[23809]: cluster 2024-03-11T21:03:48.239001+0000 osd.5 (osd.5) 246 : cluster 0 4.1d scrub ok 2024-03-11T21:03:49.932 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:49 smithi146 bash[23809]: cluster 2024-03-11T21:03:48.780096+0000 osd.2 (osd.2) 181 : cluster 0 2.0 scrub starts 2024-03-11T21:03:49.932 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:49 smithi146 bash[23809]: cluster 2024-03-11T21:03:48.784634+0000 osd.2 (osd.2) 182 : cluster 0 2.0 scrub ok 2024-03-11T21:03:49.932 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:49 smithi146 bash[23809]: cluster 2024-03-11T21:03:48.962180+0000 osd.7 (osd.7) 279 : cluster 0 3.f scrub starts 2024-03-11T21:03:49.932 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:49 smithi146 bash[23809]: cluster 2024-03-11T21:03:48.964754+0000 osd.7 (osd.7) 280 : cluster 0 3.f scrub ok 2024-03-11T21:03:49.955 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:49.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:49.961 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:03:50.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:49 smithi023 bash[21372]: cluster 2024-03-11T21:03:48.230944+0000 osd.5 (osd.5) 245 : cluster 0 4.1d scrub starts 2024-03-11T21:03:50.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:49 smithi023 bash[21372]: cluster 2024-03-11T21:03:48.239001+0000 osd.5 (osd.5) 246 : cluster 0 4.1d scrub ok 2024-03-11T21:03:50.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:49 smithi023 bash[21372]: cluster 2024-03-11T21:03:48.780096+0000 osd.2 (osd.2) 181 : cluster 0 2.0 scrub starts 2024-03-11T21:03:50.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:49 smithi023 bash[21372]: cluster 2024-03-11T21:03:48.784634+0000 osd.2 (osd.2) 182 : cluster 0 2.0 scrub ok 2024-03-11T21:03:50.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:49 smithi023 bash[21372]: cluster 2024-03-11T21:03:48.962180+0000 osd.7 (osd.7) 279 : cluster 0 3.f scrub starts 2024-03-11T21:03:50.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:49 smithi023 bash[21372]: cluster 2024-03-11T21:03:48.964754+0000 osd.7 (osd.7) 280 : cluster 0 3.f scrub ok 2024-03-11T21:03:50.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:49 smithi023 bash[29146]: cluster 2024-03-11T21:03:48.230944+0000 osd.5 (osd.5) 245 : cluster 0 4.1d scrub starts 2024-03-11T21:03:50.036 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:49 smithi023 bash[29146]: cluster 2024-03-11T21:03:48.239001+0000 osd.5 (osd.5) 246 : cluster 0 4.1d scrub ok 2024-03-11T21:03:50.036 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:49 smithi023 bash[29146]: cluster 2024-03-11T21:03:48.780096+0000 osd.2 (osd.2) 181 : cluster 0 2.0 scrub starts 2024-03-11T21:03:50.036 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:49 smithi023 bash[29146]: cluster 2024-03-11T21:03:48.784634+0000 osd.2 (osd.2) 182 : cluster 0 2.0 scrub ok 2024-03-11T21:03:50.036 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:49 smithi023 bash[29146]: cluster 2024-03-11T21:03:48.962180+0000 osd.7 (osd.7) 279 : cluster 0 3.f scrub starts 2024-03-11T21:03:50.036 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:49 smithi023 bash[29146]: cluster 2024-03-11T21:03:48.964754+0000 osd.7 (osd.7) 280 : cluster 0 3.f scrub ok 2024-03-11T21:03:50.061 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:03:50.104 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:03:50.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:50.110 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:03:50.211 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:03:50.240 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:03:50.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:50.245 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:03:50.345 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:03:50.369 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:03:50.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:50.372 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:03:50.456 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 48 16 71130 71114 92.5867 66.625 0.0126199 0.0107954 2024-03-11T21:03:50.473 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:03:50.499 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:03:50.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:50.503 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:03:50.604 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:03:50.635 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:03:50.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:50.637 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:03:50.737 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:03:50.792 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:03:50.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:50.797 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:03:50.870 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:50 smithi023 bash[21372]: cluster 2024-03-11T21:03:50.163151+0000 mgr.y (mgr.24380) 483 : cluster 0 pgmap v541: 148 pgs: 148 active+clean; 4.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 571 B/s rd, 60 MiB/s wr, 960 op/s 2024-03-11T21:03:50.870 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:50 smithi023 bash[21372]: cluster 2024-03-11T21:03:50.250040+0000 osd.6 (osd.6) 200 : cluster 0 3.10 scrub starts 2024-03-11T21:03:50.871 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:50 smithi023 bash[21372]: cluster 2024-03-11T21:03:50.252588+0000 osd.6 (osd.6) 201 : cluster 0 3.10 scrub ok 2024-03-11T21:03:50.871 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:50 smithi023 bash[21372]: audit 2024-03-11T21:03:50.266055+0000 mon.a (mon.0) 1481 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:03:50.871 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:50 smithi023 bash[29146]: cluster 2024-03-11T21:03:50.163151+0000 mgr.y (mgr.24380) 483 : cluster 0 pgmap v541: 148 pgs: 148 active+clean; 4.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 571 B/s rd, 60 MiB/s wr, 960 op/s 2024-03-11T21:03:50.871 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:50 smithi023 bash[29146]: cluster 2024-03-11T21:03:50.250040+0000 osd.6 (osd.6) 200 : cluster 0 3.10 scrub starts 2024-03-11T21:03:50.871 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:50 smithi023 bash[29146]: cluster 2024-03-11T21:03:50.252588+0000 osd.6 (osd.6) 201 : cluster 0 3.10 scrub ok 2024-03-11T21:03:50.871 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:50 smithi023 bash[29146]: audit 2024-03-11T21:03:50.266055+0000 mon.a (mon.0) 1481 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:03:50.898 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:03:50.920 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:03:50.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:50.923 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:03:51.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:50 smithi146 bash[23809]: cluster 2024-03-11T21:03:50.163151+0000 mgr.y (mgr.24380) 483 : cluster 0 pgmap v541: 148 pgs: 148 active+clean; 4.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 571 B/s rd, 60 MiB/s wr, 960 op/s 2024-03-11T21:03:51.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:50 smithi146 bash[23809]: cluster 2024-03-11T21:03:50.250040+0000 osd.6 (osd.6) 200 : cluster 0 3.10 scrub starts 2024-03-11T21:03:51.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:50 smithi146 bash[23809]: cluster 2024-03-11T21:03:50.252588+0000 osd.6 (osd.6) 201 : cluster 0 3.10 scrub ok 2024-03-11T21:03:51.013 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:50 smithi146 bash[23809]: audit 2024-03-11T21:03:50.266055+0000 mon.a (mon.0) 1481 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:03:51.020 INFO:teuthology.orchestra.run.smithi023.stdout:osd.0: {} 2024-03-11T21:03:51.024 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:03:51.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:51.059 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:03:51.059 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:03:51.099 INFO:teuthology.orchestra.run.smithi023.stdout:osd.1: {} 2024-03-11T21:03:51.138 INFO:teuthology.orchestra.run.smithi023.stdout:osd.2: {} 2024-03-11T21:03:51.160 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:03:51.192 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:03:51.192 INFO:teuthology.orchestra.run.smithi023.stdout:osd.3: {} 2024-03-11T21:03:51.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:51.197 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:03:51.246 INFO:teuthology.orchestra.run.smithi023.stdout:osd.4: {} 2024-03-11T21:03:51.283 INFO:teuthology.orchestra.run.smithi023.stdout:osd.5: {} 2024-03-11T21:03:51.298 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:51.325 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:51.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:51.328 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:03:51.335 INFO:teuthology.orchestra.run.smithi023.stdout:osd.6: {} 2024-03-11T21:03:51.381 INFO:teuthology.orchestra.run.smithi023.stdout:osd.7: {} 2024-03-11T21:03:51.391 INFO:teuthology.orchestra.run.smithi023.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:03:51.391 INFO:teuthology.orchestra.run.smithi023.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:03:51.391 INFO:teuthology.orchestra.run.smithi023.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:03:51.391 INFO:teuthology.orchestra.run.smithi023.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:03:51.391 INFO:teuthology.orchestra.run.smithi023.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:03:51.391 INFO:teuthology.orchestra.run.smithi023.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:03:51.391 INFO:teuthology.orchestra.run.smithi023.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:03:51.391 INFO:teuthology.orchestra.run.smithi023.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:03:51.429 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:51.450 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:51.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:51.452 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:03:51.456 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 49 16 72105 72089 91.9407 60.9375 0.00982413 0.0108697 2024-03-11T21:03:51.553 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:03:51.579 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:03:51.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:51.582 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:03:51.683 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:51.709 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:51.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:51.711 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:03:51.766 INFO:teuthology.orchestra.run.smithi023.stderr:noscrub is set 2024-03-11T21:03:51.774 INFO:teuthology.orchestra.run.smithi023.stderr:reweighted osd.1 to 1 (10000) 2024-03-11T21:03:51.812 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:03:51.813 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:51 smithi023 bash[21372]: cluster 2024-03-11T21:03:50.900354+0000 osd.7 (osd.7) 281 : cluster 0 2.b scrub starts 2024-03-11T21:03:51.813 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:51 smithi023 bash[21372]: cluster 2024-03-11T21:03:50.901393+0000 osd.7 (osd.7) 282 : cluster 0 2.b scrub ok 2024-03-11T21:03:51.814 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:51 smithi023 bash[21372]: audit 2024-03-11T21:03:50.952795+0000 mon.c (mon.2) 63 : audit 0 from='client.? 172.21.15.23:0/659699116' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:03:51.814 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:51 smithi023 bash[21372]: audit 2024-03-11T21:03:51.528053+0000 mon.c (mon.2) 64 : audit 1 from='client.? 172.21.15.23:0/4168545008' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-11T21:03:51.814 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:51 smithi023 bash[21372]: audit 2024-03-11T21:03:51.528810+0000 mon.a (mon.0) 1482 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-11T21:03:51.814 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:51 smithi023 bash[21372]: audit 2024-03-11T21:03:51.597444+0000 mon.a (mon.0) 1483 : audit 1 from='client.? 172.21.15.23:0/496589209' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:03:51.814 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:51 smithi023 bash[29146]: cluster 2024-03-11T21:03:50.900354+0000 osd.7 (osd.7) 281 : cluster 0 2.b scrub starts 2024-03-11T21:03:51.814 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:51 smithi023 bash[29146]: cluster 2024-03-11T21:03:50.901393+0000 osd.7 (osd.7) 282 : cluster 0 2.b scrub ok 2024-03-11T21:03:51.814 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:51 smithi023 bash[29146]: audit 2024-03-11T21:03:50.952795+0000 mon.c (mon.2) 63 : audit 0 from='client.? 172.21.15.23:0/659699116' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:03:51.815 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:51 smithi023 bash[29146]: audit 2024-03-11T21:03:51.528053+0000 mon.c (mon.2) 64 : audit 1 from='client.? 172.21.15.23:0/4168545008' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-11T21:03:51.815 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:51 smithi023 bash[29146]: audit 2024-03-11T21:03:51.528810+0000 mon.a (mon.0) 1482 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-11T21:03:51.815 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:51 smithi023 bash[29146]: audit 2024-03-11T21:03:51.597444+0000 mon.a (mon.0) 1483 : audit 1 from='client.? 172.21.15.23:0/496589209' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:03:51.850 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:03:51.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:51.855 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:03:51.956 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:51.981 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:51.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:51.984 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:03:52.013 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:51 smithi146 bash[23809]: cluster 2024-03-11T21:03:50.900354+0000 osd.7 (osd.7) 281 : cluster 0 2.b scrub starts 2024-03-11T21:03:52.013 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:51 smithi146 bash[23809]: cluster 2024-03-11T21:03:50.901393+0000 osd.7 (osd.7) 282 : cluster 0 2.b scrub ok 2024-03-11T21:03:52.013 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:51 smithi146 bash[23809]: audit 2024-03-11T21:03:50.952795+0000 mon.c (mon.2) 63 : audit 0 from='client.? 172.21.15.23:0/659699116' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:03:52.013 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:51 smithi146 bash[23809]: audit 2024-03-11T21:03:51.528053+0000 mon.c (mon.2) 64 : audit 1 from='client.? 172.21.15.23:0/4168545008' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-11T21:03:52.013 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:51 smithi146 bash[23809]: audit 2024-03-11T21:03:51.528810+0000 mon.a (mon.0) 1482 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-11T21:03:52.013 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:51 smithi146 bash[23809]: audit 2024-03-11T21:03:51.597444+0000 mon.a (mon.0) 1483 : audit 1 from='client.? 172.21.15.23:0/496589209' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:03:52.085 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:52.117 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:52.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:52.120 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:03:52.220 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:52.245 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:52.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:52.247 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:03:52.348 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:52.374 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:52.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:52.381 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:03:52.457 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 50 16 73213 73197 91.4868 69.25 0.00915681 0.0109259 2024-03-11T21:03:52.481 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:03:52.513 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:03:52.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:52.518 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:03:52.619 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:52.642 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:52.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:52.645 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:03:52.745 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:03:52.768 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:03:52.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:52.772 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:03:52.873 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:03:52.890 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:03:52.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:52.892 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:03:52.993 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:53.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:52 smithi146 bash[23809]: cluster 2024-03-11T21:03:51.297750+0000 osd.4 (osd.4) 73 : cluster 0 2.f scrub starts 2024-03-11T21:03:53.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:52 smithi146 bash[23809]: cluster 2024-03-11T21:03:51.299294+0000 osd.4 (osd.4) 74 : cluster 0 2.f scrub ok 2024-03-11T21:03:53.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:52 smithi146 bash[23809]: cluster 2024-03-11T21:03:51.742512+0000 mon.a (mon.0) 1484 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:03:53.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:52 smithi146 bash[23809]: audit 2024-03-11T21:03:51.759215+0000 mon.a (mon.0) 1485 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-11T21:03:53.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:52 smithi146 bash[23809]: audit 2024-03-11T21:03:51.759402+0000 mon.a (mon.0) 1486 : audit 1 from='client.? 172.21.15.23:0/496589209' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:03:53.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:52 smithi146 bash[23809]: cluster 2024-03-11T21:03:51.766149+0000 mon.a (mon.0) 1487 : cluster 0 osdmap e272: 8 total, 8 up, 8 in 2024-03-11T21:03:53.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:52 smithi146 bash[23809]: cluster 2024-03-11T21:03:52.163747+0000 mgr.y (mgr.24380) 484 : cluster 0 pgmap v543: 148 pgs: 148 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.1 KiB/s rd, 64 MiB/s wr, 1.02k op/s 2024-03-11T21:03:53.025 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:53.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:53.030 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:03:53.131 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:53.158 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:53.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:53.165 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:03:53.203 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:52 smithi023 bash[21372]: cluster 2024-03-11T21:03:51.297750+0000 osd.4 (osd.4) 73 : cluster 0 2.f scrub starts 2024-03-11T21:03:53.204 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:52 smithi023 bash[21372]: cluster 2024-03-11T21:03:51.299294+0000 osd.4 (osd.4) 74 : cluster 0 2.f scrub ok 2024-03-11T21:03:53.204 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:52 smithi023 bash[21372]: cluster 2024-03-11T21:03:51.742512+0000 mon.a (mon.0) 1484 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:03:53.204 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:52 smithi023 bash[21372]: audit 2024-03-11T21:03:51.759215+0000 mon.a (mon.0) 1485 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-11T21:03:53.204 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:52 smithi023 bash[21372]: audit 2024-03-11T21:03:51.759402+0000 mon.a (mon.0) 1486 : audit 1 from='client.? 172.21.15.23:0/496589209' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:03:53.204 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:52 smithi023 bash[21372]: cluster 2024-03-11T21:03:51.766149+0000 mon.a (mon.0) 1487 : cluster 0 osdmap e272: 8 total, 8 up, 8 in 2024-03-11T21:03:53.204 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:52 smithi023 bash[21372]: cluster 2024-03-11T21:03:52.163747+0000 mgr.y (mgr.24380) 484 : cluster 0 pgmap v543: 148 pgs: 148 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.1 KiB/s rd, 64 MiB/s wr, 1.02k op/s 2024-03-11T21:03:53.204 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:52 smithi023 bash[29146]: cluster 2024-03-11T21:03:51.297750+0000 osd.4 (osd.4) 73 : cluster 0 2.f scrub starts 2024-03-11T21:03:53.204 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:52 smithi023 bash[29146]: cluster 2024-03-11T21:03:51.299294+0000 osd.4 (osd.4) 74 : cluster 0 2.f scrub ok 2024-03-11T21:03:53.204 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:52 smithi023 bash[29146]: cluster 2024-03-11T21:03:51.742512+0000 mon.a (mon.0) 1484 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:03:53.204 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:52 smithi023 bash[29146]: audit 2024-03-11T21:03:51.759215+0000 mon.a (mon.0) 1485 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-11T21:03:53.204 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:52 smithi023 bash[29146]: audit 2024-03-11T21:03:51.759402+0000 mon.a (mon.0) 1486 : audit 1 from='client.? 172.21.15.23:0/496589209' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:03:53.204 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:52 smithi023 bash[29146]: cluster 2024-03-11T21:03:51.766149+0000 mon.a (mon.0) 1487 : cluster 0 osdmap e272: 8 total, 8 up, 8 in 2024-03-11T21:03:53.205 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:52 smithi023 bash[29146]: cluster 2024-03-11T21:03:52.163747+0000 mgr.y (mgr.24380) 484 : cluster 0 pgmap v543: 148 pgs: 148 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.1 KiB/s rd, 64 MiB/s wr, 1.02k op/s 2024-03-11T21:03:53.265 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:53.283 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:53.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:53.285 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:03:53.385 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:53.412 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:53.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:53.422 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:03:53.457 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 51 16 74299 74283 91.0237 67.875 0.00598956 0.0109808 2024-03-11T21:03:53.522 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:53.554 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:53.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:53.558 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:03:53.659 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:03:53.685 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:03:53.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:53.690 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:03:53.791 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:03:53.821 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:03:53.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:53.824 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:03:53.924 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:03:53.957 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:03:53.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:53.960 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:03:54.060 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:03:54.079 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:03:54.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:54.081 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:03:54.182 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:54.205 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:54.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:54.212 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:03:54.313 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:03:54.338 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:03:54.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:54.341 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:03:54.442 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:03:54.457 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 52 16 75420 75404 90.6205 70.0625 0.0284276 0.0110288 2024-03-11T21:03:54.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:54.482 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:03:54.482 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:03:54.583 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:54.612 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:54.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:54.615 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:03:54.624 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:03:54.716 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:03:54.742 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:03:54.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:54.750 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:03:54.762 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-03-11T21:03:54.762 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T21:03:54.851 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:54.886 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:54.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:54.888 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:03:54.989 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:03:55.015 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:03:55.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:55.025 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:03:55.125 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:55.158 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:55.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:55.164 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:03:55.265 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:55.281 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T21:03:55.291 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:55.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:55.296 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:03:55.397 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:55.409 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:55 smithi146 bash[23809]: cluster 2024-03-11T21:03:54.165597+0000 mgr.y (mgr.24380) 485 : cluster 0 pgmap v544: 148 pgs: 148 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 61 MiB/s wr, 973 op/s 2024-03-11T21:03:55.409 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:55 smithi146 bash[23809]: audit 2024-03-11T21:03:54.354267+0000 mon.a (mon.0) 1488 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:03:55.409 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:55 smithi146 bash[23809]: cluster 2024-03-11T21:03:55.209116+0000 mon.a (mon.0) 1489 : cluster 0 osdmap e273: 8 total, 8 up, 8 in 2024-03-11T21:03:55.431 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:55.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:55.436 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:03:55.457 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 53 16 76401 76385 90.0674 61.3125 0.0150763 0.0110976 2024-03-11T21:03:55.512 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:03:55 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:03:55.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:55 smithi023 bash[21372]: cluster 2024-03-11T21:03:54.165597+0000 mgr.y (mgr.24380) 485 : cluster 0 pgmap v544: 148 pgs: 148 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 61 MiB/s wr, 973 op/s 2024-03-11T21:03:55.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:55 smithi023 bash[21372]: audit 2024-03-11T21:03:54.354267+0000 mon.a (mon.0) 1488 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:03:55.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:55 smithi023 bash[21372]: cluster 2024-03-11T21:03:55.209116+0000 mon.a (mon.0) 1489 : cluster 0 osdmap e273: 8 total, 8 up, 8 in 2024-03-11T21:03:55.534 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:55 smithi023 bash[29146]: cluster 2024-03-11T21:03:54.165597+0000 mgr.y (mgr.24380) 485 : cluster 0 pgmap v544: 148 pgs: 148 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 61 MiB/s wr, 973 op/s 2024-03-11T21:03:55.534 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:55 smithi023 bash[29146]: audit 2024-03-11T21:03:54.354267+0000 mon.a (mon.0) 1488 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:03:55.534 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:55 smithi023 bash[29146]: cluster 2024-03-11T21:03:55.209116+0000 mon.a (mon.0) 1489 : cluster 0 osdmap e273: 8 total, 8 up, 8 in 2024-03-11T21:03:55.538 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:03:55.566 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:03:55.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:55.572 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:03:55.672 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:55.702 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:55.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:55.704 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:03:55.805 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:03:55.840 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:03:55.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:55.844 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:03:55.945 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:03:55.976 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:03:55.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:55.979 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:03:56.080 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:03:56.110 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:03:56.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:56.112 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:03:56.213 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:03:56.239 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:03:56.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:56.242 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:03:56.342 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:03:56.351 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:56 smithi146 bash[23809]: audit 2024-03-11T21:03:55.468091+0000 mgr.y (mgr.24380) 486 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:03:56.361 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:03:56.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:56.363 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:03:56.457 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 54 16 77305 77289 89.4456 56.5 0.00279169 0.0111298 2024-03-11T21:03:56.464 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:03:56.488 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:03:56.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:56.490 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:03:56.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:56 smithi023 bash[21372]: audit 2024-03-11T21:03:55.468091+0000 mgr.y (mgr.24380) 486 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:03:56.534 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:56 smithi023 bash[29146]: audit 2024-03-11T21:03:55.468091+0000 mgr.y (mgr.24380) 486 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:03:56.590 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:03:56.627 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:03:56.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:56.632 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:03:56.706 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-11T21:03:56.733 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:03:56.777 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:03:56.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:56.780 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:03:56.881 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:03:56.906 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:03:56.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:56.915 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:03:57.017 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:57.056 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:57.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:57.061 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:03:57.161 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:03:57.191 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:03:57.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:57.196 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:03:57.297 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:03:57.408 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:03:57.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:57.410 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:03:57.457 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 55 16 78260 78244 88.9044 59.6875 0.00386185 0.0112423 2024-03-11T21:03:57.511 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:03:57.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:57 smithi023 bash[21372]: cluster 2024-03-11T21:03:56.166301+0000 mgr.y (mgr.24380) 487 : cluster 0 pgmap v546: 148 pgs: 148 active+clean; 4.6 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 72 MiB/s wr, 1.16k op/s 2024-03-11T21:03:57.534 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:57 smithi023 bash[29146]: cluster 2024-03-11T21:03:56.166301+0000 mgr.y (mgr.24380) 487 : cluster 0 pgmap v546: 148 pgs: 148 active+clean; 4.6 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 72 MiB/s wr, 1.16k op/s 2024-03-11T21:03:57.541 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:03:57.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:57.544 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:03:57.611 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:57 smithi146 bash[23809]: cluster 2024-03-11T21:03:56.166301+0000 mgr.y (mgr.24380) 487 : cluster 0 pgmap v546: 148 pgs: 148 active+clean; 4.6 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 72 MiB/s wr, 1.16k op/s 2024-03-11T21:03:57.644 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:57.670 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:57.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:57.672 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:03:57.773 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:03:57.802 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:03:57.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:57.806 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:03:57.907 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:57.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:57.936 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:57.936 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:03:58.037 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:58.064 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:58.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:58.066 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:03:58.167 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:58.194 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:03:58.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:58.198 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:03:58.299 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:58.334 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:03:58.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:58.337 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:03:58.438 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:03:58.457 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 56 16 79411 79395 88.6013 71.9375 0.0123119 0.0112803 2024-03-11T21:03:58.461 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:03:58.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:58.466 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:03:58.567 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:58.594 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:58.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:58.596 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:03:58.697 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:58.727 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:58.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:58.733 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:03:58.783 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:03:58 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:03:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:03:58.834 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:03:58.870 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:03:58.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:58.872 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:03:58.973 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:03:58.991 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:03:58.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:58.993 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:03:59.093 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:59.110 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:59.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:59.113 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:03:59.214 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:03:59.235 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:03:59.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:59.237 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:03:59.337 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:03:59.357 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:03:59.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:59.359 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:03:59.457 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 57 16 79936 79920 87.6225 32.8125 0.0149604 0.0114072 2024-03-11T21:03:59.460 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:03:59.492 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:03:59.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:59.496 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:03:59.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:03:59 smithi023 bash[21372]: cluster 2024-03-11T21:03:58.166838+0000 mgr.y (mgr.24380) 488 : cluster 0 pgmap v547: 148 pgs: 148 active+clean; 4.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 56 MiB/s wr, 895 op/s 2024-03-11T21:03:59.534 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:03:59 smithi023 bash[29146]: cluster 2024-03-11T21:03:58.166838+0000 mgr.y (mgr.24380) 488 : cluster 0 pgmap v547: 148 pgs: 148 active+clean; 4.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 56 MiB/s wr, 895 op/s 2024-03-11T21:03:59.593 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:03:59.594 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:03:59.597 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:03:59.645 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:03:59.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:59.648 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:03:59.748 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:59.758 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:03:59 smithi146 bash[23809]: cluster 2024-03-11T21:03:58.166838+0000 mgr.y (mgr.24380) 488 : cluster 0 pgmap v547: 148 pgs: 148 active+clean; 4.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 56 MiB/s wr, 895 op/s 2024-03-11T21:03:59.773 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:03:59.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:59.780 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:03:59.881 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:03:59.904 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:03:59.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:03:59.908 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:04:00.009 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:04:00.038 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:04:00.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:00.043 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:04:00.144 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:04:00.184 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:04:00.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:00.187 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:04:00.288 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:04:00.319 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:04:00.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:00.323 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:04:00.424 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:04:00.457 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 58 16 80833 80817 87.0783 56.0625 0.00659735 0.011478 2024-03-11T21:04:00.464 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:04:00.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:00.467 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:04:00.568 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:00.618 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:00.619 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:04:00.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:00.620 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:04:00.721 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:04:00.774 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:04:00.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:00.777 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:04:00.877 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:04:00.905 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:04:00.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:00.907 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:04:01.008 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:04:01.033 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:04:01.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:00 smithi023 bash[21372]: audit 2024-03-11T21:03:59.867852+0000 mon.a (mon.0) 1490 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:04:01.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:00 smithi023 bash[21372]: audit 2024-03-11T21:03:59.883302+0000 mon.a (mon.0) 1491 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:04:01.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:00 smithi023 bash[21372]: cluster 2024-03-11T21:04:00.167669+0000 mgr.y (mgr.24380) 489 : cluster 0 pgmap v548: 148 pgs: 148 active+clean; 4.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 609 B/s rd, 65 MiB/s wr, 1.04k op/s 2024-03-11T21:04:01.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:00 smithi023 bash[21372]: audit 2024-03-11T21:04:00.586711+0000 mon.a (mon.0) 1492 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:04:01.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:00 smithi023 bash[21372]: audit 2024-03-11T21:04:00.598504+0000 mon.a (mon.0) 1493 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:04:01.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:00 smithi023 bash[29146]: audit 2024-03-11T21:03:59.867852+0000 mon.a (mon.0) 1490 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:04:01.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:00 smithi023 bash[29146]: audit 2024-03-11T21:03:59.883302+0000 mon.a (mon.0) 1491 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:04:01.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:00 smithi023 bash[29146]: cluster 2024-03-11T21:04:00.167669+0000 mgr.y (mgr.24380) 489 : cluster 0 pgmap v548: 148 pgs: 148 active+clean; 4.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 609 B/s rd, 65 MiB/s wr, 1.04k op/s 2024-03-11T21:04:01.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:00 smithi023 bash[29146]: audit 2024-03-11T21:04:00.586711+0000 mon.a (mon.0) 1492 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:04:01.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:00 smithi023 bash[29146]: audit 2024-03-11T21:04:00.598504+0000 mon.a (mon.0) 1493 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:04:01.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:01.039 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:04:01.140 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:01.150 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:00 smithi146 bash[23809]: audit 2024-03-11T21:03:59.867852+0000 mon.a (mon.0) 1490 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:04:01.150 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:00 smithi146 bash[23809]: audit 2024-03-11T21:03:59.883302+0000 mon.a (mon.0) 1491 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:04:01.150 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:00 smithi146 bash[23809]: cluster 2024-03-11T21:04:00.167669+0000 mgr.y (mgr.24380) 489 : cluster 0 pgmap v548: 148 pgs: 148 active+clean; 4.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 609 B/s rd, 65 MiB/s wr, 1.04k op/s 2024-03-11T21:04:01.151 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:00 smithi146 bash[23809]: audit 2024-03-11T21:04:00.586711+0000 mon.a (mon.0) 1492 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:04:01.151 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:00 smithi146 bash[23809]: audit 2024-03-11T21:04:00.598504+0000 mon.a (mon.0) 1493 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:04:01.167 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:01.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:01.169 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:04:01.270 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:04:01.293 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:04:01.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:01.296 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:04:01.397 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:04:01.439 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:04:01.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:01.444 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:04:01.457 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 59 16 81816 81800 86.6436 61.4375 0.0207141 0.0115365 2024-03-11T21:04:01.496 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-03-11T21:04:01.497 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:04:01.545 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:04:01.569 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:04:01.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:01.575 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:04:01.675 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:04:01.702 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:04:01.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:01.705 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:04:01.806 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:01.833 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:01.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:01.838 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:04:01.938 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:04:01.949 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:01 smithi023 bash[21372]: audit 2024-03-11T21:04:01.520909+0000 mon.a (mon.0) 1494 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:04:01.949 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:01 smithi023 bash[21372]: audit 2024-03-11T21:04:01.522064+0000 mon.a (mon.0) 1495 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:04:01.950 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:01 smithi023 bash[21372]: audit 2024-03-11T21:04:01.530920+0000 mon.a (mon.0) 1496 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:04:01.950 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:01 smithi023 bash[29146]: audit 2024-03-11T21:04:01.520909+0000 mon.a (mon.0) 1494 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:04:01.950 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:01 smithi023 bash[29146]: audit 2024-03-11T21:04:01.522064+0000 mon.a (mon.0) 1495 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:04:01.950 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:01 smithi023 bash[29146]: audit 2024-03-11T21:04:01.530920+0000 mon.a (mon.0) 1496 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:04:01.966 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:04:01.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:01.971 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:04:02.072 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:04:02.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:02.111 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:04:02.111 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:04:02.212 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:04:02.222 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:01 smithi146 bash[23809]: audit 2024-03-11T21:04:01.520909+0000 mon.a (mon.0) 1494 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:04:02.222 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:01 smithi146 bash[23809]: audit 2024-03-11T21:04:01.522064+0000 mon.a (mon.0) 1495 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:04:02.222 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:01 smithi146 bash[23809]: audit 2024-03-11T21:04:01.530920+0000 mon.a (mon.0) 1496 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:04:02.239 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:04:02.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:02.243 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:04:02.343 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:04:02.366 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:04:02.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:02.368 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:04:02.457 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:2024-03-11T21:04:02.462810+0000 min lat: 0.0011821 max lat: 0.549376 avg lat: 0.0115802 2024-03-11T21:04:02.458 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-11T21:04:02.458 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 60 16 82891 82875 86.3192 67.1875 0.0141229 0.0115802 2024-03-11T21:04:02.469 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:04:02.495 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:04:02.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:02.500 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:04:02.600 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:02.622 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:02.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:02.624 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:04:02.725 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:04:02.761 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:04:02.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:02.768 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:04:02.804 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-03-11T21:04:02.869 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:04:02.928 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:04:02.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:02.930 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:04:03.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:02 smithi146 bash[23809]: cluster 2024-03-11T21:04:02.168396+0000 mgr.y (mgr.24380) 490 : cluster 0 pgmap v549: 148 pgs: 148 active+clean; 4.9 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 61 MiB/s wr, 977 op/s 2024-03-11T21:04:03.031 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:03.066 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:03.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:03.070 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:04:03.171 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:04:03.183 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:02 smithi023 bash[21372]: cluster 2024-03-11T21:04:02.168396+0000 mgr.y (mgr.24380) 490 : cluster 0 pgmap v549: 148 pgs: 148 active+clean; 4.9 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 61 MiB/s wr, 977 op/s 2024-03-11T21:04:03.184 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:02 smithi023 bash[29146]: cluster 2024-03-11T21:04:02.168396+0000 mgr.y (mgr.24380) 490 : cluster 0 pgmap v549: 148 pgs: 148 active+clean; 4.9 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 61 MiB/s wr, 977 op/s 2024-03-11T21:04:03.200 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:04:03.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:03.203 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:04:03.304 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:04:03.339 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:04:03.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:03.342 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:04:03.443 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:03.457 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 61 16 83953 83937 85.9922 66.375 0.0104681 0.0116233 2024-03-11T21:04:03.470 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:03.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:03.472 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:04:03.573 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:04:03.601 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:04:03.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:03.606 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:04:03.707 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:03.732 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:03.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:03.736 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:04:03.837 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:04:03.866 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:04:03.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:03.869 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:04:03.970 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:03.988 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:03.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:03.990 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:04:04.090 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:04:04.120 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:04:04.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:04.125 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:04:04.226 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:04.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:04.254 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:04.254 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:04:04.355 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:04:04.391 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:04:04.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:04.393 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:04:04.458 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 62 16 85000 84984 85.6606 65.4375 0.0167071 0.0116688 2024-03-11T21:04:04.494 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:04.524 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:04.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:04.528 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:04:04.628 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:04:04.661 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:04:04.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:04.664 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:04:04.765 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:04:04.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:04.799 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:04:04.799 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:04:04.900 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:04:04.929 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:04:04.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:04.935 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:04:05.036 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:05.065 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:05.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:05.068 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:04:05.169 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:05.195 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:05.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:05.201 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:04:05.234 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:05 smithi023 bash[21372]: cluster 2024-03-11T21:04:04.168748+0000 mgr.y (mgr.24380) 491 : cluster 0 pgmap v550: 148 pgs: 148 active+clean; 4.9 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 61 MiB/s wr, 977 op/s 2024-03-11T21:04:05.301 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:05.325 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:05.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:05.327 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:04:05.428 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:04:05.458 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 63 16 86074 86058 85.3657 67.125 0.00985004 0.0117088 2024-03-11T21:04:05.458 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:04:05.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:05.463 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:04:05.474 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:05 smithi146 bash[23809]: cluster 2024-03-11T21:04:04.168748+0000 mgr.y (mgr.24380) 491 : cluster 0 pgmap v550: 148 pgs: 148 active+clean; 4.9 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 61 MiB/s wr, 977 op/s 2024-03-11T21:04:05.533 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:05 smithi023 bash[29146]: cluster 2024-03-11T21:04:04.168748+0000 mgr.y (mgr.24380) 491 : cluster 0 pgmap v550: 148 pgs: 148 active+clean; 4.9 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 61 MiB/s wr, 977 op/s 2024-03-11T21:04:05.564 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:05.584 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:05.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:05.589 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:04:05.689 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:05.723 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:05.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:05.728 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:04:05.728 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:04:05 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:04:05.829 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:04:05.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:05.864 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:04:05.864 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:04:05.965 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:04:06.000 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:04:06.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:06.004 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:04:06.105 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:04:06.128 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:04:06.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:06.133 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:04:06.234 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:04:06.267 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:04:06.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:06.271 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:04:06.372 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:04:06.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:06.400 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:04:06.401 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:04:06.458 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 64 16 87083 87067 85.0171 63.0625 0.020634 0.0117576 2024-03-11T21:04:06.501 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:04:06.512 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:06 smithi023 bash[21372]: audit 2024-03-11T21:04:05.291779+0000 mon.a (mon.0) 1497 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:04:06.512 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:06 smithi023 bash[21372]: audit 2024-03-11T21:04:05.293452+0000 mon.a (mon.0) 1498 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:04:06.512 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:06 smithi023 bash[21372]: audit 2024-03-11T21:04:05.476182+0000 mgr.y (mgr.24380) 492 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:04:06.513 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:06 smithi023 bash[29146]: audit 2024-03-11T21:04:05.291779+0000 mon.a (mon.0) 1497 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:04:06.513 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:06 smithi023 bash[29146]: audit 2024-03-11T21:04:05.293452+0000 mon.a (mon.0) 1498 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:04:06.513 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:06 smithi023 bash[29146]: audit 2024-03-11T21:04:05.476182+0000 mgr.y (mgr.24380) 492 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:04:06.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:06.530 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:04:06.530 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:04:06.631 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:06.640 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:06 smithi146 bash[23809]: audit 2024-03-11T21:04:05.291779+0000 mon.a (mon.0) 1497 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:04:06.641 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:06 smithi146 bash[23809]: audit 2024-03-11T21:04:05.293452+0000 mon.a (mon.0) 1498 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:04:06.641 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:06 smithi146 bash[23809]: audit 2024-03-11T21:04:05.476182+0000 mgr.y (mgr.24380) 492 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:04:06.656 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:06.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:06.660 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:04:06.761 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:04:06.788 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:04:06.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:06.793 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:04:06.894 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:06.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:06.930 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:06.931 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:04:07.034 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:04:07.080 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:04:07.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:07.083 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:04:07.184 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:04:07.207 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:04:07.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:07.209 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:04:07.310 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:04:07.348 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:04:07.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:07.351 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:04:07.452 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:04:07.458 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 65 16 88215 88199 84.7975 70.75 0.0232366 0.0117879 2024-03-11T21:04:07.461 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:07 smithi023 bash[21372]: cluster 2024-03-11T21:04:06.169608+0000 mgr.y (mgr.24380) 493 : cluster 0 pgmap v551: 148 pgs: 148 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.4 KiB/s rd, 63 MiB/s wr, 1.00k op/s 2024-03-11T21:04:07.461 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:07 smithi023 bash[29146]: cluster 2024-03-11T21:04:06.169608+0000 mgr.y (mgr.24380) 493 : cluster 0 pgmap v551: 148 pgs: 148 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.4 KiB/s rd, 63 MiB/s wr, 1.00k op/s 2024-03-11T21:04:07.479 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:04:07.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:07.481 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:04:07.582 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:04:07.618 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:04:07.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:07.621 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:04:07.681 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:04:07.712 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:07 smithi146 bash[23809]: cluster 2024-03-11T21:04:06.169608+0000 mgr.y (mgr.24380) 493 : cluster 0 pgmap v551: 148 pgs: 148 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.4 KiB/s rd, 63 MiB/s wr, 1.00k op/s 2024-03-11T21:04:07.722 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:07.751 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:07.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:07.755 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:04:07.856 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:04:07.897 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:04:07.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:07.899 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:04:08.000 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:08.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:08.031 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:08.031 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:04:08.132 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:04:08.162 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:04:08.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:08.167 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:04:08.268 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:04:08.294 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:04:08.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:08.296 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:04:08.397 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:04:08.428 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:04:08.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:08.430 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:04:08.458 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 66 15 89106 89091 84.3574 55.75 0.0129808 0.0118491 2024-03-11T21:04:08.531 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:08.561 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:08.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:08.564 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:04:08.665 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:08.695 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:08.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:08.698 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:04:08.783 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:04:08 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:04:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:04:08.799 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:08.828 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:08.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:08.832 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:04:08.933 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:04:08.958 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:04:08.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:08.959 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:04:09.060 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:04:09.086 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:04:09.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:09.090 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:04:09.191 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:04:09.218 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:04:09.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:09.220 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:04:09.321 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:09.331 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:09 smithi023 bash[21372]: cluster 2024-03-11T21:04:08.170162+0000 mgr.y (mgr.24380) 494 : cluster 0 pgmap v552: 148 pgs: 148 active+clean; 5.3 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 853 B/s rd, 54 MiB/s wr, 863 op/s 2024-03-11T21:04:09.331 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:09 smithi023 bash[29146]: cluster 2024-03-11T21:04:08.170162+0000 mgr.y (mgr.24380) 494 : cluster 0 pgmap v552: 148 pgs: 148 active+clean; 5.3 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 853 B/s rd, 54 MiB/s wr, 863 op/s 2024-03-11T21:04:09.346 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:09.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:09.350 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:04:09.451 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:04:09.458 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 67 16 90071 90055 83.9975 60.25 0.0030533 0.0118875 2024-03-11T21:04:09.472 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:04:09.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:09.474 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:04:09.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:09 smithi146 bash[23809]: cluster 2024-03-11T21:04:08.170162+0000 mgr.y (mgr.24380) 494 : cluster 0 pgmap v552: 148 pgs: 148 active+clean; 5.3 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 853 B/s rd, 54 MiB/s wr, 863 op/s 2024-03-11T21:04:09.575 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:09.601 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:09.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:09.604 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:04:09.705 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:04:09.734 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:04:09.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:09.739 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:04:09.839 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:04:09.867 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:04:09.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:09.869 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:04:09.970 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:04:09.996 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:04:09.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:10.000 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:04:10.101 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:10.127 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:10.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:10.129 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:04:10.230 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:04:10.260 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:04:10.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:10.263 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:04:10.364 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:04:10.388 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:04:10.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:10.393 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:04:10.458 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 68 16 90933 90917 83.5544 53.875 0.0105834 0.0119633 2024-03-11T21:04:10.494 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:04:10.518 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:04:10.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:10.524 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:04:10.625 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:04:10.657 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:04:10.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:10.664 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:04:10.765 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:10.800 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:10.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:10.802 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:04:10.836 INFO:teuthology.orchestra.run.smithi023.stdout:osd.0: {} 2024-03-11T21:04:10.892 INFO:teuthology.orchestra.run.smithi023.stdout:osd.1: {} 2024-03-11T21:04:10.903 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:04:10.937 INFO:teuthology.orchestra.run.smithi023.stdout:osd.2: {} 2024-03-11T21:04:10.941 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:04:10.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:10.944 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:04:10.973 INFO:teuthology.orchestra.run.smithi023.stdout:osd.3: {} 2024-03-11T21:04:11.019 INFO:teuthology.orchestra.run.smithi023.stdout:osd.4: {} 2024-03-11T21:04:11.045 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:04:11.052 INFO:teuthology.orchestra.run.smithi023.stdout:osd.5: {} 2024-03-11T21:04:11.074 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:04:11.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:11.076 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:04:11.098 INFO:teuthology.orchestra.run.smithi023.stdout:osd.6: {} 2024-03-11T21:04:11.166 INFO:teuthology.orchestra.run.smithi023.stdout:osd.7: {} 2024-03-11T21:04:11.177 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:11.178 INFO:teuthology.orchestra.run.smithi023.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:04:11.178 INFO:teuthology.orchestra.run.smithi023.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:04:11.178 INFO:teuthology.orchestra.run.smithi023.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:04:11.178 INFO:teuthology.orchestra.run.smithi023.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:04:11.179 INFO:teuthology.orchestra.run.smithi023.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:04:11.179 INFO:teuthology.orchestra.run.smithi023.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:04:11.179 INFO:teuthology.orchestra.run.smithi023.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:04:11.179 INFO:teuthology.orchestra.run.smithi023.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:04:11.198 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:11.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:11.203 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:04:11.303 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:11.325 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:11.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:11.330 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:04:11.431 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:11.440 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:11 smithi146 bash[23809]: cluster 2024-03-11T21:04:09.832435+0000 osd.7 (osd.7) 283 : cluster 0 purged_snaps scrub starts 2024-03-11T21:04:11.440 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:11 smithi146 bash[23809]: cluster 2024-03-11T21:04:09.832487+0000 osd.7 (osd.7) 284 : cluster 0 purged_snaps scrub ok 2024-03-11T21:04:11.440 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:11 smithi146 bash[23809]: cluster 2024-03-11T21:04:10.170880+0000 mgr.y (mgr.24380) 495 : cluster 0 pgmap v553: 148 pgs: 148 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 853 B/s rd, 67 MiB/s wr, 1.07k op/s 2024-03-11T21:04:11.440 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:11 smithi146 bash[23809]: cluster 2024-03-11T21:04:10.328188+0000 mon.a (mon.0) 1499 : cluster 0 osdmap e274: 8 total, 8 up, 8 in 2024-03-11T21:04:11.440 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:11 smithi146 bash[23809]: audit 2024-03-11T21:04:10.807524+0000 mon.a (mon.0) 1500 : audit 0 from='client.? 172.21.15.23:0/1610049361' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:04:11.458 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:11.458 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 69 15 91898 91883 83.2184 60.375 0.0104379 0.0120113 2024-03-11T21:04:11.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:11.460 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:04:11.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:11 smithi023 bash[21372]: cluster 2024-03-11T21:04:09.832435+0000 osd.7 (osd.7) 283 : cluster 0 purged_snaps scrub starts 2024-03-11T21:04:11.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:11 smithi023 bash[21372]: cluster 2024-03-11T21:04:09.832487+0000 osd.7 (osd.7) 284 : cluster 0 purged_snaps scrub ok 2024-03-11T21:04:11.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:11 smithi023 bash[21372]: cluster 2024-03-11T21:04:10.170880+0000 mgr.y (mgr.24380) 495 : cluster 0 pgmap v553: 148 pgs: 148 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 853 B/s rd, 67 MiB/s wr, 1.07k op/s 2024-03-11T21:04:11.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:11 smithi023 bash[21372]: cluster 2024-03-11T21:04:10.328188+0000 mon.a (mon.0) 1499 : cluster 0 osdmap e274: 8 total, 8 up, 8 in 2024-03-11T21:04:11.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:11 smithi023 bash[21372]: audit 2024-03-11T21:04:10.807524+0000 mon.a (mon.0) 1500 : audit 0 from='client.? 172.21.15.23:0/1610049361' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:04:11.534 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:11 smithi023 bash[29146]: cluster 2024-03-11T21:04:09.832435+0000 osd.7 (osd.7) 283 : cluster 0 purged_snaps scrub starts 2024-03-11T21:04:11.534 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:11 smithi023 bash[29146]: cluster 2024-03-11T21:04:09.832487+0000 osd.7 (osd.7) 284 : cluster 0 purged_snaps scrub ok 2024-03-11T21:04:11.534 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:11 smithi023 bash[29146]: cluster 2024-03-11T21:04:10.170880+0000 mgr.y (mgr.24380) 495 : cluster 0 pgmap v553: 148 pgs: 148 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 853 B/s rd, 67 MiB/s wr, 1.07k op/s 2024-03-11T21:04:11.534 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:11 smithi023 bash[29146]: cluster 2024-03-11T21:04:10.328188+0000 mon.a (mon.0) 1499 : cluster 0 osdmap e274: 8 total, 8 up, 8 in 2024-03-11T21:04:11.534 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:11 smithi023 bash[29146]: audit 2024-03-11T21:04:10.807524+0000 mon.a (mon.0) 1500 : audit 0 from='client.? 172.21.15.23:0/1610049361' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:04:11.561 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:04:11.589 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:04:11.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:11.592 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:04:11.692 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:04:11.734 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:04:11.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:11.739 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:04:11.764 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-11T21:04:11.772 INFO:teuthology.orchestra.run.smithi023.stderr:dumped all 2024-03-11T21:04:11.841 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:04:11.859 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:04:11.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:11.861 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:04:11.962 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:04:11.985 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:04:11.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:11.987 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:04:12.088 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:04:12.122 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:04:12.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:12.124 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:04:12.225 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:04:12.249 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:04:12.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:12.251 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:04:12.326 INFO:teuthology.orchestra.run.smithi023.stderr:nodeep-scrub is set 2024-03-11T21:04:12.352 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:04:12.354 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:12 smithi023 bash[21372]: audit 2024-03-11T21:04:11.809410+0000 mon.a (mon.0) 1501 : audit 1 from='client.? 172.21.15.23:0/3979314872' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:04:12.354 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:12 smithi023 bash[29146]: audit 2024-03-11T21:04:11.809410+0000 mon.a (mon.0) 1501 : audit 1 from='client.? 172.21.15.23:0/3979314872' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:04:12.379 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:04:12.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:12.383 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:04:12.458 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 70 16 92606 92590 82.6608 44.1875 0.00852309 0.0120926 2024-03-11T21:04:12.484 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:04:12.515 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:04:12.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:12.518 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:04:12.619 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:04:12.628 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:12 smithi146 bash[23809]: audit 2024-03-11T21:04:11.809410+0000 mon.a (mon.0) 1501 : audit 1 from='client.? 172.21.15.23:0/3979314872' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:04:12.647 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:04:12.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:12.650 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:04:12.751 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:12.771 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:12.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:12.773 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:04:12.874 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:04:12.899 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:04:12.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:12.901 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:04:13.002 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:04:13.032 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:04:13.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:13.034 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:04:13.136 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:04:13.171 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:04:13.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:13.172 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:04:13.273 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:04:13.298 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:04:13.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:13.304 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:04:13.405 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:13.440 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:13.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:13.445 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:04:13.458 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 71 16 93469 93453 82.2562 53.9375 0.0136869 0.0121516 2024-03-11T21:04:13.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:13 smithi146 bash[23809]: audit 2024-03-11T21:04:11.736923+0000 mgr.y (mgr.24380) 496 : audit 0 from='client.25031 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:04:13.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:13 smithi146 bash[23809]: cluster 2024-03-11T21:04:12.171562+0000 mgr.y (mgr.24380) 497 : cluster 0 pgmap v555: 148 pgs: 148 active+clean; 5.6 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 63 MiB/s wr, 1.01k op/s 2024-03-11T21:04:13.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:13 smithi146 bash[23809]: cluster 2024-03-11T21:04:12.316974+0000 mon.a (mon.0) 1502 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:04:13.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:13 smithi146 bash[23809]: audit 2024-03-11T21:04:12.326530+0000 mon.a (mon.0) 1503 : audit 1 from='client.? 172.21.15.23:0/3979314872' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:04:13.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:13 smithi146 bash[23809]: cluster 2024-03-11T21:04:12.342606+0000 mon.a (mon.0) 1504 : cluster 0 osdmap e275: 8 total, 8 up, 8 in 2024-03-11T21:04:13.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:13 smithi023 bash[21372]: audit 2024-03-11T21:04:11.736923+0000 mgr.y (mgr.24380) 496 : audit 0 from='client.25031 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:04:13.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:13 smithi023 bash[21372]: cluster 2024-03-11T21:04:12.171562+0000 mgr.y (mgr.24380) 497 : cluster 0 pgmap v555: 148 pgs: 148 active+clean; 5.6 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 63 MiB/s wr, 1.01k op/s 2024-03-11T21:04:13.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:13 smithi023 bash[21372]: cluster 2024-03-11T21:04:12.316974+0000 mon.a (mon.0) 1502 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:04:13.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:13 smithi023 bash[21372]: audit 2024-03-11T21:04:12.326530+0000 mon.a (mon.0) 1503 : audit 1 from='client.? 172.21.15.23:0/3979314872' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:04:13.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:13 smithi023 bash[21372]: cluster 2024-03-11T21:04:12.342606+0000 mon.a (mon.0) 1504 : cluster 0 osdmap e275: 8 total, 8 up, 8 in 2024-03-11T21:04:13.534 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:13 smithi023 bash[29146]: audit 2024-03-11T21:04:11.736923+0000 mgr.y (mgr.24380) 496 : audit 0 from='client.25031 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:04:13.534 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:13 smithi023 bash[29146]: cluster 2024-03-11T21:04:12.171562+0000 mgr.y (mgr.24380) 497 : cluster 0 pgmap v555: 148 pgs: 148 active+clean; 5.6 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 63 MiB/s wr, 1.01k op/s 2024-03-11T21:04:13.534 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:13 smithi023 bash[29146]: cluster 2024-03-11T21:04:12.316974+0000 mon.a (mon.0) 1502 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:04:13.534 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:13 smithi023 bash[29146]: audit 2024-03-11T21:04:12.326530+0000 mon.a (mon.0) 1503 : audit 1 from='client.? 172.21.15.23:0/3979314872' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:04:13.534 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:13 smithi023 bash[29146]: cluster 2024-03-11T21:04:12.342606+0000 mon.a (mon.0) 1504 : cluster 0 osdmap e275: 8 total, 8 up, 8 in 2024-03-11T21:04:13.545 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:04:13.577 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:04:13.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:13.585 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:04:13.686 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:04:13.713 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:04:13.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:13.715 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:04:13.816 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:04:13.850 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:04:13.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:13.857 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:04:13.958 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:04:13.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:14.000 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:04:14.001 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:04:14.102 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:04:14.129 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:04:14.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:14.133 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:04:14.169 INFO:teuthology.orchestra.run.smithi023.stdout:{"pg_ready":true,"pg_map":{"version":554,"stamp":"2024-03-11T21:04:10.325443+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5848504744,"num_objects":89430,"num_object_clones":0,"num_object_copies":268290,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89430,"num_whiteouts":0,"num_read":2044,"num_read_kb":1890,"num_write":90133,"num_write_kb":5713309,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":197,"num_bytes_recovered":1775932,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81276,"ondisk_log_size":81276,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":18916956,"kb_used_data":16925852,"kb_used_omap":21,"kb_used_meta":1991018,"kb_avail":731044260,"statfs":{"total":767960285184,"available":748589322240,"internally_reserved":0,"allocated":17332072448,"data_stored":17327015712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22089,"internal_metadata":2038802871},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":653524992,"num_objects":9972,"num_object_clones":0,"num_object_copies":29916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9972,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":9972,"num_write_kb":638208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.158417"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":435,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.222207+0000","last_change":"2024-03-11T21:00:08.811947+0000","last_active":"2024-03-11T21:03:55.222207+0000","last_peered":"2024-03-11T21:03:55.222207+0000","last_clean":"2024-03-11T21:03:55.222207+0000","last_became_active":"2024-03-11T20:50:34.338390+0000","last_became_peered":"2024-03-11T20:50:34.338390+0000","last_unstale":"2024-03-11T21:03:55.222207+0000","last_undegraded":"2024-03-11T21:03:55.222207+0000","last_fullsized":"2024-03-11T21:03:55.222207+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:00:08.811756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:00:08.811756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:03:57.806325+0000","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":439,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.222274+0000","last_change":"2024-03-11T21:00:02.764408+0000","last_active":"2024-03-11T21:03:55.222274+0000","last_peered":"2024-03-11T21:03:55.222274+0000","last_clean":"2024-03-11T21:03:55.222274+0000","last_became_active":"2024-03-11T20:50:32.210194+0000","last_became_peered":"2024-03-11T20:50:32.210194+0000","last_unstale":"2024-03-11T21:03:55.222274+0000","last_undegraded":"2024-03-11T21:03:55.222274+0000","last_fullsized":"2024-03-11T21:03:55.222274+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:00:02.764190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:00:02.764190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:03:57.806438+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":264,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.225344+0000","last_change":"2024-03-11T21:03:38.968156+0000","last_active":"2024-03-11T21:03:55.225344+0000","last_peered":"2024-03-11T21:03:55.225344+0000","last_clean":"2024-03-11T21:03:55.225344+0000","last_became_active":"2024-03-11T21:00:27.963600+0000","last_became_peered":"2024-03-11T21:00:27.963600+0000","last_unstale":"2024-03-11T21:03:55.225344+0000","last_undegraded":"2024-03-11T21:03:55.225344+0000","last_fullsized":"2024-03-11T21:03:55.225344+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:38.968078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:53:29.964907+0000","last_clean_scrub_stamp":"2024-03-11T21:03:38.968078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:05:07.073103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1e","version":"207'20","reported_seq":474,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.225007+0000","last_change":"2024-03-11T21:00:09.750591+0000","last_active":"2024-03-11T21:03:55.225007+0000","last_peered":"2024-03-11T21:03:55.225007+0000","last_clean":"2024-03-11T21:03:55.225007+0000","last_became_active":"2024-03-11T20:50:30.101721+0000","last_became_peered":"2024-03-11T20:50:30.101721+0000","last_unstale":"2024-03-11T21:03:55.225007+0000","last_undegraded":"2024-03-11T21:03:55.225007+0000","last_fullsized":"2024-03-11T21:03:55.225007+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'10","last_scrub_stamp":"2024-03-11T21:00:09.750456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:00:09.750456+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:03:57.806467+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":403,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.213580+0000","last_change":"2024-03-11T21:02:30.521163+0000","last_active":"2024-03-11T21:03:55.213580+0000","last_peered":"2024-03-11T21:03:55.213580+0000","last_clean":"2024-03-11T21:03:55.213580+0000","last_became_active":"2024-03-11T21:00:48.467854+0000","last_became_peered":"2024-03-11T21:00:48.467854+0000","last_unstale":"2024-03-11T21:03:55.213580+0000","last_undegraded":"2024-03-11T21:03:55.213580+0000","last_fullsized":"2024-03-11T21:03:55.213580+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:02:30.520851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:55.545758+0000","last_clean_scrub_stamp":"2024-03-11T21:02:30.520851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:03:58.303216+0000","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":416,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.212985+0000","last_change":"2024-03-11T21:03:31.347920+0000","last_active":"2024-03-11T21:03:55.212985+0000","last_peered":"2024-03-11T21:03:55.212985+0000","last_clean":"2024-03-11T21:03:55.212985+0000","last_became_active":"2024-03-11T21:00:48.459957+0000","last_became_peered":"2024-03-11T21:00:48.459957+0000","last_unstale":"2024-03-11T21:03:55.212985+0000","last_undegraded":"2024-03-11T21:03:55.212985+0000","last_fullsized":"2024-03-11T21:03:55.212985+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:31.347841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:54:13.660325+0000","last_clean_scrub_stamp":"2024-03-11T21:03:31.347841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:35.281947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":448,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.224856+0000","last_change":"2024-03-11T21:03:40.675667+0000","last_active":"2024-03-11T21:03:55.224856+0000","last_peered":"2024-03-11T21:03:55.224856+0000","last_clean":"2024-03-11T21:03:55.224856+0000","last_became_active":"2024-03-11T20:50:28.058274+0000","last_became_peered":"2024-03-11T20:50:28.058274+0000","last_unstale":"2024-03-11T21:03:55.224856+0000","last_undegraded":"2024-03-11T21:03:55.224856+0000","last_fullsized":"2024-03-11T21:03:55.224856+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:40.675526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:03:40.675526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:51.747922+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"207'19","reported_seq":488,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.223201+0000","last_change":"2024-03-11T21:03:12.772305+0000","last_active":"2024-03-11T21:03:55.223201+0000","last_peered":"2024-03-11T21:03:55.223201+0000","last_clean":"2024-03-11T21:03:55.223201+0000","last_became_active":"2024-03-11T20:50:30.092638+0000","last_became_peered":"2024-03-11T20:50:30.092638+0000","last_unstale":"2024-03-11T21:03:55.223201+0000","last_undegraded":"2024-03-11T21:03:55.223201+0000","last_fullsized":"2024-03-11T21:03:55.223201+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'19","last_scrub_stamp":"2024-03-11T21:03:12.772194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:03:12.772194+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:37.917374+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":448,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.221453+0000","last_change":"2024-03-11T21:02:23.769016+0000","last_active":"2024-03-11T21:03:55.221453+0000","last_peered":"2024-03-11T21:03:55.221453+0000","last_clean":"2024-03-11T21:03:55.221453+0000","last_became_active":"2024-03-11T21:00:48.460694+0000","last_became_peered":"2024-03-11T21:00:48.460694+0000","last_unstale":"2024-03-11T21:03:55.221453+0000","last_undegraded":"2024-03-11T21:03:55.221453+0000","last_fullsized":"2024-03-11T21:03:55.221453+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:02:23.768654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:53:07.792697+0000","last_clean_scrub_stamp":"2024-03-11T21:02:23.768654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:03:57.303218+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":407,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.213593+0000","last_change":"2024-03-11T21:03:36.230810+0000","last_active":"2024-03-11T21:03:55.213593+0000","last_peered":"2024-03-11T21:03:55.213593+0000","last_clean":"2024-03-11T21:03:55.213593+0000","last_became_active":"2024-03-11T21:00:48.463839+0000","last_became_peered":"2024-03-11T21:00:48.463839+0000","last_unstale":"2024-03-11T21:03:55.213593+0000","last_undegraded":"2024-03-11T21:03:55.213593+0000","last_fullsized":"2024-03-11T21:03:55.213593+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:36.230647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:03:36.230647+0000","last_clean_scrub_stamp":"2024-03-11T21:03:36.230647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:05:04.850393+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":462,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.228006+0000","last_change":"2024-03-11T21:03:36.980913+0000","last_active":"2024-03-11T21:03:55.228006+0000","last_peered":"2024-03-11T21:03:55.228006+0000","last_clean":"2024-03-11T21:03:55.228006+0000","last_became_active":"2024-03-11T20:50:28.079225+0000","last_became_peered":"2024-03-11T20:50:28.079225+0000","last_unstale":"2024-03-11T21:03:55.228006+0000","last_undegraded":"2024-03-11T21:03:55.228006+0000","last_fullsized":"2024-03-11T21:03:55.228006+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:36.980825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:03:36.980825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:48.975950+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"207'27","reported_seq":551,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.222920+0000","last_change":"2024-03-11T21:02:24.705731+0000","last_active":"2024-03-11T21:03:55.222920+0000","last_peered":"2024-03-11T21:03:55.222920+0000","last_clean":"2024-03-11T21:03:55.222920+0000","last_became_active":"2024-03-11T21:00:48.462943+0000","last_became_peered":"2024-03-11T21:00:48.462943+0000","last_unstale":"2024-03-11T21:03:55.222920+0000","last_undegraded":"2024-03-11T21:03:55.222920+0000","last_fullsized":"2024-03-11T21:03:55.222920+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'27","last_scrub_stamp":"2024-03-11T21:02:24.705531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:02:24.705531+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:03:57.303252+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":154,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.216823+0000","last_change":"2024-03-11T21:03:20.330025+0000","last_active":"2024-03-11T21:03:55.216823+0000","last_peered":"2024-03-11T21:03:55.216823+0000","last_clean":"2024-03-11T21:03:55.216823+0000","last_became_active":"2024-03-11T21:00:48.456017+0000","last_became_peered":"2024-03-11T21:00:48.456017+0000","last_unstale":"2024-03-11T21:03:55.216823+0000","last_undegraded":"2024-03-11T21:03:55.216823+0000","last_fullsized":"2024-03-11T21:03:55.216823+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:20.329913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:03:20.329913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:27.990899+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":403,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.213252+0000","last_change":"2024-03-11T21:03:16.356593+0000","last_active":"2024-03-11T21:03:55.213252+0000","last_peered":"2024-03-11T21:03:55.213252+0000","last_clean":"2024-03-11T21:03:55.213252+0000","last_became_active":"2024-03-11T21:00:27.962251+0000","last_became_peered":"2024-03-11T21:00:27.962251+0000","last_unstale":"2024-03-11T21:03:55.213252+0000","last_undegraded":"2024-03-11T21:03:55.213252+0000","last_fullsized":"2024-03-11T21:03:55.213252+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:16.356482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:53:50.599029+0000","last_clean_scrub_stamp":"2024-03-11T21:03:16.356482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:16.806473+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":462,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.223660+0000","last_change":"2024-03-11T21:02:31.898315+0000","last_active":"2024-03-11T21:03:55.223660+0000","last_peered":"2024-03-11T21:03:55.223660+0000","last_clean":"2024-03-11T21:03:55.223660+0000","last_became_active":"2024-03-11T20:50:28.068978+0000","last_became_peered":"2024-03-11T20:50:28.068978+0000","last_unstale":"2024-03-11T21:03:55.223660+0000","last_undegraded":"2024-03-11T21:03:55.223660+0000","last_fullsized":"2024-03-11T21:03:55.223660+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:02:31.898099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:54:07.155130+0000","last_clean_scrub_stamp":"2024-03-11T21:02:31.898099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:03:56.881966+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"207'23","reported_seq":489,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.225691+0000","last_change":"2024-03-11T21:02:32.859754+0000","last_active":"2024-03-11T21:03:55.225691+0000","last_peered":"2024-03-11T21:03:55.225691+0000","last_clean":"2024-03-11T21:03:55.225691+0000","last_became_active":"2024-03-11T21:00:27.952093+0000","last_became_peered":"2024-03-11T21:00:27.952093+0000","last_unstale":"2024-03-11T21:03:55.225691+0000","last_undegraded":"2024-03-11T21:03:55.225691+0000","last_fullsized":"2024-03-11T21:03:55.225691+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'23","last_scrub_stamp":"2024-03-11T21:02:32.859604+0000","last_deep_scrub":"110'17","last_deep_scrub_stamp":"2024-03-11T20:53:04.802488+0000","last_clean_scrub_stamp":"2024-03-11T21:02:32.859604+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:03:57.303270+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":436,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.229900+0000","last_change":"2024-03-11T21:00:13.778710+0000","last_active":"2024-03-11T21:03:55.229900+0000","last_peered":"2024-03-11T21:03:55.229900+0000","last_clean":"2024-03-11T21:03:55.229900+0000","last_became_active":"2024-03-11T20:50:34.334558+0000","last_became_peered":"2024-03-11T20:50:34.334558+0000","last_unstale":"2024-03-11T21:03:55.229900+0000","last_undegraded":"2024-03-11T21:03:55.229900+0000","last_fullsized":"2024-03-11T21:03:55.229900+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:00:13.778532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:55:59.892394+0000","last_clean_scrub_stamp":"2024-03-11T21:00:13.778532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:03:57.806492+0000","scrub_duration":87,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":287,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.225568+0000","last_change":"2024-03-11T21:03:48.239121+0000","last_active":"2024-03-11T21:03:55.225568+0000","last_peered":"2024-03-11T21:03:55.225568+0000","last_clean":"2024-03-11T21:03:55.225568+0000","last_became_active":"2024-03-11T21:00:48.461013+0000","last_became_peered":"2024-03-11T21:00:48.461013+0000","last_unstale":"2024-03-11T21:03:55.225568+0000","last_undegraded":"2024-03-11T21:03:55.225568+0000","last_fullsized":"2024-03-11T21:03:55.225568+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:48.239009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:03:48.239009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:51.626936+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"77'1","reported_seq":455,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.224758+0000","last_change":"2024-03-11T21:00:10.713115+0000","last_active":"2024-03-11T21:03:55.224758+0000","last_peered":"2024-03-11T21:03:55.224758+0000","last_clean":"2024-03-11T21:03:55.224758+0000","last_became_active":"2024-03-11T20:50:28.068840+0000","last_became_peered":"2024-03-11T20:50:28.068840+0000","last_unstale":"2024-03-11T21:03:55.224758+0000","last_undegraded":"2024-03-11T21:03:55.224758+0000","last_fullsized":"2024-03-11T21:03:55.224758+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-03-11T21:00:10.712812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:00:10.712812+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:03:57.806274+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"207'19","reported_seq":477,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.213491+0000","last_change":"2024-03-11T21:03:39.240266+0000","last_active":"2024-03-11T21:03:55.213491+0000","last_peered":"2024-03-11T21:03:55.213491+0000","last_clean":"2024-03-11T21:03:55.213491+0000","last_became_active":"2024-03-11T21:00:48.468434+0000","last_became_peered":"2024-03-11T21:00:48.468434+0000","last_unstale":"2024-03-11T21:03:55.213491+0000","last_undegraded":"2024-03-11T21:03:55.213491+0000","last_fullsized":"2024-03-11T21:03:55.213491+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'19","last_scrub_stamp":"2024-03-11T21:03:39.240089+0000","last_deep_scrub":"207'19","last_deep_scrub_stamp":"2024-03-11T21:03:39.240089+0000","last_clean_scrub_stamp":"2024-03-11T21:03:39.240089+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:05:07.484049+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":397,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.217326+0000","last_change":"2024-03-11T21:02:33.399743+0000","last_active":"2024-03-11T21:03:55.217326+0000","last_peered":"2024-03-11T21:03:55.217326+0000","last_clean":"2024-03-11T21:03:55.217326+0000","last_became_active":"2024-03-11T21:00:27.955250+0000","last_became_peered":"2024-03-11T21:00:27.955250+0000","last_unstale":"2024-03-11T21:03:55.217326+0000","last_undegraded":"2024-03-11T21:03:55.217326+0000","last_fullsized":"2024-03-11T21:03:55.217326+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:02:33.399510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:33.399510+0000","last_clean_scrub_stamp":"2024-03-11T21:02:33.399510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:03:58.303241+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":487,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.233036+0000","last_change":"2024-03-11T21:03:29.516265+0000","last_active":"2024-03-11T21:03:55.233036+0000","last_peered":"2024-03-11T21:03:55.233036+0000","last_clean":"2024-03-11T21:03:55.233036+0000","last_became_active":"2024-03-11T21:00:48.459135+0000","last_became_peered":"2024-03-11T21:00:48.459135+0000","last_unstale":"2024-03-11T21:03:55.233036+0000","last_undegraded":"2024-03-11T21:03:55.233036+0000","last_fullsized":"2024-03-11T21:03:55.233036+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:29.516124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:24.666325+0000","last_clean_scrub_stamp":"2024-03-11T21:03:29.516124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:38.006278+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":462,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.230015+0000","last_change":"2024-03-11T21:03:11.542784+0000","last_active":"2024-03-11T21:03:55.230015+0000","last_peered":"2024-03-11T21:03:55.230015+0000","last_clean":"2024-03-11T21:03:55.230015+0000","last_became_active":"2024-03-11T21:00:27.956935+0000","last_became_peered":"2024-03-11T21:00:27.956935+0000","last_unstale":"2024-03-11T21:03:55.230015+0000","last_undegraded":"2024-03-11T21:03:55.230015+0000","last_fullsized":"2024-03-11T21:03:55.230015+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:11.542663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:03:11.542663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:21.597227+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"207'7","reported_seq":470,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.222786+0000","last_change":"2024-03-11T21:02:32.818371+0000","last_active":"2024-03-11T21:03:55.222786+0000","last_peered":"2024-03-11T21:03:55.222786+0000","last_clean":"2024-03-11T21:03:55.222786+0000","last_became_active":"2024-03-11T21:00:27.955296+0000","last_became_peered":"2024-03-11T21:00:27.955296+0000","last_unstale":"2024-03-11T21:03:55.222786+0000","last_undegraded":"2024-03-11T21:03:55.222786+0000","last_fullsized":"2024-03-11T21:03:55.222786+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'7","last_scrub_stamp":"2024-03-11T21:02:32.818096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:02:32.818096+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":162,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.217288+0000","last_change":"2024-03-11T21:03:42.593360+0000","last_active":"2024-03-11T21:03:55.217288+0000","last_peered":"2024-03-11T21:03:55.217288+0000","last_clean":"2024-03-11T21:03:55.217288+0000","last_became_active":"2024-03-11T21:00:48.464694+0000","last_became_peered":"2024-03-11T21:00:48.464694+0000","last_unstale":"2024-03-11T21:03:55.217288+0000","last_undegraded":"2024-03-11T21:03:55.217288+0000","last_fullsized":"2024-03-11T21:03:55.217288+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:42.593243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:03:42.593243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:47.445913+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1f","version":"212'16","reported_seq":653,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:04:03.781349+0000","last_change":"2024-03-11T21:03:09.534518+0000","last_active":"2024-03-11T21:04:03.781349+0000","last_peered":"2024-03-11T21:04:03.781349+0000","last_clean":"2024-03-11T21:04:03.781349+0000","last_became_active":"2024-03-11T21:00:48.464350+0000","last_became_peered":"2024-03-11T21:00:48.464350+0000","last_unstale":"2024-03-11T21:04:03.781349+0000","last_undegraded":"2024-03-11T21:04:03.781349+0000","last_fullsized":"2024-03-11T21:04:03.781349+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"212'16","last_scrub_stamp":"2024-03-11T21:03:09.534368+0000","last_deep_scrub":"114'12","last_deep_scrub_stamp":"2024-03-11T20:53:30.905791+0000","last_clean_scrub_stamp":"2024-03-11T21:03:09.534368+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:38.232644+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"77'1","reported_seq":440,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.222361+0000","last_change":"2024-03-11T20:59:06.457866+0000","last_active":"2024-03-11T21:03:55.222361+0000","last_peered":"2024-03-11T21:03:55.222361+0000","last_clean":"2024-03-11T21:03:55.222361+0000","last_became_active":"2024-03-11T20:50:28.068827+0000","last_became_peered":"2024-03-11T20:50:28.068827+0000","last_unstale":"2024-03-11T21:03:55.222361+0000","last_undegraded":"2024-03-11T21:03:55.222361+0000","last_fullsized":"2024-03-11T21:03:55.222361+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-03-11T20:59:06.457561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:59:06.457561+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:03:57.806217+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"207'15","reported_seq":510,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.222142+0000","last_change":"2024-03-11T21:02:30.560424+0000","last_active":"2024-03-11T21:03:55.222142+0000","last_peered":"2024-03-11T21:03:55.222142+0000","last_clean":"2024-03-11T21:03:55.222142+0000","last_became_active":"2024-03-11T21:00:48.467513+0000","last_became_peered":"2024-03-11T21:00:48.467513+0000","last_unstale":"2024-03-11T21:03:55.222142+0000","last_undegraded":"2024-03-11T21:03:55.222142+0000","last_fullsized":"2024-03-11T21:03:55.222142+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'15","last_scrub_stamp":"2024-03-11T21:02:30.560293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:02:30.560293+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"105'2","reported_seq":453,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.224534+0000","last_change":"2024-03-11T21:03:41.972426+0000","last_active":"2024-03-11T21:03:55.224534+0000","last_peered":"2024-03-11T21:03:55.224534+0000","last_clean":"2024-03-11T21:03:55.224534+0000","last_became_active":"2024-03-11T20:50:34.327356+0000","last_became_peered":"2024-03-11T20:50:34.327356+0000","last_unstale":"2024-03-11T21:03:55.224534+0000","last_undegraded":"2024-03-11T21:03:55.224534+0000","last_fullsized":"2024-03-11T21:03:55.224534+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'2","last_scrub_stamp":"2024-03-11T21:03:41.971918+0000","last_deep_scrub":"105'2","last_deep_scrub_stamp":"2024-03-11T21:03:41.971918+0000","last_clean_scrub_stamp":"2024-03-11T21:03:41.971918+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:42.904105+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":263,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.228870+0000","last_change":"2024-03-11T21:02:14.767550+0000","last_active":"2024-03-11T21:03:55.228870+0000","last_peered":"2024-03-11T21:03:55.228870+0000","last_clean":"2024-03-11T21:03:55.228870+0000","last_became_active":"2024-03-11T20:56:59.804264+0000","last_became_peered":"2024-03-11T20:56:59.804264+0000","last_unstale":"2024-03-11T21:03:55.228870+0000","last_undegraded":"2024-03-11T21:03:55.228870+0000","last_fullsized":"2024-03-11T21:03:55.228870+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:02:14.767248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:59:14.605681+0000","last_clean_scrub_stamp":"2024-03-11T21:02:14.767248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:03:56.881896+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":456,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.221109+0000","last_change":"2024-03-11T21:03:09.746912+0000","last_active":"2024-03-11T21:03:55.221109+0000","last_peered":"2024-03-11T21:03:55.221109+0000","last_clean":"2024-03-11T21:03:55.221109+0000","last_became_active":"2024-03-11T21:00:27.947614+0000","last_became_peered":"2024-03-11T21:00:27.947614+0000","last_unstale":"2024-03-11T21:03:55.221109+0000","last_undegraded":"2024-03-11T21:03:55.221109+0000","last_fullsized":"2024-03-11T21:03:55.221109+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:09.746789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:52:01.880052+0000","last_clean_scrub_stamp":"2024-03-11T21:03:09.746789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:28.738045+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"207'27","reported_seq":192,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.216688+0000","last_change":"2024-03-11T21:03:27.406786+0000","last_active":"2024-03-11T21:03:55.216688+0000","last_peered":"2024-03-11T21:03:55.216688+0000","last_clean":"2024-03-11T21:03:55.216688+0000","last_became_active":"2024-03-11T21:00:48.457893+0000","last_became_peered":"2024-03-11T21:00:48.457893+0000","last_unstale":"2024-03-11T21:03:55.216688+0000","last_undegraded":"2024-03-11T21:03:55.216688+0000","last_fullsized":"2024-03-11T21:03:55.216688+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'27","last_scrub_stamp":"2024-03-11T21:03:27.406573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:03:27.406573+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:27.409569+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":435,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.225054+0000","last_change":"2024-03-11T21:00:12.754952+0000","last_active":"2024-03-11T21:03:55.225054+0000","last_peered":"2024-03-11T21:03:55.225054+0000","last_clean":"2024-03-11T21:03:55.225054+0000","last_became_active":"2024-03-11T20:50:34.337878+0000","last_became_peered":"2024-03-11T20:50:34.337878+0000","last_unstale":"2024-03-11T21:03:55.225054+0000","last_undegraded":"2024-03-11T21:03:55.225054+0000","last_fullsized":"2024-03-11T21:03:55.225054+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:00:12.754627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:52:11.462480+0000","last_clean_scrub_stamp":"2024-03-11T21:00:12.754627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:03:57.806300+0000","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":439,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.222707+0000","last_change":"2024-03-11T20:59:13.516522+0000","last_active":"2024-03-11T21:03:55.222707+0000","last_peered":"2024-03-11T21:03:55.222707+0000","last_clean":"2024-03-11T21:03:55.222707+0000","last_became_active":"2024-03-11T20:50:32.646323+0000","last_became_peered":"2024-03-11T20:50:32.646323+0000","last_unstale":"2024-03-11T21:03:55.222707+0000","last_undegraded":"2024-03-11T21:03:55.222707+0000","last_fullsized":"2024-03-11T21:03:55.222707+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:59:13.516326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:54:06.516412+0000","last_clean_scrub_stamp":"2024-03-11T20:59:13.516326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:03:57.806245+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":469,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.231001+0000","last_change":"2024-03-11T21:03:23.586644+0000","last_active":"2024-03-11T21:03:55.231001+0000","last_peered":"2024-03-11T21:03:55.231001+0000","last_clean":"2024-03-11T21:03:55.231001+0000","last_became_active":"2024-03-11T20:50:28.086513+0000","last_became_peered":"2024-03-11T20:50:28.086513+0000","last_unstale":"2024-03-11T21:03:55.231001+0000","last_undegraded":"2024-03-11T21:03:55.231001+0000","last_fullsized":"2024-03-11T21:03:55.231001+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:23.586540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:03:23.586540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:44.222880+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"207'19","reported_seq":514,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.229163+0000","last_change":"2024-03-11T21:02:21.669112+0000","last_active":"2024-03-11T21:03:55.229163+0000","last_peered":"2024-03-11T21:03:55.229163+0000","last_clean":"2024-03-11T21:03:55.229163+0000","last_became_active":"2024-03-11T21:00:48.464707+0000","last_became_peered":"2024-03-11T21:00:48.464707+0000","last_unstale":"2024-03-11T21:03:55.229163+0000","last_undegraded":"2024-03-11T21:03:55.229163+0000","last_fullsized":"2024-03-11T21:03:55.229163+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'19","last_scrub_stamp":"2024-03-11T21:02:21.668821+0000","last_deep_scrub":"145'13","last_deep_scrub_stamp":"2024-03-11T20:55:53.560287+0000","last_clean_scrub_stamp":"2024-03-11T21:02:21.668821+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:03:57.303199+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":451,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.223654+0000","last_change":"2024-03-11T21:03:24.925576+0000","last_active":"2024-03-11T21:03:55.223654+0000","last_peered":"2024-03-11T21:03:55.223654+0000","last_clean":"2024-03-11T21:03:55.223654+0000","last_became_active":"2024-03-11T21:00:27.973405+0000","last_became_peered":"2024-03-11T21:00:27.973405+0000","last_unstale":"2024-03-11T21:03:55.223654+0000","last_undegraded":"2024-03-11T21:03:55.223654+0000","last_fullsized":"2024-03-11T21:03:55.223654+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:24.925436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:03:24.925436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:46.169097+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":452,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.222249+0000","last_change":"2024-03-11T21:02:31.549151+0000","last_active":"2024-03-11T21:03:55.222249+0000","last_peered":"2024-03-11T21:03:55.222249+0000","last_clean":"2024-03-11T21:03:55.222249+0000","last_became_active":"2024-03-11T21:00:48.472812+0000","last_became_peered":"2024-03-11T21:00:48.472812+0000","last_unstale":"2024-03-11T21:03:55.222249+0000","last_undegraded":"2024-03-11T21:03:55.222249+0000","last_fullsized":"2024-03-11T21:03:55.222249+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:02:31.549032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:02:31.549032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":463,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.229066+0000","last_change":"2024-03-11T21:03:33.835941+0000","last_active":"2024-03-11T21:03:55.229066+0000","last_peered":"2024-03-11T21:03:55.229066+0000","last_clean":"2024-03-11T21:03:55.229066+0000","last_became_active":"2024-03-11T20:50:28.075760+0000","last_became_peered":"2024-03-11T20:50:28.075760+0000","last_unstale":"2024-03-11T21:03:55.229066+0000","last_undegraded":"2024-03-11T21:03:55.229066+0000","last_fullsized":"2024-03-11T21:03:55.229066+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:33.835839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:58:55.977289+0000","last_clean_scrub_stamp":"2024-03-11T21:03:33.835839+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:45.805119+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"207'12","reported_seq":308,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.230137+0000","last_change":"2024-03-11T21:03:21.879821+0000","last_active":"2024-03-11T21:03:55.230137+0000","last_peered":"2024-03-11T21:03:55.230137+0000","last_clean":"2024-03-11T21:03:55.230137+0000","last_became_active":"2024-03-11T20:56:59.794581+0000","last_became_peered":"2024-03-11T20:56:59.794581+0000","last_unstale":"2024-03-11T21:03:55.230137+0000","last_undegraded":"2024-03-11T21:03:55.230137+0000","last_fullsized":"2024-03-11T21:03:55.230137+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'12","last_scrub_stamp":"2024-03-11T21:03:21.879683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:03:21.879683+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:50.694254+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"186'2","reported_seq":437,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.222043+0000","last_change":"2024-03-11T21:00:17.739831+0000","last_active":"2024-03-11T21:03:55.222043+0000","last_peered":"2024-03-11T21:03:55.222043+0000","last_clean":"2024-03-11T21:03:55.222043+0000","last_became_active":"2024-03-11T20:50:34.339326+0000","last_became_peered":"2024-03-11T20:50:34.339326+0000","last_unstale":"2024-03-11T21:03:55.222043+0000","last_undegraded":"2024-03-11T21:03:55.222043+0000","last_fullsized":"2024-03-11T21:03:55.222043+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"186'2","last_scrub_stamp":"2024-03-11T21:00:17.739696+0000","last_deep_scrub":"186'2","last_deep_scrub_stamp":"2024-03-11T20:59:04.534850+0000","last_clean_scrub_stamp":"2024-03-11T21:00:17.739696+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:03:57.806412+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":416,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.213324+0000","last_change":"2024-03-11T21:03:45.273658+0000","last_active":"2024-03-11T21:03:55.213324+0000","last_peered":"2024-03-11T21:03:55.213324+0000","last_clean":"2024-03-11T21:03:55.213324+0000","last_became_active":"2024-03-11T21:00:48.462656+0000","last_became_peered":"2024-03-11T21:00:48.462656+0000","last_unstale":"2024-03-11T21:03:55.213324+0000","last_undegraded":"2024-03-11T21:03:55.213324+0000","last_fullsized":"2024-03-11T21:03:55.213324+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:45.273529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:03:45.273529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:05:08.865958+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":154,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.217158+0000","last_change":"2024-03-11T21:03:23.365771+0000","last_active":"2024-03-11T21:03:55.217158+0000","last_peered":"2024-03-11T21:03:55.217158+0000","last_clean":"2024-03-11T21:03:55.217158+0000","last_became_active":"2024-03-11T21:00:48.455510+0000","last_became_peered":"2024-03-11T21:00:48.455510+0000","last_unstale":"2024-03-11T21:03:55.217158+0000","last_undegraded":"2024-03-11T21:03:55.217158+0000","last_fullsized":"2024-03-11T21:03:55.217158+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:23.365625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:03:23.365625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:43.858756+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"207'25","reported_seq":409,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.217167+0000","last_change":"2024-03-11T21:03:26.409080+0000","last_active":"2024-03-11T21:03:55.217167+0000","last_peered":"2024-03-11T21:03:55.217167+0000","last_clean":"2024-03-11T21:03:55.217167+0000","last_became_active":"2024-03-11T21:00:27.959547+0000","last_became_peered":"2024-03-11T21:00:27.959547+0000","last_unstale":"2024-03-11T21:03:55.217167+0000","last_undegraded":"2024-03-11T21:03:55.217167+0000","last_fullsized":"2024-03-11T21:03:55.217167+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'25","last_scrub_stamp":"2024-03-11T21:03:26.408832+0000","last_deep_scrub":"161'15","last_deep_scrub_stamp":"2024-03-11T20:57:15.125548+0000","last_clean_scrub_stamp":"2024-03-11T21:03:26.408832+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:42.914368+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":461,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.222984+0000","last_change":"2024-03-11T21:03:27.615442+0000","last_active":"2024-03-11T21:03:55.222984+0000","last_peered":"2024-03-11T21:03:55.222984+0000","last_clean":"2024-03-11T21:03:55.222984+0000","last_became_active":"2024-03-11T21:00:48.462105+0000","last_became_peered":"2024-03-11T21:00:48.462105+0000","last_unstale":"2024-03-11T21:03:55.222984+0000","last_undegraded":"2024-03-11T21:03:55.222984+0000","last_fullsized":"2024-03-11T21:03:55.222984+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:27.615353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:03:27.615353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:32.184921+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":162,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.216842+0000","last_change":"2024-03-11T21:03:43.605939+0000","last_active":"2024-03-11T21:03:55.216842+0000","last_peered":"2024-03-11T21:03:55.216842+0000","last_clean":"2024-03-11T21:03:55.216842+0000","last_became_active":"2024-03-11T21:00:48.459043+0000","last_became_peered":"2024-03-11T21:00:48.459043+0000","last_unstale":"2024-03-11T21:03:55.216842+0000","last_undegraded":"2024-03-11T21:03:55.216842+0000","last_fullsized":"2024-03-11T21:03:55.216842+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:43.605796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:23.137832+0000","last_clean_scrub_stamp":"2024-03-11T21:03:43.605796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:05:13.569217+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":469,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.230412+0000","last_change":"2024-03-11T21:03:36.409060+0000","last_active":"2024-03-11T21:03:55.230412+0000","last_peered":"2024-03-11T21:03:55.230412+0000","last_clean":"2024-03-11T21:03:55.230412+0000","last_became_active":"2024-03-11T20:50:28.082737+0000","last_became_peered":"2024-03-11T20:50:28.082737+0000","last_unstale":"2024-03-11T21:03:55.230412+0000","last_undegraded":"2024-03-11T21:03:55.230412+0000","last_fullsized":"2024-03-11T21:03:55.230412+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:36.408942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:28.404997+0000","last_clean_scrub_stamp":"2024-03-11T21:03:36.408942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:57.355001+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"207'15","reported_seq":484,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.228179+0000","last_change":"2024-03-11T21:03:47.989635+0000","last_active":"2024-03-11T21:03:55.228179+0000","last_peered":"2024-03-11T21:03:55.228179+0000","last_clean":"2024-03-11T21:03:55.228179+0000","last_became_active":"2024-03-11T21:00:27.967624+0000","last_became_peered":"2024-03-11T21:00:27.967624+0000","last_unstale":"2024-03-11T21:03:55.228179+0000","last_undegraded":"2024-03-11T21:03:55.228179+0000","last_fullsized":"2024-03-11T21:03:55.228179+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'15","last_scrub_stamp":"2024-03-11T21:03:47.989410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:03:47.989410+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:58.974121+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":366,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.214891+0000","last_change":"2024-03-11T21:03:44.322295+0000","last_active":"2024-03-11T21:03:55.214891+0000","last_peered":"2024-03-11T21:03:55.214891+0000","last_clean":"2024-03-11T21:03:55.214891+0000","last_became_active":"2024-03-11T21:00:27.958384+0000","last_became_peered":"2024-03-11T21:00:27.958384+0000","last_unstale":"2024-03-11T21:03:55.214891+0000","last_undegraded":"2024-03-11T21:03:55.214891+0000","last_fullsized":"2024-03-11T21:03:55.214891+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:44.322168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:57:16.032302+0000","last_clean_scrub_stamp":"2024-03-11T21:03:44.322168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:05:12.080668+0000","scrub_duration":59,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.15","version":"112'12","reported_seq":623,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:04:03.781509+0000","last_change":"2024-03-11T21:03:43.933651+0000","last_active":"2024-03-11T21:04:03.781509+0000","last_peered":"2024-03-11T21:04:03.781509+0000","last_clean":"2024-03-11T21:04:03.781509+0000","last_became_active":"2024-03-11T20:50:32.650064+0000","last_became_peered":"2024-03-11T20:50:32.650064+0000","last_unstale":"2024-03-11T21:04:03.781509+0000","last_undegraded":"2024-03-11T21:04:03.781509+0000","last_fullsized":"2024-03-11T21:04:03.781509+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'12","last_scrub_stamp":"2024-03-11T21:03:43.933551+0000","last_deep_scrub":"112'12","last_deep_scrub_stamp":"2024-03-11T20:53:11.843951+0000","last_clean_scrub_stamp":"2024-03-11T21:03:43.933551+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:49.815489+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":217,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.213777+0000","last_change":"2024-03-11T21:03:11.307482+0000","last_active":"2024-03-11T21:03:55.213777+0000","last_peered":"2024-03-11T21:03:55.213777+0000","last_clean":"2024-03-11T21:03:55.213777+0000","last_became_active":"2024-03-11T21:00:27.972907+0000","last_became_peered":"2024-03-11T21:00:27.972907+0000","last_unstale":"2024-03-11T21:03:55.213777+0000","last_undegraded":"2024-03-11T21:03:55.213777+0000","last_fullsized":"2024-03-11T21:03:55.213777+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:11.307332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:03:11.307332+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:37.141260+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.12","version":"207'19","reported_seq":398,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.223511+0000","last_change":"2024-03-11T21:02:30.944467+0000","last_active":"2024-03-11T21:03:55.223511+0000","last_peered":"2024-03-11T21:03:55.223511+0000","last_clean":"2024-03-11T21:03:55.223511+0000","last_became_active":"2024-03-11T20:56:59.796618+0000","last_became_peered":"2024-03-11T20:56:59.796618+0000","last_unstale":"2024-03-11T21:03:55.223511+0000","last_undegraded":"2024-03-11T21:03:55.223511+0000","last_fullsized":"2024-03-11T21:03:55.223511+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'19","last_scrub_stamp":"2024-03-11T21:02:30.944325+0000","last_deep_scrub":"119'13","last_deep_scrub_stamp":"2024-03-11T20:54:15.443472+0000","last_clean_scrub_stamp":"2024-03-11T21:02:30.944325+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:03.906070+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":262,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.228629+0000","last_change":"2024-03-11T21:03:10.124323+0000","last_active":"2024-03-11T21:03:55.228629+0000","last_peered":"2024-03-11T21:03:55.228629+0000","last_clean":"2024-03-11T21:03:55.228629+0000","last_became_active":"2024-03-11T20:56:59.804346+0000","last_became_peered":"2024-03-11T20:56:59.804346+0000","last_unstale":"2024-03-11T21:03:55.228629+0000","last_undegraded":"2024-03-11T21:03:55.228629+0000","last_fullsized":"2024-03-11T21:03:55.228629+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:10.124075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:03:10.124075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:32.103132+0000","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"212'17","reported_seq":645,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:58.777608+0000","last_change":"2024-03-11T21:02:25.533896+0000","last_active":"2024-03-11T21:03:58.777608+0000","last_peered":"2024-03-11T21:03:58.777608+0000","last_clean":"2024-03-11T21:03:58.777608+0000","last_became_active":"2024-03-11T21:00:48.467116+0000","last_became_peered":"2024-03-11T21:00:48.467116+0000","last_unstale":"2024-03-11T21:03:58.777608+0000","last_undegraded":"2024-03-11T21:03:58.777608+0000","last_fullsized":"2024-03-11T21:03:58.777608+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"212'17","last_scrub_stamp":"2024-03-11T21:02:25.533666+0000","last_deep_scrub":"110'13","last_deep_scrub_stamp":"2024-03-11T20:53:00.434532+0000","last_clean_scrub_stamp":"2024-03-11T21:02:25.533666+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:08.785162+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":463,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.222508+0000","last_change":"2024-03-11T21:03:29.061500+0000","last_active":"2024-03-11T21:03:55.222508+0000","last_peered":"2024-03-11T21:03:55.222508+0000","last_clean":"2024-03-11T21:03:55.222508+0000","last_became_active":"2024-03-11T20:50:28.070325+0000","last_became_peered":"2024-03-11T20:50:28.070325+0000","last_unstale":"2024-03-11T21:03:55.222508+0000","last_undegraded":"2024-03-11T21:03:55.222508+0000","last_fullsized":"2024-03-11T21:03:55.222508+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:29.061407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:03:29.061407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:47.973408+0000","scrub_duration":207,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"207'24","reported_seq":495,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.223970+0000","last_change":"2024-03-11T21:03:45.977452+0000","last_active":"2024-03-11T21:03:55.223970+0000","last_peered":"2024-03-11T21:03:55.223970+0000","last_clean":"2024-03-11T21:03:55.223970+0000","last_became_active":"2024-03-11T21:00:27.967078+0000","last_became_peered":"2024-03-11T21:00:27.967078+0000","last_unstale":"2024-03-11T21:03:55.223970+0000","last_undegraded":"2024-03-11T21:03:55.223970+0000","last_fullsized":"2024-03-11T21:03:55.223970+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'24","last_scrub_stamp":"2024-03-11T21:03:45.977203+0000","last_deep_scrub":"116'16","last_deep_scrub_stamp":"2024-03-11T20:53:39.292118+0000","last_clean_scrub_stamp":"2024-03-11T21:03:45.977203+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:56.354611+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":443,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.227692+0000","last_change":"2024-03-11T21:03:39.998420+0000","last_active":"2024-03-11T21:03:55.227692+0000","last_peered":"2024-03-11T21:03:55.227692+0000","last_clean":"2024-03-11T21:03:55.227692+0000","last_became_active":"2024-03-11T21:00:27.966997+0000","last_became_peered":"2024-03-11T21:00:27.966997+0000","last_unstale":"2024-03-11T21:03:55.227692+0000","last_undegraded":"2024-03-11T21:03:55.227692+0000","last_fullsized":"2024-03-11T21:03:55.227692+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:39.998251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:55:58.016988+0000","last_clean_scrub_stamp":"2024-03-11T21:03:39.998251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:50.834709+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":460,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.229992+0000","last_change":"2024-03-11T21:03:15.618457+0000","last_active":"2024-03-11T21:03:55.229992+0000","last_peered":"2024-03-11T21:03:55.229992+0000","last_clean":"2024-03-11T21:03:55.229992+0000","last_became_active":"2024-03-11T21:00:48.468386+0000","last_became_peered":"2024-03-11T21:00:48.468386+0000","last_unstale":"2024-03-11T21:03:55.229992+0000","last_undegraded":"2024-03-11T21:03:55.229992+0000","last_fullsized":"2024-03-11T21:03:55.229992+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:15.618335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:03:15.618335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:44.520777+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"166'2","reported_seq":496,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.229687+0000","last_change":"2024-03-11T21:03:12.535685+0000","last_active":"2024-03-11T21:03:55.229687+0000","last_peered":"2024-03-11T21:03:55.229687+0000","last_clean":"2024-03-11T21:03:55.229687+0000","last_became_active":"2024-03-11T21:00:48.466071+0000","last_became_peered":"2024-03-11T21:00:48.466071+0000","last_unstale":"2024-03-11T21:03:55.229687+0000","last_undegraded":"2024-03-11T21:03:55.229687+0000","last_fullsized":"2024-03-11T21:03:55.229687+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'2","last_scrub_stamp":"2024-03-11T21:03:12.535593+0000","last_deep_scrub":"166'2","last_deep_scrub_stamp":"2024-03-11T20:57:39.868776+0000","last_clean_scrub_stamp":"2024-03-11T21:03:12.535593+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:34.204233+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1822,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"207'8","reported_seq":469,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.229005+0000","last_change":"2024-03-11T21:03:50.252798+0000","last_active":"2024-03-11T21:03:55.229005+0000","last_peered":"2024-03-11T21:03:55.229005+0000","last_clean":"2024-03-11T21:03:55.229005+0000","last_became_active":"2024-03-11T20:50:30.101572+0000","last_became_peered":"2024-03-11T20:50:30.101572+0000","last_unstale":"2024-03-11T21:03:55.229005+0000","last_undegraded":"2024-03-11T21:03:55.229005+0000","last_fullsized":"2024-03-11T21:03:55.229005+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'8","last_scrub_stamp":"2024-03-11T21:03:50.252595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:03:50.252595+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:05:15.770439+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":462,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.233115+0000","last_change":"2024-03-11T21:03:09.700041+0000","last_active":"2024-03-11T21:03:55.233115+0000","last_peered":"2024-03-11T21:03:55.233115+0000","last_clean":"2024-03-11T21:03:55.233115+0000","last_became_active":"2024-03-11T21:00:27.964140+0000","last_became_peered":"2024-03-11T21:00:27.964140+0000","last_unstale":"2024-03-11T21:03:55.233115+0000","last_undegraded":"2024-03-11T21:03:55.233115+0000","last_fullsized":"2024-03-11T21:03:55.233115+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:09.699964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:03:09.699964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:29.206714+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":412,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.232420+0000","last_change":"2024-03-11T21:03:13.732388+0000","last_active":"2024-03-11T21:03:55.232420+0000","last_peered":"2024-03-11T21:03:55.232420+0000","last_clean":"2024-03-11T21:03:55.232420+0000","last_became_active":"2024-03-11T20:56:59.802356+0000","last_became_peered":"2024-03-11T20:56:59.802356+0000","last_unstale":"2024-03-11T21:03:55.232420+0000","last_undegraded":"2024-03-11T21:03:55.232420+0000","last_fullsized":"2024-03-11T21:03:55.232420+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:13.730932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:54:09.537972+0000","last_clean_scrub_stamp":"2024-03-11T21:03:13.730932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:15.932093+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":475,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.232018+0000","last_change":"2024-03-11T21:03:22.624489+0000","last_active":"2024-03-11T21:03:55.232018+0000","last_peered":"2024-03-11T21:03:55.232018+0000","last_clean":"2024-03-11T21:03:55.232018+0000","last_became_active":"2024-03-11T21:00:48.459557+0000","last_became_peered":"2024-03-11T21:00:48.459557+0000","last_unstale":"2024-03-11T21:03:55.232018+0000","last_undegraded":"2024-03-11T21:03:55.232018+0000","last_fullsized":"2024-03-11T21:03:55.232018+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:22.623477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:18.682947+0000","last_clean_scrub_stamp":"2024-03-11T21:03:22.623477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:23.783554+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"262'24","reported_seq":487,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.222643+0000","last_change":"2024-03-11T21:03:28.129057+0000","last_active":"2024-03-11T21:03:55.222643+0000","last_peered":"2024-03-11T21:03:55.222643+0000","last_clean":"2024-03-11T21:03:55.222643+0000","last_became_active":"2024-03-11T21:00:27.970878+0000","last_became_peered":"2024-03-11T21:00:27.970878+0000","last_unstale":"2024-03-11T21:03:55.222643+0000","last_undegraded":"2024-03-11T21:03:55.222643+0000","last_fullsized":"2024-03-11T21:03:55.222643+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"262'24","last_scrub_stamp":"2024-03-11T21:03:28.128719+0000","last_deep_scrub":"262'24","last_deep_scrub_stamp":"2024-03-11T21:03:28.128719+0000","last_clean_scrub_stamp":"2024-03-11T21:03:28.128719+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:36.970292+0000","scrub_duration":127,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.4","version":"273'5563","reported_seq":5617,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:04:06.879172+0000","last_change":"2024-03-11T21:02:47.636686+0000","last_active":"2024-03-11T21:04:06.879172+0000","last_peered":"2024-03-11T21:04:06.879172+0000","last_clean":"2024-03-11T21:04:06.879172+0000","last_became_active":"2024-03-11T21:02:47.634842+0000","last_became_peered":"2024-03-11T21:02:47.634842+0000","last_unstale":"2024-03-11T21:04:06.879172+0000","last_undegraded":"2024-03-11T21:04:06.879172+0000","last_fullsized":"2024-03-11T21:04:06.879172+0000","mapping_epoch":251,"log_start":"257'400","ondisk_log_start":"257'400","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:02:46.570492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:46.570492+0000","last_clean_scrub_stamp":"2024-03-11T21:02:46.570492+0000","objects_scrubbed":0,"log_size":5163,"log_dups_size":0,"ondisk_log_size":5163,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:11.625712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":364576768,"num_objects":5563,"num_object_clones":0,"num_object_copies":16689,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5563,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5563,"num_write_kb":356032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":457,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.224995+0000","last_change":"2024-03-11T21:03:14.642466+0000","last_active":"2024-03-11T21:03:55.224995+0000","last_peered":"2024-03-11T21:03:55.224995+0000","last_clean":"2024-03-11T21:03:55.224995+0000","last_became_active":"2024-03-11T21:00:48.466853+0000","last_became_peered":"2024-03-11T21:00:48.466853+0000","last_unstale":"2024-03-11T21:03:55.224995+0000","last_undegraded":"2024-03-11T21:03:55.224995+0000","last_fullsized":"2024-03-11T21:03:55.224995+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:14.642358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:52:53.450885+0000","last_clean_scrub_stamp":"2024-03-11T21:03:14.642358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:28.543091+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"77'2","reported_seq":405,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.214918+0000","last_change":"2024-03-11T21:03:51.299386+0000","last_active":"2024-03-11T21:03:55.214918+0000","last_peered":"2024-03-11T21:03:55.214918+0000","last_clean":"2024-03-11T21:03:55.214918+0000","last_became_active":"2024-03-11T21:00:27.964018+0000","last_became_peered":"2024-03-11T21:00:27.964018+0000","last_unstale":"2024-03-11T21:03:55.214918+0000","last_undegraded":"2024-03-11T21:03:55.214918+0000","last_fullsized":"2024-03-11T21:03:55.214918+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'2","last_scrub_stamp":"2024-03-11T21:03:51.299300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:03:51.299300+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:05:01.172859+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"207'24","reported_seq":548,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.228766+0000","last_change":"2024-03-11T21:03:23.941107+0000","last_active":"2024-03-11T21:03:55.228766+0000","last_peered":"2024-03-11T21:03:55.228766+0000","last_clean":"2024-03-11T21:03:55.228766+0000","last_became_active":"2024-03-11T21:00:48.465254+0000","last_became_peered":"2024-03-11T21:00:48.465254+0000","last_unstale":"2024-03-11T21:03:55.228766+0000","last_undegraded":"2024-03-11T21:03:55.228766+0000","last_fullsized":"2024-03-11T21:03:55.228766+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'24","last_scrub_stamp":"2024-03-11T21:03:23.940968+0000","last_deep_scrub":"100'16","last_deep_scrub_stamp":"2024-03-11T20:56:08.036331+0000","last_clean_scrub_stamp":"2024-03-11T21:03:23.940968+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:33.812875+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"212'17","reported_seq":567,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:58.777596+0000","last_change":"2024-03-11T21:03:38.259033+0000","last_active":"2024-03-11T21:03:58.777596+0000","last_peered":"2024-03-11T21:03:58.777596+0000","last_clean":"2024-03-11T21:03:58.777596+0000","last_became_active":"2024-03-11T21:00:48.461768+0000","last_became_peered":"2024-03-11T21:00:48.461768+0000","last_unstale":"2024-03-11T21:03:58.777596+0000","last_undegraded":"2024-03-11T21:03:58.777596+0000","last_fullsized":"2024-03-11T21:03:58.777596+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"212'17","last_scrub_stamp":"2024-03-11T21:03:38.258850+0000","last_deep_scrub":"212'17","last_deep_scrub_stamp":"2024-03-11T21:03:38.258850+0000","last_clean_scrub_stamp":"2024-03-11T21:03:38.258850+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:55.059529+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.f","version":"273'5585","reported_seq":5639,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:04:10.094233+0000","last_change":"2024-03-11T21:02:47.637043+0000","last_active":"2024-03-11T21:04:10.094233+0000","last_peered":"2024-03-11T21:04:10.094233+0000","last_clean":"2024-03-11T21:04:10.094233+0000","last_became_active":"2024-03-11T21:02:47.636634+0000","last_became_peered":"2024-03-11T21:02:47.636634+0000","last_unstale":"2024-03-11T21:04:10.094233+0000","last_undegraded":"2024-03-11T21:04:10.094233+0000","last_fullsized":"2024-03-11T21:04:10.094233+0000","mapping_epoch":251,"log_start":"257'200","ondisk_log_start":"257'200","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:02:46.570492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:46.570492+0000","last_clean_scrub_stamp":"2024-03-11T21:02:46.570492+0000","objects_scrubbed":0,"log_size":5385,"log_dups_size":0,"ondisk_log_size":5385,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:12.432576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":366018560,"num_objects":5585,"num_object_clones":0,"num_object_copies":16755,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5585,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5585,"num_write_kb":357440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":265,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.221972+0000","last_change":"2024-03-11T21:00:24.793264+0000","last_active":"2024-03-11T21:03:55.221972+0000","last_peered":"2024-03-11T21:03:55.221972+0000","last_clean":"2024-03-11T21:03:55.221972+0000","last_became_active":"2024-03-11T20:56:59.795191+0000","last_became_peered":"2024-03-11T20:56:59.795191+0000","last_unstale":"2024-03-11T21:03:55.221972+0000","last_undegraded":"2024-03-11T21:03:55.221972+0000","last_fullsized":"2024-03-11T21:03:55.221972+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:00:24.793093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:57:21.568400+0000","last_clean_scrub_stamp":"2024-03-11T21:00:24.793093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:03:57.806517+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":162,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.217358+0000","last_change":"2024-03-11T21:03:40.610888+0000","last_active":"2024-03-11T21:03:55.217358+0000","last_peered":"2024-03-11T21:03:55.217358+0000","last_clean":"2024-03-11T21:03:55.217358+0000","last_became_active":"2024-03-11T21:00:48.455733+0000","last_became_peered":"2024-03-11T21:00:48.455733+0000","last_unstale":"2024-03-11T21:03:55.217358+0000","last_undegraded":"2024-03-11T21:03:55.217358+0000","last_fullsized":"2024-03-11T21:03:55.217358+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:40.610810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:20.096874+0000","last_clean_scrub_stamp":"2024-03-11T21:03:40.610810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:59.174036+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"207'28","reported_seq":657,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:04:03.781738+0000","last_change":"2024-03-11T21:03:15.735024+0000","last_active":"2024-03-11T21:04:03.781738+0000","last_peered":"2024-03-11T21:04:03.781738+0000","last_clean":"2024-03-11T21:04:03.781738+0000","last_became_active":"2024-03-11T20:50:30.094977+0000","last_became_peered":"2024-03-11T20:50:30.094977+0000","last_unstale":"2024-03-11T21:04:03.781738+0000","last_undegraded":"2024-03-11T21:04:03.781738+0000","last_fullsized":"2024-03-11T21:04:03.781738+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'28","last_scrub_stamp":"2024-03-11T21:03:15.734883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:03:15.734883+0000","objects_scrubbed":8,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:29.110040+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":154,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.216937+0000","last_change":"2024-03-11T21:02:30.288033+0000","last_active":"2024-03-11T21:03:55.216937+0000","last_peered":"2024-03-11T21:03:55.216937+0000","last_clean":"2024-03-11T21:03:55.216937+0000","last_became_active":"2024-03-11T21:00:48.457901+0000","last_became_peered":"2024-03-11T21:00:48.457901+0000","last_unstale":"2024-03-11T21:03:55.216937+0000","last_undegraded":"2024-03-11T21:03:55.216937+0000","last_fullsized":"2024-03-11T21:03:55.216937+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:02:30.287958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:52:10.160202+0000","last_clean_scrub_stamp":"2024-03-11T21:02:30.287958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"273'5708","reported_seq":5762,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:04:10.050700+0000","last_change":"2024-03-11T21:02:47.621307+0000","last_active":"2024-03-11T21:04:10.050700+0000","last_peered":"2024-03-11T21:04:10.050700+0000","last_clean":"2024-03-11T21:04:10.050700+0000","last_became_active":"2024-03-11T21:02:47.619716+0000","last_became_peered":"2024-03-11T21:02:47.619716+0000","last_unstale":"2024-03-11T21:04:10.050700+0000","last_undegraded":"2024-03-11T21:04:10.050700+0000","last_fullsized":"2024-03-11T21:04:10.050700+0000","mapping_epoch":251,"log_start":"259'1000","ondisk_log_start":"259'1000","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:02:46.570492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:46.570492+0000","last_clean_scrub_stamp":"2024-03-11T21:02:46.570492+0000","objects_scrubbed":0,"log_size":4708,"log_dups_size":0,"ondisk_log_size":4708,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:15.030392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":374079488,"num_objects":5708,"num_object_clones":0,"num_object_copies":17124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5708,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5708,"num_write_kb":365312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":445,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.224888+0000","last_change":"2024-03-11T21:03:28.677362+0000","last_active":"2024-03-11T21:03:55.224888+0000","last_peered":"2024-03-11T21:03:55.224888+0000","last_clean":"2024-03-11T21:03:55.224888+0000","last_became_active":"2024-03-11T21:00:27.959408+0000","last_became_peered":"2024-03-11T21:00:27.959408+0000","last_unstale":"2024-03-11T21:03:55.224888+0000","last_undegraded":"2024-03-11T21:03:55.224888+0000","last_fullsized":"2024-03-11T21:03:55.224888+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:28.677276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:03:28.677276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:49.954168+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":469,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.222569+0000","last_change":"2024-03-11T21:02:29.807630+0000","last_active":"2024-03-11T21:03:55.222569+0000","last_peered":"2024-03-11T21:03:55.222569+0000","last_clean":"2024-03-11T21:03:55.222569+0000","last_became_active":"2024-03-11T21:00:48.471212+0000","last_became_peered":"2024-03-11T21:00:48.471212+0000","last_unstale":"2024-03-11T21:03:55.222569+0000","last_undegraded":"2024-03-11T21:03:55.222569+0000","last_fullsized":"2024-03-11T21:03:55.222569+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:02:29.807044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:33.823704+0000","last_clean_scrub_stamp":"2024-03-11T21:02:29.807044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:03:57.303287+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"207'37","reported_seq":476,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.213685+0000","last_change":"2024-03-11T21:03:22.320789+0000","last_active":"2024-03-11T21:03:55.213685+0000","last_peered":"2024-03-11T21:03:55.213685+0000","last_clean":"2024-03-11T21:03:55.213685+0000","last_became_active":"2024-03-11T21:00:27.960377+0000","last_became_peered":"2024-03-11T21:00:27.960377+0000","last_unstale":"2024-03-11T21:03:55.213685+0000","last_undegraded":"2024-03-11T21:03:55.213685+0000","last_fullsized":"2024-03-11T21:03:55.213685+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'37","last_scrub_stamp":"2024-03-11T21:03:22.320643+0000","last_deep_scrub":"182'27","last_deep_scrub_stamp":"2024-03-11T20:58:52.058851+0000","last_clean_scrub_stamp":"2024-03-11T21:03:22.320643+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:34.075583+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":162,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.216989+0000","last_change":"2024-03-11T21:03:44.649559+0000","last_active":"2024-03-11T21:03:55.216989+0000","last_peered":"2024-03-11T21:03:55.216989+0000","last_clean":"2024-03-11T21:03:55.216989+0000","last_became_active":"2024-03-11T21:00:48.458008+0000","last_became_peered":"2024-03-11T21:00:48.458008+0000","last_unstale":"2024-03-11T21:03:55.216989+0000","last_undegraded":"2024-03-11T21:03:55.216989+0000","last_fullsized":"2024-03-11T21:03:55.216989+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:44.649310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:33.319221+0000","last_clean_scrub_stamp":"2024-03-11T21:03:44.649310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:05:08.200071+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.a","version":"273'5566","reported_seq":5620,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:04:10.083348+0000","last_change":"2024-03-11T21:02:47.634170+0000","last_active":"2024-03-11T21:04:10.083348+0000","last_peered":"2024-03-11T21:04:10.083348+0000","last_clean":"2024-03-11T21:04:10.083348+0000","last_became_active":"2024-03-11T21:02:47.633164+0000","last_became_peered":"2024-03-11T21:02:47.633164+0000","last_unstale":"2024-03-11T21:04:10.083348+0000","last_undegraded":"2024-03-11T21:04:10.083348+0000","last_fullsized":"2024-03-11T21:04:10.083348+0000","mapping_epoch":251,"log_start":"257'200","ondisk_log_start":"257'200","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:02:46.570492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:46.570492+0000","last_clean_scrub_stamp":"2024-03-11T21:02:46.570492+0000","objects_scrubbed":0,"log_size":5366,"log_dups_size":0,"ondisk_log_size":5366,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:14.147819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":364773376,"num_objects":5566,"num_object_clones":0,"num_object_copies":16698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5566,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5566,"num_write_kb":356224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":443,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.227781+0000","last_change":"2024-03-11T21:02:32.951250+0000","last_active":"2024-03-11T21:03:55.227781+0000","last_peered":"2024-03-11T21:03:55.227781+0000","last_clean":"2024-03-11T21:03:55.227781+0000","last_became_active":"2024-03-11T20:50:34.321038+0000","last_became_peered":"2024-03-11T20:50:34.321038+0000","last_unstale":"2024-03-11T21:03:55.227781+0000","last_undegraded":"2024-03-11T21:03:55.227781+0000","last_fullsized":"2024-03-11T21:03:55.227781+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:02:32.951104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:32.951104+0000","last_clean_scrub_stamp":"2024-03-11T21:02:32.951104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:03:56.881929+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"207'38","reported_seq":186,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.216592+0000","last_change":"2024-03-11T21:03:06.361379+0000","last_active":"2024-03-11T21:03:55.216592+0000","last_peered":"2024-03-11T21:03:55.216592+0000","last_clean":"2024-03-11T21:03:55.216592+0000","last_became_active":"2024-03-11T21:00:48.456178+0000","last_became_peered":"2024-03-11T21:00:48.456178+0000","last_unstale":"2024-03-11T21:03:55.216592+0000","last_undegraded":"2024-03-11T21:03:55.216592+0000","last_fullsized":"2024-03-11T21:03:55.216592+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'38","last_scrub_stamp":"2024-03-11T21:03:06.361192+0000","last_deep_scrub":"109'26","last_deep_scrub_stamp":"2024-03-11T20:52:49.216378+0000","last_clean_scrub_stamp":"2024-03-11T21:03:06.361192+0000","objects_scrubbed":8,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:33.252260+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":471,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.232260+0000","last_change":"2024-03-11T21:03:19.688224+0000","last_active":"2024-03-11T21:03:55.232260+0000","last_peered":"2024-03-11T21:03:55.232260+0000","last_clean":"2024-03-11T21:03:55.232260+0000","last_became_active":"2024-03-11T20:50:28.045625+0000","last_became_peered":"2024-03-11T20:50:28.045625+0000","last_unstale":"2024-03-11T21:03:55.232260+0000","last_undegraded":"2024-03-11T21:03:55.232260+0000","last_fullsized":"2024-03-11T21:03:55.232260+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:19.688076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:03:19.688076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:26.035370+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":154,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.217420+0000","last_change":"2024-03-11T21:03:11.331958+0000","last_active":"2024-03-11T21:03:55.217420+0000","last_peered":"2024-03-11T21:03:55.217420+0000","last_clean":"2024-03-11T21:03:55.217420+0000","last_became_active":"2024-03-11T21:00:48.473346+0000","last_became_peered":"2024-03-11T21:00:48.473346+0000","last_unstale":"2024-03-11T21:03:55.217420+0000","last_undegraded":"2024-03-11T21:03:55.217420+0000","last_fullsized":"2024-03-11T21:03:55.217420+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:11.331883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:03:11.331883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:14.644222+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"273'5272","reported_seq":5326,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:04:06.127889+0000","last_change":"2024-03-11T21:02:47.639919+0000","last_active":"2024-03-11T21:04:06.127889+0000","last_peered":"2024-03-11T21:04:06.127889+0000","last_clean":"2024-03-11T21:04:06.127889+0000","last_became_active":"2024-03-11T21:02:47.639328+0000","last_became_peered":"2024-03-11T21:02:47.639328+0000","last_unstale":"2024-03-11T21:04:06.127889+0000","last_undegraded":"2024-03-11T21:04:06.127889+0000","last_fullsized":"2024-03-11T21:04:06.127889+0000","mapping_epoch":251,"log_start":"258'600","ondisk_log_start":"258'600","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:02:46.570492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:46.570492+0000","last_clean_scrub_stamp":"2024-03-11T21:02:46.570492+0000","objects_scrubbed":0,"log_size":4672,"log_dups_size":0,"ondisk_log_size":4672,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:06.827406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":345505792,"num_objects":5272,"num_object_clones":0,"num_object_copies":15816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5272,"num_write_kb":337408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":445,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.230289+0000","last_change":"2024-03-11T21:03:11.800852+0000","last_active":"2024-03-11T21:03:55.230289+0000","last_peered":"2024-03-11T21:03:55.230289+0000","last_clean":"2024-03-11T21:03:55.230289+0000","last_became_active":"2024-03-11T21:00:27.962668+0000","last_became_peered":"2024-03-11T21:00:27.962668+0000","last_unstale":"2024-03-11T21:03:55.230289+0000","last_undegraded":"2024-03-11T21:03:55.230289+0000","last_fullsized":"2024-03-11T21:03:55.230289+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:11.800717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:03:11.800717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:28.883743+0000","scrub_duration":83,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"207'20","reported_seq":474,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.223281+0000","last_change":"2024-03-11T21:00:14.736355+0000","last_active":"2024-03-11T21:03:55.223281+0000","last_peered":"2024-03-11T21:03:55.223281+0000","last_clean":"2024-03-11T21:03:55.223281+0000","last_became_active":"2024-03-11T20:50:30.101747+0000","last_became_peered":"2024-03-11T20:50:30.101747+0000","last_unstale":"2024-03-11T21:03:55.223281+0000","last_undegraded":"2024-03-11T21:03:55.223281+0000","last_fullsized":"2024-03-11T21:03:55.223281+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'10","last_scrub_stamp":"2024-03-11T21:00:14.736261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:00:14.736261+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:03:57.806383+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":464,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.223023+0000","last_change":"2024-03-11T21:03:47.212843+0000","last_active":"2024-03-11T21:03:55.223023+0000","last_peered":"2024-03-11T21:03:55.223023+0000","last_clean":"2024-03-11T21:03:55.223023+0000","last_became_active":"2024-03-11T20:50:28.085503+0000","last_became_peered":"2024-03-11T20:50:28.085503+0000","last_unstale":"2024-03-11T21:03:55.223023+0000","last_undegraded":"2024-03-11T21:03:55.223023+0000","last_fullsized":"2024-03-11T21:03:55.223023+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:47.212723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:56:03.479180+0000","last_clean_scrub_stamp":"2024-03-11T21:03:47.212723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:55.569265+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"184'130","reported_seq":724,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.220814+0000","last_change":"2024-03-11T21:02:15.792152+0000","last_active":"2024-03-11T21:03:55.220814+0000","last_peered":"2024-03-11T21:03:55.220814+0000","last_clean":"2024-03-11T21:03:55.220814+0000","last_became_active":"2024-03-11T20:49:52.584543+0000","last_became_peered":"2024-03-11T20:49:52.584543+0000","last_unstale":"2024-03-11T21:03:55.220814+0000","last_undegraded":"2024-03-11T21:03:55.220814+0000","last_fullsized":"2024-03-11T21:03:55.220814+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"184'130","last_scrub_stamp":"2024-03-11T21:02:15.791905+0000","last_deep_scrub":"184'130","last_deep_scrub_stamp":"2024-03-11T20:58:56.415267+0000","last_clean_scrub_stamp":"2024-03-11T21:02:15.791905+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:03.906002+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":463,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.230721+0000","last_change":"2024-03-11T21:03:17.506757+0000","last_active":"2024-03-11T21:03:55.230721+0000","last_peered":"2024-03-11T21:03:55.230721+0000","last_clean":"2024-03-11T21:03:55.230721+0000","last_became_active":"2024-03-11T21:00:27.959467+0000","last_became_peered":"2024-03-11T21:00:27.959467+0000","last_unstale":"2024-03-11T21:03:55.230721+0000","last_undegraded":"2024-03-11T21:03:55.230721+0000","last_fullsized":"2024-03-11T21:03:55.230721+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:17.506583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:54:22.448812+0000","last_clean_scrub_stamp":"2024-03-11T21:03:17.506583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:26.066370+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.b","version":"273'5613","reported_seq":5667,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:04:10.031425+0000","last_change":"2024-03-11T21:02:47.622120+0000","last_active":"2024-03-11T21:04:10.031425+0000","last_peered":"2024-03-11T21:04:10.031425+0000","last_clean":"2024-03-11T21:04:10.031425+0000","last_became_active":"2024-03-11T21:02:47.621751+0000","last_became_peered":"2024-03-11T21:02:47.621751+0000","last_unstale":"2024-03-11T21:04:10.031425+0000","last_undegraded":"2024-03-11T21:04:10.031425+0000","last_fullsized":"2024-03-11T21:04:10.031425+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:02:46.570492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:46.570492+0000","last_clean_scrub_stamp":"2024-03-11T21:02:46.570492+0000","objects_scrubbed":0,"log_size":5613,"log_dups_size":0,"ondisk_log_size":5613,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:10.159517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":367853568,"num_objects":5613,"num_object_clones":0,"num_object_copies":16839,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5613,"num_write_kb":359232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":154,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.217281+0000","last_change":"2024-03-11T21:02:25.182012+0000","last_active":"2024-03-11T21:03:55.217281+0000","last_peered":"2024-03-11T21:03:55.217281+0000","last_clean":"2024-03-11T21:03:55.217281+0000","last_became_active":"2024-03-11T21:00:48.464596+0000","last_became_peered":"2024-03-11T21:00:48.464596+0000","last_unstale":"2024-03-11T21:03:55.217281+0000","last_undegraded":"2024-03-11T21:03:55.217281+0000","last_fullsized":"2024-03-11T21:03:55.217281+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:02:25.181915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:02:25.181915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:03:58.572981+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"207'35","reported_seq":325,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.213323+0000","last_change":"2024-03-11T21:02:32.429640+0000","last_active":"2024-03-11T21:03:55.213323+0000","last_peered":"2024-03-11T21:03:55.213323+0000","last_clean":"2024-03-11T21:03:55.213323+0000","last_became_active":"2024-03-11T21:00:27.959172+0000","last_became_peered":"2024-03-11T21:00:27.959172+0000","last_unstale":"2024-03-11T21:03:55.213323+0000","last_undegraded":"2024-03-11T21:03:55.213323+0000","last_fullsized":"2024-03-11T21:03:55.213323+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'35","last_scrub_stamp":"2024-03-11T21:02:32.429525+0000","last_deep_scrub":"100'21","last_deep_scrub_stamp":"2024-03-11T20:51:31.835302+0000","last_clean_scrub_stamp":"2024-03-11T21:02:32.429525+0000","objects_scrubbed":7,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:02.199696+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":442,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.232227+0000","last_change":"2024-03-11T21:03:48.784865+0000","last_active":"2024-03-11T21:03:55.232227+0000","last_peered":"2024-03-11T21:03:55.232227+0000","last_clean":"2024-03-11T21:03:55.232227+0000","last_became_active":"2024-03-11T21:00:48.458579+0000","last_became_peered":"2024-03-11T21:00:48.458579+0000","last_unstale":"2024-03-11T21:03:55.232227+0000","last_undegraded":"2024-03-11T21:03:55.232227+0000","last_fullsized":"2024-03-11T21:03:55.232227+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:48.784648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:03:48.784648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:05:02.043800+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":466,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.223564+0000","last_change":"2024-03-11T21:03:45.421629+0000","last_active":"2024-03-11T21:03:55.223564+0000","last_peered":"2024-03-11T21:03:55.223564+0000","last_clean":"2024-03-11T21:03:55.223564+0000","last_became_active":"2024-03-11T21:00:48.463247+0000","last_became_peered":"2024-03-11T21:00:48.463247+0000","last_unstale":"2024-03-11T21:03:55.223564+0000","last_undegraded":"2024-03-11T21:03:55.223564+0000","last_fullsized":"2024-03-11T21:03:55.223564+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:45.421446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:26.839782+0000","last_clean_scrub_stamp":"2024-03-11T21:03:45.421446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:52.927501+0000","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.e","version":"273'5688","reported_seq":5742,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:04:10.042830+0000","last_change":"2024-03-11T21:02:47.617410+0000","last_active":"2024-03-11T21:04:10.042830+0000","last_peered":"2024-03-11T21:04:10.042830+0000","last_clean":"2024-03-11T21:04:10.042830+0000","last_became_active":"2024-03-11T21:02:47.616727+0000","last_became_peered":"2024-03-11T21:02:47.616727+0000","last_unstale":"2024-03-11T21:04:10.042830+0000","last_undegraded":"2024-03-11T21:04:10.042830+0000","last_fullsized":"2024-03-11T21:04:10.042830+0000","mapping_epoch":251,"log_start":"259'1000","ondisk_log_start":"259'1000","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:02:46.570492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:46.570492+0000","last_clean_scrub_stamp":"2024-03-11T21:02:46.570492+0000","objects_scrubbed":0,"log_size":4688,"log_dups_size":0,"ondisk_log_size":4688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:15.030433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":372768768,"num_objects":5688,"num_object_clones":0,"num_object_copies":17064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5688,"num_write_kb":364032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":162,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.216611+0000","last_change":"2024-03-11T21:03:28.495517+0000","last_active":"2024-03-11T21:03:55.216611+0000","last_peered":"2024-03-11T21:03:55.216611+0000","last_clean":"2024-03-11T21:03:55.216611+0000","last_became_active":"2024-03-11T21:00:48.469688+0000","last_became_peered":"2024-03-11T21:00:48.469688+0000","last_unstale":"2024-03-11T21:03:55.216611+0000","last_undegraded":"2024-03-11T21:03:55.216611+0000","last_fullsized":"2024-03-11T21:03:55.216611+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:28.495308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:03:28.495308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:45.445522+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"145'3","reported_seq":464,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.229915+0000","last_change":"2024-03-11T21:03:44.305877+0000","last_active":"2024-03-11T21:03:55.229915+0000","last_peered":"2024-03-11T21:03:55.229915+0000","last_clean":"2024-03-11T21:03:55.229915+0000","last_became_active":"2024-03-11T20:50:32.195452+0000","last_became_peered":"2024-03-11T20:50:32.195452+0000","last_unstale":"2024-03-11T21:03:55.229915+0000","last_undegraded":"2024-03-11T21:03:55.229915+0000","last_fullsized":"2024-03-11T21:03:55.229915+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'3","last_scrub_stamp":"2024-03-11T21:03:44.305716+0000","last_deep_scrub":"145'3","last_deep_scrub_stamp":"2024-03-11T20:55:52.848043+0000","last_clean_scrub_stamp":"2024-03-11T21:03:44.305716+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:47.051450+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0.0024049620000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":456,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.227924+0000","last_change":"2024-03-11T21:02:04.522844+0000","last_active":"2024-03-11T21:03:55.227924+0000","last_peered":"2024-03-11T21:03:55.227924+0000","last_clean":"2024-03-11T21:03:55.227924+0000","last_became_active":"2024-03-11T21:00:27.966749+0000","last_became_peered":"2024-03-11T21:00:27.966749+0000","last_unstale":"2024-03-11T21:03:55.227924+0000","last_undegraded":"2024-03-11T21:03:55.227924+0000","last_fullsized":"2024-03-11T21:03:55.227924+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:02:04.522743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:02:04.522743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:03:56.881878+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"212'46","reported_seq":231,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:58.777026+0000","last_change":"2024-03-11T21:03:12.324530+0000","last_active":"2024-03-11T21:03:58.777026+0000","last_peered":"2024-03-11T21:03:58.777026+0000","last_clean":"2024-03-11T21:03:58.777026+0000","last_became_active":"2024-03-11T21:00:48.473227+0000","last_became_peered":"2024-03-11T21:00:48.473227+0000","last_unstale":"2024-03-11T21:03:58.777026+0000","last_undegraded":"2024-03-11T21:03:58.777026+0000","last_fullsized":"2024-03-11T21:03:58.777026+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"212'46","last_scrub_stamp":"2024-03-11T21:03:12.324431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:03:12.324431+0000","objects_scrubbed":10,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:37.435182+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":69,"num_read_kb":54,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"160'12","reported_seq":377,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:04:03.781549+0000","last_change":"2024-03-11T21:03:40.870833+0000","last_active":"2024-03-11T21:04:03.781549+0000","last_peered":"2024-03-11T21:04:03.781549+0000","last_clean":"2024-03-11T21:04:03.781549+0000","last_became_active":"2024-03-11T20:56:59.797805+0000","last_became_peered":"2024-03-11T20:56:59.797805+0000","last_unstale":"2024-03-11T21:04:03.781549+0000","last_undegraded":"2024-03-11T21:04:03.781549+0000","last_fullsized":"2024-03-11T21:04:03.781549+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"160'12","last_scrub_stamp":"2024-03-11T21:03:40.870714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:03:40.870714+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:53.829709+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"273'5621","reported_seq":5674,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:04:10.037600+0000","last_change":"2024-03-11T21:02:47.623642+0000","last_active":"2024-03-11T21:04:10.037600+0000","last_peered":"2024-03-11T21:04:10.037600+0000","last_clean":"2024-03-11T21:04:10.037600+0000","last_became_active":"2024-03-11T21:02:47.623292+0000","last_became_peered":"2024-03-11T21:02:47.623292+0000","last_unstale":"2024-03-11T21:04:10.037600+0000","last_undegraded":"2024-03-11T21:04:10.037600+0000","last_fullsized":"2024-03-11T21:04:10.037600+0000","mapping_epoch":251,"log_start":"259'1000","ondisk_log_start":"259'1000","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:02:46.570492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:46.570492+0000","last_clean_scrub_stamp":"2024-03-11T21:02:46.570492+0000","objects_scrubbed":0,"log_size":4621,"log_dups_size":0,"ondisk_log_size":4621,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:15.022647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":368377856,"num_objects":5621,"num_object_clones":0,"num_object_copies":16863,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5621,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5621,"num_write_kb":359744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":367,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.213402+0000","last_change":"2024-03-11T21:03:43.345403+0000","last_active":"2024-03-11T21:03:55.213402+0000","last_peered":"2024-03-11T21:03:55.213402+0000","last_clean":"2024-03-11T21:03:55.213402+0000","last_became_active":"2024-03-11T21:00:27.961447+0000","last_became_peered":"2024-03-11T21:00:27.961447+0000","last_unstale":"2024-03-11T21:03:55.213402+0000","last_undegraded":"2024-03-11T21:03:55.213402+0000","last_fullsized":"2024-03-11T21:03:55.213402+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:43.345257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:03:43.345257+0000","last_clean_scrub_stamp":"2024-03-11T21:03:43.345257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:44.667497+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":380,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.213211+0000","last_change":"2024-03-11T21:02:22.618230+0000","last_active":"2024-03-11T21:03:55.213211+0000","last_peered":"2024-03-11T21:03:55.213211+0000","last_clean":"2024-03-11T21:03:55.213211+0000","last_became_active":"2024-03-11T21:00:27.967546+0000","last_became_peered":"2024-03-11T21:00:27.967546+0000","last_unstale":"2024-03-11T21:03:55.213211+0000","last_undegraded":"2024-03-11T21:03:55.213211+0000","last_fullsized":"2024-03-11T21:03:55.213211+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:02:22.617999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:22.617999+0000","last_clean_scrub_stamp":"2024-03-11T21:02:22.617999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:03:58.303159+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"77'1","reported_seq":158,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.218771+0000","last_change":"2024-03-11T21:03:07.312601+0000","last_active":"2024-03-11T21:03:55.218771+0000","last_peered":"2024-03-11T21:03:55.218771+0000","last_clean":"2024-03-11T21:03:55.218771+0000","last_became_active":"2024-03-11T21:00:48.468682+0000","last_became_peered":"2024-03-11T21:00:48.468682+0000","last_unstale":"2024-03-11T21:03:55.218771+0000","last_undegraded":"2024-03-11T21:03:55.218771+0000","last_fullsized":"2024-03-11T21:03:55.218771+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-03-11T21:03:07.312455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:03:07.312455+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:13.125069+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"207'17","reported_seq":477,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.229743+0000","last_change":"2024-03-11T21:03:42.690547+0000","last_active":"2024-03-11T21:03:55.229743+0000","last_peered":"2024-03-11T21:03:55.229743+0000","last_clean":"2024-03-11T21:03:55.229743+0000","last_became_active":"2024-03-11T20:50:30.097669+0000","last_became_peered":"2024-03-11T20:50:30.097669+0000","last_unstale":"2024-03-11T21:03:55.229743+0000","last_undegraded":"2024-03-11T21:03:55.229743+0000","last_fullsized":"2024-03-11T21:03:55.229743+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'17","last_scrub_stamp":"2024-03-11T21:03:42.690411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:03:42.690411+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:05:11.885193+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"84'11","reported_seq":622,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:58.777573+0000","last_change":"2024-03-11T21:00:20.771711+0000","last_active":"2024-03-11T21:03:58.777573+0000","last_peered":"2024-03-11T21:03:58.777573+0000","last_clean":"2024-03-11T21:03:58.777573+0000","last_became_active":"2024-03-11T20:50:32.647041+0000","last_became_peered":"2024-03-11T20:50:32.647041+0000","last_unstale":"2024-03-11T21:03:58.777573+0000","last_undegraded":"2024-03-11T21:03:58.777573+0000","last_fullsized":"2024-03-11T21:03:58.777573+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:00:20.771539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:00:20.771539+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:08.778670+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.c","version":"273'5451","reported_seq":5505,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:04:06.126780+0000","last_change":"2024-03-11T21:02:47.630285+0000","last_active":"2024-03-11T21:04:06.126780+0000","last_peered":"2024-03-11T21:04:06.126780+0000","last_clean":"2024-03-11T21:04:06.126780+0000","last_became_active":"2024-03-11T21:02:47.629442+0000","last_became_peered":"2024-03-11T21:02:47.629442+0000","last_unstale":"2024-03-11T21:04:06.126780+0000","last_undegraded":"2024-03-11T21:04:06.126780+0000","last_fullsized":"2024-03-11T21:04:06.126780+0000","mapping_epoch":251,"log_start":"258'800","ondisk_log_start":"258'800","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:02:46.570492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:46.570492+0000","last_clean_scrub_stamp":"2024-03-11T21:02:46.570492+0000","objects_scrubbed":0,"log_size":4651,"log_dups_size":0,"ondisk_log_size":4651,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:08.785113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":357236736,"num_objects":5451,"num_object_clones":0,"num_object_copies":16353,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5451,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5451,"num_write_kb":348864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":451,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.223442+0000","last_change":"2024-03-11T21:03:09.148007+0000","last_active":"2024-03-11T21:03:55.223442+0000","last_peered":"2024-03-11T21:03:55.223442+0000","last_clean":"2024-03-11T21:03:55.223442+0000","last_became_active":"2024-03-11T20:50:34.326947+0000","last_became_peered":"2024-03-11T20:50:34.326947+0000","last_unstale":"2024-03-11T21:03:55.223442+0000","last_undegraded":"2024-03-11T21:03:55.223442+0000","last_fullsized":"2024-03-11T21:03:55.223442+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:09.147923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:03:09.147923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:36.318911+0000","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"120'6","reported_seq":1171,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:04:04.905791+0000","last_change":"2024-03-11T21:02:19.475999+0000","last_active":"2024-03-11T21:04:04.905791+0000","last_peered":"2024-03-11T21:04:04.905791+0000","last_clean":"2024-03-11T21:04:04.905791+0000","last_became_active":"2024-03-11T21:00:48.468997+0000","last_became_peered":"2024-03-11T21:00:48.468997+0000","last_unstale":"2024-03-11T21:04:04.905791+0000","last_undegraded":"2024-03-11T21:04:04.905791+0000","last_fullsized":"2024-03-11T21:04:04.905791+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"120'6","last_scrub_stamp":"2024-03-11T21:02:19.475892+0000","last_deep_scrub":"120'6","last_deep_scrub_stamp":"2024-03-11T20:54:26.468057+0000","last_clean_scrub_stamp":"2024-03-11T21:02:19.475892+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:08.778905+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0035533370000000002,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":682,"num_read_kb":677,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":470,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.230957+0000","last_change":"2024-03-11T21:03:26.508296+0000","last_active":"2024-03-11T21:03:55.230957+0000","last_peered":"2024-03-11T21:03:55.230957+0000","last_clean":"2024-03-11T21:03:55.230957+0000","last_became_active":"2024-03-11T20:50:28.071707+0000","last_became_peered":"2024-03-11T20:50:28.071707+0000","last_unstale":"2024-03-11T21:03:55.230957+0000","last_undegraded":"2024-03-11T21:03:55.230957+0000","last_fullsized":"2024-03-11T21:03:55.230957+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:26.508215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:23.417283+0000","last_clean_scrub_stamp":"2024-03-11T21:03:26.508215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:46.116991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"207'24","reported_seq":554,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.222766+0000","last_change":"2024-03-11T21:03:47.381203+0000","last_active":"2024-03-11T21:03:55.222766+0000","last_peered":"2024-03-11T21:03:55.222766+0000","last_clean":"2024-03-11T21:03:55.222766+0000","last_became_active":"2024-03-11T21:00:48.461972+0000","last_became_peered":"2024-03-11T21:00:48.461972+0000","last_unstale":"2024-03-11T21:03:55.222766+0000","last_undegraded":"2024-03-11T21:03:55.222766+0000","last_fullsized":"2024-03-11T21:03:55.222766+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'24","last_scrub_stamp":"2024-03-11T21:03:47.381061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:03:47.381061+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:05:05.348872+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":401,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.213269+0000","last_change":"2024-03-11T21:03:12.258970+0000","last_active":"2024-03-11T21:03:55.213269+0000","last_peered":"2024-03-11T21:03:55.213269+0000","last_clean":"2024-03-11T21:03:55.213269+0000","last_became_active":"2024-03-11T21:00:27.953537+0000","last_became_peered":"2024-03-11T21:00:27.953537+0000","last_unstale":"2024-03-11T21:03:55.213269+0000","last_undegraded":"2024-03-11T21:03:55.213269+0000","last_fullsized":"2024-03-11T21:03:55.213269+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:12.258842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:03:12.258842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:15.919928+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.3","version":"273'5721","reported_seq":5775,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:04:10.052693+0000","last_change":"2024-03-11T21:02:47.628471+0000","last_active":"2024-03-11T21:04:10.052693+0000","last_peered":"2024-03-11T21:04:10.052693+0000","last_clean":"2024-03-11T21:04:10.052693+0000","last_became_active":"2024-03-11T21:02:47.627896+0000","last_became_peered":"2024-03-11T21:02:47.627896+0000","last_unstale":"2024-03-11T21:04:10.052693+0000","last_undegraded":"2024-03-11T21:04:10.052693+0000","last_fullsized":"2024-03-11T21:04:10.052693+0000","mapping_epoch":251,"log_start":"259'1100","ondisk_log_start":"259'1100","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:02:46.570492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:46.570492+0000","last_clean_scrub_stamp":"2024-03-11T21:02:46.570492+0000","objects_scrubbed":0,"log_size":4621,"log_dups_size":0,"ondisk_log_size":4621,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:15.030464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":374931456,"num_objects":5721,"num_object_clones":0,"num_object_copies":17163,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5721,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5721,"num_write_kb":366144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":435,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.223138+0000","last_change":"2024-03-11T21:03:43.723734+0000","last_active":"2024-03-11T21:03:55.223138+0000","last_peered":"2024-03-11T21:03:55.223138+0000","last_clean":"2024-03-11T21:03:55.223138+0000","last_became_active":"2024-03-11T20:50:34.335370+0000","last_became_peered":"2024-03-11T20:50:34.335370+0000","last_unstale":"2024-03-11T21:03:55.223138+0000","last_undegraded":"2024-03-11T21:03:55.223138+0000","last_fullsized":"2024-03-11T21:03:55.223138+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:43.715417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:03:43.715417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:52.411766+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"249'17","reported_seq":576,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:58.777395+0000","last_change":"2024-03-11T21:03:24.365609+0000","last_active":"2024-03-11T21:03:58.777395+0000","last_peered":"2024-03-11T21:03:58.777395+0000","last_clean":"2024-03-11T21:03:58.777395+0000","last_became_active":"2024-03-11T21:02:37.524445+0000","last_became_peered":"2024-03-11T21:02:37.524445+0000","last_unstale":"2024-03-11T21:03:58.777395+0000","last_undegraded":"2024-03-11T21:03:58.777395+0000","last_fullsized":"2024-03-11T21:03:58.777395+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"249'17","last_scrub_stamp":"2024-03-11T21:03:24.365350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:03:24.365350+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:42.564922+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":468,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.220858+0000","last_change":"2024-03-11T21:03:20.066682+0000","last_active":"2024-03-11T21:03:55.220858+0000","last_peered":"2024-03-11T21:03:55.220858+0000","last_clean":"2024-03-11T21:03:55.220858+0000","last_became_active":"2024-03-11T21:00:48.466102+0000","last_became_peered":"2024-03-11T21:00:48.466102+0000","last_unstale":"2024-03-11T21:03:55.220858+0000","last_undegraded":"2024-03-11T21:03:55.220858+0000","last_fullsized":"2024-03-11T21:03:55.220858+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:20.066579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:52:09.043655+0000","last_clean_scrub_stamp":"2024-03-11T21:03:20.066579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:28.105514+0000","scrub_duration":147,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"207'18","reported_seq":401,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.213417+0000","last_change":"2024-03-11T21:02:10.604467+0000","last_active":"2024-03-11T21:03:55.213417+0000","last_peered":"2024-03-11T21:03:55.213417+0000","last_clean":"2024-03-11T21:03:55.213417+0000","last_became_active":"2024-03-11T21:00:27.967624+0000","last_became_peered":"2024-03-11T21:00:27.967624+0000","last_unstale":"2024-03-11T21:03:55.213417+0000","last_undegraded":"2024-03-11T21:03:55.213417+0000","last_fullsized":"2024-03-11T21:03:55.213417+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'18","last_scrub_stamp":"2024-03-11T21:02:10.604327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:02:10.604327+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:03:58.303190+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"273'5476","reported_seq":5528,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:04:06.135790+0000","last_change":"2024-03-11T21:02:47.641116+0000","last_active":"2024-03-11T21:04:06.135790+0000","last_peered":"2024-03-11T21:04:06.135790+0000","last_clean":"2024-03-11T21:04:06.135790+0000","last_became_active":"2024-03-11T21:02:47.640263+0000","last_became_peered":"2024-03-11T21:02:47.640263+0000","last_unstale":"2024-03-11T21:04:06.135790+0000","last_undegraded":"2024-03-11T21:04:06.135790+0000","last_fullsized":"2024-03-11T21:04:06.135790+0000","mapping_epoch":251,"log_start":"259'800","ondisk_log_start":"259'800","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:02:46.570492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:46.570492+0000","last_clean_scrub_stamp":"2024-03-11T21:02:46.570492+0000","objects_scrubbed":0,"log_size":4676,"log_dups_size":0,"ondisk_log_size":4676,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:15.757010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358875136,"num_objects":5476,"num_object_clones":0,"num_object_copies":16428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5476,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5476,"num_write_kb":350464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":457,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.225888+0000","last_change":"2024-03-11T21:02:27.775773+0000","last_active":"2024-03-11T21:03:55.225888+0000","last_peered":"2024-03-11T21:03:55.225888+0000","last_clean":"2024-03-11T21:03:55.225888+0000","last_became_active":"2024-03-11T21:00:48.460531+0000","last_became_peered":"2024-03-11T21:00:48.460531+0000","last_unstale":"2024-03-11T21:03:55.225888+0000","last_undegraded":"2024-03-11T21:03:55.225888+0000","last_fullsized":"2024-03-11T21:03:55.225888+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:02:27.775534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:27.775534+0000","last_clean_scrub_stamp":"2024-03-11T21:02:27.775534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:08.422599+0000","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":162,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.217364+0000","last_change":"2024-03-11T21:03:29.454135+0000","last_active":"2024-03-11T21:03:55.217364+0000","last_peered":"2024-03-11T21:03:55.217364+0000","last_clean":"2024-03-11T21:03:55.217364+0000","last_became_active":"2024-03-11T21:00:48.469596+0000","last_became_peered":"2024-03-11T21:00:48.469596+0000","last_unstale":"2024-03-11T21:03:55.217364+0000","last_undegraded":"2024-03-11T21:03:55.217364+0000","last_fullsized":"2024-03-11T21:03:55.217364+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:29.453952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:03:29.453952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:37.218531+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":154,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.216716+0000","last_change":"2024-03-11T21:03:26.422762+0000","last_active":"2024-03-11T21:03:55.216716+0000","last_peered":"2024-03-11T21:03:55.216716+0000","last_clean":"2024-03-11T21:03:55.216716+0000","last_became_active":"2024-03-11T21:00:48.466084+0000","last_became_peered":"2024-03-11T21:00:48.466084+0000","last_unstale":"2024-03-11T21:03:55.216716+0000","last_undegraded":"2024-03-11T21:03:55.216716+0000","last_fullsized":"2024-03-11T21:03:55.216716+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:26.422645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:40.974147+0000","last_clean_scrub_stamp":"2024-03-11T21:03:26.422645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:36.823254+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"207'34","reported_seq":554,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.223227+0000","last_change":"2024-03-11T21:03:13.518790+0000","last_active":"2024-03-11T21:03:55.223227+0000","last_peered":"2024-03-11T21:03:55.223227+0000","last_clean":"2024-03-11T21:03:55.223227+0000","last_became_active":"2024-03-11T21:00:48.472191+0000","last_became_peered":"2024-03-11T21:00:48.472191+0000","last_unstale":"2024-03-11T21:03:55.223227+0000","last_undegraded":"2024-03-11T21:03:55.223227+0000","last_fullsized":"2024-03-11T21:03:55.223227+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'34","last_scrub_stamp":"2024-03-11T21:03:13.518551+0000","last_deep_scrub":"116'22","last_deep_scrub_stamp":"2024-03-11T20:53:47.364709+0000","last_clean_scrub_stamp":"2024-03-11T21:03:13.518551+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:14.077539+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1","version":"273'5439","reported_seq":5493,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:04:06.121270+0000","last_change":"2024-03-11T21:02:47.642494+0000","last_active":"2024-03-11T21:04:06.121270+0000","last_peered":"2024-03-11T21:04:06.121270+0000","last_clean":"2024-03-11T21:04:06.121270+0000","last_became_active":"2024-03-11T21:02:47.641594+0000","last_became_peered":"2024-03-11T21:02:47.641594+0000","last_unstale":"2024-03-11T21:04:06.121270+0000","last_undegraded":"2024-03-11T21:04:06.121270+0000","last_fullsized":"2024-03-11T21:04:06.121270+0000","mapping_epoch":251,"log_start":"259'800","ondisk_log_start":"259'800","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:02:46.570492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:46.570492+0000","last_clean_scrub_stamp":"2024-03-11T21:02:46.570492+0000","objects_scrubbed":0,"log_size":4639,"log_dups_size":0,"ondisk_log_size":4639,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":356450304,"num_objects":5439,"num_object_clones":0,"num_object_copies":16317,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5439,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5439,"num_write_kb":348096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":404,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.214165+0000","last_change":"2024-03-11T21:03:33.263561+0000","last_active":"2024-03-11T21:03:55.214165+0000","last_peered":"2024-03-11T21:03:55.214165+0000","last_clean":"2024-03-11T21:03:55.214165+0000","last_became_active":"2024-03-11T21:00:48.464070+0000","last_became_peered":"2024-03-11T21:00:48.464070+0000","last_unstale":"2024-03-11T21:03:55.214165+0000","last_undegraded":"2024-03-11T21:03:55.214165+0000","last_fullsized":"2024-03-11T21:03:55.214165+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:33.263478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:03:33.263478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:52.150843+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":162,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.217106+0000","last_change":"2024-03-11T21:03:37.601354+0000","last_active":"2024-03-11T21:03:55.217106+0000","last_peered":"2024-03-11T21:03:55.217106+0000","last_clean":"2024-03-11T21:03:55.217106+0000","last_became_active":"2024-03-11T21:00:48.461367+0000","last_became_peered":"2024-03-11T21:00:48.461367+0000","last_unstale":"2024-03-11T21:03:55.217106+0000","last_undegraded":"2024-03-11T21:03:55.217106+0000","last_fullsized":"2024-03-11T21:03:55.217106+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:37.601229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:03:37.601229+0000","last_clean_scrub_stamp":"2024-03-11T21:03:37.601229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:50.353048+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"267'19","reported_seq":482,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.224898+0000","last_change":"2024-03-11T21:03:39.673560+0000","last_active":"2024-03-11T21:03:55.224898+0000","last_peered":"2024-03-11T21:03:55.224898+0000","last_clean":"2024-03-11T21:03:55.224898+0000","last_became_active":"2024-03-11T21:00:27.967572+0000","last_became_peered":"2024-03-11T21:00:27.967572+0000","last_unstale":"2024-03-11T21:03:55.224898+0000","last_undegraded":"2024-03-11T21:03:55.224898+0000","last_fullsized":"2024-03-11T21:03:55.224898+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"267'19","last_scrub_stamp":"2024-03-11T21:03:39.673317+0000","last_deep_scrub":"267'19","last_deep_scrub_stamp":"2024-03-11T21:03:39.673317+0000","last_clean_scrub_stamp":"2024-03-11T21:03:39.673317+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:48.038895+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":411,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.217110+0000","last_change":"2024-03-11T21:02:31.471685+0000","last_active":"2024-03-11T21:03:55.217110+0000","last_peered":"2024-03-11T21:03:55.217110+0000","last_clean":"2024-03-11T21:03:55.217110+0000","last_became_active":"2024-03-11T21:00:27.958537+0000","last_became_peered":"2024-03-11T21:00:27.958537+0000","last_unstale":"2024-03-11T21:03:55.217110+0000","last_undegraded":"2024-03-11T21:03:55.217110+0000","last_fullsized":"2024-03-11T21:03:55.217110+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:02:31.471513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:35.670149+0000","last_clean_scrub_stamp":"2024-03-11T21:02:31.471513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:03:55.824025+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"273'5539","reported_seq":5591,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:04:06.889157+0000","last_change":"2024-03-11T21:02:47.629420+0000","last_active":"2024-03-11T21:04:06.889157+0000","last_peered":"2024-03-11T21:04:06.889157+0000","last_clean":"2024-03-11T21:04:06.889157+0000","last_became_active":"2024-03-11T21:02:47.628916+0000","last_became_peered":"2024-03-11T21:02:47.628916+0000","last_unstale":"2024-03-11T21:04:06.889157+0000","last_undegraded":"2024-03-11T21:04:06.889157+0000","last_fullsized":"2024-03-11T21:04:06.889157+0000","mapping_epoch":251,"log_start":"257'400","ondisk_log_start":"257'400","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:02:46.570492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:46.570492+0000","last_clean_scrub_stamp":"2024-03-11T21:02:46.570492+0000","objects_scrubbed":0,"log_size":5139,"log_dups_size":0,"ondisk_log_size":5139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:07.516032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":363003904,"num_objects":5539,"num_object_clones":0,"num_object_copies":16617,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5539,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5539,"num_write_kb":354496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":459,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.233416+0000","last_change":"2024-03-11T21:02:33.768355+0000","last_active":"2024-03-11T21:03:55.233416+0000","last_peered":"2024-03-11T21:03:55.233416+0000","last_clean":"2024-03-11T21:03:55.233416+0000","last_became_active":"2024-03-11T21:00:27.959830+0000","last_became_peered":"2024-03-11T21:00:27.959830+0000","last_unstale":"2024-03-11T21:03:55.233416+0000","last_undegraded":"2024-03-11T21:03:55.233416+0000","last_fullsized":"2024-03-11T21:03:55.233416+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:02:33.767909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:02:33.767909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:03:57.859935+0000","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"249'19","reported_seq":587,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:58.777415+0000","last_change":"2024-03-11T21:03:23.347322+0000","last_active":"2024-03-11T21:03:58.777415+0000","last_peered":"2024-03-11T21:03:58.777415+0000","last_clean":"2024-03-11T21:03:58.777415+0000","last_became_active":"2024-03-11T21:02:37.805125+0000","last_became_peered":"2024-03-11T21:02:37.805125+0000","last_unstale":"2024-03-11T21:03:58.777415+0000","last_undegraded":"2024-03-11T21:03:58.777415+0000","last_fullsized":"2024-03-11T21:03:58.777415+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"249'19","last_scrub_stamp":"2024-03-11T21:03:23.347230+0000","last_deep_scrub":"146'14","last_deep_scrub_stamp":"2024-03-11T20:56:00.242211+0000","last_clean_scrub_stamp":"2024-03-11T21:03:23.347230+0000","objects_scrubbed":1,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:35.354233+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":464,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.223919+0000","last_change":"2024-03-11T21:03:50.901480+0000","last_active":"2024-03-11T21:03:55.223919+0000","last_peered":"2024-03-11T21:03:55.223919+0000","last_clean":"2024-03-11T21:03:55.223919+0000","last_became_active":"2024-03-11T21:00:27.970573+0000","last_became_peered":"2024-03-11T21:00:27.970573+0000","last_unstale":"2024-03-11T21:03:55.223919+0000","last_undegraded":"2024-03-11T21:03:55.223919+0000","last_fullsized":"2024-03-11T21:03:55.223919+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:50.901398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:52:51.161281+0000","last_clean_scrub_stamp":"2024-03-11T21:03:50.901398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:05:13.889962+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"207'44","reported_seq":599,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.230100+0000","last_change":"2024-03-11T21:03:06.547457+0000","last_active":"2024-03-11T21:03:55.230100+0000","last_peered":"2024-03-11T21:03:55.230100+0000","last_clean":"2024-03-11T21:03:55.230100+0000","last_became_active":"2024-03-11T21:00:48.469854+0000","last_became_peered":"2024-03-11T21:00:48.469854+0000","last_unstale":"2024-03-11T21:03:55.230100+0000","last_undegraded":"2024-03-11T21:03:55.230100+0000","last_fullsized":"2024-03-11T21:03:55.230100+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'44","last_scrub_stamp":"2024-03-11T21:03:06.547254+0000","last_deep_scrub":"161'28","last_deep_scrub_stamp":"2024-03-11T20:57:20.816278+0000","last_clean_scrub_stamp":"2024-03-11T21:03:06.547254+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:13.371692+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.7","version":"273'5717","reported_seq":5771,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:04:10.073773+0000","last_change":"2024-03-11T21:02:47.637634+0000","last_active":"2024-03-11T21:04:10.073773+0000","last_peered":"2024-03-11T21:04:10.073773+0000","last_clean":"2024-03-11T21:04:10.073773+0000","last_became_active":"2024-03-11T21:02:47.637110+0000","last_became_peered":"2024-03-11T21:02:47.637110+0000","last_unstale":"2024-03-11T21:04:10.073773+0000","last_undegraded":"2024-03-11T21:04:10.073773+0000","last_fullsized":"2024-03-11T21:04:10.073773+0000","mapping_epoch":251,"log_start":"257'300","ondisk_log_start":"257'300","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:02:46.570492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:46.570492+0000","last_clean_scrub_stamp":"2024-03-11T21:02:46.570492+0000","objects_scrubbed":0,"log_size":5417,"log_dups_size":0,"ondisk_log_size":5417,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:13.447035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":374669312,"num_objects":5717,"num_object_clones":0,"num_object_copies":17151,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5717,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5717,"num_write_kb":365888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":443,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.220633+0000","last_change":"2024-03-11T21:03:17.004069+0000","last_active":"2024-03-11T21:03:55.220633+0000","last_peered":"2024-03-11T21:03:55.220633+0000","last_clean":"2024-03-11T21:03:55.220633+0000","last_became_active":"2024-03-11T20:50:34.327139+0000","last_became_peered":"2024-03-11T20:50:34.327139+0000","last_unstale":"2024-03-11T21:03:55.220633+0000","last_undegraded":"2024-03-11T21:03:55.220633+0000","last_fullsized":"2024-03-11T21:03:55.220633+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:17.003975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:03:17.003975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:29.951662+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":471,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.232895+0000","last_change":"2024-03-11T21:03:15.683200+0000","last_active":"2024-03-11T21:03:55.232895+0000","last_peered":"2024-03-11T21:03:55.232895+0000","last_clean":"2024-03-11T21:03:55.232895+0000","last_became_active":"2024-03-11T20:50:28.049901+0000","last_became_peered":"2024-03-11T20:50:28.049901+0000","last_unstale":"2024-03-11T21:03:55.232895+0000","last_undegraded":"2024-03-11T21:03:55.232895+0000","last_fullsized":"2024-03-11T21:03:55.232895+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:15.683076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:57:17.447191+0000","last_clean_scrub_stamp":"2024-03-11T21:03:15.683076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:23.081557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"207'39","reported_seq":516,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.228127+0000","last_change":"2024-03-11T21:03:14.082722+0000","last_active":"2024-03-11T21:03:55.228127+0000","last_peered":"2024-03-11T21:03:55.228127+0000","last_clean":"2024-03-11T21:03:55.228127+0000","last_became_active":"2024-03-11T20:50:30.111819+0000","last_became_peered":"2024-03-11T20:50:30.111819+0000","last_unstale":"2024-03-11T21:03:55.228127+0000","last_undegraded":"2024-03-11T21:03:55.228127+0000","last_fullsized":"2024-03-11T21:03:55.228127+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'39","last_scrub_stamp":"2024-03-11T21:03:14.082576+0000","last_deep_scrub":"102'25","last_deep_scrub_stamp":"2024-03-11T20:51:35.187094+0000","last_clean_scrub_stamp":"2024-03-11T21:03:14.082576+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:43.746045+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":467,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.229767+0000","last_change":"2024-03-11T21:03:10.523931+0000","last_active":"2024-03-11T21:03:55.229767+0000","last_peered":"2024-03-11T21:03:55.229767+0000","last_clean":"2024-03-11T21:03:55.229767+0000","last_became_active":"2024-03-11T21:00:48.362389+0000","last_became_peered":"2024-03-11T21:00:48.362389+0000","last_unstale":"2024-03-11T21:03:55.229767+0000","last_undegraded":"2024-03-11T21:03:55.229767+0000","last_fullsized":"2024-03-11T21:03:55.229767+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:10.523794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:03:10.523794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:36.694395+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.6","version":"273'5655","reported_seq":5709,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:04:10.042670+0000","last_change":"2024-03-11T21:02:47.633710+0000","last_active":"2024-03-11T21:04:10.042670+0000","last_peered":"2024-03-11T21:04:10.042670+0000","last_clean":"2024-03-11T21:04:10.042670+0000","last_became_active":"2024-03-11T21:02:47.633198+0000","last_became_peered":"2024-03-11T21:02:47.633198+0000","last_unstale":"2024-03-11T21:04:10.042670+0000","last_undegraded":"2024-03-11T21:04:10.042670+0000","last_fullsized":"2024-03-11T21:04:10.042670+0000","mapping_epoch":251,"log_start":"257'200","ondisk_log_start":"257'200","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:02:46.570492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:46.570492+0000","last_clean_scrub_stamp":"2024-03-11T21:02:46.570492+0000","objects_scrubbed":0,"log_size":5455,"log_dups_size":0,"ondisk_log_size":5455,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:12.432602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370606080,"num_objects":5655,"num_object_clones":0,"num_object_copies":16965,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5655,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5655,"num_write_kb":361920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":263,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.222966+0000","last_change":"2024-03-11T21:03:11.086739+0000","last_active":"2024-03-11T21:03:55.222966+0000","last_peered":"2024-03-11T21:03:55.222966+0000","last_clean":"2024-03-11T21:03:55.222966+0000","last_became_active":"2024-03-11T20:56:59.801112+0000","last_became_peered":"2024-03-11T20:56:59.801112+0000","last_unstale":"2024-03-11T21:03:55.222966+0000","last_undegraded":"2024-03-11T21:03:55.222966+0000","last_fullsized":"2024-03-11T21:03:55.222966+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:11.086661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:55:58.888890+0000","last_clean_scrub_stamp":"2024-03-11T21:03:11.086661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:21.359717+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":154,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.217151+0000","last_change":"2024-03-11T21:03:17.298880+0000","last_active":"2024-03-11T21:03:55.217151+0000","last_peered":"2024-03-11T21:03:55.217151+0000","last_clean":"2024-03-11T21:03:55.217151+0000","last_became_active":"2024-03-11T21:00:48.469015+0000","last_became_peered":"2024-03-11T21:00:48.469015+0000","last_unstale":"2024-03-11T21:03:55.217151+0000","last_undegraded":"2024-03-11T21:03:55.217151+0000","last_fullsized":"2024-03-11T21:03:55.217151+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:17.298802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:03:17.298802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:23.332054+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"207'20","reported_seq":482,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.222810+0000","last_change":"2024-03-11T21:03:41.904778+0000","last_active":"2024-03-11T21:03:55.222810+0000","last_peered":"2024-03-11T21:03:55.222810+0000","last_clean":"2024-03-11T21:03:55.222810+0000","last_became_active":"2024-03-11T20:50:30.109549+0000","last_became_peered":"2024-03-11T20:50:30.109549+0000","last_unstale":"2024-03-11T21:03:55.222810+0000","last_undegraded":"2024-03-11T21:03:55.222810+0000","last_fullsized":"2024-03-11T21:03:55.222810+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'20","last_scrub_stamp":"2024-03-11T21:03:41.904604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:03:41.904604+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:50.209216+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":485,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.223021+0000","last_change":"2024-03-11T21:03:39.413187+0000","last_active":"2024-03-11T21:03:55.223021+0000","last_peered":"2024-03-11T21:03:55.223021+0000","last_clean":"2024-03-11T21:03:55.223021+0000","last_became_active":"2024-03-11T21:02:37.517748+0000","last_became_peered":"2024-03-11T21:02:37.517748+0000","last_unstale":"2024-03-11T21:03:55.223021+0000","last_undegraded":"2024-03-11T21:03:55.223021+0000","last_fullsized":"2024-03-11T21:03:55.223021+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:39.412996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:52:51.931295+0000","last_clean_scrub_stamp":"2024-03-11T21:03:39.412996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:05:06.183543+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.5","version":"273'5618","reported_seq":5671,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:04:10.093657+0000","last_change":"2024-03-11T21:02:47.619123+0000","last_active":"2024-03-11T21:04:10.093657+0000","last_peered":"2024-03-11T21:04:10.093657+0000","last_clean":"2024-03-11T21:04:10.093657+0000","last_became_active":"2024-03-11T21:02:47.618756+0000","last_became_peered":"2024-03-11T21:02:47.618756+0000","last_unstale":"2024-03-11T21:04:10.093657+0000","last_undegraded":"2024-03-11T21:04:10.093657+0000","last_fullsized":"2024-03-11T21:04:10.093657+0000","mapping_epoch":251,"log_start":"257'200","ondisk_log_start":"257'200","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:02:46.570492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:46.570492+0000","last_clean_scrub_stamp":"2024-03-11T21:02:46.570492+0000","objects_scrubbed":0,"log_size":5418,"log_dups_size":0,"ondisk_log_size":5418,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:12.366731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":368181248,"num_objects":5618,"num_object_clones":0,"num_object_copies":16854,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5618,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5618,"num_write_kb":359552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":452,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.222762+0000","last_change":"2024-03-11T21:03:12.778847+0000","last_active":"2024-03-11T21:03:55.222762+0000","last_peered":"2024-03-11T21:03:55.222762+0000","last_clean":"2024-03-11T21:03:55.222762+0000","last_became_active":"2024-03-11T20:50:34.337173+0000","last_became_peered":"2024-03-11T20:50:34.337173+0000","last_unstale":"2024-03-11T21:03:55.222762+0000","last_undegraded":"2024-03-11T21:03:55.222762+0000","last_fullsized":"2024-03-11T21:03:55.222762+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:12.778718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:00:14.030599+0000","last_clean_scrub_stamp":"2024-03-11T21:03:12.778718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:18.341981+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":463,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.232581+0000","last_change":"2024-03-11T21:03:12.702740+0000","last_active":"2024-03-11T21:03:55.232581+0000","last_peered":"2024-03-11T21:03:55.232581+0000","last_clean":"2024-03-11T21:03:55.232581+0000","last_became_active":"2024-03-11T20:50:28.058942+0000","last_became_peered":"2024-03-11T20:50:28.058942+0000","last_unstale":"2024-03-11T21:03:55.232581+0000","last_undegraded":"2024-03-11T21:03:55.232581+0000","last_fullsized":"2024-03-11T21:03:55.232581+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:12.702463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:03:12.702463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:30.975532+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"207'27","reported_seq":503,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.226358+0000","last_change":"2024-03-11T21:03:48.964872+0000","last_active":"2024-03-11T21:03:55.226358+0000","last_peered":"2024-03-11T21:03:55.226358+0000","last_clean":"2024-03-11T21:03:55.226358+0000","last_became_active":"2024-03-11T21:00:27.963119+0000","last_became_peered":"2024-03-11T21:00:27.963119+0000","last_unstale":"2024-03-11T21:03:55.226358+0000","last_undegraded":"2024-03-11T21:03:55.226358+0000","last_fullsized":"2024-03-11T21:03:55.226358+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'27","last_scrub_stamp":"2024-03-11T21:03:48.964758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:03:48.964758+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:05:11.347970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":448,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-11T21:03:55.221021+0000","last_change":"2024-03-11T21:03:44.899203+0000","last_active":"2024-03-11T21:03:55.221021+0000","last_peered":"2024-03-11T21:03:55.221021+0000","last_clean":"2024-03-11T21:03:55.221021+0000","last_became_active":"2024-03-11T20:50:32.652854+0000","last_became_peered":"2024-03-11T20:50:32.652854+0000","last_unstale":"2024-03-11T21:03:55.221021+0000","last_undegraded":"2024-03-11T21:03:55.221021+0000","last_fullsized":"2024-03-11T21:03:55.221021+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:44.899076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:59:00.946094+0000","last_clean_scrub_stamp":"2024-03-11T21:03:44.899076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:48.984405+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":5847908352,"num_objects":89232,"num_object_clones":0,"num_object_copies":267696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89232,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":89232,"num_write_kb":5710848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17391747072,"data_stored":17391747072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80232,"ondisk_log_size":80232,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":20,"num_read_kb":20,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1822,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1127,"num_read_kb":884,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":174,"num_bytes_recovered":2228,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3632,"internal_metadata":0},"log_size":773,"ondisk_log_size":773,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":121,"ondisk_log_size":121,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":682,"num_read_kb":677,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":68,"seq":292057776303,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3571080,"kb_used_data":3265024,"kb_used_omap":1,"kb_used_meta":306046,"kb_avail":90174072,"statfs":{"total":95995035648,"available":92338249728,"internally_reserved":0,"allocated":3343384576,"data_stored":3342740313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":313391562},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":57,"seq":244813136057,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1292280,"kb_used_data":1084784,"kb_used_omap":1,"kb_used_meta":207486,"kb_avail":92452872,"statfs":{"total":95995035648,"available":94671740928,"internally_reserved":0,"allocated":1110818816,"data_stored":1110190947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":212466119},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":211,"seq":906238099497,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3302204,"kb_used_data":3139572,"kb_used_omap":7,"kb_used_meta":162616,"kb_avail":90442948,"statfs":{"total":95995035648,"available":92613578752,"internally_reserved":0,"allocated":3214921728,"data_stored":3214289465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8078,"internal_metadata":166518898},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738614,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1656192,"kb_used_data":1364024,"kb_used_omap":1,"kb_used_meta":292158,"kb_avail":92088960,"statfs":{"total":95995035648,"available":94299095040,"internally_reserved":0,"allocated":1396760576,"data_stored":1396123447,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":299170245},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378849,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1299308,"kb_used_data":1033508,"kb_used_omap":1,"kb_used_meta":265790,"kb_avail":92445844,"statfs":{"total":95995035648,"available":94664544256,"internally_reserved":0,"allocated":1058312192,"data_stored":1057695161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":272169413},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":28,"seq":120259084502,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3172912,"kb_used_data":2772456,"kb_used_omap":1,"kb_used_meta":400446,"kb_avail":90572240,"statfs":{"total":95995035648,"available":92745973760,"internally_reserved":0,"allocated":2838994944,"data_stored":2838374193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":410057160},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":204,"seq":876173328429,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1135508,"kb_used_data":1044812,"kb_used_omap":4,"kb_used_meta":90683,"kb_avail":92609644,"statfs":{"total":95995035648,"available":94832275456,"internally_reserved":0,"allocated":1069887488,"data_stored":1069231263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4451,"internal_metadata":92860061},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":44,"seq":188978561219,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3487472,"kb_used_data":3221672,"kb_used_omap":1,"kb_used_meta":265790,"kb_avail":90257680,"statfs":{"total":95995035648,"available":92423864320,"internally_reserved":0,"allocated":3298992128,"data_stored":3298370923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":272169413},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3632,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1401880576,"data_stored":1401880576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3231907840,"data_stored":3231907840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1064108032,"data_stored":1064108032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2848456704,"data_stored":2848456704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1073545216,"data_stored":1073545216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3315859456,"data_stored":3315859456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1112932352,"data_stored":1112932352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":3343056896,"data_stored":3343056896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T21:04:14.177 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T21:04:14.234 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:04:14.265 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:04:14.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:14.268 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:04:14.368 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:04:14.382 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-03-11T21:04:14.382 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-03-11T21:04:14.382 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 30, 2024-03-11T21:04:14.383 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-03-11T21:04:14.383 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-03-11T21:04:14.406 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:04:14.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:14.409 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:04:14.462 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 72 15 94329 94314 81.8575 53.8125 0.0110956 0.0122117 2024-03-11T21:04:14.509 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:14.539 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:14.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:14.541 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:04:14.642 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:04:14.664 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T21:04:14.668 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:04:14.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:14.671 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:04:14.772 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:04:14.802 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:04:14.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:14.806 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:04:14.907 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:04:14.927 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:04:14.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:14.929 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:04:15.029 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:04:15.054 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:04:15.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:15.056 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:04:15.157 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:04:15.184 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:04:15.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:15.186 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:04:15.286 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:04:15.317 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:04:15.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:15.323 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:04:15.423 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:15.455 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:15.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:15.460 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:04:15.462 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 73 15 94965 94950 81.2806 39.75 0.0132451 0.0122981 2024-03-11T21:04:15.512 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:04:15 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:04:15.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:15 smithi146 bash[23809]: cluster 2024-03-11T21:04:14.172337+0000 mgr.y (mgr.24380) 498 : cluster 0 pgmap v557: 148 pgs: 148 active+clean; 5.6 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 43 MiB/s wr, 692 op/s 2024-03-11T21:04:15.561 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:04:15.571 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:15 smithi023 bash[21372]: cluster 2024-03-11T21:04:14.172337+0000 mgr.y (mgr.24380) 498 : cluster 0 pgmap v557: 148 pgs: 148 active+clean; 5.6 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 43 MiB/s wr, 692 op/s 2024-03-11T21:04:15.571 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:15 smithi023 bash[29146]: cluster 2024-03-11T21:04:14.172337+0000 mgr.y (mgr.24380) 498 : cluster 0 pgmap v557: 148 pgs: 148 active+clean; 5.6 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 43 MiB/s wr, 692 op/s 2024-03-11T21:04:15.601 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:04:15.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:15.603 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:04:15.704 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:04:15.715 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-03-11T21:04:15.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:15.735 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:04:15.735 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:04:15.837 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:04:15.865 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:04:15.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:15.867 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:04:15.968 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:04:15.991 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:04:15.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:15.995 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:04:16.095 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:04:16.137 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:04:16.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:16.140 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:04:16.190 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd unset noscrub 2024-03-11T21:04:16.242 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:04:16.271 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:04:16.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:16.277 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:04:16.378 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:04:16.410 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:04:16.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:16.412 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:04:16.464 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 74 16 96006 95990 81.0589 65 0.0126084 0.0123317 2024-03-11T21:04:16.512 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:16.513 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:16 smithi146 bash[23809]: audit 2024-03-11T21:04:15.484920+0000 mgr.y (mgr.24380) 499 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:04:16.539 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:16.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:16.543 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:04:16.644 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:04:16.653 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:16 smithi023 bash[21372]: audit 2024-03-11T21:04:15.484920+0000 mgr.y (mgr.24380) 499 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:04:16.654 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:16 smithi023 bash[29146]: audit 2024-03-11T21:04:15.484920+0000 mgr.y (mgr.24380) 499 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:04:16.675 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:04:16.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:16.678 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:04:16.779 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:04:16.814 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:04:16.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:16.822 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:04:16.923 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:04:16.948 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:04:16.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:16.951 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:04:17.052 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:04:17.084 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:04:17.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:17.090 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:04:17.191 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:04:17.219 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:04:17.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:17.225 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:04:17.326 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:04:17.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:17.356 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:04:17.356 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:04:17.457 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:04:17.463 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 75 16 97092 97076 80.8831 67.875 0.0176706 0.0123589 2024-03-11T21:04:17.484 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:04:17.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:17.486 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:04:17.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:17 smithi023 bash[21372]: cluster 2024-03-11T21:04:16.173233+0000 mgr.y (mgr.24380) 500 : cluster 0 pgmap v558: 148 pgs: 148 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 59 MiB/s wr, 944 op/s 2024-03-11T21:04:17.533 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:17 smithi023 bash[29146]: cluster 2024-03-11T21:04:16.173233+0000 mgr.y (mgr.24380) 500 : cluster 0 pgmap v558: 148 pgs: 148 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 59 MiB/s wr, 944 op/s 2024-03-11T21:04:17.587 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:17.613 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:17.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:17.615 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:04:17.715 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:17.725 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:17 smithi146 bash[23809]: cluster 2024-03-11T21:04:16.173233+0000 mgr.y (mgr.24380) 500 : cluster 0 pgmap v558: 148 pgs: 148 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 59 MiB/s wr, 944 op/s 2024-03-11T21:04:17.740 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:17.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:17.743 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:04:17.843 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:04:17.871 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:04:17.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:17.875 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:04:17.975 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:04:18.010 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:04:18.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:18.012 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:04:18.113 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:04:18.142 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:04:18.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:18.148 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:04:18.249 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:04:18.278 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:04:18.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:18.282 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:04:18.382 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:04:18.407 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:04:18.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:18.409 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:04:18.466 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 76 16 98206 98190 80.7349 69.625 0.017646 0.0123809 2024-03-11T21:04:18.510 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:18.533 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:04:18 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:04:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:04:18.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:18 smithi023 bash[21372]: cluster 2024-03-11T21:04:16.663365+0000 osd.0 (osd.0) 197 : cluster 0 purged_snaps scrub starts 2024-03-11T21:04:18.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:18 smithi023 bash[21372]: cluster 2024-03-11T21:04:16.663464+0000 osd.0 (osd.0) 198 : cluster 0 purged_snaps scrub ok 2024-03-11T21:04:18.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:18 smithi023 bash[21372]: cluster 2024-03-11T21:04:17.367268+0000 mon.a (mon.0) 1505 : cluster 0 osdmap e276: 8 total, 8 up, 8 in 2024-03-11T21:04:18.534 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:18 smithi023 bash[29146]: cluster 2024-03-11T21:04:16.663365+0000 osd.0 (osd.0) 197 : cluster 0 purged_snaps scrub starts 2024-03-11T21:04:18.534 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:18 smithi023 bash[29146]: cluster 2024-03-11T21:04:16.663464+0000 osd.0 (osd.0) 198 : cluster 0 purged_snaps scrub ok 2024-03-11T21:04:18.534 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:18 smithi023 bash[29146]: cluster 2024-03-11T21:04:17.367268+0000 mon.a (mon.0) 1505 : cluster 0 osdmap e276: 8 total, 8 up, 8 in 2024-03-11T21:04:18.542 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:18.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:18.545 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:04:18.646 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:04:18.682 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:04:18.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:18.686 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:04:18.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:18 smithi146 bash[23809]: cluster 2024-03-11T21:04:16.663365+0000 osd.0 (osd.0) 197 : cluster 0 purged_snaps scrub starts 2024-03-11T21:04:18.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:18 smithi146 bash[23809]: cluster 2024-03-11T21:04:16.663464+0000 osd.0 (osd.0) 198 : cluster 0 purged_snaps scrub ok 2024-03-11T21:04:18.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:18 smithi146 bash[23809]: cluster 2024-03-11T21:04:17.367268+0000 mon.a (mon.0) 1505 : cluster 0 osdmap e276: 8 total, 8 up, 8 in 2024-03-11T21:04:18.787 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:04:18.815 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:04:18.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:18.817 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:04:18.918 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:04:18.950 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:04:18.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:18.952 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:04:19.051 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:04:19.053 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:04:19.054 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:04:19.079 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:04:19.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:19.082 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:04:19.183 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:04:19.208 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:04:19.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:19.210 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:04:19.311 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:04:19.348 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:04:19.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:19.351 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:04:19.452 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:04:19.465 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 77 16 99290 99274 80.5649 67.75 0.0191111 0.0124074 2024-03-11T21:04:19.478 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:04:19.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:19.487 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:04:19.587 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:04:19.612 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:04:19.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:19.616 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:04:19.716 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:04:19.725 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:19 smithi146 bash[23809]: cluster 2024-03-11T21:04:18.173674+0000 mgr.y (mgr.24380) 501 : cluster 0 pgmap v560: 148 pgs: 148 active+clean; 5.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.3 KiB/s rd, 47 MiB/s wr, 757 op/s 2024-03-11T21:04:19.747 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:04:19.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:19.752 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:04:19.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:19 smithi023 bash[21372]: cluster 2024-03-11T21:04:18.173674+0000 mgr.y (mgr.24380) 501 : cluster 0 pgmap v560: 148 pgs: 148 active+clean; 5.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.3 KiB/s rd, 47 MiB/s wr, 757 op/s 2024-03-11T21:04:19.783 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:19 smithi023 bash[29146]: cluster 2024-03-11T21:04:18.173674+0000 mgr.y (mgr.24380) 501 : cluster 0 pgmap v560: 148 pgs: 148 active+clean; 5.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.3 KiB/s rd, 47 MiB/s wr, 757 op/s 2024-03-11T21:04:19.853 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:19.881 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:19.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:19.884 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:04:19.984 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:20.011 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:20.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:20.013 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:04:20.069 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:04:20.115 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:04:20.150 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:04:20.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:20.155 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:04:20.255 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:20.275 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:20.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:20.277 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:04:20.378 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:04:20.401 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:04:20.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:20.404 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:04:20.465 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 78 16 100127 100111 80.2026 52.3125 0.0043978 0.0124312 2024-03-11T21:04:20.506 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:20.534 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:20.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:20.536 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:04:20.584 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:04:20.637 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:04:20.661 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:04:20.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:20.663 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:04:20.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:20 smithi146 bash[23809]: audit 2024-03-11T21:04:20.320799+0000 mon.a (mon.0) 1506 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:04:20.763 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:04:20.774 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:20 smithi023 bash[21372]: audit 2024-03-11T21:04:20.320799+0000 mon.a (mon.0) 1506 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:04:20.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:20 smithi023 bash[29146]: audit 2024-03-11T21:04:20.320799+0000 mon.a (mon.0) 1506 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:04:20.793 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:04:20.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:20.797 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:04:20.898 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:04:20.920 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:04:20.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:20.926 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:04:21.027 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:04:21.072 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:04:21.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:21.076 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:04:21.177 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:04:21.203 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:04:21.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:21.208 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:04:21.308 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:04:21.339 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:04:21.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:21.344 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:04:21.445 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:04:21.465 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 79 16 100952 100936 79.8401 51.5625 0.00690771 0.0125197 2024-03-11T21:04:21.470 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:04:21.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:21.475 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:04:21.576 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:04:21.602 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:04:21.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:21.604 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:04:21.705 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:21.715 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:21 smithi023 bash[21372]: cluster 2024-03-11T21:04:20.174287+0000 mgr.y (mgr.24380) 502 : cluster 0 pgmap v561: 148 pgs: 148 active+clean; 5.9 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 45 MiB/s wr, 720 op/s 2024-03-11T21:04:21.715 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:21 smithi023 bash[29146]: cluster 2024-03-11T21:04:20.174287+0000 mgr.y (mgr.24380) 502 : cluster 0 pgmap v561: 148 pgs: 148 active+clean; 5.9 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 45 MiB/s wr, 720 op/s 2024-03-11T21:04:21.738 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:21.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:21.747 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:04:21.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:21 smithi146 bash[23809]: cluster 2024-03-11T21:04:20.174287+0000 mgr.y (mgr.24380) 502 : cluster 0 pgmap v561: 148 pgs: 148 active+clean; 5.9 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 45 MiB/s wr, 720 op/s 2024-03-11T21:04:21.848 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:21.873 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:21.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:21.877 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:04:21.978 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:04:22.005 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:04:22.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:22.010 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:04:22.111 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:22.128 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:22.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:22.130 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:04:22.231 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:04:22.253 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:04:22.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:22.255 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:04:22.356 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:04:22.377 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:04:22.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:22.382 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:04:22.465 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:2024-03-11T21:04:22.470910+0000 min lat: 0.0011821 max lat: 0.549376 avg lat: 0.012592 2024-03-11T21:04:22.465 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-11T21:04:22.466 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 80 16 101640 101624 79.3796 43 0.00578425 0.012592 2024-03-11T21:04:22.483 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:22.515 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:22.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:22.521 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:04:22.622 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:22.647 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:22.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:22.651 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:04:22.752 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:04:22.776 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:04:22.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:22.784 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:04:22.885 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:04:22.914 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:04:22.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:22.918 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:04:23.019 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:23.049 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:23.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:23.052 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:04:23.153 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:04:23.180 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:04:23.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:23.184 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:04:23.285 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:04:23.312 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:04:23.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:23.315 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:04:23.416 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:04:23.459 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:04:23.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:23.463 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:04:23.465 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 81 16 102660 102644 79.1866 63.75 0.0116929 0.0126237 2024-03-11T21:04:23.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:23 smithi146 bash[23809]: cluster 2024-03-11T21:04:22.175043+0000 mgr.y (mgr.24380) 503 : cluster 0 pgmap v562: 148 pgs: 148 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.0 KiB/s rd, 59 MiB/s wr, 949 op/s 2024-03-11T21:04:23.563 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:04:23.595 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:04:23.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:23.599 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:04:23.654 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:23 smithi023 bash[21372]: cluster 2024-03-11T21:04:22.175043+0000 mgr.y (mgr.24380) 503 : cluster 0 pgmap v562: 148 pgs: 148 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.0 KiB/s rd, 59 MiB/s wr, 949 op/s 2024-03-11T21:04:23.655 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:23 smithi023 bash[29146]: cluster 2024-03-11T21:04:22.175043+0000 mgr.y (mgr.24380) 503 : cluster 0 pgmap v562: 148 pgs: 148 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.0 KiB/s rd, 59 MiB/s wr, 949 op/s 2024-03-11T21:04:23.700 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:04:23.723 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:04:23.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:23.726 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:04:23.826 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:04:23.849 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:04:23.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:23.851 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:04:23.952 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:04:23.992 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:04:23.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:23.995 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:04:24.096 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:04:24.122 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:04:24.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:24.126 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:04:24.227 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:04:24.264 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:04:24.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:24.268 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:04:24.369 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:04:24.394 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:04:24.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:24.399 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:04:24.465 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 82 16 103492 103476 78.855 52 0.0150284 0.0126769 2024-03-11T21:04:24.500 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:24.534 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:24.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:24.537 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:04:24.638 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:04:24.669 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:04:24.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:24.671 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:04:24.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:24 smithi146 bash[23809]: cluster 2024-03-11T21:04:23.824542+0000 osd.4 (osd.4) 75 : cluster 0 purged_snaps scrub starts 2024-03-11T21:04:24.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:24 smithi146 bash[23809]: cluster 2024-03-11T21:04:23.824631+0000 osd.4 (osd.4) 76 : cluster 0 purged_snaps scrub ok 2024-03-11T21:04:24.772 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:04:24.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:24 smithi023 bash[21372]: cluster 2024-03-11T21:04:23.824542+0000 osd.4 (osd.4) 75 : cluster 0 purged_snaps scrub starts 2024-03-11T21:04:24.784 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:24 smithi023 bash[21372]: cluster 2024-03-11T21:04:23.824631+0000 osd.4 (osd.4) 76 : cluster 0 purged_snaps scrub ok 2024-03-11T21:04:24.784 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:24 smithi023 bash[29146]: cluster 2024-03-11T21:04:23.824542+0000 osd.4 (osd.4) 75 : cluster 0 purged_snaps scrub starts 2024-03-11T21:04:24.784 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:24 smithi023 bash[29146]: cluster 2024-03-11T21:04:23.824631+0000 osd.4 (osd.4) 76 : cluster 0 purged_snaps scrub ok 2024-03-11T21:04:24.801 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:04:24.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:24.805 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:04:24.906 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:04:24.937 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:04:24.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:24.940 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:04:25.041 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:04:25.064 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:04:25.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:25.071 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:04:25.172 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:04:25.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:25.209 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:04:25.210 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:04:25.311 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:04:25.342 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:04:25.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:25.345 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:04:25.446 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:04:25.457 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:25 smithi023 bash[21372]: cluster 2024-03-11T21:04:24.175485+0000 mgr.y (mgr.24380) 504 : cluster 0 pgmap v563: 148 pgs: 148 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 58 MiB/s wr, 934 op/s 2024-03-11T21:04:25.457 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:25 smithi023 bash[21372]: cluster 2024-03-11T21:04:24.446311+0000 mon.a (mon.0) 1507 : cluster 0 osdmap e277: 8 total, 8 up, 8 in 2024-03-11T21:04:25.458 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:25 smithi023 bash[29146]: cluster 2024-03-11T21:04:24.175485+0000 mgr.y (mgr.24380) 504 : cluster 0 pgmap v563: 148 pgs: 148 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 58 MiB/s wr, 934 op/s 2024-03-11T21:04:25.458 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:25 smithi023 bash[29146]: cluster 2024-03-11T21:04:24.446311+0000 mon.a (mon.0) 1507 : cluster 0 osdmap e277: 8 total, 8 up, 8 in 2024-03-11T21:04:25.465 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 83 16 104506 104490 78.6684 63.375 0.0181689 0.0127061 2024-03-11T21:04:25.478 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:04:25.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:25.483 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:04:25.583 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:25.615 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:25.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:25.620 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:04:25.720 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:04:25.745 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:04:25.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:25.750 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:04:25.762 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:04:25 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:04:25.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:25 smithi146 bash[23809]: cluster 2024-03-11T21:04:24.175485+0000 mgr.y (mgr.24380) 504 : cluster 0 pgmap v563: 148 pgs: 148 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 58 MiB/s wr, 934 op/s 2024-03-11T21:04:25.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:25 smithi146 bash[23809]: cluster 2024-03-11T21:04:24.446311+0000 mon.a (mon.0) 1507 : cluster 0 osdmap e277: 8 total, 8 up, 8 in 2024-03-11T21:04:25.851 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:04:25.881 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:04:25.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:25.884 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:04:25.985 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:04:26.012 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:04:26.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:26.016 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:04:26.117 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:26.145 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:26.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:26.148 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:04:26.249 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:26.283 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:26.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:26.290 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:04:26.391 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:04:26.423 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:04:26.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:26.426 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:04:26.465 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 84 15 105601 105586 78.5473 68.5 0.0074476 0.0127267 2024-03-11T21:04:26.527 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:04:26.556 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:04:26.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:26.561 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:04:26.662 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:04:26.693 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:26 smithi146 bash[23809]: audit 2024-03-11T21:04:25.493591+0000 mgr.y (mgr.24380) 505 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:04:26.701 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:04:26.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:26.703 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:04:26.710 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:26 smithi023 bash[21372]: audit 2024-03-11T21:04:25.493591+0000 mgr.y (mgr.24380) 505 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:04:26.711 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:26 smithi023 bash[29146]: audit 2024-03-11T21:04:25.493591+0000 mgr.y (mgr.24380) 505 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:04:26.805 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:04:26.831 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:04:26.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:26.834 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:04:26.934 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:04:26.961 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:04:26.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:26.965 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:04:26.966 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-03-11T21:04:26.966 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-03-11T21:04:26.966 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-03-11T21:04:26.966 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-03-11T21:04:26.966 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:26.966 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.24629.0:46655 7.5 7:a3f67fbd:::benchmark_data_smithi023_67959_object46654:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-03-11T21:04:26.966 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T20:55:16.113789+0000", 2024-03-11T21:04:26.967 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 550.85127360900003, 2024-03-11T21:04:26.967 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.33351519000000002, 2024-03-11T21:04:26.967 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:04:26.967 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:04:26.967 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-03-11T21:04:26.967 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.24629", 2024-03-11T21:04:26.967 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.23:0/959732609", 2024-03-11T21:04:26.967 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 46655 2024-03-11T21:04:26.967 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:26.967 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:04:26.967 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:26.967 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:04:26.967 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:16.113789+0000", 2024-03-11T21:04:26.967 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:04:26.968 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:26.968 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:26.968 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:04:26.968 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:16.113789+0000", 2024-03-11T21:04:26.968 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:04:26.968 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:26.968 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:26.968 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:04:26.968 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:16.113791+0000", 2024-03-11T21:04:26.968 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8369999999999999e-06 2024-03-11T21:04:26.968 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:26.968 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:26.968 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:04:26.968 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:16.113844+0000", 2024-03-11T21:04:26.969 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2262000000000002e-05 2024-03-11T21:04:26.969 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:26.969 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:26.969 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:04:26.969 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:16.113860+0000", 2024-03-11T21:04:26.969 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6113e-05 2024-03-11T21:04:26.969 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:26.969 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:26.969 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:04:26.969 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:16.113868+0000", 2024-03-11T21:04:26.969 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5760000000000006e-06 2024-03-11T21:04:26.969 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:26.969 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:26.969 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:04:26.970 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:16.113931+0000", 2024-03-11T21:04:26.970 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2436999999999999e-05 2024-03-11T21:04:26.970 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:26.970 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:26.970 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:04:26.970 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:16.114066+0000", 2024-03-11T21:04:26.970 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013563299999999999 2024-03-11T21:04:26.970 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:26.970 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:26.970 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0,2", 2024-03-11T21:04:26.970 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:16.114208+0000", 2024-03-11T21:04:26.970 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014155400000000001 2024-03-11T21:04:26.970 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:26.971 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:26.971 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-03-11T21:04:26.971 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:16.115707+0000", 2024-03-11T21:04:26.971 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001499067 2024-03-11T21:04:26.971 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:26.971 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:26.971 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:04:26.971 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:16.138092+0000", 2024-03-11T21:04:26.971 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022384766 2024-03-11T21:04:26.971 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:26.971 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:26.971 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:04:26.971 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:16.446954+0000", 2024-03-11T21:04:26.971 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.308861948 2024-03-11T21:04:26.972 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:26.972 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:26.972 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:04:26.972 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:16.447160+0000", 2024-03-11T21:04:26.972 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000205928 2024-03-11T21:04:26.972 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:26.972 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:26.972 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:04:26.972 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:16.447305+0000", 2024-03-11T21:04:26.972 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000145069 2024-03-11T21:04:26.972 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:04:26.972 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:04:26.972 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:04:26.972 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:26.973 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:26.973 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.24629.0:46668 7.5 7:a68282cc:::benchmark_data_smithi023_67959_object46667:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-03-11T21:04:26.973 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T20:55:16.138556+0000", 2024-03-11T21:04:26.973 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 550.82650715800003, 2024-03-11T21:04:26.973 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.31009069700000003, 2024-03-11T21:04:26.973 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:04:26.973 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:04:26.973 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-03-11T21:04:26.973 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.24629", 2024-03-11T21:04:26.973 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.23:0/959732609", 2024-03-11T21:04:26.973 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 46668 2024-03-11T21:04:26.973 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:26.973 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:04:26.973 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:26.974 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:04:26.974 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:16.138556+0000", 2024-03-11T21:04:26.974 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:04:26.974 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:26.974 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:26.974 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:04:26.974 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:16.138556+0000", 2024-03-11T21:04:26.974 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:04:26.974 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:26.974 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:26.974 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:04:26.974 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:16.138557+0000", 2024-03-11T21:04:26.975 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.376e-06 2024-03-11T21:04:26.975 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:26.975 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:26.975 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:04:26.975 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:16.138611+0000", 2024-03-11T21:04:26.975 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3594000000000002e-05 2024-03-11T21:04:26.975 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:26.975 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:26.975 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:04:26.975 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:16.138625+0000", 2024-03-11T21:04:26.975 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4365000000000001e-05 2024-03-11T21:04:26.975 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:26.975 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:26.975 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:04:26.975 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:16.138634+0000", 2024-03-11T21:04:26.976 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8089999999999999e-06 2024-03-11T21:04:26.976 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:26.976 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:26.976 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:04:26.976 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:16.138696+0000", 2024-03-11T21:04:26.976 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1667000000000004e-05 2024-03-11T21:04:26.976 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:26.976 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:26.976 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:04:26.976 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:16.138812+0000", 2024-03-11T21:04:26.977 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000115838 2024-03-11T21:04:26.978 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:26.978 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:26.978 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0,2", 2024-03-11T21:04:26.978 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:16.138958+0000", 2024-03-11T21:04:26.978 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014603700000000001 2024-03-11T21:04:26.978 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:26.978 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:26.978 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-03-11T21:04:26.978 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:16.140172+0000", 2024-03-11T21:04:26.978 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0012141470000000001 2024-03-11T21:04:26.978 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:26.978 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:26.979 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:04:26.979 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:16.141081+0000", 2024-03-11T21:04:26.979 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000909621 2024-03-11T21:04:26.979 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:26.979 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:26.979 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:04:26.979 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:16.448383+0000", 2024-03-11T21:04:26.979 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.30730185300000001 2024-03-11T21:04:26.979 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:26.979 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:26.979 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:04:26.979 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:16.448539+0000", 2024-03-11T21:04:26.979 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015612800000000001 2024-03-11T21:04:26.980 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:26.980 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:26.980 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:04:26.980 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:16.448647+0000", 2024-03-11T21:04:26.980 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000107262 2024-03-11T21:04:26.980 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:04:26.980 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:04:26.980 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:04:26.980 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:26.980 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:26.980 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.24629.0:85821 7.5 7:a4dbbed2:::benchmark_data_smithi023_67959_object85820:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-03-11T21:04:26.980 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T20:55:52.507915+0000", 2024-03-11T21:04:26.980 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 514.45714837200001, 2024-03-11T21:04:26.980 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.34389546500000001, 2024-03-11T21:04:26.981 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:04:26.981 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:04:26.981 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-03-11T21:04:26.981 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.24629", 2024-03-11T21:04:26.981 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.23:0/959732609", 2024-03-11T21:04:26.981 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 85821 2024-03-11T21:04:26.981 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:26.981 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:04:26.981 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:26.981 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:04:26.981 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:52.507915+0000", 2024-03-11T21:04:26.981 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:04:26.981 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:26.981 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:26.982 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:04:26.982 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:52.507915+0000", 2024-03-11T21:04:26.982 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:04:26.982 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:26.982 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:26.982 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:04:26.982 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:52.507916+0000", 2024-03-11T21:04:26.982 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.508e-06 2024-03-11T21:04:26.982 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:26.982 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:26.983 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:04:26.983 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:52.507953+0000", 2024-03-11T21:04:26.983 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.6743999999999999e-05 2024-03-11T21:04:26.983 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:26.983 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:26.983 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:04:26.983 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:52.507969+0000", 2024-03-11T21:04:26.983 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6429000000000001e-05 2024-03-11T21:04:26.983 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:26.983 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:26.983 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:04:26.983 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:52.507977+0000", 2024-03-11T21:04:26.983 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9640000000000003e-06 2024-03-11T21:04:26.984 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:26.984 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:26.984 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:04:26.984 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:52.508033+0000", 2024-03-11T21:04:26.984 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5436999999999998e-05 2024-03-11T21:04:26.984 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:26.984 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:26.984 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:04:26.984 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:52.508148+0000", 2024-03-11T21:04:26.984 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000114968 2024-03-11T21:04:26.984 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:26.984 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:26.984 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0,2", 2024-03-11T21:04:26.985 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:52.508281+0000", 2024-03-11T21:04:26.985 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013320399999999999 2024-03-11T21:04:26.985 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:26.985 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:26.985 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:04:26.985 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:52.510572+0000", 2024-03-11T21:04:26.985 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002290795 2024-03-11T21:04:26.985 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:26.985 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:26.985 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-03-11T21:04:26.985 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:52.510603+0000", 2024-03-11T21:04:26.985 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1170000000000001e-05 2024-03-11T21:04:26.985 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:26.985 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:26.986 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:04:26.986 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:52.851724+0000", 2024-03-11T21:04:26.986 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.34112095799999997 2024-03-11T21:04:26.986 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:26.986 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:26.986 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:04:26.986 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:52.851763+0000", 2024-03-11T21:04:26.986 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9283999999999999e-05 2024-03-11T21:04:26.986 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:26.986 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:26.986 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:04:26.986 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:52.851810+0000", 2024-03-11T21:04:26.986 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7003999999999999e-05 2024-03-11T21:04:26.987 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:04:26.987 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:04:26.987 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:04:26.987 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:26.987 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:26.987 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.24629.0:85829 7.5 7:aa3b6cf8:::benchmark_data_smithi023_67959_object85828:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-03-11T21:04:26.987 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T20:55:52.517723+0000", 2024-03-11T21:04:26.987 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 514.44733964900001, 2024-03-11T21:04:26.987 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.33424817899999998, 2024-03-11T21:04:26.987 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:04:26.987 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:04:26.987 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-03-11T21:04:26.987 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.24629", 2024-03-11T21:04:26.987 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.23:0/959732609", 2024-03-11T21:04:26.988 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 85829 2024-03-11T21:04:26.988 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:26.988 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:04:26.988 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:26.988 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:04:26.988 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:52.517723+0000", 2024-03-11T21:04:26.988 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:04:26.988 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:26.988 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:26.988 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:04:26.988 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:52.517723+0000", 2024-03-11T21:04:26.988 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:04:26.988 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:26.988 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:26.989 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:04:26.989 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:52.517725+0000", 2024-03-11T21:04:26.989 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3289999999999999e-06 2024-03-11T21:04:26.989 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:26.989 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:26.989 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:04:26.989 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:52.517823+0000", 2024-03-11T21:04:26.989 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7900999999999997e-05 2024-03-11T21:04:26.989 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:26.989 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:26.990 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:04:26.990 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:52.517838+0000", 2024-03-11T21:04:26.990 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4932e-05 2024-03-11T21:04:26.990 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:26.990 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:26.990 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:04:26.990 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:52.517846+0000", 2024-03-11T21:04:26.990 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1859999999999992e-06 2024-03-11T21:04:26.990 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:26.990 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:26.990 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:04:26.990 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:52.517903+0000", 2024-03-11T21:04:26.990 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7476999999999999e-05 2024-03-11T21:04:26.990 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:26.990 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:26.991 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:04:26.991 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:52.518029+0000", 2024-03-11T21:04:26.991 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012561199999999999 2024-03-11T21:04:26.991 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:26.991 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:26.991 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0,2", 2024-03-11T21:04:26.991 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:52.518158+0000", 2024-03-11T21:04:26.991 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012880199999999999 2024-03-11T21:04:26.991 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:26.991 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:26.991 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-03-11T21:04:26.991 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:52.519914+0000", 2024-03-11T21:04:26.991 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0017567380000000001 2024-03-11T21:04:26.991 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:26.991 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:26.992 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:04:26.992 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:52.520430+0000", 2024-03-11T21:04:26.992 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00051583800000000002 2024-03-11T21:04:26.992 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:26.992 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:26.992 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:04:26.992 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:52.851890+0000", 2024-03-11T21:04:26.992 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.33146021599999997 2024-03-11T21:04:26.992 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:26.992 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:26.992 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:04:26.992 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:52.851945+0000", 2024-03-11T21:04:26.992 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4614000000000003e-05 2024-03-11T21:04:26.992 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:26.992 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:26.993 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:04:26.993 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T20:55:52.851972+0000", 2024-03-11T21:04:26.993 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6534e-05 2024-03-11T21:04:26.993 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:04:26.993 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:04:26.993 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:04:26.993 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:26.993 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:26.993 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.15081.0:45727 9.e 9:7adeeb78:::benchmark_data_smithi023_106206_object45726:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e262)", 2024-03-11T21:04:26.993 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T21:03:27.496015+0000", 2024-03-11T21:04:26.993 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 59.469047783999997, 2024-03-11T21:04:26.993 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.42054282100000001, 2024-03-11T21:04:26.993 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:04:26.993 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:04:26.993 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-03-11T21:04:26.994 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.15081", 2024-03-11T21:04:26.994 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.23:0/2841194346", 2024-03-11T21:04:26.994 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 45727 2024-03-11T21:04:26.994 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:26.994 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:04:26.994 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:26.994 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:04:26.994 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:27.496015+0000", 2024-03-11T21:04:26.994 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:04:26.994 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:26.994 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:26.994 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:04:26.994 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:27.496015+0000", 2024-03-11T21:04:26.994 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:04:26.995 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:26.995 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:26.995 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:04:26.995 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:27.496016+0000", 2024-03-11T21:04:26.995 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0020000000000001e-06 2024-03-11T21:04:26.995 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:26.995 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:26.995 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:04:26.995 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:27.496090+0000", 2024-03-11T21:04:26.995 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3446000000000001e-05 2024-03-11T21:04:26.995 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:26.995 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:26.995 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:04:26.995 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:27.496099+0000", 2024-03-11T21:04:26.996 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8780000000000004e-06 2024-03-11T21:04:26.996 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:26.996 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:26.996 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:04:26.996 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:27.496104+0000", 2024-03-11T21:04:26.996 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3299999999999998e-06 2024-03-11T21:04:26.996 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:26.996 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:26.996 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:04:26.996 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:27.496153+0000", 2024-03-11T21:04:26.996 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8754000000000001e-05 2024-03-11T21:04:26.996 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:26.996 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:26.997 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:04:26.997 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:27.496245+0000", 2024-03-11T21:04:26.997 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2763000000000006e-05 2024-03-11T21:04:26.997 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:26.997 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:26.997 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1,4", 2024-03-11T21:04:26.997 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:27.496327+0000", 2024-03-11T21:04:26.997 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1444000000000002e-05 2024-03-11T21:04:26.997 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:26.997 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:26.997 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-03-11T21:04:26.997 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:27.499109+0000", 2024-03-11T21:04:26.997 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0027823909999999999 2024-03-11T21:04:26.997 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:26.997 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:26.998 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:04:26.998 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:27.509429+0000", 2024-03-11T21:04:26.998 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010319968000000001 2024-03-11T21:04:26.998 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:26.998 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:26.998 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:04:26.998 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:27.916352+0000", 2024-03-11T21:04:26.998 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.40692308599999999 2024-03-11T21:04:26.998 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:26.998 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:26.998 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:04:26.998 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:27.916478+0000", 2024-03-11T21:04:26.998 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012558400000000001 2024-03-11T21:04:26.998 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:26.999 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:26.999 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:04:26.999 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:27.916558+0000", 2024-03-11T21:04:26.999 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.0174999999999997e-05 2024-03-11T21:04:26.999 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:04:26.999 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:04:26.999 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:04:26.999 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:26.999 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:26.999 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.15081.0:45729 9.e 9:7553524a:::benchmark_data_smithi023_106206_object45728:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e262)", 2024-03-11T21:04:26.999 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T21:03:27.496359+0000", 2024-03-11T21:04:26.999 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 59.468704473999999, 2024-03-11T21:04:26.999 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.42950042399999999, 2024-03-11T21:04:26.999 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:04:26.999 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:04:27.000 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-03-11T21:04:27.000 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.15081", 2024-03-11T21:04:27.000 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.23:0/2841194346", 2024-03-11T21:04:27.000 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 45729 2024-03-11T21:04:27.000 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.000 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:04:27.000 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.000 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:04:27.000 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:27.496359+0000", 2024-03-11T21:04:27.000 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:04:27.000 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.000 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.000 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:04:27.000 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:27.496359+0000", 2024-03-11T21:04:27.000 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:04:27.001 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.001 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.001 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:04:27.001 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:27.496359+0000", 2024-03-11T21:04:27.001 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4600000000000004e-07 2024-03-11T21:04:27.001 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.001 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.001 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:04:27.001 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:27.496405+0000", 2024-03-11T21:04:27.001 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5654e-05 2024-03-11T21:04:27.001 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.001 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.001 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:04:27.001 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:27.496411+0000", 2024-03-11T21:04:27.001 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1009999999999997e-06 2024-03-11T21:04:27.002 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.002 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.002 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:04:27.002 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:27.496415+0000", 2024-03-11T21:04:27.002 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0330000000000002e-06 2024-03-11T21:04:27.002 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.002 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.002 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:04:27.002 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:27.497867+0000", 2024-03-11T21:04:27.002 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0014521359999999999 2024-03-11T21:04:27.002 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.002 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.002 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:04:27.002 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:27.497988+0000", 2024-03-11T21:04:27.003 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000120538 2024-03-11T21:04:27.003 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.003 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.003 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1,4", 2024-03-11T21:04:27.003 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:27.498149+0000", 2024-03-11T21:04:27.003 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016169700000000001 2024-03-11T21:04:27.003 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.003 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.003 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-03-11T21:04:27.003 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:27.499256+0000", 2024-03-11T21:04:27.003 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001106087 2024-03-11T21:04:27.003 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.003 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.003 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:04:27.003 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:27.509464+0000", 2024-03-11T21:04:27.003 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010208283 2024-03-11T21:04:27.004 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.004 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.004 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:04:27.004 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:27.925630+0000", 2024-03-11T21:04:27.004 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.41616583299999999 2024-03-11T21:04:27.004 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.004 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.004 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:04:27.004 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:27.925728+0000", 2024-03-11T21:04:27.004 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8717999999999999e-05 2024-03-11T21:04:27.004 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.004 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.004 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:04:27.004 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:27.925859+0000", 2024-03-11T21:04:27.004 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000130698 2024-03-11T21:04:27.005 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:04:27.005 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:04:27.005 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:04:27.005 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.005 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.005 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.15081.0:45786 9.e 9:76f26470:::benchmark_data_smithi023_106206_object45785:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e262)", 2024-03-11T21:04:27.005 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T21:03:27.613526+0000", 2024-03-11T21:04:27.005 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 59.351536746999997, 2024-03-11T21:04:27.005 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.31262665099999998, 2024-03-11T21:04:27.005 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:04:27.005 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:04:27.005 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-03-11T21:04:27.005 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.15081", 2024-03-11T21:04:27.005 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.23:0/2841194346", 2024-03-11T21:04:27.006 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 45786 2024-03-11T21:04:27.006 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.006 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:04:27.006 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.006 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:04:27.006 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:27.613526+0000", 2024-03-11T21:04:27.006 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:04:27.006 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.006 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.006 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:04:27.006 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:27.613526+0000", 2024-03-11T21:04:27.006 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:04:27.006 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.006 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.006 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:04:27.007 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:27.613528+0000", 2024-03-11T21:04:27.007 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2869999999999999e-06 2024-03-11T21:04:27.007 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.007 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.007 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:04:27.007 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:27.613585+0000", 2024-03-11T21:04:27.007 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7750000000000001e-05 2024-03-11T21:04:27.007 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.007 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.007 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:04:27.007 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:27.613602+0000", 2024-03-11T21:04:27.007 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6864e-05 2024-03-11T21:04:27.007 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.007 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.008 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:04:27.008 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:27.613612+0000", 2024-03-11T21:04:27.008 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8740000000000007e-06 2024-03-11T21:04:27.008 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.008 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.008 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:04:27.008 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:27.613664+0000", 2024-03-11T21:04:27.008 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1904999999999999e-05 2024-03-11T21:04:27.008 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.008 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.008 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:04:27.008 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:27.613750+0000", 2024-03-11T21:04:27.008 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5916999999999997e-05 2024-03-11T21:04:27.008 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.009 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.009 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1,4", 2024-03-11T21:04:27.009 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:27.613836+0000", 2024-03-11T21:04:27.009 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5661000000000005e-05 2024-03-11T21:04:27.009 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.009 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.009 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-03-11T21:04:27.009 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:27.617074+0000", 2024-03-11T21:04:27.009 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0032386530000000002 2024-03-11T21:04:27.009 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.009 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.009 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:04:27.009 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:27.617138+0000", 2024-03-11T21:04:27.009 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4172999999999997e-05 2024-03-11T21:04:27.009 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.010 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.010 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:04:27.010 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:27.925976+0000", 2024-03-11T21:04:27.010 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.30883720999999997 2024-03-11T21:04:27.010 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.010 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.010 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:04:27.010 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:27.926077+0000", 2024-03-11T21:04:27.010 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000101162 2024-03-11T21:04:27.010 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.010 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.010 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:04:27.010 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:27.926153+0000", 2024-03-11T21:04:27.010 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6194999999999998e-05 2024-03-11T21:04:27.010 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:04:27.011 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:04:27.011 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:04:27.011 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.011 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.011 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.15081.0:45787 9.e 9:725b6c6c:::benchmark_data_smithi023_106206_object45786:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e262)", 2024-03-11T21:04:27.011 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T21:03:27.613973+0000", 2024-03-11T21:04:27.011 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 59.351090386000003, 2024-03-11T21:04:27.011 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.31248040399999999, 2024-03-11T21:04:27.011 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:04:27.011 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:04:27.011 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-03-11T21:04:27.011 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.15081", 2024-03-11T21:04:27.011 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.23:0/2841194346", 2024-03-11T21:04:27.011 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 45787 2024-03-11T21:04:27.012 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.012 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:04:27.012 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.012 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:04:27.012 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:27.613973+0000", 2024-03-11T21:04:27.012 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:04:27.012 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.012 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.012 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:04:27.012 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:27.613973+0000", 2024-03-11T21:04:27.012 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:04:27.012 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.012 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.012 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:04:27.012 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:27.613974+0000", 2024-03-11T21:04:27.013 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.226e-06 2024-03-11T21:04:27.013 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.013 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.013 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:04:27.013 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:27.614059+0000", 2024-03-11T21:04:27.013 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5109000000000001e-05 2024-03-11T21:04:27.013 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.013 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.013 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:04:27.013 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:27.614074+0000", 2024-03-11T21:04:27.013 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5124e-05 2024-03-11T21:04:27.013 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.013 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.013 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:04:27.013 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:27.614087+0000", 2024-03-11T21:04:27.014 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2931999999999999e-05 2024-03-11T21:04:27.014 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.014 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.014 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:04:27.014 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:27.615072+0000", 2024-03-11T21:04:27.014 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0009850639999999999 2024-03-11T21:04:27.014 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.014 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.014 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:04:27.014 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:27.615237+0000", 2024-03-11T21:04:27.014 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016469400000000001 2024-03-11T21:04:27.014 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.014 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.014 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1,4", 2024-03-11T21:04:27.015 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:27.615404+0000", 2024-03-11T21:04:27.015 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000167462 2024-03-11T21:04:27.015 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.015 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.015 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:04:27.015 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:27.617206+0000", 2024-03-11T21:04:27.015 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001801225 2024-03-11T21:04:27.015 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.015 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.015 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-03-11T21:04:27.015 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:27.617231+0000", 2024-03-11T21:04:27.015 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5245e-05 2024-03-11T21:04:27.015 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.015 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.016 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:04:27.016 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:27.926260+0000", 2024-03-11T21:04:27.016 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.30902887299999998 2024-03-11T21:04:27.016 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.016 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.016 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:04:27.016 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:27.926353+0000", 2024-03-11T21:04:27.016 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.3782000000000004e-05 2024-03-11T21:04:27.016 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.016 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.016 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:04:27.016 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:27.926453+0000", 2024-03-11T21:04:27.016 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.9667999999999995e-05 2024-03-11T21:04:27.016 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:04:27.016 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:04:27.017 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:04:27.017 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.017 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.017 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.15081.0:64017 9.e 9:743c5f8a:::benchmark_data_smithi023_106206_object64016:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e271)", 2024-03-11T21:04:27.017 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T21:03:42.888748+0000", 2024-03-11T21:04:27.017 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 44.076315446000002, 2024-03-11T21:04:27.017 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.358813559, 2024-03-11T21:04:27.017 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:04:27.017 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:04:27.017 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-03-11T21:04:27.017 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.15081", 2024-03-11T21:04:27.018 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.23:0/2841194346", 2024-03-11T21:04:27.018 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 64017 2024-03-11T21:04:27.018 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.018 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:04:27.018 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.018 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:04:27.018 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:42.888748+0000", 2024-03-11T21:04:27.018 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:04:27.018 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.018 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.018 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:04:27.018 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:42.888748+0000", 2024-03-11T21:04:27.018 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:04:27.018 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.019 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.019 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:04:27.019 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:42.888749+0000", 2024-03-11T21:04:27.019 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1939999999999999e-06 2024-03-11T21:04:27.019 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.019 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.019 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:04:27.019 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:42.888798+0000", 2024-03-11T21:04:27.019 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8835000000000003e-05 2024-03-11T21:04:27.019 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.019 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.019 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:04:27.019 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:42.888809+0000", 2024-03-11T21:04:27.019 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0910000000000001e-05 2024-03-11T21:04:27.019 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.020 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.020 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:04:27.020 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:42.888814+0000", 2024-03-11T21:04:27.020 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8749999999999997e-06 2024-03-11T21:04:27.020 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.020 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.020 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:04:27.020 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:42.888873+0000", 2024-03-11T21:04:27.020 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8245999999999998e-05 2024-03-11T21:04:27.020 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.020 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.020 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:04:27.020 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:42.889100+0000", 2024-03-11T21:04:27.020 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000227001 2024-03-11T21:04:27.020 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.021 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.021 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1,4", 2024-03-11T21:04:27.021 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:42.889270+0000", 2024-03-11T21:04:27.021 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017050600000000001 2024-03-11T21:04:27.021 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.021 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.021 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:04:27.021 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:42.891656+0000", 2024-03-11T21:04:27.021 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0023857700000000002 2024-03-11T21:04:27.021 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.021 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.021 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-03-11T21:04:27.021 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:42.894103+0000", 2024-03-11T21:04:27.021 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0024474309999999999 2024-03-11T21:04:27.022 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.022 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.022 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:04:27.022 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:43.247331+0000", 2024-03-11T21:04:27.022 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.35322764699999998 2024-03-11T21:04:27.022 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.022 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.022 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:04:27.022 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:43.247460+0000", 2024-03-11T21:04:27.022 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012848499999999999 2024-03-11T21:04:27.022 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.022 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.022 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:04:27.022 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:43.247561+0000", 2024-03-11T21:04:27.023 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000101659 2024-03-11T21:04:27.023 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:04:27.023 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:04:27.023 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:04:27.023 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.023 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.023 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.15081.0:64018 9.e 9:775a0dd0:::benchmark_data_smithi023_106206_object64017:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e271)", 2024-03-11T21:04:27.023 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T21:03:42.888951+0000", 2024-03-11T21:04:27.023 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 44.076112111, 2024-03-11T21:04:27.023 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.36171358599999998, 2024-03-11T21:04:27.023 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:04:27.023 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:04:27.023 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-03-11T21:04:27.023 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.15081", 2024-03-11T21:04:27.023 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.23:0/2841194346", 2024-03-11T21:04:27.024 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 64018 2024-03-11T21:04:27.024 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.024 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:04:27.024 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.024 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:04:27.024 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:42.888951+0000", 2024-03-11T21:04:27.024 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:04:27.024 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.024 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.024 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:04:27.024 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:42.888951+0000", 2024-03-11T21:04:27.024 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:04:27.024 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.024 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.024 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:04:27.025 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:42.888952+0000", 2024-03-11T21:04:27.025 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8299999999999997e-07 2024-03-11T21:04:27.025 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.025 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.025 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:04:27.025 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:42.889006+0000", 2024-03-11T21:04:27.025 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4595999999999997e-05 2024-03-11T21:04:27.025 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.025 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.025 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:04:27.025 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:42.889013+0000", 2024-03-11T21:04:27.025 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4200000000000004e-06 2024-03-11T21:04:27.025 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.025 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.025 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:04:27.026 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:42.889017+0000", 2024-03-11T21:04:27.026 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.194e-06 2024-03-11T21:04:27.026 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.026 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.026 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:04:27.026 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:42.891692+0000", 2024-03-11T21:04:27.026 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002674868 2024-03-11T21:04:27.026 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.026 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.026 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:04:27.026 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:42.891825+0000", 2024-03-11T21:04:27.026 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013372300000000001 2024-03-11T21:04:27.026 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.026 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.027 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1,4", 2024-03-11T21:04:27.027 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:42.891978+0000", 2024-03-11T21:04:27.027 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015290099999999999 2024-03-11T21:04:27.027 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.027 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.027 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:04:27.027 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:42.894128+0000", 2024-03-11T21:04:27.027 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002149677 2024-03-11T21:04:27.027 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.027 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.027 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-03-11T21:04:27.027 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:42.894248+0000", 2024-03-11T21:04:27.027 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000120522 2024-03-11T21:04:27.027 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.027 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.028 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:04:27.028 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:43.250430+0000", 2024-03-11T21:04:27.028 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.35618196600000002 2024-03-11T21:04:27.028 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.028 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.028 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:04:27.028 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:43.250577+0000", 2024-03-11T21:04:27.028 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014681300000000001 2024-03-11T21:04:27.028 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.028 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.028 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:04:27.028 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:43.250665+0000", 2024-03-11T21:04:27.028 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.7323000000000004e-05 2024-03-11T21:04:27.028 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:04:27.028 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:04:27.029 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:04:27.029 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.029 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.029 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.15081.0:64021 9.e 9:7b151e6c:::benchmark_data_smithi023_106206_object64020:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e271)", 2024-03-11T21:04:27.029 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T21:03:42.896976+0000", 2024-03-11T21:04:27.029 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 44.068087226000003, 2024-03-11T21:04:27.029 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.35395735900000003, 2024-03-11T21:04:27.029 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:04:27.029 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:04:27.029 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-03-11T21:04:27.029 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.15081", 2024-03-11T21:04:27.029 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.23:0/2841194346", 2024-03-11T21:04:27.029 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 64021 2024-03-11T21:04:27.029 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.030 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:04:27.030 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.030 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:04:27.030 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:42.896976+0000", 2024-03-11T21:04:27.030 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:04:27.030 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.030 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.030 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:04:27.030 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:42.896976+0000", 2024-03-11T21:04:27.030 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:04:27.030 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.030 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.030 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:04:27.030 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:42.896977+0000", 2024-03-11T21:04:27.030 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.587e-06 2024-03-11T21:04:27.031 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.031 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.031 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:04:27.031 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:42.897030+0000", 2024-03-11T21:04:27.031 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2129000000000002e-05 2024-03-11T21:04:27.031 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.031 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.031 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:04:27.031 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:42.897042+0000", 2024-03-11T21:04:27.031 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2789000000000001e-05 2024-03-11T21:04:27.031 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.031 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.031 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:04:27.031 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:42.897052+0000", 2024-03-11T21:04:27.031 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2110000000000004e-06 2024-03-11T21:04:27.032 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.032 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.032 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:04:27.032 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:42.897100+0000", 2024-03-11T21:04:27.032 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7995999999999999e-05 2024-03-11T21:04:27.032 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.032 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.032 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:04:27.032 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:42.897215+0000", 2024-03-11T21:04:27.032 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000114957 2024-03-11T21:04:27.032 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.032 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.033 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1,4", 2024-03-11T21:04:27.033 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:42.897315+0000", 2024-03-11T21:04:27.033 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000100084 2024-03-11T21:04:27.033 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.033 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.033 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:04:27.033 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:42.898979+0000", 2024-03-11T21:04:27.033 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001664008 2024-03-11T21:04:27.033 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.033 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.033 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-03-11T21:04:27.033 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:42.898987+0000", 2024-03-11T21:04:27.033 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.3569999999999998e-06 2024-03-11T21:04:27.034 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.034 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.034 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:04:27.034 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:43.250749+0000", 2024-03-11T21:04:27.034 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.351761571 2024-03-11T21:04:27.034 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.034 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.034 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:04:27.034 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:43.250856+0000", 2024-03-11T21:04:27.034 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000107044 2024-03-11T21:04:27.034 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.034 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.034 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:04:27.035 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:43.250933+0000", 2024-03-11T21:04:27.035 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.7626000000000005e-05 2024-03-11T21:04:27.035 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:04:27.035 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:04:27.035 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:04:27.035 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.035 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.035 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.15081.0:66776 9.d 9:bb5d3949:::benchmark_data_smithi023_106206_object66775:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e271)", 2024-03-11T21:04:27.035 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T21:03:45.808861+0000", 2024-03-11T21:04:27.035 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 41.156202006000001, 2024-03-11T21:04:27.035 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.36772420300000003, 2024-03-11T21:04:27.035 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:04:27.035 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:04:27.035 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-03-11T21:04:27.035 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.15081", 2024-03-11T21:04:27.036 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.23:0/2841194346", 2024-03-11T21:04:27.036 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 66776 2024-03-11T21:04:27.036 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.036 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:04:27.036 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.036 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:04:27.036 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:45.808861+0000", 2024-03-11T21:04:27.036 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:04:27.036 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.036 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.036 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:04:27.036 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:45.808861+0000", 2024-03-11T21:04:27.036 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:04:27.036 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.036 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.037 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:04:27.037 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:45.808863+0000", 2024-03-11T21:04:27.037 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6029999999999999e-06 2024-03-11T21:04:27.037 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.037 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.037 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:04:27.037 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:45.808933+0000", 2024-03-11T21:04:27.037 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0706999999999999e-05 2024-03-11T21:04:27.037 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.037 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.037 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:04:27.037 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:45.808945+0000", 2024-03-11T21:04:27.037 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1678000000000001e-05 2024-03-11T21:04:27.037 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.037 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.038 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:04:27.038 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:45.808953+0000", 2024-03-11T21:04:27.038 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1759999999999993e-06 2024-03-11T21:04:27.038 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.038 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.038 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:04:27.038 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:45.809006+0000", 2024-03-11T21:04:27.038 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3007999999999998e-05 2024-03-11T21:04:27.038 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.038 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.038 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:04:27.038 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:45.809126+0000", 2024-03-11T21:04:27.038 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000119881 2024-03-11T21:04:27.039 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.039 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.039 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1,5", 2024-03-11T21:04:27.039 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:45.809227+0000", 2024-03-11T21:04:27.039 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000100838 2024-03-11T21:04:27.039 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.039 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.039 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-03-11T21:04:27.039 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:45.811007+0000", 2024-03-11T21:04:27.039 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0017801270000000001 2024-03-11T21:04:27.039 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.039 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.039 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:04:27.039 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:45.815297+0000", 2024-03-11T21:04:27.039 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0042899050000000001 2024-03-11T21:04:27.040 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.040 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.040 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:04:27.040 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:46.176082+0000", 2024-03-11T21:04:27.040 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.36078505199999999 2024-03-11T21:04:27.040 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.040 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.040 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:04:27.040 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:46.176507+0000", 2024-03-11T21:04:27.040 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00042532099999999999 2024-03-11T21:04:27.040 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.040 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.040 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:04:27.040 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:46.176585+0000", 2024-03-11T21:04:27.040 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.7906999999999998e-05 2024-03-11T21:04:27.041 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:04:27.041 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:04:27.041 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:04:27.041 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.041 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.041 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.15081.0:66794 9.9 9:9a544e5a:::benchmark_data_smithi023_106206_object66793:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e271)", 2024-03-11T21:04:27.041 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T21:03:45.830289+0000", 2024-03-11T21:04:27.041 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 41.134774303, 2024-03-11T21:04:27.041 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.32850591099999998, 2024-03-11T21:04:27.041 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:04:27.041 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:04:27.041 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-03-11T21:04:27.041 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.15081", 2024-03-11T21:04:27.041 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.23:0/2841194346", 2024-03-11T21:04:27.042 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 66794 2024-03-11T21:04:27.042 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.042 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:04:27.042 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.042 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:04:27.042 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:45.830289+0000", 2024-03-11T21:04:27.042 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:04:27.042 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.042 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.042 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:04:27.042 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:45.830289+0000", 2024-03-11T21:04:27.042 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:04:27.042 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.042 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.042 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:04:27.043 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:45.830290+0000", 2024-03-11T21:04:27.043 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.8299999999999996e-07 2024-03-11T21:04:27.043 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.043 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.043 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:04:27.043 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:45.830348+0000", 2024-03-11T21:04:27.043 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8094000000000003e-05 2024-03-11T21:04:27.043 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.043 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.043 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:04:27.043 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:45.830357+0000", 2024-03-11T21:04:27.043 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4010000000000003e-06 2024-03-11T21:04:27.043 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.043 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.043 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:04:27.044 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:45.830365+0000", 2024-03-11T21:04:27.044 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.96e-06 2024-03-11T21:04:27.044 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.046 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.046 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:04:27.046 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:45.830420+0000", 2024-03-11T21:04:27.046 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4577000000000003e-05 2024-03-11T21:04:27.046 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.046 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.046 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:04:27.046 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:45.830524+0000", 2024-03-11T21:04:27.047 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000104666 2024-03-11T21:04:27.047 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.047 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.047 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0,5", 2024-03-11T21:04:27.047 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:45.830613+0000", 2024-03-11T21:04:27.047 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8517999999999995e-05 2024-03-11T21:04:27.047 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.047 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.047 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-03-11T21:04:27.047 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:45.832392+0000", 2024-03-11T21:04:27.047 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0017789559999999999 2024-03-11T21:04:27.047 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.047 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.047 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:04:27.047 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:45.832764+0000", 2024-03-11T21:04:27.048 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00037190600000000003 2024-03-11T21:04:27.048 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.048 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.048 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:04:27.048 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:46.158062+0000", 2024-03-11T21:04:27.048 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.32529847200000001 2024-03-11T21:04:27.048 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.048 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.048 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:04:27.048 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:46.158710+0000", 2024-03-11T21:04:27.048 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00064813200000000005 2024-03-11T21:04:27.048 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.048 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.048 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:04:27.048 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:46.158795+0000", 2024-03-11T21:04:27.048 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4445999999999998e-05 2024-03-11T21:04:27.049 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:04:27.049 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:04:27.049 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:04:27.049 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.049 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.049 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.15081.0:94735 9.d 9:b0b306b5:::benchmark_data_smithi023_106206_object94734:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e275)", 2024-03-11T21:04:27.049 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T21:04:14.852238+0000", 2024-03-11T21:04:27.049 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 12.112825238999999, 2024-03-11T21:04:27.049 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.41739162899999999, 2024-03-11T21:04:27.049 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:04:27.049 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:04:27.049 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-03-11T21:04:27.049 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.15081", 2024-03-11T21:04:27.049 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.23:0/2841194346", 2024-03-11T21:04:27.050 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 94735 2024-03-11T21:04:27.050 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.050 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:04:27.050 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.050 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:04:27.050 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:14.852238+0000", 2024-03-11T21:04:27.050 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:04:27.050 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.050 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.050 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:04:27.050 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:14.852238+0000", 2024-03-11T21:04:27.050 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:04:27.050 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.050 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.051 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:04:27.051 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:14.852239+0000", 2024-03-11T21:04:27.051 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.7700000000000004e-07 2024-03-11T21:04:27.051 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.051 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.051 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:04:27.051 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:14.852292+0000", 2024-03-11T21:04:27.051 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3486999999999998e-05 2024-03-11T21:04:27.051 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.051 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.051 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:04:27.051 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:14.852302+0000", 2024-03-11T21:04:27.051 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0118e-05 2024-03-11T21:04:27.051 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.051 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.052 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:04:27.052 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:14.852309+0000", 2024-03-11T21:04:27.052 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.985e-06 2024-03-11T21:04:27.052 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.052 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.052 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:04:27.052 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:14.852356+0000", 2024-03-11T21:04:27.052 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6635999999999999e-05 2024-03-11T21:04:27.052 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.052 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.052 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:04:27.052 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:14.852445+0000", 2024-03-11T21:04:27.052 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.9392000000000001e-05 2024-03-11T21:04:27.052 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.052 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.053 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1,5", 2024-03-11T21:04:27.053 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:14.852530+0000", 2024-03-11T21:04:27.053 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4893000000000002e-05 2024-03-11T21:04:27.053 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.053 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.053 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-03-11T21:04:27.053 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:14.855129+0000", 2024-03-11T21:04:27.053 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002598731 2024-03-11T21:04:27.053 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.053 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.053 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:04:27.053 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:14.857899+0000", 2024-03-11T21:04:27.053 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002769886 2024-03-11T21:04:27.053 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.054 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.054 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:04:27.054 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:15.269508+0000", 2024-03-11T21:04:27.054 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.41160974700000003 2024-03-11T21:04:27.054 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.054 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.054 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:04:27.054 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:15.269575+0000", 2024-03-11T21:04:27.054 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6303000000000005e-05 2024-03-11T21:04:27.054 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.054 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.054 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:04:27.054 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:15.269629+0000", 2024-03-11T21:04:27.054 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4673999999999999e-05 2024-03-11T21:04:27.054 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:04:27.055 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:04:27.055 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:04:27.055 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.055 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.055 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.15081.0:94739 9.9 9:99bbc5ed:::benchmark_data_smithi023_106206_object94738:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e275)", 2024-03-11T21:04:27.055 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T21:04:14.858831+0000", 2024-03-11T21:04:27.055 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 12.106231893, 2024-03-11T21:04:27.055 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.40680787499999999, 2024-03-11T21:04:27.055 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:04:27.055 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:04:27.055 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-03-11T21:04:27.055 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.15081", 2024-03-11T21:04:27.055 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.23:0/2841194346", 2024-03-11T21:04:27.055 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 94739 2024-03-11T21:04:27.056 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.056 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:04:27.056 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.056 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:04:27.056 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:14.858831+0000", 2024-03-11T21:04:27.056 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:04:27.056 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.056 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.056 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:04:27.056 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:14.858831+0000", 2024-03-11T21:04:27.056 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:04:27.056 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.056 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.056 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:04:27.056 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:14.858833+0000", 2024-03-11T21:04:27.057 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2800000000000002e-06 2024-03-11T21:04:27.057 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.057 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.057 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:04:27.057 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:14.858891+0000", 2024-03-11T21:04:27.057 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7256000000000002e-05 2024-03-11T21:04:27.057 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.057 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.057 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:04:27.057 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:14.858916+0000", 2024-03-11T21:04:27.057 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4816000000000001e-05 2024-03-11T21:04:27.057 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.057 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.057 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:04:27.057 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:14.858931+0000", 2024-03-11T21:04:27.057 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5903e-05 2024-03-11T21:04:27.058 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.058 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.058 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:04:27.058 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:14.859017+0000", 2024-03-11T21:04:27.058 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5267999999999997e-05 2024-03-11T21:04:27.058 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.058 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.058 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:04:27.058 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:14.859111+0000", 2024-03-11T21:04:27.058 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4406000000000004e-05 2024-03-11T21:04:27.058 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.058 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.058 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0,5", 2024-03-11T21:04:27.058 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:14.859203+0000", 2024-03-11T21:04:27.059 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2260000000000001e-05 2024-03-11T21:04:27.059 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.059 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.059 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-03-11T21:04:27.059 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:14.861163+0000", 2024-03-11T21:04:27.059 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0019593829999999999 2024-03-11T21:04:27.059 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.059 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.059 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:04:27.059 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:14.862876+0000", 2024-03-11T21:04:27.059 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001713011 2024-03-11T21:04:27.059 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.059 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.059 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:04:27.059 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:15.265491+0000", 2024-03-11T21:04:27.059 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.40261487899999998 2024-03-11T21:04:27.060 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.060 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.060 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:04:27.060 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:15.265571+0000", 2024-03-11T21:04:27.060 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.0411e-05 2024-03-11T21:04:27.060 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.060 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.060 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:04:27.060 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:15.265639+0000", 2024-03-11T21:04:27.060 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8002000000000004e-05 2024-03-11T21:04:27.060 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:04:27.060 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:04:27.060 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:04:27.060 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.060 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.061 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.15081.0:94743 9.9 9:95a8cf82:::benchmark_data_smithi023_106206_object94742:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e275)", 2024-03-11T21:04:27.061 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T21:04:14.864738+0000", 2024-03-11T21:04:27.061 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 12.100325570000001, 2024-03-11T21:04:27.061 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.40752799899999997, 2024-03-11T21:04:27.061 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:04:27.061 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:04:27.061 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-03-11T21:04:27.061 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.15081", 2024-03-11T21:04:27.061 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.23:0/2841194346", 2024-03-11T21:04:27.061 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 94743 2024-03-11T21:04:27.061 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.061 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:04:27.061 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.061 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:04:27.061 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:14.864738+0000", 2024-03-11T21:04:27.062 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:04:27.062 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.062 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.062 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:04:27.062 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:14.864738+0000", 2024-03-11T21:04:27.062 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:04:27.062 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.062 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.062 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:04:27.062 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:14.864738+0000", 2024-03-11T21:04:27.062 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4300000000000003e-07 2024-03-11T21:04:27.062 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.062 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.062 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:04:27.063 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:14.864804+0000", 2024-03-11T21:04:27.063 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5454e-05 2024-03-11T21:04:27.063 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.063 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.063 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:04:27.063 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:14.864811+0000", 2024-03-11T21:04:27.063 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.097e-06 2024-03-11T21:04:27.063 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.063 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.063 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:04:27.063 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:14.864815+0000", 2024-03-11T21:04:27.063 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7609999999999998e-06 2024-03-11T21:04:27.063 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.063 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.063 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:04:27.064 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:14.864849+0000", 2024-03-11T21:04:27.064 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3481e-05 2024-03-11T21:04:27.064 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.064 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.064 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:04:27.064 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:14.864935+0000", 2024-03-11T21:04:27.064 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.6544000000000004e-05 2024-03-11T21:04:27.064 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.064 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.064 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0,5", 2024-03-11T21:04:27.064 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:14.865020+0000", 2024-03-11T21:04:27.064 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4939999999999997e-05 2024-03-11T21:04:27.064 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.064 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.065 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-03-11T21:04:27.065 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:14.866294+0000", 2024-03-11T21:04:27.065 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001274 2024-03-11T21:04:27.065 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.065 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.065 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:04:27.065 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:14.869032+0000", 2024-03-11T21:04:27.065 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0027375989999999998 2024-03-11T21:04:27.065 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.065 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.065 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:04:27.065 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:15.272132+0000", 2024-03-11T21:04:27.065 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.40310035999999999 2024-03-11T21:04:27.065 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.065 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.066 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:04:27.066 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:15.272210+0000", 2024-03-11T21:04:27.066 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.7738999999999995e-05 2024-03-11T21:04:27.066 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.066 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.066 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:04:27.066 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:15.272266+0000", 2024-03-11T21:04:27.066 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5380999999999997e-05 2024-03-11T21:04:27.066 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:04:27.066 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:04:27.066 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:04:27.066 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.066 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.066 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.15081.0:100107 9.e 9:7103d6a1:::benchmark_data_smithi023_106206_object100106:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e276)", 2024-03-11T21:04:27.067 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T21:04:20.252914+0000", 2024-03-11T21:04:27.067 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 6.7121488649999996, 2024-03-11T21:04:27.067 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.42876025899999998, 2024-03-11T21:04:27.067 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:04:27.067 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:04:27.067 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-03-11T21:04:27.067 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.15081", 2024-03-11T21:04:27.067 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.23:0/2841194346", 2024-03-11T21:04:27.067 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 100107 2024-03-11T21:04:27.067 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.067 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:04:27.067 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.067 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:04:27.067 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.252914+0000", 2024-03-11T21:04:27.067 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:04:27.068 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.068 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.068 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:04:27.068 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.252914+0000", 2024-03-11T21:04:27.068 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:04:27.068 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.068 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.068 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:04:27.068 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.252915+0000", 2024-03-11T21:04:27.068 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2190000000000001e-06 2024-03-11T21:04:27.068 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.068 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.068 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:04:27.068 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.252988+0000", 2024-03-11T21:04:27.068 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2510999999999997e-05 2024-03-11T21:04:27.069 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.069 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.069 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:04:27.069 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.253003+0000", 2024-03-11T21:04:27.069 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5152e-05 2024-03-11T21:04:27.069 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.069 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.069 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:04:27.069 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.253011+0000", 2024-03-11T21:04:27.069 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.8970000000000008e-06 2024-03-11T21:04:27.069 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.069 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.069 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:04:27.069 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.253074+0000", 2024-03-11T21:04:27.069 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3347000000000002e-05 2024-03-11T21:04:27.070 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.070 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.070 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:04:27.070 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.253211+0000", 2024-03-11T21:04:27.070 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013664600000000001 2024-03-11T21:04:27.070 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.070 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.070 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1,4", 2024-03-11T21:04:27.070 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.253370+0000", 2024-03-11T21:04:27.070 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015909999999999999 2024-03-11T21:04:27.070 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.070 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.070 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-03-11T21:04:27.071 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.256333+0000", 2024-03-11T21:04:27.071 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002962559 2024-03-11T21:04:27.071 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.071 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.071 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:04:27.071 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.264983+0000", 2024-03-11T21:04:27.071 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0086507130000000008 2024-03-11T21:04:27.071 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.071 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.071 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:04:27.071 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.681571+0000", 2024-03-11T21:04:27.071 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.416587239 2024-03-11T21:04:27.071 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.071 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.071 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:04:27.072 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.681631+0000", 2024-03-11T21:04:27.072 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0851999999999999e-05 2024-03-11T21:04:27.072 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.072 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.072 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:04:27.072 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.681674+0000", 2024-03-11T21:04:27.072 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3024e-05 2024-03-11T21:04:27.072 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:04:27.072 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:04:27.072 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:04:27.072 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.072 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.072 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.15081.0:100110 9.e 9:7d11b99e:::benchmark_data_smithi023_106206_object100109:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e276)", 2024-03-11T21:04:27.072 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T21:04:20.256346+0000", 2024-03-11T21:04:27.072 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 6.7087168620000002, 2024-03-11T21:04:27.073 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.42547889500000002, 2024-03-11T21:04:27.073 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:04:27.073 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:04:27.073 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-03-11T21:04:27.073 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.15081", 2024-03-11T21:04:27.073 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.23:0/2841194346", 2024-03-11T21:04:27.073 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 100110 2024-03-11T21:04:27.073 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.073 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:04:27.073 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.073 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:04:27.073 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.256346+0000", 2024-03-11T21:04:27.073 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:04:27.073 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.074 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.074 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:04:27.074 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.256346+0000", 2024-03-11T21:04:27.074 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:04:27.074 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.074 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.074 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:04:27.074 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.256347+0000", 2024-03-11T21:04:27.074 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2359999999999999e-06 2024-03-11T21:04:27.074 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.074 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.074 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:04:27.074 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.256378+0000", 2024-03-11T21:04:27.074 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0459000000000001e-05 2024-03-11T21:04:27.074 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.075 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.075 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:04:27.075 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.256391+0000", 2024-03-11T21:04:27.075 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2673999999999999e-05 2024-03-11T21:04:27.075 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.075 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.075 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:04:27.075 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.256400+0000", 2024-03-11T21:04:27.075 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2769999999999994e-06 2024-03-11T21:04:27.075 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.075 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.075 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:04:27.075 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.256464+0000", 2024-03-11T21:04:27.075 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3739999999999996e-05 2024-03-11T21:04:27.076 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.076 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.076 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:04:27.076 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.256594+0000", 2024-03-11T21:04:27.076 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013073699999999999 2024-03-11T21:04:27.076 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.076 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.076 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1,4", 2024-03-11T21:04:27.076 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.256739+0000", 2024-03-11T21:04:27.076 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014482399999999999 2024-03-11T21:04:27.076 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.076 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.076 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-03-11T21:04:27.076 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.264992+0000", 2024-03-11T21:04:27.076 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0082530409999999992 2024-03-11T21:04:27.077 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.077 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.077 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:04:27.077 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.265013+0000", 2024-03-11T21:04:27.077 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0673e-05 2024-03-11T21:04:27.077 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.077 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.077 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:04:27.077 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.681723+0000", 2024-03-11T21:04:27.077 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.41670987900000001 2024-03-11T21:04:27.077 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.077 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.077 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:04:27.077 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.681783+0000", 2024-03-11T21:04:27.078 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9969000000000001e-05 2024-03-11T21:04:27.078 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.078 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.078 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:04:27.078 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.681825+0000", 2024-03-11T21:04:27.078 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2385999999999997e-05 2024-03-11T21:04:27.078 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:04:27.078 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:04:27.078 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:04:27.078 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.078 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.078 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.15081.0:100124 9.3 9:ccb6310d:::benchmark_data_smithi023_106206_object100123:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e276)", 2024-03-11T21:04:27.078 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T21:04:20.270198+0000", 2024-03-11T21:04:27.079 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 6.6948651129999996, 2024-03-11T21:04:27.079 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.40965254000000001, 2024-03-11T21:04:27.079 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:04:27.079 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:04:27.079 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-03-11T21:04:27.079 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.15081", 2024-03-11T21:04:27.079 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.23:0/2841194346", 2024-03-11T21:04:27.079 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 100124 2024-03-11T21:04:27.079 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.079 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:04:27.079 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.079 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:04:27.079 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.270198+0000", 2024-03-11T21:04:27.079 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:04:27.079 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.080 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.080 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:04:27.080 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.270198+0000", 2024-03-11T21:04:27.080 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:04:27.080 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.080 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.080 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:04:27.080 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.270199+0000", 2024-03-11T21:04:27.080 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4810000000000001e-06 2024-03-11T21:04:27.080 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.080 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.080 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:04:27.080 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.270239+0000", 2024-03-11T21:04:27.080 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9388000000000003e-05 2024-03-11T21:04:27.081 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.081 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.081 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:04:27.081 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.270251+0000", 2024-03-11T21:04:27.081 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1861e-05 2024-03-11T21:04:27.081 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.081 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.081 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:04:27.081 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.270268+0000", 2024-03-11T21:04:27.081 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7051e-05 2024-03-11T21:04:27.081 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.081 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.081 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:04:27.081 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.270328+0000", 2024-03-11T21:04:27.081 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9848e-05 2024-03-11T21:04:27.082 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.082 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.082 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:04:27.082 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.270519+0000", 2024-03-11T21:04:27.082 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019136399999999999 2024-03-11T21:04:27.082 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.082 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.082 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1,3", 2024-03-11T21:04:27.082 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.270690+0000", 2024-03-11T21:04:27.082 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000171254 2024-03-11T21:04:27.082 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.082 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.082 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-03-11T21:04:27.082 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.272796+0000", 2024-03-11T21:04:27.082 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0021062350000000001 2024-03-11T21:04:27.083 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.083 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.083 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:04:27.083 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.274410+0000", 2024-03-11T21:04:27.083 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00161399 2024-03-11T21:04:27.083 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.083 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.083 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:04:27.083 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.679768+0000", 2024-03-11T21:04:27.083 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.40535767 2024-03-11T21:04:27.083 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.083 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.083 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:04:27.083 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.679810+0000", 2024-03-11T21:04:27.084 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2001e-05 2024-03-11T21:04:27.084 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.084 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.084 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:04:27.084 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.679851+0000", 2024-03-11T21:04:27.084 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0397e-05 2024-03-11T21:04:27.084 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:04:27.084 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:04:27.084 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:04:27.084 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.084 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.084 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.15081.0:100127 9.d 9:b25a6eb8:::benchmark_data_smithi023_106206_object100126:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e276)", 2024-03-11T21:04:27.084 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T21:04:20.275216+0000", 2024-03-11T21:04:27.084 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 6.689846985, 2024-03-11T21:04:27.084 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.39741401900000001, 2024-03-11T21:04:27.085 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:04:27.085 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:04:27.085 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-03-11T21:04:27.085 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.15081", 2024-03-11T21:04:27.085 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.23:0/2841194346", 2024-03-11T21:04:27.085 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 100127 2024-03-11T21:04:27.085 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.085 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:04:27.085 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.085 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:04:27.085 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.275216+0000", 2024-03-11T21:04:27.085 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:04:27.085 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.085 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.085 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:04:27.086 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.275216+0000", 2024-03-11T21:04:27.086 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:04:27.086 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.086 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.086 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:04:27.086 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.275217+0000", 2024-03-11T21:04:27.086 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3659999999999999e-06 2024-03-11T21:04:27.086 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.086 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.086 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:04:27.086 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.275266+0000", 2024-03-11T21:04:27.086 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8186000000000002e-05 2024-03-11T21:04:27.086 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.086 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.086 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:04:27.087 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.275276+0000", 2024-03-11T21:04:27.087 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0831e-05 2024-03-11T21:04:27.087 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.087 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.087 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:04:27.087 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.275282+0000", 2024-03-11T21:04:27.087 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7230000000000001e-06 2024-03-11T21:04:27.087 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.087 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.087 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:04:27.087 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.275332+0000", 2024-03-11T21:04:27.087 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9954000000000003e-05 2024-03-11T21:04:27.087 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.087 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.087 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:04:27.088 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.275427+0000", 2024-03-11T21:04:27.088 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4592999999999994e-05 2024-03-11T21:04:27.088 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.088 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.088 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1,5", 2024-03-11T21:04:27.088 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.275523+0000", 2024-03-11T21:04:27.088 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5845000000000001e-05 2024-03-11T21:04:27.088 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.088 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.088 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:04:27.088 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.277890+0000", 2024-03-11T21:04:27.088 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0023677479999999998 2024-03-11T21:04:27.088 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.088 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.088 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-03-11T21:04:27.089 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.277972+0000", 2024-03-11T21:04:27.089 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1965999999999994e-05 2024-03-11T21:04:27.089 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.089 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.089 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:04:27.089 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.672543+0000", 2024-03-11T21:04:27.089 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.39457092799999999 2024-03-11T21:04:27.089 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.089 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.089 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:04:27.089 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.672590+0000", 2024-03-11T21:04:27.089 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6281e-05 2024-03-11T21:04:27.089 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:04:27.089 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:04:27.089 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:04:27.090 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.672630+0000", 2024-03-11T21:04:27.090 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0598e-05 2024-03-11T21:04:27.090 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:04:27.090 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:04:27.090 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:04:27.090 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:04:27.090 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:04:27.090 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-03-11T21:04:27.090 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:04:27.113 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:04:27.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:27.116 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:04:27.217 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:04:27.244 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:04:27.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:27.246 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:04:27.347 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:04:27.377 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:04:27.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:27.380 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:04:27.465 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 85 16 106645 106629 78.39 65.1875 0.00875749 0.0127515 2024-03-11T21:04:27.481 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:04:27.516 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:04:27.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:27.518 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:04:27.619 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:04:27.641 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:04:27.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:27.643 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:04:27.744 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:27.773 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:27.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:27.775 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:04:27.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:27 smithi023 bash[21372]: cluster 2024-03-11T21:04:26.176260+0000 mgr.y (mgr.24380) 506 : cluster 0 pgmap v565: 148 pgs: 148 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 581 B/s rd, 55 MiB/s wr, 887 op/s 2024-03-11T21:04:27.783 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:27 smithi023 bash[29146]: cluster 2024-03-11T21:04:26.176260+0000 mgr.y (mgr.24380) 506 : cluster 0 pgmap v565: 148 pgs: 148 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 581 B/s rd, 55 MiB/s wr, 887 op/s 2024-03-11T21:04:27.876 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:04:27.903 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:04:27.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:27.906 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:04:27.951 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:27 smithi146 bash[23809]: cluster 2024-03-11T21:04:26.176260+0000 mgr.y (mgr.24380) 506 : cluster 0 pgmap v565: 148 pgs: 148 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 581 B/s rd, 55 MiB/s wr, 887 op/s 2024-03-11T21:04:28.007 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:04:28.038 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:04:28.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:28.043 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:04:28.144 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:28.178 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:28.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:28.180 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:04:28.281 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:04:28.312 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:04:28.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:28.313 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:04:28.413 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:04:28.433 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-11T21:04:28.445 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:04:28.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:28.451 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:04:28.465 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 86 16 107261 107245 77.9262 38.5 0.0399552 0.0128283 2024-03-11T21:04:28.552 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:04:28.579 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:04:28.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:28.581 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:04:28.682 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:28.709 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:28.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:28.773 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:04:28.782 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:04:28 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:04:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:04:28.874 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:04:28.907 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:04:28.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:28.911 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:04:29.012 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:04:29.035 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:04:29.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:29.041 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:04:29.142 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:04:29.166 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:04:29.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:29.170 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:04:29.243 INFO:teuthology.orchestra.run.smithi023.stdout:osd.0: {} 2024-03-11T21:04:29.270 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:29.296 INFO:teuthology.orchestra.run.smithi023.stdout:osd.1: {} 2024-03-11T21:04:29.307 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:29.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:29.312 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:04:29.347 INFO:teuthology.orchestra.run.smithi023.stdout:osd.2: {} 2024-03-11T21:04:29.392 INFO:teuthology.orchestra.run.smithi023.stdout:osd.3: {} 2024-03-11T21:04:29.413 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:04:29.438 INFO:teuthology.orchestra.run.smithi023.stdout:osd.4: {} 2024-03-11T21:04:29.445 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:04:29.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:29.453 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:04:29.465 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 87 16 108151 108135 77.6698 55.625 0.0126082 0.0128682 2024-03-11T21:04:29.472 INFO:teuthology.orchestra.run.smithi023.stderr:noscrub is unset 2024-03-11T21:04:29.483 INFO:teuthology.orchestra.run.smithi023.stdout:osd.5: {} 2024-03-11T21:04:29.523 INFO:teuthology.orchestra.run.smithi023.stdout:osd.6: {} 2024-03-11T21:04:29.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:29 smithi023 bash[21372]: cluster 2024-03-11T21:04:28.177046+0000 mgr.y (mgr.24380) 507 : cluster 0 pgmap v566: 148 pgs: 148 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 59 MiB/s wr, 950 op/s 2024-03-11T21:04:29.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:29 smithi023 bash[21372]: audit 2024-03-11T21:04:29.180125+0000 mon.a (mon.0) 1508 : audit 0 from='client.? 172.21.15.23:0/1190594059' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:04:29.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:29 smithi023 bash[21372]: audit 2024-03-11T21:04:29.225659+0000 mon.c (mon.2) 65 : audit 1 from='client.? 172.21.15.23:0/617339792' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:04:29.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:29 smithi023 bash[21372]: audit 2024-03-11T21:04:29.226270+0000 mon.a (mon.0) 1509 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:04:29.531 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:29 smithi023 bash[29146]: cluster 2024-03-11T21:04:28.177046+0000 mgr.y (mgr.24380) 507 : cluster 0 pgmap v566: 148 pgs: 148 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 59 MiB/s wr, 950 op/s 2024-03-11T21:04:29.532 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:29 smithi023 bash[29146]: audit 2024-03-11T21:04:29.180125+0000 mon.a (mon.0) 1508 : audit 0 from='client.? 172.21.15.23:0/1190594059' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:04:29.532 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:29 smithi023 bash[29146]: audit 2024-03-11T21:04:29.225659+0000 mon.c (mon.2) 65 : audit 1 from='client.? 172.21.15.23:0/617339792' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:04:29.532 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:29 smithi023 bash[29146]: audit 2024-03-11T21:04:29.226270+0000 mon.a (mon.0) 1509 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:04:29.554 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:04:29.563 INFO:teuthology.orchestra.run.smithi023.stdout:osd.7: {} 2024-03-11T21:04:29.569 INFO:teuthology.orchestra.run.smithi023.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:04:29.569 INFO:teuthology.orchestra.run.smithi023.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:04:29.569 INFO:teuthology.orchestra.run.smithi023.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:04:29.569 INFO:teuthology.orchestra.run.smithi023.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:04:29.569 INFO:teuthology.orchestra.run.smithi023.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:04:29.569 INFO:teuthology.orchestra.run.smithi023.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:04:29.569 INFO:teuthology.orchestra.run.smithi023.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:04:29.569 INFO:teuthology.orchestra.run.smithi023.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:04:29.584 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:04:29.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:29.592 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:04:29.629 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-11T21:04:29.637 INFO:teuthology.orchestra.run.smithi023.stderr:dumped all 2024-03-11T21:04:29.693 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:29.733 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:29.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:29.736 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:04:29.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:29 smithi146 bash[23809]: cluster 2024-03-11T21:04:28.177046+0000 mgr.y (mgr.24380) 507 : cluster 0 pgmap v566: 148 pgs: 148 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 59 MiB/s wr, 950 op/s 2024-03-11T21:04:29.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:29 smithi146 bash[23809]: audit 2024-03-11T21:04:29.180125+0000 mon.a (mon.0) 1508 : audit 0 from='client.? 172.21.15.23:0/1190594059' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:04:29.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:29 smithi146 bash[23809]: audit 2024-03-11T21:04:29.225659+0000 mon.c (mon.2) 65 : audit 1 from='client.? 172.21.15.23:0/617339792' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:04:29.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:29 smithi146 bash[23809]: audit 2024-03-11T21:04:29.226270+0000 mon.a (mon.0) 1509 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:04:29.837 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:04:29.870 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:04:29.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:29.875 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:04:29.976 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:04:30.004 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:04:30.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:30.008 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:04:30.109 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:04:30.136 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:04:30.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:30.140 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:04:30.241 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:04:30.269 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:04:30.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:30.274 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:04:30.375 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:30.401 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:30.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:30.404 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:04:30.466 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 88 16 109110 109094 77.4683 59.9375 0.00617688 0.0129032 2024-03-11T21:04:30.505 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:04:30.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:30.534 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:04:30.535 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:04:30.636 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:04:30.647 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:30 smithi023 bash[21372]: cluster 2024-03-11T21:04:29.442725+0000 mon.a (mon.0) 1510 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:04:30.647 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:30 smithi023 bash[21372]: audit 2024-03-11T21:04:29.458814+0000 mon.a (mon.0) 1511 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:04:30.647 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:30 smithi023 bash[21372]: cluster 2024-03-11T21:04:29.464804+0000 mon.a (mon.0) 1512 : cluster 0 osdmap e278: 8 total, 8 up, 8 in 2024-03-11T21:04:30.647 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:30 smithi023 bash[29146]: cluster 2024-03-11T21:04:29.442725+0000 mon.a (mon.0) 1510 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:04:30.647 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:30 smithi023 bash[29146]: audit 2024-03-11T21:04:29.458814+0000 mon.a (mon.0) 1511 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:04:30.647 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:30 smithi023 bash[29146]: cluster 2024-03-11T21:04:29.464804+0000 mon.a (mon.0) 1512 : cluster 0 osdmap e278: 8 total, 8 up, 8 in 2024-03-11T21:04:30.664 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:04:30.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:30.666 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:04:30.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:30 smithi146 bash[23809]: cluster 2024-03-11T21:04:29.442725+0000 mon.a (mon.0) 1510 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:04:30.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:30 smithi146 bash[23809]: audit 2024-03-11T21:04:29.458814+0000 mon.a (mon.0) 1511 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:04:30.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:30 smithi146 bash[23809]: cluster 2024-03-11T21:04:29.464804+0000 mon.a (mon.0) 1512 : cluster 0 osdmap e278: 8 total, 8 up, 8 in 2024-03-11T21:04:30.768 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:04:30.794 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:04:30.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:30.798 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:04:30.899 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:30.925 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:30.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:30.928 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:04:31.029 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:31.057 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:31.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:31.062 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:04:31.162 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:04:31.193 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:04:31.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:31.196 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:04:31.297 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:04:31.319 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:04:31.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:31.324 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:04:31.424 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:04:31.457 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:04:31.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:31.460 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:04:31.466 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 89 16 110116 110100 77.3043 62.875 0.0156366 0.0129312 2024-03-11T21:04:31.561 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:04:31.590 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:04:31.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:31.594 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:04:31.695 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:31.704 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:31 smithi146 bash[23809]: audit 2024-03-11T21:04:29.605169+0000 mgr.y (mgr.24380) 508 : audit 0 from='client.15150 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:04:31.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:31 smithi146 bash[23809]: cluster 2024-03-11T21:04:29.634081+0000 osd.6 (osd.6) 202 : cluster 0 9.b scrub starts 2024-03-11T21:04:31.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:31 smithi146 bash[23809]: cluster 2024-03-11T21:04:30.177664+0000 mgr.y (mgr.24380) 509 : cluster 0 pgmap v568: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 45 MiB/s wr, 721 op/s 2024-03-11T21:04:31.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:31 smithi146 bash[23809]: cluster 2024-03-11T21:04:30.227877+0000 osd.3 (osd.3) 272 : cluster 0 6.b scrub starts 2024-03-11T21:04:31.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:31 smithi146 bash[23809]: cluster 2024-03-11T21:04:30.333277+0000 osd.3 (osd.3) 273 : cluster 0 6.b scrub ok 2024-03-11T21:04:31.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:31 smithi146 bash[23809]: cluster 2024-03-11T21:04:31.323804+0000 osd.5 (osd.5) 247 : cluster 0 purged_snaps scrub starts 2024-03-11T21:04:31.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:31 smithi146 bash[23809]: cluster 2024-03-11T21:04:31.323849+0000 osd.5 (osd.5) 248 : cluster 0 purged_snaps scrub ok 2024-03-11T21:04:31.724 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:31.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:31.729 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:04:31.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:31 smithi023 bash[21372]: audit 2024-03-11T21:04:29.605169+0000 mgr.y (mgr.24380) 508 : audit 0 from='client.15150 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:04:31.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:31 smithi023 bash[21372]: cluster 2024-03-11T21:04:29.634081+0000 osd.6 (osd.6) 202 : cluster 0 9.b scrub starts 2024-03-11T21:04:31.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:31 smithi023 bash[21372]: cluster 2024-03-11T21:04:30.177664+0000 mgr.y (mgr.24380) 509 : cluster 0 pgmap v568: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 45 MiB/s wr, 721 op/s 2024-03-11T21:04:31.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:31 smithi023 bash[21372]: cluster 2024-03-11T21:04:30.227877+0000 osd.3 (osd.3) 272 : cluster 0 6.b scrub starts 2024-03-11T21:04:31.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:31 smithi023 bash[21372]: cluster 2024-03-11T21:04:30.333277+0000 osd.3 (osd.3) 273 : cluster 0 6.b scrub ok 2024-03-11T21:04:31.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:31 smithi023 bash[21372]: cluster 2024-03-11T21:04:31.323804+0000 osd.5 (osd.5) 247 : cluster 0 purged_snaps scrub starts 2024-03-11T21:04:31.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:31 smithi023 bash[21372]: cluster 2024-03-11T21:04:31.323849+0000 osd.5 (osd.5) 248 : cluster 0 purged_snaps scrub ok 2024-03-11T21:04:31.784 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:31 smithi023 bash[29146]: audit 2024-03-11T21:04:29.605169+0000 mgr.y (mgr.24380) 508 : audit 0 from='client.15150 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:04:31.784 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:31 smithi023 bash[29146]: cluster 2024-03-11T21:04:29.634081+0000 osd.6 (osd.6) 202 : cluster 0 9.b scrub starts 2024-03-11T21:04:31.784 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:31 smithi023 bash[29146]: cluster 2024-03-11T21:04:30.177664+0000 mgr.y (mgr.24380) 509 : cluster 0 pgmap v568: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 45 MiB/s wr, 721 op/s 2024-03-11T21:04:31.784 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:31 smithi023 bash[29146]: cluster 2024-03-11T21:04:30.227877+0000 osd.3 (osd.3) 272 : cluster 0 6.b scrub starts 2024-03-11T21:04:31.784 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:31 smithi023 bash[29146]: cluster 2024-03-11T21:04:30.333277+0000 osd.3 (osd.3) 273 : cluster 0 6.b scrub ok 2024-03-11T21:04:31.784 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:31 smithi023 bash[29146]: cluster 2024-03-11T21:04:31.323804+0000 osd.5 (osd.5) 247 : cluster 0 purged_snaps scrub starts 2024-03-11T21:04:31.784 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:31 smithi023 bash[29146]: cluster 2024-03-11T21:04:31.323849+0000 osd.5 (osd.5) 248 : cluster 0 purged_snaps scrub ok 2024-03-11T21:04:31.830 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:04:31.860 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:04:31.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:31.867 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:04:31.967 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:04:31.998 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:04:32.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:32.001 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:04:32.102 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:04:32.121 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:04:32.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:32.122 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:04:32.223 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:32.251 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:32.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:32.255 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:04:32.356 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:04:32.388 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:04:32.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:32.390 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:04:32.466 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:Total time run: 90.0055 2024-03-11T21:04:32.466 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:Total writes made: 111143 2024-03-11T21:04:32.466 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:Write size: 65536 2024-03-11T21:04:32.466 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:Object size: 65536 2024-03-11T21:04:32.467 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:Bandwidth (MB/sec): 77.1779 2024-03-11T21:04:32.467 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:Stddev Bandwidth: 30.4271 2024-03-11T21:04:32.467 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:Max bandwidth (MB/sec): 179.25 2024-03-11T21:04:32.467 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:Min bandwidth (MB/sec): 32.8125 2024-03-11T21:04:32.467 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:Average IOPS: 1234 2024-03-11T21:04:32.467 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:Stddev IOPS: 486.834 2024-03-11T21:04:32.467 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:Max IOPS: 2868 2024-03-11T21:04:32.467 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:Min IOPS: 525 2024-03-11T21:04:32.467 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:Average Latency(s): 0.0129533 2024-03-11T21:04:32.468 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:Stddev Latency(s): 0.0206588 2024-03-11T21:04:32.468 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:Max latency(s): 0.549376 2024-03-11T21:04:32.468 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:Min latency(s): 0.0011821 2024-03-11T21:04:32.471 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:Cleaning up (deleting benchmark objects) 2024-03-11T21:04:32.490 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:04:32.519 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:04:32.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:32.521 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:04:32.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:32 smithi023 bash[21372]: cluster 2024-03-11T21:04:31.271026+0000 osd.3 (osd.3) 274 : cluster 0 2.19 scrub starts 2024-03-11T21:04:32.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:32 smithi023 bash[21372]: cluster 2024-03-11T21:04:31.272750+0000 osd.3 (osd.3) 275 : cluster 0 2.19 scrub ok 2024-03-11T21:04:32.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:32 smithi023 bash[21372]: cluster 2024-03-11T21:04:31.403279+0000 osd.6 (osd.6) 203 : cluster 0 9.b scrub ok 2024-03-11T21:04:32.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:32 smithi023 bash[21372]: cluster 2024-03-11T21:04:31.470997+0000 mon.a (mon.0) 1513 : cluster 0 osdmap e279: 8 total, 8 up, 8 in 2024-03-11T21:04:32.533 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:32 smithi023 bash[29146]: cluster 2024-03-11T21:04:31.271026+0000 osd.3 (osd.3) 274 : cluster 0 2.19 scrub starts 2024-03-11T21:04:32.533 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:32 smithi023 bash[29146]: cluster 2024-03-11T21:04:31.272750+0000 osd.3 (osd.3) 275 : cluster 0 2.19 scrub ok 2024-03-11T21:04:32.533 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:32 smithi023 bash[29146]: cluster 2024-03-11T21:04:31.403279+0000 osd.6 (osd.6) 203 : cluster 0 9.b scrub ok 2024-03-11T21:04:32.534 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:32 smithi023 bash[29146]: cluster 2024-03-11T21:04:31.470997+0000 mon.a (mon.0) 1513 : cluster 0 osdmap e279: 8 total, 8 up, 8 in 2024-03-11T21:04:32.621 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:04:32.647 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:04:32.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:32.651 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:04:32.752 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:04:32.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:32 smithi146 bash[23809]: cluster 2024-03-11T21:04:31.271026+0000 osd.3 (osd.3) 274 : cluster 0 2.19 scrub starts 2024-03-11T21:04:32.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:32 smithi146 bash[23809]: cluster 2024-03-11T21:04:31.272750+0000 osd.3 (osd.3) 275 : cluster 0 2.19 scrub ok 2024-03-11T21:04:32.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:32 smithi146 bash[23809]: cluster 2024-03-11T21:04:31.403279+0000 osd.6 (osd.6) 203 : cluster 0 9.b scrub ok 2024-03-11T21:04:32.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:32 smithi146 bash[23809]: cluster 2024-03-11T21:04:31.470997+0000 mon.a (mon.0) 1513 : cluster 0 osdmap e279: 8 total, 8 up, 8 in 2024-03-11T21:04:32.785 INFO:teuthology.orchestra.run.smithi023.stdout:{"pg_ready":true,"pg_map":{"version":567,"stamp":"2024-03-11T21:04:29.459615+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6859921832,"num_objects":104863,"num_object_clones":0,"num_object_copies":314589,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104863,"num_whiteouts":0,"num_read":2064,"num_read_kb":1910,"num_write":105566,"num_write_kb":6701021,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":197,"num_bytes_recovered":1775932,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81909,"ondisk_log_size":81909,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":22165196,"kb_used_data":20033068,"kb_used_omap":21,"kb_used_meta":2131242,"kb_avail":727796020,"statfs":{"total":767960285184,"available":745263124480,"internally_reserved":0,"allocated":20513861632,"data_stored":20510512154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22089,"internal_metadata":2182392247},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,4,5,6,0,1,4,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":516358144,"num_objects":7879,"num_object_clones":0,"num_object_copies":23637,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7879,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":7879,"num_write_kb":504256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.285767"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":443,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.439143+0000","last_change":"2024-03-11T21:00:08.811947+0000","last_active":"2024-03-11T21:04:24.439143+0000","last_peered":"2024-03-11T21:04:24.439143+0000","last_clean":"2024-03-11T21:04:24.439143+0000","last_became_active":"2024-03-11T20:50:34.338390+0000","last_became_peered":"2024-03-11T20:50:34.338390+0000","last_unstale":"2024-03-11T21:04:24.439143+0000","last_undegraded":"2024-03-11T21:04:24.439143+0000","last_fullsized":"2024-03-11T21:04:24.439143+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:00:08.811756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:00:08.811756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:26.107243+0000","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":447,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.439146+0000","last_change":"2024-03-11T21:00:02.764408+0000","last_active":"2024-03-11T21:04:24.439146+0000","last_peered":"2024-03-11T21:04:24.439146+0000","last_clean":"2024-03-11T21:04:24.439146+0000","last_became_active":"2024-03-11T20:50:32.210194+0000","last_became_peered":"2024-03-11T20:50:32.210194+0000","last_unstale":"2024-03-11T21:04:24.439146+0000","last_undegraded":"2024-03-11T21:04:24.439146+0000","last_fullsized":"2024-03-11T21:04:24.439146+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:00:02.764190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:00:02.764190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:26.107391+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":272,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.433339+0000","last_change":"2024-03-11T21:03:38.968156+0000","last_active":"2024-03-11T21:04:24.433339+0000","last_peered":"2024-03-11T21:04:24.433339+0000","last_clean":"2024-03-11T21:04:24.433339+0000","last_became_active":"2024-03-11T21:00:27.963600+0000","last_became_peered":"2024-03-11T21:00:27.963600+0000","last_unstale":"2024-03-11T21:04:24.433339+0000","last_undegraded":"2024-03-11T21:04:24.433339+0000","last_fullsized":"2024-03-11T21:04:24.433339+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:38.968078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:53:29.964907+0000","last_clean_scrub_stamp":"2024-03-11T21:03:38.968078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:05:07.073103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1e","version":"207'20","reported_seq":482,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.441095+0000","last_change":"2024-03-11T21:00:09.750591+0000","last_active":"2024-03-11T21:04:24.441095+0000","last_peered":"2024-03-11T21:04:24.441095+0000","last_clean":"2024-03-11T21:04:24.441095+0000","last_became_active":"2024-03-11T20:50:30.101721+0000","last_became_peered":"2024-03-11T20:50:30.101721+0000","last_unstale":"2024-03-11T21:04:24.441095+0000","last_undegraded":"2024-03-11T21:04:24.441095+0000","last_fullsized":"2024-03-11T21:04:24.441095+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'10","last_scrub_stamp":"2024-03-11T21:00:09.750456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:00:09.750456+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:26.107431+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":411,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.427697+0000","last_change":"2024-03-11T21:02:30.521163+0000","last_active":"2024-03-11T21:04:24.427697+0000","last_peered":"2024-03-11T21:04:24.427697+0000","last_clean":"2024-03-11T21:04:24.427697+0000","last_became_active":"2024-03-11T21:00:48.467854+0000","last_became_peered":"2024-03-11T21:00:48.467854+0000","last_unstale":"2024-03-11T21:04:24.427697+0000","last_undegraded":"2024-03-11T21:04:24.427697+0000","last_fullsized":"2024-03-11T21:04:24.427697+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:02:30.520851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:55.545758+0000","last_clean_scrub_stamp":"2024-03-11T21:02:30.520851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":424,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.427042+0000","last_change":"2024-03-11T21:03:31.347920+0000","last_active":"2024-03-11T21:04:24.427042+0000","last_peered":"2024-03-11T21:04:24.427042+0000","last_clean":"2024-03-11T21:04:24.427042+0000","last_became_active":"2024-03-11T21:00:48.459957+0000","last_became_peered":"2024-03-11T21:00:48.459957+0000","last_unstale":"2024-03-11T21:04:24.427042+0000","last_undegraded":"2024-03-11T21:04:24.427042+0000","last_fullsized":"2024-03-11T21:04:24.427042+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:31.347841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:54:13.660325+0000","last_clean_scrub_stamp":"2024-03-11T21:03:31.347841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:35.281947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":456,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.440876+0000","last_change":"2024-03-11T21:03:40.675667+0000","last_active":"2024-03-11T21:04:24.440876+0000","last_peered":"2024-03-11T21:04:24.440876+0000","last_clean":"2024-03-11T21:04:24.440876+0000","last_became_active":"2024-03-11T20:50:28.058274+0000","last_became_peered":"2024-03-11T20:50:28.058274+0000","last_unstale":"2024-03-11T21:04:24.440876+0000","last_undegraded":"2024-03-11T21:04:24.440876+0000","last_fullsized":"2024-03-11T21:04:24.440876+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:40.675526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:03:40.675526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:51.747922+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"207'19","reported_seq":496,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.447307+0000","last_change":"2024-03-11T21:03:12.772305+0000","last_active":"2024-03-11T21:04:24.447307+0000","last_peered":"2024-03-11T21:04:24.447307+0000","last_clean":"2024-03-11T21:04:24.447307+0000","last_became_active":"2024-03-11T20:50:30.092638+0000","last_became_peered":"2024-03-11T20:50:30.092638+0000","last_unstale":"2024-03-11T21:04:24.447307+0000","last_undegraded":"2024-03-11T21:04:24.447307+0000","last_fullsized":"2024-03-11T21:04:24.447307+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'19","last_scrub_stamp":"2024-03-11T21:03:12.772194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:03:12.772194+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:37.917374+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":456,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.439508+0000","last_change":"2024-03-11T21:02:23.769016+0000","last_active":"2024-03-11T21:04:24.439508+0000","last_peered":"2024-03-11T21:04:24.439508+0000","last_clean":"2024-03-11T21:04:24.439508+0000","last_became_active":"2024-03-11T21:00:48.460694+0000","last_became_peered":"2024-03-11T21:00:48.460694+0000","last_unstale":"2024-03-11T21:04:24.439508+0000","last_undegraded":"2024-03-11T21:04:24.439508+0000","last_fullsized":"2024-03-11T21:04:24.439508+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:02:23.768654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:53:07.792697+0000","last_clean_scrub_stamp":"2024-03-11T21:02:23.768654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:26.440067+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":415,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.427531+0000","last_change":"2024-03-11T21:03:36.230810+0000","last_active":"2024-03-11T21:04:24.427531+0000","last_peered":"2024-03-11T21:04:24.427531+0000","last_clean":"2024-03-11T21:04:24.427531+0000","last_became_active":"2024-03-11T21:00:48.463839+0000","last_became_peered":"2024-03-11T21:00:48.463839+0000","last_unstale":"2024-03-11T21:04:24.427531+0000","last_undegraded":"2024-03-11T21:04:24.427531+0000","last_fullsized":"2024-03-11T21:04:24.427531+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:36.230647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:03:36.230647+0000","last_clean_scrub_stamp":"2024-03-11T21:03:36.230647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:05:04.850393+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":470,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.437994+0000","last_change":"2024-03-11T21:03:36.980913+0000","last_active":"2024-03-11T21:04:24.437994+0000","last_peered":"2024-03-11T21:04:24.437994+0000","last_clean":"2024-03-11T21:04:24.437994+0000","last_became_active":"2024-03-11T20:50:28.079225+0000","last_became_peered":"2024-03-11T20:50:28.079225+0000","last_unstale":"2024-03-11T21:04:24.437994+0000","last_undegraded":"2024-03-11T21:04:24.437994+0000","last_fullsized":"2024-03-11T21:04:24.437994+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:36.980825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:03:36.980825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:48.975950+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"207'27","reported_seq":559,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.440195+0000","last_change":"2024-03-11T21:02:24.705731+0000","last_active":"2024-03-11T21:04:24.440195+0000","last_peered":"2024-03-11T21:04:24.440195+0000","last_clean":"2024-03-11T21:04:24.440195+0000","last_became_active":"2024-03-11T21:00:48.462943+0000","last_became_peered":"2024-03-11T21:00:48.462943+0000","last_unstale":"2024-03-11T21:04:24.440195+0000","last_undegraded":"2024-03-11T21:04:24.440195+0000","last_fullsized":"2024-03-11T21:04:24.440195+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'27","last_scrub_stamp":"2024-03-11T21:02:24.705531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:02:24.705531+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:26.440113+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":162,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.445233+0000","last_change":"2024-03-11T21:03:20.330025+0000","last_active":"2024-03-11T21:04:24.445233+0000","last_peered":"2024-03-11T21:04:24.445233+0000","last_clean":"2024-03-11T21:04:24.445233+0000","last_became_active":"2024-03-11T21:00:48.456017+0000","last_became_peered":"2024-03-11T21:00:48.456017+0000","last_unstale":"2024-03-11T21:04:24.445233+0000","last_undegraded":"2024-03-11T21:04:24.445233+0000","last_fullsized":"2024-03-11T21:04:24.445233+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:20.329913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:03:20.329913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:27.990899+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":411,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.427377+0000","last_change":"2024-03-11T21:03:16.356593+0000","last_active":"2024-03-11T21:04:24.427377+0000","last_peered":"2024-03-11T21:04:24.427377+0000","last_clean":"2024-03-11T21:04:24.427377+0000","last_became_active":"2024-03-11T21:00:27.962251+0000","last_became_peered":"2024-03-11T21:00:27.962251+0000","last_unstale":"2024-03-11T21:04:24.427377+0000","last_undegraded":"2024-03-11T21:04:24.427377+0000","last_fullsized":"2024-03-11T21:04:24.427377+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:16.356482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:53:50.599029+0000","last_clean_scrub_stamp":"2024-03-11T21:03:16.356482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":470,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.433738+0000","last_change":"2024-03-11T21:02:31.898315+0000","last_active":"2024-03-11T21:04:24.433738+0000","last_peered":"2024-03-11T21:04:24.433738+0000","last_clean":"2024-03-11T21:04:24.433738+0000","last_became_active":"2024-03-11T20:50:28.068978+0000","last_became_peered":"2024-03-11T20:50:28.068978+0000","last_unstale":"2024-03-11T21:04:24.433738+0000","last_undegraded":"2024-03-11T21:04:24.433738+0000","last_fullsized":"2024-03-11T21:04:24.433738+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:02:31.898099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:54:07.155130+0000","last_clean_scrub_stamp":"2024-03-11T21:02:31.898099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:25.051348+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"207'23","reported_seq":497,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.443856+0000","last_change":"2024-03-11T21:02:32.859754+0000","last_active":"2024-03-11T21:04:24.443856+0000","last_peered":"2024-03-11T21:04:24.443856+0000","last_clean":"2024-03-11T21:04:24.443856+0000","last_became_active":"2024-03-11T21:00:27.952093+0000","last_became_peered":"2024-03-11T21:00:27.952093+0000","last_unstale":"2024-03-11T21:04:24.443856+0000","last_undegraded":"2024-03-11T21:04:24.443856+0000","last_fullsized":"2024-03-11T21:04:24.443856+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'23","last_scrub_stamp":"2024-03-11T21:02:32.859604+0000","last_deep_scrub":"110'17","last_deep_scrub_stamp":"2024-03-11T20:53:04.802488+0000","last_clean_scrub_stamp":"2024-03-11T21:02:32.859604+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:26.440151+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":444,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.441182+0000","last_change":"2024-03-11T21:00:13.778710+0000","last_active":"2024-03-11T21:04:24.441182+0000","last_peered":"2024-03-11T21:04:24.441182+0000","last_clean":"2024-03-11T21:04:24.441182+0000","last_became_active":"2024-03-11T20:50:34.334558+0000","last_became_peered":"2024-03-11T20:50:34.334558+0000","last_unstale":"2024-03-11T21:04:24.441182+0000","last_undegraded":"2024-03-11T21:04:24.441182+0000","last_fullsized":"2024-03-11T21:04:24.441182+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:00:13.778532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:55:59.892394+0000","last_clean_scrub_stamp":"2024-03-11T21:00:13.778532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:26.107470+0000","scrub_duration":87,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":295,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.443549+0000","last_change":"2024-03-11T21:03:48.239121+0000","last_active":"2024-03-11T21:04:24.443549+0000","last_peered":"2024-03-11T21:04:24.443549+0000","last_clean":"2024-03-11T21:04:24.443549+0000","last_became_active":"2024-03-11T21:00:48.461013+0000","last_became_peered":"2024-03-11T21:00:48.461013+0000","last_unstale":"2024-03-11T21:04:24.443549+0000","last_undegraded":"2024-03-11T21:04:24.443549+0000","last_fullsized":"2024-03-11T21:04:24.443549+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:48.239009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:03:48.239009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:51.626936+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"77'1","reported_seq":463,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.439304+0000","last_change":"2024-03-11T21:00:10.713115+0000","last_active":"2024-03-11T21:04:24.439304+0000","last_peered":"2024-03-11T21:04:24.439304+0000","last_clean":"2024-03-11T21:04:24.439304+0000","last_became_active":"2024-03-11T20:50:28.068840+0000","last_became_peered":"2024-03-11T20:50:28.068840+0000","last_unstale":"2024-03-11T21:04:24.439304+0000","last_undegraded":"2024-03-11T21:04:24.439304+0000","last_fullsized":"2024-03-11T21:04:24.439304+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-03-11T21:00:10.712812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:00:10.712812+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:26.107133+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"207'19","reported_seq":485,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.427577+0000","last_change":"2024-03-11T21:03:39.240266+0000","last_active":"2024-03-11T21:04:24.427577+0000","last_peered":"2024-03-11T21:04:24.427577+0000","last_clean":"2024-03-11T21:04:24.427577+0000","last_became_active":"2024-03-11T21:00:48.468434+0000","last_became_peered":"2024-03-11T21:00:48.468434+0000","last_unstale":"2024-03-11T21:04:24.427577+0000","last_undegraded":"2024-03-11T21:04:24.427577+0000","last_fullsized":"2024-03-11T21:04:24.427577+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'19","last_scrub_stamp":"2024-03-11T21:03:39.240089+0000","last_deep_scrub":"207'19","last_deep_scrub_stamp":"2024-03-11T21:03:39.240089+0000","last_clean_scrub_stamp":"2024-03-11T21:03:39.240089+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:05:07.484049+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":406,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.428322+0000","last_change":"2024-03-11T21:02:33.399743+0000","last_active":"2024-03-11T21:04:24.428322+0000","last_peered":"2024-03-11T21:04:24.428322+0000","last_clean":"2024-03-11T21:04:24.428322+0000","last_became_active":"2024-03-11T21:00:27.955250+0000","last_became_peered":"2024-03-11T21:00:27.955250+0000","last_unstale":"2024-03-11T21:04:24.428322+0000","last_undegraded":"2024-03-11T21:04:24.428322+0000","last_fullsized":"2024-03-11T21:04:24.428322+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:02:33.399510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:33.399510+0000","last_clean_scrub_stamp":"2024-03-11T21:02:33.399510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":495,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.444252+0000","last_change":"2024-03-11T21:03:29.516265+0000","last_active":"2024-03-11T21:04:24.444252+0000","last_peered":"2024-03-11T21:04:24.444252+0000","last_clean":"2024-03-11T21:04:24.444252+0000","last_became_active":"2024-03-11T21:00:48.459135+0000","last_became_peered":"2024-03-11T21:00:48.459135+0000","last_unstale":"2024-03-11T21:04:24.444252+0000","last_undegraded":"2024-03-11T21:04:24.444252+0000","last_fullsized":"2024-03-11T21:04:24.444252+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:29.516124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:24.666325+0000","last_clean_scrub_stamp":"2024-03-11T21:03:29.516124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:38.006278+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":470,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.443573+0000","last_change":"2024-03-11T21:03:11.542784+0000","last_active":"2024-03-11T21:04:24.443573+0000","last_peered":"2024-03-11T21:04:24.443573+0000","last_clean":"2024-03-11T21:04:24.443573+0000","last_became_active":"2024-03-11T21:00:27.956935+0000","last_became_peered":"2024-03-11T21:00:27.956935+0000","last_unstale":"2024-03-11T21:04:24.443573+0000","last_undegraded":"2024-03-11T21:04:24.443573+0000","last_fullsized":"2024-03-11T21:04:24.443573+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:11.542663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:03:11.542663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:27.631210+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"207'7","reported_seq":478,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.446683+0000","last_change":"2024-03-11T21:02:32.818371+0000","last_active":"2024-03-11T21:04:24.446683+0000","last_peered":"2024-03-11T21:04:24.446683+0000","last_clean":"2024-03-11T21:04:24.446683+0000","last_became_active":"2024-03-11T21:00:27.955296+0000","last_became_peered":"2024-03-11T21:00:27.955296+0000","last_unstale":"2024-03-11T21:04:24.446683+0000","last_undegraded":"2024-03-11T21:04:24.446683+0000","last_fullsized":"2024-03-11T21:04:24.446683+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'7","last_scrub_stamp":"2024-03-11T21:02:32.818096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:02:32.818096+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":170,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.453762+0000","last_change":"2024-03-11T21:03:42.593360+0000","last_active":"2024-03-11T21:04:24.453762+0000","last_peered":"2024-03-11T21:04:24.453762+0000","last_clean":"2024-03-11T21:04:24.453762+0000","last_became_active":"2024-03-11T21:00:48.464694+0000","last_became_peered":"2024-03-11T21:00:48.464694+0000","last_unstale":"2024-03-11T21:04:24.453762+0000","last_undegraded":"2024-03-11T21:04:24.453762+0000","last_fullsized":"2024-03-11T21:04:24.453762+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:42.593243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:03:42.593243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:47.445913+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1f","version":"212'16","reported_seq":664,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.443992+0000","last_change":"2024-03-11T21:03:09.534518+0000","last_active":"2024-03-11T21:04:24.443992+0000","last_peered":"2024-03-11T21:04:24.443992+0000","last_clean":"2024-03-11T21:04:24.443992+0000","last_became_active":"2024-03-11T21:00:48.464350+0000","last_became_peered":"2024-03-11T21:00:48.464350+0000","last_unstale":"2024-03-11T21:04:24.443992+0000","last_undegraded":"2024-03-11T21:04:24.443992+0000","last_fullsized":"2024-03-11T21:04:24.443992+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"212'16","last_scrub_stamp":"2024-03-11T21:03:09.534368+0000","last_deep_scrub":"114'12","last_deep_scrub_stamp":"2024-03-11T20:53:30.905791+0000","last_clean_scrub_stamp":"2024-03-11T21:03:09.534368+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:38.232644+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"77'1","reported_seq":448,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.439396+0000","last_change":"2024-03-11T20:59:06.457866+0000","last_active":"2024-03-11T21:04:24.439396+0000","last_peered":"2024-03-11T21:04:24.439396+0000","last_clean":"2024-03-11T21:04:24.439396+0000","last_became_active":"2024-03-11T20:50:28.068827+0000","last_became_peered":"2024-03-11T20:50:28.068827+0000","last_unstale":"2024-03-11T21:04:24.439396+0000","last_undegraded":"2024-03-11T21:04:24.439396+0000","last_fullsized":"2024-03-11T21:04:24.439396+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-03-11T20:59:06.457561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T20:59:06.457561+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:26.107038+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"207'15","reported_seq":518,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.439196+0000","last_change":"2024-03-11T21:02:30.560424+0000","last_active":"2024-03-11T21:04:24.439196+0000","last_peered":"2024-03-11T21:04:24.439196+0000","last_clean":"2024-03-11T21:04:24.439196+0000","last_became_active":"2024-03-11T21:00:48.467513+0000","last_became_peered":"2024-03-11T21:00:48.467513+0000","last_unstale":"2024-03-11T21:04:24.439196+0000","last_undegraded":"2024-03-11T21:04:24.439196+0000","last_fullsized":"2024-03-11T21:04:24.439196+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'15","last_scrub_stamp":"2024-03-11T21:02:30.560293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:02:30.560293+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:26.107558+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"105'2","reported_seq":461,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.433973+0000","last_change":"2024-03-11T21:03:41.972426+0000","last_active":"2024-03-11T21:04:24.433973+0000","last_peered":"2024-03-11T21:04:24.433973+0000","last_clean":"2024-03-11T21:04:24.433973+0000","last_became_active":"2024-03-11T20:50:34.327356+0000","last_became_peered":"2024-03-11T20:50:34.327356+0000","last_unstale":"2024-03-11T21:04:24.433973+0000","last_undegraded":"2024-03-11T21:04:24.433973+0000","last_fullsized":"2024-03-11T21:04:24.433973+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'2","last_scrub_stamp":"2024-03-11T21:03:41.971918+0000","last_deep_scrub":"105'2","last_deep_scrub_stamp":"2024-03-11T21:03:41.971918+0000","last_clean_scrub_stamp":"2024-03-11T21:03:41.971918+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:42.904105+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":271,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.434182+0000","last_change":"2024-03-11T21:02:14.767550+0000","last_active":"2024-03-11T21:04:24.434182+0000","last_peered":"2024-03-11T21:04:24.434182+0000","last_clean":"2024-03-11T21:04:24.434182+0000","last_became_active":"2024-03-11T20:56:59.804264+0000","last_became_peered":"2024-03-11T20:56:59.804264+0000","last_unstale":"2024-03-11T21:04:24.434182+0000","last_undegraded":"2024-03-11T21:04:24.434182+0000","last_fullsized":"2024-03-11T21:04:24.434182+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:02:14.767248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:59:14.605681+0000","last_clean_scrub_stamp":"2024-03-11T21:02:14.767248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:25.051235+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":464,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.439422+0000","last_change":"2024-03-11T21:03:09.746912+0000","last_active":"2024-03-11T21:04:24.439422+0000","last_peered":"2024-03-11T21:04:24.439422+0000","last_clean":"2024-03-11T21:04:24.439422+0000","last_became_active":"2024-03-11T21:00:27.947614+0000","last_became_peered":"2024-03-11T21:00:27.947614+0000","last_unstale":"2024-03-11T21:04:24.439422+0000","last_undegraded":"2024-03-11T21:04:24.439422+0000","last_fullsized":"2024-03-11T21:04:24.439422+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:09.746789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:52:01.880052+0000","last_clean_scrub_stamp":"2024-03-11T21:03:09.746789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:28.738045+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"207'27","reported_seq":200,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.444076+0000","last_change":"2024-03-11T21:03:27.406786+0000","last_active":"2024-03-11T21:04:24.444076+0000","last_peered":"2024-03-11T21:04:24.444076+0000","last_clean":"2024-03-11T21:04:24.444076+0000","last_became_active":"2024-03-11T21:00:48.457893+0000","last_became_peered":"2024-03-11T21:00:48.457893+0000","last_unstale":"2024-03-11T21:04:24.444076+0000","last_undegraded":"2024-03-11T21:04:24.444076+0000","last_fullsized":"2024-03-11T21:04:24.444076+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'27","last_scrub_stamp":"2024-03-11T21:03:27.406573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:03:27.406573+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:27.409569+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":443,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.439481+0000","last_change":"2024-03-11T21:00:12.754952+0000","last_active":"2024-03-11T21:04:24.439481+0000","last_peered":"2024-03-11T21:04:24.439481+0000","last_clean":"2024-03-11T21:04:24.439481+0000","last_became_active":"2024-03-11T20:50:34.337878+0000","last_became_peered":"2024-03-11T20:50:34.337878+0000","last_unstale":"2024-03-11T21:04:24.439481+0000","last_undegraded":"2024-03-11T21:04:24.439481+0000","last_fullsized":"2024-03-11T21:04:24.439481+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:00:12.754627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:52:11.462480+0000","last_clean_scrub_stamp":"2024-03-11T21:00:12.754627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:26.107208+0000","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":447,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.440021+0000","last_change":"2024-03-11T20:59:13.516522+0000","last_active":"2024-03-11T21:04:24.440021+0000","last_peered":"2024-03-11T21:04:24.440021+0000","last_clean":"2024-03-11T21:04:24.440021+0000","last_became_active":"2024-03-11T20:50:32.646323+0000","last_became_peered":"2024-03-11T20:50:32.646323+0000","last_unstale":"2024-03-11T21:04:24.440021+0000","last_undegraded":"2024-03-11T21:04:24.440021+0000","last_fullsized":"2024-03-11T21:04:24.440021+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:59:13.516326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:54:06.516412+0000","last_clean_scrub_stamp":"2024-03-11T20:59:13.516326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:26.107092+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":477,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.444215+0000","last_change":"2024-03-11T21:03:23.586644+0000","last_active":"2024-03-11T21:04:24.444215+0000","last_peered":"2024-03-11T21:04:24.444215+0000","last_clean":"2024-03-11T21:04:24.444215+0000","last_became_active":"2024-03-11T20:50:28.086513+0000","last_became_peered":"2024-03-11T20:50:28.086513+0000","last_unstale":"2024-03-11T21:04:24.444215+0000","last_undegraded":"2024-03-11T21:04:24.444215+0000","last_fullsized":"2024-03-11T21:04:24.444215+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:23.586540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:03:23.586540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:44.222880+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"207'19","reported_seq":522,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.440691+0000","last_change":"2024-03-11T21:02:21.669112+0000","last_active":"2024-03-11T21:04:24.440691+0000","last_peered":"2024-03-11T21:04:24.440691+0000","last_clean":"2024-03-11T21:04:24.440691+0000","last_became_active":"2024-03-11T21:00:48.464707+0000","last_became_peered":"2024-03-11T21:00:48.464707+0000","last_unstale":"2024-03-11T21:04:24.440691+0000","last_undegraded":"2024-03-11T21:04:24.440691+0000","last_fullsized":"2024-03-11T21:04:24.440691+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'19","last_scrub_stamp":"2024-03-11T21:02:21.668821+0000","last_deep_scrub":"145'13","last_deep_scrub_stamp":"2024-03-11T20:55:53.560287+0000","last_clean_scrub_stamp":"2024-03-11T21:02:21.668821+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:26.440045+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":459,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.434454+0000","last_change":"2024-03-11T21:03:24.925576+0000","last_active":"2024-03-11T21:04:24.434454+0000","last_peered":"2024-03-11T21:04:24.434454+0000","last_clean":"2024-03-11T21:04:24.434454+0000","last_became_active":"2024-03-11T21:00:27.973405+0000","last_became_peered":"2024-03-11T21:00:27.973405+0000","last_unstale":"2024-03-11T21:04:24.434454+0000","last_undegraded":"2024-03-11T21:04:24.434454+0000","last_fullsized":"2024-03-11T21:04:24.434454+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:24.925436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:03:24.925436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:46.169097+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":460,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.439586+0000","last_change":"2024-03-11T21:02:31.549151+0000","last_active":"2024-03-11T21:04:24.439586+0000","last_peered":"2024-03-11T21:04:24.439586+0000","last_clean":"2024-03-11T21:04:24.439586+0000","last_became_active":"2024-03-11T21:00:48.472812+0000","last_became_peered":"2024-03-11T21:00:48.472812+0000","last_unstale":"2024-03-11T21:04:24.439586+0000","last_undegraded":"2024-03-11T21:04:24.439586+0000","last_fullsized":"2024-03-11T21:04:24.439586+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:02:31.549032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:02:31.549032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:26.107572+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":471,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.440580+0000","last_change":"2024-03-11T21:03:33.835941+0000","last_active":"2024-03-11T21:04:24.440580+0000","last_peered":"2024-03-11T21:04:24.440580+0000","last_clean":"2024-03-11T21:04:24.440580+0000","last_became_active":"2024-03-11T20:50:28.075760+0000","last_became_peered":"2024-03-11T20:50:28.075760+0000","last_unstale":"2024-03-11T21:04:24.440580+0000","last_undegraded":"2024-03-11T21:04:24.440580+0000","last_fullsized":"2024-03-11T21:04:24.440580+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:33.835839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:58:55.977289+0000","last_clean_scrub_stamp":"2024-03-11T21:03:33.835839+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:45.805119+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"207'12","reported_seq":317,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.440436+0000","last_change":"2024-03-11T21:03:21.879821+0000","last_active":"2024-03-11T21:04:24.440436+0000","last_peered":"2024-03-11T21:04:24.440436+0000","last_clean":"2024-03-11T21:04:24.440436+0000","last_became_active":"2024-03-11T20:56:59.794581+0000","last_became_peered":"2024-03-11T20:56:59.794581+0000","last_unstale":"2024-03-11T21:04:24.440436+0000","last_undegraded":"2024-03-11T21:04:24.440436+0000","last_fullsized":"2024-03-11T21:04:24.440436+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'12","last_scrub_stamp":"2024-03-11T21:03:21.879683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:03:21.879683+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:50.694254+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"186'2","reported_seq":445,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.438920+0000","last_change":"2024-03-11T21:00:17.739831+0000","last_active":"2024-03-11T21:04:24.438920+0000","last_peered":"2024-03-11T21:04:24.438920+0000","last_clean":"2024-03-11T21:04:24.438920+0000","last_became_active":"2024-03-11T20:50:34.339326+0000","last_became_peered":"2024-03-11T20:50:34.339326+0000","last_unstale":"2024-03-11T21:04:24.438920+0000","last_undegraded":"2024-03-11T21:04:24.438920+0000","last_fullsized":"2024-03-11T21:04:24.438920+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"186'2","last_scrub_stamp":"2024-03-11T21:00:17.739696+0000","last_deep_scrub":"186'2","last_deep_scrub_stamp":"2024-03-11T20:59:04.534850+0000","last_clean_scrub_stamp":"2024-03-11T21:00:17.739696+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:26.107356+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":424,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.427305+0000","last_change":"2024-03-11T21:03:45.273658+0000","last_active":"2024-03-11T21:04:24.427305+0000","last_peered":"2024-03-11T21:04:24.427305+0000","last_clean":"2024-03-11T21:04:24.427305+0000","last_became_active":"2024-03-11T21:00:48.462656+0000","last_became_peered":"2024-03-11T21:00:48.462656+0000","last_unstale":"2024-03-11T21:04:24.427305+0000","last_undegraded":"2024-03-11T21:04:24.427305+0000","last_fullsized":"2024-03-11T21:04:24.427305+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:45.273529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:03:45.273529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:05:08.865958+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":162,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.453595+0000","last_change":"2024-03-11T21:03:23.365771+0000","last_active":"2024-03-11T21:04:24.453595+0000","last_peered":"2024-03-11T21:04:24.453595+0000","last_clean":"2024-03-11T21:04:24.453595+0000","last_became_active":"2024-03-11T21:00:48.455510+0000","last_became_peered":"2024-03-11T21:00:48.455510+0000","last_unstale":"2024-03-11T21:04:24.453595+0000","last_undegraded":"2024-03-11T21:04:24.453595+0000","last_fullsized":"2024-03-11T21:04:24.453595+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:23.365625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:03:23.365625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:43.858756+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"207'25","reported_seq":417,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.427722+0000","last_change":"2024-03-11T21:03:26.409080+0000","last_active":"2024-03-11T21:04:24.427722+0000","last_peered":"2024-03-11T21:04:24.427722+0000","last_clean":"2024-03-11T21:04:24.427722+0000","last_became_active":"2024-03-11T21:00:27.959547+0000","last_became_peered":"2024-03-11T21:00:27.959547+0000","last_unstale":"2024-03-11T21:04:24.427722+0000","last_undegraded":"2024-03-11T21:04:24.427722+0000","last_fullsized":"2024-03-11T21:04:24.427722+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'25","last_scrub_stamp":"2024-03-11T21:03:26.408832+0000","last_deep_scrub":"161'15","last_deep_scrub_stamp":"2024-03-11T20:57:15.125548+0000","last_clean_scrub_stamp":"2024-03-11T21:03:26.408832+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:42.914368+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":469,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.446121+0000","last_change":"2024-03-11T21:03:27.615442+0000","last_active":"2024-03-11T21:04:24.446121+0000","last_peered":"2024-03-11T21:04:24.446121+0000","last_clean":"2024-03-11T21:04:24.446121+0000","last_became_active":"2024-03-11T21:00:48.462105+0000","last_became_peered":"2024-03-11T21:00:48.462105+0000","last_unstale":"2024-03-11T21:04:24.446121+0000","last_undegraded":"2024-03-11T21:04:24.446121+0000","last_fullsized":"2024-03-11T21:04:24.446121+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:27.615353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:03:27.615353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:32.184921+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":170,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.447206+0000","last_change":"2024-03-11T21:03:43.605939+0000","last_active":"2024-03-11T21:04:24.447206+0000","last_peered":"2024-03-11T21:04:24.447206+0000","last_clean":"2024-03-11T21:04:24.447206+0000","last_became_active":"2024-03-11T21:00:48.459043+0000","last_became_peered":"2024-03-11T21:00:48.459043+0000","last_unstale":"2024-03-11T21:04:24.447206+0000","last_undegraded":"2024-03-11T21:04:24.447206+0000","last_fullsized":"2024-03-11T21:04:24.447206+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:43.605796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:23.137832+0000","last_clean_scrub_stamp":"2024-03-11T21:03:43.605796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:05:13.569217+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":477,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.443493+0000","last_change":"2024-03-11T21:03:36.409060+0000","last_active":"2024-03-11T21:04:24.443493+0000","last_peered":"2024-03-11T21:04:24.443493+0000","last_clean":"2024-03-11T21:04:24.443493+0000","last_became_active":"2024-03-11T20:50:28.082737+0000","last_became_peered":"2024-03-11T20:50:28.082737+0000","last_unstale":"2024-03-11T21:04:24.443493+0000","last_undegraded":"2024-03-11T21:04:24.443493+0000","last_fullsized":"2024-03-11T21:04:24.443493+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:36.408942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:28.404997+0000","last_clean_scrub_stamp":"2024-03-11T21:03:36.408942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:57.355001+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"207'15","reported_seq":492,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.438083+0000","last_change":"2024-03-11T21:03:47.989635+0000","last_active":"2024-03-11T21:04:24.438083+0000","last_peered":"2024-03-11T21:04:24.438083+0000","last_clean":"2024-03-11T21:04:24.438083+0000","last_became_active":"2024-03-11T21:00:27.967624+0000","last_became_peered":"2024-03-11T21:00:27.967624+0000","last_unstale":"2024-03-11T21:04:24.438083+0000","last_undegraded":"2024-03-11T21:04:24.438083+0000","last_fullsized":"2024-03-11T21:04:24.438083+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'15","last_scrub_stamp":"2024-03-11T21:03:47.989410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:03:47.989410+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:58.974121+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":374,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.428374+0000","last_change":"2024-03-11T21:03:44.322295+0000","last_active":"2024-03-11T21:04:24.428374+0000","last_peered":"2024-03-11T21:04:24.428374+0000","last_clean":"2024-03-11T21:04:24.428374+0000","last_became_active":"2024-03-11T21:00:27.958384+0000","last_became_peered":"2024-03-11T21:00:27.958384+0000","last_unstale":"2024-03-11T21:04:24.428374+0000","last_undegraded":"2024-03-11T21:04:24.428374+0000","last_fullsized":"2024-03-11T21:04:24.428374+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:44.322168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:57:16.032302+0000","last_clean_scrub_stamp":"2024-03-11T21:03:44.322168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:05:12.080668+0000","scrub_duration":59,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.15","version":"112'12","reported_seq":634,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.444013+0000","last_change":"2024-03-11T21:03:43.933651+0000","last_active":"2024-03-11T21:04:24.444013+0000","last_peered":"2024-03-11T21:04:24.444013+0000","last_clean":"2024-03-11T21:04:24.444013+0000","last_became_active":"2024-03-11T20:50:32.650064+0000","last_became_peered":"2024-03-11T20:50:32.650064+0000","last_unstale":"2024-03-11T21:04:24.444013+0000","last_undegraded":"2024-03-11T21:04:24.444013+0000","last_fullsized":"2024-03-11T21:04:24.444013+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'12","last_scrub_stamp":"2024-03-11T21:03:43.933551+0000","last_deep_scrub":"112'12","last_deep_scrub_stamp":"2024-03-11T20:53:11.843951+0000","last_clean_scrub_stamp":"2024-03-11T21:03:43.933551+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:49.815489+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":225,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.427732+0000","last_change":"2024-03-11T21:03:11.307482+0000","last_active":"2024-03-11T21:04:24.427732+0000","last_peered":"2024-03-11T21:04:24.427732+0000","last_clean":"2024-03-11T21:04:24.427732+0000","last_became_active":"2024-03-11T21:00:27.972907+0000","last_became_peered":"2024-03-11T21:00:27.972907+0000","last_unstale":"2024-03-11T21:04:24.427732+0000","last_undegraded":"2024-03-11T21:04:24.427732+0000","last_fullsized":"2024-03-11T21:04:24.427732+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:11.307332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:03:11.307332+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:37.141260+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.12","version":"207'19","reported_seq":406,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.434266+0000","last_change":"2024-03-11T21:02:30.944467+0000","last_active":"2024-03-11T21:04:24.434266+0000","last_peered":"2024-03-11T21:04:24.434266+0000","last_clean":"2024-03-11T21:04:24.434266+0000","last_became_active":"2024-03-11T20:56:59.796618+0000","last_became_peered":"2024-03-11T20:56:59.796618+0000","last_unstale":"2024-03-11T21:04:24.434266+0000","last_undegraded":"2024-03-11T21:04:24.434266+0000","last_fullsized":"2024-03-11T21:04:24.434266+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'19","last_scrub_stamp":"2024-03-11T21:02:30.944325+0000","last_deep_scrub":"119'13","last_deep_scrub_stamp":"2024-03-11T20:54:15.443472+0000","last_clean_scrub_stamp":"2024-03-11T21:02:30.944325+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:25.051322+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":271,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.434051+0000","last_change":"2024-03-11T21:03:10.124323+0000","last_active":"2024-03-11T21:04:24.434051+0000","last_peered":"2024-03-11T21:04:24.434051+0000","last_clean":"2024-03-11T21:04:24.434051+0000","last_became_active":"2024-03-11T20:56:59.804346+0000","last_became_peered":"2024-03-11T20:56:59.804346+0000","last_unstale":"2024-03-11T21:04:24.434051+0000","last_undegraded":"2024-03-11T21:04:24.434051+0000","last_fullsized":"2024-03-11T21:04:24.434051+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:10.124075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:03:10.124075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:32.103132+0000","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"212'17","reported_seq":657,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.441309+0000","last_change":"2024-03-11T21:02:25.533896+0000","last_active":"2024-03-11T21:04:24.441309+0000","last_peered":"2024-03-11T21:04:24.441309+0000","last_clean":"2024-03-11T21:04:24.441309+0000","last_became_active":"2024-03-11T21:00:48.467116+0000","last_became_peered":"2024-03-11T21:00:48.467116+0000","last_unstale":"2024-03-11T21:04:24.441309+0000","last_undegraded":"2024-03-11T21:04:24.441309+0000","last_fullsized":"2024-03-11T21:04:24.441309+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"212'17","last_scrub_stamp":"2024-03-11T21:02:25.533666+0000","last_deep_scrub":"110'13","last_deep_scrub_stamp":"2024-03-11T20:53:00.434532+0000","last_clean_scrub_stamp":"2024-03-11T21:02:25.533666+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:26.107541+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":471,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.439568+0000","last_change":"2024-03-11T21:03:29.061500+0000","last_active":"2024-03-11T21:04:24.439568+0000","last_peered":"2024-03-11T21:04:24.439568+0000","last_clean":"2024-03-11T21:04:24.439568+0000","last_became_active":"2024-03-11T20:50:28.070325+0000","last_became_peered":"2024-03-11T20:50:28.070325+0000","last_unstale":"2024-03-11T21:04:24.439568+0000","last_undegraded":"2024-03-11T21:04:24.439568+0000","last_fullsized":"2024-03-11T21:04:24.439568+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:29.061407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:03:29.061407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:47.973408+0000","scrub_duration":207,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"207'24","reported_seq":503,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.433688+0000","last_change":"2024-03-11T21:03:45.977452+0000","last_active":"2024-03-11T21:04:24.433688+0000","last_peered":"2024-03-11T21:04:24.433688+0000","last_clean":"2024-03-11T21:04:24.433688+0000","last_became_active":"2024-03-11T21:00:27.967078+0000","last_became_peered":"2024-03-11T21:00:27.967078+0000","last_unstale":"2024-03-11T21:04:24.433688+0000","last_undegraded":"2024-03-11T21:04:24.433688+0000","last_fullsized":"2024-03-11T21:04:24.433688+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'24","last_scrub_stamp":"2024-03-11T21:03:45.977203+0000","last_deep_scrub":"116'16","last_deep_scrub_stamp":"2024-03-11T20:53:39.292118+0000","last_clean_scrub_stamp":"2024-03-11T21:03:45.977203+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:56.354611+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":451,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.437802+0000","last_change":"2024-03-11T21:03:39.998420+0000","last_active":"2024-03-11T21:04:24.437802+0000","last_peered":"2024-03-11T21:04:24.437802+0000","last_clean":"2024-03-11T21:04:24.437802+0000","last_became_active":"2024-03-11T21:00:27.966997+0000","last_became_peered":"2024-03-11T21:00:27.966997+0000","last_unstale":"2024-03-11T21:04:24.437802+0000","last_undegraded":"2024-03-11T21:04:24.437802+0000","last_fullsized":"2024-03-11T21:04:24.437802+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:39.998251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:55:58.016988+0000","last_clean_scrub_stamp":"2024-03-11T21:03:39.998251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:50.834709+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":468,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.441283+0000","last_change":"2024-03-11T21:03:15.618457+0000","last_active":"2024-03-11T21:04:24.441283+0000","last_peered":"2024-03-11T21:04:24.441283+0000","last_clean":"2024-03-11T21:04:24.441283+0000","last_became_active":"2024-03-11T21:00:48.468386+0000","last_became_peered":"2024-03-11T21:00:48.468386+0000","last_unstale":"2024-03-11T21:04:24.441283+0000","last_undegraded":"2024-03-11T21:04:24.441283+0000","last_fullsized":"2024-03-11T21:04:24.441283+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:15.618335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:03:15.618335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:44.520777+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"166'2","reported_seq":504,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.444456+0000","last_change":"2024-03-11T21:03:12.535685+0000","last_active":"2024-03-11T21:04:24.444456+0000","last_peered":"2024-03-11T21:04:24.444456+0000","last_clean":"2024-03-11T21:04:24.444456+0000","last_became_active":"2024-03-11T21:00:48.466071+0000","last_became_peered":"2024-03-11T21:00:48.466071+0000","last_unstale":"2024-03-11T21:04:24.444456+0000","last_undegraded":"2024-03-11T21:04:24.444456+0000","last_fullsized":"2024-03-11T21:04:24.444456+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'2","last_scrub_stamp":"2024-03-11T21:03:12.535593+0000","last_deep_scrub":"166'2","last_deep_scrub_stamp":"2024-03-11T20:57:39.868776+0000","last_clean_scrub_stamp":"2024-03-11T21:03:12.535593+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:34.204233+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1822,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"207'8","reported_seq":477,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.443174+0000","last_change":"2024-03-11T21:03:50.252798+0000","last_active":"2024-03-11T21:04:24.443174+0000","last_peered":"2024-03-11T21:04:24.443174+0000","last_clean":"2024-03-11T21:04:24.443174+0000","last_became_active":"2024-03-11T20:50:30.101572+0000","last_became_peered":"2024-03-11T20:50:30.101572+0000","last_unstale":"2024-03-11T21:04:24.443174+0000","last_undegraded":"2024-03-11T21:04:24.443174+0000","last_fullsized":"2024-03-11T21:04:24.443174+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'8","last_scrub_stamp":"2024-03-11T21:03:50.252595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:03:50.252595+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:05:15.770439+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":470,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.444366+0000","last_change":"2024-03-11T21:03:09.700041+0000","last_active":"2024-03-11T21:04:24.444366+0000","last_peered":"2024-03-11T21:04:24.444366+0000","last_clean":"2024-03-11T21:04:24.444366+0000","last_became_active":"2024-03-11T21:00:27.964140+0000","last_became_peered":"2024-03-11T21:00:27.964140+0000","last_unstale":"2024-03-11T21:04:24.444366+0000","last_undegraded":"2024-03-11T21:04:24.444366+0000","last_fullsized":"2024-03-11T21:04:24.444366+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:09.699964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:03:09.699964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:29.206714+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":421,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.445688+0000","last_change":"2024-03-11T21:03:13.732388+0000","last_active":"2024-03-11T21:04:24.445688+0000","last_peered":"2024-03-11T21:04:24.445688+0000","last_clean":"2024-03-11T21:04:24.445688+0000","last_became_active":"2024-03-11T20:56:59.802356+0000","last_became_peered":"2024-03-11T20:56:59.802356+0000","last_unstale":"2024-03-11T21:04:24.445688+0000","last_undegraded":"2024-03-11T21:04:24.445688+0000","last_fullsized":"2024-03-11T21:04:24.445688+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:13.730932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:54:09.537972+0000","last_clean_scrub_stamp":"2024-03-11T21:03:13.730932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":484,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.443961+0000","last_change":"2024-03-11T21:03:22.624489+0000","last_active":"2024-03-11T21:04:24.443961+0000","last_peered":"2024-03-11T21:04:24.443961+0000","last_clean":"2024-03-11T21:04:24.443961+0000","last_became_active":"2024-03-11T21:00:48.459557+0000","last_became_peered":"2024-03-11T21:00:48.459557+0000","last_unstale":"2024-03-11T21:04:24.443961+0000","last_undegraded":"2024-03-11T21:04:24.443961+0000","last_fullsized":"2024-03-11T21:04:24.443961+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:22.623477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:18.682947+0000","last_clean_scrub_stamp":"2024-03-11T21:03:22.623477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"262'24","reported_seq":495,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.433012+0000","last_change":"2024-03-11T21:03:28.129057+0000","last_active":"2024-03-11T21:04:24.433012+0000","last_peered":"2024-03-11T21:04:24.433012+0000","last_clean":"2024-03-11T21:04:24.433012+0000","last_became_active":"2024-03-11T21:00:27.970878+0000","last_became_peered":"2024-03-11T21:00:27.970878+0000","last_unstale":"2024-03-11T21:04:24.433012+0000","last_undegraded":"2024-03-11T21:04:24.433012+0000","last_fullsized":"2024-03-11T21:04:24.433012+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"262'24","last_scrub_stamp":"2024-03-11T21:03:28.128719+0000","last_deep_scrub":"262'24","last_deep_scrub_stamp":"2024-03-11T21:03:28.128719+0000","last_clean_scrub_stamp":"2024-03-11T21:03:28.128719+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:36.970292+0000","scrub_duration":127,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.4","version":"277'6670","reported_seq":6732,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:26.872181+0000","last_change":"2024-03-11T21:02:47.636686+0000","last_active":"2024-03-11T21:04:26.872181+0000","last_peered":"2024-03-11T21:04:26.872181+0000","last_clean":"2024-03-11T21:04:26.872181+0000","last_became_active":"2024-03-11T21:02:47.634842+0000","last_became_peered":"2024-03-11T21:02:47.634842+0000","last_unstale":"2024-03-11T21:04:26.872181+0000","last_undegraded":"2024-03-11T21:04:26.872181+0000","last_fullsized":"2024-03-11T21:04:26.872181+0000","mapping_epoch":251,"log_start":"259'1500","ondisk_log_start":"259'1500","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:02:46.570492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:46.570492+0000","last_clean_scrub_stamp":"2024-03-11T21:02:46.570492+0000","objects_scrubbed":0,"log_size":5170,"log_dups_size":0,"ondisk_log_size":5170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:30.649183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":437125120,"num_objects":6670,"num_object_clones":0,"num_object_copies":20010,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6670,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6670,"num_write_kb":426880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":465,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.439422+0000","last_change":"2024-03-11T21:03:14.642466+0000","last_active":"2024-03-11T21:04:24.439422+0000","last_peered":"2024-03-11T21:04:24.439422+0000","last_clean":"2024-03-11T21:04:24.439422+0000","last_became_active":"2024-03-11T21:00:48.466853+0000","last_became_peered":"2024-03-11T21:00:48.466853+0000","last_unstale":"2024-03-11T21:04:24.439422+0000","last_undegraded":"2024-03-11T21:04:24.439422+0000","last_fullsized":"2024-03-11T21:04:24.439422+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:14.642358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:52:53.450885+0000","last_clean_scrub_stamp":"2024-03-11T21:03:14.642358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:28.543091+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"77'2","reported_seq":413,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.428411+0000","last_change":"2024-03-11T21:03:51.299386+0000","last_active":"2024-03-11T21:04:24.428411+0000","last_peered":"2024-03-11T21:04:24.428411+0000","last_clean":"2024-03-11T21:04:24.428411+0000","last_became_active":"2024-03-11T21:00:27.964018+0000","last_became_peered":"2024-03-11T21:00:27.964018+0000","last_unstale":"2024-03-11T21:04:24.428411+0000","last_undegraded":"2024-03-11T21:04:24.428411+0000","last_fullsized":"2024-03-11T21:04:24.428411+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'2","last_scrub_stamp":"2024-03-11T21:03:51.299300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:03:51.299300+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:05:01.172859+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"207'24","reported_seq":556,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.434105+0000","last_change":"2024-03-11T21:03:23.941107+0000","last_active":"2024-03-11T21:04:24.434105+0000","last_peered":"2024-03-11T21:04:24.434105+0000","last_clean":"2024-03-11T21:04:24.434105+0000","last_became_active":"2024-03-11T21:00:48.465254+0000","last_became_peered":"2024-03-11T21:00:48.465254+0000","last_unstale":"2024-03-11T21:04:24.434105+0000","last_undegraded":"2024-03-11T21:04:24.434105+0000","last_fullsized":"2024-03-11T21:04:24.434105+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'24","last_scrub_stamp":"2024-03-11T21:03:23.940968+0000","last_deep_scrub":"100'16","last_deep_scrub_stamp":"2024-03-11T20:56:08.036331+0000","last_clean_scrub_stamp":"2024-03-11T21:03:23.940968+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:33.812875+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"212'17","reported_seq":579,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.427082+0000","last_change":"2024-03-11T21:03:38.259033+0000","last_active":"2024-03-11T21:04:24.427082+0000","last_peered":"2024-03-11T21:04:24.427082+0000","last_clean":"2024-03-11T21:04:24.427082+0000","last_became_active":"2024-03-11T21:00:48.461768+0000","last_became_peered":"2024-03-11T21:00:48.461768+0000","last_unstale":"2024-03-11T21:04:24.427082+0000","last_undegraded":"2024-03-11T21:04:24.427082+0000","last_fullsized":"2024-03-11T21:04:24.427082+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"212'17","last_scrub_stamp":"2024-03-11T21:03:38.258850+0000","last_deep_scrub":"212'17","last_deep_scrub_stamp":"2024-03-11T21:03:38.258850+0000","last_clean_scrub_stamp":"2024-03-11T21:03:38.258850+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:55.059529+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.f","version":"277'6458","reported_seq":6517,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:25.239611+0000","last_change":"2024-03-11T21:02:47.637043+0000","last_active":"2024-03-11T21:04:25.239611+0000","last_peered":"2024-03-11T21:04:25.239611+0000","last_clean":"2024-03-11T21:04:25.239611+0000","last_became_active":"2024-03-11T21:02:47.636634+0000","last_became_peered":"2024-03-11T21:02:47.636634+0000","last_unstale":"2024-03-11T21:04:25.239611+0000","last_undegraded":"2024-03-11T21:04:25.239611+0000","last_fullsized":"2024-03-11T21:04:25.239611+0000","mapping_epoch":251,"log_start":"259'1100","ondisk_log_start":"259'1100","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:02:46.570492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:46.570492+0000","last_clean_scrub_stamp":"2024-03-11T21:02:46.570492+0000","objects_scrubbed":0,"log_size":5358,"log_dups_size":0,"ondisk_log_size":5358,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:26.440224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":423231488,"num_objects":6458,"num_object_clones":0,"num_object_copies":19374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6458,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6458,"num_write_kb":413312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":273,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.438833+0000","last_change":"2024-03-11T21:00:24.793264+0000","last_active":"2024-03-11T21:04:24.438833+0000","last_peered":"2024-03-11T21:04:24.438833+0000","last_clean":"2024-03-11T21:04:24.438833+0000","last_became_active":"2024-03-11T20:56:59.795191+0000","last_became_peered":"2024-03-11T20:56:59.795191+0000","last_unstale":"2024-03-11T21:04:24.438833+0000","last_undegraded":"2024-03-11T21:04:24.438833+0000","last_fullsized":"2024-03-11T21:04:24.438833+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:00:24.793093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:57:21.568400+0000","last_clean_scrub_stamp":"2024-03-11T21:00:24.793093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:26.107486+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":170,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.454519+0000","last_change":"2024-03-11T21:03:40.610888+0000","last_active":"2024-03-11T21:04:24.454519+0000","last_peered":"2024-03-11T21:04:24.454519+0000","last_clean":"2024-03-11T21:04:24.454519+0000","last_became_active":"2024-03-11T21:00:48.455733+0000","last_became_peered":"2024-03-11T21:00:48.455733+0000","last_unstale":"2024-03-11T21:04:24.454519+0000","last_undegraded":"2024-03-11T21:04:24.454519+0000","last_fullsized":"2024-03-11T21:04:24.454519+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:40.610810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:20.096874+0000","last_clean_scrub_stamp":"2024-03-11T21:03:40.610810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:59.174036+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"207'28","reported_seq":668,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.443700+0000","last_change":"2024-03-11T21:03:15.735024+0000","last_active":"2024-03-11T21:04:24.443700+0000","last_peered":"2024-03-11T21:04:24.443700+0000","last_clean":"2024-03-11T21:04:24.443700+0000","last_became_active":"2024-03-11T20:50:30.094977+0000","last_became_peered":"2024-03-11T20:50:30.094977+0000","last_unstale":"2024-03-11T21:04:24.443700+0000","last_undegraded":"2024-03-11T21:04:24.443700+0000","last_fullsized":"2024-03-11T21:04:24.443700+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'28","last_scrub_stamp":"2024-03-11T21:03:15.734883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:03:15.734883+0000","objects_scrubbed":8,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:29.110040+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":162,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.447068+0000","last_change":"2024-03-11T21:02:30.288033+0000","last_active":"2024-03-11T21:04:24.447068+0000","last_peered":"2024-03-11T21:04:24.447068+0000","last_clean":"2024-03-11T21:04:24.447068+0000","last_became_active":"2024-03-11T21:00:48.457901+0000","last_became_peered":"2024-03-11T21:00:48.457901+0000","last_unstale":"2024-03-11T21:04:24.447068+0000","last_undegraded":"2024-03-11T21:04:24.447068+0000","last_fullsized":"2024-03-11T21:04:24.447068+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:02:30.287958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:52:10.160202+0000","last_clean_scrub_stamp":"2024-03-11T21:02:30.287958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"277'6583","reported_seq":6643,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:25.066256+0000","last_change":"2024-03-11T21:02:47.621307+0000","last_active":"2024-03-11T21:04:25.066256+0000","last_peered":"2024-03-11T21:04:25.066256+0000","last_clean":"2024-03-11T21:04:25.066256+0000","last_became_active":"2024-03-11T21:02:47.619716+0000","last_became_peered":"2024-03-11T21:02:47.619716+0000","last_unstale":"2024-03-11T21:04:25.066256+0000","last_undegraded":"2024-03-11T21:04:25.066256+0000","last_fullsized":"2024-03-11T21:04:25.066256+0000","mapping_epoch":251,"log_start":"260'1900","ondisk_log_start":"260'1900","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:02:46.570492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:46.570492+0000","last_clean_scrub_stamp":"2024-03-11T21:02:46.570492+0000","objects_scrubbed":0,"log_size":4683,"log_dups_size":0,"ondisk_log_size":4683,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":431423488,"num_objects":6583,"num_object_clones":0,"num_object_copies":19749,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6583,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6583,"num_write_kb":421312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":453,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.440958+0000","last_change":"2024-03-11T21:03:28.677362+0000","last_active":"2024-03-11T21:04:24.440958+0000","last_peered":"2024-03-11T21:04:24.440958+0000","last_clean":"2024-03-11T21:04:24.440958+0000","last_became_active":"2024-03-11T21:00:27.959408+0000","last_became_peered":"2024-03-11T21:00:27.959408+0000","last_unstale":"2024-03-11T21:04:24.440958+0000","last_undegraded":"2024-03-11T21:04:24.440958+0000","last_fullsized":"2024-03-11T21:04:24.440958+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:28.677276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:03:28.677276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:49.954168+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":477,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.439621+0000","last_change":"2024-03-11T21:02:29.807630+0000","last_active":"2024-03-11T21:04:24.439621+0000","last_peered":"2024-03-11T21:04:24.439621+0000","last_clean":"2024-03-11T21:04:24.439621+0000","last_became_active":"2024-03-11T21:00:48.471212+0000","last_became_peered":"2024-03-11T21:00:48.471212+0000","last_unstale":"2024-03-11T21:04:24.439621+0000","last_undegraded":"2024-03-11T21:04:24.439621+0000","last_fullsized":"2024-03-11T21:04:24.439621+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:02:29.807044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:33.823704+0000","last_clean_scrub_stamp":"2024-03-11T21:02:29.807044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:26.440180+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"207'37","reported_seq":484,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.427633+0000","last_change":"2024-03-11T21:03:22.320789+0000","last_active":"2024-03-11T21:04:24.427633+0000","last_peered":"2024-03-11T21:04:24.427633+0000","last_clean":"2024-03-11T21:04:24.427633+0000","last_became_active":"2024-03-11T21:00:27.960377+0000","last_became_peered":"2024-03-11T21:00:27.960377+0000","last_unstale":"2024-03-11T21:04:24.427633+0000","last_undegraded":"2024-03-11T21:04:24.427633+0000","last_fullsized":"2024-03-11T21:04:24.427633+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'37","last_scrub_stamp":"2024-03-11T21:03:22.320643+0000","last_deep_scrub":"182'27","last_deep_scrub_stamp":"2024-03-11T20:58:52.058851+0000","last_clean_scrub_stamp":"2024-03-11T21:03:22.320643+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:34.075583+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":170,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.453443+0000","last_change":"2024-03-11T21:03:44.649559+0000","last_active":"2024-03-11T21:04:24.453443+0000","last_peered":"2024-03-11T21:04:24.453443+0000","last_clean":"2024-03-11T21:04:24.453443+0000","last_became_active":"2024-03-11T21:00:48.458008+0000","last_became_peered":"2024-03-11T21:00:48.458008+0000","last_unstale":"2024-03-11T21:04:24.453443+0000","last_undegraded":"2024-03-11T21:04:24.453443+0000","last_fullsized":"2024-03-11T21:04:24.453443+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:44.649310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:33.319221+0000","last_clean_scrub_stamp":"2024-03-11T21:03:44.649310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:05:08.200071+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.a","version":"277'6461","reported_seq":6523,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:25.252217+0000","last_change":"2024-03-11T21:02:47.634170+0000","last_active":"2024-03-11T21:04:25.252217+0000","last_peered":"2024-03-11T21:04:25.252217+0000","last_clean":"2024-03-11T21:04:25.252217+0000","last_became_active":"2024-03-11T21:02:47.633164+0000","last_became_peered":"2024-03-11T21:02:47.633164+0000","last_unstale":"2024-03-11T21:04:25.252217+0000","last_undegraded":"2024-03-11T21:04:25.252217+0000","last_fullsized":"2024-03-11T21:04:25.252217+0000","mapping_epoch":251,"log_start":"259'1100","ondisk_log_start":"259'1100","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:02:46.570492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:46.570492+0000","last_clean_scrub_stamp":"2024-03-11T21:02:46.570492+0000","objects_scrubbed":0,"log_size":5361,"log_dups_size":0,"ondisk_log_size":5361,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:26.440298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":423428096,"num_objects":6461,"num_object_clones":0,"num_object_copies":19383,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6461,"num_write_kb":413504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":451,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.437911+0000","last_change":"2024-03-11T21:02:32.951250+0000","last_active":"2024-03-11T21:04:24.437911+0000","last_peered":"2024-03-11T21:04:24.437911+0000","last_clean":"2024-03-11T21:04:24.437911+0000","last_became_active":"2024-03-11T20:50:34.321038+0000","last_became_peered":"2024-03-11T20:50:34.321038+0000","last_unstale":"2024-03-11T21:04:24.437911+0000","last_undegraded":"2024-03-11T21:04:24.437911+0000","last_fullsized":"2024-03-11T21:04:24.437911+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:02:32.951104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:32.951104+0000","last_clean_scrub_stamp":"2024-03-11T21:02:32.951104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:25.051291+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"207'38","reported_seq":194,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.444820+0000","last_change":"2024-03-11T21:03:06.361379+0000","last_active":"2024-03-11T21:04:24.444820+0000","last_peered":"2024-03-11T21:04:24.444820+0000","last_clean":"2024-03-11T21:04:24.444820+0000","last_became_active":"2024-03-11T21:00:48.456178+0000","last_became_peered":"2024-03-11T21:00:48.456178+0000","last_unstale":"2024-03-11T21:04:24.444820+0000","last_undegraded":"2024-03-11T21:04:24.444820+0000","last_fullsized":"2024-03-11T21:04:24.444820+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'38","last_scrub_stamp":"2024-03-11T21:03:06.361192+0000","last_deep_scrub":"109'26","last_deep_scrub_stamp":"2024-03-11T20:52:49.216378+0000","last_clean_scrub_stamp":"2024-03-11T21:03:06.361192+0000","objects_scrubbed":8,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:33.252260+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":479,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.443621+0000","last_change":"2024-03-11T21:03:19.688224+0000","last_active":"2024-03-11T21:04:24.443621+0000","last_peered":"2024-03-11T21:04:24.443621+0000","last_clean":"2024-03-11T21:04:24.443621+0000","last_became_active":"2024-03-11T20:50:28.045625+0000","last_became_peered":"2024-03-11T20:50:28.045625+0000","last_unstale":"2024-03-11T21:04:24.443621+0000","last_undegraded":"2024-03-11T21:04:24.443621+0000","last_fullsized":"2024-03-11T21:04:24.443621+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:19.688076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:03:19.688076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:26.035370+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":162,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.454264+0000","last_change":"2024-03-11T21:03:11.331958+0000","last_active":"2024-03-11T21:04:24.454264+0000","last_peered":"2024-03-11T21:04:24.454264+0000","last_clean":"2024-03-11T21:04:24.454264+0000","last_became_active":"2024-03-11T21:00:48.473346+0000","last_became_peered":"2024-03-11T21:00:48.473346+0000","last_unstale":"2024-03-11T21:04:24.454264+0000","last_undegraded":"2024-03-11T21:04:24.454264+0000","last_fullsized":"2024-03-11T21:04:24.454264+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:11.331883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:03:11.331883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"277'6405","reported_seq":6467,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:26.099108+0000","last_change":"2024-03-11T21:02:47.639919+0000","last_active":"2024-03-11T21:04:26.099108+0000","last_peered":"2024-03-11T21:04:26.099108+0000","last_clean":"2024-03-11T21:04:26.099108+0000","last_became_active":"2024-03-11T21:02:47.639328+0000","last_became_peered":"2024-03-11T21:02:47.639328+0000","last_unstale":"2024-03-11T21:04:26.099108+0000","last_undegraded":"2024-03-11T21:04:26.099108+0000","last_fullsized":"2024-03-11T21:04:26.099108+0000","mapping_epoch":251,"log_start":"259'1700","ondisk_log_start":"259'1700","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:02:46.570492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:46.570492+0000","last_clean_scrub_stamp":"2024-03-11T21:02:46.570492+0000","objects_scrubbed":0,"log_size":4705,"log_dups_size":0,"ondisk_log_size":4705,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":419758080,"num_objects":6405,"num_object_clones":0,"num_object_copies":19215,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6405,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6405,"num_write_kb":409920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":453,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.440576+0000","last_change":"2024-03-11T21:03:11.800852+0000","last_active":"2024-03-11T21:04:24.440576+0000","last_peered":"2024-03-11T21:04:24.440576+0000","last_clean":"2024-03-11T21:04:24.440576+0000","last_became_active":"2024-03-11T21:00:27.962668+0000","last_became_peered":"2024-03-11T21:00:27.962668+0000","last_unstale":"2024-03-11T21:04:24.440576+0000","last_undegraded":"2024-03-11T21:04:24.440576+0000","last_fullsized":"2024-03-11T21:04:24.440576+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:11.800717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:03:11.800717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:28.883743+0000","scrub_duration":83,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"207'20","reported_seq":482,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.439467+0000","last_change":"2024-03-11T21:00:14.736355+0000","last_active":"2024-03-11T21:04:24.439467+0000","last_peered":"2024-03-11T21:04:24.439467+0000","last_clean":"2024-03-11T21:04:24.439467+0000","last_became_active":"2024-03-11T20:50:30.101747+0000","last_became_peered":"2024-03-11T20:50:30.101747+0000","last_unstale":"2024-03-11T21:04:24.439467+0000","last_undegraded":"2024-03-11T21:04:24.439467+0000","last_fullsized":"2024-03-11T21:04:24.439467+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'10","last_scrub_stamp":"2024-03-11T21:00:14.736261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:00:14.736261+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:26.107318+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":473,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.439981+0000","last_change":"2024-03-11T21:03:47.212843+0000","last_active":"2024-03-11T21:04:24.439981+0000","last_peered":"2024-03-11T21:04:24.439981+0000","last_clean":"2024-03-11T21:04:24.439981+0000","last_became_active":"2024-03-11T20:50:28.085503+0000","last_became_peered":"2024-03-11T20:50:28.085503+0000","last_unstale":"2024-03-11T21:04:24.439981+0000","last_undegraded":"2024-03-11T21:04:24.439981+0000","last_fullsized":"2024-03-11T21:04:24.439981+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:47.212723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:56:03.479180+0000","last_clean_scrub_stamp":"2024-03-11T21:03:47.212723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:55.569265+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"184'130","reported_seq":732,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.433066+0000","last_change":"2024-03-11T21:02:15.792152+0000","last_active":"2024-03-11T21:04:24.433066+0000","last_peered":"2024-03-11T21:04:24.433066+0000","last_clean":"2024-03-11T21:04:24.433066+0000","last_became_active":"2024-03-11T20:49:52.584543+0000","last_became_peered":"2024-03-11T20:49:52.584543+0000","last_unstale":"2024-03-11T21:04:24.433066+0000","last_undegraded":"2024-03-11T21:04:24.433066+0000","last_fullsized":"2024-03-11T21:04:24.433066+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"184'130","last_scrub_stamp":"2024-03-11T21:02:15.791905+0000","last_deep_scrub":"184'130","last_deep_scrub_stamp":"2024-03-11T20:58:56.415267+0000","last_clean_scrub_stamp":"2024-03-11T21:02:15.791905+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:25.051265+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":471,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.443827+0000","last_change":"2024-03-11T21:03:17.506757+0000","last_active":"2024-03-11T21:04:24.443827+0000","last_peered":"2024-03-11T21:04:24.443827+0000","last_clean":"2024-03-11T21:04:24.443827+0000","last_became_active":"2024-03-11T21:00:27.959467+0000","last_became_peered":"2024-03-11T21:00:27.959467+0000","last_unstale":"2024-03-11T21:04:24.443827+0000","last_undegraded":"2024-03-11T21:04:24.443827+0000","last_fullsized":"2024-03-11T21:04:24.443827+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:17.506583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:54:22.448812+0000","last_clean_scrub_stamp":"2024-03-11T21:03:17.506583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:26.066370+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.b","version":"277'6473","reported_seq":6535,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:25.043496+0000","last_change":"2024-03-11T21:02:47.622120+0000","last_active":"2024-03-11T21:04:25.043496+0000","last_peered":"2024-03-11T21:04:25.043496+0000","last_clean":"2024-03-11T21:04:25.043496+0000","last_became_active":"2024-03-11T21:02:47.621751+0000","last_became_peered":"2024-03-11T21:02:47.621751+0000","last_unstale":"2024-03-11T21:04:25.043496+0000","last_undegraded":"2024-03-11T21:04:25.043496+0000","last_fullsized":"2024-03-11T21:04:25.043496+0000","mapping_epoch":251,"log_start":"257'300","ondisk_log_start":"257'300","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:02:46.570492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:46.570492+0000","last_clean_scrub_stamp":"2024-03-11T21:02:46.570492+0000","objects_scrubbed":0,"log_size":6173,"log_dups_size":0,"ondisk_log_size":6173,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:27.631184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":424214528,"num_objects":6473,"num_object_clones":0,"num_object_copies":19419,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6473,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6473,"num_write_kb":414272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":162,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.453725+0000","last_change":"2024-03-11T21:02:25.182012+0000","last_active":"2024-03-11T21:04:24.453725+0000","last_peered":"2024-03-11T21:04:24.453725+0000","last_clean":"2024-03-11T21:04:24.453725+0000","last_became_active":"2024-03-11T21:00:48.464596+0000","last_became_peered":"2024-03-11T21:00:48.464596+0000","last_unstale":"2024-03-11T21:04:24.453725+0000","last_undegraded":"2024-03-11T21:04:24.453725+0000","last_fullsized":"2024-03-11T21:04:24.453725+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:02:25.181915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:02:25.181915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"207'35","reported_seq":333,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.427369+0000","last_change":"2024-03-11T21:02:32.429640+0000","last_active":"2024-03-11T21:04:24.427369+0000","last_peered":"2024-03-11T21:04:24.427369+0000","last_clean":"2024-03-11T21:04:24.427369+0000","last_became_active":"2024-03-11T21:00:27.959172+0000","last_became_peered":"2024-03-11T21:00:27.959172+0000","last_unstale":"2024-03-11T21:04:24.427369+0000","last_undegraded":"2024-03-11T21:04:24.427369+0000","last_fullsized":"2024-03-11T21:04:24.427369+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'35","last_scrub_stamp":"2024-03-11T21:02:32.429525+0000","last_deep_scrub":"100'21","last_deep_scrub_stamp":"2024-03-11T20:51:31.835302+0000","last_clean_scrub_stamp":"2024-03-11T21:02:32.429525+0000","objects_scrubbed":7,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":450,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.444136+0000","last_change":"2024-03-11T21:03:48.784865+0000","last_active":"2024-03-11T21:04:24.444136+0000","last_peered":"2024-03-11T21:04:24.444136+0000","last_clean":"2024-03-11T21:04:24.444136+0000","last_became_active":"2024-03-11T21:00:48.458579+0000","last_became_peered":"2024-03-11T21:00:48.458579+0000","last_unstale":"2024-03-11T21:04:24.444136+0000","last_undegraded":"2024-03-11T21:04:24.444136+0000","last_fullsized":"2024-03-11T21:04:24.444136+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:48.784648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:03:48.784648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:05:02.043800+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":474,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.446884+0000","last_change":"2024-03-11T21:03:45.421629+0000","last_active":"2024-03-11T21:04:24.446884+0000","last_peered":"2024-03-11T21:04:24.446884+0000","last_clean":"2024-03-11T21:04:24.446884+0000","last_became_active":"2024-03-11T21:00:48.463247+0000","last_became_peered":"2024-03-11T21:00:48.463247+0000","last_unstale":"2024-03-11T21:04:24.446884+0000","last_undegraded":"2024-03-11T21:04:24.446884+0000","last_fullsized":"2024-03-11T21:04:24.446884+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:45.421446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:26.839782+0000","last_clean_scrub_stamp":"2024-03-11T21:03:45.421446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:52.927501+0000","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.e","version":"277'6523","reported_seq":6585,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:25.062491+0000","last_change":"2024-03-11T21:02:47.617410+0000","last_active":"2024-03-11T21:04:25.062491+0000","last_peered":"2024-03-11T21:04:25.062491+0000","last_clean":"2024-03-11T21:04:25.062491+0000","last_became_active":"2024-03-11T21:02:47.616727+0000","last_became_peered":"2024-03-11T21:02:47.616727+0000","last_unstale":"2024-03-11T21:04:25.062491+0000","last_undegraded":"2024-03-11T21:04:25.062491+0000","last_fullsized":"2024-03-11T21:04:25.062491+0000","mapping_epoch":251,"log_start":"260'1900","ondisk_log_start":"260'1900","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:02:46.570492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:46.570492+0000","last_clean_scrub_stamp":"2024-03-11T21:02:46.570492+0000","objects_scrubbed":0,"log_size":4623,"log_dups_size":0,"ondisk_log_size":4623,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":427491328,"num_objects":6523,"num_object_clones":0,"num_object_copies":19569,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6523,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6523,"num_write_kb":417472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":170,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.443974+0000","last_change":"2024-03-11T21:03:28.495517+0000","last_active":"2024-03-11T21:04:24.443974+0000","last_peered":"2024-03-11T21:04:24.443974+0000","last_clean":"2024-03-11T21:04:24.443974+0000","last_became_active":"2024-03-11T21:00:48.469688+0000","last_became_peered":"2024-03-11T21:00:48.469688+0000","last_unstale":"2024-03-11T21:04:24.443974+0000","last_undegraded":"2024-03-11T21:04:24.443974+0000","last_fullsized":"2024-03-11T21:04:24.443974+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:28.495308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:03:28.495308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:45.445522+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"145'3","reported_seq":472,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.443470+0000","last_change":"2024-03-11T21:03:44.305877+0000","last_active":"2024-03-11T21:04:24.443470+0000","last_peered":"2024-03-11T21:04:24.443470+0000","last_clean":"2024-03-11T21:04:24.443470+0000","last_became_active":"2024-03-11T20:50:32.195452+0000","last_became_peered":"2024-03-11T20:50:32.195452+0000","last_unstale":"2024-03-11T21:04:24.443470+0000","last_undegraded":"2024-03-11T21:04:24.443470+0000","last_fullsized":"2024-03-11T21:04:24.443470+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'3","last_scrub_stamp":"2024-03-11T21:03:44.305716+0000","last_deep_scrub":"145'3","last_deep_scrub_stamp":"2024-03-11T20:55:52.848043+0000","last_clean_scrub_stamp":"2024-03-11T21:03:44.305716+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:47.051450+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0.0024049620000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":464,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.437974+0000","last_change":"2024-03-11T21:02:04.522844+0000","last_active":"2024-03-11T21:04:24.437974+0000","last_peered":"2024-03-11T21:04:24.437974+0000","last_clean":"2024-03-11T21:04:24.437974+0000","last_became_active":"2024-03-11T21:00:27.966749+0000","last_became_peered":"2024-03-11T21:00:27.966749+0000","last_unstale":"2024-03-11T21:04:24.437974+0000","last_undegraded":"2024-03-11T21:04:24.437974+0000","last_fullsized":"2024-03-11T21:04:24.437974+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:02:04.522743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:02:04.522743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:25.051207+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"212'46","reported_seq":243,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.454282+0000","last_change":"2024-03-11T21:03:12.324530+0000","last_active":"2024-03-11T21:04:24.454282+0000","last_peered":"2024-03-11T21:04:24.454282+0000","last_clean":"2024-03-11T21:04:24.454282+0000","last_became_active":"2024-03-11T21:00:48.473227+0000","last_became_peered":"2024-03-11T21:00:48.473227+0000","last_unstale":"2024-03-11T21:04:24.454282+0000","last_undegraded":"2024-03-11T21:04:24.454282+0000","last_fullsized":"2024-03-11T21:04:24.454282+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"212'46","last_scrub_stamp":"2024-03-11T21:03:12.324431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:03:12.324431+0000","objects_scrubbed":10,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:37.435182+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":69,"num_read_kb":54,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"160'12","reported_seq":388,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.439687+0000","last_change":"2024-03-11T21:03:40.870833+0000","last_active":"2024-03-11T21:04:24.439687+0000","last_peered":"2024-03-11T21:04:24.439687+0000","last_clean":"2024-03-11T21:04:24.439687+0000","last_became_active":"2024-03-11T20:56:59.797805+0000","last_became_peered":"2024-03-11T20:56:59.797805+0000","last_unstale":"2024-03-11T21:04:24.439687+0000","last_undegraded":"2024-03-11T21:04:24.439687+0000","last_fullsized":"2024-03-11T21:04:24.439687+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"160'12","last_scrub_stamp":"2024-03-11T21:03:40.870714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:03:40.870714+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:53.829709+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"277'6464","reported_seq":6526,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:25.039187+0000","last_change":"2024-03-11T21:02:47.623642+0000","last_active":"2024-03-11T21:04:25.039187+0000","last_peered":"2024-03-11T21:04:25.039187+0000","last_clean":"2024-03-11T21:04:25.039187+0000","last_became_active":"2024-03-11T21:02:47.623292+0000","last_became_peered":"2024-03-11T21:02:47.623292+0000","last_unstale":"2024-03-11T21:04:25.039187+0000","last_undegraded":"2024-03-11T21:04:25.039187+0000","last_fullsized":"2024-03-11T21:04:25.039187+0000","mapping_epoch":251,"log_start":"260'1800","ondisk_log_start":"260'1800","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:02:46.570492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:46.570492+0000","last_clean_scrub_stamp":"2024-03-11T21:02:46.570492+0000","objects_scrubbed":0,"log_size":4664,"log_dups_size":0,"ondisk_log_size":4664,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:25.051396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":423624704,"num_objects":6464,"num_object_clones":0,"num_object_copies":19392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6464,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6464,"num_write_kb":413696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":375,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.427482+0000","last_change":"2024-03-11T21:03:43.345403+0000","last_active":"2024-03-11T21:04:24.427482+0000","last_peered":"2024-03-11T21:04:24.427482+0000","last_clean":"2024-03-11T21:04:24.427482+0000","last_became_active":"2024-03-11T21:00:27.961447+0000","last_became_peered":"2024-03-11T21:00:27.961447+0000","last_unstale":"2024-03-11T21:04:24.427482+0000","last_undegraded":"2024-03-11T21:04:24.427482+0000","last_fullsized":"2024-03-11T21:04:24.427482+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:43.345257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:03:43.345257+0000","last_clean_scrub_stamp":"2024-03-11T21:03:43.345257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:44.667497+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":388,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.427271+0000","last_change":"2024-03-11T21:02:22.618230+0000","last_active":"2024-03-11T21:04:24.427271+0000","last_peered":"2024-03-11T21:04:24.427271+0000","last_clean":"2024-03-11T21:04:24.427271+0000","last_became_active":"2024-03-11T21:00:27.967546+0000","last_became_peered":"2024-03-11T21:00:27.967546+0000","last_unstale":"2024-03-11T21:04:24.427271+0000","last_undegraded":"2024-03-11T21:04:24.427271+0000","last_fullsized":"2024-03-11T21:04:24.427271+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:02:22.617999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:22.617999+0000","last_clean_scrub_stamp":"2024-03-11T21:02:22.617999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"77'1","reported_seq":166,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.454729+0000","last_change":"2024-03-11T21:03:07.312601+0000","last_active":"2024-03-11T21:04:24.454729+0000","last_peered":"2024-03-11T21:04:24.454729+0000","last_clean":"2024-03-11T21:04:24.454729+0000","last_became_active":"2024-03-11T21:00:48.468682+0000","last_became_peered":"2024-03-11T21:00:48.468682+0000","last_unstale":"2024-03-11T21:04:24.454729+0000","last_undegraded":"2024-03-11T21:04:24.454729+0000","last_fullsized":"2024-03-11T21:04:24.454729+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-03-11T21:03:07.312455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:03:07.312455+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"207'17","reported_seq":485,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.441019+0000","last_change":"2024-03-11T21:03:42.690547+0000","last_active":"2024-03-11T21:04:24.441019+0000","last_peered":"2024-03-11T21:04:24.441019+0000","last_clean":"2024-03-11T21:04:24.441019+0000","last_became_active":"2024-03-11T20:50:30.097669+0000","last_became_peered":"2024-03-11T20:50:30.097669+0000","last_unstale":"2024-03-11T21:04:24.441019+0000","last_undegraded":"2024-03-11T21:04:24.441019+0000","last_fullsized":"2024-03-11T21:04:24.441019+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'17","last_scrub_stamp":"2024-03-11T21:03:42.690411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:03:42.690411+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:05:11.885193+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"84'11","reported_seq":634,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.439785+0000","last_change":"2024-03-11T21:00:20.771711+0000","last_active":"2024-03-11T21:04:24.439785+0000","last_peered":"2024-03-11T21:04:24.439785+0000","last_clean":"2024-03-11T21:04:24.439785+0000","last_became_active":"2024-03-11T20:50:32.647041+0000","last_became_peered":"2024-03-11T20:50:32.647041+0000","last_unstale":"2024-03-11T21:04:24.439785+0000","last_undegraded":"2024-03-11T21:04:24.439785+0000","last_fullsized":"2024-03-11T21:04:24.439785+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:00:20.771539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:00:20.771539+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:26.107279+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.c","version":"277'6629","reported_seq":6690,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:26.197505+0000","last_change":"2024-03-11T21:02:47.630285+0000","last_active":"2024-03-11T21:04:26.197505+0000","last_peered":"2024-03-11T21:04:26.197505+0000","last_clean":"2024-03-11T21:04:26.197505+0000","last_became_active":"2024-03-11T21:02:47.629442+0000","last_became_peered":"2024-03-11T21:02:47.629442+0000","last_unstale":"2024-03-11T21:04:26.197505+0000","last_undegraded":"2024-03-11T21:04:26.197505+0000","last_fullsized":"2024-03-11T21:04:26.197505+0000","mapping_epoch":251,"log_start":"260'2000","ondisk_log_start":"260'2000","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:02:46.570492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:46.570492+0000","last_clean_scrub_stamp":"2024-03-11T21:02:46.570492+0000","objects_scrubbed":0,"log_size":4629,"log_dups_size":0,"ondisk_log_size":4629,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:31.193750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":434438144,"num_objects":6629,"num_object_clones":0,"num_object_copies":19887,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6629,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6629,"num_write_kb":424256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":459,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.433588+0000","last_change":"2024-03-11T21:03:09.148007+0000","last_active":"2024-03-11T21:04:24.433588+0000","last_peered":"2024-03-11T21:04:24.433588+0000","last_clean":"2024-03-11T21:04:24.433588+0000","last_became_active":"2024-03-11T20:50:34.326947+0000","last_became_peered":"2024-03-11T20:50:34.326947+0000","last_unstale":"2024-03-11T21:04:24.433588+0000","last_undegraded":"2024-03-11T21:04:24.433588+0000","last_fullsized":"2024-03-11T21:04:24.433588+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:09.147923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:03:09.147923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:36.318911+0000","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"120'6","reported_seq":1199,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:25.137722+0000","last_change":"2024-03-11T21:02:19.475999+0000","last_active":"2024-03-11T21:04:25.137722+0000","last_peered":"2024-03-11T21:04:25.137722+0000","last_clean":"2024-03-11T21:04:25.137722+0000","last_became_active":"2024-03-11T21:00:48.468997+0000","last_became_peered":"2024-03-11T21:00:48.468997+0000","last_unstale":"2024-03-11T21:04:25.137722+0000","last_undegraded":"2024-03-11T21:04:25.137722+0000","last_fullsized":"2024-03-11T21:04:25.137722+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"120'6","last_scrub_stamp":"2024-03-11T21:02:19.475892+0000","last_deep_scrub":"120'6","last_deep_scrub_stamp":"2024-03-11T20:54:26.468057+0000","last_clean_scrub_stamp":"2024-03-11T21:02:19.475892+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0035533370000000002,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":702,"num_read_kb":697,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":478,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.444136+0000","last_change":"2024-03-11T21:03:26.508296+0000","last_active":"2024-03-11T21:04:24.444136+0000","last_peered":"2024-03-11T21:04:24.444136+0000","last_clean":"2024-03-11T21:04:24.444136+0000","last_became_active":"2024-03-11T20:50:28.071707+0000","last_became_peered":"2024-03-11T20:50:28.071707+0000","last_unstale":"2024-03-11T21:04:24.444136+0000","last_undegraded":"2024-03-11T21:04:24.444136+0000","last_fullsized":"2024-03-11T21:04:24.444136+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:26.508215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:23.417283+0000","last_clean_scrub_stamp":"2024-03-11T21:03:26.508215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:46.116991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"207'24","reported_seq":562,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.446659+0000","last_change":"2024-03-11T21:03:47.381203+0000","last_active":"2024-03-11T21:04:24.446659+0000","last_peered":"2024-03-11T21:04:24.446659+0000","last_clean":"2024-03-11T21:04:24.446659+0000","last_became_active":"2024-03-11T21:00:48.461972+0000","last_became_peered":"2024-03-11T21:00:48.461972+0000","last_unstale":"2024-03-11T21:04:24.446659+0000","last_undegraded":"2024-03-11T21:04:24.446659+0000","last_fullsized":"2024-03-11T21:04:24.446659+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'24","last_scrub_stamp":"2024-03-11T21:03:47.381061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:03:47.381061+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:05:05.348872+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":409,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.427296+0000","last_change":"2024-03-11T21:03:12.258970+0000","last_active":"2024-03-11T21:04:24.427296+0000","last_peered":"2024-03-11T21:04:24.427296+0000","last_clean":"2024-03-11T21:04:24.427296+0000","last_became_active":"2024-03-11T21:00:27.953537+0000","last_became_peered":"2024-03-11T21:00:27.953537+0000","last_unstale":"2024-03-11T21:04:24.427296+0000","last_undegraded":"2024-03-11T21:04:24.427296+0000","last_fullsized":"2024-03-11T21:04:24.427296+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:12.258842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:03:12.258842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.3","version":"277'6591","reported_seq":6653,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:25.021108+0000","last_change":"2024-03-11T21:02:47.628471+0000","last_active":"2024-03-11T21:04:25.021108+0000","last_peered":"2024-03-11T21:04:25.021108+0000","last_clean":"2024-03-11T21:04:25.021108+0000","last_became_active":"2024-03-11T21:02:47.627896+0000","last_became_peered":"2024-03-11T21:02:47.627896+0000","last_unstale":"2024-03-11T21:04:25.021108+0000","last_undegraded":"2024-03-11T21:04:25.021108+0000","last_fullsized":"2024-03-11T21:04:25.021108+0000","mapping_epoch":251,"log_start":"260'1900","ondisk_log_start":"260'1900","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:02:46.570492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:46.570492+0000","last_clean_scrub_stamp":"2024-03-11T21:02:46.570492+0000","objects_scrubbed":0,"log_size":4691,"log_dups_size":0,"ondisk_log_size":4691,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:25.055392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":431947776,"num_objects":6591,"num_object_clones":0,"num_object_copies":19773,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6591,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6591,"num_write_kb":421824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":444,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.440610+0000","last_change":"2024-03-11T21:03:43.723734+0000","last_active":"2024-03-11T21:04:24.440610+0000","last_peered":"2024-03-11T21:04:24.440610+0000","last_clean":"2024-03-11T21:04:24.440610+0000","last_became_active":"2024-03-11T20:50:34.335370+0000","last_became_peered":"2024-03-11T20:50:34.335370+0000","last_unstale":"2024-03-11T21:04:24.440610+0000","last_undegraded":"2024-03-11T21:04:24.440610+0000","last_fullsized":"2024-03-11T21:04:24.440610+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:43.715417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:03:43.715417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:52.411766+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"249'17","reported_seq":588,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.428324+0000","last_change":"2024-03-11T21:03:24.365609+0000","last_active":"2024-03-11T21:04:24.428324+0000","last_peered":"2024-03-11T21:04:24.428324+0000","last_clean":"2024-03-11T21:04:24.428324+0000","last_became_active":"2024-03-11T21:02:37.524445+0000","last_became_peered":"2024-03-11T21:02:37.524445+0000","last_unstale":"2024-03-11T21:04:24.428324+0000","last_undegraded":"2024-03-11T21:04:24.428324+0000","last_fullsized":"2024-03-11T21:04:24.428324+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"249'17","last_scrub_stamp":"2024-03-11T21:03:24.365350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:03:24.365350+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:42.564922+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":476,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.433201+0000","last_change":"2024-03-11T21:03:20.066682+0000","last_active":"2024-03-11T21:04:24.433201+0000","last_peered":"2024-03-11T21:04:24.433201+0000","last_clean":"2024-03-11T21:04:24.433201+0000","last_became_active":"2024-03-11T21:00:48.466102+0000","last_became_peered":"2024-03-11T21:00:48.466102+0000","last_unstale":"2024-03-11T21:04:24.433201+0000","last_undegraded":"2024-03-11T21:04:24.433201+0000","last_fullsized":"2024-03-11T21:04:24.433201+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:20.066579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:52:09.043655+0000","last_clean_scrub_stamp":"2024-03-11T21:03:20.066579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:28.105514+0000","scrub_duration":147,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"207'18","reported_seq":409,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.427454+0000","last_change":"2024-03-11T21:02:10.604467+0000","last_active":"2024-03-11T21:04:24.427454+0000","last_peered":"2024-03-11T21:04:24.427454+0000","last_clean":"2024-03-11T21:04:24.427454+0000","last_became_active":"2024-03-11T21:00:27.967624+0000","last_became_peered":"2024-03-11T21:00:27.967624+0000","last_unstale":"2024-03-11T21:04:24.427454+0000","last_undegraded":"2024-03-11T21:04:24.427454+0000","last_fullsized":"2024-03-11T21:04:24.427454+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'18","last_scrub_stamp":"2024-03-11T21:02:10.604327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:02:10.604327+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"277'6568","reported_seq":6630,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:26.172990+0000","last_change":"2024-03-11T21:02:47.641116+0000","last_active":"2024-03-11T21:04:26.172990+0000","last_peered":"2024-03-11T21:04:26.172990+0000","last_clean":"2024-03-11T21:04:26.172990+0000","last_became_active":"2024-03-11T21:02:47.640263+0000","last_became_peered":"2024-03-11T21:02:47.640263+0000","last_unstale":"2024-03-11T21:04:26.172990+0000","last_undegraded":"2024-03-11T21:04:26.172990+0000","last_fullsized":"2024-03-11T21:04:26.172990+0000","mapping_epoch":251,"log_start":"260'1900","ondisk_log_start":"260'1900","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:02:46.570492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:46.570492+0000","last_clean_scrub_stamp":"2024-03-11T21:02:46.570492+0000","objects_scrubbed":0,"log_size":4668,"log_dups_size":0,"ondisk_log_size":4668,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":430440448,"num_objects":6568,"num_object_clones":0,"num_object_copies":19704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6568,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6568,"num_write_kb":420352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":465,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.444106+0000","last_change":"2024-03-11T21:02:27.775773+0000","last_active":"2024-03-11T21:04:24.444106+0000","last_peered":"2024-03-11T21:04:24.444106+0000","last_clean":"2024-03-11T21:04:24.444106+0000","last_became_active":"2024-03-11T21:00:48.460531+0000","last_became_peered":"2024-03-11T21:00:48.460531+0000","last_unstale":"2024-03-11T21:04:24.444106+0000","last_undegraded":"2024-03-11T21:04:24.444106+0000","last_fullsized":"2024-03-11T21:04:24.444106+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:02:27.775534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:27.775534+0000","last_clean_scrub_stamp":"2024-03-11T21:02:27.775534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:26.440092+0000","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":170,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.454202+0000","last_change":"2024-03-11T21:03:29.454135+0000","last_active":"2024-03-11T21:04:24.454202+0000","last_peered":"2024-03-11T21:04:24.454202+0000","last_clean":"2024-03-11T21:04:24.454202+0000","last_became_active":"2024-03-11T21:00:48.469596+0000","last_became_peered":"2024-03-11T21:00:48.469596+0000","last_unstale":"2024-03-11T21:04:24.454202+0000","last_undegraded":"2024-03-11T21:04:24.454202+0000","last_fullsized":"2024-03-11T21:04:24.454202+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:29.453952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:03:29.453952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:37.218531+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":162,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.444170+0000","last_change":"2024-03-11T21:03:26.422762+0000","last_active":"2024-03-11T21:04:24.444170+0000","last_peered":"2024-03-11T21:04:24.444170+0000","last_clean":"2024-03-11T21:04:24.444170+0000","last_became_active":"2024-03-11T21:00:48.466084+0000","last_became_peered":"2024-03-11T21:00:48.466084+0000","last_unstale":"2024-03-11T21:04:24.444170+0000","last_undegraded":"2024-03-11T21:04:24.444170+0000","last_fullsized":"2024-03-11T21:04:24.444170+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:26.422645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:40.974147+0000","last_clean_scrub_stamp":"2024-03-11T21:03:26.422645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:36.823254+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"207'34","reported_seq":562,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.439683+0000","last_change":"2024-03-11T21:03:13.518790+0000","last_active":"2024-03-11T21:04:24.439683+0000","last_peered":"2024-03-11T21:04:24.439683+0000","last_clean":"2024-03-11T21:04:24.439683+0000","last_became_active":"2024-03-11T21:00:48.472191+0000","last_became_peered":"2024-03-11T21:00:48.472191+0000","last_unstale":"2024-03-11T21:04:24.439683+0000","last_undegraded":"2024-03-11T21:04:24.439683+0000","last_fullsized":"2024-03-11T21:04:24.439683+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'34","last_scrub_stamp":"2024-03-11T21:03:13.518551+0000","last_deep_scrub":"116'22","last_deep_scrub_stamp":"2024-03-11T20:53:47.364709+0000","last_clean_scrub_stamp":"2024-03-11T21:03:13.518551+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:26.117741+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1","version":"277'6598","reported_seq":6660,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:26.148976+0000","last_change":"2024-03-11T21:02:47.642494+0000","last_active":"2024-03-11T21:04:26.148976+0000","last_peered":"2024-03-11T21:04:26.148976+0000","last_clean":"2024-03-11T21:04:26.148976+0000","last_became_active":"2024-03-11T21:02:47.641594+0000","last_became_peered":"2024-03-11T21:02:47.641594+0000","last_unstale":"2024-03-11T21:04:26.148976+0000","last_undegraded":"2024-03-11T21:04:26.148976+0000","last_fullsized":"2024-03-11T21:04:26.148976+0000","mapping_epoch":251,"log_start":"260'1900","ondisk_log_start":"260'1900","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:02:46.570492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:46.570492+0000","last_clean_scrub_stamp":"2024-03-11T21:02:46.570492+0000","objects_scrubbed":0,"log_size":4698,"log_dups_size":0,"ondisk_log_size":4698,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":432406528,"num_objects":6598,"num_object_clones":0,"num_object_copies":19794,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6598,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6598,"num_write_kb":422272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":412,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.428191+0000","last_change":"2024-03-11T21:03:33.263561+0000","last_active":"2024-03-11T21:04:24.428191+0000","last_peered":"2024-03-11T21:04:24.428191+0000","last_clean":"2024-03-11T21:04:24.428191+0000","last_became_active":"2024-03-11T21:00:48.464070+0000","last_became_peered":"2024-03-11T21:00:48.464070+0000","last_unstale":"2024-03-11T21:04:24.428191+0000","last_undegraded":"2024-03-11T21:04:24.428191+0000","last_fullsized":"2024-03-11T21:04:24.428191+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:33.263478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:03:33.263478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:52.150843+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":170,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.453579+0000","last_change":"2024-03-11T21:03:37.601354+0000","last_active":"2024-03-11T21:04:24.453579+0000","last_peered":"2024-03-11T21:04:24.453579+0000","last_clean":"2024-03-11T21:04:24.453579+0000","last_became_active":"2024-03-11T21:00:48.461367+0000","last_became_peered":"2024-03-11T21:00:48.461367+0000","last_unstale":"2024-03-11T21:04:24.453579+0000","last_undegraded":"2024-03-11T21:04:24.453579+0000","last_fullsized":"2024-03-11T21:04:24.453579+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:37.601229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:03:37.601229+0000","last_clean_scrub_stamp":"2024-03-11T21:03:37.601229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:50.353048+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"267'19","reported_seq":490,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.439351+0000","last_change":"2024-03-11T21:03:39.673560+0000","last_active":"2024-03-11T21:04:24.439351+0000","last_peered":"2024-03-11T21:04:24.439351+0000","last_clean":"2024-03-11T21:04:24.439351+0000","last_became_active":"2024-03-11T21:00:27.967572+0000","last_became_peered":"2024-03-11T21:00:27.967572+0000","last_unstale":"2024-03-11T21:04:24.439351+0000","last_undegraded":"2024-03-11T21:04:24.439351+0000","last_fullsized":"2024-03-11T21:04:24.439351+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"267'19","last_scrub_stamp":"2024-03-11T21:03:39.673317+0000","last_deep_scrub":"267'19","last_deep_scrub_stamp":"2024-03-11T21:03:39.673317+0000","last_clean_scrub_stamp":"2024-03-11T21:03:39.673317+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:48.038895+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":419,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.427652+0000","last_change":"2024-03-11T21:02:31.471685+0000","last_active":"2024-03-11T21:04:24.427652+0000","last_peered":"2024-03-11T21:04:24.427652+0000","last_clean":"2024-03-11T21:04:24.427652+0000","last_became_active":"2024-03-11T21:00:27.958537+0000","last_became_peered":"2024-03-11T21:00:27.958537+0000","last_unstale":"2024-03-11T21:04:24.427652+0000","last_undegraded":"2024-03-11T21:04:24.427652+0000","last_fullsized":"2024-03-11T21:04:24.427652+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:02:31.471513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:35.670149+0000","last_clean_scrub_stamp":"2024-03-11T21:02:31.471513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"277'6675","reported_seq":6737,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:26.905666+0000","last_change":"2024-03-11T21:02:47.629420+0000","last_active":"2024-03-11T21:04:26.905666+0000","last_peered":"2024-03-11T21:04:26.905666+0000","last_clean":"2024-03-11T21:04:26.905666+0000","last_became_active":"2024-03-11T21:02:47.628916+0000","last_became_peered":"2024-03-11T21:02:47.628916+0000","last_unstale":"2024-03-11T21:04:26.905666+0000","last_undegraded":"2024-03-11T21:04:26.905666+0000","last_fullsized":"2024-03-11T21:04:26.905666+0000","mapping_epoch":251,"log_start":"259'1500","ondisk_log_start":"259'1500","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:02:46.570492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:46.570492+0000","last_clean_scrub_stamp":"2024-03-11T21:02:46.570492+0000","objects_scrubbed":0,"log_size":5175,"log_dups_size":0,"ondisk_log_size":5175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:30.649125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":437452800,"num_objects":6675,"num_object_clones":0,"num_object_copies":20025,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6675,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6675,"num_write_kb":427200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":468,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.444186+0000","last_change":"2024-03-11T21:02:33.768355+0000","last_active":"2024-03-11T21:04:24.444186+0000","last_peered":"2024-03-11T21:04:24.444186+0000","last_clean":"2024-03-11T21:04:24.444186+0000","last_became_active":"2024-03-11T21:00:27.959830+0000","last_became_peered":"2024-03-11T21:00:27.959830+0000","last_unstale":"2024-03-11T21:04:24.444186+0000","last_undegraded":"2024-03-11T21:04:24.444186+0000","last_fullsized":"2024-03-11T21:04:24.444186+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:02:33.767909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:02:33.767909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"249'19","reported_seq":599,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.427801+0000","last_change":"2024-03-11T21:03:23.347322+0000","last_active":"2024-03-11T21:04:24.427801+0000","last_peered":"2024-03-11T21:04:24.427801+0000","last_clean":"2024-03-11T21:04:24.427801+0000","last_became_active":"2024-03-11T21:02:37.805125+0000","last_became_peered":"2024-03-11T21:02:37.805125+0000","last_unstale":"2024-03-11T21:04:24.427801+0000","last_undegraded":"2024-03-11T21:04:24.427801+0000","last_fullsized":"2024-03-11T21:04:24.427801+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"249'19","last_scrub_stamp":"2024-03-11T21:03:23.347230+0000","last_deep_scrub":"146'14","last_deep_scrub_stamp":"2024-03-11T20:56:00.242211+0000","last_clean_scrub_stamp":"2024-03-11T21:03:23.347230+0000","objects_scrubbed":1,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:35.354233+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":472,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.433638+0000","last_change":"2024-03-11T21:03:50.901480+0000","last_active":"2024-03-11T21:04:24.433638+0000","last_peered":"2024-03-11T21:04:24.433638+0000","last_clean":"2024-03-11T21:04:24.433638+0000","last_became_active":"2024-03-11T21:00:27.970573+0000","last_became_peered":"2024-03-11T21:00:27.970573+0000","last_unstale":"2024-03-11T21:04:24.433638+0000","last_undegraded":"2024-03-11T21:04:24.433638+0000","last_fullsized":"2024-03-11T21:04:24.433638+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:50.901398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:52:51.161281+0000","last_clean_scrub_stamp":"2024-03-11T21:03:50.901398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:05:13.889962+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"207'44","reported_seq":607,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.443671+0000","last_change":"2024-03-11T21:03:06.547457+0000","last_active":"2024-03-11T21:04:24.443671+0000","last_peered":"2024-03-11T21:04:24.443671+0000","last_clean":"2024-03-11T21:04:24.443671+0000","last_became_active":"2024-03-11T21:00:48.469854+0000","last_became_peered":"2024-03-11T21:00:48.469854+0000","last_unstale":"2024-03-11T21:04:24.443671+0000","last_undegraded":"2024-03-11T21:04:24.443671+0000","last_fullsized":"2024-03-11T21:04:24.443671+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'44","last_scrub_stamp":"2024-03-11T21:03:06.547254+0000","last_deep_scrub":"161'28","last_deep_scrub_stamp":"2024-03-11T20:57:20.816278+0000","last_clean_scrub_stamp":"2024-03-11T21:03:06.547254+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:24.665975+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.7","version":"277'6597","reported_seq":6659,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:25.239510+0000","last_change":"2024-03-11T21:02:47.637634+0000","last_active":"2024-03-11T21:04:25.239510+0000","last_peered":"2024-03-11T21:04:25.239510+0000","last_clean":"2024-03-11T21:04:25.239510+0000","last_became_active":"2024-03-11T21:02:47.637110+0000","last_became_peered":"2024-03-11T21:02:47.637110+0000","last_unstale":"2024-03-11T21:04:25.239510+0000","last_undegraded":"2024-03-11T21:04:25.239510+0000","last_fullsized":"2024-03-11T21:04:25.239510+0000","mapping_epoch":251,"log_start":"259'1200","ondisk_log_start":"259'1200","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:02:46.570492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:46.570492+0000","last_clean_scrub_stamp":"2024-03-11T21:02:46.570492+0000","objects_scrubbed":0,"log_size":5397,"log_dups_size":0,"ondisk_log_size":5397,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:26.440205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":432340992,"num_objects":6597,"num_object_clones":0,"num_object_copies":19791,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6597,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6597,"num_write_kb":422208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":451,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.432855+0000","last_change":"2024-03-11T21:03:17.004069+0000","last_active":"2024-03-11T21:04:24.432855+0000","last_peered":"2024-03-11T21:04:24.432855+0000","last_clean":"2024-03-11T21:04:24.432855+0000","last_became_active":"2024-03-11T20:50:34.327139+0000","last_became_peered":"2024-03-11T20:50:34.327139+0000","last_unstale":"2024-03-11T21:04:24.432855+0000","last_undegraded":"2024-03-11T21:04:24.432855+0000","last_fullsized":"2024-03-11T21:04:24.432855+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:17.003975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:03:17.003975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:29.951662+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":479,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.444079+0000","last_change":"2024-03-11T21:03:15.683200+0000","last_active":"2024-03-11T21:04:24.444079+0000","last_peered":"2024-03-11T21:04:24.444079+0000","last_clean":"2024-03-11T21:04:24.444079+0000","last_became_active":"2024-03-11T20:50:28.049901+0000","last_became_peered":"2024-03-11T20:50:28.049901+0000","last_unstale":"2024-03-11T21:04:24.444079+0000","last_undegraded":"2024-03-11T21:04:24.444079+0000","last_fullsized":"2024-03-11T21:04:24.444079+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:15.683076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:57:17.447191+0000","last_clean_scrub_stamp":"2024-03-11T21:03:15.683076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"207'39","reported_seq":524,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.438025+0000","last_change":"2024-03-11T21:03:14.082722+0000","last_active":"2024-03-11T21:04:24.438025+0000","last_peered":"2024-03-11T21:04:24.438025+0000","last_clean":"2024-03-11T21:04:24.438025+0000","last_became_active":"2024-03-11T20:50:30.111819+0000","last_became_peered":"2024-03-11T20:50:30.111819+0000","last_unstale":"2024-03-11T21:04:24.438025+0000","last_undegraded":"2024-03-11T21:04:24.438025+0000","last_fullsized":"2024-03-11T21:04:24.438025+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'39","last_scrub_stamp":"2024-03-11T21:03:14.082576+0000","last_deep_scrub":"102'25","last_deep_scrub_stamp":"2024-03-11T20:51:35.187094+0000","last_clean_scrub_stamp":"2024-03-11T21:03:14.082576+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:43.746045+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":475,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.443369+0000","last_change":"2024-03-11T21:03:10.523931+0000","last_active":"2024-03-11T21:04:24.443369+0000","last_peered":"2024-03-11T21:04:24.443369+0000","last_clean":"2024-03-11T21:04:24.443369+0000","last_became_active":"2024-03-11T21:00:48.362389+0000","last_became_peered":"2024-03-11T21:00:48.362389+0000","last_unstale":"2024-03-11T21:04:24.443369+0000","last_undegraded":"2024-03-11T21:04:24.443369+0000","last_fullsized":"2024-03-11T21:04:24.443369+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:10.523794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:03:10.523794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:36.694395+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.6","version":"277'6531","reported_seq":6592,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:25.263658+0000","last_change":"2024-03-11T21:02:47.633710+0000","last_active":"2024-03-11T21:04:25.263658+0000","last_peered":"2024-03-11T21:04:25.263658+0000","last_clean":"2024-03-11T21:04:25.263658+0000","last_became_active":"2024-03-11T21:02:47.633198+0000","last_became_peered":"2024-03-11T21:02:47.633198+0000","last_unstale":"2024-03-11T21:04:25.263658+0000","last_undegraded":"2024-03-11T21:04:25.263658+0000","last_fullsized":"2024-03-11T21:04:25.263658+0000","mapping_epoch":251,"log_start":"259'1100","ondisk_log_start":"259'1100","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:02:46.570492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:46.570492+0000","last_clean_scrub_stamp":"2024-03-11T21:02:46.570492+0000","objects_scrubbed":0,"log_size":5431,"log_dups_size":0,"ondisk_log_size":5431,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:26.440241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":428015616,"num_objects":6531,"num_object_clones":0,"num_object_copies":19593,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6531,"num_write_kb":417984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":271,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.433320+0000","last_change":"2024-03-11T21:03:11.086739+0000","last_active":"2024-03-11T21:04:24.433320+0000","last_peered":"2024-03-11T21:04:24.433320+0000","last_clean":"2024-03-11T21:04:24.433320+0000","last_became_active":"2024-03-11T20:56:59.801112+0000","last_became_peered":"2024-03-11T20:56:59.801112+0000","last_unstale":"2024-03-11T21:04:24.433320+0000","last_undegraded":"2024-03-11T21:04:24.433320+0000","last_fullsized":"2024-03-11T21:04:24.433320+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:11.086661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:55:58.888890+0000","last_clean_scrub_stamp":"2024-03-11T21:03:11.086661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:28.112468+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":162,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.453558+0000","last_change":"2024-03-11T21:03:17.298880+0000","last_active":"2024-03-11T21:04:24.453558+0000","last_peered":"2024-03-11T21:04:24.453558+0000","last_clean":"2024-03-11T21:04:24.453558+0000","last_became_active":"2024-03-11T21:00:48.469015+0000","last_became_peered":"2024-03-11T21:00:48.469015+0000","last_unstale":"2024-03-11T21:04:24.453558+0000","last_undegraded":"2024-03-11T21:04:24.453558+0000","last_fullsized":"2024-03-11T21:04:24.453558+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:17.298802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:03:17.298802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:28.651280+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"207'20","reported_seq":490,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.440031+0000","last_change":"2024-03-11T21:03:41.904778+0000","last_active":"2024-03-11T21:04:24.440031+0000","last_peered":"2024-03-11T21:04:24.440031+0000","last_clean":"2024-03-11T21:04:24.440031+0000","last_became_active":"2024-03-11T20:50:30.109549+0000","last_became_peered":"2024-03-11T20:50:30.109549+0000","last_unstale":"2024-03-11T21:04:24.440031+0000","last_undegraded":"2024-03-11T21:04:24.440031+0000","last_fullsized":"2024-03-11T21:04:24.440031+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'20","last_scrub_stamp":"2024-03-11T21:03:41.904604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:03:41.904604+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:50.209216+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":493,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.446899+0000","last_change":"2024-03-11T21:03:39.413187+0000","last_active":"2024-03-11T21:04:24.446899+0000","last_peered":"2024-03-11T21:04:24.446899+0000","last_clean":"2024-03-11T21:04:24.446899+0000","last_became_active":"2024-03-11T21:02:37.517748+0000","last_became_peered":"2024-03-11T21:02:37.517748+0000","last_unstale":"2024-03-11T21:04:24.446899+0000","last_undegraded":"2024-03-11T21:04:24.446899+0000","last_fullsized":"2024-03-11T21:04:24.446899+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:39.412996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:52:51.931295+0000","last_clean_scrub_stamp":"2024-03-11T21:03:39.412996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:05:06.183543+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.5","version":"277'6439","reported_seq":6501,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:25.264538+0000","last_change":"2024-03-11T21:02:47.619123+0000","last_active":"2024-03-11T21:04:25.264538+0000","last_peered":"2024-03-11T21:04:25.264538+0000","last_clean":"2024-03-11T21:04:25.264538+0000","last_became_active":"2024-03-11T21:02:47.618756+0000","last_became_peered":"2024-03-11T21:02:47.618756+0000","last_unstale":"2024-03-11T21:04:25.264538+0000","last_undegraded":"2024-03-11T21:04:25.264538+0000","last_fullsized":"2024-03-11T21:04:25.264538+0000","mapping_epoch":251,"log_start":"259'1000","ondisk_log_start":"259'1000","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:02:46.570492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:46.570492+0000","last_clean_scrub_stamp":"2024-03-11T21:02:46.570492+0000","objects_scrubbed":0,"log_size":5439,"log_dups_size":0,"ondisk_log_size":5439,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:26.440258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":421986304,"num_objects":6439,"num_object_clones":0,"num_object_copies":19317,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6439,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6439,"num_write_kb":412096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":460,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.440220+0000","last_change":"2024-03-11T21:03:12.778847+0000","last_active":"2024-03-11T21:04:24.440220+0000","last_peered":"2024-03-11T21:04:24.440220+0000","last_clean":"2024-03-11T21:04:24.440220+0000","last_became_active":"2024-03-11T20:50:34.337173+0000","last_became_peered":"2024-03-11T20:50:34.337173+0000","last_unstale":"2024-03-11T21:04:24.440220+0000","last_undegraded":"2024-03-11T21:04:24.440220+0000","last_fullsized":"2024-03-11T21:04:24.440220+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:12.778718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:00:14.030599+0000","last_clean_scrub_stamp":"2024-03-11T21:03:12.778718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:26.440328+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":471,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.444550+0000","last_change":"2024-03-11T21:03:12.702740+0000","last_active":"2024-03-11T21:04:24.444550+0000","last_peered":"2024-03-11T21:04:24.444550+0000","last_clean":"2024-03-11T21:04:24.444550+0000","last_became_active":"2024-03-11T20:50:28.058942+0000","last_became_peered":"2024-03-11T20:50:28.058942+0000","last_unstale":"2024-03-11T21:04:24.444550+0000","last_undegraded":"2024-03-11T21:04:24.444550+0000","last_fullsized":"2024-03-11T21:04:24.444550+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:12.702463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:03:12.702463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:30.975532+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"207'27","reported_seq":511,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.433459+0000","last_change":"2024-03-11T21:03:48.964872+0000","last_active":"2024-03-11T21:04:24.433459+0000","last_peered":"2024-03-11T21:04:24.433459+0000","last_clean":"2024-03-11T21:04:24.433459+0000","last_became_active":"2024-03-11T21:00:27.963119+0000","last_became_peered":"2024-03-11T21:00:27.963119+0000","last_unstale":"2024-03-11T21:04:24.433459+0000","last_undegraded":"2024-03-11T21:04:24.433459+0000","last_fullsized":"2024-03-11T21:04:24.433459+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'27","last_scrub_stamp":"2024-03-11T21:03:48.964758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:03:48.964758+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:05:11.347970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":456,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-11T21:04:24.439415+0000","last_change":"2024-03-11T21:03:44.899203+0000","last_active":"2024-03-11T21:04:24.439415+0000","last_peered":"2024-03-11T21:04:24.439415+0000","last_clean":"2024-03-11T21:04:24.439415+0000","last_became_active":"2024-03-11T20:50:32.652854+0000","last_became_peered":"2024-03-11T20:50:32.652854+0000","last_unstale":"2024-03-11T21:04:24.439415+0000","last_undegraded":"2024-03-11T21:04:24.439415+0000","last_fullsized":"2024-03-11T21:04:24.439415+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:44.899076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:59:00.946094+0000","last_clean_scrub_stamp":"2024-03-11T21:03:44.899076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:48.984405+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":6859325440,"num_objects":104665,"num_object_clones":0,"num_object_copies":313995,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":104665,"num_write_kb":6698560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20614742016,"data_stored":20614742016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80865,"ondisk_log_size":80865,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":20,"num_read_kb":20,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1822,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1127,"num_read_kb":884,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":174,"num_bytes_recovered":2228,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3632,"internal_metadata":0},"log_size":773,"ondisk_log_size":773,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":121,"ondisk_log_size":121,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":702,"num_read_kb":697,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":68,"seq":292057776306,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4040400,"kb_used_data":3724552,"kb_used_omap":1,"kb_used_meta":315774,"kb_avail":89704752,"statfs":{"total":95995035648,"available":91857666048,"internally_reserved":0,"allocated":3813941248,"data_stored":3813466689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":323353034},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":57,"seq":244813136060,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1515472,"kb_used_data":1245328,"kb_used_omap":1,"kb_used_meta":269950,"kb_avail":92229680,"statfs":{"total":95995035648,"available":94443192320,"internally_reserved":0,"allocated":1275215872,"data_stored":1274815978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":276429255},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":211,"seq":906238099501,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3998364,"kb_used_data":3813524,"kb_used_omap":7,"kb_used_meta":184824,"kb_avail":89746788,"statfs":{"total":95995035648,"available":91900710912,"internally_reserved":0,"allocated":3905048576,"data_stored":3904644288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8078,"internal_metadata":189259890},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738618,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1935896,"kb_used_data":1656024,"kb_used_omap":1,"kb_used_meta":279678,"kb_avail":91809256,"statfs":{"total":95995035648,"available":94012678144,"internally_reserved":0,"allocated":1695768576,"data_stored":1695359422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":286390725},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378853,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1530180,"kb_used_data":1250308,"kb_used_omap":1,"kb_used_meta":279678,"kb_avail":92214972,"statfs":{"total":95995035648,"available":94428131328,"internally_reserved":0,"allocated":1280315392,"data_stored":1279926336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":286390725},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":28,"seq":120259084506,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3774224,"kb_used_data":3373768,"kb_used_omap":1,"kb_used_meta":400446,"kb_avail":89970928,"statfs":{"total":95995035648,"available":92130230272,"internally_reserved":0,"allocated":3454738432,"data_stored":3454345656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":410057160},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,2,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":204,"seq":876173328433,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1371500,"kb_used_data":1262572,"kb_used_omap":4,"kb_used_meta":108731,"kb_avail":92373652,"statfs":{"total":95995035648,"available":94590619648,"internally_reserved":0,"allocated":1292873728,"data_stored":1292445478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4451,"internal_metadata":111341213},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":44,"seq":188978561222,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3999160,"kb_used_data":3706992,"kb_used_omap":1,"kb_used_meta":292158,"kb_avail":89745992,"statfs":{"total":95995035648,"available":91899895808,"internally_reserved":0,"allocated":3795959808,"data_stored":3795508307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":299170245},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3632,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1704329216,"data_stored":1704329216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3912564736,"data_stored":3912564736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1287782400,"data_stored":1287782400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3454599168,"data_stored":3454599168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1299382272,"data_stored":1299382272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3827367936,"data_stored":3827367936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1281097728,"data_stored":1281097728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":3847618560,"data_stored":3847618560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T21:04:32.792 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-03-11T21:04:32.793 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-11T21:04:32.793 INFO:tasks.thrashosds.thrasher:Shrinking pool unique_pool_2 2024-03-11T21:04:32.793 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool unique_pool_2 by 10 min 8 2024-03-11T21:04:32.793 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T21:04:32.794 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:04:32.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:32.796 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:04:32.897 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:32.947 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:32.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:32.953 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:04:33.054 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:33.082 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:33.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:33.087 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:04:33.188 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:04:33.211 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:04:33.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:33.214 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:04:33.253 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:04:33.315 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:04:33.340 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:04:33.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:33.343 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:04:33.444 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:04:33.465 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:04:33.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:33.467 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:04:33.568 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:33.577 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:33 smithi146 bash[23809]: cluster 2024-03-11T21:04:31.652950+0000 osd.6 (osd.6) 204 : cluster 0 2.1a scrub starts 2024-03-11T21:04:33.577 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:33 smithi146 bash[23809]: cluster 2024-03-11T21:04:31.654827+0000 osd.6 (osd.6) 205 : cluster 0 2.1a scrub ok 2024-03-11T21:04:33.578 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:33 smithi146 bash[23809]: cluster 2024-03-11T21:04:32.180647+0000 mgr.y (mgr.24380) 510 : cluster 0 pgmap v570: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.3 KiB/s rd, 72 MiB/s wr, 1.15k op/s 2024-03-11T21:04:33.590 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:33.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:33.593 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:04:33.694 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:04:33.703 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:33 smithi023 bash[21372]: cluster 2024-03-11T21:04:31.652950+0000 osd.6 (osd.6) 204 : cluster 0 2.1a scrub starts 2024-03-11T21:04:33.704 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:33 smithi023 bash[21372]: cluster 2024-03-11T21:04:31.654827+0000 osd.6 (osd.6) 205 : cluster 0 2.1a scrub ok 2024-03-11T21:04:33.704 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:33 smithi023 bash[21372]: cluster 2024-03-11T21:04:32.180647+0000 mgr.y (mgr.24380) 510 : cluster 0 pgmap v570: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.3 KiB/s rd, 72 MiB/s wr, 1.15k op/s 2024-03-11T21:04:33.704 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:33 smithi023 bash[29146]: cluster 2024-03-11T21:04:31.652950+0000 osd.6 (osd.6) 204 : cluster 0 2.1a scrub starts 2024-03-11T21:04:33.704 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:33 smithi023 bash[29146]: cluster 2024-03-11T21:04:31.654827+0000 osd.6 (osd.6) 205 : cluster 0 2.1a scrub ok 2024-03-11T21:04:33.704 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:33 smithi023 bash[29146]: cluster 2024-03-11T21:04:32.180647+0000 mgr.y (mgr.24380) 510 : cluster 0 pgmap v570: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.3 KiB/s rd, 72 MiB/s wr, 1.15k op/s 2024-03-11T21:04:33.719 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:04:33.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:33.721 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:04:33.822 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:04:33.845 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:04:33.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:33.848 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:04:33.949 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:04:33.987 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:04:33.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:33.990 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:04:34.091 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:04:34.113 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:04:34.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:34.116 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:04:34.216 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:34.242 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:34.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:34.245 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:04:34.346 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:34.383 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:34.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:34.385 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:04:34.486 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:04:34.513 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:04:34.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:34.517 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:04:34.618 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:04:34.649 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:04:34.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:34.653 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:04:34.754 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:34.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:34 smithi146 bash[23809]: cluster 2024-03-11T21:04:33.469595+0000 osd.4 (osd.4) 77 : cluster 0 5.1 scrub starts 2024-03-11T21:04:34.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:34 smithi146 bash[23809]: cluster 2024-03-11T21:04:33.475045+0000 osd.4 (osd.4) 78 : cluster 0 5.1 scrub ok 2024-03-11T21:04:34.764 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-11T21:04:34.782 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:34 smithi023 bash[21372]: cluster 2024-03-11T21:04:33.469595+0000 osd.4 (osd.4) 77 : cluster 0 5.1 scrub starts 2024-03-11T21:04:34.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:34 smithi023 bash[21372]: cluster 2024-03-11T21:04:33.475045+0000 osd.4 (osd.4) 78 : cluster 0 5.1 scrub ok 2024-03-11T21:04:34.783 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:34 smithi023 bash[29146]: cluster 2024-03-11T21:04:33.469595+0000 osd.4 (osd.4) 77 : cluster 0 5.1 scrub starts 2024-03-11T21:04:34.783 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:34 smithi023 bash[29146]: cluster 2024-03-11T21:04:33.475045+0000 osd.4 (osd.4) 78 : cluster 0 5.1 scrub ok 2024-03-11T21:04:34.784 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:34.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:34.789 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:04:34.859 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T21:04:34.890 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:34.945 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:34.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:34.947 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:04:35.048 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:04:35.078 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:04:35.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:35.080 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:04:35.181 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:35.206 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:35.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:35.208 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:04:35.309 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:04:35.339 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:04:35.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:35.343 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:04:35.444 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:35.464 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:35.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:35.466 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:04:35.567 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:04:35.576 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:04:35 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:04:35.576 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:35 smithi146 bash[23809]: cluster 2024-03-11T21:04:34.008355+0000 osd.2 (osd.2) 183 : cluster 0 6.f scrub starts 2024-03-11T21:04:35.576 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:35 smithi146 bash[23809]: cluster 2024-03-11T21:04:34.115941+0000 osd.2 (osd.2) 184 : cluster 0 6.f scrub ok 2024-03-11T21:04:35.577 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:35 smithi146 bash[23809]: cluster 2024-03-11T21:04:34.181072+0000 mgr.y (mgr.24380) 511 : cluster 0 pgmap v571: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 49 MiB/s wr, 788 op/s 2024-03-11T21:04:35.577 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:35 smithi146 bash[23809]: audit 2024-03-11T21:04:35.334748+0000 mon.a (mon.0) 1514 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:04:35.590 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:04:35.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:35.592 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:04:35.692 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:04:35.717 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:04:35.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:35.721 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:04:35.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:35 smithi023 bash[21372]: cluster 2024-03-11T21:04:34.008355+0000 osd.2 (osd.2) 183 : cluster 0 6.f scrub starts 2024-03-11T21:04:35.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:35 smithi023 bash[21372]: cluster 2024-03-11T21:04:34.115941+0000 osd.2 (osd.2) 184 : cluster 0 6.f scrub ok 2024-03-11T21:04:35.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:35 smithi023 bash[21372]: cluster 2024-03-11T21:04:34.181072+0000 mgr.y (mgr.24380) 511 : cluster 0 pgmap v571: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 49 MiB/s wr, 788 op/s 2024-03-11T21:04:35.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:35 smithi023 bash[21372]: audit 2024-03-11T21:04:35.334748+0000 mon.a (mon.0) 1514 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:04:35.783 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:35 smithi023 bash[29146]: cluster 2024-03-11T21:04:34.008355+0000 osd.2 (osd.2) 183 : cluster 0 6.f scrub starts 2024-03-11T21:04:35.783 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:35 smithi023 bash[29146]: cluster 2024-03-11T21:04:34.115941+0000 osd.2 (osd.2) 184 : cluster 0 6.f scrub ok 2024-03-11T21:04:35.784 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:35 smithi023 bash[29146]: cluster 2024-03-11T21:04:34.181072+0000 mgr.y (mgr.24380) 511 : cluster 0 pgmap v571: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 49 MiB/s wr, 788 op/s 2024-03-11T21:04:35.784 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:35 smithi023 bash[29146]: audit 2024-03-11T21:04:35.334748+0000 mon.a (mon.0) 1514 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:04:35.821 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:35.848 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:35.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:35.851 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:04:35.952 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:35.980 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:35.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:35.983 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:04:36.084 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:36.109 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:36.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:36.111 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:04:36.212 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:04:36.238 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:04:36.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:36.242 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:04:36.342 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:04:36.368 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:04:36.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:36.373 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:04:36.473 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:04:36.504 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:04:36.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:36.507 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:04:36.607 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:04:36.637 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:04:36.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:36.640 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:04:36.741 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:04:36.750 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:36 smithi146 bash[23809]: cluster 2024-03-11T21:04:34.997517+0000 osd.2 (osd.2) 185 : cluster 0 4.16 scrub starts 2024-03-11T21:04:36.751 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:36 smithi146 bash[23809]: cluster 2024-03-11T21:04:35.004844+0000 osd.2 (osd.2) 186 : cluster 0 4.16 scrub ok 2024-03-11T21:04:36.751 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:36 smithi146 bash[23809]: cluster 2024-03-11T21:04:35.478532+0000 osd.5 (osd.5) 249 : cluster 0 3.16 scrub starts 2024-03-11T21:04:36.751 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:36 smithi146 bash[23809]: cluster 2024-03-11T21:04:35.480674+0000 osd.5 (osd.5) 250 : cluster 0 3.16 scrub ok 2024-03-11T21:04:36.751 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:36 smithi146 bash[23809]: audit 2024-03-11T21:04:35.504397+0000 mgr.y (mgr.24380) 512 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:04:36.751 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:36 smithi146 bash[23809]: cluster 2024-03-11T21:04:35.602820+0000 osd.1 (osd.1) 57 : cluster 0 6.4 scrub starts 2024-03-11T21:04:36.751 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:36 smithi146 bash[23809]: cluster 2024-03-11T21:04:35.604153+0000 osd.1 (osd.1) 58 : cluster 0 6.4 scrub ok 2024-03-11T21:04:36.751 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:36 smithi146 bash[23809]: cluster 2024-03-11T21:04:35.634662+0000 osd.4 (osd.4) 79 : cluster 0 3.9 scrub starts 2024-03-11T21:04:36.751 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:36 smithi146 bash[23809]: cluster 2024-03-11T21:04:35.636776+0000 osd.4 (osd.4) 80 : cluster 0 3.9 scrub ok 2024-03-11T21:04:36.751 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:36 smithi146 bash[23809]: cluster 2024-03-11T21:04:36.185628+0000 osd.0 (osd.0) 199 : cluster 0 3.1b scrub starts 2024-03-11T21:04:36.751 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:36 smithi146 bash[23809]: cluster 2024-03-11T21:04:36.187757+0000 osd.0 (osd.0) 200 : cluster 0 3.1b scrub ok 2024-03-11T21:04:36.751 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:36 smithi146 bash[23809]: cluster 2024-03-11T21:04:36.200381+0000 osd.7 (osd.7) 285 : cluster 0 2.5 scrub starts 2024-03-11T21:04:36.751 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:36 smithi146 bash[23809]: cluster 2024-03-11T21:04:36.201825+0000 osd.7 (osd.7) 286 : cluster 0 2.5 scrub ok 2024-03-11T21:04:36.779 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:04:36.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:36.781 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:04:36.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:36 smithi023 bash[21372]: cluster 2024-03-11T21:04:34.997517+0000 osd.2 (osd.2) 185 : cluster 0 4.16 scrub starts 2024-03-11T21:04:36.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:36 smithi023 bash[21372]: cluster 2024-03-11T21:04:35.004844+0000 osd.2 (osd.2) 186 : cluster 0 4.16 scrub ok 2024-03-11T21:04:36.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:36 smithi023 bash[21372]: cluster 2024-03-11T21:04:35.478532+0000 osd.5 (osd.5) 249 : cluster 0 3.16 scrub starts 2024-03-11T21:04:36.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:36 smithi023 bash[21372]: cluster 2024-03-11T21:04:35.480674+0000 osd.5 (osd.5) 250 : cluster 0 3.16 scrub ok 2024-03-11T21:04:36.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:36 smithi023 bash[21372]: audit 2024-03-11T21:04:35.504397+0000 mgr.y (mgr.24380) 512 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:04:36.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:36 smithi023 bash[21372]: cluster 2024-03-11T21:04:35.602820+0000 osd.1 (osd.1) 57 : cluster 0 6.4 scrub starts 2024-03-11T21:04:36.784 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:36 smithi023 bash[21372]: cluster 2024-03-11T21:04:35.604153+0000 osd.1 (osd.1) 58 : cluster 0 6.4 scrub ok 2024-03-11T21:04:36.784 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:36 smithi023 bash[21372]: cluster 2024-03-11T21:04:35.634662+0000 osd.4 (osd.4) 79 : cluster 0 3.9 scrub starts 2024-03-11T21:04:36.784 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:36 smithi023 bash[21372]: cluster 2024-03-11T21:04:35.636776+0000 osd.4 (osd.4) 80 : cluster 0 3.9 scrub ok 2024-03-11T21:04:36.784 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:36 smithi023 bash[21372]: cluster 2024-03-11T21:04:36.185628+0000 osd.0 (osd.0) 199 : cluster 0 3.1b scrub starts 2024-03-11T21:04:36.784 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:36 smithi023 bash[21372]: cluster 2024-03-11T21:04:36.187757+0000 osd.0 (osd.0) 200 : cluster 0 3.1b scrub ok 2024-03-11T21:04:36.784 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:36 smithi023 bash[21372]: cluster 2024-03-11T21:04:36.200381+0000 osd.7 (osd.7) 285 : cluster 0 2.5 scrub starts 2024-03-11T21:04:36.784 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:36 smithi023 bash[21372]: cluster 2024-03-11T21:04:36.201825+0000 osd.7 (osd.7) 286 : cluster 0 2.5 scrub ok 2024-03-11T21:04:36.784 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:36 smithi023 bash[29146]: cluster 2024-03-11T21:04:34.997517+0000 osd.2 (osd.2) 185 : cluster 0 4.16 scrub starts 2024-03-11T21:04:36.784 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:36 smithi023 bash[29146]: cluster 2024-03-11T21:04:35.004844+0000 osd.2 (osd.2) 186 : cluster 0 4.16 scrub ok 2024-03-11T21:04:36.784 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:36 smithi023 bash[29146]: cluster 2024-03-11T21:04:35.478532+0000 osd.5 (osd.5) 249 : cluster 0 3.16 scrub starts 2024-03-11T21:04:36.784 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:36 smithi023 bash[29146]: cluster 2024-03-11T21:04:35.480674+0000 osd.5 (osd.5) 250 : cluster 0 3.16 scrub ok 2024-03-11T21:04:36.784 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:36 smithi023 bash[29146]: audit 2024-03-11T21:04:35.504397+0000 mgr.y (mgr.24380) 512 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:04:36.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:36 smithi023 bash[29146]: cluster 2024-03-11T21:04:35.602820+0000 osd.1 (osd.1) 57 : cluster 0 6.4 scrub starts 2024-03-11T21:04:36.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:36 smithi023 bash[29146]: cluster 2024-03-11T21:04:35.604153+0000 osd.1 (osd.1) 58 : cluster 0 6.4 scrub ok 2024-03-11T21:04:36.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:36 smithi023 bash[29146]: cluster 2024-03-11T21:04:35.634662+0000 osd.4 (osd.4) 79 : cluster 0 3.9 scrub starts 2024-03-11T21:04:36.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:36 smithi023 bash[29146]: cluster 2024-03-11T21:04:35.636776+0000 osd.4 (osd.4) 80 : cluster 0 3.9 scrub ok 2024-03-11T21:04:36.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:36 smithi023 bash[29146]: cluster 2024-03-11T21:04:36.185628+0000 osd.0 (osd.0) 199 : cluster 0 3.1b scrub starts 2024-03-11T21:04:36.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:36 smithi023 bash[29146]: cluster 2024-03-11T21:04:36.187757+0000 osd.0 (osd.0) 200 : cluster 0 3.1b scrub ok 2024-03-11T21:04:36.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:36 smithi023 bash[29146]: cluster 2024-03-11T21:04:36.200381+0000 osd.7 (osd.7) 285 : cluster 0 2.5 scrub starts 2024-03-11T21:04:36.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:36 smithi023 bash[29146]: cluster 2024-03-11T21:04:36.201825+0000 osd.7 (osd.7) 286 : cluster 0 2.5 scrub ok 2024-03-11T21:04:36.883 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:36.910 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:36.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:36.914 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:04:37.015 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:37.039 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:37.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:37.043 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:04:37.144 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:04:37.170 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:04:37.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:37.173 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:04:37.274 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:37.307 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:37.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:37.313 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:04:37.414 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:04:37.446 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:04:37.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:37.448 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:04:37.549 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:37.579 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:37.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:37.584 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:04:37.616 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:04:37.685 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:04:37.694 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:37 smithi146 bash[23809]: cluster 2024-03-11T21:04:35.669454+0000 osd.6 (osd.6) 206 : cluster 0 3.a scrub starts 2024-03-11T21:04:37.695 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:37 smithi146 bash[23809]: cluster 2024-03-11T21:04:35.674457+0000 osd.6 (osd.6) 207 : cluster 0 3.a scrub ok 2024-03-11T21:04:37.695 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:37 smithi146 bash[23809]: cluster 2024-03-11T21:04:36.182667+0000 mgr.y (mgr.24380) 513 : cluster 0 pgmap v572: 148 pgs: 148 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 767 B/s rd, 47 MiB/s wr, 990 op/s 2024-03-11T21:04:37.710 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:04:37.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:37.712 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:04:37.813 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:04:37.823 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:37 smithi023 bash[21372]: cluster 2024-03-11T21:04:35.669454+0000 osd.6 (osd.6) 206 : cluster 0 3.a scrub starts 2024-03-11T21:04:37.823 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:37 smithi023 bash[21372]: cluster 2024-03-11T21:04:35.674457+0000 osd.6 (osd.6) 207 : cluster 0 3.a scrub ok 2024-03-11T21:04:37.823 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:37 smithi023 bash[21372]: cluster 2024-03-11T21:04:36.182667+0000 mgr.y (mgr.24380) 513 : cluster 0 pgmap v572: 148 pgs: 148 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 767 B/s rd, 47 MiB/s wr, 990 op/s 2024-03-11T21:04:37.824 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:37 smithi023 bash[29146]: cluster 2024-03-11T21:04:35.669454+0000 osd.6 (osd.6) 206 : cluster 0 3.a scrub starts 2024-03-11T21:04:37.824 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:37 smithi023 bash[29146]: cluster 2024-03-11T21:04:35.674457+0000 osd.6 (osd.6) 207 : cluster 0 3.a scrub ok 2024-03-11T21:04:37.824 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:37 smithi023 bash[29146]: cluster 2024-03-11T21:04:36.182667+0000 mgr.y (mgr.24380) 513 : cluster 0 pgmap v572: 148 pgs: 148 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 767 B/s rd, 47 MiB/s wr, 990 op/s 2024-03-11T21:04:37.838 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:04:37.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:37.840 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:04:37.941 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:37.967 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:37.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:37.971 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:04:38.071 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:38.098 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:38.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:38.100 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:04:38.200 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:04:38.224 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:04:38.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:38.228 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:04:38.329 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:04:38.352 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:04:38.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:38.359 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:04:38.459 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:38.482 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:38.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:38.484 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:04:38.585 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:04:38.594 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:04:38 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:04:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:04:38.594 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:38 smithi023 bash[29146]: cluster 2024-03-11T21:04:37.517736+0000 osd.5 (osd.5) 251 : cluster 0 6.19 scrub starts 2024-03-11T21:04:38.594 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:38 smithi023 bash[29146]: cluster 2024-03-11T21:04:37.573953+0000 osd.5 (osd.5) 252 : cluster 0 6.19 scrub ok 2024-03-11T21:04:38.594 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:38 smithi023 bash[29146]: cluster 2024-03-11T21:04:38.267900+0000 osd.7 (osd.7) 287 : cluster 0 4.1e scrub starts 2024-03-11T21:04:38.595 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:38 smithi023 bash[29146]: cluster 2024-03-11T21:04:38.282702+0000 osd.7 (osd.7) 288 : cluster 0 4.1e scrub ok 2024-03-11T21:04:38.595 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:38 smithi023 bash[21372]: cluster 2024-03-11T21:04:37.517736+0000 osd.5 (osd.5) 251 : cluster 0 6.19 scrub starts 2024-03-11T21:04:38.595 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:38 smithi023 bash[21372]: cluster 2024-03-11T21:04:37.573953+0000 osd.5 (osd.5) 252 : cluster 0 6.19 scrub ok 2024-03-11T21:04:38.595 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:38 smithi023 bash[21372]: cluster 2024-03-11T21:04:38.267900+0000 osd.7 (osd.7) 287 : cluster 0 4.1e scrub starts 2024-03-11T21:04:38.595 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:38 smithi023 bash[21372]: cluster 2024-03-11T21:04:38.282702+0000 osd.7 (osd.7) 288 : cluster 0 4.1e scrub ok 2024-03-11T21:04:38.616 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:04:38.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:38.619 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:04:38.720 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:38.730 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:38 smithi146 bash[23809]: cluster 2024-03-11T21:04:37.517736+0000 osd.5 (osd.5) 251 : cluster 0 6.19 scrub starts 2024-03-11T21:04:38.730 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:38 smithi146 bash[23809]: cluster 2024-03-11T21:04:37.573953+0000 osd.5 (osd.5) 252 : cluster 0 6.19 scrub ok 2024-03-11T21:04:38.730 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:38 smithi146 bash[23809]: cluster 2024-03-11T21:04:38.267900+0000 osd.7 (osd.7) 287 : cluster 0 4.1e scrub starts 2024-03-11T21:04:38.730 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:38 smithi146 bash[23809]: cluster 2024-03-11T21:04:38.282702+0000 osd.7 (osd.7) 288 : cluster 0 4.1e scrub ok 2024-03-11T21:04:38.749 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:38.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:38.753 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:04:38.854 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:04:38.878 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:04:38.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:38.880 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:04:38.980 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:39.013 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:39.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:39.016 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:04:39.117 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:04:39.151 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:04:39.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:39.155 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:04:39.255 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:39.286 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:39.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:39.290 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:04:39.391 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:04:39.421 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:04:39.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:39.427 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:04:39.528 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:04:39.555 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:04:39.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:39.559 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:04:39.652 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:04:39.653 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:04:39.660 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:04:39.673 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:39 smithi023 bash[29146]: cluster 2024-03-11T21:04:37.622495+0000 osd.6 (osd.6) 208 : cluster 0 4.5 scrub starts 2024-03-11T21:04:39.673 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:39 smithi023 bash[29146]: cluster 2024-03-11T21:04:37.625729+0000 osd.6 (osd.6) 209 : cluster 0 4.5 scrub ok 2024-03-11T21:04:39.673 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:39 smithi023 bash[29146]: cluster 2024-03-11T21:04:37.966213+0000 osd.2 (osd.2) 187 : cluster 0 2.c scrub starts 2024-03-11T21:04:39.673 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:39 smithi023 bash[29146]: cluster 2024-03-11T21:04:37.971864+0000 osd.2 (osd.2) 188 : cluster 0 2.c scrub ok 2024-03-11T21:04:39.673 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:39 smithi023 bash[29146]: cluster 2024-03-11T21:04:38.183304+0000 mgr.y (mgr.24380) 514 : cluster 0 pgmap v573: 148 pgs: 148 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.3 KiB/s rd, 46 MiB/s wr, 1.13k op/s 2024-03-11T21:04:39.673 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:39 smithi023 bash[29146]: cluster 2024-03-11T21:04:38.615596+0000 osd.6 (osd.6) 210 : cluster 0 2.11 scrub starts 2024-03-11T21:04:39.673 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:39 smithi023 bash[29146]: cluster 2024-03-11T21:04:38.617113+0000 osd.6 (osd.6) 211 : cluster 0 2.11 scrub ok 2024-03-11T21:04:39.673 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:39 smithi023 bash[29146]: cluster 2024-03-11T21:04:39.283364+0000 osd.7 (osd.7) 289 : cluster 0 1.0 scrub starts 2024-03-11T21:04:39.674 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:39 smithi023 bash[29146]: cluster 2024-03-11T21:04:39.297377+0000 osd.7 (osd.7) 290 : cluster 0 1.0 scrub ok 2024-03-11T21:04:39.674 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:39 smithi023 bash[21372]: cluster 2024-03-11T21:04:37.622495+0000 osd.6 (osd.6) 208 : cluster 0 4.5 scrub starts 2024-03-11T21:04:39.674 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:39 smithi023 bash[21372]: cluster 2024-03-11T21:04:37.625729+0000 osd.6 (osd.6) 209 : cluster 0 4.5 scrub ok 2024-03-11T21:04:39.674 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:39 smithi023 bash[21372]: cluster 2024-03-11T21:04:37.966213+0000 osd.2 (osd.2) 187 : cluster 0 2.c scrub starts 2024-03-11T21:04:39.674 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:39 smithi023 bash[21372]: cluster 2024-03-11T21:04:37.971864+0000 osd.2 (osd.2) 188 : cluster 0 2.c scrub ok 2024-03-11T21:04:39.674 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:39 smithi023 bash[21372]: cluster 2024-03-11T21:04:38.183304+0000 mgr.y (mgr.24380) 514 : cluster 0 pgmap v573: 148 pgs: 148 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.3 KiB/s rd, 46 MiB/s wr, 1.13k op/s 2024-03-11T21:04:39.674 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:39 smithi023 bash[21372]: cluster 2024-03-11T21:04:38.615596+0000 osd.6 (osd.6) 210 : cluster 0 2.11 scrub starts 2024-03-11T21:04:39.674 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:39 smithi023 bash[21372]: cluster 2024-03-11T21:04:38.617113+0000 osd.6 (osd.6) 211 : cluster 0 2.11 scrub ok 2024-03-11T21:04:39.674 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:39 smithi023 bash[21372]: cluster 2024-03-11T21:04:39.283364+0000 osd.7 (osd.7) 289 : cluster 0 1.0 scrub starts 2024-03-11T21:04:39.674 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:39 smithi023 bash[21372]: cluster 2024-03-11T21:04:39.297377+0000 osd.7 (osd.7) 290 : cluster 0 1.0 scrub ok 2024-03-11T21:04:39.696 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:04:39.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:39.700 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:04:39.801 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:04:39.824 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:04:39.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:39.826 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:04:39.927 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:04:39.938 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:39 smithi146 bash[23809]: cluster 2024-03-11T21:04:37.622495+0000 osd.6 (osd.6) 208 : cluster 0 4.5 scrub starts 2024-03-11T21:04:39.938 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:39 smithi146 bash[23809]: cluster 2024-03-11T21:04:37.625729+0000 osd.6 (osd.6) 209 : cluster 0 4.5 scrub ok 2024-03-11T21:04:39.938 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:39 smithi146 bash[23809]: cluster 2024-03-11T21:04:37.966213+0000 osd.2 (osd.2) 187 : cluster 0 2.c scrub starts 2024-03-11T21:04:39.938 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:39 smithi146 bash[23809]: cluster 2024-03-11T21:04:37.971864+0000 osd.2 (osd.2) 188 : cluster 0 2.c scrub ok 2024-03-11T21:04:39.938 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:39 smithi146 bash[23809]: cluster 2024-03-11T21:04:38.183304+0000 mgr.y (mgr.24380) 514 : cluster 0 pgmap v573: 148 pgs: 148 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.3 KiB/s rd, 46 MiB/s wr, 1.13k op/s 2024-03-11T21:04:39.939 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:39 smithi146 bash[23809]: cluster 2024-03-11T21:04:38.615596+0000 osd.6 (osd.6) 210 : cluster 0 2.11 scrub starts 2024-03-11T21:04:39.939 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:39 smithi146 bash[23809]: cluster 2024-03-11T21:04:38.617113+0000 osd.6 (osd.6) 211 : cluster 0 2.11 scrub ok 2024-03-11T21:04:39.939 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:39 smithi146 bash[23809]: cluster 2024-03-11T21:04:39.283364+0000 osd.7 (osd.7) 289 : cluster 0 1.0 scrub starts 2024-03-11T21:04:39.939 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:39 smithi146 bash[23809]: cluster 2024-03-11T21:04:39.297377+0000 osd.7 (osd.7) 290 : cluster 0 1.0 scrub ok 2024-03-11T21:04:39.957 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:04:39.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:39.961 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:04:40.062 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:04:40.084 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:04:40.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:40.087 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:04:40.188 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:04:40.213 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:04:40.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:40.219 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:04:40.320 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:04:40.353 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:04:40.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:40.358 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:04:40.459 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:40.490 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:40.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:40.497 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:04:40.598 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:04:40.605 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:40 smithi023 bash[21372]: cluster 2024-03-11T21:04:38.984851+0000 osd.2 (osd.2) 189 : cluster 0 2.10 scrub starts 2024-03-11T21:04:40.605 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:40 smithi023 bash[21372]: cluster 2024-03-11T21:04:38.986190+0000 osd.2 (osd.2) 190 : cluster 0 2.10 scrub ok 2024-03-11T21:04:40.605 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:40 smithi023 bash[21372]: cluster 2024-03-11T21:04:39.461953+0000 osd.5 (osd.5) 253 : cluster 0 6.d scrub starts 2024-03-11T21:04:40.605 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:40 smithi023 bash[21372]: cluster 2024-03-11T21:04:39.530822+0000 osd.5 (osd.5) 254 : cluster 0 6.d scrub ok 2024-03-11T21:04:40.605 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:40 smithi023 bash[21372]: cluster 2024-03-11T21:04:40.169602+0000 osd.3 (osd.3) 276 : cluster 0 4.11 scrub starts 2024-03-11T21:04:40.605 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:40 smithi023 bash[21372]: cluster 2024-03-11T21:04:40.177987+0000 osd.3 (osd.3) 277 : cluster 0 4.11 scrub ok 2024-03-11T21:04:40.605 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:40 smithi023 bash[29146]: cluster 2024-03-11T21:04:38.984851+0000 osd.2 (osd.2) 189 : cluster 0 2.10 scrub starts 2024-03-11T21:04:40.606 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:40 smithi023 bash[29146]: cluster 2024-03-11T21:04:38.986190+0000 osd.2 (osd.2) 190 : cluster 0 2.10 scrub ok 2024-03-11T21:04:40.606 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:40 smithi023 bash[29146]: cluster 2024-03-11T21:04:39.461953+0000 osd.5 (osd.5) 253 : cluster 0 6.d scrub starts 2024-03-11T21:04:40.606 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:40 smithi023 bash[29146]: cluster 2024-03-11T21:04:39.530822+0000 osd.5 (osd.5) 254 : cluster 0 6.d scrub ok 2024-03-11T21:04:40.606 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:40 smithi023 bash[29146]: cluster 2024-03-11T21:04:40.169602+0000 osd.3 (osd.3) 276 : cluster 0 4.11 scrub starts 2024-03-11T21:04:40.606 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:40 smithi023 bash[29146]: cluster 2024-03-11T21:04:40.177987+0000 osd.3 (osd.3) 277 : cluster 0 4.11 scrub ok 2024-03-11T21:04:40.615 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:04:40.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:40.617 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:04:40.718 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:40.726 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:40 smithi146 bash[23809]: cluster 2024-03-11T21:04:38.984851+0000 osd.2 (osd.2) 189 : cluster 0 2.10 scrub starts 2024-03-11T21:04:40.726 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:40 smithi146 bash[23809]: cluster 2024-03-11T21:04:38.986190+0000 osd.2 (osd.2) 190 : cluster 0 2.10 scrub ok 2024-03-11T21:04:40.726 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:40 smithi146 bash[23809]: cluster 2024-03-11T21:04:39.461953+0000 osd.5 (osd.5) 253 : cluster 0 6.d scrub starts 2024-03-11T21:04:40.726 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:40 smithi146 bash[23809]: cluster 2024-03-11T21:04:39.530822+0000 osd.5 (osd.5) 254 : cluster 0 6.d scrub ok 2024-03-11T21:04:40.726 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:40 smithi146 bash[23809]: cluster 2024-03-11T21:04:40.169602+0000 osd.3 (osd.3) 276 : cluster 0 4.11 scrub starts 2024-03-11T21:04:40.726 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:40 smithi146 bash[23809]: cluster 2024-03-11T21:04:40.177987+0000 osd.3 (osd.3) 277 : cluster 0 4.11 scrub ok 2024-03-11T21:04:40.740 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:40.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:40.741 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:04:40.842 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:04:40.869 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:04:40.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:40.873 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:04:40.935 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-03-11T21:04:40.935 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:04:40.974 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:04:41.009 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:04:41.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:41.012 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:04:41.112 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:04:41.136 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:04:41.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:41.138 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:04:41.240 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:04:41.270 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:04:41.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:41.273 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:04:41.374 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:04:41.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:41.395 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:04:41.395 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:04:41.496 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:04:41.531 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:04:41.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:41.534 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:04:41.635 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:41.661 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:41.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:41.664 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:04:41.764 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:41.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:41 smithi023 bash[21372]: cluster 2024-03-11T21:04:40.183919+0000 mgr.y (mgr.24380) 515 : cluster 0 pgmap v574: 148 pgs: 148 active+clean; 6.5 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.1 KiB/s rd, 31 MiB/s wr, 1.00k op/s 2024-03-11T21:04:41.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:41 smithi023 bash[21372]: cluster 2024-03-11T21:04:40.518859+0000 osd.1 (osd.1) 59 : cluster 0 4.2 scrub starts 2024-03-11T21:04:41.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:41 smithi023 bash[21372]: cluster 2024-03-11T21:04:40.529453+0000 osd.1 (osd.1) 60 : cluster 0 4.2 scrub ok 2024-03-11T21:04:41.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:41 smithi023 bash[21372]: cluster 2024-03-11T21:04:41.182820+0000 osd.0 (osd.0) 201 : cluster 0 6.10 scrub starts 2024-03-11T21:04:41.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:41 smithi023 bash[21372]: cluster 2024-03-11T21:04:41.212744+0000 osd.0 (osd.0) 202 : cluster 0 6.10 scrub ok 2024-03-11T21:04:41.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:41 smithi023 bash[21372]: cluster 2024-03-11T21:04:41.424333+0000 osd.5 (osd.5) 255 : cluster 0 3.1c scrub starts 2024-03-11T21:04:41.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:41 smithi023 bash[21372]: cluster 2024-03-11T21:04:41.440122+0000 osd.5 (osd.5) 256 : cluster 0 3.1c scrub ok 2024-03-11T21:04:41.783 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:41 smithi023 bash[29146]: cluster 2024-03-11T21:04:40.183919+0000 mgr.y (mgr.24380) 515 : cluster 0 pgmap v574: 148 pgs: 148 active+clean; 6.5 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.1 KiB/s rd, 31 MiB/s wr, 1.00k op/s 2024-03-11T21:04:41.783 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:41 smithi023 bash[29146]: cluster 2024-03-11T21:04:40.518859+0000 osd.1 (osd.1) 59 : cluster 0 4.2 scrub starts 2024-03-11T21:04:41.784 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:41 smithi023 bash[29146]: cluster 2024-03-11T21:04:40.529453+0000 osd.1 (osd.1) 60 : cluster 0 4.2 scrub ok 2024-03-11T21:04:41.784 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:41 smithi023 bash[29146]: cluster 2024-03-11T21:04:41.182820+0000 osd.0 (osd.0) 201 : cluster 0 6.10 scrub starts 2024-03-11T21:04:41.784 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:41 smithi023 bash[29146]: cluster 2024-03-11T21:04:41.212744+0000 osd.0 (osd.0) 202 : cluster 0 6.10 scrub ok 2024-03-11T21:04:41.784 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:41 smithi023 bash[29146]: cluster 2024-03-11T21:04:41.424333+0000 osd.5 (osd.5) 255 : cluster 0 3.1c scrub starts 2024-03-11T21:04:41.784 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:41 smithi023 bash[29146]: cluster 2024-03-11T21:04:41.440122+0000 osd.5 (osd.5) 256 : cluster 0 3.1c scrub ok 2024-03-11T21:04:41.788 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:41.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:41.791 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:04:41.892 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:04:41.920 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:04:41.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:41.925 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:04:42.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:41 smithi146 bash[23809]: cluster 2024-03-11T21:04:40.183919+0000 mgr.y (mgr.24380) 515 : cluster 0 pgmap v574: 148 pgs: 148 active+clean; 6.5 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.1 KiB/s rd, 31 MiB/s wr, 1.00k op/s 2024-03-11T21:04:42.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:41 smithi146 bash[23809]: cluster 2024-03-11T21:04:40.518859+0000 osd.1 (osd.1) 59 : cluster 0 4.2 scrub starts 2024-03-11T21:04:42.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:41 smithi146 bash[23809]: cluster 2024-03-11T21:04:40.529453+0000 osd.1 (osd.1) 60 : cluster 0 4.2 scrub ok 2024-03-11T21:04:42.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:41 smithi146 bash[23809]: cluster 2024-03-11T21:04:41.182820+0000 osd.0 (osd.0) 201 : cluster 0 6.10 scrub starts 2024-03-11T21:04:42.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:41 smithi146 bash[23809]: cluster 2024-03-11T21:04:41.212744+0000 osd.0 (osd.0) 202 : cluster 0 6.10 scrub ok 2024-03-11T21:04:42.013 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:41 smithi146 bash[23809]: cluster 2024-03-11T21:04:41.424333+0000 osd.5 (osd.5) 255 : cluster 0 3.1c scrub starts 2024-03-11T21:04:42.013 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:41 smithi146 bash[23809]: cluster 2024-03-11T21:04:41.440122+0000 osd.5 (osd.5) 256 : cluster 0 3.1c scrub ok 2024-03-11T21:04:42.026 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:42.056 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:42.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:42.060 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:04:42.161 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:42.207 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:42.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:42.213 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:04:42.314 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:04:42.345 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:04:42.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:42.354 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:04:42.368 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-03-11T21:04:42.455 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:04:42.490 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:04:42.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:42.495 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:04:42.596 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:42.635 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:42.637 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:42 smithi146 bash[23809]: cluster 2024-03-11T21:04:41.624604+0000 osd.4 (osd.4) 81 : cluster 0 6.1a scrub starts 2024-03-11T21:04:42.637 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:42 smithi146 bash[23809]: cluster 2024-03-11T21:04:41.638361+0000 osd.4 (osd.4) 82 : cluster 0 6.1a scrub ok 2024-03-11T21:04:42.637 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:42 smithi146 bash[23809]: cluster 2024-03-11T21:04:42.303881+0000 osd.7 (osd.7) 291 : cluster 0 6.5 scrub starts 2024-03-11T21:04:42.638 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:42 smithi146 bash[23809]: cluster 2024-03-11T21:04:42.338708+0000 osd.7 (osd.7) 292 : cluster 0 6.5 scrub ok 2024-03-11T21:04:42.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:42.638 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:04:42.739 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:04:42.765 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:04:42.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:42.770 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:04:42.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:42 smithi023 bash[21372]: cluster 2024-03-11T21:04:41.624604+0000 osd.4 (osd.4) 81 : cluster 0 6.1a scrub starts 2024-03-11T21:04:42.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:42 smithi023 bash[21372]: cluster 2024-03-11T21:04:41.638361+0000 osd.4 (osd.4) 82 : cluster 0 6.1a scrub ok 2024-03-11T21:04:42.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:42 smithi023 bash[21372]: cluster 2024-03-11T21:04:42.303881+0000 osd.7 (osd.7) 291 : cluster 0 6.5 scrub starts 2024-03-11T21:04:42.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:42 smithi023 bash[21372]: cluster 2024-03-11T21:04:42.338708+0000 osd.7 (osd.7) 292 : cluster 0 6.5 scrub ok 2024-03-11T21:04:42.783 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:42 smithi023 bash[29146]: cluster 2024-03-11T21:04:41.624604+0000 osd.4 (osd.4) 81 : cluster 0 6.1a scrub starts 2024-03-11T21:04:42.783 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:42 smithi023 bash[29146]: cluster 2024-03-11T21:04:41.638361+0000 osd.4 (osd.4) 82 : cluster 0 6.1a scrub ok 2024-03-11T21:04:42.783 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:42 smithi023 bash[29146]: cluster 2024-03-11T21:04:42.303881+0000 osd.7 (osd.7) 291 : cluster 0 6.5 scrub starts 2024-03-11T21:04:42.783 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:42 smithi023 bash[29146]: cluster 2024-03-11T21:04:42.338708+0000 osd.7 (osd.7) 292 : cluster 0 6.5 scrub ok 2024-03-11T21:04:42.871 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:42.899 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:42.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:42.902 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:04:43.003 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:43.025 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:43.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:43.027 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:04:43.128 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:43.150 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:43.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:43.154 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:04:43.255 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:43.285 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:43.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:43.288 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:04:43.389 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:04:43.415 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:04:43.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:43.419 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:04:43.520 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:43.545 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:43.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:43.550 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:04:43.650 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:43.670 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:43.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:43.671 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:04:43.772 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:04:43.800 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:04:43.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:43.807 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:04:43.907 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:04:43.917 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:43 smithi023 bash[29146]: cluster 2024-03-11T21:04:42.184745+0000 mgr.y (mgr.24380) 516 : cluster 0 pgmap v575: 148 pgs: 148 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.0 KiB/s rd, 14 MiB/s wr, 1.04k op/s 2024-03-11T21:04:43.917 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:43 smithi023 bash[29146]: cluster 2024-03-11T21:04:42.621466+0000 osd.4 (osd.4) 83 : cluster 0 6.1e scrub starts 2024-03-11T21:04:43.917 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:43 smithi023 bash[29146]: cluster 2024-03-11T21:04:42.633890+0000 osd.4 (osd.4) 84 : cluster 0 6.1e scrub ok 2024-03-11T21:04:43.917 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:43 smithi023 bash[29146]: cluster 2024-03-11T21:04:43.344455+0000 osd.7 (osd.7) 293 : cluster 0 3.12 scrub starts 2024-03-11T21:04:43.917 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:43 smithi023 bash[29146]: cluster 2024-03-11T21:04:43.346651+0000 osd.7 (osd.7) 294 : cluster 0 3.12 scrub ok 2024-03-11T21:04:43.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:43 smithi023 bash[21372]: cluster 2024-03-11T21:04:42.184745+0000 mgr.y (mgr.24380) 516 : cluster 0 pgmap v575: 148 pgs: 148 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.0 KiB/s rd, 14 MiB/s wr, 1.04k op/s 2024-03-11T21:04:43.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:43 smithi023 bash[21372]: cluster 2024-03-11T21:04:42.621466+0000 osd.4 (osd.4) 83 : cluster 0 6.1e scrub starts 2024-03-11T21:04:43.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:43 smithi023 bash[21372]: cluster 2024-03-11T21:04:42.633890+0000 osd.4 (osd.4) 84 : cluster 0 6.1e scrub ok 2024-03-11T21:04:43.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:43 smithi023 bash[21372]: cluster 2024-03-11T21:04:43.344455+0000 osd.7 (osd.7) 293 : cluster 0 3.12 scrub starts 2024-03-11T21:04:43.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:43 smithi023 bash[21372]: cluster 2024-03-11T21:04:43.346651+0000 osd.7 (osd.7) 294 : cluster 0 3.12 scrub ok 2024-03-11T21:04:43.933 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:04:43.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:43.939 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:04:44.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:43 smithi146 bash[23809]: cluster 2024-03-11T21:04:42.184745+0000 mgr.y (mgr.24380) 516 : cluster 0 pgmap v575: 148 pgs: 148 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.0 KiB/s rd, 14 MiB/s wr, 1.04k op/s 2024-03-11T21:04:44.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:43 smithi146 bash[23809]: cluster 2024-03-11T21:04:42.621466+0000 osd.4 (osd.4) 83 : cluster 0 6.1e scrub starts 2024-03-11T21:04:44.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:43 smithi146 bash[23809]: cluster 2024-03-11T21:04:42.633890+0000 osd.4 (osd.4) 84 : cluster 0 6.1e scrub ok 2024-03-11T21:04:44.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:43 smithi146 bash[23809]: cluster 2024-03-11T21:04:43.344455+0000 osd.7 (osd.7) 293 : cluster 0 3.12 scrub starts 2024-03-11T21:04:44.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:43 smithi146 bash[23809]: cluster 2024-03-11T21:04:43.346651+0000 osd.7 (osd.7) 294 : cluster 0 3.12 scrub ok 2024-03-11T21:04:44.039 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:04:44.070 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:04:44.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:44.074 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:04:44.175 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:04:44.205 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:04:44.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:44.210 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:04:44.311 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:04:44.340 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:04:44.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:44.341 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:04:44.442 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:04:44.497 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:04:44.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:44.499 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:04:44.600 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:44.630 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:44.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:44.634 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:04:44.735 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:04:44.759 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:04:44.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:44.761 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:04:44.791 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:44 smithi023 bash[21372]: cluster 2024-03-11T21:04:42.682160+0000 osd.6 (osd.6) 212 : cluster 0 4.a scrub starts 2024-03-11T21:04:44.791 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:44 smithi023 bash[21372]: cluster 2024-03-11T21:04:42.685313+0000 osd.6 (osd.6) 213 : cluster 0 4.a scrub ok 2024-03-11T21:04:44.791 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:44 smithi023 bash[21372]: cluster 2024-03-11T21:04:43.559878+0000 osd.1 (osd.1) 61 : cluster 0 9.0 scrub starts 2024-03-11T21:04:44.791 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:44 smithi023 bash[21372]: cluster 2024-03-11T21:04:43.624109+0000 osd.4 (osd.4) 85 : cluster 0 4.c scrub starts 2024-03-11T21:04:44.791 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:44 smithi023 bash[21372]: cluster 2024-03-11T21:04:43.625257+0000 osd.4 (osd.4) 86 : cluster 0 4.c scrub ok 2024-03-11T21:04:44.791 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:44 smithi023 bash[21372]: cluster 2024-03-11T21:04:44.138951+0000 osd.3 (osd.3) 278 : cluster 0 2.1b scrub starts 2024-03-11T21:04:44.791 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:44 smithi023 bash[21372]: cluster 2024-03-11T21:04:44.142986+0000 osd.3 (osd.3) 279 : cluster 0 2.1b scrub ok 2024-03-11T21:04:44.791 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:44 smithi023 bash[21372]: cluster 2024-03-11T21:04:44.185373+0000 mgr.y (mgr.24380) 517 : cluster 0 pgmap v576: 148 pgs: 148 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 938 B/s rd, 9.7 MiB/s wr, 879 op/s 2024-03-11T21:04:44.791 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:44 smithi023 bash[21372]: cluster 2024-03-11T21:04:44.345132+0000 osd.7 (osd.7) 295 : cluster 0 2.1c scrub starts 2024-03-11T21:04:44.791 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:44 smithi023 bash[21372]: cluster 2024-03-11T21:04:44.347534+0000 osd.7 (osd.7) 296 : cluster 0 2.1c scrub ok 2024-03-11T21:04:44.792 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:44 smithi023 bash[29146]: cluster 2024-03-11T21:04:42.682160+0000 osd.6 (osd.6) 212 : cluster 0 4.a scrub starts 2024-03-11T21:04:44.792 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:44 smithi023 bash[29146]: cluster 2024-03-11T21:04:42.685313+0000 osd.6 (osd.6) 213 : cluster 0 4.a scrub ok 2024-03-11T21:04:44.792 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:44 smithi023 bash[29146]: cluster 2024-03-11T21:04:43.559878+0000 osd.1 (osd.1) 61 : cluster 0 9.0 scrub starts 2024-03-11T21:04:44.792 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:44 smithi023 bash[29146]: cluster 2024-03-11T21:04:43.624109+0000 osd.4 (osd.4) 85 : cluster 0 4.c scrub starts 2024-03-11T21:04:44.792 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:44 smithi023 bash[29146]: cluster 2024-03-11T21:04:43.625257+0000 osd.4 (osd.4) 86 : cluster 0 4.c scrub ok 2024-03-11T21:04:44.792 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:44 smithi023 bash[29146]: cluster 2024-03-11T21:04:44.138951+0000 osd.3 (osd.3) 278 : cluster 0 2.1b scrub starts 2024-03-11T21:04:44.792 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:44 smithi023 bash[29146]: cluster 2024-03-11T21:04:44.142986+0000 osd.3 (osd.3) 279 : cluster 0 2.1b scrub ok 2024-03-11T21:04:44.792 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:44 smithi023 bash[29146]: cluster 2024-03-11T21:04:44.185373+0000 mgr.y (mgr.24380) 517 : cluster 0 pgmap v576: 148 pgs: 148 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 938 B/s rd, 9.7 MiB/s wr, 879 op/s 2024-03-11T21:04:44.792 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:44 smithi023 bash[29146]: cluster 2024-03-11T21:04:44.345132+0000 osd.7 (osd.7) 295 : cluster 0 2.1c scrub starts 2024-03-11T21:04:44.792 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:44 smithi023 bash[29146]: cluster 2024-03-11T21:04:44.347534+0000 osd.7 (osd.7) 296 : cluster 0 2.1c scrub ok 2024-03-11T21:04:44.862 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:44.908 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:44.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:44.910 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:04:45.011 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:04:45.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:44 smithi146 bash[23809]: cluster 2024-03-11T21:04:42.682160+0000 osd.6 (osd.6) 212 : cluster 0 4.a scrub starts 2024-03-11T21:04:45.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:44 smithi146 bash[23809]: cluster 2024-03-11T21:04:42.685313+0000 osd.6 (osd.6) 213 : cluster 0 4.a scrub ok 2024-03-11T21:04:45.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:44 smithi146 bash[23809]: cluster 2024-03-11T21:04:43.559878+0000 osd.1 (osd.1) 61 : cluster 0 9.0 scrub starts 2024-03-11T21:04:45.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:44 smithi146 bash[23809]: cluster 2024-03-11T21:04:43.624109+0000 osd.4 (osd.4) 85 : cluster 0 4.c scrub starts 2024-03-11T21:04:45.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:44 smithi146 bash[23809]: cluster 2024-03-11T21:04:43.625257+0000 osd.4 (osd.4) 86 : cluster 0 4.c scrub ok 2024-03-11T21:04:45.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:44 smithi146 bash[23809]: cluster 2024-03-11T21:04:44.138951+0000 osd.3 (osd.3) 278 : cluster 0 2.1b scrub starts 2024-03-11T21:04:45.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:44 smithi146 bash[23809]: cluster 2024-03-11T21:04:44.142986+0000 osd.3 (osd.3) 279 : cluster 0 2.1b scrub ok 2024-03-11T21:04:45.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:44 smithi146 bash[23809]: cluster 2024-03-11T21:04:44.185373+0000 mgr.y (mgr.24380) 517 : cluster 0 pgmap v576: 148 pgs: 148 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 938 B/s rd, 9.7 MiB/s wr, 879 op/s 2024-03-11T21:04:45.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:44 smithi146 bash[23809]: cluster 2024-03-11T21:04:44.345132+0000 osd.7 (osd.7) 295 : cluster 0 2.1c scrub starts 2024-03-11T21:04:45.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:44 smithi146 bash[23809]: cluster 2024-03-11T21:04:44.347534+0000 osd.7 (osd.7) 296 : cluster 0 2.1c scrub ok 2024-03-11T21:04:45.034 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:04:45.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:45.037 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:04:45.138 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:04:45.177 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:04:45.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:45.180 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:04:45.281 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:45.302 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:45.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:45.304 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:04:45.404 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:45.432 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:45.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:45.437 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:04:45.538 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:45.560 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:45.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:45.562 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:04:45.614 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:04:45 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:04:45.663 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:45.688 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:45.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:45.691 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:04:45.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:45 smithi146 bash[23809]: cluster 2024-03-11T21:04:44.015715+0000 osd.2 (osd.2) 191 : cluster 0 2.1 scrub starts 2024-03-11T21:04:45.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:45 smithi146 bash[23809]: cluster 2024-03-11T21:04:44.017037+0000 osd.2 (osd.2) 192 : cluster 0 2.1 scrub ok 2024-03-11T21:04:45.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:45 smithi146 bash[23809]: cluster 2024-03-11T21:04:44.591404+0000 osd.1 (osd.1) 62 : cluster 0 9.4 scrub starts 2024-03-11T21:04:45.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:45 smithi146 bash[23809]: audit 2024-03-11T21:04:45.512300+0000 mgr.y (mgr.24380) 518 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:04:45.793 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:04:45.803 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:45 smithi023 bash[21372]: cluster 2024-03-11T21:04:44.015715+0000 osd.2 (osd.2) 191 : cluster 0 2.1 scrub starts 2024-03-11T21:04:45.803 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:45 smithi023 bash[21372]: cluster 2024-03-11T21:04:44.017037+0000 osd.2 (osd.2) 192 : cluster 0 2.1 scrub ok 2024-03-11T21:04:45.803 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:45 smithi023 bash[21372]: cluster 2024-03-11T21:04:44.591404+0000 osd.1 (osd.1) 62 : cluster 0 9.4 scrub starts 2024-03-11T21:04:45.803 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:45 smithi023 bash[21372]: audit 2024-03-11T21:04:45.512300+0000 mgr.y (mgr.24380) 518 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:04:45.803 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:45 smithi023 bash[29146]: cluster 2024-03-11T21:04:44.015715+0000 osd.2 (osd.2) 191 : cluster 0 2.1 scrub starts 2024-03-11T21:04:45.803 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:45 smithi023 bash[29146]: cluster 2024-03-11T21:04:44.017037+0000 osd.2 (osd.2) 192 : cluster 0 2.1 scrub ok 2024-03-11T21:04:45.803 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:45 smithi023 bash[29146]: cluster 2024-03-11T21:04:44.591404+0000 osd.1 (osd.1) 62 : cluster 0 9.4 scrub starts 2024-03-11T21:04:45.803 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:45 smithi023 bash[29146]: audit 2024-03-11T21:04:45.512300+0000 mgr.y (mgr.24380) 518 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:04:45.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:45.824 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:04:45.824 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:04:45.925 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:04:45.952 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:04:45.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:45.956 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:04:46.057 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:04:46.079 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:04:46.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:46.083 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:04:46.183 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:46.211 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:46.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:46.215 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:04:46.315 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:04:46.344 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:04:46.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:46.346 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:04:46.447 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:04:46.494 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:04:46.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:46.496 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:04:46.597 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:04:46.617 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:04:46.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:46.619 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:04:46.720 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:04:46.740 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:04:46.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:46.742 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:04:46.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:46 smithi146 bash[23809]: cluster 2024-03-11T21:04:44.710296+0000 osd.6 (osd.6) 214 : cluster 0 4.1f scrub starts 2024-03-11T21:04:46.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:46 smithi146 bash[23809]: cluster 2024-03-11T21:04:44.711560+0000 osd.6 (osd.6) 215 : cluster 0 4.1f scrub ok 2024-03-11T21:04:46.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:46 smithi146 bash[23809]: cluster 2024-03-11T21:04:45.789206+0000 osd.1 (osd.1) 63 : cluster 0 9.0 scrub ok 2024-03-11T21:04:46.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:46 smithi146 bash[23809]: cluster 2024-03-11T21:04:46.186472+0000 mgr.y (mgr.24380) 519 : cluster 0 pgmap v577: 148 pgs: 148 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 938 B/s rd, 9.7 MiB/s wr, 1.13k op/s 2024-03-11T21:04:46.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:46 smithi146 bash[23809]: cluster 2024-03-11T21:04:46.327581+0000 osd.7 (osd.7) 297 : cluster 0 9.d scrub starts 2024-03-11T21:04:46.842 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:46.872 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:46.874 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:46 smithi023 bash[21372]: cluster 2024-03-11T21:04:44.710296+0000 osd.6 (osd.6) 214 : cluster 0 4.1f scrub starts 2024-03-11T21:04:46.874 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:46 smithi023 bash[21372]: cluster 2024-03-11T21:04:44.711560+0000 osd.6 (osd.6) 215 : cluster 0 4.1f scrub ok 2024-03-11T21:04:46.874 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:46 smithi023 bash[21372]: cluster 2024-03-11T21:04:45.789206+0000 osd.1 (osd.1) 63 : cluster 0 9.0 scrub ok 2024-03-11T21:04:46.874 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:46 smithi023 bash[21372]: cluster 2024-03-11T21:04:46.186472+0000 mgr.y (mgr.24380) 519 : cluster 0 pgmap v577: 148 pgs: 148 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 938 B/s rd, 9.7 MiB/s wr, 1.13k op/s 2024-03-11T21:04:46.874 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:46 smithi023 bash[21372]: cluster 2024-03-11T21:04:46.327581+0000 osd.7 (osd.7) 297 : cluster 0 9.d scrub starts 2024-03-11T21:04:46.874 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:46 smithi023 bash[29146]: cluster 2024-03-11T21:04:44.710296+0000 osd.6 (osd.6) 214 : cluster 0 4.1f scrub starts 2024-03-11T21:04:46.874 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:46 smithi023 bash[29146]: cluster 2024-03-11T21:04:44.711560+0000 osd.6 (osd.6) 215 : cluster 0 4.1f scrub ok 2024-03-11T21:04:46.875 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:46 smithi023 bash[29146]: cluster 2024-03-11T21:04:45.789206+0000 osd.1 (osd.1) 63 : cluster 0 9.0 scrub ok 2024-03-11T21:04:46.875 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:46 smithi023 bash[29146]: cluster 2024-03-11T21:04:46.186472+0000 mgr.y (mgr.24380) 519 : cluster 0 pgmap v577: 148 pgs: 148 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 938 B/s rd, 9.7 MiB/s wr, 1.13k op/s 2024-03-11T21:04:46.875 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:46 smithi023 bash[29146]: cluster 2024-03-11T21:04:46.327581+0000 osd.7 (osd.7) 297 : cluster 0 9.d scrub starts 2024-03-11T21:04:46.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:46.875 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:04:46.976 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:04:47.013 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:04:47.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:47.018 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:04:47.119 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:47.146 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:47.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:47.151 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:04:47.184 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:04:47.252 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:04:47.272 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:04:47.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:47.275 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:04:47.375 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:04:47.398 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:04:47.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:47.400 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:04:47.501 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:04:47.525 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:04:47.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:47.528 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:04:47.629 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:04:47.654 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:04:47.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:47.656 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:04:47.757 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:04:47.783 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:04:47.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:47.786 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:04:47.887 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:04:47.912 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:04:47.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:47.915 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:04:48.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:47 smithi146 bash[23809]: cluster 2024-03-11T21:04:45.871278+0000 osd.6 (osd.6) 216 : cluster 0 purged_snaps scrub starts 2024-03-11T21:04:48.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:47 smithi146 bash[23809]: cluster 2024-03-11T21:04:45.871344+0000 osd.6 (osd.6) 217 : cluster 0 purged_snaps scrub ok 2024-03-11T21:04:48.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:47 smithi146 bash[23809]: cluster 2024-03-11T21:04:46.635100+0000 mon.a (mon.0) 1515 : cluster 0 osdmap e280: 8 total, 8 up, 8 in 2024-03-11T21:04:48.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:47 smithi146 bash[23809]: cluster 2024-03-11T21:04:46.773737+0000 osd.4 (osd.4) 87 : cluster 0 3.1 scrub starts 2024-03-11T21:04:48.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:47 smithi146 bash[23809]: cluster 2024-03-11T21:04:46.779817+0000 osd.4 (osd.4) 88 : cluster 0 3.1 scrub ok 2024-03-11T21:04:48.015 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:04:48.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:47 smithi023 bash[21372]: cluster 2024-03-11T21:04:45.871278+0000 osd.6 (osd.6) 216 : cluster 0 purged_snaps scrub starts 2024-03-11T21:04:48.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:47 smithi023 bash[21372]: cluster 2024-03-11T21:04:45.871344+0000 osd.6 (osd.6) 217 : cluster 0 purged_snaps scrub ok 2024-03-11T21:04:48.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:47 smithi023 bash[21372]: cluster 2024-03-11T21:04:46.635100+0000 mon.a (mon.0) 1515 : cluster 0 osdmap e280: 8 total, 8 up, 8 in 2024-03-11T21:04:48.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:47 smithi023 bash[21372]: cluster 2024-03-11T21:04:46.773737+0000 osd.4 (osd.4) 87 : cluster 0 3.1 scrub starts 2024-03-11T21:04:48.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:47 smithi023 bash[21372]: cluster 2024-03-11T21:04:46.779817+0000 osd.4 (osd.4) 88 : cluster 0 3.1 scrub ok 2024-03-11T21:04:48.033 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:47 smithi023 bash[29146]: cluster 2024-03-11T21:04:45.871278+0000 osd.6 (osd.6) 216 : cluster 0 purged_snaps scrub starts 2024-03-11T21:04:48.033 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:47 smithi023 bash[29146]: cluster 2024-03-11T21:04:45.871344+0000 osd.6 (osd.6) 217 : cluster 0 purged_snaps scrub ok 2024-03-11T21:04:48.033 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:47 smithi023 bash[29146]: cluster 2024-03-11T21:04:46.635100+0000 mon.a (mon.0) 1515 : cluster 0 osdmap e280: 8 total, 8 up, 8 in 2024-03-11T21:04:48.033 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:47 smithi023 bash[29146]: cluster 2024-03-11T21:04:46.773737+0000 osd.4 (osd.4) 87 : cluster 0 3.1 scrub starts 2024-03-11T21:04:48.033 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:47 smithi023 bash[29146]: cluster 2024-03-11T21:04:46.779817+0000 osd.4 (osd.4) 88 : cluster 0 3.1 scrub ok 2024-03-11T21:04:48.043 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:04:48.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:48.048 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:04:48.148 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:04:48.169 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:04:48.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:48.171 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:04:48.271 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:48.304 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:48.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:48.308 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:04:48.409 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:48.445 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:48.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:48.447 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:04:48.548 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:04:48.574 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:04:48.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:48.580 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:04:48.639 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:04:48 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:04:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:04:48.681 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:04:48.709 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:04:48.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:48.714 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:04:48.814 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:04:48.828 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:48 smithi023 bash[21372]: cluster 2024-03-11T21:04:46.933485+0000 osd.2 (osd.2) 193 : cluster 0 6.14 scrub starts 2024-03-11T21:04:48.828 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:48 smithi023 bash[21372]: cluster 2024-03-11T21:04:46.938402+0000 osd.2 (osd.2) 194 : cluster 0 6.14 scrub ok 2024-03-11T21:04:48.828 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:48 smithi023 bash[21372]: cluster 2024-03-11T21:04:47.235325+0000 osd.1 (osd.1) 64 : cluster 0 9.4 scrub ok 2024-03-11T21:04:48.840 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:04:48.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:48.844 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:04:48.945 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:04:48.970 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:04:48.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:48.975 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:04:49.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:48 smithi146 bash[23809]: cluster 2024-03-11T21:04:46.933485+0000 osd.2 (osd.2) 193 : cluster 0 6.14 scrub starts 2024-03-11T21:04:49.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:48 smithi146 bash[23809]: cluster 2024-03-11T21:04:46.938402+0000 osd.2 (osd.2) 194 : cluster 0 6.14 scrub ok 2024-03-11T21:04:49.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:48 smithi146 bash[23809]: cluster 2024-03-11T21:04:47.235325+0000 osd.1 (osd.1) 64 : cluster 0 9.4 scrub ok 2024-03-11T21:04:49.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:48 smithi146 bash[23809]: cluster 2024-03-11T21:04:47.488712+0000 osd.4 (osd.4) 89 : cluster 0 4.1 scrub starts 2024-03-11T21:04:49.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:48 smithi146 bash[23809]: cluster 2024-03-11T21:04:47.495037+0000 osd.4 (osd.4) 90 : cluster 0 4.1 scrub ok 2024-03-11T21:04:49.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:48 smithi146 bash[23809]: cluster 2024-03-11T21:04:47.706846+0000 osd.6 (osd.6) 218 : cluster 0 2.17 scrub starts 2024-03-11T21:04:49.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:48 smithi146 bash[23809]: cluster 2024-03-11T21:04:47.711052+0000 osd.6 (osd.6) 219 : cluster 0 2.17 scrub ok 2024-03-11T21:04:49.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:48 smithi146 bash[23809]: cluster 2024-03-11T21:04:47.823110+0000 osd.7 (osd.7) 298 : cluster 0 9.d scrub ok 2024-03-11T21:04:49.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:48 smithi146 bash[23809]: cluster 2024-03-11T21:04:48.187067+0000 mgr.y (mgr.24380) 520 : cluster 0 pgmap v579: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 996 op/s 2024-03-11T21:04:49.057 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:48 smithi023 bash[21372]: cluster 2024-03-11T21:04:47.488712+0000 osd.4 (osd.4) 89 : cluster 0 4.1 scrub starts 2024-03-11T21:04:49.057 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:48 smithi023 bash[21372]: cluster 2024-03-11T21:04:47.495037+0000 osd.4 (osd.4) 90 : cluster 0 4.1 scrub ok 2024-03-11T21:04:49.057 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:48 smithi023 bash[21372]: cluster 2024-03-11T21:04:47.706846+0000 osd.6 (osd.6) 218 : cluster 0 2.17 scrub starts 2024-03-11T21:04:49.058 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:48 smithi023 bash[21372]: cluster 2024-03-11T21:04:47.711052+0000 osd.6 (osd.6) 219 : cluster 0 2.17 scrub ok 2024-03-11T21:04:49.058 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:48 smithi023 bash[21372]: cluster 2024-03-11T21:04:47.823110+0000 osd.7 (osd.7) 298 : cluster 0 9.d scrub ok 2024-03-11T21:04:49.058 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:48 smithi023 bash[21372]: cluster 2024-03-11T21:04:48.187067+0000 mgr.y (mgr.24380) 520 : cluster 0 pgmap v579: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 996 op/s 2024-03-11T21:04:49.058 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:48 smithi023 bash[29146]: cluster 2024-03-11T21:04:46.933485+0000 osd.2 (osd.2) 193 : cluster 0 6.14 scrub starts 2024-03-11T21:04:49.058 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:48 smithi023 bash[29146]: cluster 2024-03-11T21:04:46.938402+0000 osd.2 (osd.2) 194 : cluster 0 6.14 scrub ok 2024-03-11T21:04:49.058 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:48 smithi023 bash[29146]: cluster 2024-03-11T21:04:47.235325+0000 osd.1 (osd.1) 64 : cluster 0 9.4 scrub ok 2024-03-11T21:04:49.058 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:48 smithi023 bash[29146]: cluster 2024-03-11T21:04:47.488712+0000 osd.4 (osd.4) 89 : cluster 0 4.1 scrub starts 2024-03-11T21:04:49.058 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:48 smithi023 bash[29146]: cluster 2024-03-11T21:04:47.495037+0000 osd.4 (osd.4) 90 : cluster 0 4.1 scrub ok 2024-03-11T21:04:49.059 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:48 smithi023 bash[29146]: cluster 2024-03-11T21:04:47.706846+0000 osd.6 (osd.6) 218 : cluster 0 2.17 scrub starts 2024-03-11T21:04:49.059 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:48 smithi023 bash[29146]: cluster 2024-03-11T21:04:47.711052+0000 osd.6 (osd.6) 219 : cluster 0 2.17 scrub ok 2024-03-11T21:04:49.059 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:48 smithi023 bash[29146]: cluster 2024-03-11T21:04:47.823110+0000 osd.7 (osd.7) 298 : cluster 0 9.d scrub ok 2024-03-11T21:04:49.059 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:48 smithi023 bash[29146]: cluster 2024-03-11T21:04:48.187067+0000 mgr.y (mgr.24380) 520 : cluster 0 pgmap v579: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 996 op/s 2024-03-11T21:04:49.076 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:49.094 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:49.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:49.096 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:04:49.197 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:04:49.220 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:04:49.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:49.222 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:04:49.323 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:49.349 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:49.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:49.352 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:04:49.453 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:04:49.488 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:04:49.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:49.492 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:04:49.593 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:49.614 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:49.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:49.617 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:04:49.718 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:04:49.744 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:04:49.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:49.752 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:04:49.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:49 smithi146 bash[23809]: cluster 2024-03-11T21:04:48.498334+0000 osd.3 (osd.3) 280 : cluster 0 purged_snaps scrub starts 2024-03-11T21:04:49.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:49 smithi146 bash[23809]: cluster 2024-03-11T21:04:48.498385+0000 osd.3 (osd.3) 281 : cluster 0 purged_snaps scrub ok 2024-03-11T21:04:49.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:49 smithi146 bash[23809]: cluster 2024-03-11T21:04:48.508815+0000 osd.4 (osd.4) 91 : cluster 0 4.1a scrub starts 2024-03-11T21:04:49.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:49 smithi146 bash[23809]: cluster 2024-03-11T21:04:48.510363+0000 osd.4 (osd.4) 92 : cluster 0 4.1a scrub ok 2024-03-11T21:04:49.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:49 smithi146 bash[23809]: cluster 2024-03-11T21:04:48.639116+0000 mon.a (mon.0) 1516 : cluster 0 osdmap e281: 8 total, 8 up, 8 in 2024-03-11T21:04:49.852 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:04:49.880 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:04:49.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:49.884 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:04:49.985 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:04:50.010 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:04:50.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:50.012 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:04:50.032 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:49 smithi023 bash[29146]: cluster 2024-03-11T21:04:48.498334+0000 osd.3 (osd.3) 280 : cluster 0 purged_snaps scrub starts 2024-03-11T21:04:50.032 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:49 smithi023 bash[29146]: cluster 2024-03-11T21:04:48.498385+0000 osd.3 (osd.3) 281 : cluster 0 purged_snaps scrub ok 2024-03-11T21:04:50.033 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:49 smithi023 bash[29146]: cluster 2024-03-11T21:04:48.508815+0000 osd.4 (osd.4) 91 : cluster 0 4.1a scrub starts 2024-03-11T21:04:50.033 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:49 smithi023 bash[29146]: cluster 2024-03-11T21:04:48.510363+0000 osd.4 (osd.4) 92 : cluster 0 4.1a scrub ok 2024-03-11T21:04:50.033 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:49 smithi023 bash[29146]: cluster 2024-03-11T21:04:48.639116+0000 mon.a (mon.0) 1516 : cluster 0 osdmap e281: 8 total, 8 up, 8 in 2024-03-11T21:04:50.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:49 smithi023 bash[21372]: cluster 2024-03-11T21:04:48.498334+0000 osd.3 (osd.3) 280 : cluster 0 purged_snaps scrub starts 2024-03-11T21:04:50.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:49 smithi023 bash[21372]: cluster 2024-03-11T21:04:48.498385+0000 osd.3 (osd.3) 281 : cluster 0 purged_snaps scrub ok 2024-03-11T21:04:50.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:49 smithi023 bash[21372]: cluster 2024-03-11T21:04:48.508815+0000 osd.4 (osd.4) 91 : cluster 0 4.1a scrub starts 2024-03-11T21:04:50.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:49 smithi023 bash[21372]: cluster 2024-03-11T21:04:48.510363+0000 osd.4 (osd.4) 92 : cluster 0 4.1a scrub ok 2024-03-11T21:04:50.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:49 smithi023 bash[21372]: cluster 2024-03-11T21:04:48.639116+0000 mon.a (mon.0) 1516 : cluster 0 osdmap e281: 8 total, 8 up, 8 in 2024-03-11T21:04:50.113 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:04:50.136 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:04:50.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:50.138 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:04:50.239 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:04:50.258 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:04:50.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:50.260 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:04:50.361 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:04:50.385 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:04:50.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:50.387 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:04:50.488 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:50.515 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:50.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:50.517 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:04:50.618 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:04:50.639 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:04:50.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:50.641 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:04:50.742 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:04:50.762 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:50 smithi023 bash[21372]: cluster 2024-03-11T21:04:48.905208+0000 osd.2 (osd.2) 195 : cluster 0 2.e scrub starts 2024-03-11T21:04:50.762 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:50 smithi023 bash[21372]: cluster 2024-03-11T21:04:48.908941+0000 osd.2 (osd.2) 196 : cluster 0 2.e scrub ok 2024-03-11T21:04:50.762 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:50 smithi023 bash[21372]: cluster 2024-03-11T21:04:48.914160+0000 osd.2 (osd.2) 197 : cluster 0 purged_snaps scrub starts 2024-03-11T21:04:50.762 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:50 smithi023 bash[21372]: cluster 2024-03-11T21:04:48.914227+0000 osd.2 (osd.2) 198 : cluster 0 purged_snaps scrub ok 2024-03-11T21:04:50.762 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:50 smithi023 bash[21372]: cluster 2024-03-11T21:04:49.456152+0000 osd.5 (osd.5) 257 : cluster 0 3.1d scrub starts 2024-03-11T21:04:50.762 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:50 smithi023 bash[21372]: cluster 2024-03-11T21:04:49.467078+0000 osd.5 (osd.5) 258 : cluster 0 3.1d scrub ok 2024-03-11T21:04:50.762 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:50 smithi023 bash[21372]: cluster 2024-03-11T21:04:49.502962+0000 osd.4 (osd.4) 93 : cluster 0 3.3 scrub starts 2024-03-11T21:04:50.762 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:50 smithi023 bash[21372]: cluster 2024-03-11T21:04:49.513744+0000 osd.4 (osd.4) 94 : cluster 0 3.3 scrub ok 2024-03-11T21:04:50.763 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:50 smithi023 bash[21372]: cluster 2024-03-11T21:04:49.649925+0000 mon.a (mon.0) 1517 : cluster 0 osdmap e282: 8 total, 8 up, 8 in 2024-03-11T21:04:50.763 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:50 smithi023 bash[21372]: cluster 2024-03-11T21:04:50.146475+0000 osd.3 (osd.3) 282 : cluster 0 6.13 scrub starts 2024-03-11T21:04:50.763 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:50 smithi023 bash[21372]: cluster 2024-03-11T21:04:50.187746+0000 mgr.y (mgr.24380) 521 : cluster 0 pgmap v582: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.00k op/s 2024-03-11T21:04:50.763 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:50 smithi023 bash[21372]: cluster 2024-03-11T21:04:50.207469+0000 osd.3 (osd.3) 283 : cluster 0 6.13 scrub ok 2024-03-11T21:04:50.763 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:50 smithi023 bash[21372]: cluster 2024-03-11T21:04:50.306003+0000 osd.0 (osd.0) 203 : cluster 0 3.1f scrub starts 2024-03-11T21:04:50.763 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:50 smithi023 bash[21372]: cluster 2024-03-11T21:04:50.309316+0000 osd.0 (osd.0) 204 : cluster 0 3.1f scrub ok 2024-03-11T21:04:50.763 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:50 smithi023 bash[21372]: audit 2024-03-11T21:04:50.352124+0000 mon.a (mon.0) 1518 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:04:50.773 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:04:50.777 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:50 smithi146 bash[23809]: cluster 2024-03-11T21:04:48.905208+0000 osd.2 (osd.2) 195 : cluster 0 2.e scrub starts 2024-03-11T21:04:50.777 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:50 smithi146 bash[23809]: cluster 2024-03-11T21:04:48.908941+0000 osd.2 (osd.2) 196 : cluster 0 2.e scrub ok 2024-03-11T21:04:50.777 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:50 smithi146 bash[23809]: cluster 2024-03-11T21:04:48.914160+0000 osd.2 (osd.2) 197 : cluster 0 purged_snaps scrub starts 2024-03-11T21:04:50.778 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:50 smithi146 bash[23809]: cluster 2024-03-11T21:04:48.914227+0000 osd.2 (osd.2) 198 : cluster 0 purged_snaps scrub ok 2024-03-11T21:04:50.778 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:50 smithi146 bash[23809]: cluster 2024-03-11T21:04:49.456152+0000 osd.5 (osd.5) 257 : cluster 0 3.1d scrub starts 2024-03-11T21:04:50.778 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:50 smithi146 bash[23809]: cluster 2024-03-11T21:04:49.467078+0000 osd.5 (osd.5) 258 : cluster 0 3.1d scrub ok 2024-03-11T21:04:50.778 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:50 smithi146 bash[23809]: cluster 2024-03-11T21:04:49.502962+0000 osd.4 (osd.4) 93 : cluster 0 3.3 scrub starts 2024-03-11T21:04:50.778 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:50 smithi146 bash[23809]: cluster 2024-03-11T21:04:49.513744+0000 osd.4 (osd.4) 94 : cluster 0 3.3 scrub ok 2024-03-11T21:04:50.778 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:50 smithi146 bash[23809]: cluster 2024-03-11T21:04:49.649925+0000 mon.a (mon.0) 1517 : cluster 0 osdmap e282: 8 total, 8 up, 8 in 2024-03-11T21:04:50.778 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:50 smithi146 bash[23809]: cluster 2024-03-11T21:04:50.146475+0000 osd.3 (osd.3) 282 : cluster 0 6.13 scrub starts 2024-03-11T21:04:50.778 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:50 smithi146 bash[23809]: cluster 2024-03-11T21:04:50.187746+0000 mgr.y (mgr.24380) 521 : cluster 0 pgmap v582: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.00k op/s 2024-03-11T21:04:50.778 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:50 smithi146 bash[23809]: cluster 2024-03-11T21:04:50.207469+0000 osd.3 (osd.3) 283 : cluster 0 6.13 scrub ok 2024-03-11T21:04:50.778 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:50 smithi146 bash[23809]: cluster 2024-03-11T21:04:50.306003+0000 osd.0 (osd.0) 203 : cluster 0 3.1f scrub starts 2024-03-11T21:04:50.778 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:50 smithi146 bash[23809]: cluster 2024-03-11T21:04:50.309316+0000 osd.0 (osd.0) 204 : cluster 0 3.1f scrub ok 2024-03-11T21:04:50.778 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:50 smithi146 bash[23809]: audit 2024-03-11T21:04:50.352124+0000 mon.a (mon.0) 1518 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:04:50.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:50.779 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:04:50.880 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:04:50.905 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:04:50.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:50.908 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:04:51.008 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:04:51.033 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:50 smithi023 bash[29146]: cluster 2024-03-11T21:04:48.905208+0000 osd.2 (osd.2) 195 : cluster 0 2.e scrub starts 2024-03-11T21:04:51.033 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:50 smithi023 bash[29146]: cluster 2024-03-11T21:04:48.908941+0000 osd.2 (osd.2) 196 : cluster 0 2.e scrub ok 2024-03-11T21:04:51.033 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:50 smithi023 bash[29146]: cluster 2024-03-11T21:04:48.914160+0000 osd.2 (osd.2) 197 : cluster 0 purged_snaps scrub starts 2024-03-11T21:04:51.033 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:50 smithi023 bash[29146]: cluster 2024-03-11T21:04:48.914227+0000 osd.2 (osd.2) 198 : cluster 0 purged_snaps scrub ok 2024-03-11T21:04:51.033 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:50 smithi023 bash[29146]: cluster 2024-03-11T21:04:49.456152+0000 osd.5 (osd.5) 257 : cluster 0 3.1d scrub starts 2024-03-11T21:04:51.033 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:50 smithi023 bash[29146]: cluster 2024-03-11T21:04:49.467078+0000 osd.5 (osd.5) 258 : cluster 0 3.1d scrub ok 2024-03-11T21:04:51.033 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:50 smithi023 bash[29146]: cluster 2024-03-11T21:04:49.502962+0000 osd.4 (osd.4) 93 : cluster 0 3.3 scrub starts 2024-03-11T21:04:51.033 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:50 smithi023 bash[29146]: cluster 2024-03-11T21:04:49.513744+0000 osd.4 (osd.4) 94 : cluster 0 3.3 scrub ok 2024-03-11T21:04:51.033 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:50 smithi023 bash[29146]: cluster 2024-03-11T21:04:49.649925+0000 mon.a (mon.0) 1517 : cluster 0 osdmap e282: 8 total, 8 up, 8 in 2024-03-11T21:04:51.033 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:50 smithi023 bash[29146]: cluster 2024-03-11T21:04:50.146475+0000 osd.3 (osd.3) 282 : cluster 0 6.13 scrub starts 2024-03-11T21:04:51.033 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:50 smithi023 bash[29146]: cluster 2024-03-11T21:04:50.187746+0000 mgr.y (mgr.24380) 521 : cluster 0 pgmap v582: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.00k op/s 2024-03-11T21:04:51.033 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:50 smithi023 bash[29146]: cluster 2024-03-11T21:04:50.207469+0000 osd.3 (osd.3) 283 : cluster 0 6.13 scrub ok 2024-03-11T21:04:51.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:50 smithi023 bash[29146]: cluster 2024-03-11T21:04:50.306003+0000 osd.0 (osd.0) 203 : cluster 0 3.1f scrub starts 2024-03-11T21:04:51.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:50 smithi023 bash[29146]: cluster 2024-03-11T21:04:50.309316+0000 osd.0 (osd.0) 204 : cluster 0 3.1f scrub ok 2024-03-11T21:04:51.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:50 smithi023 bash[29146]: audit 2024-03-11T21:04:50.352124+0000 mon.a (mon.0) 1518 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:04:51.040 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:04:51.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:51.043 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:04:51.144 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:51.174 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:51.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:51.178 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:04:51.279 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:04:51.308 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:04:51.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:51.311 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:04:51.412 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:51.438 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:51.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:51.441 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:04:51.541 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:04:51.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:51.570 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:04:51.570 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:04:51.671 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:04:51.697 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:04:51.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:51.700 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:04:51.801 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:51.835 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:51.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:51.837 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:04:51.938 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:04:51.945 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:51 smithi023 bash[21372]: cluster 2024-03-11T21:04:50.535296+0000 osd.7 (osd.7) 299 : cluster 0 9.9 scrub starts 2024-03-11T21:04:51.945 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:51 smithi023 bash[21372]: cluster 2024-03-11T21:04:51.062872+0000 osd.1 (osd.1) 65 : cluster 0 purged_snaps scrub starts 2024-03-11T21:04:51.945 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:51 smithi023 bash[21372]: cluster 2024-03-11T21:04:51.062944+0000 osd.1 (osd.1) 66 : cluster 0 purged_snaps scrub ok 2024-03-11T21:04:51.945 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:51 smithi023 bash[21372]: cluster 2024-03-11T21:04:51.192388+0000 osd.3 (osd.3) 284 : cluster 0 6.1b scrub starts 2024-03-11T21:04:51.945 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:51 smithi023 bash[29146]: cluster 2024-03-11T21:04:50.535296+0000 osd.7 (osd.7) 299 : cluster 0 9.9 scrub starts 2024-03-11T21:04:51.945 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:51 smithi023 bash[29146]: cluster 2024-03-11T21:04:51.062872+0000 osd.1 (osd.1) 65 : cluster 0 purged_snaps scrub starts 2024-03-11T21:04:51.945 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:51 smithi023 bash[29146]: cluster 2024-03-11T21:04:51.062944+0000 osd.1 (osd.1) 66 : cluster 0 purged_snaps scrub ok 2024-03-11T21:04:51.945 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:51 smithi023 bash[29146]: cluster 2024-03-11T21:04:51.192388+0000 osd.3 (osd.3) 284 : cluster 0 6.1b scrub starts 2024-03-11T21:04:51.947 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:51 smithi146 bash[23809]: cluster 2024-03-11T21:04:50.535296+0000 osd.7 (osd.7) 299 : cluster 0 9.9 scrub starts 2024-03-11T21:04:51.947 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:51 smithi146 bash[23809]: cluster 2024-03-11T21:04:51.062872+0000 osd.1 (osd.1) 65 : cluster 0 purged_snaps scrub starts 2024-03-11T21:04:51.947 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:51 smithi146 bash[23809]: cluster 2024-03-11T21:04:51.062944+0000 osd.1 (osd.1) 66 : cluster 0 purged_snaps scrub ok 2024-03-11T21:04:51.947 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:51 smithi146 bash[23809]: cluster 2024-03-11T21:04:51.192388+0000 osd.3 (osd.3) 284 : cluster 0 6.1b scrub starts 2024-03-11T21:04:51.963 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:04:51.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:51.966 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:04:52.067 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:04:52.115 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:04:52.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:52.125 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:04:52.217 INFO:teuthology.orchestra.run.smithi023.stdout:osd.0: {} 2024-03-11T21:04:52.226 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:04:52.245 INFO:teuthology.orchestra.run.smithi023.stdout:osd.1: {} 2024-03-11T21:04:52.251 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:04:52.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:52.254 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:04:52.295 INFO:teuthology.orchestra.run.smithi023.stdout:osd.2: {} 2024-03-11T21:04:52.337 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-11T21:04:52.347 INFO:teuthology.orchestra.run.smithi023.stderr:dumped all 2024-03-11T21:04:52.347 INFO:teuthology.orchestra.run.smithi023.stdout:osd.3: {} 2024-03-11T21:04:52.354 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:52.385 INFO:teuthology.orchestra.run.smithi023.stdout:osd.4: {} 2024-03-11T21:04:52.404 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:52.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:52.409 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:04:52.424 INFO:teuthology.orchestra.run.smithi023.stdout:osd.5: {} 2024-03-11T21:04:52.459 INFO:teuthology.orchestra.run.smithi023.stdout:osd.6: {} 2024-03-11T21:04:52.495 INFO:teuthology.orchestra.run.smithi023.stdout:osd.7: {} 2024-03-11T21:04:52.499 INFO:teuthology.orchestra.run.smithi023.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:04:52.499 INFO:teuthology.orchestra.run.smithi023.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:04:52.499 INFO:teuthology.orchestra.run.smithi023.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:04:52.499 INFO:teuthology.orchestra.run.smithi023.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:04:52.499 INFO:teuthology.orchestra.run.smithi023.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:04:52.499 INFO:teuthology.orchestra.run.smithi023.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:04:52.499 INFO:teuthology.orchestra.run.smithi023.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:04:52.499 INFO:teuthology.orchestra.run.smithi023.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:04:52.511 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:04:52.536 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:04:52.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:52.538 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:04:52.639 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:52.669 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:52.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:52.673 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:04:52.706 INFO:teuthology.orchestra.run.smithi023.stderr:nodeep-scrub is unset 2024-03-11T21:04:52.774 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:04:52.808 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:04:52.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:52.812 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:04:52.913 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:04:52.921 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:52 smithi023 bash[21372]: cluster 2024-03-11T21:04:51.279059+0000 osd.3 (osd.3) 285 : cluster 0 6.1b scrub ok 2024-03-11T21:04:52.922 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:52 smithi023 bash[21372]: cluster 2024-03-11T21:04:51.688221+0000 mon.a (mon.0) 1519 : cluster 0 osdmap e283: 8 total, 8 up, 8 in 2024-03-11T21:04:52.922 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:52 smithi023 bash[21372]: cluster 2024-03-11T21:04:51.696700+0000 osd.6 (osd.6) 220 : cluster 0 2.7 scrub starts 2024-03-11T21:04:52.922 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:52 smithi023 bash[21372]: cluster 2024-03-11T21:04:51.700739+0000 osd.6 (osd.6) 221 : cluster 0 2.7 scrub ok 2024-03-11T21:04:52.922 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:52 smithi023 bash[21372]: cluster 2024-03-11T21:04:52.158440+0000 osd.3 (osd.3) 286 : cluster 0 4.0 scrub starts 2024-03-11T21:04:52.922 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:52 smithi023 bash[21372]: audit 2024-03-11T21:04:52.168435+0000 mon.c (mon.2) 66 : audit 0 from='client.? 172.21.15.23:0/553365793' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:04:52.922 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:52 smithi023 bash[21372]: cluster 2024-03-11T21:04:52.175874+0000 osd.3 (osd.3) 287 : cluster 0 4.0 scrub ok 2024-03-11T21:04:52.922 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:52 smithi023 bash[21372]: cluster 2024-03-11T21:04:52.188306+0000 mgr.y (mgr.24380) 522 : cluster 0 pgmap v584: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 921 B/s rd, 0 B/s wr, 856 op/s 2024-03-11T21:04:52.922 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:52 smithi023 bash[21372]: audit 2024-03-11T21:04:52.309492+0000 mgr.y (mgr.24380) 523 : audit 0 from='client.25049 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:04:52.922 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:52 smithi023 bash[21372]: cluster 2024-03-11T21:04:52.370285+0000 osd.7 (osd.7) 300 : cluster 0 9.9 scrub ok 2024-03-11T21:04:52.922 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:52 smithi023 bash[21372]: audit 2024-03-11T21:04:52.434418+0000 mon.a (mon.0) 1520 : audit 1 from='client.? 172.21.15.23:0/376389630' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:04:52.923 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:52 smithi023 bash[29146]: cluster 2024-03-11T21:04:51.279059+0000 osd.3 (osd.3) 285 : cluster 0 6.1b scrub ok 2024-03-11T21:04:52.923 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:52 smithi023 bash[29146]: cluster 2024-03-11T21:04:51.688221+0000 mon.a (mon.0) 1519 : cluster 0 osdmap e283: 8 total, 8 up, 8 in 2024-03-11T21:04:52.923 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:52 smithi023 bash[29146]: cluster 2024-03-11T21:04:51.696700+0000 osd.6 (osd.6) 220 : cluster 0 2.7 scrub starts 2024-03-11T21:04:52.923 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:52 smithi023 bash[29146]: cluster 2024-03-11T21:04:51.700739+0000 osd.6 (osd.6) 221 : cluster 0 2.7 scrub ok 2024-03-11T21:04:52.923 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:52 smithi023 bash[29146]: cluster 2024-03-11T21:04:52.158440+0000 osd.3 (osd.3) 286 : cluster 0 4.0 scrub starts 2024-03-11T21:04:52.923 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:52 smithi023 bash[29146]: audit 2024-03-11T21:04:52.168435+0000 mon.c (mon.2) 66 : audit 0 from='client.? 172.21.15.23:0/553365793' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:04:52.923 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:52 smithi023 bash[29146]: cluster 2024-03-11T21:04:52.175874+0000 osd.3 (osd.3) 287 : cluster 0 4.0 scrub ok 2024-03-11T21:04:52.923 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:52 smithi023 bash[29146]: cluster 2024-03-11T21:04:52.188306+0000 mgr.y (mgr.24380) 522 : cluster 0 pgmap v584: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 921 B/s rd, 0 B/s wr, 856 op/s 2024-03-11T21:04:52.923 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:52 smithi023 bash[29146]: audit 2024-03-11T21:04:52.309492+0000 mgr.y (mgr.24380) 523 : audit 0 from='client.25049 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:04:52.923 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:52 smithi023 bash[29146]: cluster 2024-03-11T21:04:52.370285+0000 osd.7 (osd.7) 300 : cluster 0 9.9 scrub ok 2024-03-11T21:04:52.923 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:52 smithi023 bash[29146]: audit 2024-03-11T21:04:52.434418+0000 mon.a (mon.0) 1520 : audit 1 from='client.? 172.21.15.23:0/376389630' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:04:52.939 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:04:52.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:52.941 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:04:53.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:52 smithi146 bash[23809]: cluster 2024-03-11T21:04:51.279059+0000 osd.3 (osd.3) 285 : cluster 0 6.1b scrub ok 2024-03-11T21:04:53.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:52 smithi146 bash[23809]: cluster 2024-03-11T21:04:51.688221+0000 mon.a (mon.0) 1519 : cluster 0 osdmap e283: 8 total, 8 up, 8 in 2024-03-11T21:04:53.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:52 smithi146 bash[23809]: cluster 2024-03-11T21:04:51.696700+0000 osd.6 (osd.6) 220 : cluster 0 2.7 scrub starts 2024-03-11T21:04:53.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:52 smithi146 bash[23809]: cluster 2024-03-11T21:04:51.700739+0000 osd.6 (osd.6) 221 : cluster 0 2.7 scrub ok 2024-03-11T21:04:53.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:52 smithi146 bash[23809]: cluster 2024-03-11T21:04:52.158440+0000 osd.3 (osd.3) 286 : cluster 0 4.0 scrub starts 2024-03-11T21:04:53.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:52 smithi146 bash[23809]: audit 2024-03-11T21:04:52.168435+0000 mon.c (mon.2) 66 : audit 0 from='client.? 172.21.15.23:0/553365793' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:04:53.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:52 smithi146 bash[23809]: cluster 2024-03-11T21:04:52.175874+0000 osd.3 (osd.3) 287 : cluster 0 4.0 scrub ok 2024-03-11T21:04:53.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:52 smithi146 bash[23809]: cluster 2024-03-11T21:04:52.188306+0000 mgr.y (mgr.24380) 522 : cluster 0 pgmap v584: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 921 B/s rd, 0 B/s wr, 856 op/s 2024-03-11T21:04:53.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:52 smithi146 bash[23809]: audit 2024-03-11T21:04:52.309492+0000 mgr.y (mgr.24380) 523 : audit 0 from='client.25049 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:04:53.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:52 smithi146 bash[23809]: cluster 2024-03-11T21:04:52.370285+0000 osd.7 (osd.7) 300 : cluster 0 9.9 scrub ok 2024-03-11T21:04:53.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:52 smithi146 bash[23809]: audit 2024-03-11T21:04:52.434418+0000 mon.a (mon.0) 1520 : audit 1 from='client.? 172.21.15.23:0/376389630' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:04:53.042 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:04:53.078 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:04:53.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:53.082 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:04:53.182 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:04:53.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:53.214 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:04:53.214 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:04:53.315 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:53.340 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:53.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:53.344 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:04:53.445 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:04:53.468 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:04:53.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:53.471 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:04:53.571 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:53.599 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:53.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:53.602 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:04:53.703 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:04:53.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:53.741 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:04:53.742 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:04:53.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:53 smithi146 bash[23809]: cluster 2024-03-11T21:04:51.860111+0000 osd.2 (osd.2) 199 : cluster 0 4.1c scrub starts 2024-03-11T21:04:53.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:53 smithi146 bash[23809]: cluster 2024-03-11T21:04:51.868533+0000 osd.2 (osd.2) 200 : cluster 0 4.1c scrub ok 2024-03-11T21:04:53.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:53 smithi146 bash[23809]: cluster 2024-03-11T21:04:52.683187+0000 mon.a (mon.0) 1521 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:04:53.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:53 smithi146 bash[23809]: cluster 2024-03-11T21:04:52.683222+0000 mon.a (mon.0) 1522 : cluster 1 Cluster is now healthy 2024-03-11T21:04:53.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:53 smithi146 bash[23809]: audit 2024-03-11T21:04:52.693258+0000 mon.a (mon.0) 1523 : audit 1 from='client.? 172.21.15.23:0/376389630' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:04:53.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:53 smithi146 bash[23809]: cluster 2024-03-11T21:04:52.698716+0000 mon.a (mon.0) 1524 : cluster 0 osdmap e284: 8 total, 8 up, 8 in 2024-03-11T21:04:53.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:53 smithi146 bash[23809]: cluster 2024-03-11T21:04:52.736387+0000 osd.6 (osd.6) 222 : cluster 0 5.2 scrub starts 2024-03-11T21:04:53.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:53 smithi146 bash[23809]: cluster 2024-03-11T21:04:52.756598+0000 osd.6 (osd.6) 223 : cluster 0 5.2 scrub ok 2024-03-11T21:04:53.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:53 smithi146 bash[23809]: cluster 2024-03-11T21:04:53.203173+0000 osd.3 (osd.3) 288 : cluster 0 3.2 deep-scrub starts 2024-03-11T21:04:53.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:53 smithi146 bash[23809]: cluster 2024-03-11T21:04:53.217180+0000 osd.3 (osd.3) 289 : cluster 0 3.2 deep-scrub ok 2024-03-11T21:04:53.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:53 smithi146 bash[23809]: cluster 2024-03-11T21:04:53.478570+0000 osd.7 (osd.7) 301 : cluster 0 9.e scrub starts 2024-03-11T21:04:53.837 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:53 smithi023 bash[21372]: cluster 2024-03-11T21:04:51.860111+0000 osd.2 (osd.2) 199 : cluster 0 4.1c scrub starts 2024-03-11T21:04:53.838 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:53 smithi023 bash[21372]: cluster 2024-03-11T21:04:51.868533+0000 osd.2 (osd.2) 200 : cluster 0 4.1c scrub ok 2024-03-11T21:04:53.838 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:53 smithi023 bash[21372]: cluster 2024-03-11T21:04:52.683187+0000 mon.a (mon.0) 1521 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:04:53.838 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:53 smithi023 bash[21372]: cluster 2024-03-11T21:04:52.683222+0000 mon.a (mon.0) 1522 : cluster 1 Cluster is now healthy 2024-03-11T21:04:53.838 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:53 smithi023 bash[21372]: audit 2024-03-11T21:04:52.693258+0000 mon.a (mon.0) 1523 : audit 1 from='client.? 172.21.15.23:0/376389630' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:04:53.838 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:53 smithi023 bash[21372]: cluster 2024-03-11T21:04:52.698716+0000 mon.a (mon.0) 1524 : cluster 0 osdmap e284: 8 total, 8 up, 8 in 2024-03-11T21:04:53.838 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:53 smithi023 bash[21372]: cluster 2024-03-11T21:04:52.736387+0000 osd.6 (osd.6) 222 : cluster 0 5.2 scrub starts 2024-03-11T21:04:53.838 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:53 smithi023 bash[21372]: cluster 2024-03-11T21:04:52.756598+0000 osd.6 (osd.6) 223 : cluster 0 5.2 scrub ok 2024-03-11T21:04:53.838 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:53 smithi023 bash[21372]: cluster 2024-03-11T21:04:53.203173+0000 osd.3 (osd.3) 288 : cluster 0 3.2 deep-scrub starts 2024-03-11T21:04:53.838 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:53 smithi023 bash[21372]: cluster 2024-03-11T21:04:53.217180+0000 osd.3 (osd.3) 289 : cluster 0 3.2 deep-scrub ok 2024-03-11T21:04:53.838 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:53 smithi023 bash[21372]: cluster 2024-03-11T21:04:53.478570+0000 osd.7 (osd.7) 301 : cluster 0 9.e scrub starts 2024-03-11T21:04:53.839 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:53 smithi023 bash[29146]: cluster 2024-03-11T21:04:51.860111+0000 osd.2 (osd.2) 199 : cluster 0 4.1c scrub starts 2024-03-11T21:04:53.839 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:53 smithi023 bash[29146]: cluster 2024-03-11T21:04:51.868533+0000 osd.2 (osd.2) 200 : cluster 0 4.1c scrub ok 2024-03-11T21:04:53.839 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:53 smithi023 bash[29146]: cluster 2024-03-11T21:04:52.683187+0000 mon.a (mon.0) 1521 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:04:53.839 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:53 smithi023 bash[29146]: cluster 2024-03-11T21:04:52.683222+0000 mon.a (mon.0) 1522 : cluster 1 Cluster is now healthy 2024-03-11T21:04:53.839 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:53 smithi023 bash[29146]: audit 2024-03-11T21:04:52.693258+0000 mon.a (mon.0) 1523 : audit 1 from='client.? 172.21.15.23:0/376389630' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:04:53.839 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:53 smithi023 bash[29146]: cluster 2024-03-11T21:04:52.698716+0000 mon.a (mon.0) 1524 : cluster 0 osdmap e284: 8 total, 8 up, 8 in 2024-03-11T21:04:53.839 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:53 smithi023 bash[29146]: cluster 2024-03-11T21:04:52.736387+0000 osd.6 (osd.6) 222 : cluster 0 5.2 scrub starts 2024-03-11T21:04:53.839 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:53 smithi023 bash[29146]: cluster 2024-03-11T21:04:52.756598+0000 osd.6 (osd.6) 223 : cluster 0 5.2 scrub ok 2024-03-11T21:04:53.839 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:53 smithi023 bash[29146]: cluster 2024-03-11T21:04:53.203173+0000 osd.3 (osd.3) 288 : cluster 0 3.2 deep-scrub starts 2024-03-11T21:04:53.839 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:53 smithi023 bash[29146]: cluster 2024-03-11T21:04:53.217180+0000 osd.3 (osd.3) 289 : cluster 0 3.2 deep-scrub ok 2024-03-11T21:04:53.839 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:53 smithi023 bash[29146]: cluster 2024-03-11T21:04:53.478570+0000 osd.7 (osd.7) 301 : cluster 0 9.e scrub starts 2024-03-11T21:04:53.843 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:04:53.863 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:04:53.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:53.865 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:04:53.966 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:04:53.989 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:04:53.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:53.993 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:04:54.094 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:04:54.123 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:04:54.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:54.126 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:04:54.227 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:04:54.254 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:04:54.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:54.257 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:04:54.358 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:04:54.388 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:04:54.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:54.392 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:04:54.493 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:04:54.515 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:04:54.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:54.516 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:04:54.617 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:04:54.638 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:04:54.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:54.639 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:04:54.740 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:04:54.763 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:04:54.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:54.764 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:04:54.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:54 smithi023 bash[21372]: cluster 2024-03-11T21:04:53.464465+0000 osd.4 (osd.4) 95 : cluster 0 4.18 scrub starts 2024-03-11T21:04:54.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:54 smithi023 bash[21372]: cluster 2024-03-11T21:04:53.468104+0000 osd.4 (osd.4) 96 : cluster 0 4.18 scrub ok 2024-03-11T21:04:54.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:54 smithi023 bash[21372]: cluster 2024-03-11T21:04:54.188731+0000 mgr.y (mgr.24380) 524 : cluster 0 pgmap v586: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 922 B/s rd, 0 B/s wr, 756 op/s 2024-03-11T21:04:54.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:54 smithi023 bash[21372]: cluster 2024-03-11T21:04:54.196657+0000 osd.3 (osd.3) 290 : cluster 0 6.11 scrub starts 2024-03-11T21:04:54.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:54 smithi023 bash[21372]: cluster 2024-03-11T21:04:54.215376+0000 osd.0 (osd.0) 205 : cluster 0 4.6 scrub starts 2024-03-11T21:04:54.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:54 smithi023 bash[21372]: cluster 2024-03-11T21:04:54.223065+0000 osd.3 (osd.3) 291 : cluster 0 6.11 scrub ok 2024-03-11T21:04:54.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:54 smithi023 bash[21372]: cluster 2024-03-11T21:04:54.227763+0000 osd.0 (osd.0) 206 : cluster 0 4.6 scrub ok 2024-03-11T21:04:54.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:54 smithi023 bash[21372]: cluster 2024-03-11T21:04:54.513824+0000 osd.7 (osd.7) 302 : cluster 0 9.3 scrub starts 2024-03-11T21:04:54.784 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:54 smithi023 bash[29146]: cluster 2024-03-11T21:04:53.464465+0000 osd.4 (osd.4) 95 : cluster 0 4.18 scrub starts 2024-03-11T21:04:54.784 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:54 smithi023 bash[29146]: cluster 2024-03-11T21:04:53.468104+0000 osd.4 (osd.4) 96 : cluster 0 4.18 scrub ok 2024-03-11T21:04:54.784 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:54 smithi023 bash[29146]: cluster 2024-03-11T21:04:54.188731+0000 mgr.y (mgr.24380) 524 : cluster 0 pgmap v586: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 922 B/s rd, 0 B/s wr, 756 op/s 2024-03-11T21:04:54.784 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:54 smithi023 bash[29146]: cluster 2024-03-11T21:04:54.196657+0000 osd.3 (osd.3) 290 : cluster 0 6.11 scrub starts 2024-03-11T21:04:54.784 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:54 smithi023 bash[29146]: cluster 2024-03-11T21:04:54.215376+0000 osd.0 (osd.0) 205 : cluster 0 4.6 scrub starts 2024-03-11T21:04:54.784 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:54 smithi023 bash[29146]: cluster 2024-03-11T21:04:54.223065+0000 osd.3 (osd.3) 291 : cluster 0 6.11 scrub ok 2024-03-11T21:04:54.784 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:54 smithi023 bash[29146]: cluster 2024-03-11T21:04:54.227763+0000 osd.0 (osd.0) 206 : cluster 0 4.6 scrub ok 2024-03-11T21:04:54.784 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:54 smithi023 bash[29146]: cluster 2024-03-11T21:04:54.513824+0000 osd.7 (osd.7) 302 : cluster 0 9.3 scrub starts 2024-03-11T21:04:54.866 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:54.890 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:54.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:54.896 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:04:54.997 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:04:55.027 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:04:55.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:55.031 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:04:55.132 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:04:55.142 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:54 smithi146 bash[23809]: cluster 2024-03-11T21:04:53.464465+0000 osd.4 (osd.4) 95 : cluster 0 4.18 scrub starts 2024-03-11T21:04:55.142 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:54 smithi146 bash[23809]: cluster 2024-03-11T21:04:53.468104+0000 osd.4 (osd.4) 96 : cluster 0 4.18 scrub ok 2024-03-11T21:04:55.142 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:54 smithi146 bash[23809]: cluster 2024-03-11T21:04:54.188731+0000 mgr.y (mgr.24380) 524 : cluster 0 pgmap v586: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 922 B/s rd, 0 B/s wr, 756 op/s 2024-03-11T21:04:55.143 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:54 smithi146 bash[23809]: cluster 2024-03-11T21:04:54.196657+0000 osd.3 (osd.3) 290 : cluster 0 6.11 scrub starts 2024-03-11T21:04:55.143 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:54 smithi146 bash[23809]: cluster 2024-03-11T21:04:54.215376+0000 osd.0 (osd.0) 205 : cluster 0 4.6 scrub starts 2024-03-11T21:04:55.143 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:54 smithi146 bash[23809]: cluster 2024-03-11T21:04:54.223065+0000 osd.3 (osd.3) 291 : cluster 0 6.11 scrub ok 2024-03-11T21:04:55.143 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:54 smithi146 bash[23809]: cluster 2024-03-11T21:04:54.227763+0000 osd.0 (osd.0) 206 : cluster 0 4.6 scrub ok 2024-03-11T21:04:55.143 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:54 smithi146 bash[23809]: cluster 2024-03-11T21:04:54.513824+0000 osd.7 (osd.7) 302 : cluster 0 9.3 scrub starts 2024-03-11T21:04:55.174 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:04:55.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:55.180 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:04:55.281 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:04:55.312 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:04:55.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:55.315 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:04:55.416 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:55.439 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:55.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:55.441 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:04:55.542 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:55.574 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:55.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:55.579 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:04:55.680 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:55.681 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:04:55 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:04:55.710 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:55.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:55.715 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:04:55.793 INFO:teuthology.orchestra.run.smithi023.stdout:{"pg_ready":true,"pg_map":{"version":584,"stamp":"2024-03-11T21:04:52.187928+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6134634944,"num_objects":93797,"num_object_clones":0,"num_object_copies":281391,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":93797,"num_whiteouts":0,"num_read":2090,"num_read_kb":1936,"num_write":129590,"num_write_kb":7115614,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":197,"num_bytes_recovered":1775932,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81833,"ondisk_log_size":81833,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":20232704,"kb_used_data":18042972,"kb_used_omap":21,"kb_used_meta":2189546,"kb_avail":729728512,"statfs":{"total":767960285184,"available":747241996288,"internally_reserved":0,"allocated":18476003328,"data_stored":18473374166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22089,"internal_metadata":2242095543},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-311492608,"num_objects":-4753,"num_object_clones":0,"num_object_copies":-14259,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4753,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":4753,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.556540"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":457,"reported_epoch":282,"state":"active+clean+scrubbing","last_fresh":"2024-03-11T21:04:51.192452+0000","last_change":"2024-03-11T21:04:51.184615+0000","last_active":"2024-03-11T21:04:51.192452+0000","last_peered":"2024-03-11T21:04:51.192452+0000","last_clean":"2024-03-11T21:04:51.192452+0000","last_became_active":"2024-03-11T20:50:34.338390+0000","last_became_peered":"2024-03-11T20:50:34.338390+0000","last_unstale":"2024-03-11T21:04:51.192452+0000","last_undegraded":"2024-03-11T21:04:51.192452+0000","last_fullsized":"2024-03-11T21:04:51.192452+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:00:08.811756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:00:08.811756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"scrubbing for 0s","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":457,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:49.718404+0000","last_change":"2024-03-11T21:00:02.764408+0000","last_active":"2024-03-11T21:04:49.718404+0000","last_peered":"2024-03-11T21:04:49.718404+0000","last_clean":"2024-03-11T21:04:49.718404+0000","last_became_active":"2024-03-11T20:50:32.210194+0000","last_became_peered":"2024-03-11T20:50:32.210194+0000","last_unstale":"2024-03-11T21:04:49.718404+0000","last_undegraded":"2024-03-11T21:04:49.718404+0000","last_fullsized":"2024-03-11T21:04:49.718404+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:00:02.764190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:00:02.764190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":282,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:49.652583+0000","last_change":"2024-03-11T21:03:38.968156+0000","last_active":"2024-03-11T21:04:49.652583+0000","last_peered":"2024-03-11T21:04:49.652583+0000","last_clean":"2024-03-11T21:04:49.652583+0000","last_became_active":"2024-03-11T21:00:27.963600+0000","last_became_peered":"2024-03-11T21:00:27.963600+0000","last_unstale":"2024-03-11T21:04:49.652583+0000","last_undegraded":"2024-03-11T21:04:49.652583+0000","last_fullsized":"2024-03-11T21:04:49.652583+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:38.968078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:53:29.964907+0000","last_clean_scrub_stamp":"2024-03-11T21:03:38.968078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:05:07.073103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1e","version":"207'20","reported_seq":493,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:49.709269+0000","last_change":"2024-03-11T21:00:09.750591+0000","last_active":"2024-03-11T21:04:49.709269+0000","last_peered":"2024-03-11T21:04:49.709269+0000","last_clean":"2024-03-11T21:04:49.709269+0000","last_became_active":"2024-03-11T20:50:30.101721+0000","last_became_peered":"2024-03-11T20:50:30.101721+0000","last_unstale":"2024-03-11T21:04:49.709269+0000","last_undegraded":"2024-03-11T21:04:49.709269+0000","last_fullsized":"2024-03-11T21:04:49.709269+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'10","last_scrub_stamp":"2024-03-11T21:00:09.750456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:00:09.750456+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":429,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:49.725228+0000","last_change":"2024-03-11T21:04:41.638509+0000","last_active":"2024-03-11T21:04:49.725228+0000","last_peered":"2024-03-11T21:04:49.725228+0000","last_clean":"2024-03-11T21:04:49.725228+0000","last_became_active":"2024-03-11T21:00:48.467854+0000","last_became_peered":"2024-03-11T21:00:48.467854+0000","last_unstale":"2024-03-11T21:04:49.725228+0000","last_undegraded":"2024-03-11T21:04:49.725228+0000","last_fullsized":"2024-03-11T21:04:49.725228+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:04:41.638387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:55.545758+0000","last_clean_scrub_stamp":"2024-03-11T21:04:41.638387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:06:09.198177+0000","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":434,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:49.724145+0000","last_change":"2024-03-11T21:03:31.347920+0000","last_active":"2024-03-11T21:04:49.724145+0000","last_peered":"2024-03-11T21:04:49.724145+0000","last_clean":"2024-03-11T21:04:49.724145+0000","last_became_active":"2024-03-11T21:00:48.459957+0000","last_became_peered":"2024-03-11T21:00:48.459957+0000","last_unstale":"2024-03-11T21:04:49.724145+0000","last_undegraded":"2024-03-11T21:04:49.724145+0000","last_fullsized":"2024-03-11T21:04:49.724145+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:31.347841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:54:13.660325+0000","last_clean_scrub_stamp":"2024-03-11T21:03:31.347841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":467,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:49.708831+0000","last_change":"2024-03-11T21:03:40.675667+0000","last_active":"2024-03-11T21:04:49.708831+0000","last_peered":"2024-03-11T21:04:49.708831+0000","last_clean":"2024-03-11T21:04:49.708831+0000","last_became_active":"2024-03-11T20:50:28.058274+0000","last_became_peered":"2024-03-11T20:50:28.058274+0000","last_unstale":"2024-03-11T21:04:49.708831+0000","last_undegraded":"2024-03-11T21:04:49.708831+0000","last_fullsized":"2024-03-11T21:04:49.708831+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:40.675526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:03:40.675526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:51.747922+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"207'19","reported_seq":514,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:50.309458+0000","last_change":"2024-03-11T21:04:50.309531+0000","last_active":"2024-03-11T21:04:50.309458+0000","last_peered":"2024-03-11T21:04:50.309458+0000","last_clean":"2024-03-11T21:04:50.309458+0000","last_became_active":"2024-03-11T20:50:30.092638+0000","last_became_peered":"2024-03-11T20:50:30.092638+0000","last_unstale":"2024-03-11T21:04:50.309458+0000","last_undegraded":"2024-03-11T21:04:50.309458+0000","last_fullsized":"2024-03-11T21:04:50.309458+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'19","last_scrub_stamp":"2024-03-11T21:04:50.309326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:04:50.309326+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:06:16.159866+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":474,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:49.648452+0000","last_change":"2024-03-11T21:04:37.574370+0000","last_active":"2024-03-11T21:04:49.648452+0000","last_peered":"2024-03-11T21:04:49.648452+0000","last_clean":"2024-03-11T21:04:49.648452+0000","last_became_active":"2024-03-11T21:00:48.460694+0000","last_became_peered":"2024-03-11T21:00:48.460694+0000","last_unstale":"2024-03-11T21:04:49.648452+0000","last_undegraded":"2024-03-11T21:04:49.648452+0000","last_fullsized":"2024-03-11T21:04:49.648452+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:04:37.573961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:53:07.792697+0000","last_clean_scrub_stamp":"2024-03-11T21:04:37.573961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:05:39.316155+0000","scrub_duration":59,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":425,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:49.725615+0000","last_change":"2024-03-11T21:03:36.230810+0000","last_active":"2024-03-11T21:04:49.725615+0000","last_peered":"2024-03-11T21:04:49.725615+0000","last_clean":"2024-03-11T21:04:49.725615+0000","last_became_active":"2024-03-11T21:00:48.463839+0000","last_became_peered":"2024-03-11T21:00:48.463839+0000","last_unstale":"2024-03-11T21:04:49.725615+0000","last_undegraded":"2024-03-11T21:04:49.725615+0000","last_fullsized":"2024-03-11T21:04:49.725615+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:36.230647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:03:36.230647+0000","last_clean_scrub_stamp":"2024-03-11T21:03:36.230647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:05:04.850393+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":480,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:49.652674+0000","last_change":"2024-03-11T21:03:36.980913+0000","last_active":"2024-03-11T21:04:49.652674+0000","last_peered":"2024-03-11T21:04:49.652674+0000","last_clean":"2024-03-11T21:04:49.652674+0000","last_became_active":"2024-03-11T20:50:28.079225+0000","last_became_peered":"2024-03-11T20:50:28.079225+0000","last_unstale":"2024-03-11T21:04:49.652674+0000","last_undegraded":"2024-03-11T21:04:49.652674+0000","last_fullsized":"2024-03-11T21:04:49.652674+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:36.980825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:03:36.980825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"207'27","reported_seq":577,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:49.648866+0000","last_change":"2024-03-11T21:04:41.440261+0000","last_active":"2024-03-11T21:04:49.648866+0000","last_peered":"2024-03-11T21:04:49.648866+0000","last_clean":"2024-03-11T21:04:49.648866+0000","last_became_active":"2024-03-11T21:00:48.462943+0000","last_became_peered":"2024-03-11T21:00:48.462943+0000","last_unstale":"2024-03-11T21:04:49.648866+0000","last_undegraded":"2024-03-11T21:04:49.648866+0000","last_fullsized":"2024-03-11T21:04:49.648866+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'27","last_scrub_stamp":"2024-03-11T21:04:41.440129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:04:41.440129+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:05:44.838328+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":174,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-03-11T21:04:51.689295+0000","last_change":"2024-03-11T21:03:20.330025+0000","last_active":"2024-03-11T21:04:51.689295+0000","last_peered":"2024-03-11T21:04:51.689295+0000","last_clean":"2024-03-11T21:04:51.689295+0000","last_became_active":"2024-03-11T21:00:48.456017+0000","last_became_peered":"2024-03-11T21:00:48.456017+0000","last_unstale":"2024-03-11T21:04:51.689295+0000","last_undegraded":"2024-03-11T21:04:51.689295+0000","last_fullsized":"2024-03-11T21:04:51.689295+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:20.329913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:03:20.329913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":429,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:49.724550+0000","last_change":"2024-03-11T21:04:48.510504+0000","last_active":"2024-03-11T21:04:49.724550+0000","last_peered":"2024-03-11T21:04:49.724550+0000","last_clean":"2024-03-11T21:04:49.724550+0000","last_became_active":"2024-03-11T21:00:27.962251+0000","last_became_peered":"2024-03-11T21:00:27.962251+0000","last_unstale":"2024-03-11T21:04:49.724550+0000","last_undegraded":"2024-03-11T21:04:49.724550+0000","last_fullsized":"2024-03-11T21:04:49.724550+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:04:48.510370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:53:50.599029+0000","last_clean_scrub_stamp":"2024-03-11T21:04:48.510370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:06:06.017107+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":488,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:49.652082+0000","last_change":"2024-03-11T21:04:44.347647+0000","last_active":"2024-03-11T21:04:49.652082+0000","last_peered":"2024-03-11T21:04:49.652082+0000","last_clean":"2024-03-11T21:04:49.652082+0000","last_became_active":"2024-03-11T20:50:28.068978+0000","last_became_peered":"2024-03-11T20:50:28.068978+0000","last_unstale":"2024-03-11T21:04:49.652082+0000","last_undegraded":"2024-03-11T21:04:49.652082+0000","last_fullsized":"2024-03-11T21:04:49.652082+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:04:44.347540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:54:07.155130+0000","last_clean_scrub_stamp":"2024-03-11T21:04:44.347540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:06:07.963405+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"207'23","reported_seq":515,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:49.649336+0000","last_change":"2024-03-11T21:04:49.467165+0000","last_active":"2024-03-11T21:04:49.649336+0000","last_peered":"2024-03-11T21:04:49.649336+0000","last_clean":"2024-03-11T21:04:49.649336+0000","last_became_active":"2024-03-11T21:00:27.952093+0000","last_became_peered":"2024-03-11T21:00:27.952093+0000","last_unstale":"2024-03-11T21:04:49.649336+0000","last_undegraded":"2024-03-11T21:04:49.649336+0000","last_fullsized":"2024-03-11T21:04:49.649336+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'23","last_scrub_stamp":"2024-03-11T21:04:49.467082+0000","last_deep_scrub":"110'17","last_deep_scrub_stamp":"2024-03-11T20:53:04.802488+0000","last_clean_scrub_stamp":"2024-03-11T21:04:49.467082+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:05:54.486191+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":454,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:49.724911+0000","last_change":"2024-03-11T21:00:13.778710+0000","last_active":"2024-03-11T21:04:49.724911+0000","last_peered":"2024-03-11T21:04:49.724911+0000","last_clean":"2024-03-11T21:04:49.724911+0000","last_became_active":"2024-03-11T20:50:34.334558+0000","last_became_peered":"2024-03-11T20:50:34.334558+0000","last_unstale":"2024-03-11T21:04:49.724911+0000","last_undegraded":"2024-03-11T21:04:49.724911+0000","last_fullsized":"2024-03-11T21:04:49.724911+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:00:13.778532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:55:59.892394+0000","last_clean_scrub_stamp":"2024-03-11T21:00:13.778532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":87,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":305,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:49.649079+0000","last_change":"2024-03-11T21:03:48.239121+0000","last_active":"2024-03-11T21:04:49.649079+0000","last_peered":"2024-03-11T21:04:49.649079+0000","last_clean":"2024-03-11T21:04:49.649079+0000","last_became_active":"2024-03-11T21:00:48.461013+0000","last_became_peered":"2024-03-11T21:00:48.461013+0000","last_unstale":"2024-03-11T21:04:49.649079+0000","last_undegraded":"2024-03-11T21:04:49.649079+0000","last_fullsized":"2024-03-11T21:04:49.649079+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:48.239009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:03:48.239009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:51.626936+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"77'1","reported_seq":481,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:49.713367+0000","last_change":"2024-03-11T21:04:44.146657+0000","last_active":"2024-03-11T21:04:49.713367+0000","last_peered":"2024-03-11T21:04:49.713367+0000","last_clean":"2024-03-11T21:04:49.713367+0000","last_became_active":"2024-03-11T20:50:28.068840+0000","last_became_peered":"2024-03-11T20:50:28.068840+0000","last_unstale":"2024-03-11T21:04:49.713367+0000","last_undegraded":"2024-03-11T21:04:49.713367+0000","last_fullsized":"2024-03-11T21:04:49.713367+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-03-11T21:04:44.142991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:04:44.142991+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:05:57.087853+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"207'19","reported_seq":495,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:49.725175+0000","last_change":"2024-03-11T21:03:39.240266+0000","last_active":"2024-03-11T21:04:49.725175+0000","last_peered":"2024-03-11T21:04:49.725175+0000","last_clean":"2024-03-11T21:04:49.725175+0000","last_became_active":"2024-03-11T21:00:48.468434+0000","last_became_peered":"2024-03-11T21:00:48.468434+0000","last_unstale":"2024-03-11T21:04:49.725175+0000","last_undegraded":"2024-03-11T21:04:49.725175+0000","last_fullsized":"2024-03-11T21:04:49.725175+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'19","last_scrub_stamp":"2024-03-11T21:03:39.240089+0000","last_deep_scrub":"207'19","last_deep_scrub_stamp":"2024-03-11T21:03:39.240089+0000","last_clean_scrub_stamp":"2024-03-11T21:03:39.240089+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:05:07.484049+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":424,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:49.726668+0000","last_change":"2024-03-11T21:04:42.634122+0000","last_active":"2024-03-11T21:04:49.726668+0000","last_peered":"2024-03-11T21:04:49.726668+0000","last_clean":"2024-03-11T21:04:49.726668+0000","last_became_active":"2024-03-11T21:00:27.955250+0000","last_became_peered":"2024-03-11T21:00:27.955250+0000","last_unstale":"2024-03-11T21:04:49.726668+0000","last_undegraded":"2024-03-11T21:04:49.726668+0000","last_fullsized":"2024-03-11T21:04:49.726668+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:04:42.633895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:33.399510+0000","last_clean_scrub_stamp":"2024-03-11T21:04:42.633895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:05:53.435844+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":505,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:49.648822+0000","last_change":"2024-03-11T21:03:29.516265+0000","last_active":"2024-03-11T21:04:49.648822+0000","last_peered":"2024-03-11T21:04:49.648822+0000","last_clean":"2024-03-11T21:04:49.648822+0000","last_became_active":"2024-03-11T21:00:48.459135+0000","last_became_peered":"2024-03-11T21:00:48.459135+0000","last_unstale":"2024-03-11T21:04:49.648822+0000","last_undegraded":"2024-03-11T21:04:49.648822+0000","last_fullsized":"2024-03-11T21:04:49.648822+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:29.516124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:24.666325+0000","last_clean_scrub_stamp":"2024-03-11T21:03:29.516124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":488,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:49.710753+0000","last_change":"2024-03-11T21:04:31.654959+0000","last_active":"2024-03-11T21:04:49.710753+0000","last_peered":"2024-03-11T21:04:49.710753+0000","last_clean":"2024-03-11T21:04:49.710753+0000","last_became_active":"2024-03-11T21:00:27.956935+0000","last_became_peered":"2024-03-11T21:00:27.956935+0000","last_unstale":"2024-03-11T21:04:49.710753+0000","last_undegraded":"2024-03-11T21:04:49.710753+0000","last_fullsized":"2024-03-11T21:04:49.710753+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:04:31.654834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:04:31.654834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:05:39.447632+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"207'7","reported_seq":496,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:49.717337+0000","last_change":"2024-03-11T21:04:36.187975+0000","last_active":"2024-03-11T21:04:49.717337+0000","last_peered":"2024-03-11T21:04:49.717337+0000","last_clean":"2024-03-11T21:04:49.717337+0000","last_became_active":"2024-03-11T21:00:27.955296+0000","last_became_peered":"2024-03-11T21:00:27.955296+0000","last_unstale":"2024-03-11T21:04:49.717337+0000","last_undegraded":"2024-03-11T21:04:49.717337+0000","last_fullsized":"2024-03-11T21:04:49.717337+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'7","last_scrub_stamp":"2024-03-11T21:04:36.187771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:04:36.187771+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:05:56.838761+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":182,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-03-11T21:04:51.688377+0000","last_change":"2024-03-11T21:03:42.593360+0000","last_active":"2024-03-11T21:04:51.688377+0000","last_peered":"2024-03-11T21:04:51.688377+0000","last_clean":"2024-03-11T21:04:51.688377+0000","last_became_active":"2024-03-11T21:00:48.464694+0000","last_became_peered":"2024-03-11T21:00:48.464694+0000","last_unstale":"2024-03-11T21:04:51.688377+0000","last_undegraded":"2024-03-11T21:04:51.688377+0000","last_fullsized":"2024-03-11T21:04:51.688377+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:42.593243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:03:42.593243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1f","version":"212'16","reported_seq":687,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:49.711686+0000","last_change":"2024-03-11T21:04:44.711673+0000","last_active":"2024-03-11T21:04:49.711686+0000","last_peered":"2024-03-11T21:04:49.711686+0000","last_clean":"2024-03-11T21:04:49.711686+0000","last_became_active":"2024-03-11T21:00:48.464350+0000","last_became_peered":"2024-03-11T21:00:48.464350+0000","last_unstale":"2024-03-11T21:04:49.711686+0000","last_undegraded":"2024-03-11T21:04:49.711686+0000","last_fullsized":"2024-03-11T21:04:49.711686+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"212'16","last_scrub_stamp":"2024-03-11T21:04:44.711564+0000","last_deep_scrub":"114'12","last_deep_scrub_stamp":"2024-03-11T20:53:30.905791+0000","last_clean_scrub_stamp":"2024-03-11T21:04:44.711564+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:06:01.713334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"77'1","reported_seq":466,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:49.719281+0000","last_change":"2024-03-11T21:04:31.272900+0000","last_active":"2024-03-11T21:04:49.719281+0000","last_peered":"2024-03-11T21:04:49.719281+0000","last_clean":"2024-03-11T21:04:49.719281+0000","last_became_active":"2024-03-11T20:50:28.068827+0000","last_became_peered":"2024-03-11T20:50:28.068827+0000","last_unstale":"2024-03-11T21:04:49.719281+0000","last_undegraded":"2024-03-11T21:04:49.719281+0000","last_fullsized":"2024-03-11T21:04:49.719281+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-03-11T21:04:31.272755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:04:31.272755+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:05:49.263125+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"207'15","reported_seq":528,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:49.714319+0000","last_change":"2024-03-11T21:02:30.560424+0000","last_active":"2024-03-11T21:04:49.714319+0000","last_peered":"2024-03-11T21:04:49.714319+0000","last_clean":"2024-03-11T21:04:49.714319+0000","last_became_active":"2024-03-11T21:00:48.467513+0000","last_became_peered":"2024-03-11T21:00:48.467513+0000","last_unstale":"2024-03-11T21:04:49.714319+0000","last_undegraded":"2024-03-11T21:04:49.714319+0000","last_fullsized":"2024-03-11T21:04:49.714319+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'15","last_scrub_stamp":"2024-03-11T21:02:30.560293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:02:30.560293+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"105'2","reported_seq":471,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:49.652282+0000","last_change":"2024-03-11T21:03:41.972426+0000","last_active":"2024-03-11T21:04:49.652282+0000","last_peered":"2024-03-11T21:04:49.652282+0000","last_clean":"2024-03-11T21:04:49.652282+0000","last_became_active":"2024-03-11T20:50:34.327356+0000","last_became_peered":"2024-03-11T20:50:34.327356+0000","last_unstale":"2024-03-11T21:04:49.652282+0000","last_undegraded":"2024-03-11T21:04:49.652282+0000","last_fullsized":"2024-03-11T21:04:49.652282+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'2","last_scrub_stamp":"2024-03-11T21:03:41.971918+0000","last_deep_scrub":"105'2","last_deep_scrub_stamp":"2024-03-11T21:03:41.971918+0000","last_clean_scrub_stamp":"2024-03-11T21:03:41.971918+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":289,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:49.652634+0000","last_change":"2024-03-11T21:04:38.282820+0000","last_active":"2024-03-11T21:04:49.652634+0000","last_peered":"2024-03-11T21:04:49.652634+0000","last_clean":"2024-03-11T21:04:49.652634+0000","last_became_active":"2024-03-11T20:56:59.804264+0000","last_became_peered":"2024-03-11T20:56:59.804264+0000","last_unstale":"2024-03-11T21:04:49.652634+0000","last_undegraded":"2024-03-11T21:04:49.652634+0000","last_fullsized":"2024-03-11T21:04:49.652634+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:04:38.282708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:59:14.605681+0000","last_clean_scrub_stamp":"2024-03-11T21:04:38.282708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:05:50.289383+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":474,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:49.648300+0000","last_change":"2024-03-11T21:03:09.746912+0000","last_active":"2024-03-11T21:04:49.648300+0000","last_peered":"2024-03-11T21:04:49.648300+0000","last_clean":"2024-03-11T21:04:49.648300+0000","last_became_active":"2024-03-11T21:00:27.947614+0000","last_became_peered":"2024-03-11T21:00:27.947614+0000","last_unstale":"2024-03-11T21:04:49.648300+0000","last_undegraded":"2024-03-11T21:04:49.648300+0000","last_fullsized":"2024-03-11T21:04:49.648300+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:09.746789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:52:01.880052+0000","last_clean_scrub_stamp":"2024-03-11T21:03:09.746789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"207'27","reported_seq":212,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-03-11T21:04:51.686717+0000","last_change":"2024-03-11T21:03:27.406786+0000","last_active":"2024-03-11T21:04:51.686717+0000","last_peered":"2024-03-11T21:04:51.686717+0000","last_clean":"2024-03-11T21:04:51.686717+0000","last_became_active":"2024-03-11T21:00:48.457893+0000","last_became_peered":"2024-03-11T21:00:48.457893+0000","last_unstale":"2024-03-11T21:04:51.686717+0000","last_undegraded":"2024-03-11T21:04:51.686717+0000","last_fullsized":"2024-03-11T21:04:51.686717+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'27","last_scrub_stamp":"2024-03-11T21:03:27.406573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:03:27.406573+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":462,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:50.207546+0000","last_change":"2024-03-11T21:04:50.207587+0000","last_active":"2024-03-11T21:04:50.207546+0000","last_peered":"2024-03-11T21:04:50.207546+0000","last_clean":"2024-03-11T21:04:50.207546+0000","last_became_active":"2024-03-11T20:50:34.337878+0000","last_became_peered":"2024-03-11T20:50:34.337878+0000","last_unstale":"2024-03-11T21:04:50.207546+0000","last_undegraded":"2024-03-11T21:04:50.207546+0000","last_fullsized":"2024-03-11T21:04:50.207546+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:04:50.207473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:52:11.462480+0000","last_clean_scrub_stamp":"2024-03-11T21:04:50.207473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:05:59.001511+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":466,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:49.720082+0000","last_change":"2024-03-11T21:04:40.178107+0000","last_active":"2024-03-11T21:04:49.720082+0000","last_peered":"2024-03-11T21:04:49.720082+0000","last_clean":"2024-03-11T21:04:49.720082+0000","last_became_active":"2024-03-11T20:50:32.646323+0000","last_became_peered":"2024-03-11T20:50:32.646323+0000","last_unstale":"2024-03-11T21:04:49.720082+0000","last_undegraded":"2024-03-11T21:04:49.720082+0000","last_fullsized":"2024-03-11T21:04:49.720082+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:04:40.177993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:54:06.516412+0000","last_clean_scrub_stamp":"2024-03-11T21:04:40.177993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:05:43.194670+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":495,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:49.711937+0000","last_change":"2024-03-11T21:04:47.711141+0000","last_active":"2024-03-11T21:04:49.711937+0000","last_peered":"2024-03-11T21:04:49.711937+0000","last_clean":"2024-03-11T21:04:49.711937+0000","last_became_active":"2024-03-11T20:50:28.086513+0000","last_became_peered":"2024-03-11T20:50:28.086513+0000","last_unstale":"2024-03-11T21:04:49.711937+0000","last_undegraded":"2024-03-11T21:04:49.711937+0000","last_fullsized":"2024-03-11T21:04:49.711937+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:04:47.711056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:04:47.711056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:05:48.646359+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"207'19","reported_seq":540,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:49.648779+0000","last_change":"2024-03-11T21:04:35.480773+0000","last_active":"2024-03-11T21:04:49.648779+0000","last_peered":"2024-03-11T21:04:49.648779+0000","last_clean":"2024-03-11T21:04:49.648779+0000","last_became_active":"2024-03-11T21:00:48.464707+0000","last_became_peered":"2024-03-11T21:00:48.464707+0000","last_unstale":"2024-03-11T21:04:49.648779+0000","last_undegraded":"2024-03-11T21:04:49.648779+0000","last_fullsized":"2024-03-11T21:04:49.648779+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'19","last_scrub_stamp":"2024-03-11T21:04:35.480678+0000","last_deep_scrub":"145'13","last_deep_scrub_stamp":"2024-03-11T20:55:53.560287+0000","last_clean_scrub_stamp":"2024-03-11T21:04:35.480678+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:05:45.517483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":470,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:49.652049+0000","last_change":"2024-03-11T21:03:24.925576+0000","last_active":"2024-03-11T21:04:49.652049+0000","last_peered":"2024-03-11T21:04:49.652049+0000","last_clean":"2024-03-11T21:04:49.652049+0000","last_became_active":"2024-03-11T21:00:27.973405+0000","last_became_peered":"2024-03-11T21:00:27.973405+0000","last_unstale":"2024-03-11T21:04:49.652049+0000","last_undegraded":"2024-03-11T21:04:49.652049+0000","last_fullsized":"2024-03-11T21:04:49.652049+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:24.925436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:03:24.925436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":471,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:49.714305+0000","last_change":"2024-03-11T21:02:31.549151+0000","last_active":"2024-03-11T21:04:49.714305+0000","last_peered":"2024-03-11T21:04:49.714305+0000","last_clean":"2024-03-11T21:04:49.714305+0000","last_became_active":"2024-03-11T21:00:48.472812+0000","last_became_peered":"2024-03-11T21:00:48.472812+0000","last_unstale":"2024-03-11T21:04:49.714305+0000","last_undegraded":"2024-03-11T21:04:49.714305+0000","last_fullsized":"2024-03-11T21:04:49.714305+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:02:31.549032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:02:31.549032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":481,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:49.648714+0000","last_change":"2024-03-11T21:03:33.835941+0000","last_active":"2024-03-11T21:04:49.648714+0000","last_peered":"2024-03-11T21:04:49.648714+0000","last_clean":"2024-03-11T21:04:49.648714+0000","last_became_active":"2024-03-11T20:50:28.075760+0000","last_became_peered":"2024-03-11T20:50:28.075760+0000","last_unstale":"2024-03-11T21:04:49.648714+0000","last_undegraded":"2024-03-11T21:04:49.648714+0000","last_fullsized":"2024-03-11T21:04:49.648714+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:33.835839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:58:55.977289+0000","last_clean_scrub_stamp":"2024-03-11T21:03:33.835839+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"207'12","reported_seq":327,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:49.649078+0000","last_change":"2024-03-11T21:03:21.879821+0000","last_active":"2024-03-11T21:04:49.649078+0000","last_peered":"2024-03-11T21:04:49.649078+0000","last_clean":"2024-03-11T21:04:49.649078+0000","last_became_active":"2024-03-11T20:56:59.794581+0000","last_became_peered":"2024-03-11T20:56:59.794581+0000","last_unstale":"2024-03-11T21:04:49.649078+0000","last_undegraded":"2024-03-11T21:04:49.649078+0000","last_fullsized":"2024-03-11T21:04:49.649078+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'12","last_scrub_stamp":"2024-03-11T21:03:21.879683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:03:21.879683+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:50.694254+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"186'2","reported_seq":456,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:49.717899+0000","last_change":"2024-03-11T21:00:17.739831+0000","last_active":"2024-03-11T21:04:49.717899+0000","last_peered":"2024-03-11T21:04:49.717899+0000","last_clean":"2024-03-11T21:04:49.717899+0000","last_became_active":"2024-03-11T20:50:34.339326+0000","last_became_peered":"2024-03-11T20:50:34.339326+0000","last_unstale":"2024-03-11T21:04:49.717899+0000","last_undegraded":"2024-03-11T21:04:49.717899+0000","last_fullsized":"2024-03-11T21:04:49.717899+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"186'2","last_scrub_stamp":"2024-03-11T21:00:17.739696+0000","last_deep_scrub":"186'2","last_deep_scrub_stamp":"2024-03-11T20:59:04.534850+0000","last_clean_scrub_stamp":"2024-03-11T21:00:17.739696+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":434,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:49.726292+0000","last_change":"2024-03-11T21:03:45.273658+0000","last_active":"2024-03-11T21:04:49.726292+0000","last_peered":"2024-03-11T21:04:49.726292+0000","last_clean":"2024-03-11T21:04:49.726292+0000","last_became_active":"2024-03-11T21:00:48.462656+0000","last_became_peered":"2024-03-11T21:00:48.462656+0000","last_unstale":"2024-03-11T21:04:49.726292+0000","last_undegraded":"2024-03-11T21:04:49.726292+0000","last_fullsized":"2024-03-11T21:04:49.726292+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:45.273529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:03:45.273529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:05:08.865958+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":174,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-03-11T21:04:51.688392+0000","last_change":"2024-03-11T21:03:23.365771+0000","last_active":"2024-03-11T21:04:51.688392+0000","last_peered":"2024-03-11T21:04:51.688392+0000","last_clean":"2024-03-11T21:04:51.688392+0000","last_became_active":"2024-03-11T21:00:48.455510+0000","last_became_peered":"2024-03-11T21:00:48.455510+0000","last_unstale":"2024-03-11T21:04:51.688392+0000","last_undegraded":"2024-03-11T21:04:51.688392+0000","last_fullsized":"2024-03-11T21:04:51.688392+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:23.365625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:03:23.365625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"207'25","reported_seq":427,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:49.728486+0000","last_change":"2024-03-11T21:03:26.409080+0000","last_active":"2024-03-11T21:04:49.728486+0000","last_peered":"2024-03-11T21:04:49.728486+0000","last_clean":"2024-03-11T21:04:49.728486+0000","last_became_active":"2024-03-11T21:00:27.959547+0000","last_became_peered":"2024-03-11T21:00:27.959547+0000","last_unstale":"2024-03-11T21:04:49.728486+0000","last_undegraded":"2024-03-11T21:04:49.728486+0000","last_fullsized":"2024-03-11T21:04:49.728486+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'25","last_scrub_stamp":"2024-03-11T21:03:26.408832+0000","last_deep_scrub":"161'15","last_deep_scrub_stamp":"2024-03-11T20:57:15.125548+0000","last_clean_scrub_stamp":"2024-03-11T21:03:26.408832+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":488,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:49.715656+0000","last_change":"2024-03-11T21:04:41.212905+0000","last_active":"2024-03-11T21:04:49.715656+0000","last_peered":"2024-03-11T21:04:49.715656+0000","last_clean":"2024-03-11T21:04:49.715656+0000","last_became_active":"2024-03-11T21:00:48.462105+0000","last_became_peered":"2024-03-11T21:00:48.462105+0000","last_unstale":"2024-03-11T21:04:49.715656+0000","last_undegraded":"2024-03-11T21:04:49.715656+0000","last_fullsized":"2024-03-11T21:04:49.715656+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:04:41.212752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:04:41.212752+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:05:57.804292+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":182,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-03-11T21:04:51.690467+0000","last_change":"2024-03-11T21:03:43.605939+0000","last_active":"2024-03-11T21:04:51.690467+0000","last_peered":"2024-03-11T21:04:51.690467+0000","last_clean":"2024-03-11T21:04:51.690467+0000","last_became_active":"2024-03-11T21:00:48.459043+0000","last_became_peered":"2024-03-11T21:00:48.459043+0000","last_unstale":"2024-03-11T21:04:51.690467+0000","last_undegraded":"2024-03-11T21:04:51.690467+0000","last_fullsized":"2024-03-11T21:04:51.690467+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:43.605796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:23.137832+0000","last_clean_scrub_stamp":"2024-03-11T21:03:43.605796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:05:13.569217+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":488,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:49.710804+0000","last_change":"2024-03-11T21:03:36.409060+0000","last_active":"2024-03-11T21:04:49.710804+0000","last_peered":"2024-03-11T21:04:49.710804+0000","last_clean":"2024-03-11T21:04:49.710804+0000","last_became_active":"2024-03-11T20:50:28.082737+0000","last_became_peered":"2024-03-11T20:50:28.082737+0000","last_unstale":"2024-03-11T21:04:49.710804+0000","last_undegraded":"2024-03-11T21:04:49.710804+0000","last_fullsized":"2024-03-11T21:04:49.710804+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:36.408942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:28.404997+0000","last_clean_scrub_stamp":"2024-03-11T21:03:36.408942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:57.355001+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"207'15","reported_seq":502,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:49.652827+0000","last_change":"2024-03-11T21:03:47.989635+0000","last_active":"2024-03-11T21:04:49.652827+0000","last_peered":"2024-03-11T21:04:49.652827+0000","last_clean":"2024-03-11T21:04:49.652827+0000","last_became_active":"2024-03-11T21:00:27.967624+0000","last_became_peered":"2024-03-11T21:00:27.967624+0000","last_unstale":"2024-03-11T21:04:49.652827+0000","last_undegraded":"2024-03-11T21:04:49.652827+0000","last_fullsized":"2024-03-11T21:04:49.652827+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'15","last_scrub_stamp":"2024-03-11T21:03:47.989410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:03:47.989410+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:58.974121+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":384,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:49.726799+0000","last_change":"2024-03-11T21:03:44.322295+0000","last_active":"2024-03-11T21:04:49.726799+0000","last_peered":"2024-03-11T21:04:49.726799+0000","last_clean":"2024-03-11T21:04:49.726799+0000","last_became_active":"2024-03-11T21:00:27.958384+0000","last_became_peered":"2024-03-11T21:00:27.958384+0000","last_unstale":"2024-03-11T21:04:49.726799+0000","last_undegraded":"2024-03-11T21:04:49.726799+0000","last_fullsized":"2024-03-11T21:04:49.726799+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:44.322168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:57:16.032302+0000","last_clean_scrub_stamp":"2024-03-11T21:03:44.322168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:05:12.080668+0000","scrub_duration":59,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.15","version":"112'12","reported_seq":649,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:49.649492+0000","last_change":"2024-03-11T21:03:43.933651+0000","last_active":"2024-03-11T21:04:49.649492+0000","last_peered":"2024-03-11T21:04:49.649492+0000","last_clean":"2024-03-11T21:04:49.649492+0000","last_became_active":"2024-03-11T20:50:32.650064+0000","last_became_peered":"2024-03-11T20:50:32.650064+0000","last_unstale":"2024-03-11T21:04:49.649492+0000","last_undegraded":"2024-03-11T21:04:49.649492+0000","last_fullsized":"2024-03-11T21:04:49.649492+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'12","last_scrub_stamp":"2024-03-11T21:03:43.933551+0000","last_deep_scrub":"112'12","last_deep_scrub_stamp":"2024-03-11T20:53:11.843951+0000","last_clean_scrub_stamp":"2024-03-11T21:03:43.933551+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:49.815489+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":235,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:49.725945+0000","last_change":"2024-03-11T21:03:11.307482+0000","last_active":"2024-03-11T21:04:49.725945+0000","last_peered":"2024-03-11T21:04:49.725945+0000","last_clean":"2024-03-11T21:04:49.725945+0000","last_became_active":"2024-03-11T21:00:27.972907+0000","last_became_peered":"2024-03-11T21:00:27.972907+0000","last_unstale":"2024-03-11T21:04:49.725945+0000","last_undegraded":"2024-03-11T21:04:49.725945+0000","last_fullsized":"2024-03-11T21:04:49.725945+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:11.307332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:03:11.307332+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.12","version":"207'19","reported_seq":424,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:49.651923+0000","last_change":"2024-03-11T21:04:43.346812+0000","last_active":"2024-03-11T21:04:49.651923+0000","last_peered":"2024-03-11T21:04:49.651923+0000","last_clean":"2024-03-11T21:04:49.651923+0000","last_became_active":"2024-03-11T20:56:59.796618+0000","last_became_peered":"2024-03-11T20:56:59.796618+0000","last_unstale":"2024-03-11T21:04:49.651923+0000","last_undegraded":"2024-03-11T21:04:49.651923+0000","last_fullsized":"2024-03-11T21:04:49.651923+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'19","last_scrub_stamp":"2024-03-11T21:04:43.346658+0000","last_deep_scrub":"119'13","last_deep_scrub_stamp":"2024-03-11T20:54:15.443472+0000","last_clean_scrub_stamp":"2024-03-11T21:04:43.346658+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:05:54.569465+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":281,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:49.652451+0000","last_change":"2024-03-11T21:03:10.124323+0000","last_active":"2024-03-11T21:04:49.652451+0000","last_peered":"2024-03-11T21:04:49.652451+0000","last_clean":"2024-03-11T21:04:49.652451+0000","last_became_active":"2024-03-11T20:56:59.804346+0000","last_became_peered":"2024-03-11T20:56:59.804346+0000","last_unstale":"2024-03-11T21:04:49.652451+0000","last_undegraded":"2024-03-11T21:04:49.652451+0000","last_fullsized":"2024-03-11T21:04:49.652451+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:10.124075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:03:10.124075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"212'17","reported_seq":673,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:49.709290+0000","last_change":"2024-03-11T21:02:25.533896+0000","last_active":"2024-03-11T21:04:49.709290+0000","last_peered":"2024-03-11T21:04:49.709290+0000","last_clean":"2024-03-11T21:04:49.709290+0000","last_became_active":"2024-03-11T21:00:48.467116+0000","last_became_peered":"2024-03-11T21:00:48.467116+0000","last_unstale":"2024-03-11T21:04:49.709290+0000","last_undegraded":"2024-03-11T21:04:49.709290+0000","last_fullsized":"2024-03-11T21:04:49.709290+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"212'17","last_scrub_stamp":"2024-03-11T21:02:25.533666+0000","last_deep_scrub":"110'13","last_deep_scrub_stamp":"2024-03-11T20:53:00.434532+0000","last_clean_scrub_stamp":"2024-03-11T21:02:25.533666+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":482,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:49.650575+0000","last_change":"2024-03-11T21:03:29.061500+0000","last_active":"2024-03-11T21:04:49.650575+0000","last_peered":"2024-03-11T21:04:49.650575+0000","last_clean":"2024-03-11T21:04:49.650575+0000","last_became_active":"2024-03-11T20:50:28.070325+0000","last_became_peered":"2024-03-11T20:50:28.070325+0000","last_unstale":"2024-03-11T21:04:49.650575+0000","last_undegraded":"2024-03-11T21:04:49.650575+0000","last_fullsized":"2024-03-11T21:04:49.650575+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:29.061407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:03:29.061407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":207,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"207'24","reported_seq":514,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:49.652217+0000","last_change":"2024-03-11T21:03:45.977452+0000","last_active":"2024-03-11T21:04:49.652217+0000","last_peered":"2024-03-11T21:04:49.652217+0000","last_clean":"2024-03-11T21:04:49.652217+0000","last_became_active":"2024-03-11T21:00:27.967078+0000","last_became_peered":"2024-03-11T21:00:27.967078+0000","last_unstale":"2024-03-11T21:04:49.652217+0000","last_undegraded":"2024-03-11T21:04:49.652217+0000","last_fullsized":"2024-03-11T21:04:49.652217+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'24","last_scrub_stamp":"2024-03-11T21:03:45.977203+0000","last_deep_scrub":"116'16","last_deep_scrub_stamp":"2024-03-11T20:53:39.292118+0000","last_clean_scrub_stamp":"2024-03-11T21:03:45.977203+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:56.354611+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":461,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:49.652279+0000","last_change":"2024-03-11T21:03:39.998420+0000","last_active":"2024-03-11T21:04:49.652279+0000","last_peered":"2024-03-11T21:04:49.652279+0000","last_clean":"2024-03-11T21:04:49.652279+0000","last_became_active":"2024-03-11T21:00:27.966997+0000","last_became_peered":"2024-03-11T21:00:27.966997+0000","last_unstale":"2024-03-11T21:04:49.652279+0000","last_undegraded":"2024-03-11T21:04:49.652279+0000","last_fullsized":"2024-03-11T21:04:49.652279+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:39.998251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:55:58.016988+0000","last_clean_scrub_stamp":"2024-03-11T21:03:39.998251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:50.834709+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":478,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:49.724967+0000","last_change":"2024-03-11T21:03:15.618457+0000","last_active":"2024-03-11T21:04:49.724967+0000","last_peered":"2024-03-11T21:04:49.724967+0000","last_clean":"2024-03-11T21:04:49.724967+0000","last_became_active":"2024-03-11T21:00:48.468386+0000","last_became_peered":"2024-03-11T21:00:48.468386+0000","last_unstale":"2024-03-11T21:04:49.724967+0000","last_undegraded":"2024-03-11T21:04:49.724967+0000","last_fullsized":"2024-03-11T21:04:49.724967+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:15.618335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:03:15.618335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"166'2","reported_seq":523,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:49.713769+0000","last_change":"2024-03-11T21:04:38.617364+0000","last_active":"2024-03-11T21:04:49.713769+0000","last_peered":"2024-03-11T21:04:49.713769+0000","last_clean":"2024-03-11T21:04:49.713769+0000","last_became_active":"2024-03-11T21:00:48.466071+0000","last_became_peered":"2024-03-11T21:00:48.466071+0000","last_unstale":"2024-03-11T21:04:49.713769+0000","last_undegraded":"2024-03-11T21:04:49.713769+0000","last_fullsized":"2024-03-11T21:04:49.713769+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'2","last_scrub_stamp":"2024-03-11T21:04:38.617120+0000","last_deep_scrub":"166'2","last_deep_scrub_stamp":"2024-03-11T20:57:39.868776+0000","last_clean_scrub_stamp":"2024-03-11T21:04:38.617120+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:05:52.369686+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1822,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"207'8","reported_seq":488,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:49.710293+0000","last_change":"2024-03-11T21:03:50.252798+0000","last_active":"2024-03-11T21:04:49.710293+0000","last_peered":"2024-03-11T21:04:49.710293+0000","last_clean":"2024-03-11T21:04:49.710293+0000","last_became_active":"2024-03-11T20:50:30.101572+0000","last_became_peered":"2024-03-11T20:50:30.101572+0000","last_unstale":"2024-03-11T21:04:49.710293+0000","last_undegraded":"2024-03-11T21:04:49.710293+0000","last_fullsized":"2024-03-11T21:04:49.710293+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'8","last_scrub_stamp":"2024-03-11T21:03:50.252595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:03:50.252595+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:05:15.770439+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":489,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:49.648911+0000","last_change":"2024-03-11T21:04:46.938493+0000","last_active":"2024-03-11T21:04:49.648911+0000","last_peered":"2024-03-11T21:04:49.648911+0000","last_clean":"2024-03-11T21:04:49.648911+0000","last_became_active":"2024-03-11T21:00:27.964140+0000","last_became_peered":"2024-03-11T21:00:27.964140+0000","last_unstale":"2024-03-11T21:04:49.648911+0000","last_undegraded":"2024-03-11T21:04:49.648911+0000","last_fullsized":"2024-03-11T21:04:49.648911+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:04:46.938407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:04:46.938407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:06:10.287530+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":439,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:49.649059+0000","last_change":"2024-03-11T21:04:35.005443+0000","last_active":"2024-03-11T21:04:49.649059+0000","last_peered":"2024-03-11T21:04:49.649059+0000","last_clean":"2024-03-11T21:04:49.649059+0000","last_became_active":"2024-03-11T20:56:59.802356+0000","last_became_peered":"2024-03-11T20:56:59.802356+0000","last_unstale":"2024-03-11T21:04:49.649059+0000","last_undegraded":"2024-03-11T21:04:49.649059+0000","last_fullsized":"2024-03-11T21:04:49.649059+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:04:35.004850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:54:09.537972+0000","last_clean_scrub_stamp":"2024-03-11T21:04:35.004850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:06:02.348576+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":502,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:49.648205+0000","last_change":"2024-03-11T21:04:38.986316+0000","last_active":"2024-03-11T21:04:49.648205+0000","last_peered":"2024-03-11T21:04:49.648205+0000","last_clean":"2024-03-11T21:04:49.648205+0000","last_became_active":"2024-03-11T21:00:48.459557+0000","last_became_peered":"2024-03-11T21:00:48.459557+0000","last_unstale":"2024-03-11T21:04:49.648205+0000","last_undegraded":"2024-03-11T21:04:49.648205+0000","last_fullsized":"2024-03-11T21:04:49.648205+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:04:38.986197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:18.682947+0000","last_clean_scrub_stamp":"2024-03-11T21:04:38.986197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:05:50.543638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"262'24","reported_seq":506,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:49.651969+0000","last_change":"2024-03-11T21:03:28.129057+0000","last_active":"2024-03-11T21:04:49.651969+0000","last_peered":"2024-03-11T21:04:49.651969+0000","last_clean":"2024-03-11T21:04:49.651969+0000","last_became_active":"2024-03-11T21:00:27.970878+0000","last_became_peered":"2024-03-11T21:00:27.970878+0000","last_unstale":"2024-03-11T21:04:49.651969+0000","last_undegraded":"2024-03-11T21:04:49.651969+0000","last_fullsized":"2024-03-11T21:04:49.651969+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"262'24","last_scrub_stamp":"2024-03-11T21:03:28.128719+0000","last_deep_scrub":"262'24","last_deep_scrub_stamp":"2024-03-11T21:03:28.128719+0000","last_clean_scrub_stamp":"2024-03-11T21:03:28.128719+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":127,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.4","version":"283'8174","reported_seq":8317,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-03-11T21:04:51.940412+0000","last_change":"2024-03-11T21:04:47.235472+0000","last_active":"2024-03-11T21:04:51.940412+0000","last_peered":"2024-03-11T21:04:51.940412+0000","last_clean":"2024-03-11T21:04:51.940412+0000","last_became_active":"2024-03-11T21:02:47.634842+0000","last_became_peered":"2024-03-11T21:02:47.634842+0000","last_unstale":"2024-03-11T21:04:51.940412+0000","last_undegraded":"2024-03-11T21:04:51.940412+0000","last_fullsized":"2024-03-11T21:04:51.940412+0000","mapping_epoch":251,"log_start":"262'3000","ondisk_log_start":"262'3000","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"280'7923","last_scrub_stamp":"2024-03-11T21:04:47.235330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:46.570492+0000","last_clean_scrub_stamp":"2024-03-11T21:04:47.235330+0000","objects_scrubbed":6106,"log_size":5174,"log_dups_size":0,"ondisk_log_size":5174,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:06:16.894228+0000","scrub_duration":2651,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":379584512,"num_objects":5792,"num_object_clones":0,"num_object_copies":17376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5792,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8174,"num_write_kb":446912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":483,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:49.713691+0000","last_change":"2024-03-11T21:04:30.333409+0000","last_active":"2024-03-11T21:04:49.713691+0000","last_peered":"2024-03-11T21:04:49.713691+0000","last_clean":"2024-03-11T21:04:49.713691+0000","last_became_active":"2024-03-11T21:00:48.466853+0000","last_became_peered":"2024-03-11T21:00:48.466853+0000","last_unstale":"2024-03-11T21:04:49.713691+0000","last_undegraded":"2024-03-11T21:04:49.713691+0000","last_fullsized":"2024-03-11T21:04:49.713691+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:04:30.333285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:52:53.450885+0000","last_clean_scrub_stamp":"2024-03-11T21:04:30.333285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:05:42.292842+0000","scrub_duration":107,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"77'2","reported_seq":423,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:49.726892+0000","last_change":"2024-03-11T21:03:51.299386+0000","last_active":"2024-03-11T21:04:49.726892+0000","last_peered":"2024-03-11T21:04:49.726892+0000","last_clean":"2024-03-11T21:04:49.726892+0000","last_became_active":"2024-03-11T21:00:27.964018+0000","last_became_peered":"2024-03-11T21:00:27.964018+0000","last_unstale":"2024-03-11T21:04:49.726892+0000","last_undegraded":"2024-03-11T21:04:49.726892+0000","last_fullsized":"2024-03-11T21:04:49.726892+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'2","last_scrub_stamp":"2024-03-11T21:03:51.299300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:03:51.299300+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:05:01.172859+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"207'24","reported_seq":566,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:49.652508+0000","last_change":"2024-03-11T21:03:23.941107+0000","last_active":"2024-03-11T21:04:49.652508+0000","last_peered":"2024-03-11T21:04:49.652508+0000","last_clean":"2024-03-11T21:04:49.652508+0000","last_became_active":"2024-03-11T21:00:48.465254+0000","last_became_peered":"2024-03-11T21:00:48.465254+0000","last_unstale":"2024-03-11T21:04:49.652508+0000","last_undegraded":"2024-03-11T21:04:49.652508+0000","last_fullsized":"2024-03-11T21:04:49.652508+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'24","last_scrub_stamp":"2024-03-11T21:03:23.940968+0000","last_deep_scrub":"100'16","last_deep_scrub_stamp":"2024-03-11T20:56:08.036331+0000","last_clean_scrub_stamp":"2024-03-11T21:03:23.940968+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"212'17","reported_seq":595,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:49.724354+0000","last_change":"2024-03-11T21:03:38.259033+0000","last_active":"2024-03-11T21:04:49.724354+0000","last_peered":"2024-03-11T21:04:49.724354+0000","last_clean":"2024-03-11T21:04:49.724354+0000","last_became_active":"2024-03-11T21:00:48.461768+0000","last_became_peered":"2024-03-11T21:00:48.461768+0000","last_unstale":"2024-03-11T21:04:49.724354+0000","last_undegraded":"2024-03-11T21:04:49.724354+0000","last_fullsized":"2024-03-11T21:04:49.724354+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"212'17","last_scrub_stamp":"2024-03-11T21:03:38.258850+0000","last_deep_scrub":"212'17","last_deep_scrub_stamp":"2024-03-11T21:03:38.258850+0000","last_clean_scrub_stamp":"2024-03-11T21:03:38.258850+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:55.059529+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.f","version":"282'8008","reported_seq":8080,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:50.201508+0000","last_change":"2024-03-11T21:02:47.637043+0000","last_active":"2024-03-11T21:04:50.201508+0000","last_peered":"2024-03-11T21:04:50.201508+0000","last_clean":"2024-03-11T21:04:50.201508+0000","last_became_active":"2024-03-11T21:02:47.636634+0000","last_became_peered":"2024-03-11T21:02:47.636634+0000","last_unstale":"2024-03-11T21:04:50.201508+0000","last_undegraded":"2024-03-11T21:04:50.201508+0000","last_fullsized":"2024-03-11T21:04:50.201508+0000","mapping_epoch":251,"log_start":"261'2600","ondisk_log_start":"261'2600","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:02:46.570492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:46.570492+0000","last_clean_scrub_stamp":"2024-03-11T21:02:46.570492+0000","objects_scrubbed":0,"log_size":5408,"log_dups_size":0,"ondisk_log_size":5408,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381288448,"num_objects":5818,"num_object_clones":0,"num_object_copies":17454,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5818,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8008,"num_write_kb":442432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":283,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:49.714040+0000","last_change":"2024-03-11T21:00:24.793264+0000","last_active":"2024-03-11T21:04:49.714040+0000","last_peered":"2024-03-11T21:04:49.714040+0000","last_clean":"2024-03-11T21:04:49.714040+0000","last_became_active":"2024-03-11T20:56:59.795191+0000","last_became_peered":"2024-03-11T20:56:59.795191+0000","last_unstale":"2024-03-11T21:04:49.714040+0000","last_undegraded":"2024-03-11T21:04:49.714040+0000","last_fullsized":"2024-03-11T21:04:49.714040+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:00:24.793093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:57:21.568400+0000","last_clean_scrub_stamp":"2024-03-11T21:00:24.793093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":182,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-03-11T21:04:51.688109+0000","last_change":"2024-03-11T21:03:40.610888+0000","last_active":"2024-03-11T21:04:51.688109+0000","last_peered":"2024-03-11T21:04:51.688109+0000","last_clean":"2024-03-11T21:04:51.688109+0000","last_became_active":"2024-03-11T21:00:48.455733+0000","last_became_peered":"2024-03-11T21:00:48.455733+0000","last_unstale":"2024-03-11T21:04:51.688109+0000","last_undegraded":"2024-03-11T21:04:51.688109+0000","last_fullsized":"2024-03-11T21:04:51.688109+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:40.610810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:20.096874+0000","last_clean_scrub_stamp":"2024-03-11T21:03:40.610810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:59.174036+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"207'28","reported_seq":683,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:49.649184+0000","last_change":"2024-03-11T21:03:15.735024+0000","last_active":"2024-03-11T21:04:49.649184+0000","last_peered":"2024-03-11T21:04:49.649184+0000","last_clean":"2024-03-11T21:04:49.649184+0000","last_became_active":"2024-03-11T20:50:30.094977+0000","last_became_peered":"2024-03-11T20:50:30.094977+0000","last_unstale":"2024-03-11T21:04:49.649184+0000","last_undegraded":"2024-03-11T21:04:49.649184+0000","last_fullsized":"2024-03-11T21:04:49.649184+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'28","last_scrub_stamp":"2024-03-11T21:03:15.734883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:03:15.734883+0000","objects_scrubbed":8,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":182,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-03-11T21:04:51.689669+0000","last_change":"2024-03-11T21:04:40.529581+0000","last_active":"2024-03-11T21:04:51.689669+0000","last_peered":"2024-03-11T21:04:51.689669+0000","last_clean":"2024-03-11T21:04:51.689669+0000","last_became_active":"2024-03-11T21:00:48.457901+0000","last_became_peered":"2024-03-11T21:00:48.457901+0000","last_unstale":"2024-03-11T21:04:51.689669+0000","last_undegraded":"2024-03-11T21:04:51.689669+0000","last_fullsized":"2024-03-11T21:04:51.689669+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:04:40.529460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:52:10.160202+0000","last_clean_scrub_stamp":"2024-03-11T21:04:40.529460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:05:46.208928+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"282'8074","reported_seq":8146,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:50.021047+0000","last_change":"2024-03-11T21:02:47.621307+0000","last_active":"2024-03-11T21:04:50.021047+0000","last_peered":"2024-03-11T21:04:50.021047+0000","last_clean":"2024-03-11T21:04:50.021047+0000","last_became_active":"2024-03-11T21:02:47.619716+0000","last_became_peered":"2024-03-11T21:02:47.619716+0000","last_unstale":"2024-03-11T21:04:50.021047+0000","last_undegraded":"2024-03-11T21:04:50.021047+0000","last_fullsized":"2024-03-11T21:04:50.021047+0000","mapping_epoch":251,"log_start":"266'3400","ondisk_log_start":"266'3400","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:02:46.570492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:46.570492+0000","last_clean_scrub_stamp":"2024-03-11T21:02:46.570492+0000","objects_scrubbed":0,"log_size":4674,"log_dups_size":0,"ondisk_log_size":4674,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":386269184,"num_objects":5894,"num_object_clones":0,"num_object_copies":17682,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5894,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8074,"num_write_kb":446976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":463,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:49.708997+0000","last_change":"2024-03-11T21:03:28.677362+0000","last_active":"2024-03-11T21:04:49.708997+0000","last_peered":"2024-03-11T21:04:49.708997+0000","last_clean":"2024-03-11T21:04:49.708997+0000","last_became_active":"2024-03-11T21:00:27.959408+0000","last_became_peered":"2024-03-11T21:00:27.959408+0000","last_unstale":"2024-03-11T21:04:49.708997+0000","last_undegraded":"2024-03-11T21:04:49.708997+0000","last_fullsized":"2024-03-11T21:04:49.708997+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:28.677276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:03:28.677276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:49.954168+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":487,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:49.650611+0000","last_change":"2024-03-11T21:02:29.807630+0000","last_active":"2024-03-11T21:04:49.650611+0000","last_peered":"2024-03-11T21:04:49.650611+0000","last_clean":"2024-03-11T21:04:49.650611+0000","last_became_active":"2024-03-11T21:00:48.471212+0000","last_became_peered":"2024-03-11T21:00:48.471212+0000","last_unstale":"2024-03-11T21:04:49.650611+0000","last_undegraded":"2024-03-11T21:04:49.650611+0000","last_fullsized":"2024-03-11T21:04:49.650611+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:02:29.807044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:33.823704+0000","last_clean_scrub_stamp":"2024-03-11T21:02:29.807044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"207'37","reported_seq":503,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:49.725790+0000","last_change":"2024-03-11T21:04:49.513889+0000","last_active":"2024-03-11T21:04:49.725790+0000","last_peered":"2024-03-11T21:04:49.725790+0000","last_clean":"2024-03-11T21:04:49.725790+0000","last_became_active":"2024-03-11T21:00:27.960377+0000","last_became_peered":"2024-03-11T21:00:27.960377+0000","last_unstale":"2024-03-11T21:04:49.725790+0000","last_undegraded":"2024-03-11T21:04:49.725790+0000","last_fullsized":"2024-03-11T21:04:49.725790+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'37","last_scrub_stamp":"2024-03-11T21:04:49.513751+0000","last_deep_scrub":"182'27","last_deep_scrub_stamp":"2024-03-11T20:58:52.058851+0000","last_clean_scrub_stamp":"2024-03-11T21:04:49.513751+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:05:58.443345+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":182,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-03-11T21:04:51.687624+0000","last_change":"2024-03-11T21:03:44.649559+0000","last_active":"2024-03-11T21:04:51.687624+0000","last_peered":"2024-03-11T21:04:51.687624+0000","last_clean":"2024-03-11T21:04:51.687624+0000","last_became_active":"2024-03-11T21:00:48.458008+0000","last_became_peered":"2024-03-11T21:00:48.458008+0000","last_unstale":"2024-03-11T21:04:51.687624+0000","last_undegraded":"2024-03-11T21:04:51.687624+0000","last_fullsized":"2024-03-11T21:04:51.687624+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:44.649310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:33.319221+0000","last_clean_scrub_stamp":"2024-03-11T21:03:44.649310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:05:08.200071+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.a","version":"282'7966","reported_seq":8038,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:50.155385+0000","last_change":"2024-03-11T21:02:47.634170+0000","last_active":"2024-03-11T21:04:50.155385+0000","last_peered":"2024-03-11T21:04:50.155385+0000","last_clean":"2024-03-11T21:04:50.155385+0000","last_became_active":"2024-03-11T21:02:47.633164+0000","last_became_peered":"2024-03-11T21:02:47.633164+0000","last_unstale":"2024-03-11T21:04:50.155385+0000","last_undegraded":"2024-03-11T21:04:50.155385+0000","last_fullsized":"2024-03-11T21:04:50.155385+0000","mapping_epoch":251,"log_start":"261'2600","ondisk_log_start":"261'2600","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:02:46.570492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:46.570492+0000","last_clean_scrub_stamp":"2024-03-11T21:02:46.570492+0000","objects_scrubbed":0,"log_size":5366,"log_dups_size":0,"ondisk_log_size":5366,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":379715584,"num_objects":5794,"num_object_clones":0,"num_object_copies":17382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5794,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7966,"num_write_kb":440320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":469,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:49.652404+0000","last_change":"2024-03-11T21:04:42.338906+0000","last_active":"2024-03-11T21:04:49.652404+0000","last_peered":"2024-03-11T21:04:49.652404+0000","last_clean":"2024-03-11T21:04:49.652404+0000","last_became_active":"2024-03-11T20:50:34.321038+0000","last_became_peered":"2024-03-11T20:50:34.321038+0000","last_unstale":"2024-03-11T21:04:49.652404+0000","last_undegraded":"2024-03-11T21:04:49.652404+0000","last_fullsized":"2024-03-11T21:04:49.652404+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:04:42.338712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:32.951104+0000","last_clean_scrub_stamp":"2024-03-11T21:04:42.338712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:05:55.277574+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"207'38","reported_seq":207,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-03-11T21:04:51.689057+0000","last_change":"2024-03-11T21:03:06.361379+0000","last_active":"2024-03-11T21:04:51.689057+0000","last_peered":"2024-03-11T21:04:51.689057+0000","last_clean":"2024-03-11T21:04:51.689057+0000","last_became_active":"2024-03-11T21:00:48.456178+0000","last_became_peered":"2024-03-11T21:00:48.456178+0000","last_unstale":"2024-03-11T21:04:51.689057+0000","last_undegraded":"2024-03-11T21:04:51.689057+0000","last_fullsized":"2024-03-11T21:04:51.689057+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'38","last_scrub_stamp":"2024-03-11T21:03:06.361192+0000","last_deep_scrub":"109'26","last_deep_scrub_stamp":"2024-03-11T20:52:49.216378+0000","last_clean_scrub_stamp":"2024-03-11T21:03:06.361192+0000","objects_scrubbed":8,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":497,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:49.648214+0000","last_change":"2024-03-11T21:04:44.017149+0000","last_active":"2024-03-11T21:04:49.648214+0000","last_peered":"2024-03-11T21:04:49.648214+0000","last_clean":"2024-03-11T21:04:49.648214+0000","last_became_active":"2024-03-11T20:50:28.045625+0000","last_became_peered":"2024-03-11T20:50:28.045625+0000","last_unstale":"2024-03-11T21:04:49.648214+0000","last_undegraded":"2024-03-11T21:04:49.648214+0000","last_fullsized":"2024-03-11T21:04:49.648214+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:04:44.017041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:04:44.017041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:06:01.450955+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":174,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-03-11T21:04:51.687323+0000","last_change":"2024-03-11T21:03:11.331958+0000","last_active":"2024-03-11T21:04:51.687323+0000","last_peered":"2024-03-11T21:04:51.687323+0000","last_clean":"2024-03-11T21:04:51.687323+0000","last_became_active":"2024-03-11T21:00:48.473346+0000","last_became_peered":"2024-03-11T21:00:48.473346+0000","last_unstale":"2024-03-11T21:04:51.687323+0000","last_undegraded":"2024-03-11T21:04:51.687323+0000","last_fullsized":"2024-03-11T21:04:51.687323+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:11.331883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:03:11.331883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"282'7812","reported_seq":7884,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:51.217928+0000","last_change":"2024-03-11T21:02:47.639919+0000","last_active":"2024-03-11T21:04:51.217928+0000","last_peered":"2024-03-11T21:04:51.217928+0000","last_clean":"2024-03-11T21:04:51.217928+0000","last_became_active":"2024-03-11T21:02:47.639328+0000","last_became_peered":"2024-03-11T21:02:47.639328+0000","last_unstale":"2024-03-11T21:04:51.217928+0000","last_undegraded":"2024-03-11T21:04:51.217928+0000","last_fullsized":"2024-03-11T21:04:51.217928+0000","mapping_epoch":251,"log_start":"264'3100","ondisk_log_start":"264'3100","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:02:46.570492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:46.570492+0000","last_clean_scrub_stamp":"2024-03-11T21:02:46.570492+0000","objects_scrubbed":0,"log_size":4712,"log_dups_size":0,"ondisk_log_size":4712,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":373686272,"num_objects":5702,"num_object_clones":0,"num_object_copies":17106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5702,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7812,"num_write_kb":432448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":463,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:49.649146+0000","last_change":"2024-03-11T21:03:11.800852+0000","last_active":"2024-03-11T21:04:49.649146+0000","last_peered":"2024-03-11T21:04:49.649146+0000","last_clean":"2024-03-11T21:04:49.649146+0000","last_became_active":"2024-03-11T21:00:27.962668+0000","last_became_peered":"2024-03-11T21:00:27.962668+0000","last_unstale":"2024-03-11T21:04:49.649146+0000","last_undegraded":"2024-03-11T21:04:49.649146+0000","last_fullsized":"2024-03-11T21:04:49.649146+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:11.800717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:03:11.800717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":83,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"207'20","reported_seq":492,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:49.715514+0000","last_change":"2024-03-11T21:00:14.736355+0000","last_active":"2024-03-11T21:04:49.715514+0000","last_peered":"2024-03-11T21:04:49.715514+0000","last_clean":"2024-03-11T21:04:49.715514+0000","last_became_active":"2024-03-11T20:50:30.101747+0000","last_became_peered":"2024-03-11T20:50:30.101747+0000","last_unstale":"2024-03-11T21:04:49.715514+0000","last_undegraded":"2024-03-11T21:04:49.715514+0000","last_fullsized":"2024-03-11T21:04:49.715514+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'10","last_scrub_stamp":"2024-03-11T21:00:14.736261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:00:14.736261+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":483,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:49.649097+0000","last_change":"2024-03-11T21:03:47.212843+0000","last_active":"2024-03-11T21:04:49.649097+0000","last_peered":"2024-03-11T21:04:49.649097+0000","last_clean":"2024-03-11T21:04:49.649097+0000","last_became_active":"2024-03-11T20:50:28.085503+0000","last_became_peered":"2024-03-11T20:50:28.085503+0000","last_unstale":"2024-03-11T21:04:49.649097+0000","last_undegraded":"2024-03-11T21:04:49.649097+0000","last_fullsized":"2024-03-11T21:04:49.649097+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:47.212723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:56:03.479180+0000","last_clean_scrub_stamp":"2024-03-11T21:03:47.212723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:55.569265+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"184'130","reported_seq":750,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:49.651803+0000","last_change":"2024-03-11T21:04:39.297532+0000","last_active":"2024-03-11T21:04:49.651803+0000","last_peered":"2024-03-11T21:04:49.651803+0000","last_clean":"2024-03-11T21:04:49.651803+0000","last_became_active":"2024-03-11T20:49:52.584543+0000","last_became_peered":"2024-03-11T20:49:52.584543+0000","last_unstale":"2024-03-11T21:04:49.651803+0000","last_undegraded":"2024-03-11T21:04:49.651803+0000","last_fullsized":"2024-03-11T21:04:49.651803+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"184'130","last_scrub_stamp":"2024-03-11T21:04:39.297394+0000","last_deep_scrub":"184'130","last_deep_scrub_stamp":"2024-03-11T20:58:56.415267+0000","last_clean_scrub_stamp":"2024-03-11T21:04:39.297394+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:05:50.270423+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":489,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:49.711244+0000","last_change":"2024-03-11T21:04:37.625846+0000","last_active":"2024-03-11T21:04:49.711244+0000","last_peered":"2024-03-11T21:04:49.711244+0000","last_clean":"2024-03-11T21:04:49.711244+0000","last_became_active":"2024-03-11T21:00:27.959467+0000","last_became_peered":"2024-03-11T21:00:27.959467+0000","last_unstale":"2024-03-11T21:04:49.711244+0000","last_undegraded":"2024-03-11T21:04:49.711244+0000","last_fullsized":"2024-03-11T21:04:49.711244+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:04:37.625736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:54:22.448812+0000","last_clean_scrub_stamp":"2024-03-11T21:04:37.625736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:05:49.674155+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.b","version":"282'8039","reported_seq":8188,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:50.042562+0000","last_change":"2024-03-11T21:04:31.403773+0000","last_active":"2024-03-11T21:04:50.042562+0000","last_peered":"2024-03-11T21:04:50.042562+0000","last_clean":"2024-03-11T21:04:50.042562+0000","last_became_active":"2024-03-11T21:02:47.621751+0000","last_became_peered":"2024-03-11T21:02:47.621751+0000","last_unstale":"2024-03-11T21:04:50.042562+0000","last_undegraded":"2024-03-11T21:04:50.042562+0000","last_fullsized":"2024-03-11T21:04:50.042562+0000","mapping_epoch":251,"log_start":"260'1900","ondisk_log_start":"260'1900","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'6874","last_scrub_stamp":"2024-03-11T21:04:31.403288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:46.570492+0000","last_clean_scrub_stamp":"2024-03-11T21:04:31.403288+0000","objects_scrubbed":6818,"log_size":6139,"log_dups_size":0,"ondisk_log_size":6139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:05:56.294802+0000","scrub_duration":1775,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":384106496,"num_objects":5861,"num_object_clones":0,"num_object_copies":17583,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5861,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8039,"num_write_kb":444800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":182,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-03-11T21:04:51.687840+0000","last_change":"2024-03-11T21:04:35.604251+0000","last_active":"2024-03-11T21:04:51.687840+0000","last_peered":"2024-03-11T21:04:51.687840+0000","last_clean":"2024-03-11T21:04:51.687840+0000","last_became_active":"2024-03-11T21:00:48.464596+0000","last_became_peered":"2024-03-11T21:00:48.464596+0000","last_unstale":"2024-03-11T21:04:51.687840+0000","last_undegraded":"2024-03-11T21:04:51.687840+0000","last_fullsized":"2024-03-11T21:04:51.687840+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:04:35.604158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:04:35.604158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:06:01.815373+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"207'35","reported_seq":352,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:49.724778+0000","last_change":"2024-03-11T21:04:46.779974+0000","last_active":"2024-03-11T21:04:49.724778+0000","last_peered":"2024-03-11T21:04:49.724778+0000","last_clean":"2024-03-11T21:04:49.724778+0000","last_became_active":"2024-03-11T21:00:27.959172+0000","last_became_peered":"2024-03-11T21:00:27.959172+0000","last_unstale":"2024-03-11T21:04:49.724778+0000","last_undegraded":"2024-03-11T21:04:49.724778+0000","last_fullsized":"2024-03-11T21:04:49.724778+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'35","last_scrub_stamp":"2024-03-11T21:04:46.779822+0000","last_deep_scrub":"100'21","last_deep_scrub_stamp":"2024-03-11T20:51:31.835302+0000","last_clean_scrub_stamp":"2024-03-11T21:04:46.779822+0000","objects_scrubbed":7,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:05:54.186220+0000","scrub_duration":263,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":460,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:49.648324+0000","last_change":"2024-03-11T21:03:48.784865+0000","last_active":"2024-03-11T21:04:49.648324+0000","last_peered":"2024-03-11T21:04:49.648324+0000","last_clean":"2024-03-11T21:04:49.648324+0000","last_became_active":"2024-03-11T21:00:48.458579+0000","last_became_peered":"2024-03-11T21:00:48.458579+0000","last_unstale":"2024-03-11T21:04:49.648324+0000","last_undegraded":"2024-03-11T21:04:49.648324+0000","last_fullsized":"2024-03-11T21:04:49.648324+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:48.784648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:03:48.784648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:05:02.043800+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":484,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:49.716927+0000","last_change":"2024-03-11T21:03:45.421629+0000","last_active":"2024-03-11T21:04:49.716927+0000","last_peered":"2024-03-11T21:04:49.716927+0000","last_clean":"2024-03-11T21:04:49.716927+0000","last_became_active":"2024-03-11T21:00:48.463247+0000","last_became_peered":"2024-03-11T21:00:48.463247+0000","last_unstale":"2024-03-11T21:04:49.716927+0000","last_undegraded":"2024-03-11T21:04:49.716927+0000","last_fullsized":"2024-03-11T21:04:49.716927+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:45.421446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:26.839782+0000","last_clean_scrub_stamp":"2024-03-11T21:03:45.421446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:52.927501+0000","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.e","version":"282'8072","reported_seq":8144,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:50.073351+0000","last_change":"2024-03-11T21:02:47.617410+0000","last_active":"2024-03-11T21:04:50.073351+0000","last_peered":"2024-03-11T21:04:50.073351+0000","last_clean":"2024-03-11T21:04:50.073351+0000","last_became_active":"2024-03-11T21:02:47.616727+0000","last_became_peered":"2024-03-11T21:02:47.616727+0000","last_unstale":"2024-03-11T21:04:50.073351+0000","last_undegraded":"2024-03-11T21:04:50.073351+0000","last_fullsized":"2024-03-11T21:04:50.073351+0000","mapping_epoch":251,"log_start":"267'3400","ondisk_log_start":"267'3400","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:02:46.570492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:46.570492+0000","last_clean_scrub_stamp":"2024-03-11T21:02:46.570492+0000","objects_scrubbed":0,"log_size":4672,"log_dups_size":0,"ondisk_log_size":4672,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":389283840,"num_objects":5940,"num_object_clones":0,"num_object_copies":17820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5940,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8072,"num_write_kb":448384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":182,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-03-11T21:04:51.686714+0000","last_change":"2024-03-11T21:03:28.495517+0000","last_active":"2024-03-11T21:04:51.686714+0000","last_peered":"2024-03-11T21:04:51.686714+0000","last_clean":"2024-03-11T21:04:51.686714+0000","last_became_active":"2024-03-11T21:00:48.469688+0000","last_became_peered":"2024-03-11T21:00:48.469688+0000","last_unstale":"2024-03-11T21:04:51.686714+0000","last_undegraded":"2024-03-11T21:04:51.686714+0000","last_fullsized":"2024-03-11T21:04:51.686714+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:28.495308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:03:28.495308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"145'3","reported_seq":482,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:49.711239+0000","last_change":"2024-03-11T21:03:44.305877+0000","last_active":"2024-03-11T21:04:49.711239+0000","last_peered":"2024-03-11T21:04:49.711239+0000","last_clean":"2024-03-11T21:04:49.711239+0000","last_became_active":"2024-03-11T20:50:32.195452+0000","last_became_peered":"2024-03-11T20:50:32.195452+0000","last_unstale":"2024-03-11T21:04:49.711239+0000","last_undegraded":"2024-03-11T21:04:49.711239+0000","last_fullsized":"2024-03-11T21:04:49.711239+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'3","last_scrub_stamp":"2024-03-11T21:03:44.305716+0000","last_deep_scrub":"145'3","last_deep_scrub_stamp":"2024-03-11T20:55:52.848043+0000","last_clean_scrub_stamp":"2024-03-11T21:03:44.305716+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0.0024049620000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":482,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:49.652586+0000","last_change":"2024-03-11T21:04:36.201956+0000","last_active":"2024-03-11T21:04:49.652586+0000","last_peered":"2024-03-11T21:04:49.652586+0000","last_clean":"2024-03-11T21:04:49.652586+0000","last_became_active":"2024-03-11T21:00:27.966749+0000","last_became_peered":"2024-03-11T21:00:27.966749+0000","last_unstale":"2024-03-11T21:04:49.652586+0000","last_undegraded":"2024-03-11T21:04:49.652586+0000","last_fullsized":"2024-03-11T21:04:49.652586+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:04:36.201831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:04:36.201831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:05:54.420434+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"212'46","reported_seq":260,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-03-11T21:04:51.687999+0000","last_change":"2024-03-11T21:03:12.324530+0000","last_active":"2024-03-11T21:04:51.687999+0000","last_peered":"2024-03-11T21:04:51.687999+0000","last_clean":"2024-03-11T21:04:51.687999+0000","last_became_active":"2024-03-11T21:00:48.473227+0000","last_became_peered":"2024-03-11T21:00:48.473227+0000","last_unstale":"2024-03-11T21:04:51.687999+0000","last_undegraded":"2024-03-11T21:04:51.687999+0000","last_fullsized":"2024-03-11T21:04:51.687999+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"212'46","last_scrub_stamp":"2024-03-11T21:03:12.324431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:03:12.324431+0000","objects_scrubbed":10,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":69,"num_read_kb":54,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"160'12","reported_seq":403,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:49.649000+0000","last_change":"2024-03-11T21:03:40.870833+0000","last_active":"2024-03-11T21:04:49.649000+0000","last_peered":"2024-03-11T21:04:49.649000+0000","last_clean":"2024-03-11T21:04:49.649000+0000","last_became_active":"2024-03-11T20:56:59.797805+0000","last_became_peered":"2024-03-11T20:56:59.797805+0000","last_unstale":"2024-03-11T21:04:49.649000+0000","last_undegraded":"2024-03-11T21:04:49.649000+0000","last_fullsized":"2024-03-11T21:04:49.649000+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"160'12","last_scrub_stamp":"2024-03-11T21:03:40.870714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:03:40.870714+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:53.829709+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"282'7948","reported_seq":8088,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:50.090578+0000","last_change":"2024-03-11T21:04:47.823495+0000","last_active":"2024-03-11T21:04:50.090578+0000","last_peered":"2024-03-11T21:04:50.090578+0000","last_clean":"2024-03-11T21:04:50.090578+0000","last_became_active":"2024-03-11T21:02:47.623292+0000","last_became_peered":"2024-03-11T21:02:47.623292+0000","last_unstale":"2024-03-11T21:04:50.090578+0000","last_undegraded":"2024-03-11T21:04:50.090578+0000","last_fullsized":"2024-03-11T21:04:50.090578+0000","mapping_epoch":251,"log_start":"266'3300","ondisk_log_start":"266'3300","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"280'7808","last_scrub_stamp":"2024-03-11T21:04:47.823119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:46.570492+0000","last_clean_scrub_stamp":"2024-03-11T21:04:47.823119+0000","objects_scrubbed":6048,"log_size":4648,"log_dups_size":0,"ondisk_log_size":4648,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:06:03.048471+0000","scrub_duration":1495,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":383320088,"num_objects":5850,"num_object_clones":0,"num_object_copies":17550,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5850,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7948,"num_write_kb":441473,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":386,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:49.725078+0000","last_change":"2024-03-11T21:03:43.345403+0000","last_active":"2024-03-11T21:04:49.725078+0000","last_peered":"2024-03-11T21:04:49.725078+0000","last_clean":"2024-03-11T21:04:49.725078+0000","last_became_active":"2024-03-11T21:00:27.961447+0000","last_became_peered":"2024-03-11T21:00:27.961447+0000","last_unstale":"2024-03-11T21:04:49.725078+0000","last_undegraded":"2024-03-11T21:04:49.725078+0000","last_fullsized":"2024-03-11T21:04:49.725078+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:43.345257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:03:43.345257+0000","last_clean_scrub_stamp":"2024-03-11T21:03:43.345257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":407,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:49.724566+0000","last_change":"2024-03-11T21:04:33.475294+0000","last_active":"2024-03-11T21:04:49.724566+0000","last_peered":"2024-03-11T21:04:49.724566+0000","last_clean":"2024-03-11T21:04:49.724566+0000","last_became_active":"2024-03-11T21:00:27.967546+0000","last_became_peered":"2024-03-11T21:00:27.967546+0000","last_unstale":"2024-03-11T21:04:49.724566+0000","last_undegraded":"2024-03-11T21:04:49.724566+0000","last_fullsized":"2024-03-11T21:04:49.724566+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:04:33.475057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:22.617999+0000","last_clean_scrub_stamp":"2024-03-11T21:04:33.475057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:06:01.886504+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"77'1","reported_seq":178,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-03-11T21:04:51.687097+0000","last_change":"2024-03-11T21:03:07.312601+0000","last_active":"2024-03-11T21:04:51.687097+0000","last_peered":"2024-03-11T21:04:51.687097+0000","last_clean":"2024-03-11T21:04:51.687097+0000","last_became_active":"2024-03-11T21:00:48.468682+0000","last_became_peered":"2024-03-11T21:00:48.468682+0000","last_unstale":"2024-03-11T21:04:51.687097+0000","last_undegraded":"2024-03-11T21:04:51.687097+0000","last_fullsized":"2024-03-11T21:04:51.687097+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-03-11T21:03:07.312455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:03:07.312455+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"207'17","reported_seq":495,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:49.724891+0000","last_change":"2024-03-11T21:03:42.690547+0000","last_active":"2024-03-11T21:04:49.724891+0000","last_peered":"2024-03-11T21:04:49.724891+0000","last_clean":"2024-03-11T21:04:49.724891+0000","last_became_active":"2024-03-11T20:50:30.097669+0000","last_became_peered":"2024-03-11T20:50:30.097669+0000","last_unstale":"2024-03-11T21:04:49.724891+0000","last_undegraded":"2024-03-11T21:04:49.724891+0000","last_fullsized":"2024-03-11T21:04:49.724891+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'17","last_scrub_stamp":"2024-03-11T21:03:42.690411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:03:42.690411+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:05:11.885193+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"84'11","reported_seq":650,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:49.714509+0000","last_change":"2024-03-11T21:00:20.771711+0000","last_active":"2024-03-11T21:04:49.714509+0000","last_peered":"2024-03-11T21:04:49.714509+0000","last_clean":"2024-03-11T21:04:49.714509+0000","last_became_active":"2024-03-11T20:50:32.647041+0000","last_became_peered":"2024-03-11T20:50:32.647041+0000","last_unstale":"2024-03-11T21:04:49.714509+0000","last_undegraded":"2024-03-11T21:04:49.714509+0000","last_fullsized":"2024-03-11T21:04:49.714509+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:00:20.771539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:00:20.771539+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.c","version":"282'8131","reported_seq":8203,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:51.247115+0000","last_change":"2024-03-11T21:02:47.630285+0000","last_active":"2024-03-11T21:04:51.247115+0000","last_peered":"2024-03-11T21:04:51.247115+0000","last_clean":"2024-03-11T21:04:51.247115+0000","last_became_active":"2024-03-11T21:02:47.629442+0000","last_became_peered":"2024-03-11T21:02:47.629442+0000","last_unstale":"2024-03-11T21:04:51.247115+0000","last_undegraded":"2024-03-11T21:04:51.247115+0000","last_fullsized":"2024-03-11T21:04:51.247115+0000","mapping_epoch":251,"log_start":"267'3500","ondisk_log_start":"267'3500","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:02:46.570492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:46.570492+0000","last_clean_scrub_stamp":"2024-03-11T21:02:46.570492+0000","objects_scrubbed":0,"log_size":4631,"log_dups_size":0,"ondisk_log_size":4631,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":382009344,"num_objects":5829,"num_object_clones":0,"num_object_copies":17487,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8131,"num_write_kb":446720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":469,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:49.652132+0000","last_change":"2024-03-11T21:03:09.148007+0000","last_active":"2024-03-11T21:04:49.652132+0000","last_peered":"2024-03-11T21:04:49.652132+0000","last_clean":"2024-03-11T21:04:49.652132+0000","last_became_active":"2024-03-11T20:50:34.326947+0000","last_became_peered":"2024-03-11T20:50:34.326947+0000","last_unstale":"2024-03-11T21:04:49.652132+0000","last_undegraded":"2024-03-11T21:04:49.652132+0000","last_fullsized":"2024-03-11T21:04:49.652132+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:09.147923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:03:09.147923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"120'6","reported_seq":1235,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:50.209509+0000","last_change":"2024-03-11T21:02:19.475999+0000","last_active":"2024-03-11T21:04:50.209509+0000","last_peered":"2024-03-11T21:04:50.209509+0000","last_clean":"2024-03-11T21:04:50.209509+0000","last_became_active":"2024-03-11T21:00:48.468997+0000","last_became_peered":"2024-03-11T21:00:48.468997+0000","last_unstale":"2024-03-11T21:04:50.209509+0000","last_undegraded":"2024-03-11T21:04:50.209509+0000","last_fullsized":"2024-03-11T21:04:50.209509+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"120'6","last_scrub_stamp":"2024-03-11T21:02:19.475892+0000","last_deep_scrub":"120'6","last_deep_scrub_stamp":"2024-03-11T20:54:26.468057+0000","last_clean_scrub_stamp":"2024-03-11T21:02:19.475892+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0035533370000000002,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":727,"num_read_kb":722,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":488,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:49.711788+0000","last_change":"2024-03-11T21:03:26.508296+0000","last_active":"2024-03-11T21:04:49.711788+0000","last_peered":"2024-03-11T21:04:49.711788+0000","last_clean":"2024-03-11T21:04:49.711788+0000","last_became_active":"2024-03-11T20:50:28.071707+0000","last_became_peered":"2024-03-11T20:50:28.071707+0000","last_unstale":"2024-03-11T21:04:49.711788+0000","last_undegraded":"2024-03-11T21:04:49.711788+0000","last_fullsized":"2024-03-11T21:04:49.711788+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:26.508215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:23.417283+0000","last_clean_scrub_stamp":"2024-03-11T21:03:26.508215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"207'24","reported_seq":572,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:49.716582+0000","last_change":"2024-03-11T21:03:47.381203+0000","last_active":"2024-03-11T21:04:49.716582+0000","last_peered":"2024-03-11T21:04:49.716582+0000","last_clean":"2024-03-11T21:04:49.716582+0000","last_became_active":"2024-03-11T21:00:48.461972+0000","last_became_peered":"2024-03-11T21:00:48.461972+0000","last_unstale":"2024-03-11T21:04:49.716582+0000","last_undegraded":"2024-03-11T21:04:49.716582+0000","last_fullsized":"2024-03-11T21:04:49.716582+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'24","last_scrub_stamp":"2024-03-11T21:03:47.381061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:03:47.381061+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:05:05.348872+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":428,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:49.724701+0000","last_change":"2024-03-11T21:04:47.495166+0000","last_active":"2024-03-11T21:04:49.724701+0000","last_peered":"2024-03-11T21:04:49.724701+0000","last_clean":"2024-03-11T21:04:49.724701+0000","last_became_active":"2024-03-11T21:00:27.953537+0000","last_became_peered":"2024-03-11T21:00:27.953537+0000","last_unstale":"2024-03-11T21:04:49.724701+0000","last_undegraded":"2024-03-11T21:04:49.724701+0000","last_fullsized":"2024-03-11T21:04:49.724701+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:04:47.495043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:04:47.495043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:05:59.283433+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.3","version":"282'8080","reported_seq":8152,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:50.055633+0000","last_change":"2024-03-11T21:02:47.628471+0000","last_active":"2024-03-11T21:04:50.055633+0000","last_peered":"2024-03-11T21:04:50.055633+0000","last_clean":"2024-03-11T21:04:50.055633+0000","last_became_active":"2024-03-11T21:02:47.627896+0000","last_became_peered":"2024-03-11T21:02:47.627896+0000","last_unstale":"2024-03-11T21:04:50.055633+0000","last_undegraded":"2024-03-11T21:04:50.055633+0000","last_fullsized":"2024-03-11T21:04:50.055633+0000","mapping_epoch":251,"log_start":"266'3400","ondisk_log_start":"266'3400","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:02:46.570492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:46.570492+0000","last_clean_scrub_stamp":"2024-03-11T21:02:46.570492+0000","objects_scrubbed":0,"log_size":4680,"log_dups_size":0,"ondisk_log_size":4680,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":389414912,"num_objects":5942,"num_object_clones":0,"num_object_copies":17826,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5942,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8080,"num_write_kb":448704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":454,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:49.708682+0000","last_change":"2024-03-11T21:03:43.723734+0000","last_active":"2024-03-11T21:04:49.708682+0000","last_peered":"2024-03-11T21:04:49.708682+0000","last_clean":"2024-03-11T21:04:49.708682+0000","last_became_active":"2024-03-11T20:50:34.335370+0000","last_became_peered":"2024-03-11T20:50:34.335370+0000","last_unstale":"2024-03-11T21:04:49.708682+0000","last_undegraded":"2024-03-11T21:04:49.708682+0000","last_fullsized":"2024-03-11T21:04:49.708682+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:43.715417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:03:43.715417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:52.411766+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"249'17","reported_seq":603,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:49.726693+0000","last_change":"2024-03-11T21:03:24.365609+0000","last_active":"2024-03-11T21:04:49.726693+0000","last_peered":"2024-03-11T21:04:49.726693+0000","last_clean":"2024-03-11T21:04:49.726693+0000","last_became_active":"2024-03-11T21:02:37.524445+0000","last_became_peered":"2024-03-11T21:02:37.524445+0000","last_unstale":"2024-03-11T21:04:49.726693+0000","last_undegraded":"2024-03-11T21:04:49.726693+0000","last_fullsized":"2024-03-11T21:04:49.726693+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"249'17","last_scrub_stamp":"2024-03-11T21:03:24.365350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:03:24.365350+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":486,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:49.651813+0000","last_change":"2024-03-11T21:03:20.066682+0000","last_active":"2024-03-11T21:04:49.651813+0000","last_peered":"2024-03-11T21:04:49.651813+0000","last_clean":"2024-03-11T21:04:49.651813+0000","last_became_active":"2024-03-11T21:00:48.466102+0000","last_became_peered":"2024-03-11T21:00:48.466102+0000","last_unstale":"2024-03-11T21:04:49.651813+0000","last_undegraded":"2024-03-11T21:04:49.651813+0000","last_fullsized":"2024-03-11T21:04:49.651813+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:20.066579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:52:09.043655+0000","last_clean_scrub_stamp":"2024-03-11T21:03:20.066579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":147,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"207'18","reported_seq":428,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:49.724929+0000","last_change":"2024-03-11T21:04:35.636942+0000","last_active":"2024-03-11T21:04:49.724929+0000","last_peered":"2024-03-11T21:04:49.724929+0000","last_clean":"2024-03-11T21:04:49.724929+0000","last_became_active":"2024-03-11T21:00:27.967624+0000","last_became_peered":"2024-03-11T21:00:27.967624+0000","last_unstale":"2024-03-11T21:04:49.724929+0000","last_undegraded":"2024-03-11T21:04:49.724929+0000","last_fullsized":"2024-03-11T21:04:49.724929+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'18","last_scrub_stamp":"2024-03-11T21:04:35.636784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:04:35.636784+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:05:47.818988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"282'8053","reported_seq":8125,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:51.223563+0000","last_change":"2024-03-11T21:02:47.641116+0000","last_active":"2024-03-11T21:04:51.223563+0000","last_peered":"2024-03-11T21:04:51.223563+0000","last_clean":"2024-03-11T21:04:51.223563+0000","last_became_active":"2024-03-11T21:02:47.640263+0000","last_became_peered":"2024-03-11T21:02:47.640263+0000","last_unstale":"2024-03-11T21:04:51.223563+0000","last_undegraded":"2024-03-11T21:04:51.223563+0000","last_fullsized":"2024-03-11T21:04:51.223563+0000","mapping_epoch":251,"log_start":"266'3400","ondisk_log_start":"266'3400","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:02:46.570492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:46.570492+0000","last_clean_scrub_stamp":"2024-03-11T21:02:46.570492+0000","objects_scrubbed":0,"log_size":4653,"log_dups_size":0,"ondisk_log_size":4653,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381747200,"num_objects":5825,"num_object_clones":0,"num_object_copies":17475,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5825,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8053,"num_write_kb":444096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":483,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:49.649582+0000","last_change":"2024-03-11T21:04:39.531066+0000","last_active":"2024-03-11T21:04:49.649582+0000","last_peered":"2024-03-11T21:04:49.649582+0000","last_clean":"2024-03-11T21:04:49.649582+0000","last_became_active":"2024-03-11T21:00:48.460531+0000","last_became_peered":"2024-03-11T21:00:48.460531+0000","last_unstale":"2024-03-11T21:04:49.649582+0000","last_undegraded":"2024-03-11T21:04:49.649582+0000","last_fullsized":"2024-03-11T21:04:49.649582+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:04:39.530832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:27.775534+0000","last_clean_scrub_stamp":"2024-03-11T21:04:39.530832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:06:04.624420+0000","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":182,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-03-11T21:04:51.687217+0000","last_change":"2024-03-11T21:03:29.454135+0000","last_active":"2024-03-11T21:04:51.687217+0000","last_peered":"2024-03-11T21:04:51.687217+0000","last_clean":"2024-03-11T21:04:51.687217+0000","last_became_active":"2024-03-11T21:00:48.469596+0000","last_became_peered":"2024-03-11T21:00:48.469596+0000","last_unstale":"2024-03-11T21:04:51.687217+0000","last_undegraded":"2024-03-11T21:04:51.687217+0000","last_fullsized":"2024-03-11T21:04:51.687217+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:29.453952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:03:29.453952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":174,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-03-11T21:04:51.686817+0000","last_change":"2024-03-11T21:03:26.422762+0000","last_active":"2024-03-11T21:04:51.686817+0000","last_peered":"2024-03-11T21:04:51.686817+0000","last_clean":"2024-03-11T21:04:51.686817+0000","last_became_active":"2024-03-11T21:00:48.466084+0000","last_became_peered":"2024-03-11T21:00:48.466084+0000","last_unstale":"2024-03-11T21:04:51.686817+0000","last_undegraded":"2024-03-11T21:04:51.686817+0000","last_fullsized":"2024-03-11T21:04:51.686817+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:26.422645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:40.974147+0000","last_clean_scrub_stamp":"2024-03-11T21:03:26.422645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"207'34","reported_seq":572,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:49.714769+0000","last_change":"2024-03-11T21:03:13.518790+0000","last_active":"2024-03-11T21:04:49.714769+0000","last_peered":"2024-03-11T21:04:49.714769+0000","last_clean":"2024-03-11T21:04:49.714769+0000","last_became_active":"2024-03-11T21:00:48.472191+0000","last_became_peered":"2024-03-11T21:00:48.472191+0000","last_unstale":"2024-03-11T21:04:49.714769+0000","last_undegraded":"2024-03-11T21:04:49.714769+0000","last_fullsized":"2024-03-11T21:04:49.714769+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'34","last_scrub_stamp":"2024-03-11T21:03:13.518551+0000","last_deep_scrub":"116'22","last_deep_scrub_stamp":"2024-03-11T20:53:47.364709+0000","last_clean_scrub_stamp":"2024-03-11T21:03:13.518551+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1","version":"282'8074","reported_seq":8146,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:51.228918+0000","last_change":"2024-03-11T21:02:47.642494+0000","last_active":"2024-03-11T21:04:51.228918+0000","last_peered":"2024-03-11T21:04:51.228918+0000","last_clean":"2024-03-11T21:04:51.228918+0000","last_became_active":"2024-03-11T21:02:47.641594+0000","last_became_peered":"2024-03-11T21:02:47.641594+0000","last_unstale":"2024-03-11T21:04:51.228918+0000","last_undegraded":"2024-03-11T21:04:51.228918+0000","last_fullsized":"2024-03-11T21:04:51.228918+0000","mapping_epoch":251,"log_start":"266'3400","ondisk_log_start":"266'3400","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:02:46.570492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:46.570492+0000","last_clean_scrub_stamp":"2024-03-11T21:02:46.570492+0000","objects_scrubbed":0,"log_size":4674,"log_dups_size":0,"ondisk_log_size":4674,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":382599168,"num_objects":5838,"num_object_clones":0,"num_object_copies":17514,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5838,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8074,"num_write_kb":445184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":422,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:49.726441+0000","last_change":"2024-03-11T21:03:33.263561+0000","last_active":"2024-03-11T21:04:49.726441+0000","last_peered":"2024-03-11T21:04:49.726441+0000","last_clean":"2024-03-11T21:04:49.726441+0000","last_became_active":"2024-03-11T21:00:48.464070+0000","last_became_peered":"2024-03-11T21:00:48.464070+0000","last_unstale":"2024-03-11T21:04:49.726441+0000","last_undegraded":"2024-03-11T21:04:49.726441+0000","last_fullsized":"2024-03-11T21:04:49.726441+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:33.263478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:03:33.263478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:52.150843+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":182,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-03-11T21:04:51.690765+0000","last_change":"2024-03-11T21:03:37.601354+0000","last_active":"2024-03-11T21:04:51.690765+0000","last_peered":"2024-03-11T21:04:51.690765+0000","last_clean":"2024-03-11T21:04:51.690765+0000","last_became_active":"2024-03-11T21:00:48.461367+0000","last_became_peered":"2024-03-11T21:00:48.461367+0000","last_unstale":"2024-03-11T21:04:51.690765+0000","last_undegraded":"2024-03-11T21:04:51.690765+0000","last_fullsized":"2024-03-11T21:04:51.690765+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:37.601229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:03:37.601229+0000","last_clean_scrub_stamp":"2024-03-11T21:03:37.601229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"267'19","reported_seq":500,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:49.713531+0000","last_change":"2024-03-11T21:03:39.673560+0000","last_active":"2024-03-11T21:04:49.713531+0000","last_peered":"2024-03-11T21:04:49.713531+0000","last_clean":"2024-03-11T21:04:49.713531+0000","last_became_active":"2024-03-11T21:00:27.967572+0000","last_became_peered":"2024-03-11T21:00:27.967572+0000","last_unstale":"2024-03-11T21:04:49.713531+0000","last_undegraded":"2024-03-11T21:04:49.713531+0000","last_fullsized":"2024-03-11T21:04:49.713531+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"267'19","last_scrub_stamp":"2024-03-11T21:03:39.673317+0000","last_deep_scrub":"267'19","last_deep_scrub_stamp":"2024-03-11T21:03:39.673317+0000","last_clean_scrub_stamp":"2024-03-11T21:03:39.673317+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":437,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:49.728380+0000","last_change":"2024-03-11T21:04:43.625382+0000","last_active":"2024-03-11T21:04:49.728380+0000","last_peered":"2024-03-11T21:04:49.728380+0000","last_clean":"2024-03-11T21:04:49.728380+0000","last_became_active":"2024-03-11T21:00:27.958537+0000","last_became_peered":"2024-03-11T21:00:27.958537+0000","last_unstale":"2024-03-11T21:04:49.728380+0000","last_undegraded":"2024-03-11T21:04:49.728380+0000","last_fullsized":"2024-03-11T21:04:49.728380+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:04:43.625264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:35.670149+0000","last_clean_scrub_stamp":"2024-03-11T21:04:43.625264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:05:57.169694+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"283'8157","reported_seq":8304,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-03-11T21:04:51.967070+0000","last_change":"2024-03-11T21:04:45.789459+0000","last_active":"2024-03-11T21:04:51.967070+0000","last_peered":"2024-03-11T21:04:51.967070+0000","last_clean":"2024-03-11T21:04:51.967070+0000","last_became_active":"2024-03-11T21:02:47.628916+0000","last_became_peered":"2024-03-11T21:02:47.628916+0000","last_unstale":"2024-03-11T21:04:51.967070+0000","last_undegraded":"2024-03-11T21:04:51.967070+0000","last_fullsized":"2024-03-11T21:04:51.967070+0000","mapping_epoch":251,"log_start":"262'3000","ondisk_log_start":"262'3000","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"279'7813","last_scrub_stamp":"2024-03-11T21:04:45.789212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:46.570492+0000","last_clean_scrub_stamp":"2024-03-11T21:04:45.789212+0000","objects_scrubbed":6235,"log_size":5157,"log_dups_size":0,"ondisk_log_size":5157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:05:51.377701+0000","scrub_duration":2231,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":382271488,"num_objects":5833,"num_object_clones":0,"num_object_copies":17499,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5833,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8157,"num_write_kb":447680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":486,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:49.649173+0000","last_change":"2024-03-11T21:04:34.116076+0000","last_active":"2024-03-11T21:04:49.649173+0000","last_peered":"2024-03-11T21:04:49.649173+0000","last_clean":"2024-03-11T21:04:49.649173+0000","last_became_active":"2024-03-11T21:00:27.959830+0000","last_became_peered":"2024-03-11T21:00:27.959830+0000","last_unstale":"2024-03-11T21:04:49.649173+0000","last_undegraded":"2024-03-11T21:04:49.649173+0000","last_fullsized":"2024-03-11T21:04:49.649173+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:04:34.115948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:04:34.115948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:05:42.755318+0000","scrub_duration":111,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"249'19","reported_seq":614,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:49.726359+0000","last_change":"2024-03-11T21:03:23.347322+0000","last_active":"2024-03-11T21:04:49.726359+0000","last_peered":"2024-03-11T21:04:49.726359+0000","last_clean":"2024-03-11T21:04:49.726359+0000","last_became_active":"2024-03-11T21:02:37.805125+0000","last_became_peered":"2024-03-11T21:02:37.805125+0000","last_unstale":"2024-03-11T21:04:49.726359+0000","last_undegraded":"2024-03-11T21:04:49.726359+0000","last_fullsized":"2024-03-11T21:04:49.726359+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"249'19","last_scrub_stamp":"2024-03-11T21:03:23.347230+0000","last_deep_scrub":"146'14","last_deep_scrub_stamp":"2024-03-11T20:56:00.242211+0000","last_clean_scrub_stamp":"2024-03-11T21:03:23.347230+0000","objects_scrubbed":1,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":482,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:49.652140+0000","last_change":"2024-03-11T21:03:50.901480+0000","last_active":"2024-03-11T21:04:49.652140+0000","last_peered":"2024-03-11T21:04:49.652140+0000","last_clean":"2024-03-11T21:04:49.652140+0000","last_became_active":"2024-03-11T21:00:27.970573+0000","last_became_peered":"2024-03-11T21:00:27.970573+0000","last_unstale":"2024-03-11T21:04:49.652140+0000","last_undegraded":"2024-03-11T21:04:49.652140+0000","last_fullsized":"2024-03-11T21:04:49.652140+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:50.901398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:52:51.161281+0000","last_clean_scrub_stamp":"2024-03-11T21:03:50.901398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:05:13.889962+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"207'44","reported_seq":625,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:49.710927+0000","last_change":"2024-03-11T21:04:35.674612+0000","last_active":"2024-03-11T21:04:49.710927+0000","last_peered":"2024-03-11T21:04:49.710927+0000","last_clean":"2024-03-11T21:04:49.710927+0000","last_became_active":"2024-03-11T21:00:48.469854+0000","last_became_peered":"2024-03-11T21:00:48.469854+0000","last_unstale":"2024-03-11T21:04:49.710927+0000","last_undegraded":"2024-03-11T21:04:49.710927+0000","last_fullsized":"2024-03-11T21:04:49.710927+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'44","last_scrub_stamp":"2024-03-11T21:04:35.674462+0000","last_deep_scrub":"161'28","last_deep_scrub_stamp":"2024-03-11T20:57:20.816278+0000","last_clean_scrub_stamp":"2024-03-11T21:04:35.674462+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:05:39.125762+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.7","version":"282'8160","reported_seq":8232,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:50.136891+0000","last_change":"2024-03-11T21:02:47.637634+0000","last_active":"2024-03-11T21:04:50.136891+0000","last_peered":"2024-03-11T21:04:50.136891+0000","last_clean":"2024-03-11T21:04:50.136891+0000","last_became_active":"2024-03-11T21:02:47.637110+0000","last_became_peered":"2024-03-11T21:02:47.637110+0000","last_unstale":"2024-03-11T21:04:50.136891+0000","last_undegraded":"2024-03-11T21:04:50.136891+0000","last_fullsized":"2024-03-11T21:04:50.136891+0000","mapping_epoch":251,"log_start":"261'2800","ondisk_log_start":"261'2800","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:02:46.570492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:46.570492+0000","last_clean_scrub_stamp":"2024-03-11T21:02:46.570492+0000","objects_scrubbed":0,"log_size":5360,"log_dups_size":0,"ondisk_log_size":5360,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":388759552,"num_objects":5932,"num_object_clones":0,"num_object_copies":17796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5932,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8160,"num_write_kb":450944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":461,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:49.651728+0000","last_change":"2024-03-11T21:03:17.004069+0000","last_active":"2024-03-11T21:04:49.651728+0000","last_peered":"2024-03-11T21:04:49.651728+0000","last_clean":"2024-03-11T21:04:49.651728+0000","last_became_active":"2024-03-11T20:50:34.327139+0000","last_became_peered":"2024-03-11T20:50:34.327139+0000","last_unstale":"2024-03-11T21:04:49.651728+0000","last_undegraded":"2024-03-11T21:04:49.651728+0000","last_fullsized":"2024-03-11T21:04:49.651728+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:17.003975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:03:17.003975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":497,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:49.648614+0000","last_change":"2024-03-11T21:04:37.972028+0000","last_active":"2024-03-11T21:04:49.648614+0000","last_peered":"2024-03-11T21:04:49.648614+0000","last_clean":"2024-03-11T21:04:49.648614+0000","last_became_active":"2024-03-11T20:50:28.049901+0000","last_became_peered":"2024-03-11T20:50:28.049901+0000","last_unstale":"2024-03-11T21:04:49.648614+0000","last_undegraded":"2024-03-11T21:04:49.648614+0000","last_fullsized":"2024-03-11T21:04:49.648614+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:04:37.971874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:57:17.447191+0000","last_clean_scrub_stamp":"2024-03-11T21:04:37.971874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:05:38.692199+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"207'39","reported_seq":534,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:49.652761+0000","last_change":"2024-03-11T21:03:14.082722+0000","last_active":"2024-03-11T21:04:49.652761+0000","last_peered":"2024-03-11T21:04:49.652761+0000","last_clean":"2024-03-11T21:04:49.652761+0000","last_became_active":"2024-03-11T20:50:30.111819+0000","last_became_peered":"2024-03-11T20:50:30.111819+0000","last_unstale":"2024-03-11T21:04:49.652761+0000","last_undegraded":"2024-03-11T21:04:49.652761+0000","last_fullsized":"2024-03-11T21:04:49.652761+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'39","last_scrub_stamp":"2024-03-11T21:03:14.082576+0000","last_deep_scrub":"102'25","last_deep_scrub_stamp":"2024-03-11T20:51:35.187094+0000","last_clean_scrub_stamp":"2024-03-11T21:03:14.082576+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":493,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:49.710616+0000","last_change":"2024-03-11T21:04:42.685438+0000","last_active":"2024-03-11T21:04:49.710616+0000","last_peered":"2024-03-11T21:04:49.710616+0000","last_clean":"2024-03-11T21:04:49.710616+0000","last_became_active":"2024-03-11T21:00:48.362389+0000","last_became_peered":"2024-03-11T21:00:48.362389+0000","last_unstale":"2024-03-11T21:04:49.710616+0000","last_undegraded":"2024-03-11T21:04:49.710616+0000","last_fullsized":"2024-03-11T21:04:49.710616+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:04:42.685320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:04:42.685320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:05:52.674564+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.6","version":"282'8021","reported_seq":8093,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:50.171250+0000","last_change":"2024-03-11T21:02:47.633710+0000","last_active":"2024-03-11T21:04:50.171250+0000","last_peered":"2024-03-11T21:04:50.171250+0000","last_clean":"2024-03-11T21:04:50.171250+0000","last_became_active":"2024-03-11T21:02:47.633198+0000","last_became_peered":"2024-03-11T21:02:47.633198+0000","last_unstale":"2024-03-11T21:04:50.171250+0000","last_undegraded":"2024-03-11T21:04:50.171250+0000","last_fullsized":"2024-03-11T21:04:50.171250+0000","mapping_epoch":251,"log_start":"261'2600","ondisk_log_start":"261'2600","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:02:46.570492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:46.570492+0000","last_clean_scrub_stamp":"2024-03-11T21:02:46.570492+0000","objects_scrubbed":0,"log_size":5421,"log_dups_size":0,"ondisk_log_size":5421,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":389480448,"num_objects":5943,"num_object_clones":0,"num_object_copies":17829,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5943,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8021,"num_write_kb":446848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":281,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:49.652318+0000","last_change":"2024-03-11T21:03:11.086739+0000","last_active":"2024-03-11T21:04:49.652318+0000","last_peered":"2024-03-11T21:04:49.652318+0000","last_clean":"2024-03-11T21:04:49.652318+0000","last_became_active":"2024-03-11T20:56:59.801112+0000","last_became_peered":"2024-03-11T20:56:59.801112+0000","last_unstale":"2024-03-11T21:04:49.652318+0000","last_undegraded":"2024-03-11T21:04:49.652318+0000","last_fullsized":"2024-03-11T21:04:49.652318+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:11.086661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:55:58.888890+0000","last_clean_scrub_stamp":"2024-03-11T21:03:11.086661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":174,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-03-11T21:04:51.687890+0000","last_change":"2024-03-11T21:03:17.298880+0000","last_active":"2024-03-11T21:04:51.687890+0000","last_peered":"2024-03-11T21:04:51.687890+0000","last_clean":"2024-03-11T21:04:51.687890+0000","last_became_active":"2024-03-11T21:00:48.469015+0000","last_became_peered":"2024-03-11T21:00:48.469015+0000","last_unstale":"2024-03-11T21:04:51.687890+0000","last_undegraded":"2024-03-11T21:04:51.687890+0000","last_fullsized":"2024-03-11T21:04:51.687890+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:17.298802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:03:17.298802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"207'20","reported_seq":500,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:49.648762+0000","last_change":"2024-03-11T21:03:41.904778+0000","last_active":"2024-03-11T21:04:49.648762+0000","last_peered":"2024-03-11T21:04:49.648762+0000","last_clean":"2024-03-11T21:04:49.648762+0000","last_became_active":"2024-03-11T20:50:30.109549+0000","last_became_peered":"2024-03-11T20:50:30.109549+0000","last_unstale":"2024-03-11T21:04:49.648762+0000","last_undegraded":"2024-03-11T21:04:49.648762+0000","last_fullsized":"2024-03-11T21:04:49.648762+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'20","last_scrub_stamp":"2024-03-11T21:03:41.904604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:03:41.904604+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:04:50.209216+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":503,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:49.717827+0000","last_change":"2024-03-11T21:03:39.413187+0000","last_active":"2024-03-11T21:04:49.717827+0000","last_peered":"2024-03-11T21:04:49.717827+0000","last_clean":"2024-03-11T21:04:49.717827+0000","last_became_active":"2024-03-11T21:02:37.517748+0000","last_became_peered":"2024-03-11T21:02:37.517748+0000","last_unstale":"2024-03-11T21:04:49.717827+0000","last_undegraded":"2024-03-11T21:04:49.717827+0000","last_fullsized":"2024-03-11T21:04:49.717827+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:39.412996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:52:51.931295+0000","last_clean_scrub_stamp":"2024-03-11T21:03:39.412996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:05:06.183543+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.5","version":"282'7920","reported_seq":7992,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:50.139399+0000","last_change":"2024-03-11T21:02:47.619123+0000","last_active":"2024-03-11T21:04:50.139399+0000","last_peered":"2024-03-11T21:04:50.139399+0000","last_clean":"2024-03-11T21:04:50.139399+0000","last_became_active":"2024-03-11T21:02:47.618756+0000","last_became_peered":"2024-03-11T21:02:47.618756+0000","last_unstale":"2024-03-11T21:04:50.139399+0000","last_undegraded":"2024-03-11T21:04:50.139399+0000","last_fullsized":"2024-03-11T21:04:50.139399+0000","mapping_epoch":251,"log_start":"261'2500","ondisk_log_start":"261'2500","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:02:46.570492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:46.570492+0000","last_clean_scrub_stamp":"2024-03-11T21:02:46.570492+0000","objects_scrubbed":0,"log_size":5420,"log_dups_size":0,"ondisk_log_size":5420,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":380502016,"num_objects":5806,"num_object_clones":0,"num_object_copies":17418,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7920,"num_write_kb":439232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":470,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:49.650778+0000","last_change":"2024-03-11T21:03:12.778847+0000","last_active":"2024-03-11T21:04:49.650778+0000","last_peered":"2024-03-11T21:04:49.650778+0000","last_clean":"2024-03-11T21:04:49.650778+0000","last_became_active":"2024-03-11T20:50:34.337173+0000","last_became_peered":"2024-03-11T20:50:34.337173+0000","last_unstale":"2024-03-11T21:04:49.650778+0000","last_undegraded":"2024-03-11T21:04:49.650778+0000","last_fullsized":"2024-03-11T21:04:49.650778+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:12.778718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:00:14.030599+0000","last_clean_scrub_stamp":"2024-03-11T21:03:12.778718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":489,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:49.649202+0000","last_change":"2024-03-11T21:04:48.909061+0000","last_active":"2024-03-11T21:04:49.649202+0000","last_peered":"2024-03-11T21:04:49.649202+0000","last_clean":"2024-03-11T21:04:49.649202+0000","last_became_active":"2024-03-11T20:50:28.058942+0000","last_became_peered":"2024-03-11T20:50:28.058942+0000","last_unstale":"2024-03-11T21:04:49.649202+0000","last_undegraded":"2024-03-11T21:04:49.649202+0000","last_fullsized":"2024-03-11T21:04:49.649202+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:04:48.908945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:04:48.908945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:06:14.438316+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"207'27","reported_seq":521,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:49.652825+0000","last_change":"2024-03-11T21:03:48.964872+0000","last_active":"2024-03-11T21:04:49.652825+0000","last_peered":"2024-03-11T21:04:49.652825+0000","last_clean":"2024-03-11T21:04:49.652825+0000","last_became_active":"2024-03-11T21:00:27.963119+0000","last_became_peered":"2024-03-11T21:00:27.963119+0000","last_unstale":"2024-03-11T21:04:49.652825+0000","last_undegraded":"2024-03-11T21:04:49.652825+0000","last_fullsized":"2024-03-11T21:04:49.652825+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'27","last_scrub_stamp":"2024-03-11T21:03:48.964758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:03:48.964758+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:05:11.347970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":466,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-03-11T21:04:49.648282+0000","last_change":"2024-03-11T21:03:44.899203+0000","last_active":"2024-03-11T21:04:49.648282+0000","last_peered":"2024-03-11T21:04:49.648282+0000","last_clean":"2024-03-11T21:04:49.648282+0000","last_became_active":"2024-03-11T20:50:32.652854+0000","last_became_peered":"2024-03-11T20:50:32.652854+0000","last_unstale":"2024-03-11T21:04:49.648282+0000","last_undegraded":"2024-03-11T21:04:49.648282+0000","last_fullsized":"2024-03-11T21:04:49.648282+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:03:44.899076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:59:00.946094+0000","last_clean_scrub_stamp":"2024-03-11T21:03:44.899076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":6134038552,"num_objects":93599,"num_object_clones":0,"num_object_copies":280797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":93599,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":128689,"num_write_kb":7113153,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18385547264,"data_stored":18385535048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80789,"ondisk_log_size":80789,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":20,"num_read_kb":20,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1822,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1127,"num_read_kb":884,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":174,"num_bytes_recovered":2228,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3632,"internal_metadata":0},"log_size":773,"ondisk_log_size":773,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":121,"ondisk_log_size":121,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":727,"num_read_kb":722,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":68,"seq":292057776311,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3728944,"kb_used_data":3413160,"kb_used_omap":1,"kb_used_meta":315774,"kb_avail":90016208,"statfs":{"total":95995035648,"available":92176596992,"internally_reserved":0,"allocated":3495075840,"data_stored":3494736019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":323353034},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":57,"seq":244813136065,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1412268,"kb_used_data":1138144,"kb_used_omap":1,"kb_used_meta":274110,"kb_avail":92332884,"statfs":{"total":95995035648,"available":94548873216,"internally_reserved":0,"allocated":1165459456,"data_stored":1165136055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":280689095},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":211,"seq":906238099506,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3527360,"kb_used_data":3346676,"kb_used_omap":7,"kb_used_meta":180664,"kb_avail":90217792,"statfs":{"total":95995035648,"available":92383019008,"internally_reserved":0,"allocated":3426996224,"data_stored":3426660311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8078,"internal_metadata":185000050},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738623,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1797048,"kb_used_data":1490984,"kb_used_omap":1,"kb_used_meta":306046,"kb_avail":91948104,"statfs":{"total":95995035648,"available":94154858496,"internally_reserved":0,"allocated":1526767616,"data_stored":1526434955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":313391557},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378858,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1423652,"kb_used_data":1139796,"kb_used_omap":1,"kb_used_meta":283838,"kb_avail":92321500,"statfs":{"total":95995035648,"available":94537216000,"internally_reserved":0,"allocated":1167151104,"data_stored":1166838541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":290650565},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":28,"seq":120259084511,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3395044,"kb_used_data":2984856,"kb_used_omap":1,"kb_used_meta":410174,"kb_avail":90350108,"statfs":{"total":95995035648,"available":92518510592,"internally_reserved":0,"allocated":3056492544,"data_stored":3056176261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":420018632},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":204,"seq":876173328438,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1241228,"kb_used_data":1132476,"kb_used_omap":4,"kb_used_meta":108731,"kb_avail":92503924,"statfs":{"total":95995035648,"available":94724018176,"internally_reserved":0,"allocated":1159655424,"data_stored":1159303667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4451,"internal_metadata":111341213},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":44,"seq":188978561227,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3707160,"kb_used_data":3396880,"kb_used_omap":1,"kb_used_meta":310206,"kb_avail":90037992,"statfs":{"total":95995035648,"available":92198903808,"internally_reserved":0,"allocated":3478405120,"data_stored":3478088357,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":317651397},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3632,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1523580928,"data_stored":1523580928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3418951680,"data_stored":3418947608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1161887744,"data_stored":1161887744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3053060096,"data_stored":3053060096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1153761280,"data_stored":1153761280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3447459840,"data_stored":3447455768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1158348800,"data_stored":1158348800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":3468496896,"data_stored":3468492824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T21:04:55.798 INFO:tasks.cephadm.ceph_manager.ceph:would drop below min_pgs, proj 6, currently 16 2024-03-11T21:04:55.816 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:55.878 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:55.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:55.881 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:04:55.888 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-03-11T21:04:55.888 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-03-11T21:04:55.888 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 30, 2024-03-11T21:04:55.888 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-03-11T21:04:55.888 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-03-11T21:04:55.907 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:55 smithi146 bash[23809]: cluster 2024-03-11T21:04:53.498790+0000 osd.5 (osd.5) 259 : cluster 0 2.2 scrub starts 2024-03-11T21:04:55.907 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:55 smithi146 bash[23809]: cluster 2024-03-11T21:04:53.512336+0000 osd.5 (osd.5) 260 : cluster 0 2.2 scrub ok 2024-03-11T21:04:55.907 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:55 smithi146 bash[23809]: cluster 2024-03-11T21:04:55.136091+0000 osd.7 (osd.7) 303 : cluster 0 9.e scrub ok 2024-03-11T21:04:55.907 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:55 smithi146 bash[23809]: audit 2024-03-11T21:04:55.522783+0000 mgr.y (mgr.24380) 525 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:04:55.982 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:04:56.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:56.014 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:04:56.014 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:04:56.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:55 smithi023 bash[21372]: cluster 2024-03-11T21:04:53.498790+0000 osd.5 (osd.5) 259 : cluster 0 2.2 scrub starts 2024-03-11T21:04:56.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:55 smithi023 bash[21372]: cluster 2024-03-11T21:04:53.512336+0000 osd.5 (osd.5) 260 : cluster 0 2.2 scrub ok 2024-03-11T21:04:56.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:55 smithi023 bash[21372]: cluster 2024-03-11T21:04:55.136091+0000 osd.7 (osd.7) 303 : cluster 0 9.e scrub ok 2024-03-11T21:04:56.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:55 smithi023 bash[21372]: audit 2024-03-11T21:04:55.522783+0000 mgr.y (mgr.24380) 525 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:04:56.033 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:55 smithi023 bash[29146]: cluster 2024-03-11T21:04:53.498790+0000 osd.5 (osd.5) 259 : cluster 0 2.2 scrub starts 2024-03-11T21:04:56.033 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:55 smithi023 bash[29146]: cluster 2024-03-11T21:04:53.512336+0000 osd.5 (osd.5) 260 : cluster 0 2.2 scrub ok 2024-03-11T21:04:56.033 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:55 smithi023 bash[29146]: cluster 2024-03-11T21:04:55.136091+0000 osd.7 (osd.7) 303 : cluster 0 9.e scrub ok 2024-03-11T21:04:56.033 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:55 smithi023 bash[29146]: audit 2024-03-11T21:04:55.522783+0000 mgr.y (mgr.24380) 525 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:04:56.116 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:04:56.138 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:04:56.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:56.141 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:04:56.242 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:04:56.272 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:04:56.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:56.274 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:04:56.375 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:04:56.399 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:04:56.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:56.401 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:04:56.502 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:04:56.528 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:04:56.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:56.530 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:04:56.631 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:56.659 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:56.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:56.661 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:04:56.762 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:04:56.788 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:04:56.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:56.791 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:04:56.892 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:56.901 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:56 smithi023 bash[29146]: cluster 2024-03-11T21:04:55.231036+0000 osd.3 (osd.3) 292 : cluster 0 4.19 scrub starts 2024-03-11T21:04:56.901 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:56 smithi023 bash[29146]: cluster 2024-03-11T21:04:55.242396+0000 osd.3 (osd.3) 293 : cluster 0 4.19 scrub ok 2024-03-11T21:04:56.902 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:56 smithi023 bash[29146]: cluster 2024-03-11T21:04:56.189566+0000 mgr.y (mgr.24380) 526 : cluster 0 pgmap v587: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 782 B/s rd, 0 B/s wr, 1.05k op/s 2024-03-11T21:04:56.902 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:56 smithi023 bash[29146]: cluster 2024-03-11T21:04:56.421863+0000 osd.7 (osd.7) 304 : cluster 0 9.3 scrub ok 2024-03-11T21:04:56.902 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:56 smithi023 bash[21372]: cluster 2024-03-11T21:04:55.231036+0000 osd.3 (osd.3) 292 : cluster 0 4.19 scrub starts 2024-03-11T21:04:56.902 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:56 smithi023 bash[21372]: cluster 2024-03-11T21:04:55.242396+0000 osd.3 (osd.3) 293 : cluster 0 4.19 scrub ok 2024-03-11T21:04:56.902 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:56 smithi023 bash[21372]: cluster 2024-03-11T21:04:56.189566+0000 mgr.y (mgr.24380) 526 : cluster 0 pgmap v587: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 782 B/s rd, 0 B/s wr, 1.05k op/s 2024-03-11T21:04:56.902 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:56 smithi023 bash[21372]: cluster 2024-03-11T21:04:56.421863+0000 osd.7 (osd.7) 304 : cluster 0 9.3 scrub ok 2024-03-11T21:04:56.913 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:56.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:56.917 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:04:57.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:56 smithi146 bash[23809]: cluster 2024-03-11T21:04:55.231036+0000 osd.3 (osd.3) 292 : cluster 0 4.19 scrub starts 2024-03-11T21:04:57.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:56 smithi146 bash[23809]: cluster 2024-03-11T21:04:55.242396+0000 osd.3 (osd.3) 293 : cluster 0 4.19 scrub ok 2024-03-11T21:04:57.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:56 smithi146 bash[23809]: cluster 2024-03-11T21:04:56.189566+0000 mgr.y (mgr.24380) 526 : cluster 0 pgmap v587: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 782 B/s rd, 0 B/s wr, 1.05k op/s 2024-03-11T21:04:57.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:56 smithi146 bash[23809]: cluster 2024-03-11T21:04:56.421863+0000 osd.7 (osd.7) 304 : cluster 0 9.3 scrub ok 2024-03-11T21:04:57.017 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:04:57.044 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:04:57.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:57.046 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:04:57.147 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:57.176 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:57.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:57.180 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:04:57.280 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:04:57.303 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:04:57.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:57.307 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:04:57.407 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:57.443 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:57.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:57.448 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:04:57.548 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:04:57.570 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:04:57.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:57.573 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:04:57.674 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:04:57.705 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:04:57.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:57.712 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:04:57.773 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T21:04:57.813 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:57.816 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd set noscrub 2024-03-11T21:04:57.830 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:57 smithi023 bash[21372]: cluster 2024-03-11T21:04:55.811327+0000 osd.5 (osd.5) 261 : cluster 0 9.7 scrub starts 2024-03-11T21:04:57.830 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:57 smithi023 bash[21372]: cluster 2024-03-11T21:04:56.268243+0000 osd.3 (osd.3) 294 : cluster 0 3.1e scrub starts 2024-03-11T21:04:57.830 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:57 smithi023 bash[21372]: cluster 2024-03-11T21:04:56.274669+0000 osd.3 (osd.3) 295 : cluster 0 3.1e scrub ok 2024-03-11T21:04:57.830 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:57 smithi023 bash[21372]: cluster 2024-03-11T21:04:56.488301+0000 osd.4 (osd.4) 97 : cluster 0 4.d scrub starts 2024-03-11T21:04:57.830 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:57 smithi023 bash[21372]: cluster 2024-03-11T21:04:56.489960+0000 osd.4 (osd.4) 98 : cluster 0 4.d scrub ok 2024-03-11T21:04:57.831 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:57 smithi023 bash[29146]: cluster 2024-03-11T21:04:55.811327+0000 osd.5 (osd.5) 261 : cluster 0 9.7 scrub starts 2024-03-11T21:04:57.831 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:57 smithi023 bash[29146]: cluster 2024-03-11T21:04:56.268243+0000 osd.3 (osd.3) 294 : cluster 0 3.1e scrub starts 2024-03-11T21:04:57.831 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:57 smithi023 bash[29146]: cluster 2024-03-11T21:04:56.274669+0000 osd.3 (osd.3) 295 : cluster 0 3.1e scrub ok 2024-03-11T21:04:57.831 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:57 smithi023 bash[29146]: cluster 2024-03-11T21:04:56.488301+0000 osd.4 (osd.4) 97 : cluster 0 4.d scrub starts 2024-03-11T21:04:57.831 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:57 smithi023 bash[29146]: cluster 2024-03-11T21:04:56.489960+0000 osd.4 (osd.4) 98 : cluster 0 4.d scrub ok 2024-03-11T21:04:57.838 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:57.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:57.842 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:04:57.943 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:04:58.001 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:04:58.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:58.004 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:04:58.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:57 smithi146 bash[23809]: cluster 2024-03-11T21:04:55.811327+0000 osd.5 (osd.5) 261 : cluster 0 9.7 scrub starts 2024-03-11T21:04:58.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:57 smithi146 bash[23809]: cluster 2024-03-11T21:04:56.268243+0000 osd.3 (osd.3) 294 : cluster 0 3.1e scrub starts 2024-03-11T21:04:58.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:57 smithi146 bash[23809]: cluster 2024-03-11T21:04:56.274669+0000 osd.3 (osd.3) 295 : cluster 0 3.1e scrub ok 2024-03-11T21:04:58.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:57 smithi146 bash[23809]: cluster 2024-03-11T21:04:56.488301+0000 osd.4 (osd.4) 97 : cluster 0 4.d scrub starts 2024-03-11T21:04:58.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:57 smithi146 bash[23809]: cluster 2024-03-11T21:04:56.489960+0000 osd.4 (osd.4) 98 : cluster 0 4.d scrub ok 2024-03-11T21:04:58.105 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:04:58.138 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:04:58.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:58.145 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:04:58.246 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:04:58.275 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:04:58.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:58.280 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:04:58.326 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-03-11T21:04:58.380 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:58.403 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:58.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:58.407 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:04:58.508 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:04:58.518 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:04:58 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:04:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:04:58.536 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:04:58.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:58.538 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:04:58.638 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:58.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:58.673 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:04:58.673 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:04:58.774 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:04:58.782 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:58 smithi023 bash[21372]: cluster 2024-03-11T21:04:56.834842+0000 osd.5 (osd.5) 262 : cluster 0 9.f scrub starts 2024-03-11T21:04:58.782 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:58 smithi023 bash[21372]: cluster 2024-03-11T21:04:57.593354+0000 osd.7 (osd.7) 305 : cluster 0 6.9 scrub starts 2024-03-11T21:04:58.782 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:58 smithi023 bash[21372]: cluster 2024-03-11T21:04:57.639923+0000 osd.7 (osd.7) 306 : cluster 0 6.9 scrub ok 2024-03-11T21:04:58.782 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:58 smithi023 bash[21372]: cluster 2024-03-11T21:04:57.836175+0000 osd.1 (osd.1) 67 : cluster 0 2.6 deep-scrub starts 2024-03-11T21:04:58.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:58 smithi023 bash[21372]: cluster 2024-03-11T21:04:57.850489+0000 osd.1 (osd.1) 68 : cluster 0 2.6 deep-scrub ok 2024-03-11T21:04:58.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:58 smithi023 bash[21372]: cluster 2024-03-11T21:04:58.190187+0000 mgr.y (mgr.24380) 527 : cluster 0 pgmap v588: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 919 op/s 2024-03-11T21:04:58.783 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:58 smithi023 bash[29146]: cluster 2024-03-11T21:04:56.834842+0000 osd.5 (osd.5) 262 : cluster 0 9.f scrub starts 2024-03-11T21:04:58.783 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:58 smithi023 bash[29146]: cluster 2024-03-11T21:04:57.593354+0000 osd.7 (osd.7) 305 : cluster 0 6.9 scrub starts 2024-03-11T21:04:58.783 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:58 smithi023 bash[29146]: cluster 2024-03-11T21:04:57.639923+0000 osd.7 (osd.7) 306 : cluster 0 6.9 scrub ok 2024-03-11T21:04:58.783 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:58 smithi023 bash[29146]: cluster 2024-03-11T21:04:57.836175+0000 osd.1 (osd.1) 67 : cluster 0 2.6 deep-scrub starts 2024-03-11T21:04:58.783 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:58 smithi023 bash[29146]: cluster 2024-03-11T21:04:57.850489+0000 osd.1 (osd.1) 68 : cluster 0 2.6 deep-scrub ok 2024-03-11T21:04:58.783 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:58 smithi023 bash[29146]: cluster 2024-03-11T21:04:58.190187+0000 mgr.y (mgr.24380) 527 : cluster 0 pgmap v588: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 919 op/s 2024-03-11T21:04:58.806 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:04:58.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:58.812 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:04:58.913 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:04:58.940 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:04:58.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:58.945 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:04:59.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:58 smithi146 bash[23809]: cluster 2024-03-11T21:04:56.834842+0000 osd.5 (osd.5) 262 : cluster 0 9.f scrub starts 2024-03-11T21:04:59.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:58 smithi146 bash[23809]: cluster 2024-03-11T21:04:57.593354+0000 osd.7 (osd.7) 305 : cluster 0 6.9 scrub starts 2024-03-11T21:04:59.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:58 smithi146 bash[23809]: cluster 2024-03-11T21:04:57.639923+0000 osd.7 (osd.7) 306 : cluster 0 6.9 scrub ok 2024-03-11T21:04:59.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:58 smithi146 bash[23809]: cluster 2024-03-11T21:04:57.836175+0000 osd.1 (osd.1) 67 : cluster 0 2.6 deep-scrub starts 2024-03-11T21:04:59.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:58 smithi146 bash[23809]: cluster 2024-03-11T21:04:57.850489+0000 osd.1 (osd.1) 68 : cluster 0 2.6 deep-scrub ok 2024-03-11T21:04:59.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:58 smithi146 bash[23809]: cluster 2024-03-11T21:04:58.190187+0000 mgr.y (mgr.24380) 527 : cluster 0 pgmap v588: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 919 op/s 2024-03-11T21:04:59.045 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:59.077 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:59.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:59.079 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:04:59.179 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:04:59.204 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:04:59.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:59.210 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:04:59.310 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:04:59.336 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:04:59.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:59.338 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:04:59.439 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:59.466 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:04:59.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:59.469 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:04:59.570 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:04:59.593 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:04:59.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:59.598 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:04:59.699 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:04:59.733 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:04:59.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:59.738 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:04:59.839 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:04:59.868 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:04:59.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:59.870 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:04:59.971 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:05:00.371 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:59 smithi023 bash[21372]: cluster 2024-03-11T21:04:58.251110+0000 osd.3 (osd.3) 296 : cluster 0 6.1f scrub starts 2024-03-11T21:05:00.371 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:04:59 smithi023 bash[21372]: cluster 2024-03-11T21:04:58.371321+0000 osd.3 (osd.3) 297 : cluster 0 6.1f scrub ok 2024-03-11T21:05:00.372 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:59 smithi023 bash[29146]: cluster 2024-03-11T21:04:58.251110+0000 osd.3 (osd.3) 296 : cluster 0 6.1f scrub starts 2024-03-11T21:05:00.372 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:04:59 smithi023 bash[29146]: cluster 2024-03-11T21:04:58.371321+0000 osd.3 (osd.3) 297 : cluster 0 6.1f scrub ok 2024-03-11T21:05:00.372 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:59 smithi146 bash[23809]: cluster 2024-03-11T21:04:58.251110+0000 osd.3 (osd.3) 296 : cluster 0 6.1f scrub starts 2024-03-11T21:05:00.372 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:04:59 smithi146 bash[23809]: cluster 2024-03-11T21:04:58.371321+0000 osd.3 (osd.3) 297 : cluster 0 6.1f scrub ok 2024-03-11T21:05:00.402 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:05:00.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:00.406 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:05:00.507 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:05:00.542 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:05:00.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:00.546 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:05:00.646 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:05:00.671 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:05:00.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:00.673 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:05:00.774 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:05:00.800 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 6, 7, 1] out_osds: [] dead_osds: [] live_osds: [0, 2, 3, 5, 6, 7, 4, 1] 2024-03-11T21:05:00.800 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-11T21:05:00.800 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-03-11T21:05:00.800 INFO:tasks.thrashosds.thrasher:Setting osd 4 primary_affinity to 0.000000 2024-03-11T21:05:00.800 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd primary-affinity 4 0 2024-03-11T21:05:00.807 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:05:00.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:00.812 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:05:00.912 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:05:00.936 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:05:00.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:00.938 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:05:01.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:00 smithi146 bash[23809]: cluster 2024-03-11T21:04:58.843455+0000 osd.5 (osd.5) 263 : cluster 0 9.7 scrub ok 2024-03-11T21:05:01.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:00 smithi146 bash[23809]: cluster 2024-03-11T21:04:59.475642+0000 osd.5 (osd.5) 264 : cluster 0 9.f scrub ok 2024-03-11T21:05:01.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:00 smithi146 bash[23809]: cluster 2024-03-11T21:04:59.652888+0000 osd.7 (osd.7) 307 : cluster 0 2.8 deep-scrub starts 2024-03-11T21:05:01.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:00 smithi146 bash[23809]: cluster 2024-03-11T21:04:59.654837+0000 osd.7 (osd.7) 308 : cluster 0 2.8 deep-scrub ok 2024-03-11T21:05:01.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:00 smithi146 bash[23809]: cluster 2024-03-11T21:04:59.790071+0000 osd.6 (osd.6) 224 : cluster 0 2.14 scrub starts 2024-03-11T21:05:01.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:00 smithi146 bash[23809]: cluster 2024-03-11T21:04:59.819352+0000 osd.6 (osd.6) 225 : cluster 0 2.14 scrub ok 2024-03-11T21:05:01.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:00 smithi146 bash[23809]: cluster 2024-03-11T21:05:00.190743+0000 mgr.y (mgr.24380) 528 : cluster 0 pgmap v589: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 601 B/s rd, 0 B/s wr, 795 op/s 2024-03-11T21:05:01.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:00 smithi146 bash[23809]: cluster 2024-03-11T21:05:00.225809+0000 osd.3 (osd.3) 298 : cluster 0 6.0 scrub starts 2024-03-11T21:05:01.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:00 smithi023 bash[21372]: cluster 2024-03-11T21:04:58.843455+0000 osd.5 (osd.5) 263 : cluster 0 9.7 scrub ok 2024-03-11T21:05:01.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:00 smithi023 bash[21372]: cluster 2024-03-11T21:04:59.475642+0000 osd.5 (osd.5) 264 : cluster 0 9.f scrub ok 2024-03-11T21:05:01.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:00 smithi023 bash[21372]: cluster 2024-03-11T21:04:59.652888+0000 osd.7 (osd.7) 307 : cluster 0 2.8 deep-scrub starts 2024-03-11T21:05:01.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:00 smithi023 bash[21372]: cluster 2024-03-11T21:04:59.654837+0000 osd.7 (osd.7) 308 : cluster 0 2.8 deep-scrub ok 2024-03-11T21:05:01.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:00 smithi023 bash[21372]: cluster 2024-03-11T21:04:59.790071+0000 osd.6 (osd.6) 224 : cluster 0 2.14 scrub starts 2024-03-11T21:05:01.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:00 smithi023 bash[21372]: cluster 2024-03-11T21:04:59.819352+0000 osd.6 (osd.6) 225 : cluster 0 2.14 scrub ok 2024-03-11T21:05:01.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:00 smithi023 bash[21372]: cluster 2024-03-11T21:05:00.190743+0000 mgr.y (mgr.24380) 528 : cluster 0 pgmap v589: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 601 B/s rd, 0 B/s wr, 795 op/s 2024-03-11T21:05:01.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:00 smithi023 bash[21372]: cluster 2024-03-11T21:05:00.225809+0000 osd.3 (osd.3) 298 : cluster 0 6.0 scrub starts 2024-03-11T21:05:01.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:00 smithi023 bash[29146]: cluster 2024-03-11T21:04:58.843455+0000 osd.5 (osd.5) 263 : cluster 0 9.7 scrub ok 2024-03-11T21:05:01.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:00 smithi023 bash[29146]: cluster 2024-03-11T21:04:59.475642+0000 osd.5 (osd.5) 264 : cluster 0 9.f scrub ok 2024-03-11T21:05:01.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:00 smithi023 bash[29146]: cluster 2024-03-11T21:04:59.652888+0000 osd.7 (osd.7) 307 : cluster 0 2.8 deep-scrub starts 2024-03-11T21:05:01.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:00 smithi023 bash[29146]: cluster 2024-03-11T21:04:59.654837+0000 osd.7 (osd.7) 308 : cluster 0 2.8 deep-scrub ok 2024-03-11T21:05:01.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:00 smithi023 bash[29146]: cluster 2024-03-11T21:04:59.790071+0000 osd.6 (osd.6) 224 : cluster 0 2.14 scrub starts 2024-03-11T21:05:01.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:00 smithi023 bash[29146]: cluster 2024-03-11T21:04:59.819352+0000 osd.6 (osd.6) 225 : cluster 0 2.14 scrub ok 2024-03-11T21:05:01.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:00 smithi023 bash[29146]: cluster 2024-03-11T21:05:00.190743+0000 mgr.y (mgr.24380) 528 : cluster 0 pgmap v589: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 601 B/s rd, 0 B/s wr, 795 op/s 2024-03-11T21:05:01.035 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:00 smithi023 bash[29146]: cluster 2024-03-11T21:05:00.225809+0000 osd.3 (osd.3) 298 : cluster 0 6.0 scrub starts 2024-03-11T21:05:01.038 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:01.065 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:01.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:01.072 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:05:01.173 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:05:01.197 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:05:01.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:01.199 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:05:01.300 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:05:01.327 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:05:01.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:01.331 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:05:01.432 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:01.457 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:01.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:01.460 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:05:01.561 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:01.589 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:01.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:01.592 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:05:01.692 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:01.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:01.721 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:01.721 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:05:01.822 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:01.850 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:01.853 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:01 smithi146 bash[23809]: cluster 2024-03-11T21:05:00.290720+0000 osd.3 (osd.3) 299 : cluster 0 6.0 scrub ok 2024-03-11T21:05:01.853 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:01 smithi146 bash[23809]: cluster 2024-03-11T21:05:00.431730+0000 osd.4 (osd.4) 99 : cluster 0 2.13 scrub starts 2024-03-11T21:05:01.853 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:01 smithi146 bash[23809]: cluster 2024-03-11T21:05:00.433982+0000 osd.4 (osd.4) 100 : cluster 0 2.13 scrub ok 2024-03-11T21:05:01.853 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:01 smithi146 bash[23809]: audit 2024-03-11T21:05:01.611591+0000 mon.a (mon.0) 1525 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:05:01.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:01.854 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:05:01.873 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:01 smithi023 bash[21372]: cluster 2024-03-11T21:05:00.290720+0000 osd.3 (osd.3) 299 : cluster 0 6.0 scrub ok 2024-03-11T21:05:01.873 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:01 smithi023 bash[21372]: cluster 2024-03-11T21:05:00.431730+0000 osd.4 (osd.4) 99 : cluster 0 2.13 scrub starts 2024-03-11T21:05:01.873 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:01 smithi023 bash[21372]: cluster 2024-03-11T21:05:00.433982+0000 osd.4 (osd.4) 100 : cluster 0 2.13 scrub ok 2024-03-11T21:05:01.873 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:01 smithi023 bash[21372]: audit 2024-03-11T21:05:01.611591+0000 mon.a (mon.0) 1525 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:05:01.873 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:01 smithi023 bash[29146]: cluster 2024-03-11T21:05:00.290720+0000 osd.3 (osd.3) 299 : cluster 0 6.0 scrub ok 2024-03-11T21:05:01.873 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:01 smithi023 bash[29146]: cluster 2024-03-11T21:05:00.431730+0000 osd.4 (osd.4) 99 : cluster 0 2.13 scrub starts 2024-03-11T21:05:01.873 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:01 smithi023 bash[29146]: cluster 2024-03-11T21:05:00.433982+0000 osd.4 (osd.4) 100 : cluster 0 2.13 scrub ok 2024-03-11T21:05:01.873 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:01 smithi023 bash[29146]: audit 2024-03-11T21:05:01.611591+0000 mon.a (mon.0) 1525 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:05:01.955 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:01.986 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:01.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:01.992 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:05:02.093 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:02.133 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:02.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:02.136 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:05:02.236 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:05:02.271 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:05:02.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:02.273 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:05:02.374 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:02.400 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:02.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:02.402 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:05:02.503 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:02.528 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:02.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:02.530 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:05:02.631 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:05:02.639 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:05:02.639 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:05:02.658 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:05:02.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:02.661 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:05:02.762 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:02.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:02 smithi146 bash[23809]: cluster 2024-03-11T21:05:00.821001+0000 osd.5 (osd.5) 265 : cluster 0 9.6 deep-scrub starts 2024-03-11T21:05:02.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:02 smithi146 bash[23809]: cluster 2024-03-11T21:05:01.665779+0000 osd.7 (osd.7) 309 : cluster 0 6.8 scrub starts 2024-03-11T21:05:02.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:02 smithi146 bash[23809]: cluster 2024-03-11T21:05:01.675715+0000 osd.7 (osd.7) 310 : cluster 0 6.8 scrub ok 2024-03-11T21:05:02.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:02 smithi146 bash[23809]: cluster 2024-03-11T21:05:02.191613+0000 mgr.y (mgr.24380) 529 : cluster 0 pgmap v590: 148 pgs: 148 active+clean; 5.1 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 982 op/s 2024-03-11T21:05:02.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:02 smithi146 bash[23809]: audit 2024-03-11T21:05:02.490085+0000 mon.a (mon.0) 1526 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:05:02.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:02 smithi146 bash[23809]: audit 2024-03-11T21:05:02.491216+0000 mon.a (mon.0) 1527 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:05:02.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:02 smithi146 bash[23809]: audit 2024-03-11T21:05:02.498050+0000 mon.a (mon.0) 1528 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:05:02.795 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:02.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:02.798 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:05:02.899 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:05:02.928 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:05:02.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:02.931 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:05:03.032 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:05:03.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:02 smithi023 bash[21372]: cluster 2024-03-11T21:05:00.821001+0000 osd.5 (osd.5) 265 : cluster 0 9.6 deep-scrub starts 2024-03-11T21:05:03.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:02 smithi023 bash[21372]: cluster 2024-03-11T21:05:01.665779+0000 osd.7 (osd.7) 309 : cluster 0 6.8 scrub starts 2024-03-11T21:05:03.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:02 smithi023 bash[21372]: cluster 2024-03-11T21:05:01.675715+0000 osd.7 (osd.7) 310 : cluster 0 6.8 scrub ok 2024-03-11T21:05:03.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:02 smithi023 bash[21372]: cluster 2024-03-11T21:05:02.191613+0000 mgr.y (mgr.24380) 529 : cluster 0 pgmap v590: 148 pgs: 148 active+clean; 5.1 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 982 op/s 2024-03-11T21:05:03.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:02 smithi023 bash[21372]: audit 2024-03-11T21:05:02.490085+0000 mon.a (mon.0) 1526 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:05:03.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:02 smithi023 bash[21372]: audit 2024-03-11T21:05:02.491216+0000 mon.a (mon.0) 1527 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:05:03.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:02 smithi023 bash[21372]: audit 2024-03-11T21:05:02.498050+0000 mon.a (mon.0) 1528 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:05:03.033 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:02 smithi023 bash[29146]: cluster 2024-03-11T21:05:00.821001+0000 osd.5 (osd.5) 265 : cluster 0 9.6 deep-scrub starts 2024-03-11T21:05:03.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:02 smithi023 bash[29146]: cluster 2024-03-11T21:05:01.665779+0000 osd.7 (osd.7) 309 : cluster 0 6.8 scrub starts 2024-03-11T21:05:03.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:02 smithi023 bash[29146]: cluster 2024-03-11T21:05:01.675715+0000 osd.7 (osd.7) 310 : cluster 0 6.8 scrub ok 2024-03-11T21:05:03.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:02 smithi023 bash[29146]: cluster 2024-03-11T21:05:02.191613+0000 mgr.y (mgr.24380) 529 : cluster 0 pgmap v590: 148 pgs: 148 active+clean; 5.1 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 982 op/s 2024-03-11T21:05:03.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:02 smithi023 bash[29146]: audit 2024-03-11T21:05:02.490085+0000 mon.a (mon.0) 1526 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:05:03.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:02 smithi023 bash[29146]: audit 2024-03-11T21:05:02.491216+0000 mon.a (mon.0) 1527 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:05:03.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:02 smithi023 bash[29146]: audit 2024-03-11T21:05:02.498050+0000 mon.a (mon.0) 1528 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:05:03.059 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:05:03.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:03.061 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:05:03.139 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:05:03.163 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:05:03.184 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:05:03.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:03.186 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:05:03.287 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:05:03.306 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:05:03.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:03.307 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:05:03.407 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:05:03.437 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:05:03.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:03.439 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:05:03.540 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:03.565 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:03.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:03.568 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:05:03.669 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:05:03.692 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:05:03.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:03.694 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:05:03.795 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:03.831 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:03.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:03.834 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:05:03.935 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:05:03.956 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:05:03.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:03.959 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:05:04.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:03 smithi146 bash[23809]: cluster 2024-03-11T21:05:02.795563+0000 osd.5 (osd.5) 266 : cluster 0 9.5 scrub starts 2024-03-11T21:05:04.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:03 smithi146 bash[23809]: cluster 2024-03-11T21:05:02.890610+0000 osd.1 (osd.1) 69 : cluster 0 4.7 scrub starts 2024-03-11T21:05:04.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:03 smithi146 bash[23809]: cluster 2024-03-11T21:05:02.894365+0000 osd.1 (osd.1) 70 : cluster 0 4.7 scrub ok 2024-03-11T21:05:04.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:03 smithi146 bash[23809]: cluster 2024-03-11T21:05:03.179569+0000 osd.3 (osd.3) 300 : cluster 0 5.0 scrub starts 2024-03-11T21:05:04.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:03 smithi146 bash[23809]: cluster 2024-03-11T21:05:03.181689+0000 osd.3 (osd.3) 301 : cluster 0 5.0 scrub ok 2024-03-11T21:05:04.059 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:05:04.083 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:05:04.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:04.086 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:05:04.186 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:05:04.219 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:05:04.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:04.227 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:05:04.330 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:05:04.363 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:05:04.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:04.365 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:05:04.432 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:04 smithi023 bash[21372]: cluster 2024-03-11T21:05:02.795563+0000 osd.5 (osd.5) 266 : cluster 0 9.5 scrub starts 2024-03-11T21:05:04.432 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:04 smithi023 bash[21372]: cluster 2024-03-11T21:05:02.890610+0000 osd.1 (osd.1) 69 : cluster 0 4.7 scrub starts 2024-03-11T21:05:04.432 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:04 smithi023 bash[21372]: cluster 2024-03-11T21:05:02.894365+0000 osd.1 (osd.1) 70 : cluster 0 4.7 scrub ok 2024-03-11T21:05:04.432 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:04 smithi023 bash[21372]: cluster 2024-03-11T21:05:03.179569+0000 osd.3 (osd.3) 300 : cluster 0 5.0 scrub starts 2024-03-11T21:05:04.432 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:04 smithi023 bash[21372]: cluster 2024-03-11T21:05:03.181689+0000 osd.3 (osd.3) 301 : cluster 0 5.0 scrub ok 2024-03-11T21:05:04.465 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:04.496 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:04.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:04.499 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:05:04.599 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:04.627 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:04.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:04.629 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:05:04.681 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:05:04.711 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:04 smithi023 bash[29146]: cluster 2024-03-11T21:05:02.795563+0000 osd.5 (osd.5) 266 : cluster 0 9.5 scrub starts 2024-03-11T21:05:04.711 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:04 smithi023 bash[29146]: cluster 2024-03-11T21:05:02.890610+0000 osd.1 (osd.1) 69 : cluster 0 4.7 scrub starts 2024-03-11T21:05:04.711 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:04 smithi023 bash[29146]: cluster 2024-03-11T21:05:02.894365+0000 osd.1 (osd.1) 70 : cluster 0 4.7 scrub ok 2024-03-11T21:05:04.711 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:04 smithi023 bash[29146]: cluster 2024-03-11T21:05:03.179569+0000 osd.3 (osd.3) 300 : cluster 0 5.0 scrub starts 2024-03-11T21:05:04.711 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:04 smithi023 bash[29146]: cluster 2024-03-11T21:05:03.181689+0000 osd.3 (osd.3) 301 : cluster 0 5.0 scrub ok 2024-03-11T21:05:04.730 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:04.768 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:04.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:04.772 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:05:04.872 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:04.881 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:04 smithi146 bash[23809]: cluster 2024-03-11T21:05:03.852636+0000 osd.1 (osd.1) 71 : cluster 0 2.d scrub starts 2024-03-11T21:05:04.881 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:04 smithi146 bash[23809]: cluster 2024-03-11T21:05:03.853969+0000 osd.1 (osd.1) 72 : cluster 0 2.d scrub ok 2024-03-11T21:05:04.881 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:04 smithi146 bash[23809]: cluster 2024-03-11T21:05:04.192056+0000 mgr.y (mgr.24380) 530 : cluster 0 pgmap v591: 148 pgs: 148 active+clean; 5.1 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 890 B/s rd, 0 B/s wr, 855 op/s 2024-03-11T21:05:04.892 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:04.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:04.893 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:05:04.994 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:05.020 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:05.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:05.025 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:05:05.032 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:04 smithi023 bash[29146]: cluster 2024-03-11T21:05:03.852636+0000 osd.1 (osd.1) 71 : cluster 0 2.d scrub starts 2024-03-11T21:05:05.032 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:04 smithi023 bash[29146]: cluster 2024-03-11T21:05:03.853969+0000 osd.1 (osd.1) 72 : cluster 0 2.d scrub ok 2024-03-11T21:05:05.032 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:04 smithi023 bash[29146]: cluster 2024-03-11T21:05:04.192056+0000 mgr.y (mgr.24380) 530 : cluster 0 pgmap v591: 148 pgs: 148 active+clean; 5.1 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 890 B/s rd, 0 B/s wr, 855 op/s 2024-03-11T21:05:05.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:04 smithi023 bash[21372]: cluster 2024-03-11T21:05:03.852636+0000 osd.1 (osd.1) 71 : cluster 0 2.d scrub starts 2024-03-11T21:05:05.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:04 smithi023 bash[21372]: cluster 2024-03-11T21:05:03.853969+0000 osd.1 (osd.1) 72 : cluster 0 2.d scrub ok 2024-03-11T21:05:05.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:04 smithi023 bash[21372]: cluster 2024-03-11T21:05:04.192056+0000 mgr.y (mgr.24380) 530 : cluster 0 pgmap v591: 148 pgs: 148 active+clean; 5.1 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 890 B/s rd, 0 B/s wr, 855 op/s 2024-03-11T21:05:05.126 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:05:05.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:05.156 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:05:05.156 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:05:05.257 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:05:05.281 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:05:05.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:05.286 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:05:05.387 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:05:05.406 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:05:05.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:05.408 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:05:05.508 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:05.527 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:05.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:05.529 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:05:05.630 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:05.652 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:05.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:05.655 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:05:05.756 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:05:05.769 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:05:05 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:05:05.790 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:05:05.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:05.794 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:05:05.895 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:05:05.916 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:05:05.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:05.918 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:05:06.018 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:06.030 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:05 smithi023 bash[29146]: cluster 2024-03-11T21:05:04.320331+0000 osd.5 (osd.5) 267 : cluster 0 9.5 scrub ok 2024-03-11T21:05:06.030 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:05 smithi023 bash[29146]: cluster 2024-03-11T21:05:04.909239+0000 osd.1 (osd.1) 73 : cluster 0 3.19 scrub starts 2024-03-11T21:05:06.030 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:05 smithi023 bash[29146]: cluster 2024-03-11T21:05:04.911908+0000 osd.1 (osd.1) 74 : cluster 0 3.19 scrub ok 2024-03-11T21:05:06.030 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:05 smithi023 bash[29146]: cluster 2024-03-11T21:05:05.114912+0000 osd.3 (osd.3) 302 : cluster 0 9.c scrub starts 2024-03-11T21:05:06.030 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:05 smithi023 bash[29146]: audit 2024-03-11T21:05:05.385473+0000 mon.a (mon.0) 1529 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:05:06.030 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:05 smithi023 bash[29146]: audit 2024-03-11T21:05:05.386831+0000 mon.a (mon.0) 1530 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:05:06.031 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:05 smithi023 bash[29146]: audit 2024-03-11T21:05:05.533400+0000 mgr.y (mgr.24380) 531 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:05:06.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:05 smithi023 bash[21372]: cluster 2024-03-11T21:05:04.320331+0000 osd.5 (osd.5) 267 : cluster 0 9.5 scrub ok 2024-03-11T21:05:06.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:05 smithi023 bash[21372]: cluster 2024-03-11T21:05:04.909239+0000 osd.1 (osd.1) 73 : cluster 0 3.19 scrub starts 2024-03-11T21:05:06.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:05 smithi023 bash[21372]: cluster 2024-03-11T21:05:04.911908+0000 osd.1 (osd.1) 74 : cluster 0 3.19 scrub ok 2024-03-11T21:05:06.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:05 smithi023 bash[21372]: cluster 2024-03-11T21:05:05.114912+0000 osd.3 (osd.3) 302 : cluster 0 9.c scrub starts 2024-03-11T21:05:06.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:05 smithi023 bash[21372]: audit 2024-03-11T21:05:05.385473+0000 mon.a (mon.0) 1529 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:05:06.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:05 smithi023 bash[21372]: audit 2024-03-11T21:05:05.386831+0000 mon.a (mon.0) 1530 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:05:06.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:05 smithi023 bash[21372]: audit 2024-03-11T21:05:05.533400+0000 mgr.y (mgr.24380) 531 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:05:06.048 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:06.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:06.050 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:05:06.151 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:06.160 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:05 smithi146 bash[23809]: cluster 2024-03-11T21:05:04.320331+0000 osd.5 (osd.5) 267 : cluster 0 9.5 scrub ok 2024-03-11T21:05:06.160 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:05 smithi146 bash[23809]: cluster 2024-03-11T21:05:04.909239+0000 osd.1 (osd.1) 73 : cluster 0 3.19 scrub starts 2024-03-11T21:05:06.161 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:05 smithi146 bash[23809]: cluster 2024-03-11T21:05:04.911908+0000 osd.1 (osd.1) 74 : cluster 0 3.19 scrub ok 2024-03-11T21:05:06.161 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:05 smithi146 bash[23809]: cluster 2024-03-11T21:05:05.114912+0000 osd.3 (osd.3) 302 : cluster 0 9.c scrub starts 2024-03-11T21:05:06.161 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:05 smithi146 bash[23809]: audit 2024-03-11T21:05:05.385473+0000 mon.a (mon.0) 1529 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:05:06.161 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:05 smithi146 bash[23809]: audit 2024-03-11T21:05:05.386831+0000 mon.a (mon.0) 1530 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:05:06.161 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:05 smithi146 bash[23809]: audit 2024-03-11T21:05:05.533400+0000 mgr.y (mgr.24380) 531 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:05:06.187 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:06.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:06.190 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:05:06.292 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:05:06.336 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:05:06.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:06.342 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:05:06.443 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:05:06.463 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:05:06.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:06.467 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:05:06.568 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:05:06.590 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:05:06.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:06.593 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:05:06.694 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:05:06.717 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:05:06.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:06.719 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:05:06.820 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:06.848 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:06.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:06.851 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:05:06.951 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:06.960 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:06 smithi146 bash[23809]: cluster 2024-03-11T21:05:05.651229+0000 osd.7 (osd.7) 311 : cluster 0 6.16 scrub starts 2024-03-11T21:05:06.960 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:06 smithi146 bash[23809]: cluster 2024-03-11T21:05:05.664032+0000 osd.7 (osd.7) 312 : cluster 0 6.16 scrub ok 2024-03-11T21:05:06.960 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:06 smithi146 bash[23809]: cluster 2024-03-11T21:05:05.812068+0000 osd.5 (osd.5) 268 : cluster 0 9.a scrub starts 2024-03-11T21:05:06.960 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:06 smithi146 bash[23809]: cluster 2024-03-11T21:05:05.844457+0000 osd.2 (osd.2) 201 : cluster 0 2.0 scrub starts 2024-03-11T21:05:06.960 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:06 smithi146 bash[23809]: cluster 2024-03-11T21:05:05.850065+0000 osd.2 (osd.2) 202 : cluster 0 2.0 scrub ok 2024-03-11T21:05:06.960 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:06 smithi146 bash[23809]: cluster 2024-03-11T21:05:05.985603+0000 osd.1 (osd.1) 75 : cluster 0 6.18 scrub starts 2024-03-11T21:05:06.960 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:06 smithi146 bash[23809]: cluster 2024-03-11T21:05:06.011613+0000 osd.1 (osd.1) 76 : cluster 0 6.18 scrub ok 2024-03-11T21:05:06.961 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:06 smithi146 bash[23809]: cluster 2024-03-11T21:05:06.118267+0000 osd.3 (osd.3) 303 : cluster 0 4.14 scrub starts 2024-03-11T21:05:06.961 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:06 smithi146 bash[23809]: cluster 2024-03-11T21:05:06.153730+0000 osd.3 (osd.3) 304 : cluster 0 4.14 scrub ok 2024-03-11T21:05:06.961 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:06 smithi146 bash[23809]: cluster 2024-03-11T21:05:06.193459+0000 mgr.y (mgr.24380) 532 : cluster 0 pgmap v592: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.05k op/s 2024-03-11T21:05:06.973 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:06.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:06.975 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:05:07.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:06 smithi023 bash[21372]: cluster 2024-03-11T21:05:05.651229+0000 osd.7 (osd.7) 311 : cluster 0 6.16 scrub starts 2024-03-11T21:05:07.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:06 smithi023 bash[21372]: cluster 2024-03-11T21:05:05.664032+0000 osd.7 (osd.7) 312 : cluster 0 6.16 scrub ok 2024-03-11T21:05:07.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:06 smithi023 bash[21372]: cluster 2024-03-11T21:05:05.812068+0000 osd.5 (osd.5) 268 : cluster 0 9.a scrub starts 2024-03-11T21:05:07.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:06 smithi023 bash[21372]: cluster 2024-03-11T21:05:05.844457+0000 osd.2 (osd.2) 201 : cluster 0 2.0 scrub starts 2024-03-11T21:05:07.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:06 smithi023 bash[21372]: cluster 2024-03-11T21:05:05.850065+0000 osd.2 (osd.2) 202 : cluster 0 2.0 scrub ok 2024-03-11T21:05:07.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:06 smithi023 bash[21372]: cluster 2024-03-11T21:05:05.985603+0000 osd.1 (osd.1) 75 : cluster 0 6.18 scrub starts 2024-03-11T21:05:07.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:06 smithi023 bash[21372]: cluster 2024-03-11T21:05:06.011613+0000 osd.1 (osd.1) 76 : cluster 0 6.18 scrub ok 2024-03-11T21:05:07.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:06 smithi023 bash[21372]: cluster 2024-03-11T21:05:06.118267+0000 osd.3 (osd.3) 303 : cluster 0 4.14 scrub starts 2024-03-11T21:05:07.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:06 smithi023 bash[21372]: cluster 2024-03-11T21:05:06.153730+0000 osd.3 (osd.3) 304 : cluster 0 4.14 scrub ok 2024-03-11T21:05:07.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:06 smithi023 bash[21372]: cluster 2024-03-11T21:05:06.193459+0000 mgr.y (mgr.24380) 532 : cluster 0 pgmap v592: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.05k op/s 2024-03-11T21:05:07.033 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:06 smithi023 bash[29146]: cluster 2024-03-11T21:05:05.651229+0000 osd.7 (osd.7) 311 : cluster 0 6.16 scrub starts 2024-03-11T21:05:07.033 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:06 smithi023 bash[29146]: cluster 2024-03-11T21:05:05.664032+0000 osd.7 (osd.7) 312 : cluster 0 6.16 scrub ok 2024-03-11T21:05:07.033 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:06 smithi023 bash[29146]: cluster 2024-03-11T21:05:05.812068+0000 osd.5 (osd.5) 268 : cluster 0 9.a scrub starts 2024-03-11T21:05:07.033 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:06 smithi023 bash[29146]: cluster 2024-03-11T21:05:05.844457+0000 osd.2 (osd.2) 201 : cluster 0 2.0 scrub starts 2024-03-11T21:05:07.033 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:06 smithi023 bash[29146]: cluster 2024-03-11T21:05:05.850065+0000 osd.2 (osd.2) 202 : cluster 0 2.0 scrub ok 2024-03-11T21:05:07.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:06 smithi023 bash[29146]: cluster 2024-03-11T21:05:05.985603+0000 osd.1 (osd.1) 75 : cluster 0 6.18 scrub starts 2024-03-11T21:05:07.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:06 smithi023 bash[29146]: cluster 2024-03-11T21:05:06.011613+0000 osd.1 (osd.1) 76 : cluster 0 6.18 scrub ok 2024-03-11T21:05:07.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:06 smithi023 bash[29146]: cluster 2024-03-11T21:05:06.118267+0000 osd.3 (osd.3) 303 : cluster 0 4.14 scrub starts 2024-03-11T21:05:07.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:06 smithi023 bash[29146]: cluster 2024-03-11T21:05:06.153730+0000 osd.3 (osd.3) 304 : cluster 0 4.14 scrub ok 2024-03-11T21:05:07.034 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:06 smithi023 bash[29146]: cluster 2024-03-11T21:05:06.193459+0000 mgr.y (mgr.24380) 532 : cluster 0 pgmap v592: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.05k op/s 2024-03-11T21:05:07.076 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:05:07.101 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:05:07.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:07.105 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:05:07.206 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:05:07.225 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:05:07.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:07.229 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:05:07.330 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:05:07.351 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:05:07.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:07.353 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:05:07.454 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:07.480 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:07.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:07.481 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:05:07.582 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:07.640 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:07.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:07.643 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:05:07.744 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:07.771 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:07.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:07.775 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:05:07.875 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:05:07.899 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:05:07.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:07.901 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:05:08.002 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:05:08.014 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:07 smithi146 bash[23809]: cluster 2024-03-11T21:05:06.624050+0000 osd.7 (osd.7) 313 : cluster 0 3.e scrub starts 2024-03-11T21:05:08.014 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:07 smithi146 bash[23809]: cluster 2024-03-11T21:05:06.684888+0000 osd.7 (osd.7) 314 : cluster 0 3.e scrub ok 2024-03-11T21:05:08.014 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:07 smithi146 bash[23809]: cluster 2024-03-11T21:05:07.054022+0000 osd.3 (osd.3) 305 : cluster 0 9.c scrub ok 2024-03-11T21:05:08.014 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:07 smithi146 bash[23809]: cluster 2024-03-11T21:05:07.515738+0000 osd.0 (osd.0) 207 : cluster 0 3.6 scrub starts 2024-03-11T21:05:08.027 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:05:08.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:08.031 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:05:08.132 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:08.157 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:08.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:08.161 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:05:08.262 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:08.286 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:08.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:08.288 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:05:08.294 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:07 smithi023 bash[21372]: cluster 2024-03-11T21:05:06.624050+0000 osd.7 (osd.7) 313 : cluster 0 3.e scrub starts 2024-03-11T21:05:08.294 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:07 smithi023 bash[21372]: cluster 2024-03-11T21:05:06.684888+0000 osd.7 (osd.7) 314 : cluster 0 3.e scrub ok 2024-03-11T21:05:08.295 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:07 smithi023 bash[21372]: cluster 2024-03-11T21:05:07.054022+0000 osd.3 (osd.3) 305 : cluster 0 9.c scrub ok 2024-03-11T21:05:08.295 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:07 smithi023 bash[21372]: cluster 2024-03-11T21:05:07.515738+0000 osd.0 (osd.0) 207 : cluster 0 3.6 scrub starts 2024-03-11T21:05:08.295 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:07 smithi023 bash[29146]: cluster 2024-03-11T21:05:06.624050+0000 osd.7 (osd.7) 313 : cluster 0 3.e scrub starts 2024-03-11T21:05:08.295 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:07 smithi023 bash[29146]: cluster 2024-03-11T21:05:06.684888+0000 osd.7 (osd.7) 314 : cluster 0 3.e scrub ok 2024-03-11T21:05:08.295 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:07 smithi023 bash[29146]: cluster 2024-03-11T21:05:07.054022+0000 osd.3 (osd.3) 305 : cluster 0 9.c scrub ok 2024-03-11T21:05:08.295 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:07 smithi023 bash[29146]: cluster 2024-03-11T21:05:07.515738+0000 osd.0 (osd.0) 207 : cluster 0 3.6 scrub starts 2024-03-11T21:05:08.389 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:05:08.411 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:05:08.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:08.415 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:05:08.516 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:08.541 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:08.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:08.547 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:05:08.648 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:08.674 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:08.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:08.680 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:05:08.780 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:05:08.782 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:05:08 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:05:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:05:08.805 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:05:08.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:08.808 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:05:08.908 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:05:08.935 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:05:08.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:08.943 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:05:09.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:08 smithi146 bash[23809]: cluster 2024-03-11T21:05:07.228012+0000 osd.5 (osd.5) 269 : cluster 0 9.a scrub ok 2024-03-11T21:05:09.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:08 smithi146 bash[23809]: cluster 2024-03-11T21:05:07.546484+0000 osd.0 (osd.0) 208 : cluster 0 3.6 scrub ok 2024-03-11T21:05:09.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:08 smithi146 bash[23809]: cluster 2024-03-11T21:05:07.595566+0000 osd.7 (osd.7) 315 : cluster 0 6.3 scrub starts 2024-03-11T21:05:09.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:08 smithi146 bash[23809]: cluster 2024-03-11T21:05:07.664275+0000 osd.7 (osd.7) 316 : cluster 0 6.3 scrub ok 2024-03-11T21:05:09.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:08 smithi146 bash[23809]: cluster 2024-03-11T21:05:08.019550+0000 osd.1 (osd.1) 77 : cluster 0 3.0 scrub starts 2024-03-11T21:05:09.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:08 smithi146 bash[23809]: cluster 2024-03-11T21:05:08.024664+0000 osd.1 (osd.1) 78 : cluster 0 3.0 scrub ok 2024-03-11T21:05:09.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:08 smithi146 bash[23809]: cluster 2024-03-11T21:05:08.194120+0000 mgr.y (mgr.24380) 533 : cluster 0 pgmap v593: 148 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 146 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 948 op/s 2024-03-11T21:05:09.044 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:09.065 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:09.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:09.067 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:05:09.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:08 smithi023 bash[21372]: cluster 2024-03-11T21:05:07.228012+0000 osd.5 (osd.5) 269 : cluster 0 9.a scrub ok 2024-03-11T21:05:09.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:08 smithi023 bash[21372]: cluster 2024-03-11T21:05:07.546484+0000 osd.0 (osd.0) 208 : cluster 0 3.6 scrub ok 2024-03-11T21:05:09.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:08 smithi023 bash[21372]: cluster 2024-03-11T21:05:07.595566+0000 osd.7 (osd.7) 315 : cluster 0 6.3 scrub starts 2024-03-11T21:05:09.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:08 smithi023 bash[21372]: cluster 2024-03-11T21:05:07.664275+0000 osd.7 (osd.7) 316 : cluster 0 6.3 scrub ok 2024-03-11T21:05:09.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:08 smithi023 bash[21372]: cluster 2024-03-11T21:05:08.019550+0000 osd.1 (osd.1) 77 : cluster 0 3.0 scrub starts 2024-03-11T21:05:09.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:08 smithi023 bash[21372]: cluster 2024-03-11T21:05:08.024664+0000 osd.1 (osd.1) 78 : cluster 0 3.0 scrub ok 2024-03-11T21:05:09.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:08 smithi023 bash[21372]: cluster 2024-03-11T21:05:08.194120+0000 mgr.y (mgr.24380) 533 : cluster 0 pgmap v593: 148 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 146 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 948 op/s 2024-03-11T21:05:09.091 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:08 smithi023 bash[29146]: cluster 2024-03-11T21:05:07.228012+0000 osd.5 (osd.5) 269 : cluster 0 9.a scrub ok 2024-03-11T21:05:09.091 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:08 smithi023 bash[29146]: cluster 2024-03-11T21:05:07.546484+0000 osd.0 (osd.0) 208 : cluster 0 3.6 scrub ok 2024-03-11T21:05:09.092 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:08 smithi023 bash[29146]: cluster 2024-03-11T21:05:07.595566+0000 osd.7 (osd.7) 315 : cluster 0 6.3 scrub starts 2024-03-11T21:05:09.092 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:08 smithi023 bash[29146]: cluster 2024-03-11T21:05:07.664275+0000 osd.7 (osd.7) 316 : cluster 0 6.3 scrub ok 2024-03-11T21:05:09.092 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:08 smithi023 bash[29146]: cluster 2024-03-11T21:05:08.019550+0000 osd.1 (osd.1) 77 : cluster 0 3.0 scrub starts 2024-03-11T21:05:09.092 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:08 smithi023 bash[29146]: cluster 2024-03-11T21:05:08.024664+0000 osd.1 (osd.1) 78 : cluster 0 3.0 scrub ok 2024-03-11T21:05:09.092 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:08 smithi023 bash[29146]: cluster 2024-03-11T21:05:08.194120+0000 mgr.y (mgr.24380) 533 : cluster 0 pgmap v593: 148 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 146 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 948 op/s 2024-03-11T21:05:09.168 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:09.191 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:09.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:09.193 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:05:09.294 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:05:09.313 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:05:09.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:09.315 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:05:09.416 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:05:09.433 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:05:09.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:09.435 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:05:09.536 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:09.563 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:09.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:09.565 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:05:09.666 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:09.694 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:09.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:09.696 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:05:09.797 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:09.821 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:09.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:09.827 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:05:09.928 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:09.952 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:09.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:09.958 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:05:10.059 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:05:10.069 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:09 smithi023 bash[21372]: cluster 2024-03-11T21:05:07.858531+0000 osd.5 (osd.5) 270 : cluster 0 6.a scrub starts 2024-03-11T21:05:10.069 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:09 smithi023 bash[21372]: cluster 2024-03-11T21:05:07.878150+0000 osd.5 (osd.5) 271 : cluster 0 6.a scrub ok 2024-03-11T21:05:10.069 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:09 smithi023 bash[21372]: cluster 2024-03-11T21:05:08.865936+0000 osd.5 (osd.5) 272 : cluster 0 2.18 deep-scrub starts 2024-03-11T21:05:10.070 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:09 smithi023 bash[21372]: cluster 2024-03-11T21:05:08.869145+0000 osd.5 (osd.5) 273 : cluster 0 2.18 deep-scrub ok 2024-03-11T21:05:10.070 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:09 smithi023 bash[21372]: cluster 2024-03-11T21:05:08.998541+0000 osd.1 (osd.1) 79 : cluster 0 2.9 scrub starts 2024-03-11T21:05:10.070 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:09 smithi023 bash[21372]: cluster 2024-03-11T21:05:09.001179+0000 osd.1 (osd.1) 80 : cluster 0 2.9 scrub ok 2024-03-11T21:05:10.070 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:09 smithi023 bash[21372]: cluster 2024-03-11T21:05:09.116840+0000 osd.0 (osd.0) 209 : cluster 0 4.b scrub starts 2024-03-11T21:05:10.070 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:09 smithi023 bash[21372]: cluster 2024-03-11T21:05:09.117957+0000 osd.0 (osd.0) 210 : cluster 0 4.b scrub ok 2024-03-11T21:05:10.070 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:09 smithi023 bash[21372]: cluster 2024-03-11T21:05:09.216588+0000 osd.3 (osd.3) 306 : cluster 0 3.18 scrub starts 2024-03-11T21:05:10.070 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:09 smithi023 bash[21372]: cluster 2024-03-11T21:05:09.233422+0000 osd.3 (osd.3) 307 : cluster 0 3.18 scrub ok 2024-03-11T21:05:10.070 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:09 smithi023 bash[29146]: cluster 2024-03-11T21:05:07.858531+0000 osd.5 (osd.5) 270 : cluster 0 6.a scrub starts 2024-03-11T21:05:10.070 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:09 smithi023 bash[29146]: cluster 2024-03-11T21:05:07.878150+0000 osd.5 (osd.5) 271 : cluster 0 6.a scrub ok 2024-03-11T21:05:10.070 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:09 smithi023 bash[29146]: cluster 2024-03-11T21:05:08.865936+0000 osd.5 (osd.5) 272 : cluster 0 2.18 deep-scrub starts 2024-03-11T21:05:10.070 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:09 smithi023 bash[29146]: cluster 2024-03-11T21:05:08.869145+0000 osd.5 (osd.5) 273 : cluster 0 2.18 deep-scrub ok 2024-03-11T21:05:10.071 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:09 smithi023 bash[29146]: cluster 2024-03-11T21:05:08.998541+0000 osd.1 (osd.1) 79 : cluster 0 2.9 scrub starts 2024-03-11T21:05:10.071 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:09 smithi023 bash[29146]: cluster 2024-03-11T21:05:09.001179+0000 osd.1 (osd.1) 80 : cluster 0 2.9 scrub ok 2024-03-11T21:05:10.071 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:09 smithi023 bash[29146]: cluster 2024-03-11T21:05:09.116840+0000 osd.0 (osd.0) 209 : cluster 0 4.b scrub starts 2024-03-11T21:05:10.071 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:09 smithi023 bash[29146]: cluster 2024-03-11T21:05:09.117957+0000 osd.0 (osd.0) 210 : cluster 0 4.b scrub ok 2024-03-11T21:05:10.071 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:09 smithi023 bash[29146]: cluster 2024-03-11T21:05:09.216588+0000 osd.3 (osd.3) 306 : cluster 0 3.18 scrub starts 2024-03-11T21:05:10.071 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:09 smithi023 bash[29146]: cluster 2024-03-11T21:05:09.233422+0000 osd.3 (osd.3) 307 : cluster 0 3.18 scrub ok 2024-03-11T21:05:10.111 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:05:10.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:10.113 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:05:10.214 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:05:10.228 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:09 smithi146 bash[23809]: cluster 2024-03-11T21:05:07.858531+0000 osd.5 (osd.5) 270 : cluster 0 6.a scrub starts 2024-03-11T21:05:10.228 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:09 smithi146 bash[23809]: cluster 2024-03-11T21:05:07.878150+0000 osd.5 (osd.5) 271 : cluster 0 6.a scrub ok 2024-03-11T21:05:10.228 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:09 smithi146 bash[23809]: cluster 2024-03-11T21:05:08.865936+0000 osd.5 (osd.5) 272 : cluster 0 2.18 deep-scrub starts 2024-03-11T21:05:10.228 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:09 smithi146 bash[23809]: cluster 2024-03-11T21:05:08.869145+0000 osd.5 (osd.5) 273 : cluster 0 2.18 deep-scrub ok 2024-03-11T21:05:10.228 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:09 smithi146 bash[23809]: cluster 2024-03-11T21:05:08.998541+0000 osd.1 (osd.1) 79 : cluster 0 2.9 scrub starts 2024-03-11T21:05:10.228 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:09 smithi146 bash[23809]: cluster 2024-03-11T21:05:09.001179+0000 osd.1 (osd.1) 80 : cluster 0 2.9 scrub ok 2024-03-11T21:05:10.228 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:09 smithi146 bash[23809]: cluster 2024-03-11T21:05:09.116840+0000 osd.0 (osd.0) 209 : cluster 0 4.b scrub starts 2024-03-11T21:05:10.228 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:09 smithi146 bash[23809]: cluster 2024-03-11T21:05:09.117957+0000 osd.0 (osd.0) 210 : cluster 0 4.b scrub ok 2024-03-11T21:05:10.228 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:09 smithi146 bash[23809]: cluster 2024-03-11T21:05:09.216588+0000 osd.3 (osd.3) 306 : cluster 0 3.18 scrub starts 2024-03-11T21:05:10.229 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:09 smithi146 bash[23809]: cluster 2024-03-11T21:05:09.233422+0000 osd.3 (osd.3) 307 : cluster 0 3.18 scrub ok 2024-03-11T21:05:10.251 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:05:10.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:10.256 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:05:10.356 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:05:10.381 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:05:10.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:10.383 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:05:10.484 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:05:10.506 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:05:10.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:10.508 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:05:10.609 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:10.647 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:10.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:10.649 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:05:10.750 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:05:10.784 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:05:10.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:10.788 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:05:10.889 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:05:10.921 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:05:10.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:10.926 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:05:11.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:10 smithi146 bash[23809]: cluster 2024-03-11T21:05:09.621073+0000 osd.7 (osd.7) 317 : cluster 0 3.11 scrub starts 2024-03-11T21:05:11.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:10 smithi146 bash[23809]: cluster 2024-03-11T21:05:09.637353+0000 osd.7 (osd.7) 318 : cluster 0 3.11 scrub ok 2024-03-11T21:05:11.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:10 smithi146 bash[23809]: cluster 2024-03-11T21:05:09.870330+0000 osd.4 (osd.4) 101 : cluster 0 4.e scrub starts 2024-03-11T21:05:11.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:10 smithi146 bash[23809]: cluster 2024-03-11T21:05:09.888028+0000 osd.5 (osd.5) 274 : cluster 0 6.7 scrub starts 2024-03-11T21:05:11.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:10 smithi146 bash[23809]: cluster 2024-03-11T21:05:09.913675+0000 osd.4 (osd.4) 102 : cluster 0 4.e scrub ok 2024-03-11T21:05:11.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:10 smithi146 bash[23809]: cluster 2024-03-11T21:05:09.952408+0000 osd.5 (osd.5) 275 : cluster 0 6.7 scrub ok 2024-03-11T21:05:11.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:10 smithi146 bash[23809]: cluster 2024-03-11T21:05:09.967625+0000 osd.1 (osd.1) 81 : cluster 0 4.f scrub starts 2024-03-11T21:05:11.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:10 smithi146 bash[23809]: cluster 2024-03-11T21:05:09.968767+0000 osd.1 (osd.1) 82 : cluster 0 4.f scrub ok 2024-03-11T21:05:11.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:10 smithi146 bash[23809]: cluster 2024-03-11T21:05:10.194569+0000 mgr.y (mgr.24380) 534 : cluster 0 pgmap v594: 148 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 146 active+clean; 4.8 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 814 op/s 2024-03-11T21:05:11.027 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:05:11.060 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:05:11.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:11.063 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:05:11.164 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:05:11.175 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:10 smithi023 bash[21372]: cluster 2024-03-11T21:05:09.621073+0000 osd.7 (osd.7) 317 : cluster 0 3.11 scrub starts 2024-03-11T21:05:11.176 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:10 smithi023 bash[21372]: cluster 2024-03-11T21:05:09.637353+0000 osd.7 (osd.7) 318 : cluster 0 3.11 scrub ok 2024-03-11T21:05:11.176 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:10 smithi023 bash[21372]: cluster 2024-03-11T21:05:09.870330+0000 osd.4 (osd.4) 101 : cluster 0 4.e scrub starts 2024-03-11T21:05:11.176 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:10 smithi023 bash[21372]: cluster 2024-03-11T21:05:09.888028+0000 osd.5 (osd.5) 274 : cluster 0 6.7 scrub starts 2024-03-11T21:05:11.176 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:10 smithi023 bash[21372]: cluster 2024-03-11T21:05:09.913675+0000 osd.4 (osd.4) 102 : cluster 0 4.e scrub ok 2024-03-11T21:05:11.176 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:10 smithi023 bash[21372]: cluster 2024-03-11T21:05:09.952408+0000 osd.5 (osd.5) 275 : cluster 0 6.7 scrub ok 2024-03-11T21:05:11.176 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:10 smithi023 bash[21372]: cluster 2024-03-11T21:05:09.967625+0000 osd.1 (osd.1) 81 : cluster 0 4.f scrub starts 2024-03-11T21:05:11.176 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:10 smithi023 bash[21372]: cluster 2024-03-11T21:05:09.968767+0000 osd.1 (osd.1) 82 : cluster 0 4.f scrub ok 2024-03-11T21:05:11.176 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:10 smithi023 bash[21372]: cluster 2024-03-11T21:05:10.194569+0000 mgr.y (mgr.24380) 534 : cluster 0 pgmap v594: 148 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 146 active+clean; 4.8 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 814 op/s 2024-03-11T21:05:11.177 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:10 smithi023 bash[29146]: cluster 2024-03-11T21:05:09.621073+0000 osd.7 (osd.7) 317 : cluster 0 3.11 scrub starts 2024-03-11T21:05:11.177 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:10 smithi023 bash[29146]: cluster 2024-03-11T21:05:09.637353+0000 osd.7 (osd.7) 318 : cluster 0 3.11 scrub ok 2024-03-11T21:05:11.177 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:10 smithi023 bash[29146]: cluster 2024-03-11T21:05:09.870330+0000 osd.4 (osd.4) 101 : cluster 0 4.e scrub starts 2024-03-11T21:05:11.177 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:10 smithi023 bash[29146]: cluster 2024-03-11T21:05:09.888028+0000 osd.5 (osd.5) 274 : cluster 0 6.7 scrub starts 2024-03-11T21:05:11.177 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:10 smithi023 bash[29146]: cluster 2024-03-11T21:05:09.913675+0000 osd.4 (osd.4) 102 : cluster 0 4.e scrub ok 2024-03-11T21:05:11.177 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:10 smithi023 bash[29146]: cluster 2024-03-11T21:05:09.952408+0000 osd.5 (osd.5) 275 : cluster 0 6.7 scrub ok 2024-03-11T21:05:11.177 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:10 smithi023 bash[29146]: cluster 2024-03-11T21:05:09.967625+0000 osd.1 (osd.1) 81 : cluster 0 4.f scrub starts 2024-03-11T21:05:11.177 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:10 smithi023 bash[29146]: cluster 2024-03-11T21:05:09.968767+0000 osd.1 (osd.1) 82 : cluster 0 4.f scrub ok 2024-03-11T21:05:11.177 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:10 smithi023 bash[29146]: cluster 2024-03-11T21:05:10.194569+0000 mgr.y (mgr.24380) 534 : cluster 0 pgmap v594: 148 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 146 active+clean; 4.8 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 814 op/s 2024-03-11T21:05:11.200 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:05:11.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:11.205 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:05:11.307 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:11.335 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:11.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:11.340 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:05:11.440 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:05:11.461 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:05:11.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:11.463 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:05:11.564 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:05:11.588 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:05:11.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:11.590 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:05:11.691 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:05:11.718 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:05:11.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:11.720 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:05:11.821 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:05:11.844 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:05:11.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:11.846 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:05:11.947 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:11.956 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:11 smithi023 bash[21372]: cluster 2024-03-11T21:05:10.919124+0000 osd.5 (osd.5) 276 : cluster 0 3.5 scrub starts 2024-03-11T21:05:11.957 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:11 smithi023 bash[21372]: cluster 2024-03-11T21:05:10.941961+0000 osd.5 (osd.5) 277 : cluster 0 3.5 scrub ok 2024-03-11T21:05:11.957 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:11 smithi023 bash[29146]: cluster 2024-03-11T21:05:10.919124+0000 osd.5 (osd.5) 276 : cluster 0 3.5 scrub starts 2024-03-11T21:05:11.957 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:11 smithi023 bash[29146]: cluster 2024-03-11T21:05:10.941961+0000 osd.5 (osd.5) 277 : cluster 0 3.5 scrub ok 2024-03-11T21:05:11.980 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:11.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:11.983 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:05:12.084 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:05:12.107 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:05:12.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:12.114 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:05:12.215 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:12.225 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:11 smithi146 bash[23809]: cluster 2024-03-11T21:05:10.919124+0000 osd.5 (osd.5) 276 : cluster 0 3.5 scrub starts 2024-03-11T21:05:12.225 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:11 smithi146 bash[23809]: cluster 2024-03-11T21:05:10.941961+0000 osd.5 (osd.5) 277 : cluster 0 3.5 scrub ok 2024-03-11T21:05:12.241 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:12.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:12.245 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:05:12.255 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-03-11T21:05:12.255 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-03-11T21:05:12.255 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-03-11T21:05:12.255 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-03-11T21:05:12.255 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.255 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.24281.0:2123 4.d 4:b4812045:::notify.4:head [watch reconnect cookie 94110942031104 gen 7] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-03-11T21:05:12.255 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T21:00:26.952063+0000", 2024-03-11T21:05:12.255 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 285.30677995799999, 2024-03-11T21:05:12.256 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0130510230000001, 2024-03-11T21:05:12.256 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:05:12.256 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-03-11T21:05:12.256 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-03-11T21:05:12.256 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.24281", 2024-03-11T21:05:12.256 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.23:0/2185274774", 2024-03-11T21:05:12.256 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2123 2024-03-11T21:05:12.256 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.256 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:05:12.256 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.256 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:05:12.256 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:00:26.952063+0000", 2024-03-11T21:05:12.256 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:05:12.256 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.257 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.257 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:05:12.257 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:00:26.952063+0000", 2024-03-11T21:05:12.257 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:05:12.257 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.257 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.257 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:05:12.257 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:00:26.952065+0000", 2024-03-11T21:05:12.257 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.216e-06 2024-03-11T21:05:12.257 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.257 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.257 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:05:12.257 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:00:26.952074+0000", 2024-03-11T21:05:12.257 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4460000000000001e-06 2024-03-11T21:05:12.258 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.258 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.258 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:05:12.258 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:00:26.952125+0000", 2024-03-11T21:05:12.258 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1100999999999997e-05 2024-03-11T21:05:12.258 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.258 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.258 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:05:12.258 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:00:26.952145+0000", 2024-03-11T21:05:12.258 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0072999999999999e-05 2024-03-11T21:05:12.258 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.258 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.258 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:05:12.259 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:00:26.973658+0000", 2024-03-11T21:05:12.259 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.021513365 2024-03-11T21:05:12.259 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.259 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.259 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-03-11T21:05:12.259 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:00:26.973666+0000", 2024-03-11T21:05:12.259 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2459999999999997e-06 2024-03-11T21:05:12.259 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.259 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.259 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:05:12.259 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:00:27.951840+0000", 2024-03-11T21:05:12.259 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.97817403000000003 2024-03-11T21:05:12.259 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.260 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.260 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:05:12.260 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:00:27.952903+0000", 2024-03-11T21:05:12.260 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001063479 2024-03-11T21:05:12.260 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.260 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.260 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:05:12.260 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:00:27.965114+0000", 2024-03-11T21:05:12.260 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012211067000000001 2024-03-11T21:05:12.260 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:05:12.260 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:05:12.260 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:05:12.260 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.261 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.261 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.24281.0:2124 4.9 4:93e5b521:::notify.7:head [watch reconnect cookie 94110942020736 gen 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-03-11T21:05:12.261 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T21:00:26.952169+0000", 2024-03-11T21:05:12.261 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 285.30667387599999, 2024-03-11T21:05:12.261 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0205736160000001, 2024-03-11T21:05:12.261 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:05:12.261 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-03-11T21:05:12.261 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-03-11T21:05:12.261 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.24281", 2024-03-11T21:05:12.261 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.23:0/2185274774", 2024-03-11T21:05:12.261 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2124 2024-03-11T21:05:12.261 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.261 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:05:12.262 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.262 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:05:12.262 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:00:26.952169+0000", 2024-03-11T21:05:12.262 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:05:12.262 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.262 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.262 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:05:12.262 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:00:26.952169+0000", 2024-03-11T21:05:12.262 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:05:12.262 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.262 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.262 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:05:12.262 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:00:26.952170+0000", 2024-03-11T21:05:12.262 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.128e-06 2024-03-11T21:05:12.263 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.263 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.263 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:05:12.263 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:00:26.952177+0000", 2024-03-11T21:05:12.263 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.866e-06 2024-03-11T21:05:12.263 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.263 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.263 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:05:12.263 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:00:26.952191+0000", 2024-03-11T21:05:12.263 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3855e-05 2024-03-11T21:05:12.264 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.264 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.264 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:05:12.264 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:00:26.952218+0000", 2024-03-11T21:05:12.264 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7304000000000002e-05 2024-03-11T21:05:12.264 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.264 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.265 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:05:12.265 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:00:26.973759+0000", 2024-03-11T21:05:12.265 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.021540423 2024-03-11T21:05:12.265 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.265 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.265 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-03-11T21:05:12.265 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:00:26.973774+0000", 2024-03-11T21:05:12.265 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5639000000000001e-05 2024-03-11T21:05:12.265 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.265 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.265 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:05:12.265 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:00:27.964841+0000", 2024-03-11T21:05:12.265 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.99106630900000003 2024-03-11T21:05:12.265 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.266 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.266 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:05:12.266 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:00:27.965046+0000", 2024-03-11T21:05:12.266 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00020581500000000001 2024-03-11T21:05:12.266 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.266 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.266 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:05:12.266 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:00:27.972743+0000", 2024-03-11T21:05:12.266 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0076962769999999996 2024-03-11T21:05:12.266 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:05:12.266 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:05:12.266 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:05:12.266 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.266 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.266 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.24281.0:2125 4.e 4:7759931f:::notify.3:head [watch reconnect cookie 94110942029952 gen 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-03-11T21:05:12.267 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T21:00:26.952234+0000", 2024-03-11T21:05:12.267 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 285.30660932400002, 2024-03-11T21:05:12.267 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.016309409, 2024-03-11T21:05:12.267 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:05:12.267 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-03-11T21:05:12.267 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-03-11T21:05:12.267 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.24281", 2024-03-11T21:05:12.267 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.23:0/2185274774", 2024-03-11T21:05:12.267 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2125 2024-03-11T21:05:12.267 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.267 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:05:12.267 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.267 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:05:12.267 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:00:26.952234+0000", 2024-03-11T21:05:12.267 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:05:12.268 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.268 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.268 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:05:12.268 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:00:26.952234+0000", 2024-03-11T21:05:12.268 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:05:12.268 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.268 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.268 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:05:12.268 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:00:26.952235+0000", 2024-03-11T21:05:12.268 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8400000000000003e-07 2024-03-11T21:05:12.268 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.268 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.268 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:05:12.268 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:00:26.952239+0000", 2024-03-11T21:05:12.268 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6319999999999999e-06 2024-03-11T21:05:12.269 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.269 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.269 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:05:12.269 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:00:26.952252+0000", 2024-03-11T21:05:12.269 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3183999999999999e-05 2024-03-11T21:05:12.269 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.269 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.269 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:05:12.269 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:00:26.952263+0000", 2024-03-11T21:05:12.269 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0566e-05 2024-03-11T21:05:12.269 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.269 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.269 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:05:12.269 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:00:26.971482+0000", 2024-03-11T21:05:12.269 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.019219127999999999 2024-03-11T21:05:12.269 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.270 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.270 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-03-11T21:05:12.270 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:00:26.971512+0000", 2024-03-11T21:05:12.270 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.027e-05 2024-03-11T21:05:12.270 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.270 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.270 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:05:12.270 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:00:27.956458+0000", 2024-03-11T21:05:12.270 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.98494606699999998 2024-03-11T21:05:12.270 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.270 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.270 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:05:12.270 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:00:27.956622+0000", 2024-03-11T21:05:12.271 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016309899999999999 2024-03-11T21:05:12.271 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.271 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.271 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:05:12.271 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:00:27.968543+0000", 2024-03-11T21:05:12.271 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.011921579 2024-03-11T21:05:12.271 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:05:12.271 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:05:12.271 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:05:12.271 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.271 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.271 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.14910.0:39320 8.2 8:4d140e99:::benchmark_data_smithi023_89854_object39319:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e209)", 2024-03-11T21:05:12.271 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T21:00:45.878479+0000", 2024-03-11T21:05:12.271 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 266.38036384200001, 2024-03-11T21:05:12.272 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.32651124199999998, 2024-03-11T21:05:12.272 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:05:12.272 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:05:12.272 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-03-11T21:05:12.272 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.14910", 2024-03-11T21:05:12.272 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.23:0/2457905657", 2024-03-11T21:05:12.272 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 39320 2024-03-11T21:05:12.272 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.272 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:05:12.272 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.272 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:05:12.272 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:00:45.878479+0000", 2024-03-11T21:05:12.272 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:05:12.272 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.273 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.273 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:05:12.273 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:00:45.878479+0000", 2024-03-11T21:05:12.273 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:05:12.273 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.273 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.273 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:05:12.273 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:00:45.878481+0000", 2024-03-11T21:05:12.273 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9719999999999999e-06 2024-03-11T21:05:12.273 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.273 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.274 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:05:12.274 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:00:45.878510+0000", 2024-03-11T21:05:12.274 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9132000000000002e-05 2024-03-11T21:05:12.274 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.274 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.274 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:05:12.274 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:00:45.878524+0000", 2024-03-11T21:05:12.274 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3627e-05 2024-03-11T21:05:12.274 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.274 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.274 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:05:12.274 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:00:45.878532+0000", 2024-03-11T21:05:12.274 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5210000000000002e-06 2024-03-11T21:05:12.274 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.275 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.275 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:05:12.275 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:00:45.878592+0000", 2024-03-11T21:05:12.275 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9939999999999999e-05 2024-03-11T21:05:12.275 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.275 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.275 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:05:12.275 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:00:45.878754+0000", 2024-03-11T21:05:12.275 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016116700000000001 2024-03-11T21:05:12.275 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.275 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.275 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0,5", 2024-03-11T21:05:12.275 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:00:45.878898+0000", 2024-03-11T21:05:12.275 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014414199999999999 2024-03-11T21:05:12.275 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.276 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.276 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-03-11T21:05:12.276 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:00:45.880414+0000", 2024-03-11T21:05:12.276 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001516461 2024-03-11T21:05:12.276 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.276 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.276 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:05:12.276 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:00:45.881172+0000", 2024-03-11T21:05:12.276 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00075784599999999995 2024-03-11T21:05:12.276 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.276 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.276 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:05:12.276 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:00:46.204710+0000", 2024-03-11T21:05:12.277 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.32353836800000002 2024-03-11T21:05:12.277 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.277 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.277 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:05:12.277 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:00:46.204865+0000", 2024-03-11T21:05:12.277 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000155057 2024-03-11T21:05:12.277 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.277 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.277 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:05:12.277 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:00:46.204990+0000", 2024-03-11T21:05:12.277 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012500900000000001 2024-03-11T21:05:12.277 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:05:12.277 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:05:12.277 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:05:12.277 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.277 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.278 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.14910.0:48620 8.2 8:42dff9c2:::benchmark_data_smithi023_89854_object48619:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-03-11T21:05:12.278 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T21:00:58.429887+0000", 2024-03-11T21:05:12.278 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 253.82895595599999, 2024-03-11T21:05:12.278 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.22759221299999999, 2024-03-11T21:05:12.278 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:05:12.278 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:05:12.278 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-03-11T21:05:12.278 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.14910", 2024-03-11T21:05:12.278 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.23:0/2457905657", 2024-03-11T21:05:12.278 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 48620 2024-03-11T21:05:12.278 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.278 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:05:12.278 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.278 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:05:12.279 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:00:58.429887+0000", 2024-03-11T21:05:12.279 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:05:12.279 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.279 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.279 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:05:12.279 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:00:58.429887+0000", 2024-03-11T21:05:12.279 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:05:12.279 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.279 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.279 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:05:12.279 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:00:58.429889+0000", 2024-03-11T21:05:12.279 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4410000000000001e-06 2024-03-11T21:05:12.279 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.279 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.279 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:05:12.279 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:00:58.429942+0000", 2024-03-11T21:05:12.280 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3854000000000003e-05 2024-03-11T21:05:12.280 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.280 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.280 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:05:12.280 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:00:58.429955+0000", 2024-03-11T21:05:12.280 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2422999999999999e-05 2024-03-11T21:05:12.280 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.280 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.280 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:05:12.280 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:00:58.429961+0000", 2024-03-11T21:05:12.280 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5359999999999998e-06 2024-03-11T21:05:12.280 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.280 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.280 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:05:12.280 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:00:58.430007+0000", 2024-03-11T21:05:12.281 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5581999999999998e-05 2024-03-11T21:05:12.281 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.281 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.281 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:05:12.281 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:00:58.430143+0000", 2024-03-11T21:05:12.281 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000136088 2024-03-11T21:05:12.281 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.281 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.281 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0,5", 2024-03-11T21:05:12.281 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:00:58.430249+0000", 2024-03-11T21:05:12.281 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010604500000000001 2024-03-11T21:05:12.281 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.281 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.281 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-03-11T21:05:12.281 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:00:58.431948+0000", 2024-03-11T21:05:12.281 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001698541 2024-03-11T21:05:12.282 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.282 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.282 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:05:12.282 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:00:58.432131+0000", 2024-03-11T21:05:12.282 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018376500000000001 2024-03-11T21:05:12.282 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.282 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.282 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:05:12.282 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:00:58.657193+0000", 2024-03-11T21:05:12.282 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.225061758 2024-03-11T21:05:12.282 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.282 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.282 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:05:12.282 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:00:58.657352+0000", 2024-03-11T21:05:12.282 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015921699999999999 2024-03-11T21:05:12.283 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.283 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.283 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:05:12.283 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:00:58.657479+0000", 2024-03-11T21:05:12.283 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012696300000000001 2024-03-11T21:05:12.283 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:05:12.283 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:05:12.283 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:05:12.283 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.283 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.283 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.14910.0:53233 8.2 8:4530d442:::benchmark_data_smithi023_89854_object3148:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e214)", 2024-03-11T21:05:12.283 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T21:01:05.645506+0000", 2024-03-11T21:05:12.283 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 246.613337003, 2024-03-11T21:05:12.283 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.212880494, 2024-03-11T21:05:12.284 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:05:12.284 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:05:12.284 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-03-11T21:05:12.284 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.14910", 2024-03-11T21:05:12.284 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.23:0/2457905657", 2024-03-11T21:05:12.284 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 53233 2024-03-11T21:05:12.284 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.284 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:05:12.284 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.284 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:05:12.284 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:05.645506+0000", 2024-03-11T21:05:12.284 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:05:12.284 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.284 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.284 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:05:12.285 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:05.645506+0000", 2024-03-11T21:05:12.285 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:05:12.285 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.285 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.285 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:05:12.285 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:05.645508+0000", 2024-03-11T21:05:12.285 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.883e-06 2024-03-11T21:05:12.285 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.285 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.285 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:05:12.285 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:05.645514+0000", 2024-03-11T21:05:12.285 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2190000000000001e-06 2024-03-11T21:05:12.285 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.285 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.285 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:05:12.286 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:05.645529+0000", 2024-03-11T21:05:12.286 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4598e-05 2024-03-11T21:05:12.286 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.286 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.286 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:05:12.286 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:05.645535+0000", 2024-03-11T21:05:12.286 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7139999999999996e-06 2024-03-11T21:05:12.286 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.286 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.286 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:05:12.286 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:05.645580+0000", 2024-03-11T21:05:12.286 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4635999999999998e-05 2024-03-11T21:05:12.286 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.286 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.286 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:05:12.287 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:05.645676+0000", 2024-03-11T21:05:12.289 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5526000000000007e-05 2024-03-11T21:05:12.290 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.290 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.290 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0,5", 2024-03-11T21:05:12.290 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:05.645751+0000", 2024-03-11T21:05:12.290 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5321000000000006e-05 2024-03-11T21:05:12.290 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.290 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.290 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-03-11T21:05:12.290 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:05.647326+0000", 2024-03-11T21:05:12.290 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001575119 2024-03-11T21:05:12.291 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.291 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.291 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:05:12.291 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:05.648025+0000", 2024-03-11T21:05:12.291 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00069896499999999996 2024-03-11T21:05:12.291 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.291 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.291 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:05:12.291 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:05.858276+0000", 2024-03-11T21:05:12.291 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.21025125 2024-03-11T21:05:12.291 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.291 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.291 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:05:12.291 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:05.858331+0000", 2024-03-11T21:05:12.291 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4685999999999998e-05 2024-03-11T21:05:12.292 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.292 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.292 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:05:12.292 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:05.858387+0000", 2024-03-11T21:05:12.292 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5577e-05 2024-03-11T21:05:12.292 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:05:12.292 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:05:12.292 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:05:12.292 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.292 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.292 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.14910.0:53235 8.2 8:4e742323:::benchmark_data_smithi023_89854_object3150:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e214)", 2024-03-11T21:05:12.292 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T21:01:05.648117+0000", 2024-03-11T21:05:12.292 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 246.610725952, 2024-03-11T21:05:12.293 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.21202364200000001, 2024-03-11T21:05:12.293 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:05:12.293 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:05:12.293 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-03-11T21:05:12.293 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.14910", 2024-03-11T21:05:12.293 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.23:0/2457905657", 2024-03-11T21:05:12.293 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 53235 2024-03-11T21:05:12.293 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.293 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:05:12.293 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.293 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:05:12.293 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:05.648117+0000", 2024-03-11T21:05:12.293 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:05:12.293 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.293 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.293 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:05:12.294 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:05.648117+0000", 2024-03-11T21:05:12.294 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:05:12.294 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.294 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.294 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:05:12.294 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:05.648118+0000", 2024-03-11T21:05:12.294 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8599999999999996e-07 2024-03-11T21:05:12.294 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.294 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.294 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:05:12.294 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:05.648125+0000", 2024-03-11T21:05:12.294 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5780000000000004e-06 2024-03-11T21:05:12.294 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.294 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.294 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:05:12.295 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:05.648137+0000", 2024-03-11T21:05:12.295 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2054e-05 2024-03-11T21:05:12.295 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.295 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.295 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:05:12.295 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:05.648143+0000", 2024-03-11T21:05:12.295 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4690000000000002e-06 2024-03-11T21:05:12.295 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.295 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.295 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:05:12.295 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:05.648183+0000", 2024-03-11T21:05:12.295 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9931000000000001e-05 2024-03-11T21:05:12.295 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.295 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.295 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:05:12.296 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:05.648256+0000", 2024-03-11T21:05:12.296 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2774000000000005e-05 2024-03-11T21:05:12.296 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.296 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.296 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0,5", 2024-03-11T21:05:12.296 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:05.648320+0000", 2024-03-11T21:05:12.296 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4567999999999996e-05 2024-03-11T21:05:12.296 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.296 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.296 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-03-11T21:05:12.296 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:05.649340+0000", 2024-03-11T21:05:12.296 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001019617 2024-03-11T21:05:12.296 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.297 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.297 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:05:12.297 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:05.649925+0000", 2024-03-11T21:05:12.297 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00058496500000000001 2024-03-11T21:05:12.297 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.297 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.297 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:05:12.297 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:05.860047+0000", 2024-03-11T21:05:12.297 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.210121998 2024-03-11T21:05:12.298 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.298 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.298 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:05:12.298 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:05.860085+0000", 2024-03-11T21:05:12.298 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7651999999999998e-05 2024-03-11T21:05:12.298 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.298 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.298 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:05:12.298 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:05.860141+0000", 2024-03-11T21:05:12.298 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.605e-05 2024-03-11T21:05:12.298 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:05:12.298 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:05:12.298 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:05:12.298 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.298 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.299 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.14910.0:70089 8.2 8:4075b7dd:::benchmark_data_smithi023_89854_object20004:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e219)", 2024-03-11T21:05:12.299 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T21:01:24.950419+0000", 2024-03-11T21:05:12.299 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 227.30842445900001, 2024-03-11T21:05:12.299 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.282496262, 2024-03-11T21:05:12.299 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:05:12.299 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:05:12.299 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-03-11T21:05:12.299 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.14910", 2024-03-11T21:05:12.299 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.23:0/2457905657", 2024-03-11T21:05:12.299 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 70089 2024-03-11T21:05:12.299 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.299 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:05:12.299 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.299 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:05:12.299 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:24.950419+0000", 2024-03-11T21:05:12.300 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:05:12.300 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.300 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.300 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:05:12.300 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:24.950419+0000", 2024-03-11T21:05:12.300 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:05:12.300 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.300 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.300 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:05:12.300 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:24.950420+0000", 2024-03-11T21:05:12.300 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4099999999999997e-07 2024-03-11T21:05:12.300 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.300 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.300 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:05:12.301 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:24.950424+0000", 2024-03-11T21:05:12.301 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9300000000000002e-06 2024-03-11T21:05:12.301 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.301 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.301 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:05:12.301 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:24.950433+0000", 2024-03-11T21:05:12.301 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8899999999999996e-06 2024-03-11T21:05:12.301 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.301 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.301 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:05:12.301 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:24.950441+0000", 2024-03-11T21:05:12.301 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6930000000000007e-06 2024-03-11T21:05:12.301 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.301 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.302 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:05:12.302 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:24.950495+0000", 2024-03-11T21:05:12.302 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3652000000000001e-05 2024-03-11T21:05:12.302 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.302 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.302 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:05:12.302 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:24.950594+0000", 2024-03-11T21:05:12.302 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.9628000000000002e-05 2024-03-11T21:05:12.302 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.302 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.302 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0,5", 2024-03-11T21:05:12.302 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:24.950684+0000", 2024-03-11T21:05:12.302 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.9209999999999995e-05 2024-03-11T21:05:12.302 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.303 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.303 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-03-11T21:05:12.303 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:24.952283+0000", 2024-03-11T21:05:12.303 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0015992999999999999 2024-03-11T21:05:12.303 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.303 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.303 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:05:12.303 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:24.952750+0000", 2024-03-11T21:05:12.303 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00046699099999999999 2024-03-11T21:05:12.303 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.303 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.303 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:05:12.303 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:25.232802+0000", 2024-03-11T21:05:12.303 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.28005227300000002 2024-03-11T21:05:12.303 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.303 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.304 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:05:12.304 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:25.232864+0000", 2024-03-11T21:05:12.304 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1719999999999999e-05 2024-03-11T21:05:12.304 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.304 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.304 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:05:12.304 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:25.232915+0000", 2024-03-11T21:05:12.304 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1034e-05 2024-03-11T21:05:12.304 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:05:12.304 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:05:12.304 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:05:12.304 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.304 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.305 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.14910.0:70090 8.2 8:4308c9af:::benchmark_data_smithi023_89854_object20005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e219)", 2024-03-11T21:05:12.305 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T21:01:24.952677+0000", 2024-03-11T21:05:12.305 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 227.306166142, 2024-03-11T21:05:12.305 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.28038873600000003, 2024-03-11T21:05:12.305 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:05:12.305 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:05:12.305 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-03-11T21:05:12.305 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.14910", 2024-03-11T21:05:12.305 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.23:0/2457905657", 2024-03-11T21:05:12.305 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 70090 2024-03-11T21:05:12.305 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.305 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:05:12.305 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.305 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:05:12.306 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:24.952677+0000", 2024-03-11T21:05:12.306 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:05:12.306 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.306 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.306 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:05:12.306 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:24.952677+0000", 2024-03-11T21:05:12.306 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:05:12.306 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.306 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.306 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:05:12.306 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:24.952678+0000", 2024-03-11T21:05:12.306 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4600000000000004e-07 2024-03-11T21:05:12.307 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.307 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.307 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:05:12.307 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:24.952682+0000", 2024-03-11T21:05:12.307 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4020000000000004e-06 2024-03-11T21:05:12.307 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.307 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.307 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:05:12.307 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:24.952692+0000", 2024-03-11T21:05:12.307 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0261000000000001e-05 2024-03-11T21:05:12.307 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.307 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.307 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:05:12.308 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:24.952700+0000", 2024-03-11T21:05:12.308 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.0970000000000006e-06 2024-03-11T21:05:12.308 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.308 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.308 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:05:12.308 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:24.952776+0000", 2024-03-11T21:05:12.308 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5432000000000006e-05 2024-03-11T21:05:12.308 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.308 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.308 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:05:12.308 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:24.952851+0000", 2024-03-11T21:05:12.308 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5327000000000006e-05 2024-03-11T21:05:12.308 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.308 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.309 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0,5", 2024-03-11T21:05:12.309 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:24.952916+0000", 2024-03-11T21:05:12.309 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4521000000000001e-05 2024-03-11T21:05:12.309 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.309 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.309 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-03-11T21:05:12.309 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:24.954029+0000", 2024-03-11T21:05:12.309 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0011136869999999999 2024-03-11T21:05:12.309 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.309 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.309 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:05:12.309 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:24.954639+0000", 2024-03-11T21:05:12.309 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00060926899999999996 2024-03-11T21:05:12.309 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.309 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.310 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:05:12.310 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:25.232980+0000", 2024-03-11T21:05:12.310 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.27834167500000001 2024-03-11T21:05:12.310 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.310 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.310 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:05:12.310 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:25.233028+0000", 2024-03-11T21:05:12.310 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7203999999999997e-05 2024-03-11T21:05:12.310 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.310 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.310 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:05:12.310 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:25.233066+0000", 2024-03-11T21:05:12.310 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8114999999999997e-05 2024-03-11T21:05:12.310 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:05:12.310 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:05:12.311 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:05:12.311 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.311 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.311 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.14910.0:70095 8.2 8:4ef661c3:::benchmark_data_smithi023_89854_object20010:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e219)", 2024-03-11T21:05:12.311 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T21:01:24.954954+0000", 2024-03-11T21:05:12.311 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 227.30388859499999, 2024-03-11T21:05:12.311 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.27823027299999997, 2024-03-11T21:05:12.311 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:05:12.311 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:05:12.311 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-03-11T21:05:12.311 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.14910", 2024-03-11T21:05:12.311 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.23:0/2457905657", 2024-03-11T21:05:12.311 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 70095 2024-03-11T21:05:12.311 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.311 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:05:12.312 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.312 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:05:12.312 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:24.954954+0000", 2024-03-11T21:05:12.312 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:05:12.312 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.312 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.312 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:05:12.312 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:24.954954+0000", 2024-03-11T21:05:12.312 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:05:12.312 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.312 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.312 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:05:12.312 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:24.954955+0000", 2024-03-11T21:05:12.312 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.61e-07 2024-03-11T21:05:12.312 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.312 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.313 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:05:12.313 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:24.954959+0000", 2024-03-11T21:05:12.313 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.6969999999999999e-06 2024-03-11T21:05:12.313 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.313 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.313 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:05:12.313 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:24.954968+0000", 2024-03-11T21:05:12.313 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5030000000000003e-06 2024-03-11T21:05:12.313 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.313 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.313 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:05:12.313 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:24.954974+0000", 2024-03-11T21:05:12.313 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4879999999999998e-06 2024-03-11T21:05:12.314 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.314 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.314 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:05:12.314 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:24.955023+0000", 2024-03-11T21:05:12.314 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9261999999999997e-05 2024-03-11T21:05:12.314 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.314 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.314 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:05:12.314 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:24.955148+0000", 2024-03-11T21:05:12.314 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012500900000000001 2024-03-11T21:05:12.314 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.314 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.314 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0,5", 2024-03-11T21:05:12.314 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:24.955270+0000", 2024-03-11T21:05:12.314 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000122204 2024-03-11T21:05:12.315 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.315 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.315 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-03-11T21:05:12.315 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:24.956860+0000", 2024-03-11T21:05:12.315 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0015894710000000001 2024-03-11T21:05:12.315 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.315 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.315 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:05:12.315 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:24.957041+0000", 2024-03-11T21:05:12.315 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000180676 2024-03-11T21:05:12.315 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.315 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.315 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:05:12.315 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:25.233104+0000", 2024-03-11T21:05:12.316 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.27606304999999998 2024-03-11T21:05:12.316 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.316 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.316 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:05:12.316 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:25.233149+0000", 2024-03-11T21:05:12.316 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5544999999999998e-05 2024-03-11T21:05:12.316 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.316 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.316 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:05:12.316 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:25.233185+0000", 2024-03-11T21:05:12.316 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5607e-05 2024-03-11T21:05:12.316 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:05:12.316 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:05:12.316 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:05:12.316 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.317 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.317 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.14910.0:70105 8.2 8:4a7e9a59:::benchmark_data_smithi023_89854_object20020:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e219)", 2024-03-11T21:05:12.317 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T21:01:24.972432+0000", 2024-03-11T21:05:12.317 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 227.28641092500001, 2024-03-11T21:05:12.317 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.26089491999999997, 2024-03-11T21:05:12.317 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:05:12.317 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:05:12.317 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-03-11T21:05:12.317 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.14910", 2024-03-11T21:05:12.317 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.23:0/2457905657", 2024-03-11T21:05:12.317 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 70105 2024-03-11T21:05:12.317 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.317 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:05:12.317 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.317 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:05:12.318 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:24.972432+0000", 2024-03-11T21:05:12.318 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:05:12.318 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.318 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.318 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:05:12.318 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:24.972432+0000", 2024-03-11T21:05:12.318 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:05:12.318 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.318 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.318 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:05:12.318 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:24.972434+0000", 2024-03-11T21:05:12.318 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4440000000000001e-06 2024-03-11T21:05:12.318 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.319 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.319 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:05:12.319 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:24.972439+0000", 2024-03-11T21:05:12.319 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0540000000000002e-06 2024-03-11T21:05:12.319 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.319 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.319 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:05:12.319 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:24.972450+0000", 2024-03-11T21:05:12.319 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1426e-05 2024-03-11T21:05:12.319 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.319 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.319 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:05:12.319 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:24.972457+0000", 2024-03-11T21:05:12.319 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4810000000000003e-06 2024-03-11T21:05:12.320 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.320 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.320 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:05:12.320 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:24.972507+0000", 2024-03-11T21:05:12.320 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0983000000000003e-05 2024-03-11T21:05:12.320 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.320 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.320 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:05:12.320 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:24.972599+0000", 2024-03-11T21:05:12.320 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.1885000000000005e-05 2024-03-11T21:05:12.320 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.320 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.320 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0,5", 2024-03-11T21:05:12.321 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:24.972671+0000", 2024-03-11T21:05:12.321 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1767000000000006e-05 2024-03-11T21:05:12.321 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.321 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.321 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-03-11T21:05:12.321 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:24.974681+0000", 2024-03-11T21:05:12.321 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00201001 2024-03-11T21:05:12.321 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.321 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.321 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:05:12.321 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:24.974963+0000", 2024-03-11T21:05:12.321 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00028223499999999998 2024-03-11T21:05:12.321 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.321 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.321 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:05:12.321 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:25.233230+0000", 2024-03-11T21:05:12.322 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.25826708300000001 2024-03-11T21:05:12.322 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.322 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.322 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:05:12.322 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:25.233292+0000", 2024-03-11T21:05:12.322 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1323999999999998e-05 2024-03-11T21:05:12.322 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.322 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.322 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:05:12.322 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:25.233327+0000", 2024-03-11T21:05:12.322 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5228000000000002e-05 2024-03-11T21:05:12.322 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:05:12.322 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:05:12.322 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:05:12.323 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.323 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.323 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.14910.0:70113 8.2 8:41ea1eae:::benchmark_data_smithi023_89854_object20028:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e219)", 2024-03-11T21:05:12.323 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T21:01:24.979191+0000", 2024-03-11T21:05:12.323 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 227.27965229399999, 2024-03-11T21:05:12.323 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.254253283, 2024-03-11T21:05:12.323 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:05:12.323 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:05:12.323 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-03-11T21:05:12.323 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.14910", 2024-03-11T21:05:12.323 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.23:0/2457905657", 2024-03-11T21:05:12.323 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 70113 2024-03-11T21:05:12.323 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.323 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:05:12.323 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.324 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:05:12.324 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:24.979191+0000", 2024-03-11T21:05:12.324 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:05:12.324 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.324 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.324 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:05:12.324 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:24.979191+0000", 2024-03-11T21:05:12.324 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:05:12.324 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.324 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.324 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:05:12.324 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:24.979192+0000", 2024-03-11T21:05:12.324 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2440000000000001e-06 2024-03-11T21:05:12.324 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.324 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.324 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:05:12.325 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:24.979198+0000", 2024-03-11T21:05:12.325 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8869999999999997e-06 2024-03-11T21:05:12.325 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.325 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.325 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:05:12.325 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:24.979210+0000", 2024-03-11T21:05:12.325 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2573e-05 2024-03-11T21:05:12.325 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.325 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.325 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:05:12.325 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:24.979217+0000", 2024-03-11T21:05:12.325 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1700000000000002e-06 2024-03-11T21:05:12.325 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.325 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.325 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:05:12.326 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:24.979265+0000", 2024-03-11T21:05:12.326 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.85e-05 2024-03-11T21:05:12.326 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.326 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.326 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:05:12.326 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:24.979365+0000", 2024-03-11T21:05:12.326 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.9795000000000003e-05 2024-03-11T21:05:12.326 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.326 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.326 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0,5", 2024-03-11T21:05:12.326 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:24.979432+0000", 2024-03-11T21:05:12.326 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7319999999999999e-05 2024-03-11T21:05:12.326 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.326 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.326 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-03-11T21:05:12.327 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:24.981080+0000", 2024-03-11T21:05:12.327 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001647679 2024-03-11T21:05:12.327 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.327 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.327 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:05:12.327 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:24.981504+0000", 2024-03-11T21:05:12.327 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00042458600000000002 2024-03-11T21:05:12.327 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.327 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.327 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:05:12.327 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:25.233363+0000", 2024-03-11T21:05:12.327 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.251858741 2024-03-11T21:05:12.327 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.327 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.328 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:05:12.328 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:25.233409+0000", 2024-03-11T21:05:12.328 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6158000000000001e-05 2024-03-11T21:05:12.328 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.328 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.328 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:05:12.328 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:01:25.233444+0000", 2024-03-11T21:05:12.328 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4629999999999999e-05 2024-03-11T21:05:12.328 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:05:12.328 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:05:12.328 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:05:12.328 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.328 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.328 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.15081.0:45727 9.e e262/251 9:7adeeb78:::benchmark_data_smithi023_106206_object45726:head v 262'2855, mlcod=261'2853)", 2024-03-11T21:05:12.329 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T21:03:27.496485+0000", 2024-03-11T21:05:12.329 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 104.762358353, 2024-03-11T21:05:12.329 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.41900660499999998, 2024-03-11T21:05:12.329 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:05:12.329 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:05:12.329 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:05:12.329 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.329 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:05:12.329 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:27.496485+0000", 2024-03-11T21:05:12.329 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:05:12.329 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.329 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.329 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:05:12.329 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:27.496485+0000", 2024-03-11T21:05:12.329 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:05:12.330 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.330 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.330 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:05:12.330 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:27.496485+0000", 2024-03-11T21:05:12.330 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1e-07 2024-03-11T21:05:12.330 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.330 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.330 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:05:12.330 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:27.496505+0000", 2024-03-11T21:05:12.330 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9826e-05 2024-03-11T21:05:12.330 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.330 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.330 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:05:12.330 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:27.496516+0000", 2024-03-11T21:05:12.330 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1747e-05 2024-03-11T21:05:12.331 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.331 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.331 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:05:12.331 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:27.496525+0000", 2024-03-11T21:05:12.331 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8240000000000006e-06 2024-03-11T21:05:12.331 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.331 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.331 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:05:12.331 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:27.496586+0000", 2024-03-11T21:05:12.331 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0668999999999998e-05 2024-03-11T21:05:12.331 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.331 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.331 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:05:12.331 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:27.496605+0000", 2024-03-11T21:05:12.332 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9536999999999999e-05 2024-03-11T21:05:12.332 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.332 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.332 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:05:12.332 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:27.915382+0000", 2024-03-11T21:05:12.332 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.41877668499999998 2024-03-11T21:05:12.332 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.332 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.332 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:05:12.332 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:27.915491+0000", 2024-03-11T21:05:12.332 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000109107 2024-03-11T21:05:12.332 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:05:12.332 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:05:12.333 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:05:12.333 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.333 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.333 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.15081.0:45729 9.e e262/251 9:7553524a:::benchmark_data_smithi023_106206_object45728:head v 262'2856, mlcod=261'2853)", 2024-03-11T21:05:12.333 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T21:03:27.498289+0000", 2024-03-11T21:05:12.333 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 104.76055397, 2024-03-11T21:05:12.333 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.42696921599999998, 2024-03-11T21:05:12.333 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:05:12.333 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:05:12.333 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:05:12.333 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.333 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:05:12.333 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:27.498289+0000", 2024-03-11T21:05:12.333 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:05:12.334 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.334 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.334 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:05:12.334 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:27.498289+0000", 2024-03-11T21:05:12.334 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:05:12.334 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.334 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.334 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:05:12.334 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:27.498290+0000", 2024-03-11T21:05:12.334 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2799999999999995e-07 2024-03-11T21:05:12.334 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.334 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.334 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:05:12.334 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:27.498311+0000", 2024-03-11T21:05:12.335 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1702e-05 2024-03-11T21:05:12.335 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.335 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.335 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:05:12.335 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:27.914495+0000", 2024-03-11T21:05:12.335 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.41618395899999999 2024-03-11T21:05:12.335 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.335 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.335 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:05:12.335 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:27.914529+0000", 2024-03-11T21:05:12.335 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3923e-05 2024-03-11T21:05:12.335 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.335 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.336 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:05:12.336 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:27.915869+0000", 2024-03-11T21:05:12.336 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001339556 2024-03-11T21:05:12.336 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.336 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.336 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:05:12.336 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:27.915908+0000", 2024-03-11T21:05:12.336 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8612999999999998e-05 2024-03-11T21:05:12.336 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.336 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.336 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:05:12.336 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:27.925157+0000", 2024-03-11T21:05:12.336 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0092493539999999996 2024-03-11T21:05:12.336 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.336 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.337 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:05:12.337 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:27.925258+0000", 2024-03-11T21:05:12.337 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000101381 2024-03-11T21:05:12.337 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:05:12.337 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:05:12.337 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:05:12.337 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.337 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.337 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.15081.0:64010 9.c e271/251 9:3cedf398:::benchmark_data_smithi023_106206_object64009:head v 271'4052, mlcod=271'4050)", 2024-03-11T21:05:12.337 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T21:03:42.884430+0000", 2024-03-11T21:05:12.337 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 89.374412759999998, 2024-03-11T21:05:12.337 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.35927554, 2024-03-11T21:05:12.337 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:05:12.337 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:05:12.337 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:05:12.338 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.338 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:05:12.338 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:42.884430+0000", 2024-03-11T21:05:12.338 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:05:12.338 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.338 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.338 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:05:12.338 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:42.884430+0000", 2024-03-11T21:05:12.338 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:05:12.338 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.338 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.338 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:05:12.338 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:42.884431+0000", 2024-03-11T21:05:12.338 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.33e-07 2024-03-11T21:05:12.339 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.339 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.339 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:05:12.339 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:42.884499+0000", 2024-03-11T21:05:12.339 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7316000000000004e-05 2024-03-11T21:05:12.339 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.339 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.339 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:05:12.339 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:42.884513+0000", 2024-03-11T21:05:12.339 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.483e-05 2024-03-11T21:05:12.339 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.339 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.339 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:05:12.340 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:42.884520+0000", 2024-03-11T21:05:12.340 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9070000000000001e-06 2024-03-11T21:05:12.340 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.340 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.340 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:05:12.340 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:42.884576+0000", 2024-03-11T21:05:12.340 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5838999999999998e-05 2024-03-11T21:05:12.340 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.340 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.340 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:05:12.340 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:43.239006+0000", 2024-03-11T21:05:12.340 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.35443018300000001 2024-03-11T21:05:12.340 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.341 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.341 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:05:12.341 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:43.243576+0000", 2024-03-11T21:05:12.341 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0045697969999999996 2024-03-11T21:05:12.341 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.341 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.341 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:05:12.341 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:43.243706+0000", 2024-03-11T21:05:12.341 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012973499999999999 2024-03-11T21:05:12.341 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:05:12.341 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:05:12.341 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:05:12.341 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.341 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.341 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.15081.0:64017 9.e e271/251 9:743c5f8a:::benchmark_data_smithi023_106206_object64016:head v 271'3998, mlcod=271'3996)", 2024-03-11T21:05:12.342 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T21:03:42.889411+0000", 2024-03-11T21:05:12.342 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 89.369432180999993, 2024-03-11T21:05:12.342 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.357421247, 2024-03-11T21:05:12.342 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:05:12.342 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:05:12.342 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:05:12.342 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.342 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:05:12.342 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:42.889411+0000", 2024-03-11T21:05:12.342 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:05:12.342 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.342 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.342 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:05:12.343 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:42.889411+0000", 2024-03-11T21:05:12.343 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:05:12.343 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.343 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.343 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:05:12.343 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:42.889412+0000", 2024-03-11T21:05:12.343 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0389999999999999e-06 2024-03-11T21:05:12.343 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.343 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.343 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:05:12.343 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:42.889441+0000", 2024-03-11T21:05:12.343 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8948000000000001e-05 2024-03-11T21:05:12.343 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.343 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.344 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:05:12.344 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:43.238302+0000", 2024-03-11T21:05:12.344 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.34886149599999999 2024-03-11T21:05:12.344 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.344 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.344 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:05:12.344 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:43.238334+0000", 2024-03-11T21:05:12.344 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2011999999999998e-05 2024-03-11T21:05:12.344 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.344 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.344 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:05:12.344 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:43.239241+0000", 2024-03-11T21:05:12.344 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00090631200000000005 2024-03-11T21:05:12.344 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.345 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.345 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:05:12.345 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:43.239296+0000", 2024-03-11T21:05:12.345 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5018000000000001e-05 2024-03-11T21:05:12.345 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.345 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.345 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:05:12.345 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:43.246743+0000", 2024-03-11T21:05:12.345 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0074477679999999996 2024-03-11T21:05:12.345 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.345 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.345 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:05:12.345 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:43.246832+0000", 2024-03-11T21:05:12.345 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8653999999999995e-05 2024-03-11T21:05:12.345 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:05:12.346 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:05:12.346 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:05:12.346 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.346 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.346 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.15081.0:142575 9.e e284/251 9:7bb92a1d:::benchmark_data_smithi023_106206_object31429:head v 284'8978, mlcod=284'8976)", 2024-03-11T21:05:12.346 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T21:05:05.351173+0000", 2024-03-11T21:05:12.346 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 6.9076702389999998, 2024-03-11T21:05:12.346 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.43447089100000003, 2024-03-11T21:05:12.346 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:05:12.346 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:05:12.346 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:05:12.346 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.346 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:05:12.346 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:05.351173+0000", 2024-03-11T21:05:12.347 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:05:12.347 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.347 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.347 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:05:12.347 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:05.351173+0000", 2024-03-11T21:05:12.347 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:05:12.347 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.347 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.347 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:05:12.347 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:05.351174+0000", 2024-03-11T21:05:12.347 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.234e-06 2024-03-11T21:05:12.347 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.347 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.347 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:05:12.347 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:05.351183+0000", 2024-03-11T21:05:12.348 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.9619999999999999e-06 2024-03-11T21:05:12.348 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.348 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.348 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:05:12.348 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:05.764785+0000", 2024-03-11T21:05:12.348 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.41360211099999999 2024-03-11T21:05:12.348 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.348 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.348 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:05:12.348 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:05.764809+0000", 2024-03-11T21:05:12.348 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4148999999999999e-05 2024-03-11T21:05:12.348 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.348 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.348 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:05:12.348 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:05.764951+0000", 2024-03-11T21:05:12.349 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000141254 2024-03-11T21:05:12.349 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.349 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.349 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:05:12.349 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:05.765011+0000", 2024-03-11T21:05:12.349 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0739000000000002e-05 2024-03-11T21:05:12.349 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.349 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.349 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:05:12.349 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:05.785539+0000", 2024-03-11T21:05:12.349 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.020527561999999999 2024-03-11T21:05:12.349 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.349 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.349 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:05:12.349 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:05.785644+0000", 2024-03-11T21:05:12.349 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010488 2024-03-11T21:05:12.350 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:05:12.350 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:05:12.350 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:05:12.350 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.350 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.350 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.15081.0:143669 9.c e284/251 9:33cff7cf:::benchmark_data_smithi023_106206_object32523:head v 284'9056, mlcod=284'9054)", 2024-03-11T21:05:12.350 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T21:05:07.061773+0000", 2024-03-11T21:05:12.350 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 5.1970704550000004, 2024-03-11T21:05:12.350 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.45908703099999998, 2024-03-11T21:05:12.350 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:05:12.350 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:05:12.350 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:05:12.350 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.350 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:05:12.350 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:07.061773+0000", 2024-03-11T21:05:12.351 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:05:12.351 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.351 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.351 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:05:12.351 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:07.061773+0000", 2024-03-11T21:05:12.351 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:05:12.351 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.351 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.351 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:05:12.351 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:07.061774+0000", 2024-03-11T21:05:12.351 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2800000000000005e-07 2024-03-11T21:05:12.351 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.351 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.351 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:05:12.351 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:07.061781+0000", 2024-03-11T21:05:12.352 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5650000000000004e-06 2024-03-11T21:05:12.352 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.352 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.352 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:05:12.352 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:07.061792+0000", 2024-03-11T21:05:12.352 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.081e-05 2024-03-11T21:05:12.352 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.352 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.352 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:05:12.352 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:07.061799+0000", 2024-03-11T21:05:12.352 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1280000000000002e-06 2024-03-11T21:05:12.352 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.352 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.352 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:05:12.352 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:07.061869+0000", 2024-03-11T21:05:12.352 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9726000000000003e-05 2024-03-11T21:05:12.353 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.353 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.353 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:05:12.353 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:07.061891+0000", 2024-03-11T21:05:12.353 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2189999999999999e-05 2024-03-11T21:05:12.353 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.353 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.353 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:05:12.353 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:07.520795+0000", 2024-03-11T21:05:12.353 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.45890372299999999 2024-03-11T21:05:12.353 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.353 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.353 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:05:12.353 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:07.520860+0000", 2024-03-11T21:05:12.354 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4961000000000004e-05 2024-03-11T21:05:12.354 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:05:12.354 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:05:12.354 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:05:12.354 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.354 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.354 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.15081.0:143675 9.4 e284/251 9:26a5279f:::benchmark_data_smithi023_106206_object32529:head v 284'9030, mlcod=284'9027)", 2024-03-11T21:05:12.354 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T21:05:07.066093+0000", 2024-03-11T21:05:12.354 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 5.1927504759999996, 2024-03-11T21:05:12.354 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.45874078000000001, 2024-03-11T21:05:12.354 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:05:12.354 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:05:12.354 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:05:12.354 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.354 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:05:12.355 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:07.066093+0000", 2024-03-11T21:05:12.355 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:05:12.355 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.355 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.355 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:05:12.355 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:07.066093+0000", 2024-03-11T21:05:12.355 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:05:12.355 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.355 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.355 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:05:12.355 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:07.066094+0000", 2024-03-11T21:05:12.355 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0389999999999999e-06 2024-03-11T21:05:12.355 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.355 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.356 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:05:12.356 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:07.066103+0000", 2024-03-11T21:05:12.356 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.3000000000000007e-06 2024-03-11T21:05:12.356 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.356 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.356 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:05:12.356 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:07.513902+0000", 2024-03-11T21:05:12.356 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.44779908600000001 2024-03-11T21:05:12.356 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.356 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.356 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:05:12.356 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:07.513918+0000", 2024-03-11T21:05:12.356 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5863e-05 2024-03-11T21:05:12.356 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.356 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.357 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:05:12.357 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:07.514008+0000", 2024-03-11T21:05:12.357 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0148000000000005e-05 2024-03-11T21:05:12.357 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.357 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.357 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:05:12.357 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:07.514049+0000", 2024-03-11T21:05:12.357 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1099000000000001e-05 2024-03-11T21:05:12.357 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.357 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.357 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:05:12.357 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:07.524758+0000", 2024-03-11T21:05:12.357 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010709371000000001 2024-03-11T21:05:12.357 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.357 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.357 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:05:12.358 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:07.524833+0000", 2024-03-11T21:05:12.358 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4874000000000002e-05 2024-03-11T21:05:12.358 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:05:12.358 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:05:12.358 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:05:12.358 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.358 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.358 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "MOSDScrubReserve(3.6,QREQUEST,e:284,from:0,reservation_nonce:7)", 2024-03-11T21:05:12.358 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T21:05:07.197735+0000", 2024-03-11T21:05:12.358 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 5.0611079099999996, 2024-03-11T21:05:12.358 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.31858316399999997, 2024-03-11T21:05:12.358 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:05:12.358 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-03-11T21:05:12.358 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:05:12.358 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.359 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:05:12.359 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:07.197735+0000", 2024-03-11T21:05:12.359 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:05:12.359 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.359 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.359 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:05:12.359 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:07.197735+0000", 2024-03-11T21:05:12.359 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:05:12.359 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.359 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.359 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:05:12.359 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:07.197736+0000", 2024-03-11T21:05:12.359 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6200000000000006e-07 2024-03-11T21:05:12.359 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.359 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.360 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:05:12.360 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:07.197743+0000", 2024-03-11T21:05:12.360 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6030000000000001e-06 2024-03-11T21:05:12.360 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.360 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.360 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:05:12.360 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:07.514375+0000", 2024-03-11T21:05:12.360 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.31663247700000002 2024-03-11T21:05:12.360 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.360 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.360 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:05:12.360 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:07.514384+0000", 2024-03-11T21:05:12.360 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4020000000000002e-06 2024-03-11T21:05:12.360 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.360 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.361 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:05:12.361 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:07.516230+0000", 2024-03-11T21:05:12.361 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0018461300000000001 2024-03-11T21:05:12.361 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.361 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.361 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:05:12.361 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:07.516255+0000", 2024-03-11T21:05:12.361 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5469e-05 2024-03-11T21:05:12.361 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:05:12.361 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:05:12.361 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:05:12.361 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:07.516318+0000", 2024-03-11T21:05:12.361 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3120999999999994e-05 2024-03-11T21:05:12.361 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:05:12.361 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:05:12.362 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:05:12.362 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:05:12.362 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:05:12.362 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-03-11T21:05:12.362 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:12.387 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:12.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:12.389 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:05:12.490 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:05:12.530 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:05:12.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:12.537 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:05:12.638 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:12.673 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:12.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:12.680 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:05:12.781 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:05:12.799 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:05:12.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:12.801 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:05:12.902 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:05:12.925 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:05:12.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:12.927 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:05:13.028 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:05:13.065 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:05:13.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:13.072 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:05:13.173 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:13.182 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:12 smithi146 bash[23809]: cluster 2024-03-11T21:05:12.029874+0000 osd.1 (osd.1) 83 : cluster 0 3.4 scrub starts 2024-03-11T21:05:13.182 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:12 smithi146 bash[23809]: cluster 2024-03-11T21:05:12.034706+0000 osd.1 (osd.1) 84 : cluster 0 3.4 scrub ok 2024-03-11T21:05:13.182 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:12 smithi146 bash[23809]: cluster 2024-03-11T21:05:12.195361+0000 mgr.y (mgr.24380) 535 : cluster 0 pgmap v595: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 954 op/s 2024-03-11T21:05:13.201 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:13.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:13.205 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:05:13.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:12 smithi023 bash[21372]: cluster 2024-03-11T21:05:12.029874+0000 osd.1 (osd.1) 83 : cluster 0 3.4 scrub starts 2024-03-11T21:05:13.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:12 smithi023 bash[21372]: cluster 2024-03-11T21:05:12.034706+0000 osd.1 (osd.1) 84 : cluster 0 3.4 scrub ok 2024-03-11T21:05:13.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:12 smithi023 bash[21372]: cluster 2024-03-11T21:05:12.195361+0000 mgr.y (mgr.24380) 535 : cluster 0 pgmap v595: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 954 op/s 2024-03-11T21:05:13.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:12 smithi023 bash[29146]: cluster 2024-03-11T21:05:12.029874+0000 osd.1 (osd.1) 83 : cluster 0 3.4 scrub starts 2024-03-11T21:05:13.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:12 smithi023 bash[29146]: cluster 2024-03-11T21:05:12.034706+0000 osd.1 (osd.1) 84 : cluster 0 3.4 scrub ok 2024-03-11T21:05:13.283 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:12 smithi023 bash[29146]: cluster 2024-03-11T21:05:12.195361+0000 mgr.y (mgr.24380) 535 : cluster 0 pgmap v595: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 954 op/s 2024-03-11T21:05:13.305 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:05:13.333 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:05:13.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:13.339 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:05:13.440 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:05:13.469 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:05:13.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:13.474 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:05:13.516 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-11T21:05:13.575 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:13.593 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:13.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:13.595 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:05:13.696 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:05:13.749 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:05:13.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:13.756 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:05:13.857 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:05:13.880 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:05:13.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:13.885 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:05:13.986 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:14.009 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:14.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:14.011 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:05:14.112 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:05:14.154 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:05:14.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:14.160 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:05:14.261 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:05:14.292 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:05:14.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:14.294 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:05:14.395 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:14.425 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:14.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:14.427 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:05:14.528 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:14.554 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:14.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:14.563 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:05:14.663 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:05:14.688 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:05:14.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:14.690 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:05:14.791 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:05:14.818 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:05:14.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:14.823 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:05:14.924 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:05:14.953 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:05:14.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:14.957 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:05:15.058 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:05:15.081 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:05:15.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:15.083 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:05:15.184 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:05:15.220 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:05:15.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:15.223 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:05:15.324 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:05:15.333 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:15 smithi146 bash[23809]: cluster 2024-03-11T21:05:13.699038+0000 osd.7 (osd.7) 319 : cluster 0 6.1c scrub starts 2024-03-11T21:05:15.333 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:15 smithi146 bash[23809]: cluster 2024-03-11T21:05:13.702709+0000 osd.7 (osd.7) 320 : cluster 0 6.1c scrub ok 2024-03-11T21:05:15.333 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:15 smithi146 bash[23809]: cluster 2024-03-11T21:05:14.195736+0000 mgr.y (mgr.24380) 536 : cluster 0 pgmap v596: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 653 op/s 2024-03-11T21:05:15.333 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:15 smithi146 bash[23809]: cluster 2024-03-11T21:05:15.071933+0000 osd.1 (osd.1) 85 : cluster 0 2.15 scrub starts 2024-03-11T21:05:15.333 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:15 smithi146 bash[23809]: cluster 2024-03-11T21:05:15.075519+0000 osd.1 (osd.1) 86 : cluster 0 2.15 scrub ok 2024-03-11T21:05:15.346 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:05:15.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:15.348 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:05:15.372 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:15 smithi023 bash[21372]: cluster 2024-03-11T21:05:13.699038+0000 osd.7 (osd.7) 319 : cluster 0 6.1c scrub starts 2024-03-11T21:05:15.372 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:15 smithi023 bash[21372]: cluster 2024-03-11T21:05:13.702709+0000 osd.7 (osd.7) 320 : cluster 0 6.1c scrub ok 2024-03-11T21:05:15.373 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:15 smithi023 bash[21372]: cluster 2024-03-11T21:05:14.195736+0000 mgr.y (mgr.24380) 536 : cluster 0 pgmap v596: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 653 op/s 2024-03-11T21:05:15.373 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:15 smithi023 bash[21372]: cluster 2024-03-11T21:05:15.071933+0000 osd.1 (osd.1) 85 : cluster 0 2.15 scrub starts 2024-03-11T21:05:15.373 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:15 smithi023 bash[21372]: cluster 2024-03-11T21:05:15.075519+0000 osd.1 (osd.1) 86 : cluster 0 2.15 scrub ok 2024-03-11T21:05:15.373 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:15 smithi023 bash[29146]: cluster 2024-03-11T21:05:13.699038+0000 osd.7 (osd.7) 319 : cluster 0 6.1c scrub starts 2024-03-11T21:05:15.373 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:15 smithi023 bash[29146]: cluster 2024-03-11T21:05:13.702709+0000 osd.7 (osd.7) 320 : cluster 0 6.1c scrub ok 2024-03-11T21:05:15.373 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:15 smithi023 bash[29146]: cluster 2024-03-11T21:05:14.195736+0000 mgr.y (mgr.24380) 536 : cluster 0 pgmap v596: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 653 op/s 2024-03-11T21:05:15.373 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:15 smithi023 bash[29146]: cluster 2024-03-11T21:05:15.071933+0000 osd.1 (osd.1) 85 : cluster 0 2.15 scrub starts 2024-03-11T21:05:15.373 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:15 smithi023 bash[29146]: cluster 2024-03-11T21:05:15.075519+0000 osd.1 (osd.1) 86 : cluster 0 2.15 scrub ok 2024-03-11T21:05:15.449 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:15.489 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:15.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:15.494 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:05:15.595 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:05:15.604 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:05:15 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:05:15.617 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:05:15.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:15.620 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:05:15.721 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:05:15.724 INFO:teuthology.orchestra.run.smithi023.stdout:osd.0: {} 2024-03-11T21:05:15.745 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:05:15.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:15.748 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:05:15.773 INFO:teuthology.orchestra.run.smithi023.stdout:osd.1: {} 2024-03-11T21:05:15.813 INFO:teuthology.orchestra.run.smithi023.stdout:osd.2: {} 2024-03-11T21:05:15.848 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:05:15.868 INFO:teuthology.orchestra.run.smithi023.stdout:osd.3: {} 2024-03-11T21:05:15.884 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:05:15.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:15.886 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:05:15.910 INFO:teuthology.orchestra.run.smithi023.stdout:osd.4: {} 2024-03-11T21:05:15.945 INFO:teuthology.orchestra.run.smithi023.stdout:osd.5: {} 2024-03-11T21:05:15.980 INFO:teuthology.orchestra.run.smithi023.stdout:osd.6: {} 2024-03-11T21:05:15.987 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:05:16.012 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:05:16.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:16.015 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:05:16.021 INFO:teuthology.orchestra.run.smithi023.stdout:osd.7: {} 2024-03-11T21:05:16.026 INFO:teuthology.orchestra.run.smithi023.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:05:16.026 INFO:teuthology.orchestra.run.smithi023.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:05:16.026 INFO:teuthology.orchestra.run.smithi023.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:05:16.026 INFO:teuthology.orchestra.run.smithi023.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:05:16.026 INFO:teuthology.orchestra.run.smithi023.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:05:16.027 INFO:teuthology.orchestra.run.smithi023.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:05:16.027 INFO:teuthology.orchestra.run.smithi023.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:05:16.027 INFO:teuthology.orchestra.run.smithi023.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:05:16.115 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:05:16.142 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:05:16.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:16.150 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:05:16.251 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:16.274 INFO:teuthology.orchestra.run.smithi023.stderr:set osd.4 primary-affinity to 0 (802) 2024-03-11T21:05:16.287 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:16.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:16.292 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:05:16.299 INFO:teuthology.orchestra.run.smithi023.stderr:noscrub is set 2024-03-11T21:05:16.300 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:16 smithi023 bash[21372]: cluster 2024-03-11T21:05:14.685999+0000 osd.7 (osd.7) 321 : cluster 0 3.d scrub starts 2024-03-11T21:05:16.300 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:16 smithi023 bash[21372]: cluster 2024-03-11T21:05:14.687989+0000 osd.7 (osd.7) 322 : cluster 0 3.d scrub ok 2024-03-11T21:05:16.300 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:16 smithi023 bash[21372]: audit 2024-03-11T21:05:15.540485+0000 mgr.y (mgr.24380) 537 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:05:16.300 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:16 smithi023 bash[21372]: audit 2024-03-11T21:05:15.660535+0000 mon.c (mon.2) 67 : audit 0 from='client.? 172.21.15.23:0/56760060' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:05:16.300 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:16 smithi023 bash[21372]: audit 2024-03-11T21:05:15.781139+0000 mon.c (mon.2) 68 : audit 1 from='client.? 172.21.15.23:0/1516405017' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:05:16.300 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:16 smithi023 bash[21372]: audit 2024-03-11T21:05:15.782468+0000 mon.a (mon.0) 1531 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:05:16.300 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:16 smithi023 bash[21372]: cluster 2024-03-11T21:05:16.049893+0000 osd.1 (osd.1) 87 : cluster 0 6.1 scrub starts 2024-03-11T21:05:16.300 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:16 smithi023 bash[21372]: cluster 2024-03-11T21:05:16.075791+0000 osd.1 (osd.1) 88 : cluster 0 6.1 scrub ok 2024-03-11T21:05:16.300 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:16 smithi023 bash[21372]: audit 2024-03-11T21:05:16.083557+0000 mon.a (mon.0) 1532 : audit 1 from='client.? 172.21.15.23:0/4258932753' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.0}]: dispatch 2024-03-11T21:05:16.301 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:16 smithi023 bash[29146]: cluster 2024-03-11T21:05:14.685999+0000 osd.7 (osd.7) 321 : cluster 0 3.d scrub starts 2024-03-11T21:05:16.301 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:16 smithi023 bash[29146]: cluster 2024-03-11T21:05:14.687989+0000 osd.7 (osd.7) 322 : cluster 0 3.d scrub ok 2024-03-11T21:05:16.301 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:16 smithi023 bash[29146]: audit 2024-03-11T21:05:15.540485+0000 mgr.y (mgr.24380) 537 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:05:16.301 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:16 smithi023 bash[29146]: audit 2024-03-11T21:05:15.660535+0000 mon.c (mon.2) 67 : audit 0 from='client.? 172.21.15.23:0/56760060' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:05:16.301 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:16 smithi023 bash[29146]: audit 2024-03-11T21:05:15.781139+0000 mon.c (mon.2) 68 : audit 1 from='client.? 172.21.15.23:0/1516405017' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:05:16.301 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:16 smithi023 bash[29146]: audit 2024-03-11T21:05:15.782468+0000 mon.a (mon.0) 1531 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:05:16.301 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:16 smithi023 bash[29146]: cluster 2024-03-11T21:05:16.049893+0000 osd.1 (osd.1) 87 : cluster 0 6.1 scrub starts 2024-03-11T21:05:16.301 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:16 smithi023 bash[29146]: cluster 2024-03-11T21:05:16.075791+0000 osd.1 (osd.1) 88 : cluster 0 6.1 scrub ok 2024-03-11T21:05:16.301 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:16 smithi023 bash[29146]: audit 2024-03-11T21:05:16.083557+0000 mon.a (mon.0) 1532 : audit 1 from='client.? 172.21.15.23:0/4258932753' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.0}]: dispatch 2024-03-11T21:05:16.393 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:05:16.416 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:05:16.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:16.419 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:05:16.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:16 smithi146 bash[23809]: cluster 2024-03-11T21:05:14.685999+0000 osd.7 (osd.7) 321 : cluster 0 3.d scrub starts 2024-03-11T21:05:16.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:16 smithi146 bash[23809]: cluster 2024-03-11T21:05:14.687989+0000 osd.7 (osd.7) 322 : cluster 0 3.d scrub ok 2024-03-11T21:05:16.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:16 smithi146 bash[23809]: audit 2024-03-11T21:05:15.540485+0000 mgr.y (mgr.24380) 537 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:05:16.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:16 smithi146 bash[23809]: audit 2024-03-11T21:05:15.660535+0000 mon.c (mon.2) 67 : audit 0 from='client.? 172.21.15.23:0/56760060' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:05:16.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:16 smithi146 bash[23809]: audit 2024-03-11T21:05:15.781139+0000 mon.c (mon.2) 68 : audit 1 from='client.? 172.21.15.23:0/1516405017' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:05:16.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:16 smithi146 bash[23809]: audit 2024-03-11T21:05:15.782468+0000 mon.a (mon.0) 1531 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:05:16.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:16 smithi146 bash[23809]: cluster 2024-03-11T21:05:16.049893+0000 osd.1 (osd.1) 87 : cluster 0 6.1 scrub starts 2024-03-11T21:05:16.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:16 smithi146 bash[23809]: cluster 2024-03-11T21:05:16.075791+0000 osd.1 (osd.1) 88 : cluster 0 6.1 scrub ok 2024-03-11T21:05:16.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:16 smithi146 bash[23809]: audit 2024-03-11T21:05:16.083557+0000 mon.a (mon.0) 1532 : audit 1 from='client.? 172.21.15.23:0/4258932753' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.0}]: dispatch 2024-03-11T21:05:16.519 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:16.550 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:16.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:16.551 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:05:16.652 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:05:16.684 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:05:16.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:16.688 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:05:16.789 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:05:16.811 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:05:16.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:16.814 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:05:16.914 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:16.938 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:16.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:16.940 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:05:17.041 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:17.065 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:17.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:17.068 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:05:17.169 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:05:17.191 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:05:17.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:17.195 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:05:17.295 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:05:17.306 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:17 smithi146 bash[23809]: cluster 2024-03-11T21:05:16.196639+0000 mgr.y (mgr.24380) 538 : cluster 0 pgmap v597: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 4.5 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 863 op/s 2024-03-11T21:05:17.306 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:17 smithi146 bash[23809]: cluster 2024-03-11T21:05:16.256378+0000 mon.a (mon.0) 1533 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:05:17.306 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:17 smithi146 bash[23809]: audit 2024-03-11T21:05:16.272952+0000 mon.a (mon.0) 1534 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:05:17.307 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:17 smithi146 bash[23809]: audit 2024-03-11T21:05:16.273185+0000 mon.a (mon.0) 1535 : audit 1 from='client.? 172.21.15.23:0/4258932753' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.0}]': finished 2024-03-11T21:05:17.307 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:17 smithi146 bash[23809]: cluster 2024-03-11T21:05:16.296665+0000 mon.a (mon.0) 1536 : cluster 0 osdmap e285: 8 total, 8 up, 8 in 2024-03-11T21:05:17.318 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:05:17.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:17.320 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:05:17.421 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:17.448 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:17.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:17.454 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:05:17.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:17 smithi023 bash[21372]: cluster 2024-03-11T21:05:16.196639+0000 mgr.y (mgr.24380) 538 : cluster 0 pgmap v597: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 4.5 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 863 op/s 2024-03-11T21:05:17.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:17 smithi023 bash[21372]: cluster 2024-03-11T21:05:16.256378+0000 mon.a (mon.0) 1533 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:05:17.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:17 smithi023 bash[21372]: audit 2024-03-11T21:05:16.272952+0000 mon.a (mon.0) 1534 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:05:17.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:17 smithi023 bash[21372]: audit 2024-03-11T21:05:16.273185+0000 mon.a (mon.0) 1535 : audit 1 from='client.? 172.21.15.23:0/4258932753' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.0}]': finished 2024-03-11T21:05:17.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:17 smithi023 bash[21372]: cluster 2024-03-11T21:05:16.296665+0000 mon.a (mon.0) 1536 : cluster 0 osdmap e285: 8 total, 8 up, 8 in 2024-03-11T21:05:17.531 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:17 smithi023 bash[29146]: cluster 2024-03-11T21:05:16.196639+0000 mgr.y (mgr.24380) 538 : cluster 0 pgmap v597: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 4.5 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 863 op/s 2024-03-11T21:05:17.532 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:17 smithi023 bash[29146]: cluster 2024-03-11T21:05:16.256378+0000 mon.a (mon.0) 1533 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:05:17.532 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:17 smithi023 bash[29146]: audit 2024-03-11T21:05:16.272952+0000 mon.a (mon.0) 1534 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:05:17.532 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:17 smithi023 bash[29146]: audit 2024-03-11T21:05:16.273185+0000 mon.a (mon.0) 1535 : audit 1 from='client.? 172.21.15.23:0/4258932753' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.0}]': finished 2024-03-11T21:05:17.532 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:17 smithi023 bash[29146]: cluster 2024-03-11T21:05:16.296665+0000 mon.a (mon.0) 1536 : cluster 0 osdmap e285: 8 total, 8 up, 8 in 2024-03-11T21:05:17.555 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:05:17.581 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:05:17.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:17.583 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:05:17.683 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:17.710 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:17.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:17.713 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:05:17.814 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:17.834 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:17.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:17.837 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:05:17.938 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:17.957 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:17.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:17.958 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:05:18.059 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:18.080 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:18.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:18.083 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:05:18.184 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:18.207 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:18.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:18.212 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:05:18.313 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:18.339 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:18.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:18.344 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:05:18.445 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:05:18.481 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:05:18.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:18.485 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:05:18.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:18 smithi146 bash[23809]: cluster 2024-03-11T21:05:17.306180+0000 mon.a (mon.0) 1537 : cluster 0 osdmap e286: 8 total, 8 up, 8 in 2024-03-11T21:05:18.587 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:18.599 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:18 smithi023 bash[21372]: cluster 2024-03-11T21:05:17.306180+0000 mon.a (mon.0) 1537 : cluster 0 osdmap e286: 8 total, 8 up, 8 in 2024-03-11T21:05:18.599 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:05:18 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:05:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:05:18.599 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:18 smithi023 bash[29146]: cluster 2024-03-11T21:05:17.306180+0000 mon.a (mon.0) 1537 : cluster 0 osdmap e286: 8 total, 8 up, 8 in 2024-03-11T21:05:18.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:18.628 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:18.628 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:05:18.729 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:05:18.752 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:05:18.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:18.753 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:05:18.854 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:18.884 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:18.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:18.887 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:05:18.988 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:05:19.012 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:05:19.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:19.015 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:05:19.116 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:05:19.142 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:05:19.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:19.144 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:05:19.245 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:05:19.264 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:05:19.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:19.266 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:05:19.366 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:05:19.389 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:05:19.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:19.391 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:05:19.492 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:05:19.502 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:19 smithi023 bash[21372]: cluster 2024-03-11T21:05:18.197374+0000 mgr.y (mgr.24380) 539 : cluster 0 pgmap v600: 148 pgs: 10 peering, 1 active+clean+scrubbing+deep, 137 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 930 op/s 2024-03-11T21:05:19.502 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:19 smithi023 bash[21372]: cluster 2024-03-11T21:05:18.298986+0000 mon.a (mon.0) 1538 : cluster 3 Health check failed: Reduced data availability: 9 pgs inactive, 10 pgs peering (PG_AVAILABILITY) 2024-03-11T21:05:19.502 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:19 smithi023 bash[29146]: cluster 2024-03-11T21:05:18.197374+0000 mgr.y (mgr.24380) 539 : cluster 0 pgmap v600: 148 pgs: 10 peering, 1 active+clean+scrubbing+deep, 137 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 930 op/s 2024-03-11T21:05:19.502 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:19 smithi023 bash[29146]: cluster 2024-03-11T21:05:18.298986+0000 mon.a (mon.0) 1538 : cluster 3 Health check failed: Reduced data availability: 9 pgs inactive, 10 pgs peering (PG_AVAILABILITY) 2024-03-11T21:05:19.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:19 smithi146 bash[23809]: cluster 2024-03-11T21:05:18.197374+0000 mgr.y (mgr.24380) 539 : cluster 0 pgmap v600: 148 pgs: 10 peering, 1 active+clean+scrubbing+deep, 137 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 930 op/s 2024-03-11T21:05:19.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:19 smithi146 bash[23809]: cluster 2024-03-11T21:05:18.298986+0000 mon.a (mon.0) 1538 : cluster 3 Health check failed: Reduced data availability: 9 pgs inactive, 10 pgs peering (PG_AVAILABILITY) 2024-03-11T21:05:19.522 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:05:19.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:19.527 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:05:19.628 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:19.653 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:19.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:19.659 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:05:19.760 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:05:19.783 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:05:19.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:19.787 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:05:19.888 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:19.914 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:19.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:19.918 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:05:20.019 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:05:20.055 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:05:20.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:20.059 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:05:20.160 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:05:20.194 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:05:20.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:20.199 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:05:20.299 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:05:20.325 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:05:20.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:20.332 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:05:20.433 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:05:20.463 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:05:20.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:20.465 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:05:20.566 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:05:20.588 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:05:20.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:20.594 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:05:20.695 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:05:20.718 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:05:20.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:20.723 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:05:20.824 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:05:20.861 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:05:20.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:20.864 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:05:20.965 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:05:20.990 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:05:20.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:20.995 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:05:21.096 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:05:21.128 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:05:21.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:21.131 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:05:21.139 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T21:05:21.232 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:21.262 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:21.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:21.267 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:05:21.367 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:05:21.400 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:05:21.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:21.402 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:05:21.503 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:05:21.514 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:21 smithi146 bash[23809]: cluster 2024-03-11T21:05:20.197990+0000 mgr.y (mgr.24380) 540 : cluster 0 pgmap v601: 148 pgs: 10 peering, 1 active+clean+scrubbing+deep, 137 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 383 B/s rd, 0 B/s wr, 538 op/s 2024-03-11T21:05:21.514 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:21 smithi146 bash[23809]: audit 2024-03-11T21:05:20.412515+0000 mon.a (mon.0) 1539 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:05:21.514 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:21 smithi146 bash[23809]: audit 2024-03-11T21:05:20.417368+0000 mon.a (mon.0) 1540 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:05:21.531 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:05:21.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:21.535 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:05:21.636 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:21.671 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:21.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:21.673 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:05:21.774 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:05:21.782 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:21 smithi023 bash[21372]: cluster 2024-03-11T21:05:20.197990+0000 mgr.y (mgr.24380) 540 : cluster 0 pgmap v601: 148 pgs: 10 peering, 1 active+clean+scrubbing+deep, 137 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 383 B/s rd, 0 B/s wr, 538 op/s 2024-03-11T21:05:21.782 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:21 smithi023 bash[21372]: audit 2024-03-11T21:05:20.412515+0000 mon.a (mon.0) 1539 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:05:21.782 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:21 smithi023 bash[21372]: audit 2024-03-11T21:05:20.417368+0000 mon.a (mon.0) 1540 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:05:21.782 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:21 smithi023 bash[29146]: cluster 2024-03-11T21:05:20.197990+0000 mgr.y (mgr.24380) 540 : cluster 0 pgmap v601: 148 pgs: 10 peering, 1 active+clean+scrubbing+deep, 137 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 383 B/s rd, 0 B/s wr, 538 op/s 2024-03-11T21:05:21.782 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:21 smithi023 bash[29146]: audit 2024-03-11T21:05:20.412515+0000 mon.a (mon.0) 1539 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:05:21.783 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:21 smithi023 bash[29146]: audit 2024-03-11T21:05:20.417368+0000 mon.a (mon.0) 1540 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:05:21.794 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-11T21:05:21.801 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:05:21.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:21.804 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:05:21.905 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:21.927 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:21.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:21.931 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:05:22.032 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:22.055 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:22.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:22.059 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:05:22.160 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:05:22.193 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:05:22.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:22.198 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:05:22.285 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:05:22.299 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:05:22.338 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:05:22.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:22.340 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:05:22.441 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:05:22.465 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:05:22.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:22.469 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:05:22.570 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:22.595 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:22.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:22.599 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:05:22.699 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:22.708 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:22 smithi146 bash[23809]: cluster 2024-03-11T21:05:22.410778+0000 mon.a (mon.0) 1541 : cluster 1 Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 9 pgs inactive, 10 pgs peering) 2024-03-11T21:05:22.724 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:22.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:22.726 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:05:22.782 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:22 smithi023 bash[29146]: cluster 2024-03-11T21:05:22.410778+0000 mon.a (mon.0) 1541 : cluster 1 Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 9 pgs inactive, 10 pgs peering) 2024-03-11T21:05:22.782 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:22 smithi023 bash[21372]: cluster 2024-03-11T21:05:22.410778+0000 mon.a (mon.0) 1541 : cluster 1 Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 9 pgs inactive, 10 pgs peering) 2024-03-11T21:05:22.827 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:22.846 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:22.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:22.848 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:05:22.949 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:22.970 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:22.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:22.972 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:05:23.072 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:23.091 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:23.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:23.093 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:05:23.194 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:05:23.212 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:05:23.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:23.214 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:05:23.315 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:23.346 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:23.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:23.348 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:05:23.449 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:05:23.473 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:05:23.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:23.476 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:05:23.478 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:23 smithi023 bash[21372]: cluster 2024-03-11T21:05:22.198860+0000 mgr.y (mgr.24380) 541 : cluster 0 pgmap v602: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 990 op/s 2024-03-11T21:05:23.478 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:23 smithi023 bash[29146]: cluster 2024-03-11T21:05:22.198860+0000 mgr.y (mgr.24380) 541 : cluster 0 pgmap v602: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 990 op/s 2024-03-11T21:05:23.577 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:23.586 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:23 smithi146 bash[23809]: cluster 2024-03-11T21:05:22.198860+0000 mgr.y (mgr.24380) 541 : cluster 0 pgmap v602: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 990 op/s 2024-03-11T21:05:23.604 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:23.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:23.605 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:05:23.706 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:05:23.731 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:05:23.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:23.738 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:05:23.838 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:23.864 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:23.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:23.867 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:05:23.968 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:05:23.994 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:05:23.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:23.997 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:05:24.098 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:24.122 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:24.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:24.125 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:05:24.226 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:24.256 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:24.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:24.258 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:05:24.359 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:24.384 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:24.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:24.386 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:05:24.487 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:05:24.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:24.516 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:05:24.516 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:05:24.605 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 6, 7, 1] out_osds: [] dead_osds: [] live_osds: [0, 2, 3, 5, 6, 7, 4, 1] 2024-03-11T21:05:24.605 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-11T21:05:24.606 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-03-11T21:05:24.606 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd dump -f json-pretty 2024-03-11T21:05:24.617 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:05:24.680 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:05:24.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:24.682 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:05:24.783 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:05:24.803 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:05:24.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:24.805 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:05:24.906 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:05:24.928 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:05:24.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:24.931 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:05:25.031 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:05:25.054 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:05:25.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:25.056 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:05:25.157 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:25.188 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:25.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:25.190 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:05:25.291 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:05:25.322 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:05:25.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:25.325 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:05:25.426 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:05:25.449 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:05:25.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:25.454 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:05:25.554 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:05:25.585 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:05:25.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:25.591 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:05:25.692 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:05:25.701 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:05:25 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:05:25.702 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:25 smithi146 bash[23809]: cluster 2024-03-11T21:05:24.199306+0000 mgr.y (mgr.24380) 542 : cluster 0 pgmap v603: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 674 op/s 2024-03-11T21:05:25.718 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:05:25.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:25.723 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:05:25.782 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:25 smithi023 bash[21372]: cluster 2024-03-11T21:05:24.199306+0000 mgr.y (mgr.24380) 542 : cluster 0 pgmap v603: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 674 op/s 2024-03-11T21:05:25.782 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:25 smithi023 bash[29146]: cluster 2024-03-11T21:05:24.199306+0000 mgr.y (mgr.24380) 542 : cluster 0 pgmap v603: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 674 op/s 2024-03-11T21:05:25.824 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:05:25.851 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:05:25.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:25.855 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:05:25.956 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:05:25.974 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:05:25.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:25.976 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:05:26.076 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:05:26.102 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:05:26.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:26.106 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:05:26.207 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:05:26.238 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:05:26.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:26.242 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:05:26.315 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:05:26.344 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:05:26.368 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:05:26.368 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:05:26.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:26.371 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:05:26.471 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:26.503 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:26.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:26.507 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:05:26.608 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:05:26.635 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:05:26.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:26.639 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:05:26.740 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:26.751 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:26 smithi146 bash[23809]: audit 2024-03-11T21:05:25.549008+0000 mgr.y (mgr.24380) 543 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:05:26.762 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:26.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:26.767 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:05:26.781 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:26 smithi023 bash[21372]: audit 2024-03-11T21:05:25.549008+0000 mgr.y (mgr.24380) 543 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:05:26.782 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:26 smithi023 bash[29146]: audit 2024-03-11T21:05:25.549008+0000 mgr.y (mgr.24380) 543 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:05:26.868 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:05:26.895 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:05:26.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:26.898 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:05:26.999 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:05:27.024 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:05:27.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:27.026 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:05:27.127 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:05:27.153 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:05:27.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:27.158 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:05:27.258 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:27.289 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:27.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:27.295 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:05:27.396 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:27.415 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:27.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:27.417 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:05:27.518 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:27.525 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:27 smithi023 bash[21372]: cluster 2024-03-11T21:05:26.200169+0000 mgr.y (mgr.24380) 544 : cluster 0 pgmap v604: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 516 B/s rd, 0 B/s wr, 813 op/s 2024-03-11T21:05:27.526 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:27 smithi023 bash[29146]: cluster 2024-03-11T21:05:26.200169+0000 mgr.y (mgr.24380) 544 : cluster 0 pgmap v604: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 516 B/s rd, 0 B/s wr, 813 op/s 2024-03-11T21:05:27.543 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:27.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:27.546 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:05:27.647 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:05:27.684 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:05:27.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:27.688 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:05:27.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:27 smithi146 bash[23809]: cluster 2024-03-11T21:05:26.200169+0000 mgr.y (mgr.24380) 544 : cluster 0 pgmap v604: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 516 B/s rd, 0 B/s wr, 813 op/s 2024-03-11T21:05:27.789 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:05:27.822 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:05:27.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:27.825 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:05:27.926 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:27.959 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:27.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:27.963 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:05:28.064 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:05:28.099 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:05:28.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:28.102 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:05:28.203 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:28.226 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:28.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:28.229 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:05:28.329 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:28.354 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:28.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:28.357 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:05:28.458 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:05:28.487 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:05:28.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:28.489 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:05:28.590 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:28.601 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:28 smithi023 bash[21372]: cluster 2024-03-11T21:05:26.916946+0000 osd.7 (osd.7) 323 : cluster 0 purged_snaps scrub starts 2024-03-11T21:05:28.601 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:28 smithi023 bash[21372]: cluster 2024-03-11T21:05:26.917008+0000 osd.7 (osd.7) 324 : cluster 0 purged_snaps scrub ok 2024-03-11T21:05:28.601 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:28 smithi023 bash[21372]: cluster 2024-03-11T21:05:27.492840+0000 osd.5 (osd.5) 278 : cluster 0 9.6 deep-scrub ok 2024-03-11T21:05:28.601 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:28 smithi023 bash[21372]: cluster 2024-03-11T21:05:27.507186+0000 mon.a (mon.0) 1542 : cluster 0 osdmap e287: 8 total, 8 up, 8 in 2024-03-11T21:05:28.601 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:05:28 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:05:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:05:28.602 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:28 smithi023 bash[29146]: cluster 2024-03-11T21:05:26.916946+0000 osd.7 (osd.7) 323 : cluster 0 purged_snaps scrub starts 2024-03-11T21:05:28.602 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:28 smithi023 bash[29146]: cluster 2024-03-11T21:05:26.917008+0000 osd.7 (osd.7) 324 : cluster 0 purged_snaps scrub ok 2024-03-11T21:05:28.602 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:28 smithi023 bash[29146]: cluster 2024-03-11T21:05:27.492840+0000 osd.5 (osd.5) 278 : cluster 0 9.6 deep-scrub ok 2024-03-11T21:05:28.602 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:28 smithi023 bash[29146]: cluster 2024-03-11T21:05:27.507186+0000 mon.a (mon.0) 1542 : cluster 0 osdmap e287: 8 total, 8 up, 8 in 2024-03-11T21:05:28.623 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:28.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:28.626 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:05:28.727 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:05:28.751 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:05:28.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:28.755 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:05:28.779 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:28 smithi146 bash[23809]: cluster 2024-03-11T21:05:26.916946+0000 osd.7 (osd.7) 323 : cluster 0 purged_snaps scrub starts 2024-03-11T21:05:28.779 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:28 smithi146 bash[23809]: cluster 2024-03-11T21:05:26.917008+0000 osd.7 (osd.7) 324 : cluster 0 purged_snaps scrub ok 2024-03-11T21:05:28.779 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:28 smithi146 bash[23809]: cluster 2024-03-11T21:05:27.492840+0000 osd.5 (osd.5) 278 : cluster 0 9.6 deep-scrub ok 2024-03-11T21:05:28.780 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:28 smithi146 bash[23809]: cluster 2024-03-11T21:05:27.507186+0000 mon.a (mon.0) 1542 : cluster 0 osdmap e287: 8 total, 8 up, 8 in 2024-03-11T21:05:28.856 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:28.883 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:28.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:28.886 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:05:28.987 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:05:29.014 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:05:29.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:29.019 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:05:29.120 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:05:29.153 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:05:29.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:29.158 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:05:29.258 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:05:29.286 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:05:29.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:29.289 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:05:29.296 INFO:teuthology.orchestra.run.smithi023.stdout:ERROR: (22) Invalid argument 2024-03-11T21:05:29.296 INFO:teuthology.orchestra.run.smithi023.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:05:29.389 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:05:29.417 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:05:29.418 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:05:29.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:29.422 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:05:29.522 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:29.546 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:29.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:29.549 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:05:29.640 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:29 smithi023 bash[21372]: cluster 2024-03-11T21:05:28.200590+0000 mgr.y (mgr.24380) 545 : cluster 0 pgmap v606: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 614 B/s rd, 0 B/s wr, 810 op/s 2024-03-11T21:05:29.640 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:29 smithi023 bash[29146]: cluster 2024-03-11T21:05:28.200590+0000 mgr.y (mgr.24380) 545 : cluster 0 pgmap v606: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 614 B/s rd, 0 B/s wr, 810 op/s 2024-03-11T21:05:29.650 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:05:29.675 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:05:29.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:29.678 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:05:29.778 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:05:29.804 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:05:29.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:29.807 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:05:29.908 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:05:29.934 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:05:29.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:29.940 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:05:30.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:29 smithi146 bash[23809]: cluster 2024-03-11T21:05:28.200590+0000 mgr.y (mgr.24380) 545 : cluster 0 pgmap v606: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 614 B/s rd, 0 B/s wr, 810 op/s 2024-03-11T21:05:30.041 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:30.065 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:30.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:30.070 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:05:30.171 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:30.193 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:30.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:30.197 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:05:30.298 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:05:30.327 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:05:30.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:30.329 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:05:30.430 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:05:30.449 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:05:30.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:30.450 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:05:30.551 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:05:30.596 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:05:30.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:30.599 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:05:30.700 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:30.733 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:30.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:30.737 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:05:30.838 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:05:30.866 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:05:30.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:30.870 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:05:30.971 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:05:30.995 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:05:30.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:31.000 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:05:31.101 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:31.129 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:31.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:31.134 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:05:31.235 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:05:31.614 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:05:31.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:31.626 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:05:31.727 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:05:31.752 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:05:31.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:31.755 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:05:31.782 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:31 smithi023 bash[21372]: cluster 2024-03-11T21:05:30.201112+0000 mgr.y (mgr.24380) 546 : cluster 0 pgmap v607: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 716 B/s rd, 0 B/s wr, 821 op/s 2024-03-11T21:05:31.783 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:31 smithi023 bash[29146]: cluster 2024-03-11T21:05:30.201112+0000 mgr.y (mgr.24380) 546 : cluster 0 pgmap v607: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 716 B/s rd, 0 B/s wr, 821 op/s 2024-03-11T21:05:31.855 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:05:31.866 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:31 smithi146 bash[23809]: cluster 2024-03-11T21:05:30.201112+0000 mgr.y (mgr.24380) 546 : cluster 0 pgmap v607: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 716 B/s rd, 0 B/s wr, 821 op/s 2024-03-11T21:05:31.884 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:05:31.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:31.891 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:05:31.992 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:32.019 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:32.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:32.022 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:05:32.123 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:05:32.151 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:05:32.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:32.153 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:05:32.254 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:32.285 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:32.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:32.292 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:05:32.392 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:05:32.418 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:05:32.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:32.423 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:05:32.524 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:32.553 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:32.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:32.555 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:05:32.655 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:32.677 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:32.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:32.681 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:05:32.782 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:32.808 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:32.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:32.811 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:05:32.912 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:05:32.937 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:05:32.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:32.942 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:05:33.043 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:33.071 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:33.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:33.074 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:05:33.084 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-03-11T21:05:33.175 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:33.205 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:33.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:33.211 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:05:33.312 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:33.337 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:33.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:33.339 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:05:33.440 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:33.462 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:33.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:33.468 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:05:33.569 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:05:33.590 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:05:33.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:33.594 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:05:33.695 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:33.723 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:33.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:33.730 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:05:33.830 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:33.840 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:33 smithi146 bash[23809]: cluster 2024-03-11T21:05:32.202051+0000 mgr.y (mgr.24380) 547 : cluster 0 pgmap v608: 148 pgs: 148 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 933 op/s 2024-03-11T21:05:33.859 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:33.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:33.864 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:05:33.965 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:33.987 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:33.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:33.991 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:05:34.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:33 smithi023 bash[21372]: cluster 2024-03-11T21:05:32.202051+0000 mgr.y (mgr.24380) 547 : cluster 0 pgmap v608: 148 pgs: 148 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 933 op/s 2024-03-11T21:05:34.032 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:33 smithi023 bash[29146]: cluster 2024-03-11T21:05:32.202051+0000 mgr.y (mgr.24380) 547 : cluster 0 pgmap v608: 148 pgs: 148 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 933 op/s 2024-03-11T21:05:34.091 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:34.109 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:34.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:34.111 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:05:34.211 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:05:34.240 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:05:34.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:34.247 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:05:34.348 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:05:34.374 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:05:34.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:34.378 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:05:34.478 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:05:34.505 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:05:34.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:34.508 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:05:34.608 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:05:34.634 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:05:34.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:34.637 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:05:34.738 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:05:34.765 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:05:34.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:34.768 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:05:34.869 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:05:34.879 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:34 smithi146 bash[23809]: cluster 2024-03-11T21:05:33.523781+0000 osd.5 (osd.5) 279 : cluster 0 purged_snaps scrub starts 2024-03-11T21:05:34.879 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:34 smithi146 bash[23809]: cluster 2024-03-11T21:05:33.523840+0000 osd.5 (osd.5) 280 : cluster 0 purged_snaps scrub ok 2024-03-11T21:05:34.879 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:34 smithi146 bash[23809]: cluster 2024-03-11T21:05:33.549780+0000 osd.0 (osd.0) 211 : cluster 0 purged_snaps scrub starts 2024-03-11T21:05:34.879 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:34 smithi146 bash[23809]: cluster 2024-03-11T21:05:33.549847+0000 osd.0 (osd.0) 212 : cluster 0 purged_snaps scrub ok 2024-03-11T21:05:34.879 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:34 smithi146 bash[23809]: cluster 2024-03-11T21:05:33.562533+0000 mon.a (mon.0) 1543 : cluster 0 osdmap e288: 8 total, 8 up, 8 in 2024-03-11T21:05:34.893 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:05:34.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:34.895 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:05:34.996 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:05:35.027 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:05:35.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:34 smithi023 bash[21372]: cluster 2024-03-11T21:05:33.523781+0000 osd.5 (osd.5) 279 : cluster 0 purged_snaps scrub starts 2024-03-11T21:05:35.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:34 smithi023 bash[21372]: cluster 2024-03-11T21:05:33.523840+0000 osd.5 (osd.5) 280 : cluster 0 purged_snaps scrub ok 2024-03-11T21:05:35.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:34 smithi023 bash[21372]: cluster 2024-03-11T21:05:33.549780+0000 osd.0 (osd.0) 211 : cluster 0 purged_snaps scrub starts 2024-03-11T21:05:35.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:34 smithi023 bash[21372]: cluster 2024-03-11T21:05:33.549847+0000 osd.0 (osd.0) 212 : cluster 0 purged_snaps scrub ok 2024-03-11T21:05:35.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:34 smithi023 bash[21372]: cluster 2024-03-11T21:05:33.562533+0000 mon.a (mon.0) 1543 : cluster 0 osdmap e288: 8 total, 8 up, 8 in 2024-03-11T21:05:35.029 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:34 smithi023 bash[29146]: cluster 2024-03-11T21:05:33.523781+0000 osd.5 (osd.5) 279 : cluster 0 purged_snaps scrub starts 2024-03-11T21:05:35.029 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:34 smithi023 bash[29146]: cluster 2024-03-11T21:05:33.523840+0000 osd.5 (osd.5) 280 : cluster 0 purged_snaps scrub ok 2024-03-11T21:05:35.029 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:34 smithi023 bash[29146]: cluster 2024-03-11T21:05:33.549780+0000 osd.0 (osd.0) 211 : cluster 0 purged_snaps scrub starts 2024-03-11T21:05:35.029 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:34 smithi023 bash[29146]: cluster 2024-03-11T21:05:33.549847+0000 osd.0 (osd.0) 212 : cluster 0 purged_snaps scrub ok 2024-03-11T21:05:35.029 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:34 smithi023 bash[29146]: cluster 2024-03-11T21:05:33.562533+0000 mon.a (mon.0) 1543 : cluster 0 osdmap e288: 8 total, 8 up, 8 in 2024-03-11T21:05:35.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:35.030 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:05:35.131 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:05:35.152 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:05:35.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:35.158 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:05:35.259 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:05:35.286 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:05:35.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:35.292 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:05:35.392 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:35.422 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:35.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:35.427 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:05:35.528 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:35.555 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:35.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:35.559 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:05:35.660 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:05:35.695 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:05:35.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:35.698 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:05:35.799 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:05:35.828 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:05:35.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:35.831 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:05:35.831 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:35 smithi023 bash[21372]: cluster 2024-03-11T21:05:34.202611+0000 mgr.y (mgr.24380) 548 : cluster 0 pgmap v610: 148 pgs: 148 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 833 op/s 2024-03-11T21:05:35.831 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:35 smithi023 bash[21372]: cluster 2024-03-11T21:05:34.575977+0000 mon.a (mon.0) 1544 : cluster 0 osdmap e289: 8 total, 8 up, 8 in 2024-03-11T21:05:35.831 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:35 smithi023 bash[21372]: audit 2024-03-11T21:05:35.446498+0000 mon.a (mon.0) 1545 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:05:35.832 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:35 smithi023 bash[21372]: audit 2024-03-11T21:05:35.447670+0000 mon.a (mon.0) 1546 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:05:35.832 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:35 smithi023 bash[29146]: cluster 2024-03-11T21:05:34.202611+0000 mgr.y (mgr.24380) 548 : cluster 0 pgmap v610: 148 pgs: 148 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 833 op/s 2024-03-11T21:05:35.832 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:35 smithi023 bash[29146]: cluster 2024-03-11T21:05:34.575977+0000 mon.a (mon.0) 1544 : cluster 0 osdmap e289: 8 total, 8 up, 8 in 2024-03-11T21:05:35.832 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:35 smithi023 bash[29146]: audit 2024-03-11T21:05:35.446498+0000 mon.a (mon.0) 1545 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:05:35.832 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:35 smithi023 bash[29146]: audit 2024-03-11T21:05:35.447670+0000 mon.a (mon.0) 1546 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:05:35.932 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:35.941 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:05:35 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:05:35.941 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:35 smithi146 bash[23809]: cluster 2024-03-11T21:05:34.202611+0000 mgr.y (mgr.24380) 548 : cluster 0 pgmap v610: 148 pgs: 148 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 833 op/s 2024-03-11T21:05:35.941 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:35 smithi146 bash[23809]: cluster 2024-03-11T21:05:34.575977+0000 mon.a (mon.0) 1544 : cluster 0 osdmap e289: 8 total, 8 up, 8 in 2024-03-11T21:05:35.941 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:35 smithi146 bash[23809]: audit 2024-03-11T21:05:35.446498+0000 mon.a (mon.0) 1545 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:05:35.941 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:35 smithi146 bash[23809]: audit 2024-03-11T21:05:35.447670+0000 mon.a (mon.0) 1546 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:05:35.966 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:35.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:35.968 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:05:36.069 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:36.101 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:36.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:36.107 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:05:36.208 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:05:36.225 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:05:36.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:36.227 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:05:36.328 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:05:36.361 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:05:36.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:36.363 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:05:36.464 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:36.495 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:36.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:36.501 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:05:36.602 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:05:36.630 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:05:36.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:36.634 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:05:36.735 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:36.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:36.777 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:36.778 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:05:36.781 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:36 smithi023 bash[29146]: audit 2024-03-11T21:05:35.560532+0000 mgr.y (mgr.24380) 549 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:05:36.782 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:36 smithi023 bash[21372]: audit 2024-03-11T21:05:35.560532+0000 mgr.y (mgr.24380) 549 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:05:36.878 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:05:36.885 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:36 smithi146 bash[23809]: audit 2024-03-11T21:05:35.560532+0000 mgr.y (mgr.24380) 549 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:05:36.897 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:05:36.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:36.899 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:05:37.000 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:05:37.026 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:05:37.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:37.028 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:05:37.129 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:05:37.165 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:05:37.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:37.169 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:05:37.269 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:37.290 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:37.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:37.292 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:05:37.393 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:05:37.415 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:05:37.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:37.417 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:05:37.518 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:37.547 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:37.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:37.551 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:05:37.652 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:37.661 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:37 smithi146 bash[23809]: cluster 2024-03-11T21:05:36.203481+0000 mgr.y (mgr.24380) 550 : cluster 0 pgmap v612: 148 pgs: 148 active+clean; 3.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.02k op/s 2024-03-11T21:05:37.677 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:37.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:37.682 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:05:37.783 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:37.806 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:37.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:37.810 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:05:37.911 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:05:37.920 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:37 smithi023 bash[29146]: cluster 2024-03-11T21:05:36.203481+0000 mgr.y (mgr.24380) 550 : cluster 0 pgmap v612: 148 pgs: 148 active+clean; 3.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.02k op/s 2024-03-11T21:05:37.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:37 smithi023 bash[21372]: cluster 2024-03-11T21:05:36.203481+0000 mgr.y (mgr.24380) 550 : cluster 0 pgmap v612: 148 pgs: 148 active+clean; 3.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.02k op/s 2024-03-11T21:05:37.942 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:05:37.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:37.949 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:05:38.050 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:05:38.069 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:05:38.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:38.070 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:05:38.171 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:05:38.194 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:05:38.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:38.199 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:05:38.300 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:05:38.322 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:05:38.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:38.324 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:05:38.424 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:38.457 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:38.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:38.463 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:05:38.532 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:05:38 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:05:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:05:38.564 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:38.610 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:38.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:38.621 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:05:38.721 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:05:38.747 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:05:38.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:38.750 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:05:38.850 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:05:38.881 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:05:38.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:38.886 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:05:38.987 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:05:39.010 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:05:39.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:39.013 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:05:39.027 INFO:teuthology.orchestra.run.smithi023.stdout:osd.0: {} 2024-03-11T21:05:39.086 INFO:teuthology.orchestra.run.smithi023.stdout:osd.1: {} 2024-03-11T21:05:39.114 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:39.124 INFO:teuthology.orchestra.run.smithi023.stdout:osd.2: {} 2024-03-11T21:05:39.141 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:39.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:39.146 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:05:39.163 INFO:teuthology.orchestra.run.smithi023.stdout:osd.3: {} 2024-03-11T21:05:39.199 INFO:teuthology.orchestra.run.smithi023.stdout:osd.4: {} 2024-03-11T21:05:39.244 INFO:teuthology.orchestra.run.smithi023.stdout:osd.5: {} 2024-03-11T21:05:39.247 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:05:39.280 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:05:39.281 INFO:teuthology.orchestra.run.smithi023.stdout:osd.6: {} 2024-03-11T21:05:39.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:39.282 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:05:39.316 INFO:teuthology.orchestra.run.smithi023.stdout:osd.7: {} 2024-03-11T21:05:39.324 INFO:teuthology.orchestra.run.smithi023.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:05:39.324 INFO:teuthology.orchestra.run.smithi023.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:05:39.324 INFO:teuthology.orchestra.run.smithi023.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:05:39.324 INFO:teuthology.orchestra.run.smithi023.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:05:39.325 INFO:teuthology.orchestra.run.smithi023.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:05:39.325 INFO:teuthology.orchestra.run.smithi023.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:05:39.325 INFO:teuthology.orchestra.run.smithi023.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:05:39.325 INFO:teuthology.orchestra.run.smithi023.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:05:39.383 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:39.415 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:39.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:39.420 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:05:39.520 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:39.543 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:39.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:39.548 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:05:39.630 INFO:teuthology.orchestra.run.smithi023.stderr:nodeep-scrub is set 2024-03-11T21:05:39.649 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:39.667 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:39.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:39.669 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:05:39.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:39 smithi146 bash[23809]: cluster 2024-03-11T21:05:38.204150+0000 mgr.y (mgr.24380) 551 : cluster 0 pgmap v613: 148 pgs: 148 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.22k op/s 2024-03-11T21:05:39.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:39 smithi146 bash[23809]: audit 2024-03-11T21:05:38.735610+0000 mon.c (mon.2) 69 : audit 1 from='client.? 172.21.15.23:0/2018962600' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:05:39.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:39 smithi146 bash[23809]: audit 2024-03-11T21:05:38.736669+0000 mon.a (mon.0) 1547 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:05:39.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:39 smithi146 bash[23809]: audit 2024-03-11T21:05:38.993063+0000 mon.c (mon.2) 70 : audit 0 from='client.? 172.21.15.23:0/1607266673' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:05:39.770 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:05:39.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:39 smithi023 bash[21372]: cluster 2024-03-11T21:05:38.204150+0000 mgr.y (mgr.24380) 551 : cluster 0 pgmap v613: 148 pgs: 148 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.22k op/s 2024-03-11T21:05:39.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:39 smithi023 bash[21372]: audit 2024-03-11T21:05:38.735610+0000 mon.c (mon.2) 69 : audit 1 from='client.? 172.21.15.23:0/2018962600' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:05:39.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:39 smithi023 bash[21372]: audit 2024-03-11T21:05:38.736669+0000 mon.a (mon.0) 1547 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:05:39.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:39 smithi023 bash[21372]: audit 2024-03-11T21:05:38.993063+0000 mon.c (mon.2) 70 : audit 0 from='client.? 172.21.15.23:0/1607266673' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:05:39.780 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:39 smithi023 bash[29146]: cluster 2024-03-11T21:05:38.204150+0000 mgr.y (mgr.24380) 551 : cluster 0 pgmap v613: 148 pgs: 148 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.22k op/s 2024-03-11T21:05:39.780 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:39 smithi023 bash[29146]: audit 2024-03-11T21:05:38.735610+0000 mon.c (mon.2) 69 : audit 1 from='client.? 172.21.15.23:0/2018962600' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:05:39.780 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:39 smithi023 bash[29146]: audit 2024-03-11T21:05:38.736669+0000 mon.a (mon.0) 1547 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:05:39.780 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:39 smithi023 bash[29146]: audit 2024-03-11T21:05:38.993063+0000 mon.c (mon.2) 70 : audit 0 from='client.? 172.21.15.23:0/1607266673' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:05:39.793 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:05:39.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:39.797 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:05:39.898 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:39.933 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:39.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:39.936 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:05:40.036 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:05:40.064 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:05:40.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:40.066 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:05:40.167 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:05:40.190 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:05:40.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:40.192 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:05:40.293 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:40.316 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:40.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:40.318 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:05:40.418 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:40.442 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:40.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:40.444 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:05:40.545 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:05:40.585 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:05:40.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:40.589 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:05:40.690 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:40.723 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:40.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:40.727 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:05:40.782 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:40 smithi023 bash[21372]: cluster 2024-03-11T21:05:39.599141+0000 mon.a (mon.0) 1548 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:05:40.782 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:40 smithi023 bash[21372]: audit 2024-03-11T21:05:39.615802+0000 mon.a (mon.0) 1549 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:05:40.782 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:40 smithi023 bash[21372]: cluster 2024-03-11T21:05:39.631047+0000 mon.a (mon.0) 1550 : cluster 0 osdmap e290: 8 total, 8 up, 8 in 2024-03-11T21:05:40.782 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:40 smithi023 bash[29146]: cluster 2024-03-11T21:05:39.599141+0000 mon.a (mon.0) 1548 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:05:40.782 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:40 smithi023 bash[29146]: audit 2024-03-11T21:05:39.615802+0000 mon.a (mon.0) 1549 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:05:40.782 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:40 smithi023 bash[29146]: cluster 2024-03-11T21:05:39.631047+0000 mon.a (mon.0) 1550 : cluster 0 osdmap e290: 8 total, 8 up, 8 in 2024-03-11T21:05:40.828 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:40.857 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:40.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:40.859 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:05:40.960 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:40.969 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:40 smithi146 bash[23809]: cluster 2024-03-11T21:05:39.599141+0000 mon.a (mon.0) 1548 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:05:40.970 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:40 smithi146 bash[23809]: audit 2024-03-11T21:05:39.615802+0000 mon.a (mon.0) 1549 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:05:40.970 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:40 smithi146 bash[23809]: cluster 2024-03-11T21:05:39.631047+0000 mon.a (mon.0) 1550 : cluster 0 osdmap e290: 8 total, 8 up, 8 in 2024-03-11T21:05:40.986 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:40.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:40.988 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:05:41.089 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:05:41.121 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:05:41.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:41.126 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:05:41.227 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:05:41.255 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:05:41.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:41.258 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:05:41.359 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:05:41.384 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:05:41.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:41.386 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:05:41.487 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:41.505 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:41.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:41.507 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:05:41.607 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:05:41.640 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:05:41.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:41.642 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:05:41.743 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:41.776 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:41.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:41.779 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:05:41.781 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:41 smithi023 bash[21372]: cluster 2024-03-11T21:05:40.204606+0000 mgr.y (mgr.24380) 552 : cluster 0 pgmap v615: 148 pgs: 148 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 770 B/s rd, 0 B/s wr, 803 op/s 2024-03-11T21:05:41.782 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:41 smithi023 bash[29146]: cluster 2024-03-11T21:05:40.204606+0000 mgr.y (mgr.24380) 552 : cluster 0 pgmap v615: 148 pgs: 148 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 770 B/s rd, 0 B/s wr, 803 op/s 2024-03-11T21:05:41.880 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:05:41.906 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:05:41.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:41.912 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:05:42.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:41 smithi146 bash[23809]: cluster 2024-03-11T21:05:40.204606+0000 mgr.y (mgr.24380) 552 : cluster 0 pgmap v615: 148 pgs: 148 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 770 B/s rd, 0 B/s wr, 803 op/s 2024-03-11T21:05:42.013 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:42.047 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:42.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:42.050 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:05:42.150 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:42.173 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:42.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:42.177 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:05:42.277 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:05:42.303 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:05:42.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:42.305 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:05:42.406 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:05:42.442 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:05:42.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:42.445 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:05:42.546 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:05:42.572 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:05:42.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:42.577 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:05:42.594 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:05:42.656 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-11T21:05:42.661 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-11T21:05:42.661 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 290, 2024-03-11T21:05:42.661 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "67ba00f8-dfe7-11ee-95c8-87774f69a715", 2024-03-11T21:05:42.661 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2024-03-11T20:40:13.664802+0000", 2024-03-11T21:05:42.661 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2024-03-11T21:05:39.598975+0000", 2024-03-11T21:05:42.661 INFO:teuthology.orchestra.run.smithi023.stdout: "last_up_change": "2024-03-11T21:00:47.324817+0000", 2024-03-11T21:05:42.661 INFO:teuthology.orchestra.run.smithi023.stdout: "last_in_change": "2024-03-11T20:57:23.560527+0000", 2024-03-11T21:05:42.661 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-03-11T21:05:42.662 INFO:teuthology.orchestra.run.smithi023.stdout: "flags_num": 5806080, 2024-03-11T21:05:42.662 INFO:teuthology.orchestra.run.smithi023.stdout: "flags_set": [ 2024-03-11T21:05:42.662 INFO:teuthology.orchestra.run.smithi023.stdout: "nodeep-scrub", 2024-03-11T21:05:42.662 INFO:teuthology.orchestra.run.smithi023.stdout: "noscrub", 2024-03-11T21:05:42.662 INFO:teuthology.orchestra.run.smithi023.stdout: "pglog_hardlimit", 2024-03-11T21:05:42.662 INFO:teuthology.orchestra.run.smithi023.stdout: "purged_snapdirs", 2024-03-11T21:05:42.662 INFO:teuthology.orchestra.run.smithi023.stdout: "recovery_deletes", 2024-03-11T21:05:42.662 INFO:teuthology.orchestra.run.smithi023.stdout: "sortbitwise" 2024-03-11T21:05:42.662 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-11T21:05:42.662 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_version": 20, 2024-03-11T21:05:42.662 INFO:teuthology.orchestra.run.smithi023.stdout: "full_ratio": 0.94999998807907104, 2024-03-11T21:05:42.662 INFO:teuthology.orchestra.run.smithi023.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-03-11T21:05:42.662 INFO:teuthology.orchestra.run.smithi023.stdout: "nearfull_ratio": 0.85000002384185791, 2024-03-11T21:05:42.663 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_snapshot": "", 2024-03-11T21:05:42.663 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_max": 9, 2024-03-11T21:05:42.663 INFO:teuthology.orchestra.run.smithi023.stdout: "max_osd": 8, 2024-03-11T21:05:42.663 INFO:teuthology.orchestra.run.smithi023.stdout: "require_min_compat_client": "luminous", 2024-03-11T21:05:42.663 INFO:teuthology.orchestra.run.smithi023.stdout: "min_compat_client": "luminous", 2024-03-11T21:05:42.663 INFO:teuthology.orchestra.run.smithi023.stdout: "require_osd_release": "squid", 2024-03-11T21:05:42.663 INFO:teuthology.orchestra.run.smithi023.stdout: "allow_crimson": false, 2024-03-11T21:05:42.663 INFO:teuthology.orchestra.run.smithi023.stdout: "pools": [ 2024-03-11T21:05:42.663 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:05:42.663 INFO:teuthology.orchestra.run.smithi023.stdout: "pool": 1, 2024-03-11T21:05:42.663 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_name": ".mgr", 2024-03-11T21:05:42.663 INFO:teuthology.orchestra.run.smithi023.stdout: "create_time": "2024-03-11T20:45:31.964570+0000", 2024-03-11T21:05:42.663 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-11T21:05:42.663 INFO:teuthology.orchestra.run.smithi023.stdout: "flags_names": "hashpspool", 2024-03-11T21:05:42.664 INFO:teuthology.orchestra.run.smithi023.stdout: "type": 1, 2024-03-11T21:05:42.664 INFO:teuthology.orchestra.run.smithi023.stdout: "size": 3, 2024-03-11T21:05:42.664 INFO:teuthology.orchestra.run.smithi023.stdout: "min_size": 2, 2024-03-11T21:05:42.664 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_rule": 0, 2024-03-11T21:05:42.664 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_count": 0, 2024-03-11T21:05:42.664 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_target": 0, 2024-03-11T21:05:42.664 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T21:05:42.664 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T21:05:42.664 INFO:teuthology.orchestra.run.smithi023.stdout: "object_hash": 2, 2024-03-11T21:05:42.664 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_autoscale_mode": "off", 2024-03-11T21:05:42.664 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num": 1, 2024-03-11T21:05:42.664 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_placement_num": 1, 2024-03-11T21:05:42.664 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_placement_num_target": 1, 2024-03-11T21:05:42.664 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num_target": 1, 2024-03-11T21:05:42.665 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num_pending": 1, 2024-03-11T21:05:42.665 INFO:teuthology.orchestra.run.smithi023.stdout: "last_pg_merge_meta": { 2024-03-11T21:05:42.665 INFO:teuthology.orchestra.run.smithi023.stdout: "source_pgid": "0.0", 2024-03-11T21:05:42.665 INFO:teuthology.orchestra.run.smithi023.stdout: "ready_epoch": 0, 2024-03-11T21:05:42.665 INFO:teuthology.orchestra.run.smithi023.stdout: "last_epoch_started": 0, 2024-03-11T21:05:42.665 INFO:teuthology.orchestra.run.smithi023.stdout: "last_epoch_clean": 0, 2024-03-11T21:05:42.665 INFO:teuthology.orchestra.run.smithi023.stdout: "source_version": "0'0", 2024-03-11T21:05:42.665 INFO:teuthology.orchestra.run.smithi023.stdout: "target_version": "0'0" 2024-03-11T21:05:42.665 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:05:42.665 INFO:teuthology.orchestra.run.smithi023.stdout: "last_change": "142", 2024-03-11T21:05:42.665 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend": "0", 2024-03-11T21:05:42.665 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T21:05:42.665 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T21:05:42.665 INFO:teuthology.orchestra.run.smithi023.stdout: "auid": 0, 2024-03-11T21:05:42.666 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_mode": "selfmanaged", 2024-03-11T21:05:42.666 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_seq": 0, 2024-03-11T21:05:42.666 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_epoch": 0, 2024-03-11T21:05:42.666 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_snaps": [], 2024-03-11T21:05:42.666 INFO:teuthology.orchestra.run.smithi023.stdout: "removed_snaps": "[]", 2024-03-11T21:05:42.666 INFO:teuthology.orchestra.run.smithi023.stdout: "quota_max_bytes": 0, 2024-03-11T21:05:42.666 INFO:teuthology.orchestra.run.smithi023.stdout: "quota_max_objects": 0, 2024-03-11T21:05:42.666 INFO:teuthology.orchestra.run.smithi023.stdout: "tiers": [], 2024-03-11T21:05:42.666 INFO:teuthology.orchestra.run.smithi023.stdout: "tier_of": -1, 2024-03-11T21:05:42.666 INFO:teuthology.orchestra.run.smithi023.stdout: "read_tier": -1, 2024-03-11T21:05:42.666 INFO:teuthology.orchestra.run.smithi023.stdout: "write_tier": -1, 2024-03-11T21:05:42.666 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_mode": "none", 2024-03-11T21:05:42.666 INFO:teuthology.orchestra.run.smithi023.stdout: "target_max_bytes": 0, 2024-03-11T21:05:42.666 INFO:teuthology.orchestra.run.smithi023.stdout: "target_max_objects": 0, 2024-03-11T21:05:42.667 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T21:05:42.667 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T21:05:42.667 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T21:05:42.667 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_min_flush_age": 0, 2024-03-11T21:05:42.667 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_min_evict_age": 0, 2024-03-11T21:05:42.667 INFO:teuthology.orchestra.run.smithi023.stdout: "erasure_code_profile": "", 2024-03-11T21:05:42.667 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_params": { 2024-03-11T21:05:42.667 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "none" 2024-03-11T21:05:42.667 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:05:42.667 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_period": 0, 2024-03-11T21:05:42.667 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_count": 0, 2024-03-11T21:05:42.667 INFO:teuthology.orchestra.run.smithi023.stdout: "use_gmt_hitset": true, 2024-03-11T21:05:42.668 INFO:teuthology.orchestra.run.smithi023.stdout: "min_read_recency_for_promote": 0, 2024-03-11T21:05:42.668 INFO:teuthology.orchestra.run.smithi023.stdout: "min_write_recency_for_promote": 0, 2024-03-11T21:05:42.668 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T21:05:42.668 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_search_last_n": 0, 2024-03-11T21:05:42.668 INFO:teuthology.orchestra.run.smithi023.stdout: "grade_table": [], 2024-03-11T21:05:42.668 INFO:teuthology.orchestra.run.smithi023.stdout: "stripe_width": 0, 2024-03-11T21:05:42.668 INFO:teuthology.orchestra.run.smithi023.stdout: "expected_num_objects": 0, 2024-03-11T21:05:42.668 INFO:teuthology.orchestra.run.smithi023.stdout: "fast_read": false, 2024-03-11T21:05:42.668 INFO:teuthology.orchestra.run.smithi023.stdout: "options": { 2024-03-11T21:05:42.668 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num_max": 32, 2024-03-11T21:05:42.668 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num_min": 1 2024-03-11T21:05:42.668 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:05:42.668 INFO:teuthology.orchestra.run.smithi023.stdout: "application_metadata": { 2024-03-11T21:05:42.668 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr": {} 2024-03-11T21:05:42.669 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:05:42.669 INFO:teuthology.orchestra.run.smithi023.stdout: "read_balance": { 2024-03-11T21:05:42.669 INFO:teuthology.orchestra.run.smithi023.stdout: "score_type": "Fair distribution", 2024-03-11T21:05:42.669 INFO:teuthology.orchestra.run.smithi023.stdout: "score_acting": 6.9800000190734863, 2024-03-11T21:05:42.669 INFO:teuthology.orchestra.run.smithi023.stdout: "score_stable": 6.9800000190734863, 2024-03-11T21:05:42.669 INFO:teuthology.orchestra.run.smithi023.stdout: "optimal_score": 0.43000000715255737, 2024-03-11T21:05:42.669 INFO:teuthology.orchestra.run.smithi023.stdout: "raw_score_acting": 3, 2024-03-11T21:05:42.669 INFO:teuthology.orchestra.run.smithi023.stdout: "raw_score_stable": 3, 2024-03-11T21:05:42.669 INFO:teuthology.orchestra.run.smithi023.stdout: "primary_affinity_weighted": 0.80015754699707031, 2024-03-11T21:05:42.669 INFO:teuthology.orchestra.run.smithi023.stdout: "average_primary_affinity": 0.9100000262260437, 2024-03-11T21:05:42.669 INFO:teuthology.orchestra.run.smithi023.stdout: "average_primary_affinity_weighted": 0.80000001192092896 2024-03-11T21:05:42.669 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:05:42.669 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:05:42.670 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:05:42.670 INFO:teuthology.orchestra.run.smithi023.stdout: "pool": 2, 2024-03-11T21:05:42.670 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_name": ".rgw.root", 2024-03-11T21:05:42.670 INFO:teuthology.orchestra.run.smithi023.stdout: "create_time": "2024-03-11T20:50:26.421486+0000", 2024-03-11T21:05:42.670 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-11T21:05:42.670 INFO:teuthology.orchestra.run.smithi023.stdout: "flags_names": "hashpspool", 2024-03-11T21:05:42.670 INFO:teuthology.orchestra.run.smithi023.stdout: "type": 1, 2024-03-11T21:05:42.670 INFO:teuthology.orchestra.run.smithi023.stdout: "size": 3, 2024-03-11T21:05:42.670 INFO:teuthology.orchestra.run.smithi023.stdout: "min_size": 2, 2024-03-11T21:05:42.670 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_rule": 0, 2024-03-11T21:05:42.670 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_count": 0, 2024-03-11T21:05:42.670 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_target": 0, 2024-03-11T21:05:42.670 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T21:05:42.670 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T21:05:42.671 INFO:teuthology.orchestra.run.smithi023.stdout: "object_hash": 2, 2024-03-11T21:05:42.671 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_autoscale_mode": "off", 2024-03-11T21:05:42.671 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num": 32, 2024-03-11T21:05:42.671 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_placement_num": 32, 2024-03-11T21:05:42.671 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_placement_num_target": 32, 2024-03-11T21:05:42.671 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num_target": 32, 2024-03-11T21:05:42.671 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num_pending": 32, 2024-03-11T21:05:42.671 INFO:teuthology.orchestra.run.smithi023.stdout: "last_pg_merge_meta": { 2024-03-11T21:05:42.671 INFO:teuthology.orchestra.run.smithi023.stdout: "source_pgid": "0.0", 2024-03-11T21:05:42.671 INFO:teuthology.orchestra.run.smithi023.stdout: "ready_epoch": 0, 2024-03-11T21:05:42.671 INFO:teuthology.orchestra.run.smithi023.stdout: "last_epoch_started": 0, 2024-03-11T21:05:42.671 INFO:teuthology.orchestra.run.smithi023.stdout: "last_epoch_clean": 0, 2024-03-11T21:05:42.671 INFO:teuthology.orchestra.run.smithi023.stdout: "source_version": "0'0", 2024-03-11T21:05:42.671 INFO:teuthology.orchestra.run.smithi023.stdout: "target_version": "0'0" 2024-03-11T21:05:42.671 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:05:42.672 INFO:teuthology.orchestra.run.smithi023.stdout: "last_change": "78", 2024-03-11T21:05:42.672 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend": "0", 2024-03-11T21:05:42.672 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T21:05:42.672 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T21:05:42.672 INFO:teuthology.orchestra.run.smithi023.stdout: "auid": 0, 2024-03-11T21:05:42.672 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_mode": "selfmanaged", 2024-03-11T21:05:42.672 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_seq": 0, 2024-03-11T21:05:42.672 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_epoch": 0, 2024-03-11T21:05:42.672 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_snaps": [], 2024-03-11T21:05:42.672 INFO:teuthology.orchestra.run.smithi023.stdout: "removed_snaps": "[]", 2024-03-11T21:05:42.672 INFO:teuthology.orchestra.run.smithi023.stdout: "quota_max_bytes": 0, 2024-03-11T21:05:42.672 INFO:teuthology.orchestra.run.smithi023.stdout: "quota_max_objects": 0, 2024-03-11T21:05:42.672 INFO:teuthology.orchestra.run.smithi023.stdout: "tiers": [], 2024-03-11T21:05:42.672 INFO:teuthology.orchestra.run.smithi023.stdout: "tier_of": -1, 2024-03-11T21:05:42.673 INFO:teuthology.orchestra.run.smithi023.stdout: "read_tier": -1, 2024-03-11T21:05:42.673 INFO:teuthology.orchestra.run.smithi023.stdout: "write_tier": -1, 2024-03-11T21:05:42.673 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_mode": "none", 2024-03-11T21:05:42.673 INFO:teuthology.orchestra.run.smithi023.stdout: "target_max_bytes": 0, 2024-03-11T21:05:42.673 INFO:teuthology.orchestra.run.smithi023.stdout: "target_max_objects": 0, 2024-03-11T21:05:42.673 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T21:05:42.673 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T21:05:42.673 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T21:05:42.673 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_min_flush_age": 0, 2024-03-11T21:05:42.673 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_min_evict_age": 0, 2024-03-11T21:05:42.673 INFO:teuthology.orchestra.run.smithi023.stdout: "erasure_code_profile": "", 2024-03-11T21:05:42.673 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_params": { 2024-03-11T21:05:42.673 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "none" 2024-03-11T21:05:42.673 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:05:42.674 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_period": 0, 2024-03-11T21:05:42.674 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_count": 0, 2024-03-11T21:05:42.674 INFO:teuthology.orchestra.run.smithi023.stdout: "use_gmt_hitset": true, 2024-03-11T21:05:42.674 INFO:teuthology.orchestra.run.smithi023.stdout: "min_read_recency_for_promote": 0, 2024-03-11T21:05:42.674 INFO:teuthology.orchestra.run.smithi023.stdout: "min_write_recency_for_promote": 0, 2024-03-11T21:05:42.674 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T21:05:42.674 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_search_last_n": 0, 2024-03-11T21:05:42.674 INFO:teuthology.orchestra.run.smithi023.stdout: "grade_table": [], 2024-03-11T21:05:42.674 INFO:teuthology.orchestra.run.smithi023.stdout: "stripe_width": 0, 2024-03-11T21:05:42.674 INFO:teuthology.orchestra.run.smithi023.stdout: "expected_num_objects": 0, 2024-03-11T21:05:42.674 INFO:teuthology.orchestra.run.smithi023.stdout: "fast_read": false, 2024-03-11T21:05:42.674 INFO:teuthology.orchestra.run.smithi023.stdout: "options": {}, 2024-03-11T21:05:42.675 INFO:teuthology.orchestra.run.smithi023.stdout: "application_metadata": { 2024-03-11T21:05:42.675 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw": {} 2024-03-11T21:05:42.675 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:05:42.675 INFO:teuthology.orchestra.run.smithi023.stdout: "read_balance": { 2024-03-11T21:05:42.675 INFO:teuthology.orchestra.run.smithi023.stdout: "score_type": "Fair distribution", 2024-03-11T21:05:42.675 INFO:teuthology.orchestra.run.smithi023.stdout: "score_acting": 1.5399999618530273, 2024-03-11T21:05:42.675 INFO:teuthology.orchestra.run.smithi023.stdout: "score_stable": 1.5399999618530273, 2024-03-11T21:05:42.675 INFO:teuthology.orchestra.run.smithi023.stdout: "optimal_score": 1.1399999856948853, 2024-03-11T21:05:42.675 INFO:teuthology.orchestra.run.smithi023.stdout: "raw_score_acting": 1.75, 2024-03-11T21:05:42.675 INFO:teuthology.orchestra.run.smithi023.stdout: "raw_score_stable": 1.75, 2024-03-11T21:05:42.675 INFO:teuthology.orchestra.run.smithi023.stdout: "primary_affinity_weighted": 0.80015754699707031, 2024-03-11T21:05:42.675 INFO:teuthology.orchestra.run.smithi023.stdout: "average_primary_affinity": 0.9100000262260437, 2024-03-11T21:05:42.675 INFO:teuthology.orchestra.run.smithi023.stdout: "average_primary_affinity_weighted": 0.80000001192092896 2024-03-11T21:05:42.675 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:05:42.676 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:05:42.676 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:05:42.676 INFO:teuthology.orchestra.run.smithi023.stdout: "pool": 3, 2024-03-11T21:05:42.676 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_name": "default.rgw.log", 2024-03-11T21:05:42.676 INFO:teuthology.orchestra.run.smithi023.stdout: "create_time": "2024-03-11T20:50:28.592615+0000", 2024-03-11T21:05:42.676 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-11T21:05:42.676 INFO:teuthology.orchestra.run.smithi023.stdout: "flags_names": "hashpspool", 2024-03-11T21:05:42.676 INFO:teuthology.orchestra.run.smithi023.stdout: "type": 1, 2024-03-11T21:05:42.676 INFO:teuthology.orchestra.run.smithi023.stdout: "size": 3, 2024-03-11T21:05:42.676 INFO:teuthology.orchestra.run.smithi023.stdout: "min_size": 2, 2024-03-11T21:05:42.676 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_rule": 0, 2024-03-11T21:05:42.676 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_count": 0, 2024-03-11T21:05:42.676 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_target": 0, 2024-03-11T21:05:42.676 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T21:05:42.676 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T21:05:42.677 INFO:teuthology.orchestra.run.smithi023.stdout: "object_hash": 2, 2024-03-11T21:05:42.677 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_autoscale_mode": "off", 2024-03-11T21:05:42.677 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num": 32, 2024-03-11T21:05:42.677 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_placement_num": 32, 2024-03-11T21:05:42.677 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_placement_num_target": 32, 2024-03-11T21:05:42.677 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num_target": 32, 2024-03-11T21:05:42.677 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num_pending": 32, 2024-03-11T21:05:42.677 INFO:teuthology.orchestra.run.smithi023.stdout: "last_pg_merge_meta": { 2024-03-11T21:05:42.677 INFO:teuthology.orchestra.run.smithi023.stdout: "source_pgid": "0.0", 2024-03-11T21:05:42.677 INFO:teuthology.orchestra.run.smithi023.stdout: "ready_epoch": 0, 2024-03-11T21:05:42.677 INFO:teuthology.orchestra.run.smithi023.stdout: "last_epoch_started": 0, 2024-03-11T21:05:42.677 INFO:teuthology.orchestra.run.smithi023.stdout: "last_epoch_clean": 0, 2024-03-11T21:05:42.677 INFO:teuthology.orchestra.run.smithi023.stdout: "source_version": "0'0", 2024-03-11T21:05:42.677 INFO:teuthology.orchestra.run.smithi023.stdout: "target_version": "0'0" 2024-03-11T21:05:42.678 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:05:42.678 INFO:teuthology.orchestra.run.smithi023.stdout: "last_change": "80", 2024-03-11T21:05:42.678 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend": "0", 2024-03-11T21:05:42.678 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T21:05:42.678 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T21:05:42.678 INFO:teuthology.orchestra.run.smithi023.stdout: "auid": 0, 2024-03-11T21:05:42.678 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_mode": "selfmanaged", 2024-03-11T21:05:42.678 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_seq": 0, 2024-03-11T21:05:42.678 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_epoch": 0, 2024-03-11T21:05:42.678 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_snaps": [], 2024-03-11T21:05:42.678 INFO:teuthology.orchestra.run.smithi023.stdout: "removed_snaps": "[]", 2024-03-11T21:05:42.678 INFO:teuthology.orchestra.run.smithi023.stdout: "quota_max_bytes": 0, 2024-03-11T21:05:42.678 INFO:teuthology.orchestra.run.smithi023.stdout: "quota_max_objects": 0, 2024-03-11T21:05:42.678 INFO:teuthology.orchestra.run.smithi023.stdout: "tiers": [], 2024-03-11T21:05:42.679 INFO:teuthology.orchestra.run.smithi023.stdout: "tier_of": -1, 2024-03-11T21:05:42.679 INFO:teuthology.orchestra.run.smithi023.stdout: "read_tier": -1, 2024-03-11T21:05:42.679 INFO:teuthology.orchestra.run.smithi023.stdout: "write_tier": -1, 2024-03-11T21:05:42.679 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_mode": "none", 2024-03-11T21:05:42.679 INFO:teuthology.orchestra.run.smithi023.stdout: "target_max_bytes": 0, 2024-03-11T21:05:42.679 INFO:teuthology.orchestra.run.smithi023.stdout: "target_max_objects": 0, 2024-03-11T21:05:42.679 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T21:05:42.679 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T21:05:42.679 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T21:05:42.679 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_min_flush_age": 0, 2024-03-11T21:05:42.679 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_min_evict_age": 0, 2024-03-11T21:05:42.679 INFO:teuthology.orchestra.run.smithi023.stdout: "erasure_code_profile": "", 2024-03-11T21:05:42.679 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_params": { 2024-03-11T21:05:42.679 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "none" 2024-03-11T21:05:42.680 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:05:42.680 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_period": 0, 2024-03-11T21:05:42.680 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_count": 0, 2024-03-11T21:05:42.680 INFO:teuthology.orchestra.run.smithi023.stdout: "use_gmt_hitset": true, 2024-03-11T21:05:42.680 INFO:teuthology.orchestra.run.smithi023.stdout: "min_read_recency_for_promote": 0, 2024-03-11T21:05:42.680 INFO:teuthology.orchestra.run.smithi023.stdout: "min_write_recency_for_promote": 0, 2024-03-11T21:05:42.680 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T21:05:42.680 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_search_last_n": 0, 2024-03-11T21:05:42.680 INFO:teuthology.orchestra.run.smithi023.stdout: "grade_table": [], 2024-03-11T21:05:42.680 INFO:teuthology.orchestra.run.smithi023.stdout: "stripe_width": 0, 2024-03-11T21:05:42.680 INFO:teuthology.orchestra.run.smithi023.stdout: "expected_num_objects": 0, 2024-03-11T21:05:42.680 INFO:teuthology.orchestra.run.smithi023.stdout: "fast_read": false, 2024-03-11T21:05:42.680 INFO:teuthology.orchestra.run.smithi023.stdout: "options": {}, 2024-03-11T21:05:42.680 INFO:teuthology.orchestra.run.smithi023.stdout: "application_metadata": { 2024-03-11T21:05:42.680 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw": {} 2024-03-11T21:05:42.681 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:05:42.681 INFO:teuthology.orchestra.run.smithi023.stdout: "read_balance": { 2024-03-11T21:05:42.681 INFO:teuthology.orchestra.run.smithi023.stdout: "score_type": "Fair distribution", 2024-03-11T21:05:42.681 INFO:teuthology.orchestra.run.smithi023.stdout: "score_acting": 1.75, 2024-03-11T21:05:42.681 INFO:teuthology.orchestra.run.smithi023.stdout: "score_stable": 1.75, 2024-03-11T21:05:42.681 INFO:teuthology.orchestra.run.smithi023.stdout: "optimal_score": 1.1399999856948853, 2024-03-11T21:05:42.681 INFO:teuthology.orchestra.run.smithi023.stdout: "raw_score_acting": 2, 2024-03-11T21:05:42.681 INFO:teuthology.orchestra.run.smithi023.stdout: "raw_score_stable": 2, 2024-03-11T21:05:42.681 INFO:teuthology.orchestra.run.smithi023.stdout: "primary_affinity_weighted": 0.80015754699707031, 2024-03-11T21:05:42.681 INFO:teuthology.orchestra.run.smithi023.stdout: "average_primary_affinity": 0.9100000262260437, 2024-03-11T21:05:42.681 INFO:teuthology.orchestra.run.smithi023.stdout: "average_primary_affinity_weighted": 0.80000001192092896 2024-03-11T21:05:42.681 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:05:42.681 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:05:42.681 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:05:42.681 INFO:teuthology.orchestra.run.smithi023.stdout: "pool": 4, 2024-03-11T21:05:42.682 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_name": "default.rgw.control", 2024-03-11T21:05:42.682 INFO:teuthology.orchestra.run.smithi023.stdout: "create_time": "2024-03-11T20:50:30.536371+0000", 2024-03-11T21:05:42.682 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-11T21:05:42.682 INFO:teuthology.orchestra.run.smithi023.stdout: "flags_names": "hashpspool", 2024-03-11T21:05:42.682 INFO:teuthology.orchestra.run.smithi023.stdout: "type": 1, 2024-03-11T21:05:42.682 INFO:teuthology.orchestra.run.smithi023.stdout: "size": 3, 2024-03-11T21:05:42.682 INFO:teuthology.orchestra.run.smithi023.stdout: "min_size": 2, 2024-03-11T21:05:42.682 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_rule": 0, 2024-03-11T21:05:42.682 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_count": 0, 2024-03-11T21:05:42.682 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_target": 0, 2024-03-11T21:05:42.682 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T21:05:42.682 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T21:05:42.682 INFO:teuthology.orchestra.run.smithi023.stdout: "object_hash": 2, 2024-03-11T21:05:42.682 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_autoscale_mode": "off", 2024-03-11T21:05:42.683 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num": 32, 2024-03-11T21:05:42.683 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_placement_num": 32, 2024-03-11T21:05:42.683 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_placement_num_target": 32, 2024-03-11T21:05:42.683 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num_target": 32, 2024-03-11T21:05:42.683 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num_pending": 32, 2024-03-11T21:05:42.683 INFO:teuthology.orchestra.run.smithi023.stdout: "last_pg_merge_meta": { 2024-03-11T21:05:42.683 INFO:teuthology.orchestra.run.smithi023.stdout: "source_pgid": "0.0", 2024-03-11T21:05:42.683 INFO:teuthology.orchestra.run.smithi023.stdout: "ready_epoch": 0, 2024-03-11T21:05:42.683 INFO:teuthology.orchestra.run.smithi023.stdout: "last_epoch_started": 0, 2024-03-11T21:05:42.683 INFO:teuthology.orchestra.run.smithi023.stdout: "last_epoch_clean": 0, 2024-03-11T21:05:42.683 INFO:teuthology.orchestra.run.smithi023.stdout: "source_version": "0'0", 2024-03-11T21:05:42.683 INFO:teuthology.orchestra.run.smithi023.stdout: "target_version": "0'0" 2024-03-11T21:05:42.683 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:05:42.683 INFO:teuthology.orchestra.run.smithi023.stdout: "last_change": "82", 2024-03-11T21:05:42.683 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend": "0", 2024-03-11T21:05:42.684 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T21:05:42.684 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T21:05:42.684 INFO:teuthology.orchestra.run.smithi023.stdout: "auid": 0, 2024-03-11T21:05:42.684 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_mode": "selfmanaged", 2024-03-11T21:05:42.684 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_seq": 0, 2024-03-11T21:05:42.684 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_epoch": 0, 2024-03-11T21:05:42.684 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_snaps": [], 2024-03-11T21:05:42.684 INFO:teuthology.orchestra.run.smithi023.stdout: "removed_snaps": "[]", 2024-03-11T21:05:42.684 INFO:teuthology.orchestra.run.smithi023.stdout: "quota_max_bytes": 0, 2024-03-11T21:05:42.684 INFO:teuthology.orchestra.run.smithi023.stdout: "quota_max_objects": 0, 2024-03-11T21:05:42.684 INFO:teuthology.orchestra.run.smithi023.stdout: "tiers": [], 2024-03-11T21:05:42.684 INFO:teuthology.orchestra.run.smithi023.stdout: "tier_of": -1, 2024-03-11T21:05:42.684 INFO:teuthology.orchestra.run.smithi023.stdout: "read_tier": -1, 2024-03-11T21:05:42.685 INFO:teuthology.orchestra.run.smithi023.stdout: "write_tier": -1, 2024-03-11T21:05:42.685 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_mode": "none", 2024-03-11T21:05:42.685 INFO:teuthology.orchestra.run.smithi023.stdout: "target_max_bytes": 0, 2024-03-11T21:05:42.685 INFO:teuthology.orchestra.run.smithi023.stdout: "target_max_objects": 0, 2024-03-11T21:05:42.685 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T21:05:42.685 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T21:05:42.685 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T21:05:42.685 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_min_flush_age": 0, 2024-03-11T21:05:42.685 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_min_evict_age": 0, 2024-03-11T21:05:42.685 INFO:teuthology.orchestra.run.smithi023.stdout: "erasure_code_profile": "", 2024-03-11T21:05:42.685 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_params": { 2024-03-11T21:05:42.685 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "none" 2024-03-11T21:05:42.685 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:05:42.686 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_period": 0, 2024-03-11T21:05:42.686 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_count": 0, 2024-03-11T21:05:42.686 INFO:teuthology.orchestra.run.smithi023.stdout: "use_gmt_hitset": true, 2024-03-11T21:05:42.686 INFO:teuthology.orchestra.run.smithi023.stdout: "min_read_recency_for_promote": 0, 2024-03-11T21:05:42.686 INFO:teuthology.orchestra.run.smithi023.stdout: "min_write_recency_for_promote": 0, 2024-03-11T21:05:42.686 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T21:05:42.686 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_search_last_n": 0, 2024-03-11T21:05:42.686 INFO:teuthology.orchestra.run.smithi023.stdout: "grade_table": [], 2024-03-11T21:05:42.686 INFO:teuthology.orchestra.run.smithi023.stdout: "stripe_width": 0, 2024-03-11T21:05:42.686 INFO:teuthology.orchestra.run.smithi023.stdout: "expected_num_objects": 0, 2024-03-11T21:05:42.686 INFO:teuthology.orchestra.run.smithi023.stdout: "fast_read": false, 2024-03-11T21:05:42.686 INFO:teuthology.orchestra.run.smithi023.stdout: "options": {}, 2024-03-11T21:05:42.686 INFO:teuthology.orchestra.run.smithi023.stdout: "application_metadata": { 2024-03-11T21:05:42.686 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw": {} 2024-03-11T21:05:42.686 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:05:42.687 INFO:teuthology.orchestra.run.smithi023.stdout: "read_balance": { 2024-03-11T21:05:42.687 INFO:teuthology.orchestra.run.smithi023.stdout: "score_type": "Fair distribution", 2024-03-11T21:05:42.687 INFO:teuthology.orchestra.run.smithi023.stdout: "score_acting": 1.9700000286102295, 2024-03-11T21:05:42.687 INFO:teuthology.orchestra.run.smithi023.stdout: "score_stable": 1.9700000286102295, 2024-03-11T21:05:42.687 INFO:teuthology.orchestra.run.smithi023.stdout: "optimal_score": 1.1399999856948853, 2024-03-11T21:05:42.687 INFO:teuthology.orchestra.run.smithi023.stdout: "raw_score_acting": 2.25, 2024-03-11T21:05:42.687 INFO:teuthology.orchestra.run.smithi023.stdout: "raw_score_stable": 2.25, 2024-03-11T21:05:42.687 INFO:teuthology.orchestra.run.smithi023.stdout: "primary_affinity_weighted": 0.80015754699707031, 2024-03-11T21:05:42.687 INFO:teuthology.orchestra.run.smithi023.stdout: "average_primary_affinity": 0.9100000262260437, 2024-03-11T21:05:42.687 INFO:teuthology.orchestra.run.smithi023.stdout: "average_primary_affinity_weighted": 0.80000001192092896 2024-03-11T21:05:42.687 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:05:42.687 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:05:42.687 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:05:42.687 INFO:teuthology.orchestra.run.smithi023.stdout: "pool": 5, 2024-03-11T21:05:42.688 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_name": "datapool", 2024-03-11T21:05:42.688 INFO:teuthology.orchestra.run.smithi023.stdout: "create_time": "2024-03-11T20:50:31.135845+0000", 2024-03-11T21:05:42.688 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 8193, 2024-03-11T21:05:42.688 INFO:teuthology.orchestra.run.smithi023.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-03-11T21:05:42.688 INFO:teuthology.orchestra.run.smithi023.stdout: "type": 1, 2024-03-11T21:05:42.688 INFO:teuthology.orchestra.run.smithi023.stdout: "size": 3, 2024-03-11T21:05:42.688 INFO:teuthology.orchestra.run.smithi023.stdout: "min_size": 2, 2024-03-11T21:05:42.688 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_rule": 0, 2024-03-11T21:05:42.688 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_count": 0, 2024-03-11T21:05:42.688 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_target": 0, 2024-03-11T21:05:42.688 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T21:05:42.688 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T21:05:42.688 INFO:teuthology.orchestra.run.smithi023.stdout: "object_hash": 2, 2024-03-11T21:05:42.688 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_autoscale_mode": "off", 2024-03-11T21:05:42.688 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num": 3, 2024-03-11T21:05:42.688 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_placement_num": 3, 2024-03-11T21:05:42.689 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_placement_num_target": 3, 2024-03-11T21:05:42.689 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num_target": 3, 2024-03-11T21:05:42.689 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num_pending": 3, 2024-03-11T21:05:42.689 INFO:teuthology.orchestra.run.smithi023.stdout: "last_pg_merge_meta": { 2024-03-11T21:05:42.689 INFO:teuthology.orchestra.run.smithi023.stdout: "source_pgid": "0.0", 2024-03-11T21:05:42.689 INFO:teuthology.orchestra.run.smithi023.stdout: "ready_epoch": 0, 2024-03-11T21:05:42.689 INFO:teuthology.orchestra.run.smithi023.stdout: "last_epoch_started": 0, 2024-03-11T21:05:42.689 INFO:teuthology.orchestra.run.smithi023.stdout: "last_epoch_clean": 0, 2024-03-11T21:05:42.689 INFO:teuthology.orchestra.run.smithi023.stdout: "source_version": "0'0", 2024-03-11T21:05:42.689 INFO:teuthology.orchestra.run.smithi023.stdout: "target_version": "0'0" 2024-03-11T21:05:42.689 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:05:42.689 INFO:teuthology.orchestra.run.smithi023.stdout: "last_change": "87", 2024-03-11T21:05:42.689 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend": "0", 2024-03-11T21:05:42.689 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T21:05:42.689 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T21:05:42.690 INFO:teuthology.orchestra.run.smithi023.stdout: "auid": 0, 2024-03-11T21:05:42.690 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_mode": "selfmanaged", 2024-03-11T21:05:42.690 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_seq": 2, 2024-03-11T21:05:42.690 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_epoch": 87, 2024-03-11T21:05:42.690 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_snaps": [], 2024-03-11T21:05:42.690 INFO:teuthology.orchestra.run.smithi023.stdout: "removed_snaps": "[]", 2024-03-11T21:05:42.690 INFO:teuthology.orchestra.run.smithi023.stdout: "quota_max_bytes": 0, 2024-03-11T21:05:42.690 INFO:teuthology.orchestra.run.smithi023.stdout: "quota_max_objects": 0, 2024-03-11T21:05:42.690 INFO:teuthology.orchestra.run.smithi023.stdout: "tiers": [], 2024-03-11T21:05:42.690 INFO:teuthology.orchestra.run.smithi023.stdout: "tier_of": -1, 2024-03-11T21:05:42.690 INFO:teuthology.orchestra.run.smithi023.stdout: "read_tier": -1, 2024-03-11T21:05:42.690 INFO:teuthology.orchestra.run.smithi023.stdout: "write_tier": -1, 2024-03-11T21:05:42.690 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_mode": "none", 2024-03-11T21:05:42.690 INFO:teuthology.orchestra.run.smithi023.stdout: "target_max_bytes": 0, 2024-03-11T21:05:42.691 INFO:teuthology.orchestra.run.smithi023.stdout: "target_max_objects": 0, 2024-03-11T21:05:42.691 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T21:05:42.691 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T21:05:42.691 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T21:05:42.691 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_min_flush_age": 0, 2024-03-11T21:05:42.691 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_min_evict_age": 0, 2024-03-11T21:05:42.691 INFO:teuthology.orchestra.run.smithi023.stdout: "erasure_code_profile": "", 2024-03-11T21:05:42.691 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_params": { 2024-03-11T21:05:42.691 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "none" 2024-03-11T21:05:42.691 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:05:42.691 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_period": 0, 2024-03-11T21:05:42.691 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_count": 0, 2024-03-11T21:05:42.691 INFO:teuthology.orchestra.run.smithi023.stdout: "use_gmt_hitset": true, 2024-03-11T21:05:42.691 INFO:teuthology.orchestra.run.smithi023.stdout: "min_read_recency_for_promote": 0, 2024-03-11T21:05:42.691 INFO:teuthology.orchestra.run.smithi023.stdout: "min_write_recency_for_promote": 0, 2024-03-11T21:05:42.692 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T21:05:42.692 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_search_last_n": 0, 2024-03-11T21:05:42.692 INFO:teuthology.orchestra.run.smithi023.stdout: "grade_table": [], 2024-03-11T21:05:42.692 INFO:teuthology.orchestra.run.smithi023.stdout: "stripe_width": 0, 2024-03-11T21:05:42.692 INFO:teuthology.orchestra.run.smithi023.stdout: "expected_num_objects": 0, 2024-03-11T21:05:42.692 INFO:teuthology.orchestra.run.smithi023.stdout: "fast_read": false, 2024-03-11T21:05:42.692 INFO:teuthology.orchestra.run.smithi023.stdout: "options": {}, 2024-03-11T21:05:42.692 INFO:teuthology.orchestra.run.smithi023.stdout: "application_metadata": { 2024-03-11T21:05:42.692 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd": {} 2024-03-11T21:05:42.692 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:05:42.692 INFO:teuthology.orchestra.run.smithi023.stdout: "read_balance": { 2024-03-11T21:05:42.692 INFO:teuthology.orchestra.run.smithi023.stdout: "score_type": "Fair distribution", 2024-03-11T21:05:42.692 INFO:teuthology.orchestra.run.smithi023.stdout: "score_acting": 4.6700000762939453, 2024-03-11T21:05:42.692 INFO:teuthology.orchestra.run.smithi023.stdout: "score_stable": 4.6700000762939453, 2024-03-11T21:05:42.693 INFO:teuthology.orchestra.run.smithi023.stdout: "optimal_score": 1, 2024-03-11T21:05:42.693 INFO:teuthology.orchestra.run.smithi023.stdout: "raw_score_acting": 4.6700000762939453, 2024-03-11T21:05:42.693 INFO:teuthology.orchestra.run.smithi023.stdout: "raw_score_stable": 4.6700000762939453, 2024-03-11T21:05:42.693 INFO:teuthology.orchestra.run.smithi023.stdout: "primary_affinity_weighted": 0.80015754699707031, 2024-03-11T21:05:42.693 INFO:teuthology.orchestra.run.smithi023.stdout: "average_primary_affinity": 0.9100000262260437, 2024-03-11T21:05:42.693 INFO:teuthology.orchestra.run.smithi023.stdout: "average_primary_affinity_weighted": 0.80000001192092896 2024-03-11T21:05:42.693 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:05:42.693 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:05:42.693 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:05:42.693 INFO:teuthology.orchestra.run.smithi023.stdout: "pool": 6, 2024-03-11T21:05:42.693 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_name": "default.rgw.meta", 2024-03-11T21:05:42.693 INFO:teuthology.orchestra.run.smithi023.stdout: "create_time": "2024-03-11T20:50:33.233163+0000", 2024-03-11T21:05:42.693 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-11T21:05:42.693 INFO:teuthology.orchestra.run.smithi023.stdout: "flags_names": "hashpspool", 2024-03-11T21:05:42.693 INFO:teuthology.orchestra.run.smithi023.stdout: "type": 1, 2024-03-11T21:05:42.694 INFO:teuthology.orchestra.run.smithi023.stdout: "size": 3, 2024-03-11T21:05:42.694 INFO:teuthology.orchestra.run.smithi023.stdout: "min_size": 2, 2024-03-11T21:05:42.694 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_rule": 0, 2024-03-11T21:05:42.694 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_count": 0, 2024-03-11T21:05:42.694 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_target": 0, 2024-03-11T21:05:42.694 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T21:05:42.694 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T21:05:42.694 INFO:teuthology.orchestra.run.smithi023.stdout: "object_hash": 2, 2024-03-11T21:05:42.694 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_autoscale_mode": "off", 2024-03-11T21:05:42.694 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num": 32, 2024-03-11T21:05:42.694 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_placement_num": 32, 2024-03-11T21:05:42.694 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_placement_num_target": 32, 2024-03-11T21:05:42.694 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num_target": 32, 2024-03-11T21:05:42.695 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num_pending": 32, 2024-03-11T21:05:42.695 INFO:teuthology.orchestra.run.smithi023.stdout: "last_pg_merge_meta": { 2024-03-11T21:05:42.695 INFO:teuthology.orchestra.run.smithi023.stdout: "source_pgid": "0.0", 2024-03-11T21:05:42.695 INFO:teuthology.orchestra.run.smithi023.stdout: "ready_epoch": 0, 2024-03-11T21:05:42.695 INFO:teuthology.orchestra.run.smithi023.stdout: "last_epoch_started": 0, 2024-03-11T21:05:42.695 INFO:teuthology.orchestra.run.smithi023.stdout: "last_epoch_clean": 0, 2024-03-11T21:05:42.695 INFO:teuthology.orchestra.run.smithi023.stdout: "source_version": "0'0", 2024-03-11T21:05:42.695 INFO:teuthology.orchestra.run.smithi023.stdout: "target_version": "0'0" 2024-03-11T21:05:42.695 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:05:42.695 INFO:teuthology.orchestra.run.smithi023.stdout: "last_change": "84", 2024-03-11T21:05:42.695 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend": "0", 2024-03-11T21:05:42.695 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T21:05:42.695 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T21:05:42.695 INFO:teuthology.orchestra.run.smithi023.stdout: "auid": 0, 2024-03-11T21:05:42.695 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_mode": "selfmanaged", 2024-03-11T21:05:42.696 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_seq": 0, 2024-03-11T21:05:42.696 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_epoch": 0, 2024-03-11T21:05:42.696 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_snaps": [], 2024-03-11T21:05:42.696 INFO:teuthology.orchestra.run.smithi023.stdout: "removed_snaps": "[]", 2024-03-11T21:05:42.696 INFO:teuthology.orchestra.run.smithi023.stdout: "quota_max_bytes": 0, 2024-03-11T21:05:42.696 INFO:teuthology.orchestra.run.smithi023.stdout: "quota_max_objects": 0, 2024-03-11T21:05:42.696 INFO:teuthology.orchestra.run.smithi023.stdout: "tiers": [], 2024-03-11T21:05:42.696 INFO:teuthology.orchestra.run.smithi023.stdout: "tier_of": -1, 2024-03-11T21:05:42.696 INFO:teuthology.orchestra.run.smithi023.stdout: "read_tier": -1, 2024-03-11T21:05:42.696 INFO:teuthology.orchestra.run.smithi023.stdout: "write_tier": -1, 2024-03-11T21:05:42.696 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_mode": "none", 2024-03-11T21:05:42.696 INFO:teuthology.orchestra.run.smithi023.stdout: "target_max_bytes": 0, 2024-03-11T21:05:42.696 INFO:teuthology.orchestra.run.smithi023.stdout: "target_max_objects": 0, 2024-03-11T21:05:42.696 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T21:05:42.697 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T21:05:42.697 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T21:05:42.697 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_min_flush_age": 0, 2024-03-11T21:05:42.697 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_min_evict_age": 0, 2024-03-11T21:05:42.697 INFO:teuthology.orchestra.run.smithi023.stdout: "erasure_code_profile": "", 2024-03-11T21:05:42.697 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_params": { 2024-03-11T21:05:42.697 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "none" 2024-03-11T21:05:42.697 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:05:42.697 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_period": 0, 2024-03-11T21:05:42.697 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_count": 0, 2024-03-11T21:05:42.697 INFO:teuthology.orchestra.run.smithi023.stdout: "use_gmt_hitset": true, 2024-03-11T21:05:42.697 INFO:teuthology.orchestra.run.smithi023.stdout: "min_read_recency_for_promote": 0, 2024-03-11T21:05:42.697 INFO:teuthology.orchestra.run.smithi023.stdout: "min_write_recency_for_promote": 0, 2024-03-11T21:05:42.697 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T21:05:42.698 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_search_last_n": 0, 2024-03-11T21:05:42.698 INFO:teuthology.orchestra.run.smithi023.stdout: "grade_table": [], 2024-03-11T21:05:42.698 INFO:teuthology.orchestra.run.smithi023.stdout: "stripe_width": 0, 2024-03-11T21:05:42.698 INFO:teuthology.orchestra.run.smithi023.stdout: "expected_num_objects": 0, 2024-03-11T21:05:42.698 INFO:teuthology.orchestra.run.smithi023.stdout: "fast_read": false, 2024-03-11T21:05:42.698 INFO:teuthology.orchestra.run.smithi023.stdout: "options": { 2024-03-11T21:05:42.698 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_autoscale_bias": 4 2024-03-11T21:05:42.698 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:05:42.698 INFO:teuthology.orchestra.run.smithi023.stdout: "application_metadata": { 2024-03-11T21:05:42.698 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw": {} 2024-03-11T21:05:42.698 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:05:42.698 INFO:teuthology.orchestra.run.smithi023.stdout: "read_balance": { 2024-03-11T21:05:42.698 INFO:teuthology.orchestra.run.smithi023.stdout: "score_type": "Fair distribution", 2024-03-11T21:05:42.698 INFO:teuthology.orchestra.run.smithi023.stdout: "score_acting": 1.9700000286102295, 2024-03-11T21:05:42.699 INFO:teuthology.orchestra.run.smithi023.stdout: "score_stable": 1.9700000286102295, 2024-03-11T21:05:42.699 INFO:teuthology.orchestra.run.smithi023.stdout: "optimal_score": 1.1399999856948853, 2024-03-11T21:05:42.699 INFO:teuthology.orchestra.run.smithi023.stdout: "raw_score_acting": 2.25, 2024-03-11T21:05:42.699 INFO:teuthology.orchestra.run.smithi023.stdout: "raw_score_stable": 2.25, 2024-03-11T21:05:42.699 INFO:teuthology.orchestra.run.smithi023.stdout: "primary_affinity_weighted": 0.80015754699707031, 2024-03-11T21:05:42.699 INFO:teuthology.orchestra.run.smithi023.stdout: "average_primary_affinity": 0.9100000262260437, 2024-03-11T21:05:42.699 INFO:teuthology.orchestra.run.smithi023.stdout: "average_primary_affinity_weighted": 0.80000001192092896 2024-03-11T21:05:42.699 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:05:42.699 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:05:42.699 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:05:42.699 INFO:teuthology.orchestra.run.smithi023.stdout: "pool": 9, 2024-03-11T21:05:42.699 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_name": "unique_pool_2", 2024-03-11T21:05:42.699 INFO:teuthology.orchestra.run.smithi023.stdout: "create_time": "2024-03-11T21:02:46.162698+0000", 2024-03-11T21:05:42.699 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-11T21:05:42.700 INFO:teuthology.orchestra.run.smithi023.stdout: "flags_names": "hashpspool", 2024-03-11T21:05:42.700 INFO:teuthology.orchestra.run.smithi023.stdout: "type": 1, 2024-03-11T21:05:42.700 INFO:teuthology.orchestra.run.smithi023.stdout: "size": 3, 2024-03-11T21:05:42.700 INFO:teuthology.orchestra.run.smithi023.stdout: "min_size": 2, 2024-03-11T21:05:42.700 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_rule": 0, 2024-03-11T21:05:42.700 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_count": 0, 2024-03-11T21:05:42.700 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_target": 0, 2024-03-11T21:05:42.700 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T21:05:42.700 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T21:05:42.700 INFO:teuthology.orchestra.run.smithi023.stdout: "object_hash": 2, 2024-03-11T21:05:42.700 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_autoscale_mode": "off", 2024-03-11T21:05:42.700 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num": 16, 2024-03-11T21:05:42.700 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_placement_num": 16, 2024-03-11T21:05:42.700 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_placement_num_target": 16, 2024-03-11T21:05:42.700 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num_target": 16, 2024-03-11T21:05:42.701 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num_pending": 16, 2024-03-11T21:05:42.701 INFO:teuthology.orchestra.run.smithi023.stdout: "last_pg_merge_meta": { 2024-03-11T21:05:42.701 INFO:teuthology.orchestra.run.smithi023.stdout: "source_pgid": "0.0", 2024-03-11T21:05:42.701 INFO:teuthology.orchestra.run.smithi023.stdout: "ready_epoch": 0, 2024-03-11T21:05:42.701 INFO:teuthology.orchestra.run.smithi023.stdout: "last_epoch_started": 0, 2024-03-11T21:05:42.701 INFO:teuthology.orchestra.run.smithi023.stdout: "last_epoch_clean": 0, 2024-03-11T21:05:42.701 INFO:teuthology.orchestra.run.smithi023.stdout: "source_version": "0'0", 2024-03-11T21:05:42.701 INFO:teuthology.orchestra.run.smithi023.stdout: "target_version": "0'0" 2024-03-11T21:05:42.701 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:05:42.701 INFO:teuthology.orchestra.run.smithi023.stdout: "last_change": "256", 2024-03-11T21:05:42.701 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend": "0", 2024-03-11T21:05:42.701 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T21:05:42.701 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T21:05:42.701 INFO:teuthology.orchestra.run.smithi023.stdout: "auid": 0, 2024-03-11T21:05:42.702 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_mode": "selfmanaged", 2024-03-11T21:05:42.702 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_seq": 0, 2024-03-11T21:05:42.702 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_epoch": 0, 2024-03-11T21:05:42.702 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_snaps": [], 2024-03-11T21:05:42.702 INFO:teuthology.orchestra.run.smithi023.stdout: "removed_snaps": "[]", 2024-03-11T21:05:42.702 INFO:teuthology.orchestra.run.smithi023.stdout: "quota_max_bytes": 0, 2024-03-11T21:05:42.702 INFO:teuthology.orchestra.run.smithi023.stdout: "quota_max_objects": 0, 2024-03-11T21:05:42.702 INFO:teuthology.orchestra.run.smithi023.stdout: "tiers": [], 2024-03-11T21:05:42.702 INFO:teuthology.orchestra.run.smithi023.stdout: "tier_of": -1, 2024-03-11T21:05:42.702 INFO:teuthology.orchestra.run.smithi023.stdout: "read_tier": -1, 2024-03-11T21:05:42.702 INFO:teuthology.orchestra.run.smithi023.stdout: "write_tier": -1, 2024-03-11T21:05:42.702 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_mode": "none", 2024-03-11T21:05:42.702 INFO:teuthology.orchestra.run.smithi023.stdout: "target_max_bytes": 0, 2024-03-11T21:05:42.702 INFO:teuthology.orchestra.run.smithi023.stdout: "target_max_objects": 0, 2024-03-11T21:05:42.703 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T21:05:42.703 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T21:05:42.703 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T21:05:42.703 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_min_flush_age": 0, 2024-03-11T21:05:42.703 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_min_evict_age": 0, 2024-03-11T21:05:42.703 INFO:teuthology.orchestra.run.smithi023.stdout: "erasure_code_profile": "", 2024-03-11T21:05:42.703 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_params": { 2024-03-11T21:05:42.703 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "none" 2024-03-11T21:05:42.703 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:05:42.703 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_period": 0, 2024-03-11T21:05:42.703 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_count": 0, 2024-03-11T21:05:42.703 INFO:teuthology.orchestra.run.smithi023.stdout: "use_gmt_hitset": true, 2024-03-11T21:05:42.703 INFO:teuthology.orchestra.run.smithi023.stdout: "min_read_recency_for_promote": 0, 2024-03-11T21:05:42.703 INFO:teuthology.orchestra.run.smithi023.stdout: "min_write_recency_for_promote": 0, 2024-03-11T21:05:42.703 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T21:05:42.704 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_search_last_n": 0, 2024-03-11T21:05:42.704 INFO:teuthology.orchestra.run.smithi023.stdout: "grade_table": [], 2024-03-11T21:05:42.704 INFO:teuthology.orchestra.run.smithi023.stdout: "stripe_width": 0, 2024-03-11T21:05:42.704 INFO:teuthology.orchestra.run.smithi023.stdout: "expected_num_objects": 0, 2024-03-11T21:05:42.704 INFO:teuthology.orchestra.run.smithi023.stdout: "fast_read": false, 2024-03-11T21:05:42.704 INFO:teuthology.orchestra.run.smithi023.stdout: "options": {}, 2024-03-11T21:05:42.704 INFO:teuthology.orchestra.run.smithi023.stdout: "application_metadata": { 2024-03-11T21:05:42.704 INFO:teuthology.orchestra.run.smithi023.stdout: "rados": {} 2024-03-11T21:05:42.704 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:05:42.704 INFO:teuthology.orchestra.run.smithi023.stdout: "read_balance": { 2024-03-11T21:05:42.704 INFO:teuthology.orchestra.run.smithi023.stdout: "score_type": "Fair distribution", 2024-03-11T21:05:42.704 INFO:teuthology.orchestra.run.smithi023.stdout: "score_acting": 2.190000057220459, 2024-03-11T21:05:42.704 INFO:teuthology.orchestra.run.smithi023.stdout: "score_stable": 2.190000057220459, 2024-03-11T21:05:42.704 INFO:teuthology.orchestra.run.smithi023.stdout: "optimal_score": 1.1399999856948853, 2024-03-11T21:05:42.704 INFO:teuthology.orchestra.run.smithi023.stdout: "raw_score_acting": 2.5, 2024-03-11T21:05:42.705 INFO:teuthology.orchestra.run.smithi023.stdout: "raw_score_stable": 2.5, 2024-03-11T21:05:42.705 INFO:teuthology.orchestra.run.smithi023.stdout: "primary_affinity_weighted": 0.80015754699707031, 2024-03-11T21:05:42.705 INFO:teuthology.orchestra.run.smithi023.stdout: "average_primary_affinity": 0.9100000262260437, 2024-03-11T21:05:42.705 INFO:teuthology.orchestra.run.smithi023.stdout: "average_primary_affinity_weighted": 0.80000001192092896 2024-03-11T21:05:42.705 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:05:42.705 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:05:42.705 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-11T21:05:42.705 INFO:teuthology.orchestra.run.smithi023.stdout: "osds": [ 2024-03-11T21:05:42.705 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:05:42.705 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": 0, 2024-03-11T21:05:42.705 INFO:teuthology.orchestra.run.smithi023.stdout: "uuid": "3df72b4a-8fe4-4265-b4c6-af022e1e23e1", 2024-03-11T21:05:42.705 INFO:teuthology.orchestra.run.smithi023.stdout: "up": 1, 2024-03-11T21:05:42.705 INFO:teuthology.orchestra.run.smithi023.stdout: "in": 1, 2024-03-11T21:05:42.705 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 1, 2024-03-11T21:05:42.706 INFO:teuthology.orchestra.run.smithi023.stdout: "primary_affinity": 0.4012603759765625, 2024-03-11T21:05:42.706 INFO:teuthology.orchestra.run.smithi023.stdout: "last_clean_begin": 0, 2024-03-11T21:05:42.706 INFO:teuthology.orchestra.run.smithi023.stdout: "last_clean_end": 0, 2024-03-11T21:05:42.706 INFO:teuthology.orchestra.run.smithi023.stdout: "up_from": 8, 2024-03-11T21:05:42.706 INFO:teuthology.orchestra.run.smithi023.stdout: "up_thru": 248, 2024-03-11T21:05:42.706 INFO:teuthology.orchestra.run.smithi023.stdout: "down_at": 0, 2024-03-11T21:05:42.706 INFO:teuthology.orchestra.run.smithi023.stdout: "lost_at": 0, 2024-03-11T21:05:42.706 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2024-03-11T21:05:42.706 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:05:42.706 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:05:42.706 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:05:42.706 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6802", 2024-03-11T21:05:42.706 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 2154627393 2024-03-11T21:05:42.706 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:05:42.706 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:05:42.707 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:05:42.707 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6803", 2024-03-11T21:05:42.707 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 2154627393 2024-03-11T21:05:42.707 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:05:42.707 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:05:42.707 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:05:42.707 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_addrs": { 2024-03-11T21:05:42.707 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:05:42.707 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:05:42.707 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:05:42.707 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6804", 2024-03-11T21:05:42.707 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 2154627393 2024-03-11T21:05:42.707 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:05:42.707 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:05:42.707 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:05:42.708 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6805", 2024-03-11T21:05:42.708 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 2154627393 2024-03-11T21:05:42.708 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:05:42.708 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:05:42.708 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:05:42.708 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_back_addrs": { 2024-03-11T21:05:42.708 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:05:42.708 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:05:42.708 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:05:42.708 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6808", 2024-03-11T21:05:42.708 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 2154627393 2024-03-11T21:05:42.708 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:05:42.708 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:05:42.708 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:05:42.709 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6809", 2024-03-11T21:05:42.709 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 2154627393 2024-03-11T21:05:42.709 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:05:42.709 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:05:42.709 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:05:42.709 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_front_addrs": { 2024-03-11T21:05:42.709 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:05:42.709 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:05:42.709 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:05:42.709 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6806", 2024-03-11T21:05:42.709 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 2154627393 2024-03-11T21:05:42.709 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:05:42.709 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:05:42.709 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:05:42.710 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6807", 2024-03-11T21:05:42.710 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 2154627393 2024-03-11T21:05:42.710 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:05:42.710 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:05:42.710 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:05:42.710 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6803/2154627393", 2024-03-11T21:05:42.710 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_addr": "172.21.15.23:6805/2154627393", 2024-03-11T21:05:42.710 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_back_addr": "172.21.15.23:6809/2154627393", 2024-03-11T21:05:42.710 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_front_addr": "172.21.15.23:6807/2154627393", 2024-03-11T21:05:42.710 INFO:teuthology.orchestra.run.smithi023.stdout: "state": [ 2024-03-11T21:05:42.710 INFO:teuthology.orchestra.run.smithi023.stdout: "exists", 2024-03-11T21:05:42.710 INFO:teuthology.orchestra.run.smithi023.stdout: "up" 2024-03-11T21:05:42.710 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:05:42.710 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:05:42.711 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:05:42.711 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": 1, 2024-03-11T21:05:42.711 INFO:teuthology.orchestra.run.smithi023.stdout: "uuid": "e8fd7500-9081-41a1-8077-a4d0f0e12a2c", 2024-03-11T21:05:42.711 INFO:teuthology.orchestra.run.smithi023.stdout: "up": 1, 2024-03-11T21:05:42.711 INFO:teuthology.orchestra.run.smithi023.stdout: "in": 1, 2024-03-11T21:05:42.711 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 1, 2024-03-11T21:05:42.711 INFO:teuthology.orchestra.run.smithi023.stdout: "primary_affinity": 1, 2024-03-11T21:05:42.711 INFO:teuthology.orchestra.run.smithi023.stdout: "last_clean_begin": 13, 2024-03-11T21:05:42.711 INFO:teuthology.orchestra.run.smithi023.stdout: "last_clean_end": 173, 2024-03-11T21:05:42.711 INFO:teuthology.orchestra.run.smithi023.stdout: "up_from": 211, 2024-03-11T21:05:42.711 INFO:teuthology.orchestra.run.smithi023.stdout: "up_thru": 285, 2024-03-11T21:05:42.711 INFO:teuthology.orchestra.run.smithi023.stdout: "down_at": 174, 2024-03-11T21:05:42.711 INFO:teuthology.orchestra.run.smithi023.stdout: "lost_at": 0, 2024-03-11T21:05:42.711 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2024-03-11T21:05:42.712 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:05:42.712 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:05:42.712 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:05:42.712 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6810", 2024-03-11T21:05:42.712 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 56585354 2024-03-11T21:05:42.712 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:05:42.712 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:05:42.712 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:05:42.712 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6811", 2024-03-11T21:05:42.712 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 56585354 2024-03-11T21:05:42.712 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:05:42.712 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:05:42.712 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:05:42.712 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_addrs": { 2024-03-11T21:05:42.713 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:05:42.713 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:05:42.713 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:05:42.713 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6812", 2024-03-11T21:05:42.713 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 56585354 2024-03-11T21:05:42.713 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:05:42.713 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:05:42.713 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:05:42.713 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6813", 2024-03-11T21:05:42.713 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 56585354 2024-03-11T21:05:42.713 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:05:42.713 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:05:42.713 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:05:42.713 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_back_addrs": { 2024-03-11T21:05:42.714 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:05:42.714 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:05:42.714 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:05:42.714 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6816", 2024-03-11T21:05:42.714 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 56585354 2024-03-11T21:05:42.714 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:05:42.714 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:05:42.714 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:05:42.714 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6817", 2024-03-11T21:05:42.714 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 56585354 2024-03-11T21:05:42.714 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:05:42.714 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:05:42.714 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:05:42.715 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_front_addrs": { 2024-03-11T21:05:42.715 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:05:42.715 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:05:42.715 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:05:42.715 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6814", 2024-03-11T21:05:42.715 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 56585354 2024-03-11T21:05:42.715 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:05:42.715 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:05:42.715 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:05:42.715 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6815", 2024-03-11T21:05:42.715 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 56585354 2024-03-11T21:05:42.715 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:05:42.715 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:05:42.715 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:05:42.716 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6811/56585354", 2024-03-11T21:05:42.716 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_addr": "172.21.15.23:6813/56585354", 2024-03-11T21:05:42.716 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_back_addr": "172.21.15.23:6817/56585354", 2024-03-11T21:05:42.716 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_front_addr": "172.21.15.23:6815/56585354", 2024-03-11T21:05:42.716 INFO:teuthology.orchestra.run.smithi023.stdout: "state": [ 2024-03-11T21:05:42.716 INFO:teuthology.orchestra.run.smithi023.stdout: "exists", 2024-03-11T21:05:42.716 INFO:teuthology.orchestra.run.smithi023.stdout: "up" 2024-03-11T21:05:42.716 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:05:42.716 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:05:42.716 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:05:42.716 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": 2, 2024-03-11T21:05:42.716 INFO:teuthology.orchestra.run.smithi023.stdout: "uuid": "dd85d52a-df48-4b03-8612-cecfd82b5987", 2024-03-11T21:05:42.716 INFO:teuthology.orchestra.run.smithi023.stdout: "up": 1, 2024-03-11T21:05:42.716 INFO:teuthology.orchestra.run.smithi023.stdout: "in": 1, 2024-03-11T21:05:42.717 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 1, 2024-03-11T21:05:42.717 INFO:teuthology.orchestra.run.smithi023.stdout: "primary_affinity": 1, 2024-03-11T21:05:42.717 INFO:teuthology.orchestra.run.smithi023.stdout: "last_clean_begin": 0, 2024-03-11T21:05:42.717 INFO:teuthology.orchestra.run.smithi023.stdout: "last_clean_end": 0, 2024-03-11T21:05:42.717 INFO:teuthology.orchestra.run.smithi023.stdout: "up_from": 19, 2024-03-11T21:05:42.717 INFO:teuthology.orchestra.run.smithi023.stdout: "up_thru": 285, 2024-03-11T21:05:42.717 INFO:teuthology.orchestra.run.smithi023.stdout: "down_at": 0, 2024-03-11T21:05:42.717 INFO:teuthology.orchestra.run.smithi023.stdout: "lost_at": 0, 2024-03-11T21:05:42.717 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2024-03-11T21:05:42.717 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:05:42.717 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:05:42.717 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:05:42.717 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6818", 2024-03-11T21:05:42.718 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1770748056 2024-03-11T21:05:42.718 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:05:42.718 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:05:42.718 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:05:42.718 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6819", 2024-03-11T21:05:42.718 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1770748056 2024-03-11T21:05:42.718 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:05:42.718 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:05:42.718 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:05:42.718 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_addrs": { 2024-03-11T21:05:42.718 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:05:42.718 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:05:42.718 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:05:42.718 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6820", 2024-03-11T21:05:42.719 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1770748056 2024-03-11T21:05:42.719 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:05:42.719 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:05:42.719 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:05:42.719 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6821", 2024-03-11T21:05:42.719 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1770748056 2024-03-11T21:05:42.719 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:05:42.719 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:05:42.719 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:05:42.719 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_back_addrs": { 2024-03-11T21:05:42.719 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:05:42.719 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:05:42.719 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:05:42.719 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6824", 2024-03-11T21:05:42.719 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1770748056 2024-03-11T21:05:42.720 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:05:42.720 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:05:42.720 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:05:42.720 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6825", 2024-03-11T21:05:42.720 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1770748056 2024-03-11T21:05:42.720 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:05:42.720 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:05:42.720 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:05:42.720 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_front_addrs": { 2024-03-11T21:05:42.720 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:05:42.720 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:05:42.720 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:05:42.720 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6822", 2024-03-11T21:05:42.720 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1770748056 2024-03-11T21:05:42.720 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:05:42.720 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:05:42.721 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:05:42.721 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6823", 2024-03-11T21:05:42.721 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1770748056 2024-03-11T21:05:42.721 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:05:42.721 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:05:42.721 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:05:42.721 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6819/1770748056", 2024-03-11T21:05:42.721 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_addr": "172.21.15.23:6821/1770748056", 2024-03-11T21:05:42.721 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_back_addr": "172.21.15.23:6825/1770748056", 2024-03-11T21:05:42.721 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_front_addr": "172.21.15.23:6823/1770748056", 2024-03-11T21:05:42.721 INFO:teuthology.orchestra.run.smithi023.stdout: "state": [ 2024-03-11T21:05:42.721 INFO:teuthology.orchestra.run.smithi023.stdout: "exists", 2024-03-11T21:05:42.721 INFO:teuthology.orchestra.run.smithi023.stdout: "up" 2024-03-11T21:05:42.721 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:05:42.721 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:05:42.721 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:05:42.722 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": 3, 2024-03-11T21:05:42.722 INFO:teuthology.orchestra.run.smithi023.stdout: "uuid": "e3021106-3779-4b94-9bbf-a7724544110f", 2024-03-11T21:05:42.722 INFO:teuthology.orchestra.run.smithi023.stdout: "up": 1, 2024-03-11T21:05:42.722 INFO:teuthology.orchestra.run.smithi023.stdout: "in": 1, 2024-03-11T21:05:42.722 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 1, 2024-03-11T21:05:42.722 INFO:teuthology.orchestra.run.smithi023.stdout: "primary_affinity": 1, 2024-03-11T21:05:42.722 INFO:teuthology.orchestra.run.smithi023.stdout: "last_clean_begin": 0, 2024-03-11T21:05:42.722 INFO:teuthology.orchestra.run.smithi023.stdout: "last_clean_end": 0, 2024-03-11T21:05:42.722 INFO:teuthology.orchestra.run.smithi023.stdout: "up_from": 28, 2024-03-11T21:05:42.722 INFO:teuthology.orchestra.run.smithi023.stdout: "up_thru": 285, 2024-03-11T21:05:42.722 INFO:teuthology.orchestra.run.smithi023.stdout: "down_at": 0, 2024-03-11T21:05:42.722 INFO:teuthology.orchestra.run.smithi023.stdout: "lost_at": 0, 2024-03-11T21:05:42.722 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2024-03-11T21:05:42.722 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:05:42.722 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:05:42.723 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:05:42.723 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6826", 2024-03-11T21:05:42.723 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 938496814 2024-03-11T21:05:42.723 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:05:42.723 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:05:42.723 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:05:42.723 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6827", 2024-03-11T21:05:42.723 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 938496814 2024-03-11T21:05:42.723 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:05:42.723 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:05:42.723 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:05:42.723 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_addrs": { 2024-03-11T21:05:42.723 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:05:42.723 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:05:42.723 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:05:42.724 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6828", 2024-03-11T21:05:42.724 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 938496814 2024-03-11T21:05:42.724 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:05:42.724 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:05:42.724 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:05:42.724 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6829", 2024-03-11T21:05:42.724 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 938496814 2024-03-11T21:05:42.724 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:05:42.724 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:05:42.724 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:05:42.724 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_back_addrs": { 2024-03-11T21:05:42.724 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:05:42.724 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:05:42.724 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:05:42.724 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6832", 2024-03-11T21:05:42.724 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 938496814 2024-03-11T21:05:42.725 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:05:42.725 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:05:42.725 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:05:42.725 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6833", 2024-03-11T21:05:42.725 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 938496814 2024-03-11T21:05:42.725 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:05:42.725 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:05:42.725 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:05:42.725 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_front_addrs": { 2024-03-11T21:05:42.725 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:05:42.725 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:05:42.725 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:05:42.725 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6830", 2024-03-11T21:05:42.725 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 938496814 2024-03-11T21:05:42.726 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:05:42.726 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:05:42.726 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:05:42.726 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6831", 2024-03-11T21:05:42.726 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 938496814 2024-03-11T21:05:42.726 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:05:42.726 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:05:42.726 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:05:42.726 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6827/938496814", 2024-03-11T21:05:42.726 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_addr": "172.21.15.23:6829/938496814", 2024-03-11T21:05:42.726 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_back_addr": "172.21.15.23:6833/938496814", 2024-03-11T21:05:42.726 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_front_addr": "172.21.15.23:6831/938496814", 2024-03-11T21:05:42.726 INFO:teuthology.orchestra.run.smithi023.stdout: "state": [ 2024-03-11T21:05:42.726 INFO:teuthology.orchestra.run.smithi023.stdout: "exists", 2024-03-11T21:05:42.726 INFO:teuthology.orchestra.run.smithi023.stdout: "up" 2024-03-11T21:05:42.727 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:05:42.727 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:05:42.727 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:05:42.727 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": 4, 2024-03-11T21:05:42.727 INFO:teuthology.orchestra.run.smithi023.stdout: "uuid": "4dcbc91e-7035-413d-97bc-c9d804eb16d7", 2024-03-11T21:05:42.727 INFO:teuthology.orchestra.run.smithi023.stdout: "up": 1, 2024-03-11T21:05:42.727 INFO:teuthology.orchestra.run.smithi023.stdout: "in": 1, 2024-03-11T21:05:42.727 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 1, 2024-03-11T21:05:42.727 INFO:teuthology.orchestra.run.smithi023.stdout: "primary_affinity": 0, 2024-03-11T21:05:42.727 INFO:teuthology.orchestra.run.smithi023.stdout: "last_clean_begin": 35, 2024-03-11T21:05:42.727 INFO:teuthology.orchestra.run.smithi023.stdout: "last_clean_end": 182, 2024-03-11T21:05:42.727 INFO:teuthology.orchestra.run.smithi023.stdout: "up_from": 204, 2024-03-11T21:05:42.727 INFO:teuthology.orchestra.run.smithi023.stdout: "up_thru": 248, 2024-03-11T21:05:42.727 INFO:teuthology.orchestra.run.smithi023.stdout: "down_at": 183, 2024-03-11T21:05:42.727 INFO:teuthology.orchestra.run.smithi023.stdout: "lost_at": 0, 2024-03-11T21:05:42.727 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2024-03-11T21:05:42.728 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:05:42.728 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:05:42.728 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:05:42.728 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6800", 2024-03-11T21:05:42.728 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 3858359434 2024-03-11T21:05:42.728 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:05:42.728 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:05:42.728 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:05:42.728 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6801", 2024-03-11T21:05:42.728 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 3858359434 2024-03-11T21:05:42.728 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:05:42.728 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:05:42.728 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:05:42.728 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_addrs": { 2024-03-11T21:05:42.728 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:05:42.729 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:05:42.729 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:05:42.729 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6802", 2024-03-11T21:05:42.729 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 3858359434 2024-03-11T21:05:42.729 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:05:42.729 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:05:42.729 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:05:42.729 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6803", 2024-03-11T21:05:42.729 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 3858359434 2024-03-11T21:05:42.729 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:05:42.729 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:05:42.729 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:05:42.729 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_back_addrs": { 2024-03-11T21:05:42.729 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:05:42.729 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:05:42.729 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:05:42.730 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6806", 2024-03-11T21:05:42.730 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 3858359434 2024-03-11T21:05:42.730 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:05:42.730 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:05:42.730 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:05:42.730 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6807", 2024-03-11T21:05:42.730 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 3858359434 2024-03-11T21:05:42.730 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:05:42.730 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:05:42.730 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:05:42.730 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_front_addrs": { 2024-03-11T21:05:42.730 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:05:42.730 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:05:42.730 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:05:42.731 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6804", 2024-03-11T21:05:42.731 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 3858359434 2024-03-11T21:05:42.731 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:05:42.731 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:05:42.731 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:05:42.731 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6805", 2024-03-11T21:05:42.731 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 3858359434 2024-03-11T21:05:42.731 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:05:42.731 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:05:42.731 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:05:42.731 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.146:6801/3858359434", 2024-03-11T21:05:42.731 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_addr": "172.21.15.146:6803/3858359434", 2024-03-11T21:05:42.731 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_back_addr": "172.21.15.146:6807/3858359434", 2024-03-11T21:05:42.731 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_front_addr": "172.21.15.146:6805/3858359434", 2024-03-11T21:05:42.732 INFO:teuthology.orchestra.run.smithi023.stdout: "state": [ 2024-03-11T21:05:42.732 INFO:teuthology.orchestra.run.smithi023.stdout: "exists", 2024-03-11T21:05:42.732 INFO:teuthology.orchestra.run.smithi023.stdout: "up" 2024-03-11T21:05:42.732 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:05:42.732 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:05:42.732 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:05:42.732 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": 5, 2024-03-11T21:05:42.732 INFO:teuthology.orchestra.run.smithi023.stdout: "uuid": "aefcb766-8bae-4ff4-b71e-6691c8ac02f1", 2024-03-11T21:05:42.732 INFO:teuthology.orchestra.run.smithi023.stdout: "up": 1, 2024-03-11T21:05:42.732 INFO:teuthology.orchestra.run.smithi023.stdout: "in": 1, 2024-03-11T21:05:42.732 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 1, 2024-03-11T21:05:42.732 INFO:teuthology.orchestra.run.smithi023.stdout: "primary_affinity": 1, 2024-03-11T21:05:42.732 INFO:teuthology.orchestra.run.smithi023.stdout: "last_clean_begin": 0, 2024-03-11T21:05:42.732 INFO:teuthology.orchestra.run.smithi023.stdout: "last_clean_end": 0, 2024-03-11T21:05:42.732 INFO:teuthology.orchestra.run.smithi023.stdout: "up_from": 44, 2024-03-11T21:05:42.733 INFO:teuthology.orchestra.run.smithi023.stdout: "up_thru": 285, 2024-03-11T21:05:42.733 INFO:teuthology.orchestra.run.smithi023.stdout: "down_at": 0, 2024-03-11T21:05:42.733 INFO:teuthology.orchestra.run.smithi023.stdout: "lost_at": 0, 2024-03-11T21:05:42.733 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2024-03-11T21:05:42.733 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:05:42.733 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:05:42.733 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:05:42.733 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6808", 2024-03-11T21:05:42.733 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 4003230814 2024-03-11T21:05:42.733 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:05:42.733 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:05:42.733 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:05:42.733 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6809", 2024-03-11T21:05:42.733 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 4003230814 2024-03-11T21:05:42.734 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:05:42.734 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:05:42.734 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:05:42.734 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_addrs": { 2024-03-11T21:05:42.734 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:05:42.734 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:05:42.734 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:05:42.734 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6810", 2024-03-11T21:05:42.734 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 4003230814 2024-03-11T21:05:42.734 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:05:42.734 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:05:42.734 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:05:42.734 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6811", 2024-03-11T21:05:42.734 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 4003230814 2024-03-11T21:05:42.735 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:05:42.735 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:05:42.735 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:05:42.735 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_back_addrs": { 2024-03-11T21:05:42.735 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:05:42.735 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:05:42.735 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:05:42.735 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6814", 2024-03-11T21:05:42.735 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 4003230814 2024-03-11T21:05:42.735 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:05:42.735 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:05:42.735 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:05:42.735 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6815", 2024-03-11T21:05:42.735 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 4003230814 2024-03-11T21:05:42.736 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:05:42.736 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:05:42.736 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:05:42.736 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_front_addrs": { 2024-03-11T21:05:42.736 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:05:42.736 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:05:42.736 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:05:42.736 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6812", 2024-03-11T21:05:42.736 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 4003230814 2024-03-11T21:05:42.736 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:05:42.736 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:05:42.736 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:05:42.736 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6813", 2024-03-11T21:05:42.737 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 4003230814 2024-03-11T21:05:42.737 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:05:42.737 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:05:42.737 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:05:42.737 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.146:6809/4003230814", 2024-03-11T21:05:42.737 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_addr": "172.21.15.146:6811/4003230814", 2024-03-11T21:05:42.737 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_back_addr": "172.21.15.146:6815/4003230814", 2024-03-11T21:05:42.737 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_front_addr": "172.21.15.146:6813/4003230814", 2024-03-11T21:05:42.737 INFO:teuthology.orchestra.run.smithi023.stdout: "state": [ 2024-03-11T21:05:42.737 INFO:teuthology.orchestra.run.smithi023.stdout: "exists", 2024-03-11T21:05:42.737 INFO:teuthology.orchestra.run.smithi023.stdout: "up" 2024-03-11T21:05:42.737 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:05:42.737 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:05:42.737 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:05:42.738 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": 6, 2024-03-11T21:05:42.738 INFO:teuthology.orchestra.run.smithi023.stdout: "uuid": "5cb151e8-1f9b-4617-8a86-adfb06bb9606", 2024-03-11T21:05:42.738 INFO:teuthology.orchestra.run.smithi023.stdout: "up": 1, 2024-03-11T21:05:42.738 INFO:teuthology.orchestra.run.smithi023.stdout: "in": 1, 2024-03-11T21:05:42.738 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 1, 2024-03-11T21:05:42.738 INFO:teuthology.orchestra.run.smithi023.stdout: "primary_affinity": 1, 2024-03-11T21:05:42.738 INFO:teuthology.orchestra.run.smithi023.stdout: "last_clean_begin": 0, 2024-03-11T21:05:42.738 INFO:teuthology.orchestra.run.smithi023.stdout: "last_clean_end": 0, 2024-03-11T21:05:42.738 INFO:teuthology.orchestra.run.smithi023.stdout: "up_from": 57, 2024-03-11T21:05:42.738 INFO:teuthology.orchestra.run.smithi023.stdout: "up_thru": 285, 2024-03-11T21:05:42.738 INFO:teuthology.orchestra.run.smithi023.stdout: "down_at": 0, 2024-03-11T21:05:42.738 INFO:teuthology.orchestra.run.smithi023.stdout: "lost_at": 0, 2024-03-11T21:05:42.738 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2024-03-11T21:05:42.738 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:05:42.739 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:05:42.739 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:05:42.739 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6816", 2024-03-11T21:05:42.739 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 3334092376 2024-03-11T21:05:42.739 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:05:42.739 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:05:42.739 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:05:42.739 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6817", 2024-03-11T21:05:42.739 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 3334092376 2024-03-11T21:05:42.739 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:05:42.739 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:05:42.739 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:05:42.739 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_addrs": { 2024-03-11T21:05:42.739 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:05:42.739 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:05:42.739 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:05:42.740 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6818", 2024-03-11T21:05:42.740 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 3334092376 2024-03-11T21:05:42.740 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:05:42.740 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:05:42.740 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:05:42.740 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6819", 2024-03-11T21:05:42.740 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 3334092376 2024-03-11T21:05:42.740 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:05:42.740 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:05:42.740 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:05:42.740 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_back_addrs": { 2024-03-11T21:05:42.740 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:05:42.740 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:05:42.740 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:05:42.740 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6822", 2024-03-11T21:05:42.741 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 3334092376 2024-03-11T21:05:42.741 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:05:42.741 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:05:42.741 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:05:42.741 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6823", 2024-03-11T21:05:42.741 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 3334092376 2024-03-11T21:05:42.741 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:05:42.741 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:05:42.741 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:05:42.741 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_front_addrs": { 2024-03-11T21:05:42.741 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:05:42.741 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:05:42.741 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:05:42.741 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6820", 2024-03-11T21:05:42.741 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 3334092376 2024-03-11T21:05:42.742 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:05:42.742 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:05:42.742 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:05:42.742 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6821", 2024-03-11T21:05:42.742 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 3334092376 2024-03-11T21:05:42.742 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:05:42.742 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:05:42.742 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:05:42.742 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.146:6817/3334092376", 2024-03-11T21:05:42.742 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_addr": "172.21.15.146:6819/3334092376", 2024-03-11T21:05:42.742 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_back_addr": "172.21.15.146:6823/3334092376", 2024-03-11T21:05:42.742 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_front_addr": "172.21.15.146:6821/3334092376", 2024-03-11T21:05:42.742 INFO:teuthology.orchestra.run.smithi023.stdout: "state": [ 2024-03-11T21:05:42.743 INFO:teuthology.orchestra.run.smithi023.stdout: "exists", 2024-03-11T21:05:42.743 INFO:teuthology.orchestra.run.smithi023.stdout: "up" 2024-03-11T21:05:42.743 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:05:42.743 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:05:42.743 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:05:42.743 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": 7, 2024-03-11T21:05:42.743 INFO:teuthology.orchestra.run.smithi023.stdout: "uuid": "bdbd11a2-fa09-4f82-93a2-6b2d9c750bfd", 2024-03-11T21:05:42.743 INFO:teuthology.orchestra.run.smithi023.stdout: "up": 1, 2024-03-11T21:05:42.743 INFO:teuthology.orchestra.run.smithi023.stdout: "in": 1, 2024-03-11T21:05:42.743 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 1, 2024-03-11T21:05:42.743 INFO:teuthology.orchestra.run.smithi023.stdout: "primary_affinity": 1, 2024-03-11T21:05:42.743 INFO:teuthology.orchestra.run.smithi023.stdout: "last_clean_begin": 0, 2024-03-11T21:05:42.743 INFO:teuthology.orchestra.run.smithi023.stdout: "last_clean_end": 0, 2024-03-11T21:05:42.743 INFO:teuthology.orchestra.run.smithi023.stdout: "up_from": 68, 2024-03-11T21:05:42.743 INFO:teuthology.orchestra.run.smithi023.stdout: "up_thru": 285, 2024-03-11T21:05:42.744 INFO:teuthology.orchestra.run.smithi023.stdout: "down_at": 0, 2024-03-11T21:05:42.744 INFO:teuthology.orchestra.run.smithi023.stdout: "lost_at": 0, 2024-03-11T21:05:42.744 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2024-03-11T21:05:42.744 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:05:42.744 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:05:42.744 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:05:42.744 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6824", 2024-03-11T21:05:42.744 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1678407923 2024-03-11T21:05:42.744 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:05:42.744 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:05:42.744 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:05:42.744 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6825", 2024-03-11T21:05:42.744 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1678407923 2024-03-11T21:05:42.744 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:05:42.745 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:05:42.745 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:05:42.745 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_addrs": { 2024-03-11T21:05:42.745 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:05:42.745 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:05:42.745 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:05:42.745 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6826", 2024-03-11T21:05:42.745 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1678407923 2024-03-11T21:05:42.745 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:05:42.745 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:05:42.745 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:05:42.745 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6827", 2024-03-11T21:05:42.745 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1678407923 2024-03-11T21:05:42.745 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:05:42.746 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:05:42.746 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:05:42.746 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_back_addrs": { 2024-03-11T21:05:42.746 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:05:42.746 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:05:42.746 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:05:42.746 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6830", 2024-03-11T21:05:42.746 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1678407923 2024-03-11T21:05:42.746 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:05:42.746 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:05:42.746 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:05:42.746 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6831", 2024-03-11T21:05:42.746 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1678407923 2024-03-11T21:05:42.746 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:05:42.747 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:05:42.747 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:05:42.747 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_front_addrs": { 2024-03-11T21:05:42.747 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:05:42.747 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:05:42.747 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:05:42.747 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6828", 2024-03-11T21:05:42.747 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1678407923 2024-03-11T21:05:42.747 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:05:42.747 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:05:42.747 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:05:42.747 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6829", 2024-03-11T21:05:42.747 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1678407923 2024-03-11T21:05:42.747 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:05:42.747 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:05:42.747 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:05:42.748 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.146:6825/1678407923", 2024-03-11T21:05:42.748 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_addr": "172.21.15.146:6827/1678407923", 2024-03-11T21:05:42.748 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_back_addr": "172.21.15.146:6831/1678407923", 2024-03-11T21:05:42.748 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_front_addr": "172.21.15.146:6829/1678407923", 2024-03-11T21:05:42.748 INFO:teuthology.orchestra.run.smithi023.stdout: "state": [ 2024-03-11T21:05:42.748 INFO:teuthology.orchestra.run.smithi023.stdout: "exists", 2024-03-11T21:05:42.748 INFO:teuthology.orchestra.run.smithi023.stdout: "up" 2024-03-11T21:05:42.748 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:05:42.748 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:05:42.748 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-11T21:05:42.748 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_xinfo": [ 2024-03-11T21:05:42.748 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:05:42.748 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": 0, 2024-03-11T21:05:42.748 INFO:teuthology.orchestra.run.smithi023.stdout: "down_stamp": "0.000000", 2024-03-11T21:05:42.749 INFO:teuthology.orchestra.run.smithi023.stdout: "laggy_probability": 0, 2024-03-11T21:05:42.749 INFO:teuthology.orchestra.run.smithi023.stdout: "laggy_interval": 0, 2024-03-11T21:05:42.749 INFO:teuthology.orchestra.run.smithi023.stdout: "features": 4540701547738038271, 2024-03-11T21:05:42.749 INFO:teuthology.orchestra.run.smithi023.stdout: "old_weight": 0, 2024-03-11T21:05:42.749 INFO:teuthology.orchestra.run.smithi023.stdout: "last_purged_snaps_scrub": "2024-03-11T21:05:33.549850+0000", 2024-03-11T21:05:42.749 INFO:teuthology.orchestra.run.smithi023.stdout: "dead_epoch": 0 2024-03-11T21:05:42.749 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:05:42.749 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:05:42.749 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": 1, 2024-03-11T21:05:42.749 INFO:teuthology.orchestra.run.smithi023.stdout: "down_stamp": "0.000000", 2024-03-11T21:05:42.749 INFO:teuthology.orchestra.run.smithi023.stdout: "laggy_probability": 0, 2024-03-11T21:05:42.749 INFO:teuthology.orchestra.run.smithi023.stdout: "laggy_interval": 0, 2024-03-11T21:05:42.749 INFO:teuthology.orchestra.run.smithi023.stdout: "features": 4540701547738038271, 2024-03-11T21:05:42.749 INFO:teuthology.orchestra.run.smithi023.stdout: "old_weight": 0, 2024-03-11T21:05:42.749 INFO:teuthology.orchestra.run.smithi023.stdout: "last_purged_snaps_scrub": "2024-03-11T21:04:51.062946+0000", 2024-03-11T21:05:42.750 INFO:teuthology.orchestra.run.smithi023.stdout: "dead_epoch": 173 2024-03-11T21:05:42.750 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:05:42.750 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:05:42.750 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": 2, 2024-03-11T21:05:42.750 INFO:teuthology.orchestra.run.smithi023.stdout: "down_stamp": "0.000000", 2024-03-11T21:05:42.750 INFO:teuthology.orchestra.run.smithi023.stdout: "laggy_probability": 0, 2024-03-11T21:05:42.750 INFO:teuthology.orchestra.run.smithi023.stdout: "laggy_interval": 0, 2024-03-11T21:05:42.750 INFO:teuthology.orchestra.run.smithi023.stdout: "features": 4540701547738038271, 2024-03-11T21:05:42.750 INFO:teuthology.orchestra.run.smithi023.stdout: "old_weight": 0, 2024-03-11T21:05:42.750 INFO:teuthology.orchestra.run.smithi023.stdout: "last_purged_snaps_scrub": "2024-03-11T21:04:48.914230+0000", 2024-03-11T21:05:42.750 INFO:teuthology.orchestra.run.smithi023.stdout: "dead_epoch": 0 2024-03-11T21:05:42.750 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:05:42.750 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:05:42.750 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": 3, 2024-03-11T21:05:42.751 INFO:teuthology.orchestra.run.smithi023.stdout: "down_stamp": "0.000000", 2024-03-11T21:05:42.751 INFO:teuthology.orchestra.run.smithi023.stdout: "laggy_probability": 0, 2024-03-11T21:05:42.751 INFO:teuthology.orchestra.run.smithi023.stdout: "laggy_interval": 0, 2024-03-11T21:05:42.751 INFO:teuthology.orchestra.run.smithi023.stdout: "features": 4540701547738038271, 2024-03-11T21:05:42.751 INFO:teuthology.orchestra.run.smithi023.stdout: "old_weight": 0, 2024-03-11T21:05:42.751 INFO:teuthology.orchestra.run.smithi023.stdout: "last_purged_snaps_scrub": "2024-03-11T21:04:48.498387+0000", 2024-03-11T21:05:42.751 INFO:teuthology.orchestra.run.smithi023.stdout: "dead_epoch": 0 2024-03-11T21:05:42.751 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:05:42.751 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:05:42.751 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": 4, 2024-03-11T21:05:42.751 INFO:teuthology.orchestra.run.smithi023.stdout: "down_stamp": "0.000000", 2024-03-11T21:05:42.751 INFO:teuthology.orchestra.run.smithi023.stdout: "laggy_probability": 0, 2024-03-11T21:05:42.751 INFO:teuthology.orchestra.run.smithi023.stdout: "laggy_interval": 0, 2024-03-11T21:05:42.751 INFO:teuthology.orchestra.run.smithi023.stdout: "features": 4540701547738038271, 2024-03-11T21:05:42.751 INFO:teuthology.orchestra.run.smithi023.stdout: "old_weight": 0, 2024-03-11T21:05:42.752 INFO:teuthology.orchestra.run.smithi023.stdout: "last_purged_snaps_scrub": "2024-03-11T21:04:23.824633+0000", 2024-03-11T21:05:42.752 INFO:teuthology.orchestra.run.smithi023.stdout: "dead_epoch": 182 2024-03-11T21:05:42.752 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:05:42.752 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:05:42.752 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": 5, 2024-03-11T21:05:42.752 INFO:teuthology.orchestra.run.smithi023.stdout: "down_stamp": "0.000000", 2024-03-11T21:05:42.752 INFO:teuthology.orchestra.run.smithi023.stdout: "laggy_probability": 0, 2024-03-11T21:05:42.752 INFO:teuthology.orchestra.run.smithi023.stdout: "laggy_interval": 0, 2024-03-11T21:05:42.752 INFO:teuthology.orchestra.run.smithi023.stdout: "features": 4540701547738038271, 2024-03-11T21:05:42.752 INFO:teuthology.orchestra.run.smithi023.stdout: "old_weight": 0, 2024-03-11T21:05:42.752 INFO:teuthology.orchestra.run.smithi023.stdout: "last_purged_snaps_scrub": "2024-03-11T21:05:33.523843+0000", 2024-03-11T21:05:42.752 INFO:teuthology.orchestra.run.smithi023.stdout: "dead_epoch": 0 2024-03-11T21:05:42.752 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:05:42.752 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:05:42.752 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": 6, 2024-03-11T21:05:42.753 INFO:teuthology.orchestra.run.smithi023.stdout: "down_stamp": "0.000000", 2024-03-11T21:05:42.753 INFO:teuthology.orchestra.run.smithi023.stdout: "laggy_probability": 0, 2024-03-11T21:05:42.753 INFO:teuthology.orchestra.run.smithi023.stdout: "laggy_interval": 0, 2024-03-11T21:05:42.753 INFO:teuthology.orchestra.run.smithi023.stdout: "features": 4540701547738038271, 2024-03-11T21:05:42.753 INFO:teuthology.orchestra.run.smithi023.stdout: "old_weight": 0, 2024-03-11T21:05:42.753 INFO:teuthology.orchestra.run.smithi023.stdout: "last_purged_snaps_scrub": "2024-03-11T21:04:45.871348+0000", 2024-03-11T21:05:42.753 INFO:teuthology.orchestra.run.smithi023.stdout: "dead_epoch": 0 2024-03-11T21:05:42.753 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:05:42.753 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:05:42.753 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": 7, 2024-03-11T21:05:42.753 INFO:teuthology.orchestra.run.smithi023.stdout: "down_stamp": "0.000000", 2024-03-11T21:05:42.753 INFO:teuthology.orchestra.run.smithi023.stdout: "laggy_probability": 0, 2024-03-11T21:05:42.753 INFO:teuthology.orchestra.run.smithi023.stdout: "laggy_interval": 0, 2024-03-11T21:05:42.753 INFO:teuthology.orchestra.run.smithi023.stdout: "features": 4540701547738038271, 2024-03-11T21:05:42.753 INFO:teuthology.orchestra.run.smithi023.stdout: "old_weight": 0, 2024-03-11T21:05:42.753 INFO:teuthology.orchestra.run.smithi023.stdout: "last_purged_snaps_scrub": "2024-03-11T21:05:26.917013+0000", 2024-03-11T21:05:42.754 INFO:teuthology.orchestra.run.smithi023.stdout: "dead_epoch": 0 2024-03-11T21:05:42.754 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:05:42.754 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-11T21:05:42.754 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_upmap": [], 2024-03-11T21:05:42.754 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_upmap_items": [ 2024-03-11T21:05:42.754 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:05:42.754 INFO:teuthology.orchestra.run.smithi023.stdout: "pgid": "2.0", 2024-03-11T21:05:42.754 INFO:teuthology.orchestra.run.smithi023.stdout: "mappings": [ 2024-03-11T21:05:42.754 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:05:42.754 INFO:teuthology.orchestra.run.smithi023.stdout: "from": 7, 2024-03-11T21:05:42.754 INFO:teuthology.orchestra.run.smithi023.stdout: "to": 2 2024-03-11T21:05:42.754 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:05:42.754 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:05:42.755 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:05:42.755 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:05:42.755 INFO:teuthology.orchestra.run.smithi023.stdout: "pgid": "4.b", 2024-03-11T21:05:42.755 INFO:teuthology.orchestra.run.smithi023.stdout: "mappings": [ 2024-03-11T21:05:42.755 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:05:42.755 INFO:teuthology.orchestra.run.smithi023.stdout: "from": 1, 2024-03-11T21:05:42.755 INFO:teuthology.orchestra.run.smithi023.stdout: "to": 2 2024-03-11T21:05:42.755 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:05:42.755 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:05:42.755 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:05:42.755 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:05:42.755 INFO:teuthology.orchestra.run.smithi023.stdout: "pgid": "4.d", 2024-03-11T21:05:42.755 INFO:teuthology.orchestra.run.smithi023.stdout: "mappings": [ 2024-03-11T21:05:42.755 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:05:42.755 INFO:teuthology.orchestra.run.smithi023.stdout: "from": 1, 2024-03-11T21:05:42.755 INFO:teuthology.orchestra.run.smithi023.stdout: "to": 5 2024-03-11T21:05:42.755 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:05:42.756 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:05:42.756 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:05:42.756 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:05:42.756 INFO:teuthology.orchestra.run.smithi023.stdout: "pgid": "4.e", 2024-03-11T21:05:42.756 INFO:teuthology.orchestra.run.smithi023.stdout: "mappings": [ 2024-03-11T21:05:42.756 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:05:42.756 INFO:teuthology.orchestra.run.smithi023.stdout: "from": 1, 2024-03-11T21:05:42.756 INFO:teuthology.orchestra.run.smithi023.stdout: "to": 2 2024-03-11T21:05:42.756 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:05:42.756 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:05:42.756 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:05:42.756 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:05:42.756 INFO:teuthology.orchestra.run.smithi023.stdout: "pgid": "4.16", 2024-03-11T21:05:42.756 INFO:teuthology.orchestra.run.smithi023.stdout: "mappings": [ 2024-03-11T21:05:42.756 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:05:42.756 INFO:teuthology.orchestra.run.smithi023.stdout: "from": 3, 2024-03-11T21:05:42.757 INFO:teuthology.orchestra.run.smithi023.stdout: "to": 2 2024-03-11T21:05:42.757 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:05:42.757 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:05:42.757 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:05:42.757 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:05:42.757 INFO:teuthology.orchestra.run.smithi023.stdout: "pgid": "4.1d", 2024-03-11T21:05:42.757 INFO:teuthology.orchestra.run.smithi023.stdout: "mappings": [ 2024-03-11T21:05:42.757 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:05:42.757 INFO:teuthology.orchestra.run.smithi023.stdout: "from": 3, 2024-03-11T21:05:42.757 INFO:teuthology.orchestra.run.smithi023.stdout: "to": 5 2024-03-11T21:05:42.757 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:05:42.757 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:05:42.757 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:05:42.757 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-11T21:05:42.757 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_upmap_primaries": [], 2024-03-11T21:05:42.757 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_temp": [], 2024-03-11T21:05:42.758 INFO:teuthology.orchestra.run.smithi023.stdout: "primary_temp": [], 2024-03-11T21:05:42.758 INFO:teuthology.orchestra.run.smithi023.stdout: "blocklist": { 2024-03-11T21:05:42.758 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:0/1618334186": "2024-03-12T20:40:45.361537+0000", 2024-03-11T21:05:42.758 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:6801/2813578081": "2024-03-12T20:40:45.361537+0000", 2024-03-11T21:05:42.758 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:0/2869940180": "2024-03-12T20:51:33.790205+0000", 2024-03-11T21:05:42.758 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:6800/3037564930": "2024-03-12T20:51:33.790205+0000", 2024-03-11T21:05:42.758 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:0/3297567084": "2024-03-12T20:51:33.790205+0000", 2024-03-11T21:05:42.758 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:0/3708753659": "2024-03-12T20:40:45.361537+0000", 2024-03-11T21:05:42.758 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:0/2004146740": "2024-03-12T20:40:45.361537+0000", 2024-03-11T21:05:42.758 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:0/827653812": "2024-03-12T20:41:21.301516+0000", 2024-03-11T21:05:42.758 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:6800/1057699850": "2024-03-12T20:41:21.301516+0000", 2024-03-11T21:05:42.758 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:6800/2813578081": "2024-03-12T20:40:45.361537+0000", 2024-03-11T21:05:42.758 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:6801/1057699850": "2024-03-12T20:41:21.301516+0000", 2024-03-11T21:05:42.758 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:0/3833230375": "2024-03-12T20:41:21.301516+0000", 2024-03-11T21:05:42.758 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:6801/3037564930": "2024-03-12T20:51:33.790205+0000", 2024-03-11T21:05:42.758 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:0/1182809102": "2024-03-12T20:51:33.790205+0000", 2024-03-11T21:05:42.759 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:0/2122134187": "2024-03-12T20:51:33.790205+0000", 2024-03-11T21:05:42.759 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:0/2356701899": "2024-03-12T20:41:21.301516+0000", 2024-03-11T21:05:42.759 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:0/843974100": "2024-03-12T20:51:33.790205+0000" 2024-03-11T21:05:42.759 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:05:42.759 INFO:teuthology.orchestra.run.smithi023.stdout: "range_blocklist": {}, 2024-03-11T21:05:42.759 INFO:teuthology.orchestra.run.smithi023.stdout: "erasure_code_profiles": { 2024-03-11T21:05:42.759 INFO:teuthology.orchestra.run.smithi023.stdout: "default": { 2024-03-11T21:05:42.759 INFO:teuthology.orchestra.run.smithi023.stdout: "crush-failure-domain": "osd", 2024-03-11T21:05:42.759 INFO:teuthology.orchestra.run.smithi023.stdout: "k": "2", 2024-03-11T21:05:42.759 INFO:teuthology.orchestra.run.smithi023.stdout: "m": "1", 2024-03-11T21:05:42.759 INFO:teuthology.orchestra.run.smithi023.stdout: "plugin": "jerasure", 2024-03-11T21:05:42.759 INFO:teuthology.orchestra.run.smithi023.stdout: "technique": "reed_sol_van" 2024-03-11T21:05:42.759 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:05:42.759 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:05:42.759 INFO:teuthology.orchestra.run.smithi023.stdout: "removed_snaps_queue": [], 2024-03-11T21:05:42.759 INFO:teuthology.orchestra.run.smithi023.stdout: "new_removed_snaps": [], 2024-03-11T21:05:42.760 INFO:teuthology.orchestra.run.smithi023.stdout: "new_purged_snaps": [], 2024-03-11T21:05:42.760 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_node_flags": {}, 2024-03-11T21:05:42.760 INFO:teuthology.orchestra.run.smithi023.stdout: "device_class_flags": {}, 2024-03-11T21:05:42.760 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": { 2024-03-11T21:05:42.760 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode_enabled": false, 2024-03-11T21:05:42.760 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_bucket_count": 0, 2024-03-11T21:05:42.760 INFO:teuthology.orchestra.run.smithi023.stdout: "degraded_stretch_mode": 0, 2024-03-11T21:05:42.760 INFO:teuthology.orchestra.run.smithi023.stdout: "recovering_stretch_mode": 0, 2024-03-11T21:05:42.760 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode_bucket": 0 2024-03-11T21:05:42.760 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:05:42.760 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-11T21:05:42.760 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:05:42.788 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:05:42.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:42.793 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:05:42.894 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:42.917 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:42.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:42.919 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:05:43.020 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:05:43.043 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:05:43.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:43.048 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:05:43.149 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:43.182 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:43.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:43.187 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:05:43.288 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:05:43.306 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:05:43.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:43.308 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:05:43.409 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:05:43.439 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:05:43.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:43.444 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:05:43.544 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:05:43.568 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:05:43.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:43.570 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:05:43.671 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:43.690 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:43.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:43.691 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:05:43.781 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:43 smithi023 bash[29146]: cluster 2024-03-11T21:05:42.205479+0000 mgr.y (mgr.24380) 553 : cluster 0 pgmap v616: 148 pgs: 148 active+clean; 3.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.20k op/s 2024-03-11T21:05:43.782 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:43 smithi023 bash[29146]: audit 2024-03-11T21:05:42.653660+0000 mon.a (mon.0) 1551 : audit 0 from='client.? 172.21.15.23:0/1784910155' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-11T21:05:43.782 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:43 smithi023 bash[21372]: cluster 2024-03-11T21:05:42.205479+0000 mgr.y (mgr.24380) 553 : cluster 0 pgmap v616: 148 pgs: 148 active+clean; 3.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.20k op/s 2024-03-11T21:05:43.782 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:43 smithi023 bash[21372]: audit 2024-03-11T21:05:42.653660+0000 mon.a (mon.0) 1551 : audit 0 from='client.? 172.21.15.23:0/1784910155' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-11T21:05:43.792 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:43.822 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:43.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:43.827 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:05:43.927 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:05:43.953 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:05:43.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:43.956 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:05:44.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:43 smithi146 bash[23809]: cluster 2024-03-11T21:05:42.205479+0000 mgr.y (mgr.24380) 553 : cluster 0 pgmap v616: 148 pgs: 148 active+clean; 3.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.20k op/s 2024-03-11T21:05:44.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:43 smithi146 bash[23809]: audit 2024-03-11T21:05:42.653660+0000 mon.a (mon.0) 1551 : audit 0 from='client.? 172.21.15.23:0/1784910155' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-11T21:05:44.057 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:44.057 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T21:05:44.075 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:44.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:44.076 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:05:44.093 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd unset noscrub 2024-03-11T21:05:44.177 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:44.206 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:44.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:44.210 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:05:44.310 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:44.338 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:44.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:44.343 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:05:44.444 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:44.479 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:44.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:44.483 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:05:44.584 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:05:44.617 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:05:44.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:44.619 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:05:44.720 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:05:44.748 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:05:44.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:44.750 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:05:44.781 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:44 smithi023 bash[21372]: cluster 2024-03-11T21:05:44.205942+0000 mgr.y (mgr.24380) 554 : cluster 0 pgmap v617: 148 pgs: 148 active+clean; 3.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 994 op/s 2024-03-11T21:05:44.782 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:44 smithi023 bash[29146]: cluster 2024-03-11T21:05:44.205942+0000 mgr.y (mgr.24380) 554 : cluster 0 pgmap v617: 148 pgs: 148 active+clean; 3.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 994 op/s 2024-03-11T21:05:44.851 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:05:44.877 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:05:44.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:44.883 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:05:44.984 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:44.993 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:44 smithi146 bash[23809]: cluster 2024-03-11T21:05:44.205942+0000 mgr.y (mgr.24380) 554 : cluster 0 pgmap v617: 148 pgs: 148 active+clean; 3.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 994 op/s 2024-03-11T21:05:45.015 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:45.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:45.021 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:05:45.024 INFO:tasks.thrashosds.thrasher:j is {'epoch': 290, 'fsid': '67ba00f8-dfe7-11ee-95c8-87774f69a715', 'created': '2024-03-11T20:40:13.664802+0000', 'modified': '2024-03-11T21:05:39.598975+0000', 'last_up_change': '2024-03-11T21:00:47.324817+0000', 'last_in_change': '2024-03-11T20:57:23.560527+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 20, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 9, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-11T20:45:31.964570+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '142', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6.980000019073486, 'score_stable': 6.980000019073486, 'optimal_score': 0.4300000071525574, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.8001575469970703, 'average_primary_affinity': 0.9100000262260437, 'average_primary_affinity_weighted': 0.800000011920929}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2024-03-11T20:50:26.421486+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '78', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5399999618530273, 'score_stable': 1.5399999618530273, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.8001575469970703, 'average_primary_affinity': 0.9100000262260437, 'average_primary_affinity_weighted': 0.800000011920929}}, {'pool': 3, 'pool_name': 'default.rgw.log', 'create_time': '2024-03-11T20:50:28.592615+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.8001575469970703, 'average_primary_affinity': 0.9100000262260437, 'average_primary_affinity_weighted': 0.800000011920929}}, {'pool': 4, 'pool_name': 'default.rgw.control', 'create_time': '2024-03-11T20:50:30.536371+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9700000286102295, 'score_stable': 1.9700000286102295, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.8001575469970703, 'average_primary_affinity': 0.9100000262260437, 'average_primary_affinity_weighted': 0.800000011920929}}, {'pool': 5, 'pool_name': 'datapool', 'create_time': '2024-03-11T20:50:31.135845+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 87, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 4.670000076293945, 'score_stable': 4.670000076293945, 'optimal_score': 1, 'raw_score_acting': 4.670000076293945, 'raw_score_stable': 4.670000076293945, 'primary_affinity_weighted': 0.8001575469970703, 'average_primary_affinity': 0.9100000262260437, 'average_primary_affinity_weighted': 0.800000011920929}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-03-11T20:50:33.233163+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9700000286102295, 'score_stable': 1.9700000286102295, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.8001575469970703, 'average_primary_affinity': 0.9100000262260437, 'average_primary_affinity_weighted': 0.800000011920929}}, {'pool': 9, 'pool_name': 'unique_pool_2', 'create_time': '2024-03-11T21:02:46.162698+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '256', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.190000057220459, 'score_stable': 2.190000057220459, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 0.8001575469970703, 'average_primary_affinity': 0.9100000262260437, 'average_primary_affinity_weighted': 0.800000011920929}}], 'osds': [{'osd': 0, 'uuid': '3df72b4a-8fe4-4265-b4c6-af022e1e23e1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.4012603759765625, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 248, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6802', 'nonce': 2154627393}, {'type': 'v1', 'addr': '172.21.15.23:6803', 'nonce': 2154627393}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6804', 'nonce': 2154627393}, {'type': 'v1', 'addr': '172.21.15.23:6805', 'nonce': 2154627393}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6808', 'nonce': 2154627393}, {'type': 'v1', 'addr': '172.21.15.23:6809', 'nonce': 2154627393}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6806', 'nonce': 2154627393}, {'type': 'v1', 'addr': '172.21.15.23:6807', 'nonce': 2154627393}]}, 'public_addr': '172.21.15.23:6803/2154627393', 'cluster_addr': '172.21.15.23:6805/2154627393', 'heartbeat_back_addr': '172.21.15.23:6809/2154627393', 'heartbeat_front_addr': '172.21.15.23:6807/2154627393', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'e8fd7500-9081-41a1-8077-a4d0f0e12a2c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 13, 'last_clean_end': 173, 'up_from': 211, 'up_thru': 285, 'down_at': 174, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6810', 'nonce': 56585354}, {'type': 'v1', 'addr': '172.21.15.23:6811', 'nonce': 56585354}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6812', 'nonce': 56585354}, {'type': 'v1', 'addr': '172.21.15.23:6813', 'nonce': 56585354}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6816', 'nonce': 56585354}, {'type': 'v1', 'addr': '172.21.15.23:6817', 'nonce': 56585354}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6814', 'nonce': 56585354}, {'type': 'v1', 'addr': '172.21.15.23:6815', 'nonce': 56585354}]}, 'public_addr': '172.21.15.23:6811/56585354', 'cluster_addr': '172.21.15.23:6813/56585354', 'heartbeat_back_addr': '172.21.15.23:6817/56585354', 'heartbeat_front_addr': '172.21.15.23:6815/56585354', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'dd85d52a-df48-4b03-8612-cecfd82b5987', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 19, 'up_thru': 285, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6818', 'nonce': 1770748056}, {'type': 'v1', 'addr': '172.21.15.23:6819', 'nonce': 1770748056}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6820', 'nonce': 1770748056}, {'type': 'v1', 'addr': '172.21.15.23:6821', 'nonce': 1770748056}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6824', 'nonce': 1770748056}, {'type': 'v1', 'addr': '172.21.15.23:6825', 'nonce': 1770748056}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6822', 'nonce': 1770748056}, {'type': 'v1', 'addr': '172.21.15.23:6823', 'nonce': 1770748056}]}, 'public_addr': '172.21.15.23:6819/1770748056', 'cluster_addr': '172.21.15.23:6821/1770748056', 'heartbeat_back_addr': '172.21.15.23:6825/1770748056', 'heartbeat_front_addr': '172.21.15.23:6823/1770748056', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'e3021106-3779-4b94-9bbf-a7724544110f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 28, 'up_thru': 285, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6826', 'nonce': 938496814}, {'type': 'v1', 'addr': '172.21.15.23:6827', 'nonce': 938496814}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6828', 'nonce': 938496814}, {'type': 'v1', 'addr': '172.21.15.23:6829', 'nonce': 938496814}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6832', 'nonce': 938496814}, {'type': 'v1', 'addr': '172.21.15.23:6833', 'nonce': 938496814}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6830', 'nonce': 938496814}, {'type': 'v1', 'addr': '172.21.15.23:6831', 'nonce': 938496814}]}, 'public_addr': '172.21.15.23:6827/938496814', 'cluster_addr': '172.21.15.23:6829/938496814', 'heartbeat_back_addr': '172.21.15.23:6833/938496814', 'heartbeat_front_addr': '172.21.15.23:6831/938496814', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '4dcbc91e-7035-413d-97bc-c9d804eb16d7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 35, 'last_clean_end': 182, 'up_from': 204, 'up_thru': 248, 'down_at': 183, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6800', 'nonce': 3858359434}, {'type': 'v1', 'addr': '172.21.15.146:6801', 'nonce': 3858359434}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6802', 'nonce': 3858359434}, {'type': 'v1', 'addr': '172.21.15.146:6803', 'nonce': 3858359434}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6806', 'nonce': 3858359434}, {'type': 'v1', 'addr': '172.21.15.146:6807', 'nonce': 3858359434}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6804', 'nonce': 3858359434}, {'type': 'v1', 'addr': '172.21.15.146:6805', 'nonce': 3858359434}]}, 'public_addr': '172.21.15.146:6801/3858359434', 'cluster_addr': '172.21.15.146:6803/3858359434', 'heartbeat_back_addr': '172.21.15.146:6807/3858359434', 'heartbeat_front_addr': '172.21.15.146:6805/3858359434', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'aefcb766-8bae-4ff4-b71e-6691c8ac02f1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 44, 'up_thru': 285, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6808', 'nonce': 4003230814}, {'type': 'v1', 'addr': '172.21.15.146:6809', 'nonce': 4003230814}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6810', 'nonce': 4003230814}, {'type': 'v1', 'addr': '172.21.15.146:6811', 'nonce': 4003230814}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6814', 'nonce': 4003230814}, {'type': 'v1', 'addr': '172.21.15.146:6815', 'nonce': 4003230814}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6812', 'nonce': 4003230814}, {'type': 'v1', 'addr': '172.21.15.146:6813', 'nonce': 4003230814}]}, 'public_addr': '172.21.15.146:6809/4003230814', 'cluster_addr': '172.21.15.146:6811/4003230814', 'heartbeat_back_addr': '172.21.15.146:6815/4003230814', 'heartbeat_front_addr': '172.21.15.146:6813/4003230814', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '5cb151e8-1f9b-4617-8a86-adfb06bb9606', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 57, 'up_thru': 285, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6816', 'nonce': 3334092376}, {'type': 'v1', 'addr': '172.21.15.146:6817', 'nonce': 3334092376}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6818', 'nonce': 3334092376}, {'type': 'v1', 'addr': '172.21.15.146:6819', 'nonce': 3334092376}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6822', 'nonce': 3334092376}, {'type': 'v1', 'addr': '172.21.15.146:6823', 'nonce': 3334092376}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6820', 'nonce': 3334092376}, {'type': 'v1', 'addr': '172.21.15.146:6821', 'nonce': 3334092376}]}, 'public_addr': '172.21.15.146:6817/3334092376', 'cluster_addr': '172.21.15.146:6819/3334092376', 'heartbeat_back_addr': '172.21.15.146:6823/3334092376', 'heartbeat_front_addr': '172.21.15.146:6821/3334092376', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'bdbd11a2-fa09-4f82-93a2-6b2d9c750bfd', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 68, 'up_thru': 285, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6824', 'nonce': 1678407923}, {'type': 'v1', 'addr': '172.21.15.146:6825', 'nonce': 1678407923}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6826', 'nonce': 1678407923}, {'type': 'v1', 'addr': '172.21.15.146:6827', 'nonce': 1678407923}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6830', 'nonce': 1678407923}, {'type': 'v1', 'addr': '172.21.15.146:6831', 'nonce': 1678407923}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6828', 'nonce': 1678407923}, {'type': 'v1', 'addr': '172.21.15.146:6829', 'nonce': 1678407923}]}, 'public_addr': '172.21.15.146:6825/1678407923', 'cluster_addr': '172.21.15.146:6827/1678407923', 'heartbeat_back_addr': '172.21.15.146:6831/1678407923', 'heartbeat_front_addr': '172.21.15.146:6829/1678407923', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T21:05:33.549850+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T21:04:51.062946+0000', 'dead_epoch': 173}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T21:04:48.914230+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T21:04:48.498387+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T21:04:23.824633+0000', 'dead_epoch': 182}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T21:05:33.523843+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T21:04:45.871348+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T21:05:26.917013+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '2.0', 'mappings': [{'from': 7, 'to': 2}]}, {'pgid': '4.b', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.d', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.e', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.16', 'mappings': [{'from': 3, 'to': 2}]}, {'pgid': '4.1d', 'mappings': [{'from': 3, 'to': 5}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.23:0/1618334186': '2024-03-12T20:40:45.361537+0000', '172.21.15.23:6801/2813578081': '2024-03-12T20:40:45.361537+0000', '172.21.15.23:0/2869940180': '2024-03-12T20:51:33.790205+0000', '172.21.15.23:6800/3037564930': '2024-03-12T20:51:33.790205+0000', '172.21.15.23:0/3297567084': '2024-03-12T20:51:33.790205+0000', '172.21.15.23:0/3708753659': '2024-03-12T20:40:45.361537+0000', '172.21.15.23:0/2004146740': '2024-03-12T20:40:45.361537+0000', '172.21.15.23:0/827653812': '2024-03-12T20:41:21.301516+0000', '172.21.15.23:6800/1057699850': '2024-03-12T20:41:21.301516+0000', '172.21.15.23:6800/2813578081': '2024-03-12T20:40:45.361537+0000', '172.21.15.23:6801/1057699850': '2024-03-12T20:41:21.301516+0000', '172.21.15.23:0/3833230375': '2024-03-12T20:41:21.301516+0000', '172.21.15.23:6801/3037564930': '2024-03-12T20:51:33.790205+0000', '172.21.15.23:0/1182809102': '2024-03-12T20:51:33.790205+0000', '172.21.15.23:0/2122134187': '2024-03-12T20:51:33.790205+0000', '172.21.15.23:0/2356701899': '2024-03-12T20:41:21.301516+0000', '172.21.15.23:0/843974100': '2024-03-12T20:51:33.790205+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-03-11T21:05:45.024 INFO:tasks.thrashosds.thrasher:No pg_upmap entries; doing nothing 2024-03-11T21:05:45.122 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:05:45.145 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:05:45.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:45.148 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:05:45.248 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:45.280 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:45.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:45.282 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:05:45.383 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:45.417 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:45.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:45.420 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:05:45.521 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:05:45.559 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:05:45.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:45.561 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:05:45.662 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:05:45.692 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:05:45.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:45.695 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:05:45.796 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:05:45.813 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:05:45.814 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:45 smithi146 bash[23809]: audit 2024-03-11T21:05:45.569417+0000 mgr.y (mgr.24380) 555 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:05:45.814 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:05:45 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:05:45.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:45.815 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:05:45.816 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:45 smithi023 bash[21372]: audit 2024-03-11T21:05:45.569417+0000 mgr.y (mgr.24380) 555 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:05:45.817 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:45 smithi023 bash[29146]: audit 2024-03-11T21:05:45.569417+0000 mgr.y (mgr.24380) 555 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:05:45.916 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:45.940 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:45.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:45.942 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:05:46.043 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:46.066 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:46.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:46.068 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:05:46.168 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:05:46.204 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:05:46.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:46.207 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:05:46.308 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:05:46.331 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:05:46.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:46.334 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:05:46.434 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:05:46.459 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:05:46.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:46.463 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:05:46.563 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:05:46.587 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:05:46.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:46.589 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:05:46.690 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:46.698 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:46 smithi023 bash[21372]: cluster 2024-03-11T21:05:46.206759+0000 mgr.y (mgr.24380) 556 : cluster 0 pgmap v618: 148 pgs: 148 active+clean; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 948 op/s 2024-03-11T21:05:46.700 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:46 smithi023 bash[29146]: cluster 2024-03-11T21:05:46.206759+0000 mgr.y (mgr.24380) 556 : cluster 0 pgmap v618: 148 pgs: 148 active+clean; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 948 op/s 2024-03-11T21:05:46.714 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:46.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:46.717 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:05:46.818 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:46.843 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:46.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:46.846 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:05:46.947 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:46.955 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:46 smithi146 bash[23809]: cluster 2024-03-11T21:05:46.206759+0000 mgr.y (mgr.24380) 556 : cluster 0 pgmap v618: 148 pgs: 148 active+clean; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 948 op/s 2024-03-11T21:05:46.972 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:46.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:46.977 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:05:47.077 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:05:47.101 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:05:47.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:47.106 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:05:47.207 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:05:47.231 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:05:47.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:47.235 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:05:47.336 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:47.363 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:47.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:47.367 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:05:47.468 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:05:47.495 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:05:47.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:47.498 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:05:47.599 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:05:47.635 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:05:47.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:47.638 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:05:47.739 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:47.762 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:47.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:47.767 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:05:47.868 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:05:47.894 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:05:47.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:47.896 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:05:47.997 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:48.031 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:48.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:48.034 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:05:48.135 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:48.161 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:48.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:48.166 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:05:48.266 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:48.290 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:48.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:48.292 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:05:48.306 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-11T21:05:48.306 INFO:teuthology.orchestra.run.smithi023.stdout: "ops": [], 2024-03-11T21:05:48.306 INFO:teuthology.orchestra.run.smithi023.stdout: "complaint_time": 30, 2024-03-11T21:05:48.306 INFO:teuthology.orchestra.run.smithi023.stdout: "num_blocked_ops": 0 2024-03-11T21:05:48.306 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-11T21:05:48.393 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:05:48.418 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:05:48.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:48.420 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:05:48.500 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:05:48 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:05:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:05:48.521 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:05:48.545 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:05:48.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:48.548 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:05:48.648 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:05:48.681 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:05:48.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:48.685 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:05:48.786 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:05:48.803 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:05:48.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:48.805 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:05:48.906 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:05:48.934 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:05:48.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:48.936 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:05:49.037 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:49.066 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:49.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:49.069 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:05:49.170 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:49.193 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:49.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:49.199 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:05:49.300 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:05:49.325 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:05:49.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:49.333 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:05:49.434 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:05:49.464 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-03-11T21:05:49.467 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:05:49.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:49.471 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:05:49.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:49 smithi146 bash[23809]: cluster 2024-03-11T21:05:48.207348+0000 mgr.y (mgr.24380) 557 : cluster 0 pgmap v619: 148 pgs: 148 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 936 op/s 2024-03-11T21:05:49.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:49 smithi023 bash[21372]: cluster 2024-03-11T21:05:48.207348+0000 mgr.y (mgr.24380) 557 : cluster 0 pgmap v619: 148 pgs: 148 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 936 op/s 2024-03-11T21:05:49.532 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:49 smithi023 bash[29146]: cluster 2024-03-11T21:05:48.207348+0000 mgr.y (mgr.24380) 557 : cluster 0 pgmap v619: 148 pgs: 148 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 936 op/s 2024-03-11T21:05:49.572 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:05:49.611 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:05:49.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:49.616 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:05:49.716 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:05:49.755 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:05:49.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:49.759 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:05:49.860 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:49.874 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:05:49.874 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:05:49.884 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:49.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:49.887 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:05:49.988 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:05:50.025 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 6, 7, 1] out_osds: [] dead_osds: [] live_osds: [0, 2, 3, 5, 6, 7, 4, 1] 2024-03-11T21:05:50.025 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-11T21:05:50.025 INFO:tasks.thrashosds.thrasher:inject_pause on osd.6 2024-03-11T21:05:50.025 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-03-11T21:05:50.025 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-03-11T21:05:50.026 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph daemon osd.6 config set filestore_inject_stall 3 2024-03-11T21:05:50.027 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:05:50.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:50.029 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:05:50.130 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:05:50.171 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:05:50.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:50.175 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:05:50.276 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:50.309 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:50.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:50.314 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:05:50.415 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:05:50.451 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:05:50.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:50.454 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:05:50.554 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:05:50.582 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:05:50.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:50.585 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:05:50.685 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:50.719 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:50.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:50.722 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:05:50.822 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:50.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:50.856 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:50.857 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:05:50.958 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:50.993 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:50.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:50.998 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:05:51.099 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:51.125 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:51.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:51.128 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:05:51.229 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:05:51.254 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:05:51.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:51.258 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:05:51.358 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:51.388 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:51.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:51.394 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:05:51.402 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:51 smithi023 bash[21372]: cluster 2024-03-11T21:05:49.824907+0000 osd.4 (osd.4) 103 : cluster 0 purged_snaps scrub starts 2024-03-11T21:05:51.403 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:51 smithi023 bash[21372]: cluster 2024-03-11T21:05:49.825012+0000 osd.4 (osd.4) 104 : cluster 0 purged_snaps scrub ok 2024-03-11T21:05:51.403 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:51 smithi023 bash[21372]: cluster 2024-03-11T21:05:50.207886+0000 mgr.y (mgr.24380) 558 : cluster 0 pgmap v620: 148 pgs: 148 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 968 B/s rd, 0 B/s wr, 915 op/s 2024-03-11T21:05:51.403 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:51 smithi023 bash[21372]: cluster 2024-03-11T21:05:50.273126+0000 mon.a (mon.0) 1552 : cluster 0 osdmap e291: 8 total, 8 up, 8 in 2024-03-11T21:05:51.403 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:51 smithi023 bash[21372]: audit 2024-03-11T21:05:50.464881+0000 mon.a (mon.0) 1553 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:05:51.494 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:05:51.504 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:51 smithi023 bash[29146]: cluster 2024-03-11T21:05:49.824907+0000 osd.4 (osd.4) 103 : cluster 0 purged_snaps scrub starts 2024-03-11T21:05:51.504 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:51 smithi023 bash[29146]: cluster 2024-03-11T21:05:49.825012+0000 osd.4 (osd.4) 104 : cluster 0 purged_snaps scrub ok 2024-03-11T21:05:51.504 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:51 smithi023 bash[29146]: cluster 2024-03-11T21:05:50.207886+0000 mgr.y (mgr.24380) 558 : cluster 0 pgmap v620: 148 pgs: 148 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 968 B/s rd, 0 B/s wr, 915 op/s 2024-03-11T21:05:51.505 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:51 smithi023 bash[29146]: cluster 2024-03-11T21:05:50.273126+0000 mon.a (mon.0) 1552 : cluster 0 osdmap e291: 8 total, 8 up, 8 in 2024-03-11T21:05:51.505 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:51 smithi023 bash[29146]: audit 2024-03-11T21:05:50.464881+0000 mon.a (mon.0) 1553 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:05:51.520 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:05:51.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:51.526 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:05:51.626 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:51.635 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:51 smithi146 bash[23809]: cluster 2024-03-11T21:05:49.824907+0000 osd.4 (osd.4) 103 : cluster 0 purged_snaps scrub starts 2024-03-11T21:05:51.635 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:51 smithi146 bash[23809]: cluster 2024-03-11T21:05:49.825012+0000 osd.4 (osd.4) 104 : cluster 0 purged_snaps scrub ok 2024-03-11T21:05:51.635 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:51 smithi146 bash[23809]: cluster 2024-03-11T21:05:50.207886+0000 mgr.y (mgr.24380) 558 : cluster 0 pgmap v620: 148 pgs: 148 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 968 B/s rd, 0 B/s wr, 915 op/s 2024-03-11T21:05:51.635 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:51 smithi146 bash[23809]: cluster 2024-03-11T21:05:50.273126+0000 mon.a (mon.0) 1552 : cluster 0 osdmap e291: 8 total, 8 up, 8 in 2024-03-11T21:05:51.635 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:51 smithi146 bash[23809]: audit 2024-03-11T21:05:50.464881+0000 mon.a (mon.0) 1553 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:05:51.650 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:51.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:51.656 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:05:51.757 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:51.789 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:51.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:51.791 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:05:51.892 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:05:51.910 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:05:51.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:51.912 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:05:52.014 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:05:52.057 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:05:52.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:52.059 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:05:52.160 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:05:52.189 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:05:52.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:52.194 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:05:52.295 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:05:52.323 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:05:52.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:52.325 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:05:52.426 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:52.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:52.459 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:52.459 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:05:52.560 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:52.588 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:52.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:52.591 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:05:52.692 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:05:52.733 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:05:52.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:52.738 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:05:52.838 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:05:52.868 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:05:52.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:52.870 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:05:52.971 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:05:53.002 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:05:53.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:53.005 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:05:53.106 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:53.124 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:53.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:53.126 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:05:53.227 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:05:53.250 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:05:53.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:53.257 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:05:53.358 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:53.390 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:53.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:53.392 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:05:53.493 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:05:53.520 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:05:53.528 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:53 smithi146 bash[23809]: cluster 2024-03-11T21:05:52.208708+0000 mgr.y (mgr.24380) 559 : cluster 0 pgmap v622: 148 pgs: 148 active+clean; 2.4 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.00k op/s 2024-03-11T21:05:53.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:53.529 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:05:53.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:53 smithi023 bash[21372]: cluster 2024-03-11T21:05:52.208708+0000 mgr.y (mgr.24380) 559 : cluster 0 pgmap v622: 148 pgs: 148 active+clean; 2.4 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.00k op/s 2024-03-11T21:05:53.531 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:53 smithi023 bash[29146]: cluster 2024-03-11T21:05:52.208708+0000 mgr.y (mgr.24380) 559 : cluster 0 pgmap v622: 148 pgs: 148 active+clean; 2.4 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.00k op/s 2024-03-11T21:05:53.630 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:53.660 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:53.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:53.662 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:05:53.763 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:53.789 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:53.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:53.794 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:05:53.895 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:53.919 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:53.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:53.921 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:05:54.022 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:05:54.039 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:05:54.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:54.041 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:05:54.142 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:54.169 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:54.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:54.174 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:05:54.275 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:05:54.309 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:05:54.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:54.314 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:05:54.414 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:54.449 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:54.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:54.454 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:05:54.555 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:05:54.585 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:05:54.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:54.590 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:05:54.691 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:54.711 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:54.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:54.713 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:05:54.814 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:54.845 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:54.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:54.849 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:05:54.852 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:05:54.949 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:05:54.976 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:05:54.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:54.981 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:05:55.082 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:05:55.097 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:05:55.107 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:05:55.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:55.111 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:05:55.211 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:55.237 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:55.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:55.239 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:05:55.339 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:05:55.364 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:05:55.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:55.369 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:05:55.469 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:05:55.478 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:55 smithi146 bash[23809]: cluster 2024-03-11T21:05:54.209174+0000 mgr.y (mgr.24380) 560 : cluster 0 pgmap v623: 148 pgs: 148 active+clean; 2.4 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.00k op/s 2024-03-11T21:05:55.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:55.495 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:05:55.495 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:05:55.596 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:05:55.616 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:55 smithi023 bash[21372]: cluster 2024-03-11T21:05:54.209174+0000 mgr.y (mgr.24380) 560 : cluster 0 pgmap v623: 148 pgs: 148 active+clean; 2.4 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.00k op/s 2024-03-11T21:05:55.616 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:55 smithi023 bash[29146]: cluster 2024-03-11T21:05:54.209174+0000 mgr.y (mgr.24380) 560 : cluster 0 pgmap v623: 148 pgs: 148 active+clean; 2.4 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.00k op/s 2024-03-11T21:05:55.623 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:05:55.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:55.625 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:05:55.726 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:05:55.735 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:05:55 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:05:55.746 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:05:55.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:55.748 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:05:55.848 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:55.873 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:55.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:55.875 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:05:55.975 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:05:56.008 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:05:56.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:56.011 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:05:56.112 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:05:56.146 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:05:56.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:56.151 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:05:56.251 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:05:56.275 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:05:56.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:56.281 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:05:56.382 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:05:56.408 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:05:56.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:56.410 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:05:56.511 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:05:56.519 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:56 smithi023 bash[21372]: audit 2024-03-11T21:05:55.577153+0000 mgr.y (mgr.24380) 561 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:05:56.519 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:56 smithi023 bash[21372]: cluster 2024-03-11T21:05:55.827516+0000 osd.2 (osd.2) 203 : cluster 0 purged_snaps scrub starts 2024-03-11T21:05:56.519 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:56 smithi023 bash[21372]: cluster 2024-03-11T21:05:55.827609+0000 osd.2 (osd.2) 204 : cluster 0 purged_snaps scrub ok 2024-03-11T21:05:56.519 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:56 smithi023 bash[29146]: audit 2024-03-11T21:05:55.577153+0000 mgr.y (mgr.24380) 561 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:05:56.519 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:56 smithi023 bash[29146]: cluster 2024-03-11T21:05:55.827516+0000 osd.2 (osd.2) 203 : cluster 0 purged_snaps scrub starts 2024-03-11T21:05:56.519 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:56 smithi023 bash[29146]: cluster 2024-03-11T21:05:55.827609+0000 osd.2 (osd.2) 204 : cluster 0 purged_snaps scrub ok 2024-03-11T21:05:56.529 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:05:56.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:56.532 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:05:56.564 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:56 smithi146 bash[23809]: audit 2024-03-11T21:05:55.577153+0000 mgr.y (mgr.24380) 561 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:05:56.564 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:56 smithi146 bash[23809]: cluster 2024-03-11T21:05:55.827516+0000 osd.2 (osd.2) 203 : cluster 0 purged_snaps scrub starts 2024-03-11T21:05:56.564 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:56 smithi146 bash[23809]: cluster 2024-03-11T21:05:55.827609+0000 osd.2 (osd.2) 204 : cluster 0 purged_snaps scrub ok 2024-03-11T21:05:56.633 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:56.657 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:56.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:56.659 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:05:56.760 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:05:56.791 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:05:56.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:56.794 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:05:56.895 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:56.912 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:56.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:56.914 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:05:57.014 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:05:57.043 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:05:57.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:57.046 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:05:57.147 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:57.173 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:57.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:57.175 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:05:57.276 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:57.309 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:57.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:57.311 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:05:57.412 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:57.438 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:57.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:57.442 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:05:57.543 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:05:57.553 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:57 smithi146 bash[23809]: cluster 2024-03-11T21:05:56.209958+0000 mgr.y (mgr.24380) 562 : cluster 0 pgmap v624: 148 pgs: 148 active+clean; 2.2 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.03k op/s 2024-03-11T21:05:57.553 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:57 smithi146 bash[23809]: cluster 2024-03-11T21:05:56.301872+0000 mon.a (mon.0) 1554 : cluster 0 osdmap e292: 8 total, 8 up, 8 in 2024-03-11T21:05:57.571 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:05:57.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:57.575 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:05:57.676 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:05:57.704 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:05:57.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:57.706 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:05:57.739 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:57 smithi023 bash[21372]: cluster 2024-03-11T21:05:56.209958+0000 mgr.y (mgr.24380) 562 : cluster 0 pgmap v624: 148 pgs: 148 active+clean; 2.2 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.03k op/s 2024-03-11T21:05:57.739 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:57 smithi023 bash[21372]: cluster 2024-03-11T21:05:56.301872+0000 mon.a (mon.0) 1554 : cluster 0 osdmap e292: 8 total, 8 up, 8 in 2024-03-11T21:05:57.739 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:57 smithi023 bash[29146]: cluster 2024-03-11T21:05:56.209958+0000 mgr.y (mgr.24380) 562 : cluster 0 pgmap v624: 148 pgs: 148 active+clean; 2.2 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.03k op/s 2024-03-11T21:05:57.739 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:57 smithi023 bash[29146]: cluster 2024-03-11T21:05:56.301872+0000 mon.a (mon.0) 1554 : cluster 0 osdmap e292: 8 total, 8 up, 8 in 2024-03-11T21:05:57.807 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:57.838 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:57.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:57.841 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:05:57.941 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:05:57.970 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:05:57.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:57.972 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:05:58.027 INFO:teuthology.orchestra.run.smithi023.stdout:osd.0: {} 2024-03-11T21:05:58.073 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:58.086 INFO:teuthology.orchestra.run.smithi023.stdout:osd.1: {} 2024-03-11T21:05:58.103 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:58.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:58.106 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:05:58.127 INFO:teuthology.orchestra.run.smithi023.stdout:osd.2: {} 2024-03-11T21:05:58.176 INFO:teuthology.orchestra.run.smithi023.stdout:osd.3: {} 2024-03-11T21:05:58.208 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:05:58.222 INFO:teuthology.orchestra.run.smithi023.stdout:osd.4: {} 2024-03-11T21:05:58.234 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:05:58.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:58.239 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:05:58.252 INFO:teuthology.orchestra.run.smithi023.stdout:osd.5: {} 2024-03-11T21:05:58.284 INFO:teuthology.orchestra.run.smithi023.stdout:osd.6: {} 2024-03-11T21:05:58.327 INFO:teuthology.orchestra.run.smithi023.stdout:osd.7: {} 2024-03-11T21:05:58.340 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:58.344 INFO:teuthology.orchestra.run.smithi023.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:05:58.344 INFO:teuthology.orchestra.run.smithi023.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:05:58.344 INFO:teuthology.orchestra.run.smithi023.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:05:58.344 INFO:teuthology.orchestra.run.smithi023.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:05:58.344 INFO:teuthology.orchestra.run.smithi023.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:05:58.345 INFO:teuthology.orchestra.run.smithi023.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:05:58.345 INFO:teuthology.orchestra.run.smithi023.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:05:58.345 INFO:teuthology.orchestra.run.smithi023.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:05:58.391 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:05:58.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:58.396 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:05:58.497 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:05:58.520 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:05:58.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:58.522 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:05:58.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:58 smithi023 bash[21372]: audit 2024-03-11T21:05:57.968147+0000 mon.c (mon.2) 71 : audit 0 from='client.? 172.21.15.23:0/1423577897' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:05:58.531 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:05:58 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:05:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:05:58.531 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:58 smithi023 bash[29146]: audit 2024-03-11T21:05:57.968147+0000 mon.c (mon.2) 71 : audit 0 from='client.? 172.21.15.23:0/1423577897' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:05:58.593 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:58 smithi146 bash[23809]: audit 2024-03-11T21:05:57.968147+0000 mon.c (mon.2) 71 : audit 0 from='client.? 172.21.15.23:0/1423577897' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:05:58.622 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:05:58.652 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:05:58.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:58.654 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:05:58.755 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:05:58.784 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:05:58.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:58.789 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:05:58.889 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:05:58.914 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:05:58.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:58.917 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:05:59.018 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:05:59.039 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:05:59.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:59.040 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:05:59.141 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:05:59.170 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:05:59.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:59.174 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:05:59.275 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:05:59.309 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:05:59.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:59.312 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:05:59.333 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:59 smithi023 bash[21372]: cluster 2024-03-11T21:05:58.210658+0000 mgr.y (mgr.24380) 563 : cluster 0 pgmap v626: 148 pgs: 148 active+clean; 2.1 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.25k op/s 2024-03-11T21:05:59.333 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:59 smithi023 bash[21372]: audit 2024-03-11T21:05:58.632746+0000 mon.c (mon.2) 72 : audit 1 from='client.? 172.21.15.23:0/3959174158' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:05:59.333 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:05:59 smithi023 bash[21372]: audit 2024-03-11T21:05:58.633571+0000 mon.a (mon.0) 1555 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:05:59.334 INFO:teuthology.orchestra.run.smithi023.stderr:noscrub is unset 2024-03-11T21:05:59.414 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:59.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:59 smithi146 bash[23809]: cluster 2024-03-11T21:05:58.210658+0000 mgr.y (mgr.24380) 563 : cluster 0 pgmap v626: 148 pgs: 148 active+clean; 2.1 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.25k op/s 2024-03-11T21:05:59.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:59 smithi146 bash[23809]: audit 2024-03-11T21:05:58.632746+0000 mon.c (mon.2) 72 : audit 1 from='client.? 172.21.15.23:0/3959174158' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:05:59.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:05:59 smithi146 bash[23809]: audit 2024-03-11T21:05:58.633571+0000 mon.a (mon.0) 1555 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:05:59.443 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:59.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:59.450 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:05:59.551 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:05:59.568 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:05:59.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:59.570 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:05:59.671 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:59.693 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:05:59.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:59.695 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:05:59.795 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:59.812 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:59 smithi023 bash[29146]: cluster 2024-03-11T21:05:58.210658+0000 mgr.y (mgr.24380) 563 : cluster 0 pgmap v626: 148 pgs: 148 active+clean; 2.1 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.25k op/s 2024-03-11T21:05:59.813 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:59 smithi023 bash[29146]: audit 2024-03-11T21:05:58.632746+0000 mon.c (mon.2) 72 : audit 1 from='client.? 172.21.15.23:0/3959174158' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:05:59.813 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:05:59 smithi023 bash[29146]: audit 2024-03-11T21:05:58.633571+0000 mon.a (mon.0) 1555 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:05:59.814 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:05:59.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:59.816 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:05:59.917 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:05:59.943 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:05:59.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:05:59.948 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:06:00.048 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:06:00.070 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:06:00.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:00.072 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:06:00.173 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:06:00.204 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:06:00.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:00.209 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:06:00.310 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:00.348 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:00.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:00.350 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:06:00.451 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:06:00.460 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:00 smithi146 bash[23809]: cluster 2024-03-11T21:05:59.317968+0000 mon.a (mon.0) 1556 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:06:00.460 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:00 smithi146 bash[23809]: audit 2024-03-11T21:05:59.321333+0000 mon.a (mon.0) 1557 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:06:00.460 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:00 smithi146 bash[23809]: cluster 2024-03-11T21:05:59.330793+0000 mon.a (mon.0) 1558 : cluster 0 osdmap e293: 8 total, 8 up, 8 in 2024-03-11T21:06:00.471 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:06:00.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:00.473 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:06:00.574 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:06:00.598 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:06:00.599 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:00 smithi023 bash[29146]: cluster 2024-03-11T21:05:59.317968+0000 mon.a (mon.0) 1556 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:06:00.599 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:00 smithi023 bash[29146]: audit 2024-03-11T21:05:59.321333+0000 mon.a (mon.0) 1557 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:06:00.599 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:00 smithi023 bash[29146]: cluster 2024-03-11T21:05:59.330793+0000 mon.a (mon.0) 1558 : cluster 0 osdmap e293: 8 total, 8 up, 8 in 2024-03-11T21:06:00.599 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:00 smithi023 bash[21372]: cluster 2024-03-11T21:05:59.317968+0000 mon.a (mon.0) 1556 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:06:00.600 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:00 smithi023 bash[21372]: audit 2024-03-11T21:05:59.321333+0000 mon.a (mon.0) 1557 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:06:00.600 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:00 smithi023 bash[21372]: cluster 2024-03-11T21:05:59.330793+0000 mon.a (mon.0) 1558 : cluster 0 osdmap e293: 8 total, 8 up, 8 in 2024-03-11T21:06:00.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:00.600 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:06:00.701 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:00.726 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:00.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:00.728 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:06:00.828 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:06:00.864 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:06:00.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:00.867 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:06:00.968 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:06:00.995 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:06:00.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:00.997 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:06:01.098 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:06:01.126 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:06:01.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:01.129 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:06:01.230 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:06:01.265 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:06:01.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:01.268 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:06:01.369 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:06:01.394 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:06:01.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:01.396 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:06:01.497 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:01.529 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:01.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:01 smithi023 bash[21372]: cluster 2024-03-11T21:06:00.211287+0000 mgr.y (mgr.24380) 564 : cluster 0 pgmap v628: 148 pgs: 148 active+clean; 2.1 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 665 op/s 2024-03-11T21:06:01.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:01 smithi023 bash[21372]: cluster 2024-03-11T21:06:00.603674+0000 osd.6 (osd.6) 226 : cluster 0 purged_snaps scrub starts 2024-03-11T21:06:01.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:01 smithi023 bash[21372]: cluster 2024-03-11T21:06:00.603769+0000 osd.6 (osd.6) 227 : cluster 0 purged_snaps scrub ok 2024-03-11T21:06:01.531 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:01 smithi023 bash[29146]: cluster 2024-03-11T21:06:00.211287+0000 mgr.y (mgr.24380) 564 : cluster 0 pgmap v628: 148 pgs: 148 active+clean; 2.1 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 665 op/s 2024-03-11T21:06:01.531 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:01 smithi023 bash[29146]: cluster 2024-03-11T21:06:00.603674+0000 osd.6 (osd.6) 226 : cluster 0 purged_snaps scrub starts 2024-03-11T21:06:01.531 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:01 smithi023 bash[29146]: cluster 2024-03-11T21:06:00.603769+0000 osd.6 (osd.6) 227 : cluster 0 purged_snaps scrub ok 2024-03-11T21:06:01.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:01.532 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:06:01.633 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:06:01.660 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:06:01.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:01.663 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:06:01.708 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-03-11T21:06:01.708 INFO:teuthology.orchestra.run.smithi146.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-03-11T21:06:01.708 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-03-11T21:06:01.721 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:01 smithi146 bash[23809]: cluster 2024-03-11T21:06:00.211287+0000 mgr.y (mgr.24380) 564 : cluster 0 pgmap v628: 148 pgs: 148 active+clean; 2.1 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 665 op/s 2024-03-11T21:06:01.721 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:01 smithi146 bash[23809]: cluster 2024-03-11T21:06:00.603674+0000 osd.6 (osd.6) 226 : cluster 0 purged_snaps scrub starts 2024-03-11T21:06:01.721 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:01 smithi146 bash[23809]: cluster 2024-03-11T21:06:00.603769+0000 osd.6 (osd.6) 227 : cluster 0 purged_snaps scrub ok 2024-03-11T21:06:01.764 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:06:01.791 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:06:01.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:01.794 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:06:01.895 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:01.938 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:01.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:01.943 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:06:02.044 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:02.070 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:02.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:02.075 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:06:02.175 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:06:02.201 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:06:02.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:02.203 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:06:02.304 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:06:02.325 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:06:02.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:02.329 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:06:02.429 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:06:02.971 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:02 smithi146 bash[23809]: cluster 2024-03-11T21:06:00.828904+0000 osd.7 (osd.7) 325 : cluster 0 6.12 scrub starts 2024-03-11T21:06:02.972 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:02 smithi146 bash[23809]: cluster 2024-03-11T21:06:00.837714+0000 osd.7 (osd.7) 326 : cluster 0 6.12 scrub ok 2024-03-11T21:06:02.972 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:02 smithi146 bash[23809]: cluster 2024-03-11T21:06:01.200373+0000 osd.3 (osd.3) 308 : cluster 0 4.10 scrub starts 2024-03-11T21:06:02.972 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:02 smithi146 bash[23809]: cluster 2024-03-11T21:06:01.202141+0000 osd.3 (osd.3) 309 : cluster 0 4.10 scrub ok 2024-03-11T21:06:02.972 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:02 smithi146 bash[23809]: cluster 2024-03-11T21:06:01.357930+0000 mon.a (mon.0) 1559 : cluster 0 osdmap e294: 8 total, 8 up, 8 in 2024-03-11T21:06:02.980 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:02 smithi023 bash[21372]: cluster 2024-03-11T21:06:00.828904+0000 osd.7 (osd.7) 325 : cluster 0 6.12 scrub starts 2024-03-11T21:06:02.980 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:02 smithi023 bash[21372]: cluster 2024-03-11T21:06:00.837714+0000 osd.7 (osd.7) 326 : cluster 0 6.12 scrub ok 2024-03-11T21:06:02.980 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:02 smithi023 bash[21372]: cluster 2024-03-11T21:06:01.200373+0000 osd.3 (osd.3) 308 : cluster 0 4.10 scrub starts 2024-03-11T21:06:02.980 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:02 smithi023 bash[21372]: cluster 2024-03-11T21:06:01.202141+0000 osd.3 (osd.3) 309 : cluster 0 4.10 scrub ok 2024-03-11T21:06:02.980 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:02 smithi023 bash[21372]: cluster 2024-03-11T21:06:01.357930+0000 mon.a (mon.0) 1559 : cluster 0 osdmap e294: 8 total, 8 up, 8 in 2024-03-11T21:06:02.980 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:02 smithi023 bash[29146]: cluster 2024-03-11T21:06:00.828904+0000 osd.7 (osd.7) 325 : cluster 0 6.12 scrub starts 2024-03-11T21:06:02.981 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:02 smithi023 bash[29146]: cluster 2024-03-11T21:06:00.837714+0000 osd.7 (osd.7) 326 : cluster 0 6.12 scrub ok 2024-03-11T21:06:02.981 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:02 smithi023 bash[29146]: cluster 2024-03-11T21:06:01.200373+0000 osd.3 (osd.3) 308 : cluster 0 4.10 scrub starts 2024-03-11T21:06:02.981 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:02 smithi023 bash[29146]: cluster 2024-03-11T21:06:01.202141+0000 osd.3 (osd.3) 309 : cluster 0 4.10 scrub ok 2024-03-11T21:06:02.981 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:02 smithi023 bash[29146]: cluster 2024-03-11T21:06:01.357930+0000 mon.a (mon.0) 1559 : cluster 0 osdmap e294: 8 total, 8 up, 8 in 2024-03-11T21:06:03.005 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:06:03.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:03.012 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:06:03.114 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:06:03.141 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:06:03.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:03.144 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:06:03.244 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:03.266 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:03.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:03.268 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:06:03.369 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:03.395 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:03.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:03.398 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:06:03.499 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:06:03.509 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:03 smithi146 bash[23809]: cluster 2024-03-11T21:06:02.171148+0000 osd.5 (osd.5) 281 : cluster 0 2.16 scrub starts 2024-03-11T21:06:03.509 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:03 smithi146 bash[23809]: cluster 2024-03-11T21:06:02.177494+0000 osd.5 (osd.5) 282 : cluster 0 2.16 scrub ok 2024-03-11T21:06:03.509 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:03 smithi146 bash[23809]: cluster 2024-03-11T21:06:02.211985+0000 mgr.y (mgr.24380) 565 : cluster 0 pgmap v630: 148 pgs: 148 active+clean; 1.8 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.02k op/s 2024-03-11T21:06:03.509 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:03 smithi146 bash[23809]: cluster 2024-03-11T21:06:02.322322+0000 osd.2 (osd.2) 205 : cluster 0 6.17 scrub starts 2024-03-11T21:06:03.509 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:03 smithi146 bash[23809]: cluster 2024-03-11T21:06:02.414528+0000 osd.2 (osd.2) 206 : cluster 0 6.17 scrub ok 2024-03-11T21:06:03.509 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:03 smithi146 bash[23809]: cluster 2024-03-11T21:06:02.418471+0000 osd.3 (osd.3) 310 : cluster 0 purged_snaps scrub starts 2024-03-11T21:06:03.509 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:03 smithi146 bash[23809]: cluster 2024-03-11T21:06:02.418535+0000 osd.3 (osd.3) 311 : cluster 0 purged_snaps scrub ok 2024-03-11T21:06:03.509 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:03 smithi146 bash[23809]: audit 2024-03-11T21:06:02.579022+0000 mon.a (mon.0) 1560 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:06:03.532 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:06:03.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:03.536 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:06:03.594 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:03 smithi023 bash[29146]: cluster 2024-03-11T21:06:02.171148+0000 osd.5 (osd.5) 281 : cluster 0 2.16 scrub starts 2024-03-11T21:06:03.594 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:03 smithi023 bash[29146]: cluster 2024-03-11T21:06:02.177494+0000 osd.5 (osd.5) 282 : cluster 0 2.16 scrub ok 2024-03-11T21:06:03.594 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:03 smithi023 bash[29146]: cluster 2024-03-11T21:06:02.211985+0000 mgr.y (mgr.24380) 565 : cluster 0 pgmap v630: 148 pgs: 148 active+clean; 1.8 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.02k op/s 2024-03-11T21:06:03.594 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:03 smithi023 bash[29146]: cluster 2024-03-11T21:06:02.322322+0000 osd.2 (osd.2) 205 : cluster 0 6.17 scrub starts 2024-03-11T21:06:03.594 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:03 smithi023 bash[29146]: cluster 2024-03-11T21:06:02.414528+0000 osd.2 (osd.2) 206 : cluster 0 6.17 scrub ok 2024-03-11T21:06:03.594 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:03 smithi023 bash[29146]: cluster 2024-03-11T21:06:02.418471+0000 osd.3 (osd.3) 310 : cluster 0 purged_snaps scrub starts 2024-03-11T21:06:03.594 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:03 smithi023 bash[29146]: cluster 2024-03-11T21:06:02.418535+0000 osd.3 (osd.3) 311 : cluster 0 purged_snaps scrub ok 2024-03-11T21:06:03.595 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:03 smithi023 bash[29146]: audit 2024-03-11T21:06:02.579022+0000 mon.a (mon.0) 1560 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:06:03.595 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:03 smithi023 bash[21372]: cluster 2024-03-11T21:06:02.171148+0000 osd.5 (osd.5) 281 : cluster 0 2.16 scrub starts 2024-03-11T21:06:03.595 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:03 smithi023 bash[21372]: cluster 2024-03-11T21:06:02.177494+0000 osd.5 (osd.5) 282 : cluster 0 2.16 scrub ok 2024-03-11T21:06:03.595 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:03 smithi023 bash[21372]: cluster 2024-03-11T21:06:02.211985+0000 mgr.y (mgr.24380) 565 : cluster 0 pgmap v630: 148 pgs: 148 active+clean; 1.8 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.02k op/s 2024-03-11T21:06:03.595 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:03 smithi023 bash[21372]: cluster 2024-03-11T21:06:02.322322+0000 osd.2 (osd.2) 205 : cluster 0 6.17 scrub starts 2024-03-11T21:06:03.595 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:03 smithi023 bash[21372]: cluster 2024-03-11T21:06:02.414528+0000 osd.2 (osd.2) 206 : cluster 0 6.17 scrub ok 2024-03-11T21:06:03.595 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:03 smithi023 bash[21372]: cluster 2024-03-11T21:06:02.418471+0000 osd.3 (osd.3) 310 : cluster 0 purged_snaps scrub starts 2024-03-11T21:06:03.595 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:03 smithi023 bash[21372]: cluster 2024-03-11T21:06:02.418535+0000 osd.3 (osd.3) 311 : cluster 0 purged_snaps scrub ok 2024-03-11T21:06:03.595 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:03 smithi023 bash[21372]: audit 2024-03-11T21:06:02.579022+0000 mon.a (mon.0) 1560 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:06:03.637 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:06:03.670 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:06:03.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:03.674 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:06:03.775 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:06:03.810 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:06:03.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:03.814 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:06:03.828 INFO:teuthology.orchestra.run.smithi023.stdout:ERROR: (22) Invalid argument 2024-03-11T21:06:03.828 INFO:teuthology.orchestra.run.smithi023.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:06:03.915 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:06:03.939 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:06:03.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:03.943 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:06:03.997 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T21:06:04.043 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:04.087 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:04.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:04.102 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:06:04.203 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:04.226 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:04.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:04.231 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:06:04.332 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:04.350 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:04.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:04.351 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:06:04.453 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:06:04.476 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:06:04.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:04.478 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:06:04.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:04 smithi146 bash[23809]: cluster 2024-03-11T21:06:03.214854+0000 osd.3 (osd.3) 312 : cluster 0 9.8 scrub starts 2024-03-11T21:06:04.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:04 smithi146 bash[23809]: cluster 2024-03-11T21:06:03.385500+0000 mon.a (mon.0) 1561 : cluster 0 osdmap e295: 8 total, 8 up, 8 in 2024-03-11T21:06:04.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:04 smithi146 bash[23809]: audit 2024-03-11T21:06:03.480293+0000 mon.a (mon.0) 1562 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:06:04.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:04 smithi146 bash[23809]: audit 2024-03-11T21:06:03.482724+0000 mon.a (mon.0) 1563 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:06:04.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:04 smithi146 bash[23809]: audit 2024-03-11T21:06:03.490326+0000 mon.a (mon.0) 1564 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:06:04.579 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:06:04.605 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:06:04.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:04.607 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:06:04.708 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:06:04.722 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:04 smithi023 bash[21372]: cluster 2024-03-11T21:06:03.214854+0000 osd.3 (osd.3) 312 : cluster 0 9.8 scrub starts 2024-03-11T21:06:04.722 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:04 smithi023 bash[21372]: cluster 2024-03-11T21:06:03.385500+0000 mon.a (mon.0) 1561 : cluster 0 osdmap e295: 8 total, 8 up, 8 in 2024-03-11T21:06:04.722 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:04 smithi023 bash[21372]: audit 2024-03-11T21:06:03.480293+0000 mon.a (mon.0) 1562 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:06:04.722 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:04 smithi023 bash[21372]: audit 2024-03-11T21:06:03.482724+0000 mon.a (mon.0) 1563 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:06:04.722 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:04 smithi023 bash[21372]: audit 2024-03-11T21:06:03.490326+0000 mon.a (mon.0) 1564 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:06:04.722 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:04 smithi023 bash[29146]: cluster 2024-03-11T21:06:03.214854+0000 osd.3 (osd.3) 312 : cluster 0 9.8 scrub starts 2024-03-11T21:06:04.722 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:04 smithi023 bash[29146]: cluster 2024-03-11T21:06:03.385500+0000 mon.a (mon.0) 1561 : cluster 0 osdmap e295: 8 total, 8 up, 8 in 2024-03-11T21:06:04.722 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:04 smithi023 bash[29146]: audit 2024-03-11T21:06:03.480293+0000 mon.a (mon.0) 1562 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:06:04.722 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:04 smithi023 bash[29146]: audit 2024-03-11T21:06:03.482724+0000 mon.a (mon.0) 1563 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:06:04.723 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:04 smithi023 bash[29146]: audit 2024-03-11T21:06:03.490326+0000 mon.a (mon.0) 1564 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:06:04.738 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:06:04.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:04.744 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:06:04.845 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:06:04.870 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:06:04.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:04.873 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:06:04.974 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:06:05.000 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:06:05.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:05.005 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:06:05.106 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:06:05.140 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:06:05.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:05.143 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:06:05.200 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-11T21:06:05.244 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:05.276 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:05.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:05.283 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:06:05.338 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-11T21:06:05.384 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:06:05.427 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:06:05.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:05.438 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:06:05.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:05 smithi023 bash[21372]: cluster 2024-03-11T21:06:03.786366+0000 osd.7 (osd.7) 327 : cluster 0 2.1d scrub starts 2024-03-11T21:06:05.532 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:05 smithi023 bash[21372]: cluster 2024-03-11T21:06:03.793947+0000 osd.7 (osd.7) 328 : cluster 0 2.1d scrub ok 2024-03-11T21:06:05.532 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:05 smithi023 bash[21372]: cluster 2024-03-11T21:06:04.026121+0000 osd.3 (osd.3) 313 : cluster 0 9.8 scrub ok 2024-03-11T21:06:05.532 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:05 smithi023 bash[21372]: cluster 2024-03-11T21:06:04.111100+0000 osd.5 (osd.5) 283 : cluster 0 2.12 scrub starts 2024-03-11T21:06:05.532 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:05 smithi023 bash[21372]: cluster 2024-03-11T21:06:04.112375+0000 osd.5 (osd.5) 284 : cluster 0 2.12 scrub ok 2024-03-11T21:06:05.532 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:05 smithi023 bash[21372]: cluster 2024-03-11T21:06:04.212398+0000 mgr.y (mgr.24380) 566 : cluster 0 pgmap v632: 148 pgs: 148 active+clean; 1.8 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 721 op/s 2024-03-11T21:06:05.532 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:05 smithi023 bash[21372]: audit 2024-03-11T21:06:05.482601+0000 mon.a (mon.0) 1565 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:06:05.532 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:05 smithi023 bash[29146]: cluster 2024-03-11T21:06:03.786366+0000 osd.7 (osd.7) 327 : cluster 0 2.1d scrub starts 2024-03-11T21:06:05.532 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:05 smithi023 bash[29146]: cluster 2024-03-11T21:06:03.793947+0000 osd.7 (osd.7) 328 : cluster 0 2.1d scrub ok 2024-03-11T21:06:05.532 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:05 smithi023 bash[29146]: cluster 2024-03-11T21:06:04.026121+0000 osd.3 (osd.3) 313 : cluster 0 9.8 scrub ok 2024-03-11T21:06:05.532 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:05 smithi023 bash[29146]: cluster 2024-03-11T21:06:04.111100+0000 osd.5 (osd.5) 283 : cluster 0 2.12 scrub starts 2024-03-11T21:06:05.532 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:05 smithi023 bash[29146]: cluster 2024-03-11T21:06:04.112375+0000 osd.5 (osd.5) 284 : cluster 0 2.12 scrub ok 2024-03-11T21:06:05.533 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:05 smithi023 bash[29146]: cluster 2024-03-11T21:06:04.212398+0000 mgr.y (mgr.24380) 566 : cluster 0 pgmap v632: 148 pgs: 148 active+clean; 1.8 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 721 op/s 2024-03-11T21:06:05.533 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:05 smithi023 bash[29146]: audit 2024-03-11T21:06:05.482601+0000 mon.a (mon.0) 1565 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:06:05.539 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:06:05.572 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:06:05.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:05.574 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:06:05.675 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:05.702 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:05.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:05.707 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:06:05.760 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:06:05 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:06:05.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:05 smithi146 bash[23809]: cluster 2024-03-11T21:06:03.786366+0000 osd.7 (osd.7) 327 : cluster 0 2.1d scrub starts 2024-03-11T21:06:05.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:05 smithi146 bash[23809]: cluster 2024-03-11T21:06:03.793947+0000 osd.7 (osd.7) 328 : cluster 0 2.1d scrub ok 2024-03-11T21:06:05.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:05 smithi146 bash[23809]: cluster 2024-03-11T21:06:04.026121+0000 osd.3 (osd.3) 313 : cluster 0 9.8 scrub ok 2024-03-11T21:06:05.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:05 smithi146 bash[23809]: cluster 2024-03-11T21:06:04.111100+0000 osd.5 (osd.5) 283 : cluster 0 2.12 scrub starts 2024-03-11T21:06:05.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:05 smithi146 bash[23809]: cluster 2024-03-11T21:06:04.112375+0000 osd.5 (osd.5) 284 : cluster 0 2.12 scrub ok 2024-03-11T21:06:05.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:05 smithi146 bash[23809]: cluster 2024-03-11T21:06:04.212398+0000 mgr.y (mgr.24380) 566 : cluster 0 pgmap v632: 148 pgs: 148 active+clean; 1.8 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 721 op/s 2024-03-11T21:06:05.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:05 smithi146 bash[23809]: audit 2024-03-11T21:06:05.482601+0000 mon.a (mon.0) 1565 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:06:05.808 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:05.835 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:05.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:05.840 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:06:05.941 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:05.967 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:05.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:05.970 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:06:06.071 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:06.095 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:06.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:06.097 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:06:06.198 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:06.222 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:06.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:06.224 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:06:06.325 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:06.368 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:06.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:06.376 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:06:06.477 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:06:06.502 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:06:06.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:06.509 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:06:06.610 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:06.640 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:06.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:06.643 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:06:06.744 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:06.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:06 smithi146 bash[23809]: cluster 2024-03-11T21:06:04.793354+0000 osd.7 (osd.7) 329 : cluster 0 6.15 scrub starts 2024-03-11T21:06:06.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:06 smithi146 bash[23809]: cluster 2024-03-11T21:06:04.835420+0000 osd.7 (osd.7) 330 : cluster 0 6.15 scrub ok 2024-03-11T21:06:06.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:06 smithi146 bash[23809]: cluster 2024-03-11T21:06:04.971928+0000 osd.0 (osd.0) 213 : cluster 0 3.1b scrub starts 2024-03-11T21:06:06.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:06 smithi146 bash[23809]: cluster 2024-03-11T21:06:04.976047+0000 osd.0 (osd.0) 214 : cluster 0 3.1b scrub ok 2024-03-11T21:06:06.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:06 smithi146 bash[23809]: cluster 2024-03-11T21:06:05.043150+0000 osd.6 (osd.6) 228 : cluster 0 4.13 scrub starts 2024-03-11T21:06:06.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:06 smithi146 bash[23809]: cluster 2024-03-11T21:06:05.051631+0000 osd.6 (osd.6) 229 : cluster 0 4.13 scrub ok 2024-03-11T21:06:06.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:06 smithi146 bash[23809]: cluster 2024-03-11T21:06:05.085121+0000 osd.5 (osd.5) 285 : cluster 0 4.8 scrub starts 2024-03-11T21:06:06.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:06 smithi146 bash[23809]: cluster 2024-03-11T21:06:05.086935+0000 osd.5 (osd.5) 286 : cluster 0 4.8 scrub ok 2024-03-11T21:06:06.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:06 smithi146 bash[23809]: cluster 2024-03-11T21:06:05.279711+0000 osd.2 (osd.2) 207 : cluster 0 2.c scrub starts 2024-03-11T21:06:06.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:06 smithi146 bash[23809]: cluster 2024-03-11T21:06:05.281617+0000 osd.2 (osd.2) 208 : cluster 0 2.c scrub ok 2024-03-11T21:06:06.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:06 smithi146 bash[23809]: cluster 2024-03-11T21:06:05.299631+0000 osd.3 (osd.3) 314 : cluster 0 9.1 scrub starts 2024-03-11T21:06:06.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:06 smithi146 bash[23809]: cluster 2024-03-11T21:06:05.487527+0000 osd.1 (osd.1) 89 : cluster 0 3.1a scrub starts 2024-03-11T21:06:06.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:06 smithi146 bash[23809]: cluster 2024-03-11T21:06:05.503833+0000 osd.1 (osd.1) 90 : cluster 0 3.1a scrub ok 2024-03-11T21:06:06.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:06 smithi146 bash[23809]: audit 2024-03-11T21:06:05.588747+0000 mgr.y (mgr.24380) 567 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:06:06.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:06 smithi146 bash[23809]: cluster 2024-03-11T21:06:05.847020+0000 osd.3 (osd.3) 315 : cluster 0 9.1 scrub ok 2024-03-11T21:06:06.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:06 smithi146 bash[23809]: cluster 2024-03-11T21:06:06.171126+0000 osd.1 (osd.1) 91 : cluster 0 purged_snaps scrub starts 2024-03-11T21:06:06.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:06 smithi146 bash[23809]: cluster 2024-03-11T21:06:06.171222+0000 osd.1 (osd.1) 92 : cluster 0 purged_snaps scrub ok 2024-03-11T21:06:06.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:06 smithi146 bash[23809]: cluster 2024-03-11T21:06:06.429224+0000 osd.1 (osd.1) 93 : cluster 0 4.9 scrub starts 2024-03-11T21:06:06.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:06 smithi146 bash[23809]: cluster 2024-03-11T21:06:06.430742+0000 osd.1 (osd.1) 94 : cluster 0 4.9 scrub ok 2024-03-11T21:06:06.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:06.770 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:06.771 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:06:06.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:06 smithi023 bash[21372]: cluster 2024-03-11T21:06:04.793354+0000 osd.7 (osd.7) 329 : cluster 0 6.15 scrub starts 2024-03-11T21:06:06.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:06 smithi023 bash[21372]: cluster 2024-03-11T21:06:04.835420+0000 osd.7 (osd.7) 330 : cluster 0 6.15 scrub ok 2024-03-11T21:06:06.784 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:06 smithi023 bash[21372]: cluster 2024-03-11T21:06:04.971928+0000 osd.0 (osd.0) 213 : cluster 0 3.1b scrub starts 2024-03-11T21:06:06.784 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:06 smithi023 bash[21372]: cluster 2024-03-11T21:06:04.976047+0000 osd.0 (osd.0) 214 : cluster 0 3.1b scrub ok 2024-03-11T21:06:06.784 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:06 smithi023 bash[21372]: cluster 2024-03-11T21:06:05.043150+0000 osd.6 (osd.6) 228 : cluster 0 4.13 scrub starts 2024-03-11T21:06:06.784 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:06 smithi023 bash[21372]: cluster 2024-03-11T21:06:05.051631+0000 osd.6 (osd.6) 229 : cluster 0 4.13 scrub ok 2024-03-11T21:06:06.784 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:06 smithi023 bash[21372]: cluster 2024-03-11T21:06:05.085121+0000 osd.5 (osd.5) 285 : cluster 0 4.8 scrub starts 2024-03-11T21:06:06.784 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:06 smithi023 bash[21372]: cluster 2024-03-11T21:06:05.086935+0000 osd.5 (osd.5) 286 : cluster 0 4.8 scrub ok 2024-03-11T21:06:06.784 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:06 smithi023 bash[21372]: cluster 2024-03-11T21:06:05.279711+0000 osd.2 (osd.2) 207 : cluster 0 2.c scrub starts 2024-03-11T21:06:06.784 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:06 smithi023 bash[21372]: cluster 2024-03-11T21:06:05.281617+0000 osd.2 (osd.2) 208 : cluster 0 2.c scrub ok 2024-03-11T21:06:06.784 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:06 smithi023 bash[21372]: cluster 2024-03-11T21:06:05.299631+0000 osd.3 (osd.3) 314 : cluster 0 9.1 scrub starts 2024-03-11T21:06:06.784 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:06 smithi023 bash[21372]: cluster 2024-03-11T21:06:05.487527+0000 osd.1 (osd.1) 89 : cluster 0 3.1a scrub starts 2024-03-11T21:06:06.784 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:06 smithi023 bash[21372]: cluster 2024-03-11T21:06:05.503833+0000 osd.1 (osd.1) 90 : cluster 0 3.1a scrub ok 2024-03-11T21:06:06.784 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:06 smithi023 bash[21372]: audit 2024-03-11T21:06:05.588747+0000 mgr.y (mgr.24380) 567 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:06:06.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:06 smithi023 bash[21372]: cluster 2024-03-11T21:06:05.847020+0000 osd.3 (osd.3) 315 : cluster 0 9.1 scrub ok 2024-03-11T21:06:06.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:06 smithi023 bash[21372]: cluster 2024-03-11T21:06:06.171126+0000 osd.1 (osd.1) 91 : cluster 0 purged_snaps scrub starts 2024-03-11T21:06:06.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:06 smithi023 bash[21372]: cluster 2024-03-11T21:06:06.171222+0000 osd.1 (osd.1) 92 : cluster 0 purged_snaps scrub ok 2024-03-11T21:06:06.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:06 smithi023 bash[21372]: cluster 2024-03-11T21:06:06.429224+0000 osd.1 (osd.1) 93 : cluster 0 4.9 scrub starts 2024-03-11T21:06:06.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:06 smithi023 bash[21372]: cluster 2024-03-11T21:06:06.430742+0000 osd.1 (osd.1) 94 : cluster 0 4.9 scrub ok 2024-03-11T21:06:06.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:06 smithi023 bash[29146]: cluster 2024-03-11T21:06:04.793354+0000 osd.7 (osd.7) 329 : cluster 0 6.15 scrub starts 2024-03-11T21:06:06.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:06 smithi023 bash[29146]: cluster 2024-03-11T21:06:04.835420+0000 osd.7 (osd.7) 330 : cluster 0 6.15 scrub ok 2024-03-11T21:06:06.785 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:06 smithi023 bash[29146]: cluster 2024-03-11T21:06:04.971928+0000 osd.0 (osd.0) 213 : cluster 0 3.1b scrub starts 2024-03-11T21:06:06.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:06 smithi023 bash[29146]: cluster 2024-03-11T21:06:04.976047+0000 osd.0 (osd.0) 214 : cluster 0 3.1b scrub ok 2024-03-11T21:06:06.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:06 smithi023 bash[29146]: cluster 2024-03-11T21:06:05.043150+0000 osd.6 (osd.6) 228 : cluster 0 4.13 scrub starts 2024-03-11T21:06:06.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:06 smithi023 bash[29146]: cluster 2024-03-11T21:06:05.051631+0000 osd.6 (osd.6) 229 : cluster 0 4.13 scrub ok 2024-03-11T21:06:06.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:06 smithi023 bash[29146]: cluster 2024-03-11T21:06:05.085121+0000 osd.5 (osd.5) 285 : cluster 0 4.8 scrub starts 2024-03-11T21:06:06.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:06 smithi023 bash[29146]: cluster 2024-03-11T21:06:05.086935+0000 osd.5 (osd.5) 286 : cluster 0 4.8 scrub ok 2024-03-11T21:06:06.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:06 smithi023 bash[29146]: cluster 2024-03-11T21:06:05.279711+0000 osd.2 (osd.2) 207 : cluster 0 2.c scrub starts 2024-03-11T21:06:06.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:06 smithi023 bash[29146]: cluster 2024-03-11T21:06:05.281617+0000 osd.2 (osd.2) 208 : cluster 0 2.c scrub ok 2024-03-11T21:06:06.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:06 smithi023 bash[29146]: cluster 2024-03-11T21:06:05.299631+0000 osd.3 (osd.3) 314 : cluster 0 9.1 scrub starts 2024-03-11T21:06:06.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:06 smithi023 bash[29146]: cluster 2024-03-11T21:06:05.487527+0000 osd.1 (osd.1) 89 : cluster 0 3.1a scrub starts 2024-03-11T21:06:06.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:06 smithi023 bash[29146]: cluster 2024-03-11T21:06:05.503833+0000 osd.1 (osd.1) 90 : cluster 0 3.1a scrub ok 2024-03-11T21:06:06.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:06 smithi023 bash[29146]: audit 2024-03-11T21:06:05.588747+0000 mgr.y (mgr.24380) 567 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:06:06.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:06 smithi023 bash[29146]: cluster 2024-03-11T21:06:05.847020+0000 osd.3 (osd.3) 315 : cluster 0 9.1 scrub ok 2024-03-11T21:06:06.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:06 smithi023 bash[29146]: cluster 2024-03-11T21:06:06.171126+0000 osd.1 (osd.1) 91 : cluster 0 purged_snaps scrub starts 2024-03-11T21:06:06.786 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:06 smithi023 bash[29146]: cluster 2024-03-11T21:06:06.171222+0000 osd.1 (osd.1) 92 : cluster 0 purged_snaps scrub ok 2024-03-11T21:06:06.787 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:06 smithi023 bash[29146]: cluster 2024-03-11T21:06:06.429224+0000 osd.1 (osd.1) 93 : cluster 0 4.9 scrub starts 2024-03-11T21:06:06.787 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:06 smithi023 bash[29146]: cluster 2024-03-11T21:06:06.430742+0000 osd.1 (osd.1) 94 : cluster 0 4.9 scrub ok 2024-03-11T21:06:06.871 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:06:06.900 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:06:06.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:06.904 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:06:07.005 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:06:07.034 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:06:07.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:07.039 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:06:07.140 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:07.165 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:07.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:07.170 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:06:07.270 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:06:07.295 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:06:07.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:07.297 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:06:07.398 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:07.422 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:07.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:07.426 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:06:07.526 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:06:07.536 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:07 smithi146 bash[23809]: cluster 2024-03-11T21:06:05.957193+0000 osd.0 (osd.0) 215 : cluster 0 6.10 scrub starts 2024-03-11T21:06:07.536 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:07 smithi146 bash[23809]: cluster 2024-03-11T21:06:05.984412+0000 osd.0 (osd.0) 216 : cluster 0 6.10 scrub ok 2024-03-11T21:06:07.536 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:07 smithi146 bash[23809]: cluster 2024-03-11T21:06:06.215437+0000 mgr.y (mgr.24380) 568 : cluster 0 pgmap v633: 148 pgs: 148 active+clean; 1.6 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 743 B/s rd, 0 B/s wr, 1.03k op/s 2024-03-11T21:06:07.536 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:07 smithi146 bash[23809]: cluster 2024-03-11T21:06:06.264100+0000 osd.2 (osd.2) 209 : cluster 0 6.f scrub starts 2024-03-11T21:06:07.536 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:07 smithi146 bash[23809]: cluster 2024-03-11T21:06:06.419242+0000 osd.2 (osd.2) 210 : cluster 0 6.f scrub ok 2024-03-11T21:06:07.537 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:07 smithi146 bash[23809]: cluster 2024-03-11T21:06:06.529226+0000 mon.a (mon.0) 1566 : cluster 0 osdmap e296: 8 total, 8 up, 8 in 2024-03-11T21:06:07.555 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:06:07.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:07.560 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:06:07.661 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:07.688 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:07.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:07.693 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:06:07.781 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:07 smithi023 bash[21372]: cluster 2024-03-11T21:06:05.957193+0000 osd.0 (osd.0) 215 : cluster 0 6.10 scrub starts 2024-03-11T21:06:07.781 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:07 smithi023 bash[21372]: cluster 2024-03-11T21:06:05.984412+0000 osd.0 (osd.0) 216 : cluster 0 6.10 scrub ok 2024-03-11T21:06:07.781 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:07 smithi023 bash[21372]: cluster 2024-03-11T21:06:06.215437+0000 mgr.y (mgr.24380) 568 : cluster 0 pgmap v633: 148 pgs: 148 active+clean; 1.6 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 743 B/s rd, 0 B/s wr, 1.03k op/s 2024-03-11T21:06:07.781 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:07 smithi023 bash[21372]: cluster 2024-03-11T21:06:06.264100+0000 osd.2 (osd.2) 209 : cluster 0 6.f scrub starts 2024-03-11T21:06:07.781 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:07 smithi023 bash[21372]: cluster 2024-03-11T21:06:06.419242+0000 osd.2 (osd.2) 210 : cluster 0 6.f scrub ok 2024-03-11T21:06:07.782 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:07 smithi023 bash[21372]: cluster 2024-03-11T21:06:06.529226+0000 mon.a (mon.0) 1566 : cluster 0 osdmap e296: 8 total, 8 up, 8 in 2024-03-11T21:06:07.782 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:07 smithi023 bash[29146]: cluster 2024-03-11T21:06:05.957193+0000 osd.0 (osd.0) 215 : cluster 0 6.10 scrub starts 2024-03-11T21:06:07.782 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:07 smithi023 bash[29146]: cluster 2024-03-11T21:06:05.984412+0000 osd.0 (osd.0) 216 : cluster 0 6.10 scrub ok 2024-03-11T21:06:07.782 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:07 smithi023 bash[29146]: cluster 2024-03-11T21:06:06.215437+0000 mgr.y (mgr.24380) 568 : cluster 0 pgmap v633: 148 pgs: 148 active+clean; 1.6 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 743 B/s rd, 0 B/s wr, 1.03k op/s 2024-03-11T21:06:07.782 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:07 smithi023 bash[29146]: cluster 2024-03-11T21:06:06.264100+0000 osd.2 (osd.2) 209 : cluster 0 6.f scrub starts 2024-03-11T21:06:07.782 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:07 smithi023 bash[29146]: cluster 2024-03-11T21:06:06.419242+0000 osd.2 (osd.2) 210 : cluster 0 6.f scrub ok 2024-03-11T21:06:07.782 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:07 smithi023 bash[29146]: cluster 2024-03-11T21:06:06.529226+0000 mon.a (mon.0) 1566 : cluster 0 osdmap e296: 8 total, 8 up, 8 in 2024-03-11T21:06:07.794 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:07.821 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:07.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:07.825 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:06:07.926 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:07.949 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:07.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:07.953 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:06:07.974 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 6, 7, 1] out_osds: [] dead_osds: [] live_osds: [0, 2, 3, 5, 6, 7, 4, 1] 2024-03-11T21:06:07.974 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-11T21:06:07.974 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-03-11T21:06:07.974 INFO:tasks.thrashosds.thrasher:Setting osd 0 primary_affinity to 1.000000 2024-03-11T21:06:07.975 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd primary-affinity 0 1 2024-03-11T21:06:08.053 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:06:08.082 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:06:08.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:08.084 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:06:08.185 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:06:08.222 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:06:08.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:08.230 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:06:08.331 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:08.356 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:08.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:08.360 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:06:08.461 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:08.503 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:08.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:08.506 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:06:08.607 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:06:08.631 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:06:08.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:08.636 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:06:08.736 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:08.765 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:08.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:08.767 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:06:08.781 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:06:08 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:06:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:06:08.849 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:06:08.868 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:06:08.892 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:06:08.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:08.896 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:06:08.997 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:06:09.035 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:06:09.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:09.038 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:06:09.138 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:06:09.161 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:06:09.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:09.165 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:06:09.266 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:06:09.286 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:06:09.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:09.289 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:06:09.390 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:09.421 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:09.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:09.425 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:06:09.526 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:09.538 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:09 smithi146 bash[23809]: cluster 2024-03-11T21:06:08.216035+0000 mgr.y (mgr.24380) 569 : cluster 0 pgmap v635: 148 pgs: 148 active+clean; 1.5 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 890 op/s 2024-03-11T21:06:09.539 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:09 smithi146 bash[23809]: cluster 2024-03-11T21:06:08.461514+0000 osd.1 (osd.1) 95 : cluster 0 6.1d scrub starts 2024-03-11T21:06:09.539 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:09 smithi146 bash[23809]: cluster 2024-03-11T21:06:08.497217+0000 osd.1 (osd.1) 96 : cluster 0 6.1d scrub ok 2024-03-11T21:06:09.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:09.554 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:09.555 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:06:09.655 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:06:09.687 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:06:09.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:09.689 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:06:09.781 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:09 smithi023 bash[21372]: cluster 2024-03-11T21:06:08.216035+0000 mgr.y (mgr.24380) 569 : cluster 0 pgmap v635: 148 pgs: 148 active+clean; 1.5 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 890 op/s 2024-03-11T21:06:09.781 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:09 smithi023 bash[21372]: cluster 2024-03-11T21:06:08.461514+0000 osd.1 (osd.1) 95 : cluster 0 6.1d scrub starts 2024-03-11T21:06:09.781 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:09 smithi023 bash[21372]: cluster 2024-03-11T21:06:08.497217+0000 osd.1 (osd.1) 96 : cluster 0 6.1d scrub ok 2024-03-11T21:06:09.782 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:09 smithi023 bash[29146]: cluster 2024-03-11T21:06:08.216035+0000 mgr.y (mgr.24380) 569 : cluster 0 pgmap v635: 148 pgs: 148 active+clean; 1.5 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 890 op/s 2024-03-11T21:06:09.782 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:09 smithi023 bash[29146]: cluster 2024-03-11T21:06:08.461514+0000 osd.1 (osd.1) 95 : cluster 0 6.1d scrub starts 2024-03-11T21:06:09.782 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:09 smithi023 bash[29146]: cluster 2024-03-11T21:06:08.497217+0000 osd.1 (osd.1) 96 : cluster 0 6.1d scrub ok 2024-03-11T21:06:09.791 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:06:09.827 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:06:09.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:09.830 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:06:09.870 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:06:09.870 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:06:09.931 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:06:09.964 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:06:09.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:09.967 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:06:10.068 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:06:10.088 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:06:10.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:10.090 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:06:10.191 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:06:10.227 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:06:10.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:10.236 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:06:10.337 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:10.360 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:10.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:10.363 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:06:10.463 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:10.491 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:10.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:10.495 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:06:10.595 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:10.620 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:10.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:10.622 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:06:10.723 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:06:10.745 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:06:10.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:10.749 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:06:10.850 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:06:10.877 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:06:10.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:10.881 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:06:10.982 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:06:11.005 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:06:11.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:11.007 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:06:11.108 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:11.140 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:11.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:11.144 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:06:11.245 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:06:11.272 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:06:11.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:11.276 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:06:11.376 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:06:11.403 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:06:11.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:11.405 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:06:11.506 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:11.537 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:11.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:11.542 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:06:11.643 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:11.652 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:11 smithi146 bash[23809]: cluster 2024-03-11T21:06:10.216621+0000 mgr.y (mgr.24380) 570 : cluster 0 pgmap v636: 148 pgs: 148 active+clean; 1.5 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 605 op/s 2024-03-11T21:06:11.673 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:11.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:11.676 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:06:11.777 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:06:11.805 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:06:11.807 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:11 smithi023 bash[29146]: cluster 2024-03-11T21:06:10.216621+0000 mgr.y (mgr.24380) 570 : cluster 0 pgmap v636: 148 pgs: 148 active+clean; 1.5 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 605 op/s 2024-03-11T21:06:11.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:11.809 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:06:11.809 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:11 smithi023 bash[21372]: cluster 2024-03-11T21:06:10.216621+0000 mgr.y (mgr.24380) 570 : cluster 0 pgmap v636: 148 pgs: 148 active+clean; 1.5 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 605 op/s 2024-03-11T21:06:11.910 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:06:11.911 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:06:11.937 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:06:11.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:11.939 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:06:12.040 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:06:12.065 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:06:12.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:12.068 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:06:12.169 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:12.205 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:12.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:12.212 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:06:12.313 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:06:12.348 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:06:12.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:12.353 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:06:12.453 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:06:12.471 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:06:12.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:12.473 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:06:12.573 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:06:12.603 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:06:12.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:12.607 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:06:12.707 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:12.754 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:12.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:12.758 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:06:12.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:12 smithi146 bash[23809]: cluster 2024-03-11T21:06:10.934114+0000 osd.7 (osd.7) 331 : cluster 0 3.14 scrub starts 2024-03-11T21:06:12.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:12 smithi146 bash[23809]: cluster 2024-03-11T21:06:10.944539+0000 osd.7 (osd.7) 332 : cluster 0 3.14 scrub ok 2024-03-11T21:06:12.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:12 smithi146 bash[23809]: cluster 2024-03-11T21:06:11.042583+0000 osd.5 (osd.5) 287 : cluster 0 4.15 scrub starts 2024-03-11T21:06:12.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:12 smithi146 bash[23809]: cluster 2024-03-11T21:06:11.049947+0000 osd.5 (osd.5) 288 : cluster 0 4.15 scrub ok 2024-03-11T21:06:12.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:12 smithi146 bash[23809]: cluster 2024-03-11T21:06:11.344555+0000 osd.3 (osd.3) 316 : cluster 0 3.8 scrub starts 2024-03-11T21:06:12.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:12 smithi146 bash[23809]: cluster 2024-03-11T21:06:11.347856+0000 osd.3 (osd.3) 317 : cluster 0 3.8 scrub ok 2024-03-11T21:06:12.859 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:06:12.868 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:12 smithi023 bash[29146]: cluster 2024-03-11T21:06:10.934114+0000 osd.7 (osd.7) 331 : cluster 0 3.14 scrub starts 2024-03-11T21:06:12.868 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:12 smithi023 bash[29146]: cluster 2024-03-11T21:06:10.944539+0000 osd.7 (osd.7) 332 : cluster 0 3.14 scrub ok 2024-03-11T21:06:12.868 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:12 smithi023 bash[29146]: cluster 2024-03-11T21:06:11.042583+0000 osd.5 (osd.5) 287 : cluster 0 4.15 scrub starts 2024-03-11T21:06:12.868 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:12 smithi023 bash[29146]: cluster 2024-03-11T21:06:11.049947+0000 osd.5 (osd.5) 288 : cluster 0 4.15 scrub ok 2024-03-11T21:06:12.868 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:12 smithi023 bash[29146]: cluster 2024-03-11T21:06:11.344555+0000 osd.3 (osd.3) 316 : cluster 0 3.8 scrub starts 2024-03-11T21:06:12.868 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:12 smithi023 bash[29146]: cluster 2024-03-11T21:06:11.347856+0000 osd.3 (osd.3) 317 : cluster 0 3.8 scrub ok 2024-03-11T21:06:12.868 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:12 smithi023 bash[21372]: cluster 2024-03-11T21:06:10.934114+0000 osd.7 (osd.7) 331 : cluster 0 3.14 scrub starts 2024-03-11T21:06:12.868 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:12 smithi023 bash[21372]: cluster 2024-03-11T21:06:10.944539+0000 osd.7 (osd.7) 332 : cluster 0 3.14 scrub ok 2024-03-11T21:06:12.868 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:12 smithi023 bash[21372]: cluster 2024-03-11T21:06:11.042583+0000 osd.5 (osd.5) 287 : cluster 0 4.15 scrub starts 2024-03-11T21:06:12.868 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:12 smithi023 bash[21372]: cluster 2024-03-11T21:06:11.049947+0000 osd.5 (osd.5) 288 : cluster 0 4.15 scrub ok 2024-03-11T21:06:12.869 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:12 smithi023 bash[21372]: cluster 2024-03-11T21:06:11.344555+0000 osd.3 (osd.3) 316 : cluster 0 3.8 scrub starts 2024-03-11T21:06:12.869 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:12 smithi023 bash[21372]: cluster 2024-03-11T21:06:11.347856+0000 osd.3 (osd.3) 317 : cluster 0 3.8 scrub ok 2024-03-11T21:06:12.878 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:06:12.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:12.880 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:06:12.981 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:13.015 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:13.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:13.019 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:06:13.120 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:13.146 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:13.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:13.149 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:06:13.250 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:13.278 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:13.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:13.283 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:06:13.383 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:06:13.401 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:06:13.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:13.403 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:06:13.506 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:06:13.532 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:06:13.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:13.534 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:06:13.634 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:06:13.644 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:13 smithi023 bash[21372]: cluster 2024-03-11T21:06:11.980744+0000 osd.5 (osd.5) 289 : cluster 0 3.c scrub starts 2024-03-11T21:06:13.644 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:13 smithi023 bash[21372]: cluster 2024-03-11T21:06:11.984950+0000 osd.5 (osd.5) 290 : cluster 0 3.c scrub ok 2024-03-11T21:06:13.644 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:13 smithi023 bash[21372]: cluster 2024-03-11T21:06:12.217509+0000 mgr.y (mgr.24380) 571 : cluster 0 pgmap v637: 148 pgs: 148 active+clean; 1.2 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.14k op/s 2024-03-11T21:06:13.644 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:13 smithi023 bash[21372]: cluster 2024-03-11T21:06:12.233411+0000 osd.2 (osd.2) 211 : cluster 0 2.10 scrub starts 2024-03-11T21:06:13.645 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:13 smithi023 bash[21372]: cluster 2024-03-11T21:06:12.235790+0000 osd.2 (osd.2) 212 : cluster 0 2.10 scrub ok 2024-03-11T21:06:13.645 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:13 smithi023 bash[21372]: cluster 2024-03-11T21:06:12.331873+0000 osd.3 (osd.3) 318 : cluster 0 9.2 scrub starts 2024-03-11T21:06:13.645 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:13 smithi023 bash[21372]: cluster 2024-03-11T21:06:12.528561+0000 osd.1 (osd.1) 97 : cluster 0 2.a scrub starts 2024-03-11T21:06:13.645 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:13 smithi023 bash[21372]: cluster 2024-03-11T21:06:12.536049+0000 osd.1 (osd.1) 98 : cluster 0 2.a scrub ok 2024-03-11T21:06:13.645 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:13 smithi023 bash[29146]: cluster 2024-03-11T21:06:11.980744+0000 osd.5 (osd.5) 289 : cluster 0 3.c scrub starts 2024-03-11T21:06:13.645 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:13 smithi023 bash[29146]: cluster 2024-03-11T21:06:11.984950+0000 osd.5 (osd.5) 290 : cluster 0 3.c scrub ok 2024-03-11T21:06:13.645 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:13 smithi023 bash[29146]: cluster 2024-03-11T21:06:12.217509+0000 mgr.y (mgr.24380) 571 : cluster 0 pgmap v637: 148 pgs: 148 active+clean; 1.2 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.14k op/s 2024-03-11T21:06:13.645 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:13 smithi023 bash[29146]: cluster 2024-03-11T21:06:12.233411+0000 osd.2 (osd.2) 211 : cluster 0 2.10 scrub starts 2024-03-11T21:06:13.645 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:13 smithi023 bash[29146]: cluster 2024-03-11T21:06:12.235790+0000 osd.2 (osd.2) 212 : cluster 0 2.10 scrub ok 2024-03-11T21:06:13.645 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:13 smithi023 bash[29146]: cluster 2024-03-11T21:06:12.331873+0000 osd.3 (osd.3) 318 : cluster 0 9.2 scrub starts 2024-03-11T21:06:13.645 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:13 smithi023 bash[29146]: cluster 2024-03-11T21:06:12.528561+0000 osd.1 (osd.1) 97 : cluster 0 2.a scrub starts 2024-03-11T21:06:13.645 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:13 smithi023 bash[29146]: cluster 2024-03-11T21:06:12.536049+0000 osd.1 (osd.1) 98 : cluster 0 2.a scrub ok 2024-03-11T21:06:13.663 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:06:13.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:13.665 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:06:13.766 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:06:13.789 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:06:13.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:13.795 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:06:13.895 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:06:13.905 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:13 smithi146 bash[23809]: cluster 2024-03-11T21:06:11.980744+0000 osd.5 (osd.5) 289 : cluster 0 3.c scrub starts 2024-03-11T21:06:13.906 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:13 smithi146 bash[23809]: cluster 2024-03-11T21:06:11.984950+0000 osd.5 (osd.5) 290 : cluster 0 3.c scrub ok 2024-03-11T21:06:13.906 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:13 smithi146 bash[23809]: cluster 2024-03-11T21:06:12.217509+0000 mgr.y (mgr.24380) 571 : cluster 0 pgmap v637: 148 pgs: 148 active+clean; 1.2 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.14k op/s 2024-03-11T21:06:13.906 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:13 smithi146 bash[23809]: cluster 2024-03-11T21:06:12.233411+0000 osd.2 (osd.2) 211 : cluster 0 2.10 scrub starts 2024-03-11T21:06:13.906 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:13 smithi146 bash[23809]: cluster 2024-03-11T21:06:12.235790+0000 osd.2 (osd.2) 212 : cluster 0 2.10 scrub ok 2024-03-11T21:06:13.906 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:13 smithi146 bash[23809]: cluster 2024-03-11T21:06:12.331873+0000 osd.3 (osd.3) 318 : cluster 0 9.2 scrub starts 2024-03-11T21:06:13.906 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:13 smithi146 bash[23809]: cluster 2024-03-11T21:06:12.528561+0000 osd.1 (osd.1) 97 : cluster 0 2.a scrub starts 2024-03-11T21:06:13.906 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:13 smithi146 bash[23809]: cluster 2024-03-11T21:06:12.536049+0000 osd.1 (osd.1) 98 : cluster 0 2.a scrub ok 2024-03-11T21:06:13.921 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:06:13.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:13.924 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:06:14.025 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:14.047 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:14.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:14.049 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:06:14.150 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:06:14.180 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:06:14.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:14.185 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:06:14.286 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:06:14.311 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:06:14.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:14.316 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:06:14.417 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:06:14.443 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:06:14.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:14.446 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:06:14.546 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:14.568 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:14.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:14.569 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:06:14.670 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:06:14.686 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:14 smithi023 bash[21372]: cluster 2024-03-11T21:06:12.910602+0000 osd.7 (osd.7) 333 : cluster 0 3.13 scrub starts 2024-03-11T21:06:14.686 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:14 smithi023 bash[21372]: cluster 2024-03-11T21:06:12.914342+0000 osd.7 (osd.7) 334 : cluster 0 3.13 scrub ok 2024-03-11T21:06:14.686 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:14 smithi023 bash[21372]: cluster 2024-03-11T21:06:13.027940+0000 osd.3 (osd.3) 319 : cluster 0 9.2 scrub ok 2024-03-11T21:06:14.686 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:14 smithi023 bash[29146]: cluster 2024-03-11T21:06:12.910602+0000 osd.7 (osd.7) 333 : cluster 0 3.13 scrub starts 2024-03-11T21:06:14.687 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:14 smithi023 bash[29146]: cluster 2024-03-11T21:06:12.914342+0000 osd.7 (osd.7) 334 : cluster 0 3.13 scrub ok 2024-03-11T21:06:14.687 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:14 smithi023 bash[29146]: cluster 2024-03-11T21:06:13.027940+0000 osd.3 (osd.3) 319 : cluster 0 9.2 scrub ok 2024-03-11T21:06:14.697 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:06:14.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:14.702 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:06:14.803 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:14.838 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:14.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:14.841 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:06:14.942 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:14.953 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:14 smithi146 bash[23809]: cluster 2024-03-11T21:06:12.910602+0000 osd.7 (osd.7) 333 : cluster 0 3.13 scrub starts 2024-03-11T21:06:14.953 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:14 smithi146 bash[23809]: cluster 2024-03-11T21:06:12.914342+0000 osd.7 (osd.7) 334 : cluster 0 3.13 scrub ok 2024-03-11T21:06:14.954 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:14 smithi146 bash[23809]: cluster 2024-03-11T21:06:13.027940+0000 osd.3 (osd.3) 319 : cluster 0 9.2 scrub ok 2024-03-11T21:06:14.966 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:14.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:14.970 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:06:15.071 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:15.097 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:15.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:15.099 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:06:15.200 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:06:15.233 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:06:15.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:15.235 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:06:15.336 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:15.358 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:15.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:15.366 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:06:15.466 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:06:15.495 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:06:15.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:15.497 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:06:15.598 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:06:15.627 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:06:15.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:15.633 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:06:15.734 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:06:15.742 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:06:15 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:06:15.742 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:15 smithi146 bash[23809]: cluster 2024-03-11T21:06:13.938174+0000 osd.7 (osd.7) 335 : cluster 0 3.15 scrub starts 2024-03-11T21:06:15.743 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:15 smithi146 bash[23809]: cluster 2024-03-11T21:06:13.947345+0000 osd.7 (osd.7) 336 : cluster 0 3.15 scrub ok 2024-03-11T21:06:15.743 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:15 smithi146 bash[23809]: cluster 2024-03-11T21:06:14.219428+0000 mgr.y (mgr.24380) 572 : cluster 0 pgmap v638: 148 pgs: 148 active+clean; 1.2 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.01k op/s 2024-03-11T21:06:15.743 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:15 smithi146 bash[23809]: cluster 2024-03-11T21:06:14.899332+0000 osd.0 (osd.0) 217 : cluster 0 4.6 scrub starts 2024-03-11T21:06:15.743 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:15 smithi146 bash[23809]: cluster 2024-03-11T21:06:14.986983+0000 osd.0 (osd.0) 218 : cluster 0 4.6 scrub ok 2024-03-11T21:06:15.768 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:06:15.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:15.771 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:06:15.872 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:06:15.882 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:15 smithi023 bash[21372]: cluster 2024-03-11T21:06:13.938174+0000 osd.7 (osd.7) 335 : cluster 0 3.15 scrub starts 2024-03-11T21:06:15.882 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:15 smithi023 bash[21372]: cluster 2024-03-11T21:06:13.947345+0000 osd.7 (osd.7) 336 : cluster 0 3.15 scrub ok 2024-03-11T21:06:15.882 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:15 smithi023 bash[21372]: cluster 2024-03-11T21:06:14.219428+0000 mgr.y (mgr.24380) 572 : cluster 0 pgmap v638: 148 pgs: 148 active+clean; 1.2 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.01k op/s 2024-03-11T21:06:15.882 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:15 smithi023 bash[21372]: cluster 2024-03-11T21:06:14.899332+0000 osd.0 (osd.0) 217 : cluster 0 4.6 scrub starts 2024-03-11T21:06:15.882 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:15 smithi023 bash[21372]: cluster 2024-03-11T21:06:14.986983+0000 osd.0 (osd.0) 218 : cluster 0 4.6 scrub ok 2024-03-11T21:06:15.883 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:15 smithi023 bash[29146]: cluster 2024-03-11T21:06:13.938174+0000 osd.7 (osd.7) 335 : cluster 0 3.15 scrub starts 2024-03-11T21:06:15.883 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:15 smithi023 bash[29146]: cluster 2024-03-11T21:06:13.947345+0000 osd.7 (osd.7) 336 : cluster 0 3.15 scrub ok 2024-03-11T21:06:15.883 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:15 smithi023 bash[29146]: cluster 2024-03-11T21:06:14.219428+0000 mgr.y (mgr.24380) 572 : cluster 0 pgmap v638: 148 pgs: 148 active+clean; 1.2 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.01k op/s 2024-03-11T21:06:15.883 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:15 smithi023 bash[29146]: cluster 2024-03-11T21:06:14.899332+0000 osd.0 (osd.0) 217 : cluster 0 4.6 scrub starts 2024-03-11T21:06:15.883 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:15 smithi023 bash[29146]: cluster 2024-03-11T21:06:14.986983+0000 osd.0 (osd.0) 218 : cluster 0 4.6 scrub ok 2024-03-11T21:06:15.903 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:06:15.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:15.905 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:06:16.006 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:16.049 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:16.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:16.055 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:06:16.156 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:06:16.182 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:06:16.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:16.190 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:06:16.291 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:06:16.311 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:06:16.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:16.312 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:06:16.413 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:06:16.438 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:06:16.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:16.444 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:06:16.545 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:06:16.579 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:06:16.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:16.585 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:06:16.686 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:06:16.708 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:06:16.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:16.712 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:06:16.782 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:16 smithi023 bash[21372]: cluster 2024-03-11T21:06:14.976968+0000 osd.7 (osd.7) 337 : cluster 0 2.f scrub starts 2024-03-11T21:06:16.782 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:16 smithi023 bash[21372]: cluster 2024-03-11T21:06:14.979069+0000 osd.7 (osd.7) 338 : cluster 0 2.f scrub ok 2024-03-11T21:06:16.782 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:16 smithi023 bash[21372]: cluster 2024-03-11T21:06:15.038641+0000 osd.5 (osd.5) 291 : cluster 0 3.17 scrub starts 2024-03-11T21:06:16.782 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:16 smithi023 bash[21372]: cluster 2024-03-11T21:06:15.040495+0000 osd.5 (osd.5) 292 : cluster 0 3.17 scrub ok 2024-03-11T21:06:16.782 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:16 smithi023 bash[21372]: audit 2024-03-11T21:06:15.600453+0000 mgr.y (mgr.24380) 573 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:06:16.783 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:16 smithi023 bash[29146]: cluster 2024-03-11T21:06:14.976968+0000 osd.7 (osd.7) 337 : cluster 0 2.f scrub starts 2024-03-11T21:06:16.783 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:16 smithi023 bash[29146]: cluster 2024-03-11T21:06:14.979069+0000 osd.7 (osd.7) 338 : cluster 0 2.f scrub ok 2024-03-11T21:06:16.783 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:16 smithi023 bash[29146]: cluster 2024-03-11T21:06:15.038641+0000 osd.5 (osd.5) 291 : cluster 0 3.17 scrub starts 2024-03-11T21:06:16.783 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:16 smithi023 bash[29146]: cluster 2024-03-11T21:06:15.040495+0000 osd.5 (osd.5) 292 : cluster 0 3.17 scrub ok 2024-03-11T21:06:16.783 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:16 smithi023 bash[29146]: audit 2024-03-11T21:06:15.600453+0000 mgr.y (mgr.24380) 573 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:06:16.812 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:16.845 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:16.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:16.848 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:06:16.949 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:06:16.958 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:16 smithi146 bash[23809]: cluster 2024-03-11T21:06:14.976968+0000 osd.7 (osd.7) 337 : cluster 0 2.f scrub starts 2024-03-11T21:06:16.958 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:16 smithi146 bash[23809]: cluster 2024-03-11T21:06:14.979069+0000 osd.7 (osd.7) 338 : cluster 0 2.f scrub ok 2024-03-11T21:06:16.958 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:16 smithi146 bash[23809]: cluster 2024-03-11T21:06:15.038641+0000 osd.5 (osd.5) 291 : cluster 0 3.17 scrub starts 2024-03-11T21:06:16.958 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:16 smithi146 bash[23809]: cluster 2024-03-11T21:06:15.040495+0000 osd.5 (osd.5) 292 : cluster 0 3.17 scrub ok 2024-03-11T21:06:16.958 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:16 smithi146 bash[23809]: audit 2024-03-11T21:06:15.600453+0000 mgr.y (mgr.24380) 573 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:06:16.982 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:06:16.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:16.984 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:06:17.085 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:06:17.125 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:06:17.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:17.129 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:06:17.230 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:06:17.267 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:06:17.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:17.271 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:06:17.372 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:06:17.407 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:06:17.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:17.408 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:06:17.509 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:06:17.529 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:06:17.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:17.530 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:06:17.631 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:06:17.641 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:17 smithi146 bash[23809]: cluster 2024-03-11T21:06:15.985908+0000 osd.7 (osd.7) 339 : cluster 0 2.1f scrub starts 2024-03-11T21:06:17.641 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:17 smithi146 bash[23809]: cluster 2024-03-11T21:06:15.987230+0000 osd.7 (osd.7) 340 : cluster 0 2.1f scrub ok 2024-03-11T21:06:17.641 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:17 smithi146 bash[23809]: cluster 2024-03-11T21:06:16.220026+0000 mgr.y (mgr.24380) 574 : cluster 0 pgmap v639: 148 pgs: 148 active+clean; 1.0 GiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.05k op/s 2024-03-11T21:06:17.641 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:17 smithi146 bash[23809]: cluster 2024-03-11T21:06:16.507274+0000 osd.3 (osd.3) 320 : cluster 0 4.17 scrub starts 2024-03-11T21:06:17.641 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:17 smithi146 bash[23809]: cluster 2024-03-11T21:06:16.515111+0000 osd.3 (osd.3) 321 : cluster 0 4.17 scrub ok 2024-03-11T21:06:17.654 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:06:17.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:17.656 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:06:17.757 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:06:17.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:17 smithi023 bash[21372]: cluster 2024-03-11T21:06:15.985908+0000 osd.7 (osd.7) 339 : cluster 0 2.1f scrub starts 2024-03-11T21:06:17.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:17 smithi023 bash[21372]: cluster 2024-03-11T21:06:15.987230+0000 osd.7 (osd.7) 340 : cluster 0 2.1f scrub ok 2024-03-11T21:06:17.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:17 smithi023 bash[21372]: cluster 2024-03-11T21:06:16.220026+0000 mgr.y (mgr.24380) 574 : cluster 0 pgmap v639: 148 pgs: 148 active+clean; 1.0 GiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.05k op/s 2024-03-11T21:06:17.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:17 smithi023 bash[21372]: cluster 2024-03-11T21:06:16.507274+0000 osd.3 (osd.3) 320 : cluster 0 4.17 scrub starts 2024-03-11T21:06:17.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:17 smithi023 bash[21372]: cluster 2024-03-11T21:06:16.515111+0000 osd.3 (osd.3) 321 : cluster 0 4.17 scrub ok 2024-03-11T21:06:17.770 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:17 smithi023 bash[29146]: cluster 2024-03-11T21:06:15.985908+0000 osd.7 (osd.7) 339 : cluster 0 2.1f scrub starts 2024-03-11T21:06:17.770 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:17 smithi023 bash[29146]: cluster 2024-03-11T21:06:15.987230+0000 osd.7 (osd.7) 340 : cluster 0 2.1f scrub ok 2024-03-11T21:06:17.770 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:17 smithi023 bash[29146]: cluster 2024-03-11T21:06:16.220026+0000 mgr.y (mgr.24380) 574 : cluster 0 pgmap v639: 148 pgs: 148 active+clean; 1.0 GiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.05k op/s 2024-03-11T21:06:17.770 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:17 smithi023 bash[29146]: cluster 2024-03-11T21:06:16.507274+0000 osd.3 (osd.3) 320 : cluster 0 4.17 scrub starts 2024-03-11T21:06:17.770 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:17 smithi023 bash[29146]: cluster 2024-03-11T21:06:16.515111+0000 osd.3 (osd.3) 321 : cluster 0 4.17 scrub ok 2024-03-11T21:06:17.784 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:06:17.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:17.786 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:06:17.887 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:06:17.918 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:06:17.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:17.922 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:06:18.023 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:06:18.055 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:06:18.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:18.058 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:06:18.159 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:18.189 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:18.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:18.194 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:06:18.295 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:06:18.319 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:06:18.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:18.326 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:06:18.427 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:18.457 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:18.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:18.460 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:06:18.561 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:06:18.582 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:06:18.586 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:06:18 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:06:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:06:18.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:18.588 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:06:18.689 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:18.713 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:18.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:18.717 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:06:18.781 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:18 smithi023 bash[21372]: cluster 2024-03-11T21:06:17.022167+0000 osd.7 (osd.7) 341 : cluster 0 3.f scrub starts 2024-03-11T21:06:18.781 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:18 smithi023 bash[21372]: cluster 2024-03-11T21:06:17.026562+0000 osd.7 (osd.7) 342 : cluster 0 3.f scrub ok 2024-03-11T21:06:18.781 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:18 smithi023 bash[21372]: cluster 2024-03-11T21:06:17.060028+0000 osd.5 (osd.5) 293 : cluster 0 4.1d scrub starts 2024-03-11T21:06:18.781 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:18 smithi023 bash[21372]: cluster 2024-03-11T21:06:17.063290+0000 osd.5 (osd.5) 294 : cluster 0 4.1d scrub ok 2024-03-11T21:06:18.781 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:18 smithi023 bash[21372]: cluster 2024-03-11T21:06:17.875548+0000 osd.1 (osd.1) 99 : cluster 0 2.4 scrub starts 2024-03-11T21:06:18.781 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:18 smithi023 bash[21372]: cluster 2024-03-11T21:06:17.889143+0000 osd.1 (osd.1) 100 : cluster 0 2.4 scrub ok 2024-03-11T21:06:18.782 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:18 smithi023 bash[29146]: cluster 2024-03-11T21:06:17.022167+0000 osd.7 (osd.7) 341 : cluster 0 3.f scrub starts 2024-03-11T21:06:18.782 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:18 smithi023 bash[29146]: cluster 2024-03-11T21:06:17.026562+0000 osd.7 (osd.7) 342 : cluster 0 3.f scrub ok 2024-03-11T21:06:18.782 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:18 smithi023 bash[29146]: cluster 2024-03-11T21:06:17.060028+0000 osd.5 (osd.5) 293 : cluster 0 4.1d scrub starts 2024-03-11T21:06:18.782 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:18 smithi023 bash[29146]: cluster 2024-03-11T21:06:17.063290+0000 osd.5 (osd.5) 294 : cluster 0 4.1d scrub ok 2024-03-11T21:06:18.782 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:18 smithi023 bash[29146]: cluster 2024-03-11T21:06:17.875548+0000 osd.1 (osd.1) 99 : cluster 0 2.4 scrub starts 2024-03-11T21:06:18.782 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:18 smithi023 bash[29146]: cluster 2024-03-11T21:06:17.889143+0000 osd.1 (osd.1) 100 : cluster 0 2.4 scrub ok 2024-03-11T21:06:18.818 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:06:18.826 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:18 smithi146 bash[23809]: cluster 2024-03-11T21:06:17.022167+0000 osd.7 (osd.7) 341 : cluster 0 3.f scrub starts 2024-03-11T21:06:18.827 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:18 smithi146 bash[23809]: cluster 2024-03-11T21:06:17.026562+0000 osd.7 (osd.7) 342 : cluster 0 3.f scrub ok 2024-03-11T21:06:18.827 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:18 smithi146 bash[23809]: cluster 2024-03-11T21:06:17.060028+0000 osd.5 (osd.5) 293 : cluster 0 4.1d scrub starts 2024-03-11T21:06:18.827 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:18 smithi146 bash[23809]: cluster 2024-03-11T21:06:17.063290+0000 osd.5 (osd.5) 294 : cluster 0 4.1d scrub ok 2024-03-11T21:06:18.827 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:18 smithi146 bash[23809]: cluster 2024-03-11T21:06:17.875548+0000 osd.1 (osd.1) 99 : cluster 0 2.4 scrub starts 2024-03-11T21:06:18.827 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:18 smithi146 bash[23809]: cluster 2024-03-11T21:06:17.889143+0000 osd.1 (osd.1) 100 : cluster 0 2.4 scrub ok 2024-03-11T21:06:18.852 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:06:18.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:18.855 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:06:18.955 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:18.981 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:18.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:18.983 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:06:19.084 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:06:19.116 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:06:19.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:19.119 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:06:19.220 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:19.258 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:19.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:19.263 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:06:19.364 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:19.390 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:19.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:19.393 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:06:19.494 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:19.523 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:19.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:19.526 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:06:19.626 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:19.643 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:19.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:19.646 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:06:19.746 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:06:19.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:19 smithi146 bash[23809]: cluster 2024-03-11T21:06:17.988588+0000 osd.6 (osd.6) 230 : cluster 0 3.10 scrub starts 2024-03-11T21:06:19.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:19 smithi146 bash[23809]: cluster 2024-03-11T21:06:17.989345+0000 osd.7 (osd.7) 343 : cluster 0 2.b scrub starts 2024-03-11T21:06:19.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:19 smithi146 bash[23809]: cluster 2024-03-11T21:06:17.992302+0000 osd.6 (osd.6) 231 : cluster 0 3.10 scrub ok 2024-03-11T21:06:19.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:19 smithi146 bash[23809]: cluster 2024-03-11T21:06:18.003079+0000 osd.7 (osd.7) 344 : cluster 0 2.b scrub ok 2024-03-11T21:06:19.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:19 smithi146 bash[23809]: cluster 2024-03-11T21:06:18.040000+0000 osd.5 (osd.5) 295 : cluster 0 4.3 scrub starts 2024-03-11T21:06:19.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:19 smithi146 bash[23809]: cluster 2024-03-11T21:06:18.041374+0000 osd.5 (osd.5) 296 : cluster 0 4.3 scrub ok 2024-03-11T21:06:19.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:19 smithi146 bash[23809]: cluster 2024-03-11T21:06:18.220640+0000 mgr.y (mgr.24380) 575 : cluster 0 pgmap v640: 148 pgs: 148 active+clean; 924 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 875 B/s rd, 0 B/s wr, 943 op/s 2024-03-11T21:06:19.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:19 smithi146 bash[23809]: cluster 2024-03-11T21:06:18.839138+0000 osd.1 (osd.1) 101 : cluster 0 6.e scrub starts 2024-03-11T21:06:19.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:19 smithi146 bash[23809]: cluster 2024-03-11T21:06:18.848218+0000 osd.1 (osd.1) 102 : cluster 0 6.e scrub ok 2024-03-11T21:06:19.776 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:06:19.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:19.779 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:06:19.781 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:19 smithi023 bash[21372]: cluster 2024-03-11T21:06:17.988588+0000 osd.6 (osd.6) 230 : cluster 0 3.10 scrub starts 2024-03-11T21:06:19.781 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:19 smithi023 bash[21372]: cluster 2024-03-11T21:06:17.989345+0000 osd.7 (osd.7) 343 : cluster 0 2.b scrub starts 2024-03-11T21:06:19.781 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:19 smithi023 bash[21372]: cluster 2024-03-11T21:06:17.992302+0000 osd.6 (osd.6) 231 : cluster 0 3.10 scrub ok 2024-03-11T21:06:19.781 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:19 smithi023 bash[21372]: cluster 2024-03-11T21:06:18.003079+0000 osd.7 (osd.7) 344 : cluster 0 2.b scrub ok 2024-03-11T21:06:19.781 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:19 smithi023 bash[21372]: cluster 2024-03-11T21:06:18.040000+0000 osd.5 (osd.5) 295 : cluster 0 4.3 scrub starts 2024-03-11T21:06:19.781 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:19 smithi023 bash[21372]: cluster 2024-03-11T21:06:18.041374+0000 osd.5 (osd.5) 296 : cluster 0 4.3 scrub ok 2024-03-11T21:06:19.781 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:19 smithi023 bash[21372]: cluster 2024-03-11T21:06:18.220640+0000 mgr.y (mgr.24380) 575 : cluster 0 pgmap v640: 148 pgs: 148 active+clean; 924 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 875 B/s rd, 0 B/s wr, 943 op/s 2024-03-11T21:06:19.781 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:19 smithi023 bash[21372]: cluster 2024-03-11T21:06:18.839138+0000 osd.1 (osd.1) 101 : cluster 0 6.e scrub starts 2024-03-11T21:06:19.781 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:19 smithi023 bash[21372]: cluster 2024-03-11T21:06:18.848218+0000 osd.1 (osd.1) 102 : cluster 0 6.e scrub ok 2024-03-11T21:06:19.782 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:19 smithi023 bash[29146]: cluster 2024-03-11T21:06:17.988588+0000 osd.6 (osd.6) 230 : cluster 0 3.10 scrub starts 2024-03-11T21:06:19.782 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:19 smithi023 bash[29146]: cluster 2024-03-11T21:06:17.989345+0000 osd.7 (osd.7) 343 : cluster 0 2.b scrub starts 2024-03-11T21:06:19.782 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:19 smithi023 bash[29146]: cluster 2024-03-11T21:06:17.992302+0000 osd.6 (osd.6) 231 : cluster 0 3.10 scrub ok 2024-03-11T21:06:19.782 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:19 smithi023 bash[29146]: cluster 2024-03-11T21:06:18.003079+0000 osd.7 (osd.7) 344 : cluster 0 2.b scrub ok 2024-03-11T21:06:19.782 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:19 smithi023 bash[29146]: cluster 2024-03-11T21:06:18.040000+0000 osd.5 (osd.5) 295 : cluster 0 4.3 scrub starts 2024-03-11T21:06:19.782 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:19 smithi023 bash[29146]: cluster 2024-03-11T21:06:18.041374+0000 osd.5 (osd.5) 296 : cluster 0 4.3 scrub ok 2024-03-11T21:06:19.782 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:19 smithi023 bash[29146]: cluster 2024-03-11T21:06:18.220640+0000 mgr.y (mgr.24380) 575 : cluster 0 pgmap v640: 148 pgs: 148 active+clean; 924 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 875 B/s rd, 0 B/s wr, 943 op/s 2024-03-11T21:06:19.782 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:19 smithi023 bash[29146]: cluster 2024-03-11T21:06:18.839138+0000 osd.1 (osd.1) 101 : cluster 0 6.e scrub starts 2024-03-11T21:06:19.782 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:19 smithi023 bash[29146]: cluster 2024-03-11T21:06:18.848218+0000 osd.1 (osd.1) 102 : cluster 0 6.e scrub ok 2024-03-11T21:06:19.880 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:19.908 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:19.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:19.911 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:06:20.012 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:20.033 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:20.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:20.035 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:06:20.136 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:06:20.162 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:06:20.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:20.166 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:06:20.267 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:06:20.297 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:06:20.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:20.302 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:06:20.403 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:20.429 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:20.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:20.434 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:06:20.534 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:06:20.574 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:06:20.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:20.580 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:06:20.681 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:06:20.692 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:20 smithi146 bash[23809]: cluster 2024-03-11T21:06:19.074486+0000 osd.5 (osd.5) 297 : cluster 0 2.3 scrub starts 2024-03-11T21:06:20.692 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:20 smithi146 bash[23809]: cluster 2024-03-11T21:06:19.075651+0000 osd.5 (osd.5) 298 : cluster 0 2.3 scrub ok 2024-03-11T21:06:20.692 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:20 smithi146 bash[23809]: cluster 2024-03-11T21:06:19.513458+0000 osd.3 (osd.3) 322 : cluster 0 3.b scrub starts 2024-03-11T21:06:20.692 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:20 smithi146 bash[23809]: cluster 2024-03-11T21:06:19.516289+0000 osd.3 (osd.3) 323 : cluster 0 3.b scrub ok 2024-03-11T21:06:20.693 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:20 smithi146 bash[23809]: cluster 2024-03-11T21:06:19.853177+0000 osd.1 (osd.1) 103 : cluster 0 4.4 scrub starts 2024-03-11T21:06:20.693 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:20 smithi146 bash[23809]: cluster 2024-03-11T21:06:19.855850+0000 osd.1 (osd.1) 104 : cluster 0 4.4 scrub ok 2024-03-11T21:06:20.693 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:20 smithi146 bash[23809]: audit 2024-03-11T21:06:20.506042+0000 mon.a (mon.0) 1567 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:06:20.716 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:06:20.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:20.721 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:06:20.822 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:06:20.832 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:20 smithi023 bash[21372]: cluster 2024-03-11T21:06:19.074486+0000 osd.5 (osd.5) 297 : cluster 0 2.3 scrub starts 2024-03-11T21:06:20.832 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:20 smithi023 bash[21372]: cluster 2024-03-11T21:06:19.075651+0000 osd.5 (osd.5) 298 : cluster 0 2.3 scrub ok 2024-03-11T21:06:20.832 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:20 smithi023 bash[21372]: cluster 2024-03-11T21:06:19.513458+0000 osd.3 (osd.3) 322 : cluster 0 3.b scrub starts 2024-03-11T21:06:20.832 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:20 smithi023 bash[21372]: cluster 2024-03-11T21:06:19.516289+0000 osd.3 (osd.3) 323 : cluster 0 3.b scrub ok 2024-03-11T21:06:20.832 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:20 smithi023 bash[21372]: cluster 2024-03-11T21:06:19.853177+0000 osd.1 (osd.1) 103 : cluster 0 4.4 scrub starts 2024-03-11T21:06:20.832 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:20 smithi023 bash[21372]: cluster 2024-03-11T21:06:19.855850+0000 osd.1 (osd.1) 104 : cluster 0 4.4 scrub ok 2024-03-11T21:06:20.832 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:20 smithi023 bash[21372]: audit 2024-03-11T21:06:20.506042+0000 mon.a (mon.0) 1567 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:06:20.833 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:20 smithi023 bash[29146]: cluster 2024-03-11T21:06:19.074486+0000 osd.5 (osd.5) 297 : cluster 0 2.3 scrub starts 2024-03-11T21:06:20.833 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:20 smithi023 bash[29146]: cluster 2024-03-11T21:06:19.075651+0000 osd.5 (osd.5) 298 : cluster 0 2.3 scrub ok 2024-03-11T21:06:20.833 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:20 smithi023 bash[29146]: cluster 2024-03-11T21:06:19.513458+0000 osd.3 (osd.3) 322 : cluster 0 3.b scrub starts 2024-03-11T21:06:20.833 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:20 smithi023 bash[29146]: cluster 2024-03-11T21:06:19.516289+0000 osd.3 (osd.3) 323 : cluster 0 3.b scrub ok 2024-03-11T21:06:20.833 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:20 smithi023 bash[29146]: cluster 2024-03-11T21:06:19.853177+0000 osd.1 (osd.1) 103 : cluster 0 4.4 scrub starts 2024-03-11T21:06:20.833 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:20 smithi023 bash[29146]: cluster 2024-03-11T21:06:19.855850+0000 osd.1 (osd.1) 104 : cluster 0 4.4 scrub ok 2024-03-11T21:06:20.833 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:20 smithi023 bash[29146]: audit 2024-03-11T21:06:20.506042+0000 mon.a (mon.0) 1567 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:06:20.847 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:06:20.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:20.851 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:06:20.952 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:20.979 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:20.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:20.985 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:06:21.086 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:06:21.114 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:06:21.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:21.118 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:06:21.219 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:06:21.241 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:06:21.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:21.243 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:06:21.344 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:21.389 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:21.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:21.392 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:06:21.493 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:06:21.510 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:06:21.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:21.512 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:06:21.613 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:21.643 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:21.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:21.645 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:06:21.746 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:21.773 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:21.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:21.778 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:06:21.879 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:21.889 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:21 smithi023 bash[21372]: cluster 2024-03-11T21:06:20.221092+0000 mgr.y (mgr.24380) 576 : cluster 0 pgmap v641: 148 pgs: 148 active+clean; 907 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 893 op/s 2024-03-11T21:06:21.890 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:21 smithi023 bash[21372]: cluster 2024-03-11T21:06:20.819722+0000 osd.0 (osd.0) 219 : cluster 0 4.b scrub starts 2024-03-11T21:06:21.890 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:21 smithi023 bash[21372]: cluster 2024-03-11T21:06:20.821698+0000 osd.0 (osd.0) 220 : cluster 0 4.b scrub ok 2024-03-11T21:06:21.890 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:21 smithi023 bash[29146]: cluster 2024-03-11T21:06:20.221092+0000 mgr.y (mgr.24380) 576 : cluster 0 pgmap v641: 148 pgs: 148 active+clean; 907 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 893 op/s 2024-03-11T21:06:21.894 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:21 smithi023 bash[29146]: cluster 2024-03-11T21:06:20.819722+0000 osd.0 (osd.0) 219 : cluster 0 4.b scrub starts 2024-03-11T21:06:21.894 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:21 smithi023 bash[29146]: cluster 2024-03-11T21:06:20.821698+0000 osd.0 (osd.0) 220 : cluster 0 4.b scrub ok 2024-03-11T21:06:21.906 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:21.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:21.909 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:06:22.010 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:22.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:21 smithi146 bash[23809]: cluster 2024-03-11T21:06:20.221092+0000 mgr.y (mgr.24380) 576 : cluster 0 pgmap v641: 148 pgs: 148 active+clean; 907 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 893 op/s 2024-03-11T21:06:22.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:21 smithi146 bash[23809]: cluster 2024-03-11T21:06:20.819722+0000 osd.0 (osd.0) 219 : cluster 0 4.b scrub starts 2024-03-11T21:06:22.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:21 smithi146 bash[23809]: cluster 2024-03-11T21:06:20.821698+0000 osd.0 (osd.0) 220 : cluster 0 4.b scrub ok 2024-03-11T21:06:22.037 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:22.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:22.043 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:06:22.144 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:06:22.166 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:06:22.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:22.168 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:06:22.269 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:06:22.293 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:06:22.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:22.295 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:06:22.396 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:06:22.417 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:06:22.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:22.419 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:06:22.520 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:22.558 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:22.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:22.563 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:06:22.664 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:22.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:22 smithi146 bash[23809]: cluster 2024-03-11T21:06:21.799932+0000 osd.0 (osd.0) 221 : cluster 0 3.1f scrub starts 2024-03-11T21:06:22.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:22 smithi146 bash[23809]: cluster 2024-03-11T21:06:21.802966+0000 osd.0 (osd.0) 222 : cluster 0 3.1f scrub ok 2024-03-11T21:06:22.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:22 smithi146 bash[23809]: cluster 2024-03-11T21:06:22.221928+0000 mgr.y (mgr.24380) 577 : cluster 0 pgmap v642: 148 pgs: 148 active+clean; 616 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.25k op/s 2024-03-11T21:06:22.696 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:22.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:22.699 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:06:22.781 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:22 smithi023 bash[21372]: cluster 2024-03-11T21:06:21.799932+0000 osd.0 (osd.0) 221 : cluster 0 3.1f scrub starts 2024-03-11T21:06:22.781 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:22 smithi023 bash[21372]: cluster 2024-03-11T21:06:21.802966+0000 osd.0 (osd.0) 222 : cluster 0 3.1f scrub ok 2024-03-11T21:06:22.781 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:22 smithi023 bash[21372]: cluster 2024-03-11T21:06:22.221928+0000 mgr.y (mgr.24380) 577 : cluster 0 pgmap v642: 148 pgs: 148 active+clean; 616 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.25k op/s 2024-03-11T21:06:22.781 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:22 smithi023 bash[29146]: cluster 2024-03-11T21:06:21.799932+0000 osd.0 (osd.0) 221 : cluster 0 3.1f scrub starts 2024-03-11T21:06:22.781 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:22 smithi023 bash[29146]: cluster 2024-03-11T21:06:21.802966+0000 osd.0 (osd.0) 222 : cluster 0 3.1f scrub ok 2024-03-11T21:06:22.781 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:22 smithi023 bash[29146]: cluster 2024-03-11T21:06:22.221928+0000 mgr.y (mgr.24380) 577 : cluster 0 pgmap v642: 148 pgs: 148 active+clean; 616 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.25k op/s 2024-03-11T21:06:22.800 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:22.827 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:22.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:22.830 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:06:22.931 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:06:22.956 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:06:22.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:22.962 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:06:23.062 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:06:23.091 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:06:23.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:23.095 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:06:23.197 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:23.230 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:23.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:23.237 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:06:23.338 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:23.371 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:23.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:23.377 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:06:23.478 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:06:23.479 INFO:teuthology.orchestra.run.smithi023.stdout:ERROR: (22) Invalid argument 2024-03-11T21:06:23.480 INFO:teuthology.orchestra.run.smithi023.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:06:23.509 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:06:23.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:23.513 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:06:23.614 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:06:23.643 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:06:23.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:23.648 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:06:23.749 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:06:23.758 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:23 smithi023 bash[21372]: cluster 2024-03-11T21:06:22.809220+0000 osd.1 (osd.1) 105 : cluster 0 4.12 scrub starts 2024-03-11T21:06:23.758 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:23 smithi023 bash[21372]: cluster 2024-03-11T21:06:22.817539+0000 osd.1 (osd.1) 106 : cluster 0 4.12 scrub ok 2024-03-11T21:06:23.758 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:23 smithi023 bash[29146]: cluster 2024-03-11T21:06:22.809220+0000 osd.1 (osd.1) 105 : cluster 0 4.12 scrub starts 2024-03-11T21:06:23.758 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:23 smithi023 bash[29146]: cluster 2024-03-11T21:06:22.817539+0000 osd.1 (osd.1) 106 : cluster 0 4.12 scrub ok 2024-03-11T21:06:23.777 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:06:23.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:23.779 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:06:23.880 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:06:23.909 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:06:23.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:23.912 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:06:24.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:23 smithi146 bash[23809]: cluster 2024-03-11T21:06:22.809220+0000 osd.1 (osd.1) 105 : cluster 0 4.12 scrub starts 2024-03-11T21:06:24.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:23 smithi146 bash[23809]: cluster 2024-03-11T21:06:22.817539+0000 osd.1 (osd.1) 106 : cluster 0 4.12 scrub ok 2024-03-11T21:06:24.013 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:24.051 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:24.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:24.054 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:06:24.155 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:06:24.176 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:06:24.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:24.182 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:06:24.283 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:24.307 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:24.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:24.316 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:06:24.416 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:24.441 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:24.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:24.452 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:06:24.553 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:06:24.580 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:06:24.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:24.584 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:06:24.685 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:24.712 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:24.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:24.714 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:06:24.815 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:06:24.825 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:24 smithi023 bash[21372]: cluster 2024-03-11T21:06:23.835326+0000 osd.0 (osd.0) 223 : cluster 0 3.6 scrub starts 2024-03-11T21:06:24.826 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:24 smithi023 bash[21372]: cluster 2024-03-11T21:06:23.850537+0000 osd.0 (osd.0) 224 : cluster 0 3.6 scrub ok 2024-03-11T21:06:24.826 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:24 smithi023 bash[21372]: cluster 2024-03-11T21:06:24.222357+0000 mgr.y (mgr.24380) 578 : cluster 0 pgmap v643: 148 pgs: 148 active+clean; 616 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 817 op/s 2024-03-11T21:06:24.826 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:24 smithi023 bash[29146]: cluster 2024-03-11T21:06:23.835326+0000 osd.0 (osd.0) 223 : cluster 0 3.6 scrub starts 2024-03-11T21:06:24.826 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:24 smithi023 bash[29146]: cluster 2024-03-11T21:06:23.850537+0000 osd.0 (osd.0) 224 : cluster 0 3.6 scrub ok 2024-03-11T21:06:24.826 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:24 smithi023 bash[29146]: cluster 2024-03-11T21:06:24.222357+0000 mgr.y (mgr.24380) 578 : cluster 0 pgmap v643: 148 pgs: 148 active+clean; 616 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 817 op/s 2024-03-11T21:06:24.842 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:06:24.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:24.844 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:06:24.945 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:06:24.976 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:06:24.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:24.979 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:06:25.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:24 smithi146 bash[23809]: cluster 2024-03-11T21:06:23.835326+0000 osd.0 (osd.0) 223 : cluster 0 3.6 scrub starts 2024-03-11T21:06:25.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:24 smithi146 bash[23809]: cluster 2024-03-11T21:06:23.850537+0000 osd.0 (osd.0) 224 : cluster 0 3.6 scrub ok 2024-03-11T21:06:25.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:24 smithi146 bash[23809]: cluster 2024-03-11T21:06:24.222357+0000 mgr.y (mgr.24380) 578 : cluster 0 pgmap v643: 148 pgs: 148 active+clean; 616 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 817 op/s 2024-03-11T21:06:25.079 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:06:25.114 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:06:25.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:25.119 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:06:25.219 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:06:25.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:25.253 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:06:25.254 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:06:25.354 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:06:25.385 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:06:25.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:25.391 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:06:25.491 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:25.524 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:25.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:25.525 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:06:25.626 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:06:25.634 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:06:25 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:06:25.646 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:06:25.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:25.648 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:06:25.748 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:25.771 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:25.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:25.775 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:06:25.879 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:25.908 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:25.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:25.912 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:06:26.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:25 smithi146 bash[23809]: cluster 2024-03-11T21:06:23.919111+0000 osd.5 (osd.5) 299 : cluster 0 6.19 scrub starts 2024-03-11T21:06:26.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:25 smithi146 bash[23809]: cluster 2024-03-11T21:06:23.971348+0000 osd.6 (osd.6) 232 : cluster 0 3.a scrub starts 2024-03-11T21:06:26.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:25 smithi146 bash[23809]: cluster 2024-03-11T21:06:23.973508+0000 osd.6 (osd.6) 233 : cluster 0 3.a scrub ok 2024-03-11T21:06:26.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:25 smithi146 bash[23809]: cluster 2024-03-11T21:06:23.997255+0000 osd.5 (osd.5) 300 : cluster 0 6.19 scrub ok 2024-03-11T21:06:26.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:25 smithi146 bash[23809]: cluster 2024-03-11T21:06:24.063185+0000 osd.2 (osd.2) 213 : cluster 0 4.1c scrub starts 2024-03-11T21:06:26.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:25 smithi146 bash[23809]: cluster 2024-03-11T21:06:24.067937+0000 osd.2 (osd.2) 214 : cluster 0 4.1c scrub ok 2024-03-11T21:06:26.013 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:25 smithi146 bash[23809]: audit 2024-03-11T21:06:25.612970+0000 mgr.y (mgr.24380) 579 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:06:26.013 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:06:26.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:25 smithi023 bash[21372]: cluster 2024-03-11T21:06:23.919111+0000 osd.5 (osd.5) 299 : cluster 0 6.19 scrub starts 2024-03-11T21:06:26.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:25 smithi023 bash[21372]: cluster 2024-03-11T21:06:23.971348+0000 osd.6 (osd.6) 232 : cluster 0 3.a scrub starts 2024-03-11T21:06:26.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:25 smithi023 bash[21372]: cluster 2024-03-11T21:06:23.973508+0000 osd.6 (osd.6) 233 : cluster 0 3.a scrub ok 2024-03-11T21:06:26.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:25 smithi023 bash[21372]: cluster 2024-03-11T21:06:23.997255+0000 osd.5 (osd.5) 300 : cluster 0 6.19 scrub ok 2024-03-11T21:06:26.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:25 smithi023 bash[21372]: cluster 2024-03-11T21:06:24.063185+0000 osd.2 (osd.2) 213 : cluster 0 4.1c scrub starts 2024-03-11T21:06:26.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:25 smithi023 bash[21372]: cluster 2024-03-11T21:06:24.067937+0000 osd.2 (osd.2) 214 : cluster 0 4.1c scrub ok 2024-03-11T21:06:26.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:25 smithi023 bash[21372]: audit 2024-03-11T21:06:25.612970+0000 mgr.y (mgr.24380) 579 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:06:26.031 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:25 smithi023 bash[29146]: cluster 2024-03-11T21:06:23.919111+0000 osd.5 (osd.5) 299 : cluster 0 6.19 scrub starts 2024-03-11T21:06:26.031 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:25 smithi023 bash[29146]: cluster 2024-03-11T21:06:23.971348+0000 osd.6 (osd.6) 232 : cluster 0 3.a scrub starts 2024-03-11T21:06:26.032 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:25 smithi023 bash[29146]: cluster 2024-03-11T21:06:23.973508+0000 osd.6 (osd.6) 233 : cluster 0 3.a scrub ok 2024-03-11T21:06:26.032 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:25 smithi023 bash[29146]: cluster 2024-03-11T21:06:23.997255+0000 osd.5 (osd.5) 300 : cluster 0 6.19 scrub ok 2024-03-11T21:06:26.032 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:25 smithi023 bash[29146]: cluster 2024-03-11T21:06:24.063185+0000 osd.2 (osd.2) 213 : cluster 0 4.1c scrub starts 2024-03-11T21:06:26.032 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:25 smithi023 bash[29146]: cluster 2024-03-11T21:06:24.067937+0000 osd.2 (osd.2) 214 : cluster 0 4.1c scrub ok 2024-03-11T21:06:26.032 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:25 smithi023 bash[29146]: audit 2024-03-11T21:06:25.612970+0000 mgr.y (mgr.24380) 579 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:06:26.034 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:06:26.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:26.037 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:06:26.138 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:06:26.161 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-03-11T21:06:26.168 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:06:26.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:26.173 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:06:26.274 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:26.297 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:26.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:26.301 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:06:26.402 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:26.442 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:26.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:26.446 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:06:26.547 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:06:26.578 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:06:26.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:26.586 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:06:26.686 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:06:26.729 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:06:26.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:26.736 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:06:26.781 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:26 smithi023 bash[21372]: cluster 2024-03-11T21:06:24.908716+0000 osd.5 (osd.5) 301 : cluster 0 3.1c scrub starts 2024-03-11T21:06:26.781 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:26 smithi023 bash[21372]: cluster 2024-03-11T21:06:24.913976+0000 osd.5 (osd.5) 302 : cluster 0 3.1c scrub ok 2024-03-11T21:06:26.781 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:26 smithi023 bash[21372]: cluster 2024-03-11T21:06:25.507026+0000 osd.3 (osd.3) 324 : cluster 0 6.6 scrub starts 2024-03-11T21:06:26.781 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:26 smithi023 bash[21372]: cluster 2024-03-11T21:06:25.518000+0000 osd.3 (osd.3) 325 : cluster 0 6.6 scrub ok 2024-03-11T21:06:26.781 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:26 smithi023 bash[21372]: cluster 2024-03-11T21:06:25.838370+0000 osd.1 (osd.1) 107 : cluster 0 4.2 scrub starts 2024-03-11T21:06:26.781 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:26 smithi023 bash[21372]: cluster 2024-03-11T21:06:25.845783+0000 osd.1 (osd.1) 108 : cluster 0 4.2 scrub ok 2024-03-11T21:06:26.781 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:26 smithi023 bash[21372]: cluster 2024-03-11T21:06:26.223011+0000 mgr.y (mgr.24380) 580 : cluster 0 pgmap v644: 148 pgs: 148 active+clean; 412 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.09k op/s 2024-03-11T21:06:26.782 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:26 smithi023 bash[29146]: cluster 2024-03-11T21:06:24.908716+0000 osd.5 (osd.5) 301 : cluster 0 3.1c scrub starts 2024-03-11T21:06:26.782 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:26 smithi023 bash[29146]: cluster 2024-03-11T21:06:24.913976+0000 osd.5 (osd.5) 302 : cluster 0 3.1c scrub ok 2024-03-11T21:06:26.782 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:26 smithi023 bash[29146]: cluster 2024-03-11T21:06:25.507026+0000 osd.3 (osd.3) 324 : cluster 0 6.6 scrub starts 2024-03-11T21:06:26.782 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:26 smithi023 bash[29146]: cluster 2024-03-11T21:06:25.518000+0000 osd.3 (osd.3) 325 : cluster 0 6.6 scrub ok 2024-03-11T21:06:26.782 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:26 smithi023 bash[29146]: cluster 2024-03-11T21:06:25.838370+0000 osd.1 (osd.1) 107 : cluster 0 4.2 scrub starts 2024-03-11T21:06:26.782 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:26 smithi023 bash[29146]: cluster 2024-03-11T21:06:25.845783+0000 osd.1 (osd.1) 108 : cluster 0 4.2 scrub ok 2024-03-11T21:06:26.782 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:26 smithi023 bash[29146]: cluster 2024-03-11T21:06:26.223011+0000 mgr.y (mgr.24380) 580 : cluster 0 pgmap v644: 148 pgs: 148 active+clean; 412 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.09k op/s 2024-03-11T21:06:26.837 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:26.865 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:26.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:26.870 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:06:26.910 INFO:teuthology.orchestra.run.smithi023.stdout:osd.0: {} 2024-03-11T21:06:26.952 INFO:teuthology.orchestra.run.smithi023.stdout:osd.1: {} 2024-03-11T21:06:26.971 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:06:26.998 INFO:teuthology.orchestra.run.smithi023.stdout:osd.2: {} 2024-03-11T21:06:27.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:26 smithi146 bash[23809]: cluster 2024-03-11T21:06:24.908716+0000 osd.5 (osd.5) 301 : cluster 0 3.1c scrub starts 2024-03-11T21:06:27.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:26 smithi146 bash[23809]: cluster 2024-03-11T21:06:24.913976+0000 osd.5 (osd.5) 302 : cluster 0 3.1c scrub ok 2024-03-11T21:06:27.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:26 smithi146 bash[23809]: cluster 2024-03-11T21:06:25.507026+0000 osd.3 (osd.3) 324 : cluster 0 6.6 scrub starts 2024-03-11T21:06:27.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:26 smithi146 bash[23809]: cluster 2024-03-11T21:06:25.518000+0000 osd.3 (osd.3) 325 : cluster 0 6.6 scrub ok 2024-03-11T21:06:27.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:26 smithi146 bash[23809]: cluster 2024-03-11T21:06:25.838370+0000 osd.1 (osd.1) 107 : cluster 0 4.2 scrub starts 2024-03-11T21:06:27.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:26 smithi146 bash[23809]: cluster 2024-03-11T21:06:25.845783+0000 osd.1 (osd.1) 108 : cluster 0 4.2 scrub ok 2024-03-11T21:06:27.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:26 smithi146 bash[23809]: cluster 2024-03-11T21:06:26.223011+0000 mgr.y (mgr.24380) 580 : cluster 0 pgmap v644: 148 pgs: 148 active+clean; 412 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.09k op/s 2024-03-11T21:06:27.015 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:06:27.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:27.018 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:06:27.049 INFO:teuthology.orchestra.run.smithi023.stdout:osd.3: {} 2024-03-11T21:06:27.084 INFO:teuthology.orchestra.run.smithi023.stdout:osd.4: {} 2024-03-11T21:06:27.117 INFO:teuthology.orchestra.run.smithi023.stdout:osd.5: {} 2024-03-11T21:06:27.119 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:06:27.147 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:06:27.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:27.149 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:06:27.152 INFO:teuthology.orchestra.run.smithi023.stdout:osd.6: {} 2024-03-11T21:06:27.186 INFO:teuthology.orchestra.run.smithi023.stdout:osd.7: {} 2024-03-11T21:06:27.189 INFO:teuthology.orchestra.run.smithi023.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:06:27.189 INFO:teuthology.orchestra.run.smithi023.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:06:27.189 INFO:teuthology.orchestra.run.smithi023.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:06:27.189 INFO:teuthology.orchestra.run.smithi023.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:06:27.189 INFO:teuthology.orchestra.run.smithi023.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:06:27.190 INFO:teuthology.orchestra.run.smithi023.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:06:27.190 INFO:teuthology.orchestra.run.smithi023.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:06:27.190 INFO:teuthology.orchestra.run.smithi023.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:06:27.249 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:27.274 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:27.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:27.282 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:06:27.382 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:06:27.405 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:06:27.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:27.411 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:06:27.512 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:06:27.538 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:06:27.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:27.541 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:06:27.642 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:06:27.668 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:06:27.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:27.671 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:06:27.717 INFO:teuthology.orchestra.run.smithi023.stderr:nodeep-scrub is unset 2024-03-11T21:06:27.724 INFO:teuthology.orchestra.run.smithi023.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-03-11T21:06:27.772 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:27.781 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:27 smithi023 bash[21372]: cluster 2024-03-11T21:06:26.528204+0000 osd.3 (osd.3) 326 : cluster 0 2.1e scrub starts 2024-03-11T21:06:27.781 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:27 smithi023 bash[21372]: cluster 2024-03-11T21:06:26.534038+0000 osd.3 (osd.3) 327 : cluster 0 2.1e scrub ok 2024-03-11T21:06:27.781 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:27 smithi023 bash[21372]: audit 2024-03-11T21:06:26.855760+0000 mon.a (mon.0) 1568 : audit 0 from='client.? 172.21.15.23:0/3183091206' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:06:27.781 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:27 smithi023 bash[21372]: cluster 2024-03-11T21:06:27.002702+0000 osd.7 (osd.7) 345 : cluster 0 1.0 scrub starts 2024-03-11T21:06:27.782 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:27 smithi023 bash[21372]: cluster 2024-03-11T21:06:27.008689+0000 osd.7 (osd.7) 346 : cluster 0 1.0 scrub ok 2024-03-11T21:06:27.782 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:27 smithi023 bash[21372]: audit 2024-03-11T21:06:27.227287+0000 mon.c (mon.2) 73 : audit 1 from='client.? 172.21.15.23:0/114351649' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-03-11T21:06:27.782 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:27 smithi023 bash[21372]: audit 2024-03-11T21:06:27.227880+0000 mon.a (mon.0) 1569 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-03-11T21:06:27.782 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:27 smithi023 bash[21372]: audit 2024-03-11T21:06:27.261527+0000 mon.b (mon.1) 40 : audit 1 from='client.? 172.21.15.23:0/2700258791' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:06:27.782 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:27 smithi023 bash[21372]: audit 2024-03-11T21:06:27.261807+0000 mon.a (mon.0) 1570 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:06:27.782 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:27 smithi023 bash[29146]: cluster 2024-03-11T21:06:26.528204+0000 osd.3 (osd.3) 326 : cluster 0 2.1e scrub starts 2024-03-11T21:06:27.783 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:27 smithi023 bash[29146]: cluster 2024-03-11T21:06:26.534038+0000 osd.3 (osd.3) 327 : cluster 0 2.1e scrub ok 2024-03-11T21:06:27.783 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:27 smithi023 bash[29146]: audit 2024-03-11T21:06:26.855760+0000 mon.a (mon.0) 1568 : audit 0 from='client.? 172.21.15.23:0/3183091206' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:06:27.783 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:27 smithi023 bash[29146]: cluster 2024-03-11T21:06:27.002702+0000 osd.7 (osd.7) 345 : cluster 0 1.0 scrub starts 2024-03-11T21:06:27.783 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:27 smithi023 bash[29146]: cluster 2024-03-11T21:06:27.008689+0000 osd.7 (osd.7) 346 : cluster 0 1.0 scrub ok 2024-03-11T21:06:27.783 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:27 smithi023 bash[29146]: audit 2024-03-11T21:06:27.227287+0000 mon.c (mon.2) 73 : audit 1 from='client.? 172.21.15.23:0/114351649' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-03-11T21:06:27.783 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:27 smithi023 bash[29146]: audit 2024-03-11T21:06:27.227880+0000 mon.a (mon.0) 1569 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-03-11T21:06:27.783 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:27 smithi023 bash[29146]: audit 2024-03-11T21:06:27.261527+0000 mon.b (mon.1) 40 : audit 1 from='client.? 172.21.15.23:0/2700258791' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:06:27.783 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:27 smithi023 bash[29146]: audit 2024-03-11T21:06:27.261807+0000 mon.a (mon.0) 1570 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:06:27.794 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:27.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:27.797 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:06:27.897 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:27.919 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:27.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:27.922 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:06:28.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:27 smithi146 bash[23809]: cluster 2024-03-11T21:06:26.528204+0000 osd.3 (osd.3) 326 : cluster 0 2.1e scrub starts 2024-03-11T21:06:28.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:27 smithi146 bash[23809]: cluster 2024-03-11T21:06:26.534038+0000 osd.3 (osd.3) 327 : cluster 0 2.1e scrub ok 2024-03-11T21:06:28.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:27 smithi146 bash[23809]: audit 2024-03-11T21:06:26.855760+0000 mon.a (mon.0) 1568 : audit 0 from='client.? 172.21.15.23:0/3183091206' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:06:28.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:27 smithi146 bash[23809]: cluster 2024-03-11T21:06:27.002702+0000 osd.7 (osd.7) 345 : cluster 0 1.0 scrub starts 2024-03-11T21:06:28.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:27 smithi146 bash[23809]: cluster 2024-03-11T21:06:27.008689+0000 osd.7 (osd.7) 346 : cluster 0 1.0 scrub ok 2024-03-11T21:06:28.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:27 smithi146 bash[23809]: audit 2024-03-11T21:06:27.227287+0000 mon.c (mon.2) 73 : audit 1 from='client.? 172.21.15.23:0/114351649' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-03-11T21:06:28.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:27 smithi146 bash[23809]: audit 2024-03-11T21:06:27.227880+0000 mon.a (mon.0) 1569 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-03-11T21:06:28.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:27 smithi146 bash[23809]: audit 2024-03-11T21:06:27.261527+0000 mon.b (mon.1) 40 : audit 1 from='client.? 172.21.15.23:0/2700258791' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:06:28.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:27 smithi146 bash[23809]: audit 2024-03-11T21:06:27.261807+0000 mon.a (mon.0) 1570 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:06:28.023 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:28.049 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:28.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:28.052 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:06:28.153 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:06:28.181 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:06:28.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:28.183 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:06:28.284 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:06:28.311 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:06:28.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:28.314 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:06:28.414 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:28.442 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:28.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:28.445 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:06:28.546 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:28.557 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:06:28 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:06:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:06:28.575 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:28.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:28.578 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:06:28.678 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:06:28.707 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:06:28.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:28.709 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:06:28.810 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:28.840 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:28.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:28.844 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:06:28.945 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:28.952 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:28 smithi023 bash[21372]: cluster 2024-03-11T21:06:26.986575+0000 osd.6 (osd.6) 234 : cluster 0 2.1a scrub starts 2024-03-11T21:06:28.952 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:28 smithi023 bash[21372]: cluster 2024-03-11T21:06:26.987811+0000 osd.6 (osd.6) 235 : cluster 0 2.1a scrub ok 2024-03-11T21:06:28.953 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:28 smithi023 bash[21372]: cluster 2024-03-11T21:06:26.990488+0000 osd.5 (osd.5) 303 : cluster 0 3.16 scrub starts 2024-03-11T21:06:28.953 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:28 smithi023 bash[21372]: cluster 2024-03-11T21:06:27.015129+0000 osd.5 (osd.5) 304 : cluster 0 3.16 scrub ok 2024-03-11T21:06:28.953 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:28 smithi023 bash[21372]: cluster 2024-03-11T21:06:27.705968+0000 mon.a (mon.0) 1571 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:06:28.953 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:28 smithi023 bash[21372]: cluster 2024-03-11T21:06:27.706007+0000 mon.a (mon.0) 1572 : cluster 1 Cluster is now healthy 2024-03-11T21:06:28.953 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:28 smithi023 bash[21372]: audit 2024-03-11T21:06:27.707416+0000 mon.a (mon.0) 1573 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-03-11T21:06:28.953 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:28 smithi023 bash[21372]: audit 2024-03-11T21:06:27.707631+0000 mon.a (mon.0) 1574 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:06:28.953 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:28 smithi023 bash[21372]: cluster 2024-03-11T21:06:27.721340+0000 mon.a (mon.0) 1575 : cluster 0 osdmap e297: 8 total, 8 up, 8 in 2024-03-11T21:06:28.953 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:28 smithi023 bash[21372]: cluster 2024-03-11T21:06:28.040911+0000 osd.7 (osd.7) 347 : cluster 0 2.5 scrub starts 2024-03-11T21:06:28.953 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:28 smithi023 bash[21372]: cluster 2024-03-11T21:06:28.042194+0000 osd.7 (osd.7) 348 : cluster 0 2.5 scrub ok 2024-03-11T21:06:28.954 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:28 smithi023 bash[21372]: cluster 2024-03-11T21:06:28.223764+0000 mgr.y (mgr.24380) 581 : cluster 0 pgmap v646: 148 pgs: 148 active+clean; 298 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.00k op/s 2024-03-11T21:06:28.954 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:28 smithi023 bash[29146]: cluster 2024-03-11T21:06:26.986575+0000 osd.6 (osd.6) 234 : cluster 0 2.1a scrub starts 2024-03-11T21:06:28.954 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:28 smithi023 bash[29146]: cluster 2024-03-11T21:06:26.987811+0000 osd.6 (osd.6) 235 : cluster 0 2.1a scrub ok 2024-03-11T21:06:28.954 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:28 smithi023 bash[29146]: cluster 2024-03-11T21:06:26.990488+0000 osd.5 (osd.5) 303 : cluster 0 3.16 scrub starts 2024-03-11T21:06:28.954 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:28 smithi023 bash[29146]: cluster 2024-03-11T21:06:27.015129+0000 osd.5 (osd.5) 304 : cluster 0 3.16 scrub ok 2024-03-11T21:06:28.954 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:28 smithi023 bash[29146]: cluster 2024-03-11T21:06:27.705968+0000 mon.a (mon.0) 1571 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:06:28.955 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:28 smithi023 bash[29146]: cluster 2024-03-11T21:06:27.706007+0000 mon.a (mon.0) 1572 : cluster 1 Cluster is now healthy 2024-03-11T21:06:28.955 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:28 smithi023 bash[29146]: audit 2024-03-11T21:06:27.707416+0000 mon.a (mon.0) 1573 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-03-11T21:06:28.955 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:28 smithi023 bash[29146]: audit 2024-03-11T21:06:27.707631+0000 mon.a (mon.0) 1574 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:06:28.955 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:28 smithi023 bash[29146]: cluster 2024-03-11T21:06:27.721340+0000 mon.a (mon.0) 1575 : cluster 0 osdmap e297: 8 total, 8 up, 8 in 2024-03-11T21:06:28.955 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:28 smithi023 bash[29146]: cluster 2024-03-11T21:06:28.040911+0000 osd.7 (osd.7) 347 : cluster 0 2.5 scrub starts 2024-03-11T21:06:28.955 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:28 smithi023 bash[29146]: cluster 2024-03-11T21:06:28.042194+0000 osd.7 (osd.7) 348 : cluster 0 2.5 scrub ok 2024-03-11T21:06:28.955 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:28 smithi023 bash[29146]: cluster 2024-03-11T21:06:28.223764+0000 mgr.y (mgr.24380) 581 : cluster 0 pgmap v646: 148 pgs: 148 active+clean; 298 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.00k op/s 2024-03-11T21:06:28.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:28.975 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:28.976 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:06:29.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:28 smithi146 bash[23809]: cluster 2024-03-11T21:06:26.986575+0000 osd.6 (osd.6) 234 : cluster 0 2.1a scrub starts 2024-03-11T21:06:29.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:28 smithi146 bash[23809]: cluster 2024-03-11T21:06:26.987811+0000 osd.6 (osd.6) 235 : cluster 0 2.1a scrub ok 2024-03-11T21:06:29.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:28 smithi146 bash[23809]: cluster 2024-03-11T21:06:26.990488+0000 osd.5 (osd.5) 303 : cluster 0 3.16 scrub starts 2024-03-11T21:06:29.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:28 smithi146 bash[23809]: cluster 2024-03-11T21:06:27.015129+0000 osd.5 (osd.5) 304 : cluster 0 3.16 scrub ok 2024-03-11T21:06:29.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:28 smithi146 bash[23809]: cluster 2024-03-11T21:06:27.705968+0000 mon.a (mon.0) 1571 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:06:29.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:28 smithi146 bash[23809]: cluster 2024-03-11T21:06:27.706007+0000 mon.a (mon.0) 1572 : cluster 1 Cluster is now healthy 2024-03-11T21:06:29.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:28 smithi146 bash[23809]: audit 2024-03-11T21:06:27.707416+0000 mon.a (mon.0) 1573 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-03-11T21:06:29.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:28 smithi146 bash[23809]: audit 2024-03-11T21:06:27.707631+0000 mon.a (mon.0) 1574 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:06:29.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:28 smithi146 bash[23809]: cluster 2024-03-11T21:06:27.721340+0000 mon.a (mon.0) 1575 : cluster 0 osdmap e297: 8 total, 8 up, 8 in 2024-03-11T21:06:29.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:28 smithi146 bash[23809]: cluster 2024-03-11T21:06:28.040911+0000 osd.7 (osd.7) 347 : cluster 0 2.5 scrub starts 2024-03-11T21:06:29.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:28 smithi146 bash[23809]: cluster 2024-03-11T21:06:28.042194+0000 osd.7 (osd.7) 348 : cluster 0 2.5 scrub ok 2024-03-11T21:06:29.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:28 smithi146 bash[23809]: cluster 2024-03-11T21:06:28.223764+0000 mgr.y (mgr.24380) 581 : cluster 0 pgmap v646: 148 pgs: 148 active+clean; 298 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.00k op/s 2024-03-11T21:06:29.077 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:06:29.111 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:06:29.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:29.115 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:06:29.216 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:29.238 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:29.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:29.240 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:06:29.340 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:29.369 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:29.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:29.374 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:06:29.476 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:29.501 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:29.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:29.506 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:06:29.607 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:06:29.626 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:06:29.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:29.627 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:06:29.728 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:06:29.747 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:06:29.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:29.749 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:06:29.850 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:06:29.861 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:29 smithi023 bash[29146]: cluster 2024-03-11T21:06:27.998479+0000 osd.5 (osd.5) 305 : cluster 0 6.1a scrub starts 2024-03-11T21:06:29.861 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:29 smithi023 bash[29146]: cluster 2024-03-11T21:06:28.005263+0000 osd.5 (osd.5) 306 : cluster 0 6.1a scrub ok 2024-03-11T21:06:29.861 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:29 smithi023 bash[29146]: cluster 2024-03-11T21:06:28.745385+0000 mon.a (mon.0) 1576 : cluster 0 osdmap e298: 8 total, 8 up, 8 in 2024-03-11T21:06:29.861 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:29 smithi023 bash[29146]: cluster 2024-03-11T21:06:29.091613+0000 osd.7 (osd.7) 349 : cluster 0 6.5 scrub starts 2024-03-11T21:06:29.861 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:29 smithi023 bash[29146]: cluster 2024-03-11T21:06:29.140967+0000 osd.7 (osd.7) 350 : cluster 0 6.5 scrub ok 2024-03-11T21:06:29.862 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:29 smithi023 bash[21372]: cluster 2024-03-11T21:06:27.998479+0000 osd.5 (osd.5) 305 : cluster 0 6.1a scrub starts 2024-03-11T21:06:29.862 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:29 smithi023 bash[21372]: cluster 2024-03-11T21:06:28.005263+0000 osd.5 (osd.5) 306 : cluster 0 6.1a scrub ok 2024-03-11T21:06:29.863 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:29 smithi023 bash[21372]: cluster 2024-03-11T21:06:28.745385+0000 mon.a (mon.0) 1576 : cluster 0 osdmap e298: 8 total, 8 up, 8 in 2024-03-11T21:06:29.863 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:29 smithi023 bash[21372]: cluster 2024-03-11T21:06:29.091613+0000 osd.7 (osd.7) 349 : cluster 0 6.5 scrub starts 2024-03-11T21:06:29.863 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:29 smithi023 bash[21372]: cluster 2024-03-11T21:06:29.140967+0000 osd.7 (osd.7) 350 : cluster 0 6.5 scrub ok 2024-03-11T21:06:29.878 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:06:29.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:29.882 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:06:29.983 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:29.993 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:29 smithi146 bash[23809]: cluster 2024-03-11T21:06:27.998479+0000 osd.5 (osd.5) 305 : cluster 0 6.1a scrub starts 2024-03-11T21:06:29.993 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:29 smithi146 bash[23809]: cluster 2024-03-11T21:06:28.005263+0000 osd.5 (osd.5) 306 : cluster 0 6.1a scrub ok 2024-03-11T21:06:29.993 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:29 smithi146 bash[23809]: cluster 2024-03-11T21:06:28.745385+0000 mon.a (mon.0) 1576 : cluster 0 osdmap e298: 8 total, 8 up, 8 in 2024-03-11T21:06:29.993 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:29 smithi146 bash[23809]: cluster 2024-03-11T21:06:29.091613+0000 osd.7 (osd.7) 349 : cluster 0 6.5 scrub starts 2024-03-11T21:06:29.993 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:29 smithi146 bash[23809]: cluster 2024-03-11T21:06:29.140967+0000 osd.7 (osd.7) 350 : cluster 0 6.5 scrub ok 2024-03-11T21:06:30.013 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:30.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:30.019 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:06:30.120 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:30.145 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:30.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:30.149 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:06:30.250 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:30.273 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:30.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:30.279 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:06:30.380 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:30.406 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:30.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:30.409 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:06:30.510 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:06:30.539 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:06:30.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:30.541 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:06:30.642 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:30.669 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:30.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:30.674 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:06:30.775 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:06:30.781 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:30 smithi023 bash[21372]: cluster 2024-03-11T21:06:29.041405+0000 osd.5 (osd.5) 307 : cluster 0 3.1d scrub starts 2024-03-11T21:06:30.781 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:30 smithi023 bash[21372]: cluster 2024-03-11T21:06:29.043954+0000 osd.5 (osd.5) 308 : cluster 0 3.1d scrub ok 2024-03-11T21:06:30.781 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:30 smithi023 bash[21372]: cluster 2024-03-11T21:06:30.224517+0000 mgr.y (mgr.24380) 582 : cluster 0 pgmap v648: 148 pgs: 148 active+clean; 298 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 634 op/s 2024-03-11T21:06:30.781 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:30 smithi023 bash[29146]: cluster 2024-03-11T21:06:29.041405+0000 osd.5 (osd.5) 307 : cluster 0 3.1d scrub starts 2024-03-11T21:06:30.781 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:30 smithi023 bash[29146]: cluster 2024-03-11T21:06:29.043954+0000 osd.5 (osd.5) 308 : cluster 0 3.1d scrub ok 2024-03-11T21:06:30.781 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:30 smithi023 bash[29146]: cluster 2024-03-11T21:06:30.224517+0000 mgr.y (mgr.24380) 582 : cluster 0 pgmap v648: 148 pgs: 148 active+clean; 298 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 634 op/s 2024-03-11T21:06:30.801 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:06:30.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:30.807 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:06:30.908 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:06:30.932 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:06:30.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:30.937 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:06:31.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:30 smithi146 bash[23809]: cluster 2024-03-11T21:06:29.041405+0000 osd.5 (osd.5) 307 : cluster 0 3.1d scrub starts 2024-03-11T21:06:31.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:30 smithi146 bash[23809]: cluster 2024-03-11T21:06:29.043954+0000 osd.5 (osd.5) 308 : cluster 0 3.1d scrub ok 2024-03-11T21:06:31.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:30 smithi146 bash[23809]: cluster 2024-03-11T21:06:30.224517+0000 mgr.y (mgr.24380) 582 : cluster 0 pgmap v648: 148 pgs: 148 active+clean; 298 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 634 op/s 2024-03-11T21:06:31.038 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:31.062 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:31.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:31.069 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:06:31.170 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:06:31.203 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:06:31.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:31.210 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:06:31.310 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:31.344 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:31.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:31.347 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:06:31.448 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:31.474 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:31.475 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T21:06:31.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:31.478 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:06:31.579 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:31.613 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:31.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:31.622 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:06:31.705 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:Removed 111143 objects 2024-03-11T21:06:31.706 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:Clean up completed and total clean up time :119.236 2024-03-11T21:06:31.723 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:06:31.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:31.742 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:06:31.742 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:06:31.843 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:06:31.866 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:06:31.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:31.868 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:06:31.969 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:06:31.996 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:06:31.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:31.999 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:06:32.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:31 smithi146 bash[23809]: cluster 2024-03-11T21:06:30.554648+0000 osd.3 (osd.3) 328 : cluster 0 6.c scrub starts 2024-03-11T21:06:32.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:31 smithi146 bash[23809]: cluster 2024-03-11T21:06:30.558247+0000 osd.3 (osd.3) 329 : cluster 0 6.c scrub ok 2024-03-11T21:06:32.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:31 smithi146 bash[23809]: cluster 2024-03-11T21:06:30.979096+0000 osd.0 (osd.0) 225 : cluster 0 9.b scrub starts 2024-03-11T21:06:32.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:31 smithi146 bash[23809]: cluster 2024-03-11T21:06:31.016338+0000 osd.0 (osd.0) 226 : cluster 0 9.b scrub ok 2024-03-11T21:06:32.030 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:31 smithi023 bash[29146]: cluster 2024-03-11T21:06:30.554648+0000 osd.3 (osd.3) 328 : cluster 0 6.c scrub starts 2024-03-11T21:06:32.031 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:31 smithi023 bash[29146]: cluster 2024-03-11T21:06:30.558247+0000 osd.3 (osd.3) 329 : cluster 0 6.c scrub ok 2024-03-11T21:06:32.031 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:31 smithi023 bash[29146]: cluster 2024-03-11T21:06:30.979096+0000 osd.0 (osd.0) 225 : cluster 0 9.b scrub starts 2024-03-11T21:06:32.031 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:31 smithi023 bash[29146]: cluster 2024-03-11T21:06:31.016338+0000 osd.0 (osd.0) 226 : cluster 0 9.b scrub ok 2024-03-11T21:06:32.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:31 smithi023 bash[21372]: cluster 2024-03-11T21:06:30.554648+0000 osd.3 (osd.3) 328 : cluster 0 6.c scrub starts 2024-03-11T21:06:32.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:31 smithi023 bash[21372]: cluster 2024-03-11T21:06:30.558247+0000 osd.3 (osd.3) 329 : cluster 0 6.c scrub ok 2024-03-11T21:06:32.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:31 smithi023 bash[21372]: cluster 2024-03-11T21:06:30.979096+0000 osd.0 (osd.0) 225 : cluster 0 9.b scrub starts 2024-03-11T21:06:32.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:31 smithi023 bash[21372]: cluster 2024-03-11T21:06:31.016338+0000 osd.0 (osd.0) 226 : cluster 0 9.b scrub ok 2024-03-11T21:06:32.100 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:06:32.127 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:06:32.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:32.129 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:06:32.230 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:06:32.250 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:06:32.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:32.252 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:06:32.353 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:06:32.381 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:06:32.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:32.383 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:06:32.484 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:32.511 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:32.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:32.513 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:06:32.614 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:32.642 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:32.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:32.644 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:06:32.703 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd set noscrub 2024-03-11T21:06:32.745 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:06:32.771 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:06:32.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:32.773 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:06:32.874 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:32.899 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:32.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:32.901 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:06:33.002 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:33.010 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:32 smithi023 bash[29146]: cluster 2024-03-11T21:06:31.529649+0000 osd.3 (osd.3) 330 : cluster 0 6.2 scrub starts 2024-03-11T21:06:33.010 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:32 smithi023 bash[29146]: cluster 2024-03-11T21:06:31.535013+0000 osd.3 (osd.3) 331 : cluster 0 6.2 scrub ok 2024-03-11T21:06:33.010 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:32 smithi023 bash[29146]: cluster 2024-03-11T21:06:32.049879+0000 osd.7 (osd.7) 351 : cluster 0 6.8 scrub starts 2024-03-11T21:06:33.011 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:32 smithi023 bash[29146]: cluster 2024-03-11T21:06:32.052031+0000 osd.7 (osd.7) 352 : cluster 0 6.8 scrub ok 2024-03-11T21:06:33.011 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:32 smithi023 bash[29146]: cluster 2024-03-11T21:06:32.225323+0000 mgr.y (mgr.24380) 583 : cluster 0 pgmap v649: 148 pgs: 148 active+clean; 56 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.12k op/s 2024-03-11T21:06:33.011 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:32 smithi023 bash[21372]: cluster 2024-03-11T21:06:31.529649+0000 osd.3 (osd.3) 330 : cluster 0 6.2 scrub starts 2024-03-11T21:06:33.011 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:32 smithi023 bash[21372]: cluster 2024-03-11T21:06:31.535013+0000 osd.3 (osd.3) 331 : cluster 0 6.2 scrub ok 2024-03-11T21:06:33.011 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:32 smithi023 bash[21372]: cluster 2024-03-11T21:06:32.049879+0000 osd.7 (osd.7) 351 : cluster 0 6.8 scrub starts 2024-03-11T21:06:33.011 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:32 smithi023 bash[21372]: cluster 2024-03-11T21:06:32.052031+0000 osd.7 (osd.7) 352 : cluster 0 6.8 scrub ok 2024-03-11T21:06:33.011 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:32 smithi023 bash[21372]: cluster 2024-03-11T21:06:32.225323+0000 mgr.y (mgr.24380) 583 : cluster 0 pgmap v649: 148 pgs: 148 active+clean; 56 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.12k op/s 2024-03-11T21:06:33.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:32 smithi146 bash[23809]: cluster 2024-03-11T21:06:31.529649+0000 osd.3 (osd.3) 330 : cluster 0 6.2 scrub starts 2024-03-11T21:06:33.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:32 smithi146 bash[23809]: cluster 2024-03-11T21:06:31.535013+0000 osd.3 (osd.3) 331 : cluster 0 6.2 scrub ok 2024-03-11T21:06:33.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:32 smithi146 bash[23809]: cluster 2024-03-11T21:06:32.049879+0000 osd.7 (osd.7) 351 : cluster 0 6.8 scrub starts 2024-03-11T21:06:33.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:32 smithi146 bash[23809]: cluster 2024-03-11T21:06:32.052031+0000 osd.7 (osd.7) 352 : cluster 0 6.8 scrub ok 2024-03-11T21:06:33.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:32 smithi146 bash[23809]: cluster 2024-03-11T21:06:32.225323+0000 mgr.y (mgr.24380) 583 : cluster 0 pgmap v649: 148 pgs: 148 active+clean; 56 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.12k op/s 2024-03-11T21:06:33.024 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:33.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:33.026 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:06:33.127 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:33.154 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:33.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:33.156 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:06:33.257 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:33.282 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:33.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:33.284 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:06:33.385 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:33.414 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:33.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:33.416 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:06:33.517 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:06:33.546 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:06:33.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:33.548 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:06:33.649 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:06:33.718 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:06:33.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:33.721 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:06:33.823 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:06:33.846 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:06:33.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:33.849 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:06:33.950 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:33.961 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:33 smithi146 bash[23809]: cluster 2024-03-11T21:06:31.989966+0000 osd.5 (osd.5) 309 : cluster 0 4.1 scrub starts 2024-03-11T21:06:33.961 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:33 smithi146 bash[23809]: cluster 2024-03-11T21:06:31.992046+0000 osd.5 (osd.5) 310 : cluster 0 4.1 scrub ok 2024-03-11T21:06:33.961 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:33 smithi146 bash[23809]: cluster 2024-03-11T21:06:32.571948+0000 osd.3 (osd.3) 332 : cluster 0 4.1b scrub starts 2024-03-11T21:06:33.961 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:33 smithi146 bash[23809]: cluster 2024-03-11T21:06:32.574190+0000 osd.3 (osd.3) 333 : cluster 0 4.1b scrub ok 2024-03-11T21:06:33.961 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:33 smithi146 bash[23809]: cluster 2024-03-11T21:06:32.917827+0000 osd.2 (osd.2) 215 : cluster 0 3.9 scrub starts 2024-03-11T21:06:33.961 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:33 smithi146 bash[23809]: cluster 2024-03-11T21:06:32.921039+0000 osd.2 (osd.2) 216 : cluster 0 3.9 scrub ok 2024-03-11T21:06:33.961 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:33 smithi146 bash[23809]: cluster 2024-03-11T21:06:32.967065+0000 osd.0 (osd.0) 227 : cluster 0 4.1e deep-scrub starts 2024-03-11T21:06:33.962 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:33 smithi146 bash[23809]: cluster 2024-03-11T21:06:32.968646+0000 osd.0 (osd.0) 228 : cluster 0 4.1e deep-scrub ok 2024-03-11T21:06:33.962 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:33 smithi146 bash[23809]: cluster 2024-03-11T21:06:33.046338+0000 osd.7 (osd.7) 353 : cluster 0 9.d scrub starts 2024-03-11T21:06:33.962 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:33 smithi146 bash[23809]: cluster 2024-03-11T21:06:33.079611+0000 osd.7 (osd.7) 354 : cluster 0 9.d scrub ok 2024-03-11T21:06:33.977 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:33.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:33.979 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:06:34.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:33 smithi023 bash[21372]: cluster 2024-03-11T21:06:31.989966+0000 osd.5 (osd.5) 309 : cluster 0 4.1 scrub starts 2024-03-11T21:06:34.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:33 smithi023 bash[21372]: cluster 2024-03-11T21:06:31.992046+0000 osd.5 (osd.5) 310 : cluster 0 4.1 scrub ok 2024-03-11T21:06:34.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:33 smithi023 bash[21372]: cluster 2024-03-11T21:06:32.571948+0000 osd.3 (osd.3) 332 : cluster 0 4.1b scrub starts 2024-03-11T21:06:34.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:33 smithi023 bash[21372]: cluster 2024-03-11T21:06:32.574190+0000 osd.3 (osd.3) 333 : cluster 0 4.1b scrub ok 2024-03-11T21:06:34.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:33 smithi023 bash[21372]: cluster 2024-03-11T21:06:32.917827+0000 osd.2 (osd.2) 215 : cluster 0 3.9 scrub starts 2024-03-11T21:06:34.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:33 smithi023 bash[21372]: cluster 2024-03-11T21:06:32.921039+0000 osd.2 (osd.2) 216 : cluster 0 3.9 scrub ok 2024-03-11T21:06:34.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:33 smithi023 bash[21372]: cluster 2024-03-11T21:06:32.967065+0000 osd.0 (osd.0) 227 : cluster 0 4.1e deep-scrub starts 2024-03-11T21:06:34.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:33 smithi023 bash[21372]: cluster 2024-03-11T21:06:32.968646+0000 osd.0 (osd.0) 228 : cluster 0 4.1e deep-scrub ok 2024-03-11T21:06:34.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:33 smithi023 bash[21372]: cluster 2024-03-11T21:06:33.046338+0000 osd.7 (osd.7) 353 : cluster 0 9.d scrub starts 2024-03-11T21:06:34.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:33 smithi023 bash[21372]: cluster 2024-03-11T21:06:33.079611+0000 osd.7 (osd.7) 354 : cluster 0 9.d scrub ok 2024-03-11T21:06:34.032 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:33 smithi023 bash[29146]: cluster 2024-03-11T21:06:31.989966+0000 osd.5 (osd.5) 309 : cluster 0 4.1 scrub starts 2024-03-11T21:06:34.032 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:33 smithi023 bash[29146]: cluster 2024-03-11T21:06:31.992046+0000 osd.5 (osd.5) 310 : cluster 0 4.1 scrub ok 2024-03-11T21:06:34.032 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:33 smithi023 bash[29146]: cluster 2024-03-11T21:06:32.571948+0000 osd.3 (osd.3) 332 : cluster 0 4.1b scrub starts 2024-03-11T21:06:34.032 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:33 smithi023 bash[29146]: cluster 2024-03-11T21:06:32.574190+0000 osd.3 (osd.3) 333 : cluster 0 4.1b scrub ok 2024-03-11T21:06:34.032 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:33 smithi023 bash[29146]: cluster 2024-03-11T21:06:32.917827+0000 osd.2 (osd.2) 215 : cluster 0 3.9 scrub starts 2024-03-11T21:06:34.032 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:33 smithi023 bash[29146]: cluster 2024-03-11T21:06:32.921039+0000 osd.2 (osd.2) 216 : cluster 0 3.9 scrub ok 2024-03-11T21:06:34.032 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:33 smithi023 bash[29146]: cluster 2024-03-11T21:06:32.967065+0000 osd.0 (osd.0) 227 : cluster 0 4.1e deep-scrub starts 2024-03-11T21:06:34.032 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:33 smithi023 bash[29146]: cluster 2024-03-11T21:06:32.968646+0000 osd.0 (osd.0) 228 : cluster 0 4.1e deep-scrub ok 2024-03-11T21:06:34.032 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:33 smithi023 bash[29146]: cluster 2024-03-11T21:06:33.046338+0000 osd.7 (osd.7) 353 : cluster 0 9.d scrub starts 2024-03-11T21:06:34.032 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:33 smithi023 bash[29146]: cluster 2024-03-11T21:06:33.079611+0000 osd.7 (osd.7) 354 : cluster 0 9.d scrub ok 2024-03-11T21:06:34.080 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:06:34.106 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:06:34.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:34.108 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:06:34.208 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:06:34.225 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:06:34.237 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:06:34.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:34.239 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:06:34.340 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:06:34.367 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:06:34.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:34.369 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:06:34.470 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:06:34.498 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:06:34.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:34.500 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:06:34.601 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:34.630 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:34.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:34.631 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:06:34.732 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:34.762 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:34.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:34.764 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:06:34.865 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:34.877 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:34 smithi146 bash[23809]: cluster 2024-03-11T21:06:33.714176+0000 osd.1 (osd.1) 109 : cluster 0 9.0 scrub starts 2024-03-11T21:06:34.877 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:34 smithi146 bash[23809]: cluster 2024-03-11T21:06:33.744825+0000 osd.1 (osd.1) 110 : cluster 0 9.0 scrub ok 2024-03-11T21:06:34.877 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:34 smithi146 bash[23809]: cluster 2024-03-11T21:06:33.934517+0000 osd.2 (osd.2) 217 : cluster 0 2.1 scrub starts 2024-03-11T21:06:34.878 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:34 smithi146 bash[23809]: cluster 2024-03-11T21:06:33.936401+0000 osd.2 (osd.2) 218 : cluster 0 2.1 scrub ok 2024-03-11T21:06:34.878 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:34 smithi146 bash[23809]: cluster 2024-03-11T21:06:34.060667+0000 osd.7 (osd.7) 355 : cluster 0 9.3 scrub starts 2024-03-11T21:06:34.878 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:34 smithi146 bash[23809]: cluster 2024-03-11T21:06:34.092210+0000 osd.7 (osd.7) 356 : cluster 0 9.3 scrub ok 2024-03-11T21:06:34.878 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:34 smithi146 bash[23809]: cluster 2024-03-11T21:06:34.225969+0000 mgr.y (mgr.24380) 584 : cluster 0 pgmap v650: 148 pgs: 148 active+clean; 56 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 712 op/s 2024-03-11T21:06:34.892 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:34.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:34.895 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:06:34.995 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:35.022 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:35.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:35.024 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:06:35.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:35 smithi023 bash[21372]: cluster 2024-03-11T21:06:33.714176+0000 osd.1 (osd.1) 109 : cluster 0 9.0 scrub starts 2024-03-11T21:06:35.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:35 smithi023 bash[21372]: cluster 2024-03-11T21:06:33.744825+0000 osd.1 (osd.1) 110 : cluster 0 9.0 scrub ok 2024-03-11T21:06:35.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:35 smithi023 bash[21372]: cluster 2024-03-11T21:06:33.934517+0000 osd.2 (osd.2) 217 : cluster 0 2.1 scrub starts 2024-03-11T21:06:35.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:35 smithi023 bash[21372]: cluster 2024-03-11T21:06:33.936401+0000 osd.2 (osd.2) 218 : cluster 0 2.1 scrub ok 2024-03-11T21:06:35.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:35 smithi023 bash[21372]: cluster 2024-03-11T21:06:34.060667+0000 osd.7 (osd.7) 355 : cluster 0 9.3 scrub starts 2024-03-11T21:06:35.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:35 smithi023 bash[21372]: cluster 2024-03-11T21:06:34.092210+0000 osd.7 (osd.7) 356 : cluster 0 9.3 scrub ok 2024-03-11T21:06:35.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:35 smithi023 bash[21372]: cluster 2024-03-11T21:06:34.225969+0000 mgr.y (mgr.24380) 584 : cluster 0 pgmap v650: 148 pgs: 148 active+clean; 56 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 712 op/s 2024-03-11T21:06:35.031 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:35 smithi023 bash[29146]: cluster 2024-03-11T21:06:33.714176+0000 osd.1 (osd.1) 109 : cluster 0 9.0 scrub starts 2024-03-11T21:06:35.032 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:35 smithi023 bash[29146]: cluster 2024-03-11T21:06:33.744825+0000 osd.1 (osd.1) 110 : cluster 0 9.0 scrub ok 2024-03-11T21:06:35.032 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:35 smithi023 bash[29146]: cluster 2024-03-11T21:06:33.934517+0000 osd.2 (osd.2) 217 : cluster 0 2.1 scrub starts 2024-03-11T21:06:35.032 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:35 smithi023 bash[29146]: cluster 2024-03-11T21:06:33.936401+0000 osd.2 (osd.2) 218 : cluster 0 2.1 scrub ok 2024-03-11T21:06:35.032 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:35 smithi023 bash[29146]: cluster 2024-03-11T21:06:34.060667+0000 osd.7 (osd.7) 355 : cluster 0 9.3 scrub starts 2024-03-11T21:06:35.032 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:35 smithi023 bash[29146]: cluster 2024-03-11T21:06:34.092210+0000 osd.7 (osd.7) 356 : cluster 0 9.3 scrub ok 2024-03-11T21:06:35.032 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:35 smithi023 bash[29146]: cluster 2024-03-11T21:06:34.225969+0000 mgr.y (mgr.24380) 584 : cluster 0 pgmap v650: 148 pgs: 148 active+clean; 56 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 712 op/s 2024-03-11T21:06:35.125 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:06:35.154 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:06:35.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:35.156 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:06:35.257 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:06:35.283 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:06:35.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:35.286 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:06:35.386 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:06:35.413 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:06:35.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:35.415 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:06:35.516 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:06:35.543 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:06:35.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:35.545 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:06:35.646 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:35.674 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:35.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:35.677 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:06:35.722 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 6, 7, 1] out_osds: [] dead_osds: [] live_osds: [0, 2, 3, 5, 6, 7, 4, 1] 2024-03-11T21:06:35.722 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-11T21:06:35.723 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-03-11T21:06:35.723 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd dump -f json-pretty 2024-03-11T21:06:35.777 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:35.788 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:06:35 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:06:35.805 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:35.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:35.807 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:06:35.908 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:06:35.928 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:06:35.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:35.930 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:06:36.022 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_2 2024-03-11T21:06:36.022 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd pool rm unique_pool_2 unique_pool_2 --yes-i-really-really-mean-it 2024-03-11T21:06:36.030 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:36.057 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:36.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:36.059 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:06:36.160 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:36.186 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:36.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:36.188 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:06:36.267 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:06:36.289 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:06:36.308 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:06:36.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:36.311 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:06:36.411 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:06:36.440 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:06:36.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:36.442 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:06:36.543 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:36.571 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:36.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:36.574 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:06:36.675 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:36.687 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:36 smithi023 bash[21372]: audit 2024-03-11T21:06:35.541849+0000 mon.a (mon.0) 1577 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:06:36.687 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:36 smithi023 bash[21372]: audit 2024-03-11T21:06:35.544248+0000 mon.a (mon.0) 1578 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:06:36.687 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:36 smithi023 bash[21372]: audit 2024-03-11T21:06:35.624137+0000 mgr.y (mgr.24380) 585 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:06:36.687 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:36 smithi023 bash[21372]: cluster 2024-03-11T21:06:35.938657+0000 osd.2 (osd.2) 219 : cluster 0 4.d scrub starts 2024-03-11T21:06:36.688 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:36 smithi023 bash[21372]: cluster 2024-03-11T21:06:35.940963+0000 osd.2 (osd.2) 220 : cluster 0 4.d scrub ok 2024-03-11T21:06:36.688 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:36 smithi023 bash[29146]: audit 2024-03-11T21:06:35.541849+0000 mon.a (mon.0) 1577 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:06:36.688 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:36 smithi023 bash[29146]: audit 2024-03-11T21:06:35.544248+0000 mon.a (mon.0) 1578 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:06:36.688 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:36 smithi023 bash[29146]: audit 2024-03-11T21:06:35.624137+0000 mgr.y (mgr.24380) 585 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:06:36.688 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:36 smithi023 bash[29146]: cluster 2024-03-11T21:06:35.938657+0000 osd.2 (osd.2) 219 : cluster 0 4.d scrub starts 2024-03-11T21:06:36.688 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:36 smithi023 bash[29146]: cluster 2024-03-11T21:06:35.940963+0000 osd.2 (osd.2) 220 : cluster 0 4.d scrub ok 2024-03-11T21:06:36.703 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:36.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:36.705 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:06:36.806 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:06:36.817 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:36 smithi146 bash[23809]: audit 2024-03-11T21:06:35.541849+0000 mon.a (mon.0) 1577 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:06:36.817 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:36 smithi146 bash[23809]: audit 2024-03-11T21:06:35.544248+0000 mon.a (mon.0) 1578 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:06:36.817 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:36 smithi146 bash[23809]: audit 2024-03-11T21:06:35.624137+0000 mgr.y (mgr.24380) 585 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:06:36.818 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:36 smithi146 bash[23809]: cluster 2024-03-11T21:06:35.938657+0000 osd.2 (osd.2) 219 : cluster 0 4.d scrub starts 2024-03-11T21:06:36.818 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:36 smithi146 bash[23809]: cluster 2024-03-11T21:06:35.940963+0000 osd.2 (osd.2) 220 : cluster 0 4.d scrub ok 2024-03-11T21:06:36.834 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:06:36.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:36.836 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:06:36.937 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:36.961 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:36.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:36.964 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:06:37.064 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:06:37.092 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:06:37.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:37.095 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:06:37.196 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:37.225 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:37.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:37.227 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:06:37.328 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:37.355 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:37.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:37.357 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:06:37.458 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:06:37.485 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:06:37.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:37.487 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:06:37.588 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:06:37.614 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:06:37.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:37.616 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:06:37.717 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:06:37.729 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:37 smithi023 bash[21372]: cluster 2024-03-11T21:06:36.006929+0000 osd.5 (osd.5) 311 : cluster 0 9.7 scrub starts 2024-03-11T21:06:37.729 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:37 smithi023 bash[21372]: cluster 2024-03-11T21:06:36.029475+0000 osd.5 (osd.5) 312 : cluster 0 9.7 scrub ok 2024-03-11T21:06:37.729 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:37 smithi023 bash[21372]: cluster 2024-03-11T21:06:36.119919+0000 osd.7 (osd.7) 357 : cluster 0 2.1c scrub starts 2024-03-11T21:06:37.729 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:37 smithi023 bash[21372]: cluster 2024-03-11T21:06:36.121416+0000 osd.7 (osd.7) 358 : cluster 0 2.1c scrub ok 2024-03-11T21:06:37.729 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:37 smithi023 bash[21372]: cluster 2024-03-11T21:06:36.226770+0000 mgr.y (mgr.24380) 586 : cluster 0 pgmap v651: 148 pgs: 148 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 722 B/s rd, 0 B/s wr, 560 op/s 2024-03-11T21:06:37.729 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:37 smithi023 bash[21372]: cluster 2024-03-11T21:06:36.907153+0000 osd.2 (osd.2) 221 : cluster 0 6.14 scrub starts 2024-03-11T21:06:37.729 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:37 smithi023 bash[21372]: cluster 2024-03-11T21:06:36.909561+0000 osd.2 (osd.2) 222 : cluster 0 6.14 scrub ok 2024-03-11T21:06:37.729 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:37 smithi023 bash[29146]: cluster 2024-03-11T21:06:36.006929+0000 osd.5 (osd.5) 311 : cluster 0 9.7 scrub starts 2024-03-11T21:06:37.730 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:37 smithi023 bash[29146]: cluster 2024-03-11T21:06:36.029475+0000 osd.5 (osd.5) 312 : cluster 0 9.7 scrub ok 2024-03-11T21:06:37.730 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:37 smithi023 bash[29146]: cluster 2024-03-11T21:06:36.119919+0000 osd.7 (osd.7) 357 : cluster 0 2.1c scrub starts 2024-03-11T21:06:37.730 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:37 smithi023 bash[29146]: cluster 2024-03-11T21:06:36.121416+0000 osd.7 (osd.7) 358 : cluster 0 2.1c scrub ok 2024-03-11T21:06:37.730 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:37 smithi023 bash[29146]: cluster 2024-03-11T21:06:36.226770+0000 mgr.y (mgr.24380) 586 : cluster 0 pgmap v651: 148 pgs: 148 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 722 B/s rd, 0 B/s wr, 560 op/s 2024-03-11T21:06:37.730 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:37 smithi023 bash[29146]: cluster 2024-03-11T21:06:36.907153+0000 osd.2 (osd.2) 221 : cluster 0 6.14 scrub starts 2024-03-11T21:06:37.730 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:37 smithi023 bash[29146]: cluster 2024-03-11T21:06:36.909561+0000 osd.2 (osd.2) 222 : cluster 0 6.14 scrub ok 2024-03-11T21:06:37.745 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:06:37.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:37.747 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:06:37.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:37 smithi146 bash[23809]: cluster 2024-03-11T21:06:36.006929+0000 osd.5 (osd.5) 311 : cluster 0 9.7 scrub starts 2024-03-11T21:06:37.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:37 smithi146 bash[23809]: cluster 2024-03-11T21:06:36.029475+0000 osd.5 (osd.5) 312 : cluster 0 9.7 scrub ok 2024-03-11T21:06:37.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:37 smithi146 bash[23809]: cluster 2024-03-11T21:06:36.119919+0000 osd.7 (osd.7) 357 : cluster 0 2.1c scrub starts 2024-03-11T21:06:37.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:37 smithi146 bash[23809]: cluster 2024-03-11T21:06:36.121416+0000 osd.7 (osd.7) 358 : cluster 0 2.1c scrub ok 2024-03-11T21:06:37.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:37 smithi146 bash[23809]: cluster 2024-03-11T21:06:36.226770+0000 mgr.y (mgr.24380) 586 : cluster 0 pgmap v651: 148 pgs: 148 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 722 B/s rd, 0 B/s wr, 560 op/s 2024-03-11T21:06:37.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:37 smithi146 bash[23809]: cluster 2024-03-11T21:06:36.907153+0000 osd.2 (osd.2) 221 : cluster 0 6.14 scrub starts 2024-03-11T21:06:37.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:37 smithi146 bash[23809]: cluster 2024-03-11T21:06:36.909561+0000 osd.2 (osd.2) 222 : cluster 0 6.14 scrub ok 2024-03-11T21:06:37.847 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:37.873 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:37.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:37.876 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:06:37.976 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:06:38.005 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:06:38.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:38.008 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:06:38.109 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:38.129 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:38.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:38.131 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:06:38.232 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:38.257 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:38.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:38.259 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:06:38.360 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:38.386 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:38.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:38.388 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:06:38.489 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:06:38.501 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:06:38 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:06:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:06:38.516 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:06:38.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:38.519 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:06:38.524 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:06:38.620 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:06:38.647 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:06:38.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:38.649 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:06:38.750 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:38.776 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:38.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:38.778 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:06:38.879 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:38.906 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:38.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:38.908 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:06:39.009 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:39.037 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:39.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:39.039 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:06:39.140 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:06:39.160 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:06:39.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:39.162 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:06:39.263 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:06:39.291 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:06:39.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:39.293 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:06:39.394 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:06:39.421 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:06:39.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:39.423 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:06:39.523 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:06:39.545 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:06:39.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:39.548 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:06:39.648 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:39.659 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:39 smithi146 bash[23809]: cluster 2024-03-11T21:06:37.969433+0000 osd.6 (osd.6) 236 : cluster 0 2.17 scrub starts 2024-03-11T21:06:39.660 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:39 smithi146 bash[23809]: cluster 2024-03-11T21:06:37.971977+0000 osd.6 (osd.6) 237 : cluster 0 2.17 scrub ok 2024-03-11T21:06:39.660 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:39 smithi146 bash[23809]: cluster 2024-03-11T21:06:37.992013+0000 osd.5 (osd.5) 313 : cluster 0 6.d scrub starts 2024-03-11T21:06:39.660 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:39 smithi146 bash[23809]: cluster 2024-03-11T21:06:38.053762+0000 osd.5 (osd.5) 314 : cluster 0 6.d scrub ok 2024-03-11T21:06:39.660 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:39 smithi146 bash[23809]: cluster 2024-03-11T21:06:38.227678+0000 mgr.y (mgr.24380) 587 : cluster 0 pgmap v652: 148 pgs: 148 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 614 B/s rd, 0 B/s wr, 477 op/s 2024-03-11T21:06:39.674 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:39.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:39.676 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:06:39.777 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:06:39.805 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:06:39.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:39.807 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:06:39.866 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:39 smithi023 bash[29146]: cluster 2024-03-11T21:06:37.969433+0000 osd.6 (osd.6) 236 : cluster 0 2.17 scrub starts 2024-03-11T21:06:39.866 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:39 smithi023 bash[29146]: cluster 2024-03-11T21:06:37.971977+0000 osd.6 (osd.6) 237 : cluster 0 2.17 scrub ok 2024-03-11T21:06:39.866 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:39 smithi023 bash[29146]: cluster 2024-03-11T21:06:37.992013+0000 osd.5 (osd.5) 313 : cluster 0 6.d scrub starts 2024-03-11T21:06:39.866 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:39 smithi023 bash[29146]: cluster 2024-03-11T21:06:38.053762+0000 osd.5 (osd.5) 314 : cluster 0 6.d scrub ok 2024-03-11T21:06:39.866 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:39 smithi023 bash[29146]: cluster 2024-03-11T21:06:38.227678+0000 mgr.y (mgr.24380) 587 : cluster 0 pgmap v652: 148 pgs: 148 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 614 B/s rd, 0 B/s wr, 477 op/s 2024-03-11T21:06:39.908 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:06:39.937 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:06:39.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:39.939 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:06:40.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:39 smithi023 bash[21372]: cluster 2024-03-11T21:06:37.969433+0000 osd.6 (osd.6) 236 : cluster 0 2.17 scrub starts 2024-03-11T21:06:40.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:39 smithi023 bash[21372]: cluster 2024-03-11T21:06:37.971977+0000 osd.6 (osd.6) 237 : cluster 0 2.17 scrub ok 2024-03-11T21:06:40.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:39 smithi023 bash[21372]: cluster 2024-03-11T21:06:37.992013+0000 osd.5 (osd.5) 313 : cluster 0 6.d scrub starts 2024-03-11T21:06:40.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:39 smithi023 bash[21372]: cluster 2024-03-11T21:06:38.053762+0000 osd.5 (osd.5) 314 : cluster 0 6.d scrub ok 2024-03-11T21:06:40.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:39 smithi023 bash[21372]: cluster 2024-03-11T21:06:38.227678+0000 mgr.y (mgr.24380) 587 : cluster 0 pgmap v652: 148 pgs: 148 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 614 B/s rd, 0 B/s wr, 477 op/s 2024-03-11T21:06:40.040 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:40.067 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:40.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:40.070 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:06:40.170 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:06:40.195 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:06:40.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:40.198 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:06:40.299 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:06:40.326 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:06:40.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:40.328 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:06:40.428 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:06:40.457 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:06:40.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:40.459 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:06:40.560 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:40.586 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:40.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:40.589 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:06:40.690 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:06:40.701 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:40 smithi023 bash[21372]: cluster 2024-03-11T21:06:38.934978+0000 osd.6 (osd.6) 238 : cluster 0 4.5 scrub starts 2024-03-11T21:06:40.701 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:40 smithi023 bash[21372]: cluster 2024-03-11T21:06:38.936439+0000 osd.6 (osd.6) 239 : cluster 0 4.5 scrub ok 2024-03-11T21:06:40.702 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:40 smithi023 bash[29146]: cluster 2024-03-11T21:06:38.934978+0000 osd.6 (osd.6) 238 : cluster 0 4.5 scrub starts 2024-03-11T21:06:40.702 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:40 smithi023 bash[29146]: cluster 2024-03-11T21:06:38.936439+0000 osd.6 (osd.6) 239 : cluster 0 4.5 scrub ok 2024-03-11T21:06:40.718 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:06:40.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:40.720 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:06:40.821 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:06:40.848 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:06:40.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:40.850 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:06:40.952 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:40.963 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:40 smithi146 bash[23809]: cluster 2024-03-11T21:06:38.934978+0000 osd.6 (osd.6) 238 : cluster 0 4.5 scrub starts 2024-03-11T21:06:40.963 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:40 smithi146 bash[23809]: cluster 2024-03-11T21:06:38.936439+0000 osd.6 (osd.6) 239 : cluster 0 4.5 scrub ok 2024-03-11T21:06:40.979 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:40.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:40.982 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:06:41.082 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:06:41.111 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:06:41.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:41.113 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:06:41.214 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:41.242 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:41.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:41.244 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:06:41.345 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:41.373 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:41.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:41.375 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:06:41.476 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:41.502 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:41.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:41.505 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:06:41.606 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:06:41.635 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:06:41.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:41.637 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:06:41.738 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:06:41.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:41 smithi146 bash[23809]: cluster 2024-03-11T21:06:39.968143+0000 osd.6 (osd.6) 240 : cluster 0 2.11 scrub starts 2024-03-11T21:06:41.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:41 smithi146 bash[23809]: cluster 2024-03-11T21:06:39.971149+0000 osd.6 (osd.6) 241 : cluster 0 2.11 scrub ok 2024-03-11T21:06:41.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:41 smithi146 bash[23809]: cluster 2024-03-11T21:06:40.228509+0000 mgr.y (mgr.24380) 588 : cluster 0 pgmap v653: 148 pgs: 148 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 980 B/s rd, 0 B/s wr, 415 op/s 2024-03-11T21:06:41.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:41 smithi146 bash[23809]: cluster 2024-03-11T21:06:40.938480+0000 osd.2 (osd.2) 223 : cluster 0 2.e scrub starts 2024-03-11T21:06:41.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:41 smithi146 bash[23809]: cluster 2024-03-11T21:06:40.940895+0000 osd.2 (osd.2) 224 : cluster 0 2.e scrub ok 2024-03-11T21:06:41.764 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:06:41.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:41.767 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:06:41.867 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:06:41.878 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:41 smithi023 bash[21372]: cluster 2024-03-11T21:06:39.968143+0000 osd.6 (osd.6) 240 : cluster 0 2.11 scrub starts 2024-03-11T21:06:41.879 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:41 smithi023 bash[21372]: cluster 2024-03-11T21:06:39.971149+0000 osd.6 (osd.6) 241 : cluster 0 2.11 scrub ok 2024-03-11T21:06:41.879 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:41 smithi023 bash[21372]: cluster 2024-03-11T21:06:40.228509+0000 mgr.y (mgr.24380) 588 : cluster 0 pgmap v653: 148 pgs: 148 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 980 B/s rd, 0 B/s wr, 415 op/s 2024-03-11T21:06:41.879 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:41 smithi023 bash[21372]: cluster 2024-03-11T21:06:40.938480+0000 osd.2 (osd.2) 223 : cluster 0 2.e scrub starts 2024-03-11T21:06:41.879 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:41 smithi023 bash[21372]: cluster 2024-03-11T21:06:40.940895+0000 osd.2 (osd.2) 224 : cluster 0 2.e scrub ok 2024-03-11T21:06:41.879 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:41 smithi023 bash[29146]: cluster 2024-03-11T21:06:39.968143+0000 osd.6 (osd.6) 240 : cluster 0 2.11 scrub starts 2024-03-11T21:06:41.879 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:41 smithi023 bash[29146]: cluster 2024-03-11T21:06:39.971149+0000 osd.6 (osd.6) 241 : cluster 0 2.11 scrub ok 2024-03-11T21:06:41.879 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:41 smithi023 bash[29146]: cluster 2024-03-11T21:06:40.228509+0000 mgr.y (mgr.24380) 588 : cluster 0 pgmap v653: 148 pgs: 148 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 980 B/s rd, 0 B/s wr, 415 op/s 2024-03-11T21:06:41.879 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:41 smithi023 bash[29146]: cluster 2024-03-11T21:06:40.938480+0000 osd.2 (osd.2) 223 : cluster 0 2.e scrub starts 2024-03-11T21:06:41.879 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:41 smithi023 bash[29146]: cluster 2024-03-11T21:06:40.940895+0000 osd.2 (osd.2) 224 : cluster 0 2.e scrub ok 2024-03-11T21:06:41.892 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:06:41.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:41.895 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:06:41.995 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:06:42.022 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:06:42.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:42.025 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:06:42.125 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:42.155 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:42.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:42.157 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:06:42.258 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:06:42.284 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:06:42.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:42.287 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:06:42.349 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:06:42.350 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:06:42.387 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:42.409 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:42.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:42.411 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:06:42.512 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:06:42.540 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:06:42.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:42.543 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:06:42.644 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:42.671 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:42.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:42.673 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:06:42.716 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:42 smithi023 bash[21372]: cluster 2024-03-11T21:06:40.980796+0000 osd.6 (osd.6) 242 : cluster 0 4.a scrub starts 2024-03-11T21:06:42.717 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:42 smithi023 bash[21372]: cluster 2024-03-11T21:06:40.982421+0000 osd.6 (osd.6) 243 : cluster 0 4.a scrub ok 2024-03-11T21:06:42.717 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:42 smithi023 bash[21372]: cluster 2024-03-11T21:06:41.231452+0000 osd.7 (osd.7) 359 : cluster 0 3.e scrub starts 2024-03-11T21:06:42.717 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:42 smithi023 bash[21372]: cluster 2024-03-11T21:06:41.235412+0000 osd.7 (osd.7) 360 : cluster 0 3.e scrub ok 2024-03-11T21:06:42.717 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:42 smithi023 bash[21372]: cluster 2024-03-11T21:06:41.471897+0000 osd.3 (osd.3) 334 : cluster 0 3.7 scrub starts 2024-03-11T21:06:42.717 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:42 smithi023 bash[21372]: cluster 2024-03-11T21:06:41.475860+0000 osd.3 (osd.3) 335 : cluster 0 3.7 scrub ok 2024-03-11T21:06:42.717 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:42 smithi023 bash[29146]: cluster 2024-03-11T21:06:40.980796+0000 osd.6 (osd.6) 242 : cluster 0 4.a scrub starts 2024-03-11T21:06:42.717 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:42 smithi023 bash[29146]: cluster 2024-03-11T21:06:40.982421+0000 osd.6 (osd.6) 243 : cluster 0 4.a scrub ok 2024-03-11T21:06:42.717 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:42 smithi023 bash[29146]: cluster 2024-03-11T21:06:41.231452+0000 osd.7 (osd.7) 359 : cluster 0 3.e scrub starts 2024-03-11T21:06:42.717 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:42 smithi023 bash[29146]: cluster 2024-03-11T21:06:41.235412+0000 osd.7 (osd.7) 360 : cluster 0 3.e scrub ok 2024-03-11T21:06:42.717 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:42 smithi023 bash[29146]: cluster 2024-03-11T21:06:41.471897+0000 osd.3 (osd.3) 334 : cluster 0 3.7 scrub starts 2024-03-11T21:06:42.717 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:42 smithi023 bash[29146]: cluster 2024-03-11T21:06:41.475860+0000 osd.3 (osd.3) 335 : cluster 0 3.7 scrub ok 2024-03-11T21:06:42.774 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:06:42.795 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:06:42.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:42.798 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:06:42.899 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:06:42.902 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-11T21:06:42.902 INFO:teuthology.orchestra.run.smithi023.stdout: "ops": [], 2024-03-11T21:06:42.902 INFO:teuthology.orchestra.run.smithi023.stdout: "complaint_time": 30, 2024-03-11T21:06:42.903 INFO:teuthology.orchestra.run.smithi023.stdout: "num_blocked_ops": 0 2024-03-11T21:06:42.903 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-11T21:06:42.920 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:06:42.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:42.922 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:06:43.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:42 smithi146 bash[23809]: cluster 2024-03-11T21:06:40.980796+0000 osd.6 (osd.6) 242 : cluster 0 4.a scrub starts 2024-03-11T21:06:43.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:42 smithi146 bash[23809]: cluster 2024-03-11T21:06:40.982421+0000 osd.6 (osd.6) 243 : cluster 0 4.a scrub ok 2024-03-11T21:06:43.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:42 smithi146 bash[23809]: cluster 2024-03-11T21:06:41.231452+0000 osd.7 (osd.7) 359 : cluster 0 3.e scrub starts 2024-03-11T21:06:43.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:42 smithi146 bash[23809]: cluster 2024-03-11T21:06:41.235412+0000 osd.7 (osd.7) 360 : cluster 0 3.e scrub ok 2024-03-11T21:06:43.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:42 smithi146 bash[23809]: cluster 2024-03-11T21:06:41.471897+0000 osd.3 (osd.3) 334 : cluster 0 3.7 scrub starts 2024-03-11T21:06:43.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:42 smithi146 bash[23809]: cluster 2024-03-11T21:06:41.475860+0000 osd.3 (osd.3) 335 : cluster 0 3.7 scrub ok 2024-03-11T21:06:43.023 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:06:43.044 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:06:43.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:43.046 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:06:43.147 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:06:43.168 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:06:43.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:43.170 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:06:43.271 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:06:43.296 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:06:43.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:43.298 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:06:43.399 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:43.426 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:43.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:43.428 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:06:43.529 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:06:43.554 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:06:43.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:43.557 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:06:43.658 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:43.669 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:43 smithi146 bash[23809]: cluster 2024-03-11T21:06:41.761117+0000 osd.1 (osd.1) 111 : cluster 0 6.4 scrub starts 2024-03-11T21:06:43.670 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:43 smithi146 bash[23809]: cluster 2024-03-11T21:06:41.763431+0000 osd.1 (osd.1) 112 : cluster 0 6.4 scrub ok 2024-03-11T21:06:43.670 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:43 smithi146 bash[23809]: cluster 2024-03-11T21:06:42.229529+0000 mgr.y (mgr.24380) 589 : cluster 0 pgmap v654: 148 pgs: 148 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 938 B/s rd, 0 B/s wr, 398 op/s 2024-03-11T21:06:43.670 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:43 smithi146 bash[23809]: cluster 2024-03-11T21:06:42.240158+0000 osd.7 (osd.7) 361 : cluster 0 2.8 deep-scrub starts 2024-03-11T21:06:43.670 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:43 smithi146 bash[23809]: cluster 2024-03-11T21:06:42.242790+0000 osd.7 (osd.7) 362 : cluster 0 2.8 deep-scrub ok 2024-03-11T21:06:43.670 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:43 smithi146 bash[23809]: cluster 2024-03-11T21:06:42.943260+0000 osd.2 (osd.2) 225 : cluster 0 2.0 scrub starts 2024-03-11T21:06:43.670 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:43 smithi146 bash[23809]: cluster 2024-03-11T21:06:42.945661+0000 osd.2 (osd.2) 226 : cluster 0 2.0 scrub ok 2024-03-11T21:06:43.685 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:43.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:43.687 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:06:43.788 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:06:43.816 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:06:43.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:43.819 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:06:43.919 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:06:43.931 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:43 smithi023 bash[21372]: cluster 2024-03-11T21:06:41.761117+0000 osd.1 (osd.1) 111 : cluster 0 6.4 scrub starts 2024-03-11T21:06:43.932 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:43 smithi023 bash[21372]: cluster 2024-03-11T21:06:41.763431+0000 osd.1 (osd.1) 112 : cluster 0 6.4 scrub ok 2024-03-11T21:06:43.932 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:43 smithi023 bash[21372]: cluster 2024-03-11T21:06:42.229529+0000 mgr.y (mgr.24380) 589 : cluster 0 pgmap v654: 148 pgs: 148 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 938 B/s rd, 0 B/s wr, 398 op/s 2024-03-11T21:06:43.932 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:43 smithi023 bash[21372]: cluster 2024-03-11T21:06:42.240158+0000 osd.7 (osd.7) 361 : cluster 0 2.8 deep-scrub starts 2024-03-11T21:06:43.932 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:43 smithi023 bash[21372]: cluster 2024-03-11T21:06:42.242790+0000 osd.7 (osd.7) 362 : cluster 0 2.8 deep-scrub ok 2024-03-11T21:06:43.932 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:43 smithi023 bash[21372]: cluster 2024-03-11T21:06:42.943260+0000 osd.2 (osd.2) 225 : cluster 0 2.0 scrub starts 2024-03-11T21:06:43.932 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:43 smithi023 bash[21372]: cluster 2024-03-11T21:06:42.945661+0000 osd.2 (osd.2) 226 : cluster 0 2.0 scrub ok 2024-03-11T21:06:43.932 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:43 smithi023 bash[29146]: cluster 2024-03-11T21:06:41.761117+0000 osd.1 (osd.1) 111 : cluster 0 6.4 scrub starts 2024-03-11T21:06:43.932 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:43 smithi023 bash[29146]: cluster 2024-03-11T21:06:41.763431+0000 osd.1 (osd.1) 112 : cluster 0 6.4 scrub ok 2024-03-11T21:06:43.932 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:43 smithi023 bash[29146]: cluster 2024-03-11T21:06:42.229529+0000 mgr.y (mgr.24380) 589 : cluster 0 pgmap v654: 148 pgs: 148 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 938 B/s rd, 0 B/s wr, 398 op/s 2024-03-11T21:06:43.932 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:43 smithi023 bash[29146]: cluster 2024-03-11T21:06:42.240158+0000 osd.7 (osd.7) 361 : cluster 0 2.8 deep-scrub starts 2024-03-11T21:06:43.933 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:43 smithi023 bash[29146]: cluster 2024-03-11T21:06:42.242790+0000 osd.7 (osd.7) 362 : cluster 0 2.8 deep-scrub ok 2024-03-11T21:06:43.933 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:43 smithi023 bash[29146]: cluster 2024-03-11T21:06:42.943260+0000 osd.2 (osd.2) 225 : cluster 0 2.0 scrub starts 2024-03-11T21:06:43.933 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:43 smithi023 bash[29146]: cluster 2024-03-11T21:06:42.945661+0000 osd.2 (osd.2) 226 : cluster 0 2.0 scrub ok 2024-03-11T21:06:43.947 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:06:43.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:43.950 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:06:44.051 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:06:44.080 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:06:44.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:44.083 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:06:44.183 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:44.209 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:44.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:44.212 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:06:44.313 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:44.342 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:44.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:44.344 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:06:44.445 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:06:44.473 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:06:44.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:44.475 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:06:44.576 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:06:44.606 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:06:44.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:44.608 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:06:44.709 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:44.737 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:44.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:44.740 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:06:44.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:44 smithi146 bash[23809]: cluster 2024-03-11T21:06:42.770883+0000 osd.1 (osd.1) 113 : cluster 0 4.f scrub starts 2024-03-11T21:06:44.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:44 smithi146 bash[23809]: cluster 2024-03-11T21:06:42.772612+0000 osd.1 (osd.1) 114 : cluster 0 4.f scrub ok 2024-03-11T21:06:44.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:44 smithi146 bash[23809]: cluster 2024-03-11T21:06:42.898292+0000 osd.0 (osd.0) 229 : cluster 0 9.9 scrub starts 2024-03-11T21:06:44.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:44 smithi146 bash[23809]: cluster 2024-03-11T21:06:42.903575+0000 osd.5 (osd.5) 315 : cluster 0 2.2 scrub starts 2024-03-11T21:06:44.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:44 smithi146 bash[23809]: cluster 2024-03-11T21:06:42.906018+0000 osd.5 (osd.5) 316 : cluster 0 2.2 scrub ok 2024-03-11T21:06:44.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:44 smithi146 bash[23809]: cluster 2024-03-11T21:06:42.932578+0000 osd.0 (osd.0) 230 : cluster 0 9.9 scrub ok 2024-03-11T21:06:44.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:44 smithi146 bash[23809]: cluster 2024-03-11T21:06:43.199885+0000 osd.7 (osd.7) 363 : cluster 0 6.3 deep-scrub starts 2024-03-11T21:06:44.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:44 smithi146 bash[23809]: cluster 2024-03-11T21:06:43.271445+0000 osd.7 (osd.7) 364 : cluster 0 6.3 deep-scrub ok 2024-03-11T21:06:44.841 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:44.870 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:44.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:44.872 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:06:44.973 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:06:45.000 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:06:45.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:45.003 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:06:45.070 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:44 smithi023 bash[21372]: cluster 2024-03-11T21:06:42.770883+0000 osd.1 (osd.1) 113 : cluster 0 4.f scrub starts 2024-03-11T21:06:45.070 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:44 smithi023 bash[21372]: cluster 2024-03-11T21:06:42.772612+0000 osd.1 (osd.1) 114 : cluster 0 4.f scrub ok 2024-03-11T21:06:45.070 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:44 smithi023 bash[21372]: cluster 2024-03-11T21:06:42.898292+0000 osd.0 (osd.0) 229 : cluster 0 9.9 scrub starts 2024-03-11T21:06:45.070 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:44 smithi023 bash[21372]: cluster 2024-03-11T21:06:42.903575+0000 osd.5 (osd.5) 315 : cluster 0 2.2 scrub starts 2024-03-11T21:06:45.070 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:44 smithi023 bash[21372]: cluster 2024-03-11T21:06:42.906018+0000 osd.5 (osd.5) 316 : cluster 0 2.2 scrub ok 2024-03-11T21:06:45.070 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:44 smithi023 bash[21372]: cluster 2024-03-11T21:06:42.932578+0000 osd.0 (osd.0) 230 : cluster 0 9.9 scrub ok 2024-03-11T21:06:45.070 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:44 smithi023 bash[21372]: cluster 2024-03-11T21:06:43.199885+0000 osd.7 (osd.7) 363 : cluster 0 6.3 deep-scrub starts 2024-03-11T21:06:45.070 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:44 smithi023 bash[21372]: cluster 2024-03-11T21:06:43.271445+0000 osd.7 (osd.7) 364 : cluster 0 6.3 deep-scrub ok 2024-03-11T21:06:45.070 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:44 smithi023 bash[29146]: cluster 2024-03-11T21:06:42.770883+0000 osd.1 (osd.1) 113 : cluster 0 4.f scrub starts 2024-03-11T21:06:45.071 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:44 smithi023 bash[29146]: cluster 2024-03-11T21:06:42.772612+0000 osd.1 (osd.1) 114 : cluster 0 4.f scrub ok 2024-03-11T21:06:45.071 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:44 smithi023 bash[29146]: cluster 2024-03-11T21:06:42.898292+0000 osd.0 (osd.0) 229 : cluster 0 9.9 scrub starts 2024-03-11T21:06:45.071 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:44 smithi023 bash[29146]: cluster 2024-03-11T21:06:42.903575+0000 osd.5 (osd.5) 315 : cluster 0 2.2 scrub starts 2024-03-11T21:06:45.071 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:44 smithi023 bash[29146]: cluster 2024-03-11T21:06:42.906018+0000 osd.5 (osd.5) 316 : cluster 0 2.2 scrub ok 2024-03-11T21:06:45.071 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:44 smithi023 bash[29146]: cluster 2024-03-11T21:06:42.932578+0000 osd.0 (osd.0) 230 : cluster 0 9.9 scrub ok 2024-03-11T21:06:45.071 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:44 smithi023 bash[29146]: cluster 2024-03-11T21:06:43.199885+0000 osd.7 (osd.7) 363 : cluster 0 6.3 deep-scrub starts 2024-03-11T21:06:45.071 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:44 smithi023 bash[29146]: cluster 2024-03-11T21:06:43.271445+0000 osd.7 (osd.7) 364 : cluster 0 6.3 deep-scrub ok 2024-03-11T21:06:45.103 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:06:45.129 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:06:45.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:45.131 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:06:45.232 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:45.258 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:45.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:45.260 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:06:45.361 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:06:45.389 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:06:45.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:45.391 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:06:45.492 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:45.519 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:45.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:45.521 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:06:45.622 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:45.649 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:45.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:45.651 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:06:45.752 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:06:45.764 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:45 smithi023 bash[21372]: cluster 2024-03-11T21:06:43.726333+0000 osd.1 (osd.1) 115 : cluster 0 9.4 scrub starts 2024-03-11T21:06:45.764 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:45 smithi023 bash[21372]: cluster 2024-03-11T21:06:43.762900+0000 osd.1 (osd.1) 116 : cluster 0 9.4 scrub ok 2024-03-11T21:06:45.764 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:45 smithi023 bash[21372]: cluster 2024-03-11T21:06:43.953936+0000 osd.6 (osd.6) 244 : cluster 0 6.1e scrub starts 2024-03-11T21:06:45.764 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:45 smithi023 bash[21372]: cluster 2024-03-11T21:06:43.967723+0000 osd.6 (osd.6) 245 : cluster 0 6.1e scrub ok 2024-03-11T21:06:45.764 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:45 smithi023 bash[21372]: cluster 2024-03-11T21:06:44.230271+0000 mgr.y (mgr.24380) 590 : cluster 0 pgmap v655: 148 pgs: 148 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 682 B/s rd, 0 B/s wr, 74 op/s 2024-03-11T21:06:45.764 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:45 smithi023 bash[21372]: cluster 2024-03-11T21:06:44.386949+0000 osd.3 (osd.3) 336 : cluster 0 5.1 scrub starts 2024-03-11T21:06:45.764 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:45 smithi023 bash[21372]: cluster 2024-03-11T21:06:44.388826+0000 osd.3 (osd.3) 337 : cluster 0 5.1 scrub ok 2024-03-11T21:06:45.765 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:45 smithi023 bash[29146]: cluster 2024-03-11T21:06:43.726333+0000 osd.1 (osd.1) 115 : cluster 0 9.4 scrub starts 2024-03-11T21:06:45.765 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:45 smithi023 bash[29146]: cluster 2024-03-11T21:06:43.762900+0000 osd.1 (osd.1) 116 : cluster 0 9.4 scrub ok 2024-03-11T21:06:45.765 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:45 smithi023 bash[29146]: cluster 2024-03-11T21:06:43.953936+0000 osd.6 (osd.6) 244 : cluster 0 6.1e scrub starts 2024-03-11T21:06:45.765 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:45 smithi023 bash[29146]: cluster 2024-03-11T21:06:43.967723+0000 osd.6 (osd.6) 245 : cluster 0 6.1e scrub ok 2024-03-11T21:06:45.765 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:45 smithi023 bash[29146]: cluster 2024-03-11T21:06:44.230271+0000 mgr.y (mgr.24380) 590 : cluster 0 pgmap v655: 148 pgs: 148 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 682 B/s rd, 0 B/s wr, 74 op/s 2024-03-11T21:06:45.765 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:45 smithi023 bash[29146]: cluster 2024-03-11T21:06:44.386949+0000 osd.3 (osd.3) 336 : cluster 0 5.1 scrub starts 2024-03-11T21:06:45.765 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:45 smithi023 bash[29146]: cluster 2024-03-11T21:06:44.388826+0000 osd.3 (osd.3) 337 : cluster 0 5.1 scrub ok 2024-03-11T21:06:45.781 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:06:45.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:45.784 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:06:45.885 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:06:45.896 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:06:45 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:06:45.897 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:45 smithi146 bash[23809]: cluster 2024-03-11T21:06:43.726333+0000 osd.1 (osd.1) 115 : cluster 0 9.4 scrub starts 2024-03-11T21:06:45.897 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:45 smithi146 bash[23809]: cluster 2024-03-11T21:06:43.762900+0000 osd.1 (osd.1) 116 : cluster 0 9.4 scrub ok 2024-03-11T21:06:45.897 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:45 smithi146 bash[23809]: cluster 2024-03-11T21:06:43.953936+0000 osd.6 (osd.6) 244 : cluster 0 6.1e scrub starts 2024-03-11T21:06:45.897 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:45 smithi146 bash[23809]: cluster 2024-03-11T21:06:43.967723+0000 osd.6 (osd.6) 245 : cluster 0 6.1e scrub ok 2024-03-11T21:06:45.897 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:45 smithi146 bash[23809]: cluster 2024-03-11T21:06:44.230271+0000 mgr.y (mgr.24380) 590 : cluster 0 pgmap v655: 148 pgs: 148 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 682 B/s rd, 0 B/s wr, 74 op/s 2024-03-11T21:06:45.897 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:45 smithi146 bash[23809]: cluster 2024-03-11T21:06:44.386949+0000 osd.3 (osd.3) 336 : cluster 0 5.1 scrub starts 2024-03-11T21:06:45.897 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:45 smithi146 bash[23809]: cluster 2024-03-11T21:06:44.388826+0000 osd.3 (osd.3) 337 : cluster 0 5.1 scrub ok 2024-03-11T21:06:45.910 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:06:45.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:45.913 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:06:46.013 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:06:46.041 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:06:46.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:46.043 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:06:46.144 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:46.170 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:46.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:46.172 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:06:46.273 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:46.299 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:46.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:46.302 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:06:46.402 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:06:46.431 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:06:46.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:46.433 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:06:46.534 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:06:46.562 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:06:46.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:46.564 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:06:46.612 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-03-11T21:06:46.666 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:06:46.677 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:46 smithi146 bash[23809]: cluster 2024-03-11T21:06:44.903800+0000 osd.0 (osd.0) 231 : cluster 0 4.16 scrub starts 2024-03-11T21:06:46.677 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:46 smithi146 bash[23809]: cluster 2024-03-11T21:06:44.905334+0000 osd.0 (osd.0) 232 : cluster 0 4.16 scrub ok 2024-03-11T21:06:46.677 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:46 smithi146 bash[23809]: audit 2024-03-11T21:06:45.633152+0000 mgr.y (mgr.24380) 591 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:06:46.693 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:06:46.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:46.696 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:06:46.797 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:46.825 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:46.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:46.827 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:06:46.928 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:06:46.939 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:46 smithi023 bash[29146]: cluster 2024-03-11T21:06:44.903800+0000 osd.0 (osd.0) 231 : cluster 0 4.16 scrub starts 2024-03-11T21:06:46.940 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:46 smithi023 bash[29146]: cluster 2024-03-11T21:06:44.905334+0000 osd.0 (osd.0) 232 : cluster 0 4.16 scrub ok 2024-03-11T21:06:46.940 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:46 smithi023 bash[29146]: audit 2024-03-11T21:06:45.633152+0000 mgr.y (mgr.24380) 591 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:06:46.940 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:46 smithi023 bash[21372]: cluster 2024-03-11T21:06:44.903800+0000 osd.0 (osd.0) 231 : cluster 0 4.16 scrub starts 2024-03-11T21:06:46.940 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:46 smithi023 bash[21372]: cluster 2024-03-11T21:06:44.905334+0000 osd.0 (osd.0) 232 : cluster 0 4.16 scrub ok 2024-03-11T21:06:46.940 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:46 smithi023 bash[21372]: audit 2024-03-11T21:06:45.633152+0000 mgr.y (mgr.24380) 591 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:06:46.955 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:06:46.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:46.958 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:06:47.059 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:06:47.086 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:06:47.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:47.088 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:06:47.189 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:06:47.218 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:06:47.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:47.221 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:06:47.322 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:47.349 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:47.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:47.351 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:06:47.452 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:06:47.481 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:06:47.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:47.483 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:06:47.584 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:06:47.613 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:06:47.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:47.615 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:06:47.716 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:47.741 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:47.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:47.744 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:06:47.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:47 smithi023 bash[21372]: cluster 2024-03-11T21:06:45.689372+0000 osd.1 (osd.1) 117 : cluster 0 3.19 scrub starts 2024-03-11T21:06:47.781 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:47 smithi023 bash[21372]: cluster 2024-03-11T21:06:45.693848+0000 osd.1 (osd.1) 118 : cluster 0 3.19 scrub ok 2024-03-11T21:06:47.781 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:47 smithi023 bash[21372]: cluster 2024-03-11T21:06:45.883180+0000 osd.0 (osd.0) 233 : cluster 0 3.12 scrub starts 2024-03-11T21:06:47.781 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:47 smithi023 bash[21372]: cluster 2024-03-11T21:06:45.887044+0000 osd.0 (osd.0) 234 : cluster 0 3.12 scrub ok 2024-03-11T21:06:47.781 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:47 smithi023 bash[21372]: cluster 2024-03-11T21:06:46.231882+0000 mgr.y (mgr.24380) 592 : cluster 0 pgmap v656: 148 pgs: 148 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 74 op/s 2024-03-11T21:06:47.781 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:47 smithi023 bash[21372]: cluster 2024-03-11T21:06:46.830093+0000 osd.5 (osd.5) 317 : cluster 0 2.18 scrub starts 2024-03-11T21:06:47.781 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:47 smithi023 bash[21372]: cluster 2024-03-11T21:06:46.831401+0000 osd.5 (osd.5) 318 : cluster 0 2.18 scrub ok 2024-03-11T21:06:47.781 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:47 smithi023 bash[29146]: cluster 2024-03-11T21:06:45.689372+0000 osd.1 (osd.1) 117 : cluster 0 3.19 scrub starts 2024-03-11T21:06:47.781 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:47 smithi023 bash[29146]: cluster 2024-03-11T21:06:45.693848+0000 osd.1 (osd.1) 118 : cluster 0 3.19 scrub ok 2024-03-11T21:06:47.781 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:47 smithi023 bash[29146]: cluster 2024-03-11T21:06:45.883180+0000 osd.0 (osd.0) 233 : cluster 0 3.12 scrub starts 2024-03-11T21:06:47.781 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:47 smithi023 bash[29146]: cluster 2024-03-11T21:06:45.887044+0000 osd.0 (osd.0) 234 : cluster 0 3.12 scrub ok 2024-03-11T21:06:47.781 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:47 smithi023 bash[29146]: cluster 2024-03-11T21:06:46.231882+0000 mgr.y (mgr.24380) 592 : cluster 0 pgmap v656: 148 pgs: 148 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 74 op/s 2024-03-11T21:06:47.782 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:47 smithi023 bash[29146]: cluster 2024-03-11T21:06:46.830093+0000 osd.5 (osd.5) 317 : cluster 0 2.18 scrub starts 2024-03-11T21:06:47.782 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:47 smithi023 bash[29146]: cluster 2024-03-11T21:06:46.831401+0000 osd.5 (osd.5) 318 : cluster 0 2.18 scrub ok 2024-03-11T21:06:47.844 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:47.855 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:47 smithi146 bash[23809]: cluster 2024-03-11T21:06:45.689372+0000 osd.1 (osd.1) 117 : cluster 0 3.19 scrub starts 2024-03-11T21:06:47.855 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:47 smithi146 bash[23809]: cluster 2024-03-11T21:06:45.693848+0000 osd.1 (osd.1) 118 : cluster 0 3.19 scrub ok 2024-03-11T21:06:47.855 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:47 smithi146 bash[23809]: cluster 2024-03-11T21:06:45.883180+0000 osd.0 (osd.0) 233 : cluster 0 3.12 scrub starts 2024-03-11T21:06:47.855 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:47 smithi146 bash[23809]: cluster 2024-03-11T21:06:45.887044+0000 osd.0 (osd.0) 234 : cluster 0 3.12 scrub ok 2024-03-11T21:06:47.855 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:47 smithi146 bash[23809]: cluster 2024-03-11T21:06:46.231882+0000 mgr.y (mgr.24380) 592 : cluster 0 pgmap v656: 148 pgs: 148 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 74 op/s 2024-03-11T21:06:47.855 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:47 smithi146 bash[23809]: cluster 2024-03-11T21:06:46.830093+0000 osd.5 (osd.5) 317 : cluster 0 2.18 scrub starts 2024-03-11T21:06:47.855 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:47 smithi146 bash[23809]: cluster 2024-03-11T21:06:46.831401+0000 osd.5 (osd.5) 318 : cluster 0 2.18 scrub ok 2024-03-11T21:06:47.869 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:47.869 INFO:teuthology.orchestra.run.smithi023.stdout:osd.0: {} 2024-03-11T21:06:47.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:47.871 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:06:47.890 INFO:teuthology.orchestra.run.smithi023.stdout:osd.1: {} 2024-03-11T21:06:47.910 INFO:teuthology.orchestra.run.smithi023.stdout:osd.2: {} 2024-03-11T21:06:47.931 INFO:teuthology.orchestra.run.smithi023.stdout:osd.3: {} 2024-03-11T21:06:47.956 INFO:teuthology.orchestra.run.smithi023.stdout:osd.4: {} 2024-03-11T21:06:47.973 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:47.981 INFO:teuthology.orchestra.run.smithi023.stdout:osd.5: {} 2024-03-11T21:06:48.000 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:48.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:48.003 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:06:48.005 INFO:teuthology.orchestra.run.smithi023.stdout:osd.6: {} 2024-03-11T21:06:48.027 INFO:teuthology.orchestra.run.smithi023.stdout:osd.7: {} 2024-03-11T21:06:48.029 INFO:teuthology.orchestra.run.smithi023.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:06:48.029 INFO:teuthology.orchestra.run.smithi023.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:06:48.029 INFO:teuthology.orchestra.run.smithi023.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:06:48.029 INFO:teuthology.orchestra.run.smithi023.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:06:48.029 INFO:teuthology.orchestra.run.smithi023.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:06:48.029 INFO:teuthology.orchestra.run.smithi023.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:06:48.030 INFO:teuthology.orchestra.run.smithi023.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:06:48.030 INFO:teuthology.orchestra.run.smithi023.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:06:48.104 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:06:48.130 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:06:48.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:48.132 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:06:48.233 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:48.261 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:48.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:48.264 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:06:48.364 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:06:48.393 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:06:48.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:48.395 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:06:48.477 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:06:48 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:06:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:06:48.496 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:06:48.521 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:06:48.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:48.523 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:06:48.624 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:06:48.651 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:06:48.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:48.654 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:06:48.755 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:48.782 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:48.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:48.785 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:06:48.886 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:48.895 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:48 smithi146 bash[23809]: cluster 2024-03-11T21:06:46.931348+0000 osd.0 (osd.0) 235 : cluster 0 6.0 scrub starts 2024-03-11T21:06:48.895 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:48 smithi146 bash[23809]: cluster 2024-03-11T21:06:46.969132+0000 osd.0 (osd.0) 236 : cluster 0 6.0 scrub ok 2024-03-11T21:06:48.895 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:48 smithi146 bash[23809]: cluster 2024-03-11T21:06:47.190201+0000 osd.7 (osd.7) 365 : cluster 0 9.e scrub starts 2024-03-11T21:06:48.895 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:48 smithi146 bash[23809]: cluster 2024-03-11T21:06:47.231594+0000 osd.7 (osd.7) 366 : cluster 0 9.e scrub ok 2024-03-11T21:06:48.896 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:48 smithi146 bash[23809]: cluster 2024-03-11T21:06:47.463222+0000 osd.3 (osd.3) 338 : cluster 0 6.b scrub starts 2024-03-11T21:06:48.896 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:48 smithi146 bash[23809]: cluster 2024-03-11T21:06:47.558206+0000 osd.3 (osd.3) 339 : cluster 0 6.b scrub ok 2024-03-11T21:06:48.896 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:48 smithi146 bash[23809]: cluster 2024-03-11T21:06:47.642555+0000 osd.1 (osd.1) 119 : cluster 0 2.6 deep-scrub starts 2024-03-11T21:06:48.896 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:48 smithi146 bash[23809]: cluster 2024-03-11T21:06:47.645670+0000 osd.1 (osd.1) 120 : cluster 0 2.6 deep-scrub ok 2024-03-11T21:06:48.896 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:48 smithi146 bash[23809]: audit 2024-03-11T21:06:47.855814+0000 mon.a (mon.0) 1579 : audit 0 from='client.? 172.21.15.23:0/3771995967' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:06:48.911 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:48.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:48.913 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:06:49.014 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:06:49.027 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:48 smithi023 bash[21372]: cluster 2024-03-11T21:06:46.931348+0000 osd.0 (osd.0) 235 : cluster 0 6.0 scrub starts 2024-03-11T21:06:49.027 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:48 smithi023 bash[21372]: cluster 2024-03-11T21:06:46.969132+0000 osd.0 (osd.0) 236 : cluster 0 6.0 scrub ok 2024-03-11T21:06:49.027 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:48 smithi023 bash[21372]: cluster 2024-03-11T21:06:47.190201+0000 osd.7 (osd.7) 365 : cluster 0 9.e scrub starts 2024-03-11T21:06:49.027 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:48 smithi023 bash[21372]: cluster 2024-03-11T21:06:47.231594+0000 osd.7 (osd.7) 366 : cluster 0 9.e scrub ok 2024-03-11T21:06:49.027 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:48 smithi023 bash[21372]: cluster 2024-03-11T21:06:47.463222+0000 osd.3 (osd.3) 338 : cluster 0 6.b scrub starts 2024-03-11T21:06:49.027 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:48 smithi023 bash[21372]: cluster 2024-03-11T21:06:47.558206+0000 osd.3 (osd.3) 339 : cluster 0 6.b scrub ok 2024-03-11T21:06:49.027 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:48 smithi023 bash[21372]: cluster 2024-03-11T21:06:47.642555+0000 osd.1 (osd.1) 119 : cluster 0 2.6 deep-scrub starts 2024-03-11T21:06:49.027 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:48 smithi023 bash[21372]: cluster 2024-03-11T21:06:47.645670+0000 osd.1 (osd.1) 120 : cluster 0 2.6 deep-scrub ok 2024-03-11T21:06:49.027 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:48 smithi023 bash[21372]: audit 2024-03-11T21:06:47.855814+0000 mon.a (mon.0) 1579 : audit 0 from='client.? 172.21.15.23:0/3771995967' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:06:49.028 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:48 smithi023 bash[29146]: cluster 2024-03-11T21:06:46.931348+0000 osd.0 (osd.0) 235 : cluster 0 6.0 scrub starts 2024-03-11T21:06:49.028 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:48 smithi023 bash[29146]: cluster 2024-03-11T21:06:46.969132+0000 osd.0 (osd.0) 236 : cluster 0 6.0 scrub ok 2024-03-11T21:06:49.028 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:48 smithi023 bash[29146]: cluster 2024-03-11T21:06:47.190201+0000 osd.7 (osd.7) 365 : cluster 0 9.e scrub starts 2024-03-11T21:06:49.028 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:48 smithi023 bash[29146]: cluster 2024-03-11T21:06:47.231594+0000 osd.7 (osd.7) 366 : cluster 0 9.e scrub ok 2024-03-11T21:06:49.028 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:48 smithi023 bash[29146]: cluster 2024-03-11T21:06:47.463222+0000 osd.3 (osd.3) 338 : cluster 0 6.b scrub starts 2024-03-11T21:06:49.028 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:48 smithi023 bash[29146]: cluster 2024-03-11T21:06:47.558206+0000 osd.3 (osd.3) 339 : cluster 0 6.b scrub ok 2024-03-11T21:06:49.028 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:48 smithi023 bash[29146]: cluster 2024-03-11T21:06:47.642555+0000 osd.1 (osd.1) 119 : cluster 0 2.6 deep-scrub starts 2024-03-11T21:06:49.028 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:48 smithi023 bash[29146]: cluster 2024-03-11T21:06:47.645670+0000 osd.1 (osd.1) 120 : cluster 0 2.6 deep-scrub ok 2024-03-11T21:06:49.028 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:48 smithi023 bash[29146]: audit 2024-03-11T21:06:47.855814+0000 mon.a (mon.0) 1579 : audit 0 from='client.? 172.21.15.23:0/3771995967' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:06:49.045 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:06:49.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:49.048 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:06:49.148 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:06:49.189 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:06:49.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:49.193 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:06:49.293 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:49.326 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:49.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:49.329 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:06:49.430 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:49.463 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:49.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:49.465 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:06:49.566 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:06:49.595 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:06:49.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:49.597 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:06:49.698 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:06:49.726 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:06:49.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:49.728 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:06:49.781 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:49 smithi023 bash[21372]: cluster 2024-03-11T21:06:47.819547+0000 osd.6 (osd.6) 246 : cluster 0 4.1f deep-scrub starts 2024-03-11T21:06:49.781 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:49 smithi023 bash[21372]: cluster 2024-03-11T21:06:47.821216+0000 osd.6 (osd.6) 247 : cluster 0 4.1f deep-scrub ok 2024-03-11T21:06:49.781 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:49 smithi023 bash[21372]: cluster 2024-03-11T21:06:48.232784+0000 mgr.y (mgr.24380) 593 : cluster 0 pgmap v657: 148 pgs: 148 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:06:49.781 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:49 smithi023 bash[21372]: cluster 2024-03-11T21:06:48.788183+0000 osd.5 (osd.5) 319 : cluster 0 9.f scrub starts 2024-03-11T21:06:49.781 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:49 smithi023 bash[21372]: cluster 2024-03-11T21:06:48.799722+0000 osd.5 (osd.5) 320 : cluster 0 9.f scrub ok 2024-03-11T21:06:49.781 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:49 smithi023 bash[29146]: cluster 2024-03-11T21:06:47.819547+0000 osd.6 (osd.6) 246 : cluster 0 4.1f deep-scrub starts 2024-03-11T21:06:49.781 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:49 smithi023 bash[29146]: cluster 2024-03-11T21:06:47.821216+0000 osd.6 (osd.6) 247 : cluster 0 4.1f deep-scrub ok 2024-03-11T21:06:49.782 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:49 smithi023 bash[29146]: cluster 2024-03-11T21:06:48.232784+0000 mgr.y (mgr.24380) 593 : cluster 0 pgmap v657: 148 pgs: 148 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:06:49.782 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:49 smithi023 bash[29146]: cluster 2024-03-11T21:06:48.788183+0000 osd.5 (osd.5) 319 : cluster 0 9.f scrub starts 2024-03-11T21:06:49.782 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:49 smithi023 bash[29146]: cluster 2024-03-11T21:06:48.799722+0000 osd.5 (osd.5) 320 : cluster 0 9.f scrub ok 2024-03-11T21:06:49.828 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:49.855 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:49.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:49.857 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:06:49.957 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:49.969 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:49 smithi146 bash[23809]: cluster 2024-03-11T21:06:47.819547+0000 osd.6 (osd.6) 246 : cluster 0 4.1f deep-scrub starts 2024-03-11T21:06:49.969 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:49 smithi146 bash[23809]: cluster 2024-03-11T21:06:47.821216+0000 osd.6 (osd.6) 247 : cluster 0 4.1f deep-scrub ok 2024-03-11T21:06:49.969 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:49 smithi146 bash[23809]: cluster 2024-03-11T21:06:48.232784+0000 mgr.y (mgr.24380) 593 : cluster 0 pgmap v657: 148 pgs: 148 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:06:49.969 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:49 smithi146 bash[23809]: cluster 2024-03-11T21:06:48.788183+0000 osd.5 (osd.5) 319 : cluster 0 9.f scrub starts 2024-03-11T21:06:49.969 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:49 smithi146 bash[23809]: cluster 2024-03-11T21:06:48.799722+0000 osd.5 (osd.5) 320 : cluster 0 9.f scrub ok 2024-03-11T21:06:49.985 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:49.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:49.987 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:06:50.088 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:50.118 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:50.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:50.120 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:06:50.221 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:06:50.249 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:06:50.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:50.251 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:06:50.352 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:06:50.379 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:06:50.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:50.381 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:06:50.481 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:50.504 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:50.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:50.506 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:06:50.607 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:06:50.634 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:06:50.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:50.637 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:06:50.738 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:50.746 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:50 smithi023 bash[29146]: cluster 2024-03-11T21:06:49.622428+0000 osd.1 (osd.1) 121 : cluster 0 2.15 scrub starts 2024-03-11T21:06:50.746 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:50 smithi023 bash[29146]: cluster 2024-03-11T21:06:49.624111+0000 osd.1 (osd.1) 122 : cluster 0 2.15 scrub ok 2024-03-11T21:06:50.746 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:50 smithi023 bash[29146]: cluster 2024-03-11T21:06:50.234041+0000 mgr.y (mgr.24380) 594 : cluster 0 pgmap v658: 148 pgs: 148 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:06:50.747 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:50 smithi023 bash[29146]: audit 2024-03-11T21:06:50.575735+0000 mon.a (mon.0) 1580 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:06:50.747 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:50 smithi023 bash[21372]: cluster 2024-03-11T21:06:49.622428+0000 osd.1 (osd.1) 121 : cluster 0 2.15 scrub starts 2024-03-11T21:06:50.747 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:50 smithi023 bash[21372]: cluster 2024-03-11T21:06:49.624111+0000 osd.1 (osd.1) 122 : cluster 0 2.15 scrub ok 2024-03-11T21:06:50.747 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:50 smithi023 bash[21372]: cluster 2024-03-11T21:06:50.234041+0000 mgr.y (mgr.24380) 594 : cluster 0 pgmap v658: 148 pgs: 148 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:06:50.747 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:50 smithi023 bash[21372]: audit 2024-03-11T21:06:50.575735+0000 mon.a (mon.0) 1580 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:06:50.758 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:50.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:50.761 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:06:50.861 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:50.873 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:50 smithi146 bash[23809]: cluster 2024-03-11T21:06:49.622428+0000 osd.1 (osd.1) 121 : cluster 0 2.15 scrub starts 2024-03-11T21:06:50.873 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:50 smithi146 bash[23809]: cluster 2024-03-11T21:06:49.624111+0000 osd.1 (osd.1) 122 : cluster 0 2.15 scrub ok 2024-03-11T21:06:50.873 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:50 smithi146 bash[23809]: cluster 2024-03-11T21:06:50.234041+0000 mgr.y (mgr.24380) 594 : cluster 0 pgmap v658: 148 pgs: 148 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:06:50.873 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:50 smithi146 bash[23809]: audit 2024-03-11T21:06:50.575735+0000 mon.a (mon.0) 1580 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:06:50.886 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:50.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:50.888 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:06:50.989 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:51.019 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:51.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:51.021 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:06:51.122 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:06:51.149 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:06:51.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:51.152 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:06:51.252 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:51.281 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:51.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:51.283 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:06:51.384 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:06:51.410 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:06:51.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:51.412 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:06:51.512 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:51.539 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:51.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:51.541 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:06:51.641 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:51.671 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:51.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:51.674 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:06:51.775 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:06:51.786 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:51 smithi146 bash[23809]: cluster 2024-03-11T21:06:49.893266+0000 osd.0 (osd.0) 237 : cluster 0 3.1 scrub starts 2024-03-11T21:06:51.786 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:51 smithi146 bash[23809]: cluster 2024-03-11T21:06:49.897262+0000 osd.0 (osd.0) 238 : cluster 0 3.1 scrub ok 2024-03-11T21:06:51.786 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:51 smithi146 bash[23809]: cluster 2024-03-11T21:06:50.840121+0000 osd.5 (osd.5) 321 : cluster 0 9.a scrub starts 2024-03-11T21:06:51.786 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:51 smithi146 bash[23809]: cluster 2024-03-11T21:06:50.858093+0000 osd.0 (osd.0) 239 : cluster 0 9.5 scrub starts 2024-03-11T21:06:51.786 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:51 smithi146 bash[23809]: cluster 2024-03-11T21:06:50.895691+0000 osd.0 (osd.0) 240 : cluster 0 9.5 scrub ok 2024-03-11T21:06:51.801 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:06:51.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:51.806 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:06:51.906 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:06:51.934 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:06:51.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:51.937 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:06:52.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:51 smithi023 bash[21372]: cluster 2024-03-11T21:06:49.893266+0000 osd.0 (osd.0) 237 : cluster 0 3.1 scrub starts 2024-03-11T21:06:52.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:51 smithi023 bash[21372]: cluster 2024-03-11T21:06:49.897262+0000 osd.0 (osd.0) 238 : cluster 0 3.1 scrub ok 2024-03-11T21:06:52.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:51 smithi023 bash[21372]: cluster 2024-03-11T21:06:50.840121+0000 osd.5 (osd.5) 321 : cluster 0 9.a scrub starts 2024-03-11T21:06:52.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:51 smithi023 bash[21372]: cluster 2024-03-11T21:06:50.858093+0000 osd.0 (osd.0) 239 : cluster 0 9.5 scrub starts 2024-03-11T21:06:52.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:51 smithi023 bash[21372]: cluster 2024-03-11T21:06:50.895691+0000 osd.0 (osd.0) 240 : cluster 0 9.5 scrub ok 2024-03-11T21:06:52.031 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:51 smithi023 bash[29146]: cluster 2024-03-11T21:06:49.893266+0000 osd.0 (osd.0) 237 : cluster 0 3.1 scrub starts 2024-03-11T21:06:52.031 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:51 smithi023 bash[29146]: cluster 2024-03-11T21:06:49.897262+0000 osd.0 (osd.0) 238 : cluster 0 3.1 scrub ok 2024-03-11T21:06:52.031 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:51 smithi023 bash[29146]: cluster 2024-03-11T21:06:50.840121+0000 osd.5 (osd.5) 321 : cluster 0 9.a scrub starts 2024-03-11T21:06:52.032 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:51 smithi023 bash[29146]: cluster 2024-03-11T21:06:50.858093+0000 osd.0 (osd.0) 239 : cluster 0 9.5 scrub starts 2024-03-11T21:06:52.032 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:51 smithi023 bash[29146]: cluster 2024-03-11T21:06:50.895691+0000 osd.0 (osd.0) 240 : cluster 0 9.5 scrub ok 2024-03-11T21:06:52.038 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:52.065 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:52.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:52.068 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:06:52.169 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:06:52.194 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:06:52.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:52.196 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:06:52.297 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:52.324 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:52.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:52.326 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:06:52.427 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:06:52.454 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:06:52.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:52.456 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:06:52.557 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:06:52.585 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:06:52.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:52.588 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:06:52.689 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:52.700 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:52 smithi023 bash[21372]: cluster 2024-03-11T21:06:50.900406+0000 osd.5 (osd.5) 322 : cluster 0 9.a scrub ok 2024-03-11T21:06:52.700 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:52 smithi023 bash[21372]: cluster 2024-03-11T21:06:51.618249+0000 osd.1 (osd.1) 123 : cluster 0 3.0 scrub starts 2024-03-11T21:06:52.700 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:52 smithi023 bash[21372]: cluster 2024-03-11T21:06:51.623837+0000 osd.1 (osd.1) 124 : cluster 0 3.0 scrub ok 2024-03-11T21:06:52.701 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:52 smithi023 bash[21372]: cluster 2024-03-11T21:06:51.744831+0000 osd.6 (osd.6) 248 : cluster 0 4.18 scrub starts 2024-03-11T21:06:52.701 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:52 smithi023 bash[21372]: cluster 2024-03-11T21:06:51.747178+0000 osd.6 (osd.6) 249 : cluster 0 4.18 scrub ok 2024-03-11T21:06:52.701 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:52 smithi023 bash[21372]: cluster 2024-03-11T21:06:52.235073+0000 mgr.y (mgr.24380) 595 : cluster 0 pgmap v659: 148 pgs: 148 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:06:52.701 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:52 smithi023 bash[21372]: cluster 2024-03-11T21:06:52.268507+0000 osd.7 (osd.7) 367 : cluster 0 3.11 scrub starts 2024-03-11T21:06:52.701 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:52 smithi023 bash[21372]: cluster 2024-03-11T21:06:52.270672+0000 osd.7 (osd.7) 368 : cluster 0 3.11 scrub ok 2024-03-11T21:06:52.701 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:52 smithi023 bash[29146]: cluster 2024-03-11T21:06:50.900406+0000 osd.5 (osd.5) 322 : cluster 0 9.a scrub ok 2024-03-11T21:06:52.701 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:52 smithi023 bash[29146]: cluster 2024-03-11T21:06:51.618249+0000 osd.1 (osd.1) 123 : cluster 0 3.0 scrub starts 2024-03-11T21:06:52.701 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:52 smithi023 bash[29146]: cluster 2024-03-11T21:06:51.623837+0000 osd.1 (osd.1) 124 : cluster 0 3.0 scrub ok 2024-03-11T21:06:52.701 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:52 smithi023 bash[29146]: cluster 2024-03-11T21:06:51.744831+0000 osd.6 (osd.6) 248 : cluster 0 4.18 scrub starts 2024-03-11T21:06:52.701 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:52 smithi023 bash[29146]: cluster 2024-03-11T21:06:51.747178+0000 osd.6 (osd.6) 249 : cluster 0 4.18 scrub ok 2024-03-11T21:06:52.701 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:52 smithi023 bash[29146]: cluster 2024-03-11T21:06:52.235073+0000 mgr.y (mgr.24380) 595 : cluster 0 pgmap v659: 148 pgs: 148 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:06:52.701 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:52 smithi023 bash[29146]: cluster 2024-03-11T21:06:52.268507+0000 osd.7 (osd.7) 367 : cluster 0 3.11 scrub starts 2024-03-11T21:06:52.702 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:52 smithi023 bash[29146]: cluster 2024-03-11T21:06:52.270672+0000 osd.7 (osd.7) 368 : cluster 0 3.11 scrub ok 2024-03-11T21:06:52.713 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:52.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:52.715 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:06:52.728 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T21:06:52.817 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:52.829 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:52 smithi146 bash[23809]: cluster 2024-03-11T21:06:50.900406+0000 osd.5 (osd.5) 322 : cluster 0 9.a scrub ok 2024-03-11T21:06:52.829 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:52 smithi146 bash[23809]: cluster 2024-03-11T21:06:51.618249+0000 osd.1 (osd.1) 123 : cluster 0 3.0 scrub starts 2024-03-11T21:06:52.829 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:52 smithi146 bash[23809]: cluster 2024-03-11T21:06:51.623837+0000 osd.1 (osd.1) 124 : cluster 0 3.0 scrub ok 2024-03-11T21:06:52.829 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:52 smithi146 bash[23809]: cluster 2024-03-11T21:06:51.744831+0000 osd.6 (osd.6) 248 : cluster 0 4.18 scrub starts 2024-03-11T21:06:52.829 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:52 smithi146 bash[23809]: cluster 2024-03-11T21:06:51.747178+0000 osd.6 (osd.6) 249 : cluster 0 4.18 scrub ok 2024-03-11T21:06:52.829 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:52 smithi146 bash[23809]: cluster 2024-03-11T21:06:52.235073+0000 mgr.y (mgr.24380) 595 : cluster 0 pgmap v659: 148 pgs: 148 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:06:52.829 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:52 smithi146 bash[23809]: cluster 2024-03-11T21:06:52.268507+0000 osd.7 (osd.7) 367 : cluster 0 3.11 scrub starts 2024-03-11T21:06:52.829 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:52 smithi146 bash[23809]: cluster 2024-03-11T21:06:52.270672+0000 osd.7 (osd.7) 368 : cluster 0 3.11 scrub ok 2024-03-11T21:06:52.846 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:52.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:52.848 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:06:52.949 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:06:52.974 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:06:52.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:52.976 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:06:53.077 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:53.103 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:53.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:53.105 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:06:53.206 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:53.234 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:53.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:53.236 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:06:53.336 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:53.364 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:53.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:53.367 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:06:53.467 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:53.494 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:53.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:53.496 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:06:53.597 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:06:53.625 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:06:53.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:53.628 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:06:53.707 INFO:teuthology.orchestra.run.smithi023.stderr:noscrub is set 2024-03-11T21:06:53.729 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:53.740 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:53 smithi146 bash[23809]: cluster 2024-03-11T21:06:52.861505+0000 osd.0 (osd.0) 241 : cluster 0 2.13 deep-scrub starts 2024-03-11T21:06:53.740 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:53 smithi146 bash[23809]: cluster 2024-03-11T21:06:52.862962+0000 osd.0 (osd.0) 242 : cluster 0 2.13 deep-scrub ok 2024-03-11T21:06:53.740 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:53 smithi146 bash[23809]: audit 2024-03-11T21:06:52.955191+0000 mon.c (mon.2) 74 : audit 1 from='client.? 172.21.15.23:0/2290056978' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:06:53.740 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:53 smithi146 bash[23809]: audit 2024-03-11T21:06:52.955506+0000 mon.a (mon.0) 1581 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:06:53.740 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:53 smithi146 bash[23809]: cluster 2024-03-11T21:06:53.460878+0000 osd.7 (osd.7) 369 : cluster 0 purged_snaps scrub starts 2024-03-11T21:06:53.741 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:53 smithi146 bash[23809]: cluster 2024-03-11T21:06:53.461024+0000 osd.7 (osd.7) 370 : cluster 0 purged_snaps scrub ok 2024-03-11T21:06:53.755 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:53.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:53.757 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:06:53.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:53 smithi023 bash[21372]: cluster 2024-03-11T21:06:52.861505+0000 osd.0 (osd.0) 241 : cluster 0 2.13 deep-scrub starts 2024-03-11T21:06:53.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:53 smithi023 bash[21372]: cluster 2024-03-11T21:06:52.862962+0000 osd.0 (osd.0) 242 : cluster 0 2.13 deep-scrub ok 2024-03-11T21:06:53.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:53 smithi023 bash[21372]: audit 2024-03-11T21:06:52.955191+0000 mon.c (mon.2) 74 : audit 1 from='client.? 172.21.15.23:0/2290056978' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:06:53.781 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:53 smithi023 bash[21372]: audit 2024-03-11T21:06:52.955506+0000 mon.a (mon.0) 1581 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:06:53.781 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:53 smithi023 bash[21372]: cluster 2024-03-11T21:06:53.460878+0000 osd.7 (osd.7) 369 : cluster 0 purged_snaps scrub starts 2024-03-11T21:06:53.781 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:53 smithi023 bash[21372]: cluster 2024-03-11T21:06:53.461024+0000 osd.7 (osd.7) 370 : cluster 0 purged_snaps scrub ok 2024-03-11T21:06:53.781 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:53 smithi023 bash[29146]: cluster 2024-03-11T21:06:52.861505+0000 osd.0 (osd.0) 241 : cluster 0 2.13 deep-scrub starts 2024-03-11T21:06:53.781 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:53 smithi023 bash[29146]: cluster 2024-03-11T21:06:52.862962+0000 osd.0 (osd.0) 242 : cluster 0 2.13 deep-scrub ok 2024-03-11T21:06:53.781 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:53 smithi023 bash[29146]: audit 2024-03-11T21:06:52.955191+0000 mon.c (mon.2) 74 : audit 1 from='client.? 172.21.15.23:0/2290056978' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:06:53.781 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:53 smithi023 bash[29146]: audit 2024-03-11T21:06:52.955506+0000 mon.a (mon.0) 1581 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:06:53.781 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:53 smithi023 bash[29146]: cluster 2024-03-11T21:06:53.460878+0000 osd.7 (osd.7) 369 : cluster 0 purged_snaps scrub starts 2024-03-11T21:06:53.782 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:53 smithi023 bash[29146]: cluster 2024-03-11T21:06:53.461024+0000 osd.7 (osd.7) 370 : cluster 0 purged_snaps scrub ok 2024-03-11T21:06:53.858 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:53.886 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:53.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:53.888 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:06:53.989 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:06:54.017 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:06:54.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:54.020 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:06:54.121 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:54.147 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:54.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:54.149 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:06:54.250 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:54.273 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:54.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:54.275 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:06:54.376 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:54.402 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:54.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:54.404 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:06:54.505 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:54.532 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:54.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:54.534 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:06:54.635 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:54.661 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:54.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:54.663 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:06:54.764 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:54.790 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:54.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:54.793 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:06:54.893 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:54.905 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:54 smithi146 bash[23809]: cluster 2024-03-11T21:06:53.531203+0000 osd.3 (osd.3) 340 : cluster 0 4.11 deep-scrub starts 2024-03-11T21:06:54.905 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:54 smithi146 bash[23809]: cluster 2024-03-11T21:06:53.533341+0000 osd.3 (osd.3) 341 : cluster 0 4.11 deep-scrub ok 2024-03-11T21:06:54.905 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:54 smithi146 bash[23809]: cluster 2024-03-11T21:06:53.676506+0000 mon.a (mon.0) 1582 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:06:54.905 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:54 smithi146 bash[23809]: audit 2024-03-11T21:06:53.696718+0000 mon.a (mon.0) 1583 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:06:54.905 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:54 smithi146 bash[23809]: cluster 2024-03-11T21:06:53.709901+0000 mon.a (mon.0) 1584 : cluster 0 osdmap e299: 8 total, 8 up, 8 in 2024-03-11T21:06:54.905 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:54 smithi146 bash[23809]: cluster 2024-03-11T21:06:54.235555+0000 mgr.y (mgr.24380) 596 : cluster 0 pgmap v661: 148 pgs: 148 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:06:54.921 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:54.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:54.923 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:06:55.024 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:55.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:54 smithi023 bash[21372]: cluster 2024-03-11T21:06:53.531203+0000 osd.3 (osd.3) 340 : cluster 0 4.11 deep-scrub starts 2024-03-11T21:06:55.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:54 smithi023 bash[21372]: cluster 2024-03-11T21:06:53.533341+0000 osd.3 (osd.3) 341 : cluster 0 4.11 deep-scrub ok 2024-03-11T21:06:55.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:54 smithi023 bash[21372]: cluster 2024-03-11T21:06:53.676506+0000 mon.a (mon.0) 1582 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:06:55.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:54 smithi023 bash[21372]: audit 2024-03-11T21:06:53.696718+0000 mon.a (mon.0) 1583 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:06:55.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:54 smithi023 bash[21372]: cluster 2024-03-11T21:06:53.709901+0000 mon.a (mon.0) 1584 : cluster 0 osdmap e299: 8 total, 8 up, 8 in 2024-03-11T21:06:55.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:54 smithi023 bash[21372]: cluster 2024-03-11T21:06:54.235555+0000 mgr.y (mgr.24380) 596 : cluster 0 pgmap v661: 148 pgs: 148 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:06:55.031 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:54 smithi023 bash[29146]: cluster 2024-03-11T21:06:53.531203+0000 osd.3 (osd.3) 340 : cluster 0 4.11 deep-scrub starts 2024-03-11T21:06:55.031 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:54 smithi023 bash[29146]: cluster 2024-03-11T21:06:53.533341+0000 osd.3 (osd.3) 341 : cluster 0 4.11 deep-scrub ok 2024-03-11T21:06:55.031 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:54 smithi023 bash[29146]: cluster 2024-03-11T21:06:53.676506+0000 mon.a (mon.0) 1582 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:06:55.031 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:54 smithi023 bash[29146]: audit 2024-03-11T21:06:53.696718+0000 mon.a (mon.0) 1583 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:06:55.031 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:54 smithi023 bash[29146]: cluster 2024-03-11T21:06:53.709901+0000 mon.a (mon.0) 1584 : cluster 0 osdmap e299: 8 total, 8 up, 8 in 2024-03-11T21:06:55.031 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:54 smithi023 bash[29146]: cluster 2024-03-11T21:06:54.235555+0000 mgr.y (mgr.24380) 596 : cluster 0 pgmap v661: 148 pgs: 148 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:06:55.051 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:55.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:55.053 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:06:55.153 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:55.178 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:55.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:55.180 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:06:55.281 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:06:55.307 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:06:55.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:55.309 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:06:55.410 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:06:55.437 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:06:55.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:55.439 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:06:55.540 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:55.567 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:55.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:55.569 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:06:55.670 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:55.696 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:55.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:55.699 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:06:55.800 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:55.820 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:55.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:55.823 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:06:55.924 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:55.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:06:55 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:06:55.935 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:55 smithi146 bash[23809]: cluster 2024-03-11T21:06:54.687919+0000 osd.5 (osd.5) 323 : cluster 0 purged_snaps scrub starts 2024-03-11T21:06:55.935 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:55 smithi146 bash[23809]: cluster 2024-03-11T21:06:54.688043+0000 osd.5 (osd.5) 324 : cluster 0 purged_snaps scrub ok 2024-03-11T21:06:55.935 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:55 smithi146 bash[23809]: cluster 2024-03-11T21:06:55.700341+0000 mon.a (mon.0) 1585 : cluster 0 osdmap e300: 8 total, 8 up, 8 in 2024-03-11T21:06:55.950 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:55.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:55.951 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:06:56.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:55 smithi023 bash[21372]: cluster 2024-03-11T21:06:54.687919+0000 osd.5 (osd.5) 323 : cluster 0 purged_snaps scrub starts 2024-03-11T21:06:56.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:55 smithi023 bash[21372]: cluster 2024-03-11T21:06:54.688043+0000 osd.5 (osd.5) 324 : cluster 0 purged_snaps scrub ok 2024-03-11T21:06:56.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:55 smithi023 bash[21372]: cluster 2024-03-11T21:06:55.700341+0000 mon.a (mon.0) 1585 : cluster 0 osdmap e300: 8 total, 8 up, 8 in 2024-03-11T21:06:56.031 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:55 smithi023 bash[29146]: cluster 2024-03-11T21:06:54.687919+0000 osd.5 (osd.5) 323 : cluster 0 purged_snaps scrub starts 2024-03-11T21:06:56.031 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:55 smithi023 bash[29146]: cluster 2024-03-11T21:06:54.688043+0000 osd.5 (osd.5) 324 : cluster 0 purged_snaps scrub ok 2024-03-11T21:06:56.031 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:55 smithi023 bash[29146]: cluster 2024-03-11T21:06:55.700341+0000 mon.a (mon.0) 1585 : cluster 0 osdmap e300: 8 total, 8 up, 8 in 2024-03-11T21:06:56.052 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:06:56.072 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:06:56.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:56.074 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:06:56.135 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-11T21:06:56.135 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-11T21:06:56.135 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 300, 2024-03-11T21:06:56.135 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "67ba00f8-dfe7-11ee-95c8-87774f69a715", 2024-03-11T21:06:56.135 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2024-03-11T20:40:13.664802+0000", 2024-03-11T21:06:56.135 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2024-03-11T21:06:55.687752+0000", 2024-03-11T21:06:56.135 INFO:teuthology.orchestra.run.smithi023.stdout: "last_up_change": "2024-03-11T21:00:47.324817+0000", 2024-03-11T21:06:56.135 INFO:teuthology.orchestra.run.smithi023.stdout: "last_in_change": "2024-03-11T20:57:23.560527+0000", 2024-03-11T21:06:56.135 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-03-11T21:06:56.135 INFO:teuthology.orchestra.run.smithi023.stdout: "flags_num": 5801984, 2024-03-11T21:06:56.136 INFO:teuthology.orchestra.run.smithi023.stdout: "flags_set": [ 2024-03-11T21:06:56.136 INFO:teuthology.orchestra.run.smithi023.stdout: "noscrub", 2024-03-11T21:06:56.136 INFO:teuthology.orchestra.run.smithi023.stdout: "pglog_hardlimit", 2024-03-11T21:06:56.136 INFO:teuthology.orchestra.run.smithi023.stdout: "purged_snapdirs", 2024-03-11T21:06:56.136 INFO:teuthology.orchestra.run.smithi023.stdout: "recovery_deletes", 2024-03-11T21:06:56.136 INFO:teuthology.orchestra.run.smithi023.stdout: "sortbitwise" 2024-03-11T21:06:56.136 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-11T21:06:56.136 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_version": 20, 2024-03-11T21:06:56.136 INFO:teuthology.orchestra.run.smithi023.stdout: "full_ratio": 0.94999998807907104, 2024-03-11T21:06:56.136 INFO:teuthology.orchestra.run.smithi023.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-03-11T21:06:56.136 INFO:teuthology.orchestra.run.smithi023.stdout: "nearfull_ratio": 0.85000002384185791, 2024-03-11T21:06:56.136 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_snapshot": "", 2024-03-11T21:06:56.136 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_max": 9, 2024-03-11T21:06:56.137 INFO:teuthology.orchestra.run.smithi023.stdout: "max_osd": 8, 2024-03-11T21:06:56.137 INFO:teuthology.orchestra.run.smithi023.stdout: "require_min_compat_client": "luminous", 2024-03-11T21:06:56.137 INFO:teuthology.orchestra.run.smithi023.stdout: "min_compat_client": "luminous", 2024-03-11T21:06:56.137 INFO:teuthology.orchestra.run.smithi023.stdout: "require_osd_release": "squid", 2024-03-11T21:06:56.137 INFO:teuthology.orchestra.run.smithi023.stdout: "allow_crimson": false, 2024-03-11T21:06:56.137 INFO:teuthology.orchestra.run.smithi023.stdout: "pools": [ 2024-03-11T21:06:56.137 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:06:56.137 INFO:teuthology.orchestra.run.smithi023.stdout: "pool": 1, 2024-03-11T21:06:56.137 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_name": ".mgr", 2024-03-11T21:06:56.137 INFO:teuthology.orchestra.run.smithi023.stdout: "create_time": "2024-03-11T20:45:31.964570+0000", 2024-03-11T21:06:56.137 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-11T21:06:56.137 INFO:teuthology.orchestra.run.smithi023.stdout: "flags_names": "hashpspool", 2024-03-11T21:06:56.137 INFO:teuthology.orchestra.run.smithi023.stdout: "type": 1, 2024-03-11T21:06:56.138 INFO:teuthology.orchestra.run.smithi023.stdout: "size": 3, 2024-03-11T21:06:56.138 INFO:teuthology.orchestra.run.smithi023.stdout: "min_size": 2, 2024-03-11T21:06:56.138 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_rule": 0, 2024-03-11T21:06:56.138 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_count": 0, 2024-03-11T21:06:56.138 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_target": 0, 2024-03-11T21:06:56.138 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T21:06:56.138 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T21:06:56.138 INFO:teuthology.orchestra.run.smithi023.stdout: "object_hash": 2, 2024-03-11T21:06:56.138 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_autoscale_mode": "off", 2024-03-11T21:06:56.138 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num": 1, 2024-03-11T21:06:56.138 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_placement_num": 1, 2024-03-11T21:06:56.138 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_placement_num_target": 1, 2024-03-11T21:06:56.138 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num_target": 1, 2024-03-11T21:06:56.139 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num_pending": 1, 2024-03-11T21:06:56.139 INFO:teuthology.orchestra.run.smithi023.stdout: "last_pg_merge_meta": { 2024-03-11T21:06:56.139 INFO:teuthology.orchestra.run.smithi023.stdout: "source_pgid": "0.0", 2024-03-11T21:06:56.139 INFO:teuthology.orchestra.run.smithi023.stdout: "ready_epoch": 0, 2024-03-11T21:06:56.139 INFO:teuthology.orchestra.run.smithi023.stdout: "last_epoch_started": 0, 2024-03-11T21:06:56.139 INFO:teuthology.orchestra.run.smithi023.stdout: "last_epoch_clean": 0, 2024-03-11T21:06:56.139 INFO:teuthology.orchestra.run.smithi023.stdout: "source_version": "0'0", 2024-03-11T21:06:56.139 INFO:teuthology.orchestra.run.smithi023.stdout: "target_version": "0'0" 2024-03-11T21:06:56.139 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:06:56.139 INFO:teuthology.orchestra.run.smithi023.stdout: "last_change": "142", 2024-03-11T21:06:56.139 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend": "0", 2024-03-11T21:06:56.139 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T21:06:56.139 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T21:06:56.140 INFO:teuthology.orchestra.run.smithi023.stdout: "auid": 0, 2024-03-11T21:06:56.140 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_mode": "selfmanaged", 2024-03-11T21:06:56.140 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_seq": 0, 2024-03-11T21:06:56.140 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_epoch": 0, 2024-03-11T21:06:56.140 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_snaps": [], 2024-03-11T21:06:56.140 INFO:teuthology.orchestra.run.smithi023.stdout: "removed_snaps": "[]", 2024-03-11T21:06:56.140 INFO:teuthology.orchestra.run.smithi023.stdout: "quota_max_bytes": 0, 2024-03-11T21:06:56.140 INFO:teuthology.orchestra.run.smithi023.stdout: "quota_max_objects": 0, 2024-03-11T21:06:56.140 INFO:teuthology.orchestra.run.smithi023.stdout: "tiers": [], 2024-03-11T21:06:56.140 INFO:teuthology.orchestra.run.smithi023.stdout: "tier_of": -1, 2024-03-11T21:06:56.140 INFO:teuthology.orchestra.run.smithi023.stdout: "read_tier": -1, 2024-03-11T21:06:56.140 INFO:teuthology.orchestra.run.smithi023.stdout: "write_tier": -1, 2024-03-11T21:06:56.140 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_mode": "none", 2024-03-11T21:06:56.140 INFO:teuthology.orchestra.run.smithi023.stdout: "target_max_bytes": 0, 2024-03-11T21:06:56.140 INFO:teuthology.orchestra.run.smithi023.stdout: "target_max_objects": 0, 2024-03-11T21:06:56.141 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T21:06:56.141 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T21:06:56.141 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T21:06:56.141 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_min_flush_age": 0, 2024-03-11T21:06:56.141 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_min_evict_age": 0, 2024-03-11T21:06:56.141 INFO:teuthology.orchestra.run.smithi023.stdout: "erasure_code_profile": "", 2024-03-11T21:06:56.141 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_params": { 2024-03-11T21:06:56.141 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "none" 2024-03-11T21:06:56.141 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:06:56.141 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_period": 0, 2024-03-11T21:06:56.141 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_count": 0, 2024-03-11T21:06:56.141 INFO:teuthology.orchestra.run.smithi023.stdout: "use_gmt_hitset": true, 2024-03-11T21:06:56.141 INFO:teuthology.orchestra.run.smithi023.stdout: "min_read_recency_for_promote": 0, 2024-03-11T21:06:56.141 INFO:teuthology.orchestra.run.smithi023.stdout: "min_write_recency_for_promote": 0, 2024-03-11T21:06:56.142 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T21:06:56.142 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_search_last_n": 0, 2024-03-11T21:06:56.142 INFO:teuthology.orchestra.run.smithi023.stdout: "grade_table": [], 2024-03-11T21:06:56.142 INFO:teuthology.orchestra.run.smithi023.stdout: "stripe_width": 0, 2024-03-11T21:06:56.142 INFO:teuthology.orchestra.run.smithi023.stdout: "expected_num_objects": 0, 2024-03-11T21:06:56.142 INFO:teuthology.orchestra.run.smithi023.stdout: "fast_read": false, 2024-03-11T21:06:56.142 INFO:teuthology.orchestra.run.smithi023.stdout: "options": { 2024-03-11T21:06:56.142 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num_max": 32, 2024-03-11T21:06:56.142 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num_min": 1 2024-03-11T21:06:56.142 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:06:56.142 INFO:teuthology.orchestra.run.smithi023.stdout: "application_metadata": { 2024-03-11T21:06:56.142 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr": {} 2024-03-11T21:06:56.142 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:06:56.143 INFO:teuthology.orchestra.run.smithi023.stdout: "read_balance": { 2024-03-11T21:06:56.143 INFO:teuthology.orchestra.run.smithi023.stdout: "score_type": "Fair distribution", 2024-03-11T21:06:56.143 INFO:teuthology.orchestra.run.smithi023.stdout: "score_acting": 6.9800000190734863, 2024-03-11T21:06:56.143 INFO:teuthology.orchestra.run.smithi023.stdout: "score_stable": 6.9800000190734863, 2024-03-11T21:06:56.143 INFO:teuthology.orchestra.run.smithi023.stdout: "optimal_score": 0.43000000715255737, 2024-03-11T21:06:56.143 INFO:teuthology.orchestra.run.smithi023.stdout: "raw_score_acting": 3, 2024-03-11T21:06:56.143 INFO:teuthology.orchestra.run.smithi023.stdout: "raw_score_stable": 3, 2024-03-11T21:06:56.143 INFO:teuthology.orchestra.run.smithi023.stdout: "primary_affinity_weighted": 0.875, 2024-03-11T21:06:56.143 INFO:teuthology.orchestra.run.smithi023.stdout: "average_primary_affinity": 1, 2024-03-11T21:06:56.143 INFO:teuthology.orchestra.run.smithi023.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-03-11T21:06:56.143 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:06:56.143 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:06:56.143 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:06:56.143 INFO:teuthology.orchestra.run.smithi023.stdout: "pool": 2, 2024-03-11T21:06:56.144 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_name": ".rgw.root", 2024-03-11T21:06:56.144 INFO:teuthology.orchestra.run.smithi023.stdout: "create_time": "2024-03-11T20:50:26.421486+0000", 2024-03-11T21:06:56.144 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-11T21:06:56.144 INFO:teuthology.orchestra.run.smithi023.stdout: "flags_names": "hashpspool", 2024-03-11T21:06:56.144 INFO:teuthology.orchestra.run.smithi023.stdout: "type": 1, 2024-03-11T21:06:56.144 INFO:teuthology.orchestra.run.smithi023.stdout: "size": 3, 2024-03-11T21:06:56.144 INFO:teuthology.orchestra.run.smithi023.stdout: "min_size": 2, 2024-03-11T21:06:56.144 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_rule": 0, 2024-03-11T21:06:56.144 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_count": 0, 2024-03-11T21:06:56.144 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_target": 0, 2024-03-11T21:06:56.144 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T21:06:56.144 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T21:06:56.144 INFO:teuthology.orchestra.run.smithi023.stdout: "object_hash": 2, 2024-03-11T21:06:56.144 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_autoscale_mode": "off", 2024-03-11T21:06:56.145 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num": 32, 2024-03-11T21:06:56.145 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_placement_num": 32, 2024-03-11T21:06:56.145 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_placement_num_target": 32, 2024-03-11T21:06:56.145 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num_target": 32, 2024-03-11T21:06:56.145 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num_pending": 32, 2024-03-11T21:06:56.145 INFO:teuthology.orchestra.run.smithi023.stdout: "last_pg_merge_meta": { 2024-03-11T21:06:56.145 INFO:teuthology.orchestra.run.smithi023.stdout: "source_pgid": "0.0", 2024-03-11T21:06:56.145 INFO:teuthology.orchestra.run.smithi023.stdout: "ready_epoch": 0, 2024-03-11T21:06:56.145 INFO:teuthology.orchestra.run.smithi023.stdout: "last_epoch_started": 0, 2024-03-11T21:06:56.145 INFO:teuthology.orchestra.run.smithi023.stdout: "last_epoch_clean": 0, 2024-03-11T21:06:56.145 INFO:teuthology.orchestra.run.smithi023.stdout: "source_version": "0'0", 2024-03-11T21:06:56.145 INFO:teuthology.orchestra.run.smithi023.stdout: "target_version": "0'0" 2024-03-11T21:06:56.145 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:06:56.145 INFO:teuthology.orchestra.run.smithi023.stdout: "last_change": "78", 2024-03-11T21:06:56.145 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend": "0", 2024-03-11T21:06:56.146 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T21:06:56.146 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T21:06:56.146 INFO:teuthology.orchestra.run.smithi023.stdout: "auid": 0, 2024-03-11T21:06:56.146 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_mode": "selfmanaged", 2024-03-11T21:06:56.146 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_seq": 0, 2024-03-11T21:06:56.146 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_epoch": 0, 2024-03-11T21:06:56.146 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_snaps": [], 2024-03-11T21:06:56.146 INFO:teuthology.orchestra.run.smithi023.stdout: "removed_snaps": "[]", 2024-03-11T21:06:56.146 INFO:teuthology.orchestra.run.smithi023.stdout: "quota_max_bytes": 0, 2024-03-11T21:06:56.146 INFO:teuthology.orchestra.run.smithi023.stdout: "quota_max_objects": 0, 2024-03-11T21:06:56.146 INFO:teuthology.orchestra.run.smithi023.stdout: "tiers": [], 2024-03-11T21:06:56.146 INFO:teuthology.orchestra.run.smithi023.stdout: "tier_of": -1, 2024-03-11T21:06:56.146 INFO:teuthology.orchestra.run.smithi023.stdout: "read_tier": -1, 2024-03-11T21:06:56.146 INFO:teuthology.orchestra.run.smithi023.stdout: "write_tier": -1, 2024-03-11T21:06:56.147 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_mode": "none", 2024-03-11T21:06:56.147 INFO:teuthology.orchestra.run.smithi023.stdout: "target_max_bytes": 0, 2024-03-11T21:06:56.147 INFO:teuthology.orchestra.run.smithi023.stdout: "target_max_objects": 0, 2024-03-11T21:06:56.147 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T21:06:56.147 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T21:06:56.147 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T21:06:56.147 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_min_flush_age": 0, 2024-03-11T21:06:56.147 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_min_evict_age": 0, 2024-03-11T21:06:56.147 INFO:teuthology.orchestra.run.smithi023.stdout: "erasure_code_profile": "", 2024-03-11T21:06:56.147 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_params": { 2024-03-11T21:06:56.147 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "none" 2024-03-11T21:06:56.147 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:06:56.147 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_period": 0, 2024-03-11T21:06:56.147 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_count": 0, 2024-03-11T21:06:56.147 INFO:teuthology.orchestra.run.smithi023.stdout: "use_gmt_hitset": true, 2024-03-11T21:06:56.147 INFO:teuthology.orchestra.run.smithi023.stdout: "min_read_recency_for_promote": 0, 2024-03-11T21:06:56.148 INFO:teuthology.orchestra.run.smithi023.stdout: "min_write_recency_for_promote": 0, 2024-03-11T21:06:56.148 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T21:06:56.148 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_search_last_n": 0, 2024-03-11T21:06:56.148 INFO:teuthology.orchestra.run.smithi023.stdout: "grade_table": [], 2024-03-11T21:06:56.148 INFO:teuthology.orchestra.run.smithi023.stdout: "stripe_width": 0, 2024-03-11T21:06:56.148 INFO:teuthology.orchestra.run.smithi023.stdout: "expected_num_objects": 0, 2024-03-11T21:06:56.148 INFO:teuthology.orchestra.run.smithi023.stdout: "fast_read": false, 2024-03-11T21:06:56.148 INFO:teuthology.orchestra.run.smithi023.stdout: "options": {}, 2024-03-11T21:06:56.148 INFO:teuthology.orchestra.run.smithi023.stdout: "application_metadata": { 2024-03-11T21:06:56.148 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw": {} 2024-03-11T21:06:56.148 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:06:56.148 INFO:teuthology.orchestra.run.smithi023.stdout: "read_balance": { 2024-03-11T21:06:56.148 INFO:teuthology.orchestra.run.smithi023.stdout: "score_type": "Fair distribution", 2024-03-11T21:06:56.148 INFO:teuthology.orchestra.run.smithi023.stdout: "score_acting": 1.3200000524520874, 2024-03-11T21:06:56.149 INFO:teuthology.orchestra.run.smithi023.stdout: "score_stable": 1.3200000524520874, 2024-03-11T21:06:56.149 INFO:teuthology.orchestra.run.smithi023.stdout: "optimal_score": 1.1399999856948853, 2024-03-11T21:06:56.149 INFO:teuthology.orchestra.run.smithi023.stdout: "raw_score_acting": 1.5, 2024-03-11T21:06:56.149 INFO:teuthology.orchestra.run.smithi023.stdout: "raw_score_stable": 1.5, 2024-03-11T21:06:56.149 INFO:teuthology.orchestra.run.smithi023.stdout: "primary_affinity_weighted": 0.875, 2024-03-11T21:06:56.149 INFO:teuthology.orchestra.run.smithi023.stdout: "average_primary_affinity": 1, 2024-03-11T21:06:56.149 INFO:teuthology.orchestra.run.smithi023.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-03-11T21:06:56.149 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:06:56.149 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:06:56.149 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:06:56.149 INFO:teuthology.orchestra.run.smithi023.stdout: "pool": 3, 2024-03-11T21:06:56.149 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_name": "default.rgw.log", 2024-03-11T21:06:56.149 INFO:teuthology.orchestra.run.smithi023.stdout: "create_time": "2024-03-11T20:50:28.592615+0000", 2024-03-11T21:06:56.149 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-11T21:06:56.149 INFO:teuthology.orchestra.run.smithi023.stdout: "flags_names": "hashpspool", 2024-03-11T21:06:56.150 INFO:teuthology.orchestra.run.smithi023.stdout: "type": 1, 2024-03-11T21:06:56.150 INFO:teuthology.orchestra.run.smithi023.stdout: "size": 3, 2024-03-11T21:06:56.150 INFO:teuthology.orchestra.run.smithi023.stdout: "min_size": 2, 2024-03-11T21:06:56.150 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_rule": 0, 2024-03-11T21:06:56.150 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_count": 0, 2024-03-11T21:06:56.150 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_target": 0, 2024-03-11T21:06:56.150 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T21:06:56.150 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T21:06:56.150 INFO:teuthology.orchestra.run.smithi023.stdout: "object_hash": 2, 2024-03-11T21:06:56.150 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_autoscale_mode": "off", 2024-03-11T21:06:56.150 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num": 32, 2024-03-11T21:06:56.150 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_placement_num": 32, 2024-03-11T21:06:56.150 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_placement_num_target": 32, 2024-03-11T21:06:56.150 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num_target": 32, 2024-03-11T21:06:56.151 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num_pending": 32, 2024-03-11T21:06:56.151 INFO:teuthology.orchestra.run.smithi023.stdout: "last_pg_merge_meta": { 2024-03-11T21:06:56.151 INFO:teuthology.orchestra.run.smithi023.stdout: "source_pgid": "0.0", 2024-03-11T21:06:56.151 INFO:teuthology.orchestra.run.smithi023.stdout: "ready_epoch": 0, 2024-03-11T21:06:56.151 INFO:teuthology.orchestra.run.smithi023.stdout: "last_epoch_started": 0, 2024-03-11T21:06:56.151 INFO:teuthology.orchestra.run.smithi023.stdout: "last_epoch_clean": 0, 2024-03-11T21:06:56.151 INFO:teuthology.orchestra.run.smithi023.stdout: "source_version": "0'0", 2024-03-11T21:06:56.151 INFO:teuthology.orchestra.run.smithi023.stdout: "target_version": "0'0" 2024-03-11T21:06:56.151 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:06:56.151 INFO:teuthology.orchestra.run.smithi023.stdout: "last_change": "80", 2024-03-11T21:06:56.151 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend": "0", 2024-03-11T21:06:56.151 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T21:06:56.151 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T21:06:56.152 INFO:teuthology.orchestra.run.smithi023.stdout: "auid": 0, 2024-03-11T21:06:56.152 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_mode": "selfmanaged", 2024-03-11T21:06:56.152 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_seq": 0, 2024-03-11T21:06:56.152 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_epoch": 0, 2024-03-11T21:06:56.152 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_snaps": [], 2024-03-11T21:06:56.152 INFO:teuthology.orchestra.run.smithi023.stdout: "removed_snaps": "[]", 2024-03-11T21:06:56.152 INFO:teuthology.orchestra.run.smithi023.stdout: "quota_max_bytes": 0, 2024-03-11T21:06:56.152 INFO:teuthology.orchestra.run.smithi023.stdout: "quota_max_objects": 0, 2024-03-11T21:06:56.152 INFO:teuthology.orchestra.run.smithi023.stdout: "tiers": [], 2024-03-11T21:06:56.152 INFO:teuthology.orchestra.run.smithi023.stdout: "tier_of": -1, 2024-03-11T21:06:56.152 INFO:teuthology.orchestra.run.smithi023.stdout: "read_tier": -1, 2024-03-11T21:06:56.152 INFO:teuthology.orchestra.run.smithi023.stdout: "write_tier": -1, 2024-03-11T21:06:56.152 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_mode": "none", 2024-03-11T21:06:56.152 INFO:teuthology.orchestra.run.smithi023.stdout: "target_max_bytes": 0, 2024-03-11T21:06:56.152 INFO:teuthology.orchestra.run.smithi023.stdout: "target_max_objects": 0, 2024-03-11T21:06:56.152 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T21:06:56.153 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T21:06:56.153 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T21:06:56.153 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_min_flush_age": 0, 2024-03-11T21:06:56.153 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_min_evict_age": 0, 2024-03-11T21:06:56.153 INFO:teuthology.orchestra.run.smithi023.stdout: "erasure_code_profile": "", 2024-03-11T21:06:56.153 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_params": { 2024-03-11T21:06:56.153 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "none" 2024-03-11T21:06:56.153 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:06:56.153 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_period": 0, 2024-03-11T21:06:56.153 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_count": 0, 2024-03-11T21:06:56.153 INFO:teuthology.orchestra.run.smithi023.stdout: "use_gmt_hitset": true, 2024-03-11T21:06:56.154 INFO:teuthology.orchestra.run.smithi023.stdout: "min_read_recency_for_promote": 0, 2024-03-11T21:06:56.154 INFO:teuthology.orchestra.run.smithi023.stdout: "min_write_recency_for_promote": 0, 2024-03-11T21:06:56.154 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T21:06:56.154 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_search_last_n": 0, 2024-03-11T21:06:56.154 INFO:teuthology.orchestra.run.smithi023.stdout: "grade_table": [], 2024-03-11T21:06:56.154 INFO:teuthology.orchestra.run.smithi023.stdout: "stripe_width": 0, 2024-03-11T21:06:56.154 INFO:teuthology.orchestra.run.smithi023.stdout: "expected_num_objects": 0, 2024-03-11T21:06:56.154 INFO:teuthology.orchestra.run.smithi023.stdout: "fast_read": false, 2024-03-11T21:06:56.154 INFO:teuthology.orchestra.run.smithi023.stdout: "options": {}, 2024-03-11T21:06:56.155 INFO:teuthology.orchestra.run.smithi023.stdout: "application_metadata": { 2024-03-11T21:06:56.155 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw": {} 2024-03-11T21:06:56.155 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:06:56.155 INFO:teuthology.orchestra.run.smithi023.stdout: "read_balance": { 2024-03-11T21:06:56.155 INFO:teuthology.orchestra.run.smithi023.stdout: "score_type": "Fair distribution", 2024-03-11T21:06:56.155 INFO:teuthology.orchestra.run.smithi023.stdout: "score_acting": 1.5399999618530273, 2024-03-11T21:06:56.155 INFO:teuthology.orchestra.run.smithi023.stdout: "score_stable": 1.5399999618530273, 2024-03-11T21:06:56.155 INFO:teuthology.orchestra.run.smithi023.stdout: "optimal_score": 1.1399999856948853, 2024-03-11T21:06:56.155 INFO:teuthology.orchestra.run.smithi023.stdout: "raw_score_acting": 1.75, 2024-03-11T21:06:56.155 INFO:teuthology.orchestra.run.smithi023.stdout: "raw_score_stable": 1.75, 2024-03-11T21:06:56.155 INFO:teuthology.orchestra.run.smithi023.stdout: "primary_affinity_weighted": 0.875, 2024-03-11T21:06:56.155 INFO:teuthology.orchestra.run.smithi023.stdout: "average_primary_affinity": 1, 2024-03-11T21:06:56.156 INFO:teuthology.orchestra.run.smithi023.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-03-11T21:06:56.156 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:06:56.156 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:06:56.156 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:06:56.156 INFO:teuthology.orchestra.run.smithi023.stdout: "pool": 4, 2024-03-11T21:06:56.156 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_name": "default.rgw.control", 2024-03-11T21:06:56.156 INFO:teuthology.orchestra.run.smithi023.stdout: "create_time": "2024-03-11T20:50:30.536371+0000", 2024-03-11T21:06:56.156 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-11T21:06:56.156 INFO:teuthology.orchestra.run.smithi023.stdout: "flags_names": "hashpspool", 2024-03-11T21:06:56.156 INFO:teuthology.orchestra.run.smithi023.stdout: "type": 1, 2024-03-11T21:06:56.156 INFO:teuthology.orchestra.run.smithi023.stdout: "size": 3, 2024-03-11T21:06:56.156 INFO:teuthology.orchestra.run.smithi023.stdout: "min_size": 2, 2024-03-11T21:06:56.157 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_rule": 0, 2024-03-11T21:06:56.157 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_count": 0, 2024-03-11T21:06:56.157 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_target": 0, 2024-03-11T21:06:56.157 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T21:06:56.157 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T21:06:56.157 INFO:teuthology.orchestra.run.smithi023.stdout: "object_hash": 2, 2024-03-11T21:06:56.157 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_autoscale_mode": "off", 2024-03-11T21:06:56.157 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num": 32, 2024-03-11T21:06:56.157 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_placement_num": 32, 2024-03-11T21:06:56.157 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_placement_num_target": 32, 2024-03-11T21:06:56.157 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num_target": 32, 2024-03-11T21:06:56.157 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num_pending": 32, 2024-03-11T21:06:56.157 INFO:teuthology.orchestra.run.smithi023.stdout: "last_pg_merge_meta": { 2024-03-11T21:06:56.158 INFO:teuthology.orchestra.run.smithi023.stdout: "source_pgid": "0.0", 2024-03-11T21:06:56.158 INFO:teuthology.orchestra.run.smithi023.stdout: "ready_epoch": 0, 2024-03-11T21:06:56.158 INFO:teuthology.orchestra.run.smithi023.stdout: "last_epoch_started": 0, 2024-03-11T21:06:56.158 INFO:teuthology.orchestra.run.smithi023.stdout: "last_epoch_clean": 0, 2024-03-11T21:06:56.158 INFO:teuthology.orchestra.run.smithi023.stdout: "source_version": "0'0", 2024-03-11T21:06:56.158 INFO:teuthology.orchestra.run.smithi023.stdout: "target_version": "0'0" 2024-03-11T21:06:56.158 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:06:56.158 INFO:teuthology.orchestra.run.smithi023.stdout: "last_change": "82", 2024-03-11T21:06:56.158 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend": "0", 2024-03-11T21:06:56.158 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T21:06:56.158 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T21:06:56.158 INFO:teuthology.orchestra.run.smithi023.stdout: "auid": 0, 2024-03-11T21:06:56.159 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_mode": "selfmanaged", 2024-03-11T21:06:56.159 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_seq": 0, 2024-03-11T21:06:56.159 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_epoch": 0, 2024-03-11T21:06:56.159 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_snaps": [], 2024-03-11T21:06:56.159 INFO:teuthology.orchestra.run.smithi023.stdout: "removed_snaps": "[]", 2024-03-11T21:06:56.159 INFO:teuthology.orchestra.run.smithi023.stdout: "quota_max_bytes": 0, 2024-03-11T21:06:56.159 INFO:teuthology.orchestra.run.smithi023.stdout: "quota_max_objects": 0, 2024-03-11T21:06:56.159 INFO:teuthology.orchestra.run.smithi023.stdout: "tiers": [], 2024-03-11T21:06:56.159 INFO:teuthology.orchestra.run.smithi023.stdout: "tier_of": -1, 2024-03-11T21:06:56.159 INFO:teuthology.orchestra.run.smithi023.stdout: "read_tier": -1, 2024-03-11T21:06:56.159 INFO:teuthology.orchestra.run.smithi023.stdout: "write_tier": -1, 2024-03-11T21:06:56.159 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_mode": "none", 2024-03-11T21:06:56.160 INFO:teuthology.orchestra.run.smithi023.stdout: "target_max_bytes": 0, 2024-03-11T21:06:56.160 INFO:teuthology.orchestra.run.smithi023.stdout: "target_max_objects": 0, 2024-03-11T21:06:56.160 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T21:06:56.160 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T21:06:56.160 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T21:06:56.160 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_min_flush_age": 0, 2024-03-11T21:06:56.160 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_min_evict_age": 0, 2024-03-11T21:06:56.160 INFO:teuthology.orchestra.run.smithi023.stdout: "erasure_code_profile": "", 2024-03-11T21:06:56.160 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_params": { 2024-03-11T21:06:56.160 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "none" 2024-03-11T21:06:56.160 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:06:56.160 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_period": 0, 2024-03-11T21:06:56.161 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_count": 0, 2024-03-11T21:06:56.161 INFO:teuthology.orchestra.run.smithi023.stdout: "use_gmt_hitset": true, 2024-03-11T21:06:56.161 INFO:teuthology.orchestra.run.smithi023.stdout: "min_read_recency_for_promote": 0, 2024-03-11T21:06:56.161 INFO:teuthology.orchestra.run.smithi023.stdout: "min_write_recency_for_promote": 0, 2024-03-11T21:06:56.161 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T21:06:56.161 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_search_last_n": 0, 2024-03-11T21:06:56.161 INFO:teuthology.orchestra.run.smithi023.stdout: "grade_table": [], 2024-03-11T21:06:56.161 INFO:teuthology.orchestra.run.smithi023.stdout: "stripe_width": 0, 2024-03-11T21:06:56.161 INFO:teuthology.orchestra.run.smithi023.stdout: "expected_num_objects": 0, 2024-03-11T21:06:56.161 INFO:teuthology.orchestra.run.smithi023.stdout: "fast_read": false, 2024-03-11T21:06:56.161 INFO:teuthology.orchestra.run.smithi023.stdout: "options": {}, 2024-03-11T21:06:56.161 INFO:teuthology.orchestra.run.smithi023.stdout: "application_metadata": { 2024-03-11T21:06:56.161 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw": {} 2024-03-11T21:06:56.162 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:06:56.162 INFO:teuthology.orchestra.run.smithi023.stdout: "read_balance": { 2024-03-11T21:06:56.162 INFO:teuthology.orchestra.run.smithi023.stdout: "score_type": "Fair distribution", 2024-03-11T21:06:56.162 INFO:teuthology.orchestra.run.smithi023.stdout: "score_acting": 1.9700000286102295, 2024-03-11T21:06:56.162 INFO:teuthology.orchestra.run.smithi023.stdout: "score_stable": 1.9700000286102295, 2024-03-11T21:06:56.162 INFO:teuthology.orchestra.run.smithi023.stdout: "optimal_score": 1.1399999856948853, 2024-03-11T21:06:56.162 INFO:teuthology.orchestra.run.smithi023.stdout: "raw_score_acting": 2.25, 2024-03-11T21:06:56.162 INFO:teuthology.orchestra.run.smithi023.stdout: "raw_score_stable": 2.25, 2024-03-11T21:06:56.162 INFO:teuthology.orchestra.run.smithi023.stdout: "primary_affinity_weighted": 0.875, 2024-03-11T21:06:56.162 INFO:teuthology.orchestra.run.smithi023.stdout: "average_primary_affinity": 1, 2024-03-11T21:06:56.162 INFO:teuthology.orchestra.run.smithi023.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-03-11T21:06:56.162 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:06:56.163 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:06:56.163 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:06:56.163 INFO:teuthology.orchestra.run.smithi023.stdout: "pool": 5, 2024-03-11T21:06:56.163 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_name": "datapool", 2024-03-11T21:06:56.163 INFO:teuthology.orchestra.run.smithi023.stdout: "create_time": "2024-03-11T20:50:31.135845+0000", 2024-03-11T21:06:56.163 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 8193, 2024-03-11T21:06:56.163 INFO:teuthology.orchestra.run.smithi023.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-03-11T21:06:56.163 INFO:teuthology.orchestra.run.smithi023.stdout: "type": 1, 2024-03-11T21:06:56.163 INFO:teuthology.orchestra.run.smithi023.stdout: "size": 3, 2024-03-11T21:06:56.163 INFO:teuthology.orchestra.run.smithi023.stdout: "min_size": 2, 2024-03-11T21:06:56.163 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_rule": 0, 2024-03-11T21:06:56.163 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_count": 0, 2024-03-11T21:06:56.163 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_target": 0, 2024-03-11T21:06:56.164 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T21:06:56.164 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T21:06:56.164 INFO:teuthology.orchestra.run.smithi023.stdout: "object_hash": 2, 2024-03-11T21:06:56.164 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_autoscale_mode": "off", 2024-03-11T21:06:56.164 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num": 3, 2024-03-11T21:06:56.164 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_placement_num": 3, 2024-03-11T21:06:56.164 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_placement_num_target": 3, 2024-03-11T21:06:56.164 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num_target": 3, 2024-03-11T21:06:56.164 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num_pending": 3, 2024-03-11T21:06:56.164 INFO:teuthology.orchestra.run.smithi023.stdout: "last_pg_merge_meta": { 2024-03-11T21:06:56.164 INFO:teuthology.orchestra.run.smithi023.stdout: "source_pgid": "0.0", 2024-03-11T21:06:56.164 INFO:teuthology.orchestra.run.smithi023.stdout: "ready_epoch": 0, 2024-03-11T21:06:56.165 INFO:teuthology.orchestra.run.smithi023.stdout: "last_epoch_started": 0, 2024-03-11T21:06:56.165 INFO:teuthology.orchestra.run.smithi023.stdout: "last_epoch_clean": 0, 2024-03-11T21:06:56.165 INFO:teuthology.orchestra.run.smithi023.stdout: "source_version": "0'0", 2024-03-11T21:06:56.165 INFO:teuthology.orchestra.run.smithi023.stdout: "target_version": "0'0" 2024-03-11T21:06:56.165 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:06:56.165 INFO:teuthology.orchestra.run.smithi023.stdout: "last_change": "87", 2024-03-11T21:06:56.165 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend": "0", 2024-03-11T21:06:56.165 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T21:06:56.165 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T21:06:56.165 INFO:teuthology.orchestra.run.smithi023.stdout: "auid": 0, 2024-03-11T21:06:56.165 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_mode": "selfmanaged", 2024-03-11T21:06:56.165 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_seq": 2, 2024-03-11T21:06:56.165 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_epoch": 87, 2024-03-11T21:06:56.166 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_snaps": [], 2024-03-11T21:06:56.166 INFO:teuthology.orchestra.run.smithi023.stdout: "removed_snaps": "[]", 2024-03-11T21:06:56.166 INFO:teuthology.orchestra.run.smithi023.stdout: "quota_max_bytes": 0, 2024-03-11T21:06:56.166 INFO:teuthology.orchestra.run.smithi023.stdout: "quota_max_objects": 0, 2024-03-11T21:06:56.166 INFO:teuthology.orchestra.run.smithi023.stdout: "tiers": [], 2024-03-11T21:06:56.166 INFO:teuthology.orchestra.run.smithi023.stdout: "tier_of": -1, 2024-03-11T21:06:56.166 INFO:teuthology.orchestra.run.smithi023.stdout: "read_tier": -1, 2024-03-11T21:06:56.166 INFO:teuthology.orchestra.run.smithi023.stdout: "write_tier": -1, 2024-03-11T21:06:56.166 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_mode": "none", 2024-03-11T21:06:56.166 INFO:teuthology.orchestra.run.smithi023.stdout: "target_max_bytes": 0, 2024-03-11T21:06:56.166 INFO:teuthology.orchestra.run.smithi023.stdout: "target_max_objects": 0, 2024-03-11T21:06:56.166 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T21:06:56.167 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T21:06:56.167 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T21:06:56.167 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_min_flush_age": 0, 2024-03-11T21:06:56.167 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_min_evict_age": 0, 2024-03-11T21:06:56.167 INFO:teuthology.orchestra.run.smithi023.stdout: "erasure_code_profile": "", 2024-03-11T21:06:56.167 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_params": { 2024-03-11T21:06:56.167 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "none" 2024-03-11T21:06:56.167 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:06:56.167 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_period": 0, 2024-03-11T21:06:56.167 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_count": 0, 2024-03-11T21:06:56.167 INFO:teuthology.orchestra.run.smithi023.stdout: "use_gmt_hitset": true, 2024-03-11T21:06:56.167 INFO:teuthology.orchestra.run.smithi023.stdout: "min_read_recency_for_promote": 0, 2024-03-11T21:06:56.167 INFO:teuthology.orchestra.run.smithi023.stdout: "min_write_recency_for_promote": 0, 2024-03-11T21:06:56.168 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T21:06:56.168 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_search_last_n": 0, 2024-03-11T21:06:56.168 INFO:teuthology.orchestra.run.smithi023.stdout: "grade_table": [], 2024-03-11T21:06:56.168 INFO:teuthology.orchestra.run.smithi023.stdout: "stripe_width": 0, 2024-03-11T21:06:56.168 INFO:teuthology.orchestra.run.smithi023.stdout: "expected_num_objects": 0, 2024-03-11T21:06:56.168 INFO:teuthology.orchestra.run.smithi023.stdout: "fast_read": false, 2024-03-11T21:06:56.168 INFO:teuthology.orchestra.run.smithi023.stdout: "options": {}, 2024-03-11T21:06:56.168 INFO:teuthology.orchestra.run.smithi023.stdout: "application_metadata": { 2024-03-11T21:06:56.168 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd": {} 2024-03-11T21:06:56.168 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:06:56.168 INFO:teuthology.orchestra.run.smithi023.stdout: "read_balance": { 2024-03-11T21:06:56.168 INFO:teuthology.orchestra.run.smithi023.stdout: "score_type": "Fair distribution", 2024-03-11T21:06:56.168 INFO:teuthology.orchestra.run.smithi023.stdout: "score_acting": 4.6700000762939453, 2024-03-11T21:06:56.169 INFO:teuthology.orchestra.run.smithi023.stdout: "score_stable": 4.6700000762939453, 2024-03-11T21:06:56.169 INFO:teuthology.orchestra.run.smithi023.stdout: "optimal_score": 1, 2024-03-11T21:06:56.169 INFO:teuthology.orchestra.run.smithi023.stdout: "raw_score_acting": 4.6700000762939453, 2024-03-11T21:06:56.169 INFO:teuthology.orchestra.run.smithi023.stdout: "raw_score_stable": 4.6700000762939453, 2024-03-11T21:06:56.169 INFO:teuthology.orchestra.run.smithi023.stdout: "primary_affinity_weighted": 0.875, 2024-03-11T21:06:56.169 INFO:teuthology.orchestra.run.smithi023.stdout: "average_primary_affinity": 1, 2024-03-11T21:06:56.169 INFO:teuthology.orchestra.run.smithi023.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-03-11T21:06:56.169 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:06:56.169 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:06:56.169 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:06:56.169 INFO:teuthology.orchestra.run.smithi023.stdout: "pool": 6, 2024-03-11T21:06:56.169 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_name": "default.rgw.meta", 2024-03-11T21:06:56.170 INFO:teuthology.orchestra.run.smithi023.stdout: "create_time": "2024-03-11T20:50:33.233163+0000", 2024-03-11T21:06:56.170 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-11T21:06:56.170 INFO:teuthology.orchestra.run.smithi023.stdout: "flags_names": "hashpspool", 2024-03-11T21:06:56.170 INFO:teuthology.orchestra.run.smithi023.stdout: "type": 1, 2024-03-11T21:06:56.170 INFO:teuthology.orchestra.run.smithi023.stdout: "size": 3, 2024-03-11T21:06:56.170 INFO:teuthology.orchestra.run.smithi023.stdout: "min_size": 2, 2024-03-11T21:06:56.170 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_rule": 0, 2024-03-11T21:06:56.170 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_count": 0, 2024-03-11T21:06:56.170 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_target": 0, 2024-03-11T21:06:56.170 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T21:06:56.170 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T21:06:56.170 INFO:teuthology.orchestra.run.smithi023.stdout: "object_hash": 2, 2024-03-11T21:06:56.171 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_autoscale_mode": "off", 2024-03-11T21:06:56.171 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num": 32, 2024-03-11T21:06:56.171 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_placement_num": 32, 2024-03-11T21:06:56.171 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_placement_num_target": 32, 2024-03-11T21:06:56.171 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num_target": 32, 2024-03-11T21:06:56.171 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num_pending": 32, 2024-03-11T21:06:56.171 INFO:teuthology.orchestra.run.smithi023.stdout: "last_pg_merge_meta": { 2024-03-11T21:06:56.171 INFO:teuthology.orchestra.run.smithi023.stdout: "source_pgid": "0.0", 2024-03-11T21:06:56.171 INFO:teuthology.orchestra.run.smithi023.stdout: "ready_epoch": 0, 2024-03-11T21:06:56.171 INFO:teuthology.orchestra.run.smithi023.stdout: "last_epoch_started": 0, 2024-03-11T21:06:56.171 INFO:teuthology.orchestra.run.smithi023.stdout: "last_epoch_clean": 0, 2024-03-11T21:06:56.171 INFO:teuthology.orchestra.run.smithi023.stdout: "source_version": "0'0", 2024-03-11T21:06:56.171 INFO:teuthology.orchestra.run.smithi023.stdout: "target_version": "0'0" 2024-03-11T21:06:56.172 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:06:56.172 INFO:teuthology.orchestra.run.smithi023.stdout: "last_change": "84", 2024-03-11T21:06:56.172 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend": "0", 2024-03-11T21:06:56.172 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T21:06:56.172 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T21:06:56.172 INFO:teuthology.orchestra.run.smithi023.stdout: "auid": 0, 2024-03-11T21:06:56.172 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_mode": "selfmanaged", 2024-03-11T21:06:56.172 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_seq": 0, 2024-03-11T21:06:56.172 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_epoch": 0, 2024-03-11T21:06:56.172 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_snaps": [], 2024-03-11T21:06:56.172 INFO:teuthology.orchestra.run.smithi023.stdout: "removed_snaps": "[]", 2024-03-11T21:06:56.172 INFO:teuthology.orchestra.run.smithi023.stdout: "quota_max_bytes": 0, 2024-03-11T21:06:56.172 INFO:teuthology.orchestra.run.smithi023.stdout: "quota_max_objects": 0, 2024-03-11T21:06:56.173 INFO:teuthology.orchestra.run.smithi023.stdout: "tiers": [], 2024-03-11T21:06:56.173 INFO:teuthology.orchestra.run.smithi023.stdout: "tier_of": -1, 2024-03-11T21:06:56.173 INFO:teuthology.orchestra.run.smithi023.stdout: "read_tier": -1, 2024-03-11T21:06:56.173 INFO:teuthology.orchestra.run.smithi023.stdout: "write_tier": -1, 2024-03-11T21:06:56.173 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_mode": "none", 2024-03-11T21:06:56.173 INFO:teuthology.orchestra.run.smithi023.stdout: "target_max_bytes": 0, 2024-03-11T21:06:56.173 INFO:teuthology.orchestra.run.smithi023.stdout: "target_max_objects": 0, 2024-03-11T21:06:56.173 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T21:06:56.173 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T21:06:56.173 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T21:06:56.173 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_min_flush_age": 0, 2024-03-11T21:06:56.174 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_min_evict_age": 0, 2024-03-11T21:06:56.174 INFO:teuthology.orchestra.run.smithi023.stdout: "erasure_code_profile": "", 2024-03-11T21:06:56.174 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_params": { 2024-03-11T21:06:56.174 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "none" 2024-03-11T21:06:56.174 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:06:56.174 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_period": 0, 2024-03-11T21:06:56.174 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_count": 0, 2024-03-11T21:06:56.174 INFO:teuthology.orchestra.run.smithi023.stdout: "use_gmt_hitset": true, 2024-03-11T21:06:56.174 INFO:teuthology.orchestra.run.smithi023.stdout: "min_read_recency_for_promote": 0, 2024-03-11T21:06:56.174 INFO:teuthology.orchestra.run.smithi023.stdout: "min_write_recency_for_promote": 0, 2024-03-11T21:06:56.174 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T21:06:56.175 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_search_last_n": 0, 2024-03-11T21:06:56.175 INFO:teuthology.orchestra.run.smithi023.stdout: "grade_table": [], 2024-03-11T21:06:56.175 INFO:teuthology.orchestra.run.smithi023.stdout: "stripe_width": 0, 2024-03-11T21:06:56.175 INFO:teuthology.orchestra.run.smithi023.stdout: "expected_num_objects": 0, 2024-03-11T21:06:56.175 INFO:teuthology.orchestra.run.smithi023.stdout: "fast_read": false, 2024-03-11T21:06:56.175 INFO:teuthology.orchestra.run.smithi023.stdout: "options": { 2024-03-11T21:06:56.175 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_autoscale_bias": 4 2024-03-11T21:06:56.175 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:06:56.175 INFO:teuthology.orchestra.run.smithi023.stdout: "application_metadata": { 2024-03-11T21:06:56.175 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw": {} 2024-03-11T21:06:56.175 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:06:56.175 INFO:teuthology.orchestra.run.smithi023.stdout: "read_balance": { 2024-03-11T21:06:56.176 INFO:teuthology.orchestra.run.smithi023.stdout: "score_type": "Fair distribution", 2024-03-11T21:06:56.176 INFO:teuthology.orchestra.run.smithi023.stdout: "score_acting": 1.75, 2024-03-11T21:06:56.176 INFO:teuthology.orchestra.run.smithi023.stdout: "score_stable": 1.75, 2024-03-11T21:06:56.176 INFO:teuthology.orchestra.run.smithi023.stdout: "optimal_score": 1.1399999856948853, 2024-03-11T21:06:56.176 INFO:teuthology.orchestra.run.smithi023.stdout: "raw_score_acting": 2, 2024-03-11T21:06:56.176 INFO:teuthology.orchestra.run.smithi023.stdout: "raw_score_stable": 2, 2024-03-11T21:06:56.176 INFO:teuthology.orchestra.run.smithi023.stdout: "primary_affinity_weighted": 0.875, 2024-03-11T21:06:56.176 INFO:teuthology.orchestra.run.smithi023.stdout: "average_primary_affinity": 1, 2024-03-11T21:06:56.176 INFO:teuthology.orchestra.run.smithi023.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-03-11T21:06:56.176 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:06:56.176 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:06:56.176 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:06:56.176 INFO:teuthology.orchestra.run.smithi023.stdout: "pool": 9, 2024-03-11T21:06:56.177 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_name": "unique_pool_2", 2024-03-11T21:06:56.177 INFO:teuthology.orchestra.run.smithi023.stdout: "create_time": "2024-03-11T21:02:46.162698+0000", 2024-03-11T21:06:56.177 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-11T21:06:56.177 INFO:teuthology.orchestra.run.smithi023.stdout: "flags_names": "hashpspool", 2024-03-11T21:06:56.177 INFO:teuthology.orchestra.run.smithi023.stdout: "type": 1, 2024-03-11T21:06:56.177 INFO:teuthology.orchestra.run.smithi023.stdout: "size": 3, 2024-03-11T21:06:56.177 INFO:teuthology.orchestra.run.smithi023.stdout: "min_size": 2, 2024-03-11T21:06:56.177 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_rule": 0, 2024-03-11T21:06:56.177 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_count": 0, 2024-03-11T21:06:56.177 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_target": 0, 2024-03-11T21:06:56.177 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T21:06:56.177 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T21:06:56.178 INFO:teuthology.orchestra.run.smithi023.stdout: "object_hash": 2, 2024-03-11T21:06:56.178 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_autoscale_mode": "off", 2024-03-11T21:06:56.178 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num": 16, 2024-03-11T21:06:56.178 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_placement_num": 16, 2024-03-11T21:06:56.178 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_placement_num_target": 16, 2024-03-11T21:06:56.178 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num_target": 16, 2024-03-11T21:06:56.178 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num_pending": 16, 2024-03-11T21:06:56.178 INFO:teuthology.orchestra.run.smithi023.stdout: "last_pg_merge_meta": { 2024-03-11T21:06:56.178 INFO:teuthology.orchestra.run.smithi023.stdout: "source_pgid": "0.0", 2024-03-11T21:06:56.178 INFO:teuthology.orchestra.run.smithi023.stdout: "ready_epoch": 0, 2024-03-11T21:06:56.178 INFO:teuthology.orchestra.run.smithi023.stdout: "last_epoch_started": 0, 2024-03-11T21:06:56.178 INFO:teuthology.orchestra.run.smithi023.stdout: "last_epoch_clean": 0, 2024-03-11T21:06:56.179 INFO:teuthology.orchestra.run.smithi023.stdout: "source_version": "0'0", 2024-03-11T21:06:56.179 INFO:teuthology.orchestra.run.smithi023.stdout: "target_version": "0'0" 2024-03-11T21:06:56.179 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:06:56.179 INFO:teuthology.orchestra.run.smithi023.stdout: "last_change": "256", 2024-03-11T21:06:56.179 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend": "0", 2024-03-11T21:06:56.179 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T21:06:56.179 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T21:06:56.179 INFO:teuthology.orchestra.run.smithi023.stdout: "auid": 0, 2024-03-11T21:06:56.179 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_mode": "selfmanaged", 2024-03-11T21:06:56.179 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_seq": 0, 2024-03-11T21:06:56.179 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_epoch": 0, 2024-03-11T21:06:56.179 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_snaps": [], 2024-03-11T21:06:56.179 INFO:teuthology.orchestra.run.smithi023.stdout: "removed_snaps": "[]", 2024-03-11T21:06:56.180 INFO:teuthology.orchestra.run.smithi023.stdout: "quota_max_bytes": 0, 2024-03-11T21:06:56.180 INFO:teuthology.orchestra.run.smithi023.stdout: "quota_max_objects": 0, 2024-03-11T21:06:56.180 INFO:teuthology.orchestra.run.smithi023.stdout: "tiers": [], 2024-03-11T21:06:56.180 INFO:teuthology.orchestra.run.smithi023.stdout: "tier_of": -1, 2024-03-11T21:06:56.180 INFO:teuthology.orchestra.run.smithi023.stdout: "read_tier": -1, 2024-03-11T21:06:56.180 INFO:teuthology.orchestra.run.smithi023.stdout: "write_tier": -1, 2024-03-11T21:06:56.180 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_mode": "none", 2024-03-11T21:06:56.180 INFO:teuthology.orchestra.run.smithi023.stdout: "target_max_bytes": 0, 2024-03-11T21:06:56.180 INFO:teuthology.orchestra.run.smithi023.stdout: "target_max_objects": 0, 2024-03-11T21:06:56.180 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T21:06:56.180 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T21:06:56.180 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T21:06:56.181 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_min_flush_age": 0, 2024-03-11T21:06:56.181 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_min_evict_age": 0, 2024-03-11T21:06:56.181 INFO:teuthology.orchestra.run.smithi023.stdout: "erasure_code_profile": "", 2024-03-11T21:06:56.181 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_params": { 2024-03-11T21:06:56.181 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "none" 2024-03-11T21:06:56.181 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:06:56.181 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_period": 0, 2024-03-11T21:06:56.181 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_count": 0, 2024-03-11T21:06:56.181 INFO:teuthology.orchestra.run.smithi023.stdout: "use_gmt_hitset": true, 2024-03-11T21:06:56.181 INFO:teuthology.orchestra.run.smithi023.stdout: "min_read_recency_for_promote": 0, 2024-03-11T21:06:56.181 INFO:teuthology.orchestra.run.smithi023.stdout: "min_write_recency_for_promote": 0, 2024-03-11T21:06:56.181 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T21:06:56.182 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_search_last_n": 0, 2024-03-11T21:06:56.182 INFO:teuthology.orchestra.run.smithi023.stdout: "grade_table": [], 2024-03-11T21:06:56.182 INFO:teuthology.orchestra.run.smithi023.stdout: "stripe_width": 0, 2024-03-11T21:06:56.182 INFO:teuthology.orchestra.run.smithi023.stdout: "expected_num_objects": 0, 2024-03-11T21:06:56.182 INFO:teuthology.orchestra.run.smithi023.stdout: "fast_read": false, 2024-03-11T21:06:56.182 INFO:teuthology.orchestra.run.smithi023.stdout: "options": {}, 2024-03-11T21:06:56.182 INFO:teuthology.orchestra.run.smithi023.stdout: "application_metadata": { 2024-03-11T21:06:56.182 INFO:teuthology.orchestra.run.smithi023.stdout: "rados": {} 2024-03-11T21:06:56.182 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:06:56.182 INFO:teuthology.orchestra.run.smithi023.stdout: "read_balance": { 2024-03-11T21:06:56.182 INFO:teuthology.orchestra.run.smithi023.stdout: "score_type": "Fair distribution", 2024-03-11T21:06:56.182 INFO:teuthology.orchestra.run.smithi023.stdout: "score_acting": 1.75, 2024-03-11T21:06:56.182 INFO:teuthology.orchestra.run.smithi023.stdout: "score_stable": 1.75, 2024-03-11T21:06:56.183 INFO:teuthology.orchestra.run.smithi023.stdout: "optimal_score": 1.1399999856948853, 2024-03-11T21:06:56.183 INFO:teuthology.orchestra.run.smithi023.stdout: "raw_score_acting": 2, 2024-03-11T21:06:56.183 INFO:teuthology.orchestra.run.smithi023.stdout: "raw_score_stable": 2, 2024-03-11T21:06:56.183 INFO:teuthology.orchestra.run.smithi023.stdout: "primary_affinity_weighted": 0.875, 2024-03-11T21:06:56.183 INFO:teuthology.orchestra.run.smithi023.stdout: "average_primary_affinity": 1, 2024-03-11T21:06:56.183 INFO:teuthology.orchestra.run.smithi023.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-03-11T21:06:56.183 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:06:56.183 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:06:56.183 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-11T21:06:56.183 INFO:teuthology.orchestra.run.smithi023.stdout: "osds": [ 2024-03-11T21:06:56.183 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:06:56.183 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": 0, 2024-03-11T21:06:56.183 INFO:teuthology.orchestra.run.smithi023.stdout: "uuid": "3df72b4a-8fe4-4265-b4c6-af022e1e23e1", 2024-03-11T21:06:56.184 INFO:teuthology.orchestra.run.smithi023.stdout: "up": 1, 2024-03-11T21:06:56.184 INFO:teuthology.orchestra.run.smithi023.stdout: "in": 1, 2024-03-11T21:06:56.184 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 1, 2024-03-11T21:06:56.184 INFO:teuthology.orchestra.run.smithi023.stdout: "primary_affinity": 1, 2024-03-11T21:06:56.184 INFO:teuthology.orchestra.run.smithi023.stdout: "last_clean_begin": 0, 2024-03-11T21:06:56.184 INFO:teuthology.orchestra.run.smithi023.stdout: "last_clean_end": 0, 2024-03-11T21:06:56.184 INFO:teuthology.orchestra.run.smithi023.stdout: "up_from": 8, 2024-03-11T21:06:56.184 INFO:teuthology.orchestra.run.smithi023.stdout: "up_thru": 297, 2024-03-11T21:06:56.184 INFO:teuthology.orchestra.run.smithi023.stdout: "down_at": 0, 2024-03-11T21:06:56.184 INFO:teuthology.orchestra.run.smithi023.stdout: "lost_at": 0, 2024-03-11T21:06:56.184 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2024-03-11T21:06:56.184 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:06:56.185 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:06:56.185 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:06:56.185 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6802", 2024-03-11T21:06:56.185 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 2154627393 2024-03-11T21:06:56.185 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:06:56.185 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:06:56.185 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:06:56.185 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6803", 2024-03-11T21:06:56.185 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 2154627393 2024-03-11T21:06:56.185 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:06:56.185 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:06:56.185 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:06:56.185 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_addrs": { 2024-03-11T21:06:56.186 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:06:56.186 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:06:56.186 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:06:56.186 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6804", 2024-03-11T21:06:56.186 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 2154627393 2024-03-11T21:06:56.186 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:06:56.186 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:06:56.186 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:06:56.186 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6805", 2024-03-11T21:06:56.186 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 2154627393 2024-03-11T21:06:56.186 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:06:56.186 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:06:56.186 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:06:56.187 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_back_addrs": { 2024-03-11T21:06:56.187 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:06:56.187 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:06:56.187 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:06:56.187 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6808", 2024-03-11T21:06:56.187 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 2154627393 2024-03-11T21:06:56.187 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:06:56.187 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:06:56.187 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:06:56.187 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6809", 2024-03-11T21:06:56.187 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 2154627393 2024-03-11T21:06:56.187 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:06:56.187 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:06:56.188 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:06:56.188 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_front_addrs": { 2024-03-11T21:06:56.188 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:06:56.188 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:06:56.188 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:06:56.188 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6806", 2024-03-11T21:06:56.188 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 2154627393 2024-03-11T21:06:56.188 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:06:56.188 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:06:56.188 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:06:56.188 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6807", 2024-03-11T21:06:56.188 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 2154627393 2024-03-11T21:06:56.188 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:06:56.189 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:06:56.189 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:06:56.189 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6803/2154627393", 2024-03-11T21:06:56.189 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_addr": "172.21.15.23:6805/2154627393", 2024-03-11T21:06:56.189 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_back_addr": "172.21.15.23:6809/2154627393", 2024-03-11T21:06:56.189 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_front_addr": "172.21.15.23:6807/2154627393", 2024-03-11T21:06:56.189 INFO:teuthology.orchestra.run.smithi023.stdout: "state": [ 2024-03-11T21:06:56.189 INFO:teuthology.orchestra.run.smithi023.stdout: "exists", 2024-03-11T21:06:56.189 INFO:teuthology.orchestra.run.smithi023.stdout: "up" 2024-03-11T21:06:56.189 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:06:56.189 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:06:56.189 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:06:56.189 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": 1, 2024-03-11T21:06:56.189 INFO:teuthology.orchestra.run.smithi023.stdout: "uuid": "e8fd7500-9081-41a1-8077-a4d0f0e12a2c", 2024-03-11T21:06:56.190 INFO:teuthology.orchestra.run.smithi023.stdout: "up": 1, 2024-03-11T21:06:56.190 INFO:teuthology.orchestra.run.smithi023.stdout: "in": 1, 2024-03-11T21:06:56.190 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 1, 2024-03-11T21:06:56.190 INFO:teuthology.orchestra.run.smithi023.stdout: "primary_affinity": 1, 2024-03-11T21:06:56.190 INFO:teuthology.orchestra.run.smithi023.stdout: "last_clean_begin": 13, 2024-03-11T21:06:56.190 INFO:teuthology.orchestra.run.smithi023.stdout: "last_clean_end": 173, 2024-03-11T21:06:56.190 INFO:teuthology.orchestra.run.smithi023.stdout: "up_from": 211, 2024-03-11T21:06:56.190 INFO:teuthology.orchestra.run.smithi023.stdout: "up_thru": 285, 2024-03-11T21:06:56.190 INFO:teuthology.orchestra.run.smithi023.stdout: "down_at": 174, 2024-03-11T21:06:56.190 INFO:teuthology.orchestra.run.smithi023.stdout: "lost_at": 0, 2024-03-11T21:06:56.190 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2024-03-11T21:06:56.190 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:06:56.191 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:06:56.191 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:06:56.191 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6810", 2024-03-11T21:06:56.191 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 56585354 2024-03-11T21:06:56.191 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:06:56.191 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:06:56.191 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:06:56.191 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6811", 2024-03-11T21:06:56.191 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 56585354 2024-03-11T21:06:56.191 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:06:56.191 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:06:56.191 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:06:56.192 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_addrs": { 2024-03-11T21:06:56.192 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:06:56.192 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:06:56.192 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:06:56.192 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6812", 2024-03-11T21:06:56.192 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 56585354 2024-03-11T21:06:56.192 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:06:56.192 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:06:56.192 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:06:56.192 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6813", 2024-03-11T21:06:56.192 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 56585354 2024-03-11T21:06:56.192 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:06:56.193 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:06:56.193 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:06:56.193 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_back_addrs": { 2024-03-11T21:06:56.193 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:06:56.193 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:06:56.193 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:06:56.193 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6816", 2024-03-11T21:06:56.193 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 56585354 2024-03-11T21:06:56.193 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:06:56.193 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:06:56.193 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:06:56.193 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6817", 2024-03-11T21:06:56.193 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 56585354 2024-03-11T21:06:56.194 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:06:56.194 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:06:56.194 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:06:56.194 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_front_addrs": { 2024-03-11T21:06:56.194 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:06:56.194 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:06:56.194 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:06:56.194 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6814", 2024-03-11T21:06:56.194 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 56585354 2024-03-11T21:06:56.194 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:06:56.194 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:06:56.194 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:06:56.194 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6815", 2024-03-11T21:06:56.195 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 56585354 2024-03-11T21:06:56.195 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:06:56.195 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:06:56.195 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:06:56.195 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6811/56585354", 2024-03-11T21:06:56.195 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_addr": "172.21.15.23:6813/56585354", 2024-03-11T21:06:56.195 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_back_addr": "172.21.15.23:6817/56585354", 2024-03-11T21:06:56.195 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_front_addr": "172.21.15.23:6815/56585354", 2024-03-11T21:06:56.195 INFO:teuthology.orchestra.run.smithi023.stdout: "state": [ 2024-03-11T21:06:56.195 INFO:teuthology.orchestra.run.smithi023.stdout: "exists", 2024-03-11T21:06:56.195 INFO:teuthology.orchestra.run.smithi023.stdout: "up" 2024-03-11T21:06:56.195 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:06:56.196 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:06:56.196 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:06:56.196 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": 2, 2024-03-11T21:06:56.196 INFO:teuthology.orchestra.run.smithi023.stdout: "uuid": "dd85d52a-df48-4b03-8612-cecfd82b5987", 2024-03-11T21:06:56.196 INFO:teuthology.orchestra.run.smithi023.stdout: "up": 1, 2024-03-11T21:06:56.196 INFO:teuthology.orchestra.run.smithi023.stdout: "in": 1, 2024-03-11T21:06:56.196 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 1, 2024-03-11T21:06:56.196 INFO:teuthology.orchestra.run.smithi023.stdout: "primary_affinity": 1, 2024-03-11T21:06:56.196 INFO:teuthology.orchestra.run.smithi023.stdout: "last_clean_begin": 0, 2024-03-11T21:06:56.196 INFO:teuthology.orchestra.run.smithi023.stdout: "last_clean_end": 0, 2024-03-11T21:06:56.196 INFO:teuthology.orchestra.run.smithi023.stdout: "up_from": 19, 2024-03-11T21:06:56.196 INFO:teuthology.orchestra.run.smithi023.stdout: "up_thru": 285, 2024-03-11T21:06:56.196 INFO:teuthology.orchestra.run.smithi023.stdout: "down_at": 0, 2024-03-11T21:06:56.197 INFO:teuthology.orchestra.run.smithi023.stdout: "lost_at": 0, 2024-03-11T21:06:56.197 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2024-03-11T21:06:56.197 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:06:56.197 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:06:56.197 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:06:56.197 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6818", 2024-03-11T21:06:56.197 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1770748056 2024-03-11T21:06:56.197 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:06:56.197 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:06:56.197 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:06:56.197 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6819", 2024-03-11T21:06:56.197 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1770748056 2024-03-11T21:06:56.198 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:06:56.198 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:06:56.198 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:06:56.198 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_addrs": { 2024-03-11T21:06:56.198 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:06:56.198 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:06:56.198 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:06:56.198 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6820", 2024-03-11T21:06:56.198 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1770748056 2024-03-11T21:06:56.198 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:06:56.198 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:06:56.199 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:06:56.199 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6821", 2024-03-11T21:06:56.199 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1770748056 2024-03-11T21:06:56.199 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:06:56.199 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:06:56.199 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:06:56.199 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_back_addrs": { 2024-03-11T21:06:56.199 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:06:56.199 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:06:56.199 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:06:56.199 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6824", 2024-03-11T21:06:56.199 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1770748056 2024-03-11T21:06:56.199 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:06:56.200 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:06:56.200 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:06:56.200 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6825", 2024-03-11T21:06:56.200 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1770748056 2024-03-11T21:06:56.200 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:06:56.200 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:06:56.200 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:06:56.200 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_front_addrs": { 2024-03-11T21:06:56.200 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:06:56.200 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:06:56.200 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:06:56.200 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6822", 2024-03-11T21:06:56.201 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1770748056 2024-03-11T21:06:56.201 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:06:56.201 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:06:56.201 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:06:56.201 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6823", 2024-03-11T21:06:56.201 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1770748056 2024-03-11T21:06:56.201 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:06:56.201 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:06:56.201 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:06:56.201 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6819/1770748056", 2024-03-11T21:06:56.201 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_addr": "172.21.15.23:6821/1770748056", 2024-03-11T21:06:56.201 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_back_addr": "172.21.15.23:6825/1770748056", 2024-03-11T21:06:56.201 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_front_addr": "172.21.15.23:6823/1770748056", 2024-03-11T21:06:56.202 INFO:teuthology.orchestra.run.smithi023.stdout: "state": [ 2024-03-11T21:06:56.202 INFO:teuthology.orchestra.run.smithi023.stdout: "exists", 2024-03-11T21:06:56.202 INFO:teuthology.orchestra.run.smithi023.stdout: "up" 2024-03-11T21:06:56.202 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:06:56.202 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:06:56.202 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:06:56.202 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": 3, 2024-03-11T21:06:56.202 INFO:teuthology.orchestra.run.smithi023.stdout: "uuid": "e3021106-3779-4b94-9bbf-a7724544110f", 2024-03-11T21:06:56.202 INFO:teuthology.orchestra.run.smithi023.stdout: "up": 1, 2024-03-11T21:06:56.202 INFO:teuthology.orchestra.run.smithi023.stdout: "in": 1, 2024-03-11T21:06:56.202 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 1, 2024-03-11T21:06:56.202 INFO:teuthology.orchestra.run.smithi023.stdout: "primary_affinity": 1, 2024-03-11T21:06:56.202 INFO:teuthology.orchestra.run.smithi023.stdout: "last_clean_begin": 0, 2024-03-11T21:06:56.203 INFO:teuthology.orchestra.run.smithi023.stdout: "last_clean_end": 0, 2024-03-11T21:06:56.203 INFO:teuthology.orchestra.run.smithi023.stdout: "up_from": 28, 2024-03-11T21:06:56.203 INFO:teuthology.orchestra.run.smithi023.stdout: "up_thru": 285, 2024-03-11T21:06:56.203 INFO:teuthology.orchestra.run.smithi023.stdout: "down_at": 0, 2024-03-11T21:06:56.203 INFO:teuthology.orchestra.run.smithi023.stdout: "lost_at": 0, 2024-03-11T21:06:56.203 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2024-03-11T21:06:56.203 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:06:56.203 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:06:56.203 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:06:56.203 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6826", 2024-03-11T21:06:56.203 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 938496814 2024-03-11T21:06:56.203 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:06:56.204 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:06:56.204 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:06:56.204 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6827", 2024-03-11T21:06:56.204 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 938496814 2024-03-11T21:06:56.204 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:06:56.204 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:06:56.204 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:06:56.204 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_addrs": { 2024-03-11T21:06:56.204 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:06:56.204 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:06:56.204 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:06:56.204 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6828", 2024-03-11T21:06:56.204 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 938496814 2024-03-11T21:06:56.205 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:06:56.205 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:06:56.205 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:06:56.205 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6829", 2024-03-11T21:06:56.205 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 938496814 2024-03-11T21:06:56.205 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:06:56.205 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:06:56.205 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:06:56.205 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_back_addrs": { 2024-03-11T21:06:56.205 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:06:56.205 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:06:56.205 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:06:56.205 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6832", 2024-03-11T21:06:56.206 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 938496814 2024-03-11T21:06:56.206 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:06:56.206 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:06:56.206 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:06:56.206 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6833", 2024-03-11T21:06:56.206 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 938496814 2024-03-11T21:06:56.206 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:06:56.206 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:06:56.206 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:06:56.206 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_front_addrs": { 2024-03-11T21:06:56.206 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:06:56.206 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:06:56.206 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:06:56.207 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6830", 2024-03-11T21:06:56.207 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 938496814 2024-03-11T21:06:56.207 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:06:56.207 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:06:56.207 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:06:56.207 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6831", 2024-03-11T21:06:56.207 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 938496814 2024-03-11T21:06:56.207 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:06:56.207 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:06:56.207 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:06:56.207 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6827/938496814", 2024-03-11T21:06:56.207 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_addr": "172.21.15.23:6829/938496814", 2024-03-11T21:06:56.207 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_back_addr": "172.21.15.23:6833/938496814", 2024-03-11T21:06:56.208 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_front_addr": "172.21.15.23:6831/938496814", 2024-03-11T21:06:56.208 INFO:teuthology.orchestra.run.smithi023.stdout: "state": [ 2024-03-11T21:06:56.208 INFO:teuthology.orchestra.run.smithi023.stdout: "exists", 2024-03-11T21:06:56.208 INFO:teuthology.orchestra.run.smithi023.stdout: "up" 2024-03-11T21:06:56.208 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:06:56.208 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:06:56.208 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:06:56.208 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": 4, 2024-03-11T21:06:56.208 INFO:teuthology.orchestra.run.smithi023.stdout: "uuid": "4dcbc91e-7035-413d-97bc-c9d804eb16d7", 2024-03-11T21:06:56.208 INFO:teuthology.orchestra.run.smithi023.stdout: "up": 1, 2024-03-11T21:06:56.208 INFO:teuthology.orchestra.run.smithi023.stdout: "in": 1, 2024-03-11T21:06:56.208 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 1, 2024-03-11T21:06:56.209 INFO:teuthology.orchestra.run.smithi023.stdout: "primary_affinity": 0, 2024-03-11T21:06:56.209 INFO:teuthology.orchestra.run.smithi023.stdout: "last_clean_begin": 35, 2024-03-11T21:06:56.209 INFO:teuthology.orchestra.run.smithi023.stdout: "last_clean_end": 182, 2024-03-11T21:06:56.209 INFO:teuthology.orchestra.run.smithi023.stdout: "up_from": 204, 2024-03-11T21:06:56.209 INFO:teuthology.orchestra.run.smithi023.stdout: "up_thru": 248, 2024-03-11T21:06:56.209 INFO:teuthology.orchestra.run.smithi023.stdout: "down_at": 183, 2024-03-11T21:06:56.209 INFO:teuthology.orchestra.run.smithi023.stdout: "lost_at": 0, 2024-03-11T21:06:56.209 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2024-03-11T21:06:56.209 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:06:56.209 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:06:56.209 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:06:56.209 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6800", 2024-03-11T21:06:56.210 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 3858359434 2024-03-11T21:06:56.210 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:06:56.210 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:06:56.210 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:06:56.210 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6801", 2024-03-11T21:06:56.210 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 3858359434 2024-03-11T21:06:56.210 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:06:56.210 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:06:56.210 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:06:56.210 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_addrs": { 2024-03-11T21:06:56.210 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:06:56.210 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:06:56.211 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:06:56.211 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6802", 2024-03-11T21:06:56.211 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 3858359434 2024-03-11T21:06:56.211 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:06:56.211 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:06:56.211 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:06:56.211 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6803", 2024-03-11T21:06:56.211 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 3858359434 2024-03-11T21:06:56.211 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:06:56.211 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:06:56.211 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:06:56.211 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_back_addrs": { 2024-03-11T21:06:56.211 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:06:56.212 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:06:56.212 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:06:56.212 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6806", 2024-03-11T21:06:56.212 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 3858359434 2024-03-11T21:06:56.212 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:06:56.212 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:06:56.212 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:06:56.212 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6807", 2024-03-11T21:06:56.212 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 3858359434 2024-03-11T21:06:56.212 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:06:56.212 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:06:56.212 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:06:56.212 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_front_addrs": { 2024-03-11T21:06:56.212 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:06:56.213 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:06:56.213 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:06:56.213 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6804", 2024-03-11T21:06:56.213 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 3858359434 2024-03-11T21:06:56.213 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:06:56.213 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:06:56.213 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:06:56.213 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6805", 2024-03-11T21:06:56.213 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 3858359434 2024-03-11T21:06:56.213 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:06:56.213 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:06:56.213 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:06:56.213 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.146:6801/3858359434", 2024-03-11T21:06:56.214 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_addr": "172.21.15.146:6803/3858359434", 2024-03-11T21:06:56.214 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_back_addr": "172.21.15.146:6807/3858359434", 2024-03-11T21:06:56.214 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_front_addr": "172.21.15.146:6805/3858359434", 2024-03-11T21:06:56.214 INFO:teuthology.orchestra.run.smithi023.stdout: "state": [ 2024-03-11T21:06:56.214 INFO:teuthology.orchestra.run.smithi023.stdout: "exists", 2024-03-11T21:06:56.214 INFO:teuthology.orchestra.run.smithi023.stdout: "up" 2024-03-11T21:06:56.214 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:06:56.214 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:06:56.214 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:06:56.214 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": 5, 2024-03-11T21:06:56.214 INFO:teuthology.orchestra.run.smithi023.stdout: "uuid": "aefcb766-8bae-4ff4-b71e-6691c8ac02f1", 2024-03-11T21:06:56.214 INFO:teuthology.orchestra.run.smithi023.stdout: "up": 1, 2024-03-11T21:06:56.215 INFO:teuthology.orchestra.run.smithi023.stdout: "in": 1, 2024-03-11T21:06:56.215 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 1, 2024-03-11T21:06:56.215 INFO:teuthology.orchestra.run.smithi023.stdout: "primary_affinity": 1, 2024-03-11T21:06:56.215 INFO:teuthology.orchestra.run.smithi023.stdout: "last_clean_begin": 0, 2024-03-11T21:06:56.215 INFO:teuthology.orchestra.run.smithi023.stdout: "last_clean_end": 0, 2024-03-11T21:06:56.215 INFO:teuthology.orchestra.run.smithi023.stdout: "up_from": 44, 2024-03-11T21:06:56.215 INFO:teuthology.orchestra.run.smithi023.stdout: "up_thru": 285, 2024-03-11T21:06:56.215 INFO:teuthology.orchestra.run.smithi023.stdout: "down_at": 0, 2024-03-11T21:06:56.215 INFO:teuthology.orchestra.run.smithi023.stdout: "lost_at": 0, 2024-03-11T21:06:56.215 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2024-03-11T21:06:56.215 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:06:56.215 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:06:56.216 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:06:56.216 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6808", 2024-03-11T21:06:56.216 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 4003230814 2024-03-11T21:06:56.216 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:06:56.216 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:06:56.216 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:06:56.216 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6809", 2024-03-11T21:06:56.216 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 4003230814 2024-03-11T21:06:56.216 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:06:56.216 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:06:56.216 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:06:56.217 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_addrs": { 2024-03-11T21:06:56.217 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:06:56.217 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:06:56.217 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:06:56.217 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6810", 2024-03-11T21:06:56.217 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 4003230814 2024-03-11T21:06:56.217 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:06:56.217 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:06:56.217 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:06:56.217 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6811", 2024-03-11T21:06:56.217 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 4003230814 2024-03-11T21:06:56.217 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:06:56.217 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:06:56.218 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:06:56.218 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_back_addrs": { 2024-03-11T21:06:56.218 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:06:56.218 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:06:56.218 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:06:56.218 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6814", 2024-03-11T21:06:56.218 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 4003230814 2024-03-11T21:06:56.218 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:06:56.218 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:06:56.218 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:06:56.218 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6815", 2024-03-11T21:06:56.218 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 4003230814 2024-03-11T21:06:56.219 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:06:56.219 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:06:56.219 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:06:56.219 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_front_addrs": { 2024-03-11T21:06:56.219 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:06:56.219 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:06:56.219 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:06:56.219 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6812", 2024-03-11T21:06:56.219 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 4003230814 2024-03-11T21:06:56.219 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:06:56.219 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:06:56.219 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:06:56.219 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6813", 2024-03-11T21:06:56.219 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 4003230814 2024-03-11T21:06:56.220 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:06:56.220 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:06:56.220 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:06:56.220 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.146:6809/4003230814", 2024-03-11T21:06:56.220 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_addr": "172.21.15.146:6811/4003230814", 2024-03-11T21:06:56.220 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_back_addr": "172.21.15.146:6815/4003230814", 2024-03-11T21:06:56.220 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_front_addr": "172.21.15.146:6813/4003230814", 2024-03-11T21:06:56.220 INFO:teuthology.orchestra.run.smithi023.stdout: "state": [ 2024-03-11T21:06:56.220 INFO:teuthology.orchestra.run.smithi023.stdout: "exists", 2024-03-11T21:06:56.220 INFO:teuthology.orchestra.run.smithi023.stdout: "up" 2024-03-11T21:06:56.220 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:06:56.220 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:06:56.220 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:06:56.220 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": 6, 2024-03-11T21:06:56.221 INFO:teuthology.orchestra.run.smithi023.stdout: "uuid": "5cb151e8-1f9b-4617-8a86-adfb06bb9606", 2024-03-11T21:06:56.221 INFO:teuthology.orchestra.run.smithi023.stdout: "up": 1, 2024-03-11T21:06:56.221 INFO:teuthology.orchestra.run.smithi023.stdout: "in": 1, 2024-03-11T21:06:56.221 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 1, 2024-03-11T21:06:56.221 INFO:teuthology.orchestra.run.smithi023.stdout: "primary_affinity": 1, 2024-03-11T21:06:56.221 INFO:teuthology.orchestra.run.smithi023.stdout: "last_clean_begin": 0, 2024-03-11T21:06:56.221 INFO:teuthology.orchestra.run.smithi023.stdout: "last_clean_end": 0, 2024-03-11T21:06:56.221 INFO:teuthology.orchestra.run.smithi023.stdout: "up_from": 57, 2024-03-11T21:06:56.221 INFO:teuthology.orchestra.run.smithi023.stdout: "up_thru": 285, 2024-03-11T21:06:56.221 INFO:teuthology.orchestra.run.smithi023.stdout: "down_at": 0, 2024-03-11T21:06:56.221 INFO:teuthology.orchestra.run.smithi023.stdout: "lost_at": 0, 2024-03-11T21:06:56.221 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2024-03-11T21:06:56.221 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:06:56.221 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:06:56.222 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:06:56.222 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6816", 2024-03-11T21:06:56.222 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 3334092376 2024-03-11T21:06:56.222 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:06:56.222 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:06:56.222 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:06:56.222 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6817", 2024-03-11T21:06:56.222 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 3334092376 2024-03-11T21:06:56.222 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:06:56.222 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:06:56.222 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:06:56.222 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_addrs": { 2024-03-11T21:06:56.223 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:06:56.223 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:06:56.223 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:06:56.223 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6818", 2024-03-11T21:06:56.223 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 3334092376 2024-03-11T21:06:56.223 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:06:56.223 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:06:56.223 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:06:56.223 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6819", 2024-03-11T21:06:56.223 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 3334092376 2024-03-11T21:06:56.223 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:06:56.223 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:06:56.224 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:06:56.224 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_back_addrs": { 2024-03-11T21:06:56.224 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:06:56.224 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:06:56.224 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:06:56.224 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6822", 2024-03-11T21:06:56.224 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 3334092376 2024-03-11T21:06:56.224 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:06:56.224 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:06:56.224 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:06:56.224 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6823", 2024-03-11T21:06:56.224 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 3334092376 2024-03-11T21:06:56.224 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:06:56.225 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:06:56.225 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:06:56.225 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_front_addrs": { 2024-03-11T21:06:56.225 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:06:56.225 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:06:56.225 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:06:56.225 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6820", 2024-03-11T21:06:56.225 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 3334092376 2024-03-11T21:06:56.225 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:06:56.225 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:06:56.225 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:06:56.225 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6821", 2024-03-11T21:06:56.225 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 3334092376 2024-03-11T21:06:56.226 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:06:56.226 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:06:56.226 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:06:56.226 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.146:6817/3334092376", 2024-03-11T21:06:56.226 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_addr": "172.21.15.146:6819/3334092376", 2024-03-11T21:06:56.226 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_back_addr": "172.21.15.146:6823/3334092376", 2024-03-11T21:06:56.226 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_front_addr": "172.21.15.146:6821/3334092376", 2024-03-11T21:06:56.226 INFO:teuthology.orchestra.run.smithi023.stdout: "state": [ 2024-03-11T21:06:56.227 INFO:teuthology.orchestra.run.smithi023.stdout: "exists", 2024-03-11T21:06:56.227 INFO:teuthology.orchestra.run.smithi023.stdout: "up" 2024-03-11T21:06:56.227 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:06:56.227 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:06:56.227 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:06:56.227 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": 7, 2024-03-11T21:06:56.227 INFO:teuthology.orchestra.run.smithi023.stdout: "uuid": "bdbd11a2-fa09-4f82-93a2-6b2d9c750bfd", 2024-03-11T21:06:56.227 INFO:teuthology.orchestra.run.smithi023.stdout: "up": 1, 2024-03-11T21:06:56.227 INFO:teuthology.orchestra.run.smithi023.stdout: "in": 1, 2024-03-11T21:06:56.227 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 1, 2024-03-11T21:06:56.227 INFO:teuthology.orchestra.run.smithi023.stdout: "primary_affinity": 1, 2024-03-11T21:06:56.227 INFO:teuthology.orchestra.run.smithi023.stdout: "last_clean_begin": 0, 2024-03-11T21:06:56.227 INFO:teuthology.orchestra.run.smithi023.stdout: "last_clean_end": 0, 2024-03-11T21:06:56.228 INFO:teuthology.orchestra.run.smithi023.stdout: "up_from": 68, 2024-03-11T21:06:56.228 INFO:teuthology.orchestra.run.smithi023.stdout: "up_thru": 285, 2024-03-11T21:06:56.228 INFO:teuthology.orchestra.run.smithi023.stdout: "down_at": 0, 2024-03-11T21:06:56.228 INFO:teuthology.orchestra.run.smithi023.stdout: "lost_at": 0, 2024-03-11T21:06:56.228 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2024-03-11T21:06:56.228 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:06:56.228 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:06:56.228 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:06:56.228 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6824", 2024-03-11T21:06:56.228 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1678407923 2024-03-11T21:06:56.228 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:06:56.228 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:06:56.229 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:06:56.229 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6825", 2024-03-11T21:06:56.229 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1678407923 2024-03-11T21:06:56.229 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:06:56.229 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:06:56.229 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:06:56.229 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_addrs": { 2024-03-11T21:06:56.229 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:06:56.229 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:06:56.229 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:06:56.229 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6826", 2024-03-11T21:06:56.229 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1678407923 2024-03-11T21:06:56.230 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:06:56.230 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:06:56.230 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:06:56.230 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6827", 2024-03-11T21:06:56.230 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1678407923 2024-03-11T21:06:56.230 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:06:56.230 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:06:56.230 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:06:56.230 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_back_addrs": { 2024-03-11T21:06:56.230 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:06:56.230 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:06:56.231 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:06:56.231 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6830", 2024-03-11T21:06:56.231 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1678407923 2024-03-11T21:06:56.231 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:06:56.231 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:06:56.231 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:06:56.231 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6831", 2024-03-11T21:06:56.231 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1678407923 2024-03-11T21:06:56.231 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:06:56.231 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:06:56.231 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:06:56.231 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_front_addrs": { 2024-03-11T21:06:56.231 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:06:56.232 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:06:56.232 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:06:56.232 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6828", 2024-03-11T21:06:56.232 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1678407923 2024-03-11T21:06:56.232 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:06:56.232 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:06:56.232 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:06:56.232 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6829", 2024-03-11T21:06:56.232 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1678407923 2024-03-11T21:06:56.232 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:06:56.232 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:06:56.232 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:06:56.233 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.146:6825/1678407923", 2024-03-11T21:06:56.233 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_addr": "172.21.15.146:6827/1678407923", 2024-03-11T21:06:56.233 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_back_addr": "172.21.15.146:6831/1678407923", 2024-03-11T21:06:56.233 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_front_addr": "172.21.15.146:6829/1678407923", 2024-03-11T21:06:56.233 INFO:teuthology.orchestra.run.smithi023.stdout: "state": [ 2024-03-11T21:06:56.233 INFO:teuthology.orchestra.run.smithi023.stdout: "exists", 2024-03-11T21:06:56.233 INFO:teuthology.orchestra.run.smithi023.stdout: "up" 2024-03-11T21:06:56.233 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:06:56.233 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:06:56.233 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-11T21:06:56.233 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_xinfo": [ 2024-03-11T21:06:56.233 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:06:56.233 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": 0, 2024-03-11T21:06:56.234 INFO:teuthology.orchestra.run.smithi023.stdout: "down_stamp": "0.000000", 2024-03-11T21:06:56.234 INFO:teuthology.orchestra.run.smithi023.stdout: "laggy_probability": 0, 2024-03-11T21:06:56.234 INFO:teuthology.orchestra.run.smithi023.stdout: "laggy_interval": 0, 2024-03-11T21:06:56.234 INFO:teuthology.orchestra.run.smithi023.stdout: "features": 4540701547738038271, 2024-03-11T21:06:56.234 INFO:teuthology.orchestra.run.smithi023.stdout: "old_weight": 0, 2024-03-11T21:06:56.234 INFO:teuthology.orchestra.run.smithi023.stdout: "last_purged_snaps_scrub": "2024-03-11T21:05:33.549850+0000", 2024-03-11T21:06:56.234 INFO:teuthology.orchestra.run.smithi023.stdout: "dead_epoch": 0 2024-03-11T21:06:56.234 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:06:56.234 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:06:56.234 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": 1, 2024-03-11T21:06:56.234 INFO:teuthology.orchestra.run.smithi023.stdout: "down_stamp": "0.000000", 2024-03-11T21:06:56.234 INFO:teuthology.orchestra.run.smithi023.stdout: "laggy_probability": 0, 2024-03-11T21:06:56.235 INFO:teuthology.orchestra.run.smithi023.stdout: "laggy_interval": 0, 2024-03-11T21:06:56.235 INFO:teuthology.orchestra.run.smithi023.stdout: "features": 4540701547738038271, 2024-03-11T21:06:56.235 INFO:teuthology.orchestra.run.smithi023.stdout: "old_weight": 0, 2024-03-11T21:06:56.235 INFO:teuthology.orchestra.run.smithi023.stdout: "last_purged_snaps_scrub": "2024-03-11T21:06:06.171226+0000", 2024-03-11T21:06:56.235 INFO:teuthology.orchestra.run.smithi023.stdout: "dead_epoch": 173 2024-03-11T21:06:56.235 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:06:56.235 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:06:56.235 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": 2, 2024-03-11T21:06:56.235 INFO:teuthology.orchestra.run.smithi023.stdout: "down_stamp": "0.000000", 2024-03-11T21:06:56.235 INFO:teuthology.orchestra.run.smithi023.stdout: "laggy_probability": 0, 2024-03-11T21:06:56.235 INFO:teuthology.orchestra.run.smithi023.stdout: "laggy_interval": 0, 2024-03-11T21:06:56.236 INFO:teuthology.orchestra.run.smithi023.stdout: "features": 4540701547738038271, 2024-03-11T21:06:56.236 INFO:teuthology.orchestra.run.smithi023.stdout: "old_weight": 0, 2024-03-11T21:06:56.236 INFO:teuthology.orchestra.run.smithi023.stdout: "last_purged_snaps_scrub": "2024-03-11T21:05:55.827614+0000", 2024-03-11T21:06:56.236 INFO:teuthology.orchestra.run.smithi023.stdout: "dead_epoch": 0 2024-03-11T21:06:56.236 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:06:56.236 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:06:56.236 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": 3, 2024-03-11T21:06:56.236 INFO:teuthology.orchestra.run.smithi023.stdout: "down_stamp": "0.000000", 2024-03-11T21:06:56.236 INFO:teuthology.orchestra.run.smithi023.stdout: "laggy_probability": 0, 2024-03-11T21:06:56.236 INFO:teuthology.orchestra.run.smithi023.stdout: "laggy_interval": 0, 2024-03-11T21:06:56.236 INFO:teuthology.orchestra.run.smithi023.stdout: "features": 4540701547738038271, 2024-03-11T21:06:56.236 INFO:teuthology.orchestra.run.smithi023.stdout: "old_weight": 0, 2024-03-11T21:06:56.236 INFO:teuthology.orchestra.run.smithi023.stdout: "last_purged_snaps_scrub": "2024-03-11T21:06:02.418537+0000", 2024-03-11T21:06:56.237 INFO:teuthology.orchestra.run.smithi023.stdout: "dead_epoch": 0 2024-03-11T21:06:56.237 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:06:56.237 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:06:56.237 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": 4, 2024-03-11T21:06:56.237 INFO:teuthology.orchestra.run.smithi023.stdout: "down_stamp": "0.000000", 2024-03-11T21:06:56.237 INFO:teuthology.orchestra.run.smithi023.stdout: "laggy_probability": 0, 2024-03-11T21:06:56.237 INFO:teuthology.orchestra.run.smithi023.stdout: "laggy_interval": 0, 2024-03-11T21:06:56.237 INFO:teuthology.orchestra.run.smithi023.stdout: "features": 4540701547738038271, 2024-03-11T21:06:56.237 INFO:teuthology.orchestra.run.smithi023.stdout: "old_weight": 0, 2024-03-11T21:06:56.237 INFO:teuthology.orchestra.run.smithi023.stdout: "last_purged_snaps_scrub": "2024-03-11T21:05:49.825016+0000", 2024-03-11T21:06:56.237 INFO:teuthology.orchestra.run.smithi023.stdout: "dead_epoch": 182 2024-03-11T21:06:56.237 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:06:56.237 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:06:56.238 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": 5, 2024-03-11T21:06:56.238 INFO:teuthology.orchestra.run.smithi023.stdout: "down_stamp": "0.000000", 2024-03-11T21:06:56.238 INFO:teuthology.orchestra.run.smithi023.stdout: "laggy_probability": 0, 2024-03-11T21:06:56.238 INFO:teuthology.orchestra.run.smithi023.stdout: "laggy_interval": 0, 2024-03-11T21:06:56.238 INFO:teuthology.orchestra.run.smithi023.stdout: "features": 4540701547738038271, 2024-03-11T21:06:56.238 INFO:teuthology.orchestra.run.smithi023.stdout: "old_weight": 0, 2024-03-11T21:06:56.238 INFO:teuthology.orchestra.run.smithi023.stdout: "last_purged_snaps_scrub": "2024-03-11T21:06:54.688050+0000", 2024-03-11T21:06:56.238 INFO:teuthology.orchestra.run.smithi023.stdout: "dead_epoch": 0 2024-03-11T21:06:56.238 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:06:56.238 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:06:56.238 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": 6, 2024-03-11T21:06:56.239 INFO:teuthology.orchestra.run.smithi023.stdout: "down_stamp": "0.000000", 2024-03-11T21:06:56.239 INFO:teuthology.orchestra.run.smithi023.stdout: "laggy_probability": 0, 2024-03-11T21:06:56.239 INFO:teuthology.orchestra.run.smithi023.stdout: "laggy_interval": 0, 2024-03-11T21:06:56.239 INFO:teuthology.orchestra.run.smithi023.stdout: "features": 4540701547738038271, 2024-03-11T21:06:56.239 INFO:teuthology.orchestra.run.smithi023.stdout: "old_weight": 0, 2024-03-11T21:06:56.239 INFO:teuthology.orchestra.run.smithi023.stdout: "last_purged_snaps_scrub": "2024-03-11T21:06:00.603774+0000", 2024-03-11T21:06:56.239 INFO:teuthology.orchestra.run.smithi023.stdout: "dead_epoch": 0 2024-03-11T21:06:56.239 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:06:56.239 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:06:56.239 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": 7, 2024-03-11T21:06:56.239 INFO:teuthology.orchestra.run.smithi023.stdout: "down_stamp": "0.000000", 2024-03-11T21:06:56.239 INFO:teuthology.orchestra.run.smithi023.stdout: "laggy_probability": 0, 2024-03-11T21:06:56.239 INFO:teuthology.orchestra.run.smithi023.stdout: "laggy_interval": 0, 2024-03-11T21:06:56.240 INFO:teuthology.orchestra.run.smithi023.stdout: "features": 4540701547738038271, 2024-03-11T21:06:56.240 INFO:teuthology.orchestra.run.smithi023.stdout: "old_weight": 0, 2024-03-11T21:06:56.240 INFO:teuthology.orchestra.run.smithi023.stdout: "last_purged_snaps_scrub": "2024-03-11T21:06:53.461029+0000", 2024-03-11T21:06:56.240 INFO:teuthology.orchestra.run.smithi023.stdout: "dead_epoch": 0 2024-03-11T21:06:56.240 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:06:56.240 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-11T21:06:56.240 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_upmap": [], 2024-03-11T21:06:56.240 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_upmap_items": [ 2024-03-11T21:06:56.240 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:06:56.240 INFO:teuthology.orchestra.run.smithi023.stdout: "pgid": "2.0", 2024-03-11T21:06:56.240 INFO:teuthology.orchestra.run.smithi023.stdout: "mappings": [ 2024-03-11T21:06:56.240 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:06:56.240 INFO:teuthology.orchestra.run.smithi023.stdout: "from": 7, 2024-03-11T21:06:56.240 INFO:teuthology.orchestra.run.smithi023.stdout: "to": 2 2024-03-11T21:06:56.241 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:06:56.241 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:06:56.241 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:06:56.241 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:06:56.241 INFO:teuthology.orchestra.run.smithi023.stdout: "pgid": "4.b", 2024-03-11T21:06:56.241 INFO:teuthology.orchestra.run.smithi023.stdout: "mappings": [ 2024-03-11T21:06:56.241 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:06:56.241 INFO:teuthology.orchestra.run.smithi023.stdout: "from": 1, 2024-03-11T21:06:56.241 INFO:teuthology.orchestra.run.smithi023.stdout: "to": 2 2024-03-11T21:06:56.241 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:06:56.241 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:06:56.241 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:06:56.241 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:06:56.242 INFO:teuthology.orchestra.run.smithi023.stdout: "pgid": "4.d", 2024-03-11T21:06:56.242 INFO:teuthology.orchestra.run.smithi023.stdout: "mappings": [ 2024-03-11T21:06:56.242 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:06:56.242 INFO:teuthology.orchestra.run.smithi023.stdout: "from": 1, 2024-03-11T21:06:56.242 INFO:teuthology.orchestra.run.smithi023.stdout: "to": 5 2024-03-11T21:06:56.242 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:06:56.242 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:06:56.242 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:06:56.242 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:06:56.242 INFO:teuthology.orchestra.run.smithi023.stdout: "pgid": "4.e", 2024-03-11T21:06:56.242 INFO:teuthology.orchestra.run.smithi023.stdout: "mappings": [ 2024-03-11T21:06:56.242 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:06:56.243 INFO:teuthology.orchestra.run.smithi023.stdout: "from": 1, 2024-03-11T21:06:56.243 INFO:teuthology.orchestra.run.smithi023.stdout: "to": 2 2024-03-11T21:06:56.243 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:06:56.243 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:06:56.243 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:06:56.243 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:06:56.243 INFO:teuthology.orchestra.run.smithi023.stdout: "pgid": "4.16", 2024-03-11T21:06:56.243 INFO:teuthology.orchestra.run.smithi023.stdout: "mappings": [ 2024-03-11T21:06:56.243 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:06:56.243 INFO:teuthology.orchestra.run.smithi023.stdout: "from": 3, 2024-03-11T21:06:56.243 INFO:teuthology.orchestra.run.smithi023.stdout: "to": 2 2024-03-11T21:06:56.243 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:06:56.244 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:06:56.244 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:06:56.244 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:06:56.244 INFO:teuthology.orchestra.run.smithi023.stdout: "pgid": "4.1d", 2024-03-11T21:06:56.244 INFO:teuthology.orchestra.run.smithi023.stdout: "mappings": [ 2024-03-11T21:06:56.244 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:06:56.244 INFO:teuthology.orchestra.run.smithi023.stdout: "from": 3, 2024-03-11T21:06:56.244 INFO:teuthology.orchestra.run.smithi023.stdout: "to": 5 2024-03-11T21:06:56.244 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:06:56.244 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:06:56.244 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:06:56.244 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-11T21:06:56.244 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_upmap_primaries": [], 2024-03-11T21:06:56.245 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_temp": [], 2024-03-11T21:06:56.245 INFO:teuthology.orchestra.run.smithi023.stdout: "primary_temp": [], 2024-03-11T21:06:56.245 INFO:teuthology.orchestra.run.smithi023.stdout: "blocklist": { 2024-03-11T21:06:56.245 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:0/1618334186": "2024-03-12T20:40:45.361537+0000", 2024-03-11T21:06:56.245 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:6801/2813578081": "2024-03-12T20:40:45.361537+0000", 2024-03-11T21:06:56.245 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:0/2869940180": "2024-03-12T20:51:33.790205+0000", 2024-03-11T21:06:56.245 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:6800/3037564930": "2024-03-12T20:51:33.790205+0000", 2024-03-11T21:06:56.245 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:0/3297567084": "2024-03-12T20:51:33.790205+0000", 2024-03-11T21:06:56.245 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:0/3708753659": "2024-03-12T20:40:45.361537+0000", 2024-03-11T21:06:56.245 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:0/2004146740": "2024-03-12T20:40:45.361537+0000", 2024-03-11T21:06:56.245 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:0/827653812": "2024-03-12T20:41:21.301516+0000", 2024-03-11T21:06:56.245 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:6800/1057699850": "2024-03-12T20:41:21.301516+0000", 2024-03-11T21:06:56.246 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:6800/2813578081": "2024-03-12T20:40:45.361537+0000", 2024-03-11T21:06:56.246 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:6801/1057699850": "2024-03-12T20:41:21.301516+0000", 2024-03-11T21:06:56.246 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:0/3833230375": "2024-03-12T20:41:21.301516+0000", 2024-03-11T21:06:56.246 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:6801/3037564930": "2024-03-12T20:51:33.790205+0000", 2024-03-11T21:06:56.246 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:0/1182809102": "2024-03-12T20:51:33.790205+0000", 2024-03-11T21:06:56.246 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:0/2122134187": "2024-03-12T20:51:33.790205+0000", 2024-03-11T21:06:56.246 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:0/2356701899": "2024-03-12T20:41:21.301516+0000", 2024-03-11T21:06:56.246 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:0/843974100": "2024-03-12T20:51:33.790205+0000" 2024-03-11T21:06:56.246 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:06:56.246 INFO:teuthology.orchestra.run.smithi023.stdout: "range_blocklist": {}, 2024-03-11T21:06:56.246 INFO:teuthology.orchestra.run.smithi023.stdout: "erasure_code_profiles": { 2024-03-11T21:06:56.246 INFO:teuthology.orchestra.run.smithi023.stdout: "default": { 2024-03-11T21:06:56.247 INFO:teuthology.orchestra.run.smithi023.stdout: "crush-failure-domain": "osd", 2024-03-11T21:06:56.247 INFO:teuthology.orchestra.run.smithi023.stdout: "k": "2", 2024-03-11T21:06:56.247 INFO:teuthology.orchestra.run.smithi023.stdout: "m": "1", 2024-03-11T21:06:56.247 INFO:teuthology.orchestra.run.smithi023.stdout: "plugin": "jerasure", 2024-03-11T21:06:56.247 INFO:teuthology.orchestra.run.smithi023.stdout: "technique": "reed_sol_van" 2024-03-11T21:06:56.247 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:06:56.247 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:06:56.247 INFO:teuthology.orchestra.run.smithi023.stdout: "removed_snaps_queue": [], 2024-03-11T21:06:56.247 INFO:teuthology.orchestra.run.smithi023.stdout: "new_removed_snaps": [], 2024-03-11T21:06:56.247 INFO:teuthology.orchestra.run.smithi023.stdout: "new_purged_snaps": [], 2024-03-11T21:06:56.247 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_node_flags": {}, 2024-03-11T21:06:56.247 INFO:teuthology.orchestra.run.smithi023.stdout: "device_class_flags": {}, 2024-03-11T21:06:56.247 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": { 2024-03-11T21:06:56.247 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode_enabled": false, 2024-03-11T21:06:56.248 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_bucket_count": 0, 2024-03-11T21:06:56.248 INFO:teuthology.orchestra.run.smithi023.stdout: "degraded_stretch_mode": 0, 2024-03-11T21:06:56.248 INFO:teuthology.orchestra.run.smithi023.stdout: "recovering_stretch_mode": 0, 2024-03-11T21:06:56.248 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode_bucket": 0 2024-03-11T21:06:56.248 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:06:56.248 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-11T21:06:56.248 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:06:56.268 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:06:56.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:56.269 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:06:56.370 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:06:56.394 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:06:56.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:56.396 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:06:56.497 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:06:56.521 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:06:56.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:56.523 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:06:56.624 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:56.651 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:56.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:56.653 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:06:56.751 INFO:teuthology.orchestra.run.smithi023.stderr:pool 'unique_pool_2' removed 2024-03-11T21:06:56.754 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:06:56.792 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:06:56.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:56.796 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:06:56.897 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:06:56.918 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:06:56.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:56.920 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:06:57.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:56 smithi146 bash[23809]: audit 2024-03-11T21:06:55.641329+0000 mgr.y (mgr.24380) 597 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:06:57.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:56 smithi146 bash[23809]: audit 2024-03-11T21:06:56.139633+0000 mon.c (mon.2) 75 : audit 0 from='client.? 172.21.15.23:0/2864173891' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-11T21:06:57.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:56 smithi146 bash[23809]: audit 2024-03-11T21:06:56.215109+0000 mon.c (mon.2) 76 : audit 1 from='client.? 172.21.15.23:0/1322551593' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T21:06:57.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:56 smithi146 bash[23809]: audit 2024-03-11T21:06:56.215430+0000 mon.a (mon.0) 1586 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T21:06:57.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:56 smithi146 bash[23809]: cluster 2024-03-11T21:06:56.236027+0000 mgr.y (mgr.24380) 598 : cluster 0 pgmap v663: 148 pgs: 148 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:06:57.020 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:06:57.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:56 smithi023 bash[21372]: audit 2024-03-11T21:06:55.641329+0000 mgr.y (mgr.24380) 597 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:06:57.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:56 smithi023 bash[21372]: audit 2024-03-11T21:06:56.139633+0000 mon.c (mon.2) 75 : audit 0 from='client.? 172.21.15.23:0/2864173891' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-11T21:06:57.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:56 smithi023 bash[21372]: audit 2024-03-11T21:06:56.215109+0000 mon.c (mon.2) 76 : audit 1 from='client.? 172.21.15.23:0/1322551593' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T21:06:57.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:56 smithi023 bash[21372]: audit 2024-03-11T21:06:56.215430+0000 mon.a (mon.0) 1586 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T21:06:57.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:56 smithi023 bash[21372]: cluster 2024-03-11T21:06:56.236027+0000 mgr.y (mgr.24380) 598 : cluster 0 pgmap v663: 148 pgs: 148 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:06:57.031 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:56 smithi023 bash[29146]: audit 2024-03-11T21:06:55.641329+0000 mgr.y (mgr.24380) 597 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:06:57.031 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:56 smithi023 bash[29146]: audit 2024-03-11T21:06:56.139633+0000 mon.c (mon.2) 75 : audit 0 from='client.? 172.21.15.23:0/2864173891' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-11T21:06:57.031 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:56 smithi023 bash[29146]: audit 2024-03-11T21:06:56.215109+0000 mon.c (mon.2) 76 : audit 1 from='client.? 172.21.15.23:0/1322551593' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T21:06:57.032 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:56 smithi023 bash[29146]: audit 2024-03-11T21:06:56.215430+0000 mon.a (mon.0) 1586 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T21:06:57.032 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:56 smithi023 bash[29146]: cluster 2024-03-11T21:06:56.236027+0000 mgr.y (mgr.24380) 598 : cluster 0 pgmap v663: 148 pgs: 148 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:06:57.046 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:06:57.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:57.049 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:06:57.150 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:57.174 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:06:57.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:57.177 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:06:57.188 INFO:tasks.thrashosds.thrasher:j is {'epoch': 300, 'fsid': '67ba00f8-dfe7-11ee-95c8-87774f69a715', 'created': '2024-03-11T20:40:13.664802+0000', 'modified': '2024-03-11T21:06:55.687752+0000', 'last_up_change': '2024-03-11T21:00:47.324817+0000', 'last_in_change': '2024-03-11T20:57:23.560527+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 20, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 9, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-11T20:45:31.964570+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '142', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6.980000019073486, 'score_stable': 6.980000019073486, 'optimal_score': 0.4300000071525574, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2024-03-11T20:50:26.421486+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '78', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3200000524520874, 'score_stable': 1.3200000524520874, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 3, 'pool_name': 'default.rgw.log', 'create_time': '2024-03-11T20:50:28.592615+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5399999618530273, 'score_stable': 1.5399999618530273, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 4, 'pool_name': 'default.rgw.control', 'create_time': '2024-03-11T20:50:30.536371+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9700000286102295, 'score_stable': 1.9700000286102295, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 5, 'pool_name': 'datapool', 'create_time': '2024-03-11T20:50:31.135845+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 87, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 4.670000076293945, 'score_stable': 4.670000076293945, 'optimal_score': 1, 'raw_score_acting': 4.670000076293945, 'raw_score_stable': 4.670000076293945, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-03-11T20:50:33.233163+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 9, 'pool_name': 'unique_pool_2', 'create_time': '2024-03-11T21:02:46.162698+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '256', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}], 'osds': [{'osd': 0, 'uuid': '3df72b4a-8fe4-4265-b4c6-af022e1e23e1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 297, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6802', 'nonce': 2154627393}, {'type': 'v1', 'addr': '172.21.15.23:6803', 'nonce': 2154627393}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6804', 'nonce': 2154627393}, {'type': 'v1', 'addr': '172.21.15.23:6805', 'nonce': 2154627393}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6808', 'nonce': 2154627393}, {'type': 'v1', 'addr': '172.21.15.23:6809', 'nonce': 2154627393}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6806', 'nonce': 2154627393}, {'type': 'v1', 'addr': '172.21.15.23:6807', 'nonce': 2154627393}]}, 'public_addr': '172.21.15.23:6803/2154627393', 'cluster_addr': '172.21.15.23:6805/2154627393', 'heartbeat_back_addr': '172.21.15.23:6809/2154627393', 'heartbeat_front_addr': '172.21.15.23:6807/2154627393', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'e8fd7500-9081-41a1-8077-a4d0f0e12a2c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 13, 'last_clean_end': 173, 'up_from': 211, 'up_thru': 285, 'down_at': 174, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6810', 'nonce': 56585354}, {'type': 'v1', 'addr': '172.21.15.23:6811', 'nonce': 56585354}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6812', 'nonce': 56585354}, {'type': 'v1', 'addr': '172.21.15.23:6813', 'nonce': 56585354}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6816', 'nonce': 56585354}, {'type': 'v1', 'addr': '172.21.15.23:6817', 'nonce': 56585354}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6814', 'nonce': 56585354}, {'type': 'v1', 'addr': '172.21.15.23:6815', 'nonce': 56585354}]}, 'public_addr': '172.21.15.23:6811/56585354', 'cluster_addr': '172.21.15.23:6813/56585354', 'heartbeat_back_addr': '172.21.15.23:6817/56585354', 'heartbeat_front_addr': '172.21.15.23:6815/56585354', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'dd85d52a-df48-4b03-8612-cecfd82b5987', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 19, 'up_thru': 285, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6818', 'nonce': 1770748056}, {'type': 'v1', 'addr': '172.21.15.23:6819', 'nonce': 1770748056}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6820', 'nonce': 1770748056}, {'type': 'v1', 'addr': '172.21.15.23:6821', 'nonce': 1770748056}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6824', 'nonce': 1770748056}, {'type': 'v1', 'addr': '172.21.15.23:6825', 'nonce': 1770748056}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6822', 'nonce': 1770748056}, {'type': 'v1', 'addr': '172.21.15.23:6823', 'nonce': 1770748056}]}, 'public_addr': '172.21.15.23:6819/1770748056', 'cluster_addr': '172.21.15.23:6821/1770748056', 'heartbeat_back_addr': '172.21.15.23:6825/1770748056', 'heartbeat_front_addr': '172.21.15.23:6823/1770748056', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'e3021106-3779-4b94-9bbf-a7724544110f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 28, 'up_thru': 285, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6826', 'nonce': 938496814}, {'type': 'v1', 'addr': '172.21.15.23:6827', 'nonce': 938496814}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6828', 'nonce': 938496814}, {'type': 'v1', 'addr': '172.21.15.23:6829', 'nonce': 938496814}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6832', 'nonce': 938496814}, {'type': 'v1', 'addr': '172.21.15.23:6833', 'nonce': 938496814}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6830', 'nonce': 938496814}, {'type': 'v1', 'addr': '172.21.15.23:6831', 'nonce': 938496814}]}, 'public_addr': '172.21.15.23:6827/938496814', 'cluster_addr': '172.21.15.23:6829/938496814', 'heartbeat_back_addr': '172.21.15.23:6833/938496814', 'heartbeat_front_addr': '172.21.15.23:6831/938496814', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '4dcbc91e-7035-413d-97bc-c9d804eb16d7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 35, 'last_clean_end': 182, 'up_from': 204, 'up_thru': 248, 'down_at': 183, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6800', 'nonce': 3858359434}, {'type': 'v1', 'addr': '172.21.15.146:6801', 'nonce': 3858359434}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6802', 'nonce': 3858359434}, {'type': 'v1', 'addr': '172.21.15.146:6803', 'nonce': 3858359434}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6806', 'nonce': 3858359434}, {'type': 'v1', 'addr': '172.21.15.146:6807', 'nonce': 3858359434}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6804', 'nonce': 3858359434}, {'type': 'v1', 'addr': '172.21.15.146:6805', 'nonce': 3858359434}]}, 'public_addr': '172.21.15.146:6801/3858359434', 'cluster_addr': '172.21.15.146:6803/3858359434', 'heartbeat_back_addr': '172.21.15.146:6807/3858359434', 'heartbeat_front_addr': '172.21.15.146:6805/3858359434', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'aefcb766-8bae-4ff4-b71e-6691c8ac02f1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 44, 'up_thru': 285, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6808', 'nonce': 4003230814}, {'type': 'v1', 'addr': '172.21.15.146:6809', 'nonce': 4003230814}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6810', 'nonce': 4003230814}, {'type': 'v1', 'addr': '172.21.15.146:6811', 'nonce': 4003230814}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6814', 'nonce': 4003230814}, {'type': 'v1', 'addr': '172.21.15.146:6815', 'nonce': 4003230814}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6812', 'nonce': 4003230814}, {'type': 'v1', 'addr': '172.21.15.146:6813', 'nonce': 4003230814}]}, 'public_addr': '172.21.15.146:6809/4003230814', 'cluster_addr': '172.21.15.146:6811/4003230814', 'heartbeat_back_addr': '172.21.15.146:6815/4003230814', 'heartbeat_front_addr': '172.21.15.146:6813/4003230814', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '5cb151e8-1f9b-4617-8a86-adfb06bb9606', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 57, 'up_thru': 285, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6816', 'nonce': 3334092376}, {'type': 'v1', 'addr': '172.21.15.146:6817', 'nonce': 3334092376}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6818', 'nonce': 3334092376}, {'type': 'v1', 'addr': '172.21.15.146:6819', 'nonce': 3334092376}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6822', 'nonce': 3334092376}, {'type': 'v1', 'addr': '172.21.15.146:6823', 'nonce': 3334092376}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6820', 'nonce': 3334092376}, {'type': 'v1', 'addr': '172.21.15.146:6821', 'nonce': 3334092376}]}, 'public_addr': '172.21.15.146:6817/3334092376', 'cluster_addr': '172.21.15.146:6819/3334092376', 'heartbeat_back_addr': '172.21.15.146:6823/3334092376', 'heartbeat_front_addr': '172.21.15.146:6821/3334092376', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'bdbd11a2-fa09-4f82-93a2-6b2d9c750bfd', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 68, 'up_thru': 285, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6824', 'nonce': 1678407923}, {'type': 'v1', 'addr': '172.21.15.146:6825', 'nonce': 1678407923}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6826', 'nonce': 1678407923}, {'type': 'v1', 'addr': '172.21.15.146:6827', 'nonce': 1678407923}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6830', 'nonce': 1678407923}, {'type': 'v1', 'addr': '172.21.15.146:6831', 'nonce': 1678407923}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6828', 'nonce': 1678407923}, {'type': 'v1', 'addr': '172.21.15.146:6829', 'nonce': 1678407923}]}, 'public_addr': '172.21.15.146:6825/1678407923', 'cluster_addr': '172.21.15.146:6827/1678407923', 'heartbeat_back_addr': '172.21.15.146:6831/1678407923', 'heartbeat_front_addr': '172.21.15.146:6829/1678407923', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T21:05:33.549850+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T21:06:06.171226+0000', 'dead_epoch': 173}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T21:05:55.827614+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T21:06:02.418537+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T21:05:49.825016+0000', 'dead_epoch': 182}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T21:06:54.688050+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T21:06:00.603774+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T21:06:53.461029+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '2.0', 'mappings': [{'from': 7, 'to': 2}]}, {'pgid': '4.b', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.d', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.e', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.16', 'mappings': [{'from': 3, 'to': 2}]}, {'pgid': '4.1d', 'mappings': [{'from': 3, 'to': 5}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.23:0/1618334186': '2024-03-12T20:40:45.361537+0000', '172.21.15.23:6801/2813578081': '2024-03-12T20:40:45.361537+0000', '172.21.15.23:0/2869940180': '2024-03-12T20:51:33.790205+0000', '172.21.15.23:6800/3037564930': '2024-03-12T20:51:33.790205+0000', '172.21.15.23:0/3297567084': '2024-03-12T20:51:33.790205+0000', '172.21.15.23:0/3708753659': '2024-03-12T20:40:45.361537+0000', '172.21.15.23:0/2004146740': '2024-03-12T20:40:45.361537+0000', '172.21.15.23:0/827653812': '2024-03-12T20:41:21.301516+0000', '172.21.15.23:6800/1057699850': '2024-03-12T20:41:21.301516+0000', '172.21.15.23:6800/2813578081': '2024-03-12T20:40:45.361537+0000', '172.21.15.23:6801/1057699850': '2024-03-12T20:41:21.301516+0000', '172.21.15.23:0/3833230375': '2024-03-12T20:41:21.301516+0000', '172.21.15.23:6801/3037564930': '2024-03-12T20:51:33.790205+0000', '172.21.15.23:0/1182809102': '2024-03-12T20:51:33.790205+0000', '172.21.15.23:0/2122134187': '2024-03-12T20:51:33.790205+0000', '172.21.15.23:0/2356701899': '2024-03-12T20:41:21.301516+0000', '172.21.15.23:0/843974100': '2024-03-12T20:51:33.790205+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-03-11T21:06:57.189 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T21:06:57.278 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:06:57.303 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:06:57.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:57.306 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:06:57.407 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:57.433 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:57.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:57.435 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:06:57.479 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-11T21:06:57.535 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:06:57.560 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:06:57.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:57.563 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:06:57.663 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:57.690 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:57.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:57.692 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:06:57.781 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:57 smithi023 bash[21372]: audit 2024-03-11T21:06:56.714049+0000 mon.a (mon.0) 1587 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-03-11T21:06:57.781 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:57 smithi023 bash[21372]: cluster 2024-03-11T21:06:56.725998+0000 mon.a (mon.0) 1588 : cluster 0 osdmap e301: 8 total, 8 up, 8 in 2024-03-11T21:06:57.781 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:57 smithi023 bash[29146]: audit 2024-03-11T21:06:56.714049+0000 mon.a (mon.0) 1587 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-03-11T21:06:57.781 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:57 smithi023 bash[29146]: cluster 2024-03-11T21:06:56.725998+0000 mon.a (mon.0) 1588 : cluster 0 osdmap e301: 8 total, 8 up, 8 in 2024-03-11T21:06:57.792 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:06:57.817 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:06:57.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:57.819 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:06:57.879 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-03-11T21:06:57.879 INFO:tasks.radosbench:Beginning radosbench... 2024-03-11T21:06:57.880 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_3 2024-03-11T21:06:57.880 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd pool create unique_pool_3 16 2024-03-11T21:06:57.920 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:57.931 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:57 smithi146 bash[23809]: audit 2024-03-11T21:06:56.714049+0000 mon.a (mon.0) 1587 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-03-11T21:06:57.931 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:57 smithi146 bash[23809]: cluster 2024-03-11T21:06:56.725998+0000 mon.a (mon.0) 1588 : cluster 0 osdmap e301: 8 total, 8 up, 8 in 2024-03-11T21:06:57.946 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:57.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:57.949 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:06:58.050 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:58.077 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:58.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:58.079 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:06:58.180 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:58.206 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:58.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:58.208 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:06:58.309 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:06:58.334 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:06:58.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:58.337 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:06:58.344 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:06:58.438 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:06:58.465 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:06:58.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:58.468 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:06:58.569 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:06:58.595 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:06:58.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:58.598 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:06:58.698 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:58.725 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:58.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:58.727 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:06:58.728 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:06:58 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:06:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:06:58.828 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:58.839 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:06:58 smithi146 bash[23809]: cluster 2024-03-11T21:06:58.236862+0000 mgr.y (mgr.24380) 599 : cluster 0 pgmap v665: 132 pgs: 132 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-11T21:06:58.854 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:58.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:58.857 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:06:58.958 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:06:58.984 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:06:58.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:58.986 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:06:59.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:06:58 smithi023 bash[21372]: cluster 2024-03-11T21:06:58.236862+0000 mgr.y (mgr.24380) 599 : cluster 0 pgmap v665: 132 pgs: 132 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-11T21:06:59.031 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:06:58 smithi023 bash[29146]: cluster 2024-03-11T21:06:58.236862+0000 mgr.y (mgr.24380) 599 : cluster 0 pgmap v665: 132 pgs: 132 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-11T21:06:59.087 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:59.114 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:59.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:59.117 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:06:59.217 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:59.246 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:06:59.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:59.248 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:06:59.349 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:06:59.366 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:06:59.377 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:06:59.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:59.379 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:06:59.480 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:06:59.508 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:06:59.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:59.511 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:06:59.612 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:59.638 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:06:59.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:59.640 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:06:59.741 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:06:59.766 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:06:59.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:59.769 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:06:59.869 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:06:59.896 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:06:59.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:59.898 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:06:59.999 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:07:00.026 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:07:00.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:00.029 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:07:00.130 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:00.153 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:00.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:00.155 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:07:00.256 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:07:00.284 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:07:00.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:00.287 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:07:00.387 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:07:00.416 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:07:00.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:00.425 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:07:00.526 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:07:00.552 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:07:00.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:00.554 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:07:00.655 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:07:00.682 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:07:00.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:00.685 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:07:00.786 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:00.814 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:00.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:00.816 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:07:00.917 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:07:00.944 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:07:00.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:00.947 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:07:01.047 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:07:01.075 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:07:01.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:01.078 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:07:01.179 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:07:01.208 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:07:01.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:01.210 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:07:01.311 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:01.338 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:01.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:01.341 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:07:01.442 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:07:01.453 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:01 smithi023 bash[21372]: cluster 2024-03-11T21:07:00.079996+0000 osd.0 (osd.0) 243 : cluster 0 purged_snaps scrub starts 2024-03-11T21:07:01.453 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:01 smithi023 bash[21372]: cluster 2024-03-11T21:07:00.080076+0000 osd.0 (osd.0) 244 : cluster 0 purged_snaps scrub ok 2024-03-11T21:07:01.454 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:01 smithi023 bash[21372]: cluster 2024-03-11T21:07:00.144247+0000 mon.a (mon.0) 1589 : cluster 0 osdmap e302: 8 total, 8 up, 8 in 2024-03-11T21:07:01.454 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:01 smithi023 bash[21372]: cluster 2024-03-11T21:07:00.237622+0000 mgr.y (mgr.24380) 600 : cluster 0 pgmap v667: 132 pgs: 132 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-11T21:07:01.454 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:01 smithi023 bash[29146]: cluster 2024-03-11T21:07:00.079996+0000 osd.0 (osd.0) 243 : cluster 0 purged_snaps scrub starts 2024-03-11T21:07:01.454 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:01 smithi023 bash[29146]: cluster 2024-03-11T21:07:00.080076+0000 osd.0 (osd.0) 244 : cluster 0 purged_snaps scrub ok 2024-03-11T21:07:01.454 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:01 smithi023 bash[29146]: cluster 2024-03-11T21:07:00.144247+0000 mon.a (mon.0) 1589 : cluster 0 osdmap e302: 8 total, 8 up, 8 in 2024-03-11T21:07:01.454 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:01 smithi023 bash[29146]: cluster 2024-03-11T21:07:00.237622+0000 mgr.y (mgr.24380) 600 : cluster 0 pgmap v667: 132 pgs: 132 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-11T21:07:01.469 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:07:01.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:01.471 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:07:01.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:01 smithi146 bash[23809]: cluster 2024-03-11T21:07:00.079996+0000 osd.0 (osd.0) 243 : cluster 0 purged_snaps scrub starts 2024-03-11T21:07:01.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:01 smithi146 bash[23809]: cluster 2024-03-11T21:07:00.080076+0000 osd.0 (osd.0) 244 : cluster 0 purged_snaps scrub ok 2024-03-11T21:07:01.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:01 smithi146 bash[23809]: cluster 2024-03-11T21:07:00.144247+0000 mon.a (mon.0) 1589 : cluster 0 osdmap e302: 8 total, 8 up, 8 in 2024-03-11T21:07:01.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:01 smithi146 bash[23809]: cluster 2024-03-11T21:07:00.237622+0000 mgr.y (mgr.24380) 600 : cluster 0 pgmap v667: 132 pgs: 132 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-11T21:07:01.572 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:07:01.598 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:07:01.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:01.601 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:07:01.702 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:07:01.722 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:07:01.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:01.725 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:07:01.825 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:07:01.854 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:07:01.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:01.856 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:07:01.957 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:07:01.985 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:07:01.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:01.987 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:07:02.088 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:07:02.116 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:07:02.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:02.118 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:07:02.219 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:07:02.244 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:07:02.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:02.247 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:07:02.348 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:07:02.371 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:07:02.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:02.373 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:07:02.440 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:07:02.441 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:07:02.441 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:07:02.474 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:02.502 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:02.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:02.504 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:07:02.605 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:07:02.634 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:07:02.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:02.639 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:07:02.740 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:02.766 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:02.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:02.768 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:07:02.869 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:02.897 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:02.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:02.900 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:07:03.000 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:03.029 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:03.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:03.031 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:07:03.131 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:03.159 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:03.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:03.161 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:07:03.262 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:07:03.290 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:07:03.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:03.292 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:07:03.393 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:07:03.422 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:07:03.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:03.424 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:07:03.525 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:07:03.552 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:07:03.553 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:03 smithi023 bash[21372]: cluster 2024-03-11T21:07:02.238227+0000 mgr.y (mgr.24380) 601 : cluster 0 pgmap v668: 132 pgs: 132 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 625 B/s rd, 0 op/s 2024-03-11T21:07:03.553 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:03 smithi023 bash[29146]: cluster 2024-03-11T21:07:02.238227+0000 mgr.y (mgr.24380) 601 : cluster 0 pgmap v668: 132 pgs: 132 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 625 B/s rd, 0 op/s 2024-03-11T21:07:03.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:03.554 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:07:03.655 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:03.666 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:03 smithi146 bash[23809]: cluster 2024-03-11T21:07:02.238227+0000 mgr.y (mgr.24380) 601 : cluster 0 pgmap v668: 132 pgs: 132 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 625 B/s rd, 0 op/s 2024-03-11T21:07:03.680 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:03.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:03.682 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:07:03.783 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:03.807 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:03.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:03.809 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:07:03.910 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:07:03.934 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:07:03.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:03.936 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:07:04.037 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:04.059 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:04.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:04.061 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:07:04.162 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:07:04.186 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:07:04.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:04.189 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:07:04.290 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:04.318 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:04.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:04.320 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:07:04.421 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:07:04.445 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:07:04.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:04.447 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:07:04.548 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:04.559 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:04 smithi146 bash[23809]: audit 2024-03-11T21:07:03.585901+0000 mon.a (mon.0) 1590 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:07:04.559 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:04 smithi146 bash[23809]: audit 2024-03-11T21:07:04.274906+0000 mon.a (mon.0) 1591 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:07:04.559 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:04 smithi146 bash[23809]: audit 2024-03-11T21:07:04.276474+0000 mon.a (mon.0) 1592 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:07:04.559 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:04 smithi146 bash[23809]: audit 2024-03-11T21:07:04.288285+0000 mon.a (mon.0) 1593 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:07:04.572 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:04.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:04.574 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:07:04.632 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:04 smithi023 bash[21372]: audit 2024-03-11T21:07:03.585901+0000 mon.a (mon.0) 1590 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:07:04.639 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:04 smithi023 bash[21372]: audit 2024-03-11T21:07:04.274906+0000 mon.a (mon.0) 1591 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:07:04.639 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:04 smithi023 bash[21372]: audit 2024-03-11T21:07:04.276474+0000 mon.a (mon.0) 1592 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:07:04.640 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:04 smithi023 bash[21372]: audit 2024-03-11T21:07:04.288285+0000 mon.a (mon.0) 1593 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:07:04.640 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:04 smithi023 bash[29146]: audit 2024-03-11T21:07:03.585901+0000 mon.a (mon.0) 1590 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:07:04.640 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:04 smithi023 bash[29146]: audit 2024-03-11T21:07:04.274906+0000 mon.a (mon.0) 1591 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:07:04.640 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:04 smithi023 bash[29146]: audit 2024-03-11T21:07:04.276474+0000 mon.a (mon.0) 1592 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:07:04.640 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:04 smithi023 bash[29146]: audit 2024-03-11T21:07:04.288285+0000 mon.a (mon.0) 1593 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:07:04.677 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:04.705 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:04.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:04.707 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:07:04.808 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:07:04.832 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:07:04.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:04.834 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:07:04.862 INFO:teuthology.orchestra.run.smithi023.stdout:ERROR: (22) Invalid argument 2024-03-11T21:07:04.862 INFO:teuthology.orchestra.run.smithi023.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:07:04.935 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:07:04.959 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:07:04.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:04.961 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:07:05.062 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:05.090 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:05.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:05.092 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:07:05.193 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:05.219 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:05.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:05.222 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:07:05.323 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:07:05.348 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:07:05.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:05.351 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:07:05.452 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:07:05.463 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:05 smithi023 bash[21372]: cluster 2024-03-11T21:07:04.238674+0000 mgr.y (mgr.24380) 602 : cluster 0 pgmap v669: 132 pgs: 132 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-11T21:07:05.464 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:05 smithi023 bash[29146]: cluster 2024-03-11T21:07:04.238674+0000 mgr.y (mgr.24380) 602 : cluster 0 pgmap v669: 132 pgs: 132 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-11T21:07:05.480 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:07:05.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:05.483 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:07:05.583 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:07:05.618 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:07:05.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:05.621 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:07:05.638 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:05 smithi146 bash[23809]: cluster 2024-03-11T21:07:04.238674+0000 mgr.y (mgr.24380) 602 : cluster 0 pgmap v669: 132 pgs: 132 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-11T21:07:05.722 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:07:05.750 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:07:05.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:05.752 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:07:05.853 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:05.881 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:05.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:05.883 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:07:05.984 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:05.995 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:07:05 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:07:06.010 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:06.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:06.013 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:07:06.114 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:07:06.142 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:07:06.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:06.145 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:07:06.246 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:07:06.273 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:07:06.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:06.276 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:07:06.376 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:06.403 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:06.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:06.406 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:07:06.507 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:07:06.534 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:07:06.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:06.536 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:07:06.637 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:06.649 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:06 smithi146 bash[23809]: audit 2024-03-11T21:07:05.602623+0000 mon.a (mon.0) 1594 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:07:06.649 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:06 smithi146 bash[23809]: audit 2024-03-11T21:07:05.605199+0000 mon.a (mon.0) 1595 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:07:06.665 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:06.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:06.668 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:07:06.768 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:07:06.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:06 smithi023 bash[21372]: audit 2024-03-11T21:07:05.602623+0000 mon.a (mon.0) 1594 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:07:06.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:06 smithi023 bash[21372]: audit 2024-03-11T21:07:05.605199+0000 mon.a (mon.0) 1595 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:07:06.781 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:06 smithi023 bash[29146]: audit 2024-03-11T21:07:05.602623+0000 mon.a (mon.0) 1594 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:07:06.781 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:06 smithi023 bash[29146]: audit 2024-03-11T21:07:05.605199+0000 mon.a (mon.0) 1595 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:07:06.798 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:07:06.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:06.800 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:07:06.901 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:07:06.930 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:07:06.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:06.932 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:07:07.033 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:07:07.063 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:07:07.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:07.065 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:07:07.166 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:07.193 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:07.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:07.195 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:07:07.296 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:07:07.323 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:07:07.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:07.326 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:07:07.426 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:07:07.453 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:07:07.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:07.455 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:07:07.556 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:07.583 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:07.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:07.585 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:07:07.686 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:07:07.708 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:07:07.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:07.713 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:07:07.721 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-11T21:07:07.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:07 smithi146 bash[23809]: audit 2024-03-11T21:07:05.645258+0000 mgr.y (mgr.24380) 603 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:07:07.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:07 smithi146 bash[23809]: cluster 2024-03-11T21:07:06.240111+0000 mgr.y (mgr.24380) 604 : cluster 0 pgmap v670: 132 pgs: 132 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T21:07:07.813 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:07:07.834 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:07:07.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:07.836 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:07:07.861 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:07 smithi023 bash[21372]: audit 2024-03-11T21:07:05.645258+0000 mgr.y (mgr.24380) 603 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:07:07.862 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:07 smithi023 bash[21372]: cluster 2024-03-11T21:07:06.240111+0000 mgr.y (mgr.24380) 604 : cluster 0 pgmap v670: 132 pgs: 132 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T21:07:07.862 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:07 smithi023 bash[29146]: audit 2024-03-11T21:07:05.645258+0000 mgr.y (mgr.24380) 603 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:07:07.862 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:07 smithi023 bash[29146]: cluster 2024-03-11T21:07:06.240111+0000 mgr.y (mgr.24380) 604 : cluster 0 pgmap v670: 132 pgs: 132 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T21:07:07.937 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:07:07.966 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:07:07.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:07.968 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:07:08.069 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:07:08.094 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:07:08.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:08.096 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:07:08.197 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:08.198 INFO:teuthology.orchestra.run.smithi023.stdout:osd.0: {} 2024-03-11T21:07:08.218 INFO:teuthology.orchestra.run.smithi023.stdout:osd.1: {} 2024-03-11T21:07:08.224 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:08.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:08.227 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:07:08.238 INFO:teuthology.orchestra.run.smithi023.stdout:osd.2: {} 2024-03-11T21:07:08.259 INFO:teuthology.orchestra.run.smithi023.stdout:osd.3: {} 2024-03-11T21:07:08.282 INFO:teuthology.orchestra.run.smithi023.stdout:osd.4: {} 2024-03-11T21:07:08.307 INFO:teuthology.orchestra.run.smithi023.stdout:osd.5: {} 2024-03-11T21:07:08.328 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:08.331 INFO:teuthology.orchestra.run.smithi023.stdout:osd.6: {} 2024-03-11T21:07:08.355 INFO:teuthology.orchestra.run.smithi023.stdout:osd.7: {} 2024-03-11T21:07:08.355 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:08.356 INFO:teuthology.orchestra.run.smithi023.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:07:08.356 INFO:teuthology.orchestra.run.smithi023.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:07:08.356 INFO:teuthology.orchestra.run.smithi023.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:07:08.356 INFO:teuthology.orchestra.run.smithi023.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:07:08.357 INFO:teuthology.orchestra.run.smithi023.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:07:08.357 INFO:teuthology.orchestra.run.smithi023.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:07:08.357 INFO:teuthology.orchestra.run.smithi023.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:07:08.357 INFO:teuthology.orchestra.run.smithi023.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:07:08.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:08.358 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:07:08.459 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:08.484 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:08.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:08.487 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:07:08.587 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:07:08.599 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:07:08 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:07:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:07:08.619 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:07:08.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:08.625 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:07:08.726 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:07:08.738 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:08 smithi146 bash[23809]: cluster 2024-03-11T21:07:07.744958+0000 osd.3 (osd.3) 342 : cluster 0 purged_snaps scrub starts 2024-03-11T21:07:08.738 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:08 smithi146 bash[23809]: cluster 2024-03-11T21:07:07.745035+0000 osd.3 (osd.3) 343 : cluster 0 purged_snaps scrub ok 2024-03-11T21:07:08.738 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:08 smithi146 bash[23809]: audit 2024-03-11T21:07:08.183801+0000 mon.a (mon.0) 1596 : audit 0 from='client.? 172.21.15.23:0/361570030' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:07:08.754 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:07:08.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:08.756 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:07:08.857 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:07:08.868 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:08 smithi023 bash[29146]: cluster 2024-03-11T21:07:07.744958+0000 osd.3 (osd.3) 342 : cluster 0 purged_snaps scrub starts 2024-03-11T21:07:08.868 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:08 smithi023 bash[29146]: cluster 2024-03-11T21:07:07.745035+0000 osd.3 (osd.3) 343 : cluster 0 purged_snaps scrub ok 2024-03-11T21:07:08.869 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:08 smithi023 bash[29146]: audit 2024-03-11T21:07:08.183801+0000 mon.a (mon.0) 1596 : audit 0 from='client.? 172.21.15.23:0/361570030' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:07:08.869 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:08 smithi023 bash[21372]: cluster 2024-03-11T21:07:07.744958+0000 osd.3 (osd.3) 342 : cluster 0 purged_snaps scrub starts 2024-03-11T21:07:08.869 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:08 smithi023 bash[21372]: cluster 2024-03-11T21:07:07.745035+0000 osd.3 (osd.3) 343 : cluster 0 purged_snaps scrub ok 2024-03-11T21:07:08.869 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:08 smithi023 bash[21372]: audit 2024-03-11T21:07:08.183801+0000 mon.a (mon.0) 1596 : audit 0 from='client.? 172.21.15.23:0/361570030' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:07:08.885 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:07:08.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:08.888 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:07:08.989 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:09.016 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:09.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:09.018 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:07:09.119 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:07:09.146 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:07:09.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:09.148 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:07:09.249 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:07:09.275 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:07:09.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:09.277 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:07:09.378 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:09.405 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:09.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:09.408 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:07:09.509 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:09.537 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:09.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:09.539 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:07:09.640 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:07:09.669 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:07:09.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:09.671 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:07:09.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:09 smithi146 bash[23809]: cluster 2024-03-11T21:07:08.240618+0000 mgr.y (mgr.24380) 605 : cluster 0 pgmap v671: 132 pgs: 132 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:07:09.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:09 smithi146 bash[23809]: cluster 2024-03-11T21:07:08.630868+0000 mon.a (mon.0) 1597 : cluster 0 osdmap e303: 8 total, 8 up, 8 in 2024-03-11T21:07:09.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:09 smithi146 bash[23809]: cluster 2024-03-11T21:07:08.637372+0000 osd.1 (osd.1) 125 : cluster 0 purged_snaps scrub starts 2024-03-11T21:07:09.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:09 smithi146 bash[23809]: cluster 2024-03-11T21:07:08.637469+0000 osd.1 (osd.1) 126 : cluster 0 purged_snaps scrub ok 2024-03-11T21:07:09.772 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:07:09.798 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:07:09.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:09.800 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:07:09.858 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:09 smithi023 bash[29146]: cluster 2024-03-11T21:07:08.240618+0000 mgr.y (mgr.24380) 605 : cluster 0 pgmap v671: 132 pgs: 132 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:07:09.858 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:09 smithi023 bash[29146]: cluster 2024-03-11T21:07:08.630868+0000 mon.a (mon.0) 1597 : cluster 0 osdmap e303: 8 total, 8 up, 8 in 2024-03-11T21:07:09.859 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:09 smithi023 bash[29146]: cluster 2024-03-11T21:07:08.637372+0000 osd.1 (osd.1) 125 : cluster 0 purged_snaps scrub starts 2024-03-11T21:07:09.859 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:09 smithi023 bash[29146]: cluster 2024-03-11T21:07:08.637469+0000 osd.1 (osd.1) 126 : cluster 0 purged_snaps scrub ok 2024-03-11T21:07:09.859 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:09 smithi023 bash[21372]: cluster 2024-03-11T21:07:08.240618+0000 mgr.y (mgr.24380) 605 : cluster 0 pgmap v671: 132 pgs: 132 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:07:09.859 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:09 smithi023 bash[21372]: cluster 2024-03-11T21:07:08.630868+0000 mon.a (mon.0) 1597 : cluster 0 osdmap e303: 8 total, 8 up, 8 in 2024-03-11T21:07:09.859 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:09 smithi023 bash[21372]: cluster 2024-03-11T21:07:08.637372+0000 osd.1 (osd.1) 125 : cluster 0 purged_snaps scrub starts 2024-03-11T21:07:09.859 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:09 smithi023 bash[21372]: cluster 2024-03-11T21:07:08.637469+0000 osd.1 (osd.1) 126 : cluster 0 purged_snaps scrub ok 2024-03-11T21:07:09.900 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:09.926 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:09.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:09.929 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:07:10.030 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:10.057 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:10.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:10.060 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:07:10.160 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:07:10.190 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:07:10.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:10.192 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:07:10.293 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:07:10.316 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:07:10.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:10.319 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:07:10.422 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:07:10.450 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:07:10.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:10.452 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:07:10.553 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:10.581 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:10.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:10.583 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:07:10.683 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:07:10.707 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:07:10.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:10.709 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:07:10.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:10 smithi146 bash[23809]: cluster 2024-03-11T21:07:09.652095+0000 mon.a (mon.0) 1598 : cluster 0 osdmap e304: 8 total, 8 up, 8 in 2024-03-11T21:07:10.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:10 smithi146 bash[23809]: cluster 2024-03-11T21:07:10.241439+0000 mgr.y (mgr.24380) 606 : cluster 0 pgmap v674: 132 pgs: 132 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:07:10.810 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:10.838 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:10.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:10.840 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:07:10.941 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:07:10.953 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:10 smithi023 bash[21372]: cluster 2024-03-11T21:07:09.652095+0000 mon.a (mon.0) 1598 : cluster 0 osdmap e304: 8 total, 8 up, 8 in 2024-03-11T21:07:10.953 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:10 smithi023 bash[21372]: cluster 2024-03-11T21:07:10.241439+0000 mgr.y (mgr.24380) 606 : cluster 0 pgmap v674: 132 pgs: 132 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:07:10.953 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:10 smithi023 bash[29146]: cluster 2024-03-11T21:07:09.652095+0000 mon.a (mon.0) 1598 : cluster 0 osdmap e304: 8 total, 8 up, 8 in 2024-03-11T21:07:10.953 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:10 smithi023 bash[29146]: cluster 2024-03-11T21:07:10.241439+0000 mgr.y (mgr.24380) 606 : cluster 0 pgmap v674: 132 pgs: 132 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:07:10.969 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:07:10.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:10.971 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:07:11.072 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:07:11.100 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:07:11.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:11.102 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:07:11.203 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:11.234 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:11.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:11.235 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:07:11.336 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:11.362 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:11.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:11.365 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:07:11.466 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:11.494 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:11.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:11.496 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:07:11.597 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:07:11.625 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:07:11.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:11.627 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:07:11.728 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:11.739 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:11 smithi146 bash[23809]: cluster 2024-03-11T21:07:10.439839+0000 osd.2 (osd.2) 227 : cluster 0 purged_snaps scrub starts 2024-03-11T21:07:11.739 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:11 smithi146 bash[23809]: cluster 2024-03-11T21:07:10.439951+0000 osd.2 (osd.2) 228 : cluster 0 purged_snaps scrub ok 2024-03-11T21:07:11.739 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:11 smithi146 bash[23809]: cluster 2024-03-11T21:07:10.660826+0000 mon.a (mon.0) 1599 : cluster 0 osdmap e305: 8 total, 8 up, 8 in 2024-03-11T21:07:11.754 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:11.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:11.756 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:07:11.857 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:07:11.883 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:07:11.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:11.885 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:07:11.986 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:07:11.995 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:11 smithi023 bash[29146]: cluster 2024-03-11T21:07:10.439839+0000 osd.2 (osd.2) 227 : cluster 0 purged_snaps scrub starts 2024-03-11T21:07:11.995 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:11 smithi023 bash[29146]: cluster 2024-03-11T21:07:10.439951+0000 osd.2 (osd.2) 228 : cluster 0 purged_snaps scrub ok 2024-03-11T21:07:11.995 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:11 smithi023 bash[29146]: cluster 2024-03-11T21:07:10.660826+0000 mon.a (mon.0) 1599 : cluster 0 osdmap e305: 8 total, 8 up, 8 in 2024-03-11T21:07:11.996 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:11 smithi023 bash[21372]: cluster 2024-03-11T21:07:10.439839+0000 osd.2 (osd.2) 227 : cluster 0 purged_snaps scrub starts 2024-03-11T21:07:11.996 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:11 smithi023 bash[21372]: cluster 2024-03-11T21:07:10.439951+0000 osd.2 (osd.2) 228 : cluster 0 purged_snaps scrub ok 2024-03-11T21:07:11.996 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:11 smithi023 bash[21372]: cluster 2024-03-11T21:07:10.660826+0000 mon.a (mon.0) 1599 : cluster 0 osdmap e305: 8 total, 8 up, 8 in 2024-03-11T21:07:12.008 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:07:12.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:12.011 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:07:12.112 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:12.137 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:12.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:12.139 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:07:12.240 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:07:12.267 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:07:12.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:12.269 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:07:12.370 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:07:12.397 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:07:12.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:12.400 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:07:12.501 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:07:12.528 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:07:12.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:12.531 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:07:12.632 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:07:12.654 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T21:07:12.662 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:07:12.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:12.664 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:07:12.765 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:07:12.776 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:12 smithi023 bash[21372]: cluster 2024-03-11T21:07:12.242054+0000 mgr.y (mgr.24380) 607 : cluster 0 pgmap v676: 132 pgs: 132 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:07:12.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:12 smithi023 bash[29146]: cluster 2024-03-11T21:07:12.242054+0000 mgr.y (mgr.24380) 607 : cluster 0 pgmap v676: 132 pgs: 132 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:07:12.789 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:07:12.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:12.791 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:07:12.892 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:07:12.903 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:12 smithi146 bash[23809]: cluster 2024-03-11T21:07:12.242054+0000 mgr.y (mgr.24380) 607 : cluster 0 pgmap v676: 132 pgs: 132 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:07:12.919 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:07:12.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:12.922 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:07:13.022 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:13.049 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:13.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:13.052 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:07:13.152 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:13.176 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:13.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:13.179 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:07:13.279 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:07:13.304 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:07:13.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:13.306 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:07:13.383 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-11T21:07:13.386 INFO:teuthology.orchestra.run.smithi023.stderr:dumped all 2024-03-11T21:07:13.407 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:07:13.426 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:07:13.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:13.428 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:07:13.529 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:07:13.554 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:07:13.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:13.555 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:07:13.656 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:07:13.683 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:07:13.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:13.685 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:07:13.786 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:07:13.810 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:07:13.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:13.812 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:07:13.857 INFO:teuthology.orchestra.run.smithi023.stderr:pool 'unique_pool_3' created 2024-03-11T21:07:13.860 INFO:teuthology.orchestra.run.smithi023.stderr:nodeep-scrub is set 2024-03-11T21:07:13.913 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:07:13.924 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:13 smithi146 bash[23809]: audit 2024-03-11T21:07:13.364573+0000 mon.a (mon.0) 1600 : audit 1 from='client.? 172.21.15.23:0/807158889' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-03-11T21:07:13.925 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:13 smithi146 bash[23809]: audit 2024-03-11T21:07:13.367625+0000 mgr.y (mgr.24380) 608 : audit 0 from='client.15252 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:07:13.925 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:13 smithi146 bash[23809]: audit 2024-03-11T21:07:13.416558+0000 mon.c (mon.2) 77 : audit 1 from='client.? 172.21.15.23:0/509053598' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:07:13.925 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:13 smithi146 bash[23809]: audit 2024-03-11T21:07:13.798040+0000 mon.a (mon.0) 1601 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:07:13.941 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:07:13.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:13.943 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:07:14.044 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:07:14.070 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:07:14.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:14.073 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:07:14.174 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:07:14.185 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:13 smithi023 bash[21372]: audit 2024-03-11T21:07:13.364573+0000 mon.a (mon.0) 1600 : audit 1 from='client.? 172.21.15.23:0/807158889' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-03-11T21:07:14.185 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:13 smithi023 bash[21372]: audit 2024-03-11T21:07:13.367625+0000 mgr.y (mgr.24380) 608 : audit 0 from='client.15252 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:07:14.185 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:13 smithi023 bash[21372]: audit 2024-03-11T21:07:13.416558+0000 mon.c (mon.2) 77 : audit 1 from='client.? 172.21.15.23:0/509053598' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:07:14.185 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:13 smithi023 bash[21372]: audit 2024-03-11T21:07:13.798040+0000 mon.a (mon.0) 1601 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:07:14.185 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:13 smithi023 bash[29146]: audit 2024-03-11T21:07:13.364573+0000 mon.a (mon.0) 1600 : audit 1 from='client.? 172.21.15.23:0/807158889' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-03-11T21:07:14.186 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:13 smithi023 bash[29146]: audit 2024-03-11T21:07:13.367625+0000 mgr.y (mgr.24380) 608 : audit 0 from='client.15252 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:07:14.186 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:13 smithi023 bash[29146]: audit 2024-03-11T21:07:13.416558+0000 mon.c (mon.2) 77 : audit 1 from='client.? 172.21.15.23:0/509053598' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:07:14.186 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:13 smithi023 bash[29146]: audit 2024-03-11T21:07:13.798040+0000 mon.a (mon.0) 1601 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:07:14.201 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:07:14.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:14.204 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:07:14.305 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:07:14.332 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:07:14.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:14.335 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:07:14.427 INFO:teuthology.orchestra.run.smithi023.stdout:{"pg_ready":true,"pg_map":{"version":676,"stamp":"2024-03-11T21:07:12.241646+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596392,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":2220,"num_read_kb":2066,"num_write":901,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":189,"num_bytes_recovered":1775932,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1054,"ondisk_log_size":1054,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2536544,"kb_used_data":8128,"kb_used_omap":21,"kb_used_meta":2528298,"kb_avail":747424672,"statfs":{"total":767960285184,"available":765362864128,"internally_reserved":0,"allocated":8323072,"data_stored":5686743,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22089,"internal_metadata":2588977591},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002721"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":504,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-11T21:07:09.049967+0000","last_change":"2024-03-11T21:04:51.279176+0000","last_active":"2024-03-11T21:07:09.049967+0000","last_peered":"2024-03-11T21:07:09.049967+0000","last_clean":"2024-03-11T21:07:09.049967+0000","last_became_active":"2024-03-11T20:50:34.338390+0000","last_became_peered":"2024-03-11T20:50:34.338390+0000","last_unstale":"2024-03-11T21:07:09.049967+0000","last_undegraded":"2024-03-11T21:07:09.049967+0000","last_fullsized":"2024-03-11T21:07:09.049967+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:04:51.279065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:04:51.279065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":508,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-11T21:07:09.048836+0000","last_change":"2024-03-11T21:04:55.242530+0000","last_active":"2024-03-11T21:07:09.048836+0000","last_peered":"2024-03-11T21:07:09.048836+0000","last_clean":"2024-03-11T21:07:09.048836+0000","last_became_active":"2024-03-11T20:50:32.210194+0000","last_became_peered":"2024-03-11T20:50:32.210194+0000","last_unstale":"2024-03-11T21:07:09.048836+0000","last_undegraded":"2024-03-11T21:07:09.048836+0000","last_fullsized":"2024-03-11T21:07:09.048836+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:04:55.242403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:04:55.242403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":220,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-11T21:07:09.048966+0000","last_change":"2024-03-11T21:06:28.760676+0000","last_active":"2024-03-11T21:07:09.048966+0000","last_peered":"2024-03-11T21:07:09.048966+0000","last_clean":"2024-03-11T21:07:09.048966+0000","last_became_active":"2024-03-11T21:06:28.760306+0000","last_became_peered":"2024-03-11T21:06:28.760306+0000","last_unstale":"2024-03-11T21:07:09.048966+0000","last_undegraded":"2024-03-11T21:07:09.048966+0000","last_fullsized":"2024-03-11T21:07:09.048966+0000","mapping_epoch":297,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:06:15.987233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:53:29.964907+0000","last_clean_scrub_stamp":"2024-03-11T21:06:15.987233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:07:40.884136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"207'20","reported_seq":545,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-11T21:07:09.052998+0000","last_change":"2024-03-11T21:04:56.274776+0000","last_active":"2024-03-11T21:07:09.052998+0000","last_peered":"2024-03-11T21:07:09.052998+0000","last_clean":"2024-03-11T21:07:09.052998+0000","last_became_active":"2024-03-11T20:50:30.101721+0000","last_became_peered":"2024-03-11T20:50:30.101721+0000","last_unstale":"2024-03-11T21:07:09.052998+0000","last_undegraded":"2024-03-11T21:07:09.052998+0000","last_fullsized":"2024-03-11T21:07:09.052998+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'20","last_scrub_stamp":"2024-03-11T21:04:56.274676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:04:56.274676+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":339,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-03-11T21:07:10.051279+0000","last_change":"2024-03-11T21:06:28.005367+0000","last_active":"2024-03-11T21:07:10.051279+0000","last_peered":"2024-03-11T21:07:10.051279+0000","last_clean":"2024-03-11T21:07:10.051279+0000","last_became_active":"2024-03-11T21:05:17.316714+0000","last_became_peered":"2024-03-11T21:05:17.316714+0000","last_unstale":"2024-03-11T21:07:10.051279+0000","last_undegraded":"2024-03-11T21:07:10.051279+0000","last_fullsized":"2024-03-11T21:07:10.051279+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:06:28.005267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:55.545758+0000","last_clean_scrub_stamp":"2024-03-11T21:06:28.005267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:07:29.057359+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":341,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-11T21:07:09.047986+0000","last_change":"2024-03-11T21:06:51.747404+0000","last_active":"2024-03-11T21:07:09.047986+0000","last_peered":"2024-03-11T21:07:09.047986+0000","last_clean":"2024-03-11T21:07:09.047986+0000","last_became_active":"2024-03-11T21:05:17.319272+0000","last_became_peered":"2024-03-11T21:05:17.319272+0000","last_unstale":"2024-03-11T21:07:09.047986+0000","last_undegraded":"2024-03-11T21:07:09.047986+0000","last_fullsized":"2024-03-11T21:07:09.047986+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:06:51.747194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:54:13.660325+0000","last_clean_scrub_stamp":"2024-03-11T21:06:51.747194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:08:15.176787+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":518,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-11T21:07:09.052675+0000","last_change":"2024-03-11T21:06:26.534331+0000","last_active":"2024-03-11T21:07:09.052675+0000","last_peered":"2024-03-11T21:07:09.052675+0000","last_clean":"2024-03-11T21:07:09.052675+0000","last_became_active":"2024-03-11T20:50:28.058274+0000","last_became_peered":"2024-03-11T20:50:28.058274+0000","last_unstale":"2024-03-11T21:07:09.052675+0000","last_undegraded":"2024-03-11T21:07:09.052675+0000","last_fullsized":"2024-03-11T21:07:09.052675+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:06:26.534045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:06:26.534045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:07:50.977042+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"207'19","reported_seq":566,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-11T21:07:09.052065+0000","last_change":"2024-03-11T21:06:21.803105+0000","last_active":"2024-03-11T21:07:09.052065+0000","last_peered":"2024-03-11T21:07:09.052065+0000","last_clean":"2024-03-11T21:07:09.052065+0000","last_became_active":"2024-03-11T20:50:30.092638+0000","last_became_peered":"2024-03-11T20:50:30.092638+0000","last_unstale":"2024-03-11T21:07:09.052065+0000","last_undegraded":"2024-03-11T21:07:09.052065+0000","last_fullsized":"2024-03-11T21:07:09.052065+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'19","last_scrub_stamp":"2024-03-11T21:06:21.802973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:06:21.802973+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:07:33.218740+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":527,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-03-11T21:07:10.051556+0000","last_change":"2024-03-11T21:06:23.997404+0000","last_active":"2024-03-11T21:07:10.051556+0000","last_peered":"2024-03-11T21:07:10.051556+0000","last_clean":"2024-03-11T21:07:10.051556+0000","last_became_active":"2024-03-11T21:00:48.460694+0000","last_became_peered":"2024-03-11T21:00:48.460694+0000","last_unstale":"2024-03-11T21:07:10.051556+0000","last_undegraded":"2024-03-11T21:07:10.051556+0000","last_fullsized":"2024-03-11T21:07:10.051556+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:06:23.997263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:53:07.792697+0000","last_clean_scrub_stamp":"2024-03-11T21:06:23.997263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:07:27.253308+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":333,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-11T21:07:09.050317+0000","last_change":"2024-03-11T21:06:32.574420+0000","last_active":"2024-03-11T21:07:09.050317+0000","last_peered":"2024-03-11T21:07:09.050317+0000","last_clean":"2024-03-11T21:07:09.050317+0000","last_became_active":"2024-03-11T21:05:17.316491+0000","last_became_peered":"2024-03-11T21:05:17.316491+0000","last_unstale":"2024-03-11T21:07:09.050317+0000","last_undegraded":"2024-03-11T21:07:09.050317+0000","last_fullsized":"2024-03-11T21:07:09.050317+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:06:32.574201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:03:36.230647+0000","last_clean_scrub_stamp":"2024-03-11T21:06:32.574201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:07:43.215087+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":530,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-11T21:07:08.732614+0000","last_change":"2024-03-11T21:06:03.794075+0000","last_active":"2024-03-11T21:07:08.732614+0000","last_peered":"2024-03-11T21:07:08.732614+0000","last_clean":"2024-03-11T21:07:08.732614+0000","last_became_active":"2024-03-11T20:50:28.079225+0000","last_became_peered":"2024-03-11T20:50:28.079225+0000","last_unstale":"2024-03-11T21:07:08.732614+0000","last_undegraded":"2024-03-11T21:07:08.732614+0000","last_fullsized":"2024-03-11T21:07:08.732614+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:06:03.793954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:06:03.793954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:07:12.294919+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"207'27","reported_seq":630,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-03-11T21:07:10.056889+0000","last_change":"2024-03-11T21:06:24.914121+0000","last_active":"2024-03-11T21:07:10.056889+0000","last_peered":"2024-03-11T21:07:10.056889+0000","last_clean":"2024-03-11T21:07:10.056889+0000","last_became_active":"2024-03-11T21:00:48.462943+0000","last_became_peered":"2024-03-11T21:00:48.462943+0000","last_unstale":"2024-03-11T21:07:10.056889+0000","last_undegraded":"2024-03-11T21:07:10.056889+0000","last_fullsized":"2024-03-11T21:07:10.056889+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'27","last_scrub_stamp":"2024-03-11T21:06:24.913983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:06:24.913983+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:07:28.575136+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":213,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-11T21:07:09.047652+0000","last_change":"2024-03-11T21:06:28.757644+0000","last_active":"2024-03-11T21:07:09.047652+0000","last_peered":"2024-03-11T21:07:09.047652+0000","last_clean":"2024-03-11T21:07:09.047652+0000","last_became_active":"2024-03-11T21:06:28.757201+0000","last_became_peered":"2024-03-11T21:06:28.757201+0000","last_unstale":"2024-03-11T21:07:09.047652+0000","last_undegraded":"2024-03-11T21:07:09.047652+0000","last_fullsized":"2024-03-11T21:07:09.047652+0000","mapping_epoch":297,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:05:06.011620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:05:06.011620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:07:13.743065+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":295,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-11T21:07:09.049880+0000","last_change":"2024-03-11T21:05:17.316044+0000","last_active":"2024-03-11T21:07:09.049880+0000","last_peered":"2024-03-11T21:07:09.049880+0000","last_clean":"2024-03-11T21:07:09.049880+0000","last_became_active":"2024-03-11T21:05:17.313388+0000","last_became_peered":"2024-03-11T21:05:17.313388+0000","last_unstale":"2024-03-11T21:07:09.049880+0000","last_undegraded":"2024-03-11T21:07:09.049880+0000","last_fullsized":"2024-03-11T21:07:09.049880+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:04:48.510370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:53:50.599029+0000","last_clean_scrub_stamp":"2024-03-11T21:04:48.510370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":538,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-11T21:07:08.731779+0000","last_change":"2024-03-11T21:06:36.121588+0000","last_active":"2024-03-11T21:07:08.731779+0000","last_peered":"2024-03-11T21:07:08.731779+0000","last_clean":"2024-03-11T21:07:08.731779+0000","last_became_active":"2024-03-11T20:50:28.068978+0000","last_became_peered":"2024-03-11T20:50:28.068978+0000","last_unstale":"2024-03-11T21:07:08.731779+0000","last_undegraded":"2024-03-11T21:07:08.731779+0000","last_fullsized":"2024-03-11T21:07:08.731779+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:06:36.121422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:54:07.155130+0000","last_clean_scrub_stamp":"2024-03-11T21:06:36.121422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:07:44.543947+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"207'23","reported_seq":568,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-03-11T21:07:10.057019+0000","last_change":"2024-03-11T21:06:29.044108+0000","last_active":"2024-03-11T21:07:10.057019+0000","last_peered":"2024-03-11T21:07:10.057019+0000","last_clean":"2024-03-11T21:07:10.057019+0000","last_became_active":"2024-03-11T21:00:27.952093+0000","last_became_peered":"2024-03-11T21:00:27.952093+0000","last_unstale":"2024-03-11T21:07:10.057019+0000","last_undegraded":"2024-03-11T21:07:10.057019+0000","last_fullsized":"2024-03-11T21:07:10.057019+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'23","last_scrub_stamp":"2024-03-11T21:06:29.043962+0000","last_deep_scrub":"110'17","last_deep_scrub_stamp":"2024-03-11T20:53:04.802488+0000","last_clean_scrub_stamp":"2024-03-11T21:06:29.043962+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:07:58.373314+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":505,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-11T21:07:09.053215+0000","last_change":"2024-03-11T21:04:58.371455+0000","last_active":"2024-03-11T21:07:09.053215+0000","last_peered":"2024-03-11T21:07:09.053215+0000","last_clean":"2024-03-11T21:07:09.053215+0000","last_became_active":"2024-03-11T20:50:34.334558+0000","last_became_peered":"2024-03-11T20:50:34.334558+0000","last_unstale":"2024-03-11T21:07:09.053215+0000","last_undegraded":"2024-03-11T21:07:09.053215+0000","last_fullsized":"2024-03-11T21:07:09.053215+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:04:58.371326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:55:59.892394+0000","last_clean_scrub_stamp":"2024-03-11T21:04:58.371326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":127,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":358,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-03-11T21:07:10.056603+0000","last_change":"2024-03-11T21:06:17.063473+0000","last_active":"2024-03-11T21:07:10.056603+0000","last_peered":"2024-03-11T21:07:10.056603+0000","last_clean":"2024-03-11T21:07:10.056603+0000","last_became_active":"2024-03-11T21:00:48.461013+0000","last_became_peered":"2024-03-11T21:00:48.461013+0000","last_unstale":"2024-03-11T21:07:10.056603+0000","last_undegraded":"2024-03-11T21:07:10.056603+0000","last_fullsized":"2024-03-11T21:07:10.056603+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:06:17.063294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:06:17.063294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:07:44.378224+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"77'1","reported_seq":524,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-11T21:07:09.050499+0000","last_change":"2024-03-11T21:04:44.146657+0000","last_active":"2024-03-11T21:07:09.050499+0000","last_peered":"2024-03-11T21:07:09.050499+0000","last_clean":"2024-03-11T21:07:09.050499+0000","last_became_active":"2024-03-11T20:50:28.068840+0000","last_became_peered":"2024-03-11T20:50:28.068840+0000","last_unstale":"2024-03-11T21:07:09.050499+0000","last_undegraded":"2024-03-11T21:07:09.050499+0000","last_fullsized":"2024-03-11T21:07:09.050499+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-03-11T21:04:44.142991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:04:44.142991+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"207'19","reported_seq":379,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-03-11T21:07:10.658678+0000","last_change":"2024-03-11T21:06:05.504033+0000","last_active":"2024-03-11T21:07:10.658678+0000","last_peered":"2024-03-11T21:07:10.658678+0000","last_clean":"2024-03-11T21:07:10.658678+0000","last_became_active":"2024-03-11T21:05:17.443030+0000","last_became_peered":"2024-03-11T21:05:17.443030+0000","last_unstale":"2024-03-11T21:07:10.658678+0000","last_undegraded":"2024-03-11T21:07:10.658678+0000","last_fullsized":"2024-03-11T21:07:10.658678+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'19","last_scrub_stamp":"2024-03-11T21:06:05.503839+0000","last_deep_scrub":"207'19","last_deep_scrub_stamp":"2024-03-11T21:03:39.240089+0000","last_clean_scrub_stamp":"2024-03-11T21:06:05.503839+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:07:15.564989+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":297,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-11T21:07:09.049260+0000","last_change":"2024-03-11T21:06:43.968085+0000","last_active":"2024-03-11T21:07:09.049260+0000","last_peered":"2024-03-11T21:07:09.049260+0000","last_clean":"2024-03-11T21:07:09.049260+0000","last_became_active":"2024-03-11T21:05:17.313986+0000","last_became_peered":"2024-03-11T21:05:17.313986+0000","last_unstale":"2024-03-11T21:07:09.049260+0000","last_undegraded":"2024-03-11T21:07:09.049260+0000","last_fullsized":"2024-03-11T21:07:09.049260+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:06:43.967741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:33.399510+0000","last_clean_scrub_stamp":"2024-03-11T21:06:43.967741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:07:46.814001+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":565,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-03-11T21:07:09.651928+0000","last_change":"2024-03-11T21:06:24.068081+0000","last_active":"2024-03-11T21:07:09.651928+0000","last_peered":"2024-03-11T21:07:09.651928+0000","last_clean":"2024-03-11T21:07:09.651928+0000","last_became_active":"2024-03-11T21:00:48.459135+0000","last_became_peered":"2024-03-11T21:00:48.459135+0000","last_unstale":"2024-03-11T21:07:09.651928+0000","last_undegraded":"2024-03-11T21:07:09.651928+0000","last_fullsized":"2024-03-11T21:07:09.651928+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:06:24.067944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:24.666325+0000","last_clean_scrub_stamp":"2024-03-11T21:06:24.067944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:07:44.194649+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":539,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-11T21:07:09.049248+0000","last_change":"2024-03-11T21:06:26.987929+0000","last_active":"2024-03-11T21:07:09.049248+0000","last_peered":"2024-03-11T21:07:09.049248+0000","last_clean":"2024-03-11T21:07:09.049248+0000","last_became_active":"2024-03-11T21:00:27.956935+0000","last_became_peered":"2024-03-11T21:00:27.956935+0000","last_unstale":"2024-03-11T21:07:09.049248+0000","last_undegraded":"2024-03-11T21:07:09.049248+0000","last_fullsized":"2024-03-11T21:07:09.049248+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:06:26.987816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:06:26.987816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:07:42.809815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"207'7","reported_seq":547,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-11T21:07:09.051074+0000","last_change":"2024-03-11T21:06:04.976306+0000","last_active":"2024-03-11T21:07:09.051074+0000","last_peered":"2024-03-11T21:07:09.051074+0000","last_clean":"2024-03-11T21:07:09.051074+0000","last_became_active":"2024-03-11T21:00:27.955296+0000","last_became_peered":"2024-03-11T21:00:27.955296+0000","last_unstale":"2024-03-11T21:07:09.051074+0000","last_undegraded":"2024-03-11T21:07:09.051074+0000","last_fullsized":"2024-03-11T21:07:09.051074+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'7","last_scrub_stamp":"2024-03-11T21:06:04.976055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:06:04.976055+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:07:27.605253+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":235,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-03-11T21:07:10.659308+0000","last_change":"2024-03-11T21:06:08.497752+0000","last_active":"2024-03-11T21:07:10.659308+0000","last_peered":"2024-03-11T21:07:10.659308+0000","last_clean":"2024-03-11T21:07:10.659308+0000","last_became_active":"2024-03-11T21:00:48.464694+0000","last_became_peered":"2024-03-11T21:00:48.464694+0000","last_unstale":"2024-03-11T21:07:10.659308+0000","last_undegraded":"2024-03-11T21:07:10.659308+0000","last_fullsized":"2024-03-11T21:07:10.659308+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:06:08.497227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:06:08.497227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:07:35.840537+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1f","version":"212'16","reported_seq":767,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-11T21:07:09.049096+0000","last_change":"2024-03-11T21:06:47.821325+0000","last_active":"2024-03-11T21:07:09.049096+0000","last_peered":"2024-03-11T21:07:09.049096+0000","last_clean":"2024-03-11T21:07:09.049096+0000","last_became_active":"2024-03-11T21:00:48.464350+0000","last_became_peered":"2024-03-11T21:00:48.464350+0000","last_unstale":"2024-03-11T21:07:09.049096+0000","last_undegraded":"2024-03-11T21:07:09.049096+0000","last_fullsized":"2024-03-11T21:07:09.049096+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"212'16","last_scrub_stamp":"2024-03-11T21:06:47.821221+0000","last_deep_scrub":"212'16","last_deep_scrub_stamp":"2024-03-11T21:06:47.821221+0000","last_clean_scrub_stamp":"2024-03-11T21:06:47.821221+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:08:10.792166+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"77'1","reported_seq":509,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-11T21:07:09.048924+0000","last_change":"2024-03-11T21:04:31.272900+0000","last_active":"2024-03-11T21:07:09.048924+0000","last_peered":"2024-03-11T21:07:09.048924+0000","last_clean":"2024-03-11T21:07:09.048924+0000","last_became_active":"2024-03-11T20:50:28.068827+0000","last_became_peered":"2024-03-11T20:50:28.068827+0000","last_unstale":"2024-03-11T21:07:09.048924+0000","last_undegraded":"2024-03-11T21:07:09.048924+0000","last_fullsized":"2024-03-11T21:07:09.048924+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-03-11T21:04:31.272755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:04:31.272755+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"207'15","reported_seq":579,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-11T21:07:09.048481+0000","last_change":"2024-03-11T21:05:09.233528+0000","last_active":"2024-03-11T21:07:09.048481+0000","last_peered":"2024-03-11T21:07:09.048481+0000","last_clean":"2024-03-11T21:07:09.048481+0000","last_became_active":"2024-03-11T21:00:48.467513+0000","last_became_peered":"2024-03-11T21:00:48.467513+0000","last_unstale":"2024-03-11T21:07:09.048481+0000","last_undegraded":"2024-03-11T21:07:09.048481+0000","last_fullsized":"2024-03-11T21:07:09.048481+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'15","last_scrub_stamp":"2024-03-11T21:05:09.233427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:05:09.233427+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"105'2","reported_seq":521,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-11T21:07:08.732526+0000","last_change":"2024-03-11T21:05:13.702864+0000","last_active":"2024-03-11T21:07:08.732526+0000","last_peered":"2024-03-11T21:07:08.732526+0000","last_clean":"2024-03-11T21:07:08.732526+0000","last_became_active":"2024-03-11T20:50:34.327356+0000","last_became_peered":"2024-03-11T20:50:34.327356+0000","last_unstale":"2024-03-11T21:07:08.732526+0000","last_undegraded":"2024-03-11T21:07:08.732526+0000","last_fullsized":"2024-03-11T21:07:08.732526+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'2","last_scrub_stamp":"2024-03-11T21:05:13.702717+0000","last_deep_scrub":"105'2","last_deep_scrub_stamp":"2024-03-11T21:03:41.971918+0000","last_clean_scrub_stamp":"2024-03-11T21:05:13.702717+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":220,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-11T21:07:09.049776+0000","last_change":"2024-03-11T21:06:32.968730+0000","last_active":"2024-03-11T21:07:09.049776+0000","last_peered":"2024-03-11T21:07:09.049776+0000","last_clean":"2024-03-11T21:07:09.049776+0000","last_became_active":"2024-03-11T21:06:28.759784+0000","last_became_peered":"2024-03-11T21:06:28.759784+0000","last_unstale":"2024-03-11T21:07:09.049776+0000","last_undegraded":"2024-03-11T21:07:09.049776+0000","last_fullsized":"2024-03-11T21:07:09.049776+0000","mapping_epoch":297,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:06:32.968650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:06:32.968650+0000","last_clean_scrub_stamp":"2024-03-11T21:06:32.968650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:07:37.214290+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":535,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-03-11T21:07:10.051765+0000","last_change":"2024-03-11T21:06:46.831529+0000","last_active":"2024-03-11T21:07:10.051765+0000","last_peered":"2024-03-11T21:07:10.051765+0000","last_clean":"2024-03-11T21:07:10.051765+0000","last_became_active":"2024-03-11T21:00:27.947614+0000","last_became_peered":"2024-03-11T21:00:27.947614+0000","last_unstale":"2024-03-11T21:07:10.051765+0000","last_undegraded":"2024-03-11T21:07:10.051765+0000","last_fullsized":"2024-03-11T21:07:10.051765+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:06:46.831406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:05:08.869152+0000","last_clean_scrub_stamp":"2024-03-11T21:06:46.831406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:08:06.425479+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"207'27","reported_seq":273,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-03-11T21:07:10.658055+0000","last_change":"2024-03-11T21:06:45.694196+0000","last_active":"2024-03-11T21:07:10.658055+0000","last_peered":"2024-03-11T21:07:10.658055+0000","last_clean":"2024-03-11T21:07:10.658055+0000","last_became_active":"2024-03-11T21:00:48.457893+0000","last_became_peered":"2024-03-11T21:00:48.457893+0000","last_unstale":"2024-03-11T21:07:10.658055+0000","last_undegraded":"2024-03-11T21:07:10.658055+0000","last_fullsized":"2024-03-11T21:07:10.658055+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'27","last_scrub_stamp":"2024-03-11T21:06:45.693859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:06:45.693859+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:07:48.006023+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":505,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-11T21:07:09.050928+0000","last_change":"2024-03-11T21:04:50.207587+0000","last_active":"2024-03-11T21:07:09.050928+0000","last_peered":"2024-03-11T21:07:09.050928+0000","last_clean":"2024-03-11T21:07:09.050928+0000","last_became_active":"2024-03-11T20:50:34.337878+0000","last_became_peered":"2024-03-11T20:50:34.337878+0000","last_unstale":"2024-03-11T21:07:09.050928+0000","last_undegraded":"2024-03-11T21:07:09.050928+0000","last_fullsized":"2024-03-11T21:07:09.050928+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:04:50.207473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:52:11.462480+0000","last_clean_scrub_stamp":"2024-03-11T21:04:50.207473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":517,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-11T21:07:09.049564+0000","last_change":"2024-03-11T21:06:53.533564+0000","last_active":"2024-03-11T21:07:09.049564+0000","last_peered":"2024-03-11T21:07:09.049564+0000","last_clean":"2024-03-11T21:07:09.049564+0000","last_became_active":"2024-03-11T20:50:32.646323+0000","last_became_peered":"2024-03-11T20:50:32.646323+0000","last_unstale":"2024-03-11T21:07:09.049564+0000","last_undegraded":"2024-03-11T21:07:09.049564+0000","last_fullsized":"2024-03-11T21:07:09.049564+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:06:53.533352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:06:53.533352+0000","last_clean_scrub_stamp":"2024-03-11T21:06:53.533352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:07:58.410631+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":546,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-11T21:07:09.049362+0000","last_change":"2024-03-11T21:06:37.972213+0000","last_active":"2024-03-11T21:07:09.049362+0000","last_peered":"2024-03-11T21:07:09.049362+0000","last_clean":"2024-03-11T21:07:09.049362+0000","last_became_active":"2024-03-11T20:50:28.086513+0000","last_became_peered":"2024-03-11T20:50:28.086513+0000","last_unstale":"2024-03-11T21:07:09.049362+0000","last_undegraded":"2024-03-11T21:07:09.049362+0000","last_fullsized":"2024-03-11T21:07:09.049362+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:06:37.971990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:06:37.971990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:08:07.065988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"207'19","reported_seq":593,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-03-11T21:07:10.057695+0000","last_change":"2024-03-11T21:06:27.015232+0000","last_active":"2024-03-11T21:07:10.057695+0000","last_peered":"2024-03-11T21:07:10.057695+0000","last_clean":"2024-03-11T21:07:10.057695+0000","last_became_active":"2024-03-11T21:00:48.464707+0000","last_became_peered":"2024-03-11T21:00:48.464707+0000","last_unstale":"2024-03-11T21:07:10.057695+0000","last_undegraded":"2024-03-11T21:07:10.057695+0000","last_fullsized":"2024-03-11T21:07:10.057695+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'19","last_scrub_stamp":"2024-03-11T21:06:27.015135+0000","last_deep_scrub":"145'13","last_deep_scrub_stamp":"2024-03-11T20:55:53.560287+0000","last_clean_scrub_stamp":"2024-03-11T21:06:27.015135+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:07:38.662854+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":520,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-11T21:07:08.731449+0000","last_change":"2024-03-11T21:06:00.837953+0000","last_active":"2024-03-11T21:07:08.731449+0000","last_peered":"2024-03-11T21:07:08.731449+0000","last_clean":"2024-03-11T21:07:08.731449+0000","last_became_active":"2024-03-11T21:00:27.973405+0000","last_became_peered":"2024-03-11T21:00:27.973405+0000","last_unstale":"2024-03-11T21:07:08.731449+0000","last_undegraded":"2024-03-11T21:07:08.731449+0000","last_fullsized":"2024-03-11T21:07:08.731449+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:06:00.837723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:06:00.837723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:07:19.562927+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":522,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-11T21:07:09.048453+0000","last_change":"2024-03-11T21:06:01.202252+0000","last_active":"2024-03-11T21:07:09.048453+0000","last_peered":"2024-03-11T21:07:09.048453+0000","last_clean":"2024-03-11T21:07:09.048453+0000","last_became_active":"2024-03-11T21:00:48.472812+0000","last_became_peered":"2024-03-11T21:00:48.472812+0000","last_unstale":"2024-03-11T21:07:09.048453+0000","last_undegraded":"2024-03-11T21:07:09.048453+0000","last_fullsized":"2024-03-11T21:07:09.048453+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:06:01.202148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:06:01.202148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:07:22.072171+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":534,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-03-11T21:07:10.057436+0000","last_change":"2024-03-11T21:06:02.177624+0000","last_active":"2024-03-11T21:07:10.057436+0000","last_peered":"2024-03-11T21:07:10.057436+0000","last_clean":"2024-03-11T21:07:10.057436+0000","last_became_active":"2024-03-11T20:50:28.075760+0000","last_became_peered":"2024-03-11T20:50:28.075760+0000","last_unstale":"2024-03-11T21:07:10.057436+0000","last_undegraded":"2024-03-11T21:07:10.057436+0000","last_fullsized":"2024-03-11T21:07:10.057436+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:06:02.177501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:58:55.977289+0000","last_clean_scrub_stamp":"2024-03-11T21:06:02.177501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:07:12.273327+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"207'12","reported_seq":188,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-11T21:07:09.051688+0000","last_change":"2024-03-11T21:06:28.764470+0000","last_active":"2024-03-11T21:07:09.051688+0000","last_peered":"2024-03-11T21:07:09.051688+0000","last_clean":"2024-03-11T21:07:09.051688+0000","last_became_active":"2024-03-11T21:06:28.763971+0000","last_became_peered":"2024-03-11T21:06:28.763971+0000","last_unstale":"2024-03-11T21:07:09.051688+0000","last_undegraded":"2024-03-11T21:07:09.051688+0000","last_fullsized":"2024-03-11T21:07:09.051688+0000","mapping_epoch":297,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'12","last_scrub_stamp":"2024-03-11T21:06:15.040503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:06:15.040503+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:07:27.449982+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":16,"num_read_kb":13,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.11","version":"186'2","reported_seq":507,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-11T21:07:09.048428+0000","last_change":"2024-03-11T21:04:54.223438+0000","last_active":"2024-03-11T21:07:09.048428+0000","last_peered":"2024-03-11T21:07:09.048428+0000","last_clean":"2024-03-11T21:07:09.048428+0000","last_became_active":"2024-03-11T20:50:34.339326+0000","last_became_peered":"2024-03-11T20:50:34.339326+0000","last_unstale":"2024-03-11T21:07:09.048428+0000","last_undegraded":"2024-03-11T21:07:09.048428+0000","last_fullsized":"2024-03-11T21:07:09.048428+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"186'2","last_scrub_stamp":"2024-03-11T21:04:54.223074+0000","last_deep_scrub":"186'2","last_deep_scrub_stamp":"2024-03-11T20:59:04.534850+0000","last_clean_scrub_stamp":"2024-03-11T21:04:54.223074+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":341,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-11T21:07:09.049166+0000","last_change":"2024-03-11T21:06:05.051766+0000","last_active":"2024-03-11T21:07:09.049166+0000","last_peered":"2024-03-11T21:07:09.049166+0000","last_clean":"2024-03-11T21:07:09.049166+0000","last_became_active":"2024-03-11T21:05:17.314174+0000","last_became_peered":"2024-03-11T21:05:17.314174+0000","last_unstale":"2024-03-11T21:07:09.049166+0000","last_undegraded":"2024-03-11T21:07:09.049166+0000","last_fullsized":"2024-03-11T21:07:09.049166+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:06:05.051636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:06:05.051636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:07:11.713356+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":235,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-03-11T21:07:10.659085+0000","last_change":"2024-03-11T21:06:49.624252+0000","last_active":"2024-03-11T21:07:10.659085+0000","last_peered":"2024-03-11T21:07:10.659085+0000","last_clean":"2024-03-11T21:07:10.659085+0000","last_became_active":"2024-03-11T21:00:48.455510+0000","last_became_peered":"2024-03-11T21:00:48.455510+0000","last_unstale":"2024-03-11T21:07:10.659085+0000","last_undegraded":"2024-03-11T21:07:10.659085+0000","last_fullsized":"2024-03-11T21:07:10.659085+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:06:49.624119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:06:49.624119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:07:50.913463+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"207'25","reported_seq":311,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-11T21:07:08.732062+0000","last_change":"2024-03-11T21:06:10.944686+0000","last_active":"2024-03-11T21:07:08.732062+0000","last_peered":"2024-03-11T21:07:08.732062+0000","last_clean":"2024-03-11T21:07:08.732062+0000","last_became_active":"2024-03-11T21:05:17.314507+0000","last_became_peered":"2024-03-11T21:05:17.314507+0000","last_unstale":"2024-03-11T21:07:08.732062+0000","last_undegraded":"2024-03-11T21:07:08.732062+0000","last_fullsized":"2024-03-11T21:07:08.732062+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'25","last_scrub_stamp":"2024-03-11T21:06:10.944546+0000","last_deep_scrub":"161'15","last_deep_scrub_stamp":"2024-03-11T20:57:15.125548+0000","last_clean_scrub_stamp":"2024-03-11T21:06:10.944546+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:07:22.123477+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":539,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-11T21:07:09.049332+0000","last_change":"2024-03-11T21:06:05.984545+0000","last_active":"2024-03-11T21:07:09.049332+0000","last_peered":"2024-03-11T21:07:09.049332+0000","last_clean":"2024-03-11T21:07:09.049332+0000","last_became_active":"2024-03-11T21:00:48.462105+0000","last_became_peered":"2024-03-11T21:00:48.462105+0000","last_unstale":"2024-03-11T21:07:09.049332+0000","last_undegraded":"2024-03-11T21:07:09.049332+0000","last_fullsized":"2024-03-11T21:07:09.049332+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:06:05.984419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:06:05.984419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:07:30.930515+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":235,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-03-11T21:07:10.658507+0000","last_change":"2024-03-11T21:06:22.817666+0000","last_active":"2024-03-11T21:07:10.658507+0000","last_peered":"2024-03-11T21:07:10.658507+0000","last_clean":"2024-03-11T21:07:10.658507+0000","last_became_active":"2024-03-11T21:00:48.459043+0000","last_became_peered":"2024-03-11T21:00:48.459043+0000","last_unstale":"2024-03-11T21:07:10.658507+0000","last_undegraded":"2024-03-11T21:07:10.658507+0000","last_fullsized":"2024-03-11T21:07:10.658507+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:06:22.817543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:23.137832+0000","last_clean_scrub_stamp":"2024-03-11T21:06:22.817543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:07:46.013197+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":539,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-11T21:07:09.048530+0000","last_change":"2024-03-11T21:04:59.819489+0000","last_active":"2024-03-11T21:07:09.048530+0000","last_peered":"2024-03-11T21:07:09.048530+0000","last_clean":"2024-03-11T21:07:09.048530+0000","last_became_active":"2024-03-11T20:50:28.082737+0000","last_became_peered":"2024-03-11T20:50:28.082737+0000","last_unstale":"2024-03-11T21:07:09.048530+0000","last_undegraded":"2024-03-11T21:07:09.048530+0000","last_fullsized":"2024-03-11T21:07:09.048530+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:04:59.819359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:28.404997+0000","last_clean_scrub_stamp":"2024-03-11T21:04:59.819359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"207'15","reported_seq":552,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-11T21:07:08.732909+0000","last_change":"2024-03-11T21:06:13.947437+0000","last_active":"2024-03-11T21:07:08.732909+0000","last_peered":"2024-03-11T21:07:08.732909+0000","last_clean":"2024-03-11T21:07:08.732909+0000","last_became_active":"2024-03-11T21:00:27.967624+0000","last_became_peered":"2024-03-11T21:00:27.967624+0000","last_unstale":"2024-03-11T21:07:08.732909+0000","last_undegraded":"2024-03-11T21:07:08.732909+0000","last_fullsized":"2024-03-11T21:07:08.732909+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'15","last_scrub_stamp":"2024-03-11T21:06:13.947349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:06:13.947349+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:07:41.353356+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":259,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-03-11T21:07:09.653430+0000","last_change":"2024-03-11T21:06:02.414808+0000","last_active":"2024-03-11T21:07:09.653430+0000","last_peered":"2024-03-11T21:07:09.653430+0000","last_clean":"2024-03-11T21:07:09.653430+0000","last_became_active":"2024-03-11T21:05:17.428544+0000","last_became_peered":"2024-03-11T21:05:17.428544+0000","last_unstale":"2024-03-11T21:07:09.653430+0000","last_undegraded":"2024-03-11T21:07:09.653430+0000","last_fullsized":"2024-03-11T21:07:09.653430+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:06:02.414533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:57:16.032302+0000","last_clean_scrub_stamp":"2024-03-11T21:06:02.414533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:07:14.453050+0000","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"112'12","reported_seq":730,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-03-11T21:07:10.057284+0000","last_change":"2024-03-11T21:06:11.050064+0000","last_active":"2024-03-11T21:07:10.057284+0000","last_peered":"2024-03-11T21:07:10.057284+0000","last_clean":"2024-03-11T21:07:10.057284+0000","last_became_active":"2024-03-11T20:50:32.650064+0000","last_became_peered":"2024-03-11T20:50:32.650064+0000","last_unstale":"2024-03-11T21:07:10.057284+0000","last_undegraded":"2024-03-11T21:07:10.057284+0000","last_fullsized":"2024-03-11T21:07:10.057284+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'12","last_scrub_stamp":"2024-03-11T21:06:11.049954+0000","last_deep_scrub":"112'12","last_deep_scrub_stamp":"2024-03-11T20:53:11.843951+0000","last_clean_scrub_stamp":"2024-03-11T21:06:11.049954+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:07:32.971452+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":235,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-11T21:07:09.048496+0000","last_change":"2024-03-11T21:06:52.863103+0000","last_active":"2024-03-11T21:07:09.048496+0000","last_peered":"2024-03-11T21:07:09.048496+0000","last_clean":"2024-03-11T21:07:09.048496+0000","last_became_active":"2024-03-11T21:06:28.756041+0000","last_became_peered":"2024-03-11T21:06:28.756041+0000","last_unstale":"2024-03-11T21:07:09.048496+0000","last_undegraded":"2024-03-11T21:07:09.048496+0000","last_fullsized":"2024-03-11T21:07:09.048496+0000","mapping_epoch":297,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:06:52.862971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:06:52.862971+0000","last_clean_scrub_stamp":"2024-03-11T21:06:52.862971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:08:20.522422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"207'19","reported_seq":286,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-11T21:07:09.047666+0000","last_change":"2024-03-11T21:06:45.887284+0000","last_active":"2024-03-11T21:07:09.047666+0000","last_peered":"2024-03-11T21:07:09.047666+0000","last_clean":"2024-03-11T21:07:09.047666+0000","last_became_active":"2024-03-11T21:06:28.758223+0000","last_became_peered":"2024-03-11T21:06:28.758223+0000","last_unstale":"2024-03-11T21:07:09.047666+0000","last_undegraded":"2024-03-11T21:07:09.047666+0000","last_fullsized":"2024-03-11T21:07:09.047666+0000","mapping_epoch":297,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'19","last_scrub_stamp":"2024-03-11T21:06:45.887055+0000","last_deep_scrub":"119'13","last_deep_scrub_stamp":"2024-03-11T20:54:15.443472+0000","last_clean_scrub_stamp":"2024-03-11T21:06:45.887055+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:08:04.084738+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":207,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-11T21:07:09.049340+0000","last_change":"2024-03-11T21:06:28.760549+0000","last_active":"2024-03-11T21:07:09.049340+0000","last_peered":"2024-03-11T21:07:09.049340+0000","last_clean":"2024-03-11T21:07:09.049340+0000","last_became_active":"2024-03-11T21:06:28.758634+0000","last_became_peered":"2024-03-11T21:06:28.758634+0000","last_unstale":"2024-03-11T21:07:09.049340+0000","last_undegraded":"2024-03-11T21:07:09.049340+0000","last_fullsized":"2024-03-11T21:07:09.049340+0000","mapping_epoch":297,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:05:05.664042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:05:05.664042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:07:13.743153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"212'17","reported_seq":752,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-11T21:07:09.052742+0000","last_change":"2024-03-11T21:05:06.153853+0000","last_active":"2024-03-11T21:07:09.052742+0000","last_peered":"2024-03-11T21:07:09.052742+0000","last_clean":"2024-03-11T21:07:09.052742+0000","last_became_active":"2024-03-11T21:00:48.467116+0000","last_became_peered":"2024-03-11T21:00:48.467116+0000","last_unstale":"2024-03-11T21:07:09.052742+0000","last_undegraded":"2024-03-11T21:07:09.052742+0000","last_fullsized":"2024-03-11T21:07:09.052742+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"212'17","last_scrub_stamp":"2024-03-11T21:05:06.153737+0000","last_deep_scrub":"110'13","last_deep_scrub_stamp":"2024-03-11T20:53:00.434532+0000","last_clean_scrub_stamp":"2024-03-11T21:05:06.153737+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":535,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-03-11T21:07:10.054502+0000","last_change":"2024-03-11T21:06:04.112492+0000","last_active":"2024-03-11T21:07:10.054502+0000","last_peered":"2024-03-11T21:07:10.054502+0000","last_clean":"2024-03-11T21:07:10.054502+0000","last_became_active":"2024-03-11T20:50:28.070325+0000","last_became_peered":"2024-03-11T20:50:28.070325+0000","last_unstale":"2024-03-11T21:07:10.054502+0000","last_undegraded":"2024-03-11T21:07:10.054502+0000","last_fullsized":"2024-03-11T21:07:10.054502+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:06:04.112381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:06:04.112381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:07:23.159452+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"207'24","reported_seq":564,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-11T21:07:08.731653+0000","last_change":"2024-03-11T21:06:12.914676+0000","last_active":"2024-03-11T21:07:08.731653+0000","last_peered":"2024-03-11T21:07:08.731653+0000","last_clean":"2024-03-11T21:07:08.731653+0000","last_became_active":"2024-03-11T21:00:27.967078+0000","last_became_peered":"2024-03-11T21:00:27.967078+0000","last_unstale":"2024-03-11T21:07:08.731653+0000","last_undegraded":"2024-03-11T21:07:08.731653+0000","last_fullsized":"2024-03-11T21:07:08.731653+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'24","last_scrub_stamp":"2024-03-11T21:06:12.914355+0000","last_deep_scrub":"116'16","last_deep_scrub_stamp":"2024-03-11T20:53:39.292118+0000","last_clean_scrub_stamp":"2024-03-11T21:06:12.914355+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:07:16.298472+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":511,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-11T21:07:08.731576+0000","last_change":"2024-03-11T21:06:04.835538+0000","last_active":"2024-03-11T21:07:08.731576+0000","last_peered":"2024-03-11T21:07:08.731576+0000","last_clean":"2024-03-11T21:07:08.731576+0000","last_became_active":"2024-03-11T21:00:27.966997+0000","last_became_peered":"2024-03-11T21:00:27.966997+0000","last_unstale":"2024-03-11T21:07:08.731576+0000","last_undegraded":"2024-03-11T21:07:08.731576+0000","last_fullsized":"2024-03-11T21:07:08.731576+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:06:04.835427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:55:58.016988+0000","last_clean_scrub_stamp":"2024-03-11T21:06:04.835427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:07:14.300980+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":529,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-11T21:07:09.053368+0000","last_change":"2024-03-11T21:06:16.515576+0000","last_active":"2024-03-11T21:07:09.053368+0000","last_peered":"2024-03-11T21:07:09.053368+0000","last_clean":"2024-03-11T21:07:09.053368+0000","last_became_active":"2024-03-11T21:00:48.468386+0000","last_became_peered":"2024-03-11T21:00:48.468386+0000","last_unstale":"2024-03-11T21:07:09.053368+0000","last_undegraded":"2024-03-11T21:07:09.053368+0000","last_fullsized":"2024-03-11T21:07:09.053368+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:06:16.515117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:06:16.515117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:07:40.102483+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"166'2","reported_seq":574,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-11T21:07:09.047717+0000","last_change":"2024-03-11T21:06:39.971394+0000","last_active":"2024-03-11T21:07:09.047717+0000","last_peered":"2024-03-11T21:07:09.047717+0000","last_clean":"2024-03-11T21:07:09.047717+0000","last_became_active":"2024-03-11T21:00:48.466071+0000","last_became_peered":"2024-03-11T21:00:48.466071+0000","last_unstale":"2024-03-11T21:07:09.047717+0000","last_undegraded":"2024-03-11T21:07:09.047717+0000","last_fullsized":"2024-03-11T21:07:09.047717+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'2","last_scrub_stamp":"2024-03-11T21:06:39.971160+0000","last_deep_scrub":"166'2","last_deep_scrub_stamp":"2024-03-11T20:57:39.868776+0000","last_clean_scrub_stamp":"2024-03-11T21:06:39.971160+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:07:54.280554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1822,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"207'8","reported_seq":539,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-11T21:07:09.047581+0000","last_change":"2024-03-11T21:06:17.992450+0000","last_active":"2024-03-11T21:07:09.047581+0000","last_peered":"2024-03-11T21:07:09.047581+0000","last_clean":"2024-03-11T21:07:09.047581+0000","last_became_active":"2024-03-11T20:50:30.101572+0000","last_became_peered":"2024-03-11T20:50:30.101572+0000","last_unstale":"2024-03-11T21:07:09.047581+0000","last_undegraded":"2024-03-11T21:07:09.047581+0000","last_fullsized":"2024-03-11T21:07:09.047581+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'8","last_scrub_stamp":"2024-03-11T21:06:17.992309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:06:17.992309+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:07:39.118549+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":541,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-03-11T21:07:09.653136+0000","last_change":"2024-03-11T21:06:36.909780+0000","last_active":"2024-03-11T21:07:09.653136+0000","last_peered":"2024-03-11T21:07:09.653136+0000","last_clean":"2024-03-11T21:07:09.653136+0000","last_became_active":"2024-03-11T21:00:27.964140+0000","last_became_peered":"2024-03-11T21:00:27.964140+0000","last_unstale":"2024-03-11T21:07:09.653136+0000","last_undegraded":"2024-03-11T21:07:09.653136+0000","last_fullsized":"2024-03-11T21:07:09.653136+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:06:36.909571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:06:36.909571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:07:40.594835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":233,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-11T21:07:09.048893+0000","last_change":"2024-03-11T21:06:44.905488+0000","last_active":"2024-03-11T21:07:09.048893+0000","last_peered":"2024-03-11T21:07:09.048893+0000","last_clean":"2024-03-11T21:07:09.048893+0000","last_became_active":"2024-03-11T21:06:28.758133+0000","last_became_peered":"2024-03-11T21:06:28.758133+0000","last_unstale":"2024-03-11T21:07:09.048893+0000","last_undegraded":"2024-03-11T21:07:09.048893+0000","last_fullsized":"2024-03-11T21:07:09.048893+0000","mapping_epoch":297,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:06:44.905349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:54:09.537972+0000","last_clean_scrub_stamp":"2024-03-11T21:06:44.905349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:07:52.693074+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":554,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-03-11T21:07:09.648594+0000","last_change":"2024-03-11T21:06:12.235882+0000","last_active":"2024-03-11T21:07:09.648594+0000","last_peered":"2024-03-11T21:07:09.648594+0000","last_clean":"2024-03-11T21:07:09.648594+0000","last_became_active":"2024-03-11T21:00:48.459557+0000","last_became_peered":"2024-03-11T21:00:48.459557+0000","last_unstale":"2024-03-11T21:07:09.648594+0000","last_undegraded":"2024-03-11T21:07:09.648594+0000","last_fullsized":"2024-03-11T21:07:09.648594+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:06:12.235794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:18.682947+0000","last_clean_scrub_stamp":"2024-03-11T21:06:12.235794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:07:27.774582+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"262'24","reported_seq":564,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-11T21:07:08.730744+0000","last_change":"2024-03-11T21:06:52.270836+0000","last_active":"2024-03-11T21:07:08.730744+0000","last_peered":"2024-03-11T21:07:08.730744+0000","last_clean":"2024-03-11T21:07:08.730744+0000","last_became_active":"2024-03-11T21:00:27.970878+0000","last_became_peered":"2024-03-11T21:00:27.970878+0000","last_unstale":"2024-03-11T21:07:08.730744+0000","last_undegraded":"2024-03-11T21:07:08.730744+0000","last_fullsized":"2024-03-11T21:07:08.730744+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"262'24","last_scrub_stamp":"2024-03-11T21:06:52.270681+0000","last_deep_scrub":"262'24","last_deep_scrub_stamp":"2024-03-11T21:03:28.128719+0000","last_clean_scrub_stamp":"2024-03-11T21:06:52.270681+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:08:01.103138+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":534,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-11T21:07:09.050789+0000","last_change":"2024-03-11T21:06:47.558351+0000","last_active":"2024-03-11T21:07:09.050789+0000","last_peered":"2024-03-11T21:07:09.050789+0000","last_clean":"2024-03-11T21:07:09.050789+0000","last_became_active":"2024-03-11T21:00:48.466853+0000","last_became_peered":"2024-03-11T21:00:48.466853+0000","last_unstale":"2024-03-11T21:07:09.050789+0000","last_undegraded":"2024-03-11T21:07:09.050789+0000","last_fullsized":"2024-03-11T21:07:09.050789+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:06:47.558214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:52:53.450885+0000","last_clean_scrub_stamp":"2024-03-11T21:06:47.558214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:07:54.369398+0000","scrub_duration":95,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"77'2","reported_seq":31,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-11T21:07:09.049434+0000","last_change":"2024-03-11T21:06:28.760531+0000","last_active":"2024-03-11T21:07:09.049434+0000","last_peered":"2024-03-11T21:07:09.049434+0000","last_clean":"2024-03-11T21:07:09.049434+0000","last_became_active":"2024-03-11T21:06:28.759838+0000","last_became_peered":"2024-03-11T21:06:28.759838+0000","last_unstale":"2024-03-11T21:07:09.049434+0000","last_undegraded":"2024-03-11T21:07:09.049434+0000","last_fullsized":"2024-03-11T21:07:09.049434+0000","mapping_epoch":297,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'2","last_scrub_stamp":"2024-03-11T21:06:14.979075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:06:14.979075+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:07:31.330432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"207'24","reported_seq":624,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-11T21:07:08.731967+0000","last_change":"2024-03-11T21:06:41.235642+0000","last_active":"2024-03-11T21:07:08.731967+0000","last_peered":"2024-03-11T21:07:08.731967+0000","last_clean":"2024-03-11T21:07:08.731967+0000","last_became_active":"2024-03-11T21:00:48.465254+0000","last_became_peered":"2024-03-11T21:00:48.465254+0000","last_unstale":"2024-03-11T21:07:08.731967+0000","last_undegraded":"2024-03-11T21:07:08.731967+0000","last_fullsized":"2024-03-11T21:07:08.731967+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'24","last_scrub_stamp":"2024-03-11T21:06:41.235422+0000","last_deep_scrub":"100'16","last_deep_scrub_stamp":"2024-03-11T20:56:08.036331+0000","last_clean_scrub_stamp":"2024-03-11T21:06:41.235422+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:07:58.085154+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"286'18","reported_seq":461,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-03-11T21:07:10.657880+0000","last_change":"2024-03-11T21:06:06.430872+0000","last_active":"2024-03-11T21:07:10.657880+0000","last_peered":"2024-03-11T21:07:10.657880+0000","last_clean":"2024-03-11T21:07:10.657880+0000","last_became_active":"2024-03-11T21:05:17.317915+0000","last_became_peered":"2024-03-11T21:05:17.317915+0000","last_unstale":"2024-03-11T21:07:10.657880+0000","last_undegraded":"2024-03-11T21:07:10.657880+0000","last_fullsized":"2024-03-11T21:07:10.657880+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"286'18","last_scrub_stamp":"2024-03-11T21:06:06.430748+0000","last_deep_scrub":"212'17","last_deep_scrub_stamp":"2024-03-11T21:03:38.258850+0000","last_clean_scrub_stamp":"2024-03-11T21:06:06.430748+0000","objects_scrubbed":1,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:07:18.070057+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":216,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-11T21:07:09.047629+0000","last_change":"2024-03-11T21:06:46.969281+0000","last_active":"2024-03-11T21:07:09.047629+0000","last_peered":"2024-03-11T21:07:09.047629+0000","last_clean":"2024-03-11T21:07:09.047629+0000","last_became_active":"2024-03-11T21:06:28.755689+0000","last_became_peered":"2024-03-11T21:06:28.755689+0000","last_unstale":"2024-03-11T21:07:09.047629+0000","last_undegraded":"2024-03-11T21:07:09.047629+0000","last_fullsized":"2024-03-11T21:07:09.047629+0000","mapping_epoch":297,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:06:46.969140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:57:21.568400+0000","last_clean_scrub_stamp":"2024-03-11T21:06:46.969140+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:07:59.046083+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":236,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-03-11T21:07:10.659096+0000","last_change":"2024-03-11T21:06:17.889312+0000","last_active":"2024-03-11T21:07:10.659096+0000","last_peered":"2024-03-11T21:07:10.659096+0000","last_clean":"2024-03-11T21:07:10.659096+0000","last_became_active":"2024-03-11T21:00:48.455733+0000","last_became_peered":"2024-03-11T21:00:48.455733+0000","last_unstale":"2024-03-11T21:07:10.659096+0000","last_undegraded":"2024-03-11T21:07:10.659096+0000","last_fullsized":"2024-03-11T21:07:10.659096+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:06:17.889154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:20.096874+0000","last_clean_scrub_stamp":"2024-03-11T21:06:17.889154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:07:32.960980+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"207'28","reported_seq":765,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-03-11T21:07:10.057112+0000","last_change":"2024-03-11T21:05:10.942104+0000","last_active":"2024-03-11T21:07:10.057112+0000","last_peered":"2024-03-11T21:07:10.057112+0000","last_clean":"2024-03-11T21:07:10.057112+0000","last_became_active":"2024-03-11T20:50:30.094977+0000","last_became_peered":"2024-03-11T20:50:30.094977+0000","last_unstale":"2024-03-11T21:07:10.057112+0000","last_undegraded":"2024-03-11T21:07:10.057112+0000","last_fullsized":"2024-03-11T21:07:10.057112+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'28","last_scrub_stamp":"2024-03-11T21:05:10.941968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:05:10.941968+0000","objects_scrubbed":8,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":236,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-03-11T21:07:10.658380+0000","last_change":"2024-03-11T21:06:25.845901+0000","last_active":"2024-03-11T21:07:10.658380+0000","last_peered":"2024-03-11T21:07:10.658380+0000","last_clean":"2024-03-11T21:07:10.658380+0000","last_became_active":"2024-03-11T21:00:48.457901+0000","last_became_peered":"2024-03-11T21:00:48.457901+0000","last_unstale":"2024-03-11T21:07:10.658380+0000","last_undegraded":"2024-03-11T21:07:10.658380+0000","last_fullsized":"2024-03-11T21:07:10.658380+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:06:25.845791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:52:10.160202+0000","last_clean_scrub_stamp":"2024-03-11T21:06:25.845791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:07:33.329280+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":514,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-11T21:07:09.052689+0000","last_change":"2024-03-11T21:06:25.518117+0000","last_active":"2024-03-11T21:07:09.052689+0000","last_peered":"2024-03-11T21:07:09.052689+0000","last_clean":"2024-03-11T21:07:09.052689+0000","last_became_active":"2024-03-11T21:00:27.959408+0000","last_became_peered":"2024-03-11T21:00:27.959408+0000","last_unstale":"2024-03-11T21:07:09.052689+0000","last_undegraded":"2024-03-11T21:07:09.052689+0000","last_fullsized":"2024-03-11T21:07:09.052689+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:06:25.518004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:06:25.518004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:07:47.062377+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":549,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-03-11T21:07:10.054774+0000","last_change":"2024-03-11T21:06:42.906241+0000","last_active":"2024-03-11T21:07:10.054774+0000","last_peered":"2024-03-11T21:07:10.054774+0000","last_clean":"2024-03-11T21:07:10.054774+0000","last_became_active":"2024-03-11T21:00:48.471212+0000","last_became_peered":"2024-03-11T21:00:48.471212+0000","last_unstale":"2024-03-11T21:07:10.054774+0000","last_undegraded":"2024-03-11T21:07:10.054774+0000","last_fullsized":"2024-03-11T21:07:10.054774+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:06:42.906028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:33.823704+0000","last_clean_scrub_stamp":"2024-03-11T21:06:42.906028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:08:00.492117+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"207'37","reported_seq":355,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-11T21:07:09.048162+0000","last_change":"2024-03-11T21:06:28.757392+0000","last_active":"2024-03-11T21:07:09.048162+0000","last_peered":"2024-03-11T21:07:09.048162+0000","last_clean":"2024-03-11T21:07:09.048162+0000","last_became_active":"2024-03-11T21:06:28.756995+0000","last_became_peered":"2024-03-11T21:06:28.756995+0000","last_unstale":"2024-03-11T21:07:09.048162+0000","last_undegraded":"2024-03-11T21:07:09.048162+0000","last_fullsized":"2024-03-11T21:07:09.048162+0000","mapping_epoch":297,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'37","last_scrub_stamp":"2024-03-11T21:04:49.513751+0000","last_deep_scrub":"182'27","last_deep_scrub_stamp":"2024-03-11T20:58:52.058851+0000","last_clean_scrub_stamp":"2024-03-11T21:04:49.513751+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:07:13.743013+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":236,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-03-11T21:07:10.658004+0000","last_change":"2024-03-11T21:06:19.855978+0000","last_active":"2024-03-11T21:07:10.658004+0000","last_peered":"2024-03-11T21:07:10.658004+0000","last_clean":"2024-03-11T21:07:10.658004+0000","last_became_active":"2024-03-11T21:00:48.458008+0000","last_became_peered":"2024-03-11T21:00:48.458008+0000","last_unstale":"2024-03-11T21:07:10.658004+0000","last_undegraded":"2024-03-11T21:07:10.658004+0000","last_fullsized":"2024-03-11T21:07:10.658004+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:06:19.855856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:33.319221+0000","last_clean_scrub_stamp":"2024-03-11T21:06:19.855856+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:07:23.419113+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":519,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-11T21:07:08.731590+0000","last_change":"2024-03-11T21:06:29.141103+0000","last_active":"2024-03-11T21:07:08.731590+0000","last_peered":"2024-03-11T21:07:08.731590+0000","last_clean":"2024-03-11T21:07:08.731590+0000","last_became_active":"2024-03-11T20:50:34.321038+0000","last_became_peered":"2024-03-11T20:50:34.321038+0000","last_unstale":"2024-03-11T21:07:08.731590+0000","last_undegraded":"2024-03-11T21:07:08.731590+0000","last_fullsized":"2024-03-11T21:07:08.731590+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:06:29.140971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:32.951104+0000","last_clean_scrub_stamp":"2024-03-11T21:06:29.140971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:07:49.086045+0000","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"207'38","reported_seq":268,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-03-11T21:07:10.657689+0000","last_change":"2024-03-11T21:06:51.624094+0000","last_active":"2024-03-11T21:07:10.657689+0000","last_peered":"2024-03-11T21:07:10.657689+0000","last_clean":"2024-03-11T21:07:10.657689+0000","last_became_active":"2024-03-11T21:00:48.456178+0000","last_became_peered":"2024-03-11T21:00:48.456178+0000","last_unstale":"2024-03-11T21:07:10.657689+0000","last_undegraded":"2024-03-11T21:07:10.657689+0000","last_fullsized":"2024-03-11T21:07:10.657689+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'38","last_scrub_stamp":"2024-03-11T21:06:51.623848+0000","last_deep_scrub":"109'26","last_deep_scrub_stamp":"2024-03-11T20:52:49.216378+0000","last_clean_scrub_stamp":"2024-03-11T21:06:51.623848+0000","objects_scrubbed":8,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:07:54.209917+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":550,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-03-11T21:07:09.649597+0000","last_change":"2024-03-11T21:06:33.936507+0000","last_active":"2024-03-11T21:07:09.649597+0000","last_peered":"2024-03-11T21:07:09.649597+0000","last_clean":"2024-03-11T21:07:09.649597+0000","last_became_active":"2024-03-11T20:50:28.045625+0000","last_became_peered":"2024-03-11T20:50:28.045625+0000","last_unstale":"2024-03-11T21:07:09.649597+0000","last_undegraded":"2024-03-11T21:07:09.649597+0000","last_fullsized":"2024-03-11T21:07:09.649597+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:06:33.936406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:06:33.936406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:07:37.966038+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":227,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-03-11T21:07:10.659386+0000","last_change":"2024-03-11T21:05:02.894444+0000","last_active":"2024-03-11T21:07:10.659386+0000","last_peered":"2024-03-11T21:07:10.659386+0000","last_clean":"2024-03-11T21:07:10.659386+0000","last_became_active":"2024-03-11T21:00:48.473346+0000","last_became_peered":"2024-03-11T21:00:48.473346+0000","last_unstale":"2024-03-11T21:07:10.659386+0000","last_undegraded":"2024-03-11T21:07:10.659386+0000","last_fullsized":"2024-03-11T21:07:10.659386+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:05:02.894369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:05:02.894369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":516,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-03-11T21:07:10.057967+0000","last_change":"2024-03-11T21:05:09.952496+0000","last_active":"2024-03-11T21:07:10.057967+0000","last_peered":"2024-03-11T21:07:10.057967+0000","last_clean":"2024-03-11T21:07:10.057967+0000","last_became_active":"2024-03-11T21:00:27.962668+0000","last_became_peered":"2024-03-11T21:00:27.962668+0000","last_unstale":"2024-03-11T21:07:10.057967+0000","last_undegraded":"2024-03-11T21:07:10.057967+0000","last_fullsized":"2024-03-11T21:07:10.057967+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:05:09.952413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:05:09.952413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"284'25","reported_seq":549,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-11T21:07:09.050181+0000","last_change":"2024-03-11T21:04:53.217494+0000","last_active":"2024-03-11T21:07:09.050181+0000","last_peered":"2024-03-11T21:07:09.050181+0000","last_clean":"2024-03-11T21:07:09.050181+0000","last_became_active":"2024-03-11T20:50:30.101747+0000","last_became_peered":"2024-03-11T20:50:30.101747+0000","last_unstale":"2024-03-11T21:07:09.050181+0000","last_undegraded":"2024-03-11T21:07:09.050181+0000","last_fullsized":"2024-03-11T21:07:09.050181+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"284'25","last_scrub_stamp":"2024-03-11T21:04:53.217188+0000","last_deep_scrub":"284'25","last_deep_scrub_stamp":"2024-03-11T21:04:53.217188+0000","last_clean_scrub_stamp":"2024-03-11T21:04:53.217188+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:07:14.441728+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":536,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-03-11T21:07:10.054888+0000","last_change":"2024-03-11T21:06:19.075727+0000","last_active":"2024-03-11T21:07:10.054888+0000","last_peered":"2024-03-11T21:07:10.054888+0000","last_clean":"2024-03-11T21:07:10.054888+0000","last_became_active":"2024-03-11T20:50:28.085503+0000","last_became_peered":"2024-03-11T20:50:28.085503+0000","last_unstale":"2024-03-11T21:07:10.054888+0000","last_undegraded":"2024-03-11T21:07:10.054888+0000","last_fullsized":"2024-03-11T21:07:10.054888+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:06:19.075654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:56:03.479180+0000","last_clean_scrub_stamp":"2024-03-11T21:06:19.075654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:07:27.472135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"184'130","reported_seq":800,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-11T21:07:08.730928+0000","last_change":"2024-03-11T21:06:27.008858+0000","last_active":"2024-03-11T21:07:08.730928+0000","last_peered":"2024-03-11T21:07:08.730928+0000","last_clean":"2024-03-11T21:07:08.730928+0000","last_became_active":"2024-03-11T20:49:52.584543+0000","last_became_peered":"2024-03-11T20:49:52.584543+0000","last_unstale":"2024-03-11T21:07:08.730928+0000","last_undegraded":"2024-03-11T21:07:08.730928+0000","last_fullsized":"2024-03-11T21:07:08.730928+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"184'130","last_scrub_stamp":"2024-03-11T21:06:27.008698+0000","last_deep_scrub":"184'130","last_deep_scrub_stamp":"2024-03-11T20:58:56.415267+0000","last_clean_scrub_stamp":"2024-03-11T21:06:27.008698+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:07:32.807388+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":540,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-11T21:07:09.048803+0000","last_change":"2024-03-11T21:06:38.936544+0000","last_active":"2024-03-11T21:07:09.048803+0000","last_peered":"2024-03-11T21:07:09.048803+0000","last_clean":"2024-03-11T21:07:09.048803+0000","last_became_active":"2024-03-11T21:00:27.959467+0000","last_became_peered":"2024-03-11T21:00:27.959467+0000","last_unstale":"2024-03-11T21:07:09.048803+0000","last_undegraded":"2024-03-11T21:07:09.048803+0000","last_fullsized":"2024-03-11T21:07:09.048803+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:06:38.936445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:54:22.448812+0000","last_clean_scrub_stamp":"2024-03-11T21:06:38.936445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:07:51.279867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":236,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-03-11T21:07:10.658925+0000","last_change":"2024-03-11T21:06:41.763723+0000","last_active":"2024-03-11T21:07:10.658925+0000","last_peered":"2024-03-11T21:07:10.658925+0000","last_clean":"2024-03-11T21:07:10.658925+0000","last_became_active":"2024-03-11T21:00:48.464596+0000","last_became_peered":"2024-03-11T21:00:48.464596+0000","last_unstale":"2024-03-11T21:07:10.658925+0000","last_undegraded":"2024-03-11T21:07:10.658925+0000","last_fullsized":"2024-03-11T21:07:10.658925+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:06:41.763442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:06:41.763442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:08:03.621444+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"207'35","reported_seq":212,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-11T21:07:09.048166+0000","last_change":"2024-03-11T21:06:49.897492+0000","last_active":"2024-03-11T21:07:09.048166+0000","last_peered":"2024-03-11T21:07:09.048166+0000","last_clean":"2024-03-11T21:07:09.048166+0000","last_became_active":"2024-03-11T21:06:28.761186+0000","last_became_peered":"2024-03-11T21:06:28.761186+0000","last_unstale":"2024-03-11T21:07:09.048166+0000","last_undegraded":"2024-03-11T21:07:09.048166+0000","last_fullsized":"2024-03-11T21:07:09.048166+0000","mapping_epoch":297,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'35","last_scrub_stamp":"2024-03-11T21:06:49.897272+0000","last_deep_scrub":"100'21","last_deep_scrub_stamp":"2024-03-11T20:51:31.835302+0000","last_clean_scrub_stamp":"2024-03-11T21:06:49.897272+0000","objects_scrubbed":7,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:08:11.681353+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":520,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-03-11T21:07:09.649103+0000","last_change":"2024-03-11T21:06:42.945752+0000","last_active":"2024-03-11T21:07:09.649103+0000","last_peered":"2024-03-11T21:07:09.649103+0000","last_clean":"2024-03-11T21:07:09.649103+0000","last_became_active":"2024-03-11T21:00:48.458579+0000","last_became_peered":"2024-03-11T21:00:48.458579+0000","last_unstale":"2024-03-11T21:07:09.649103+0000","last_undegraded":"2024-03-11T21:07:09.649103+0000","last_fullsized":"2024-03-11T21:07:09.649103+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:06:42.945665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:06:42.945665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:07:48.277174+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":543,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-11T21:07:09.051891+0000","last_change":"2024-03-11T21:06:14.987279+0000","last_active":"2024-03-11T21:07:09.051891+0000","last_peered":"2024-03-11T21:07:09.051891+0000","last_clean":"2024-03-11T21:07:09.051891+0000","last_became_active":"2024-03-11T21:00:48.463247+0000","last_became_peered":"2024-03-11T21:00:48.463247+0000","last_unstale":"2024-03-11T21:07:09.051891+0000","last_undegraded":"2024-03-11T21:07:09.051891+0000","last_fullsized":"2024-03-11T21:07:09.051891+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:06:14.986987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:26.839782+0000","last_clean_scrub_stamp":"2024-03-11T21:06:14.986987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:07:43.966158+0000","scrub_duration":87,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":236,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-03-11T21:07:10.658226+0000","last_change":"2024-03-11T21:05:16.075970+0000","last_active":"2024-03-11T21:07:10.658226+0000","last_peered":"2024-03-11T21:07:10.658226+0000","last_clean":"2024-03-11T21:07:10.658226+0000","last_became_active":"2024-03-11T21:00:48.469688+0000","last_became_peered":"2024-03-11T21:00:48.469688+0000","last_unstale":"2024-03-11T21:07:10.658226+0000","last_undegraded":"2024-03-11T21:07:10.658226+0000","last_fullsized":"2024-03-11T21:07:10.658226+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:05:16.075797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:05:16.075797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"145'3","reported_seq":534,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-11T21:07:09.049185+0000","last_change":"2024-03-11T21:04:52.756712+0000","last_active":"2024-03-11T21:07:09.049185+0000","last_peered":"2024-03-11T21:07:09.049185+0000","last_clean":"2024-03-11T21:07:09.049185+0000","last_became_active":"2024-03-11T20:50:32.195452+0000","last_became_peered":"2024-03-11T20:50:32.195452+0000","last_unstale":"2024-03-11T21:07:09.049185+0000","last_undegraded":"2024-03-11T21:07:09.049185+0000","last_fullsized":"2024-03-11T21:07:09.049185+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'3","last_scrub_stamp":"2024-03-11T21:04:52.756603+0000","last_deep_scrub":"145'3","last_deep_scrub_stamp":"2024-03-11T20:55:52.848043+0000","last_clean_scrub_stamp":"2024-03-11T21:04:52.756603+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:07:14.713060+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0.0024049620000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":532,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-11T21:07:08.732120+0000","last_change":"2024-03-11T21:06:28.042306+0000","last_active":"2024-03-11T21:07:08.732120+0000","last_peered":"2024-03-11T21:07:08.732120+0000","last_clean":"2024-03-11T21:07:08.732120+0000","last_became_active":"2024-03-11T21:00:27.966749+0000","last_became_peered":"2024-03-11T21:00:27.966749+0000","last_unstale":"2024-03-11T21:07:08.732120+0000","last_undegraded":"2024-03-11T21:07:08.732120+0000","last_fullsized":"2024-03-11T21:07:08.732120+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:06:28.042198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:06:28.042198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:07:42.369255+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"212'46","reported_seq":342,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-03-11T21:07:10.659091+0000","last_change":"2024-03-11T21:05:12.034872+0000","last_active":"2024-03-11T21:07:10.659091+0000","last_peered":"2024-03-11T21:07:10.659091+0000","last_clean":"2024-03-11T21:07:10.659091+0000","last_became_active":"2024-03-11T21:00:48.473227+0000","last_became_peered":"2024-03-11T21:00:48.473227+0000","last_unstale":"2024-03-11T21:07:10.659091+0000","last_undegraded":"2024-03-11T21:07:10.659091+0000","last_fullsized":"2024-03-11T21:07:10.659091+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"212'46","last_scrub_stamp":"2024-03-11T21:05:12.034714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:05:12.034714+0000","objects_scrubbed":10,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":69,"num_read_kb":54,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"298'13","reported_seq":311,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-11T21:07:09.047776+0000","last_change":"2024-03-11T21:06:28.763326+0000","last_active":"2024-03-11T21:07:09.047776+0000","last_peered":"2024-03-11T21:07:09.047776+0000","last_clean":"2024-03-11T21:07:09.047776+0000","last_became_active":"2024-03-11T21:06:28.762397+0000","last_became_peered":"2024-03-11T21:06:28.762397+0000","last_unstale":"2024-03-11T21:07:09.047776+0000","last_undegraded":"2024-03-11T21:07:09.047776+0000","last_fullsized":"2024-03-11T21:07:09.047776+0000","mapping_epoch":297,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"160'12","last_scrub_stamp":"2024-03-11T21:06:18.041380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:06:18.041380+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:07:33.607582+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":259,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-11T21:07:09.050359+0000","last_change":"2024-03-11T21:06:31.535147+0000","last_active":"2024-03-11T21:07:09.050359+0000","last_peered":"2024-03-11T21:07:09.050359+0000","last_clean":"2024-03-11T21:07:09.050359+0000","last_became_active":"2024-03-11T21:05:17.441521+0000","last_became_peered":"2024-03-11T21:05:17.441521+0000","last_unstale":"2024-03-11T21:07:09.050359+0000","last_undegraded":"2024-03-11T21:07:09.050359+0000","last_fullsized":"2024-03-11T21:07:09.050359+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:06:31.535021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:03:43.345257+0000","last_clean_scrub_stamp":"2024-03-11T21:06:31.535021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:07:43.687816+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":280,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-11T21:07:09.049476+0000","last_change":"2024-03-11T21:06:44.388965+0000","last_active":"2024-03-11T21:07:09.049476+0000","last_peered":"2024-03-11T21:07:09.049476+0000","last_clean":"2024-03-11T21:07:09.049476+0000","last_became_active":"2024-03-11T21:05:17.315844+0000","last_became_peered":"2024-03-11T21:05:17.315844+0000","last_unstale":"2024-03-11T21:07:09.049476+0000","last_undegraded":"2024-03-11T21:07:09.049476+0000","last_fullsized":"2024-03-11T21:07:09.049476+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:06:44.388832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:22.617999+0000","last_clean_scrub_stamp":"2024-03-11T21:06:44.388832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:07:45.686204+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"284'2","reported_seq":240,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-03-11T21:07:10.659674+0000","last_change":"2024-03-11T21:06:47.645929+0000","last_active":"2024-03-11T21:07:10.659674+0000","last_peered":"2024-03-11T21:07:10.659674+0000","last_clean":"2024-03-11T21:07:10.659674+0000","last_became_active":"2024-03-11T21:00:48.468682+0000","last_became_peered":"2024-03-11T21:00:48.468682+0000","last_unstale":"2024-03-11T21:07:10.659674+0000","last_undegraded":"2024-03-11T21:07:10.659674+0000","last_fullsized":"2024-03-11T21:07:10.659674+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"284'2","last_scrub_stamp":"2024-03-11T21:06:47.645682+0000","last_deep_scrub":"284'2","last_deep_scrub_stamp":"2024-03-11T21:06:47.645682+0000","last_clean_scrub_stamp":"2024-03-11T21:06:47.645682+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:07:53.144959+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"207'17","reported_seq":546,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-11T21:07:09.053024+0000","last_change":"2024-03-11T21:06:41.476100+0000","last_active":"2024-03-11T21:07:09.053024+0000","last_peered":"2024-03-11T21:07:09.053024+0000","last_clean":"2024-03-11T21:07:09.053024+0000","last_became_active":"2024-03-11T20:50:30.097669+0000","last_became_peered":"2024-03-11T20:50:30.097669+0000","last_unstale":"2024-03-11T21:07:09.053024+0000","last_undegraded":"2024-03-11T21:07:09.053024+0000","last_fullsized":"2024-03-11T21:07:09.053024+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'17","last_scrub_stamp":"2024-03-11T21:06:41.475870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:06:41.475870+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:07:55.611411+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"84'11","reported_seq":729,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-11T21:07:09.048826+0000","last_change":"2024-03-11T21:04:52.176137+0000","last_active":"2024-03-11T21:07:09.048826+0000","last_peered":"2024-03-11T21:07:09.048826+0000","last_clean":"2024-03-11T21:07:09.048826+0000","last_became_active":"2024-03-11T20:50:32.647041+0000","last_became_peered":"2024-03-11T20:50:32.647041+0000","last_unstale":"2024-03-11T21:07:09.048826+0000","last_undegraded":"2024-03-11T21:07:09.048826+0000","last_fullsized":"2024-03-11T21:07:09.048826+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:04:52.175881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:04:52.175881+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":527,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-11T21:07:08.731304+0000","last_change":"2024-03-11T21:06:43.271663+0000","last_active":"2024-03-11T21:07:08.731304+0000","last_peered":"2024-03-11T21:07:08.731304+0000","last_clean":"2024-03-11T21:07:08.731304+0000","last_became_active":"2024-03-11T20:50:34.326947+0000","last_became_peered":"2024-03-11T20:50:34.326947+0000","last_unstale":"2024-03-11T21:07:08.731304+0000","last_undegraded":"2024-03-11T21:07:08.731304+0000","last_fullsized":"2024-03-11T21:07:08.731304+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:06:43.271453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:06:43.271453+0000","last_clean_scrub_stamp":"2024-03-11T21:06:43.271453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:07:56.724552+0000","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"120'6","reported_seq":1423,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-11T21:07:09.049208+0000","last_change":"2024-03-11T21:05:03.181822+0000","last_active":"2024-03-11T21:07:09.049208+0000","last_peered":"2024-03-11T21:07:09.049208+0000","last_clean":"2024-03-11T21:07:09.049208+0000","last_became_active":"2024-03-11T21:00:48.468997+0000","last_became_peered":"2024-03-11T21:00:48.468997+0000","last_unstale":"2024-03-11T21:07:09.049208+0000","last_undegraded":"2024-03-11T21:07:09.049208+0000","last_fullsized":"2024-03-11T21:07:09.049208+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"120'6","last_scrub_stamp":"2024-03-11T21:05:03.181697+0000","last_deep_scrub":"120'6","last_deep_scrub_stamp":"2024-03-11T20:54:26.468057+0000","last_clean_scrub_stamp":"2024-03-11T21:05:03.181697+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0035533370000000002,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":864,"num_read_kb":859,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":539,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-11T21:07:09.049154+0000","last_change":"2024-03-11T21:04:51.700881+0000","last_active":"2024-03-11T21:07:09.049154+0000","last_peered":"2024-03-11T21:07:09.049154+0000","last_clean":"2024-03-11T21:07:09.049154+0000","last_became_active":"2024-03-11T20:50:28.071707+0000","last_became_peered":"2024-03-11T20:50:28.071707+0000","last_unstale":"2024-03-11T21:07:09.049154+0000","last_undegraded":"2024-03-11T21:07:09.049154+0000","last_fullsized":"2024-03-11T21:07:09.049154+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:04:51.700746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:23.417283+0000","last_clean_scrub_stamp":"2024-03-11T21:04:51.700746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"207'24","reported_seq":631,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-11T21:07:09.051441+0000","last_change":"2024-03-11T21:06:23.850668+0000","last_active":"2024-03-11T21:07:09.051441+0000","last_peered":"2024-03-11T21:07:09.051441+0000","last_clean":"2024-03-11T21:07:09.051441+0000","last_became_active":"2024-03-11T21:00:48.461972+0000","last_became_peered":"2024-03-11T21:00:48.461972+0000","last_unstale":"2024-03-11T21:07:09.051441+0000","last_undegraded":"2024-03-11T21:07:09.051441+0000","last_fullsized":"2024-03-11T21:07:09.051441+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'24","last_scrub_stamp":"2024-03-11T21:06:23.850544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:06:23.850544+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:07:24.484162+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":303,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-03-11T21:07:10.051395+0000","last_change":"2024-03-11T21:06:31.992255+0000","last_active":"2024-03-11T21:07:10.051395+0000","last_peered":"2024-03-11T21:07:10.051395+0000","last_clean":"2024-03-11T21:07:10.051395+0000","last_became_active":"2024-03-11T21:05:17.315807+0000","last_became_peered":"2024-03-11T21:05:17.315807+0000","last_unstale":"2024-03-11T21:07:10.051395+0000","last_undegraded":"2024-03-11T21:07:10.051395+0000","last_fullsized":"2024-03-11T21:07:10.051395+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:06:31.992056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:06:31.992056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:07:47.274231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":505,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-11T21:07:09.051819+0000","last_change":"2024-03-11T21:06:30.558401+0000","last_active":"2024-03-11T21:07:09.051819+0000","last_peered":"2024-03-11T21:07:09.051819+0000","last_clean":"2024-03-11T21:07:09.051819+0000","last_became_active":"2024-03-11T20:50:34.335370+0000","last_became_peered":"2024-03-11T20:50:34.335370+0000","last_unstale":"2024-03-11T21:07:09.051819+0000","last_undegraded":"2024-03-11T21:07:09.051819+0000","last_fullsized":"2024-03-11T21:07:09.051819+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:06:30.558258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:06:30.558258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:07:47.720090+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"286'18","reported_seq":141,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-11T21:07:09.049422+0000","last_change":"2024-03-11T21:05:17.449952+0000","last_active":"2024-03-11T21:07:09.049422+0000","last_peered":"2024-03-11T21:07:09.049422+0000","last_clean":"2024-03-11T21:07:09.049422+0000","last_became_active":"2024-03-11T21:05:17.449213+0000","last_became_peered":"2024-03-11T21:05:17.449213+0000","last_unstale":"2024-03-11T21:07:09.049422+0000","last_undegraded":"2024-03-11T21:07:09.049422+0000","last_fullsized":"2024-03-11T21:07:09.049422+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"249'17","last_scrub_stamp":"2024-03-11T21:05:09.913682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:05:09.913682+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":544,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-11T21:07:08.730998+0000","last_change":"2024-03-11T21:06:42.243027+0000","last_active":"2024-03-11T21:07:08.730998+0000","last_peered":"2024-03-11T21:07:08.730998+0000","last_clean":"2024-03-11T21:07:08.730998+0000","last_became_active":"2024-03-11T21:00:48.466102+0000","last_became_peered":"2024-03-11T21:00:48.466102+0000","last_unstale":"2024-03-11T21:07:08.730998+0000","last_undegraded":"2024-03-11T21:07:08.730998+0000","last_fullsized":"2024-03-11T21:07:08.730998+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:06:42.242801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:06:42.242801+0000","last_clean_scrub_stamp":"2024-03-11T21:06:42.242801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:07:59.098752+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"207'18","reported_seq":313,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-03-11T21:07:09.649343+0000","last_change":"2024-03-11T21:06:32.921142+0000","last_active":"2024-03-11T21:07:09.649343+0000","last_peered":"2024-03-11T21:07:09.649343+0000","last_clean":"2024-03-11T21:07:09.649343+0000","last_became_active":"2024-03-11T21:05:17.428742+0000","last_became_peered":"2024-03-11T21:05:17.428742+0000","last_unstale":"2024-03-11T21:07:09.649343+0000","last_undegraded":"2024-03-11T21:07:09.649343+0000","last_fullsized":"2024-03-11T21:07:09.649343+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'18","last_scrub_stamp":"2024-03-11T21:06:32.921043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:06:32.921043+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:07:44.793581+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":23,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":536,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-03-11T21:07:10.057268+0000","last_change":"2024-03-11T21:06:38.053998+0000","last_active":"2024-03-11T21:07:10.057268+0000","last_peered":"2024-03-11T21:07:10.057268+0000","last_clean":"2024-03-11T21:07:10.057268+0000","last_became_active":"2024-03-11T21:00:48.460531+0000","last_became_peered":"2024-03-11T21:00:48.460531+0000","last_unstale":"2024-03-11T21:07:10.057268+0000","last_undegraded":"2024-03-11T21:07:10.057268+0000","last_fullsized":"2024-03-11T21:07:10.057268+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:06:38.053772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:27.775534+0000","last_clean_scrub_stamp":"2024-03-11T21:06:38.053772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:07:51.184629+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":243,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-03-11T21:07:10.659326+0000","last_change":"2024-03-11T21:06:42.772823+0000","last_active":"2024-03-11T21:07:10.659326+0000","last_peered":"2024-03-11T21:07:10.659326+0000","last_clean":"2024-03-11T21:07:10.659326+0000","last_became_active":"2024-03-11T21:00:48.469596+0000","last_became_peered":"2024-03-11T21:00:48.469596+0000","last_unstale":"2024-03-11T21:07:10.659326+0000","last_undegraded":"2024-03-11T21:07:10.659326+0000","last_fullsized":"2024-03-11T21:07:10.659326+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:06:42.772624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:06:42.772624+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:08:01.652524+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":227,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-03-11T21:07:10.658231+0000","last_change":"2024-03-11T21:05:09.001323+0000","last_active":"2024-03-11T21:07:10.658231+0000","last_peered":"2024-03-11T21:07:10.658231+0000","last_clean":"2024-03-11T21:07:10.658231+0000","last_became_active":"2024-03-11T21:00:48.466084+0000","last_became_peered":"2024-03-11T21:00:48.466084+0000","last_unstale":"2024-03-11T21:07:10.658231+0000","last_undegraded":"2024-03-11T21:07:10.658231+0000","last_fullsized":"2024-03-11T21:07:10.658231+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:05:09.001185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:40.974147+0000","last_clean_scrub_stamp":"2024-03-11T21:05:09.001185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"207'34","reported_seq":623,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-11T21:07:09.048742+0000","last_change":"2024-03-11T21:06:11.347952+0000","last_active":"2024-03-11T21:07:09.048742+0000","last_peered":"2024-03-11T21:07:09.048742+0000","last_clean":"2024-03-11T21:07:09.048742+0000","last_became_active":"2024-03-11T21:00:48.472191+0000","last_became_peered":"2024-03-11T21:00:48.472191+0000","last_unstale":"2024-03-11T21:07:09.048742+0000","last_undegraded":"2024-03-11T21:07:09.048742+0000","last_fullsized":"2024-03-11T21:07:09.048742+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'34","last_scrub_stamp":"2024-03-11T21:06:11.347860+0000","last_deep_scrub":"116'22","last_deep_scrub_stamp":"2024-03-11T20:53:47.364709+0000","last_clean_scrub_stamp":"2024-03-11T21:06:11.347860+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:07:31.716188+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":314,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-03-11T21:07:10.659661+0000","last_change":"2024-03-11T21:06:18.848362+0000","last_active":"2024-03-11T21:07:10.659661+0000","last_peered":"2024-03-11T21:07:10.659661+0000","last_clean":"2024-03-11T21:07:10.659661+0000","last_became_active":"2024-03-11T21:05:17.455236+0000","last_became_peered":"2024-03-11T21:05:17.455236+0000","last_unstale":"2024-03-11T21:07:10.659661+0000","last_undegraded":"2024-03-11T21:07:10.659661+0000","last_fullsized":"2024-03-11T21:07:10.659661+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:06:18.848225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:06:18.848225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:07:39.837386+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":235,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-03-11T21:07:10.658755+0000","last_change":"2024-03-11T21:06:12.536259+0000","last_active":"2024-03-11T21:07:10.658755+0000","last_peered":"2024-03-11T21:07:10.658755+0000","last_clean":"2024-03-11T21:07:10.658755+0000","last_became_active":"2024-03-11T21:00:48.461367+0000","last_became_peered":"2024-03-11T21:00:48.461367+0000","last_unstale":"2024-03-11T21:07:10.658755+0000","last_undegraded":"2024-03-11T21:07:10.658755+0000","last_fullsized":"2024-03-11T21:07:10.658755+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:06:12.536058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:03:37.601229+0000","last_clean_scrub_stamp":"2024-03-11T21:06:12.536058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:07:30.007155+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"267'19","reported_seq":551,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-11T21:07:09.050556+0000","last_change":"2024-03-11T21:06:19.516422+0000","last_active":"2024-03-11T21:07:09.050556+0000","last_peered":"2024-03-11T21:07:09.050556+0000","last_clean":"2024-03-11T21:07:09.050556+0000","last_became_active":"2024-03-11T21:00:27.967572+0000","last_became_peered":"2024-03-11T21:00:27.967572+0000","last_unstale":"2024-03-11T21:07:09.050556+0000","last_undegraded":"2024-03-11T21:07:09.050556+0000","last_fullsized":"2024-03-11T21:07:09.050556+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"267'19","last_scrub_stamp":"2024-03-11T21:06:19.516296+0000","last_deep_scrub":"267'19","last_deep_scrub_stamp":"2024-03-11T21:03:39.673317+0000","last_clean_scrub_stamp":"2024-03-11T21:06:19.516296+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:07:30.664815+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":303,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-11T21:07:09.052498+0000","last_change":"2024-03-11T21:05:17.316687+0000","last_active":"2024-03-11T21:07:09.052498+0000","last_peered":"2024-03-11T21:07:09.052498+0000","last_clean":"2024-03-11T21:07:09.052498+0000","last_became_active":"2024-03-11T21:05:17.316136+0000","last_became_peered":"2024-03-11T21:05:17.316136+0000","last_unstale":"2024-03-11T21:07:09.052498+0000","last_undegraded":"2024-03-11T21:07:09.052498+0000","last_fullsized":"2024-03-11T21:07:09.052498+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:04:43.625264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:35.670149+0000","last_clean_scrub_stamp":"2024-03-11T21:04:43.625264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":538,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-03-11T21:07:09.653342+0000","last_change":"2024-03-11T21:06:06.419472+0000","last_active":"2024-03-11T21:07:09.653342+0000","last_peered":"2024-03-11T21:07:09.653342+0000","last_clean":"2024-03-11T21:07:09.653342+0000","last_became_active":"2024-03-11T21:00:27.959830+0000","last_became_peered":"2024-03-11T21:00:27.959830+0000","last_unstale":"2024-03-11T21:07:09.653342+0000","last_undegraded":"2024-03-11T21:07:09.653342+0000","last_fullsized":"2024-03-11T21:07:09.653342+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:06:06.419249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:06:06.419249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:07:12.287112+0000","scrub_duration":159,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"286'20","reported_seq":383,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-03-11T21:07:09.653472+0000","last_change":"2024-03-11T21:06:35.941070+0000","last_active":"2024-03-11T21:07:09.653472+0000","last_peered":"2024-03-11T21:07:09.653472+0000","last_clean":"2024-03-11T21:07:09.653472+0000","last_became_active":"2024-03-11T21:05:17.428970+0000","last_became_peered":"2024-03-11T21:05:17.428970+0000","last_unstale":"2024-03-11T21:07:09.653472+0000","last_undegraded":"2024-03-11T21:07:09.653472+0000","last_fullsized":"2024-03-11T21:07:09.653472+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"286'20","last_scrub_stamp":"2024-03-11T21:06:35.940967+0000","last_deep_scrub":"146'14","last_deep_scrub_stamp":"2024-03-11T20:56:00.242211+0000","last_clean_scrub_stamp":"2024-03-11T21:06:35.940967+0000","objects_scrubbed":1,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:08:02.967330+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":532,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-11T21:07:08.731577+0000","last_change":"2024-03-11T21:06:18.003215+0000","last_active":"2024-03-11T21:07:08.731577+0000","last_peered":"2024-03-11T21:07:08.731577+0000","last_clean":"2024-03-11T21:07:08.731577+0000","last_became_active":"2024-03-11T21:00:27.970573+0000","last_became_peered":"2024-03-11T21:00:27.970573+0000","last_unstale":"2024-03-11T21:07:08.731577+0000","last_undegraded":"2024-03-11T21:07:08.731577+0000","last_fullsized":"2024-03-11T21:07:08.731577+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:06:18.003086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:52:51.161281+0000","last_clean_scrub_stamp":"2024-03-11T21:06:18.003086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:07:32.102645+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"207'44","reported_seq":676,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-11T21:07:09.049257+0000","last_change":"2024-03-11T21:06:23.973658+0000","last_active":"2024-03-11T21:07:09.049257+0000","last_peered":"2024-03-11T21:07:09.049257+0000","last_clean":"2024-03-11T21:07:09.049257+0000","last_became_active":"2024-03-11T21:00:48.469854+0000","last_became_peered":"2024-03-11T21:00:48.469854+0000","last_unstale":"2024-03-11T21:07:09.049257+0000","last_undegraded":"2024-03-11T21:07:09.049257+0000","last_fullsized":"2024-03-11T21:07:09.049257+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'44","last_scrub_stamp":"2024-03-11T21:06:23.973513+0000","last_deep_scrub":"161'28","last_deep_scrub_stamp":"2024-03-11T20:57:20.816278+0000","last_clean_scrub_stamp":"2024-03-11T21:06:23.973513+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:07:47.170354+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":519,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-11T21:07:08.730590+0000","last_change":"2024-03-11T21:06:32.052246+0000","last_active":"2024-03-11T21:07:08.730590+0000","last_peered":"2024-03-11T21:07:08.730590+0000","last_clean":"2024-03-11T21:07:08.730590+0000","last_became_active":"2024-03-11T20:50:34.327139+0000","last_became_peered":"2024-03-11T20:50:34.327139+0000","last_unstale":"2024-03-11T21:07:08.730590+0000","last_undegraded":"2024-03-11T21:07:08.730590+0000","last_fullsized":"2024-03-11T21:07:08.730590+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:06:32.052041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:06:32.052041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:07:36.523508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":549,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-03-11T21:07:09.651624+0000","last_change":"2024-03-11T21:06:05.281706+0000","last_active":"2024-03-11T21:07:09.651624+0000","last_peered":"2024-03-11T21:07:09.651624+0000","last_clean":"2024-03-11T21:07:09.651624+0000","last_became_active":"2024-03-11T20:50:28.049901+0000","last_became_peered":"2024-03-11T20:50:28.049901+0000","last_unstale":"2024-03-11T21:07:09.651624+0000","last_undegraded":"2024-03-11T21:07:09.651624+0000","last_fullsized":"2024-03-11T21:07:09.651624+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:06:05.281621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:57:17.447191+0000","last_clean_scrub_stamp":"2024-03-11T21:06:05.281621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"207'39","reported_seq":584,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-11T21:07:08.732759+0000","last_change":"2024-03-11T21:05:14.688119+0000","last_active":"2024-03-11T21:07:08.732759+0000","last_peered":"2024-03-11T21:07:08.732759+0000","last_clean":"2024-03-11T21:07:08.732759+0000","last_became_active":"2024-03-11T20:50:30.111819+0000","last_became_peered":"2024-03-11T20:50:30.111819+0000","last_unstale":"2024-03-11T21:07:08.732759+0000","last_undegraded":"2024-03-11T21:07:08.732759+0000","last_fullsized":"2024-03-11T21:07:08.732759+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'39","last_scrub_stamp":"2024-03-11T21:05:14.687993+0000","last_deep_scrub":"102'25","last_deep_scrub_stamp":"2024-03-11T20:51:35.187094+0000","last_clean_scrub_stamp":"2024-03-11T21:05:14.687993+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":544,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-11T21:07:09.049021+0000","last_change":"2024-03-11T21:06:40.982570+0000","last_active":"2024-03-11T21:07:09.049021+0000","last_peered":"2024-03-11T21:07:09.049021+0000","last_clean":"2024-03-11T21:07:09.049021+0000","last_became_active":"2024-03-11T21:00:48.362389+0000","last_became_peered":"2024-03-11T21:00:48.362389+0000","last_unstale":"2024-03-11T21:07:09.049021+0000","last_undegraded":"2024-03-11T21:07:09.049021+0000","last_fullsized":"2024-03-11T21:07:09.049021+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:06:40.982427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:06:40.982427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:07:51.542869+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":215,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-11T21:07:09.047642+0000","last_change":"2024-03-11T21:06:28.760544+0000","last_active":"2024-03-11T21:07:09.047642+0000","last_peered":"2024-03-11T21:07:09.047642+0000","last_clean":"2024-03-11T21:07:09.047642+0000","last_became_active":"2024-03-11T21:06:28.758451+0000","last_became_peered":"2024-03-11T21:06:28.758451+0000","last_unstale":"2024-03-11T21:07:09.047642+0000","last_undegraded":"2024-03-11T21:07:09.047642+0000","last_fullsized":"2024-03-11T21:07:09.047642+0000","mapping_epoch":297,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:04:57.639932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:55:58.888890+0000","last_clean_scrub_stamp":"2024-03-11T21:04:57.639932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:07:13.743109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":227,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-03-11T21:07:10.659003+0000","last_change":"2024-03-11T21:05:03.854094+0000","last_active":"2024-03-11T21:07:10.659003+0000","last_peered":"2024-03-11T21:07:10.659003+0000","last_clean":"2024-03-11T21:07:10.659003+0000","last_became_active":"2024-03-11T21:00:48.469015+0000","last_became_peered":"2024-03-11T21:00:48.469015+0000","last_unstale":"2024-03-11T21:07:10.659003+0000","last_undegraded":"2024-03-11T21:07:10.659003+0000","last_fullsized":"2024-03-11T21:07:10.659003+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:05:03.853977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:05:03.853977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"207'20","reported_seq":553,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-03-11T21:07:10.056710+0000","last_change":"2024-03-11T21:06:11.985090+0000","last_active":"2024-03-11T21:07:10.056710+0000","last_peered":"2024-03-11T21:07:10.056710+0000","last_clean":"2024-03-11T21:07:10.056710+0000","last_became_active":"2024-03-11T20:50:30.109549+0000","last_became_peered":"2024-03-11T20:50:30.109549+0000","last_unstale":"2024-03-11T21:07:10.056710+0000","last_undegraded":"2024-03-11T21:07:10.056710+0000","last_fullsized":"2024-03-11T21:07:10.056710+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'20","last_scrub_stamp":"2024-03-11T21:06:11.984956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:06:11.984956+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:07:18.759202+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":562,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-11T21:07:09.051715+0000","last_change":"2024-03-11T21:06:20.821862+0000","last_active":"2024-03-11T21:07:09.051715+0000","last_peered":"2024-03-11T21:07:09.051715+0000","last_clean":"2024-03-11T21:07:09.051715+0000","last_became_active":"2024-03-11T21:02:37.517748+0000","last_became_peered":"2024-03-11T21:02:37.517748+0000","last_unstale":"2024-03-11T21:07:09.051715+0000","last_undegraded":"2024-03-11T21:07:09.051715+0000","last_fullsized":"2024-03-11T21:07:09.051715+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:06:20.821707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:52:51.931295+0000","last_clean_scrub_stamp":"2024-03-11T21:06:20.821707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:07:22.287117+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":523,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-03-11T21:07:10.055401+0000","last_change":"2024-03-11T21:05:07.878264+0000","last_active":"2024-03-11T21:07:10.055401+0000","last_peered":"2024-03-11T21:07:10.055401+0000","last_clean":"2024-03-11T21:07:10.055401+0000","last_became_active":"2024-03-11T20:50:34.337173+0000","last_became_peered":"2024-03-11T20:50:34.337173+0000","last_unstale":"2024-03-11T21:07:10.055401+0000","last_undegraded":"2024-03-11T21:07:10.055401+0000","last_fullsized":"2024-03-11T21:07:10.055401+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:05:07.878156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:00:14.030599+0000","last_clean_scrub_stamp":"2024-03-11T21:05:07.878156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":541,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-03-11T21:07:09.653741+0000","last_change":"2024-03-11T21:06:40.941136+0000","last_active":"2024-03-11T21:07:09.653741+0000","last_peered":"2024-03-11T21:07:09.653741+0000","last_clean":"2024-03-11T21:07:09.653741+0000","last_became_active":"2024-03-11T20:50:28.058942+0000","last_became_peered":"2024-03-11T20:50:28.058942+0000","last_unstale":"2024-03-11T21:07:09.653741+0000","last_undegraded":"2024-03-11T21:07:09.653741+0000","last_fullsized":"2024-03-11T21:07:09.653741+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:06:40.940908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:06:40.940908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:07:54.905984+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"207'27","reported_seq":571,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-11T21:07:08.732328+0000","last_change":"2024-03-11T21:06:17.026702+0000","last_active":"2024-03-11T21:07:08.732328+0000","last_peered":"2024-03-11T21:07:08.732328+0000","last_clean":"2024-03-11T21:07:08.732328+0000","last_became_active":"2024-03-11T21:00:27.963119+0000","last_became_peered":"2024-03-11T21:00:27.963119+0000","last_unstale":"2024-03-11T21:07:08.732328+0000","last_undegraded":"2024-03-11T21:07:08.732328+0000","last_fullsized":"2024-03-11T21:07:08.732328+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'27","last_scrub_stamp":"2024-03-11T21:06:17.026570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:06:17.026570+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:07:45.885004+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":519,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-03-11T21:07:10.051492+0000","last_change":"2024-03-11T21:06:05.087050+0000","last_active":"2024-03-11T21:07:10.051492+0000","last_peered":"2024-03-11T21:07:10.051492+0000","last_clean":"2024-03-11T21:07:10.051492+0000","last_became_active":"2024-03-11T20:50:32.652854+0000","last_became_peered":"2024-03-11T20:50:32.652854+0000","last_unstale":"2024-03-11T21:07:10.051492+0000","last_undegraded":"2024-03-11T21:07:10.051492+0000","last_fullsized":"2024-03-11T21:07:10.051492+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:06:05.086941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:59:00.946094+0000","last_clean_scrub_stamp":"2024-03-11T21:06:05.086941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:07:22.839796+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":20,"num_read_kb":20,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1822,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1121,"num_read_kb":878,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":171,"num_bytes_recovered":2228,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3632,"internal_metadata":0},"log_size":778,"ondisk_log_size":778,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":125,"ondisk_log_size":125,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":864,"num_read_kb":859,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":68,"seq":292057776339,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":383796,"kb_used_data":1384,"kb_used_omap":1,"kb_used_meta":382398,"kb_avail":93361356,"statfs":{"total":95995035648,"available":95602028544,"internally_reserved":0,"allocated":1417216,"data_stored":1077158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":391576010},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":57,"seq":244813136093,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":281060,"kb_used_data":1368,"kb_used_omap":1,"kb_used_meta":279678,"kb_avail":93464092,"statfs":{"total":95995035648,"available":95707230208,"internally_reserved":0,"allocated":1400832,"data_stored":1077168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":286390727},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":211,"seq":906238099534,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":307840,"kb_used_data":820,"kb_used_omap":7,"kb_used_meta":307000,"kb_avail":93437312,"statfs":{"total":95995035648,"available":95679807488,"internally_reserved":0,"allocated":839680,"data_stored":502800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8078,"internal_metadata":314368114},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738651,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":317164,"kb_used_data":1376,"kb_used_omap":1,"kb_used_meta":315774,"kb_avail":93427988,"statfs":{"total":95995035648,"available":95670259712,"internally_reserved":0,"allocated":1409024,"data_stored":1076100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":323353029},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378886,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":324888,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":324094,"kb_avail":93420264,"statfs":{"total":95995035648,"available":95662350336,"internally_reserved":0,"allocated":798720,"data_stored":485894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":331872709},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":28,"seq":120259084539,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":451228,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":450430,"kb_avail":93293924,"statfs":{"total":95995035648,"available":95532978176,"internally_reserved":0,"allocated":802816,"data_stored":486270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":461240776},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":204,"seq":876173328466,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":113740,"kb_used_data":832,"kb_used_omap":4,"kb_used_meta":112891,"kb_avail":93631412,"statfs":{"total":95995035648,"available":95878565888,"internally_reserved":0,"allocated":851968,"data_stored":495537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4451,"internal_metadata":115601053},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":44,"seq":188978561255,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":356828,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":356030,"kb_avail":93388324,"statfs":{"total":95995035648,"available":95629643776,"internally_reserved":0,"allocated":802816,"data_stored":485816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":364575173},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3632,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T21:07:14.433 INFO:tasks.thrashosds.thrasher:Setting 6.a to [0, 4, 7] 2024-03-11T21:07:14.433 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '6.a', '0', '4', '7'] 2024-03-11T21:07:14.433 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd pg-upmap 6.a 0 4 7 2024-03-11T21:07:14.436 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:14.461 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:14.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:14.463 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:07:14.564 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:14.590 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:14.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:14.593 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:07:14.693 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:07:14.718 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:07:14.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:14.720 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:07:14.821 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:07:14.849 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:07:14.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:14.852 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:07:14.953 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:07:14.964 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:14 smithi023 bash[21372]: cluster 2024-03-11T21:07:13.847525+0000 mon.a (mon.0) 1602 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:07:14.964 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:14 smithi023 bash[21372]: audit 2024-03-11T21:07:13.851212+0000 mon.a (mon.0) 1603 : audit 1 from='client.? 172.21.15.23:0/807158889' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-03-11T21:07:14.964 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:14 smithi023 bash[21372]: audit 2024-03-11T21:07:13.851397+0000 mon.a (mon.0) 1604 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:07:14.964 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:14 smithi023 bash[21372]: cluster 2024-03-11T21:07:13.863560+0000 mon.a (mon.0) 1605 : cluster 0 osdmap e306: 8 total, 8 up, 8 in 2024-03-11T21:07:14.964 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:14 smithi023 bash[21372]: cluster 2024-03-11T21:07:14.242594+0000 mgr.y (mgr.24380) 609 : cluster 0 pgmap v678: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 911 B/s rd, 0 op/s 2024-03-11T21:07:14.964 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:14 smithi023 bash[21372]: cluster 2024-03-11T21:07:14.864157+0000 mon.a (mon.0) 1606 : cluster 0 osdmap e307: 8 total, 8 up, 8 in 2024-03-11T21:07:14.965 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:14 smithi023 bash[29146]: cluster 2024-03-11T21:07:13.847525+0000 mon.a (mon.0) 1602 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:07:14.965 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:14 smithi023 bash[29146]: audit 2024-03-11T21:07:13.851212+0000 mon.a (mon.0) 1603 : audit 1 from='client.? 172.21.15.23:0/807158889' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-03-11T21:07:14.965 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:14 smithi023 bash[29146]: audit 2024-03-11T21:07:13.851397+0000 mon.a (mon.0) 1604 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:07:14.965 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:14 smithi023 bash[29146]: cluster 2024-03-11T21:07:13.863560+0000 mon.a (mon.0) 1605 : cluster 0 osdmap e306: 8 total, 8 up, 8 in 2024-03-11T21:07:14.965 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:14 smithi023 bash[29146]: cluster 2024-03-11T21:07:14.242594+0000 mgr.y (mgr.24380) 609 : cluster 0 pgmap v678: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 911 B/s rd, 0 op/s 2024-03-11T21:07:14.965 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:14 smithi023 bash[29146]: cluster 2024-03-11T21:07:14.864157+0000 mon.a (mon.0) 1606 : cluster 0 osdmap e307: 8 total, 8 up, 8 in 2024-03-11T21:07:14.980 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:07:14.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:14.982 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:07:15.083 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:15.094 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:14 smithi146 bash[23809]: cluster 2024-03-11T21:07:13.847525+0000 mon.a (mon.0) 1602 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:07:15.094 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:14 smithi146 bash[23809]: audit 2024-03-11T21:07:13.851212+0000 mon.a (mon.0) 1603 : audit 1 from='client.? 172.21.15.23:0/807158889' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-03-11T21:07:15.094 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:14 smithi146 bash[23809]: audit 2024-03-11T21:07:13.851397+0000 mon.a (mon.0) 1604 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:07:15.095 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:14 smithi146 bash[23809]: cluster 2024-03-11T21:07:13.863560+0000 mon.a (mon.0) 1605 : cluster 0 osdmap e306: 8 total, 8 up, 8 in 2024-03-11T21:07:15.095 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:14 smithi146 bash[23809]: cluster 2024-03-11T21:07:14.242594+0000 mgr.y (mgr.24380) 609 : cluster 0 pgmap v678: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 911 B/s rd, 0 op/s 2024-03-11T21:07:15.095 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:14 smithi146 bash[23809]: cluster 2024-03-11T21:07:14.864157+0000 mon.a (mon.0) 1606 : cluster 0 osdmap e307: 8 total, 8 up, 8 in 2024-03-11T21:07:15.109 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:15.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:15.112 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:07:15.212 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:15.241 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:15.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:15.243 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:07:15.343 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:07:15.372 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:07:15.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:15.374 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:07:15.475 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:07:15.502 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:07:15.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:15.505 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:07:15.606 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:15.634 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:15.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:15.636 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:07:15.737 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:15.748 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:07:15 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:07:15.765 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:15.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:15.768 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:07:15.821 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd pool application enable unique_pool_3 rados --yes-i-really-mean-it || true 2024-03-11T21:07:15.868 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:07:15.894 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:07:15.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:15.897 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:07:15.997 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:07:16.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:15 smithi146 bash[23809]: cluster 2024-03-11T21:07:14.627293+0000 osd.4 (osd.4) 105 : cluster 0 purged_snaps scrub starts 2024-03-11T21:07:16.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:15 smithi146 bash[23809]: cluster 2024-03-11T21:07:14.627446+0000 osd.4 (osd.4) 106 : cluster 0 purged_snaps scrub ok 2024-03-11T21:07:16.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:15 smithi146 bash[23809]: cluster 2024-03-11T21:07:14.869083+0000 mon.a (mon.0) 1607 : cluster 3 Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:07:16.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:15 smithi146 bash[23809]: cluster 2024-03-11T21:07:15.888443+0000 mon.a (mon.0) 1608 : cluster 0 osdmap e308: 8 total, 8 up, 8 in 2024-03-11T21:07:16.023 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:07:16.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:16.025 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:07:16.126 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:07:16.153 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:07:16.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:16.155 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:07:16.187 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:07:16.210 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:15 smithi023 bash[29146]: cluster 2024-03-11T21:07:14.627293+0000 osd.4 (osd.4) 105 : cluster 0 purged_snaps scrub starts 2024-03-11T21:07:16.210 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:15 smithi023 bash[29146]: cluster 2024-03-11T21:07:14.627446+0000 osd.4 (osd.4) 106 : cluster 0 purged_snaps scrub ok 2024-03-11T21:07:16.210 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:15 smithi023 bash[29146]: cluster 2024-03-11T21:07:14.869083+0000 mon.a (mon.0) 1607 : cluster 3 Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:07:16.210 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:15 smithi023 bash[29146]: cluster 2024-03-11T21:07:15.888443+0000 mon.a (mon.0) 1608 : cluster 0 osdmap e308: 8 total, 8 up, 8 in 2024-03-11T21:07:16.211 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:15 smithi023 bash[21372]: cluster 2024-03-11T21:07:14.627293+0000 osd.4 (osd.4) 105 : cluster 0 purged_snaps scrub starts 2024-03-11T21:07:16.211 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:15 smithi023 bash[21372]: cluster 2024-03-11T21:07:14.627446+0000 osd.4 (osd.4) 106 : cluster 0 purged_snaps scrub ok 2024-03-11T21:07:16.211 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:15 smithi023 bash[21372]: cluster 2024-03-11T21:07:14.869083+0000 mon.a (mon.0) 1607 : cluster 3 Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:07:16.211 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:15 smithi023 bash[21372]: cluster 2024-03-11T21:07:15.888443+0000 mon.a (mon.0) 1608 : cluster 0 osdmap e308: 8 total, 8 up, 8 in 2024-03-11T21:07:16.256 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:16.285 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:16.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:16.288 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:07:16.389 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:07:16.414 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:07:16.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:16.417 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:07:16.518 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:16.545 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:16.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:16.547 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:07:16.648 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:16.676 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:16.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:16.678 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:07:16.779 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:07:16.811 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:07:16.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:16.813 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:07:16.914 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:07:16.926 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:16 smithi146 bash[23809]: audit 2024-03-11T21:07:15.655888+0000 mgr.y (mgr.24380) 610 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:07:16.926 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:16 smithi146 bash[23809]: cluster 2024-03-11T21:07:16.243092+0000 mgr.y (mgr.24380) 611 : cluster 0 pgmap v681: 148 pgs: 13 creating+peering, 1 unknown, 134 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 916 B/s rd, 0 op/s 2024-03-11T21:07:16.942 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:07:16.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:16.944 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:07:17.045 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:07:17.072 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:07:17.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:17.075 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:07:17.104 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:16 smithi023 bash[21372]: audit 2024-03-11T21:07:15.655888+0000 mgr.y (mgr.24380) 610 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:07:17.104 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:16 smithi023 bash[21372]: cluster 2024-03-11T21:07:16.243092+0000 mgr.y (mgr.24380) 611 : cluster 0 pgmap v681: 148 pgs: 13 creating+peering, 1 unknown, 134 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 916 B/s rd, 0 op/s 2024-03-11T21:07:17.105 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:16 smithi023 bash[29146]: audit 2024-03-11T21:07:15.655888+0000 mgr.y (mgr.24380) 610 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:07:17.105 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:16 smithi023 bash[29146]: cluster 2024-03-11T21:07:16.243092+0000 mgr.y (mgr.24380) 611 : cluster 0 pgmap v681: 148 pgs: 13 creating+peering, 1 unknown, 134 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 916 B/s rd, 0 op/s 2024-03-11T21:07:17.175 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:07:17.196 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:07:17.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:17.198 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:07:17.299 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:17.327 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:17.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:17.329 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:07:17.430 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:17.455 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:17.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:17.458 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:07:17.558 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:07:17.588 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:07:17.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:17.590 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:07:17.691 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:17.716 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:17.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:17.719 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:07:17.813 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd unset noscrub 2024-03-11T21:07:17.819 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:07:17.882 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:07:17.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:17.885 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:07:17.985 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:07:18.013 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:07:18.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:18.016 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:07:18.117 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:07:18.144 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:07:18.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:18.147 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:07:18.247 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:18.274 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:18.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:18.276 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:07:18.377 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:07:18.404 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:07:18.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:18.406 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:07:18.489 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:07:18.507 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:18.534 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:18.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:18.537 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:07:18.638 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:07:18.650 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:07:18 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:07:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:07:18.665 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:07:18.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:18.668 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:07:18.769 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:07:18.797 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:07:18.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:18.800 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:07:18.901 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:07:18.930 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:07:18.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:18.933 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:07:19.033 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:07:19.061 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:07:19.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:19.063 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:07:19.164 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:19.192 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:19.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:19.195 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:07:19.295 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:07:19.323 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:07:19.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:19.325 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:07:19.353 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:07:19.427 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:07:19.449 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:07:19.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:19.451 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:07:19.552 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:07:19.565 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:19 smithi023 bash[21372]: cluster 2024-03-11T21:07:18.244056+0000 mgr.y (mgr.24380) 612 : cluster 0 pgmap v682: 148 pgs: 13 creating+peering, 135 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:07:19.566 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:19 smithi023 bash[29146]: cluster 2024-03-11T21:07:18.244056+0000 mgr.y (mgr.24380) 612 : cluster 0 pgmap v682: 148 pgs: 13 creating+peering, 135 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:07:19.581 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:07:19.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:19.584 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:07:19.685 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:07:19.707 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:07:19.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:19.710 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:07:19.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:19 smithi146 bash[23809]: cluster 2024-03-11T21:07:18.244056+0000 mgr.y (mgr.24380) 612 : cluster 0 pgmap v682: 148 pgs: 13 creating+peering, 135 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:07:19.810 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:07:19.837 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:07:19.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:19.839 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:07:19.940 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:19.965 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:19.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:19.968 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:07:20.069 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:07:20.096 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:07:20.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:20.099 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:07:20.200 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:07:20.229 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:07:20.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:20.231 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:07:20.332 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:20.361 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:20.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:20.363 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:07:20.376 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:07:20.394 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:20 smithi023 bash[21372]: cluster 2024-03-11T21:07:19.678990+0000 osd.6 (osd.6) 250 : cluster 0 purged_snaps scrub starts 2024-03-11T21:07:20.394 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:20 smithi023 bash[21372]: cluster 2024-03-11T21:07:19.679100+0000 osd.6 (osd.6) 251 : cluster 0 purged_snaps scrub ok 2024-03-11T21:07:20.394 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:20 smithi023 bash[29146]: cluster 2024-03-11T21:07:19.678990+0000 osd.6 (osd.6) 250 : cluster 0 purged_snaps scrub starts 2024-03-11T21:07:20.394 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:20 smithi023 bash[29146]: cluster 2024-03-11T21:07:19.679100+0000 osd.6 (osd.6) 251 : cluster 0 purged_snaps scrub ok 2024-03-11T21:07:20.464 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:20.490 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:20.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:20.492 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:07:20.593 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:07:20.620 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:07:20.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:20.622 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:07:20.723 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:07:20.752 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:07:20.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:20.755 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:07:20.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:20 smithi146 bash[23809]: cluster 2024-03-11T21:07:19.678990+0000 osd.6 (osd.6) 250 : cluster 0 purged_snaps scrub starts 2024-03-11T21:07:20.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:20 smithi146 bash[23809]: cluster 2024-03-11T21:07:19.679100+0000 osd.6 (osd.6) 251 : cluster 0 purged_snaps scrub ok 2024-03-11T21:07:20.855 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:07:20.886 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:07:20.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:20.889 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:07:20.990 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:21.017 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:21.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:21.019 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:07:21.120 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:07:21.148 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:07:21.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:21.150 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:07:21.251 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:07:21.278 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:07:21.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:21.280 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:07:21.326 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-11T21:07:21.326 INFO:teuthology.orchestra.run.smithi023.stdout: "ops": [], 2024-03-11T21:07:21.327 INFO:teuthology.orchestra.run.smithi023.stdout: "num_ops": 0 2024-03-11T21:07:21.327 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-11T21:07:21.381 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:21.409 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:21.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:21.412 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:07:21.512 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:21.523 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:21 smithi146 bash[23809]: cluster 2024-03-11T21:07:20.245005+0000 mgr.y (mgr.24380) 613 : cluster 0 pgmap v683: 148 pgs: 148 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-03-11T21:07:21.524 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:21 smithi146 bash[23809]: cluster 2024-03-11T21:07:20.318075+0000 mon.a (mon.0) 1609 : cluster 0 osdmap e309: 8 total, 8 up, 8 in 2024-03-11T21:07:21.524 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:21 smithi146 bash[23809]: audit 2024-03-11T21:07:20.639846+0000 mon.a (mon.0) 1610 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:07:21.524 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:21 smithi146 bash[23809]: audit 2024-03-11T21:07:20.641009+0000 mon.a (mon.0) 1611 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:07:21.541 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:21.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:21.543 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:07:21.548 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:21 smithi023 bash[21372]: cluster 2024-03-11T21:07:20.245005+0000 mgr.y (mgr.24380) 613 : cluster 0 pgmap v683: 148 pgs: 148 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-03-11T21:07:21.548 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:21 smithi023 bash[21372]: cluster 2024-03-11T21:07:20.318075+0000 mon.a (mon.0) 1609 : cluster 0 osdmap e309: 8 total, 8 up, 8 in 2024-03-11T21:07:21.548 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:21 smithi023 bash[21372]: audit 2024-03-11T21:07:20.639846+0000 mon.a (mon.0) 1610 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:07:21.548 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:21 smithi023 bash[21372]: audit 2024-03-11T21:07:20.641009+0000 mon.a (mon.0) 1611 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:07:21.549 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:21 smithi023 bash[29146]: cluster 2024-03-11T21:07:20.245005+0000 mgr.y (mgr.24380) 613 : cluster 0 pgmap v683: 148 pgs: 148 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-03-11T21:07:21.549 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:21 smithi023 bash[29146]: cluster 2024-03-11T21:07:20.318075+0000 mon.a (mon.0) 1609 : cluster 0 osdmap e309: 8 total, 8 up, 8 in 2024-03-11T21:07:21.549 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:21 smithi023 bash[29146]: audit 2024-03-11T21:07:20.639846+0000 mon.a (mon.0) 1610 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:07:21.549 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:21 smithi023 bash[29146]: audit 2024-03-11T21:07:20.641009+0000 mon.a (mon.0) 1611 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:07:21.644 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:07:21.673 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:07:21.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:21.675 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:07:21.776 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:21.803 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:21.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:21.805 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:07:21.906 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:07:21.931 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:07:21.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:21.933 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:07:22.034 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:07:22.059 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:07:22.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:22.061 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:07:22.162 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:07:22.193 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:07:22.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:22.196 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:07:22.296 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:07:22.326 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:07:22.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:22.328 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:07:22.429 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:07:22.457 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:07:22.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:22.459 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:07:22.560 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:07:22.589 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:07:22.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:22.592 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:07:22.692 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:07:22.722 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:07:22.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:22.726 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:07:22.826 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:22.853 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:22.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:22.855 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:07:22.956 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:07:22.980 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:07:22.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:22.982 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:07:23.083 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:23.108 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:23.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:23.110 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:07:23.211 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:07:23.234 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:07:23.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:23.237 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:07:23.338 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:07:23.363 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:07:23.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:23.366 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:07:23.466 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:07:23.478 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:23 smithi023 bash[21372]: cluster 2024-03-11T21:07:22.245837+0000 mgr.y (mgr.24380) 614 : cluster 0 pgmap v685: 148 pgs: 148 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 693 B/s rd, 0 op/s 2024-03-11T21:07:23.478 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:23 smithi023 bash[29146]: cluster 2024-03-11T21:07:22.245837+0000 mgr.y (mgr.24380) 614 : cluster 0 pgmap v685: 148 pgs: 148 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 693 B/s rd, 0 op/s 2024-03-11T21:07:23.496 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:07:23.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:23.498 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:07:23.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:23 smithi146 bash[23809]: cluster 2024-03-11T21:07:22.245837+0000 mgr.y (mgr.24380) 614 : cluster 0 pgmap v685: 148 pgs: 148 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 693 B/s rd, 0 op/s 2024-03-11T21:07:23.599 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:23.625 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:23.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:23.627 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:07:23.728 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:23.753 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:23.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:23.755 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:07:23.856 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:23.884 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:23.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:23.886 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:07:23.987 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:07:24.016 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:07:24.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:24.019 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:07:24.120 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:07:24.124 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-03-11T21:07:24.147 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:07:24.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:24.149 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:07:24.250 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:07:24.278 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:07:24.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:24.280 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:07:24.381 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:07:24.402 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:07:24.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:24.405 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:07:24.506 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:07:24.536 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:07:24.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:24.539 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:07:24.640 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:24.666 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:24.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:24.669 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:07:24.769 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:07:24.795 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:07:24.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:24.797 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:07:24.898 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:24.924 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:24.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:24.926 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:07:25.027 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:07:25.054 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:07:25.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:25.056 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:07:25.157 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:25.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:25.187 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:25.187 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:07:25.288 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:07:25.310 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:07:25.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:25.312 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:07:25.413 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:07:25.440 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:07:25.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:25.443 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:07:25.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:25 smithi146 bash[23809]: cluster 2024-03-11T21:07:24.246221+0000 mgr.y (mgr.24380) 615 : cluster 0 pgmap v686: 148 pgs: 148 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 612 B/s rd, 0 op/s 2024-03-11T21:07:25.543 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:25.563 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:25 smithi023 bash[21372]: cluster 2024-03-11T21:07:24.246221+0000 mgr.y (mgr.24380) 615 : cluster 0 pgmap v686: 148 pgs: 148 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 612 B/s rd, 0 op/s 2024-03-11T21:07:25.564 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:25 smithi023 bash[29146]: cluster 2024-03-11T21:07:24.246221+0000 mgr.y (mgr.24380) 615 : cluster 0 pgmap v686: 148 pgs: 148 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 612 B/s rd, 0 op/s 2024-03-11T21:07:25.570 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:25.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:25.575 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:07:25.589 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:07:25.676 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:07:25.703 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:07:25.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:25.705 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:07:25.806 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:07:25.833 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:07:25.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:25.836 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:07:25.937 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:07:25.963 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:07:25.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:25.966 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:07:26.010 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:07:25 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:07:26.066 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:07:26.095 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:07:26.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:26.098 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:07:26.199 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:26.228 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:26.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:26.230 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:07:26.330 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:07:26.361 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:07:26.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:26.363 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:07:26.464 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:07:26.493 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:07:26.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:26.495 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:07:26.597 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:07:26.626 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:07:26.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:26.628 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:07:26.729 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:26.757 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:26.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:26.759 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:07:26.860 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:07:26.887 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:07:26.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:26.890 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:07:26.991 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:07:27.018 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:07:27.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:27.021 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:07:27.122 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:07:27.150 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:07:27.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:27.152 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:07:27.253 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:27.280 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:27.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:27.283 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:07:27.383 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:07:27.395 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:27 smithi023 bash[21372]: audit 2024-03-11T21:07:25.666409+0000 mgr.y (mgr.24380) 616 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:07:27.395 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:27 smithi023 bash[21372]: cluster 2024-03-11T21:07:26.247796+0000 mgr.y (mgr.24380) 617 : cluster 0 pgmap v687: 148 pgs: 148 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:07:27.395 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:27 smithi023 bash[29146]: audit 2024-03-11T21:07:25.666409+0000 mgr.y (mgr.24380) 616 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:07:27.395 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:27 smithi023 bash[29146]: cluster 2024-03-11T21:07:26.247796+0000 mgr.y (mgr.24380) 617 : cluster 0 pgmap v687: 148 pgs: 148 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:07:27.412 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:07:27.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:27.415 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:07:27.516 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:07:27.527 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:27 smithi146 bash[23809]: audit 2024-03-11T21:07:25.666409+0000 mgr.y (mgr.24380) 616 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:07:27.527 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:27 smithi146 bash[23809]: cluster 2024-03-11T21:07:26.247796+0000 mgr.y (mgr.24380) 617 : cluster 0 pgmap v687: 148 pgs: 148 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:07:27.542 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:07:27.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:27.545 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:07:27.646 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:07:27.673 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:07:27.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:27.676 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:07:27.776 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:27.805 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:27.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:27.807 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:07:27.908 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:07:27.938 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:07:27.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:27.940 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:07:28.041 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:07:28.070 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:07:28.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:28.072 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:07:28.173 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:07:28.195 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:07:28.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:28.197 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:07:28.298 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:07:28.326 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:07:28.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:28.328 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:07:28.429 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:07:28.436 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:07:28 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:07:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:07:28.450 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:07:28.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:28.452 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:07:28.553 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:07:28.575 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:07:28.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:28.577 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:07:28.678 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:07:28.705 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:07:28.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:28.707 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:07:28.808 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:07:28.836 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:07:28.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:28.839 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:07:28.940 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:07:28.969 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:07:28.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:28.971 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:07:29.072 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:29.101 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:29.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:29.103 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:07:29.204 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:07:29.232 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:07:29.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:29.235 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:07:29.335 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:07:29.365 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:07:29.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:29.368 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:07:29.468 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:29.495 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:29.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:29.497 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:07:29.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:29 smithi023 bash[21372]: cluster 2024-03-11T21:07:28.248438+0000 mgr.y (mgr.24380) 618 : cluster 0 pgmap v688: 148 pgs: 148 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:07:29.531 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:29 smithi023 bash[29146]: cluster 2024-03-11T21:07:28.248438+0000 mgr.y (mgr.24380) 618 : cluster 0 pgmap v688: 148 pgs: 148 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:07:29.598 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:07:29.623 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:07:29.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:29.625 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:07:29.726 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:07:29.754 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:07:29.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:29.756 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:07:29.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:29 smithi146 bash[23809]: cluster 2024-03-11T21:07:28.248438+0000 mgr.y (mgr.24380) 618 : cluster 0 pgmap v688: 148 pgs: 148 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:07:29.857 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:07:29.886 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:07:29.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:29.888 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:07:29.989 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:07:30.018 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:07:30.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:30.020 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:07:30.121 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:07:30.147 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:07:30.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:30.150 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:07:30.251 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:07:30.278 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:07:30.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:30.280 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:07:30.381 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:07:30.409 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:07:30.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:30.411 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:07:30.512 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:07:30.540 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:07:30.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:30.543 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:07:30.643 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:07:30.672 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:07:30.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:30.675 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:07:30.776 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:07:30.806 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:07:30.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:30.809 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:07:30.909 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:07:30.932 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:07:30.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:30.935 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:07:31.035 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:07:31.055 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:07:31.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:31.059 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:07:31.129 INFO:teuthology.orchestra.run.smithi023.stdout:osd.0: {} 2024-03-11T21:07:31.149 INFO:teuthology.orchestra.run.smithi023.stdout:osd.1: {} 2024-03-11T21:07:31.160 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:07:31.170 INFO:teuthology.orchestra.run.smithi023.stdout:osd.2: {} 2024-03-11T21:07:31.180 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:07:31.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:31.182 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:07:31.191 INFO:teuthology.orchestra.run.smithi023.stdout:osd.3: {} 2024-03-11T21:07:31.216 INFO:teuthology.orchestra.run.smithi023.stdout:osd.4: {} 2024-03-11T21:07:31.241 INFO:teuthology.orchestra.run.smithi023.stdout:osd.5: {} 2024-03-11T21:07:31.266 INFO:teuthology.orchestra.run.smithi023.stdout:osd.6: {} 2024-03-11T21:07:31.283 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:07:31.291 INFO:teuthology.orchestra.run.smithi023.stdout:osd.7: {} 2024-03-11T21:07:31.293 INFO:teuthology.orchestra.run.smithi023.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:07:31.293 INFO:teuthology.orchestra.run.smithi023.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:07:31.293 INFO:teuthology.orchestra.run.smithi023.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:07:31.293 INFO:teuthology.orchestra.run.smithi023.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:07:31.293 INFO:teuthology.orchestra.run.smithi023.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:07:31.293 INFO:teuthology.orchestra.run.smithi023.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:07:31.293 INFO:teuthology.orchestra.run.smithi023.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:07:31.293 INFO:teuthology.orchestra.run.smithi023.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:07:31.303 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:07:31.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:31.305 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:07:31.406 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:07:31.424 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:07:31.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:31.427 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:07:31.528 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:07:31.549 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:07:31.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:31.551 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:07:31.576 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:31 smithi023 bash[29146]: cluster 2024-03-11T21:07:30.249900+0000 mgr.y (mgr.24380) 619 : cluster 0 pgmap v689: 148 pgs: 148 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:07:31.577 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:31 smithi023 bash[29146]: audit 2024-03-11T21:07:31.114317+0000 mon.c (mon.2) 78 : audit 0 from='client.? 172.21.15.23:0/3267102996' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:07:31.577 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:31 smithi023 bash[21372]: cluster 2024-03-11T21:07:30.249900+0000 mgr.y (mgr.24380) 619 : cluster 0 pgmap v689: 148 pgs: 148 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:07:31.577 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:31 smithi023 bash[21372]: audit 2024-03-11T21:07:31.114317+0000 mon.c (mon.2) 78 : audit 0 from='client.? 172.21.15.23:0/3267102996' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:07:31.652 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:07:31.664 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:31 smithi146 bash[23809]: cluster 2024-03-11T21:07:30.249900+0000 mgr.y (mgr.24380) 619 : cluster 0 pgmap v689: 148 pgs: 148 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:07:31.664 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:31 smithi146 bash[23809]: audit 2024-03-11T21:07:31.114317+0000 mon.c (mon.2) 78 : audit 0 from='client.? 172.21.15.23:0/3267102996' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:07:31.679 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:07:31.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:31.681 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:07:31.782 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:31.808 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:31.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:31.810 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:07:31.911 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:07:31.937 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:07:31.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:31.939 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:07:32.040 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:07:32.066 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:07:32.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:32.069 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:07:32.170 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:07:32.197 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:07:32.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:32.199 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:07:32.300 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:07:32.325 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:07:32.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:32.327 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:07:32.363 INFO:teuthology.orchestra.run.smithi023.stderr:enabled application 'rados' on pool 'unique_pool_3' 2024-03-11T21:07:32.429 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:32.440 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:32 smithi146 bash[23809]: audit 2024-03-11T21:07:31.789234+0000 mon.c (mon.2) 79 : audit 1 from='client.? 172.21.15.23:0/3461142101' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:07:32.440 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:32 smithi146 bash[23809]: audit 2024-03-11T21:07:31.789813+0000 mon.a (mon.0) 1612 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:07:32.457 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:32.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:32.459 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:07:32.560 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:07:32.586 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:07:32.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:32.589 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:07:32.690 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:32.717 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:32.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:32.719 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:07:32.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:32 smithi023 bash[21372]: audit 2024-03-11T21:07:31.789234+0000 mon.c (mon.2) 79 : audit 1 from='client.? 172.21.15.23:0/3461142101' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:07:32.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:32 smithi023 bash[21372]: audit 2024-03-11T21:07:31.789813+0000 mon.a (mon.0) 1612 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:07:32.780 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:32 smithi023 bash[29146]: audit 2024-03-11T21:07:31.789234+0000 mon.c (mon.2) 79 : audit 1 from='client.? 172.21.15.23:0/3461142101' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:07:32.780 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:32 smithi023 bash[29146]: audit 2024-03-11T21:07:31.789813+0000 mon.a (mon.0) 1612 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:07:32.820 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:32.850 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:32.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:32.852 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:07:32.953 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:07:32.980 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:07:32.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:32.982 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:07:33.083 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:07:33.110 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:07:33.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:33.112 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:07:33.213 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:07:33.241 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:07:33.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:33.243 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:07:33.344 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:07:33.368 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:07:33.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:33.370 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:07:33.384 INFO:teuthology.orchestra.run.smithi023.stderr:set 6.a pg_upmap mapping to [0,4,7] 2024-03-11T21:07:33.471 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:33.496 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:33.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:33.498 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:07:33.598 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:07:33.609 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:33 smithi146 bash[23809]: cluster 2024-03-11T21:07:32.250902+0000 mgr.y (mgr.24380) 620 : cluster 0 pgmap v690: 148 pgs: 148 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 857 B/s rd, 0 op/s 2024-03-11T21:07:33.610 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:33 smithi146 bash[23809]: audit 2024-03-11T21:07:32.364176+0000 mon.a (mon.0) 1613 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-11T21:07:33.610 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:33 smithi146 bash[23809]: cluster 2024-03-11T21:07:32.367560+0000 mon.a (mon.0) 1614 : cluster 0 osdmap e310: 8 total, 8 up, 8 in 2024-03-11T21:07:33.610 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:33 smithi146 bash[23809]: audit 2024-03-11T21:07:32.375845+0000 mon.a (mon.0) 1615 : audit 1 from='client.? 172.21.15.23:0/4184944165' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "6.a", "id": [0, 4, 7]}]: dispatch 2024-03-11T21:07:33.624 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:07:33.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:33.627 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:07:33.727 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:07:33.739 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:33 smithi023 bash[21372]: cluster 2024-03-11T21:07:32.250902+0000 mgr.y (mgr.24380) 620 : cluster 0 pgmap v690: 148 pgs: 148 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 857 B/s rd, 0 op/s 2024-03-11T21:07:33.740 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:33 smithi023 bash[21372]: audit 2024-03-11T21:07:32.364176+0000 mon.a (mon.0) 1613 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-11T21:07:33.740 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:33 smithi023 bash[21372]: cluster 2024-03-11T21:07:32.367560+0000 mon.a (mon.0) 1614 : cluster 0 osdmap e310: 8 total, 8 up, 8 in 2024-03-11T21:07:33.740 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:33 smithi023 bash[21372]: audit 2024-03-11T21:07:32.375845+0000 mon.a (mon.0) 1615 : audit 1 from='client.? 172.21.15.23:0/4184944165' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "6.a", "id": [0, 4, 7]}]: dispatch 2024-03-11T21:07:33.740 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:33 smithi023 bash[29146]: cluster 2024-03-11T21:07:32.250902+0000 mgr.y (mgr.24380) 620 : cluster 0 pgmap v690: 148 pgs: 148 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 857 B/s rd, 0 op/s 2024-03-11T21:07:33.740 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:33 smithi023 bash[29146]: audit 2024-03-11T21:07:32.364176+0000 mon.a (mon.0) 1613 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-11T21:07:33.740 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:33 smithi023 bash[29146]: cluster 2024-03-11T21:07:32.367560+0000 mon.a (mon.0) 1614 : cluster 0 osdmap e310: 8 total, 8 up, 8 in 2024-03-11T21:07:33.740 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:33 smithi023 bash[29146]: audit 2024-03-11T21:07:32.375845+0000 mon.a (mon.0) 1615 : audit 1 from='client.? 172.21.15.23:0/4184944165' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "6.a", "id": [0, 4, 7]}]: dispatch 2024-03-11T21:07:33.756 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:07:33.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:33.759 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:07:33.860 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:33.888 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:33.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:33.891 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:07:33.991 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:07:34.018 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:07:34.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:34.020 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:07:34.121 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:07:34.150 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:07:34.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:34.153 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:07:34.253 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:07:34.281 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:07:34.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:34.284 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:07:34.385 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:34.386 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:34 smithi023 bash[21372]: audit 2024-03-11T21:07:33.384027+0000 mon.a (mon.0) 1616 : audit 1 from='client.? 172.21.15.23:0/4184944165' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "6.a", "id": [0, 4, 7]}]': finished 2024-03-11T21:07:34.386 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:34 smithi023 bash[21372]: cluster 2024-03-11T21:07:33.390285+0000 mon.a (mon.0) 1617 : cluster 0 osdmap e311: 8 total, 8 up, 8 in 2024-03-11T21:07:34.413 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:34.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:34.416 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:07:34.516 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:34.544 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:34.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:34.546 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:07:34.647 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:07:34.658 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:34 smithi146 bash[23809]: audit 2024-03-11T21:07:33.384027+0000 mon.a (mon.0) 1616 : audit 1 from='client.? 172.21.15.23:0/4184944165' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "6.a", "id": [0, 4, 7]}]': finished 2024-03-11T21:07:34.658 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:34 smithi146 bash[23809]: cluster 2024-03-11T21:07:33.390285+0000 mon.a (mon.0) 1617 : cluster 0 osdmap e311: 8 total, 8 up, 8 in 2024-03-11T21:07:34.674 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:07:34.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:34.676 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:07:34.777 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:34.780 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:34 smithi023 bash[29146]: audit 2024-03-11T21:07:33.384027+0000 mon.a (mon.0) 1616 : audit 1 from='client.? 172.21.15.23:0/4184944165' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "6.a", "id": [0, 4, 7]}]': finished 2024-03-11T21:07:34.780 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:34 smithi023 bash[29146]: cluster 2024-03-11T21:07:33.390285+0000 mon.a (mon.0) 1617 : cluster 0 osdmap e311: 8 total, 8 up, 8 in 2024-03-11T21:07:34.803 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:34.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:34.806 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:07:34.907 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:07:34.937 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:07:34.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:34.939 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:07:35.040 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:07:35.068 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:07:35.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:35.070 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:07:35.171 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:07:35.198 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:07:35.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:35.200 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:07:35.301 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:35.327 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:35.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:35.330 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:07:35.431 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:07:35.442 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:35 smithi146 bash[23809]: cluster 2024-03-11T21:07:34.251599+0000 mgr.y (mgr.24380) 621 : cluster 0 pgmap v693: 148 pgs: 148 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:07:35.442 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:35 smithi146 bash[23809]: cluster 2024-03-11T21:07:34.384588+0000 mon.a (mon.0) 1618 : cluster 1 Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-11T21:07:35.442 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:35 smithi146 bash[23809]: cluster 2024-03-11T21:07:34.409307+0000 mon.a (mon.0) 1619 : cluster 0 osdmap e312: 8 total, 8 up, 8 in 2024-03-11T21:07:35.442 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:35 smithi146 bash[23809]: cluster 2024-03-11T21:07:35.278110+0000 mon.a (mon.0) 1620 : cluster 0 osdmap e313: 8 total, 8 up, 8 in 2024-03-11T21:07:35.458 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:07:35.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:35.460 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:07:35.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:35 smithi023 bash[21372]: cluster 2024-03-11T21:07:34.251599+0000 mgr.y (mgr.24380) 621 : cluster 0 pgmap v693: 148 pgs: 148 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:07:35.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:35 smithi023 bash[21372]: cluster 2024-03-11T21:07:34.384588+0000 mon.a (mon.0) 1618 : cluster 1 Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-11T21:07:35.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:35 smithi023 bash[21372]: cluster 2024-03-11T21:07:34.409307+0000 mon.a (mon.0) 1619 : cluster 0 osdmap e312: 8 total, 8 up, 8 in 2024-03-11T21:07:35.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:35 smithi023 bash[21372]: cluster 2024-03-11T21:07:35.278110+0000 mon.a (mon.0) 1620 : cluster 0 osdmap e313: 8 total, 8 up, 8 in 2024-03-11T21:07:35.530 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:35 smithi023 bash[29146]: cluster 2024-03-11T21:07:34.251599+0000 mgr.y (mgr.24380) 621 : cluster 0 pgmap v693: 148 pgs: 148 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:07:35.531 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:35 smithi023 bash[29146]: cluster 2024-03-11T21:07:34.384588+0000 mon.a (mon.0) 1618 : cluster 1 Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-11T21:07:35.531 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:35 smithi023 bash[29146]: cluster 2024-03-11T21:07:34.409307+0000 mon.a (mon.0) 1619 : cluster 0 osdmap e312: 8 total, 8 up, 8 in 2024-03-11T21:07:35.531 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:35 smithi023 bash[29146]: cluster 2024-03-11T21:07:35.278110+0000 mon.a (mon.0) 1620 : cluster 0 osdmap e313: 8 total, 8 up, 8 in 2024-03-11T21:07:35.561 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:07:35.568 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T21:07:35.590 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:07:35.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:35.593 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:07:35.693 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:07:35.716 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:07:35.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:35.719 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:07:35.760 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:07:35 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:07:35.820 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:35.846 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:35.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:35.848 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:07:35.949 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:07:35.977 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:07:35.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:35.979 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:07:36.080 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:07:36.103 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:07:36.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:36.105 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:07:36.183 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:07:36.206 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:07:36.226 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:07:36.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:36.228 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:07:36.317 DEBUG:teuthology.orchestra.run.smithi023:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_3 bench 90 write' 2024-03-11T21:07:36.320 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-03-11T21:07:36.321 INFO:teuthology.orchestra.run:waiting for 3000 2024-03-11T21:07:36.329 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:36.357 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:36.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:36.359 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:07:36.364 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:hints = 1 2024-03-11T21:07:36.364 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-03-11T21:07:36.364 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:Object prefix: benchmark_data_smithi023_124944 2024-03-11T21:07:36.365 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-11T21:07:36.365 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 0 0 0 0 0 0 - 0 2024-03-11T21:07:36.460 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:07:36.494 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:07:36.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:36.498 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:07:36.598 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:07:36.624 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:07:36.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:36.627 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:07:36.671 INFO:teuthology.orchestra.run.smithi023.stderr:noscrub is unset 2024-03-11T21:07:36.727 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:07:36.754 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:07:36.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:36.757 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:07:36.858 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:07:36.867 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:36 smithi023 bash[21372]: audit 2024-03-11T21:07:35.660809+0000 mon.a (mon.0) 1621 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:07:36.868 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:36 smithi023 bash[21372]: audit 2024-03-11T21:07:35.662256+0000 mon.a (mon.0) 1622 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:07:36.868 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:36 smithi023 bash[21372]: audit 2024-03-11T21:07:35.677597+0000 mgr.y (mgr.24380) 622 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:07:36.868 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:36 smithi023 bash[21372]: cluster 2024-03-11T21:07:36.252127+0000 mgr.y (mgr.24380) 623 : cluster 0 pgmap v696: 148 pgs: 148 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:07:36.868 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:36 smithi023 bash[21372]: audit 2024-03-11T21:07:36.394303+0000 mon.a (mon.0) 1623 : audit 1 from='client.? 172.21.15.23:0/4069755843' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:07:36.868 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:36 smithi023 bash[29146]: audit 2024-03-11T21:07:35.660809+0000 mon.a (mon.0) 1621 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:07:36.869 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:36 smithi023 bash[29146]: audit 2024-03-11T21:07:35.662256+0000 mon.a (mon.0) 1622 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:07:36.869 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:36 smithi023 bash[29146]: audit 2024-03-11T21:07:35.677597+0000 mgr.y (mgr.24380) 622 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:07:36.869 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:36 smithi023 bash[29146]: cluster 2024-03-11T21:07:36.252127+0000 mgr.y (mgr.24380) 623 : cluster 0 pgmap v696: 148 pgs: 148 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:07:36.869 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:36 smithi023 bash[29146]: audit 2024-03-11T21:07:36.394303+0000 mon.a (mon.0) 1623 : audit 1 from='client.? 172.21.15.23:0/4069755843' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:07:36.880 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:07:36.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:36.882 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:07:36.983 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:07:36.994 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:36 smithi146 bash[23809]: audit 2024-03-11T21:07:35.660809+0000 mon.a (mon.0) 1621 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:07:36.994 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:36 smithi146 bash[23809]: audit 2024-03-11T21:07:35.662256+0000 mon.a (mon.0) 1622 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:07:36.994 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:36 smithi146 bash[23809]: audit 2024-03-11T21:07:35.677597+0000 mgr.y (mgr.24380) 622 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:07:36.994 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:36 smithi146 bash[23809]: cluster 2024-03-11T21:07:36.252127+0000 mgr.y (mgr.24380) 623 : cluster 0 pgmap v696: 148 pgs: 148 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:07:36.994 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:36 smithi146 bash[23809]: audit 2024-03-11T21:07:36.394303+0000 mon.a (mon.0) 1623 : audit 1 from='client.? 172.21.15.23:0/4069755843' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:07:37.015 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:07:37.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:37.017 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:07:37.118 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:07:37.148 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:07:37.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:37.151 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:07:37.252 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:07:37.279 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:07:37.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:37.282 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:07:37.365 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 1 16 2874 2858 178.615 178.625 0.00453827 0.00557207 2024-03-11T21:07:37.384 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:07:37.408 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:07:37.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:37.411 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:07:37.512 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:07:37.540 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:07:37.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:37.542 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:07:37.643 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:37.662 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:37.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:37.664 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:07:37.764 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:07:37.797 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:07:37.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:37.799 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:07:37.900 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:07:37.912 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:37 smithi146 bash[23809]: cluster 2024-03-11T21:07:36.658735+0000 mon.a (mon.0) 1624 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:07:37.912 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:37 smithi146 bash[23809]: audit 2024-03-11T21:07:36.674641+0000 mon.a (mon.0) 1625 : audit 1 from='client.? 172.21.15.23:0/4069755843' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:07:37.912 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:37 smithi146 bash[23809]: cluster 2024-03-11T21:07:36.683301+0000 mon.a (mon.0) 1626 : cluster 0 osdmap e314: 8 total, 8 up, 8 in 2024-03-11T21:07:37.929 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:07:37.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:37.932 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:07:38.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:37 smithi023 bash[21372]: cluster 2024-03-11T21:07:36.658735+0000 mon.a (mon.0) 1624 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:07:38.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:37 smithi023 bash[21372]: audit 2024-03-11T21:07:36.674641+0000 mon.a (mon.0) 1625 : audit 1 from='client.? 172.21.15.23:0/4069755843' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:07:38.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:37 smithi023 bash[21372]: cluster 2024-03-11T21:07:36.683301+0000 mon.a (mon.0) 1626 : cluster 0 osdmap e314: 8 total, 8 up, 8 in 2024-03-11T21:07:38.031 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:37 smithi023 bash[29146]: cluster 2024-03-11T21:07:36.658735+0000 mon.a (mon.0) 1624 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:07:38.032 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:37 smithi023 bash[29146]: audit 2024-03-11T21:07:36.674641+0000 mon.a (mon.0) 1625 : audit 1 from='client.? 172.21.15.23:0/4069755843' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:07:38.032 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:37 smithi023 bash[29146]: cluster 2024-03-11T21:07:36.683301+0000 mon.a (mon.0) 1626 : cluster 0 osdmap e314: 8 total, 8 up, 8 in 2024-03-11T21:07:38.032 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:07:38.068 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:07:38.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:38.070 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:07:38.171 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:07:38.200 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:07:38.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:38.203 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:07:38.303 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:38.334 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:38.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:38.337 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:07:38.365 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 2 16 4897 4881 152.521 126.438 0.00454576 0.00647332 2024-03-11T21:07:38.438 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:38.466 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:38.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:38.469 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:07:38.569 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:07:38.590 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:07:38.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:38.592 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:07:38.682 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:07:38 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:07:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:07:38.692 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:07:38.728 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:07:38.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:38.731 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:07:38.832 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:07:38.839 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:38 smithi146 bash[23809]: cluster 2024-03-11T21:07:37.446563+0000 osd.3 (osd.3) 344 : cluster 0 2.19 scrub starts 2024-03-11T21:07:38.839 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:38 smithi146 bash[23809]: cluster 2024-03-11T21:07:37.448696+0000 osd.3 (osd.3) 345 : cluster 0 2.19 scrub ok 2024-03-11T21:07:38.840 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:38 smithi146 bash[23809]: cluster 2024-03-11T21:07:37.775447+0000 osd.5 (osd.5) 325 : cluster 0 3.1c scrub starts 2024-03-11T21:07:38.840 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:38 smithi146 bash[23809]: cluster 2024-03-11T21:07:37.781627+0000 osd.5 (osd.5) 326 : cluster 0 3.1c scrub ok 2024-03-11T21:07:38.840 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:38 smithi146 bash[23809]: cluster 2024-03-11T21:07:38.252586+0000 mgr.y (mgr.24380) 624 : cluster 0 pgmap v698: 148 pgs: 148 active+clean; 11 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 2.2 MiB/s wr, 35 op/s 2024-03-11T21:07:38.851 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:07:38.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:38.853 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:07:38.954 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:38.979 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:38.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:38.982 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:07:39.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:38 smithi023 bash[21372]: cluster 2024-03-11T21:07:37.446563+0000 osd.3 (osd.3) 344 : cluster 0 2.19 scrub starts 2024-03-11T21:07:39.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:38 smithi023 bash[21372]: cluster 2024-03-11T21:07:37.448696+0000 osd.3 (osd.3) 345 : cluster 0 2.19 scrub ok 2024-03-11T21:07:39.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:38 smithi023 bash[21372]: cluster 2024-03-11T21:07:37.775447+0000 osd.5 (osd.5) 325 : cluster 0 3.1c scrub starts 2024-03-11T21:07:39.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:38 smithi023 bash[21372]: cluster 2024-03-11T21:07:37.781627+0000 osd.5 (osd.5) 326 : cluster 0 3.1c scrub ok 2024-03-11T21:07:39.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:38 smithi023 bash[21372]: cluster 2024-03-11T21:07:38.252586+0000 mgr.y (mgr.24380) 624 : cluster 0 pgmap v698: 148 pgs: 148 active+clean; 11 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 2.2 MiB/s wr, 35 op/s 2024-03-11T21:07:39.030 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:38 smithi023 bash[29146]: cluster 2024-03-11T21:07:37.446563+0000 osd.3 (osd.3) 344 : cluster 0 2.19 scrub starts 2024-03-11T21:07:39.031 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:38 smithi023 bash[29146]: cluster 2024-03-11T21:07:37.448696+0000 osd.3 (osd.3) 345 : cluster 0 2.19 scrub ok 2024-03-11T21:07:39.031 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:38 smithi023 bash[29146]: cluster 2024-03-11T21:07:37.775447+0000 osd.5 (osd.5) 325 : cluster 0 3.1c scrub starts 2024-03-11T21:07:39.031 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:38 smithi023 bash[29146]: cluster 2024-03-11T21:07:37.781627+0000 osd.5 (osd.5) 326 : cluster 0 3.1c scrub ok 2024-03-11T21:07:39.031 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:38 smithi023 bash[29146]: cluster 2024-03-11T21:07:38.252586+0000 mgr.y (mgr.24380) 624 : cluster 0 pgmap v698: 148 pgs: 148 active+clean; 11 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 2.2 MiB/s wr, 35 op/s 2024-03-11T21:07:39.082 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:07:39.109 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:07:39.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:39.113 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:07:39.213 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:07:39.245 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:07:39.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:39.248 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:07:39.349 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:07:39.365 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 3 16 6893 6877 143.26 124.75 0.0113928 0.00696867 2024-03-11T21:07:39.374 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:07:39.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:39.376 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:07:39.477 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:39.505 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:39.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:39.507 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:07:39.608 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:07:39.634 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:07:39.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:39.637 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:07:39.738 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:07:39.747 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:39 smithi023 bash[21372]: cluster 2024-03-11T21:07:38.301944+0000 osd.2 (osd.2) 229 : cluster 0 6.f scrub starts 2024-03-11T21:07:39.747 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:39 smithi023 bash[21372]: cluster 2024-03-11T21:07:38.354480+0000 osd.1 (osd.1) 127 : cluster 0 2.d scrub starts 2024-03-11T21:07:39.748 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:39 smithi023 bash[21372]: cluster 2024-03-11T21:07:38.356543+0000 osd.1 (osd.1) 128 : cluster 0 2.d scrub ok 2024-03-11T21:07:39.748 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:39 smithi023 bash[21372]: cluster 2024-03-11T21:07:38.419787+0000 osd.3 (osd.3) 346 : cluster 0 2.1b scrub starts 2024-03-11T21:07:39.748 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:39 smithi023 bash[21372]: cluster 2024-03-11T21:07:38.421785+0000 osd.3 (osd.3) 347 : cluster 0 2.1b scrub ok 2024-03-11T21:07:39.748 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:39 smithi023 bash[21372]: cluster 2024-03-11T21:07:38.448231+0000 osd.2 (osd.2) 230 : cluster 0 6.f scrub ok 2024-03-11T21:07:39.748 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:39 smithi023 bash[21372]: cluster 2024-03-11T21:07:38.772376+0000 osd.0 (osd.0) 245 : cluster 0 4.1e scrub starts 2024-03-11T21:07:39.748 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:39 smithi023 bash[21372]: cluster 2024-03-11T21:07:38.775831+0000 osd.0 (osd.0) 246 : cluster 0 4.1e scrub ok 2024-03-11T21:07:39.748 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:39 smithi023 bash[29146]: cluster 2024-03-11T21:07:38.301944+0000 osd.2 (osd.2) 229 : cluster 0 6.f scrub starts 2024-03-11T21:07:39.748 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:39 smithi023 bash[29146]: cluster 2024-03-11T21:07:38.354480+0000 osd.1 (osd.1) 127 : cluster 0 2.d scrub starts 2024-03-11T21:07:39.748 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:39 smithi023 bash[29146]: cluster 2024-03-11T21:07:38.356543+0000 osd.1 (osd.1) 128 : cluster 0 2.d scrub ok 2024-03-11T21:07:39.748 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:39 smithi023 bash[29146]: cluster 2024-03-11T21:07:38.419787+0000 osd.3 (osd.3) 346 : cluster 0 2.1b scrub starts 2024-03-11T21:07:39.748 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:39 smithi023 bash[29146]: cluster 2024-03-11T21:07:38.421785+0000 osd.3 (osd.3) 347 : cluster 0 2.1b scrub ok 2024-03-11T21:07:39.748 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:39 smithi023 bash[29146]: cluster 2024-03-11T21:07:38.448231+0000 osd.2 (osd.2) 230 : cluster 0 6.f scrub ok 2024-03-11T21:07:39.748 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:39 smithi023 bash[29146]: cluster 2024-03-11T21:07:38.772376+0000 osd.0 (osd.0) 245 : cluster 0 4.1e scrub starts 2024-03-11T21:07:39.749 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:39 smithi023 bash[29146]: cluster 2024-03-11T21:07:38.775831+0000 osd.0 (osd.0) 246 : cluster 0 4.1e scrub ok 2024-03-11T21:07:39.764 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:07:39.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:39.768 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:07:39.869 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:07:39.892 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:07:39.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:39.895 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:07:39.996 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:40.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:39 smithi146 bash[23809]: cluster 2024-03-11T21:07:38.301944+0000 osd.2 (osd.2) 229 : cluster 0 6.f scrub starts 2024-03-11T21:07:40.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:39 smithi146 bash[23809]: cluster 2024-03-11T21:07:38.354480+0000 osd.1 (osd.1) 127 : cluster 0 2.d scrub starts 2024-03-11T21:07:40.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:39 smithi146 bash[23809]: cluster 2024-03-11T21:07:38.356543+0000 osd.1 (osd.1) 128 : cluster 0 2.d scrub ok 2024-03-11T21:07:40.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:39 smithi146 bash[23809]: cluster 2024-03-11T21:07:38.419787+0000 osd.3 (osd.3) 346 : cluster 0 2.1b scrub starts 2024-03-11T21:07:40.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:39 smithi146 bash[23809]: cluster 2024-03-11T21:07:38.421785+0000 osd.3 (osd.3) 347 : cluster 0 2.1b scrub ok 2024-03-11T21:07:40.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:39 smithi146 bash[23809]: cluster 2024-03-11T21:07:38.448231+0000 osd.2 (osd.2) 230 : cluster 0 6.f scrub ok 2024-03-11T21:07:40.006 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:39 smithi146 bash[23809]: cluster 2024-03-11T21:07:38.772376+0000 osd.0 (osd.0) 245 : cluster 0 4.1e scrub starts 2024-03-11T21:07:40.006 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:39 smithi146 bash[23809]: cluster 2024-03-11T21:07:38.775831+0000 osd.0 (osd.0) 246 : cluster 0 4.1e scrub ok 2024-03-11T21:07:40.023 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:40.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:40.025 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:07:40.126 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:40.154 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:40.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:40.157 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:07:40.257 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:40.288 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:40.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:40.290 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:07:40.365 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 4 16 9594 9578 149.645 168.812 0.00764258 0.0066727 2024-03-11T21:07:40.391 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:07:40.420 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:07:40.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:40.422 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:07:40.523 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:07:40.549 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:07:40.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:40.552 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:07:40.653 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:07:40.678 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:07:40.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:40.680 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:07:40.781 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:40.791 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:40 smithi146 bash[23809]: cluster 2024-03-11T21:07:39.349516+0000 osd.2 (osd.2) 231 : cluster 0 2.c scrub starts 2024-03-11T21:07:40.791 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:40 smithi146 bash[23809]: cluster 2024-03-11T21:07:39.353175+0000 osd.2 (osd.2) 232 : cluster 0 2.c scrub ok 2024-03-11T21:07:40.791 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:40 smithi146 bash[23809]: cluster 2024-03-11T21:07:39.369273+0000 osd.1 (osd.1) 129 : cluster 0 4.7 scrub starts 2024-03-11T21:07:40.791 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:40 smithi146 bash[23809]: cluster 2024-03-11T21:07:39.371629+0000 osd.1 (osd.1) 130 : cluster 0 4.7 scrub ok 2024-03-11T21:07:40.791 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:40 smithi146 bash[23809]: cluster 2024-03-11T21:07:40.253427+0000 mgr.y (mgr.24380) 625 : cluster 0 pgmap v699: 148 pgs: 148 active+clean; 423 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.7 KiB/s rd, 70 MiB/s wr, 1.13k op/s 2024-03-11T21:07:40.791 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:40 smithi146 bash[23809]: cluster 2024-03-11T21:07:40.274393+0000 mon.a (mon.0) 1627 : cluster 0 osdmap e315: 8 total, 8 up, 8 in 2024-03-11T21:07:40.798 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 6, 7, 1] out_osds: [] dead_osds: [] live_osds: [0, 2, 3, 5, 6, 7, 4, 1] 2024-03-11T21:07:40.799 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-11T21:07:40.799 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-03-11T21:07:40.799 INFO:tasks.thrashosds.thrasher:Setting osd 2 primary_affinity to 0.000000 2024-03-11T21:07:40.799 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd primary-affinity 2 0 2024-03-11T21:07:40.814 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:40.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:40.816 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:07:40.917 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:07:40.944 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:07:40.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:40.946 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:07:41.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:40 smithi023 bash[21372]: cluster 2024-03-11T21:07:39.349516+0000 osd.2 (osd.2) 231 : cluster 0 2.c scrub starts 2024-03-11T21:07:41.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:40 smithi023 bash[21372]: cluster 2024-03-11T21:07:39.353175+0000 osd.2 (osd.2) 232 : cluster 0 2.c scrub ok 2024-03-11T21:07:41.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:40 smithi023 bash[21372]: cluster 2024-03-11T21:07:39.369273+0000 osd.1 (osd.1) 129 : cluster 0 4.7 scrub starts 2024-03-11T21:07:41.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:40 smithi023 bash[21372]: cluster 2024-03-11T21:07:39.371629+0000 osd.1 (osd.1) 130 : cluster 0 4.7 scrub ok 2024-03-11T21:07:41.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:40 smithi023 bash[21372]: cluster 2024-03-11T21:07:40.253427+0000 mgr.y (mgr.24380) 625 : cluster 0 pgmap v699: 148 pgs: 148 active+clean; 423 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.7 KiB/s rd, 70 MiB/s wr, 1.13k op/s 2024-03-11T21:07:41.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:40 smithi023 bash[21372]: cluster 2024-03-11T21:07:40.274393+0000 mon.a (mon.0) 1627 : cluster 0 osdmap e315: 8 total, 8 up, 8 in 2024-03-11T21:07:41.031 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:40 smithi023 bash[29146]: cluster 2024-03-11T21:07:39.349516+0000 osd.2 (osd.2) 231 : cluster 0 2.c scrub starts 2024-03-11T21:07:41.031 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:40 smithi023 bash[29146]: cluster 2024-03-11T21:07:39.353175+0000 osd.2 (osd.2) 232 : cluster 0 2.c scrub ok 2024-03-11T21:07:41.031 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:40 smithi023 bash[29146]: cluster 2024-03-11T21:07:39.369273+0000 osd.1 (osd.1) 129 : cluster 0 4.7 scrub starts 2024-03-11T21:07:41.031 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:40 smithi023 bash[29146]: cluster 2024-03-11T21:07:39.371629+0000 osd.1 (osd.1) 130 : cluster 0 4.7 scrub ok 2024-03-11T21:07:41.031 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:40 smithi023 bash[29146]: cluster 2024-03-11T21:07:40.253427+0000 mgr.y (mgr.24380) 625 : cluster 0 pgmap v699: 148 pgs: 148 active+clean; 423 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.7 KiB/s rd, 70 MiB/s wr, 1.13k op/s 2024-03-11T21:07:41.031 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:40 smithi023 bash[29146]: cluster 2024-03-11T21:07:40.274393+0000 mon.a (mon.0) 1627 : cluster 0 osdmap e315: 8 total, 8 up, 8 in 2024-03-11T21:07:41.046 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:07:41.075 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:07:41.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:41.077 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:07:41.178 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:41.200 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:41.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:41.202 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:07:41.303 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:07:41.336 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:07:41.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:41.341 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:07:41.365 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 5 16 12085 12069 150.851 155.688 0.00587968 0.00662011 2024-03-11T21:07:41.441 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:41.462 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:41.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:41.465 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:07:41.471 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-11T21:07:41.565 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:07:41.607 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:07:41.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:41.612 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:07:41.713 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:41.742 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:41.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:41.744 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:07:41.845 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:07:41.855 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:41 smithi023 bash[21372]: cluster 2024-03-11T21:07:40.328432+0000 osd.2 (osd.2) 233 : cluster 0 6.17 scrub starts 2024-03-11T21:07:41.855 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:41 smithi023 bash[21372]: cluster 2024-03-11T21:07:40.457220+0000 osd.2 (osd.2) 234 : cluster 0 6.17 scrub ok 2024-03-11T21:07:41.856 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:41 smithi023 bash[21372]: cluster 2024-03-11T21:07:40.562252+0000 osd.6 (osd.6) 252 : cluster 0 2.7 scrub starts 2024-03-11T21:07:41.856 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:41 smithi023 bash[21372]: cluster 2024-03-11T21:07:40.565073+0000 osd.6 (osd.6) 253 : cluster 0 2.7 scrub ok 2024-03-11T21:07:41.856 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:41 smithi023 bash[21372]: cluster 2024-03-11T21:07:41.177269+0000 osd.7 (osd.7) 371 : cluster 0 3.d scrub starts 2024-03-11T21:07:41.856 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:41 smithi023 bash[21372]: cluster 2024-03-11T21:07:41.182187+0000 osd.7 (osd.7) 372 : cluster 0 3.d scrub ok 2024-03-11T21:07:41.857 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:41 smithi023 bash[29146]: cluster 2024-03-11T21:07:40.328432+0000 osd.2 (osd.2) 233 : cluster 0 6.17 scrub starts 2024-03-11T21:07:41.857 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:41 smithi023 bash[29146]: cluster 2024-03-11T21:07:40.457220+0000 osd.2 (osd.2) 234 : cluster 0 6.17 scrub ok 2024-03-11T21:07:41.857 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:41 smithi023 bash[29146]: cluster 2024-03-11T21:07:40.562252+0000 osd.6 (osd.6) 252 : cluster 0 2.7 scrub starts 2024-03-11T21:07:41.857 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:41 smithi023 bash[29146]: cluster 2024-03-11T21:07:40.565073+0000 osd.6 (osd.6) 253 : cluster 0 2.7 scrub ok 2024-03-11T21:07:41.857 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:41 smithi023 bash[29146]: cluster 2024-03-11T21:07:41.177269+0000 osd.7 (osd.7) 371 : cluster 0 3.d scrub starts 2024-03-11T21:07:41.857 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:41 smithi023 bash[29146]: cluster 2024-03-11T21:07:41.182187+0000 osd.7 (osd.7) 372 : cluster 0 3.d scrub ok 2024-03-11T21:07:41.869 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:07:41.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:41.872 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:07:41.973 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:07:41.997 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:07:41.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:42.000 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:07:42.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:41 smithi146 bash[23809]: cluster 2024-03-11T21:07:40.328432+0000 osd.2 (osd.2) 233 : cluster 0 6.17 scrub starts 2024-03-11T21:07:42.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:41 smithi146 bash[23809]: cluster 2024-03-11T21:07:40.457220+0000 osd.2 (osd.2) 234 : cluster 0 6.17 scrub ok 2024-03-11T21:07:42.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:41 smithi146 bash[23809]: cluster 2024-03-11T21:07:40.562252+0000 osd.6 (osd.6) 252 : cluster 0 2.7 scrub starts 2024-03-11T21:07:42.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:41 smithi146 bash[23809]: cluster 2024-03-11T21:07:40.565073+0000 osd.6 (osd.6) 253 : cluster 0 2.7 scrub ok 2024-03-11T21:07:42.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:41 smithi146 bash[23809]: cluster 2024-03-11T21:07:41.177269+0000 osd.7 (osd.7) 371 : cluster 0 3.d scrub starts 2024-03-11T21:07:42.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:41 smithi146 bash[23809]: cluster 2024-03-11T21:07:41.182187+0000 osd.7 (osd.7) 372 : cluster 0 3.d scrub ok 2024-03-11T21:07:42.101 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:07:42.132 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:07:42.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:42.135 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:07:42.236 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:07:42.262 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:07:42.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:42.265 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:07:42.365 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 6 16 14272 14256 148.488 136.688 0.00141784 0.00663965 2024-03-11T21:07:42.366 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:07:42.385 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:07:42.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:42.386 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:07:42.487 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:42.511 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:42.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:42.512 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:07:42.613 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:07:42.640 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:07:42.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:42.642 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:07:42.743 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:07:42.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:42 smithi146 bash[23809]: cluster 2024-03-11T21:07:41.363704+0000 osd.3 (osd.3) 348 : cluster 0 6.13 scrub starts 2024-03-11T21:07:42.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:42 smithi146 bash[23809]: cluster 2024-03-11T21:07:41.374669+0000 osd.2 (osd.2) 235 : cluster 0 2.10 scrub starts 2024-03-11T21:07:42.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:42 smithi146 bash[23809]: cluster 2024-03-11T21:07:41.377279+0000 osd.2 (osd.2) 236 : cluster 0 2.10 scrub ok 2024-03-11T21:07:42.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:42 smithi146 bash[23809]: cluster 2024-03-11T21:07:41.394035+0000 osd.1 (osd.1) 131 : cluster 0 2.9 scrub starts 2024-03-11T21:07:42.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:42 smithi146 bash[23809]: cluster 2024-03-11T21:07:41.397953+0000 osd.1 (osd.1) 132 : cluster 0 2.9 scrub ok 2024-03-11T21:07:42.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:42 smithi146 bash[23809]: cluster 2024-03-11T21:07:41.485042+0000 osd.3 (osd.3) 349 : cluster 0 6.13 scrub ok 2024-03-11T21:07:42.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:42 smithi146 bash[23809]: cluster 2024-03-11T21:07:42.254113+0000 mgr.y (mgr.24380) 626 : cluster 0 pgmap v701: 148 pgs: 148 active+clean; 544 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 733 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-03-11T21:07:42.778 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:07:42.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:42.781 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:07:42.882 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:42.908 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:42.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:42.910 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:07:43.011 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:43.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:42 smithi023 bash[21372]: cluster 2024-03-11T21:07:41.363704+0000 osd.3 (osd.3) 348 : cluster 0 6.13 scrub starts 2024-03-11T21:07:43.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:42 smithi023 bash[21372]: cluster 2024-03-11T21:07:41.374669+0000 osd.2 (osd.2) 235 : cluster 0 2.10 scrub starts 2024-03-11T21:07:43.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:42 smithi023 bash[21372]: cluster 2024-03-11T21:07:41.377279+0000 osd.2 (osd.2) 236 : cluster 0 2.10 scrub ok 2024-03-11T21:07:43.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:42 smithi023 bash[21372]: cluster 2024-03-11T21:07:41.394035+0000 osd.1 (osd.1) 131 : cluster 0 2.9 scrub starts 2024-03-11T21:07:43.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:42 smithi023 bash[21372]: cluster 2024-03-11T21:07:41.397953+0000 osd.1 (osd.1) 132 : cluster 0 2.9 scrub ok 2024-03-11T21:07:43.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:42 smithi023 bash[21372]: cluster 2024-03-11T21:07:41.485042+0000 osd.3 (osd.3) 349 : cluster 0 6.13 scrub ok 2024-03-11T21:07:43.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:42 smithi023 bash[21372]: cluster 2024-03-11T21:07:42.254113+0000 mgr.y (mgr.24380) 626 : cluster 0 pgmap v701: 148 pgs: 148 active+clean; 544 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 733 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-03-11T21:07:43.031 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:42 smithi023 bash[29146]: cluster 2024-03-11T21:07:41.363704+0000 osd.3 (osd.3) 348 : cluster 0 6.13 scrub starts 2024-03-11T21:07:43.031 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:42 smithi023 bash[29146]: cluster 2024-03-11T21:07:41.374669+0000 osd.2 (osd.2) 235 : cluster 0 2.10 scrub starts 2024-03-11T21:07:43.031 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:42 smithi023 bash[29146]: cluster 2024-03-11T21:07:41.377279+0000 osd.2 (osd.2) 236 : cluster 0 2.10 scrub ok 2024-03-11T21:07:43.031 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:42 smithi023 bash[29146]: cluster 2024-03-11T21:07:41.394035+0000 osd.1 (osd.1) 131 : cluster 0 2.9 scrub starts 2024-03-11T21:07:43.031 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:42 smithi023 bash[29146]: cluster 2024-03-11T21:07:41.397953+0000 osd.1 (osd.1) 132 : cluster 0 2.9 scrub ok 2024-03-11T21:07:43.031 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:42 smithi023 bash[29146]: cluster 2024-03-11T21:07:41.485042+0000 osd.3 (osd.3) 349 : cluster 0 6.13 scrub ok 2024-03-11T21:07:43.031 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:42 smithi023 bash[29146]: cluster 2024-03-11T21:07:42.254113+0000 mgr.y (mgr.24380) 626 : cluster 0 pgmap v701: 148 pgs: 148 active+clean; 544 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 733 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-03-11T21:07:43.033 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:43.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:43.035 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:07:43.136 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:07:43.168 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:07:43.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:43.171 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:07:43.272 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:07:43.300 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:07:43.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:43.304 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:07:43.365 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 7 16 15637 15621 139.461 85.3125 0.00859642 0.00716167 2024-03-11T21:07:43.405 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:43.430 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:43.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:43.433 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:07:43.534 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:07:43.559 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:07:43.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:43.563 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:07:43.664 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:07:43.679 INFO:teuthology.orchestra.run.smithi023.stdout:ERROR: (22) Invalid argument 2024-03-11T21:07:43.680 INFO:teuthology.orchestra.run.smithi023.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:07:43.699 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:07:43.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:43.703 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:07:43.804 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:07:43.831 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:07:43.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:43.834 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:07:43.916 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:43 smithi023 bash[21372]: cluster 2024-03-11T21:07:41.764850+0000 osd.0 (osd.0) 247 : cluster 0 3.3 scrub starts 2024-03-11T21:07:43.916 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:43 smithi023 bash[21372]: cluster 2024-03-11T21:07:41.769298+0000 osd.0 (osd.0) 248 : cluster 0 3.3 scrub ok 2024-03-11T21:07:43.917 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:43 smithi023 bash[21372]: cluster 2024-03-11T21:07:43.184886+0000 osd.7 (osd.7) 373 : cluster 0 6.1c scrub starts 2024-03-11T21:07:43.917 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:43 smithi023 bash[21372]: cluster 2024-03-11T21:07:43.187422+0000 osd.7 (osd.7) 374 : cluster 0 6.1c scrub ok 2024-03-11T21:07:43.917 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:43 smithi023 bash[29146]: cluster 2024-03-11T21:07:41.764850+0000 osd.0 (osd.0) 247 : cluster 0 3.3 scrub starts 2024-03-11T21:07:43.917 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:43 smithi023 bash[29146]: cluster 2024-03-11T21:07:41.769298+0000 osd.0 (osd.0) 248 : cluster 0 3.3 scrub ok 2024-03-11T21:07:43.917 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:43 smithi023 bash[29146]: cluster 2024-03-11T21:07:43.184886+0000 osd.7 (osd.7) 373 : cluster 0 6.1c scrub starts 2024-03-11T21:07:43.917 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:43 smithi023 bash[29146]: cluster 2024-03-11T21:07:43.187422+0000 osd.7 (osd.7) 374 : cluster 0 6.1c scrub ok 2024-03-11T21:07:43.935 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:07:43.962 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:07:43.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:43.965 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:07:44.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:43 smithi146 bash[23809]: cluster 2024-03-11T21:07:41.764850+0000 osd.0 (osd.0) 247 : cluster 0 3.3 scrub starts 2024-03-11T21:07:44.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:43 smithi146 bash[23809]: cluster 2024-03-11T21:07:41.769298+0000 osd.0 (osd.0) 248 : cluster 0 3.3 scrub ok 2024-03-11T21:07:44.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:43 smithi146 bash[23809]: cluster 2024-03-11T21:07:43.184886+0000 osd.7 (osd.7) 373 : cluster 0 6.1c scrub starts 2024-03-11T21:07:44.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:43 smithi146 bash[23809]: cluster 2024-03-11T21:07:43.187422+0000 osd.7 (osd.7) 374 : cluster 0 6.1c scrub ok 2024-03-11T21:07:44.012 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:07:44.066 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:44.095 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:44.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:44.097 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:07:44.198 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:07:44.222 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:07:44.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:44.225 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:07:44.326 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:07:44.350 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:07:44.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:44.353 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:07:44.365 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 8 16 17803 17787 138.949 135.375 0.00653174 0.00718918 2024-03-11T21:07:44.453 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:44.479 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:44.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:44.481 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:07:44.582 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:07:44.619 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:07:44.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:44.621 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:07:44.722 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:07:44.745 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:07:44.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:44.747 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:07:44.847 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:44.883 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:44.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:44.886 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:07:44.986 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:07:44.996 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:44 smithi023 bash[21372]: cluster 2024-03-11T21:07:43.345176+0000 osd.2 (osd.2) 237 : cluster 0 2.1 scrub starts 2024-03-11T21:07:44.996 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:44 smithi023 bash[21372]: cluster 2024-03-11T21:07:43.347179+0000 osd.2 (osd.2) 238 : cluster 0 2.1 scrub ok 2024-03-11T21:07:44.996 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:44 smithi023 bash[21372]: cluster 2024-03-11T21:07:43.588642+0000 osd.6 (osd.6) 254 : cluster 0 2.14 scrub starts 2024-03-11T21:07:44.997 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:44 smithi023 bash[21372]: cluster 2024-03-11T21:07:43.593277+0000 osd.6 (osd.6) 255 : cluster 0 2.14 scrub ok 2024-03-11T21:07:44.997 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:44 smithi023 bash[21372]: cluster 2024-03-11T21:07:44.224137+0000 osd.7 (osd.7) 375 : cluster 0 2.1d scrub starts 2024-03-11T21:07:44.997 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:44 smithi023 bash[21372]: cluster 2024-03-11T21:07:44.227432+0000 osd.7 (osd.7) 376 : cluster 0 2.1d scrub ok 2024-03-11T21:07:44.997 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:44 smithi023 bash[21372]: cluster 2024-03-11T21:07:44.254855+0000 mgr.y (mgr.24380) 627 : cluster 0 pgmap v702: 148 pgs: 148 active+clean; 544 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 68 MiB/s wr, 1.09k op/s 2024-03-11T21:07:44.997 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:44 smithi023 bash[29146]: cluster 2024-03-11T21:07:43.345176+0000 osd.2 (osd.2) 237 : cluster 0 2.1 scrub starts 2024-03-11T21:07:44.997 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:44 smithi023 bash[29146]: cluster 2024-03-11T21:07:43.347179+0000 osd.2 (osd.2) 238 : cluster 0 2.1 scrub ok 2024-03-11T21:07:44.997 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:44 smithi023 bash[29146]: cluster 2024-03-11T21:07:43.588642+0000 osd.6 (osd.6) 254 : cluster 0 2.14 scrub starts 2024-03-11T21:07:44.997 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:44 smithi023 bash[29146]: cluster 2024-03-11T21:07:43.593277+0000 osd.6 (osd.6) 255 : cluster 0 2.14 scrub ok 2024-03-11T21:07:44.997 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:44 smithi023 bash[29146]: cluster 2024-03-11T21:07:44.224137+0000 osd.7 (osd.7) 375 : cluster 0 2.1d scrub starts 2024-03-11T21:07:44.997 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:44 smithi023 bash[29146]: cluster 2024-03-11T21:07:44.227432+0000 osd.7 (osd.7) 376 : cluster 0 2.1d scrub ok 2024-03-11T21:07:44.997 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:44 smithi023 bash[29146]: cluster 2024-03-11T21:07:44.254855+0000 mgr.y (mgr.24380) 627 : cluster 0 pgmap v702: 148 pgs: 148 active+clean; 544 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 68 MiB/s wr, 1.09k op/s 2024-03-11T21:07:45.009 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:44 smithi146 bash[23809]: cluster 2024-03-11T21:07:43.345176+0000 osd.2 (osd.2) 237 : cluster 0 2.1 scrub starts 2024-03-11T21:07:45.009 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:44 smithi146 bash[23809]: cluster 2024-03-11T21:07:43.347179+0000 osd.2 (osd.2) 238 : cluster 0 2.1 scrub ok 2024-03-11T21:07:45.009 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:44 smithi146 bash[23809]: cluster 2024-03-11T21:07:43.588642+0000 osd.6 (osd.6) 254 : cluster 0 2.14 scrub starts 2024-03-11T21:07:45.009 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:44 smithi146 bash[23809]: cluster 2024-03-11T21:07:43.593277+0000 osd.6 (osd.6) 255 : cluster 0 2.14 scrub ok 2024-03-11T21:07:45.009 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:44 smithi146 bash[23809]: cluster 2024-03-11T21:07:44.224137+0000 osd.7 (osd.7) 375 : cluster 0 2.1d scrub starts 2024-03-11T21:07:45.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:44 smithi146 bash[23809]: cluster 2024-03-11T21:07:44.227432+0000 osd.7 (osd.7) 376 : cluster 0 2.1d scrub ok 2024-03-11T21:07:45.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:44 smithi146 bash[23809]: cluster 2024-03-11T21:07:44.254855+0000 mgr.y (mgr.24380) 627 : cluster 0 pgmap v702: 148 pgs: 148 active+clean; 544 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 68 MiB/s wr, 1.09k op/s 2024-03-11T21:07:45.013 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:07:45.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:45.015 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:07:45.116 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:45.140 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:45.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:45.144 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:07:45.245 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:07:45.277 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:07:45.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:45.282 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:07:45.365 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 9 16 19631 19615 136.204 114.25 0.00455843 0.00733418 2024-03-11T21:07:45.383 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:45.407 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:45.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:45.411 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:07:45.512 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:07:45.536 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:07:45.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:45.538 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:07:45.639 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:07:45.659 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:07:45.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:45.660 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:07:45.761 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:07:45.786 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:07:45.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:45.788 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:07:45.889 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:07:45.906 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:07:45.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:45.908 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:07:45.914 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:45 smithi023 bash[21372]: cluster 2024-03-11T21:07:43.789004+0000 osd.0 (osd.0) 249 : cluster 0 6.18 scrub starts 2024-03-11T21:07:45.914 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:45 smithi023 bash[21372]: cluster 2024-03-11T21:07:43.867056+0000 osd.0 (osd.0) 250 : cluster 0 6.18 scrub ok 2024-03-11T21:07:45.914 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:45 smithi023 bash[21372]: cluster 2024-03-11T21:07:45.175573+0000 osd.7 (osd.7) 377 : cluster 0 6.15 scrub starts 2024-03-11T21:07:45.914 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:45 smithi023 bash[21372]: cluster 2024-03-11T21:07:45.229572+0000 osd.7 (osd.7) 378 : cluster 0 6.15 scrub ok 2024-03-11T21:07:45.914 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:45 smithi023 bash[21372]: cluster 2024-03-11T21:07:45.286822+0000 mon.a (mon.0) 1628 : cluster 0 osdmap e316: 8 total, 8 up, 8 in 2024-03-11T21:07:45.915 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:45 smithi023 bash[29146]: cluster 2024-03-11T21:07:43.789004+0000 osd.0 (osd.0) 249 : cluster 0 6.18 scrub starts 2024-03-11T21:07:45.915 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:45 smithi023 bash[29146]: cluster 2024-03-11T21:07:43.867056+0000 osd.0 (osd.0) 250 : cluster 0 6.18 scrub ok 2024-03-11T21:07:45.915 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:45 smithi023 bash[29146]: cluster 2024-03-11T21:07:45.175573+0000 osd.7 (osd.7) 377 : cluster 0 6.15 scrub starts 2024-03-11T21:07:45.915 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:45 smithi023 bash[29146]: cluster 2024-03-11T21:07:45.229572+0000 osd.7 (osd.7) 378 : cluster 0 6.15 scrub ok 2024-03-11T21:07:45.915 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:45 smithi023 bash[29146]: cluster 2024-03-11T21:07:45.286822+0000 mon.a (mon.0) 1628 : cluster 0 osdmap e316: 8 total, 8 up, 8 in 2024-03-11T21:07:46.009 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:07:46.010 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:07:45 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:07:46.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:45 smithi146 bash[23809]: cluster 2024-03-11T21:07:43.789004+0000 osd.0 (osd.0) 249 : cluster 0 6.18 scrub starts 2024-03-11T21:07:46.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:45 smithi146 bash[23809]: cluster 2024-03-11T21:07:43.867056+0000 osd.0 (osd.0) 250 : cluster 0 6.18 scrub ok 2024-03-11T21:07:46.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:45 smithi146 bash[23809]: cluster 2024-03-11T21:07:45.175573+0000 osd.7 (osd.7) 377 : cluster 0 6.15 scrub starts 2024-03-11T21:07:46.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:45 smithi146 bash[23809]: cluster 2024-03-11T21:07:45.229572+0000 osd.7 (osd.7) 378 : cluster 0 6.15 scrub ok 2024-03-11T21:07:46.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:45 smithi146 bash[23809]: cluster 2024-03-11T21:07:45.286822+0000 mon.a (mon.0) 1628 : cluster 0 osdmap e316: 8 total, 8 up, 8 in 2024-03-11T21:07:46.028 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:07:46.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:46.030 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:07:46.130 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:07:46.159 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:07:46.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:46.161 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:07:46.262 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:07:46.289 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:07:46.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:46.293 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:07:46.366 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 10 16 21540 21524 134.513 119.312 0.00315522 0.00742731 2024-03-11T21:07:46.394 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:07:46.416 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:07:46.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:46.421 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:07:46.436 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-03-11T21:07:46.521 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:07:46.552 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:07:46.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:46.554 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:07:46.655 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:07:46.681 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:07:46.685 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:07:46.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:46.689 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:07:46.790 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:07:46.815 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:07:46.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:46.817 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:07:46.918 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:07:46.943 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:07:46.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:46.947 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:07:47.009 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:46 smithi146 bash[23809]: cluster 2024-03-11T21:07:44.806242+0000 osd.0 (osd.0) 251 : cluster 0 6.9 scrub starts 2024-03-11T21:07:47.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:46 smithi146 bash[23809]: cluster 2024-03-11T21:07:44.901219+0000 osd.0 (osd.0) 252 : cluster 0 6.9 scrub ok 2024-03-11T21:07:47.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:46 smithi146 bash[23809]: cluster 2024-03-11T21:07:45.353374+0000 osd.2 (osd.2) 239 : cluster 0 6.14 scrub starts 2024-03-11T21:07:47.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:46 smithi146 bash[23809]: cluster 2024-03-11T21:07:45.355414+0000 osd.2 (osd.2) 240 : cluster 0 6.14 scrub ok 2024-03-11T21:07:47.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:46 smithi146 bash[23809]: cluster 2024-03-11T21:07:45.370090+0000 osd.1 (osd.1) 133 : cluster 0 6.1 scrub starts 2024-03-11T21:07:47.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:46 smithi146 bash[23809]: cluster 2024-03-11T21:07:45.416769+0000 osd.1 (osd.1) 134 : cluster 0 6.1 scrub ok 2024-03-11T21:07:47.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:46 smithi146 bash[23809]: cluster 2024-03-11T21:07:45.525573+0000 osd.6 (osd.6) 256 : cluster 0 5.2 scrub starts 2024-03-11T21:07:47.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:46 smithi146 bash[23809]: cluster 2024-03-11T21:07:45.552522+0000 osd.6 (osd.6) 257 : cluster 0 5.2 scrub ok 2024-03-11T21:07:47.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:46 smithi146 bash[23809]: cluster 2024-03-11T21:07:45.681325+0000 osd.5 (osd.5) 327 : cluster 0 3.5 scrub starts 2024-03-11T21:07:47.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:46 smithi146 bash[23809]: audit 2024-03-11T21:07:45.685092+0000 mgr.y (mgr.24380) 628 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:07:47.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:46 smithi146 bash[23809]: cluster 2024-03-11T21:07:45.686945+0000 osd.5 (osd.5) 328 : cluster 0 3.5 scrub ok 2024-03-11T21:07:47.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:46 smithi146 bash[23809]: cluster 2024-03-11T21:07:46.198776+0000 osd.7 (osd.7) 379 : cluster 0 3.13 scrub starts 2024-03-11T21:07:47.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:46 smithi146 bash[23809]: cluster 2024-03-11T21:07:46.201236+0000 osd.7 (osd.7) 380 : cluster 0 3.13 scrub ok 2024-03-11T21:07:47.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:46 smithi146 bash[23809]: cluster 2024-03-11T21:07:46.255384+0000 mgr.y (mgr.24380) 629 : cluster 0 pgmap v704: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.1 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 142 MiB/s wr, 2.27k op/s 2024-03-11T21:07:47.048 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:47.076 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:47.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:47.077 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:07:47.159 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:46 smithi023 bash[21372]: cluster 2024-03-11T21:07:44.806242+0000 osd.0 (osd.0) 251 : cluster 0 6.9 scrub starts 2024-03-11T21:07:47.159 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:46 smithi023 bash[21372]: cluster 2024-03-11T21:07:44.901219+0000 osd.0 (osd.0) 252 : cluster 0 6.9 scrub ok 2024-03-11T21:07:47.159 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:46 smithi023 bash[21372]: cluster 2024-03-11T21:07:45.353374+0000 osd.2 (osd.2) 239 : cluster 0 6.14 scrub starts 2024-03-11T21:07:47.159 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:46 smithi023 bash[21372]: cluster 2024-03-11T21:07:45.355414+0000 osd.2 (osd.2) 240 : cluster 0 6.14 scrub ok 2024-03-11T21:07:47.159 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:46 smithi023 bash[21372]: cluster 2024-03-11T21:07:45.370090+0000 osd.1 (osd.1) 133 : cluster 0 6.1 scrub starts 2024-03-11T21:07:47.159 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:46 smithi023 bash[21372]: cluster 2024-03-11T21:07:45.416769+0000 osd.1 (osd.1) 134 : cluster 0 6.1 scrub ok 2024-03-11T21:07:47.160 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:46 smithi023 bash[21372]: cluster 2024-03-11T21:07:45.525573+0000 osd.6 (osd.6) 256 : cluster 0 5.2 scrub starts 2024-03-11T21:07:47.160 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:46 smithi023 bash[21372]: cluster 2024-03-11T21:07:45.552522+0000 osd.6 (osd.6) 257 : cluster 0 5.2 scrub ok 2024-03-11T21:07:47.160 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:46 smithi023 bash[21372]: cluster 2024-03-11T21:07:45.681325+0000 osd.5 (osd.5) 327 : cluster 0 3.5 scrub starts 2024-03-11T21:07:47.160 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:46 smithi023 bash[21372]: audit 2024-03-11T21:07:45.685092+0000 mgr.y (mgr.24380) 628 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:07:47.160 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:46 smithi023 bash[21372]: cluster 2024-03-11T21:07:45.686945+0000 osd.5 (osd.5) 328 : cluster 0 3.5 scrub ok 2024-03-11T21:07:47.160 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:46 smithi023 bash[21372]: cluster 2024-03-11T21:07:46.198776+0000 osd.7 (osd.7) 379 : cluster 0 3.13 scrub starts 2024-03-11T21:07:47.160 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:46 smithi023 bash[21372]: cluster 2024-03-11T21:07:46.201236+0000 osd.7 (osd.7) 380 : cluster 0 3.13 scrub ok 2024-03-11T21:07:47.160 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:46 smithi023 bash[21372]: cluster 2024-03-11T21:07:46.255384+0000 mgr.y (mgr.24380) 629 : cluster 0 pgmap v704: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.1 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 142 MiB/s wr, 2.27k op/s 2024-03-11T21:07:47.160 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:46 smithi023 bash[29146]: cluster 2024-03-11T21:07:44.806242+0000 osd.0 (osd.0) 251 : cluster 0 6.9 scrub starts 2024-03-11T21:07:47.160 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:46 smithi023 bash[29146]: cluster 2024-03-11T21:07:44.901219+0000 osd.0 (osd.0) 252 : cluster 0 6.9 scrub ok 2024-03-11T21:07:47.160 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:46 smithi023 bash[29146]: cluster 2024-03-11T21:07:45.353374+0000 osd.2 (osd.2) 239 : cluster 0 6.14 scrub starts 2024-03-11T21:07:47.161 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:46 smithi023 bash[29146]: cluster 2024-03-11T21:07:45.355414+0000 osd.2 (osd.2) 240 : cluster 0 6.14 scrub ok 2024-03-11T21:07:47.161 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:46 smithi023 bash[29146]: cluster 2024-03-11T21:07:45.370090+0000 osd.1 (osd.1) 133 : cluster 0 6.1 scrub starts 2024-03-11T21:07:47.161 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:46 smithi023 bash[29146]: cluster 2024-03-11T21:07:45.416769+0000 osd.1 (osd.1) 134 : cluster 0 6.1 scrub ok 2024-03-11T21:07:47.161 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:46 smithi023 bash[29146]: cluster 2024-03-11T21:07:45.525573+0000 osd.6 (osd.6) 256 : cluster 0 5.2 scrub starts 2024-03-11T21:07:47.161 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:46 smithi023 bash[29146]: cluster 2024-03-11T21:07:45.552522+0000 osd.6 (osd.6) 257 : cluster 0 5.2 scrub ok 2024-03-11T21:07:47.161 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:46 smithi023 bash[29146]: cluster 2024-03-11T21:07:45.681325+0000 osd.5 (osd.5) 327 : cluster 0 3.5 scrub starts 2024-03-11T21:07:47.161 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:46 smithi023 bash[29146]: audit 2024-03-11T21:07:45.685092+0000 mgr.y (mgr.24380) 628 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:07:47.161 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:46 smithi023 bash[29146]: cluster 2024-03-11T21:07:45.686945+0000 osd.5 (osd.5) 328 : cluster 0 3.5 scrub ok 2024-03-11T21:07:47.161 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:46 smithi023 bash[29146]: cluster 2024-03-11T21:07:46.198776+0000 osd.7 (osd.7) 379 : cluster 0 3.13 scrub starts 2024-03-11T21:07:47.161 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:46 smithi023 bash[29146]: cluster 2024-03-11T21:07:46.201236+0000 osd.7 (osd.7) 380 : cluster 0 3.13 scrub ok 2024-03-11T21:07:47.161 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:46 smithi023 bash[29146]: cluster 2024-03-11T21:07:46.255384+0000 mgr.y (mgr.24380) 629 : cluster 0 pgmap v704: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.1 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 142 MiB/s wr, 2.27k op/s 2024-03-11T21:07:47.178 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:47.206 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:47.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:47.210 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:07:47.310 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:07:47.335 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:07:47.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:47.341 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:07:47.365 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 11 15 23411 23396 132.92 117 0.00843355 0.00751663 2024-03-11T21:07:47.442 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:07:47.468 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:07:47.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:47.474 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:07:47.575 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:07:47.599 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:07:47.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:47.603 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:07:47.704 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:47.731 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:47.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:47.737 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:07:47.838 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:07:47.848 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:47 smithi146 bash[23809]: cluster 2024-03-11T21:07:46.370120+0000 osd.3 (osd.3) 350 : cluster 0 3.1e scrub starts 2024-03-11T21:07:47.848 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:47 smithi146 bash[23809]: cluster 2024-03-11T21:07:46.374355+0000 osd.3 (osd.3) 351 : cluster 0 3.1e scrub ok 2024-03-11T21:07:47.848 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:47 smithi146 bash[23809]: cluster 2024-03-11T21:07:47.157053+0000 osd.7 (osd.7) 381 : cluster 0 6.12 scrub starts 2024-03-11T21:07:47.848 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:47 smithi146 bash[23809]: cluster 2024-03-11T21:07:47.158669+0000 osd.7 (osd.7) 382 : cluster 0 6.12 scrub ok 2024-03-11T21:07:47.872 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:07:47.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:47.875 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:07:47.975 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:48.002 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:48.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:48.005 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:07:48.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:47 smithi023 bash[21372]: cluster 2024-03-11T21:07:46.370120+0000 osd.3 (osd.3) 350 : cluster 0 3.1e scrub starts 2024-03-11T21:07:48.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:47 smithi023 bash[21372]: cluster 2024-03-11T21:07:46.374355+0000 osd.3 (osd.3) 351 : cluster 0 3.1e scrub ok 2024-03-11T21:07:48.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:47 smithi023 bash[21372]: cluster 2024-03-11T21:07:47.157053+0000 osd.7 (osd.7) 381 : cluster 0 6.12 scrub starts 2024-03-11T21:07:48.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:47 smithi023 bash[21372]: cluster 2024-03-11T21:07:47.158669+0000 osd.7 (osd.7) 382 : cluster 0 6.12 scrub ok 2024-03-11T21:07:48.030 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:47 smithi023 bash[29146]: cluster 2024-03-11T21:07:46.370120+0000 osd.3 (osd.3) 350 : cluster 0 3.1e scrub starts 2024-03-11T21:07:48.030 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:47 smithi023 bash[29146]: cluster 2024-03-11T21:07:46.374355+0000 osd.3 (osd.3) 351 : cluster 0 3.1e scrub ok 2024-03-11T21:07:48.030 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:47 smithi023 bash[29146]: cluster 2024-03-11T21:07:47.157053+0000 osd.7 (osd.7) 381 : cluster 0 6.12 scrub starts 2024-03-11T21:07:48.030 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:47 smithi023 bash[29146]: cluster 2024-03-11T21:07:47.158669+0000 osd.7 (osd.7) 382 : cluster 0 6.12 scrub ok 2024-03-11T21:07:48.106 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:07:48.131 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:07:48.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:48.136 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:07:48.237 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:07:48.257 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:07:48.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:48.259 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:07:48.360 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:48.366 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 12 16 25166 25150 130.978 109.625 0.00197018 0.00761153 2024-03-11T21:07:48.377 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:48.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:48.379 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:07:48.480 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:07:48.497 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:07:48 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:07:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:07:48.515 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:07:48.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:48.519 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:07:48.619 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:48.655 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:48.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:48.658 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:07:48.759 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:07:48.794 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:07:48.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:48.799 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:07:48.899 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:07:48.909 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:48 smithi146 bash[23809]: cluster 2024-03-11T21:07:46.825469+0000 osd.0 (osd.0) 253 : cluster 0 6.a scrub starts 2024-03-11T21:07:48.910 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:48 smithi146 bash[23809]: cluster 2024-03-11T21:07:46.870639+0000 osd.0 (osd.0) 254 : cluster 0 6.a scrub ok 2024-03-11T21:07:48.910 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:48 smithi146 bash[23809]: cluster 2024-03-11T21:07:47.444906+0000 osd.1 (osd.1) 135 : cluster 0 3.4 scrub starts 2024-03-11T21:07:48.910 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:48 smithi146 bash[23809]: cluster 2024-03-11T21:07:47.448521+0000 osd.1 (osd.1) 136 : cluster 0 3.4 scrub ok 2024-03-11T21:07:48.910 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:48 smithi146 bash[23809]: cluster 2024-03-11T21:07:48.256084+0000 mgr.y (mgr.24380) 630 : cluster 0 pgmap v705: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.2 GiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 95 MiB/s wr, 1.52k op/s 2024-03-11T21:07:48.922 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:07:48.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:48.925 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:07:49.025 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:07:49.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:48 smithi023 bash[21372]: cluster 2024-03-11T21:07:46.825469+0000 osd.0 (osd.0) 253 : cluster 0 6.a scrub starts 2024-03-11T21:07:49.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:48 smithi023 bash[21372]: cluster 2024-03-11T21:07:46.870639+0000 osd.0 (osd.0) 254 : cluster 0 6.a scrub ok 2024-03-11T21:07:49.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:48 smithi023 bash[21372]: cluster 2024-03-11T21:07:47.444906+0000 osd.1 (osd.1) 135 : cluster 0 3.4 scrub starts 2024-03-11T21:07:49.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:48 smithi023 bash[21372]: cluster 2024-03-11T21:07:47.448521+0000 osd.1 (osd.1) 136 : cluster 0 3.4 scrub ok 2024-03-11T21:07:49.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:48 smithi023 bash[21372]: cluster 2024-03-11T21:07:48.256084+0000 mgr.y (mgr.24380) 630 : cluster 0 pgmap v705: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.2 GiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 95 MiB/s wr, 1.52k op/s 2024-03-11T21:07:49.030 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:48 smithi023 bash[29146]: cluster 2024-03-11T21:07:46.825469+0000 osd.0 (osd.0) 253 : cluster 0 6.a scrub starts 2024-03-11T21:07:49.030 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:48 smithi023 bash[29146]: cluster 2024-03-11T21:07:46.870639+0000 osd.0 (osd.0) 254 : cluster 0 6.a scrub ok 2024-03-11T21:07:49.030 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:48 smithi023 bash[29146]: cluster 2024-03-11T21:07:47.444906+0000 osd.1 (osd.1) 135 : cluster 0 3.4 scrub starts 2024-03-11T21:07:49.030 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:48 smithi023 bash[29146]: cluster 2024-03-11T21:07:47.448521+0000 osd.1 (osd.1) 136 : cluster 0 3.4 scrub ok 2024-03-11T21:07:49.030 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:48 smithi023 bash[29146]: cluster 2024-03-11T21:07:48.256084+0000 mgr.y (mgr.24380) 630 : cluster 0 pgmap v705: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.2 GiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 95 MiB/s wr, 1.52k op/s 2024-03-11T21:07:49.045 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:07:49.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:49.047 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:07:49.147 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:49.171 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:49.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:49.173 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:07:49.273 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:07:49.298 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:07:49.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:49.300 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:07:49.366 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 13 15 26127 26112 125.527 60.125 0.0259622 0.00795795 2024-03-11T21:07:49.401 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:49.424 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:49.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:49.429 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:07:49.529 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:07:49.557 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:07:49.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:49.559 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:07:49.659 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:07:49.688 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:07:49.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:49.691 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:07:49.791 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:49.817 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:49.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:49.821 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:07:49.921 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:07:49.931 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:49 smithi023 bash[21372]: cluster 2024-03-11T21:07:48.493813+0000 osd.1 (osd.1) 137 : cluster 0 3.1a scrub starts 2024-03-11T21:07:49.932 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:49 smithi023 bash[21372]: cluster 2024-03-11T21:07:48.498476+0000 osd.2 (osd.2) 241 : cluster 0 4.1c scrub starts 2024-03-11T21:07:49.932 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:49 smithi023 bash[21372]: cluster 2024-03-11T21:07:48.503131+0000 osd.1 (osd.1) 138 : cluster 0 3.1a scrub ok 2024-03-11T21:07:49.932 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:49 smithi023 bash[21372]: cluster 2024-03-11T21:07:48.513221+0000 osd.2 (osd.2) 242 : cluster 0 4.1c scrub ok 2024-03-11T21:07:49.932 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:49 smithi023 bash[21372]: cluster 2024-03-11T21:07:48.807040+0000 osd.0 (osd.0) 255 : cluster 0 6.16 scrub starts 2024-03-11T21:07:49.932 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:49 smithi023 bash[21372]: cluster 2024-03-11T21:07:49.149180+0000 osd.7 (osd.7) 383 : cluster 0 3.14 scrub starts 2024-03-11T21:07:49.932 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:49 smithi023 bash[21372]: cluster 2024-03-11T21:07:49.153326+0000 osd.7 (osd.7) 384 : cluster 0 3.14 scrub ok 2024-03-11T21:07:49.932 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:49 smithi023 bash[29146]: cluster 2024-03-11T21:07:48.493813+0000 osd.1 (osd.1) 137 : cluster 0 3.1a scrub starts 2024-03-11T21:07:49.932 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:49 smithi023 bash[29146]: cluster 2024-03-11T21:07:48.498476+0000 osd.2 (osd.2) 241 : cluster 0 4.1c scrub starts 2024-03-11T21:07:49.933 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:49 smithi023 bash[29146]: cluster 2024-03-11T21:07:48.503131+0000 osd.1 (osd.1) 138 : cluster 0 3.1a scrub ok 2024-03-11T21:07:49.933 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:49 smithi023 bash[29146]: cluster 2024-03-11T21:07:48.513221+0000 osd.2 (osd.2) 242 : cluster 0 4.1c scrub ok 2024-03-11T21:07:49.933 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:49 smithi023 bash[29146]: cluster 2024-03-11T21:07:48.807040+0000 osd.0 (osd.0) 255 : cluster 0 6.16 scrub starts 2024-03-11T21:07:49.933 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:49 smithi023 bash[29146]: cluster 2024-03-11T21:07:49.149180+0000 osd.7 (osd.7) 383 : cluster 0 3.14 scrub starts 2024-03-11T21:07:49.933 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:49 smithi023 bash[29146]: cluster 2024-03-11T21:07:49.153326+0000 osd.7 (osd.7) 384 : cluster 0 3.14 scrub ok 2024-03-11T21:07:49.948 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:07:49.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:49.952 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:07:50.053 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:07:50.063 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:49 smithi146 bash[23809]: cluster 2024-03-11T21:07:48.493813+0000 osd.1 (osd.1) 137 : cluster 0 3.1a scrub starts 2024-03-11T21:07:50.063 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:49 smithi146 bash[23809]: cluster 2024-03-11T21:07:48.498476+0000 osd.2 (osd.2) 241 : cluster 0 4.1c scrub starts 2024-03-11T21:07:50.063 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:49 smithi146 bash[23809]: cluster 2024-03-11T21:07:48.503131+0000 osd.1 (osd.1) 138 : cluster 0 3.1a scrub ok 2024-03-11T21:07:50.063 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:49 smithi146 bash[23809]: cluster 2024-03-11T21:07:48.513221+0000 osd.2 (osd.2) 242 : cluster 0 4.1c scrub ok 2024-03-11T21:07:50.063 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:49 smithi146 bash[23809]: cluster 2024-03-11T21:07:48.807040+0000 osd.0 (osd.0) 255 : cluster 0 6.16 scrub starts 2024-03-11T21:07:50.063 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:49 smithi146 bash[23809]: cluster 2024-03-11T21:07:49.149180+0000 osd.7 (osd.7) 383 : cluster 0 3.14 scrub starts 2024-03-11T21:07:50.063 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:49 smithi146 bash[23809]: cluster 2024-03-11T21:07:49.153326+0000 osd.7 (osd.7) 384 : cluster 0 3.14 scrub ok 2024-03-11T21:07:50.078 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:07:50.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:50.081 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:07:50.181 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:07:50.212 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:07:50.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:50.214 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:07:50.315 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:50.339 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:50.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:50.342 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:07:50.366 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 14 16 27626 27610 123.248 93.625 0.00659873 0.00810761 2024-03-11T21:07:50.442 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:50.466 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:50.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:50.468 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:07:50.569 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:07:50.572 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:07:50.605 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:07:50.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:50.608 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:07:50.709 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:07:50.807 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:07:50.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:50.811 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:07:50.900 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:50 smithi023 bash[29146]: cluster 2024-03-11T21:07:48.864442+0000 osd.0 (osd.0) 256 : cluster 0 6.16 scrub ok 2024-03-11T21:07:50.900 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:50 smithi023 bash[29146]: cluster 2024-03-11T21:07:49.529653+0000 osd.2 (osd.2) 243 : cluster 0 3.9 scrub starts 2024-03-11T21:07:50.900 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:50 smithi023 bash[29146]: cluster 2024-03-11T21:07:49.533413+0000 osd.2 (osd.2) 244 : cluster 0 3.9 scrub ok 2024-03-11T21:07:50.900 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:50 smithi023 bash[29146]: cluster 2024-03-11T21:07:50.257020+0000 mgr.y (mgr.24380) 631 : cluster 0 pgmap v706: 148 pgs: 148 active+clean; 1.6 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 1.0 KiB/s rd, 118 MiB/s wr, 1.89k op/s 2024-03-11T21:07:50.900 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:50 smithi023 bash[29146]: cluster 2024-03-11T21:07:50.488487+0000 osd.1 (osd.1) 139 : cluster 0 4.9 scrub starts 2024-03-11T21:07:50.900 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:50 smithi023 bash[29146]: cluster 2024-03-11T21:07:50.490045+0000 osd.1 (osd.1) 140 : cluster 0 4.9 scrub ok 2024-03-11T21:07:50.901 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:50 smithi023 bash[29146]: audit 2024-03-11T21:07:50.675493+0000 mon.a (mon.0) 1629 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:07:50.901 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:50 smithi023 bash[21372]: cluster 2024-03-11T21:07:48.864442+0000 osd.0 (osd.0) 256 : cluster 0 6.16 scrub ok 2024-03-11T21:07:50.901 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:50 smithi023 bash[21372]: cluster 2024-03-11T21:07:49.529653+0000 osd.2 (osd.2) 243 : cluster 0 3.9 scrub starts 2024-03-11T21:07:50.901 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:50 smithi023 bash[21372]: cluster 2024-03-11T21:07:49.533413+0000 osd.2 (osd.2) 244 : cluster 0 3.9 scrub ok 2024-03-11T21:07:50.901 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:50 smithi023 bash[21372]: cluster 2024-03-11T21:07:50.257020+0000 mgr.y (mgr.24380) 631 : cluster 0 pgmap v706: 148 pgs: 148 active+clean; 1.6 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 1.0 KiB/s rd, 118 MiB/s wr, 1.89k op/s 2024-03-11T21:07:50.901 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:50 smithi023 bash[21372]: cluster 2024-03-11T21:07:50.488487+0000 osd.1 (osd.1) 139 : cluster 0 4.9 scrub starts 2024-03-11T21:07:50.901 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:50 smithi023 bash[21372]: cluster 2024-03-11T21:07:50.490045+0000 osd.1 (osd.1) 140 : cluster 0 4.9 scrub ok 2024-03-11T21:07:50.901 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:50 smithi023 bash[21372]: audit 2024-03-11T21:07:50.675493+0000 mon.a (mon.0) 1629 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:07:50.912 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:07:50.945 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:07:50.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:50.950 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:07:51.050 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:07:51.075 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:07:51.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:51.078 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:07:51.179 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:51.190 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:50 smithi146 bash[23809]: cluster 2024-03-11T21:07:48.864442+0000 osd.0 (osd.0) 256 : cluster 0 6.16 scrub ok 2024-03-11T21:07:51.190 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:50 smithi146 bash[23809]: cluster 2024-03-11T21:07:49.529653+0000 osd.2 (osd.2) 243 : cluster 0 3.9 scrub starts 2024-03-11T21:07:51.190 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:50 smithi146 bash[23809]: cluster 2024-03-11T21:07:49.533413+0000 osd.2 (osd.2) 244 : cluster 0 3.9 scrub ok 2024-03-11T21:07:51.190 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:50 smithi146 bash[23809]: cluster 2024-03-11T21:07:50.257020+0000 mgr.y (mgr.24380) 631 : cluster 0 pgmap v706: 148 pgs: 148 active+clean; 1.6 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 1.0 KiB/s rd, 118 MiB/s wr, 1.89k op/s 2024-03-11T21:07:51.190 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:50 smithi146 bash[23809]: cluster 2024-03-11T21:07:50.488487+0000 osd.1 (osd.1) 139 : cluster 0 4.9 scrub starts 2024-03-11T21:07:51.190 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:50 smithi146 bash[23809]: cluster 2024-03-11T21:07:50.490045+0000 osd.1 (osd.1) 140 : cluster 0 4.9 scrub ok 2024-03-11T21:07:51.190 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:50 smithi146 bash[23809]: audit 2024-03-11T21:07:50.675493+0000 mon.a (mon.0) 1629 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:07:51.214 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:51.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:51.216 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:07:51.317 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:07:51.338 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:07:51.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:51.340 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:07:51.366 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 15 16 29340 29324 122.172 107.125 0.0175044 0.00817624 2024-03-11T21:07:51.441 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:51.473 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:51.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:51.477 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:07:51.578 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:07:51.606 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:07:51.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:51.609 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:07:51.709 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:51.736 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:51.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:51.739 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:07:51.839 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:07:51.866 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:07:51.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:51.870 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:07:51.970 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:07:51.979 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:51 smithi146 bash[23809]: cluster 2024-03-11T21:07:50.777866+0000 osd.0 (osd.0) 257 : cluster 0 4.b scrub starts 2024-03-11T21:07:51.979 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:51 smithi146 bash[23809]: cluster 2024-03-11T21:07:50.781420+0000 osd.0 (osd.0) 258 : cluster 0 4.b scrub ok 2024-03-11T21:07:51.992 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:07:51.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:51.994 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:07:52.095 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:07:52.119 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:07:52.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:52.120 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:07:52.142 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:51 smithi023 bash[21372]: cluster 2024-03-11T21:07:50.777866+0000 osd.0 (osd.0) 257 : cluster 0 4.b scrub starts 2024-03-11T21:07:52.142 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:51 smithi023 bash[21372]: cluster 2024-03-11T21:07:50.781420+0000 osd.0 (osd.0) 258 : cluster 0 4.b scrub ok 2024-03-11T21:07:52.143 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:51 smithi023 bash[29146]: cluster 2024-03-11T21:07:50.777866+0000 osd.0 (osd.0) 257 : cluster 0 4.b scrub starts 2024-03-11T21:07:52.143 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:51 smithi023 bash[29146]: cluster 2024-03-11T21:07:50.781420+0000 osd.0 (osd.0) 258 : cluster 0 4.b scrub ok 2024-03-11T21:07:52.221 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:07:52.260 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:07:52.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:52.263 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:07:52.364 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:07:52.366 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 16 16 30679 30663 119.767 83.6875 0.00478321 0.0083435 2024-03-11T21:07:52.391 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:07:52.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:52.393 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:07:52.494 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:52.514 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:52.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:52.516 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:07:52.617 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:07:52.652 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:07:52.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:52.658 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:07:52.759 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:52.786 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:52.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:52.790 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:07:52.890 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:07:52.900 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:52 smithi023 bash[29146]: cluster 2024-03-11T21:07:51.764941+0000 osd.5 (osd.5) 329 : cluster 0 6.7 scrub starts 2024-03-11T21:07:52.900 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:52 smithi023 bash[29146]: cluster 2024-03-11T21:07:52.163729+0000 osd.7 (osd.7) 385 : cluster 0 2.b scrub starts 2024-03-11T21:07:52.900 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:52 smithi023 bash[29146]: cluster 2024-03-11T21:07:52.165105+0000 osd.7 (osd.7) 386 : cluster 0 2.b scrub ok 2024-03-11T21:07:52.900 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:52 smithi023 bash[29146]: cluster 2024-03-11T21:07:52.257739+0000 mgr.y (mgr.24380) 632 : cluster 0 pgmap v707: 148 pgs: 148 active+clean; 1.7 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 115 MiB/s wr, 1.84k op/s 2024-03-11T21:07:52.901 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:52 smithi023 bash[21372]: cluster 2024-03-11T21:07:51.764941+0000 osd.5 (osd.5) 329 : cluster 0 6.7 scrub starts 2024-03-11T21:07:52.901 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:52 smithi023 bash[21372]: cluster 2024-03-11T21:07:52.163729+0000 osd.7 (osd.7) 385 : cluster 0 2.b scrub starts 2024-03-11T21:07:52.901 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:52 smithi023 bash[21372]: cluster 2024-03-11T21:07:52.165105+0000 osd.7 (osd.7) 386 : cluster 0 2.b scrub ok 2024-03-11T21:07:52.901 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:52 smithi023 bash[21372]: cluster 2024-03-11T21:07:52.257739+0000 mgr.y (mgr.24380) 632 : cluster 0 pgmap v707: 148 pgs: 148 active+clean; 1.7 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 115 MiB/s wr, 1.84k op/s 2024-03-11T21:07:52.918 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:07:52.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:52.920 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:07:53.021 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:07:53.047 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:07:53.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:53.050 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:07:53.150 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:07:53.182 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:07:53.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:53.187 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:07:53.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:52 smithi146 bash[23809]: cluster 2024-03-11T21:07:51.764941+0000 osd.5 (osd.5) 329 : cluster 0 6.7 scrub starts 2024-03-11T21:07:53.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:52 smithi146 bash[23809]: cluster 2024-03-11T21:07:52.163729+0000 osd.7 (osd.7) 385 : cluster 0 2.b scrub starts 2024-03-11T21:07:53.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:52 smithi146 bash[23809]: cluster 2024-03-11T21:07:52.165105+0000 osd.7 (osd.7) 386 : cluster 0 2.b scrub ok 2024-03-11T21:07:53.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:52 smithi146 bash[23809]: cluster 2024-03-11T21:07:52.257739+0000 mgr.y (mgr.24380) 632 : cluster 0 pgmap v707: 148 pgs: 148 active+clean; 1.7 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 115 MiB/s wr, 1.84k op/s 2024-03-11T21:07:53.288 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:07:53.323 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:07:53.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:53.326 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:07:53.366 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 17 16 31741 31725 116.626 66.375 0.00886008 0.00856836 2024-03-11T21:07:53.428 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:53.455 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:53.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:53.457 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:07:53.557 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:53.581 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:53.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:53.586 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:07:53.687 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:53.715 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:53.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:53.717 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:07:53.818 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:07:53.846 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:07:53.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:53.849 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:07:53.950 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:07:53.976 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:07:53.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:53.981 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:07:54.009 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:53 smithi146 bash[23809]: cluster 2024-03-11T21:07:51.859880+0000 osd.5 (osd.5) 330 : cluster 0 6.7 scrub ok 2024-03-11T21:07:54.009 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:53 smithi146 bash[23809]: cluster 2024-03-11T21:07:53.180986+0000 osd.7 (osd.7) 387 : cluster 0 1.0 scrub starts 2024-03-11T21:07:54.009 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:53 smithi146 bash[23809]: cluster 2024-03-11T21:07:53.187849+0000 osd.7 (osd.7) 388 : cluster 0 1.0 scrub ok 2024-03-11T21:07:54.082 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:07:54.108 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:07:54.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:54.114 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:07:54.214 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:07:54.236 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:07:54.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:54.237 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:07:54.338 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:07:54.362 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:07:54.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:54.365 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:07:54.366 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:53 smithi023 bash[21372]: cluster 2024-03-11T21:07:51.859880+0000 osd.5 (osd.5) 330 : cluster 0 6.7 scrub ok 2024-03-11T21:07:54.366 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:53 smithi023 bash[21372]: cluster 2024-03-11T21:07:53.180986+0000 osd.7 (osd.7) 387 : cluster 0 1.0 scrub starts 2024-03-11T21:07:54.366 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:53 smithi023 bash[21372]: cluster 2024-03-11T21:07:53.187849+0000 osd.7 (osd.7) 388 : cluster 0 1.0 scrub ok 2024-03-11T21:07:54.366 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:53 smithi023 bash[29146]: cluster 2024-03-11T21:07:51.859880+0000 osd.5 (osd.5) 330 : cluster 0 6.7 scrub ok 2024-03-11T21:07:54.366 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:53 smithi023 bash[29146]: cluster 2024-03-11T21:07:53.180986+0000 osd.7 (osd.7) 387 : cluster 0 1.0 scrub starts 2024-03-11T21:07:54.367 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:53 smithi023 bash[29146]: cluster 2024-03-11T21:07:53.187849+0000 osd.7 (osd.7) 388 : cluster 0 1.0 scrub ok 2024-03-11T21:07:54.367 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 18 16 33115 33099 114.916 85.875 0.00471389 0.00858669 2024-03-11T21:07:54.465 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:07:54.497 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:07:54.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:54.500 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:07:54.600 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:07:54.630 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:07:54.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:54.635 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:07:54.736 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:54.759 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:54.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:54.761 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:07:54.862 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:07:54.887 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:07:54.891 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:54 smithi023 bash[21372]: cluster 2024-03-11T21:07:52.832642+0000 osd.0 (osd.0) 259 : cluster 0 3.6 scrub starts 2024-03-11T21:07:54.891 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:54 smithi023 bash[21372]: cluster 2024-03-11T21:07:52.837960+0000 osd.0 (osd.0) 260 : cluster 0 3.6 scrub ok 2024-03-11T21:07:54.891 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:54 smithi023 bash[21372]: cluster 2024-03-11T21:07:54.258279+0000 mgr.y (mgr.24380) 633 : cluster 0 pgmap v708: 148 pgs: 148 active+clean; 1.7 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 115 MiB/s wr, 1.84k op/s 2024-03-11T21:07:54.891 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:54 smithi023 bash[29146]: cluster 2024-03-11T21:07:52.832642+0000 osd.0 (osd.0) 259 : cluster 0 3.6 scrub starts 2024-03-11T21:07:54.891 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:54 smithi023 bash[29146]: cluster 2024-03-11T21:07:52.837960+0000 osd.0 (osd.0) 260 : cluster 0 3.6 scrub ok 2024-03-11T21:07:54.891 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:54 smithi023 bash[29146]: cluster 2024-03-11T21:07:54.258279+0000 mgr.y (mgr.24380) 633 : cluster 0 pgmap v708: 148 pgs: 148 active+clean; 1.7 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 115 MiB/s wr, 1.84k op/s 2024-03-11T21:07:54.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:54.892 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:07:54.993 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:55.001 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:54 smithi146 bash[23809]: cluster 2024-03-11T21:07:52.832642+0000 osd.0 (osd.0) 259 : cluster 0 3.6 scrub starts 2024-03-11T21:07:55.001 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:54 smithi146 bash[23809]: cluster 2024-03-11T21:07:52.837960+0000 osd.0 (osd.0) 260 : cluster 0 3.6 scrub ok 2024-03-11T21:07:55.001 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:54 smithi146 bash[23809]: cluster 2024-03-11T21:07:54.258279+0000 mgr.y (mgr.24380) 633 : cluster 0 pgmap v708: 148 pgs: 148 active+clean; 1.7 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 115 MiB/s wr, 1.84k op/s 2024-03-11T21:07:55.015 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:55.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:55.018 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:07:55.119 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:07:55.149 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:07:55.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:55.153 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:07:55.254 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:07:55.286 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:07:55.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:55.289 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:07:55.366 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 19 16 34855 34839 114.591 108.75 0.00313262 0.00872048 2024-03-11T21:07:55.390 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:07:55.415 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:07:55.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:55.418 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:07:55.421 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:07:55.520 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:55.539 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:55.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:55.540 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:07:55.641 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:07:55.666 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:07:55.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:55.668 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:07:55.769 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:07:55.797 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:07:55.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:55.800 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:07:55.807 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:07:55 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:07:55.900 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:07:55.909 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:55 smithi023 bash[21372]: cluster 2024-03-11T21:07:54.596303+0000 osd.2 (osd.2) 245 : cluster 0 2.0 scrub starts 2024-03-11T21:07:55.909 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:55 smithi023 bash[21372]: cluster 2024-03-11T21:07:54.604966+0000 osd.2 (osd.2) 246 : cluster 0 2.0 scrub ok 2024-03-11T21:07:55.909 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:55 smithi023 bash[21372]: cluster 2024-03-11T21:07:55.692445+0000 osd.6 (osd.6) 258 : cluster 0 4.e scrub starts 2024-03-11T21:07:55.909 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:55 smithi023 bash[21372]: cluster 2024-03-11T21:07:55.698173+0000 osd.6 (osd.6) 259 : cluster 0 4.e scrub ok 2024-03-11T21:07:55.910 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:55 smithi023 bash[29146]: cluster 2024-03-11T21:07:54.596303+0000 osd.2 (osd.2) 245 : cluster 0 2.0 scrub starts 2024-03-11T21:07:55.910 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:55 smithi023 bash[29146]: cluster 2024-03-11T21:07:54.604966+0000 osd.2 (osd.2) 246 : cluster 0 2.0 scrub ok 2024-03-11T21:07:55.910 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:55 smithi023 bash[29146]: cluster 2024-03-11T21:07:55.692445+0000 osd.6 (osd.6) 258 : cluster 0 4.e scrub starts 2024-03-11T21:07:55.910 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:55 smithi023 bash[29146]: cluster 2024-03-11T21:07:55.698173+0000 osd.6 (osd.6) 259 : cluster 0 4.e scrub ok 2024-03-11T21:07:55.924 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:07:55.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:55.926 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:07:55.953 INFO:teuthology.orchestra.run.smithi023.stdout:osd.0: {} 2024-03-11T21:07:55.993 INFO:teuthology.orchestra.run.smithi023.stdout:osd.1: {} 2024-03-11T21:07:56.009 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:55 smithi146 bash[23809]: cluster 2024-03-11T21:07:54.596303+0000 osd.2 (osd.2) 245 : cluster 0 2.0 scrub starts 2024-03-11T21:07:56.009 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:55 smithi146 bash[23809]: cluster 2024-03-11T21:07:54.604966+0000 osd.2 (osd.2) 246 : cluster 0 2.0 scrub ok 2024-03-11T21:07:56.009 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:55 smithi146 bash[23809]: cluster 2024-03-11T21:07:55.692445+0000 osd.6 (osd.6) 258 : cluster 0 4.e scrub starts 2024-03-11T21:07:56.009 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:55 smithi146 bash[23809]: cluster 2024-03-11T21:07:55.698173+0000 osd.6 (osd.6) 259 : cluster 0 4.e scrub ok 2024-03-11T21:07:56.028 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:07:56.037 INFO:teuthology.orchestra.run.smithi023.stdout:osd.2: {} 2024-03-11T21:07:56.063 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:07:56.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:56.069 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:07:56.080 INFO:teuthology.orchestra.run.smithi023.stdout:osd.3: {} 2024-03-11T21:07:56.123 INFO:teuthology.orchestra.run.smithi023.stdout:osd.4: {} 2024-03-11T21:07:56.164 INFO:teuthology.orchestra.run.smithi023.stdout:osd.5: {} 2024-03-11T21:07:56.169 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:07:56.200 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:07:56.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:56.203 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:07:56.203 INFO:teuthology.orchestra.run.smithi023.stdout:osd.6: {} 2024-03-11T21:07:56.241 INFO:teuthology.orchestra.run.smithi023.stdout:osd.7: {} 2024-03-11T21:07:56.243 INFO:teuthology.orchestra.run.smithi023.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:07:56.243 INFO:teuthology.orchestra.run.smithi023.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:07:56.243 INFO:teuthology.orchestra.run.smithi023.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:07:56.244 INFO:teuthology.orchestra.run.smithi023.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:07:56.244 INFO:teuthology.orchestra.run.smithi023.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:07:56.244 INFO:teuthology.orchestra.run.smithi023.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:07:56.244 INFO:teuthology.orchestra.run.smithi023.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:07:56.244 INFO:teuthology.orchestra.run.smithi023.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:07:56.304 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:07:56.335 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:07:56.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:56.338 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:07:56.366 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:2024-03-11T21:07:56.375962+0000 min lat: 0.00119683 max lat: 0.491549 avg lat: 0.0088505 2024-03-11T21:07:56.366 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-11T21:07:56.366 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 20 16 36129 36113 112.842 79.625 0.00226939 0.0088505 2024-03-11T21:07:56.439 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:07:56.461 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:07:56.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:56.463 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:07:56.564 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:07:56.589 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:07:56.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:56.591 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:07:56.691 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:56.718 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:56.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:56.721 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:07:56.821 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:07:56.845 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:07:56.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:56.847 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:07:56.948 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:07:56.977 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:07:56.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:56.981 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:07:57.009 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:56 smithi146 bash[23809]: audit 2024-03-11T21:07:55.693343+0000 mgr.y (mgr.24380) 634 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:07:57.009 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:56 smithi146 bash[23809]: audit 2024-03-11T21:07:55.905719+0000 mon.a (mon.0) 1630 : audit 0 from='client.? 172.21.15.23:0/3274983958' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:07:57.009 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:56 smithi146 bash[23809]: cluster 2024-03-11T21:07:56.106903+0000 osd.7 (osd.7) 389 : cluster 0 6.8 scrub starts 2024-03-11T21:07:57.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:56 smithi146 bash[23809]: cluster 2024-03-11T21:07:56.111585+0000 osd.7 (osd.7) 390 : cluster 0 6.8 scrub ok 2024-03-11T21:07:57.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:56 smithi146 bash[23809]: cluster 2024-03-11T21:07:56.259136+0000 mgr.y (mgr.24380) 635 : cluster 0 pgmap v709: 148 pgs: 148 active+clean; 2.1 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 932 B/s rd, 96 MiB/s wr, 1.53k op/s 2024-03-11T21:07:57.082 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:07:57.114 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:07:57.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:57.116 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:07:57.217 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:07:57.245 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:07:57.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:57.248 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:07:57.349 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:57.366 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 21 16 36979 36963 109.998 53.125 0.00342136 0.00903475 2024-03-11T21:07:57.375 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:57.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:57.377 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:07:57.417 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:56 smithi023 bash[29146]: audit 2024-03-11T21:07:55.693343+0000 mgr.y (mgr.24380) 634 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:07:57.417 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:56 smithi023 bash[29146]: audit 2024-03-11T21:07:55.905719+0000 mon.a (mon.0) 1630 : audit 0 from='client.? 172.21.15.23:0/3274983958' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:07:57.418 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:56 smithi023 bash[29146]: cluster 2024-03-11T21:07:56.106903+0000 osd.7 (osd.7) 389 : cluster 0 6.8 scrub starts 2024-03-11T21:07:57.418 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:56 smithi023 bash[29146]: cluster 2024-03-11T21:07:56.111585+0000 osd.7 (osd.7) 390 : cluster 0 6.8 scrub ok 2024-03-11T21:07:57.418 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:56 smithi023 bash[29146]: cluster 2024-03-11T21:07:56.259136+0000 mgr.y (mgr.24380) 635 : cluster 0 pgmap v709: 148 pgs: 148 active+clean; 2.1 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 932 B/s rd, 96 MiB/s wr, 1.53k op/s 2024-03-11T21:07:57.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:56 smithi023 bash[21372]: audit 2024-03-11T21:07:55.693343+0000 mgr.y (mgr.24380) 634 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:07:57.419 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:56 smithi023 bash[21372]: audit 2024-03-11T21:07:55.905719+0000 mon.a (mon.0) 1630 : audit 0 from='client.? 172.21.15.23:0/3274983958' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:07:57.419 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:56 smithi023 bash[21372]: cluster 2024-03-11T21:07:56.106903+0000 osd.7 (osd.7) 389 : cluster 0 6.8 scrub starts 2024-03-11T21:07:57.419 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:56 smithi023 bash[21372]: cluster 2024-03-11T21:07:56.111585+0000 osd.7 (osd.7) 390 : cluster 0 6.8 scrub ok 2024-03-11T21:07:57.419 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:56 smithi023 bash[21372]: cluster 2024-03-11T21:07:56.259136+0000 mgr.y (mgr.24380) 635 : cluster 0 pgmap v709: 148 pgs: 148 active+clean; 2.1 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 932 B/s rd, 96 MiB/s wr, 1.53k op/s 2024-03-11T21:07:57.478 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:07:57.506 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:07:57.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:57.509 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:07:57.609 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:07:57.637 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:07:57.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:57.639 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:07:57.740 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:07:57.758 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:07:57.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:57.759 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:07:57.860 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:57.891 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:57.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:57.895 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:07:57.996 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:58.020 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:07:58.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:58.029 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:07:58.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:57 smithi023 bash[21372]: cluster 2024-03-11T21:07:57.479944+0000 osd.6 (osd.6) 260 : cluster 0 4.13 scrub starts 2024-03-11T21:07:58.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:57 smithi023 bash[21372]: cluster 2024-03-11T21:07:57.484156+0000 osd.6 (osd.6) 261 : cluster 0 4.13 scrub ok 2024-03-11T21:07:58.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:57 smithi023 bash[21372]: cluster 2024-03-11T21:07:57.618750+0000 osd.7 (osd.7) 391 : cluster 0 purged_snaps scrub starts 2024-03-11T21:07:58.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:57 smithi023 bash[21372]: cluster 2024-03-11T21:07:57.618818+0000 osd.7 (osd.7) 392 : cluster 0 purged_snaps scrub ok 2024-03-11T21:07:58.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:57 smithi023 bash[21372]: cluster 2024-03-11T21:07:57.622063+0000 osd.1 (osd.1) 141 : cluster 0 4.4 scrub starts 2024-03-11T21:07:58.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:57 smithi023 bash[21372]: cluster 2024-03-11T21:07:57.635656+0000 osd.1 (osd.1) 142 : cluster 0 4.4 scrub ok 2024-03-11T21:07:58.030 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:57 smithi023 bash[29146]: cluster 2024-03-11T21:07:57.479944+0000 osd.6 (osd.6) 260 : cluster 0 4.13 scrub starts 2024-03-11T21:07:58.030 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:57 smithi023 bash[29146]: cluster 2024-03-11T21:07:57.484156+0000 osd.6 (osd.6) 261 : cluster 0 4.13 scrub ok 2024-03-11T21:07:58.031 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:57 smithi023 bash[29146]: cluster 2024-03-11T21:07:57.618750+0000 osd.7 (osd.7) 391 : cluster 0 purged_snaps scrub starts 2024-03-11T21:07:58.031 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:57 smithi023 bash[29146]: cluster 2024-03-11T21:07:57.618818+0000 osd.7 (osd.7) 392 : cluster 0 purged_snaps scrub ok 2024-03-11T21:07:58.031 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:57 smithi023 bash[29146]: cluster 2024-03-11T21:07:57.622063+0000 osd.1 (osd.1) 141 : cluster 0 4.4 scrub starts 2024-03-11T21:07:58.031 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:57 smithi023 bash[29146]: cluster 2024-03-11T21:07:57.635656+0000 osd.1 (osd.1) 142 : cluster 0 4.4 scrub ok 2024-03-11T21:07:58.130 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:07:58.158 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:07:58.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:58.166 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:07:58.259 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:57 smithi146 bash[23809]: cluster 2024-03-11T21:07:57.479944+0000 osd.6 (osd.6) 260 : cluster 0 4.13 scrub starts 2024-03-11T21:07:58.259 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:57 smithi146 bash[23809]: cluster 2024-03-11T21:07:57.484156+0000 osd.6 (osd.6) 261 : cluster 0 4.13 scrub ok 2024-03-11T21:07:58.259 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:57 smithi146 bash[23809]: cluster 2024-03-11T21:07:57.618750+0000 osd.7 (osd.7) 391 : cluster 0 purged_snaps scrub starts 2024-03-11T21:07:58.259 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:57 smithi146 bash[23809]: cluster 2024-03-11T21:07:57.618818+0000 osd.7 (osd.7) 392 : cluster 0 purged_snaps scrub ok 2024-03-11T21:07:58.259 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:57 smithi146 bash[23809]: cluster 2024-03-11T21:07:57.622063+0000 osd.1 (osd.1) 141 : cluster 0 4.4 scrub starts 2024-03-11T21:07:58.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:57 smithi146 bash[23809]: cluster 2024-03-11T21:07:57.635656+0000 osd.1 (osd.1) 142 : cluster 0 4.4 scrub ok 2024-03-11T21:07:58.267 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:07:58.292 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:07:58.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:58.295 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:07:58.367 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 22 16 38362 38346 108.927 86.4375 0.00786027 0.00917482 2024-03-11T21:07:58.396 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:58.421 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:07:58.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:58.426 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:07:58.526 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:07:58.530 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:07:58 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:07:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:07:58.560 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:07:58.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:58.563 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:07:58.664 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:07:58.691 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:07:58.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:58.693 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:07:58.794 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:07:58.812 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:07:58.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:58.814 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:07:58.915 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:07:58.943 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:07:58.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:58.946 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:07:59.009 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:58 smithi146 bash[23809]: cluster 2024-03-11T21:07:56.865684+0000 osd.5 (osd.5) 331 : cluster 0 2.16 scrub starts 2024-03-11T21:07:59.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:58 smithi146 bash[23809]: cluster 2024-03-11T21:07:56.868475+0000 osd.5 (osd.5) 332 : cluster 0 2.16 scrub ok 2024-03-11T21:07:59.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:58 smithi146 bash[23809]: cluster 2024-03-11T21:07:57.316998+0000 osd.3 (osd.3) 352 : cluster 0 6.1b scrub starts 2024-03-11T21:07:59.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:58 smithi146 bash[23809]: cluster 2024-03-11T21:07:57.433160+0000 osd.3 (osd.3) 353 : cluster 0 6.1b scrub ok 2024-03-11T21:07:59.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:58 smithi146 bash[23809]: cluster 2024-03-11T21:07:57.814382+0000 osd.2 (osd.2) 247 : cluster 0 2.e scrub starts 2024-03-11T21:07:59.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:58 smithi146 bash[23809]: cluster 2024-03-11T21:07:57.816875+0000 osd.2 (osd.2) 248 : cluster 0 2.e scrub ok 2024-03-11T21:07:59.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:58 smithi146 bash[23809]: cluster 2024-03-11T21:07:57.844746+0000 mon.a (mon.0) 1631 : cluster 0 osdmap e317: 8 total, 8 up, 8 in 2024-03-11T21:07:59.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:58 smithi146 bash[23809]: cluster 2024-03-11T21:07:58.085306+0000 osd.7 (osd.7) 393 : cluster 0 3.15 scrub starts 2024-03-11T21:07:59.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:58 smithi146 bash[23809]: cluster 2024-03-11T21:07:58.088376+0000 osd.7 (osd.7) 394 : cluster 0 3.15 scrub ok 2024-03-11T21:07:59.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:58 smithi146 bash[23809]: cluster 2024-03-11T21:07:58.259600+0000 mgr.y (mgr.24380) 636 : cluster 0 pgmap v711: 148 pgs: 148 active+clean; 2.1 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 96 MiB/s wr, 1.54k op/s 2024-03-11T21:07:59.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:58 smithi146 bash[23809]: cluster 2024-03-11T21:07:58.610718+0000 osd.1 (osd.1) 143 : cluster 0 2.a scrub starts 2024-03-11T21:07:59.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:58 smithi146 bash[23809]: cluster 2024-03-11T21:07:58.612414+0000 osd.1 (osd.1) 144 : cluster 0 2.a scrub ok 2024-03-11T21:07:59.047 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:07:59.069 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:07:59.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:59.072 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:07:59.172 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:07:59.193 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:07:59.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:59.195 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:07:59.280 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:58 smithi023 bash[21372]: cluster 2024-03-11T21:07:56.865684+0000 osd.5 (osd.5) 331 : cluster 0 2.16 scrub starts 2024-03-11T21:07:59.280 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:58 smithi023 bash[21372]: cluster 2024-03-11T21:07:56.868475+0000 osd.5 (osd.5) 332 : cluster 0 2.16 scrub ok 2024-03-11T21:07:59.280 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:58 smithi023 bash[21372]: cluster 2024-03-11T21:07:57.316998+0000 osd.3 (osd.3) 352 : cluster 0 6.1b scrub starts 2024-03-11T21:07:59.280 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:58 smithi023 bash[21372]: cluster 2024-03-11T21:07:57.433160+0000 osd.3 (osd.3) 353 : cluster 0 6.1b scrub ok 2024-03-11T21:07:59.280 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:58 smithi023 bash[21372]: cluster 2024-03-11T21:07:57.814382+0000 osd.2 (osd.2) 247 : cluster 0 2.e scrub starts 2024-03-11T21:07:59.280 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:58 smithi023 bash[21372]: cluster 2024-03-11T21:07:57.816875+0000 osd.2 (osd.2) 248 : cluster 0 2.e scrub ok 2024-03-11T21:07:59.280 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:58 smithi023 bash[21372]: cluster 2024-03-11T21:07:57.844746+0000 mon.a (mon.0) 1631 : cluster 0 osdmap e317: 8 total, 8 up, 8 in 2024-03-11T21:07:59.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:58 smithi023 bash[21372]: cluster 2024-03-11T21:07:58.085306+0000 osd.7 (osd.7) 393 : cluster 0 3.15 scrub starts 2024-03-11T21:07:59.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:58 smithi023 bash[21372]: cluster 2024-03-11T21:07:58.088376+0000 osd.7 (osd.7) 394 : cluster 0 3.15 scrub ok 2024-03-11T21:07:59.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:58 smithi023 bash[21372]: cluster 2024-03-11T21:07:58.259600+0000 mgr.y (mgr.24380) 636 : cluster 0 pgmap v711: 148 pgs: 148 active+clean; 2.1 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 96 MiB/s wr, 1.54k op/s 2024-03-11T21:07:59.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:58 smithi023 bash[21372]: cluster 2024-03-11T21:07:58.610718+0000 osd.1 (osd.1) 143 : cluster 0 2.a scrub starts 2024-03-11T21:07:59.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:58 smithi023 bash[21372]: cluster 2024-03-11T21:07:58.612414+0000 osd.1 (osd.1) 144 : cluster 0 2.a scrub ok 2024-03-11T21:07:59.281 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:58 smithi023 bash[29146]: cluster 2024-03-11T21:07:56.865684+0000 osd.5 (osd.5) 331 : cluster 0 2.16 scrub starts 2024-03-11T21:07:59.281 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:58 smithi023 bash[29146]: cluster 2024-03-11T21:07:56.868475+0000 osd.5 (osd.5) 332 : cluster 0 2.16 scrub ok 2024-03-11T21:07:59.281 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:58 smithi023 bash[29146]: cluster 2024-03-11T21:07:57.316998+0000 osd.3 (osd.3) 352 : cluster 0 6.1b scrub starts 2024-03-11T21:07:59.281 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:58 smithi023 bash[29146]: cluster 2024-03-11T21:07:57.433160+0000 osd.3 (osd.3) 353 : cluster 0 6.1b scrub ok 2024-03-11T21:07:59.282 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:58 smithi023 bash[29146]: cluster 2024-03-11T21:07:57.814382+0000 osd.2 (osd.2) 247 : cluster 0 2.e scrub starts 2024-03-11T21:07:59.282 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:58 smithi023 bash[29146]: cluster 2024-03-11T21:07:57.816875+0000 osd.2 (osd.2) 248 : cluster 0 2.e scrub ok 2024-03-11T21:07:59.282 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:58 smithi023 bash[29146]: cluster 2024-03-11T21:07:57.844746+0000 mon.a (mon.0) 1631 : cluster 0 osdmap e317: 8 total, 8 up, 8 in 2024-03-11T21:07:59.282 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:58 smithi023 bash[29146]: cluster 2024-03-11T21:07:58.085306+0000 osd.7 (osd.7) 393 : cluster 0 3.15 scrub starts 2024-03-11T21:07:59.282 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:58 smithi023 bash[29146]: cluster 2024-03-11T21:07:58.088376+0000 osd.7 (osd.7) 394 : cluster 0 3.15 scrub ok 2024-03-11T21:07:59.282 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:58 smithi023 bash[29146]: cluster 2024-03-11T21:07:58.259600+0000 mgr.y (mgr.24380) 636 : cluster 0 pgmap v711: 148 pgs: 148 active+clean; 2.1 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 96 MiB/s wr, 1.54k op/s 2024-03-11T21:07:59.282 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:58 smithi023 bash[29146]: cluster 2024-03-11T21:07:58.610718+0000 osd.1 (osd.1) 143 : cluster 0 2.a scrub starts 2024-03-11T21:07:59.282 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:58 smithi023 bash[29146]: cluster 2024-03-11T21:07:58.612414+0000 osd.1 (osd.1) 144 : cluster 0 2.a scrub ok 2024-03-11T21:07:59.296 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:07:59.334 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:07:59.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:59.337 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:07:59.367 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 23 16 39576 39560 107.489 75.875 0.00619402 0.00929739 2024-03-11T21:07:59.438 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:07:59.466 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:07:59.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:59.469 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:07:59.570 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:07:59.597 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:07:59.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:59.600 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:07:59.701 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:07:59.721 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:07:59.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:59.722 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:07:59.823 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:07:59.850 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:07:59.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:59.853 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:07:59.954 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:07:59.983 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:07:59.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:07:59.986 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:08:00.087 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:08:00.097 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:59 smithi023 bash[29146]: cluster 2024-03-11T21:07:57.885330+0000 osd.0 (osd.0) 261 : cluster 0 3.17 scrub starts 2024-03-11T21:08:00.097 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:59 smithi023 bash[29146]: cluster 2024-03-11T21:07:57.887313+0000 osd.0 (osd.0) 262 : cluster 0 3.17 scrub ok 2024-03-11T21:08:00.097 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:59 smithi023 bash[29146]: cluster 2024-03-11T21:07:59.095033+0000 osd.7 (osd.7) 395 : cluster 0 2.5 scrub starts 2024-03-11T21:08:00.097 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:59 smithi023 bash[29146]: cluster 2024-03-11T21:07:59.256983+0000 osd.7 (osd.7) 396 : cluster 0 2.5 scrub ok 2024-03-11T21:08:00.098 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:59 smithi023 bash[29146]: cluster 2024-03-11T21:07:59.459442+0000 osd.6 (osd.6) 262 : cluster 0 3.10 scrub starts 2024-03-11T21:08:00.098 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:07:59 smithi023 bash[29146]: cluster 2024-03-11T21:07:59.463848+0000 osd.6 (osd.6) 263 : cluster 0 3.10 scrub ok 2024-03-11T21:08:00.098 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:59 smithi023 bash[21372]: cluster 2024-03-11T21:07:57.885330+0000 osd.0 (osd.0) 261 : cluster 0 3.17 scrub starts 2024-03-11T21:08:00.098 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:59 smithi023 bash[21372]: cluster 2024-03-11T21:07:57.887313+0000 osd.0 (osd.0) 262 : cluster 0 3.17 scrub ok 2024-03-11T21:08:00.098 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:59 smithi023 bash[21372]: cluster 2024-03-11T21:07:59.095033+0000 osd.7 (osd.7) 395 : cluster 0 2.5 scrub starts 2024-03-11T21:08:00.098 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:59 smithi023 bash[21372]: cluster 2024-03-11T21:07:59.256983+0000 osd.7 (osd.7) 396 : cluster 0 2.5 scrub ok 2024-03-11T21:08:00.098 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:59 smithi023 bash[21372]: cluster 2024-03-11T21:07:59.459442+0000 osd.6 (osd.6) 262 : cluster 0 3.10 scrub starts 2024-03-11T21:08:00.098 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:07:59 smithi023 bash[21372]: cluster 2024-03-11T21:07:59.463848+0000 osd.6 (osd.6) 263 : cluster 0 3.10 scrub ok 2024-03-11T21:08:00.119 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:08:00.120 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:59 smithi146 bash[23809]: cluster 2024-03-11T21:07:57.885330+0000 osd.0 (osd.0) 261 : cluster 0 3.17 scrub starts 2024-03-11T21:08:00.120 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:59 smithi146 bash[23809]: cluster 2024-03-11T21:07:57.887313+0000 osd.0 (osd.0) 262 : cluster 0 3.17 scrub ok 2024-03-11T21:08:00.120 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:59 smithi146 bash[23809]: cluster 2024-03-11T21:07:59.095033+0000 osd.7 (osd.7) 395 : cluster 0 2.5 scrub starts 2024-03-11T21:08:00.120 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:59 smithi146 bash[23809]: cluster 2024-03-11T21:07:59.256983+0000 osd.7 (osd.7) 396 : cluster 0 2.5 scrub ok 2024-03-11T21:08:00.120 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:59 smithi146 bash[23809]: cluster 2024-03-11T21:07:59.459442+0000 osd.6 (osd.6) 262 : cluster 0 3.10 scrub starts 2024-03-11T21:08:00.121 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:07:59 smithi146 bash[23809]: cluster 2024-03-11T21:07:59.463848+0000 osd.6 (osd.6) 263 : cluster 0 3.10 scrub ok 2024-03-11T21:08:00.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:00.121 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:08:00.222 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:00.249 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:00.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:00.252 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:08:00.353 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:00.367 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 24 16 40938 40922 106.557 85.125 0.0115008 0.00937909 2024-03-11T21:08:00.386 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:00.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:00.389 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:08:00.490 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:00.523 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:00.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:00.526 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:08:00.626 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:08:00.653 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:08:00.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:00.657 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:08:00.758 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:08:00.778 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:08:00.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:00.780 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:08:00.881 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:08:00.916 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:08:00.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:00.919 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:08:01.020 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:01.052 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:01.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:01.055 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:08:01.156 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:08:01.184 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:08:01.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:01.187 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:08:01.288 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:01.313 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:01.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:01.318 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:08:01.367 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 25 16 42387 42371 105.917 90.5625 0.0135755 0.00943563 2024-03-11T21:08:01.418 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:08:01.429 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:01 smithi023 bash[21372]: cluster 2024-03-11T21:08:00.260327+0000 mgr.y (mgr.24380) 637 : cluster 0 pgmap v712: 148 pgs: 148 active+clean; 2.4 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 85 MiB/s wr, 1.36k op/s 2024-03-11T21:08:01.429 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:01 smithi023 bash[21372]: cluster 2024-03-11T21:08:00.281481+0000 mon.a (mon.0) 1632 : cluster 0 osdmap e318: 8 total, 8 up, 8 in 2024-03-11T21:08:01.429 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:01 smithi023 bash[21372]: cluster 2024-03-11T21:08:00.318415+0000 osd.3 (osd.3) 354 : cluster 0 6.11 scrub starts 2024-03-11T21:08:01.430 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:01 smithi023 bash[21372]: cluster 2024-03-11T21:08:00.343685+0000 osd.3 (osd.3) 355 : cluster 0 6.11 scrub ok 2024-03-11T21:08:01.430 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:01 smithi023 bash[21372]: cluster 2024-03-11T21:08:00.624802+0000 osd.1 (osd.1) 145 : cluster 0 2.4 scrub starts 2024-03-11T21:08:01.430 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:01 smithi023 bash[21372]: cluster 2024-03-11T21:08:00.625898+0000 osd.1 (osd.1) 146 : cluster 0 2.4 scrub ok 2024-03-11T21:08:01.430 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:01 smithi023 bash[21372]: cluster 2024-03-11T21:08:00.879349+0000 osd.0 (osd.0) 263 : cluster 0 3.1b scrub starts 2024-03-11T21:08:01.430 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:01 smithi023 bash[29146]: cluster 2024-03-11T21:08:00.260327+0000 mgr.y (mgr.24380) 637 : cluster 0 pgmap v712: 148 pgs: 148 active+clean; 2.4 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 85 MiB/s wr, 1.36k op/s 2024-03-11T21:08:01.430 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:01 smithi023 bash[29146]: cluster 2024-03-11T21:08:00.281481+0000 mon.a (mon.0) 1632 : cluster 0 osdmap e318: 8 total, 8 up, 8 in 2024-03-11T21:08:01.430 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:01 smithi023 bash[29146]: cluster 2024-03-11T21:08:00.318415+0000 osd.3 (osd.3) 354 : cluster 0 6.11 scrub starts 2024-03-11T21:08:01.430 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:01 smithi023 bash[29146]: cluster 2024-03-11T21:08:00.343685+0000 osd.3 (osd.3) 355 : cluster 0 6.11 scrub ok 2024-03-11T21:08:01.430 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:01 smithi023 bash[29146]: cluster 2024-03-11T21:08:00.624802+0000 osd.1 (osd.1) 145 : cluster 0 2.4 scrub starts 2024-03-11T21:08:01.430 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:01 smithi023 bash[29146]: cluster 2024-03-11T21:08:00.625898+0000 osd.1 (osd.1) 146 : cluster 0 2.4 scrub ok 2024-03-11T21:08:01.430 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:01 smithi023 bash[29146]: cluster 2024-03-11T21:08:00.879349+0000 osd.0 (osd.0) 263 : cluster 0 3.1b scrub starts 2024-03-11T21:08:01.447 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:08:01.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:01.450 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:08:01.551 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:08:01.560 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:01 smithi146 bash[23809]: cluster 2024-03-11T21:08:00.260327+0000 mgr.y (mgr.24380) 637 : cluster 0 pgmap v712: 148 pgs: 148 active+clean; 2.4 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 85 MiB/s wr, 1.36k op/s 2024-03-11T21:08:01.560 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:01 smithi146 bash[23809]: cluster 2024-03-11T21:08:00.281481+0000 mon.a (mon.0) 1632 : cluster 0 osdmap e318: 8 total, 8 up, 8 in 2024-03-11T21:08:01.560 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:01 smithi146 bash[23809]: cluster 2024-03-11T21:08:00.318415+0000 osd.3 (osd.3) 354 : cluster 0 6.11 scrub starts 2024-03-11T21:08:01.560 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:01 smithi146 bash[23809]: cluster 2024-03-11T21:08:00.343685+0000 osd.3 (osd.3) 355 : cluster 0 6.11 scrub ok 2024-03-11T21:08:01.560 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:01 smithi146 bash[23809]: cluster 2024-03-11T21:08:00.624802+0000 osd.1 (osd.1) 145 : cluster 0 2.4 scrub starts 2024-03-11T21:08:01.561 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:01 smithi146 bash[23809]: cluster 2024-03-11T21:08:00.625898+0000 osd.1 (osd.1) 146 : cluster 0 2.4 scrub ok 2024-03-11T21:08:01.561 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:01 smithi146 bash[23809]: cluster 2024-03-11T21:08:00.879349+0000 osd.0 (osd.0) 263 : cluster 0 3.1b scrub starts 2024-03-11T21:08:01.575 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:08:01.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:01.579 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:08:01.680 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:08:01.699 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:08:01.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:01.701 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:08:01.802 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:08:01.827 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:08:01.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:01.832 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:08:01.933 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:08:01.964 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:08:01.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:01.970 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:08:02.070 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:08:02.089 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:08:02.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:02.091 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:08:02.191 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:02.217 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:02.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:02.224 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:08:02.325 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:02.355 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:02.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:02.360 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:08:02.367 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 26 16 43790 43774 105.216 87.6875 0.00665026 0.00949836 2024-03-11T21:08:02.461 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:02.484 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:02.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:02.486 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:08:02.509 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:02 smithi146 bash[23809]: cluster 2024-03-11T21:08:00.885110+0000 osd.0 (osd.0) 264 : cluster 0 3.1b scrub ok 2024-03-11T21:08:02.509 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:02 smithi146 bash[23809]: cluster 2024-03-11T21:08:01.358287+0000 osd.3 (osd.3) 356 : cluster 0 4.1a scrub starts 2024-03-11T21:08:02.509 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:02 smithi146 bash[23809]: cluster 2024-03-11T21:08:01.359872+0000 osd.3 (osd.3) 357 : cluster 0 4.1a scrub ok 2024-03-11T21:08:02.509 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:02 smithi146 bash[23809]: cluster 2024-03-11T21:08:01.479623+0000 osd.6 (osd.6) 264 : cluster 0 2.1a scrub starts 2024-03-11T21:08:02.509 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:02 smithi146 bash[23809]: cluster 2024-03-11T21:08:01.481605+0000 osd.6 (osd.6) 265 : cluster 0 2.1a scrub ok 2024-03-11T21:08:02.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:02 smithi146 bash[23809]: cluster 2024-03-11T21:08:01.884335+0000 osd.0 (osd.0) 265 : cluster 0 6.10 scrub starts 2024-03-11T21:08:02.548 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:02 smithi023 bash[21372]: cluster 2024-03-11T21:08:00.885110+0000 osd.0 (osd.0) 264 : cluster 0 3.1b scrub ok 2024-03-11T21:08:02.548 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:02 smithi023 bash[21372]: cluster 2024-03-11T21:08:01.358287+0000 osd.3 (osd.3) 356 : cluster 0 4.1a scrub starts 2024-03-11T21:08:02.548 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:02 smithi023 bash[21372]: cluster 2024-03-11T21:08:01.359872+0000 osd.3 (osd.3) 357 : cluster 0 4.1a scrub ok 2024-03-11T21:08:02.548 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:02 smithi023 bash[21372]: cluster 2024-03-11T21:08:01.479623+0000 osd.6 (osd.6) 264 : cluster 0 2.1a scrub starts 2024-03-11T21:08:02.548 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:02 smithi023 bash[21372]: cluster 2024-03-11T21:08:01.481605+0000 osd.6 (osd.6) 265 : cluster 0 2.1a scrub ok 2024-03-11T21:08:02.548 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:02 smithi023 bash[21372]: cluster 2024-03-11T21:08:01.884335+0000 osd.0 (osd.0) 265 : cluster 0 6.10 scrub starts 2024-03-11T21:08:02.548 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:02 smithi023 bash[29146]: cluster 2024-03-11T21:08:00.885110+0000 osd.0 (osd.0) 264 : cluster 0 3.1b scrub ok 2024-03-11T21:08:02.548 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:02 smithi023 bash[29146]: cluster 2024-03-11T21:08:01.358287+0000 osd.3 (osd.3) 356 : cluster 0 4.1a scrub starts 2024-03-11T21:08:02.548 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:02 smithi023 bash[29146]: cluster 2024-03-11T21:08:01.359872+0000 osd.3 (osd.3) 357 : cluster 0 4.1a scrub ok 2024-03-11T21:08:02.549 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:02 smithi023 bash[29146]: cluster 2024-03-11T21:08:01.479623+0000 osd.6 (osd.6) 264 : cluster 0 2.1a scrub starts 2024-03-11T21:08:02.549 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:02 smithi023 bash[29146]: cluster 2024-03-11T21:08:01.481605+0000 osd.6 (osd.6) 265 : cluster 0 2.1a scrub ok 2024-03-11T21:08:02.549 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:02 smithi023 bash[29146]: cluster 2024-03-11T21:08:01.884335+0000 osd.0 (osd.0) 265 : cluster 0 6.10 scrub starts 2024-03-11T21:08:02.587 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:02.614 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:02.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:02.617 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:08:02.718 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:02.744 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:02.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:02.747 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:08:02.848 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:02.876 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:02.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:02.881 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:08:02.982 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:08:03.007 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:08:03.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:03.010 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:08:03.111 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:08:03.134 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:08:03.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:03.137 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:08:03.225 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T21:08:03.238 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:08:03.265 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:08:03.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:03.270 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:08:03.367 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 27 16 45096 45080 104.342 81.625 0.00273402 0.00953725 2024-03-11T21:08:03.371 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:03.393 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:03.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:03.395 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:08:03.496 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:08:03.508 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:03 smithi146 bash[23809]: cluster 2024-03-11T21:08:01.870364+0000 osd.5 (osd.5) 333 : cluster 0 3.c scrub starts 2024-03-11T21:08:03.508 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:03 smithi146 bash[23809]: cluster 2024-03-11T21:08:01.874648+0000 osd.5 (osd.5) 334 : cluster 0 3.c scrub ok 2024-03-11T21:08:03.508 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:03 smithi146 bash[23809]: cluster 2024-03-11T21:08:01.931034+0000 osd.0 (osd.0) 266 : cluster 0 6.10 scrub ok 2024-03-11T21:08:03.508 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:03 smithi146 bash[23809]: cluster 2024-03-11T21:08:02.262151+0000 mgr.y (mgr.24380) 638 : cluster 0 pgmap v714: 148 pgs: 148 active+clean; 2.5 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 104 MiB/s wr, 1.67k op/s 2024-03-11T21:08:03.508 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:03 smithi146 bash[23809]: cluster 2024-03-11T21:08:02.475504+0000 osd.6 (osd.6) 266 : cluster 0 6.1e scrub starts 2024-03-11T21:08:03.508 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:03 smithi146 bash[23809]: cluster 2024-03-11T21:08:02.490694+0000 osd.6 (osd.6) 267 : cluster 0 6.1e scrub ok 2024-03-11T21:08:03.520 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:08:03.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:03.522 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:08:03.527 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:03 smithi023 bash[21372]: cluster 2024-03-11T21:08:01.870364+0000 osd.5 (osd.5) 333 : cluster 0 3.c scrub starts 2024-03-11T21:08:03.527 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:03 smithi023 bash[21372]: cluster 2024-03-11T21:08:01.874648+0000 osd.5 (osd.5) 334 : cluster 0 3.c scrub ok 2024-03-11T21:08:03.528 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:03 smithi023 bash[21372]: cluster 2024-03-11T21:08:01.931034+0000 osd.0 (osd.0) 266 : cluster 0 6.10 scrub ok 2024-03-11T21:08:03.528 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:03 smithi023 bash[21372]: cluster 2024-03-11T21:08:02.262151+0000 mgr.y (mgr.24380) 638 : cluster 0 pgmap v714: 148 pgs: 148 active+clean; 2.5 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 104 MiB/s wr, 1.67k op/s 2024-03-11T21:08:03.528 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:03 smithi023 bash[21372]: cluster 2024-03-11T21:08:02.475504+0000 osd.6 (osd.6) 266 : cluster 0 6.1e scrub starts 2024-03-11T21:08:03.528 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:03 smithi023 bash[21372]: cluster 2024-03-11T21:08:02.490694+0000 osd.6 (osd.6) 267 : cluster 0 6.1e scrub ok 2024-03-11T21:08:03.528 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:03 smithi023 bash[29146]: cluster 2024-03-11T21:08:01.870364+0000 osd.5 (osd.5) 333 : cluster 0 3.c scrub starts 2024-03-11T21:08:03.528 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:03 smithi023 bash[29146]: cluster 2024-03-11T21:08:01.874648+0000 osd.5 (osd.5) 334 : cluster 0 3.c scrub ok 2024-03-11T21:08:03.528 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:03 smithi023 bash[29146]: cluster 2024-03-11T21:08:01.931034+0000 osd.0 (osd.0) 266 : cluster 0 6.10 scrub ok 2024-03-11T21:08:03.528 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:03 smithi023 bash[29146]: cluster 2024-03-11T21:08:02.262151+0000 mgr.y (mgr.24380) 638 : cluster 0 pgmap v714: 148 pgs: 148 active+clean; 2.5 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 104 MiB/s wr, 1.67k op/s 2024-03-11T21:08:03.528 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:03 smithi023 bash[29146]: cluster 2024-03-11T21:08:02.475504+0000 osd.6 (osd.6) 266 : cluster 0 6.1e scrub starts 2024-03-11T21:08:03.529 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:03 smithi023 bash[29146]: cluster 2024-03-11T21:08:02.490694+0000 osd.6 (osd.6) 267 : cluster 0 6.1e scrub ok 2024-03-11T21:08:03.623 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:03.643 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:03.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:03.645 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:08:03.745 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:03.771 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:03.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:03.774 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:08:03.875 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:08:03.906 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:08:03.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:03.907 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:08:04.008 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:08:04.037 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:08:04.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:04.040 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:08:04.141 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:04.165 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:04.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:04.167 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:08:04.268 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:04.286 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:04.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:04.290 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:08:04.305 INFO:teuthology.orchestra.run.smithi023.stderr:set osd.2 primary-affinity to 0 (802) 2024-03-11T21:08:04.367 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 28 16 45778 45762 102.137 42.625 0.00260479 0.00974532 2024-03-11T21:08:04.392 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:04.402 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:04 smithi146 bash[23809]: cluster 2024-03-11T21:08:02.975371+0000 osd.2 (osd.2) 249 : cluster 0 4.d scrub starts 2024-03-11T21:08:04.402 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:04 smithi146 bash[23809]: cluster 2024-03-11T21:08:02.983964+0000 osd.2 (osd.2) 250 : cluster 0 4.d scrub ok 2024-03-11T21:08:04.402 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:04 smithi146 bash[23809]: cluster 2024-03-11T21:08:03.092207+0000 osd.7 (osd.7) 397 : cluster 0 2.1c scrub starts 2024-03-11T21:08:04.402 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:04 smithi146 bash[23809]: cluster 2024-03-11T21:08:03.095336+0000 osd.7 (osd.7) 398 : cluster 0 2.1c scrub ok 2024-03-11T21:08:04.402 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:04 smithi146 bash[23809]: cluster 2024-03-11T21:08:03.330646+0000 osd.3 (osd.3) 358 : cluster 0 4.c scrub starts 2024-03-11T21:08:04.402 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:04 smithi146 bash[23809]: cluster 2024-03-11T21:08:03.331824+0000 osd.3 (osd.3) 359 : cluster 0 4.c scrub ok 2024-03-11T21:08:04.403 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:04 smithi146 bash[23809]: cluster 2024-03-11T21:08:03.485005+0000 osd.6 (osd.6) 268 : cluster 0 3.a scrub starts 2024-03-11T21:08:04.403 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:04 smithi146 bash[23809]: cluster 2024-03-11T21:08:03.489919+0000 osd.6 (osd.6) 269 : cluster 0 3.a scrub ok 2024-03-11T21:08:04.403 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:04 smithi146 bash[23809]: audit 2024-03-11T21:08:03.628964+0000 mon.c (mon.2) 80 : audit 1 from='client.? 172.21.15.23:0/3502134568' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.0}]: dispatch 2024-03-11T21:08:04.403 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:04 smithi146 bash[23809]: audit 2024-03-11T21:08:03.629376+0000 mon.a (mon.0) 1633 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.0}]: dispatch 2024-03-11T21:08:04.403 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:04 smithi146 bash[23809]: cluster 2024-03-11T21:08:03.814744+0000 osd.0 (osd.0) 267 : cluster 0 purged_snaps scrub starts 2024-03-11T21:08:04.403 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:04 smithi146 bash[23809]: cluster 2024-03-11T21:08:03.814836+0000 osd.0 (osd.0) 268 : cluster 0 purged_snaps scrub ok 2024-03-11T21:08:04.419 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:04.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:04.423 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:08:04.524 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:04.529 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:04 smithi023 bash[21372]: cluster 2024-03-11T21:08:02.975371+0000 osd.2 (osd.2) 249 : cluster 0 4.d scrub starts 2024-03-11T21:08:04.529 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:04 smithi023 bash[21372]: cluster 2024-03-11T21:08:02.983964+0000 osd.2 (osd.2) 250 : cluster 0 4.d scrub ok 2024-03-11T21:08:04.529 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:04 smithi023 bash[21372]: cluster 2024-03-11T21:08:03.092207+0000 osd.7 (osd.7) 397 : cluster 0 2.1c scrub starts 2024-03-11T21:08:04.529 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:04 smithi023 bash[21372]: cluster 2024-03-11T21:08:03.095336+0000 osd.7 (osd.7) 398 : cluster 0 2.1c scrub ok 2024-03-11T21:08:04.529 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:04 smithi023 bash[21372]: cluster 2024-03-11T21:08:03.330646+0000 osd.3 (osd.3) 358 : cluster 0 4.c scrub starts 2024-03-11T21:08:04.529 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:04 smithi023 bash[21372]: cluster 2024-03-11T21:08:03.331824+0000 osd.3 (osd.3) 359 : cluster 0 4.c scrub ok 2024-03-11T21:08:04.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:04 smithi023 bash[21372]: cluster 2024-03-11T21:08:03.485005+0000 osd.6 (osd.6) 268 : cluster 0 3.a scrub starts 2024-03-11T21:08:04.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:04 smithi023 bash[21372]: cluster 2024-03-11T21:08:03.489919+0000 osd.6 (osd.6) 269 : cluster 0 3.a scrub ok 2024-03-11T21:08:04.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:04 smithi023 bash[21372]: audit 2024-03-11T21:08:03.628964+0000 mon.c (mon.2) 80 : audit 1 from='client.? 172.21.15.23:0/3502134568' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.0}]: dispatch 2024-03-11T21:08:04.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:04 smithi023 bash[21372]: audit 2024-03-11T21:08:03.629376+0000 mon.a (mon.0) 1633 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.0}]: dispatch 2024-03-11T21:08:04.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:04 smithi023 bash[21372]: cluster 2024-03-11T21:08:03.814744+0000 osd.0 (osd.0) 267 : cluster 0 purged_snaps scrub starts 2024-03-11T21:08:04.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:04 smithi023 bash[21372]: cluster 2024-03-11T21:08:03.814836+0000 osd.0 (osd.0) 268 : cluster 0 purged_snaps scrub ok 2024-03-11T21:08:04.530 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:04 smithi023 bash[29146]: cluster 2024-03-11T21:08:02.975371+0000 osd.2 (osd.2) 249 : cluster 0 4.d scrub starts 2024-03-11T21:08:04.530 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:04 smithi023 bash[29146]: cluster 2024-03-11T21:08:02.983964+0000 osd.2 (osd.2) 250 : cluster 0 4.d scrub ok 2024-03-11T21:08:04.530 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:04 smithi023 bash[29146]: cluster 2024-03-11T21:08:03.092207+0000 osd.7 (osd.7) 397 : cluster 0 2.1c scrub starts 2024-03-11T21:08:04.530 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:04 smithi023 bash[29146]: cluster 2024-03-11T21:08:03.095336+0000 osd.7 (osd.7) 398 : cluster 0 2.1c scrub ok 2024-03-11T21:08:04.530 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:04 smithi023 bash[29146]: cluster 2024-03-11T21:08:03.330646+0000 osd.3 (osd.3) 358 : cluster 0 4.c scrub starts 2024-03-11T21:08:04.531 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:04 smithi023 bash[29146]: cluster 2024-03-11T21:08:03.331824+0000 osd.3 (osd.3) 359 : cluster 0 4.c scrub ok 2024-03-11T21:08:04.531 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:04 smithi023 bash[29146]: cluster 2024-03-11T21:08:03.485005+0000 osd.6 (osd.6) 268 : cluster 0 3.a scrub starts 2024-03-11T21:08:04.531 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:04 smithi023 bash[29146]: cluster 2024-03-11T21:08:03.489919+0000 osd.6 (osd.6) 269 : cluster 0 3.a scrub ok 2024-03-11T21:08:04.531 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:04 smithi023 bash[29146]: audit 2024-03-11T21:08:03.628964+0000 mon.c (mon.2) 80 : audit 1 from='client.? 172.21.15.23:0/3502134568' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.0}]: dispatch 2024-03-11T21:08:04.531 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:04 smithi023 bash[29146]: audit 2024-03-11T21:08:03.629376+0000 mon.a (mon.0) 1633 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.0}]: dispatch 2024-03-11T21:08:04.531 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:04 smithi023 bash[29146]: cluster 2024-03-11T21:08:03.814744+0000 osd.0 (osd.0) 267 : cluster 0 purged_snaps scrub starts 2024-03-11T21:08:04.531 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:04 smithi023 bash[29146]: cluster 2024-03-11T21:08:03.814836+0000 osd.0 (osd.0) 268 : cluster 0 purged_snaps scrub ok 2024-03-11T21:08:04.541 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:04.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:04.543 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:08:04.644 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:04.666 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:04.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:04.668 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:08:04.769 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:08:04.788 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:08:04.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:04.790 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:08:04.891 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:04.911 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:04.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:04.913 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:08:05.014 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:08:05.036 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:08:05.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:05.039 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:08:05.140 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:05.165 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:05.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:05.167 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:08:05.268 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:08:05.300 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:08:05.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:05.302 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:08:05.367 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 29 15 45966 45951 99.0227 11.8125 0.0101457 0.010093 2024-03-11T21:08:05.403 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:05.413 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:05 smithi146 bash[23809]: cluster 2024-03-11T21:08:03.927234+0000 osd.5 (osd.5) 335 : cluster 0 4.8 scrub starts 2024-03-11T21:08:05.413 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:05 smithi146 bash[23809]: cluster 2024-03-11T21:08:03.929604+0000 osd.5 (osd.5) 336 : cluster 0 4.8 scrub ok 2024-03-11T21:08:05.413 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:05 smithi146 bash[23809]: cluster 2024-03-11T21:08:04.101979+0000 osd.7 (osd.7) 399 : cluster 0 3.f scrub starts 2024-03-11T21:08:05.413 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:05 smithi146 bash[23809]: cluster 2024-03-11T21:08:04.105915+0000 osd.7 (osd.7) 400 : cluster 0 3.f scrub ok 2024-03-11T21:08:05.413 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:05 smithi146 bash[23809]: cluster 2024-03-11T21:08:04.262627+0000 mgr.y (mgr.24380) 639 : cluster 0 pgmap v715: 148 pgs: 148 active+clean; 2.5 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 52 MiB/s wr, 824 op/s 2024-03-11T21:08:05.413 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:05 smithi146 bash[23809]: audit 2024-03-11T21:08:04.296533+0000 mon.a (mon.0) 1634 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.0}]': finished 2024-03-11T21:08:05.413 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:05 smithi146 bash[23809]: cluster 2024-03-11T21:08:04.303778+0000 mon.a (mon.0) 1635 : cluster 0 osdmap e319: 8 total, 8 up, 8 in 2024-03-11T21:08:05.413 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:05 smithi146 bash[23809]: audit 2024-03-11T21:08:04.364600+0000 mon.a (mon.0) 1636 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:08:05.414 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:05 smithi146 bash[23809]: cluster 2024-03-11T21:08:04.423300+0000 osd.3 (osd.3) 360 : cluster 0 4.0 scrub starts 2024-03-11T21:08:05.414 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:05 smithi146 bash[23809]: cluster 2024-03-11T21:08:04.428600+0000 osd.3 (osd.3) 361 : cluster 0 4.0 scrub ok 2024-03-11T21:08:05.414 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:05 smithi146 bash[23809]: cluster 2024-03-11T21:08:04.506891+0000 osd.6 (osd.6) 270 : cluster 0 4.5 scrub starts 2024-03-11T21:08:05.414 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:05 smithi146 bash[23809]: cluster 2024-03-11T21:08:04.509033+0000 osd.6 (osd.6) 271 : cluster 0 4.5 scrub ok 2024-03-11T21:08:05.414 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:05 smithi146 bash[23809]: cluster 2024-03-11T21:08:04.821729+0000 osd.5 (osd.5) 337 : cluster 0 purged_snaps scrub starts 2024-03-11T21:08:05.414 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:05 smithi146 bash[23809]: cluster 2024-03-11T21:08:04.821812+0000 osd.5 (osd.5) 338 : cluster 0 purged_snaps scrub ok 2024-03-11T21:08:05.414 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:05 smithi146 bash[23809]: audit 2024-03-11T21:08:05.041039+0000 mon.a (mon.0) 1637 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:08:05.414 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:05 smithi146 bash[23809]: audit 2024-03-11T21:08:05.041874+0000 mon.a (mon.0) 1638 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:08:05.414 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:05 smithi146 bash[23809]: audit 2024-03-11T21:08:05.050747+0000 mon.a (mon.0) 1639 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:08:05.414 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:05 smithi146 bash[23809]: cluster 2024-03-11T21:08:05.289953+0000 mon.a (mon.0) 1640 : cluster 0 osdmap e320: 8 total, 8 up, 8 in 2024-03-11T21:08:05.432 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:05.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:05.434 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:08:05.535 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:08:05.565 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:08:05.568 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:05 smithi023 bash[21372]: cluster 2024-03-11T21:08:03.927234+0000 osd.5 (osd.5) 335 : cluster 0 4.8 scrub starts 2024-03-11T21:08:05.568 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:05 smithi023 bash[21372]: cluster 2024-03-11T21:08:03.929604+0000 osd.5 (osd.5) 336 : cluster 0 4.8 scrub ok 2024-03-11T21:08:05.568 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:05 smithi023 bash[21372]: cluster 2024-03-11T21:08:04.101979+0000 osd.7 (osd.7) 399 : cluster 0 3.f scrub starts 2024-03-11T21:08:05.568 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:05 smithi023 bash[21372]: cluster 2024-03-11T21:08:04.105915+0000 osd.7 (osd.7) 400 : cluster 0 3.f scrub ok 2024-03-11T21:08:05.568 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:05 smithi023 bash[21372]: cluster 2024-03-11T21:08:04.262627+0000 mgr.y (mgr.24380) 639 : cluster 0 pgmap v715: 148 pgs: 148 active+clean; 2.5 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 52 MiB/s wr, 824 op/s 2024-03-11T21:08:05.569 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:05 smithi023 bash[21372]: audit 2024-03-11T21:08:04.296533+0000 mon.a (mon.0) 1634 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.0}]': finished 2024-03-11T21:08:05.569 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:05 smithi023 bash[21372]: cluster 2024-03-11T21:08:04.303778+0000 mon.a (mon.0) 1635 : cluster 0 osdmap e319: 8 total, 8 up, 8 in 2024-03-11T21:08:05.569 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:05 smithi023 bash[21372]: audit 2024-03-11T21:08:04.364600+0000 mon.a (mon.0) 1636 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:08:05.569 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:05 smithi023 bash[21372]: cluster 2024-03-11T21:08:04.423300+0000 osd.3 (osd.3) 360 : cluster 0 4.0 scrub starts 2024-03-11T21:08:05.569 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:05 smithi023 bash[21372]: cluster 2024-03-11T21:08:04.428600+0000 osd.3 (osd.3) 361 : cluster 0 4.0 scrub ok 2024-03-11T21:08:05.569 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:05 smithi023 bash[21372]: cluster 2024-03-11T21:08:04.506891+0000 osd.6 (osd.6) 270 : cluster 0 4.5 scrub starts 2024-03-11T21:08:05.569 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:05 smithi023 bash[21372]: cluster 2024-03-11T21:08:04.509033+0000 osd.6 (osd.6) 271 : cluster 0 4.5 scrub ok 2024-03-11T21:08:05.569 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:05 smithi023 bash[21372]: cluster 2024-03-11T21:08:04.821729+0000 osd.5 (osd.5) 337 : cluster 0 purged_snaps scrub starts 2024-03-11T21:08:05.569 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:05 smithi023 bash[21372]: cluster 2024-03-11T21:08:04.821812+0000 osd.5 (osd.5) 338 : cluster 0 purged_snaps scrub ok 2024-03-11T21:08:05.569 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:05 smithi023 bash[21372]: audit 2024-03-11T21:08:05.041039+0000 mon.a (mon.0) 1637 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:08:05.569 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:05 smithi023 bash[21372]: audit 2024-03-11T21:08:05.041874+0000 mon.a (mon.0) 1638 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:08:05.569 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:05 smithi023 bash[21372]: audit 2024-03-11T21:08:05.050747+0000 mon.a (mon.0) 1639 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:08:05.569 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:05 smithi023 bash[21372]: cluster 2024-03-11T21:08:05.289953+0000 mon.a (mon.0) 1640 : cluster 0 osdmap e320: 8 total, 8 up, 8 in 2024-03-11T21:08:05.570 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:05 smithi023 bash[29146]: cluster 2024-03-11T21:08:03.927234+0000 osd.5 (osd.5) 335 : cluster 0 4.8 scrub starts 2024-03-11T21:08:05.570 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:05 smithi023 bash[29146]: cluster 2024-03-11T21:08:03.929604+0000 osd.5 (osd.5) 336 : cluster 0 4.8 scrub ok 2024-03-11T21:08:05.570 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:05 smithi023 bash[29146]: cluster 2024-03-11T21:08:04.101979+0000 osd.7 (osd.7) 399 : cluster 0 3.f scrub starts 2024-03-11T21:08:05.570 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:05 smithi023 bash[29146]: cluster 2024-03-11T21:08:04.105915+0000 osd.7 (osd.7) 400 : cluster 0 3.f scrub ok 2024-03-11T21:08:05.570 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:05 smithi023 bash[29146]: cluster 2024-03-11T21:08:04.262627+0000 mgr.y (mgr.24380) 639 : cluster 0 pgmap v715: 148 pgs: 148 active+clean; 2.5 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 52 MiB/s wr, 824 op/s 2024-03-11T21:08:05.570 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:05 smithi023 bash[29146]: audit 2024-03-11T21:08:04.296533+0000 mon.a (mon.0) 1634 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.0}]': finished 2024-03-11T21:08:05.570 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:05 smithi023 bash[29146]: cluster 2024-03-11T21:08:04.303778+0000 mon.a (mon.0) 1635 : cluster 0 osdmap e319: 8 total, 8 up, 8 in 2024-03-11T21:08:05.570 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:05 smithi023 bash[29146]: audit 2024-03-11T21:08:04.364600+0000 mon.a (mon.0) 1636 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:08:05.570 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:05 smithi023 bash[29146]: cluster 2024-03-11T21:08:04.423300+0000 osd.3 (osd.3) 360 : cluster 0 4.0 scrub starts 2024-03-11T21:08:05.571 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:05 smithi023 bash[29146]: cluster 2024-03-11T21:08:04.428600+0000 osd.3 (osd.3) 361 : cluster 0 4.0 scrub ok 2024-03-11T21:08:05.571 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:05 smithi023 bash[29146]: cluster 2024-03-11T21:08:04.506891+0000 osd.6 (osd.6) 270 : cluster 0 4.5 scrub starts 2024-03-11T21:08:05.571 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:05 smithi023 bash[29146]: cluster 2024-03-11T21:08:04.509033+0000 osd.6 (osd.6) 271 : cluster 0 4.5 scrub ok 2024-03-11T21:08:05.571 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:05 smithi023 bash[29146]: cluster 2024-03-11T21:08:04.821729+0000 osd.5 (osd.5) 337 : cluster 0 purged_snaps scrub starts 2024-03-11T21:08:05.571 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:05 smithi023 bash[29146]: cluster 2024-03-11T21:08:04.821812+0000 osd.5 (osd.5) 338 : cluster 0 purged_snaps scrub ok 2024-03-11T21:08:05.571 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:05 smithi023 bash[29146]: audit 2024-03-11T21:08:05.041039+0000 mon.a (mon.0) 1637 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:08:05.571 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:05 smithi023 bash[29146]: audit 2024-03-11T21:08:05.041874+0000 mon.a (mon.0) 1638 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:08:05.571 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:05 smithi023 bash[29146]: audit 2024-03-11T21:08:05.050747+0000 mon.a (mon.0) 1639 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:08:05.571 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:05 smithi023 bash[29146]: cluster 2024-03-11T21:08:05.289953+0000 mon.a (mon.0) 1640 : cluster 0 osdmap e320: 8 total, 8 up, 8 in 2024-03-11T21:08:05.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:05.572 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:08:05.673 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:05.700 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:05.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:05.702 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:08:05.803 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:08:05.828 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:08:05.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:05.831 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:08:05.932 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:08:05.951 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:08:05.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:05.953 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:08:06.009 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:08:05 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:08:06.054 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:08:06.080 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:08:06.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:06.086 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:08:06.187 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:08:06.212 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:08:06.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:06.214 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:08:06.315 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:06.325 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:06 smithi146 bash[23809]: cluster 2024-03-11T21:08:04.910745+0000 osd.0 (osd.0) 269 : cluster 0 2.f scrub starts 2024-03-11T21:08:06.325 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:06 smithi146 bash[23809]: cluster 2024-03-11T21:08:04.912943+0000 osd.0 (osd.0) 270 : cluster 0 2.f scrub ok 2024-03-11T21:08:06.325 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:06 smithi146 bash[23809]: cluster 2024-03-11T21:08:04.972043+0000 osd.5 (osd.5) 339 : cluster 0 2.12 scrub starts 2024-03-11T21:08:06.325 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:06 smithi146 bash[23809]: cluster 2024-03-11T21:08:04.974090+0000 osd.5 (osd.5) 340 : cluster 0 2.12 scrub ok 2024-03-11T21:08:06.325 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:06 smithi146 bash[23809]: cluster 2024-03-11T21:08:05.061017+0000 osd.7 (osd.7) 401 : cluster 0 6.5 scrub starts 2024-03-11T21:08:06.325 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:06 smithi146 bash[23809]: cluster 2024-03-11T21:08:05.120641+0000 osd.7 (osd.7) 402 : cluster 0 6.5 scrub ok 2024-03-11T21:08:06.325 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:06 smithi146 bash[23809]: cluster 2024-03-11T21:08:05.494222+0000 osd.6 (osd.6) 272 : cluster 0 4.a scrub starts 2024-03-11T21:08:06.325 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:06 smithi146 bash[23809]: cluster 2024-03-11T21:08:05.495699+0000 osd.6 (osd.6) 273 : cluster 0 4.a scrub ok 2024-03-11T21:08:06.325 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:06 smithi146 bash[23809]: audit 2024-03-11T21:08:05.696219+0000 mon.a (mon.0) 1641 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:08:06.343 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:06.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:06.346 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:08:06.367 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 30 16 47313 47297 98.5259 84.125 0.00753682 0.0101447 2024-03-11T21:08:06.447 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:08:06.475 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:08:06.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:06.478 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:08:06.478 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:06 smithi023 bash[21372]: cluster 2024-03-11T21:08:04.910745+0000 osd.0 (osd.0) 269 : cluster 0 2.f scrub starts 2024-03-11T21:08:06.479 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:06 smithi023 bash[21372]: cluster 2024-03-11T21:08:04.912943+0000 osd.0 (osd.0) 270 : cluster 0 2.f scrub ok 2024-03-11T21:08:06.479 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:06 smithi023 bash[21372]: cluster 2024-03-11T21:08:04.972043+0000 osd.5 (osd.5) 339 : cluster 0 2.12 scrub starts 2024-03-11T21:08:06.479 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:06 smithi023 bash[21372]: cluster 2024-03-11T21:08:04.974090+0000 osd.5 (osd.5) 340 : cluster 0 2.12 scrub ok 2024-03-11T21:08:06.479 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:06 smithi023 bash[21372]: cluster 2024-03-11T21:08:05.061017+0000 osd.7 (osd.7) 401 : cluster 0 6.5 scrub starts 2024-03-11T21:08:06.479 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:06 smithi023 bash[21372]: cluster 2024-03-11T21:08:05.120641+0000 osd.7 (osd.7) 402 : cluster 0 6.5 scrub ok 2024-03-11T21:08:06.479 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:06 smithi023 bash[21372]: cluster 2024-03-11T21:08:05.494222+0000 osd.6 (osd.6) 272 : cluster 0 4.a scrub starts 2024-03-11T21:08:06.479 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:06 smithi023 bash[21372]: cluster 2024-03-11T21:08:05.495699+0000 osd.6 (osd.6) 273 : cluster 0 4.a scrub ok 2024-03-11T21:08:06.479 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:06 smithi023 bash[21372]: audit 2024-03-11T21:08:05.696219+0000 mon.a (mon.0) 1641 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:08:06.479 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:06 smithi023 bash[29146]: cluster 2024-03-11T21:08:04.910745+0000 osd.0 (osd.0) 269 : cluster 0 2.f scrub starts 2024-03-11T21:08:06.479 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:06 smithi023 bash[29146]: cluster 2024-03-11T21:08:04.912943+0000 osd.0 (osd.0) 270 : cluster 0 2.f scrub ok 2024-03-11T21:08:06.479 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:06 smithi023 bash[29146]: cluster 2024-03-11T21:08:04.972043+0000 osd.5 (osd.5) 339 : cluster 0 2.12 scrub starts 2024-03-11T21:08:06.480 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:06 smithi023 bash[29146]: cluster 2024-03-11T21:08:04.974090+0000 osd.5 (osd.5) 340 : cluster 0 2.12 scrub ok 2024-03-11T21:08:06.480 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:06 smithi023 bash[29146]: cluster 2024-03-11T21:08:05.061017+0000 osd.7 (osd.7) 401 : cluster 0 6.5 scrub starts 2024-03-11T21:08:06.480 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:06 smithi023 bash[29146]: cluster 2024-03-11T21:08:05.120641+0000 osd.7 (osd.7) 402 : cluster 0 6.5 scrub ok 2024-03-11T21:08:06.480 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:06 smithi023 bash[29146]: cluster 2024-03-11T21:08:05.494222+0000 osd.6 (osd.6) 272 : cluster 0 4.a scrub starts 2024-03-11T21:08:06.480 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:06 smithi023 bash[29146]: cluster 2024-03-11T21:08:05.495699+0000 osd.6 (osd.6) 273 : cluster 0 4.a scrub ok 2024-03-11T21:08:06.480 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:06 smithi023 bash[29146]: audit 2024-03-11T21:08:05.696219+0000 mon.a (mon.0) 1641 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:08:06.580 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:08:06.609 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:08:06.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:06.610 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:08:06.711 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:08:06.732 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:08:06.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:06.736 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:08:06.837 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:08:06.878 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:08:06.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:06.884 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:08:06.985 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:08:07.010 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:08:07.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:07.012 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:08:07.113 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:07.136 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:07.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:07.138 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:08:07.239 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:08:07.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:07.269 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:08:07.269 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:08:07.355 INFO:teuthology.orchestra.run.smithi023.stderr:nodeep-scrub is unset 2024-03-11T21:08:07.367 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 31 16 48639 48623 98.0208 82.875 0.0114481 0.0101916 2024-03-11T21:08:07.370 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:08:07.388 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:08:07.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:07.390 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:08:07.490 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:08:07.505 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:07 smithi146 bash[23809]: audit 2024-03-11T21:08:05.704082+0000 mgr.y (mgr.24380) 640 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:08:07.505 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:07 smithi146 bash[23809]: cluster 2024-03-11T21:08:05.900339+0000 osd.0 (osd.0) 271 : cluster 0 3.1f scrub starts 2024-03-11T21:08:07.505 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:07 smithi146 bash[23809]: cluster 2024-03-11T21:08:05.905689+0000 osd.0 (osd.0) 272 : cluster 0 3.1f scrub ok 2024-03-11T21:08:07.505 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:07 smithi146 bash[23809]: cluster 2024-03-11T21:08:05.955136+0000 osd.5 (osd.5) 341 : cluster 0 6.19 scrub starts 2024-03-11T21:08:07.505 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:07 smithi146 bash[23809]: cluster 2024-03-11T21:08:05.992333+0000 osd.5 (osd.5) 342 : cluster 0 6.19 scrub ok 2024-03-11T21:08:07.505 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:07 smithi146 bash[23809]: cluster 2024-03-11T21:08:06.263979+0000 mgr.y (mgr.24380) 641 : cluster 0 pgmap v718: 148 pgs: 1 active+clean+scrubbing, 6 peering, 141 active+clean; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 852 B/s rd, 68 MiB/s wr, 1.09k op/s; 0 B/s, 0 objects/s recovering 2024-03-11T21:08:07.505 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:07 smithi146 bash[23809]: cluster 2024-03-11T21:08:06.317590+0000 mon.a (mon.0) 1642 : cluster 3 Health check failed: Reduced data availability: 4 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-03-11T21:08:07.505 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:07 smithi146 bash[23809]: audit 2024-03-11T21:08:07.290269+0000 mon.c (mon.2) 81 : audit 1 from='client.? 172.21.15.23:0/2458173214' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:08:07.505 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:07 smithi146 bash[23809]: audit 2024-03-11T21:08:07.290783+0000 mon.a (mon.0) 1643 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:08:07.520 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:08:07.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:07.523 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:08:07.624 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:08:07.643 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:08:07.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:07.645 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:08:07.745 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:07.772 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:07.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:07.774 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:08:07.857 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:07 smithi023 bash[21372]: audit 2024-03-11T21:08:05.704082+0000 mgr.y (mgr.24380) 640 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:08:07.857 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:07 smithi023 bash[21372]: cluster 2024-03-11T21:08:05.900339+0000 osd.0 (osd.0) 271 : cluster 0 3.1f scrub starts 2024-03-11T21:08:07.857 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:07 smithi023 bash[21372]: cluster 2024-03-11T21:08:05.905689+0000 osd.0 (osd.0) 272 : cluster 0 3.1f scrub ok 2024-03-11T21:08:07.857 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:07 smithi023 bash[21372]: cluster 2024-03-11T21:08:05.955136+0000 osd.5 (osd.5) 341 : cluster 0 6.19 scrub starts 2024-03-11T21:08:07.857 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:07 smithi023 bash[21372]: cluster 2024-03-11T21:08:05.992333+0000 osd.5 (osd.5) 342 : cluster 0 6.19 scrub ok 2024-03-11T21:08:07.857 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:07 smithi023 bash[21372]: cluster 2024-03-11T21:08:06.263979+0000 mgr.y (mgr.24380) 641 : cluster 0 pgmap v718: 148 pgs: 1 active+clean+scrubbing, 6 peering, 141 active+clean; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 852 B/s rd, 68 MiB/s wr, 1.09k op/s; 0 B/s, 0 objects/s recovering 2024-03-11T21:08:07.857 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:07 smithi023 bash[21372]: cluster 2024-03-11T21:08:06.317590+0000 mon.a (mon.0) 1642 : cluster 3 Health check failed: Reduced data availability: 4 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-03-11T21:08:07.857 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:07 smithi023 bash[21372]: audit 2024-03-11T21:08:07.290269+0000 mon.c (mon.2) 81 : audit 1 from='client.? 172.21.15.23:0/2458173214' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:08:07.858 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:07 smithi023 bash[21372]: audit 2024-03-11T21:08:07.290783+0000 mon.a (mon.0) 1643 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:08:07.858 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:07 smithi023 bash[29146]: audit 2024-03-11T21:08:05.704082+0000 mgr.y (mgr.24380) 640 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:08:07.858 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:07 smithi023 bash[29146]: cluster 2024-03-11T21:08:05.900339+0000 osd.0 (osd.0) 271 : cluster 0 3.1f scrub starts 2024-03-11T21:08:07.858 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:07 smithi023 bash[29146]: cluster 2024-03-11T21:08:05.905689+0000 osd.0 (osd.0) 272 : cluster 0 3.1f scrub ok 2024-03-11T21:08:07.858 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:07 smithi023 bash[29146]: cluster 2024-03-11T21:08:05.955136+0000 osd.5 (osd.5) 341 : cluster 0 6.19 scrub starts 2024-03-11T21:08:07.858 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:07 smithi023 bash[29146]: cluster 2024-03-11T21:08:05.992333+0000 osd.5 (osd.5) 342 : cluster 0 6.19 scrub ok 2024-03-11T21:08:07.858 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:07 smithi023 bash[29146]: cluster 2024-03-11T21:08:06.263979+0000 mgr.y (mgr.24380) 641 : cluster 0 pgmap v718: 148 pgs: 1 active+clean+scrubbing, 6 peering, 141 active+clean; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 852 B/s rd, 68 MiB/s wr, 1.09k op/s; 0 B/s, 0 objects/s recovering 2024-03-11T21:08:07.858 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:07 smithi023 bash[29146]: cluster 2024-03-11T21:08:06.317590+0000 mon.a (mon.0) 1642 : cluster 3 Health check failed: Reduced data availability: 4 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-03-11T21:08:07.858 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:07 smithi023 bash[29146]: audit 2024-03-11T21:08:07.290269+0000 mon.c (mon.2) 81 : audit 1 from='client.? 172.21.15.23:0/2458173214' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:08:07.858 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:07 smithi023 bash[29146]: audit 2024-03-11T21:08:07.290783+0000 mon.a (mon.0) 1643 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:08:07.875 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:08:07.907 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:08:07.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:07.910 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:08:08.011 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:08:08.037 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:08:08.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:08.039 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:08:08.140 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:08:08.175 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:08:08.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:08.179 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:08:08.280 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:08:08.298 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:08:08.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:08.300 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:08:08.367 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 32 16 49108 49092 95.8735 29.3125 0.00313746 0.0104249 2024-03-11T21:08:08.401 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:08:08.428 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:08:08.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:08.430 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:08:08.531 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:08:08.558 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:08:08.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:08.561 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:08:08.662 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:08.685 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:08.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:08.688 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:08:08.691 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:08 smithi023 bash[21372]: cluster 2024-03-11T21:08:07.123043+0000 osd.7 (osd.7) 403 : cluster 0 6.3 scrub starts 2024-03-11T21:08:08.691 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:08 smithi023 bash[21372]: cluster 2024-03-11T21:08:07.202120+0000 osd.7 (osd.7) 404 : cluster 0 6.3 scrub ok 2024-03-11T21:08:08.691 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:08 smithi023 bash[21372]: cluster 2024-03-11T21:08:07.325595+0000 mon.a (mon.0) 1644 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:08:08.691 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:08 smithi023 bash[21372]: audit 2024-03-11T21:08:07.356001+0000 mon.a (mon.0) 1645 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:08:08.691 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:08 smithi023 bash[21372]: cluster 2024-03-11T21:08:07.360225+0000 mon.a (mon.0) 1646 : cluster 0 osdmap e321: 8 total, 8 up, 8 in 2024-03-11T21:08:08.692 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:08 smithi023 bash[21372]: cluster 2024-03-11T21:08:07.478580+0000 osd.6 (osd.6) 274 : cluster 0 2.11 scrub starts 2024-03-11T21:08:08.692 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:08 smithi023 bash[21372]: cluster 2024-03-11T21:08:07.483160+0000 osd.6 (osd.6) 275 : cluster 0 2.11 scrub ok 2024-03-11T21:08:08.692 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:08 smithi023 bash[21372]: cluster 2024-03-11T21:08:07.563731+0000 osd.1 (osd.1) 147 : cluster 0 4.2 scrub starts 2024-03-11T21:08:08.692 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:08 smithi023 bash[21372]: cluster 2024-03-11T21:08:07.565353+0000 osd.1 (osd.1) 148 : cluster 0 4.2 scrub ok 2024-03-11T21:08:08.692 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:08:08 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:08:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:08:08.692 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:08 smithi023 bash[29146]: cluster 2024-03-11T21:08:07.123043+0000 osd.7 (osd.7) 403 : cluster 0 6.3 scrub starts 2024-03-11T21:08:08.692 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:08 smithi023 bash[29146]: cluster 2024-03-11T21:08:07.202120+0000 osd.7 (osd.7) 404 : cluster 0 6.3 scrub ok 2024-03-11T21:08:08.692 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:08 smithi023 bash[29146]: cluster 2024-03-11T21:08:07.325595+0000 mon.a (mon.0) 1644 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:08:08.692 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:08 smithi023 bash[29146]: audit 2024-03-11T21:08:07.356001+0000 mon.a (mon.0) 1645 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:08:08.692 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:08 smithi023 bash[29146]: cluster 2024-03-11T21:08:07.360225+0000 mon.a (mon.0) 1646 : cluster 0 osdmap e321: 8 total, 8 up, 8 in 2024-03-11T21:08:08.693 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:08 smithi023 bash[29146]: cluster 2024-03-11T21:08:07.478580+0000 osd.6 (osd.6) 274 : cluster 0 2.11 scrub starts 2024-03-11T21:08:08.693 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:08 smithi023 bash[29146]: cluster 2024-03-11T21:08:07.483160+0000 osd.6 (osd.6) 275 : cluster 0 2.11 scrub ok 2024-03-11T21:08:08.693 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:08 smithi023 bash[29146]: cluster 2024-03-11T21:08:07.563731+0000 osd.1 (osd.1) 147 : cluster 0 4.2 scrub starts 2024-03-11T21:08:08.693 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:08 smithi023 bash[29146]: cluster 2024-03-11T21:08:07.565353+0000 osd.1 (osd.1) 148 : cluster 0 4.2 scrub ok 2024-03-11T21:08:08.759 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:08 smithi146 bash[23809]: cluster 2024-03-11T21:08:07.123043+0000 osd.7 (osd.7) 403 : cluster 0 6.3 scrub starts 2024-03-11T21:08:08.759 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:08 smithi146 bash[23809]: cluster 2024-03-11T21:08:07.202120+0000 osd.7 (osd.7) 404 : cluster 0 6.3 scrub ok 2024-03-11T21:08:08.759 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:08 smithi146 bash[23809]: cluster 2024-03-11T21:08:07.325595+0000 mon.a (mon.0) 1644 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:08:08.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:08 smithi146 bash[23809]: audit 2024-03-11T21:08:07.356001+0000 mon.a (mon.0) 1645 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:08:08.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:08 smithi146 bash[23809]: cluster 2024-03-11T21:08:07.360225+0000 mon.a (mon.0) 1646 : cluster 0 osdmap e321: 8 total, 8 up, 8 in 2024-03-11T21:08:08.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:08 smithi146 bash[23809]: cluster 2024-03-11T21:08:07.478580+0000 osd.6 (osd.6) 274 : cluster 0 2.11 scrub starts 2024-03-11T21:08:08.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:08 smithi146 bash[23809]: cluster 2024-03-11T21:08:07.483160+0000 osd.6 (osd.6) 275 : cluster 0 2.11 scrub ok 2024-03-11T21:08:08.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:08 smithi146 bash[23809]: cluster 2024-03-11T21:08:07.563731+0000 osd.1 (osd.1) 147 : cluster 0 4.2 scrub starts 2024-03-11T21:08:08.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:08 smithi146 bash[23809]: cluster 2024-03-11T21:08:07.565353+0000 osd.1 (osd.1) 148 : cluster 0 4.2 scrub ok 2024-03-11T21:08:08.789 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:08:08.817 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:08:08.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:08.821 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:08:08.921 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:08:08.946 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:08:08.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:08.951 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:08:09.052 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:08:09.074 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:08:09.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:09.077 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:08:09.178 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:09.203 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:09.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:09.209 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:08:09.310 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:09.335 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:09.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:09.340 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:08:09.368 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 33 15 50517 50502 95.6385 88.125 0.00769269 0.0104503 2024-03-11T21:08:09.440 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:08:09.451 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:09 smithi023 bash[29146]: cluster 2024-03-11T21:08:08.176118+0000 osd.0 (osd.0) 273 : cluster 0 4.3 scrub starts 2024-03-11T21:08:09.451 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:09 smithi023 bash[29146]: cluster 2024-03-11T21:08:08.180996+0000 osd.0 (osd.0) 274 : cluster 0 4.3 scrub ok 2024-03-11T21:08:09.451 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:09 smithi023 bash[29146]: cluster 2024-03-11T21:08:08.264528+0000 mgr.y (mgr.24380) 642 : cluster 0 pgmap v720: 148 pgs: 1 active+clean+scrubbing, 6 peering, 141 active+clean; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 852 B/s rd, 59 MiB/s wr, 944 op/s; 0 B/s, 0 objects/s recovering 2024-03-11T21:08:09.452 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:09 smithi023 bash[29146]: cluster 2024-03-11T21:08:08.519312+0000 osd.1 (osd.1) 149 : cluster 0 6.1d scrub starts 2024-03-11T21:08:09.452 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:09 smithi023 bash[29146]: cluster 2024-03-11T21:08:08.622050+0000 osd.1 (osd.1) 150 : cluster 0 6.1d scrub ok 2024-03-11T21:08:09.452 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:09 smithi023 bash[21372]: cluster 2024-03-11T21:08:08.176118+0000 osd.0 (osd.0) 273 : cluster 0 4.3 scrub starts 2024-03-11T21:08:09.452 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:09 smithi023 bash[21372]: cluster 2024-03-11T21:08:08.180996+0000 osd.0 (osd.0) 274 : cluster 0 4.3 scrub ok 2024-03-11T21:08:09.452 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:09 smithi023 bash[21372]: cluster 2024-03-11T21:08:08.264528+0000 mgr.y (mgr.24380) 642 : cluster 0 pgmap v720: 148 pgs: 1 active+clean+scrubbing, 6 peering, 141 active+clean; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 852 B/s rd, 59 MiB/s wr, 944 op/s; 0 B/s, 0 objects/s recovering 2024-03-11T21:08:09.452 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:09 smithi023 bash[21372]: cluster 2024-03-11T21:08:08.519312+0000 osd.1 (osd.1) 149 : cluster 0 6.1d scrub starts 2024-03-11T21:08:09.452 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:09 smithi023 bash[21372]: cluster 2024-03-11T21:08:08.622050+0000 osd.1 (osd.1) 150 : cluster 0 6.1d scrub ok 2024-03-11T21:08:09.467 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:08:09.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:09.471 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:08:09.571 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:08:09.596 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:08:09.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:09.600 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:08:09.701 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:08:09.734 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:08:09.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:09.738 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:08:09.759 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:09 smithi146 bash[23809]: cluster 2024-03-11T21:08:08.176118+0000 osd.0 (osd.0) 273 : cluster 0 4.3 scrub starts 2024-03-11T21:08:09.759 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:09 smithi146 bash[23809]: cluster 2024-03-11T21:08:08.180996+0000 osd.0 (osd.0) 274 : cluster 0 4.3 scrub ok 2024-03-11T21:08:09.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:09 smithi146 bash[23809]: cluster 2024-03-11T21:08:08.264528+0000 mgr.y (mgr.24380) 642 : cluster 0 pgmap v720: 148 pgs: 1 active+clean+scrubbing, 6 peering, 141 active+clean; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 852 B/s rd, 59 MiB/s wr, 944 op/s; 0 B/s, 0 objects/s recovering 2024-03-11T21:08:09.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:09 smithi146 bash[23809]: cluster 2024-03-11T21:08:08.519312+0000 osd.1 (osd.1) 149 : cluster 0 6.1d scrub starts 2024-03-11T21:08:09.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:09 smithi146 bash[23809]: cluster 2024-03-11T21:08:08.622050+0000 osd.1 (osd.1) 150 : cluster 0 6.1d scrub ok 2024-03-11T21:08:09.839 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:09.879 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:09.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:09.883 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:08:09.984 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:08:10.018 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:08:10.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:10.026 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:08:10.126 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:10.150 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:10.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:10.153 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:08:10.165 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-11T21:08:10.165 INFO:teuthology.orchestra.run.smithi023.stdout: "size": 20, 2024-03-11T21:08:10.165 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 600, 2024-03-11T21:08:10.165 INFO:teuthology.orchestra.run.smithi023.stdout: "ops": [ 2024-03-11T21:08:10.165 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.165 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_op(client.24281.0:1928 4.d 4.a204812d (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-03-11T21:08:10.165 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T20:58:53.171214+0000", 2024-03-11T21:08:10.165 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 556.99871016500003, 2024-03-11T21:08:10.165 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.41157037000000002, 2024-03-11T21:08:10.166 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:08:10.166 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "reached pg", 2024-03-11T21:08:10.166 INFO:teuthology.orchestra.run.smithi023.stdout: "client_info": { 2024-03-11T21:08:10.166 INFO:teuthology.orchestra.run.smithi023.stdout: "client": "client.24281", 2024-03-11T21:08:10.166 INFO:teuthology.orchestra.run.smithi023.stdout: "client_addr": "172.21.15.23:0/2185274774", 2024-03-11T21:08:10.166 INFO:teuthology.orchestra.run.smithi023.stdout: "tid": 1928 2024-03-11T21:08:10.166 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.166 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:08:10.166 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.166 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:08:10.166 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:58:53.171214+0000", 2024-03-11T21:08:10.166 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:08:10.166 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.167 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.167 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:08:10.167 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:58:53.171214+0000", 2024-03-11T21:08:10.167 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:08:10.167 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.167 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.167 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:08:10.167 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:58:53.171216+0000", 2024-03-11T21:08:10.167 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.0980000000000001e-06 2024-03-11T21:08:10.167 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.167 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.167 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:08:10.168 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:58:53.171224+0000", 2024-03-11T21:08:10.168 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 8.4689999999999997e-06 2024-03-11T21:08:10.168 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.168 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.168 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:08:10.168 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:58:53.171248+0000", 2024-03-11T21:08:10.168 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.4068999999999999e-05 2024-03-11T21:08:10.168 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.168 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.168 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:08:10.169 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:58:53.171262+0000", 2024-03-11T21:08:10.169 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.3465000000000001e-05 2024-03-11T21:08:10.169 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.169 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.169 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:08:10.169 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:58:53.171353+0000", 2024-03-11T21:08:10.169 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 9.0943e-05 2024-03-11T21:08:10.169 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.170 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.170 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for peered", 2024-03-11T21:08:10.170 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:58:53.171361+0000", 2024-03-11T21:08:10.170 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 8.5459999999999992e-06 2024-03-11T21:08:10.170 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.170 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.170 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:08:10.170 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:58:53.582528+0000", 2024-03-11T21:08:10.170 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.41116678099999998 2024-03-11T21:08:10.170 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.170 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.170 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:08:10.171 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T20:58:53.582784+0000", 2024-03-11T21:08:10.171 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00025599900000000003 2024-03-11T21:08:10.171 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:08:10.171 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:08:10.171 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:08:10.171 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.171 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.171 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_op(client.15291.0:36114 10.5 10:afeeceab:::benchmark_data_smithi023_124944_object36113:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e316)", 2024-03-11T21:08:10.171 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T21:07:56.357102+0000", 2024-03-11T21:08:10.171 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 13.812821599999999, 2024-03-11T21:08:10.171 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.29702194500000001, 2024-03-11T21:08:10.171 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:08:10.171 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:08:10.172 INFO:teuthology.orchestra.run.smithi023.stdout: "client_info": { 2024-03-11T21:08:10.172 INFO:teuthology.orchestra.run.smithi023.stdout: "client": "client.15291", 2024-03-11T21:08:10.172 INFO:teuthology.orchestra.run.smithi023.stdout: "client_addr": "172.21.15.23:0/729656300", 2024-03-11T21:08:10.172 INFO:teuthology.orchestra.run.smithi023.stdout: "tid": 36114 2024-03-11T21:08:10.172 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.172 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:08:10.172 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.172 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:08:10.172 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:07:56.357102+0000", 2024-03-11T21:08:10.172 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:08:10.172 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.172 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.172 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:08:10.172 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:07:56.357102+0000", 2024-03-11T21:08:10.172 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:08:10.173 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.173 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.173 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:08:10.173 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:07:56.357103+0000", 2024-03-11T21:08:10.173 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.2020000000000001e-06 2024-03-11T21:08:10.173 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.173 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.173 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:08:10.173 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:07:56.357123+0000", 2024-03-11T21:08:10.173 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.9905e-05 2024-03-11T21:08:10.173 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.173 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.173 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:08:10.173 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:07:56.357133+0000", 2024-03-11T21:08:10.173 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 9.9790000000000005e-06 2024-03-11T21:08:10.174 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.174 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.174 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:08:10.174 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:07:56.357138+0000", 2024-03-11T21:08:10.174 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 5.0710000000000002e-06 2024-03-11T21:08:10.174 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.174 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.174 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:08:10.174 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:07:56.357183+0000", 2024-03-11T21:08:10.175 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 4.4190999999999998e-05 2024-03-11T21:08:10.175 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.175 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.175 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:08:10.175 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:07:56.357286+0000", 2024-03-11T21:08:10.175 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000103942 2024-03-11T21:08:10.175 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.175 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.175 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for subops from 0,3", 2024-03-11T21:08:10.175 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:07:56.357394+0000", 2024-03-11T21:08:10.175 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000107849 2024-03-11T21:08:10.175 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.175 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.175 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "op_commit", 2024-03-11T21:08:10.175 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:07:56.358570+0000", 2024-03-11T21:08:10.176 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.001175253 2024-03-11T21:08:10.176 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.176 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.176 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:08:10.176 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:07:56.360099+0000", 2024-03-11T21:08:10.176 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0015296789999999999 2024-03-11T21:08:10.176 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.176 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.176 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:08:10.176 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:07:56.653554+0000", 2024-03-11T21:08:10.176 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.29345461299999998 2024-03-11T21:08:10.176 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.176 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.176 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T21:08:10.176 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:07:56.653974+0000", 2024-03-11T21:08:10.177 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00042009700000000003 2024-03-11T21:08:10.177 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.177 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.177 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:08:10.177 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:07:56.654124+0000", 2024-03-11T21:08:10.177 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00015016399999999999 2024-03-11T21:08:10.177 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:08:10.177 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:08:10.177 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:08:10.177 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.177 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.177 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_op(client.15291.0:36121 10.5 10:a834e12d:::benchmark_data_smithi023_124944_object36120:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e316)", 2024-03-11T21:08:10.177 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T21:07:56.362220+0000", 2024-03-11T21:08:10.177 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 13.80770351, 2024-03-11T21:08:10.177 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.29291954100000001, 2024-03-11T21:08:10.177 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:08:10.178 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:08:10.178 INFO:teuthology.orchestra.run.smithi023.stdout: "client_info": { 2024-03-11T21:08:10.178 INFO:teuthology.orchestra.run.smithi023.stdout: "client": "client.15291", 2024-03-11T21:08:10.178 INFO:teuthology.orchestra.run.smithi023.stdout: "client_addr": "172.21.15.23:0/729656300", 2024-03-11T21:08:10.178 INFO:teuthology.orchestra.run.smithi023.stdout: "tid": 36121 2024-03-11T21:08:10.178 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.178 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:08:10.178 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.178 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:08:10.178 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:07:56.362220+0000", 2024-03-11T21:08:10.178 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:08:10.178 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.178 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.178 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:08:10.179 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:07:56.362220+0000", 2024-03-11T21:08:10.179 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:08:10.179 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.179 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.179 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:08:10.179 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:07:56.362222+0000", 2024-03-11T21:08:10.179 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.669e-06 2024-03-11T21:08:10.179 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.179 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.179 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:08:10.179 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:07:56.362244+0000", 2024-03-11T21:08:10.179 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.1911000000000001e-05 2024-03-11T21:08:10.179 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.179 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.179 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:08:10.180 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:07:56.362259+0000", 2024-03-11T21:08:10.180 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.4791e-05 2024-03-11T21:08:10.180 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.180 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.180 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:08:10.180 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:07:56.362267+0000", 2024-03-11T21:08:10.180 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.9719999999999992e-06 2024-03-11T21:08:10.180 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.180 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.180 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:08:10.180 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:07:56.362317+0000", 2024-03-11T21:08:10.180 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 4.9932000000000002e-05 2024-03-11T21:08:10.180 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.180 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.181 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:08:10.181 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:07:56.362418+0000", 2024-03-11T21:08:10.181 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00010139799999999999 2024-03-11T21:08:10.181 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.181 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.181 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for subops from 0,3", 2024-03-11T21:08:10.181 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:07:56.362516+0000", 2024-03-11T21:08:10.181 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 9.8024999999999997e-05 2024-03-11T21:08:10.181 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.181 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.181 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "op_commit", 2024-03-11T21:08:10.181 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:07:56.363889+0000", 2024-03-11T21:08:10.181 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0013730000000000001 2024-03-11T21:08:10.181 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.181 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.181 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:08:10.182 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:07:56.364083+0000", 2024-03-11T21:08:10.182 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000193597 2024-03-11T21:08:10.182 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.182 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.182 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:08:10.182 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:07:56.654319+0000", 2024-03-11T21:08:10.182 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.29023663599999999 2024-03-11T21:08:10.182 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.182 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.182 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T21:08:10.182 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:07:56.655045+0000", 2024-03-11T21:08:10.182 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000725988 2024-03-11T21:08:10.182 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.182 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.183 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:08:10.183 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:07:56.655140+0000", 2024-03-11T21:08:10.183 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 9.4622000000000003e-05 2024-03-11T21:08:10.183 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:08:10.183 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:08:10.183 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:08:10.183 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.183 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.183 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_op(client.15291.0:36131 10.5 10:a34df68a:::benchmark_data_smithi023_124944_object36130:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e316)", 2024-03-11T21:08:10.183 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T21:07:56.379043+0000", 2024-03-11T21:08:10.183 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 13.790880537, 2024-03-11T21:08:10.183 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.27728798399999999, 2024-03-11T21:08:10.183 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:08:10.183 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:08:10.183 INFO:teuthology.orchestra.run.smithi023.stdout: "client_info": { 2024-03-11T21:08:10.184 INFO:teuthology.orchestra.run.smithi023.stdout: "client": "client.15291", 2024-03-11T21:08:10.184 INFO:teuthology.orchestra.run.smithi023.stdout: "client_addr": "172.21.15.23:0/729656300", 2024-03-11T21:08:10.184 INFO:teuthology.orchestra.run.smithi023.stdout: "tid": 36131 2024-03-11T21:08:10.184 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.184 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:08:10.184 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.184 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:08:10.184 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:07:56.379043+0000", 2024-03-11T21:08:10.184 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:08:10.184 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.184 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.184 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:08:10.184 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:07:56.379043+0000", 2024-03-11T21:08:10.184 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:08:10.184 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.184 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.185 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:08:10.185 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:07:56.379044+0000", 2024-03-11T21:08:10.185 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.28e-06 2024-03-11T21:08:10.185 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.185 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.185 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:08:10.185 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:07:56.379064+0000", 2024-03-11T21:08:10.185 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.9349000000000001e-05 2024-03-11T21:08:10.185 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.185 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.185 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:08:10.185 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:07:56.379075+0000", 2024-03-11T21:08:10.185 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.1409999999999999e-05 2024-03-11T21:08:10.185 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.185 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.186 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:08:10.186 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:07:56.379081+0000", 2024-03-11T21:08:10.186 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 5.7189999999999998e-06 2024-03-11T21:08:10.186 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.186 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.186 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:08:10.186 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:07:56.379134+0000", 2024-03-11T21:08:10.186 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 5.2941e-05 2024-03-11T21:08:10.186 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.186 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.186 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:08:10.186 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:07:56.379239+0000", 2024-03-11T21:08:10.186 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000104589 2024-03-11T21:08:10.186 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.187 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.187 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for subops from 0,3", 2024-03-11T21:08:10.187 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:07:56.379339+0000", 2024-03-11T21:08:10.187 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00010030399999999999 2024-03-11T21:08:10.187 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.187 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.187 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "op_commit", 2024-03-11T21:08:10.187 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:07:56.380690+0000", 2024-03-11T21:08:10.187 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0013515249999999999 2024-03-11T21:08:10.187 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.187 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.187 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:08:10.187 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:07:56.380986+0000", 2024-03-11T21:08:10.187 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000295474 2024-03-11T21:08:10.187 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.188 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.188 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:08:10.188 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:07:56.655246+0000", 2024-03-11T21:08:10.188 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.27426044100000002 2024-03-11T21:08:10.188 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.188 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.188 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T21:08:10.188 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:07:56.656022+0000", 2024-03-11T21:08:10.188 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00077543200000000005 2024-03-11T21:08:10.188 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.188 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.188 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:08:10.188 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:07:56.656331+0000", 2024-03-11T21:08:10.188 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00030951999999999998 2024-03-11T21:08:10.188 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:08:10.189 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:08:10.189 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:08:10.189 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.189 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.189 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_op(client.15291.0:45062 10.5 10:a3d6b3da:::benchmark_data_smithi023_124944_object45061:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e318)", 2024-03-11T21:08:10.189 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T21:08:03.238438+0000", 2024-03-11T21:08:10.189 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 6.9314861050000003, 2024-03-11T21:08:10.189 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.30270804099999998, 2024-03-11T21:08:10.189 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:08:10.189 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:08:10.189 INFO:teuthology.orchestra.run.smithi023.stdout: "client_info": { 2024-03-11T21:08:10.189 INFO:teuthology.orchestra.run.smithi023.stdout: "client": "client.15291", 2024-03-11T21:08:10.189 INFO:teuthology.orchestra.run.smithi023.stdout: "client_addr": "172.21.15.23:0/729656300", 2024-03-11T21:08:10.189 INFO:teuthology.orchestra.run.smithi023.stdout: "tid": 45062 2024-03-11T21:08:10.189 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.189 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:08:10.190 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.190 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:08:10.190 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.238438+0000", 2024-03-11T21:08:10.190 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:08:10.190 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.190 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.190 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:08:10.190 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.238438+0000", 2024-03-11T21:08:10.190 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:08:10.190 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.190 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.190 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:08:10.190 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.238439+0000", 2024-03-11T21:08:10.190 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.302e-06 2024-03-11T21:08:10.191 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.191 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.191 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:08:10.191 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.238460+0000", 2024-03-11T21:08:10.191 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.1270000000000001e-05 2024-03-11T21:08:10.191 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.191 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.191 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:08:10.191 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.238471+0000", 2024-03-11T21:08:10.191 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.1117999999999999e-05 2024-03-11T21:08:10.191 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.191 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.191 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:08:10.191 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.238479+0000", 2024-03-11T21:08:10.191 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.3300000000000001e-06 2024-03-11T21:08:10.191 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.191 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.192 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:08:10.192 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.238530+0000", 2024-03-11T21:08:10.192 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 5.1811000000000003e-05 2024-03-11T21:08:10.192 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.192 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.192 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:08:10.192 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.238649+0000", 2024-03-11T21:08:10.192 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00011900000000000001 2024-03-11T21:08:10.192 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.192 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.192 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for subops from 0,3", 2024-03-11T21:08:10.192 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.238781+0000", 2024-03-11T21:08:10.192 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00013152500000000001 2024-03-11T21:08:10.192 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.192 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.192 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "op_commit", 2024-03-11T21:08:10.193 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.240785+0000", 2024-03-11T21:08:10.193 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00200431 2024-03-11T21:08:10.193 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.193 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.193 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:08:10.193 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.540829+0000", 2024-03-11T21:08:10.193 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.300043485 2024-03-11T21:08:10.193 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.193 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.193 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:08:10.193 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.540961+0000", 2024-03-11T21:08:10.193 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000132373 2024-03-11T21:08:10.193 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.193 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.193 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T21:08:10.194 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.541091+0000", 2024-03-11T21:08:10.194 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00012990199999999999 2024-03-11T21:08:10.194 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.194 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.194 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:08:10.194 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.541146+0000", 2024-03-11T21:08:10.194 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 5.4614999999999998e-05 2024-03-11T21:08:10.194 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:08:10.194 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:08:10.194 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:08:10.194 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.194 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.194 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_op(client.15291.0:45065 10.5 10:a028b1f5:::benchmark_data_smithi023_124944_object45064:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e318)", 2024-03-11T21:08:10.195 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T21:08:03.242285+0000", 2024-03-11T21:08:10.195 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 6.9276390140000004, 2024-03-11T21:08:10.195 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.30386189400000002, 2024-03-11T21:08:10.195 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:08:10.195 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:08:10.195 INFO:teuthology.orchestra.run.smithi023.stdout: "client_info": { 2024-03-11T21:08:10.195 INFO:teuthology.orchestra.run.smithi023.stdout: "client": "client.15291", 2024-03-11T21:08:10.195 INFO:teuthology.orchestra.run.smithi023.stdout: "client_addr": "172.21.15.23:0/729656300", 2024-03-11T21:08:10.195 INFO:teuthology.orchestra.run.smithi023.stdout: "tid": 45065 2024-03-11T21:08:10.195 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.195 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:08:10.195 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.195 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:08:10.195 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.242285+0000", 2024-03-11T21:08:10.195 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:08:10.196 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.196 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.196 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:08:10.196 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.242285+0000", 2024-03-11T21:08:10.196 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:08:10.196 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.196 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.196 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:08:10.196 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.242286+0000", 2024-03-11T21:08:10.196 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.6509999999999999e-06 2024-03-11T21:08:10.196 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.196 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.196 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:08:10.196 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.242305+0000", 2024-03-11T21:08:10.196 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.9097999999999999e-05 2024-03-11T21:08:10.197 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.197 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.197 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:08:10.197 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.242315+0000", 2024-03-11T21:08:10.197 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 9.7040000000000006e-06 2024-03-11T21:08:10.197 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.197 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.197 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:08:10.197 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.242322+0000", 2024-03-11T21:08:10.197 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.3659999999999997e-06 2024-03-11T21:08:10.197 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.197 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.197 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:08:10.197 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.242387+0000", 2024-03-11T21:08:10.197 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.5164000000000002e-05 2024-03-11T21:08:10.198 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.198 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.198 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:08:10.198 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.242498+0000", 2024-03-11T21:08:10.198 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000111625 2024-03-11T21:08:10.198 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.198 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.198 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for subops from 0,3", 2024-03-11T21:08:10.198 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.242599+0000", 2024-03-11T21:08:10.198 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00010038400000000001 2024-03-11T21:08:10.198 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.198 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.198 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:08:10.198 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.540897+0000", 2024-03-11T21:08:10.199 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.29829849000000003 2024-03-11T21:08:10.199 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.199 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.199 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:08:10.199 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.541195+0000", 2024-03-11T21:08:10.199 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00029787200000000002 2024-03-11T21:08:10.199 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.199 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.199 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "op_commit", 2024-03-11T21:08:10.199 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.546006+0000", 2024-03-11T21:08:10.199 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00481132 2024-03-11T21:08:10.199 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.199 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.199 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T21:08:10.199 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.546083+0000", 2024-03-11T21:08:10.200 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.6267999999999996e-05 2024-03-11T21:08:10.200 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.200 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.200 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:08:10.200 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.546147+0000", 2024-03-11T21:08:10.200 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.3952e-05 2024-03-11T21:08:10.200 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:08:10.200 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:08:10.200 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:08:10.200 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.200 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.200 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_op(client.15291.0:45066 10.3 10:c8f8b29c:::benchmark_data_smithi023_124944_object45065:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e318)", 2024-03-11T21:08:10.200 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T21:08:03.244472+0000", 2024-03-11T21:08:10.200 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 6.9254512550000005, 2024-03-11T21:08:10.200 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.29773306900000002, 2024-03-11T21:08:10.201 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:08:10.201 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:08:10.201 INFO:teuthology.orchestra.run.smithi023.stdout: "client_info": { 2024-03-11T21:08:10.201 INFO:teuthology.orchestra.run.smithi023.stdout: "client": "client.15291", 2024-03-11T21:08:10.201 INFO:teuthology.orchestra.run.smithi023.stdout: "client_addr": "172.21.15.23:0/729656300", 2024-03-11T21:08:10.201 INFO:teuthology.orchestra.run.smithi023.stdout: "tid": 45066 2024-03-11T21:08:10.201 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.201 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:08:10.201 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.201 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:08:10.201 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.244472+0000", 2024-03-11T21:08:10.201 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:08:10.201 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.201 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.201 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:08:10.202 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.244472+0000", 2024-03-11T21:08:10.202 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:08:10.202 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.202 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.202 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:08:10.202 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.244474+0000", 2024-03-11T21:08:10.202 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.5850000000000001e-06 2024-03-11T21:08:10.202 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.202 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.202 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:08:10.202 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.244501+0000", 2024-03-11T21:08:10.202 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.7251999999999999e-05 2024-03-11T21:08:10.202 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.202 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.203 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:08:10.203 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.244515+0000", 2024-03-11T21:08:10.203 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.3946e-05 2024-03-11T21:08:10.203 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.203 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.203 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:08:10.203 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.244524+0000", 2024-03-11T21:08:10.203 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 8.3809999999999999e-06 2024-03-11T21:08:10.203 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.203 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.203 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:08:10.203 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.244676+0000", 2024-03-11T21:08:10.203 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00015238400000000001 2024-03-11T21:08:10.203 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.203 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.203 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:08:10.204 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.244837+0000", 2024-03-11T21:08:10.204 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00016081900000000001 2024-03-11T21:08:10.204 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.204 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.204 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for subops from 3,5", 2024-03-11T21:08:10.204 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.245039+0000", 2024-03-11T21:08:10.204 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00020164799999999999 2024-03-11T21:08:10.204 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.204 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.204 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:08:10.204 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.246993+0000", 2024-03-11T21:08:10.204 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0019543830000000001 2024-03-11T21:08:10.204 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.204 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.204 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "op_commit", 2024-03-11T21:08:10.205 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.247209+0000", 2024-03-11T21:08:10.205 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00021650700000000001 2024-03-11T21:08:10.205 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.205 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.205 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:08:10.205 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.542068+0000", 2024-03-11T21:08:10.205 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.29485842499999998 2024-03-11T21:08:10.205 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.205 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.205 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T21:08:10.205 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.542165+0000", 2024-03-11T21:08:10.205 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 9.6874999999999997e-05 2024-03-11T21:08:10.205 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.205 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.205 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:08:10.206 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.542206+0000", 2024-03-11T21:08:10.206 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 4.0864e-05 2024-03-11T21:08:10.206 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:08:10.206 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:08:10.206 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:08:10.206 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.206 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.206 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_op(client.15291.0:45067 10.5 10:a13e9247:::benchmark_data_smithi023_124944_object45066:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e318)", 2024-03-11T21:08:10.206 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T21:08:03.244550+0000", 2024-03-11T21:08:10.206 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 6.9253732929999998, 2024-03-11T21:08:10.206 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.30287361499999998, 2024-03-11T21:08:10.206 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:08:10.206 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:08:10.206 INFO:teuthology.orchestra.run.smithi023.stdout: "client_info": { 2024-03-11T21:08:10.207 INFO:teuthology.orchestra.run.smithi023.stdout: "client": "client.15291", 2024-03-11T21:08:10.207 INFO:teuthology.orchestra.run.smithi023.stdout: "client_addr": "172.21.15.23:0/729656300", 2024-03-11T21:08:10.207 INFO:teuthology.orchestra.run.smithi023.stdout: "tid": 45067 2024-03-11T21:08:10.207 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.207 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:08:10.207 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.207 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:08:10.207 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.244550+0000", 2024-03-11T21:08:10.207 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:08:10.207 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.207 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.207 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:08:10.207 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.244550+0000", 2024-03-11T21:08:10.207 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:08:10.207 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.207 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.208 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:08:10.208 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.244551+0000", 2024-03-11T21:08:10.208 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.2699999999999999e-07 2024-03-11T21:08:10.208 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.208 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.208 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:08:10.208 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.244572+0000", 2024-03-11T21:08:10.208 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.0678000000000001e-05 2024-03-11T21:08:10.208 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.208 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.208 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:08:10.208 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.244581+0000", 2024-03-11T21:08:10.208 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 8.8720000000000008e-06 2024-03-11T21:08:10.208 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.208 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.209 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:08:10.209 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.244588+0000", 2024-03-11T21:08:10.209 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.0430000000000002e-06 2024-03-11T21:08:10.209 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.209 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.209 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:08:10.209 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.247066+0000", 2024-03-11T21:08:10.209 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0024784070000000002 2024-03-11T21:08:10.209 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.209 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.209 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:08:10.209 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.247201+0000", 2024-03-11T21:08:10.209 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000134585 2024-03-11T21:08:10.209 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.209 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.210 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for subops from 0,3", 2024-03-11T21:08:10.210 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.537674+0000", 2024-03-11T21:08:10.210 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.29047304600000001 2024-03-11T21:08:10.210 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.210 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.210 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "op_commit", 2024-03-11T21:08:10.210 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.546160+0000", 2024-03-11T21:08:10.210 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0084859849999999997 2024-03-11T21:08:10.210 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.210 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.210 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:08:10.210 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.546892+0000", 2024-03-11T21:08:10.210 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00073225700000000005 2024-03-11T21:08:10.211 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.211 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.211 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:08:10.211 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.547326+0000", 2024-03-11T21:08:10.211 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000434435 2024-03-11T21:08:10.211 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.211 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.211 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T21:08:10.211 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.547384+0000", 2024-03-11T21:08:10.211 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 5.7858000000000001e-05 2024-03-11T21:08:10.211 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.211 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.211 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:08:10.211 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.547424+0000", 2024-03-11T21:08:10.211 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 3.9721999999999997e-05 2024-03-11T21:08:10.212 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:08:10.212 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:08:10.212 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:08:10.212 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.212 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.212 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_op(client.15291.0:45070 10.3 10:c860362f:::benchmark_data_smithi023_124944_object45069:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e318)", 2024-03-11T21:08:10.212 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T21:08:03.248420+0000", 2024-03-11T21:08:10.212 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 6.9215036989999996, 2024-03-11T21:08:10.212 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.298756935, 2024-03-11T21:08:10.212 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:08:10.212 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:08:10.212 INFO:teuthology.orchestra.run.smithi023.stdout: "client_info": { 2024-03-11T21:08:10.212 INFO:teuthology.orchestra.run.smithi023.stdout: "client": "client.15291", 2024-03-11T21:08:10.212 INFO:teuthology.orchestra.run.smithi023.stdout: "client_addr": "172.21.15.23:0/729656300", 2024-03-11T21:08:10.212 INFO:teuthology.orchestra.run.smithi023.stdout: "tid": 45070 2024-03-11T21:08:10.212 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.213 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:08:10.213 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.213 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:08:10.213 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.248420+0000", 2024-03-11T21:08:10.213 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:08:10.213 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.213 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.213 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:08:10.213 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.248420+0000", 2024-03-11T21:08:10.213 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:08:10.213 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.213 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.213 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:08:10.213 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.248422+0000", 2024-03-11T21:08:10.213 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.0600000000000002e-06 2024-03-11T21:08:10.213 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.213 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.213 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:08:10.214 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.248460+0000", 2024-03-11T21:08:10.214 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 3.7824999999999998e-05 2024-03-11T21:08:10.214 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.214 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.214 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:08:10.214 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.535541+0000", 2024-03-11T21:08:10.214 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.28708099100000001 2024-03-11T21:08:10.214 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.214 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.214 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:08:10.214 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.535577+0000", 2024-03-11T21:08:10.214 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 3.5615999999999999e-05 2024-03-11T21:08:10.214 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.214 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.215 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:08:10.215 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.536442+0000", 2024-03-11T21:08:10.215 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00086580000000000001 2024-03-11T21:08:10.215 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.215 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.215 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:08:10.215 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.536875+0000", 2024-03-11T21:08:10.215 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000432397 2024-03-11T21:08:10.215 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.215 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.215 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for subops from 3,5", 2024-03-11T21:08:10.215 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.537295+0000", 2024-03-11T21:08:10.215 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00042025 2024-03-11T21:08:10.215 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.215 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.215 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "op_commit", 2024-03-11T21:08:10.215 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.542220+0000", 2024-03-11T21:08:10.216 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0049246589999999996 2024-03-11T21:08:10.216 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.216 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.216 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:08:10.216 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.544296+0000", 2024-03-11T21:08:10.216 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.002076479 2024-03-11T21:08:10.216 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.216 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.216 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:08:10.216 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.547068+0000", 2024-03-11T21:08:10.216 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.002772189 2024-03-11T21:08:10.216 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.216 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.216 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T21:08:10.216 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.547132+0000", 2024-03-11T21:08:10.216 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.4035999999999995e-05 2024-03-11T21:08:10.217 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.217 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.217 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:08:10.217 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.547177+0000", 2024-03-11T21:08:10.217 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 4.4632999999999998e-05 2024-03-11T21:08:10.217 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:08:10.217 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:08:10.217 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:08:10.217 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.217 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.217 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_repop_reply(client.15291.0:45062 10.5 e318/306 ondisk, result = 0)", 2024-03-11T21:08:10.217 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T21:08:03.255834+0000", 2024-03-11T21:08:10.217 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 6.9140899789999999, 2024-03-11T21:08:10.217 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.28500816299999998, 2024-03-11T21:08:10.217 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:08:10.217 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "started", 2024-03-11T21:08:10.218 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:08:10.218 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.218 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:08:10.218 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.255834+0000", 2024-03-11T21:08:10.218 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:08:10.218 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.218 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.218 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:08:10.218 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.255834+0000", 2024-03-11T21:08:10.218 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:08:10.218 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.218 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.218 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:08:10.218 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.255840+0000", 2024-03-11T21:08:10.218 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 5.8409999999999996e-06 2024-03-11T21:08:10.219 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.219 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.219 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:08:10.219 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.255845+0000", 2024-03-11T21:08:10.219 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 5.6810000000000003e-06 2024-03-11T21:08:10.219 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.219 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.219 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:08:10.219 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.535738+0000", 2024-03-11T21:08:10.219 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.27989238300000002 2024-03-11T21:08:10.219 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.219 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.219 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:08:10.219 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.535747+0000", 2024-03-11T21:08:10.219 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 9.149e-06 2024-03-11T21:08:10.220 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.220 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.220 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:08:10.220 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.540805+0000", 2024-03-11T21:08:10.220 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.005058585 2024-03-11T21:08:10.220 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.220 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.220 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:08:10.220 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.540818+0000", 2024-03-11T21:08:10.220 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.289e-05 2024-03-11T21:08:10.220 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.220 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.220 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:08:10.220 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.540842+0000", 2024-03-11T21:08:10.221 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.3634000000000001e-05 2024-03-11T21:08:10.221 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:08:10.221 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:08:10.221 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:08:10.221 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.221 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.221 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_repop(client.15291.0:45080 10.9 e318/306 10:9c9e49b5:::benchmark_data_smithi023_124944_object45079:head v 318'2801, mlcod=318'2799)", 2024-03-11T21:08:10.221 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T21:08:03.258296+0000", 2024-03-11T21:08:10.221 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 6.91162776, 2024-03-11T21:08:10.221 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.281699225, 2024-03-11T21:08:10.221 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:08:10.221 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:08:10.221 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:08:10.221 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.221 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:08:10.222 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.258296+0000", 2024-03-11T21:08:10.222 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:08:10.222 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.222 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.222 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:08:10.222 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.258296+0000", 2024-03-11T21:08:10.222 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:08:10.222 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.222 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.222 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:08:10.222 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.258299+0000", 2024-03-11T21:08:10.222 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 3.388e-06 2024-03-11T21:08:10.222 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.222 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.222 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:08:10.223 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.258339+0000", 2024-03-11T21:08:10.223 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 3.9169999999999999e-05 2024-03-11T21:08:10.223 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.223 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.223 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:08:10.223 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.535739+0000", 2024-03-11T21:08:10.223 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.27740067200000001 2024-03-11T21:08:10.223 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.223 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.223 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:08:10.223 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.535750+0000", 2024-03-11T21:08:10.223 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.0841e-05 2024-03-11T21:08:10.223 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.223 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.223 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:08:10.224 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.536073+0000", 2024-03-11T21:08:10.224 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00032279299999999998 2024-03-11T21:08:10.224 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.224 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.224 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:08:10.224 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.536093+0000", 2024-03-11T21:08:10.224 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.0530000000000002e-05 2024-03-11T21:08:10.224 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.224 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.224 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T21:08:10.224 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.539948+0000", 2024-03-11T21:08:10.224 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0038548639999999999 2024-03-11T21:08:10.224 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.224 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.224 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:08:10.225 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.539995+0000", 2024-03-11T21:08:10.225 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 4.6966999999999999e-05 2024-03-11T21:08:10.225 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:08:10.225 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:08:10.225 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:08:10.225 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.225 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.225 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_op(client.15291.0:45074 10.5 10:a7c299e2:::benchmark_data_smithi023_124944_object45073:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e318)", 2024-03-11T21:08:10.225 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T21:08:03.535866+0000", 2024-03-11T21:08:10.225 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 6.6340578729999997, 2024-03-11T21:08:10.225 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.23795728999999999, 2024-03-11T21:08:10.225 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:08:10.225 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:08:10.225 INFO:teuthology.orchestra.run.smithi023.stdout: "client_info": { 2024-03-11T21:08:10.225 INFO:teuthology.orchestra.run.smithi023.stdout: "client": "client.15291", 2024-03-11T21:08:10.226 INFO:teuthology.orchestra.run.smithi023.stdout: "client_addr": "172.21.15.23:0/729656300", 2024-03-11T21:08:10.226 INFO:teuthology.orchestra.run.smithi023.stdout: "tid": 45074 2024-03-11T21:08:10.226 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.226 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:08:10.226 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.226 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:08:10.226 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.535866+0000", 2024-03-11T21:08:10.226 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:08:10.226 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.226 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.226 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:08:10.226 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.535866+0000", 2024-03-11T21:08:10.226 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:08:10.226 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.227 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.227 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:08:10.227 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.535868+0000", 2024-03-11T21:08:10.227 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.807e-06 2024-03-11T21:08:10.227 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.227 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.227 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:08:10.227 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.535996+0000", 2024-03-11T21:08:10.227 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00012783299999999999 2024-03-11T21:08:10.227 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.227 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.227 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:08:10.227 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.536020+0000", 2024-03-11T21:08:10.227 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.4788000000000001e-05 2024-03-11T21:08:10.227 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.227 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.227 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:08:10.228 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.536037+0000", 2024-03-11T21:08:10.228 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.6789000000000001e-05 2024-03-11T21:08:10.228 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.228 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.228 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:08:10.228 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.541239+0000", 2024-03-11T21:08:10.228 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0052018589999999996 2024-03-11T21:08:10.228 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.228 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.228 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:08:10.228 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.541426+0000", 2024-03-11T21:08:10.228 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00018716800000000001 2024-03-11T21:08:10.228 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.228 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.228 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for subops from 0,3", 2024-03-11T21:08:10.229 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.541614+0000", 2024-03-11T21:08:10.229 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000187469 2024-03-11T21:08:10.229 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.229 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.229 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "op_commit", 2024-03-11T21:08:10.229 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.546166+0000", 2024-03-11T21:08:10.229 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.004552553 2024-03-11T21:08:10.229 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.229 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.229 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:08:10.229 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.547287+0000", 2024-03-11T21:08:10.229 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0011213009999999999 2024-03-11T21:08:10.229 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.229 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.229 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:08:10.229 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.773211+0000", 2024-03-11T21:08:10.230 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.225923241 2024-03-11T21:08:10.230 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.230 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.230 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T21:08:10.230 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.773713+0000", 2024-03-11T21:08:10.230 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00050204999999999998 2024-03-11T21:08:10.230 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.230 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.230 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:08:10.230 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.773823+0000", 2024-03-11T21:08:10.230 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000110432 2024-03-11T21:08:10.230 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:08:10.231 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:08:10.231 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:08:10.231 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.231 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.231 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_op(client.15291.0:45093 10.5 10:aa9cc63b:::benchmark_data_smithi023_124944_object45092:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e318)", 2024-03-11T21:08:10.231 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T21:08:03.536063+0000", 2024-03-11T21:08:10.231 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 6.6338608509999997, 2024-03-11T21:08:10.231 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.238867463, 2024-03-11T21:08:10.231 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:08:10.231 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:08:10.231 INFO:teuthology.orchestra.run.smithi023.stdout: "client_info": { 2024-03-11T21:08:10.231 INFO:teuthology.orchestra.run.smithi023.stdout: "client": "client.15291", 2024-03-11T21:08:10.231 INFO:teuthology.orchestra.run.smithi023.stdout: "client_addr": "172.21.15.23:0/729656300", 2024-03-11T21:08:10.231 INFO:teuthology.orchestra.run.smithi023.stdout: "tid": 45093 2024-03-11T21:08:10.231 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.232 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:08:10.232 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.232 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:08:10.232 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.536063+0000", 2024-03-11T21:08:10.232 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:08:10.232 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.232 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.232 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:08:10.232 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.536063+0000", 2024-03-11T21:08:10.232 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:08:10.232 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.232 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.232 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:08:10.232 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.536064+0000", 2024-03-11T21:08:10.232 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.4169999999999999e-06 2024-03-11T21:08:10.233 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.233 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.233 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:08:10.233 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.536161+0000", 2024-03-11T21:08:10.233 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 9.624e-05 2024-03-11T21:08:10.233 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.233 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.233 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:08:10.233 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.536179+0000", 2024-03-11T21:08:10.233 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.8550999999999999e-05 2024-03-11T21:08:10.233 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.234 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.234 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:08:10.234 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.536194+0000", 2024-03-11T21:08:10.234 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.448e-05 2024-03-11T21:08:10.234 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.234 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.234 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:08:10.234 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.543437+0000", 2024-03-11T21:08:10.234 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0072437329999999996 2024-03-11T21:08:10.234 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.234 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.234 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:08:10.234 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.543611+0000", 2024-03-11T21:08:10.234 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000173179 2024-03-11T21:08:10.235 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.235 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.235 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for subops from 0,3", 2024-03-11T21:08:10.235 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.543788+0000", 2024-03-11T21:08:10.235 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00017776099999999999 2024-03-11T21:08:10.235 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.235 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.235 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "op_commit", 2024-03-11T21:08:10.235 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.546172+0000", 2024-03-11T21:08:10.235 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0023838930000000002 2024-03-11T21:08:10.235 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.235 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.235 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:08:10.235 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.773957+0000", 2024-03-11T21:08:10.235 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.22778480300000001 2024-03-11T21:08:10.236 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.236 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.236 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:08:10.236 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.774660+0000", 2024-03-11T21:08:10.236 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000702562 2024-03-11T21:08:10.236 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.236 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.236 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T21:08:10.236 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.774822+0000", 2024-03-11T21:08:10.236 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00016226500000000001 2024-03-11T21:08:10.236 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.236 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.236 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:08:10.236 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.774930+0000", 2024-03-11T21:08:10.236 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00010857900000000001 2024-03-11T21:08:10.237 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:08:10.237 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:08:10.237 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:08:10.237 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.237 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.237 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_op(client.15291.0:45096 10.3 10:c53b345f:::benchmark_data_smithi023_124944_object45095:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e318)", 2024-03-11T21:08:10.237 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T21:08:03.536224+0000", 2024-03-11T21:08:10.237 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 6.6336997630000001, 2024-03-11T21:08:10.237 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.23753238099999999, 2024-03-11T21:08:10.237 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:08:10.237 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:08:10.237 INFO:teuthology.orchestra.run.smithi023.stdout: "client_info": { 2024-03-11T21:08:10.237 INFO:teuthology.orchestra.run.smithi023.stdout: "client": "client.15291", 2024-03-11T21:08:10.237 INFO:teuthology.orchestra.run.smithi023.stdout: "client_addr": "172.21.15.23:0/729656300", 2024-03-11T21:08:10.237 INFO:teuthology.orchestra.run.smithi023.stdout: "tid": 45096 2024-03-11T21:08:10.238 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.238 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:08:10.238 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.238 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:08:10.238 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.536224+0000", 2024-03-11T21:08:10.238 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:08:10.238 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.238 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.238 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:08:10.238 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.536224+0000", 2024-03-11T21:08:10.238 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:08:10.238 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.238 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.238 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:08:10.239 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.536225+0000", 2024-03-11T21:08:10.239 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.099e-06 2024-03-11T21:08:10.239 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.239 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.239 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:08:10.239 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.536318+0000", 2024-03-11T21:08:10.239 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 9.2600000000000001e-05 2024-03-11T21:08:10.239 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.239 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.239 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:08:10.239 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.536335+0000", 2024-03-11T21:08:10.239 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.7328000000000001e-05 2024-03-11T21:08:10.239 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.239 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.239 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:08:10.240 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.536349+0000", 2024-03-11T21:08:10.240 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.3569e-05 2024-03-11T21:08:10.240 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.240 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.240 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:08:10.240 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.542251+0000", 2024-03-11T21:08:10.240 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0059028240000000001 2024-03-11T21:08:10.240 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.240 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.240 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:08:10.240 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.542367+0000", 2024-03-11T21:08:10.240 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000115691 2024-03-11T21:08:10.240 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.240 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.240 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for subops from 3,5", 2024-03-11T21:08:10.240 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.542495+0000", 2024-03-11T21:08:10.241 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000128174 2024-03-11T21:08:10.241 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.241 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.241 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "op_commit", 2024-03-11T21:08:10.241 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.546189+0000", 2024-03-11T21:08:10.241 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0036939389999999998 2024-03-11T21:08:10.241 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.241 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.241 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:08:10.241 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.773289+0000", 2024-03-11T21:08:10.241 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.22709961300000001 2024-03-11T21:08:10.241 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.241 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.242 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:08:10.242 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.773475+0000", 2024-03-11T21:08:10.242 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00018666900000000001 2024-03-11T21:08:10.242 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.242 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.242 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T21:08:10.242 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.773632+0000", 2024-03-11T21:08:10.242 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000156732 2024-03-11T21:08:10.242 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.242 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.242 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:08:10.242 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.773756+0000", 2024-03-11T21:08:10.242 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00012414300000000001 2024-03-11T21:08:10.242 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:08:10.242 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:08:10.243 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:08:10.243 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.243 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.243 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_repop_reply(client.15291.0:45074 10.5 e318/306 ondisk, result = 0)", 2024-03-11T21:08:10.243 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T21:08:03.546784+0000", 2024-03-11T21:08:10.243 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 6.6231395920000002, 2024-03-11T21:08:10.243 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.22706923600000001, 2024-03-11T21:08:10.243 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:08:10.243 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "started", 2024-03-11T21:08:10.243 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:08:10.243 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.243 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:08:10.243 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.546784+0000", 2024-03-11T21:08:10.243 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:08:10.244 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.244 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.244 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:08:10.244 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.546784+0000", 2024-03-11T21:08:10.244 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:08:10.244 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.244 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.244 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:08:10.244 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.546784+0000", 2024-03-11T21:08:10.244 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.96e-07 2024-03-11T21:08:10.244 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.244 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.244 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:08:10.244 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.546787+0000", 2024-03-11T21:08:10.244 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.5399999999999998e-06 2024-03-11T21:08:10.244 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.244 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.245 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:08:10.245 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.546795+0000", 2024-03-11T21:08:10.245 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.6380000000000004e-06 2024-03-11T21:08:10.245 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.245 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.245 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:08:10.245 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.546800+0000", 2024-03-11T21:08:10.245 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 5.7019999999999998e-06 2024-03-11T21:08:10.245 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.245 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.245 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:08:10.245 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.547464+0000", 2024-03-11T21:08:10.245 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00066383100000000003 2024-03-11T21:08:10.245 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.245 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.246 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:08:10.246 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.547471+0000", 2024-03-11T21:08:10.246 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.7889999999999997e-06 2024-03-11T21:08:10.246 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.246 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.246 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:08:10.246 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.773853+0000", 2024-03-11T21:08:10.246 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.22638243999999999 2024-03-11T21:08:10.246 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:08:10.246 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:08:10.246 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:08:10.246 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.246 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.246 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_repop_reply(client.15291.0:45096 10.3 e318/306 ondisk, result = 0)", 2024-03-11T21:08:10.246 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T21:08:03.546951+0000", 2024-03-11T21:08:10.247 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 6.6229728799999998, 2024-03-11T21:08:10.247 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.226831644, 2024-03-11T21:08:10.247 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:08:10.247 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "started", 2024-03-11T21:08:10.247 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:08:10.247 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.247 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:08:10.247 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.546951+0000", 2024-03-11T21:08:10.247 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:08:10.247 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.247 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.247 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:08:10.247 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.546951+0000", 2024-03-11T21:08:10.247 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:08:10.247 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.247 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.248 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:08:10.248 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.546952+0000", 2024-03-11T21:08:10.248 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 8.1399999999999996e-07 2024-03-11T21:08:10.248 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.248 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.248 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:08:10.248 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.546956+0000", 2024-03-11T21:08:10.248 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 4.3830000000000002e-06 2024-03-11T21:08:10.248 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.248 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.248 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:08:10.248 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.546966+0000", 2024-03-11T21:08:10.248 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 9.5829999999999996e-06 2024-03-11T21:08:10.248 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.248 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.248 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:08:10.249 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.546972+0000", 2024-03-11T21:08:10.249 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.4359999999999999e-06 2024-03-11T21:08:10.249 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.249 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.249 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:08:10.249 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.773441+0000", 2024-03-11T21:08:10.249 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.22646924399999999 2024-03-11T21:08:10.249 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.249 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.249 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:08:10.249 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.773460+0000", 2024-03-11T21:08:10.249 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.8887000000000001e-05 2024-03-11T21:08:10.249 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.249 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.249 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:08:10.249 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.773783+0000", 2024-03-11T21:08:10.250 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00032229699999999998 2024-03-11T21:08:10.250 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:08:10.250 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:08:10.250 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:08:10.250 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.250 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.250 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_repop_reply(client.15291.0:45093 10.5 e318/306 ondisk, result = 0)", 2024-03-11T21:08:10.250 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T21:08:03.546966+0000", 2024-03-11T21:08:10.250 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 6.6229575470000004, 2024-03-11T21:08:10.250 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.227020376, 2024-03-11T21:08:10.250 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:08:10.250 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "started", 2024-03-11T21:08:10.250 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:08:10.250 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.251 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:08:10.251 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.546966+0000", 2024-03-11T21:08:10.251 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:08:10.251 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.251 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.251 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:08:10.251 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.546966+0000", 2024-03-11T21:08:10.251 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:08:10.251 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.251 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.251 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:08:10.251 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.546967+0000", 2024-03-11T21:08:10.251 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 3.3599999999999999e-07 2024-03-11T21:08:10.251 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.251 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.252 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:08:10.252 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.546969+0000", 2024-03-11T21:08:10.252 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.2910000000000002e-06 2024-03-11T21:08:10.252 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.252 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.252 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:08:10.252 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.546975+0000", 2024-03-11T21:08:10.252 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 5.9059999999999999e-06 2024-03-11T21:08:10.252 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.252 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.252 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:08:10.252 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.546979+0000", 2024-03-11T21:08:10.252 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 4.5519999999999998e-06 2024-03-11T21:08:10.252 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.252 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.253 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:08:10.253 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.773923+0000", 2024-03-11T21:08:10.253 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.22694391799999999 2024-03-11T21:08:10.253 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.253 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.253 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:08:10.253 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.773942+0000", 2024-03-11T21:08:10.253 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.8763e-05 2024-03-11T21:08:10.253 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.253 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.253 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:08:10.253 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.773987+0000", 2024-03-11T21:08:10.253 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 4.4610000000000001e-05 2024-03-11T21:08:10.253 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:08:10.253 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:08:10.253 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:08:10.253 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.254 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.254 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_op(client.15291.0:45099 10.3 10:cf0a25b2:::benchmark_data_smithi023_124944_object45098:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e318)", 2024-03-11T21:08:10.254 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T21:08:03.547029+0000", 2024-03-11T21:08:10.254 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 6.6228947140000001, 2024-03-11T21:08:10.254 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.23888295400000001, 2024-03-11T21:08:10.254 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:08:10.254 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:08:10.254 INFO:teuthology.orchestra.run.smithi023.stdout: "client_info": { 2024-03-11T21:08:10.254 INFO:teuthology.orchestra.run.smithi023.stdout: "client": "client.15291", 2024-03-11T21:08:10.254 INFO:teuthology.orchestra.run.smithi023.stdout: "client_addr": "172.21.15.23:0/729656300", 2024-03-11T21:08:10.254 INFO:teuthology.orchestra.run.smithi023.stdout: "tid": 45099 2024-03-11T21:08:10.254 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.254 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:08:10.255 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.255 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:08:10.255 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.547029+0000", 2024-03-11T21:08:10.255 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:08:10.255 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.255 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.255 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:08:10.255 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.547029+0000", 2024-03-11T21:08:10.255 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:08:10.255 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.255 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.255 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:08:10.255 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.547030+0000", 2024-03-11T21:08:10.255 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.2890000000000001e-06 2024-03-11T21:08:10.255 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.256 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.256 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:08:10.256 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.547051+0000", 2024-03-11T21:08:10.256 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.0741e-05 2024-03-11T21:08:10.256 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.256 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.256 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:08:10.256 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.547062+0000", 2024-03-11T21:08:10.256 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.0468999999999999e-05 2024-03-11T21:08:10.256 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.256 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.256 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:08:10.256 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.547069+0000", 2024-03-11T21:08:10.256 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.311e-06 2024-03-11T21:08:10.256 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.257 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.257 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:08:10.257 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.773865+0000", 2024-03-11T21:08:10.257 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.226796634 2024-03-11T21:08:10.257 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.257 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.257 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:08:10.257 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.774194+0000", 2024-03-11T21:08:10.257 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00032826099999999999 2024-03-11T21:08:10.257 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.257 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.257 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for subops from 3,5", 2024-03-11T21:08:10.257 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.774584+0000", 2024-03-11T21:08:10.257 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00039006799999999998 2024-03-11T21:08:10.257 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.258 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.258 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:08:10.258 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.785444+0000", 2024-03-11T21:08:10.258 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.010860137000000001 2024-03-11T21:08:10.258 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.258 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.258 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:08:10.258 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.785565+0000", 2024-03-11T21:08:10.258 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000120596 2024-03-11T21:08:10.258 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.258 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.258 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "op_commit", 2024-03-11T21:08:10.258 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.785600+0000", 2024-03-11T21:08:10.258 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 3.5463000000000002e-05 2024-03-11T21:08:10.258 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.258 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.259 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T21:08:10.259 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.785775+0000", 2024-03-11T21:08:10.259 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00017501300000000001 2024-03-11T21:08:10.259 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.259 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.259 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:08:10.259 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.785912+0000", 2024-03-11T21:08:10.259 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00013697199999999999 2024-03-11T21:08:10.259 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:08:10.259 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:08:10.259 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:08:10.259 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.259 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.259 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_op(client.15291.0:45109 10.5 10:a5649fa4:::benchmark_data_smithi023_124944_object45108:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e318)", 2024-03-11T21:08:10.259 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T21:08:03.548069+0000", 2024-03-11T21:08:10.260 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 6.6218542679999999, 2024-03-11T21:08:10.260 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.245461016, 2024-03-11T21:08:10.260 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:08:10.260 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:08:10.260 INFO:teuthology.orchestra.run.smithi023.stdout: "client_info": { 2024-03-11T21:08:10.260 INFO:teuthology.orchestra.run.smithi023.stdout: "client": "client.15291", 2024-03-11T21:08:10.260 INFO:teuthology.orchestra.run.smithi023.stdout: "client_addr": "172.21.15.23:0/729656300", 2024-03-11T21:08:10.260 INFO:teuthology.orchestra.run.smithi023.stdout: "tid": 45109 2024-03-11T21:08:10.260 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.260 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:08:10.260 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.260 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:08:10.260 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.548069+0000", 2024-03-11T21:08:10.260 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:08:10.261 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.261 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.261 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:08:10.261 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.548069+0000", 2024-03-11T21:08:10.261 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:08:10.261 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.261 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.261 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:08:10.261 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.548071+0000", 2024-03-11T21:08:10.261 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.097e-06 2024-03-11T21:08:10.261 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.261 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.261 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:08:10.261 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.548095+0000", 2024-03-11T21:08:10.261 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.4893e-05 2024-03-11T21:08:10.261 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.261 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.262 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:08:10.262 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.773244+0000", 2024-03-11T21:08:10.262 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.225148719 2024-03-11T21:08:10.262 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.262 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.262 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:08:10.262 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.773284+0000", 2024-03-11T21:08:10.262 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 3.9433e-05 2024-03-11T21:08:10.262 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.262 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.262 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:08:10.262 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.775090+0000", 2024-03-11T21:08:10.262 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.001806001 2024-03-11T21:08:10.262 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.263 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.263 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:08:10.263 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.775418+0000", 2024-03-11T21:08:10.263 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00032868099999999999 2024-03-11T21:08:10.263 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.263 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.263 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for subops from 0,3", 2024-03-11T21:08:10.263 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.775836+0000", 2024-03-11T21:08:10.263 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00041763400000000001 2024-03-11T21:08:10.263 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.263 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.263 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "op_commit", 2024-03-11T21:08:10.263 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.780021+0000", 2024-03-11T21:08:10.263 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0041853100000000002 2024-03-11T21:08:10.263 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.263 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.263 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:08:10.264 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.785472+0000", 2024-03-11T21:08:10.264 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0054502739999999997 2024-03-11T21:08:10.264 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.264 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.264 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:08:10.264 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.793289+0000", 2024-03-11T21:08:10.264 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0078170159999999995 2024-03-11T21:08:10.264 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.264 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.264 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T21:08:10.264 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.793432+0000", 2024-03-11T21:08:10.264 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000143745 2024-03-11T21:08:10.264 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.264 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.264 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:08:10.264 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.793531+0000", 2024-03-11T21:08:10.265 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 9.8213000000000003e-05 2024-03-11T21:08:10.265 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:08:10.265 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:08:10.265 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:08:10.265 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.265 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.265 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_repop(client.15291.0:45103 10.8 e318/306 10:196c74ce:::benchmark_data_smithi023_124944_object45102:head v 318'2855, mlcod=318'2853)", 2024-03-11T21:08:10.265 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T21:08:03.548073+0000", 2024-03-11T21:08:10.265 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 6.6218511319999998, 2024-03-11T21:08:10.265 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.242017859, 2024-03-11T21:08:10.265 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:08:10.265 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:08:10.265 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:08:10.265 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.265 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:08:10.265 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.548073+0000", 2024-03-11T21:08:10.266 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:08:10.266 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.266 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.266 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:08:10.266 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.548073+0000", 2024-03-11T21:08:10.266 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:08:10.266 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.266 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.266 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:08:10.266 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.548073+0000", 2024-03-11T21:08:10.266 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.0999999999999998e-07 2024-03-11T21:08:10.266 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.266 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.266 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:08:10.267 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.548103+0000", 2024-03-11T21:08:10.267 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.9651000000000001e-05 2024-03-11T21:08:10.267 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.267 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.267 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:08:10.267 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.773233+0000", 2024-03-11T21:08:10.267 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.22513018900000001 2024-03-11T21:08:10.267 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.267 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.267 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:08:10.267 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.773268+0000", 2024-03-11T21:08:10.267 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 3.5102999999999999e-05 2024-03-11T21:08:10.267 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.267 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.267 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:08:10.268 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.773422+0000", 2024-03-11T21:08:10.268 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000153969 2024-03-11T21:08:10.268 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.268 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.268 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:08:10.268 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.773482+0000", 2024-03-11T21:08:10.268 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 5.9695999999999999e-05 2024-03-11T21:08:10.268 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.268 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.268 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T21:08:10.268 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.789989+0000", 2024-03-11T21:08:10.268 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.016507404999999999 2024-03-11T21:08:10.268 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:10.268 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:10.268 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:08:10.268 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.790090+0000", 2024-03-11T21:08:10.269 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000101136 2024-03-11T21:08:10.269 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:08:10.269 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:08:10.269 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:08:10.269 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:08:10.269 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:08:10.269 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-11T21:08:10.269 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:08:10.294 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:08:10.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:10.296 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:08:10.368 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 34 16 51907 51891 95.3786 86.8125 0.0264589 0.010479 2024-03-11T21:08:10.397 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:08:10.408 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:10 smithi146 bash[23809]: cluster 2024-03-11T21:08:09.341706+0000 osd.3 (osd.3) 362 : cluster 0 4.19 scrub starts 2024-03-11T21:08:10.408 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:10 smithi146 bash[23809]: cluster 2024-03-11T21:08:09.344635+0000 osd.3 (osd.3) 363 : cluster 0 4.19 scrub ok 2024-03-11T21:08:10.409 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:10 smithi146 bash[23809]: cluster 2024-03-11T21:08:09.467373+0000 osd.6 (osd.6) 276 : cluster 0 2.17 scrub starts 2024-03-11T21:08:10.409 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:10 smithi146 bash[23809]: cluster 2024-03-11T21:08:09.468336+0000 osd.6 (osd.6) 277 : cluster 0 2.17 scrub ok 2024-03-11T21:08:10.409 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:10 smithi146 bash[23809]: cluster 2024-03-11T21:08:10.285984+0000 mon.a (mon.0) 1647 : cluster 0 osdmap e322: 8 total, 8 up, 8 in 2024-03-11T21:08:10.424 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:08:10.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:10.429 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:08:10.529 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:10 smithi023 bash[21372]: cluster 2024-03-11T21:08:09.341706+0000 osd.3 (osd.3) 362 : cluster 0 4.19 scrub starts 2024-03-11T21:08:10.529 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:10 smithi023 bash[21372]: cluster 2024-03-11T21:08:09.344635+0000 osd.3 (osd.3) 363 : cluster 0 4.19 scrub ok 2024-03-11T21:08:10.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:10 smithi023 bash[21372]: cluster 2024-03-11T21:08:09.467373+0000 osd.6 (osd.6) 276 : cluster 0 2.17 scrub starts 2024-03-11T21:08:10.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:10 smithi023 bash[21372]: cluster 2024-03-11T21:08:09.468336+0000 osd.6 (osd.6) 277 : cluster 0 2.17 scrub ok 2024-03-11T21:08:10.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:10 smithi023 bash[21372]: cluster 2024-03-11T21:08:10.285984+0000 mon.a (mon.0) 1647 : cluster 0 osdmap e322: 8 total, 8 up, 8 in 2024-03-11T21:08:10.530 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:10 smithi023 bash[29146]: cluster 2024-03-11T21:08:09.341706+0000 osd.3 (osd.3) 362 : cluster 0 4.19 scrub starts 2024-03-11T21:08:10.530 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:10 smithi023 bash[29146]: cluster 2024-03-11T21:08:09.344635+0000 osd.3 (osd.3) 363 : cluster 0 4.19 scrub ok 2024-03-11T21:08:10.530 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:10 smithi023 bash[29146]: cluster 2024-03-11T21:08:09.467373+0000 osd.6 (osd.6) 276 : cluster 0 2.17 scrub starts 2024-03-11T21:08:10.530 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:10 smithi023 bash[29146]: cluster 2024-03-11T21:08:09.468336+0000 osd.6 (osd.6) 277 : cluster 0 2.17 scrub ok 2024-03-11T21:08:10.530 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:10 smithi023 bash[29146]: cluster 2024-03-11T21:08:10.285984+0000 mon.a (mon.0) 1647 : cluster 0 osdmap e322: 8 total, 8 up, 8 in 2024-03-11T21:08:10.531 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:10.566 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:10.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:10.569 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:08:10.669 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:08:10.705 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:08:10.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:10.709 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:08:10.809 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:10.831 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:10.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:10.837 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:08:10.938 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:08:10.971 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:08:10.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:10.976 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:08:11.077 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:08:11.106 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:08:11.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:11.111 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:08:11.211 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:11.240 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:11.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:11.241 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:08:11.342 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:11.368 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 35 16 53328 53312 95.1907 88.8125 0.00522695 0.0105003 2024-03-11T21:08:11.376 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:11.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:11.381 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:08:11.441 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-11T21:08:11.482 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:08:11.509 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:11 smithi146 bash[23809]: cluster 2024-03-11T21:08:09.962214+0000 osd.5 (osd.5) 343 : cluster 0 2.3 scrub starts 2024-03-11T21:08:11.509 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:11 smithi146 bash[23809]: cluster 2024-03-11T21:08:09.965141+0000 osd.5 (osd.5) 344 : cluster 0 2.3 scrub ok 2024-03-11T21:08:11.509 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:11 smithi146 bash[23809]: cluster 2024-03-11T21:08:10.265432+0000 mgr.y (mgr.24380) 643 : cluster 0 pgmap v721: 148 pgs: 148 active+clean; 3.1 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1.7 KiB/s rd, 101 MiB/s wr, 1.61k op/s; 0 B/s, 0 objects/s recovering 2024-03-11T21:08:11.509 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:11 smithi146 bash[23809]: cluster 2024-03-11T21:08:10.364759+0000 mon.a (mon.0) 1648 : cluster 1 Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 5 pgs peering) 2024-03-11T21:08:11.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:11 smithi146 bash[23809]: cluster 2024-03-11T21:08:10.364815+0000 mon.a (mon.0) 1649 : cluster 1 Cluster is now healthy 2024-03-11T21:08:11.537 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:08:11.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:11.541 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:08:11.637 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 6, 7, 1] out_osds: [] dead_osds: [] live_osds: [0, 2, 3, 5, 6, 7, 4, 1] 2024-03-11T21:08:11.637 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-11T21:08:11.638 INFO:tasks.thrashosds.thrasher:Shrinking pool unique_pool_3 2024-03-11T21:08:11.638 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool unique_pool_3 by 10 min 8 2024-03-11T21:08:11.638 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T21:08:11.642 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:08:11.649 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:11 smithi023 bash[21372]: cluster 2024-03-11T21:08:09.962214+0000 osd.5 (osd.5) 343 : cluster 0 2.3 scrub starts 2024-03-11T21:08:11.649 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:11 smithi023 bash[21372]: cluster 2024-03-11T21:08:09.965141+0000 osd.5 (osd.5) 344 : cluster 0 2.3 scrub ok 2024-03-11T21:08:11.649 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:11 smithi023 bash[21372]: cluster 2024-03-11T21:08:10.265432+0000 mgr.y (mgr.24380) 643 : cluster 0 pgmap v721: 148 pgs: 148 active+clean; 3.1 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1.7 KiB/s rd, 101 MiB/s wr, 1.61k op/s; 0 B/s, 0 objects/s recovering 2024-03-11T21:08:11.649 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:11 smithi023 bash[21372]: cluster 2024-03-11T21:08:10.364759+0000 mon.a (mon.0) 1648 : cluster 1 Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 5 pgs peering) 2024-03-11T21:08:11.649 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:11 smithi023 bash[21372]: cluster 2024-03-11T21:08:10.364815+0000 mon.a (mon.0) 1649 : cluster 1 Cluster is now healthy 2024-03-11T21:08:11.649 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:11 smithi023 bash[29146]: cluster 2024-03-11T21:08:09.962214+0000 osd.5 (osd.5) 343 : cluster 0 2.3 scrub starts 2024-03-11T21:08:11.649 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:11 smithi023 bash[29146]: cluster 2024-03-11T21:08:09.965141+0000 osd.5 (osd.5) 344 : cluster 0 2.3 scrub ok 2024-03-11T21:08:11.650 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:11 smithi023 bash[29146]: cluster 2024-03-11T21:08:10.265432+0000 mgr.y (mgr.24380) 643 : cluster 0 pgmap v721: 148 pgs: 148 active+clean; 3.1 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1.7 KiB/s rd, 101 MiB/s wr, 1.61k op/s; 0 B/s, 0 objects/s recovering 2024-03-11T21:08:11.650 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:11 smithi023 bash[29146]: cluster 2024-03-11T21:08:10.364759+0000 mon.a (mon.0) 1648 : cluster 1 Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 5 pgs peering) 2024-03-11T21:08:11.650 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:11 smithi023 bash[29146]: cluster 2024-03-11T21:08:10.364815+0000 mon.a (mon.0) 1649 : cluster 1 Cluster is now healthy 2024-03-11T21:08:11.689 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:08:11.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:11.693 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:08:11.794 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:11.827 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:11.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:11.828 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:08:11.895 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd set noscrub 2024-03-11T21:08:11.929 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:11.951 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:11.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:11.957 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:08:12.058 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:08:12.097 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:08:12.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:12.102 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:08:12.203 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:08:12.234 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:08:12.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:12.241 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:08:12.341 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:12.368 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 36 16 54597 54581 94.7495 79.3125 0.0148607 0.0105488 2024-03-11T21:08:12.372 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:12.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:12.376 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:08:12.415 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:08:12.476 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:08:12.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:12.519 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:08:12.519 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:08:12.620 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:08:12.645 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:08:12.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:12.648 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:08:12.660 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:12 smithi023 bash[21372]: cluster 2024-03-11T21:08:10.965111+0000 osd.5 (osd.5) 345 : cluster 0 6.1a scrub starts 2024-03-11T21:08:12.660 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:12 smithi023 bash[21372]: cluster 2024-03-11T21:08:10.985433+0000 osd.5 (osd.5) 346 : cluster 0 6.1a scrub ok 2024-03-11T21:08:12.660 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:12 smithi023 bash[21372]: cluster 2024-03-11T21:08:11.412558+0000 osd.3 (osd.3) 364 : cluster 0 3.2 scrub starts 2024-03-11T21:08:12.660 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:12 smithi023 bash[21372]: cluster 2024-03-11T21:08:11.417680+0000 osd.3 (osd.3) 365 : cluster 0 3.2 scrub ok 2024-03-11T21:08:12.660 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:12 smithi023 bash[21372]: cluster 2024-03-11T21:08:11.445143+0000 osd.6 (osd.6) 278 : cluster 0 4.1f scrub starts 2024-03-11T21:08:12.660 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:12 smithi023 bash[21372]: cluster 2024-03-11T21:08:11.449699+0000 osd.6 (osd.6) 279 : cluster 0 4.1f scrub ok 2024-03-11T21:08:12.660 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:12 smithi023 bash[21372]: cluster 2024-03-11T21:08:11.488047+0000 osd.1 (osd.1) 151 : cluster 0 6.e scrub starts 2024-03-11T21:08:12.660 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:12 smithi023 bash[21372]: cluster 2024-03-11T21:08:11.530866+0000 osd.1 (osd.1) 152 : cluster 0 6.e scrub ok 2024-03-11T21:08:12.660 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:12 smithi023 bash[29146]: cluster 2024-03-11T21:08:10.965111+0000 osd.5 (osd.5) 345 : cluster 0 6.1a scrub starts 2024-03-11T21:08:12.660 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:12 smithi023 bash[29146]: cluster 2024-03-11T21:08:10.985433+0000 osd.5 (osd.5) 346 : cluster 0 6.1a scrub ok 2024-03-11T21:08:12.661 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:12 smithi023 bash[29146]: cluster 2024-03-11T21:08:11.412558+0000 osd.3 (osd.3) 364 : cluster 0 3.2 scrub starts 2024-03-11T21:08:12.661 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:12 smithi023 bash[29146]: cluster 2024-03-11T21:08:11.417680+0000 osd.3 (osd.3) 365 : cluster 0 3.2 scrub ok 2024-03-11T21:08:12.661 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:12 smithi023 bash[29146]: cluster 2024-03-11T21:08:11.445143+0000 osd.6 (osd.6) 278 : cluster 0 4.1f scrub starts 2024-03-11T21:08:12.661 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:12 smithi023 bash[29146]: cluster 2024-03-11T21:08:11.449699+0000 osd.6 (osd.6) 279 : cluster 0 4.1f scrub ok 2024-03-11T21:08:12.661 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:12 smithi023 bash[29146]: cluster 2024-03-11T21:08:11.488047+0000 osd.1 (osd.1) 151 : cluster 0 6.e scrub starts 2024-03-11T21:08:12.661 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:12 smithi023 bash[29146]: cluster 2024-03-11T21:08:11.530866+0000 osd.1 (osd.1) 152 : cluster 0 6.e scrub ok 2024-03-11T21:08:12.750 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:12.759 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:12 smithi146 bash[23809]: cluster 2024-03-11T21:08:10.965111+0000 osd.5 (osd.5) 345 : cluster 0 6.1a scrub starts 2024-03-11T21:08:12.759 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:12 smithi146 bash[23809]: cluster 2024-03-11T21:08:10.985433+0000 osd.5 (osd.5) 346 : cluster 0 6.1a scrub ok 2024-03-11T21:08:12.759 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:12 smithi146 bash[23809]: cluster 2024-03-11T21:08:11.412558+0000 osd.3 (osd.3) 364 : cluster 0 3.2 scrub starts 2024-03-11T21:08:12.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:12 smithi146 bash[23809]: cluster 2024-03-11T21:08:11.417680+0000 osd.3 (osd.3) 365 : cluster 0 3.2 scrub ok 2024-03-11T21:08:12.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:12 smithi146 bash[23809]: cluster 2024-03-11T21:08:11.445143+0000 osd.6 (osd.6) 278 : cluster 0 4.1f scrub starts 2024-03-11T21:08:12.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:12 smithi146 bash[23809]: cluster 2024-03-11T21:08:11.449699+0000 osd.6 (osd.6) 279 : cluster 0 4.1f scrub ok 2024-03-11T21:08:12.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:12 smithi146 bash[23809]: cluster 2024-03-11T21:08:11.488047+0000 osd.1 (osd.1) 151 : cluster 0 6.e scrub starts 2024-03-11T21:08:12.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:12 smithi146 bash[23809]: cluster 2024-03-11T21:08:11.530866+0000 osd.1 (osd.1) 152 : cluster 0 6.e scrub ok 2024-03-11T21:08:12.775 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:12.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:12.778 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:08:12.878 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:12.910 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:12.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:12.913 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:08:13.014 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:13.039 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:13.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:13.043 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:08:13.144 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:13.179 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:13.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:13.185 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:08:13.286 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:08:13.309 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:08:13.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:13.318 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:08:13.368 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 37 16 55940 55924 94.4571 83.9375 0.0130512 0.0105821 2024-03-11T21:08:13.419 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:13.446 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:13.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:13.449 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:08:13.550 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:13.583 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:13.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:13.585 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:08:13.686 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:08:13.696 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:13 smithi023 bash[21372]: cluster 2024-03-11T21:08:11.969811+0000 osd.5 (osd.5) 347 : cluster 0 4.15 scrub starts 2024-03-11T21:08:13.696 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:13 smithi023 bash[21372]: cluster 2024-03-11T21:08:11.973397+0000 osd.5 (osd.5) 348 : cluster 0 4.15 scrub ok 2024-03-11T21:08:13.696 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:13 smithi023 bash[21372]: cluster 2024-03-11T21:08:12.266576+0000 mgr.y (mgr.24380) 644 : cluster 0 pgmap v723: 148 pgs: 148 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 733 B/s rd, 60 MiB/s wr, 963 op/s; 0 B/s, 0 objects/s recovering 2024-03-11T21:08:13.697 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:13 smithi023 bash[29146]: cluster 2024-03-11T21:08:11.969811+0000 osd.5 (osd.5) 347 : cluster 0 4.15 scrub starts 2024-03-11T21:08:13.697 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:13 smithi023 bash[29146]: cluster 2024-03-11T21:08:11.973397+0000 osd.5 (osd.5) 348 : cluster 0 4.15 scrub ok 2024-03-11T21:08:13.697 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:13 smithi023 bash[29146]: cluster 2024-03-11T21:08:12.266576+0000 mgr.y (mgr.24380) 644 : cluster 0 pgmap v723: 148 pgs: 148 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 733 B/s rd, 60 MiB/s wr, 963 op/s; 0 B/s, 0 objects/s recovering 2024-03-11T21:08:13.716 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:08:13.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:13.720 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:08:13.759 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:13 smithi146 bash[23809]: cluster 2024-03-11T21:08:11.969811+0000 osd.5 (osd.5) 347 : cluster 0 4.15 scrub starts 2024-03-11T21:08:13.759 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:13 smithi146 bash[23809]: cluster 2024-03-11T21:08:11.973397+0000 osd.5 (osd.5) 348 : cluster 0 4.15 scrub ok 2024-03-11T21:08:13.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:13 smithi146 bash[23809]: cluster 2024-03-11T21:08:12.266576+0000 mgr.y (mgr.24380) 644 : cluster 0 pgmap v723: 148 pgs: 148 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 733 B/s rd, 60 MiB/s wr, 963 op/s; 0 B/s, 0 objects/s recovering 2024-03-11T21:08:13.821 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:08:13.847 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:08:13.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:13.851 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:08:13.952 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:08:13.978 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:08:13.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:13.982 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:08:14.083 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:14.111 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:14.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:14.113 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:08:14.214 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:08:14.243 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:08:14.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:14.245 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:08:14.346 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:08:14.368 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 38 16 57279 57263 94.1735 83.6875 0.00990107 0.010612 2024-03-11T21:08:14.385 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:08:14.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:14.388 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:08:14.489 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:08:14.509 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:14 smithi146 bash[23809]: cluster 2024-03-11T21:08:12.931523+0000 osd.5 (osd.5) 349 : cluster 0 3.16 scrub starts 2024-03-11T21:08:14.509 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:14 smithi146 bash[23809]: cluster 2024-03-11T21:08:12.934627+0000 osd.5 (osd.5) 350 : cluster 0 3.16 scrub ok 2024-03-11T21:08:14.509 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:14 smithi146 bash[23809]: cluster 2024-03-11T21:08:13.133865+0000 osd.0 (osd.0) 275 : cluster 0 2.1f deep-scrub starts 2024-03-11T21:08:14.509 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:14 smithi146 bash[23809]: cluster 2024-03-11T21:08:13.139765+0000 osd.0 (osd.0) 276 : cluster 0 2.1f deep-scrub ok 2024-03-11T21:08:14.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:14 smithi146 bash[23809]: cluster 2024-03-11T21:08:13.519745+0000 osd.2 (osd.2) 251 : cluster 0 purged_snaps scrub starts 2024-03-11T21:08:14.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:14 smithi146 bash[23809]: cluster 2024-03-11T21:08:13.519809+0000 osd.2 (osd.2) 252 : cluster 0 purged_snaps scrub ok 2024-03-11T21:08:14.515 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:08:14.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:14.517 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:08:14.618 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:14.647 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:14.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:14.650 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:08:14.694 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:14 smithi023 bash[21372]: cluster 2024-03-11T21:08:12.931523+0000 osd.5 (osd.5) 349 : cluster 0 3.16 scrub starts 2024-03-11T21:08:14.694 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:14 smithi023 bash[21372]: cluster 2024-03-11T21:08:12.934627+0000 osd.5 (osd.5) 350 : cluster 0 3.16 scrub ok 2024-03-11T21:08:14.694 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:14 smithi023 bash[21372]: cluster 2024-03-11T21:08:13.133865+0000 osd.0 (osd.0) 275 : cluster 0 2.1f deep-scrub starts 2024-03-11T21:08:14.694 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:14 smithi023 bash[21372]: cluster 2024-03-11T21:08:13.139765+0000 osd.0 (osd.0) 276 : cluster 0 2.1f deep-scrub ok 2024-03-11T21:08:14.694 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:14 smithi023 bash[21372]: cluster 2024-03-11T21:08:13.519745+0000 osd.2 (osd.2) 251 : cluster 0 purged_snaps scrub starts 2024-03-11T21:08:14.694 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:14 smithi023 bash[21372]: cluster 2024-03-11T21:08:13.519809+0000 osd.2 (osd.2) 252 : cluster 0 purged_snaps scrub ok 2024-03-11T21:08:14.695 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:14 smithi023 bash[29146]: cluster 2024-03-11T21:08:12.931523+0000 osd.5 (osd.5) 349 : cluster 0 3.16 scrub starts 2024-03-11T21:08:14.695 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:14 smithi023 bash[29146]: cluster 2024-03-11T21:08:12.934627+0000 osd.5 (osd.5) 350 : cluster 0 3.16 scrub ok 2024-03-11T21:08:14.695 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:14 smithi023 bash[29146]: cluster 2024-03-11T21:08:13.133865+0000 osd.0 (osd.0) 275 : cluster 0 2.1f deep-scrub starts 2024-03-11T21:08:14.695 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:14 smithi023 bash[29146]: cluster 2024-03-11T21:08:13.139765+0000 osd.0 (osd.0) 276 : cluster 0 2.1f deep-scrub ok 2024-03-11T21:08:14.695 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:14 smithi023 bash[29146]: cluster 2024-03-11T21:08:13.519745+0000 osd.2 (osd.2) 251 : cluster 0 purged_snaps scrub starts 2024-03-11T21:08:14.695 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:14 smithi023 bash[29146]: cluster 2024-03-11T21:08:13.519809+0000 osd.2 (osd.2) 252 : cluster 0 purged_snaps scrub ok 2024-03-11T21:08:14.754 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:08:14.776 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:08:14.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:14.779 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:08:14.880 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:14.917 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:14.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:14.922 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:08:15.023 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:08:15.043 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:08:15.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:15.044 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:08:15.145 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:15.168 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:15.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:15.170 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:08:15.271 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:08:15.303 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:08:15.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:15.308 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:08:15.368 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 39 16 58513 58497 93.7362 77.125 0.0134077 0.0106635 2024-03-11T21:08:15.409 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:15.438 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:15.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:15.441 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:08:15.542 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:08:15.565 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:08:15.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:15.568 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:08:15.568 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:15 smithi023 bash[21372]: cluster 2024-03-11T21:08:13.895608+0000 osd.5 (osd.5) 351 : cluster 0 4.1d scrub starts 2024-03-11T21:08:15.568 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:15 smithi023 bash[21372]: cluster 2024-03-11T21:08:13.903056+0000 osd.5 (osd.5) 352 : cluster 0 4.1d scrub ok 2024-03-11T21:08:15.568 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:15 smithi023 bash[21372]: cluster 2024-03-11T21:08:14.072685+0000 osd.7 (osd.7) 405 : cluster 0 3.e scrub starts 2024-03-11T21:08:15.568 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:15 smithi023 bash[21372]: cluster 2024-03-11T21:08:14.074638+0000 osd.7 (osd.7) 406 : cluster 0 3.e scrub ok 2024-03-11T21:08:15.568 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:15 smithi023 bash[21372]: cluster 2024-03-11T21:08:14.267000+0000 mgr.y (mgr.24380) 645 : cluster 0 pgmap v724: 148 pgs: 148 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 45 MiB/s wr, 719 op/s 2024-03-11T21:08:15.568 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:15 smithi023 bash[21372]: cluster 2024-03-11T21:08:14.422816+0000 mon.a (mon.0) 1650 : cluster 0 osdmap e323: 8 total, 8 up, 8 in 2024-03-11T21:08:15.568 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:15 smithi023 bash[21372]: cluster 2024-03-11T21:08:15.298126+0000 mon.a (mon.0) 1651 : cluster 0 osdmap e324: 8 total, 8 up, 8 in 2024-03-11T21:08:15.568 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:15 smithi023 bash[29146]: cluster 2024-03-11T21:08:13.895608+0000 osd.5 (osd.5) 351 : cluster 0 4.1d scrub starts 2024-03-11T21:08:15.569 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:15 smithi023 bash[29146]: cluster 2024-03-11T21:08:13.903056+0000 osd.5 (osd.5) 352 : cluster 0 4.1d scrub ok 2024-03-11T21:08:15.569 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:15 smithi023 bash[29146]: cluster 2024-03-11T21:08:14.072685+0000 osd.7 (osd.7) 405 : cluster 0 3.e scrub starts 2024-03-11T21:08:15.569 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:15 smithi023 bash[29146]: cluster 2024-03-11T21:08:14.074638+0000 osd.7 (osd.7) 406 : cluster 0 3.e scrub ok 2024-03-11T21:08:15.569 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:15 smithi023 bash[29146]: cluster 2024-03-11T21:08:14.267000+0000 mgr.y (mgr.24380) 645 : cluster 0 pgmap v724: 148 pgs: 148 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 45 MiB/s wr, 719 op/s 2024-03-11T21:08:15.569 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:15 smithi023 bash[29146]: cluster 2024-03-11T21:08:14.422816+0000 mon.a (mon.0) 1650 : cluster 0 osdmap e323: 8 total, 8 up, 8 in 2024-03-11T21:08:15.569 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:15 smithi023 bash[29146]: cluster 2024-03-11T21:08:15.298126+0000 mon.a (mon.0) 1651 : cluster 0 osdmap e324: 8 total, 8 up, 8 in 2024-03-11T21:08:15.669 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:15.693 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:15.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:15.696 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:08:15.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:15 smithi146 bash[23809]: cluster 2024-03-11T21:08:13.895608+0000 osd.5 (osd.5) 351 : cluster 0 4.1d scrub starts 2024-03-11T21:08:15.707 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:15 smithi146 bash[23809]: cluster 2024-03-11T21:08:13.903056+0000 osd.5 (osd.5) 352 : cluster 0 4.1d scrub ok 2024-03-11T21:08:15.707 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:15 smithi146 bash[23809]: cluster 2024-03-11T21:08:14.072685+0000 osd.7 (osd.7) 405 : cluster 0 3.e scrub starts 2024-03-11T21:08:15.707 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:15 smithi146 bash[23809]: cluster 2024-03-11T21:08:14.074638+0000 osd.7 (osd.7) 406 : cluster 0 3.e scrub ok 2024-03-11T21:08:15.707 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:15 smithi146 bash[23809]: cluster 2024-03-11T21:08:14.267000+0000 mgr.y (mgr.24380) 645 : cluster 0 pgmap v724: 148 pgs: 148 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 45 MiB/s wr, 719 op/s 2024-03-11T21:08:15.707 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:15 smithi146 bash[23809]: cluster 2024-03-11T21:08:14.422816+0000 mon.a (mon.0) 1650 : cluster 0 osdmap e323: 8 total, 8 up, 8 in 2024-03-11T21:08:15.707 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:15 smithi146 bash[23809]: cluster 2024-03-11T21:08:15.298126+0000 mon.a (mon.0) 1651 : cluster 0 osdmap e324: 8 total, 8 up, 8 in 2024-03-11T21:08:15.797 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:08:15.827 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:08:15.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:15.831 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:08:15.932 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:08:15.954 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:08:15.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:15.959 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:08:16.009 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:08:15 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:08:16.060 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:08:16.082 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:08:16.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:16.084 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:08:16.185 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:08:16.209 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:08:16.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:16.211 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:08:16.311 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:08:16.312 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:16.314 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:08:16.349 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:16.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:16.353 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:08:16.368 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:2024-03-11T21:08:16.377853+0000 min lat: 0.00119683 max lat: 1.06642 avg lat: 0.0106972 2024-03-11T21:08:16.368 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-11T21:08:16.368 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 40 16 59821 59805 93.4363 81.75 0.0124575 0.0106972 2024-03-11T21:08:16.454 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:08:16.480 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:08:16.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:16.484 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:08:16.490 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:08:16.585 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:16.610 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:16.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:16.612 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:08:16.713 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:08:16.750 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:08:16.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:16.752 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:08:16.853 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:08:16.882 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:08:16.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:16.888 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:08:16.989 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:08:17.016 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:08:17.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:17.021 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:08:17.122 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:08:17.144 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:08:17.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:17.146 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:08:17.247 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:08:17.278 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:08:17.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:17.280 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:08:17.368 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 41 16 61149 61133 93.1816 83 0.0157876 0.0107269 2024-03-11T21:08:17.381 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:08:17.407 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:08:17.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:17.411 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:08:17.511 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:08:17.538 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:08:17.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:17.544 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:08:17.645 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:08:17.654 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:17 smithi023 bash[21372]: audit 2024-03-11T21:08:15.713137+0000 mgr.y (mgr.24380) 646 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:08:17.655 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:17 smithi023 bash[21372]: cluster 2024-03-11T21:08:15.914152+0000 osd.5 (osd.5) 353 : cluster 0 4.1 scrub starts 2024-03-11T21:08:17.655 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:17 smithi023 bash[21372]: cluster 2024-03-11T21:08:15.915368+0000 osd.5 (osd.5) 354 : cluster 0 4.1 scrub ok 2024-03-11T21:08:17.655 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:17 smithi023 bash[21372]: cluster 2024-03-11T21:08:16.133508+0000 osd.7 (osd.7) 407 : cluster 0 2.8 scrub starts 2024-03-11T21:08:17.655 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:17 smithi023 bash[21372]: cluster 2024-03-11T21:08:16.136844+0000 osd.7 (osd.7) 408 : cluster 0 2.8 scrub ok 2024-03-11T21:08:17.655 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:17 smithi023 bash[21372]: cluster 2024-03-11T21:08:16.211471+0000 osd.0 (osd.0) 277 : cluster 0 4.6 scrub starts 2024-03-11T21:08:17.655 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:17 smithi023 bash[21372]: cluster 2024-03-11T21:08:16.214355+0000 osd.0 (osd.0) 278 : cluster 0 4.6 scrub ok 2024-03-11T21:08:17.655 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:17 smithi023 bash[21372]: cluster 2024-03-11T21:08:16.267584+0000 mgr.y (mgr.24380) 647 : cluster 0 pgmap v727: 148 pgs: 148 active+clean; 3.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 852 B/s rd, 80 MiB/s wr, 1.29k op/s 2024-03-11T21:08:17.655 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:17 smithi023 bash[21372]: cluster 2024-03-11T21:08:16.450490+0000 osd.6 (osd.6) 280 : cluster 0 4.18 scrub starts 2024-03-11T21:08:17.655 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:17 smithi023 bash[21372]: cluster 2024-03-11T21:08:16.455019+0000 osd.6 (osd.6) 281 : cluster 0 4.18 scrub ok 2024-03-11T21:08:17.655 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:17 smithi023 bash[29146]: audit 2024-03-11T21:08:15.713137+0000 mgr.y (mgr.24380) 646 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:08:17.655 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:17 smithi023 bash[29146]: cluster 2024-03-11T21:08:15.914152+0000 osd.5 (osd.5) 353 : cluster 0 4.1 scrub starts 2024-03-11T21:08:17.655 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:17 smithi023 bash[29146]: cluster 2024-03-11T21:08:15.915368+0000 osd.5 (osd.5) 354 : cluster 0 4.1 scrub ok 2024-03-11T21:08:17.656 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:17 smithi023 bash[29146]: cluster 2024-03-11T21:08:16.133508+0000 osd.7 (osd.7) 407 : cluster 0 2.8 scrub starts 2024-03-11T21:08:17.656 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:17 smithi023 bash[29146]: cluster 2024-03-11T21:08:16.136844+0000 osd.7 (osd.7) 408 : cluster 0 2.8 scrub ok 2024-03-11T21:08:17.656 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:17 smithi023 bash[29146]: cluster 2024-03-11T21:08:16.211471+0000 osd.0 (osd.0) 277 : cluster 0 4.6 scrub starts 2024-03-11T21:08:17.656 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:17 smithi023 bash[29146]: cluster 2024-03-11T21:08:16.214355+0000 osd.0 (osd.0) 278 : cluster 0 4.6 scrub ok 2024-03-11T21:08:17.656 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:17 smithi023 bash[29146]: cluster 2024-03-11T21:08:16.267584+0000 mgr.y (mgr.24380) 647 : cluster 0 pgmap v727: 148 pgs: 148 active+clean; 3.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 852 B/s rd, 80 MiB/s wr, 1.29k op/s 2024-03-11T21:08:17.656 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:17 smithi023 bash[29146]: cluster 2024-03-11T21:08:16.450490+0000 osd.6 (osd.6) 280 : cluster 0 4.18 scrub starts 2024-03-11T21:08:17.656 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:17 smithi023 bash[29146]: cluster 2024-03-11T21:08:16.455019+0000 osd.6 (osd.6) 281 : cluster 0 4.18 scrub ok 2024-03-11T21:08:17.690 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:08:17.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:17.694 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:08:17.759 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:17 smithi146 bash[23809]: audit 2024-03-11T21:08:15.713137+0000 mgr.y (mgr.24380) 646 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:08:17.759 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:17 smithi146 bash[23809]: cluster 2024-03-11T21:08:15.914152+0000 osd.5 (osd.5) 353 : cluster 0 4.1 scrub starts 2024-03-11T21:08:17.759 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:17 smithi146 bash[23809]: cluster 2024-03-11T21:08:15.915368+0000 osd.5 (osd.5) 354 : cluster 0 4.1 scrub ok 2024-03-11T21:08:17.759 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:17 smithi146 bash[23809]: cluster 2024-03-11T21:08:16.133508+0000 osd.7 (osd.7) 407 : cluster 0 2.8 scrub starts 2024-03-11T21:08:17.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:17 smithi146 bash[23809]: cluster 2024-03-11T21:08:16.136844+0000 osd.7 (osd.7) 408 : cluster 0 2.8 scrub ok 2024-03-11T21:08:17.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:17 smithi146 bash[23809]: cluster 2024-03-11T21:08:16.211471+0000 osd.0 (osd.0) 277 : cluster 0 4.6 scrub starts 2024-03-11T21:08:17.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:17 smithi146 bash[23809]: cluster 2024-03-11T21:08:16.214355+0000 osd.0 (osd.0) 278 : cluster 0 4.6 scrub ok 2024-03-11T21:08:17.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:17 smithi146 bash[23809]: cluster 2024-03-11T21:08:16.267584+0000 mgr.y (mgr.24380) 647 : cluster 0 pgmap v727: 148 pgs: 148 active+clean; 3.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 852 B/s rd, 80 MiB/s wr, 1.29k op/s 2024-03-11T21:08:17.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:17 smithi146 bash[23809]: cluster 2024-03-11T21:08:16.450490+0000 osd.6 (osd.6) 280 : cluster 0 4.18 scrub starts 2024-03-11T21:08:17.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:17 smithi146 bash[23809]: cluster 2024-03-11T21:08:16.455019+0000 osd.6 (osd.6) 281 : cluster 0 4.18 scrub ok 2024-03-11T21:08:17.794 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:08:17.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:17.820 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:08:17.820 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:08:17.921 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:08:17.948 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:08:17.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:17.951 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:08:18.052 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:08:18.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:18.081 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:08:18.081 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:08:18.181 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:18.219 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:18.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:18.224 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:08:18.325 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:18.366 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:18.368 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 42 16 62444 62428 92.8899 80.9375 0.0188692 0.0107591 2024-03-11T21:08:18.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:18.370 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:08:18.471 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:18.496 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:18.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:18.499 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:08:18.600 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:18.641 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:18.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:18.645 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:08:18.746 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:08:18.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:18 smithi146 bash[23809]: cluster 2024-03-11T21:08:17.474915+0000 osd.1 (osd.1) 153 : cluster 0 4.12 scrub starts 2024-03-11T21:08:18.756 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:18 smithi146 bash[23809]: cluster 2024-03-11T21:08:17.480518+0000 osd.1 (osd.1) 154 : cluster 0 4.12 scrub ok 2024-03-11T21:08:18.773 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:08:18.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:18.776 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:08:18.779 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:08:18 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:08:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:08:18.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:18 smithi023 bash[21372]: cluster 2024-03-11T21:08:17.474915+0000 osd.1 (osd.1) 153 : cluster 0 4.12 scrub starts 2024-03-11T21:08:18.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:18 smithi023 bash[21372]: cluster 2024-03-11T21:08:17.480518+0000 osd.1 (osd.1) 154 : cluster 0 4.12 scrub ok 2024-03-11T21:08:18.780 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:18 smithi023 bash[29146]: cluster 2024-03-11T21:08:17.474915+0000 osd.1 (osd.1) 153 : cluster 0 4.12 scrub starts 2024-03-11T21:08:18.780 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:18 smithi023 bash[29146]: cluster 2024-03-11T21:08:17.480518+0000 osd.1 (osd.1) 154 : cluster 0 4.12 scrub ok 2024-03-11T21:08:18.876 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:08:18.910 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:08:18.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:18.912 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:08:19.013 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:08:19.042 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:08:19.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:19.048 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:08:19.149 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:19.172 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:19.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:19.176 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:08:19.277 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:19.304 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:19.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:19.306 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:08:19.370 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 43 16 63593 63577 92.3996 71.8125 0.0140557 0.0108175 2024-03-11T21:08:19.407 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:08:19.438 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:08:19.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:19.441 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:08:19.542 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:19.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:19.574 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:19.574 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:08:19.675 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:08:19.705 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:08:19.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:19.711 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:08:19.711 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:19 smithi146 bash[23809]: cluster 2024-03-11T21:08:18.121573+0000 osd.7 (osd.7) 409 : cluster 0 3.11 scrub starts 2024-03-11T21:08:19.711 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:19 smithi146 bash[23809]: cluster 2024-03-11T21:08:18.124014+0000 osd.7 (osd.7) 410 : cluster 0 3.11 scrub ok 2024-03-11T21:08:19.711 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:19 smithi146 bash[23809]: cluster 2024-03-11T21:08:18.228347+0000 osd.0 (osd.0) 279 : cluster 0 4.16 scrub starts 2024-03-11T21:08:19.711 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:19 smithi146 bash[23809]: cluster 2024-03-11T21:08:18.233537+0000 osd.0 (osd.0) 280 : cluster 0 4.16 scrub ok 2024-03-11T21:08:19.711 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:19 smithi146 bash[23809]: cluster 2024-03-11T21:08:18.269350+0000 mgr.y (mgr.24380) 648 : cluster 0 pgmap v728: 148 pgs: 148 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 641 B/s rd, 63 MiB/s wr, 1.01k op/s 2024-03-11T21:08:19.711 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:19 smithi146 bash[23809]: cluster 2024-03-11T21:08:18.378917+0000 osd.3 (osd.3) 366 : cluster 0 6.1f scrub starts 2024-03-11T21:08:19.711 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:19 smithi146 bash[23809]: cluster 2024-03-11T21:08:18.541425+0000 osd.3 (osd.3) 367 : cluster 0 6.1f scrub ok 2024-03-11T21:08:19.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:19 smithi023 bash[21372]: cluster 2024-03-11T21:08:18.121573+0000 osd.7 (osd.7) 409 : cluster 0 3.11 scrub starts 2024-03-11T21:08:19.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:19 smithi023 bash[21372]: cluster 2024-03-11T21:08:18.124014+0000 osd.7 (osd.7) 410 : cluster 0 3.11 scrub ok 2024-03-11T21:08:19.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:19 smithi023 bash[21372]: cluster 2024-03-11T21:08:18.228347+0000 osd.0 (osd.0) 279 : cluster 0 4.16 scrub starts 2024-03-11T21:08:19.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:19 smithi023 bash[21372]: cluster 2024-03-11T21:08:18.233537+0000 osd.0 (osd.0) 280 : cluster 0 4.16 scrub ok 2024-03-11T21:08:19.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:19 smithi023 bash[21372]: cluster 2024-03-11T21:08:18.269350+0000 mgr.y (mgr.24380) 648 : cluster 0 pgmap v728: 148 pgs: 148 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 641 B/s rd, 63 MiB/s wr, 1.01k op/s 2024-03-11T21:08:19.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:19 smithi023 bash[21372]: cluster 2024-03-11T21:08:18.378917+0000 osd.3 (osd.3) 366 : cluster 0 6.1f scrub starts 2024-03-11T21:08:19.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:19 smithi023 bash[21372]: cluster 2024-03-11T21:08:18.541425+0000 osd.3 (osd.3) 367 : cluster 0 6.1f scrub ok 2024-03-11T21:08:19.780 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:19 smithi023 bash[29146]: cluster 2024-03-11T21:08:18.121573+0000 osd.7 (osd.7) 409 : cluster 0 3.11 scrub starts 2024-03-11T21:08:19.780 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:19 smithi023 bash[29146]: cluster 2024-03-11T21:08:18.124014+0000 osd.7 (osd.7) 410 : cluster 0 3.11 scrub ok 2024-03-11T21:08:19.780 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:19 smithi023 bash[29146]: cluster 2024-03-11T21:08:18.228347+0000 osd.0 (osd.0) 279 : cluster 0 4.16 scrub starts 2024-03-11T21:08:19.781 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:19 smithi023 bash[29146]: cluster 2024-03-11T21:08:18.233537+0000 osd.0 (osd.0) 280 : cluster 0 4.16 scrub ok 2024-03-11T21:08:19.781 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:19 smithi023 bash[29146]: cluster 2024-03-11T21:08:18.269350+0000 mgr.y (mgr.24380) 648 : cluster 0 pgmap v728: 148 pgs: 148 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 641 B/s rd, 63 MiB/s wr, 1.01k op/s 2024-03-11T21:08:19.781 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:19 smithi023 bash[29146]: cluster 2024-03-11T21:08:18.378917+0000 osd.3 (osd.3) 366 : cluster 0 6.1f scrub starts 2024-03-11T21:08:19.781 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:19 smithi023 bash[29146]: cluster 2024-03-11T21:08:18.541425+0000 osd.3 (osd.3) 367 : cluster 0 6.1f scrub ok 2024-03-11T21:08:19.811 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:08:19.837 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:08:19.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:19.842 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:08:19.942 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:08:19.970 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:08:19.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:19.975 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:08:20.076 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:08:20.106 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:08:20.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:20.111 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:08:20.211 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:20.242 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:20.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:20.245 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:08:20.345 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:08:20.368 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 44 16 64872 64856 92.1162 79.9375 0.0136102 0.0108506 2024-03-11T21:08:20.372 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:08:20.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:20.376 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:08:20.477 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:20.489 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:20 smithi146 bash[23809]: cluster 2024-03-11T21:08:19.339479+0000 osd.3 (osd.3) 368 : cluster 0 5.0 scrub starts 2024-03-11T21:08:20.490 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:20 smithi146 bash[23809]: cluster 2024-03-11T21:08:19.340639+0000 osd.3 (osd.3) 369 : cluster 0 5.0 scrub ok 2024-03-11T21:08:20.490 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:20 smithi146 bash[23809]: cluster 2024-03-11T21:08:19.533565+0000 osd.1 (osd.1) 155 : cluster 0 3.19 scrub starts 2024-03-11T21:08:20.490 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:20 smithi146 bash[23809]: cluster 2024-03-11T21:08:19.539505+0000 osd.1 (osd.1) 156 : cluster 0 3.19 scrub ok 2024-03-11T21:08:20.501 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:20.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:20.503 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:08:20.604 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:08:20.610 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:20 smithi023 bash[21372]: cluster 2024-03-11T21:08:19.339479+0000 osd.3 (osd.3) 368 : cluster 0 5.0 scrub starts 2024-03-11T21:08:20.610 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:20 smithi023 bash[21372]: cluster 2024-03-11T21:08:19.340639+0000 osd.3 (osd.3) 369 : cluster 0 5.0 scrub ok 2024-03-11T21:08:20.610 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:20 smithi023 bash[21372]: cluster 2024-03-11T21:08:19.533565+0000 osd.1 (osd.1) 155 : cluster 0 3.19 scrub starts 2024-03-11T21:08:20.610 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:20 smithi023 bash[21372]: cluster 2024-03-11T21:08:19.539505+0000 osd.1 (osd.1) 156 : cluster 0 3.19 scrub ok 2024-03-11T21:08:20.611 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:20 smithi023 bash[29146]: cluster 2024-03-11T21:08:19.339479+0000 osd.3 (osd.3) 368 : cluster 0 5.0 scrub starts 2024-03-11T21:08:20.611 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:20 smithi023 bash[29146]: cluster 2024-03-11T21:08:19.340639+0000 osd.3 (osd.3) 369 : cluster 0 5.0 scrub ok 2024-03-11T21:08:20.611 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:20 smithi023 bash[29146]: cluster 2024-03-11T21:08:19.533565+0000 osd.1 (osd.1) 155 : cluster 0 3.19 scrub starts 2024-03-11T21:08:20.611 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:20 smithi023 bash[29146]: cluster 2024-03-11T21:08:19.539505+0000 osd.1 (osd.1) 156 : cluster 0 3.19 scrub ok 2024-03-11T21:08:20.639 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:08:20.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:20.641 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:08:20.702 INFO:teuthology.orchestra.run.smithi023.stdout:osd.0: {} 2024-03-11T21:08:20.743 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:08:20.760 INFO:teuthology.orchestra.run.smithi023.stdout:osd.1: {} 2024-03-11T21:08:20.778 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:08:20.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:20.784 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:08:20.805 INFO:teuthology.orchestra.run.smithi023.stdout:osd.2: {} 2024-03-11T21:08:20.849 INFO:teuthology.orchestra.run.smithi023.stdout:osd.3: {} 2024-03-11T21:08:20.886 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:08:20.895 INFO:teuthology.orchestra.run.smithi023.stdout:osd.4: {} 2024-03-11T21:08:20.909 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:08:20.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:20.914 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:08:20.937 INFO:teuthology.orchestra.run.smithi023.stdout:osd.5: {} 2024-03-11T21:08:20.974 INFO:teuthology.orchestra.run.smithi023.stdout:osd.6: {} 2024-03-11T21:08:21.015 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:08:21.019 INFO:teuthology.orchestra.run.smithi023.stdout:osd.7: {} 2024-03-11T21:08:21.024 INFO:teuthology.orchestra.run.smithi023.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:08:21.024 INFO:teuthology.orchestra.run.smithi023.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:08:21.024 INFO:teuthology.orchestra.run.smithi023.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:08:21.024 INFO:teuthology.orchestra.run.smithi023.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:08:21.024 INFO:teuthology.orchestra.run.smithi023.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:08:21.024 INFO:teuthology.orchestra.run.smithi023.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:08:21.025 INFO:teuthology.orchestra.run.smithi023.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:08:21.025 INFO:teuthology.orchestra.run.smithi023.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:08:21.042 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:08:21.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:21.046 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:08:21.147 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:21.174 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:21.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:21.180 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:08:21.280 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:21.317 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:21.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:21.320 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:08:21.368 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 45 15 66003 65988 91.6413 70.75 0.00574897 0.0109071 2024-03-11T21:08:21.421 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:21.566 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:21.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:21.573 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:08:21.674 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:21.684 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:21 smithi023 bash[21372]: cluster 2024-03-11T21:08:20.252611+0000 osd.0 (osd.0) 281 : cluster 0 6.0 scrub starts 2024-03-11T21:08:21.684 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:21 smithi023 bash[21372]: cluster 2024-03-11T21:08:20.270175+0000 mgr.y (mgr.24380) 649 : cluster 0 pgmap v729: 148 pgs: 148 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 89 MiB/s wr, 1.43k op/s 2024-03-11T21:08:21.684 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:21 smithi023 bash[21372]: cluster 2024-03-11T21:08:20.317774+0000 osd.0 (osd.0) 282 : cluster 0 6.0 scrub ok 2024-03-11T21:08:21.684 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:21 smithi023 bash[21372]: cluster 2024-03-11T21:08:20.325157+0000 osd.3 (osd.3) 370 : cluster 0 4.14 scrub starts 2024-03-11T21:08:21.684 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:21 smithi023 bash[21372]: cluster 2024-03-11T21:08:20.327684+0000 osd.3 (osd.3) 371 : cluster 0 4.14 scrub ok 2024-03-11T21:08:21.684 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:21 smithi023 bash[21372]: audit 2024-03-11T21:08:20.647294+0000 mon.c (mon.2) 82 : audit 0 from='client.? 172.21.15.23:0/997466785' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:08:21.684 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:21 smithi023 bash[21372]: audit 2024-03-11T21:08:20.726409+0000 mon.a (mon.0) 1652 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:08:21.684 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:21 smithi023 bash[21372]: audit 2024-03-11T21:08:20.728389+0000 mon.a (mon.0) 1653 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:08:21.685 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:21 smithi023 bash[29146]: cluster 2024-03-11T21:08:20.252611+0000 osd.0 (osd.0) 281 : cluster 0 6.0 scrub starts 2024-03-11T21:08:21.685 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:21 smithi023 bash[29146]: cluster 2024-03-11T21:08:20.270175+0000 mgr.y (mgr.24380) 649 : cluster 0 pgmap v729: 148 pgs: 148 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 89 MiB/s wr, 1.43k op/s 2024-03-11T21:08:21.685 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:21 smithi023 bash[29146]: cluster 2024-03-11T21:08:20.317774+0000 osd.0 (osd.0) 282 : cluster 0 6.0 scrub ok 2024-03-11T21:08:21.685 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:21 smithi023 bash[29146]: cluster 2024-03-11T21:08:20.325157+0000 osd.3 (osd.3) 370 : cluster 0 4.14 scrub starts 2024-03-11T21:08:21.685 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:21 smithi023 bash[29146]: cluster 2024-03-11T21:08:20.327684+0000 osd.3 (osd.3) 371 : cluster 0 4.14 scrub ok 2024-03-11T21:08:21.685 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:21 smithi023 bash[29146]: audit 2024-03-11T21:08:20.647294+0000 mon.c (mon.2) 82 : audit 0 from='client.? 172.21.15.23:0/997466785' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:08:21.685 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:21 smithi023 bash[29146]: audit 2024-03-11T21:08:20.726409+0000 mon.a (mon.0) 1652 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:08:21.685 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:21 smithi023 bash[29146]: audit 2024-03-11T21:08:20.728389+0000 mon.a (mon.0) 1653 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:08:21.699 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:21.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:21.703 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:08:21.759 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:21 smithi146 bash[23809]: cluster 2024-03-11T21:08:20.252611+0000 osd.0 (osd.0) 281 : cluster 0 6.0 scrub starts 2024-03-11T21:08:21.759 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:21 smithi146 bash[23809]: cluster 2024-03-11T21:08:20.270175+0000 mgr.y (mgr.24380) 649 : cluster 0 pgmap v729: 148 pgs: 148 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 89 MiB/s wr, 1.43k op/s 2024-03-11T21:08:21.759 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:21 smithi146 bash[23809]: cluster 2024-03-11T21:08:20.317774+0000 osd.0 (osd.0) 282 : cluster 0 6.0 scrub ok 2024-03-11T21:08:21.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:21 smithi146 bash[23809]: cluster 2024-03-11T21:08:20.325157+0000 osd.3 (osd.3) 370 : cluster 0 4.14 scrub starts 2024-03-11T21:08:21.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:21 smithi146 bash[23809]: cluster 2024-03-11T21:08:20.327684+0000 osd.3 (osd.3) 371 : cluster 0 4.14 scrub ok 2024-03-11T21:08:21.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:21 smithi146 bash[23809]: audit 2024-03-11T21:08:20.647294+0000 mon.c (mon.2) 82 : audit 0 from='client.? 172.21.15.23:0/997466785' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:08:21.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:21 smithi146 bash[23809]: audit 2024-03-11T21:08:20.726409+0000 mon.a (mon.0) 1652 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:08:21.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:21 smithi146 bash[23809]: audit 2024-03-11T21:08:20.728389+0000 mon.a (mon.0) 1653 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:08:21.804 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:21.826 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:21.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:21.829 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:08:21.930 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:21.960 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:21.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:21.962 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:08:22.063 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:08:22.089 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:08:22.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:22.092 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:08:22.193 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:22.215 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:22.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:22.217 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:08:22.318 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:22.350 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:22.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:22.352 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:08:22.368 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 46 16 67164 67148 91.225 72.5 0.0328695 0.0109563 2024-03-11T21:08:22.453 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:22.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:22.488 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:22.488 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:08:22.589 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:22.598 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:22 smithi146 bash[23809]: cluster 2024-03-11T21:08:21.284238+0000 osd.3 (osd.3) 372 : cluster 0 3.18 scrub starts 2024-03-11T21:08:22.598 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:22 smithi146 bash[23809]: cluster 2024-03-11T21:08:21.295906+0000 osd.3 (osd.3) 373 : cluster 0 3.18 scrub ok 2024-03-11T21:08:22.598 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:22 smithi146 bash[23809]: cluster 2024-03-11T21:08:21.314839+0000 osd.0 (osd.0) 283 : cluster 0 3.12 scrub starts 2024-03-11T21:08:22.598 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:22 smithi146 bash[23809]: cluster 2024-03-11T21:08:21.327564+0000 osd.0 (osd.0) 284 : cluster 0 3.12 scrub ok 2024-03-11T21:08:22.598 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:22 smithi146 bash[23809]: cluster 2024-03-11T21:08:21.457343+0000 osd.1 (osd.1) 157 : cluster 0 purged_snaps scrub starts 2024-03-11T21:08:22.599 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:22 smithi146 bash[23809]: cluster 2024-03-11T21:08:21.457447+0000 osd.1 (osd.1) 158 : cluster 0 purged_snaps scrub ok 2024-03-11T21:08:22.599 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:22 smithi146 bash[23809]: cluster 2024-03-11T21:08:21.527036+0000 osd.1 (osd.1) 159 : cluster 0 2.15 scrub starts 2024-03-11T21:08:22.599 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:22 smithi146 bash[23809]: cluster 2024-03-11T21:08:21.529635+0000 osd.1 (osd.1) 160 : cluster 0 2.15 scrub ok 2024-03-11T21:08:22.616 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:22.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:22.623 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:08:22.724 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:08:22.744 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:08:22.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:22.746 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:08:22.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:22 smithi023 bash[21372]: cluster 2024-03-11T21:08:21.284238+0000 osd.3 (osd.3) 372 : cluster 0 3.18 scrub starts 2024-03-11T21:08:22.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:22 smithi023 bash[21372]: cluster 2024-03-11T21:08:21.295906+0000 osd.3 (osd.3) 373 : cluster 0 3.18 scrub ok 2024-03-11T21:08:22.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:22 smithi023 bash[21372]: cluster 2024-03-11T21:08:21.314839+0000 osd.0 (osd.0) 283 : cluster 0 3.12 scrub starts 2024-03-11T21:08:22.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:22 smithi023 bash[21372]: cluster 2024-03-11T21:08:21.327564+0000 osd.0 (osd.0) 284 : cluster 0 3.12 scrub ok 2024-03-11T21:08:22.769 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:22 smithi023 bash[21372]: cluster 2024-03-11T21:08:21.457343+0000 osd.1 (osd.1) 157 : cluster 0 purged_snaps scrub starts 2024-03-11T21:08:22.769 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:22 smithi023 bash[21372]: cluster 2024-03-11T21:08:21.457447+0000 osd.1 (osd.1) 158 : cluster 0 purged_snaps scrub ok 2024-03-11T21:08:22.769 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:22 smithi023 bash[21372]: cluster 2024-03-11T21:08:21.527036+0000 osd.1 (osd.1) 159 : cluster 0 2.15 scrub starts 2024-03-11T21:08:22.769 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:22 smithi023 bash[21372]: cluster 2024-03-11T21:08:21.529635+0000 osd.1 (osd.1) 160 : cluster 0 2.15 scrub ok 2024-03-11T21:08:22.769 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:22 smithi023 bash[29146]: cluster 2024-03-11T21:08:21.284238+0000 osd.3 (osd.3) 372 : cluster 0 3.18 scrub starts 2024-03-11T21:08:22.769 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:22 smithi023 bash[29146]: cluster 2024-03-11T21:08:21.295906+0000 osd.3 (osd.3) 373 : cluster 0 3.18 scrub ok 2024-03-11T21:08:22.769 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:22 smithi023 bash[29146]: cluster 2024-03-11T21:08:21.314839+0000 osd.0 (osd.0) 283 : cluster 0 3.12 scrub starts 2024-03-11T21:08:22.769 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:22 smithi023 bash[29146]: cluster 2024-03-11T21:08:21.327564+0000 osd.0 (osd.0) 284 : cluster 0 3.12 scrub ok 2024-03-11T21:08:22.769 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:22 smithi023 bash[29146]: cluster 2024-03-11T21:08:21.457343+0000 osd.1 (osd.1) 157 : cluster 0 purged_snaps scrub starts 2024-03-11T21:08:22.769 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:22 smithi023 bash[29146]: cluster 2024-03-11T21:08:21.457447+0000 osd.1 (osd.1) 158 : cluster 0 purged_snaps scrub ok 2024-03-11T21:08:22.769 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:22 smithi023 bash[29146]: cluster 2024-03-11T21:08:21.527036+0000 osd.1 (osd.1) 159 : cluster 0 2.15 scrub starts 2024-03-11T21:08:22.769 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:22 smithi023 bash[29146]: cluster 2024-03-11T21:08:21.529635+0000 osd.1 (osd.1) 160 : cluster 0 2.15 scrub ok 2024-03-11T21:08:22.847 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:22.890 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:22.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:22.893 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:08:22.994 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:08:23.014 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:08:23.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:23.016 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:08:23.117 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:23.143 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:23.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:23.146 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:08:23.247 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:08:23.269 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:08:23.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:23.272 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:08:23.369 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 47 16 68098 68082 90.526 58.375 0.00722739 0.0110416 2024-03-11T21:08:23.373 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:08:23.409 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:08:23.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:23.413 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:08:23.514 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:23.529 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:23 smithi023 bash[21372]: cluster 2024-03-11T21:08:22.115618+0000 osd.7 (osd.7) 411 : cluster 0 10.1 scrub starts 2024-03-11T21:08:23.529 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:23 smithi023 bash[21372]: cluster 2024-03-11T21:08:22.270815+0000 mgr.y (mgr.24380) 650 : cluster 0 pgmap v730: 148 pgs: 148 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 101 MiB/s wr, 1.62k op/s 2024-03-11T21:08:23.529 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:23 smithi023 bash[21372]: cluster 2024-03-11T21:08:22.479423+0000 osd.1 (osd.1) 161 : cluster 0 2.6 scrub starts 2024-03-11T21:08:23.529 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:23 smithi023 bash[21372]: cluster 2024-03-11T21:08:22.484102+0000 mon.a (mon.0) 1654 : cluster 0 osdmap e325: 8 total, 8 up, 8 in 2024-03-11T21:08:23.529 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:23 smithi023 bash[21372]: cluster 2024-03-11T21:08:22.484791+0000 osd.1 (osd.1) 162 : cluster 0 2.6 scrub ok 2024-03-11T21:08:23.529 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:23 smithi023 bash[21372]: cluster 2024-03-11T21:08:22.794519+0000 osd.5 (osd.5) 355 : cluster 0 6.d scrub starts 2024-03-11T21:08:23.530 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:23 smithi023 bash[29146]: cluster 2024-03-11T21:08:22.115618+0000 osd.7 (osd.7) 411 : cluster 0 10.1 scrub starts 2024-03-11T21:08:23.530 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:23 smithi023 bash[29146]: cluster 2024-03-11T21:08:22.270815+0000 mgr.y (mgr.24380) 650 : cluster 0 pgmap v730: 148 pgs: 148 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 101 MiB/s wr, 1.62k op/s 2024-03-11T21:08:23.530 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:23 smithi023 bash[29146]: cluster 2024-03-11T21:08:22.479423+0000 osd.1 (osd.1) 161 : cluster 0 2.6 scrub starts 2024-03-11T21:08:23.530 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:23 smithi023 bash[29146]: cluster 2024-03-11T21:08:22.484102+0000 mon.a (mon.0) 1654 : cluster 0 osdmap e325: 8 total, 8 up, 8 in 2024-03-11T21:08:23.530 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:23 smithi023 bash[29146]: cluster 2024-03-11T21:08:22.484791+0000 osd.1 (osd.1) 162 : cluster 0 2.6 scrub ok 2024-03-11T21:08:23.530 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:23 smithi023 bash[29146]: cluster 2024-03-11T21:08:22.794519+0000 osd.5 (osd.5) 355 : cluster 0 6.d scrub starts 2024-03-11T21:08:23.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:23.552 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:23.552 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:08:23.653 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:08:23.679 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:08:23.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:23.685 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:08:23.759 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:23 smithi146 bash[23809]: cluster 2024-03-11T21:08:22.115618+0000 osd.7 (osd.7) 411 : cluster 0 10.1 scrub starts 2024-03-11T21:08:23.759 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:23 smithi146 bash[23809]: cluster 2024-03-11T21:08:22.270815+0000 mgr.y (mgr.24380) 650 : cluster 0 pgmap v730: 148 pgs: 148 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 101 MiB/s wr, 1.62k op/s 2024-03-11T21:08:23.759 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:23 smithi146 bash[23809]: cluster 2024-03-11T21:08:22.479423+0000 osd.1 (osd.1) 161 : cluster 0 2.6 scrub starts 2024-03-11T21:08:23.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:23 smithi146 bash[23809]: cluster 2024-03-11T21:08:22.484102+0000 mon.a (mon.0) 1654 : cluster 0 osdmap e325: 8 total, 8 up, 8 in 2024-03-11T21:08:23.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:23 smithi146 bash[23809]: cluster 2024-03-11T21:08:22.484791+0000 osd.1 (osd.1) 162 : cluster 0 2.6 scrub ok 2024-03-11T21:08:23.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:23 smithi146 bash[23809]: cluster 2024-03-11T21:08:22.794519+0000 osd.5 (osd.5) 355 : cluster 0 6.d scrub starts 2024-03-11T21:08:23.786 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:08:23.811 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:08:23.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:23.816 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:08:23.916 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:08:23.944 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:08:23.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:23.947 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:08:24.048 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:08:24.074 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:08:24.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:24.078 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:08:24.179 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:08:24.216 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:08:24.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:24.218 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:08:24.319 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:08:24.344 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:08:24.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:24.346 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:08:24.369 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 48 16 69231 69215 90.1152 70.8125 0.00632361 0.0110922 2024-03-11T21:08:24.447 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:24.465 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:24.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:24.466 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:08:24.567 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:08:24.599 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:08:24.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:24.603 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:08:24.704 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:08:24.713 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:24 smithi146 bash[23809]: cluster 2024-03-11T21:08:22.890682+0000 osd.5 (osd.5) 356 : cluster 0 6.d scrub ok 2024-03-11T21:08:24.713 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:24 smithi146 bash[23809]: cluster 2024-03-11T21:08:23.465599+0000 osd.1 (osd.1) 163 : cluster 0 3.0 scrub starts 2024-03-11T21:08:24.713 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:24 smithi146 bash[23809]: cluster 2024-03-11T21:08:23.471353+0000 osd.1 (osd.1) 164 : cluster 0 3.0 scrub ok 2024-03-11T21:08:24.713 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:24 smithi146 bash[23809]: cluster 2024-03-11T21:08:23.497924+0000 osd.7 (osd.7) 412 : cluster 0 10.1 scrub ok 2024-03-11T21:08:24.713 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:24 smithi146 bash[23809]: cluster 2024-03-11T21:08:23.784672+0000 osd.5 (osd.5) 357 : cluster 0 3.1d scrub starts 2024-03-11T21:08:24.713 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:24 smithi146 bash[23809]: cluster 2024-03-11T21:08:23.793238+0000 osd.5 (osd.5) 358 : cluster 0 3.1d scrub ok 2024-03-11T21:08:24.728 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:08:24.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:24.730 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:08:24.808 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:24 smithi023 bash[21372]: cluster 2024-03-11T21:08:22.890682+0000 osd.5 (osd.5) 356 : cluster 0 6.d scrub ok 2024-03-11T21:08:24.808 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:24 smithi023 bash[21372]: cluster 2024-03-11T21:08:23.465599+0000 osd.1 (osd.1) 163 : cluster 0 3.0 scrub starts 2024-03-11T21:08:24.808 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:24 smithi023 bash[21372]: cluster 2024-03-11T21:08:23.471353+0000 osd.1 (osd.1) 164 : cluster 0 3.0 scrub ok 2024-03-11T21:08:24.808 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:24 smithi023 bash[21372]: cluster 2024-03-11T21:08:23.497924+0000 osd.7 (osd.7) 412 : cluster 0 10.1 scrub ok 2024-03-11T21:08:24.808 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:24 smithi023 bash[21372]: cluster 2024-03-11T21:08:23.784672+0000 osd.5 (osd.5) 357 : cluster 0 3.1d scrub starts 2024-03-11T21:08:24.808 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:24 smithi023 bash[21372]: cluster 2024-03-11T21:08:23.793238+0000 osd.5 (osd.5) 358 : cluster 0 3.1d scrub ok 2024-03-11T21:08:24.808 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:24 smithi023 bash[29146]: cluster 2024-03-11T21:08:22.890682+0000 osd.5 (osd.5) 356 : cluster 0 6.d scrub ok 2024-03-11T21:08:24.808 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:24 smithi023 bash[29146]: cluster 2024-03-11T21:08:23.465599+0000 osd.1 (osd.1) 163 : cluster 0 3.0 scrub starts 2024-03-11T21:08:24.809 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:24 smithi023 bash[29146]: cluster 2024-03-11T21:08:23.471353+0000 osd.1 (osd.1) 164 : cluster 0 3.0 scrub ok 2024-03-11T21:08:24.809 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:24 smithi023 bash[29146]: cluster 2024-03-11T21:08:23.497924+0000 osd.7 (osd.7) 412 : cluster 0 10.1 scrub ok 2024-03-11T21:08:24.809 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:24 smithi023 bash[29146]: cluster 2024-03-11T21:08:23.784672+0000 osd.5 (osd.5) 357 : cluster 0 3.1d scrub starts 2024-03-11T21:08:24.809 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:24 smithi023 bash[29146]: cluster 2024-03-11T21:08:23.793238+0000 osd.5 (osd.5) 358 : cluster 0 3.1d scrub ok 2024-03-11T21:08:24.831 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:08:24.866 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:08:24.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:24.868 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:08:24.969 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:08:24.996 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:08:25.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:25.001 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:08:25.102 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:25.137 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:25.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:25.142 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:08:25.243 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:08:25.270 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:08:25.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:25.272 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:08:25.369 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 49 16 70373 70357 89.7326 71.375 0.00848534 0.0111392 2024-03-11T21:08:25.373 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:08:25.401 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:08:25.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:25.403 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:08:25.504 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:08:25.509 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:25 smithi146 bash[23809]: cluster 2024-03-11T21:08:23.899434+0000 osd.3 (osd.3) 374 : cluster 0 purged_snaps scrub starts 2024-03-11T21:08:25.509 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:25 smithi146 bash[23809]: cluster 2024-03-11T21:08:23.899521+0000 osd.3 (osd.3) 375 : cluster 0 purged_snaps scrub ok 2024-03-11T21:08:25.509 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:25 smithi146 bash[23809]: cluster 2024-03-11T21:08:24.271257+0000 mgr.y (mgr.24380) 651 : cluster 0 pgmap v732: 148 pgs: 148 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 570 B/s rd, 56 MiB/s wr, 888 op/s 2024-03-11T21:08:25.509 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:25 smithi146 bash[23809]: cluster 2024-03-11T21:08:24.316006+0000 osd.0 (osd.0) 285 : cluster 0 3.1 scrub starts 2024-03-11T21:08:25.509 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:25 smithi146 bash[23809]: cluster 2024-03-11T21:08:24.322396+0000 osd.3 (osd.3) 376 : cluster 0 4.10 scrub starts 2024-03-11T21:08:25.509 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:25 smithi146 bash[23809]: cluster 2024-03-11T21:08:24.323596+0000 osd.3 (osd.3) 377 : cluster 0 4.10 scrub ok 2024-03-11T21:08:25.509 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:25 smithi146 bash[23809]: cluster 2024-03-11T21:08:24.324620+0000 osd.0 (osd.0) 286 : cluster 0 3.1 scrub ok 2024-03-11T21:08:25.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:25 smithi146 bash[23809]: cluster 2024-03-11T21:08:24.492411+0000 mon.a (mon.0) 1655 : cluster 0 osdmap e326: 8 total, 8 up, 8 in 2024-03-11T21:08:25.537 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:08:25.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:25.540 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:08:25.629 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:25 smithi023 bash[21372]: cluster 2024-03-11T21:08:23.899434+0000 osd.3 (osd.3) 374 : cluster 0 purged_snaps scrub starts 2024-03-11T21:08:25.629 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:25 smithi023 bash[21372]: cluster 2024-03-11T21:08:23.899521+0000 osd.3 (osd.3) 375 : cluster 0 purged_snaps scrub ok 2024-03-11T21:08:25.629 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:25 smithi023 bash[21372]: cluster 2024-03-11T21:08:24.271257+0000 mgr.y (mgr.24380) 651 : cluster 0 pgmap v732: 148 pgs: 148 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 570 B/s rd, 56 MiB/s wr, 888 op/s 2024-03-11T21:08:25.629 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:25 smithi023 bash[21372]: cluster 2024-03-11T21:08:24.316006+0000 osd.0 (osd.0) 285 : cluster 0 3.1 scrub starts 2024-03-11T21:08:25.629 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:25 smithi023 bash[21372]: cluster 2024-03-11T21:08:24.322396+0000 osd.3 (osd.3) 376 : cluster 0 4.10 scrub starts 2024-03-11T21:08:25.629 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:25 smithi023 bash[21372]: cluster 2024-03-11T21:08:24.323596+0000 osd.3 (osd.3) 377 : cluster 0 4.10 scrub ok 2024-03-11T21:08:25.629 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:25 smithi023 bash[21372]: cluster 2024-03-11T21:08:24.324620+0000 osd.0 (osd.0) 286 : cluster 0 3.1 scrub ok 2024-03-11T21:08:25.629 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:25 smithi023 bash[21372]: cluster 2024-03-11T21:08:24.492411+0000 mon.a (mon.0) 1655 : cluster 0 osdmap e326: 8 total, 8 up, 8 in 2024-03-11T21:08:25.641 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:08:25.651 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:25 smithi023 bash[29146]: cluster 2024-03-11T21:08:23.899434+0000 osd.3 (osd.3) 374 : cluster 0 purged_snaps scrub starts 2024-03-11T21:08:25.652 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:25 smithi023 bash[29146]: cluster 2024-03-11T21:08:23.899521+0000 osd.3 (osd.3) 375 : cluster 0 purged_snaps scrub ok 2024-03-11T21:08:25.652 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:25 smithi023 bash[29146]: cluster 2024-03-11T21:08:24.271257+0000 mgr.y (mgr.24380) 651 : cluster 0 pgmap v732: 148 pgs: 148 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 570 B/s rd, 56 MiB/s wr, 888 op/s 2024-03-11T21:08:25.652 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:25 smithi023 bash[29146]: cluster 2024-03-11T21:08:24.316006+0000 osd.0 (osd.0) 285 : cluster 0 3.1 scrub starts 2024-03-11T21:08:25.652 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:25 smithi023 bash[29146]: cluster 2024-03-11T21:08:24.322396+0000 osd.3 (osd.3) 376 : cluster 0 4.10 scrub starts 2024-03-11T21:08:25.652 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:25 smithi023 bash[29146]: cluster 2024-03-11T21:08:24.323596+0000 osd.3 (osd.3) 377 : cluster 0 4.10 scrub ok 2024-03-11T21:08:25.652 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:25 smithi023 bash[29146]: cluster 2024-03-11T21:08:24.324620+0000 osd.0 (osd.0) 286 : cluster 0 3.1 scrub ok 2024-03-11T21:08:25.652 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:25 smithi023 bash[29146]: cluster 2024-03-11T21:08:24.492411+0000 mon.a (mon.0) 1655 : cluster 0 osdmap e326: 8 total, 8 up, 8 in 2024-03-11T21:08:25.669 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:08:25.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:25.675 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:08:25.759 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:08:25 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:08:25.775 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:25.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:25.804 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:25.804 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:08:25.905 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:08:25.930 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:08:25.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:25.932 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:08:26.033 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:08:26.050 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:08:26.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:26.052 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:08:26.153 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:08:26.185 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:08:26.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:26.188 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:08:26.288 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:08:26.321 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:08:26.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:26.324 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:08:26.369 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 50 16 71358 71342 89.1691 61.5625 0.00689365 0.0112093 2024-03-11T21:08:26.424 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:08:26.455 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:08:26.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:26.460 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:08:26.560 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:26.571 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:26 smithi146 bash[23809]: cluster 2024-03-11T21:08:25.322364+0000 osd.6 (osd.6) 282 : cluster 0 10.f scrub starts 2024-03-11T21:08:26.571 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:26 smithi146 bash[23809]: cluster 2024-03-11T21:08:25.790160+0000 osd.5 (osd.5) 359 : cluster 0 2.2 scrub starts 2024-03-11T21:08:26.571 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:26 smithi146 bash[23809]: cluster 2024-03-11T21:08:25.793943+0000 osd.5 (osd.5) 360 : cluster 0 2.2 scrub ok 2024-03-11T21:08:26.589 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:26.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:26.591 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:08:26.692 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:26.724 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:26.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:26.729 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:08:26.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:26 smithi023 bash[21372]: cluster 2024-03-11T21:08:25.322364+0000 osd.6 (osd.6) 282 : cluster 0 10.f scrub starts 2024-03-11T21:08:26.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:26 smithi023 bash[21372]: cluster 2024-03-11T21:08:25.790160+0000 osd.5 (osd.5) 359 : cluster 0 2.2 scrub starts 2024-03-11T21:08:26.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:26 smithi023 bash[21372]: cluster 2024-03-11T21:08:25.793943+0000 osd.5 (osd.5) 360 : cluster 0 2.2 scrub ok 2024-03-11T21:08:26.780 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:26 smithi023 bash[29146]: cluster 2024-03-11T21:08:25.322364+0000 osd.6 (osd.6) 282 : cluster 0 10.f scrub starts 2024-03-11T21:08:26.780 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:26 smithi023 bash[29146]: cluster 2024-03-11T21:08:25.790160+0000 osd.5 (osd.5) 359 : cluster 0 2.2 scrub starts 2024-03-11T21:08:26.780 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:26 smithi023 bash[29146]: cluster 2024-03-11T21:08:25.793943+0000 osd.5 (osd.5) 360 : cluster 0 2.2 scrub ok 2024-03-11T21:08:26.829 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:08:26.861 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:08:26.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:26.863 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:08:26.964 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:26.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:27.000 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:27.000 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:08:27.101 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:08:27.129 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:08:27.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:27.133 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:08:27.234 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:27.263 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:27.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:27.267 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:08:27.368 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:27.369 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 51 16 72484 72468 88.8004 70.375 0.0188864 0.0112557 2024-03-11T21:08:27.399 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:27.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:27.401 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:08:27.502 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:08:27.526 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:08:27.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:27.532 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:08:27.633 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:08:27.658 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:08:27.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:27.661 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:08:27.759 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:27 smithi146 bash[23809]: audit 2024-03-11T21:08:25.725191+0000 mgr.y (mgr.24380) 652 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:08:27.759 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:27 smithi146 bash[23809]: cluster 2024-03-11T21:08:26.272171+0000 mgr.y (mgr.24380) 653 : cluster 0 pgmap v734: 148 pgs: 148 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 90 MiB/s wr, 1.44k op/s 2024-03-11T21:08:27.759 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:27 smithi146 bash[23809]: cluster 2024-03-11T21:08:26.375560+0000 osd.1 (osd.1) 165 : cluster 0 4.f scrub starts 2024-03-11T21:08:27.759 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:27 smithi146 bash[23809]: cluster 2024-03-11T21:08:26.382270+0000 osd.1 (osd.1) 166 : cluster 0 4.f scrub ok 2024-03-11T21:08:27.762 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:27.786 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:27.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:27.790 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:08:27.791 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:27 smithi023 bash[21372]: audit 2024-03-11T21:08:25.725191+0000 mgr.y (mgr.24380) 652 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:08:27.791 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:27 smithi023 bash[21372]: cluster 2024-03-11T21:08:26.272171+0000 mgr.y (mgr.24380) 653 : cluster 0 pgmap v734: 148 pgs: 148 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 90 MiB/s wr, 1.44k op/s 2024-03-11T21:08:27.791 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:27 smithi023 bash[21372]: cluster 2024-03-11T21:08:26.375560+0000 osd.1 (osd.1) 165 : cluster 0 4.f scrub starts 2024-03-11T21:08:27.791 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:27 smithi023 bash[21372]: cluster 2024-03-11T21:08:26.382270+0000 osd.1 (osd.1) 166 : cluster 0 4.f scrub ok 2024-03-11T21:08:27.791 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:27 smithi023 bash[29146]: audit 2024-03-11T21:08:25.725191+0000 mgr.y (mgr.24380) 652 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:08:27.791 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:27 smithi023 bash[29146]: cluster 2024-03-11T21:08:26.272171+0000 mgr.y (mgr.24380) 653 : cluster 0 pgmap v734: 148 pgs: 148 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 90 MiB/s wr, 1.44k op/s 2024-03-11T21:08:27.791 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:27 smithi023 bash[29146]: cluster 2024-03-11T21:08:26.375560+0000 osd.1 (osd.1) 165 : cluster 0 4.f scrub starts 2024-03-11T21:08:27.791 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:27 smithi023 bash[29146]: cluster 2024-03-11T21:08:26.382270+0000 osd.1 (osd.1) 166 : cluster 0 4.f scrub ok 2024-03-11T21:08:27.891 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:27.909 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:27.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:27.910 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:08:28.011 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:28.041 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:28.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:28.046 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:08:28.147 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:08:28.167 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:08:28.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:28.169 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:08:28.270 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:28.305 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:28.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:28.309 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:08:28.369 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 52 16 73384 73368 88.1744 56.25 0.00919457 0.0113364 2024-03-11T21:08:28.409 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:28.449 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:28.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:28.453 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:08:28.554 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:08:28.585 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:08:28.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:28.589 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:08:28.690 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:08:28.700 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:28 smithi146 bash[23809]: cluster 2024-03-11T21:08:27.083414+0000 osd.6 (osd.6) 283 : cluster 0 10.f scrub ok 2024-03-11T21:08:28.700 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:28 smithi146 bash[23809]: cluster 2024-03-11T21:08:27.752832+0000 osd.5 (osd.5) 361 : cluster 0 2.18 deep-scrub starts 2024-03-11T21:08:28.700 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:28 smithi146 bash[23809]: cluster 2024-03-11T21:08:27.756029+0000 osd.5 (osd.5) 362 : cluster 0 2.18 deep-scrub ok 2024-03-11T21:08:28.721 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:08:28.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:28.726 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:08:28.779 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:08:28 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:08:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:08:28.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:28 smithi023 bash[21372]: cluster 2024-03-11T21:08:27.083414+0000 osd.6 (osd.6) 283 : cluster 0 10.f scrub ok 2024-03-11T21:08:28.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:28 smithi023 bash[21372]: cluster 2024-03-11T21:08:27.752832+0000 osd.5 (osd.5) 361 : cluster 0 2.18 deep-scrub starts 2024-03-11T21:08:28.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:28 smithi023 bash[21372]: cluster 2024-03-11T21:08:27.756029+0000 osd.5 (osd.5) 362 : cluster 0 2.18 deep-scrub ok 2024-03-11T21:08:28.780 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:28 smithi023 bash[29146]: cluster 2024-03-11T21:08:27.083414+0000 osd.6 (osd.6) 283 : cluster 0 10.f scrub ok 2024-03-11T21:08:28.780 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:28 smithi023 bash[29146]: cluster 2024-03-11T21:08:27.752832+0000 osd.5 (osd.5) 361 : cluster 0 2.18 deep-scrub starts 2024-03-11T21:08:28.781 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:28 smithi023 bash[29146]: cluster 2024-03-11T21:08:27.756029+0000 osd.5 (osd.5) 362 : cluster 0 2.18 deep-scrub ok 2024-03-11T21:08:28.826 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:08:28.850 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:08:28.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:28.853 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:08:28.954 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:08:28.982 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:08:28.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:28.986 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:08:29.086 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:08:29.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:29.126 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:08:29.126 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:08:29.173 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T21:08:29.227 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:08:29.250 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:08:29.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:29.252 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:08:29.353 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:29.369 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 53 16 74475 74459 87.7966 68.1875 0.00464708 0.0113851 2024-03-11T21:08:29.376 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:29.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:29.387 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:08:29.487 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:08:29.513 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:29 smithi023 bash[21372]: cluster 2024-03-11T21:08:28.273001+0000 mgr.y (mgr.24380) 654 : cluster 0 pgmap v735: 148 pgs: 148 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 55 MiB/s wr, 885 op/s 2024-03-11T21:08:29.513 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:29 smithi023 bash[21372]: cluster 2024-03-11T21:08:28.364387+0000 osd.3 (osd.3) 378 : cluster 0 3.b scrub starts 2024-03-11T21:08:29.513 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:29 smithi023 bash[21372]: cluster 2024-03-11T21:08:28.368552+0000 osd.3 (osd.3) 379 : cluster 0 3.b scrub ok 2024-03-11T21:08:29.514 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:08:29.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:29.516 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:08:29.577 INFO:teuthology.orchestra.run.smithi023.stdout:ERROR: (22) Invalid argument 2024-03-11T21:08:29.578 INFO:teuthology.orchestra.run.smithi023.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:08:29.617 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:08:29.623 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:29 smithi023 bash[29146]: cluster 2024-03-11T21:08:28.273001+0000 mgr.y (mgr.24380) 654 : cluster 0 pgmap v735: 148 pgs: 148 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 55 MiB/s wr, 885 op/s 2024-03-11T21:08:29.624 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:29 smithi023 bash[29146]: cluster 2024-03-11T21:08:28.364387+0000 osd.3 (osd.3) 378 : cluster 0 3.b scrub starts 2024-03-11T21:08:29.624 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:29 smithi023 bash[29146]: cluster 2024-03-11T21:08:28.368552+0000 osd.3 (osd.3) 379 : cluster 0 3.b scrub ok 2024-03-11T21:08:29.631 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-11T21:08:29.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:29.649 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:08:29.649 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:08:29.663 INFO:teuthology.orchestra.run.smithi023.stderr:dumped all 2024-03-11T21:08:29.750 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:29.759 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:29 smithi146 bash[23809]: cluster 2024-03-11T21:08:28.273001+0000 mgr.y (mgr.24380) 654 : cluster 0 pgmap v735: 148 pgs: 148 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 55 MiB/s wr, 885 op/s 2024-03-11T21:08:29.759 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:29 smithi146 bash[23809]: cluster 2024-03-11T21:08:28.364387+0000 osd.3 (osd.3) 378 : cluster 0 3.b scrub starts 2024-03-11T21:08:29.759 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:29 smithi146 bash[23809]: cluster 2024-03-11T21:08:28.368552+0000 osd.3 (osd.3) 379 : cluster 0 3.b scrub ok 2024-03-11T21:08:29.777 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:29.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:29.781 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:08:29.882 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:08:29.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:29.925 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:08:29.925 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:08:30.026 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:30.052 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:30.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:30.054 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:08:30.155 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:30.180 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:30.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:30.184 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:08:30.285 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:30.312 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:30.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:30.314 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:08:30.370 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 54 16 75481 75465 87.335 62.875 0.005908 0.0114452 2024-03-11T21:08:30.415 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:30.446 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:30.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:30.452 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:08:30.539 INFO:teuthology.orchestra.run.smithi023.stderr:noscrub is set 2024-03-11T21:08:30.553 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:30.573 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:30.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:30.576 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:08:30.677 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:08:30.686 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:30 smithi146 bash[23809]: cluster 2024-03-11T21:08:29.499528+0000 osd.1 (osd.1) 167 : cluster 0 6.4 scrub starts 2024-03-11T21:08:30.686 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:30 smithi146 bash[23809]: cluster 2024-03-11T21:08:29.503920+0000 osd.1 (osd.1) 168 : cluster 0 6.4 scrub ok 2024-03-11T21:08:30.686 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:30 smithi146 bash[23809]: audit 2024-03-11T21:08:29.616420+0000 mgr.y (mgr.24380) 655 : audit 0 from='client.24847 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:08:30.686 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:30 smithi146 bash[23809]: audit 2024-03-11T21:08:30.087362+0000 mon.b (mon.1) 41 : audit 1 from='client.? 172.21.15.23:0/768702493' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:08:30.686 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:30 smithi146 bash[23809]: audit 2024-03-11T21:08:30.088672+0000 mon.a (mon.0) 1656 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:08:30.705 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:08:30.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:30.708 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:08:30.780 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:30 smithi023 bash[29146]: cluster 2024-03-11T21:08:29.499528+0000 osd.1 (osd.1) 167 : cluster 0 6.4 scrub starts 2024-03-11T21:08:30.780 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:30 smithi023 bash[29146]: cluster 2024-03-11T21:08:29.503920+0000 osd.1 (osd.1) 168 : cluster 0 6.4 scrub ok 2024-03-11T21:08:30.780 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:30 smithi023 bash[29146]: audit 2024-03-11T21:08:29.616420+0000 mgr.y (mgr.24380) 655 : audit 0 from='client.24847 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:08:30.780 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:30 smithi023 bash[29146]: audit 2024-03-11T21:08:30.087362+0000 mon.b (mon.1) 41 : audit 1 from='client.? 172.21.15.23:0/768702493' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:08:30.780 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:30 smithi023 bash[29146]: audit 2024-03-11T21:08:30.088672+0000 mon.a (mon.0) 1656 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:08:30.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:30 smithi023 bash[21372]: cluster 2024-03-11T21:08:29.499528+0000 osd.1 (osd.1) 167 : cluster 0 6.4 scrub starts 2024-03-11T21:08:30.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:30 smithi023 bash[21372]: cluster 2024-03-11T21:08:29.503920+0000 osd.1 (osd.1) 168 : cluster 0 6.4 scrub ok 2024-03-11T21:08:30.781 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:30 smithi023 bash[21372]: audit 2024-03-11T21:08:29.616420+0000 mgr.y (mgr.24380) 655 : audit 0 from='client.24847 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:08:30.781 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:30 smithi023 bash[21372]: audit 2024-03-11T21:08:30.087362+0000 mon.b (mon.1) 41 : audit 1 from='client.? 172.21.15.23:0/768702493' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:08:30.781 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:30 smithi023 bash[21372]: audit 2024-03-11T21:08:30.088672+0000 mon.a (mon.0) 1656 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:08:30.809 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:30.832 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:30.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:30.837 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:08:30.938 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:08:30.966 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:08:30.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:30.971 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:08:31.072 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:08:31.107 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:08:31.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:31.111 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:08:31.212 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:08:31.239 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:08:31.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:31.241 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:08:31.342 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:08:31.368 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:08:31.369 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 55 16 76562 76546 86.9755 67.5625 0.0159726 0.0114924 2024-03-11T21:08:31.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:31.370 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:08:31.471 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:08:31.497 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:08:31.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:31.500 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:08:31.600 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:08:31.632 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:08:31.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:31.638 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:08:31.738 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:08:31.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:31 smithi146 bash[23809]: cluster 2024-03-11T21:08:30.273752+0000 mgr.y (mgr.24380) 656 : cluster 0 pgmap v736: 148 pgs: 148 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 74 MiB/s wr, 1.18k op/s 2024-03-11T21:08:31.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:31 smithi146 bash[23809]: cluster 2024-03-11T21:08:30.520122+0000 mon.a (mon.0) 1657 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:08:31.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:31 smithi146 bash[23809]: audit 2024-03-11T21:08:30.535849+0000 mon.a (mon.0) 1658 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:08:31.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:31 smithi146 bash[23809]: cluster 2024-03-11T21:08:30.540163+0000 mon.a (mon.0) 1659 : cluster 0 osdmap e327: 8 total, 8 up, 8 in 2024-03-11T21:08:31.767 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:08:31.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:31.772 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:08:31.853 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:31 smithi023 bash[21372]: cluster 2024-03-11T21:08:30.273752+0000 mgr.y (mgr.24380) 656 : cluster 0 pgmap v736: 148 pgs: 148 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 74 MiB/s wr, 1.18k op/s 2024-03-11T21:08:31.853 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:31 smithi023 bash[21372]: cluster 2024-03-11T21:08:30.520122+0000 mon.a (mon.0) 1657 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:08:31.853 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:31 smithi023 bash[21372]: audit 2024-03-11T21:08:30.535849+0000 mon.a (mon.0) 1658 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:08:31.853 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:31 smithi023 bash[21372]: cluster 2024-03-11T21:08:30.540163+0000 mon.a (mon.0) 1659 : cluster 0 osdmap e327: 8 total, 8 up, 8 in 2024-03-11T21:08:31.853 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:31 smithi023 bash[29146]: cluster 2024-03-11T21:08:30.273752+0000 mgr.y (mgr.24380) 656 : cluster 0 pgmap v736: 148 pgs: 148 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 74 MiB/s wr, 1.18k op/s 2024-03-11T21:08:31.853 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:31 smithi023 bash[29146]: cluster 2024-03-11T21:08:30.520122+0000 mon.a (mon.0) 1657 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:08:31.853 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:31 smithi023 bash[29146]: audit 2024-03-11T21:08:30.535849+0000 mon.a (mon.0) 1658 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:08:31.854 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:31 smithi023 bash[29146]: cluster 2024-03-11T21:08:30.540163+0000 mon.a (mon.0) 1659 : cluster 0 osdmap e327: 8 total, 8 up, 8 in 2024-03-11T21:08:31.874 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:08:31.904 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:08:31.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:31.909 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:08:32.010 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:08:32.036 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:08:32.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:32.041 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:08:32.142 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:32.164 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:32.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:32.170 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:08:32.271 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:08:32.296 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:08:32.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:32.297 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:08:32.370 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 56 16 77602 77586 86.583 65 0.00692556 0.0115451 2024-03-11T21:08:32.398 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:08:32.424 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:08:32.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:32.429 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:08:32.530 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:32.553 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:32.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:32.555 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:08:32.656 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:08:32.688 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:08:32.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:32.691 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:08:32.776 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-03-11T21:08:32.792 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:32.825 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:32.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:32.827 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:08:32.873 INFO:teuthology.orchestra.run.smithi023.stdout:{"pg_ready":true,"pg_map":{"version":735,"stamp":"2024-03-11T21:08:28.272385+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4591524264,"num_objects":70250,"num_object_clones":0,"num_object_copies":210750,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70250,"num_whiteouts":0,"num_read":2295,"num_read_kb":2141,"num_write":70953,"num_write_kb":4485789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":190,"num_bytes_recovered":1775932,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71107,"ondisk_log_size":71107,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":16221960,"kb_used_data":13358976,"kb_used_omap":21,"kb_used_meta":2862890,"kb_avail":733739256,"statfs":{"total":767960285184,"available":751348998144,"internally_reserved":0,"allocated":13679591424,"data_stored":13676246105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22089,"internal_metadata":2931599799},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1,6,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":464257024,"num_objects":7084,"num_object_clones":0,"num_object_copies":21252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7084,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":7084,"num_write_kb":453376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002869"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":559,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.490658+0000","last_change":"2024-03-11T21:07:57.433433+0000","last_active":"2024-03-11T21:08:24.490658+0000","last_peered":"2024-03-11T21:08:24.490658+0000","last_clean":"2024-03-11T21:08:24.490658+0000","last_became_active":"2024-03-11T20:50:34.338390+0000","last_became_peered":"2024-03-11T20:50:34.338390+0000","last_unstale":"2024-03-11T21:08:24.490658+0000","last_undegraded":"2024-03-11T21:08:24.490658+0000","last_fullsized":"2024-03-11T21:08:24.490658+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:07:57.433167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:07:57.433167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:11.759348+0000","scrub_duration":119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":563,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.490208+0000","last_change":"2024-03-11T21:08:09.344727+0000","last_active":"2024-03-11T21:08:24.490208+0000","last_peered":"2024-03-11T21:08:24.490208+0000","last_clean":"2024-03-11T21:08:24.490208+0000","last_became_active":"2024-03-11T20:50:32.210194+0000","last_became_peered":"2024-03-11T20:50:32.210194+0000","last_unstale":"2024-03-11T21:08:24.490208+0000","last_undegraded":"2024-03-11T21:08:24.490208+0000","last_fullsized":"2024-03-11T21:08:24.490208+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:08:09.344640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:08:09.344640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:22.075739+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":274,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.511616+0000","last_change":"2024-03-11T21:08:13.139913+0000","last_active":"2024-03-11T21:08:24.511616+0000","last_peered":"2024-03-11T21:08:24.511616+0000","last_clean":"2024-03-11T21:08:24.511616+0000","last_became_active":"2024-03-11T21:06:28.760306+0000","last_became_peered":"2024-03-11T21:06:28.760306+0000","last_unstale":"2024-03-11T21:08:24.511616+0000","last_undegraded":"2024-03-11T21:08:24.511616+0000","last_fullsized":"2024-03-11T21:08:24.511616+0000","mapping_epoch":297,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:08:13.139773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:08:13.139773+0000","last_clean_scrub_stamp":"2024-03-11T21:08:13.139773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:27.982920+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"207'20","reported_seq":599,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.490675+0000","last_change":"2024-03-11T21:07:46.374511+0000","last_active":"2024-03-11T21:08:24.490675+0000","last_peered":"2024-03-11T21:08:24.490675+0000","last_clean":"2024-03-11T21:08:24.490675+0000","last_became_active":"2024-03-11T20:50:30.101721+0000","last_became_peered":"2024-03-11T20:50:30.101721+0000","last_unstale":"2024-03-11T21:08:24.490675+0000","last_undegraded":"2024-03-11T21:08:24.490675+0000","last_fullsized":"2024-03-11T21:08:24.490675+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'20","last_scrub_stamp":"2024-03-11T21:07:46.374362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:07:46.374362+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:08:52.320899+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":392,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.508329+0000","last_change":"2024-03-11T21:08:10.985718+0000","last_active":"2024-03-11T21:08:24.508329+0000","last_peered":"2024-03-11T21:08:24.508329+0000","last_clean":"2024-03-11T21:08:24.508329+0000","last_became_active":"2024-03-11T21:05:17.316714+0000","last_became_peered":"2024-03-11T21:05:17.316714+0000","last_unstale":"2024-03-11T21:08:24.508329+0000","last_undegraded":"2024-03-11T21:08:24.508329+0000","last_fullsized":"2024-03-11T21:08:24.508329+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:08:10.985486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:55.545758+0000","last_clean_scrub_stamp":"2024-03-11T21:08:10.985486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:19.423326+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":396,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.484940+0000","last_change":"2024-03-11T21:08:16.455139+0000","last_active":"2024-03-11T21:08:24.484940+0000","last_peered":"2024-03-11T21:08:24.484940+0000","last_clean":"2024-03-11T21:08:24.484940+0000","last_became_active":"2024-03-11T21:05:17.319272+0000","last_became_peered":"2024-03-11T21:05:17.319272+0000","last_unstale":"2024-03-11T21:08:24.484940+0000","last_undegraded":"2024-03-11T21:08:24.484940+0000","last_fullsized":"2024-03-11T21:08:24.484940+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:08:16.455026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:54:13.660325+0000","last_clean_scrub_stamp":"2024-03-11T21:08:16.455026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:37.154803+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":564,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.490547+0000","last_change":"2024-03-11T21:06:26.534331+0000","last_active":"2024-03-11T21:08:24.490547+0000","last_peered":"2024-03-11T21:08:24.490547+0000","last_clean":"2024-03-11T21:08:24.490547+0000","last_became_active":"2024-03-11T20:50:28.058274+0000","last_became_peered":"2024-03-11T20:50:28.058274+0000","last_unstale":"2024-03-11T21:08:24.490547+0000","last_undegraded":"2024-03-11T21:08:24.490547+0000","last_fullsized":"2024-03-11T21:08:24.490547+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:06:26.534045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:06:26.534045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"207'19","reported_seq":620,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.511543+0000","last_change":"2024-03-11T21:08:05.905831+0000","last_active":"2024-03-11T21:08:24.511543+0000","last_peered":"2024-03-11T21:08:24.511543+0000","last_clean":"2024-03-11T21:08:24.511543+0000","last_became_active":"2024-03-11T20:50:30.092638+0000","last_became_peered":"2024-03-11T20:50:30.092638+0000","last_unstale":"2024-03-11T21:08:24.511543+0000","last_undegraded":"2024-03-11T21:08:24.511543+0000","last_fullsized":"2024-03-11T21:08:24.511543+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'19","last_scrub_stamp":"2024-03-11T21:08:05.905696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:08:05.905696+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:31.482408+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":580,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.506994+0000","last_change":"2024-03-11T21:08:05.992436+0000","last_active":"2024-03-11T21:08:24.506994+0000","last_peered":"2024-03-11T21:08:24.506994+0000","last_clean":"2024-03-11T21:08:24.506994+0000","last_became_active":"2024-03-11T21:00:48.460694+0000","last_became_peered":"2024-03-11T21:00:48.460694+0000","last_unstale":"2024-03-11T21:08:24.506994+0000","last_undegraded":"2024-03-11T21:08:24.506994+0000","last_fullsized":"2024-03-11T21:08:24.506994+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:08:05.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:53:07.792697+0000","last_clean_scrub_stamp":"2024-03-11T21:08:05.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:22.040040+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":380,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.490753+0000","last_change":"2024-03-11T21:06:32.574420+0000","last_active":"2024-03-11T21:08:24.490753+0000","last_peered":"2024-03-11T21:08:24.490753+0000","last_clean":"2024-03-11T21:08:24.490753+0000","last_became_active":"2024-03-11T21:05:17.316491+0000","last_became_peered":"2024-03-11T21:05:17.316491+0000","last_unstale":"2024-03-11T21:08:24.490753+0000","last_undegraded":"2024-03-11T21:08:24.490753+0000","last_fullsized":"2024-03-11T21:08:24.490753+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:06:32.574201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:03:36.230647+0000","last_clean_scrub_stamp":"2024-03-11T21:06:32.574201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":585,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.512995+0000","last_change":"2024-03-11T21:07:44.227628+0000","last_active":"2024-03-11T21:08:24.512995+0000","last_peered":"2024-03-11T21:08:24.512995+0000","last_clean":"2024-03-11T21:08:24.512995+0000","last_became_active":"2024-03-11T20:50:28.079225+0000","last_became_peered":"2024-03-11T20:50:28.079225+0000","last_unstale":"2024-03-11T21:08:24.512995+0000","last_undegraded":"2024-03-11T21:08:24.512995+0000","last_fullsized":"2024-03-11T21:08:24.512995+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:07:44.227438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:07:44.227438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:10.870994+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"207'27","reported_seq":683,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.507319+0000","last_change":"2024-03-11T21:07:37.782008+0000","last_active":"2024-03-11T21:08:24.507319+0000","last_peered":"2024-03-11T21:08:24.507319+0000","last_clean":"2024-03-11T21:08:24.507319+0000","last_became_active":"2024-03-11T21:00:48.462943+0000","last_became_peered":"2024-03-11T21:00:48.462943+0000","last_unstale":"2024-03-11T21:08:24.507319+0000","last_undegraded":"2024-03-11T21:08:24.507319+0000","last_fullsized":"2024-03-11T21:08:24.507319+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'27","last_scrub_stamp":"2024-03-11T21:07:37.781649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:07:37.781649+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:08:59.411336+0000","scrub_duration":139,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":267,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.511715+0000","last_change":"2024-03-11T21:07:43.867303+0000","last_active":"2024-03-11T21:08:24.511715+0000","last_peered":"2024-03-11T21:08:24.511715+0000","last_clean":"2024-03-11T21:08:24.511715+0000","last_became_active":"2024-03-11T21:06:28.757201+0000","last_became_peered":"2024-03-11T21:06:28.757201+0000","last_unstale":"2024-03-11T21:08:24.511715+0000","last_undegraded":"2024-03-11T21:08:24.511715+0000","last_fullsized":"2024-03-11T21:08:24.511715+0000","mapping_epoch":297,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:07:43.867063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:07:43.867063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:08:58.839728+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":350,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.490157+0000","last_change":"2024-03-11T21:08:01.360006+0000","last_active":"2024-03-11T21:08:24.490157+0000","last_peered":"2024-03-11T21:08:24.490157+0000","last_clean":"2024-03-11T21:08:24.490157+0000","last_became_active":"2024-03-11T21:05:17.313388+0000","last_became_peered":"2024-03-11T21:05:17.313388+0000","last_unstale":"2024-03-11T21:08:24.490157+0000","last_undegraded":"2024-03-11T21:08:24.490157+0000","last_fullsized":"2024-03-11T21:08:24.490157+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:08:01.359878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:53:50.599029+0000","last_clean_scrub_stamp":"2024-03-11T21:08:01.359878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:23.039160+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":593,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.512636+0000","last_change":"2024-03-11T21:08:03.095456+0000","last_active":"2024-03-11T21:08:24.512636+0000","last_peered":"2024-03-11T21:08:24.512636+0000","last_clean":"2024-03-11T21:08:24.512636+0000","last_became_active":"2024-03-11T20:50:28.068978+0000","last_became_peered":"2024-03-11T20:50:28.068978+0000","last_unstale":"2024-03-11T21:08:24.512636+0000","last_undegraded":"2024-03-11T21:08:24.512636+0000","last_fullsized":"2024-03-11T21:08:24.512636+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:08:03.095340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:54:07.155130+0000","last_clean_scrub_stamp":"2024-03-11T21:08:03.095340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:18.246317+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"207'23","reported_seq":621,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.508864+0000","last_change":"2024-03-11T21:08:23.793392+0000","last_active":"2024-03-11T21:08:24.508864+0000","last_peered":"2024-03-11T21:08:24.508864+0000","last_clean":"2024-03-11T21:08:24.508864+0000","last_became_active":"2024-03-11T21:00:27.952093+0000","last_became_peered":"2024-03-11T21:00:27.952093+0000","last_unstale":"2024-03-11T21:08:24.508864+0000","last_undegraded":"2024-03-11T21:08:24.508864+0000","last_fullsized":"2024-03-11T21:08:24.508864+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'23","last_scrub_stamp":"2024-03-11T21:08:23.793245+0000","last_deep_scrub":"110'17","last_deep_scrub_stamp":"2024-03-11T20:53:04.802488+0000","last_clean_scrub_stamp":"2024-03-11T21:08:23.793245+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:41.066649+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":560,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.490968+0000","last_change":"2024-03-11T21:08:18.541547+0000","last_active":"2024-03-11T21:08:24.490968+0000","last_peered":"2024-03-11T21:08:24.490968+0000","last_clean":"2024-03-11T21:08:24.490968+0000","last_became_active":"2024-03-11T20:50:34.334558+0000","last_became_peered":"2024-03-11T20:50:34.334558+0000","last_unstale":"2024-03-11T21:08:24.490968+0000","last_undegraded":"2024-03-11T21:08:24.490968+0000","last_fullsized":"2024-03-11T21:08:24.490968+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:08:18.541431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:55:59.892394+0000","last_clean_scrub_stamp":"2024-03-11T21:08:18.541431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:22.496111+0000","scrub_duration":163,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":410,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.508511+0000","last_change":"2024-03-11T21:08:13.903187+0000","last_active":"2024-03-11T21:08:24.508511+0000","last_peered":"2024-03-11T21:08:24.508511+0000","last_clean":"2024-03-11T21:08:24.508511+0000","last_became_active":"2024-03-11T21:00:48.461013+0000","last_became_peered":"2024-03-11T21:00:48.461013+0000","last_unstale":"2024-03-11T21:08:24.508511+0000","last_undegraded":"2024-03-11T21:08:24.508511+0000","last_fullsized":"2024-03-11T21:08:24.508511+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:08:13.903063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:08:13.903063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:14.522260+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"77'1","reported_seq":579,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.490778+0000","last_change":"2024-03-11T21:07:38.421921+0000","last_active":"2024-03-11T21:08:24.490778+0000","last_peered":"2024-03-11T21:08:24.490778+0000","last_clean":"2024-03-11T21:08:24.490778+0000","last_became_active":"2024-03-11T20:50:28.068840+0000","last_became_peered":"2024-03-11T20:50:28.068840+0000","last_unstale":"2024-03-11T21:08:24.490778+0000","last_undegraded":"2024-03-11T21:08:24.490778+0000","last_fullsized":"2024-03-11T21:08:24.490778+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-03-11T21:07:38.421793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:07:38.421793+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:04.626281+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"207'19","reported_seq":430,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.486159+0000","last_change":"2024-03-11T21:07:48.503708+0000","last_active":"2024-03-11T21:08:24.486159+0000","last_peered":"2024-03-11T21:08:24.486159+0000","last_clean":"2024-03-11T21:08:24.486159+0000","last_became_active":"2024-03-11T21:05:17.443030+0000","last_became_peered":"2024-03-11T21:05:17.443030+0000","last_unstale":"2024-03-11T21:08:24.486159+0000","last_undegraded":"2024-03-11T21:08:24.486159+0000","last_fullsized":"2024-03-11T21:08:24.486159+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'19","last_scrub_stamp":"2024-03-11T21:07:48.503147+0000","last_deep_scrub":"207'19","last_deep_scrub_stamp":"2024-03-11T21:03:39.240089+0000","last_clean_scrub_stamp":"2024-03-11T21:07:48.503147+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:11.342210+0000","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":352,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.485419+0000","last_change":"2024-03-11T21:08:02.490787+0000","last_active":"2024-03-11T21:08:24.485419+0000","last_peered":"2024-03-11T21:08:24.485419+0000","last_clean":"2024-03-11T21:08:24.485419+0000","last_became_active":"2024-03-11T21:05:17.313986+0000","last_became_peered":"2024-03-11T21:05:17.313986+0000","last_unstale":"2024-03-11T21:08:24.485419+0000","last_undegraded":"2024-03-11T21:08:24.485419+0000","last_fullsized":"2024-03-11T21:08:24.485419+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:08:02.490701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:33.399510+0000","last_clean_scrub_stamp":"2024-03-11T21:08:02.490701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:32.053442+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":363,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.486290+0000","last_change":"2024-03-11T21:08:05.299006+0000","last_active":"2024-03-11T21:08:24.486290+0000","last_peered":"2024-03-11T21:08:24.486290+0000","last_clean":"2024-03-11T21:08:24.486290+0000","last_became_active":"2024-03-11T21:08:05.298552+0000","last_became_peered":"2024-03-11T21:08:05.298552+0000","last_unstale":"2024-03-11T21:08:24.486290+0000","last_undegraded":"2024-03-11T21:08:24.486290+0000","last_fullsized":"2024-03-11T21:08:24.486290+0000","mapping_epoch":319,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:07:48.513229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:24.666325+0000","last_clean_scrub_stamp":"2024-03-11T21:07:48.513229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:08:55.380668+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":594,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.485255+0000","last_change":"2024-03-11T21:08:01.481718+0000","last_active":"2024-03-11T21:08:24.485255+0000","last_peered":"2024-03-11T21:08:24.485255+0000","last_clean":"2024-03-11T21:08:24.485255+0000","last_became_active":"2024-03-11T21:00:27.956935+0000","last_became_peered":"2024-03-11T21:00:27.956935+0000","last_unstale":"2024-03-11T21:08:24.485255+0000","last_undegraded":"2024-03-11T21:08:24.485255+0000","last_fullsized":"2024-03-11T21:08:24.485255+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:08:01.481609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:08:01.481609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:11.212257+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"207'7","reported_seq":602,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.513019+0000","last_change":"2024-03-11T21:08:00.885293+0000","last_active":"2024-03-11T21:08:24.513019+0000","last_peered":"2024-03-11T21:08:24.513019+0000","last_clean":"2024-03-11T21:08:24.513019+0000","last_became_active":"2024-03-11T21:00:27.955296+0000","last_became_peered":"2024-03-11T21:00:27.955296+0000","last_unstale":"2024-03-11T21:08:24.513019+0000","last_undegraded":"2024-03-11T21:08:24.513019+0000","last_fullsized":"2024-03-11T21:08:24.513019+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'7","last_scrub_stamp":"2024-03-11T21:08:00.885119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:08:00.885119+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:12.183663+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":286,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.486409+0000","last_change":"2024-03-11T21:08:08.622255+0000","last_active":"2024-03-11T21:08:24.486409+0000","last_peered":"2024-03-11T21:08:24.486409+0000","last_clean":"2024-03-11T21:08:24.486409+0000","last_became_active":"2024-03-11T21:00:48.464694+0000","last_became_peered":"2024-03-11T21:00:48.464694+0000","last_unstale":"2024-03-11T21:08:24.486409+0000","last_undegraded":"2024-03-11T21:08:24.486409+0000","last_fullsized":"2024-03-11T21:08:24.486409+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:08:08.622054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:08:08.622054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:36.228544+0000","scrub_duration":107,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1f","version":"212'16","reported_seq":836,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.485306+0000","last_change":"2024-03-11T21:08:11.449942+0000","last_active":"2024-03-11T21:08:24.485306+0000","last_peered":"2024-03-11T21:08:24.485306+0000","last_clean":"2024-03-11T21:08:24.485306+0000","last_became_active":"2024-03-11T21:00:48.464350+0000","last_became_peered":"2024-03-11T21:00:48.464350+0000","last_unstale":"2024-03-11T21:08:24.485306+0000","last_undegraded":"2024-03-11T21:08:24.485306+0000","last_fullsized":"2024-03-11T21:08:24.485306+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"212'16","last_scrub_stamp":"2024-03-11T21:08:11.449706+0000","last_deep_scrub":"212'16","last_deep_scrub_stamp":"2024-03-11T21:06:47.821221+0000","last_clean_scrub_stamp":"2024-03-11T21:08:11.449706+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:13.593200+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"77'1","reported_seq":564,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.490235+0000","last_change":"2024-03-11T21:07:37.448905+0000","last_active":"2024-03-11T21:08:24.490235+0000","last_peered":"2024-03-11T21:08:24.490235+0000","last_clean":"2024-03-11T21:08:24.490235+0000","last_became_active":"2024-03-11T20:50:28.068827+0000","last_became_peered":"2024-03-11T20:50:28.068827+0000","last_unstale":"2024-03-11T21:08:24.490235+0000","last_undegraded":"2024-03-11T21:08:24.490235+0000","last_fullsized":"2024-03-11T21:08:24.490235+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-03-11T21:07:37.448705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:07:37.448705+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:04.563738+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"207'15","reported_seq":634,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.490121+0000","last_change":"2024-03-11T21:08:21.296049+0000","last_active":"2024-03-11T21:08:24.490121+0000","last_peered":"2024-03-11T21:08:24.490121+0000","last_clean":"2024-03-11T21:08:24.490121+0000","last_became_active":"2024-03-11T21:00:48.467513+0000","last_became_peered":"2024-03-11T21:00:48.467513+0000","last_unstale":"2024-03-11T21:08:24.490121+0000","last_undegraded":"2024-03-11T21:08:24.490121+0000","last_fullsized":"2024-03-11T21:08:24.490121+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'15","last_scrub_stamp":"2024-03-11T21:08:21.295914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:08:21.295914+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:50.946102+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"105'2","reported_seq":576,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.513456+0000","last_change":"2024-03-11T21:07:43.187688+0000","last_active":"2024-03-11T21:08:24.513456+0000","last_peered":"2024-03-11T21:08:24.513456+0000","last_clean":"2024-03-11T21:08:24.513456+0000","last_became_active":"2024-03-11T20:50:34.327356+0000","last_became_peered":"2024-03-11T20:50:34.327356+0000","last_unstale":"2024-03-11T21:08:24.513456+0000","last_undegraded":"2024-03-11T21:08:24.513456+0000","last_fullsized":"2024-03-11T21:08:24.513456+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'2","last_scrub_stamp":"2024-03-11T21:07:43.187429+0000","last_deep_scrub":"105'2","last_deep_scrub_stamp":"2024-03-11T21:03:41.971918+0000","last_clean_scrub_stamp":"2024-03-11T21:07:43.187429+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:03.259623+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":274,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.513739+0000","last_change":"2024-03-11T21:07:38.776015+0000","last_active":"2024-03-11T21:08:24.513739+0000","last_peered":"2024-03-11T21:08:24.513739+0000","last_clean":"2024-03-11T21:08:24.513739+0000","last_became_active":"2024-03-11T21:06:28.759784+0000","last_became_peered":"2024-03-11T21:06:28.759784+0000","last_unstale":"2024-03-11T21:08:24.513739+0000","last_undegraded":"2024-03-11T21:08:24.513739+0000","last_fullsized":"2024-03-11T21:08:24.513739+0000","mapping_epoch":297,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:07:38.775839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:06:32.968650+0000","last_clean_scrub_stamp":"2024-03-11T21:07:38.775839+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:05.777072+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":580,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.508052+0000","last_change":"2024-03-11T21:06:46.831529+0000","last_active":"2024-03-11T21:08:24.508052+0000","last_peered":"2024-03-11T21:08:24.508052+0000","last_clean":"2024-03-11T21:08:24.508052+0000","last_became_active":"2024-03-11T21:00:27.947614+0000","last_became_peered":"2024-03-11T21:00:27.947614+0000","last_unstale":"2024-03-11T21:08:24.508052+0000","last_undegraded":"2024-03-11T21:08:24.508052+0000","last_fullsized":"2024-03-11T21:08:24.508052+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:06:46.831406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:05:08.869152+0000","last_clean_scrub_stamp":"2024-03-11T21:06:46.831406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"207'27","reported_seq":324,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.485955+0000","last_change":"2024-03-11T21:08:19.539727+0000","last_active":"2024-03-11T21:08:24.485955+0000","last_peered":"2024-03-11T21:08:24.485955+0000","last_clean":"2024-03-11T21:08:24.485955+0000","last_became_active":"2024-03-11T21:00:48.457893+0000","last_became_peered":"2024-03-11T21:00:48.457893+0000","last_unstale":"2024-03-11T21:08:24.485955+0000","last_undegraded":"2024-03-11T21:08:24.485955+0000","last_fullsized":"2024-03-11T21:08:24.485955+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'27","last_scrub_stamp":"2024-03-11T21:08:19.539512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:08:19.539512+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:47.582896+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":559,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.490864+0000","last_change":"2024-03-11T21:07:41.485168+0000","last_active":"2024-03-11T21:08:24.490864+0000","last_peered":"2024-03-11T21:08:24.490864+0000","last_clean":"2024-03-11T21:08:24.490864+0000","last_became_active":"2024-03-11T20:50:34.337878+0000","last_became_peered":"2024-03-11T20:50:34.337878+0000","last_unstale":"2024-03-11T21:08:24.490864+0000","last_undegraded":"2024-03-11T21:08:24.490864+0000","last_fullsized":"2024-03-11T21:08:24.490864+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:07:41.485049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:52:11.462480+0000","last_clean_scrub_stamp":"2024-03-11T21:07:41.485049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:08:49.813123+0000","scrub_duration":123,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":563,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.490530+0000","last_change":"2024-03-11T21:06:53.533564+0000","last_active":"2024-03-11T21:08:24.490530+0000","last_peered":"2024-03-11T21:08:24.490530+0000","last_clean":"2024-03-11T21:08:24.490530+0000","last_became_active":"2024-03-11T20:50:32.646323+0000","last_became_peered":"2024-03-11T20:50:32.646323+0000","last_unstale":"2024-03-11T21:08:24.490530+0000","last_undegraded":"2024-03-11T21:08:24.490530+0000","last_fullsized":"2024-03-11T21:08:24.490530+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:06:53.533352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:06:53.533352+0000","last_clean_scrub_stamp":"2024-03-11T21:06:53.533352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":601,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.485513+0000","last_change":"2024-03-11T21:08:09.468417+0000","last_active":"2024-03-11T21:08:24.485513+0000","last_peered":"2024-03-11T21:08:24.485513+0000","last_clean":"2024-03-11T21:08:24.485513+0000","last_became_active":"2024-03-11T20:50:28.086513+0000","last_became_peered":"2024-03-11T20:50:28.086513+0000","last_unstale":"2024-03-11T21:08:24.485513+0000","last_undegraded":"2024-03-11T21:08:24.485513+0000","last_fullsized":"2024-03-11T21:08:24.485513+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:08:09.468339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:08:09.468339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:31.864241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"207'19","reported_seq":646,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.508729+0000","last_change":"2024-03-11T21:08:12.934811+0000","last_active":"2024-03-11T21:08:24.508729+0000","last_peered":"2024-03-11T21:08:24.508729+0000","last_clean":"2024-03-11T21:08:24.508729+0000","last_became_active":"2024-03-11T21:00:48.464707+0000","last_became_peered":"2024-03-11T21:00:48.464707+0000","last_unstale":"2024-03-11T21:08:24.508729+0000","last_undegraded":"2024-03-11T21:08:24.508729+0000","last_fullsized":"2024-03-11T21:08:24.508729+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'19","last_scrub_stamp":"2024-03-11T21:08:12.934635+0000","last_deep_scrub":"145'13","last_deep_scrub_stamp":"2024-03-11T20:55:53.560287+0000","last_clean_scrub_stamp":"2024-03-11T21:08:12.934635+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:30.699153+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":574,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.512887+0000","last_change":"2024-03-11T21:07:47.158773+0000","last_active":"2024-03-11T21:08:24.512887+0000","last_peered":"2024-03-11T21:08:24.512887+0000","last_clean":"2024-03-11T21:08:24.512887+0000","last_became_active":"2024-03-11T21:00:27.973405+0000","last_became_peered":"2024-03-11T21:00:27.973405+0000","last_unstale":"2024-03-11T21:08:24.512887+0000","last_undegraded":"2024-03-11T21:08:24.512887+0000","last_fullsized":"2024-03-11T21:08:24.512887+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:07:47.158672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:07:47.158672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:08:59.062029+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":576,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.490158+0000","last_change":"2024-03-11T21:08:24.323683+0000","last_active":"2024-03-11T21:08:24.490158+0000","last_peered":"2024-03-11T21:08:24.490158+0000","last_clean":"2024-03-11T21:08:24.490158+0000","last_became_active":"2024-03-11T21:00:48.472812+0000","last_became_peered":"2024-03-11T21:00:48.472812+0000","last_unstale":"2024-03-11T21:08:24.490158+0000","last_undegraded":"2024-03-11T21:08:24.490158+0000","last_fullsized":"2024-03-11T21:08:24.490158+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:08:24.323601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:08:24.323601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:27.123101+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":587,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.508628+0000","last_change":"2024-03-11T21:07:56.868732+0000","last_active":"2024-03-11T21:08:24.508628+0000","last_peered":"2024-03-11T21:08:24.508628+0000","last_clean":"2024-03-11T21:08:24.508628+0000","last_became_active":"2024-03-11T20:50:28.075760+0000","last_became_peered":"2024-03-11T20:50:28.075760+0000","last_unstale":"2024-03-11T21:08:24.508628+0000","last_undegraded":"2024-03-11T21:08:24.508628+0000","last_fullsized":"2024-03-11T21:08:24.508628+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:07:56.868482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:58:55.977289+0000","last_clean_scrub_stamp":"2024-03-11T21:07:56.868482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:17.637518+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"207'12","reported_seq":242,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.511426+0000","last_change":"2024-03-11T21:07:57.887460+0000","last_active":"2024-03-11T21:08:24.511426+0000","last_peered":"2024-03-11T21:08:24.511426+0000","last_clean":"2024-03-11T21:08:24.511426+0000","last_became_active":"2024-03-11T21:06:28.763971+0000","last_became_peered":"2024-03-11T21:06:28.763971+0000","last_unstale":"2024-03-11T21:08:24.511426+0000","last_undegraded":"2024-03-11T21:08:24.511426+0000","last_fullsized":"2024-03-11T21:08:24.511426+0000","mapping_epoch":297,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'12","last_scrub_stamp":"2024-03-11T21:07:57.887318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:07:57.887318+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:12.897002+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":16,"num_read_kb":13,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.11","version":"186'2","reported_seq":561,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.490085+0000","last_change":"2024-03-11T21:08:00.343845+0000","last_active":"2024-03-11T21:08:24.490085+0000","last_peered":"2024-03-11T21:08:24.490085+0000","last_clean":"2024-03-11T21:08:24.490085+0000","last_became_active":"2024-03-11T20:50:34.339326+0000","last_became_peered":"2024-03-11T20:50:34.339326+0000","last_unstale":"2024-03-11T21:08:24.490085+0000","last_undegraded":"2024-03-11T21:08:24.490085+0000","last_fullsized":"2024-03-11T21:08:24.490085+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"186'2","last_scrub_stamp":"2024-03-11T21:08:00.343694+0000","last_deep_scrub":"186'2","last_deep_scrub_stamp":"2024-03-11T20:59:04.534850+0000","last_clean_scrub_stamp":"2024-03-11T21:08:00.343694+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:08.445542+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":396,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.485294+0000","last_change":"2024-03-11T21:07:57.484363+0000","last_active":"2024-03-11T21:08:24.485294+0000","last_peered":"2024-03-11T21:08:24.485294+0000","last_clean":"2024-03-11T21:08:24.485294+0000","last_became_active":"2024-03-11T21:05:17.314174+0000","last_became_peered":"2024-03-11T21:05:17.314174+0000","last_unstale":"2024-03-11T21:08:24.485294+0000","last_undegraded":"2024-03-11T21:08:24.485294+0000","last_fullsized":"2024-03-11T21:08:24.485294+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:07:57.484167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:07:57.484167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:04.140139+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":286,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.486241+0000","last_change":"2024-03-11T21:08:21.529802+0000","last_active":"2024-03-11T21:08:24.486241+0000","last_peered":"2024-03-11T21:08:24.486241+0000","last_clean":"2024-03-11T21:08:24.486241+0000","last_became_active":"2024-03-11T21:00:48.455510+0000","last_became_peered":"2024-03-11T21:00:48.455510+0000","last_unstale":"2024-03-11T21:08:24.486241+0000","last_undegraded":"2024-03-11T21:08:24.486241+0000","last_fullsized":"2024-03-11T21:08:24.486241+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:08:21.529639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:08:21.529639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:40.099922+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"207'25","reported_seq":366,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.513296+0000","last_change":"2024-03-11T21:07:49.153573+0000","last_active":"2024-03-11T21:08:24.513296+0000","last_peered":"2024-03-11T21:08:24.513296+0000","last_clean":"2024-03-11T21:08:24.513296+0000","last_became_active":"2024-03-11T21:05:17.314507+0000","last_became_peered":"2024-03-11T21:05:17.314507+0000","last_unstale":"2024-03-11T21:08:24.513296+0000","last_undegraded":"2024-03-11T21:08:24.513296+0000","last_fullsized":"2024-03-11T21:08:24.513296+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'25","last_scrub_stamp":"2024-03-11T21:07:49.153339+0000","last_deep_scrub":"161'15","last_deep_scrub_stamp":"2024-03-11T20:57:15.125548+0000","last_clean_scrub_stamp":"2024-03-11T21:07:49.153339+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:00.489856+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":593,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.511801+0000","last_change":"2024-03-11T21:08:01.931240+0000","last_active":"2024-03-11T21:08:24.511801+0000","last_peered":"2024-03-11T21:08:24.511801+0000","last_clean":"2024-03-11T21:08:24.511801+0000","last_became_active":"2024-03-11T21:00:48.462105+0000","last_became_peered":"2024-03-11T21:00:48.462105+0000","last_unstale":"2024-03-11T21:08:24.511801+0000","last_undegraded":"2024-03-11T21:08:24.511801+0000","last_fullsized":"2024-03-11T21:08:24.511801+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:08:01.931041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:08:01.931041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:07.195438+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":286,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.485997+0000","last_change":"2024-03-11T21:08:17.480632+0000","last_active":"2024-03-11T21:08:24.485997+0000","last_peered":"2024-03-11T21:08:24.485997+0000","last_clean":"2024-03-11T21:08:24.485997+0000","last_became_active":"2024-03-11T21:00:48.459043+0000","last_became_peered":"2024-03-11T21:00:48.459043+0000","last_unstale":"2024-03-11T21:08:24.485997+0000","last_undegraded":"2024-03-11T21:08:24.485997+0000","last_fullsized":"2024-03-11T21:08:24.485997+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:08:17.480534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:23.137832+0000","last_clean_scrub_stamp":"2024-03-11T21:08:17.480534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:43.791504+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":593,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.485097+0000","last_change":"2024-03-11T21:07:43.593361+0000","last_active":"2024-03-11T21:08:24.485097+0000","last_peered":"2024-03-11T21:08:24.485097+0000","last_clean":"2024-03-11T21:08:24.485097+0000","last_became_active":"2024-03-11T20:50:28.082737+0000","last_became_peered":"2024-03-11T20:50:28.082737+0000","last_unstale":"2024-03-11T21:08:24.485097+0000","last_undegraded":"2024-03-11T21:08:24.485097+0000","last_fullsized":"2024-03-11T21:08:24.485097+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:07:43.593282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:28.404997+0000","last_clean_scrub_stamp":"2024-03-11T21:07:43.593282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:08:43.835258+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"207'15","reported_seq":607,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.513187+0000","last_change":"2024-03-11T21:07:58.088523+0000","last_active":"2024-03-11T21:08:24.513187+0000","last_peered":"2024-03-11T21:08:24.513187+0000","last_clean":"2024-03-11T21:08:24.513187+0000","last_became_active":"2024-03-11T21:00:27.967624+0000","last_became_peered":"2024-03-11T21:00:27.967624+0000","last_unstale":"2024-03-11T21:08:24.513187+0000","last_undegraded":"2024-03-11T21:08:24.513187+0000","last_fullsized":"2024-03-11T21:08:24.513187+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'15","last_scrub_stamp":"2024-03-11T21:07:58.088382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:07:58.088382+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:08:59.615528+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":230,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.509087+0000","last_change":"2024-03-11T21:08:05.300696+0000","last_active":"2024-03-11T21:08:24.509087+0000","last_peered":"2024-03-11T21:08:24.509087+0000","last_clean":"2024-03-11T21:08:24.509087+0000","last_became_active":"2024-03-11T21:08:05.299913+0000","last_became_peered":"2024-03-11T21:08:05.299913+0000","last_unstale":"2024-03-11T21:08:24.509087+0000","last_undegraded":"2024-03-11T21:08:24.509087+0000","last_fullsized":"2024-03-11T21:08:24.509087+0000","mapping_epoch":319,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:07:40.457227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:57:16.032302+0000","last_clean_scrub_stamp":"2024-03-11T21:07:40.457227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:08:49.058965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"112'12","reported_seq":798,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.508784+0000","last_change":"2024-03-11T21:08:11.973519+0000","last_active":"2024-03-11T21:08:24.508784+0000","last_peered":"2024-03-11T21:08:24.508784+0000","last_clean":"2024-03-11T21:08:24.508784+0000","last_became_active":"2024-03-11T20:50:32.650064+0000","last_became_peered":"2024-03-11T20:50:32.650064+0000","last_unstale":"2024-03-11T21:08:24.508784+0000","last_undegraded":"2024-03-11T21:08:24.508784+0000","last_fullsized":"2024-03-11T21:08:24.508784+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'12","last_scrub_stamp":"2024-03-11T21:08:11.973404+0000","last_deep_scrub":"112'12","last_deep_scrub_stamp":"2024-03-11T20:53:11.843951+0000","last_clean_scrub_stamp":"2024-03-11T21:08:11.973404+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:18.838738+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":281,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.512843+0000","last_change":"2024-03-11T21:06:52.863103+0000","last_active":"2024-03-11T21:08:24.512843+0000","last_peered":"2024-03-11T21:08:24.512843+0000","last_clean":"2024-03-11T21:08:24.512843+0000","last_became_active":"2024-03-11T21:06:28.756041+0000","last_became_peered":"2024-03-11T21:06:28.756041+0000","last_unstale":"2024-03-11T21:08:24.512843+0000","last_undegraded":"2024-03-11T21:08:24.512843+0000","last_fullsized":"2024-03-11T21:08:24.512843+0000","mapping_epoch":297,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:06:52.862971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:06:52.862971+0000","last_clean_scrub_stamp":"2024-03-11T21:06:52.862971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"207'19","reported_seq":340,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.511497+0000","last_change":"2024-03-11T21:08:21.328028+0000","last_active":"2024-03-11T21:08:24.511497+0000","last_peered":"2024-03-11T21:08:24.511497+0000","last_clean":"2024-03-11T21:08:24.511497+0000","last_became_active":"2024-03-11T21:06:28.758223+0000","last_became_peered":"2024-03-11T21:06:28.758223+0000","last_unstale":"2024-03-11T21:08:24.511497+0000","last_undegraded":"2024-03-11T21:08:24.511497+0000","last_fullsized":"2024-03-11T21:08:24.511497+0000","mapping_epoch":297,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'19","last_scrub_stamp":"2024-03-11T21:08:21.327572+0000","last_deep_scrub":"119'13","last_deep_scrub_stamp":"2024-03-11T20:54:15.443472+0000","last_clean_scrub_stamp":"2024-03-11T21:08:21.327572+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:42.323400+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":261,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.514024+0000","last_change":"2024-03-11T21:07:48.864558+0000","last_active":"2024-03-11T21:08:24.514024+0000","last_peered":"2024-03-11T21:08:24.514024+0000","last_clean":"2024-03-11T21:08:24.514024+0000","last_became_active":"2024-03-11T21:06:28.758634+0000","last_became_peered":"2024-03-11T21:06:28.758634+0000","last_unstale":"2024-03-11T21:08:24.514024+0000","last_undegraded":"2024-03-11T21:08:24.514024+0000","last_fullsized":"2024-03-11T21:08:24.514024+0000","mapping_epoch":297,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:07:48.864448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:07:48.864448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:05.940170+0000","scrub_duration":59,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"212'17","reported_seq":821,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.492123+0000","last_change":"2024-03-11T21:08:20.327834+0000","last_active":"2024-03-11T21:08:24.492123+0000","last_peered":"2024-03-11T21:08:24.492123+0000","last_clean":"2024-03-11T21:08:24.492123+0000","last_became_active":"2024-03-11T21:00:48.467116+0000","last_became_peered":"2024-03-11T21:00:48.467116+0000","last_unstale":"2024-03-11T21:08:24.492123+0000","last_undegraded":"2024-03-11T21:08:24.492123+0000","last_fullsized":"2024-03-11T21:08:24.492123+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"212'17","last_scrub_stamp":"2024-03-11T21:08:20.327696+0000","last_deep_scrub":"110'13","last_deep_scrub_stamp":"2024-03-11T20:53:00.434532+0000","last_clean_scrub_stamp":"2024-03-11T21:08:20.327696+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:34.643918+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":587,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.508427+0000","last_change":"2024-03-11T21:08:04.974358+0000","last_active":"2024-03-11T21:08:24.508427+0000","last_peered":"2024-03-11T21:08:24.508427+0000","last_clean":"2024-03-11T21:08:24.508427+0000","last_became_active":"2024-03-11T20:50:28.070325+0000","last_became_peered":"2024-03-11T20:50:28.070325+0000","last_unstale":"2024-03-11T21:08:24.508427+0000","last_undegraded":"2024-03-11T21:08:24.508427+0000","last_fullsized":"2024-03-11T21:08:24.508427+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:08:04.974102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:08:04.974102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:13.492851+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"207'24","reported_seq":618,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.513013+0000","last_change":"2024-03-11T21:07:46.201396+0000","last_active":"2024-03-11T21:08:24.513013+0000","last_peered":"2024-03-11T21:08:24.513013+0000","last_clean":"2024-03-11T21:08:24.513013+0000","last_became_active":"2024-03-11T21:00:27.967078+0000","last_became_peered":"2024-03-11T21:00:27.967078+0000","last_unstale":"2024-03-11T21:08:24.513013+0000","last_undegraded":"2024-03-11T21:08:24.513013+0000","last_fullsized":"2024-03-11T21:08:24.513013+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'24","last_scrub_stamp":"2024-03-11T21:07:46.201244+0000","last_deep_scrub":"116'16","last_deep_scrub_stamp":"2024-03-11T20:53:39.292118+0000","last_clean_scrub_stamp":"2024-03-11T21:07:46.201244+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:08:48.584972+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":566,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.512728+0000","last_change":"2024-03-11T21:07:45.229735+0000","last_active":"2024-03-11T21:08:24.512728+0000","last_peered":"2024-03-11T21:08:24.512728+0000","last_clean":"2024-03-11T21:08:24.512728+0000","last_became_active":"2024-03-11T21:00:27.966997+0000","last_became_peered":"2024-03-11T21:00:27.966997+0000","last_unstale":"2024-03-11T21:08:24.512728+0000","last_undegraded":"2024-03-11T21:08:24.512728+0000","last_fullsized":"2024-03-11T21:08:24.512728+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:07:45.229582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:55:58.016988+0000","last_clean_scrub_stamp":"2024-03-11T21:07:45.229582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:07.735366+0000","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":576,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.491030+0000","last_change":"2024-03-11T21:06:16.515576+0000","last_active":"2024-03-11T21:08:24.491030+0000","last_peered":"2024-03-11T21:08:24.491030+0000","last_clean":"2024-03-11T21:08:24.491030+0000","last_became_active":"2024-03-11T21:00:48.468386+0000","last_became_peered":"2024-03-11T21:00:48.468386+0000","last_unstale":"2024-03-11T21:08:24.491030+0000","last_undegraded":"2024-03-11T21:08:24.491030+0000","last_fullsized":"2024-03-11T21:08:24.491030+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:06:16.515117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:06:16.515117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"166'2","reported_seq":628,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.484803+0000","last_change":"2024-03-11T21:08:07.483308+0000","last_active":"2024-03-11T21:08:24.484803+0000","last_peered":"2024-03-11T21:08:24.484803+0000","last_clean":"2024-03-11T21:08:24.484803+0000","last_became_active":"2024-03-11T21:00:48.466071+0000","last_became_peered":"2024-03-11T21:00:48.466071+0000","last_unstale":"2024-03-11T21:08:24.484803+0000","last_undegraded":"2024-03-11T21:08:24.484803+0000","last_fullsized":"2024-03-11T21:08:24.484803+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'2","last_scrub_stamp":"2024-03-11T21:08:07.483168+0000","last_deep_scrub":"166'2","last_deep_scrub_stamp":"2024-03-11T20:57:39.868776+0000","last_clean_scrub_stamp":"2024-03-11T21:08:07.483168+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:09.957537+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1822,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"207'8","reported_seq":593,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.484740+0000","last_change":"2024-03-11T21:07:59.465898+0000","last_active":"2024-03-11T21:08:24.484740+0000","last_peered":"2024-03-11T21:08:24.484740+0000","last_clean":"2024-03-11T21:08:24.484740+0000","last_became_active":"2024-03-11T20:50:30.101572+0000","last_became_peered":"2024-03-11T20:50:30.101572+0000","last_unstale":"2024-03-11T21:08:24.484740+0000","last_undegraded":"2024-03-11T21:08:24.484740+0000","last_fullsized":"2024-03-11T21:08:24.484740+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'8","last_scrub_stamp":"2024-03-11T21:07:59.463853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:07:59.463853+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:08.969430+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":329,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.513507+0000","last_change":"2024-03-11T21:08:05.297690+0000","last_active":"2024-03-11T21:08:24.513507+0000","last_peered":"2024-03-11T21:08:24.513507+0000","last_clean":"2024-03-11T21:08:24.513507+0000","last_became_active":"2024-03-11T21:08:05.296676+0000","last_became_peered":"2024-03-11T21:08:05.296676+0000","last_unstale":"2024-03-11T21:08:24.513507+0000","last_undegraded":"2024-03-11T21:08:24.513507+0000","last_fullsized":"2024-03-11T21:08:24.513507+0000","mapping_epoch":319,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:07:45.355422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:07:45.355422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:03.741554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":287,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.514043+0000","last_change":"2024-03-11T21:08:18.233619+0000","last_active":"2024-03-11T21:08:24.514043+0000","last_peered":"2024-03-11T21:08:24.514043+0000","last_clean":"2024-03-11T21:08:24.514043+0000","last_became_active":"2024-03-11T21:06:28.758133+0000","last_became_peered":"2024-03-11T21:06:28.758133+0000","last_unstale":"2024-03-11T21:08:24.514043+0000","last_undegraded":"2024-03-11T21:08:24.514043+0000","last_fullsized":"2024-03-11T21:08:24.514043+0000","mapping_epoch":297,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:08:18.233542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:54:09.537972+0000","last_clean_scrub_stamp":"2024-03-11T21:08:18.233542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:18.981775+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":351,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.485790+0000","last_change":"2024-03-11T21:08:05.301250+0000","last_active":"2024-03-11T21:08:24.485790+0000","last_peered":"2024-03-11T21:08:24.485790+0000","last_clean":"2024-03-11T21:08:24.485790+0000","last_became_active":"2024-03-11T21:08:05.300426+0000","last_became_peered":"2024-03-11T21:08:05.300426+0000","last_unstale":"2024-03-11T21:08:24.485790+0000","last_undegraded":"2024-03-11T21:08:24.485790+0000","last_fullsized":"2024-03-11T21:08:24.485790+0000","mapping_epoch":319,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:07:41.377285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:18.682947+0000","last_clean_scrub_stamp":"2024-03-11T21:07:41.377285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:08:42.527160+0000","scrub_duration":211,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"262'24","reported_seq":618,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.512451+0000","last_change":"2024-03-11T21:08:18.124252+0000","last_active":"2024-03-11T21:08:24.512451+0000","last_peered":"2024-03-11T21:08:24.512451+0000","last_clean":"2024-03-11T21:08:24.512451+0000","last_became_active":"2024-03-11T21:00:27.970878+0000","last_became_peered":"2024-03-11T21:00:27.970878+0000","last_unstale":"2024-03-11T21:08:24.512451+0000","last_undegraded":"2024-03-11T21:08:24.512451+0000","last_fullsized":"2024-03-11T21:08:24.512451+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"262'24","last_scrub_stamp":"2024-03-11T21:08:18.124021+0000","last_deep_scrub":"262'24","last_deep_scrub_stamp":"2024-03-11T21:03:28.128719+0000","last_clean_scrub_stamp":"2024-03-11T21:08:18.124021+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:36.507595+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.7","version":"326'4388","reported_seq":4438,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.861672+0000","last_change":"2024-03-11T21:07:14.960555+0000","last_active":"2024-03-11T21:08:24.861672+0000","last_peered":"2024-03-11T21:08:24.861672+0000","last_clean":"2024-03-11T21:08:24.861672+0000","last_became_active":"2024-03-11T21:07:14.959711+0000","last_became_peered":"2024-03-11T21:07:14.959711+0000","last_unstale":"2024-03-11T21:08:24.861672+0000","last_undegraded":"2024-03-11T21:08:24.861672+0000","last_fullsized":"2024-03-11T21:08:24.861672+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":306,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:07:13.846944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:07:13.846944+0000","last_clean_scrub_stamp":"2024-03-11T21:07:13.846944+0000","objects_scrubbed":0,"log_size":4388,"log_dups_size":0,"ondisk_log_size":4388,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":287571968,"num_objects":4388,"num_object_clones":0,"num_object_copies":13164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4388,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4388,"num_write_kb":280832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":580,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.490835+0000","last_change":"2024-03-11T21:06:47.558351+0000","last_active":"2024-03-11T21:08:24.490835+0000","last_peered":"2024-03-11T21:08:24.490835+0000","last_clean":"2024-03-11T21:08:24.490835+0000","last_became_active":"2024-03-11T21:00:48.466853+0000","last_became_peered":"2024-03-11T21:00:48.466853+0000","last_unstale":"2024-03-11T21:08:24.490835+0000","last_undegraded":"2024-03-11T21:08:24.490835+0000","last_fullsized":"2024-03-11T21:08:24.490835+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:06:47.558214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:52:53.450885+0000","last_clean_scrub_stamp":"2024-03-11T21:06:47.558214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":95,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"77'2","reported_seq":86,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.511242+0000","last_change":"2024-03-11T21:08:04.913072+0000","last_active":"2024-03-11T21:08:24.511242+0000","last_peered":"2024-03-11T21:08:24.511242+0000","last_clean":"2024-03-11T21:08:24.511242+0000","last_became_active":"2024-03-11T21:06:28.759838+0000","last_became_peered":"2024-03-11T21:06:28.759838+0000","last_unstale":"2024-03-11T21:08:24.511242+0000","last_undegraded":"2024-03-11T21:08:24.511242+0000","last_fullsized":"2024-03-11T21:08:24.511242+0000","mapping_epoch":297,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'2","last_scrub_stamp":"2024-03-11T21:08:04.912948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:08:04.912948+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:25.763732+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"207'24","reported_seq":678,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.513050+0000","last_change":"2024-03-11T21:08:14.074734+0000","last_active":"2024-03-11T21:08:24.513050+0000","last_peered":"2024-03-11T21:08:24.513050+0000","last_clean":"2024-03-11T21:08:24.513050+0000","last_became_active":"2024-03-11T21:00:48.465254+0000","last_became_peered":"2024-03-11T21:00:48.465254+0000","last_unstale":"2024-03-11T21:08:24.513050+0000","last_undegraded":"2024-03-11T21:08:24.513050+0000","last_fullsized":"2024-03-11T21:08:24.513050+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'24","last_scrub_stamp":"2024-03-11T21:08:14.074642+0000","last_deep_scrub":"100'16","last_deep_scrub_stamp":"2024-03-11T20:56:08.036331+0000","last_clean_scrub_stamp":"2024-03-11T21:08:14.074642+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:38.968786+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"286'18","reported_seq":527,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.485824+0000","last_change":"2024-03-11T21:07:50.490186+0000","last_active":"2024-03-11T21:08:24.485824+0000","last_peered":"2024-03-11T21:08:24.485824+0000","last_clean":"2024-03-11T21:08:24.485824+0000","last_became_active":"2024-03-11T21:05:17.317915+0000","last_became_peered":"2024-03-11T21:05:17.317915+0000","last_unstale":"2024-03-11T21:08:24.485824+0000","last_undegraded":"2024-03-11T21:08:24.485824+0000","last_fullsized":"2024-03-11T21:08:24.485824+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"286'18","last_scrub_stamp":"2024-03-11T21:07:50.490054+0000","last_deep_scrub":"212'17","last_deep_scrub_stamp":"2024-03-11T21:03:38.258850+0000","last_clean_scrub_stamp":"2024-03-11T21:07:50.490054+0000","objects_scrubbed":1,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:17.332753+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.c","version":"326'4287","reported_seq":4337,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.816133+0000","last_change":"2024-03-11T21:07:14.952881+0000","last_active":"2024-03-11T21:08:24.816133+0000","last_peered":"2024-03-11T21:08:24.816133+0000","last_clean":"2024-03-11T21:08:24.816133+0000","last_became_active":"2024-03-11T21:07:14.952564+0000","last_became_peered":"2024-03-11T21:07:14.952564+0000","last_unstale":"2024-03-11T21:08:24.816133+0000","last_undegraded":"2024-03-11T21:08:24.816133+0000","last_fullsized":"2024-03-11T21:08:24.816133+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":306,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:07:13.846944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:07:13.846944+0000","last_clean_scrub_stamp":"2024-03-11T21:07:13.846944+0000","objects_scrubbed":0,"log_size":4287,"log_dups_size":0,"ondisk_log_size":4287,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:08:34.418948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":280952832,"num_objects":4287,"num_object_clones":0,"num_object_copies":12861,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4287,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4287,"num_write_kb":274368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":270,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.511880+0000","last_change":"2024-03-11T21:08:20.317891+0000","last_active":"2024-03-11T21:08:24.511880+0000","last_peered":"2024-03-11T21:08:24.511880+0000","last_clean":"2024-03-11T21:08:24.511880+0000","last_became_active":"2024-03-11T21:06:28.755689+0000","last_became_peered":"2024-03-11T21:06:28.755689+0000","last_unstale":"2024-03-11T21:08:24.511880+0000","last_undegraded":"2024-03-11T21:08:24.511880+0000","last_fullsized":"2024-03-11T21:08:24.511880+0000","mapping_epoch":297,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:08:20.317781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:57:21.568400+0000","last_clean_scrub_stamp":"2024-03-11T21:08:20.317781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:32.488402+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":286,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.486390+0000","last_change":"2024-03-11T21:08:00.626028+0000","last_active":"2024-03-11T21:08:24.486390+0000","last_peered":"2024-03-11T21:08:24.486390+0000","last_clean":"2024-03-11T21:08:24.486390+0000","last_became_active":"2024-03-11T21:00:48.455733+0000","last_became_peered":"2024-03-11T21:00:48.455733+0000","last_unstale":"2024-03-11T21:08:24.486390+0000","last_undegraded":"2024-03-11T21:08:24.486390+0000","last_fullsized":"2024-03-11T21:08:24.486390+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:08:00.625904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:20.096874+0000","last_clean_scrub_stamp":"2024-03-11T21:08:00.625904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:07.475499+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"207'28","reported_seq":832,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.508606+0000","last_change":"2024-03-11T21:07:45.687177+0000","last_active":"2024-03-11T21:08:24.508606+0000","last_peered":"2024-03-11T21:08:24.508606+0000","last_clean":"2024-03-11T21:08:24.508606+0000","last_became_active":"2024-03-11T20:50:30.094977+0000","last_became_peered":"2024-03-11T20:50:30.094977+0000","last_unstale":"2024-03-11T21:08:24.508606+0000","last_undegraded":"2024-03-11T21:08:24.508606+0000","last_fullsized":"2024-03-11T21:08:24.508606+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'28","last_scrub_stamp":"2024-03-11T21:07:45.686955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:07:45.686955+0000","objects_scrubbed":8,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:08:51.870122+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":286,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.485871+0000","last_change":"2024-03-11T21:08:07.565477+0000","last_active":"2024-03-11T21:08:24.485871+0000","last_peered":"2024-03-11T21:08:24.485871+0000","last_clean":"2024-03-11T21:08:24.485871+0000","last_became_active":"2024-03-11T21:00:48.457901+0000","last_became_peered":"2024-03-11T21:00:48.457901+0000","last_unstale":"2024-03-11T21:08:24.485871+0000","last_undegraded":"2024-03-11T21:08:24.485871+0000","last_fullsized":"2024-03-11T21:08:24.485871+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:08:07.565360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:52:10.160202+0000","last_clean_scrub_stamp":"2024-03-11T21:08:07.565360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:10.499003+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.a","version":"326'4336","reported_seq":4386,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.852911+0000","last_change":"2024-03-11T21:07:14.883958+0000","last_active":"2024-03-11T21:08:24.852911+0000","last_peered":"2024-03-11T21:08:24.852911+0000","last_clean":"2024-03-11T21:08:24.852911+0000","last_became_active":"2024-03-11T21:07:14.883606+0000","last_became_peered":"2024-03-11T21:07:14.883606+0000","last_unstale":"2024-03-11T21:08:24.852911+0000","last_undegraded":"2024-03-11T21:08:24.852911+0000","last_fullsized":"2024-03-11T21:08:24.852911+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":306,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:07:13.846944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:07:13.846944+0000","last_clean_scrub_stamp":"2024-03-11T21:07:13.846944+0000","objects_scrubbed":0,"log_size":4336,"log_dups_size":0,"ondisk_log_size":4336,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:08:32.077438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":284164096,"num_objects":4336,"num_object_clones":0,"num_object_copies":13008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4336,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4336,"num_write_kb":277504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":561,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.490552+0000","last_change":"2024-03-11T21:06:25.518117+0000","last_active":"2024-03-11T21:08:24.490552+0000","last_peered":"2024-03-11T21:08:24.490552+0000","last_clean":"2024-03-11T21:08:24.490552+0000","last_became_active":"2024-03-11T21:00:27.959408+0000","last_became_peered":"2024-03-11T21:00:27.959408+0000","last_unstale":"2024-03-11T21:08:24.490552+0000","last_undegraded":"2024-03-11T21:08:24.490552+0000","last_fullsized":"2024-03-11T21:08:24.490552+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:06:25.518004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:06:25.518004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":593,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.508609+0000","last_change":"2024-03-11T21:06:42.906241+0000","last_active":"2024-03-11T21:08:24.508609+0000","last_peered":"2024-03-11T21:08:24.508609+0000","last_clean":"2024-03-11T21:08:24.508609+0000","last_became_active":"2024-03-11T21:00:48.471212+0000","last_became_peered":"2024-03-11T21:00:48.471212+0000","last_unstale":"2024-03-11T21:08:24.508609+0000","last_undegraded":"2024-03-11T21:08:24.508609+0000","last_fullsized":"2024-03-11T21:08:24.508609+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:06:42.906028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:33.823704+0000","last_clean_scrub_stamp":"2024-03-11T21:06:42.906028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"207'37","reported_seq":409,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.513014+0000","last_change":"2024-03-11T21:07:41.769444+0000","last_active":"2024-03-11T21:08:24.513014+0000","last_peered":"2024-03-11T21:08:24.513014+0000","last_clean":"2024-03-11T21:08:24.513014+0000","last_became_active":"2024-03-11T21:06:28.756995+0000","last_became_peered":"2024-03-11T21:06:28.756995+0000","last_unstale":"2024-03-11T21:08:24.513014+0000","last_undegraded":"2024-03-11T21:08:24.513014+0000","last_fullsized":"2024-03-11T21:08:24.513014+0000","mapping_epoch":297,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'37","last_scrub_stamp":"2024-03-11T21:07:41.769305+0000","last_deep_scrub":"182'27","last_deep_scrub_stamp":"2024-03-11T20:58:52.058851+0000","last_clean_scrub_stamp":"2024-03-11T21:07:41.769305+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:08:55.330725+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":286,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.486016+0000","last_change":"2024-03-11T21:07:57.635813+0000","last_active":"2024-03-11T21:08:24.486016+0000","last_peered":"2024-03-11T21:08:24.486016+0000","last_clean":"2024-03-11T21:08:24.486016+0000","last_became_active":"2024-03-11T21:00:48.458008+0000","last_became_peered":"2024-03-11T21:00:48.458008+0000","last_unstale":"2024-03-11T21:08:24.486016+0000","last_undegraded":"2024-03-11T21:08:24.486016+0000","last_fullsized":"2024-03-11T21:08:24.486016+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:07:57.635663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:33.319221+0000","last_clean_scrub_stamp":"2024-03-11T21:07:57.635663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:24.155564+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.9","version":"326'4419","reported_seq":4469,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:25.461158+0000","last_change":"2024-03-11T21:07:14.877587+0000","last_active":"2024-03-11T21:08:25.461158+0000","last_peered":"2024-03-11T21:08:25.461158+0000","last_clean":"2024-03-11T21:08:25.461158+0000","last_became_active":"2024-03-11T21:07:14.877262+0000","last_became_peered":"2024-03-11T21:07:14.877262+0000","last_unstale":"2024-03-11T21:08:25.461158+0000","last_undegraded":"2024-03-11T21:08:25.461158+0000","last_fullsized":"2024-03-11T21:08:25.461158+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":306,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:07:13.846944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:07:13.846944+0000","last_clean_scrub_stamp":"2024-03-11T21:07:13.846944+0000","objects_scrubbed":0,"log_size":4419,"log_dups_size":0,"ondisk_log_size":4419,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:08:34.531701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":289603584,"num_objects":4419,"num_object_clones":0,"num_object_copies":13257,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4419,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4419,"num_write_kb":282816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":574,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.512746+0000","last_change":"2024-03-11T21:08:05.120894+0000","last_active":"2024-03-11T21:08:24.512746+0000","last_peered":"2024-03-11T21:08:24.512746+0000","last_clean":"2024-03-11T21:08:24.512746+0000","last_became_active":"2024-03-11T20:50:34.321038+0000","last_became_peered":"2024-03-11T20:50:34.321038+0000","last_unstale":"2024-03-11T21:08:24.512746+0000","last_undegraded":"2024-03-11T21:08:24.512746+0000","last_fullsized":"2024-03-11T21:08:24.512746+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:08:05.120651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:32.951104+0000","last_clean_scrub_stamp":"2024-03-11T21:08:05.120651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:10.572863+0000","scrub_duration":59,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"207'38","reported_seq":318,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.485754+0000","last_change":"2024-03-11T21:08:23.471517+0000","last_active":"2024-03-11T21:08:24.485754+0000","last_peered":"2024-03-11T21:08:24.485754+0000","last_clean":"2024-03-11T21:08:24.485754+0000","last_became_active":"2024-03-11T21:00:48.456178+0000","last_became_peered":"2024-03-11T21:00:48.456178+0000","last_unstale":"2024-03-11T21:08:24.485754+0000","last_undegraded":"2024-03-11T21:08:24.485754+0000","last_fullsized":"2024-03-11T21:08:24.485754+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'38","last_scrub_stamp":"2024-03-11T21:08:23.471361+0000","last_deep_scrub":"109'26","last_deep_scrub_stamp":"2024-03-11T20:52:49.216378+0000","last_clean_scrub_stamp":"2024-03-11T21:08:23.471361+0000","objects_scrubbed":8,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:42.364904+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":25,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.490353+0000","last_change":"2024-03-11T21:08:05.308997+0000","last_active":"2024-03-11T21:08:24.490353+0000","last_peered":"2024-03-11T21:08:24.490353+0000","last_clean":"2024-03-11T21:08:24.490353+0000","last_became_active":"2024-03-11T21:08:05.308608+0000","last_became_peered":"2024-03-11T21:08:05.308608+0000","last_unstale":"2024-03-11T21:08:24.490353+0000","last_undegraded":"2024-03-11T21:08:24.490353+0000","last_fullsized":"2024-03-11T21:08:24.490353+0000","mapping_epoch":319,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:07:43.347185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:07:43.347185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:08:56.936436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":278,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.486599+0000","last_change":"2024-03-11T21:07:39.371711+0000","last_active":"2024-03-11T21:08:24.486599+0000","last_peered":"2024-03-11T21:08:24.486599+0000","last_clean":"2024-03-11T21:08:24.486599+0000","last_became_active":"2024-03-11T21:00:48.473346+0000","last_became_peered":"2024-03-11T21:00:48.473346+0000","last_unstale":"2024-03-11T21:08:24.486599+0000","last_undegraded":"2024-03-11T21:08:24.486599+0000","last_fullsized":"2024-03-11T21:08:24.486599+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:07:39.371633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:07:39.371633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:08:41.104048+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"326'4266","reported_seq":4316,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.843064+0000","last_change":"2024-03-11T21:07:14.893219+0000","last_active":"2024-03-11T21:08:24.843064+0000","last_peered":"2024-03-11T21:08:24.843064+0000","last_clean":"2024-03-11T21:08:24.843064+0000","last_became_active":"2024-03-11T21:07:14.892853+0000","last_became_peered":"2024-03-11T21:07:14.892853+0000","last_unstale":"2024-03-11T21:08:24.843064+0000","last_undegraded":"2024-03-11T21:08:24.843064+0000","last_fullsized":"2024-03-11T21:08:24.843064+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":306,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:07:13.846944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:07:13.846944+0000","last_clean_scrub_stamp":"2024-03-11T21:07:13.846944+0000","objects_scrubbed":0,"log_size":4266,"log_dups_size":0,"ondisk_log_size":4266,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":279576576,"num_objects":4266,"num_object_clones":0,"num_object_copies":12798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4266,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4266,"num_write_kb":273024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":569,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.509043+0000","last_change":"2024-03-11T21:07:51.860019+0000","last_active":"2024-03-11T21:08:24.509043+0000","last_peered":"2024-03-11T21:08:24.509043+0000","last_clean":"2024-03-11T21:08:24.509043+0000","last_became_active":"2024-03-11T21:00:27.962668+0000","last_became_peered":"2024-03-11T21:00:27.962668+0000","last_unstale":"2024-03-11T21:08:24.509043+0000","last_undegraded":"2024-03-11T21:08:24.509043+0000","last_fullsized":"2024-03-11T21:08:24.509043+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:07:51.859887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:07:51.859887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:00.536649+0000","scrub_duration":95,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"284'25","reported_seq":603,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.490324+0000","last_change":"2024-03-11T21:08:11.417836+0000","last_active":"2024-03-11T21:08:24.490324+0000","last_peered":"2024-03-11T21:08:24.490324+0000","last_clean":"2024-03-11T21:08:24.490324+0000","last_became_active":"2024-03-11T20:50:30.101747+0000","last_became_peered":"2024-03-11T20:50:30.101747+0000","last_unstale":"2024-03-11T21:08:24.490324+0000","last_undegraded":"2024-03-11T21:08:24.490324+0000","last_fullsized":"2024-03-11T21:08:24.490324+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"284'25","last_scrub_stamp":"2024-03-11T21:08:11.417688+0000","last_deep_scrub":"284'25","last_deep_scrub_stamp":"2024-03-11T21:04:53.217188+0000","last_clean_scrub_stamp":"2024-03-11T21:08:11.417688+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:21.214862+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":589,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.508222+0000","last_change":"2024-03-11T21:08:09.965355+0000","last_active":"2024-03-11T21:08:24.508222+0000","last_peered":"2024-03-11T21:08:24.508222+0000","last_clean":"2024-03-11T21:08:24.508222+0000","last_became_active":"2024-03-11T20:50:28.085503+0000","last_became_peered":"2024-03-11T20:50:28.085503+0000","last_unstale":"2024-03-11T21:08:24.508222+0000","last_undegraded":"2024-03-11T21:08:24.508222+0000","last_fullsized":"2024-03-11T21:08:24.508222+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:08:09.965148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:56:03.479180+0000","last_clean_scrub_stamp":"2024-03-11T21:08:09.965148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:36.540699+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"184'130","reported_seq":855,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.512529+0000","last_change":"2024-03-11T21:07:53.188030+0000","last_active":"2024-03-11T21:08:24.512529+0000","last_peered":"2024-03-11T21:08:24.512529+0000","last_clean":"2024-03-11T21:08:24.512529+0000","last_became_active":"2024-03-11T20:49:52.584543+0000","last_became_peered":"2024-03-11T20:49:52.584543+0000","last_unstale":"2024-03-11T21:08:24.512529+0000","last_undegraded":"2024-03-11T21:08:24.512529+0000","last_fullsized":"2024-03-11T21:08:24.512529+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"184'130","last_scrub_stamp":"2024-03-11T21:07:53.187858+0000","last_deep_scrub":"184'130","last_deep_scrub_stamp":"2024-03-11T20:58:56.415267+0000","last_clean_scrub_stamp":"2024-03-11T21:07:53.187858+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:19.647152+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":595,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.485509+0000","last_change":"2024-03-11T21:08:04.509143+0000","last_active":"2024-03-11T21:08:24.485509+0000","last_peered":"2024-03-11T21:08:24.485509+0000","last_clean":"2024-03-11T21:08:24.485509+0000","last_became_active":"2024-03-11T21:00:27.959467+0000","last_became_peered":"2024-03-11T21:00:27.959467+0000","last_unstale":"2024-03-11T21:08:24.485509+0000","last_undegraded":"2024-03-11T21:08:24.485509+0000","last_fullsized":"2024-03-11T21:08:24.485509+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:08:04.509037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:54:22.448812+0000","last_clean_scrub_stamp":"2024-03-11T21:08:04.509037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:33.092321+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.8","version":"326'4386","reported_seq":4436,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.906653+0000","last_change":"2024-03-11T21:07:14.892247+0000","last_active":"2024-03-11T21:08:24.906653+0000","last_peered":"2024-03-11T21:08:24.906653+0000","last_clean":"2024-03-11T21:08:24.906653+0000","last_became_active":"2024-03-11T21:07:14.891196+0000","last_became_peered":"2024-03-11T21:07:14.891196+0000","last_unstale":"2024-03-11T21:08:24.906653+0000","last_undegraded":"2024-03-11T21:08:24.906653+0000","last_fullsized":"2024-03-11T21:08:24.906653+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":306,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:07:13.846944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:07:13.846944+0000","last_clean_scrub_stamp":"2024-03-11T21:07:13.846944+0000","objects_scrubbed":0,"log_size":4386,"log_dups_size":0,"ondisk_log_size":4386,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:08:31.145275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":287440896,"num_objects":4386,"num_object_clones":0,"num_object_copies":13158,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4386,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4386,"num_write_kb":280704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":278,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.486248+0000","last_change":"2024-03-11T21:06:41.763723+0000","last_active":"2024-03-11T21:08:24.486248+0000","last_peered":"2024-03-11T21:08:24.486248+0000","last_clean":"2024-03-11T21:08:24.486248+0000","last_became_active":"2024-03-11T21:00:48.464596+0000","last_became_peered":"2024-03-11T21:00:48.464596+0000","last_unstale":"2024-03-11T21:08:24.486248+0000","last_undegraded":"2024-03-11T21:08:24.486248+0000","last_fullsized":"2024-03-11T21:08:24.486248+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:06:41.763442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:06:41.763442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"207'35","reported_seq":266,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.511144+0000","last_change":"2024-03-11T21:08:24.324884+0000","last_active":"2024-03-11T21:08:24.511144+0000","last_peered":"2024-03-11T21:08:24.511144+0000","last_clean":"2024-03-11T21:08:24.511144+0000","last_became_active":"2024-03-11T21:06:28.761186+0000","last_became_peered":"2024-03-11T21:06:28.761186+0000","last_unstale":"2024-03-11T21:08:24.511144+0000","last_undegraded":"2024-03-11T21:08:24.511144+0000","last_fullsized":"2024-03-11T21:08:24.511144+0000","mapping_epoch":297,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'35","last_scrub_stamp":"2024-03-11T21:08:24.324628+0000","last_deep_scrub":"100'21","last_deep_scrub_stamp":"2024-03-11T20:51:31.835302+0000","last_clean_scrub_stamp":"2024-03-11T21:08:24.324628+0000","objects_scrubbed":7,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:25.322331+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":317,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.486086+0000","last_change":"2024-03-11T21:08:05.301922+0000","last_active":"2024-03-11T21:08:24.486086+0000","last_peered":"2024-03-11T21:08:24.486086+0000","last_clean":"2024-03-11T21:08:24.486086+0000","last_became_active":"2024-03-11T21:08:05.301024+0000","last_became_peered":"2024-03-11T21:08:05.301024+0000","last_unstale":"2024-03-11T21:08:24.486086+0000","last_undegraded":"2024-03-11T21:08:24.486086+0000","last_fullsized":"2024-03-11T21:08:24.486086+0000","mapping_epoch":319,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:07:54.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:07:54.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:13.882295+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":598,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.513889+0000","last_change":"2024-03-11T21:08:16.214623+0000","last_active":"2024-03-11T21:08:24.513889+0000","last_peered":"2024-03-11T21:08:24.513889+0000","last_clean":"2024-03-11T21:08:24.513889+0000","last_became_active":"2024-03-11T21:00:48.463247+0000","last_became_peered":"2024-03-11T21:00:48.463247+0000","last_unstale":"2024-03-11T21:08:24.513889+0000","last_undegraded":"2024-03-11T21:08:24.513889+0000","last_fullsized":"2024-03-11T21:08:24.513889+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:08:16.214363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:26.839782+0000","last_clean_scrub_stamp":"2024-03-11T21:08:16.214363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:35.817015+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.d","version":"326'4389","reported_seq":4439,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.828748+0000","last_change":"2024-03-11T21:07:14.951803+0000","last_active":"2024-03-11T21:08:24.828748+0000","last_peered":"2024-03-11T21:08:24.828748+0000","last_clean":"2024-03-11T21:08:24.828748+0000","last_became_active":"2024-03-11T21:07:14.951361+0000","last_became_peered":"2024-03-11T21:07:14.951361+0000","last_unstale":"2024-03-11T21:08:24.828748+0000","last_undegraded":"2024-03-11T21:08:24.828748+0000","last_fullsized":"2024-03-11T21:08:24.828748+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":306,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:07:13.846944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:07:13.846944+0000","last_clean_scrub_stamp":"2024-03-11T21:07:13.846944+0000","objects_scrubbed":0,"log_size":4389,"log_dups_size":0,"ondisk_log_size":4389,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":287637504,"num_objects":4389,"num_object_clones":0,"num_object_copies":13167,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4389,"num_write_kb":280896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":286,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.485871+0000","last_change":"2024-03-11T21:07:45.417128+0000","last_active":"2024-03-11T21:08:24.485871+0000","last_peered":"2024-03-11T21:08:24.485871+0000","last_clean":"2024-03-11T21:08:24.485871+0000","last_became_active":"2024-03-11T21:00:48.469688+0000","last_became_peered":"2024-03-11T21:00:48.469688+0000","last_unstale":"2024-03-11T21:08:24.485871+0000","last_undegraded":"2024-03-11T21:08:24.485871+0000","last_fullsized":"2024-03-11T21:08:24.485871+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:07:45.416778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:07:45.416778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:05.943677+0000","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"145'3","reported_seq":588,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.485235+0000","last_change":"2024-03-11T21:07:45.552642+0000","last_active":"2024-03-11T21:08:24.485235+0000","last_peered":"2024-03-11T21:08:24.485235+0000","last_clean":"2024-03-11T21:08:24.485235+0000","last_became_active":"2024-03-11T20:50:32.195452+0000","last_became_peered":"2024-03-11T20:50:32.195452+0000","last_unstale":"2024-03-11T21:08:24.485235+0000","last_undegraded":"2024-03-11T21:08:24.485235+0000","last_fullsized":"2024-03-11T21:08:24.485235+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'3","last_scrub_stamp":"2024-03-11T21:07:45.552526+0000","last_deep_scrub":"145'3","last_deep_scrub_stamp":"2024-03-11T20:55:52.848043+0000","last_clean_scrub_stamp":"2024-03-11T21:07:45.552526+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:08:59.099650+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0.0024049620000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":587,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.512851+0000","last_change":"2024-03-11T21:07:59.257140+0000","last_active":"2024-03-11T21:08:24.512851+0000","last_peered":"2024-03-11T21:08:24.512851+0000","last_clean":"2024-03-11T21:08:24.512851+0000","last_became_active":"2024-03-11T21:00:27.966749+0000","last_became_peered":"2024-03-11T21:00:27.966749+0000","last_unstale":"2024-03-11T21:08:24.512851+0000","last_undegraded":"2024-03-11T21:08:24.512851+0000","last_fullsized":"2024-03-11T21:08:24.512851+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:07:59.256994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:07:59.256994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:16.309231+0000","scrub_duration":163,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"212'46","reported_seq":407,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.486367+0000","last_change":"2024-03-11T21:07:47.448694+0000","last_active":"2024-03-11T21:08:24.486367+0000","last_peered":"2024-03-11T21:08:24.486367+0000","last_clean":"2024-03-11T21:08:24.486367+0000","last_became_active":"2024-03-11T21:00:48.473227+0000","last_became_peered":"2024-03-11T21:00:48.473227+0000","last_unstale":"2024-03-11T21:08:24.486367+0000","last_undegraded":"2024-03-11T21:08:24.486367+0000","last_fullsized":"2024-03-11T21:08:24.486367+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"212'46","last_scrub_stamp":"2024-03-11T21:07:47.448528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:07:47.448528+0000","objects_scrubbed":10,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:08:50.895708+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":69,"num_read_kb":54,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"298'13","reported_seq":380,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.513121+0000","last_change":"2024-03-11T21:08:08.181167+0000","last_active":"2024-03-11T21:08:24.513121+0000","last_peered":"2024-03-11T21:08:24.513121+0000","last_clean":"2024-03-11T21:08:24.513121+0000","last_became_active":"2024-03-11T21:06:28.762397+0000","last_became_peered":"2024-03-11T21:06:28.762397+0000","last_unstale":"2024-03-11T21:08:24.513121+0000","last_undegraded":"2024-03-11T21:08:24.513121+0000","last_fullsized":"2024-03-11T21:08:24.513121+0000","mapping_epoch":297,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"298'13","last_scrub_stamp":"2024-03-11T21:08:08.181013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:08:08.181013+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:25.674580+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.e","version":"326'4334","reported_seq":4384,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.844143+0000","last_change":"2024-03-11T21:07:14.954025+0000","last_active":"2024-03-11T21:08:24.844143+0000","last_peered":"2024-03-11T21:08:24.844143+0000","last_clean":"2024-03-11T21:08:24.844143+0000","last_became_active":"2024-03-11T21:07:14.953126+0000","last_became_peered":"2024-03-11T21:07:14.953126+0000","last_unstale":"2024-03-11T21:08:24.844143+0000","last_undegraded":"2024-03-11T21:08:24.844143+0000","last_fullsized":"2024-03-11T21:08:24.844143+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":306,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:07:13.846944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:07:13.846944+0000","last_clean_scrub_stamp":"2024-03-11T21:07:13.846944+0000","objects_scrubbed":0,"log_size":4334,"log_dups_size":0,"ondisk_log_size":4334,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:08:32.880001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":284033024,"num_objects":4334,"num_object_clones":0,"num_object_copies":13002,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4334,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4334,"num_write_kb":277376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":306,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.490506+0000","last_change":"2024-03-11T21:06:31.535147+0000","last_active":"2024-03-11T21:08:24.490506+0000","last_peered":"2024-03-11T21:08:24.490506+0000","last_clean":"2024-03-11T21:08:24.490506+0000","last_became_active":"2024-03-11T21:05:17.441521+0000","last_became_peered":"2024-03-11T21:05:17.441521+0000","last_unstale":"2024-03-11T21:08:24.490506+0000","last_undegraded":"2024-03-11T21:08:24.490506+0000","last_fullsized":"2024-03-11T21:08:24.490506+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:06:31.535021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:03:43.345257+0000","last_clean_scrub_stamp":"2024-03-11T21:06:31.535021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":327,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.490463+0000","last_change":"2024-03-11T21:06:44.388965+0000","last_active":"2024-03-11T21:08:24.490463+0000","last_peered":"2024-03-11T21:08:24.490463+0000","last_clean":"2024-03-11T21:08:24.490463+0000","last_became_active":"2024-03-11T21:05:17.315844+0000","last_became_peered":"2024-03-11T21:05:17.315844+0000","last_unstale":"2024-03-11T21:08:24.490463+0000","last_undegraded":"2024-03-11T21:08:24.490463+0000","last_fullsized":"2024-03-11T21:08:24.490463+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:06:44.388832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:22.617999+0000","last_clean_scrub_stamp":"2024-03-11T21:06:44.388832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"284'2","reported_seq":291,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.486350+0000","last_change":"2024-03-11T21:08:22.484946+0000","last_active":"2024-03-11T21:08:24.486350+0000","last_peered":"2024-03-11T21:08:24.486350+0000","last_clean":"2024-03-11T21:08:24.486350+0000","last_became_active":"2024-03-11T21:00:48.468682+0000","last_became_peered":"2024-03-11T21:00:48.468682+0000","last_unstale":"2024-03-11T21:08:24.486350+0000","last_undegraded":"2024-03-11T21:08:24.486350+0000","last_fullsized":"2024-03-11T21:08:24.486350+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"284'2","last_scrub_stamp":"2024-03-11T21:08:22.484799+0000","last_deep_scrub":"284'2","last_deep_scrub_stamp":"2024-03-11T21:06:47.645682+0000","last_clean_scrub_stamp":"2024-03-11T21:08:22.484799+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:33.894145+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"207'17","reported_seq":593,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.490889+0000","last_change":"2024-03-11T21:06:41.476100+0000","last_active":"2024-03-11T21:08:24.490889+0000","last_peered":"2024-03-11T21:08:24.490889+0000","last_clean":"2024-03-11T21:08:24.490889+0000","last_became_active":"2024-03-11T20:50:30.097669+0000","last_became_peered":"2024-03-11T20:50:30.097669+0000","last_unstale":"2024-03-11T21:08:24.490889+0000","last_undegraded":"2024-03-11T21:08:24.490889+0000","last_fullsized":"2024-03-11T21:08:24.490889+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'17","last_scrub_stamp":"2024-03-11T21:06:41.475870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:06:41.475870+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"84'11","reported_seq":798,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.490301+0000","last_change":"2024-03-11T21:08:04.428744+0000","last_active":"2024-03-11T21:08:24.490301+0000","last_peered":"2024-03-11T21:08:24.490301+0000","last_clean":"2024-03-11T21:08:24.490301+0000","last_became_active":"2024-03-11T20:50:32.647041+0000","last_became_peered":"2024-03-11T20:50:32.647041+0000","last_unstale":"2024-03-11T21:08:24.490301+0000","last_undegraded":"2024-03-11T21:08:24.490301+0000","last_fullsized":"2024-03-11T21:08:24.490301+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:08:04.428606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:08:04.428606+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:24.794391+0000","scrub_duration":83,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.f","version":"326'4476","reported_seq":4525,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.862103+0000","last_change":"2024-03-11T21:07:14.960488+0000","last_active":"2024-03-11T21:08:24.862103+0000","last_peered":"2024-03-11T21:08:24.862103+0000","last_clean":"2024-03-11T21:08:24.862103+0000","last_became_active":"2024-03-11T21:07:14.959566+0000","last_became_peered":"2024-03-11T21:07:14.959566+0000","last_unstale":"2024-03-11T21:08:24.862103+0000","last_undegraded":"2024-03-11T21:08:24.862103+0000","last_fullsized":"2024-03-11T21:08:24.862103+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":306,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:07:13.846944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:07:13.846944+0000","last_clean_scrub_stamp":"2024-03-11T21:07:13.846944+0000","objects_scrubbed":0,"log_size":4476,"log_dups_size":0,"ondisk_log_size":4476,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":293339136,"num_objects":4476,"num_object_clones":0,"num_object_copies":13428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4476,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4476,"num_write_kb":286464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":582,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.512619+0000","last_change":"2024-03-11T21:08:07.202258+0000","last_active":"2024-03-11T21:08:24.512619+0000","last_peered":"2024-03-11T21:08:24.512619+0000","last_clean":"2024-03-11T21:08:24.512619+0000","last_became_active":"2024-03-11T20:50:34.326947+0000","last_became_peered":"2024-03-11T20:50:34.326947+0000","last_unstale":"2024-03-11T21:08:24.512619+0000","last_undegraded":"2024-03-11T21:08:24.512619+0000","last_fullsized":"2024-03-11T21:08:24.512619+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:08:07.202131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:06:43.271453+0000","last_clean_scrub_stamp":"2024-03-11T21:08:07.202131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:22.106352+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"120'6","reported_seq":1552,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.490389+0000","last_change":"2024-03-11T21:08:19.340719+0000","last_active":"2024-03-11T21:08:24.490389+0000","last_peered":"2024-03-11T21:08:24.490389+0000","last_clean":"2024-03-11T21:08:24.490389+0000","last_became_active":"2024-03-11T21:00:48.468997+0000","last_became_peered":"2024-03-11T21:00:48.468997+0000","last_unstale":"2024-03-11T21:08:24.490389+0000","last_undegraded":"2024-03-11T21:08:24.490389+0000","last_fullsized":"2024-03-11T21:08:24.490389+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"120'6","last_scrub_stamp":"2024-03-11T21:08:19.340643+0000","last_deep_scrub":"120'6","last_deep_scrub_stamp":"2024-03-11T20:54:26.468057+0000","last_clean_scrub_stamp":"2024-03-11T21:08:19.340643+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:34.137288+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0035533370000000002,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":939,"num_read_kb":934,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":594,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.485422+0000","last_change":"2024-03-11T21:07:40.565165+0000","last_active":"2024-03-11T21:08:24.485422+0000","last_peered":"2024-03-11T21:08:24.485422+0000","last_clean":"2024-03-11T21:08:24.485422+0000","last_became_active":"2024-03-11T20:50:28.071707+0000","last_became_peered":"2024-03-11T20:50:28.071707+0000","last_unstale":"2024-03-11T21:08:24.485422+0000","last_undegraded":"2024-03-11T21:08:24.485422+0000","last_fullsized":"2024-03-11T21:08:24.485422+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:07:40.565077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:23.417283+0000","last_clean_scrub_stamp":"2024-03-11T21:07:40.565077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:08:51.841430+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"207'24","reported_seq":686,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.513512+0000","last_change":"2024-03-11T21:07:52.838119+0000","last_active":"2024-03-11T21:08:24.513512+0000","last_peered":"2024-03-11T21:08:24.513512+0000","last_clean":"2024-03-11T21:08:24.513512+0000","last_became_active":"2024-03-11T21:00:48.461972+0000","last_became_peered":"2024-03-11T21:00:48.461972+0000","last_unstale":"2024-03-11T21:08:24.513512+0000","last_undegraded":"2024-03-11T21:08:24.513512+0000","last_fullsized":"2024-03-11T21:08:24.513512+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'24","last_scrub_stamp":"2024-03-11T21:07:52.837968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:07:52.837968+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:17.874543+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":356,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.507210+0000","last_change":"2024-03-11T21:08:15.915483+0000","last_active":"2024-03-11T21:08:24.507210+0000","last_peered":"2024-03-11T21:08:24.507210+0000","last_clean":"2024-03-11T21:08:24.507210+0000","last_became_active":"2024-03-11T21:05:17.315807+0000","last_became_peered":"2024-03-11T21:05:17.315807+0000","last_unstale":"2024-03-11T21:08:24.507210+0000","last_undegraded":"2024-03-11T21:08:24.507210+0000","last_fullsized":"2024-03-11T21:08:24.507210+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:08:15.915374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:08:15.915374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:35.617260+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.0","version":"326'4398","reported_seq":4447,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.844032+0000","last_change":"2024-03-11T21:07:14.953886+0000","last_active":"2024-03-11T21:08:24.844032+0000","last_peered":"2024-03-11T21:08:24.844032+0000","last_clean":"2024-03-11T21:08:24.844032+0000","last_became_active":"2024-03-11T21:07:14.952617+0000","last_became_peered":"2024-03-11T21:07:14.952617+0000","last_unstale":"2024-03-11T21:08:24.844032+0000","last_undegraded":"2024-03-11T21:08:24.844032+0000","last_fullsized":"2024-03-11T21:08:24.844032+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":306,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:07:13.846944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:07:13.846944+0000","last_clean_scrub_stamp":"2024-03-11T21:07:13.846944+0000","objects_scrubbed":0,"log_size":4398,"log_dups_size":0,"ondisk_log_size":4398,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:08:26.395702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":288227328,"num_objects":4398,"num_object_clones":0,"num_object_copies":13194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4398,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4398,"num_write_kb":281472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":551,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.491845+0000","last_change":"2024-03-11T21:06:30.558401+0000","last_active":"2024-03-11T21:08:24.491845+0000","last_peered":"2024-03-11T21:08:24.491845+0000","last_clean":"2024-03-11T21:08:24.491845+0000","last_became_active":"2024-03-11T20:50:34.335370+0000","last_became_peered":"2024-03-11T20:50:34.335370+0000","last_unstale":"2024-03-11T21:08:24.491845+0000","last_undegraded":"2024-03-11T21:08:24.491845+0000","last_fullsized":"2024-03-11T21:08:24.491845+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:06:30.558258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:06:30.558258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"286'18","reported_seq":211,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.485492+0000","last_change":"2024-03-11T21:07:55.699039+0000","last_active":"2024-03-11T21:08:24.485492+0000","last_peered":"2024-03-11T21:08:24.485492+0000","last_clean":"2024-03-11T21:08:24.485492+0000","last_became_active":"2024-03-11T21:05:17.449213+0000","last_became_peered":"2024-03-11T21:05:17.449213+0000","last_unstale":"2024-03-11T21:08:24.485492+0000","last_undegraded":"2024-03-11T21:08:24.485492+0000","last_fullsized":"2024-03-11T21:08:24.485492+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"286'18","last_scrub_stamp":"2024-03-11T21:07:55.698212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:07:55.698212+0000","objects_scrubbed":1,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:13.452670+0000","scrub_duration":195,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":599,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.512641+0000","last_change":"2024-03-11T21:08:16.137000+0000","last_active":"2024-03-11T21:08:24.512641+0000","last_peered":"2024-03-11T21:08:24.512641+0000","last_clean":"2024-03-11T21:08:24.512641+0000","last_became_active":"2024-03-11T21:00:48.466102+0000","last_became_peered":"2024-03-11T21:00:48.466102+0000","last_unstale":"2024-03-11T21:08:24.512641+0000","last_undegraded":"2024-03-11T21:08:24.512641+0000","last_fullsized":"2024-03-11T21:08:24.512641+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:08:16.136851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:06:42.242801+0000","last_clean_scrub_stamp":"2024-03-11T21:08:16.136851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:37.876869+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"207'18","reported_seq":282,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.512518+0000","last_change":"2024-03-11T21:08:05.299437+0000","last_active":"2024-03-11T21:08:24.512518+0000","last_peered":"2024-03-11T21:08:24.512518+0000","last_clean":"2024-03-11T21:08:24.512518+0000","last_became_active":"2024-03-11T21:08:05.298170+0000","last_became_peered":"2024-03-11T21:08:05.298170+0000","last_unstale":"2024-03-11T21:08:24.512518+0000","last_undegraded":"2024-03-11T21:08:24.512518+0000","last_fullsized":"2024-03-11T21:08:24.512518+0000","mapping_epoch":319,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'18","last_scrub_stamp":"2024-03-11T21:07:49.533424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:07:49.533424+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:08:55.100382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":23,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1","version":"326'4355","reported_seq":4455,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.831767+0000","last_change":"2024-03-11T21:08:23.498439+0000","last_active":"2024-03-11T21:08:24.831767+0000","last_peered":"2024-03-11T21:08:24.831767+0000","last_clean":"2024-03-11T21:08:24.831767+0000","last_became_active":"2024-03-11T21:07:14.951704+0000","last_became_peered":"2024-03-11T21:07:14.951704+0000","last_unstale":"2024-03-11T21:08:24.831767+0000","last_undegraded":"2024-03-11T21:08:24.831767+0000","last_fullsized":"2024-03-11T21:08:24.831767+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":306,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'4267","last_scrub_stamp":"2024-03-11T21:08:23.497931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:07:13.846944+0000","last_clean_scrub_stamp":"2024-03-11T21:08:23.497931+0000","objects_scrubbed":4222,"log_size":4355,"log_dups_size":0,"ondisk_log_size":4355,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:39.426552+0000","scrub_duration":1383,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":285409280,"num_objects":4355,"num_object_clones":0,"num_object_copies":13065,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4355,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4355,"num_write_kb":278720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":588,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.508978+0000","last_change":"2024-03-11T21:08:22.890801+0000","last_active":"2024-03-11T21:08:24.508978+0000","last_peered":"2024-03-11T21:08:24.508978+0000","last_clean":"2024-03-11T21:08:24.508978+0000","last_became_active":"2024-03-11T21:00:48.460531+0000","last_became_peered":"2024-03-11T21:00:48.460531+0000","last_unstale":"2024-03-11T21:08:24.508978+0000","last_undegraded":"2024-03-11T21:08:24.508978+0000","last_fullsized":"2024-03-11T21:08:24.508978+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:08:22.890690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:27.775534+0000","last_clean_scrub_stamp":"2024-03-11T21:08:22.890690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:42.927325+0000","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":294,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:26.382373+0000","last_change":"2024-03-11T21:08:26.382408+0000","last_active":"2024-03-11T21:08:26.382373+0000","last_peered":"2024-03-11T21:08:26.382373+0000","last_clean":"2024-03-11T21:08:26.382373+0000","last_became_active":"2024-03-11T21:00:48.469596+0000","last_became_peered":"2024-03-11T21:00:48.469596+0000","last_unstale":"2024-03-11T21:08:26.382373+0000","last_undegraded":"2024-03-11T21:08:26.382373+0000","last_fullsized":"2024-03-11T21:08:26.382373+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:08:26.382277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:08:26.382277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:31.988971+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":278,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.485996+0000","last_change":"2024-03-11T21:07:41.398091+0000","last_active":"2024-03-11T21:08:24.485996+0000","last_peered":"2024-03-11T21:08:24.485996+0000","last_clean":"2024-03-11T21:08:24.485996+0000","last_became_active":"2024-03-11T21:00:48.466084+0000","last_became_peered":"2024-03-11T21:00:48.466084+0000","last_unstale":"2024-03-11T21:08:24.485996+0000","last_undegraded":"2024-03-11T21:08:24.485996+0000","last_fullsized":"2024-03-11T21:08:24.485996+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:07:41.397960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:40.974147+0000","last_clean_scrub_stamp":"2024-03-11T21:07:41.397960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:08:50.008845+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"207'34","reported_seq":670,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.490221+0000","last_change":"2024-03-11T21:06:11.347952+0000","last_active":"2024-03-11T21:08:24.490221+0000","last_peered":"2024-03-11T21:08:24.490221+0000","last_clean":"2024-03-11T21:08:24.490221+0000","last_became_active":"2024-03-11T21:00:48.472191+0000","last_became_peered":"2024-03-11T21:00:48.472191+0000","last_unstale":"2024-03-11T21:08:24.490221+0000","last_undegraded":"2024-03-11T21:08:24.490221+0000","last_fullsized":"2024-03-11T21:08:24.490221+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'34","last_scrub_stamp":"2024-03-11T21:06:11.347860+0000","last_deep_scrub":"116'22","last_deep_scrub_stamp":"2024-03-11T20:53:47.364709+0000","last_clean_scrub_stamp":"2024-03-11T21:06:11.347860+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"326'4520","reported_seq":4570,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:27.216316+0000","last_change":"2024-03-11T21:07:14.952811+0000","last_active":"2024-03-11T21:08:27.216316+0000","last_peered":"2024-03-11T21:08:27.216316+0000","last_clean":"2024-03-11T21:08:27.216316+0000","last_became_active":"2024-03-11T21:07:14.951968+0000","last_became_peered":"2024-03-11T21:07:14.951968+0000","last_unstale":"2024-03-11T21:08:27.216316+0000","last_undegraded":"2024-03-11T21:08:27.216316+0000","last_fullsized":"2024-03-11T21:08:27.216316+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":306,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:07:13.846944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:07:13.846944+0000","last_clean_scrub_stamp":"2024-03-11T21:07:13.846944+0000","objects_scrubbed":0,"log_size":4520,"log_dups_size":0,"ondisk_log_size":4520,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":296222720,"num_objects":4520,"num_object_clones":0,"num_object_copies":13560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4520,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4520,"num_write_kb":289280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":365,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.486286+0000","last_change":"2024-03-11T21:08:11.531203+0000","last_active":"2024-03-11T21:08:24.486286+0000","last_peered":"2024-03-11T21:08:24.486286+0000","last_clean":"2024-03-11T21:08:24.486286+0000","last_became_active":"2024-03-11T21:05:17.455236+0000","last_became_peered":"2024-03-11T21:05:17.455236+0000","last_unstale":"2024-03-11T21:08:24.486286+0000","last_undegraded":"2024-03-11T21:08:24.486286+0000","last_fullsized":"2024-03-11T21:08:24.486286+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:08:11.530874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:08:11.530874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:27.560603+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":286,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.486220+0000","last_change":"2024-03-11T21:07:58.612531+0000","last_active":"2024-03-11T21:08:24.486220+0000","last_peered":"2024-03-11T21:08:24.486220+0000","last_clean":"2024-03-11T21:08:24.486220+0000","last_became_active":"2024-03-11T21:00:48.461367+0000","last_became_peered":"2024-03-11T21:00:48.461367+0000","last_unstale":"2024-03-11T21:08:24.486220+0000","last_undegraded":"2024-03-11T21:08:24.486220+0000","last_fullsized":"2024-03-11T21:08:24.486220+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:07:58.612421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:03:37.601229+0000","last_clean_scrub_stamp":"2024-03-11T21:07:58.612421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:24.827499+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"267'19","reported_seq":597,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.490806+0000","last_change":"2024-03-11T21:06:19.516422+0000","last_active":"2024-03-11T21:08:24.490806+0000","last_peered":"2024-03-11T21:08:24.490806+0000","last_clean":"2024-03-11T21:08:24.490806+0000","last_became_active":"2024-03-11T21:00:27.967572+0000","last_became_peered":"2024-03-11T21:00:27.967572+0000","last_unstale":"2024-03-11T21:08:24.490806+0000","last_undegraded":"2024-03-11T21:08:24.490806+0000","last_fullsized":"2024-03-11T21:08:24.490806+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"267'19","last_scrub_stamp":"2024-03-11T21:06:19.516296+0000","last_deep_scrub":"267'19","last_deep_scrub_stamp":"2024-03-11T21:03:39.673317+0000","last_clean_scrub_stamp":"2024-03-11T21:06:19.516296+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":358,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.492040+0000","last_change":"2024-03-11T21:08:03.331915+0000","last_active":"2024-03-11T21:08:24.492040+0000","last_peered":"2024-03-11T21:08:24.492040+0000","last_clean":"2024-03-11T21:08:24.492040+0000","last_became_active":"2024-03-11T21:05:17.316136+0000","last_became_peered":"2024-03-11T21:05:17.316136+0000","last_unstale":"2024-03-11T21:08:24.492040+0000","last_undegraded":"2024-03-11T21:08:24.492040+0000","last_fullsized":"2024-03-11T21:08:24.492040+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:08:03.331832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:35.670149+0000","last_clean_scrub_stamp":"2024-03-11T21:08:03.331832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:16.310959+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"326'4325","reported_seq":4381,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:25.441122+0000","last_change":"2024-03-11T21:08:05.305057+0000","last_active":"2024-03-11T21:08:25.441122+0000","last_peered":"2024-03-11T21:08:25.441122+0000","last_clean":"2024-03-11T21:08:25.441122+0000","last_became_active":"2024-03-11T21:08:05.298758+0000","last_became_peered":"2024-03-11T21:08:05.298758+0000","last_unstale":"2024-03-11T21:08:25.441122+0000","last_undegraded":"2024-03-11T21:08:25.441122+0000","last_fullsized":"2024-03-11T21:08:25.441122+0000","mapping_epoch":319,"log_start":"0'0","ondisk_log_start":"0'0","created":306,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:07:13.846944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:07:13.846944+0000","last_clean_scrub_stamp":"2024-03-11T21:07:13.846944+0000","objects_scrubbed":0,"log_size":4325,"log_dups_size":0,"ondisk_log_size":4325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:08:35.842052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":283443200,"num_objects":4325,"num_object_clones":0,"num_object_copies":12975,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4325,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4325,"num_write_kb":276800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":329,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.490793+0000","last_change":"2024-03-11T21:08:05.305618+0000","last_active":"2024-03-11T21:08:24.490793+0000","last_peered":"2024-03-11T21:08:24.490793+0000","last_clean":"2024-03-11T21:08:24.490793+0000","last_became_active":"2024-03-11T21:08:05.304963+0000","last_became_peered":"2024-03-11T21:08:05.304963+0000","last_unstale":"2024-03-11T21:08:24.490793+0000","last_undegraded":"2024-03-11T21:08:24.490793+0000","last_fullsized":"2024-03-11T21:08:24.490793+0000","mapping_epoch":319,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:07:38.448238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:07:38.448238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:08:45.878342+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"320'21","reported_seq":528,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.508392+0000","last_change":"2024-03-11T21:08:05.299421+0000","last_active":"2024-03-11T21:08:24.508392+0000","last_peered":"2024-03-11T21:08:24.508392+0000","last_clean":"2024-03-11T21:08:24.508392+0000","last_became_active":"2024-03-11T21:08:05.298887+0000","last_became_peered":"2024-03-11T21:08:05.298887+0000","last_unstale":"2024-03-11T21:08:24.508392+0000","last_undegraded":"2024-03-11T21:08:24.508392+0000","last_fullsized":"2024-03-11T21:08:24.508392+0000","mapping_epoch":319,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":0,"last_scrub":"286'20","last_scrub_stamp":"2024-03-11T21:08:02.983969+0000","last_deep_scrub":"146'14","last_deep_scrub_stamp":"2024-03-11T20:56:00.242211+0000","last_clean_scrub_stamp":"2024-03-11T21:08:02.983969+0000","objects_scrubbed":1,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:21.708236+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":586,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.512844+0000","last_change":"2024-03-11T21:07:52.165209+0000","last_active":"2024-03-11T21:08:24.512844+0000","last_peered":"2024-03-11T21:08:24.512844+0000","last_clean":"2024-03-11T21:08:24.512844+0000","last_became_active":"2024-03-11T21:00:27.970573+0000","last_became_peered":"2024-03-11T21:00:27.970573+0000","last_unstale":"2024-03-11T21:08:24.512844+0000","last_undegraded":"2024-03-11T21:08:24.512844+0000","last_fullsized":"2024-03-11T21:08:24.512844+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:07:52.165109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:52:51.161281+0000","last_clean_scrub_stamp":"2024-03-11T21:07:52.165109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:08.174733+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"207'44","reported_seq":731,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.485344+0000","last_change":"2024-03-11T21:08:03.490157+0000","last_active":"2024-03-11T21:08:24.485344+0000","last_peered":"2024-03-11T21:08:24.485344+0000","last_clean":"2024-03-11T21:08:24.485344+0000","last_became_active":"2024-03-11T21:00:48.469854+0000","last_became_peered":"2024-03-11T21:00:48.469854+0000","last_unstale":"2024-03-11T21:08:24.485344+0000","last_undegraded":"2024-03-11T21:08:24.485344+0000","last_fullsized":"2024-03-11T21:08:24.485344+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'44","last_scrub_stamp":"2024-03-11T21:08:03.489929+0000","last_deep_scrub":"161'28","last_deep_scrub_stamp":"2024-03-11T20:57:20.816278+0000","last_clean_scrub_stamp":"2024-03-11T21:08:03.489929+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:14.596807+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.4","version":"326'4326","reported_seq":4376,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.926278+0000","last_change":"2024-03-11T21:07:14.960911+0000","last_active":"2024-03-11T21:08:24.926278+0000","last_peered":"2024-03-11T21:08:24.926278+0000","last_clean":"2024-03-11T21:08:24.926278+0000","last_became_active":"2024-03-11T21:07:14.959964+0000","last_became_peered":"2024-03-11T21:07:14.959964+0000","last_unstale":"2024-03-11T21:08:24.926278+0000","last_undegraded":"2024-03-11T21:08:24.926278+0000","last_fullsized":"2024-03-11T21:08:24.926278+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":306,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:07:13.846944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:07:13.846944+0000","last_clean_scrub_stamp":"2024-03-11T21:07:13.846944+0000","objects_scrubbed":0,"log_size":4326,"log_dups_size":0,"ondisk_log_size":4326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:08:35.763800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":283508736,"num_objects":4326,"num_object_clones":0,"num_object_copies":12978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4326,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4326,"num_write_kb":276864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":574,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.512353+0000","last_change":"2024-03-11T21:07:56.111952+0000","last_active":"2024-03-11T21:08:24.512353+0000","last_peered":"2024-03-11T21:08:24.512353+0000","last_clean":"2024-03-11T21:08:24.512353+0000","last_became_active":"2024-03-11T20:50:34.327139+0000","last_became_peered":"2024-03-11T20:50:34.327139+0000","last_unstale":"2024-03-11T21:08:24.512353+0000","last_undegraded":"2024-03-11T21:08:24.512353+0000","last_fullsized":"2024-03-11T21:08:24.512353+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:07:56.111591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:07:56.111591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:14.482857+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":25,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.507809+0000","last_change":"2024-03-11T21:08:05.303485+0000","last_active":"2024-03-11T21:08:24.507809+0000","last_peered":"2024-03-11T21:08:24.507809+0000","last_clean":"2024-03-11T21:08:24.507809+0000","last_became_active":"2024-03-11T21:08:05.303226+0000","last_became_peered":"2024-03-11T21:08:05.303226+0000","last_unstale":"2024-03-11T21:08:24.507809+0000","last_undegraded":"2024-03-11T21:08:24.507809+0000","last_fullsized":"2024-03-11T21:08:24.507809+0000","mapping_epoch":319,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:07:39.353184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:57:17.447191+0000","last_clean_scrub_stamp":"2024-03-11T21:07:39.353184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:06.110363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"207'39","reported_seq":638,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.513088+0000","last_change":"2024-03-11T21:07:41.182291+0000","last_active":"2024-03-11T21:08:24.513088+0000","last_peered":"2024-03-11T21:08:24.513088+0000","last_clean":"2024-03-11T21:08:24.513088+0000","last_became_active":"2024-03-11T20:50:30.111819+0000","last_became_peered":"2024-03-11T20:50:30.111819+0000","last_unstale":"2024-03-11T21:08:24.513088+0000","last_undegraded":"2024-03-11T21:08:24.513088+0000","last_fullsized":"2024-03-11T21:08:24.513088+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'39","last_scrub_stamp":"2024-03-11T21:07:41.182193+0000","last_deep_scrub":"102'25","last_deep_scrub_stamp":"2024-03-11T20:51:35.187094+0000","last_clean_scrub_stamp":"2024-03-11T21:07:41.182193+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:08.028067+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":599,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.485115+0000","last_change":"2024-03-11T21:08:05.495968+0000","last_active":"2024-03-11T21:08:24.485115+0000","last_peered":"2024-03-11T21:08:24.485115+0000","last_clean":"2024-03-11T21:08:24.485115+0000","last_became_active":"2024-03-11T21:00:48.362389+0000","last_became_peered":"2024-03-11T21:00:48.362389+0000","last_unstale":"2024-03-11T21:08:24.485115+0000","last_undegraded":"2024-03-11T21:08:24.485115+0000","last_fullsized":"2024-03-11T21:08:24.485115+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:08:05.495776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:08:05.495776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:11.113630+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.5","version":"326'4417","reported_seq":4471,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.919095+0000","last_change":"2024-03-11T21:08:05.315370+0000","last_active":"2024-03-11T21:08:24.919095+0000","last_peered":"2024-03-11T21:08:24.919095+0000","last_clean":"2024-03-11T21:08:24.919095+0000","last_became_active":"2024-03-11T21:08:05.304912+0000","last_became_peered":"2024-03-11T21:08:05.304912+0000","last_unstale":"2024-03-11T21:08:24.919095+0000","last_undegraded":"2024-03-11T21:08:24.919095+0000","last_fullsized":"2024-03-11T21:08:24.919095+0000","mapping_epoch":319,"log_start":"0'0","ondisk_log_start":"0'0","created":306,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:07:13.846944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:07:13.846944+0000","last_clean_scrub_stamp":"2024-03-11T21:07:13.846944+0000","objects_scrubbed":0,"log_size":4417,"log_dups_size":0,"ondisk_log_size":4417,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":289472512,"num_objects":4417,"num_object_clones":0,"num_object_copies":13251,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4417,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4417,"num_write_kb":282688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":270,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.511390+0000","last_change":"2024-03-11T21:07:44.901299+0000","last_active":"2024-03-11T21:08:24.511390+0000","last_peered":"2024-03-11T21:08:24.511390+0000","last_clean":"2024-03-11T21:08:24.511390+0000","last_became_active":"2024-03-11T21:06:28.758451+0000","last_became_peered":"2024-03-11T21:06:28.758451+0000","last_unstale":"2024-03-11T21:08:24.511390+0000","last_undegraded":"2024-03-11T21:08:24.511390+0000","last_fullsized":"2024-03-11T21:08:24.511390+0000","mapping_epoch":297,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:07:44.901223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:55:58.888890+0000","last_clean_scrub_stamp":"2024-03-11T21:07:44.901223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:08:56.046804+0000","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":278,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.486173+0000","last_change":"2024-03-11T21:07:38.356703+0000","last_active":"2024-03-11T21:08:24.486173+0000","last_peered":"2024-03-11T21:08:24.486173+0000","last_clean":"2024-03-11T21:08:24.486173+0000","last_became_active":"2024-03-11T21:00:48.469015+0000","last_became_peered":"2024-03-11T21:00:48.469015+0000","last_unstale":"2024-03-11T21:08:24.486173+0000","last_undegraded":"2024-03-11T21:08:24.486173+0000","last_fullsized":"2024-03-11T21:08:24.486173+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:07:38.356551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:07:38.356551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:08:41.014665+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"207'20","reported_seq":605,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.507725+0000","last_change":"2024-03-11T21:08:01.874794+0000","last_active":"2024-03-11T21:08:24.507725+0000","last_peered":"2024-03-11T21:08:24.507725+0000","last_clean":"2024-03-11T21:08:24.507725+0000","last_became_active":"2024-03-11T20:50:30.109549+0000","last_became_peered":"2024-03-11T20:50:30.109549+0000","last_unstale":"2024-03-11T21:08:24.507725+0000","last_undegraded":"2024-03-11T21:08:24.507725+0000","last_fullsized":"2024-03-11T21:08:24.507725+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'20","last_scrub_stamp":"2024-03-11T21:08:01.874656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:08:01.874656+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:14.866467+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":616,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.513185+0000","last_change":"2024-03-11T21:07:50.781503+0000","last_active":"2024-03-11T21:08:24.513185+0000","last_peered":"2024-03-11T21:08:24.513185+0000","last_clean":"2024-03-11T21:08:24.513185+0000","last_became_active":"2024-03-11T21:02:37.517748+0000","last_became_peered":"2024-03-11T21:02:37.517748+0000","last_unstale":"2024-03-11T21:08:24.513185+0000","last_undegraded":"2024-03-11T21:08:24.513185+0000","last_fullsized":"2024-03-11T21:08:24.513185+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:07:50.781425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:52:51.931295+0000","last_clean_scrub_stamp":"2024-03-11T21:07:50.781425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:08:59.606631+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.6","version":"326'4430","reported_seq":4479,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:25.454161+0000","last_change":"2024-03-11T21:07:14.953797+0000","last_active":"2024-03-11T21:08:25.454161+0000","last_peered":"2024-03-11T21:08:25.454161+0000","last_clean":"2024-03-11T21:08:25.454161+0000","last_became_active":"2024-03-11T21:07:14.952904+0000","last_became_peered":"2024-03-11T21:07:14.952904+0000","last_unstale":"2024-03-11T21:08:25.454161+0000","last_undegraded":"2024-03-11T21:08:25.454161+0000","last_fullsized":"2024-03-11T21:08:25.454161+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":306,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:07:13.846944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:07:13.846944+0000","last_clean_scrub_stamp":"2024-03-11T21:07:13.846944+0000","objects_scrubbed":0,"log_size":4430,"log_dups_size":0,"ondisk_log_size":4430,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:08:40.812663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":290324480,"num_objects":4430,"num_object_clones":0,"num_object_copies":13290,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4430,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4430,"num_write_kb":283520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":48,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.511598+0000","last_change":"2024-03-11T21:07:46.870757+0000","last_active":"2024-03-11T21:08:24.511598+0000","last_peered":"2024-03-11T21:08:24.511598+0000","last_clean":"2024-03-11T21:08:24.511598+0000","last_became_active":"2024-03-11T21:07:34.504579+0000","last_became_peered":"2024-03-11T21:07:34.504579+0000","last_unstale":"2024-03-11T21:08:24.511598+0000","last_undegraded":"2024-03-11T21:08:24.511598+0000","last_fullsized":"2024-03-11T21:08:24.511598+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:07:46.870646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:00:14.030599+0000","last_clean_scrub_stamp":"2024-03-11T21:07:46.870646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:10.091794+0000","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":25,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.490664+0000","last_change":"2024-03-11T21:08:05.308620+0000","last_active":"2024-03-11T21:08:24.490664+0000","last_peered":"2024-03-11T21:08:24.490664+0000","last_clean":"2024-03-11T21:08:24.490664+0000","last_became_active":"2024-03-11T21:08:05.307984+0000","last_became_peered":"2024-03-11T21:08:05.307984+0000","last_unstale":"2024-03-11T21:08:24.490664+0000","last_undegraded":"2024-03-11T21:08:24.490664+0000","last_fullsized":"2024-03-11T21:08:24.490664+0000","mapping_epoch":319,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:07:57.816881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:07:57.816881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:08:57.826082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.f","version":"207'27","reported_seq":625,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.513254+0000","last_change":"2024-03-11T21:08:04.106067+0000","last_active":"2024-03-11T21:08:24.513254+0000","last_peered":"2024-03-11T21:08:24.513254+0000","last_clean":"2024-03-11T21:08:24.513254+0000","last_became_active":"2024-03-11T21:00:27.963119+0000","last_became_peered":"2024-03-11T21:00:27.963119+0000","last_unstale":"2024-03-11T21:08:24.513254+0000","last_undegraded":"2024-03-11T21:08:24.513254+0000","last_fullsized":"2024-03-11T21:08:24.513254+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'27","last_scrub_stamp":"2024-03-11T21:08:04.105922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:08:04.105922+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:25.134101+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":572,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-11T21:08:24.507959+0000","last_change":"2024-03-11T21:08:03.929764+0000","last_active":"2024-03-11T21:08:24.507959+0000","last_peered":"2024-03-11T21:08:24.507959+0000","last_clean":"2024-03-11T21:08:24.507959+0000","last_became_active":"2024-03-11T20:50:32.652854+0000","last_became_peered":"2024-03-11T20:50:32.652854+0000","last_unstale":"2024-03-11T21:08:24.507959+0000","last_undegraded":"2024-03-11T21:08:24.507959+0000","last_fullsized":"2024-03-11T21:08:24.507959+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:08:03.929608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:59:00.946094+0000","last_clean_scrub_stamp":"2024-03-11T21:08:03.929608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:27.258446+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":4590927872,"num_objects":70052,"num_object_clones":0,"num_object_copies":210156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70052,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":70052,"num_write_kb":4483328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13808435200,"data_stored":13808435200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70052,"ondisk_log_size":70052,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":20,"num_read_kb":20,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1822,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1121,"num_read_kb":878,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":171,"num_bytes_recovered":2228,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3632,"internal_metadata":0},"log_size":778,"ondisk_log_size":778,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":126,"ondisk_log_size":126,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":939,"num_read_kb":934,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":68,"seq":292057776354,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1515000,"kb_used_data":1100656,"kb_used_omap":1,"kb_used_meta":414334,"kb_avail":92230152,"statfs":{"total":95995035648,"available":94443675648,"internally_reserved":0,"allocated":1127071744,"data_stored":1126642913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":424278474},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":57,"seq":244813136108,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2262248,"kb_used_data":1942304,"kb_used_omap":1,"kb_used_meta":319934,"kb_avail":91482904,"statfs":{"total":95995035648,"available":93678493696,"internally_reserved":0,"allocated":1988919296,"data_stored":1988506859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":327612871},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1,6,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":211,"seq":906238099549,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1183288,"kb_used_data":854064,"kb_used_omap":7,"kb_used_meta":329208,"kb_avail":92561864,"statfs":{"total":95995035648,"available":94783348736,"internally_reserved":0,"allocated":874561536,"data_stored":874140692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8078,"internal_metadata":337109106},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738666,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3114416,"kb_used_data":2762536,"kb_used_omap":1,"kb_used_meta":351870,"kb_avail":90630736,"statfs":{"total":95995035648,"available":92805873664,"internally_reserved":0,"allocated":2828836864,"data_stored":2828415167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":360315333},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378901,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2010912,"kb_used_data":1659028,"kb_used_omap":1,"kb_used_meta":351870,"kb_avail":91734240,"statfs":{"total":95995035648,"available":93935861760,"internally_reserved":0,"allocated":1698844672,"data_stored":1698443073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":360315333},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":28,"seq":120259084554,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1906784,"kb_used_data":1388312,"kb_used_omap":1,"kb_used_meta":518462,"kb_avail":91838368,"statfs":{"total":95995035648,"available":94042488832,"internally_reserved":0,"allocated":1421631488,"data_stored":1421226169,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":530905544},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":204,"seq":876173328481,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2449552,"kb_used_data":2260296,"kb_used_omap":4,"kb_used_meta":189243,"kb_avail":91295600,"statfs":{"total":95995035648,"available":93486694400,"internally_reserved":0,"allocated":2314543104,"data_stored":2314098298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4451,"internal_metadata":193785501},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":44,"seq":188978561270,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1779760,"kb_used_data":1391780,"kb_used_omap":1,"kb_used_meta":387966,"kb_avail":91965392,"statfs":{"total":95995035648,"available":94172561408,"internally_reserved":0,"allocated":1425182720,"data_stored":1424772934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":397277637},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3632,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2854486016,"data_stored":2854486016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":885129216,"data_stored":885129216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1720909824,"data_stored":1720909824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1433665536,"data_stored":1433665536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2328625152,"data_stored":2328625152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1440481280,"data_stored":1440481280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2005008384,"data_stored":2005008384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1140129792,"data_stored":1140129792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T21:08:32.880 INFO:tasks.cephadm.ceph_manager.ceph:would drop below min_pgs, proj 6, currently 16 2024-03-11T21:08:32.928 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:08:32.957 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:08:32.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:32.960 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:08:33.061 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:08:33.098 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:08:33.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:33.100 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:08:33.201 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:08:33.229 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:08:33.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:33.232 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:08:33.333 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:33.365 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:33.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:33.367 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:08:33.369 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 57 16 78670 78654 86.2349 66.75 0.0101999 0.0115904 2024-03-11T21:08:33.468 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:33.501 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:33.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:33.506 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:08:33.610 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:08:33.619 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:33 smithi146 bash[23809]: cluster 2024-03-11T21:08:32.274616+0000 mgr.y (mgr.24380) 657 : cluster 0 pgmap v738: 148 pgs: 148 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 83 MiB/s wr, 1.33k op/s 2024-03-11T21:08:33.633 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:08:33.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:33.635 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:08:33.736 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:08:33.764 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:08:33.766 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:33 smithi023 bash[21372]: cluster 2024-03-11T21:08:32.274616+0000 mgr.y (mgr.24380) 657 : cluster 0 pgmap v738: 148 pgs: 148 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 83 MiB/s wr, 1.33k op/s 2024-03-11T21:08:33.767 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:33 smithi023 bash[29146]: cluster 2024-03-11T21:08:32.274616+0000 mgr.y (mgr.24380) 657 : cluster 0 pgmap v738: 148 pgs: 148 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 83 MiB/s wr, 1.33k op/s 2024-03-11T21:08:33.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:33.768 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:08:33.868 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:08:33.892 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:08:33.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:33.894 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:08:33.995 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:34.026 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:34.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:34.033 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:08:34.134 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:34.163 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:34.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:34.169 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:08:34.269 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:08:34.298 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:08:34.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:34.300 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:08:34.370 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 58 16 79787 79771 85.9517 69.8125 0.0154856 0.0116292 2024-03-11T21:08:34.401 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:34.426 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:34.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:34.432 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:08:34.532 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:34.560 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:34.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:34.563 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:08:34.664 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:34.690 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:34.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:34.694 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:08:34.794 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:08:34.818 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:08:34.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:34.823 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:08:34.924 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:08:34.957 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:08:34.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:34.959 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:08:34.973 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-11T21:08:35.060 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:35.087 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:35.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:35.090 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:08:35.191 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:08:35.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:35.223 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:08:35.223 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:08:35.324 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:08:35.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:35.354 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:08:35.354 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:08:35.370 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 59 16 80851 80835 85.6219 66.5 0.0142697 0.0116745 2024-03-11T21:08:35.455 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:08:35.478 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:08:35.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:35.482 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:08:35.582 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:08:35.592 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:35 smithi146 bash[23809]: cluster 2024-03-11T21:08:34.275124+0000 mgr.y (mgr.24380) 658 : cluster 0 pgmap v739: 148 pgs: 148 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.0 KiB/s rd, 68 MiB/s wr, 1.09k op/s 2024-03-11T21:08:35.592 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:35 smithi146 bash[23809]: cluster 2024-03-11T21:08:35.295724+0000 mon.a (mon.0) 1660 : cluster 0 osdmap e328: 8 total, 8 up, 8 in 2024-03-11T21:08:35.611 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:08:35.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:35.615 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:08:35.716 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:35.757 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:35.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:35.760 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:08:35.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:35 smithi023 bash[21372]: cluster 2024-03-11T21:08:34.275124+0000 mgr.y (mgr.24380) 658 : cluster 0 pgmap v739: 148 pgs: 148 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.0 KiB/s rd, 68 MiB/s wr, 1.09k op/s 2024-03-11T21:08:35.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:35 smithi023 bash[21372]: cluster 2024-03-11T21:08:35.295724+0000 mon.a (mon.0) 1660 : cluster 0 osdmap e328: 8 total, 8 up, 8 in 2024-03-11T21:08:35.779 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:35 smithi023 bash[29146]: cluster 2024-03-11T21:08:34.275124+0000 mgr.y (mgr.24380) 658 : cluster 0 pgmap v739: 148 pgs: 148 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.0 KiB/s rd, 68 MiB/s wr, 1.09k op/s 2024-03-11T21:08:35.779 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:35 smithi023 bash[29146]: cluster 2024-03-11T21:08:35.295724+0000 mon.a (mon.0) 1660 : cluster 0 osdmap e328: 8 total, 8 up, 8 in 2024-03-11T21:08:35.861 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:35.870 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:08:35 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:08:35.884 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:35.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:35.886 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:08:35.987 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:08:36.016 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:08:36.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:36.022 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:08:36.123 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:36.159 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:36.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:36.161 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:08:36.262 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:36.295 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:36.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:36.296 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:08:36.370 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:2024-03-11T21:08:36.379900+0000 min lat: 0.00119683 max lat: 1.06642 avg lat: 0.0117118 2024-03-11T21:08:36.370 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-11T21:08:36.370 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 60 16 81955 81939 85.3447 69 0.0145862 0.0117118 2024-03-11T21:08:36.397 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:08:36.428 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:08:36.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:36.432 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:08:36.499 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:08:36.533 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:36.558 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:36.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:36.563 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:08:36.664 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:36.691 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:36.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:36.697 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:08:36.798 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:08:36.884 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:36 smithi146 bash[23809]: audit 2024-03-11T21:08:35.733186+0000 mgr.y (mgr.24380) 659 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:08:36.884 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:36 smithi146 bash[23809]: audit 2024-03-11T21:08:35.754399+0000 mon.a (mon.0) 1661 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:08:36.884 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:36 smithi146 bash[23809]: audit 2024-03-11T21:08:35.755675+0000 mon.a (mon.0) 1662 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:08:36.884 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:36 smithi146 bash[23809]: cluster 2024-03-11T21:08:36.275663+0000 mgr.y (mgr.24380) 660 : cluster 0 pgmap v741: 148 pgs: 148 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 80 MiB/s wr, 1.28k op/s 2024-03-11T21:08:36.899 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:08:36.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:36.900 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:08:37.001 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:08:37.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:36 smithi023 bash[21372]: audit 2024-03-11T21:08:35.733186+0000 mgr.y (mgr.24380) 659 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:08:37.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:36 smithi023 bash[21372]: audit 2024-03-11T21:08:35.754399+0000 mon.a (mon.0) 1661 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:08:37.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:36 smithi023 bash[21372]: audit 2024-03-11T21:08:35.755675+0000 mon.a (mon.0) 1662 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:08:37.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:36 smithi023 bash[21372]: cluster 2024-03-11T21:08:36.275663+0000 mgr.y (mgr.24380) 660 : cluster 0 pgmap v741: 148 pgs: 148 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 80 MiB/s wr, 1.28k op/s 2024-03-11T21:08:37.030 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:36 smithi023 bash[29146]: audit 2024-03-11T21:08:35.733186+0000 mgr.y (mgr.24380) 659 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:08:37.030 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:36 smithi023 bash[29146]: audit 2024-03-11T21:08:35.754399+0000 mon.a (mon.0) 1661 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:08:37.030 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:36 smithi023 bash[29146]: audit 2024-03-11T21:08:35.755675+0000 mon.a (mon.0) 1662 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:08:37.030 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:36 smithi023 bash[29146]: cluster 2024-03-11T21:08:36.275663+0000 mgr.y (mgr.24380) 660 : cluster 0 pgmap v741: 148 pgs: 148 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 80 MiB/s wr, 1.28k op/s 2024-03-11T21:08:37.030 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:08:37.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:37.035 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:08:37.136 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:37.161 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:37.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:37.165 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:08:37.265 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:37.285 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:37.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:37.287 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:08:37.370 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 61 16 82856 82840 84.8687 56.3125 0.00699198 0.011745 2024-03-11T21:08:37.388 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:08:37.413 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:08:37.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:37.415 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:08:37.516 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:37.525 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:08:37.540 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:37.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:37.542 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:08:37.643 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:37.681 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:37.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:37.684 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:08:37.784 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:08:37.821 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:08:37.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:37.827 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:08:37.881 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 6, 7, 1] out_osds: [] dead_osds: [] live_osds: [0, 2, 3, 5, 6, 7, 4, 1] 2024-03-11T21:08:37.882 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-11T21:08:37.882 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-03-11T21:08:37.882 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd dump -f json-pretty 2024-03-11T21:08:37.928 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:08:37.951 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:08:37.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:37.955 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:08:38.055 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:08:38.080 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:08:38.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:38.086 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:08:38.186 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:08:38.204 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:08:38.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:38.206 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:08:38.307 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:08:38.332 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:08:38.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:38.334 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:08:38.370 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 62 15 83464 83449 84.1137 38.0625 0.0270296 0.0118834 2024-03-11T21:08:38.435 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:08:38.459 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:08:38.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:38.464 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:08:38.529 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:08:38 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:08:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:08:38.565 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:08:38.591 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:08:38.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:38.594 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:08:38.695 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:08:38.715 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:08:38.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:38.717 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:08:38.818 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:38.840 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:38.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:38.842 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:08:38.943 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:38.974 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:38.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:38.977 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:08:39.078 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:39.114 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:39.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:39.118 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:08:39.219 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:08:39.265 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:08:39.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:39.268 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:08:39.368 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:39.370 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 63 16 84259 84243 83.5662 49.625 0.0179545 0.0119619 2024-03-11T21:08:39.396 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:39.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:39.401 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:08:39.502 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:08:39.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:39 smithi146 bash[23809]: cluster 2024-03-11T21:08:38.276596+0000 mgr.y (mgr.24380) 661 : cluster 0 pgmap v742: 148 pgs: 148 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 52 MiB/s wr, 832 op/s 2024-03-11T21:08:39.532 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:08:39.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:39.536 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:08:39.574 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:08:39.580 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:39 smithi023 bash[29146]: cluster 2024-03-11T21:08:38.276596+0000 mgr.y (mgr.24380) 661 : cluster 0 pgmap v742: 148 pgs: 148 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 52 MiB/s wr, 832 op/s 2024-03-11T21:08:39.582 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:39 smithi023 bash[21372]: cluster 2024-03-11T21:08:38.276596+0000 mgr.y (mgr.24380) 661 : cluster 0 pgmap v742: 148 pgs: 148 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 52 MiB/s wr, 832 op/s 2024-03-11T21:08:39.637 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:08:39.665 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:08:39.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:39.671 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:08:39.772 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:39.799 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:39.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:39.804 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:08:39.905 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:08:39.937 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:08:39.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:39.940 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:08:40.041 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:08:40.069 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:08:40.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:40.071 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:08:40.172 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:08:40.189 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:08:40.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:40.191 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:08:40.292 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:08:40.321 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:08:40.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:40.325 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:08:40.370 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 64 16 85150 85134 83.1305 55.6875 0.00820003 0.0120247 2024-03-11T21:08:40.426 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:08:40.452 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:08:40.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:40.459 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:08:40.560 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:08:40.585 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:08:40.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:40.590 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:08:40.691 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:08:40.725 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:08:40.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:40.728 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:08:40.829 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:08:40.853 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:08:40.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:40.857 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:08:40.958 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:40.989 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:40.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:40.994 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:08:41.095 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:08:41.122 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:08:41.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:41.130 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:08:41.231 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:41.265 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:41.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:41.269 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:08:41.370 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:08:41.371 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 65 16 86056 86040 82.7227 56.625 0.00626195 0.012084 2024-03-11T21:08:41.397 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:08:41.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:41.399 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:08:41.500 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:08:41.509 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:41 smithi146 bash[23809]: cluster 2024-03-11T21:08:40.277480+0000 mgr.y (mgr.24380) 662 : cluster 0 pgmap v743: 148 pgs: 148 active+clean; 5.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.0 KiB/s rd, 58 MiB/s wr, 929 op/s 2024-03-11T21:08:41.534 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:08:41.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:41.537 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:08:41.638 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:41.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:41.672 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:41.672 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:08:41.682 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:41 smithi023 bash[21372]: cluster 2024-03-11T21:08:40.277480+0000 mgr.y (mgr.24380) 662 : cluster 0 pgmap v743: 148 pgs: 148 active+clean; 5.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.0 KiB/s rd, 58 MiB/s wr, 929 op/s 2024-03-11T21:08:41.683 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:41 smithi023 bash[29146]: cluster 2024-03-11T21:08:40.277480+0000 mgr.y (mgr.24380) 662 : cluster 0 pgmap v743: 148 pgs: 148 active+clean; 5.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.0 KiB/s rd, 58 MiB/s wr, 929 op/s 2024-03-11T21:08:41.773 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:41.793 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:41.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:41.795 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:08:41.896 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:08:41.921 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:08:41.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:41.923 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:08:42.023 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:42.063 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:42.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:42.065 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:08:42.166 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:42.200 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:42.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:42.204 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:08:42.305 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:42.329 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:42.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:42.334 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:08:42.370 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 66 16 86826 86810 82.1984 48.125 0.0189814 0.0121598 2024-03-11T21:08:42.435 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:08:42.460 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:08:42.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:42.467 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:08:42.568 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:42.586 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:42.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:42.589 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:08:42.690 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:08:42.724 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:08:42.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:42.727 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:08:42.828 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:42.854 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:42.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:42.856 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:08:42.957 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:42.980 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:42.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:42.983 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:08:43.084 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:43.110 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:43.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:43.115 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:08:43.216 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:43.242 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:43.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:43.245 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:08:43.345 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:08:43.346 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:43 smithi023 bash[21372]: cluster 2024-03-11T21:08:42.219825+0000 osd.4 (osd.4) 107 : cluster 0 purged_snaps scrub starts 2024-03-11T21:08:43.347 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:43 smithi023 bash[21372]: cluster 2024-03-11T21:08:42.219910+0000 osd.4 (osd.4) 108 : cluster 0 purged_snaps scrub ok 2024-03-11T21:08:43.347 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:43 smithi023 bash[21372]: cluster 2024-03-11T21:08:42.278003+0000 mgr.y (mgr.24380) 663 : cluster 0 pgmap v744: 148 pgs: 148 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 61 MiB/s wr, 982 op/s 2024-03-11T21:08:43.347 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:43 smithi023 bash[21372]: cluster 2024-03-11T21:08:42.352811+0000 mon.a (mon.0) 1663 : cluster 0 osdmap e329: 8 total, 8 up, 8 in 2024-03-11T21:08:43.372 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 67 16 87762 87746 81.8425 58.5 0.00732811 0.0122128 2024-03-11T21:08:43.376 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:08:43.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:43.380 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:08:43.481 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:43.490 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:43 smithi146 bash[23809]: cluster 2024-03-11T21:08:42.219825+0000 osd.4 (osd.4) 107 : cluster 0 purged_snaps scrub starts 2024-03-11T21:08:43.490 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:43 smithi146 bash[23809]: cluster 2024-03-11T21:08:42.219910+0000 osd.4 (osd.4) 108 : cluster 0 purged_snaps scrub ok 2024-03-11T21:08:43.490 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:43 smithi146 bash[23809]: cluster 2024-03-11T21:08:42.278003+0000 mgr.y (mgr.24380) 663 : cluster 0 pgmap v744: 148 pgs: 148 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 61 MiB/s wr, 982 op/s 2024-03-11T21:08:43.490 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:43 smithi146 bash[23809]: cluster 2024-03-11T21:08:42.352811+0000 mon.a (mon.0) 1663 : cluster 0 osdmap e329: 8 total, 8 up, 8 in 2024-03-11T21:08:43.507 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:43.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:43.510 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:08:43.611 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:43.621 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:43 smithi023 bash[29146]: cluster 2024-03-11T21:08:42.219825+0000 osd.4 (osd.4) 107 : cluster 0 purged_snaps scrub starts 2024-03-11T21:08:43.621 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:43 smithi023 bash[29146]: cluster 2024-03-11T21:08:42.219910+0000 osd.4 (osd.4) 108 : cluster 0 purged_snaps scrub ok 2024-03-11T21:08:43.621 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:43 smithi023 bash[29146]: cluster 2024-03-11T21:08:42.278003+0000 mgr.y (mgr.24380) 663 : cluster 0 pgmap v744: 148 pgs: 148 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 61 MiB/s wr, 982 op/s 2024-03-11T21:08:43.621 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:43 smithi023 bash[29146]: cluster 2024-03-11T21:08:42.352811+0000 mon.a (mon.0) 1663 : cluster 0 osdmap e329: 8 total, 8 up, 8 in 2024-03-11T21:08:43.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:43.650 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:43.650 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:08:43.751 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:43.777 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:43.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:43.780 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:08:43.881 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:08:43.905 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:08:43.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:43.909 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:08:44.010 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:08:44.042 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:08:44.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:44.047 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:08:44.148 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:08:44.171 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:08:44.180 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:08:44.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:44.189 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:08:44.290 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:08:44.314 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:08:44.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:44.320 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:08:44.372 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 68 16 88797 88781 81.5902 64.6875 0.0178031 0.0122512 2024-03-11T21:08:44.421 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:08:44.447 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:08:44.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:44.449 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:08:44.550 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:44.576 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:44.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:44.579 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:08:44.681 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:44.713 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:44.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:44.715 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:08:44.816 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:44.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:44.847 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:44.847 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:08:44.948 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:44.977 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:44.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:44.981 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:08:45.081 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:08:45.106 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:08:45.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:45.109 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:08:45.210 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:08:45.240 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:08:45.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:45.242 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:08:45.343 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:08:45.371 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:08:45.372 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 69 16 89827 89811 81.3406 64.375 0.0177119 0.012288 2024-03-11T21:08:45.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:45.377 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:08:45.478 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:08:45.505 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:08:45.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:45.506 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:08:45.607 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:08:45.617 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:45 smithi146 bash[23809]: cluster 2024-03-11T21:08:44.278379+0000 mgr.y (mgr.24380) 664 : cluster 0 pgmap v746: 148 pgs: 148 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 569 B/s rd, 40 MiB/s wr, 640 op/s 2024-03-11T21:08:45.617 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:45 smithi146 bash[23809]: cluster 2024-03-11T21:08:45.299238+0000 mon.a (mon.0) 1664 : cluster 0 osdmap e330: 8 total, 8 up, 8 in 2024-03-11T21:08:45.637 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:08:45.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:45.640 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:08:45.741 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:45.748 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:08:45 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:08:45.759 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:45.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:45.762 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:08:45.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:45 smithi023 bash[21372]: cluster 2024-03-11T21:08:44.278379+0000 mgr.y (mgr.24380) 664 : cluster 0 pgmap v746: 148 pgs: 148 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 569 B/s rd, 40 MiB/s wr, 640 op/s 2024-03-11T21:08:45.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:45 smithi023 bash[21372]: cluster 2024-03-11T21:08:45.299238+0000 mon.a (mon.0) 1664 : cluster 0 osdmap e330: 8 total, 8 up, 8 in 2024-03-11T21:08:45.779 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:45 smithi023 bash[29146]: cluster 2024-03-11T21:08:44.278379+0000 mgr.y (mgr.24380) 664 : cluster 0 pgmap v746: 148 pgs: 148 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 569 B/s rd, 40 MiB/s wr, 640 op/s 2024-03-11T21:08:45.779 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:45 smithi023 bash[29146]: cluster 2024-03-11T21:08:45.299238+0000 mon.a (mon.0) 1664 : cluster 0 osdmap e330: 8 total, 8 up, 8 in 2024-03-11T21:08:45.863 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:08:45.887 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:08:45.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:45.892 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:08:45.993 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:08:46.016 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:08:46.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:46.020 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:08:46.120 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:46.139 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:46.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:46.140 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:08:46.241 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:08:46.276 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:08:46.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:46.279 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:08:46.372 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 70 16 90780 90764 81.0294 59.5625 0.023746 0.0123357 2024-03-11T21:08:46.380 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:08:46.408 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:08:46.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:46.411 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:08:46.512 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:08:46.534 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:08:46.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:46.536 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:08:46.637 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:08:46.669 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:08:46.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:46.676 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:08:46.777 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:08:46.802 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:08:46.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:46.808 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:08:46.909 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:08:46.930 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:08:46.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:46.932 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:08:47.033 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:08:47.061 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:08:47.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:47.065 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:08:47.166 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:47.198 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:47.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:47.204 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:08:47.304 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:08:47.327 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:08:47.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:47.330 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:08:47.373 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 71 16 91723 91707 80.7182 58.9375 0.0150442 0.0123837 2024-03-11T21:08:47.431 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:08:47.453 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:08:47.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:47.456 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:08:47.557 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:08:47.591 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:08:47.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:47.598 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:08:47.629 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:47 smithi023 bash[21372]: audit 2024-03-11T21:08:45.741212+0000 mgr.y (mgr.24380) 665 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:08:47.629 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:47 smithi023 bash[21372]: cluster 2024-03-11T21:08:45.838006+0000 osd.6 (osd.6) 284 : cluster 0 purged_snaps scrub starts 2024-03-11T21:08:47.629 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:47 smithi023 bash[21372]: cluster 2024-03-11T21:08:45.838069+0000 osd.6 (osd.6) 285 : cluster 0 purged_snaps scrub ok 2024-03-11T21:08:47.630 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:47 smithi023 bash[21372]: cluster 2024-03-11T21:08:46.279213+0000 mgr.y (mgr.24380) 666 : cluster 0 pgmap v748: 148 pgs: 148 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 68 MiB/s wr, 1.09k op/s 2024-03-11T21:08:47.630 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:47 smithi023 bash[21372]: cluster 2024-03-11T21:08:46.367631+0000 mon.a (mon.0) 1665 : cluster 0 osdmap e331: 8 total, 8 up, 8 in 2024-03-11T21:08:47.634 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:47 smithi023 bash[29146]: audit 2024-03-11T21:08:45.741212+0000 mgr.y (mgr.24380) 665 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:08:47.634 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:47 smithi023 bash[29146]: cluster 2024-03-11T21:08:45.838006+0000 osd.6 (osd.6) 284 : cluster 0 purged_snaps scrub starts 2024-03-11T21:08:47.634 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:47 smithi023 bash[29146]: cluster 2024-03-11T21:08:45.838069+0000 osd.6 (osd.6) 285 : cluster 0 purged_snaps scrub ok 2024-03-11T21:08:47.634 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:47 smithi023 bash[29146]: cluster 2024-03-11T21:08:46.279213+0000 mgr.y (mgr.24380) 666 : cluster 0 pgmap v748: 148 pgs: 148 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 68 MiB/s wr, 1.09k op/s 2024-03-11T21:08:47.634 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:47 smithi023 bash[29146]: cluster 2024-03-11T21:08:46.367631+0000 mon.a (mon.0) 1665 : cluster 0 osdmap e331: 8 total, 8 up, 8 in 2024-03-11T21:08:47.698 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:47.707 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:47 smithi146 bash[23809]: audit 2024-03-11T21:08:45.741212+0000 mgr.y (mgr.24380) 665 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:08:47.708 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:47 smithi146 bash[23809]: cluster 2024-03-11T21:08:45.838006+0000 osd.6 (osd.6) 284 : cluster 0 purged_snaps scrub starts 2024-03-11T21:08:47.708 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:47 smithi146 bash[23809]: cluster 2024-03-11T21:08:45.838069+0000 osd.6 (osd.6) 285 : cluster 0 purged_snaps scrub ok 2024-03-11T21:08:47.708 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:47 smithi146 bash[23809]: cluster 2024-03-11T21:08:46.279213+0000 mgr.y (mgr.24380) 666 : cluster 0 pgmap v748: 148 pgs: 148 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 68 MiB/s wr, 1.09k op/s 2024-03-11T21:08:47.708 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:47 smithi146 bash[23809]: cluster 2024-03-11T21:08:46.367631+0000 mon.a (mon.0) 1665 : cluster 0 osdmap e331: 8 total, 8 up, 8 in 2024-03-11T21:08:47.721 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:47.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:47.725 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:08:47.825 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:47.860 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:47.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:47.864 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:08:47.965 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:08:48.002 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:08:48.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:48.004 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:08:48.104 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:08:48.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:48.136 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:08:48.136 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:08:48.237 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:08:48.264 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:08:48.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:48.269 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:08:48.370 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:48.372 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 72 16 92728 92712 80.4695 62.8125 0.0130684 0.0124211 2024-03-11T21:08:48.399 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:48.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:48.401 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:08:48.502 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:08:48.512 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:08:48 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:08:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:08:48.541 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:08:48.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:48.544 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:08:48.644 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:08:48.664 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:08:48.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:48.666 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:08:48.767 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:48.800 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:48.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:48.804 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:08:48.905 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:08:48.927 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:08:48.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:48.929 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:08:49.030 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:08:49.073 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:08:49.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:49.078 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:08:49.179 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:08:49.211 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:08:49.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:49.213 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:08:49.314 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:49.336 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:49.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:49.338 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:08:49.373 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 73 16 93266 93250 79.8277 33.625 0.0104061 0.0125055 2024-03-11T21:08:49.438 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:49.465 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:49.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:49.469 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:08:49.569 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:49.599 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:49.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:49.605 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:08:49.706 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:49.713 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:49 smithi146 bash[23809]: cluster 2024-03-11T21:08:48.279696+0000 mgr.y (mgr.24380) 667 : cluster 0 pgmap v750: 148 pgs: 148 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 48 MiB/s wr, 763 op/s 2024-03-11T21:08:49.728 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:49.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:49.731 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:08:49.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:49 smithi023 bash[21372]: cluster 2024-03-11T21:08:48.279696+0000 mgr.y (mgr.24380) 667 : cluster 0 pgmap v750: 148 pgs: 148 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 48 MiB/s wr, 763 op/s 2024-03-11T21:08:49.779 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:49 smithi023 bash[29146]: cluster 2024-03-11T21:08:48.279696+0000 mgr.y (mgr.24380) 667 : cluster 0 pgmap v750: 148 pgs: 148 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 48 MiB/s wr, 763 op/s 2024-03-11T21:08:49.832 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:08:49.857 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:08:49.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:49.865 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:08:49.965 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:50.002 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:50.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:50.009 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:08:50.110 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:50.132 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:50.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:50.136 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:08:50.194 INFO:teuthology.orchestra.run.smithi023.stdout:osd.0: {} 2024-03-11T21:08:50.237 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:50.244 INFO:teuthology.orchestra.run.smithi023.stdout:osd.1: {} 2024-03-11T21:08:50.279 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:50.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:50.284 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:08:50.285 INFO:teuthology.orchestra.run.smithi023.stdout:osd.2: {} 2024-03-11T21:08:50.325 INFO:teuthology.orchestra.run.smithi023.stdout:ERROR: (22) Invalid argument 2024-03-11T21:08:50.326 INFO:teuthology.orchestra.run.smithi023.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:08:50.361 INFO:teuthology.orchestra.run.smithi023.stdout:osd.3: {} 2024-03-11T21:08:50.373 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 74 15 94136 94121 79.4845 54.4375 0.00847216 0.0125761 2024-03-11T21:08:50.385 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:08:50.394 INFO:teuthology.orchestra.run.smithi023.stdout:osd.4: {} 2024-03-11T21:08:50.403 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:08:50.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:50.405 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:08:50.415 INFO:teuthology.orchestra.run.smithi023.stdout:osd.5: {} 2024-03-11T21:08:50.436 INFO:teuthology.orchestra.run.smithi023.stdout:osd.6: {} 2024-03-11T21:08:50.456 INFO:teuthology.orchestra.run.smithi023.stdout:osd.7: {} 2024-03-11T21:08:50.457 INFO:teuthology.orchestra.run.smithi023.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:08:50.457 INFO:teuthology.orchestra.run.smithi023.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:08:50.457 INFO:teuthology.orchestra.run.smithi023.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:08:50.457 INFO:teuthology.orchestra.run.smithi023.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:08:50.457 INFO:teuthology.orchestra.run.smithi023.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:08:50.457 INFO:teuthology.orchestra.run.smithi023.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:08:50.457 INFO:teuthology.orchestra.run.smithi023.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:08:50.458 INFO:teuthology.orchestra.run.smithi023.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:08:50.506 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:08:50.534 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:08:50.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:50.539 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:08:50.640 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:50.664 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:50.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:50.667 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:08:50.673 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:50 smithi023 bash[21372]: audit 2024-03-11T21:08:50.125083+0000 mon.c (mon.2) 83 : audit 0 from='client.? 172.21.15.23:0/2946715645' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:08:50.673 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:50 smithi023 bash[21372]: cluster 2024-03-11T21:08:50.308671+0000 mon.a (mon.0) 1666 : cluster 0 osdmap e332: 8 total, 8 up, 8 in 2024-03-11T21:08:50.674 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:50 smithi023 bash[29146]: audit 2024-03-11T21:08:50.125083+0000 mon.c (mon.2) 83 : audit 0 from='client.? 172.21.15.23:0/2946715645' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:08:50.674 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:50 smithi023 bash[29146]: cluster 2024-03-11T21:08:50.308671+0000 mon.a (mon.0) 1666 : cluster 0 osdmap e332: 8 total, 8 up, 8 in 2024-03-11T21:08:50.758 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:50 smithi146 bash[23809]: audit 2024-03-11T21:08:50.125083+0000 mon.c (mon.2) 83 : audit 0 from='client.? 172.21.15.23:0/2946715645' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:08:50.759 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:50 smithi146 bash[23809]: cluster 2024-03-11T21:08:50.308671+0000 mon.a (mon.0) 1666 : cluster 0 osdmap e332: 8 total, 8 up, 8 in 2024-03-11T21:08:50.768 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:08:50.793 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:08:50.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:50.797 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:08:50.898 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:50.952 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:50.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:50.956 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:08:51.057 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:08:51.088 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:08:51.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:51.096 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:08:51.197 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:51.218 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:51.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:51.220 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:08:51.321 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:51.349 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:51.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:51.353 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:08:51.373 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 75 16 95016 95000 79.1572 54.9375 0.00943037 0.0126285 2024-03-11T21:08:51.412 INFO:teuthology.orchestra.run.smithi023.stderr:nodeep-scrub is set 2024-03-11T21:08:51.454 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:08:51.492 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:08:51.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:51.495 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:08:51.509 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:51 smithi146 bash[23809]: cluster 2024-03-11T21:08:50.284852+0000 mgr.y (mgr.24380) 668 : cluster 0 pgmap v751: 148 pgs: 148 active+clean; 5.7 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.3 KiB/s rd, 62 MiB/s wr, 995 op/s 2024-03-11T21:08:51.509 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:51 smithi146 bash[23809]: audit 2024-03-11T21:08:50.775037+0000 mon.a (mon.0) 1667 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:08:51.509 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:51 smithi146 bash[23809]: audit 2024-03-11T21:08:50.959497+0000 mon.a (mon.0) 1668 : audit 1 from='client.? 172.21.15.23:0/2087692487' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:08:51.596 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:08:51.624 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:08:51.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:51.630 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:08:51.731 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:51.767 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:51.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:51.772 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:08:51.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:51 smithi023 bash[21372]: cluster 2024-03-11T21:08:50.284852+0000 mgr.y (mgr.24380) 668 : cluster 0 pgmap v751: 148 pgs: 148 active+clean; 5.7 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.3 KiB/s rd, 62 MiB/s wr, 995 op/s 2024-03-11T21:08:51.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:51 smithi023 bash[21372]: audit 2024-03-11T21:08:50.775037+0000 mon.a (mon.0) 1667 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:08:51.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:51 smithi023 bash[21372]: audit 2024-03-11T21:08:50.959497+0000 mon.a (mon.0) 1668 : audit 1 from='client.? 172.21.15.23:0/2087692487' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:08:51.779 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:51 smithi023 bash[29146]: cluster 2024-03-11T21:08:50.284852+0000 mgr.y (mgr.24380) 668 : cluster 0 pgmap v751: 148 pgs: 148 active+clean; 5.7 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.3 KiB/s rd, 62 MiB/s wr, 995 op/s 2024-03-11T21:08:51.779 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:51 smithi023 bash[29146]: audit 2024-03-11T21:08:50.775037+0000 mon.a (mon.0) 1667 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:08:51.779 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:51 smithi023 bash[29146]: audit 2024-03-11T21:08:50.959497+0000 mon.a (mon.0) 1668 : audit 1 from='client.? 172.21.15.23:0/2087692487' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:08:51.873 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:08:51.897 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:08:51.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:51.902 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:08:52.002 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:52.034 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:52.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:52.036 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:08:52.137 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:08:52.154 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:08:52.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:52.156 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:08:52.256 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:08:52.283 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:08:52.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:52.286 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:08:52.373 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 76 16 95929 95913 78.8664 57.0625 0.0100709 0.0126749 2024-03-11T21:08:52.386 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:52.417 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:52.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:52.421 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:08:52.521 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:08:52.541 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:08:52.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:52.543 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:08:52.643 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:08:52.653 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:52 smithi146 bash[23809]: cluster 2024-03-11T21:08:51.393680+0000 mon.a (mon.0) 1669 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:08:52.654 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:52 smithi146 bash[23809]: audit 2024-03-11T21:08:51.415047+0000 mon.a (mon.0) 1670 : audit 1 from='client.? 172.21.15.23:0/2087692487' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:08:52.654 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:52 smithi146 bash[23809]: cluster 2024-03-11T21:08:51.427287+0000 mon.a (mon.0) 1671 : cluster 0 osdmap e333: 8 total, 8 up, 8 in 2024-03-11T21:08:52.660 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:52 smithi023 bash[21372]: cluster 2024-03-11T21:08:51.393680+0000 mon.a (mon.0) 1669 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:08:52.660 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:52 smithi023 bash[21372]: audit 2024-03-11T21:08:51.415047+0000 mon.a (mon.0) 1670 : audit 1 from='client.? 172.21.15.23:0/2087692487' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:08:52.660 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:52 smithi023 bash[21372]: cluster 2024-03-11T21:08:51.427287+0000 mon.a (mon.0) 1671 : cluster 0 osdmap e333: 8 total, 8 up, 8 in 2024-03-11T21:08:52.661 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:52 smithi023 bash[29146]: cluster 2024-03-11T21:08:51.393680+0000 mon.a (mon.0) 1669 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:08:52.661 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:52 smithi023 bash[29146]: audit 2024-03-11T21:08:51.415047+0000 mon.a (mon.0) 1670 : audit 1 from='client.? 172.21.15.23:0/2087692487' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:08:52.661 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:52 smithi023 bash[29146]: cluster 2024-03-11T21:08:51.427287+0000 mon.a (mon.0) 1671 : cluster 0 osdmap e333: 8 total, 8 up, 8 in 2024-03-11T21:08:52.668 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:08:52.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:52.671 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:08:52.771 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:08:52.807 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:08:52.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:52.812 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:08:52.912 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:08:52.945 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:08:52.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:52.948 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:08:53.049 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:53.080 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:53.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:53.085 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:08:53.186 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:08:53.212 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:08:53.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:53.220 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:08:53.321 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:08:53.354 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:08:53.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:53.357 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:08:53.373 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 77 16 96719 96703 78.4833 49.375 0.0076521 0.0127368 2024-03-11T21:08:53.457 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:53.483 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:53.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:53.490 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:08:53.529 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:53 smithi023 bash[21372]: cluster 2024-03-11T21:08:52.285390+0000 mgr.y (mgr.24380) 669 : cluster 0 pgmap v754: 148 pgs: 148 active+clean; 5.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 852 B/s rd, 49 MiB/s wr, 778 op/s 2024-03-11T21:08:53.529 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:53 smithi023 bash[29146]: cluster 2024-03-11T21:08:52.285390+0000 mgr.y (mgr.24380) 669 : cluster 0 pgmap v754: 148 pgs: 148 active+clean; 5.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 852 B/s rd, 49 MiB/s wr, 778 op/s 2024-03-11T21:08:53.590 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:08:53.618 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:08:53.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:53.624 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:08:53.725 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:08:53.758 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:53 smithi146 bash[23809]: cluster 2024-03-11T21:08:52.285390+0000 mgr.y (mgr.24380) 669 : cluster 0 pgmap v754: 148 pgs: 148 active+clean; 5.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 852 B/s rd, 49 MiB/s wr, 778 op/s 2024-03-11T21:08:53.759 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:08:53.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:53.763 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:08:53.864 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:53.890 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:53.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:53.893 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:08:53.994 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:08:54.021 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:08:54.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:54.024 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:08:54.125 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:08:54.152 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:08:54.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:54.155 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:08:54.256 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:08:54.283 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:08:54.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:54.286 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:08:54.373 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 78 16 97660 97644 78.2311 58.8125 0.00634761 0.0127783 2024-03-11T21:08:54.387 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:08:54.414 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:08:54.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:54.416 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:08:54.517 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:08:54.540 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:08:54.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:54.542 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:08:54.643 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:54.668 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:54.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:54.670 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:08:54.770 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:08:54.803 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:08:54.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:54.807 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:08:54.908 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:54.933 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:54.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:54.938 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:08:55.039 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:55.068 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:55.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:55.071 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:08:55.084 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-03-11T21:08:55.172 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:55.194 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:55.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:55.198 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:08:55.299 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:55.334 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:55.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:55.339 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:08:55.376 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 79 16 98629 98613 78.004 60.5625 0.00913904 0.0128138 2024-03-11T21:08:55.440 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:55.464 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:55.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:55.467 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:08:55.567 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:08:55.601 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:08:55.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:55.604 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:08:55.605 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:55 smithi023 bash[21372]: cluster 2024-03-11T21:08:54.285920+0000 mgr.y (mgr.24380) 670 : cluster 0 pgmap v755: 148 pgs: 148 active+clean; 5.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 646 B/s rd, 37 MiB/s wr, 590 op/s 2024-03-11T21:08:55.605 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:55 smithi023 bash[21372]: cluster 2024-03-11T21:08:55.305688+0000 mon.a (mon.0) 1672 : cluster 0 osdmap e334: 8 total, 8 up, 8 in 2024-03-11T21:08:55.605 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:55 smithi023 bash[29146]: cluster 2024-03-11T21:08:54.285920+0000 mgr.y (mgr.24380) 670 : cluster 0 pgmap v755: 148 pgs: 148 active+clean; 5.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 646 B/s rd, 37 MiB/s wr, 590 op/s 2024-03-11T21:08:55.605 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:55 smithi023 bash[29146]: cluster 2024-03-11T21:08:55.305688+0000 mon.a (mon.0) 1672 : cluster 0 osdmap e334: 8 total, 8 up, 8 in 2024-03-11T21:08:55.705 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:55.730 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:55.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:55.734 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:08:55.742 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:55 smithi146 bash[23809]: cluster 2024-03-11T21:08:54.285920+0000 mgr.y (mgr.24380) 670 : cluster 0 pgmap v755: 148 pgs: 148 active+clean; 5.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 646 B/s rd, 37 MiB/s wr, 590 op/s 2024-03-11T21:08:55.742 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:55 smithi146 bash[23809]: cluster 2024-03-11T21:08:55.305688+0000 mon.a (mon.0) 1672 : cluster 0 osdmap e334: 8 total, 8 up, 8 in 2024-03-11T21:08:55.835 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:55.862 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:55.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:55.864 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:08:55.965 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:08:55.996 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:08:56.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:56.001 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:08:56.001 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:08:55 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:08:56.064 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-11T21:08:56.065 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-11T21:08:56.065 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 334, 2024-03-11T21:08:56.065 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "67ba00f8-dfe7-11ee-95c8-87774f69a715", 2024-03-11T21:08:56.065 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2024-03-11T20:40:13.664802+0000", 2024-03-11T21:08:56.065 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2024-03-11T21:08:55.295434+0000", 2024-03-11T21:08:56.065 INFO:teuthology.orchestra.run.smithi023.stdout: "last_up_change": "2024-03-11T21:00:47.324817+0000", 2024-03-11T21:08:56.065 INFO:teuthology.orchestra.run.smithi023.stdout: "last_in_change": "2024-03-11T20:57:23.560527+0000", 2024-03-11T21:08:56.065 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-03-11T21:08:56.065 INFO:teuthology.orchestra.run.smithi023.stdout: "flags_num": 5806080, 2024-03-11T21:08:56.065 INFO:teuthology.orchestra.run.smithi023.stdout: "flags_set": [ 2024-03-11T21:08:56.065 INFO:teuthology.orchestra.run.smithi023.stdout: "nodeep-scrub", 2024-03-11T21:08:56.066 INFO:teuthology.orchestra.run.smithi023.stdout: "noscrub", 2024-03-11T21:08:56.066 INFO:teuthology.orchestra.run.smithi023.stdout: "pglog_hardlimit", 2024-03-11T21:08:56.066 INFO:teuthology.orchestra.run.smithi023.stdout: "purged_snapdirs", 2024-03-11T21:08:56.066 INFO:teuthology.orchestra.run.smithi023.stdout: "recovery_deletes", 2024-03-11T21:08:56.066 INFO:teuthology.orchestra.run.smithi023.stdout: "sortbitwise" 2024-03-11T21:08:56.066 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-11T21:08:56.066 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_version": 20, 2024-03-11T21:08:56.066 INFO:teuthology.orchestra.run.smithi023.stdout: "full_ratio": 0.94999998807907104, 2024-03-11T21:08:56.066 INFO:teuthology.orchestra.run.smithi023.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-03-11T21:08:56.066 INFO:teuthology.orchestra.run.smithi023.stdout: "nearfull_ratio": 0.85000002384185791, 2024-03-11T21:08:56.066 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_snapshot": "", 2024-03-11T21:08:56.066 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_max": 10, 2024-03-11T21:08:56.066 INFO:teuthology.orchestra.run.smithi023.stdout: "max_osd": 8, 2024-03-11T21:08:56.067 INFO:teuthology.orchestra.run.smithi023.stdout: "require_min_compat_client": "luminous", 2024-03-11T21:08:56.067 INFO:teuthology.orchestra.run.smithi023.stdout: "min_compat_client": "luminous", 2024-03-11T21:08:56.067 INFO:teuthology.orchestra.run.smithi023.stdout: "require_osd_release": "squid", 2024-03-11T21:08:56.067 INFO:teuthology.orchestra.run.smithi023.stdout: "allow_crimson": false, 2024-03-11T21:08:56.067 INFO:teuthology.orchestra.run.smithi023.stdout: "pools": [ 2024-03-11T21:08:56.067 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:56.067 INFO:teuthology.orchestra.run.smithi023.stdout: "pool": 1, 2024-03-11T21:08:56.067 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_name": ".mgr", 2024-03-11T21:08:56.067 INFO:teuthology.orchestra.run.smithi023.stdout: "create_time": "2024-03-11T20:45:31.964570+0000", 2024-03-11T21:08:56.067 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-11T21:08:56.067 INFO:teuthology.orchestra.run.smithi023.stdout: "flags_names": "hashpspool", 2024-03-11T21:08:56.067 INFO:teuthology.orchestra.run.smithi023.stdout: "type": 1, 2024-03-11T21:08:56.067 INFO:teuthology.orchestra.run.smithi023.stdout: "size": 3, 2024-03-11T21:08:56.067 INFO:teuthology.orchestra.run.smithi023.stdout: "min_size": 2, 2024-03-11T21:08:56.068 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_rule": 0, 2024-03-11T21:08:56.068 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_count": 0, 2024-03-11T21:08:56.068 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_target": 0, 2024-03-11T21:08:56.068 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T21:08:56.068 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T21:08:56.068 INFO:teuthology.orchestra.run.smithi023.stdout: "object_hash": 2, 2024-03-11T21:08:56.068 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_autoscale_mode": "off", 2024-03-11T21:08:56.068 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num": 1, 2024-03-11T21:08:56.068 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_placement_num": 1, 2024-03-11T21:08:56.068 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_placement_num_target": 1, 2024-03-11T21:08:56.068 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num_target": 1, 2024-03-11T21:08:56.068 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num_pending": 1, 2024-03-11T21:08:56.069 INFO:teuthology.orchestra.run.smithi023.stdout: "last_pg_merge_meta": { 2024-03-11T21:08:56.069 INFO:teuthology.orchestra.run.smithi023.stdout: "source_pgid": "0.0", 2024-03-11T21:08:56.069 INFO:teuthology.orchestra.run.smithi023.stdout: "ready_epoch": 0, 2024-03-11T21:08:56.069 INFO:teuthology.orchestra.run.smithi023.stdout: "last_epoch_started": 0, 2024-03-11T21:08:56.069 INFO:teuthology.orchestra.run.smithi023.stdout: "last_epoch_clean": 0, 2024-03-11T21:08:56.069 INFO:teuthology.orchestra.run.smithi023.stdout: "source_version": "0'0", 2024-03-11T21:08:56.069 INFO:teuthology.orchestra.run.smithi023.stdout: "target_version": "0'0" 2024-03-11T21:08:56.069 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:56.070 INFO:teuthology.orchestra.run.smithi023.stdout: "last_change": "142", 2024-03-11T21:08:56.070 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend": "0", 2024-03-11T21:08:56.070 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T21:08:56.070 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T21:08:56.070 INFO:teuthology.orchestra.run.smithi023.stdout: "auid": 0, 2024-03-11T21:08:56.070 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_mode": "selfmanaged", 2024-03-11T21:08:56.070 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_seq": 0, 2024-03-11T21:08:56.070 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_epoch": 0, 2024-03-11T21:08:56.070 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_snaps": [], 2024-03-11T21:08:56.070 INFO:teuthology.orchestra.run.smithi023.stdout: "removed_snaps": "[]", 2024-03-11T21:08:56.071 INFO:teuthology.orchestra.run.smithi023.stdout: "quota_max_bytes": 0, 2024-03-11T21:08:56.071 INFO:teuthology.orchestra.run.smithi023.stdout: "quota_max_objects": 0, 2024-03-11T21:08:56.071 INFO:teuthology.orchestra.run.smithi023.stdout: "tiers": [], 2024-03-11T21:08:56.071 INFO:teuthology.orchestra.run.smithi023.stdout: "tier_of": -1, 2024-03-11T21:08:56.071 INFO:teuthology.orchestra.run.smithi023.stdout: "read_tier": -1, 2024-03-11T21:08:56.071 INFO:teuthology.orchestra.run.smithi023.stdout: "write_tier": -1, 2024-03-11T21:08:56.071 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_mode": "none", 2024-03-11T21:08:56.071 INFO:teuthology.orchestra.run.smithi023.stdout: "target_max_bytes": 0, 2024-03-11T21:08:56.071 INFO:teuthology.orchestra.run.smithi023.stdout: "target_max_objects": 0, 2024-03-11T21:08:56.071 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T21:08:56.071 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T21:08:56.071 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T21:08:56.071 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_min_flush_age": 0, 2024-03-11T21:08:56.071 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_min_evict_age": 0, 2024-03-11T21:08:56.072 INFO:teuthology.orchestra.run.smithi023.stdout: "erasure_code_profile": "", 2024-03-11T21:08:56.072 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_params": { 2024-03-11T21:08:56.072 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "none" 2024-03-11T21:08:56.072 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:56.072 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_period": 0, 2024-03-11T21:08:56.072 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_count": 0, 2024-03-11T21:08:56.072 INFO:teuthology.orchestra.run.smithi023.stdout: "use_gmt_hitset": true, 2024-03-11T21:08:56.072 INFO:teuthology.orchestra.run.smithi023.stdout: "min_read_recency_for_promote": 0, 2024-03-11T21:08:56.072 INFO:teuthology.orchestra.run.smithi023.stdout: "min_write_recency_for_promote": 0, 2024-03-11T21:08:56.072 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T21:08:56.072 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_search_last_n": 0, 2024-03-11T21:08:56.072 INFO:teuthology.orchestra.run.smithi023.stdout: "grade_table": [], 2024-03-11T21:08:56.072 INFO:teuthology.orchestra.run.smithi023.stdout: "stripe_width": 0, 2024-03-11T21:08:56.072 INFO:teuthology.orchestra.run.smithi023.stdout: "expected_num_objects": 0, 2024-03-11T21:08:56.073 INFO:teuthology.orchestra.run.smithi023.stdout: "fast_read": false, 2024-03-11T21:08:56.073 INFO:teuthology.orchestra.run.smithi023.stdout: "options": { 2024-03-11T21:08:56.073 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num_max": 32, 2024-03-11T21:08:56.073 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num_min": 1 2024-03-11T21:08:56.073 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:56.073 INFO:teuthology.orchestra.run.smithi023.stdout: "application_metadata": { 2024-03-11T21:08:56.073 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr": {} 2024-03-11T21:08:56.073 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:56.073 INFO:teuthology.orchestra.run.smithi023.stdout: "read_balance": { 2024-03-11T21:08:56.073 INFO:teuthology.orchestra.run.smithi023.stdout: "score_type": "Fair distribution", 2024-03-11T21:08:56.073 INFO:teuthology.orchestra.run.smithi023.stdout: "score_acting": 6, 2024-03-11T21:08:56.073 INFO:teuthology.orchestra.run.smithi023.stdout: "score_stable": 6, 2024-03-11T21:08:56.073 INFO:teuthology.orchestra.run.smithi023.stdout: "optimal_score": 0.5, 2024-03-11T21:08:56.074 INFO:teuthology.orchestra.run.smithi023.stdout: "raw_score_acting": 3, 2024-03-11T21:08:56.074 INFO:teuthology.orchestra.run.smithi023.stdout: "raw_score_stable": 3, 2024-03-11T21:08:56.074 INFO:teuthology.orchestra.run.smithi023.stdout: "primary_affinity_weighted": 0.75, 2024-03-11T21:08:56.074 INFO:teuthology.orchestra.run.smithi023.stdout: "average_primary_affinity": 1, 2024-03-11T21:08:56.074 INFO:teuthology.orchestra.run.smithi023.stdout: "average_primary_affinity_weighted": 0.75 2024-03-11T21:08:56.074 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:08:56.074 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:56.074 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:56.074 INFO:teuthology.orchestra.run.smithi023.stdout: "pool": 2, 2024-03-11T21:08:56.074 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_name": ".rgw.root", 2024-03-11T21:08:56.074 INFO:teuthology.orchestra.run.smithi023.stdout: "create_time": "2024-03-11T20:50:26.421486+0000", 2024-03-11T21:08:56.075 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-11T21:08:56.075 INFO:teuthology.orchestra.run.smithi023.stdout: "flags_names": "hashpspool", 2024-03-11T21:08:56.075 INFO:teuthology.orchestra.run.smithi023.stdout: "type": 1, 2024-03-11T21:08:56.075 INFO:teuthology.orchestra.run.smithi023.stdout: "size": 3, 2024-03-11T21:08:56.075 INFO:teuthology.orchestra.run.smithi023.stdout: "min_size": 2, 2024-03-11T21:08:56.075 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_rule": 0, 2024-03-11T21:08:56.075 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_count": 0, 2024-03-11T21:08:56.075 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_target": 0, 2024-03-11T21:08:56.075 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T21:08:56.075 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T21:08:56.075 INFO:teuthology.orchestra.run.smithi023.stdout: "object_hash": 2, 2024-03-11T21:08:56.075 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_autoscale_mode": "off", 2024-03-11T21:08:56.075 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num": 32, 2024-03-11T21:08:56.076 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_placement_num": 32, 2024-03-11T21:08:56.076 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_placement_num_target": 32, 2024-03-11T21:08:56.076 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num_target": 32, 2024-03-11T21:08:56.076 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num_pending": 32, 2024-03-11T21:08:56.076 INFO:teuthology.orchestra.run.smithi023.stdout: "last_pg_merge_meta": { 2024-03-11T21:08:56.076 INFO:teuthology.orchestra.run.smithi023.stdout: "source_pgid": "0.0", 2024-03-11T21:08:56.076 INFO:teuthology.orchestra.run.smithi023.stdout: "ready_epoch": 0, 2024-03-11T21:08:56.076 INFO:teuthology.orchestra.run.smithi023.stdout: "last_epoch_started": 0, 2024-03-11T21:08:56.076 INFO:teuthology.orchestra.run.smithi023.stdout: "last_epoch_clean": 0, 2024-03-11T21:08:56.076 INFO:teuthology.orchestra.run.smithi023.stdout: "source_version": "0'0", 2024-03-11T21:08:56.076 INFO:teuthology.orchestra.run.smithi023.stdout: "target_version": "0'0" 2024-03-11T21:08:56.076 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:56.076 INFO:teuthology.orchestra.run.smithi023.stdout: "last_change": "78", 2024-03-11T21:08:56.076 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend": "0", 2024-03-11T21:08:56.077 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T21:08:56.077 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T21:08:56.077 INFO:teuthology.orchestra.run.smithi023.stdout: "auid": 0, 2024-03-11T21:08:56.077 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_mode": "selfmanaged", 2024-03-11T21:08:56.077 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_seq": 0, 2024-03-11T21:08:56.077 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_epoch": 0, 2024-03-11T21:08:56.077 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_snaps": [], 2024-03-11T21:08:56.077 INFO:teuthology.orchestra.run.smithi023.stdout: "removed_snaps": "[]", 2024-03-11T21:08:56.077 INFO:teuthology.orchestra.run.smithi023.stdout: "quota_max_bytes": 0, 2024-03-11T21:08:56.077 INFO:teuthology.orchestra.run.smithi023.stdout: "quota_max_objects": 0, 2024-03-11T21:08:56.077 INFO:teuthology.orchestra.run.smithi023.stdout: "tiers": [], 2024-03-11T21:08:56.077 INFO:teuthology.orchestra.run.smithi023.stdout: "tier_of": -1, 2024-03-11T21:08:56.077 INFO:teuthology.orchestra.run.smithi023.stdout: "read_tier": -1, 2024-03-11T21:08:56.077 INFO:teuthology.orchestra.run.smithi023.stdout: "write_tier": -1, 2024-03-11T21:08:56.078 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_mode": "none", 2024-03-11T21:08:56.078 INFO:teuthology.orchestra.run.smithi023.stdout: "target_max_bytes": 0, 2024-03-11T21:08:56.078 INFO:teuthology.orchestra.run.smithi023.stdout: "target_max_objects": 0, 2024-03-11T21:08:56.078 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T21:08:56.078 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T21:08:56.078 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T21:08:56.078 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_min_flush_age": 0, 2024-03-11T21:08:56.078 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_min_evict_age": 0, 2024-03-11T21:08:56.078 INFO:teuthology.orchestra.run.smithi023.stdout: "erasure_code_profile": "", 2024-03-11T21:08:56.078 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_params": { 2024-03-11T21:08:56.078 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "none" 2024-03-11T21:08:56.078 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:56.078 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_period": 0, 2024-03-11T21:08:56.078 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_count": 0, 2024-03-11T21:08:56.079 INFO:teuthology.orchestra.run.smithi023.stdout: "use_gmt_hitset": true, 2024-03-11T21:08:56.079 INFO:teuthology.orchestra.run.smithi023.stdout: "min_read_recency_for_promote": 0, 2024-03-11T21:08:56.079 INFO:teuthology.orchestra.run.smithi023.stdout: "min_write_recency_for_promote": 0, 2024-03-11T21:08:56.079 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T21:08:56.079 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_search_last_n": 0, 2024-03-11T21:08:56.079 INFO:teuthology.orchestra.run.smithi023.stdout: "grade_table": [], 2024-03-11T21:08:56.079 INFO:teuthology.orchestra.run.smithi023.stdout: "stripe_width": 0, 2024-03-11T21:08:56.079 INFO:teuthology.orchestra.run.smithi023.stdout: "expected_num_objects": 0, 2024-03-11T21:08:56.079 INFO:teuthology.orchestra.run.smithi023.stdout: "fast_read": false, 2024-03-11T21:08:56.079 INFO:teuthology.orchestra.run.smithi023.stdout: "options": {}, 2024-03-11T21:08:56.079 INFO:teuthology.orchestra.run.smithi023.stdout: "application_metadata": { 2024-03-11T21:08:56.079 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw": {} 2024-03-11T21:08:56.079 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:56.079 INFO:teuthology.orchestra.run.smithi023.stdout: "read_balance": { 2024-03-11T21:08:56.080 INFO:teuthology.orchestra.run.smithi023.stdout: "score_type": "Fair distribution", 2024-03-11T21:08:56.080 INFO:teuthology.orchestra.run.smithi023.stdout: "score_acting": 1.5, 2024-03-11T21:08:56.080 INFO:teuthology.orchestra.run.smithi023.stdout: "score_stable": 1.5, 2024-03-11T21:08:56.080 INFO:teuthology.orchestra.run.smithi023.stdout: "optimal_score": 1.3300000429153442, 2024-03-11T21:08:56.080 INFO:teuthology.orchestra.run.smithi023.stdout: "raw_score_acting": 2, 2024-03-11T21:08:56.080 INFO:teuthology.orchestra.run.smithi023.stdout: "raw_score_stable": 2, 2024-03-11T21:08:56.080 INFO:teuthology.orchestra.run.smithi023.stdout: "primary_affinity_weighted": 0.75, 2024-03-11T21:08:56.080 INFO:teuthology.orchestra.run.smithi023.stdout: "average_primary_affinity": 1, 2024-03-11T21:08:56.081 INFO:teuthology.orchestra.run.smithi023.stdout: "average_primary_affinity_weighted": 0.75 2024-03-11T21:08:56.081 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:08:56.081 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:56.081 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:56.081 INFO:teuthology.orchestra.run.smithi023.stdout: "pool": 3, 2024-03-11T21:08:56.081 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_name": "default.rgw.log", 2024-03-11T21:08:56.081 INFO:teuthology.orchestra.run.smithi023.stdout: "create_time": "2024-03-11T20:50:28.592615+0000", 2024-03-11T21:08:56.081 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-11T21:08:56.081 INFO:teuthology.orchestra.run.smithi023.stdout: "flags_names": "hashpspool", 2024-03-11T21:08:56.081 INFO:teuthology.orchestra.run.smithi023.stdout: "type": 1, 2024-03-11T21:08:56.081 INFO:teuthology.orchestra.run.smithi023.stdout: "size": 3, 2024-03-11T21:08:56.081 INFO:teuthology.orchestra.run.smithi023.stdout: "min_size": 2, 2024-03-11T21:08:56.081 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_rule": 0, 2024-03-11T21:08:56.081 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_count": 0, 2024-03-11T21:08:56.081 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_target": 0, 2024-03-11T21:08:56.082 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T21:08:56.082 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T21:08:56.082 INFO:teuthology.orchestra.run.smithi023.stdout: "object_hash": 2, 2024-03-11T21:08:56.082 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_autoscale_mode": "off", 2024-03-11T21:08:56.082 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num": 32, 2024-03-11T21:08:56.082 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_placement_num": 32, 2024-03-11T21:08:56.082 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_placement_num_target": 32, 2024-03-11T21:08:56.082 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num_target": 32, 2024-03-11T21:08:56.082 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num_pending": 32, 2024-03-11T21:08:56.083 INFO:teuthology.orchestra.run.smithi023.stdout: "last_pg_merge_meta": { 2024-03-11T21:08:56.083 INFO:teuthology.orchestra.run.smithi023.stdout: "source_pgid": "0.0", 2024-03-11T21:08:56.083 INFO:teuthology.orchestra.run.smithi023.stdout: "ready_epoch": 0, 2024-03-11T21:08:56.083 INFO:teuthology.orchestra.run.smithi023.stdout: "last_epoch_started": 0, 2024-03-11T21:08:56.083 INFO:teuthology.orchestra.run.smithi023.stdout: "last_epoch_clean": 0, 2024-03-11T21:08:56.083 INFO:teuthology.orchestra.run.smithi023.stdout: "source_version": "0'0", 2024-03-11T21:08:56.083 INFO:teuthology.orchestra.run.smithi023.stdout: "target_version": "0'0" 2024-03-11T21:08:56.084 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:56.084 INFO:teuthology.orchestra.run.smithi023.stdout: "last_change": "80", 2024-03-11T21:08:56.084 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend": "0", 2024-03-11T21:08:56.084 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T21:08:56.084 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T21:08:56.085 INFO:teuthology.orchestra.run.smithi023.stdout: "auid": 0, 2024-03-11T21:08:56.085 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_mode": "selfmanaged", 2024-03-11T21:08:56.085 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_seq": 0, 2024-03-11T21:08:56.085 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_epoch": 0, 2024-03-11T21:08:56.085 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_snaps": [], 2024-03-11T21:08:56.085 INFO:teuthology.orchestra.run.smithi023.stdout: "removed_snaps": "[]", 2024-03-11T21:08:56.085 INFO:teuthology.orchestra.run.smithi023.stdout: "quota_max_bytes": 0, 2024-03-11T21:08:56.085 INFO:teuthology.orchestra.run.smithi023.stdout: "quota_max_objects": 0, 2024-03-11T21:08:56.085 INFO:teuthology.orchestra.run.smithi023.stdout: "tiers": [], 2024-03-11T21:08:56.085 INFO:teuthology.orchestra.run.smithi023.stdout: "tier_of": -1, 2024-03-11T21:08:56.085 INFO:teuthology.orchestra.run.smithi023.stdout: "read_tier": -1, 2024-03-11T21:08:56.085 INFO:teuthology.orchestra.run.smithi023.stdout: "write_tier": -1, 2024-03-11T21:08:56.085 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_mode": "none", 2024-03-11T21:08:56.086 INFO:teuthology.orchestra.run.smithi023.stdout: "target_max_bytes": 0, 2024-03-11T21:08:56.086 INFO:teuthology.orchestra.run.smithi023.stdout: "target_max_objects": 0, 2024-03-11T21:08:56.086 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T21:08:56.086 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T21:08:56.086 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T21:08:56.086 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_min_flush_age": 0, 2024-03-11T21:08:56.086 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_min_evict_age": 0, 2024-03-11T21:08:56.086 INFO:teuthology.orchestra.run.smithi023.stdout: "erasure_code_profile": "", 2024-03-11T21:08:56.086 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_params": { 2024-03-11T21:08:56.086 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "none" 2024-03-11T21:08:56.086 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:56.086 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_period": 0, 2024-03-11T21:08:56.086 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_count": 0, 2024-03-11T21:08:56.087 INFO:teuthology.orchestra.run.smithi023.stdout: "use_gmt_hitset": true, 2024-03-11T21:08:56.087 INFO:teuthology.orchestra.run.smithi023.stdout: "min_read_recency_for_promote": 0, 2024-03-11T21:08:56.087 INFO:teuthology.orchestra.run.smithi023.stdout: "min_write_recency_for_promote": 0, 2024-03-11T21:08:56.087 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T21:08:56.087 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_search_last_n": 0, 2024-03-11T21:08:56.087 INFO:teuthology.orchestra.run.smithi023.stdout: "grade_table": [], 2024-03-11T21:08:56.087 INFO:teuthology.orchestra.run.smithi023.stdout: "stripe_width": 0, 2024-03-11T21:08:56.087 INFO:teuthology.orchestra.run.smithi023.stdout: "expected_num_objects": 0, 2024-03-11T21:08:56.087 INFO:teuthology.orchestra.run.smithi023.stdout: "fast_read": false, 2024-03-11T21:08:56.087 INFO:teuthology.orchestra.run.smithi023.stdout: "options": {}, 2024-03-11T21:08:56.087 INFO:teuthology.orchestra.run.smithi023.stdout: "application_metadata": { 2024-03-11T21:08:56.087 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw": {} 2024-03-11T21:08:56.088 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:56.088 INFO:teuthology.orchestra.run.smithi023.stdout: "read_balance": { 2024-03-11T21:08:56.088 INFO:teuthology.orchestra.run.smithi023.stdout: "score_type": "Fair distribution", 2024-03-11T21:08:56.088 INFO:teuthology.orchestra.run.smithi023.stdout: "score_acting": 1.5, 2024-03-11T21:08:56.088 INFO:teuthology.orchestra.run.smithi023.stdout: "score_stable": 1.5, 2024-03-11T21:08:56.088 INFO:teuthology.orchestra.run.smithi023.stdout: "optimal_score": 1.3300000429153442, 2024-03-11T21:08:56.088 INFO:teuthology.orchestra.run.smithi023.stdout: "raw_score_acting": 2, 2024-03-11T21:08:56.088 INFO:teuthology.orchestra.run.smithi023.stdout: "raw_score_stable": 2, 2024-03-11T21:08:56.088 INFO:teuthology.orchestra.run.smithi023.stdout: "primary_affinity_weighted": 0.75, 2024-03-11T21:08:56.088 INFO:teuthology.orchestra.run.smithi023.stdout: "average_primary_affinity": 1, 2024-03-11T21:08:56.088 INFO:teuthology.orchestra.run.smithi023.stdout: "average_primary_affinity_weighted": 0.75 2024-03-11T21:08:56.088 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:08:56.088 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:56.089 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:56.089 INFO:teuthology.orchestra.run.smithi023.stdout: "pool": 4, 2024-03-11T21:08:56.089 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_name": "default.rgw.control", 2024-03-11T21:08:56.089 INFO:teuthology.orchestra.run.smithi023.stdout: "create_time": "2024-03-11T20:50:30.536371+0000", 2024-03-11T21:08:56.089 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-11T21:08:56.089 INFO:teuthology.orchestra.run.smithi023.stdout: "flags_names": "hashpspool", 2024-03-11T21:08:56.089 INFO:teuthology.orchestra.run.smithi023.stdout: "type": 1, 2024-03-11T21:08:56.089 INFO:teuthology.orchestra.run.smithi023.stdout: "size": 3, 2024-03-11T21:08:56.089 INFO:teuthology.orchestra.run.smithi023.stdout: "min_size": 2, 2024-03-11T21:08:56.089 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_rule": 0, 2024-03-11T21:08:56.089 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_count": 0, 2024-03-11T21:08:56.090 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_target": 0, 2024-03-11T21:08:56.090 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T21:08:56.090 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T21:08:56.090 INFO:teuthology.orchestra.run.smithi023.stdout: "object_hash": 2, 2024-03-11T21:08:56.090 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_autoscale_mode": "off", 2024-03-11T21:08:56.090 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num": 32, 2024-03-11T21:08:56.090 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_placement_num": 32, 2024-03-11T21:08:56.090 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_placement_num_target": 32, 2024-03-11T21:08:56.090 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num_target": 32, 2024-03-11T21:08:56.090 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num_pending": 32, 2024-03-11T21:08:56.091 INFO:teuthology.orchestra.run.smithi023.stdout: "last_pg_merge_meta": { 2024-03-11T21:08:56.091 INFO:teuthology.orchestra.run.smithi023.stdout: "source_pgid": "0.0", 2024-03-11T21:08:56.091 INFO:teuthology.orchestra.run.smithi023.stdout: "ready_epoch": 0, 2024-03-11T21:08:56.091 INFO:teuthology.orchestra.run.smithi023.stdout: "last_epoch_started": 0, 2024-03-11T21:08:56.091 INFO:teuthology.orchestra.run.smithi023.stdout: "last_epoch_clean": 0, 2024-03-11T21:08:56.091 INFO:teuthology.orchestra.run.smithi023.stdout: "source_version": "0'0", 2024-03-11T21:08:56.091 INFO:teuthology.orchestra.run.smithi023.stdout: "target_version": "0'0" 2024-03-11T21:08:56.091 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:56.091 INFO:teuthology.orchestra.run.smithi023.stdout: "last_change": "82", 2024-03-11T21:08:56.091 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend": "0", 2024-03-11T21:08:56.091 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T21:08:56.092 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T21:08:56.092 INFO:teuthology.orchestra.run.smithi023.stdout: "auid": 0, 2024-03-11T21:08:56.092 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_mode": "selfmanaged", 2024-03-11T21:08:56.092 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_seq": 0, 2024-03-11T21:08:56.092 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_epoch": 0, 2024-03-11T21:08:56.092 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_snaps": [], 2024-03-11T21:08:56.092 INFO:teuthology.orchestra.run.smithi023.stdout: "removed_snaps": "[]", 2024-03-11T21:08:56.092 INFO:teuthology.orchestra.run.smithi023.stdout: "quota_max_bytes": 0, 2024-03-11T21:08:56.092 INFO:teuthology.orchestra.run.smithi023.stdout: "quota_max_objects": 0, 2024-03-11T21:08:56.092 INFO:teuthology.orchestra.run.smithi023.stdout: "tiers": [], 2024-03-11T21:08:56.092 INFO:teuthology.orchestra.run.smithi023.stdout: "tier_of": -1, 2024-03-11T21:08:56.092 INFO:teuthology.orchestra.run.smithi023.stdout: "read_tier": -1, 2024-03-11T21:08:56.092 INFO:teuthology.orchestra.run.smithi023.stdout: "write_tier": -1, 2024-03-11T21:08:56.092 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_mode": "none", 2024-03-11T21:08:56.093 INFO:teuthology.orchestra.run.smithi023.stdout: "target_max_bytes": 0, 2024-03-11T21:08:56.093 INFO:teuthology.orchestra.run.smithi023.stdout: "target_max_objects": 0, 2024-03-11T21:08:56.093 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T21:08:56.093 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T21:08:56.093 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T21:08:56.093 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_min_flush_age": 0, 2024-03-11T21:08:56.093 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_min_evict_age": 0, 2024-03-11T21:08:56.093 INFO:teuthology.orchestra.run.smithi023.stdout: "erasure_code_profile": "", 2024-03-11T21:08:56.093 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_params": { 2024-03-11T21:08:56.093 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "none" 2024-03-11T21:08:56.093 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:56.093 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_period": 0, 2024-03-11T21:08:56.093 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_count": 0, 2024-03-11T21:08:56.093 INFO:teuthology.orchestra.run.smithi023.stdout: "use_gmt_hitset": true, 2024-03-11T21:08:56.093 INFO:teuthology.orchestra.run.smithi023.stdout: "min_read_recency_for_promote": 0, 2024-03-11T21:08:56.094 INFO:teuthology.orchestra.run.smithi023.stdout: "min_write_recency_for_promote": 0, 2024-03-11T21:08:56.094 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T21:08:56.094 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_search_last_n": 0, 2024-03-11T21:08:56.094 INFO:teuthology.orchestra.run.smithi023.stdout: "grade_table": [], 2024-03-11T21:08:56.094 INFO:teuthology.orchestra.run.smithi023.stdout: "stripe_width": 0, 2024-03-11T21:08:56.094 INFO:teuthology.orchestra.run.smithi023.stdout: "expected_num_objects": 0, 2024-03-11T21:08:56.094 INFO:teuthology.orchestra.run.smithi023.stdout: "fast_read": false, 2024-03-11T21:08:56.094 INFO:teuthology.orchestra.run.smithi023.stdout: "options": {}, 2024-03-11T21:08:56.094 INFO:teuthology.orchestra.run.smithi023.stdout: "application_metadata": { 2024-03-11T21:08:56.094 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw": {} 2024-03-11T21:08:56.094 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:56.094 INFO:teuthology.orchestra.run.smithi023.stdout: "read_balance": { 2024-03-11T21:08:56.094 INFO:teuthology.orchestra.run.smithi023.stdout: "score_type": "Fair distribution", 2024-03-11T21:08:56.095 INFO:teuthology.orchestra.run.smithi023.stdout: "score_acting": 1.690000057220459, 2024-03-11T21:08:56.095 INFO:teuthology.orchestra.run.smithi023.stdout: "score_stable": 1.690000057220459, 2024-03-11T21:08:56.095 INFO:teuthology.orchestra.run.smithi023.stdout: "optimal_score": 1.3300000429153442, 2024-03-11T21:08:56.095 INFO:teuthology.orchestra.run.smithi023.stdout: "raw_score_acting": 2.25, 2024-03-11T21:08:56.095 INFO:teuthology.orchestra.run.smithi023.stdout: "raw_score_stable": 2.25, 2024-03-11T21:08:56.095 INFO:teuthology.orchestra.run.smithi023.stdout: "primary_affinity_weighted": 0.75, 2024-03-11T21:08:56.095 INFO:teuthology.orchestra.run.smithi023.stdout: "average_primary_affinity": 1, 2024-03-11T21:08:56.096 INFO:teuthology.orchestra.run.smithi023.stdout: "average_primary_affinity_weighted": 0.75 2024-03-11T21:08:56.096 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:08:56.096 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:56.096 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:56.096 INFO:teuthology.orchestra.run.smithi023.stdout: "pool": 5, 2024-03-11T21:08:56.096 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_name": "datapool", 2024-03-11T21:08:56.096 INFO:teuthology.orchestra.run.smithi023.stdout: "create_time": "2024-03-11T20:50:31.135845+0000", 2024-03-11T21:08:56.097 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 8193, 2024-03-11T21:08:56.097 INFO:teuthology.orchestra.run.smithi023.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-03-11T21:08:56.097 INFO:teuthology.orchestra.run.smithi023.stdout: "type": 1, 2024-03-11T21:08:56.097 INFO:teuthology.orchestra.run.smithi023.stdout: "size": 3, 2024-03-11T21:08:56.097 INFO:teuthology.orchestra.run.smithi023.stdout: "min_size": 2, 2024-03-11T21:08:56.097 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_rule": 0, 2024-03-11T21:08:56.097 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_count": 0, 2024-03-11T21:08:56.098 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_target": 0, 2024-03-11T21:08:56.098 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T21:08:56.098 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T21:08:56.098 INFO:teuthology.orchestra.run.smithi023.stdout: "object_hash": 2, 2024-03-11T21:08:56.098 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_autoscale_mode": "off", 2024-03-11T21:08:56.098 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num": 3, 2024-03-11T21:08:56.099 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_placement_num": 3, 2024-03-11T21:08:56.099 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_placement_num_target": 3, 2024-03-11T21:08:56.099 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num_target": 3, 2024-03-11T21:08:56.099 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num_pending": 3, 2024-03-11T21:08:56.099 INFO:teuthology.orchestra.run.smithi023.stdout: "last_pg_merge_meta": { 2024-03-11T21:08:56.099 INFO:teuthology.orchestra.run.smithi023.stdout: "source_pgid": "0.0", 2024-03-11T21:08:56.100 INFO:teuthology.orchestra.run.smithi023.stdout: "ready_epoch": 0, 2024-03-11T21:08:56.100 INFO:teuthology.orchestra.run.smithi023.stdout: "last_epoch_started": 0, 2024-03-11T21:08:56.100 INFO:teuthology.orchestra.run.smithi023.stdout: "last_epoch_clean": 0, 2024-03-11T21:08:56.100 INFO:teuthology.orchestra.run.smithi023.stdout: "source_version": "0'0", 2024-03-11T21:08:56.100 INFO:teuthology.orchestra.run.smithi023.stdout: "target_version": "0'0" 2024-03-11T21:08:56.100 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:56.100 INFO:teuthology.orchestra.run.smithi023.stdout: "last_change": "87", 2024-03-11T21:08:56.101 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend": "0", 2024-03-11T21:08:56.101 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T21:08:56.101 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T21:08:56.101 INFO:teuthology.orchestra.run.smithi023.stdout: "auid": 0, 2024-03-11T21:08:56.101 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_mode": "selfmanaged", 2024-03-11T21:08:56.101 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_seq": 2, 2024-03-11T21:08:56.102 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_epoch": 87, 2024-03-11T21:08:56.102 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_snaps": [], 2024-03-11T21:08:56.102 INFO:teuthology.orchestra.run.smithi023.stdout: "removed_snaps": "[]", 2024-03-11T21:08:56.102 INFO:teuthology.orchestra.run.smithi023.stdout: "quota_max_bytes": 0, 2024-03-11T21:08:56.102 INFO:teuthology.orchestra.run.smithi023.stdout: "quota_max_objects": 0, 2024-03-11T21:08:56.103 INFO:teuthology.orchestra.run.smithi023.stdout: "tiers": [], 2024-03-11T21:08:56.103 INFO:teuthology.orchestra.run.smithi023.stdout: "tier_of": -1, 2024-03-11T21:08:56.103 INFO:teuthology.orchestra.run.smithi023.stdout: "read_tier": -1, 2024-03-11T21:08:56.103 INFO:teuthology.orchestra.run.smithi023.stdout: "write_tier": -1, 2024-03-11T21:08:56.103 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_mode": "none", 2024-03-11T21:08:56.103 INFO:teuthology.orchestra.run.smithi023.stdout: "target_max_bytes": 0, 2024-03-11T21:08:56.104 INFO:teuthology.orchestra.run.smithi023.stdout: "target_max_objects": 0, 2024-03-11T21:08:56.104 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T21:08:56.104 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T21:08:56.104 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T21:08:56.104 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_min_flush_age": 0, 2024-03-11T21:08:56.104 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_min_evict_age": 0, 2024-03-11T21:08:56.105 INFO:teuthology.orchestra.run.smithi023.stdout: "erasure_code_profile": "", 2024-03-11T21:08:56.105 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_params": { 2024-03-11T21:08:56.105 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "none" 2024-03-11T21:08:56.105 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:56.105 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_period": 0, 2024-03-11T21:08:56.105 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_count": 0, 2024-03-11T21:08:56.105 INFO:teuthology.orchestra.run.smithi023.stdout: "use_gmt_hitset": true, 2024-03-11T21:08:56.105 INFO:teuthology.orchestra.run.smithi023.stdout: "min_read_recency_for_promote": 0, 2024-03-11T21:08:56.105 INFO:teuthology.orchestra.run.smithi023.stdout: "min_write_recency_for_promote": 0, 2024-03-11T21:08:56.106 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T21:08:56.106 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_search_last_n": 0, 2024-03-11T21:08:56.106 INFO:teuthology.orchestra.run.smithi023.stdout: "grade_table": [], 2024-03-11T21:08:56.106 INFO:teuthology.orchestra.run.smithi023.stdout: "stripe_width": 0, 2024-03-11T21:08:56.106 INFO:teuthology.orchestra.run.smithi023.stdout: "expected_num_objects": 0, 2024-03-11T21:08:56.106 INFO:teuthology.orchestra.run.smithi023.stdout: "fast_read": false, 2024-03-11T21:08:56.106 INFO:teuthology.orchestra.run.smithi023.stdout: "options": {}, 2024-03-11T21:08:56.107 INFO:teuthology.orchestra.run.smithi023.stdout: "application_metadata": { 2024-03-11T21:08:56.107 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd": {} 2024-03-11T21:08:56.107 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:56.107 INFO:teuthology.orchestra.run.smithi023.stdout: "read_balance": { 2024-03-11T21:08:56.107 INFO:teuthology.orchestra.run.smithi023.stdout: "score_type": "Fair distribution", 2024-03-11T21:08:56.107 INFO:teuthology.orchestra.run.smithi023.stdout: "score_acting": 3.9900000095367432, 2024-03-11T21:08:56.108 INFO:teuthology.orchestra.run.smithi023.stdout: "score_stable": 3.9900000095367432, 2024-03-11T21:08:56.108 INFO:teuthology.orchestra.run.smithi023.stdout: "optimal_score": 1.1699999570846558, 2024-03-11T21:08:56.108 INFO:teuthology.orchestra.run.smithi023.stdout: "raw_score_acting": 4.6700000762939453, 2024-03-11T21:08:56.108 INFO:teuthology.orchestra.run.smithi023.stdout: "raw_score_stable": 4.6700000762939453, 2024-03-11T21:08:56.108 INFO:teuthology.orchestra.run.smithi023.stdout: "primary_affinity_weighted": 0.75, 2024-03-11T21:08:56.108 INFO:teuthology.orchestra.run.smithi023.stdout: "average_primary_affinity": 1, 2024-03-11T21:08:56.108 INFO:teuthology.orchestra.run.smithi023.stdout: "average_primary_affinity_weighted": 0.75 2024-03-11T21:08:56.109 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:08:56.109 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:56.109 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:56.109 INFO:teuthology.orchestra.run.smithi023.stdout: "pool": 6, 2024-03-11T21:08:56.109 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_name": "default.rgw.meta", 2024-03-11T21:08:56.109 INFO:teuthology.orchestra.run.smithi023.stdout: "create_time": "2024-03-11T20:50:33.233163+0000", 2024-03-11T21:08:56.110 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-11T21:08:56.110 INFO:teuthology.orchestra.run.smithi023.stdout: "flags_names": "hashpspool", 2024-03-11T21:08:56.110 INFO:teuthology.orchestra.run.smithi023.stdout: "type": 1, 2024-03-11T21:08:56.110 INFO:teuthology.orchestra.run.smithi023.stdout: "size": 3, 2024-03-11T21:08:56.110 INFO:teuthology.orchestra.run.smithi023.stdout: "min_size": 2, 2024-03-11T21:08:56.110 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_rule": 0, 2024-03-11T21:08:56.111 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_count": 0, 2024-03-11T21:08:56.111 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_target": 0, 2024-03-11T21:08:56.111 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T21:08:56.111 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T21:08:56.111 INFO:teuthology.orchestra.run.smithi023.stdout: "object_hash": 2, 2024-03-11T21:08:56.111 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_autoscale_mode": "off", 2024-03-11T21:08:56.112 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num": 32, 2024-03-11T21:08:56.112 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_placement_num": 32, 2024-03-11T21:08:56.112 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_placement_num_target": 32, 2024-03-11T21:08:56.112 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num_target": 32, 2024-03-11T21:08:56.112 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num_pending": 32, 2024-03-11T21:08:56.112 INFO:teuthology.orchestra.run.smithi023.stdout: "last_pg_merge_meta": { 2024-03-11T21:08:56.113 INFO:teuthology.orchestra.run.smithi023.stdout: "source_pgid": "0.0", 2024-03-11T21:08:56.113 INFO:teuthology.orchestra.run.smithi023.stdout: "ready_epoch": 0, 2024-03-11T21:08:56.113 INFO:teuthology.orchestra.run.smithi023.stdout: "last_epoch_started": 0, 2024-03-11T21:08:56.113 INFO:teuthology.orchestra.run.smithi023.stdout: "last_epoch_clean": 0, 2024-03-11T21:08:56.113 INFO:teuthology.orchestra.run.smithi023.stdout: "source_version": "0'0", 2024-03-11T21:08:56.113 INFO:teuthology.orchestra.run.smithi023.stdout: "target_version": "0'0" 2024-03-11T21:08:56.114 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:56.114 INFO:teuthology.orchestra.run.smithi023.stdout: "last_change": "84", 2024-03-11T21:08:56.114 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend": "0", 2024-03-11T21:08:56.114 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T21:08:56.114 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T21:08:56.114 INFO:teuthology.orchestra.run.smithi023.stdout: "auid": 0, 2024-03-11T21:08:56.115 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_mode": "selfmanaged", 2024-03-11T21:08:56.115 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_seq": 0, 2024-03-11T21:08:56.115 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_epoch": 0, 2024-03-11T21:08:56.115 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_snaps": [], 2024-03-11T21:08:56.115 INFO:teuthology.orchestra.run.smithi023.stdout: "removed_snaps": "[]", 2024-03-11T21:08:56.115 INFO:teuthology.orchestra.run.smithi023.stdout: "quota_max_bytes": 0, 2024-03-11T21:08:56.115 INFO:teuthology.orchestra.run.smithi023.stdout: "quota_max_objects": 0, 2024-03-11T21:08:56.116 INFO:teuthology.orchestra.run.smithi023.stdout: "tiers": [], 2024-03-11T21:08:56.116 INFO:teuthology.orchestra.run.smithi023.stdout: "tier_of": -1, 2024-03-11T21:08:56.116 INFO:teuthology.orchestra.run.smithi023.stdout: "read_tier": -1, 2024-03-11T21:08:56.116 INFO:teuthology.orchestra.run.smithi023.stdout: "write_tier": -1, 2024-03-11T21:08:56.116 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_mode": "none", 2024-03-11T21:08:56.117 INFO:teuthology.orchestra.run.smithi023.stdout: "target_max_bytes": 0, 2024-03-11T21:08:56.117 INFO:teuthology.orchestra.run.smithi023.stdout: "target_max_objects": 0, 2024-03-11T21:08:56.117 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T21:08:56.117 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T21:08:56.117 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T21:08:56.117 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_min_flush_age": 0, 2024-03-11T21:08:56.118 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_min_evict_age": 0, 2024-03-11T21:08:56.118 INFO:teuthology.orchestra.run.smithi023.stdout: "erasure_code_profile": "", 2024-03-11T21:08:56.118 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_params": { 2024-03-11T21:08:56.118 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "none" 2024-03-11T21:08:56.118 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:56.118 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_period": 0, 2024-03-11T21:08:56.118 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_count": 0, 2024-03-11T21:08:56.119 INFO:teuthology.orchestra.run.smithi023.stdout: "use_gmt_hitset": true, 2024-03-11T21:08:56.119 INFO:teuthology.orchestra.run.smithi023.stdout: "min_read_recency_for_promote": 0, 2024-03-11T21:08:56.119 INFO:teuthology.orchestra.run.smithi023.stdout: "min_write_recency_for_promote": 0, 2024-03-11T21:08:56.119 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T21:08:56.119 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_search_last_n": 0, 2024-03-11T21:08:56.119 INFO:teuthology.orchestra.run.smithi023.stdout: "grade_table": [], 2024-03-11T21:08:56.119 INFO:teuthology.orchestra.run.smithi023.stdout: "stripe_width": 0, 2024-03-11T21:08:56.119 INFO:teuthology.orchestra.run.smithi023.stdout: "expected_num_objects": 0, 2024-03-11T21:08:56.119 INFO:teuthology.orchestra.run.smithi023.stdout: "fast_read": false, 2024-03-11T21:08:56.120 INFO:teuthology.orchestra.run.smithi023.stdout: "options": { 2024-03-11T21:08:56.120 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_autoscale_bias": 4 2024-03-11T21:08:56.120 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:56.120 INFO:teuthology.orchestra.run.smithi023.stdout: "application_metadata": { 2024-03-11T21:08:56.120 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw": {} 2024-03-11T21:08:56.120 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:56.120 INFO:teuthology.orchestra.run.smithi023.stdout: "read_balance": { 2024-03-11T21:08:56.120 INFO:teuthology.orchestra.run.smithi023.stdout: "score_type": "Fair distribution", 2024-03-11T21:08:56.121 INFO:teuthology.orchestra.run.smithi023.stdout: "score_acting": 1.690000057220459, 2024-03-11T21:08:56.121 INFO:teuthology.orchestra.run.smithi023.stdout: "score_stable": 1.690000057220459, 2024-03-11T21:08:56.121 INFO:teuthology.orchestra.run.smithi023.stdout: "optimal_score": 1.3300000429153442, 2024-03-11T21:08:56.121 INFO:teuthology.orchestra.run.smithi023.stdout: "raw_score_acting": 2.25, 2024-03-11T21:08:56.121 INFO:teuthology.orchestra.run.smithi023.stdout: "raw_score_stable": 2.25, 2024-03-11T21:08:56.121 INFO:teuthology.orchestra.run.smithi023.stdout: "primary_affinity_weighted": 0.75, 2024-03-11T21:08:56.121 INFO:teuthology.orchestra.run.smithi023.stdout: "average_primary_affinity": 1, 2024-03-11T21:08:56.121 INFO:teuthology.orchestra.run.smithi023.stdout: "average_primary_affinity_weighted": 0.75 2024-03-11T21:08:56.122 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:08:56.122 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:56.122 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:56.122 INFO:teuthology.orchestra.run.smithi023.stdout: "pool": 10, 2024-03-11T21:08:56.122 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_name": "unique_pool_3", 2024-03-11T21:08:56.122 INFO:teuthology.orchestra.run.smithi023.stdout: "create_time": "2024-03-11T21:07:13.796707+0000", 2024-03-11T21:08:56.122 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-11T21:08:56.123 INFO:teuthology.orchestra.run.smithi023.stdout: "flags_names": "hashpspool", 2024-03-11T21:08:56.123 INFO:teuthology.orchestra.run.smithi023.stdout: "type": 1, 2024-03-11T21:08:56.123 INFO:teuthology.orchestra.run.smithi023.stdout: "size": 3, 2024-03-11T21:08:56.123 INFO:teuthology.orchestra.run.smithi023.stdout: "min_size": 2, 2024-03-11T21:08:56.123 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_rule": 0, 2024-03-11T21:08:56.123 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_count": 0, 2024-03-11T21:08:56.123 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_target": 0, 2024-03-11T21:08:56.123 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T21:08:56.123 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T21:08:56.123 INFO:teuthology.orchestra.run.smithi023.stdout: "object_hash": 2, 2024-03-11T21:08:56.123 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_autoscale_mode": "off", 2024-03-11T21:08:56.124 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num": 16, 2024-03-11T21:08:56.124 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_placement_num": 16, 2024-03-11T21:08:56.124 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_placement_num_target": 16, 2024-03-11T21:08:56.124 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num_target": 16, 2024-03-11T21:08:56.124 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num_pending": 16, 2024-03-11T21:08:56.124 INFO:teuthology.orchestra.run.smithi023.stdout: "last_pg_merge_meta": { 2024-03-11T21:08:56.124 INFO:teuthology.orchestra.run.smithi023.stdout: "source_pgid": "0.0", 2024-03-11T21:08:56.125 INFO:teuthology.orchestra.run.smithi023.stdout: "ready_epoch": 0, 2024-03-11T21:08:56.125 INFO:teuthology.orchestra.run.smithi023.stdout: "last_epoch_started": 0, 2024-03-11T21:08:56.125 INFO:teuthology.orchestra.run.smithi023.stdout: "last_epoch_clean": 0, 2024-03-11T21:08:56.125 INFO:teuthology.orchestra.run.smithi023.stdout: "source_version": "0'0", 2024-03-11T21:08:56.125 INFO:teuthology.orchestra.run.smithi023.stdout: "target_version": "0'0" 2024-03-11T21:08:56.125 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:56.126 INFO:teuthology.orchestra.run.smithi023.stdout: "last_change": "310", 2024-03-11T21:08:56.126 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend": "0", 2024-03-11T21:08:56.126 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T21:08:56.126 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T21:08:56.126 INFO:teuthology.orchestra.run.smithi023.stdout: "auid": 0, 2024-03-11T21:08:56.126 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_mode": "selfmanaged", 2024-03-11T21:08:56.127 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_seq": 0, 2024-03-11T21:08:56.127 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_epoch": 0, 2024-03-11T21:08:56.127 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_snaps": [], 2024-03-11T21:08:56.127 INFO:teuthology.orchestra.run.smithi023.stdout: "removed_snaps": "[]", 2024-03-11T21:08:56.127 INFO:teuthology.orchestra.run.smithi023.stdout: "quota_max_bytes": 0, 2024-03-11T21:08:56.127 INFO:teuthology.orchestra.run.smithi023.stdout: "quota_max_objects": 0, 2024-03-11T21:08:56.127 INFO:teuthology.orchestra.run.smithi023.stdout: "tiers": [], 2024-03-11T21:08:56.128 INFO:teuthology.orchestra.run.smithi023.stdout: "tier_of": -1, 2024-03-11T21:08:56.128 INFO:teuthology.orchestra.run.smithi023.stdout: "read_tier": -1, 2024-03-11T21:08:56.128 INFO:teuthology.orchestra.run.smithi023.stdout: "write_tier": -1, 2024-03-11T21:08:56.128 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_mode": "none", 2024-03-11T21:08:56.128 INFO:teuthology.orchestra.run.smithi023.stdout: "target_max_bytes": 0, 2024-03-11T21:08:56.128 INFO:teuthology.orchestra.run.smithi023.stdout: "target_max_objects": 0, 2024-03-11T21:08:56.129 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T21:08:56.129 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T21:08:56.129 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T21:08:56.129 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_min_flush_age": 0, 2024-03-11T21:08:56.129 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_min_evict_age": 0, 2024-03-11T21:08:56.129 INFO:teuthology.orchestra.run.smithi023.stdout: "erasure_code_profile": "", 2024-03-11T21:08:56.130 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_params": { 2024-03-11T21:08:56.130 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "none" 2024-03-11T21:08:56.130 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:56.130 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_period": 0, 2024-03-11T21:08:56.130 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_count": 0, 2024-03-11T21:08:56.130 INFO:teuthology.orchestra.run.smithi023.stdout: "use_gmt_hitset": true, 2024-03-11T21:08:56.131 INFO:teuthology.orchestra.run.smithi023.stdout: "min_read_recency_for_promote": 0, 2024-03-11T21:08:56.131 INFO:teuthology.orchestra.run.smithi023.stdout: "min_write_recency_for_promote": 0, 2024-03-11T21:08:56.131 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T21:08:56.131 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_search_last_n": 0, 2024-03-11T21:08:56.131 INFO:teuthology.orchestra.run.smithi023.stdout: "grade_table": [], 2024-03-11T21:08:56.131 INFO:teuthology.orchestra.run.smithi023.stdout: "stripe_width": 0, 2024-03-11T21:08:56.132 INFO:teuthology.orchestra.run.smithi023.stdout: "expected_num_objects": 0, 2024-03-11T21:08:56.132 INFO:teuthology.orchestra.run.smithi023.stdout: "fast_read": false, 2024-03-11T21:08:56.132 INFO:teuthology.orchestra.run.smithi023.stdout: "options": {}, 2024-03-11T21:08:56.132 INFO:teuthology.orchestra.run.smithi023.stdout: "application_metadata": { 2024-03-11T21:08:56.132 INFO:teuthology.orchestra.run.smithi023.stdout: "rados": {} 2024-03-11T21:08:56.132 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:56.132 INFO:teuthology.orchestra.run.smithi023.stdout: "read_balance": { 2024-03-11T21:08:56.133 INFO:teuthology.orchestra.run.smithi023.stdout: "score_type": "Fair distribution", 2024-03-11T21:08:56.133 INFO:teuthology.orchestra.run.smithi023.stdout: "score_acting": 2.630000114440918, 2024-03-11T21:08:56.133 INFO:teuthology.orchestra.run.smithi023.stdout: "score_stable": 2.630000114440918, 2024-03-11T21:08:56.133 INFO:teuthology.orchestra.run.smithi023.stdout: "optimal_score": 1.3300000429153442, 2024-03-11T21:08:56.133 INFO:teuthology.orchestra.run.smithi023.stdout: "raw_score_acting": 3.5, 2024-03-11T21:08:56.133 INFO:teuthology.orchestra.run.smithi023.stdout: "raw_score_stable": 3.5, 2024-03-11T21:08:56.133 INFO:teuthology.orchestra.run.smithi023.stdout: "primary_affinity_weighted": 0.75, 2024-03-11T21:08:56.134 INFO:teuthology.orchestra.run.smithi023.stdout: "average_primary_affinity": 1, 2024-03-11T21:08:56.134 INFO:teuthology.orchestra.run.smithi023.stdout: "average_primary_affinity_weighted": 0.75 2024-03-11T21:08:56.134 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:08:56.134 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:08:56.134 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-11T21:08:56.134 INFO:teuthology.orchestra.run.smithi023.stdout: "osds": [ 2024-03-11T21:08:56.134 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:56.135 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": 0, 2024-03-11T21:08:56.135 INFO:teuthology.orchestra.run.smithi023.stdout: "uuid": "3df72b4a-8fe4-4265-b4c6-af022e1e23e1", 2024-03-11T21:08:56.135 INFO:teuthology.orchestra.run.smithi023.stdout: "up": 1, 2024-03-11T21:08:56.135 INFO:teuthology.orchestra.run.smithi023.stdout: "in": 1, 2024-03-11T21:08:56.135 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 1, 2024-03-11T21:08:56.135 INFO:teuthology.orchestra.run.smithi023.stdout: "primary_affinity": 1, 2024-03-11T21:08:56.136 INFO:teuthology.orchestra.run.smithi023.stdout: "last_clean_begin": 0, 2024-03-11T21:08:56.136 INFO:teuthology.orchestra.run.smithi023.stdout: "last_clean_end": 0, 2024-03-11T21:08:56.136 INFO:teuthology.orchestra.run.smithi023.stdout: "up_from": 8, 2024-03-11T21:08:56.136 INFO:teuthology.orchestra.run.smithi023.stdout: "up_thru": 319, 2024-03-11T21:08:56.136 INFO:teuthology.orchestra.run.smithi023.stdout: "down_at": 0, 2024-03-11T21:08:56.136 INFO:teuthology.orchestra.run.smithi023.stdout: "lost_at": 0, 2024-03-11T21:08:56.136 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2024-03-11T21:08:56.137 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:08:56.137 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:56.137 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:08:56.137 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6802", 2024-03-11T21:08:56.137 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 2154627393 2024-03-11T21:08:56.137 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:56.138 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:56.138 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:08:56.138 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6803", 2024-03-11T21:08:56.138 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 2154627393 2024-03-11T21:08:56.138 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:08:56.138 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:08:56.139 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:56.139 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_addrs": { 2024-03-11T21:08:56.139 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:08:56.139 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:56.139 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:08:56.139 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6804", 2024-03-11T21:08:56.140 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 2154627393 2024-03-11T21:08:56.140 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:56.140 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:56.140 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:08:56.140 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6805", 2024-03-11T21:08:56.140 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 2154627393 2024-03-11T21:08:56.141 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:08:56.141 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:08:56.141 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:56.141 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_back_addrs": { 2024-03-11T21:08:56.141 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:08:56.141 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:56.142 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:08:56.142 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6808", 2024-03-11T21:08:56.142 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 2154627393 2024-03-11T21:08:56.142 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:56.142 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:56.142 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:08:56.142 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6809", 2024-03-11T21:08:56.143 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 2154627393 2024-03-11T21:08:56.143 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:08:56.143 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:08:56.143 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:56.143 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_front_addrs": { 2024-03-11T21:08:56.144 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:08:56.144 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:56.144 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:08:56.144 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6806", 2024-03-11T21:08:56.144 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 2154627393 2024-03-11T21:08:56.144 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:56.145 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:56.145 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:08:56.145 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6807", 2024-03-11T21:08:56.145 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 2154627393 2024-03-11T21:08:56.145 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:08:56.145 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:08:56.146 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:56.146 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6803/2154627393", 2024-03-11T21:08:56.146 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_addr": "172.21.15.23:6805/2154627393", 2024-03-11T21:08:56.146 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_back_addr": "172.21.15.23:6809/2154627393", 2024-03-11T21:08:56.146 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_front_addr": "172.21.15.23:6807/2154627393", 2024-03-11T21:08:56.147 INFO:teuthology.orchestra.run.smithi023.stdout: "state": [ 2024-03-11T21:08:56.147 INFO:teuthology.orchestra.run.smithi023.stdout: "exists", 2024-03-11T21:08:56.147 INFO:teuthology.orchestra.run.smithi023.stdout: "up" 2024-03-11T21:08:56.147 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:08:56.147 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:56.147 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:56.148 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": 1, 2024-03-11T21:08:56.148 INFO:teuthology.orchestra.run.smithi023.stdout: "uuid": "e8fd7500-9081-41a1-8077-a4d0f0e12a2c", 2024-03-11T21:08:56.148 INFO:teuthology.orchestra.run.smithi023.stdout: "up": 1, 2024-03-11T21:08:56.148 INFO:teuthology.orchestra.run.smithi023.stdout: "in": 1, 2024-03-11T21:08:56.148 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 1, 2024-03-11T21:08:56.148 INFO:teuthology.orchestra.run.smithi023.stdout: "primary_affinity": 1, 2024-03-11T21:08:56.149 INFO:teuthology.orchestra.run.smithi023.stdout: "last_clean_begin": 13, 2024-03-11T21:08:56.149 INFO:teuthology.orchestra.run.smithi023.stdout: "last_clean_end": 173, 2024-03-11T21:08:56.149 INFO:teuthology.orchestra.run.smithi023.stdout: "up_from": 211, 2024-03-11T21:08:56.149 INFO:teuthology.orchestra.run.smithi023.stdout: "up_thru": 319, 2024-03-11T21:08:56.149 INFO:teuthology.orchestra.run.smithi023.stdout: "down_at": 174, 2024-03-11T21:08:56.150 INFO:teuthology.orchestra.run.smithi023.stdout: "lost_at": 0, 2024-03-11T21:08:56.150 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2024-03-11T21:08:56.150 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:08:56.150 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:56.150 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:08:56.151 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6810", 2024-03-11T21:08:56.151 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 56585354 2024-03-11T21:08:56.151 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:56.151 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:56.151 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:08:56.151 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6811", 2024-03-11T21:08:56.152 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 56585354 2024-03-11T21:08:56.152 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:08:56.152 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:08:56.152 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:56.152 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_addrs": { 2024-03-11T21:08:56.152 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:08:56.153 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:56.153 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:08:56.153 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6812", 2024-03-11T21:08:56.153 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 56585354 2024-03-11T21:08:56.153 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:56.153 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:56.153 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:08:56.153 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6813", 2024-03-11T21:08:56.153 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 56585354 2024-03-11T21:08:56.153 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:08:56.153 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:08:56.154 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:56.154 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_back_addrs": { 2024-03-11T21:08:56.154 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:08:56.154 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:56.154 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:08:56.154 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6816", 2024-03-11T21:08:56.154 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 56585354 2024-03-11T21:08:56.154 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:56.154 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:56.154 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:08:56.154 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6817", 2024-03-11T21:08:56.154 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 56585354 2024-03-11T21:08:56.154 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:08:56.155 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:08:56.155 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:56.155 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_front_addrs": { 2024-03-11T21:08:56.155 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:08:56.155 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:56.155 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:08:56.155 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6814", 2024-03-11T21:08:56.155 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 56585354 2024-03-11T21:08:56.155 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:56.155 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:56.155 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:08:56.155 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6815", 2024-03-11T21:08:56.155 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 56585354 2024-03-11T21:08:56.155 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:08:56.155 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:08:56.156 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:56.156 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6811/56585354", 2024-03-11T21:08:56.156 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_addr": "172.21.15.23:6813/56585354", 2024-03-11T21:08:56.156 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_back_addr": "172.21.15.23:6817/56585354", 2024-03-11T21:08:56.156 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_front_addr": "172.21.15.23:6815/56585354", 2024-03-11T21:08:56.156 INFO:teuthology.orchestra.run.smithi023.stdout: "state": [ 2024-03-11T21:08:56.156 INFO:teuthology.orchestra.run.smithi023.stdout: "exists", 2024-03-11T21:08:56.156 INFO:teuthology.orchestra.run.smithi023.stdout: "up" 2024-03-11T21:08:56.156 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:08:56.156 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:56.156 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:56.156 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": 2, 2024-03-11T21:08:56.156 INFO:teuthology.orchestra.run.smithi023.stdout: "uuid": "dd85d52a-df48-4b03-8612-cecfd82b5987", 2024-03-11T21:08:56.157 INFO:teuthology.orchestra.run.smithi023.stdout: "up": 1, 2024-03-11T21:08:56.157 INFO:teuthology.orchestra.run.smithi023.stdout: "in": 1, 2024-03-11T21:08:56.157 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 1, 2024-03-11T21:08:56.157 INFO:teuthology.orchestra.run.smithi023.stdout: "primary_affinity": 0, 2024-03-11T21:08:56.157 INFO:teuthology.orchestra.run.smithi023.stdout: "last_clean_begin": 0, 2024-03-11T21:08:56.157 INFO:teuthology.orchestra.run.smithi023.stdout: "last_clean_end": 0, 2024-03-11T21:08:56.157 INFO:teuthology.orchestra.run.smithi023.stdout: "up_from": 19, 2024-03-11T21:08:56.157 INFO:teuthology.orchestra.run.smithi023.stdout: "up_thru": 306, 2024-03-11T21:08:56.157 INFO:teuthology.orchestra.run.smithi023.stdout: "down_at": 0, 2024-03-11T21:08:56.157 INFO:teuthology.orchestra.run.smithi023.stdout: "lost_at": 0, 2024-03-11T21:08:56.157 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2024-03-11T21:08:56.157 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:08:56.157 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:56.157 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:08:56.157 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6818", 2024-03-11T21:08:56.158 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1770748056 2024-03-11T21:08:56.158 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:56.158 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:56.158 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:08:56.158 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6819", 2024-03-11T21:08:56.158 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1770748056 2024-03-11T21:08:56.158 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:08:56.158 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:08:56.158 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:56.158 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_addrs": { 2024-03-11T21:08:56.158 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:08:56.158 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:56.158 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:08:56.158 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6820", 2024-03-11T21:08:56.158 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1770748056 2024-03-11T21:08:56.159 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:56.159 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:56.159 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:08:56.159 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6821", 2024-03-11T21:08:56.159 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1770748056 2024-03-11T21:08:56.159 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:08:56.159 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:08:56.159 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:56.159 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_back_addrs": { 2024-03-11T21:08:56.159 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:08:56.159 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:56.159 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:08:56.159 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6824", 2024-03-11T21:08:56.159 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1770748056 2024-03-11T21:08:56.159 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:56.160 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:56.160 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:08:56.160 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6825", 2024-03-11T21:08:56.160 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1770748056 2024-03-11T21:08:56.160 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:08:56.160 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:08:56.160 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:56.160 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_front_addrs": { 2024-03-11T21:08:56.160 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:08:56.160 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:56.160 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:08:56.160 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6822", 2024-03-11T21:08:56.160 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1770748056 2024-03-11T21:08:56.160 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:56.161 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:56.161 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:08:56.161 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6823", 2024-03-11T21:08:56.161 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1770748056 2024-03-11T21:08:56.161 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:08:56.161 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:08:56.161 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:56.161 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6819/1770748056", 2024-03-11T21:08:56.161 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_addr": "172.21.15.23:6821/1770748056", 2024-03-11T21:08:56.161 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_back_addr": "172.21.15.23:6825/1770748056", 2024-03-11T21:08:56.161 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_front_addr": "172.21.15.23:6823/1770748056", 2024-03-11T21:08:56.161 INFO:teuthology.orchestra.run.smithi023.stdout: "state": [ 2024-03-11T21:08:56.161 INFO:teuthology.orchestra.run.smithi023.stdout: "exists", 2024-03-11T21:08:56.161 INFO:teuthology.orchestra.run.smithi023.stdout: "up" 2024-03-11T21:08:56.161 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:08:56.162 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:56.162 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:56.162 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": 3, 2024-03-11T21:08:56.162 INFO:teuthology.orchestra.run.smithi023.stdout: "uuid": "e3021106-3779-4b94-9bbf-a7724544110f", 2024-03-11T21:08:56.162 INFO:teuthology.orchestra.run.smithi023.stdout: "up": 1, 2024-03-11T21:08:56.162 INFO:teuthology.orchestra.run.smithi023.stdout: "in": 1, 2024-03-11T21:08:56.162 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 1, 2024-03-11T21:08:56.162 INFO:teuthology.orchestra.run.smithi023.stdout: "primary_affinity": 1, 2024-03-11T21:08:56.162 INFO:teuthology.orchestra.run.smithi023.stdout: "last_clean_begin": 0, 2024-03-11T21:08:56.162 INFO:teuthology.orchestra.run.smithi023.stdout: "last_clean_end": 0, 2024-03-11T21:08:56.162 INFO:teuthology.orchestra.run.smithi023.stdout: "up_from": 28, 2024-03-11T21:08:56.162 INFO:teuthology.orchestra.run.smithi023.stdout: "up_thru": 319, 2024-03-11T21:08:56.162 INFO:teuthology.orchestra.run.smithi023.stdout: "down_at": 0, 2024-03-11T21:08:56.162 INFO:teuthology.orchestra.run.smithi023.stdout: "lost_at": 0, 2024-03-11T21:08:56.162 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2024-03-11T21:08:56.163 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:08:56.163 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:56.163 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:08:56.163 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6826", 2024-03-11T21:08:56.163 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 938496814 2024-03-11T21:08:56.163 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:56.163 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:56.163 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:08:56.163 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6827", 2024-03-11T21:08:56.163 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 938496814 2024-03-11T21:08:56.163 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:08:56.163 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:08:56.163 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:56.163 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_addrs": { 2024-03-11T21:08:56.163 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:08:56.164 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:56.164 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:08:56.164 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6828", 2024-03-11T21:08:56.164 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 938496814 2024-03-11T21:08:56.164 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:56.164 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:56.164 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:08:56.164 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6829", 2024-03-11T21:08:56.164 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 938496814 2024-03-11T21:08:56.164 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:08:56.164 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:08:56.164 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:56.164 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_back_addrs": { 2024-03-11T21:08:56.164 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:08:56.164 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:56.165 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:08:56.165 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6832", 2024-03-11T21:08:56.165 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 938496814 2024-03-11T21:08:56.165 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:56.165 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:56.165 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:08:56.165 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6833", 2024-03-11T21:08:56.165 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 938496814 2024-03-11T21:08:56.165 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:08:56.165 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:08:56.165 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:56.165 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_front_addrs": { 2024-03-11T21:08:56.165 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:08:56.165 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:56.165 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:08:56.166 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6830", 2024-03-11T21:08:56.166 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 938496814 2024-03-11T21:08:56.166 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:56.166 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:56.166 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:08:56.166 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6831", 2024-03-11T21:08:56.166 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 938496814 2024-03-11T21:08:56.166 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:08:56.166 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:08:56.166 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:56.166 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6827/938496814", 2024-03-11T21:08:56.166 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_addr": "172.21.15.23:6829/938496814", 2024-03-11T21:08:56.166 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_back_addr": "172.21.15.23:6833/938496814", 2024-03-11T21:08:56.166 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_front_addr": "172.21.15.23:6831/938496814", 2024-03-11T21:08:56.166 INFO:teuthology.orchestra.run.smithi023.stdout: "state": [ 2024-03-11T21:08:56.167 INFO:teuthology.orchestra.run.smithi023.stdout: "exists", 2024-03-11T21:08:56.167 INFO:teuthology.orchestra.run.smithi023.stdout: "up" 2024-03-11T21:08:56.167 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:08:56.167 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:56.167 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:56.167 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": 4, 2024-03-11T21:08:56.167 INFO:teuthology.orchestra.run.smithi023.stdout: "uuid": "4dcbc91e-7035-413d-97bc-c9d804eb16d7", 2024-03-11T21:08:56.167 INFO:teuthology.orchestra.run.smithi023.stdout: "up": 1, 2024-03-11T21:08:56.167 INFO:teuthology.orchestra.run.smithi023.stdout: "in": 1, 2024-03-11T21:08:56.167 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 1, 2024-03-11T21:08:56.167 INFO:teuthology.orchestra.run.smithi023.stdout: "primary_affinity": 0, 2024-03-11T21:08:56.167 INFO:teuthology.orchestra.run.smithi023.stdout: "last_clean_begin": 35, 2024-03-11T21:08:56.167 INFO:teuthology.orchestra.run.smithi023.stdout: "last_clean_end": 182, 2024-03-11T21:08:56.167 INFO:teuthology.orchestra.run.smithi023.stdout: "up_from": 204, 2024-03-11T21:08:56.168 INFO:teuthology.orchestra.run.smithi023.stdout: "up_thru": 248, 2024-03-11T21:08:56.168 INFO:teuthology.orchestra.run.smithi023.stdout: "down_at": 183, 2024-03-11T21:08:56.168 INFO:teuthology.orchestra.run.smithi023.stdout: "lost_at": 0, 2024-03-11T21:08:56.168 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2024-03-11T21:08:56.168 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:08:56.168 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:56.168 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:08:56.168 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6800", 2024-03-11T21:08:56.168 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 3858359434 2024-03-11T21:08:56.168 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:56.168 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:56.168 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:08:56.168 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6801", 2024-03-11T21:08:56.168 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 3858359434 2024-03-11T21:08:56.168 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:08:56.169 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:08:56.169 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:56.169 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_addrs": { 2024-03-11T21:08:56.169 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:08:56.169 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:56.169 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:08:56.169 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6802", 2024-03-11T21:08:56.169 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 3858359434 2024-03-11T21:08:56.169 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:56.169 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:56.169 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:08:56.169 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6803", 2024-03-11T21:08:56.169 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 3858359434 2024-03-11T21:08:56.169 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:08:56.170 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:08:56.170 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:56.170 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_back_addrs": { 2024-03-11T21:08:56.170 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:08:56.170 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:56.170 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:08:56.170 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6806", 2024-03-11T21:08:56.170 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 3858359434 2024-03-11T21:08:56.170 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:56.170 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:56.170 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:08:56.170 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6807", 2024-03-11T21:08:56.170 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 3858359434 2024-03-11T21:08:56.170 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:08:56.170 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:08:56.171 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:56.171 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_front_addrs": { 2024-03-11T21:08:56.171 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:08:56.171 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:56.171 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:08:56.171 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6804", 2024-03-11T21:08:56.171 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 3858359434 2024-03-11T21:08:56.171 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:56.171 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:56.171 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:08:56.171 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6805", 2024-03-11T21:08:56.171 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 3858359434 2024-03-11T21:08:56.171 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:08:56.171 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:08:56.172 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:56.172 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.146:6801/3858359434", 2024-03-11T21:08:56.172 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_addr": "172.21.15.146:6803/3858359434", 2024-03-11T21:08:56.172 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_back_addr": "172.21.15.146:6807/3858359434", 2024-03-11T21:08:56.172 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_front_addr": "172.21.15.146:6805/3858359434", 2024-03-11T21:08:56.172 INFO:teuthology.orchestra.run.smithi023.stdout: "state": [ 2024-03-11T21:08:56.172 INFO:teuthology.orchestra.run.smithi023.stdout: "exists", 2024-03-11T21:08:56.172 INFO:teuthology.orchestra.run.smithi023.stdout: "up" 2024-03-11T21:08:56.172 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:08:56.172 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:56.172 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:56.172 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": 5, 2024-03-11T21:08:56.172 INFO:teuthology.orchestra.run.smithi023.stdout: "uuid": "aefcb766-8bae-4ff4-b71e-6691c8ac02f1", 2024-03-11T21:08:56.172 INFO:teuthology.orchestra.run.smithi023.stdout: "up": 1, 2024-03-11T21:08:56.172 INFO:teuthology.orchestra.run.smithi023.stdout: "in": 1, 2024-03-11T21:08:56.173 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 1, 2024-03-11T21:08:56.173 INFO:teuthology.orchestra.run.smithi023.stdout: "primary_affinity": 1, 2024-03-11T21:08:56.173 INFO:teuthology.orchestra.run.smithi023.stdout: "last_clean_begin": 0, 2024-03-11T21:08:56.173 INFO:teuthology.orchestra.run.smithi023.stdout: "last_clean_end": 0, 2024-03-11T21:08:56.173 INFO:teuthology.orchestra.run.smithi023.stdout: "up_from": 44, 2024-03-11T21:08:56.173 INFO:teuthology.orchestra.run.smithi023.stdout: "up_thru": 319, 2024-03-11T21:08:56.173 INFO:teuthology.orchestra.run.smithi023.stdout: "down_at": 0, 2024-03-11T21:08:56.173 INFO:teuthology.orchestra.run.smithi023.stdout: "lost_at": 0, 2024-03-11T21:08:56.173 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2024-03-11T21:08:56.173 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:08:56.173 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:56.173 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:08:56.173 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6808", 2024-03-11T21:08:56.173 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 4003230814 2024-03-11T21:08:56.174 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:56.174 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:56.174 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:08:56.174 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6809", 2024-03-11T21:08:56.174 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 4003230814 2024-03-11T21:08:56.174 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:08:56.174 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:08:56.174 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:56.174 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_addrs": { 2024-03-11T21:08:56.174 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:08:56.174 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:56.174 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:08:56.174 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6810", 2024-03-11T21:08:56.174 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 4003230814 2024-03-11T21:08:56.174 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:56.175 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:56.175 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:08:56.175 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6811", 2024-03-11T21:08:56.175 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 4003230814 2024-03-11T21:08:56.175 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:08:56.175 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:08:56.175 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:56.175 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_back_addrs": { 2024-03-11T21:08:56.175 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:08:56.175 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:56.175 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:08:56.175 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6814", 2024-03-11T21:08:56.175 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 4003230814 2024-03-11T21:08:56.175 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:56.176 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:56.176 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:08:56.176 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6815", 2024-03-11T21:08:56.176 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 4003230814 2024-03-11T21:08:56.176 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:08:56.176 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:08:56.176 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:56.176 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_front_addrs": { 2024-03-11T21:08:56.176 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:08:56.176 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:56.176 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:08:56.176 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6812", 2024-03-11T21:08:56.176 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 4003230814 2024-03-11T21:08:56.176 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:56.177 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:56.177 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:08:56.177 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6813", 2024-03-11T21:08:56.177 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 4003230814 2024-03-11T21:08:56.177 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:08:56.177 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:08:56.177 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:56.177 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.146:6809/4003230814", 2024-03-11T21:08:56.177 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_addr": "172.21.15.146:6811/4003230814", 2024-03-11T21:08:56.177 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_back_addr": "172.21.15.146:6815/4003230814", 2024-03-11T21:08:56.177 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_front_addr": "172.21.15.146:6813/4003230814", 2024-03-11T21:08:56.177 INFO:teuthology.orchestra.run.smithi023.stdout: "state": [ 2024-03-11T21:08:56.177 INFO:teuthology.orchestra.run.smithi023.stdout: "exists", 2024-03-11T21:08:56.177 INFO:teuthology.orchestra.run.smithi023.stdout: "up" 2024-03-11T21:08:56.177 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:08:56.178 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:56.178 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:56.178 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": 6, 2024-03-11T21:08:56.178 INFO:teuthology.orchestra.run.smithi023.stdout: "uuid": "5cb151e8-1f9b-4617-8a86-adfb06bb9606", 2024-03-11T21:08:56.178 INFO:teuthology.orchestra.run.smithi023.stdout: "up": 1, 2024-03-11T21:08:56.178 INFO:teuthology.orchestra.run.smithi023.stdout: "in": 1, 2024-03-11T21:08:56.178 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 1, 2024-03-11T21:08:56.178 INFO:teuthology.orchestra.run.smithi023.stdout: "primary_affinity": 1, 2024-03-11T21:08:56.178 INFO:teuthology.orchestra.run.smithi023.stdout: "last_clean_begin": 0, 2024-03-11T21:08:56.178 INFO:teuthology.orchestra.run.smithi023.stdout: "last_clean_end": 0, 2024-03-11T21:08:56.178 INFO:teuthology.orchestra.run.smithi023.stdout: "up_from": 57, 2024-03-11T21:08:56.178 INFO:teuthology.orchestra.run.smithi023.stdout: "up_thru": 306, 2024-03-11T21:08:56.178 INFO:teuthology.orchestra.run.smithi023.stdout: "down_at": 0, 2024-03-11T21:08:56.178 INFO:teuthology.orchestra.run.smithi023.stdout: "lost_at": 0, 2024-03-11T21:08:56.178 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2024-03-11T21:08:56.179 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:08:56.179 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:56.179 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:08:56.179 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6816", 2024-03-11T21:08:56.179 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 3334092376 2024-03-11T21:08:56.179 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:56.179 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:56.179 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:08:56.179 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6817", 2024-03-11T21:08:56.179 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 3334092376 2024-03-11T21:08:56.179 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:08:56.179 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:08:56.179 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:56.179 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_addrs": { 2024-03-11T21:08:56.179 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:08:56.180 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:56.180 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:08:56.180 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6818", 2024-03-11T21:08:56.180 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 3334092376 2024-03-11T21:08:56.180 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:56.180 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:56.180 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:08:56.180 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6819", 2024-03-11T21:08:56.180 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 3334092376 2024-03-11T21:08:56.180 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:08:56.180 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:08:56.180 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:56.180 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_back_addrs": { 2024-03-11T21:08:56.180 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:08:56.180 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:56.181 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:08:56.181 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6822", 2024-03-11T21:08:56.181 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 3334092376 2024-03-11T21:08:56.181 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:56.181 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:56.181 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:08:56.181 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6823", 2024-03-11T21:08:56.181 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 3334092376 2024-03-11T21:08:56.181 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:08:56.181 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:08:56.181 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:56.181 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_front_addrs": { 2024-03-11T21:08:56.181 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:08:56.181 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:56.181 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:08:56.182 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6820", 2024-03-11T21:08:56.182 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 3334092376 2024-03-11T21:08:56.182 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:56.182 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:56.182 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:08:56.182 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6821", 2024-03-11T21:08:56.182 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 3334092376 2024-03-11T21:08:56.182 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:08:56.182 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:08:56.182 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:56.182 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.146:6817/3334092376", 2024-03-11T21:08:56.182 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_addr": "172.21.15.146:6819/3334092376", 2024-03-11T21:08:56.182 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_back_addr": "172.21.15.146:6823/3334092376", 2024-03-11T21:08:56.182 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_front_addr": "172.21.15.146:6821/3334092376", 2024-03-11T21:08:56.182 INFO:teuthology.orchestra.run.smithi023.stdout: "state": [ 2024-03-11T21:08:56.183 INFO:teuthology.orchestra.run.smithi023.stdout: "exists", 2024-03-11T21:08:56.183 INFO:teuthology.orchestra.run.smithi023.stdout: "up" 2024-03-11T21:08:56.183 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:08:56.183 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:56.183 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:56.183 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": 7, 2024-03-11T21:08:56.183 INFO:teuthology.orchestra.run.smithi023.stdout: "uuid": "bdbd11a2-fa09-4f82-93a2-6b2d9c750bfd", 2024-03-11T21:08:56.183 INFO:teuthology.orchestra.run.smithi023.stdout: "up": 1, 2024-03-11T21:08:56.183 INFO:teuthology.orchestra.run.smithi023.stdout: "in": 1, 2024-03-11T21:08:56.183 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 1, 2024-03-11T21:08:56.183 INFO:teuthology.orchestra.run.smithi023.stdout: "primary_affinity": 1, 2024-03-11T21:08:56.183 INFO:teuthology.orchestra.run.smithi023.stdout: "last_clean_begin": 0, 2024-03-11T21:08:56.183 INFO:teuthology.orchestra.run.smithi023.stdout: "last_clean_end": 0, 2024-03-11T21:08:56.183 INFO:teuthology.orchestra.run.smithi023.stdout: "up_from": 68, 2024-03-11T21:08:56.184 INFO:teuthology.orchestra.run.smithi023.stdout: "up_thru": 319, 2024-03-11T21:08:56.184 INFO:teuthology.orchestra.run.smithi023.stdout: "down_at": 0, 2024-03-11T21:08:56.184 INFO:teuthology.orchestra.run.smithi023.stdout: "lost_at": 0, 2024-03-11T21:08:56.184 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2024-03-11T21:08:56.184 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:08:56.184 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:56.184 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:08:56.184 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6824", 2024-03-11T21:08:56.184 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1678407923 2024-03-11T21:08:56.184 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:56.184 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:56.184 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:08:56.184 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6825", 2024-03-11T21:08:56.184 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1678407923 2024-03-11T21:08:56.184 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:08:56.185 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:08:56.185 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:56.185 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_addrs": { 2024-03-11T21:08:56.185 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:08:56.185 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:56.185 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:08:56.185 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6826", 2024-03-11T21:08:56.185 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1678407923 2024-03-11T21:08:56.185 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:56.185 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:56.185 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:08:56.185 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6827", 2024-03-11T21:08:56.185 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1678407923 2024-03-11T21:08:56.185 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:08:56.186 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:08:56.186 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:56.186 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_back_addrs": { 2024-03-11T21:08:56.186 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:08:56.186 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:56.186 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:08:56.186 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6830", 2024-03-11T21:08:56.186 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1678407923 2024-03-11T21:08:56.186 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:56.186 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:56.186 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:08:56.186 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6831", 2024-03-11T21:08:56.186 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1678407923 2024-03-11T21:08:56.186 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:08:56.187 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:08:56.187 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:56.187 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_front_addrs": { 2024-03-11T21:08:56.187 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:08:56.187 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:56.187 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:08:56.187 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6828", 2024-03-11T21:08:56.187 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1678407923 2024-03-11T21:08:56.187 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:56.187 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:56.187 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:08:56.187 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6829", 2024-03-11T21:08:56.187 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1678407923 2024-03-11T21:08:56.187 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:08:56.187 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:08:56.188 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:56.188 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.146:6825/1678407923", 2024-03-11T21:08:56.188 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_addr": "172.21.15.146:6827/1678407923", 2024-03-11T21:08:56.188 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_back_addr": "172.21.15.146:6831/1678407923", 2024-03-11T21:08:56.188 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_front_addr": "172.21.15.146:6829/1678407923", 2024-03-11T21:08:56.188 INFO:teuthology.orchestra.run.smithi023.stdout: "state": [ 2024-03-11T21:08:56.188 INFO:teuthology.orchestra.run.smithi023.stdout: "exists", 2024-03-11T21:08:56.188 INFO:teuthology.orchestra.run.smithi023.stdout: "up" 2024-03-11T21:08:56.188 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:08:56.188 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:08:56.188 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-11T21:08:56.188 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_xinfo": [ 2024-03-11T21:08:56.188 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:56.188 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": 0, 2024-03-11T21:08:56.189 INFO:teuthology.orchestra.run.smithi023.stdout: "down_stamp": "0.000000", 2024-03-11T21:08:56.189 INFO:teuthology.orchestra.run.smithi023.stdout: "laggy_probability": 0, 2024-03-11T21:08:56.189 INFO:teuthology.orchestra.run.smithi023.stdout: "laggy_interval": 0, 2024-03-11T21:08:56.189 INFO:teuthology.orchestra.run.smithi023.stdout: "features": 4540701547738038271, 2024-03-11T21:08:56.189 INFO:teuthology.orchestra.run.smithi023.stdout: "old_weight": 0, 2024-03-11T21:08:56.189 INFO:teuthology.orchestra.run.smithi023.stdout: "last_purged_snaps_scrub": "2024-03-11T21:08:03.814842+0000", 2024-03-11T21:08:56.189 INFO:teuthology.orchestra.run.smithi023.stdout: "dead_epoch": 0 2024-03-11T21:08:56.189 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:56.189 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:56.189 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": 1, 2024-03-11T21:08:56.189 INFO:teuthology.orchestra.run.smithi023.stdout: "down_stamp": "0.000000", 2024-03-11T21:08:56.189 INFO:teuthology.orchestra.run.smithi023.stdout: "laggy_probability": 0, 2024-03-11T21:08:56.189 INFO:teuthology.orchestra.run.smithi023.stdout: "laggy_interval": 0, 2024-03-11T21:08:56.189 INFO:teuthology.orchestra.run.smithi023.stdout: "features": 4540701547738038271, 2024-03-11T21:08:56.189 INFO:teuthology.orchestra.run.smithi023.stdout: "old_weight": 0, 2024-03-11T21:08:56.190 INFO:teuthology.orchestra.run.smithi023.stdout: "last_purged_snaps_scrub": "2024-03-11T21:08:21.457450+0000", 2024-03-11T21:08:56.190 INFO:teuthology.orchestra.run.smithi023.stdout: "dead_epoch": 173 2024-03-11T21:08:56.190 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:56.190 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:56.190 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": 2, 2024-03-11T21:08:56.190 INFO:teuthology.orchestra.run.smithi023.stdout: "down_stamp": "0.000000", 2024-03-11T21:08:56.190 INFO:teuthology.orchestra.run.smithi023.stdout: "laggy_probability": 0, 2024-03-11T21:08:56.190 INFO:teuthology.orchestra.run.smithi023.stdout: "laggy_interval": 0, 2024-03-11T21:08:56.190 INFO:teuthology.orchestra.run.smithi023.stdout: "features": 4540701547738038271, 2024-03-11T21:08:56.190 INFO:teuthology.orchestra.run.smithi023.stdout: "old_weight": 0, 2024-03-11T21:08:56.190 INFO:teuthology.orchestra.run.smithi023.stdout: "last_purged_snaps_scrub": "2024-03-11T21:08:13.519812+0000", 2024-03-11T21:08:56.190 INFO:teuthology.orchestra.run.smithi023.stdout: "dead_epoch": 0 2024-03-11T21:08:56.190 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:56.190 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:56.191 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": 3, 2024-03-11T21:08:56.191 INFO:teuthology.orchestra.run.smithi023.stdout: "down_stamp": "0.000000", 2024-03-11T21:08:56.191 INFO:teuthology.orchestra.run.smithi023.stdout: "laggy_probability": 0, 2024-03-11T21:08:56.191 INFO:teuthology.orchestra.run.smithi023.stdout: "laggy_interval": 0, 2024-03-11T21:08:56.191 INFO:teuthology.orchestra.run.smithi023.stdout: "features": 4540701547738038271, 2024-03-11T21:08:56.191 INFO:teuthology.orchestra.run.smithi023.stdout: "old_weight": 0, 2024-03-11T21:08:56.191 INFO:teuthology.orchestra.run.smithi023.stdout: "last_purged_snaps_scrub": "2024-03-11T21:08:23.899523+0000", 2024-03-11T21:08:56.191 INFO:teuthology.orchestra.run.smithi023.stdout: "dead_epoch": 0 2024-03-11T21:08:56.191 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:56.191 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:56.191 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": 4, 2024-03-11T21:08:56.191 INFO:teuthology.orchestra.run.smithi023.stdout: "down_stamp": "0.000000", 2024-03-11T21:08:56.191 INFO:teuthology.orchestra.run.smithi023.stdout: "laggy_probability": 0, 2024-03-11T21:08:56.192 INFO:teuthology.orchestra.run.smithi023.stdout: "laggy_interval": 0, 2024-03-11T21:08:56.192 INFO:teuthology.orchestra.run.smithi023.stdout: "features": 4540701547738038271, 2024-03-11T21:08:56.192 INFO:teuthology.orchestra.run.smithi023.stdout: "old_weight": 0, 2024-03-11T21:08:56.192 INFO:teuthology.orchestra.run.smithi023.stdout: "last_purged_snaps_scrub": "2024-03-11T21:08:42.219924+0000", 2024-03-11T21:08:56.192 INFO:teuthology.orchestra.run.smithi023.stdout: "dead_epoch": 182 2024-03-11T21:08:56.192 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:56.192 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:56.192 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": 5, 2024-03-11T21:08:56.192 INFO:teuthology.orchestra.run.smithi023.stdout: "down_stamp": "0.000000", 2024-03-11T21:08:56.192 INFO:teuthology.orchestra.run.smithi023.stdout: "laggy_probability": 0, 2024-03-11T21:08:56.192 INFO:teuthology.orchestra.run.smithi023.stdout: "laggy_interval": 0, 2024-03-11T21:08:56.192 INFO:teuthology.orchestra.run.smithi023.stdout: "features": 4540701547738038271, 2024-03-11T21:08:56.192 INFO:teuthology.orchestra.run.smithi023.stdout: "old_weight": 0, 2024-03-11T21:08:56.192 INFO:teuthology.orchestra.run.smithi023.stdout: "last_purged_snaps_scrub": "2024-03-11T21:08:04.821815+0000", 2024-03-11T21:08:56.192 INFO:teuthology.orchestra.run.smithi023.stdout: "dead_epoch": 0 2024-03-11T21:08:56.193 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:56.193 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:56.193 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": 6, 2024-03-11T21:08:56.193 INFO:teuthology.orchestra.run.smithi023.stdout: "down_stamp": "0.000000", 2024-03-11T21:08:56.193 INFO:teuthology.orchestra.run.smithi023.stdout: "laggy_probability": 0, 2024-03-11T21:08:56.193 INFO:teuthology.orchestra.run.smithi023.stdout: "laggy_interval": 0, 2024-03-11T21:08:56.193 INFO:teuthology.orchestra.run.smithi023.stdout: "features": 4540701547738038271, 2024-03-11T21:08:56.193 INFO:teuthology.orchestra.run.smithi023.stdout: "old_weight": 0, 2024-03-11T21:08:56.193 INFO:teuthology.orchestra.run.smithi023.stdout: "last_purged_snaps_scrub": "2024-03-11T21:08:45.838074+0000", 2024-03-11T21:08:56.193 INFO:teuthology.orchestra.run.smithi023.stdout: "dead_epoch": 0 2024-03-11T21:08:56.193 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:56.193 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:56.193 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": 7, 2024-03-11T21:08:56.193 INFO:teuthology.orchestra.run.smithi023.stdout: "down_stamp": "0.000000", 2024-03-11T21:08:56.193 INFO:teuthology.orchestra.run.smithi023.stdout: "laggy_probability": 0, 2024-03-11T21:08:56.194 INFO:teuthology.orchestra.run.smithi023.stdout: "laggy_interval": 0, 2024-03-11T21:08:56.194 INFO:teuthology.orchestra.run.smithi023.stdout: "features": 4540701547738038271, 2024-03-11T21:08:56.194 INFO:teuthology.orchestra.run.smithi023.stdout: "old_weight": 0, 2024-03-11T21:08:56.194 INFO:teuthology.orchestra.run.smithi023.stdout: "last_purged_snaps_scrub": "2024-03-11T21:07:57.618823+0000", 2024-03-11T21:08:56.194 INFO:teuthology.orchestra.run.smithi023.stdout: "dead_epoch": 0 2024-03-11T21:08:56.194 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:08:56.194 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-11T21:08:56.194 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_upmap": [ 2024-03-11T21:08:56.194 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:56.194 INFO:teuthology.orchestra.run.smithi023.stdout: "pgid": "6.a", 2024-03-11T21:08:56.194 INFO:teuthology.orchestra.run.smithi023.stdout: "osds": [ 2024-03-11T21:08:56.194 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2024-03-11T21:08:56.194 INFO:teuthology.orchestra.run.smithi023.stdout: 4, 2024-03-11T21:08:56.194 INFO:teuthology.orchestra.run.smithi023.stdout: 7 2024-03-11T21:08:56.194 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:08:56.195 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:08:56.195 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-11T21:08:56.195 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_upmap_items": [ 2024-03-11T21:08:56.195 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:56.195 INFO:teuthology.orchestra.run.smithi023.stdout: "pgid": "2.0", 2024-03-11T21:08:56.195 INFO:teuthology.orchestra.run.smithi023.stdout: "mappings": [ 2024-03-11T21:08:56.195 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:56.195 INFO:teuthology.orchestra.run.smithi023.stdout: "from": 7, 2024-03-11T21:08:56.195 INFO:teuthology.orchestra.run.smithi023.stdout: "to": 2 2024-03-11T21:08:56.195 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:08:56.195 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:08:56.195 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:56.195 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:56.195 INFO:teuthology.orchestra.run.smithi023.stdout: "pgid": "4.b", 2024-03-11T21:08:56.195 INFO:teuthology.orchestra.run.smithi023.stdout: "mappings": [ 2024-03-11T21:08:56.196 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:56.196 INFO:teuthology.orchestra.run.smithi023.stdout: "from": 1, 2024-03-11T21:08:56.196 INFO:teuthology.orchestra.run.smithi023.stdout: "to": 2 2024-03-11T21:08:56.196 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:08:56.196 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:08:56.196 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:56.196 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:56.196 INFO:teuthology.orchestra.run.smithi023.stdout: "pgid": "4.d", 2024-03-11T21:08:56.196 INFO:teuthology.orchestra.run.smithi023.stdout: "mappings": [ 2024-03-11T21:08:56.196 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:56.196 INFO:teuthology.orchestra.run.smithi023.stdout: "from": 1, 2024-03-11T21:08:56.196 INFO:teuthology.orchestra.run.smithi023.stdout: "to": 5 2024-03-11T21:08:56.196 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:08:56.196 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:08:56.196 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:56.197 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:56.197 INFO:teuthology.orchestra.run.smithi023.stdout: "pgid": "4.e", 2024-03-11T21:08:56.197 INFO:teuthology.orchestra.run.smithi023.stdout: "mappings": [ 2024-03-11T21:08:56.197 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:56.197 INFO:teuthology.orchestra.run.smithi023.stdout: "from": 1, 2024-03-11T21:08:56.197 INFO:teuthology.orchestra.run.smithi023.stdout: "to": 2 2024-03-11T21:08:56.197 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:08:56.197 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:08:56.197 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:56.197 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:56.197 INFO:teuthology.orchestra.run.smithi023.stdout: "pgid": "4.16", 2024-03-11T21:08:56.197 INFO:teuthology.orchestra.run.smithi023.stdout: "mappings": [ 2024-03-11T21:08:56.197 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:56.197 INFO:teuthology.orchestra.run.smithi023.stdout: "from": 3, 2024-03-11T21:08:56.197 INFO:teuthology.orchestra.run.smithi023.stdout: "to": 2 2024-03-11T21:08:56.197 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:08:56.198 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:08:56.198 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:56.198 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:56.198 INFO:teuthology.orchestra.run.smithi023.stdout: "pgid": "4.1d", 2024-03-11T21:08:56.198 INFO:teuthology.orchestra.run.smithi023.stdout: "mappings": [ 2024-03-11T21:08:56.198 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:08:56.198 INFO:teuthology.orchestra.run.smithi023.stdout: "from": 3, 2024-03-11T21:08:56.198 INFO:teuthology.orchestra.run.smithi023.stdout: "to": 5 2024-03-11T21:08:56.198 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:08:56.198 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:08:56.198 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:08:56.198 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-11T21:08:56.198 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_upmap_primaries": [], 2024-03-11T21:08:56.198 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_temp": [], 2024-03-11T21:08:56.199 INFO:teuthology.orchestra.run.smithi023.stdout: "primary_temp": [], 2024-03-11T21:08:56.199 INFO:teuthology.orchestra.run.smithi023.stdout: "blocklist": { 2024-03-11T21:08:56.199 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:0/1618334186": "2024-03-12T20:40:45.361537+0000", 2024-03-11T21:08:56.199 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:6801/2813578081": "2024-03-12T20:40:45.361537+0000", 2024-03-11T21:08:56.199 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:0/2869940180": "2024-03-12T20:51:33.790205+0000", 2024-03-11T21:08:56.199 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:6800/3037564930": "2024-03-12T20:51:33.790205+0000", 2024-03-11T21:08:56.199 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:0/3297567084": "2024-03-12T20:51:33.790205+0000", 2024-03-11T21:08:56.199 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:0/3708753659": "2024-03-12T20:40:45.361537+0000", 2024-03-11T21:08:56.199 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:0/2004146740": "2024-03-12T20:40:45.361537+0000", 2024-03-11T21:08:56.199 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:0/827653812": "2024-03-12T20:41:21.301516+0000", 2024-03-11T21:08:56.199 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:6800/1057699850": "2024-03-12T20:41:21.301516+0000", 2024-03-11T21:08:56.199 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:6800/2813578081": "2024-03-12T20:40:45.361537+0000", 2024-03-11T21:08:56.199 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:6801/1057699850": "2024-03-12T20:41:21.301516+0000", 2024-03-11T21:08:56.199 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:0/3833230375": "2024-03-12T20:41:21.301516+0000", 2024-03-11T21:08:56.199 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:6801/3037564930": "2024-03-12T20:51:33.790205+0000", 2024-03-11T21:08:56.199 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:0/1182809102": "2024-03-12T20:51:33.790205+0000", 2024-03-11T21:08:56.200 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:0/2122134187": "2024-03-12T20:51:33.790205+0000", 2024-03-11T21:08:56.200 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:0/2356701899": "2024-03-12T20:41:21.301516+0000", 2024-03-11T21:08:56.200 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:0/843974100": "2024-03-12T20:51:33.790205+0000" 2024-03-11T21:08:56.200 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:56.200 INFO:teuthology.orchestra.run.smithi023.stdout: "range_blocklist": {}, 2024-03-11T21:08:56.200 INFO:teuthology.orchestra.run.smithi023.stdout: "erasure_code_profiles": { 2024-03-11T21:08:56.200 INFO:teuthology.orchestra.run.smithi023.stdout: "default": { 2024-03-11T21:08:56.200 INFO:teuthology.orchestra.run.smithi023.stdout: "crush-failure-domain": "osd", 2024-03-11T21:08:56.200 INFO:teuthology.orchestra.run.smithi023.stdout: "k": "2", 2024-03-11T21:08:56.200 INFO:teuthology.orchestra.run.smithi023.stdout: "m": "1", 2024-03-11T21:08:56.200 INFO:teuthology.orchestra.run.smithi023.stdout: "plugin": "jerasure", 2024-03-11T21:08:56.200 INFO:teuthology.orchestra.run.smithi023.stdout: "technique": "reed_sol_van" 2024-03-11T21:08:56.200 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:08:56.200 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:08:56.200 INFO:teuthology.orchestra.run.smithi023.stdout: "removed_snaps_queue": [], 2024-03-11T21:08:56.201 INFO:teuthology.orchestra.run.smithi023.stdout: "new_removed_snaps": [], 2024-03-11T21:08:56.201 INFO:teuthology.orchestra.run.smithi023.stdout: "new_purged_snaps": [], 2024-03-11T21:08:56.201 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_node_flags": {}, 2024-03-11T21:08:56.201 INFO:teuthology.orchestra.run.smithi023.stdout: "device_class_flags": {}, 2024-03-11T21:08:56.201 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": { 2024-03-11T21:08:56.201 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode_enabled": false, 2024-03-11T21:08:56.201 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_bucket_count": 0, 2024-03-11T21:08:56.201 INFO:teuthology.orchestra.run.smithi023.stdout: "degraded_stretch_mode": 0, 2024-03-11T21:08:56.201 INFO:teuthology.orchestra.run.smithi023.stdout: "recovering_stretch_mode": 0, 2024-03-11T21:08:56.201 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode_bucket": 0 2024-03-11T21:08:56.201 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:08:56.201 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-11T21:08:56.202 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:56.227 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:56.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:56.230 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:08:56.331 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:56.355 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:56.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:56.360 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:08:56.377 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:2024-03-11T21:08:56.386834+0000 min lat: 0.00119683 max lat: 1.06642 avg lat: 0.0128495 2024-03-11T21:08:56.377 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-11T21:08:56.377 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 80 16 99607 99591 77.793 61.125 0.0197655 0.0128495 2024-03-11T21:08:56.461 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:08:56.471 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:56 smithi146 bash[23809]: audit 2024-03-11T21:08:56.065847+0000 mon.a (mon.0) 1673 : audit 0 from='client.? 172.21.15.23:0/3679105572' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-11T21:08:56.489 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:08:56.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:56.491 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:08:56.592 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:08:56.630 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:08:56.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:56.637 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:08:56.738 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:56.768 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:56.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:56.773 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:08:56.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:56 smithi023 bash[21372]: audit 2024-03-11T21:08:56.065847+0000 mon.a (mon.0) 1673 : audit 0 from='client.? 172.21.15.23:0/3679105572' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-11T21:08:56.779 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:56 smithi023 bash[29146]: audit 2024-03-11T21:08:56.065847+0000 mon.a (mon.0) 1673 : audit 0 from='client.? 172.21.15.23:0/3679105572' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-11T21:08:56.873 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:56.906 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:56.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:56.908 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:08:57.009 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:57.032 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:57.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:57.036 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:08:57.085 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T21:08:57.113 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd unset noscrub 2024-03-11T21:08:57.137 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:57.204 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:57.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:57.209 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:08:57.309 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:08:57.337 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:08:57.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:57.347 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:08:57.379 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 81 16 100587 100571 77.5865 61.25 0.0122561 0.0128838 2024-03-11T21:08:57.398 INFO:tasks.thrashosds.thrasher:j is {'epoch': 334, 'fsid': '67ba00f8-dfe7-11ee-95c8-87774f69a715', 'created': '2024-03-11T20:40:13.664802+0000', 'modified': '2024-03-11T21:08:55.295434+0000', 'last_up_change': '2024-03-11T21:00:47.324817+0000', 'last_in_change': '2024-03-11T20:57:23.560527+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 20, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 10, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-11T20:45:31.964570+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '142', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6, 'score_stable': 6, 'optimal_score': 0.5, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.75, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.75}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2024-03-11T20:50:26.421486+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '78', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1.3300000429153442, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.75, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.75}}, {'pool': 3, 'pool_name': 'default.rgw.log', 'create_time': '2024-03-11T20:50:28.592615+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1.3300000429153442, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.75, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.75}}, {'pool': 4, 'pool_name': 'default.rgw.control', 'create_time': '2024-03-11T20:50:30.536371+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.690000057220459, 'score_stable': 1.690000057220459, 'optimal_score': 1.3300000429153442, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.75, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.75}}, {'pool': 5, 'pool_name': 'datapool', 'create_time': '2024-03-11T20:50:31.135845+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 87, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3.990000009536743, 'score_stable': 3.990000009536743, 'optimal_score': 1.1699999570846558, 'raw_score_acting': 4.670000076293945, 'raw_score_stable': 4.670000076293945, 'primary_affinity_weighted': 0.75, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.75}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-03-11T20:50:33.233163+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.690000057220459, 'score_stable': 1.690000057220459, 'optimal_score': 1.3300000429153442, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.75, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.75}}, {'pool': 10, 'pool_name': 'unique_pool_3', 'create_time': '2024-03-11T21:07:13.796707+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '310', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.630000114440918, 'score_stable': 2.630000114440918, 'optimal_score': 1.3300000429153442, 'raw_score_acting': 3.5, 'raw_score_stable': 3.5, 'primary_affinity_weighted': 0.75, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.75}}], 'osds': [{'osd': 0, 'uuid': '3df72b4a-8fe4-4265-b4c6-af022e1e23e1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 319, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6802', 'nonce': 2154627393}, {'type': 'v1', 'addr': '172.21.15.23:6803', 'nonce': 2154627393}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6804', 'nonce': 2154627393}, {'type': 'v1', 'addr': '172.21.15.23:6805', 'nonce': 2154627393}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6808', 'nonce': 2154627393}, {'type': 'v1', 'addr': '172.21.15.23:6809', 'nonce': 2154627393}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6806', 'nonce': 2154627393}, {'type': 'v1', 'addr': '172.21.15.23:6807', 'nonce': 2154627393}]}, 'public_addr': '172.21.15.23:6803/2154627393', 'cluster_addr': '172.21.15.23:6805/2154627393', 'heartbeat_back_addr': '172.21.15.23:6809/2154627393', 'heartbeat_front_addr': '172.21.15.23:6807/2154627393', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'e8fd7500-9081-41a1-8077-a4d0f0e12a2c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 13, 'last_clean_end': 173, 'up_from': 211, 'up_thru': 319, 'down_at': 174, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6810', 'nonce': 56585354}, {'type': 'v1', 'addr': '172.21.15.23:6811', 'nonce': 56585354}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6812', 'nonce': 56585354}, {'type': 'v1', 'addr': '172.21.15.23:6813', 'nonce': 56585354}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6816', 'nonce': 56585354}, {'type': 'v1', 'addr': '172.21.15.23:6817', 'nonce': 56585354}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6814', 'nonce': 56585354}, {'type': 'v1', 'addr': '172.21.15.23:6815', 'nonce': 56585354}]}, 'public_addr': '172.21.15.23:6811/56585354', 'cluster_addr': '172.21.15.23:6813/56585354', 'heartbeat_back_addr': '172.21.15.23:6817/56585354', 'heartbeat_front_addr': '172.21.15.23:6815/56585354', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'dd85d52a-df48-4b03-8612-cecfd82b5987', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 19, 'up_thru': 306, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6818', 'nonce': 1770748056}, {'type': 'v1', 'addr': '172.21.15.23:6819', 'nonce': 1770748056}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6820', 'nonce': 1770748056}, {'type': 'v1', 'addr': '172.21.15.23:6821', 'nonce': 1770748056}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6824', 'nonce': 1770748056}, {'type': 'v1', 'addr': '172.21.15.23:6825', 'nonce': 1770748056}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6822', 'nonce': 1770748056}, {'type': 'v1', 'addr': '172.21.15.23:6823', 'nonce': 1770748056}]}, 'public_addr': '172.21.15.23:6819/1770748056', 'cluster_addr': '172.21.15.23:6821/1770748056', 'heartbeat_back_addr': '172.21.15.23:6825/1770748056', 'heartbeat_front_addr': '172.21.15.23:6823/1770748056', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'e3021106-3779-4b94-9bbf-a7724544110f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 28, 'up_thru': 319, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6826', 'nonce': 938496814}, {'type': 'v1', 'addr': '172.21.15.23:6827', 'nonce': 938496814}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6828', 'nonce': 938496814}, {'type': 'v1', 'addr': '172.21.15.23:6829', 'nonce': 938496814}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6832', 'nonce': 938496814}, {'type': 'v1', 'addr': '172.21.15.23:6833', 'nonce': 938496814}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6830', 'nonce': 938496814}, {'type': 'v1', 'addr': '172.21.15.23:6831', 'nonce': 938496814}]}, 'public_addr': '172.21.15.23:6827/938496814', 'cluster_addr': '172.21.15.23:6829/938496814', 'heartbeat_back_addr': '172.21.15.23:6833/938496814', 'heartbeat_front_addr': '172.21.15.23:6831/938496814', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '4dcbc91e-7035-413d-97bc-c9d804eb16d7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 35, 'last_clean_end': 182, 'up_from': 204, 'up_thru': 248, 'down_at': 183, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6800', 'nonce': 3858359434}, {'type': 'v1', 'addr': '172.21.15.146:6801', 'nonce': 3858359434}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6802', 'nonce': 3858359434}, {'type': 'v1', 'addr': '172.21.15.146:6803', 'nonce': 3858359434}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6806', 'nonce': 3858359434}, {'type': 'v1', 'addr': '172.21.15.146:6807', 'nonce': 3858359434}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6804', 'nonce': 3858359434}, {'type': 'v1', 'addr': '172.21.15.146:6805', 'nonce': 3858359434}]}, 'public_addr': '172.21.15.146:6801/3858359434', 'cluster_addr': '172.21.15.146:6803/3858359434', 'heartbeat_back_addr': '172.21.15.146:6807/3858359434', 'heartbeat_front_addr': '172.21.15.146:6805/3858359434', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'aefcb766-8bae-4ff4-b71e-6691c8ac02f1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 44, 'up_thru': 319, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6808', 'nonce': 4003230814}, {'type': 'v1', 'addr': '172.21.15.146:6809', 'nonce': 4003230814}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6810', 'nonce': 4003230814}, {'type': 'v1', 'addr': '172.21.15.146:6811', 'nonce': 4003230814}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6814', 'nonce': 4003230814}, {'type': 'v1', 'addr': '172.21.15.146:6815', 'nonce': 4003230814}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6812', 'nonce': 4003230814}, {'type': 'v1', 'addr': '172.21.15.146:6813', 'nonce': 4003230814}]}, 'public_addr': '172.21.15.146:6809/4003230814', 'cluster_addr': '172.21.15.146:6811/4003230814', 'heartbeat_back_addr': '172.21.15.146:6815/4003230814', 'heartbeat_front_addr': '172.21.15.146:6813/4003230814', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '5cb151e8-1f9b-4617-8a86-adfb06bb9606', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 57, 'up_thru': 306, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6816', 'nonce': 3334092376}, {'type': 'v1', 'addr': '172.21.15.146:6817', 'nonce': 3334092376}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6818', 'nonce': 3334092376}, {'type': 'v1', 'addr': '172.21.15.146:6819', 'nonce': 3334092376}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6822', 'nonce': 3334092376}, {'type': 'v1', 'addr': '172.21.15.146:6823', 'nonce': 3334092376}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6820', 'nonce': 3334092376}, {'type': 'v1', 'addr': '172.21.15.146:6821', 'nonce': 3334092376}]}, 'public_addr': '172.21.15.146:6817/3334092376', 'cluster_addr': '172.21.15.146:6819/3334092376', 'heartbeat_back_addr': '172.21.15.146:6823/3334092376', 'heartbeat_front_addr': '172.21.15.146:6821/3334092376', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'bdbd11a2-fa09-4f82-93a2-6b2d9c750bfd', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 68, 'up_thru': 319, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6824', 'nonce': 1678407923}, {'type': 'v1', 'addr': '172.21.15.146:6825', 'nonce': 1678407923}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6826', 'nonce': 1678407923}, {'type': 'v1', 'addr': '172.21.15.146:6827', 'nonce': 1678407923}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6830', 'nonce': 1678407923}, {'type': 'v1', 'addr': '172.21.15.146:6831', 'nonce': 1678407923}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6828', 'nonce': 1678407923}, {'type': 'v1', 'addr': '172.21.15.146:6829', 'nonce': 1678407923}]}, 'public_addr': '172.21.15.146:6825/1678407923', 'cluster_addr': '172.21.15.146:6827/1678407923', 'heartbeat_back_addr': '172.21.15.146:6831/1678407923', 'heartbeat_front_addr': '172.21.15.146:6829/1678407923', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T21:08:03.814842+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T21:08:21.457450+0000', 'dead_epoch': 173}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T21:08:13.519812+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T21:08:23.899523+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T21:08:42.219924+0000', 'dead_epoch': 182}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T21:08:04.821815+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T21:08:45.838074+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T21:07:57.618823+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '6.a', 'osds': [0, 4, 7]}], 'pg_upmap_items': [{'pgid': '2.0', 'mappings': [{'from': 7, 'to': 2}]}, {'pgid': '4.b', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.d', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.e', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.16', 'mappings': [{'from': 3, 'to': 2}]}, {'pgid': '4.1d', 'mappings': [{'from': 3, 'to': 5}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.23:0/1618334186': '2024-03-12T20:40:45.361537+0000', '172.21.15.23:6801/2813578081': '2024-03-12T20:40:45.361537+0000', '172.21.15.23:0/2869940180': '2024-03-12T20:51:33.790205+0000', '172.21.15.23:6800/3037564930': '2024-03-12T20:51:33.790205+0000', '172.21.15.23:0/3297567084': '2024-03-12T20:51:33.790205+0000', '172.21.15.23:0/3708753659': '2024-03-12T20:40:45.361537+0000', '172.21.15.23:0/2004146740': '2024-03-12T20:40:45.361537+0000', '172.21.15.23:0/827653812': '2024-03-12T20:41:21.301516+0000', '172.21.15.23:6800/1057699850': '2024-03-12T20:41:21.301516+0000', '172.21.15.23:6800/2813578081': '2024-03-12T20:40:45.361537+0000', '172.21.15.23:6801/1057699850': '2024-03-12T20:41:21.301516+0000', '172.21.15.23:0/3833230375': '2024-03-12T20:41:21.301516+0000', '172.21.15.23:6801/3037564930': '2024-03-12T20:51:33.790205+0000', '172.21.15.23:0/1182809102': '2024-03-12T20:51:33.790205+0000', '172.21.15.23:0/2122134187': '2024-03-12T20:51:33.790205+0000', '172.21.15.23:0/2356701899': '2024-03-12T20:41:21.301516+0000', '172.21.15.23:0/843974100': '2024-03-12T20:51:33.790205+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-03-11T21:08:57.399 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T21:08:57.448 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:57.532 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:57.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:57.539 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:08:57.640 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:57.653 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:57 smithi023 bash[21372]: audit 2024-03-11T21:08:55.751301+0000 mgr.y (mgr.24380) 671 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:08:57.653 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:57 smithi023 bash[21372]: cluster 2024-03-11T21:08:56.286549+0000 mgr.y (mgr.24380) 672 : cluster 0 pgmap v757: 148 pgs: 148 active+clean; 6.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 853 B/s rd, 55 MiB/s wr, 874 op/s 2024-03-11T21:08:57.653 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:57 smithi023 bash[29146]: audit 2024-03-11T21:08:55.751301+0000 mgr.y (mgr.24380) 671 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:08:57.654 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:57 smithi023 bash[29146]: cluster 2024-03-11T21:08:56.286549+0000 mgr.y (mgr.24380) 672 : cluster 0 pgmap v757: 148 pgs: 148 active+clean; 6.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 853 B/s rd, 55 MiB/s wr, 874 op/s 2024-03-11T21:08:57.676 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:57.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:57.683 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:08:57.759 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:57 smithi146 bash[23809]: audit 2024-03-11T21:08:55.751301+0000 mgr.y (mgr.24380) 671 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:08:57.759 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:57 smithi146 bash[23809]: cluster 2024-03-11T21:08:56.286549+0000 mgr.y (mgr.24380) 672 : cluster 0 pgmap v757: 148 pgs: 148 active+clean; 6.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 853 B/s rd, 55 MiB/s wr, 874 op/s 2024-03-11T21:08:57.783 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:08:57.808 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:08:57.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:57.811 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:08:57.912 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:08:57.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:57.942 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:08:57.942 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:08:58.043 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:08:58.071 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:08:58.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:58.073 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:08:58.174 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:08:58.198 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:08:58.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:58.206 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:08:58.306 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:08:58.340 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:08:58.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:58.344 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:08:58.379 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 82 16 101579 101563 77.3963 62 0.016702 0.0129148 2024-03-11T21:08:58.445 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:08:58.468 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:08:58.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:58.475 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:08:58.529 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:08:58 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:08:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:08:58.576 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:58.600 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:58.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:58.606 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:08:58.707 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:58.733 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:08:58.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:58.738 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:08:58.839 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:58.864 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:58.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:58.866 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:08:58.967 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:08:58.993 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:08:58.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:58.995 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:08:59.096 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:59.132 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:59.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:59.136 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:08:59.237 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:08:59.271 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:08:59.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:59.274 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:08:59.375 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:08:59.379 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 83 16 102605 102589 77.2364 64.125 0.0152585 0.0129418 2024-03-11T21:08:59.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:59.403 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:08:59.403 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:08:59.504 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:59.509 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:08:59 smithi146 bash[23809]: cluster 2024-03-11T21:08:58.287119+0000 mgr.y (mgr.24380) 673 : cluster 0 pgmap v758: 148 pgs: 148 active+clean; 6.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 641 B/s rd, 43 MiB/s wr, 692 op/s 2024-03-11T21:08:59.533 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:08:59.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:59.535 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:08:59.636 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:08:59.660 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:08:59.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:59.668 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:08:59.769 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:08:59.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:08:59 smithi023 bash[21372]: cluster 2024-03-11T21:08:58.287119+0000 mgr.y (mgr.24380) 673 : cluster 0 pgmap v758: 148 pgs: 148 active+clean; 6.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 641 B/s rd, 43 MiB/s wr, 692 op/s 2024-03-11T21:08:59.779 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:08:59 smithi023 bash[29146]: cluster 2024-03-11T21:08:58.287119+0000 mgr.y (mgr.24380) 673 : cluster 0 pgmap v758: 148 pgs: 148 active+clean; 6.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 641 B/s rd, 43 MiB/s wr, 692 op/s 2024-03-11T21:08:59.791 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:08:59.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:59.796 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:08:59.896 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:59.925 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:08:59.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:08:59.929 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:09:00.030 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:09:00.065 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:09:00.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:00.071 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:09:00.172 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:09:00.201 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:09:00.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:00.203 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:09:00.304 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:09:00.336 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:09:00.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:00.338 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:09:00.379 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 84 16 103588 103572 77.0482 61.4375 0.0189165 0.0129734 2024-03-11T21:09:00.439 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:00.471 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:00.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:00.473 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:09:00.574 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:09:00.599 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:09:00.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:00.602 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:09:00.703 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:00.730 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:00.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:00.732 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:09:00.833 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:00.858 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:00.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:00.861 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:09:00.914 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:09:00.962 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:09:00.997 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:09:01.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:01.002 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:09:01.103 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:09:01.133 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:09:01.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:01.136 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:09:01.237 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:09:01.273 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:09:01.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:01.277 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:09:01.377 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:01.379 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 85 16 104604 104588 76.8888 63.5 0.0098045 0.0130005 2024-03-11T21:09:01.413 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:01.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:01.415 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:09:01.509 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:01 smithi146 bash[23809]: cluster 2024-03-11T21:09:00.288228+0000 mgr.y (mgr.24380) 674 : cluster 0 pgmap v759: 148 pgs: 148 active+clean; 6.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.1 KiB/s rd, 60 MiB/s wr, 960 op/s 2024-03-11T21:09:01.516 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:09:01.545 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:09:01.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:01.549 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:09:01.650 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:09:01.676 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:09:01.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:01.680 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:09:01.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:01 smithi023 bash[21372]: cluster 2024-03-11T21:09:00.288228+0000 mgr.y (mgr.24380) 674 : cluster 0 pgmap v759: 148 pgs: 148 active+clean; 6.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.1 KiB/s rd, 60 MiB/s wr, 960 op/s 2024-03-11T21:09:01.779 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:01 smithi023 bash[29146]: cluster 2024-03-11T21:09:00.288228+0000 mgr.y (mgr.24380) 674 : cluster 0 pgmap v759: 148 pgs: 148 active+clean; 6.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.1 KiB/s rd, 60 MiB/s wr, 960 op/s 2024-03-11T21:09:01.780 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:09:01.816 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:09:01.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:01.820 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:09:01.921 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:09:01.948 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:09:01.948 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:09:01.954 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:09:01.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:01.954 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:09:01.955 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:09:02.056 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:02.084 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:02.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:02.089 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:09:02.190 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:09:02.214 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:09:02.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:02.217 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:09:02.318 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:09:02.347 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:09:02.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:02.349 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:09:02.379 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 86 16 105611 105595 76.7265 62.9375 0.0080915 0.0130282 2024-03-11T21:09:02.492 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:09:02.524 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:09:02.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:02.530 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:09:02.631 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:09:02.657 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:09:02.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:02.663 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:09:02.763 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:09:02.791 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:09:02.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:02.794 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:09:02.895 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:09:02.920 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:09:02.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:02.927 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:09:03.028 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:09:03.046 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:09:03.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:03.048 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:09:03.149 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:03.178 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:03.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:03.184 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:09:03.285 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:09:03.309 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:09:03.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:03.314 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:09:03.380 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 87 16 106475 106459 76.4653 54 0.0133599 0.0130728 2024-03-11T21:09:03.414 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:09:03.439 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:09:03.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:03.450 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:09:03.551 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:03.582 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:03.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:03.584 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:09:03.685 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:09:03.715 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:09:03.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:03.720 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:09:03.758 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:03 smithi146 bash[23809]: cluster 2024-03-11T21:09:02.288912+0000 mgr.y (mgr.24380) 675 : cluster 0 pgmap v760: 148 pgs: 148 active+clean; 6.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 59 MiB/s wr, 946 op/s 2024-03-11T21:09:03.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:03 smithi023 bash[21372]: cluster 2024-03-11T21:09:02.288912+0000 mgr.y (mgr.24380) 675 : cluster 0 pgmap v760: 148 pgs: 148 active+clean; 6.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 59 MiB/s wr, 946 op/s 2024-03-11T21:09:03.779 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:03 smithi023 bash[29146]: cluster 2024-03-11T21:09:02.288912+0000 mgr.y (mgr.24380) 675 : cluster 0 pgmap v760: 148 pgs: 148 active+clean; 6.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 59 MiB/s wr, 946 op/s 2024-03-11T21:09:03.820 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:09:03.849 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:09:03.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:03.851 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:09:03.952 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:09:03.979 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:09:03.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:03.984 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:09:04.084 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:09:04.112 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:09:04.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:04.119 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:09:04.220 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:09:04.243 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:09:04.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:04.246 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:09:04.347 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:09:04.365 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:09:04.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:04.367 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:09:04.380 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 88 16 107411 107395 76.2611 58.5 0.00325755 0.0130949 2024-03-11T21:09:04.468 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:09:04.496 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:09:04.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:04.500 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:09:04.601 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:04.630 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:04.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:04.635 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:09:04.736 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:09:04.763 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:09:04.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:04.769 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:09:04.870 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:09:04.902 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:09:04.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:04.905 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:09:05.006 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:09:05.036 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:09:05.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:05.039 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:09:05.140 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:09:05.175 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:09:05.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:05.182 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:09:05.283 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:05.318 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:05.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:05.321 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:09:05.380 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 89 15 108329 108314 76.0495 57.4375 0.013675 0.0131442 2024-03-11T21:09:05.421 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:09:05.458 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:09:05.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:05.462 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:09:05.563 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:09:05.600 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:09:05.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:05.605 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:09:05.706 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:09:05.716 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:05 smithi023 bash[21372]: cluster 2024-03-11T21:09:04.289409+0000 mgr.y (mgr.24380) 676 : cluster 0 pgmap v761: 148 pgs: 148 active+clean; 6.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 59 MiB/s wr, 946 op/s 2024-03-11T21:09:05.717 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:05 smithi023 bash[21372]: audit 2024-03-11T21:09:05.110311+0000 mon.a (mon.0) 1674 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:09:05.717 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:05 smithi023 bash[29146]: cluster 2024-03-11T21:09:04.289409+0000 mgr.y (mgr.24380) 676 : cluster 0 pgmap v761: 148 pgs: 148 active+clean; 6.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 59 MiB/s wr, 946 op/s 2024-03-11T21:09:05.717 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:05 smithi023 bash[29146]: audit 2024-03-11T21:09:05.110311+0000 mon.a (mon.0) 1674 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:09:05.751 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:09:05.752 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:05 smithi146 bash[23809]: cluster 2024-03-11T21:09:04.289409+0000 mgr.y (mgr.24380) 676 : cluster 0 pgmap v761: 148 pgs: 148 active+clean; 6.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 59 MiB/s wr, 946 op/s 2024-03-11T21:09:05.752 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:05 smithi146 bash[23809]: audit 2024-03-11T21:09:05.110311+0000 mon.a (mon.0) 1674 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:09:05.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:05.754 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:09:05.855 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:09:05.892 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:09:05.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:05.901 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:09:06.002 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:09:06.008 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:09:05 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:09:06.027 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:09:06.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:06.031 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:09:06.132 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:09:06.157 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:09:06.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:06.161 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:09:06.261 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:09:06.283 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:09:06.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:06.285 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:09:06.380 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 90 16 109009 108993 75.676 42.4375 0.00304645 0.0131806 2024-03-11T21:09:06.385 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:06.405 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:06.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:06.411 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:09:06.511 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:09:06.536 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:09:06.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:06.537 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:09:06.638 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:09:06.660 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:06 smithi023 bash[21372]: audit 2024-03-11T21:09:05.793326+0000 mon.a (mon.0) 1675 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:09:06.661 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:06 smithi023 bash[21372]: audit 2024-03-11T21:09:06.027591+0000 mon.a (mon.0) 1676 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:09:06.661 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:06 smithi023 bash[21372]: audit 2024-03-11T21:09:06.029987+0000 mon.a (mon.0) 1677 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:09:06.661 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:06 smithi023 bash[21372]: audit 2024-03-11T21:09:06.038002+0000 mon.a (mon.0) 1678 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:09:06.661 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:06 smithi023 bash[29146]: audit 2024-03-11T21:09:05.793326+0000 mon.a (mon.0) 1675 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:09:06.661 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:06 smithi023 bash[29146]: audit 2024-03-11T21:09:06.027591+0000 mon.a (mon.0) 1676 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:09:06.661 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:06 smithi023 bash[29146]: audit 2024-03-11T21:09:06.029987+0000 mon.a (mon.0) 1677 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:09:06.661 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:06 smithi023 bash[29146]: audit 2024-03-11T21:09:06.038002+0000 mon.a (mon.0) 1678 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:09:06.667 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:09:06.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:06.669 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:09:06.759 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:06 smithi146 bash[23809]: audit 2024-03-11T21:09:05.793326+0000 mon.a (mon.0) 1675 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:09:06.759 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:06 smithi146 bash[23809]: audit 2024-03-11T21:09:06.027591+0000 mon.a (mon.0) 1676 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:09:06.759 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:06 smithi146 bash[23809]: audit 2024-03-11T21:09:06.029987+0000 mon.a (mon.0) 1677 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:09:06.759 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:06 smithi146 bash[23809]: audit 2024-03-11T21:09:06.038002+0000 mon.a (mon.0) 1678 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:09:06.770 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:09:06.796 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:09:06.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:06.798 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:09:06.899 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:09:06.919 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:09:06.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:06.922 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:09:07.023 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:09:07.051 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:09:07.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:07.053 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:09:07.154 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:07.178 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:07.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:07.180 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:09:07.281 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:09:07.308 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:09:07.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:07.310 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:09:07.380 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:Total time run: 90.0505 2024-03-11T21:09:07.380 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:Total writes made: 109009 2024-03-11T21:09:07.380 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:Write size: 65536 2024-03-11T21:09:07.381 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:Object size: 65536 2024-03-11T21:09:07.381 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:Bandwidth (MB/sec): 75.6582 2024-03-11T21:09:07.381 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:Stddev Bandwidth: 28.2516 2024-03-11T21:09:07.381 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:Max bandwidth (MB/sec): 178.625 2024-03-11T21:09:07.381 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:Min bandwidth (MB/sec): 11.8125 2024-03-11T21:09:07.381 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:Average IOPS: 1210 2024-03-11T21:09:07.381 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:Stddev IOPS: 452.026 2024-03-11T21:09:07.381 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:Max IOPS: 2858 2024-03-11T21:09:07.381 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:Min IOPS: 189 2024-03-11T21:09:07.381 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:Average Latency(s): 0.013213 2024-03-11T21:09:07.381 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:Stddev Latency(s): 0.0214927 2024-03-11T21:09:07.381 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:Max latency(s): 1.06642 2024-03-11T21:09:07.381 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:Min latency(s): 0.00119683 2024-03-11T21:09:07.389 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:Cleaning up (deleting benchmark objects) 2024-03-11T21:09:07.412 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:09:07.446 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:09:07.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:07.449 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:09:07.550 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:09:07.560 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:07 smithi023 bash[21372]: audit 2024-03-11T21:09:05.761192+0000 mgr.y (mgr.24380) 677 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:09:07.561 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:07 smithi023 bash[21372]: cluster 2024-03-11T21:09:06.290756+0000 mgr.y (mgr.24380) 678 : cluster 0 pgmap v762: 148 pgs: 148 active+clean; 6.6 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 932 B/s rd, 60 MiB/s wr, 961 op/s 2024-03-11T21:09:07.561 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:07 smithi023 bash[29146]: audit 2024-03-11T21:09:05.761192+0000 mgr.y (mgr.24380) 677 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:09:07.561 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:07 smithi023 bash[29146]: cluster 2024-03-11T21:09:06.290756+0000 mgr.y (mgr.24380) 678 : cluster 0 pgmap v762: 148 pgs: 148 active+clean; 6.6 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 932 B/s rd, 60 MiB/s wr, 961 op/s 2024-03-11T21:09:07.577 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:09:07.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:07.580 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:09:07.681 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:09:07.716 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:09:07.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:07.718 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:09:07.758 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:07 smithi146 bash[23809]: audit 2024-03-11T21:09:05.761192+0000 mgr.y (mgr.24380) 677 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:09:07.759 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:07 smithi146 bash[23809]: cluster 2024-03-11T21:09:06.290756+0000 mgr.y (mgr.24380) 678 : cluster 0 pgmap v762: 148 pgs: 148 active+clean; 6.6 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 932 B/s rd, 60 MiB/s wr, 961 op/s 2024-03-11T21:09:07.819 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:09:07.838 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:09:07.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:07.840 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:09:07.941 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:09:07.961 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:09:07.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:07.963 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:09:08.064 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:08.089 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:08.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:08.092 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:09:08.193 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:08.227 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:08.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:08.235 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:09:08.336 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:09:08.356 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:09:08.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:08.358 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:09:08.459 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:09:08.485 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:09:08.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:08.487 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:09:08.588 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:09:08.613 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:09:08.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:08.616 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:09:08.717 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:08.751 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:08.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:08.756 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:09:08.778 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:09:08 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:09:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:09:08.857 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:08.875 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:08.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:08.877 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:09:08.977 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:09:09.001 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:09:09.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:09.008 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:09:09.109 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:09:09.137 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:09:09.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:09.139 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:09:09.240 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:09:09.265 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:09:09.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:09.267 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:09:09.368 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:09:09.403 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:09:09.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:09.408 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:09:09.509 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:09.537 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:09.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:09.539 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:09:09.640 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:09:09.651 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:09 smithi146 bash[23809]: cluster 2024-03-11T21:09:08.291257+0000 mgr.y (mgr.24380) 679 : cluster 0 pgmap v763: 148 pgs: 148 active+clean; 6.6 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 852 B/s rd, 52 MiB/s wr, 828 op/s 2024-03-11T21:09:09.666 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:09:09.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:09.671 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:09:09.771 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:09:09.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:09 smithi023 bash[21372]: cluster 2024-03-11T21:09:08.291257+0000 mgr.y (mgr.24380) 679 : cluster 0 pgmap v763: 148 pgs: 148 active+clean; 6.6 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 852 B/s rd, 52 MiB/s wr, 828 op/s 2024-03-11T21:09:09.778 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:09 smithi023 bash[29146]: cluster 2024-03-11T21:09:08.291257+0000 mgr.y (mgr.24380) 679 : cluster 0 pgmap v763: 148 pgs: 148 active+clean; 6.6 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 852 B/s rd, 52 MiB/s wr, 828 op/s 2024-03-11T21:09:09.799 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:09:09.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:09.802 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:09:09.903 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:09:09.930 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:09:09.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:09.935 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:09:10.035 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:09:10.065 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:09:10.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:10.072 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:09:10.172 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:09:10.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:10.205 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:09:10.205 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:09:10.306 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:09:10.342 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:09:10.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:10.345 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:09:10.445 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:09:10.463 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:09:10.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:10.470 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:09:10.571 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:09:10.598 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:09:10.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:10.605 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:09:10.705 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:09:10.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:10.736 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:09:10.736 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:09:10.837 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:09:10.862 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:09:10.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:10.864 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:09:10.965 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:09:11.007 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:09:11.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:11.012 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:09:11.113 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:11.144 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:11.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:11.150 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:09:11.250 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:09:11.274 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:09:11.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:11.279 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:09:11.380 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:09:11.410 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:09:11.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:11.415 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:09:11.516 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:09:11.526 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:11 smithi146 bash[23809]: cluster 2024-03-11T21:09:10.292134+0000 mgr.y (mgr.24380) 680 : cluster 0 pgmap v764: 148 pgs: 148 active+clean; 6.5 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.3 KiB/s rd, 54 MiB/s wr, 1.02k op/s 2024-03-11T21:09:11.544 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:09:11.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:11.548 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:09:11.649 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:09:11.659 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:11 smithi023 bash[21372]: cluster 2024-03-11T21:09:10.292134+0000 mgr.y (mgr.24380) 680 : cluster 0 pgmap v764: 148 pgs: 148 active+clean; 6.5 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.3 KiB/s rd, 54 MiB/s wr, 1.02k op/s 2024-03-11T21:09:11.659 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:11 smithi023 bash[29146]: cluster 2024-03-11T21:09:10.292134+0000 mgr.y (mgr.24380) 680 : cluster 0 pgmap v764: 148 pgs: 148 active+clean; 6.5 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.3 KiB/s rd, 54 MiB/s wr, 1.02k op/s 2024-03-11T21:09:11.678 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:09:11.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:11.682 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:09:11.783 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:09:11.808 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:09:11.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:11.815 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:09:11.915 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:09:11.946 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:09:11.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:11.950 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:09:12.051 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:09:12.090 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:09:12.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:12.092 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:09:12.193 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:09:12.225 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:09:12.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:12.228 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:09:12.329 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:12.350 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:12.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:12.355 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:09:12.456 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:09:12.478 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:09:12.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:12.480 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:09:12.580 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:09:12.609 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:09:12.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:12.613 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:09:12.714 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:09:12.742 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:09:12.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:12.745 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:09:12.846 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:09:12.883 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:09:12.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:12.886 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:09:12.987 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:09:13.029 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:09:13.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:13.033 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:09:13.134 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:09:13.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:13.166 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:09:13.166 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:09:13.267 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:13.300 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:13.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:13.302 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:09:13.403 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:09:13.424 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:09:13.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:13.426 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:09:13.526 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:13.537 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:13 smithi146 bash[23809]: cluster 2024-03-11T21:09:12.294094+0000 mgr.y (mgr.24380) 681 : cluster 0 pgmap v765: 148 pgs: 148 active+clean; 6.5 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 938 B/s rd, 36 MiB/s wr, 781 op/s 2024-03-11T21:09:13.560 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:13.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:13.565 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:09:13.665 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:09:13.683 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:09:13.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:13.685 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:09:13.785 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:13.807 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:13 smithi023 bash[21372]: cluster 2024-03-11T21:09:12.294094+0000 mgr.y (mgr.24380) 681 : cluster 0 pgmap v765: 148 pgs: 148 active+clean; 6.5 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 938 B/s rd, 36 MiB/s wr, 781 op/s 2024-03-11T21:09:13.808 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:13 smithi023 bash[29146]: cluster 2024-03-11T21:09:12.294094+0000 mgr.y (mgr.24380) 681 : cluster 0 pgmap v765: 148 pgs: 148 active+clean; 6.5 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 938 B/s rd, 36 MiB/s wr, 781 op/s 2024-03-11T21:09:13.813 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:13.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:13.816 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:09:13.916 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:09:13.951 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:09:13.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:13.955 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:09:14.057 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:14.092 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:14.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:14.097 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:09:14.197 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:09:14.209 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-11T21:09:14.209 INFO:teuthology.orchestra.run.smithi023.stdout: "size": 20, 2024-03-11T21:09:14.209 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 600, 2024-03-11T21:09:14.209 INFO:teuthology.orchestra.run.smithi023.stdout: "ops": [ 2024-03-11T21:09:14.209 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.209 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_op(client.15081.0:45756 9.c 9:3befdc6d:::benchmark_data_smithi023_106206_object45755:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e262)", 2024-03-11T21:09:14.210 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T21:03:27.584807+0000", 2024-03-11T21:09:14.210 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 346.62257120700002, 2024-03-11T21:09:14.210 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.50962572399999995, 2024-03-11T21:09:14.210 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:09:14.210 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:09:14.210 INFO:teuthology.orchestra.run.smithi023.stdout: "client_info": { 2024-03-11T21:09:14.210 INFO:teuthology.orchestra.run.smithi023.stdout: "client": "client.15081", 2024-03-11T21:09:14.210 INFO:teuthology.orchestra.run.smithi023.stdout: "client_addr": "172.21.15.23:0/2841194346", 2024-03-11T21:09:14.210 INFO:teuthology.orchestra.run.smithi023.stdout: "tid": 45756 2024-03-11T21:09:14.210 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.210 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:09:14.210 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.210 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:09:14.210 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:03:27.584807+0000", 2024-03-11T21:09:14.210 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:09:14.211 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.211 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.211 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:09:14.211 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:03:27.584807+0000", 2024-03-11T21:09:14.211 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:09:14.211 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.211 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.211 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:09:14.211 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:03:27.584808+0000", 2024-03-11T21:09:14.211 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.2950000000000001e-06 2024-03-11T21:09:14.211 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.211 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.211 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:09:14.211 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:03:27.584831+0000", 2024-03-11T21:09:14.212 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.2594000000000001e-05 2024-03-11T21:09:14.212 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.212 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.212 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:09:14.212 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:03:27.584847+0000", 2024-03-11T21:09:14.212 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.6095000000000001e-05 2024-03-11T21:09:14.212 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.212 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.212 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:09:14.212 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:03:27.584856+0000", 2024-03-11T21:09:14.212 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 9.4590000000000004e-06 2024-03-11T21:09:14.212 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.212 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.212 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:09:14.212 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:03:27.584911+0000", 2024-03-11T21:09:14.213 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 5.5065000000000002e-05 2024-03-11T21:09:14.213 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.213 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.213 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:09:14.213 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:03:27.585029+0000", 2024-03-11T21:09:14.213 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00011733 2024-03-11T21:09:14.213 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.213 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.213 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for subops from 4,7", 2024-03-11T21:09:14.213 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:03:27.585130+0000", 2024-03-11T21:09:14.213 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000101259 2024-03-11T21:09:14.213 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.213 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.213 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:09:14.213 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:03:27.587489+0000", 2024-03-11T21:09:14.214 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0023592790000000002 2024-03-11T21:09:14.214 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.214 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.214 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "op_commit", 2024-03-11T21:09:14.214 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:03:27.591794+0000", 2024-03-11T21:09:14.214 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0043047370000000003 2024-03-11T21:09:14.214 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.214 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.214 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:09:14.214 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:03:28.094297+0000", 2024-03-11T21:09:14.214 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.50250288899999995 2024-03-11T21:09:14.214 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.214 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.214 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T21:09:14.215 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:03:28.094380+0000", 2024-03-11T21:09:14.215 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 8.2978000000000004e-05 2024-03-11T21:09:14.215 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.215 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.215 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:09:14.215 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:03:28.094433+0000", 2024-03-11T21:09:14.215 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 5.2744000000000002e-05 2024-03-11T21:09:14.215 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:09:14.215 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:09:14.215 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:09:14.215 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.215 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.215 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_op(client.15081.0:45763 9.c 9:303e561d:::benchmark_data_smithi023_106206_object45762:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e262)", 2024-03-11T21:09:14.215 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T21:03:27.592221+0000", 2024-03-11T21:09:14.216 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 346.61515687000002, 2024-03-11T21:09:14.216 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.50515164499999998, 2024-03-11T21:09:14.216 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:09:14.216 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:09:14.216 INFO:teuthology.orchestra.run.smithi023.stdout: "client_info": { 2024-03-11T21:09:14.216 INFO:teuthology.orchestra.run.smithi023.stdout: "client": "client.15081", 2024-03-11T21:09:14.216 INFO:teuthology.orchestra.run.smithi023.stdout: "client_addr": "172.21.15.23:0/2841194346", 2024-03-11T21:09:14.216 INFO:teuthology.orchestra.run.smithi023.stdout: "tid": 45763 2024-03-11T21:09:14.216 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.216 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:09:14.216 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.216 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:09:14.216 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:03:27.592221+0000", 2024-03-11T21:09:14.216 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:09:14.216 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.217 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.217 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:09:14.217 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:03:27.592221+0000", 2024-03-11T21:09:14.217 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:09:14.217 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.217 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.217 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:09:14.217 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:03:27.592222+0000", 2024-03-11T21:09:14.217 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.4799999999999998e-07 2024-03-11T21:09:14.217 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.217 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.217 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:09:14.217 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:03:27.592257+0000", 2024-03-11T21:09:14.217 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 3.5562999999999998e-05 2024-03-11T21:09:14.217 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.218 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.218 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:09:14.218 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:03:27.592272+0000", 2024-03-11T21:09:14.218 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.414e-05 2024-03-11T21:09:14.218 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.218 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.218 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:09:14.218 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:03:27.592280+0000", 2024-03-11T21:09:14.218 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 8.8079999999999994e-06 2024-03-11T21:09:14.218 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.218 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.218 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:09:14.218 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:03:27.592336+0000", 2024-03-11T21:09:14.218 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 5.5797000000000001e-05 2024-03-11T21:09:14.219 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.219 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.219 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:09:14.219 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:03:27.592465+0000", 2024-03-11T21:09:14.219 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00012849399999999999 2024-03-11T21:09:14.219 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.219 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.219 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for subops from 4,7", 2024-03-11T21:09:14.219 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:03:27.592608+0000", 2024-03-11T21:09:14.219 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000143609 2024-03-11T21:09:14.219 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.219 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.219 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "op_commit", 2024-03-11T21:09:14.219 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:03:27.594758+0000", 2024-03-11T21:09:14.219 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0021495469999999999 2024-03-11T21:09:14.220 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.220 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.220 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:09:14.220 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:03:27.594909+0000", 2024-03-11T21:09:14.220 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000151469 2024-03-11T21:09:14.220 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.220 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.220 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:09:14.220 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:03:28.097228+0000", 2024-03-11T21:09:14.220 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.502318508 2024-03-11T21:09:14.220 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.220 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.220 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T21:09:14.220 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:03:28.097294+0000", 2024-03-11T21:09:14.220 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.6700999999999997e-05 2024-03-11T21:09:14.220 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.221 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.221 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:09:14.221 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:03:28.097373+0000", 2024-03-11T21:09:14.221 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.8361000000000004e-05 2024-03-11T21:09:14.221 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:09:14.221 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:09:14.221 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:09:14.221 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.221 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.221 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_op(client.15081.0:45789 9.c 9:37dfeafd:::benchmark_data_smithi023_106206_object45788:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e262)", 2024-03-11T21:09:14.221 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T21:03:27.616515+0000", 2024-03-11T21:09:14.221 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 346.59086273000003, 2024-03-11T21:09:14.221 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.48967892600000001, 2024-03-11T21:09:14.221 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:09:14.221 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:09:14.222 INFO:teuthology.orchestra.run.smithi023.stdout: "client_info": { 2024-03-11T21:09:14.222 INFO:teuthology.orchestra.run.smithi023.stdout: "client": "client.15081", 2024-03-11T21:09:14.222 INFO:teuthology.orchestra.run.smithi023.stdout: "client_addr": "172.21.15.23:0/2841194346", 2024-03-11T21:09:14.222 INFO:teuthology.orchestra.run.smithi023.stdout: "tid": 45789 2024-03-11T21:09:14.222 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.222 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:09:14.222 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.222 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:09:14.222 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:03:27.616515+0000", 2024-03-11T21:09:14.222 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:09:14.222 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.222 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.222 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:09:14.223 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:03:27.616515+0000", 2024-03-11T21:09:14.223 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:09:14.223 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.223 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.223 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:09:14.223 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:03:27.616516+0000", 2024-03-11T21:09:14.223 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.164e-06 2024-03-11T21:09:14.223 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.223 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.223 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:09:14.223 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:03:27.616540+0000", 2024-03-11T21:09:14.223 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.3407000000000001e-05 2024-03-11T21:09:14.223 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.223 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.223 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:09:14.223 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:03:27.616550+0000", 2024-03-11T21:09:14.224 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 9.6269999999999995e-06 2024-03-11T21:09:14.224 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.224 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.224 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:09:14.224 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:03:27.616555+0000", 2024-03-11T21:09:14.224 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 5.3709999999999999e-06 2024-03-11T21:09:14.224 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.224 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.224 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:09:14.224 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:03:27.616618+0000", 2024-03-11T21:09:14.224 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.2697e-05 2024-03-11T21:09:14.224 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.224 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.224 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:09:14.224 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:03:27.616786+0000", 2024-03-11T21:09:14.225 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000168381 2024-03-11T21:09:14.225 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.225 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.225 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for subops from 4,7", 2024-03-11T21:09:14.225 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:03:27.616960+0000", 2024-03-11T21:09:14.225 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00017410599999999999 2024-03-11T21:09:14.225 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.225 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.225 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "op_commit", 2024-03-11T21:09:14.225 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:03:27.618532+0000", 2024-03-11T21:09:14.225 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0015720420000000001 2024-03-11T21:09:14.225 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.225 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.225 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:09:14.225 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:03:27.619541+0000", 2024-03-11T21:09:14.226 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0010091189999999999 2024-03-11T21:09:14.226 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.226 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.226 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:09:14.226 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:03:28.106029+0000", 2024-03-11T21:09:14.226 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.48648761600000001 2024-03-11T21:09:14.226 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.226 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.226 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T21:09:14.226 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:03:28.106123+0000", 2024-03-11T21:09:14.226 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 9.4512999999999994e-05 2024-03-11T21:09:14.226 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.226 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.226 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:09:14.226 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:03:28.106194+0000", 2024-03-11T21:09:14.227 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.0883000000000006e-05 2024-03-11T21:09:14.227 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:09:14.227 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:09:14.227 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:09:14.227 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.227 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.227 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_op(client.15081.0:45808 9.c 9:37d7c2ac:::benchmark_data_smithi023_106206_object45807:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e262)", 2024-03-11T21:09:14.227 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T21:03:27.636607+0000", 2024-03-11T21:09:14.227 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 346.57077060900002, 2024-03-11T21:09:14.227 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.473137749, 2024-03-11T21:09:14.227 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:09:14.227 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:09:14.227 INFO:teuthology.orchestra.run.smithi023.stdout: "client_info": { 2024-03-11T21:09:14.227 INFO:teuthology.orchestra.run.smithi023.stdout: "client": "client.15081", 2024-03-11T21:09:14.227 INFO:teuthology.orchestra.run.smithi023.stdout: "client_addr": "172.21.15.23:0/2841194346", 2024-03-11T21:09:14.228 INFO:teuthology.orchestra.run.smithi023.stdout: "tid": 45808 2024-03-11T21:09:14.228 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.228 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:09:14.228 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.228 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:09:14.228 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:03:27.636607+0000", 2024-03-11T21:09:14.228 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:09:14.228 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.228 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.228 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:09:14.228 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:03:27.636607+0000", 2024-03-11T21:09:14.228 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:09:14.228 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.228 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.229 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:09:14.229 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:03:27.636608+0000", 2024-03-11T21:09:14.229 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 9.33e-07 2024-03-11T21:09:14.229 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.229 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.229 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:09:14.229 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:03:27.636630+0000", 2024-03-11T21:09:14.229 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.1158999999999999e-05 2024-03-11T21:09:14.229 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.229 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.229 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:09:14.229 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:03:27.636640+0000", 2024-03-11T21:09:14.229 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.0163000000000001e-05 2024-03-11T21:09:14.229 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.229 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.230 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:09:14.230 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:03:27.636647+0000", 2024-03-11T21:09:14.230 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.8059999999999995e-06 2024-03-11T21:09:14.230 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.230 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.230 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:09:14.230 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:03:27.636697+0000", 2024-03-11T21:09:14.230 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 4.973e-05 2024-03-11T21:09:14.230 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.230 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.230 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:09:14.230 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:03:27.636814+0000", 2024-03-11T21:09:14.230 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00011692 2024-03-11T21:09:14.230 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.230 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.231 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for subops from 4,7", 2024-03-11T21:09:14.231 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:03:27.636904+0000", 2024-03-11T21:09:14.231 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 8.9927999999999997e-05 2024-03-11T21:09:14.231 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.231 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.231 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "op_commit", 2024-03-11T21:09:14.231 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:03:27.638431+0000", 2024-03-11T21:09:14.231 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0015269839999999999 2024-03-11T21:09:14.231 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.231 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.231 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:09:14.231 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:03:27.638952+0000", 2024-03-11T21:09:14.231 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00052124699999999996 2024-03-11T21:09:14.231 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.231 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.231 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:09:14.232 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:03:28.109209+0000", 2024-03-11T21:09:14.232 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.47025634199999999 2024-03-11T21:09:14.232 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.232 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.232 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T21:09:14.232 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:03:28.109363+0000", 2024-03-11T21:09:14.232 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000154019 2024-03-11T21:09:14.232 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.232 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.232 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:09:14.232 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:03:28.109745+0000", 2024-03-11T21:09:14.232 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00038251799999999999 2024-03-11T21:09:14.232 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:09:14.232 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:09:14.232 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:09:14.233 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.233 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.233 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_op(client.15081.0:45813 9.c 9:3eeef15a:::benchmark_data_smithi023_106206_object45812:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e262)", 2024-03-11T21:09:14.233 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T21:03:27.647248+0000", 2024-03-11T21:09:14.233 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 346.56012974100003, 2024-03-11T21:09:14.233 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.47378036099999998, 2024-03-11T21:09:14.233 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:09:14.233 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:09:14.233 INFO:teuthology.orchestra.run.smithi023.stdout: "client_info": { 2024-03-11T21:09:14.233 INFO:teuthology.orchestra.run.smithi023.stdout: "client": "client.15081", 2024-03-11T21:09:14.233 INFO:teuthology.orchestra.run.smithi023.stdout: "client_addr": "172.21.15.23:0/2841194346", 2024-03-11T21:09:14.233 INFO:teuthology.orchestra.run.smithi023.stdout: "tid": 45813 2024-03-11T21:09:14.233 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.233 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:09:14.233 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.233 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:09:14.234 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:03:27.647248+0000", 2024-03-11T21:09:14.234 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:09:14.234 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.234 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.234 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:09:14.234 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:03:27.647248+0000", 2024-03-11T21:09:14.234 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:09:14.234 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.234 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.234 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:09:14.234 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:03:27.647250+0000", 2024-03-11T21:09:14.234 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.3629999999999999e-06 2024-03-11T21:09:14.234 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.235 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.235 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:09:14.235 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:03:27.647272+0000", 2024-03-11T21:09:14.235 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.1940999999999999e-05 2024-03-11T21:09:14.235 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.235 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.235 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:09:14.235 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:03:27.647285+0000", 2024-03-11T21:09:14.235 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.3346000000000001e-05 2024-03-11T21:09:14.235 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.235 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.235 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:09:14.235 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:03:27.647293+0000", 2024-03-11T21:09:14.235 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.8120000000000008e-06 2024-03-11T21:09:14.235 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.236 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.236 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:09:14.236 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:03:27.647338+0000", 2024-03-11T21:09:14.236 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 4.5630000000000002e-05 2024-03-11T21:09:14.236 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.236 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.236 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:09:14.236 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:03:27.647441+0000", 2024-03-11T21:09:14.236 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00010220100000000001 2024-03-11T21:09:14.236 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.236 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.236 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for subops from 4,7", 2024-03-11T21:09:14.236 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:03:27.647546+0000", 2024-03-11T21:09:14.236 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00010577 2024-03-11T21:09:14.236 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.237 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.237 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "op_commit", 2024-03-11T21:09:14.237 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:03:27.648877+0000", 2024-03-11T21:09:14.237 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0013308180000000001 2024-03-11T21:09:14.237 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.237 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.237 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:09:14.237 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:03:27.649645+0000", 2024-03-11T21:09:14.237 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00076745800000000005 2024-03-11T21:09:14.237 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.237 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.237 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:09:14.237 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:03:28.120829+0000", 2024-03-11T21:09:14.237 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.47118484300000002 2024-03-11T21:09:14.237 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.238 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.238 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T21:09:14.238 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:03:28.120925+0000", 2024-03-11T21:09:14.238 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 9.5610000000000001e-05 2024-03-11T21:09:14.238 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.238 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.238 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:09:14.238 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:03:28.121029+0000", 2024-03-11T21:09:14.238 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000103569 2024-03-11T21:09:14.238 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:09:14.238 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:09:14.238 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:09:14.238 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.238 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.239 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_op(client.15081.0:143669 9.c 9:33cff7cf:::benchmark_data_smithi023_106206_object32523:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e284)", 2024-03-11T21:09:14.239 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T21:05:07.060388+0000", 2024-03-11T21:09:14.239 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 247.14699018100001, 2024-03-11T21:09:14.239 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.46200499299999997, 2024-03-11T21:09:14.239 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:09:14.239 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:09:14.239 INFO:teuthology.orchestra.run.smithi023.stdout: "client_info": { 2024-03-11T21:09:14.239 INFO:teuthology.orchestra.run.smithi023.stdout: "client": "client.15081", 2024-03-11T21:09:14.239 INFO:teuthology.orchestra.run.smithi023.stdout: "client_addr": "172.21.15.23:0/2841194346", 2024-03-11T21:09:14.239 INFO:teuthology.orchestra.run.smithi023.stdout: "tid": 143669 2024-03-11T21:09:14.239 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.239 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:09:14.239 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.239 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:09:14.239 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:05:07.060388+0000", 2024-03-11T21:09:14.240 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:09:14.240 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.240 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.240 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:09:14.240 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:05:07.060388+0000", 2024-03-11T21:09:14.240 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:09:14.240 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.240 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.240 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:09:14.240 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:05:07.060389+0000", 2024-03-11T21:09:14.240 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.342e-06 2024-03-11T21:09:14.240 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.240 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.240 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:09:14.240 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:05:07.060394+0000", 2024-03-11T21:09:14.240 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 5.2460000000000003e-06 2024-03-11T21:09:14.241 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.241 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.241 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:09:14.241 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:05:07.060407+0000", 2024-03-11T21:09:14.241 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.2084e-05 2024-03-11T21:09:14.241 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.241 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.241 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:09:14.241 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:05:07.060415+0000", 2024-03-11T21:09:14.241 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 8.337e-06 2024-03-11T21:09:14.241 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.241 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.241 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:09:14.241 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:05:07.060470+0000", 2024-03-11T21:09:14.241 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 5.5424999999999999e-05 2024-03-11T21:09:14.242 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.242 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.242 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:09:14.242 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:05:07.060560+0000", 2024-03-11T21:09:14.242 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 8.9806e-05 2024-03-11T21:09:14.242 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.242 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.242 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for subops from 4,7", 2024-03-11T21:09:14.242 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:05:07.060673+0000", 2024-03-11T21:09:14.242 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000112817 2024-03-11T21:09:14.242 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.242 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.242 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "op_commit", 2024-03-11T21:09:14.242 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:05:07.070367+0000", 2024-03-11T21:09:14.243 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0096941089999999994 2024-03-11T21:09:14.243 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.243 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.243 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:09:14.243 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:05:07.070420+0000", 2024-03-11T21:09:14.243 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 5.2757000000000003e-05 2024-03-11T21:09:14.243 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.243 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.243 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:09:14.243 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:05:07.522231+0000", 2024-03-11T21:09:14.243 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.45181091499999998 2024-03-11T21:09:14.243 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.243 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.243 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T21:09:14.243 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:05:07.522316+0000", 2024-03-11T21:09:14.243 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 8.5266000000000007e-05 2024-03-11T21:09:14.244 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.244 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.244 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:09:14.244 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:05:07.522393+0000", 2024-03-11T21:09:14.244 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.6889000000000002e-05 2024-03-11T21:09:14.244 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:09:14.244 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:09:14.244 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:09:14.244 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.244 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.244 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_op(client.15081.0:143671 9.c 9:3224b99c:::benchmark_data_smithi023_106206_object32525:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e284)", 2024-03-11T21:09:14.244 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T21:05:07.061479+0000", 2024-03-11T21:09:14.244 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 247.14589863699999, 2024-03-11T21:09:14.244 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.46640475100000001, 2024-03-11T21:09:14.244 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:09:14.245 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:09:14.245 INFO:teuthology.orchestra.run.smithi023.stdout: "client_info": { 2024-03-11T21:09:14.245 INFO:teuthology.orchestra.run.smithi023.stdout: "client": "client.15081", 2024-03-11T21:09:14.245 INFO:teuthology.orchestra.run.smithi023.stdout: "client_addr": "172.21.15.23:0/2841194346", 2024-03-11T21:09:14.245 INFO:teuthology.orchestra.run.smithi023.stdout: "tid": 143671 2024-03-11T21:09:14.245 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.245 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:09:14.245 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.245 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:09:14.245 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:05:07.061479+0000", 2024-03-11T21:09:14.245 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:09:14.245 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.245 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.245 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:09:14.245 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:05:07.061479+0000", 2024-03-11T21:09:14.246 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:09:14.246 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.246 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.246 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:09:14.246 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:05:07.061480+0000", 2024-03-11T21:09:14.246 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.7700000000000004e-07 2024-03-11T21:09:14.246 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.246 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.246 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:09:14.246 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:05:07.061485+0000", 2024-03-11T21:09:14.246 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 4.5040000000000004e-06 2024-03-11T21:09:14.246 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.246 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.246 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:09:14.246 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:05:07.061495+0000", 2024-03-11T21:09:14.247 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 9.9399999999999997e-06 2024-03-11T21:09:14.247 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.247 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.247 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:09:14.247 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:05:07.061502+0000", 2024-03-11T21:09:14.247 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.1620000000000002e-06 2024-03-11T21:09:14.247 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.247 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.247 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:09:14.247 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:05:07.065251+0000", 2024-03-11T21:09:14.247 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0037487620000000001 2024-03-11T21:09:14.247 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.247 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.247 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:09:14.247 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:05:07.065375+0000", 2024-03-11T21:09:14.248 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00012423800000000001 2024-03-11T21:09:14.248 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.248 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.248 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for subops from 4,7", 2024-03-11T21:09:14.248 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:05:07.065490+0000", 2024-03-11T21:09:14.248 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000115135 2024-03-11T21:09:14.248 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.248 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.248 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "op_commit", 2024-03-11T21:09:14.248 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:05:07.070373+0000", 2024-03-11T21:09:14.248 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00488347 2024-03-11T21:09:14.248 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.248 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.248 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:09:14.248 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:05:07.073608+0000", 2024-03-11T21:09:14.249 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0032345920000000001 2024-03-11T21:09:14.249 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.249 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.249 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:09:14.249 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:05:07.527765+0000", 2024-03-11T21:09:14.249 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.45415671299999999 2024-03-11T21:09:14.249 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.249 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.249 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T21:09:14.249 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:05:07.527831+0000", 2024-03-11T21:09:14.249 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.6167000000000005e-05 2024-03-11T21:09:14.249 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.249 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.249 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:09:14.249 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:05:07.527884+0000", 2024-03-11T21:09:14.249 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 5.3291000000000002e-05 2024-03-11T21:09:14.250 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:09:14.250 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:09:14.250 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:09:14.250 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.250 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.250 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_op(client.15081.0:143677 9.c 9:3b3baede:::benchmark_data_smithi023_106206_object32531:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e284)", 2024-03-11T21:09:14.250 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T21:05:07.066845+0000", 2024-03-11T21:09:14.250 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 247.140532761, 2024-03-11T21:09:14.250 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.46154951999999999, 2024-03-11T21:09:14.250 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:09:14.250 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:09:14.250 INFO:teuthology.orchestra.run.smithi023.stdout: "client_info": { 2024-03-11T21:09:14.250 INFO:teuthology.orchestra.run.smithi023.stdout: "client": "client.15081", 2024-03-11T21:09:14.250 INFO:teuthology.orchestra.run.smithi023.stdout: "client_addr": "172.21.15.23:0/2841194346", 2024-03-11T21:09:14.251 INFO:teuthology.orchestra.run.smithi023.stdout: "tid": 143677 2024-03-11T21:09:14.251 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.251 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:09:14.251 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.251 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:09:14.251 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:05:07.066845+0000", 2024-03-11T21:09:14.251 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:09:14.251 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.251 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.251 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:09:14.251 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:05:07.066845+0000", 2024-03-11T21:09:14.251 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:09:14.251 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.251 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.251 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:09:14.252 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:05:07.066846+0000", 2024-03-11T21:09:14.252 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 4.6400000000000003e-07 2024-03-11T21:09:14.252 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.252 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.252 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:09:14.252 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:05:07.066849+0000", 2024-03-11T21:09:14.252 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.8140000000000001e-06 2024-03-11T21:09:14.252 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.252 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.252 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:09:14.252 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:05:07.066855+0000", 2024-03-11T21:09:14.252 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.7390000000000002e-06 2024-03-11T21:09:14.252 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.252 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.252 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:09:14.253 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:05:07.066861+0000", 2024-03-11T21:09:14.253 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 5.8590000000000001e-06 2024-03-11T21:09:14.253 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.253 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.253 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:09:14.253 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:05:07.070465+0000", 2024-03-11T21:09:14.253 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0036038239999999998 2024-03-11T21:09:14.253 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.253 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.253 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:09:14.253 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:05:07.070788+0000", 2024-03-11T21:09:14.253 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00032268800000000001 2024-03-11T21:09:14.253 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.253 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.253 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for subops from 4,7", 2024-03-11T21:09:14.254 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:05:07.070907+0000", 2024-03-11T21:09:14.254 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000119658 2024-03-11T21:09:14.254 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.254 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.254 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "op_commit", 2024-03-11T21:09:14.254 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:05:07.074217+0000", 2024-03-11T21:09:14.254 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0033095780000000001 2024-03-11T21:09:14.254 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.254 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.254 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:09:14.254 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:05:07.078805+0000", 2024-03-11T21:09:14.254 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.004588295 2024-03-11T21:09:14.254 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.254 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.254 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:09:14.255 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:05:07.528205+0000", 2024-03-11T21:09:14.255 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.44939938899999998 2024-03-11T21:09:14.255 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.255 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.255 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T21:09:14.255 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:05:07.528289+0000", 2024-03-11T21:09:14.255 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 8.4639000000000001e-05 2024-03-11T21:09:14.255 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.255 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.255 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:09:14.255 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:05:07.528395+0000", 2024-03-11T21:09:14.255 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000105573 2024-03-11T21:09:14.255 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:09:14.255 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:09:14.255 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:09:14.256 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.256 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.256 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_repop(client.15291.0:48630 10.7 e320/306 10:edacf55f:::benchmark_data_smithi023_124944_object48629:head v 320'3028, mlcod=320'3026)", 2024-03-11T21:09:14.256 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T21:08:07.361395+0000", 2024-03-11T21:09:14.256 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 66.845982835000001, 2024-03-11T21:09:14.256 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.091003474000000001, 2024-03-11T21:09:14.256 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:09:14.256 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:09:14.256 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:09:14.256 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.256 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:09:14.256 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:07.361395+0000", 2024-03-11T21:09:14.256 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:09:14.256 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.256 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.256 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:09:14.257 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:07.361395+0000", 2024-03-11T21:09:14.257 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:09:14.257 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.257 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.257 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:09:14.257 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:07.361396+0000", 2024-03-11T21:09:14.257 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 9.1399999999999995e-07 2024-03-11T21:09:14.257 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.257 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.257 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:09:14.257 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:07.361462+0000", 2024-03-11T21:09:14.257 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.5718999999999999e-05 2024-03-11T21:09:14.257 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.257 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.257 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:09:14.258 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:07.445315+0000", 2024-03-11T21:09:14.258 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.083853340999999998 2024-03-11T21:09:14.258 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.258 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.258 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:09:14.258 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:07.445327+0000", 2024-03-11T21:09:14.258 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.1335000000000001e-05 2024-03-11T21:09:14.258 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.258 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.258 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:09:14.258 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:07.446258+0000", 2024-03-11T21:09:14.258 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00093180300000000004 2024-03-11T21:09:14.258 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.258 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.258 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:09:14.259 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:07.446278+0000", 2024-03-11T21:09:14.259 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.9680000000000001e-05 2024-03-11T21:09:14.259 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.259 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.259 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T21:09:14.259 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:07.452336+0000", 2024-03-11T21:09:14.259 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0060584719999999996 2024-03-11T21:09:14.259 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.259 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.259 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:09:14.259 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:07.452399+0000", 2024-03-11T21:09:14.259 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.2210000000000002e-05 2024-03-11T21:09:14.259 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:09:14.259 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:09:14.259 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:09:14.259 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.260 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.260 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_repop(client.15291.0:48635 10.5 e320/319 10:a63a665b:::benchmark_data_smithi023_124944_object48634:head v 320'3062, mlcod=320'3060)", 2024-03-11T21:09:14.260 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T21:08:07.365754+0000", 2024-03-11T21:09:14.260 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 66.841623986000002, 2024-03-11T21:09:14.260 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.083663839000000004, 2024-03-11T21:09:14.260 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:09:14.260 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:09:14.260 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:09:14.260 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.260 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:09:14.260 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:07.365754+0000", 2024-03-11T21:09:14.260 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:09:14.260 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.261 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.261 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:09:14.261 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:07.365754+0000", 2024-03-11T21:09:14.261 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:09:14.261 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.261 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.261 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:09:14.261 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:07.365755+0000", 2024-03-11T21:09:14.261 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.234e-06 2024-03-11T21:09:14.261 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.261 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.261 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:09:14.261 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:07.365780+0000", 2024-03-11T21:09:14.261 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.4870000000000001e-05 2024-03-11T21:09:14.261 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.262 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.262 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:09:14.262 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:07.445344+0000", 2024-03-11T21:09:14.262 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.079564139000000006 2024-03-11T21:09:14.262 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.262 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.262 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:09:14.262 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:07.445350+0000", 2024-03-11T21:09:14.262 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.0229999999999998e-06 2024-03-11T21:09:14.262 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.262 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.262 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:09:14.262 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:07.446171+0000", 2024-03-11T21:09:14.262 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00082114799999999995 2024-03-11T21:09:14.262 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.263 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.263 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:09:14.263 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:07.446195+0000", 2024-03-11T21:09:14.263 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.3689e-05 2024-03-11T21:09:14.263 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.263 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.263 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T21:09:14.263 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:07.449374+0000", 2024-03-11T21:09:14.263 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.003178915 2024-03-11T21:09:14.263 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.263 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.263 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:09:14.263 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:07.449418+0000", 2024-03-11T21:09:14.263 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 4.3820999999999999e-05 2024-03-11T21:09:14.264 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:09:14.264 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:09:14.264 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:09:14.264 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.264 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.264 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_repop(client.15291.0:62449 10.7 e324/306 10:ec3fdc32:::benchmark_data_smithi023_124944_object62448:head v 324'3963, mlcod=324'3961)", 2024-03-11T21:09:14.264 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T21:08:18.380953+0000", 2024-03-11T21:09:14.264 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 55.826425458999999, 2024-03-11T21:09:14.264 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.073206926000000005, 2024-03-11T21:09:14.264 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:09:14.264 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:09:14.264 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:09:14.264 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.264 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:09:14.265 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:18.380953+0000", 2024-03-11T21:09:14.265 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:09:14.265 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.265 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.265 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:09:14.265 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:18.380953+0000", 2024-03-11T21:09:14.265 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:09:14.265 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.265 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.265 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:09:14.265 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:18.380953+0000", 2024-03-11T21:09:14.265 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 8.4600000000000003e-07 2024-03-11T21:09:14.265 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.265 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.265 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:09:14.266 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:18.380998+0000", 2024-03-11T21:09:14.266 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 4.4200999999999999e-05 2024-03-11T21:09:14.266 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.266 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.266 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:09:14.266 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:18.381012+0000", 2024-03-11T21:09:14.266 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.4144999999999999e-05 2024-03-11T21:09:14.266 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.266 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.266 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:09:14.266 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:18.381020+0000", 2024-03-11T21:09:14.266 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 8.1960000000000008e-06 2024-03-11T21:09:14.266 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.266 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.267 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:09:14.267 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:18.381099+0000", 2024-03-11T21:09:14.267 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.9406999999999994e-05 2024-03-11T21:09:14.267 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.267 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.267 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:09:14.267 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:18.381152+0000", 2024-03-11T21:09:14.267 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 5.3050000000000002e-05 2024-03-11T21:09:14.267 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.267 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.267 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T21:09:14.267 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:18.454129+0000", 2024-03-11T21:09:14.267 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.072976245999999995 2024-03-11T21:09:14.267 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.267 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.267 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:09:14.268 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:18.454160+0000", 2024-03-11T21:09:14.268 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 3.0834999999999999e-05 2024-03-11T21:09:14.268 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:09:14.268 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:09:14.268 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:09:14.268 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.268 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.268 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_repop(client.15291.0:62459 10.f e324/306 10:f7e607bb:::benchmark_data_smithi023_124944_object62458:head v 324'3977, mlcod=324'3975)", 2024-03-11T21:09:14.268 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T21:08:18.389930+0000", 2024-03-11T21:09:14.268 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 55.817448507000002, 2024-03-11T21:09:14.268 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.064251094999999994, 2024-03-11T21:09:14.268 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:09:14.268 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:09:14.268 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:09:14.268 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.269 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:09:14.269 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:18.389930+0000", 2024-03-11T21:09:14.269 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:09:14.269 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.269 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.269 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:09:14.269 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:18.389930+0000", 2024-03-11T21:09:14.269 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:09:14.269 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.269 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.269 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:09:14.269 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:18.389931+0000", 2024-03-11T21:09:14.270 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 9.7499999999999998e-07 2024-03-11T21:09:14.270 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.270 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.270 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:09:14.270 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:18.389969+0000", 2024-03-11T21:09:14.270 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 3.8482999999999997e-05 2024-03-11T21:09:14.270 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.270 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.270 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:09:14.270 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:18.389982+0000", 2024-03-11T21:09:14.270 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.3095000000000001e-05 2024-03-11T21:09:14.270 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.270 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.270 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:09:14.271 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:18.389990+0000", 2024-03-11T21:09:14.271 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.5889999999999997e-06 2024-03-11T21:09:14.271 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.271 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.271 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:09:14.271 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:18.390068+0000", 2024-03-11T21:09:14.271 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.8107000000000003e-05 2024-03-11T21:09:14.271 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.271 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.271 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:09:14.271 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:18.390088+0000", 2024-03-11T21:09:14.271 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.0710999999999999e-05 2024-03-11T21:09:14.271 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.271 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.271 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T21:09:14.272 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:18.454163+0000", 2024-03-11T21:09:14.272 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.064074189000000004 2024-03-11T21:09:14.272 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.272 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.272 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:09:14.272 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:18.454181+0000", 2024-03-11T21:09:14.272 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.7946000000000001e-05 2024-03-11T21:09:14.272 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:09:14.272 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:09:14.272 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:09:14.272 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.272 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.272 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_repop(client.15291.0:62462 10.f e324/306 10:f9f473e3:::benchmark_data_smithi023_124944_object62461:head v 324'3978, mlcod=324'3975)", 2024-03-11T21:09:14.272 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T21:08:18.391727+0000", 2024-03-11T21:09:14.272 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 55.815650898999998, 2024-03-11T21:09:14.273 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.062471146999999998, 2024-03-11T21:09:14.273 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:09:14.273 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:09:14.273 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:09:14.273 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.273 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:09:14.273 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:18.391727+0000", 2024-03-11T21:09:14.273 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:09:14.273 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.273 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.273 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:09:14.273 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:18.391727+0000", 2024-03-11T21:09:14.273 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:09:14.273 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.273 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.274 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:09:14.274 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:18.391728+0000", 2024-03-11T21:09:14.274 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.99e-07 2024-03-11T21:09:14.274 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.274 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.274 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:09:14.274 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:18.391819+0000", 2024-03-11T21:09:14.274 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 9.1138999999999996e-05 2024-03-11T21:09:14.274 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.274 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.274 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:09:14.274 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:18.391835+0000", 2024-03-11T21:09:14.274 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.6082e-05 2024-03-11T21:09:14.274 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.275 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.275 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:09:14.275 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:18.391842+0000", 2024-03-11T21:09:14.275 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.2969999999999998e-06 2024-03-11T21:09:14.275 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.275 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.275 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:09:14.275 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:18.391970+0000", 2024-03-11T21:09:14.275 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00012799799999999999 2024-03-11T21:09:14.275 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.275 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.275 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:09:14.275 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:18.391989+0000", 2024-03-11T21:09:14.275 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.8374000000000001e-05 2024-03-11T21:09:14.275 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.276 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.276 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T21:09:14.276 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:18.454182+0000", 2024-03-11T21:09:14.276 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.062193495000000001 2024-03-11T21:09:14.276 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.276 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.276 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:09:14.276 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:18.454198+0000", 2024-03-11T21:09:14.276 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.6062999999999999e-05 2024-03-11T21:09:14.276 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:09:14.276 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:09:14.276 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:09:14.276 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.276 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.276 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_repop(client.15291.0:62465 10.f e324/306 10:f60699eb:::benchmark_data_smithi023_124944_object62464:head v 324'3979, mlcod=324'3975)", 2024-03-11T21:09:14.277 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T21:08:18.393299+0000", 2024-03-11T21:09:14.277 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 55.814078971000001, 2024-03-11T21:09:14.277 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.06091568, 2024-03-11T21:09:14.277 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:09:14.277 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:09:14.277 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:09:14.277 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.277 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:09:14.277 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:18.393299+0000", 2024-03-11T21:09:14.277 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:09:14.277 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.277 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.277 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:09:14.277 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:18.393299+0000", 2024-03-11T21:09:14.277 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:09:14.278 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.278 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.278 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:09:14.278 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:18.393300+0000", 2024-03-11T21:09:14.278 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.61e-07 2024-03-11T21:09:14.278 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.278 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.278 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:09:14.278 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:18.393384+0000", 2024-03-11T21:09:14.278 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 8.4525999999999997e-05 2024-03-11T21:09:14.278 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.278 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.278 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:09:14.278 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:18.393399+0000", 2024-03-11T21:09:14.278 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.5041e-05 2024-03-11T21:09:14.279 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.279 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.279 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:09:14.279 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:18.393407+0000", 2024-03-11T21:09:14.279 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.1559999999999998e-06 2024-03-11T21:09:14.279 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.279 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.279 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:09:14.279 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:18.394418+0000", 2024-03-11T21:09:14.279 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0010110659999999999 2024-03-11T21:09:14.279 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.279 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.279 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:09:14.279 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:18.394479+0000", 2024-03-11T21:09:14.280 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.1861999999999998e-05 2024-03-11T21:09:14.280 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.280 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.280 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T21:09:14.280 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:18.454200+0000", 2024-03-11T21:09:14.280 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.059720487000000003 2024-03-11T21:09:14.280 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.280 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.280 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:09:14.280 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:18.454215+0000", 2024-03-11T21:09:14.280 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.4781e-05 2024-03-11T21:09:14.280 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:09:14.280 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:09:14.280 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:09:14.280 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.281 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.281 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_repop(client.15291.0:62468 10.7 e324/306 10:e31c7c50:::benchmark_data_smithi023_124944_object62467:head v 324'3964, mlcod=324'3961)", 2024-03-11T21:09:14.281 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T21:08:18.395682+0000", 2024-03-11T21:09:14.281 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 55.811695800000003, 2024-03-11T21:09:14.281 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.058551302, 2024-03-11T21:09:14.281 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:09:14.281 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:09:14.281 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:09:14.281 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.281 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:09:14.281 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:18.395682+0000", 2024-03-11T21:09:14.281 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:09:14.281 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.281 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.281 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:09:14.282 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:18.395682+0000", 2024-03-11T21:09:14.282 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:09:14.282 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.282 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.282 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:09:14.282 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:18.395683+0000", 2024-03-11T21:09:14.282 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 8.5899999999999995e-07 2024-03-11T21:09:14.282 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.282 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.282 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:09:14.282 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:18.395795+0000", 2024-03-11T21:09:14.282 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000111926 2024-03-11T21:09:14.282 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.282 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.282 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:09:14.283 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:18.395810+0000", 2024-03-11T21:09:14.283 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.5177999999999999e-05 2024-03-11T21:09:14.283 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.283 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.283 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:09:14.283 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:18.395818+0000", 2024-03-11T21:09:14.283 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.7570000000000004e-06 2024-03-11T21:09:14.283 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.283 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.283 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:09:14.283 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:18.396799+0000", 2024-03-11T21:09:14.283 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00098151399999999995 2024-03-11T21:09:14.283 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.283 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.283 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:09:14.284 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:18.396820+0000", 2024-03-11T21:09:14.284 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.0098999999999998e-05 2024-03-11T21:09:14.284 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.284 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.284 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T21:09:14.284 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:18.454216+0000", 2024-03-11T21:09:14.284 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.057396717999999999 2024-03-11T21:09:14.284 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.284 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.284 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:09:14.284 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:18.454234+0000", 2024-03-11T21:09:14.284 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.7251000000000001e-05 2024-03-11T21:09:14.284 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:09:14.284 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:09:14.284 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:09:14.284 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.285 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.285 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_repop(client.15291.0:62475 10.f e324/306 10:f843260d:::benchmark_data_smithi023_124944_object62474:head v 324'3980, mlcod=324'3975)", 2024-03-11T21:09:14.285 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T21:08:18.400172+0000", 2024-03-11T21:09:14.285 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 55.807206280999999, 2024-03-11T21:09:14.285 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.054077811000000003, 2024-03-11T21:09:14.285 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:09:14.285 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:09:14.285 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:09:14.285 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.285 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:09:14.285 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:18.400172+0000", 2024-03-11T21:09:14.285 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:09:14.285 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.285 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.285 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:09:14.286 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:18.400172+0000", 2024-03-11T21:09:14.286 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:09:14.286 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.286 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.286 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:09:14.286 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:18.400173+0000", 2024-03-11T21:09:14.286 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 8.7199999999999997e-07 2024-03-11T21:09:14.286 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.286 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.286 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:09:14.286 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:18.400236+0000", 2024-03-11T21:09:14.286 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.3184999999999999e-05 2024-03-11T21:09:14.286 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.287 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.287 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:09:14.287 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:18.400247+0000", 2024-03-11T21:09:14.287 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.1357e-05 2024-03-11T21:09:14.287 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.287 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.287 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:09:14.287 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:18.400254+0000", 2024-03-11T21:09:14.287 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.9770000000000003e-06 2024-03-11T21:09:14.287 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.287 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.287 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:09:14.287 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:18.403372+0000", 2024-03-11T21:09:14.287 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0031174480000000001 2024-03-11T21:09:14.287 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.288 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.288 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:09:14.288 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:18.403392+0000", 2024-03-11T21:09:14.288 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.0526e-05 2024-03-11T21:09:14.288 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.288 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.288 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T21:09:14.288 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:18.454236+0000", 2024-03-11T21:09:14.288 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.050843653000000003 2024-03-11T21:09:14.288 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.288 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.288 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:09:14.288 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:18.454250+0000", 2024-03-11T21:09:14.288 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.3793e-05 2024-03-11T21:09:14.288 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:09:14.289 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:09:14.289 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:09:14.289 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.289 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.289 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_repop(client.15291.0:108989 10.4 e334/306 10:2bd5283d:::benchmark_data_smithi023_124944_object108988:head v 334'6767, mlcod=334'6763)", 2024-03-11T21:09:14.289 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T21:09:06.184528+0000", 2024-03-11T21:09:14.289 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 8.0228504009999995, 2024-03-11T21:09:14.289 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.215858519, 2024-03-11T21:09:14.289 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:09:14.289 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:09:14.289 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:09:14.289 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.289 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:09:14.289 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:09:06.184528+0000", 2024-03-11T21:09:14.289 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:09:14.289 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.290 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.290 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:09:14.290 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:09:06.184528+0000", 2024-03-11T21:09:14.290 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:09:14.290 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.290 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.290 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:09:14.290 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:09:06.184528+0000", 2024-03-11T21:09:14.290 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 8.0400000000000005e-07 2024-03-11T21:09:14.290 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.290 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.290 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:09:14.290 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:09:06.184551+0000", 2024-03-11T21:09:14.290 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.2259000000000002e-05 2024-03-11T21:09:14.291 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.291 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.291 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:09:14.291 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:09:06.184558+0000", 2024-03-11T21:09:14.291 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.7419999999999997e-06 2024-03-11T21:09:14.291 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.291 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.291 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:09:14.291 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:09:06.184564+0000", 2024-03-11T21:09:14.291 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 5.3650000000000003e-06 2024-03-11T21:09:14.291 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.291 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.291 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:09:14.291 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:09:06.184616+0000", 2024-03-11T21:09:14.291 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 5.1700000000000003e-05 2024-03-11T21:09:14.292 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.292 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.292 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:09:14.292 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:09:06.184634+0000", 2024-03-11T21:09:14.292 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.8442999999999999e-05 2024-03-11T21:09:14.292 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.292 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.292 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T21:09:14.292 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:09:06.400269+0000", 2024-03-11T21:09:14.292 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.21563542999999999 2024-03-11T21:09:14.292 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.292 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.292 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:09:14.292 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:09:06.400386+0000", 2024-03-11T21:09:14.292 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00011677599999999999 2024-03-11T21:09:14.293 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:09:14.293 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:09:14.293 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:09:14.293 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.293 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.293 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_repop(client.15291.0:108988 10.5 e334/319 10:a9bcfff1:::benchmark_data_smithi023_124944_object108987:head v 334'6875, mlcod=334'6872)", 2024-03-11T21:09:14.293 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T21:09:06.185727+0000", 2024-03-11T21:09:14.293 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 8.0216506580000004, 2024-03-11T21:09:14.293 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.22287127200000001, 2024-03-11T21:09:14.293 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:09:14.293 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:09:14.293 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:09:14.293 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.293 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:09:14.293 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:09:06.185727+0000", 2024-03-11T21:09:14.293 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:09:14.294 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.294 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.294 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:09:14.294 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:09:06.185727+0000", 2024-03-11T21:09:14.294 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:09:14.294 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.294 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.294 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:09:14.294 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:09:06.185728+0000", 2024-03-11T21:09:14.294 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.6899999999999997e-07 2024-03-11T21:09:14.294 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.294 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.294 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:09:14.294 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:09:06.185748+0000", 2024-03-11T21:09:14.295 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.0273000000000001e-05 2024-03-11T21:09:14.295 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.295 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.295 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:09:14.295 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:09:06.185758+0000", 2024-03-11T21:09:14.295 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.0176e-05 2024-03-11T21:09:14.295 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.295 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.295 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:09:14.295 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:09:06.185765+0000", 2024-03-11T21:09:14.295 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.3709999999999996e-06 2024-03-11T21:09:14.295 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.295 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.295 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:09:14.295 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:09:06.185816+0000", 2024-03-11T21:09:14.295 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 5.1201e-05 2024-03-11T21:09:14.296 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.296 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.296 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:09:14.296 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:09:06.185835+0000", 2024-03-11T21:09:14.296 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.8822999999999999e-05 2024-03-11T21:09:14.296 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.296 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.296 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T21:09:14.296 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:09:06.408557+0000", 2024-03-11T21:09:14.296 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.22272240900000001 2024-03-11T21:09:14.296 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.296 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.296 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:09:14.296 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:09:06.408599+0000", 2024-03-11T21:09:14.296 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 4.1350000000000002e-05 2024-03-11T21:09:14.297 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:09:14.297 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:09:14.297 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:09:14.297 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.297 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.297 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_repop(client.15291.0:108994 10.5 e334/319 10:a88f9fa4:::benchmark_data_smithi023_124944_object108993:head v 334'6876, mlcod=334'6872)", 2024-03-11T21:09:14.297 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T21:09:06.188474+0000", 2024-03-11T21:09:14.297 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 8.0189038480000008, 2024-03-11T21:09:14.297 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.23066687599999999, 2024-03-11T21:09:14.297 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:09:14.297 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:09:14.297 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:09:14.297 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.297 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:09:14.297 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:09:06.188474+0000", 2024-03-11T21:09:14.298 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:09:14.298 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.298 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.298 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:09:14.298 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:09:06.188474+0000", 2024-03-11T21:09:14.298 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:09:14.298 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.298 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.298 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:09:14.298 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:09:06.188475+0000", 2024-03-11T21:09:14.298 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 8.1399999999999996e-07 2024-03-11T21:09:14.298 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.298 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.298 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:09:14.298 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:09:06.188494+0000", 2024-03-11T21:09:14.299 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.9207000000000001e-05 2024-03-11T21:09:14.299 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.299 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.299 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:09:14.299 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:09:06.400087+0000", 2024-03-11T21:09:14.299 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.211592947 2024-03-11T21:09:14.299 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.299 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.299 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:09:14.299 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:09:06.400119+0000", 2024-03-11T21:09:14.299 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 3.1890000000000001e-05 2024-03-11T21:09:14.299 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.299 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.299 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:09:14.299 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:09:06.405843+0000", 2024-03-11T21:09:14.300 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0057238419999999998 2024-03-11T21:09:14.300 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.300 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.300 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:09:14.300 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:09:06.405877+0000", 2024-03-11T21:09:14.300 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 3.4604999999999998e-05 2024-03-11T21:09:14.300 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.300 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.300 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T21:09:14.300 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:09:06.419012+0000", 2024-03-11T21:09:14.300 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.013134115 2024-03-11T21:09:14.301 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.301 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.301 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:09:14.301 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:09:06.419141+0000", 2024-03-11T21:09:14.301 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00012945599999999999 2024-03-11T21:09:14.301 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:09:14.301 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:09:14.301 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:09:14.301 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.301 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.301 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_repop(client.15291.0:109002 10.f e334/306 10:f23a0525:::benchmark_data_smithi023_124944_object109001:head v 334'6926, mlcod=334'6924)", 2024-03-11T21:09:14.301 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T21:09:06.192991+0000", 2024-03-11T21:09:14.301 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 8.0143865719999994, 2024-03-11T21:09:14.301 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.219322829, 2024-03-11T21:09:14.301 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:09:14.302 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:09:14.302 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:09:14.302 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.302 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:09:14.302 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:09:06.192991+0000", 2024-03-11T21:09:14.302 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:09:14.302 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.302 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.302 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:09:14.302 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:09:06.192991+0000", 2024-03-11T21:09:14.302 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:09:14.302 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.302 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.302 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:09:14.303 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:09:06.192993+0000", 2024-03-11T21:09:14.303 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.102e-06 2024-03-11T21:09:14.303 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.303 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.303 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:09:14.303 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:09:06.193061+0000", 2024-03-11T21:09:14.303 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.8356999999999996e-05 2024-03-11T21:09:14.303 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.303 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.303 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:09:14.303 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:09:06.400253+0000", 2024-03-11T21:09:14.303 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.20719238000000001 2024-03-11T21:09:14.303 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.303 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.303 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:09:14.304 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:09:06.400278+0000", 2024-03-11T21:09:14.304 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.4422000000000001e-05 2024-03-11T21:09:14.304 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.304 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.304 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:09:14.304 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:09:06.400482+0000", 2024-03-11T21:09:14.304 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000204728 2024-03-11T21:09:14.304 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.304 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.304 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:09:14.304 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:09:06.400538+0000", 2024-03-11T21:09:14.304 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 5.5705999999999998e-05 2024-03-11T21:09:14.304 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.304 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.304 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T21:09:14.305 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:09:06.412138+0000", 2024-03-11T21:09:14.305 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.011600157 2024-03-11T21:09:14.305 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:09:14.305 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:09:14.305 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:09:14.305 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:09:06.412314+0000", 2024-03-11T21:09:14.305 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00017597700000000001 2024-03-11T21:09:14.305 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:09:14.305 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:09:14.305 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:09:14.305 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:09:14.305 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:09:14.305 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-11T21:09:14.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:14.306 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:09:14.307 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:09:14.407 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:09:14.428 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:09:14.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:14.432 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:09:14.533 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:09:14.560 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:09:14.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:14.565 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:09:14.666 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:09:14.675 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:14 smithi146 bash[23809]: cluster 2024-03-11T21:09:12.812484+0000 osd.7 (osd.7) 413 : cluster 0 purged_snaps scrub starts 2024-03-11T21:09:14.675 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:14 smithi146 bash[23809]: cluster 2024-03-11T21:09:12.812542+0000 osd.7 (osd.7) 414 : cluster 0 purged_snaps scrub ok 2024-03-11T21:09:14.675 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:14 smithi146 bash[23809]: cluster 2024-03-11T21:09:13.506817+0000 mon.a (mon.0) 1679 : cluster 0 osdmap e335: 8 total, 8 up, 8 in 2024-03-11T21:09:14.703 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:09:14.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:14.706 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:09:14.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:14 smithi023 bash[21372]: cluster 2024-03-11T21:09:12.812484+0000 osd.7 (osd.7) 413 : cluster 0 purged_snaps scrub starts 2024-03-11T21:09:14.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:14 smithi023 bash[21372]: cluster 2024-03-11T21:09:12.812542+0000 osd.7 (osd.7) 414 : cluster 0 purged_snaps scrub ok 2024-03-11T21:09:14.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:14 smithi023 bash[21372]: cluster 2024-03-11T21:09:13.506817+0000 mon.a (mon.0) 1679 : cluster 0 osdmap e335: 8 total, 8 up, 8 in 2024-03-11T21:09:14.779 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:14 smithi023 bash[29146]: cluster 2024-03-11T21:09:12.812484+0000 osd.7 (osd.7) 413 : cluster 0 purged_snaps scrub starts 2024-03-11T21:09:14.779 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:14 smithi023 bash[29146]: cluster 2024-03-11T21:09:12.812542+0000 osd.7 (osd.7) 414 : cluster 0 purged_snaps scrub ok 2024-03-11T21:09:14.779 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:14 smithi023 bash[29146]: cluster 2024-03-11T21:09:13.506817+0000 mon.a (mon.0) 1679 : cluster 0 osdmap e335: 8 total, 8 up, 8 in 2024-03-11T21:09:14.806 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:09:14.840 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:09:14.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:14.845 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:09:14.946 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:09:14.965 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:09:14.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:14.967 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:09:15.068 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:09:15.098 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:09:15.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:15.101 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:09:15.201 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:09:15.230 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:09:15.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:15.233 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:09:15.334 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:09:15.359 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:09:15.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:15.361 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:09:15.462 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:15.493 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:15.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:15.496 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:09:15.597 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:09:15.638 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:09:15.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:15.639 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:09:15.740 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:09:15.749 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:15 smithi146 bash[23809]: cluster 2024-03-11T21:09:14.294571+0000 mgr.y (mgr.24380) 682 : cluster 0 pgmap v767: 148 pgs: 148 active+clean; 6.5 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.1 KiB/s rd, 35 MiB/s wr, 814 op/s 2024-03-11T21:09:15.749 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:15 smithi146 bash[23809]: cluster 2024-03-11T21:09:15.307241+0000 mon.a (mon.0) 1680 : cluster 0 osdmap e336: 8 total, 8 up, 8 in 2024-03-11T21:09:15.761 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:09:15.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:15.763 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:09:15.790 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:15 smithi023 bash[21372]: cluster 2024-03-11T21:09:14.294571+0000 mgr.y (mgr.24380) 682 : cluster 0 pgmap v767: 148 pgs: 148 active+clean; 6.5 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.1 KiB/s rd, 35 MiB/s wr, 814 op/s 2024-03-11T21:09:15.790 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:15 smithi023 bash[21372]: cluster 2024-03-11T21:09:15.307241+0000 mon.a (mon.0) 1680 : cluster 0 osdmap e336: 8 total, 8 up, 8 in 2024-03-11T21:09:15.791 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:15 smithi023 bash[29146]: cluster 2024-03-11T21:09:14.294571+0000 mgr.y (mgr.24380) 682 : cluster 0 pgmap v767: 148 pgs: 148 active+clean; 6.5 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.1 KiB/s rd, 35 MiB/s wr, 814 op/s 2024-03-11T21:09:15.791 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:15 smithi023 bash[29146]: cluster 2024-03-11T21:09:15.307241+0000 mon.a (mon.0) 1680 : cluster 0 osdmap e336: 8 total, 8 up, 8 in 2024-03-11T21:09:15.863 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:15.872 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:09:15 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:09:15.889 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:15.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:15.891 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:09:15.992 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:09:16.030 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:09:16.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:16.036 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:09:16.137 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:09:16.166 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:09:16.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:16.172 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:09:16.272 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:09:16.301 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:09:16.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:16.305 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:09:16.406 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:09:16.425 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:09:16.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:16.427 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:09:16.527 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:09:16.550 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:09:16.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:16.552 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:09:16.553 INFO:teuthology.orchestra.run.smithi023.stderr:noscrub is unset 2024-03-11T21:09:16.653 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:09:16.692 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:09:16.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:16.694 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:09:16.795 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:09:16.806 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:16 smithi023 bash[21372]: audit 2024-03-11T21:09:16.416578+0000 mon.a (mon.0) 1681 : audit 1 from='client.? 172.21.15.23:0/76632662' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:09:16.807 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:16 smithi023 bash[29146]: audit 2024-03-11T21:09:16.416578+0000 mon.a (mon.0) 1681 : audit 1 from='client.? 172.21.15.23:0/76632662' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:09:16.823 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:09:16.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:16.827 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:09:16.928 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:16.937 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:16 smithi146 bash[23809]: audit 2024-03-11T21:09:16.416578+0000 mon.a (mon.0) 1681 : audit 1 from='client.? 172.21.15.23:0/76632662' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:09:16.959 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:16.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:16.964 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:09:17.066 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:09:17.097 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:09:17.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:17.099 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:09:17.200 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:09:17.230 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:09:17.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:17.233 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:09:17.299 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-11T21:09:17.334 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:09:17.365 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:09:17.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:17.367 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:09:17.468 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:09:17.488 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:09:17.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:17.490 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:09:17.591 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:17.601 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:17 smithi146 bash[23809]: audit 2024-03-11T21:09:15.769256+0000 mgr.y (mgr.24380) 683 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:09:17.601 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:17 smithi146 bash[23809]: cluster 2024-03-11T21:09:16.295080+0000 mgr.y (mgr.24380) 684 : cluster 0 pgmap v769: 148 pgs: 148 active+clean; 6.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.4 KiB/s rd, 7.2 MiB/s wr, 989 op/s 2024-03-11T21:09:17.601 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:17 smithi146 bash[23809]: cluster 2024-03-11T21:09:16.538842+0000 mon.a (mon.0) 1682 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:09:17.601 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:17 smithi146 bash[23809]: audit 2024-03-11T21:09:16.559222+0000 mon.a (mon.0) 1683 : audit 1 from='client.? 172.21.15.23:0/76632662' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:09:17.601 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:17 smithi146 bash[23809]: cluster 2024-03-11T21:09:16.572284+0000 mon.a (mon.0) 1684 : cluster 0 osdmap e337: 8 total, 8 up, 8 in 2024-03-11T21:09:17.601 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:17 smithi146 bash[23809]: cluster 2024-03-11T21:09:16.815101+0000 osd.5 (osd.5) 363 : cluster 0 10.9 scrub starts 2024-03-11T21:09:17.634 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:17.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:17.641 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:09:17.711 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:17 smithi023 bash[21372]: audit 2024-03-11T21:09:15.769256+0000 mgr.y (mgr.24380) 683 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:09:17.711 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:17 smithi023 bash[21372]: cluster 2024-03-11T21:09:16.295080+0000 mgr.y (mgr.24380) 684 : cluster 0 pgmap v769: 148 pgs: 148 active+clean; 6.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.4 KiB/s rd, 7.2 MiB/s wr, 989 op/s 2024-03-11T21:09:17.711 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:17 smithi023 bash[21372]: cluster 2024-03-11T21:09:16.538842+0000 mon.a (mon.0) 1682 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:09:17.711 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:17 smithi023 bash[21372]: audit 2024-03-11T21:09:16.559222+0000 mon.a (mon.0) 1683 : audit 1 from='client.? 172.21.15.23:0/76632662' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:09:17.711 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:17 smithi023 bash[21372]: cluster 2024-03-11T21:09:16.572284+0000 mon.a (mon.0) 1684 : cluster 0 osdmap e337: 8 total, 8 up, 8 in 2024-03-11T21:09:17.711 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:17 smithi023 bash[21372]: cluster 2024-03-11T21:09:16.815101+0000 osd.5 (osd.5) 363 : cluster 0 10.9 scrub starts 2024-03-11T21:09:17.741 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:09:17.767 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:09:17.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:17.769 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:09:17.870 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:09:17.890 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:09:17.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:17.894 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:09:17.995 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:09:18.031 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:09:18.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:18.034 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:09:18.038 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:17 smithi023 bash[29146]: audit 2024-03-11T21:09:15.769256+0000 mgr.y (mgr.24380) 683 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:09:18.038 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:17 smithi023 bash[29146]: cluster 2024-03-11T21:09:16.295080+0000 mgr.y (mgr.24380) 684 : cluster 0 pgmap v769: 148 pgs: 148 active+clean; 6.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.4 KiB/s rd, 7.2 MiB/s wr, 989 op/s 2024-03-11T21:09:18.038 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:17 smithi023 bash[29146]: cluster 2024-03-11T21:09:16.538842+0000 mon.a (mon.0) 1682 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:09:18.038 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:17 smithi023 bash[29146]: audit 2024-03-11T21:09:16.559222+0000 mon.a (mon.0) 1683 : audit 1 from='client.? 172.21.15.23:0/76632662' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:09:18.039 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:17 smithi023 bash[29146]: cluster 2024-03-11T21:09:16.572284+0000 mon.a (mon.0) 1684 : cluster 0 osdmap e337: 8 total, 8 up, 8 in 2024-03-11T21:09:18.039 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:17 smithi023 bash[29146]: cluster 2024-03-11T21:09:16.815101+0000 osd.5 (osd.5) 363 : cluster 0 10.9 scrub starts 2024-03-11T21:09:18.135 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:09:18.161 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:09:18.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:18.166 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:09:18.267 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:09:18.295 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:09:18.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:18.299 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:09:18.400 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:18.442 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:18.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:18.448 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:09:18.450 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:09:18 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:09:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:09:18.548 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:09:18.567 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:09:18.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:18.569 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:09:18.670 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:09:18.694 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:09:18.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:18.699 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:09:18.800 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:09:18.803 INFO:teuthology.orchestra.run.smithi023.stdout:osd.0: {} 2024-03-11T21:09:18.811 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:18 smithi146 bash[23809]: cluster 2024-03-11T21:09:17.234944+0000 osd.7 (osd.7) 415 : cluster 0 10.c scrub starts 2024-03-11T21:09:18.840 INFO:teuthology.orchestra.run.smithi023.stdout:osd.1: {} 2024-03-11T21:09:18.853 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:09:18.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:18.857 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:09:18.875 INFO:teuthology.orchestra.run.smithi023.stdout:osd.2: {} 2024-03-11T21:09:18.913 INFO:teuthology.orchestra.run.smithi023.stdout:osd.3: {} 2024-03-11T21:09:18.952 INFO:teuthology.orchestra.run.smithi023.stdout:osd.4: {} 2024-03-11T21:09:18.958 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:09:18.967 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:18 smithi023 bash[21372]: cluster 2024-03-11T21:09:17.234944+0000 osd.7 (osd.7) 415 : cluster 0 10.c scrub starts 2024-03-11T21:09:18.967 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:18 smithi023 bash[29146]: cluster 2024-03-11T21:09:17.234944+0000 osd.7 (osd.7) 415 : cluster 0 10.c scrub starts 2024-03-11T21:09:18.989 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:09:18.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:18.993 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:09:19.009 INFO:teuthology.orchestra.run.smithi023.stdout:osd.5: {} 2024-03-11T21:09:19.060 INFO:teuthology.orchestra.run.smithi023.stdout:osd.6: {} 2024-03-11T21:09:19.094 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:09:19.113 INFO:teuthology.orchestra.run.smithi023.stdout:osd.7: {} 2024-03-11T21:09:19.117 INFO:teuthology.orchestra.run.smithi023.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:09:19.117 INFO:teuthology.orchestra.run.smithi023.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:09:19.117 INFO:teuthology.orchestra.run.smithi023.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:09:19.118 INFO:teuthology.orchestra.run.smithi023.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:09:19.118 INFO:teuthology.orchestra.run.smithi023.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:09:19.118 INFO:teuthology.orchestra.run.smithi023.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:09:19.118 INFO:teuthology.orchestra.run.smithi023.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:09:19.118 INFO:teuthology.orchestra.run.smithi023.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:09:19.129 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:09:19.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:19.132 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:09:19.233 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:09:19.263 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:09:19.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:19.266 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:09:19.367 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:09:19.405 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:09:19.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:19.409 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:09:19.509 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:09:19.536 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:09:19.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:19.540 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:09:19.608 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-11T21:09:19.616 INFO:teuthology.orchestra.run.smithi023.stderr:dumped all 2024-03-11T21:09:19.641 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:09:19.652 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:19 smithi146 bash[23809]: cluster 2024-03-11T21:09:18.221522+0000 osd.1 (osd.1) 169 : cluster 0 4.9 scrub starts 2024-03-11T21:09:19.652 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:19 smithi146 bash[23809]: cluster 2024-03-11T21:09:18.222633+0000 osd.1 (osd.1) 170 : cluster 0 4.9 scrub ok 2024-03-11T21:09:19.652 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:19 smithi146 bash[23809]: cluster 2024-03-11T21:09:18.295927+0000 mgr.y (mgr.24380) 685 : cluster 0 pgmap v771: 148 pgs: 148 active+clean; 6.2 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 782 op/s 2024-03-11T21:09:19.652 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:19 smithi146 bash[23809]: audit 2024-03-11T21:09:18.638532+0000 mon.c (mon.2) 84 : audit 0 from='client.? 172.21.15.23:0/2022787017' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:09:19.652 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:19 smithi146 bash[23809]: cluster 2024-03-11T21:09:18.840875+0000 osd.5 (osd.5) 364 : cluster 0 10.3 scrub starts 2024-03-11T21:09:19.673 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:09:19.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:19.676 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:09:19.777 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:09:19.814 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:09:19.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:19.818 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:09:19.919 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:19.958 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:19.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:19.965 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:09:20.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:19 smithi023 bash[21372]: cluster 2024-03-11T21:09:18.221522+0000 osd.1 (osd.1) 169 : cluster 0 4.9 scrub starts 2024-03-11T21:09:20.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:19 smithi023 bash[21372]: cluster 2024-03-11T21:09:18.222633+0000 osd.1 (osd.1) 170 : cluster 0 4.9 scrub ok 2024-03-11T21:09:20.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:19 smithi023 bash[21372]: cluster 2024-03-11T21:09:18.295927+0000 mgr.y (mgr.24380) 685 : cluster 0 pgmap v771: 148 pgs: 148 active+clean; 6.2 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 782 op/s 2024-03-11T21:09:20.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:19 smithi023 bash[21372]: audit 2024-03-11T21:09:18.638532+0000 mon.c (mon.2) 84 : audit 0 from='client.? 172.21.15.23:0/2022787017' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:09:20.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:19 smithi023 bash[21372]: cluster 2024-03-11T21:09:18.840875+0000 osd.5 (osd.5) 364 : cluster 0 10.3 scrub starts 2024-03-11T21:09:20.029 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:19 smithi023 bash[29146]: cluster 2024-03-11T21:09:18.221522+0000 osd.1 (osd.1) 169 : cluster 0 4.9 scrub starts 2024-03-11T21:09:20.029 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:19 smithi023 bash[29146]: cluster 2024-03-11T21:09:18.222633+0000 osd.1 (osd.1) 170 : cluster 0 4.9 scrub ok 2024-03-11T21:09:20.029 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:19 smithi023 bash[29146]: cluster 2024-03-11T21:09:18.295927+0000 mgr.y (mgr.24380) 685 : cluster 0 pgmap v771: 148 pgs: 148 active+clean; 6.2 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 782 op/s 2024-03-11T21:09:20.029 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:19 smithi023 bash[29146]: audit 2024-03-11T21:09:18.638532+0000 mon.c (mon.2) 84 : audit 0 from='client.? 172.21.15.23:0/2022787017' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:09:20.029 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:19 smithi023 bash[29146]: cluster 2024-03-11T21:09:18.840875+0000 osd.5 (osd.5) 364 : cluster 0 10.3 scrub starts 2024-03-11T21:09:20.065 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:09:20.088 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:09:20.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:20.091 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:09:20.192 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:09:20.221 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:09:20.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:20.228 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:09:20.328 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:09:20.345 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:09:20.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:20.347 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:09:20.448 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:09:20.481 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:09:20.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:20.485 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:09:20.586 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:09:20.623 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:09:20.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:20.627 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:09:20.728 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:09:20.737 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:20 smithi146 bash[23809]: cluster 2024-03-11T21:09:19.224928+0000 osd.5 (osd.5) 365 : cluster 0 10.9 scrub ok 2024-03-11T21:09:20.737 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:20 smithi146 bash[23809]: audit 2024-03-11T21:09:19.590043+0000 mgr.y (mgr.24380) 686 : audit 0 from='client.15345 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:09:20.737 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:20 smithi146 bash[23809]: cluster 2024-03-11T21:09:19.600736+0000 osd.6 (osd.6) 286 : cluster 0 10.7 scrub starts 2024-03-11T21:09:20.737 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:20 smithi146 bash[23809]: cluster 2024-03-11T21:09:19.774476+0000 osd.3 (osd.3) 380 : cluster 0 3.8 scrub starts 2024-03-11T21:09:20.737 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:20 smithi146 bash[23809]: cluster 2024-03-11T21:09:19.777310+0000 osd.3 (osd.3) 381 : cluster 0 3.8 scrub ok 2024-03-11T21:09:20.737 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:20 smithi146 bash[23809]: cluster 2024-03-11T21:09:20.310346+0000 mon.a (mon.0) 1685 : cluster 0 osdmap e338: 8 total, 8 up, 8 in 2024-03-11T21:09:20.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:20.753 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:09:20.753 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:09:20.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:20 smithi023 bash[21372]: cluster 2024-03-11T21:09:19.224928+0000 osd.5 (osd.5) 365 : cluster 0 10.9 scrub ok 2024-03-11T21:09:20.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:20 smithi023 bash[21372]: audit 2024-03-11T21:09:19.590043+0000 mgr.y (mgr.24380) 686 : audit 0 from='client.15345 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:09:20.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:20 smithi023 bash[21372]: cluster 2024-03-11T21:09:19.600736+0000 osd.6 (osd.6) 286 : cluster 0 10.7 scrub starts 2024-03-11T21:09:20.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:20 smithi023 bash[21372]: cluster 2024-03-11T21:09:19.774476+0000 osd.3 (osd.3) 380 : cluster 0 3.8 scrub starts 2024-03-11T21:09:20.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:20 smithi023 bash[21372]: cluster 2024-03-11T21:09:19.777310+0000 osd.3 (osd.3) 381 : cluster 0 3.8 scrub ok 2024-03-11T21:09:20.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:20 smithi023 bash[21372]: cluster 2024-03-11T21:09:20.310346+0000 mon.a (mon.0) 1685 : cluster 0 osdmap e338: 8 total, 8 up, 8 in 2024-03-11T21:09:20.779 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:20 smithi023 bash[29146]: cluster 2024-03-11T21:09:19.224928+0000 osd.5 (osd.5) 365 : cluster 0 10.9 scrub ok 2024-03-11T21:09:20.779 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:20 smithi023 bash[29146]: audit 2024-03-11T21:09:19.590043+0000 mgr.y (mgr.24380) 686 : audit 0 from='client.15345 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:09:20.779 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:20 smithi023 bash[29146]: cluster 2024-03-11T21:09:19.600736+0000 osd.6 (osd.6) 286 : cluster 0 10.7 scrub starts 2024-03-11T21:09:20.779 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:20 smithi023 bash[29146]: cluster 2024-03-11T21:09:19.774476+0000 osd.3 (osd.3) 380 : cluster 0 3.8 scrub starts 2024-03-11T21:09:20.779 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:20 smithi023 bash[29146]: cluster 2024-03-11T21:09:19.777310+0000 osd.3 (osd.3) 381 : cluster 0 3.8 scrub ok 2024-03-11T21:09:20.779 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:20 smithi023 bash[29146]: cluster 2024-03-11T21:09:20.310346+0000 mon.a (mon.0) 1685 : cluster 0 osdmap e338: 8 total, 8 up, 8 in 2024-03-11T21:09:20.854 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:09:20.882 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:09:20.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:20.884 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:09:20.985 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:09:21.013 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:09:21.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:21.018 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:09:21.119 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:09:21.124 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-11T21:09:21.156 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:09:21.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:21.161 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:09:21.184 INFO:teuthology.orchestra.run.smithi023.stdout:{"pg_ready":true,"pg_map":{"version":771,"stamp":"2024-03-11T21:09:18.295275+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6669998528,"num_objects":101966,"num_object_clones":0,"num_object_copies":305898,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101966,"num_whiteouts":0,"num_read":2346,"num_read_kb":2192,"num_write":117153,"num_write_kb":6979038,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":190,"num_bytes_recovered":1775932,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91207,"ondisk_log_size":91207,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":22749924,"kb_used_data":19607488,"kb_used_omap":21,"kb_used_meta":3141930,"kb_avail":727211292,"statfs":{"total":767960285184,"available":744664363008,"internally_reserved":0,"allocated":20078067712,"data_stored":20074390650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22089,"internal_metadata":3217336759},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,2,6,2,7],"upper_bound":32},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-307625984,"num_objects":-4694,"num_object_clones":0,"num_object_copies":-14082,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4694,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":4694,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001682"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":577,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-11T21:09:13.516537+0000","last_change":"2024-03-11T21:07:57.433433+0000","last_active":"2024-03-11T21:09:13.516537+0000","last_peered":"2024-03-11T21:09:13.516537+0000","last_clean":"2024-03-11T21:09:13.516537+0000","last_became_active":"2024-03-11T20:50:34.338390+0000","last_became_peered":"2024-03-11T20:50:34.338390+0000","last_unstale":"2024-03-11T21:09:13.516537+0000","last_undegraded":"2024-03-11T21:09:13.516537+0000","last_fullsized":"2024-03-11T21:09:13.516537+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:07:57.433167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:07:57.433167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":581,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-11T21:09:13.516374+0000","last_change":"2024-03-11T21:08:09.344727+0000","last_active":"2024-03-11T21:09:13.516374+0000","last_peered":"2024-03-11T21:09:13.516374+0000","last_clean":"2024-03-11T21:09:13.516374+0000","last_became_active":"2024-03-11T20:50:32.210194+0000","last_became_peered":"2024-03-11T20:50:32.210194+0000","last_unstale":"2024-03-11T21:09:13.516374+0000","last_undegraded":"2024-03-11T21:09:13.516374+0000","last_fullsized":"2024-03-11T21:09:13.516374+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:08:09.344640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:08:09.344640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:22.075739+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":293,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-11T21:09:13.523996+0000","last_change":"2024-03-11T21:08:13.139913+0000","last_active":"2024-03-11T21:09:13.523996+0000","last_peered":"2024-03-11T21:09:13.523996+0000","last_clean":"2024-03-11T21:09:13.523996+0000","last_became_active":"2024-03-11T21:06:28.760306+0000","last_became_peered":"2024-03-11T21:06:28.760306+0000","last_unstale":"2024-03-11T21:09:13.523996+0000","last_undegraded":"2024-03-11T21:09:13.523996+0000","last_fullsized":"2024-03-11T21:09:13.523996+0000","mapping_epoch":297,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:08:13.139773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:08:13.139773+0000","last_clean_scrub_stamp":"2024-03-11T21:08:13.139773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:27.982920+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"207'20","reported_seq":618,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-11T21:09:13.518731+0000","last_change":"2024-03-11T21:07:46.374511+0000","last_active":"2024-03-11T21:09:13.518731+0000","last_peered":"2024-03-11T21:09:13.518731+0000","last_clean":"2024-03-11T21:09:13.518731+0000","last_became_active":"2024-03-11T20:50:30.101721+0000","last_became_peered":"2024-03-11T20:50:30.101721+0000","last_unstale":"2024-03-11T21:09:13.518731+0000","last_undegraded":"2024-03-11T21:09:13.518731+0000","last_fullsized":"2024-03-11T21:09:13.518731+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'20","last_scrub_stamp":"2024-03-11T21:07:46.374362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:07:46.374362+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":412,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-03-11T21:09:15.330574+0000","last_change":"2024-03-11T21:08:10.985718+0000","last_active":"2024-03-11T21:09:15.330574+0000","last_peered":"2024-03-11T21:09:15.330574+0000","last_clean":"2024-03-11T21:09:15.330574+0000","last_became_active":"2024-03-11T21:05:17.316714+0000","last_became_peered":"2024-03-11T21:05:17.316714+0000","last_unstale":"2024-03-11T21:09:15.330574+0000","last_undegraded":"2024-03-11T21:09:15.330574+0000","last_fullsized":"2024-03-11T21:09:15.330574+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:08:10.985486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:55.545758+0000","last_clean_scrub_stamp":"2024-03-11T21:08:10.985486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:19.423326+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":414,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-11T21:09:13.508964+0000","last_change":"2024-03-11T21:08:16.455139+0000","last_active":"2024-03-11T21:09:13.508964+0000","last_peered":"2024-03-11T21:09:13.508964+0000","last_clean":"2024-03-11T21:09:13.508964+0000","last_became_active":"2024-03-11T21:05:17.319272+0000","last_became_peered":"2024-03-11T21:05:17.319272+0000","last_unstale":"2024-03-11T21:09:13.508964+0000","last_undegraded":"2024-03-11T21:09:13.508964+0000","last_fullsized":"2024-03-11T21:09:13.508964+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:08:16.455026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:54:13.660325+0000","last_clean_scrub_stamp":"2024-03-11T21:08:16.455026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:37.154803+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":583,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-11T21:09:13.517947+0000","last_change":"2024-03-11T21:06:26.534331+0000","last_active":"2024-03-11T21:09:13.517947+0000","last_peered":"2024-03-11T21:09:13.517947+0000","last_clean":"2024-03-11T21:09:13.517947+0000","last_became_active":"2024-03-11T20:50:28.058274+0000","last_became_peered":"2024-03-11T20:50:28.058274+0000","last_unstale":"2024-03-11T21:09:13.517947+0000","last_undegraded":"2024-03-11T21:09:13.517947+0000","last_fullsized":"2024-03-11T21:09:13.517947+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:06:26.534045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:06:26.534045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"207'19","reported_seq":639,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-11T21:09:13.523833+0000","last_change":"2024-03-11T21:08:05.905831+0000","last_active":"2024-03-11T21:09:13.523833+0000","last_peered":"2024-03-11T21:09:13.523833+0000","last_clean":"2024-03-11T21:09:13.523833+0000","last_became_active":"2024-03-11T20:50:30.092638+0000","last_became_peered":"2024-03-11T20:50:30.092638+0000","last_unstale":"2024-03-11T21:09:13.523833+0000","last_undegraded":"2024-03-11T21:09:13.523833+0000","last_fullsized":"2024-03-11T21:09:13.523833+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'19","last_scrub_stamp":"2024-03-11T21:08:05.905696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:08:05.905696+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:31.482408+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":600,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-03-11T21:09:15.330880+0000","last_change":"2024-03-11T21:08:05.992436+0000","last_active":"2024-03-11T21:09:15.330880+0000","last_peered":"2024-03-11T21:09:15.330880+0000","last_clean":"2024-03-11T21:09:15.330880+0000","last_became_active":"2024-03-11T21:00:48.460694+0000","last_became_peered":"2024-03-11T21:00:48.460694+0000","last_unstale":"2024-03-11T21:09:15.330880+0000","last_undegraded":"2024-03-11T21:09:15.330880+0000","last_fullsized":"2024-03-11T21:09:15.330880+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:08:05.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:53:07.792697+0000","last_clean_scrub_stamp":"2024-03-11T21:08:05.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:22.040040+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":398,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-11T21:09:13.516686+0000","last_change":"2024-03-11T21:06:32.574420+0000","last_active":"2024-03-11T21:09:13.516686+0000","last_peered":"2024-03-11T21:09:13.516686+0000","last_clean":"2024-03-11T21:09:13.516686+0000","last_became_active":"2024-03-11T21:05:17.316491+0000","last_became_peered":"2024-03-11T21:05:17.316491+0000","last_unstale":"2024-03-11T21:09:13.516686+0000","last_undegraded":"2024-03-11T21:09:13.516686+0000","last_fullsized":"2024-03-11T21:09:13.516686+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:06:32.574201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:03:36.230647+0000","last_clean_scrub_stamp":"2024-03-11T21:06:32.574201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":603,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-11T21:09:13.522757+0000","last_change":"2024-03-11T21:07:44.227628+0000","last_active":"2024-03-11T21:09:13.522757+0000","last_peered":"2024-03-11T21:09:13.522757+0000","last_clean":"2024-03-11T21:09:13.522757+0000","last_became_active":"2024-03-11T20:50:28.079225+0000","last_became_peered":"2024-03-11T20:50:28.079225+0000","last_unstale":"2024-03-11T21:09:13.522757+0000","last_undegraded":"2024-03-11T21:09:13.522757+0000","last_fullsized":"2024-03-11T21:09:13.522757+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:07:44.227438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:07:44.227438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:16.294919+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"207'27","reported_seq":703,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-03-11T21:09:15.331011+0000","last_change":"2024-03-11T21:07:37.782008+0000","last_active":"2024-03-11T21:09:15.331011+0000","last_peered":"2024-03-11T21:09:15.331011+0000","last_clean":"2024-03-11T21:09:15.331011+0000","last_became_active":"2024-03-11T21:00:48.462943+0000","last_became_peered":"2024-03-11T21:00:48.462943+0000","last_unstale":"2024-03-11T21:09:15.331011+0000","last_undegraded":"2024-03-11T21:09:15.331011+0000","last_fullsized":"2024-03-11T21:09:15.331011+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'27","last_scrub_stamp":"2024-03-11T21:07:37.781649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:07:37.781649+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:18.847626+0000","scrub_duration":139,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":286,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-11T21:09:13.522210+0000","last_change":"2024-03-11T21:07:43.867303+0000","last_active":"2024-03-11T21:09:13.522210+0000","last_peered":"2024-03-11T21:09:13.522210+0000","last_clean":"2024-03-11T21:09:13.522210+0000","last_became_active":"2024-03-11T21:06:28.757201+0000","last_became_peered":"2024-03-11T21:06:28.757201+0000","last_unstale":"2024-03-11T21:09:13.522210+0000","last_undegraded":"2024-03-11T21:09:13.522210+0000","last_fullsized":"2024-03-11T21:09:13.522210+0000","mapping_epoch":297,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:07:43.867063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:07:43.867063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":368,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-11T21:09:13.516227+0000","last_change":"2024-03-11T21:08:01.360006+0000","last_active":"2024-03-11T21:09:13.516227+0000","last_peered":"2024-03-11T21:09:13.516227+0000","last_clean":"2024-03-11T21:09:13.516227+0000","last_became_active":"2024-03-11T21:05:17.313388+0000","last_became_peered":"2024-03-11T21:05:17.313388+0000","last_unstale":"2024-03-11T21:09:13.516227+0000","last_undegraded":"2024-03-11T21:09:13.516227+0000","last_fullsized":"2024-03-11T21:09:13.516227+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:08:01.359878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:53:50.599029+0000","last_clean_scrub_stamp":"2024-03-11T21:08:01.359878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:23.039160+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":611,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-11T21:09:13.522193+0000","last_change":"2024-03-11T21:08:03.095456+0000","last_active":"2024-03-11T21:09:13.522193+0000","last_peered":"2024-03-11T21:09:13.522193+0000","last_clean":"2024-03-11T21:09:13.522193+0000","last_became_active":"2024-03-11T20:50:28.068978+0000","last_became_peered":"2024-03-11T20:50:28.068978+0000","last_unstale":"2024-03-11T21:09:13.522193+0000","last_undegraded":"2024-03-11T21:09:13.522193+0000","last_fullsized":"2024-03-11T21:09:13.522193+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:08:03.095340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:54:07.155130+0000","last_clean_scrub_stamp":"2024-03-11T21:08:03.095340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:18.246317+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"207'23","reported_seq":641,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-03-11T21:09:15.331489+0000","last_change":"2024-03-11T21:08:23.793392+0000","last_active":"2024-03-11T21:09:15.331489+0000","last_peered":"2024-03-11T21:09:15.331489+0000","last_clean":"2024-03-11T21:09:15.331489+0000","last_became_active":"2024-03-11T21:00:27.952093+0000","last_became_peered":"2024-03-11T21:00:27.952093+0000","last_unstale":"2024-03-11T21:09:15.331489+0000","last_undegraded":"2024-03-11T21:09:15.331489+0000","last_fullsized":"2024-03-11T21:09:15.331489+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'23","last_scrub_stamp":"2024-03-11T21:08:23.793245+0000","last_deep_scrub":"110'17","last_deep_scrub_stamp":"2024-03-11T20:53:04.802488+0000","last_clean_scrub_stamp":"2024-03-11T21:08:23.793245+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:41.066649+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":578,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-11T21:09:13.522387+0000","last_change":"2024-03-11T21:08:18.541547+0000","last_active":"2024-03-11T21:09:13.522387+0000","last_peered":"2024-03-11T21:09:13.522387+0000","last_clean":"2024-03-11T21:09:13.522387+0000","last_became_active":"2024-03-11T20:50:34.334558+0000","last_became_peered":"2024-03-11T20:50:34.334558+0000","last_unstale":"2024-03-11T21:09:13.522387+0000","last_undegraded":"2024-03-11T21:09:13.522387+0000","last_fullsized":"2024-03-11T21:09:13.522387+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:08:18.541431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:55:59.892394+0000","last_clean_scrub_stamp":"2024-03-11T21:08:18.541431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:22.496111+0000","scrub_duration":163,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":431,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-03-11T21:09:15.331173+0000","last_change":"2024-03-11T21:08:13.903187+0000","last_active":"2024-03-11T21:09:15.331173+0000","last_peered":"2024-03-11T21:09:15.331173+0000","last_clean":"2024-03-11T21:09:15.331173+0000","last_became_active":"2024-03-11T21:00:48.461013+0000","last_became_peered":"2024-03-11T21:00:48.461013+0000","last_unstale":"2024-03-11T21:09:15.331173+0000","last_undegraded":"2024-03-11T21:09:15.331173+0000","last_fullsized":"2024-03-11T21:09:15.331173+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:08:13.903063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:08:13.903063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"77'1","reported_seq":597,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-11T21:09:13.516768+0000","last_change":"2024-03-11T21:07:38.421921+0000","last_active":"2024-03-11T21:09:13.516768+0000","last_peered":"2024-03-11T21:09:13.516768+0000","last_clean":"2024-03-11T21:09:13.516768+0000","last_became_active":"2024-03-11T20:50:28.068840+0000","last_became_peered":"2024-03-11T20:50:28.068840+0000","last_unstale":"2024-03-11T21:09:13.516768+0000","last_undegraded":"2024-03-11T21:09:13.516768+0000","last_fullsized":"2024-03-11T21:09:13.516768+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-03-11T21:07:38.421793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:07:38.421793+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"207'19","reported_seq":452,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:09:16.570308+0000","last_change":"2024-03-11T21:07:48.503708+0000","last_active":"2024-03-11T21:09:16.570308+0000","last_peered":"2024-03-11T21:09:16.570308+0000","last_clean":"2024-03-11T21:09:16.570308+0000","last_became_active":"2024-03-11T21:05:17.443030+0000","last_became_peered":"2024-03-11T21:05:17.443030+0000","last_unstale":"2024-03-11T21:09:16.570308+0000","last_undegraded":"2024-03-11T21:09:16.570308+0000","last_fullsized":"2024-03-11T21:09:16.570308+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'19","last_scrub_stamp":"2024-03-11T21:07:48.503147+0000","last_deep_scrub":"207'19","last_deep_scrub_stamp":"2024-03-11T21:03:39.240089+0000","last_clean_scrub_stamp":"2024-03-11T21:07:48.503147+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:20.236544+0000","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":370,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-11T21:09:13.510482+0000","last_change":"2024-03-11T21:08:02.490787+0000","last_active":"2024-03-11T21:09:13.510482+0000","last_peered":"2024-03-11T21:09:13.510482+0000","last_clean":"2024-03-11T21:09:13.510482+0000","last_became_active":"2024-03-11T21:05:17.313986+0000","last_became_peered":"2024-03-11T21:05:17.313986+0000","last_unstale":"2024-03-11T21:09:13.510482+0000","last_undegraded":"2024-03-11T21:09:13.510482+0000","last_fullsized":"2024-03-11T21:09:13.510482+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:08:02.490701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:33.399510+0000","last_clean_scrub_stamp":"2024-03-11T21:08:02.490701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:32.053442+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":385,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:09:16.570032+0000","last_change":"2024-03-11T21:08:05.299006+0000","last_active":"2024-03-11T21:09:16.570032+0000","last_peered":"2024-03-11T21:09:16.570032+0000","last_clean":"2024-03-11T21:09:16.570032+0000","last_became_active":"2024-03-11T21:08:05.298552+0000","last_became_peered":"2024-03-11T21:08:05.298552+0000","last_unstale":"2024-03-11T21:09:16.570032+0000","last_undegraded":"2024-03-11T21:09:16.570032+0000","last_fullsized":"2024-03-11T21:09:16.570032+0000","mapping_epoch":319,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:07:48.513229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:24.666325+0000","last_clean_scrub_stamp":"2024-03-11T21:07:48.513229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:20.236448+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":612,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-11T21:09:13.509399+0000","last_change":"2024-03-11T21:08:01.481718+0000","last_active":"2024-03-11T21:09:13.509399+0000","last_peered":"2024-03-11T21:09:13.509399+0000","last_clean":"2024-03-11T21:09:13.509399+0000","last_became_active":"2024-03-11T21:00:27.956935+0000","last_became_peered":"2024-03-11T21:00:27.956935+0000","last_unstale":"2024-03-11T21:09:13.509399+0000","last_undegraded":"2024-03-11T21:09:13.509399+0000","last_fullsized":"2024-03-11T21:09:13.509399+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:08:01.481609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:08:01.481609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"207'7","reported_seq":620,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-11T21:09:13.522521+0000","last_change":"2024-03-11T21:08:00.885293+0000","last_active":"2024-03-11T21:09:13.522521+0000","last_peered":"2024-03-11T21:09:13.522521+0000","last_clean":"2024-03-11T21:09:13.522521+0000","last_became_active":"2024-03-11T21:00:27.955296+0000","last_became_peered":"2024-03-11T21:00:27.955296+0000","last_unstale":"2024-03-11T21:09:13.522521+0000","last_undegraded":"2024-03-11T21:09:13.522521+0000","last_fullsized":"2024-03-11T21:09:13.522521+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'7","last_scrub_stamp":"2024-03-11T21:08:00.885119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:08:00.885119+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":308,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:09:16.570497+0000","last_change":"2024-03-11T21:08:08.622255+0000","last_active":"2024-03-11T21:09:16.570497+0000","last_peered":"2024-03-11T21:09:16.570497+0000","last_clean":"2024-03-11T21:09:16.570497+0000","last_became_active":"2024-03-11T21:00:48.464694+0000","last_became_peered":"2024-03-11T21:00:48.464694+0000","last_unstale":"2024-03-11T21:09:16.570497+0000","last_undegraded":"2024-03-11T21:09:16.570497+0000","last_fullsized":"2024-03-11T21:09:16.570497+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:08:08.622054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:08:08.622054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:36.228544+0000","scrub_duration":107,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1f","version":"212'16","reported_seq":865,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-11T21:09:14.024274+0000","last_change":"2024-03-11T21:08:11.449942+0000","last_active":"2024-03-11T21:09:14.024274+0000","last_peered":"2024-03-11T21:09:14.024274+0000","last_clean":"2024-03-11T21:09:14.024274+0000","last_became_active":"2024-03-11T21:00:48.464350+0000","last_became_peered":"2024-03-11T21:00:48.464350+0000","last_unstale":"2024-03-11T21:09:14.024274+0000","last_undegraded":"2024-03-11T21:09:14.024274+0000","last_fullsized":"2024-03-11T21:09:14.024274+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"212'16","last_scrub_stamp":"2024-03-11T21:08:11.449706+0000","last_deep_scrub":"212'16","last_deep_scrub_stamp":"2024-03-11T21:06:47.821221+0000","last_clean_scrub_stamp":"2024-03-11T21:08:11.449706+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:19.579717+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"77'1","reported_seq":582,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-11T21:09:13.516467+0000","last_change":"2024-03-11T21:07:37.448905+0000","last_active":"2024-03-11T21:09:13.516467+0000","last_peered":"2024-03-11T21:09:13.516467+0000","last_clean":"2024-03-11T21:09:13.516467+0000","last_became_active":"2024-03-11T20:50:28.068827+0000","last_became_peered":"2024-03-11T20:50:28.068827+0000","last_unstale":"2024-03-11T21:09:13.516467+0000","last_undegraded":"2024-03-11T21:09:13.516467+0000","last_fullsized":"2024-03-11T21:09:13.516467+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-03-11T21:07:37.448705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:07:37.448705+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"207'15","reported_seq":652,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-11T21:09:13.515851+0000","last_change":"2024-03-11T21:08:21.296049+0000","last_active":"2024-03-11T21:09:13.515851+0000","last_peered":"2024-03-11T21:09:13.515851+0000","last_clean":"2024-03-11T21:09:13.515851+0000","last_became_active":"2024-03-11T21:00:48.467513+0000","last_became_peered":"2024-03-11T21:00:48.467513+0000","last_unstale":"2024-03-11T21:09:13.515851+0000","last_undegraded":"2024-03-11T21:09:13.515851+0000","last_fullsized":"2024-03-11T21:09:13.515851+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'15","last_scrub_stamp":"2024-03-11T21:08:21.295914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:08:21.295914+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:50.946102+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"105'2","reported_seq":594,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-11T21:09:13.522495+0000","last_change":"2024-03-11T21:07:43.187688+0000","last_active":"2024-03-11T21:09:13.522495+0000","last_peered":"2024-03-11T21:09:13.522495+0000","last_clean":"2024-03-11T21:09:13.522495+0000","last_became_active":"2024-03-11T20:50:34.327356+0000","last_became_peered":"2024-03-11T20:50:34.327356+0000","last_unstale":"2024-03-11T21:09:13.522495+0000","last_undegraded":"2024-03-11T21:09:13.522495+0000","last_fullsized":"2024-03-11T21:09:13.522495+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'2","last_scrub_stamp":"2024-03-11T21:07:43.187429+0000","last_deep_scrub":"105'2","last_deep_scrub_stamp":"2024-03-11T21:03:41.971918+0000","last_clean_scrub_stamp":"2024-03-11T21:07:43.187429+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:14.240835+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":293,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-11T21:09:13.523440+0000","last_change":"2024-03-11T21:07:38.776015+0000","last_active":"2024-03-11T21:09:13.523440+0000","last_peered":"2024-03-11T21:09:13.523440+0000","last_clean":"2024-03-11T21:09:13.523440+0000","last_became_active":"2024-03-11T21:06:28.759784+0000","last_became_peered":"2024-03-11T21:06:28.759784+0000","last_unstale":"2024-03-11T21:09:13.523440+0000","last_undegraded":"2024-03-11T21:09:13.523440+0000","last_fullsized":"2024-03-11T21:09:13.523440+0000","mapping_epoch":297,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:07:38.775839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:06:32.968650+0000","last_clean_scrub_stamp":"2024-03-11T21:07:38.775839+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":608,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-03-11T21:09:15.330236+0000","last_change":"2024-03-11T21:08:27.756153+0000","last_active":"2024-03-11T21:09:15.330236+0000","last_peered":"2024-03-11T21:09:15.330236+0000","last_clean":"2024-03-11T21:09:15.330236+0000","last_became_active":"2024-03-11T21:00:27.947614+0000","last_became_peered":"2024-03-11T21:00:27.947614+0000","last_unstale":"2024-03-11T21:09:15.330236+0000","last_undegraded":"2024-03-11T21:09:15.330236+0000","last_fullsized":"2024-03-11T21:09:15.330236+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:08:27.756036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:08:27.756036+0000","last_clean_scrub_stamp":"2024-03-11T21:08:27.756036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:38.771269+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"207'27","reported_seq":346,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:09:16.569182+0000","last_change":"2024-03-11T21:08:19.539727+0000","last_active":"2024-03-11T21:09:16.569182+0000","last_peered":"2024-03-11T21:09:16.569182+0000","last_clean":"2024-03-11T21:09:16.569182+0000","last_became_active":"2024-03-11T21:00:48.457893+0000","last_became_peered":"2024-03-11T21:00:48.457893+0000","last_unstale":"2024-03-11T21:09:16.569182+0000","last_undegraded":"2024-03-11T21:09:16.569182+0000","last_fullsized":"2024-03-11T21:09:16.569182+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'27","last_scrub_stamp":"2024-03-11T21:08:19.539512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:08:19.539512+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:47.582896+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":577,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-11T21:09:13.517079+0000","last_change":"2024-03-11T21:07:41.485168+0000","last_active":"2024-03-11T21:09:13.517079+0000","last_peered":"2024-03-11T21:09:13.517079+0000","last_clean":"2024-03-11T21:09:13.517079+0000","last_became_active":"2024-03-11T20:50:34.337878+0000","last_became_peered":"2024-03-11T20:50:34.337878+0000","last_unstale":"2024-03-11T21:09:13.517079+0000","last_undegraded":"2024-03-11T21:09:13.517079+0000","last_fullsized":"2024-03-11T21:09:13.517079+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:07:41.485049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:52:11.462480+0000","last_clean_scrub_stamp":"2024-03-11T21:07:41.485049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":123,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":582,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-11T21:09:13.517189+0000","last_change":"2024-03-11T21:06:53.533564+0000","last_active":"2024-03-11T21:09:13.517189+0000","last_peered":"2024-03-11T21:09:13.517189+0000","last_clean":"2024-03-11T21:09:13.517189+0000","last_became_active":"2024-03-11T20:50:32.646323+0000","last_became_peered":"2024-03-11T20:50:32.646323+0000","last_unstale":"2024-03-11T21:09:13.517189+0000","last_undegraded":"2024-03-11T21:09:13.517189+0000","last_fullsized":"2024-03-11T21:09:13.517189+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:06:53.533352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:06:53.533352+0000","last_clean_scrub_stamp":"2024-03-11T21:06:53.533352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:18.713159+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":619,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-11T21:09:13.510136+0000","last_change":"2024-03-11T21:08:09.468417+0000","last_active":"2024-03-11T21:09:13.510136+0000","last_peered":"2024-03-11T21:09:13.510136+0000","last_clean":"2024-03-11T21:09:13.510136+0000","last_became_active":"2024-03-11T20:50:28.086513+0000","last_became_peered":"2024-03-11T20:50:28.086513+0000","last_unstale":"2024-03-11T21:09:13.510136+0000","last_undegraded":"2024-03-11T21:09:13.510136+0000","last_fullsized":"2024-03-11T21:09:13.510136+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:08:09.468339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:08:09.468339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:31.864241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"207'19","reported_seq":666,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-03-11T21:09:15.333155+0000","last_change":"2024-03-11T21:08:12.934811+0000","last_active":"2024-03-11T21:09:15.333155+0000","last_peered":"2024-03-11T21:09:15.333155+0000","last_clean":"2024-03-11T21:09:15.333155+0000","last_became_active":"2024-03-11T21:00:48.464707+0000","last_became_peered":"2024-03-11T21:00:48.464707+0000","last_unstale":"2024-03-11T21:09:15.333155+0000","last_undegraded":"2024-03-11T21:09:15.333155+0000","last_fullsized":"2024-03-11T21:09:15.333155+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'19","last_scrub_stamp":"2024-03-11T21:08:12.934635+0000","last_deep_scrub":"145'13","last_deep_scrub_stamp":"2024-03-11T20:55:53.560287+0000","last_clean_scrub_stamp":"2024-03-11T21:08:12.934635+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:30.699153+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":593,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-11T21:09:13.522011+0000","last_change":"2024-03-11T21:07:47.158773+0000","last_active":"2024-03-11T21:09:13.522011+0000","last_peered":"2024-03-11T21:09:13.522011+0000","last_clean":"2024-03-11T21:09:13.522011+0000","last_became_active":"2024-03-11T21:00:27.973405+0000","last_became_peered":"2024-03-11T21:00:27.973405+0000","last_unstale":"2024-03-11T21:09:13.522011+0000","last_undegraded":"2024-03-11T21:09:13.522011+0000","last_fullsized":"2024-03-11T21:09:13.522011+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:07:47.158672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:07:47.158672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":595,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-11T21:09:13.515907+0000","last_change":"2024-03-11T21:08:24.323683+0000","last_active":"2024-03-11T21:09:13.515907+0000","last_peered":"2024-03-11T21:09:13.515907+0000","last_clean":"2024-03-11T21:09:13.515907+0000","last_became_active":"2024-03-11T21:00:48.472812+0000","last_became_peered":"2024-03-11T21:00:48.472812+0000","last_unstale":"2024-03-11T21:09:13.515907+0000","last_undegraded":"2024-03-11T21:09:13.515907+0000","last_fullsized":"2024-03-11T21:09:13.515907+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:08:24.323601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:08:24.323601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:27.123101+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":607,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-03-11T21:09:15.333057+0000","last_change":"2024-03-11T21:07:56.868732+0000","last_active":"2024-03-11T21:09:15.333057+0000","last_peered":"2024-03-11T21:09:15.333057+0000","last_clean":"2024-03-11T21:09:15.333057+0000","last_became_active":"2024-03-11T20:50:28.075760+0000","last_became_peered":"2024-03-11T20:50:28.075760+0000","last_unstale":"2024-03-11T21:09:15.333057+0000","last_undegraded":"2024-03-11T21:09:15.333057+0000","last_fullsized":"2024-03-11T21:09:15.333057+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:07:56.868482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:58:55.977289+0000","last_clean_scrub_stamp":"2024-03-11T21:07:56.868482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:17.637518+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"207'12","reported_seq":261,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-11T21:09:13.523623+0000","last_change":"2024-03-11T21:07:57.887460+0000","last_active":"2024-03-11T21:09:13.523623+0000","last_peered":"2024-03-11T21:09:13.523623+0000","last_clean":"2024-03-11T21:09:13.523623+0000","last_became_active":"2024-03-11T21:06:28.763971+0000","last_became_peered":"2024-03-11T21:06:28.763971+0000","last_unstale":"2024-03-11T21:09:13.523623+0000","last_undegraded":"2024-03-11T21:09:13.523623+0000","last_fullsized":"2024-03-11T21:09:13.523623+0000","mapping_epoch":297,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'12","last_scrub_stamp":"2024-03-11T21:07:57.887318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:07:57.887318+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":16,"num_read_kb":13,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.11","version":"186'2","reported_seq":579,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-11T21:09:13.516003+0000","last_change":"2024-03-11T21:08:00.343845+0000","last_active":"2024-03-11T21:09:13.516003+0000","last_peered":"2024-03-11T21:09:13.516003+0000","last_clean":"2024-03-11T21:09:13.516003+0000","last_became_active":"2024-03-11T20:50:34.339326+0000","last_became_peered":"2024-03-11T20:50:34.339326+0000","last_unstale":"2024-03-11T21:09:13.516003+0000","last_undegraded":"2024-03-11T21:09:13.516003+0000","last_fullsized":"2024-03-11T21:09:13.516003+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"186'2","last_scrub_stamp":"2024-03-11T21:08:00.343694+0000","last_deep_scrub":"186'2","last_deep_scrub_stamp":"2024-03-11T20:59:04.534850+0000","last_clean_scrub_stamp":"2024-03-11T21:08:00.343694+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:13.649759+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":414,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-11T21:09:13.510152+0000","last_change":"2024-03-11T21:07:57.484363+0000","last_active":"2024-03-11T21:09:13.510152+0000","last_peered":"2024-03-11T21:09:13.510152+0000","last_clean":"2024-03-11T21:09:13.510152+0000","last_became_active":"2024-03-11T21:05:17.314174+0000","last_became_peered":"2024-03-11T21:05:17.314174+0000","last_unstale":"2024-03-11T21:09:13.510152+0000","last_undegraded":"2024-03-11T21:09:13.510152+0000","last_fullsized":"2024-03-11T21:09:13.510152+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:07:57.484167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:07:57.484167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":308,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:09:16.570342+0000","last_change":"2024-03-11T21:08:21.529802+0000","last_active":"2024-03-11T21:09:16.570342+0000","last_peered":"2024-03-11T21:09:16.570342+0000","last_clean":"2024-03-11T21:09:16.570342+0000","last_became_active":"2024-03-11T21:00:48.455510+0000","last_became_peered":"2024-03-11T21:00:48.455510+0000","last_unstale":"2024-03-11T21:09:16.570342+0000","last_undegraded":"2024-03-11T21:09:16.570342+0000","last_fullsized":"2024-03-11T21:09:16.570342+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:08:21.529639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:08:21.529639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:40.099922+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"207'25","reported_seq":384,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-11T21:09:13.522382+0000","last_change":"2024-03-11T21:07:49.153573+0000","last_active":"2024-03-11T21:09:13.522382+0000","last_peered":"2024-03-11T21:09:13.522382+0000","last_clean":"2024-03-11T21:09:13.522382+0000","last_became_active":"2024-03-11T21:05:17.314507+0000","last_became_peered":"2024-03-11T21:05:17.314507+0000","last_unstale":"2024-03-11T21:09:13.522382+0000","last_undegraded":"2024-03-11T21:09:13.522382+0000","last_fullsized":"2024-03-11T21:09:13.522382+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'25","last_scrub_stamp":"2024-03-11T21:07:49.153339+0000","last_deep_scrub":"161'15","last_deep_scrub_stamp":"2024-03-11T20:57:15.125548+0000","last_clean_scrub_stamp":"2024-03-11T21:07:49.153339+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:14.240802+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":612,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-11T21:09:13.522346+0000","last_change":"2024-03-11T21:08:01.931240+0000","last_active":"2024-03-11T21:09:13.522346+0000","last_peered":"2024-03-11T21:09:13.522346+0000","last_clean":"2024-03-11T21:09:13.522346+0000","last_became_active":"2024-03-11T21:00:48.462105+0000","last_became_peered":"2024-03-11T21:00:48.462105+0000","last_unstale":"2024-03-11T21:09:13.522346+0000","last_undegraded":"2024-03-11T21:09:13.522346+0000","last_fullsized":"2024-03-11T21:09:13.522346+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:08:01.931041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:08:01.931041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:19.047459+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":308,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:09:16.569698+0000","last_change":"2024-03-11T21:08:17.480632+0000","last_active":"2024-03-11T21:09:16.569698+0000","last_peered":"2024-03-11T21:09:16.569698+0000","last_clean":"2024-03-11T21:09:16.569698+0000","last_became_active":"2024-03-11T21:00:48.459043+0000","last_became_peered":"2024-03-11T21:00:48.459043+0000","last_unstale":"2024-03-11T21:09:16.569698+0000","last_undegraded":"2024-03-11T21:09:16.569698+0000","last_fullsized":"2024-03-11T21:09:16.569698+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:08:17.480534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:23.137832+0000","last_clean_scrub_stamp":"2024-03-11T21:08:17.480534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:43.791504+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":612,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-11T21:09:13.511011+0000","last_change":"2024-03-11T21:07:43.593361+0000","last_active":"2024-03-11T21:09:13.511011+0000","last_peered":"2024-03-11T21:09:13.511011+0000","last_clean":"2024-03-11T21:09:13.511011+0000","last_became_active":"2024-03-11T20:50:28.082737+0000","last_became_peered":"2024-03-11T20:50:28.082737+0000","last_unstale":"2024-03-11T21:09:13.511011+0000","last_undegraded":"2024-03-11T21:09:13.511011+0000","last_fullsized":"2024-03-11T21:09:13.511011+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:07:43.593282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:28.404997+0000","last_clean_scrub_stamp":"2024-03-11T21:07:43.593282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:19.579556+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"207'15","reported_seq":625,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-11T21:09:13.522941+0000","last_change":"2024-03-11T21:07:58.088523+0000","last_active":"2024-03-11T21:09:13.522941+0000","last_peered":"2024-03-11T21:09:13.522941+0000","last_clean":"2024-03-11T21:09:13.522941+0000","last_became_active":"2024-03-11T21:00:27.967624+0000","last_became_peered":"2024-03-11T21:00:27.967624+0000","last_unstale":"2024-03-11T21:09:13.522941+0000","last_undegraded":"2024-03-11T21:09:13.522941+0000","last_fullsized":"2024-03-11T21:09:13.522941+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'15","last_scrub_stamp":"2024-03-11T21:07:58.088382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:07:58.088382+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:14.240774+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":250,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-03-11T21:09:15.332982+0000","last_change":"2024-03-11T21:08:05.300696+0000","last_active":"2024-03-11T21:09:15.332982+0000","last_peered":"2024-03-11T21:09:15.332982+0000","last_clean":"2024-03-11T21:09:15.332982+0000","last_became_active":"2024-03-11T21:08:05.299913+0000","last_became_peered":"2024-03-11T21:08:05.299913+0000","last_unstale":"2024-03-11T21:09:15.332982+0000","last_undegraded":"2024-03-11T21:09:15.332982+0000","last_fullsized":"2024-03-11T21:09:15.332982+0000","mapping_epoch":319,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:07:40.457227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:57:16.032302+0000","last_clean_scrub_stamp":"2024-03-11T21:07:40.457227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"112'12","reported_seq":828,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-03-11T21:09:15.331643+0000","last_change":"2024-03-11T21:08:11.973519+0000","last_active":"2024-03-11T21:09:15.331643+0000","last_peered":"2024-03-11T21:09:15.331643+0000","last_clean":"2024-03-11T21:09:15.331643+0000","last_became_active":"2024-03-11T20:50:32.650064+0000","last_became_peered":"2024-03-11T20:50:32.650064+0000","last_unstale":"2024-03-11T21:09:15.331643+0000","last_undegraded":"2024-03-11T21:09:15.331643+0000","last_fullsized":"2024-03-11T21:09:15.331643+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'12","last_scrub_stamp":"2024-03-11T21:08:11.973404+0000","last_deep_scrub":"112'12","last_deep_scrub_stamp":"2024-03-11T20:53:11.843951+0000","last_clean_scrub_stamp":"2024-03-11T21:08:11.973404+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:18.838738+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":300,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-11T21:09:13.520904+0000","last_change":"2024-03-11T21:06:52.863103+0000","last_active":"2024-03-11T21:09:13.520904+0000","last_peered":"2024-03-11T21:09:13.520904+0000","last_clean":"2024-03-11T21:09:13.520904+0000","last_became_active":"2024-03-11T21:06:28.756041+0000","last_became_peered":"2024-03-11T21:06:28.756041+0000","last_unstale":"2024-03-11T21:09:13.520904+0000","last_undegraded":"2024-03-11T21:09:13.520904+0000","last_fullsized":"2024-03-11T21:09:13.520904+0000","mapping_epoch":297,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:06:52.862971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:06:52.862971+0000","last_clean_scrub_stamp":"2024-03-11T21:06:52.862971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"207'19","reported_seq":359,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-11T21:09:13.521331+0000","last_change":"2024-03-11T21:08:21.328028+0000","last_active":"2024-03-11T21:09:13.521331+0000","last_peered":"2024-03-11T21:09:13.521331+0000","last_clean":"2024-03-11T21:09:13.521331+0000","last_became_active":"2024-03-11T21:06:28.758223+0000","last_became_peered":"2024-03-11T21:06:28.758223+0000","last_unstale":"2024-03-11T21:09:13.521331+0000","last_undegraded":"2024-03-11T21:09:13.521331+0000","last_fullsized":"2024-03-11T21:09:13.521331+0000","mapping_epoch":297,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'19","last_scrub_stamp":"2024-03-11T21:08:21.327572+0000","last_deep_scrub":"119'13","last_deep_scrub_stamp":"2024-03-11T20:54:15.443472+0000","last_clean_scrub_stamp":"2024-03-11T21:08:21.327572+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:42.323400+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":280,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-11T21:09:13.523791+0000","last_change":"2024-03-11T21:07:48.864558+0000","last_active":"2024-03-11T21:09:13.523791+0000","last_peered":"2024-03-11T21:09:13.523791+0000","last_clean":"2024-03-11T21:09:13.523791+0000","last_became_active":"2024-03-11T21:06:28.758634+0000","last_became_peered":"2024-03-11T21:06:28.758634+0000","last_unstale":"2024-03-11T21:09:13.523791+0000","last_undegraded":"2024-03-11T21:09:13.523791+0000","last_fullsized":"2024-03-11T21:09:13.523791+0000","mapping_epoch":297,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:07:48.864448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:07:48.864448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":59,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"212'17","reported_seq":849,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-11T21:09:13.518277+0000","last_change":"2024-03-11T21:08:20.327834+0000","last_active":"2024-03-11T21:09:13.518277+0000","last_peered":"2024-03-11T21:09:13.518277+0000","last_clean":"2024-03-11T21:09:13.518277+0000","last_became_active":"2024-03-11T21:00:48.467116+0000","last_became_peered":"2024-03-11T21:00:48.467116+0000","last_unstale":"2024-03-11T21:09:13.518277+0000","last_undegraded":"2024-03-11T21:09:13.518277+0000","last_fullsized":"2024-03-11T21:09:13.518277+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"212'17","last_scrub_stamp":"2024-03-11T21:08:20.327696+0000","last_deep_scrub":"110'13","last_deep_scrub_stamp":"2024-03-11T20:53:00.434532+0000","last_clean_scrub_stamp":"2024-03-11T21:08:20.327696+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:34.643918+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":608,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-03-11T21:09:15.330628+0000","last_change":"2024-03-11T21:08:04.974358+0000","last_active":"2024-03-11T21:09:15.330628+0000","last_peered":"2024-03-11T21:09:15.330628+0000","last_clean":"2024-03-11T21:09:15.330628+0000","last_became_active":"2024-03-11T20:50:28.070325+0000","last_became_peered":"2024-03-11T20:50:28.070325+0000","last_unstale":"2024-03-11T21:09:15.330628+0000","last_undegraded":"2024-03-11T21:09:15.330628+0000","last_fullsized":"2024-03-11T21:09:15.330628+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:08:04.974102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:08:04.974102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:18.847716+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"207'24","reported_seq":637,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-11T21:09:13.522334+0000","last_change":"2024-03-11T21:07:46.201396+0000","last_active":"2024-03-11T21:09:13.522334+0000","last_peered":"2024-03-11T21:09:13.522334+0000","last_clean":"2024-03-11T21:09:13.522334+0000","last_became_active":"2024-03-11T21:00:27.967078+0000","last_became_peered":"2024-03-11T21:00:27.967078+0000","last_unstale":"2024-03-11T21:09:13.522334+0000","last_undegraded":"2024-03-11T21:09:13.522334+0000","last_fullsized":"2024-03-11T21:09:13.522334+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'24","last_scrub_stamp":"2024-03-11T21:07:46.201244+0000","last_deep_scrub":"116'16","last_deep_scrub_stamp":"2024-03-11T20:53:39.292118+0000","last_clean_scrub_stamp":"2024-03-11T21:07:46.201244+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:14.240686+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":584,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-11T21:09:13.522282+0000","last_change":"2024-03-11T21:07:45.229735+0000","last_active":"2024-03-11T21:09:13.522282+0000","last_peered":"2024-03-11T21:09:13.522282+0000","last_clean":"2024-03-11T21:09:13.522282+0000","last_became_active":"2024-03-11T21:00:27.966997+0000","last_became_peered":"2024-03-11T21:00:27.966997+0000","last_unstale":"2024-03-11T21:09:13.522282+0000","last_undegraded":"2024-03-11T21:09:13.522282+0000","last_fullsized":"2024-03-11T21:09:13.522282+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:07:45.229582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:55:58.016988+0000","last_clean_scrub_stamp":"2024-03-11T21:07:45.229582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:14.240877+0000","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":594,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-11T21:09:13.522511+0000","last_change":"2024-03-11T21:06:16.515576+0000","last_active":"2024-03-11T21:09:13.522511+0000","last_peered":"2024-03-11T21:09:13.522511+0000","last_clean":"2024-03-11T21:09:13.522511+0000","last_became_active":"2024-03-11T21:00:48.468386+0000","last_became_peered":"2024-03-11T21:00:48.468386+0000","last_unstale":"2024-03-11T21:09:13.522511+0000","last_undegraded":"2024-03-11T21:09:13.522511+0000","last_fullsized":"2024-03-11T21:09:13.522511+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:06:16.515117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:06:16.515117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"166'2","reported_seq":647,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-11T21:09:13.508737+0000","last_change":"2024-03-11T21:08:07.483308+0000","last_active":"2024-03-11T21:09:13.508737+0000","last_peered":"2024-03-11T21:09:13.508737+0000","last_clean":"2024-03-11T21:09:13.508737+0000","last_became_active":"2024-03-11T21:00:48.466071+0000","last_became_peered":"2024-03-11T21:00:48.466071+0000","last_unstale":"2024-03-11T21:09:13.508737+0000","last_undegraded":"2024-03-11T21:09:13.508737+0000","last_fullsized":"2024-03-11T21:09:13.508737+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'2","last_scrub_stamp":"2024-03-11T21:08:07.483168+0000","last_deep_scrub":"166'2","last_deep_scrub_stamp":"2024-03-11T20:57:39.868776+0000","last_clean_scrub_stamp":"2024-03-11T21:08:07.483168+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1822,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"207'8","reported_seq":612,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-11T21:09:13.508759+0000","last_change":"2024-03-11T21:07:59.465898+0000","last_active":"2024-03-11T21:09:13.508759+0000","last_peered":"2024-03-11T21:09:13.508759+0000","last_clean":"2024-03-11T21:09:13.508759+0000","last_became_active":"2024-03-11T20:50:30.101572+0000","last_became_peered":"2024-03-11T20:50:30.101572+0000","last_unstale":"2024-03-11T21:09:13.508759+0000","last_undegraded":"2024-03-11T21:09:13.508759+0000","last_fullsized":"2024-03-11T21:09:13.508759+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'8","last_scrub_stamp":"2024-03-11T21:07:59.463853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:07:59.463853+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":347,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-11T21:09:13.522570+0000","last_change":"2024-03-11T21:08:05.297690+0000","last_active":"2024-03-11T21:09:13.522570+0000","last_peered":"2024-03-11T21:09:13.522570+0000","last_clean":"2024-03-11T21:09:13.522570+0000","last_became_active":"2024-03-11T21:08:05.296676+0000","last_became_peered":"2024-03-11T21:08:05.296676+0000","last_unstale":"2024-03-11T21:09:13.522570+0000","last_undegraded":"2024-03-11T21:09:13.522570+0000","last_fullsized":"2024-03-11T21:09:13.522570+0000","mapping_epoch":319,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:07:45.355422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:07:45.355422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:14.240861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":306,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-11T21:09:13.523924+0000","last_change":"2024-03-11T21:08:18.233619+0000","last_active":"2024-03-11T21:09:13.523924+0000","last_peered":"2024-03-11T21:09:13.523924+0000","last_clean":"2024-03-11T21:09:13.523924+0000","last_became_active":"2024-03-11T21:06:28.758133+0000","last_became_peered":"2024-03-11T21:06:28.758133+0000","last_unstale":"2024-03-11T21:09:13.523924+0000","last_undegraded":"2024-03-11T21:09:13.523924+0000","last_fullsized":"2024-03-11T21:09:13.523924+0000","mapping_epoch":297,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:08:18.233542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:54:09.537972+0000","last_clean_scrub_stamp":"2024-03-11T21:08:18.233542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:18.981775+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":373,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:09:16.569119+0000","last_change":"2024-03-11T21:08:05.301250+0000","last_active":"2024-03-11T21:09:16.569119+0000","last_peered":"2024-03-11T21:09:16.569119+0000","last_clean":"2024-03-11T21:09:16.569119+0000","last_became_active":"2024-03-11T21:08:05.300426+0000","last_became_peered":"2024-03-11T21:08:05.300426+0000","last_unstale":"2024-03-11T21:09:16.569119+0000","last_undegraded":"2024-03-11T21:09:16.569119+0000","last_fullsized":"2024-03-11T21:09:16.569119+0000","mapping_epoch":319,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:07:41.377285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:18.682947+0000","last_clean_scrub_stamp":"2024-03-11T21:07:41.377285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:20.236367+0000","scrub_duration":211,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"262'24","reported_seq":637,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-11T21:09:13.521491+0000","last_change":"2024-03-11T21:08:18.124252+0000","last_active":"2024-03-11T21:09:13.521491+0000","last_peered":"2024-03-11T21:09:13.521491+0000","last_clean":"2024-03-11T21:09:13.521491+0000","last_became_active":"2024-03-11T21:00:27.970878+0000","last_became_peered":"2024-03-11T21:00:27.970878+0000","last_unstale":"2024-03-11T21:09:13.521491+0000","last_undegraded":"2024-03-11T21:09:13.521491+0000","last_fullsized":"2024-03-11T21:09:13.521491+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"262'24","last_scrub_stamp":"2024-03-11T21:08:18.124021+0000","last_deep_scrub":"262'24","last_deep_scrub_stamp":"2024-03-11T21:03:28.128719+0000","last_clean_scrub_stamp":"2024-03-11T21:08:18.124021+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:36.507595+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.7","version":"335'7301","reported_seq":7369,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-11T21:09:14.900206+0000","last_change":"2024-03-11T21:07:14.960555+0000","last_active":"2024-03-11T21:09:14.900206+0000","last_peered":"2024-03-11T21:09:14.900206+0000","last_clean":"2024-03-11T21:09:14.900206+0000","last_became_active":"2024-03-11T21:07:14.959711+0000","last_became_peered":"2024-03-11T21:07:14.959711+0000","last_unstale":"2024-03-11T21:09:14.900206+0000","last_undegraded":"2024-03-11T21:09:14.900206+0000","last_fullsized":"2024-03-11T21:09:14.900206+0000","mapping_epoch":306,"log_start":"316'1500","ondisk_log_start":"316'1500","created":306,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:07:13.846944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:07:13.846944+0000","last_clean_scrub_stamp":"2024-03-11T21:07:13.846944+0000","objects_scrubbed":0,"log_size":5801,"log_dups_size":0,"ondisk_log_size":5801,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:19.579411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":422772736,"num_objects":6451,"num_object_clones":0,"num_object_copies":19353,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6451,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7301,"num_write_kb":440064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":599,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-11T21:09:13.516985+0000","last_change":"2024-03-11T21:06:47.558351+0000","last_active":"2024-03-11T21:09:13.516985+0000","last_peered":"2024-03-11T21:09:13.516985+0000","last_clean":"2024-03-11T21:09:13.516985+0000","last_became_active":"2024-03-11T21:00:48.466853+0000","last_became_peered":"2024-03-11T21:00:48.466853+0000","last_unstale":"2024-03-11T21:09:13.516985+0000","last_undegraded":"2024-03-11T21:09:13.516985+0000","last_fullsized":"2024-03-11T21:09:13.516985+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:06:47.558214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:52:53.450885+0000","last_clean_scrub_stamp":"2024-03-11T21:06:47.558214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":95,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"77'2","reported_seq":104,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-11T21:09:13.523377+0000","last_change":"2024-03-11T21:08:04.913072+0000","last_active":"2024-03-11T21:09:13.523377+0000","last_peered":"2024-03-11T21:09:13.523377+0000","last_clean":"2024-03-11T21:09:13.523377+0000","last_became_active":"2024-03-11T21:06:28.759838+0000","last_became_peered":"2024-03-11T21:06:28.759838+0000","last_unstale":"2024-03-11T21:09:13.523377+0000","last_undegraded":"2024-03-11T21:09:13.523377+0000","last_fullsized":"2024-03-11T21:09:13.523377+0000","mapping_epoch":297,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'2","last_scrub_stamp":"2024-03-11T21:08:04.912948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:08:04.912948+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:25.763732+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"207'24","reported_seq":697,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-11T21:09:13.522849+0000","last_change":"2024-03-11T21:08:14.074734+0000","last_active":"2024-03-11T21:09:13.522849+0000","last_peered":"2024-03-11T21:09:13.522849+0000","last_clean":"2024-03-11T21:09:13.522849+0000","last_became_active":"2024-03-11T21:00:48.465254+0000","last_became_peered":"2024-03-11T21:00:48.465254+0000","last_unstale":"2024-03-11T21:09:13.522849+0000","last_undegraded":"2024-03-11T21:09:13.522849+0000","last_fullsized":"2024-03-11T21:09:13.522849+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'24","last_scrub_stamp":"2024-03-11T21:08:14.074642+0000","last_deep_scrub":"100'16","last_deep_scrub_stamp":"2024-03-11T20:56:08.036331+0000","last_clean_scrub_stamp":"2024-03-11T21:08:14.074642+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:38.968786+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"286'18","reported_seq":559,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:09:16.568975+0000","last_change":"2024-03-11T21:07:50.490186+0000","last_active":"2024-03-11T21:09:16.568975+0000","last_peered":"2024-03-11T21:09:16.568975+0000","last_clean":"2024-03-11T21:09:16.568975+0000","last_became_active":"2024-03-11T21:05:17.317915+0000","last_became_peered":"2024-03-11T21:05:17.317915+0000","last_unstale":"2024-03-11T21:09:16.568975+0000","last_undegraded":"2024-03-11T21:09:16.568975+0000","last_fullsized":"2024-03-11T21:09:16.568975+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"286'18","last_scrub_stamp":"2024-03-11T21:07:50.490054+0000","last_deep_scrub":"212'17","last_deep_scrub_stamp":"2024-03-11T21:03:38.258850+0000","last_clean_scrub_stamp":"2024-03-11T21:07:50.490054+0000","objects_scrubbed":1,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:17.332753+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.c","version":"335'7130","reported_seq":7198,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-11T21:09:14.822614+0000","last_change":"2024-03-11T21:07:14.952881+0000","last_active":"2024-03-11T21:09:14.822614+0000","last_peered":"2024-03-11T21:09:14.822614+0000","last_clean":"2024-03-11T21:09:14.822614+0000","last_became_active":"2024-03-11T21:07:14.952564+0000","last_became_peered":"2024-03-11T21:07:14.952564+0000","last_unstale":"2024-03-11T21:09:14.822614+0000","last_undegraded":"2024-03-11T21:09:14.822614+0000","last_fullsized":"2024-03-11T21:09:14.822614+0000","mapping_epoch":306,"log_start":"316'2200","ondisk_log_start":"316'2200","created":306,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:07:13.846944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:07:13.846944+0000","last_clean_scrub_stamp":"2024-03-11T21:07:13.846944+0000","objects_scrubbed":0,"log_size":4930,"log_dups_size":0,"ondisk_log_size":4930,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":411566080,"num_objects":6280,"num_object_clones":0,"num_object_copies":18840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6280,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7130,"num_write_kb":429120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":289,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-11T21:09:13.522392+0000","last_change":"2024-03-11T21:08:20.317891+0000","last_active":"2024-03-11T21:09:13.522392+0000","last_peered":"2024-03-11T21:09:13.522392+0000","last_clean":"2024-03-11T21:09:13.522392+0000","last_became_active":"2024-03-11T21:06:28.755689+0000","last_became_peered":"2024-03-11T21:06:28.755689+0000","last_unstale":"2024-03-11T21:09:13.522392+0000","last_undegraded":"2024-03-11T21:09:13.522392+0000","last_fullsized":"2024-03-11T21:09:13.522392+0000","mapping_epoch":297,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:08:20.317781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:57:21.568400+0000","last_clean_scrub_stamp":"2024-03-11T21:08:20.317781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:32.488402+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":309,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:09:16.570214+0000","last_change":"2024-03-11T21:08:00.626028+0000","last_active":"2024-03-11T21:09:16.570214+0000","last_peered":"2024-03-11T21:09:16.570214+0000","last_clean":"2024-03-11T21:09:16.570214+0000","last_became_active":"2024-03-11T21:00:48.455733+0000","last_became_peered":"2024-03-11T21:00:48.455733+0000","last_unstale":"2024-03-11T21:09:16.570214+0000","last_undegraded":"2024-03-11T21:09:16.570214+0000","last_fullsized":"2024-03-11T21:09:16.570214+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:08:00.625904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:20.096874+0000","last_clean_scrub_stamp":"2024-03-11T21:08:00.625904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:20.236510+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"207'28","reported_seq":863,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-03-11T21:09:15.331376+0000","last_change":"2024-03-11T21:07:45.687177+0000","last_active":"2024-03-11T21:09:15.331376+0000","last_peered":"2024-03-11T21:09:15.331376+0000","last_clean":"2024-03-11T21:09:15.331376+0000","last_became_active":"2024-03-11T20:50:30.094977+0000","last_became_peered":"2024-03-11T20:50:30.094977+0000","last_unstale":"2024-03-11T21:09:15.331376+0000","last_undegraded":"2024-03-11T21:09:15.331376+0000","last_fullsized":"2024-03-11T21:09:15.331376+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'28","last_scrub_stamp":"2024-03-11T21:07:45.686955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:07:45.686955+0000","objects_scrubbed":8,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:18.847589+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":309,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:09:16.569735+0000","last_change":"2024-03-11T21:08:07.565477+0000","last_active":"2024-03-11T21:09:16.569735+0000","last_peered":"2024-03-11T21:09:16.569735+0000","last_clean":"2024-03-11T21:09:16.569735+0000","last_became_active":"2024-03-11T21:00:48.457901+0000","last_became_peered":"2024-03-11T21:00:48.457901+0000","last_unstale":"2024-03-11T21:09:16.569735+0000","last_undegraded":"2024-03-11T21:09:16.569735+0000","last_fullsized":"2024-03-11T21:09:16.569735+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:08:07.565360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:52:10.160202+0000","last_clean_scrub_stamp":"2024-03-11T21:08:07.565360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:20.236526+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.a","version":"335'7198","reported_seq":7265,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-11T21:09:14.880569+0000","last_change":"2024-03-11T21:07:14.883958+0000","last_active":"2024-03-11T21:09:14.880569+0000","last_peered":"2024-03-11T21:09:14.880569+0000","last_clean":"2024-03-11T21:09:14.880569+0000","last_became_active":"2024-03-11T21:07:14.883606+0000","last_became_peered":"2024-03-11T21:07:14.883606+0000","last_unstale":"2024-03-11T21:09:14.880569+0000","last_undegraded":"2024-03-11T21:09:14.880569+0000","last_fullsized":"2024-03-11T21:09:14.880569+0000","mapping_epoch":306,"log_start":"316'1400","ondisk_log_start":"316'1400","created":306,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:07:13.846944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:07:13.846944+0000","last_clean_scrub_stamp":"2024-03-11T21:07:13.846944+0000","objects_scrubbed":0,"log_size":5798,"log_dups_size":0,"ondisk_log_size":5798,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:19.579493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":416415744,"num_objects":6354,"num_object_clones":0,"num_object_copies":19062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6354,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7198,"num_write_kb":433664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":579,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-11T21:09:13.518148+0000","last_change":"2024-03-11T21:06:25.518117+0000","last_active":"2024-03-11T21:09:13.518148+0000","last_peered":"2024-03-11T21:09:13.518148+0000","last_clean":"2024-03-11T21:09:13.518148+0000","last_became_active":"2024-03-11T21:00:27.959408+0000","last_became_peered":"2024-03-11T21:00:27.959408+0000","last_unstale":"2024-03-11T21:09:13.518148+0000","last_undegraded":"2024-03-11T21:09:13.518148+0000","last_fullsized":"2024-03-11T21:09:13.518148+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:06:25.518004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:06:25.518004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":622,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-03-11T21:09:15.332599+0000","last_change":"2024-03-11T21:08:25.794071+0000","last_active":"2024-03-11T21:09:15.332599+0000","last_peered":"2024-03-11T21:09:15.332599+0000","last_clean":"2024-03-11T21:09:15.332599+0000","last_became_active":"2024-03-11T21:00:48.471212+0000","last_became_peered":"2024-03-11T21:00:48.471212+0000","last_unstale":"2024-03-11T21:09:15.332599+0000","last_undegraded":"2024-03-11T21:09:15.332599+0000","last_fullsized":"2024-03-11T21:09:15.332599+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:08:25.793949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:33.823704+0000","last_clean_scrub_stamp":"2024-03-11T21:08:25.793949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:36.396192+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"207'37","reported_seq":428,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-11T21:09:13.521020+0000","last_change":"2024-03-11T21:07:41.769444+0000","last_active":"2024-03-11T21:09:13.521020+0000","last_peered":"2024-03-11T21:09:13.521020+0000","last_clean":"2024-03-11T21:09:13.521020+0000","last_became_active":"2024-03-11T21:06:28.756995+0000","last_became_peered":"2024-03-11T21:06:28.756995+0000","last_unstale":"2024-03-11T21:09:13.521020+0000","last_undegraded":"2024-03-11T21:09:13.521020+0000","last_fullsized":"2024-03-11T21:09:13.521020+0000","mapping_epoch":297,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'37","last_scrub_stamp":"2024-03-11T21:07:41.769305+0000","last_deep_scrub":"182'27","last_deep_scrub_stamp":"2024-03-11T20:58:52.058851+0000","last_clean_scrub_stamp":"2024-03-11T21:07:41.769305+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":309,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:09:16.569759+0000","last_change":"2024-03-11T21:07:57.635813+0000","last_active":"2024-03-11T21:09:16.569759+0000","last_peered":"2024-03-11T21:09:16.569759+0000","last_clean":"2024-03-11T21:09:16.569759+0000","last_became_active":"2024-03-11T21:00:48.458008+0000","last_became_peered":"2024-03-11T21:00:48.458008+0000","last_unstale":"2024-03-11T21:09:16.569759+0000","last_undegraded":"2024-03-11T21:09:16.569759+0000","last_fullsized":"2024-03-11T21:09:16.569759+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:07:57.635663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:33.319221+0000","last_clean_scrub_stamp":"2024-03-11T21:07:57.635663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:24.155564+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.9","version":"336'7220","reported_seq":7290,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-03-11T21:09:15.515178+0000","last_change":"2024-03-11T21:07:14.877587+0000","last_active":"2024-03-11T21:09:15.515178+0000","last_peered":"2024-03-11T21:09:15.515178+0000","last_clean":"2024-03-11T21:09:15.515178+0000","last_became_active":"2024-03-11T21:07:14.877262+0000","last_became_peered":"2024-03-11T21:07:14.877262+0000","last_unstale":"2024-03-11T21:09:15.515178+0000","last_undegraded":"2024-03-11T21:09:15.515178+0000","last_fullsized":"2024-03-11T21:09:15.515178+0000","mapping_epoch":306,"log_start":"316'1300","ondisk_log_start":"316'1300","created":306,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:07:13.846944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:07:13.846944+0000","last_clean_scrub_stamp":"2024-03-11T21:07:13.846944+0000","objects_scrubbed":0,"log_size":5920,"log_dups_size":0,"ondisk_log_size":5920,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":415367168,"num_objects":6338,"num_object_clones":0,"num_object_copies":19014,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6338,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7220,"num_write_kb":433856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":592,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-11T21:09:13.522450+0000","last_change":"2024-03-11T21:08:05.120894+0000","last_active":"2024-03-11T21:09:13.522450+0000","last_peered":"2024-03-11T21:09:13.522450+0000","last_clean":"2024-03-11T21:09:13.522450+0000","last_became_active":"2024-03-11T20:50:34.321038+0000","last_became_peered":"2024-03-11T20:50:34.321038+0000","last_unstale":"2024-03-11T21:09:13.522450+0000","last_undegraded":"2024-03-11T21:09:13.522450+0000","last_fullsized":"2024-03-11T21:09:13.522450+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:08:05.120651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:32.951104+0000","last_clean_scrub_stamp":"2024-03-11T21:08:05.120651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:16.294887+0000","scrub_duration":59,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"207'38","reported_seq":341,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:09:16.568921+0000","last_change":"2024-03-11T21:08:23.471517+0000","last_active":"2024-03-11T21:09:16.568921+0000","last_peered":"2024-03-11T21:09:16.568921+0000","last_clean":"2024-03-11T21:09:16.568921+0000","last_became_active":"2024-03-11T21:00:48.456178+0000","last_became_peered":"2024-03-11T21:00:48.456178+0000","last_unstale":"2024-03-11T21:09:16.568921+0000","last_undegraded":"2024-03-11T21:09:16.568921+0000","last_fullsized":"2024-03-11T21:09:16.568921+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'38","last_scrub_stamp":"2024-03-11T21:08:23.471361+0000","last_deep_scrub":"109'26","last_deep_scrub_stamp":"2024-03-11T20:52:49.216378+0000","last_clean_scrub_stamp":"2024-03-11T21:08:23.471361+0000","objects_scrubbed":8,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:42.364904+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":43,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-11T21:09:13.516714+0000","last_change":"2024-03-11T21:08:05.308997+0000","last_active":"2024-03-11T21:09:13.516714+0000","last_peered":"2024-03-11T21:09:13.516714+0000","last_clean":"2024-03-11T21:09:13.516714+0000","last_became_active":"2024-03-11T21:08:05.308608+0000","last_became_peered":"2024-03-11T21:08:05.308608+0000","last_unstale":"2024-03-11T21:09:13.516714+0000","last_undegraded":"2024-03-11T21:09:13.516714+0000","last_fullsized":"2024-03-11T21:09:13.516714+0000","mapping_epoch":319,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:07:43.347185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:07:43.347185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":300,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:09:16.570833+0000","last_change":"2024-03-11T21:07:39.371711+0000","last_active":"2024-03-11T21:09:16.570833+0000","last_peered":"2024-03-11T21:09:16.570833+0000","last_clean":"2024-03-11T21:09:16.570833+0000","last_became_active":"2024-03-11T21:00:48.473346+0000","last_became_peered":"2024-03-11T21:00:48.473346+0000","last_unstale":"2024-03-11T21:09:16.570833+0000","last_undegraded":"2024-03-11T21:09:16.570833+0000","last_fullsized":"2024-03-11T21:09:16.570833+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:07:39.371633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:07:39.371633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:20.236338+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"335'7223","reported_seq":7290,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-11T21:09:14.896579+0000","last_change":"2024-03-11T21:07:14.893219+0000","last_active":"2024-03-11T21:09:14.896579+0000","last_peered":"2024-03-11T21:09:14.896579+0000","last_clean":"2024-03-11T21:09:14.896579+0000","last_became_active":"2024-03-11T21:07:14.892853+0000","last_became_peered":"2024-03-11T21:07:14.892853+0000","last_unstale":"2024-03-11T21:09:14.896579+0000","last_undegraded":"2024-03-11T21:09:14.896579+0000","last_fullsized":"2024-03-11T21:09:14.896579+0000","mapping_epoch":306,"log_start":"316'1400","ondisk_log_start":"316'1400","created":306,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:07:13.846944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:07:13.846944+0000","last_clean_scrub_stamp":"2024-03-11T21:07:13.846944+0000","objects_scrubbed":0,"log_size":5823,"log_dups_size":0,"ondisk_log_size":5823,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:19.579447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":417267712,"num_objects":6367,"num_object_clones":0,"num_object_copies":19101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7223,"num_write_kb":434880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":589,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-03-11T21:09:15.333237+0000","last_change":"2024-03-11T21:07:51.860019+0000","last_active":"2024-03-11T21:09:15.333237+0000","last_peered":"2024-03-11T21:09:15.333237+0000","last_clean":"2024-03-11T21:09:15.333237+0000","last_became_active":"2024-03-11T21:00:27.962668+0000","last_became_peered":"2024-03-11T21:00:27.962668+0000","last_unstale":"2024-03-11T21:09:15.333237+0000","last_undegraded":"2024-03-11T21:09:15.333237+0000","last_fullsized":"2024-03-11T21:09:15.333237+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:07:51.859887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:07:51.859887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:18.847656+0000","scrub_duration":95,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"284'25","reported_seq":622,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-11T21:09:13.517402+0000","last_change":"2024-03-11T21:08:11.417836+0000","last_active":"2024-03-11T21:09:13.517402+0000","last_peered":"2024-03-11T21:09:13.517402+0000","last_clean":"2024-03-11T21:09:13.517402+0000","last_became_active":"2024-03-11T20:50:30.101747+0000","last_became_peered":"2024-03-11T20:50:30.101747+0000","last_unstale":"2024-03-11T21:09:13.517402+0000","last_undegraded":"2024-03-11T21:09:13.517402+0000","last_fullsized":"2024-03-11T21:09:13.517402+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"284'25","last_scrub_stamp":"2024-03-11T21:08:11.417688+0000","last_deep_scrub":"284'25","last_deep_scrub_stamp":"2024-03-11T21:04:53.217188+0000","last_clean_scrub_stamp":"2024-03-11T21:08:11.417688+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:21.214862+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":609,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-03-11T21:09:15.330730+0000","last_change":"2024-03-11T21:08:09.965355+0000","last_active":"2024-03-11T21:09:15.330730+0000","last_peered":"2024-03-11T21:09:15.330730+0000","last_clean":"2024-03-11T21:09:15.330730+0000","last_became_active":"2024-03-11T20:50:28.085503+0000","last_became_peered":"2024-03-11T20:50:28.085503+0000","last_unstale":"2024-03-11T21:09:15.330730+0000","last_undegraded":"2024-03-11T21:09:15.330730+0000","last_fullsized":"2024-03-11T21:09:15.330730+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:08:09.965148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:56:03.479180+0000","last_clean_scrub_stamp":"2024-03-11T21:08:09.965148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:36.540699+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"184'130","reported_seq":873,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-11T21:09:13.521491+0000","last_change":"2024-03-11T21:07:53.188030+0000","last_active":"2024-03-11T21:09:13.521491+0000","last_peered":"2024-03-11T21:09:13.521491+0000","last_clean":"2024-03-11T21:09:13.521491+0000","last_became_active":"2024-03-11T20:49:52.584543+0000","last_became_peered":"2024-03-11T20:49:52.584543+0000","last_unstale":"2024-03-11T21:09:13.521491+0000","last_undegraded":"2024-03-11T21:09:13.521491+0000","last_fullsized":"2024-03-11T21:09:13.521491+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"184'130","last_scrub_stamp":"2024-03-11T21:07:53.187858+0000","last_deep_scrub":"184'130","last_deep_scrub_stamp":"2024-03-11T20:58:56.415267+0000","last_clean_scrub_stamp":"2024-03-11T21:07:53.187858+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:19.647152+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":613,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-11T21:09:13.509875+0000","last_change":"2024-03-11T21:08:04.509143+0000","last_active":"2024-03-11T21:09:13.509875+0000","last_peered":"2024-03-11T21:09:13.509875+0000","last_clean":"2024-03-11T21:09:13.509875+0000","last_became_active":"2024-03-11T21:00:27.959467+0000","last_became_peered":"2024-03-11T21:00:27.959467+0000","last_unstale":"2024-03-11T21:09:13.509875+0000","last_undegraded":"2024-03-11T21:09:13.509875+0000","last_fullsized":"2024-03-11T21:09:13.509875+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:08:04.509037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:54:22.448812+0000","last_clean_scrub_stamp":"2024-03-11T21:08:04.509037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:33.092321+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.8","version":"335'7284","reported_seq":7352,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-11T21:09:15.128493+0000","last_change":"2024-03-11T21:07:14.892247+0000","last_active":"2024-03-11T21:09:15.128493+0000","last_peered":"2024-03-11T21:09:15.128493+0000","last_clean":"2024-03-11T21:09:15.128493+0000","last_became_active":"2024-03-11T21:07:14.891196+0000","last_became_peered":"2024-03-11T21:07:14.891196+0000","last_unstale":"2024-03-11T21:09:15.128493+0000","last_undegraded":"2024-03-11T21:09:15.128493+0000","last_fullsized":"2024-03-11T21:09:15.128493+0000","mapping_epoch":306,"log_start":"316'1900","ondisk_log_start":"316'1900","created":306,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:07:13.846944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:07:13.846944+0000","last_clean_scrub_stamp":"2024-03-11T21:07:13.846944+0000","objects_scrubbed":0,"log_size":5384,"log_dups_size":0,"ondisk_log_size":5384,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:19.047316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":415498240,"num_objects":6340,"num_object_clones":0,"num_object_copies":19020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6340,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7284,"num_write_kb":435968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":309,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:09:16.569982+0000","last_change":"2024-03-11T21:08:29.504076+0000","last_active":"2024-03-11T21:09:16.569982+0000","last_peered":"2024-03-11T21:09:16.569982+0000","last_clean":"2024-03-11T21:09:16.569982+0000","last_became_active":"2024-03-11T21:00:48.464596+0000","last_became_peered":"2024-03-11T21:00:48.464596+0000","last_unstale":"2024-03-11T21:09:16.569982+0000","last_undegraded":"2024-03-11T21:09:16.569982+0000","last_fullsized":"2024-03-11T21:09:16.569982+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:08:29.503928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:08:29.503928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:35.828734+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"207'35","reported_seq":285,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-11T21:09:13.521383+0000","last_change":"2024-03-11T21:08:24.324884+0000","last_active":"2024-03-11T21:09:13.521383+0000","last_peered":"2024-03-11T21:09:13.521383+0000","last_clean":"2024-03-11T21:09:13.521383+0000","last_became_active":"2024-03-11T21:06:28.761186+0000","last_became_peered":"2024-03-11T21:06:28.761186+0000","last_unstale":"2024-03-11T21:09:13.521383+0000","last_undegraded":"2024-03-11T21:09:13.521383+0000","last_fullsized":"2024-03-11T21:09:13.521383+0000","mapping_epoch":297,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'35","last_scrub_stamp":"2024-03-11T21:08:24.324628+0000","last_deep_scrub":"100'21","last_deep_scrub_stamp":"2024-03-11T20:51:31.835302+0000","last_clean_scrub_stamp":"2024-03-11T21:08:24.324628+0000","objects_scrubbed":7,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:25.322331+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":339,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:09:16.569421+0000","last_change":"2024-03-11T21:08:05.301922+0000","last_active":"2024-03-11T21:09:16.569421+0000","last_peered":"2024-03-11T21:09:16.569421+0000","last_clean":"2024-03-11T21:09:16.569421+0000","last_became_active":"2024-03-11T21:08:05.301024+0000","last_became_peered":"2024-03-11T21:08:05.301024+0000","last_unstale":"2024-03-11T21:09:16.569421+0000","last_undegraded":"2024-03-11T21:09:16.569421+0000","last_fullsized":"2024-03-11T21:09:16.569421+0000","mapping_epoch":319,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:07:54.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:07:54.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:20.236579+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":616,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-11T21:09:13.523670+0000","last_change":"2024-03-11T21:08:16.214623+0000","last_active":"2024-03-11T21:09:13.523670+0000","last_peered":"2024-03-11T21:09:13.523670+0000","last_clean":"2024-03-11T21:09:13.523670+0000","last_became_active":"2024-03-11T21:00:48.463247+0000","last_became_peered":"2024-03-11T21:00:48.463247+0000","last_unstale":"2024-03-11T21:09:13.523670+0000","last_undegraded":"2024-03-11T21:09:13.523670+0000","last_fullsized":"2024-03-11T21:09:13.523670+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:08:16.214363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:26.839782+0000","last_clean_scrub_stamp":"2024-03-11T21:08:16.214363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:35.817015+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.d","version":"335'7286","reported_seq":7352,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-11T21:09:14.894232+0000","last_change":"2024-03-11T21:07:14.951803+0000","last_active":"2024-03-11T21:09:14.894232+0000","last_peered":"2024-03-11T21:09:14.894232+0000","last_clean":"2024-03-11T21:09:14.894232+0000","last_became_active":"2024-03-11T21:07:14.951361+0000","last_became_peered":"2024-03-11T21:07:14.951361+0000","last_unstale":"2024-03-11T21:09:14.894232+0000","last_undegraded":"2024-03-11T21:09:14.894232+0000","last_fullsized":"2024-03-11T21:09:14.894232+0000","mapping_epoch":306,"log_start":"316'1500","ondisk_log_start":"316'1500","created":306,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:07:13.846944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:07:13.846944+0000","last_clean_scrub_stamp":"2024-03-11T21:07:13.846944+0000","objects_scrubbed":0,"log_size":5786,"log_dups_size":0,"ondisk_log_size":5786,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:19.579430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":416612376,"num_objects":6358,"num_object_clones":0,"num_object_copies":19074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6358,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7286,"num_write_kb":436545,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":309,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:09:16.569083+0000","last_change":"2024-03-11T21:07:45.417128+0000","last_active":"2024-03-11T21:09:16.569083+0000","last_peered":"2024-03-11T21:09:16.569083+0000","last_clean":"2024-03-11T21:09:16.569083+0000","last_became_active":"2024-03-11T21:00:48.469688+0000","last_became_peered":"2024-03-11T21:00:48.469688+0000","last_unstale":"2024-03-11T21:09:16.569083+0000","last_undegraded":"2024-03-11T21:09:16.569083+0000","last_fullsized":"2024-03-11T21:09:16.569083+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:07:45.416778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:07:45.416778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:20.236482+0000","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"145'3","reported_seq":607,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-11T21:09:13.509337+0000","last_change":"2024-03-11T21:07:45.552642+0000","last_active":"2024-03-11T21:09:13.509337+0000","last_peered":"2024-03-11T21:09:13.509337+0000","last_clean":"2024-03-11T21:09:13.509337+0000","last_became_active":"2024-03-11T20:50:32.195452+0000","last_became_peered":"2024-03-11T20:50:32.195452+0000","last_unstale":"2024-03-11T21:09:13.509337+0000","last_undegraded":"2024-03-11T21:09:13.509337+0000","last_fullsized":"2024-03-11T21:09:13.509337+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'3","last_scrub_stamp":"2024-03-11T21:07:45.552526+0000","last_deep_scrub":"145'3","last_deep_scrub_stamp":"2024-03-11T20:55:52.848043+0000","last_clean_scrub_stamp":"2024-03-11T21:07:45.552526+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:19.579603+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0.0024049620000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":605,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-11T21:09:13.522674+0000","last_change":"2024-03-11T21:07:59.257140+0000","last_active":"2024-03-11T21:09:13.522674+0000","last_peered":"2024-03-11T21:09:13.522674+0000","last_clean":"2024-03-11T21:09:13.522674+0000","last_became_active":"2024-03-11T21:00:27.966749+0000","last_became_peered":"2024-03-11T21:00:27.966749+0000","last_unstale":"2024-03-11T21:09:13.522674+0000","last_undegraded":"2024-03-11T21:09:13.522674+0000","last_fullsized":"2024-03-11T21:09:13.522674+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:07:59.256994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:07:59.256994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:16.309231+0000","scrub_duration":163,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"212'46","reported_seq":440,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:09:16.570085+0000","last_change":"2024-03-11T21:07:47.448694+0000","last_active":"2024-03-11T21:09:16.570085+0000","last_peered":"2024-03-11T21:09:16.570085+0000","last_clean":"2024-03-11T21:09:16.570085+0000","last_became_active":"2024-03-11T21:00:48.473227+0000","last_became_peered":"2024-03-11T21:00:48.473227+0000","last_unstale":"2024-03-11T21:09:16.570085+0000","last_undegraded":"2024-03-11T21:09:16.570085+0000","last_fullsized":"2024-03-11T21:09:16.570085+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"212'46","last_scrub_stamp":"2024-03-11T21:07:47.448528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:07:47.448528+0000","objects_scrubbed":10,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:20.236413+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":69,"num_read_kb":54,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"298'13","reported_seq":409,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-11T21:09:13.522647+0000","last_change":"2024-03-11T21:08:08.181167+0000","last_active":"2024-03-11T21:09:13.522647+0000","last_peered":"2024-03-11T21:09:13.522647+0000","last_clean":"2024-03-11T21:09:13.522647+0000","last_became_active":"2024-03-11T21:06:28.762397+0000","last_became_peered":"2024-03-11T21:06:28.762397+0000","last_unstale":"2024-03-11T21:09:13.522647+0000","last_undegraded":"2024-03-11T21:09:13.522647+0000","last_fullsized":"2024-03-11T21:09:13.522647+0000","mapping_epoch":297,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"298'13","last_scrub_stamp":"2024-03-11T21:08:08.181013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:08:08.181013+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:25.674580+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.e","version":"335'7221","reported_seq":7289,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-11T21:09:14.823046+0000","last_change":"2024-03-11T21:07:14.954025+0000","last_active":"2024-03-11T21:09:14.823046+0000","last_peered":"2024-03-11T21:09:14.823046+0000","last_clean":"2024-03-11T21:09:14.823046+0000","last_became_active":"2024-03-11T21:07:14.953126+0000","last_became_peered":"2024-03-11T21:07:14.953126+0000","last_unstale":"2024-03-11T21:09:14.823046+0000","last_undegraded":"2024-03-11T21:09:14.823046+0000","last_fullsized":"2024-03-11T21:09:14.823046+0000","mapping_epoch":306,"log_start":"316'1400","ondisk_log_start":"316'1400","created":306,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:07:13.846944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:07:13.846944+0000","last_clean_scrub_stamp":"2024-03-11T21:07:13.846944+0000","objects_scrubbed":0,"log_size":5821,"log_dups_size":0,"ondisk_log_size":5821,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:19.579527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":420151296,"num_objects":6411,"num_object_clones":0,"num_object_copies":19233,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6411,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7221,"num_write_kb":436224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":324,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-11T21:09:13.516476+0000","last_change":"2024-03-11T21:06:31.535147+0000","last_active":"2024-03-11T21:09:13.516476+0000","last_peered":"2024-03-11T21:09:13.516476+0000","last_clean":"2024-03-11T21:09:13.516476+0000","last_became_active":"2024-03-11T21:05:17.441521+0000","last_became_peered":"2024-03-11T21:05:17.441521+0000","last_unstale":"2024-03-11T21:09:13.516476+0000","last_undegraded":"2024-03-11T21:09:13.516476+0000","last_fullsized":"2024-03-11T21:09:13.516476+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:06:31.535021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:03:43.345257+0000","last_clean_scrub_stamp":"2024-03-11T21:06:31.535021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":345,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-11T21:09:13.516995+0000","last_change":"2024-03-11T21:06:44.388965+0000","last_active":"2024-03-11T21:09:13.516995+0000","last_peered":"2024-03-11T21:09:13.516995+0000","last_clean":"2024-03-11T21:09:13.516995+0000","last_became_active":"2024-03-11T21:05:17.315844+0000","last_became_peered":"2024-03-11T21:05:17.315844+0000","last_unstale":"2024-03-11T21:09:13.516995+0000","last_undegraded":"2024-03-11T21:09:13.516995+0000","last_fullsized":"2024-03-11T21:09:13.516995+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:06:44.388832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:22.617999+0000","last_clean_scrub_stamp":"2024-03-11T21:06:44.388832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"284'2","reported_seq":313,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:09:16.570612+0000","last_change":"2024-03-11T21:08:22.484946+0000","last_active":"2024-03-11T21:09:16.570612+0000","last_peered":"2024-03-11T21:09:16.570612+0000","last_clean":"2024-03-11T21:09:16.570612+0000","last_became_active":"2024-03-11T21:00:48.468682+0000","last_became_peered":"2024-03-11T21:00:48.468682+0000","last_unstale":"2024-03-11T21:09:16.570612+0000","last_undegraded":"2024-03-11T21:09:16.570612+0000","last_fullsized":"2024-03-11T21:09:16.570612+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"284'2","last_scrub_stamp":"2024-03-11T21:08:22.484799+0000","last_deep_scrub":"284'2","last_deep_scrub_stamp":"2024-03-11T21:06:47.645682+0000","last_clean_scrub_stamp":"2024-03-11T21:08:22.484799+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:33.894145+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"207'17","reported_seq":611,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-11T21:09:13.522337+0000","last_change":"2024-03-11T21:06:41.476100+0000","last_active":"2024-03-11T21:09:13.522337+0000","last_peered":"2024-03-11T21:09:13.522337+0000","last_clean":"2024-03-11T21:09:13.522337+0000","last_became_active":"2024-03-11T20:50:30.097669+0000","last_became_peered":"2024-03-11T20:50:30.097669+0000","last_unstale":"2024-03-11T21:09:13.522337+0000","last_undegraded":"2024-03-11T21:09:13.522337+0000","last_fullsized":"2024-03-11T21:09:13.522337+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'17","last_scrub_stamp":"2024-03-11T21:06:41.475870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:06:41.475870+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"84'11","reported_seq":827,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-11T21:09:13.516056+0000","last_change":"2024-03-11T21:08:04.428744+0000","last_active":"2024-03-11T21:09:13.516056+0000","last_peered":"2024-03-11T21:09:13.516056+0000","last_clean":"2024-03-11T21:09:13.516056+0000","last_became_active":"2024-03-11T20:50:32.647041+0000","last_became_peered":"2024-03-11T20:50:32.647041+0000","last_unstale":"2024-03-11T21:09:13.516056+0000","last_undegraded":"2024-03-11T21:09:13.516056+0000","last_fullsized":"2024-03-11T21:09:13.516056+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:08:04.428606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:08:04.428606+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:24.794391+0000","scrub_duration":83,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.f","version":"335'7405","reported_seq":7528,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-11T21:09:14.888886+0000","last_change":"2024-03-11T21:08:27.083555+0000","last_active":"2024-03-11T21:09:14.888886+0000","last_peered":"2024-03-11T21:09:14.888886+0000","last_clean":"2024-03-11T21:09:14.888886+0000","last_became_active":"2024-03-11T21:07:14.959566+0000","last_became_peered":"2024-03-11T21:07:14.959566+0000","last_unstale":"2024-03-11T21:09:14.888886+0000","last_undegraded":"2024-03-11T21:09:14.888886+0000","last_fullsized":"2024-03-11T21:09:14.888886+0000","mapping_epoch":306,"log_start":"316'1600","ondisk_log_start":"316'1600","created":306,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'4622","last_scrub_stamp":"2024-03-11T21:08:27.083422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:07:13.846944+0000","last_clean_scrub_stamp":"2024-03-11T21:08:27.083422+0000","objects_scrubbed":4556,"log_size":5805,"log_dups_size":0,"ondisk_log_size":5805,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:48.019551+0000","scrub_duration":1783,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":422641664,"num_objects":6449,"num_object_clones":0,"num_object_copies":19347,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6449,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7405,"num_write_kb":443328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":600,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-11T21:09:13.522141+0000","last_change":"2024-03-11T21:08:07.202258+0000","last_active":"2024-03-11T21:09:13.522141+0000","last_peered":"2024-03-11T21:09:13.522141+0000","last_clean":"2024-03-11T21:09:13.522141+0000","last_became_active":"2024-03-11T20:50:34.326947+0000","last_became_peered":"2024-03-11T20:50:34.326947+0000","last_unstale":"2024-03-11T21:09:13.522141+0000","last_undegraded":"2024-03-11T21:09:13.522141+0000","last_fullsized":"2024-03-11T21:09:13.522141+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:08:07.202131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:06:43.271453+0000","last_clean_scrub_stamp":"2024-03-11T21:08:07.202131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:22.106352+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"120'6","reported_seq":1621,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-11T21:09:13.781704+0000","last_change":"2024-03-11T21:08:19.340719+0000","last_active":"2024-03-11T21:09:13.781704+0000","last_peered":"2024-03-11T21:09:13.781704+0000","last_clean":"2024-03-11T21:09:13.781704+0000","last_became_active":"2024-03-11T21:00:48.468997+0000","last_became_peered":"2024-03-11T21:00:48.468997+0000","last_unstale":"2024-03-11T21:09:13.781704+0000","last_undegraded":"2024-03-11T21:09:13.781704+0000","last_fullsized":"2024-03-11T21:09:13.781704+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"120'6","last_scrub_stamp":"2024-03-11T21:08:19.340643+0000","last_deep_scrub":"120'6","last_deep_scrub_stamp":"2024-03-11T20:54:26.468057+0000","last_clean_scrub_stamp":"2024-03-11T21:08:19.340643+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:34.137288+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0035533370000000002,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":989,"num_read_kb":984,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":612,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-11T21:09:13.510124+0000","last_change":"2024-03-11T21:07:40.565165+0000","last_active":"2024-03-11T21:09:13.510124+0000","last_peered":"2024-03-11T21:09:13.510124+0000","last_clean":"2024-03-11T21:09:13.510124+0000","last_became_active":"2024-03-11T20:50:28.071707+0000","last_became_peered":"2024-03-11T20:50:28.071707+0000","last_unstale":"2024-03-11T21:09:13.510124+0000","last_undegraded":"2024-03-11T21:09:13.510124+0000","last_fullsized":"2024-03-11T21:09:13.510124+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:07:40.565077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:23.417283+0000","last_clean_scrub_stamp":"2024-03-11T21:07:40.565077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"207'24","reported_seq":704,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-11T21:09:13.523218+0000","last_change":"2024-03-11T21:07:52.838119+0000","last_active":"2024-03-11T21:09:13.523218+0000","last_peered":"2024-03-11T21:09:13.523218+0000","last_clean":"2024-03-11T21:09:13.523218+0000","last_became_active":"2024-03-11T21:00:48.461972+0000","last_became_peered":"2024-03-11T21:00:48.461972+0000","last_unstale":"2024-03-11T21:09:13.523218+0000","last_undegraded":"2024-03-11T21:09:13.523218+0000","last_fullsized":"2024-03-11T21:09:13.523218+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'24","last_scrub_stamp":"2024-03-11T21:07:52.837968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:07:52.837968+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:17.874543+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":376,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-03-11T21:09:15.330383+0000","last_change":"2024-03-11T21:08:15.915483+0000","last_active":"2024-03-11T21:09:15.330383+0000","last_peered":"2024-03-11T21:09:15.330383+0000","last_clean":"2024-03-11T21:09:15.330383+0000","last_became_active":"2024-03-11T21:05:17.315807+0000","last_became_peered":"2024-03-11T21:05:17.315807+0000","last_unstale":"2024-03-11T21:09:15.330383+0000","last_undegraded":"2024-03-11T21:09:15.330383+0000","last_fullsized":"2024-03-11T21:09:15.330383+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:08:15.915374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:08:15.915374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:35.617260+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.0","version":"335'7250","reported_seq":7317,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-11T21:09:14.894264+0000","last_change":"2024-03-11T21:07:14.953886+0000","last_active":"2024-03-11T21:09:14.894264+0000","last_peered":"2024-03-11T21:09:14.894264+0000","last_clean":"2024-03-11T21:09:14.894264+0000","last_became_active":"2024-03-11T21:07:14.952617+0000","last_became_peered":"2024-03-11T21:07:14.952617+0000","last_unstale":"2024-03-11T21:09:14.894264+0000","last_undegraded":"2024-03-11T21:09:14.894264+0000","last_fullsized":"2024-03-11T21:09:14.894264+0000","mapping_epoch":306,"log_start":"316'1400","ondisk_log_start":"316'1400","created":306,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:07:13.846944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:07:13.846944+0000","last_clean_scrub_stamp":"2024-03-11T21:07:13.846944+0000","objects_scrubbed":0,"log_size":5850,"log_dups_size":0,"ondisk_log_size":5850,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:19.579464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":419692544,"num_objects":6404,"num_object_clones":0,"num_object_copies":19212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6404,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7250,"num_write_kb":436928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":570,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-11T21:09:13.518698+0000","last_change":"2024-03-11T21:06:30.558401+0000","last_active":"2024-03-11T21:09:13.518698+0000","last_peered":"2024-03-11T21:09:13.518698+0000","last_clean":"2024-03-11T21:09:13.518698+0000","last_became_active":"2024-03-11T20:50:34.335370+0000","last_became_peered":"2024-03-11T20:50:34.335370+0000","last_unstale":"2024-03-11T21:09:13.518698+0000","last_undegraded":"2024-03-11T21:09:13.518698+0000","last_fullsized":"2024-03-11T21:09:13.518698+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:06:30.558258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:06:30.558258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"286'18","reported_seq":239,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-11T21:09:13.510732+0000","last_change":"2024-03-11T21:07:55.699039+0000","last_active":"2024-03-11T21:09:13.510732+0000","last_peered":"2024-03-11T21:09:13.510732+0000","last_clean":"2024-03-11T21:09:13.510732+0000","last_became_active":"2024-03-11T21:05:17.449213+0000","last_became_peered":"2024-03-11T21:05:17.449213+0000","last_unstale":"2024-03-11T21:09:13.510732+0000","last_undegraded":"2024-03-11T21:09:13.510732+0000","last_fullsized":"2024-03-11T21:09:13.510732+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"286'18","last_scrub_stamp":"2024-03-11T21:07:55.698212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:07:55.698212+0000","objects_scrubbed":1,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":195,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":617,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-11T21:09:13.521845+0000","last_change":"2024-03-11T21:08:16.137000+0000","last_active":"2024-03-11T21:09:13.521845+0000","last_peered":"2024-03-11T21:09:13.521845+0000","last_clean":"2024-03-11T21:09:13.521845+0000","last_became_active":"2024-03-11T21:00:48.466102+0000","last_became_peered":"2024-03-11T21:00:48.466102+0000","last_unstale":"2024-03-11T21:09:13.521845+0000","last_undegraded":"2024-03-11T21:09:13.521845+0000","last_fullsized":"2024-03-11T21:09:13.521845+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:08:16.136851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:06:42.242801+0000","last_clean_scrub_stamp":"2024-03-11T21:08:16.136851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:37.876869+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"207'18","reported_seq":300,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-11T21:09:13.522583+0000","last_change":"2024-03-11T21:08:05.299437+0000","last_active":"2024-03-11T21:09:13.522583+0000","last_peered":"2024-03-11T21:09:13.522583+0000","last_clean":"2024-03-11T21:09:13.522583+0000","last_became_active":"2024-03-11T21:08:05.298170+0000","last_became_peered":"2024-03-11T21:08:05.298170+0000","last_unstale":"2024-03-11T21:09:13.522583+0000","last_undegraded":"2024-03-11T21:09:13.522583+0000","last_fullsized":"2024-03-11T21:09:13.522583+0000","mapping_epoch":319,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'18","last_scrub_stamp":"2024-03-11T21:07:49.533424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:07:49.533424+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:14.240722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":23,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1","version":"335'7257","reported_seq":7375,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-11T21:09:14.843463+0000","last_change":"2024-03-11T21:08:23.498439+0000","last_active":"2024-03-11T21:09:14.843463+0000","last_peered":"2024-03-11T21:09:14.843463+0000","last_clean":"2024-03-11T21:09:14.843463+0000","last_became_active":"2024-03-11T21:07:14.951704+0000","last_became_peered":"2024-03-11T21:07:14.951704+0000","last_unstale":"2024-03-11T21:09:14.843463+0000","last_undegraded":"2024-03-11T21:09:14.843463+0000","last_fullsized":"2024-03-11T21:09:14.843463+0000","mapping_epoch":306,"log_start":"316'2300","ondisk_log_start":"316'2300","created":306,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'4267","last_scrub_stamp":"2024-03-11T21:08:23.497931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:07:13.846944+0000","last_clean_scrub_stamp":"2024-03-11T21:08:23.497931+0000","objects_scrubbed":4222,"log_size":4957,"log_dups_size":0,"ondisk_log_size":4957,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:39.426552+0000","scrub_duration":1383,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":418447360,"num_objects":6385,"num_object_clones":0,"num_object_copies":19155,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6385,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7257,"num_write_kb":436544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":609,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-03-11T21:09:15.331729+0000","last_change":"2024-03-11T21:08:22.890801+0000","last_active":"2024-03-11T21:09:15.331729+0000","last_peered":"2024-03-11T21:09:15.331729+0000","last_clean":"2024-03-11T21:09:15.331729+0000","last_became_active":"2024-03-11T21:00:48.460531+0000","last_became_peered":"2024-03-11T21:00:48.460531+0000","last_unstale":"2024-03-11T21:09:15.331729+0000","last_undegraded":"2024-03-11T21:09:15.331729+0000","last_fullsized":"2024-03-11T21:09:15.331729+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:08:22.890690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:27.775534+0000","last_clean_scrub_stamp":"2024-03-11T21:08:22.890690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:42.927325+0000","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":316,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:09:16.570798+0000","last_change":"2024-03-11T21:08:26.382408+0000","last_active":"2024-03-11T21:09:16.570798+0000","last_peered":"2024-03-11T21:09:16.570798+0000","last_clean":"2024-03-11T21:09:16.570798+0000","last_became_active":"2024-03-11T21:00:48.469596+0000","last_became_peered":"2024-03-11T21:00:48.469596+0000","last_unstale":"2024-03-11T21:09:16.570798+0000","last_undegraded":"2024-03-11T21:09:16.570798+0000","last_fullsized":"2024-03-11T21:09:16.570798+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:08:26.382277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:08:26.382277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:31.988971+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":300,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:09:16.569225+0000","last_change":"2024-03-11T21:07:41.398091+0000","last_active":"2024-03-11T21:09:16.569225+0000","last_peered":"2024-03-11T21:09:16.569225+0000","last_clean":"2024-03-11T21:09:16.569225+0000","last_became_active":"2024-03-11T21:00:48.466084+0000","last_became_peered":"2024-03-11T21:00:48.466084+0000","last_unstale":"2024-03-11T21:09:16.569225+0000","last_undegraded":"2024-03-11T21:09:16.569225+0000","last_fullsized":"2024-03-11T21:09:16.569225+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:07:41.397960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:40.974147+0000","last_clean_scrub_stamp":"2024-03-11T21:07:41.397960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:20.236382+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"207'34","reported_seq":688,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-11T21:09:13.515980+0000","last_change":"2024-03-11T21:06:11.347952+0000","last_active":"2024-03-11T21:09:13.515980+0000","last_peered":"2024-03-11T21:09:13.515980+0000","last_clean":"2024-03-11T21:09:13.515980+0000","last_became_active":"2024-03-11T21:00:48.472191+0000","last_became_peered":"2024-03-11T21:00:48.472191+0000","last_unstale":"2024-03-11T21:09:13.515980+0000","last_undegraded":"2024-03-11T21:09:13.515980+0000","last_fullsized":"2024-03-11T21:09:13.515980+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'34","last_scrub_stamp":"2024-03-11T21:06:11.347860+0000","last_deep_scrub":"116'22","last_deep_scrub_stamp":"2024-03-11T20:53:47.364709+0000","last_clean_scrub_stamp":"2024-03-11T21:06:11.347860+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"337'7382","reported_seq":7454,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:09:17.227896+0000","last_change":"2024-03-11T21:07:14.952811+0000","last_active":"2024-03-11T21:09:17.227896+0000","last_peered":"2024-03-11T21:09:17.227896+0000","last_clean":"2024-03-11T21:09:17.227896+0000","last_became_active":"2024-03-11T21:07:14.951968+0000","last_became_peered":"2024-03-11T21:07:14.951968+0000","last_unstale":"2024-03-11T21:09:17.227896+0000","last_undegraded":"2024-03-11T21:09:17.227896+0000","last_fullsized":"2024-03-11T21:09:17.227896+0000","mapping_epoch":306,"log_start":"316'1700","ondisk_log_start":"316'1700","created":306,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:07:13.846944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:07:13.846944+0000","last_clean_scrub_stamp":"2024-03-11T21:07:13.846944+0000","objects_scrubbed":0,"log_size":5682,"log_dups_size":0,"ondisk_log_size":5682,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:20.236285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":411172864,"num_objects":6274,"num_object_clones":0,"num_object_copies":18822,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6274,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7382,"num_write_kb":436992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":387,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:09:16.570537+0000","last_change":"2024-03-11T21:08:11.531203+0000","last_active":"2024-03-11T21:09:16.570537+0000","last_peered":"2024-03-11T21:09:16.570537+0000","last_clean":"2024-03-11T21:09:16.570537+0000","last_became_active":"2024-03-11T21:05:17.455236+0000","last_became_peered":"2024-03-11T21:05:17.455236+0000","last_unstale":"2024-03-11T21:09:16.570537+0000","last_undegraded":"2024-03-11T21:09:16.570537+0000","last_fullsized":"2024-03-11T21:09:16.570537+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:08:11.530874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:08:11.530874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:27.560603+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":308,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:09:16.570415+0000","last_change":"2024-03-11T21:07:58.612531+0000","last_active":"2024-03-11T21:09:16.570415+0000","last_peered":"2024-03-11T21:09:16.570415+0000","last_clean":"2024-03-11T21:09:16.570415+0000","last_became_active":"2024-03-11T21:00:48.461367+0000","last_became_peered":"2024-03-11T21:00:48.461367+0000","last_unstale":"2024-03-11T21:09:16.570415+0000","last_undegraded":"2024-03-11T21:09:16.570415+0000","last_fullsized":"2024-03-11T21:09:16.570415+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:07:58.612421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:03:37.601229+0000","last_clean_scrub_stamp":"2024-03-11T21:07:58.612421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:24.827499+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"267'19","reported_seq":624,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-11T21:09:13.516880+0000","last_change":"2024-03-11T21:08:28.368669+0000","last_active":"2024-03-11T21:09:13.516880+0000","last_peered":"2024-03-11T21:09:13.516880+0000","last_clean":"2024-03-11T21:09:13.516880+0000","last_became_active":"2024-03-11T21:00:27.967572+0000","last_became_peered":"2024-03-11T21:00:27.967572+0000","last_unstale":"2024-03-11T21:09:13.516880+0000","last_undegraded":"2024-03-11T21:09:13.516880+0000","last_fullsized":"2024-03-11T21:09:13.516880+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"267'19","last_scrub_stamp":"2024-03-11T21:08:28.368557+0000","last_deep_scrub":"267'19","last_deep_scrub_stamp":"2024-03-11T21:03:39.673317+0000","last_clean_scrub_stamp":"2024-03-11T21:08:28.368557+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:28.486379+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":376,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-11T21:09:13.518089+0000","last_change":"2024-03-11T21:08:03.331915+0000","last_active":"2024-03-11T21:09:13.518089+0000","last_peered":"2024-03-11T21:09:13.518089+0000","last_clean":"2024-03-11T21:09:13.518089+0000","last_became_active":"2024-03-11T21:05:17.316136+0000","last_became_peered":"2024-03-11T21:05:17.316136+0000","last_unstale":"2024-03-11T21:09:13.518089+0000","last_undegraded":"2024-03-11T21:09:13.518089+0000","last_fullsized":"2024-03-11T21:09:13.518089+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:08:03.331832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:35.670149+0000","last_clean_scrub_stamp":"2024-03-11T21:08:03.331832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:16.310959+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"336'7205","reported_seq":7281,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-03-11T21:09:15.472199+0000","last_change":"2024-03-11T21:08:05.305057+0000","last_active":"2024-03-11T21:09:15.472199+0000","last_peered":"2024-03-11T21:09:15.472199+0000","last_clean":"2024-03-11T21:09:15.472199+0000","last_became_active":"2024-03-11T21:08:05.298758+0000","last_became_peered":"2024-03-11T21:08:05.298758+0000","last_unstale":"2024-03-11T21:09:15.472199+0000","last_undegraded":"2024-03-11T21:09:15.472199+0000","last_fullsized":"2024-03-11T21:09:15.472199+0000","mapping_epoch":319,"log_start":"316'1300","ondisk_log_start":"316'1300","created":306,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:07:13.846944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:07:13.846944+0000","last_clean_scrub_stamp":"2024-03-11T21:07:13.846944+0000","objects_scrubbed":0,"log_size":5905,"log_dups_size":0,"ondisk_log_size":5905,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":410583040,"num_objects":6265,"num_object_clones":0,"num_object_copies":18795,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6265,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7205,"num_write_kb":431040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":347,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-11T21:09:13.522163+0000","last_change":"2024-03-11T21:08:05.305618+0000","last_active":"2024-03-11T21:09:13.522163+0000","last_peered":"2024-03-11T21:09:13.522163+0000","last_clean":"2024-03-11T21:09:13.522163+0000","last_became_active":"2024-03-11T21:08:05.304963+0000","last_became_peered":"2024-03-11T21:08:05.304963+0000","last_unstale":"2024-03-11T21:09:13.522163+0000","last_undegraded":"2024-03-11T21:09:13.522163+0000","last_fullsized":"2024-03-11T21:09:13.522163+0000","mapping_epoch":319,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:07:38.448238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:07:38.448238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"320'21","reported_seq":558,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-03-11T21:09:15.330510+0000","last_change":"2024-03-11T21:08:05.299421+0000","last_active":"2024-03-11T21:09:15.330510+0000","last_peered":"2024-03-11T21:09:15.330510+0000","last_clean":"2024-03-11T21:09:15.330510+0000","last_became_active":"2024-03-11T21:08:05.298887+0000","last_became_peered":"2024-03-11T21:08:05.298887+0000","last_unstale":"2024-03-11T21:09:15.330510+0000","last_undegraded":"2024-03-11T21:09:15.330510+0000","last_fullsized":"2024-03-11T21:09:15.330510+0000","mapping_epoch":319,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":0,"last_scrub":"286'20","last_scrub_stamp":"2024-03-11T21:08:02.983969+0000","last_deep_scrub":"146'14","last_deep_scrub_stamp":"2024-03-11T20:56:00.242211+0000","last_clean_scrub_stamp":"2024-03-11T21:08:02.983969+0000","objects_scrubbed":1,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:21.708236+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":605,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-11T21:09:13.522237+0000","last_change":"2024-03-11T21:07:52.165209+0000","last_active":"2024-03-11T21:09:13.522237+0000","last_peered":"2024-03-11T21:09:13.522237+0000","last_clean":"2024-03-11T21:09:13.522237+0000","last_became_active":"2024-03-11T21:00:27.970573+0000","last_became_peered":"2024-03-11T21:00:27.970573+0000","last_unstale":"2024-03-11T21:09:13.522237+0000","last_undegraded":"2024-03-11T21:09:13.522237+0000","last_fullsized":"2024-03-11T21:09:13.522237+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:07:52.165109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:52:51.161281+0000","last_clean_scrub_stamp":"2024-03-11T21:07:52.165109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:14.240915+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"207'44","reported_seq":749,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-11T21:09:13.509529+0000","last_change":"2024-03-11T21:08:03.490157+0000","last_active":"2024-03-11T21:09:13.509529+0000","last_peered":"2024-03-11T21:09:13.509529+0000","last_clean":"2024-03-11T21:09:13.509529+0000","last_became_active":"2024-03-11T21:00:48.469854+0000","last_became_peered":"2024-03-11T21:00:48.469854+0000","last_unstale":"2024-03-11T21:09:13.509529+0000","last_undegraded":"2024-03-11T21:09:13.509529+0000","last_fullsized":"2024-03-11T21:09:13.509529+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'44","last_scrub_stamp":"2024-03-11T21:08:03.489929+0000","last_deep_scrub":"161'28","last_deep_scrub_stamp":"2024-03-11T20:57:20.816278+0000","last_clean_scrub_stamp":"2024-03-11T21:08:03.489929+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:14.596807+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.4","version":"335'7191","reported_seq":7259,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-11T21:09:15.084903+0000","last_change":"2024-03-11T21:07:14.960911+0000","last_active":"2024-03-11T21:09:15.084903+0000","last_peered":"2024-03-11T21:09:15.084903+0000","last_clean":"2024-03-11T21:09:15.084903+0000","last_became_active":"2024-03-11T21:07:14.959964+0000","last_became_peered":"2024-03-11T21:07:14.959964+0000","last_unstale":"2024-03-11T21:09:15.084903+0000","last_undegraded":"2024-03-11T21:09:15.084903+0000","last_fullsized":"2024-03-11T21:09:15.084903+0000","mapping_epoch":306,"log_start":"316'1800","ondisk_log_start":"316'1800","created":306,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:07:13.846944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:07:13.846944+0000","last_clean_scrub_stamp":"2024-03-11T21:07:13.846944+0000","objects_scrubbed":0,"log_size":5391,"log_dups_size":0,"ondisk_log_size":5391,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:19.047341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":415956992,"num_objects":6347,"num_object_clones":0,"num_object_copies":19041,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6347,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7191,"num_write_kb":433216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":592,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-11T21:09:13.521263+0000","last_change":"2024-03-11T21:07:56.111952+0000","last_active":"2024-03-11T21:09:13.521263+0000","last_peered":"2024-03-11T21:09:13.521263+0000","last_clean":"2024-03-11T21:09:13.521263+0000","last_became_active":"2024-03-11T20:50:34.327139+0000","last_became_peered":"2024-03-11T20:50:34.327139+0000","last_unstale":"2024-03-11T21:09:13.521263+0000","last_undegraded":"2024-03-11T21:09:13.521263+0000","last_fullsized":"2024-03-11T21:09:13.521263+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:07:56.111591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:07:56.111591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:14.482857+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":45,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-03-11T21:09:15.330930+0000","last_change":"2024-03-11T21:08:05.303485+0000","last_active":"2024-03-11T21:09:15.330930+0000","last_peered":"2024-03-11T21:09:15.330930+0000","last_clean":"2024-03-11T21:09:15.330930+0000","last_became_active":"2024-03-11T21:08:05.303226+0000","last_became_peered":"2024-03-11T21:08:05.303226+0000","last_unstale":"2024-03-11T21:09:15.330930+0000","last_undegraded":"2024-03-11T21:09:15.330930+0000","last_fullsized":"2024-03-11T21:09:15.330930+0000","mapping_epoch":319,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:07:39.353184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:57:17.447191+0000","last_clean_scrub_stamp":"2024-03-11T21:07:39.353184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:18.847686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"207'39","reported_seq":657,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-11T21:09:13.522844+0000","last_change":"2024-03-11T21:07:41.182291+0000","last_active":"2024-03-11T21:09:13.522844+0000","last_peered":"2024-03-11T21:09:13.522844+0000","last_clean":"2024-03-11T21:09:13.522844+0000","last_became_active":"2024-03-11T20:50:30.111819+0000","last_became_peered":"2024-03-11T20:50:30.111819+0000","last_unstale":"2024-03-11T21:09:13.522844+0000","last_undegraded":"2024-03-11T21:09:13.522844+0000","last_fullsized":"2024-03-11T21:09:13.522844+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'39","last_scrub_stamp":"2024-03-11T21:07:41.182193+0000","last_deep_scrub":"102'25","last_deep_scrub_stamp":"2024-03-11T20:51:35.187094+0000","last_clean_scrub_stamp":"2024-03-11T21:07:41.182193+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:14.240895+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":617,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-11T21:09:13.509205+0000","last_change":"2024-03-11T21:08:05.495968+0000","last_active":"2024-03-11T21:09:13.509205+0000","last_peered":"2024-03-11T21:09:13.509205+0000","last_clean":"2024-03-11T21:09:13.509205+0000","last_became_active":"2024-03-11T21:00:48.362389+0000","last_became_peered":"2024-03-11T21:00:48.362389+0000","last_unstale":"2024-03-11T21:09:13.509205+0000","last_undegraded":"2024-03-11T21:09:13.509205+0000","last_fullsized":"2024-03-11T21:09:13.509205+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:08:05.495776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:08:05.495776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.5","version":"335'7363","reported_seq":7434,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-11T21:09:15.114820+0000","last_change":"2024-03-11T21:08:05.315370+0000","last_active":"2024-03-11T21:09:15.114820+0000","last_peered":"2024-03-11T21:09:15.114820+0000","last_clean":"2024-03-11T21:09:15.114820+0000","last_became_active":"2024-03-11T21:08:05.304912+0000","last_became_peered":"2024-03-11T21:08:05.304912+0000","last_unstale":"2024-03-11T21:09:15.114820+0000","last_undegraded":"2024-03-11T21:09:15.114820+0000","last_fullsized":"2024-03-11T21:09:15.114820+0000","mapping_epoch":319,"log_start":"316'2000","ondisk_log_start":"316'2000","created":306,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:07:13.846944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:07:13.846944+0000","last_clean_scrub_stamp":"2024-03-11T21:07:13.846944+0000","objects_scrubbed":0,"log_size":5363,"log_dups_size":0,"ondisk_log_size":5363,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:19.042562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":418840576,"num_objects":6391,"num_object_clones":0,"num_object_copies":19173,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7363,"num_write_kb":440128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":288,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-11T21:09:13.521772+0000","last_change":"2024-03-11T21:07:44.901299+0000","last_active":"2024-03-11T21:09:13.521772+0000","last_peered":"2024-03-11T21:09:13.521772+0000","last_clean":"2024-03-11T21:09:13.521772+0000","last_became_active":"2024-03-11T21:06:28.758451+0000","last_became_peered":"2024-03-11T21:06:28.758451+0000","last_unstale":"2024-03-11T21:09:13.521772+0000","last_undegraded":"2024-03-11T21:09:13.521772+0000","last_fullsized":"2024-03-11T21:09:13.521772+0000","mapping_epoch":297,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:07:44.901223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:55:58.888890+0000","last_clean_scrub_stamp":"2024-03-11T21:07:44.901223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":300,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:09:16.570255+0000","last_change":"2024-03-11T21:07:38.356703+0000","last_active":"2024-03-11T21:09:16.570255+0000","last_peered":"2024-03-11T21:09:16.570255+0000","last_clean":"2024-03-11T21:09:16.570255+0000","last_became_active":"2024-03-11T21:00:48.469015+0000","last_became_peered":"2024-03-11T21:00:48.469015+0000","last_unstale":"2024-03-11T21:09:16.570255+0000","last_undegraded":"2024-03-11T21:09:16.570255+0000","last_fullsized":"2024-03-11T21:09:16.570255+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:07:38.356551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:07:38.356551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:20.236305+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"207'20","reported_seq":626,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-03-11T21:09:15.330838+0000","last_change":"2024-03-11T21:08:01.874794+0000","last_active":"2024-03-11T21:09:15.330838+0000","last_peered":"2024-03-11T21:09:15.330838+0000","last_clean":"2024-03-11T21:09:15.330838+0000","last_became_active":"2024-03-11T20:50:30.109549+0000","last_became_peered":"2024-03-11T20:50:30.109549+0000","last_unstale":"2024-03-11T21:09:15.330838+0000","last_undegraded":"2024-03-11T21:09:15.330838+0000","last_fullsized":"2024-03-11T21:09:15.330838+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'20","last_scrub_stamp":"2024-03-11T21:08:01.874656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:08:01.874656+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":635,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-11T21:09:13.522738+0000","last_change":"2024-03-11T21:07:50.781503+0000","last_active":"2024-03-11T21:09:13.522738+0000","last_peered":"2024-03-11T21:09:13.522738+0000","last_clean":"2024-03-11T21:09:13.522738+0000","last_became_active":"2024-03-11T21:02:37.517748+0000","last_became_peered":"2024-03-11T21:02:37.517748+0000","last_unstale":"2024-03-11T21:09:13.522738+0000","last_undegraded":"2024-03-11T21:09:13.522738+0000","last_fullsized":"2024-03-11T21:09:13.522738+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:07:50.781425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:52:51.931295+0000","last_clean_scrub_stamp":"2024-03-11T21:07:50.781425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.6","version":"336'7336","reported_seq":7406,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-03-11T21:09:15.497638+0000","last_change":"2024-03-11T21:07:14.953797+0000","last_active":"2024-03-11T21:09:15.497638+0000","last_peered":"2024-03-11T21:09:15.497638+0000","last_clean":"2024-03-11T21:09:15.497638+0000","last_became_active":"2024-03-11T21:07:14.952904+0000","last_became_peered":"2024-03-11T21:07:14.952904+0000","last_unstale":"2024-03-11T21:09:15.497638+0000","last_undegraded":"2024-03-11T21:09:15.497638+0000","last_fullsized":"2024-03-11T21:09:15.497638+0000","mapping_epoch":306,"log_start":"316'1400","ondisk_log_start":"316'1400","created":306,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:07:13.846944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:07:13.846944+0000","last_clean_scrub_stamp":"2024-03-11T21:07:13.846944+0000","objects_scrubbed":0,"log_size":5936,"log_dups_size":0,"ondisk_log_size":5936,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":416415744,"num_objects":6354,"num_object_clones":0,"num_object_copies":19062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6354,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7336,"num_write_kb":438080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":66,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-11T21:09:13.521431+0000","last_change":"2024-03-11T21:07:46.870757+0000","last_active":"2024-03-11T21:09:13.521431+0000","last_peered":"2024-03-11T21:09:13.521431+0000","last_clean":"2024-03-11T21:09:13.521431+0000","last_became_active":"2024-03-11T21:07:34.504579+0000","last_became_peered":"2024-03-11T21:07:34.504579+0000","last_unstale":"2024-03-11T21:09:13.521431+0000","last_undegraded":"2024-03-11T21:09:13.521431+0000","last_fullsized":"2024-03-11T21:09:13.521431+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:07:46.870646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:00:14.030599+0000","last_clean_scrub_stamp":"2024-03-11T21:07:46.870646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":43,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-11T21:09:13.518324+0000","last_change":"2024-03-11T21:08:05.308620+0000","last_active":"2024-03-11T21:09:13.518324+0000","last_peered":"2024-03-11T21:09:13.518324+0000","last_clean":"2024-03-11T21:09:13.518324+0000","last_became_active":"2024-03-11T21:08:05.307984+0000","last_became_peered":"2024-03-11T21:08:05.307984+0000","last_unstale":"2024-03-11T21:09:13.518324+0000","last_undegraded":"2024-03-11T21:09:13.518324+0000","last_fullsized":"2024-03-11T21:09:13.518324+0000","mapping_epoch":319,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:07:57.816881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:07:57.816881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.f","version":"207'27","reported_seq":644,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-03-11T21:09:13.523176+0000","last_change":"2024-03-11T21:08:04.106067+0000","last_active":"2024-03-11T21:09:13.523176+0000","last_peered":"2024-03-11T21:09:13.523176+0000","last_clean":"2024-03-11T21:09:13.523176+0000","last_became_active":"2024-03-11T21:00:27.963119+0000","last_became_peered":"2024-03-11T21:00:27.963119+0000","last_unstale":"2024-03-11T21:09:13.523176+0000","last_undegraded":"2024-03-11T21:09:13.523176+0000","last_fullsized":"2024-03-11T21:09:13.523176+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'27","last_scrub_stamp":"2024-03-11T21:08:04.105922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:08:04.105922+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:25.134101+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":592,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-03-11T21:09:15.330145+0000","last_change":"2024-03-11T21:08:03.929764+0000","last_active":"2024-03-11T21:09:15.330145+0000","last_peered":"2024-03-11T21:09:15.330145+0000","last_clean":"2024-03-11T21:09:15.330145+0000","last_became_active":"2024-03-11T20:50:32.652854+0000","last_became_peered":"2024-03-11T20:50:32.652854+0000","last_unstale":"2024-03-11T21:09:15.330145+0000","last_undegraded":"2024-03-11T21:09:15.330145+0000","last_fullsized":"2024-03-11T21:09:15.330145+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:08:03.929608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:59:00.946094+0000","last_clean_scrub_stamp":"2024-03-11T21:08:03.929608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:09:27.258446+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":6669402136,"num_objects":101768,"num_object_clones":0,"num_object_copies":305304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101768,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":116252,"num_write_kb":6976577,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":19985346560,"data_stored":19985334344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90152,"ondisk_log_size":90152,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":20,"num_read_kb":20,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1822,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1121,"num_read_kb":878,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":171,"num_bytes_recovered":2228,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3632,"internal_metadata":0},"log_size":778,"ondisk_log_size":778,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":126,"ondisk_log_size":126,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":989,"num_read_kb":984,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":68,"seq":292057776364,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2083956,"kb_used_data":1637672,"kb_used_omap":1,"kb_used_meta":446270,"kb_avail":91661196,"statfs":{"total":95995035648,"available":93861064704,"internally_reserved":0,"allocated":1676976128,"data_stored":1676506867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":456980938},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":57,"seq":244813136118,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3239720,"kb_used_data":2879452,"kb_used_omap":1,"kb_used_meta":360190,"kb_avail":90505432,"statfs":{"total":95995035648,"available":92677562368,"internally_reserved":0,"allocated":2948558848,"data_stored":2948101909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":368835015},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":211,"seq":906238099559,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1542452,"kb_used_data":1213224,"kb_used_omap":7,"kb_used_meta":329208,"kb_avail":92202700,"statfs":{"total":95995035648,"available":94415564800,"internally_reserved":0,"allocated":1242341376,"data_stored":1241880052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8078,"internal_metadata":337109106},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738676,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4513776,"kb_used_data":4071588,"kb_used_omap":1,"kb_used_meta":442110,"kb_avail":89231376,"statfs":{"total":95995035648,"available":91372929024,"internally_reserved":0,"allocated":4169306112,"data_stored":4168839913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":452721093},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378911,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2816984,"kb_used_data":2465100,"kb_used_omap":1,"kb_used_meta":351870,"kb_avail":90928168,"statfs":{"total":95995035648,"available":93110444032,"internally_reserved":0,"allocated":2524262400,"data_stored":2523820371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":360315333},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":28,"seq":120259084564,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2594716,"kb_used_data":2049808,"kb_used_omap":1,"kb_used_meta":544830,"kb_avail":91150436,"statfs":{"total":95995035648,"available":93338046464,"internally_reserved":0,"allocated":2099003392,"data_stored":2098557643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":557906376},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":204,"seq":876173328491,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3505876,"kb_used_data":3248388,"kb_used_omap":4,"kb_used_meta":257275,"kb_avail":90239276,"statfs":{"total":95995035648,"available":92405018624,"internally_reserved":0,"allocated":3326349312,"data_stored":3325859954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4451,"internal_metadata":263450269},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":44,"seq":188978561280,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2452444,"kb_used_data":2042256,"kb_used_omap":1,"kb_used_meta":410174,"kb_avail":91292708,"statfs":{"total":95995035648,"available":93483732992,"internally_reserved":0,"allocated":2091270144,"data_stored":2090823941,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":420018629},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3632,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4164489216,"data_stored":4164485144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1232732160,"data_stored":1232732160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2508652544,"data_stored":2508652544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2094399488,"data_stored":2094399488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":3308720128,"data_stored":3308716056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2076835840,"data_stored":2076835840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2935492608,"data_stored":2935488536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1664024576,"data_stored":1664024576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T21:09:21.191 INFO:tasks.thrashosds.thrasher:Setting 6.a to [3, 5, 2, 7, 6, 1] 2024-03-11T21:09:21.191 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '6.a', '3', '5', '2', '7', '6', '1'] 2024-03-11T21:09:21.191 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd pg-upmap-items 6.a 3 5 2 7 6 1 2024-03-11T21:09:21.262 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:09:21.300 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:09:21.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:21.307 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:09:21.408 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:21.436 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:21.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:21.439 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:09:21.540 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:09:21.571 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:09:21.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:21.578 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:09:21.678 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:09:21.710 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:09:21.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:21.713 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:09:21.759 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:21 smithi146 bash[23809]: cluster 2024-03-11T21:09:19.776140+0000 osd.7 (osd.7) 416 : cluster 0 10.c scrub ok 2024-03-11T21:09:21.759 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:21 smithi146 bash[23809]: cluster 2024-03-11T21:09:20.296727+0000 mgr.y (mgr.24380) 687 : cluster 0 pgmap v772: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.12k op/s 2024-03-11T21:09:21.759 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:21 smithi146 bash[23809]: cluster 2024-03-11T21:09:20.787955+0000 osd.5 (osd.5) 366 : cluster 0 10.6 scrub starts 2024-03-11T21:09:21.759 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:21 smithi146 bash[23809]: audit 2024-03-11T21:09:20.874119+0000 mon.a (mon.0) 1686 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:09:21.814 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:09:21.863 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:09:21.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:21.869 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:09:21.970 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:22.022 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:22.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:22.025 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:09:22.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:21 smithi023 bash[21372]: cluster 2024-03-11T21:09:19.776140+0000 osd.7 (osd.7) 416 : cluster 0 10.c scrub ok 2024-03-11T21:09:22.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:21 smithi023 bash[21372]: cluster 2024-03-11T21:09:20.296727+0000 mgr.y (mgr.24380) 687 : cluster 0 pgmap v772: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.12k op/s 2024-03-11T21:09:22.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:21 smithi023 bash[21372]: cluster 2024-03-11T21:09:20.787955+0000 osd.5 (osd.5) 366 : cluster 0 10.6 scrub starts 2024-03-11T21:09:22.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:21 smithi023 bash[21372]: audit 2024-03-11T21:09:20.874119+0000 mon.a (mon.0) 1686 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:09:22.029 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:21 smithi023 bash[29146]: cluster 2024-03-11T21:09:19.776140+0000 osd.7 (osd.7) 416 : cluster 0 10.c scrub ok 2024-03-11T21:09:22.029 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:21 smithi023 bash[29146]: cluster 2024-03-11T21:09:20.296727+0000 mgr.y (mgr.24380) 687 : cluster 0 pgmap v772: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.12k op/s 2024-03-11T21:09:22.029 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:21 smithi023 bash[29146]: cluster 2024-03-11T21:09:20.787955+0000 osd.5 (osd.5) 366 : cluster 0 10.6 scrub starts 2024-03-11T21:09:22.029 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:21 smithi023 bash[29146]: audit 2024-03-11T21:09:20.874119+0000 mon.a (mon.0) 1686 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:09:22.084 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:09:22.125 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:09:22.153 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:09:22.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:22.156 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:09:22.257 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:09:22.281 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:09:22.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:22.287 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:09:22.388 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:09:22.419 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:09:22.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:22.423 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:09:22.494 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T21:09:22.524 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:09:22.555 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:09:22.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:22.558 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:09:22.659 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:09:22.686 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:09:22.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:22.688 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:09:22.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:22 smithi023 bash[21372]: cluster 2024-03-11T21:09:21.036455+0000 osd.5 (osd.5) 367 : cluster 0 10.3 scrub ok 2024-03-11T21:09:22.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:22 smithi023 bash[21372]: cluster 2024-03-11T21:09:21.304489+0000 osd.7 (osd.7) 417 : cluster 0 3.13 scrub starts 2024-03-11T21:09:22.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:22 smithi023 bash[21372]: cluster 2024-03-11T21:09:21.307270+0000 osd.7 (osd.7) 418 : cluster 0 3.13 scrub ok 2024-03-11T21:09:22.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:22 smithi023 bash[21372]: cluster 2024-03-11T21:09:21.670506+0000 osd.6 (osd.6) 287 : cluster 0 10.d scrub starts 2024-03-11T21:09:22.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:22 smithi023 bash[21372]: cluster 2024-03-11T21:09:21.752275+0000 osd.5 (osd.5) 368 : cluster 0 6.17 scrub starts 2024-03-11T21:09:22.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:22 smithi023 bash[21372]: cluster 2024-03-11T21:09:22.036244+0000 osd.0 (osd.0) 287 : cluster 0 10.5 scrub starts 2024-03-11T21:09:22.779 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:22 smithi023 bash[29146]: cluster 2024-03-11T21:09:21.036455+0000 osd.5 (osd.5) 367 : cluster 0 10.3 scrub ok 2024-03-11T21:09:22.779 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:22 smithi023 bash[29146]: cluster 2024-03-11T21:09:21.304489+0000 osd.7 (osd.7) 417 : cluster 0 3.13 scrub starts 2024-03-11T21:09:22.779 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:22 smithi023 bash[29146]: cluster 2024-03-11T21:09:21.307270+0000 osd.7 (osd.7) 418 : cluster 0 3.13 scrub ok 2024-03-11T21:09:22.779 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:22 smithi023 bash[29146]: cluster 2024-03-11T21:09:21.670506+0000 osd.6 (osd.6) 287 : cluster 0 10.d scrub starts 2024-03-11T21:09:22.779 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:22 smithi023 bash[29146]: cluster 2024-03-11T21:09:21.752275+0000 osd.5 (osd.5) 368 : cluster 0 6.17 scrub starts 2024-03-11T21:09:22.779 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:22 smithi023 bash[29146]: cluster 2024-03-11T21:09:22.036244+0000 osd.0 (osd.0) 287 : cluster 0 10.5 scrub starts 2024-03-11T21:09:22.789 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:09:22.799 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:22 smithi146 bash[23809]: cluster 2024-03-11T21:09:21.036455+0000 osd.5 (osd.5) 367 : cluster 0 10.3 scrub ok 2024-03-11T21:09:22.799 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:22 smithi146 bash[23809]: cluster 2024-03-11T21:09:21.304489+0000 osd.7 (osd.7) 417 : cluster 0 3.13 scrub starts 2024-03-11T21:09:22.799 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:22 smithi146 bash[23809]: cluster 2024-03-11T21:09:21.307270+0000 osd.7 (osd.7) 418 : cluster 0 3.13 scrub ok 2024-03-11T21:09:22.799 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:22 smithi146 bash[23809]: cluster 2024-03-11T21:09:21.670506+0000 osd.6 (osd.6) 287 : cluster 0 10.d scrub starts 2024-03-11T21:09:22.800 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:22 smithi146 bash[23809]: cluster 2024-03-11T21:09:21.752275+0000 osd.5 (osd.5) 368 : cluster 0 6.17 scrub starts 2024-03-11T21:09:22.800 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:22 smithi146 bash[23809]: cluster 2024-03-11T21:09:22.036244+0000 osd.0 (osd.0) 287 : cluster 0 10.5 scrub starts 2024-03-11T21:09:22.813 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:09:22.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:22.816 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:09:22.917 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:09:22.949 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:09:22.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:22.954 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:09:23.055 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:09:23.080 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:09:23.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:23.084 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:09:23.185 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:23.217 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:23.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:23.223 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:09:23.324 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:23.342 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:23.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:23.343 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:09:23.444 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:09:23.474 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:09:23.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:23.477 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:09:23.578 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:09:23.608 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:09:23.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:23.610 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:09:23.711 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:09:23.737 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:09:23.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:23.739 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:09:23.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:23 smithi023 bash[21372]: cluster 2024-03-11T21:09:21.916309+0000 osd.5 (osd.5) 369 : cluster 0 6.17 scrub ok 2024-03-11T21:09:23.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:23 smithi023 bash[21372]: cluster 2024-03-11T21:09:22.297236+0000 mgr.y (mgr.24380) 688 : cluster 0 pgmap v774: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 6.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 732 B/s rd, 0 B/s wr, 713 op/s 2024-03-11T21:09:23.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:23 smithi023 bash[21372]: cluster 2024-03-11T21:09:22.474001+0000 osd.6 (osd.6) 288 : cluster 0 10.7 scrub ok 2024-03-11T21:09:23.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:23 smithi023 bash[21372]: cluster 2024-03-11T21:09:22.639578+0000 osd.6 (osd.6) 289 : cluster 0 10.b scrub starts 2024-03-11T21:09:23.779 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:23 smithi023 bash[29146]: cluster 2024-03-11T21:09:21.916309+0000 osd.5 (osd.5) 369 : cluster 0 6.17 scrub ok 2024-03-11T21:09:23.779 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:23 smithi023 bash[29146]: cluster 2024-03-11T21:09:22.297236+0000 mgr.y (mgr.24380) 688 : cluster 0 pgmap v774: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 6.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 732 B/s rd, 0 B/s wr, 713 op/s 2024-03-11T21:09:23.779 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:23 smithi023 bash[29146]: cluster 2024-03-11T21:09:22.474001+0000 osd.6 (osd.6) 288 : cluster 0 10.7 scrub ok 2024-03-11T21:09:23.779 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:23 smithi023 bash[29146]: cluster 2024-03-11T21:09:22.639578+0000 osd.6 (osd.6) 289 : cluster 0 10.b scrub starts 2024-03-11T21:09:23.840 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:09:23.884 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:09:23.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:23.892 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:09:23.939 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:23 smithi146 bash[23809]: cluster 2024-03-11T21:09:21.916309+0000 osd.5 (osd.5) 369 : cluster 0 6.17 scrub ok 2024-03-11T21:09:23.939 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:23 smithi146 bash[23809]: cluster 2024-03-11T21:09:22.297236+0000 mgr.y (mgr.24380) 688 : cluster 0 pgmap v774: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 6.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 732 B/s rd, 0 B/s wr, 713 op/s 2024-03-11T21:09:23.993 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:09:24.028 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:09:24.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:24.038 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:09:24.139 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:09:24.173 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:09:24.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:24.177 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:09:24.258 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:23 smithi146 bash[23809]: cluster 2024-03-11T21:09:22.474001+0000 osd.6 (osd.6) 288 : cluster 0 10.7 scrub ok 2024-03-11T21:09:24.259 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:23 smithi146 bash[23809]: cluster 2024-03-11T21:09:22.639578+0000 osd.6 (osd.6) 289 : cluster 0 10.b scrub starts 2024-03-11T21:09:24.278 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:09:24.305 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:09:24.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:24.307 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:09:24.408 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:09:24.434 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:09:24.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:24.440 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:09:24.541 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:09:24.574 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:09:24.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:24.579 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:09:24.680 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:09:24.709 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:09:24.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:24.713 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:09:24.813 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:09:24.835 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:09:24.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:24.837 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:09:24.938 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:09:24.947 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:24 smithi146 bash[23809]: cluster 2024-03-11T21:09:23.070436+0000 osd.5 (osd.5) 370 : cluster 0 10.6 scrub ok 2024-03-11T21:09:24.947 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:24 smithi146 bash[23809]: cluster 2024-03-11T21:09:23.327504+0000 osd.1 (osd.1) 171 : cluster 0 10.2 scrub starts 2024-03-11T21:09:24.948 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:24 smithi146 bash[23809]: cluster 2024-03-11T21:09:23.385902+0000 osd.7 (osd.7) 419 : cluster 0 3.9 scrub starts 2024-03-11T21:09:24.948 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:24 smithi146 bash[23809]: cluster 2024-03-11T21:09:23.387995+0000 osd.7 (osd.7) 420 : cluster 0 3.9 scrub ok 2024-03-11T21:09:24.948 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:24 smithi146 bash[23809]: cluster 2024-03-11T21:09:23.604327+0000 osd.6 (osd.6) 290 : cluster 0 10.0 scrub starts 2024-03-11T21:09:24.948 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:24 smithi146 bash[23809]: cluster 2024-03-11T21:09:23.813141+0000 osd.0 (osd.0) 288 : cluster 0 10.5 scrub ok 2024-03-11T21:09:24.948 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:24 smithi146 bash[23809]: cluster 2024-03-11T21:09:23.872748+0000 osd.5 (osd.5) 371 : cluster 0 purged_snaps scrub starts 2024-03-11T21:09:24.948 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:24 smithi146 bash[23809]: cluster 2024-03-11T21:09:23.872987+0000 osd.5 (osd.5) 372 : cluster 0 purged_snaps scrub ok 2024-03-11T21:09:24.963 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:09:24.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:24.968 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:09:25.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:24 smithi023 bash[21372]: cluster 2024-03-11T21:09:23.070436+0000 osd.5 (osd.5) 370 : cluster 0 10.6 scrub ok 2024-03-11T21:09:25.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:24 smithi023 bash[21372]: cluster 2024-03-11T21:09:23.327504+0000 osd.1 (osd.1) 171 : cluster 0 10.2 scrub starts 2024-03-11T21:09:25.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:24 smithi023 bash[21372]: cluster 2024-03-11T21:09:23.385902+0000 osd.7 (osd.7) 419 : cluster 0 3.9 scrub starts 2024-03-11T21:09:25.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:24 smithi023 bash[21372]: cluster 2024-03-11T21:09:23.387995+0000 osd.7 (osd.7) 420 : cluster 0 3.9 scrub ok 2024-03-11T21:09:25.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:24 smithi023 bash[21372]: cluster 2024-03-11T21:09:23.604327+0000 osd.6 (osd.6) 290 : cluster 0 10.0 scrub starts 2024-03-11T21:09:25.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:24 smithi023 bash[21372]: cluster 2024-03-11T21:09:23.813141+0000 osd.0 (osd.0) 288 : cluster 0 10.5 scrub ok 2024-03-11T21:09:25.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:24 smithi023 bash[21372]: cluster 2024-03-11T21:09:23.872748+0000 osd.5 (osd.5) 371 : cluster 0 purged_snaps scrub starts 2024-03-11T21:09:25.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:24 smithi023 bash[21372]: cluster 2024-03-11T21:09:23.872987+0000 osd.5 (osd.5) 372 : cluster 0 purged_snaps scrub ok 2024-03-11T21:09:25.029 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:24 smithi023 bash[29146]: cluster 2024-03-11T21:09:23.070436+0000 osd.5 (osd.5) 370 : cluster 0 10.6 scrub ok 2024-03-11T21:09:25.029 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:24 smithi023 bash[29146]: cluster 2024-03-11T21:09:23.327504+0000 osd.1 (osd.1) 171 : cluster 0 10.2 scrub starts 2024-03-11T21:09:25.029 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:24 smithi023 bash[29146]: cluster 2024-03-11T21:09:23.385902+0000 osd.7 (osd.7) 419 : cluster 0 3.9 scrub starts 2024-03-11T21:09:25.029 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:24 smithi023 bash[29146]: cluster 2024-03-11T21:09:23.387995+0000 osd.7 (osd.7) 420 : cluster 0 3.9 scrub ok 2024-03-11T21:09:25.030 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:24 smithi023 bash[29146]: cluster 2024-03-11T21:09:23.604327+0000 osd.6 (osd.6) 290 : cluster 0 10.0 scrub starts 2024-03-11T21:09:25.030 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:24 smithi023 bash[29146]: cluster 2024-03-11T21:09:23.813141+0000 osd.0 (osd.0) 288 : cluster 0 10.5 scrub ok 2024-03-11T21:09:25.030 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:24 smithi023 bash[29146]: cluster 2024-03-11T21:09:23.872748+0000 osd.5 (osd.5) 371 : cluster 0 purged_snaps scrub starts 2024-03-11T21:09:25.030 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:24 smithi023 bash[29146]: cluster 2024-03-11T21:09:23.872987+0000 osd.5 (osd.5) 372 : cluster 0 purged_snaps scrub ok 2024-03-11T21:09:25.069 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:09:25.104 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:09:25.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:25.108 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:09:25.210 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:09:25.256 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:09:25.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:25.261 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:09:25.361 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:09:25.393 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:09:25.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:25.395 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:09:25.496 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:09:25.569 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:09:25.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:25.573 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:09:25.674 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:09:25.700 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:09:25.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:25.704 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:09:25.805 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:09:25.828 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:09:25.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:25.830 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:09:25.931 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:09:25.941 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:25 smithi023 bash[21372]: cluster 2024-03-11T21:09:24.297664+0000 mgr.y (mgr.24380) 689 : cluster 0 pgmap v775: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 6.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 512 op/s 2024-03-11T21:09:25.941 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:25 smithi023 bash[21372]: cluster 2024-03-11T21:09:24.323258+0000 osd.6 (osd.6) 291 : cluster 0 10.d scrub ok 2024-03-11T21:09:25.941 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:25 smithi023 bash[21372]: cluster 2024-03-11T21:09:24.324663+0000 osd.0 (osd.0) 289 : cluster 0 2.13 scrub starts 2024-03-11T21:09:25.941 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:25 smithi023 bash[21372]: cluster 2024-03-11T21:09:24.330318+0000 osd.0 (osd.0) 290 : cluster 0 2.13 scrub ok 2024-03-11T21:09:25.941 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:25 smithi023 bash[21372]: cluster 2024-03-11T21:09:24.336858+0000 osd.1 (osd.1) 172 : cluster 0 2.d scrub starts 2024-03-11T21:09:25.941 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:25 smithi023 bash[21372]: cluster 2024-03-11T21:09:24.339162+0000 osd.1 (osd.1) 173 : cluster 0 2.d scrub ok 2024-03-11T21:09:25.941 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:25 smithi023 bash[21372]: cluster 2024-03-11T21:09:24.634693+0000 osd.6 (osd.6) 292 : cluster 0 10.a scrub starts 2024-03-11T21:09:25.941 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:25 smithi023 bash[21372]: cluster 2024-03-11T21:09:24.638916+0000 mon.a (mon.0) 1687 : cluster 0 osdmap e339: 8 total, 8 up, 8 in 2024-03-11T21:09:25.941 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:25 smithi023 bash[21372]: cluster 2024-03-11T21:09:24.802615+0000 osd.6 (osd.6) 293 : cluster 0 10.b scrub ok 2024-03-11T21:09:25.941 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:25 smithi023 bash[21372]: cluster 2024-03-11T21:09:24.809307+0000 osd.3 (osd.3) 382 : cluster 0 4.17 scrub starts 2024-03-11T21:09:25.942 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:25 smithi023 bash[21372]: cluster 2024-03-11T21:09:24.811781+0000 osd.3 (osd.3) 383 : cluster 0 4.17 scrub ok 2024-03-11T21:09:25.942 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:25 smithi023 bash[21372]: cluster 2024-03-11T21:09:25.054153+0000 osd.0 (osd.0) 291 : cluster 0 10.8 scrub starts 2024-03-11T21:09:25.942 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:25 smithi023 bash[21372]: cluster 2024-03-11T21:09:25.310041+0000 mon.a (mon.0) 1688 : cluster 0 osdmap e340: 8 total, 8 up, 8 in 2024-03-11T21:09:25.942 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:25 smithi023 bash[29146]: cluster 2024-03-11T21:09:24.297664+0000 mgr.y (mgr.24380) 689 : cluster 0 pgmap v775: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 6.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 512 op/s 2024-03-11T21:09:25.942 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:25 smithi023 bash[29146]: cluster 2024-03-11T21:09:24.323258+0000 osd.6 (osd.6) 291 : cluster 0 10.d scrub ok 2024-03-11T21:09:25.942 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:25 smithi023 bash[29146]: cluster 2024-03-11T21:09:24.324663+0000 osd.0 (osd.0) 289 : cluster 0 2.13 scrub starts 2024-03-11T21:09:25.942 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:25 smithi023 bash[29146]: cluster 2024-03-11T21:09:24.330318+0000 osd.0 (osd.0) 290 : cluster 0 2.13 scrub ok 2024-03-11T21:09:25.942 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:25 smithi023 bash[29146]: cluster 2024-03-11T21:09:24.336858+0000 osd.1 (osd.1) 172 : cluster 0 2.d scrub starts 2024-03-11T21:09:25.942 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:25 smithi023 bash[29146]: cluster 2024-03-11T21:09:24.339162+0000 osd.1 (osd.1) 173 : cluster 0 2.d scrub ok 2024-03-11T21:09:25.942 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:25 smithi023 bash[29146]: cluster 2024-03-11T21:09:24.634693+0000 osd.6 (osd.6) 292 : cluster 0 10.a scrub starts 2024-03-11T21:09:25.942 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:25 smithi023 bash[29146]: cluster 2024-03-11T21:09:24.638916+0000 mon.a (mon.0) 1687 : cluster 0 osdmap e339: 8 total, 8 up, 8 in 2024-03-11T21:09:25.943 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:25 smithi023 bash[29146]: cluster 2024-03-11T21:09:24.802615+0000 osd.6 (osd.6) 293 : cluster 0 10.b scrub ok 2024-03-11T21:09:25.943 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:25 smithi023 bash[29146]: cluster 2024-03-11T21:09:24.809307+0000 osd.3 (osd.3) 382 : cluster 0 4.17 scrub starts 2024-03-11T21:09:25.943 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:25 smithi023 bash[29146]: cluster 2024-03-11T21:09:24.811781+0000 osd.3 (osd.3) 383 : cluster 0 4.17 scrub ok 2024-03-11T21:09:25.943 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:25 smithi023 bash[29146]: cluster 2024-03-11T21:09:25.054153+0000 osd.0 (osd.0) 291 : cluster 0 10.8 scrub starts 2024-03-11T21:09:25.943 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:25 smithi023 bash[29146]: cluster 2024-03-11T21:09:25.310041+0000 mon.a (mon.0) 1688 : cluster 0 osdmap e340: 8 total, 8 up, 8 in 2024-03-11T21:09:25.943 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:09:25.945 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:09:25.973 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:09:25.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:25.982 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:09:26.008 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:09:25 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:09:26.009 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:25 smithi146 bash[23809]: cluster 2024-03-11T21:09:24.297664+0000 mgr.y (mgr.24380) 689 : cluster 0 pgmap v775: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 6.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 512 op/s 2024-03-11T21:09:26.009 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:25 smithi146 bash[23809]: cluster 2024-03-11T21:09:24.323258+0000 osd.6 (osd.6) 291 : cluster 0 10.d scrub ok 2024-03-11T21:09:26.009 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:25 smithi146 bash[23809]: cluster 2024-03-11T21:09:24.324663+0000 osd.0 (osd.0) 289 : cluster 0 2.13 scrub starts 2024-03-11T21:09:26.009 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:25 smithi146 bash[23809]: cluster 2024-03-11T21:09:24.330318+0000 osd.0 (osd.0) 290 : cluster 0 2.13 scrub ok 2024-03-11T21:09:26.009 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:25 smithi146 bash[23809]: cluster 2024-03-11T21:09:24.336858+0000 osd.1 (osd.1) 172 : cluster 0 2.d scrub starts 2024-03-11T21:09:26.009 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:25 smithi146 bash[23809]: cluster 2024-03-11T21:09:24.339162+0000 osd.1 (osd.1) 173 : cluster 0 2.d scrub ok 2024-03-11T21:09:26.009 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:25 smithi146 bash[23809]: cluster 2024-03-11T21:09:24.634693+0000 osd.6 (osd.6) 292 : cluster 0 10.a scrub starts 2024-03-11T21:09:26.009 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:25 smithi146 bash[23809]: cluster 2024-03-11T21:09:24.638916+0000 mon.a (mon.0) 1687 : cluster 0 osdmap e339: 8 total, 8 up, 8 in 2024-03-11T21:09:26.009 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:25 smithi146 bash[23809]: cluster 2024-03-11T21:09:24.802615+0000 osd.6 (osd.6) 293 : cluster 0 10.b scrub ok 2024-03-11T21:09:26.009 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:25 smithi146 bash[23809]: cluster 2024-03-11T21:09:24.809307+0000 osd.3 (osd.3) 382 : cluster 0 4.17 scrub starts 2024-03-11T21:09:26.009 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:25 smithi146 bash[23809]: cluster 2024-03-11T21:09:24.811781+0000 osd.3 (osd.3) 383 : cluster 0 4.17 scrub ok 2024-03-11T21:09:26.009 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:25 smithi146 bash[23809]: cluster 2024-03-11T21:09:25.054153+0000 osd.0 (osd.0) 291 : cluster 0 10.8 scrub starts 2024-03-11T21:09:26.009 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:25 smithi146 bash[23809]: cluster 2024-03-11T21:09:25.310041+0000 mon.a (mon.0) 1688 : cluster 0 osdmap e340: 8 total, 8 up, 8 in 2024-03-11T21:09:26.083 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:26.107 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:26.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:26.109 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:09:26.210 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:09:26.341 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:09:26.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:26.346 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:09:26.447 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:09:26.468 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:09:26.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:26.476 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:09:26.577 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:09:26.607 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:09:26.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:26.609 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:09:26.710 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:09:26.720 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:26 smithi146 bash[23809]: cluster 2024-03-11T21:09:25.047203+0000 osd.1 (osd.1) 174 : cluster 0 10.2 scrub ok 2024-03-11T21:09:26.721 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:26 smithi146 bash[23809]: cluster 2024-03-11T21:09:25.309948+0000 osd.7 (osd.7) 421 : cluster 0 6.12 scrub starts 2024-03-11T21:09:26.721 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:26 smithi146 bash[23809]: cluster 2024-03-11T21:09:25.312965+0000 osd.7 (osd.7) 422 : cluster 0 6.12 scrub ok 2024-03-11T21:09:26.721 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:26 smithi146 bash[23809]: cluster 2024-03-11T21:09:25.607503+0000 osd.3 (osd.3) 384 : cluster 0 purged_snaps scrub starts 2024-03-11T21:09:26.721 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:26 smithi146 bash[23809]: cluster 2024-03-11T21:09:25.607554+0000 osd.3 (osd.3) 385 : cluster 0 purged_snaps scrub ok 2024-03-11T21:09:26.721 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:26 smithi146 bash[23809]: cluster 2024-03-11T21:09:25.671573+0000 osd.6 (osd.6) 294 : cluster 0 10.e scrub starts 2024-03-11T21:09:26.721 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:26 smithi146 bash[23809]: cluster 2024-03-11T21:09:25.774284+0000 osd.6 (osd.6) 295 : cluster 0 10.0 scrub ok 2024-03-11T21:09:26.721 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:26 smithi146 bash[23809]: cluster 2024-03-11T21:09:26.312549+0000 mon.a (mon.0) 1689 : cluster 0 osdmap e341: 8 total, 8 up, 8 in 2024-03-11T21:09:26.748 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:09:26.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:26.751 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:09:26.852 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:09:26.959 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:09:26.986 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:26 smithi023 bash[21372]: cluster 2024-03-11T21:09:25.047203+0000 osd.1 (osd.1) 174 : cluster 0 10.2 scrub ok 2024-03-11T21:09:26.986 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:26 smithi023 bash[21372]: cluster 2024-03-11T21:09:25.309948+0000 osd.7 (osd.7) 421 : cluster 0 6.12 scrub starts 2024-03-11T21:09:26.986 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:26 smithi023 bash[21372]: cluster 2024-03-11T21:09:25.312965+0000 osd.7 (osd.7) 422 : cluster 0 6.12 scrub ok 2024-03-11T21:09:26.986 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:26 smithi023 bash[21372]: cluster 2024-03-11T21:09:25.607503+0000 osd.3 (osd.3) 384 : cluster 0 purged_snaps scrub starts 2024-03-11T21:09:26.986 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:26 smithi023 bash[21372]: cluster 2024-03-11T21:09:25.607554+0000 osd.3 (osd.3) 385 : cluster 0 purged_snaps scrub ok 2024-03-11T21:09:26.987 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:26 smithi023 bash[21372]: cluster 2024-03-11T21:09:25.671573+0000 osd.6 (osd.6) 294 : cluster 0 10.e scrub starts 2024-03-11T21:09:26.987 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:26 smithi023 bash[21372]: cluster 2024-03-11T21:09:25.774284+0000 osd.6 (osd.6) 295 : cluster 0 10.0 scrub ok 2024-03-11T21:09:26.987 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:26 smithi023 bash[21372]: cluster 2024-03-11T21:09:26.312549+0000 mon.a (mon.0) 1689 : cluster 0 osdmap e341: 8 total, 8 up, 8 in 2024-03-11T21:09:26.987 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:26 smithi023 bash[29146]: cluster 2024-03-11T21:09:25.047203+0000 osd.1 (osd.1) 174 : cluster 0 10.2 scrub ok 2024-03-11T21:09:26.987 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:26 smithi023 bash[29146]: cluster 2024-03-11T21:09:25.309948+0000 osd.7 (osd.7) 421 : cluster 0 6.12 scrub starts 2024-03-11T21:09:26.987 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:26 smithi023 bash[29146]: cluster 2024-03-11T21:09:25.312965+0000 osd.7 (osd.7) 422 : cluster 0 6.12 scrub ok 2024-03-11T21:09:26.987 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:26 smithi023 bash[29146]: cluster 2024-03-11T21:09:25.607503+0000 osd.3 (osd.3) 384 : cluster 0 purged_snaps scrub starts 2024-03-11T21:09:26.987 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:26 smithi023 bash[29146]: cluster 2024-03-11T21:09:25.607554+0000 osd.3 (osd.3) 385 : cluster 0 purged_snaps scrub ok 2024-03-11T21:09:26.987 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:26 smithi023 bash[29146]: cluster 2024-03-11T21:09:25.671573+0000 osd.6 (osd.6) 294 : cluster 0 10.e scrub starts 2024-03-11T21:09:26.988 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:26 smithi023 bash[29146]: cluster 2024-03-11T21:09:25.774284+0000 osd.6 (osd.6) 295 : cluster 0 10.0 scrub ok 2024-03-11T21:09:26.988 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:26 smithi023 bash[29146]: cluster 2024-03-11T21:09:26.312549+0000 mon.a (mon.0) 1689 : cluster 0 osdmap e341: 8 total, 8 up, 8 in 2024-03-11T21:09:27.041 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:09:27.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:27.043 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:09:27.144 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:09:27.173 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:09:27.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:27.174 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:09:27.275 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:09:27.299 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:09:27.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:27.302 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:09:27.403 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:09:27.435 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:09:27.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:27.441 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:09:27.542 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:09:27.635 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:09:27.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:27.641 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:09:27.742 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:09:27.767 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:09:27.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:27.777 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:09:27.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:27 smithi023 bash[21372]: audit 2024-03-11T21:09:25.777186+0000 mgr.y (mgr.24380) 690 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:09:27.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:27 smithi023 bash[21372]: cluster 2024-03-11T21:09:26.294505+0000 osd.1 (osd.1) 175 : cluster 0 4.7 scrub starts 2024-03-11T21:09:27.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:27 smithi023 bash[21372]: cluster 2024-03-11T21:09:26.296702+0000 osd.1 (osd.1) 176 : cluster 0 4.7 scrub ok 2024-03-11T21:09:27.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:27 smithi023 bash[21372]: cluster 2024-03-11T21:09:26.298356+0000 mgr.y (mgr.24380) 691 : cluster 0 pgmap v778: 148 pgs: 3 active+clean+scrubbing, 145 active+clean; 5.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 828 op/s 2024-03-11T21:09:27.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:27 smithi023 bash[21372]: cluster 2024-03-11T21:09:26.872078+0000 osd.0 (osd.0) 292 : cluster 0 10.8 scrub ok 2024-03-11T21:09:27.779 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:27 smithi023 bash[29146]: audit 2024-03-11T21:09:25.777186+0000 mgr.y (mgr.24380) 690 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:09:27.779 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:27 smithi023 bash[29146]: cluster 2024-03-11T21:09:26.294505+0000 osd.1 (osd.1) 175 : cluster 0 4.7 scrub starts 2024-03-11T21:09:27.779 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:27 smithi023 bash[29146]: cluster 2024-03-11T21:09:26.296702+0000 osd.1 (osd.1) 176 : cluster 0 4.7 scrub ok 2024-03-11T21:09:27.779 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:27 smithi023 bash[29146]: cluster 2024-03-11T21:09:26.298356+0000 mgr.y (mgr.24380) 691 : cluster 0 pgmap v778: 148 pgs: 3 active+clean+scrubbing, 145 active+clean; 5.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 828 op/s 2024-03-11T21:09:27.779 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:27 smithi023 bash[29146]: cluster 2024-03-11T21:09:26.872078+0000 osd.0 (osd.0) 292 : cluster 0 10.8 scrub ok 2024-03-11T21:09:27.877 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:09:27.887 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:27 smithi146 bash[23809]: audit 2024-03-11T21:09:25.777186+0000 mgr.y (mgr.24380) 690 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:09:27.887 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:27 smithi146 bash[23809]: cluster 2024-03-11T21:09:26.294505+0000 osd.1 (osd.1) 175 : cluster 0 4.7 scrub starts 2024-03-11T21:09:27.888 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:27 smithi146 bash[23809]: cluster 2024-03-11T21:09:26.296702+0000 osd.1 (osd.1) 176 : cluster 0 4.7 scrub ok 2024-03-11T21:09:27.888 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:27 smithi146 bash[23809]: cluster 2024-03-11T21:09:26.298356+0000 mgr.y (mgr.24380) 691 : cluster 0 pgmap v778: 148 pgs: 3 active+clean+scrubbing, 145 active+clean; 5.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 828 op/s 2024-03-11T21:09:27.888 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:27 smithi146 bash[23809]: cluster 2024-03-11T21:09:26.872078+0000 osd.0 (osd.0) 292 : cluster 0 10.8 scrub ok 2024-03-11T21:09:27.911 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:09:27.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:27.918 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:09:28.019 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:09:28.090 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:09:28.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:28.092 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:09:28.193 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:09:28.222 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:09:28.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:28.226 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:09:28.327 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:28.352 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:28.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:28.356 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:09:28.457 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:09:28.467 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:09:28 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:09:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:09:28.490 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:09:28.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:28.492 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:09:28.593 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:28.630 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:28.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:28.634 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:09:28.735 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:09:28.770 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:09:28.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:28.775 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:09:28.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:28 smithi023 bash[21372]: cluster 2024-03-11T21:09:27.146322+0000 osd.6 (osd.6) 296 : cluster 0 10.a scrub ok 2024-03-11T21:09:28.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:28 smithi023 bash[21372]: cluster 2024-03-11T21:09:27.651084+0000 osd.6 (osd.6) 297 : cluster 0 2.14 scrub starts 2024-03-11T21:09:28.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:28 smithi023 bash[21372]: cluster 2024-03-11T21:09:27.653218+0000 osd.6 (osd.6) 298 : cluster 0 2.14 scrub ok 2024-03-11T21:09:28.778 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:28 smithi023 bash[29146]: cluster 2024-03-11T21:09:27.146322+0000 osd.6 (osd.6) 296 : cluster 0 10.a scrub ok 2024-03-11T21:09:28.778 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:28 smithi023 bash[29146]: cluster 2024-03-11T21:09:27.651084+0000 osd.6 (osd.6) 297 : cluster 0 2.14 scrub starts 2024-03-11T21:09:28.779 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:28 smithi023 bash[29146]: cluster 2024-03-11T21:09:27.653218+0000 osd.6 (osd.6) 298 : cluster 0 2.14 scrub ok 2024-03-11T21:09:28.876 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:09:28.904 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:09:28.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:28.908 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:09:29.008 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:28 smithi146 bash[23809]: cluster 2024-03-11T21:09:27.146322+0000 osd.6 (osd.6) 296 : cluster 0 10.a scrub ok 2024-03-11T21:09:29.009 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:28 smithi146 bash[23809]: cluster 2024-03-11T21:09:27.651084+0000 osd.6 (osd.6) 297 : cluster 0 2.14 scrub starts 2024-03-11T21:09:29.009 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:28 smithi146 bash[23809]: cluster 2024-03-11T21:09:27.653218+0000 osd.6 (osd.6) 298 : cluster 0 2.14 scrub ok 2024-03-11T21:09:29.009 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:29.026 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:29.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:29.028 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:09:29.129 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:09:29.156 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:09:29.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:29.158 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:09:29.259 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:09:29.403 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:09:29.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:29.406 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:09:29.507 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:09:29.530 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:09:29.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:29.534 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:09:29.635 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:09:29.671 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:09:29.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:29.675 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:09:29.776 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:29.821 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:29.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:29.825 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:09:29.926 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:09:29.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:29.965 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:09:29.965 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:09:29.965 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:29 smithi023 bash[29146]: cluster 2024-03-11T21:09:27.964489+0000 osd.6 (osd.6) 299 : cluster 0 10.e scrub ok 2024-03-11T21:09:29.965 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:29 smithi023 bash[29146]: cluster 2024-03-11T21:09:28.260643+0000 osd.1 (osd.1) 177 : cluster 0 2.10 scrub starts 2024-03-11T21:09:29.965 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:29 smithi023 bash[29146]: cluster 2024-03-11T21:09:28.261900+0000 osd.1 (osd.1) 178 : cluster 0 2.10 scrub ok 2024-03-11T21:09:29.966 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:29 smithi023 bash[29146]: cluster 2024-03-11T21:09:28.300837+0000 mgr.y (mgr.24380) 692 : cluster 0 pgmap v780: 148 pgs: 3 active+clean+scrubbing, 145 active+clean; 5.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 715 op/s 2024-03-11T21:09:29.966 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:29 smithi023 bash[29146]: cluster 2024-03-11T21:09:28.303967+0000 osd.7 (osd.7) 423 : cluster 0 3.15 scrub starts 2024-03-11T21:09:29.966 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:29 smithi023 bash[29146]: cluster 2024-03-11T21:09:28.305995+0000 osd.7 (osd.7) 424 : cluster 0 3.15 scrub ok 2024-03-11T21:09:29.966 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:29 smithi023 bash[29146]: cluster 2024-03-11T21:09:28.640999+0000 osd.6 (osd.6) 300 : cluster 0 2.7 scrub starts 2024-03-11T21:09:29.966 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:29 smithi023 bash[29146]: cluster 2024-03-11T21:09:28.645469+0000 osd.6 (osd.6) 301 : cluster 0 2.7 scrub ok 2024-03-11T21:09:29.966 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:29 smithi023 bash[21372]: cluster 2024-03-11T21:09:27.964489+0000 osd.6 (osd.6) 299 : cluster 0 10.e scrub ok 2024-03-11T21:09:29.966 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:29 smithi023 bash[21372]: cluster 2024-03-11T21:09:28.260643+0000 osd.1 (osd.1) 177 : cluster 0 2.10 scrub starts 2024-03-11T21:09:29.966 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:29 smithi023 bash[21372]: cluster 2024-03-11T21:09:28.261900+0000 osd.1 (osd.1) 178 : cluster 0 2.10 scrub ok 2024-03-11T21:09:29.966 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:29 smithi023 bash[21372]: cluster 2024-03-11T21:09:28.300837+0000 mgr.y (mgr.24380) 692 : cluster 0 pgmap v780: 148 pgs: 3 active+clean+scrubbing, 145 active+clean; 5.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 715 op/s 2024-03-11T21:09:29.966 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:29 smithi023 bash[21372]: cluster 2024-03-11T21:09:28.303967+0000 osd.7 (osd.7) 423 : cluster 0 3.15 scrub starts 2024-03-11T21:09:29.966 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:29 smithi023 bash[21372]: cluster 2024-03-11T21:09:28.305995+0000 osd.7 (osd.7) 424 : cluster 0 3.15 scrub ok 2024-03-11T21:09:29.967 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:29 smithi023 bash[21372]: cluster 2024-03-11T21:09:28.640999+0000 osd.6 (osd.6) 300 : cluster 0 2.7 scrub starts 2024-03-11T21:09:29.967 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:29 smithi023 bash[21372]: cluster 2024-03-11T21:09:28.645469+0000 osd.6 (osd.6) 301 : cluster 0 2.7 scrub ok 2024-03-11T21:09:30.066 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:09:30.091 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:09:30.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:30.093 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:09:30.194 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:09:30.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:29 smithi146 bash[23809]: cluster 2024-03-11T21:09:27.964489+0000 osd.6 (osd.6) 299 : cluster 0 10.e scrub ok 2024-03-11T21:09:30.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:29 smithi146 bash[23809]: cluster 2024-03-11T21:09:28.260643+0000 osd.1 (osd.1) 177 : cluster 0 2.10 scrub starts 2024-03-11T21:09:30.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:29 smithi146 bash[23809]: cluster 2024-03-11T21:09:28.261900+0000 osd.1 (osd.1) 178 : cluster 0 2.10 scrub ok 2024-03-11T21:09:30.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:29 smithi146 bash[23809]: cluster 2024-03-11T21:09:28.300837+0000 mgr.y (mgr.24380) 692 : cluster 0 pgmap v780: 148 pgs: 3 active+clean+scrubbing, 145 active+clean; 5.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 715 op/s 2024-03-11T21:09:30.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:29 smithi146 bash[23809]: cluster 2024-03-11T21:09:28.303967+0000 osd.7 (osd.7) 423 : cluster 0 3.15 scrub starts 2024-03-11T21:09:30.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:29 smithi146 bash[23809]: cluster 2024-03-11T21:09:28.305995+0000 osd.7 (osd.7) 424 : cluster 0 3.15 scrub ok 2024-03-11T21:09:30.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:29 smithi146 bash[23809]: cluster 2024-03-11T21:09:28.640999+0000 osd.6 (osd.6) 300 : cluster 0 2.7 scrub starts 2024-03-11T21:09:30.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:29 smithi146 bash[23809]: cluster 2024-03-11T21:09:28.645469+0000 osd.6 (osd.6) 301 : cluster 0 2.7 scrub ok 2024-03-11T21:09:30.223 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:09:30.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:30.226 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:09:30.327 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:30.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:30.362 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:30.362 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:09:30.463 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:09:30.491 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:09:30.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:30.494 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:09:30.595 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:09:30.617 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:09:30.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:30.621 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:09:30.722 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:09:30.754 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:09:30.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:30.759 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:09:30.759 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:30 smithi146 bash[23809]: cluster 2024-03-11T21:09:29.561974+0000 osd.1 (osd.1) 179 : cluster 0 purged_snaps scrub starts 2024-03-11T21:09:30.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:30 smithi146 bash[23809]: cluster 2024-03-11T21:09:29.562122+0000 osd.1 (osd.1) 180 : cluster 0 purged_snaps scrub ok 2024-03-11T21:09:30.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:30 smithi146 bash[23809]: cluster 2024-03-11T21:09:29.670688+0000 mon.a (mon.0) 1690 : cluster 0 osdmap e342: 8 total, 8 up, 8 in 2024-03-11T21:09:30.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:30 smithi146 bash[23809]: cluster 2024-03-11T21:09:29.886514+0000 osd.3 (osd.3) 386 : cluster 0 4.1b scrub starts 2024-03-11T21:09:30.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:30 smithi146 bash[23809]: cluster 2024-03-11T21:09:29.891595+0000 osd.3 (osd.3) 387 : cluster 0 4.1b scrub ok 2024-03-11T21:09:30.860 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:09:30.869 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:30 smithi023 bash[21372]: cluster 2024-03-11T21:09:29.561974+0000 osd.1 (osd.1) 179 : cluster 0 purged_snaps scrub starts 2024-03-11T21:09:30.870 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:30 smithi023 bash[21372]: cluster 2024-03-11T21:09:29.562122+0000 osd.1 (osd.1) 180 : cluster 0 purged_snaps scrub ok 2024-03-11T21:09:30.870 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:30 smithi023 bash[21372]: cluster 2024-03-11T21:09:29.670688+0000 mon.a (mon.0) 1690 : cluster 0 osdmap e342: 8 total, 8 up, 8 in 2024-03-11T21:09:30.870 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:30 smithi023 bash[21372]: cluster 2024-03-11T21:09:29.886514+0000 osd.3 (osd.3) 386 : cluster 0 4.1b scrub starts 2024-03-11T21:09:30.870 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:30 smithi023 bash[21372]: cluster 2024-03-11T21:09:29.891595+0000 osd.3 (osd.3) 387 : cluster 0 4.1b scrub ok 2024-03-11T21:09:30.870 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:30 smithi023 bash[29146]: cluster 2024-03-11T21:09:29.561974+0000 osd.1 (osd.1) 179 : cluster 0 purged_snaps scrub starts 2024-03-11T21:09:30.870 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:30 smithi023 bash[29146]: cluster 2024-03-11T21:09:29.562122+0000 osd.1 (osd.1) 180 : cluster 0 purged_snaps scrub ok 2024-03-11T21:09:30.870 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:30 smithi023 bash[29146]: cluster 2024-03-11T21:09:29.670688+0000 mon.a (mon.0) 1690 : cluster 0 osdmap e342: 8 total, 8 up, 8 in 2024-03-11T21:09:30.870 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:30 smithi023 bash[29146]: cluster 2024-03-11T21:09:29.886514+0000 osd.3 (osd.3) 386 : cluster 0 4.1b scrub starts 2024-03-11T21:09:30.870 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:30 smithi023 bash[29146]: cluster 2024-03-11T21:09:29.891595+0000 osd.3 (osd.3) 387 : cluster 0 4.1b scrub ok 2024-03-11T21:09:30.890 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:09:30.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:30.895 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:09:30.996 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:09:31.016 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:09:31.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:31.017 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:09:31.118 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:09:31.142 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:09:31.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:31.144 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:09:31.245 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:09:31.589 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:09:31.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:31.591 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:09:31.692 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:09:31.725 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:09:31.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:31.727 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:09:31.828 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:09:31.838 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:31 smithi023 bash[21372]: cluster 2024-03-11T21:09:30.301543+0000 mgr.y (mgr.24380) 693 : cluster 0 pgmap v782: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 1.32k op/s 2024-03-11T21:09:31.838 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:31 smithi023 bash[21372]: cluster 2024-03-11T21:09:30.614213+0000 osd.6 (osd.6) 302 : cluster 0 5.2 scrub starts 2024-03-11T21:09:31.839 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:31 smithi023 bash[21372]: cluster 2024-03-11T21:09:30.672116+0000 osd.6 (osd.6) 303 : cluster 0 5.2 scrub ok 2024-03-11T21:09:31.839 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:31 smithi023 bash[21372]: cluster 2024-03-11T21:09:30.723137+0000 osd.5 (osd.5) 373 : cluster 0 3.5 scrub starts 2024-03-11T21:09:31.839 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:31 smithi023 bash[21372]: cluster 2024-03-11T21:09:30.729164+0000 osd.5 (osd.5) 374 : cluster 0 3.5 scrub ok 2024-03-11T21:09:31.839 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:31 smithi023 bash[29146]: cluster 2024-03-11T21:09:30.301543+0000 mgr.y (mgr.24380) 693 : cluster 0 pgmap v782: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 1.32k op/s 2024-03-11T21:09:31.839 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:31 smithi023 bash[29146]: cluster 2024-03-11T21:09:30.614213+0000 osd.6 (osd.6) 302 : cluster 0 5.2 scrub starts 2024-03-11T21:09:31.839 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:31 smithi023 bash[29146]: cluster 2024-03-11T21:09:30.672116+0000 osd.6 (osd.6) 303 : cluster 0 5.2 scrub ok 2024-03-11T21:09:31.839 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:31 smithi023 bash[29146]: cluster 2024-03-11T21:09:30.723137+0000 osd.5 (osd.5) 373 : cluster 0 3.5 scrub starts 2024-03-11T21:09:31.839 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:31 smithi023 bash[29146]: cluster 2024-03-11T21:09:30.729164+0000 osd.5 (osd.5) 374 : cluster 0 3.5 scrub ok 2024-03-11T21:09:31.869 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:09:31.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:31.872 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:09:31.972 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:31.982 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:31 smithi146 bash[23809]: cluster 2024-03-11T21:09:30.301543+0000 mgr.y (mgr.24380) 693 : cluster 0 pgmap v782: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 1.32k op/s 2024-03-11T21:09:31.983 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:31 smithi146 bash[23809]: cluster 2024-03-11T21:09:30.614213+0000 osd.6 (osd.6) 302 : cluster 0 5.2 scrub starts 2024-03-11T21:09:31.983 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:31 smithi146 bash[23809]: cluster 2024-03-11T21:09:30.672116+0000 osd.6 (osd.6) 303 : cluster 0 5.2 scrub ok 2024-03-11T21:09:31.983 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:31 smithi146 bash[23809]: cluster 2024-03-11T21:09:30.723137+0000 osd.5 (osd.5) 373 : cluster 0 3.5 scrub starts 2024-03-11T21:09:31.983 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:31 smithi146 bash[23809]: cluster 2024-03-11T21:09:30.729164+0000 osd.5 (osd.5) 374 : cluster 0 3.5 scrub ok 2024-03-11T21:09:31.997 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:32.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:32.002 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:09:32.103 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:09:32.131 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:09:32.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:32.137 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:09:32.238 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:09:32.282 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:09:32.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:32.287 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:09:32.388 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:09:32.550 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:09:32.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:32.551 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:09:32.652 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:09:32.674 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:09:32.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:32.676 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:09:32.777 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:09:32.915 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:09:32.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:32.917 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:09:33.008 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:32 smithi146 bash[23809]: cluster 2024-03-11T21:09:31.397517+0000 osd.2 (osd.2) 253 : cluster 0 purged_snaps scrub starts 2024-03-11T21:09:33.009 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:32 smithi146 bash[23809]: cluster 2024-03-11T21:09:31.397592+0000 osd.2 (osd.2) 254 : cluster 0 purged_snaps scrub ok 2024-03-11T21:09:33.009 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:32 smithi146 bash[23809]: cluster 2024-03-11T21:09:31.574052+0000 osd.0 (osd.0) 293 : cluster 0 purged_snaps scrub starts 2024-03-11T21:09:33.009 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:32 smithi146 bash[23809]: cluster 2024-03-11T21:09:31.574191+0000 osd.0 (osd.0) 294 : cluster 0 purged_snaps scrub ok 2024-03-11T21:09:33.009 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:32 smithi146 bash[23809]: cluster 2024-03-11T21:09:31.698711+0000 mon.a (mon.0) 1691 : cluster 0 osdmap e343: 8 total, 8 up, 8 in 2024-03-11T21:09:33.009 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:32 smithi146 bash[23809]: cluster 2024-03-11T21:09:31.713890+0000 osd.5 (osd.5) 375 : cluster 0 3.1c scrub starts 2024-03-11T21:09:33.009 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:32 smithi146 bash[23809]: cluster 2024-03-11T21:09:31.716285+0000 osd.5 (osd.5) 376 : cluster 0 3.1c scrub ok 2024-03-11T21:09:33.009 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:32 smithi146 bash[23809]: cluster 2024-03-11T21:09:32.098185+0000 osd.0 (osd.0) 295 : cluster 0 10.4 scrub starts 2024-03-11T21:09:33.009 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:32 smithi146 bash[23809]: cluster 2024-03-11T21:09:32.302113+0000 mgr.y (mgr.24380) 694 : cluster 0 pgmap v784: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 761 op/s 2024-03-11T21:09:33.009 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:32 smithi146 bash[23809]: cluster 2024-03-11T21:09:32.382348+0000 osd.7 (osd.7) 425 : cluster 0 3.14 scrub starts 2024-03-11T21:09:33.009 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:32 smithi146 bash[23809]: cluster 2024-03-11T21:09:32.383957+0000 osd.7 (osd.7) 426 : cluster 0 3.14 scrub ok 2024-03-11T21:09:33.018 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:09:33.049 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:09:33.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:33.051 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:09:33.053 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:32 smithi023 bash[21372]: cluster 2024-03-11T21:09:31.397517+0000 osd.2 (osd.2) 253 : cluster 0 purged_snaps scrub starts 2024-03-11T21:09:33.053 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:32 smithi023 bash[21372]: cluster 2024-03-11T21:09:31.397592+0000 osd.2 (osd.2) 254 : cluster 0 purged_snaps scrub ok 2024-03-11T21:09:33.053 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:32 smithi023 bash[21372]: cluster 2024-03-11T21:09:31.574052+0000 osd.0 (osd.0) 293 : cluster 0 purged_snaps scrub starts 2024-03-11T21:09:33.054 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:32 smithi023 bash[21372]: cluster 2024-03-11T21:09:31.574191+0000 osd.0 (osd.0) 294 : cluster 0 purged_snaps scrub ok 2024-03-11T21:09:33.054 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:32 smithi023 bash[21372]: cluster 2024-03-11T21:09:31.698711+0000 mon.a (mon.0) 1691 : cluster 0 osdmap e343: 8 total, 8 up, 8 in 2024-03-11T21:09:33.054 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:32 smithi023 bash[21372]: cluster 2024-03-11T21:09:31.713890+0000 osd.5 (osd.5) 375 : cluster 0 3.1c scrub starts 2024-03-11T21:09:33.054 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:32 smithi023 bash[21372]: cluster 2024-03-11T21:09:31.716285+0000 osd.5 (osd.5) 376 : cluster 0 3.1c scrub ok 2024-03-11T21:09:33.054 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:32 smithi023 bash[21372]: cluster 2024-03-11T21:09:32.098185+0000 osd.0 (osd.0) 295 : cluster 0 10.4 scrub starts 2024-03-11T21:09:33.054 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:32 smithi023 bash[21372]: cluster 2024-03-11T21:09:32.302113+0000 mgr.y (mgr.24380) 694 : cluster 0 pgmap v784: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 761 op/s 2024-03-11T21:09:33.054 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:32 smithi023 bash[21372]: cluster 2024-03-11T21:09:32.382348+0000 osd.7 (osd.7) 425 : cluster 0 3.14 scrub starts 2024-03-11T21:09:33.054 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:32 smithi023 bash[21372]: cluster 2024-03-11T21:09:32.383957+0000 osd.7 (osd.7) 426 : cluster 0 3.14 scrub ok 2024-03-11T21:09:33.055 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:32 smithi023 bash[29146]: cluster 2024-03-11T21:09:31.397517+0000 osd.2 (osd.2) 253 : cluster 0 purged_snaps scrub starts 2024-03-11T21:09:33.055 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:32 smithi023 bash[29146]: cluster 2024-03-11T21:09:31.397592+0000 osd.2 (osd.2) 254 : cluster 0 purged_snaps scrub ok 2024-03-11T21:09:33.055 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:32 smithi023 bash[29146]: cluster 2024-03-11T21:09:31.574052+0000 osd.0 (osd.0) 293 : cluster 0 purged_snaps scrub starts 2024-03-11T21:09:33.055 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:32 smithi023 bash[29146]: cluster 2024-03-11T21:09:31.574191+0000 osd.0 (osd.0) 294 : cluster 0 purged_snaps scrub ok 2024-03-11T21:09:33.055 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:32 smithi023 bash[29146]: cluster 2024-03-11T21:09:31.698711+0000 mon.a (mon.0) 1691 : cluster 0 osdmap e343: 8 total, 8 up, 8 in 2024-03-11T21:09:33.055 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:32 smithi023 bash[29146]: cluster 2024-03-11T21:09:31.713890+0000 osd.5 (osd.5) 375 : cluster 0 3.1c scrub starts 2024-03-11T21:09:33.055 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:32 smithi023 bash[29146]: cluster 2024-03-11T21:09:31.716285+0000 osd.5 (osd.5) 376 : cluster 0 3.1c scrub ok 2024-03-11T21:09:33.055 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:32 smithi023 bash[29146]: cluster 2024-03-11T21:09:32.098185+0000 osd.0 (osd.0) 295 : cluster 0 10.4 scrub starts 2024-03-11T21:09:33.055 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:32 smithi023 bash[29146]: cluster 2024-03-11T21:09:32.302113+0000 mgr.y (mgr.24380) 694 : cluster 0 pgmap v784: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 761 op/s 2024-03-11T21:09:33.055 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:32 smithi023 bash[29146]: cluster 2024-03-11T21:09:32.382348+0000 osd.7 (osd.7) 425 : cluster 0 3.14 scrub starts 2024-03-11T21:09:33.056 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:32 smithi023 bash[29146]: cluster 2024-03-11T21:09:32.383957+0000 osd.7 (osd.7) 426 : cluster 0 3.14 scrub ok 2024-03-11T21:09:33.162 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:09:33.194 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:09:33.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:33.198 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:09:33.299 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:09:33.325 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:09:33.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:33.327 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:09:33.428 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:09:33.458 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:09:33.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:33.460 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:09:33.560 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:33.588 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:33.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:33.591 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:09:33.692 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:09:33.733 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:09:33.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:33.735 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:09:33.836 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:33.868 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:33.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:33.870 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:09:33.897 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-03-11T21:09:33.897 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:09:33.972 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:09:33.981 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:33 smithi146 bash[23809]: cluster 2024-03-11T21:09:32.310161+0000 osd.1 (osd.1) 181 : cluster 0 2.9 scrub starts 2024-03-11T21:09:33.982 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:33 smithi146 bash[23809]: cluster 2024-03-11T21:09:32.311700+0000 osd.1 (osd.1) 182 : cluster 0 2.9 scrub ok 2024-03-11T21:09:33.982 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:33 smithi146 bash[23809]: cluster 2024-03-11T21:09:32.538565+0000 osd.6 (osd.6) 304 : cluster 0 4.13 scrub starts 2024-03-11T21:09:33.982 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:33 smithi146 bash[23809]: cluster 2024-03-11T21:09:32.540821+0000 osd.6 (osd.6) 305 : cluster 0 4.13 scrub ok 2024-03-11T21:09:33.982 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:33 smithi146 bash[23809]: cluster 2024-03-11T21:09:32.733500+0000 osd.5 (osd.5) 377 : cluster 0 6.7 scrub starts 2024-03-11T21:09:33.982 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:33 smithi146 bash[23809]: cluster 2024-03-11T21:09:32.825157+0000 osd.5 (osd.5) 378 : cluster 0 6.7 scrub ok 2024-03-11T21:09:33.996 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:09:33.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:33.999 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:09:34.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:33 smithi023 bash[21372]: cluster 2024-03-11T21:09:32.310161+0000 osd.1 (osd.1) 181 : cluster 0 2.9 scrub starts 2024-03-11T21:09:34.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:33 smithi023 bash[21372]: cluster 2024-03-11T21:09:32.311700+0000 osd.1 (osd.1) 182 : cluster 0 2.9 scrub ok 2024-03-11T21:09:34.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:33 smithi023 bash[21372]: cluster 2024-03-11T21:09:32.538565+0000 osd.6 (osd.6) 304 : cluster 0 4.13 scrub starts 2024-03-11T21:09:34.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:33 smithi023 bash[21372]: cluster 2024-03-11T21:09:32.540821+0000 osd.6 (osd.6) 305 : cluster 0 4.13 scrub ok 2024-03-11T21:09:34.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:33 smithi023 bash[21372]: cluster 2024-03-11T21:09:32.733500+0000 osd.5 (osd.5) 377 : cluster 0 6.7 scrub starts 2024-03-11T21:09:34.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:33 smithi023 bash[21372]: cluster 2024-03-11T21:09:32.825157+0000 osd.5 (osd.5) 378 : cluster 0 6.7 scrub ok 2024-03-11T21:09:34.029 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:33 smithi023 bash[29146]: cluster 2024-03-11T21:09:32.310161+0000 osd.1 (osd.1) 181 : cluster 0 2.9 scrub starts 2024-03-11T21:09:34.029 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:33 smithi023 bash[29146]: cluster 2024-03-11T21:09:32.311700+0000 osd.1 (osd.1) 182 : cluster 0 2.9 scrub ok 2024-03-11T21:09:34.029 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:33 smithi023 bash[29146]: cluster 2024-03-11T21:09:32.538565+0000 osd.6 (osd.6) 304 : cluster 0 4.13 scrub starts 2024-03-11T21:09:34.029 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:33 smithi023 bash[29146]: cluster 2024-03-11T21:09:32.540821+0000 osd.6 (osd.6) 305 : cluster 0 4.13 scrub ok 2024-03-11T21:09:34.030 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:33 smithi023 bash[29146]: cluster 2024-03-11T21:09:32.733500+0000 osd.5 (osd.5) 377 : cluster 0 6.7 scrub starts 2024-03-11T21:09:34.030 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:33 smithi023 bash[29146]: cluster 2024-03-11T21:09:32.825157+0000 osd.5 (osd.5) 378 : cluster 0 6.7 scrub ok 2024-03-11T21:09:34.099 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:34.124 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:34.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:34.126 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:09:34.227 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:09:34.256 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:09:34.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:34.260 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:09:34.360 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:09:34.616 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:09:34.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:34.623 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:09:34.723 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:09:34.752 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:09:34.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:34.754 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:09:34.855 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:09:34.865 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:34 smithi023 bash[21372]: cluster 2024-03-11T21:09:33.318075+0000 osd.1 (osd.1) 183 : cluster 0 3.4 scrub starts 2024-03-11T21:09:34.866 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:34 smithi023 bash[21372]: cluster 2024-03-11T21:09:33.321336+0000 osd.1 (osd.1) 184 : cluster 0 3.4 scrub ok 2024-03-11T21:09:34.866 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:34 smithi023 bash[21372]: cluster 2024-03-11T21:09:33.761069+0000 osd.5 (osd.5) 379 : cluster 0 2.c scrub starts 2024-03-11T21:09:34.866 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:34 smithi023 bash[21372]: cluster 2024-03-11T21:09:33.763590+0000 osd.5 (osd.5) 380 : cluster 0 2.c scrub ok 2024-03-11T21:09:34.866 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:34 smithi023 bash[21372]: cluster 2024-03-11T21:09:34.130389+0000 osd.0 (osd.0) 296 : cluster 0 10.4 scrub ok 2024-03-11T21:09:34.866 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:34 smithi023 bash[21372]: cluster 2024-03-11T21:09:34.304939+0000 mgr.y (mgr.24380) 695 : cluster 0 pgmap v785: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 640 B/s rd, 0 B/s wr, 571 op/s 2024-03-11T21:09:34.867 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:34 smithi023 bash[29146]: cluster 2024-03-11T21:09:33.318075+0000 osd.1 (osd.1) 183 : cluster 0 3.4 scrub starts 2024-03-11T21:09:34.867 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:34 smithi023 bash[29146]: cluster 2024-03-11T21:09:33.321336+0000 osd.1 (osd.1) 184 : cluster 0 3.4 scrub ok 2024-03-11T21:09:34.867 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:34 smithi023 bash[29146]: cluster 2024-03-11T21:09:33.761069+0000 osd.5 (osd.5) 379 : cluster 0 2.c scrub starts 2024-03-11T21:09:34.867 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:34 smithi023 bash[29146]: cluster 2024-03-11T21:09:33.763590+0000 osd.5 (osd.5) 380 : cluster 0 2.c scrub ok 2024-03-11T21:09:34.867 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:34 smithi023 bash[29146]: cluster 2024-03-11T21:09:34.130389+0000 osd.0 (osd.0) 296 : cluster 0 10.4 scrub ok 2024-03-11T21:09:34.867 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:34 smithi023 bash[29146]: cluster 2024-03-11T21:09:34.304939+0000 mgr.y (mgr.24380) 695 : cluster 0 pgmap v785: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 640 B/s rd, 0 B/s wr, 571 op/s 2024-03-11T21:09:34.892 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:09:34.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:34.895 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:09:34.997 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:35.006 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:34 smithi146 bash[23809]: cluster 2024-03-11T21:09:33.318075+0000 osd.1 (osd.1) 183 : cluster 0 3.4 scrub starts 2024-03-11T21:09:35.006 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:34 smithi146 bash[23809]: cluster 2024-03-11T21:09:33.321336+0000 osd.1 (osd.1) 184 : cluster 0 3.4 scrub ok 2024-03-11T21:09:35.007 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:34 smithi146 bash[23809]: cluster 2024-03-11T21:09:33.761069+0000 osd.5 (osd.5) 379 : cluster 0 2.c scrub starts 2024-03-11T21:09:35.007 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:34 smithi146 bash[23809]: cluster 2024-03-11T21:09:33.763590+0000 osd.5 (osd.5) 380 : cluster 0 2.c scrub ok 2024-03-11T21:09:35.007 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:34 smithi146 bash[23809]: cluster 2024-03-11T21:09:34.130389+0000 osd.0 (osd.0) 296 : cluster 0 10.4 scrub ok 2024-03-11T21:09:35.007 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:34 smithi146 bash[23809]: cluster 2024-03-11T21:09:34.304939+0000 mgr.y (mgr.24380) 695 : cluster 0 pgmap v785: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 640 B/s rd, 0 B/s wr, 571 op/s 2024-03-11T21:09:35.029 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:35.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:35.030 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:09:35.124 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-03-11T21:09:35.131 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:09:35.195 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:09:35.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:35.197 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:09:35.297 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:09:35.339 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:09:35.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:35.344 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:09:35.445 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:09:35.477 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:09:35.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:35.479 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:09:35.579 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:09:35.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:35.673 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:09:35.673 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:09:35.774 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:35.795 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:35.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:35.797 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:09:35.898 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:09:36.008 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:09:35 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:09:36.008 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:35 smithi146 bash[23809]: cluster 2024-03-11T21:09:34.552983+0000 osd.6 (osd.6) 306 : cluster 0 3.10 scrub starts 2024-03-11T21:09:36.008 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:35 smithi146 bash[23809]: cluster 2024-03-11T21:09:34.557840+0000 osd.6 (osd.6) 307 : cluster 0 3.10 scrub ok 2024-03-11T21:09:36.009 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:35 smithi146 bash[23809]: cluster 2024-03-11T21:09:34.733193+0000 osd.5 (osd.5) 381 : cluster 0 2.12 scrub starts 2024-03-11T21:09:36.009 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:35 smithi146 bash[23809]: cluster 2024-03-11T21:09:34.741843+0000 osd.5 (osd.5) 382 : cluster 0 2.12 scrub ok 2024-03-11T21:09:36.009 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:35 smithi146 bash[23809]: cluster 2024-03-11T21:09:34.947619+0000 osd.3 (osd.3) 388 : cluster 0 6.2 scrub starts 2024-03-11T21:09:36.009 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:35 smithi146 bash[23809]: cluster 2024-03-11T21:09:34.951010+0000 osd.3 (osd.3) 389 : cluster 0 6.2 scrub ok 2024-03-11T21:09:36.009 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:35 smithi146 bash[23809]: cluster 2024-03-11T21:09:35.317117+0000 mon.a (mon.0) 1692 : cluster 0 osdmap e344: 8 total, 8 up, 8 in 2024-03-11T21:09:36.009 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:35 smithi146 bash[23809]: cluster 2024-03-11T21:09:35.467761+0000 osd.7 (osd.7) 427 : cluster 0 6.1c scrub starts 2024-03-11T21:09:36.009 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:35 smithi146 bash[23809]: cluster 2024-03-11T21:09:35.469564+0000 osd.7 (osd.7) 428 : cluster 0 6.1c scrub ok 2024-03-11T21:09:36.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:35 smithi023 bash[21372]: cluster 2024-03-11T21:09:34.552983+0000 osd.6 (osd.6) 306 : cluster 0 3.10 scrub starts 2024-03-11T21:09:36.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:35 smithi023 bash[21372]: cluster 2024-03-11T21:09:34.557840+0000 osd.6 (osd.6) 307 : cluster 0 3.10 scrub ok 2024-03-11T21:09:36.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:35 smithi023 bash[21372]: cluster 2024-03-11T21:09:34.733193+0000 osd.5 (osd.5) 381 : cluster 0 2.12 scrub starts 2024-03-11T21:09:36.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:35 smithi023 bash[21372]: cluster 2024-03-11T21:09:34.741843+0000 osd.5 (osd.5) 382 : cluster 0 2.12 scrub ok 2024-03-11T21:09:36.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:35 smithi023 bash[21372]: cluster 2024-03-11T21:09:34.947619+0000 osd.3 (osd.3) 388 : cluster 0 6.2 scrub starts 2024-03-11T21:09:36.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:35 smithi023 bash[21372]: cluster 2024-03-11T21:09:34.951010+0000 osd.3 (osd.3) 389 : cluster 0 6.2 scrub ok 2024-03-11T21:09:36.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:35 smithi023 bash[21372]: cluster 2024-03-11T21:09:35.317117+0000 mon.a (mon.0) 1692 : cluster 0 osdmap e344: 8 total, 8 up, 8 in 2024-03-11T21:09:36.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:35 smithi023 bash[21372]: cluster 2024-03-11T21:09:35.467761+0000 osd.7 (osd.7) 427 : cluster 0 6.1c scrub starts 2024-03-11T21:09:36.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:35 smithi023 bash[21372]: cluster 2024-03-11T21:09:35.469564+0000 osd.7 (osd.7) 428 : cluster 0 6.1c scrub ok 2024-03-11T21:09:36.029 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:35 smithi023 bash[29146]: cluster 2024-03-11T21:09:34.552983+0000 osd.6 (osd.6) 306 : cluster 0 3.10 scrub starts 2024-03-11T21:09:36.029 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:35 smithi023 bash[29146]: cluster 2024-03-11T21:09:34.557840+0000 osd.6 (osd.6) 307 : cluster 0 3.10 scrub ok 2024-03-11T21:09:36.029 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:35 smithi023 bash[29146]: cluster 2024-03-11T21:09:34.733193+0000 osd.5 (osd.5) 381 : cluster 0 2.12 scrub starts 2024-03-11T21:09:36.030 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:35 smithi023 bash[29146]: cluster 2024-03-11T21:09:34.741843+0000 osd.5 (osd.5) 382 : cluster 0 2.12 scrub ok 2024-03-11T21:09:36.030 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:35 smithi023 bash[29146]: cluster 2024-03-11T21:09:34.947619+0000 osd.3 (osd.3) 388 : cluster 0 6.2 scrub starts 2024-03-11T21:09:36.030 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:35 smithi023 bash[29146]: cluster 2024-03-11T21:09:34.951010+0000 osd.3 (osd.3) 389 : cluster 0 6.2 scrub ok 2024-03-11T21:09:36.030 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:35 smithi023 bash[29146]: cluster 2024-03-11T21:09:35.317117+0000 mon.a (mon.0) 1692 : cluster 0 osdmap e344: 8 total, 8 up, 8 in 2024-03-11T21:09:36.030 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:35 smithi023 bash[29146]: cluster 2024-03-11T21:09:35.467761+0000 osd.7 (osd.7) 427 : cluster 0 6.1c scrub starts 2024-03-11T21:09:36.030 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:35 smithi023 bash[29146]: cluster 2024-03-11T21:09:35.469564+0000 osd.7 (osd.7) 428 : cluster 0 6.1c scrub ok 2024-03-11T21:09:36.502 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:09:36.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:36.504 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:09:36.605 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:36.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:36.635 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:36.636 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:09:36.736 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:09:36.768 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:09:36.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:36.772 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:09:36.873 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:36.882 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:36 smithi146 bash[23809]: cluster 2024-03-11T21:09:35.256093+0000 osd.0 (osd.0) 297 : cluster 0 3.3 scrub starts 2024-03-11T21:09:36.882 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:36 smithi146 bash[23809]: cluster 2024-03-11T21:09:35.261968+0000 osd.0 (osd.0) 298 : cluster 0 3.3 scrub ok 2024-03-11T21:09:36.882 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:36 smithi146 bash[23809]: cluster 2024-03-11T21:09:35.733276+0000 osd.5 (osd.5) 383 : cluster 0 4.1d scrub starts 2024-03-11T21:09:36.882 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:36 smithi146 bash[23809]: cluster 2024-03-11T21:09:35.734331+0000 osd.5 (osd.5) 384 : cluster 0 4.1d scrub ok 2024-03-11T21:09:36.882 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:36 smithi146 bash[23809]: audit 2024-03-11T21:09:35.788039+0000 mgr.y (mgr.24380) 696 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:09:36.882 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:36 smithi146 bash[23809]: audit 2024-03-11T21:09:35.889559+0000 mon.a (mon.0) 1693 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:09:36.883 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:36 smithi146 bash[23809]: cluster 2024-03-11T21:09:35.934259+0000 osd.3 (osd.3) 390 : cluster 0 5.1 scrub starts 2024-03-11T21:09:36.883 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:36 smithi146 bash[23809]: cluster 2024-03-11T21:09:35.935553+0000 osd.3 (osd.3) 391 : cluster 0 5.1 scrub ok 2024-03-11T21:09:36.883 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:36 smithi146 bash[23809]: cluster 2024-03-11T21:09:36.305505+0000 mgr.y (mgr.24380) 697 : cluster 0 pgmap v787: 148 pgs: 148 active+clean; 5.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.24k op/s 2024-03-11T21:09:36.895 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:36.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:36.896 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:09:36.997 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:09:37.007 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:36 smithi023 bash[29146]: cluster 2024-03-11T21:09:35.256093+0000 osd.0 (osd.0) 297 : cluster 0 3.3 scrub starts 2024-03-11T21:09:37.007 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:36 smithi023 bash[29146]: cluster 2024-03-11T21:09:35.261968+0000 osd.0 (osd.0) 298 : cluster 0 3.3 scrub ok 2024-03-11T21:09:37.008 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:36 smithi023 bash[29146]: cluster 2024-03-11T21:09:35.733276+0000 osd.5 (osd.5) 383 : cluster 0 4.1d scrub starts 2024-03-11T21:09:37.008 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:36 smithi023 bash[29146]: cluster 2024-03-11T21:09:35.734331+0000 osd.5 (osd.5) 384 : cluster 0 4.1d scrub ok 2024-03-11T21:09:37.008 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:36 smithi023 bash[29146]: audit 2024-03-11T21:09:35.788039+0000 mgr.y (mgr.24380) 696 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:09:37.008 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:36 smithi023 bash[29146]: audit 2024-03-11T21:09:35.889559+0000 mon.a (mon.0) 1693 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:09:37.008 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:36 smithi023 bash[29146]: cluster 2024-03-11T21:09:35.934259+0000 osd.3 (osd.3) 390 : cluster 0 5.1 scrub starts 2024-03-11T21:09:37.008 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:36 smithi023 bash[29146]: cluster 2024-03-11T21:09:35.935553+0000 osd.3 (osd.3) 391 : cluster 0 5.1 scrub ok 2024-03-11T21:09:37.008 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:36 smithi023 bash[29146]: cluster 2024-03-11T21:09:36.305505+0000 mgr.y (mgr.24380) 697 : cluster 0 pgmap v787: 148 pgs: 148 active+clean; 5.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.24k op/s 2024-03-11T21:09:37.008 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:36 smithi023 bash[21372]: cluster 2024-03-11T21:09:35.256093+0000 osd.0 (osd.0) 297 : cluster 0 3.3 scrub starts 2024-03-11T21:09:37.008 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:36 smithi023 bash[21372]: cluster 2024-03-11T21:09:35.261968+0000 osd.0 (osd.0) 298 : cluster 0 3.3 scrub ok 2024-03-11T21:09:37.008 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:36 smithi023 bash[21372]: cluster 2024-03-11T21:09:35.733276+0000 osd.5 (osd.5) 383 : cluster 0 4.1d scrub starts 2024-03-11T21:09:37.008 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:36 smithi023 bash[21372]: cluster 2024-03-11T21:09:35.734331+0000 osd.5 (osd.5) 384 : cluster 0 4.1d scrub ok 2024-03-11T21:09:37.009 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:36 smithi023 bash[21372]: audit 2024-03-11T21:09:35.788039+0000 mgr.y (mgr.24380) 696 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:09:37.009 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:36 smithi023 bash[21372]: audit 2024-03-11T21:09:35.889559+0000 mon.a (mon.0) 1693 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:09:37.009 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:36 smithi023 bash[21372]: cluster 2024-03-11T21:09:35.934259+0000 osd.3 (osd.3) 390 : cluster 0 5.1 scrub starts 2024-03-11T21:09:37.009 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:36 smithi023 bash[21372]: cluster 2024-03-11T21:09:35.935553+0000 osd.3 (osd.3) 391 : cluster 0 5.1 scrub ok 2024-03-11T21:09:37.009 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:36 smithi023 bash[21372]: cluster 2024-03-11T21:09:36.305505+0000 mgr.y (mgr.24380) 697 : cluster 0 pgmap v787: 148 pgs: 148 active+clean; 5.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.24k op/s 2024-03-11T21:09:37.025 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:09:37.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:37.030 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:09:37.130 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:09:37.157 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:09:37.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:37.163 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:09:37.263 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:09:37.292 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:09:37.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:37.297 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:09:37.398 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:09:37.432 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:09:37.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:37.435 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:09:37.537 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:09:37.568 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:09:37.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:37.570 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:09:37.671 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:09:37.698 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:09:37.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:37.705 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:09:37.806 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:09:37.815 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:37 smithi146 bash[23809]: cluster 2024-03-11T21:09:36.744040+0000 osd.5 (osd.5) 385 : cluster 0 3.c scrub starts 2024-03-11T21:09:37.815 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:37 smithi146 bash[23809]: cluster 2024-03-11T21:09:36.752167+0000 osd.5 (osd.5) 386 : cluster 0 3.c scrub ok 2024-03-11T21:09:37.815 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:37 smithi146 bash[23809]: cluster 2024-03-11T21:09:37.471400+0000 osd.7 (osd.7) 429 : cluster 0 6.14 scrub starts 2024-03-11T21:09:37.816 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:37 smithi146 bash[23809]: cluster 2024-03-11T21:09:37.476079+0000 osd.7 (osd.7) 430 : cluster 0 6.14 scrub ok 2024-03-11T21:09:37.832 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:09:37.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:37.834 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:09:37.935 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:09:37.964 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:09:37.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:37.968 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:09:38.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:37 smithi023 bash[21372]: cluster 2024-03-11T21:09:36.744040+0000 osd.5 (osd.5) 385 : cluster 0 3.c scrub starts 2024-03-11T21:09:38.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:37 smithi023 bash[21372]: cluster 2024-03-11T21:09:36.752167+0000 osd.5 (osd.5) 386 : cluster 0 3.c scrub ok 2024-03-11T21:09:38.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:37 smithi023 bash[21372]: cluster 2024-03-11T21:09:37.471400+0000 osd.7 (osd.7) 429 : cluster 0 6.14 scrub starts 2024-03-11T21:09:38.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:37 smithi023 bash[21372]: cluster 2024-03-11T21:09:37.476079+0000 osd.7 (osd.7) 430 : cluster 0 6.14 scrub ok 2024-03-11T21:09:38.029 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:37 smithi023 bash[29146]: cluster 2024-03-11T21:09:36.744040+0000 osd.5 (osd.5) 385 : cluster 0 3.c scrub starts 2024-03-11T21:09:38.029 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:37 smithi023 bash[29146]: cluster 2024-03-11T21:09:36.752167+0000 osd.5 (osd.5) 386 : cluster 0 3.c scrub ok 2024-03-11T21:09:38.029 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:37 smithi023 bash[29146]: cluster 2024-03-11T21:09:37.471400+0000 osd.7 (osd.7) 429 : cluster 0 6.14 scrub starts 2024-03-11T21:09:38.029 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:37 smithi023 bash[29146]: cluster 2024-03-11T21:09:37.476079+0000 osd.7 (osd.7) 430 : cluster 0 6.14 scrub ok 2024-03-11T21:09:38.069 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:09:38.411 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:09:38.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:38.415 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:09:38.516 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:09:38.542 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:09:38.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:38.547 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:09:38.647 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:09:38.687 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:09:38.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:38.693 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:09:38.746 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:09:38 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:09:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:09:38.793 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:09:38.802 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:38 smithi146 bash[23809]: cluster 2024-03-11T21:09:37.709804+0000 osd.5 (osd.5) 387 : cluster 0 2.16 scrub starts 2024-03-11T21:09:38.803 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:38 smithi146 bash[23809]: cluster 2024-03-11T21:09:37.712655+0000 osd.5 (osd.5) 388 : cluster 0 2.16 scrub ok 2024-03-11T21:09:38.803 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:38 smithi146 bash[23809]: cluster 2024-03-11T21:09:38.306062+0000 mgr.y (mgr.24380) 698 : cluster 0 pgmap v788: 148 pgs: 148 active+clean; 5.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 659 op/s 2024-03-11T21:09:38.803 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:38 smithi146 bash[23809]: cluster 2024-03-11T21:09:38.469412+0000 osd.7 (osd.7) 431 : cluster 0 6.15 scrub starts 2024-03-11T21:09:38.803 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:38 smithi146 bash[23809]: cluster 2024-03-11T21:09:38.526223+0000 osd.7 (osd.7) 432 : cluster 0 6.15 scrub ok 2024-03-11T21:09:38.826 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:09:38.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:38.830 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:09:38.931 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:09:38.960 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:09:38.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:38.962 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:09:39.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:38 smithi023 bash[21372]: cluster 2024-03-11T21:09:37.709804+0000 osd.5 (osd.5) 387 : cluster 0 2.16 scrub starts 2024-03-11T21:09:39.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:38 smithi023 bash[21372]: cluster 2024-03-11T21:09:37.712655+0000 osd.5 (osd.5) 388 : cluster 0 2.16 scrub ok 2024-03-11T21:09:39.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:38 smithi023 bash[21372]: cluster 2024-03-11T21:09:38.306062+0000 mgr.y (mgr.24380) 698 : cluster 0 pgmap v788: 148 pgs: 148 active+clean; 5.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 659 op/s 2024-03-11T21:09:39.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:38 smithi023 bash[21372]: cluster 2024-03-11T21:09:38.469412+0000 osd.7 (osd.7) 431 : cluster 0 6.15 scrub starts 2024-03-11T21:09:39.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:38 smithi023 bash[21372]: cluster 2024-03-11T21:09:38.526223+0000 osd.7 (osd.7) 432 : cluster 0 6.15 scrub ok 2024-03-11T21:09:39.029 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:38 smithi023 bash[29146]: cluster 2024-03-11T21:09:37.709804+0000 osd.5 (osd.5) 387 : cluster 0 2.16 scrub starts 2024-03-11T21:09:39.029 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:38 smithi023 bash[29146]: cluster 2024-03-11T21:09:37.712655+0000 osd.5 (osd.5) 388 : cluster 0 2.16 scrub ok 2024-03-11T21:09:39.029 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:38 smithi023 bash[29146]: cluster 2024-03-11T21:09:38.306062+0000 mgr.y (mgr.24380) 698 : cluster 0 pgmap v788: 148 pgs: 148 active+clean; 5.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 659 op/s 2024-03-11T21:09:39.029 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:38 smithi023 bash[29146]: cluster 2024-03-11T21:09:38.469412+0000 osd.7 (osd.7) 431 : cluster 0 6.15 scrub starts 2024-03-11T21:09:39.029 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:38 smithi023 bash[29146]: cluster 2024-03-11T21:09:38.526223+0000 osd.7 (osd.7) 432 : cluster 0 6.15 scrub ok 2024-03-11T21:09:39.064 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:09:39.093 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:09:39.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:39.094 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:09:39.195 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:09:39.230 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:09:39.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:39.233 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:09:39.334 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:09:39.377 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:09:39.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:39.382 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:09:39.483 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:09:39.778 INFO:teuthology.orchestra.run.smithi023.stdout:osd.0: {} 2024-03-11T21:09:39.778 INFO:teuthology.orchestra.run.smithi023.stderr:nodeep-scrub is unset 2024-03-11T21:09:39.779 INFO:teuthology.orchestra.run.smithi023.stdout:osd.1: {} 2024-03-11T21:09:39.779 INFO:teuthology.orchestra.run.smithi023.stdout:osd.2: {} 2024-03-11T21:09:39.779 INFO:teuthology.orchestra.run.smithi023.stdout:osd.3: {} 2024-03-11T21:09:39.800 INFO:teuthology.orchestra.run.smithi023.stdout:osd.4: {} 2024-03-11T21:09:39.839 INFO:teuthology.orchestra.run.smithi023.stdout:osd.5: {} 2024-03-11T21:09:39.868 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:09:39.881 INFO:teuthology.orchestra.run.smithi023.stdout:osd.6: {} 2024-03-11T21:09:39.913 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:09:39.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:39.916 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:09:39.925 INFO:teuthology.orchestra.run.smithi023.stdout:osd.7: {} 2024-03-11T21:09:39.935 INFO:teuthology.orchestra.run.smithi023.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:09:39.935 INFO:teuthology.orchestra.run.smithi023.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:09:39.935 INFO:teuthology.orchestra.run.smithi023.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:09:39.935 INFO:teuthology.orchestra.run.smithi023.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:09:39.935 INFO:teuthology.orchestra.run.smithi023.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:09:39.935 INFO:teuthology.orchestra.run.smithi023.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:09:39.935 INFO:teuthology.orchestra.run.smithi023.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:09:39.935 INFO:teuthology.orchestra.run.smithi023.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:09:40.017 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:09:40.025 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:39 smithi146 bash[23809]: audit 2024-03-11T21:09:39.659793+0000 mon.c (mon.2) 85 : audit 0 from='client.? 172.21.15.23:0/3078368960' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:09:40.025 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:39 smithi146 bash[23809]: audit 2024-03-11T21:09:39.723693+0000 mon.b (mon.1) 42 : audit 1 from='client.? 172.21.15.23:0/964451796' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:09:40.026 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:39 smithi146 bash[23809]: audit 2024-03-11T21:09:39.723966+0000 mon.a (mon.0) 1694 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:09:40.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:39 smithi023 bash[21372]: audit 2024-03-11T21:09:39.659793+0000 mon.c (mon.2) 85 : audit 0 from='client.? 172.21.15.23:0/3078368960' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:09:40.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:39 smithi023 bash[21372]: audit 2024-03-11T21:09:39.723693+0000 mon.b (mon.1) 42 : audit 1 from='client.? 172.21.15.23:0/964451796' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:09:40.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:39 smithi023 bash[21372]: audit 2024-03-11T21:09:39.723966+0000 mon.a (mon.0) 1694 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:09:40.030 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:39 smithi023 bash[29146]: audit 2024-03-11T21:09:39.659793+0000 mon.c (mon.2) 85 : audit 0 from='client.? 172.21.15.23:0/3078368960' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:09:40.030 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:39 smithi023 bash[29146]: audit 2024-03-11T21:09:39.723693+0000 mon.b (mon.1) 42 : audit 1 from='client.? 172.21.15.23:0/964451796' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:09:40.030 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:39 smithi023 bash[29146]: audit 2024-03-11T21:09:39.723966+0000 mon.a (mon.0) 1694 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:09:40.040 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:09:40.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:40.045 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:09:40.146 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:09:40.164 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:09:40.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:40.166 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:09:40.266 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:09:40.298 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:09:40.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:40.301 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:09:40.402 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:09:40.444 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:09:40.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:40.447 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:09:40.548 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:40.571 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:40.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:40.573 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:09:40.674 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:09:40.701 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:09:40.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:40.702 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:09:40.803 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:40.812 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:40 smithi146 bash[23809]: cluster 2024-03-11T21:09:39.203742+0000 osd.1 (osd.1) 185 : cluster 0 4.1c scrub starts 2024-03-11T21:09:40.812 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:40 smithi146 bash[23809]: cluster 2024-03-11T21:09:39.217517+0000 osd.1 (osd.1) 186 : cluster 0 4.1c scrub ok 2024-03-11T21:09:40.812 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:40 smithi146 bash[23809]: cluster 2024-03-11T21:09:39.308570+0000 osd.0 (osd.0) 299 : cluster 0 6.9 scrub starts 2024-03-11T21:09:40.812 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:40 smithi146 bash[23809]: cluster 2024-03-11T21:09:39.408067+0000 osd.0 (osd.0) 300 : cluster 0 6.9 scrub ok 2024-03-11T21:09:40.812 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:40 smithi146 bash[23809]: cluster 2024-03-11T21:09:39.755748+0000 mon.a (mon.0) 1695 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:09:40.812 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:40 smithi146 bash[23809]: cluster 2024-03-11T21:09:39.755771+0000 mon.a (mon.0) 1696 : cluster 1 Cluster is now healthy 2024-03-11T21:09:40.812 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:40 smithi146 bash[23809]: audit 2024-03-11T21:09:39.757099+0000 mon.a (mon.0) 1697 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:09:40.812 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:40 smithi146 bash[23809]: cluster 2024-03-11T21:09:39.763925+0000 mon.a (mon.0) 1698 : cluster 0 osdmap e345: 8 total, 8 up, 8 in 2024-03-11T21:09:40.812 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:40 smithi146 bash[23809]: cluster 2024-03-11T21:09:39.872211+0000 osd.3 (osd.3) 392 : cluster 0 6.6 scrub starts 2024-03-11T21:09:40.813 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:40 smithi146 bash[23809]: cluster 2024-03-11T21:09:39.887587+0000 osd.3 (osd.3) 393 : cluster 0 6.6 scrub ok 2024-03-11T21:09:40.813 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:40 smithi146 bash[23809]: cluster 2024-03-11T21:09:40.307412+0000 mgr.y (mgr.24380) 699 : cluster 0 pgmap v790: 148 pgs: 148 active+clean; 5.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 948 op/s 2024-03-11T21:09:40.813 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:40 smithi146 bash[23809]: cluster 2024-03-11T21:09:40.319315+0000 mon.a (mon.0) 1699 : cluster 0 osdmap e346: 8 total, 8 up, 8 in 2024-03-11T21:09:40.813 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:40 smithi146 bash[23809]: cluster 2024-03-11T21:09:40.483718+0000 osd.7 (osd.7) 433 : cluster 0 3.d scrub starts 2024-03-11T21:09:40.813 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:40 smithi146 bash[23809]: cluster 2024-03-11T21:09:40.486309+0000 osd.7 (osd.7) 434 : cluster 0 3.d scrub ok 2024-03-11T21:09:40.826 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:40.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:40.828 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:09:40.929 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:09:40.939 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:40 smithi023 bash[21372]: cluster 2024-03-11T21:09:39.203742+0000 osd.1 (osd.1) 185 : cluster 0 4.1c scrub starts 2024-03-11T21:09:40.939 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:40 smithi023 bash[21372]: cluster 2024-03-11T21:09:39.217517+0000 osd.1 (osd.1) 186 : cluster 0 4.1c scrub ok 2024-03-11T21:09:40.939 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:40 smithi023 bash[21372]: cluster 2024-03-11T21:09:39.308570+0000 osd.0 (osd.0) 299 : cluster 0 6.9 scrub starts 2024-03-11T21:09:40.939 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:40 smithi023 bash[21372]: cluster 2024-03-11T21:09:39.408067+0000 osd.0 (osd.0) 300 : cluster 0 6.9 scrub ok 2024-03-11T21:09:40.939 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:40 smithi023 bash[21372]: cluster 2024-03-11T21:09:39.755748+0000 mon.a (mon.0) 1695 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:09:40.939 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:40 smithi023 bash[21372]: cluster 2024-03-11T21:09:39.755771+0000 mon.a (mon.0) 1696 : cluster 1 Cluster is now healthy 2024-03-11T21:09:40.939 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:40 smithi023 bash[21372]: audit 2024-03-11T21:09:39.757099+0000 mon.a (mon.0) 1697 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:09:40.939 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:40 smithi023 bash[21372]: cluster 2024-03-11T21:09:39.763925+0000 mon.a (mon.0) 1698 : cluster 0 osdmap e345: 8 total, 8 up, 8 in 2024-03-11T21:09:40.939 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:40 smithi023 bash[21372]: cluster 2024-03-11T21:09:39.872211+0000 osd.3 (osd.3) 392 : cluster 0 6.6 scrub starts 2024-03-11T21:09:40.939 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:40 smithi023 bash[21372]: cluster 2024-03-11T21:09:39.887587+0000 osd.3 (osd.3) 393 : cluster 0 6.6 scrub ok 2024-03-11T21:09:40.939 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:40 smithi023 bash[21372]: cluster 2024-03-11T21:09:40.307412+0000 mgr.y (mgr.24380) 699 : cluster 0 pgmap v790: 148 pgs: 148 active+clean; 5.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 948 op/s 2024-03-11T21:09:40.939 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:40 smithi023 bash[21372]: cluster 2024-03-11T21:09:40.319315+0000 mon.a (mon.0) 1699 : cluster 0 osdmap e346: 8 total, 8 up, 8 in 2024-03-11T21:09:40.940 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:40 smithi023 bash[21372]: cluster 2024-03-11T21:09:40.483718+0000 osd.7 (osd.7) 433 : cluster 0 3.d scrub starts 2024-03-11T21:09:40.940 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:40 smithi023 bash[21372]: cluster 2024-03-11T21:09:40.486309+0000 osd.7 (osd.7) 434 : cluster 0 3.d scrub ok 2024-03-11T21:09:40.940 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:40 smithi023 bash[29146]: cluster 2024-03-11T21:09:39.203742+0000 osd.1 (osd.1) 185 : cluster 0 4.1c scrub starts 2024-03-11T21:09:40.940 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:40 smithi023 bash[29146]: cluster 2024-03-11T21:09:39.217517+0000 osd.1 (osd.1) 186 : cluster 0 4.1c scrub ok 2024-03-11T21:09:40.940 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:40 smithi023 bash[29146]: cluster 2024-03-11T21:09:39.308570+0000 osd.0 (osd.0) 299 : cluster 0 6.9 scrub starts 2024-03-11T21:09:40.940 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:40 smithi023 bash[29146]: cluster 2024-03-11T21:09:39.408067+0000 osd.0 (osd.0) 300 : cluster 0 6.9 scrub ok 2024-03-11T21:09:40.940 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:40 smithi023 bash[29146]: cluster 2024-03-11T21:09:39.755748+0000 mon.a (mon.0) 1695 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:09:40.940 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:40 smithi023 bash[29146]: cluster 2024-03-11T21:09:39.755771+0000 mon.a (mon.0) 1696 : cluster 1 Cluster is now healthy 2024-03-11T21:09:40.940 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:40 smithi023 bash[29146]: audit 2024-03-11T21:09:39.757099+0000 mon.a (mon.0) 1697 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:09:40.940 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:40 smithi023 bash[29146]: cluster 2024-03-11T21:09:39.763925+0000 mon.a (mon.0) 1698 : cluster 0 osdmap e345: 8 total, 8 up, 8 in 2024-03-11T21:09:40.941 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:40 smithi023 bash[29146]: cluster 2024-03-11T21:09:39.872211+0000 osd.3 (osd.3) 392 : cluster 0 6.6 scrub starts 2024-03-11T21:09:40.941 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:40 smithi023 bash[29146]: cluster 2024-03-11T21:09:39.887587+0000 osd.3 (osd.3) 393 : cluster 0 6.6 scrub ok 2024-03-11T21:09:40.941 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:40 smithi023 bash[29146]: cluster 2024-03-11T21:09:40.307412+0000 mgr.y (mgr.24380) 699 : cluster 0 pgmap v790: 148 pgs: 148 active+clean; 5.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 948 op/s 2024-03-11T21:09:40.941 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:40 smithi023 bash[29146]: cluster 2024-03-11T21:09:40.319315+0000 mon.a (mon.0) 1699 : cluster 0 osdmap e346: 8 total, 8 up, 8 in 2024-03-11T21:09:40.941 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:40 smithi023 bash[29146]: cluster 2024-03-11T21:09:40.483718+0000 osd.7 (osd.7) 433 : cluster 0 3.d scrub starts 2024-03-11T21:09:40.941 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:40 smithi023 bash[29146]: cluster 2024-03-11T21:09:40.486309+0000 osd.7 (osd.7) 434 : cluster 0 3.d scrub ok 2024-03-11T21:09:41.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:41.018 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:09:41.019 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:09:41.119 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:09:41.150 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:09:41.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:41.152 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:09:41.253 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:09:41.275 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:09:41.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:41.278 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:09:41.379 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:41.410 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:41.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:41.412 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:09:41.513 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:09:41.550 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:09:41.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:41.557 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:09:41.657 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:09:41.676 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:09:41.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:41.679 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:09:41.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:41 smithi023 bash[21372]: cluster 2024-03-11T21:09:40.198623+0000 osd.1 (osd.1) 187 : cluster 0 6.1 scrub starts 2024-03-11T21:09:41.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:41 smithi023 bash[21372]: cluster 2024-03-11T21:09:40.245930+0000 osd.1 (osd.1) 188 : cluster 0 6.1 scrub ok 2024-03-11T21:09:41.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:41 smithi023 bash[21372]: cluster 2024-03-11T21:09:40.698666+0000 osd.5 (osd.5) 389 : cluster 0 4.15 scrub starts 2024-03-11T21:09:41.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:41 smithi023 bash[21372]: cluster 2024-03-11T21:09:40.700208+0000 osd.5 (osd.5) 390 : cluster 0 4.15 scrub ok 2024-03-11T21:09:41.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:41 smithi023 bash[21372]: cluster 2024-03-11T21:09:40.871224+0000 osd.3 (osd.3) 394 : cluster 0 6.c scrub starts 2024-03-11T21:09:41.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:41 smithi023 bash[21372]: cluster 2024-03-11T21:09:40.876674+0000 osd.3 (osd.3) 395 : cluster 0 6.c scrub ok 2024-03-11T21:09:41.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:41 smithi023 bash[21372]: audit 2024-03-11T21:09:40.960235+0000 mon.a (mon.0) 1700 : audit 1 from='client.? 172.21.15.23:0/4102935051' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "6.a", "id": [3, 5, 2, 7, 6, 1]}]: dispatch 2024-03-11T21:09:41.779 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:09:41.801 INFO:teuthology.orchestra.run.smithi023.stderr:set 6.a pg_upmap_items mapping to [3->5,2->7,6->1] 2024-03-11T21:09:41.809 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:09:41.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:41.815 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:09:41.916 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:41.943 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:41.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:41.953 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:09:42.028 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:41 smithi023 bash[29146]: cluster 2024-03-11T21:09:40.198623+0000 osd.1 (osd.1) 187 : cluster 0 6.1 scrub starts 2024-03-11T21:09:42.028 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:41 smithi023 bash[29146]: cluster 2024-03-11T21:09:40.245930+0000 osd.1 (osd.1) 188 : cluster 0 6.1 scrub ok 2024-03-11T21:09:42.028 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:41 smithi023 bash[29146]: cluster 2024-03-11T21:09:40.698666+0000 osd.5 (osd.5) 389 : cluster 0 4.15 scrub starts 2024-03-11T21:09:42.028 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:41 smithi023 bash[29146]: cluster 2024-03-11T21:09:40.700208+0000 osd.5 (osd.5) 390 : cluster 0 4.15 scrub ok 2024-03-11T21:09:42.028 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:41 smithi023 bash[29146]: cluster 2024-03-11T21:09:40.871224+0000 osd.3 (osd.3) 394 : cluster 0 6.c scrub starts 2024-03-11T21:09:42.028 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:41 smithi023 bash[29146]: cluster 2024-03-11T21:09:40.876674+0000 osd.3 (osd.3) 395 : cluster 0 6.c scrub ok 2024-03-11T21:09:42.028 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:41 smithi023 bash[29146]: audit 2024-03-11T21:09:40.960235+0000 mon.a (mon.0) 1700 : audit 1 from='client.? 172.21.15.23:0/4102935051' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "6.a", "id": [3, 5, 2, 7, 6, 1]}]: dispatch 2024-03-11T21:09:42.053 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:09:42.077 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:09:42.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:42.081 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:09:42.182 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:09:42.205 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:09:42.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:42.207 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:09:42.258 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:41 smithi146 bash[23809]: cluster 2024-03-11T21:09:40.198623+0000 osd.1 (osd.1) 187 : cluster 0 6.1 scrub starts 2024-03-11T21:09:42.258 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:41 smithi146 bash[23809]: cluster 2024-03-11T21:09:40.245930+0000 osd.1 (osd.1) 188 : cluster 0 6.1 scrub ok 2024-03-11T21:09:42.258 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:41 smithi146 bash[23809]: cluster 2024-03-11T21:09:40.698666+0000 osd.5 (osd.5) 389 : cluster 0 4.15 scrub starts 2024-03-11T21:09:42.258 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:41 smithi146 bash[23809]: cluster 2024-03-11T21:09:40.700208+0000 osd.5 (osd.5) 390 : cluster 0 4.15 scrub ok 2024-03-11T21:09:42.259 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:41 smithi146 bash[23809]: cluster 2024-03-11T21:09:40.871224+0000 osd.3 (osd.3) 394 : cluster 0 6.c scrub starts 2024-03-11T21:09:42.259 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:41 smithi146 bash[23809]: cluster 2024-03-11T21:09:40.876674+0000 osd.3 (osd.3) 395 : cluster 0 6.c scrub ok 2024-03-11T21:09:42.259 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:41 smithi146 bash[23809]: audit 2024-03-11T21:09:40.960235+0000 mon.a (mon.0) 1700 : audit 1 from='client.? 172.21.15.23:0/4102935051' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "6.a", "id": [3, 5, 2, 7, 6, 1]}]: dispatch 2024-03-11T21:09:42.308 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:42.334 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:42.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:42.340 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:09:42.440 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:09:42.475 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:09:42.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:42.477 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:09:42.578 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:09:42.608 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:09:42.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:42.611 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:09:42.712 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:09:42.740 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:09:42.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:42.743 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:09:42.844 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:09:42.872 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:09:42.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:42.874 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:09:42.970 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:42 smithi023 bash[21372]: cluster 2024-03-11T21:09:41.239941+0000 osd.1 (osd.1) 189 : cluster 0 2.4 scrub starts 2024-03-11T21:09:42.970 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:42 smithi023 bash[21372]: cluster 2024-03-11T21:09:41.244058+0000 osd.1 (osd.1) 190 : cluster 0 2.4 scrub ok 2024-03-11T21:09:42.970 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:42 smithi023 bash[21372]: cluster 2024-03-11T21:09:41.529527+0000 osd.6 (osd.6) 308 : cluster 0 2.11 scrub starts 2024-03-11T21:09:42.971 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:42 smithi023 bash[21372]: cluster 2024-03-11T21:09:41.540180+0000 osd.6 (osd.6) 309 : cluster 0 2.11 scrub ok 2024-03-11T21:09:42.971 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:42 smithi023 bash[21372]: cluster 2024-03-11T21:09:41.726989+0000 osd.5 (osd.5) 391 : cluster 0 6.1a scrub starts 2024-03-11T21:09:42.971 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:42 smithi023 bash[21372]: cluster 2024-03-11T21:09:41.738646+0000 osd.5 (osd.5) 392 : cluster 0 6.1a scrub ok 2024-03-11T21:09:42.971 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:42 smithi023 bash[21372]: audit 2024-03-11T21:09:41.804255+0000 mon.a (mon.0) 1701 : audit 1 from='client.? 172.21.15.23:0/4102935051' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "6.a", "id": [3, 5, 2, 7, 6, 1]}]': finished 2024-03-11T21:09:42.971 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:42 smithi023 bash[21372]: cluster 2024-03-11T21:09:41.811931+0000 mon.a (mon.0) 1702 : cluster 0 osdmap e347: 8 total, 8 up, 8 in 2024-03-11T21:09:42.971 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:42 smithi023 bash[21372]: cluster 2024-03-11T21:09:42.307978+0000 mgr.y (mgr.24380) 700 : cluster 0 pgmap v793: 148 pgs: 148 active+clean; 4.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 796 op/s 2024-03-11T21:09:42.971 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:42 smithi023 bash[29146]: cluster 2024-03-11T21:09:41.239941+0000 osd.1 (osd.1) 189 : cluster 0 2.4 scrub starts 2024-03-11T21:09:42.971 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:42 smithi023 bash[29146]: cluster 2024-03-11T21:09:41.244058+0000 osd.1 (osd.1) 190 : cluster 0 2.4 scrub ok 2024-03-11T21:09:42.971 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:42 smithi023 bash[29146]: cluster 2024-03-11T21:09:41.529527+0000 osd.6 (osd.6) 308 : cluster 0 2.11 scrub starts 2024-03-11T21:09:42.972 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:42 smithi023 bash[29146]: cluster 2024-03-11T21:09:41.540180+0000 osd.6 (osd.6) 309 : cluster 0 2.11 scrub ok 2024-03-11T21:09:42.972 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:42 smithi023 bash[29146]: cluster 2024-03-11T21:09:41.726989+0000 osd.5 (osd.5) 391 : cluster 0 6.1a scrub starts 2024-03-11T21:09:42.972 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:42 smithi023 bash[29146]: cluster 2024-03-11T21:09:41.738646+0000 osd.5 (osd.5) 392 : cluster 0 6.1a scrub ok 2024-03-11T21:09:42.972 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:42 smithi023 bash[29146]: audit 2024-03-11T21:09:41.804255+0000 mon.a (mon.0) 1701 : audit 1 from='client.? 172.21.15.23:0/4102935051' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "6.a", "id": [3, 5, 2, 7, 6, 1]}]': finished 2024-03-11T21:09:42.972 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:42 smithi023 bash[29146]: cluster 2024-03-11T21:09:41.811931+0000 mon.a (mon.0) 1702 : cluster 0 osdmap e347: 8 total, 8 up, 8 in 2024-03-11T21:09:42.972 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:42 smithi023 bash[29146]: cluster 2024-03-11T21:09:42.307978+0000 mgr.y (mgr.24380) 700 : cluster 0 pgmap v793: 148 pgs: 148 active+clean; 4.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 796 op/s 2024-03-11T21:09:42.975 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:09:43.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:43.002 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:09:43.002 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:09:43.103 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:09:43.137 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:09:43.138 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:42 smithi146 bash[23809]: cluster 2024-03-11T21:09:41.239941+0000 osd.1 (osd.1) 189 : cluster 0 2.4 scrub starts 2024-03-11T21:09:43.139 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:42 smithi146 bash[23809]: cluster 2024-03-11T21:09:41.244058+0000 osd.1 (osd.1) 190 : cluster 0 2.4 scrub ok 2024-03-11T21:09:43.139 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:42 smithi146 bash[23809]: cluster 2024-03-11T21:09:41.529527+0000 osd.6 (osd.6) 308 : cluster 0 2.11 scrub starts 2024-03-11T21:09:43.139 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:42 smithi146 bash[23809]: cluster 2024-03-11T21:09:41.540180+0000 osd.6 (osd.6) 309 : cluster 0 2.11 scrub ok 2024-03-11T21:09:43.139 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:42 smithi146 bash[23809]: cluster 2024-03-11T21:09:41.726989+0000 osd.5 (osd.5) 391 : cluster 0 6.1a scrub starts 2024-03-11T21:09:43.139 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:42 smithi146 bash[23809]: cluster 2024-03-11T21:09:41.738646+0000 osd.5 (osd.5) 392 : cluster 0 6.1a scrub ok 2024-03-11T21:09:43.139 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:42 smithi146 bash[23809]: audit 2024-03-11T21:09:41.804255+0000 mon.a (mon.0) 1701 : audit 1 from='client.? 172.21.15.23:0/4102935051' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "6.a", "id": [3, 5, 2, 7, 6, 1]}]': finished 2024-03-11T21:09:43.139 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:42 smithi146 bash[23809]: cluster 2024-03-11T21:09:41.811931+0000 mon.a (mon.0) 1702 : cluster 0 osdmap e347: 8 total, 8 up, 8 in 2024-03-11T21:09:43.139 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:42 smithi146 bash[23809]: cluster 2024-03-11T21:09:42.307978+0000 mgr.y (mgr.24380) 700 : cluster 0 pgmap v793: 148 pgs: 148 active+clean; 4.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 796 op/s 2024-03-11T21:09:43.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:43.140 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:09:43.241 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:43.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:43.270 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:43.270 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:09:43.371 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:43.396 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:43.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:43.402 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:09:43.503 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:09:43.523 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:09:43.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:43.526 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:09:43.627 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:09:43.650 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:09:43.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:43.656 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:09:43.757 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:09:43.785 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:09:43.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:43.788 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:09:43.889 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:43.901 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:43 smithi146 bash[23809]: cluster 2024-03-11T21:09:42.229795+0000 osd.1 (osd.1) 191 : cluster 0 4.2 scrub starts 2024-03-11T21:09:43.902 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:43 smithi146 bash[23809]: cluster 2024-03-11T21:09:42.231190+0000 osd.1 (osd.1) 192 : cluster 0 4.2 scrub ok 2024-03-11T21:09:43.902 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:43 smithi146 bash[23809]: cluster 2024-03-11T21:09:42.584722+0000 osd.6 (osd.6) 310 : cluster 0 4.a scrub starts 2024-03-11T21:09:43.902 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:43 smithi146 bash[23809]: cluster 2024-03-11T21:09:42.596106+0000 osd.6 (osd.6) 311 : cluster 0 4.a scrub ok 2024-03-11T21:09:43.902 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:43 smithi146 bash[23809]: cluster 2024-03-11T21:09:43.237477+0000 osd.1 (osd.1) 193 : cluster 0 3.1a scrub starts 2024-03-11T21:09:43.902 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:43 smithi146 bash[23809]: cluster 2024-03-11T21:09:43.241392+0000 osd.1 (osd.1) 194 : cluster 0 3.1a scrub ok 2024-03-11T21:09:43.902 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:43 smithi146 bash[23809]: cluster 2024-03-11T21:09:43.383242+0000 osd.7 (osd.7) 435 : cluster 0 2.b scrub starts 2024-03-11T21:09:43.902 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:43 smithi146 bash[23809]: cluster 2024-03-11T21:09:43.387703+0000 osd.7 (osd.7) 436 : cluster 0 2.b scrub ok 2024-03-11T21:09:43.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:43.916 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:43.916 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:09:44.017 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:09:44.027 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:43 smithi023 bash[21372]: cluster 2024-03-11T21:09:42.229795+0000 osd.1 (osd.1) 191 : cluster 0 4.2 scrub starts 2024-03-11T21:09:44.027 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:43 smithi023 bash[21372]: cluster 2024-03-11T21:09:42.231190+0000 osd.1 (osd.1) 192 : cluster 0 4.2 scrub ok 2024-03-11T21:09:44.027 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:43 smithi023 bash[21372]: cluster 2024-03-11T21:09:42.584722+0000 osd.6 (osd.6) 310 : cluster 0 4.a scrub starts 2024-03-11T21:09:44.027 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:43 smithi023 bash[21372]: cluster 2024-03-11T21:09:42.596106+0000 osd.6 (osd.6) 311 : cluster 0 4.a scrub ok 2024-03-11T21:09:44.027 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:43 smithi023 bash[21372]: cluster 2024-03-11T21:09:43.237477+0000 osd.1 (osd.1) 193 : cluster 0 3.1a scrub starts 2024-03-11T21:09:44.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:43 smithi023 bash[21372]: cluster 2024-03-11T21:09:43.241392+0000 osd.1 (osd.1) 194 : cluster 0 3.1a scrub ok 2024-03-11T21:09:44.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:43 smithi023 bash[21372]: cluster 2024-03-11T21:09:43.383242+0000 osd.7 (osd.7) 435 : cluster 0 2.b scrub starts 2024-03-11T21:09:44.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:43 smithi023 bash[21372]: cluster 2024-03-11T21:09:43.387703+0000 osd.7 (osd.7) 436 : cluster 0 2.b scrub ok 2024-03-11T21:09:44.028 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:43 smithi023 bash[29146]: cluster 2024-03-11T21:09:42.229795+0000 osd.1 (osd.1) 191 : cluster 0 4.2 scrub starts 2024-03-11T21:09:44.028 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:43 smithi023 bash[29146]: cluster 2024-03-11T21:09:42.231190+0000 osd.1 (osd.1) 192 : cluster 0 4.2 scrub ok 2024-03-11T21:09:44.028 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:43 smithi023 bash[29146]: cluster 2024-03-11T21:09:42.584722+0000 osd.6 (osd.6) 310 : cluster 0 4.a scrub starts 2024-03-11T21:09:44.028 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:43 smithi023 bash[29146]: cluster 2024-03-11T21:09:42.596106+0000 osd.6 (osd.6) 311 : cluster 0 4.a scrub ok 2024-03-11T21:09:44.028 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:43 smithi023 bash[29146]: cluster 2024-03-11T21:09:43.237477+0000 osd.1 (osd.1) 193 : cluster 0 3.1a scrub starts 2024-03-11T21:09:44.028 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:43 smithi023 bash[29146]: cluster 2024-03-11T21:09:43.241392+0000 osd.1 (osd.1) 194 : cluster 0 3.1a scrub ok 2024-03-11T21:09:44.028 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:43 smithi023 bash[29146]: cluster 2024-03-11T21:09:43.383242+0000 osd.7 (osd.7) 435 : cluster 0 2.b scrub starts 2024-03-11T21:09:44.029 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:43 smithi023 bash[29146]: cluster 2024-03-11T21:09:43.387703+0000 osd.7 (osd.7) 436 : cluster 0 2.b scrub ok 2024-03-11T21:09:44.051 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:09:44.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:44.054 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:09:44.155 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:09:44.189 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:09:44.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:44.193 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:09:44.293 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:09:44.319 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:09:44.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:44.324 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:09:44.425 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:09:44.454 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:09:44.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:44.460 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:09:44.561 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:09:44.596 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:09:44.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:44.598 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:09:44.699 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:09:44.730 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:09:44.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:44.736 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:09:44.836 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:44.864 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:44.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:44.867 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:09:44.968 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:09:44.973 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T21:09:44.993 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:09:44.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:44.996 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:09:45.026 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd set noscrub 2024-03-11T21:09:45.097 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:09:45.111 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:44 smithi023 bash[21372]: cluster 2024-03-11T21:09:44.308392+0000 mgr.y (mgr.24380) 701 : cluster 0 pgmap v794: 148 pgs: 148 active+clean; 4.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 749 op/s 2024-03-11T21:09:45.111 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:44 smithi023 bash[29146]: cluster 2024-03-11T21:09:44.308392+0000 mgr.y (mgr.24380) 701 : cluster 0 pgmap v794: 148 pgs: 148 active+clean; 4.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 749 op/s 2024-03-11T21:09:45.139 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:09:45.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:45.144 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:09:45.211 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:44 smithi146 bash[23809]: cluster 2024-03-11T21:09:44.308392+0000 mgr.y (mgr.24380) 701 : cluster 0 pgmap v794: 148 pgs: 148 active+clean; 4.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 749 op/s 2024-03-11T21:09:45.245 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:09:45.279 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:09:45.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:45.281 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:09:45.382 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:45.415 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:45.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:45.418 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:09:45.519 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:45.549 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:45.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:45.551 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:09:45.652 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:09:45.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:45.686 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:09:45.686 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:09:45.787 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:09:45.812 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:09:45.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:45.815 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:09:45.917 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:09:45.928 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:45 smithi023 bash[21372]: cluster 2024-03-11T21:09:44.521029+0000 osd.6 (osd.6) 312 : cluster 0 2.1a scrub starts 2024-03-11T21:09:45.928 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:45 smithi023 bash[21372]: cluster 2024-03-11T21:09:44.522795+0000 osd.6 (osd.6) 313 : cluster 0 2.1a scrub ok 2024-03-11T21:09:45.928 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:45 smithi023 bash[21372]: cluster 2024-03-11T21:09:45.322726+0000 mon.a (mon.0) 1703 : cluster 0 osdmap e348: 8 total, 8 up, 8 in 2024-03-11T21:09:45.928 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:45 smithi023 bash[29146]: cluster 2024-03-11T21:09:44.521029+0000 osd.6 (osd.6) 312 : cluster 0 2.1a scrub starts 2024-03-11T21:09:45.928 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:45 smithi023 bash[29146]: cluster 2024-03-11T21:09:44.522795+0000 osd.6 (osd.6) 313 : cluster 0 2.1a scrub ok 2024-03-11T21:09:45.928 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:45 smithi023 bash[29146]: cluster 2024-03-11T21:09:45.322726+0000 mon.a (mon.0) 1703 : cluster 0 osdmap e348: 8 total, 8 up, 8 in 2024-03-11T21:09:45.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:45.955 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:09:45.956 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:09:46.056 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:09:46.084 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:09:46.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:46.087 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:09:46.188 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:09:46.232 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:09:46.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:46.235 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:09:46.258 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:45 smithi146 bash[23809]: cluster 2024-03-11T21:09:44.521029+0000 osd.6 (osd.6) 312 : cluster 0 2.1a scrub starts 2024-03-11T21:09:46.258 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:45 smithi146 bash[23809]: cluster 2024-03-11T21:09:44.522795+0000 osd.6 (osd.6) 313 : cluster 0 2.1a scrub ok 2024-03-11T21:09:46.258 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:45 smithi146 bash[23809]: cluster 2024-03-11T21:09:45.322726+0000 mon.a (mon.0) 1703 : cluster 0 osdmap e348: 8 total, 8 up, 8 in 2024-03-11T21:09:46.259 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:09:45 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:09:46.336 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:09:46.365 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:09:46.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:46.368 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:09:46.468 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:09:46.495 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:09:46.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:46.499 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:09:46.600 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:09:46.630 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:09:46.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:46.632 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:09:46.733 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:09:46.757 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:09:46.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:46.761 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:09:46.862 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:09:46.890 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:09:46.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:46.893 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:09:46.994 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:09:47.004 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:46 smithi023 bash[21372]: cluster 2024-03-11T21:09:45.480658+0000 osd.6 (osd.6) 314 : cluster 0 4.e scrub starts 2024-03-11T21:09:47.004 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:46 smithi023 bash[21372]: cluster 2024-03-11T21:09:45.491540+0000 osd.6 (osd.6) 315 : cluster 0 4.e scrub ok 2024-03-11T21:09:47.004 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:46 smithi023 bash[21372]: audit 2024-03-11T21:09:45.797147+0000 mgr.y (mgr.24380) 702 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:09:47.004 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:46 smithi023 bash[21372]: cluster 2024-03-11T21:09:46.195907+0000 osd.1 (osd.1) 195 : cluster 0 2.0 scrub starts 2024-03-11T21:09:47.004 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:46 smithi023 bash[21372]: cluster 2024-03-11T21:09:46.204483+0000 osd.1 (osd.1) 196 : cluster 0 2.0 scrub ok 2024-03-11T21:09:47.004 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:46 smithi023 bash[21372]: cluster 2024-03-11T21:09:46.308934+0000 mgr.y (mgr.24380) 703 : cluster 0 pgmap v796: 148 pgs: 148 active+clean; 4.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 946 op/s 2024-03-11T21:09:47.004 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:46 smithi023 bash[21372]: cluster 2024-03-11T21:09:46.369086+0000 osd.7 (osd.7) 437 : cluster 0 6.5 scrub starts 2024-03-11T21:09:47.004 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:46 smithi023 bash[21372]: cluster 2024-03-11T21:09:46.434455+0000 osd.7 (osd.7) 438 : cluster 0 6.5 scrub ok 2024-03-11T21:09:47.005 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:46 smithi023 bash[29146]: cluster 2024-03-11T21:09:45.480658+0000 osd.6 (osd.6) 314 : cluster 0 4.e scrub starts 2024-03-11T21:09:47.005 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:46 smithi023 bash[29146]: cluster 2024-03-11T21:09:45.491540+0000 osd.6 (osd.6) 315 : cluster 0 4.e scrub ok 2024-03-11T21:09:47.005 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:46 smithi023 bash[29146]: audit 2024-03-11T21:09:45.797147+0000 mgr.y (mgr.24380) 702 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:09:47.005 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:46 smithi023 bash[29146]: cluster 2024-03-11T21:09:46.195907+0000 osd.1 (osd.1) 195 : cluster 0 2.0 scrub starts 2024-03-11T21:09:47.005 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:46 smithi023 bash[29146]: cluster 2024-03-11T21:09:46.204483+0000 osd.1 (osd.1) 196 : cluster 0 2.0 scrub ok 2024-03-11T21:09:47.005 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:46 smithi023 bash[29146]: cluster 2024-03-11T21:09:46.308934+0000 mgr.y (mgr.24380) 703 : cluster 0 pgmap v796: 148 pgs: 148 active+clean; 4.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 946 op/s 2024-03-11T21:09:47.005 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:46 smithi023 bash[29146]: cluster 2024-03-11T21:09:46.369086+0000 osd.7 (osd.7) 437 : cluster 0 6.5 scrub starts 2024-03-11T21:09:47.005 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:46 smithi023 bash[29146]: cluster 2024-03-11T21:09:46.434455+0000 osd.7 (osd.7) 438 : cluster 0 6.5 scrub ok 2024-03-11T21:09:47.022 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:09:47.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:47.026 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:09:47.123 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:46 smithi146 bash[23809]: cluster 2024-03-11T21:09:45.480658+0000 osd.6 (osd.6) 314 : cluster 0 4.e scrub starts 2024-03-11T21:09:47.123 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:46 smithi146 bash[23809]: cluster 2024-03-11T21:09:45.491540+0000 osd.6 (osd.6) 315 : cluster 0 4.e scrub ok 2024-03-11T21:09:47.123 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:46 smithi146 bash[23809]: audit 2024-03-11T21:09:45.797147+0000 mgr.y (mgr.24380) 702 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:09:47.123 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:46 smithi146 bash[23809]: cluster 2024-03-11T21:09:46.195907+0000 osd.1 (osd.1) 195 : cluster 0 2.0 scrub starts 2024-03-11T21:09:47.123 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:46 smithi146 bash[23809]: cluster 2024-03-11T21:09:46.204483+0000 osd.1 (osd.1) 196 : cluster 0 2.0 scrub ok 2024-03-11T21:09:47.123 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:46 smithi146 bash[23809]: cluster 2024-03-11T21:09:46.308934+0000 mgr.y (mgr.24380) 703 : cluster 0 pgmap v796: 148 pgs: 148 active+clean; 4.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 946 op/s 2024-03-11T21:09:47.123 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:46 smithi146 bash[23809]: cluster 2024-03-11T21:09:46.369086+0000 osd.7 (osd.7) 437 : cluster 0 6.5 scrub starts 2024-03-11T21:09:47.123 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:46 smithi146 bash[23809]: cluster 2024-03-11T21:09:46.434455+0000 osd.7 (osd.7) 438 : cluster 0 6.5 scrub ok 2024-03-11T21:09:47.127 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:47.154 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:47.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:47.161 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:09:47.262 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:09:47.295 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:09:47.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:47.301 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:09:47.380 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-03-11T21:09:47.380 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-03-11T21:09:47.381 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 30, 2024-03-11T21:09:47.381 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-03-11T21:09:47.381 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-03-11T21:09:47.402 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:09:47.431 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:09:47.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:47.434 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:09:47.535 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:09:47.606 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:09:47.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:47.609 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:09:47.710 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:09:47.739 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:09:47.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:47.744 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:09:47.845 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:09:47.866 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:47 smithi023 bash[29146]: cluster 2024-03-11T21:09:46.824128+0000 osd.5 (osd.5) 393 : cluster 0 4.d scrub starts 2024-03-11T21:09:47.867 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:47 smithi023 bash[29146]: cluster 2024-03-11T21:09:46.828790+0000 osd.5 (osd.5) 394 : cluster 0 4.d scrub ok 2024-03-11T21:09:47.874 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:09:47.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:47.878 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:09:47.979 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:09:47.988 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:47 smithi146 bash[23809]: cluster 2024-03-11T21:09:46.824128+0000 osd.5 (osd.5) 393 : cluster 0 4.d scrub starts 2024-03-11T21:09:47.988 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:47 smithi146 bash[23809]: cluster 2024-03-11T21:09:46.828790+0000 osd.5 (osd.5) 394 : cluster 0 4.d scrub ok 2024-03-11T21:09:48.008 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:09:48.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:48.012 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:09:48.113 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:09:48.123 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:47 smithi023 bash[21372]: cluster 2024-03-11T21:09:46.824128+0000 osd.5 (osd.5) 393 : cluster 0 4.d scrub starts 2024-03-11T21:09:48.123 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:47 smithi023 bash[21372]: cluster 2024-03-11T21:09:46.828790+0000 osd.5 (osd.5) 394 : cluster 0 4.d scrub ok 2024-03-11T21:09:48.144 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:09:48.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:48.149 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:09:48.250 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:09:48.272 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:09:48.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:48.274 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:09:48.375 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:09:48.398 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:09:48.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:48.400 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:09:48.501 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:09:48.512 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:09:48 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:09:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:09:48.532 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:09:48.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:48.534 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:09:48.635 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:09:48.676 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:09:48.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:48.682 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:09:48.783 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:09:48.811 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:09:48.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:48.818 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:09:48.840 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 6, 7, 1] out_osds: [] dead_osds: [] live_osds: [0, 2, 3, 5, 6, 7, 4, 1] 2024-03-11T21:09:48.840 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-11T21:09:48.840 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-03-11T21:09:48.840 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd dump -f json-pretty 2024-03-11T21:09:48.875 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-03-11T21:09:48.919 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:09:48.959 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:09:48.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:48.962 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:09:49.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:48 smithi146 bash[23809]: cluster 2024-03-11T21:09:47.330012+0000 osd.0 (osd.0) 301 : cluster 0 6.18 scrub starts 2024-03-11T21:09:49.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:48 smithi146 bash[23809]: cluster 2024-03-11T21:09:47.379728+0000 osd.0 (osd.0) 302 : cluster 0 6.18 scrub ok 2024-03-11T21:09:49.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:48 smithi146 bash[23809]: cluster 2024-03-11T21:09:47.514746+0000 osd.6 (osd.6) 316 : cluster 0 4.1f scrub starts 2024-03-11T21:09:49.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:48 smithi146 bash[23809]: cluster 2024-03-11T21:09:47.536613+0000 osd.6 (osd.6) 317 : cluster 0 4.1f scrub ok 2024-03-11T21:09:49.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:48 smithi146 bash[23809]: cluster 2024-03-11T21:09:47.825141+0000 osd.5 (osd.5) 395 : cluster 0 6.19 scrub starts 2024-03-11T21:09:49.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:48 smithi146 bash[23809]: cluster 2024-03-11T21:09:47.905764+0000 osd.5 (osd.5) 396 : cluster 0 6.19 scrub ok 2024-03-11T21:09:49.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:48 smithi146 bash[23809]: cluster 2024-03-11T21:09:48.131058+0000 osd.1 (osd.1) 197 : cluster 0 4.4 scrub starts 2024-03-11T21:09:49.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:48 smithi146 bash[23809]: cluster 2024-03-11T21:09:48.134721+0000 osd.1 (osd.1) 198 : cluster 0 4.4 scrub ok 2024-03-11T21:09:49.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:48 smithi146 bash[23809]: cluster 2024-03-11T21:09:48.309753+0000 mgr.y (mgr.24380) 704 : cluster 0 pgmap v797: 148 pgs: 148 active+clean; 4.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 640 B/s rd, 0 B/s wr, 754 op/s 2024-03-11T21:09:49.063 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:09:49.071 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:48 smithi023 bash[21372]: cluster 2024-03-11T21:09:47.330012+0000 osd.0 (osd.0) 301 : cluster 0 6.18 scrub starts 2024-03-11T21:09:49.071 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:48 smithi023 bash[21372]: cluster 2024-03-11T21:09:47.379728+0000 osd.0 (osd.0) 302 : cluster 0 6.18 scrub ok 2024-03-11T21:09:49.071 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:48 smithi023 bash[21372]: cluster 2024-03-11T21:09:47.514746+0000 osd.6 (osd.6) 316 : cluster 0 4.1f scrub starts 2024-03-11T21:09:49.071 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:48 smithi023 bash[21372]: cluster 2024-03-11T21:09:47.536613+0000 osd.6 (osd.6) 317 : cluster 0 4.1f scrub ok 2024-03-11T21:09:49.071 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:48 smithi023 bash[21372]: cluster 2024-03-11T21:09:47.825141+0000 osd.5 (osd.5) 395 : cluster 0 6.19 scrub starts 2024-03-11T21:09:49.071 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:48 smithi023 bash[21372]: cluster 2024-03-11T21:09:47.905764+0000 osd.5 (osd.5) 396 : cluster 0 6.19 scrub ok 2024-03-11T21:09:49.072 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:48 smithi023 bash[21372]: cluster 2024-03-11T21:09:48.131058+0000 osd.1 (osd.1) 197 : cluster 0 4.4 scrub starts 2024-03-11T21:09:49.072 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:48 smithi023 bash[21372]: cluster 2024-03-11T21:09:48.134721+0000 osd.1 (osd.1) 198 : cluster 0 4.4 scrub ok 2024-03-11T21:09:49.072 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:48 smithi023 bash[21372]: cluster 2024-03-11T21:09:48.309753+0000 mgr.y (mgr.24380) 704 : cluster 0 pgmap v797: 148 pgs: 148 active+clean; 4.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 640 B/s rd, 0 B/s wr, 754 op/s 2024-03-11T21:09:49.072 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:48 smithi023 bash[29146]: cluster 2024-03-11T21:09:47.330012+0000 osd.0 (osd.0) 301 : cluster 0 6.18 scrub starts 2024-03-11T21:09:49.072 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:48 smithi023 bash[29146]: cluster 2024-03-11T21:09:47.379728+0000 osd.0 (osd.0) 302 : cluster 0 6.18 scrub ok 2024-03-11T21:09:49.072 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:48 smithi023 bash[29146]: cluster 2024-03-11T21:09:47.514746+0000 osd.6 (osd.6) 316 : cluster 0 4.1f scrub starts 2024-03-11T21:09:49.072 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:48 smithi023 bash[29146]: cluster 2024-03-11T21:09:47.536613+0000 osd.6 (osd.6) 317 : cluster 0 4.1f scrub ok 2024-03-11T21:09:49.072 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:48 smithi023 bash[29146]: cluster 2024-03-11T21:09:47.825141+0000 osd.5 (osd.5) 395 : cluster 0 6.19 scrub starts 2024-03-11T21:09:49.072 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:48 smithi023 bash[29146]: cluster 2024-03-11T21:09:47.905764+0000 osd.5 (osd.5) 396 : cluster 0 6.19 scrub ok 2024-03-11T21:09:49.073 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:48 smithi023 bash[29146]: cluster 2024-03-11T21:09:48.131058+0000 osd.1 (osd.1) 197 : cluster 0 4.4 scrub starts 2024-03-11T21:09:49.073 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:48 smithi023 bash[29146]: cluster 2024-03-11T21:09:48.134721+0000 osd.1 (osd.1) 198 : cluster 0 4.4 scrub ok 2024-03-11T21:09:49.073 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:48 smithi023 bash[29146]: cluster 2024-03-11T21:09:48.309753+0000 mgr.y (mgr.24380) 704 : cluster 0 pgmap v797: 148 pgs: 148 active+clean; 4.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 640 B/s rd, 0 B/s wr, 754 op/s 2024-03-11T21:09:49.098 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:09:49.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:49.104 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:09:49.205 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:09:49.232 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:09:49.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:49.235 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:09:49.336 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:09:49.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:49.375 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:09:49.375 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:09:49.476 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:09:49.500 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:09:49.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:49.504 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:09:49.605 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:09:49.635 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:09:49.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:49.637 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:09:49.738 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:09:49.772 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:09:49.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:49.777 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:09:49.878 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:49.881 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:09:49.883 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:09:49.899 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:49.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:49.903 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:09:50.004 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:09:50.047 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:09:50.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:50.054 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:09:50.155 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:09:50.166 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:49 smithi023 bash[21372]: cluster 2024-03-11T21:09:48.525988+0000 osd.6 (osd.6) 318 : cluster 0 3.a scrub starts 2024-03-11T21:09:50.166 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:49 smithi023 bash[21372]: cluster 2024-03-11T21:09:48.539732+0000 osd.6 (osd.6) 319 : cluster 0 3.a scrub ok 2024-03-11T21:09:50.166 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:49 smithi023 bash[21372]: cluster 2024-03-11T21:09:48.834405+0000 osd.5 (osd.5) 397 : cluster 0 4.8 scrub starts 2024-03-11T21:09:50.166 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:49 smithi023 bash[21372]: cluster 2024-03-11T21:09:48.842638+0000 osd.5 (osd.5) 398 : cluster 0 4.8 scrub ok 2024-03-11T21:09:50.166 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:49 smithi023 bash[21372]: cluster 2024-03-11T21:09:49.141187+0000 osd.1 (osd.1) 199 : cluster 0 2.a scrub starts 2024-03-11T21:09:50.166 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:49 smithi023 bash[21372]: cluster 2024-03-11T21:09:49.147410+0000 osd.1 (osd.1) 200 : cluster 0 2.a scrub ok 2024-03-11T21:09:50.167 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:49 smithi023 bash[29146]: cluster 2024-03-11T21:09:48.525988+0000 osd.6 (osd.6) 318 : cluster 0 3.a scrub starts 2024-03-11T21:09:50.167 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:49 smithi023 bash[29146]: cluster 2024-03-11T21:09:48.539732+0000 osd.6 (osd.6) 319 : cluster 0 3.a scrub ok 2024-03-11T21:09:50.167 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:49 smithi023 bash[29146]: cluster 2024-03-11T21:09:48.834405+0000 osd.5 (osd.5) 397 : cluster 0 4.8 scrub starts 2024-03-11T21:09:50.167 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:49 smithi023 bash[29146]: cluster 2024-03-11T21:09:48.842638+0000 osd.5 (osd.5) 398 : cluster 0 4.8 scrub ok 2024-03-11T21:09:50.167 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:49 smithi023 bash[29146]: cluster 2024-03-11T21:09:49.141187+0000 osd.1 (osd.1) 199 : cluster 0 2.a scrub starts 2024-03-11T21:09:50.167 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:49 smithi023 bash[29146]: cluster 2024-03-11T21:09:49.147410+0000 osd.1 (osd.1) 200 : cluster 0 2.a scrub ok 2024-03-11T21:09:50.182 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:09:50.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:50.189 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:09:50.259 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:49 smithi146 bash[23809]: cluster 2024-03-11T21:09:48.525988+0000 osd.6 (osd.6) 318 : cluster 0 3.a scrub starts 2024-03-11T21:09:50.259 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:49 smithi146 bash[23809]: cluster 2024-03-11T21:09:48.539732+0000 osd.6 (osd.6) 319 : cluster 0 3.a scrub ok 2024-03-11T21:09:50.259 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:49 smithi146 bash[23809]: cluster 2024-03-11T21:09:48.834405+0000 osd.5 (osd.5) 397 : cluster 0 4.8 scrub starts 2024-03-11T21:09:50.259 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:49 smithi146 bash[23809]: cluster 2024-03-11T21:09:48.842638+0000 osd.5 (osd.5) 398 : cluster 0 4.8 scrub ok 2024-03-11T21:09:50.259 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:49 smithi146 bash[23809]: cluster 2024-03-11T21:09:49.141187+0000 osd.1 (osd.1) 199 : cluster 0 2.a scrub starts 2024-03-11T21:09:50.259 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:49 smithi146 bash[23809]: cluster 2024-03-11T21:09:49.147410+0000 osd.1 (osd.1) 200 : cluster 0 2.a scrub ok 2024-03-11T21:09:50.290 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:09:50.323 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:09:50.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:50.326 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:09:50.427 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:09:50.451 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:09:50.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:50.459 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:09:50.560 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:50.585 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:50.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:50.588 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:09:50.689 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:09:50.716 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:09:50.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:50.719 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:09:50.820 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:09:50.843 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:09:50.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:50.845 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:09:50.946 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:09:50.965 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:09:50.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:50.966 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:09:51.017 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:51 smithi023 bash[29146]: cluster 2024-03-11T21:09:49.340789+0000 osd.0 (osd.0) 303 : cluster 0 4.b scrub starts 2024-03-11T21:09:51.017 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:51 smithi023 bash[29146]: cluster 2024-03-11T21:09:49.344588+0000 osd.0 (osd.0) 304 : cluster 0 4.b scrub ok 2024-03-11T21:09:51.017 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:51 smithi023 bash[29146]: cluster 2024-03-11T21:09:49.508493+0000 osd.6 (osd.6) 320 : cluster 0 2.17 scrub starts 2024-03-11T21:09:51.018 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:51 smithi023 bash[29146]: cluster 2024-03-11T21:09:49.510477+0000 osd.6 (osd.6) 321 : cluster 0 2.17 scrub ok 2024-03-11T21:09:51.018 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:51 smithi023 bash[29146]: cluster 2024-03-11T21:09:49.804617+0000 osd.5 (osd.5) 399 : cluster 0 3.16 scrub starts 2024-03-11T21:09:51.018 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:51 smithi023 bash[29146]: cluster 2024-03-11T21:09:49.809672+0000 osd.5 (osd.5) 400 : cluster 0 3.16 scrub ok 2024-03-11T21:09:51.018 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:51 smithi023 bash[29146]: cluster 2024-03-11T21:09:49.932025+0000 osd.3 (osd.3) 396 : cluster 0 2.1e scrub starts 2024-03-11T21:09:51.018 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:51 smithi023 bash[29146]: cluster 2024-03-11T21:09:49.933184+0000 osd.3 (osd.3) 397 : cluster 0 2.1e scrub ok 2024-03-11T21:09:51.018 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:51 smithi023 bash[29146]: cluster 2024-03-11T21:09:50.153362+0000 osd.1 (osd.1) 201 : cluster 0 6.e scrub starts 2024-03-11T21:09:51.018 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:51 smithi023 bash[29146]: cluster 2024-03-11T21:09:50.191068+0000 osd.1 (osd.1) 202 : cluster 0 6.e scrub ok 2024-03-11T21:09:51.018 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:51 smithi023 bash[29146]: cluster 2024-03-11T21:09:50.310557+0000 mgr.y (mgr.24380) 705 : cluster 0 pgmap v798: 148 pgs: 148 active+clean; 4.5 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 925 op/s 2024-03-11T21:09:51.068 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:09:51.099 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:09:51.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:51.103 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:09:51.204 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:09:51.213 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:50 smithi146 bash[23809]: cluster 2024-03-11T21:09:49.340789+0000 osd.0 (osd.0) 303 : cluster 0 4.b scrub starts 2024-03-11T21:09:51.213 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:50 smithi146 bash[23809]: cluster 2024-03-11T21:09:49.344588+0000 osd.0 (osd.0) 304 : cluster 0 4.b scrub ok 2024-03-11T21:09:51.213 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:50 smithi146 bash[23809]: cluster 2024-03-11T21:09:49.508493+0000 osd.6 (osd.6) 320 : cluster 0 2.17 scrub starts 2024-03-11T21:09:51.213 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:50 smithi146 bash[23809]: cluster 2024-03-11T21:09:49.510477+0000 osd.6 (osd.6) 321 : cluster 0 2.17 scrub ok 2024-03-11T21:09:51.213 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:50 smithi146 bash[23809]: cluster 2024-03-11T21:09:49.804617+0000 osd.5 (osd.5) 399 : cluster 0 3.16 scrub starts 2024-03-11T21:09:51.213 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:50 smithi146 bash[23809]: cluster 2024-03-11T21:09:49.809672+0000 osd.5 (osd.5) 400 : cluster 0 3.16 scrub ok 2024-03-11T21:09:51.213 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:50 smithi146 bash[23809]: cluster 2024-03-11T21:09:49.932025+0000 osd.3 (osd.3) 396 : cluster 0 2.1e scrub starts 2024-03-11T21:09:51.213 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:50 smithi146 bash[23809]: cluster 2024-03-11T21:09:49.933184+0000 osd.3 (osd.3) 397 : cluster 0 2.1e scrub ok 2024-03-11T21:09:51.213 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:50 smithi146 bash[23809]: cluster 2024-03-11T21:09:50.153362+0000 osd.1 (osd.1) 201 : cluster 0 6.e scrub starts 2024-03-11T21:09:51.214 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:50 smithi146 bash[23809]: cluster 2024-03-11T21:09:50.191068+0000 osd.1 (osd.1) 202 : cluster 0 6.e scrub ok 2024-03-11T21:09:51.214 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:50 smithi146 bash[23809]: cluster 2024-03-11T21:09:50.310557+0000 mgr.y (mgr.24380) 705 : cluster 0 pgmap v798: 148 pgs: 148 active+clean; 4.5 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 925 op/s 2024-03-11T21:09:51.233 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:09:51.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:51.236 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:09:51.278 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:51 smithi023 bash[21372]: cluster 2024-03-11T21:09:49.340789+0000 osd.0 (osd.0) 303 : cluster 0 4.b scrub starts 2024-03-11T21:09:51.278 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:51 smithi023 bash[21372]: cluster 2024-03-11T21:09:49.344588+0000 osd.0 (osd.0) 304 : cluster 0 4.b scrub ok 2024-03-11T21:09:51.278 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:51 smithi023 bash[21372]: cluster 2024-03-11T21:09:49.508493+0000 osd.6 (osd.6) 320 : cluster 0 2.17 scrub starts 2024-03-11T21:09:51.278 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:51 smithi023 bash[21372]: cluster 2024-03-11T21:09:49.510477+0000 osd.6 (osd.6) 321 : cluster 0 2.17 scrub ok 2024-03-11T21:09:51.279 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:51 smithi023 bash[21372]: cluster 2024-03-11T21:09:49.804617+0000 osd.5 (osd.5) 399 : cluster 0 3.16 scrub starts 2024-03-11T21:09:51.279 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:51 smithi023 bash[21372]: cluster 2024-03-11T21:09:49.809672+0000 osd.5 (osd.5) 400 : cluster 0 3.16 scrub ok 2024-03-11T21:09:51.279 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:51 smithi023 bash[21372]: cluster 2024-03-11T21:09:49.932025+0000 osd.3 (osd.3) 396 : cluster 0 2.1e scrub starts 2024-03-11T21:09:51.279 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:51 smithi023 bash[21372]: cluster 2024-03-11T21:09:49.933184+0000 osd.3 (osd.3) 397 : cluster 0 2.1e scrub ok 2024-03-11T21:09:51.279 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:51 smithi023 bash[21372]: cluster 2024-03-11T21:09:50.153362+0000 osd.1 (osd.1) 201 : cluster 0 6.e scrub starts 2024-03-11T21:09:51.279 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:51 smithi023 bash[21372]: cluster 2024-03-11T21:09:50.191068+0000 osd.1 (osd.1) 202 : cluster 0 6.e scrub ok 2024-03-11T21:09:51.279 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:51 smithi023 bash[21372]: cluster 2024-03-11T21:09:50.310557+0000 mgr.y (mgr.24380) 705 : cluster 0 pgmap v798: 148 pgs: 148 active+clean; 4.5 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 925 op/s 2024-03-11T21:09:51.337 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:09:51.363 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:09:51.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:51.368 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:09:51.468 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:09:51.513 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:09:51.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:51.517 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:09:51.618 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:09:51.647 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:09:51.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:51.650 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:09:51.750 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:09:51.792 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:09:51.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:51.795 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:09:51.896 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:09:51.925 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:09:51.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:51.927 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:09:52.028 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:52.054 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:52.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:52.057 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:09:52.157 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:09:52.167 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:51 smithi023 bash[21372]: audit 2024-03-11T21:09:50.952084+0000 mon.a (mon.0) 1704 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:09:52.167 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:51 smithi023 bash[21372]: audit 2024-03-11T21:09:50.953165+0000 mon.a (mon.0) 1705 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:09:52.167 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:51 smithi023 bash[21372]: cluster 2024-03-11T21:09:50.972685+0000 osd.3 (osd.3) 398 : cluster 0 6.b scrub starts 2024-03-11T21:09:52.167 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:51 smithi023 bash[21372]: cluster 2024-03-11T21:09:51.109817+0000 osd.3 (osd.3) 399 : cluster 0 6.b scrub ok 2024-03-11T21:09:52.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:51 smithi023 bash[21372]: cluster 2024-03-11T21:09:51.518709+0000 osd.6 (osd.6) 322 : cluster 0 6.1e scrub starts 2024-03-11T21:09:52.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:51 smithi023 bash[21372]: cluster 2024-03-11T21:09:51.529426+0000 osd.6 (osd.6) 323 : cluster 0 6.1e scrub ok 2024-03-11T21:09:52.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:51 smithi023 bash[21372]: cluster 2024-03-11T21:09:51.793404+0000 osd.5 (osd.5) 401 : cluster 0 4.1 deep-scrub starts 2024-03-11T21:09:52.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:51 smithi023 bash[21372]: cluster 2024-03-11T21:09:51.796374+0000 osd.5 (osd.5) 402 : cluster 0 4.1 deep-scrub ok 2024-03-11T21:09:52.168 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:51 smithi023 bash[29146]: audit 2024-03-11T21:09:50.952084+0000 mon.a (mon.0) 1704 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:09:52.168 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:51 smithi023 bash[29146]: audit 2024-03-11T21:09:50.953165+0000 mon.a (mon.0) 1705 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:09:52.168 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:51 smithi023 bash[29146]: cluster 2024-03-11T21:09:50.972685+0000 osd.3 (osd.3) 398 : cluster 0 6.b scrub starts 2024-03-11T21:09:52.168 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:51 smithi023 bash[29146]: cluster 2024-03-11T21:09:51.109817+0000 osd.3 (osd.3) 399 : cluster 0 6.b scrub ok 2024-03-11T21:09:52.168 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:51 smithi023 bash[29146]: cluster 2024-03-11T21:09:51.518709+0000 osd.6 (osd.6) 322 : cluster 0 6.1e scrub starts 2024-03-11T21:09:52.168 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:51 smithi023 bash[29146]: cluster 2024-03-11T21:09:51.529426+0000 osd.6 (osd.6) 323 : cluster 0 6.1e scrub ok 2024-03-11T21:09:52.168 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:51 smithi023 bash[29146]: cluster 2024-03-11T21:09:51.793404+0000 osd.5 (osd.5) 401 : cluster 0 4.1 deep-scrub starts 2024-03-11T21:09:52.169 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:51 smithi023 bash[29146]: cluster 2024-03-11T21:09:51.796374+0000 osd.5 (osd.5) 402 : cluster 0 4.1 deep-scrub ok 2024-03-11T21:09:52.188 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:09:52.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:52.191 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:09:52.258 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:51 smithi146 bash[23809]: audit 2024-03-11T21:09:50.952084+0000 mon.a (mon.0) 1704 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:09:52.258 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:51 smithi146 bash[23809]: audit 2024-03-11T21:09:50.953165+0000 mon.a (mon.0) 1705 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:09:52.258 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:51 smithi146 bash[23809]: cluster 2024-03-11T21:09:50.972685+0000 osd.3 (osd.3) 398 : cluster 0 6.b scrub starts 2024-03-11T21:09:52.258 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:51 smithi146 bash[23809]: cluster 2024-03-11T21:09:51.109817+0000 osd.3 (osd.3) 399 : cluster 0 6.b scrub ok 2024-03-11T21:09:52.258 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:51 smithi146 bash[23809]: cluster 2024-03-11T21:09:51.518709+0000 osd.6 (osd.6) 322 : cluster 0 6.1e scrub starts 2024-03-11T21:09:52.259 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:51 smithi146 bash[23809]: cluster 2024-03-11T21:09:51.529426+0000 osd.6 (osd.6) 323 : cluster 0 6.1e scrub ok 2024-03-11T21:09:52.259 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:51 smithi146 bash[23809]: cluster 2024-03-11T21:09:51.793404+0000 osd.5 (osd.5) 401 : cluster 0 4.1 deep-scrub starts 2024-03-11T21:09:52.259 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:51 smithi146 bash[23809]: cluster 2024-03-11T21:09:51.796374+0000 osd.5 (osd.5) 402 : cluster 0 4.1 deep-scrub ok 2024-03-11T21:09:52.291 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:09:52.313 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:09:52.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:52.317 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:09:52.418 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:09:52.452 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:09:52.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:52.460 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:09:52.560 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:09:52.593 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:09:52.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:52.598 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:09:52.699 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:09:52.723 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:09:52.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:52.727 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:09:52.828 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:09:52.860 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:09:52.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:52.867 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:09:52.937 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:09:52.956 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:52 smithi023 bash[29146]: cluster 2024-03-11T21:09:52.311346+0000 mgr.y (mgr.24380) 706 : cluster 0 pgmap v799: 148 pgs: 148 active+clean; 4.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 971 op/s 2024-03-11T21:09:52.956 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:52 smithi023 bash[29146]: cluster 2024-03-11T21:09:52.417853+0000 osd.7 (osd.7) 439 : cluster 0 2.1d scrub starts 2024-03-11T21:09:52.956 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:52 smithi023 bash[29146]: cluster 2024-03-11T21:09:52.421817+0000 osd.7 (osd.7) 440 : cluster 0 2.1d scrub ok 2024-03-11T21:09:52.968 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:09:52.997 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:09:52.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:53.000 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:09:53.100 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:09:53.128 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:09:53.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:53.132 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:09:53.233 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:09:53.242 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:52 smithi146 bash[23809]: cluster 2024-03-11T21:09:52.311346+0000 mgr.y (mgr.24380) 706 : cluster 0 pgmap v799: 148 pgs: 148 active+clean; 4.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 971 op/s 2024-03-11T21:09:53.242 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:52 smithi146 bash[23809]: cluster 2024-03-11T21:09:52.417853+0000 osd.7 (osd.7) 439 : cluster 0 2.1d scrub starts 2024-03-11T21:09:53.242 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:52 smithi146 bash[23809]: cluster 2024-03-11T21:09:52.421817+0000 osd.7 (osd.7) 440 : cluster 0 2.1d scrub ok 2024-03-11T21:09:53.243 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:52 smithi146 bash[23809]: cluster 2024-03-11T21:09:52.499063+0000 osd.6 (osd.6) 324 : cluster 0 4.5 scrub starts 2024-03-11T21:09:53.243 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:52 smithi146 bash[23809]: cluster 2024-03-11T21:09:52.505782+0000 osd.6 (osd.6) 325 : cluster 0 4.5 scrub ok 2024-03-11T21:09:53.259 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:09:53.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:53.266 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:09:53.278 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:52 smithi023 bash[21372]: cluster 2024-03-11T21:09:52.311346+0000 mgr.y (mgr.24380) 706 : cluster 0 pgmap v799: 148 pgs: 148 active+clean; 4.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 971 op/s 2024-03-11T21:09:53.278 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:52 smithi023 bash[21372]: cluster 2024-03-11T21:09:52.417853+0000 osd.7 (osd.7) 439 : cluster 0 2.1d scrub starts 2024-03-11T21:09:53.278 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:52 smithi023 bash[21372]: cluster 2024-03-11T21:09:52.421817+0000 osd.7 (osd.7) 440 : cluster 0 2.1d scrub ok 2024-03-11T21:09:53.278 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:52 smithi023 bash[21372]: cluster 2024-03-11T21:09:52.499063+0000 osd.6 (osd.6) 324 : cluster 0 4.5 scrub starts 2024-03-11T21:09:53.278 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:52 smithi023 bash[21372]: cluster 2024-03-11T21:09:52.505782+0000 osd.6 (osd.6) 325 : cluster 0 4.5 scrub ok 2024-03-11T21:09:53.279 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:52 smithi023 bash[29146]: cluster 2024-03-11T21:09:52.499063+0000 osd.6 (osd.6) 324 : cluster 0 4.5 scrub starts 2024-03-11T21:09:53.279 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:52 smithi023 bash[29146]: cluster 2024-03-11T21:09:52.505782+0000 osd.6 (osd.6) 325 : cluster 0 4.5 scrub ok 2024-03-11T21:09:53.366 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:09:53.404 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:09:53.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:53.409 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:09:53.509 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:53.532 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:53.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:53.534 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:09:53.635 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:09:53.667 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:09:53.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:53.672 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:09:53.677 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:09:53.774 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:09:53.804 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:09:53.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:53.810 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:09:53.911 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:53.943 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:53.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:53.946 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:09:54.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:53 smithi023 bash[21372]: cluster 2024-03-11T21:09:53.195520+0000 osd.1 (osd.1) 203 : cluster 0 4.f scrub starts 2024-03-11T21:09:54.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:53 smithi023 bash[21372]: cluster 2024-03-11T21:09:53.196737+0000 osd.1 (osd.1) 204 : cluster 0 4.f scrub ok 2024-03-11T21:09:54.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:53 smithi023 bash[21372]: cluster 2024-03-11T21:09:53.433203+0000 osd.7 (osd.7) 441 : cluster 0 6.8 deep-scrub starts 2024-03-11T21:09:54.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:53 smithi023 bash[21372]: cluster 2024-03-11T21:09:53.436477+0000 osd.7 (osd.7) 442 : cluster 0 6.8 deep-scrub ok 2024-03-11T21:09:54.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:53 smithi023 bash[21372]: cluster 2024-03-11T21:09:53.476480+0000 osd.6 (osd.6) 326 : cluster 0 4.18 scrub starts 2024-03-11T21:09:54.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:53 smithi023 bash[21372]: cluster 2024-03-11T21:09:53.477468+0000 osd.6 (osd.6) 327 : cluster 0 4.18 scrub ok 2024-03-11T21:09:54.029 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:53 smithi023 bash[29146]: cluster 2024-03-11T21:09:53.195520+0000 osd.1 (osd.1) 203 : cluster 0 4.f scrub starts 2024-03-11T21:09:54.029 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:53 smithi023 bash[29146]: cluster 2024-03-11T21:09:53.196737+0000 osd.1 (osd.1) 204 : cluster 0 4.f scrub ok 2024-03-11T21:09:54.029 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:53 smithi023 bash[29146]: cluster 2024-03-11T21:09:53.433203+0000 osd.7 (osd.7) 441 : cluster 0 6.8 deep-scrub starts 2024-03-11T21:09:54.029 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:53 smithi023 bash[29146]: cluster 2024-03-11T21:09:53.436477+0000 osd.7 (osd.7) 442 : cluster 0 6.8 deep-scrub ok 2024-03-11T21:09:54.029 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:53 smithi023 bash[29146]: cluster 2024-03-11T21:09:53.476480+0000 osd.6 (osd.6) 326 : cluster 0 4.18 scrub starts 2024-03-11T21:09:54.029 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:53 smithi023 bash[29146]: cluster 2024-03-11T21:09:53.477468+0000 osd.6 (osd.6) 327 : cluster 0 4.18 scrub ok 2024-03-11T21:09:54.047 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:09:54.051 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:53 smithi146 bash[23809]: cluster 2024-03-11T21:09:53.195520+0000 osd.1 (osd.1) 203 : cluster 0 4.f scrub starts 2024-03-11T21:09:54.051 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:53 smithi146 bash[23809]: cluster 2024-03-11T21:09:53.196737+0000 osd.1 (osd.1) 204 : cluster 0 4.f scrub ok 2024-03-11T21:09:54.051 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:53 smithi146 bash[23809]: cluster 2024-03-11T21:09:53.433203+0000 osd.7 (osd.7) 441 : cluster 0 6.8 deep-scrub starts 2024-03-11T21:09:54.051 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:53 smithi146 bash[23809]: cluster 2024-03-11T21:09:53.436477+0000 osd.7 (osd.7) 442 : cluster 0 6.8 deep-scrub ok 2024-03-11T21:09:54.052 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:53 smithi146 bash[23809]: cluster 2024-03-11T21:09:53.476480+0000 osd.6 (osd.6) 326 : cluster 0 4.18 scrub starts 2024-03-11T21:09:54.052 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:53 smithi146 bash[23809]: cluster 2024-03-11T21:09:53.477468+0000 osd.6 (osd.6) 327 : cluster 0 4.18 scrub ok 2024-03-11T21:09:54.079 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:09:54.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:54.084 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:09:54.185 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:09:54.207 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:09:54.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:54.209 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:09:54.310 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:09:54.343 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:09:54.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:54.347 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:09:54.448 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:54.478 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:54.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:54.481 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:09:54.582 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:09:54.599 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:09:54.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:54.601 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:09:54.702 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:09:54.740 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:09:54.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:54.743 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:09:54.844 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:09:54.868 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:09:54.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:54.871 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:09:54.971 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:09:54.990 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:09:54.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:54.992 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:09:55.008 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:54 smithi146 bash[23809]: cluster 2024-03-11T21:09:54.011580+0000 osd.3 (osd.3) 400 : cluster 0 3.7 scrub starts 2024-03-11T21:09:55.008 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:54 smithi146 bash[23809]: cluster 2024-03-11T21:09:54.016572+0000 osd.3 (osd.3) 401 : cluster 0 3.7 scrub ok 2024-03-11T21:09:55.008 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:54 smithi146 bash[23809]: cluster 2024-03-11T21:09:54.211185+0000 osd.1 (osd.1) 205 : cluster 0 2.6 scrub starts 2024-03-11T21:09:55.008 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:54 smithi146 bash[23809]: cluster 2024-03-11T21:09:54.213462+0000 osd.1 (osd.1) 206 : cluster 0 2.6 scrub ok 2024-03-11T21:09:55.008 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:54 smithi146 bash[23809]: cluster 2024-03-11T21:09:54.311791+0000 mgr.y (mgr.24380) 707 : cluster 0 pgmap v800: 148 pgs: 148 active+clean; 4.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 971 op/s 2024-03-11T21:09:55.008 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:54 smithi146 bash[23809]: cluster 2024-03-11T21:09:54.726762+0000 osd.6 (osd.6) 328 : cluster 0 purged_snaps scrub starts 2024-03-11T21:09:55.008 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:54 smithi146 bash[23809]: cluster 2024-03-11T21:09:54.726879+0000 osd.6 (osd.6) 329 : cluster 0 purged_snaps scrub ok 2024-03-11T21:09:55.093 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:09:55.118 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:09:55.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:55.119 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:09:55.220 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:09:55.246 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:09:55.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:55.248 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:09:55.279 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:54 smithi023 bash[21372]: cluster 2024-03-11T21:09:54.011580+0000 osd.3 (osd.3) 400 : cluster 0 3.7 scrub starts 2024-03-11T21:09:55.279 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:54 smithi023 bash[21372]: cluster 2024-03-11T21:09:54.016572+0000 osd.3 (osd.3) 401 : cluster 0 3.7 scrub ok 2024-03-11T21:09:55.279 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:54 smithi023 bash[21372]: cluster 2024-03-11T21:09:54.211185+0000 osd.1 (osd.1) 205 : cluster 0 2.6 scrub starts 2024-03-11T21:09:55.279 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:54 smithi023 bash[21372]: cluster 2024-03-11T21:09:54.213462+0000 osd.1 (osd.1) 206 : cluster 0 2.6 scrub ok 2024-03-11T21:09:55.279 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:54 smithi023 bash[21372]: cluster 2024-03-11T21:09:54.311791+0000 mgr.y (mgr.24380) 707 : cluster 0 pgmap v800: 148 pgs: 148 active+clean; 4.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 971 op/s 2024-03-11T21:09:55.279 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:54 smithi023 bash[21372]: cluster 2024-03-11T21:09:54.726762+0000 osd.6 (osd.6) 328 : cluster 0 purged_snaps scrub starts 2024-03-11T21:09:55.279 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:54 smithi023 bash[21372]: cluster 2024-03-11T21:09:54.726879+0000 osd.6 (osd.6) 329 : cluster 0 purged_snaps scrub ok 2024-03-11T21:09:55.279 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:54 smithi023 bash[29146]: cluster 2024-03-11T21:09:54.011580+0000 osd.3 (osd.3) 400 : cluster 0 3.7 scrub starts 2024-03-11T21:09:55.280 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:54 smithi023 bash[29146]: cluster 2024-03-11T21:09:54.016572+0000 osd.3 (osd.3) 401 : cluster 0 3.7 scrub ok 2024-03-11T21:09:55.280 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:54 smithi023 bash[29146]: cluster 2024-03-11T21:09:54.211185+0000 osd.1 (osd.1) 205 : cluster 0 2.6 scrub starts 2024-03-11T21:09:55.280 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:54 smithi023 bash[29146]: cluster 2024-03-11T21:09:54.213462+0000 osd.1 (osd.1) 206 : cluster 0 2.6 scrub ok 2024-03-11T21:09:55.280 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:54 smithi023 bash[29146]: cluster 2024-03-11T21:09:54.311791+0000 mgr.y (mgr.24380) 707 : cluster 0 pgmap v800: 148 pgs: 148 active+clean; 4.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 971 op/s 2024-03-11T21:09:55.280 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:54 smithi023 bash[29146]: cluster 2024-03-11T21:09:54.726762+0000 osd.6 (osd.6) 328 : cluster 0 purged_snaps scrub starts 2024-03-11T21:09:55.280 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:54 smithi023 bash[29146]: cluster 2024-03-11T21:09:54.726879+0000 osd.6 (osd.6) 329 : cluster 0 purged_snaps scrub ok 2024-03-11T21:09:55.348 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:09:55.368 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:09:55.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:55.370 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:09:55.471 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:09:55.498 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:09:55.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:55.501 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:09:55.602 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:55.627 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:55.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:55.629 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:09:55.729 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:09:55.764 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:09:55.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:55.768 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:09:55.869 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:09:55.895 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:09:55.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:55.898 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:09:55.950 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:09:55 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:09:55.998 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:09:56.026 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:09:56.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:56.028 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:09:56.129 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:09:56.138 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:55 smithi146 bash[23809]: cluster 2024-03-11T21:09:54.400144+0000 osd.0 (osd.0) 305 : cluster 0 4.1e scrub starts 2024-03-11T21:09:56.138 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:55 smithi146 bash[23809]: cluster 2024-03-11T21:09:54.403130+0000 osd.0 (osd.0) 306 : cluster 0 4.1e scrub ok 2024-03-11T21:09:56.138 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:55 smithi146 bash[23809]: cluster 2024-03-11T21:09:54.984856+0000 mon.a (mon.0) 1706 : cluster 0 osdmap e349: 8 total, 8 up, 8 in 2024-03-11T21:09:56.138 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:55 smithi146 bash[23809]: cluster 2024-03-11T21:09:55.526104+0000 osd.7 (osd.7) 443 : cluster 0 2.5 scrub starts 2024-03-11T21:09:56.138 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:55 smithi146 bash[23809]: cluster 2024-03-11T21:09:55.530105+0000 osd.7 (osd.7) 444 : cluster 0 2.5 scrub ok 2024-03-11T21:09:56.138 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:55 smithi146 bash[23809]: cluster 2024-03-11T21:09:55.785107+0000 osd.5 (osd.5) 403 : cluster 0 2.2 deep-scrub starts 2024-03-11T21:09:56.138 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:55 smithi146 bash[23809]: cluster 2024-03-11T21:09:55.786341+0000 osd.5 (osd.5) 404 : cluster 0 2.2 deep-scrub ok 2024-03-11T21:09:56.150 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:09:56.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:56.153 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:09:56.253 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:09:56.278 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:09:56.279 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:55 smithi023 bash[21372]: cluster 2024-03-11T21:09:54.400144+0000 osd.0 (osd.0) 305 : cluster 0 4.1e scrub starts 2024-03-11T21:09:56.279 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:55 smithi023 bash[21372]: cluster 2024-03-11T21:09:54.403130+0000 osd.0 (osd.0) 306 : cluster 0 4.1e scrub ok 2024-03-11T21:09:56.279 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:55 smithi023 bash[21372]: cluster 2024-03-11T21:09:54.984856+0000 mon.a (mon.0) 1706 : cluster 0 osdmap e349: 8 total, 8 up, 8 in 2024-03-11T21:09:56.279 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:55 smithi023 bash[21372]: cluster 2024-03-11T21:09:55.526104+0000 osd.7 (osd.7) 443 : cluster 0 2.5 scrub starts 2024-03-11T21:09:56.279 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:55 smithi023 bash[21372]: cluster 2024-03-11T21:09:55.530105+0000 osd.7 (osd.7) 444 : cluster 0 2.5 scrub ok 2024-03-11T21:09:56.279 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:55 smithi023 bash[21372]: cluster 2024-03-11T21:09:55.785107+0000 osd.5 (osd.5) 403 : cluster 0 2.2 deep-scrub starts 2024-03-11T21:09:56.279 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:55 smithi023 bash[21372]: cluster 2024-03-11T21:09:55.786341+0000 osd.5 (osd.5) 404 : cluster 0 2.2 deep-scrub ok 2024-03-11T21:09:56.280 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:56 smithi023 bash[29146]: cluster 2024-03-11T21:09:54.400144+0000 osd.0 (osd.0) 305 : cluster 0 4.1e scrub starts 2024-03-11T21:09:56.280 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:56 smithi023 bash[29146]: cluster 2024-03-11T21:09:54.403130+0000 osd.0 (osd.0) 306 : cluster 0 4.1e scrub ok 2024-03-11T21:09:56.280 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:56 smithi023 bash[29146]: cluster 2024-03-11T21:09:54.984856+0000 mon.a (mon.0) 1706 : cluster 0 osdmap e349: 8 total, 8 up, 8 in 2024-03-11T21:09:56.280 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:56 smithi023 bash[29146]: cluster 2024-03-11T21:09:55.526104+0000 osd.7 (osd.7) 443 : cluster 0 2.5 scrub starts 2024-03-11T21:09:56.280 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:56 smithi023 bash[29146]: cluster 2024-03-11T21:09:55.530105+0000 osd.7 (osd.7) 444 : cluster 0 2.5 scrub ok 2024-03-11T21:09:56.280 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:56 smithi023 bash[29146]: cluster 2024-03-11T21:09:55.785107+0000 osd.5 (osd.5) 403 : cluster 0 2.2 deep-scrub starts 2024-03-11T21:09:56.280 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:56 smithi023 bash[29146]: cluster 2024-03-11T21:09:55.786341+0000 osd.5 (osd.5) 404 : cluster 0 2.2 deep-scrub ok 2024-03-11T21:09:56.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:56.281 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:09:56.382 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:56.408 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:56.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:56.411 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:09:56.512 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:09:56.536 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:09:56.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:56.537 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:09:56.638 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:09:56.668 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:09:56.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:56.671 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:09:56.772 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:09:56.793 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:09:56.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:56.795 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:09:56.896 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:09:56.934 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:09:56.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:56.938 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:09:57.038 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:57.066 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:57.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:57.071 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:09:57.171 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:09:57.181 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:56 smithi023 bash[21372]: audit 2024-03-11T21:09:55.805211+0000 mgr.y (mgr.24380) 708 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:09:57.181 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:56 smithi023 bash[21372]: cluster 2024-03-11T21:09:56.168462+0000 osd.1 (osd.1) 207 : cluster 0 6.4 scrub starts 2024-03-11T21:09:57.181 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:56 smithi023 bash[21372]: cluster 2024-03-11T21:09:56.169828+0000 osd.1 (osd.1) 208 : cluster 0 6.4 scrub ok 2024-03-11T21:09:57.181 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:56 smithi023 bash[21372]: cluster 2024-03-11T21:09:56.312530+0000 mgr.y (mgr.24380) 709 : cluster 0 pgmap v802: 148 pgs: 148 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 916 op/s 2024-03-11T21:09:57.182 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:56 smithi023 bash[21372]: cluster 2024-03-11T21:09:56.780076+0000 osd.5 (osd.5) 405 : cluster 0 2.3 scrub starts 2024-03-11T21:09:57.182 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:56 smithi023 bash[21372]: cluster 2024-03-11T21:09:56.781831+0000 osd.5 (osd.5) 406 : cluster 0 2.3 scrub ok 2024-03-11T21:09:57.182 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:57 smithi023 bash[29146]: audit 2024-03-11T21:09:55.805211+0000 mgr.y (mgr.24380) 708 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:09:57.182 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:57 smithi023 bash[29146]: cluster 2024-03-11T21:09:56.168462+0000 osd.1 (osd.1) 207 : cluster 0 6.4 scrub starts 2024-03-11T21:09:57.182 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:57 smithi023 bash[29146]: cluster 2024-03-11T21:09:56.169828+0000 osd.1 (osd.1) 208 : cluster 0 6.4 scrub ok 2024-03-11T21:09:57.182 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:57 smithi023 bash[29146]: cluster 2024-03-11T21:09:56.312530+0000 mgr.y (mgr.24380) 709 : cluster 0 pgmap v802: 148 pgs: 148 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 916 op/s 2024-03-11T21:09:57.182 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:57 smithi023 bash[29146]: cluster 2024-03-11T21:09:56.780076+0000 osd.5 (osd.5) 405 : cluster 0 2.3 scrub starts 2024-03-11T21:09:57.182 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:57 smithi023 bash[29146]: cluster 2024-03-11T21:09:56.781831+0000 osd.5 (osd.5) 406 : cluster 0 2.3 scrub ok 2024-03-11T21:09:57.202 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:09:57.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:57.204 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:09:57.258 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:57 smithi146 bash[23809]: audit 2024-03-11T21:09:55.805211+0000 mgr.y (mgr.24380) 708 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:09:57.258 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:57 smithi146 bash[23809]: cluster 2024-03-11T21:09:56.168462+0000 osd.1 (osd.1) 207 : cluster 0 6.4 scrub starts 2024-03-11T21:09:57.258 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:57 smithi146 bash[23809]: cluster 2024-03-11T21:09:56.169828+0000 osd.1 (osd.1) 208 : cluster 0 6.4 scrub ok 2024-03-11T21:09:57.258 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:57 smithi146 bash[23809]: cluster 2024-03-11T21:09:56.312530+0000 mgr.y (mgr.24380) 709 : cluster 0 pgmap v802: 148 pgs: 148 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 916 op/s 2024-03-11T21:09:57.258 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:57 smithi146 bash[23809]: cluster 2024-03-11T21:09:56.780076+0000 osd.5 (osd.5) 405 : cluster 0 2.3 scrub starts 2024-03-11T21:09:57.259 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:57 smithi146 bash[23809]: cluster 2024-03-11T21:09:56.781831+0000 osd.5 (osd.5) 406 : cluster 0 2.3 scrub ok 2024-03-11T21:09:57.305 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:09:57.330 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:09:57.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:57.333 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:09:57.434 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:09:57.462 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:09:57.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:57.465 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:09:57.565 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:09:57.583 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:09:57.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:57.588 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:09:57.689 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:09:57.712 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:09:57.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:57.717 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:09:57.817 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:09:57.849 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:09:57.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:57.854 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:09:57.954 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:57.997 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:09:57.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:57.999 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:09:58.100 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:09:58.133 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:09:58.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:58.135 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:09:58.236 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:09:58.259 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:09:58.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:58.261 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:09:58.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:58 smithi023 bash[21372]: cluster 2024-03-11T21:09:57.171628+0000 osd.1 (osd.1) 209 : cluster 0 6.1d scrub starts 2024-03-11T21:09:58.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:58 smithi023 bash[21372]: cluster 2024-03-11T21:09:57.285354+0000 osd.1 (osd.1) 210 : cluster 0 6.1d scrub ok 2024-03-11T21:09:58.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:58 smithi023 bash[21372]: cluster 2024-03-11T21:09:57.516319+0000 osd.6 (osd.6) 330 : cluster 0 10.f scrub starts 2024-03-11T21:09:58.341 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:58 smithi023 bash[21372]: cluster 2024-03-11T21:09:57.531001+0000 osd.7 (osd.7) 445 : cluster 0 2.1c scrub starts 2024-03-11T21:09:58.341 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:58 smithi023 bash[21372]: cluster 2024-03-11T21:09:57.534960+0000 osd.7 (osd.7) 446 : cluster 0 2.1c scrub ok 2024-03-11T21:09:58.341 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:58 smithi023 bash[29146]: cluster 2024-03-11T21:09:57.171628+0000 osd.1 (osd.1) 209 : cluster 0 6.1d scrub starts 2024-03-11T21:09:58.341 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:58 smithi023 bash[29146]: cluster 2024-03-11T21:09:57.285354+0000 osd.1 (osd.1) 210 : cluster 0 6.1d scrub ok 2024-03-11T21:09:58.341 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:58 smithi023 bash[29146]: cluster 2024-03-11T21:09:57.516319+0000 osd.6 (osd.6) 330 : cluster 0 10.f scrub starts 2024-03-11T21:09:58.341 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:58 smithi023 bash[29146]: cluster 2024-03-11T21:09:57.531001+0000 osd.7 (osd.7) 445 : cluster 0 2.1c scrub starts 2024-03-11T21:09:58.341 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:58 smithi023 bash[29146]: cluster 2024-03-11T21:09:57.534960+0000 osd.7 (osd.7) 446 : cluster 0 2.1c scrub ok 2024-03-11T21:09:58.362 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:09:58.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:58.391 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:09:58.391 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:09:58.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:58 smithi146 bash[23809]: cluster 2024-03-11T21:09:57.171628+0000 osd.1 (osd.1) 209 : cluster 0 6.1d scrub starts 2024-03-11T21:09:58.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:58 smithi146 bash[23809]: cluster 2024-03-11T21:09:57.285354+0000 osd.1 (osd.1) 210 : cluster 0 6.1d scrub ok 2024-03-11T21:09:58.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:58 smithi146 bash[23809]: cluster 2024-03-11T21:09:57.516319+0000 osd.6 (osd.6) 330 : cluster 0 10.f scrub starts 2024-03-11T21:09:58.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:58 smithi146 bash[23809]: cluster 2024-03-11T21:09:57.531001+0000 osd.7 (osd.7) 445 : cluster 0 2.1c scrub starts 2024-03-11T21:09:58.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:58 smithi146 bash[23809]: cluster 2024-03-11T21:09:57.534960+0000 osd.7 (osd.7) 446 : cluster 0 2.1c scrub ok 2024-03-11T21:09:58.492 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:09:58.513 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:09:58.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:58.515 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:09:58.616 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:09:58.654 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:09:58.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:58.656 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:09:58.757 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:09:58.778 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:09:58 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:09:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:09:58.785 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:09:58.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:58.794 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:09:58.895 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:09:58.927 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:09:58.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:58.932 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:09:59.033 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:09:59.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:59.070 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:09:59.070 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:09:59.082 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:09:59 smithi146 bash[23809]: cluster 2024-03-11T21:09:58.313306+0000 mgr.y (mgr.24380) 710 : cluster 0 pgmap v803: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 905 op/s 2024-03-11T21:09:59.171 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:09:59.181 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:09:59 smithi023 bash[21372]: cluster 2024-03-11T21:09:58.313306+0000 mgr.y (mgr.24380) 710 : cluster 0 pgmap v803: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 905 op/s 2024-03-11T21:09:59.181 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:09:59 smithi023 bash[29146]: cluster 2024-03-11T21:09:58.313306+0000 mgr.y (mgr.24380) 710 : cluster 0 pgmap v803: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 905 op/s 2024-03-11T21:09:59.195 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:09:59.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:59.201 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:09:59.302 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:09:59.332 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:09:59.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:59.337 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:09:59.438 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:09:59.468 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:09:59.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:59.470 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:09:59.571 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:09:59.593 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:09:59.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:59.596 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:09:59.697 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:09:59.733 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:09:59.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:59.738 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:09:59.838 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:09:59.856 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:09:59.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:59.859 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:09:59.959 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:09:59.980 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:09:59.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:59.981 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:10:00.082 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:00.108 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:00.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:00.111 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:10:00.211 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:00.249 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:00.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:00.253 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:10:00.258 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:00 smithi146 bash[23809]: cluster 2024-03-11T21:09:58.987097+0000 osd.6 (osd.6) 331 : cluster 0 10.f scrub ok 2024-03-11T21:10:00.258 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:00 smithi146 bash[23809]: cluster 2024-03-11T21:09:59.506368+0000 osd.7 (osd.7) 447 : cluster 0 1.0 scrub starts 2024-03-11T21:10:00.258 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:00 smithi146 bash[23809]: cluster 2024-03-11T21:09:59.511437+0000 osd.7 (osd.7) 448 : cluster 0 1.0 scrub ok 2024-03-11T21:10:00.258 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:00 smithi146 bash[23809]: cluster 2024-03-11T21:10:00.000118+0000 mon.a (mon.0) 1707 : cluster 1 overall HEALTH_OK 2024-03-11T21:10:00.278 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:00 smithi023 bash[21372]: cluster 2024-03-11T21:09:58.987097+0000 osd.6 (osd.6) 331 : cluster 0 10.f scrub ok 2024-03-11T21:10:00.278 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:00 smithi023 bash[21372]: cluster 2024-03-11T21:09:59.506368+0000 osd.7 (osd.7) 447 : cluster 0 1.0 scrub starts 2024-03-11T21:10:00.278 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:00 smithi023 bash[21372]: cluster 2024-03-11T21:09:59.511437+0000 osd.7 (osd.7) 448 : cluster 0 1.0 scrub ok 2024-03-11T21:10:00.278 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:00 smithi023 bash[21372]: cluster 2024-03-11T21:10:00.000118+0000 mon.a (mon.0) 1707 : cluster 1 overall HEALTH_OK 2024-03-11T21:10:00.278 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:00 smithi023 bash[29146]: cluster 2024-03-11T21:09:58.987097+0000 osd.6 (osd.6) 331 : cluster 0 10.f scrub ok 2024-03-11T21:10:00.278 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:00 smithi023 bash[29146]: cluster 2024-03-11T21:09:59.506368+0000 osd.7 (osd.7) 447 : cluster 0 1.0 scrub starts 2024-03-11T21:10:00.278 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:00 smithi023 bash[29146]: cluster 2024-03-11T21:09:59.511437+0000 osd.7 (osd.7) 448 : cluster 0 1.0 scrub ok 2024-03-11T21:10:00.279 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:00 smithi023 bash[29146]: cluster 2024-03-11T21:10:00.000118+0000 mon.a (mon.0) 1707 : cluster 1 overall HEALTH_OK 2024-03-11T21:10:00.354 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:00.383 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:00.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:00.386 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:10:00.487 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:10:00.514 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:10:00.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:00.515 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:10:00.616 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:10:00.647 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:10:00.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:00.649 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:10:00.749 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:00.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:00.783 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:00.783 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:10:00.884 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:00.908 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:00.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:00.910 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:10:01.011 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:10:01.029 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:10:01.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:01.031 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:10:01.088 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-03-11T21:10:01.088 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-03-11T21:10:01.088 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-03-11T21:10:01.088 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-03-11T21:10:01.088 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.088 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.15081.0:94727 9.5 9:ac54a978:::benchmark_data_smithi023_106206_object94726:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e275)", 2024-03-11T21:10:01.088 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T21:04:14.845756+0000", 2024-03-11T21:10:01.088 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 346.24986093799998, 2024-03-11T21:10:01.088 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.42099270700000002, 2024-03-11T21:10:01.089 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:10:01.089 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:10:01.089 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-03-11T21:10:01.089 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.15081", 2024-03-11T21:10:01.089 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.23:0/2841194346", 2024-03-11T21:10:01.089 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 94727 2024-03-11T21:10:01.089 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.089 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:10:01.089 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.089 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:10:01.089 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:14.845756+0000", 2024-03-11T21:10:01.090 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:10:01.090 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.090 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.090 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:10:01.090 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:14.845756+0000", 2024-03-11T21:10:01.090 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:10:01.090 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.090 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.090 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:10:01.090 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:14.845757+0000", 2024-03-11T21:10:01.090 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0750000000000001e-06 2024-03-11T21:10:01.090 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.090 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.090 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:10:01.091 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:14.845811+0000", 2024-03-11T21:10:01.091 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3581000000000001e-05 2024-03-11T21:10:01.091 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.091 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.091 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:10:01.091 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:14.845819+0000", 2024-03-11T21:10:01.091 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.3510000000000002e-06 2024-03-11T21:10:01.091 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.091 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.091 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:10:01.091 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:14.845825+0000", 2024-03-11T21:10:01.091 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7189999999999998e-06 2024-03-11T21:10:01.091 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.091 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.091 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:10:01.092 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:14.845870+0000", 2024-03-11T21:10:01.092 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5540000000000001e-05 2024-03-11T21:10:01.092 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.092 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.092 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:10:01.092 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:14.845950+0000", 2024-03-11T21:10:01.092 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.0154999999999994e-05 2024-03-11T21:10:01.092 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.092 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.092 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0,7", 2024-03-11T21:10:01.092 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:14.846036+0000", 2024-03-11T21:10:01.093 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5017000000000003e-05 2024-03-11T21:10:01.093 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.093 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.093 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-03-11T21:10:01.093 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:14.850969+0000", 2024-03-11T21:10:01.093 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0049334069999999999 2024-03-11T21:10:01.093 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.093 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.093 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:10:01.093 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:14.851655+0000", 2024-03-11T21:10:01.094 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00068594299999999995 2024-03-11T21:10:01.094 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.094 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.094 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:10:01.094 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:15.266615+0000", 2024-03-11T21:10:01.095 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.41496038800000001 2024-03-11T21:10:01.095 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.095 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.095 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:10:01.095 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:15.266690+0000", 2024-03-11T21:10:01.095 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5174999999999998e-05 2024-03-11T21:10:01.095 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.095 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.095 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:10:01.095 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:15.266749+0000", 2024-03-11T21:10:01.096 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8356000000000002e-05 2024-03-11T21:10:01.096 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:10:01.096 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:10:01.096 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:10:01.096 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.096 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.096 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.15081.0:94733 9.5 9:aed2b800:::benchmark_data_smithi023_106206_object94732:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e275)", 2024-03-11T21:10:01.096 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T21:04:14.851956+0000", 2024-03-11T21:10:01.096 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 346.24366097199999, 2024-03-11T21:10:01.096 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.43228125699999997, 2024-03-11T21:10:01.096 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:10:01.096 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:10:01.096 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-03-11T21:10:01.096 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.15081", 2024-03-11T21:10:01.096 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.23:0/2841194346", 2024-03-11T21:10:01.097 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 94733 2024-03-11T21:10:01.097 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.097 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:10:01.097 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.097 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:10:01.097 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:14.851956+0000", 2024-03-11T21:10:01.097 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:10:01.097 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.097 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.097 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:10:01.097 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:14.851956+0000", 2024-03-11T21:10:01.097 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:10:01.097 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.097 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.098 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:10:01.098 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:14.851957+0000", 2024-03-11T21:10:01.098 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3399999999999998e-07 2024-03-11T21:10:01.098 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.098 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.098 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:10:01.098 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:14.852043+0000", 2024-03-11T21:10:01.098 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5818999999999999e-05 2024-03-11T21:10:01.098 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.098 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.098 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:10:01.098 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:14.852052+0000", 2024-03-11T21:10:01.098 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4550000000000001e-06 2024-03-11T21:10:01.098 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.098 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.099 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:10:01.099 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:14.852058+0000", 2024-03-11T21:10:01.099 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8370000000000002e-06 2024-03-11T21:10:01.099 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.099 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.099 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:10:01.099 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:15.255757+0000", 2024-03-11T21:10:01.099 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.40369865799999999 2024-03-11T21:10:01.099 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.099 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.099 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:10:01.099 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:15.255994+0000", 2024-03-11T21:10:01.099 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000237495 2024-03-11T21:10:01.099 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.099 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.099 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0,7", 2024-03-11T21:10:01.100 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:15.259981+0000", 2024-03-11T21:10:01.100 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0039865710000000004 2024-03-11T21:10:01.100 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.100 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.100 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:10:01.100 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:15.270685+0000", 2024-03-11T21:10:01.100 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010704284999999999 2024-03-11T21:10:01.100 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.100 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.100 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-03-11T21:10:01.100 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:15.272267+0000", 2024-03-11T21:10:01.100 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0015820459999999999 2024-03-11T21:10:01.100 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.100 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.100 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:10:01.101 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:15.284093+0000", 2024-03-11T21:10:01.101 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.011826058 2024-03-11T21:10:01.101 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.101 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.101 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:10:01.101 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:15.284166+0000", 2024-03-11T21:10:01.101 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3372000000000002e-05 2024-03-11T21:10:01.101 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.101 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.101 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:10:01.101 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:15.284237+0000", 2024-03-11T21:10:01.101 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0926999999999994e-05 2024-03-11T21:10:01.101 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:10:01.101 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:10:01.101 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:10:01.101 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.102 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.102 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.15081.0:94734 9.f 9:f5b9225d:::benchmark_data_smithi023_106206_object94733:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e275)", 2024-03-11T21:10:01.102 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T21:04:14.852175+0000", 2024-03-11T21:10:01.102 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 346.24344179000002, 2024-03-11T21:10:01.102 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.43143531699999998, 2024-03-11T21:10:01.102 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:10:01.102 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:10:01.102 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-03-11T21:10:01.102 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.15081", 2024-03-11T21:10:01.102 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.23:0/2841194346", 2024-03-11T21:10:01.102 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 94734 2024-03-11T21:10:01.102 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.102 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:10:01.102 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.103 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:10:01.103 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:14.852175+0000", 2024-03-11T21:10:01.103 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:10:01.103 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.103 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.103 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:10:01.103 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:14.852175+0000", 2024-03-11T21:10:01.103 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:10:01.103 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.103 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.103 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:10:01.103 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:14.852176+0000", 2024-03-11T21:10:01.104 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9299999999999997e-07 2024-03-11T21:10:01.104 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.104 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.104 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:10:01.104 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:14.852257+0000", 2024-03-11T21:10:01.104 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.0612999999999995e-05 2024-03-11T21:10:01.104 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.104 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.104 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:10:01.104 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:15.254335+0000", 2024-03-11T21:10:01.104 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.402078932 2024-03-11T21:10:01.104 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.104 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.105 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:10:01.105 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:15.254373+0000", 2024-03-11T21:10:01.105 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7585e-05 2024-03-11T21:10:01.105 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.105 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.105 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:10:01.105 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:15.255687+0000", 2024-03-11T21:10:01.105 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0013136300000000001 2024-03-11T21:10:01.105 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.105 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.105 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:10:01.105 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:15.255847+0000", 2024-03-11T21:10:01.105 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0001607 2024-03-11T21:10:01.105 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.105 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.106 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1,3", 2024-03-11T21:10:01.106 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:15.256015+0000", 2024-03-11T21:10:01.106 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000167178 2024-03-11T21:10:01.106 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.106 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.106 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-03-11T21:10:01.106 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:15.264060+0000", 2024-03-11T21:10:01.106 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.008045689 2024-03-11T21:10:01.106 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.106 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.107 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:10:01.107 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:15.274948+0000", 2024-03-11T21:10:01.107 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010887354 2024-03-11T21:10:01.107 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.107 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.107 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:10:01.107 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:15.283479+0000", 2024-03-11T21:10:01.107 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0085318860000000007 2024-03-11T21:10:01.107 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.107 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.107 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:10:01.107 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:15.283557+0000", 2024-03-11T21:10:01.107 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.7155000000000002e-05 2024-03-11T21:10:01.107 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.107 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.108 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:10:01.108 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:15.283611+0000", 2024-03-11T21:10:01.108 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3902e-05 2024-03-11T21:10:01.108 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:10:01.108 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:10:01.108 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:10:01.108 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.108 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.108 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.15081.0:100109 9.a 9:5a1d3607:::benchmark_data_smithi023_106206_object100108:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e276)", 2024-03-11T21:10:01.108 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T21:04:20.256189+0000", 2024-03-11T21:10:01.108 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 340.83942755800001, 2024-03-11T21:10:01.108 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.42929683000000002, 2024-03-11T21:10:01.108 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:10:01.108 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:10:01.108 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-03-11T21:10:01.109 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.15081", 2024-03-11T21:10:01.109 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.23:0/2841194346", 2024-03-11T21:10:01.109 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 100109 2024-03-11T21:10:01.109 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.109 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:10:01.109 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.109 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:10:01.109 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.256189+0000", 2024-03-11T21:10:01.109 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:10:01.109 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.109 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.109 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:10:01.109 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.256189+0000", 2024-03-11T21:10:01.109 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:10:01.109 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.110 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.110 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:10:01.110 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.256190+0000", 2024-03-11T21:10:01.110 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.9800000000000002e-07 2024-03-11T21:10:01.110 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.110 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.110 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:10:01.110 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.256242+0000", 2024-03-11T21:10:01.110 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1220000000000001e-05 2024-03-11T21:10:01.110 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.110 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.110 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:10:01.111 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.256253+0000", 2024-03-11T21:10:01.111 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1856e-05 2024-03-11T21:10:01.111 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.111 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.111 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:10:01.111 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.256260+0000", 2024-03-11T21:10:01.111 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3210000000000001e-06 2024-03-11T21:10:01.111 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.111 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.111 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:10:01.111 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.256529+0000", 2024-03-11T21:10:01.111 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00026921300000000003 2024-03-11T21:10:01.112 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.112 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.112 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:10:01.112 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.256720+0000", 2024-03-11T21:10:01.112 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019099899999999999 2024-03-11T21:10:01.112 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.112 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.112 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1,7", 2024-03-11T21:10:01.112 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.257135+0000", 2024-03-11T21:10:01.112 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00041547900000000001 2024-03-11T21:10:01.112 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.113 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.113 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-03-11T21:10:01.113 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.263529+0000", 2024-03-11T21:10:01.113 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0063933000000000002 2024-03-11T21:10:01.113 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.113 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.113 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:10:01.113 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.265151+0000", 2024-03-11T21:10:01.113 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001622638 2024-03-11T21:10:01.113 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.113 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.113 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:10:01.113 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.685386+0000", 2024-03-11T21:10:01.113 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.42023454500000001 2024-03-11T21:10:01.114 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.114 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.114 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:10:01.114 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.685442+0000", 2024-03-11T21:10:01.114 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5816999999999998e-05 2024-03-11T21:10:01.114 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.114 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.114 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:10:01.114 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.685486+0000", 2024-03-11T21:10:01.114 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4543999999999999e-05 2024-03-11T21:10:01.115 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:10:01.115 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:10:01.115 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:10:01.115 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.115 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.115 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.15081.0:100111 9.6 9:683dd3a3:::benchmark_data_smithi023_106206_object100110:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e276)", 2024-03-11T21:10:01.115 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T21:04:20.256276+0000", 2024-03-11T21:10:01.115 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 340.83934124799998, 2024-03-11T21:10:01.115 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.45657501099999998, 2024-03-11T21:10:01.115 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:10:01.115 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:10:01.115 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-03-11T21:10:01.116 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.15081", 2024-03-11T21:10:01.116 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.23:0/2841194346", 2024-03-11T21:10:01.116 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 100111 2024-03-11T21:10:01.116 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.116 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:10:01.116 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.116 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:10:01.116 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.256276+0000", 2024-03-11T21:10:01.116 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:10:01.116 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.116 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.116 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:10:01.117 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.256276+0000", 2024-03-11T21:10:01.117 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:10:01.117 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.117 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.117 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:10:01.117 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.256276+0000", 2024-03-11T21:10:01.117 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3599999999999999e-07 2024-03-11T21:10:01.117 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.117 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.118 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:10:01.118 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.256316+0000", 2024-03-11T21:10:01.118 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0000000000000003e-05 2024-03-11T21:10:01.118 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.118 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.118 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:10:01.118 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.256324+0000", 2024-03-11T21:10:01.118 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.7929999999999998e-06 2024-03-11T21:10:01.118 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.118 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.119 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:10:01.119 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.256329+0000", 2024-03-11T21:10:01.119 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6240000000000001e-06 2024-03-11T21:10:01.119 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.119 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.119 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:10:01.119 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.256402+0000", 2024-03-11T21:10:01.119 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3787000000000004e-05 2024-03-11T21:10:01.119 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.119 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.119 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:10:01.119 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.256506+0000", 2024-03-11T21:10:01.119 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000103358 2024-03-11T21:10:01.119 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.119 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.120 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1,2", 2024-03-11T21:10:01.120 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.256595+0000", 2024-03-11T21:10:01.120 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.9660999999999994e-05 2024-03-11T21:10:01.120 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.120 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.120 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:10:01.120 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.262460+0000", 2024-03-11T21:10:01.120 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.005864513 2024-03-11T21:10:01.120 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.120 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.120 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-03-11T21:10:01.120 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.263640+0000", 2024-03-11T21:10:01.120 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0011800789999999999 2024-03-11T21:10:01.120 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.120 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.121 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:10:01.121 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.712743+0000", 2024-03-11T21:10:01.121 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.44910283200000001 2024-03-11T21:10:01.121 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.121 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.121 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:10:01.121 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.712802+0000", 2024-03-11T21:10:01.121 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8785e-05 2024-03-11T21:10:01.121 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.121 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.121 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:10:01.121 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.712851+0000", 2024-03-11T21:10:01.121 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9143e-05 2024-03-11T21:10:01.121 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:10:01.121 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:10:01.121 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:10:01.122 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.122 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.122 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.15081.0:100112 9.f 9:f20d529a:::benchmark_data_smithi023_106206_object100111:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e276)", 2024-03-11T21:10:01.122 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T21:04:20.256800+0000", 2024-03-11T21:10:01.122 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 340.83881675100002, 2024-03-11T21:10:01.122 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.42604328299999999, 2024-03-11T21:10:01.122 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:10:01.122 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:10:01.122 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-03-11T21:10:01.122 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.15081", 2024-03-11T21:10:01.122 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.23:0/2841194346", 2024-03-11T21:10:01.122 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 100112 2024-03-11T21:10:01.122 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.122 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:10:01.123 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.123 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:10:01.123 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.256800+0000", 2024-03-11T21:10:01.123 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:10:01.123 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.123 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.123 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:10:01.123 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.256800+0000", 2024-03-11T21:10:01.123 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:10:01.123 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.123 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.123 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:10:01.123 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.256818+0000", 2024-03-11T21:10:01.123 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8084000000000002e-05 2024-03-11T21:10:01.124 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.124 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.124 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:10:01.124 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.256878+0000", 2024-03-11T21:10:01.124 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9679000000000002e-05 2024-03-11T21:10:01.124 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.124 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.124 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:10:01.124 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.256887+0000", 2024-03-11T21:10:01.124 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4210000000000001e-06 2024-03-11T21:10:01.124 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.124 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.124 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:10:01.124 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.256894+0000", 2024-03-11T21:10:01.124 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9299999999999997e-06 2024-03-11T21:10:01.124 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.125 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.125 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:10:01.125 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.261963+0000", 2024-03-11T21:10:01.125 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0050689519999999998 2024-03-11T21:10:01.125 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.125 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.125 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:10:01.125 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.262138+0000", 2024-03-11T21:10:01.125 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000175021 2024-03-11T21:10:01.125 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.125 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.125 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1,3", 2024-03-11T21:10:01.125 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.262268+0000", 2024-03-11T21:10:01.125 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012967499999999999 2024-03-11T21:10:01.125 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.126 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.126 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-03-11T21:10:01.126 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.265345+0000", 2024-03-11T21:10:01.126 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030769719999999999 2024-03-11T21:10:01.126 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.126 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.126 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:10:01.126 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.265750+0000", 2024-03-11T21:10:01.126 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00040489600000000002 2024-03-11T21:10:01.126 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.126 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.126 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:10:01.126 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.682753+0000", 2024-03-11T21:10:01.126 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.41700277099999999 2024-03-11T21:10:01.126 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.126 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.127 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:10:01.127 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.682794+0000", 2024-03-11T21:10:01.127 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1459999999999999e-05 2024-03-11T21:10:01.127 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.127 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.127 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:10:01.127 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.682844+0000", 2024-03-11T21:10:01.127 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9422000000000002e-05 2024-03-11T21:10:01.127 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:10:01.127 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:10:01.127 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:10:01.127 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.127 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.127 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.15081.0:100114 9.6 9:6130f34f:::benchmark_data_smithi023_106206_object100113:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e276)", 2024-03-11T21:10:01.127 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T21:04:20.256950+0000", 2024-03-11T21:10:01.128 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 340.83866676399998, 2024-03-11T21:10:01.128 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.45608137599999998, 2024-03-11T21:10:01.128 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:10:01.128 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:10:01.128 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-03-11T21:10:01.128 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.15081", 2024-03-11T21:10:01.128 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.23:0/2841194346", 2024-03-11T21:10:01.128 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 100114 2024-03-11T21:10:01.128 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.128 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:10:01.128 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.128 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:10:01.128 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.256950+0000", 2024-03-11T21:10:01.128 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:10:01.128 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.128 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.129 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:10:01.129 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.256950+0000", 2024-03-11T21:10:01.129 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:10:01.129 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.129 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.129 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:10:01.129 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.256951+0000", 2024-03-11T21:10:01.129 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0699999999999996e-07 2024-03-11T21:10:01.129 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.129 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.129 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:10:01.129 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.257017+0000", 2024-03-11T21:10:01.129 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6227000000000001e-05 2024-03-11T21:10:01.129 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.129 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.129 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:10:01.130 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.257025+0000", 2024-03-11T21:10:01.130 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3520000000000001e-06 2024-03-11T21:10:01.130 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.130 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.130 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:10:01.130 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.257030+0000", 2024-03-11T21:10:01.130 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5520000000000004e-06 2024-03-11T21:10:01.130 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.130 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.130 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:10:01.130 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.259177+0000", 2024-03-11T21:10:01.130 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0021465799999999999 2024-03-11T21:10:01.130 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.130 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.130 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:10:01.131 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.259311+0000", 2024-03-11T21:10:01.131 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000134108 2024-03-11T21:10:01.131 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.131 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.131 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1,2", 2024-03-11T21:10:01.131 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.259458+0000", 2024-03-11T21:10:01.131 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014676700000000001 2024-03-11T21:10:01.131 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.131 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.131 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:10:01.131 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.262496+0000", 2024-03-11T21:10:01.131 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030383390000000001 2024-03-11T21:10:01.131 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.131 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.132 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-03-11T21:10:01.132 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.263646+0000", 2024-03-11T21:10:01.132 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0011498179999999999 2024-03-11T21:10:01.132 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.132 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.132 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:10:01.132 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.712902+0000", 2024-03-11T21:10:01.132 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.44925606000000001 2024-03-11T21:10:01.132 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.132 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.132 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:10:01.132 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.712996+0000", 2024-03-11T21:10:01.132 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.3765999999999996e-05 2024-03-11T21:10:01.132 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.132 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.133 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:10:01.133 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.713032+0000", 2024-03-11T21:10:01.133 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.6100000000000003e-05 2024-03-11T21:10:01.133 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:10:01.133 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:10:01.133 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:10:01.133 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.133 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.133 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.15081.0:100116 9.a 9:5dbcef92:::benchmark_data_smithi023_106206_object100115:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e276)", 2024-03-11T21:10:01.133 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T21:04:20.257187+0000", 2024-03-11T21:10:01.133 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 340.838430071, 2024-03-11T21:10:01.133 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.428418509, 2024-03-11T21:10:01.133 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:10:01.133 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:10:01.133 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-03-11T21:10:01.133 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.15081", 2024-03-11T21:10:01.134 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.23:0/2841194346", 2024-03-11T21:10:01.134 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 100116 2024-03-11T21:10:01.134 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.134 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:10:01.134 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.134 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:10:01.134 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.257187+0000", 2024-03-11T21:10:01.134 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:10:01.134 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.134 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.134 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:10:01.134 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.257187+0000", 2024-03-11T21:10:01.134 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:10:01.134 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.134 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.135 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:10:01.135 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.257188+0000", 2024-03-11T21:10:01.135 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4900000000000005e-07 2024-03-11T21:10:01.135 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.135 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.135 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:10:01.135 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.257297+0000", 2024-03-11T21:10:01.135 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000109429 2024-03-11T21:10:01.135 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.135 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.135 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:10:01.135 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.257307+0000", 2024-03-11T21:10:01.135 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0097e-05 2024-03-11T21:10:01.135 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.135 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.136 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:10:01.136 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.257315+0000", 2024-03-11T21:10:01.136 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4329999999999998e-06 2024-03-11T21:10:01.136 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.136 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.136 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:10:01.136 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.260550+0000", 2024-03-11T21:10:01.136 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0032356860000000002 2024-03-11T21:10:01.136 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.136 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.136 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:10:01.136 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.260668+0000", 2024-03-11T21:10:01.136 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000117458 2024-03-11T21:10:01.136 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.136 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.136 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1,7", 2024-03-11T21:10:01.137 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.260831+0000", 2024-03-11T21:10:01.137 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000163282 2024-03-11T21:10:01.137 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.137 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.137 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-03-11T21:10:01.137 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.263654+0000", 2024-03-11T21:10:01.137 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002823239 2024-03-11T21:10:01.137 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.137 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.137 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:10:01.137 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.265186+0000", 2024-03-11T21:10:01.137 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0015315800000000001 2024-03-11T21:10:01.137 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.137 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.137 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:10:01.137 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.685526+0000", 2024-03-11T21:10:01.138 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.420340573 2024-03-11T21:10:01.138 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.138 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.138 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:10:01.138 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.685567+0000", 2024-03-11T21:10:01.138 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0864e-05 2024-03-11T21:10:01.138 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.138 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.138 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:10:01.138 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.685605+0000", 2024-03-11T21:10:01.138 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8118999999999999e-05 2024-03-11T21:10:01.138 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:10:01.138 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:10:01.138 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:10:01.138 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.139 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.139 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.15081.0:100121 9.a 9:53a9f732:::benchmark_data_smithi023_106206_object100120:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e276)", 2024-03-11T21:10:01.139 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T21:04:20.265494+0000", 2024-03-11T21:10:01.139 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 340.83012288399999, 2024-03-11T21:10:01.139 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.42204219599999998, 2024-03-11T21:10:01.139 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:10:01.139 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:10:01.139 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-03-11T21:10:01.139 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.15081", 2024-03-11T21:10:01.139 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.23:0/2841194346", 2024-03-11T21:10:01.139 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 100121 2024-03-11T21:10:01.139 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.139 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:10:01.139 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.139 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:10:01.139 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.265494+0000", 2024-03-11T21:10:01.140 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:10:01.140 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.140 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.140 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:10:01.140 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.265494+0000", 2024-03-11T21:10:01.140 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:10:01.140 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.140 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.140 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:10:01.140 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.265495+0000", 2024-03-11T21:10:01.140 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.139e-06 2024-03-11T21:10:01.140 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.140 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.140 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:10:01.140 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.265560+0000", 2024-03-11T21:10:01.140 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4920999999999997e-05 2024-03-11T21:10:01.141 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.141 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.141 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:10:01.141 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.265578+0000", 2024-03-11T21:10:01.141 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7520000000000002e-05 2024-03-11T21:10:01.141 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.141 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.141 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:10:01.141 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.265583+0000", 2024-03-11T21:10:01.141 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1710000000000001e-06 2024-03-11T21:10:01.141 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.141 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.142 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:10:01.142 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.265628+0000", 2024-03-11T21:10:01.142 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4708e-05 2024-03-11T21:10:01.142 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.142 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.142 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:10:01.142 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.265730+0000", 2024-03-11T21:10:01.142 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000102796 2024-03-11T21:10:01.142 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.142 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.142 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1,7", 2024-03-11T21:10:01.142 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.265835+0000", 2024-03-11T21:10:01.143 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010480399999999999 2024-03-11T21:10:01.143 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.143 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.143 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-03-11T21:10:01.143 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.268130+0000", 2024-03-11T21:10:01.143 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0022952929999999999 2024-03-11T21:10:01.143 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.143 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.143 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:10:01.143 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.270694+0000", 2024-03-11T21:10:01.143 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0025635800000000002 2024-03-11T21:10:01.143 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.143 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.143 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:10:01.144 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.687437+0000", 2024-03-11T21:10:01.144 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.41674264300000002 2024-03-11T21:10:01.144 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.144 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.144 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:10:01.144 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.687491+0000", 2024-03-11T21:10:01.144 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4503000000000003e-05 2024-03-11T21:10:01.144 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.144 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.144 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:10:01.144 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.687536+0000", 2024-03-11T21:10:01.144 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5117999999999997e-05 2024-03-11T21:10:01.145 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:10:01.145 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:10:01.145 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:10:01.145 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.145 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.145 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.15081.0:132286 9.f 9:f95388ac:::benchmark_data_smithi023_106206_object21140:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e284)", 2024-03-11T21:10:01.145 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T21:04:54.385987+0000", 2024-03-11T21:10:01.145 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 306.70962962800002, 2024-03-11T21:10:01.145 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.42386572, 2024-03-11T21:10:01.145 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:10:01.145 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:10:01.145 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-03-11T21:10:01.146 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.15081", 2024-03-11T21:10:01.146 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.23:0/2841194346", 2024-03-11T21:10:01.146 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 132286 2024-03-11T21:10:01.146 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.146 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:10:01.146 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.146 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:10:01.146 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:54.385987+0000", 2024-03-11T21:10:01.146 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:10:01.146 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.146 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.146 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:10:01.146 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:54.385987+0000", 2024-03-11T21:10:01.147 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:10:01.147 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.147 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.147 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:10:01.147 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:54.385988+0000", 2024-03-11T21:10:01.147 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9200000000000001e-07 2024-03-11T21:10:01.147 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.147 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.147 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:10:01.147 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:54.385991+0000", 2024-03-11T21:10:01.147 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9909999999999998e-06 2024-03-11T21:10:01.147 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.148 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.148 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:10:01.148 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:54.385999+0000", 2024-03-11T21:10:01.148 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9959999999999993e-06 2024-03-11T21:10:01.148 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.148 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.148 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:10:01.148 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:54.386005+0000", 2024-03-11T21:10:01.148 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9549999999999997e-06 2024-03-11T21:10:01.148 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.148 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.148 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:10:01.148 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:54.386052+0000", 2024-03-11T21:10:01.148 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6918e-05 2024-03-11T21:10:01.149 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.149 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.149 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:10:01.149 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:54.386163+0000", 2024-03-11T21:10:01.149 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000111264 2024-03-11T21:10:01.149 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.149 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.149 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1,3", 2024-03-11T21:10:01.149 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:54.386267+0000", 2024-03-11T21:10:01.149 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010361699999999999 2024-03-11T21:10:01.149 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.150 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.150 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-03-11T21:10:01.150 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:54.393844+0000", 2024-03-11T21:10:01.150 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0075774950000000001 2024-03-11T21:10:01.150 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.150 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.150 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:10:01.150 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:54.395305+0000", 2024-03-11T21:10:01.150 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001460952 2024-03-11T21:10:01.150 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.150 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.151 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:10:01.151 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:54.809751+0000", 2024-03-11T21:10:01.151 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.41444547799999998 2024-03-11T21:10:01.151 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.151 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.151 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:10:01.151 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:54.809806+0000", 2024-03-11T21:10:01.151 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5699000000000003e-05 2024-03-11T21:10:01.151 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.151 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.151 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:10:01.151 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:54.809853+0000", 2024-03-11T21:10:01.151 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6762999999999999e-05 2024-03-11T21:10:01.152 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:10:01.152 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:10:01.152 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:10:01.152 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.152 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.152 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.24281.0:3503 4.d 4:b4812045:::notify.4:head [watch reconnect cookie 94110942031104 gen 11] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e319)", 2024-03-11T21:10:01.152 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T21:08:04.307325+0000", 2024-03-11T21:10:01.152 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 116.78829204199999, 2024-03-11T21:10:01.152 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.000399638, 2024-03-11T21:10:01.153 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:10:01.153 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-03-11T21:10:01.153 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-03-11T21:10:01.153 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.24281", 2024-03-11T21:10:01.153 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.23:0/2185274774", 2024-03-11T21:10:01.153 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3503 2024-03-11T21:10:01.153 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.153 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:10:01.153 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.153 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:10:01.154 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:08:04.307325+0000", 2024-03-11T21:10:01.154 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:10:01.154 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.154 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.154 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:10:01.154 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:08:04.307325+0000", 2024-03-11T21:10:01.154 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:10:01.154 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.154 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.154 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:10:01.154 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:08:04.307327+0000", 2024-03-11T21:10:01.154 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9099999999999999e-06 2024-03-11T21:10:01.154 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.154 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.155 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:10:01.155 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:08:04.307337+0000", 2024-03-11T21:10:01.155 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8349999999999999e-06 2024-03-11T21:10:01.155 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.155 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.155 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:10:01.155 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:08:04.307350+0000", 2024-03-11T21:10:01.155 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3703000000000001e-05 2024-03-11T21:10:01.155 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.155 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.155 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:10:01.155 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:08:04.307362+0000", 2024-03-11T21:10:01.155 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1415e-05 2024-03-11T21:10:01.155 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.155 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.155 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:10:01.156 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:08:04.307428+0000", 2024-03-11T21:10:01.156 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6039999999999998e-05 2024-03-11T21:10:01.156 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.156 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.156 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-03-11T21:10:01.156 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:08:04.307440+0000", 2024-03-11T21:10:01.156 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2394e-05 2024-03-11T21:10:01.156 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.156 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.156 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:10:01.156 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:08:05.299740+0000", 2024-03-11T21:10:01.156 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.99229994600000004 2024-03-11T21:10:01.156 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.156 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.157 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:10:01.157 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:08:05.299924+0000", 2024-03-11T21:10:01.157 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000183654 2024-03-11T21:10:01.157 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.157 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.157 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:10:01.157 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:08:05.307725+0000", 2024-03-11T21:10:01.157 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0078007409999999999 2024-03-11T21:10:01.157 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:10:01.157 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:10:01.157 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:10:01.157 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.157 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.157 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.15291.0:45784 10.3 10:c7ed7d5d:::benchmark_data_smithi023_124944_object45783:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e319)", 2024-03-11T21:10:01.157 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T21:08:04.423587+0000", 2024-03-11T21:10:01.158 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 116.672029693, 2024-03-11T21:10:01.158 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.88668345299999995, 2024-03-11T21:10:01.158 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:10:01.158 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:10:01.158 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-03-11T21:10:01.158 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.15291", 2024-03-11T21:10:01.158 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.23:0/729656300", 2024-03-11T21:10:01.158 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 45784 2024-03-11T21:10:01.158 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.158 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:10:01.158 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.158 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:10:01.158 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:08:04.423587+0000", 2024-03-11T21:10:01.158 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:10:01.158 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.159 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.159 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:10:01.159 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:08:04.423587+0000", 2024-03-11T21:10:01.159 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:10:01.159 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.159 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.159 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:10:01.159 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:08:04.423589+0000", 2024-03-11T21:10:01.159 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8250000000000001e-06 2024-03-11T21:10:01.159 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.159 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.159 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:10:01.159 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:08:04.423658+0000", 2024-03-11T21:10:01.159 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8998000000000006e-05 2024-03-11T21:10:01.159 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.160 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.160 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:10:01.160 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:08:04.423669+0000", 2024-03-11T21:10:01.160 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0864e-05 2024-03-11T21:10:01.160 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.160 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.160 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:10:01.160 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:08:04.423677+0000", 2024-03-11T21:10:01.160 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5079999999999999e-06 2024-03-11T21:10:01.160 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.160 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.160 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:10:01.160 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:08:04.423750+0000", 2024-03-11T21:10:01.160 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3149e-05 2024-03-11T21:10:01.160 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.161 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.161 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-03-11T21:10:01.161 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:08:04.423764+0000", 2024-03-11T21:10:01.161 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4817000000000001e-05 2024-03-11T21:10:01.161 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.161 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.161 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:10:01.161 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:08:05.301970+0000", 2024-03-11T21:10:01.161 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.878205915 2024-03-11T21:10:01.161 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.161 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.161 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:10:01.161 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:08:05.303284+0000", 2024-03-11T21:10:01.161 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0013135239999999999 2024-03-11T21:10:01.161 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.161 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.162 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2,3", 2024-03-11T21:10:01.162 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:08:05.303482+0000", 2024-03-11T21:10:01.162 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000198402 2024-03-11T21:10:01.162 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.162 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.162 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-03-11T21:10:01.162 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:08:05.307393+0000", 2024-03-11T21:10:01.162 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0039102069999999997 2024-03-11T21:10:01.162 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.162 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.162 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:10:01.162 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:08:05.310121+0000", 2024-03-11T21:10:01.162 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0027289179999999999 2024-03-11T21:10:01.162 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.162 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.163 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:10:01.163 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:08:05.310200+0000", 2024-03-11T21:10:01.163 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.8106000000000001e-05 2024-03-11T21:10:01.163 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.163 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.163 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:10:01.163 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:08:05.310239+0000", 2024-03-11T21:10:01.163 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9161e-05 2024-03-11T21:10:01.163 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.163 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.163 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:10:01.163 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:08:05.310271+0000", 2024-03-11T21:10:01.163 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2058999999999999e-05 2024-03-11T21:10:01.163 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:10:01.164 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:10:01.164 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:10:01.164 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.164 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.164 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.15291.0:45813 10.3 10:cd016041:::benchmark_data_smithi023_124944_object45812:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e319)", 2024-03-11T21:10:01.164 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T21:08:04.444961+0000", 2024-03-11T21:10:01.164 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 116.650655883, 2024-03-11T21:10:01.164 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.87241040000000003, 2024-03-11T21:10:01.164 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:10:01.164 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:10:01.164 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-03-11T21:10:01.164 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.15291", 2024-03-11T21:10:01.164 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.23:0/729656300", 2024-03-11T21:10:01.164 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 45813 2024-03-11T21:10:01.164 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.164 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:10:01.165 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.165 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:10:01.165 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:08:04.444961+0000", 2024-03-11T21:10:01.165 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:10:01.165 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.165 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.165 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:10:01.165 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:08:04.444961+0000", 2024-03-11T21:10:01.165 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:10:01.165 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.165 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.165 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:10:01.165 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:08:04.444962+0000", 2024-03-11T21:10:01.165 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8700000000000004e-07 2024-03-11T21:10:01.165 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.165 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.166 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:10:01.166 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:08:04.444991+0000", 2024-03-11T21:10:01.166 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8581e-05 2024-03-11T21:10:01.166 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.166 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.166 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:10:01.166 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:08:04.445005+0000", 2024-03-11T21:10:01.166 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4762e-05 2024-03-11T21:10:01.166 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.166 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.166 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:10:01.167 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:08:04.445013+0000", 2024-03-11T21:10:01.167 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.8879999999999997e-06 2024-03-11T21:10:01.167 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.167 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.167 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:10:01.167 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:08:04.445094+0000", 2024-03-11T21:10:01.168 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.0790000000000004e-05 2024-03-11T21:10:01.168 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.168 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.168 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-03-11T21:10:01.168 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:08:04.445104+0000", 2024-03-11T21:10:01.168 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6099999999999995e-06 2024-03-11T21:10:01.168 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.168 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.168 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:10:01.168 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:08:05.305294+0000", 2024-03-11T21:10:01.168 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.86019053099999998 2024-03-11T21:10:01.168 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.168 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.168 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:10:01.169 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:08:05.305448+0000", 2024-03-11T21:10:01.169 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015348099999999999 2024-03-11T21:10:01.169 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.169 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.169 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2,3", 2024-03-11T21:10:01.169 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:08:05.305601+0000", 2024-03-11T21:10:01.169 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015284000000000001 2024-03-11T21:10:01.169 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.169 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.169 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:10:01.169 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:08:05.310317+0000", 2024-03-11T21:10:01.169 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0047161449999999997 2024-03-11T21:10:01.169 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.169 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.169 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-03-11T21:10:01.169 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:08:05.310331+0000", 2024-03-11T21:10:01.170 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4521999999999999e-05 2024-03-11T21:10:01.170 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.170 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.170 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:10:01.170 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:08:05.317246+0000", 2024-03-11T21:10:01.170 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0069142789999999997 2024-03-11T21:10:01.170 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.170 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.170 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:10:01.170 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:08:05.317319+0000", 2024-03-11T21:10:01.170 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3467999999999995e-05 2024-03-11T21:10:01.170 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.170 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.170 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:10:01.170 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:08:05.317372+0000", 2024-03-11T21:10:01.171 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2516000000000003e-05 2024-03-11T21:10:01.171 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:10:01.171 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:10:01.171 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:10:01.171 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.171 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.171 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.15291.0:45836 10.3 10:c103eff8:::benchmark_data_smithi023_124944_object45835:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e319)", 2024-03-11T21:10:01.171 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T21:08:04.463824+0000", 2024-03-11T21:10:01.171 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 116.631793403, 2024-03-11T21:10:01.171 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.85371886200000002, 2024-03-11T21:10:01.171 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:10:01.171 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:10:01.171 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-03-11T21:10:01.171 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.15291", 2024-03-11T21:10:01.172 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.23:0/729656300", 2024-03-11T21:10:01.172 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 45836 2024-03-11T21:10:01.172 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.172 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:10:01.172 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.172 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:10:01.172 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:08:04.463824+0000", 2024-03-11T21:10:01.172 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:10:01.172 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.172 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.172 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:10:01.172 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:08:04.463824+0000", 2024-03-11T21:10:01.172 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:10:01.172 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.172 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.173 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:10:01.173 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:08:04.463825+0000", 2024-03-11T21:10:01.173 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5600000000000001e-06 2024-03-11T21:10:01.173 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.175 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.175 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:10:01.175 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:08:04.463888+0000", 2024-03-11T21:10:01.175 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2822000000000003e-05 2024-03-11T21:10:01.176 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.176 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.177 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:10:01.178 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:08:04.463905+0000", 2024-03-11T21:10:01.178 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6872e-05 2024-03-11T21:10:01.178 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.178 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.178 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:10:01.178 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:08:04.463912+0000", 2024-03-11T21:10:01.179 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4780000000000002e-06 2024-03-11T21:10:01.179 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.179 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.179 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:10:01.179 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:08:04.463987+0000", 2024-03-11T21:10:01.179 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4412000000000005e-05 2024-03-11T21:10:01.179 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.179 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.180 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-03-11T21:10:01.180 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:08:04.464000+0000", 2024-03-11T21:10:01.180 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2775000000000001e-05 2024-03-11T21:10:01.180 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.180 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.180 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:10:01.180 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:08:05.307462+0000", 2024-03-11T21:10:01.180 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.84346295800000004 2024-03-11T21:10:01.180 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.180 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.180 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:10:01.180 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:08:05.307551+0000", 2024-03-11T21:10:01.181 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8609000000000005e-05 2024-03-11T21:10:01.181 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.181 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.181 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2,3", 2024-03-11T21:10:01.181 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:08:05.307696+0000", 2024-03-11T21:10:01.181 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000144566 2024-03-11T21:10:01.181 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.181 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.181 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-03-11T21:10:01.181 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:08:05.310338+0000", 2024-03-11T21:10:01.181 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002642625 2024-03-11T21:10:01.181 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.181 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.181 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:10:01.181 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:08:05.314685+0000", 2024-03-11T21:10:01.182 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.004346557 2024-03-11T21:10:01.182 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.182 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.182 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:10:01.182 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:08:05.317441+0000", 2024-03-11T21:10:01.182 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0027558320000000002 2024-03-11T21:10:01.182 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.182 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.182 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:10:01.182 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:08:05.317500+0000", 2024-03-11T21:10:01.182 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9753999999999997e-05 2024-03-11T21:10:01.183 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.183 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.183 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:10:01.183 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:08:05.317542+0000", 2024-03-11T21:10:01.183 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2042000000000002e-05 2024-03-11T21:10:01.183 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:10:01.183 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:10:01.183 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:10:01.184 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.184 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.184 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.15291.0:45893 10.3 10:cbae7b72:::benchmark_data_smithi023_124944_object45892:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e319)", 2024-03-11T21:10:01.184 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T21:08:04.519244+0000", 2024-03-11T21:10:01.184 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 116.57637299300001, 2024-03-11T21:10:01.184 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.80627590699999996, 2024-03-11T21:10:01.184 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:10:01.184 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:10:01.185 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-03-11T21:10:01.185 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.15291", 2024-03-11T21:10:01.185 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.23:0/729656300", 2024-03-11T21:10:01.185 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 45893 2024-03-11T21:10:01.185 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.186 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:10:01.186 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.186 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:10:01.186 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:08:04.519244+0000", 2024-03-11T21:10:01.186 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:10:01.186 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.186 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.186 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:10:01.187 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:08:04.519244+0000", 2024-03-11T21:10:01.187 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:10:01.187 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.187 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.187 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:10:01.187 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:08:04.519246+0000", 2024-03-11T21:10:01.187 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.491e-06 2024-03-11T21:10:01.187 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.187 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.188 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:10:01.188 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:08:04.519279+0000", 2024-03-11T21:10:01.188 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3819999999999998e-05 2024-03-11T21:10:01.188 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.188 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.188 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:10:01.188 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:08:04.519300+0000", 2024-03-11T21:10:01.188 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0740000000000001e-05 2024-03-11T21:10:01.189 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.189 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.189 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:10:01.189 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:08:04.519311+0000", 2024-03-11T21:10:01.189 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1042e-05 2024-03-11T21:10:01.189 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.189 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.189 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:10:01.189 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:08:04.519392+0000", 2024-03-11T21:10:01.190 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1359000000000005e-05 2024-03-11T21:10:01.190 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.190 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.190 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-03-11T21:10:01.190 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:08:04.519405+0000", 2024-03-11T21:10:01.190 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2327000000000001e-05 2024-03-11T21:10:01.190 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.190 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.191 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:10:01.191 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:08:05.308734+0000", 2024-03-11T21:10:01.191 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.78932875499999999 2024-03-11T21:10:01.191 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.191 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.191 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:10:01.191 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:08:05.308851+0000", 2024-03-11T21:10:01.191 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000117458 2024-03-11T21:10:01.191 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.192 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.192 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2,3", 2024-03-11T21:10:01.192 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:08:05.309013+0000", 2024-03-11T21:10:01.192 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016154400000000001 2024-03-11T21:10:01.192 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.192 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.192 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-03-11T21:10:01.192 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:08:05.310344+0000", 2024-03-11T21:10:01.192 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0013312790000000001 2024-03-11T21:10:01.193 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.193 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.193 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:10:01.193 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:08:05.314862+0000", 2024-03-11T21:10:01.193 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0045184149999999996 2024-03-11T21:10:01.193 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.193 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.193 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:10:01.193 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:08:05.325442+0000", 2024-03-11T21:10:01.194 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010579663 2024-03-11T21:10:01.194 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.194 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.194 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:10:01.194 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:08:05.325484+0000", 2024-03-11T21:10:01.194 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1881000000000001e-05 2024-03-11T21:10:01.194 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.194 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.194 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:10:01.195 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:08:05.325520+0000", 2024-03-11T21:10:01.195 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.6133000000000001e-05 2024-03-11T21:10:01.195 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:10:01.195 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:10:01.195 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:10:01.195 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.195 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.195 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.15291.0:151207 10.9 10:9401ef5c:::benchmark_data_smithi023_124944_object42195:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e348)", 2024-03-11T21:10:01.195 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T21:09:54.894845+0000", 2024-03-11T21:10:01.196 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 6.2007716039999998, 2024-03-11T21:10:01.196 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.63431103799999999, 2024-03-11T21:10:01.196 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:10:01.196 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:10:01.196 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-03-11T21:10:01.196 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.15291", 2024-03-11T21:10:01.196 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.23:0/729656300", 2024-03-11T21:10:01.196 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 151207 2024-03-11T21:10:01.196 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.197 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:10:01.197 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.197 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:10:01.197 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.894845+0000", 2024-03-11T21:10:01.197 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:10:01.197 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.197 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.197 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:10:01.197 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.894845+0000", 2024-03-11T21:10:01.198 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:10:01.198 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.198 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.198 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:10:01.198 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.894846+0000", 2024-03-11T21:10:01.198 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0200000000000001e-07 2024-03-11T21:10:01.198 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.198 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.199 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:10:01.199 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.894850+0000", 2024-03-11T21:10:01.199 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0980000000000004e-06 2024-03-11T21:10:01.199 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.199 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.199 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:10:01.199 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.894859+0000", 2024-03-11T21:10:01.199 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8119999999999997e-06 2024-03-11T21:10:01.199 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.199 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.199 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:10:01.199 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.894865+0000", 2024-03-11T21:10:01.199 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4909999999999996e-06 2024-03-11T21:10:01.200 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.200 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.200 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:10:01.200 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.900458+0000", 2024-03-11T21:10:01.200 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0055930090000000003 2024-03-11T21:10:01.200 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.200 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.200 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:10:01.200 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.900560+0000", 2024-03-11T21:10:01.200 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010234999999999999 2024-03-11T21:10:01.200 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.200 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.200 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2,7", 2024-03-11T21:10:01.201 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.900664+0000", 2024-03-11T21:10:01.201 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010407399999999999 2024-03-11T21:10:01.201 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.201 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.201 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-03-11T21:10:01.201 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.903593+0000", 2024-03-11T21:10:01.201 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002929301 2024-03-11T21:10:01.201 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.201 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.201 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:10:01.201 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.914541+0000", 2024-03-11T21:10:01.201 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010947389 2024-03-11T21:10:01.201 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.201 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.201 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:10:01.202 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:55.529049+0000", 2024-03-11T21:10:01.202 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.61450856099999995 2024-03-11T21:10:01.202 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.202 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.202 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:10:01.202 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:55.529105+0000", 2024-03-11T21:10:01.202 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5501999999999998e-05 2024-03-11T21:10:01.202 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.202 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.202 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:10:01.202 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:55.529156+0000", 2024-03-11T21:10:01.202 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1749000000000002e-05 2024-03-11T21:10:01.202 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:10:01.203 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:10:01.203 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:10:01.203 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.203 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.203 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.15291.0:151216 10.9 10:9dcd14dc:::benchmark_data_smithi023_124944_object42204:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e348)", 2024-03-11T21:10:01.203 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T21:09:54.904477+0000", 2024-03-11T21:10:01.203 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 6.1911397340000001, 2024-03-11T21:10:01.203 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.62534909100000002, 2024-03-11T21:10:01.203 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:10:01.203 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:10:01.203 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-03-11T21:10:01.204 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.15291", 2024-03-11T21:10:01.204 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.23:0/729656300", 2024-03-11T21:10:01.204 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 151216 2024-03-11T21:10:01.204 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.204 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:10:01.204 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.204 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:10:01.204 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.904477+0000", 2024-03-11T21:10:01.204 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:10:01.204 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.204 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.204 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:10:01.204 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.904477+0000", 2024-03-11T21:10:01.205 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:10:01.205 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.205 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.205 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:10:01.205 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.904479+0000", 2024-03-11T21:10:01.205 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3540000000000001e-06 2024-03-11T21:10:01.205 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.205 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.205 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:10:01.205 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.904484+0000", 2024-03-11T21:10:01.205 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9130000000000002e-06 2024-03-11T21:10:01.205 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.205 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.205 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:10:01.205 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.904494+0000", 2024-03-11T21:10:01.206 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0655999999999999e-05 2024-03-11T21:10:01.206 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.206 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.206 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:10:01.206 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.904502+0000", 2024-03-11T21:10:01.206 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6790000000000005e-06 2024-03-11T21:10:01.206 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.206 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.206 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:10:01.206 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.904553+0000", 2024-03-11T21:10:01.206 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1517999999999997e-05 2024-03-11T21:10:01.206 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.206 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.206 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:10:01.207 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.904679+0000", 2024-03-11T21:10:01.207 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012542599999999999 2024-03-11T21:10:01.207 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.207 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.207 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2,7", 2024-03-11T21:10:01.207 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.904808+0000", 2024-03-11T21:10:01.207 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012926999999999999 2024-03-11T21:10:01.207 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.207 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.207 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-03-11T21:10:01.207 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.914512+0000", 2024-03-11T21:10:01.207 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.009704318 2024-03-11T21:10:01.207 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.207 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.207 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:10:01.208 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.914593+0000", 2024-03-11T21:10:01.208 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.0570999999999998e-05 2024-03-11T21:10:01.208 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.208 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.208 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:10:01.208 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:55.529683+0000", 2024-03-11T21:10:01.208 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.61509043699999999 2024-03-11T21:10:01.208 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.208 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.208 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:10:01.208 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:55.529779+0000", 2024-03-11T21:10:01.208 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5272999999999994e-05 2024-03-11T21:10:01.208 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.208 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.208 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:10:01.209 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:55.529826+0000", 2024-03-11T21:10:01.209 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7676000000000002e-05 2024-03-11T21:10:01.209 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:10:01.209 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:10:01.209 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:10:01.209 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.209 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.209 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.15291.0:151223 10.9 10:91839f98:::benchmark_data_smithi023_124944_object42211:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e348)", 2024-03-11T21:10:01.209 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T21:09:54.907846+0000", 2024-03-11T21:10:01.209 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 6.1877707119999998, 2024-03-11T21:10:01.209 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.622239184, 2024-03-11T21:10:01.209 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:10:01.209 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:10:01.209 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-03-11T21:10:01.209 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.15291", 2024-03-11T21:10:01.210 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.23:0/729656300", 2024-03-11T21:10:01.210 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 151223 2024-03-11T21:10:01.210 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.210 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:10:01.210 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.210 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:10:01.210 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.907846+0000", 2024-03-11T21:10:01.210 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:10:01.210 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.210 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.210 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:10:01.210 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.907846+0000", 2024-03-11T21:10:01.210 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:10:01.211 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.211 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.211 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:10:01.211 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.907847+0000", 2024-03-11T21:10:01.211 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3300000000000001e-07 2024-03-11T21:10:01.211 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.211 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.211 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:10:01.211 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.907851+0000", 2024-03-11T21:10:01.211 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.16e-06 2024-03-11T21:10:01.211 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.211 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.211 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:10:01.211 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.907861+0000", 2024-03-11T21:10:01.211 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.9529999999999994e-06 2024-03-11T21:10:01.212 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.212 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.212 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:10:01.212 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.907868+0000", 2024-03-11T21:10:01.212 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3000000000000004e-06 2024-03-11T21:10:01.212 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.212 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.212 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:10:01.212 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.911667+0000", 2024-03-11T21:10:01.212 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0037984960000000002 2024-03-11T21:10:01.212 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.212 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.212 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:10:01.212 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.911777+0000", 2024-03-11T21:10:01.212 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000109931 2024-03-11T21:10:01.213 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.213 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.213 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2,7", 2024-03-11T21:10:01.213 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.911890+0000", 2024-03-11T21:10:01.213 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011264000000000001 2024-03-11T21:10:01.213 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.213 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.213 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-03-11T21:10:01.213 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.914608+0000", 2024-03-11T21:10:01.213 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0027184710000000001 2024-03-11T21:10:01.213 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.213 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.213 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:10:01.213 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.917016+0000", 2024-03-11T21:10:01.213 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002407944 2024-03-11T21:10:01.214 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.214 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.214 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:10:01.214 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:55.529985+0000", 2024-03-11T21:10:01.214 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.61296886799999994 2024-03-11T21:10:01.214 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.214 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.214 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:10:01.214 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:55.530044+0000", 2024-03-11T21:10:01.214 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9293999999999998e-05 2024-03-11T21:10:01.214 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.214 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.214 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:10:01.214 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:55.530085+0000", 2024-03-11T21:10:01.214 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1393999999999997e-05 2024-03-11T21:10:01.215 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:10:01.215 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:10:01.215 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:10:01.215 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.215 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.215 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.15291.0:151225 10.9 10:9bc89164:::benchmark_data_smithi023_124944_object42213:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e348)", 2024-03-11T21:10:01.215 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T21:09:54.911488+0000", 2024-03-11T21:10:01.215 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 6.1841294019999999, 2024-03-11T21:10:01.215 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.61898846799999996, 2024-03-11T21:10:01.215 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:10:01.215 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:10:01.215 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-03-11T21:10:01.215 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.15291", 2024-03-11T21:10:01.215 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.23:0/729656300", 2024-03-11T21:10:01.215 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 151225 2024-03-11T21:10:01.216 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.216 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:10:01.216 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.216 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:10:01.216 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.911488+0000", 2024-03-11T21:10:01.216 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:10:01.216 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.216 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.216 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:10:01.216 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.911488+0000", 2024-03-11T21:10:01.216 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:10:01.216 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.216 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.216 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:10:01.216 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.911489+0000", 2024-03-11T21:10:01.217 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.37e-06 2024-03-11T21:10:01.217 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.217 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.217 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:10:01.217 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.911493+0000", 2024-03-11T21:10:01.217 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9899999999999999e-06 2024-03-11T21:10:01.217 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.217 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.217 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:10:01.217 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.911502+0000", 2024-03-11T21:10:01.217 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.3479999999999993e-06 2024-03-11T21:10:01.217 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.217 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.217 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:10:01.217 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.911509+0000", 2024-03-11T21:10:01.218 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4810000000000003e-06 2024-03-11T21:10:01.218 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.218 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.218 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:10:01.218 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.914625+0000", 2024-03-11T21:10:01.218 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003116204 2024-03-11T21:10:01.218 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.218 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.218 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:10:01.218 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.914694+0000", 2024-03-11T21:10:01.218 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9068999999999999e-05 2024-03-11T21:10:01.218 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.218 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.218 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2,7", 2024-03-11T21:10:01.218 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.914759+0000", 2024-03-11T21:10:01.219 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4880000000000002e-05 2024-03-11T21:10:01.219 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.219 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.219 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:10:01.219 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.954832+0000", 2024-03-11T21:10:01.219 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040073444 2024-03-11T21:10:01.219 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.219 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.219 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-03-11T21:10:01.219 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.954846+0000", 2024-03-11T21:10:01.219 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4107000000000001e-05 2024-03-11T21:10:01.219 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.219 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.219 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:10:01.219 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:55.530200+0000", 2024-03-11T21:10:01.220 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.57535300499999997 2024-03-11T21:10:01.220 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.220 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.220 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:10:01.220 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:55.530375+0000", 2024-03-11T21:10:01.220 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017544300000000001 2024-03-11T21:10:01.220 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.220 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.220 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:10:01.220 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:55.530476+0000", 2024-03-11T21:10:01.220 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000101127 2024-03-11T21:10:01.220 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:10:01.220 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:10:01.220 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:10:01.220 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.220 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.221 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.15291.0:151257 10.6 10:6eba6ab7:::benchmark_data_smithi023_124944_object42245:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e348)", 2024-03-11T21:10:01.221 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T21:09:54.970721+0000", 2024-03-11T21:10:01.221 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 6.124896101, 2024-03-11T21:10:01.221 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.55136036099999997, 2024-03-11T21:10:01.221 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:10:01.221 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:10:01.221 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-03-11T21:10:01.221 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.15291", 2024-03-11T21:10:01.221 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.23:0/729656300", 2024-03-11T21:10:01.221 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 151257 2024-03-11T21:10:01.221 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.221 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:10:01.221 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.221 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:10:01.221 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.970721+0000", 2024-03-11T21:10:01.222 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:10:01.222 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.222 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.222 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:10:01.222 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.970721+0000", 2024-03-11T21:10:01.222 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:10:01.222 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.222 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.222 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:10:01.222 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.970722+0000", 2024-03-11T21:10:01.222 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.7499999999999999e-07 2024-03-11T21:10:01.222 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.222 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.222 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:10:01.222 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.970726+0000", 2024-03-11T21:10:01.223 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9979999999999997e-06 2024-03-11T21:10:01.223 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.223 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.223 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:10:01.223 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.970738+0000", 2024-03-11T21:10:01.223 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2543e-05 2024-03-11T21:10:01.223 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.223 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.223 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:10:01.223 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.970746+0000", 2024-03-11T21:10:01.223 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.498e-06 2024-03-11T21:10:01.223 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.223 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.224 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:10:01.224 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.973032+0000", 2024-03-11T21:10:01.224 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002286742 2024-03-11T21:10:01.224 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.224 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.224 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:10:01.224 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.973140+0000", 2024-03-11T21:10:01.224 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000107544 2024-03-11T21:10:01.224 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.224 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.224 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0,4", 2024-03-11T21:10:01.224 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.973257+0000", 2024-03-11T21:10:01.224 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000117323 2024-03-11T21:10:01.224 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.224 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.224 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-03-11T21:10:01.225 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.976000+0000", 2024-03-11T21:10:01.225 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0027431650000000001 2024-03-11T21:10:01.225 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.225 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.225 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:10:01.225 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.976548+0000", 2024-03-11T21:10:01.225 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00054784199999999999 2024-03-11T21:10:01.225 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.225 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.225 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:10:01.225 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:55.521870+0000", 2024-03-11T21:10:01.225 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.545321532 2024-03-11T21:10:01.225 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.225 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.226 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:10:01.226 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:55.522020+0000", 2024-03-11T21:10:01.226 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015009800000000001 2024-03-11T21:10:01.226 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:10:01.226 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:10:01.226 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:10:01.226 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:55.522081+0000", 2024-03-11T21:10:01.226 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1301000000000001e-05 2024-03-11T21:10:01.226 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:10:01.226 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:10:01.226 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:10:01.226 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:10:01.226 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:10:01.226 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-03-11T21:10:01.227 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:01.252 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:01.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:01.253 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:10:01.354 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:10:01.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:01.374 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:10:01.374 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:10:01.476 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:10:01.488 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:01 smithi023 bash[29146]: cluster 2024-03-11T21:10:00.314130+0000 mgr.y (mgr.24380) 711 : cluster 0 pgmap v804: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 3.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 857 op/s 2024-03-11T21:10:01.489 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:01 smithi023 bash[21372]: cluster 2024-03-11T21:10:00.314130+0000 mgr.y (mgr.24380) 711 : cluster 0 pgmap v804: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 3.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 857 op/s 2024-03-11T21:10:01.504 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:10:01.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:01.507 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:10:01.508 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:01 smithi146 bash[23809]: cluster 2024-03-11T21:10:00.314130+0000 mgr.y (mgr.24380) 711 : cluster 0 pgmap v804: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 3.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 857 op/s 2024-03-11T21:10:01.609 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:10:01.647 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:10:01.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:01.651 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:10:01.752 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:10:01.772 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:10:01.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:01.773 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:10:01.874 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:10:01.901 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:10:01.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:01.903 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:10:02.004 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:02.029 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:02.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:02.031 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:10:02.132 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:10:02.154 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:10:02.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:02.156 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:10:02.256 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:10:02.282 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:10:02.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:02.288 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:10:02.388 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:02.415 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:02.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:02.418 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:10:02.470 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:02 smithi023 bash[21372]: cluster 2024-03-11T21:10:01.217352+0000 osd.1 (osd.1) 211 : cluster 0 2.15 scrub starts 2024-03-11T21:10:02.470 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:02 smithi023 bash[21372]: cluster 2024-03-11T21:10:01.220713+0000 osd.1 (osd.1) 212 : cluster 0 2.15 scrub ok 2024-03-11T21:10:02.470 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:02 smithi023 bash[21372]: cluster 2024-03-11T21:10:01.675262+0000 osd.5 (osd.5) 407 : cluster 0 2.18 scrub starts 2024-03-11T21:10:02.470 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:02 smithi023 bash[21372]: cluster 2024-03-11T21:10:01.677273+0000 osd.5 (osd.5) 408 : cluster 0 2.18 scrub ok 2024-03-11T21:10:02.470 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:02 smithi023 bash[29146]: cluster 2024-03-11T21:10:01.217352+0000 osd.1 (osd.1) 211 : cluster 0 2.15 scrub starts 2024-03-11T21:10:02.471 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:02 smithi023 bash[29146]: cluster 2024-03-11T21:10:01.220713+0000 osd.1 (osd.1) 212 : cluster 0 2.15 scrub ok 2024-03-11T21:10:02.471 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:02 smithi023 bash[29146]: cluster 2024-03-11T21:10:01.675262+0000 osd.5 (osd.5) 407 : cluster 0 2.18 scrub starts 2024-03-11T21:10:02.471 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:02 smithi023 bash[29146]: cluster 2024-03-11T21:10:01.677273+0000 osd.5 (osd.5) 408 : cluster 0 2.18 scrub ok 2024-03-11T21:10:02.508 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:02 smithi146 bash[23809]: cluster 2024-03-11T21:10:01.217352+0000 osd.1 (osd.1) 211 : cluster 0 2.15 scrub starts 2024-03-11T21:10:02.508 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:02 smithi146 bash[23809]: cluster 2024-03-11T21:10:01.220713+0000 osd.1 (osd.1) 212 : cluster 0 2.15 scrub ok 2024-03-11T21:10:02.508 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:02 smithi146 bash[23809]: cluster 2024-03-11T21:10:01.675262+0000 osd.5 (osd.5) 407 : cluster 0 2.18 scrub starts 2024-03-11T21:10:02.508 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:02 smithi146 bash[23809]: cluster 2024-03-11T21:10:01.677273+0000 osd.5 (osd.5) 408 : cluster 0 2.18 scrub ok 2024-03-11T21:10:02.521 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:10:02.558 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:10:02.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:02.560 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:10:02.583 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-11T21:10:02.633 INFO:teuthology.orchestra.run.smithi023.stdout:osd.0: {} 2024-03-11T21:10:02.661 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:02.688 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:02.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:02.690 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:10:02.744 INFO:teuthology.orchestra.run.smithi023.stdout:osd.1: {} 2024-03-11T21:10:02.784 INFO:teuthology.orchestra.run.smithi023.stdout:osd.2: {} 2024-03-11T21:10:02.791 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:02.807 INFO:teuthology.orchestra.run.smithi023.stdout:osd.3: {} 2024-03-11T21:10:02.809 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:02.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:02.815 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:10:02.828 INFO:teuthology.orchestra.run.smithi023.stdout:osd.4: {} 2024-03-11T21:10:02.863 INFO:teuthology.orchestra.run.smithi023.stdout:osd.5: {} 2024-03-11T21:10:02.903 INFO:teuthology.orchestra.run.smithi023.stdout:osd.6: {} 2024-03-11T21:10:02.915 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:10:02.933 INFO:teuthology.orchestra.run.smithi023.stdout:osd.7: {} 2024-03-11T21:10:02.938 INFO:teuthology.orchestra.run.smithi023.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:10:02.938 INFO:teuthology.orchestra.run.smithi023.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:10:02.938 INFO:teuthology.orchestra.run.smithi023.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:10:02.939 INFO:teuthology.orchestra.run.smithi023.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:10:02.939 INFO:teuthology.orchestra.run.smithi023.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:10:02.939 INFO:teuthology.orchestra.run.smithi023.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:10:02.939 INFO:teuthology.orchestra.run.smithi023.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:10:02.939 INFO:teuthology.orchestra.run.smithi023.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:10:02.945 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:10:02.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:02.948 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:10:03.048 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:10:03.073 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:10:03.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:03.076 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:10:03.176 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:03.218 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:03.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:03.222 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:10:03.323 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:10:03.332 INFO:teuthology.orchestra.run.smithi023.stderr:noscrub is set 2024-03-11T21:10:03.360 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:10:03.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:03.363 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:10:03.464 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:10:03.490 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:10:03.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:03.495 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:10:03.508 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:03 smithi146 bash[23809]: cluster 2024-03-11T21:10:01.448110+0000 osd.0 (osd.0) 307 : cluster 0 6.16 scrub starts 2024-03-11T21:10:03.508 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:03 smithi146 bash[23809]: cluster 2024-03-11T21:10:01.497624+0000 osd.0 (osd.0) 308 : cluster 0 6.16 scrub ok 2024-03-11T21:10:03.508 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:03 smithi146 bash[23809]: cluster 2024-03-11T21:10:02.093633+0000 osd.3 (osd.3) 402 : cluster 0 4.11 deep-scrub starts 2024-03-11T21:10:03.508 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:03 smithi146 bash[23809]: cluster 2024-03-11T21:10:02.099645+0000 osd.3 (osd.3) 403 : cluster 0 4.11 deep-scrub ok 2024-03-11T21:10:03.508 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:03 smithi146 bash[23809]: cluster 2024-03-11T21:10:02.239816+0000 osd.1 (osd.1) 213 : cluster 0 3.0 scrub starts 2024-03-11T21:10:03.508 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:03 smithi146 bash[23809]: cluster 2024-03-11T21:10:02.247646+0000 osd.1 (osd.1) 214 : cluster 0 3.0 scrub ok 2024-03-11T21:10:03.508 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:03 smithi146 bash[23809]: cluster 2024-03-11T21:10:02.314830+0000 mgr.y (mgr.24380) 712 : cluster 0 pgmap v805: 148 pgs: 148 active+clean; 3.8 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 818 op/s 2024-03-11T21:10:03.509 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:03 smithi146 bash[23809]: cluster 2024-03-11T21:10:02.551523+0000 osd.7 (osd.7) 449 : cluster 0 6.3 scrub starts 2024-03-11T21:10:03.509 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:03 smithi146 bash[23809]: audit 2024-03-11T21:10:02.618184+0000 mon.b (mon.1) 43 : audit 0 from='client.? 172.21.15.23:0/1119350268' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:10:03.509 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:03 smithi146 bash[23809]: audit 2024-03-11T21:10:02.936272+0000 mon.a (mon.0) 1708 : audit 1 from='client.? 172.21.15.23:0/1418634504' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:10:03.596 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:03.626 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:03.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:03.628 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:10:03.642 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:03 smithi023 bash[21372]: cluster 2024-03-11T21:10:01.448110+0000 osd.0 (osd.0) 307 : cluster 0 6.16 scrub starts 2024-03-11T21:10:03.643 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:03 smithi023 bash[21372]: cluster 2024-03-11T21:10:01.497624+0000 osd.0 (osd.0) 308 : cluster 0 6.16 scrub ok 2024-03-11T21:10:03.643 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:03 smithi023 bash[21372]: cluster 2024-03-11T21:10:02.093633+0000 osd.3 (osd.3) 402 : cluster 0 4.11 deep-scrub starts 2024-03-11T21:10:03.643 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:03 smithi023 bash[21372]: cluster 2024-03-11T21:10:02.099645+0000 osd.3 (osd.3) 403 : cluster 0 4.11 deep-scrub ok 2024-03-11T21:10:03.643 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:03 smithi023 bash[21372]: cluster 2024-03-11T21:10:02.239816+0000 osd.1 (osd.1) 213 : cluster 0 3.0 scrub starts 2024-03-11T21:10:03.643 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:03 smithi023 bash[21372]: cluster 2024-03-11T21:10:02.247646+0000 osd.1 (osd.1) 214 : cluster 0 3.0 scrub ok 2024-03-11T21:10:03.643 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:03 smithi023 bash[21372]: cluster 2024-03-11T21:10:02.314830+0000 mgr.y (mgr.24380) 712 : cluster 0 pgmap v805: 148 pgs: 148 active+clean; 3.8 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 818 op/s 2024-03-11T21:10:03.643 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:03 smithi023 bash[21372]: cluster 2024-03-11T21:10:02.551523+0000 osd.7 (osd.7) 449 : cluster 0 6.3 scrub starts 2024-03-11T21:10:03.643 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:03 smithi023 bash[21372]: audit 2024-03-11T21:10:02.618184+0000 mon.b (mon.1) 43 : audit 0 from='client.? 172.21.15.23:0/1119350268' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:10:03.643 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:03 smithi023 bash[21372]: audit 2024-03-11T21:10:02.936272+0000 mon.a (mon.0) 1708 : audit 1 from='client.? 172.21.15.23:0/1418634504' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:10:03.643 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:03 smithi023 bash[29146]: cluster 2024-03-11T21:10:01.448110+0000 osd.0 (osd.0) 307 : cluster 0 6.16 scrub starts 2024-03-11T21:10:03.644 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:03 smithi023 bash[29146]: cluster 2024-03-11T21:10:01.497624+0000 osd.0 (osd.0) 308 : cluster 0 6.16 scrub ok 2024-03-11T21:10:03.644 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:03 smithi023 bash[29146]: cluster 2024-03-11T21:10:02.093633+0000 osd.3 (osd.3) 402 : cluster 0 4.11 deep-scrub starts 2024-03-11T21:10:03.645 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:03 smithi023 bash[29146]: cluster 2024-03-11T21:10:02.099645+0000 osd.3 (osd.3) 403 : cluster 0 4.11 deep-scrub ok 2024-03-11T21:10:03.645 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:03 smithi023 bash[29146]: cluster 2024-03-11T21:10:02.239816+0000 osd.1 (osd.1) 213 : cluster 0 3.0 scrub starts 2024-03-11T21:10:03.645 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:03 smithi023 bash[29146]: cluster 2024-03-11T21:10:02.247646+0000 osd.1 (osd.1) 214 : cluster 0 3.0 scrub ok 2024-03-11T21:10:03.645 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:03 smithi023 bash[29146]: cluster 2024-03-11T21:10:02.314830+0000 mgr.y (mgr.24380) 712 : cluster 0 pgmap v805: 148 pgs: 148 active+clean; 3.8 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 818 op/s 2024-03-11T21:10:03.645 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:03 smithi023 bash[29146]: cluster 2024-03-11T21:10:02.551523+0000 osd.7 (osd.7) 449 : cluster 0 6.3 scrub starts 2024-03-11T21:10:03.645 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:03 smithi023 bash[29146]: audit 2024-03-11T21:10:02.618184+0000 mon.b (mon.1) 43 : audit 0 from='client.? 172.21.15.23:0/1119350268' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:10:03.645 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:03 smithi023 bash[29146]: audit 2024-03-11T21:10:02.936272+0000 mon.a (mon.0) 1708 : audit 1 from='client.? 172.21.15.23:0/1418634504' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:10:03.729 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:10:03.756 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:10:03.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:03.759 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:10:03.860 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:10:03.895 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:10:03.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:03.900 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:10:04.001 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:10:04.034 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:10:04.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:04.037 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:10:04.138 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:04.190 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:04.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:04.195 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:10:04.300 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:10:04.339 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:10:04.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:04.346 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:10:04.447 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:10:04.473 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:10:04.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:04.476 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:10:04.577 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:10:04.590 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:04 smithi146 bash[23809]: cluster 2024-03-11T21:10:02.424562+0000 osd.0 (osd.0) 309 : cluster 0 6.10 scrub starts 2024-03-11T21:10:04.590 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:04 smithi146 bash[23809]: cluster 2024-03-11T21:10:02.496957+0000 osd.0 (osd.0) 310 : cluster 0 6.10 scrub ok 2024-03-11T21:10:04.590 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:04 smithi146 bash[23809]: cluster 2024-03-11T21:10:02.768090+0000 osd.7 (osd.7) 450 : cluster 0 6.3 scrub ok 2024-03-11T21:10:04.591 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:04 smithi146 bash[23809]: cluster 2024-03-11T21:10:02.908432+0000 osd.3 (osd.3) 404 : cluster 0 6.f scrub starts 2024-03-11T21:10:04.591 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:04 smithi146 bash[23809]: cluster 2024-03-11T21:10:03.144870+0000 osd.3 (osd.3) 405 : cluster 0 6.f scrub ok 2024-03-11T21:10:04.591 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:04 smithi146 bash[23809]: cluster 2024-03-11T21:10:03.270464+0000 osd.1 (osd.1) 215 : cluster 0 4.12 scrub starts 2024-03-11T21:10:04.591 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:04 smithi146 bash[23809]: cluster 2024-03-11T21:10:03.274259+0000 osd.1 (osd.1) 216 : cluster 0 4.12 scrub ok 2024-03-11T21:10:04.591 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:04 smithi146 bash[23809]: cluster 2024-03-11T21:10:03.331016+0000 mon.a (mon.0) 1709 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:10:04.591 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:04 smithi146 bash[23809]: audit 2024-03-11T21:10:03.332743+0000 mon.a (mon.0) 1710 : audit 1 from='client.? 172.21.15.23:0/1418634504' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:10:04.592 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:04 smithi146 bash[23809]: cluster 2024-03-11T21:10:03.336751+0000 mon.a (mon.0) 1711 : cluster 0 osdmap e350: 8 total, 8 up, 8 in 2024-03-11T21:10:04.600 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:10:04.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:04.602 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:10:04.703 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:10:04.730 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:10:04.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:04.735 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:10:04.835 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:04.884 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:04.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:04.887 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:10:04.935 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:04 smithi023 bash[21372]: cluster 2024-03-11T21:10:02.424562+0000 osd.0 (osd.0) 309 : cluster 0 6.10 scrub starts 2024-03-11T21:10:04.935 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:04 smithi023 bash[21372]: cluster 2024-03-11T21:10:02.496957+0000 osd.0 (osd.0) 310 : cluster 0 6.10 scrub ok 2024-03-11T21:10:04.935 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:04 smithi023 bash[21372]: cluster 2024-03-11T21:10:02.768090+0000 osd.7 (osd.7) 450 : cluster 0 6.3 scrub ok 2024-03-11T21:10:04.935 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:04 smithi023 bash[21372]: cluster 2024-03-11T21:10:02.908432+0000 osd.3 (osd.3) 404 : cluster 0 6.f scrub starts 2024-03-11T21:10:04.936 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:04 smithi023 bash[21372]: cluster 2024-03-11T21:10:03.144870+0000 osd.3 (osd.3) 405 : cluster 0 6.f scrub ok 2024-03-11T21:10:04.936 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:04 smithi023 bash[21372]: cluster 2024-03-11T21:10:03.270464+0000 osd.1 (osd.1) 215 : cluster 0 4.12 scrub starts 2024-03-11T21:10:04.936 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:04 smithi023 bash[21372]: cluster 2024-03-11T21:10:03.274259+0000 osd.1 (osd.1) 216 : cluster 0 4.12 scrub ok 2024-03-11T21:10:04.936 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:04 smithi023 bash[21372]: cluster 2024-03-11T21:10:03.331016+0000 mon.a (mon.0) 1709 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:10:04.936 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:04 smithi023 bash[21372]: audit 2024-03-11T21:10:03.332743+0000 mon.a (mon.0) 1710 : audit 1 from='client.? 172.21.15.23:0/1418634504' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:10:04.936 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:04 smithi023 bash[21372]: cluster 2024-03-11T21:10:03.336751+0000 mon.a (mon.0) 1711 : cluster 0 osdmap e350: 8 total, 8 up, 8 in 2024-03-11T21:10:04.936 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:04 smithi023 bash[29146]: cluster 2024-03-11T21:10:02.424562+0000 osd.0 (osd.0) 309 : cluster 0 6.10 scrub starts 2024-03-11T21:10:04.936 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:04 smithi023 bash[29146]: cluster 2024-03-11T21:10:02.496957+0000 osd.0 (osd.0) 310 : cluster 0 6.10 scrub ok 2024-03-11T21:10:04.936 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:04 smithi023 bash[29146]: cluster 2024-03-11T21:10:02.768090+0000 osd.7 (osd.7) 450 : cluster 0 6.3 scrub ok 2024-03-11T21:10:04.936 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:04 smithi023 bash[29146]: cluster 2024-03-11T21:10:02.908432+0000 osd.3 (osd.3) 404 : cluster 0 6.f scrub starts 2024-03-11T21:10:04.936 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:04 smithi023 bash[29146]: cluster 2024-03-11T21:10:03.144870+0000 osd.3 (osd.3) 405 : cluster 0 6.f scrub ok 2024-03-11T21:10:04.937 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:04 smithi023 bash[29146]: cluster 2024-03-11T21:10:03.270464+0000 osd.1 (osd.1) 215 : cluster 0 4.12 scrub starts 2024-03-11T21:10:04.937 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:04 smithi023 bash[29146]: cluster 2024-03-11T21:10:03.274259+0000 osd.1 (osd.1) 216 : cluster 0 4.12 scrub ok 2024-03-11T21:10:04.937 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:04 smithi023 bash[29146]: cluster 2024-03-11T21:10:03.331016+0000 mon.a (mon.0) 1709 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:10:04.937 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:04 smithi023 bash[29146]: audit 2024-03-11T21:10:03.332743+0000 mon.a (mon.0) 1710 : audit 1 from='client.? 172.21.15.23:0/1418634504' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:10:04.937 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:04 smithi023 bash[29146]: cluster 2024-03-11T21:10:03.336751+0000 mon.a (mon.0) 1711 : cluster 0 osdmap e350: 8 total, 8 up, 8 in 2024-03-11T21:10:04.988 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:05.020 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:05.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:05.022 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:10:05.060 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-11T21:10:05.060 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-11T21:10:05.060 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 350, 2024-03-11T21:10:05.061 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "67ba00f8-dfe7-11ee-95c8-87774f69a715", 2024-03-11T21:10:05.061 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2024-03-11T20:40:13.664802+0000", 2024-03-11T21:10:05.061 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2024-03-11T21:10:03.330707+0000", 2024-03-11T21:10:05.061 INFO:teuthology.orchestra.run.smithi023.stdout: "last_up_change": "2024-03-11T21:00:47.324817+0000", 2024-03-11T21:10:05.061 INFO:teuthology.orchestra.run.smithi023.stdout: "last_in_change": "2024-03-11T20:57:23.560527+0000", 2024-03-11T21:10:05.061 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-03-11T21:10:05.061 INFO:teuthology.orchestra.run.smithi023.stdout: "flags_num": 5801984, 2024-03-11T21:10:05.061 INFO:teuthology.orchestra.run.smithi023.stdout: "flags_set": [ 2024-03-11T21:10:05.061 INFO:teuthology.orchestra.run.smithi023.stdout: "noscrub", 2024-03-11T21:10:05.061 INFO:teuthology.orchestra.run.smithi023.stdout: "pglog_hardlimit", 2024-03-11T21:10:05.061 INFO:teuthology.orchestra.run.smithi023.stdout: "purged_snapdirs", 2024-03-11T21:10:05.061 INFO:teuthology.orchestra.run.smithi023.stdout: "recovery_deletes", 2024-03-11T21:10:05.062 INFO:teuthology.orchestra.run.smithi023.stdout: "sortbitwise" 2024-03-11T21:10:05.062 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-11T21:10:05.062 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_version": 20, 2024-03-11T21:10:05.062 INFO:teuthology.orchestra.run.smithi023.stdout: "full_ratio": 0.94999998807907104, 2024-03-11T21:10:05.062 INFO:teuthology.orchestra.run.smithi023.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-03-11T21:10:05.062 INFO:teuthology.orchestra.run.smithi023.stdout: "nearfull_ratio": 0.85000002384185791, 2024-03-11T21:10:05.062 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_snapshot": "", 2024-03-11T21:10:05.062 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_max": 10, 2024-03-11T21:10:05.062 INFO:teuthology.orchestra.run.smithi023.stdout: "max_osd": 8, 2024-03-11T21:10:05.062 INFO:teuthology.orchestra.run.smithi023.stdout: "require_min_compat_client": "luminous", 2024-03-11T21:10:05.062 INFO:teuthology.orchestra.run.smithi023.stdout: "min_compat_client": "luminous", 2024-03-11T21:10:05.062 INFO:teuthology.orchestra.run.smithi023.stdout: "require_osd_release": "squid", 2024-03-11T21:10:05.063 INFO:teuthology.orchestra.run.smithi023.stdout: "allow_crimson": false, 2024-03-11T21:10:05.063 INFO:teuthology.orchestra.run.smithi023.stdout: "pools": [ 2024-03-11T21:10:05.063 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:10:05.063 INFO:teuthology.orchestra.run.smithi023.stdout: "pool": 1, 2024-03-11T21:10:05.063 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_name": ".mgr", 2024-03-11T21:10:05.063 INFO:teuthology.orchestra.run.smithi023.stdout: "create_time": "2024-03-11T20:45:31.964570+0000", 2024-03-11T21:10:05.063 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-11T21:10:05.063 INFO:teuthology.orchestra.run.smithi023.stdout: "flags_names": "hashpspool", 2024-03-11T21:10:05.063 INFO:teuthology.orchestra.run.smithi023.stdout: "type": 1, 2024-03-11T21:10:05.063 INFO:teuthology.orchestra.run.smithi023.stdout: "size": 3, 2024-03-11T21:10:05.063 INFO:teuthology.orchestra.run.smithi023.stdout: "min_size": 2, 2024-03-11T21:10:05.063 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_rule": 0, 2024-03-11T21:10:05.063 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_count": 0, 2024-03-11T21:10:05.063 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_target": 0, 2024-03-11T21:10:05.064 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T21:10:05.064 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T21:10:05.064 INFO:teuthology.orchestra.run.smithi023.stdout: "object_hash": 2, 2024-03-11T21:10:05.064 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_autoscale_mode": "off", 2024-03-11T21:10:05.064 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num": 1, 2024-03-11T21:10:05.064 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_placement_num": 1, 2024-03-11T21:10:05.064 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_placement_num_target": 1, 2024-03-11T21:10:05.064 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num_target": 1, 2024-03-11T21:10:05.064 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num_pending": 1, 2024-03-11T21:10:05.064 INFO:teuthology.orchestra.run.smithi023.stdout: "last_pg_merge_meta": { 2024-03-11T21:10:05.064 INFO:teuthology.orchestra.run.smithi023.stdout: "source_pgid": "0.0", 2024-03-11T21:10:05.064 INFO:teuthology.orchestra.run.smithi023.stdout: "ready_epoch": 0, 2024-03-11T21:10:05.064 INFO:teuthology.orchestra.run.smithi023.stdout: "last_epoch_started": 0, 2024-03-11T21:10:05.065 INFO:teuthology.orchestra.run.smithi023.stdout: "last_epoch_clean": 0, 2024-03-11T21:10:05.065 INFO:teuthology.orchestra.run.smithi023.stdout: "source_version": "0'0", 2024-03-11T21:10:05.065 INFO:teuthology.orchestra.run.smithi023.stdout: "target_version": "0'0" 2024-03-11T21:10:05.065 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:10:05.065 INFO:teuthology.orchestra.run.smithi023.stdout: "last_change": "142", 2024-03-11T21:10:05.065 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend": "0", 2024-03-11T21:10:05.065 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T21:10:05.065 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T21:10:05.065 INFO:teuthology.orchestra.run.smithi023.stdout: "auid": 0, 2024-03-11T21:10:05.065 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_mode": "selfmanaged", 2024-03-11T21:10:05.065 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_seq": 0, 2024-03-11T21:10:05.065 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_epoch": 0, 2024-03-11T21:10:05.065 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_snaps": [], 2024-03-11T21:10:05.066 INFO:teuthology.orchestra.run.smithi023.stdout: "removed_snaps": "[]", 2024-03-11T21:10:05.066 INFO:teuthology.orchestra.run.smithi023.stdout: "quota_max_bytes": 0, 2024-03-11T21:10:05.066 INFO:teuthology.orchestra.run.smithi023.stdout: "quota_max_objects": 0, 2024-03-11T21:10:05.066 INFO:teuthology.orchestra.run.smithi023.stdout: "tiers": [], 2024-03-11T21:10:05.066 INFO:teuthology.orchestra.run.smithi023.stdout: "tier_of": -1, 2024-03-11T21:10:05.066 INFO:teuthology.orchestra.run.smithi023.stdout: "read_tier": -1, 2024-03-11T21:10:05.066 INFO:teuthology.orchestra.run.smithi023.stdout: "write_tier": -1, 2024-03-11T21:10:05.066 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_mode": "none", 2024-03-11T21:10:05.066 INFO:teuthology.orchestra.run.smithi023.stdout: "target_max_bytes": 0, 2024-03-11T21:10:05.066 INFO:teuthology.orchestra.run.smithi023.stdout: "target_max_objects": 0, 2024-03-11T21:10:05.066 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T21:10:05.066 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T21:10:05.066 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T21:10:05.067 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_min_flush_age": 0, 2024-03-11T21:10:05.067 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_min_evict_age": 0, 2024-03-11T21:10:05.067 INFO:teuthology.orchestra.run.smithi023.stdout: "erasure_code_profile": "", 2024-03-11T21:10:05.067 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_params": { 2024-03-11T21:10:05.067 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "none" 2024-03-11T21:10:05.067 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:10:05.067 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_period": 0, 2024-03-11T21:10:05.067 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_count": 0, 2024-03-11T21:10:05.067 INFO:teuthology.orchestra.run.smithi023.stdout: "use_gmt_hitset": true, 2024-03-11T21:10:05.067 INFO:teuthology.orchestra.run.smithi023.stdout: "min_read_recency_for_promote": 0, 2024-03-11T21:10:05.067 INFO:teuthology.orchestra.run.smithi023.stdout: "min_write_recency_for_promote": 0, 2024-03-11T21:10:05.067 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T21:10:05.067 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_search_last_n": 0, 2024-03-11T21:10:05.068 INFO:teuthology.orchestra.run.smithi023.stdout: "grade_table": [], 2024-03-11T21:10:05.068 INFO:teuthology.orchestra.run.smithi023.stdout: "stripe_width": 0, 2024-03-11T21:10:05.068 INFO:teuthology.orchestra.run.smithi023.stdout: "expected_num_objects": 0, 2024-03-11T21:10:05.068 INFO:teuthology.orchestra.run.smithi023.stdout: "fast_read": false, 2024-03-11T21:10:05.068 INFO:teuthology.orchestra.run.smithi023.stdout: "options": { 2024-03-11T21:10:05.068 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num_max": 32, 2024-03-11T21:10:05.068 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num_min": 1 2024-03-11T21:10:05.068 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:10:05.068 INFO:teuthology.orchestra.run.smithi023.stdout: "application_metadata": { 2024-03-11T21:10:05.068 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr": {} 2024-03-11T21:10:05.068 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:10:05.068 INFO:teuthology.orchestra.run.smithi023.stdout: "read_balance": { 2024-03-11T21:10:05.068 INFO:teuthology.orchestra.run.smithi023.stdout: "score_type": "Fair distribution", 2024-03-11T21:10:05.069 INFO:teuthology.orchestra.run.smithi023.stdout: "score_acting": 6, 2024-03-11T21:10:05.069 INFO:teuthology.orchestra.run.smithi023.stdout: "score_stable": 6, 2024-03-11T21:10:05.069 INFO:teuthology.orchestra.run.smithi023.stdout: "optimal_score": 0.5, 2024-03-11T21:10:05.069 INFO:teuthology.orchestra.run.smithi023.stdout: "raw_score_acting": 3, 2024-03-11T21:10:05.069 INFO:teuthology.orchestra.run.smithi023.stdout: "raw_score_stable": 3, 2024-03-11T21:10:05.069 INFO:teuthology.orchestra.run.smithi023.stdout: "primary_affinity_weighted": 0.75, 2024-03-11T21:10:05.069 INFO:teuthology.orchestra.run.smithi023.stdout: "average_primary_affinity": 1, 2024-03-11T21:10:05.069 INFO:teuthology.orchestra.run.smithi023.stdout: "average_primary_affinity_weighted": 0.75 2024-03-11T21:10:05.069 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:10:05.069 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:10:05.069 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:10:05.069 INFO:teuthology.orchestra.run.smithi023.stdout: "pool": 2, 2024-03-11T21:10:05.069 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_name": ".rgw.root", 2024-03-11T21:10:05.070 INFO:teuthology.orchestra.run.smithi023.stdout: "create_time": "2024-03-11T20:50:26.421486+0000", 2024-03-11T21:10:05.070 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-11T21:10:05.070 INFO:teuthology.orchestra.run.smithi023.stdout: "flags_names": "hashpspool", 2024-03-11T21:10:05.070 INFO:teuthology.orchestra.run.smithi023.stdout: "type": 1, 2024-03-11T21:10:05.070 INFO:teuthology.orchestra.run.smithi023.stdout: "size": 3, 2024-03-11T21:10:05.070 INFO:teuthology.orchestra.run.smithi023.stdout: "min_size": 2, 2024-03-11T21:10:05.070 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_rule": 0, 2024-03-11T21:10:05.070 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_count": 0, 2024-03-11T21:10:05.070 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_target": 0, 2024-03-11T21:10:05.070 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T21:10:05.070 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T21:10:05.070 INFO:teuthology.orchestra.run.smithi023.stdout: "object_hash": 2, 2024-03-11T21:10:05.070 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_autoscale_mode": "off", 2024-03-11T21:10:05.071 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num": 32, 2024-03-11T21:10:05.071 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_placement_num": 32, 2024-03-11T21:10:05.071 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_placement_num_target": 32, 2024-03-11T21:10:05.071 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num_target": 32, 2024-03-11T21:10:05.071 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num_pending": 32, 2024-03-11T21:10:05.071 INFO:teuthology.orchestra.run.smithi023.stdout: "last_pg_merge_meta": { 2024-03-11T21:10:05.071 INFO:teuthology.orchestra.run.smithi023.stdout: "source_pgid": "0.0", 2024-03-11T21:10:05.071 INFO:teuthology.orchestra.run.smithi023.stdout: "ready_epoch": 0, 2024-03-11T21:10:05.071 INFO:teuthology.orchestra.run.smithi023.stdout: "last_epoch_started": 0, 2024-03-11T21:10:05.071 INFO:teuthology.orchestra.run.smithi023.stdout: "last_epoch_clean": 0, 2024-03-11T21:10:05.071 INFO:teuthology.orchestra.run.smithi023.stdout: "source_version": "0'0", 2024-03-11T21:10:05.071 INFO:teuthology.orchestra.run.smithi023.stdout: "target_version": "0'0" 2024-03-11T21:10:05.071 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:10:05.072 INFO:teuthology.orchestra.run.smithi023.stdout: "last_change": "78", 2024-03-11T21:10:05.072 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend": "0", 2024-03-11T21:10:05.072 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T21:10:05.072 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T21:10:05.072 INFO:teuthology.orchestra.run.smithi023.stdout: "auid": 0, 2024-03-11T21:10:05.072 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_mode": "selfmanaged", 2024-03-11T21:10:05.072 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_seq": 0, 2024-03-11T21:10:05.072 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_epoch": 0, 2024-03-11T21:10:05.072 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_snaps": [], 2024-03-11T21:10:05.072 INFO:teuthology.orchestra.run.smithi023.stdout: "removed_snaps": "[]", 2024-03-11T21:10:05.072 INFO:teuthology.orchestra.run.smithi023.stdout: "quota_max_bytes": 0, 2024-03-11T21:10:05.072 INFO:teuthology.orchestra.run.smithi023.stdout: "quota_max_objects": 0, 2024-03-11T21:10:05.072 INFO:teuthology.orchestra.run.smithi023.stdout: "tiers": [], 2024-03-11T21:10:05.072 INFO:teuthology.orchestra.run.smithi023.stdout: "tier_of": -1, 2024-03-11T21:10:05.073 INFO:teuthology.orchestra.run.smithi023.stdout: "read_tier": -1, 2024-03-11T21:10:05.073 INFO:teuthology.orchestra.run.smithi023.stdout: "write_tier": -1, 2024-03-11T21:10:05.073 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_mode": "none", 2024-03-11T21:10:05.073 INFO:teuthology.orchestra.run.smithi023.stdout: "target_max_bytes": 0, 2024-03-11T21:10:05.073 INFO:teuthology.orchestra.run.smithi023.stdout: "target_max_objects": 0, 2024-03-11T21:10:05.073 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T21:10:05.073 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T21:10:05.073 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T21:10:05.073 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_min_flush_age": 0, 2024-03-11T21:10:05.073 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_min_evict_age": 0, 2024-03-11T21:10:05.073 INFO:teuthology.orchestra.run.smithi023.stdout: "erasure_code_profile": "", 2024-03-11T21:10:05.073 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_params": { 2024-03-11T21:10:05.073 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "none" 2024-03-11T21:10:05.073 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:10:05.074 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_period": 0, 2024-03-11T21:10:05.074 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_count": 0, 2024-03-11T21:10:05.074 INFO:teuthology.orchestra.run.smithi023.stdout: "use_gmt_hitset": true, 2024-03-11T21:10:05.074 INFO:teuthology.orchestra.run.smithi023.stdout: "min_read_recency_for_promote": 0, 2024-03-11T21:10:05.074 INFO:teuthology.orchestra.run.smithi023.stdout: "min_write_recency_for_promote": 0, 2024-03-11T21:10:05.074 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T21:10:05.074 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_search_last_n": 0, 2024-03-11T21:10:05.074 INFO:teuthology.orchestra.run.smithi023.stdout: "grade_table": [], 2024-03-11T21:10:05.074 INFO:teuthology.orchestra.run.smithi023.stdout: "stripe_width": 0, 2024-03-11T21:10:05.074 INFO:teuthology.orchestra.run.smithi023.stdout: "expected_num_objects": 0, 2024-03-11T21:10:05.075 INFO:teuthology.orchestra.run.smithi023.stdout: "fast_read": false, 2024-03-11T21:10:05.075 INFO:teuthology.orchestra.run.smithi023.stdout: "options": {}, 2024-03-11T21:10:05.075 INFO:teuthology.orchestra.run.smithi023.stdout: "application_metadata": { 2024-03-11T21:10:05.075 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw": {} 2024-03-11T21:10:05.075 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:10:05.075 INFO:teuthology.orchestra.run.smithi023.stdout: "read_balance": { 2024-03-11T21:10:05.075 INFO:teuthology.orchestra.run.smithi023.stdout: "score_type": "Fair distribution", 2024-03-11T21:10:05.075 INFO:teuthology.orchestra.run.smithi023.stdout: "score_acting": 1.5, 2024-03-11T21:10:05.076 INFO:teuthology.orchestra.run.smithi023.stdout: "score_stable": 1.5, 2024-03-11T21:10:05.076 INFO:teuthology.orchestra.run.smithi023.stdout: "optimal_score": 1.3300000429153442, 2024-03-11T21:10:05.076 INFO:teuthology.orchestra.run.smithi023.stdout: "raw_score_acting": 2, 2024-03-11T21:10:05.076 INFO:teuthology.orchestra.run.smithi023.stdout: "raw_score_stable": 2, 2024-03-11T21:10:05.076 INFO:teuthology.orchestra.run.smithi023.stdout: "primary_affinity_weighted": 0.75, 2024-03-11T21:10:05.076 INFO:teuthology.orchestra.run.smithi023.stdout: "average_primary_affinity": 1, 2024-03-11T21:10:05.076 INFO:teuthology.orchestra.run.smithi023.stdout: "average_primary_affinity_weighted": 0.75 2024-03-11T21:10:05.076 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:10:05.076 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:10:05.076 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:10:05.076 INFO:teuthology.orchestra.run.smithi023.stdout: "pool": 3, 2024-03-11T21:10:05.076 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_name": "default.rgw.log", 2024-03-11T21:10:05.077 INFO:teuthology.orchestra.run.smithi023.stdout: "create_time": "2024-03-11T20:50:28.592615+0000", 2024-03-11T21:10:05.077 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-11T21:10:05.077 INFO:teuthology.orchestra.run.smithi023.stdout: "flags_names": "hashpspool", 2024-03-11T21:10:05.077 INFO:teuthology.orchestra.run.smithi023.stdout: "type": 1, 2024-03-11T21:10:05.077 INFO:teuthology.orchestra.run.smithi023.stdout: "size": 3, 2024-03-11T21:10:05.077 INFO:teuthology.orchestra.run.smithi023.stdout: "min_size": 2, 2024-03-11T21:10:05.077 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_rule": 0, 2024-03-11T21:10:05.077 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_count": 0, 2024-03-11T21:10:05.077 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_target": 0, 2024-03-11T21:10:05.077 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T21:10:05.077 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T21:10:05.077 INFO:teuthology.orchestra.run.smithi023.stdout: "object_hash": 2, 2024-03-11T21:10:05.078 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_autoscale_mode": "off", 2024-03-11T21:10:05.078 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num": 32, 2024-03-11T21:10:05.078 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_placement_num": 32, 2024-03-11T21:10:05.078 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_placement_num_target": 32, 2024-03-11T21:10:05.078 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num_target": 32, 2024-03-11T21:10:05.078 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num_pending": 32, 2024-03-11T21:10:05.078 INFO:teuthology.orchestra.run.smithi023.stdout: "last_pg_merge_meta": { 2024-03-11T21:10:05.078 INFO:teuthology.orchestra.run.smithi023.stdout: "source_pgid": "0.0", 2024-03-11T21:10:05.078 INFO:teuthology.orchestra.run.smithi023.stdout: "ready_epoch": 0, 2024-03-11T21:10:05.078 INFO:teuthology.orchestra.run.smithi023.stdout: "last_epoch_started": 0, 2024-03-11T21:10:05.078 INFO:teuthology.orchestra.run.smithi023.stdout: "last_epoch_clean": 0, 2024-03-11T21:10:05.078 INFO:teuthology.orchestra.run.smithi023.stdout: "source_version": "0'0", 2024-03-11T21:10:05.078 INFO:teuthology.orchestra.run.smithi023.stdout: "target_version": "0'0" 2024-03-11T21:10:05.079 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:10:05.079 INFO:teuthology.orchestra.run.smithi023.stdout: "last_change": "80", 2024-03-11T21:10:05.079 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend": "0", 2024-03-11T21:10:05.079 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T21:10:05.079 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T21:10:05.079 INFO:teuthology.orchestra.run.smithi023.stdout: "auid": 0, 2024-03-11T21:10:05.079 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_mode": "selfmanaged", 2024-03-11T21:10:05.079 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_seq": 0, 2024-03-11T21:10:05.079 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_epoch": 0, 2024-03-11T21:10:05.079 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_snaps": [], 2024-03-11T21:10:05.079 INFO:teuthology.orchestra.run.smithi023.stdout: "removed_snaps": "[]", 2024-03-11T21:10:05.079 INFO:teuthology.orchestra.run.smithi023.stdout: "quota_max_bytes": 0, 2024-03-11T21:10:05.080 INFO:teuthology.orchestra.run.smithi023.stdout: "quota_max_objects": 0, 2024-03-11T21:10:05.080 INFO:teuthology.orchestra.run.smithi023.stdout: "tiers": [], 2024-03-11T21:10:05.080 INFO:teuthology.orchestra.run.smithi023.stdout: "tier_of": -1, 2024-03-11T21:10:05.080 INFO:teuthology.orchestra.run.smithi023.stdout: "read_tier": -1, 2024-03-11T21:10:05.080 INFO:teuthology.orchestra.run.smithi023.stdout: "write_tier": -1, 2024-03-11T21:10:05.080 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_mode": "none", 2024-03-11T21:10:05.080 INFO:teuthology.orchestra.run.smithi023.stdout: "target_max_bytes": 0, 2024-03-11T21:10:05.080 INFO:teuthology.orchestra.run.smithi023.stdout: "target_max_objects": 0, 2024-03-11T21:10:05.080 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T21:10:05.080 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T21:10:05.080 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T21:10:05.080 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_min_flush_age": 0, 2024-03-11T21:10:05.081 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_min_evict_age": 0, 2024-03-11T21:10:05.081 INFO:teuthology.orchestra.run.smithi023.stdout: "erasure_code_profile": "", 2024-03-11T21:10:05.081 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_params": { 2024-03-11T21:10:05.081 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "none" 2024-03-11T21:10:05.081 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:10:05.081 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_period": 0, 2024-03-11T21:10:05.081 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_count": 0, 2024-03-11T21:10:05.081 INFO:teuthology.orchestra.run.smithi023.stdout: "use_gmt_hitset": true, 2024-03-11T21:10:05.081 INFO:teuthology.orchestra.run.smithi023.stdout: "min_read_recency_for_promote": 0, 2024-03-11T21:10:05.081 INFO:teuthology.orchestra.run.smithi023.stdout: "min_write_recency_for_promote": 0, 2024-03-11T21:10:05.081 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T21:10:05.081 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_search_last_n": 0, 2024-03-11T21:10:05.082 INFO:teuthology.orchestra.run.smithi023.stdout: "grade_table": [], 2024-03-11T21:10:05.082 INFO:teuthology.orchestra.run.smithi023.stdout: "stripe_width": 0, 2024-03-11T21:10:05.082 INFO:teuthology.orchestra.run.smithi023.stdout: "expected_num_objects": 0, 2024-03-11T21:10:05.082 INFO:teuthology.orchestra.run.smithi023.stdout: "fast_read": false, 2024-03-11T21:10:05.082 INFO:teuthology.orchestra.run.smithi023.stdout: "options": {}, 2024-03-11T21:10:05.082 INFO:teuthology.orchestra.run.smithi023.stdout: "application_metadata": { 2024-03-11T21:10:05.082 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw": {} 2024-03-11T21:10:05.082 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:10:05.082 INFO:teuthology.orchestra.run.smithi023.stdout: "read_balance": { 2024-03-11T21:10:05.082 INFO:teuthology.orchestra.run.smithi023.stdout: "score_type": "Fair distribution", 2024-03-11T21:10:05.082 INFO:teuthology.orchestra.run.smithi023.stdout: "score_acting": 1.5, 2024-03-11T21:10:05.083 INFO:teuthology.orchestra.run.smithi023.stdout: "score_stable": 1.5, 2024-03-11T21:10:05.083 INFO:teuthology.orchestra.run.smithi023.stdout: "optimal_score": 1.3300000429153442, 2024-03-11T21:10:05.083 INFO:teuthology.orchestra.run.smithi023.stdout: "raw_score_acting": 2, 2024-03-11T21:10:05.083 INFO:teuthology.orchestra.run.smithi023.stdout: "raw_score_stable": 2, 2024-03-11T21:10:05.083 INFO:teuthology.orchestra.run.smithi023.stdout: "primary_affinity_weighted": 0.75, 2024-03-11T21:10:05.083 INFO:teuthology.orchestra.run.smithi023.stdout: "average_primary_affinity": 1, 2024-03-11T21:10:05.083 INFO:teuthology.orchestra.run.smithi023.stdout: "average_primary_affinity_weighted": 0.75 2024-03-11T21:10:05.083 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:10:05.083 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:10:05.083 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:10:05.084 INFO:teuthology.orchestra.run.smithi023.stdout: "pool": 4, 2024-03-11T21:10:05.084 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_name": "default.rgw.control", 2024-03-11T21:10:05.084 INFO:teuthology.orchestra.run.smithi023.stdout: "create_time": "2024-03-11T20:50:30.536371+0000", 2024-03-11T21:10:05.084 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-11T21:10:05.084 INFO:teuthology.orchestra.run.smithi023.stdout: "flags_names": "hashpspool", 2024-03-11T21:10:05.084 INFO:teuthology.orchestra.run.smithi023.stdout: "type": 1, 2024-03-11T21:10:05.084 INFO:teuthology.orchestra.run.smithi023.stdout: "size": 3, 2024-03-11T21:10:05.084 INFO:teuthology.orchestra.run.smithi023.stdout: "min_size": 2, 2024-03-11T21:10:05.084 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_rule": 0, 2024-03-11T21:10:05.084 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_count": 0, 2024-03-11T21:10:05.084 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_target": 0, 2024-03-11T21:10:05.084 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T21:10:05.085 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T21:10:05.085 INFO:teuthology.orchestra.run.smithi023.stdout: "object_hash": 2, 2024-03-11T21:10:05.085 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_autoscale_mode": "off", 2024-03-11T21:10:05.085 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num": 32, 2024-03-11T21:10:05.085 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_placement_num": 32, 2024-03-11T21:10:05.085 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_placement_num_target": 32, 2024-03-11T21:10:05.085 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num_target": 32, 2024-03-11T21:10:05.085 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num_pending": 32, 2024-03-11T21:10:05.085 INFO:teuthology.orchestra.run.smithi023.stdout: "last_pg_merge_meta": { 2024-03-11T21:10:05.085 INFO:teuthology.orchestra.run.smithi023.stdout: "source_pgid": "0.0", 2024-03-11T21:10:05.085 INFO:teuthology.orchestra.run.smithi023.stdout: "ready_epoch": 0, 2024-03-11T21:10:05.085 INFO:teuthology.orchestra.run.smithi023.stdout: "last_epoch_started": 0, 2024-03-11T21:10:05.085 INFO:teuthology.orchestra.run.smithi023.stdout: "last_epoch_clean": 0, 2024-03-11T21:10:05.086 INFO:teuthology.orchestra.run.smithi023.stdout: "source_version": "0'0", 2024-03-11T21:10:05.086 INFO:teuthology.orchestra.run.smithi023.stdout: "target_version": "0'0" 2024-03-11T21:10:05.086 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:10:05.086 INFO:teuthology.orchestra.run.smithi023.stdout: "last_change": "82", 2024-03-11T21:10:05.086 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend": "0", 2024-03-11T21:10:05.086 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T21:10:05.086 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T21:10:05.086 INFO:teuthology.orchestra.run.smithi023.stdout: "auid": 0, 2024-03-11T21:10:05.087 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_mode": "selfmanaged", 2024-03-11T21:10:05.087 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_seq": 0, 2024-03-11T21:10:05.087 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_epoch": 0, 2024-03-11T21:10:05.087 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_snaps": [], 2024-03-11T21:10:05.087 INFO:teuthology.orchestra.run.smithi023.stdout: "removed_snaps": "[]", 2024-03-11T21:10:05.087 INFO:teuthology.orchestra.run.smithi023.stdout: "quota_max_bytes": 0, 2024-03-11T21:10:05.087 INFO:teuthology.orchestra.run.smithi023.stdout: "quota_max_objects": 0, 2024-03-11T21:10:05.087 INFO:teuthology.orchestra.run.smithi023.stdout: "tiers": [], 2024-03-11T21:10:05.087 INFO:teuthology.orchestra.run.smithi023.stdout: "tier_of": -1, 2024-03-11T21:10:05.087 INFO:teuthology.orchestra.run.smithi023.stdout: "read_tier": -1, 2024-03-11T21:10:05.088 INFO:teuthology.orchestra.run.smithi023.stdout: "write_tier": -1, 2024-03-11T21:10:05.088 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_mode": "none", 2024-03-11T21:10:05.088 INFO:teuthology.orchestra.run.smithi023.stdout: "target_max_bytes": 0, 2024-03-11T21:10:05.088 INFO:teuthology.orchestra.run.smithi023.stdout: "target_max_objects": 0, 2024-03-11T21:10:05.088 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T21:10:05.088 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T21:10:05.088 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T21:10:05.088 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_min_flush_age": 0, 2024-03-11T21:10:05.089 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_min_evict_age": 0, 2024-03-11T21:10:05.089 INFO:teuthology.orchestra.run.smithi023.stdout: "erasure_code_profile": "", 2024-03-11T21:10:05.089 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_params": { 2024-03-11T21:10:05.089 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "none" 2024-03-11T21:10:05.089 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:10:05.089 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_period": 0, 2024-03-11T21:10:05.089 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_count": 0, 2024-03-11T21:10:05.089 INFO:teuthology.orchestra.run.smithi023.stdout: "use_gmt_hitset": true, 2024-03-11T21:10:05.089 INFO:teuthology.orchestra.run.smithi023.stdout: "min_read_recency_for_promote": 0, 2024-03-11T21:10:05.089 INFO:teuthology.orchestra.run.smithi023.stdout: "min_write_recency_for_promote": 0, 2024-03-11T21:10:05.089 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T21:10:05.090 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_search_last_n": 0, 2024-03-11T21:10:05.090 INFO:teuthology.orchestra.run.smithi023.stdout: "grade_table": [], 2024-03-11T21:10:05.090 INFO:teuthology.orchestra.run.smithi023.stdout: "stripe_width": 0, 2024-03-11T21:10:05.090 INFO:teuthology.orchestra.run.smithi023.stdout: "expected_num_objects": 0, 2024-03-11T21:10:05.090 INFO:teuthology.orchestra.run.smithi023.stdout: "fast_read": false, 2024-03-11T21:10:05.090 INFO:teuthology.orchestra.run.smithi023.stdout: "options": {}, 2024-03-11T21:10:05.090 INFO:teuthology.orchestra.run.smithi023.stdout: "application_metadata": { 2024-03-11T21:10:05.090 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw": {} 2024-03-11T21:10:05.090 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:10:05.090 INFO:teuthology.orchestra.run.smithi023.stdout: "read_balance": { 2024-03-11T21:10:05.090 INFO:teuthology.orchestra.run.smithi023.stdout: "score_type": "Fair distribution", 2024-03-11T21:10:05.091 INFO:teuthology.orchestra.run.smithi023.stdout: "score_acting": 1.690000057220459, 2024-03-11T21:10:05.091 INFO:teuthology.orchestra.run.smithi023.stdout: "score_stable": 1.690000057220459, 2024-03-11T21:10:05.091 INFO:teuthology.orchestra.run.smithi023.stdout: "optimal_score": 1.3300000429153442, 2024-03-11T21:10:05.091 INFO:teuthology.orchestra.run.smithi023.stdout: "raw_score_acting": 2.25, 2024-03-11T21:10:05.091 INFO:teuthology.orchestra.run.smithi023.stdout: "raw_score_stable": 2.25, 2024-03-11T21:10:05.091 INFO:teuthology.orchestra.run.smithi023.stdout: "primary_affinity_weighted": 0.75, 2024-03-11T21:10:05.091 INFO:teuthology.orchestra.run.smithi023.stdout: "average_primary_affinity": 1, 2024-03-11T21:10:05.091 INFO:teuthology.orchestra.run.smithi023.stdout: "average_primary_affinity_weighted": 0.75 2024-03-11T21:10:05.091 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:10:05.091 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:10:05.091 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:10:05.091 INFO:teuthology.orchestra.run.smithi023.stdout: "pool": 5, 2024-03-11T21:10:05.091 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_name": "datapool", 2024-03-11T21:10:05.092 INFO:teuthology.orchestra.run.smithi023.stdout: "create_time": "2024-03-11T20:50:31.135845+0000", 2024-03-11T21:10:05.092 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 8193, 2024-03-11T21:10:05.092 INFO:teuthology.orchestra.run.smithi023.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-03-11T21:10:05.092 INFO:teuthology.orchestra.run.smithi023.stdout: "type": 1, 2024-03-11T21:10:05.092 INFO:teuthology.orchestra.run.smithi023.stdout: "size": 3, 2024-03-11T21:10:05.092 INFO:teuthology.orchestra.run.smithi023.stdout: "min_size": 2, 2024-03-11T21:10:05.092 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_rule": 0, 2024-03-11T21:10:05.092 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_count": 0, 2024-03-11T21:10:05.092 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_target": 0, 2024-03-11T21:10:05.092 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T21:10:05.093 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T21:10:05.093 INFO:teuthology.orchestra.run.smithi023.stdout: "object_hash": 2, 2024-03-11T21:10:05.093 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_autoscale_mode": "off", 2024-03-11T21:10:05.093 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num": 3, 2024-03-11T21:10:05.093 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_placement_num": 3, 2024-03-11T21:10:05.093 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_placement_num_target": 3, 2024-03-11T21:10:05.093 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num_target": 3, 2024-03-11T21:10:05.093 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num_pending": 3, 2024-03-11T21:10:05.093 INFO:teuthology.orchestra.run.smithi023.stdout: "last_pg_merge_meta": { 2024-03-11T21:10:05.094 INFO:teuthology.orchestra.run.smithi023.stdout: "source_pgid": "0.0", 2024-03-11T21:10:05.094 INFO:teuthology.orchestra.run.smithi023.stdout: "ready_epoch": 0, 2024-03-11T21:10:05.094 INFO:teuthology.orchestra.run.smithi023.stdout: "last_epoch_started": 0, 2024-03-11T21:10:05.094 INFO:teuthology.orchestra.run.smithi023.stdout: "last_epoch_clean": 0, 2024-03-11T21:10:05.094 INFO:teuthology.orchestra.run.smithi023.stdout: "source_version": "0'0", 2024-03-11T21:10:05.094 INFO:teuthology.orchestra.run.smithi023.stdout: "target_version": "0'0" 2024-03-11T21:10:05.094 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:10:05.094 INFO:teuthology.orchestra.run.smithi023.stdout: "last_change": "87", 2024-03-11T21:10:05.094 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend": "0", 2024-03-11T21:10:05.094 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T21:10:05.094 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T21:10:05.095 INFO:teuthology.orchestra.run.smithi023.stdout: "auid": 0, 2024-03-11T21:10:05.095 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_mode": "selfmanaged", 2024-03-11T21:10:05.095 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_seq": 2, 2024-03-11T21:10:05.095 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_epoch": 87, 2024-03-11T21:10:05.095 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_snaps": [], 2024-03-11T21:10:05.095 INFO:teuthology.orchestra.run.smithi023.stdout: "removed_snaps": "[]", 2024-03-11T21:10:05.095 INFO:teuthology.orchestra.run.smithi023.stdout: "quota_max_bytes": 0, 2024-03-11T21:10:05.095 INFO:teuthology.orchestra.run.smithi023.stdout: "quota_max_objects": 0, 2024-03-11T21:10:05.095 INFO:teuthology.orchestra.run.smithi023.stdout: "tiers": [], 2024-03-11T21:10:05.095 INFO:teuthology.orchestra.run.smithi023.stdout: "tier_of": -1, 2024-03-11T21:10:05.095 INFO:teuthology.orchestra.run.smithi023.stdout: "read_tier": -1, 2024-03-11T21:10:05.095 INFO:teuthology.orchestra.run.smithi023.stdout: "write_tier": -1, 2024-03-11T21:10:05.095 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_mode": "none", 2024-03-11T21:10:05.096 INFO:teuthology.orchestra.run.smithi023.stdout: "target_max_bytes": 0, 2024-03-11T21:10:05.096 INFO:teuthology.orchestra.run.smithi023.stdout: "target_max_objects": 0, 2024-03-11T21:10:05.096 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T21:10:05.096 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T21:10:05.096 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T21:10:05.096 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_min_flush_age": 0, 2024-03-11T21:10:05.096 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_min_evict_age": 0, 2024-03-11T21:10:05.096 INFO:teuthology.orchestra.run.smithi023.stdout: "erasure_code_profile": "", 2024-03-11T21:10:05.096 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_params": { 2024-03-11T21:10:05.096 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "none" 2024-03-11T21:10:05.096 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:10:05.096 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_period": 0, 2024-03-11T21:10:05.097 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_count": 0, 2024-03-11T21:10:05.097 INFO:teuthology.orchestra.run.smithi023.stdout: "use_gmt_hitset": true, 2024-03-11T21:10:05.097 INFO:teuthology.orchestra.run.smithi023.stdout: "min_read_recency_for_promote": 0, 2024-03-11T21:10:05.097 INFO:teuthology.orchestra.run.smithi023.stdout: "min_write_recency_for_promote": 0, 2024-03-11T21:10:05.097 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T21:10:05.097 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_search_last_n": 0, 2024-03-11T21:10:05.097 INFO:teuthology.orchestra.run.smithi023.stdout: "grade_table": [], 2024-03-11T21:10:05.097 INFO:teuthology.orchestra.run.smithi023.stdout: "stripe_width": 0, 2024-03-11T21:10:05.097 INFO:teuthology.orchestra.run.smithi023.stdout: "expected_num_objects": 0, 2024-03-11T21:10:05.097 INFO:teuthology.orchestra.run.smithi023.stdout: "fast_read": false, 2024-03-11T21:10:05.097 INFO:teuthology.orchestra.run.smithi023.stdout: "options": {}, 2024-03-11T21:10:05.097 INFO:teuthology.orchestra.run.smithi023.stdout: "application_metadata": { 2024-03-11T21:10:05.097 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd": {} 2024-03-11T21:10:05.098 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:10:05.098 INFO:teuthology.orchestra.run.smithi023.stdout: "read_balance": { 2024-03-11T21:10:05.098 INFO:teuthology.orchestra.run.smithi023.stdout: "score_type": "Fair distribution", 2024-03-11T21:10:05.098 INFO:teuthology.orchestra.run.smithi023.stdout: "score_acting": 3.9900000095367432, 2024-03-11T21:10:05.098 INFO:teuthology.orchestra.run.smithi023.stdout: "score_stable": 3.9900000095367432, 2024-03-11T21:10:05.098 INFO:teuthology.orchestra.run.smithi023.stdout: "optimal_score": 1.1699999570846558, 2024-03-11T21:10:05.098 INFO:teuthology.orchestra.run.smithi023.stdout: "raw_score_acting": 4.6700000762939453, 2024-03-11T21:10:05.098 INFO:teuthology.orchestra.run.smithi023.stdout: "raw_score_stable": 4.6700000762939453, 2024-03-11T21:10:05.098 INFO:teuthology.orchestra.run.smithi023.stdout: "primary_affinity_weighted": 0.75, 2024-03-11T21:10:05.098 INFO:teuthology.orchestra.run.smithi023.stdout: "average_primary_affinity": 1, 2024-03-11T21:10:05.098 INFO:teuthology.orchestra.run.smithi023.stdout: "average_primary_affinity_weighted": 0.75 2024-03-11T21:10:05.098 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:10:05.099 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:10:05.099 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:10:05.099 INFO:teuthology.orchestra.run.smithi023.stdout: "pool": 6, 2024-03-11T21:10:05.099 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_name": "default.rgw.meta", 2024-03-11T21:10:05.099 INFO:teuthology.orchestra.run.smithi023.stdout: "create_time": "2024-03-11T20:50:33.233163+0000", 2024-03-11T21:10:05.099 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-11T21:10:05.099 INFO:teuthology.orchestra.run.smithi023.stdout: "flags_names": "hashpspool", 2024-03-11T21:10:05.099 INFO:teuthology.orchestra.run.smithi023.stdout: "type": 1, 2024-03-11T21:10:05.099 INFO:teuthology.orchestra.run.smithi023.stdout: "size": 3, 2024-03-11T21:10:05.099 INFO:teuthology.orchestra.run.smithi023.stdout: "min_size": 2, 2024-03-11T21:10:05.099 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_rule": 0, 2024-03-11T21:10:05.099 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_count": 0, 2024-03-11T21:10:05.099 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_target": 0, 2024-03-11T21:10:05.099 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T21:10:05.099 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T21:10:05.100 INFO:teuthology.orchestra.run.smithi023.stdout: "object_hash": 2, 2024-03-11T21:10:05.100 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_autoscale_mode": "off", 2024-03-11T21:10:05.100 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num": 32, 2024-03-11T21:10:05.100 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_placement_num": 32, 2024-03-11T21:10:05.100 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_placement_num_target": 32, 2024-03-11T21:10:05.100 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num_target": 32, 2024-03-11T21:10:05.100 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num_pending": 32, 2024-03-11T21:10:05.100 INFO:teuthology.orchestra.run.smithi023.stdout: "last_pg_merge_meta": { 2024-03-11T21:10:05.100 INFO:teuthology.orchestra.run.smithi023.stdout: "source_pgid": "0.0", 2024-03-11T21:10:05.100 INFO:teuthology.orchestra.run.smithi023.stdout: "ready_epoch": 0, 2024-03-11T21:10:05.100 INFO:teuthology.orchestra.run.smithi023.stdout: "last_epoch_started": 0, 2024-03-11T21:10:05.100 INFO:teuthology.orchestra.run.smithi023.stdout: "last_epoch_clean": 0, 2024-03-11T21:10:05.100 INFO:teuthology.orchestra.run.smithi023.stdout: "source_version": "0'0", 2024-03-11T21:10:05.100 INFO:teuthology.orchestra.run.smithi023.stdout: "target_version": "0'0" 2024-03-11T21:10:05.101 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:10:05.101 INFO:teuthology.orchestra.run.smithi023.stdout: "last_change": "84", 2024-03-11T21:10:05.101 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend": "0", 2024-03-11T21:10:05.101 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T21:10:05.101 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T21:10:05.101 INFO:teuthology.orchestra.run.smithi023.stdout: "auid": 0, 2024-03-11T21:10:05.101 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_mode": "selfmanaged", 2024-03-11T21:10:05.102 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_seq": 0, 2024-03-11T21:10:05.102 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_epoch": 0, 2024-03-11T21:10:05.102 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_snaps": [], 2024-03-11T21:10:05.102 INFO:teuthology.orchestra.run.smithi023.stdout: "removed_snaps": "[]", 2024-03-11T21:10:05.102 INFO:teuthology.orchestra.run.smithi023.stdout: "quota_max_bytes": 0, 2024-03-11T21:10:05.102 INFO:teuthology.orchestra.run.smithi023.stdout: "quota_max_objects": 0, 2024-03-11T21:10:05.102 INFO:teuthology.orchestra.run.smithi023.stdout: "tiers": [], 2024-03-11T21:10:05.102 INFO:teuthology.orchestra.run.smithi023.stdout: "tier_of": -1, 2024-03-11T21:10:05.102 INFO:teuthology.orchestra.run.smithi023.stdout: "read_tier": -1, 2024-03-11T21:10:05.102 INFO:teuthology.orchestra.run.smithi023.stdout: "write_tier": -1, 2024-03-11T21:10:05.102 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_mode": "none", 2024-03-11T21:10:05.103 INFO:teuthology.orchestra.run.smithi023.stdout: "target_max_bytes": 0, 2024-03-11T21:10:05.103 INFO:teuthology.orchestra.run.smithi023.stdout: "target_max_objects": 0, 2024-03-11T21:10:05.103 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T21:10:05.103 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T21:10:05.103 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T21:10:05.103 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_min_flush_age": 0, 2024-03-11T21:10:05.103 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_min_evict_age": 0, 2024-03-11T21:10:05.103 INFO:teuthology.orchestra.run.smithi023.stdout: "erasure_code_profile": "", 2024-03-11T21:10:05.103 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_params": { 2024-03-11T21:10:05.103 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "none" 2024-03-11T21:10:05.103 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:10:05.103 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_period": 0, 2024-03-11T21:10:05.103 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_count": 0, 2024-03-11T21:10:05.104 INFO:teuthology.orchestra.run.smithi023.stdout: "use_gmt_hitset": true, 2024-03-11T21:10:05.104 INFO:teuthology.orchestra.run.smithi023.stdout: "min_read_recency_for_promote": 0, 2024-03-11T21:10:05.104 INFO:teuthology.orchestra.run.smithi023.stdout: "min_write_recency_for_promote": 0, 2024-03-11T21:10:05.104 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T21:10:05.104 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_search_last_n": 0, 2024-03-11T21:10:05.104 INFO:teuthology.orchestra.run.smithi023.stdout: "grade_table": [], 2024-03-11T21:10:05.104 INFO:teuthology.orchestra.run.smithi023.stdout: "stripe_width": 0, 2024-03-11T21:10:05.104 INFO:teuthology.orchestra.run.smithi023.stdout: "expected_num_objects": 0, 2024-03-11T21:10:05.104 INFO:teuthology.orchestra.run.smithi023.stdout: "fast_read": false, 2024-03-11T21:10:05.104 INFO:teuthology.orchestra.run.smithi023.stdout: "options": { 2024-03-11T21:10:05.104 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_autoscale_bias": 4 2024-03-11T21:10:05.104 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:10:05.105 INFO:teuthology.orchestra.run.smithi023.stdout: "application_metadata": { 2024-03-11T21:10:05.105 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw": {} 2024-03-11T21:10:05.105 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:10:05.105 INFO:teuthology.orchestra.run.smithi023.stdout: "read_balance": { 2024-03-11T21:10:05.105 INFO:teuthology.orchestra.run.smithi023.stdout: "score_type": "Fair distribution", 2024-03-11T21:10:05.105 INFO:teuthology.orchestra.run.smithi023.stdout: "score_acting": 1.690000057220459, 2024-03-11T21:10:05.105 INFO:teuthology.orchestra.run.smithi023.stdout: "score_stable": 1.690000057220459, 2024-03-11T21:10:05.105 INFO:teuthology.orchestra.run.smithi023.stdout: "optimal_score": 1.3300000429153442, 2024-03-11T21:10:05.105 INFO:teuthology.orchestra.run.smithi023.stdout: "raw_score_acting": 2.25, 2024-03-11T21:10:05.105 INFO:teuthology.orchestra.run.smithi023.stdout: "raw_score_stable": 2.25, 2024-03-11T21:10:05.105 INFO:teuthology.orchestra.run.smithi023.stdout: "primary_affinity_weighted": 0.75, 2024-03-11T21:10:05.105 INFO:teuthology.orchestra.run.smithi023.stdout: "average_primary_affinity": 1, 2024-03-11T21:10:05.105 INFO:teuthology.orchestra.run.smithi023.stdout: "average_primary_affinity_weighted": 0.75 2024-03-11T21:10:05.106 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:10:05.106 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:10:05.106 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:10:05.106 INFO:teuthology.orchestra.run.smithi023.stdout: "pool": 10, 2024-03-11T21:10:05.106 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_name": "unique_pool_3", 2024-03-11T21:10:05.106 INFO:teuthology.orchestra.run.smithi023.stdout: "create_time": "2024-03-11T21:07:13.796707+0000", 2024-03-11T21:10:05.106 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-11T21:10:05.107 INFO:teuthology.orchestra.run.smithi023.stdout: "flags_names": "hashpspool", 2024-03-11T21:10:05.107 INFO:teuthology.orchestra.run.smithi023.stdout: "type": 1, 2024-03-11T21:10:05.107 INFO:teuthology.orchestra.run.smithi023.stdout: "size": 3, 2024-03-11T21:10:05.107 INFO:teuthology.orchestra.run.smithi023.stdout: "min_size": 2, 2024-03-11T21:10:05.107 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_rule": 0, 2024-03-11T21:10:05.107 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_count": 0, 2024-03-11T21:10:05.107 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_target": 0, 2024-03-11T21:10:05.107 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T21:10:05.108 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T21:10:05.108 INFO:teuthology.orchestra.run.smithi023.stdout: "object_hash": 2, 2024-03-11T21:10:05.109 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_autoscale_mode": "off", 2024-03-11T21:10:05.109 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num": 16, 2024-03-11T21:10:05.109 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_placement_num": 16, 2024-03-11T21:10:05.109 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_placement_num_target": 16, 2024-03-11T21:10:05.109 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num_target": 16, 2024-03-11T21:10:05.109 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num_pending": 16, 2024-03-11T21:10:05.109 INFO:teuthology.orchestra.run.smithi023.stdout: "last_pg_merge_meta": { 2024-03-11T21:10:05.109 INFO:teuthology.orchestra.run.smithi023.stdout: "source_pgid": "0.0", 2024-03-11T21:10:05.109 INFO:teuthology.orchestra.run.smithi023.stdout: "ready_epoch": 0, 2024-03-11T21:10:05.109 INFO:teuthology.orchestra.run.smithi023.stdout: "last_epoch_started": 0, 2024-03-11T21:10:05.109 INFO:teuthology.orchestra.run.smithi023.stdout: "last_epoch_clean": 0, 2024-03-11T21:10:05.109 INFO:teuthology.orchestra.run.smithi023.stdout: "source_version": "0'0", 2024-03-11T21:10:05.109 INFO:teuthology.orchestra.run.smithi023.stdout: "target_version": "0'0" 2024-03-11T21:10:05.109 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:10:05.110 INFO:teuthology.orchestra.run.smithi023.stdout: "last_change": "310", 2024-03-11T21:10:05.110 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend": "0", 2024-03-11T21:10:05.110 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T21:10:05.110 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T21:10:05.110 INFO:teuthology.orchestra.run.smithi023.stdout: "auid": 0, 2024-03-11T21:10:05.110 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_mode": "selfmanaged", 2024-03-11T21:10:05.110 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_seq": 0, 2024-03-11T21:10:05.110 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_epoch": 0, 2024-03-11T21:10:05.110 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_snaps": [], 2024-03-11T21:10:05.110 INFO:teuthology.orchestra.run.smithi023.stdout: "removed_snaps": "[]", 2024-03-11T21:10:05.110 INFO:teuthology.orchestra.run.smithi023.stdout: "quota_max_bytes": 0, 2024-03-11T21:10:05.110 INFO:teuthology.orchestra.run.smithi023.stdout: "quota_max_objects": 0, 2024-03-11T21:10:05.110 INFO:teuthology.orchestra.run.smithi023.stdout: "tiers": [], 2024-03-11T21:10:05.110 INFO:teuthology.orchestra.run.smithi023.stdout: "tier_of": -1, 2024-03-11T21:10:05.110 INFO:teuthology.orchestra.run.smithi023.stdout: "read_tier": -1, 2024-03-11T21:10:05.111 INFO:teuthology.orchestra.run.smithi023.stdout: "write_tier": -1, 2024-03-11T21:10:05.111 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_mode": "none", 2024-03-11T21:10:05.111 INFO:teuthology.orchestra.run.smithi023.stdout: "target_max_bytes": 0, 2024-03-11T21:10:05.111 INFO:teuthology.orchestra.run.smithi023.stdout: "target_max_objects": 0, 2024-03-11T21:10:05.111 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T21:10:05.111 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T21:10:05.111 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T21:10:05.111 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_min_flush_age": 0, 2024-03-11T21:10:05.111 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_min_evict_age": 0, 2024-03-11T21:10:05.111 INFO:teuthology.orchestra.run.smithi023.stdout: "erasure_code_profile": "", 2024-03-11T21:10:05.111 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_params": { 2024-03-11T21:10:05.111 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "none" 2024-03-11T21:10:05.111 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:10:05.111 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_period": 0, 2024-03-11T21:10:05.111 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_count": 0, 2024-03-11T21:10:05.112 INFO:teuthology.orchestra.run.smithi023.stdout: "use_gmt_hitset": true, 2024-03-11T21:10:05.112 INFO:teuthology.orchestra.run.smithi023.stdout: "min_read_recency_for_promote": 0, 2024-03-11T21:10:05.112 INFO:teuthology.orchestra.run.smithi023.stdout: "min_write_recency_for_promote": 0, 2024-03-11T21:10:05.112 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T21:10:05.112 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_search_last_n": 0, 2024-03-11T21:10:05.112 INFO:teuthology.orchestra.run.smithi023.stdout: "grade_table": [], 2024-03-11T21:10:05.112 INFO:teuthology.orchestra.run.smithi023.stdout: "stripe_width": 0, 2024-03-11T21:10:05.112 INFO:teuthology.orchestra.run.smithi023.stdout: "expected_num_objects": 0, 2024-03-11T21:10:05.112 INFO:teuthology.orchestra.run.smithi023.stdout: "fast_read": false, 2024-03-11T21:10:05.112 INFO:teuthology.orchestra.run.smithi023.stdout: "options": {}, 2024-03-11T21:10:05.112 INFO:teuthology.orchestra.run.smithi023.stdout: "application_metadata": { 2024-03-11T21:10:05.112 INFO:teuthology.orchestra.run.smithi023.stdout: "rados": {} 2024-03-11T21:10:05.112 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:10:05.112 INFO:teuthology.orchestra.run.smithi023.stdout: "read_balance": { 2024-03-11T21:10:05.112 INFO:teuthology.orchestra.run.smithi023.stdout: "score_type": "Fair distribution", 2024-03-11T21:10:05.112 INFO:teuthology.orchestra.run.smithi023.stdout: "score_acting": 2.630000114440918, 2024-03-11T21:10:05.113 INFO:teuthology.orchestra.run.smithi023.stdout: "score_stable": 2.630000114440918, 2024-03-11T21:10:05.113 INFO:teuthology.orchestra.run.smithi023.stdout: "optimal_score": 1.3300000429153442, 2024-03-11T21:10:05.113 INFO:teuthology.orchestra.run.smithi023.stdout: "raw_score_acting": 3.5, 2024-03-11T21:10:05.113 INFO:teuthology.orchestra.run.smithi023.stdout: "raw_score_stable": 3.5, 2024-03-11T21:10:05.113 INFO:teuthology.orchestra.run.smithi023.stdout: "primary_affinity_weighted": 0.75, 2024-03-11T21:10:05.113 INFO:teuthology.orchestra.run.smithi023.stdout: "average_primary_affinity": 1, 2024-03-11T21:10:05.113 INFO:teuthology.orchestra.run.smithi023.stdout: "average_primary_affinity_weighted": 0.75 2024-03-11T21:10:05.113 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:10:05.113 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:10:05.113 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-11T21:10:05.113 INFO:teuthology.orchestra.run.smithi023.stdout: "osds": [ 2024-03-11T21:10:05.113 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:10:05.113 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": 0, 2024-03-11T21:10:05.113 INFO:teuthology.orchestra.run.smithi023.stdout: "uuid": "3df72b4a-8fe4-4265-b4c6-af022e1e23e1", 2024-03-11T21:10:05.113 INFO:teuthology.orchestra.run.smithi023.stdout: "up": 1, 2024-03-11T21:10:05.114 INFO:teuthology.orchestra.run.smithi023.stdout: "in": 1, 2024-03-11T21:10:05.114 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 1, 2024-03-11T21:10:05.114 INFO:teuthology.orchestra.run.smithi023.stdout: "primary_affinity": 1, 2024-03-11T21:10:05.114 INFO:teuthology.orchestra.run.smithi023.stdout: "last_clean_begin": 0, 2024-03-11T21:10:05.114 INFO:teuthology.orchestra.run.smithi023.stdout: "last_clean_end": 0, 2024-03-11T21:10:05.114 INFO:teuthology.orchestra.run.smithi023.stdout: "up_from": 8, 2024-03-11T21:10:05.114 INFO:teuthology.orchestra.run.smithi023.stdout: "up_thru": 319, 2024-03-11T21:10:05.114 INFO:teuthology.orchestra.run.smithi023.stdout: "down_at": 0, 2024-03-11T21:10:05.114 INFO:teuthology.orchestra.run.smithi023.stdout: "lost_at": 0, 2024-03-11T21:10:05.114 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2024-03-11T21:10:05.114 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:10:05.114 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:10:05.114 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:10:05.114 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6802", 2024-03-11T21:10:05.114 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 2154627393 2024-03-11T21:10:05.115 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:10:05.115 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:10:05.115 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:10:05.115 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6803", 2024-03-11T21:10:05.115 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 2154627393 2024-03-11T21:10:05.115 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:10:05.115 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:10:05.115 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:10:05.115 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_addrs": { 2024-03-11T21:10:05.115 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:10:05.115 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:10:05.115 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:10:05.116 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6804", 2024-03-11T21:10:05.116 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 2154627393 2024-03-11T21:10:05.116 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:10:05.116 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:10:05.116 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:10:05.116 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6805", 2024-03-11T21:10:05.116 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 2154627393 2024-03-11T21:10:05.116 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:10:05.116 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:10:05.116 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:10:05.116 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_back_addrs": { 2024-03-11T21:10:05.116 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:10:05.116 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:10:05.116 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:10:05.116 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6808", 2024-03-11T21:10:05.117 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 2154627393 2024-03-11T21:10:05.117 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:10:05.117 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:10:05.117 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:10:05.117 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6809", 2024-03-11T21:10:05.117 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 2154627393 2024-03-11T21:10:05.117 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:10:05.117 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:10:05.117 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:10:05.117 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_front_addrs": { 2024-03-11T21:10:05.117 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:10:05.117 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:10:05.117 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:10:05.117 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6806", 2024-03-11T21:10:05.117 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 2154627393 2024-03-11T21:10:05.117 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:10:05.118 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:10:05.118 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:10:05.118 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6807", 2024-03-11T21:10:05.118 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 2154627393 2024-03-11T21:10:05.118 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:10:05.118 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:10:05.118 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:10:05.118 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6803/2154627393", 2024-03-11T21:10:05.118 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_addr": "172.21.15.23:6805/2154627393", 2024-03-11T21:10:05.118 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_back_addr": "172.21.15.23:6809/2154627393", 2024-03-11T21:10:05.118 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_front_addr": "172.21.15.23:6807/2154627393", 2024-03-11T21:10:05.118 INFO:teuthology.orchestra.run.smithi023.stdout: "state": [ 2024-03-11T21:10:05.118 INFO:teuthology.orchestra.run.smithi023.stdout: "exists", 2024-03-11T21:10:05.119 INFO:teuthology.orchestra.run.smithi023.stdout: "up" 2024-03-11T21:10:05.119 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:10:05.119 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:10:05.119 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:10:05.119 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": 1, 2024-03-11T21:10:05.119 INFO:teuthology.orchestra.run.smithi023.stdout: "uuid": "e8fd7500-9081-41a1-8077-a4d0f0e12a2c", 2024-03-11T21:10:05.119 INFO:teuthology.orchestra.run.smithi023.stdout: "up": 1, 2024-03-11T21:10:05.119 INFO:teuthology.orchestra.run.smithi023.stdout: "in": 1, 2024-03-11T21:10:05.119 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 1, 2024-03-11T21:10:05.119 INFO:teuthology.orchestra.run.smithi023.stdout: "primary_affinity": 1, 2024-03-11T21:10:05.119 INFO:teuthology.orchestra.run.smithi023.stdout: "last_clean_begin": 13, 2024-03-11T21:10:05.119 INFO:teuthology.orchestra.run.smithi023.stdout: "last_clean_end": 173, 2024-03-11T21:10:05.119 INFO:teuthology.orchestra.run.smithi023.stdout: "up_from": 211, 2024-03-11T21:10:05.119 INFO:teuthology.orchestra.run.smithi023.stdout: "up_thru": 319, 2024-03-11T21:10:05.119 INFO:teuthology.orchestra.run.smithi023.stdout: "down_at": 174, 2024-03-11T21:10:05.120 INFO:teuthology.orchestra.run.smithi023.stdout: "lost_at": 0, 2024-03-11T21:10:05.120 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2024-03-11T21:10:05.120 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:10:05.120 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:10:05.120 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:10:05.120 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6810", 2024-03-11T21:10:05.120 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 56585354 2024-03-11T21:10:05.120 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:10:05.120 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:10:05.120 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:10:05.120 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6811", 2024-03-11T21:10:05.120 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 56585354 2024-03-11T21:10:05.120 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:10:05.120 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:10:05.120 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:10:05.121 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_addrs": { 2024-03-11T21:10:05.121 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:10:05.121 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:10:05.121 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:10:05.121 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6812", 2024-03-11T21:10:05.121 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 56585354 2024-03-11T21:10:05.121 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:10:05.121 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:10:05.121 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:10:05.121 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6813", 2024-03-11T21:10:05.121 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 56585354 2024-03-11T21:10:05.121 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:10:05.121 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:10:05.122 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:10:05.122 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_back_addrs": { 2024-03-11T21:10:05.122 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:10:05.122 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:10:05.122 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:10:05.122 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6816", 2024-03-11T21:10:05.122 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 56585354 2024-03-11T21:10:05.122 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:10:05.122 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:10:05.122 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:10:05.122 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6817", 2024-03-11T21:10:05.122 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 56585354 2024-03-11T21:10:05.122 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:10:05.123 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:10:05.123 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:10:05.123 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_front_addrs": { 2024-03-11T21:10:05.123 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:10:05.123 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:10:05.123 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:10:05.123 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6814", 2024-03-11T21:10:05.123 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 56585354 2024-03-11T21:10:05.123 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:10:05.123 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:10:05.123 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:10:05.123 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6815", 2024-03-11T21:10:05.123 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 56585354 2024-03-11T21:10:05.124 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:10:05.124 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:10:05.124 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:10:05.124 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6811/56585354", 2024-03-11T21:10:05.124 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_addr": "172.21.15.23:6813/56585354", 2024-03-11T21:10:05.124 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_back_addr": "172.21.15.23:6817/56585354", 2024-03-11T21:10:05.124 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_front_addr": "172.21.15.23:6815/56585354", 2024-03-11T21:10:05.124 INFO:teuthology.orchestra.run.smithi023.stdout: "state": [ 2024-03-11T21:10:05.124 INFO:teuthology.orchestra.run.smithi023.stdout: "exists", 2024-03-11T21:10:05.124 INFO:teuthology.orchestra.run.smithi023.stdout: "up" 2024-03-11T21:10:05.124 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:10:05.124 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:10:05.124 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:10:05.124 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": 2, 2024-03-11T21:10:05.124 INFO:teuthology.orchestra.run.smithi023.stdout: "uuid": "dd85d52a-df48-4b03-8612-cecfd82b5987", 2024-03-11T21:10:05.125 INFO:teuthology.orchestra.run.smithi023.stdout: "up": 1, 2024-03-11T21:10:05.125 INFO:teuthology.orchestra.run.smithi023.stdout: "in": 1, 2024-03-11T21:10:05.125 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 1, 2024-03-11T21:10:05.125 INFO:teuthology.orchestra.run.smithi023.stdout: "primary_affinity": 0, 2024-03-11T21:10:05.125 INFO:teuthology.orchestra.run.smithi023.stdout: "last_clean_begin": 0, 2024-03-11T21:10:05.125 INFO:teuthology.orchestra.run.smithi023.stdout: "last_clean_end": 0, 2024-03-11T21:10:05.125 INFO:teuthology.orchestra.run.smithi023.stdout: "up_from": 19, 2024-03-11T21:10:05.125 INFO:teuthology.orchestra.run.smithi023.stdout: "up_thru": 306, 2024-03-11T21:10:05.125 INFO:teuthology.orchestra.run.smithi023.stdout: "down_at": 0, 2024-03-11T21:10:05.125 INFO:teuthology.orchestra.run.smithi023.stdout: "lost_at": 0, 2024-03-11T21:10:05.125 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2024-03-11T21:10:05.125 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:10:05.125 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:10:05.125 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:10:05.125 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6818", 2024-03-11T21:10:05.126 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1770748056 2024-03-11T21:10:05.126 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:10:05.126 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:10:05.126 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:10:05.126 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6819", 2024-03-11T21:10:05.126 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1770748056 2024-03-11T21:10:05.126 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:10:05.126 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:10:05.126 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:10:05.126 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_addrs": { 2024-03-11T21:10:05.126 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:10:05.126 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:10:05.126 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:10:05.126 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6820", 2024-03-11T21:10:05.126 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1770748056 2024-03-11T21:10:05.127 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:10:05.127 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:10:05.127 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:10:05.127 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6821", 2024-03-11T21:10:05.127 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1770748056 2024-03-11T21:10:05.127 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:10:05.127 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:10:05.127 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:10:05.127 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_back_addrs": { 2024-03-11T21:10:05.127 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:10:05.127 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:10:05.127 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:10:05.127 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6824", 2024-03-11T21:10:05.127 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1770748056 2024-03-11T21:10:05.127 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:10:05.128 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:10:05.128 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:10:05.128 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6825", 2024-03-11T21:10:05.128 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1770748056 2024-03-11T21:10:05.128 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:10:05.128 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:10:05.128 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:10:05.128 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_front_addrs": { 2024-03-11T21:10:05.128 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:10:05.128 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:10:05.128 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:10:05.128 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6822", 2024-03-11T21:10:05.128 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1770748056 2024-03-11T21:10:05.128 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:10:05.128 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:10:05.129 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:10:05.129 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6823", 2024-03-11T21:10:05.129 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1770748056 2024-03-11T21:10:05.129 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:10:05.129 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:10:05.129 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:10:05.129 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6819/1770748056", 2024-03-11T21:10:05.129 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_addr": "172.21.15.23:6821/1770748056", 2024-03-11T21:10:05.129 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_back_addr": "172.21.15.23:6825/1770748056", 2024-03-11T21:10:05.129 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_front_addr": "172.21.15.23:6823/1770748056", 2024-03-11T21:10:05.129 INFO:teuthology.orchestra.run.smithi023.stdout: "state": [ 2024-03-11T21:10:05.129 INFO:teuthology.orchestra.run.smithi023.stdout: "exists", 2024-03-11T21:10:05.129 INFO:teuthology.orchestra.run.smithi023.stdout: "up" 2024-03-11T21:10:05.129 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:10:05.129 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:10:05.129 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:10:05.130 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": 3, 2024-03-11T21:10:05.130 INFO:teuthology.orchestra.run.smithi023.stdout: "uuid": "e3021106-3779-4b94-9bbf-a7724544110f", 2024-03-11T21:10:05.130 INFO:teuthology.orchestra.run.smithi023.stdout: "up": 1, 2024-03-11T21:10:05.130 INFO:teuthology.orchestra.run.smithi023.stdout: "in": 1, 2024-03-11T21:10:05.130 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 1, 2024-03-11T21:10:05.130 INFO:teuthology.orchestra.run.smithi023.stdout: "primary_affinity": 1, 2024-03-11T21:10:05.130 INFO:teuthology.orchestra.run.smithi023.stdout: "last_clean_begin": 0, 2024-03-11T21:10:05.130 INFO:teuthology.orchestra.run.smithi023.stdout: "last_clean_end": 0, 2024-03-11T21:10:05.130 INFO:teuthology.orchestra.run.smithi023.stdout: "up_from": 28, 2024-03-11T21:10:05.130 INFO:teuthology.orchestra.run.smithi023.stdout: "up_thru": 319, 2024-03-11T21:10:05.130 INFO:teuthology.orchestra.run.smithi023.stdout: "down_at": 0, 2024-03-11T21:10:05.130 INFO:teuthology.orchestra.run.smithi023.stdout: "lost_at": 0, 2024-03-11T21:10:05.130 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2024-03-11T21:10:05.130 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:10:05.130 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:10:05.131 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:10:05.131 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6826", 2024-03-11T21:10:05.131 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 938496814 2024-03-11T21:10:05.131 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:10:05.131 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:10:05.131 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:10:05.131 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6827", 2024-03-11T21:10:05.131 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 938496814 2024-03-11T21:10:05.131 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:10:05.131 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:10:05.131 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:10:05.131 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_addrs": { 2024-03-11T21:10:05.131 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:10:05.131 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:10:05.131 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:10:05.132 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6828", 2024-03-11T21:10:05.132 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 938496814 2024-03-11T21:10:05.132 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:10:05.132 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:10:05.132 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:10:05.132 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6829", 2024-03-11T21:10:05.132 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 938496814 2024-03-11T21:10:05.132 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:10:05.132 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:10:05.132 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:10:05.132 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_back_addrs": { 2024-03-11T21:10:05.132 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:10:05.132 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:10:05.132 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:10:05.132 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6832", 2024-03-11T21:10:05.132 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 938496814 2024-03-11T21:10:05.133 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:10:05.133 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:10:05.133 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:10:05.133 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6833", 2024-03-11T21:10:05.133 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 938496814 2024-03-11T21:10:05.133 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:10:05.133 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:10:05.133 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:10:05.133 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_front_addrs": { 2024-03-11T21:10:05.133 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:10:05.133 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:10:05.133 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:10:05.133 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6830", 2024-03-11T21:10:05.133 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 938496814 2024-03-11T21:10:05.133 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:10:05.133 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:10:05.134 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:10:05.134 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6831", 2024-03-11T21:10:05.134 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 938496814 2024-03-11T21:10:05.134 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:10:05.134 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:10:05.134 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:10:05.134 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6827/938496814", 2024-03-11T21:10:05.134 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_addr": "172.21.15.23:6829/938496814", 2024-03-11T21:10:05.134 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_back_addr": "172.21.15.23:6833/938496814", 2024-03-11T21:10:05.134 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_front_addr": "172.21.15.23:6831/938496814", 2024-03-11T21:10:05.134 INFO:teuthology.orchestra.run.smithi023.stdout: "state": [ 2024-03-11T21:10:05.134 INFO:teuthology.orchestra.run.smithi023.stdout: "exists", 2024-03-11T21:10:05.134 INFO:teuthology.orchestra.run.smithi023.stdout: "up" 2024-03-11T21:10:05.134 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:10:05.135 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:10:05.135 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:10:05.135 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": 4, 2024-03-11T21:10:05.135 INFO:teuthology.orchestra.run.smithi023.stdout: "uuid": "4dcbc91e-7035-413d-97bc-c9d804eb16d7", 2024-03-11T21:10:05.135 INFO:teuthology.orchestra.run.smithi023.stdout: "up": 1, 2024-03-11T21:10:05.135 INFO:teuthology.orchestra.run.smithi023.stdout: "in": 1, 2024-03-11T21:10:05.135 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 1, 2024-03-11T21:10:05.135 INFO:teuthology.orchestra.run.smithi023.stdout: "primary_affinity": 0, 2024-03-11T21:10:05.138 INFO:teuthology.orchestra.run.smithi023.stdout: "last_clean_begin": 35, 2024-03-11T21:10:05.138 INFO:teuthology.orchestra.run.smithi023.stdout: "last_clean_end": 182, 2024-03-11T21:10:05.138 INFO:teuthology.orchestra.run.smithi023.stdout: "up_from": 204, 2024-03-11T21:10:05.138 INFO:teuthology.orchestra.run.smithi023.stdout: "up_thru": 248, 2024-03-11T21:10:05.138 INFO:teuthology.orchestra.run.smithi023.stdout: "down_at": 183, 2024-03-11T21:10:05.138 INFO:teuthology.orchestra.run.smithi023.stdout: "lost_at": 0, 2024-03-11T21:10:05.138 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2024-03-11T21:10:05.138 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:10:05.138 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:10:05.138 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:10:05.138 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6800", 2024-03-11T21:10:05.138 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 3858359434 2024-03-11T21:10:05.138 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:10:05.138 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:10:05.139 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:10:05.139 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6801", 2024-03-11T21:10:05.139 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 3858359434 2024-03-11T21:10:05.139 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:10:05.139 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:10:05.139 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:10:05.139 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_addrs": { 2024-03-11T21:10:05.139 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:10:05.139 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:10:05.139 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:10:05.139 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6802", 2024-03-11T21:10:05.139 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 3858359434 2024-03-11T21:10:05.139 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:10:05.139 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:10:05.139 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:10:05.140 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6803", 2024-03-11T21:10:05.140 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 3858359434 2024-03-11T21:10:05.140 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:10:05.140 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:10:05.140 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:10:05.140 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_back_addrs": { 2024-03-11T21:10:05.140 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:10:05.140 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:10:05.140 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:10:05.140 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6806", 2024-03-11T21:10:05.140 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 3858359434 2024-03-11T21:10:05.140 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:10:05.140 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:10:05.140 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:10:05.140 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6807", 2024-03-11T21:10:05.141 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 3858359434 2024-03-11T21:10:05.141 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:10:05.141 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:10:05.141 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:10:05.141 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_front_addrs": { 2024-03-11T21:10:05.141 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:10:05.141 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:10:05.141 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:10:05.141 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6804", 2024-03-11T21:10:05.141 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 3858359434 2024-03-11T21:10:05.141 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:10:05.141 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:10:05.141 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:10:05.141 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6805", 2024-03-11T21:10:05.141 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 3858359434 2024-03-11T21:10:05.142 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:10:05.142 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:10:05.142 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:10:05.142 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.146:6801/3858359434", 2024-03-11T21:10:05.142 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_addr": "172.21.15.146:6803/3858359434", 2024-03-11T21:10:05.142 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_back_addr": "172.21.15.146:6807/3858359434", 2024-03-11T21:10:05.142 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_front_addr": "172.21.15.146:6805/3858359434", 2024-03-11T21:10:05.142 INFO:teuthology.orchestra.run.smithi023.stdout: "state": [ 2024-03-11T21:10:05.142 INFO:teuthology.orchestra.run.smithi023.stdout: "exists", 2024-03-11T21:10:05.142 INFO:teuthology.orchestra.run.smithi023.stdout: "up" 2024-03-11T21:10:05.142 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:10:05.142 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:10:05.142 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:10:05.142 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": 5, 2024-03-11T21:10:05.142 INFO:teuthology.orchestra.run.smithi023.stdout: "uuid": "aefcb766-8bae-4ff4-b71e-6691c8ac02f1", 2024-03-11T21:10:05.143 INFO:teuthology.orchestra.run.smithi023.stdout: "up": 1, 2024-03-11T21:10:05.143 INFO:teuthology.orchestra.run.smithi023.stdout: "in": 1, 2024-03-11T21:10:05.143 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 1, 2024-03-11T21:10:05.143 INFO:teuthology.orchestra.run.smithi023.stdout: "primary_affinity": 1, 2024-03-11T21:10:05.143 INFO:teuthology.orchestra.run.smithi023.stdout: "last_clean_begin": 0, 2024-03-11T21:10:05.143 INFO:teuthology.orchestra.run.smithi023.stdout: "last_clean_end": 0, 2024-03-11T21:10:05.143 INFO:teuthology.orchestra.run.smithi023.stdout: "up_from": 44, 2024-03-11T21:10:05.143 INFO:teuthology.orchestra.run.smithi023.stdout: "up_thru": 319, 2024-03-11T21:10:05.143 INFO:teuthology.orchestra.run.smithi023.stdout: "down_at": 0, 2024-03-11T21:10:05.143 INFO:teuthology.orchestra.run.smithi023.stdout: "lost_at": 0, 2024-03-11T21:10:05.143 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2024-03-11T21:10:05.143 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:10:05.143 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:10:05.144 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:10:05.144 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6808", 2024-03-11T21:10:05.144 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 4003230814 2024-03-11T21:10:05.144 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:10:05.144 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:10:05.144 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:10:05.144 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6809", 2024-03-11T21:10:05.144 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 4003230814 2024-03-11T21:10:05.144 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:10:05.144 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:10:05.144 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:10:05.144 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_addrs": { 2024-03-11T21:10:05.144 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:10:05.144 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:10:05.144 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:10:05.145 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6810", 2024-03-11T21:10:05.145 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 4003230814 2024-03-11T21:10:05.145 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:10:05.145 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:10:05.145 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:10:05.145 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6811", 2024-03-11T21:10:05.145 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 4003230814 2024-03-11T21:10:05.145 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:10:05.145 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:10:05.145 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:10:05.145 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_back_addrs": { 2024-03-11T21:10:05.145 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:10:05.145 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:10:05.145 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:10:05.145 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6814", 2024-03-11T21:10:05.146 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 4003230814 2024-03-11T21:10:05.146 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:10:05.146 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:10:05.146 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:10:05.146 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6815", 2024-03-11T21:10:05.146 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 4003230814 2024-03-11T21:10:05.146 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:10:05.146 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:10:05.146 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:10:05.146 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_front_addrs": { 2024-03-11T21:10:05.146 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:10:05.146 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:10:05.146 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:10:05.146 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6812", 2024-03-11T21:10:05.147 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 4003230814 2024-03-11T21:10:05.147 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:10:05.147 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:10:05.147 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:10:05.147 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6813", 2024-03-11T21:10:05.147 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 4003230814 2024-03-11T21:10:05.147 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:10:05.147 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:10:05.147 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:10:05.147 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.146:6809/4003230814", 2024-03-11T21:10:05.147 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_addr": "172.21.15.146:6811/4003230814", 2024-03-11T21:10:05.147 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_back_addr": "172.21.15.146:6815/4003230814", 2024-03-11T21:10:05.148 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_front_addr": "172.21.15.146:6813/4003230814", 2024-03-11T21:10:05.148 INFO:teuthology.orchestra.run.smithi023.stdout: "state": [ 2024-03-11T21:10:05.148 INFO:teuthology.orchestra.run.smithi023.stdout: "exists", 2024-03-11T21:10:05.148 INFO:teuthology.orchestra.run.smithi023.stdout: "up" 2024-03-11T21:10:05.148 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:10:05.148 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:10:05.148 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:10:05.148 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": 6, 2024-03-11T21:10:05.148 INFO:teuthology.orchestra.run.smithi023.stdout: "uuid": "5cb151e8-1f9b-4617-8a86-adfb06bb9606", 2024-03-11T21:10:05.148 INFO:teuthology.orchestra.run.smithi023.stdout: "up": 1, 2024-03-11T21:10:05.148 INFO:teuthology.orchestra.run.smithi023.stdout: "in": 1, 2024-03-11T21:10:05.148 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 1, 2024-03-11T21:10:05.148 INFO:teuthology.orchestra.run.smithi023.stdout: "primary_affinity": 1, 2024-03-11T21:10:05.148 INFO:teuthology.orchestra.run.smithi023.stdout: "last_clean_begin": 0, 2024-03-11T21:10:05.148 INFO:teuthology.orchestra.run.smithi023.stdout: "last_clean_end": 0, 2024-03-11T21:10:05.149 INFO:teuthology.orchestra.run.smithi023.stdout: "up_from": 57, 2024-03-11T21:10:05.149 INFO:teuthology.orchestra.run.smithi023.stdout: "up_thru": 306, 2024-03-11T21:10:05.149 INFO:teuthology.orchestra.run.smithi023.stdout: "down_at": 0, 2024-03-11T21:10:05.149 INFO:teuthology.orchestra.run.smithi023.stdout: "lost_at": 0, 2024-03-11T21:10:05.149 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2024-03-11T21:10:05.149 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:10:05.149 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:10:05.149 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:10:05.149 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6816", 2024-03-11T21:10:05.149 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 3334092376 2024-03-11T21:10:05.149 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:10:05.149 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:10:05.149 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:10:05.149 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6817", 2024-03-11T21:10:05.149 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 3334092376 2024-03-11T21:10:05.149 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:10:05.150 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:10:05.150 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:10:05.150 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_addrs": { 2024-03-11T21:10:05.150 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:10:05.150 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:10:05.150 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:10:05.150 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6818", 2024-03-11T21:10:05.150 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 3334092376 2024-03-11T21:10:05.150 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:10:05.150 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:10:05.150 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:10:05.150 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6819", 2024-03-11T21:10:05.150 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 3334092376 2024-03-11T21:10:05.150 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:10:05.151 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:10:05.151 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:10:05.151 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_back_addrs": { 2024-03-11T21:10:05.151 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:10:05.151 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:10:05.151 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:10:05.151 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6822", 2024-03-11T21:10:05.151 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 3334092376 2024-03-11T21:10:05.151 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:10:05.151 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:10:05.151 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:10:05.151 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6823", 2024-03-11T21:10:05.151 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 3334092376 2024-03-11T21:10:05.151 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:10:05.151 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:10:05.151 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:10:05.152 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_front_addrs": { 2024-03-11T21:10:05.152 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:10:05.152 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:10:05.152 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:10:05.152 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6820", 2024-03-11T21:10:05.152 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 3334092376 2024-03-11T21:10:05.152 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:10:05.152 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:10:05.152 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:10:05.152 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6821", 2024-03-11T21:10:05.152 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 3334092376 2024-03-11T21:10:05.152 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:10:05.152 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:10:05.152 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:10:05.152 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.146:6817/3334092376", 2024-03-11T21:10:05.153 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_addr": "172.21.15.146:6819/3334092376", 2024-03-11T21:10:05.153 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_back_addr": "172.21.15.146:6823/3334092376", 2024-03-11T21:10:05.153 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_front_addr": "172.21.15.146:6821/3334092376", 2024-03-11T21:10:05.153 INFO:teuthology.orchestra.run.smithi023.stdout: "state": [ 2024-03-11T21:10:05.153 INFO:teuthology.orchestra.run.smithi023.stdout: "exists", 2024-03-11T21:10:05.153 INFO:teuthology.orchestra.run.smithi023.stdout: "up" 2024-03-11T21:10:05.153 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:10:05.153 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:10:05.153 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:10:05.153 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": 7, 2024-03-11T21:10:05.153 INFO:teuthology.orchestra.run.smithi023.stdout: "uuid": "bdbd11a2-fa09-4f82-93a2-6b2d9c750bfd", 2024-03-11T21:10:05.153 INFO:teuthology.orchestra.run.smithi023.stdout: "up": 1, 2024-03-11T21:10:05.153 INFO:teuthology.orchestra.run.smithi023.stdout: "in": 1, 2024-03-11T21:10:05.153 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 1, 2024-03-11T21:10:05.153 INFO:teuthology.orchestra.run.smithi023.stdout: "primary_affinity": 1, 2024-03-11T21:10:05.154 INFO:teuthology.orchestra.run.smithi023.stdout: "last_clean_begin": 0, 2024-03-11T21:10:05.154 INFO:teuthology.orchestra.run.smithi023.stdout: "last_clean_end": 0, 2024-03-11T21:10:05.154 INFO:teuthology.orchestra.run.smithi023.stdout: "up_from": 68, 2024-03-11T21:10:05.154 INFO:teuthology.orchestra.run.smithi023.stdout: "up_thru": 319, 2024-03-11T21:10:05.154 INFO:teuthology.orchestra.run.smithi023.stdout: "down_at": 0, 2024-03-11T21:10:05.154 INFO:teuthology.orchestra.run.smithi023.stdout: "lost_at": 0, 2024-03-11T21:10:05.154 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2024-03-11T21:10:05.154 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:10:05.154 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:10:05.154 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:10:05.154 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6824", 2024-03-11T21:10:05.154 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1678407923 2024-03-11T21:10:05.154 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:10:05.154 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:10:05.154 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:10:05.155 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6825", 2024-03-11T21:10:05.155 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1678407923 2024-03-11T21:10:05.155 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:10:05.155 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:10:05.155 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:10:05.155 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_addrs": { 2024-03-11T21:10:05.155 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:10:05.155 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:10:05.155 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:10:05.155 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6826", 2024-03-11T21:10:05.155 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1678407923 2024-03-11T21:10:05.155 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:10:05.155 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:10:05.155 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:10:05.155 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6827", 2024-03-11T21:10:05.156 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1678407923 2024-03-11T21:10:05.156 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:10:05.156 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:10:05.156 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:10:05.156 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_back_addrs": { 2024-03-11T21:10:05.156 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:10:05.156 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:10:05.156 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:10:05.156 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6830", 2024-03-11T21:10:05.156 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1678407923 2024-03-11T21:10:05.156 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:10:05.156 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:10:05.156 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:10:05.156 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6831", 2024-03-11T21:10:05.156 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1678407923 2024-03-11T21:10:05.157 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:10:05.157 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:10:05.157 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:10:05.157 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_front_addrs": { 2024-03-11T21:10:05.157 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:10:05.157 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:10:05.157 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:10:05.157 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6828", 2024-03-11T21:10:05.157 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1678407923 2024-03-11T21:10:05.157 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:10:05.157 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:10:05.157 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:10:05.157 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6829", 2024-03-11T21:10:05.157 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1678407923 2024-03-11T21:10:05.157 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:10:05.158 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:10:05.158 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:10:05.158 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.146:6825/1678407923", 2024-03-11T21:10:05.158 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_addr": "172.21.15.146:6827/1678407923", 2024-03-11T21:10:05.158 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_back_addr": "172.21.15.146:6831/1678407923", 2024-03-11T21:10:05.158 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_front_addr": "172.21.15.146:6829/1678407923", 2024-03-11T21:10:05.158 INFO:teuthology.orchestra.run.smithi023.stdout: "state": [ 2024-03-11T21:10:05.158 INFO:teuthology.orchestra.run.smithi023.stdout: "exists", 2024-03-11T21:10:05.158 INFO:teuthology.orchestra.run.smithi023.stdout: "up" 2024-03-11T21:10:05.158 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:10:05.158 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:10:05.158 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-11T21:10:05.158 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_xinfo": [ 2024-03-11T21:10:05.158 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:10:05.158 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": 0, 2024-03-11T21:10:05.159 INFO:teuthology.orchestra.run.smithi023.stdout: "down_stamp": "0.000000", 2024-03-11T21:10:05.159 INFO:teuthology.orchestra.run.smithi023.stdout: "laggy_probability": 0, 2024-03-11T21:10:05.159 INFO:teuthology.orchestra.run.smithi023.stdout: "laggy_interval": 0, 2024-03-11T21:10:05.159 INFO:teuthology.orchestra.run.smithi023.stdout: "features": 4540701547738038271, 2024-03-11T21:10:05.159 INFO:teuthology.orchestra.run.smithi023.stdout: "old_weight": 0, 2024-03-11T21:10:05.159 INFO:teuthology.orchestra.run.smithi023.stdout: "last_purged_snaps_scrub": "2024-03-11T21:09:31.574203+0000", 2024-03-11T21:10:05.159 INFO:teuthology.orchestra.run.smithi023.stdout: "dead_epoch": 0 2024-03-11T21:10:05.159 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:10:05.159 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:10:05.159 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": 1, 2024-03-11T21:10:05.159 INFO:teuthology.orchestra.run.smithi023.stdout: "down_stamp": "0.000000", 2024-03-11T21:10:05.159 INFO:teuthology.orchestra.run.smithi023.stdout: "laggy_probability": 0, 2024-03-11T21:10:05.159 INFO:teuthology.orchestra.run.smithi023.stdout: "laggy_interval": 0, 2024-03-11T21:10:05.159 INFO:teuthology.orchestra.run.smithi023.stdout: "features": 4540701547738038271, 2024-03-11T21:10:05.159 INFO:teuthology.orchestra.run.smithi023.stdout: "old_weight": 0, 2024-03-11T21:10:05.160 INFO:teuthology.orchestra.run.smithi023.stdout: "last_purged_snaps_scrub": "2024-03-11T21:09:29.562128+0000", 2024-03-11T21:10:05.160 INFO:teuthology.orchestra.run.smithi023.stdout: "dead_epoch": 173 2024-03-11T21:10:05.160 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:10:05.160 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:10:05.160 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": 2, 2024-03-11T21:10:05.160 INFO:teuthology.orchestra.run.smithi023.stdout: "down_stamp": "0.000000", 2024-03-11T21:10:05.160 INFO:teuthology.orchestra.run.smithi023.stdout: "laggy_probability": 0, 2024-03-11T21:10:05.160 INFO:teuthology.orchestra.run.smithi023.stdout: "laggy_interval": 0, 2024-03-11T21:10:05.160 INFO:teuthology.orchestra.run.smithi023.stdout: "features": 4540701547738038271, 2024-03-11T21:10:05.160 INFO:teuthology.orchestra.run.smithi023.stdout: "old_weight": 0, 2024-03-11T21:10:05.160 INFO:teuthology.orchestra.run.smithi023.stdout: "last_purged_snaps_scrub": "2024-03-11T21:09:31.397597+0000", 2024-03-11T21:10:05.160 INFO:teuthology.orchestra.run.smithi023.stdout: "dead_epoch": 0 2024-03-11T21:10:05.160 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:10:05.160 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:10:05.160 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": 3, 2024-03-11T21:10:05.161 INFO:teuthology.orchestra.run.smithi023.stdout: "down_stamp": "0.000000", 2024-03-11T21:10:05.161 INFO:teuthology.orchestra.run.smithi023.stdout: "laggy_probability": 0, 2024-03-11T21:10:05.161 INFO:teuthology.orchestra.run.smithi023.stdout: "laggy_interval": 0, 2024-03-11T21:10:05.161 INFO:teuthology.orchestra.run.smithi023.stdout: "features": 4540701547738038271, 2024-03-11T21:10:05.161 INFO:teuthology.orchestra.run.smithi023.stdout: "old_weight": 0, 2024-03-11T21:10:05.161 INFO:teuthology.orchestra.run.smithi023.stdout: "last_purged_snaps_scrub": "2024-03-11T21:09:25.607558+0000", 2024-03-11T21:10:05.161 INFO:teuthology.orchestra.run.smithi023.stdout: "dead_epoch": 0 2024-03-11T21:10:05.161 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:10:05.161 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:10:05.161 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": 4, 2024-03-11T21:10:05.161 INFO:teuthology.orchestra.run.smithi023.stdout: "down_stamp": "0.000000", 2024-03-11T21:10:05.161 INFO:teuthology.orchestra.run.smithi023.stdout: "laggy_probability": 0, 2024-03-11T21:10:05.161 INFO:teuthology.orchestra.run.smithi023.stdout: "laggy_interval": 0, 2024-03-11T21:10:05.161 INFO:teuthology.orchestra.run.smithi023.stdout: "features": 4540701547738038271, 2024-03-11T21:10:05.161 INFO:teuthology.orchestra.run.smithi023.stdout: "old_weight": 0, 2024-03-11T21:10:05.162 INFO:teuthology.orchestra.run.smithi023.stdout: "last_purged_snaps_scrub": "2024-03-11T21:08:42.219924+0000", 2024-03-11T21:10:05.162 INFO:teuthology.orchestra.run.smithi023.stdout: "dead_epoch": 182 2024-03-11T21:10:05.162 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:10:05.162 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:10:05.162 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": 5, 2024-03-11T21:10:05.162 INFO:teuthology.orchestra.run.smithi023.stdout: "down_stamp": "0.000000", 2024-03-11T21:10:05.162 INFO:teuthology.orchestra.run.smithi023.stdout: "laggy_probability": 0, 2024-03-11T21:10:05.162 INFO:teuthology.orchestra.run.smithi023.stdout: "laggy_interval": 0, 2024-03-11T21:10:05.162 INFO:teuthology.orchestra.run.smithi023.stdout: "features": 4540701547738038271, 2024-03-11T21:10:05.162 INFO:teuthology.orchestra.run.smithi023.stdout: "old_weight": 0, 2024-03-11T21:10:05.162 INFO:teuthology.orchestra.run.smithi023.stdout: "last_purged_snaps_scrub": "2024-03-11T21:09:23.872993+0000", 2024-03-11T21:10:05.162 INFO:teuthology.orchestra.run.smithi023.stdout: "dead_epoch": 0 2024-03-11T21:10:05.162 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:10:05.162 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:10:05.162 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": 6, 2024-03-11T21:10:05.163 INFO:teuthology.orchestra.run.smithi023.stdout: "down_stamp": "0.000000", 2024-03-11T21:10:05.163 INFO:teuthology.orchestra.run.smithi023.stdout: "laggy_probability": 0, 2024-03-11T21:10:05.163 INFO:teuthology.orchestra.run.smithi023.stdout: "laggy_interval": 0, 2024-03-11T21:10:05.163 INFO:teuthology.orchestra.run.smithi023.stdout: "features": 4540701547738038271, 2024-03-11T21:10:05.163 INFO:teuthology.orchestra.run.smithi023.stdout: "old_weight": 0, 2024-03-11T21:10:05.163 INFO:teuthology.orchestra.run.smithi023.stdout: "last_purged_snaps_scrub": "2024-03-11T21:09:54.726885+0000", 2024-03-11T21:10:05.163 INFO:teuthology.orchestra.run.smithi023.stdout: "dead_epoch": 0 2024-03-11T21:10:05.163 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:10:05.163 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:10:05.163 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": 7, 2024-03-11T21:10:05.163 INFO:teuthology.orchestra.run.smithi023.stdout: "down_stamp": "0.000000", 2024-03-11T21:10:05.163 INFO:teuthology.orchestra.run.smithi023.stdout: "laggy_probability": 0, 2024-03-11T21:10:05.163 INFO:teuthology.orchestra.run.smithi023.stdout: "laggy_interval": 0, 2024-03-11T21:10:05.163 INFO:teuthology.orchestra.run.smithi023.stdout: "features": 4540701547738038271, 2024-03-11T21:10:05.163 INFO:teuthology.orchestra.run.smithi023.stdout: "old_weight": 0, 2024-03-11T21:10:05.164 INFO:teuthology.orchestra.run.smithi023.stdout: "last_purged_snaps_scrub": "2024-03-11T21:09:12.812545+0000", 2024-03-11T21:10:05.164 INFO:teuthology.orchestra.run.smithi023.stdout: "dead_epoch": 0 2024-03-11T21:10:05.164 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:10:05.164 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-11T21:10:05.164 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_upmap": [ 2024-03-11T21:10:05.164 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:10:05.164 INFO:teuthology.orchestra.run.smithi023.stdout: "pgid": "6.a", 2024-03-11T21:10:05.164 INFO:teuthology.orchestra.run.smithi023.stdout: "osds": [ 2024-03-11T21:10:05.164 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2024-03-11T21:10:05.164 INFO:teuthology.orchestra.run.smithi023.stdout: 4, 2024-03-11T21:10:05.164 INFO:teuthology.orchestra.run.smithi023.stdout: 7 2024-03-11T21:10:05.164 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:10:05.164 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:10:05.164 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-11T21:10:05.164 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_upmap_items": [ 2024-03-11T21:10:05.164 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:10:05.165 INFO:teuthology.orchestra.run.smithi023.stdout: "pgid": "2.0", 2024-03-11T21:10:05.165 INFO:teuthology.orchestra.run.smithi023.stdout: "mappings": [ 2024-03-11T21:10:05.165 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:10:05.165 INFO:teuthology.orchestra.run.smithi023.stdout: "from": 7, 2024-03-11T21:10:05.165 INFO:teuthology.orchestra.run.smithi023.stdout: "to": 2 2024-03-11T21:10:05.165 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:10:05.165 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:10:05.165 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:10:05.165 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:10:05.165 INFO:teuthology.orchestra.run.smithi023.stdout: "pgid": "4.b", 2024-03-11T21:10:05.165 INFO:teuthology.orchestra.run.smithi023.stdout: "mappings": [ 2024-03-11T21:10:05.165 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:10:05.165 INFO:teuthology.orchestra.run.smithi023.stdout: "from": 1, 2024-03-11T21:10:05.165 INFO:teuthology.orchestra.run.smithi023.stdout: "to": 2 2024-03-11T21:10:05.165 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:10:05.166 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:10:05.166 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:10:05.166 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:10:05.166 INFO:teuthology.orchestra.run.smithi023.stdout: "pgid": "4.d", 2024-03-11T21:10:05.166 INFO:teuthology.orchestra.run.smithi023.stdout: "mappings": [ 2024-03-11T21:10:05.166 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:10:05.166 INFO:teuthology.orchestra.run.smithi023.stdout: "from": 1, 2024-03-11T21:10:05.166 INFO:teuthology.orchestra.run.smithi023.stdout: "to": 5 2024-03-11T21:10:05.166 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:10:05.166 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:10:05.166 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:10:05.166 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:10:05.166 INFO:teuthology.orchestra.run.smithi023.stdout: "pgid": "4.e", 2024-03-11T21:10:05.166 INFO:teuthology.orchestra.run.smithi023.stdout: "mappings": [ 2024-03-11T21:10:05.166 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:10:05.166 INFO:teuthology.orchestra.run.smithi023.stdout: "from": 1, 2024-03-11T21:10:05.167 INFO:teuthology.orchestra.run.smithi023.stdout: "to": 2 2024-03-11T21:10:05.167 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:10:05.167 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:10:05.167 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:10:05.167 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:10:05.167 INFO:teuthology.orchestra.run.smithi023.stdout: "pgid": "4.16", 2024-03-11T21:10:05.167 INFO:teuthology.orchestra.run.smithi023.stdout: "mappings": [ 2024-03-11T21:10:05.167 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:10:05.167 INFO:teuthology.orchestra.run.smithi023.stdout: "from": 3, 2024-03-11T21:10:05.167 INFO:teuthology.orchestra.run.smithi023.stdout: "to": 2 2024-03-11T21:10:05.167 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:10:05.167 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:10:05.167 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:10:05.167 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:10:05.167 INFO:teuthology.orchestra.run.smithi023.stdout: "pgid": "4.1d", 2024-03-11T21:10:05.167 INFO:teuthology.orchestra.run.smithi023.stdout: "mappings": [ 2024-03-11T21:10:05.168 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:10:05.168 INFO:teuthology.orchestra.run.smithi023.stdout: "from": 3, 2024-03-11T21:10:05.168 INFO:teuthology.orchestra.run.smithi023.stdout: "to": 5 2024-03-11T21:10:05.168 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:10:05.168 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:10:05.168 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:10:05.168 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:10:05.168 INFO:teuthology.orchestra.run.smithi023.stdout: "pgid": "6.a", 2024-03-11T21:10:05.168 INFO:teuthology.orchestra.run.smithi023.stdout: "mappings": [ 2024-03-11T21:10:05.168 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:10:05.168 INFO:teuthology.orchestra.run.smithi023.stdout: "from": 6, 2024-03-11T21:10:05.168 INFO:teuthology.orchestra.run.smithi023.stdout: "to": 1 2024-03-11T21:10:05.168 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:10:05.168 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:10:05.168 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:10:05.169 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-11T21:10:05.169 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_upmap_primaries": [], 2024-03-11T21:10:05.169 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_temp": [], 2024-03-11T21:10:05.169 INFO:teuthology.orchestra.run.smithi023.stdout: "primary_temp": [], 2024-03-11T21:10:05.169 INFO:teuthology.orchestra.run.smithi023.stdout: "blocklist": { 2024-03-11T21:10:05.169 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:0/1618334186": "2024-03-12T20:40:45.361537+0000", 2024-03-11T21:10:05.169 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:6801/2813578081": "2024-03-12T20:40:45.361537+0000", 2024-03-11T21:10:05.169 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:0/2869940180": "2024-03-12T20:51:33.790205+0000", 2024-03-11T21:10:05.169 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:6800/3037564930": "2024-03-12T20:51:33.790205+0000", 2024-03-11T21:10:05.169 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:0/3297567084": "2024-03-12T20:51:33.790205+0000", 2024-03-11T21:10:05.169 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:0/3708753659": "2024-03-12T20:40:45.361537+0000", 2024-03-11T21:10:05.169 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:0/2004146740": "2024-03-12T20:40:45.361537+0000", 2024-03-11T21:10:05.169 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:0/827653812": "2024-03-12T20:41:21.301516+0000", 2024-03-11T21:10:05.169 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:6800/1057699850": "2024-03-12T20:41:21.301516+0000", 2024-03-11T21:10:05.169 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:6800/2813578081": "2024-03-12T20:40:45.361537+0000", 2024-03-11T21:10:05.169 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:6801/1057699850": "2024-03-12T20:41:21.301516+0000", 2024-03-11T21:10:05.170 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:0/3833230375": "2024-03-12T20:41:21.301516+0000", 2024-03-11T21:10:05.170 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:6801/3037564930": "2024-03-12T20:51:33.790205+0000", 2024-03-11T21:10:05.170 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:0/1182809102": "2024-03-12T20:51:33.790205+0000", 2024-03-11T21:10:05.170 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:0/2122134187": "2024-03-12T20:51:33.790205+0000", 2024-03-11T21:10:05.170 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:0/2356701899": "2024-03-12T20:41:21.301516+0000", 2024-03-11T21:10:05.170 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:0/843974100": "2024-03-12T20:51:33.790205+0000" 2024-03-11T21:10:05.170 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:10:05.170 INFO:teuthology.orchestra.run.smithi023.stdout: "range_blocklist": {}, 2024-03-11T21:10:05.170 INFO:teuthology.orchestra.run.smithi023.stdout: "erasure_code_profiles": { 2024-03-11T21:10:05.170 INFO:teuthology.orchestra.run.smithi023.stdout: "default": { 2024-03-11T21:10:05.170 INFO:teuthology.orchestra.run.smithi023.stdout: "crush-failure-domain": "osd", 2024-03-11T21:10:05.170 INFO:teuthology.orchestra.run.smithi023.stdout: "k": "2", 2024-03-11T21:10:05.170 INFO:teuthology.orchestra.run.smithi023.stdout: "m": "1", 2024-03-11T21:10:05.170 INFO:teuthology.orchestra.run.smithi023.stdout: "plugin": "jerasure", 2024-03-11T21:10:05.170 INFO:teuthology.orchestra.run.smithi023.stdout: "technique": "reed_sol_van" 2024-03-11T21:10:05.171 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:10:05.171 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:10:05.171 INFO:teuthology.orchestra.run.smithi023.stdout: "removed_snaps_queue": [], 2024-03-11T21:10:05.171 INFO:teuthology.orchestra.run.smithi023.stdout: "new_removed_snaps": [], 2024-03-11T21:10:05.171 INFO:teuthology.orchestra.run.smithi023.stdout: "new_purged_snaps": [], 2024-03-11T21:10:05.171 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_node_flags": {}, 2024-03-11T21:10:05.171 INFO:teuthology.orchestra.run.smithi023.stdout: "device_class_flags": {}, 2024-03-11T21:10:05.171 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": { 2024-03-11T21:10:05.171 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode_enabled": false, 2024-03-11T21:10:05.171 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_bucket_count": 0, 2024-03-11T21:10:05.171 INFO:teuthology.orchestra.run.smithi023.stdout: "degraded_stretch_mode": 0, 2024-03-11T21:10:05.171 INFO:teuthology.orchestra.run.smithi023.stdout: "recovering_stretch_mode": 0, 2024-03-11T21:10:05.171 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode_bucket": 0 2024-03-11T21:10:05.171 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:10:05.171 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-11T21:10:05.172 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:10:05.196 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:10:05.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:05.198 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:10:05.299 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:10:05.326 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:10:05.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:05.331 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:10:05.432 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:10:05.436 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:05 smithi023 bash[21372]: cluster 2024-03-11T21:10:04.315213+0000 mgr.y (mgr.24380) 713 : cluster 0 pgmap v807: 148 pgs: 148 active+clean; 3.8 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 548 B/s rd, 0 B/s wr, 817 op/s 2024-03-11T21:10:05.436 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:05 smithi023 bash[21372]: audit 2024-03-11T21:10:05.066212+0000 mon.b (mon.1) 44 : audit 0 from='client.? 172.21.15.23:0/4088553224' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-11T21:10:05.462 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:10:05.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:05.467 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:10:05.568 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:05.577 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:05 smithi146 bash[23809]: cluster 2024-03-11T21:10:04.315213+0000 mgr.y (mgr.24380) 713 : cluster 0 pgmap v807: 148 pgs: 148 active+clean; 3.8 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 548 B/s rd, 0 B/s wr, 817 op/s 2024-03-11T21:10:05.577 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:05 smithi146 bash[23809]: audit 2024-03-11T21:10:05.066212+0000 mon.b (mon.1) 44 : audit 0 from='client.? 172.21.15.23:0/4088553224' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-11T21:10:05.593 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:05.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:05.595 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:10:05.695 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:10:05.706 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:05 smithi023 bash[29146]: cluster 2024-03-11T21:10:04.315213+0000 mgr.y (mgr.24380) 713 : cluster 0 pgmap v807: 148 pgs: 148 active+clean; 3.8 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 548 B/s rd, 0 B/s wr, 817 op/s 2024-03-11T21:10:05.706 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:05 smithi023 bash[29146]: audit 2024-03-11T21:10:05.066212+0000 mon.b (mon.1) 44 : audit 0 from='client.? 172.21.15.23:0/4088553224' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-11T21:10:05.727 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:10:05.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:05.730 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:10:05.830 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:05.856 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:05.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:05.858 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:10:05.959 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:10:05.968 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:10:05 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:10:05.994 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:10:05.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:05.995 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:10:06.096 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:10:06.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:06.127 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:10:06.127 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:10:06.228 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:10:06.289 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:10:06.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:06.293 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:10:06.394 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:06.448 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:06.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:06.452 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:10:06.553 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:06.598 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:06.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:06.604 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:10:06.705 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:10:06.739 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:10:06.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:06.742 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:10:06.843 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:10:06.867 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:10:06.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:06.869 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:10:06.970 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:10:06.998 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:10:07.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:07.003 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:10:07.104 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:07.113 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:06 smithi146 bash[23809]: audit 2024-03-11T21:10:05.814588+0000 mgr.y (mgr.24380) 714 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:10:07.113 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:06 smithi146 bash[23809]: audit 2024-03-11T21:10:05.971328+0000 mon.a (mon.0) 1712 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:10:07.113 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:06 smithi146 bash[23809]: audit 2024-03-11T21:10:05.972314+0000 mon.a (mon.0) 1713 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:10:07.113 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:06 smithi146 bash[23809]: audit 2024-03-11T21:10:06.124482+0000 mon.a (mon.0) 1714 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:10:07.113 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:06 smithi146 bash[23809]: cluster 2024-03-11T21:10:06.316167+0000 mgr.y (mgr.24380) 715 : cluster 0 pgmap v808: 148 pgs: 148 active+clean; 3.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 806 op/s 2024-03-11T21:10:07.129 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:07.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:07.132 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:10:07.205 INFO:tasks.thrashosds.thrasher:j is {'epoch': 350, 'fsid': '67ba00f8-dfe7-11ee-95c8-87774f69a715', 'created': '2024-03-11T20:40:13.664802+0000', 'modified': '2024-03-11T21:10:03.330707+0000', 'last_up_change': '2024-03-11T21:00:47.324817+0000', 'last_in_change': '2024-03-11T20:57:23.560527+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 20, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 10, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-11T20:45:31.964570+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '142', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6, 'score_stable': 6, 'optimal_score': 0.5, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.75, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.75}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2024-03-11T20:50:26.421486+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '78', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1.3300000429153442, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.75, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.75}}, {'pool': 3, 'pool_name': 'default.rgw.log', 'create_time': '2024-03-11T20:50:28.592615+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1.3300000429153442, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.75, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.75}}, {'pool': 4, 'pool_name': 'default.rgw.control', 'create_time': '2024-03-11T20:50:30.536371+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.690000057220459, 'score_stable': 1.690000057220459, 'optimal_score': 1.3300000429153442, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.75, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.75}}, {'pool': 5, 'pool_name': 'datapool', 'create_time': '2024-03-11T20:50:31.135845+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 87, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3.990000009536743, 'score_stable': 3.990000009536743, 'optimal_score': 1.1699999570846558, 'raw_score_acting': 4.670000076293945, 'raw_score_stable': 4.670000076293945, 'primary_affinity_weighted': 0.75, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.75}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-03-11T20:50:33.233163+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.690000057220459, 'score_stable': 1.690000057220459, 'optimal_score': 1.3300000429153442, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.75, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.75}}, {'pool': 10, 'pool_name': 'unique_pool_3', 'create_time': '2024-03-11T21:07:13.796707+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '310', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.630000114440918, 'score_stable': 2.630000114440918, 'optimal_score': 1.3300000429153442, 'raw_score_acting': 3.5, 'raw_score_stable': 3.5, 'primary_affinity_weighted': 0.75, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.75}}], 'osds': [{'osd': 0, 'uuid': '3df72b4a-8fe4-4265-b4c6-af022e1e23e1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 319, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6802', 'nonce': 2154627393}, {'type': 'v1', 'addr': '172.21.15.23:6803', 'nonce': 2154627393}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6804', 'nonce': 2154627393}, {'type': 'v1', 'addr': '172.21.15.23:6805', 'nonce': 2154627393}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6808', 'nonce': 2154627393}, {'type': 'v1', 'addr': '172.21.15.23:6809', 'nonce': 2154627393}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6806', 'nonce': 2154627393}, {'type': 'v1', 'addr': '172.21.15.23:6807', 'nonce': 2154627393}]}, 'public_addr': '172.21.15.23:6803/2154627393', 'cluster_addr': '172.21.15.23:6805/2154627393', 'heartbeat_back_addr': '172.21.15.23:6809/2154627393', 'heartbeat_front_addr': '172.21.15.23:6807/2154627393', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'e8fd7500-9081-41a1-8077-a4d0f0e12a2c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 13, 'last_clean_end': 173, 'up_from': 211, 'up_thru': 319, 'down_at': 174, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6810', 'nonce': 56585354}, {'type': 'v1', 'addr': '172.21.15.23:6811', 'nonce': 56585354}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6812', 'nonce': 56585354}, {'type': 'v1', 'addr': '172.21.15.23:6813', 'nonce': 56585354}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6816', 'nonce': 56585354}, {'type': 'v1', 'addr': '172.21.15.23:6817', 'nonce': 56585354}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6814', 'nonce': 56585354}, {'type': 'v1', 'addr': '172.21.15.23:6815', 'nonce': 56585354}]}, 'public_addr': '172.21.15.23:6811/56585354', 'cluster_addr': '172.21.15.23:6813/56585354', 'heartbeat_back_addr': '172.21.15.23:6817/56585354', 'heartbeat_front_addr': '172.21.15.23:6815/56585354', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'dd85d52a-df48-4b03-8612-cecfd82b5987', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 19, 'up_thru': 306, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6818', 'nonce': 1770748056}, {'type': 'v1', 'addr': '172.21.15.23:6819', 'nonce': 1770748056}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6820', 'nonce': 1770748056}, {'type': 'v1', 'addr': '172.21.15.23:6821', 'nonce': 1770748056}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6824', 'nonce': 1770748056}, {'type': 'v1', 'addr': '172.21.15.23:6825', 'nonce': 1770748056}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6822', 'nonce': 1770748056}, {'type': 'v1', 'addr': '172.21.15.23:6823', 'nonce': 1770748056}]}, 'public_addr': '172.21.15.23:6819/1770748056', 'cluster_addr': '172.21.15.23:6821/1770748056', 'heartbeat_back_addr': '172.21.15.23:6825/1770748056', 'heartbeat_front_addr': '172.21.15.23:6823/1770748056', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'e3021106-3779-4b94-9bbf-a7724544110f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 28, 'up_thru': 319, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6826', 'nonce': 938496814}, {'type': 'v1', 'addr': '172.21.15.23:6827', 'nonce': 938496814}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6828', 'nonce': 938496814}, {'type': 'v1', 'addr': '172.21.15.23:6829', 'nonce': 938496814}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6832', 'nonce': 938496814}, {'type': 'v1', 'addr': '172.21.15.23:6833', 'nonce': 938496814}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6830', 'nonce': 938496814}, {'type': 'v1', 'addr': '172.21.15.23:6831', 'nonce': 938496814}]}, 'public_addr': '172.21.15.23:6827/938496814', 'cluster_addr': '172.21.15.23:6829/938496814', 'heartbeat_back_addr': '172.21.15.23:6833/938496814', 'heartbeat_front_addr': '172.21.15.23:6831/938496814', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '4dcbc91e-7035-413d-97bc-c9d804eb16d7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 35, 'last_clean_end': 182, 'up_from': 204, 'up_thru': 248, 'down_at': 183, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6800', 'nonce': 3858359434}, {'type': 'v1', 'addr': '172.21.15.146:6801', 'nonce': 3858359434}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6802', 'nonce': 3858359434}, {'type': 'v1', 'addr': '172.21.15.146:6803', 'nonce': 3858359434}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6806', 'nonce': 3858359434}, {'type': 'v1', 'addr': '172.21.15.146:6807', 'nonce': 3858359434}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6804', 'nonce': 3858359434}, {'type': 'v1', 'addr': '172.21.15.146:6805', 'nonce': 3858359434}]}, 'public_addr': '172.21.15.146:6801/3858359434', 'cluster_addr': '172.21.15.146:6803/3858359434', 'heartbeat_back_addr': '172.21.15.146:6807/3858359434', 'heartbeat_front_addr': '172.21.15.146:6805/3858359434', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'aefcb766-8bae-4ff4-b71e-6691c8ac02f1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 44, 'up_thru': 319, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6808', 'nonce': 4003230814}, {'type': 'v1', 'addr': '172.21.15.146:6809', 'nonce': 4003230814}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6810', 'nonce': 4003230814}, {'type': 'v1', 'addr': '172.21.15.146:6811', 'nonce': 4003230814}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6814', 'nonce': 4003230814}, {'type': 'v1', 'addr': '172.21.15.146:6815', 'nonce': 4003230814}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6812', 'nonce': 4003230814}, {'type': 'v1', 'addr': '172.21.15.146:6813', 'nonce': 4003230814}]}, 'public_addr': '172.21.15.146:6809/4003230814', 'cluster_addr': '172.21.15.146:6811/4003230814', 'heartbeat_back_addr': '172.21.15.146:6815/4003230814', 'heartbeat_front_addr': '172.21.15.146:6813/4003230814', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '5cb151e8-1f9b-4617-8a86-adfb06bb9606', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 57, 'up_thru': 306, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6816', 'nonce': 3334092376}, {'type': 'v1', 'addr': '172.21.15.146:6817', 'nonce': 3334092376}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6818', 'nonce': 3334092376}, {'type': 'v1', 'addr': '172.21.15.146:6819', 'nonce': 3334092376}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6822', 'nonce': 3334092376}, {'type': 'v1', 'addr': '172.21.15.146:6823', 'nonce': 3334092376}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6820', 'nonce': 3334092376}, {'type': 'v1', 'addr': '172.21.15.146:6821', 'nonce': 3334092376}]}, 'public_addr': '172.21.15.146:6817/3334092376', 'cluster_addr': '172.21.15.146:6819/3334092376', 'heartbeat_back_addr': '172.21.15.146:6823/3334092376', 'heartbeat_front_addr': '172.21.15.146:6821/3334092376', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'bdbd11a2-fa09-4f82-93a2-6b2d9c750bfd', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 68, 'up_thru': 319, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6824', 'nonce': 1678407923}, {'type': 'v1', 'addr': '172.21.15.146:6825', 'nonce': 1678407923}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6826', 'nonce': 1678407923}, {'type': 'v1', 'addr': '172.21.15.146:6827', 'nonce': 1678407923}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6830', 'nonce': 1678407923}, {'type': 'v1', 'addr': '172.21.15.146:6831', 'nonce': 1678407923}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6828', 'nonce': 1678407923}, {'type': 'v1', 'addr': '172.21.15.146:6829', 'nonce': 1678407923}]}, 'public_addr': '172.21.15.146:6825/1678407923', 'cluster_addr': '172.21.15.146:6827/1678407923', 'heartbeat_back_addr': '172.21.15.146:6831/1678407923', 'heartbeat_front_addr': '172.21.15.146:6829/1678407923', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T21:09:31.574203+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T21:09:29.562128+0000', 'dead_epoch': 173}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T21:09:31.397597+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T21:09:25.607558+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T21:08:42.219924+0000', 'dead_epoch': 182}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T21:09:23.872993+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T21:09:54.726885+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T21:09:12.812545+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '6.a', 'osds': [0, 4, 7]}], 'pg_upmap_items': [{'pgid': '2.0', 'mappings': [{'from': 7, 'to': 2}]}, {'pgid': '4.b', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.d', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.e', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.16', 'mappings': [{'from': 3, 'to': 2}]}, {'pgid': '4.1d', 'mappings': [{'from': 3, 'to': 5}]}, {'pgid': '6.a', 'mappings': [{'from': 6, 'to': 1}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.23:0/1618334186': '2024-03-12T20:40:45.361537+0000', '172.21.15.23:6801/2813578081': '2024-03-12T20:40:45.361537+0000', '172.21.15.23:0/2869940180': '2024-03-12T20:51:33.790205+0000', '172.21.15.23:6800/3037564930': '2024-03-12T20:51:33.790205+0000', '172.21.15.23:0/3297567084': '2024-03-12T20:51:33.790205+0000', '172.21.15.23:0/3708753659': '2024-03-12T20:40:45.361537+0000', '172.21.15.23:0/2004146740': '2024-03-12T20:40:45.361537+0000', '172.21.15.23:0/827653812': '2024-03-12T20:41:21.301516+0000', '172.21.15.23:6800/1057699850': '2024-03-12T20:41:21.301516+0000', '172.21.15.23:6800/2813578081': '2024-03-12T20:40:45.361537+0000', '172.21.15.23:6801/1057699850': '2024-03-12T20:41:21.301516+0000', '172.21.15.23:0/3833230375': '2024-03-12T20:41:21.301516+0000', '172.21.15.23:6801/3037564930': '2024-03-12T20:51:33.790205+0000', '172.21.15.23:0/1182809102': '2024-03-12T20:51:33.790205+0000', '172.21.15.23:0/2122134187': '2024-03-12T20:51:33.790205+0000', '172.21.15.23:0/2356701899': '2024-03-12T20:41:21.301516+0000', '172.21.15.23:0/843974100': '2024-03-12T20:51:33.790205+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-03-11T21:10:07.205 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T21:10:07.233 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:10:07.257 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:06 smithi023 bash[21372]: audit 2024-03-11T21:10:05.814588+0000 mgr.y (mgr.24380) 714 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:10:07.257 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:06 smithi023 bash[21372]: audit 2024-03-11T21:10:05.971328+0000 mon.a (mon.0) 1712 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:10:07.257 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:06 smithi023 bash[21372]: audit 2024-03-11T21:10:05.972314+0000 mon.a (mon.0) 1713 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:10:07.257 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:06 smithi023 bash[21372]: audit 2024-03-11T21:10:06.124482+0000 mon.a (mon.0) 1714 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:10:07.257 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:06 smithi023 bash[21372]: cluster 2024-03-11T21:10:06.316167+0000 mgr.y (mgr.24380) 715 : cluster 0 pgmap v808: 148 pgs: 148 active+clean; 3.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 806 op/s 2024-03-11T21:10:07.257 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:06 smithi023 bash[29146]: audit 2024-03-11T21:10:05.814588+0000 mgr.y (mgr.24380) 714 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:10:07.257 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:06 smithi023 bash[29146]: audit 2024-03-11T21:10:05.971328+0000 mon.a (mon.0) 1712 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:10:07.257 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:06 smithi023 bash[29146]: audit 2024-03-11T21:10:05.972314+0000 mon.a (mon.0) 1713 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:10:07.257 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:06 smithi023 bash[29146]: audit 2024-03-11T21:10:06.124482+0000 mon.a (mon.0) 1714 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:10:07.258 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:06 smithi023 bash[29146]: cluster 2024-03-11T21:10:06.316167+0000 mgr.y (mgr.24380) 715 : cluster 0 pgmap v808: 148 pgs: 148 active+clean; 3.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 806 op/s 2024-03-11T21:10:07.285 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:10:07.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:07.289 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:10:07.390 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:10:07.393 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:10:07.412 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:10:07.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:07.414 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:10:07.515 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:10:07.547 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:10:07.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:07.551 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:10:07.651 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:07.682 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:07.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:07.684 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:10:07.785 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:10:07.816 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:10:07.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:07.820 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:10:07.921 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:07.944 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:07.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:07.947 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:10:08.048 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:10:08.071 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:10:08.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:08.074 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:10:08.175 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:10:08.185 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:07 smithi023 bash[21372]: cluster 2024-03-11T21:10:06.721149+0000 osd.4 (osd.4) 109 : cluster 0 purged_snaps scrub starts 2024-03-11T21:10:08.185 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:07 smithi023 bash[21372]: cluster 2024-03-11T21:10:06.721242+0000 osd.4 (osd.4) 110 : cluster 0 purged_snaps scrub ok 2024-03-11T21:10:08.185 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:07 smithi023 bash[21372]: cluster 2024-03-11T21:10:06.985154+0000 mon.a (mon.0) 1715 : cluster 0 osdmap e351: 8 total, 8 up, 8 in 2024-03-11T21:10:08.185 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:07 smithi023 bash[21372]: audit 2024-03-11T21:10:07.032315+0000 mon.a (mon.0) 1716 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:10:08.185 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:07 smithi023 bash[21372]: audit 2024-03-11T21:10:07.033688+0000 mon.a (mon.0) 1717 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:10:08.185 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:07 smithi023 bash[21372]: audit 2024-03-11T21:10:07.040191+0000 mon.a (mon.0) 1718 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:10:08.186 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:07 smithi023 bash[29146]: cluster 2024-03-11T21:10:06.721149+0000 osd.4 (osd.4) 109 : cluster 0 purged_snaps scrub starts 2024-03-11T21:10:08.186 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:07 smithi023 bash[29146]: cluster 2024-03-11T21:10:06.721242+0000 osd.4 (osd.4) 110 : cluster 0 purged_snaps scrub ok 2024-03-11T21:10:08.186 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:07 smithi023 bash[29146]: cluster 2024-03-11T21:10:06.985154+0000 mon.a (mon.0) 1715 : cluster 0 osdmap e351: 8 total, 8 up, 8 in 2024-03-11T21:10:08.186 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:07 smithi023 bash[29146]: audit 2024-03-11T21:10:07.032315+0000 mon.a (mon.0) 1716 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:10:08.186 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:07 smithi023 bash[29146]: audit 2024-03-11T21:10:07.033688+0000 mon.a (mon.0) 1717 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:10:08.186 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:07 smithi023 bash[29146]: audit 2024-03-11T21:10:07.040191+0000 mon.a (mon.0) 1718 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:10:08.201 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:10:08.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:08.204 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:10:08.258 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:07 smithi146 bash[23809]: cluster 2024-03-11T21:10:06.721149+0000 osd.4 (osd.4) 109 : cluster 0 purged_snaps scrub starts 2024-03-11T21:10:08.258 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:07 smithi146 bash[23809]: cluster 2024-03-11T21:10:06.721242+0000 osd.4 (osd.4) 110 : cluster 0 purged_snaps scrub ok 2024-03-11T21:10:08.258 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:07 smithi146 bash[23809]: cluster 2024-03-11T21:10:06.985154+0000 mon.a (mon.0) 1715 : cluster 0 osdmap e351: 8 total, 8 up, 8 in 2024-03-11T21:10:08.258 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:07 smithi146 bash[23809]: audit 2024-03-11T21:10:07.032315+0000 mon.a (mon.0) 1716 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:10:08.259 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:07 smithi146 bash[23809]: audit 2024-03-11T21:10:07.033688+0000 mon.a (mon.0) 1717 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:10:08.259 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:07 smithi146 bash[23809]: audit 2024-03-11T21:10:07.040191+0000 mon.a (mon.0) 1718 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:10:08.304 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:10:08.335 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:10:08.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:08.341 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:10:08.414 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T21:10:08.441 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:10:08.469 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-11T21:10:08.473 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:10:08 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:10:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:10:08.504 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:10:08.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:08.510 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:10:08.611 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:08.633 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:08.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:08.636 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:10:08.736 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:10:08.782 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:10:08.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:08.786 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:10:08.887 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:08.911 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:08.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:08.916 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:10:09.017 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:09.026 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:08 smithi023 bash[21372]: cluster 2024-03-11T21:10:08.316697+0000 mgr.y (mgr.24380) 716 : cluster 0 pgmap v810: 148 pgs: 148 active+clean; 3.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 743 op/s 2024-03-11T21:10:09.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:08 smithi023 bash[29146]: cluster 2024-03-11T21:10:08.316697+0000 mgr.y (mgr.24380) 716 : cluster 0 pgmap v810: 148 pgs: 148 active+clean; 3.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 743 op/s 2024-03-11T21:10:09.042 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:09.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:09.044 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:10:09.145 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:10:09.154 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:08 smithi146 bash[23809]: cluster 2024-03-11T21:10:08.316697+0000 mgr.y (mgr.24380) 716 : cluster 0 pgmap v810: 148 pgs: 148 active+clean; 3.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 743 op/s 2024-03-11T21:10:09.166 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:10:09.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:09.170 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:10:09.271 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:10:09.296 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:10:09.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:09.300 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:10:09.401 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:09.424 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:09.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:09.426 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:10:09.527 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:10:09.562 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:10:09.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:09.564 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:10:09.664 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:09.691 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:09.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:09.696 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:10:09.797 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:10:09.821 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:10:09.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:09.824 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:10:09.925 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:10:09.964 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:10:09.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:09.969 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:10:10.070 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:10:10.094 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:10:10.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:10.099 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:10:10.200 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:10:10.227 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:10:10.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:10.229 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:10:10.330 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:10:10.354 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:10:10.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:10.361 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:10:10.462 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:10.494 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:10.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:10.498 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:10:10.598 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:10:10.623 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:10:10.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:10.628 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:10:10.729 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:10.758 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:10.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:10.761 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:10:10.861 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:10:10.886 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:10:10.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:10.891 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:10:10.992 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:10:11.017 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:10:11.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:11.021 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:10:11.122 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:10:11.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:11.155 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:10:11.155 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:10:11.255 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:11.295 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:11.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:11.296 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:10:11.399 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:10:11.424 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:10:11.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:11.428 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:10:11.528 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:10:11.539 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:11 smithi146 bash[23809]: cluster 2024-03-11T21:10:10.317617+0000 mgr.y (mgr.24380) 717 : cluster 0 pgmap v811: 148 pgs: 148 active+clean; 3.4 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 886 op/s 2024-03-11T21:10:11.559 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:10:11.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:11.560 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:10:11.661 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:10:11.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:11.699 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:10:11.700 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:10:11.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:11 smithi023 bash[21372]: cluster 2024-03-11T21:10:10.317617+0000 mgr.y (mgr.24380) 717 : cluster 0 pgmap v811: 148 pgs: 148 active+clean; 3.4 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 886 op/s 2024-03-11T21:10:11.778 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:11 smithi023 bash[29146]: cluster 2024-03-11T21:10:10.317617+0000 mgr.y (mgr.24380) 717 : cluster 0 pgmap v811: 148 pgs: 148 active+clean; 3.4 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 886 op/s 2024-03-11T21:10:11.801 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:10:11.829 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:10:11.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:11.834 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:10:11.935 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:11.954 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:11.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:11.955 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:10:12.054 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:10:12.056 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:10:12.102 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:10:12.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:12.105 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:10:12.206 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:10:12.231 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:10:12.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:12.237 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:10:12.338 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:10:12.369 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:10:12.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:12.371 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:10:12.472 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:10:12.494 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:10:12.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:12.497 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:10:12.598 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:12.621 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:12.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:12.623 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:10:12.724 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:10:12.757 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:10:12.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:12.761 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:10:12.861 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:12.884 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:12.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:12.889 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:10:12.990 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:10:13.023 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:10:13.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:13.029 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:10:13.073 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:10:13.073 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:10:13.130 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:10:13.172 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:10:13.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:13.174 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:10:13.274 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:10:13.307 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:10:13.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:13.309 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:10:13.410 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:10:13.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:13 smithi023 bash[21372]: cluster 2024-03-11T21:10:12.319394+0000 mgr.y (mgr.24380) 718 : cluster 0 pgmap v812: 148 pgs: 148 active+clean; 3.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.03k op/s 2024-03-11T21:10:13.418 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:13 smithi023 bash[29146]: cluster 2024-03-11T21:10:12.319394+0000 mgr.y (mgr.24380) 718 : cluster 0 pgmap v812: 148 pgs: 148 active+clean; 3.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.03k op/s 2024-03-11T21:10:13.428 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:10:13.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:13.430 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:10:13.508 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:13 smithi146 bash[23809]: cluster 2024-03-11T21:10:12.319394+0000 mgr.y (mgr.24380) 718 : cluster 0 pgmap v812: 148 pgs: 148 active+clean; 3.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.03k op/s 2024-03-11T21:10:13.530 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:13.553 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:13.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:13.555 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:10:13.656 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:13.683 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:13.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:13.685 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:10:13.786 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:10:13.816 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:10:13.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:13.821 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:10:13.922 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:13.949 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:13.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:13.957 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:10:14.058 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:14.096 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:14.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:14.099 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:10:14.199 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:10:14.232 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:10:14.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:14.235 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:10:14.336 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:10:14.362 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:10:14.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:14.365 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:10:14.465 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:10:14.506 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:10:14.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:14.511 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:10:14.612 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:10:14.646 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:10:14.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:14.649 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:10:14.750 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:10:14.775 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:10:14.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:14.781 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:10:14.882 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:14.914 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:14.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:14.917 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:10:15.018 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:10:15.062 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:10:15.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:15.064 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:10:15.099 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-03-11T21:10:15.099 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:10:15.165 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:15.195 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:15.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:15.200 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:10:15.301 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:15.333 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:15.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:15.335 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:10:15.436 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:10:15.445 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:15 smithi023 bash[21372]: cluster 2024-03-11T21:10:14.319828+0000 mgr.y (mgr.24380) 719 : cluster 0 pgmap v813: 148 pgs: 148 active+clean; 3.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 927 op/s 2024-03-11T21:10:15.445 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:15 smithi023 bash[29146]: cluster 2024-03-11T21:10:14.319828+0000 mgr.y (mgr.24380) 719 : cluster 0 pgmap v813: 148 pgs: 148 active+clean; 3.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 927 op/s 2024-03-11T21:10:15.459 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:10:15.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:15.461 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:10:15.561 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:15.574 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:15 smithi146 bash[23809]: cluster 2024-03-11T21:10:14.319828+0000 mgr.y (mgr.24380) 719 : cluster 0 pgmap v813: 148 pgs: 148 active+clean; 3.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 927 op/s 2024-03-11T21:10:15.594 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:15.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:15.599 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:10:15.700 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:10:15.720 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:10:15.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:15.727 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:10:15.827 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:10:15.851 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:10:15.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:15.859 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:10:15.960 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:10:15.970 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:10:15 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:10:15.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:15.990 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:10:15.990 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:10:16.091 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:10:16.115 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:10:16.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:16.119 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:10:16.219 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:10:16.237 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:10:16.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:16.240 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:10:16.340 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:10:16.369 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:10:16.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:16.371 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:10:16.472 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:10:16.506 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:10:16.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:16.510 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:10:16.611 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:16.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:16.650 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:16.650 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:10:16.751 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:10:16.774 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:10:16.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:16.781 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:10:16.827 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-03-11T21:10:16.882 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:10:16.907 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:10:16.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:16.913 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:10:17.014 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:10:17.036 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:10:17.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:17.040 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:10:17.141 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:10:17.170 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:10:17.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:17.177 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:10:17.278 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:10:17.306 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:10:17.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:17.311 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:10:17.412 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:17.445 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:17.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:17.448 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:10:17.508 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:17 smithi146 bash[23809]: audit 2024-03-11T21:10:15.825224+0000 mgr.y (mgr.24380) 720 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:10:17.508 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:17 smithi146 bash[23809]: cluster 2024-03-11T21:10:16.320689+0000 mgr.y (mgr.24380) 721 : cluster 0 pgmap v814: 148 pgs: 148 active+clean; 3.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 921 B/s rd, 0 B/s wr, 998 op/s 2024-03-11T21:10:17.516 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:17 smithi023 bash[21372]: audit 2024-03-11T21:10:15.825224+0000 mgr.y (mgr.24380) 720 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:10:17.516 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:17 smithi023 bash[21372]: cluster 2024-03-11T21:10:16.320689+0000 mgr.y (mgr.24380) 721 : cluster 0 pgmap v814: 148 pgs: 148 active+clean; 3.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 921 B/s rd, 0 B/s wr, 998 op/s 2024-03-11T21:10:17.516 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:17 smithi023 bash[29146]: audit 2024-03-11T21:10:15.825224+0000 mgr.y (mgr.24380) 720 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:10:17.516 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:17 smithi023 bash[29146]: cluster 2024-03-11T21:10:16.320689+0000 mgr.y (mgr.24380) 721 : cluster 0 pgmap v814: 148 pgs: 148 active+clean; 3.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 921 B/s rd, 0 B/s wr, 998 op/s 2024-03-11T21:10:17.549 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:10:17.578 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:10:17.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:17.582 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:10:17.683 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:10:17.719 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:10:17.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:17.722 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:10:17.823 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:10:17.854 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:10:17.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:17.860 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:10:17.961 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:10:17.984 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:10:17.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:17.986 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:10:18.087 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:10:18.126 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:10:18.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:18.130 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:10:18.231 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:10:18.259 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:10:18.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:18.266 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:10:18.367 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:18.392 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:18.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:18.395 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:10:18.496 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:10:18.504 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:10:18 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:10:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:10:18.518 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:10:18.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:18.520 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:10:18.621 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:10:18.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:18.653 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:10:18.653 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:10:18.754 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:10:18.780 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:10:18.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:18.783 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:10:18.883 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:10:18.917 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:10:18.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:18.921 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:10:19.022 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:10:19.062 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:10:19.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:19.064 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:10:19.165 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:10:19.195 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:10:19.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:19.199 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:10:19.300 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:19.323 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:19.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:19.330 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:10:19.431 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:10:19.456 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:10:19.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:19.458 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:10:19.559 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:19.570 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:19 smithi146 bash[23809]: cluster 2024-03-11T21:10:18.321132+0000 mgr.y (mgr.24380) 722 : cluster 0 pgmap v815: 148 pgs: 148 active+clean; 3.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 812 B/s rd, 0 B/s wr, 906 op/s 2024-03-11T21:10:19.602 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:19.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:19.606 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:10:19.706 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:19.733 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:19.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:19.737 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:10:19.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:19 smithi023 bash[21372]: cluster 2024-03-11T21:10:18.321132+0000 mgr.y (mgr.24380) 722 : cluster 0 pgmap v815: 148 pgs: 148 active+clean; 3.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 812 B/s rd, 0 B/s wr, 906 op/s 2024-03-11T21:10:19.778 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:19 smithi023 bash[29146]: cluster 2024-03-11T21:10:18.321132+0000 mgr.y (mgr.24380) 722 : cluster 0 pgmap v815: 148 pgs: 148 active+clean; 3.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 812 B/s rd, 0 B/s wr, 906 op/s 2024-03-11T21:10:19.838 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:19.865 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:19.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:19.869 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:10:19.970 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:10:19.994 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:10:19.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:20.000 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:10:20.101 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:10:20.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:20.145 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:10:20.145 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:10:20.246 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:20.272 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:20.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:20.273 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:10:20.374 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:10:20.401 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:10:20.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:20.403 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:10:20.504 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:10:20.535 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:10:20.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:20.538 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:10:20.639 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:10:20.678 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:10:20.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:20.683 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:10:20.784 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:10:20.808 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:10:20.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:20.809 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:10:20.910 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:10:20.936 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:10:20.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:20.940 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:10:21.040 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:10:21.072 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:10:21.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:21.076 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:10:21.177 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:10:21.225 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:10:21.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:21.232 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:10:21.333 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:10:21.357 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:10:21.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:21.362 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:10:21.463 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:10:21.474 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:21 smithi023 bash[21372]: cluster 2024-03-11T21:10:20.321835+0000 mgr.y (mgr.24380) 723 : cluster 0 pgmap v816: 148 pgs: 148 active+clean; 2.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.06k op/s 2024-03-11T21:10:21.474 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:21 smithi023 bash[21372]: audit 2024-03-11T21:10:20.987039+0000 mon.a (mon.0) 1719 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:10:21.475 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:21 smithi023 bash[29146]: cluster 2024-03-11T21:10:20.321835+0000 mgr.y (mgr.24380) 723 : cluster 0 pgmap v816: 148 pgs: 148 active+clean; 2.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.06k op/s 2024-03-11T21:10:21.475 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:21 smithi023 bash[29146]: audit 2024-03-11T21:10:20.987039+0000 mon.a (mon.0) 1719 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:10:21.499 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:10:21.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:21.502 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:10:21.549 INFO:teuthology.orchestra.run.smithi023.stdout:osd.0: {} 2024-03-11T21:10:21.586 INFO:teuthology.orchestra.run.smithi023.stdout:osd.1: {} 2024-03-11T21:10:21.602 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:10:21.602 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:21.622 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:21 smithi146 bash[23809]: cluster 2024-03-11T21:10:20.321835+0000 mgr.y (mgr.24380) 723 : cluster 0 pgmap v816: 148 pgs: 148 active+clean; 2.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.06k op/s 2024-03-11T21:10:21.622 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:21 smithi146 bash[23809]: audit 2024-03-11T21:10:20.987039+0000 mon.a (mon.0) 1719 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:10:21.637 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:21.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:21.642 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:10:21.646 INFO:teuthology.orchestra.run.smithi023.stdout:osd.2: {} 2024-03-11T21:10:21.691 INFO:teuthology.orchestra.run.smithi023.stdout:osd.3: {} 2024-03-11T21:10:21.737 INFO:teuthology.orchestra.run.smithi023.stdout:osd.4: {} 2024-03-11T21:10:21.743 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:21.764 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:21.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:21.768 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:10:21.779 INFO:teuthology.orchestra.run.smithi023.stdout:osd.5: {} 2024-03-11T21:10:21.869 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:21.889 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:21.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:21.891 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:10:21.897 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-11T21:10:21.900 INFO:teuthology.orchestra.run.smithi023.stderr:dumped all 2024-03-11T21:10:21.921 INFO:teuthology.orchestra.run.smithi023.stdout:osd.6: {} 2024-03-11T21:10:21.959 INFO:teuthology.orchestra.run.smithi023.stdout:osd.7: {} 2024-03-11T21:10:21.960 INFO:teuthology.orchestra.run.smithi023.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:10:21.960 INFO:teuthology.orchestra.run.smithi023.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:10:21.960 INFO:teuthology.orchestra.run.smithi023.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:10:21.960 INFO:teuthology.orchestra.run.smithi023.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:10:21.960 INFO:teuthology.orchestra.run.smithi023.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:10:21.960 INFO:teuthology.orchestra.run.smithi023.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:10:21.960 INFO:teuthology.orchestra.run.smithi023.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:10:21.961 INFO:teuthology.orchestra.run.smithi023.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:10:21.992 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:10:22.020 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:10:22.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:22.023 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:10:22.123 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:22.148 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:22.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:22.153 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:10:22.254 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:10:22.282 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:10:22.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:22.285 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:10:22.386 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:10:22.412 INFO:teuthology.orchestra.run.smithi023.stderr:nodeep-scrub is set 2024-03-11T21:10:22.419 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:10:22.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:22.424 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:10:22.482 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:22 smithi023 bash[21372]: audit 2024-03-11T21:10:21.518028+0000 mon.a (mon.0) 1720 : audit 0 from='client.? 172.21.15.23:0/1145288766' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:10:22.482 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:22 smithi023 bash[21372]: audit 2024-03-11T21:10:21.891091+0000 mon.a (mon.0) 1721 : audit 1 from='client.? 172.21.15.23:0/1207459533' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:10:22.482 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:22 smithi023 bash[29146]: audit 2024-03-11T21:10:21.518028+0000 mon.a (mon.0) 1720 : audit 0 from='client.? 172.21.15.23:0/1145288766' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:10:22.482 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:22 smithi023 bash[29146]: audit 2024-03-11T21:10:21.891091+0000 mon.a (mon.0) 1721 : audit 1 from='client.? 172.21.15.23:0/1207459533' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:10:22.525 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:10:22.556 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:10:22.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:22.560 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:10:22.661 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:10:22.693 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:10:22.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:22.697 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:10:22.758 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:22 smithi146 bash[23809]: audit 2024-03-11T21:10:21.518028+0000 mon.a (mon.0) 1720 : audit 0 from='client.? 172.21.15.23:0/1145288766' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:10:22.758 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:22 smithi146 bash[23809]: audit 2024-03-11T21:10:21.891091+0000 mon.a (mon.0) 1721 : audit 1 from='client.? 172.21.15.23:0/1207459533' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:10:22.798 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:10:22.829 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:10:22.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:22.834 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:10:22.934 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:10:22.961 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:10:22.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:22.964 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:10:23.065 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:23.083 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:23.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:23.085 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:10:23.186 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:23.205 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:23.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:23.206 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:10:23.307 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:10:23.328 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:10:23.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:23.330 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:10:23.431 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:10:23.458 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:10:23.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:23.460 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:10:23.508 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:23 smithi146 bash[23809]: audit 2024-03-11T21:10:21.890935+0000 mgr.y (mgr.24380) 724 : audit 0 from='client.24919 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:10:23.508 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:23 smithi146 bash[23809]: cluster 2024-03-11T21:10:22.322588+0000 mgr.y (mgr.24380) 725 : cluster 0 pgmap v817: 148 pgs: 148 active+clean; 2.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 767 B/s rd, 0 B/s wr, 1.01k op/s 2024-03-11T21:10:23.508 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:23 smithi146 bash[23809]: cluster 2024-03-11T21:10:22.413783+0000 mon.a (mon.0) 1722 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:10:23.508 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:23 smithi146 bash[23809]: audit 2024-03-11T21:10:22.417284+0000 mon.a (mon.0) 1723 : audit 1 from='client.? 172.21.15.23:0/1207459533' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:10:23.508 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:23 smithi146 bash[23809]: cluster 2024-03-11T21:10:22.425304+0000 mon.a (mon.0) 1724 : cluster 0 osdmap e352: 8 total, 8 up, 8 in 2024-03-11T21:10:23.561 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:10:23.586 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:10:23.588 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:23 smithi023 bash[21372]: audit 2024-03-11T21:10:21.890935+0000 mgr.y (mgr.24380) 724 : audit 0 from='client.24919 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:10:23.588 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:23 smithi023 bash[21372]: cluster 2024-03-11T21:10:22.322588+0000 mgr.y (mgr.24380) 725 : cluster 0 pgmap v817: 148 pgs: 148 active+clean; 2.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 767 B/s rd, 0 B/s wr, 1.01k op/s 2024-03-11T21:10:23.588 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:23 smithi023 bash[21372]: cluster 2024-03-11T21:10:22.413783+0000 mon.a (mon.0) 1722 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:10:23.588 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:23 smithi023 bash[21372]: audit 2024-03-11T21:10:22.417284+0000 mon.a (mon.0) 1723 : audit 1 from='client.? 172.21.15.23:0/1207459533' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:10:23.588 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:23 smithi023 bash[21372]: cluster 2024-03-11T21:10:22.425304+0000 mon.a (mon.0) 1724 : cluster 0 osdmap e352: 8 total, 8 up, 8 in 2024-03-11T21:10:23.588 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:23 smithi023 bash[29146]: audit 2024-03-11T21:10:21.890935+0000 mgr.y (mgr.24380) 724 : audit 0 from='client.24919 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:10:23.589 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:23 smithi023 bash[29146]: cluster 2024-03-11T21:10:22.322588+0000 mgr.y (mgr.24380) 725 : cluster 0 pgmap v817: 148 pgs: 148 active+clean; 2.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 767 B/s rd, 0 B/s wr, 1.01k op/s 2024-03-11T21:10:23.589 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:23 smithi023 bash[29146]: cluster 2024-03-11T21:10:22.413783+0000 mon.a (mon.0) 1722 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:10:23.589 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:23 smithi023 bash[29146]: audit 2024-03-11T21:10:22.417284+0000 mon.a (mon.0) 1723 : audit 1 from='client.? 172.21.15.23:0/1207459533' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:10:23.589 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:23 smithi023 bash[29146]: cluster 2024-03-11T21:10:22.425304+0000 mon.a (mon.0) 1724 : cluster 0 osdmap e352: 8 total, 8 up, 8 in 2024-03-11T21:10:23.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:23.590 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:10:23.691 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:10:23.725 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:10:23.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:23.728 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:10:23.829 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:23.857 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:23.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:23.862 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:10:23.963 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:10:23.989 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:10:23.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:23.993 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:10:24.094 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:10:24.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:24.125 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:10:24.125 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:10:24.226 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:10:24.254 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:10:24.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:24.257 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:10:24.358 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:10:24.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:24.397 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:10:24.398 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:10:24.498 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:10:24.525 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:10:24.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:24.530 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:10:24.631 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:24.655 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:24.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:24.659 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:10:24.759 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:10:24.923 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:10:24.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:24.927 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:10:25.027 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:10:25.057 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:10:25.063 INFO:teuthology.orchestra.run.smithi023.stdout:{"pg_ready":true,"pg_map":{"version":816,"stamp":"2024-03-11T21:10:20.321295+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3003849152,"num_objects":46025,"num_object_clones":0,"num_object_copies":138075,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46025,"num_whiteouts":0,"num_read":2410,"num_read_kb":2256,"num_write":173094,"num_write_kb":6979038,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":190,"num_bytes_recovered":1775932,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91448,"ondisk_log_size":91448,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":12308424,"kb_used_data":8999368,"kb_used_omap":21,"kb_used_meta":3308522,"kb_avail":737652792,"statfs":{"total":767960285184,"available":755356459008,"internally_reserved":0,"allocated":9215352832,"data_stored":9212644984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22089,"internal_metadata":3387926967},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-835584000,"num_objects":-12750,"num_object_clones":0,"num_object_copies":-38250,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-12750,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":12750,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004955"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":610,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.995538+0000","last_change":"2024-03-11T21:07:57.433433+0000","last_active":"2024-03-11T21:10:06.995538+0000","last_peered":"2024-03-11T21:10:06.995538+0000","last_clean":"2024-03-11T21:10:06.995538+0000","last_became_active":"2024-03-11T20:50:34.338390+0000","last_became_peered":"2024-03-11T20:50:34.338390+0000","last_unstale":"2024-03-11T21:10:06.995538+0000","last_undegraded":"2024-03-11T21:10:06.995538+0000","last_fullsized":"2024-03-11T21:10:06.995538+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:07:57.433167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:07:57.433167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":614,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.994577+0000","last_change":"2024-03-11T21:08:09.344727+0000","last_active":"2024-03-11T21:10:06.994577+0000","last_peered":"2024-03-11T21:10:06.994577+0000","last_clean":"2024-03-11T21:10:06.994577+0000","last_became_active":"2024-03-11T20:50:32.210194+0000","last_became_peered":"2024-03-11T20:50:32.210194+0000","last_unstale":"2024-03-11T21:10:06.994577+0000","last_undegraded":"2024-03-11T21:10:06.994577+0000","last_fullsized":"2024-03-11T21:10:06.994577+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:08:09.344640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:08:09.344640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":325,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.989598+0000","last_change":"2024-03-11T21:08:13.139913+0000","last_active":"2024-03-11T21:10:06.989598+0000","last_peered":"2024-03-11T21:10:06.989598+0000","last_clean":"2024-03-11T21:10:06.989598+0000","last_became_active":"2024-03-11T21:06:28.760306+0000","last_became_peered":"2024-03-11T21:06:28.760306+0000","last_unstale":"2024-03-11T21:10:06.989598+0000","last_undegraded":"2024-03-11T21:10:06.989598+0000","last_fullsized":"2024-03-11T21:10:06.989598+0000","mapping_epoch":297,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:08:13.139773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:08:13.139773+0000","last_clean_scrub_stamp":"2024-03-11T21:08:13.139773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:08.412112+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"207'20","reported_seq":650,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.995892+0000","last_change":"2024-03-11T21:07:46.374511+0000","last_active":"2024-03-11T21:10:06.995892+0000","last_peered":"2024-03-11T21:10:06.995892+0000","last_clean":"2024-03-11T21:10:06.995892+0000","last_became_active":"2024-03-11T20:50:30.101721+0000","last_became_peered":"2024-03-11T20:50:30.101721+0000","last_unstale":"2024-03-11T21:10:06.995892+0000","last_undegraded":"2024-03-11T21:10:06.995892+0000","last_fullsized":"2024-03-11T21:10:06.995892+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'20","last_scrub_stamp":"2024-03-11T21:07:46.374362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:07:46.374362+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":450,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.988533+0000","last_change":"2024-03-11T21:09:41.738732+0000","last_active":"2024-03-11T21:10:06.988533+0000","last_peered":"2024-03-11T21:10:06.988533+0000","last_clean":"2024-03-11T21:10:06.988533+0000","last_became_active":"2024-03-11T21:05:17.316714+0000","last_became_peered":"2024-03-11T21:05:17.316714+0000","last_unstale":"2024-03-11T21:10:06.988533+0000","last_undegraded":"2024-03-11T21:10:06.988533+0000","last_fullsized":"2024-03-11T21:10:06.988533+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:41.738650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:55.545758+0000","last_clean_scrub_stamp":"2024-03-11T21:09:41.738650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:43.463899+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":455,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.989688+0000","last_change":"2024-03-11T21:09:53.477578+0000","last_active":"2024-03-11T21:10:06.989688+0000","last_peered":"2024-03-11T21:10:06.989688+0000","last_clean":"2024-03-11T21:10:06.989688+0000","last_became_active":"2024-03-11T21:05:17.319272+0000","last_became_peered":"2024-03-11T21:05:17.319272+0000","last_unstale":"2024-03-11T21:10:06.989688+0000","last_undegraded":"2024-03-11T21:10:06.989688+0000","last_fullsized":"2024-03-11T21:10:06.989688+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:53.477473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:54:13.660325+0000","last_clean_scrub_stamp":"2024-03-11T21:09:53.477473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:17.368384+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":623,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.995511+0000","last_change":"2024-03-11T21:09:49.933320+0000","last_active":"2024-03-11T21:10:06.995511+0000","last_peered":"2024-03-11T21:10:06.995511+0000","last_clean":"2024-03-11T21:10:06.995511+0000","last_became_active":"2024-03-11T20:50:28.058274+0000","last_became_peered":"2024-03-11T20:50:28.058274+0000","last_unstale":"2024-03-11T21:10:06.995511+0000","last_undegraded":"2024-03-11T21:10:06.995511+0000","last_fullsized":"2024-03-11T21:10:06.995511+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:49.933191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:09:49.933191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:11.026507+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"207'19","reported_seq":671,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.989550+0000","last_change":"2024-03-11T21:08:05.905831+0000","last_active":"2024-03-11T21:10:06.989550+0000","last_peered":"2024-03-11T21:10:06.989550+0000","last_clean":"2024-03-11T21:10:06.989550+0000","last_became_active":"2024-03-11T20:50:30.092638+0000","last_became_peered":"2024-03-11T20:50:30.092638+0000","last_unstale":"2024-03-11T21:10:06.989550+0000","last_undegraded":"2024-03-11T21:10:06.989550+0000","last_fullsized":"2024-03-11T21:10:06.989550+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'19","last_scrub_stamp":"2024-03-11T21:08:05.905696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:08:05.905696+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:08.412248+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":639,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.986345+0000","last_change":"2024-03-11T21:09:47.906047+0000","last_active":"2024-03-11T21:10:06.986345+0000","last_peered":"2024-03-11T21:10:06.986345+0000","last_clean":"2024-03-11T21:10:06.986345+0000","last_became_active":"2024-03-11T21:00:48.460694+0000","last_became_peered":"2024-03-11T21:00:48.460694+0000","last_unstale":"2024-03-11T21:10:06.986345+0000","last_undegraded":"2024-03-11T21:10:06.986345+0000","last_fullsized":"2024-03-11T21:10:06.986345+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:47.905770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:53:07.792697+0000","last_clean_scrub_stamp":"2024-03-11T21:09:47.905770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:54.823926+0000","scrub_duration":83,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":439,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.995770+0000","last_change":"2024-03-11T21:09:29.891712+0000","last_active":"2024-03-11T21:10:06.995770+0000","last_peered":"2024-03-11T21:10:06.995770+0000","last_clean":"2024-03-11T21:10:06.995770+0000","last_became_active":"2024-03-11T21:05:17.316491+0000","last_became_peered":"2024-03-11T21:05:17.316491+0000","last_unstale":"2024-03-11T21:10:06.995770+0000","last_undegraded":"2024-03-11T21:10:06.995770+0000","last_fullsized":"2024-03-11T21:10:06.995770+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:29.891601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:03:36.230647+0000","last_clean_scrub_stamp":"2024-03-11T21:09:29.891601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:40.655256+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":643,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.986431+0000","last_change":"2024-03-11T21:09:52.421945+0000","last_active":"2024-03-11T21:10:06.986431+0000","last_peered":"2024-03-11T21:10:06.986431+0000","last_clean":"2024-03-11T21:10:06.986431+0000","last_became_active":"2024-03-11T20:50:28.079225+0000","last_became_peered":"2024-03-11T20:50:28.079225+0000","last_unstale":"2024-03-11T21:10:06.986431+0000","last_undegraded":"2024-03-11T21:10:06.986431+0000","last_fullsized":"2024-03-11T21:10:06.986431+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:52.421824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:09:52.421824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:12.646223+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"207'27","reported_seq":741,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.985904+0000","last_change":"2024-03-11T21:09:31.716392+0000","last_active":"2024-03-11T21:10:06.985904+0000","last_peered":"2024-03-11T21:10:06.985904+0000","last_clean":"2024-03-11T21:10:06.985904+0000","last_became_active":"2024-03-11T21:00:48.462943+0000","last_became_peered":"2024-03-11T21:00:48.462943+0000","last_unstale":"2024-03-11T21:10:06.985904+0000","last_undegraded":"2024-03-11T21:10:06.985904+0000","last_fullsized":"2024-03-11T21:10:06.985904+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'27","last_scrub_stamp":"2024-03-11T21:09:31.716290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:09:31.716290+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:01.463544+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":326,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.987518+0000","last_change":"2024-03-11T21:09:47.379851+0000","last_active":"2024-03-11T21:10:06.987518+0000","last_peered":"2024-03-11T21:10:06.987518+0000","last_clean":"2024-03-11T21:10:06.987518+0000","last_became_active":"2024-03-11T21:06:28.757201+0000","last_became_peered":"2024-03-11T21:06:28.757201+0000","last_unstale":"2024-03-11T21:10:06.987518+0000","last_undegraded":"2024-03-11T21:10:06.987518+0000","last_fullsized":"2024-03-11T21:10:06.987518+0000","mapping_epoch":297,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:47.379734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:09:47.379734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:54.404545+0000","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":401,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.994675+0000","last_change":"2024-03-11T21:08:01.360006+0000","last_active":"2024-03-11T21:10:06.994675+0000","last_peered":"2024-03-11T21:10:06.994675+0000","last_clean":"2024-03-11T21:10:06.994675+0000","last_became_active":"2024-03-11T21:05:17.313388+0000","last_became_peered":"2024-03-11T21:05:17.313388+0000","last_unstale":"2024-03-11T21:10:06.994675+0000","last_undegraded":"2024-03-11T21:10:06.994675+0000","last_fullsized":"2024-03-11T21:10:06.994675+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:08:01.359878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:53:50.599029+0000","last_clean_scrub_stamp":"2024-03-11T21:08:01.359878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":651,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.986636+0000","last_change":"2024-03-11T21:09:57.535072+0000","last_active":"2024-03-11T21:10:06.986636+0000","last_peered":"2024-03-11T21:10:06.986636+0000","last_clean":"2024-03-11T21:10:06.986636+0000","last_became_active":"2024-03-11T20:50:28.068978+0000","last_became_peered":"2024-03-11T20:50:28.068978+0000","last_unstale":"2024-03-11T21:10:06.986636+0000","last_undegraded":"2024-03-11T21:10:06.986636+0000","last_fullsized":"2024-03-11T21:10:06.986636+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:57.534967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:54:07.155130+0000","last_clean_scrub_stamp":"2024-03-11T21:09:57.534967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:10.993525+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"207'23","reported_seq":671,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.986148+0000","last_change":"2024-03-11T21:08:23.793392+0000","last_active":"2024-03-11T21:10:06.986148+0000","last_peered":"2024-03-11T21:10:06.986148+0000","last_clean":"2024-03-11T21:10:06.986148+0000","last_became_active":"2024-03-11T21:00:27.952093+0000","last_became_peered":"2024-03-11T21:00:27.952093+0000","last_unstale":"2024-03-11T21:10:06.986148+0000","last_undegraded":"2024-03-11T21:10:06.986148+0000","last_fullsized":"2024-03-11T21:10:06.986148+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'23","last_scrub_stamp":"2024-03-11T21:08:23.793245+0000","last_deep_scrub":"110'17","last_deep_scrub_stamp":"2024-03-11T20:53:04.802488+0000","last_clean_scrub_stamp":"2024-03-11T21:08:23.793245+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:09.668546+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":611,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.995866+0000","last_change":"2024-03-11T21:08:18.541547+0000","last_active":"2024-03-11T21:10:06.995866+0000","last_peered":"2024-03-11T21:10:06.995866+0000","last_clean":"2024-03-11T21:10:06.995866+0000","last_became_active":"2024-03-11T20:50:34.334558+0000","last_became_peered":"2024-03-11T20:50:34.334558+0000","last_unstale":"2024-03-11T21:10:06.995866+0000","last_undegraded":"2024-03-11T21:10:06.995866+0000","last_fullsized":"2024-03-11T21:10:06.995866+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:08:18.541431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:55:59.892394+0000","last_clean_scrub_stamp":"2024-03-11T21:08:18.541431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":163,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":469,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.985578+0000","last_change":"2024-03-11T21:09:35.734411+0000","last_active":"2024-03-11T21:10:06.985578+0000","last_peered":"2024-03-11T21:10:06.985578+0000","last_clean":"2024-03-11T21:10:06.985578+0000","last_became_active":"2024-03-11T21:00:48.461013+0000","last_became_peered":"2024-03-11T21:00:48.461013+0000","last_unstale":"2024-03-11T21:10:06.985578+0000","last_undegraded":"2024-03-11T21:10:06.985578+0000","last_fullsized":"2024-03-11T21:10:06.985578+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:35.734334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:09:35.734334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:59.870287+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"77'1","reported_seq":629,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.995940+0000","last_change":"2024-03-11T21:07:38.421921+0000","last_active":"2024-03-11T21:10:06.995940+0000","last_peered":"2024-03-11T21:10:06.995940+0000","last_clean":"2024-03-11T21:10:06.995940+0000","last_became_active":"2024-03-11T20:50:28.068840+0000","last_became_peered":"2024-03-11T20:50:28.068840+0000","last_unstale":"2024-03-11T21:10:06.995940+0000","last_undegraded":"2024-03-11T21:10:06.995940+0000","last_fullsized":"2024-03-11T21:10:06.995940+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-03-11T21:07:38.421793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:07:38.421793+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"207'19","reported_seq":489,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.986079+0000","last_change":"2024-03-11T21:09:43.241588+0000","last_active":"2024-03-11T21:10:06.986079+0000","last_peered":"2024-03-11T21:10:06.986079+0000","last_clean":"2024-03-11T21:10:06.986079+0000","last_became_active":"2024-03-11T21:05:17.443030+0000","last_became_peered":"2024-03-11T21:05:17.443030+0000","last_unstale":"2024-03-11T21:10:06.986079+0000","last_undegraded":"2024-03-11T21:10:06.986079+0000","last_fullsized":"2024-03-11T21:10:06.986079+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'19","last_scrub_stamp":"2024-03-11T21:09:43.241403+0000","last_deep_scrub":"207'19","last_deep_scrub_stamp":"2024-03-11T21:03:39.240089+0000","last_clean_scrub_stamp":"2024-03-11T21:09:43.241403+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:01.424382+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":411,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.990126+0000","last_change":"2024-03-11T21:09:51.529563+0000","last_active":"2024-03-11T21:10:06.990126+0000","last_peered":"2024-03-11T21:10:06.990126+0000","last_clean":"2024-03-11T21:10:06.990126+0000","last_became_active":"2024-03-11T21:05:17.313986+0000","last_became_peered":"2024-03-11T21:05:17.313986+0000","last_unstale":"2024-03-11T21:10:06.990126+0000","last_undegraded":"2024-03-11T21:10:06.990126+0000","last_fullsized":"2024-03-11T21:10:06.990126+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:51.529432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:33.399510+0000","last_clean_scrub_stamp":"2024-03-11T21:09:51.529432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:21.400657+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":421,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.992443+0000","last_change":"2024-03-11T21:09:39.217669+0000","last_active":"2024-03-11T21:10:06.992443+0000","last_peered":"2024-03-11T21:10:06.992443+0000","last_clean":"2024-03-11T21:10:06.992443+0000","last_became_active":"2024-03-11T21:08:05.298552+0000","last_became_peered":"2024-03-11T21:08:05.298552+0000","last_unstale":"2024-03-11T21:10:06.992443+0000","last_undegraded":"2024-03-11T21:10:06.992443+0000","last_fullsized":"2024-03-11T21:10:06.992443+0000","mapping_epoch":319,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:39.217523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:24.666325+0000","last_clean_scrub_stamp":"2024-03-11T21:09:39.217523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:45.381255+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":652,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.990078+0000","last_change":"2024-03-11T21:09:44.522913+0000","last_active":"2024-03-11T21:10:06.990078+0000","last_peered":"2024-03-11T21:10:06.990078+0000","last_clean":"2024-03-11T21:10:06.990078+0000","last_became_active":"2024-03-11T21:00:27.956935+0000","last_became_peered":"2024-03-11T21:00:27.956935+0000","last_unstale":"2024-03-11T21:10:06.990078+0000","last_undegraded":"2024-03-11T21:10:06.990078+0000","last_fullsized":"2024-03-11T21:10:06.990078+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:44.522800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:09:44.522800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:08.467519+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"207'7","reported_seq":652,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.988597+0000","last_change":"2024-03-11T21:08:00.885293+0000","last_active":"2024-03-11T21:10:06.988597+0000","last_peered":"2024-03-11T21:10:06.988597+0000","last_clean":"2024-03-11T21:10:06.988597+0000","last_became_active":"2024-03-11T21:00:27.955296+0000","last_became_peered":"2024-03-11T21:00:27.955296+0000","last_unstale":"2024-03-11T21:10:06.988597+0000","last_undegraded":"2024-03-11T21:10:06.988597+0000","last_fullsized":"2024-03-11T21:10:06.988597+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'7","last_scrub_stamp":"2024-03-11T21:08:00.885119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:08:00.885119+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:08.411912+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":345,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.992485+0000","last_change":"2024-03-11T21:09:57.285484+0000","last_active":"2024-03-11T21:10:06.992485+0000","last_peered":"2024-03-11T21:10:06.992485+0000","last_clean":"2024-03-11T21:10:06.992485+0000","last_became_active":"2024-03-11T21:00:48.464694+0000","last_became_peered":"2024-03-11T21:00:48.464694+0000","last_unstale":"2024-03-11T21:10:06.992485+0000","last_undegraded":"2024-03-11T21:10:06.992485+0000","last_fullsized":"2024-03-11T21:10:06.992485+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:57.285360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:09:57.285360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:07.169016+0000","scrub_duration":119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1f","version":"212'16","reported_seq":918,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:14.077108+0000","last_change":"2024-03-11T21:09:47.536775+0000","last_active":"2024-03-11T21:10:14.077108+0000","last_peered":"2024-03-11T21:10:14.077108+0000","last_clean":"2024-03-11T21:10:14.077108+0000","last_became_active":"2024-03-11T21:00:48.464350+0000","last_became_peered":"2024-03-11T21:00:48.464350+0000","last_unstale":"2024-03-11T21:10:14.077108+0000","last_undegraded":"2024-03-11T21:10:14.077108+0000","last_fullsized":"2024-03-11T21:10:14.077108+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"212'16","last_scrub_stamp":"2024-03-11T21:09:47.536622+0000","last_deep_scrub":"212'16","last_deep_scrub_stamp":"2024-03-11T21:06:47.821221+0000","last_clean_scrub_stamp":"2024-03-11T21:09:47.536622+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:07.472659+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"77'1","reported_seq":615,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.994660+0000","last_change":"2024-03-11T21:07:37.448905+0000","last_active":"2024-03-11T21:10:06.994660+0000","last_peered":"2024-03-11T21:10:06.994660+0000","last_clean":"2024-03-11T21:10:06.994660+0000","last_became_active":"2024-03-11T20:50:28.068827+0000","last_became_peered":"2024-03-11T20:50:28.068827+0000","last_unstale":"2024-03-11T21:10:06.994660+0000","last_undegraded":"2024-03-11T21:10:06.994660+0000","last_fullsized":"2024-03-11T21:10:06.994660+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-03-11T21:07:37.448705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:07:37.448705+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"207'15","reported_seq":685,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.993790+0000","last_change":"2024-03-11T21:08:21.296049+0000","last_active":"2024-03-11T21:10:06.993790+0000","last_peered":"2024-03-11T21:10:06.993790+0000","last_clean":"2024-03-11T21:10:06.993790+0000","last_became_active":"2024-03-11T21:00:48.467513+0000","last_became_peered":"2024-03-11T21:00:48.467513+0000","last_unstale":"2024-03-11T21:10:06.993790+0000","last_undegraded":"2024-03-11T21:10:06.993790+0000","last_fullsized":"2024-03-11T21:10:06.993790+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'15","last_scrub_stamp":"2024-03-11T21:08:21.295914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:08:21.295914+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"105'2","reported_seq":634,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.987067+0000","last_change":"2024-03-11T21:09:35.469722+0000","last_active":"2024-03-11T21:10:06.987067+0000","last_peered":"2024-03-11T21:10:06.987067+0000","last_clean":"2024-03-11T21:10:06.987067+0000","last_became_active":"2024-03-11T20:50:34.327356+0000","last_became_peered":"2024-03-11T20:50:34.327356+0000","last_unstale":"2024-03-11T21:10:06.987067+0000","last_undegraded":"2024-03-11T21:10:06.987067+0000","last_fullsized":"2024-03-11T21:10:06.987067+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'2","last_scrub_stamp":"2024-03-11T21:09:35.469576+0000","last_deep_scrub":"105'2","last_deep_scrub_stamp":"2024-03-11T21:03:41.971918+0000","last_clean_scrub_stamp":"2024-03-11T21:09:35.469576+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:41.413387+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":333,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.999407+0000","last_change":"2024-03-11T21:09:54.403649+0000","last_active":"2024-03-11T21:10:06.999407+0000","last_peered":"2024-03-11T21:10:06.999407+0000","last_clean":"2024-03-11T21:10:06.999407+0000","last_became_active":"2024-03-11T21:06:28.759784+0000","last_became_peered":"2024-03-11T21:06:28.759784+0000","last_unstale":"2024-03-11T21:10:06.999407+0000","last_undegraded":"2024-03-11T21:10:06.999407+0000","last_fullsized":"2024-03-11T21:10:06.999407+0000","mapping_epoch":297,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:54.403136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:06:32.968650+0000","last_clean_scrub_stamp":"2024-03-11T21:09:54.403136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:21.584616+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":647,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.985565+0000","last_change":"2024-03-11T21:10:01.677396+0000","last_active":"2024-03-11T21:10:06.985565+0000","last_peered":"2024-03-11T21:10:06.985565+0000","last_clean":"2024-03-11T21:10:06.985565+0000","last_became_active":"2024-03-11T21:00:27.947614+0000","last_became_peered":"2024-03-11T21:00:27.947614+0000","last_unstale":"2024-03-11T21:10:06.985565+0000","last_undegraded":"2024-03-11T21:10:06.985565+0000","last_fullsized":"2024-03-11T21:10:06.985565+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:10:01.677279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:08:27.756036+0000","last_clean_scrub_stamp":"2024-03-11T21:10:01.677279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:21.767300+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"207'27","reported_seq":374,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.991708+0000","last_change":"2024-03-11T21:08:19.539727+0000","last_active":"2024-03-11T21:10:06.991708+0000","last_peered":"2024-03-11T21:10:06.991708+0000","last_clean":"2024-03-11T21:10:06.991708+0000","last_became_active":"2024-03-11T21:00:48.457893+0000","last_became_peered":"2024-03-11T21:00:48.457893+0000","last_unstale":"2024-03-11T21:10:06.991708+0000","last_undegraded":"2024-03-11T21:10:06.991708+0000","last_fullsized":"2024-03-11T21:10:06.991708+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'27","last_scrub_stamp":"2024-03-11T21:08:19.539512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:08:19.539512+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:09.279353+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":610,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.996144+0000","last_change":"2024-03-11T21:07:41.485168+0000","last_active":"2024-03-11T21:10:06.996144+0000","last_peered":"2024-03-11T21:10:06.996144+0000","last_clean":"2024-03-11T21:10:06.996144+0000","last_became_active":"2024-03-11T20:50:34.337878+0000","last_became_peered":"2024-03-11T20:50:34.337878+0000","last_unstale":"2024-03-11T21:10:06.996144+0000","last_undegraded":"2024-03-11T21:10:06.996144+0000","last_fullsized":"2024-03-11T21:10:06.996144+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:07:41.485049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:52:11.462480+0000","last_clean_scrub_stamp":"2024-03-11T21:07:41.485049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":123,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":622,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.995063+0000","last_change":"2024-03-11T21:10:02.099820+0000","last_active":"2024-03-11T21:10:06.995063+0000","last_peered":"2024-03-11T21:10:06.995063+0000","last_clean":"2024-03-11T21:10:06.995063+0000","last_became_active":"2024-03-11T20:50:32.646323+0000","last_became_peered":"2024-03-11T20:50:32.646323+0000","last_unstale":"2024-03-11T21:10:06.995063+0000","last_undegraded":"2024-03-11T21:10:06.995063+0000","last_fullsized":"2024-03-11T21:10:06.995063+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:10:02.099653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:10:02.099653+0000","last_clean_scrub_stamp":"2024-03-11T21:10:02.099653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:32.017072+0000","scrub_duration":179,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":660,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.990470+0000","last_change":"2024-03-11T21:09:49.510590+0000","last_active":"2024-03-11T21:10:06.990470+0000","last_peered":"2024-03-11T21:10:06.990470+0000","last_clean":"2024-03-11T21:10:06.990470+0000","last_became_active":"2024-03-11T20:50:28.086513+0000","last_became_peered":"2024-03-11T20:50:28.086513+0000","last_unstale":"2024-03-11T21:10:06.990470+0000","last_undegraded":"2024-03-11T21:10:06.990470+0000","last_fullsized":"2024-03-11T21:10:06.990470+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:49.510484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:09:49.510484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:59.053464+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"207'19","reported_seq":705,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.986267+0000","last_change":"2024-03-11T21:09:49.809926+0000","last_active":"2024-03-11T21:10:06.986267+0000","last_peered":"2024-03-11T21:10:06.986267+0000","last_clean":"2024-03-11T21:10:06.986267+0000","last_became_active":"2024-03-11T21:00:48.464707+0000","last_became_peered":"2024-03-11T21:00:48.464707+0000","last_unstale":"2024-03-11T21:10:06.986267+0000","last_undegraded":"2024-03-11T21:10:06.986267+0000","last_fullsized":"2024-03-11T21:10:06.986267+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'19","last_scrub_stamp":"2024-03-11T21:09:49.809679+0000","last_deep_scrub":"145'13","last_deep_scrub_stamp":"2024-03-11T20:55:53.560287+0000","last_clean_scrub_stamp":"2024-03-11T21:09:49.809679+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:02.002638+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":633,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.986169+0000","last_change":"2024-03-11T21:09:25.313083+0000","last_active":"2024-03-11T21:10:06.986169+0000","last_peered":"2024-03-11T21:10:06.986169+0000","last_clean":"2024-03-11T21:10:06.986169+0000","last_became_active":"2024-03-11T21:00:27.973405+0000","last_became_peered":"2024-03-11T21:00:27.973405+0000","last_unstale":"2024-03-11T21:10:06.986169+0000","last_undegraded":"2024-03-11T21:10:06.986169+0000","last_fullsized":"2024-03-11T21:10:06.986169+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:25.312971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:09:25.312971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:52.239778+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":627,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.994059+0000","last_change":"2024-03-11T21:08:24.323683+0000","last_active":"2024-03-11T21:10:06.994059+0000","last_peered":"2024-03-11T21:10:06.994059+0000","last_clean":"2024-03-11T21:10:06.994059+0000","last_became_active":"2024-03-11T21:00:48.472812+0000","last_became_peered":"2024-03-11T21:00:48.472812+0000","last_unstale":"2024-03-11T21:10:06.994059+0000","last_undegraded":"2024-03-11T21:10:06.994059+0000","last_fullsized":"2024-03-11T21:10:06.994059+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:08:24.323601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:08:24.323601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":646,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.986175+0000","last_change":"2024-03-11T21:09:37.712819+0000","last_active":"2024-03-11T21:10:06.986175+0000","last_peered":"2024-03-11T21:10:06.986175+0000","last_clean":"2024-03-11T21:10:06.986175+0000","last_became_active":"2024-03-11T20:50:28.075760+0000","last_became_peered":"2024-03-11T20:50:28.075760+0000","last_unstale":"2024-03-11T21:10:06.986175+0000","last_undegraded":"2024-03-11T21:10:06.986175+0000","last_fullsized":"2024-03-11T21:10:06.986175+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:37.712659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:58:55.977289+0000","last_clean_scrub_stamp":"2024-03-11T21:09:37.712659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:06.735736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"207'12","reported_seq":293,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.989499+0000","last_change":"2024-03-11T21:07:57.887460+0000","last_active":"2024-03-11T21:10:06.989499+0000","last_peered":"2024-03-11T21:10:06.989499+0000","last_clean":"2024-03-11T21:10:06.989499+0000","last_became_active":"2024-03-11T21:06:28.763971+0000","last_became_peered":"2024-03-11T21:06:28.763971+0000","last_unstale":"2024-03-11T21:10:06.989499+0000","last_undegraded":"2024-03-11T21:10:06.989499+0000","last_fullsized":"2024-03-11T21:10:06.989499+0000","mapping_epoch":297,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'12","last_scrub_stamp":"2024-03-11T21:07:57.887318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:07:57.887318+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:08.411943+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":16,"num_read_kb":13,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.11","version":"186'2","reported_seq":612,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.993788+0000","last_change":"2024-03-11T21:08:00.343845+0000","last_active":"2024-03-11T21:10:06.993788+0000","last_peered":"2024-03-11T21:10:06.993788+0000","last_clean":"2024-03-11T21:10:06.993788+0000","last_became_active":"2024-03-11T20:50:34.339326+0000","last_became_peered":"2024-03-11T20:50:34.339326+0000","last_unstale":"2024-03-11T21:10:06.993788+0000","last_undegraded":"2024-03-11T21:10:06.993788+0000","last_fullsized":"2024-03-11T21:10:06.993788+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"186'2","last_scrub_stamp":"2024-03-11T21:08:00.343694+0000","last_deep_scrub":"186'2","last_deep_scrub_stamp":"2024-03-11T20:59:04.534850+0000","last_clean_scrub_stamp":"2024-03-11T21:08:00.343694+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":455,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.989929+0000","last_change":"2024-03-11T21:09:32.541123+0000","last_active":"2024-03-11T21:10:06.989929+0000","last_peered":"2024-03-11T21:10:06.989929+0000","last_clean":"2024-03-11T21:10:06.989929+0000","last_became_active":"2024-03-11T21:05:17.314174+0000","last_became_peered":"2024-03-11T21:05:17.314174+0000","last_unstale":"2024-03-11T21:10:06.989929+0000","last_undegraded":"2024-03-11T21:10:06.989929+0000","last_fullsized":"2024-03-11T21:10:06.989929+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:32.540834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:09:32.540834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:45.351255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":344,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.992319+0000","last_change":"2024-03-11T21:10:01.220880+0000","last_active":"2024-03-11T21:10:06.992319+0000","last_peered":"2024-03-11T21:10:06.992319+0000","last_clean":"2024-03-11T21:10:06.992319+0000","last_became_active":"2024-03-11T21:00:48.455510+0000","last_became_peered":"2024-03-11T21:00:48.455510+0000","last_unstale":"2024-03-11T21:10:06.992319+0000","last_undegraded":"2024-03-11T21:10:06.992319+0000","last_fullsized":"2024-03-11T21:10:06.992319+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:10:01.220751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:10:01.220751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:22.534857+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"207'25","reported_seq":425,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.986913+0000","last_change":"2024-03-11T21:09:32.384059+0000","last_active":"2024-03-11T21:10:06.986913+0000","last_peered":"2024-03-11T21:10:06.986913+0000","last_clean":"2024-03-11T21:10:06.986913+0000","last_became_active":"2024-03-11T21:05:17.314507+0000","last_became_peered":"2024-03-11T21:05:17.314507+0000","last_unstale":"2024-03-11T21:10:06.986913+0000","last_undegraded":"2024-03-11T21:10:06.986913+0000","last_fullsized":"2024-03-11T21:10:06.986913+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'25","last_scrub_stamp":"2024-03-11T21:09:32.383962+0000","last_deep_scrub":"161'15","last_deep_scrub_stamp":"2024-03-11T20:57:15.125548+0000","last_clean_scrub_stamp":"2024-03-11T21:09:32.383962+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:00.885090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":652,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.988025+0000","last_change":"2024-03-11T21:10:02.497069+0000","last_active":"2024-03-11T21:10:06.988025+0000","last_peered":"2024-03-11T21:10:06.988025+0000","last_clean":"2024-03-11T21:10:06.988025+0000","last_became_active":"2024-03-11T21:00:48.462105+0000","last_became_peered":"2024-03-11T21:00:48.462105+0000","last_unstale":"2024-03-11T21:10:06.988025+0000","last_undegraded":"2024-03-11T21:10:06.988025+0000","last_fullsized":"2024-03-11T21:10:06.988025+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:10:02.496964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:10:02.496964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:31.206754+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":345,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.985724+0000","last_change":"2024-03-11T21:10:03.274386+0000","last_active":"2024-03-11T21:10:06.985724+0000","last_peered":"2024-03-11T21:10:06.985724+0000","last_clean":"2024-03-11T21:10:06.985724+0000","last_became_active":"2024-03-11T21:00:48.459043+0000","last_became_peered":"2024-03-11T21:00:48.459043+0000","last_unstale":"2024-03-11T21:10:06.985724+0000","last_undegraded":"2024-03-11T21:10:06.985724+0000","last_fullsized":"2024-03-11T21:10:06.985724+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:10:03.274266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:23.137832+0000","last_clean_scrub_stamp":"2024-03-11T21:10:03.274266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:07.308003+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":652,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.989922+0000","last_change":"2024-03-11T21:09:27.653392+0000","last_active":"2024-03-11T21:10:06.989922+0000","last_peered":"2024-03-11T21:10:06.989922+0000","last_clean":"2024-03-11T21:10:06.989922+0000","last_became_active":"2024-03-11T20:50:28.082737+0000","last_became_peered":"2024-03-11T20:50:28.082737+0000","last_unstale":"2024-03-11T21:10:06.989922+0000","last_undegraded":"2024-03-11T21:10:06.989922+0000","last_fullsized":"2024-03-11T21:10:06.989922+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:27.653224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:28.404997+0000","last_clean_scrub_stamp":"2024-03-11T21:09:27.653224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:36.930458+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"207'15","reported_seq":666,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.986533+0000","last_change":"2024-03-11T21:09:28.306131+0000","last_active":"2024-03-11T21:10:06.986533+0000","last_peered":"2024-03-11T21:10:06.986533+0000","last_clean":"2024-03-11T21:10:06.986533+0000","last_became_active":"2024-03-11T21:00:27.967624+0000","last_became_peered":"2024-03-11T21:00:27.967624+0000","last_unstale":"2024-03-11T21:10:06.986533+0000","last_undegraded":"2024-03-11T21:10:06.986533+0000","last_fullsized":"2024-03-11T21:10:06.986533+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'15","last_scrub_stamp":"2024-03-11T21:09:28.306002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:09:28.306002+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:48.387782+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":288,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.985920+0000","last_change":"2024-03-11T21:09:21.916431+0000","last_active":"2024-03-11T21:10:06.985920+0000","last_peered":"2024-03-11T21:10:06.985920+0000","last_clean":"2024-03-11T21:10:06.985920+0000","last_became_active":"2024-03-11T21:08:05.299913+0000","last_became_peered":"2024-03-11T21:08:05.299913+0000","last_unstale":"2024-03-11T21:10:06.985920+0000","last_undegraded":"2024-03-11T21:10:06.985920+0000","last_fullsized":"2024-03-11T21:10:06.985920+0000","mapping_epoch":319,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:21.916317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:57:16.032302+0000","last_clean_scrub_stamp":"2024-03-11T21:09:21.916317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:31.402897+0000","scrub_duration":167,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"112'12","reported_seq":879,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:09.068187+0000","last_change":"2024-03-11T21:09:40.700296+0000","last_active":"2024-03-11T21:10:09.068187+0000","last_peered":"2024-03-11T21:10:09.068187+0000","last_clean":"2024-03-11T21:10:09.068187+0000","last_became_active":"2024-03-11T20:50:32.650064+0000","last_became_peered":"2024-03-11T20:50:32.650064+0000","last_unstale":"2024-03-11T21:10:09.068187+0000","last_undegraded":"2024-03-11T21:10:09.068187+0000","last_fullsized":"2024-03-11T21:10:09.068187+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'12","last_scrub_stamp":"2024-03-11T21:09:40.700213+0000","last_deep_scrub":"112'12","last_deep_scrub_stamp":"2024-03-11T20:53:11.843951+0000","last_clean_scrub_stamp":"2024-03-11T21:09:40.700213+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:45.674977+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":340,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.988494+0000","last_change":"2024-03-11T21:09:24.330593+0000","last_active":"2024-03-11T21:10:06.988494+0000","last_peered":"2024-03-11T21:10:06.988494+0000","last_clean":"2024-03-11T21:10:06.988494+0000","last_became_active":"2024-03-11T21:06:28.756041+0000","last_became_peered":"2024-03-11T21:06:28.756041+0000","last_unstale":"2024-03-11T21:10:06.988494+0000","last_undegraded":"2024-03-11T21:10:06.988494+0000","last_fullsized":"2024-03-11T21:10:06.988494+0000","mapping_epoch":297,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:24.330325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:06:52.862971+0000","last_clean_scrub_stamp":"2024-03-11T21:09:24.330325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:41.008348+0000","scrub_duration":247,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"207'19","reported_seq":391,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.996328+0000","last_change":"2024-03-11T21:08:21.328028+0000","last_active":"2024-03-11T21:10:06.996328+0000","last_peered":"2024-03-11T21:10:06.996328+0000","last_clean":"2024-03-11T21:10:06.996328+0000","last_became_active":"2024-03-11T21:06:28.758223+0000","last_became_peered":"2024-03-11T21:06:28.758223+0000","last_unstale":"2024-03-11T21:10:06.996328+0000","last_undegraded":"2024-03-11T21:10:06.996328+0000","last_fullsized":"2024-03-11T21:10:06.996328+0000","mapping_epoch":297,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'19","last_scrub_stamp":"2024-03-11T21:08:21.327572+0000","last_deep_scrub":"119'13","last_deep_scrub_stamp":"2024-03-11T20:54:15.443472+0000","last_clean_scrub_stamp":"2024-03-11T21:08:21.327572+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:08.412335+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":320,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.999643+0000","last_change":"2024-03-11T21:10:01.497745+0000","last_active":"2024-03-11T21:10:06.999643+0000","last_peered":"2024-03-11T21:10:06.999643+0000","last_clean":"2024-03-11T21:10:06.999643+0000","last_became_active":"2024-03-11T21:06:28.758634+0000","last_became_peered":"2024-03-11T21:06:28.758634+0000","last_unstale":"2024-03-11T21:10:06.999643+0000","last_undegraded":"2024-03-11T21:10:06.999643+0000","last_fullsized":"2024-03-11T21:10:06.999643+0000","mapping_epoch":297,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:10:01.497631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:10:01.497631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:04.566028+0000","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"212'17","reported_seq":895,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:14.076272+0000","last_change":"2024-03-11T21:08:20.327834+0000","last_active":"2024-03-11T21:10:14.076272+0000","last_peered":"2024-03-11T21:10:14.076272+0000","last_clean":"2024-03-11T21:10:14.076272+0000","last_became_active":"2024-03-11T21:00:48.467116+0000","last_became_peered":"2024-03-11T21:00:48.467116+0000","last_unstale":"2024-03-11T21:10:14.076272+0000","last_undegraded":"2024-03-11T21:10:14.076272+0000","last_fullsized":"2024-03-11T21:10:14.076272+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"212'17","last_scrub_stamp":"2024-03-11T21:08:20.327696+0000","last_deep_scrub":"110'13","last_deep_scrub_stamp":"2024-03-11T20:53:00.434532+0000","last_clean_scrub_stamp":"2024-03-11T21:08:20.327696+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":646,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.988647+0000","last_change":"2024-03-11T21:09:34.741973+0000","last_active":"2024-03-11T21:10:06.988647+0000","last_peered":"2024-03-11T21:10:06.988647+0000","last_clean":"2024-03-11T21:10:06.988647+0000","last_became_active":"2024-03-11T20:50:28.070325+0000","last_became_peered":"2024-03-11T20:50:28.070325+0000","last_unstale":"2024-03-11T21:10:06.988647+0000","last_undegraded":"2024-03-11T21:10:06.988647+0000","last_fullsized":"2024-03-11T21:10:06.988647+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:34.741850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:09:34.741850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:44.232955+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"207'24","reported_seq":677,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.986504+0000","last_change":"2024-03-11T21:09:21.307369+0000","last_active":"2024-03-11T21:10:06.986504+0000","last_peered":"2024-03-11T21:10:06.986504+0000","last_clean":"2024-03-11T21:10:06.986504+0000","last_became_active":"2024-03-11T21:00:27.967078+0000","last_became_peered":"2024-03-11T21:00:27.967078+0000","last_unstale":"2024-03-11T21:10:06.986504+0000","last_undegraded":"2024-03-11T21:10:06.986504+0000","last_fullsized":"2024-03-11T21:10:06.986504+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'24","last_scrub_stamp":"2024-03-11T21:09:21.307274+0000","last_deep_scrub":"116'16","last_deep_scrub_stamp":"2024-03-11T20:53:39.292118+0000","last_clean_scrub_stamp":"2024-03-11T21:09:21.307274+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:23.339937+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":625,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.986072+0000","last_change":"2024-03-11T21:09:38.526414+0000","last_active":"2024-03-11T21:10:06.986072+0000","last_peered":"2024-03-11T21:10:06.986072+0000","last_clean":"2024-03-11T21:10:06.986072+0000","last_became_active":"2024-03-11T21:00:27.966997+0000","last_became_peered":"2024-03-11T21:00:27.966997+0000","last_unstale":"2024-03-11T21:10:06.986072+0000","last_undegraded":"2024-03-11T21:10:06.986072+0000","last_fullsized":"2024-03-11T21:10:06.986072+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:38.526227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:55:58.016988+0000","last_clean_scrub_stamp":"2024-03-11T21:09:38.526227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:44.500997+0000","scrub_duration":59,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":635,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.995925+0000","last_change":"2024-03-11T21:09:24.812080+0000","last_active":"2024-03-11T21:10:06.995925+0000","last_peered":"2024-03-11T21:10:06.995925+0000","last_clean":"2024-03-11T21:10:06.995925+0000","last_became_active":"2024-03-11T21:00:48.468386+0000","last_became_peered":"2024-03-11T21:00:48.468386+0000","last_unstale":"2024-03-11T21:10:06.995925+0000","last_undegraded":"2024-03-11T21:10:06.995925+0000","last_fullsized":"2024-03-11T21:10:06.995925+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:24.811789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:09:24.811789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:53.045151+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"166'2","reported_seq":687,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.989622+0000","last_change":"2024-03-11T21:09:41.540284+0000","last_active":"2024-03-11T21:10:06.989622+0000","last_peered":"2024-03-11T21:10:06.989622+0000","last_clean":"2024-03-11T21:10:06.989622+0000","last_became_active":"2024-03-11T21:00:48.466071+0000","last_became_peered":"2024-03-11T21:00:48.466071+0000","last_unstale":"2024-03-11T21:10:06.989622+0000","last_undegraded":"2024-03-11T21:10:06.989622+0000","last_fullsized":"2024-03-11T21:10:06.989622+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'2","last_scrub_stamp":"2024-03-11T21:09:41.540185+0000","last_deep_scrub":"166'2","last_deep_scrub_stamp":"2024-03-11T20:57:39.868776+0000","last_clean_scrub_stamp":"2024-03-11T21:09:41.540185+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:04.813488+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1822,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"207'8","reported_seq":652,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.989546+0000","last_change":"2024-03-11T21:09:34.561108+0000","last_active":"2024-03-11T21:10:06.989546+0000","last_peered":"2024-03-11T21:10:06.989546+0000","last_clean":"2024-03-11T21:10:06.989546+0000","last_became_active":"2024-03-11T20:50:30.101572+0000","last_became_peered":"2024-03-11T20:50:30.101572+0000","last_unstale":"2024-03-11T21:10:06.989546+0000","last_undegraded":"2024-03-11T21:10:06.989546+0000","last_fullsized":"2024-03-11T21:10:06.989546+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'8","last_scrub_stamp":"2024-03-11T21:09:34.557847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:09:34.557847+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:57.326034+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":387,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.987170+0000","last_change":"2024-03-11T21:09:37.476199+0000","last_active":"2024-03-11T21:10:06.987170+0000","last_peered":"2024-03-11T21:10:06.987170+0000","last_clean":"2024-03-11T21:10:06.987170+0000","last_became_active":"2024-03-11T21:08:05.296676+0000","last_became_peered":"2024-03-11T21:08:05.296676+0000","last_unstale":"2024-03-11T21:10:06.987170+0000","last_undegraded":"2024-03-11T21:10:06.987170+0000","last_fullsized":"2024-03-11T21:10:06.987170+0000","mapping_epoch":319,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:37.476087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:09:37.476087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:46.780290+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":338,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.999735+0000","last_change":"2024-03-11T21:08:18.233619+0000","last_active":"2024-03-11T21:10:06.999735+0000","last_peered":"2024-03-11T21:10:06.999735+0000","last_clean":"2024-03-11T21:10:06.999735+0000","last_became_active":"2024-03-11T21:06:28.758133+0000","last_became_peered":"2024-03-11T21:06:28.758133+0000","last_unstale":"2024-03-11T21:10:06.999735+0000","last_undegraded":"2024-03-11T21:10:06.999735+0000","last_fullsized":"2024-03-11T21:10:06.999735+0000","mapping_epoch":297,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:08:18.233542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:54:09.537972+0000","last_clean_scrub_stamp":"2024-03-11T21:08:18.233542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:08.411999+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":409,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.988897+0000","last_change":"2024-03-11T21:09:28.262105+0000","last_active":"2024-03-11T21:10:06.988897+0000","last_peered":"2024-03-11T21:10:06.988897+0000","last_clean":"2024-03-11T21:10:06.988897+0000","last_became_active":"2024-03-11T21:08:05.300426+0000","last_became_peered":"2024-03-11T21:08:05.300426+0000","last_unstale":"2024-03-11T21:10:06.988897+0000","last_undegraded":"2024-03-11T21:10:06.988897+0000","last_fullsized":"2024-03-11T21:10:06.988897+0000","mapping_epoch":319,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:28.261906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:18.682947+0000","last_clean_scrub_stamp":"2024-03-11T21:09:28.261906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:36.633753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"262'24","reported_seq":669,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.985105+0000","last_change":"2024-03-11T21:08:18.124252+0000","last_active":"2024-03-11T21:10:06.985105+0000","last_peered":"2024-03-11T21:10:06.985105+0000","last_clean":"2024-03-11T21:10:06.985105+0000","last_became_active":"2024-03-11T21:00:27.970878+0000","last_became_peered":"2024-03-11T21:00:27.970878+0000","last_unstale":"2024-03-11T21:10:06.985105+0000","last_undegraded":"2024-03-11T21:10:06.985105+0000","last_fullsized":"2024-03-11T21:10:06.985105+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"262'24","last_scrub_stamp":"2024-03-11T21:08:18.124021+0000","last_deep_scrub":"262'24","last_deep_scrub_stamp":"2024-03-11T21:03:28.128719+0000","last_clean_scrub_stamp":"2024-03-11T21:08:18.124021+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:10.585064+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.7","version":"351'10999","reported_seq":11171,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:20.200836+0000","last_change":"2024-03-11T21:09:22.474696+0000","last_active":"2024-03-11T21:10:20.200836+0000","last_peered":"2024-03-11T21:10:20.200836+0000","last_clean":"2024-03-11T21:10:20.200836+0000","last_became_active":"2024-03-11T21:07:14.959711+0000","last_became_peered":"2024-03-11T21:07:14.959711+0000","last_unstale":"2024-03-11T21:10:20.200836+0000","last_undegraded":"2024-03-11T21:10:20.200836+0000","last_fullsized":"2024-03-11T21:10:20.200836+0000","mapping_epoch":306,"log_start":"328'5200","ondisk_log_start":"328'5200","created":306,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'7694","last_scrub_stamp":"2024-03-11T21:09:22.474009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:07:13.846944+0000","last_clean_scrub_stamp":"2024-03-11T21:09:22.474009+0000","objects_scrubbed":6135,"log_size":5799,"log_dups_size":0,"ondisk_log_size":5799,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:44.945898+0000","scrub_duration":2875,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180420608,"num_objects":2753,"num_object_clones":0,"num_object_copies":8259,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2753,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10999,"num_write_kb":440064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":639,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.996108+0000","last_change":"2024-03-11T21:09:51.110151+0000","last_active":"2024-03-11T21:10:06.996108+0000","last_peered":"2024-03-11T21:10:06.996108+0000","last_clean":"2024-03-11T21:10:06.996108+0000","last_became_active":"2024-03-11T21:00:48.466853+0000","last_became_peered":"2024-03-11T21:00:48.466853+0000","last_unstale":"2024-03-11T21:10:06.996108+0000","last_undegraded":"2024-03-11T21:10:06.996108+0000","last_fullsized":"2024-03-11T21:10:06.996108+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:51.109823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:52:53.450885+0000","last_clean_scrub_stamp":"2024-03-11T21:09:51.109823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:07.631014+0000","scrub_duration":139,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"77'2","reported_seq":136,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.989411+0000","last_change":"2024-03-11T21:08:04.913072+0000","last_active":"2024-03-11T21:10:06.989411+0000","last_peered":"2024-03-11T21:10:06.989411+0000","last_clean":"2024-03-11T21:10:06.989411+0000","last_became_active":"2024-03-11T21:06:28.759838+0000","last_became_peered":"2024-03-11T21:06:28.759838+0000","last_unstale":"2024-03-11T21:10:06.989411+0000","last_undegraded":"2024-03-11T21:10:06.989411+0000","last_fullsized":"2024-03-11T21:10:06.989411+0000","mapping_epoch":297,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'2","last_scrub_stamp":"2024-03-11T21:08:04.912948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:08:04.912948+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:08.412087+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"207'24","reported_seq":729,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.986312+0000","last_change":"2024-03-11T21:08:14.074734+0000","last_active":"2024-03-11T21:10:06.986312+0000","last_peered":"2024-03-11T21:10:06.986312+0000","last_clean":"2024-03-11T21:10:06.986312+0000","last_became_active":"2024-03-11T21:00:48.465254+0000","last_became_peered":"2024-03-11T21:00:48.465254+0000","last_unstale":"2024-03-11T21:10:06.986312+0000","last_undegraded":"2024-03-11T21:10:06.986312+0000","last_fullsized":"2024-03-11T21:10:06.986312+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'24","last_scrub_stamp":"2024-03-11T21:08:14.074642+0000","last_deep_scrub":"100'16","last_deep_scrub_stamp":"2024-03-11T20:56:08.036331+0000","last_clean_scrub_stamp":"2024-03-11T21:08:14.074642+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:10.585119+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"286'18","reported_seq":608,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:09.067980+0000","last_change":"2024-03-11T21:09:18.222711+0000","last_active":"2024-03-11T21:10:09.067980+0000","last_peered":"2024-03-11T21:10:09.067980+0000","last_clean":"2024-03-11T21:10:09.067980+0000","last_became_active":"2024-03-11T21:05:17.317915+0000","last_became_peered":"2024-03-11T21:05:17.317915+0000","last_unstale":"2024-03-11T21:10:09.067980+0000","last_undegraded":"2024-03-11T21:10:09.067980+0000","last_fullsized":"2024-03-11T21:10:09.067980+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"286'18","last_scrub_stamp":"2024-03-11T21:09:18.222637+0000","last_deep_scrub":"212'17","last_deep_scrub_stamp":"2024-03-11T21:03:38.258850+0000","last_clean_scrub_stamp":"2024-03-11T21:09:18.222637+0000","objects_scrubbed":1,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:29.926701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.c","version":"351'10661","reported_seq":10832,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:19.849936+0000","last_change":"2024-03-11T21:09:19.776322+0000","last_active":"2024-03-11T21:10:19.849936+0000","last_peered":"2024-03-11T21:10:19.849936+0000","last_clean":"2024-03-11T21:10:19.849936+0000","last_became_active":"2024-03-11T21:07:14.952564+0000","last_became_peered":"2024-03-11T21:07:14.952564+0000","last_unstale":"2024-03-11T21:10:19.849936+0000","last_undegraded":"2024-03-11T21:10:19.849936+0000","last_fullsized":"2024-03-11T21:10:19.849936+0000","mapping_epoch":306,"log_start":"331'5700","ondisk_log_start":"331'5700","created":306,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"337'7359","last_scrub_stamp":"2024-03-11T21:09:19.776149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:07:13.846944+0000","last_clean_scrub_stamp":"2024-03-11T21:09:19.776149+0000","objects_scrubbed":6099,"log_size":4961,"log_dups_size":0,"ondisk_log_size":4961,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:47.626345+0000","scrub_duration":2543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180158464,"num_objects":2749,"num_object_clones":0,"num_object_copies":8247,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2749,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10661,"num_write_kb":429120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":321,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.988098+0000","last_change":"2024-03-11T21:08:20.317891+0000","last_active":"2024-03-11T21:10:06.988098+0000","last_peered":"2024-03-11T21:10:06.988098+0000","last_clean":"2024-03-11T21:10:06.988098+0000","last_became_active":"2024-03-11T21:06:28.755689+0000","last_became_peered":"2024-03-11T21:06:28.755689+0000","last_unstale":"2024-03-11T21:10:06.988098+0000","last_undegraded":"2024-03-11T21:10:06.988098+0000","last_fullsized":"2024-03-11T21:10:06.988098+0000","mapping_epoch":297,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:08:20.317781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:57:21.568400+0000","last_clean_scrub_stamp":"2024-03-11T21:08:20.317781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:08.412280+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":345,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.992655+0000","last_change":"2024-03-11T21:09:41.244145+0000","last_active":"2024-03-11T21:10:06.992655+0000","last_peered":"2024-03-11T21:10:06.992655+0000","last_clean":"2024-03-11T21:10:06.992655+0000","last_became_active":"2024-03-11T21:00:48.455733+0000","last_became_peered":"2024-03-11T21:00:48.455733+0000","last_unstale":"2024-03-11T21:10:06.992655+0000","last_undegraded":"2024-03-11T21:10:06.992655+0000","last_fullsized":"2024-03-11T21:10:06.992655+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:41.244062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:20.096874+0000","last_clean_scrub_stamp":"2024-03-11T21:09:41.244062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:05.438898+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"207'28","reported_seq":913,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:09.068301+0000","last_change":"2024-03-11T21:09:30.729303+0000","last_active":"2024-03-11T21:10:09.068301+0000","last_peered":"2024-03-11T21:10:09.068301+0000","last_clean":"2024-03-11T21:10:09.068301+0000","last_became_active":"2024-03-11T20:50:30.094977+0000","last_became_peered":"2024-03-11T20:50:30.094977+0000","last_unstale":"2024-03-11T21:10:09.068301+0000","last_undegraded":"2024-03-11T21:10:09.068301+0000","last_fullsized":"2024-03-11T21:10:09.068301+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'28","last_scrub_stamp":"2024-03-11T21:09:30.729172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:09:30.729172+0000","objects_scrubbed":8,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:36.914077+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":345,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.985490+0000","last_change":"2024-03-11T21:09:42.231430+0000","last_active":"2024-03-11T21:10:06.985490+0000","last_peered":"2024-03-11T21:10:06.985490+0000","last_clean":"2024-03-11T21:10:06.985490+0000","last_became_active":"2024-03-11T21:00:48.457901+0000","last_became_peered":"2024-03-11T21:00:48.457901+0000","last_unstale":"2024-03-11T21:10:06.985490+0000","last_undegraded":"2024-03-11T21:10:06.985490+0000","last_fullsized":"2024-03-11T21:10:06.985490+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:42.231197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:52:10.160202+0000","last_clean_scrub_stamp":"2024-03-11T21:09:42.231197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:57.587423+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.a","version":"351'10843","reported_seq":11010,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:20.197351+0000","last_change":"2024-03-11T21:09:27.146570+0000","last_active":"2024-03-11T21:10:20.197351+0000","last_peered":"2024-03-11T21:10:20.197351+0000","last_clean":"2024-03-11T21:10:20.197351+0000","last_became_active":"2024-03-11T21:07:14.883606+0000","last_became_peered":"2024-03-11T21:07:14.883606+0000","last_unstale":"2024-03-11T21:10:20.197351+0000","last_undegraded":"2024-03-11T21:10:20.197351+0000","last_fullsized":"2024-03-11T21:10:20.197351+0000","mapping_epoch":306,"log_start":"327'5000","ondisk_log_start":"327'5000","created":306,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"341'7819","last_scrub_stamp":"2024-03-11T21:09:27.146329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:07:13.846944+0000","last_clean_scrub_stamp":"2024-03-11T21:09:27.146329+0000","objects_scrubbed":5786,"log_size":5843,"log_dups_size":0,"ondisk_log_size":5843,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:42.309460+0000","scrub_duration":2515,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":177537024,"num_objects":2709,"num_object_clones":0,"num_object_copies":8127,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2709,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10843,"num_write_kb":433664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":620,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.995604+0000","last_change":"2024-03-11T21:09:39.887725+0000","last_active":"2024-03-11T21:10:06.995604+0000","last_peered":"2024-03-11T21:10:06.995604+0000","last_clean":"2024-03-11T21:10:06.995604+0000","last_became_active":"2024-03-11T21:00:27.959408+0000","last_became_peered":"2024-03-11T21:00:27.959408+0000","last_unstale":"2024-03-11T21:10:06.995604+0000","last_undegraded":"2024-03-11T21:10:06.995604+0000","last_fullsized":"2024-03-11T21:10:06.995604+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:39.887594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:09:39.887594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:04.704389+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":660,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.988816+0000","last_change":"2024-03-11T21:09:55.786453+0000","last_active":"2024-03-11T21:10:06.988816+0000","last_peered":"2024-03-11T21:10:06.988816+0000","last_clean":"2024-03-11T21:10:06.988816+0000","last_became_active":"2024-03-11T21:00:48.471212+0000","last_became_peered":"2024-03-11T21:00:48.471212+0000","last_unstale":"2024-03-11T21:10:06.988816+0000","last_undegraded":"2024-03-11T21:10:06.988816+0000","last_fullsized":"2024-03-11T21:10:06.988816+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:55.786349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:09:55.786349+0000","last_clean_scrub_stamp":"2024-03-11T21:09:55.786349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:19.421312+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"207'37","reported_seq":468,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.988546+0000","last_change":"2024-03-11T21:09:35.262587+0000","last_active":"2024-03-11T21:10:06.988546+0000","last_peered":"2024-03-11T21:10:06.988546+0000","last_clean":"2024-03-11T21:10:06.988546+0000","last_became_active":"2024-03-11T21:06:28.756995+0000","last_became_peered":"2024-03-11T21:06:28.756995+0000","last_unstale":"2024-03-11T21:10:06.988546+0000","last_undegraded":"2024-03-11T21:10:06.988546+0000","last_fullsized":"2024-03-11T21:10:06.988546+0000","mapping_epoch":297,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'37","last_scrub_stamp":"2024-03-11T21:09:35.261983+0000","last_deep_scrub":"182'27","last_deep_scrub_stamp":"2024-03-11T20:58:52.058851+0000","last_clean_scrub_stamp":"2024-03-11T21:09:35.261983+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:04.285876+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":345,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.992081+0000","last_change":"2024-03-11T21:09:48.134846+0000","last_active":"2024-03-11T21:10:06.992081+0000","last_peered":"2024-03-11T21:10:06.992081+0000","last_clean":"2024-03-11T21:10:06.992081+0000","last_became_active":"2024-03-11T21:00:48.458008+0000","last_became_peered":"2024-03-11T21:00:48.458008+0000","last_unstale":"2024-03-11T21:10:06.992081+0000","last_undegraded":"2024-03-11T21:10:06.992081+0000","last_fullsized":"2024-03-11T21:10:06.992081+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:48.134729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:33.319221+0000","last_clean_scrub_stamp":"2024-03-11T21:09:48.134729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:18.134249+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.9","version":"351'10562","reported_seq":10735,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:15.594452+0000","last_change":"2024-03-11T21:09:19.225140+0000","last_active":"2024-03-11T21:10:15.594452+0000","last_peered":"2024-03-11T21:10:15.594452+0000","last_clean":"2024-03-11T21:10:15.594452+0000","last_became_active":"2024-03-11T21:07:14.877262+0000","last_became_peered":"2024-03-11T21:07:14.877262+0000","last_unstale":"2024-03-11T21:10:15.594452+0000","last_undegraded":"2024-03-11T21:10:15.594452+0000","last_fullsized":"2024-03-11T21:10:15.594452+0000","mapping_epoch":306,"log_start":"326'4600","ondisk_log_start":"326'4600","created":306,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"337'7384","last_scrub_stamp":"2024-03-11T21:09:19.224937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:07:13.846944+0000","last_clean_scrub_stamp":"2024-03-11T21:09:19.224937+0000","objects_scrubbed":6230,"log_size":5962,"log_dups_size":0,"ondisk_log_size":5962,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:41.615452+0000","scrub_duration":2407,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196345856,"num_objects":2996,"num_object_clones":0,"num_object_copies":8988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2996,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10562,"num_write_kb":433856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":633,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.986171+0000","last_change":"2024-03-11T21:09:46.434579+0000","last_active":"2024-03-11T21:10:06.986171+0000","last_peered":"2024-03-11T21:10:06.986171+0000","last_clean":"2024-03-11T21:10:06.986171+0000","last_became_active":"2024-03-11T20:50:34.321038+0000","last_became_peered":"2024-03-11T20:50:34.321038+0000","last_unstale":"2024-03-11T21:10:06.986171+0000","last_undegraded":"2024-03-11T21:10:06.986171+0000","last_fullsized":"2024-03-11T21:10:06.986171+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:46.434462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:32.951104+0000","last_clean_scrub_stamp":"2024-03-11T21:09:46.434462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:05.403803+0000","scrub_duration":83,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"207'38","reported_seq":377,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.988405+0000","last_change":"2024-03-11T21:10:02.247772+0000","last_active":"2024-03-11T21:10:06.988405+0000","last_peered":"2024-03-11T21:10:06.988405+0000","last_clean":"2024-03-11T21:10:06.988405+0000","last_became_active":"2024-03-11T21:00:48.456178+0000","last_became_peered":"2024-03-11T21:00:48.456178+0000","last_unstale":"2024-03-11T21:10:06.988405+0000","last_undegraded":"2024-03-11T21:10:06.988405+0000","last_fullsized":"2024-03-11T21:10:06.988405+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'38","last_scrub_stamp":"2024-03-11T21:10:02.247651+0000","last_deep_scrub":"109'26","last_deep_scrub_stamp":"2024-03-11T20:52:49.216378+0000","last_clean_scrub_stamp":"2024-03-11T21:10:02.247651+0000","objects_scrubbed":8,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:21.873016+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":76,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.994828+0000","last_change":"2024-03-11T21:08:05.308997+0000","last_active":"2024-03-11T21:10:06.994828+0000","last_peered":"2024-03-11T21:10:06.994828+0000","last_clean":"2024-03-11T21:10:06.994828+0000","last_became_active":"2024-03-11T21:08:05.308608+0000","last_became_peered":"2024-03-11T21:08:05.308608+0000","last_unstale":"2024-03-11T21:10:06.994828+0000","last_undegraded":"2024-03-11T21:10:06.994828+0000","last_fullsized":"2024-03-11T21:10:06.994828+0000","mapping_epoch":319,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:07:43.347185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:07:43.347185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:14.759555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":337,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.993182+0000","last_change":"2024-03-11T21:09:26.296868+0000","last_active":"2024-03-11T21:10:06.993182+0000","last_peered":"2024-03-11T21:10:06.993182+0000","last_clean":"2024-03-11T21:10:06.993182+0000","last_became_active":"2024-03-11T21:00:48.473346+0000","last_became_peered":"2024-03-11T21:00:48.473346+0000","last_unstale":"2024-03-11T21:10:06.993182+0000","last_undegraded":"2024-03-11T21:10:06.993182+0000","last_fullsized":"2024-03-11T21:10:06.993182+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:26.296711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:09:26.296711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:53.259367+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"351'10809","reported_seq":10978,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:20.188017+0000","last_change":"2024-03-11T21:09:24.802751+0000","last_active":"2024-03-11T21:10:20.188017+0000","last_peered":"2024-03-11T21:10:20.188017+0000","last_clean":"2024-03-11T21:10:20.188017+0000","last_became_active":"2024-03-11T21:07:14.892853+0000","last_became_peered":"2024-03-11T21:07:14.892853+0000","last_unstale":"2024-03-11T21:10:20.188017+0000","last_undegraded":"2024-03-11T21:10:20.188017+0000","last_fullsized":"2024-03-11T21:10:20.188017+0000","mapping_epoch":306,"log_start":"328'5000","ondisk_log_start":"328'5000","created":306,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"339'7705","last_scrub_stamp":"2024-03-11T21:09:24.802619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:07:13.846944+0000","last_clean_scrub_stamp":"2024-03-11T21:09:24.802619+0000","objects_scrubbed":5943,"log_size":5809,"log_dups_size":0,"ondisk_log_size":5809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:46.561935+0000","scrub_duration":2167,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":182255616,"num_objects":2781,"num_object_clones":0,"num_object_copies":8343,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2781,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10809,"num_write_kb":434880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":628,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.986136+0000","last_change":"2024-03-11T21:09:32.825302+0000","last_active":"2024-03-11T21:10:06.986136+0000","last_peered":"2024-03-11T21:10:06.986136+0000","last_clean":"2024-03-11T21:10:06.986136+0000","last_became_active":"2024-03-11T21:00:27.962668+0000","last_became_peered":"2024-03-11T21:00:27.962668+0000","last_unstale":"2024-03-11T21:10:06.986136+0000","last_undegraded":"2024-03-11T21:10:06.986136+0000","last_fullsized":"2024-03-11T21:10:06.986136+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:32.825166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:09:32.825166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:01.301038+0000","scrub_duration":95,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"284'25","reported_seq":654,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.995226+0000","last_change":"2024-03-11T21:08:11.417836+0000","last_active":"2024-03-11T21:10:06.995226+0000","last_peered":"2024-03-11T21:10:06.995226+0000","last_clean":"2024-03-11T21:10:06.995226+0000","last_became_active":"2024-03-11T20:50:30.101747+0000","last_became_peered":"2024-03-11T20:50:30.101747+0000","last_unstale":"2024-03-11T21:10:06.995226+0000","last_undegraded":"2024-03-11T21:10:06.995226+0000","last_fullsized":"2024-03-11T21:10:06.995226+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"284'25","last_scrub_stamp":"2024-03-11T21:08:11.417688+0000","last_deep_scrub":"284'25","last_deep_scrub_stamp":"2024-03-11T21:04:53.217188+0000","last_clean_scrub_stamp":"2024-03-11T21:08:11.417688+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":647,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.985450+0000","last_change":"2024-03-11T21:09:56.781956+0000","last_active":"2024-03-11T21:10:06.985450+0000","last_peered":"2024-03-11T21:10:06.985450+0000","last_clean":"2024-03-11T21:10:06.985450+0000","last_became_active":"2024-03-11T20:50:28.085503+0000","last_became_peered":"2024-03-11T20:50:28.085503+0000","last_unstale":"2024-03-11T21:10:06.985450+0000","last_undegraded":"2024-03-11T21:10:06.985450+0000","last_fullsized":"2024-03-11T21:10:06.985450+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:56.781841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:56:03.479180+0000","last_clean_scrub_stamp":"2024-03-11T21:09:56.781841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:22.735910+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"184'130","reported_seq":914,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.985422+0000","last_change":"2024-03-11T21:09:59.511581+0000","last_active":"2024-03-11T21:10:06.985422+0000","last_peered":"2024-03-11T21:10:06.985422+0000","last_clean":"2024-03-11T21:10:06.985422+0000","last_became_active":"2024-03-11T20:49:52.584543+0000","last_became_peered":"2024-03-11T20:49:52.584543+0000","last_unstale":"2024-03-11T21:10:06.985422+0000","last_undegraded":"2024-03-11T21:10:06.985422+0000","last_fullsized":"2024-03-11T21:10:06.985422+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"184'130","last_scrub_stamp":"2024-03-11T21:09:59.511442+0000","last_deep_scrub":"184'130","last_deep_scrub_stamp":"2024-03-11T20:58:56.415267+0000","last_clean_scrub_stamp":"2024-03-11T21:09:59.511442+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:14.704627+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":654,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.990361+0000","last_change":"2024-03-11T21:09:52.505903+0000","last_active":"2024-03-11T21:10:06.990361+0000","last_peered":"2024-03-11T21:10:06.990361+0000","last_clean":"2024-03-11T21:10:06.990361+0000","last_became_active":"2024-03-11T21:00:27.959467+0000","last_became_peered":"2024-03-11T21:00:27.959467+0000","last_unstale":"2024-03-11T21:10:06.990361+0000","last_undegraded":"2024-03-11T21:10:06.990361+0000","last_fullsized":"2024-03-11T21:10:06.990361+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:52.505786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:54:22.448812+0000","last_clean_scrub_stamp":"2024-03-11T21:09:52.505786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:03.085316+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.8","version":"351'10603","reported_seq":10771,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:15.291576+0000","last_change":"2024-03-11T21:09:26.872217+0000","last_active":"2024-03-11T21:10:15.291576+0000","last_peered":"2024-03-11T21:10:15.291576+0000","last_clean":"2024-03-11T21:10:15.291576+0000","last_became_active":"2024-03-11T21:07:14.891196+0000","last_became_peered":"2024-03-11T21:07:14.891196+0000","last_unstale":"2024-03-11T21:10:15.291576+0000","last_undegraded":"2024-03-11T21:10:15.291576+0000","last_fullsized":"2024-03-11T21:10:15.291576+0000","mapping_epoch":306,"log_start":"328'5200","ondisk_log_start":"328'5200","created":306,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"341'7890","last_scrub_stamp":"2024-03-11T21:09:26.872082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:07:13.846944+0000","last_clean_scrub_stamp":"2024-03-11T21:09:26.872082+0000","objects_scrubbed":5788,"log_size":5403,"log_dups_size":0,"ondisk_log_size":5403,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:44.954640+0000","scrub_duration":1827,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197984256,"num_objects":3021,"num_object_clones":0,"num_object_copies":9063,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10603,"num_write_kb":435968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":345,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.992334+0000","last_change":"2024-03-11T21:09:56.170013+0000","last_active":"2024-03-11T21:10:06.992334+0000","last_peered":"2024-03-11T21:10:06.992334+0000","last_clean":"2024-03-11T21:10:06.992334+0000","last_became_active":"2024-03-11T21:00:48.464596+0000","last_became_peered":"2024-03-11T21:00:48.464596+0000","last_unstale":"2024-03-11T21:10:06.992334+0000","last_undegraded":"2024-03-11T21:10:06.992334+0000","last_fullsized":"2024-03-11T21:10:06.992334+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:56.169836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:09:56.169836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:13.046765+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"207'35","reported_seq":317,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.988885+0000","last_change":"2024-03-11T21:08:24.324884+0000","last_active":"2024-03-11T21:10:06.988885+0000","last_peered":"2024-03-11T21:10:06.988885+0000","last_clean":"2024-03-11T21:10:06.988885+0000","last_became_active":"2024-03-11T21:06:28.761186+0000","last_became_peered":"2024-03-11T21:06:28.761186+0000","last_unstale":"2024-03-11T21:10:06.988885+0000","last_undegraded":"2024-03-11T21:10:06.988885+0000","last_fullsized":"2024-03-11T21:10:06.988885+0000","mapping_epoch":297,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'35","last_scrub_stamp":"2024-03-11T21:08:24.324628+0000","last_deep_scrub":"100'21","last_deep_scrub_stamp":"2024-03-11T20:51:31.835302+0000","last_clean_scrub_stamp":"2024-03-11T21:08:24.324628+0000","objects_scrubbed":7,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:08.412028+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":376,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.997861+0000","last_change":"2024-03-11T21:09:46.204912+0000","last_active":"2024-03-11T21:10:06.997861+0000","last_peered":"2024-03-11T21:10:06.997861+0000","last_clean":"2024-03-11T21:10:06.997861+0000","last_became_active":"2024-03-11T21:08:05.301024+0000","last_became_peered":"2024-03-11T21:08:05.301024+0000","last_unstale":"2024-03-11T21:10:06.997861+0000","last_undegraded":"2024-03-11T21:10:06.997861+0000","last_fullsized":"2024-03-11T21:10:06.997861+0000","mapping_epoch":319,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:46.204490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:09:46.204490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:01.338837+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":649,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.999550+0000","last_change":"2024-03-11T21:08:16.214623+0000","last_active":"2024-03-11T21:10:06.999550+0000","last_peered":"2024-03-11T21:10:06.999550+0000","last_clean":"2024-03-11T21:10:06.999550+0000","last_became_active":"2024-03-11T21:00:48.463247+0000","last_became_peered":"2024-03-11T21:00:48.463247+0000","last_unstale":"2024-03-11T21:10:06.999550+0000","last_undegraded":"2024-03-11T21:10:06.999550+0000","last_fullsized":"2024-03-11T21:10:06.999550+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:08:16.214363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:26.839782+0000","last_clean_scrub_stamp":"2024-03-11T21:08:16.214363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:08.412306+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.d","version":"351'10911","reported_seq":11081,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:20.160062+0000","last_change":"2024-03-11T21:09:24.323453+0000","last_active":"2024-03-11T21:10:20.160062+0000","last_peered":"2024-03-11T21:10:20.160062+0000","last_clean":"2024-03-11T21:10:20.160062+0000","last_became_active":"2024-03-11T21:07:14.951361+0000","last_became_peered":"2024-03-11T21:07:14.951361+0000","last_unstale":"2024-03-11T21:10:20.160062+0000","last_undegraded":"2024-03-11T21:10:20.160062+0000","last_fullsized":"2024-03-11T21:10:20.160062+0000","mapping_epoch":306,"log_start":"328'5100","ondisk_log_start":"328'5100","created":306,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'7756","last_scrub_stamp":"2024-03-11T21:09:24.323264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:07:13.846944+0000","last_clean_scrub_stamp":"2024-03-11T21:09:24.323264+0000","objects_scrubbed":5952,"log_size":5811,"log_dups_size":0,"ondisk_log_size":5811,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:29.958414+0000","scrub_duration":2667,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":179044376,"num_objects":2733,"num_object_clones":0,"num_object_copies":8199,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2733,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10911,"num_write_kb":436545,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":345,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.991613+0000","last_change":"2024-03-11T21:09:40.246017+0000","last_active":"2024-03-11T21:10:06.991613+0000","last_peered":"2024-03-11T21:10:06.991613+0000","last_clean":"2024-03-11T21:10:06.991613+0000","last_became_active":"2024-03-11T21:00:48.469688+0000","last_became_peered":"2024-03-11T21:00:48.469688+0000","last_unstale":"2024-03-11T21:10:06.991613+0000","last_undegraded":"2024-03-11T21:10:06.991613+0000","last_fullsized":"2024-03-11T21:10:06.991613+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:40.245934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:09:40.245934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:04.261098+0000","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"145'3","reported_seq":647,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.990037+0000","last_change":"2024-03-11T21:09:30.672258+0000","last_active":"2024-03-11T21:10:06.990037+0000","last_peered":"2024-03-11T21:10:06.990037+0000","last_clean":"2024-03-11T21:10:06.990037+0000","last_became_active":"2024-03-11T20:50:32.195452+0000","last_became_peered":"2024-03-11T20:50:32.195452+0000","last_unstale":"2024-03-11T21:10:06.990037+0000","last_undegraded":"2024-03-11T21:10:06.990037+0000","last_fullsized":"2024-03-11T21:10:06.990037+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'3","last_scrub_stamp":"2024-03-11T21:09:30.672123+0000","last_deep_scrub":"145'3","last_deep_scrub_stamp":"2024-03-11T20:55:52.848043+0000","last_clean_scrub_stamp":"2024-03-11T21:09:30.672123+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:38.511160+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0.0024049620000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":646,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.986353+0000","last_change":"2024-03-11T21:09:55.530420+0000","last_active":"2024-03-11T21:10:06.986353+0000","last_peered":"2024-03-11T21:10:06.986353+0000","last_clean":"2024-03-11T21:10:06.986353+0000","last_became_active":"2024-03-11T21:00:27.966749+0000","last_became_peered":"2024-03-11T21:00:27.966749+0000","last_unstale":"2024-03-11T21:10:06.986353+0000","last_undegraded":"2024-03-11T21:10:06.986353+0000","last_fullsized":"2024-03-11T21:10:06.986353+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:55.530113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:09:55.530113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:06.152784+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"212'46","reported_seq":488,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:09.067991+0000","last_change":"2024-03-11T21:09:33.321434+0000","last_active":"2024-03-11T21:10:09.067991+0000","last_peered":"2024-03-11T21:10:09.067991+0000","last_clean":"2024-03-11T21:10:09.067991+0000","last_became_active":"2024-03-11T21:00:48.473227+0000","last_became_peered":"2024-03-11T21:00:48.473227+0000","last_unstale":"2024-03-11T21:10:09.067991+0000","last_undegraded":"2024-03-11T21:10:09.067991+0000","last_fullsized":"2024-03-11T21:10:09.067991+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"212'46","last_scrub_stamp":"2024-03-11T21:09:33.321341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:09:33.321341+0000","objects_scrubbed":10,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:52.366893+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":69,"num_read_kb":54,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"298'13","reported_seq":453,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:09.067985+0000","last_change":"2024-03-11T21:08:08.181167+0000","last_active":"2024-03-11T21:10:09.067985+0000","last_peered":"2024-03-11T21:10:09.067985+0000","last_clean":"2024-03-11T21:10:09.067985+0000","last_became_active":"2024-03-11T21:06:28.762397+0000","last_became_peered":"2024-03-11T21:06:28.762397+0000","last_unstale":"2024-03-11T21:10:09.067985+0000","last_undegraded":"2024-03-11T21:10:09.067985+0000","last_fullsized":"2024-03-11T21:10:09.067985+0000","mapping_epoch":297,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"298'13","last_scrub_stamp":"2024-03-11T21:08:08.181013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:08:08.181013+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.e","version":"351'10878","reported_seq":11046,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:20.184684+0000","last_change":"2024-03-11T21:09:27.964811+0000","last_active":"2024-03-11T21:10:20.184684+0000","last_peered":"2024-03-11T21:10:20.184684+0000","last_clean":"2024-03-11T21:10:20.184684+0000","last_became_active":"2024-03-11T21:07:14.953126+0000","last_became_peered":"2024-03-11T21:07:14.953126+0000","last_unstale":"2024-03-11T21:10:20.184684+0000","last_undegraded":"2024-03-11T21:10:20.184684+0000","last_fullsized":"2024-03-11T21:10:20.184684+0000","mapping_epoch":306,"log_start":"328'5100","ondisk_log_start":"328'5100","created":306,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"341'7918","last_scrub_stamp":"2024-03-11T21:09:27.964498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:07:13.846944+0000","last_clean_scrub_stamp":"2024-03-11T21:09:27.964498+0000","objects_scrubbed":5787,"log_size":5778,"log_dups_size":0,"ondisk_log_size":5778,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:39.102404+0000","scrub_duration":2295,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180486144,"num_objects":2754,"num_object_clones":0,"num_object_copies":8262,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2754,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10878,"num_write_kb":436224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":365,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.995373+0000","last_change":"2024-03-11T21:09:34.951119+0000","last_active":"2024-03-11T21:10:06.995373+0000","last_peered":"2024-03-11T21:10:06.995373+0000","last_clean":"2024-03-11T21:10:06.995373+0000","last_became_active":"2024-03-11T21:05:17.441521+0000","last_became_peered":"2024-03-11T21:05:17.441521+0000","last_unstale":"2024-03-11T21:10:06.995373+0000","last_undegraded":"2024-03-11T21:10:06.995373+0000","last_fullsized":"2024-03-11T21:10:06.995373+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:34.951014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:03:43.345257+0000","last_clean_scrub_stamp":"2024-03-11T21:09:34.951014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:38.035059+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":386,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.994983+0000","last_change":"2024-03-11T21:09:35.935667+0000","last_active":"2024-03-11T21:10:06.994983+0000","last_peered":"2024-03-11T21:10:06.994983+0000","last_clean":"2024-03-11T21:10:06.994983+0000","last_became_active":"2024-03-11T21:05:17.315844+0000","last_became_peered":"2024-03-11T21:05:17.315844+0000","last_unstale":"2024-03-11T21:10:06.994983+0000","last_undegraded":"2024-03-11T21:10:06.994983+0000","last_fullsized":"2024-03-11T21:10:06.994983+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:35.935559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:22.617999+0000","last_clean_scrub_stamp":"2024-03-11T21:09:35.935559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:57.074238+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"284'2","reported_seq":350,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.992756+0000","last_change":"2024-03-11T21:09:54.213683+0000","last_active":"2024-03-11T21:10:06.992756+0000","last_peered":"2024-03-11T21:10:06.992756+0000","last_clean":"2024-03-11T21:10:06.992756+0000","last_became_active":"2024-03-11T21:00:48.468682+0000","last_became_peered":"2024-03-11T21:00:48.468682+0000","last_unstale":"2024-03-11T21:10:06.992756+0000","last_undegraded":"2024-03-11T21:10:06.992756+0000","last_fullsized":"2024-03-11T21:10:06.992756+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"284'2","last_scrub_stamp":"2024-03-11T21:09:54.213473+0000","last_deep_scrub":"284'2","last_deep_scrub_stamp":"2024-03-11T21:06:47.645682+0000","last_clean_scrub_stamp":"2024-03-11T21:09:54.213473+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:13.303666+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"207'17","reported_seq":652,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.995710+0000","last_change":"2024-03-11T21:09:54.016666+0000","last_active":"2024-03-11T21:10:06.995710+0000","last_peered":"2024-03-11T21:10:06.995710+0000","last_clean":"2024-03-11T21:10:06.995710+0000","last_became_active":"2024-03-11T20:50:30.097669+0000","last_became_peered":"2024-03-11T20:50:30.097669+0000","last_unstale":"2024-03-11T21:10:06.995710+0000","last_undegraded":"2024-03-11T21:10:06.995710+0000","last_fullsized":"2024-03-11T21:10:06.995710+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'17","last_scrub_stamp":"2024-03-11T21:09:54.016577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:09:54.016577+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:21.301423+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"84'11","reported_seq":872,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:14.076262+0000","last_change":"2024-03-11T21:08:04.428744+0000","last_active":"2024-03-11T21:10:14.076262+0000","last_peered":"2024-03-11T21:10:14.076262+0000","last_clean":"2024-03-11T21:10:14.076262+0000","last_became_active":"2024-03-11T20:50:32.647041+0000","last_became_peered":"2024-03-11T20:50:32.647041+0000","last_unstale":"2024-03-11T21:10:14.076262+0000","last_undegraded":"2024-03-11T21:10:14.076262+0000","last_fullsized":"2024-03-11T21:10:14.076262+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:08:04.428606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:08:04.428606+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":83,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.f","version":"351'11086","reported_seq":11290,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:20.200604+0000","last_change":"2024-03-11T21:09:58.987446+0000","last_active":"2024-03-11T21:10:20.200604+0000","last_peered":"2024-03-11T21:10:20.200604+0000","last_clean":"2024-03-11T21:10:20.200604+0000","last_became_active":"2024-03-11T21:07:14.959566+0000","last_became_peered":"2024-03-11T21:07:14.959566+0000","last_unstale":"2024-03-11T21:10:20.200604+0000","last_undegraded":"2024-03-11T21:10:20.200604+0000","last_fullsized":"2024-03-11T21:10:20.200604+0000","mapping_epoch":306,"log_start":"328'5300","ondisk_log_start":"328'5300","created":306,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"349'9796","last_scrub_stamp":"2024-03-11T21:09:58.987104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:07:13.846944+0000","last_clean_scrub_stamp":"2024-03-11T21:09:58.987104+0000","objects_scrubbed":4098,"log_size":5786,"log_dups_size":0,"ondisk_log_size":5786,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:00.717053+0000","scrub_duration":1479,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":181403648,"num_objects":2768,"num_object_clones":0,"num_object_copies":8304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2768,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11086,"num_write_kb":443328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":641,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.986339+0000","last_change":"2024-03-11T21:10:02.768209+0000","last_active":"2024-03-11T21:10:06.986339+0000","last_peered":"2024-03-11T21:10:06.986339+0000","last_clean":"2024-03-11T21:10:06.986339+0000","last_became_active":"2024-03-11T20:50:34.326947+0000","last_became_peered":"2024-03-11T20:50:34.326947+0000","last_unstale":"2024-03-11T21:10:06.986339+0000","last_undegraded":"2024-03-11T21:10:06.986339+0000","last_fullsized":"2024-03-11T21:10:06.986339+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:10:02.768094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:06:43.271453+0000","last_clean_scrub_stamp":"2024-03-11T21:10:02.768094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:08.043060+0000","scrub_duration":219,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"120'6","reported_seq":1717,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:17.924991+0000","last_change":"2024-03-11T21:08:19.340719+0000","last_active":"2024-03-11T21:10:17.924991+0000","last_peered":"2024-03-11T21:10:17.924991+0000","last_clean":"2024-03-11T21:10:17.924991+0000","last_became_active":"2024-03-11T21:00:48.468997+0000","last_became_peered":"2024-03-11T21:00:48.468997+0000","last_unstale":"2024-03-11T21:10:17.924991+0000","last_undegraded":"2024-03-11T21:10:17.924991+0000","last_fullsized":"2024-03-11T21:10:17.924991+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"120'6","last_scrub_stamp":"2024-03-11T21:08:19.340643+0000","last_deep_scrub":"120'6","last_deep_scrub_stamp":"2024-03-11T20:54:26.468057+0000","last_clean_scrub_stamp":"2024-03-11T21:08:19.340643+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0035533370000000002,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1053,"num_read_kb":1048,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":653,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.990423+0000","last_change":"2024-03-11T21:09:28.645584+0000","last_active":"2024-03-11T21:10:06.990423+0000","last_peered":"2024-03-11T21:10:06.990423+0000","last_clean":"2024-03-11T21:10:06.990423+0000","last_became_active":"2024-03-11T20:50:28.071707+0000","last_became_peered":"2024-03-11T20:50:28.071707+0000","last_unstale":"2024-03-11T21:10:06.990423+0000","last_undegraded":"2024-03-11T21:10:06.990423+0000","last_fullsized":"2024-03-11T21:10:06.990423+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:28.645476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:23.417283+0000","last_clean_scrub_stamp":"2024-03-11T21:09:28.645476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:34.508383+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"207'24","reported_seq":737,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.999366+0000","last_change":"2024-03-11T21:07:52.838119+0000","last_active":"2024-03-11T21:10:06.999366+0000","last_peered":"2024-03-11T21:10:06.999366+0000","last_clean":"2024-03-11T21:10:06.999366+0000","last_became_active":"2024-03-11T21:00:48.461972+0000","last_became_peered":"2024-03-11T21:00:48.461972+0000","last_unstale":"2024-03-11T21:10:06.999366+0000","last_undegraded":"2024-03-11T21:10:06.999366+0000","last_fullsized":"2024-03-11T21:10:06.999366+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'24","last_scrub_stamp":"2024-03-11T21:07:52.837968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:07:52.837968+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:08.411973+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":415,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.985545+0000","last_change":"2024-03-11T21:09:51.796505+0000","last_active":"2024-03-11T21:10:06.985545+0000","last_peered":"2024-03-11T21:10:06.985545+0000","last_clean":"2024-03-11T21:10:06.985545+0000","last_became_active":"2024-03-11T21:05:17.315807+0000","last_became_peered":"2024-03-11T21:05:17.315807+0000","last_unstale":"2024-03-11T21:10:06.985545+0000","last_undegraded":"2024-03-11T21:10:06.985545+0000","last_fullsized":"2024-03-11T21:10:06.985545+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:51.796383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:09:51.796383+0000","last_clean_scrub_stamp":"2024-03-11T21:09:51.796383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:18.488750+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.0","version":"351'10945","reported_seq":11114,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:20.194259+0000","last_change":"2024-03-11T21:09:25.774724+0000","last_active":"2024-03-11T21:10:20.194259+0000","last_peered":"2024-03-11T21:10:20.194259+0000","last_clean":"2024-03-11T21:10:20.194259+0000","last_became_active":"2024-03-11T21:07:14.952617+0000","last_became_peered":"2024-03-11T21:07:14.952617+0000","last_unstale":"2024-03-11T21:10:20.194259+0000","last_undegraded":"2024-03-11T21:10:20.194259+0000","last_fullsized":"2024-03-11T21:10:20.194259+0000","mapping_epoch":306,"log_start":"327'5100","ondisk_log_start":"327'5100","created":306,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"340'7831","last_scrub_stamp":"2024-03-11T21:09:25.774291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:07:13.846944+0000","last_clean_scrub_stamp":"2024-03-11T21:09:25.774291+0000","objects_scrubbed":5890,"log_size":5845,"log_dups_size":0,"ondisk_log_size":5845,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:29.783691+0000","scrub_duration":2179,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":177537024,"num_objects":2709,"num_object_clones":0,"num_object_copies":8127,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2709,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10945,"num_write_kb":436928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":610,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.996278+0000","last_change":"2024-03-11T21:09:40.876804+0000","last_active":"2024-03-11T21:10:06.996278+0000","last_peered":"2024-03-11T21:10:06.996278+0000","last_clean":"2024-03-11T21:10:06.996278+0000","last_became_active":"2024-03-11T20:50:34.335370+0000","last_became_peered":"2024-03-11T20:50:34.335370+0000","last_unstale":"2024-03-11T21:10:06.996278+0000","last_undegraded":"2024-03-11T21:10:06.996278+0000","last_fullsized":"2024-03-11T21:10:06.996278+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:40.876681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:09:40.876681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:04.620155+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"286'18","reported_seq":293,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:14.077175+0000","last_change":"2024-03-11T21:09:45.491799+0000","last_active":"2024-03-11T21:10:14.077175+0000","last_peered":"2024-03-11T21:10:14.077175+0000","last_clean":"2024-03-11T21:10:14.077175+0000","last_became_active":"2024-03-11T21:05:17.449213+0000","last_became_peered":"2024-03-11T21:05:17.449213+0000","last_unstale":"2024-03-11T21:10:14.077175+0000","last_undegraded":"2024-03-11T21:10:14.077175+0000","last_fullsized":"2024-03-11T21:10:14.077175+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"286'18","last_scrub_stamp":"2024-03-11T21:09:45.491546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:09:45.491546+0000","objects_scrubbed":1,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:07.992752+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":650,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.985519+0000","last_change":"2024-03-11T21:08:16.137000+0000","last_active":"2024-03-11T21:10:06.985519+0000","last_peered":"2024-03-11T21:10:06.985519+0000","last_clean":"2024-03-11T21:10:06.985519+0000","last_became_active":"2024-03-11T21:00:48.466102+0000","last_became_peered":"2024-03-11T21:00:48.466102+0000","last_unstale":"2024-03-11T21:10:06.985519+0000","last_undegraded":"2024-03-11T21:10:06.985519+0000","last_fullsized":"2024-03-11T21:10:06.985519+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:08:16.136851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:06:42.242801+0000","last_clean_scrub_stamp":"2024-03-11T21:08:16.136851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:10.585090+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"207'18","reported_seq":341,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.986751+0000","last_change":"2024-03-11T21:09:23.388366+0000","last_active":"2024-03-11T21:10:06.986751+0000","last_peered":"2024-03-11T21:10:06.986751+0000","last_clean":"2024-03-11T21:10:06.986751+0000","last_became_active":"2024-03-11T21:08:05.298170+0000","last_became_peered":"2024-03-11T21:08:05.298170+0000","last_unstale":"2024-03-11T21:10:06.986751+0000","last_undegraded":"2024-03-11T21:10:06.986751+0000","last_fullsized":"2024-03-11T21:10:06.986751+0000","mapping_epoch":319,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'18","last_scrub_stamp":"2024-03-11T21:09:23.388002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:09:23.388002+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:25.122629+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":23,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1","version":"351'10872","reported_seq":11023,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:19.890889+0000","last_change":"2024-03-11T21:08:23.498439+0000","last_active":"2024-03-11T21:10:19.890889+0000","last_peered":"2024-03-11T21:10:19.890889+0000","last_clean":"2024-03-11T21:10:19.890889+0000","last_became_active":"2024-03-11T21:07:14.951704+0000","last_became_peered":"2024-03-11T21:07:14.951704+0000","last_unstale":"2024-03-11T21:10:19.890889+0000","last_undegraded":"2024-03-11T21:10:19.890889+0000","last_fullsized":"2024-03-11T21:10:19.890889+0000","mapping_epoch":306,"log_start":"332'5900","ondisk_log_start":"332'5900","created":306,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'4267","last_scrub_stamp":"2024-03-11T21:08:23.497931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:07:13.846944+0000","last_clean_scrub_stamp":"2024-03-11T21:08:23.497931+0000","objects_scrubbed":4222,"log_size":4972,"log_dups_size":0,"ondisk_log_size":4972,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:23.664818+0000","scrub_duration":1383,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":181534720,"num_objects":2770,"num_object_clones":0,"num_object_copies":8310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2770,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10872,"num_write_kb":436544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":639,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.986056+0000","last_change":"2024-03-11T21:08:22.890801+0000","last_active":"2024-03-11T21:10:06.986056+0000","last_peered":"2024-03-11T21:10:06.986056+0000","last_clean":"2024-03-11T21:10:06.986056+0000","last_became_active":"2024-03-11T21:00:48.460531+0000","last_became_peered":"2024-03-11T21:00:48.460531+0000","last_unstale":"2024-03-11T21:10:06.986056+0000","last_undegraded":"2024-03-11T21:10:06.986056+0000","last_fullsized":"2024-03-11T21:10:06.986056+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:08:22.890690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:27.775534+0000","last_clean_scrub_stamp":"2024-03-11T21:08:22.890690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:09.668599+0000","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":353,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.993103+0000","last_change":"2024-03-11T21:09:53.196862+0000","last_active":"2024-03-11T21:10:06.993103+0000","last_peered":"2024-03-11T21:10:06.993103+0000","last_clean":"2024-03-11T21:10:06.993103+0000","last_became_active":"2024-03-11T21:00:48.469596+0000","last_became_peered":"2024-03-11T21:00:48.469596+0000","last_unstale":"2024-03-11T21:10:06.993103+0000","last_undegraded":"2024-03-11T21:10:06.993103+0000","last_fullsized":"2024-03-11T21:10:06.993103+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:53.196741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:09:53.196741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:17.301704+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":337,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.991797+0000","last_change":"2024-03-11T21:09:32.311912+0000","last_active":"2024-03-11T21:10:06.991797+0000","last_peered":"2024-03-11T21:10:06.991797+0000","last_clean":"2024-03-11T21:10:06.991797+0000","last_became_active":"2024-03-11T21:00:48.466084+0000","last_became_peered":"2024-03-11T21:00:48.466084+0000","last_unstale":"2024-03-11T21:10:06.991797+0000","last_undegraded":"2024-03-11T21:10:06.991797+0000","last_fullsized":"2024-03-11T21:10:06.991797+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:32.311707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:40.974147+0000","last_clean_scrub_stamp":"2024-03-11T21:09:32.311707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:37.731689+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"207'34","reported_seq":729,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.994144+0000","last_change":"2024-03-11T21:09:19.777457+0000","last_active":"2024-03-11T21:10:06.994144+0000","last_peered":"2024-03-11T21:10:06.994144+0000","last_clean":"2024-03-11T21:10:06.994144+0000","last_became_active":"2024-03-11T21:00:48.472191+0000","last_became_peered":"2024-03-11T21:00:48.472191+0000","last_unstale":"2024-03-11T21:10:06.994144+0000","last_undegraded":"2024-03-11T21:10:06.994144+0000","last_fullsized":"2024-03-11T21:10:06.994144+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'34","last_scrub_stamp":"2024-03-11T21:09:19.777317+0000","last_deep_scrub":"116'22","last_deep_scrub_stamp":"2024-03-11T20:53:47.364709+0000","last_clean_scrub_stamp":"2024-03-11T21:09:19.777317+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:26.269324+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"351'10742","reported_seq":10910,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:17.276596+0000","last_change":"2024-03-11T21:09:25.047442+0000","last_active":"2024-03-11T21:10:17.276596+0000","last_peered":"2024-03-11T21:10:17.276596+0000","last_clean":"2024-03-11T21:10:17.276596+0000","last_became_active":"2024-03-11T21:07:14.951968+0000","last_became_peered":"2024-03-11T21:07:14.951968+0000","last_unstale":"2024-03-11T21:10:17.276596+0000","last_undegraded":"2024-03-11T21:10:17.276596+0000","last_fullsized":"2024-03-11T21:10:17.276596+0000","mapping_epoch":306,"log_start":"327'5000","ondisk_log_start":"327'5000","created":306,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"339'7779","last_scrub_stamp":"2024-03-11T21:09:25.047208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:07:13.846944+0000","last_clean_scrub_stamp":"2024-03-11T21:09:25.047208+0000","objects_scrubbed":5932,"log_size":5742,"log_dups_size":0,"ondisk_log_size":5742,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:45.529146+0000","scrub_duration":1719,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":190971904,"num_objects":2914,"num_object_clones":0,"num_object_copies":8742,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2914,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10742,"num_write_kb":436992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":424,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.992661+0000","last_change":"2024-03-11T21:09:50.191402+0000","last_active":"2024-03-11T21:10:06.992661+0000","last_peered":"2024-03-11T21:10:06.992661+0000","last_clean":"2024-03-11T21:10:06.992661+0000","last_became_active":"2024-03-11T21:05:17.455236+0000","last_became_peered":"2024-03-11T21:05:17.455236+0000","last_unstale":"2024-03-11T21:10:06.992661+0000","last_undegraded":"2024-03-11T21:10:06.992661+0000","last_fullsized":"2024-03-11T21:10:06.992661+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:50.191075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:09:50.191075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:56.493600+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":345,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.986125+0000","last_change":"2024-03-11T21:09:49.147884+0000","last_active":"2024-03-11T21:10:06.986125+0000","last_peered":"2024-03-11T21:10:06.986125+0000","last_clean":"2024-03-11T21:10:06.986125+0000","last_became_active":"2024-03-11T21:00:48.461367+0000","last_became_peered":"2024-03-11T21:00:48.461367+0000","last_unstale":"2024-03-11T21:10:06.986125+0000","last_undegraded":"2024-03-11T21:10:06.986125+0000","last_fullsized":"2024-03-11T21:10:06.986125+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:49.147414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:03:37.601229+0000","last_clean_scrub_stamp":"2024-03-11T21:09:49.147414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:04.968002+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"267'19","reported_seq":656,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.996042+0000","last_change":"2024-03-11T21:08:28.368669+0000","last_active":"2024-03-11T21:10:06.996042+0000","last_peered":"2024-03-11T21:10:06.996042+0000","last_clean":"2024-03-11T21:10:06.996042+0000","last_became_active":"2024-03-11T21:00:27.967572+0000","last_became_peered":"2024-03-11T21:00:27.967572+0000","last_unstale":"2024-03-11T21:10:06.996042+0000","last_undegraded":"2024-03-11T21:10:06.996042+0000","last_fullsized":"2024-03-11T21:10:06.996042+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"267'19","last_scrub_stamp":"2024-03-11T21:08:28.368557+0000","last_deep_scrub":"267'19","last_deep_scrub_stamp":"2024-03-11T21:03:39.673317+0000","last_clean_scrub_stamp":"2024-03-11T21:08:28.368557+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":409,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.996393+0000","last_change":"2024-03-11T21:08:03.331915+0000","last_active":"2024-03-11T21:10:06.996393+0000","last_peered":"2024-03-11T21:10:06.996393+0000","last_clean":"2024-03-11T21:10:06.996393+0000","last_became_active":"2024-03-11T21:05:17.316136+0000","last_became_peered":"2024-03-11T21:05:17.316136+0000","last_unstale":"2024-03-11T21:10:06.996393+0000","last_undegraded":"2024-03-11T21:10:06.996393+0000","last_fullsized":"2024-03-11T21:10:06.996393+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:08:03.331832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:35.670149+0000","last_clean_scrub_stamp":"2024-03-11T21:08:03.331832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"351'10473","reported_seq":10649,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:15.457344+0000","last_change":"2024-03-11T21:09:21.036734+0000","last_active":"2024-03-11T21:10:15.457344+0000","last_peered":"2024-03-11T21:10:15.457344+0000","last_clean":"2024-03-11T21:10:15.457344+0000","last_became_active":"2024-03-11T21:08:05.298758+0000","last_became_peered":"2024-03-11T21:08:05.298758+0000","last_unstale":"2024-03-11T21:10:15.457344+0000","last_undegraded":"2024-03-11T21:10:15.457344+0000","last_fullsized":"2024-03-11T21:10:15.457344+0000","mapping_epoch":319,"log_start":"326'4500","ondisk_log_start":"326'4500","created":306,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'7464","last_scrub_stamp":"2024-03-11T21:09:21.036463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:07:13.846944+0000","last_clean_scrub_stamp":"2024-03-11T21:09:21.036463+0000","objects_scrubbed":6066,"log_size":5973,"log_dups_size":0,"ondisk_log_size":5973,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:49.543563+0000","scrub_duration":2195,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196411392,"num_objects":2997,"num_object_clones":0,"num_object_copies":8991,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10473,"num_write_kb":431040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":388,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.996018+0000","last_change":"2024-03-11T21:10:03.144979+0000","last_active":"2024-03-11T21:10:06.996018+0000","last_peered":"2024-03-11T21:10:06.996018+0000","last_clean":"2024-03-11T21:10:06.996018+0000","last_became_active":"2024-03-11T21:08:05.304963+0000","last_became_peered":"2024-03-11T21:08:05.304963+0000","last_unstale":"2024-03-11T21:10:06.996018+0000","last_undegraded":"2024-03-11T21:10:06.996018+0000","last_fullsized":"2024-03-11T21:10:06.996018+0000","mapping_epoch":319,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:10:03.144875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:10:03.144875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:04.198120+0000","scrub_duration":239,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"320'21","reported_seq":608,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:09.068219+0000","last_change":"2024-03-11T21:09:46.828876+0000","last_active":"2024-03-11T21:10:09.068219+0000","last_peered":"2024-03-11T21:10:09.068219+0000","last_clean":"2024-03-11T21:10:09.068219+0000","last_became_active":"2024-03-11T21:08:05.298887+0000","last_became_peered":"2024-03-11T21:08:05.298887+0000","last_unstale":"2024-03-11T21:10:09.068219+0000","last_undegraded":"2024-03-11T21:10:09.068219+0000","last_fullsized":"2024-03-11T21:10:09.068219+0000","mapping_epoch":319,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'21","last_scrub_stamp":"2024-03-11T21:09:46.828795+0000","last_deep_scrub":"146'14","last_deep_scrub_stamp":"2024-03-11T20:56:00.242211+0000","last_clean_scrub_stamp":"2024-03-11T21:09:46.828795+0000","objects_scrubbed":1,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:03.069048+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":645,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.986432+0000","last_change":"2024-03-11T21:09:43.387853+0000","last_active":"2024-03-11T21:10:06.986432+0000","last_peered":"2024-03-11T21:10:06.986432+0000","last_clean":"2024-03-11T21:10:06.986432+0000","last_became_active":"2024-03-11T21:00:27.970573+0000","last_became_peered":"2024-03-11T21:00:27.970573+0000","last_unstale":"2024-03-11T21:10:06.986432+0000","last_undegraded":"2024-03-11T21:10:06.986432+0000","last_fullsized":"2024-03-11T21:10:06.986432+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:43.387711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:52:51.161281+0000","last_clean_scrub_stamp":"2024-03-11T21:09:43.387711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:52.740057+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"207'44","reported_seq":790,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.990133+0000","last_change":"2024-03-11T21:09:48.540145+0000","last_active":"2024-03-11T21:10:06.990133+0000","last_peered":"2024-03-11T21:10:06.990133+0000","last_clean":"2024-03-11T21:10:06.990133+0000","last_became_active":"2024-03-11T21:00:48.469854+0000","last_became_peered":"2024-03-11T21:00:48.469854+0000","last_unstale":"2024-03-11T21:10:06.990133+0000","last_undegraded":"2024-03-11T21:10:06.990133+0000","last_fullsized":"2024-03-11T21:10:06.990133+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'44","last_scrub_stamp":"2024-03-11T21:09:48.539752+0000","last_deep_scrub":"161'28","last_deep_scrub_stamp":"2024-03-11T20:57:20.816278+0000","last_clean_scrub_stamp":"2024-03-11T21:09:48.539752+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:08.766428+0000","scrub_duration":59,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.4","version":"351'10505","reported_seq":10667,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:15.309640+0000","last_change":"2024-03-11T21:09:34.130627+0000","last_active":"2024-03-11T21:10:15.309640+0000","last_peered":"2024-03-11T21:10:15.309640+0000","last_clean":"2024-03-11T21:10:15.309640+0000","last_became_active":"2024-03-11T21:07:14.959964+0000","last_became_peered":"2024-03-11T21:07:14.959964+0000","last_unstale":"2024-03-11T21:10:15.309640+0000","last_undegraded":"2024-03-11T21:10:15.309640+0000","last_fullsized":"2024-03-11T21:10:15.309640+0000","mapping_epoch":306,"log_start":"328'5100","ondisk_log_start":"328'5100","created":306,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"343'8155","last_scrub_stamp":"2024-03-11T21:09:34.130396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:07:13.846944+0000","last_clean_scrub_stamp":"2024-03-11T21:09:34.130396+0000","objects_scrubbed":5417,"log_size":5405,"log_dups_size":0,"ondisk_log_size":5405,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:53.930420+0000","scrub_duration":2035,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":198770688,"num_objects":3033,"num_object_clones":0,"num_object_copies":9099,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3033,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10505,"num_write_kb":433216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":633,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.985266+0000","last_change":"2024-03-11T21:09:53.436563+0000","last_active":"2024-03-11T21:10:06.985266+0000","last_peered":"2024-03-11T21:10:06.985266+0000","last_clean":"2024-03-11T21:10:06.985266+0000","last_became_active":"2024-03-11T20:50:34.327139+0000","last_became_peered":"2024-03-11T20:50:34.327139+0000","last_unstale":"2024-03-11T21:10:06.985266+0000","last_undegraded":"2024-03-11T21:10:06.985266+0000","last_fullsized":"2024-03-11T21:10:06.985266+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:53.436482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:09:53.436482+0000","last_clean_scrub_stamp":"2024-03-11T21:09:53.436482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:16.930710+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":83,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.985813+0000","last_change":"2024-03-11T21:09:33.763673+0000","last_active":"2024-03-11T21:10:06.985813+0000","last_peered":"2024-03-11T21:10:06.985813+0000","last_clean":"2024-03-11T21:10:06.985813+0000","last_became_active":"2024-03-11T21:08:05.303226+0000","last_became_peered":"2024-03-11T21:08:05.303226+0000","last_unstale":"2024-03-11T21:10:06.985813+0000","last_undegraded":"2024-03-11T21:10:06.985813+0000","last_fullsized":"2024-03-11T21:10:06.985813+0000","mapping_epoch":319,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:33.763593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:57:17.447191+0000","last_clean_scrub_stamp":"2024-03-11T21:09:33.763593+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:38.695565+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"207'39","reported_seq":697,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.986535+0000","last_change":"2024-03-11T21:09:40.486417+0000","last_active":"2024-03-11T21:10:06.986535+0000","last_peered":"2024-03-11T21:10:06.986535+0000","last_clean":"2024-03-11T21:10:06.986535+0000","last_became_active":"2024-03-11T20:50:30.111819+0000","last_became_peered":"2024-03-11T20:50:30.111819+0000","last_unstale":"2024-03-11T21:10:06.986535+0000","last_undegraded":"2024-03-11T21:10:06.986535+0000","last_fullsized":"2024-03-11T21:10:06.986535+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'39","last_scrub_stamp":"2024-03-11T21:09:40.486313+0000","last_deep_scrub":"102'25","last_deep_scrub_stamp":"2024-03-11T20:51:35.187094+0000","last_clean_scrub_stamp":"2024-03-11T21:09:40.486313+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:56.721971+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":658,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.990047+0000","last_change":"2024-03-11T21:09:42.597032+0000","last_active":"2024-03-11T21:10:06.990047+0000","last_peered":"2024-03-11T21:10:06.990047+0000","last_clean":"2024-03-11T21:10:06.990047+0000","last_became_active":"2024-03-11T21:00:48.362389+0000","last_became_peered":"2024-03-11T21:00:48.362389+0000","last_unstale":"2024-03-11T21:10:06.990047+0000","last_undegraded":"2024-03-11T21:10:06.990047+0000","last_fullsized":"2024-03-11T21:10:06.990047+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:42.596113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:09:42.596113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:53.893451+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.5","version":"351'10661","reported_seq":10834,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:15.302977+0000","last_change":"2024-03-11T21:09:23.813448+0000","last_active":"2024-03-11T21:10:15.302977+0000","last_peered":"2024-03-11T21:10:15.302977+0000","last_clean":"2024-03-11T21:10:15.302977+0000","last_became_active":"2024-03-11T21:08:05.304912+0000","last_became_peered":"2024-03-11T21:08:05.304912+0000","last_unstale":"2024-03-11T21:10:15.302977+0000","last_undegraded":"2024-03-11T21:10:15.302977+0000","last_fullsized":"2024-03-11T21:10:15.302977+0000","mapping_epoch":319,"log_start":"328'5300","ondisk_log_start":"328'5300","created":306,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'7788","last_scrub_stamp":"2024-03-11T21:09:23.813149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:07:13.846944+0000","last_clean_scrub_stamp":"2024-03-11T21:09:23.813149+0000","objects_scrubbed":6007,"log_size":5361,"log_dups_size":0,"ondisk_log_size":5361,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:51.716947+0000","scrub_duration":1779,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":202702848,"num_objects":3093,"num_object_clones":0,"num_object_copies":9279,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3093,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10661,"num_write_kb":440128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":329,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.989001+0000","last_change":"2024-03-11T21:09:39.408174+0000","last_active":"2024-03-11T21:10:06.989001+0000","last_peered":"2024-03-11T21:10:06.989001+0000","last_clean":"2024-03-11T21:10:06.989001+0000","last_became_active":"2024-03-11T21:06:28.758451+0000","last_became_peered":"2024-03-11T21:06:28.758451+0000","last_unstale":"2024-03-11T21:10:06.989001+0000","last_undegraded":"2024-03-11T21:10:06.989001+0000","last_fullsized":"2024-03-11T21:10:06.989001+0000","mapping_epoch":297,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:39.408071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:55:58.888890+0000","last_clean_scrub_stamp":"2024-03-11T21:09:39.408071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:47.153652+0000","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":337,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.992221+0000","last_change":"2024-03-11T21:09:24.339301+0000","last_active":"2024-03-11T21:10:06.992221+0000","last_peered":"2024-03-11T21:10:06.992221+0000","last_clean":"2024-03-11T21:10:06.992221+0000","last_became_active":"2024-03-11T21:00:48.469015+0000","last_became_peered":"2024-03-11T21:00:48.469015+0000","last_unstale":"2024-03-11T21:10:06.992221+0000","last_undegraded":"2024-03-11T21:10:06.992221+0000","last_fullsized":"2024-03-11T21:10:06.992221+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:24.339169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:09:24.339169+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:49.809729+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"207'20","reported_seq":664,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.986592+0000","last_change":"2024-03-11T21:09:36.752271+0000","last_active":"2024-03-11T21:10:06.986592+0000","last_peered":"2024-03-11T21:10:06.986592+0000","last_clean":"2024-03-11T21:10:06.986592+0000","last_became_active":"2024-03-11T20:50:30.109549+0000","last_became_peered":"2024-03-11T20:50:30.109549+0000","last_unstale":"2024-03-11T21:10:06.986592+0000","last_undegraded":"2024-03-11T21:10:06.986592+0000","last_fullsized":"2024-03-11T21:10:06.986592+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'20","last_scrub_stamp":"2024-03-11T21:09:36.752172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:09:36.752172+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:46.990100+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":675,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.988731+0000","last_change":"2024-03-11T21:09:49.344698+0000","last_active":"2024-03-11T21:10:06.988731+0000","last_peered":"2024-03-11T21:10:06.988731+0000","last_clean":"2024-03-11T21:10:06.988731+0000","last_became_active":"2024-03-11T21:02:37.517748+0000","last_became_peered":"2024-03-11T21:02:37.517748+0000","last_unstale":"2024-03-11T21:10:06.988731+0000","last_undegraded":"2024-03-11T21:10:06.988731+0000","last_fullsized":"2024-03-11T21:10:06.988731+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:49.344594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:52:51.931295+0000","last_clean_scrub_stamp":"2024-03-11T21:09:49.344594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:17.641189+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.6","version":"351'10643","reported_seq":10813,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:15.473525+0000","last_change":"2024-03-11T21:09:23.070615+0000","last_active":"2024-03-11T21:10:15.473525+0000","last_peered":"2024-03-11T21:10:15.473525+0000","last_clean":"2024-03-11T21:10:15.473525+0000","last_became_active":"2024-03-11T21:07:14.952904+0000","last_became_peered":"2024-03-11T21:07:14.952904+0000","last_unstale":"2024-03-11T21:10:15.473525+0000","last_undegraded":"2024-03-11T21:10:15.473525+0000","last_fullsized":"2024-03-11T21:10:15.473525+0000","mapping_epoch":306,"log_start":"326'4700","ondisk_log_start":"326'4700","created":306,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'7752","last_scrub_stamp":"2024-03-11T21:09:23.070441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:07:13.846944+0000","last_clean_scrub_stamp":"2024-03-11T21:09:23.070441+0000","objects_scrubbed":6006,"log_size":5943,"log_dups_size":0,"ondisk_log_size":5943,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:44.728555+0000","scrub_duration":2283,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":199688192,"num_objects":3047,"num_object_clones":0,"num_object_copies":9141,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3047,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10643,"num_write_kb":438080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":99,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.995866+0000","last_change":"2024-03-11T21:07:46.870757+0000","last_active":"2024-03-11T21:10:06.995866+0000","last_peered":"2024-03-11T21:10:06.995866+0000","last_clean":"2024-03-11T21:10:06.995866+0000","last_became_active":"2024-03-11T21:07:34.504579+0000","last_became_peered":"2024-03-11T21:07:34.504579+0000","last_unstale":"2024-03-11T21:10:06.995866+0000","last_undegraded":"2024-03-11T21:10:06.995866+0000","last_fullsized":"2024-03-11T21:10:06.995866+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:07:46.870646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:00:14.030599+0000","last_clean_scrub_stamp":"2024-03-11T21:07:46.870646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:08.411879+0000","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":76,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.995807+0000","last_change":"2024-03-11T21:08:05.308620+0000","last_active":"2024-03-11T21:10:06.995807+0000","last_peered":"2024-03-11T21:10:06.995807+0000","last_clean":"2024-03-11T21:10:06.995807+0000","last_became_active":"2024-03-11T21:08:05.307984+0000","last_became_peered":"2024-03-11T21:08:05.307984+0000","last_unstale":"2024-03-11T21:10:06.995807+0000","last_undegraded":"2024-03-11T21:10:06.995807+0000","last_fullsized":"2024-03-11T21:10:06.995807+0000","mapping_epoch":319,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:07:57.816881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:07:57.816881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.f","version":"207'27","reported_seq":676,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.986405+0000","last_change":"2024-03-11T21:08:04.106067+0000","last_active":"2024-03-11T21:10:06.986405+0000","last_peered":"2024-03-11T21:10:06.986405+0000","last_clean":"2024-03-11T21:10:06.986405+0000","last_became_active":"2024-03-11T21:00:27.963119+0000","last_became_peered":"2024-03-11T21:00:27.963119+0000","last_unstale":"2024-03-11T21:10:06.986405+0000","last_undegraded":"2024-03-11T21:10:06.986405+0000","last_fullsized":"2024-03-11T21:10:06.986405+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'27","last_scrub_stamp":"2024-03-11T21:08:04.105922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:08:04.105922+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:10.585032+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":631,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-11T21:10:06.985475+0000","last_change":"2024-03-11T21:09:48.842757+0000","last_active":"2024-03-11T21:10:06.985475+0000","last_peered":"2024-03-11T21:10:06.985475+0000","last_clean":"2024-03-11T21:10:06.985475+0000","last_became_active":"2024-03-11T20:50:32.652854+0000","last_became_peered":"2024-03-11T20:50:32.652854+0000","last_unstale":"2024-03-11T21:10:06.985475+0000","last_undegraded":"2024-03-11T21:10:06.985475+0000","last_fullsized":"2024-03-11T21:10:06.985475+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:48.842645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:59:00.946094+0000","last_clean_scrub_stamp":"2024-03-11T21:09:48.842645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:54.267900+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":3003252760,"num_objects":45827,"num_object_clones":0,"num_object_copies":137481,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45827,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":172193,"num_write_kb":6976577,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":9091362816,"data_stored":9091350600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90393,"ondisk_log_size":90393,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":20,"num_read_kb":20,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1822,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1121,"num_read_kb":878,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":171,"num_bytes_recovered":2228,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3632,"internal_metadata":0},"log_size":778,"ondisk_log_size":778,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":126,"ondisk_log_size":126,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1053,"num_read_kb":1048,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":68,"seq":292057776377,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1198804,"kb_used_data":708108,"kb_used_omap":1,"kb_used_meta":490686,"kb_avail":92546348,"statfs":{"total":95995035648,"available":94767460352,"internally_reserved":0,"allocated":725102592,"data_stored":724753467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":502462922},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":57,"seq":244813136131,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1625160,"kb_used_data":1255232,"kb_used_omap":1,"kb_used_meta":369918,"kb_avail":92119992,"statfs":{"total":95995035648,"available":94330871808,"internally_reserved":0,"allocated":1285357568,"data_stored":1285020765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":378796487},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":211,"seq":906238099571,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":929736,"kb_used_data":574144,"kb_used_omap":7,"kb_used_meta":355576,"kb_avail":92815416,"statfs":{"total":95995035648,"available":95042985984,"internally_reserved":0,"allocated":587923456,"data_stored":587586331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8078,"internal_metadata":364109938},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738688,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2438544,"kb_used_data":1978184,"kb_used_omap":1,"kb_used_meta":460158,"kb_avail":91306608,"statfs":{"total":95995035648,"available":93497966592,"internally_reserved":0,"allocated":2025660416,"data_stored":2025314353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":471202245},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378924,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1447672,"kb_used_data":1059568,"kb_used_omap":1,"kb_used_meta":387966,"kb_avail":92297480,"statfs":{"total":95995035648,"available":94512619520,"internally_reserved":0,"allocated":1084997632,"data_stored":1084675739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":397277637},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":28,"seq":120259084577,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1438780,"kb_used_data":889780,"kb_used_omap":1,"kb_used_meta":548990,"kb_avail":92306372,"statfs":{"total":95995035648,"available":94521724928,"internally_reserved":0,"allocated":911134720,"data_stored":910809107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":562166216},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":204,"seq":876173328503,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1821156,"kb_used_data":1549980,"kb_used_omap":4,"kb_used_meta":271163,"kb_avail":91923996,"statfs":{"total":95995035648,"available":94130171904,"internally_reserved":0,"allocated":1587179520,"data_stored":1586814361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4451,"internal_metadata":277671581},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":44,"seq":188978561292,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1408572,"kb_used_data":984372,"kb_used_omap":1,"kb_used_meta":424062,"kb_avail":92336580,"statfs":{"total":95995035648,"available":94552657920,"internally_reserved":0,"allocated":1007996928,"data_stored":1007670861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":434239941},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3632,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1990922240,"data_stored":1990918168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":576192512,"data_stored":576192512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1083834368,"data_stored":1083834368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":906231808,"data_stored":906231808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1568542720,"data_stored":1568538648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":988020736,"data_stored":988020736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1258491904,"data_stored":1258487832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":719126528,"data_stored":719126528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T21:10:25.071 INFO:tasks.thrashosds.thrasher:Setting 4.1d to [0, 3, 4, 2, 6, 1] 2024-03-11T21:10:25.072 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '4.1d', '0', '3', '4', '2', '6', '1'] 2024-03-11T21:10:25.072 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd pg-upmap-items 4.1d 0 3 4 2 6 1 2024-03-11T21:10:25.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:25.073 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:10:25.175 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:10:25.218 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:10:25.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:25.221 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:10:25.322 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:10:25.363 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:10:25.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:25.367 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:10:25.468 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:10:25.478 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:25 smithi146 bash[23809]: cluster 2024-03-11T21:10:24.322973+0000 mgr.y (mgr.24380) 726 : cluster 0 pgmap v819: 148 pgs: 148 active+clean; 2.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 921 B/s rd, 0 B/s wr, 992 op/s 2024-03-11T21:10:25.494 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:10:25.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:25.499 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:10:25.600 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:25.618 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:25 smithi023 bash[21372]: cluster 2024-03-11T21:10:24.322973+0000 mgr.y (mgr.24380) 726 : cluster 0 pgmap v819: 148 pgs: 148 active+clean; 2.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 921 B/s rd, 0 B/s wr, 992 op/s 2024-03-11T21:10:25.618 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:25 smithi023 bash[29146]: cluster 2024-03-11T21:10:24.322973+0000 mgr.y (mgr.24380) 726 : cluster 0 pgmap v819: 148 pgs: 148 active+clean; 2.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 921 B/s rd, 0 B/s wr, 992 op/s 2024-03-11T21:10:25.638 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:25.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:25.644 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:10:25.744 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:10:25.770 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:10:25.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:25.775 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:10:25.875 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:26.008 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:10:25 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:10:26.632 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:26.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:26.636 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:10:26.737 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:10:26.778 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:10:26.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:26.781 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:10:26.882 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:10:26.907 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:10:26.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:26.910 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:10:27.011 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:10:27.038 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:10:27.121 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T21:10:27.142 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd unset noscrub 2024-03-11T21:10:27.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:27.264 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:10:27.365 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:10:27.410 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:10:27.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:27.420 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:10:27.460 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:27 smithi023 bash[29146]: audit 2024-03-11T21:10:25.833480+0000 mgr.y (mgr.24380) 727 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:10:27.460 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:27 smithi023 bash[29146]: cluster 2024-03-11T21:10:26.324118+0000 mgr.y (mgr.24380) 728 : cluster 0 pgmap v820: 148 pgs: 148 active+clean; 2.4 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 957 op/s 2024-03-11T21:10:27.460 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:27 smithi023 bash[29146]: cluster 2024-03-11T21:10:26.504034+0000 osd.5 (osd.5) 409 : cluster 0 purged_snaps scrub starts 2024-03-11T21:10:27.460 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:27 smithi023 bash[29146]: cluster 2024-03-11T21:10:26.504086+0000 osd.5 (osd.5) 410 : cluster 0 purged_snaps scrub ok 2024-03-11T21:10:27.521 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:10:27.750 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:27 smithi146 bash[23809]: audit 2024-03-11T21:10:25.833480+0000 mgr.y (mgr.24380) 727 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:10:27.751 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:27 smithi146 bash[23809]: cluster 2024-03-11T21:10:26.324118+0000 mgr.y (mgr.24380) 728 : cluster 0 pgmap v820: 148 pgs: 148 active+clean; 2.4 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 957 op/s 2024-03-11T21:10:27.751 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:27 smithi146 bash[23809]: cluster 2024-03-11T21:10:26.504034+0000 osd.5 (osd.5) 409 : cluster 0 purged_snaps scrub starts 2024-03-11T21:10:27.751 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:27 smithi146 bash[23809]: cluster 2024-03-11T21:10:26.504086+0000 osd.5 (osd.5) 410 : cluster 0 purged_snaps scrub ok 2024-03-11T21:10:27.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:27 smithi023 bash[21372]: audit 2024-03-11T21:10:25.833480+0000 mgr.y (mgr.24380) 727 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:10:27.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:27 smithi023 bash[21372]: cluster 2024-03-11T21:10:26.324118+0000 mgr.y (mgr.24380) 728 : cluster 0 pgmap v820: 148 pgs: 148 active+clean; 2.4 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 957 op/s 2024-03-11T21:10:27.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:27 smithi023 bash[21372]: cluster 2024-03-11T21:10:26.504034+0000 osd.5 (osd.5) 409 : cluster 0 purged_snaps scrub starts 2024-03-11T21:10:27.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:27 smithi023 bash[21372]: cluster 2024-03-11T21:10:26.504086+0000 osd.5 (osd.5) 410 : cluster 0 purged_snaps scrub ok 2024-03-11T21:10:27.833 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:10:27.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:27.838 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:10:27.938 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:27.979 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:27.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:27.982 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:10:28.083 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:10:28.107 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:10:28.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:28.110 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:10:28.210 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:10:28.242 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:10:28.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:28.246 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:10:28.347 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:10:28.372 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:10:28.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:28.376 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:10:28.476 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:28.483 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:10:28 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:10:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:10:28.503 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:28.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:28.509 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:10:28.609 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:10:28.634 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-03-11T21:10:28.634 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-03-11T21:10:28.634 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 30, 2024-03-11T21:10:28.634 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-03-11T21:10:28.634 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-03-11T21:10:28.650 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:28 smithi146 bash[23809]: cluster 2024-03-11T21:10:27.462801+0000 mon.a (mon.0) 1725 : cluster 0 osdmap e353: 8 total, 8 up, 8 in 2024-03-11T21:10:28.650 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:28 smithi146 bash[23809]: cluster 2024-03-11T21:10:27.498682+0000 osd.3 (osd.3) 406 : cluster 0 purged_snaps scrub starts 2024-03-11T21:10:28.650 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:28 smithi146 bash[23809]: cluster 2024-03-11T21:10:27.498747+0000 osd.3 (osd.3) 407 : cluster 0 purged_snaps scrub ok 2024-03-11T21:10:28.732 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:10:28.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:28.738 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:10:28.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:28 smithi023 bash[21372]: cluster 2024-03-11T21:10:27.462801+0000 mon.a (mon.0) 1725 : cluster 0 osdmap e353: 8 total, 8 up, 8 in 2024-03-11T21:10:28.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:28 smithi023 bash[21372]: cluster 2024-03-11T21:10:27.498682+0000 osd.3 (osd.3) 406 : cluster 0 purged_snaps scrub starts 2024-03-11T21:10:28.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:28 smithi023 bash[21372]: cluster 2024-03-11T21:10:27.498747+0000 osd.3 (osd.3) 407 : cluster 0 purged_snaps scrub ok 2024-03-11T21:10:28.779 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:28 smithi023 bash[29146]: cluster 2024-03-11T21:10:27.462801+0000 mon.a (mon.0) 1725 : cluster 0 osdmap e353: 8 total, 8 up, 8 in 2024-03-11T21:10:28.779 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:28 smithi023 bash[29146]: cluster 2024-03-11T21:10:27.498682+0000 osd.3 (osd.3) 406 : cluster 0 purged_snaps scrub starts 2024-03-11T21:10:28.779 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:28 smithi023 bash[29146]: cluster 2024-03-11T21:10:27.498747+0000 osd.3 (osd.3) 407 : cluster 0 purged_snaps scrub ok 2024-03-11T21:10:28.840 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:28.870 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:28.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:28.875 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:10:28.975 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:29.006 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:29.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:29.010 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:10:29.111 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:10:29.135 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:10:29.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:29.137 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:10:29.239 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:10:29.266 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:10:29.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:29.270 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:10:29.371 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:10:29.399 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:10:29.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:29.402 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:10:29.503 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:10:29.530 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:10:29.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:29.532 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:10:29.633 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:10:29.641 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:29 smithi023 bash[21372]: cluster 2024-03-11T21:10:28.324529+0000 mgr.y (mgr.24380) 729 : cluster 0 pgmap v822: 148 pgs: 148 active+clean; 2.4 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 847 op/s 2024-03-11T21:10:29.642 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:29 smithi023 bash[21372]: cluster 2024-03-11T21:10:28.506682+0000 mon.a (mon.0) 1726 : cluster 0 osdmap e354: 8 total, 8 up, 8 in 2024-03-11T21:10:29.642 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:29 smithi023 bash[29146]: cluster 2024-03-11T21:10:28.324529+0000 mgr.y (mgr.24380) 729 : cluster 0 pgmap v822: 148 pgs: 148 active+clean; 2.4 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 847 op/s 2024-03-11T21:10:29.642 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:29 smithi023 bash[29146]: cluster 2024-03-11T21:10:28.506682+0000 mon.a (mon.0) 1726 : cluster 0 osdmap e354: 8 total, 8 up, 8 in 2024-03-11T21:10:29.662 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:10:29.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:29.666 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:10:29.757 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:29 smithi146 bash[23809]: cluster 2024-03-11T21:10:28.324529+0000 mgr.y (mgr.24380) 729 : cluster 0 pgmap v822: 148 pgs: 148 active+clean; 2.4 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 847 op/s 2024-03-11T21:10:29.758 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:29 smithi146 bash[23809]: cluster 2024-03-11T21:10:28.506682+0000 mon.a (mon.0) 1726 : cluster 0 osdmap e354: 8 total, 8 up, 8 in 2024-03-11T21:10:29.767 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:10:29.801 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:10:29.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:29.808 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:10:29.909 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:10:29.917 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:10:30.093 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:10:30.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:30.096 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:10:30.162 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-03-11T21:10:30.197 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:30.271 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:30.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:30.279 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:10:30.379 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:30.412 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:30.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:30.421 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:10:30.521 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:30.547 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:30.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:30.549 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:10:30.650 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:10:30.686 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:10:30.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:30.692 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:10:30.792 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:10:30.815 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:10:30.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:30.819 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:10:30.920 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:30.955 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:30.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:30.961 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:10:31.062 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:10:31.087 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:10:31.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:31.091 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:10:31.193 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:10:31.232 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:10:31.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:31.238 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:10:31.339 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:31.363 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:31.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:31.366 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:10:31.467 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:31.490 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:31.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:31.497 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:10:31.598 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:10:31.619 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:31 smithi146 bash[23809]: cluster 2024-03-11T21:10:30.325558+0000 mgr.y (mgr.24380) 730 : cluster 0 pgmap v824: 148 pgs: 148 active+clean; 2.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 953 op/s 2024-03-11T21:10:31.640 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:10:31.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:31.644 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:10:31.646 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:31 smithi023 bash[21372]: cluster 2024-03-11T21:10:30.325558+0000 mgr.y (mgr.24380) 730 : cluster 0 pgmap v824: 148 pgs: 148 active+clean; 2.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 953 op/s 2024-03-11T21:10:31.647 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:31 smithi023 bash[29146]: cluster 2024-03-11T21:10:30.325558+0000 mgr.y (mgr.24380) 730 : cluster 0 pgmap v824: 148 pgs: 148 active+clean; 2.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 953 op/s 2024-03-11T21:10:31.745 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:10:31.833 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:10:31.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:31.836 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:10:31.936 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:10:31.962 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:10:31.962 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:10:31.969 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:10:31.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:31.978 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:10:32.079 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:10:32.104 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:10:32.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:32.110 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:10:32.211 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:10:32.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:32.252 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:10:32.253 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:10:32.353 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:10:32.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:32.383 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:10:32.383 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:10:32.484 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:10:32.517 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:10:32.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:32.520 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:10:32.621 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:10:32.646 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:10:32.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:32.651 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:10:32.752 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:10:32.780 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:10:32.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:32.783 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:10:32.884 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:32.923 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:32.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:32.928 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:10:33.029 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:33.074 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:33.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:33.083 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:10:33.184 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:10:33.212 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:10:33.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:33.214 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:10:33.315 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:10:33.340 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:10:33.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:33.342 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:10:33.443 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:33.470 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:33.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:33.481 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:10:33.528 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:33 smithi023 bash[21372]: cluster 2024-03-11T21:10:31.968632+0000 osd.7 (osd.7) 451 : cluster 0 purged_snaps scrub starts 2024-03-11T21:10:33.528 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:33 smithi023 bash[21372]: cluster 2024-03-11T21:10:31.968703+0000 osd.7 (osd.7) 452 : cluster 0 purged_snaps scrub ok 2024-03-11T21:10:33.528 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:33 smithi023 bash[21372]: cluster 2024-03-11T21:10:32.326366+0000 mgr.y (mgr.24380) 731 : cluster 0 pgmap v825: 148 pgs: 148 active+clean; 2.1 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.23k op/s 2024-03-11T21:10:33.529 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:33 smithi023 bash[21372]: cluster 2024-03-11T21:10:32.365011+0000 mon.a (mon.0) 1727 : cluster 0 osdmap e355: 8 total, 8 up, 8 in 2024-03-11T21:10:33.529 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:33 smithi023 bash[29146]: cluster 2024-03-11T21:10:31.968632+0000 osd.7 (osd.7) 451 : cluster 0 purged_snaps scrub starts 2024-03-11T21:10:33.529 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:33 smithi023 bash[29146]: cluster 2024-03-11T21:10:31.968703+0000 osd.7 (osd.7) 452 : cluster 0 purged_snaps scrub ok 2024-03-11T21:10:33.529 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:33 smithi023 bash[29146]: cluster 2024-03-11T21:10:32.326366+0000 mgr.y (mgr.24380) 731 : cluster 0 pgmap v825: 148 pgs: 148 active+clean; 2.1 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.23k op/s 2024-03-11T21:10:33.529 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:33 smithi023 bash[29146]: cluster 2024-03-11T21:10:32.365011+0000 mon.a (mon.0) 1727 : cluster 0 osdmap e355: 8 total, 8 up, 8 in 2024-03-11T21:10:33.581 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:33.605 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:33.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:33.607 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:10:33.607 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:33 smithi146 bash[23809]: cluster 2024-03-11T21:10:31.968632+0000 osd.7 (osd.7) 451 : cluster 0 purged_snaps scrub starts 2024-03-11T21:10:33.607 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:33 smithi146 bash[23809]: cluster 2024-03-11T21:10:31.968703+0000 osd.7 (osd.7) 452 : cluster 0 purged_snaps scrub ok 2024-03-11T21:10:33.607 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:33 smithi146 bash[23809]: cluster 2024-03-11T21:10:32.326366+0000 mgr.y (mgr.24380) 731 : cluster 0 pgmap v825: 148 pgs: 148 active+clean; 2.1 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.23k op/s 2024-03-11T21:10:33.607 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:33 smithi146 bash[23809]: cluster 2024-03-11T21:10:32.365011+0000 mon.a (mon.0) 1727 : cluster 0 osdmap e355: 8 total, 8 up, 8 in 2024-03-11T21:10:33.709 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:10:33.733 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:10:33.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:33.734 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:10:33.835 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:10:33.862 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:10:33.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:33.865 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:10:33.968 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:34.000 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:34.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:34.004 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:10:34.105 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:10:34.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:34.137 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:10:34.137 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:10:34.238 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:10:34.260 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:10:34.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:34.266 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:10:34.367 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:34.399 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:34.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:34.401 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:10:34.502 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:10:34.541 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:10:34.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:34.545 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:10:34.646 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:10:34.670 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:10:34.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:34.675 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:10:34.776 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:34.798 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:34.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:34.805 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:10:34.906 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:10:34.938 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:10:34.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:34.943 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:10:35.040 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:10:35.044 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:10:35.071 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:10:35.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:35.082 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:10:35.184 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:10:35.216 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:10:35.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:35.220 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:10:35.320 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:10:35.351 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:10:35.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:35.359 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:10:35.460 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:10:35.486 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:10:35.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:35.490 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:10:35.542 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:35 smithi023 bash[21372]: cluster 2024-03-11T21:10:34.326768+0000 mgr.y (mgr.24380) 732 : cluster 0 pgmap v827: 148 pgs: 148 active+clean; 2.1 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 745 B/s rd, 0 B/s wr, 764 op/s 2024-03-11T21:10:35.543 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:35 smithi023 bash[29146]: cluster 2024-03-11T21:10:34.326768+0000 mgr.y (mgr.24380) 732 : cluster 0 pgmap v827: 148 pgs: 148 active+clean; 2.1 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 745 B/s rd, 0 B/s wr, 764 op/s 2024-03-11T21:10:35.591 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:10:35.619 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:10:35.619 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:35 smithi146 bash[23809]: cluster 2024-03-11T21:10:34.326768+0000 mgr.y (mgr.24380) 732 : cluster 0 pgmap v827: 148 pgs: 148 active+clean; 2.1 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 745 B/s rd, 0 B/s wr, 764 op/s 2024-03-11T21:10:35.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:35.620 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:10:35.721 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:10:35.753 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:10:35.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:35.758 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:10:35.859 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:10:35.892 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:10:35.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:35.899 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:10:36.000 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:10:36.008 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:10:35 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:10:36.031 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:10:36.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:36.034 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:10:36.135 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:10:36.585 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:10:36.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:36.588 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:10:36.688 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:10:36.697 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:36 smithi146 bash[23809]: audit 2024-03-11T21:10:36.012064+0000 mon.a (mon.0) 1728 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:10:36.708 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:10:36.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:36.709 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:10:36.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:36 smithi023 bash[21372]: audit 2024-03-11T21:10:36.012064+0000 mon.a (mon.0) 1728 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:10:36.778 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:36 smithi023 bash[29146]: audit 2024-03-11T21:10:36.012064+0000 mon.a (mon.0) 1728 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:10:36.810 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:10:36.841 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:10:36.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:36.843 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:10:36.944 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:10:36.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:36.969 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:10:36.969 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:10:37.070 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:10:37.093 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:10:37.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:37.095 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:10:37.197 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:10:37.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:37.223 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:10:37.224 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:10:37.325 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:10:37.351 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:10:37.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:37.353 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:10:37.397 INFO:teuthology.orchestra.run.smithi023.stderr:set 4.1d pg_upmap_items mapping to [0->3,4->2,6->1] 2024-03-11T21:10:37.454 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:37.494 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:37.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:37.498 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:10:37.528 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:37 smithi023 bash[21372]: audit 2024-03-11T21:10:35.841166+0000 mgr.y (mgr.24380) 733 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:10:37.528 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:37 smithi023 bash[21372]: cluster 2024-03-11T21:10:36.228201+0000 osd.0 (osd.0) 311 : cluster 0 purged_snaps scrub starts 2024-03-11T21:10:37.528 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:37 smithi023 bash[21372]: cluster 2024-03-11T21:10:36.228258+0000 osd.0 (osd.0) 312 : cluster 0 purged_snaps scrub ok 2024-03-11T21:10:37.529 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:37 smithi023 bash[21372]: cluster 2024-03-11T21:10:36.327624+0000 mgr.y (mgr.24380) 734 : cluster 0 pgmap v828: 148 pgs: 148 active+clean; 1.8 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.23k op/s 2024-03-11T21:10:37.529 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:37 smithi023 bash[21372]: cluster 2024-03-11T21:10:36.381267+0000 mon.a (mon.0) 1729 : cluster 0 osdmap e356: 8 total, 8 up, 8 in 2024-03-11T21:10:37.529 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:37 smithi023 bash[21372]: audit 2024-03-11T21:10:36.852074+0000 mon.c (mon.2) 86 : audit 1 from='client.? 172.21.15.23:0/3482616095' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "4.1d", "id": [0, 3, 4, 2, 6, 1]}]: dispatch 2024-03-11T21:10:37.529 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:37 smithi023 bash[21372]: audit 2024-03-11T21:10:36.854748+0000 mon.a (mon.0) 1730 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "4.1d", "id": [0, 3, 4, 2, 6, 1]}]: dispatch 2024-03-11T21:10:37.529 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:37 smithi023 bash[29146]: audit 2024-03-11T21:10:35.841166+0000 mgr.y (mgr.24380) 733 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:10:37.529 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:37 smithi023 bash[29146]: cluster 2024-03-11T21:10:36.228201+0000 osd.0 (osd.0) 311 : cluster 0 purged_snaps scrub starts 2024-03-11T21:10:37.529 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:37 smithi023 bash[29146]: cluster 2024-03-11T21:10:36.228258+0000 osd.0 (osd.0) 312 : cluster 0 purged_snaps scrub ok 2024-03-11T21:10:37.529 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:37 smithi023 bash[29146]: cluster 2024-03-11T21:10:36.327624+0000 mgr.y (mgr.24380) 734 : cluster 0 pgmap v828: 148 pgs: 148 active+clean; 1.8 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.23k op/s 2024-03-11T21:10:37.529 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:37 smithi023 bash[29146]: cluster 2024-03-11T21:10:36.381267+0000 mon.a (mon.0) 1729 : cluster 0 osdmap e356: 8 total, 8 up, 8 in 2024-03-11T21:10:37.529 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:37 smithi023 bash[29146]: audit 2024-03-11T21:10:36.852074+0000 mon.c (mon.2) 86 : audit 1 from='client.? 172.21.15.23:0/3482616095' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "4.1d", "id": [0, 3, 4, 2, 6, 1]}]: dispatch 2024-03-11T21:10:37.529 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:37 smithi023 bash[29146]: audit 2024-03-11T21:10:36.854748+0000 mon.a (mon.0) 1730 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "4.1d", "id": [0, 3, 4, 2, 6, 1]}]: dispatch 2024-03-11T21:10:37.598 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:37.633 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:37.635 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:37 smithi146 bash[23809]: audit 2024-03-11T21:10:35.841166+0000 mgr.y (mgr.24380) 733 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:10:37.636 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:37 smithi146 bash[23809]: cluster 2024-03-11T21:10:36.228201+0000 osd.0 (osd.0) 311 : cluster 0 purged_snaps scrub starts 2024-03-11T21:10:37.636 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:37 smithi146 bash[23809]: cluster 2024-03-11T21:10:36.228258+0000 osd.0 (osd.0) 312 : cluster 0 purged_snaps scrub ok 2024-03-11T21:10:37.636 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:37 smithi146 bash[23809]: cluster 2024-03-11T21:10:36.327624+0000 mgr.y (mgr.24380) 734 : cluster 0 pgmap v828: 148 pgs: 148 active+clean; 1.8 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.23k op/s 2024-03-11T21:10:37.636 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:37 smithi146 bash[23809]: cluster 2024-03-11T21:10:36.381267+0000 mon.a (mon.0) 1729 : cluster 0 osdmap e356: 8 total, 8 up, 8 in 2024-03-11T21:10:37.636 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:37 smithi146 bash[23809]: audit 2024-03-11T21:10:36.852074+0000 mon.c (mon.2) 86 : audit 1 from='client.? 172.21.15.23:0/3482616095' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "4.1d", "id": [0, 3, 4, 2, 6, 1]}]: dispatch 2024-03-11T21:10:37.636 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:37 smithi146 bash[23809]: audit 2024-03-11T21:10:36.854748+0000 mon.a (mon.0) 1730 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "4.1d", "id": [0, 3, 4, 2, 6, 1]}]: dispatch 2024-03-11T21:10:37.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:37.637 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:10:37.738 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:10:37.772 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:10:37.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:37.776 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:10:37.877 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:10:37.898 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:10:37.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:37.900 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:10:38.001 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:10:38.025 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:10:38.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:38.029 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:10:38.129 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:10:38.158 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:10:38.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:38.164 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:10:38.265 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:10:38.306 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:10:38.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:38.311 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:10:38.403 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:10:38 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:10:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:10:38.412 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:10:38.444 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:10:38.448 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:38 smithi023 bash[21372]: audit 2024-03-11T21:10:37.386757+0000 mon.a (mon.0) 1731 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "4.1d", "id": [0, 3, 4, 2, 6, 1]}]': finished 2024-03-11T21:10:38.449 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:38 smithi023 bash[21372]: cluster 2024-03-11T21:10:37.404999+0000 mon.a (mon.0) 1732 : cluster 0 osdmap e357: 8 total, 8 up, 8 in 2024-03-11T21:10:38.449 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:38 smithi023 bash[29146]: audit 2024-03-11T21:10:37.386757+0000 mon.a (mon.0) 1731 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "4.1d", "id": [0, 3, 4, 2, 6, 1]}]': finished 2024-03-11T21:10:38.449 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:38 smithi023 bash[29146]: cluster 2024-03-11T21:10:37.404999+0000 mon.a (mon.0) 1732 : cluster 0 osdmap e357: 8 total, 8 up, 8 in 2024-03-11T21:10:38.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:38.450 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:10:38.551 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:10:38.581 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:10:38.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:38.584 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:10:38.685 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:10:38.758 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:38 smithi146 bash[23809]: audit 2024-03-11T21:10:37.386757+0000 mon.a (mon.0) 1731 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "4.1d", "id": [0, 3, 4, 2, 6, 1]}]': finished 2024-03-11T21:10:38.758 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:38 smithi146 bash[23809]: cluster 2024-03-11T21:10:37.404999+0000 mon.a (mon.0) 1732 : cluster 0 osdmap e357: 8 total, 8 up, 8 in 2024-03-11T21:10:38.855 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:10:38.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:38.860 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:10:38.961 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:38.991 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:38.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:38.996 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:10:39.097 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:10:39.115 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:10:39.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:39.117 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:10:39.218 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:39.235 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:39.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:39.237 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:10:39.338 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:10:39.375 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:10:39.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:39.383 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:10:39.484 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:10:39.507 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:39 smithi146 bash[23809]: cluster 2024-03-11T21:10:38.328048+0000 mgr.y (mgr.24380) 735 : cluster 0 pgmap v831: 148 pgs: 148 active+clean; 1.8 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 4.3 KiB/s rd, 0 B/s wr, 813 op/s 2024-03-11T21:10:39.508 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:39 smithi146 bash[23809]: cluster 2024-03-11T21:10:38.422253+0000 mon.a (mon.0) 1733 : cluster 0 osdmap e358: 8 total, 8 up, 8 in 2024-03-11T21:10:39.512 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:10:39.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:39.517 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:10:39.618 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:10:39.647 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:10:39.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:39.652 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:10:39.709 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:39 smithi023 bash[21372]: cluster 2024-03-11T21:10:38.328048+0000 mgr.y (mgr.24380) 735 : cluster 0 pgmap v831: 148 pgs: 148 active+clean; 1.8 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 4.3 KiB/s rd, 0 B/s wr, 813 op/s 2024-03-11T21:10:39.709 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:39 smithi023 bash[21372]: cluster 2024-03-11T21:10:38.422253+0000 mon.a (mon.0) 1733 : cluster 0 osdmap e358: 8 total, 8 up, 8 in 2024-03-11T21:10:39.709 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:39 smithi023 bash[29146]: cluster 2024-03-11T21:10:38.328048+0000 mgr.y (mgr.24380) 735 : cluster 0 pgmap v831: 148 pgs: 148 active+clean; 1.8 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 4.3 KiB/s rd, 0 B/s wr, 813 op/s 2024-03-11T21:10:39.710 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:39 smithi023 bash[29146]: cluster 2024-03-11T21:10:38.422253+0000 mon.a (mon.0) 1733 : cluster 0 osdmap e358: 8 total, 8 up, 8 in 2024-03-11T21:10:39.752 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:10:39.782 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:10:39.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:39.784 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:10:39.885 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:10:39.914 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:10:39.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:39.919 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:10:40.020 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:40.039 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:40.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:40.041 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:10:40.141 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:10:40.164 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:10:40.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:40.166 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:10:40.267 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:10:40.312 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:10:40.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:40.317 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:10:40.417 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:10:40.718 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:10:40.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:40.720 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:10:40.821 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:10:40.850 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:10:40.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:40.862 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:10:40.963 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:40.990 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:40.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:40.999 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:10:41.100 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:10:41.122 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:10:41.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:41.131 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:10:41.231 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:10:41.260 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:10:41.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:41.266 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:10:41.367 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:10:41.392 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:10:41.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:41.395 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:10:41.495 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:10:41.509 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:41 smithi023 bash[21372]: cluster 2024-03-11T21:10:39.819504+0000 osd.2 (osd.2) 255 : cluster 0 purged_snaps scrub starts 2024-03-11T21:10:41.509 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:41 smithi023 bash[21372]: cluster 2024-03-11T21:10:39.819575+0000 osd.2 (osd.2) 256 : cluster 0 purged_snaps scrub ok 2024-03-11T21:10:41.509 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:41 smithi023 bash[21372]: cluster 2024-03-11T21:10:40.328825+0000 mgr.y (mgr.24380) 736 : cluster 0 pgmap v833: 148 pgs: 148 active+clean; 1.6 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 1.23k op/s 2024-03-11T21:10:41.510 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:41 smithi023 bash[21372]: cluster 2024-03-11T21:10:40.418975+0000 mon.a (mon.0) 1734 : cluster 0 osdmap e359: 8 total, 8 up, 8 in 2024-03-11T21:10:41.510 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:41 smithi023 bash[21372]: cluster 2024-03-11T21:10:40.458889+0000 osd.1 (osd.1) 217 : cluster 0 purged_snaps scrub starts 2024-03-11T21:10:41.510 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:41 smithi023 bash[21372]: cluster 2024-03-11T21:10:40.458992+0000 osd.1 (osd.1) 218 : cluster 0 purged_snaps scrub ok 2024-03-11T21:10:41.510 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:41 smithi023 bash[29146]: cluster 2024-03-11T21:10:39.819504+0000 osd.2 (osd.2) 255 : cluster 0 purged_snaps scrub starts 2024-03-11T21:10:41.510 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:41 smithi023 bash[29146]: cluster 2024-03-11T21:10:39.819575+0000 osd.2 (osd.2) 256 : cluster 0 purged_snaps scrub ok 2024-03-11T21:10:41.510 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:41 smithi023 bash[29146]: cluster 2024-03-11T21:10:40.328825+0000 mgr.y (mgr.24380) 736 : cluster 0 pgmap v833: 148 pgs: 148 active+clean; 1.6 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 1.23k op/s 2024-03-11T21:10:41.510 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:41 smithi023 bash[29146]: cluster 2024-03-11T21:10:40.418975+0000 mon.a (mon.0) 1734 : cluster 0 osdmap e359: 8 total, 8 up, 8 in 2024-03-11T21:10:41.510 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:41 smithi023 bash[29146]: cluster 2024-03-11T21:10:40.458889+0000 osd.1 (osd.1) 217 : cluster 0 purged_snaps scrub starts 2024-03-11T21:10:41.510 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:41 smithi023 bash[29146]: cluster 2024-03-11T21:10:40.458992+0000 osd.1 (osd.1) 218 : cluster 0 purged_snaps scrub ok 2024-03-11T21:10:41.533 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:10:41.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:41.538 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:10:41.638 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:10:41.667 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:10:41.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:41.671 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:10:41.697 INFO:teuthology.orchestra.run.smithi023.stdout:osd.0: {} 2024-03-11T21:10:41.725 INFO:teuthology.orchestra.run.smithi023.stdout:osd.1: {} 2024-03-11T21:10:41.759 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:41 smithi146 bash[23809]: cluster 2024-03-11T21:10:39.819504+0000 osd.2 (osd.2) 255 : cluster 0 purged_snaps scrub starts 2024-03-11T21:10:41.759 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:41 smithi146 bash[23809]: cluster 2024-03-11T21:10:39.819575+0000 osd.2 (osd.2) 256 : cluster 0 purged_snaps scrub ok 2024-03-11T21:10:41.759 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:41 smithi146 bash[23809]: cluster 2024-03-11T21:10:40.328825+0000 mgr.y (mgr.24380) 736 : cluster 0 pgmap v833: 148 pgs: 148 active+clean; 1.6 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 1.23k op/s 2024-03-11T21:10:41.759 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:41 smithi146 bash[23809]: cluster 2024-03-11T21:10:40.418975+0000 mon.a (mon.0) 1734 : cluster 0 osdmap e359: 8 total, 8 up, 8 in 2024-03-11T21:10:41.759 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:41 smithi146 bash[23809]: cluster 2024-03-11T21:10:40.458889+0000 osd.1 (osd.1) 217 : cluster 0 purged_snaps scrub starts 2024-03-11T21:10:41.759 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:41 smithi146 bash[23809]: cluster 2024-03-11T21:10:40.458992+0000 osd.1 (osd.1) 218 : cluster 0 purged_snaps scrub ok 2024-03-11T21:10:41.760 INFO:teuthology.orchestra.run.smithi023.stdout:osd.2: {} 2024-03-11T21:10:41.772 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:41.795 INFO:teuthology.orchestra.run.smithi023.stdout:osd.3: {} 2024-03-11T21:10:41.805 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:41.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:41.807 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:10:41.845 INFO:teuthology.orchestra.run.smithi023.stdout:osd.4: {} 2024-03-11T21:10:41.881 INFO:teuthology.orchestra.run.smithi023.stdout:osd.5: {} 2024-03-11T21:10:41.908 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:10:41.914 INFO:teuthology.orchestra.run.smithi023.stdout:osd.6: {} 2024-03-11T21:10:41.952 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:10:41.953 INFO:teuthology.orchestra.run.smithi023.stdout:osd.7: {} 2024-03-11T21:10:41.953 INFO:teuthology.orchestra.run.smithi023.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:10:41.953 INFO:teuthology.orchestra.run.smithi023.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:10:41.953 INFO:teuthology.orchestra.run.smithi023.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:10:41.953 INFO:teuthology.orchestra.run.smithi023.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:10:41.953 INFO:teuthology.orchestra.run.smithi023.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:10:41.953 INFO:teuthology.orchestra.run.smithi023.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:10:41.953 INFO:teuthology.orchestra.run.smithi023.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:10:41.953 INFO:teuthology.orchestra.run.smithi023.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:10:41.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:41.956 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:10:42.057 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:10:42.085 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:10:42.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:42.087 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:10:42.187 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:42.265 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:42.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:42.267 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:10:42.367 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:10:42.385 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:10:42.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:42.387 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:10:42.451 INFO:teuthology.orchestra.run.smithi023.stderr:noscrub is unset 2024-03-11T21:10:42.489 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:42.540 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-03-11T21:10:42.541 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:10:42.549 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:42.555 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:42 smithi146 bash[23809]: cluster 2024-03-11T21:10:41.438171+0000 mon.a (mon.0) 1735 : cluster 0 osdmap e360: 8 total, 8 up, 8 in 2024-03-11T21:10:42.555 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:42 smithi146 bash[23809]: audit 2024-03-11T21:10:41.666186+0000 mon.c (mon.2) 87 : audit 0 from='client.? 172.21.15.23:0/254414462' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:10:42.555 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:42 smithi146 bash[23809]: audit 2024-03-11T21:10:42.082214+0000 mon.a (mon.0) 1736 : audit 1 from='client.? 172.21.15.23:0/4239925468' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:10:42.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:42.556 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:10:42.657 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:42.687 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:42.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:42.691 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:10:42.777 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:42 smithi023 bash[21372]: cluster 2024-03-11T21:10:41.438171+0000 mon.a (mon.0) 1735 : cluster 0 osdmap e360: 8 total, 8 up, 8 in 2024-03-11T21:10:42.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:42 smithi023 bash[21372]: audit 2024-03-11T21:10:41.666186+0000 mon.c (mon.2) 87 : audit 0 from='client.? 172.21.15.23:0/254414462' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:10:42.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:42 smithi023 bash[21372]: audit 2024-03-11T21:10:42.082214+0000 mon.a (mon.0) 1736 : audit 1 from='client.? 172.21.15.23:0/4239925468' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:10:42.778 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:42 smithi023 bash[29146]: cluster 2024-03-11T21:10:41.438171+0000 mon.a (mon.0) 1735 : cluster 0 osdmap e360: 8 total, 8 up, 8 in 2024-03-11T21:10:42.778 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:42 smithi023 bash[29146]: audit 2024-03-11T21:10:41.666186+0000 mon.c (mon.2) 87 : audit 0 from='client.? 172.21.15.23:0/254414462' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:10:42.778 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:42 smithi023 bash[29146]: audit 2024-03-11T21:10:42.082214+0000 mon.a (mon.0) 1736 : audit 1 from='client.? 172.21.15.23:0/4239925468' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:10:42.792 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:10:42.821 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:10:42.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:42.826 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:10:42.927 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:10:42.950 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:10:42.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:42.952 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:10:43.053 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:10:43.083 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:10:43.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:43.085 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:10:43.185 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:10:43.219 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:10:43.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:43.225 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:10:43.325 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:10:43.356 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:10:43.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:43.363 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:10:43.464 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:43.487 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:43.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:43.490 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:10:43.590 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:43.620 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:43 smithi023 bash[29146]: cluster 2024-03-11T21:10:42.329279+0000 mgr.y (mgr.24380) 737 : cluster 0 pgmap v836: 148 pgs: 148 active+clean; 1.5 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 84 KiB/s rd, 0 B/s wr, 916 op/s 2024-03-11T21:10:43.620 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:43 smithi023 bash[29146]: cluster 2024-03-11T21:10:42.431026+0000 mon.a (mon.0) 1737 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:10:43.620 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:43 smithi023 bash[29146]: audit 2024-03-11T21:10:42.449353+0000 mon.a (mon.0) 1738 : audit 1 from='client.? 172.21.15.23:0/4239925468' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:10:43.620 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:43 smithi023 bash[29146]: cluster 2024-03-11T21:10:42.456643+0000 mon.a (mon.0) 1739 : cluster 0 osdmap e361: 8 total, 8 up, 8 in 2024-03-11T21:10:43.621 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:43.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:43.628 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:10:43.729 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:10:43.738 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:43 smithi146 bash[23809]: cluster 2024-03-11T21:10:42.329279+0000 mgr.y (mgr.24380) 737 : cluster 0 pgmap v836: 148 pgs: 148 active+clean; 1.5 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 84 KiB/s rd, 0 B/s wr, 916 op/s 2024-03-11T21:10:43.738 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:43 smithi146 bash[23809]: cluster 2024-03-11T21:10:42.431026+0000 mon.a (mon.0) 1737 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:10:43.739 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:43 smithi146 bash[23809]: audit 2024-03-11T21:10:42.449353+0000 mon.a (mon.0) 1738 : audit 1 from='client.? 172.21.15.23:0/4239925468' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:10:43.739 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:43 smithi146 bash[23809]: cluster 2024-03-11T21:10:42.456643+0000 mon.a (mon.0) 1739 : cluster 0 osdmap e361: 8 total, 8 up, 8 in 2024-03-11T21:10:43.755 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:10:43.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:43.759 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:10:43.859 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:10:43.902 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:10:43.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:43.905 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:10:43.962 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-11T21:10:44.006 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:44.007 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:43 smithi023 bash[21372]: cluster 2024-03-11T21:10:42.329279+0000 mgr.y (mgr.24380) 737 : cluster 0 pgmap v836: 148 pgs: 148 active+clean; 1.5 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 84 KiB/s rd, 0 B/s wr, 916 op/s 2024-03-11T21:10:44.007 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:43 smithi023 bash[21372]: cluster 2024-03-11T21:10:42.431026+0000 mon.a (mon.0) 1737 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:10:44.007 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:43 smithi023 bash[21372]: audit 2024-03-11T21:10:42.449353+0000 mon.a (mon.0) 1738 : audit 1 from='client.? 172.21.15.23:0/4239925468' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:10:44.007 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:43 smithi023 bash[21372]: cluster 2024-03-11T21:10:42.456643+0000 mon.a (mon.0) 1739 : cluster 0 osdmap e361: 8 total, 8 up, 8 in 2024-03-11T21:10:44.040 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:44.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:44.046 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:10:44.146 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:10:44.181 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:10:44.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:44.184 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:10:44.285 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:44.317 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:44.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:44.319 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:10:44.420 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:10:44.456 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:10:44.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:44.460 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:10:44.561 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:10:44.588 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:10:44.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:44.592 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:10:44.693 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:44.704 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:44 smithi146 bash[23809]: cluster 2024-03-11T21:10:42.612690+0000 osd.5 (osd.5) 411 : cluster 0 3.5 scrub starts 2024-03-11T21:10:44.704 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:44 smithi146 bash[23809]: cluster 2024-03-11T21:10:42.616209+0000 osd.5 (osd.5) 412 : cluster 0 3.5 scrub ok 2024-03-11T21:10:44.704 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:44 smithi146 bash[23809]: cluster 2024-03-11T21:10:43.492057+0000 osd.6 (osd.6) 332 : cluster 0 10.a scrub starts 2024-03-11T21:10:44.704 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:44 smithi146 bash[23809]: cluster 2024-03-11T21:10:44.027956+0000 osd.6 (osd.6) 333 : cluster 0 10.a scrub ok 2024-03-11T21:10:44.722 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:44.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:44.726 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:10:44.777 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:44 smithi023 bash[21372]: cluster 2024-03-11T21:10:42.612690+0000 osd.5 (osd.5) 411 : cluster 0 3.5 scrub starts 2024-03-11T21:10:44.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:44 smithi023 bash[21372]: cluster 2024-03-11T21:10:42.616209+0000 osd.5 (osd.5) 412 : cluster 0 3.5 scrub ok 2024-03-11T21:10:44.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:44 smithi023 bash[21372]: cluster 2024-03-11T21:10:43.492057+0000 osd.6 (osd.6) 332 : cluster 0 10.a scrub starts 2024-03-11T21:10:44.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:44 smithi023 bash[21372]: cluster 2024-03-11T21:10:44.027956+0000 osd.6 (osd.6) 333 : cluster 0 10.a scrub ok 2024-03-11T21:10:44.778 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:44 smithi023 bash[29146]: cluster 2024-03-11T21:10:42.612690+0000 osd.5 (osd.5) 411 : cluster 0 3.5 scrub starts 2024-03-11T21:10:44.778 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:44 smithi023 bash[29146]: cluster 2024-03-11T21:10:42.616209+0000 osd.5 (osd.5) 412 : cluster 0 3.5 scrub ok 2024-03-11T21:10:44.778 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:44 smithi023 bash[29146]: cluster 2024-03-11T21:10:43.492057+0000 osd.6 (osd.6) 332 : cluster 0 10.a scrub starts 2024-03-11T21:10:44.778 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:44 smithi023 bash[29146]: cluster 2024-03-11T21:10:44.027956+0000 osd.6 (osd.6) 333 : cluster 0 10.a scrub ok 2024-03-11T21:10:44.827 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:10:44.864 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:10:44.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:44.867 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:10:44.967 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:10:44.994 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:10:45.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:45.000 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:10:45.101 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:10:45.131 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:10:45.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:45.133 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:10:45.218 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T21:10:45.235 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:45.313 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:45.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:45.315 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:10:45.353 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 6, 7, 1] out_osds: [] dead_osds: [] live_osds: [0, 2, 3, 5, 6, 7, 4, 1] 2024-03-11T21:10:45.354 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-11T21:10:45.354 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-03-11T21:10:45.354 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T21:10:45.416 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:45.439 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:45.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:45.448 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:10:45.549 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:10:45.572 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:10:45.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:45.575 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:10:45.676 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:10:45.685 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:45 smithi146 bash[23809]: cluster 2024-03-11T21:10:44.329665+0000 mgr.y (mgr.24380) 738 : cluster 0 pgmap v838: 148 pgs: 148 active+clean; 1.5 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 765 op/s 2024-03-11T21:10:45.686 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:45 smithi146 bash[23809]: cluster 2024-03-11T21:10:44.558801+0000 osd.5 (osd.5) 413 : cluster 0 10.9 scrub starts 2024-03-11T21:10:45.686 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:45 smithi146 bash[23809]: cluster 2024-03-11T21:10:44.941566+0000 osd.5 (osd.5) 414 : cluster 0 10.9 scrub ok 2024-03-11T21:10:45.707 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:10:45.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:45.710 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:10:45.777 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:45 smithi023 bash[21372]: cluster 2024-03-11T21:10:44.329665+0000 mgr.y (mgr.24380) 738 : cluster 0 pgmap v838: 148 pgs: 148 active+clean; 1.5 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 765 op/s 2024-03-11T21:10:45.777 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:45 smithi023 bash[21372]: cluster 2024-03-11T21:10:44.558801+0000 osd.5 (osd.5) 413 : cluster 0 10.9 scrub starts 2024-03-11T21:10:45.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:45 smithi023 bash[21372]: cluster 2024-03-11T21:10:44.941566+0000 osd.5 (osd.5) 414 : cluster 0 10.9 scrub ok 2024-03-11T21:10:45.778 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:45 smithi023 bash[29146]: cluster 2024-03-11T21:10:44.329665+0000 mgr.y (mgr.24380) 738 : cluster 0 pgmap v838: 148 pgs: 148 active+clean; 1.5 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 765 op/s 2024-03-11T21:10:45.778 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:45 smithi023 bash[29146]: cluster 2024-03-11T21:10:44.558801+0000 osd.5 (osd.5) 413 : cluster 0 10.9 scrub starts 2024-03-11T21:10:45.778 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:45 smithi023 bash[29146]: cluster 2024-03-11T21:10:44.941566+0000 osd.5 (osd.5) 414 : cluster 0 10.9 scrub ok 2024-03-11T21:10:45.811 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:10:45.833 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:10:45.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:45.836 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:10:45.937 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:45.968 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:45.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:45.971 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:10:46.007 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:10:45 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:10:46.009 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-11T21:10:46.072 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:46.102 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:46.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:46.106 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:10:46.207 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:10:46.244 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:10:46.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:46.245 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:10:46.346 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:10:46.382 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:10:46.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:46.384 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:10:46.485 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:10:46.580 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:10:46.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:46.584 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:10:46.684 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:46.694 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:46 smithi023 bash[21372]: cluster 2024-03-11T21:10:45.522015+0000 osd.6 (osd.6) 334 : cluster 0 10.0 scrub starts 2024-03-11T21:10:46.694 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:46 smithi023 bash[21372]: cluster 2024-03-11T21:10:45.962973+0000 osd.6 (osd.6) 335 : cluster 0 10.0 scrub ok 2024-03-11T21:10:46.695 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:46 smithi023 bash[29146]: cluster 2024-03-11T21:10:45.522015+0000 osd.6 (osd.6) 334 : cluster 0 10.0 scrub starts 2024-03-11T21:10:46.695 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:46 smithi023 bash[29146]: cluster 2024-03-11T21:10:45.962973+0000 osd.6 (osd.6) 335 : cluster 0 10.0 scrub ok 2024-03-11T21:10:46.714 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:46.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:46.717 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:10:46.758 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:46 smithi146 bash[23809]: cluster 2024-03-11T21:10:45.522015+0000 osd.6 (osd.6) 334 : cluster 0 10.0 scrub starts 2024-03-11T21:10:46.758 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:46 smithi146 bash[23809]: cluster 2024-03-11T21:10:45.962973+0000 osd.6 (osd.6) 335 : cluster 0 10.0 scrub ok 2024-03-11T21:10:46.818 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:46.856 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:46.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:46.858 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:10:46.959 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:10:46.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:46.991 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:10:46.991 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:10:47.092 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:47.123 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:47.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:47.126 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:10:47.227 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:10:47.263 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:10:47.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:47.268 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:10:47.368 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:10:47.403 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:10:47.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:47.408 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:10:47.508 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:47 smithi146 bash[23809]: audit 2024-03-11T21:10:45.849148+0000 mgr.y (mgr.24380) 739 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:10:47.508 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:47 smithi146 bash[23809]: cluster 2024-03-11T21:10:46.330389+0000 mgr.y (mgr.24380) 740 : cluster 0 pgmap v839: 148 pgs: 148 active+clean; 1.3 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 34 KiB/s rd, 0 B/s wr, 1.08k op/s 2024-03-11T21:10:47.508 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:47 smithi146 bash[23809]: cluster 2024-03-11T21:10:46.536232+0000 osd.6 (osd.6) 336 : cluster 0 10.d scrub starts 2024-03-11T21:10:47.508 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:47 smithi146 bash[23809]: cluster 2024-03-11T21:10:46.920600+0000 osd.6 (osd.6) 337 : cluster 0 10.d scrub ok 2024-03-11T21:10:47.509 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:10:47.536 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:10:47.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:47.539 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:10:47.640 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:10:47.668 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:10:47.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:47.673 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:10:47.774 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:47.777 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:47 smithi023 bash[21372]: audit 2024-03-11T21:10:45.849148+0000 mgr.y (mgr.24380) 739 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:10:47.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:47 smithi023 bash[21372]: cluster 2024-03-11T21:10:46.330389+0000 mgr.y (mgr.24380) 740 : cluster 0 pgmap v839: 148 pgs: 148 active+clean; 1.3 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 34 KiB/s rd, 0 B/s wr, 1.08k op/s 2024-03-11T21:10:47.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:47 smithi023 bash[21372]: cluster 2024-03-11T21:10:46.536232+0000 osd.6 (osd.6) 336 : cluster 0 10.d scrub starts 2024-03-11T21:10:47.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:47 smithi023 bash[21372]: cluster 2024-03-11T21:10:46.920600+0000 osd.6 (osd.6) 337 : cluster 0 10.d scrub ok 2024-03-11T21:10:47.778 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:47 smithi023 bash[29146]: audit 2024-03-11T21:10:45.849148+0000 mgr.y (mgr.24380) 739 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:10:47.778 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:47 smithi023 bash[29146]: cluster 2024-03-11T21:10:46.330389+0000 mgr.y (mgr.24380) 740 : cluster 0 pgmap v839: 148 pgs: 148 active+clean; 1.3 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 34 KiB/s rd, 0 B/s wr, 1.08k op/s 2024-03-11T21:10:47.778 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:47 smithi023 bash[29146]: cluster 2024-03-11T21:10:46.536232+0000 osd.6 (osd.6) 336 : cluster 0 10.d scrub starts 2024-03-11T21:10:47.778 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:47 smithi023 bash[29146]: cluster 2024-03-11T21:10:46.920600+0000 osd.6 (osd.6) 337 : cluster 0 10.d scrub ok 2024-03-11T21:10:47.799 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:47.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:47.802 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:10:47.903 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:10:47.940 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:10:47.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:47.942 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:10:48.043 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:10:48.068 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:10:48.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:48.073 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:10:48.174 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:10:48.209 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:10:48.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:48.212 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:10:48.312 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:10:48.356 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:10:48.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:48.359 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:10:48.459 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:10:48.486 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:10:48.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:48.492 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:10:48.592 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:10:48.602 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:48 smithi146 bash[23809]: cluster 2024-03-11T21:10:47.272619+0000 osd.1 (osd.1) 219 : cluster 0 3.19 scrub starts 2024-03-11T21:10:48.602 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:48 smithi146 bash[23809]: cluster 2024-03-11T21:10:47.277317+0000 osd.1 (osd.1) 220 : cluster 0 3.19 scrub ok 2024-03-11T21:10:48.602 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:48 smithi146 bash[23809]: cluster 2024-03-11T21:10:47.570640+0000 osd.6 (osd.6) 338 : cluster 0 2.7 scrub starts 2024-03-11T21:10:48.602 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:48 smithi146 bash[23809]: cluster 2024-03-11T21:10:47.579836+0000 osd.6 (osd.6) 339 : cluster 0 2.7 scrub ok 2024-03-11T21:10:48.624 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:10:48.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:48.629 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:10:48.730 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:10:48.742 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:48 smithi023 bash[21372]: cluster 2024-03-11T21:10:47.272619+0000 osd.1 (osd.1) 219 : cluster 0 3.19 scrub starts 2024-03-11T21:10:48.743 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:48 smithi023 bash[21372]: cluster 2024-03-11T21:10:47.277317+0000 osd.1 (osd.1) 220 : cluster 0 3.19 scrub ok 2024-03-11T21:10:48.743 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:48 smithi023 bash[21372]: cluster 2024-03-11T21:10:47.570640+0000 osd.6 (osd.6) 338 : cluster 0 2.7 scrub starts 2024-03-11T21:10:48.743 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:48 smithi023 bash[21372]: cluster 2024-03-11T21:10:47.579836+0000 osd.6 (osd.6) 339 : cluster 0 2.7 scrub ok 2024-03-11T21:10:48.743 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:10:48 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:10:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:10:48.743 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:48 smithi023 bash[29146]: cluster 2024-03-11T21:10:47.272619+0000 osd.1 (osd.1) 219 : cluster 0 3.19 scrub starts 2024-03-11T21:10:48.743 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:48 smithi023 bash[29146]: cluster 2024-03-11T21:10:47.277317+0000 osd.1 (osd.1) 220 : cluster 0 3.19 scrub ok 2024-03-11T21:10:48.743 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:48 smithi023 bash[29146]: cluster 2024-03-11T21:10:47.570640+0000 osd.6 (osd.6) 338 : cluster 0 2.7 scrub starts 2024-03-11T21:10:48.744 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:48 smithi023 bash[29146]: cluster 2024-03-11T21:10:47.579836+0000 osd.6 (osd.6) 339 : cluster 0 2.7 scrub ok 2024-03-11T21:10:48.756 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:10:48.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:48.763 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:10:48.841 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:10:48.864 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:10:48.890 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:10:48.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:48.895 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:10:48.996 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:10:49.028 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:10:49.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:49.030 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:10:49.131 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:10:49.165 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:10:49.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:49.171 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:10:49.271 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:10:49.300 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:10:49.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:49.303 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:10:49.404 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:10:49.428 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:10:49.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:49.430 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:10:49.531 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:10:49.555 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:10:49.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:49.558 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:10:49.659 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:10:49.680 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:10:49.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:49.682 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:10:49.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:49 smithi023 bash[21372]: cluster 2024-03-11T21:10:48.262016+0000 osd.1 (osd.1) 221 : cluster 0 4.9 scrub starts 2024-03-11T21:10:49.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:49 smithi023 bash[21372]: cluster 2024-03-11T21:10:48.267034+0000 osd.1 (osd.1) 222 : cluster 0 4.9 scrub ok 2024-03-11T21:10:49.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:49 smithi023 bash[21372]: cluster 2024-03-11T21:10:48.330831+0000 mgr.y (mgr.24380) 741 : cluster 0 pgmap v840: 148 pgs: 148 active+clean; 1.2 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 753 op/s 2024-03-11T21:10:49.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:49 smithi023 bash[21372]: cluster 2024-03-11T21:10:48.505680+0000 osd.5 (osd.5) 415 : cluster 0 3.1d scrub starts 2024-03-11T21:10:49.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:49 smithi023 bash[21372]: cluster 2024-03-11T21:10:48.516554+0000 osd.5 (osd.5) 416 : cluster 0 3.1d scrub ok 2024-03-11T21:10:49.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:49 smithi023 bash[21372]: cluster 2024-03-11T21:10:48.963051+0000 osd.3 (osd.3) 408 : cluster 0 6.13 scrub starts 2024-03-11T21:10:49.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:49 smithi023 bash[21372]: cluster 2024-03-11T21:10:49.083020+0000 osd.3 (osd.3) 409 : cluster 0 6.13 scrub ok 2024-03-11T21:10:49.779 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:49 smithi023 bash[29146]: cluster 2024-03-11T21:10:48.262016+0000 osd.1 (osd.1) 221 : cluster 0 4.9 scrub starts 2024-03-11T21:10:49.779 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:49 smithi023 bash[29146]: cluster 2024-03-11T21:10:48.267034+0000 osd.1 (osd.1) 222 : cluster 0 4.9 scrub ok 2024-03-11T21:10:49.779 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:49 smithi023 bash[29146]: cluster 2024-03-11T21:10:48.330831+0000 mgr.y (mgr.24380) 741 : cluster 0 pgmap v840: 148 pgs: 148 active+clean; 1.2 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 753 op/s 2024-03-11T21:10:49.779 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:49 smithi023 bash[29146]: cluster 2024-03-11T21:10:48.505680+0000 osd.5 (osd.5) 415 : cluster 0 3.1d scrub starts 2024-03-11T21:10:49.779 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:49 smithi023 bash[29146]: cluster 2024-03-11T21:10:48.516554+0000 osd.5 (osd.5) 416 : cluster 0 3.1d scrub ok 2024-03-11T21:10:49.779 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:49 smithi023 bash[29146]: cluster 2024-03-11T21:10:48.963051+0000 osd.3 (osd.3) 408 : cluster 0 6.13 scrub starts 2024-03-11T21:10:49.779 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:49 smithi023 bash[29146]: cluster 2024-03-11T21:10:49.083020+0000 osd.3 (osd.3) 409 : cluster 0 6.13 scrub ok 2024-03-11T21:10:49.783 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:10:49.810 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:10:49.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:49.813 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:10:49.914 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:49.943 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:49.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:49.945 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:10:50.001 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:49 smithi146 bash[23809]: cluster 2024-03-11T21:10:48.262016+0000 osd.1 (osd.1) 221 : cluster 0 4.9 scrub starts 2024-03-11T21:10:50.001 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:49 smithi146 bash[23809]: cluster 2024-03-11T21:10:48.267034+0000 osd.1 (osd.1) 222 : cluster 0 4.9 scrub ok 2024-03-11T21:10:50.001 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:49 smithi146 bash[23809]: cluster 2024-03-11T21:10:48.330831+0000 mgr.y (mgr.24380) 741 : cluster 0 pgmap v840: 148 pgs: 148 active+clean; 1.2 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 753 op/s 2024-03-11T21:10:50.001 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:49 smithi146 bash[23809]: cluster 2024-03-11T21:10:48.505680+0000 osd.5 (osd.5) 415 : cluster 0 3.1d scrub starts 2024-03-11T21:10:50.001 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:49 smithi146 bash[23809]: cluster 2024-03-11T21:10:48.516554+0000 osd.5 (osd.5) 416 : cluster 0 3.1d scrub ok 2024-03-11T21:10:50.001 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:49 smithi146 bash[23809]: cluster 2024-03-11T21:10:48.963051+0000 osd.3 (osd.3) 408 : cluster 0 6.13 scrub starts 2024-03-11T21:10:50.001 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:49 smithi146 bash[23809]: cluster 2024-03-11T21:10:49.083020+0000 osd.3 (osd.3) 409 : cluster 0 6.13 scrub ok 2024-03-11T21:10:50.046 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:10:50.070 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:10:50.070 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:10:50.070 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:10:50.073 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:10:50.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:50.075 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:10:50.175 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:10:50.208 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:10:50.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:50.219 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:10:50.320 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:50.342 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:50.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:50.345 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:10:50.445 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:50.480 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:50.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:50.483 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:10:50.583 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:50.595 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:50 smithi146 bash[23809]: cluster 2024-03-11T21:10:48.845392+0000 osd.7 (osd.7) 453 : cluster 0 3.f scrub starts 2024-03-11T21:10:50.595 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:50 smithi146 bash[23809]: cluster 2024-03-11T21:10:48.856940+0000 osd.7 (osd.7) 454 : cluster 0 3.f scrub ok 2024-03-11T21:10:50.595 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:50 smithi146 bash[23809]: cluster 2024-03-11T21:10:50.015605+0000 osd.0 (osd.0) 313 : cluster 0 6.a scrub starts 2024-03-11T21:10:50.595 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:50 smithi146 bash[23809]: cluster 2024-03-11T21:10:50.090569+0000 osd.0 (osd.0) 314 : cluster 0 6.a scrub ok 2024-03-11T21:10:50.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:50.613 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:50.613 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:10:50.714 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:50.724 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:50 smithi023 bash[21372]: cluster 2024-03-11T21:10:48.845392+0000 osd.7 (osd.7) 453 : cluster 0 3.f scrub starts 2024-03-11T21:10:50.724 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:50 smithi023 bash[21372]: cluster 2024-03-11T21:10:48.856940+0000 osd.7 (osd.7) 454 : cluster 0 3.f scrub ok 2024-03-11T21:10:50.724 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:50 smithi023 bash[21372]: cluster 2024-03-11T21:10:50.015605+0000 osd.0 (osd.0) 313 : cluster 0 6.a scrub starts 2024-03-11T21:10:50.724 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:50 smithi023 bash[21372]: cluster 2024-03-11T21:10:50.090569+0000 osd.0 (osd.0) 314 : cluster 0 6.a scrub ok 2024-03-11T21:10:50.725 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:50 smithi023 bash[29146]: cluster 2024-03-11T21:10:48.845392+0000 osd.7 (osd.7) 453 : cluster 0 3.f scrub starts 2024-03-11T21:10:50.725 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:50 smithi023 bash[29146]: cluster 2024-03-11T21:10:48.856940+0000 osd.7 (osd.7) 454 : cluster 0 3.f scrub ok 2024-03-11T21:10:50.725 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:50 smithi023 bash[29146]: cluster 2024-03-11T21:10:50.015605+0000 osd.0 (osd.0) 313 : cluster 0 6.a scrub starts 2024-03-11T21:10:50.725 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:50 smithi023 bash[29146]: cluster 2024-03-11T21:10:50.090569+0000 osd.0 (osd.0) 314 : cluster 0 6.a scrub ok 2024-03-11T21:10:50.745 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:50.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:50.753 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:10:50.854 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:10:50.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:50.898 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:10:50.899 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:10:50.999 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:10:51.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:51.039 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:10:51.039 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:10:51.140 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:10:51.204 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:10:51.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:51.205 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:10:51.306 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:10:51.324 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:10:51.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:51.326 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:10:51.428 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:10:51.451 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:10:51.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:51.453 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:10:51.554 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:51.603 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:51.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:51.606 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:10:51.707 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:10:51.718 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:51 smithi146 bash[23809]: cluster 2024-03-11T21:10:50.014533+0000 osd.7 (osd.7) 455 : cluster 0 3.11 scrub starts 2024-03-11T21:10:51.718 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:51 smithi146 bash[23809]: cluster 2024-03-11T21:10:50.019124+0000 osd.7 (osd.7) 456 : cluster 0 3.11 scrub ok 2024-03-11T21:10:51.719 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:51 smithi146 bash[23809]: cluster 2024-03-11T21:10:50.331384+0000 mgr.y (mgr.24380) 742 : cluster 0 pgmap v841: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.1 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 5.8 KiB/s rd, 0 B/s wr, 877 op/s 2024-03-11T21:10:51.719 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:51 smithi146 bash[23809]: cluster 2024-03-11T21:10:50.550404+0000 osd.6 (osd.6) 340 : cluster 0 2.14 scrub starts 2024-03-11T21:10:51.719 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:51 smithi146 bash[23809]: cluster 2024-03-11T21:10:50.551114+0000 osd.5 (osd.5) 417 : cluster 0 6.d scrub starts 2024-03-11T21:10:51.719 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:51 smithi146 bash[23809]: cluster 2024-03-11T21:10:50.551637+0000 osd.6 (osd.6) 341 : cluster 0 2.14 scrub ok 2024-03-11T21:10:51.719 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:51 smithi146 bash[23809]: cluster 2024-03-11T21:10:50.652480+0000 osd.5 (osd.5) 418 : cluster 0 6.d scrub ok 2024-03-11T21:10:51.719 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:51 smithi146 bash[23809]: audit 2024-03-11T21:10:51.037055+0000 mon.a (mon.0) 1740 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:10:51.739 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:10:51.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:51.745 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:10:51.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:51 smithi023 bash[21372]: cluster 2024-03-11T21:10:50.014533+0000 osd.7 (osd.7) 455 : cluster 0 3.11 scrub starts 2024-03-11T21:10:51.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:51 smithi023 bash[21372]: cluster 2024-03-11T21:10:50.019124+0000 osd.7 (osd.7) 456 : cluster 0 3.11 scrub ok 2024-03-11T21:10:51.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:51 smithi023 bash[21372]: cluster 2024-03-11T21:10:50.331384+0000 mgr.y (mgr.24380) 742 : cluster 0 pgmap v841: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.1 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 5.8 KiB/s rd, 0 B/s wr, 877 op/s 2024-03-11T21:10:51.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:51 smithi023 bash[21372]: cluster 2024-03-11T21:10:50.550404+0000 osd.6 (osd.6) 340 : cluster 0 2.14 scrub starts 2024-03-11T21:10:51.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:51 smithi023 bash[21372]: cluster 2024-03-11T21:10:50.551114+0000 osd.5 (osd.5) 417 : cluster 0 6.d scrub starts 2024-03-11T21:10:51.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:51 smithi023 bash[21372]: cluster 2024-03-11T21:10:50.551637+0000 osd.6 (osd.6) 341 : cluster 0 2.14 scrub ok 2024-03-11T21:10:51.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:51 smithi023 bash[21372]: cluster 2024-03-11T21:10:50.652480+0000 osd.5 (osd.5) 418 : cluster 0 6.d scrub ok 2024-03-11T21:10:51.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:51 smithi023 bash[21372]: audit 2024-03-11T21:10:51.037055+0000 mon.a (mon.0) 1740 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:10:51.778 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:51 smithi023 bash[29146]: cluster 2024-03-11T21:10:50.014533+0000 osd.7 (osd.7) 455 : cluster 0 3.11 scrub starts 2024-03-11T21:10:51.779 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:51 smithi023 bash[29146]: cluster 2024-03-11T21:10:50.019124+0000 osd.7 (osd.7) 456 : cluster 0 3.11 scrub ok 2024-03-11T21:10:51.779 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:51 smithi023 bash[29146]: cluster 2024-03-11T21:10:50.331384+0000 mgr.y (mgr.24380) 742 : cluster 0 pgmap v841: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.1 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 5.8 KiB/s rd, 0 B/s wr, 877 op/s 2024-03-11T21:10:51.779 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:51 smithi023 bash[29146]: cluster 2024-03-11T21:10:50.550404+0000 osd.6 (osd.6) 340 : cluster 0 2.14 scrub starts 2024-03-11T21:10:51.779 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:51 smithi023 bash[29146]: cluster 2024-03-11T21:10:50.551114+0000 osd.5 (osd.5) 417 : cluster 0 6.d scrub starts 2024-03-11T21:10:51.779 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:51 smithi023 bash[29146]: cluster 2024-03-11T21:10:50.551637+0000 osd.6 (osd.6) 341 : cluster 0 2.14 scrub ok 2024-03-11T21:10:51.779 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:51 smithi023 bash[29146]: cluster 2024-03-11T21:10:50.652480+0000 osd.5 (osd.5) 418 : cluster 0 6.d scrub ok 2024-03-11T21:10:51.779 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:51 smithi023 bash[29146]: audit 2024-03-11T21:10:51.037055+0000 mon.a (mon.0) 1740 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:10:51.845 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:10:51.880 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:10:51.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:51.883 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:10:51.983 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:10:52.007 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:10:52.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:52.010 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:10:52.111 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:10:52.136 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:10:52.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:52.140 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:10:52.241 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:10:52.264 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:10:52.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:52.267 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:10:52.368 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:52.396 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:52.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:52.399 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:10:52.500 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:52.527 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:52.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:52.528 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:10:52.629 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:10:52.648 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:10:52.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:52.713 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:10:52.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:52 smithi023 bash[21372]: cluster 2024-03-11T21:10:51.554920+0000 osd.5 (osd.5) 419 : cluster 0 6.17 scrub starts 2024-03-11T21:10:52.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:52 smithi023 bash[21372]: cluster 2024-03-11T21:10:51.719376+0000 osd.5 (osd.5) 420 : cluster 0 6.17 scrub ok 2024-03-11T21:10:52.778 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:52 smithi023 bash[29146]: cluster 2024-03-11T21:10:51.554920+0000 osd.5 (osd.5) 419 : cluster 0 6.17 scrub starts 2024-03-11T21:10:52.778 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:52 smithi023 bash[29146]: cluster 2024-03-11T21:10:51.719376+0000 osd.5 (osd.5) 420 : cluster 0 6.17 scrub ok 2024-03-11T21:10:52.814 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:10:52.849 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:10:52.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:52.854 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:10:52.955 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:10:52.984 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:10:52.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:52.991 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:10:53.007 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:52 smithi146 bash[23809]: cluster 2024-03-11T21:10:51.554920+0000 osd.5 (osd.5) 419 : cluster 0 6.17 scrub starts 2024-03-11T21:10:53.008 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:52 smithi146 bash[23809]: cluster 2024-03-11T21:10:51.719376+0000 osd.5 (osd.5) 420 : cluster 0 6.17 scrub ok 2024-03-11T21:10:53.091 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:10:53.119 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:10:53.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:53.121 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:10:53.222 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:10:53.254 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:10:53.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:53.258 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:10:53.359 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:53.391 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:53.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:53.395 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:10:53.496 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:10:53.521 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:10:53.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:53.526 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:10:53.627 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:53.638 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:53 smithi023 bash[21372]: cluster 2024-03-11T21:10:52.023337+0000 osd.3 (osd.3) 410 : cluster 0 3.1e scrub starts 2024-03-11T21:10:53.638 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:53 smithi023 bash[21372]: cluster 2024-03-11T21:10:52.029822+0000 osd.3 (osd.3) 411 : cluster 0 3.1e scrub ok 2024-03-11T21:10:53.638 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:53 smithi023 bash[21372]: cluster 2024-03-11T21:10:52.332065+0000 mgr.y (mgr.24380) 743 : cluster 0 pgmap v842: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 983 MiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 940 op/s 2024-03-11T21:10:53.638 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:53 smithi023 bash[21372]: cluster 2024-03-11T21:10:52.519992+0000 osd.1 (osd.1) 223 : cluster 0 2.10 scrub starts 2024-03-11T21:10:53.639 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:53 smithi023 bash[21372]: cluster 2024-03-11T21:10:52.523591+0000 osd.1 (osd.1) 224 : cluster 0 2.10 scrub ok 2024-03-11T21:10:53.639 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:53 smithi023 bash[21372]: cluster 2024-03-11T21:10:52.834695+0000 osd.0 (osd.0) 315 : cluster 0 3.1b scrub starts 2024-03-11T21:10:53.639 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:53 smithi023 bash[21372]: cluster 2024-03-11T21:10:52.845039+0000 osd.0 (osd.0) 316 : cluster 0 3.1b scrub ok 2024-03-11T21:10:53.639 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:53 smithi023 bash[29146]: cluster 2024-03-11T21:10:52.023337+0000 osd.3 (osd.3) 410 : cluster 0 3.1e scrub starts 2024-03-11T21:10:53.639 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:53 smithi023 bash[29146]: cluster 2024-03-11T21:10:52.029822+0000 osd.3 (osd.3) 411 : cluster 0 3.1e scrub ok 2024-03-11T21:10:53.639 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:53 smithi023 bash[29146]: cluster 2024-03-11T21:10:52.332065+0000 mgr.y (mgr.24380) 743 : cluster 0 pgmap v842: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 983 MiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 940 op/s 2024-03-11T21:10:53.639 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:53 smithi023 bash[29146]: cluster 2024-03-11T21:10:52.519992+0000 osd.1 (osd.1) 223 : cluster 0 2.10 scrub starts 2024-03-11T21:10:53.640 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:53 smithi023 bash[29146]: cluster 2024-03-11T21:10:52.523591+0000 osd.1 (osd.1) 224 : cluster 0 2.10 scrub ok 2024-03-11T21:10:53.640 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:53 smithi023 bash[29146]: cluster 2024-03-11T21:10:52.834695+0000 osd.0 (osd.0) 315 : cluster 0 3.1b scrub starts 2024-03-11T21:10:53.640 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:53 smithi023 bash[29146]: cluster 2024-03-11T21:10:52.845039+0000 osd.0 (osd.0) 316 : cluster 0 3.1b scrub ok 2024-03-11T21:10:53.652 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:53.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:53.657 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:10:53.758 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:10:53.779 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:10:53.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:53.781 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:10:53.882 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:10:53.892 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:53 smithi146 bash[23809]: cluster 2024-03-11T21:10:52.023337+0000 osd.3 (osd.3) 410 : cluster 0 3.1e scrub starts 2024-03-11T21:10:53.892 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:53 smithi146 bash[23809]: cluster 2024-03-11T21:10:52.029822+0000 osd.3 (osd.3) 411 : cluster 0 3.1e scrub ok 2024-03-11T21:10:53.892 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:53 smithi146 bash[23809]: cluster 2024-03-11T21:10:52.332065+0000 mgr.y (mgr.24380) 743 : cluster 0 pgmap v842: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 983 MiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 940 op/s 2024-03-11T21:10:53.892 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:53 smithi146 bash[23809]: cluster 2024-03-11T21:10:52.519992+0000 osd.1 (osd.1) 223 : cluster 0 2.10 scrub starts 2024-03-11T21:10:53.893 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:53 smithi146 bash[23809]: cluster 2024-03-11T21:10:52.523591+0000 osd.1 (osd.1) 224 : cluster 0 2.10 scrub ok 2024-03-11T21:10:53.893 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:53 smithi146 bash[23809]: cluster 2024-03-11T21:10:52.834695+0000 osd.0 (osd.0) 315 : cluster 0 3.1b scrub starts 2024-03-11T21:10:53.893 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:53 smithi146 bash[23809]: cluster 2024-03-11T21:10:52.845039+0000 osd.0 (osd.0) 316 : cluster 0 3.1b scrub ok 2024-03-11T21:10:53.910 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:10:53.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:53.919 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:10:54.020 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:10:54.057 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:10:54.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:54.063 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:10:54.163 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:10:54.222 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:10:54.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:54.225 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:10:54.325 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:10:54.350 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:10:54.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:54.358 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:10:54.459 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:10:54.490 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:10:54.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:54.495 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:10:54.595 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:54.618 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:54.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:54.622 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:10:54.723 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:10:54.757 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:10:54.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:54.767 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:10:54.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:54 smithi023 bash[21372]: cluster 2024-03-11T21:10:52.819668+0000 osd.7 (osd.7) 457 : cluster 0 2.8 scrub starts 2024-03-11T21:10:54.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:54 smithi023 bash[21372]: cluster 2024-03-11T21:10:52.823169+0000 osd.7 (osd.7) 458 : cluster 0 2.8 scrub ok 2024-03-11T21:10:54.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:54 smithi023 bash[21372]: cluster 2024-03-11T21:10:53.559157+0000 osd.1 (osd.1) 225 : cluster 0 2.9 scrub starts 2024-03-11T21:10:54.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:54 smithi023 bash[21372]: cluster 2024-03-11T21:10:53.563847+0000 osd.1 (osd.1) 226 : cluster 0 2.9 scrub ok 2024-03-11T21:10:54.778 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:54 smithi023 bash[29146]: cluster 2024-03-11T21:10:52.819668+0000 osd.7 (osd.7) 457 : cluster 0 2.8 scrub starts 2024-03-11T21:10:54.779 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:54 smithi023 bash[29146]: cluster 2024-03-11T21:10:52.823169+0000 osd.7 (osd.7) 458 : cluster 0 2.8 scrub ok 2024-03-11T21:10:54.779 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:54 smithi023 bash[29146]: cluster 2024-03-11T21:10:53.559157+0000 osd.1 (osd.1) 225 : cluster 0 2.9 scrub starts 2024-03-11T21:10:54.779 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:54 smithi023 bash[29146]: cluster 2024-03-11T21:10:53.563847+0000 osd.1 (osd.1) 226 : cluster 0 2.9 scrub ok 2024-03-11T21:10:54.867 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:10:54.894 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:10:54.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:54.896 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:10:54.997 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:55.007 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:54 smithi146 bash[23809]: cluster 2024-03-11T21:10:52.819668+0000 osd.7 (osd.7) 457 : cluster 0 2.8 scrub starts 2024-03-11T21:10:55.007 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:54 smithi146 bash[23809]: cluster 2024-03-11T21:10:52.823169+0000 osd.7 (osd.7) 458 : cluster 0 2.8 scrub ok 2024-03-11T21:10:55.007 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:54 smithi146 bash[23809]: cluster 2024-03-11T21:10:53.559157+0000 osd.1 (osd.1) 225 : cluster 0 2.9 scrub starts 2024-03-11T21:10:55.007 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:54 smithi146 bash[23809]: cluster 2024-03-11T21:10:53.563847+0000 osd.1 (osd.1) 226 : cluster 0 2.9 scrub ok 2024-03-11T21:10:55.027 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:55.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:55.031 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:10:55.132 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:10:55.156 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:10:55.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:55.158 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:10:55.259 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:10:55.291 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:10:55.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:55.295 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:10:55.396 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:10:55.425 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-03-11T21:10:55.425 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:10:55.426 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:10:55.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:55.430 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:10:55.531 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:10:55.565 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:10:55.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:55.568 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:10:55.668 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:55.757 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:55 smithi146 bash[23809]: cluster 2024-03-11T21:10:53.742021+0000 osd.7 (osd.7) 459 : cluster 0 3.e scrub starts 2024-03-11T21:10:55.757 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:55 smithi146 bash[23809]: cluster 2024-03-11T21:10:53.748593+0000 osd.7 (osd.7) 460 : cluster 0 3.e scrub ok 2024-03-11T21:10:55.758 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:55 smithi146 bash[23809]: cluster 2024-03-11T21:10:54.332516+0000 mgr.y (mgr.24380) 744 : cluster 0 pgmap v843: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 983 MiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 861 B/s rd, 0 B/s wr, 791 op/s 2024-03-11T21:10:55.758 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:55 smithi146 bash[23809]: cluster 2024-03-11T21:10:54.795617+0000 osd.0 (osd.0) 317 : cluster 0 3.17 scrub starts 2024-03-11T21:10:55.758 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:55 smithi146 bash[23809]: cluster 2024-03-11T21:10:54.801305+0000 osd.0 (osd.0) 318 : cluster 0 3.17 scrub ok 2024-03-11T21:10:55.758 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:55 smithi146 bash[23809]: cluster 2024-03-11T21:10:55.010589+0000 osd.3 (osd.3) 412 : cluster 0 2.1 scrub starts 2024-03-11T21:10:55.758 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:55 smithi146 bash[23809]: cluster 2024-03-11T21:10:55.016266+0000 osd.3 (osd.3) 413 : cluster 0 2.1 scrub ok 2024-03-11T21:10:55.762 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:55 smithi023 bash[21372]: cluster 2024-03-11T21:10:53.742021+0000 osd.7 (osd.7) 459 : cluster 0 3.e scrub starts 2024-03-11T21:10:55.762 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:55 smithi023 bash[21372]: cluster 2024-03-11T21:10:53.748593+0000 osd.7 (osd.7) 460 : cluster 0 3.e scrub ok 2024-03-11T21:10:55.762 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:55 smithi023 bash[21372]: cluster 2024-03-11T21:10:54.332516+0000 mgr.y (mgr.24380) 744 : cluster 0 pgmap v843: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 983 MiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 861 B/s rd, 0 B/s wr, 791 op/s 2024-03-11T21:10:55.762 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:55 smithi023 bash[21372]: cluster 2024-03-11T21:10:54.795617+0000 osd.0 (osd.0) 317 : cluster 0 3.17 scrub starts 2024-03-11T21:10:55.762 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:55 smithi023 bash[21372]: cluster 2024-03-11T21:10:54.801305+0000 osd.0 (osd.0) 318 : cluster 0 3.17 scrub ok 2024-03-11T21:10:55.762 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:55 smithi023 bash[21372]: cluster 2024-03-11T21:10:55.010589+0000 osd.3 (osd.3) 412 : cluster 0 2.1 scrub starts 2024-03-11T21:10:55.762 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:55 smithi023 bash[21372]: cluster 2024-03-11T21:10:55.016266+0000 osd.3 (osd.3) 413 : cluster 0 2.1 scrub ok 2024-03-11T21:10:55.763 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:55 smithi023 bash[29146]: cluster 2024-03-11T21:10:53.742021+0000 osd.7 (osd.7) 459 : cluster 0 3.e scrub starts 2024-03-11T21:10:55.763 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:55 smithi023 bash[29146]: cluster 2024-03-11T21:10:53.748593+0000 osd.7 (osd.7) 460 : cluster 0 3.e scrub ok 2024-03-11T21:10:55.763 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:55 smithi023 bash[29146]: cluster 2024-03-11T21:10:54.332516+0000 mgr.y (mgr.24380) 744 : cluster 0 pgmap v843: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 983 MiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 861 B/s rd, 0 B/s wr, 791 op/s 2024-03-11T21:10:55.763 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:55 smithi023 bash[29146]: cluster 2024-03-11T21:10:54.795617+0000 osd.0 (osd.0) 317 : cluster 0 3.17 scrub starts 2024-03-11T21:10:55.763 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:55 smithi023 bash[29146]: cluster 2024-03-11T21:10:54.801305+0000 osd.0 (osd.0) 318 : cluster 0 3.17 scrub ok 2024-03-11T21:10:55.763 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:55 smithi023 bash[29146]: cluster 2024-03-11T21:10:55.010589+0000 osd.3 (osd.3) 412 : cluster 0 2.1 scrub starts 2024-03-11T21:10:55.763 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:55 smithi023 bash[29146]: cluster 2024-03-11T21:10:55.016266+0000 osd.3 (osd.3) 413 : cluster 0 2.1 scrub ok 2024-03-11T21:10:56.113 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:56.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:56.115 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:10:56.140 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:10:55 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:10:56.216 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:10:56.242 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:10:56.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:56.247 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:10:56.348 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:10:56.365 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:10:56.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:56.367 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:10:56.468 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:10:56.494 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:10:56.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:56.502 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:10:56.602 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:10:56.613 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:56 smithi146 bash[23809]: cluster 2024-03-11T21:10:55.538763+0000 osd.1 (osd.1) 227 : cluster 0 4.1c scrub starts 2024-03-11T21:10:56.613 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:56 smithi146 bash[23809]: cluster 2024-03-11T21:10:55.542638+0000 osd.1 (osd.1) 228 : cluster 0 4.1c scrub ok 2024-03-11T21:10:56.613 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:56 smithi146 bash[23809]: cluster 2024-03-11T21:10:55.717292+0000 osd.6 (osd.6) 342 : cluster 0 5.2 scrub starts 2024-03-11T21:10:56.613 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:56 smithi146 bash[23809]: cluster 2024-03-11T21:10:55.753804+0000 osd.6 (osd.6) 343 : cluster 0 5.2 scrub ok 2024-03-11T21:10:56.613 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:56 smithi146 bash[23809]: cluster 2024-03-11T21:10:55.757483+0000 osd.0 (osd.0) 319 : cluster 0 3.6 scrub starts 2024-03-11T21:10:56.613 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:56 smithi146 bash[23809]: cluster 2024-03-11T21:10:55.759331+0000 osd.0 (osd.0) 320 : cluster 0 3.6 scrub ok 2024-03-11T21:10:56.638 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:10:56.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:56.642 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:10:56.743 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:10:56.767 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:10:56.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:56.771 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:10:56.872 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:10:56.893 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:10:56.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:56.895 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:10:56.996 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:57.006 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:56 smithi023 bash[21372]: cluster 2024-03-11T21:10:55.538763+0000 osd.1 (osd.1) 227 : cluster 0 4.1c scrub starts 2024-03-11T21:10:57.007 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:56 smithi023 bash[21372]: cluster 2024-03-11T21:10:55.542638+0000 osd.1 (osd.1) 228 : cluster 0 4.1c scrub ok 2024-03-11T21:10:57.007 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:56 smithi023 bash[21372]: cluster 2024-03-11T21:10:55.717292+0000 osd.6 (osd.6) 342 : cluster 0 5.2 scrub starts 2024-03-11T21:10:57.007 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:56 smithi023 bash[21372]: cluster 2024-03-11T21:10:55.753804+0000 osd.6 (osd.6) 343 : cluster 0 5.2 scrub ok 2024-03-11T21:10:57.007 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:56 smithi023 bash[21372]: cluster 2024-03-11T21:10:55.757483+0000 osd.0 (osd.0) 319 : cluster 0 3.6 scrub starts 2024-03-11T21:10:57.007 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:56 smithi023 bash[21372]: cluster 2024-03-11T21:10:55.759331+0000 osd.0 (osd.0) 320 : cluster 0 3.6 scrub ok 2024-03-11T21:10:57.007 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:56 smithi023 bash[29146]: cluster 2024-03-11T21:10:55.538763+0000 osd.1 (osd.1) 227 : cluster 0 4.1c scrub starts 2024-03-11T21:10:57.007 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:56 smithi023 bash[29146]: cluster 2024-03-11T21:10:55.542638+0000 osd.1 (osd.1) 228 : cluster 0 4.1c scrub ok 2024-03-11T21:10:57.007 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:56 smithi023 bash[29146]: cluster 2024-03-11T21:10:55.717292+0000 osd.6 (osd.6) 342 : cluster 0 5.2 scrub starts 2024-03-11T21:10:57.007 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:56 smithi023 bash[29146]: cluster 2024-03-11T21:10:55.753804+0000 osd.6 (osd.6) 343 : cluster 0 5.2 scrub ok 2024-03-11T21:10:57.007 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:56 smithi023 bash[29146]: cluster 2024-03-11T21:10:55.757483+0000 osd.0 (osd.0) 319 : cluster 0 3.6 scrub starts 2024-03-11T21:10:57.008 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:56 smithi023 bash[29146]: cluster 2024-03-11T21:10:55.759331+0000 osd.0 (osd.0) 320 : cluster 0 3.6 scrub ok 2024-03-11T21:10:57.020 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:57.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:57.022 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:10:57.123 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:10:57.149 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:10:57.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:57.151 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:10:57.252 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:57.289 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:57.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:57.294 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:10:57.395 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:10:57.425 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:10:57.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:57.427 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:10:57.529 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:10:57.551 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:10:57.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:57.553 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:10:57.654 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:10:57.674 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:10:57.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:57.678 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:10:57.779 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:57.797 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:57.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:57.799 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:10:57.899 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:58.007 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:57 smithi146 bash[23809]: cluster 2024-03-11T21:10:55.818651+0000 osd.7 (osd.7) 461 : cluster 0 10.1 scrub starts 2024-03-11T21:10:58.008 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:57 smithi146 bash[23809]: audit 2024-03-11T21:10:55.857091+0000 mgr.y (mgr.24380) 745 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:10:58.008 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:57 smithi146 bash[23809]: cluster 2024-03-11T21:10:55.964488+0000 osd.7 (osd.7) 462 : cluster 0 10.1 scrub ok 2024-03-11T21:10:58.008 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:57 smithi146 bash[23809]: cluster 2024-03-11T21:10:56.333223+0000 mgr.y (mgr.24380) 746 : cluster 0 pgmap v844: 148 pgs: 148 active+clean; 721 MiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.13k op/s 2024-03-11T21:10:58.008 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:57 smithi146 bash[23809]: cluster 2024-03-11T21:10:56.737575+0000 osd.6 (osd.6) 344 : cluster 0 10.e scrub starts 2024-03-11T21:10:58.008 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:57 smithi146 bash[23809]: cluster 2024-03-11T21:10:56.776971+0000 osd.0 (osd.0) 321 : cluster 0 4.16 scrub starts 2024-03-11T21:10:58.008 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:57 smithi146 bash[23809]: cluster 2024-03-11T21:10:56.778707+0000 osd.0 (osd.0) 322 : cluster 0 4.16 scrub ok 2024-03-11T21:10:58.027 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:57 smithi023 bash[21372]: cluster 2024-03-11T21:10:55.818651+0000 osd.7 (osd.7) 461 : cluster 0 10.1 scrub starts 2024-03-11T21:10:58.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:57 smithi023 bash[21372]: audit 2024-03-11T21:10:55.857091+0000 mgr.y (mgr.24380) 745 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:10:58.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:57 smithi023 bash[21372]: cluster 2024-03-11T21:10:55.964488+0000 osd.7 (osd.7) 462 : cluster 0 10.1 scrub ok 2024-03-11T21:10:58.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:57 smithi023 bash[21372]: cluster 2024-03-11T21:10:56.333223+0000 mgr.y (mgr.24380) 746 : cluster 0 pgmap v844: 148 pgs: 148 active+clean; 721 MiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.13k op/s 2024-03-11T21:10:58.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:57 smithi023 bash[21372]: cluster 2024-03-11T21:10:56.737575+0000 osd.6 (osd.6) 344 : cluster 0 10.e scrub starts 2024-03-11T21:10:58.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:57 smithi023 bash[21372]: cluster 2024-03-11T21:10:56.776971+0000 osd.0 (osd.0) 321 : cluster 0 4.16 scrub starts 2024-03-11T21:10:58.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:57 smithi023 bash[21372]: cluster 2024-03-11T21:10:56.778707+0000 osd.0 (osd.0) 322 : cluster 0 4.16 scrub ok 2024-03-11T21:10:58.028 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:57 smithi023 bash[29146]: cluster 2024-03-11T21:10:55.818651+0000 osd.7 (osd.7) 461 : cluster 0 10.1 scrub starts 2024-03-11T21:10:58.028 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:57 smithi023 bash[29146]: audit 2024-03-11T21:10:55.857091+0000 mgr.y (mgr.24380) 745 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:10:58.028 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:57 smithi023 bash[29146]: cluster 2024-03-11T21:10:55.964488+0000 osd.7 (osd.7) 462 : cluster 0 10.1 scrub ok 2024-03-11T21:10:58.029 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:57 smithi023 bash[29146]: cluster 2024-03-11T21:10:56.333223+0000 mgr.y (mgr.24380) 746 : cluster 0 pgmap v844: 148 pgs: 148 active+clean; 721 MiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.13k op/s 2024-03-11T21:10:58.029 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:57 smithi023 bash[29146]: cluster 2024-03-11T21:10:56.737575+0000 osd.6 (osd.6) 344 : cluster 0 10.e scrub starts 2024-03-11T21:10:58.029 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:57 smithi023 bash[29146]: cluster 2024-03-11T21:10:56.776971+0000 osd.0 (osd.0) 321 : cluster 0 4.16 scrub starts 2024-03-11T21:10:58.029 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:57 smithi023 bash[29146]: cluster 2024-03-11T21:10:56.778707+0000 osd.0 (osd.0) 322 : cluster 0 4.16 scrub ok 2024-03-11T21:10:58.049 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:58.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:58.054 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:10:58.092 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-03-11T21:10:58.155 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:10:58.187 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:10:58.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:58.190 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:10:58.291 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:58.320 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:10:58.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:58.326 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:10:58.428 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:10:58.437 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:10:58 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:10:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:10:58.453 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:10:58.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:58.456 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:10:58.557 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:10:58.591 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:10:58.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:58.598 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:10:58.699 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:10:58.709 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:58 smithi023 bash[21372]: cluster 2024-03-11T21:10:56.858262+0000 osd.7 (osd.7) 463 : cluster 0 3.13 scrub starts 2024-03-11T21:10:58.709 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:58 smithi023 bash[21372]: cluster 2024-03-11T21:10:56.860168+0000 osd.7 (osd.7) 464 : cluster 0 3.13 scrub ok 2024-03-11T21:10:58.710 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:58 smithi023 bash[21372]: cluster 2024-03-11T21:10:57.502304+0000 osd.5 (osd.5) 421 : cluster 0 2.c scrub starts 2024-03-11T21:10:58.710 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:58 smithi023 bash[21372]: cluster 2024-03-11T21:10:57.507327+0000 osd.6 (osd.6) 345 : cluster 0 10.e scrub ok 2024-03-11T21:10:58.710 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:58 smithi023 bash[21372]: cluster 2024-03-11T21:10:57.526929+0000 osd.5 (osd.5) 422 : cluster 0 2.c scrub ok 2024-03-11T21:10:58.710 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:58 smithi023 bash[21372]: cluster 2024-03-11T21:10:57.575339+0000 osd.1 (osd.1) 229 : cluster 0 10.2 scrub starts 2024-03-11T21:10:58.710 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:58 smithi023 bash[21372]: cluster 2024-03-11T21:10:57.769919+0000 osd.1 (osd.1) 230 : cluster 0 10.2 scrub ok 2024-03-11T21:10:58.710 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:58 smithi023 bash[29146]: cluster 2024-03-11T21:10:56.858262+0000 osd.7 (osd.7) 463 : cluster 0 3.13 scrub starts 2024-03-11T21:10:58.710 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:58 smithi023 bash[29146]: cluster 2024-03-11T21:10:56.860168+0000 osd.7 (osd.7) 464 : cluster 0 3.13 scrub ok 2024-03-11T21:10:58.710 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:58 smithi023 bash[29146]: cluster 2024-03-11T21:10:57.502304+0000 osd.5 (osd.5) 421 : cluster 0 2.c scrub starts 2024-03-11T21:10:58.710 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:58 smithi023 bash[29146]: cluster 2024-03-11T21:10:57.507327+0000 osd.6 (osd.6) 345 : cluster 0 10.e scrub ok 2024-03-11T21:10:58.710 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:58 smithi023 bash[29146]: cluster 2024-03-11T21:10:57.526929+0000 osd.5 (osd.5) 422 : cluster 0 2.c scrub ok 2024-03-11T21:10:58.710 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:58 smithi023 bash[29146]: cluster 2024-03-11T21:10:57.575339+0000 osd.1 (osd.1) 229 : cluster 0 10.2 scrub starts 2024-03-11T21:10:58.710 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:58 smithi023 bash[29146]: cluster 2024-03-11T21:10:57.769919+0000 osd.1 (osd.1) 230 : cluster 0 10.2 scrub ok 2024-03-11T21:10:58.727 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:10:58.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:58.730 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:10:58.831 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:10:58.853 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:10:58.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:58.857 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:10:58.958 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:10:58.986 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:10:58.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:58.990 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:10:59.007 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:58 smithi146 bash[23809]: cluster 2024-03-11T21:10:56.858262+0000 osd.7 (osd.7) 463 : cluster 0 3.13 scrub starts 2024-03-11T21:10:59.008 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:58 smithi146 bash[23809]: cluster 2024-03-11T21:10:56.860168+0000 osd.7 (osd.7) 464 : cluster 0 3.13 scrub ok 2024-03-11T21:10:59.008 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:58 smithi146 bash[23809]: cluster 2024-03-11T21:10:57.502304+0000 osd.5 (osd.5) 421 : cluster 0 2.c scrub starts 2024-03-11T21:10:59.008 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:58 smithi146 bash[23809]: cluster 2024-03-11T21:10:57.507327+0000 osd.6 (osd.6) 345 : cluster 0 10.e scrub ok 2024-03-11T21:10:59.008 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:58 smithi146 bash[23809]: cluster 2024-03-11T21:10:57.526929+0000 osd.5 (osd.5) 422 : cluster 0 2.c scrub ok 2024-03-11T21:10:59.008 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:58 smithi146 bash[23809]: cluster 2024-03-11T21:10:57.575339+0000 osd.1 (osd.1) 229 : cluster 0 10.2 scrub starts 2024-03-11T21:10:59.008 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:58 smithi146 bash[23809]: cluster 2024-03-11T21:10:57.769919+0000 osd.1 (osd.1) 230 : cluster 0 10.2 scrub ok 2024-03-11T21:10:59.091 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:10:59.130 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:10:59.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:59.136 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:10:59.237 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:10:59.275 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:10:59.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:59.279 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:10:59.380 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:59.399 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:59.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:59.400 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:10:59.501 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:10:59.529 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:10:59.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:59.531 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:10:59.632 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:10:59.642 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:59 smithi146 bash[23809]: cluster 2024-03-11T21:10:57.861365+0000 osd.7 (osd.7) 465 : cluster 0 3.9 scrub starts 2024-03-11T21:10:59.642 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:59 smithi146 bash[23809]: cluster 2024-03-11T21:10:57.864796+0000 osd.7 (osd.7) 466 : cluster 0 3.9 scrub ok 2024-03-11T21:10:59.643 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:59 smithi146 bash[23809]: cluster 2024-03-11T21:10:58.336357+0000 mgr.y (mgr.24380) 747 : cluster 0 pgmap v845: 148 pgs: 148 active+clean; 704 MiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 783 op/s 2024-03-11T21:10:59.643 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:59 smithi146 bash[23809]: cluster 2024-03-11T21:10:58.596801+0000 osd.1 (osd.1) 231 : cluster 0 2.d scrub starts 2024-03-11T21:10:59.643 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:59 smithi146 bash[23809]: cluster 2024-03-11T21:10:58.600338+0000 osd.1 (osd.1) 232 : cluster 0 2.d scrub ok 2024-03-11T21:10:59.643 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:59 smithi146 bash[23809]: cluster 2024-03-11T21:10:58.701010+0000 osd.6 (osd.6) 346 : cluster 0 10.7 scrub starts 2024-03-11T21:10:59.643 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:59 smithi146 bash[23809]: cluster 2024-03-11T21:10:58.711371+0000 osd.0 (osd.0) 323 : cluster 0 3.1 scrub starts 2024-03-11T21:10:59.643 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:59 smithi146 bash[23809]: cluster 2024-03-11T21:10:58.718009+0000 osd.0 (osd.0) 324 : cluster 0 3.1 scrub ok 2024-03-11T21:10:59.643 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:10:59 smithi146 bash[23809]: cluster 2024-03-11T21:10:59.034152+0000 osd.6 (osd.6) 347 : cluster 0 10.7 scrub ok 2024-03-11T21:10:59.659 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:10:59.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:59.662 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:10:59.763 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:59.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:59 smithi023 bash[21372]: cluster 2024-03-11T21:10:57.861365+0000 osd.7 (osd.7) 465 : cluster 0 3.9 scrub starts 2024-03-11T21:10:59.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:59 smithi023 bash[21372]: cluster 2024-03-11T21:10:57.864796+0000 osd.7 (osd.7) 466 : cluster 0 3.9 scrub ok 2024-03-11T21:10:59.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:59 smithi023 bash[21372]: cluster 2024-03-11T21:10:58.336357+0000 mgr.y (mgr.24380) 747 : cluster 0 pgmap v845: 148 pgs: 148 active+clean; 704 MiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 783 op/s 2024-03-11T21:10:59.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:59 smithi023 bash[21372]: cluster 2024-03-11T21:10:58.596801+0000 osd.1 (osd.1) 231 : cluster 0 2.d scrub starts 2024-03-11T21:10:59.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:59 smithi023 bash[21372]: cluster 2024-03-11T21:10:58.600338+0000 osd.1 (osd.1) 232 : cluster 0 2.d scrub ok 2024-03-11T21:10:59.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:59 smithi023 bash[21372]: cluster 2024-03-11T21:10:58.701010+0000 osd.6 (osd.6) 346 : cluster 0 10.7 scrub starts 2024-03-11T21:10:59.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:59 smithi023 bash[21372]: cluster 2024-03-11T21:10:58.711371+0000 osd.0 (osd.0) 323 : cluster 0 3.1 scrub starts 2024-03-11T21:10:59.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:59 smithi023 bash[21372]: cluster 2024-03-11T21:10:58.718009+0000 osd.0 (osd.0) 324 : cluster 0 3.1 scrub ok 2024-03-11T21:10:59.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:10:59 smithi023 bash[21372]: cluster 2024-03-11T21:10:59.034152+0000 osd.6 (osd.6) 347 : cluster 0 10.7 scrub ok 2024-03-11T21:10:59.779 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:59 smithi023 bash[29146]: cluster 2024-03-11T21:10:57.861365+0000 osd.7 (osd.7) 465 : cluster 0 3.9 scrub starts 2024-03-11T21:10:59.780 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:59 smithi023 bash[29146]: cluster 2024-03-11T21:10:57.864796+0000 osd.7 (osd.7) 466 : cluster 0 3.9 scrub ok 2024-03-11T21:10:59.780 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:59 smithi023 bash[29146]: cluster 2024-03-11T21:10:58.336357+0000 mgr.y (mgr.24380) 747 : cluster 0 pgmap v845: 148 pgs: 148 active+clean; 704 MiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 783 op/s 2024-03-11T21:10:59.780 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:59 smithi023 bash[29146]: cluster 2024-03-11T21:10:58.596801+0000 osd.1 (osd.1) 231 : cluster 0 2.d scrub starts 2024-03-11T21:10:59.780 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:59 smithi023 bash[29146]: cluster 2024-03-11T21:10:58.600338+0000 osd.1 (osd.1) 232 : cluster 0 2.d scrub ok 2024-03-11T21:10:59.780 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:59 smithi023 bash[29146]: cluster 2024-03-11T21:10:58.701010+0000 osd.6 (osd.6) 346 : cluster 0 10.7 scrub starts 2024-03-11T21:10:59.780 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:59 smithi023 bash[29146]: cluster 2024-03-11T21:10:58.711371+0000 osd.0 (osd.0) 323 : cluster 0 3.1 scrub starts 2024-03-11T21:10:59.781 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:59 smithi023 bash[29146]: cluster 2024-03-11T21:10:58.718009+0000 osd.0 (osd.0) 324 : cluster 0 3.1 scrub ok 2024-03-11T21:10:59.781 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:10:59 smithi023 bash[29146]: cluster 2024-03-11T21:10:59.034152+0000 osd.6 (osd.6) 347 : cluster 0 10.7 scrub ok 2024-03-11T21:10:59.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:59.796 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:10:59.796 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:10:59.897 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:10:59.926 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:10:59.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:10:59.932 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:11:00.032 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:00.074 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:00.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:00.079 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:11:00.180 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:00.209 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:00.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:00.211 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:11:00.312 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:00.348 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:00.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:00.353 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:11:00.454 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:11:00.498 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:11:00.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:00.500 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:11:00.601 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:11:00.629 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:11:00.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:00.631 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:11:00.732 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:11:00.740 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:00 smithi146 bash[23809]: cluster 2024-03-11T21:10:59.618213+0000 osd.1 (osd.1) 233 : cluster 0 3.4 scrub starts 2024-03-11T21:11:00.740 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:00 smithi146 bash[23809]: cluster 2024-03-11T21:10:59.629623+0000 osd.1 (osd.1) 234 : cluster 0 3.4 scrub ok 2024-03-11T21:11:00.740 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:00 smithi146 bash[23809]: cluster 2024-03-11T21:11:00.429274+0000 osd.5 (osd.5) 423 : cluster 0 6.1a scrub starts 2024-03-11T21:11:00.754 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:11:00.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:00.759 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:11:00.860 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:00.890 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:00.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:00.895 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:11:00.996 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:11:01.035 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:11:01.036 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:00 smithi023 bash[21372]: cluster 2024-03-11T21:10:59.618213+0000 osd.1 (osd.1) 233 : cluster 0 3.4 scrub starts 2024-03-11T21:11:01.036 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:00 smithi023 bash[21372]: cluster 2024-03-11T21:10:59.629623+0000 osd.1 (osd.1) 234 : cluster 0 3.4 scrub ok 2024-03-11T21:11:01.036 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:00 smithi023 bash[21372]: cluster 2024-03-11T21:11:00.429274+0000 osd.5 (osd.5) 423 : cluster 0 6.1a scrub starts 2024-03-11T21:11:01.036 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:00 smithi023 bash[29146]: cluster 2024-03-11T21:10:59.618213+0000 osd.1 (osd.1) 233 : cluster 0 3.4 scrub starts 2024-03-11T21:11:01.036 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:00 smithi023 bash[29146]: cluster 2024-03-11T21:10:59.629623+0000 osd.1 (osd.1) 234 : cluster 0 3.4 scrub ok 2024-03-11T21:11:01.036 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:00 smithi023 bash[29146]: cluster 2024-03-11T21:11:00.429274+0000 osd.5 (osd.5) 423 : cluster 0 6.1a scrub starts 2024-03-11T21:11:01.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:01.037 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:11:01.138 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:11:01.160 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:11:01.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:01.207 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:11:01.207 INFO:teuthology.orchestra.run.smithi023.stdout:osd.0: {} 2024-03-11T21:11:01.242 INFO:teuthology.orchestra.run.smithi023.stdout:osd.1: {} 2024-03-11T21:11:01.285 INFO:teuthology.orchestra.run.smithi023.stdout:osd.2: {} 2024-03-11T21:11:01.307 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:11:01.344 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:11:01.347 INFO:teuthology.orchestra.run.smithi023.stdout:osd.3: {} 2024-03-11T21:11:01.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:01.348 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:11:01.388 INFO:teuthology.orchestra.run.smithi023.stdout:osd.4: {} 2024-03-11T21:11:01.438 INFO:teuthology.orchestra.run.smithi023.stdout:osd.5: {} 2024-03-11T21:11:01.450 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:11:01.476 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:11:01.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:01.479 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:11:01.499 INFO:teuthology.orchestra.run.smithi023.stdout:osd.6: {} 2024-03-11T21:11:01.539 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-11T21:11:01.553 INFO:teuthology.orchestra.run.smithi023.stdout:osd.7: {} 2024-03-11T21:11:01.555 INFO:teuthology.orchestra.run.smithi023.stderr:dumped all 2024-03-11T21:11:01.570 INFO:teuthology.orchestra.run.smithi023.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:11:01.570 INFO:teuthology.orchestra.run.smithi023.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:11:01.570 INFO:teuthology.orchestra.run.smithi023.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:11:01.570 INFO:teuthology.orchestra.run.smithi023.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:11:01.570 INFO:teuthology.orchestra.run.smithi023.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:11:01.570 INFO:teuthology.orchestra.run.smithi023.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:11:01.570 INFO:teuthology.orchestra.run.smithi023.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:11:01.570 INFO:teuthology.orchestra.run.smithi023.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:11:01.580 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:11:01.608 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:11:01.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:01.615 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:11:01.638 INFO:teuthology.orchestra.run.smithi023.stderr:nodeep-scrub is unset 2024-03-11T21:11:01.716 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:01.727 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:01 smithi146 bash[23809]: cluster 2024-03-11T21:11:00.337101+0000 mgr.y (mgr.24380) 748 : cluster 0 pgmap v846: 148 pgs: 148 active+clean; 558 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 952 op/s 2024-03-11T21:11:01.727 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:01 smithi146 bash[23809]: cluster 2024-03-11T21:11:00.484534+0000 osd.5 (osd.5) 424 : cluster 0 6.1a scrub ok 2024-03-11T21:11:01.728 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:01 smithi146 bash[23809]: cluster 2024-03-11T21:11:00.765468+0000 osd.0 (osd.0) 325 : cluster 0 4.3 scrub starts 2024-03-11T21:11:01.728 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:01 smithi146 bash[23809]: cluster 2024-03-11T21:11:00.767149+0000 osd.0 (osd.0) 326 : cluster 0 4.3 scrub ok 2024-03-11T21:11:01.728 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:01 smithi146 bash[23809]: audit 2024-03-11T21:11:01.139382+0000 mon.b (mon.1) 45 : audit 0 from='client.? 172.21.15.23:0/1127833597' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:11:01.728 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:01 smithi146 bash[23809]: cluster 2024-03-11T21:11:01.145465+0000 osd.3 (osd.3) 414 : cluster 0 2.e scrub starts 2024-03-11T21:11:01.728 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:01 smithi146 bash[23809]: cluster 2024-03-11T21:11:01.150112+0000 osd.3 (osd.3) 415 : cluster 0 2.e scrub ok 2024-03-11T21:11:01.728 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:01 smithi146 bash[23809]: cluster 2024-03-11T21:11:01.412516+0000 osd.5 (osd.5) 425 : cluster 0 2.12 scrub starts 2024-03-11T21:11:01.729 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:01 smithi146 bash[23809]: cluster 2024-03-11T21:11:01.413624+0000 osd.5 (osd.5) 426 : cluster 0 2.12 scrub ok 2024-03-11T21:11:01.729 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:01 smithi146 bash[23809]: audit 2024-03-11T21:11:01.576238+0000 mon.a (mon.0) 1741 : audit 1 from='client.? 172.21.15.23:0/1444504463' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:11:01.744 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:01.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:01.746 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:11:01.847 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:01.868 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:01 smithi023 bash[21372]: cluster 2024-03-11T21:11:00.337101+0000 mgr.y (mgr.24380) 748 : cluster 0 pgmap v846: 148 pgs: 148 active+clean; 558 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 952 op/s 2024-03-11T21:11:01.868 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:01 smithi023 bash[21372]: cluster 2024-03-11T21:11:00.484534+0000 osd.5 (osd.5) 424 : cluster 0 6.1a scrub ok 2024-03-11T21:11:01.868 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:01 smithi023 bash[21372]: cluster 2024-03-11T21:11:00.765468+0000 osd.0 (osd.0) 325 : cluster 0 4.3 scrub starts 2024-03-11T21:11:01.868 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:01 smithi023 bash[21372]: cluster 2024-03-11T21:11:00.767149+0000 osd.0 (osd.0) 326 : cluster 0 4.3 scrub ok 2024-03-11T21:11:01.868 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:01 smithi023 bash[21372]: audit 2024-03-11T21:11:01.139382+0000 mon.b (mon.1) 45 : audit 0 from='client.? 172.21.15.23:0/1127833597' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:11:01.869 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:01 smithi023 bash[21372]: cluster 2024-03-11T21:11:01.145465+0000 osd.3 (osd.3) 414 : cluster 0 2.e scrub starts 2024-03-11T21:11:01.869 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:01 smithi023 bash[21372]: cluster 2024-03-11T21:11:01.150112+0000 osd.3 (osd.3) 415 : cluster 0 2.e scrub ok 2024-03-11T21:11:01.869 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:01 smithi023 bash[21372]: cluster 2024-03-11T21:11:01.412516+0000 osd.5 (osd.5) 425 : cluster 0 2.12 scrub starts 2024-03-11T21:11:01.869 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:01 smithi023 bash[21372]: cluster 2024-03-11T21:11:01.413624+0000 osd.5 (osd.5) 426 : cluster 0 2.12 scrub ok 2024-03-11T21:11:01.869 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:01 smithi023 bash[21372]: audit 2024-03-11T21:11:01.576238+0000 mon.a (mon.0) 1741 : audit 1 from='client.? 172.21.15.23:0/1444504463' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:11:01.870 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:01 smithi023 bash[29146]: cluster 2024-03-11T21:11:00.337101+0000 mgr.y (mgr.24380) 748 : cluster 0 pgmap v846: 148 pgs: 148 active+clean; 558 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 952 op/s 2024-03-11T21:11:01.870 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:01 smithi023 bash[29146]: cluster 2024-03-11T21:11:00.484534+0000 osd.5 (osd.5) 424 : cluster 0 6.1a scrub ok 2024-03-11T21:11:01.870 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:01 smithi023 bash[29146]: cluster 2024-03-11T21:11:00.765468+0000 osd.0 (osd.0) 325 : cluster 0 4.3 scrub starts 2024-03-11T21:11:01.870 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:01 smithi023 bash[29146]: cluster 2024-03-11T21:11:00.767149+0000 osd.0 (osd.0) 326 : cluster 0 4.3 scrub ok 2024-03-11T21:11:01.871 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:01 smithi023 bash[29146]: audit 2024-03-11T21:11:01.139382+0000 mon.b (mon.1) 45 : audit 0 from='client.? 172.21.15.23:0/1127833597' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:11:01.871 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:01 smithi023 bash[29146]: cluster 2024-03-11T21:11:01.145465+0000 osd.3 (osd.3) 414 : cluster 0 2.e scrub starts 2024-03-11T21:11:01.871 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:01 smithi023 bash[29146]: cluster 2024-03-11T21:11:01.150112+0000 osd.3 (osd.3) 415 : cluster 0 2.e scrub ok 2024-03-11T21:11:01.871 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:01 smithi023 bash[29146]: cluster 2024-03-11T21:11:01.412516+0000 osd.5 (osd.5) 425 : cluster 0 2.12 scrub starts 2024-03-11T21:11:01.871 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:01 smithi023 bash[29146]: cluster 2024-03-11T21:11:01.413624+0000 osd.5 (osd.5) 426 : cluster 0 2.12 scrub ok 2024-03-11T21:11:01.872 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:01 smithi023 bash[29146]: audit 2024-03-11T21:11:01.576238+0000 mon.a (mon.0) 1741 : audit 1 from='client.? 172.21.15.23:0/1444504463' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:11:01.877 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:01.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:01.880 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:11:01.981 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:02.009 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:02.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:02.011 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:11:02.112 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:02.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:02.140 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:02.141 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:11:02.241 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:02.270 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:02.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:02.275 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:11:02.376 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:02.407 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:02.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:02.411 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:11:02.512 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:02.539 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:02.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:02.541 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:11:02.642 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:02.675 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:02.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:02.679 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:11:02.777 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:02 smithi023 bash[21372]: audit 2024-03-11T21:11:01.528728+0000 mgr.y (mgr.24380) 749 : audit 0 from='client.15423 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:11:02.777 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:02 smithi023 bash[21372]: cluster 2024-03-11T21:11:01.627538+0000 mon.a (mon.0) 1742 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:11:02.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:02 smithi023 bash[21372]: cluster 2024-03-11T21:11:01.627572+0000 mon.a (mon.0) 1743 : cluster 1 Cluster is now healthy 2024-03-11T21:11:02.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:02 smithi023 bash[21372]: audit 2024-03-11T21:11:01.640276+0000 mon.a (mon.0) 1744 : audit 1 from='client.? 172.21.15.23:0/1444504463' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:11:02.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:02 smithi023 bash[21372]: cluster 2024-03-11T21:11:01.644644+0000 mon.a (mon.0) 1745 : cluster 0 osdmap e362: 8 total, 8 up, 8 in 2024-03-11T21:11:02.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:02 smithi023 bash[21372]: cluster 2024-03-11T21:11:01.674579+0000 osd.1 (osd.1) 235 : cluster 0 4.7 scrub starts 2024-03-11T21:11:02.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:02 smithi023 bash[21372]: cluster 2024-03-11T21:11:01.676430+0000 osd.1 (osd.1) 236 : cluster 0 4.7 scrub ok 2024-03-11T21:11:02.778 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:02 smithi023 bash[29146]: audit 2024-03-11T21:11:01.528728+0000 mgr.y (mgr.24380) 749 : audit 0 from='client.15423 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:11:02.778 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:02 smithi023 bash[29146]: cluster 2024-03-11T21:11:01.627538+0000 mon.a (mon.0) 1742 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:11:02.778 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:02 smithi023 bash[29146]: cluster 2024-03-11T21:11:01.627572+0000 mon.a (mon.0) 1743 : cluster 1 Cluster is now healthy 2024-03-11T21:11:02.778 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:02 smithi023 bash[29146]: audit 2024-03-11T21:11:01.640276+0000 mon.a (mon.0) 1744 : audit 1 from='client.? 172.21.15.23:0/1444504463' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:11:02.778 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:02 smithi023 bash[29146]: cluster 2024-03-11T21:11:01.644644+0000 mon.a (mon.0) 1745 : cluster 0 osdmap e362: 8 total, 8 up, 8 in 2024-03-11T21:11:02.779 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:02 smithi023 bash[29146]: cluster 2024-03-11T21:11:01.674579+0000 osd.1 (osd.1) 235 : cluster 0 4.7 scrub starts 2024-03-11T21:11:02.779 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:02 smithi023 bash[29146]: cluster 2024-03-11T21:11:01.676430+0000 osd.1 (osd.1) 236 : cluster 0 4.7 scrub ok 2024-03-11T21:11:02.780 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:11:02.814 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:11:02.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:02.817 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:11:02.918 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:11:02.938 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:11:02.938 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:11:02.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:02.939 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:11:02.953 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:02 smithi146 bash[23809]: audit 2024-03-11T21:11:01.528728+0000 mgr.y (mgr.24380) 749 : audit 0 from='client.15423 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:11:02.953 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:02 smithi146 bash[23809]: cluster 2024-03-11T21:11:01.627538+0000 mon.a (mon.0) 1742 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:11:02.953 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:02 smithi146 bash[23809]: cluster 2024-03-11T21:11:01.627572+0000 mon.a (mon.0) 1743 : cluster 1 Cluster is now healthy 2024-03-11T21:11:02.953 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:02 smithi146 bash[23809]: audit 2024-03-11T21:11:01.640276+0000 mon.a (mon.0) 1744 : audit 1 from='client.? 172.21.15.23:0/1444504463' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:11:02.953 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:02 smithi146 bash[23809]: cluster 2024-03-11T21:11:01.644644+0000 mon.a (mon.0) 1745 : cluster 0 osdmap e362: 8 total, 8 up, 8 in 2024-03-11T21:11:02.953 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:02 smithi146 bash[23809]: cluster 2024-03-11T21:11:01.674579+0000 osd.1 (osd.1) 235 : cluster 0 4.7 scrub starts 2024-03-11T21:11:02.953 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:02 smithi146 bash[23809]: cluster 2024-03-11T21:11:01.676430+0000 osd.1 (osd.1) 236 : cluster 0 4.7 scrub ok 2024-03-11T21:11:03.040 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:11:03.065 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:11:03.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:03.067 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:11:03.167 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:11:03.187 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:11:03.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:03.189 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:11:03.290 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:11:03.317 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:11:03.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:03.319 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:11:03.420 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:03.467 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:03.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:03.472 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:11:03.572 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:03.600 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:03.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:03.606 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:11:03.707 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:03.717 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:03 smithi023 bash[21372]: cluster 2024-03-11T21:11:01.829537+0000 osd.7 (osd.7) 467 : cluster 0 6.1c scrub starts 2024-03-11T21:11:03.717 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:03 smithi023 bash[21372]: cluster 2024-03-11T21:11:01.833444+0000 osd.7 (osd.7) 468 : cluster 0 6.1c scrub ok 2024-03-11T21:11:03.717 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:03 smithi023 bash[21372]: cluster 2024-03-11T21:11:02.337643+0000 mgr.y (mgr.24380) 750 : cluster 0 pgmap v848: 148 pgs: 148 active+clean; 450 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 854 op/s 2024-03-11T21:11:03.717 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:03 smithi023 bash[21372]: cluster 2024-03-11T21:11:02.678213+0000 osd.1 (osd.1) 237 : cluster 0 6.e scrub starts 2024-03-11T21:11:03.717 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:03 smithi023 bash[21372]: cluster 2024-03-11T21:11:02.715737+0000 osd.1 (osd.1) 238 : cluster 0 6.e scrub ok 2024-03-11T21:11:03.717 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:03 smithi023 bash[21372]: cluster 2024-03-11T21:11:02.743461+0000 osd.6 (osd.6) 348 : cluster 0 4.13 scrub starts 2024-03-11T21:11:03.717 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:03 smithi023 bash[21372]: cluster 2024-03-11T21:11:02.744637+0000 osd.6 (osd.6) 349 : cluster 0 4.13 scrub ok 2024-03-11T21:11:03.717 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:03 smithi023 bash[21372]: cluster 2024-03-11T21:11:03.445381+0000 osd.3 (osd.3) 416 : cluster 0 2.19 deep-scrub starts 2024-03-11T21:11:03.717 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:03 smithi023 bash[21372]: cluster 2024-03-11T21:11:03.482671+0000 osd.3 (osd.3) 417 : cluster 0 2.19 deep-scrub ok 2024-03-11T21:11:03.718 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:03 smithi023 bash[29146]: cluster 2024-03-11T21:11:01.829537+0000 osd.7 (osd.7) 467 : cluster 0 6.1c scrub starts 2024-03-11T21:11:03.718 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:03 smithi023 bash[29146]: cluster 2024-03-11T21:11:01.833444+0000 osd.7 (osd.7) 468 : cluster 0 6.1c scrub ok 2024-03-11T21:11:03.718 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:03 smithi023 bash[29146]: cluster 2024-03-11T21:11:02.337643+0000 mgr.y (mgr.24380) 750 : cluster 0 pgmap v848: 148 pgs: 148 active+clean; 450 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 854 op/s 2024-03-11T21:11:03.718 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:03 smithi023 bash[29146]: cluster 2024-03-11T21:11:02.678213+0000 osd.1 (osd.1) 237 : cluster 0 6.e scrub starts 2024-03-11T21:11:03.718 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:03 smithi023 bash[29146]: cluster 2024-03-11T21:11:02.715737+0000 osd.1 (osd.1) 238 : cluster 0 6.e scrub ok 2024-03-11T21:11:03.718 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:03 smithi023 bash[29146]: cluster 2024-03-11T21:11:02.743461+0000 osd.6 (osd.6) 348 : cluster 0 4.13 scrub starts 2024-03-11T21:11:03.718 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:03 smithi023 bash[29146]: cluster 2024-03-11T21:11:02.744637+0000 osd.6 (osd.6) 349 : cluster 0 4.13 scrub ok 2024-03-11T21:11:03.718 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:03 smithi023 bash[29146]: cluster 2024-03-11T21:11:03.445381+0000 osd.3 (osd.3) 416 : cluster 0 2.19 deep-scrub starts 2024-03-11T21:11:03.718 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:03 smithi023 bash[29146]: cluster 2024-03-11T21:11:03.482671+0000 osd.3 (osd.3) 417 : cluster 0 2.19 deep-scrub ok 2024-03-11T21:11:03.732 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:03.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:03.735 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:11:03.757 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:03 smithi146 bash[23809]: cluster 2024-03-11T21:11:01.829537+0000 osd.7 (osd.7) 467 : cluster 0 6.1c scrub starts 2024-03-11T21:11:03.757 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:03 smithi146 bash[23809]: cluster 2024-03-11T21:11:01.833444+0000 osd.7 (osd.7) 468 : cluster 0 6.1c scrub ok 2024-03-11T21:11:03.757 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:03 smithi146 bash[23809]: cluster 2024-03-11T21:11:02.337643+0000 mgr.y (mgr.24380) 750 : cluster 0 pgmap v848: 148 pgs: 148 active+clean; 450 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 854 op/s 2024-03-11T21:11:03.757 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:03 smithi146 bash[23809]: cluster 2024-03-11T21:11:02.678213+0000 osd.1 (osd.1) 237 : cluster 0 6.e scrub starts 2024-03-11T21:11:03.757 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:03 smithi146 bash[23809]: cluster 2024-03-11T21:11:02.715737+0000 osd.1 (osd.1) 238 : cluster 0 6.e scrub ok 2024-03-11T21:11:03.758 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:03 smithi146 bash[23809]: cluster 2024-03-11T21:11:02.743461+0000 osd.6 (osd.6) 348 : cluster 0 4.13 scrub starts 2024-03-11T21:11:03.758 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:03 smithi146 bash[23809]: cluster 2024-03-11T21:11:02.744637+0000 osd.6 (osd.6) 349 : cluster 0 4.13 scrub ok 2024-03-11T21:11:03.758 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:03 smithi146 bash[23809]: cluster 2024-03-11T21:11:03.445381+0000 osd.3 (osd.3) 416 : cluster 0 2.19 deep-scrub starts 2024-03-11T21:11:03.758 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:03 smithi146 bash[23809]: cluster 2024-03-11T21:11:03.482671+0000 osd.3 (osd.3) 417 : cluster 0 2.19 deep-scrub ok 2024-03-11T21:11:03.836 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:03.870 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:03.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:03.872 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:11:03.973 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:04.014 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:04.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:04.016 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:11:04.117 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:11:04.141 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:11:04.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:04.143 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:11:04.244 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:11:04.271 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:11:04.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:04.273 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:11:04.374 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:11:04.405 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:11:04.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:04.409 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:11:04.510 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:11:04.535 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:11:04.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:04.540 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:11:04.641 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:11:04.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:04.678 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:11:04.678 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:11:04.711 INFO:teuthology.orchestra.run.smithi023.stdout:{"pg_ready":true,"pg_map":{"version":846,"stamp":"2024-03-11T21:11:00.336549+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":584653248,"num_objects":9111,"num_object_clones":0,"num_object_copies":27333,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9111,"num_whiteouts":0,"num_read":2879,"num_read_kb":2725,"num_write":210294,"num_write_kb":6979038,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":190,"num_bytes_recovered":1775932,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91948,"ondisk_log_size":91948,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":5288008,"kb_used_data":1890376,"kb_used_omap":21,"kb_used_meta":3397354,"kb_avail":744673208,"statfs":{"total":767960285184,"available":762545364992,"internally_reserved":0,"allocated":1935745024,"data_stored":1933313336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22089,"internal_metadata":3478890935},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-748814336,"num_objects":-11426,"num_object_clones":0,"num_object_copies":-34278,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-11426,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":11426,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005995"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":630,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:42.475148+0000","last_change":"2024-03-11T21:07:57.433433+0000","last_active":"2024-03-11T21:10:42.475148+0000","last_peered":"2024-03-11T21:10:42.475148+0000","last_clean":"2024-03-11T21:10:42.475148+0000","last_became_active":"2024-03-11T20:50:34.338390+0000","last_became_peered":"2024-03-11T20:50:34.338390+0000","last_unstale":"2024-03-11T21:10:42.475148+0000","last_undegraded":"2024-03-11T21:10:42.475148+0000","last_fullsized":"2024-03-11T21:10:42.475148+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:07:57.433167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:07:57.433167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:45.870967+0000","scrub_duration":119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":634,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:42.473770+0000","last_change":"2024-03-11T21:08:09.344727+0000","last_active":"2024-03-11T21:10:42.473770+0000","last_peered":"2024-03-11T21:10:42.473770+0000","last_clean":"2024-03-11T21:10:42.473770+0000","last_became_active":"2024-03-11T20:50:32.210194+0000","last_became_peered":"2024-03-11T20:50:32.210194+0000","last_unstale":"2024-03-11T21:10:42.473770+0000","last_undegraded":"2024-03-11T21:10:42.473770+0000","last_fullsized":"2024-03-11T21:10:42.473770+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:08:09.344640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:08:09.344640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:45.871055+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":346,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:42.469838+0000","last_change":"2024-03-11T21:08:13.139913+0000","last_active":"2024-03-11T21:10:42.469838+0000","last_peered":"2024-03-11T21:10:42.469838+0000","last_clean":"2024-03-11T21:10:42.469838+0000","last_became_active":"2024-03-11T21:06:28.760306+0000","last_became_peered":"2024-03-11T21:06:28.760306+0000","last_unstale":"2024-03-11T21:10:42.469838+0000","last_undegraded":"2024-03-11T21:10:42.469838+0000","last_fullsized":"2024-03-11T21:10:42.469838+0000","mapping_epoch":297,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:08:13.139773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:08:13.139773+0000","last_clean_scrub_stamp":"2024-03-11T21:08:13.139773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:43.664658+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"358'30","reported_seq":694,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:52.029948+0000","last_change":"2024-03-11T21:10:52.029948+0000","last_active":"2024-03-11T21:10:52.029948+0000","last_peered":"2024-03-11T21:10:52.029948+0000","last_clean":"2024-03-11T21:10:52.029948+0000","last_became_active":"2024-03-11T20:50:30.101721+0000","last_became_peered":"2024-03-11T20:50:30.101721+0000","last_unstale":"2024-03-11T21:10:52.029948+0000","last_undegraded":"2024-03-11T21:10:52.029948+0000","last_fullsized":"2024-03-11T21:10:52.029948+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'30","last_scrub_stamp":"2024-03-11T21:10:52.029827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:10:52.029827+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:12:18.208429+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":471,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:42.469008+0000","last_change":"2024-03-11T21:09:41.738732+0000","last_active":"2024-03-11T21:10:42.469008+0000","last_peered":"2024-03-11T21:10:42.469008+0000","last_clean":"2024-03-11T21:10:42.469008+0000","last_became_active":"2024-03-11T21:05:17.316714+0000","last_became_peered":"2024-03-11T21:05:17.316714+0000","last_unstale":"2024-03-11T21:10:42.469008+0000","last_undegraded":"2024-03-11T21:10:42.469008+0000","last_fullsized":"2024-03-11T21:10:42.469008+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:41.738650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:55.545758+0000","last_clean_scrub_stamp":"2024-03-11T21:09:41.738650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:43.463899+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":475,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:42.476302+0000","last_change":"2024-03-11T21:09:53.477578+0000","last_active":"2024-03-11T21:10:42.476302+0000","last_peered":"2024-03-11T21:10:42.476302+0000","last_clean":"2024-03-11T21:10:42.476302+0000","last_became_active":"2024-03-11T21:05:17.319272+0000","last_became_peered":"2024-03-11T21:05:17.319272+0000","last_unstale":"2024-03-11T21:10:42.476302+0000","last_undegraded":"2024-03-11T21:10:42.476302+0000","last_fullsized":"2024-03-11T21:10:42.476302+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:53.477473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:54:13.660325+0000","last_clean_scrub_stamp":"2024-03-11T21:09:53.477473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:17.368384+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":644,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:42.477497+0000","last_change":"2024-03-11T21:09:49.933320+0000","last_active":"2024-03-11T21:10:42.477497+0000","last_peered":"2024-03-11T21:10:42.477497+0000","last_clean":"2024-03-11T21:10:42.477497+0000","last_became_active":"2024-03-11T20:50:28.058274+0000","last_became_peered":"2024-03-11T20:50:28.058274+0000","last_unstale":"2024-03-11T21:10:42.477497+0000","last_undegraded":"2024-03-11T21:10:42.477497+0000","last_fullsized":"2024-03-11T21:10:42.477497+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:49.933191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:09:49.933191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:11.026507+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"358'27","reported_seq":703,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:42.469755+0000","last_change":"2024-03-11T21:08:05.905831+0000","last_active":"2024-03-11T21:10:42.469755+0000","last_peered":"2024-03-11T21:10:42.469755+0000","last_clean":"2024-03-11T21:10:42.469755+0000","last_became_active":"2024-03-11T20:50:30.092638+0000","last_became_peered":"2024-03-11T20:50:30.092638+0000","last_unstale":"2024-03-11T21:10:42.469755+0000","last_undegraded":"2024-03-11T21:10:42.469755+0000","last_fullsized":"2024-03-11T21:10:42.469755+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'19","last_scrub_stamp":"2024-03-11T21:08:05.905696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:08:05.905696+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:43.664679+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":659,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:42.469875+0000","last_change":"2024-03-11T21:09:47.906047+0000","last_active":"2024-03-11T21:10:42.469875+0000","last_peered":"2024-03-11T21:10:42.469875+0000","last_clean":"2024-03-11T21:10:42.469875+0000","last_became_active":"2024-03-11T21:00:48.460694+0000","last_became_peered":"2024-03-11T21:00:48.460694+0000","last_unstale":"2024-03-11T21:10:42.469875+0000","last_undegraded":"2024-03-11T21:10:42.469875+0000","last_fullsized":"2024-03-11T21:10:42.469875+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:47.905770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:53:07.792697+0000","last_clean_scrub_stamp":"2024-03-11T21:09:47.905770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:54.823926+0000","scrub_duration":83,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":459,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:42.475363+0000","last_change":"2024-03-11T21:09:29.891712+0000","last_active":"2024-03-11T21:10:42.475363+0000","last_peered":"2024-03-11T21:10:42.475363+0000","last_clean":"2024-03-11T21:10:42.475363+0000","last_became_active":"2024-03-11T21:05:17.316491+0000","last_became_peered":"2024-03-11T21:05:17.316491+0000","last_unstale":"2024-03-11T21:10:42.475363+0000","last_undegraded":"2024-03-11T21:10:42.475363+0000","last_fullsized":"2024-03-11T21:10:42.475363+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:29.891601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:03:36.230647+0000","last_clean_scrub_stamp":"2024-03-11T21:09:29.891601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:45.871391+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":664,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:42.476621+0000","last_change":"2024-03-11T21:09:52.421945+0000","last_active":"2024-03-11T21:10:42.476621+0000","last_peered":"2024-03-11T21:10:42.476621+0000","last_clean":"2024-03-11T21:10:42.476621+0000","last_became_active":"2024-03-11T20:50:28.079225+0000","last_became_peered":"2024-03-11T20:50:28.079225+0000","last_unstale":"2024-03-11T21:10:42.476621+0000","last_undegraded":"2024-03-11T21:10:42.476621+0000","last_fullsized":"2024-03-11T21:10:42.476621+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:52.421824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:09:52.421824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:12.646223+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"358'39","reported_seq":780,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:42.469866+0000","last_change":"2024-03-11T21:09:31.716392+0000","last_active":"2024-03-11T21:10:42.469866+0000","last_peered":"2024-03-11T21:10:42.469866+0000","last_clean":"2024-03-11T21:10:42.469866+0000","last_became_active":"2024-03-11T21:00:48.462943+0000","last_became_peered":"2024-03-11T21:00:48.462943+0000","last_unstale":"2024-03-11T21:10:42.469866+0000","last_undegraded":"2024-03-11T21:10:42.469866+0000","last_fullsized":"2024-03-11T21:10:42.469866+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'27","last_scrub_stamp":"2024-03-11T21:09:31.716290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:09:31.716290+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:01.463544+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":346,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:42.469106+0000","last_change":"2024-03-11T21:09:47.379851+0000","last_active":"2024-03-11T21:10:42.469106+0000","last_peered":"2024-03-11T21:10:42.469106+0000","last_clean":"2024-03-11T21:10:42.469106+0000","last_became_active":"2024-03-11T21:06:28.757201+0000","last_became_peered":"2024-03-11T21:06:28.757201+0000","last_unstale":"2024-03-11T21:10:42.469106+0000","last_undegraded":"2024-03-11T21:10:42.469106+0000","last_fullsized":"2024-03-11T21:10:42.469106+0000","mapping_epoch":297,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:47.379734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:09:47.379734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:54.404545+0000","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":421,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:42.475000+0000","last_change":"2024-03-11T21:08:01.360006+0000","last_active":"2024-03-11T21:10:42.475000+0000","last_peered":"2024-03-11T21:10:42.475000+0000","last_clean":"2024-03-11T21:10:42.475000+0000","last_became_active":"2024-03-11T21:05:17.313388+0000","last_became_peered":"2024-03-11T21:05:17.313388+0000","last_unstale":"2024-03-11T21:10:42.475000+0000","last_undegraded":"2024-03-11T21:10:42.475000+0000","last_fullsized":"2024-03-11T21:10:42.475000+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:08:01.359878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:53:50.599029+0000","last_clean_scrub_stamp":"2024-03-11T21:08:01.359878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:45.871117+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":672,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:42.474248+0000","last_change":"2024-03-11T21:09:57.535072+0000","last_active":"2024-03-11T21:10:42.474248+0000","last_peered":"2024-03-11T21:10:42.474248+0000","last_clean":"2024-03-11T21:10:42.474248+0000","last_became_active":"2024-03-11T20:50:28.068978+0000","last_became_peered":"2024-03-11T20:50:28.068978+0000","last_unstale":"2024-03-11T21:10:42.474248+0000","last_undegraded":"2024-03-11T21:10:42.474248+0000","last_fullsized":"2024-03-11T21:10:42.474248+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:57.534967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:54:07.155130+0000","last_clean_scrub_stamp":"2024-03-11T21:09:57.534967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:10.993525+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"358'29","reported_seq":709,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:48.516649+0000","last_change":"2024-03-11T21:10:48.516700+0000","last_active":"2024-03-11T21:10:48.516649+0000","last_peered":"2024-03-11T21:10:48.516649+0000","last_clean":"2024-03-11T21:10:48.516649+0000","last_became_active":"2024-03-11T21:00:27.952093+0000","last_became_peered":"2024-03-11T21:00:27.952093+0000","last_unstale":"2024-03-11T21:10:48.516649+0000","last_undegraded":"2024-03-11T21:10:48.516649+0000","last_fullsized":"2024-03-11T21:10:48.516649+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'29","last_scrub_stamp":"2024-03-11T21:10:48.516563+0000","last_deep_scrub":"110'17","last_deep_scrub_stamp":"2024-03-11T20:53:04.802488+0000","last_clean_scrub_stamp":"2024-03-11T21:10:48.516563+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:12:00.091300+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":631,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:42.476499+0000","last_change":"2024-03-11T21:08:18.541547+0000","last_active":"2024-03-11T21:10:42.476499+0000","last_peered":"2024-03-11T21:10:42.476499+0000","last_clean":"2024-03-11T21:10:42.476499+0000","last_became_active":"2024-03-11T20:50:34.334558+0000","last_became_peered":"2024-03-11T20:50:34.334558+0000","last_unstale":"2024-03-11T21:10:42.476499+0000","last_undegraded":"2024-03-11T21:10:42.476499+0000","last_fullsized":"2024-03-11T21:10:42.476499+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:08:18.541431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:55:59.892394+0000","last_clean_scrub_stamp":"2024-03-11T21:08:18.541431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:45.871083+0000","scrub_duration":163,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":17,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:42.475570+0000","last_change":"2024-03-11T21:10:38.442610+0000","last_active":"2024-03-11T21:10:42.475570+0000","last_peered":"2024-03-11T21:10:42.475570+0000","last_clean":"2024-03-11T21:10:42.475570+0000","last_became_active":"2024-03-11T21:10:38.441649+0000","last_became_peered":"2024-03-11T21:10:38.441649+0000","last_unstale":"2024-03-11T21:10:42.475570+0000","last_undegraded":"2024-03-11T21:10:42.475570+0000","last_fullsized":"2024-03-11T21:10:42.475570+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:35.734334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:09:35.734334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:52.142128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"77'1","reported_seq":650,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:42.475468+0000","last_change":"2024-03-11T21:07:38.421921+0000","last_active":"2024-03-11T21:10:42.475468+0000","last_peered":"2024-03-11T21:10:42.475468+0000","last_clean":"2024-03-11T21:10:42.475468+0000","last_became_active":"2024-03-11T20:50:28.068840+0000","last_became_peered":"2024-03-11T20:50:28.068840+0000","last_unstale":"2024-03-11T21:10:42.475468+0000","last_undegraded":"2024-03-11T21:10:42.475468+0000","last_fullsized":"2024-03-11T21:10:42.475468+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-03-11T21:07:38.421793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:07:38.421793+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:45.870912+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"357'25","reported_seq":518,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:42.456861+0000","last_change":"2024-03-11T21:09:43.241588+0000","last_active":"2024-03-11T21:10:42.456861+0000","last_peered":"2024-03-11T21:10:42.456861+0000","last_clean":"2024-03-11T21:10:42.456861+0000","last_became_active":"2024-03-11T21:05:17.443030+0000","last_became_peered":"2024-03-11T21:05:17.443030+0000","last_unstale":"2024-03-11T21:10:42.456861+0000","last_undegraded":"2024-03-11T21:10:42.456861+0000","last_fullsized":"2024-03-11T21:10:42.456861+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'19","last_scrub_stamp":"2024-03-11T21:09:43.241403+0000","last_deep_scrub":"207'19","last_deep_scrub_stamp":"2024-03-11T21:03:39.240089+0000","last_clean_scrub_stamp":"2024-03-11T21:09:43.241403+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:01.424382+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":431,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:42.463797+0000","last_change":"2024-03-11T21:09:51.529563+0000","last_active":"2024-03-11T21:10:42.463797+0000","last_peered":"2024-03-11T21:10:42.463797+0000","last_clean":"2024-03-11T21:10:42.463797+0000","last_became_active":"2024-03-11T21:05:17.313986+0000","last_became_peered":"2024-03-11T21:05:17.313986+0000","last_unstale":"2024-03-11T21:10:42.463797+0000","last_undegraded":"2024-03-11T21:10:42.463797+0000","last_fullsized":"2024-03-11T21:10:42.463797+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:51.529432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:33.399510+0000","last_clean_scrub_stamp":"2024-03-11T21:09:51.529432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:21.400657+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":450,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:55.542734+0000","last_change":"2024-03-11T21:10:55.542768+0000","last_active":"2024-03-11T21:10:55.542734+0000","last_peered":"2024-03-11T21:10:55.542734+0000","last_clean":"2024-03-11T21:10:55.542734+0000","last_became_active":"2024-03-11T21:08:05.298552+0000","last_became_peered":"2024-03-11T21:08:05.298552+0000","last_unstale":"2024-03-11T21:10:55.542734+0000","last_undegraded":"2024-03-11T21:10:55.542734+0000","last_fullsized":"2024-03-11T21:10:55.542734+0000","mapping_epoch":319,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:10:55.542644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:24.666325+0000","last_clean_scrub_stamp":"2024-03-11T21:10:55.542644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:12:12.365126+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":673,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:42.464886+0000","last_change":"2024-03-11T21:09:44.522913+0000","last_active":"2024-03-11T21:10:42.464886+0000","last_peered":"2024-03-11T21:10:42.464886+0000","last_clean":"2024-03-11T21:10:42.464886+0000","last_became_active":"2024-03-11T21:00:27.956935+0000","last_became_peered":"2024-03-11T21:00:27.956935+0000","last_unstale":"2024-03-11T21:10:42.464886+0000","last_undegraded":"2024-03-11T21:10:42.464886+0000","last_fullsized":"2024-03-11T21:10:42.464886+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:44.522800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:09:44.522800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:08.467519+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"357'9","reported_seq":684,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:52.845194+0000","last_change":"2024-03-11T21:10:52.845265+0000","last_active":"2024-03-11T21:10:52.845194+0000","last_peered":"2024-03-11T21:10:52.845194+0000","last_clean":"2024-03-11T21:10:52.845194+0000","last_became_active":"2024-03-11T21:00:27.955296+0000","last_became_peered":"2024-03-11T21:00:27.955296+0000","last_unstale":"2024-03-11T21:10:52.845194+0000","last_undegraded":"2024-03-11T21:10:52.845194+0000","last_fullsized":"2024-03-11T21:10:52.845194+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"357'9","last_scrub_stamp":"2024-03-11T21:10:52.845050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:10:52.845050+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:12:03.949558+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":13,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":365,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:42.457405+0000","last_change":"2024-03-11T21:09:57.285484+0000","last_active":"2024-03-11T21:10:42.457405+0000","last_peered":"2024-03-11T21:10:42.457405+0000","last_clean":"2024-03-11T21:10:42.457405+0000","last_became_active":"2024-03-11T21:00:48.464694+0000","last_became_peered":"2024-03-11T21:00:48.464694+0000","last_unstale":"2024-03-11T21:10:42.457405+0000","last_undegraded":"2024-03-11T21:10:42.457405+0000","last_fullsized":"2024-03-11T21:10:42.457405+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:57.285360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:09:57.285360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:07.169016+0000","scrub_duration":119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1f","version":"212'16","reported_seq":946,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:54.109224+0000","last_change":"2024-03-11T21:09:47.536775+0000","last_active":"2024-03-11T21:10:54.109224+0000","last_peered":"2024-03-11T21:10:54.109224+0000","last_clean":"2024-03-11T21:10:54.109224+0000","last_became_active":"2024-03-11T21:00:48.464350+0000","last_became_peered":"2024-03-11T21:00:48.464350+0000","last_unstale":"2024-03-11T21:10:54.109224+0000","last_undegraded":"2024-03-11T21:10:54.109224+0000","last_fullsized":"2024-03-11T21:10:54.109224+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"212'16","last_scrub_stamp":"2024-03-11T21:09:47.536622+0000","last_deep_scrub":"212'16","last_deep_scrub_stamp":"2024-03-11T21:06:47.821221+0000","last_clean_scrub_stamp":"2024-03-11T21:09:47.536622+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:07.472659+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"77'1","reported_seq":635,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:42.473856+0000","last_change":"2024-03-11T21:07:37.448905+0000","last_active":"2024-03-11T21:10:42.473856+0000","last_peered":"2024-03-11T21:10:42.473856+0000","last_clean":"2024-03-11T21:10:42.473856+0000","last_became_active":"2024-03-11T20:50:28.068827+0000","last_became_peered":"2024-03-11T20:50:28.068827+0000","last_unstale":"2024-03-11T21:10:42.473856+0000","last_undegraded":"2024-03-11T21:10:42.473856+0000","last_fullsized":"2024-03-11T21:10:42.473856+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-03-11T21:07:37.448705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:07:37.448705+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:45.870883+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"357'21","reported_seq":714,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:42.474485+0000","last_change":"2024-03-11T21:08:21.296049+0000","last_active":"2024-03-11T21:10:42.474485+0000","last_peered":"2024-03-11T21:10:42.474485+0000","last_clean":"2024-03-11T21:10:42.474485+0000","last_became_active":"2024-03-11T21:00:48.467513+0000","last_became_peered":"2024-03-11T21:00:48.467513+0000","last_unstale":"2024-03-11T21:10:42.474485+0000","last_undegraded":"2024-03-11T21:10:42.474485+0000","last_fullsized":"2024-03-11T21:10:42.474485+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'15","last_scrub_stamp":"2024-03-11T21:08:21.295914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:08:21.295914+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:45.871306+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"105'2","reported_seq":655,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:42.475538+0000","last_change":"2024-03-11T21:09:35.469722+0000","last_active":"2024-03-11T21:10:42.475538+0000","last_peered":"2024-03-11T21:10:42.475538+0000","last_clean":"2024-03-11T21:10:42.475538+0000","last_became_active":"2024-03-11T20:50:34.327356+0000","last_became_peered":"2024-03-11T20:50:34.327356+0000","last_unstale":"2024-03-11T21:10:42.475538+0000","last_undegraded":"2024-03-11T21:10:42.475538+0000","last_fullsized":"2024-03-11T21:10:42.475538+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'2","last_scrub_stamp":"2024-03-11T21:09:35.469576+0000","last_deep_scrub":"105'2","last_deep_scrub_stamp":"2024-03-11T21:03:41.971918+0000","last_clean_scrub_stamp":"2024-03-11T21:09:35.469576+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:46.851703+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":354,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:42.472992+0000","last_change":"2024-03-11T21:09:54.403649+0000","last_active":"2024-03-11T21:10:42.472992+0000","last_peered":"2024-03-11T21:10:42.472992+0000","last_clean":"2024-03-11T21:10:42.472992+0000","last_became_active":"2024-03-11T21:06:28.759784+0000","last_became_peered":"2024-03-11T21:06:28.759784+0000","last_unstale":"2024-03-11T21:10:42.472992+0000","last_undegraded":"2024-03-11T21:10:42.472992+0000","last_fullsized":"2024-03-11T21:10:42.472992+0000","mapping_epoch":297,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:54.403136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:06:32.968650+0000","last_clean_scrub_stamp":"2024-03-11T21:09:54.403136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:21.584616+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":667,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:42.469206+0000","last_change":"2024-03-11T21:10:01.677396+0000","last_active":"2024-03-11T21:10:42.469206+0000","last_peered":"2024-03-11T21:10:42.469206+0000","last_clean":"2024-03-11T21:10:42.469206+0000","last_became_active":"2024-03-11T21:00:27.947614+0000","last_became_peered":"2024-03-11T21:00:27.947614+0000","last_unstale":"2024-03-11T21:10:42.469206+0000","last_undegraded":"2024-03-11T21:10:42.469206+0000","last_fullsized":"2024-03-11T21:10:42.469206+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:10:01.677279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:08:27.756036+0000","last_clean_scrub_stamp":"2024-03-11T21:10:01.677279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:21.767300+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"358'39","reported_seq":421,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:47.277414+0000","last_change":"2024-03-11T21:10:47.277481+0000","last_active":"2024-03-11T21:10:47.277414+0000","last_peered":"2024-03-11T21:10:47.277414+0000","last_clean":"2024-03-11T21:10:47.277414+0000","last_became_active":"2024-03-11T21:00:48.457893+0000","last_became_peered":"2024-03-11T21:00:48.457893+0000","last_unstale":"2024-03-11T21:10:47.277414+0000","last_undegraded":"2024-03-11T21:10:47.277414+0000","last_fullsized":"2024-03-11T21:10:47.277414+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'39","last_scrub_stamp":"2024-03-11T21:10:47.277324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:10:47.277324+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:12:15.705510+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":638,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:49.083119+0000","last_change":"2024-03-11T21:10:49.083160+0000","last_active":"2024-03-11T21:10:49.083119+0000","last_peered":"2024-03-11T21:10:49.083119+0000","last_clean":"2024-03-11T21:10:49.083119+0000","last_became_active":"2024-03-11T20:50:34.337878+0000","last_became_peered":"2024-03-11T20:50:34.337878+0000","last_unstale":"2024-03-11T21:10:49.083119+0000","last_undegraded":"2024-03-11T21:10:49.083119+0000","last_fullsized":"2024-03-11T21:10:49.083119+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:10:49.083027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:52:11.462480+0000","last_clean_scrub_stamp":"2024-03-11T21:10:49.083027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:59.535242+0000","scrub_duration":119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":642,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:42.474321+0000","last_change":"2024-03-11T21:10:02.099820+0000","last_active":"2024-03-11T21:10:42.474321+0000","last_peered":"2024-03-11T21:10:42.474321+0000","last_clean":"2024-03-11T21:10:42.474321+0000","last_became_active":"2024-03-11T20:50:32.646323+0000","last_became_peered":"2024-03-11T20:50:32.646323+0000","last_unstale":"2024-03-11T21:10:42.474321+0000","last_undegraded":"2024-03-11T21:10:42.474321+0000","last_fullsized":"2024-03-11T21:10:42.474321+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:10:02.099653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:10:02.099653+0000","last_clean_scrub_stamp":"2024-03-11T21:10:02.099653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:32.017072+0000","scrub_duration":179,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":680,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:42.469393+0000","last_change":"2024-03-11T21:09:49.510590+0000","last_active":"2024-03-11T21:10:42.469393+0000","last_peered":"2024-03-11T21:10:42.469393+0000","last_clean":"2024-03-11T21:10:42.469393+0000","last_became_active":"2024-03-11T20:50:28.086513+0000","last_became_peered":"2024-03-11T20:50:28.086513+0000","last_unstale":"2024-03-11T21:10:42.469393+0000","last_undegraded":"2024-03-11T21:10:42.469393+0000","last_fullsized":"2024-03-11T21:10:42.469393+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:49.510484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:09:49.510484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:59.053464+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"357'25","reported_seq":734,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:42.470911+0000","last_change":"2024-03-11T21:09:49.809926+0000","last_active":"2024-03-11T21:10:42.470911+0000","last_peered":"2024-03-11T21:10:42.470911+0000","last_clean":"2024-03-11T21:10:42.470911+0000","last_became_active":"2024-03-11T21:00:48.464707+0000","last_became_peered":"2024-03-11T21:00:48.464707+0000","last_unstale":"2024-03-11T21:10:42.470911+0000","last_undegraded":"2024-03-11T21:10:42.470911+0000","last_fullsized":"2024-03-11T21:10:42.470911+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'19","last_scrub_stamp":"2024-03-11T21:09:49.809679+0000","last_deep_scrub":"145'13","last_deep_scrub_stamp":"2024-03-11T20:55:53.560287+0000","last_clean_scrub_stamp":"2024-03-11T21:09:49.809679+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:02.002638+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":653,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:42.474572+0000","last_change":"2024-03-11T21:09:25.313083+0000","last_active":"2024-03-11T21:10:42.474572+0000","last_peered":"2024-03-11T21:10:42.474572+0000","last_clean":"2024-03-11T21:10:42.474572+0000","last_became_active":"2024-03-11T21:00:27.973405+0000","last_became_peered":"2024-03-11T21:00:27.973405+0000","last_unstale":"2024-03-11T21:10:42.474572+0000","last_undegraded":"2024-03-11T21:10:42.474572+0000","last_fullsized":"2024-03-11T21:10:42.474572+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:25.312971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:09:25.312971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:52.239778+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":647,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:42.474578+0000","last_change":"2024-03-11T21:08:24.323683+0000","last_active":"2024-03-11T21:10:42.474578+0000","last_peered":"2024-03-11T21:10:42.474578+0000","last_clean":"2024-03-11T21:10:42.474578+0000","last_became_active":"2024-03-11T21:00:48.472812+0000","last_became_peered":"2024-03-11T21:00:48.472812+0000","last_unstale":"2024-03-11T21:10:42.474578+0000","last_undegraded":"2024-03-11T21:10:42.474578+0000","last_fullsized":"2024-03-11T21:10:42.474578+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:08:24.323601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:08:24.323601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:45.871173+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":666,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:42.470732+0000","last_change":"2024-03-11T21:09:37.712819+0000","last_active":"2024-03-11T21:10:42.470732+0000","last_peered":"2024-03-11T21:10:42.470732+0000","last_clean":"2024-03-11T21:10:42.470732+0000","last_became_active":"2024-03-11T20:50:28.075760+0000","last_became_peered":"2024-03-11T20:50:28.075760+0000","last_unstale":"2024-03-11T21:10:42.470732+0000","last_undegraded":"2024-03-11T21:10:42.470732+0000","last_fullsized":"2024-03-11T21:10:42.470732+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:37.712659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:58:55.977289+0000","last_clean_scrub_stamp":"2024-03-11T21:09:37.712659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:06.735736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"358'18","reported_seq":330,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:54.801391+0000","last_change":"2024-03-11T21:10:54.801465+0000","last_active":"2024-03-11T21:10:54.801391+0000","last_peered":"2024-03-11T21:10:54.801391+0000","last_clean":"2024-03-11T21:10:54.801391+0000","last_became_active":"2024-03-11T21:06:28.763971+0000","last_became_peered":"2024-03-11T21:06:28.763971+0000","last_unstale":"2024-03-11T21:10:54.801391+0000","last_undegraded":"2024-03-11T21:10:54.801391+0000","last_fullsized":"2024-03-11T21:10:54.801391+0000","mapping_epoch":297,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'18","last_scrub_stamp":"2024-03-11T21:10:54.801312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:10:54.801312+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:12:23.639619+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":25,"num_read_kb":22,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.11","version":"186'2","reported_seq":632,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:42.473550+0000","last_change":"2024-03-11T21:08:00.343845+0000","last_active":"2024-03-11T21:10:42.473550+0000","last_peered":"2024-03-11T21:10:42.473550+0000","last_clean":"2024-03-11T21:10:42.473550+0000","last_became_active":"2024-03-11T20:50:34.339326+0000","last_became_peered":"2024-03-11T20:50:34.339326+0000","last_unstale":"2024-03-11T21:10:42.473550+0000","last_undegraded":"2024-03-11T21:10:42.473550+0000","last_fullsized":"2024-03-11T21:10:42.473550+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"186'2","last_scrub_stamp":"2024-03-11T21:08:00.343694+0000","last_deep_scrub":"186'2","last_deep_scrub_stamp":"2024-03-11T20:59:04.534850+0000","last_clean_scrub_stamp":"2024-03-11T21:08:00.343694+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:45.870942+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":475,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:42.465484+0000","last_change":"2024-03-11T21:09:32.541123+0000","last_active":"2024-03-11T21:10:42.465484+0000","last_peered":"2024-03-11T21:10:42.465484+0000","last_clean":"2024-03-11T21:10:42.465484+0000","last_became_active":"2024-03-11T21:05:17.314174+0000","last_became_peered":"2024-03-11T21:05:17.314174+0000","last_unstale":"2024-03-11T21:10:42.465484+0000","last_undegraded":"2024-03-11T21:10:42.465484+0000","last_fullsized":"2024-03-11T21:10:42.465484+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:32.540834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:09:32.540834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:45.351255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":365,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:42.457258+0000","last_change":"2024-03-11T21:10:01.220880+0000","last_active":"2024-03-11T21:10:42.457258+0000","last_peered":"2024-03-11T21:10:42.457258+0000","last_clean":"2024-03-11T21:10:42.457258+0000","last_became_active":"2024-03-11T21:00:48.455510+0000","last_became_peered":"2024-03-11T21:00:48.455510+0000","last_unstale":"2024-03-11T21:10:42.457258+0000","last_undegraded":"2024-03-11T21:10:42.457258+0000","last_fullsized":"2024-03-11T21:10:42.457258+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:10:01.220751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:10:01.220751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:22.534857+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"358'35","reported_seq":460,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:42.475254+0000","last_change":"2024-03-11T21:09:32.384059+0000","last_active":"2024-03-11T21:10:42.475254+0000","last_peered":"2024-03-11T21:10:42.475254+0000","last_clean":"2024-03-11T21:10:42.475254+0000","last_became_active":"2024-03-11T21:05:17.314507+0000","last_became_peered":"2024-03-11T21:05:17.314507+0000","last_unstale":"2024-03-11T21:10:42.475254+0000","last_undegraded":"2024-03-11T21:10:42.475254+0000","last_fullsized":"2024-03-11T21:10:42.475254+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'25","last_scrub_stamp":"2024-03-11T21:09:32.383962+0000","last_deep_scrub":"161'15","last_deep_scrub_stamp":"2024-03-11T20:57:15.125548+0000","last_clean_scrub_stamp":"2024-03-11T21:09:32.383962+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:00.885090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":672,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:42.469244+0000","last_change":"2024-03-11T21:10:02.497069+0000","last_active":"2024-03-11T21:10:42.469244+0000","last_peered":"2024-03-11T21:10:42.469244+0000","last_clean":"2024-03-11T21:10:42.469244+0000","last_became_active":"2024-03-11T21:00:48.462105+0000","last_became_peered":"2024-03-11T21:00:48.462105+0000","last_unstale":"2024-03-11T21:10:42.469244+0000","last_undegraded":"2024-03-11T21:10:42.469244+0000","last_fullsized":"2024-03-11T21:10:42.469244+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:10:02.496964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:10:02.496964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:31.206754+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":365,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:42.456400+0000","last_change":"2024-03-11T21:10:03.274386+0000","last_active":"2024-03-11T21:10:42.456400+0000","last_peered":"2024-03-11T21:10:42.456400+0000","last_clean":"2024-03-11T21:10:42.456400+0000","last_became_active":"2024-03-11T21:00:48.459043+0000","last_became_peered":"2024-03-11T21:00:48.459043+0000","last_unstale":"2024-03-11T21:10:42.456400+0000","last_undegraded":"2024-03-11T21:10:42.456400+0000","last_fullsized":"2024-03-11T21:10:42.456400+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:10:03.274266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:23.137832+0000","last_clean_scrub_stamp":"2024-03-11T21:10:03.274266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:07.308003+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":680,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:50.551711+0000","last_change":"2024-03-11T21:10:50.551744+0000","last_active":"2024-03-11T21:10:50.551711+0000","last_peered":"2024-03-11T21:10:50.551711+0000","last_clean":"2024-03-11T21:10:50.551711+0000","last_became_active":"2024-03-11T20:50:28.082737+0000","last_became_peered":"2024-03-11T20:50:28.082737+0000","last_unstale":"2024-03-11T21:10:50.551711+0000","last_undegraded":"2024-03-11T21:10:50.551711+0000","last_fullsized":"2024-03-11T21:10:50.551711+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:10:50.551642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:28.404997+0000","last_clean_scrub_stamp":"2024-03-11T21:10:50.551642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:53.871657+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"358'21","reported_seq":695,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:42.476784+0000","last_change":"2024-03-11T21:09:28.306131+0000","last_active":"2024-03-11T21:10:42.476784+0000","last_peered":"2024-03-11T21:10:42.476784+0000","last_clean":"2024-03-11T21:10:42.476784+0000","last_became_active":"2024-03-11T21:00:27.967624+0000","last_became_peered":"2024-03-11T21:00:27.967624+0000","last_unstale":"2024-03-11T21:10:42.476784+0000","last_undegraded":"2024-03-11T21:10:42.476784+0000","last_fullsized":"2024-03-11T21:10:42.476784+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'15","last_scrub_stamp":"2024-03-11T21:09:28.306002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:09:28.306002+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:48.387782+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":317,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:51.719465+0000","last_change":"2024-03-11T21:10:51.719504+0000","last_active":"2024-03-11T21:10:51.719465+0000","last_peered":"2024-03-11T21:10:51.719465+0000","last_clean":"2024-03-11T21:10:51.719465+0000","last_became_active":"2024-03-11T21:08:05.299913+0000","last_became_peered":"2024-03-11T21:08:05.299913+0000","last_unstale":"2024-03-11T21:10:51.719465+0000","last_undegraded":"2024-03-11T21:10:51.719465+0000","last_fullsized":"2024-03-11T21:10:51.719465+0000","mapping_epoch":319,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:10:51.719385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:57:16.032302+0000","last_clean_scrub_stamp":"2024-03-11T21:10:51.719385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:12:19.761008+0000","scrub_duration":191,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"112'12","reported_seq":907,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:49.104644+0000","last_change":"2024-03-11T21:09:40.700296+0000","last_active":"2024-03-11T21:10:49.104644+0000","last_peered":"2024-03-11T21:10:49.104644+0000","last_clean":"2024-03-11T21:10:49.104644+0000","last_became_active":"2024-03-11T20:50:32.650064+0000","last_became_peered":"2024-03-11T20:50:32.650064+0000","last_unstale":"2024-03-11T21:10:49.104644+0000","last_undegraded":"2024-03-11T21:10:49.104644+0000","last_fullsized":"2024-03-11T21:10:49.104644+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'12","last_scrub_stamp":"2024-03-11T21:09:40.700213+0000","last_deep_scrub":"112'12","last_deep_scrub_stamp":"2024-03-11T20:53:11.843951+0000","last_clean_scrub_stamp":"2024-03-11T21:09:40.700213+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":360,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:42.469608+0000","last_change":"2024-03-11T21:09:24.330593+0000","last_active":"2024-03-11T21:10:42.469608+0000","last_peered":"2024-03-11T21:10:42.469608+0000","last_clean":"2024-03-11T21:10:42.469608+0000","last_became_active":"2024-03-11T21:06:28.756041+0000","last_became_peered":"2024-03-11T21:06:28.756041+0000","last_unstale":"2024-03-11T21:10:42.469608+0000","last_undegraded":"2024-03-11T21:10:42.469608+0000","last_fullsized":"2024-03-11T21:10:42.469608+0000","mapping_epoch":297,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:24.330325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:06:52.862971+0000","last_clean_scrub_stamp":"2024-03-11T21:09:24.330325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":247,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"357'25","reported_seq":420,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:42.471240+0000","last_change":"2024-03-11T21:08:21.328028+0000","last_active":"2024-03-11T21:10:42.471240+0000","last_peered":"2024-03-11T21:10:42.471240+0000","last_clean":"2024-03-11T21:10:42.471240+0000","last_became_active":"2024-03-11T21:06:28.758223+0000","last_became_peered":"2024-03-11T21:06:28.758223+0000","last_unstale":"2024-03-11T21:10:42.471240+0000","last_undegraded":"2024-03-11T21:10:42.471240+0000","last_fullsized":"2024-03-11T21:10:42.471240+0000","mapping_epoch":297,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'19","last_scrub_stamp":"2024-03-11T21:08:21.327572+0000","last_deep_scrub":"119'13","last_deep_scrub_stamp":"2024-03-11T20:54:15.443472+0000","last_clean_scrub_stamp":"2024-03-11T21:08:21.327572+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:43.664727+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":340,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:42.473174+0000","last_change":"2024-03-11T21:10:01.497745+0000","last_active":"2024-03-11T21:10:42.473174+0000","last_peered":"2024-03-11T21:10:42.473174+0000","last_clean":"2024-03-11T21:10:42.473174+0000","last_became_active":"2024-03-11T21:06:28.758634+0000","last_became_peered":"2024-03-11T21:06:28.758634+0000","last_unstale":"2024-03-11T21:10:42.473174+0000","last_undegraded":"2024-03-11T21:10:42.473174+0000","last_fullsized":"2024-03-11T21:10:42.473174+0000","mapping_epoch":297,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:10:01.497631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:10:01.497631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:04.566028+0000","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"212'17","reported_seq":923,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:54.110109+0000","last_change":"2024-03-11T21:08:20.327834+0000","last_active":"2024-03-11T21:10:54.110109+0000","last_peered":"2024-03-11T21:10:54.110109+0000","last_clean":"2024-03-11T21:10:54.110109+0000","last_became_active":"2024-03-11T21:00:48.467116+0000","last_became_peered":"2024-03-11T21:00:48.467116+0000","last_unstale":"2024-03-11T21:10:54.110109+0000","last_undegraded":"2024-03-11T21:10:54.110109+0000","last_fullsized":"2024-03-11T21:10:54.110109+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"212'17","last_scrub_stamp":"2024-03-11T21:08:20.327696+0000","last_deep_scrub":"110'13","last_deep_scrub_stamp":"2024-03-11T20:53:00.434532+0000","last_clean_scrub_stamp":"2024-03-11T21:08:20.327696+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":666,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:42.469122+0000","last_change":"2024-03-11T21:09:34.741973+0000","last_active":"2024-03-11T21:10:42.469122+0000","last_peered":"2024-03-11T21:10:42.469122+0000","last_clean":"2024-03-11T21:10:42.469122+0000","last_became_active":"2024-03-11T20:50:28.070325+0000","last_became_peered":"2024-03-11T20:50:28.070325+0000","last_unstale":"2024-03-11T21:10:42.469122+0000","last_undegraded":"2024-03-11T21:10:42.469122+0000","last_fullsized":"2024-03-11T21:10:42.469122+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:34.741850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:09:34.741850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:44.232955+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"358'32","reported_seq":717,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:56.860246+0000","last_change":"2024-03-11T21:10:56.860289+0000","last_active":"2024-03-11T21:10:56.860246+0000","last_peered":"2024-03-11T21:10:56.860246+0000","last_clean":"2024-03-11T21:10:56.860246+0000","last_became_active":"2024-03-11T21:00:27.967078+0000","last_became_peered":"2024-03-11T21:00:27.967078+0000","last_unstale":"2024-03-11T21:10:56.860246+0000","last_undegraded":"2024-03-11T21:10:56.860246+0000","last_fullsized":"2024-03-11T21:10:56.860246+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'32","last_scrub_stamp":"2024-03-11T21:10:56.860175+0000","last_deep_scrub":"116'16","last_deep_scrub_stamp":"2024-03-11T20:53:39.292118+0000","last_clean_scrub_stamp":"2024-03-11T21:10:56.860175+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:12:12.317656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":645,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:42.476312+0000","last_change":"2024-03-11T21:09:38.526414+0000","last_active":"2024-03-11T21:10:42.476312+0000","last_peered":"2024-03-11T21:10:42.476312+0000","last_clean":"2024-03-11T21:10:42.476312+0000","last_became_active":"2024-03-11T21:00:27.966997+0000","last_became_peered":"2024-03-11T21:00:27.966997+0000","last_unstale":"2024-03-11T21:10:42.476312+0000","last_undegraded":"2024-03-11T21:10:42.476312+0000","last_fullsized":"2024-03-11T21:10:42.476312+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:38.526227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:55:58.016988+0000","last_clean_scrub_stamp":"2024-03-11T21:09:38.526227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:44.500997+0000","scrub_duration":59,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":655,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:42.476507+0000","last_change":"2024-03-11T21:09:24.812080+0000","last_active":"2024-03-11T21:10:42.476507+0000","last_peered":"2024-03-11T21:10:42.476507+0000","last_clean":"2024-03-11T21:10:42.476507+0000","last_became_active":"2024-03-11T21:00:48.468386+0000","last_became_peered":"2024-03-11T21:00:48.468386+0000","last_unstale":"2024-03-11T21:10:42.476507+0000","last_undegraded":"2024-03-11T21:10:42.476507+0000","last_fullsized":"2024-03-11T21:10:42.476507+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:24.811789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:09:24.811789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:53.045151+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"166'2","reported_seq":707,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:42.463875+0000","last_change":"2024-03-11T21:09:41.540284+0000","last_active":"2024-03-11T21:10:42.463875+0000","last_peered":"2024-03-11T21:10:42.463875+0000","last_clean":"2024-03-11T21:10:42.463875+0000","last_became_active":"2024-03-11T21:00:48.466071+0000","last_became_peered":"2024-03-11T21:00:48.466071+0000","last_unstale":"2024-03-11T21:10:42.463875+0000","last_undegraded":"2024-03-11T21:10:42.463875+0000","last_fullsized":"2024-03-11T21:10:42.463875+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'2","last_scrub_stamp":"2024-03-11T21:09:41.540185+0000","last_deep_scrub":"166'2","last_deep_scrub_stamp":"2024-03-11T20:57:39.868776+0000","last_clean_scrub_stamp":"2024-03-11T21:09:41.540185+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:04.813488+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1822,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"357'12","reported_seq":678,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:42.476071+0000","last_change":"2024-03-11T21:09:34.561108+0000","last_active":"2024-03-11T21:10:42.476071+0000","last_peered":"2024-03-11T21:10:42.476071+0000","last_clean":"2024-03-11T21:10:42.476071+0000","last_became_active":"2024-03-11T20:50:30.101572+0000","last_became_peered":"2024-03-11T20:50:30.101572+0000","last_unstale":"2024-03-11T21:10:42.476071+0000","last_undegraded":"2024-03-11T21:10:42.476071+0000","last_fullsized":"2024-03-11T21:10:42.476071+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'8","last_scrub_stamp":"2024-03-11T21:09:34.557847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:09:34.557847+0000","objects_scrubbed":2,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:57.326034+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":16,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":408,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:42.477326+0000","last_change":"2024-03-11T21:09:37.476199+0000","last_active":"2024-03-11T21:10:42.477326+0000","last_peered":"2024-03-11T21:10:42.477326+0000","last_clean":"2024-03-11T21:10:42.477326+0000","last_became_active":"2024-03-11T21:08:05.296676+0000","last_became_peered":"2024-03-11T21:08:05.296676+0000","last_unstale":"2024-03-11T21:10:42.477326+0000","last_undegraded":"2024-03-11T21:10:42.477326+0000","last_fullsized":"2024-03-11T21:10:42.477326+0000","mapping_epoch":319,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:37.476087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:09:37.476087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:46.780290+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":358,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:42.473243+0000","last_change":"2024-03-11T21:08:18.233619+0000","last_active":"2024-03-11T21:10:42.473243+0000","last_peered":"2024-03-11T21:10:42.473243+0000","last_clean":"2024-03-11T21:10:42.473243+0000","last_became_active":"2024-03-11T21:06:28.758133+0000","last_became_peered":"2024-03-11T21:06:28.758133+0000","last_unstale":"2024-03-11T21:10:42.473243+0000","last_undegraded":"2024-03-11T21:10:42.473243+0000","last_fullsized":"2024-03-11T21:10:42.473243+0000","mapping_epoch":297,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:08:18.233542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:54:09.537972+0000","last_clean_scrub_stamp":"2024-03-11T21:08:18.233542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:43.664538+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":438,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:52.523746+0000","last_change":"2024-03-11T21:10:52.523855+0000","last_active":"2024-03-11T21:10:52.523746+0000","last_peered":"2024-03-11T21:10:52.523746+0000","last_clean":"2024-03-11T21:10:52.523746+0000","last_became_active":"2024-03-11T21:08:05.300426+0000","last_became_peered":"2024-03-11T21:08:05.300426+0000","last_unstale":"2024-03-11T21:10:52.523746+0000","last_undegraded":"2024-03-11T21:10:52.523746+0000","last_fullsized":"2024-03-11T21:10:52.523746+0000","mapping_epoch":319,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:10:52.523598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:18.682947+0000","last_clean_scrub_stamp":"2024-03-11T21:10:52.523598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:12:05.626155+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"358'32","reported_seq":709,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:50.019288+0000","last_change":"2024-03-11T21:10:50.019374+0000","last_active":"2024-03-11T21:10:50.019288+0000","last_peered":"2024-03-11T21:10:50.019288+0000","last_clean":"2024-03-11T21:10:50.019288+0000","last_became_active":"2024-03-11T21:00:27.970878+0000","last_became_peered":"2024-03-11T21:00:27.970878+0000","last_unstale":"2024-03-11T21:10:50.019288+0000","last_undegraded":"2024-03-11T21:10:50.019288+0000","last_fullsized":"2024-03-11T21:10:50.019288+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'32","last_scrub_stamp":"2024-03-11T21:10:50.019137+0000","last_deep_scrub":"262'24","last_deep_scrub_stamp":"2024-03-11T21:03:28.128719+0000","last_clean_scrub_stamp":"2024-03-11T21:10:50.019137+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:12:02.744582+0000","scrub_duration":179,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.7","version":"361'13294","reported_seq":13501,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:11:00.279791+0000","last_change":"2024-03-11T21:10:59.034513+0000","last_active":"2024-03-11T21:11:00.279791+0000","last_peered":"2024-03-11T21:11:00.279791+0000","last_clean":"2024-03-11T21:11:00.279791+0000","last_became_active":"2024-03-11T21:07:14.959711+0000","last_became_peered":"2024-03-11T21:07:14.959711+0000","last_unstale":"2024-03-11T21:11:00.279791+0000","last_undegraded":"2024-03-11T21:11:00.279791+0000","last_fullsized":"2024-03-11T21:11:00.279791+0000","mapping_epoch":306,"log_start":"337'7500","ondisk_log_start":"337'7500","created":306,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"361'13224","last_scrub_stamp":"2024-03-11T21:10:59.034160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:07:13.846944+0000","last_clean_scrub_stamp":"2024-03-11T21:10:59.034160+0000","objects_scrubbed":541,"log_size":5794,"log_dups_size":0,"ondisk_log_size":5794,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:12:13.782468+0000","scrub_duration":339,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30015488,"num_objects":458,"num_object_clones":0,"num_object_copies":1374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":458,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13294,"num_write_kb":440064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":660,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:42.476306+0000","last_change":"2024-03-11T21:09:51.110151+0000","last_active":"2024-03-11T21:10:42.476306+0000","last_peered":"2024-03-11T21:10:42.476306+0000","last_clean":"2024-03-11T21:10:42.476306+0000","last_became_active":"2024-03-11T21:00:48.466853+0000","last_became_peered":"2024-03-11T21:00:48.466853+0000","last_unstale":"2024-03-11T21:10:42.476306+0000","last_undegraded":"2024-03-11T21:10:42.476306+0000","last_fullsized":"2024-03-11T21:10:42.476306+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:51.109823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:52:53.450885+0000","last_clean_scrub_stamp":"2024-03-11T21:09:51.109823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:07.631014+0000","scrub_duration":139,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"77'2","reported_seq":157,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:42.469404+0000","last_change":"2024-03-11T21:08:04.913072+0000","last_active":"2024-03-11T21:10:42.469404+0000","last_peered":"2024-03-11T21:10:42.469404+0000","last_clean":"2024-03-11T21:10:42.469404+0000","last_became_active":"2024-03-11T21:06:28.759838+0000","last_became_peered":"2024-03-11T21:06:28.759838+0000","last_unstale":"2024-03-11T21:10:42.469404+0000","last_undegraded":"2024-03-11T21:10:42.469404+0000","last_fullsized":"2024-03-11T21:10:42.469404+0000","mapping_epoch":297,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'2","last_scrub_stamp":"2024-03-11T21:08:04.912948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:08:04.912948+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:43.664628+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"358'32","reported_seq":770,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:53.748697+0000","last_change":"2024-03-11T21:10:53.748734+0000","last_active":"2024-03-11T21:10:53.748697+0000","last_peered":"2024-03-11T21:10:53.748697+0000","last_clean":"2024-03-11T21:10:53.748697+0000","last_became_active":"2024-03-11T21:00:48.465254+0000","last_became_peered":"2024-03-11T21:00:48.465254+0000","last_unstale":"2024-03-11T21:10:53.748697+0000","last_undegraded":"2024-03-11T21:10:53.748697+0000","last_fullsized":"2024-03-11T21:10:53.748697+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'32","last_scrub_stamp":"2024-03-11T21:10:53.748601+0000","last_deep_scrub":"100'16","last_deep_scrub_stamp":"2024-03-11T20:56:08.036331+0000","last_clean_scrub_stamp":"2024-03-11T21:10:53.748601+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:56.698446+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"286'18","reported_seq":644,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:49.106059+0000","last_change":"2024-03-11T21:10:48.267171+0000","last_active":"2024-03-11T21:10:49.106059+0000","last_peered":"2024-03-11T21:10:49.106059+0000","last_clean":"2024-03-11T21:10:49.106059+0000","last_became_active":"2024-03-11T21:05:17.317915+0000","last_became_peered":"2024-03-11T21:05:17.317915+0000","last_unstale":"2024-03-11T21:10:49.106059+0000","last_undegraded":"2024-03-11T21:10:49.106059+0000","last_fullsized":"2024-03-11T21:10:49.106059+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"286'18","last_scrub_stamp":"2024-03-11T21:10:48.267041+0000","last_deep_scrub":"212'17","last_deep_scrub_stamp":"2024-03-11T21:03:38.258850+0000","last_clean_scrub_stamp":"2024-03-11T21:10:48.267041+0000","objects_scrubbed":1,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:12:05.174671+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.c","version":"361'12948","reported_seq":13139,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:59.900739+0000","last_change":"2024-03-11T21:09:19.776322+0000","last_active":"2024-03-11T21:10:59.900739+0000","last_peered":"2024-03-11T21:10:59.900739+0000","last_clean":"2024-03-11T21:10:59.900739+0000","last_became_active":"2024-03-11T21:07:14.952564+0000","last_became_peered":"2024-03-11T21:07:14.952564+0000","last_unstale":"2024-03-11T21:10:59.900739+0000","last_undegraded":"2024-03-11T21:10:59.900739+0000","last_fullsized":"2024-03-11T21:10:59.900739+0000","mapping_epoch":306,"log_start":"343'8000","ondisk_log_start":"343'8000","created":306,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"337'7359","last_scrub_stamp":"2024-03-11T21:09:19.776149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:07:13.846944+0000","last_clean_scrub_stamp":"2024-03-11T21:09:19.776149+0000","objects_scrubbed":6099,"log_size":4948,"log_dups_size":0,"ondisk_log_size":4948,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30277632,"num_objects":462,"num_object_clones":0,"num_object_copies":1386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":462,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12948,"num_write_kb":429120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":342,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:42.469738+0000","last_change":"2024-03-11T21:08:20.317891+0000","last_active":"2024-03-11T21:10:42.469738+0000","last_peered":"2024-03-11T21:10:42.469738+0000","last_clean":"2024-03-11T21:10:42.469738+0000","last_became_active":"2024-03-11T21:06:28.755689+0000","last_became_peered":"2024-03-11T21:06:28.755689+0000","last_unstale":"2024-03-11T21:10:42.469738+0000","last_undegraded":"2024-03-11T21:10:42.469738+0000","last_fullsized":"2024-03-11T21:10:42.469738+0000","mapping_epoch":297,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:08:20.317781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:57:21.568400+0000","last_clean_scrub_stamp":"2024-03-11T21:08:20.317781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:43.664694+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":365,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:42.456857+0000","last_change":"2024-03-11T21:09:41.244145+0000","last_active":"2024-03-11T21:10:42.456857+0000","last_peered":"2024-03-11T21:10:42.456857+0000","last_clean":"2024-03-11T21:10:42.456857+0000","last_became_active":"2024-03-11T21:00:48.455733+0000","last_became_peered":"2024-03-11T21:00:48.455733+0000","last_unstale":"2024-03-11T21:10:42.456857+0000","last_undegraded":"2024-03-11T21:10:42.456857+0000","last_fullsized":"2024-03-11T21:10:42.456857+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:41.244062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:20.096874+0000","last_clean_scrub_stamp":"2024-03-11T21:09:41.244062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:05.438898+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"358'40","reported_seq":967,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:49.104513+0000","last_change":"2024-03-11T21:10:42.616379+0000","last_active":"2024-03-11T21:10:49.104513+0000","last_peered":"2024-03-11T21:10:49.104513+0000","last_clean":"2024-03-11T21:10:49.104513+0000","last_became_active":"2024-03-11T20:50:30.094977+0000","last_became_peered":"2024-03-11T20:50:30.094977+0000","last_unstale":"2024-03-11T21:10:49.104513+0000","last_undegraded":"2024-03-11T21:10:49.104513+0000","last_fullsized":"2024-03-11T21:10:49.104513+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'40","last_scrub_stamp":"2024-03-11T21:10:42.616218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:10:42.616218+0000","objects_scrubbed":8,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:48.765700+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":61,"num_read_kb":51,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":365,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:42.455660+0000","last_change":"2024-03-11T21:09:42.231430+0000","last_active":"2024-03-11T21:10:42.455660+0000","last_peered":"2024-03-11T21:10:42.455660+0000","last_clean":"2024-03-11T21:10:42.455660+0000","last_became_active":"2024-03-11T21:00:48.457901+0000","last_became_peered":"2024-03-11T21:00:48.457901+0000","last_unstale":"2024-03-11T21:10:42.455660+0000","last_undegraded":"2024-03-11T21:10:42.455660+0000","last_fullsized":"2024-03-11T21:10:42.455660+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:42.231197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:52:10.160202+0000","last_clean_scrub_stamp":"2024-03-11T21:09:42.231197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:57.587423+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.a","version":"361'13077","reported_seq":13289,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:11:00.293064+0000","last_change":"2024-03-11T21:10:44.028892+0000","last_active":"2024-03-11T21:11:00.293064+0000","last_peered":"2024-03-11T21:11:00.293064+0000","last_clean":"2024-03-11T21:11:00.293064+0000","last_became_active":"2024-03-11T21:07:14.883606+0000","last_became_peered":"2024-03-11T21:07:14.883606+0000","last_unstale":"2024-03-11T21:11:00.293064+0000","last_undegraded":"2024-03-11T21:11:00.293064+0000","last_fullsized":"2024-03-11T21:11:00.293064+0000","mapping_epoch":306,"log_start":"337'7300","ondisk_log_start":"337'7300","created":306,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"361'12161","last_scrub_stamp":"2024-03-11T21:10:44.027960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:07:13.846944+0000","last_clean_scrub_stamp":"2024-03-11T21:10:44.027960+0000","objects_scrubbed":1410,"log_size":5777,"log_dups_size":0,"ondisk_log_size":5777,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:50.733905+0000","scrub_duration":535,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31129600,"num_objects":475,"num_object_clones":0,"num_object_copies":1425,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":475,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13077,"num_write_kb":433664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":640,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:42.477584+0000","last_change":"2024-03-11T21:09:39.887725+0000","last_active":"2024-03-11T21:10:42.477584+0000","last_peered":"2024-03-11T21:10:42.477584+0000","last_clean":"2024-03-11T21:10:42.477584+0000","last_became_active":"2024-03-11T21:00:27.959408+0000","last_became_peered":"2024-03-11T21:00:27.959408+0000","last_unstale":"2024-03-11T21:10:42.477584+0000","last_undegraded":"2024-03-11T21:10:42.477584+0000","last_fullsized":"2024-03-11T21:10:42.477584+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:39.887594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:09:39.887594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:04.704389+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":680,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:42.469309+0000","last_change":"2024-03-11T21:09:55.786453+0000","last_active":"2024-03-11T21:10:42.469309+0000","last_peered":"2024-03-11T21:10:42.469309+0000","last_clean":"2024-03-11T21:10:42.469309+0000","last_became_active":"2024-03-11T21:00:48.471212+0000","last_became_peered":"2024-03-11T21:00:48.471212+0000","last_unstale":"2024-03-11T21:10:42.469309+0000","last_undegraded":"2024-03-11T21:10:42.469309+0000","last_fullsized":"2024-03-11T21:10:42.469309+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:55.786349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:09:55.786349+0000","last_clean_scrub_stamp":"2024-03-11T21:09:55.786349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:19.421312+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"358'47","reported_seq":504,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:42.469689+0000","last_change":"2024-03-11T21:09:35.262587+0000","last_active":"2024-03-11T21:10:42.469689+0000","last_peered":"2024-03-11T21:10:42.469689+0000","last_clean":"2024-03-11T21:10:42.469689+0000","last_became_active":"2024-03-11T21:06:28.756995+0000","last_became_peered":"2024-03-11T21:06:28.756995+0000","last_unstale":"2024-03-11T21:10:42.469689+0000","last_undegraded":"2024-03-11T21:10:42.469689+0000","last_fullsized":"2024-03-11T21:10:42.469689+0000","mapping_epoch":297,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'37","last_scrub_stamp":"2024-03-11T21:09:35.261983+0000","last_deep_scrub":"182'27","last_deep_scrub_stamp":"2024-03-11T20:58:52.058851+0000","last_clean_scrub_stamp":"2024-03-11T21:09:35.261983+0000","objects_scrubbed":8,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:04.285876+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":68,"num_read_kb":54,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":365,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:42.456383+0000","last_change":"2024-03-11T21:09:48.134846+0000","last_active":"2024-03-11T21:10:42.456383+0000","last_peered":"2024-03-11T21:10:42.456383+0000","last_clean":"2024-03-11T21:10:42.456383+0000","last_became_active":"2024-03-11T21:00:48.458008+0000","last_became_peered":"2024-03-11T21:00:48.458008+0000","last_unstale":"2024-03-11T21:10:42.456383+0000","last_undegraded":"2024-03-11T21:10:42.456383+0000","last_fullsized":"2024-03-11T21:10:42.456383+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:48.134729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:33.319221+0000","last_clean_scrub_stamp":"2024-03-11T21:09:48.134729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:18.134249+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.9","version":"361'12892","reported_seq":13106,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:55.576000+0000","last_change":"2024-03-11T21:10:44.941691+0000","last_active":"2024-03-11T21:10:55.576000+0000","last_peered":"2024-03-11T21:10:55.576000+0000","last_clean":"2024-03-11T21:10:55.576000+0000","last_became_active":"2024-03-11T21:07:14.877262+0000","last_became_peered":"2024-03-11T21:07:14.877262+0000","last_unstale":"2024-03-11T21:10:55.576000+0000","last_undegraded":"2024-03-11T21:10:55.576000+0000","last_fullsized":"2024-03-11T21:10:55.576000+0000","mapping_epoch":306,"log_start":"334'6800","ondisk_log_start":"334'6800","created":306,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"361'12295","last_scrub_stamp":"2024-03-11T21:10:44.941571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:07:13.846944+0000","last_clean_scrub_stamp":"2024-03-11T21:10:44.941571+0000","objects_scrubbed":1274,"log_size":6092,"log_dups_size":0,"ondisk_log_size":6092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:54.530789+0000","scrub_duration":387,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43646976,"num_objects":666,"num_object_clones":0,"num_object_copies":1998,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":666,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12892,"num_write_kb":433856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":653,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:42.476407+0000","last_change":"2024-03-11T21:09:46.434579+0000","last_active":"2024-03-11T21:10:42.476407+0000","last_peered":"2024-03-11T21:10:42.476407+0000","last_clean":"2024-03-11T21:10:42.476407+0000","last_became_active":"2024-03-11T20:50:34.321038+0000","last_became_peered":"2024-03-11T20:50:34.321038+0000","last_unstale":"2024-03-11T21:10:42.476407+0000","last_undegraded":"2024-03-11T21:10:42.476407+0000","last_fullsized":"2024-03-11T21:10:42.476407+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:46.434462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:32.951104+0000","last_clean_scrub_stamp":"2024-03-11T21:09:46.434462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:05.403803+0000","scrub_duration":83,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"358'50","reported_seq":416,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:42.455843+0000","last_change":"2024-03-11T21:10:02.247772+0000","last_active":"2024-03-11T21:10:42.455843+0000","last_peered":"2024-03-11T21:10:42.455843+0000","last_clean":"2024-03-11T21:10:42.455843+0000","last_became_active":"2024-03-11T21:00:48.456178+0000","last_became_peered":"2024-03-11T21:00:48.456178+0000","last_unstale":"2024-03-11T21:10:42.455843+0000","last_undegraded":"2024-03-11T21:10:42.455843+0000","last_fullsized":"2024-03-11T21:10:42.455843+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'38","last_scrub_stamp":"2024-03-11T21:10:02.247651+0000","last_deep_scrub":"109'26","last_deep_scrub_stamp":"2024-03-11T20:52:49.216378+0000","last_clean_scrub_stamp":"2024-03-11T21:10:02.247651+0000","objects_scrubbed":8,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:21.873016+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":58,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":104,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:55.016350+0000","last_change":"2024-03-11T21:10:55.016392+0000","last_active":"2024-03-11T21:10:55.016350+0000","last_peered":"2024-03-11T21:10:55.016350+0000","last_clean":"2024-03-11T21:10:55.016350+0000","last_became_active":"2024-03-11T21:08:05.308608+0000","last_became_peered":"2024-03-11T21:08:05.308608+0000","last_unstale":"2024-03-11T21:10:55.016350+0000","last_undegraded":"2024-03-11T21:10:55.016350+0000","last_fullsized":"2024-03-11T21:10:55.016350+0000","mapping_epoch":319,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:10:55.016272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:10:55.016272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:12:06.189069+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":357,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:42.457159+0000","last_change":"2024-03-11T21:09:26.296868+0000","last_active":"2024-03-11T21:10:42.457159+0000","last_peered":"2024-03-11T21:10:42.457159+0000","last_clean":"2024-03-11T21:10:42.457159+0000","last_became_active":"2024-03-11T21:00:48.473346+0000","last_became_peered":"2024-03-11T21:00:48.473346+0000","last_unstale":"2024-03-11T21:10:42.457159+0000","last_undegraded":"2024-03-11T21:10:42.457159+0000","last_fullsized":"2024-03-11T21:10:42.457159+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:26.296711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:09:26.296711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:53.259367+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"361'13115","reported_seq":13304,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:11:00.267789+0000","last_change":"2024-03-11T21:09:24.802751+0000","last_active":"2024-03-11T21:11:00.267789+0000","last_peered":"2024-03-11T21:11:00.267789+0000","last_clean":"2024-03-11T21:11:00.267789+0000","last_became_active":"2024-03-11T21:07:14.892853+0000","last_became_peered":"2024-03-11T21:07:14.892853+0000","last_unstale":"2024-03-11T21:11:00.267789+0000","last_undegraded":"2024-03-11T21:11:00.267789+0000","last_fullsized":"2024-03-11T21:11:00.267789+0000","mapping_epoch":306,"log_start":"336'7300","ondisk_log_start":"336'7300","created":306,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"339'7705","last_scrub_stamp":"2024-03-11T21:09:24.802619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:07:13.846944+0000","last_clean_scrub_stamp":"2024-03-11T21:09:24.802619+0000","objects_scrubbed":5943,"log_size":5815,"log_dups_size":0,"ondisk_log_size":5815,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2167,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31129600,"num_objects":475,"num_object_clones":0,"num_object_copies":1425,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":475,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13115,"num_write_kb":434880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":648,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:42.471430+0000","last_change":"2024-03-11T21:09:32.825302+0000","last_active":"2024-03-11T21:10:42.471430+0000","last_peered":"2024-03-11T21:10:42.471430+0000","last_clean":"2024-03-11T21:10:42.471430+0000","last_became_active":"2024-03-11T21:00:27.962668+0000","last_became_peered":"2024-03-11T21:00:27.962668+0000","last_unstale":"2024-03-11T21:10:42.471430+0000","last_undegraded":"2024-03-11T21:10:42.471430+0000","last_fullsized":"2024-03-11T21:10:42.471430+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:32.825166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:09:32.825166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:01.301038+0000","scrub_duration":95,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"358'35","reported_seq":689,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:42.475096+0000","last_change":"2024-03-11T21:08:11.417836+0000","last_active":"2024-03-11T21:10:42.475096+0000","last_peered":"2024-03-11T21:10:42.475096+0000","last_clean":"2024-03-11T21:10:42.475096+0000","last_became_active":"2024-03-11T20:50:30.101747+0000","last_became_peered":"2024-03-11T20:50:30.101747+0000","last_unstale":"2024-03-11T21:10:42.475096+0000","last_undegraded":"2024-03-11T21:10:42.475096+0000","last_fullsized":"2024-03-11T21:10:42.475096+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"284'25","last_scrub_stamp":"2024-03-11T21:08:11.417688+0000","last_deep_scrub":"284'25","last_deep_scrub_stamp":"2024-03-11T21:04:53.217188+0000","last_clean_scrub_stamp":"2024-03-11T21:08:11.417688+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:45.871029+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":668,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:42.469095+0000","last_change":"2024-03-11T21:09:56.781956+0000","last_active":"2024-03-11T21:10:42.469095+0000","last_peered":"2024-03-11T21:10:42.469095+0000","last_clean":"2024-03-11T21:10:42.469095+0000","last_became_active":"2024-03-11T20:50:28.085503+0000","last_became_peered":"2024-03-11T20:50:28.085503+0000","last_unstale":"2024-03-11T21:10:42.469095+0000","last_undegraded":"2024-03-11T21:10:42.469095+0000","last_fullsized":"2024-03-11T21:10:42.469095+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:56.781841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:56:03.479180+0000","last_clean_scrub_stamp":"2024-03-11T21:09:56.781841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:22.735910+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"184'130","reported_seq":934,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:42.474333+0000","last_change":"2024-03-11T21:09:59.511581+0000","last_active":"2024-03-11T21:10:42.474333+0000","last_peered":"2024-03-11T21:10:42.474333+0000","last_clean":"2024-03-11T21:10:42.474333+0000","last_became_active":"2024-03-11T20:49:52.584543+0000","last_became_peered":"2024-03-11T20:49:52.584543+0000","last_unstale":"2024-03-11T21:10:42.474333+0000","last_undegraded":"2024-03-11T21:10:42.474333+0000","last_fullsized":"2024-03-11T21:10:42.474333+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"184'130","last_scrub_stamp":"2024-03-11T21:09:59.511442+0000","last_deep_scrub":"184'130","last_deep_scrub_stamp":"2024-03-11T20:58:56.415267+0000","last_clean_scrub_stamp":"2024-03-11T21:09:59.511442+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:14.704627+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":674,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:42.465739+0000","last_change":"2024-03-11T21:09:52.505903+0000","last_active":"2024-03-11T21:10:42.465739+0000","last_peered":"2024-03-11T21:10:42.465739+0000","last_clean":"2024-03-11T21:10:42.465739+0000","last_became_active":"2024-03-11T21:00:27.959467+0000","last_became_peered":"2024-03-11T21:00:27.959467+0000","last_unstale":"2024-03-11T21:10:42.465739+0000","last_undegraded":"2024-03-11T21:10:42.465739+0000","last_fullsized":"2024-03-11T21:10:42.465739+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:52.505786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:54:22.448812+0000","last_clean_scrub_stamp":"2024-03-11T21:09:52.505786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:03.085316+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.8","version":"361'12928","reported_seq":13115,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:55.403971+0000","last_change":"2024-03-11T21:09:26.872217+0000","last_active":"2024-03-11T21:10:55.403971+0000","last_peered":"2024-03-11T21:10:55.403971+0000","last_clean":"2024-03-11T21:10:55.403971+0000","last_became_active":"2024-03-11T21:07:14.891196+0000","last_became_peered":"2024-03-11T21:07:14.891196+0000","last_unstale":"2024-03-11T21:10:55.403971+0000","last_undegraded":"2024-03-11T21:10:55.403971+0000","last_fullsized":"2024-03-11T21:10:55.403971+0000","mapping_epoch":306,"log_start":"337'7500","ondisk_log_start":"337'7500","created":306,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"341'7890","last_scrub_stamp":"2024-03-11T21:09:26.872082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:07:13.846944+0000","last_clean_scrub_stamp":"2024-03-11T21:09:26.872082+0000","objects_scrubbed":5788,"log_size":5428,"log_dups_size":0,"ondisk_log_size":5428,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1827,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45613056,"num_objects":696,"num_object_clones":0,"num_object_copies":2088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":696,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12928,"num_write_kb":435968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":365,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:42.456610+0000","last_change":"2024-03-11T21:09:56.170013+0000","last_active":"2024-03-11T21:10:42.456610+0000","last_peered":"2024-03-11T21:10:42.456610+0000","last_clean":"2024-03-11T21:10:42.456610+0000","last_became_active":"2024-03-11T21:00:48.464596+0000","last_became_peered":"2024-03-11T21:00:48.464596+0000","last_unstale":"2024-03-11T21:10:42.456610+0000","last_undegraded":"2024-03-11T21:10:42.456610+0000","last_fullsized":"2024-03-11T21:10:42.456610+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:56.169836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:09:56.169836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:13.046765+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"358'49","reported_seq":359,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:42.471755+0000","last_change":"2024-03-11T21:08:24.324884+0000","last_active":"2024-03-11T21:10:42.471755+0000","last_peered":"2024-03-11T21:10:42.471755+0000","last_clean":"2024-03-11T21:10:42.471755+0000","last_became_active":"2024-03-11T21:06:28.761186+0000","last_became_peered":"2024-03-11T21:06:28.761186+0000","last_unstale":"2024-03-11T21:10:42.471755+0000","last_undegraded":"2024-03-11T21:10:42.471755+0000","last_fullsized":"2024-03-11T21:10:42.471755+0000","mapping_epoch":297,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'35","last_scrub_stamp":"2024-03-11T21:08:24.324628+0000","last_deep_scrub":"100'21","last_deep_scrub_stamp":"2024-03-11T20:51:31.835302+0000","last_clean_scrub_stamp":"2024-03-11T21:08:24.324628+0000","objects_scrubbed":7,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:43.664568+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":55,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":396,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:42.456181+0000","last_change":"2024-03-11T21:09:46.204912+0000","last_active":"2024-03-11T21:10:42.456181+0000","last_peered":"2024-03-11T21:10:42.456181+0000","last_clean":"2024-03-11T21:10:42.456181+0000","last_became_active":"2024-03-11T21:08:05.301024+0000","last_became_peered":"2024-03-11T21:08:05.301024+0000","last_unstale":"2024-03-11T21:10:42.456181+0000","last_undegraded":"2024-03-11T21:10:42.456181+0000","last_fullsized":"2024-03-11T21:10:42.456181+0000","mapping_epoch":319,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:46.204490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:09:46.204490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:01.338837+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":669,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:42.473085+0000","last_change":"2024-03-11T21:08:16.214623+0000","last_active":"2024-03-11T21:10:42.473085+0000","last_peered":"2024-03-11T21:10:42.473085+0000","last_clean":"2024-03-11T21:10:42.473085+0000","last_became_active":"2024-03-11T21:00:48.463247+0000","last_became_peered":"2024-03-11T21:00:48.463247+0000","last_unstale":"2024-03-11T21:10:42.473085+0000","last_undegraded":"2024-03-11T21:10:42.473085+0000","last_fullsized":"2024-03-11T21:10:42.473085+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:08:16.214363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:26.839782+0000","last_clean_scrub_stamp":"2024-03-11T21:08:16.214363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:43.664709+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.d","version":"361'13181","reported_seq":13391,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:11:00.256934+0000","last_change":"2024-03-11T21:10:46.920834+0000","last_active":"2024-03-11T21:11:00.256934+0000","last_peered":"2024-03-11T21:11:00.256934+0000","last_clean":"2024-03-11T21:11:00.256934+0000","last_became_active":"2024-03-11T21:07:14.951361+0000","last_became_peered":"2024-03-11T21:07:14.951361+0000","last_unstale":"2024-03-11T21:11:00.256934+0000","last_undegraded":"2024-03-11T21:11:00.256934+0000","last_fullsized":"2024-03-11T21:11:00.256934+0000","mapping_epoch":306,"log_start":"337'7400","ondisk_log_start":"337'7400","created":306,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"361'12469","last_scrub_stamp":"2024-03-11T21:10:46.920608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:07:13.846944+0000","last_clean_scrub_stamp":"2024-03-11T21:10:46.920608+0000","objects_scrubbed":1196,"log_size":5781,"log_dups_size":0,"ondisk_log_size":5781,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:12:10.697643+0000","scrub_duration":383,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30277656,"num_objects":463,"num_object_clones":0,"num_object_copies":1389,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":463,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13181,"num_write_kb":436545,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":365,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:42.455776+0000","last_change":"2024-03-11T21:09:40.246017+0000","last_active":"2024-03-11T21:10:42.455776+0000","last_peered":"2024-03-11T21:10:42.455776+0000","last_clean":"2024-03-11T21:10:42.455776+0000","last_became_active":"2024-03-11T21:00:48.469688+0000","last_became_peered":"2024-03-11T21:00:48.469688+0000","last_unstale":"2024-03-11T21:10:42.455776+0000","last_undegraded":"2024-03-11T21:10:42.455776+0000","last_fullsized":"2024-03-11T21:10:42.455776+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:40.245934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:09:40.245934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:04.261098+0000","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"145'3","reported_seq":675,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:55.753875+0000","last_change":"2024-03-11T21:10:55.753902+0000","last_active":"2024-03-11T21:10:55.753875+0000","last_peered":"2024-03-11T21:10:55.753875+0000","last_clean":"2024-03-11T21:10:55.753875+0000","last_became_active":"2024-03-11T20:50:32.195452+0000","last_became_peered":"2024-03-11T20:50:32.195452+0000","last_unstale":"2024-03-11T21:10:55.753875+0000","last_undegraded":"2024-03-11T21:10:55.753875+0000","last_fullsized":"2024-03-11T21:10:55.753875+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'3","last_scrub_stamp":"2024-03-11T21:10:55.753809+0000","last_deep_scrub":"145'3","last_deep_scrub_stamp":"2024-03-11T20:55:52.848043+0000","last_clean_scrub_stamp":"2024-03-11T21:10:55.753809+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:12:21.469593+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0.0024049620000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":666,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:42.476545+0000","last_change":"2024-03-11T21:09:55.530420+0000","last_active":"2024-03-11T21:10:42.476545+0000","last_peered":"2024-03-11T21:10:42.476545+0000","last_clean":"2024-03-11T21:10:42.476545+0000","last_became_active":"2024-03-11T21:00:27.966749+0000","last_became_peered":"2024-03-11T21:00:27.966749+0000","last_unstale":"2024-03-11T21:10:42.476545+0000","last_undegraded":"2024-03-11T21:10:42.476545+0000","last_fullsized":"2024-03-11T21:10:42.476545+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:55.530113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:09:55.530113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:06.152784+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"358'58","reported_seq":534,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:49.106348+0000","last_change":"2024-03-11T21:09:33.321434+0000","last_active":"2024-03-11T21:10:49.106348+0000","last_peered":"2024-03-11T21:10:49.106348+0000","last_clean":"2024-03-11T21:10:49.106348+0000","last_became_active":"2024-03-11T21:00:48.473227+0000","last_became_peered":"2024-03-11T21:00:48.473227+0000","last_unstale":"2024-03-11T21:10:49.106348+0000","last_undegraded":"2024-03-11T21:10:49.106348+0000","last_fullsized":"2024-03-11T21:10:49.106348+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"212'46","last_scrub_stamp":"2024-03-11T21:09:33.321341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:09:33.321341+0000","objects_scrubbed":10,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":87,"num_read_kb":72,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"298'13","reported_seq":482,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:49.104631+0000","last_change":"2024-03-11T21:08:08.181167+0000","last_active":"2024-03-11T21:10:49.104631+0000","last_peered":"2024-03-11T21:10:49.104631+0000","last_clean":"2024-03-11T21:10:49.104631+0000","last_became_active":"2024-03-11T21:06:28.762397+0000","last_became_peered":"2024-03-11T21:06:28.762397+0000","last_unstale":"2024-03-11T21:10:49.104631+0000","last_undegraded":"2024-03-11T21:10:49.104631+0000","last_fullsized":"2024-03-11T21:10:49.104631+0000","mapping_epoch":297,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"298'13","last_scrub_stamp":"2024-03-11T21:08:08.181013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:08:08.181013+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.e","version":"361'13181","reported_seq":13385,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:11:00.301681+0000","last_change":"2024-03-11T21:10:57.507626+0000","last_active":"2024-03-11T21:11:00.301681+0000","last_peered":"2024-03-11T21:11:00.301681+0000","last_clean":"2024-03-11T21:11:00.301681+0000","last_became_active":"2024-03-11T21:07:14.953126+0000","last_became_peered":"2024-03-11T21:07:14.953126+0000","last_unstale":"2024-03-11T21:11:00.301681+0000","last_undegraded":"2024-03-11T21:11:00.301681+0000","last_fullsized":"2024-03-11T21:11:00.301681+0000","mapping_epoch":306,"log_start":"337'7400","ondisk_log_start":"337'7400","created":306,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"361'13011","last_scrub_stamp":"2024-03-11T21:10:57.507336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:07:13.846944+0000","last_clean_scrub_stamp":"2024-03-11T21:10:57.507336+0000","objects_scrubbed":631,"log_size":5781,"log_dups_size":0,"ondisk_log_size":5781,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:12:02.129646+0000","scrub_duration":779,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29556736,"num_objects":451,"num_object_clones":0,"num_object_copies":1353,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":451,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13181,"num_write_kb":436224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":385,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:42.475185+0000","last_change":"2024-03-11T21:09:34.951119+0000","last_active":"2024-03-11T21:10:42.475185+0000","last_peered":"2024-03-11T21:10:42.475185+0000","last_clean":"2024-03-11T21:10:42.475185+0000","last_became_active":"2024-03-11T21:05:17.441521+0000","last_became_peered":"2024-03-11T21:05:17.441521+0000","last_unstale":"2024-03-11T21:10:42.475185+0000","last_undegraded":"2024-03-11T21:10:42.475185+0000","last_fullsized":"2024-03-11T21:10:42.475185+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:34.951014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:03:43.345257+0000","last_clean_scrub_stamp":"2024-03-11T21:09:34.951014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:45.871364+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":406,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:42.474229+0000","last_change":"2024-03-11T21:09:35.935667+0000","last_active":"2024-03-11T21:10:42.474229+0000","last_peered":"2024-03-11T21:10:42.474229+0000","last_clean":"2024-03-11T21:10:42.474229+0000","last_became_active":"2024-03-11T21:05:17.315844+0000","last_became_peered":"2024-03-11T21:05:17.315844+0000","last_unstale":"2024-03-11T21:10:42.474229+0000","last_undegraded":"2024-03-11T21:10:42.474229+0000","last_fullsized":"2024-03-11T21:10:42.474229+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:35.935559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:22.617999+0000","last_clean_scrub_stamp":"2024-03-11T21:09:35.935559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:57.074238+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"284'2","reported_seq":370,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:42.457229+0000","last_change":"2024-03-11T21:09:54.213683+0000","last_active":"2024-03-11T21:10:42.457229+0000","last_peered":"2024-03-11T21:10:42.457229+0000","last_clean":"2024-03-11T21:10:42.457229+0000","last_became_active":"2024-03-11T21:00:48.468682+0000","last_became_peered":"2024-03-11T21:00:48.468682+0000","last_unstale":"2024-03-11T21:10:42.457229+0000","last_undegraded":"2024-03-11T21:10:42.457229+0000","last_fullsized":"2024-03-11T21:10:42.457229+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"284'2","last_scrub_stamp":"2024-03-11T21:09:54.213473+0000","last_deep_scrub":"284'2","last_deep_scrub_stamp":"2024-03-11T21:06:47.645682+0000","last_clean_scrub_stamp":"2024-03-11T21:09:54.213473+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:13.303666+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"357'21","reported_seq":678,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:42.476364+0000","last_change":"2024-03-11T21:09:54.016666+0000","last_active":"2024-03-11T21:10:42.476364+0000","last_peered":"2024-03-11T21:10:42.476364+0000","last_clean":"2024-03-11T21:10:42.476364+0000","last_became_active":"2024-03-11T20:50:30.097669+0000","last_became_peered":"2024-03-11T20:50:30.097669+0000","last_unstale":"2024-03-11T21:10:42.476364+0000","last_undegraded":"2024-03-11T21:10:42.476364+0000","last_fullsized":"2024-03-11T21:10:42.476364+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'17","last_scrub_stamp":"2024-03-11T21:09:54.016577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:09:54.016577+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:21.301423+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":31,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"84'11","reported_seq":901,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:54.110045+0000","last_change":"2024-03-11T21:08:04.428744+0000","last_active":"2024-03-11T21:10:54.110045+0000","last_peered":"2024-03-11T21:10:54.110045+0000","last_clean":"2024-03-11T21:10:54.110045+0000","last_became_active":"2024-03-11T20:50:32.647041+0000","last_became_peered":"2024-03-11T20:50:32.647041+0000","last_unstale":"2024-03-11T21:10:54.110045+0000","last_undegraded":"2024-03-11T21:10:54.110045+0000","last_fullsized":"2024-03-11T21:10:54.110045+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:08:04.428606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:08:04.428606+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":83,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.f","version":"361'13424","reported_seq":13648,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:11:00.241297+0000","last_change":"2024-03-11T21:09:58.987446+0000","last_active":"2024-03-11T21:11:00.241297+0000","last_peered":"2024-03-11T21:11:00.241297+0000","last_clean":"2024-03-11T21:11:00.241297+0000","last_became_active":"2024-03-11T21:07:14.959566+0000","last_became_peered":"2024-03-11T21:07:14.959566+0000","last_unstale":"2024-03-11T21:11:00.241297+0000","last_undegraded":"2024-03-11T21:11:00.241297+0000","last_fullsized":"2024-03-11T21:11:00.241297+0000","mapping_epoch":306,"log_start":"337'7600","ondisk_log_start":"337'7600","created":306,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"349'9796","last_scrub_stamp":"2024-03-11T21:09:58.987104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:07:13.846944+0000","last_clean_scrub_stamp":"2024-03-11T21:09:58.987104+0000","objects_scrubbed":4098,"log_size":5824,"log_dups_size":0,"ondisk_log_size":5824,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:00.717053+0000","scrub_duration":1479,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28180480,"num_objects":430,"num_object_clones":0,"num_object_copies":1290,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":430,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13424,"num_write_kb":443328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":661,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:42.474764+0000","last_change":"2024-03-11T21:10:02.768209+0000","last_active":"2024-03-11T21:10:42.474764+0000","last_peered":"2024-03-11T21:10:42.474764+0000","last_clean":"2024-03-11T21:10:42.474764+0000","last_became_active":"2024-03-11T20:50:34.326947+0000","last_became_peered":"2024-03-11T20:50:34.326947+0000","last_unstale":"2024-03-11T21:10:42.474764+0000","last_undegraded":"2024-03-11T21:10:42.474764+0000","last_fullsized":"2024-03-11T21:10:42.474764+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:10:02.768094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:06:43.271453+0000","last_clean_scrub_stamp":"2024-03-11T21:10:02.768094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:08.043060+0000","scrub_duration":219,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"120'6","reported_seq":1778,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:58.010375+0000","last_change":"2024-03-11T21:08:19.340719+0000","last_active":"2024-03-11T21:10:58.010375+0000","last_peered":"2024-03-11T21:10:58.010375+0000","last_clean":"2024-03-11T21:10:58.010375+0000","last_became_active":"2024-03-11T21:00:48.468997+0000","last_became_peered":"2024-03-11T21:00:48.468997+0000","last_unstale":"2024-03-11T21:10:58.010375+0000","last_undegraded":"2024-03-11T21:10:58.010375+0000","last_fullsized":"2024-03-11T21:10:58.010375+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"120'6","last_scrub_stamp":"2024-03-11T21:08:19.340643+0000","last_deep_scrub":"120'6","last_deep_scrub_stamp":"2024-03-11T20:54:26.468057+0000","last_clean_scrub_stamp":"2024-03-11T21:08:19.340643+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0035533370000000002,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1093,"num_read_kb":1088,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":681,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:47.579938+0000","last_change":"2024-03-11T21:10:47.579963+0000","last_active":"2024-03-11T21:10:47.579938+0000","last_peered":"2024-03-11T21:10:47.579938+0000","last_clean":"2024-03-11T21:10:47.579938+0000","last_became_active":"2024-03-11T20:50:28.071707+0000","last_became_peered":"2024-03-11T20:50:28.071707+0000","last_unstale":"2024-03-11T21:10:47.579938+0000","last_undegraded":"2024-03-11T21:10:47.579938+0000","last_fullsized":"2024-03-11T21:10:47.579938+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:10:47.579844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:23.417283+0000","last_clean_scrub_stamp":"2024-03-11T21:10:47.579844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:12:06.595102+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"358'36","reported_seq":775,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:42.472785+0000","last_change":"2024-03-11T21:07:52.838119+0000","last_active":"2024-03-11T21:10:42.472785+0000","last_peered":"2024-03-11T21:10:42.472785+0000","last_clean":"2024-03-11T21:10:42.472785+0000","last_became_active":"2024-03-11T21:00:48.461972+0000","last_became_peered":"2024-03-11T21:00:48.461972+0000","last_unstale":"2024-03-11T21:10:42.472785+0000","last_undegraded":"2024-03-11T21:10:42.472785+0000","last_fullsized":"2024-03-11T21:10:42.472785+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'24","last_scrub_stamp":"2024-03-11T21:07:52.837968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:07:52.837968+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:43.664511+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":435,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:42.469753+0000","last_change":"2024-03-11T21:09:51.796505+0000","last_active":"2024-03-11T21:10:42.469753+0000","last_peered":"2024-03-11T21:10:42.469753+0000","last_clean":"2024-03-11T21:10:42.469753+0000","last_became_active":"2024-03-11T21:05:17.315807+0000","last_became_peered":"2024-03-11T21:05:17.315807+0000","last_unstale":"2024-03-11T21:10:42.469753+0000","last_undegraded":"2024-03-11T21:10:42.469753+0000","last_fullsized":"2024-03-11T21:10:42.469753+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:51.796383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:09:51.796383+0000","last_clean_scrub_stamp":"2024-03-11T21:09:51.796383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:18.488750+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.0","version":"361'13212","reported_seq":13423,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:11:00.291061+0000","last_change":"2024-03-11T21:10:45.963070+0000","last_active":"2024-03-11T21:11:00.291061+0000","last_peered":"2024-03-11T21:11:00.291061+0000","last_clean":"2024-03-11T21:11:00.291061+0000","last_became_active":"2024-03-11T21:07:14.952617+0000","last_became_peered":"2024-03-11T21:07:14.952617+0000","last_unstale":"2024-03-11T21:11:00.291061+0000","last_undegraded":"2024-03-11T21:11:00.291061+0000","last_fullsized":"2024-03-11T21:11:00.291061+0000","mapping_epoch":306,"log_start":"337'7400","ondisk_log_start":"337'7400","created":306,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"361'12436","last_scrub_stamp":"2024-03-11T21:10:45.962978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:07:13.846944+0000","last_clean_scrub_stamp":"2024-03-11T21:10:45.962978+0000","objects_scrubbed":1231,"log_size":5812,"log_dups_size":0,"ondisk_log_size":5812,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:48.968651+0000","scrub_duration":447,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28966912,"num_objects":442,"num_object_clones":0,"num_object_copies":1326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":442,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13212,"num_write_kb":436928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":631,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:42.475784+0000","last_change":"2024-03-11T21:09:40.876804+0000","last_active":"2024-03-11T21:10:42.475784+0000","last_peered":"2024-03-11T21:10:42.475784+0000","last_clean":"2024-03-11T21:10:42.475784+0000","last_became_active":"2024-03-11T20:50:34.335370+0000","last_became_peered":"2024-03-11T20:50:34.335370+0000","last_unstale":"2024-03-11T21:10:42.475784+0000","last_undegraded":"2024-03-11T21:10:42.475784+0000","last_fullsized":"2024-03-11T21:10:42.475784+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:40.876681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:09:40.876681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:04.620155+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"286'18","reported_seq":321,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:54.108351+0000","last_change":"2024-03-11T21:09:45.491799+0000","last_active":"2024-03-11T21:10:54.108351+0000","last_peered":"2024-03-11T21:10:54.108351+0000","last_clean":"2024-03-11T21:10:54.108351+0000","last_became_active":"2024-03-11T21:05:17.449213+0000","last_became_peered":"2024-03-11T21:05:17.449213+0000","last_unstale":"2024-03-11T21:10:54.108351+0000","last_undegraded":"2024-03-11T21:10:54.108351+0000","last_fullsized":"2024-03-11T21:10:54.108351+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"286'18","last_scrub_stamp":"2024-03-11T21:09:45.491546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:09:45.491546+0000","objects_scrubbed":1,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:07.992752+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":678,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:52.823328+0000","last_change":"2024-03-11T21:10:52.823384+0000","last_active":"2024-03-11T21:10:52.823328+0000","last_peered":"2024-03-11T21:10:52.823328+0000","last_clean":"2024-03-11T21:10:52.823328+0000","last_became_active":"2024-03-11T21:00:48.466102+0000","last_became_peered":"2024-03-11T21:00:48.466102+0000","last_unstale":"2024-03-11T21:10:52.823328+0000","last_undegraded":"2024-03-11T21:10:52.823328+0000","last_fullsized":"2024-03-11T21:10:52.823328+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:10:52.823180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:06:42.242801+0000","last_clean_scrub_stamp":"2024-03-11T21:10:52.823180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:12:10.873197+0000","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"357'24","reported_seq":378,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:57.864972+0000","last_change":"2024-03-11T21:10:57.865042+0000","last_active":"2024-03-11T21:10:57.864972+0000","last_peered":"2024-03-11T21:10:57.864972+0000","last_clean":"2024-03-11T21:10:57.864972+0000","last_became_active":"2024-03-11T21:08:05.298170+0000","last_became_peered":"2024-03-11T21:08:05.298170+0000","last_unstale":"2024-03-11T21:10:57.864972+0000","last_undegraded":"2024-03-11T21:10:57.864972+0000","last_fullsized":"2024-03-11T21:10:57.864972+0000","mapping_epoch":319,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":0,"last_scrub":"357'24","last_scrub_stamp":"2024-03-11T21:10:57.864805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:10:57.864805+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:12:07.587067+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":41,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1","version":"361'13153","reported_seq":13336,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:59.928759+0000","last_change":"2024-03-11T21:10:55.964913+0000","last_active":"2024-03-11T21:10:59.928759+0000","last_peered":"2024-03-11T21:10:59.928759+0000","last_clean":"2024-03-11T21:10:59.928759+0000","last_became_active":"2024-03-11T21:07:14.951704+0000","last_became_peered":"2024-03-11T21:07:14.951704+0000","last_unstale":"2024-03-11T21:10:59.928759+0000","last_undegraded":"2024-03-11T21:10:59.928759+0000","last_fullsized":"2024-03-11T21:10:59.928759+0000","mapping_epoch":306,"log_start":"343'8200","ondisk_log_start":"343'8200","created":306,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"361'12950","last_scrub_stamp":"2024-03-11T21:10:55.964498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:07:13.846944+0000","last_clean_scrub_stamp":"2024-03-11T21:10:55.964498+0000","objects_scrubbed":698,"log_size":4953,"log_dups_size":0,"ondisk_log_size":4953,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:12:07.508743+0000","scrub_duration":151,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32047104,"num_objects":489,"num_object_clones":0,"num_object_copies":1467,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":489,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13153,"num_write_kb":436544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":668,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:50.652544+0000","last_change":"2024-03-11T21:10:50.652568+0000","last_active":"2024-03-11T21:10:50.652544+0000","last_peered":"2024-03-11T21:10:50.652544+0000","last_clean":"2024-03-11T21:10:50.652544+0000","last_became_active":"2024-03-11T21:00:48.460531+0000","last_became_peered":"2024-03-11T21:00:48.460531+0000","last_unstale":"2024-03-11T21:10:50.652544+0000","last_undegraded":"2024-03-11T21:10:50.652544+0000","last_fullsized":"2024-03-11T21:10:50.652544+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:10:50.652484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:27.775534+0000","last_clean_scrub_stamp":"2024-03-11T21:10:50.652484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:12:12.434661+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":373,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:42.457047+0000","last_change":"2024-03-11T21:09:53.196862+0000","last_active":"2024-03-11T21:10:42.457047+0000","last_peered":"2024-03-11T21:10:42.457047+0000","last_clean":"2024-03-11T21:10:42.457047+0000","last_became_active":"2024-03-11T21:00:48.469596+0000","last_became_peered":"2024-03-11T21:00:48.469596+0000","last_unstale":"2024-03-11T21:10:42.457047+0000","last_undegraded":"2024-03-11T21:10:42.457047+0000","last_fullsized":"2024-03-11T21:10:42.457047+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:53.196741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:09:53.196741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:17.301704+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":365,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:53.563952+0000","last_change":"2024-03-11T21:10:53.563996+0000","last_active":"2024-03-11T21:10:53.563952+0000","last_peered":"2024-03-11T21:10:53.563952+0000","last_clean":"2024-03-11T21:10:53.563952+0000","last_became_active":"2024-03-11T21:00:48.466084+0000","last_became_peered":"2024-03-11T21:00:48.466084+0000","last_unstale":"2024-03-11T21:10:53.563952+0000","last_undegraded":"2024-03-11T21:10:53.563952+0000","last_fullsized":"2024-03-11T21:10:53.563952+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:10:53.563854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:40.974147+0000","last_clean_scrub_stamp":"2024-03-11T21:10:53.563854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:12:20.012133+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"358'46","reported_seq":767,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:42.474658+0000","last_change":"2024-03-11T21:09:19.777457+0000","last_active":"2024-03-11T21:10:42.474658+0000","last_peered":"2024-03-11T21:10:42.474658+0000","last_clean":"2024-03-11T21:10:42.474658+0000","last_became_active":"2024-03-11T21:00:48.472191+0000","last_became_peered":"2024-03-11T21:00:48.472191+0000","last_unstale":"2024-03-11T21:10:42.474658+0000","last_undegraded":"2024-03-11T21:10:42.474658+0000","last_fullsized":"2024-03-11T21:10:42.474658+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'34","last_scrub_stamp":"2024-03-11T21:09:19.777317+0000","last_deep_scrub":"116'22","last_deep_scrub_stamp":"2024-03-11T20:53:47.364709+0000","last_clean_scrub_stamp":"2024-03-11T21:09:19.777317+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:45.871337+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"361'13047","reported_seq":13235,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:57.287530+0000","last_change":"2024-03-11T21:09:25.047442+0000","last_active":"2024-03-11T21:10:57.287530+0000","last_peered":"2024-03-11T21:10:57.287530+0000","last_clean":"2024-03-11T21:10:57.287530+0000","last_became_active":"2024-03-11T21:07:14.951968+0000","last_became_peered":"2024-03-11T21:07:14.951968+0000","last_unstale":"2024-03-11T21:10:57.287530+0000","last_undegraded":"2024-03-11T21:10:57.287530+0000","last_fullsized":"2024-03-11T21:10:57.287530+0000","mapping_epoch":306,"log_start":"336'7300","ondisk_log_start":"336'7300","created":306,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"339'7779","last_scrub_stamp":"2024-03-11T21:09:25.047208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:07:13.846944+0000","last_clean_scrub_stamp":"2024-03-11T21:09:25.047208+0000","objects_scrubbed":5932,"log_size":5747,"log_dups_size":0,"ondisk_log_size":5747,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1719,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39911424,"num_objects":609,"num_object_clones":0,"num_object_copies":1827,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":609,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13047,"num_write_kb":436992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":444,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:42.457000+0000","last_change":"2024-03-11T21:09:50.191402+0000","last_active":"2024-03-11T21:10:42.457000+0000","last_peered":"2024-03-11T21:10:42.457000+0000","last_clean":"2024-03-11T21:10:42.457000+0000","last_became_active":"2024-03-11T21:05:17.455236+0000","last_became_peered":"2024-03-11T21:05:17.455236+0000","last_unstale":"2024-03-11T21:10:42.457000+0000","last_undegraded":"2024-03-11T21:10:42.457000+0000","last_fullsized":"2024-03-11T21:10:42.457000+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:50.191075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:09:50.191075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:56.493600+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":365,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:42.456991+0000","last_change":"2024-03-11T21:09:49.147884+0000","last_active":"2024-03-11T21:10:42.456991+0000","last_peered":"2024-03-11T21:10:42.456991+0000","last_clean":"2024-03-11T21:10:42.456991+0000","last_became_active":"2024-03-11T21:00:48.461367+0000","last_became_peered":"2024-03-11T21:00:48.461367+0000","last_unstale":"2024-03-11T21:10:42.456991+0000","last_undegraded":"2024-03-11T21:10:42.456991+0000","last_fullsized":"2024-03-11T21:10:42.456991+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:49.147414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:03:37.601229+0000","last_clean_scrub_stamp":"2024-03-11T21:09:49.147414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:04.968002+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"358'25","reported_seq":686,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:42.476199+0000","last_change":"2024-03-11T21:08:28.368669+0000","last_active":"2024-03-11T21:10:42.476199+0000","last_peered":"2024-03-11T21:10:42.476199+0000","last_clean":"2024-03-11T21:10:42.476199+0000","last_became_active":"2024-03-11T21:00:27.967572+0000","last_became_peered":"2024-03-11T21:00:27.967572+0000","last_unstale":"2024-03-11T21:10:42.476199+0000","last_undegraded":"2024-03-11T21:10:42.476199+0000","last_fullsized":"2024-03-11T21:10:42.476199+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"267'19","last_scrub_stamp":"2024-03-11T21:08:28.368557+0000","last_deep_scrub":"267'19","last_deep_scrub_stamp":"2024-03-11T21:03:39.673317+0000","last_clean_scrub_stamp":"2024-03-11T21:08:28.368557+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:45.871203+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":429,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:42.475966+0000","last_change":"2024-03-11T21:08:03.331915+0000","last_active":"2024-03-11T21:10:42.475966+0000","last_peered":"2024-03-11T21:10:42.475966+0000","last_clean":"2024-03-11T21:10:42.475966+0000","last_became_active":"2024-03-11T21:05:17.316136+0000","last_became_peered":"2024-03-11T21:05:17.316136+0000","last_unstale":"2024-03-11T21:10:42.475966+0000","last_undegraded":"2024-03-11T21:10:42.475966+0000","last_fullsized":"2024-03-11T21:10:42.475966+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:08:03.331832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:35.670149+0000","last_clean_scrub_stamp":"2024-03-11T21:08:03.331832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:45.870993+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"361'12818","reported_seq":13014,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:55.579867+0000","last_change":"2024-03-11T21:09:21.036734+0000","last_active":"2024-03-11T21:10:55.579867+0000","last_peered":"2024-03-11T21:10:55.579867+0000","last_clean":"2024-03-11T21:10:55.579867+0000","last_became_active":"2024-03-11T21:08:05.298758+0000","last_became_peered":"2024-03-11T21:08:05.298758+0000","last_unstale":"2024-03-11T21:10:55.579867+0000","last_undegraded":"2024-03-11T21:10:55.579867+0000","last_fullsized":"2024-03-11T21:10:55.579867+0000","mapping_epoch":319,"log_start":"334'6800","ondisk_log_start":"334'6800","created":306,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'7464","last_scrub_stamp":"2024-03-11T21:09:21.036463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:07:13.846944+0000","last_clean_scrub_stamp":"2024-03-11T21:09:21.036463+0000","objects_scrubbed":6066,"log_size":6018,"log_dups_size":0,"ondisk_log_size":6018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2195,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42729472,"num_objects":652,"num_object_clones":0,"num_object_copies":1956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":652,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12818,"num_write_kb":431040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":408,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:42.476194+0000","last_change":"2024-03-11T21:10:03.144979+0000","last_active":"2024-03-11T21:10:42.476194+0000","last_peered":"2024-03-11T21:10:42.476194+0000","last_clean":"2024-03-11T21:10:42.476194+0000","last_became_active":"2024-03-11T21:08:05.304963+0000","last_became_peered":"2024-03-11T21:08:05.304963+0000","last_unstale":"2024-03-11T21:10:42.476194+0000","last_undegraded":"2024-03-11T21:10:42.476194+0000","last_fullsized":"2024-03-11T21:10:42.476194+0000","mapping_epoch":319,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:10:03.144875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:10:03.144875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:04.198120+0000","scrub_duration":239,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"320'21","reported_seq":637,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:49.104289+0000","last_change":"2024-03-11T21:09:46.828876+0000","last_active":"2024-03-11T21:10:49.104289+0000","last_peered":"2024-03-11T21:10:49.104289+0000","last_clean":"2024-03-11T21:10:49.104289+0000","last_became_active":"2024-03-11T21:08:05.298887+0000","last_became_peered":"2024-03-11T21:08:05.298887+0000","last_unstale":"2024-03-11T21:10:49.104289+0000","last_undegraded":"2024-03-11T21:10:49.104289+0000","last_fullsized":"2024-03-11T21:10:49.104289+0000","mapping_epoch":319,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'21","last_scrub_stamp":"2024-03-11T21:09:46.828795+0000","last_deep_scrub":"146'14","last_deep_scrub_stamp":"2024-03-11T20:56:00.242211+0000","last_clean_scrub_stamp":"2024-03-11T21:09:46.828795+0000","objects_scrubbed":1,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:03.069048+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":666,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:42.474873+0000","last_change":"2024-03-11T21:09:43.387853+0000","last_active":"2024-03-11T21:10:42.474873+0000","last_peered":"2024-03-11T21:10:42.474873+0000","last_clean":"2024-03-11T21:10:42.474873+0000","last_became_active":"2024-03-11T21:00:27.970573+0000","last_became_peered":"2024-03-11T21:00:27.970573+0000","last_unstale":"2024-03-11T21:10:42.474873+0000","last_undegraded":"2024-03-11T21:10:42.474873+0000","last_fullsized":"2024-03-11T21:10:42.474873+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:43.387711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:52:51.161281+0000","last_clean_scrub_stamp":"2024-03-11T21:09:43.387711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:52.740057+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"358'60","reported_seq":834,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:42.464990+0000","last_change":"2024-03-11T21:09:48.540145+0000","last_active":"2024-03-11T21:10:42.464990+0000","last_peered":"2024-03-11T21:10:42.464990+0000","last_clean":"2024-03-11T21:10:42.464990+0000","last_became_active":"2024-03-11T21:00:48.469854+0000","last_became_peered":"2024-03-11T21:00:48.469854+0000","last_unstale":"2024-03-11T21:10:42.464990+0000","last_undegraded":"2024-03-11T21:10:42.464990+0000","last_fullsized":"2024-03-11T21:10:42.464990+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'44","last_scrub_stamp":"2024-03-11T21:09:48.539752+0000","last_deep_scrub":"161'28","last_deep_scrub_stamp":"2024-03-11T20:57:20.816278+0000","last_clean_scrub_stamp":"2024-03-11T21:09:48.539752+0000","objects_scrubbed":9,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:08.766428+0000","scrub_duration":59,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":80,"num_read_kb":69,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.4","version":"361'12833","reported_seq":13016,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:55.435121+0000","last_change":"2024-03-11T21:09:34.130627+0000","last_active":"2024-03-11T21:10:55.435121+0000","last_peered":"2024-03-11T21:10:55.435121+0000","last_clean":"2024-03-11T21:10:55.435121+0000","last_became_active":"2024-03-11T21:07:14.959964+0000","last_became_peered":"2024-03-11T21:07:14.959964+0000","last_unstale":"2024-03-11T21:10:55.435121+0000","last_undegraded":"2024-03-11T21:10:55.435121+0000","last_fullsized":"2024-03-11T21:10:55.435121+0000","mapping_epoch":306,"log_start":"337'7400","ondisk_log_start":"337'7400","created":306,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"343'8155","last_scrub_stamp":"2024-03-11T21:09:34.130396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:07:13.846944+0000","last_clean_scrub_stamp":"2024-03-11T21:09:34.130396+0000","objects_scrubbed":5417,"log_size":5433,"log_dups_size":0,"ondisk_log_size":5433,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2035,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46202880,"num_objects":705,"num_object_clones":0,"num_object_copies":2115,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":705,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12833,"num_write_kb":433216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":653,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:42.474154+0000","last_change":"2024-03-11T21:09:53.436563+0000","last_active":"2024-03-11T21:10:42.474154+0000","last_peered":"2024-03-11T21:10:42.474154+0000","last_clean":"2024-03-11T21:10:42.474154+0000","last_became_active":"2024-03-11T20:50:34.327139+0000","last_became_peered":"2024-03-11T20:50:34.327139+0000","last_unstale":"2024-03-11T21:10:42.474154+0000","last_undegraded":"2024-03-11T21:10:42.474154+0000","last_fullsized":"2024-03-11T21:10:42.474154+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:53.436482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:09:53.436482+0000","last_clean_scrub_stamp":"2024-03-11T21:09:53.436482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:16.930710+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":104,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:42.469780+0000","last_change":"2024-03-11T21:09:33.763673+0000","last_active":"2024-03-11T21:10:42.469780+0000","last_peered":"2024-03-11T21:10:42.469780+0000","last_clean":"2024-03-11T21:10:42.469780+0000","last_became_active":"2024-03-11T21:08:05.303226+0000","last_became_peered":"2024-03-11T21:08:05.303226+0000","last_unstale":"2024-03-11T21:10:42.469780+0000","last_undegraded":"2024-03-11T21:10:42.469780+0000","last_fullsized":"2024-03-11T21:10:42.469780+0000","mapping_epoch":319,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:33.763593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:57:17.447191+0000","last_clean_scrub_stamp":"2024-03-11T21:09:33.763593+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:45.542533+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"358'53","reported_seq":739,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:42.476738+0000","last_change":"2024-03-11T21:09:40.486417+0000","last_active":"2024-03-11T21:10:42.476738+0000","last_peered":"2024-03-11T21:10:42.476738+0000","last_clean":"2024-03-11T21:10:42.476738+0000","last_became_active":"2024-03-11T20:50:30.111819+0000","last_became_peered":"2024-03-11T20:50:30.111819+0000","last_unstale":"2024-03-11T21:10:42.476738+0000","last_undegraded":"2024-03-11T21:10:42.476738+0000","last_fullsized":"2024-03-11T21:10:42.476738+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'39","last_scrub_stamp":"2024-03-11T21:09:40.486313+0000","last_deep_scrub":"102'25","last_deep_scrub_stamp":"2024-03-11T20:51:35.187094+0000","last_clean_scrub_stamp":"2024-03-11T21:09:40.486313+0000","objects_scrubbed":8,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:56.721971+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":61,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":678,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:42.464728+0000","last_change":"2024-03-11T21:09:42.597032+0000","last_active":"2024-03-11T21:10:42.464728+0000","last_peered":"2024-03-11T21:10:42.464728+0000","last_clean":"2024-03-11T21:10:42.464728+0000","last_became_active":"2024-03-11T21:00:48.362389+0000","last_became_peered":"2024-03-11T21:00:48.362389+0000","last_unstale":"2024-03-11T21:10:42.464728+0000","last_undegraded":"2024-03-11T21:10:42.464728+0000","last_fullsized":"2024-03-11T21:10:42.464728+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:42.596113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:09:42.596113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:53.893451+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.5","version":"361'13065","reported_seq":13257,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:55.416679+0000","last_change":"2024-03-11T21:09:23.813448+0000","last_active":"2024-03-11T21:10:55.416679+0000","last_peered":"2024-03-11T21:10:55.416679+0000","last_clean":"2024-03-11T21:10:55.416679+0000","last_became_active":"2024-03-11T21:08:05.304912+0000","last_became_peered":"2024-03-11T21:08:05.304912+0000","last_unstale":"2024-03-11T21:10:55.416679+0000","last_undegraded":"2024-03-11T21:10:55.416679+0000","last_fullsized":"2024-03-11T21:10:55.416679+0000","mapping_epoch":319,"log_start":"338'7700","ondisk_log_start":"338'7700","created":306,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'7788","last_scrub_stamp":"2024-03-11T21:09:23.813149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:07:13.846944+0000","last_clean_scrub_stamp":"2024-03-11T21:09:23.813149+0000","objects_scrubbed":6007,"log_size":5365,"log_dups_size":0,"ondisk_log_size":5365,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1779,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45154304,"num_objects":689,"num_object_clones":0,"num_object_copies":2067,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":689,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13065,"num_write_kb":440128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":349,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:42.471975+0000","last_change":"2024-03-11T21:09:39.408174+0000","last_active":"2024-03-11T21:10:42.471975+0000","last_peered":"2024-03-11T21:10:42.471975+0000","last_clean":"2024-03-11T21:10:42.471975+0000","last_became_active":"2024-03-11T21:06:28.758451+0000","last_became_peered":"2024-03-11T21:06:28.758451+0000","last_unstale":"2024-03-11T21:10:42.471975+0000","last_undegraded":"2024-03-11T21:10:42.471975+0000","last_fullsized":"2024-03-11T21:10:42.471975+0000","mapping_epoch":297,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:39.408071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:55:58.888890+0000","last_clean_scrub_stamp":"2024-03-11T21:09:39.408071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:47.153652+0000","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":357,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:42.457156+0000","last_change":"2024-03-11T21:09:24.339301+0000","last_active":"2024-03-11T21:10:42.457156+0000","last_peered":"2024-03-11T21:10:42.457156+0000","last_clean":"2024-03-11T21:10:42.457156+0000","last_became_active":"2024-03-11T21:00:48.469015+0000","last_became_peered":"2024-03-11T21:00:48.469015+0000","last_unstale":"2024-03-11T21:10:42.457156+0000","last_undegraded":"2024-03-11T21:10:42.457156+0000","last_fullsized":"2024-03-11T21:10:42.457156+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:24.339169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:09:24.339169+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:49.809729+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"358'30","reported_seq":700,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:42.469728+0000","last_change":"2024-03-11T21:09:36.752271+0000","last_active":"2024-03-11T21:10:42.469728+0000","last_peered":"2024-03-11T21:10:42.469728+0000","last_clean":"2024-03-11T21:10:42.469728+0000","last_became_active":"2024-03-11T20:50:30.109549+0000","last_became_peered":"2024-03-11T20:50:30.109549+0000","last_unstale":"2024-03-11T21:10:42.469728+0000","last_undegraded":"2024-03-11T21:10:42.469728+0000","last_fullsized":"2024-03-11T21:10:42.469728+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'20","last_scrub_stamp":"2024-03-11T21:09:36.752172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:09:36.752172+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:46.990100+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":696,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:42.470090+0000","last_change":"2024-03-11T21:09:49.344698+0000","last_active":"2024-03-11T21:10:42.470090+0000","last_peered":"2024-03-11T21:10:42.470090+0000","last_clean":"2024-03-11T21:10:42.470090+0000","last_became_active":"2024-03-11T21:02:37.517748+0000","last_became_peered":"2024-03-11T21:02:37.517748+0000","last_unstale":"2024-03-11T21:10:42.470090+0000","last_undegraded":"2024-03-11T21:10:42.470090+0000","last_fullsized":"2024-03-11T21:10:42.470090+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:49.344594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:52:51.931295+0000","last_clean_scrub_stamp":"2024-03-11T21:09:49.344594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:17.641189+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.6","version":"361'12939","reported_seq":13129,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:55.593871+0000","last_change":"2024-03-11T21:09:23.070615+0000","last_active":"2024-03-11T21:10:55.593871+0000","last_peered":"2024-03-11T21:10:55.593871+0000","last_clean":"2024-03-11T21:10:55.593871+0000","last_became_active":"2024-03-11T21:07:14.952904+0000","last_became_peered":"2024-03-11T21:07:14.952904+0000","last_unstale":"2024-03-11T21:10:55.593871+0000","last_undegraded":"2024-03-11T21:10:55.593871+0000","last_fullsized":"2024-03-11T21:10:55.593871+0000","mapping_epoch":306,"log_start":"334'6900","ondisk_log_start":"334'6900","created":306,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'7752","last_scrub_stamp":"2024-03-11T21:09:23.070441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:07:13.846944+0000","last_clean_scrub_stamp":"2024-03-11T21:09:23.070441+0000","objects_scrubbed":6006,"log_size":6039,"log_dups_size":0,"ondisk_log_size":6039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2283,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49217536,"num_objects":751,"num_object_clones":0,"num_object_copies":2253,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":751,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12939,"num_write_kb":438080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":127,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:50.090633+0000","last_change":"2024-03-11T21:10:50.090657+0000","last_active":"2024-03-11T21:10:50.090633+0000","last_peered":"2024-03-11T21:10:50.090633+0000","last_clean":"2024-03-11T21:10:50.090633+0000","last_became_active":"2024-03-11T21:07:34.504579+0000","last_became_peered":"2024-03-11T21:07:34.504579+0000","last_unstale":"2024-03-11T21:10:50.090633+0000","last_undegraded":"2024-03-11T21:10:50.090633+0000","last_fullsized":"2024-03-11T21:10:50.090633+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:10:50.090575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:00:14.030599+0000","last_clean_scrub_stamp":"2024-03-11T21:10:50.090575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:12:10.607701+0000","scrub_duration":387,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":96,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:42.477671+0000","last_change":"2024-03-11T21:08:05.308620+0000","last_active":"2024-03-11T21:10:42.477671+0000","last_peered":"2024-03-11T21:10:42.477671+0000","last_clean":"2024-03-11T21:10:42.477671+0000","last_became_active":"2024-03-11T21:08:05.307984+0000","last_became_peered":"2024-03-11T21:08:05.307984+0000","last_unstale":"2024-03-11T21:10:42.477671+0000","last_undegraded":"2024-03-11T21:10:42.477671+0000","last_fullsized":"2024-03-11T21:10:42.477671+0000","mapping_epoch":319,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:07:57.816881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:07:57.816881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:45.870853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.f","version":"358'39","reported_seq":723,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:48.857063+0000","last_change":"2024-03-11T21:10:48.857063+0000","last_active":"2024-03-11T21:10:48.857063+0000","last_peered":"2024-03-11T21:10:48.857063+0000","last_clean":"2024-03-11T21:10:48.857063+0000","last_became_active":"2024-03-11T21:00:27.963119+0000","last_became_peered":"2024-03-11T21:00:27.963119+0000","last_unstale":"2024-03-11T21:10:48.857063+0000","last_undegraded":"2024-03-11T21:10:48.857063+0000","last_fullsized":"2024-03-11T21:10:48.857063+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'39","last_scrub_stamp":"2024-03-11T21:10:48.856947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:10:48.856947+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:12:12.155046+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":651,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-11T21:10:42.469166+0000","last_change":"2024-03-11T21:09:48.842757+0000","last_active":"2024-03-11T21:10:42.469166+0000","last_peered":"2024-03-11T21:10:42.469166+0000","last_clean":"2024-03-11T21:10:42.469166+0000","last_became_active":"2024-03-11T20:50:32.652854+0000","last_became_peered":"2024-03-11T20:50:32.652854+0000","last_unstale":"2024-03-11T21:10:42.469166+0000","last_undegraded":"2024-03-11T21:10:42.469166+0000","last_fullsized":"2024-03-11T21:10:42.469166+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:48.842645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:59:00.946094+0000","last_clean_scrub_stamp":"2024-03-11T21:09:48.842645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:10:54.267900+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":584056856,"num_objects":8913,"num_object_clones":0,"num_object_copies":26739,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8913,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":209107,"num_write_kb":6976577,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1835610112,"data_stored":1835597896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90607,"ondisk_log_size":90607,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":20,"num_read_kb":20,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1822,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1550,"num_read_kb":1307,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":171,"num_bytes_recovered":2228,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3632,"internal_metadata":0},"log_size":1064,"ondisk_log_size":1064,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":126,"ondisk_log_size":126,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1093,"num_read_kb":1088,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":68,"seq":292057776385,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":616692,"kb_used_data":125996,"kb_used_omap":1,"kb_used_meta":490686,"kb_avail":93128460,"statfs":{"total":95995035648,"available":95363543040,"internally_reserved":0,"allocated":129019904,"data_stored":128705299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":502462922},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":57,"seq":244813136139,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":602600,"kb_used_data":224224,"kb_used_omap":1,"kb_used_meta":378238,"kb_avail":93142552,"statfs":{"total":95995035648,"available":95377973248,"internally_reserved":0,"allocated":229605376,"data_stored":229303093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":387316167},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":211,"seq":906238099579,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":490536,"kb_used_data":125216,"kb_used_omap":7,"kb_used_meta":365304,"kb_avail":93254616,"statfs":{"total":95995035648,"available":95492726784,"internally_reserved":0,"allocated":128221184,"data_stored":127918579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8078,"internal_metadata":374071410},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738696,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":971248,"kb_used_data":480488,"kb_used_omap":1,"kb_used_meta":490686,"kb_avail":92773904,"statfs":{"total":95995035648,"available":95000477696,"internally_reserved":0,"allocated":492019712,"data_stored":491708169,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":502462917},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378932,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":582744,"kb_used_data":194768,"kb_used_omap":1,"kb_used_meta":387966,"kb_avail":93162408,"statfs":{"total":95995035648,"available":95398305792,"internally_reserved":0,"allocated":199442432,"data_stored":199155059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":397277637},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":28,"seq":120259084585,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":723292,"kb_used_data":164564,"kb_used_omap":1,"kb_used_meta":558718,"kb_avail":93021860,"statfs":{"total":95995035648,"available":95254384640,"internally_reserved":0,"allocated":168513536,"data_stored":168222443,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":572127688},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":204,"seq":876173328511,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":639492,"kb_used_data":350268,"kb_used_omap":4,"kb_used_meta":289211,"kb_avail":93105660,"statfs":{"total":95995035648,"available":95340195840,"internally_reserved":0,"allocated":358674432,"data_stored":358343793,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4451,"internal_metadata":296152733},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":44,"seq":188978561300,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":661404,"kb_used_data":224852,"kb_used_omap":1,"kb_used_meta":436542,"kb_avail":93083748,"statfs":{"total":95995035648,"available":95317757952,"internally_reserved":0,"allocated":230248448,"data_stored":229956901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":447019461},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3632,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":464850944,"data_stored":464846872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":122355712,"data_stored":122355712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":184942592,"data_stored":184942592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":155058176,"data_stored":155058176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":346296320,"data_stored":346292248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":227409920,"data_stored":227409920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":209063936,"data_stored":209059864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":125632512,"data_stored":125632512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T21:11:04.718 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-03-11T21:11:04.718 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd pool set .mgr pg_num 11 2024-03-11T21:11:04.779 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:04.800 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:04.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:04.802 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:11:04.903 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:04.915 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:04 smithi023 bash[21372]: cluster 2024-03-11T21:11:02.832774+0000 osd.7 (osd.7) 469 : cluster 0 6.15 deep-scrub starts 2024-03-11T21:11:04.915 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:04 smithi023 bash[21372]: cluster 2024-03-11T21:11:02.899461+0000 osd.7 (osd.7) 470 : cluster 0 6.15 deep-scrub ok 2024-03-11T21:11:04.916 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:04 smithi023 bash[21372]: cluster 2024-03-11T21:11:03.791233+0000 osd.0 (osd.0) 327 : cluster 0 2.f scrub starts 2024-03-11T21:11:04.916 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:04 smithi023 bash[21372]: cluster 2024-03-11T21:11:03.794160+0000 osd.0 (osd.0) 328 : cluster 0 2.f scrub ok 2024-03-11T21:11:04.916 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:04 smithi023 bash[29146]: cluster 2024-03-11T21:11:02.832774+0000 osd.7 (osd.7) 469 : cluster 0 6.15 deep-scrub starts 2024-03-11T21:11:04.916 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:04 smithi023 bash[29146]: cluster 2024-03-11T21:11:02.899461+0000 osd.7 (osd.7) 470 : cluster 0 6.15 deep-scrub ok 2024-03-11T21:11:04.916 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:04 smithi023 bash[29146]: cluster 2024-03-11T21:11:03.791233+0000 osd.0 (osd.0) 327 : cluster 0 2.f scrub starts 2024-03-11T21:11:04.916 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:04 smithi023 bash[29146]: cluster 2024-03-11T21:11:03.794160+0000 osd.0 (osd.0) 328 : cluster 0 2.f scrub ok 2024-03-11T21:11:04.932 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:04.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:04.936 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:11:05.007 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:04 smithi146 bash[23809]: cluster 2024-03-11T21:11:02.832774+0000 osd.7 (osd.7) 469 : cluster 0 6.15 deep-scrub starts 2024-03-11T21:11:05.008 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:04 smithi146 bash[23809]: cluster 2024-03-11T21:11:02.899461+0000 osd.7 (osd.7) 470 : cluster 0 6.15 deep-scrub ok 2024-03-11T21:11:05.008 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:04 smithi146 bash[23809]: cluster 2024-03-11T21:11:03.791233+0000 osd.0 (osd.0) 327 : cluster 0 2.f scrub starts 2024-03-11T21:11:05.008 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:04 smithi146 bash[23809]: cluster 2024-03-11T21:11:03.794160+0000 osd.0 (osd.0) 328 : cluster 0 2.f scrub ok 2024-03-11T21:11:05.036 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:05.073 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:05.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:05.077 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:11:05.178 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:11:05.212 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:11:05.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:05.217 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:11:05.318 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:11:05.349 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:11:05.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:05.351 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:11:05.452 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:05.476 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:05.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:05.479 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:11:05.579 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:05.603 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:05.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:05.605 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:11:05.706 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:11:05.738 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:11:05.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:05.740 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:11:05.757 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:05 smithi146 bash[23809]: cluster 2024-03-11T21:11:03.859613+0000 osd.7 (osd.7) 471 : cluster 0 6.14 scrub starts 2024-03-11T21:11:05.757 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:05 smithi146 bash[23809]: cluster 2024-03-11T21:11:03.865221+0000 osd.7 (osd.7) 472 : cluster 0 6.14 scrub ok 2024-03-11T21:11:05.757 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:05 smithi146 bash[23809]: cluster 2024-03-11T21:11:04.338076+0000 mgr.y (mgr.24380) 751 : cluster 0 pgmap v849: 148 pgs: 148 active+clean; 450 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 854 op/s 2024-03-11T21:11:05.757 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:05 smithi146 bash[23809]: cluster 2024-03-11T21:11:05.186349+0000 osd.3 (osd.3) 418 : cluster 0 2.1b scrub starts 2024-03-11T21:11:05.757 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:05 smithi146 bash[23809]: cluster 2024-03-11T21:11:05.190553+0000 osd.3 (osd.3) 419 : cluster 0 2.1b scrub ok 2024-03-11T21:11:05.841 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:11:05.870 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:11:05.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:05.873 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:11:05.974 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:11:05.983 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:05 smithi023 bash[29146]: cluster 2024-03-11T21:11:03.859613+0000 osd.7 (osd.7) 471 : cluster 0 6.14 scrub starts 2024-03-11T21:11:05.983 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:05 smithi023 bash[29146]: cluster 2024-03-11T21:11:03.865221+0000 osd.7 (osd.7) 472 : cluster 0 6.14 scrub ok 2024-03-11T21:11:05.983 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:05 smithi023 bash[29146]: cluster 2024-03-11T21:11:04.338076+0000 mgr.y (mgr.24380) 751 : cluster 0 pgmap v849: 148 pgs: 148 active+clean; 450 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 854 op/s 2024-03-11T21:11:05.983 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:05 smithi023 bash[29146]: cluster 2024-03-11T21:11:05.186349+0000 osd.3 (osd.3) 418 : cluster 0 2.1b scrub starts 2024-03-11T21:11:05.983 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:05 smithi023 bash[29146]: cluster 2024-03-11T21:11:05.190553+0000 osd.3 (osd.3) 419 : cluster 0 2.1b scrub ok 2024-03-11T21:11:05.984 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:05 smithi023 bash[21372]: cluster 2024-03-11T21:11:03.859613+0000 osd.7 (osd.7) 471 : cluster 0 6.14 scrub starts 2024-03-11T21:11:05.984 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:05 smithi023 bash[21372]: cluster 2024-03-11T21:11:03.865221+0000 osd.7 (osd.7) 472 : cluster 0 6.14 scrub ok 2024-03-11T21:11:05.984 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:05 smithi023 bash[21372]: cluster 2024-03-11T21:11:04.338076+0000 mgr.y (mgr.24380) 751 : cluster 0 pgmap v849: 148 pgs: 148 active+clean; 450 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 854 op/s 2024-03-11T21:11:05.984 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:05 smithi023 bash[21372]: cluster 2024-03-11T21:11:05.186349+0000 osd.3 (osd.3) 418 : cluster 0 2.1b scrub starts 2024-03-11T21:11:05.984 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:05 smithi023 bash[21372]: cluster 2024-03-11T21:11:05.190553+0000 osd.3 (osd.3) 419 : cluster 0 2.1b scrub ok 2024-03-11T21:11:05.995 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:11:05.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:05.997 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:11:06.098 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:11:06.123 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:11:06.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:06.125 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:11:06.226 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:11:06.257 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:11:06.258 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:11:05 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:11:06.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:06.261 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:11:06.361 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:11:06.536 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:11:06.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:06.538 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:11:06.639 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:06.671 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:06.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:06.673 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:11:06.774 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:06.800 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:06.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:06.805 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:11:06.809 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T21:11:06.818 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd set noscrub 2024-03-11T21:11:06.906 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:06.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:06 smithi146 bash[23809]: cluster 2024-03-11T21:11:04.813765+0000 osd.7 (osd.7) 473 : cluster 0 10.c scrub starts 2024-03-11T21:11:06.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:06 smithi146 bash[23809]: cluster 2024-03-11T21:11:04.859254+0000 osd.7 (osd.7) 474 : cluster 0 10.c scrub ok 2024-03-11T21:11:06.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:06 smithi146 bash[23809]: cluster 2024-03-11T21:11:05.879511+0000 osd.0 (osd.0) 329 : cluster 0 2.1f deep-scrub starts 2024-03-11T21:11:06.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:06 smithi146 bash[23809]: cluster 2024-03-11T21:11:05.884893+0000 osd.0 (osd.0) 330 : cluster 0 2.1f deep-scrub ok 2024-03-11T21:11:06.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:06 smithi146 bash[23809]: audit 2024-03-11T21:11:06.064007+0000 mon.a (mon.0) 1746 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:11:06.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:06 smithi146 bash[23809]: audit 2024-03-11T21:11:06.065086+0000 mon.a (mon.0) 1747 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:11:06.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:06 smithi146 bash[23809]: cluster 2024-03-11T21:11:06.375454+0000 osd.5 (osd.5) 427 : cluster 0 10.6 scrub starts 2024-03-11T21:11:06.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:06 smithi146 bash[23809]: cluster 2024-03-11T21:11:06.473031+0000 osd.5 (osd.5) 428 : cluster 0 10.6 scrub ok 2024-03-11T21:11:06.946 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:06.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:06.952 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:11:07.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:06 smithi023 bash[21372]: cluster 2024-03-11T21:11:04.813765+0000 osd.7 (osd.7) 473 : cluster 0 10.c scrub starts 2024-03-11T21:11:07.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:06 smithi023 bash[21372]: cluster 2024-03-11T21:11:04.859254+0000 osd.7 (osd.7) 474 : cluster 0 10.c scrub ok 2024-03-11T21:11:07.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:06 smithi023 bash[21372]: cluster 2024-03-11T21:11:05.879511+0000 osd.0 (osd.0) 329 : cluster 0 2.1f deep-scrub starts 2024-03-11T21:11:07.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:06 smithi023 bash[21372]: cluster 2024-03-11T21:11:05.884893+0000 osd.0 (osd.0) 330 : cluster 0 2.1f deep-scrub ok 2024-03-11T21:11:07.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:06 smithi023 bash[21372]: audit 2024-03-11T21:11:06.064007+0000 mon.a (mon.0) 1746 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:11:07.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:06 smithi023 bash[21372]: audit 2024-03-11T21:11:06.065086+0000 mon.a (mon.0) 1747 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:11:07.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:06 smithi023 bash[21372]: cluster 2024-03-11T21:11:06.375454+0000 osd.5 (osd.5) 427 : cluster 0 10.6 scrub starts 2024-03-11T21:11:07.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:06 smithi023 bash[21372]: cluster 2024-03-11T21:11:06.473031+0000 osd.5 (osd.5) 428 : cluster 0 10.6 scrub ok 2024-03-11T21:11:07.029 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:06 smithi023 bash[29146]: cluster 2024-03-11T21:11:04.813765+0000 osd.7 (osd.7) 473 : cluster 0 10.c scrub starts 2024-03-11T21:11:07.029 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:06 smithi023 bash[29146]: cluster 2024-03-11T21:11:04.859254+0000 osd.7 (osd.7) 474 : cluster 0 10.c scrub ok 2024-03-11T21:11:07.029 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:06 smithi023 bash[29146]: cluster 2024-03-11T21:11:05.879511+0000 osd.0 (osd.0) 329 : cluster 0 2.1f deep-scrub starts 2024-03-11T21:11:07.029 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:06 smithi023 bash[29146]: cluster 2024-03-11T21:11:05.884893+0000 osd.0 (osd.0) 330 : cluster 0 2.1f deep-scrub ok 2024-03-11T21:11:07.029 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:06 smithi023 bash[29146]: audit 2024-03-11T21:11:06.064007+0000 mon.a (mon.0) 1746 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:11:07.029 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:06 smithi023 bash[29146]: audit 2024-03-11T21:11:06.065086+0000 mon.a (mon.0) 1747 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:11:07.029 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:06 smithi023 bash[29146]: cluster 2024-03-11T21:11:06.375454+0000 osd.5 (osd.5) 427 : cluster 0 10.6 scrub starts 2024-03-11T21:11:07.029 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:06 smithi023 bash[29146]: cluster 2024-03-11T21:11:06.473031+0000 osd.5 (osd.5) 428 : cluster 0 10.6 scrub ok 2024-03-11T21:11:07.053 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:11:07.097 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:11:07.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:07.105 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:11:07.205 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:07.455 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:07.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:07.459 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:11:07.560 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:11:07.598 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:11:07.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:07.603 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:11:07.704 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:11:07.714 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:07 smithi146 bash[23809]: cluster 2024-03-11T21:11:05.812942+0000 osd.7 (osd.7) 475 : cluster 0 3.15 scrub starts 2024-03-11T21:11:07.714 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:07 smithi146 bash[23809]: cluster 2024-03-11T21:11:05.817501+0000 osd.7 (osd.7) 476 : cluster 0 3.15 scrub ok 2024-03-11T21:11:07.714 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:07 smithi146 bash[23809]: audit 2024-03-11T21:11:05.866106+0000 mgr.y (mgr.24380) 752 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:11:07.714 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:07 smithi146 bash[23809]: cluster 2024-03-11T21:11:06.338931+0000 mgr.y (mgr.24380) 753 : cluster 0 pgmap v850: 148 pgs: 148 active+clean; 192 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 848 op/s 2024-03-11T21:11:07.714 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:07 smithi146 bash[23809]: cluster 2024-03-11T21:11:06.616804+0000 osd.1 (osd.1) 239 : cluster 0 4.2 scrub starts 2024-03-11T21:11:07.715 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:07 smithi146 bash[23809]: cluster 2024-03-11T21:11:06.619017+0000 osd.1 (osd.1) 240 : cluster 0 4.2 scrub ok 2024-03-11T21:11:07.715 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:07 smithi146 bash[23809]: audit 2024-03-11T21:11:07.151243+0000 mon.a (mon.0) 1748 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:11:07.715 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:07 smithi146 bash[23809]: cluster 2024-03-11T21:11:07.398413+0000 osd.5 (osd.5) 429 : cluster 0 4.15 scrub starts 2024-03-11T21:11:07.715 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:07 smithi146 bash[23809]: cluster 2024-03-11T21:11:07.401713+0000 osd.5 (osd.5) 430 : cluster 0 4.15 scrub ok 2024-03-11T21:11:07.741 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:11:07.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:07.746 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:11:07.847 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:11:07.878 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:11:07.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:07.884 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:11:07.984 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:11:08.014 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:11:08.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:08.018 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:11:08.027 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:07 smithi023 bash[29146]: cluster 2024-03-11T21:11:05.812942+0000 osd.7 (osd.7) 475 : cluster 0 3.15 scrub starts 2024-03-11T21:11:08.027 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:07 smithi023 bash[29146]: cluster 2024-03-11T21:11:05.817501+0000 osd.7 (osd.7) 476 : cluster 0 3.15 scrub ok 2024-03-11T21:11:08.027 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:07 smithi023 bash[29146]: audit 2024-03-11T21:11:05.866106+0000 mgr.y (mgr.24380) 752 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:11:08.027 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:07 smithi023 bash[29146]: cluster 2024-03-11T21:11:06.338931+0000 mgr.y (mgr.24380) 753 : cluster 0 pgmap v850: 148 pgs: 148 active+clean; 192 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 848 op/s 2024-03-11T21:11:08.027 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:07 smithi023 bash[29146]: cluster 2024-03-11T21:11:06.616804+0000 osd.1 (osd.1) 239 : cluster 0 4.2 scrub starts 2024-03-11T21:11:08.027 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:07 smithi023 bash[29146]: cluster 2024-03-11T21:11:06.619017+0000 osd.1 (osd.1) 240 : cluster 0 4.2 scrub ok 2024-03-11T21:11:08.028 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:07 smithi023 bash[29146]: audit 2024-03-11T21:11:07.151243+0000 mon.a (mon.0) 1748 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:11:08.028 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:07 smithi023 bash[29146]: cluster 2024-03-11T21:11:07.398413+0000 osd.5 (osd.5) 429 : cluster 0 4.15 scrub starts 2024-03-11T21:11:08.028 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:07 smithi023 bash[29146]: cluster 2024-03-11T21:11:07.401713+0000 osd.5 (osd.5) 430 : cluster 0 4.15 scrub ok 2024-03-11T21:11:08.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:07 smithi023 bash[21372]: cluster 2024-03-11T21:11:05.812942+0000 osd.7 (osd.7) 475 : cluster 0 3.15 scrub starts 2024-03-11T21:11:08.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:07 smithi023 bash[21372]: cluster 2024-03-11T21:11:05.817501+0000 osd.7 (osd.7) 476 : cluster 0 3.15 scrub ok 2024-03-11T21:11:08.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:07 smithi023 bash[21372]: audit 2024-03-11T21:11:05.866106+0000 mgr.y (mgr.24380) 752 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:11:08.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:07 smithi023 bash[21372]: cluster 2024-03-11T21:11:06.338931+0000 mgr.y (mgr.24380) 753 : cluster 0 pgmap v850: 148 pgs: 148 active+clean; 192 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 848 op/s 2024-03-11T21:11:08.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:07 smithi023 bash[21372]: cluster 2024-03-11T21:11:06.616804+0000 osd.1 (osd.1) 239 : cluster 0 4.2 scrub starts 2024-03-11T21:11:08.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:07 smithi023 bash[21372]: cluster 2024-03-11T21:11:06.619017+0000 osd.1 (osd.1) 240 : cluster 0 4.2 scrub ok 2024-03-11T21:11:08.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:07 smithi023 bash[21372]: audit 2024-03-11T21:11:07.151243+0000 mon.a (mon.0) 1748 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:11:08.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:07 smithi023 bash[21372]: cluster 2024-03-11T21:11:07.398413+0000 osd.5 (osd.5) 429 : cluster 0 4.15 scrub starts 2024-03-11T21:11:08.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:07 smithi023 bash[21372]: cluster 2024-03-11T21:11:07.401713+0000 osd.5 (osd.5) 430 : cluster 0 4.15 scrub ok 2024-03-11T21:11:08.118 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:11:08.137 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:11:08.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:08.139 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:11:08.240 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:11:08.274 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:11:08.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:08.277 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:11:08.378 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:11:08.410 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:11:08.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:08.414 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:11:08.515 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:08.527 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:11:08 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:11:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:11:08.553 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:08.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:08.557 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:11:08.658 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:11:08.687 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:11:08.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:08.689 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:11:08.765 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:Removed 109009 objects 2024-03-11T21:11:08.765 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:Clean up completed and total clean up time :121.377 2024-03-11T21:11:08.790 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:08.850 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:08.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:08.852 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:11:08.953 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:08.964 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:08 smithi146 bash[23809]: cluster 2024-03-11T21:11:07.627211+0000 osd.1 (osd.1) 241 : cluster 0 2.0 scrub starts 2024-03-11T21:11:08.964 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:08 smithi146 bash[23809]: cluster 2024-03-11T21:11:07.632636+0000 osd.1 (osd.1) 242 : cluster 0 2.0 scrub ok 2024-03-11T21:11:08.964 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:08 smithi146 bash[23809]: audit 2024-03-11T21:11:08.348070+0000 mon.a (mon.0) 1749 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:11:08.964 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:08 smithi146 bash[23809]: audit 2024-03-11T21:11:08.349014+0000 mon.a (mon.0) 1750 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:11:08.964 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:08 smithi146 bash[23809]: audit 2024-03-11T21:11:08.356186+0000 mon.a (mon.0) 1751 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:11:08.978 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:08.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:08.981 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:11:09.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:08 smithi023 bash[21372]: cluster 2024-03-11T21:11:07.627211+0000 osd.1 (osd.1) 241 : cluster 0 2.0 scrub starts 2024-03-11T21:11:09.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:08 smithi023 bash[21372]: cluster 2024-03-11T21:11:07.632636+0000 osd.1 (osd.1) 242 : cluster 0 2.0 scrub ok 2024-03-11T21:11:09.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:08 smithi023 bash[21372]: audit 2024-03-11T21:11:08.348070+0000 mon.a (mon.0) 1749 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:11:09.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:08 smithi023 bash[21372]: audit 2024-03-11T21:11:08.349014+0000 mon.a (mon.0) 1750 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:11:09.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:08 smithi023 bash[21372]: audit 2024-03-11T21:11:08.356186+0000 mon.a (mon.0) 1751 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:11:09.028 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:08 smithi023 bash[29146]: cluster 2024-03-11T21:11:07.627211+0000 osd.1 (osd.1) 241 : cluster 0 2.0 scrub starts 2024-03-11T21:11:09.028 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:08 smithi023 bash[29146]: cluster 2024-03-11T21:11:07.632636+0000 osd.1 (osd.1) 242 : cluster 0 2.0 scrub ok 2024-03-11T21:11:09.028 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:08 smithi023 bash[29146]: audit 2024-03-11T21:11:08.348070+0000 mon.a (mon.0) 1749 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:11:09.029 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:08 smithi023 bash[29146]: audit 2024-03-11T21:11:08.349014+0000 mon.a (mon.0) 1750 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:11:09.029 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:08 smithi023 bash[29146]: audit 2024-03-11T21:11:08.356186+0000 mon.a (mon.0) 1751 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:11:09.081 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:09.107 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:09.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:09.109 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:11:09.210 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:09.231 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:09.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:09.233 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:11:09.334 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:09.363 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:09.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:09.365 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:11:09.466 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:11:09.491 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:11:09.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:09.493 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:11:09.567 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:11:09.594 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:11:09.614 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:11:09.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:09.616 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:11:09.717 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:11:09.728 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:09 smithi146 bash[23809]: cluster 2024-03-11T21:11:07.797362+0000 osd.7 (osd.7) 477 : cluster 0 6.12 scrub starts 2024-03-11T21:11:09.728 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:09 smithi146 bash[23809]: cluster 2024-03-11T21:11:07.805298+0000 osd.7 (osd.7) 478 : cluster 0 6.12 scrub ok 2024-03-11T21:11:09.728 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:09 smithi146 bash[23809]: cluster 2024-03-11T21:11:08.339486+0000 mgr.y (mgr.24380) 754 : cluster 0 pgmap v851: 148 pgs: 148 active+clean; 176 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 845 op/s 2024-03-11T21:11:09.728 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:09 smithi146 bash[23809]: cluster 2024-03-11T21:11:08.789236+0000 osd.0 (osd.0) 331 : cluster 0 3.1f scrub starts 2024-03-11T21:11:09.728 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:09 smithi146 bash[23809]: cluster 2024-03-11T21:11:08.790999+0000 osd.0 (osd.0) 332 : cluster 0 3.1f scrub ok 2024-03-11T21:11:09.741 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:11:09.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:09.742 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:11:09.746 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-03-11T21:11:09.746 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-03-11T21:11:09.746 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 30, 2024-03-11T21:11:09.746 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-03-11T21:11:09.746 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-03-11T21:11:09.844 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:09.864 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:09.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:09.865 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:11:09.966 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:09.980 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:09 smithi023 bash[29146]: cluster 2024-03-11T21:11:07.797362+0000 osd.7 (osd.7) 477 : cluster 0 6.12 scrub starts 2024-03-11T21:11:09.980 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:09 smithi023 bash[29146]: cluster 2024-03-11T21:11:07.805298+0000 osd.7 (osd.7) 478 : cluster 0 6.12 scrub ok 2024-03-11T21:11:09.980 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:09 smithi023 bash[29146]: cluster 2024-03-11T21:11:08.339486+0000 mgr.y (mgr.24380) 754 : cluster 0 pgmap v851: 148 pgs: 148 active+clean; 176 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 845 op/s 2024-03-11T21:11:09.980 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:09 smithi023 bash[29146]: cluster 2024-03-11T21:11:08.789236+0000 osd.0 (osd.0) 331 : cluster 0 3.1f scrub starts 2024-03-11T21:11:09.981 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:09 smithi023 bash[29146]: cluster 2024-03-11T21:11:08.790999+0000 osd.0 (osd.0) 332 : cluster 0 3.1f scrub ok 2024-03-11T21:11:09.981 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:09 smithi023 bash[21372]: cluster 2024-03-11T21:11:07.797362+0000 osd.7 (osd.7) 477 : cluster 0 6.12 scrub starts 2024-03-11T21:11:09.981 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:09 smithi023 bash[21372]: cluster 2024-03-11T21:11:07.805298+0000 osd.7 (osd.7) 478 : cluster 0 6.12 scrub ok 2024-03-11T21:11:09.981 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:09 smithi023 bash[21372]: cluster 2024-03-11T21:11:08.339486+0000 mgr.y (mgr.24380) 754 : cluster 0 pgmap v851: 148 pgs: 148 active+clean; 176 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 845 op/s 2024-03-11T21:11:09.981 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:09 smithi023 bash[21372]: cluster 2024-03-11T21:11:08.789236+0000 osd.0 (osd.0) 331 : cluster 0 3.1f scrub starts 2024-03-11T21:11:09.981 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:09 smithi023 bash[21372]: cluster 2024-03-11T21:11:08.790999+0000 osd.0 (osd.0) 332 : cluster 0 3.1f scrub ok 2024-03-11T21:11:09.995 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:09.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:09.998 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:11:10.101 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:10.130 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:10.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:10.133 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:11:10.234 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:10.260 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:10.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:10.262 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:11:10.363 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:11:10.391 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:11:10.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:10.394 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:11:10.494 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:11:10.523 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:11:10.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:10.526 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:11:10.627 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:10.652 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:10.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:10.655 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:11:10.756 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:11:10.763 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-03-11T21:11:10.767 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:10 smithi023 bash[21372]: cluster 2024-03-11T21:11:09.680993+0000 osd.1 (osd.1) 243 : cluster 0 3.1a scrub starts 2024-03-11T21:11:10.767 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:10 smithi023 bash[21372]: cluster 2024-03-11T21:11:09.683712+0000 osd.1 (osd.1) 244 : cluster 0 3.1a scrub ok 2024-03-11T21:11:10.767 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:10 smithi023 bash[21372]: cluster 2024-03-11T21:11:09.787373+0000 osd.6 (osd.6) 350 : cluster 0 10.b scrub starts 2024-03-11T21:11:10.767 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:10 smithi023 bash[21372]: cluster 2024-03-11T21:11:09.802838+0000 osd.6 (osd.6) 351 : cluster 0 10.b scrub ok 2024-03-11T21:11:10.767 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:10 smithi023 bash[21372]: cluster 2024-03-11T21:11:09.938792+0000 osd.4 (osd.4) 111 : cluster 0 purged_snaps scrub starts 2024-03-11T21:11:10.767 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:10 smithi023 bash[21372]: cluster 2024-03-11T21:11:09.938926+0000 osd.4 (osd.4) 112 : cluster 0 purged_snaps scrub ok 2024-03-11T21:11:10.767 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:10 smithi023 bash[29146]: cluster 2024-03-11T21:11:09.680993+0000 osd.1 (osd.1) 243 : cluster 0 3.1a scrub starts 2024-03-11T21:11:10.768 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:10 smithi023 bash[29146]: cluster 2024-03-11T21:11:09.683712+0000 osd.1 (osd.1) 244 : cluster 0 3.1a scrub ok 2024-03-11T21:11:10.768 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:10 smithi023 bash[29146]: cluster 2024-03-11T21:11:09.787373+0000 osd.6 (osd.6) 350 : cluster 0 10.b scrub starts 2024-03-11T21:11:10.768 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:10 smithi023 bash[29146]: cluster 2024-03-11T21:11:09.802838+0000 osd.6 (osd.6) 351 : cluster 0 10.b scrub ok 2024-03-11T21:11:10.768 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:10 smithi023 bash[29146]: cluster 2024-03-11T21:11:09.938792+0000 osd.4 (osd.4) 111 : cluster 0 purged_snaps scrub starts 2024-03-11T21:11:10.768 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:10 smithi023 bash[29146]: cluster 2024-03-11T21:11:09.938926+0000 osd.4 (osd.4) 112 : cluster 0 purged_snaps scrub ok 2024-03-11T21:11:10.781 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:11:10.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:10.783 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:11:10.884 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:11:10.912 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:11:10.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:10.914 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:11:11.007 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:10 smithi146 bash[23809]: cluster 2024-03-11T21:11:09.680993+0000 osd.1 (osd.1) 243 : cluster 0 3.1a scrub starts 2024-03-11T21:11:11.008 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:10 smithi146 bash[23809]: cluster 2024-03-11T21:11:09.683712+0000 osd.1 (osd.1) 244 : cluster 0 3.1a scrub ok 2024-03-11T21:11:11.008 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:10 smithi146 bash[23809]: cluster 2024-03-11T21:11:09.787373+0000 osd.6 (osd.6) 350 : cluster 0 10.b scrub starts 2024-03-11T21:11:11.008 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:10 smithi146 bash[23809]: cluster 2024-03-11T21:11:09.802838+0000 osd.6 (osd.6) 351 : cluster 0 10.b scrub ok 2024-03-11T21:11:11.008 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:10 smithi146 bash[23809]: cluster 2024-03-11T21:11:09.938792+0000 osd.4 (osd.4) 111 : cluster 0 purged_snaps scrub starts 2024-03-11T21:11:11.008 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:10 smithi146 bash[23809]: cluster 2024-03-11T21:11:09.938926+0000 osd.4 (osd.4) 112 : cluster 0 purged_snaps scrub ok 2024-03-11T21:11:11.015 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:11:11.042 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:11:11.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:11.044 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:11:11.145 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:11.174 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:11.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:11.176 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:11:11.277 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:11:11.304 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:11:11.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:11.307 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:11:11.408 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:11.437 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:11.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:11.440 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:11:11.541 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:11:11.570 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:11:11.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:11.573 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:11:11.603 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:11:11.603 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:11:11.674 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:11.698 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:11.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:11.700 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:11:11.801 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:11:11.832 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:11:11.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:11.835 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:11:11.936 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:11:11.949 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:11 smithi023 bash[21372]: cluster 2024-03-11T21:11:10.340615+0000 mgr.y (mgr.24380) 755 : cluster 0 pgmap v852: 148 pgs: 148 active+clean; 71 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 779 op/s 2024-03-11T21:11:11.949 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:11 smithi023 bash[21372]: cluster 2024-03-11T21:11:10.676118+0000 osd.1 (osd.1) 245 : cluster 0 6.1 deep-scrub starts 2024-03-11T21:11:11.949 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:11 smithi023 bash[21372]: cluster 2024-03-11T21:11:10.720381+0000 osd.1 (osd.1) 246 : cluster 0 6.1 deep-scrub ok 2024-03-11T21:11:11.949 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:11 smithi023 bash[21372]: cluster 2024-03-11T21:11:10.726313+0000 mon.a (mon.0) 1752 : cluster 0 osdmap e363: 8 total, 8 up, 8 in 2024-03-11T21:11:11.949 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:11 smithi023 bash[21372]: cluster 2024-03-11T21:11:11.064095+0000 osd.3 (osd.3) 420 : cluster 0 6.11 scrub starts 2024-03-11T21:11:11.949 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:11 smithi023 bash[21372]: cluster 2024-03-11T21:11:11.081128+0000 osd.3 (osd.3) 421 : cluster 0 6.11 scrub ok 2024-03-11T21:11:11.950 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:11 smithi023 bash[29146]: cluster 2024-03-11T21:11:10.340615+0000 mgr.y (mgr.24380) 755 : cluster 0 pgmap v852: 148 pgs: 148 active+clean; 71 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 779 op/s 2024-03-11T21:11:11.950 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:11 smithi023 bash[29146]: cluster 2024-03-11T21:11:10.676118+0000 osd.1 (osd.1) 245 : cluster 0 6.1 deep-scrub starts 2024-03-11T21:11:11.950 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:11 smithi023 bash[29146]: cluster 2024-03-11T21:11:10.720381+0000 osd.1 (osd.1) 246 : cluster 0 6.1 deep-scrub ok 2024-03-11T21:11:11.950 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:11 smithi023 bash[29146]: cluster 2024-03-11T21:11:10.726313+0000 mon.a (mon.0) 1752 : cluster 0 osdmap e363: 8 total, 8 up, 8 in 2024-03-11T21:11:11.950 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:11 smithi023 bash[29146]: cluster 2024-03-11T21:11:11.064095+0000 osd.3 (osd.3) 420 : cluster 0 6.11 scrub starts 2024-03-11T21:11:11.950 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:11 smithi023 bash[29146]: cluster 2024-03-11T21:11:11.081128+0000 osd.3 (osd.3) 421 : cluster 0 6.11 scrub ok 2024-03-11T21:11:11.968 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:11:11.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:11.970 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:11:12.007 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:11 smithi146 bash[23809]: cluster 2024-03-11T21:11:10.340615+0000 mgr.y (mgr.24380) 755 : cluster 0 pgmap v852: 148 pgs: 148 active+clean; 71 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 779 op/s 2024-03-11T21:11:12.008 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:11 smithi146 bash[23809]: cluster 2024-03-11T21:11:10.676118+0000 osd.1 (osd.1) 245 : cluster 0 6.1 deep-scrub starts 2024-03-11T21:11:12.008 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:11 smithi146 bash[23809]: cluster 2024-03-11T21:11:10.720381+0000 osd.1 (osd.1) 246 : cluster 0 6.1 deep-scrub ok 2024-03-11T21:11:12.008 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:11 smithi146 bash[23809]: cluster 2024-03-11T21:11:10.726313+0000 mon.a (mon.0) 1752 : cluster 0 osdmap e363: 8 total, 8 up, 8 in 2024-03-11T21:11:12.008 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:11 smithi146 bash[23809]: cluster 2024-03-11T21:11:11.064095+0000 osd.3 (osd.3) 420 : cluster 0 6.11 scrub starts 2024-03-11T21:11:12.008 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:11 smithi146 bash[23809]: cluster 2024-03-11T21:11:11.081128+0000 osd.3 (osd.3) 421 : cluster 0 6.11 scrub ok 2024-03-11T21:11:12.071 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:11:12.100 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:11:12.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:12.102 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:11:12.203 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:11:12.230 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:11:12.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:12.232 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:11:12.333 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:12.361 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:12.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:12.366 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:11:12.467 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:12.495 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:12.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:12.497 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:11:12.598 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:11:12.623 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:11:12.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:12.626 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:11:12.727 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:11:12.754 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:11:12.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:12.757 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:11:12.757 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:12 smithi146 bash[23809]: cluster 2024-03-11T21:11:11.806616+0000 osd.6 (osd.6) 352 : cluster 0 purged_snaps scrub starts 2024-03-11T21:11:12.757 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:12 smithi146 bash[23809]: cluster 2024-03-11T21:11:11.806731+0000 osd.6 (osd.6) 353 : cluster 0 purged_snaps scrub ok 2024-03-11T21:11:12.757 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:12 smithi146 bash[23809]: cluster 2024-03-11T21:11:12.341498+0000 mgr.y (mgr.24380) 756 : cluster 0 pgmap v854: 148 pgs: 148 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 719 op/s 2024-03-11T21:11:12.757 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:12 smithi146 bash[23809]: cluster 2024-03-11T21:11:12.401006+0000 osd.5 (osd.5) 431 : cluster 0 3.c scrub starts 2024-03-11T21:11:12.758 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:12 smithi146 bash[23809]: cluster 2024-03-11T21:11:12.404523+0000 osd.5 (osd.5) 432 : cluster 0 3.c scrub ok 2024-03-11T21:11:12.857 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:12.883 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:12.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:12.885 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:11:12.986 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:11:13.012 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:11:13.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:13.014 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:11:13.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:12 smithi023 bash[21372]: cluster 2024-03-11T21:11:11.806616+0000 osd.6 (osd.6) 352 : cluster 0 purged_snaps scrub starts 2024-03-11T21:11:13.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:12 smithi023 bash[21372]: cluster 2024-03-11T21:11:11.806731+0000 osd.6 (osd.6) 353 : cluster 0 purged_snaps scrub ok 2024-03-11T21:11:13.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:12 smithi023 bash[21372]: cluster 2024-03-11T21:11:12.341498+0000 mgr.y (mgr.24380) 756 : cluster 0 pgmap v854: 148 pgs: 148 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 719 op/s 2024-03-11T21:11:13.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:12 smithi023 bash[21372]: cluster 2024-03-11T21:11:12.401006+0000 osd.5 (osd.5) 431 : cluster 0 3.c scrub starts 2024-03-11T21:11:13.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:12 smithi023 bash[21372]: cluster 2024-03-11T21:11:12.404523+0000 osd.5 (osd.5) 432 : cluster 0 3.c scrub ok 2024-03-11T21:11:13.028 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:12 smithi023 bash[29146]: cluster 2024-03-11T21:11:11.806616+0000 osd.6 (osd.6) 352 : cluster 0 purged_snaps scrub starts 2024-03-11T21:11:13.028 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:12 smithi023 bash[29146]: cluster 2024-03-11T21:11:11.806731+0000 osd.6 (osd.6) 353 : cluster 0 purged_snaps scrub ok 2024-03-11T21:11:13.028 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:12 smithi023 bash[29146]: cluster 2024-03-11T21:11:12.341498+0000 mgr.y (mgr.24380) 756 : cluster 0 pgmap v854: 148 pgs: 148 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 719 op/s 2024-03-11T21:11:13.028 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:12 smithi023 bash[29146]: cluster 2024-03-11T21:11:12.401006+0000 osd.5 (osd.5) 431 : cluster 0 3.c scrub starts 2024-03-11T21:11:13.028 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:12 smithi023 bash[29146]: cluster 2024-03-11T21:11:12.404523+0000 osd.5 (osd.5) 432 : cluster 0 3.c scrub ok 2024-03-11T21:11:13.115 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:13.141 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:13.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:13.143 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:11:13.244 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:13.270 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:13.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:13.272 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:11:13.373 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:11:13.402 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:11:13.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:13.404 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:11:13.505 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:11:13.528 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:11:13.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:13.530 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:11:13.631 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:13.658 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:13.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:13.660 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:11:13.760 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:11:13.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:13 smithi023 bash[21372]: cluster 2024-03-11T21:11:12.742311+0000 mon.a (mon.0) 1753 : cluster 0 osdmap e364: 8 total, 8 up, 8 in 2024-03-11T21:11:13.772 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:13 smithi023 bash[29146]: cluster 2024-03-11T21:11:12.742311+0000 mon.a (mon.0) 1753 : cluster 0 osdmap e364: 8 total, 8 up, 8 in 2024-03-11T21:11:13.788 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:11:13.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:13.791 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:11:13.891 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:11:13.903 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:13 smithi146 bash[23809]: cluster 2024-03-11T21:11:12.742311+0000 mon.a (mon.0) 1753 : cluster 0 osdmap e364: 8 total, 8 up, 8 in 2024-03-11T21:11:13.920 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:11:13.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:13.923 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:11:14.024 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:11:14.053 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:11:14.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:14.056 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:11:14.156 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:14.186 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:14.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:14.188 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:11:14.289 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:14.318 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:14.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:14.321 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:11:14.421 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:14.448 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:14.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:14.451 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:11:14.551 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:11:14.579 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:11:14.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:14.582 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:11:14.683 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:14.711 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:14.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:14.713 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:11:14.814 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:11:14.826 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:14 smithi023 bash[21372]: cluster 2024-03-11T21:11:14.341987+0000 mgr.y (mgr.24380) 757 : cluster 0 pgmap v856: 148 pgs: 148 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 382 op/s 2024-03-11T21:11:14.827 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:14 smithi023 bash[21372]: cluster 2024-03-11T21:11:14.422566+0000 osd.5 (osd.5) 433 : cluster 0 10.3 scrub starts 2024-03-11T21:11:14.827 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:14 smithi023 bash[21372]: cluster 2024-03-11T21:11:14.444584+0000 osd.5 (osd.5) 434 : cluster 0 10.3 scrub ok 2024-03-11T21:11:14.828 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:14 smithi023 bash[29146]: cluster 2024-03-11T21:11:14.341987+0000 mgr.y (mgr.24380) 757 : cluster 0 pgmap v856: 148 pgs: 148 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 382 op/s 2024-03-11T21:11:14.828 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:14 smithi023 bash[29146]: cluster 2024-03-11T21:11:14.422566+0000 osd.5 (osd.5) 433 : cluster 0 10.3 scrub starts 2024-03-11T21:11:14.828 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:14 smithi023 bash[29146]: cluster 2024-03-11T21:11:14.444584+0000 osd.5 (osd.5) 434 : cluster 0 10.3 scrub ok 2024-03-11T21:11:14.841 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:11:14.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:14.844 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:11:14.945 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:14.972 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:14.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:14.974 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:11:15.007 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:14 smithi146 bash[23809]: cluster 2024-03-11T21:11:14.341987+0000 mgr.y (mgr.24380) 757 : cluster 0 pgmap v856: 148 pgs: 148 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 382 op/s 2024-03-11T21:11:15.007 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:14 smithi146 bash[23809]: cluster 2024-03-11T21:11:14.422566+0000 osd.5 (osd.5) 433 : cluster 0 10.3 scrub starts 2024-03-11T21:11:15.007 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:14 smithi146 bash[23809]: cluster 2024-03-11T21:11:14.444584+0000 osd.5 (osd.5) 434 : cluster 0 10.3 scrub ok 2024-03-11T21:11:15.074 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:11:15.100 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:11:15.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:15.102 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:11:15.203 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:11:15.224 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:11:15.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:15.226 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:11:15.327 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:11:15.357 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:11:15.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:15.359 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:11:15.460 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:15.481 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:15.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:15.483 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:11:15.551 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:11:15.583 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:11:15.605 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:11:15.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:15.607 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:11:15.708 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:11:15.729 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:11:15.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:15.732 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:11:15.832 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:15.844 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:15 smithi146 bash[23809]: cluster 2024-03-11T21:11:14.778437+0000 osd.6 (osd.6) 354 : cluster 0 4.a scrub starts 2024-03-11T21:11:15.844 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:15 smithi146 bash[23809]: cluster 2024-03-11T21:11:14.780658+0000 osd.6 (osd.6) 355 : cluster 0 4.a scrub ok 2024-03-11T21:11:15.844 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:15 smithi146 bash[23809]: cluster 2024-03-11T21:11:14.834427+0000 osd.7 (osd.7) 479 : cluster 0 2.b deep-scrub starts 2024-03-11T21:11:15.844 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:15 smithi146 bash[23809]: cluster 2024-03-11T21:11:14.836798+0000 osd.7 (osd.7) 480 : cluster 0 2.b deep-scrub ok 2024-03-11T21:11:15.844 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:15 smithi146 bash[23809]: cluster 2024-03-11T21:11:14.928897+0000 osd.0 (osd.0) 333 : cluster 0 6.0 scrub starts 2024-03-11T21:11:15.845 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:15 smithi146 bash[23809]: cluster 2024-03-11T21:11:14.978937+0000 osd.0 (osd.0) 334 : cluster 0 6.0 scrub ok 2024-03-11T21:11:15.845 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:15 smithi146 bash[23809]: cluster 2024-03-11T21:11:15.097209+0000 osd.3 (osd.3) 422 : cluster 0 6.1b scrub starts 2024-03-11T21:11:15.845 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:15 smithi146 bash[23809]: cluster 2024-03-11T21:11:15.221778+0000 osd.3 (osd.3) 423 : cluster 0 6.1b scrub ok 2024-03-11T21:11:15.859 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:15.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:15.861 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:11:15.962 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:15.990 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:15.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:15.992 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:11:16.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:15 smithi023 bash[21372]: cluster 2024-03-11T21:11:14.778437+0000 osd.6 (osd.6) 354 : cluster 0 4.a scrub starts 2024-03-11T21:11:16.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:15 smithi023 bash[21372]: cluster 2024-03-11T21:11:14.780658+0000 osd.6 (osd.6) 355 : cluster 0 4.a scrub ok 2024-03-11T21:11:16.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:15 smithi023 bash[21372]: cluster 2024-03-11T21:11:14.834427+0000 osd.7 (osd.7) 479 : cluster 0 2.b deep-scrub starts 2024-03-11T21:11:16.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:15 smithi023 bash[21372]: cluster 2024-03-11T21:11:14.836798+0000 osd.7 (osd.7) 480 : cluster 0 2.b deep-scrub ok 2024-03-11T21:11:16.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:15 smithi023 bash[21372]: cluster 2024-03-11T21:11:14.928897+0000 osd.0 (osd.0) 333 : cluster 0 6.0 scrub starts 2024-03-11T21:11:16.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:15 smithi023 bash[21372]: cluster 2024-03-11T21:11:14.978937+0000 osd.0 (osd.0) 334 : cluster 0 6.0 scrub ok 2024-03-11T21:11:16.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:15 smithi023 bash[21372]: cluster 2024-03-11T21:11:15.097209+0000 osd.3 (osd.3) 422 : cluster 0 6.1b scrub starts 2024-03-11T21:11:16.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:15 smithi023 bash[21372]: cluster 2024-03-11T21:11:15.221778+0000 osd.3 (osd.3) 423 : cluster 0 6.1b scrub ok 2024-03-11T21:11:16.028 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:15 smithi023 bash[29146]: cluster 2024-03-11T21:11:14.778437+0000 osd.6 (osd.6) 354 : cluster 0 4.a scrub starts 2024-03-11T21:11:16.029 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:15 smithi023 bash[29146]: cluster 2024-03-11T21:11:14.780658+0000 osd.6 (osd.6) 355 : cluster 0 4.a scrub ok 2024-03-11T21:11:16.029 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:15 smithi023 bash[29146]: cluster 2024-03-11T21:11:14.834427+0000 osd.7 (osd.7) 479 : cluster 0 2.b deep-scrub starts 2024-03-11T21:11:16.029 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:15 smithi023 bash[29146]: cluster 2024-03-11T21:11:14.836798+0000 osd.7 (osd.7) 480 : cluster 0 2.b deep-scrub ok 2024-03-11T21:11:16.029 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:15 smithi023 bash[29146]: cluster 2024-03-11T21:11:14.928897+0000 osd.0 (osd.0) 333 : cluster 0 6.0 scrub starts 2024-03-11T21:11:16.029 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:15 smithi023 bash[29146]: cluster 2024-03-11T21:11:14.978937+0000 osd.0 (osd.0) 334 : cluster 0 6.0 scrub ok 2024-03-11T21:11:16.029 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:15 smithi023 bash[29146]: cluster 2024-03-11T21:11:15.097209+0000 osd.3 (osd.3) 422 : cluster 0 6.1b scrub starts 2024-03-11T21:11:16.029 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:15 smithi023 bash[29146]: cluster 2024-03-11T21:11:15.221778+0000 osd.3 (osd.3) 423 : cluster 0 6.1b scrub ok 2024-03-11T21:11:16.093 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:16.104 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:11:15 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:11:16.122 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:16.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:16.124 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:11:16.225 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:11:16.254 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:11:16.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:16.257 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:11:16.358 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:11:16.384 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:11:16.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:16.386 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:11:16.487 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:11:16.514 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:11:16.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:16.516 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:11:16.617 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:11:16.645 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:11:16.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:16.648 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:11:16.749 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:11:16.775 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:11:16.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:16.777 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:11:16.878 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:16.889 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:16 smithi146 bash[23809]: cluster 2024-03-11T21:11:15.813005+0000 osd.6 (osd.6) 356 : cluster 0 3.10 deep-scrub starts 2024-03-11T21:11:16.890 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:16 smithi146 bash[23809]: cluster 2024-03-11T21:11:15.819233+0000 osd.6 (osd.6) 357 : cluster 0 3.10 deep-scrub ok 2024-03-11T21:11:16.890 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:16 smithi146 bash[23809]: cluster 2024-03-11T21:11:15.834685+0000 osd.7 (osd.7) 481 : cluster 0 3.d scrub starts 2024-03-11T21:11:16.890 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:16 smithi146 bash[23809]: cluster 2024-03-11T21:11:15.839667+0000 osd.7 (osd.7) 482 : cluster 0 3.d scrub ok 2024-03-11T21:11:16.890 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:16 smithi146 bash[23809]: audit 2024-03-11T21:11:15.873287+0000 mgr.y (mgr.24380) 758 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:11:16.890 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:16 smithi146 bash[23809]: cluster 2024-03-11T21:11:15.881264+0000 osd.0 (osd.0) 335 : cluster 0 4.6 scrub starts 2024-03-11T21:11:16.890 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:16 smithi146 bash[23809]: cluster 2024-03-11T21:11:15.883953+0000 osd.0 (osd.0) 336 : cluster 0 4.6 scrub ok 2024-03-11T21:11:16.890 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:16 smithi146 bash[23809]: cluster 2024-03-11T21:11:16.343506+0000 mgr.y (mgr.24380) 759 : cluster 0 pgmap v857: 148 pgs: 148 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 351 op/s 2024-03-11T21:11:16.906 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:16.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:16.908 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:11:17.009 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:17.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:16 smithi023 bash[21372]: cluster 2024-03-11T21:11:15.813005+0000 osd.6 (osd.6) 356 : cluster 0 3.10 deep-scrub starts 2024-03-11T21:11:17.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:16 smithi023 bash[21372]: cluster 2024-03-11T21:11:15.819233+0000 osd.6 (osd.6) 357 : cluster 0 3.10 deep-scrub ok 2024-03-11T21:11:17.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:16 smithi023 bash[21372]: cluster 2024-03-11T21:11:15.834685+0000 osd.7 (osd.7) 481 : cluster 0 3.d scrub starts 2024-03-11T21:11:17.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:16 smithi023 bash[21372]: cluster 2024-03-11T21:11:15.839667+0000 osd.7 (osd.7) 482 : cluster 0 3.d scrub ok 2024-03-11T21:11:17.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:16 smithi023 bash[21372]: audit 2024-03-11T21:11:15.873287+0000 mgr.y (mgr.24380) 758 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:11:17.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:16 smithi023 bash[21372]: cluster 2024-03-11T21:11:15.881264+0000 osd.0 (osd.0) 335 : cluster 0 4.6 scrub starts 2024-03-11T21:11:17.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:16 smithi023 bash[21372]: cluster 2024-03-11T21:11:15.883953+0000 osd.0 (osd.0) 336 : cluster 0 4.6 scrub ok 2024-03-11T21:11:17.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:16 smithi023 bash[21372]: cluster 2024-03-11T21:11:16.343506+0000 mgr.y (mgr.24380) 759 : cluster 0 pgmap v857: 148 pgs: 148 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 351 op/s 2024-03-11T21:11:17.028 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:16 smithi023 bash[29146]: cluster 2024-03-11T21:11:15.813005+0000 osd.6 (osd.6) 356 : cluster 0 3.10 deep-scrub starts 2024-03-11T21:11:17.029 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:16 smithi023 bash[29146]: cluster 2024-03-11T21:11:15.819233+0000 osd.6 (osd.6) 357 : cluster 0 3.10 deep-scrub ok 2024-03-11T21:11:17.029 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:16 smithi023 bash[29146]: cluster 2024-03-11T21:11:15.834685+0000 osd.7 (osd.7) 481 : cluster 0 3.d scrub starts 2024-03-11T21:11:17.029 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:16 smithi023 bash[29146]: cluster 2024-03-11T21:11:15.839667+0000 osd.7 (osd.7) 482 : cluster 0 3.d scrub ok 2024-03-11T21:11:17.029 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:16 smithi023 bash[29146]: audit 2024-03-11T21:11:15.873287+0000 mgr.y (mgr.24380) 758 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:11:17.029 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:16 smithi023 bash[29146]: cluster 2024-03-11T21:11:15.881264+0000 osd.0 (osd.0) 335 : cluster 0 4.6 scrub starts 2024-03-11T21:11:17.029 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:16 smithi023 bash[29146]: cluster 2024-03-11T21:11:15.883953+0000 osd.0 (osd.0) 336 : cluster 0 4.6 scrub ok 2024-03-11T21:11:17.029 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:16 smithi023 bash[29146]: cluster 2024-03-11T21:11:16.343506+0000 mgr.y (mgr.24380) 759 : cluster 0 pgmap v857: 148 pgs: 148 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 351 op/s 2024-03-11T21:11:17.030 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:17.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:17.032 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:11:17.133 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:11:17.162 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:11:17.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:17.164 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:11:17.265 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:11:17.292 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:11:17.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:17.294 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:11:17.395 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:17.422 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:17.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:17.424 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:11:17.525 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:17.554 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:17.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:17.556 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:11:17.657 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:11:17.686 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:11:17.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:17.688 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:11:17.789 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:11:17.816 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:11:17.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:17.819 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:11:17.920 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:11:17.929 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:17 smithi023 bash[21372]: cluster 2024-03-11T21:11:16.861307+0000 osd.6 (osd.6) 358 : cluster 0 2.17 scrub starts 2024-03-11T21:11:17.930 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:17 smithi023 bash[21372]: cluster 2024-03-11T21:11:16.861927+0000 osd.7 (osd.7) 483 : cluster 0 3.14 scrub starts 2024-03-11T21:11:17.930 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:17 smithi023 bash[21372]: cluster 2024-03-11T21:11:16.864236+0000 osd.6 (osd.6) 359 : cluster 0 2.17 scrub ok 2024-03-11T21:11:17.930 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:17 smithi023 bash[21372]: cluster 2024-03-11T21:11:16.867316+0000 osd.7 (osd.7) 484 : cluster 0 3.14 scrub ok 2024-03-11T21:11:17.930 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:17 smithi023 bash[29146]: cluster 2024-03-11T21:11:16.861307+0000 osd.6 (osd.6) 358 : cluster 0 2.17 scrub starts 2024-03-11T21:11:17.930 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:17 smithi023 bash[29146]: cluster 2024-03-11T21:11:16.861927+0000 osd.7 (osd.7) 483 : cluster 0 3.14 scrub starts 2024-03-11T21:11:17.930 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:17 smithi023 bash[29146]: cluster 2024-03-11T21:11:16.864236+0000 osd.6 (osd.6) 359 : cluster 0 2.17 scrub ok 2024-03-11T21:11:17.930 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:17 smithi023 bash[29146]: cluster 2024-03-11T21:11:16.867316+0000 osd.7 (osd.7) 484 : cluster 0 3.14 scrub ok 2024-03-11T21:11:17.944 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:11:17.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:17.947 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:11:18.007 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:17 smithi146 bash[23809]: cluster 2024-03-11T21:11:16.861307+0000 osd.6 (osd.6) 358 : cluster 0 2.17 scrub starts 2024-03-11T21:11:18.007 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:17 smithi146 bash[23809]: cluster 2024-03-11T21:11:16.861927+0000 osd.7 (osd.7) 483 : cluster 0 3.14 scrub starts 2024-03-11T21:11:18.007 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:17 smithi146 bash[23809]: cluster 2024-03-11T21:11:16.864236+0000 osd.6 (osd.6) 359 : cluster 0 2.17 scrub ok 2024-03-11T21:11:18.008 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:17 smithi146 bash[23809]: cluster 2024-03-11T21:11:16.867316+0000 osd.7 (osd.7) 484 : cluster 0 3.14 scrub ok 2024-03-11T21:11:18.047 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:11:18.069 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:11:18.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:18.071 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:11:18.171 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:18.197 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:18.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:18.200 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:11:18.301 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:18.328 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:18.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:18.330 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:11:18.431 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:18.458 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:18.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:18.460 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:11:18.561 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:11:18.587 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:11:18.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:18.589 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:11:18.690 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:18.719 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:18.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:18.723 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:11:18.777 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:11:18 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:11:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:11:18.823 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:18.848 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:18.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:18.850 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:11:18.951 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:11:18.979 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:11:18.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:18.981 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:11:19.007 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:18 smithi146 bash[23809]: cluster 2024-03-11T21:11:17.942370+0000 osd.0 (osd.0) 337 : cluster 0 3.12 scrub starts 2024-03-11T21:11:19.007 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:18 smithi146 bash[23809]: cluster 2024-03-11T21:11:17.944539+0000 osd.0 (osd.0) 338 : cluster 0 3.12 scrub ok 2024-03-11T21:11:19.008 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:18 smithi146 bash[23809]: cluster 2024-03-11T21:11:18.344062+0000 mgr.y (mgr.24380) 760 : cluster 0 pgmap v858: 148 pgs: 148 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 141 op/s 2024-03-11T21:11:19.008 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:18 smithi146 bash[23809]: cluster 2024-03-11T21:11:18.369744+0000 osd.5 (osd.5) 435 : cluster 0 4.8 scrub starts 2024-03-11T21:11:19.008 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:18 smithi146 bash[23809]: cluster 2024-03-11T21:11:18.372271+0000 osd.5 (osd.5) 436 : cluster 0 4.8 scrub ok 2024-03-11T21:11:19.082 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:11:19.110 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:11:19.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:19.112 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:11:19.213 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:19.224 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:18 smithi023 bash[29146]: cluster 2024-03-11T21:11:17.942370+0000 osd.0 (osd.0) 337 : cluster 0 3.12 scrub starts 2024-03-11T21:11:19.224 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:18 smithi023 bash[29146]: cluster 2024-03-11T21:11:17.944539+0000 osd.0 (osd.0) 338 : cluster 0 3.12 scrub ok 2024-03-11T21:11:19.224 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:18 smithi023 bash[29146]: cluster 2024-03-11T21:11:18.344062+0000 mgr.y (mgr.24380) 760 : cluster 0 pgmap v858: 148 pgs: 148 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 141 op/s 2024-03-11T21:11:19.224 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:18 smithi023 bash[29146]: cluster 2024-03-11T21:11:18.369744+0000 osd.5 (osd.5) 435 : cluster 0 4.8 scrub starts 2024-03-11T21:11:19.224 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:18 smithi023 bash[29146]: cluster 2024-03-11T21:11:18.372271+0000 osd.5 (osd.5) 436 : cluster 0 4.8 scrub ok 2024-03-11T21:11:19.225 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:18 smithi023 bash[21372]: cluster 2024-03-11T21:11:17.942370+0000 osd.0 (osd.0) 337 : cluster 0 3.12 scrub starts 2024-03-11T21:11:19.225 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:18 smithi023 bash[21372]: cluster 2024-03-11T21:11:17.944539+0000 osd.0 (osd.0) 338 : cluster 0 3.12 scrub ok 2024-03-11T21:11:19.225 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:18 smithi023 bash[21372]: cluster 2024-03-11T21:11:18.344062+0000 mgr.y (mgr.24380) 760 : cluster 0 pgmap v858: 148 pgs: 148 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 141 op/s 2024-03-11T21:11:19.225 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:18 smithi023 bash[21372]: cluster 2024-03-11T21:11:18.369744+0000 osd.5 (osd.5) 435 : cluster 0 4.8 scrub starts 2024-03-11T21:11:19.225 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:18 smithi023 bash[21372]: cluster 2024-03-11T21:11:18.372271+0000 osd.5 (osd.5) 436 : cluster 0 4.8 scrub ok 2024-03-11T21:11:19.242 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:19.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:19.244 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:11:19.322 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-03-11T21:11:19.323 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-03-11T21:11:19.323 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-03-11T21:11:19.323 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-03-11T21:11:19.323 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.323 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.15081.0:45729 9.e 9:7553524a:::benchmark_data_smithi023_106206_object45728:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e262)", 2024-03-11T21:11:19.323 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T21:03:27.496359+0000", 2024-03-11T21:11:19.323 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 471.834996735, 2024-03-11T21:11:19.323 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.42950042399999999, 2024-03-11T21:11:19.323 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:11:19.323 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:11:19.323 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-03-11T21:11:19.323 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.15081", 2024-03-11T21:11:19.323 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.23:0/2841194346", 2024-03-11T21:11:19.324 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 45729 2024-03-11T21:11:19.324 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.324 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:11:19.324 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.324 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:11:19.324 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:27.496359+0000", 2024-03-11T21:11:19.324 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:11:19.324 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.324 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.324 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:11:19.324 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:27.496359+0000", 2024-03-11T21:11:19.324 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:11:19.324 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.324 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.324 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:11:19.325 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:27.496359+0000", 2024-03-11T21:11:19.325 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4600000000000004e-07 2024-03-11T21:11:19.325 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.325 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.325 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:11:19.325 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:27.496405+0000", 2024-03-11T21:11:19.325 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5654e-05 2024-03-11T21:11:19.325 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.325 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.325 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:11:19.325 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:27.496411+0000", 2024-03-11T21:11:19.325 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1009999999999997e-06 2024-03-11T21:11:19.325 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.325 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.326 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:11:19.326 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:27.496415+0000", 2024-03-11T21:11:19.326 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0330000000000002e-06 2024-03-11T21:11:19.326 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.326 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.326 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:11:19.326 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:27.497867+0000", 2024-03-11T21:11:19.326 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0014521359999999999 2024-03-11T21:11:19.326 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.326 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.326 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:11:19.326 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:27.497988+0000", 2024-03-11T21:11:19.326 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000120538 2024-03-11T21:11:19.326 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.327 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.327 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1,4", 2024-03-11T21:11:19.327 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:27.498149+0000", 2024-03-11T21:11:19.327 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016169700000000001 2024-03-11T21:11:19.327 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.327 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.327 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-03-11T21:11:19.327 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:27.499256+0000", 2024-03-11T21:11:19.327 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001106087 2024-03-11T21:11:19.327 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.327 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.327 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:11:19.328 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:27.509464+0000", 2024-03-11T21:11:19.328 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010208283 2024-03-11T21:11:19.328 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.328 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.328 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:11:19.328 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:27.925630+0000", 2024-03-11T21:11:19.328 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.41616583299999999 2024-03-11T21:11:19.328 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.328 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.328 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:11:19.328 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:27.925728+0000", 2024-03-11T21:11:19.328 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8717999999999999e-05 2024-03-11T21:11:19.328 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.329 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.329 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:11:19.329 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:27.925859+0000", 2024-03-11T21:11:19.329 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000130698 2024-03-11T21:11:19.329 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:11:19.329 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:11:19.329 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:11:19.329 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.329 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.329 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.15081.0:100107 9.e 9:7103d6a1:::benchmark_data_smithi023_106206_object100106:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e276)", 2024-03-11T21:11:19.329 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T21:04:20.252914+0000", 2024-03-11T21:11:19.330 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 419.07844112599997, 2024-03-11T21:11:19.330 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.42876025899999998, 2024-03-11T21:11:19.330 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:11:19.330 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:11:19.330 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-03-11T21:11:19.330 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.15081", 2024-03-11T21:11:19.330 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.23:0/2841194346", 2024-03-11T21:11:19.330 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 100107 2024-03-11T21:11:19.330 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.330 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:11:19.330 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.330 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:11:19.330 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.252914+0000", 2024-03-11T21:11:19.330 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:11:19.331 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.331 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.331 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:11:19.331 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.252914+0000", 2024-03-11T21:11:19.331 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:11:19.331 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.331 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.331 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:11:19.331 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.252915+0000", 2024-03-11T21:11:19.331 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2190000000000001e-06 2024-03-11T21:11:19.331 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.332 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.332 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:11:19.332 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.252988+0000", 2024-03-11T21:11:19.332 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2510999999999997e-05 2024-03-11T21:11:19.332 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.332 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.332 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:11:19.332 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.253003+0000", 2024-03-11T21:11:19.332 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5152e-05 2024-03-11T21:11:19.332 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.332 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.333 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:11:19.333 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.253011+0000", 2024-03-11T21:11:19.333 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.8970000000000008e-06 2024-03-11T21:11:19.333 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.333 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.333 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:11:19.333 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.253074+0000", 2024-03-11T21:11:19.333 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3347000000000002e-05 2024-03-11T21:11:19.333 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.333 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.333 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:11:19.334 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.253211+0000", 2024-03-11T21:11:19.334 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013664600000000001 2024-03-11T21:11:19.334 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.334 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.334 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1,4", 2024-03-11T21:11:19.334 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.253370+0000", 2024-03-11T21:11:19.334 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015909999999999999 2024-03-11T21:11:19.334 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.334 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.334 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-03-11T21:11:19.334 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.256333+0000", 2024-03-11T21:11:19.334 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002962559 2024-03-11T21:11:19.334 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.334 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.337 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:11:19.337 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.264983+0000", 2024-03-11T21:11:19.337 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0086507130000000008 2024-03-11T21:11:19.337 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.338 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.338 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:11:19.338 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.681571+0000", 2024-03-11T21:11:19.338 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.416587239 2024-03-11T21:11:19.338 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.338 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.338 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:11:19.338 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.681631+0000", 2024-03-11T21:11:19.338 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0851999999999999e-05 2024-03-11T21:11:19.338 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.338 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.338 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:11:19.338 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:04:20.681674+0000", 2024-03-11T21:11:19.338 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3024e-05 2024-03-11T21:11:19.338 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:11:19.339 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:11:19.339 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:11:19.339 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.339 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.339 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.15081.0:142575 9.e 9:7bb92a1d:::benchmark_data_smithi023_106206_object31429:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e284)", 2024-03-11T21:11:19.339 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T21:05:05.350793+0000", 2024-03-11T21:11:19.339 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 373.98056218300002, 2024-03-11T21:11:19.339 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.43533081899999998, 2024-03-11T21:11:19.339 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:11:19.339 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:11:19.339 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-03-11T21:11:19.339 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.15081", 2024-03-11T21:11:19.339 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.23:0/2841194346", 2024-03-11T21:11:19.339 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 142575 2024-03-11T21:11:19.339 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.340 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:11:19.340 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.340 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:11:19.340 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:05.350793+0000", 2024-03-11T21:11:19.340 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:11:19.340 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.340 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.340 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:11:19.340 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:05.350793+0000", 2024-03-11T21:11:19.340 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:11:19.340 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.340 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.340 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:11:19.341 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:05.350794+0000", 2024-03-11T21:11:19.341 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2839999999999999e-06 2024-03-11T21:11:19.341 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.341 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.341 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:11:19.341 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:05.350800+0000", 2024-03-11T21:11:19.341 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1410000000000004e-06 2024-03-11T21:11:19.341 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.341 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.341 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:11:19.341 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:05.350811+0000", 2024-03-11T21:11:19.341 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1378e-05 2024-03-11T21:11:19.341 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.341 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.342 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:11:19.342 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:05.350816+0000", 2024-03-11T21:11:19.342 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2950000000000001e-06 2024-03-11T21:11:19.342 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.342 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.342 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:11:19.342 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:05.350867+0000", 2024-03-11T21:11:19.342 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1014999999999999e-05 2024-03-11T21:11:19.342 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.343 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.343 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:11:19.343 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:05.350986+0000", 2024-03-11T21:11:19.343 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000118992 2024-03-11T21:11:19.343 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.343 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.343 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1,4", 2024-03-11T21:11:19.343 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:05.351070+0000", 2024-03-11T21:11:19.343 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4194000000000001e-05 2024-03-11T21:11:19.343 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.343 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.343 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-03-11T21:11:19.343 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:05.353885+0000", 2024-03-11T21:11:19.343 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0028146849999999999 2024-03-11T21:11:19.344 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.344 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.344 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:11:19.344 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:05.358539+0000", 2024-03-11T21:11:19.344 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0046539210000000001 2024-03-11T21:11:19.344 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.344 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.344 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:11:19.344 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:05.786022+0000", 2024-03-11T21:11:19.344 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.42748251599999998 2024-03-11T21:11:19.344 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.344 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.344 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:11:19.344 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:05.786074+0000", 2024-03-11T21:11:19.345 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2646999999999999e-05 2024-03-11T21:11:19.345 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.345 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.345 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:11:19.345 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:05.786124+0000", 2024-03-11T21:11:19.345 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9750999999999999e-05 2024-03-11T21:11:19.345 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:11:19.345 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:11:19.345 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:11:19.345 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.345 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.345 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.15081.0:142578 9.e 9:7d9686ca:::benchmark_data_smithi023_106206_object31432:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e284)", 2024-03-11T21:11:19.345 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T21:05:05.354172+0000", 2024-03-11T21:11:19.345 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 373.977183056, 2024-03-11T21:11:19.345 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.43209350899999999, 2024-03-11T21:11:19.346 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:11:19.346 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:11:19.346 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-03-11T21:11:19.346 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.15081", 2024-03-11T21:11:19.346 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.23:0/2841194346", 2024-03-11T21:11:19.346 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 142578 2024-03-11T21:11:19.346 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.346 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:11:19.346 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.346 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:11:19.347 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:05.354172+0000", 2024-03-11T21:11:19.347 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:11:19.347 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.347 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.347 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:11:19.347 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:05.354172+0000", 2024-03-11T21:11:19.347 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:11:19.347 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.347 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.347 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:11:19.347 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:05.354173+0000", 2024-03-11T21:11:19.347 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3e-07 2024-03-11T21:11:19.347 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.347 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.347 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:11:19.348 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:05.354177+0000", 2024-03-11T21:11:19.348 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0640000000000004e-06 2024-03-11T21:11:19.348 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.348 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.348 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:11:19.348 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:05.354185+0000", 2024-03-11T21:11:19.348 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5700000000000004e-06 2024-03-11T21:11:19.348 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.348 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.348 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:11:19.348 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:05.354189+0000", 2024-03-11T21:11:19.348 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9779999999999999e-06 2024-03-11T21:11:19.348 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.348 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.348 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:11:19.348 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:05.354241+0000", 2024-03-11T21:11:19.349 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2862000000000003e-05 2024-03-11T21:11:19.349 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.349 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.349 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:11:19.349 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:05.354336+0000", 2024-03-11T21:11:19.349 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4877000000000007e-05 2024-03-11T21:11:19.349 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.349 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.349 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1,4", 2024-03-11T21:11:19.349 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:05.354452+0000", 2024-03-11T21:11:19.349 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011523699999999999 2024-03-11T21:11:19.349 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.349 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.349 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-03-11T21:11:19.350 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:05.356836+0000", 2024-03-11T21:11:19.350 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0023840860000000001 2024-03-11T21:11:19.350 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.350 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.350 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:11:19.350 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:05.369905+0000", 2024-03-11T21:11:19.350 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.013068999 2024-03-11T21:11:19.350 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.350 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.350 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:11:19.350 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:05.786170+0000", 2024-03-11T21:11:19.351 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.41626552900000002 2024-03-11T21:11:19.351 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.351 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.351 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:11:19.351 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:05.786223+0000", 2024-03-11T21:11:19.351 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2420000000000003e-05 2024-03-11T21:11:19.351 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.351 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.351 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:11:19.351 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:05.786266+0000", 2024-03-11T21:11:19.351 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3157e-05 2024-03-11T21:11:19.351 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:11:19.351 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:11:19.351 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:11:19.351 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.352 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.352 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.15081.0:143697 9.e 9:76cbab00:::benchmark_data_smithi023_106206_object32551:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e284)", 2024-03-11T21:11:19.352 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T21:05:07.092590+0000", 2024-03-11T21:11:19.352 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 372.238765371, 2024-03-11T21:11:19.352 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.42721914300000002, 2024-03-11T21:11:19.352 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:11:19.352 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:11:19.352 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-03-11T21:11:19.352 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.15081", 2024-03-11T21:11:19.352 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.23:0/2841194346", 2024-03-11T21:11:19.352 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 143697 2024-03-11T21:11:19.352 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.352 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:11:19.353 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.353 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:11:19.353 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:07.092590+0000", 2024-03-11T21:11:19.353 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:11:19.353 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.353 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.353 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:11:19.353 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:07.092590+0000", 2024-03-11T21:11:19.353 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:11:19.353 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.353 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.353 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:11:19.354 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:07.092592+0000", 2024-03-11T21:11:19.354 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.748e-06 2024-03-11T21:11:19.354 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.354 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.354 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:11:19.354 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:07.092599+0000", 2024-03-11T21:11:19.354 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9720000000000003e-06 2024-03-11T21:11:19.354 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.354 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.354 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:11:19.354 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:07.092611+0000", 2024-03-11T21:11:19.354 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2279000000000001e-05 2024-03-11T21:11:19.354 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.355 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.355 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:11:19.355 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:07.092620+0000", 2024-03-11T21:11:19.355 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.6160000000000002e-06 2024-03-11T21:11:19.355 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.355 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.355 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:11:19.355 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:07.092684+0000", 2024-03-11T21:11:19.355 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4286000000000001e-05 2024-03-11T21:11:19.355 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.355 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.355 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:11:19.355 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:07.092791+0000", 2024-03-11T21:11:19.355 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000107433 2024-03-11T21:11:19.355 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.356 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.356 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1,4", 2024-03-11T21:11:19.356 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:07.092896+0000", 2024-03-11T21:11:19.356 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010506900000000001 2024-03-11T21:11:19.356 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.356 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.356 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-03-11T21:11:19.356 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:07.095825+0000", 2024-03-11T21:11:19.356 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002928239 2024-03-11T21:11:19.356 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.356 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.356 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:11:19.356 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:07.101611+0000", 2024-03-11T21:11:19.356 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0057866669999999997 2024-03-11T21:11:19.356 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.357 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.357 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:11:19.357 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:07.519680+0000", 2024-03-11T21:11:19.357 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.418069101 2024-03-11T21:11:19.357 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.357 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.357 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:11:19.357 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:07.519749+0000", 2024-03-11T21:11:19.357 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8872000000000001e-05 2024-03-11T21:11:19.357 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.357 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.357 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:11:19.357 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:07.519809+0000", 2024-03-11T21:11:19.357 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9861000000000002e-05 2024-03-11T21:11:19.357 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:11:19.358 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:11:19.358 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:11:19.358 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.358 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.358 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.15291.0:151207 10.9 e348/306 10:9401ef5c:::benchmark_data_smithi023_124944_object42195:head v 348'9409, mlcod=348'9404)", 2024-03-11T21:11:19.358 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T21:09:54.901249+0000", 2024-03-11T21:11:19.358 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 84.430106581000004, 2024-03-11T21:11:19.358 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.62746928800000001, 2024-03-11T21:11:19.358 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:11:19.358 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:11:19.358 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:11:19.358 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.358 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:11:19.358 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.901249+0000", 2024-03-11T21:11:19.359 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:11:19.359 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.359 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.359 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:11:19.359 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.901249+0000", 2024-03-11T21:11:19.359 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:11:19.359 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.359 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.359 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:11:19.359 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.901249+0000", 2024-03-11T21:11:19.359 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4400000000000001e-07 2024-03-11T21:11:19.359 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.359 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.359 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:11:19.359 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.901252+0000", 2024-03-11T21:11:19.360 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2969999999999999e-06 2024-03-11T21:11:19.360 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.360 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.360 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:11:19.360 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.901257+0000", 2024-03-11T21:11:19.360 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8509999999999998e-06 2024-03-11T21:11:19.360 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.360 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.360 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:11:19.360 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.901261+0000", 2024-03-11T21:11:19.361 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8539999999999999e-06 2024-03-11T21:11:19.361 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.361 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.361 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:11:19.361 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.905788+0000", 2024-03-11T21:11:19.361 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.004526581 2024-03-11T21:11:19.361 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.361 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.361 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:11:19.361 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.905805+0000", 2024-03-11T21:11:19.361 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7011e-05 2024-03-11T21:11:19.361 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.361 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.361 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:11:19.362 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:55.528655+0000", 2024-03-11T21:11:19.362 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.62285065399999995 2024-03-11T21:11:19.362 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.362 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.362 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:11:19.362 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:55.528718+0000", 2024-03-11T21:11:19.362 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2796e-05 2024-03-11T21:11:19.362 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:11:19.362 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:11:19.362 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:11:19.362 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.362 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.362 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.15291.0:151216 10.9 e348/306 10:9dcd14dc:::benchmark_data_smithi023_124944_object42204:head v 348'9410, mlcod=348'9405)", 2024-03-11T21:11:19.363 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T21:09:54.904972+0000", 2024-03-11T21:11:19.363 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 84.426383549999997, 2024-03-11T21:11:19.363 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.62446994499999997, 2024-03-11T21:11:19.363 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:11:19.363 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:11:19.363 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:11:19.363 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.363 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:11:19.363 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.904972+0000", 2024-03-11T21:11:19.363 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:11:19.363 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.363 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.363 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:11:19.363 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.904972+0000", 2024-03-11T21:11:19.363 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:11:19.364 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.364 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.364 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:11:19.364 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.904972+0000", 2024-03-11T21:11:19.364 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.99e-07 2024-03-11T21:11:19.364 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.364 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.364 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:11:19.364 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.904978+0000", 2024-03-11T21:11:19.365 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.631e-06 2024-03-11T21:11:19.365 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.365 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.365 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:11:19.365 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.904988+0000", 2024-03-11T21:11:19.365 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6160000000000008e-06 2024-03-11T21:11:19.365 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.365 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.365 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:11:19.365 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.904994+0000", 2024-03-11T21:11:19.365 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2779999999999999e-06 2024-03-11T21:11:19.365 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.366 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.366 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:11:19.366 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.913349+0000", 2024-03-11T21:11:19.366 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0083554149999999997 2024-03-11T21:11:19.366 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.366 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.366 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:11:19.366 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.913376+0000", 2024-03-11T21:11:19.366 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6174999999999999e-05 2024-03-11T21:11:19.366 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.366 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.366 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:11:19.367 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:55.529317+0000", 2024-03-11T21:11:19.367 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.61594110899999999 2024-03-11T21:11:19.367 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.367 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.367 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:11:19.367 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:55.529442+0000", 2024-03-11T21:11:19.367 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000125022 2024-03-11T21:11:19.367 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:11:19.367 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:11:19.367 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:11:19.367 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.367 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.367 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.15291.0:151218 10.1 10:8560beb6:::benchmark_data_smithi023_124944_object42206:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e348)", 2024-03-11T21:11:19.367 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T21:09:54.905457+0000", 2024-03-11T21:11:19.368 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 84.425897989000006, 2024-03-11T21:11:19.368 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.62374859599999999, 2024-03-11T21:11:19.368 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:11:19.368 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:11:19.368 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-03-11T21:11:19.368 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.15291", 2024-03-11T21:11:19.368 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.23:0/729656300", 2024-03-11T21:11:19.368 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 151218 2024-03-11T21:11:19.368 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.368 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:11:19.368 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.368 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:11:19.368 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.905457+0000", 2024-03-11T21:11:19.368 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:11:19.369 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.369 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.369 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:11:19.369 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.905457+0000", 2024-03-11T21:11:19.369 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:11:19.369 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.369 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.369 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:11:19.369 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.905459+0000", 2024-03-11T21:11:19.369 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6780000000000001e-06 2024-03-11T21:11:19.369 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.369 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.369 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:11:19.369 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.905464+0000", 2024-03-11T21:11:19.369 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4229999999999996e-06 2024-03-11T21:11:19.370 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.370 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.370 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:11:19.370 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.905476+0000", 2024-03-11T21:11:19.370 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1501999999999999e-05 2024-03-11T21:11:19.370 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.370 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.370 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:11:19.370 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.905483+0000", 2024-03-11T21:11:19.370 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.289e-06 2024-03-11T21:11:19.370 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.370 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.370 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:11:19.370 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.908731+0000", 2024-03-11T21:11:19.371 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0032476530000000001 2024-03-11T21:11:19.371 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.371 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.371 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:11:19.371 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.908834+0000", 2024-03-11T21:11:19.371 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010323399999999999 2024-03-11T21:11:19.371 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.371 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.371 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0,4", 2024-03-11T21:11:19.371 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.908940+0000", 2024-03-11T21:11:19.371 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010635 2024-03-11T21:11:19.371 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.371 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.371 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:11:19.371 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:55.519545+0000", 2024-03-11T21:11:19.372 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.61060473599999998 2024-03-11T21:11:19.372 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.372 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.372 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:11:19.372 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:55.519704+0000", 2024-03-11T21:11:19.372 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015914799999999999 2024-03-11T21:11:19.372 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.372 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.372 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-03-11T21:11:19.372 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:55.529084+0000", 2024-03-11T21:11:19.372 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0093796260000000003 2024-03-11T21:11:19.372 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.372 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.372 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:11:19.372 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:55.529153+0000", 2024-03-11T21:11:19.373 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8571999999999993e-05 2024-03-11T21:11:19.373 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.373 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.373 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:11:19.373 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:55.529206+0000", 2024-03-11T21:11:19.373 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3384999999999998e-05 2024-03-11T21:11:19.373 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:11:19.373 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:11:19.373 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:11:19.373 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.373 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.373 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.15291.0:151219 10.1 10:812dde62:::benchmark_data_smithi023_124944_object42207:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e348)", 2024-03-11T21:11:19.373 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T21:09:54.906454+0000", 2024-03-11T21:11:19.373 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 84.424901810999998, 2024-03-11T21:11:19.373 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.62285371700000003, 2024-03-11T21:11:19.374 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:11:19.374 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:11:19.374 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-03-11T21:11:19.374 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.15291", 2024-03-11T21:11:19.374 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.23:0/729656300", 2024-03-11T21:11:19.374 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 151219 2024-03-11T21:11:19.374 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.374 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:11:19.374 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.374 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:11:19.374 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.906454+0000", 2024-03-11T21:11:19.374 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:11:19.374 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.374 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.375 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:11:19.375 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.906454+0000", 2024-03-11T21:11:19.375 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:11:19.375 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.375 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.375 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:11:19.375 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.906454+0000", 2024-03-11T21:11:19.375 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.8899999999999998e-07 2024-03-11T21:11:19.375 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.376 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.376 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:11:19.376 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.906461+0000", 2024-03-11T21:11:19.376 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1840000000000004e-06 2024-03-11T21:11:19.376 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.376 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.376 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:11:19.376 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.906470+0000", 2024-03-11T21:11:19.376 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2420000000000006e-06 2024-03-11T21:11:19.376 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.376 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.376 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:11:19.376 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.906475+0000", 2024-03-11T21:11:19.376 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5400000000000003e-06 2024-03-11T21:11:19.377 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.377 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.377 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:11:19.377 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.911564+0000", 2024-03-11T21:11:19.377 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0050889769999999997 2024-03-11T21:11:19.377 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.377 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.377 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:11:19.377 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.911680+0000", 2024-03-11T21:11:19.377 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000115442 2024-03-11T21:11:19.377 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.377 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.378 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0,4", 2024-03-11T21:11:19.378 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.911791+0000", 2024-03-11T21:11:19.378 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000111524 2024-03-11T21:11:19.378 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.378 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.378 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:11:19.378 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:55.519634+0000", 2024-03-11T21:11:19.378 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.60784307500000001 2024-03-11T21:11:19.378 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.378 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.378 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:11:19.378 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:55.529065+0000", 2024-03-11T21:11:19.379 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0094311389999999998 2024-03-11T21:11:19.379 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.379 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.379 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-03-11T21:11:19.379 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:55.529218+0000", 2024-03-11T21:11:19.379 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015231899999999999 2024-03-11T21:11:19.379 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.379 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.379 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:11:19.379 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:55.529267+0000", 2024-03-11T21:11:19.379 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9112e-05 2024-03-11T21:11:19.379 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.379 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.379 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:11:19.380 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:55.529307+0000", 2024-03-11T21:11:19.380 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0373999999999997e-05 2024-03-11T21:11:19.380 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:11:19.380 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:11:19.380 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:11:19.380 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.380 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.380 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.15291.0:151228 10.c 10:3ef0b427:::benchmark_data_smithi023_124944_object42216:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e348)", 2024-03-11T21:11:19.380 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T21:09:54.913459+0000", 2024-03-11T21:11:19.380 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 84.417896127999995, 2024-03-11T21:11:19.380 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.61638670200000001, 2024-03-11T21:11:19.381 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:11:19.381 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:11:19.381 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-03-11T21:11:19.381 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.15291", 2024-03-11T21:11:19.381 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.23:0/729656300", 2024-03-11T21:11:19.381 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 151228 2024-03-11T21:11:19.381 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.381 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:11:19.381 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.381 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:11:19.381 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.913459+0000", 2024-03-11T21:11:19.381 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:11:19.381 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.382 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.382 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:11:19.382 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.913459+0000", 2024-03-11T21:11:19.382 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:11:19.382 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.382 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.382 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:11:19.382 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.913461+0000", 2024-03-11T21:11:19.382 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9259999999999999e-06 2024-03-11T21:11:19.382 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.382 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.383 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:11:19.383 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.913467+0000", 2024-03-11T21:11:19.383 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8850000000000004e-06 2024-03-11T21:11:19.383 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.383 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.383 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:11:19.383 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:55.514068+0000", 2024-03-11T21:11:19.383 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.60060141499999997 2024-03-11T21:11:19.383 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.383 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.383 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:11:19.383 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:55.514112+0000", 2024-03-11T21:11:19.384 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3680000000000002e-05 2024-03-11T21:11:19.384 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.384 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.384 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:11:19.384 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:55.515531+0000", 2024-03-11T21:11:19.384 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0014188969999999999 2024-03-11T21:11:19.384 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.384 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.384 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:11:19.384 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:55.515741+0000", 2024-03-11T21:11:19.384 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00021031199999999999 2024-03-11T21:11:19.384 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.385 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.385 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0,4", 2024-03-11T21:11:19.385 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:55.515930+0000", 2024-03-11T21:11:19.385 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000188629 2024-03-11T21:11:19.385 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.385 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.385 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-03-11T21:11:19.385 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:55.523837+0000", 2024-03-11T21:11:19.385 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0079067800000000004 2024-03-11T21:11:19.385 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.385 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.385 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:11:19.385 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:55.524494+0000", 2024-03-11T21:11:19.385 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00065699599999999997 2024-03-11T21:11:19.386 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.386 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.386 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:11:19.386 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:55.529609+0000", 2024-03-11T21:11:19.386 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0051154479999999999 2024-03-11T21:11:19.386 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.386 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.386 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:11:19.386 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:55.529745+0000", 2024-03-11T21:11:19.386 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013553799999999999 2024-03-11T21:11:19.386 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.386 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.386 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:11:19.386 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:55.529846+0000", 2024-03-11T21:11:19.387 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000101196 2024-03-11T21:11:19.387 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:11:19.387 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:11:19.387 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:11:19.387 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.387 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.387 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.15291.0:177067 10.9 e352/306 10:9c159633:::benchmark_data_smithi023_124944_object68055:head v 352'11048, mlcod=352'11045)", 2024-03-11T21:11:19.387 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T21:10:23.156757+0000", 2024-03-11T21:11:19.387 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 56.174598666999998, 2024-03-11T21:11:19.387 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.470378927, 2024-03-11T21:11:19.387 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:11:19.387 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:11:19.387 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:11:19.387 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.387 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:11:19.388 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:10:23.156757+0000", 2024-03-11T21:11:19.388 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:11:19.388 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.388 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.388 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:11:19.388 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:10:23.156757+0000", 2024-03-11T21:11:19.388 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:11:19.388 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.388 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.388 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:11:19.388 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:10:23.156757+0000", 2024-03-11T21:11:19.388 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.7100000000000001e-07 2024-03-11T21:11:19.388 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.388 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.388 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:11:19.389 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:10:23.156763+0000", 2024-03-11T21:11:19.389 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8289999999999996e-06 2024-03-11T21:11:19.389 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.389 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.389 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:11:19.389 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:10:23.156773+0000", 2024-03-11T21:11:19.389 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4420000000000004e-06 2024-03-11T21:11:19.389 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.389 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.389 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:11:19.389 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:10:23.156779+0000", 2024-03-11T21:11:19.389 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3679999999999998e-06 2024-03-11T21:11:19.389 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.389 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.390 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:11:19.390 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:10:23.156819+0000", 2024-03-11T21:11:19.390 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9768000000000003e-05 2024-03-11T21:11:19.390 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.390 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.390 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:11:19.390 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:10:23.156832+0000", 2024-03-11T21:11:19.390 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2958e-05 2024-03-11T21:11:19.390 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.390 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.390 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:11:19.390 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:10:23.627008+0000", 2024-03-11T21:11:19.390 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.47017576 2024-03-11T21:11:19.390 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.390 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.391 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:11:19.391 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:10:23.627136+0000", 2024-03-11T21:11:19.391 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012803099999999999 2024-03-11T21:11:19.391 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:11:19.391 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:11:19.391 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:11:19.391 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.391 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.391 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.15291.0:177069 10.9 e352/306 10:9f127a96:::benchmark_data_smithi023_124944_object68057:head v 352'11049, mlcod=352'11046)", 2024-03-11T21:11:19.391 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T21:10:23.160470+0000", 2024-03-11T21:11:19.391 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 56.170885202000001, 2024-03-11T21:11:19.391 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.46676260600000002, 2024-03-11T21:11:19.391 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:11:19.391 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:11:19.391 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:11:19.392 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.392 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:11:19.392 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:10:23.160470+0000", 2024-03-11T21:11:19.392 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:11:19.392 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.392 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.392 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:11:19.392 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:10:23.160470+0000", 2024-03-11T21:11:19.392 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:11:19.392 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.392 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.392 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:11:19.392 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:10:23.160471+0000", 2024-03-11T21:11:19.392 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5199999999999995e-07 2024-03-11T21:11:19.392 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.393 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.393 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:11:19.393 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:10:23.160479+0000", 2024-03-11T21:11:19.393 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6879999999999999e-06 2024-03-11T21:11:19.393 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.393 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.393 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:11:19.393 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:10:23.160490+0000", 2024-03-11T21:11:19.393 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1601e-05 2024-03-11T21:11:19.393 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.393 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.393 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:11:19.393 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:10:23.160498+0000", 2024-03-11T21:11:19.393 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6299999999999998e-06 2024-03-11T21:11:19.393 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.394 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.394 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:11:19.394 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:10:23.162886+0000", 2024-03-11T21:11:19.394 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0023885439999999998 2024-03-11T21:11:19.394 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.394 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.394 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:11:19.394 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:10:23.162909+0000", 2024-03-11T21:11:19.394 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2819e-05 2024-03-11T21:11:19.394 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.394 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.394 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:11:19.394 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:10:23.627145+0000", 2024-03-11T21:11:19.394 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.46423582499999999 2024-03-11T21:11:19.395 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.395 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.395 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:11:19.395 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:10:23.627233+0000", 2024-03-11T21:11:19.395 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.7646999999999996e-05 2024-03-11T21:11:19.395 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:11:19.395 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:11:19.395 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:11:19.395 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.395 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.395 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.15291.0:177076 10.9 e352/306 10:94e83af3:::benchmark_data_smithi023_124944_object68064:head v 352'11050, mlcod=352'11046)", 2024-03-11T21:11:19.395 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T21:10:23.164714+0000", 2024-03-11T21:11:19.395 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 56.166641102, 2024-03-11T21:11:19.395 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.462617379, 2024-03-11T21:11:19.395 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:11:19.396 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:11:19.396 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:11:19.396 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.396 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:11:19.396 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:10:23.164714+0000", 2024-03-11T21:11:19.396 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:11:19.396 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.396 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.396 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:11:19.396 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:10:23.164714+0000", 2024-03-11T21:11:19.396 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:11:19.396 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.396 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.396 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:11:19.396 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:10:23.164715+0000", 2024-03-11T21:11:19.397 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7899999999999998e-07 2024-03-11T21:11:19.397 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.397 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.397 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:11:19.397 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:10:23.164721+0000", 2024-03-11T21:11:19.397 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0499999999999997e-06 2024-03-11T21:11:19.397 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.397 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.397 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:11:19.397 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:10:23.164730+0000", 2024-03-11T21:11:19.397 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8829999999999995e-06 2024-03-11T21:11:19.397 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.397 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.397 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:11:19.398 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:10:23.164736+0000", 2024-03-11T21:11:19.398 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.976e-06 2024-03-11T21:11:19.398 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.398 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.398 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:11:19.398 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:10:23.165985+0000", 2024-03-11T21:11:19.398 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001249111 2024-03-11T21:11:19.398 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.398 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.398 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:11:19.398 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:10:23.166003+0000", 2024-03-11T21:11:19.398 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7731e-05 2024-03-11T21:11:19.398 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.398 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.398 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:11:19.399 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:10:23.627241+0000", 2024-03-11T21:11:19.399 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.46123812199999997 2024-03-11T21:11:19.399 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.399 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.399 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:11:19.399 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:10:23.627332+0000", 2024-03-11T21:11:19.399 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0927000000000005e-05 2024-03-11T21:11:19.399 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:11:19.399 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:11:19.399 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:11:19.399 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.399 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.399 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.15291.0:177082 10.c 10:34fe5a2c:::benchmark_data_smithi023_124944_object68070:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e352)", 2024-03-11T21:11:19.399 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T21:10:23.164866+0000", 2024-03-11T21:11:19.400 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 56.166489824000003, 2024-03-11T21:11:19.400 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.46279958199999999, 2024-03-11T21:11:19.400 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:11:19.400 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:11:19.400 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-03-11T21:11:19.400 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.15291", 2024-03-11T21:11:19.400 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.23:0/729656300", 2024-03-11T21:11:19.400 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 177082 2024-03-11T21:11:19.400 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.400 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:11:19.400 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.400 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:11:19.400 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:10:23.164866+0000", 2024-03-11T21:11:19.400 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:11:19.401 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.401 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.401 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:11:19.401 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:10:23.164866+0000", 2024-03-11T21:11:19.401 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:11:19.401 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.401 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.401 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:11:19.401 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:10:23.164867+0000", 2024-03-11T21:11:19.401 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5710000000000001e-06 2024-03-11T21:11:19.401 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.401 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.401 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:11:19.401 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:10:23.164871+0000", 2024-03-11T21:11:19.401 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3150000000000002e-06 2024-03-11T21:11:19.402 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.402 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.402 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:11:19.402 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:10:23.164882+0000", 2024-03-11T21:11:19.402 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1015e-05 2024-03-11T21:11:19.402 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.402 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.402 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:11:19.402 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:10:23.164889+0000", 2024-03-11T21:11:19.402 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0469999999999996e-06 2024-03-11T21:11:19.402 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.402 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.402 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:11:19.403 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:10:23.164952+0000", 2024-03-11T21:11:19.403 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2998999999999998e-05 2024-03-11T21:11:19.403 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.403 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.403 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:11:19.403 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:10:23.165078+0000", 2024-03-11T21:11:19.403 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012513699999999999 2024-03-11T21:11:19.403 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.403 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.403 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0,4", 2024-03-11T21:11:19.403 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:10:23.165184+0000", 2024-03-11T21:11:19.403 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010674400000000001 2024-03-11T21:11:19.403 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.403 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.403 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:11:19.404 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:10:23.168408+0000", 2024-03-11T21:11:19.404 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0032236109999999999 2024-03-11T21:11:19.404 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.404 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.404 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-03-11T21:11:19.404 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:10:23.627025+0000", 2024-03-11T21:11:19.404 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.45861719000000001 2024-03-11T21:11:19.404 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.404 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.404 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:11:19.404 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:10:23.627417+0000", 2024-03-11T21:11:19.404 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00039231699999999999 2024-03-11T21:11:19.404 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.404 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.404 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:11:19.405 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:10:23.627548+0000", 2024-03-11T21:11:19.405 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013096599999999999 2024-03-11T21:11:19.405 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.405 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.405 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:11:19.405 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:10:23.627665+0000", 2024-03-11T21:11:19.405 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011667000000000001 2024-03-11T21:11:19.405 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:11:19.405 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:11:19.405 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:11:19.405 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.405 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.405 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop_reply(client.15291.0:177082 10.c e352/306 ondisk, result = 0)", 2024-03-11T21:11:19.405 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T21:10:23.168227+0000", 2024-03-11T21:11:19.405 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 56.163128751000002, 2024-03-11T21:11:19.406 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.45870508100000001, 2024-03-11T21:11:19.406 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:11:19.406 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-03-11T21:11:19.406 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:11:19.406 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.406 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:11:19.406 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:10:23.168227+0000", 2024-03-11T21:11:19.406 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:11:19.406 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.406 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.406 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:11:19.406 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:10:23.168227+0000", 2024-03-11T21:11:19.406 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:11:19.406 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.407 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.407 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:11:19.407 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:10:23.168227+0000", 2024-03-11T21:11:19.407 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5600000000000005e-07 2024-03-11T21:11:19.407 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.407 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.407 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:11:19.407 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:10:23.168232+0000", 2024-03-11T21:11:19.407 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9540000000000003e-06 2024-03-11T21:11:19.407 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.407 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.407 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:11:19.407 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:10:23.168243+0000", 2024-03-11T21:11:19.407 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0747999999999999e-05 2024-03-11T21:11:19.407 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.407 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.408 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:11:19.408 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:10:23.168250+0000", 2024-03-11T21:11:19.408 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5089999999999996e-06 2024-03-11T21:11:19.408 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.408 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.408 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:11:19.408 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:10:23.168396+0000", 2024-03-11T21:11:19.408 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014547299999999999 2024-03-11T21:11:19.408 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.408 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.408 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:11:19.408 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:10:23.168403+0000", 2024-03-11T21:11:19.408 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8750000000000002e-06 2024-03-11T21:11:19.408 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.408 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.409 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:11:19.409 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:10:23.626932+0000", 2024-03-11T21:11:19.409 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.45852886599999998 2024-03-11T21:11:19.409 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:11:19.409 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:11:19.409 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:11:19.409 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.409 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.409 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop_reply(client.15291.0:177082 10.c e352/306 ondisk, result = 0)", 2024-03-11T21:11:19.409 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T21:10:23.169982+0000", 2024-03-11T21:11:19.409 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 56.161373210000001, 2024-03-11T21:11:19.409 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.45770138199999999, 2024-03-11T21:11:19.409 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:11:19.409 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-03-11T21:11:19.409 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:11:19.410 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.410 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:11:19.410 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:10:23.169982+0000", 2024-03-11T21:11:19.410 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:11:19.410 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.410 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.410 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:11:19.410 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:10:23.169982+0000", 2024-03-11T21:11:19.410 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:11:19.410 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.410 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.410 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:11:19.410 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:10:23.169983+0000", 2024-03-11T21:11:19.410 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.0999999999999997e-07 2024-03-11T21:11:19.411 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.411 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.411 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:11:19.411 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:10:23.169988+0000", 2024-03-11T21:11:19.411 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0649999999999998e-06 2024-03-11T21:11:19.411 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.411 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.411 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:11:19.411 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:10:23.627024+0000", 2024-03-11T21:11:19.411 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.45703628000000002 2024-03-11T21:11:19.411 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.411 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.411 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:11:19.411 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:10:23.627052+0000", 2024-03-11T21:11:19.412 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.783e-05 2024-03-11T21:11:19.412 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.412 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.412 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:11:19.412 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:10:23.627391+0000", 2024-03-11T21:11:19.412 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00033927799999999999 2024-03-11T21:11:19.412 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.412 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.412 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:11:19.412 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:10:23.627406+0000", 2024-03-11T21:11:19.412 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4283e-05 2024-03-11T21:11:19.412 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.412 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.412 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:11:19.412 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:10:23.627684+0000", 2024-03-11T21:11:19.413 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000277836 2024-03-11T21:11:19.413 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:11:19.413 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:11:19.413 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:11:19.413 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.413 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.413 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.15291.0:177085 10.1 10:8107ca44:::benchmark_data_smithi023_124944_object68073:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e352)", 2024-03-11T21:11:19.413 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T21:10:23.170048+0000", 2024-03-11T21:11:19.413 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 56.161307399999998, 2024-03-11T21:11:19.413 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.47424861400000001, 2024-03-11T21:11:19.413 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:11:19.413 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:11:19.413 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-03-11T21:11:19.413 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.15291", 2024-03-11T21:11:19.413 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.23:0/729656300", 2024-03-11T21:11:19.414 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 177085 2024-03-11T21:11:19.414 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.414 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:11:19.414 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.414 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:11:19.414 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:10:23.170048+0000", 2024-03-11T21:11:19.414 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:11:19.414 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.414 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.414 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:11:19.414 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:10:23.170048+0000", 2024-03-11T21:11:19.414 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:11:19.414 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.414 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.415 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:11:19.415 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:10:23.170050+0000", 2024-03-11T21:11:19.415 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.579e-06 2024-03-11T21:11:19.415 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.415 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.415 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:11:19.415 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:10:23.170055+0000", 2024-03-11T21:11:19.415 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8300000000000001e-06 2024-03-11T21:11:19.415 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.415 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.415 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:11:19.415 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:10:23.626918+0000", 2024-03-11T21:11:19.415 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.45686248099999999 2024-03-11T21:11:19.415 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.415 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.416 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:11:19.416 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:10:23.626960+0000", 2024-03-11T21:11:19.416 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2281999999999999e-05 2024-03-11T21:11:19.416 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.416 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.416 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:11:19.416 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:10:23.627189+0000", 2024-03-11T21:11:19.416 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00022865099999999999 2024-03-11T21:11:19.416 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.416 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.416 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:11:19.416 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:10:23.627420+0000", 2024-03-11T21:11:19.416 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00023091799999999999 2024-03-11T21:11:19.416 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.417 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.417 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0,4", 2024-03-11T21:11:19.417 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:10:23.627556+0000", 2024-03-11T21:11:19.417 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013605900000000001 2024-03-11T21:11:19.417 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.417 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.417 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:11:19.417 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:10:23.636042+0000", 2024-03-11T21:11:19.417 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0084859029999999995 2024-03-11T21:11:19.417 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.417 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.417 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-03-11T21:11:19.417 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:10:23.639978+0000", 2024-03-11T21:11:19.417 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0039366879999999998 2024-03-11T21:11:19.418 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.418 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.418 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:11:19.418 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:10:23.644178+0000", 2024-03-11T21:11:19.418 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.004200067 2024-03-11T21:11:19.418 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.418 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.418 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:11:19.418 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:10:23.644245+0000", 2024-03-11T21:11:19.418 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6540999999999999e-05 2024-03-11T21:11:19.418 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.418 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.418 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:11:19.418 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:10:23.644297+0000", 2024-03-11T21:11:19.418 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1615e-05 2024-03-11T21:11:19.419 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:11:19.419 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:11:19.419 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:11:19.419 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.419 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.419 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.15291.0:213169 10.1 10:858a16f6:::benchmark_data_smithi023_124944_object104157:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e362)", 2024-03-11T21:11:19.419 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T21:11:02.913004+0000", 2024-03-11T21:11:19.419 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 16.418350869000001, 2024-03-11T21:11:19.419 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.52518250600000005, 2024-03-11T21:11:19.419 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:11:19.419 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:11:19.419 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-03-11T21:11:19.419 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.15291", 2024-03-11T21:11:19.419 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.23:0/729656300", 2024-03-11T21:11:19.420 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 213169 2024-03-11T21:11:19.420 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.420 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:11:19.420 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.420 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:11:19.420 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:11:02.913004+0000", 2024-03-11T21:11:19.420 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:11:19.420 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.420 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.420 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:11:19.420 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:11:02.913004+0000", 2024-03-11T21:11:19.420 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:11:19.420 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.420 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.420 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:11:19.421 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:11:02.913006+0000", 2024-03-11T21:11:19.421 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.282e-06 2024-03-11T21:11:19.421 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.421 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.421 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:11:19.421 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:11:02.913010+0000", 2024-03-11T21:11:19.421 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8260000000000003e-06 2024-03-11T21:11:19.421 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.421 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.421 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:11:19.421 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:11:02.913021+0000", 2024-03-11T21:11:19.421 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1209999999999999e-05 2024-03-11T21:11:19.421 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.421 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.421 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:11:19.422 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:11:02.913029+0000", 2024-03-11T21:11:19.422 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.7910000000000005e-06 2024-03-11T21:11:19.422 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.422 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.422 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:11:19.422 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:11:02.925258+0000", 2024-03-11T21:11:19.422 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012229741000000001 2024-03-11T21:11:19.422 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.422 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.422 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:11:19.422 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:11:02.925389+0000", 2024-03-11T21:11:19.422 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013090700000000001 2024-03-11T21:11:19.422 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.422 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.423 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0,4", 2024-03-11T21:11:19.423 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:11:02.925509+0000", 2024-03-11T21:11:19.423 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012012500000000001 2024-03-11T21:11:19.423 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.423 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.423 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-03-11T21:11:19.423 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:11:02.927449+0000", 2024-03-11T21:11:19.423 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001939547 2024-03-11T21:11:19.423 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.423 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.423 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:11:19.423 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:11:02.934375+0000", 2024-03-11T21:11:19.423 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00692597 2024-03-11T21:11:19.423 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.424 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.424 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:11:19.424 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:11:03.438001+0000", 2024-03-11T21:11:19.424 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.50362628499999995 2024-03-11T21:11:19.424 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.424 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.424 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:11:19.424 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:11:03.438093+0000", 2024-03-11T21:11:19.424 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2317000000000004e-05 2024-03-11T21:11:19.424 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.424 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.424 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:11:19.424 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:11:03.438187+0000", 2024-03-11T21:11:19.424 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.3504999999999993e-05 2024-03-11T21:11:19.424 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:11:19.425 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:11:19.425 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:11:19.425 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.425 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.425 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.15291.0:213182 10.1 10:85a997fa:::benchmark_data_smithi023_124944_object104170:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e362)", 2024-03-11T21:11:19.425 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T21:11:02.928333+0000", 2024-03-11T21:11:19.425 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 16.403022559, 2024-03-11T21:11:19.425 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.51015336300000003, 2024-03-11T21:11:19.425 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:11:19.425 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:11:19.425 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-03-11T21:11:19.425 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.15291", 2024-03-11T21:11:19.425 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.23:0/729656300", 2024-03-11T21:11:19.425 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 213182 2024-03-11T21:11:19.426 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.426 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:11:19.426 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.426 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:11:19.426 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:11:02.928333+0000", 2024-03-11T21:11:19.426 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:11:19.426 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.426 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.426 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:11:19.426 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:11:02.928333+0000", 2024-03-11T21:11:19.426 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:11:19.426 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.426 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.426 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:11:19.426 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:11:02.928333+0000", 2024-03-11T21:11:19.427 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9599999999999999e-07 2024-03-11T21:11:19.427 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.427 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.427 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:11:19.427 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:11:02.928337+0000", 2024-03-11T21:11:19.427 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3919999999999999e-06 2024-03-11T21:11:19.427 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.427 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.427 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:11:19.427 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:11:02.928345+0000", 2024-03-11T21:11:19.427 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.2190000000000004e-06 2024-03-11T21:11:19.427 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.427 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.428 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:11:19.428 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:11:02.928352+0000", 2024-03-11T21:11:19.428 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5860000000000001e-06 2024-03-11T21:11:19.428 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.428 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.428 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:11:19.428 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:11:02.928394+0000", 2024-03-11T21:11:19.428 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2284999999999999e-05 2024-03-11T21:11:19.428 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.428 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.428 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:11:19.428 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:11:02.928500+0000", 2024-03-11T21:11:19.428 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010647899999999999 2024-03-11T21:11:19.428 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.428 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.429 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0,4", 2024-03-11T21:11:19.429 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:11:02.928628+0000", 2024-03-11T21:11:19.429 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000127415 2024-03-11T21:11:19.429 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.429 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.429 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-03-11T21:11:19.429 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:11:02.934441+0000", 2024-03-11T21:11:19.429 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0058136689999999996 2024-03-11T21:11:19.429 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.429 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.429 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:11:19.429 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:11:02.935849+0000", 2024-03-11T21:11:19.429 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001407829 2024-03-11T21:11:19.429 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.430 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.430 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:11:19.430 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:11:03.438316+0000", 2024-03-11T21:11:19.430 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.50246687800000001 2024-03-11T21:11:19.430 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.430 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.430 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:11:19.430 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:11:03.438411+0000", 2024-03-11T21:11:19.430 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5239000000000001e-05 2024-03-11T21:11:19.430 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.430 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.430 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:11:19.430 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:11:03.438486+0000", 2024-03-11T21:11:19.431 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4776000000000004e-05 2024-03-11T21:11:19.431 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:11:19.431 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:11:19.431 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:11:19.431 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.431 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.431 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.15291.0:213183 10.1 10:82dbc83d:::benchmark_data_smithi023_124944_object104171:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e362)", 2024-03-11T21:11:19.431 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T21:11:02.928366+0000", 2024-03-11T21:11:19.431 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 16.402989722000001, 2024-03-11T21:11:19.431 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.51034267200000005, 2024-03-11T21:11:19.431 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:11:19.431 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:11:19.431 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-03-11T21:11:19.431 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.15291", 2024-03-11T21:11:19.431 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.23:0/729656300", 2024-03-11T21:11:19.432 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 213183 2024-03-11T21:11:19.432 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.432 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:11:19.432 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.432 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:11:19.432 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:11:02.928366+0000", 2024-03-11T21:11:19.432 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:11:19.432 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.432 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.432 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:11:19.432 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:11:02.928366+0000", 2024-03-11T21:11:19.432 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:11:19.432 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.432 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.432 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:11:19.433 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:11:02.928366+0000", 2024-03-11T21:11:19.433 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0999999999999999e-07 2024-03-11T21:11:19.433 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.433 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.433 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:11:19.433 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:11:02.928369+0000", 2024-03-11T21:11:19.433 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0649999999999999e-06 2024-03-11T21:11:19.433 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.433 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.433 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:11:19.433 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:11:02.928376+0000", 2024-03-11T21:11:19.433 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4030000000000004e-06 2024-03-11T21:11:19.433 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.433 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.433 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:11:19.433 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:11:02.928381+0000", 2024-03-11T21:11:19.434 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2340000000000002e-06 2024-03-11T21:11:19.434 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.434 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.434 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:11:19.434 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:11:02.931386+0000", 2024-03-11T21:11:19.434 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030050379999999998 2024-03-11T21:11:19.434 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.434 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.434 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:11:19.434 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:11:02.931489+0000", 2024-03-11T21:11:19.434 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010298699999999999 2024-03-11T21:11:19.434 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.434 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.434 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0,4", 2024-03-11T21:11:19.435 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:11:02.931597+0000", 2024-03-11T21:11:19.435 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010853600000000001 2024-03-11T21:11:19.435 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.435 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.435 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-03-11T21:11:19.435 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:11:02.934444+0000", 2024-03-11T21:11:19.435 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0028470840000000002 2024-03-11T21:11:19.435 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.435 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.435 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:11:19.435 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:11:02.939140+0000", 2024-03-11T21:11:19.435 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0046960120000000003 2024-03-11T21:11:19.435 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.435 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.435 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:11:19.436 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:11:03.438571+0000", 2024-03-11T21:11:19.436 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.49943084599999998 2024-03-11T21:11:19.436 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.436 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.436 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:11:19.436 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:11:03.438647+0000", 2024-03-11T21:11:19.436 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6151999999999999e-05 2024-03-11T21:11:19.436 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:19.436 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:19.436 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:11:19.436 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:11:03.438708+0000", 2024-03-11T21:11:19.436 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0905e-05 2024-03-11T21:11:19.436 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:11:19.436 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:11:19.436 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:11:19.437 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:11:19.437 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:11:19.437 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-03-11T21:11:19.437 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:19.466 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:19.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:19.468 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:11:19.569 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:11:19.598 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:11:19.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:19.600 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:11:19.701 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:11:19.723 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:11:19.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:19.726 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:11:19.826 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:19.838 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:19 smithi023 bash[21372]: cluster 2024-03-11T21:11:18.622788+0000 osd.1 (osd.1) 247 : cluster 0 2.a scrub starts 2024-03-11T21:11:19.838 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:19 smithi023 bash[21372]: cluster 2024-03-11T21:11:18.625285+0000 osd.1 (osd.1) 248 : cluster 0 2.a scrub ok 2024-03-11T21:11:19.838 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:19 smithi023 bash[21372]: cluster 2024-03-11T21:11:18.953517+0000 osd.0 (osd.0) 339 : cluster 0 2.13 scrub starts 2024-03-11T21:11:19.838 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:19 smithi023 bash[21372]: cluster 2024-03-11T21:11:18.956559+0000 osd.0 (osd.0) 340 : cluster 0 2.13 scrub ok 2024-03-11T21:11:19.838 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:19 smithi023 bash[21372]: cluster 2024-03-11T21:11:19.325486+0000 osd.5 (osd.5) 437 : cluster 0 6.19 deep-scrub starts 2024-03-11T21:11:19.838 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:19 smithi023 bash[21372]: cluster 2024-03-11T21:11:19.401760+0000 osd.5 (osd.5) 438 : cluster 0 6.19 deep-scrub ok 2024-03-11T21:11:19.838 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:19 smithi023 bash[29146]: cluster 2024-03-11T21:11:18.622788+0000 osd.1 (osd.1) 247 : cluster 0 2.a scrub starts 2024-03-11T21:11:19.839 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:19 smithi023 bash[29146]: cluster 2024-03-11T21:11:18.625285+0000 osd.1 (osd.1) 248 : cluster 0 2.a scrub ok 2024-03-11T21:11:19.839 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:19 smithi023 bash[29146]: cluster 2024-03-11T21:11:18.953517+0000 osd.0 (osd.0) 339 : cluster 0 2.13 scrub starts 2024-03-11T21:11:19.839 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:19 smithi023 bash[29146]: cluster 2024-03-11T21:11:18.956559+0000 osd.0 (osd.0) 340 : cluster 0 2.13 scrub ok 2024-03-11T21:11:19.839 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:19 smithi023 bash[29146]: cluster 2024-03-11T21:11:19.325486+0000 osd.5 (osd.5) 437 : cluster 0 6.19 deep-scrub starts 2024-03-11T21:11:19.839 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:19 smithi023 bash[29146]: cluster 2024-03-11T21:11:19.401760+0000 osd.5 (osd.5) 438 : cluster 0 6.19 deep-scrub ok 2024-03-11T21:11:19.853 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:19.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:19.856 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:11:19.957 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:19.979 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:19.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:19.981 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:11:20.082 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:11:20.093 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:19 smithi146 bash[23809]: cluster 2024-03-11T21:11:18.622788+0000 osd.1 (osd.1) 247 : cluster 0 2.a scrub starts 2024-03-11T21:11:20.093 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:19 smithi146 bash[23809]: cluster 2024-03-11T21:11:18.625285+0000 osd.1 (osd.1) 248 : cluster 0 2.a scrub ok 2024-03-11T21:11:20.093 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:19 smithi146 bash[23809]: cluster 2024-03-11T21:11:18.953517+0000 osd.0 (osd.0) 339 : cluster 0 2.13 scrub starts 2024-03-11T21:11:20.094 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:19 smithi146 bash[23809]: cluster 2024-03-11T21:11:18.956559+0000 osd.0 (osd.0) 340 : cluster 0 2.13 scrub ok 2024-03-11T21:11:20.094 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:19 smithi146 bash[23809]: cluster 2024-03-11T21:11:19.325486+0000 osd.5 (osd.5) 437 : cluster 0 6.19 deep-scrub starts 2024-03-11T21:11:20.094 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:19 smithi146 bash[23809]: cluster 2024-03-11T21:11:19.401760+0000 osd.5 (osd.5) 438 : cluster 0 6.19 deep-scrub ok 2024-03-11T21:11:20.110 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:11:20.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:20.112 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:11:20.177 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-11T21:11:20.213 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:11:20.239 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:11:20.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:20.241 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:11:20.342 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:11:20.364 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:11:20.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:20.366 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:11:20.467 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:20.492 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:20.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:20.494 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:11:20.595 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:20.621 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:20.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:20.623 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:11:20.724 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:20.752 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:20.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:20.754 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:11:20.810 INFO:teuthology.orchestra.run.smithi023.stderr:set pool 1 pg_num to 11 2024-03-11T21:11:20.816 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:20 smithi146 bash[23809]: cluster 2024-03-11T21:11:19.643958+0000 osd.1 (osd.1) 249 : cluster 0 2.4 scrub starts 2024-03-11T21:11:20.856 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:11:20.857 INFO:teuthology.orchestra.run.smithi023.stdout:osd.0: {} 2024-03-11T21:11:20.876 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:11:20.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:20.878 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:11:20.880 INFO:teuthology.orchestra.run.smithi023.stdout:osd.1: {} 2024-03-11T21:11:20.901 INFO:teuthology.orchestra.run.smithi023.stdout:osd.2: {} 2024-03-11T21:11:20.943 INFO:teuthology.orchestra.run.smithi023.stdout:osd.3: {} 2024-03-11T21:11:20.979 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:11:20.989 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:20 smithi023 bash[29146]: cluster 2024-03-11T21:11:19.643958+0000 osd.1 (osd.1) 249 : cluster 0 2.4 scrub starts 2024-03-11T21:11:20.989 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:20 smithi023 bash[29146]: cluster 2024-03-11T21:11:19.645819+0000 osd.1 (osd.1) 250 : cluster 0 2.4 scrub ok 2024-03-11T21:11:20.989 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:20 smithi023 bash[29146]: cluster 2024-03-11T21:11:19.943518+0000 osd.0 (osd.0) 341 : cluster 0 10.8 deep-scrub starts 2024-03-11T21:11:20.989 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:20 smithi023 bash[29146]: cluster 2024-03-11T21:11:19.962670+0000 osd.0 (osd.0) 342 : cluster 0 10.8 deep-scrub ok 2024-03-11T21:11:20.989 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:20 smithi023 bash[29146]: cluster 2024-03-11T21:11:20.157804+0000 osd.3 (osd.3) 424 : cluster 0 4.c scrub starts 2024-03-11T21:11:20.989 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:20 smithi023 bash[29146]: cluster 2024-03-11T21:11:20.159630+0000 osd.3 (osd.3) 425 : cluster 0 4.c scrub ok 2024-03-11T21:11:20.989 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:20 smithi023 bash[29146]: cluster 2024-03-11T21:11:20.344670+0000 mgr.y (mgr.24380) 761 : cluster 0 pgmap v859: 148 pgs: 148 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-11T21:11:20.989 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:20 smithi023 bash[29146]: audit 2024-03-11T21:11:20.510863+0000 mon.c (mon.2) 88 : audit 1 from='client.? 172.21.15.23:0/824432861' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-03-11T21:11:20.990 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:20 smithi023 bash[29146]: audit 2024-03-11T21:11:20.511233+0000 mon.a (mon.0) 1754 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-03-11T21:11:20.990 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:20 smithi023 bash[21372]: cluster 2024-03-11T21:11:19.643958+0000 osd.1 (osd.1) 249 : cluster 0 2.4 scrub starts 2024-03-11T21:11:20.990 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:20 smithi023 bash[21372]: cluster 2024-03-11T21:11:19.645819+0000 osd.1 (osd.1) 250 : cluster 0 2.4 scrub ok 2024-03-11T21:11:20.990 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:20 smithi023 bash[21372]: cluster 2024-03-11T21:11:19.943518+0000 osd.0 (osd.0) 341 : cluster 0 10.8 deep-scrub starts 2024-03-11T21:11:20.990 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:20 smithi023 bash[21372]: cluster 2024-03-11T21:11:19.962670+0000 osd.0 (osd.0) 342 : cluster 0 10.8 deep-scrub ok 2024-03-11T21:11:20.990 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:20 smithi023 bash[21372]: cluster 2024-03-11T21:11:20.157804+0000 osd.3 (osd.3) 424 : cluster 0 4.c scrub starts 2024-03-11T21:11:20.990 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:20 smithi023 bash[21372]: cluster 2024-03-11T21:11:20.159630+0000 osd.3 (osd.3) 425 : cluster 0 4.c scrub ok 2024-03-11T21:11:20.990 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:20 smithi023 bash[21372]: cluster 2024-03-11T21:11:20.344670+0000 mgr.y (mgr.24380) 761 : cluster 0 pgmap v859: 148 pgs: 148 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-11T21:11:20.990 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:20 smithi023 bash[21372]: audit 2024-03-11T21:11:20.510863+0000 mon.c (mon.2) 88 : audit 1 from='client.? 172.21.15.23:0/824432861' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-03-11T21:11:20.990 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:20 smithi023 bash[21372]: audit 2024-03-11T21:11:20.511233+0000 mon.a (mon.0) 1754 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-03-11T21:11:20.991 INFO:teuthology.orchestra.run.smithi023.stdout:osd.4: {} 2024-03-11T21:11:21.006 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:11:21.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:21.008 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:11:21.014 INFO:teuthology.orchestra.run.smithi023.stdout:osd.5: {} 2024-03-11T21:11:21.056 INFO:teuthology.orchestra.run.smithi023.stdout:osd.6: {} 2024-03-11T21:11:21.079 INFO:teuthology.orchestra.run.smithi023.stdout:osd.7: {} 2024-03-11T21:11:21.080 INFO:teuthology.orchestra.run.smithi023.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:11:21.080 INFO:teuthology.orchestra.run.smithi023.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:11:21.081 INFO:teuthology.orchestra.run.smithi023.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:11:21.081 INFO:teuthology.orchestra.run.smithi023.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:11:21.081 INFO:teuthology.orchestra.run.smithi023.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:11:21.081 INFO:teuthology.orchestra.run.smithi023.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:11:21.081 INFO:teuthology.orchestra.run.smithi023.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:11:21.081 INFO:teuthology.orchestra.run.smithi023.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:11:21.109 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:21.128 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:21.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:21.130 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:11:21.231 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:11:21.252 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:11:21.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:21.254 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:11:21.257 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:20 smithi146 bash[23809]: cluster 2024-03-11T21:11:19.645819+0000 osd.1 (osd.1) 250 : cluster 0 2.4 scrub ok 2024-03-11T21:11:21.257 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:20 smithi146 bash[23809]: cluster 2024-03-11T21:11:19.943518+0000 osd.0 (osd.0) 341 : cluster 0 10.8 deep-scrub starts 2024-03-11T21:11:21.257 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:20 smithi146 bash[23809]: cluster 2024-03-11T21:11:19.962670+0000 osd.0 (osd.0) 342 : cluster 0 10.8 deep-scrub ok 2024-03-11T21:11:21.258 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:20 smithi146 bash[23809]: cluster 2024-03-11T21:11:20.157804+0000 osd.3 (osd.3) 424 : cluster 0 4.c scrub starts 2024-03-11T21:11:21.258 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:20 smithi146 bash[23809]: cluster 2024-03-11T21:11:20.159630+0000 osd.3 (osd.3) 425 : cluster 0 4.c scrub ok 2024-03-11T21:11:21.258 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:20 smithi146 bash[23809]: cluster 2024-03-11T21:11:20.344670+0000 mgr.y (mgr.24380) 761 : cluster 0 pgmap v859: 148 pgs: 148 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-11T21:11:21.258 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:20 smithi146 bash[23809]: audit 2024-03-11T21:11:20.510863+0000 mon.c (mon.2) 88 : audit 1 from='client.? 172.21.15.23:0/824432861' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-03-11T21:11:21.258 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:20 smithi146 bash[23809]: audit 2024-03-11T21:11:20.511233+0000 mon.a (mon.0) 1754 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-03-11T21:11:21.354 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:21.385 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:21.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:21.387 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:11:21.488 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:11:21.518 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:11:21.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:21.520 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:11:21.621 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:21.650 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:21.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:21.653 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:11:21.754 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:11:21.785 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:11:21.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:21.787 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:11:21.841 INFO:teuthology.orchestra.run.smithi023.stderr:noscrub is set 2024-03-11T21:11:21.888 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:11:21.909 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:11:21.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:21.911 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:11:22.012 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:11:22.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:21 smithi023 bash[21372]: cluster 2024-03-11T21:11:20.614221+0000 osd.1 (osd.1) 251 : cluster 0 6.1d scrub starts 2024-03-11T21:11:22.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:21 smithi023 bash[21372]: cluster 2024-03-11T21:11:20.727929+0000 osd.1 (osd.1) 252 : cluster 0 6.1d scrub ok 2024-03-11T21:11:22.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:21 smithi023 bash[21372]: audit 2024-03-11T21:11:20.807224+0000 mon.a (mon.0) 1755 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-03-11T21:11:22.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:21 smithi023 bash[21372]: cluster 2024-03-11T21:11:20.815853+0000 mon.a (mon.0) 1756 : cluster 0 osdmap e365: 8 total, 8 up, 8 in 2024-03-11T21:11:22.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:21 smithi023 bash[21372]: audit 2024-03-11T21:11:20.845904+0000 mon.a (mon.0) 1757 : audit 0 from='client.? 172.21.15.23:0/987696378' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:11:22.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:21 smithi023 bash[21372]: cluster 2024-03-11T21:11:20.912304+0000 osd.7 (osd.7) 485 : cluster 0 6.5 deep-scrub starts 2024-03-11T21:11:22.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:21 smithi023 bash[21372]: cluster 2024-03-11T21:11:20.933176+0000 osd.0 (osd.0) 343 : cluster 0 6.9 scrub starts 2024-03-11T21:11:22.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:21 smithi023 bash[21372]: cluster 2024-03-11T21:11:20.989372+0000 osd.7 (osd.7) 486 : cluster 0 6.5 deep-scrub ok 2024-03-11T21:11:22.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:21 smithi023 bash[21372]: cluster 2024-03-11T21:11:21.007489+0000 osd.0 (osd.0) 344 : cluster 0 6.9 scrub ok 2024-03-11T21:11:22.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:21 smithi023 bash[21372]: audit 2024-03-11T21:11:21.077403+0000 mon.a (mon.0) 1758 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:11:22.023 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:21 smithi023 bash[21372]: audit 2024-03-11T21:11:21.143929+0000 mon.a (mon.0) 1759 : audit 1 from='client.? 172.21.15.23:0/3176250792' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:11:22.023 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:21 smithi023 bash[29146]: cluster 2024-03-11T21:11:20.614221+0000 osd.1 (osd.1) 251 : cluster 0 6.1d scrub starts 2024-03-11T21:11:22.023 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:21 smithi023 bash[29146]: cluster 2024-03-11T21:11:20.727929+0000 osd.1 (osd.1) 252 : cluster 0 6.1d scrub ok 2024-03-11T21:11:22.023 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:21 smithi023 bash[29146]: audit 2024-03-11T21:11:20.807224+0000 mon.a (mon.0) 1755 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-03-11T21:11:22.023 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:21 smithi023 bash[29146]: cluster 2024-03-11T21:11:20.815853+0000 mon.a (mon.0) 1756 : cluster 0 osdmap e365: 8 total, 8 up, 8 in 2024-03-11T21:11:22.023 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:21 smithi023 bash[29146]: audit 2024-03-11T21:11:20.845904+0000 mon.a (mon.0) 1757 : audit 0 from='client.? 172.21.15.23:0/987696378' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:11:22.023 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:21 smithi023 bash[29146]: cluster 2024-03-11T21:11:20.912304+0000 osd.7 (osd.7) 485 : cluster 0 6.5 deep-scrub starts 2024-03-11T21:11:22.023 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:21 smithi023 bash[29146]: cluster 2024-03-11T21:11:20.933176+0000 osd.0 (osd.0) 343 : cluster 0 6.9 scrub starts 2024-03-11T21:11:22.023 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:21 smithi023 bash[29146]: cluster 2024-03-11T21:11:20.989372+0000 osd.7 (osd.7) 486 : cluster 0 6.5 deep-scrub ok 2024-03-11T21:11:22.023 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:21 smithi023 bash[29146]: cluster 2024-03-11T21:11:21.007489+0000 osd.0 (osd.0) 344 : cluster 0 6.9 scrub ok 2024-03-11T21:11:22.023 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:21 smithi023 bash[29146]: audit 2024-03-11T21:11:21.077403+0000 mon.a (mon.0) 1758 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:11:22.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:21 smithi023 bash[29146]: audit 2024-03-11T21:11:21.143929+0000 mon.a (mon.0) 1759 : audit 1 from='client.? 172.21.15.23:0/3176250792' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:11:22.035 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:11:22.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:22.037 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:11:22.138 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:11:22.151 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:21 smithi146 bash[23809]: cluster 2024-03-11T21:11:20.614221+0000 osd.1 (osd.1) 251 : cluster 0 6.1d scrub starts 2024-03-11T21:11:22.151 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:21 smithi146 bash[23809]: cluster 2024-03-11T21:11:20.727929+0000 osd.1 (osd.1) 252 : cluster 0 6.1d scrub ok 2024-03-11T21:11:22.151 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:21 smithi146 bash[23809]: audit 2024-03-11T21:11:20.807224+0000 mon.a (mon.0) 1755 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-03-11T21:11:22.151 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:21 smithi146 bash[23809]: cluster 2024-03-11T21:11:20.815853+0000 mon.a (mon.0) 1756 : cluster 0 osdmap e365: 8 total, 8 up, 8 in 2024-03-11T21:11:22.152 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:21 smithi146 bash[23809]: audit 2024-03-11T21:11:20.845904+0000 mon.a (mon.0) 1757 : audit 0 from='client.? 172.21.15.23:0/987696378' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:11:22.152 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:21 smithi146 bash[23809]: cluster 2024-03-11T21:11:20.912304+0000 osd.7 (osd.7) 485 : cluster 0 6.5 deep-scrub starts 2024-03-11T21:11:22.152 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:21 smithi146 bash[23809]: cluster 2024-03-11T21:11:20.933176+0000 osd.0 (osd.0) 343 : cluster 0 6.9 scrub starts 2024-03-11T21:11:22.152 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:21 smithi146 bash[23809]: cluster 2024-03-11T21:11:20.989372+0000 osd.7 (osd.7) 486 : cluster 0 6.5 deep-scrub ok 2024-03-11T21:11:22.152 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:21 smithi146 bash[23809]: cluster 2024-03-11T21:11:21.007489+0000 osd.0 (osd.0) 344 : cluster 0 6.9 scrub ok 2024-03-11T21:11:22.152 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:21 smithi146 bash[23809]: audit 2024-03-11T21:11:21.077403+0000 mon.a (mon.0) 1758 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:11:22.152 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:21 smithi146 bash[23809]: audit 2024-03-11T21:11:21.143929+0000 mon.a (mon.0) 1759 : audit 1 from='client.? 172.21.15.23:0/3176250792' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:11:22.167 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:11:22.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:22.169 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:11:22.269 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:11:22.298 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:11:22.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:22.300 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:11:22.364 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_3 2024-03-11T21:11:22.365 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd pool rm unique_pool_3 unique_pool_3 --yes-i-really-really-mean-it 2024-03-11T21:11:22.401 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:11:22.437 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:11:22.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:22.439 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:11:22.540 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:22.568 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:22.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:22.570 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:11:22.671 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:22.698 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:22.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:22.701 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:11:22.802 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:22.827 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:22.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:22.830 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:11:22.930 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:22.951 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:22.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:22.954 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:11:23.027 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:22 smithi023 bash[21372]: cluster 2024-03-11T21:11:21.663129+0000 osd.1 (osd.1) 253 : cluster 0 4.12 scrub starts 2024-03-11T21:11:23.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:22 smithi023 bash[21372]: cluster 2024-03-11T21:11:21.664983+0000 osd.1 (osd.1) 254 : cluster 0 4.12 scrub ok 2024-03-11T21:11:23.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:22 smithi023 bash[21372]: cluster 2024-03-11T21:11:21.807299+0000 osd.6 (osd.6) 360 : cluster 0 10.f scrub starts 2024-03-11T21:11:23.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:22 smithi023 bash[21372]: cluster 2024-03-11T21:11:21.817947+0000 osd.6 (osd.6) 361 : cluster 0 10.f scrub ok 2024-03-11T21:11:23.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:22 smithi023 bash[21372]: cluster 2024-03-11T21:11:21.825967+0000 mon.a (mon.0) 1760 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:11:23.028 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:22 smithi023 bash[29146]: cluster 2024-03-11T21:11:21.663129+0000 osd.1 (osd.1) 253 : cluster 0 4.12 scrub starts 2024-03-11T21:11:23.028 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:22 smithi023 bash[29146]: cluster 2024-03-11T21:11:21.664983+0000 osd.1 (osd.1) 254 : cluster 0 4.12 scrub ok 2024-03-11T21:11:23.028 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:22 smithi023 bash[29146]: cluster 2024-03-11T21:11:21.807299+0000 osd.6 (osd.6) 360 : cluster 0 10.f scrub starts 2024-03-11T21:11:23.028 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:22 smithi023 bash[29146]: cluster 2024-03-11T21:11:21.817947+0000 osd.6 (osd.6) 361 : cluster 0 10.f scrub ok 2024-03-11T21:11:23.028 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:22 smithi023 bash[29146]: cluster 2024-03-11T21:11:21.825967+0000 mon.a (mon.0) 1760 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:11:23.055 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:23.078 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:23.079 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:22 smithi146 bash[23809]: cluster 2024-03-11T21:11:21.663129+0000 osd.1 (osd.1) 253 : cluster 0 4.12 scrub starts 2024-03-11T21:11:23.079 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:22 smithi146 bash[23809]: cluster 2024-03-11T21:11:21.664983+0000 osd.1 (osd.1) 254 : cluster 0 4.12 scrub ok 2024-03-11T21:11:23.079 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:22 smithi146 bash[23809]: cluster 2024-03-11T21:11:21.807299+0000 osd.6 (osd.6) 360 : cluster 0 10.f scrub starts 2024-03-11T21:11:23.079 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:22 smithi146 bash[23809]: cluster 2024-03-11T21:11:21.817947+0000 osd.6 (osd.6) 361 : cluster 0 10.f scrub ok 2024-03-11T21:11:23.079 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:22 smithi146 bash[23809]: cluster 2024-03-11T21:11:21.825967+0000 mon.a (mon.0) 1760 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:11:23.079 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:22 smithi146 bash[23809]: audit 2024-03-11T21:11:21.848038+0000 mon.a (mon.0) 1761 : audit 1 from='client.? 172.21.15.23:0/3176250792' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:11:23.079 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:22 smithi146 bash[23809]: cluster 2024-03-11T21:11:21.862178+0000 mon.a (mon.0) 1762 : cluster 0 osdmap e366: 8 total, 8 up, 8 in 2024-03-11T21:11:23.080 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:22 smithi146 bash[23809]: cluster 2024-03-11T21:11:21.959960+0000 osd.0 (osd.0) 345 : cluster 0 10.5 scrub starts 2024-03-11T21:11:23.080 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:22 smithi146 bash[23809]: cluster 2024-03-11T21:11:22.345509+0000 mgr.y (mgr.24380) 762 : cluster 0 pgmap v862: 148 pgs: 148 active+clean; 582 KiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:11:23.080 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:22 smithi146 bash[23809]: audit 2024-03-11T21:11:22.347180+0000 mon.a (mon.0) 1763 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-03-11T21:11:23.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:23.081 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:11:23.181 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:11:23.210 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:11:23.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:23.212 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:11:23.278 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:23 smithi023 bash[21372]: audit 2024-03-11T21:11:21.848038+0000 mon.a (mon.0) 1761 : audit 1 from='client.? 172.21.15.23:0/3176250792' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:11:23.278 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:23 smithi023 bash[21372]: cluster 2024-03-11T21:11:21.862178+0000 mon.a (mon.0) 1762 : cluster 0 osdmap e366: 8 total, 8 up, 8 in 2024-03-11T21:11:23.278 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:23 smithi023 bash[21372]: cluster 2024-03-11T21:11:21.959960+0000 osd.0 (osd.0) 345 : cluster 0 10.5 scrub starts 2024-03-11T21:11:23.278 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:23 smithi023 bash[21372]: cluster 2024-03-11T21:11:22.345509+0000 mgr.y (mgr.24380) 762 : cluster 0 pgmap v862: 148 pgs: 148 active+clean; 582 KiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:11:23.279 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:23 smithi023 bash[21372]: audit 2024-03-11T21:11:22.347180+0000 mon.a (mon.0) 1763 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-03-11T21:11:23.279 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:23 smithi023 bash[29146]: audit 2024-03-11T21:11:21.848038+0000 mon.a (mon.0) 1761 : audit 1 from='client.? 172.21.15.23:0/3176250792' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:11:23.279 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:23 smithi023 bash[29146]: cluster 2024-03-11T21:11:21.862178+0000 mon.a (mon.0) 1762 : cluster 0 osdmap e366: 8 total, 8 up, 8 in 2024-03-11T21:11:23.279 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:23 smithi023 bash[29146]: cluster 2024-03-11T21:11:21.959960+0000 osd.0 (osd.0) 345 : cluster 0 10.5 scrub starts 2024-03-11T21:11:23.279 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:23 smithi023 bash[29146]: cluster 2024-03-11T21:11:22.345509+0000 mgr.y (mgr.24380) 762 : cluster 0 pgmap v862: 148 pgs: 148 active+clean; 582 KiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:11:23.279 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:23 smithi023 bash[29146]: audit 2024-03-11T21:11:22.347180+0000 mon.a (mon.0) 1763 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-03-11T21:11:23.313 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:23.340 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:23.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:23.342 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:11:23.443 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:11:23.470 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:11:23.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:23.473 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:11:23.574 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:23.603 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:23.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:23.605 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:11:23.706 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:11:23.733 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:11:23.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:23.736 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:11:23.838 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:23.866 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:23.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:23.868 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:11:23.969 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:11:23.994 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:11:23.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:23.996 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:11:24.097 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:24.110 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:24 smithi146 bash[23809]: audit 2024-03-11T21:11:23.059804+0000 mon.a (mon.0) 1764 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-03-11T21:11:24.110 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:24 smithi146 bash[23809]: cluster 2024-03-11T21:11:23.070629+0000 mon.a (mon.0) 1765 : cluster 0 osdmap e367: 8 total, 8 up, 8 in 2024-03-11T21:11:24.125 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:24.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:24.127 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:11:24.228 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:11:24.254 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:11:24.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:24.256 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:11:24.357 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:24.384 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:24.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:24.387 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:11:24.487 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:24.512 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:24.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:24.515 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:11:24.527 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:24 smithi023 bash[21372]: audit 2024-03-11T21:11:23.059804+0000 mon.a (mon.0) 1764 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-03-11T21:11:24.527 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:24 smithi023 bash[21372]: cluster 2024-03-11T21:11:23.070629+0000 mon.a (mon.0) 1765 : cluster 0 osdmap e367: 8 total, 8 up, 8 in 2024-03-11T21:11:24.527 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:24 smithi023 bash[29146]: audit 2024-03-11T21:11:23.059804+0000 mon.a (mon.0) 1764 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-03-11T21:11:24.528 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:24 smithi023 bash[29146]: cluster 2024-03-11T21:11:23.070629+0000 mon.a (mon.0) 1765 : cluster 0 osdmap e367: 8 total, 8 up, 8 in 2024-03-11T21:11:24.615 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:24.643 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:24.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:24.645 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:11:24.746 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:11:24.774 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:11:24.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:24.777 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:11:24.878 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:11:24.906 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:11:24.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:24.908 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:11:24.955 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:11:25.008 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:25.032 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:25.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:25.034 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:11:25.135 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:11:25.145 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:25 smithi146 bash[23809]: cluster 2024-03-11T21:11:24.089210+0000 mon.a (mon.0) 1766 : cluster 0 osdmap e368: 8 total, 8 up, 8 in 2024-03-11T21:11:25.146 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:25 smithi146 bash[23809]: cluster 2024-03-11T21:11:24.346466+0000 mgr.y (mgr.24380) 763 : cluster 0 pgmap v865: 158 pgs: 10 unknown, 148 active+clean; 582 KiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2024-03-11T21:11:25.146 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:25 smithi146 bash[23809]: audit 2024-03-11T21:11:24.348682+0000 mon.a (mon.0) 1767 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-03-11T21:11:25.161 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:11:25.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:25.163 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:11:25.264 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:11:25.291 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:11:25.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:25.294 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:11:25.395 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:11:25.406 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:25 smithi023 bash[29146]: cluster 2024-03-11T21:11:24.089210+0000 mon.a (mon.0) 1766 : cluster 0 osdmap e368: 8 total, 8 up, 8 in 2024-03-11T21:11:25.406 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:25 smithi023 bash[29146]: cluster 2024-03-11T21:11:24.346466+0000 mgr.y (mgr.24380) 763 : cluster 0 pgmap v865: 158 pgs: 10 unknown, 148 active+clean; 582 KiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2024-03-11T21:11:25.406 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:25 smithi023 bash[29146]: audit 2024-03-11T21:11:24.348682+0000 mon.a (mon.0) 1767 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-03-11T21:11:25.406 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:25 smithi023 bash[21372]: cluster 2024-03-11T21:11:24.089210+0000 mon.a (mon.0) 1766 : cluster 0 osdmap e368: 8 total, 8 up, 8 in 2024-03-11T21:11:25.406 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:25 smithi023 bash[21372]: cluster 2024-03-11T21:11:24.346466+0000 mgr.y (mgr.24380) 763 : cluster 0 pgmap v865: 158 pgs: 10 unknown, 148 active+clean; 582 KiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2024-03-11T21:11:25.406 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:25 smithi023 bash[21372]: audit 2024-03-11T21:11:24.348682+0000 mon.a (mon.0) 1767 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-03-11T21:11:25.419 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:11:25.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:25.422 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:11:25.523 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:25.550 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:25.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:25.552 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:11:25.653 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:11:25.680 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:11:25.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:25.683 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:11:25.775 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T21:11:25.783 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:25.819 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-11T21:11:25.848 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:25.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:25.850 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:11:25.951 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:25.971 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:25.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:25.973 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:11:26.007 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:11:25 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:11:26.075 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:11:26.104 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:11:26.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:26.106 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:11:26.207 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:26.218 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:26 smithi146 bash[23809]: audit 2024-03-11T21:11:25.096687+0000 mon.a (mon.0) 1768 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-03-11T21:11:26.218 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:26 smithi146 bash[23809]: cluster 2024-03-11T21:11:25.111265+0000 mon.a (mon.0) 1769 : cluster 0 osdmap e369: 8 total, 8 up, 8 in 2024-03-11T21:11:26.219 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:26 smithi146 bash[23809]: cluster 2024-03-11T21:11:25.345390+0000 mon.a (mon.0) 1770 : cluster 0 osdmap e370: 8 total, 8 up, 8 in 2024-03-11T21:11:26.237 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:26.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:26.239 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:11:26.339 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:11:26.351 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:26 smithi023 bash[21372]: audit 2024-03-11T21:11:25.096687+0000 mon.a (mon.0) 1768 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-03-11T21:11:26.352 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:26 smithi023 bash[21372]: cluster 2024-03-11T21:11:25.111265+0000 mon.a (mon.0) 1769 : cluster 0 osdmap e369: 8 total, 8 up, 8 in 2024-03-11T21:11:26.352 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:26 smithi023 bash[21372]: cluster 2024-03-11T21:11:25.345390+0000 mon.a (mon.0) 1770 : cluster 0 osdmap e370: 8 total, 8 up, 8 in 2024-03-11T21:11:26.352 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:26 smithi023 bash[29146]: audit 2024-03-11T21:11:25.096687+0000 mon.a (mon.0) 1768 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-03-11T21:11:26.352 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:26 smithi023 bash[29146]: cluster 2024-03-11T21:11:25.111265+0000 mon.a (mon.0) 1769 : cluster 0 osdmap e369: 8 total, 8 up, 8 in 2024-03-11T21:11:26.352 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:26 smithi023 bash[29146]: cluster 2024-03-11T21:11:25.345390+0000 mon.a (mon.0) 1770 : cluster 0 osdmap e370: 8 total, 8 up, 8 in 2024-03-11T21:11:26.373 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:11:26.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:26.376 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:11:26.476 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:11:26.506 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:11:26.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:26.508 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:11:26.609 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:11:26.639 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:11:26.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:26.642 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:11:26.743 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:26.768 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:26.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:26.770 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:11:26.871 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:26.899 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:26.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:26.901 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:11:27.002 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:11:27.031 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:11:27.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:27.034 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:11:27.135 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:11:27.152 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:11:27.162 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:11:27.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:27.165 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:11:27.266 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:27.294 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:27.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:27.296 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:11:27.365 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 6, 7, 1] out_osds: [] dead_osds: [] live_osds: [0, 2, 3, 5, 6, 7, 4, 1] 2024-03-11T21:11:27.365 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-11T21:11:27.365 INFO:tasks.thrashosds.thrasher:inject_pause on osd.5 2024-03-11T21:11:27.365 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-03-11T21:11:27.365 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-03-11T21:11:27.366 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph daemon osd.5 config set filestore_inject_stall 3 2024-03-11T21:11:27.397 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:11:27.424 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:11:27.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:27.426 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:11:27.430 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:27 smithi146 bash[23809]: audit 2024-03-11T21:11:25.884423+0000 mgr.y (mgr.24380) 764 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:11:27.430 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:27 smithi146 bash[23809]: cluster 2024-03-11T21:11:26.347989+0000 mgr.y (mgr.24380) 765 : cluster 0 pgmap v868: 158 pgs: 5 remapped+peering, 3 unknown, 150 active+clean; 582 KiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:11:27.430 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:27 smithi146 bash[23809]: cluster 2024-03-11T21:11:26.369484+0000 mon.a (mon.0) 1771 : cluster 0 osdmap e371: 8 total, 8 up, 8 in 2024-03-11T21:11:27.431 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:27 smithi146 bash[23809]: cluster 2024-03-11T21:11:27.098709+0000 mon.a (mon.0) 1772 : cluster 3 Health check failed: Reduced data availability: 5 pgs peering (PG_AVAILABILITY) 2024-03-11T21:11:27.527 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:11:27.528 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:27 smithi023 bash[21372]: audit 2024-03-11T21:11:25.884423+0000 mgr.y (mgr.24380) 764 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:11:27.528 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:27 smithi023 bash[21372]: cluster 2024-03-11T21:11:26.347989+0000 mgr.y (mgr.24380) 765 : cluster 0 pgmap v868: 158 pgs: 5 remapped+peering, 3 unknown, 150 active+clean; 582 KiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:11:27.528 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:27 smithi023 bash[21372]: cluster 2024-03-11T21:11:26.369484+0000 mon.a (mon.0) 1771 : cluster 0 osdmap e371: 8 total, 8 up, 8 in 2024-03-11T21:11:27.528 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:27 smithi023 bash[21372]: cluster 2024-03-11T21:11:27.098709+0000 mon.a (mon.0) 1772 : cluster 3 Health check failed: Reduced data availability: 5 pgs peering (PG_AVAILABILITY) 2024-03-11T21:11:27.528 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:27 smithi023 bash[29146]: audit 2024-03-11T21:11:25.884423+0000 mgr.y (mgr.24380) 764 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:11:27.529 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:27 smithi023 bash[29146]: cluster 2024-03-11T21:11:26.347989+0000 mgr.y (mgr.24380) 765 : cluster 0 pgmap v868: 158 pgs: 5 remapped+peering, 3 unknown, 150 active+clean; 582 KiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:11:27.529 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:27 smithi023 bash[29146]: cluster 2024-03-11T21:11:26.369484+0000 mon.a (mon.0) 1771 : cluster 0 osdmap e371: 8 total, 8 up, 8 in 2024-03-11T21:11:27.529 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:27 smithi023 bash[29146]: cluster 2024-03-11T21:11:27.098709+0000 mon.a (mon.0) 1772 : cluster 3 Health check failed: Reduced data availability: 5 pgs peering (PG_AVAILABILITY) 2024-03-11T21:11:27.557 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:11:27.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:27.560 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:11:27.661 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:11:27.685 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:11:27.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:27.687 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:11:27.788 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:27.814 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:27.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:27.816 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:11:27.917 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:11:27.943 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:11:27.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:27.945 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:11:28.046 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:11:28.074 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:11:28.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:28.076 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:11:28.177 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:28.203 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:28.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:28.205 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:11:28.306 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:11:28.334 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:11:28.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:28.336 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:11:28.437 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:28.457 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:28.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:28.459 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:11:28.560 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:11:28.583 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:11:28.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:28.585 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:11:28.621 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-03-11T21:11:28.621 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:11:28.632 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:28 smithi146 bash[23809]: cluster 2024-03-11T21:11:27.373205+0000 mon.a (mon.0) 1773 : cluster 0 osdmap e372: 8 total, 8 up, 8 in 2024-03-11T21:11:28.632 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:28 smithi146 bash[23809]: cluster 2024-03-11T21:11:28.383280+0000 mon.a (mon.0) 1774 : cluster 0 osdmap e373: 8 total, 8 up, 8 in 2024-03-11T21:11:28.666 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:11:28 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:11:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:11:28.666 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:28 smithi023 bash[21372]: cluster 2024-03-11T21:11:27.373205+0000 mon.a (mon.0) 1773 : cluster 0 osdmap e372: 8 total, 8 up, 8 in 2024-03-11T21:11:28.666 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:28 smithi023 bash[21372]: cluster 2024-03-11T21:11:28.383280+0000 mon.a (mon.0) 1774 : cluster 0 osdmap e373: 8 total, 8 up, 8 in 2024-03-11T21:11:28.667 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:28 smithi023 bash[29146]: cluster 2024-03-11T21:11:27.373205+0000 mon.a (mon.0) 1773 : cluster 0 osdmap e372: 8 total, 8 up, 8 in 2024-03-11T21:11:28.667 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:28 smithi023 bash[29146]: cluster 2024-03-11T21:11:28.383280+0000 mon.a (mon.0) 1774 : cluster 0 osdmap e373: 8 total, 8 up, 8 in 2024-03-11T21:11:28.686 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:11:28.708 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:11:28.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:28.711 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:11:28.811 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:28.987 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:28.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:28.990 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:11:29.091 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:11:29.118 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:11:29.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:29.120 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:11:29.221 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:11:29.249 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:11:29.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:29.252 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:11:29.353 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:29.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:29.384 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:29.385 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:11:29.485 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:29.497 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:29 smithi146 bash[23809]: cluster 2024-03-11T21:11:28.348887+0000 mgr.y (mgr.24380) 766 : cluster 0 pgmap v871: 158 pgs: 5 remapped+peering, 3 unknown, 150 active+clean; 582 KiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:11:29.513 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:29.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:29.516 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:11:29.617 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:29.643 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:29.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:29.646 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:11:29.706 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-03-11T21:11:29.747 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:11:29.759 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:29 smithi023 bash[21372]: cluster 2024-03-11T21:11:28.348887+0000 mgr.y (mgr.24380) 766 : cluster 0 pgmap v871: 158 pgs: 5 remapped+peering, 3 unknown, 150 active+clean; 582 KiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:11:29.760 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:29 smithi023 bash[29146]: cluster 2024-03-11T21:11:28.348887+0000 mgr.y (mgr.24380) 766 : cluster 0 pgmap v871: 158 pgs: 5 remapped+peering, 3 unknown, 150 active+clean; 582 KiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:11:29.776 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:11:29.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:29.778 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:11:29.879 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:11:29.899 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:11:29.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:29.901 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:11:30.001 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:11:30.028 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:11:30.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:30.031 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:11:30.131 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:30.158 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:30.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:30.161 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:11:30.203 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:11:30.204 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:11:30.262 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:30.287 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:30.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:30.289 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:11:30.390 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:11:30.414 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:11:30.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:30.417 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:11:30.518 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:30.545 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:30.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:30.547 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:11:30.648 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:30.675 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:30.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:30.677 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:11:30.778 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:30.806 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:30.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:30.808 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:11:30.909 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:11:30.939 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:11:30.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:30.941 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:11:31.042 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:11:31.070 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:11:31.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:31.072 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:11:31.173 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:11:31.202 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:11:31.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:31.204 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:11:31.305 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:31.335 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:31.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:31.337 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:11:31.438 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:11:31.468 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:11:31.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:31.470 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:11:31.571 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:31.600 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:31.602 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:31 smithi023 bash[21372]: cluster 2024-03-11T21:11:29.695059+0000 osd.5 (osd.5) 439 : cluster 0 purged_snaps scrub starts 2024-03-11T21:11:31.602 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:31 smithi023 bash[21372]: cluster 2024-03-11T21:11:29.695140+0000 osd.5 (osd.5) 440 : cluster 0 purged_snaps scrub ok 2024-03-11T21:11:31.602 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:31 smithi023 bash[21372]: cluster 2024-03-11T21:11:30.349771+0000 mgr.y (mgr.24380) 767 : cluster 0 pgmap v874: 158 pgs: 4 remapped+peering, 154 active+clean; 582 KiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 80 KiB/s, 0 objects/s recovering 2024-03-11T21:11:31.602 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:31 smithi023 bash[21372]: cluster 2024-03-11T21:11:30.353889+0000 mon.a (mon.0) 1775 : cluster 0 osdmap e374: 8 total, 8 up, 8 in 2024-03-11T21:11:31.602 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:31 smithi023 bash[29146]: cluster 2024-03-11T21:11:29.695059+0000 osd.5 (osd.5) 439 : cluster 0 purged_snaps scrub starts 2024-03-11T21:11:31.602 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:31 smithi023 bash[29146]: cluster 2024-03-11T21:11:29.695140+0000 osd.5 (osd.5) 440 : cluster 0 purged_snaps scrub ok 2024-03-11T21:11:31.602 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:31 smithi023 bash[29146]: cluster 2024-03-11T21:11:30.349771+0000 mgr.y (mgr.24380) 767 : cluster 0 pgmap v874: 158 pgs: 4 remapped+peering, 154 active+clean; 582 KiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 80 KiB/s, 0 objects/s recovering 2024-03-11T21:11:31.603 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:31 smithi023 bash[29146]: cluster 2024-03-11T21:11:30.353889+0000 mon.a (mon.0) 1775 : cluster 0 osdmap e374: 8 total, 8 up, 8 in 2024-03-11T21:11:31.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:31.603 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:11:31.704 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:11:31.715 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:31 smithi146 bash[23809]: cluster 2024-03-11T21:11:29.695059+0000 osd.5 (osd.5) 439 : cluster 0 purged_snaps scrub starts 2024-03-11T21:11:31.715 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:31 smithi146 bash[23809]: cluster 2024-03-11T21:11:29.695140+0000 osd.5 (osd.5) 440 : cluster 0 purged_snaps scrub ok 2024-03-11T21:11:31.715 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:31 smithi146 bash[23809]: cluster 2024-03-11T21:11:30.349771+0000 mgr.y (mgr.24380) 767 : cluster 0 pgmap v874: 158 pgs: 4 remapped+peering, 154 active+clean; 582 KiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 80 KiB/s, 0 objects/s recovering 2024-03-11T21:11:31.715 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:31 smithi146 bash[23809]: cluster 2024-03-11T21:11:30.353889+0000 mon.a (mon.0) 1775 : cluster 0 osdmap e374: 8 total, 8 up, 8 in 2024-03-11T21:11:31.730 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:11:31.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:31.733 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:11:31.833 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:31.863 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:31.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:31.865 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:11:31.966 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:31.992 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:31.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:31.994 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:11:32.095 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:11:32.118 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:11:32.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:32.121 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:11:32.216 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:11:32.221 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:32.245 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:32.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:32.247 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:11:32.348 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:32.356 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:32 smithi023 bash[21372]: audit 2024-03-11T21:11:32.216348+0000 mon.a (mon.0) 1776 : audit 1 from='client.? 172.21.15.23:0/2575024680' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T21:11:32.381 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:32.385 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:32 smithi023 bash[29146]: audit 2024-03-11T21:11:32.216348+0000 mon.a (mon.0) 1776 : audit 1 from='client.? 172.21.15.23:0/2575024680' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T21:11:32.418 INFO:teuthology.orchestra.run.smithi023.stderr:pool 'unique_pool_3' removed 2024-03-11T21:11:32.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:32.419 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:11:32.507 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:32 smithi146 bash[23809]: audit 2024-03-11T21:11:32.216348+0000 mon.a (mon.0) 1776 : audit 1 from='client.? 172.21.15.23:0/2575024680' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T21:11:32.520 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:32.547 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:32.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:32.549 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:11:32.650 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:11:32.678 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:11:32.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:32.680 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:11:32.781 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:11:32.823 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:11:32.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:32.827 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:11:32.927 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:11:32.949 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:11:32.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:32.955 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:11:33.055 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:11:33.083 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:11:33.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:33.086 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:11:33.186 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:11:33.213 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:11:33.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:33.215 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:11:33.316 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:33.342 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:33.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:33.344 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:11:33.445 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:11:33.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:33 smithi146 bash[23809]: cluster 2024-03-11T21:11:32.350388+0000 mgr.y (mgr.24380) 768 : cluster 0 pgmap v875: 158 pgs: 158 active+clean; 582 KiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 855 B/s rd, 0 op/s; 96 KiB/s, 0 objects/s recovering 2024-03-11T21:11:33.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:33 smithi146 bash[23809]: cluster 2024-03-11T21:11:32.360999+0000 mon.a (mon.0) 1777 : cluster 1 Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs peering) 2024-03-11T21:11:33.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:33 smithi146 bash[23809]: audit 2024-03-11T21:11:32.376177+0000 mon.a (mon.0) 1778 : audit 1 from='client.? 172.21.15.23:0/2575024680' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-03-11T21:11:33.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:33 smithi146 bash[23809]: cluster 2024-03-11T21:11:32.380043+0000 mon.a (mon.0) 1779 : cluster 0 osdmap e375: 8 total, 8 up, 8 in 2024-03-11T21:11:33.469 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:11:33.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:33.471 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:11:33.572 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:33.593 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:33.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:33.595 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:11:33.696 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:11:33.708 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:33 smithi023 bash[21372]: cluster 2024-03-11T21:11:32.350388+0000 mgr.y (mgr.24380) 768 : cluster 0 pgmap v875: 158 pgs: 158 active+clean; 582 KiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 855 B/s rd, 0 op/s; 96 KiB/s, 0 objects/s recovering 2024-03-11T21:11:33.708 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:33 smithi023 bash[21372]: cluster 2024-03-11T21:11:32.360999+0000 mon.a (mon.0) 1777 : cluster 1 Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs peering) 2024-03-11T21:11:33.708 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:33 smithi023 bash[21372]: audit 2024-03-11T21:11:32.376177+0000 mon.a (mon.0) 1778 : audit 1 from='client.? 172.21.15.23:0/2575024680' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-03-11T21:11:33.708 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:33 smithi023 bash[21372]: cluster 2024-03-11T21:11:32.380043+0000 mon.a (mon.0) 1779 : cluster 0 osdmap e375: 8 total, 8 up, 8 in 2024-03-11T21:11:33.708 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:33 smithi023 bash[29146]: cluster 2024-03-11T21:11:32.350388+0000 mgr.y (mgr.24380) 768 : cluster 0 pgmap v875: 158 pgs: 158 active+clean; 582 KiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 855 B/s rd, 0 op/s; 96 KiB/s, 0 objects/s recovering 2024-03-11T21:11:33.708 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:33 smithi023 bash[29146]: cluster 2024-03-11T21:11:32.360999+0000 mon.a (mon.0) 1777 : cluster 1 Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs peering) 2024-03-11T21:11:33.708 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:33 smithi023 bash[29146]: audit 2024-03-11T21:11:32.376177+0000 mon.a (mon.0) 1778 : audit 1 from='client.? 172.21.15.23:0/2575024680' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-03-11T21:11:33.709 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:33 smithi023 bash[29146]: cluster 2024-03-11T21:11:32.380043+0000 mon.a (mon.0) 1779 : cluster 0 osdmap e375: 8 total, 8 up, 8 in 2024-03-11T21:11:33.724 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:11:33.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:33.726 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:11:33.827 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:33.852 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:33.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:33.855 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:11:33.955 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:11:33.986 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:11:33.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:33.988 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:11:34.089 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:34.115 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:34.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:34.117 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:11:34.218 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:34.248 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:34.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:34.251 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:11:34.351 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:34.378 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:34.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:34.381 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:11:34.481 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:34.509 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:34.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:34.512 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:11:34.613 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:11:34.641 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:11:34.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:34.644 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:11:34.744 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:11:34.773 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:11:34.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:34.776 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:11:34.782 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-03-11T21:11:34.783 INFO:tasks.radosbench:Beginning radosbench... 2024-03-11T21:11:34.783 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_4 2024-03-11T21:11:34.783 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd pool create unique_pool_4 16 2024-03-11T21:11:34.877 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:11:34.902 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:11:34.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:34.904 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:11:35.005 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:35.016 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:11:35.029 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:35.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:35.032 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:11:35.133 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:11:35.162 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:11:35.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:35.164 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:11:35.265 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:11:35.294 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:11:35.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:35.297 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:11:35.397 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:11:35.408 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:35 smithi023 bash[21372]: cluster 2024-03-11T21:11:34.350857+0000 mgr.y (mgr.24380) 769 : cluster 0 pgmap v877: 142 pgs: 142 active+clean; 582 KiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 96 KiB/s, 0 objects/s recovering 2024-03-11T21:11:35.408 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:35 smithi023 bash[21372]: cluster 2024-03-11T21:11:35.348127+0000 mon.a (mon.0) 1780 : cluster 0 osdmap e376: 8 total, 8 up, 8 in 2024-03-11T21:11:35.409 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:35 smithi023 bash[29146]: cluster 2024-03-11T21:11:34.350857+0000 mgr.y (mgr.24380) 769 : cluster 0 pgmap v877: 142 pgs: 142 active+clean; 582 KiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 96 KiB/s, 0 objects/s recovering 2024-03-11T21:11:35.409 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:35 smithi023 bash[29146]: cluster 2024-03-11T21:11:35.348127+0000 mon.a (mon.0) 1780 : cluster 0 osdmap e376: 8 total, 8 up, 8 in 2024-03-11T21:11:35.425 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:11:35.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:35.427 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:11:35.528 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:35.557 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:35.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:35.560 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:11:35.661 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:35.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:35 smithi146 bash[23809]: cluster 2024-03-11T21:11:34.350857+0000 mgr.y (mgr.24380) 769 : cluster 0 pgmap v877: 142 pgs: 142 active+clean; 582 KiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 96 KiB/s, 0 objects/s recovering 2024-03-11T21:11:35.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:35 smithi146 bash[23809]: cluster 2024-03-11T21:11:35.348127+0000 mon.a (mon.0) 1780 : cluster 0 osdmap e376: 8 total, 8 up, 8 in 2024-03-11T21:11:35.688 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:35.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:35.690 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:11:35.791 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:35.818 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:35.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:35.820 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:11:35.920 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:11:35.950 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:11:35.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:35.952 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:11:36.007 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:11:35 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:11:36.053 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:11:36.080 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:11:36.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:36.082 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:11:36.183 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:36.210 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:36.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:36.213 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:11:36.314 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:11:36.342 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:11:36.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:36.345 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:11:36.446 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:36.468 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:36.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:36.470 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:11:36.571 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:36.588 INFO:teuthology.orchestra.run.smithi023.stdout:osd.0: {} 2024-03-11T21:11:36.598 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:36.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:36.600 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:11:36.609 INFO:teuthology.orchestra.run.smithi023.stdout:osd.1: {} 2024-03-11T21:11:36.630 INFO:teuthology.orchestra.run.smithi023.stdout:osd.2: {} 2024-03-11T21:11:36.654 INFO:teuthology.orchestra.run.smithi023.stdout:osd.3: {} 2024-03-11T21:11:36.675 INFO:teuthology.orchestra.run.smithi023.stdout:osd.4: {} 2024-03-11T21:11:36.696 INFO:teuthology.orchestra.run.smithi023.stdout:osd.5: {} 2024-03-11T21:11:36.701 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:11:36.718 INFO:teuthology.orchestra.run.smithi023.stdout:osd.6: {} 2024-03-11T21:11:36.722 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:11:36.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:36.724 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:11:36.735 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-03-11T21:11:36.735 INFO:teuthology.orchestra.run.smithi146.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-03-11T21:11:36.735 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-03-11T21:11:36.754 INFO:teuthology.orchestra.run.smithi023.stdout:osd.7: {} 2024-03-11T21:11:36.755 INFO:teuthology.orchestra.run.smithi023.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:11:36.756 INFO:teuthology.orchestra.run.smithi023.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:11:36.756 INFO:teuthology.orchestra.run.smithi023.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:11:36.756 INFO:teuthology.orchestra.run.smithi023.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:11:36.756 INFO:teuthology.orchestra.run.smithi023.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:11:36.756 INFO:teuthology.orchestra.run.smithi023.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:11:36.756 INFO:teuthology.orchestra.run.smithi023.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:11:36.756 INFO:teuthology.orchestra.run.smithi023.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:11:36.824 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:11:36.844 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:11:36.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:36.846 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:11:36.947 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:36.968 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:36.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:36.971 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:11:37.071 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:11:37.102 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:11:37.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:37.107 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:11:37.107 INFO:teuthology.orchestra.run.smithi023.stderr:nodeep-scrub is set 2024-03-11T21:11:37.208 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:11:37.234 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:37 smithi146 bash[23809]: audit 2024-03-11T21:11:35.893422+0000 mgr.y (mgr.24380) 770 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:11:37.234 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:37 smithi146 bash[23809]: audit 2024-03-11T21:11:36.099754+0000 mon.a (mon.0) 1781 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:11:37.234 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:37 smithi146 bash[23809]: audit 2024-03-11T21:11:36.101044+0000 mon.a (mon.0) 1782 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:11:37.234 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:37 smithi146 bash[23809]: cluster 2024-03-11T21:11:36.351706+0000 mgr.y (mgr.24380) 771 : cluster 0 pgmap v879: 142 pgs: 142 active+clean; 582 KiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 43 KiB/s, 0 objects/s recovering 2024-03-11T21:11:37.234 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:37 smithi146 bash[23809]: audit 2024-03-11T21:11:36.561324+0000 mon.a (mon.0) 1783 : audit 0 from='client.? 172.21.15.23:0/650562602' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:11:37.234 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:37 smithi146 bash[23809]: audit 2024-03-11T21:11:36.792515+0000 mon.a (mon.0) 1784 : audit 1 from='client.? 172.21.15.23:0/1803641934' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:11:37.235 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:11:37.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:37.237 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:11:37.338 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:37.349 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:37 smithi023 bash[29146]: audit 2024-03-11T21:11:35.893422+0000 mgr.y (mgr.24380) 770 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:11:37.350 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:37 smithi023 bash[29146]: audit 2024-03-11T21:11:36.099754+0000 mon.a (mon.0) 1781 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:11:37.350 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:37 smithi023 bash[29146]: audit 2024-03-11T21:11:36.101044+0000 mon.a (mon.0) 1782 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:11:37.350 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:37 smithi023 bash[29146]: cluster 2024-03-11T21:11:36.351706+0000 mgr.y (mgr.24380) 771 : cluster 0 pgmap v879: 142 pgs: 142 active+clean; 582 KiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 43 KiB/s, 0 objects/s recovering 2024-03-11T21:11:37.350 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:37 smithi023 bash[29146]: audit 2024-03-11T21:11:36.561324+0000 mon.a (mon.0) 1783 : audit 0 from='client.? 172.21.15.23:0/650562602' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:11:37.350 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:37 smithi023 bash[29146]: audit 2024-03-11T21:11:36.792515+0000 mon.a (mon.0) 1784 : audit 1 from='client.? 172.21.15.23:0/1803641934' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:11:37.350 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:37 smithi023 bash[21372]: audit 2024-03-11T21:11:35.893422+0000 mgr.y (mgr.24380) 770 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:11:37.350 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:37 smithi023 bash[21372]: audit 2024-03-11T21:11:36.099754+0000 mon.a (mon.0) 1781 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:11:37.350 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:37 smithi023 bash[21372]: audit 2024-03-11T21:11:36.101044+0000 mon.a (mon.0) 1782 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:11:37.350 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:37 smithi023 bash[21372]: cluster 2024-03-11T21:11:36.351706+0000 mgr.y (mgr.24380) 771 : cluster 0 pgmap v879: 142 pgs: 142 active+clean; 582 KiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 43 KiB/s, 0 objects/s recovering 2024-03-11T21:11:37.351 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:37 smithi023 bash[21372]: audit 2024-03-11T21:11:36.561324+0000 mon.a (mon.0) 1783 : audit 0 from='client.? 172.21.15.23:0/650562602' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:11:37.351 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:37 smithi023 bash[21372]: audit 2024-03-11T21:11:36.792515+0000 mon.a (mon.0) 1784 : audit 1 from='client.? 172.21.15.23:0/1803641934' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:11:37.363 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:37.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:37.366 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:11:37.467 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:11:37.494 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:11:37.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:37.496 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:11:37.598 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:37.627 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:37.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:37.630 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:11:37.731 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:11:37.757 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:11:37.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:37.760 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:11:37.860 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:37.890 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:37.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:37.893 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:11:37.994 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:11:38.023 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:11:38.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:38.025 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:11:38.126 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:11:38.153 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:11:38.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:38.156 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:11:38.257 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:38.277 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:38 smithi023 bash[29146]: cluster 2024-03-11T21:11:37.097895+0000 mon.a (mon.0) 1785 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:11:38.277 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:38 smithi023 bash[29146]: audit 2024-03-11T21:11:37.108878+0000 mon.a (mon.0) 1786 : audit 1 from='client.? 172.21.15.23:0/1803641934' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:11:38.278 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:38 smithi023 bash[29146]: cluster 2024-03-11T21:11:37.117149+0000 mon.a (mon.0) 1787 : cluster 0 osdmap e377: 8 total, 8 up, 8 in 2024-03-11T21:11:38.278 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:38 smithi023 bash[21372]: cluster 2024-03-11T21:11:37.097895+0000 mon.a (mon.0) 1785 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:11:38.278 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:38 smithi023 bash[21372]: audit 2024-03-11T21:11:37.108878+0000 mon.a (mon.0) 1786 : audit 1 from='client.? 172.21.15.23:0/1803641934' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:11:38.278 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:38 smithi023 bash[21372]: cluster 2024-03-11T21:11:37.117149+0000 mon.a (mon.0) 1787 : cluster 0 osdmap e377: 8 total, 8 up, 8 in 2024-03-11T21:11:38.284 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:38.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:38.286 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:11:38.387 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:38.398 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:11:38 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:11:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:11:38.415 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:38.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:38.417 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:11:38.507 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:38 smithi146 bash[23809]: cluster 2024-03-11T21:11:37.097895+0000 mon.a (mon.0) 1785 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:11:38.507 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:38 smithi146 bash[23809]: audit 2024-03-11T21:11:37.108878+0000 mon.a (mon.0) 1786 : audit 1 from='client.? 172.21.15.23:0/1803641934' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:11:38.507 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:38 smithi146 bash[23809]: cluster 2024-03-11T21:11:37.117149+0000 mon.a (mon.0) 1787 : cluster 0 osdmap e377: 8 total, 8 up, 8 in 2024-03-11T21:11:38.518 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:38.546 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:38.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:38.548 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:11:38.649 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:38.675 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:38.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:38.678 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:11:38.779 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:38.801 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:38.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:38.804 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:11:38.905 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:11:38.934 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:11:38.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:38.937 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:11:39.037 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:11:39.067 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:11:39.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:39.069 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:11:39.170 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:39.198 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:39.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:39.200 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:11:39.301 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:39.329 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:39.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:39.332 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:11:39.420 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:39 smithi146 bash[23809]: cluster 2024-03-11T21:11:38.352774+0000 mgr.y (mgr.24380) 772 : cluster 0 pgmap v881: 142 pgs: 142 active+clean; 582 KiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-11T21:11:39.433 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:39.457 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:39.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:39.459 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:11:39.527 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:39 smithi023 bash[21372]: cluster 2024-03-11T21:11:38.352774+0000 mgr.y (mgr.24380) 772 : cluster 0 pgmap v881: 142 pgs: 142 active+clean; 582 KiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-11T21:11:39.527 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:39 smithi023 bash[29146]: cluster 2024-03-11T21:11:38.352774+0000 mgr.y (mgr.24380) 772 : cluster 0 pgmap v881: 142 pgs: 142 active+clean; 582 KiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-11T21:11:39.560 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:11:39.580 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:11:39.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:39.582 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:11:39.621 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-03-11T21:11:39.621 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-03-11T21:11:39.622 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 30, 2024-03-11T21:11:39.622 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-03-11T21:11:39.622 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-03-11T21:11:39.683 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:11:39.713 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:11:39.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:39.715 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:11:39.740 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T21:11:39.817 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:11:39.839 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:11:39.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:39.841 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:11:39.942 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:11:39.963 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:11:39.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:39.965 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:11:40.066 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:11:40.095 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:11:40.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:40.097 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:11:40.198 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:11:40.226 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:11:40.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:40.229 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:11:40.315 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd unset noscrub 2024-03-11T21:11:40.330 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:11:40.382 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:11:40.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:40.385 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:11:40.486 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:11:40.488 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-03-11T21:11:40.513 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:11:40.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:40.515 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:11:40.616 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:40.639 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:40.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:40.642 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:11:40.742 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:11:40.767 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:11:40.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:40.769 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:11:40.870 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:11:40.899 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:11:40.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:40.902 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:11:41.003 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:11:41.030 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:11:41.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:41.032 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:11:41.133 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:11:41.162 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:11:41.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:41.165 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:11:41.266 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:41.279 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:11:41.293 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:41.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:41.295 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:11:41.396 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:11:41.423 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:11:41.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:41.425 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:11:41.526 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:11:41.527 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:41 smithi023 bash[21372]: cluster 2024-03-11T21:11:40.349574+0000 mon.a (mon.0) 1788 : cluster 0 osdmap e378: 8 total, 8 up, 8 in 2024-03-11T21:11:41.527 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:41 smithi023 bash[21372]: cluster 2024-03-11T21:11:40.353392+0000 mgr.y (mgr.24380) 773 : cluster 0 pgmap v883: 142 pgs: 142 active+clean; 582 KiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-11T21:11:41.527 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:41 smithi023 bash[29146]: cluster 2024-03-11T21:11:40.349574+0000 mon.a (mon.0) 1788 : cluster 0 osdmap e378: 8 total, 8 up, 8 in 2024-03-11T21:11:41.528 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:41 smithi023 bash[29146]: cluster 2024-03-11T21:11:40.353392+0000 mgr.y (mgr.24380) 773 : cluster 0 pgmap v883: 142 pgs: 142 active+clean; 582 KiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-11T21:11:41.555 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:11:41.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:41.557 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:11:41.658 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:11:41.686 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:11:41.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:41.689 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:11:41.757 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:41 smithi146 bash[23809]: cluster 2024-03-11T21:11:40.349574+0000 mon.a (mon.0) 1788 : cluster 0 osdmap e378: 8 total, 8 up, 8 in 2024-03-11T21:11:41.757 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:41 smithi146 bash[23809]: cluster 2024-03-11T21:11:40.353392+0000 mgr.y (mgr.24380) 773 : cluster 0 pgmap v883: 142 pgs: 142 active+clean; 582 KiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-11T21:11:41.790 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:41.818 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:41.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:41.820 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:11:41.921 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:11:41.948 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:11:41.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:41.951 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:11:42.052 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:42.080 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:42.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:42.083 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:11:42.184 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:42.209 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:42.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:42.211 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:11:42.312 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:11:42.333 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:11:42.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:42.335 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:11:42.436 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:11:42.466 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:11:42.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:42.468 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:11:42.569 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:11:42.598 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:11:42.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:42.600 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:11:42.701 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:42.729 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:42.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:42.731 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:11:42.834 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:11:42.861 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:11:42.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:42.864 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:11:42.964 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:11:42.991 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:11:42.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:42.994 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:11:43.095 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:43.123 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:43.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:43.125 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:11:43.226 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:11:43.249 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:11:43.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:43.252 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:11:43.305 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 6, 7, 1] out_osds: [] dead_osds: [] live_osds: [0, 2, 3, 5, 6, 7, 4, 1] 2024-03-11T21:11:43.305 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-11T21:11:43.306 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T21:11:43.352 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:11:43.382 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:11:43.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:43.384 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:11:43.485 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:43.507 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:43 smithi146 bash[23809]: cluster 2024-03-11T21:11:42.354213+0000 mgr.y (mgr.24380) 774 : cluster 0 pgmap v884: 142 pgs: 142 active+clean; 582 KiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 730 B/s rd, 0 op/s 2024-03-11T21:11:43.507 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:43 smithi146 bash[23809]: cluster 2024-03-11T21:11:42.713538+0000 osd.7 (osd.7) 487 : cluster 0 purged_snaps scrub starts 2024-03-11T21:11:43.507 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:43 smithi146 bash[23809]: cluster 2024-03-11T21:11:42.713656+0000 osd.7 (osd.7) 488 : cluster 0 purged_snaps scrub ok 2024-03-11T21:11:43.508 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:43.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:43.509 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:11:43.610 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:43.622 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:43 smithi023 bash[21372]: cluster 2024-03-11T21:11:42.354213+0000 mgr.y (mgr.24380) 774 : cluster 0 pgmap v884: 142 pgs: 142 active+clean; 582 KiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 730 B/s rd, 0 op/s 2024-03-11T21:11:43.622 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:43 smithi023 bash[21372]: cluster 2024-03-11T21:11:42.713538+0000 osd.7 (osd.7) 487 : cluster 0 purged_snaps scrub starts 2024-03-11T21:11:43.622 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:43 smithi023 bash[21372]: cluster 2024-03-11T21:11:42.713656+0000 osd.7 (osd.7) 488 : cluster 0 purged_snaps scrub ok 2024-03-11T21:11:43.622 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:43 smithi023 bash[29146]: cluster 2024-03-11T21:11:42.354213+0000 mgr.y (mgr.24380) 774 : cluster 0 pgmap v884: 142 pgs: 142 active+clean; 582 KiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 730 B/s rd, 0 op/s 2024-03-11T21:11:43.622 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:43 smithi023 bash[29146]: cluster 2024-03-11T21:11:42.713538+0000 osd.7 (osd.7) 487 : cluster 0 purged_snaps scrub starts 2024-03-11T21:11:43.622 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:43 smithi023 bash[29146]: cluster 2024-03-11T21:11:42.713656+0000 osd.7 (osd.7) 488 : cluster 0 purged_snaps scrub ok 2024-03-11T21:11:43.636 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:43.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:43.638 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:11:43.738 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:43.764 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:43.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:43.766 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:11:43.867 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:11:43.895 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:11:43.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:43.897 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:11:43.997 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:11:44.026 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:11:44.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:44.028 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:11:44.129 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:11:44.158 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:11:44.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:44.160 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:11:44.261 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:44.287 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:44.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:44.290 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:11:44.327 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:11:44.328 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:11:44.390 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:44.415 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:44.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:44.416 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:11:44.517 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:44.546 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:44.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:44.548 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:11:44.649 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:11:44.660 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:44 smithi023 bash[21372]: cluster 2024-03-11T21:11:42.954895+0000 osd.3 (osd.3) 426 : cluster 0 purged_snaps scrub starts 2024-03-11T21:11:44.660 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:44 smithi023 bash[21372]: cluster 2024-03-11T21:11:42.955051+0000 osd.3 (osd.3) 427 : cluster 0 purged_snaps scrub ok 2024-03-11T21:11:44.660 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:44 smithi023 bash[21372]: cluster 2024-03-11T21:11:43.383434+0000 mon.a (mon.0) 1789 : cluster 0 osdmap e379: 8 total, 8 up, 8 in 2024-03-11T21:11:44.661 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:44 smithi023 bash[29146]: cluster 2024-03-11T21:11:42.954895+0000 osd.3 (osd.3) 426 : cluster 0 purged_snaps scrub starts 2024-03-11T21:11:44.661 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:44 smithi023 bash[29146]: cluster 2024-03-11T21:11:42.955051+0000 osd.3 (osd.3) 427 : cluster 0 purged_snaps scrub ok 2024-03-11T21:11:44.661 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:44 smithi023 bash[29146]: cluster 2024-03-11T21:11:43.383434+0000 mon.a (mon.0) 1789 : cluster 0 osdmap e379: 8 total, 8 up, 8 in 2024-03-11T21:11:44.678 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:11:44.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:44.680 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:11:44.757 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:44 smithi146 bash[23809]: cluster 2024-03-11T21:11:42.954895+0000 osd.3 (osd.3) 426 : cluster 0 purged_snaps scrub starts 2024-03-11T21:11:44.757 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:44 smithi146 bash[23809]: cluster 2024-03-11T21:11:42.955051+0000 osd.3 (osd.3) 427 : cluster 0 purged_snaps scrub ok 2024-03-11T21:11:44.757 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:44 smithi146 bash[23809]: cluster 2024-03-11T21:11:43.383434+0000 mon.a (mon.0) 1789 : cluster 0 osdmap e379: 8 total, 8 up, 8 in 2024-03-11T21:11:44.781 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:44.806 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:44.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:44.809 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:11:44.910 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:11:44.935 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:11:44.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:44.938 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:11:45.038 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:11:45.066 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:11:45.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:45.068 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:11:45.169 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:11:45.197 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:11:45.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:45.199 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:11:45.274 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:11:45.301 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:11:45.325 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:11:45.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:45.327 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:11:45.428 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:11:45.450 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:11:45.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:45.452 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:11:45.507 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:45 smithi146 bash[23809]: cluster 2024-03-11T21:11:44.354721+0000 mgr.y (mgr.24380) 775 : cluster 0 pgmap v886: 142 pgs: 142 active+clean; 582 KiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 706 B/s rd, 0 op/s 2024-03-11T21:11:45.553 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:11:45.577 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:11:45.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:45.580 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:11:45.664 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:45 smithi023 bash[21372]: cluster 2024-03-11T21:11:44.354721+0000 mgr.y (mgr.24380) 775 : cluster 0 pgmap v886: 142 pgs: 142 active+clean; 582 KiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 706 B/s rd, 0 op/s 2024-03-11T21:11:45.665 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:45 smithi023 bash[29146]: cluster 2024-03-11T21:11:44.354721+0000 mgr.y (mgr.24380) 775 : cluster 0 pgmap v886: 142 pgs: 142 active+clean; 582 KiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 706 B/s rd, 0 op/s 2024-03-11T21:11:45.681 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:11:45.702 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:11:45.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:45.706 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:11:45.806 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:45.835 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:45.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:45.837 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:11:45.938 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:11:45.959 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:11:45.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:45.961 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:11:46.062 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:46.082 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:46.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:46.085 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:11:46.185 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:11:46.204 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:11:46.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:46.207 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:11:46.257 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:11:45 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:11:46.307 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:11:46.335 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:11:46.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:46.337 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:11:46.438 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:46.464 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:46.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:46.466 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:11:46.544 INFO:teuthology.orchestra.run.smithi023.stderr:pool 'unique_pool_4' created 2024-03-11T21:11:46.568 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:11:46.578 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:46 smithi146 bash[23809]: audit 2024-03-11T21:11:46.056674+0000 mon.c (mon.2) 89 : audit 1 from='client.? 172.21.15.23:0/791121069' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-03-11T21:11:46.578 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:46 smithi146 bash[23809]: audit 2024-03-11T21:11:46.057088+0000 mon.a (mon.0) 1790 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-03-11T21:11:46.592 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:11:46.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:46.594 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:11:46.695 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:11:46.722 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:11:46.722 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:46 smithi023 bash[21372]: audit 2024-03-11T21:11:46.056674+0000 mon.c (mon.2) 89 : audit 1 from='client.? 172.21.15.23:0/791121069' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-03-11T21:11:46.722 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:46 smithi023 bash[21372]: audit 2024-03-11T21:11:46.057088+0000 mon.a (mon.0) 1790 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-03-11T21:11:46.723 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:46 smithi023 bash[29146]: audit 2024-03-11T21:11:46.056674+0000 mon.c (mon.2) 89 : audit 1 from='client.? 172.21.15.23:0/791121069' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-03-11T21:11:46.723 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:46 smithi023 bash[29146]: audit 2024-03-11T21:11:46.057088+0000 mon.a (mon.0) 1790 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-03-11T21:11:46.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:46.724 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:11:46.825 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:46.851 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:46.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:46.853 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:11:46.954 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:11:46.981 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:11:46.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:46.983 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:11:47.084 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:11:47.111 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:11:47.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:47.113 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:11:47.214 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:47.241 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:47.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:47.243 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:11:47.344 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:47.371 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:47.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:47.373 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:11:47.474 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:11:47.502 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:11:47.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:47.504 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:11:47.605 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:11:47.616 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:47 smithi146 bash[23809]: audit 2024-03-11T21:11:45.901263+0000 mgr.y (mgr.24380) 776 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:11:47.617 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:47 smithi146 bash[23809]: cluster 2024-03-11T21:11:46.356101+0000 mgr.y (mgr.24380) 777 : cluster 0 pgmap v887: 142 pgs: 142 active+clean; 582 KiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:11:47.617 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:47 smithi146 bash[23809]: audit 2024-03-11T21:11:46.538580+0000 mon.a (mon.0) 1791 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-03-11T21:11:47.617 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:47 smithi146 bash[23809]: cluster 2024-03-11T21:11:46.545788+0000 mon.a (mon.0) 1792 : cluster 0 osdmap e380: 8 total, 8 up, 8 in 2024-03-11T21:11:47.617 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:47 smithi146 bash[23809]: cluster 2024-03-11T21:11:47.545199+0000 mon.a (mon.0) 1793 : cluster 0 osdmap e381: 8 total, 8 up, 8 in 2024-03-11T21:11:47.632 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:11:47.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:47.635 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:11:47.735 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:11:47.747 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:47 smithi023 bash[21372]: audit 2024-03-11T21:11:45.901263+0000 mgr.y (mgr.24380) 776 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:11:47.747 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:47 smithi023 bash[21372]: cluster 2024-03-11T21:11:46.356101+0000 mgr.y (mgr.24380) 777 : cluster 0 pgmap v887: 142 pgs: 142 active+clean; 582 KiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:11:47.747 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:47 smithi023 bash[21372]: audit 2024-03-11T21:11:46.538580+0000 mon.a (mon.0) 1791 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-03-11T21:11:47.747 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:47 smithi023 bash[21372]: cluster 2024-03-11T21:11:46.545788+0000 mon.a (mon.0) 1792 : cluster 0 osdmap e380: 8 total, 8 up, 8 in 2024-03-11T21:11:47.748 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:47 smithi023 bash[21372]: cluster 2024-03-11T21:11:47.545199+0000 mon.a (mon.0) 1793 : cluster 0 osdmap e381: 8 total, 8 up, 8 in 2024-03-11T21:11:47.748 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:47 smithi023 bash[29146]: audit 2024-03-11T21:11:45.901263+0000 mgr.y (mgr.24380) 776 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:11:47.748 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:47 smithi023 bash[29146]: cluster 2024-03-11T21:11:46.356101+0000 mgr.y (mgr.24380) 777 : cluster 0 pgmap v887: 142 pgs: 142 active+clean; 582 KiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:11:47.748 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:47 smithi023 bash[29146]: audit 2024-03-11T21:11:46.538580+0000 mon.a (mon.0) 1791 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-03-11T21:11:47.748 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:47 smithi023 bash[29146]: cluster 2024-03-11T21:11:46.545788+0000 mon.a (mon.0) 1792 : cluster 0 osdmap e380: 8 total, 8 up, 8 in 2024-03-11T21:11:47.748 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:47 smithi023 bash[29146]: cluster 2024-03-11T21:11:47.545199+0000 mon.a (mon.0) 1793 : cluster 0 osdmap e381: 8 total, 8 up, 8 in 2024-03-11T21:11:47.763 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:11:47.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:47.766 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:11:47.867 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:47.897 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:47.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:47.900 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:11:48.001 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:48.035 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:48.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:48.036 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:11:48.137 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:11:48.167 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:11:48.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:48.169 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:11:48.270 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:11:48.303 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:11:48.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:48.306 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:11:48.407 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:11:48.438 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:11:48.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:48.440 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:11:48.541 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:11:48.566 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:11:48.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:48.568 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:11:48.669 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:11:48.681 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:11:48 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:11:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:11:48.693 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:11:48.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:48.695 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:11:48.796 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:11:48.824 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:11:48.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:48.827 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:11:48.927 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:48.949 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:48.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:48.951 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:11:48.986 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-03-11T21:11:48.986 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-03-11T21:11:48.986 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-03-11T21:11:48.986 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-03-11T21:11:48.986 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:48.986 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.15081.0:45727 9.e e262/251 9:7adeeb78:::benchmark_data_smithi023_106206_object45726:head v 262'2855, mlcod=261'2853)", 2024-03-11T21:11:48.986 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T21:03:27.496485+0000", 2024-03-11T21:11:48.986 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 501.49860511899999, 2024-03-11T21:11:48.986 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.41900660499999998, 2024-03-11T21:11:48.986 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:11:48.987 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:11:48.987 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:11:48.987 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:48.987 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:11:48.987 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:27.496485+0000", 2024-03-11T21:11:48.987 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:11:48.987 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:48.987 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:48.987 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:11:48.987 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:27.496485+0000", 2024-03-11T21:11:48.987 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:11:48.987 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:48.987 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:48.987 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:11:48.988 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:27.496485+0000", 2024-03-11T21:11:48.988 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1e-07 2024-03-11T21:11:48.988 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:48.988 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:48.988 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:11:48.988 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:27.496505+0000", 2024-03-11T21:11:48.988 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9826e-05 2024-03-11T21:11:48.988 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:48.988 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:48.988 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:11:48.988 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:27.496516+0000", 2024-03-11T21:11:48.988 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1747e-05 2024-03-11T21:11:48.988 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:48.988 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:48.988 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:11:48.989 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:27.496525+0000", 2024-03-11T21:11:48.989 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8240000000000006e-06 2024-03-11T21:11:48.989 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:48.989 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:48.989 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:11:48.989 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:27.496586+0000", 2024-03-11T21:11:48.989 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0668999999999998e-05 2024-03-11T21:11:48.989 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:48.989 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:48.989 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:11:48.989 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:27.496605+0000", 2024-03-11T21:11:48.989 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9536999999999999e-05 2024-03-11T21:11:48.989 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:48.990 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:48.990 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:11:48.990 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:27.915382+0000", 2024-03-11T21:11:48.990 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.41877668499999998 2024-03-11T21:11:48.990 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:48.990 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:48.990 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:11:48.990 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:27.915491+0000", 2024-03-11T21:11:48.990 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000109107 2024-03-11T21:11:48.990 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:11:48.990 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:11:48.990 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:11:48.990 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:48.991 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:48.991 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.15081.0:45729 9.e e262/251 9:7553524a:::benchmark_data_smithi023_106206_object45728:head v 262'2856, mlcod=261'2853)", 2024-03-11T21:11:48.991 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T21:03:27.498289+0000", 2024-03-11T21:11:48.991 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 501.49680073600001, 2024-03-11T21:11:48.991 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.42696921599999998, 2024-03-11T21:11:48.991 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:11:48.991 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:11:48.991 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:11:48.991 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:48.991 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:11:48.991 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:27.498289+0000", 2024-03-11T21:11:48.991 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:11:48.991 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:48.991 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:48.992 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:11:48.992 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:27.498289+0000", 2024-03-11T21:11:48.992 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:11:48.992 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:48.992 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:48.992 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:11:48.992 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:27.498290+0000", 2024-03-11T21:11:48.992 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2799999999999995e-07 2024-03-11T21:11:48.992 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:48.992 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:48.992 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:11:48.993 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:27.498311+0000", 2024-03-11T21:11:48.993 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1702e-05 2024-03-11T21:11:48.993 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:48.993 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:48.993 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:11:48.993 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:27.914495+0000", 2024-03-11T21:11:48.993 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.41618395899999999 2024-03-11T21:11:48.993 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:48.993 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:48.993 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:11:48.993 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:27.914529+0000", 2024-03-11T21:11:48.993 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3923e-05 2024-03-11T21:11:48.993 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:48.993 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:48.994 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:11:48.994 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:27.915869+0000", 2024-03-11T21:11:48.994 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001339556 2024-03-11T21:11:48.994 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:48.994 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:48.994 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:11:48.994 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:27.915908+0000", 2024-03-11T21:11:48.994 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8612999999999998e-05 2024-03-11T21:11:48.994 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:48.994 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:48.994 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:11:48.994 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:27.925157+0000", 2024-03-11T21:11:48.994 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0092493539999999996 2024-03-11T21:11:48.994 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:48.995 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:48.995 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:11:48.995 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:27.925258+0000", 2024-03-11T21:11:48.995 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000101381 2024-03-11T21:11:48.995 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:11:48.995 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:11:48.995 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:11:48.995 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:48.995 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:48.995 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.15081.0:64010 9.c e271/251 9:3cedf398:::benchmark_data_smithi023_106206_object64009:head v 271'4052, mlcod=271'4050)", 2024-03-11T21:11:48.995 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T21:03:42.884430+0000", 2024-03-11T21:11:48.995 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 486.11065952600001, 2024-03-11T21:11:48.995 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.35927554, 2024-03-11T21:11:48.995 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:11:48.995 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:11:48.996 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:11:48.996 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:48.996 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:11:48.996 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:42.884430+0000", 2024-03-11T21:11:48.996 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:11:48.996 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:48.996 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:48.996 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:11:48.996 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:42.884430+0000", 2024-03-11T21:11:48.996 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:11:48.996 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:48.996 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:48.996 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:11:48.996 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:42.884431+0000", 2024-03-11T21:11:48.996 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.33e-07 2024-03-11T21:11:48.996 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:48.996 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:48.997 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:11:48.997 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:42.884499+0000", 2024-03-11T21:11:48.997 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7316000000000004e-05 2024-03-11T21:11:48.997 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:48.997 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:48.997 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:11:48.997 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:42.884513+0000", 2024-03-11T21:11:48.997 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.483e-05 2024-03-11T21:11:48.997 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:48.997 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:48.997 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:11:48.997 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:42.884520+0000", 2024-03-11T21:11:48.997 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9070000000000001e-06 2024-03-11T21:11:48.997 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:48.997 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:48.998 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:11:48.998 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:42.884576+0000", 2024-03-11T21:11:48.998 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5838999999999998e-05 2024-03-11T21:11:48.998 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:48.998 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:48.998 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:11:48.998 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:43.239006+0000", 2024-03-11T21:11:48.998 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.35443018300000001 2024-03-11T21:11:48.998 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:48.998 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:48.998 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:11:48.998 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:43.243576+0000", 2024-03-11T21:11:48.998 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0045697969999999996 2024-03-11T21:11:48.998 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:48.998 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:48.999 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:11:48.999 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:43.243706+0000", 2024-03-11T21:11:48.999 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012973499999999999 2024-03-11T21:11:48.999 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:11:48.999 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:11:48.999 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:11:48.999 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:48.999 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:48.999 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.15081.0:64017 9.e e271/251 9:743c5f8a:::benchmark_data_smithi023_106206_object64016:head v 271'3998, mlcod=271'3996)", 2024-03-11T21:11:48.999 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T21:03:42.889411+0000", 2024-03-11T21:11:48.999 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 486.105678947, 2024-03-11T21:11:48.999 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.357421247, 2024-03-11T21:11:48.999 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:11:48.999 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:11:48.999 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:11:48.999 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.000 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:11:49.000 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:42.889411+0000", 2024-03-11T21:11:49.000 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:11:49.000 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.000 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.000 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:11:49.000 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:42.889411+0000", 2024-03-11T21:11:49.000 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:11:49.000 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.000 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.000 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:11:49.000 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:42.889412+0000", 2024-03-11T21:11:49.000 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0389999999999999e-06 2024-03-11T21:11:49.000 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.000 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.001 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:11:49.001 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:42.889441+0000", 2024-03-11T21:11:49.001 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8948000000000001e-05 2024-03-11T21:11:49.001 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.001 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.001 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:11:49.001 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:43.238302+0000", 2024-03-11T21:11:49.001 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.34886149599999999 2024-03-11T21:11:49.001 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.001 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.001 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:11:49.001 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:43.238334+0000", 2024-03-11T21:11:49.001 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2011999999999998e-05 2024-03-11T21:11:49.001 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.001 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.001 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:11:49.002 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:43.239241+0000", 2024-03-11T21:11:49.002 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00090631200000000005 2024-03-11T21:11:49.002 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.002 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.002 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:11:49.002 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:43.239296+0000", 2024-03-11T21:11:49.002 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5018000000000001e-05 2024-03-11T21:11:49.002 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.002 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.002 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:11:49.002 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:43.246743+0000", 2024-03-11T21:11:49.002 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0074477679999999996 2024-03-11T21:11:49.002 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.003 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.003 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:11:49.003 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:03:43.246832+0000", 2024-03-11T21:11:49.003 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8653999999999995e-05 2024-03-11T21:11:49.003 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:11:49.003 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:11:49.003 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:11:49.003 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.003 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.003 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.15081.0:142575 9.e e284/251 9:7bb92a1d:::benchmark_data_smithi023_106206_object31429:head v 284'8978, mlcod=284'8976)", 2024-03-11T21:11:49.004 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T21:05:05.351173+0000", 2024-03-11T21:11:49.004 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 403.64391700499999, 2024-03-11T21:11:49.004 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.43447089100000003, 2024-03-11T21:11:49.004 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:11:49.004 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:11:49.004 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:11:49.004 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.004 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:11:49.004 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:05.351173+0000", 2024-03-11T21:11:49.004 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:11:49.004 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.004 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.004 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:11:49.004 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:05.351173+0000", 2024-03-11T21:11:49.005 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:11:49.005 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.005 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.005 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:11:49.005 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:05.351174+0000", 2024-03-11T21:11:49.005 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.234e-06 2024-03-11T21:11:49.005 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.005 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.005 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:11:49.005 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:05.351183+0000", 2024-03-11T21:11:49.005 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.9619999999999999e-06 2024-03-11T21:11:49.005 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.005 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.005 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:11:49.006 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:05.764785+0000", 2024-03-11T21:11:49.006 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.41360211099999999 2024-03-11T21:11:49.006 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.006 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.006 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:11:49.006 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:05.764809+0000", 2024-03-11T21:11:49.006 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4148999999999999e-05 2024-03-11T21:11:49.006 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.006 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.006 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:11:49.006 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:05.764951+0000", 2024-03-11T21:11:49.006 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000141254 2024-03-11T21:11:49.006 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.007 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.007 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:11:49.007 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:05.765011+0000", 2024-03-11T21:11:49.007 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0739000000000002e-05 2024-03-11T21:11:49.007 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.007 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.007 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:11:49.007 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:05.785539+0000", 2024-03-11T21:11:49.007 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.020527561999999999 2024-03-11T21:11:49.007 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.007 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.007 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:11:49.007 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:05.785644+0000", 2024-03-11T21:11:49.007 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010488 2024-03-11T21:11:49.007 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:11:49.008 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:11:49.008 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:11:49.008 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.008 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.008 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.15081.0:143669 9.c e284/251 9:33cff7cf:::benchmark_data_smithi023_106206_object32523:head v 284'9056, mlcod=284'9054)", 2024-03-11T21:11:49.008 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T21:05:07.061773+0000", 2024-03-11T21:11:49.008 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 401.93331722099998, 2024-03-11T21:11:49.008 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.45908703099999998, 2024-03-11T21:11:49.008 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:11:49.008 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:11:49.008 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:11:49.008 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.008 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:11:49.008 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:07.061773+0000", 2024-03-11T21:11:49.008 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:11:49.009 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.009 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.009 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:11:49.009 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:07.061773+0000", 2024-03-11T21:11:49.009 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:11:49.009 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.009 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.009 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:11:49.009 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:07.061774+0000", 2024-03-11T21:11:49.009 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2800000000000005e-07 2024-03-11T21:11:49.009 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.009 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.009 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:11:49.009 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:07.061781+0000", 2024-03-11T21:11:49.010 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5650000000000004e-06 2024-03-11T21:11:49.010 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.010 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.010 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:11:49.010 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:07.061792+0000", 2024-03-11T21:11:49.010 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.081e-05 2024-03-11T21:11:49.010 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.010 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.010 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:11:49.010 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:07.061799+0000", 2024-03-11T21:11:49.010 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1280000000000002e-06 2024-03-11T21:11:49.010 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.010 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.010 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:11:49.011 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:07.061869+0000", 2024-03-11T21:11:49.011 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9726000000000003e-05 2024-03-11T21:11:49.011 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.011 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.011 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:11:49.011 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:07.061891+0000", 2024-03-11T21:11:49.011 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2189999999999999e-05 2024-03-11T21:11:49.011 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.011 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.011 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:11:49.011 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:07.520795+0000", 2024-03-11T21:11:49.011 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.45890372299999999 2024-03-11T21:11:49.011 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.011 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.011 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:11:49.011 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:07.520860+0000", 2024-03-11T21:11:49.012 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4961000000000004e-05 2024-03-11T21:11:49.012 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:11:49.012 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:11:49.012 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:11:49.012 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.012 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.012 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.15081.0:143675 9.4 e284/251 9:26a5279f:::benchmark_data_smithi023_106206_object32529:head v 284'9030, mlcod=284'9027)", 2024-03-11T21:11:49.012 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T21:05:07.066093+0000", 2024-03-11T21:11:49.012 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 401.92899724199998, 2024-03-11T21:11:49.012 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.45874078000000001, 2024-03-11T21:11:49.012 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:11:49.012 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:11:49.012 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:11:49.012 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.013 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:11:49.013 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:07.066093+0000", 2024-03-11T21:11:49.013 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:11:49.013 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.013 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.013 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:11:49.013 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:07.066093+0000", 2024-03-11T21:11:49.013 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:11:49.013 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.013 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.013 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:11:49.013 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:07.066094+0000", 2024-03-11T21:11:49.013 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0389999999999999e-06 2024-03-11T21:11:49.013 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.013 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.014 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:11:49.014 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:07.066103+0000", 2024-03-11T21:11:49.014 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.3000000000000007e-06 2024-03-11T21:11:49.014 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.014 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.014 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:11:49.014 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:07.513902+0000", 2024-03-11T21:11:49.014 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.44779908600000001 2024-03-11T21:11:49.014 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.014 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.014 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:11:49.014 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:07.513918+0000", 2024-03-11T21:11:49.014 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5863e-05 2024-03-11T21:11:49.014 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.015 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.015 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:11:49.015 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:07.514008+0000", 2024-03-11T21:11:49.015 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0148000000000005e-05 2024-03-11T21:11:49.015 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.015 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.015 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:11:49.015 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:07.514049+0000", 2024-03-11T21:11:49.015 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1099000000000001e-05 2024-03-11T21:11:49.015 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.015 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.016 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:11:49.016 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:07.524758+0000", 2024-03-11T21:11:49.016 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010709371000000001 2024-03-11T21:11:49.016 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.016 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.016 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:11:49.016 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:07.524833+0000", 2024-03-11T21:11:49.016 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4874000000000002e-05 2024-03-11T21:11:49.016 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:11:49.016 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:11:49.016 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:11:49.016 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.016 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.016 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "MOSDScrubReserve(3.6,QREQUEST,e:284,from:0,reservation_nonce:7)", 2024-03-11T21:11:49.016 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T21:05:07.197735+0000", 2024-03-11T21:11:49.017 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 401.797354676, 2024-03-11T21:11:49.017 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.31858316399999997, 2024-03-11T21:11:49.017 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:11:49.017 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-03-11T21:11:49.017 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:11:49.017 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.017 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:11:49.017 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:07.197735+0000", 2024-03-11T21:11:49.017 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:11:49.017 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.017 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.017 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:11:49.017 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:07.197735+0000", 2024-03-11T21:11:49.017 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:11:49.018 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.018 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.018 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:11:49.018 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:07.197736+0000", 2024-03-11T21:11:49.018 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6200000000000006e-07 2024-03-11T21:11:49.018 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.018 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.018 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:11:49.018 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:07.197743+0000", 2024-03-11T21:11:49.018 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6030000000000001e-06 2024-03-11T21:11:49.018 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.018 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.018 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:11:49.018 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:07.514375+0000", 2024-03-11T21:11:49.019 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.31663247700000002 2024-03-11T21:11:49.019 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.019 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.019 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:11:49.019 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:07.514384+0000", 2024-03-11T21:11:49.019 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4020000000000002e-06 2024-03-11T21:11:49.019 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.019 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.019 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:11:49.019 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:07.516230+0000", 2024-03-11T21:11:49.019 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0018461300000000001 2024-03-11T21:11:49.019 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.019 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.019 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:11:49.019 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:07.516255+0000", 2024-03-11T21:11:49.020 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5469e-05 2024-03-11T21:11:49.020 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.020 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.020 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:11:49.020 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:05:07.516318+0000", 2024-03-11T21:11:49.020 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3120999999999994e-05 2024-03-11T21:11:49.020 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:11:49.020 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:11:49.020 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:11:49.020 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.020 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.020 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.15291.0:145246 10.c e348/306 10:3ff2d9eb:::benchmark_data_smithi023_124944_object36234:head v 348'8909, mlcod=348'8907)", 2024-03-11T21:11:49.020 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T21:09:48.144453+0000", 2024-03-11T21:11:49.020 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 120.850637002, 2024-03-11T21:11:49.020 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.40722984400000001, 2024-03-11T21:11:49.021 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:11:49.021 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:11:49.021 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:11:49.021 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.021 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:11:49.021 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:48.144453+0000", 2024-03-11T21:11:49.021 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:11:49.021 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.021 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.021 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:11:49.021 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:48.144453+0000", 2024-03-11T21:11:49.021 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:11:49.021 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.021 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.021 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:11:49.022 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:48.144453+0000", 2024-03-11T21:11:49.022 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7299999999999996e-07 2024-03-11T21:11:49.022 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.022 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.022 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:11:49.022 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:48.144458+0000", 2024-03-11T21:11:49.022 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.318e-06 2024-03-11T21:11:49.022 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.022 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.022 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:11:49.022 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:48.144466+0000", 2024-03-11T21:11:49.022 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.8189999999999992e-06 2024-03-11T21:11:49.022 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.023 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.023 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:11:49.023 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:48.144470+0000", 2024-03-11T21:11:49.023 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8269999999999997e-06 2024-03-11T21:11:49.023 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.023 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.023 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:11:49.023 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:48.144523+0000", 2024-03-11T21:11:49.023 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2326e-05 2024-03-11T21:11:49.023 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.023 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.023 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:11:49.023 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:48.144541+0000", 2024-03-11T21:11:49.023 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8689e-05 2024-03-11T21:11:49.024 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.024 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.024 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:11:49.024 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:48.551356+0000", 2024-03-11T21:11:49.024 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.40681461499999999 2024-03-11T21:11:49.024 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.024 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.024 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:11:49.024 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:48.551683+0000", 2024-03-11T21:11:49.024 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00032667699999999999 2024-03-11T21:11:49.024 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:11:49.024 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:11:49.024 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:11:49.024 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.024 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.025 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.15291.0:145251 10.6 e348/306 10:6edae7d1:::benchmark_data_smithi023_124944_object36239:head v 348'9126, mlcod=348'9122)", 2024-03-11T21:11:49.025 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T21:09:48.145332+0000", 2024-03-11T21:11:49.025 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 120.849757446, 2024-03-11T21:11:49.025 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.40149501599999998, 2024-03-11T21:11:49.025 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:11:49.025 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:11:49.025 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:11:49.025 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.025 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:11:49.025 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:48.145332+0000", 2024-03-11T21:11:49.025 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:11:49.025 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.025 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.025 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:11:49.025 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:48.145332+0000", 2024-03-11T21:11:49.026 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:11:49.026 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.026 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.026 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:11:49.026 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:48.145333+0000", 2024-03-11T21:11:49.026 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0200000000000001e-07 2024-03-11T21:11:49.026 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.026 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.026 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:11:49.026 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:48.145339+0000", 2024-03-11T21:11:49.026 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5639999999999996e-06 2024-03-11T21:11:49.026 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.026 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.026 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:11:49.026 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:48.519739+0000", 2024-03-11T21:11:49.027 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.37440064099999998 2024-03-11T21:11:49.027 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.027 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.027 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:11:49.027 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:48.519766+0000", 2024-03-11T21:11:49.027 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7073e-05 2024-03-11T21:11:49.027 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.027 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.027 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:11:49.027 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:48.520050+0000", 2024-03-11T21:11:49.027 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000283999 2024-03-11T21:11:49.027 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.027 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.027 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:11:49.028 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:48.520112+0000", 2024-03-11T21:11:49.028 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1346000000000005e-05 2024-03-11T21:11:49.028 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.028 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.028 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:11:49.028 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:48.546763+0000", 2024-03-11T21:11:49.028 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026651463 2024-03-11T21:11:49.028 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.028 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.028 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:11:49.028 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:48.546827+0000", 2024-03-11T21:11:49.028 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4227999999999996e-05 2024-03-11T21:11:49.028 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:11:49.028 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:11:49.028 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:11:49.029 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.029 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.029 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.15291.0:145252 10.2 e348/306 10:45fd85ed:::benchmark_data_smithi023_124944_object36240:head v 348'9154, mlcod=348'9152)", 2024-03-11T21:11:49.029 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T21:09:48.146034+0000", 2024-03-11T21:11:49.029 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 120.84905552399999, 2024-03-11T21:11:49.029 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.38561209400000002, 2024-03-11T21:11:49.029 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:11:49.029 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:11:49.029 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:11:49.029 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.029 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:11:49.029 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:48.146034+0000", 2024-03-11T21:11:49.029 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:11:49.029 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.029 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.030 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:11:49.030 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:48.146034+0000", 2024-03-11T21:11:49.030 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:11:49.030 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.030 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.030 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:11:49.030 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:48.146035+0000", 2024-03-11T21:11:49.030 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0100000000000004e-07 2024-03-11T21:11:49.030 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.030 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.030 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:11:49.030 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:48.146044+0000", 2024-03-11T21:11:49.030 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.7409999999999998e-06 2024-03-11T21:11:49.030 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.031 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.031 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:11:49.031 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:48.519752+0000", 2024-03-11T21:11:49.031 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.37370835600000002 2024-03-11T21:11:49.031 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.031 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.031 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:11:49.031 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:48.519778+0000", 2024-03-11T21:11:49.031 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5953e-05 2024-03-11T21:11:49.031 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.031 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.031 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:11:49.031 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:48.520037+0000", 2024-03-11T21:11:49.031 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000259313 2024-03-11T21:11:49.032 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.032 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.032 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:11:49.032 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:48.520110+0000", 2024-03-11T21:11:49.032 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2612000000000002e-05 2024-03-11T21:11:49.032 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.032 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.032 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:11:49.032 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:48.531382+0000", 2024-03-11T21:11:49.032 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.011271898000000001 2024-03-11T21:11:49.032 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.032 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.032 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:11:49.032 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:48.531646+0000", 2024-03-11T21:11:49.032 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00026452000000000002 2024-03-11T21:11:49.032 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:11:49.033 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:11:49.033 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:11:49.033 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.033 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.033 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.15291.0:145258 10.e e348/306 10:755cc923:::benchmark_data_smithi023_124944_object36246:head v 348'9071, mlcod=348'9069)", 2024-03-11T21:11:49.033 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T21:09:48.156081+0000", 2024-03-11T21:11:49.033 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 120.83900872, 2024-03-11T21:11:49.033 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.38723326600000002, 2024-03-11T21:11:49.033 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:11:49.033 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:11:49.033 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:11:49.033 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.033 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:11:49.033 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:48.156081+0000", 2024-03-11T21:11:49.034 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:11:49.034 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.034 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.034 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:11:49.034 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:48.156081+0000", 2024-03-11T21:11:49.034 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:11:49.034 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.034 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.034 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:11:49.034 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:48.156082+0000", 2024-03-11T21:11:49.034 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.2600000000000001e-07 2024-03-11T21:11:49.034 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.034 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.035 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:11:49.035 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:48.156089+0000", 2024-03-11T21:11:49.035 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8349999999999998e-06 2024-03-11T21:11:49.035 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.035 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.035 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:11:49.035 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:48.519722+0000", 2024-03-11T21:11:49.035 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.36363317899999997 2024-03-11T21:11:49.035 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.035 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.035 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:11:49.035 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:48.519781+0000", 2024-03-11T21:11:49.035 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9129999999999998e-05 2024-03-11T21:11:49.035 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.035 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.036 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:11:49.036 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:48.520287+0000", 2024-03-11T21:11:49.036 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00050546200000000003 2024-03-11T21:11:49.036 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.036 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.036 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:11:49.036 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:48.520328+0000", 2024-03-11T21:11:49.036 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1353999999999997e-05 2024-03-11T21:11:49.036 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.036 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.036 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:11:49.036 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:48.543262+0000", 2024-03-11T21:11:49.036 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022933884000000002 2024-03-11T21:11:49.036 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.037 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.037 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:11:49.037 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:48.543314+0000", 2024-03-11T21:11:49.037 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2596000000000003e-05 2024-03-11T21:11:49.037 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:11:49.037 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:11:49.037 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:11:49.037 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.037 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.037 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.15291.0:151256 10.6 e348/306 10:6d84f87f:::benchmark_data_smithi023_124944_object42244:head v 348'9494, mlcod=348'9492)", 2024-03-11T21:11:49.037 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T21:09:54.970558+0000", 2024-03-11T21:11:49.037 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 114.02453165, 2024-03-11T21:11:49.037 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.54364309899999996, 2024-03-11T21:11:49.037 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:11:49.037 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:11:49.038 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:11:49.038 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.038 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:11:49.038 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.970558+0000", 2024-03-11T21:11:49.038 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:11:49.038 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.038 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.038 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:11:49.038 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.970558+0000", 2024-03-11T21:11:49.038 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:11:49.038 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.038 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.038 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:11:49.038 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.970559+0000", 2024-03-11T21:11:49.039 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.9500000000000001e-07 2024-03-11T21:11:49.039 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.039 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.039 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:11:49.039 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.970567+0000", 2024-03-11T21:11:49.039 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6839999999999996e-06 2024-03-11T21:11:49.039 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.039 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.039 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:11:49.039 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.970580+0000", 2024-03-11T21:11:49.039 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3405e-05 2024-03-11T21:11:49.039 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.039 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.039 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:11:49.039 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.970588+0000", 2024-03-11T21:11:49.040 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5089999999999996e-06 2024-03-11T21:11:49.040 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.040 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.040 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:11:49.040 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.970634+0000", 2024-03-11T21:11:49.040 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5977999999999999e-05 2024-03-11T21:11:49.040 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.040 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.040 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:11:49.040 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.970650+0000", 2024-03-11T21:11:49.040 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6620999999999999e-05 2024-03-11T21:11:49.040 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.040 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.040 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:11:49.040 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.973305+0000", 2024-03-11T21:11:49.041 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002654558 2024-03-11T21:11:49.041 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.041 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.041 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:11:49.041 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:55.514201+0000", 2024-03-11T21:11:49.041 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.540896449 2024-03-11T21:11:49.041 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:11:49.041 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:11:49.041 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:11:49.041 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.041 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.041 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.15291.0:151258 10.4 e348/306 10:27695507:::benchmark_data_smithi023_124944_object42246:head v 348'9381, mlcod=348'9379)", 2024-03-11T21:11:49.041 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T21:09:54.972262+0000", 2024-03-11T21:11:49.041 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 114.022827523, 2024-03-11T21:11:49.041 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.54207450300000004, 2024-03-11T21:11:49.042 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:11:49.042 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:11:49.042 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:11:49.042 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.042 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:11:49.042 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.972262+0000", 2024-03-11T21:11:49.042 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:11:49.042 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.042 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.042 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:11:49.042 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.972262+0000", 2024-03-11T21:11:49.042 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:11:49.042 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.043 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.043 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:11:49.043 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.972263+0000", 2024-03-11T21:11:49.043 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5600000000000005e-07 2024-03-11T21:11:49.043 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.043 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.043 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:11:49.043 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.972269+0000", 2024-03-11T21:11:49.043 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5960000000000003e-06 2024-03-11T21:11:49.043 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.043 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.043 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:11:49.043 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.972279+0000", 2024-03-11T21:11:49.043 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.9410000000000002e-06 2024-03-11T21:11:49.043 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.044 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.044 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:11:49.044 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.972286+0000", 2024-03-11T21:11:49.044 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0670000000000003e-06 2024-03-11T21:11:49.044 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.044 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.044 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:11:49.044 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.972335+0000", 2024-03-11T21:11:49.044 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9332999999999997e-05 2024-03-11T21:11:49.044 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.044 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.044 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:11:49.044 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.972353+0000", 2024-03-11T21:11:49.044 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8490000000000001e-05 2024-03-11T21:11:49.045 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.045 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.045 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:11:49.045 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:55.514213+0000", 2024-03-11T21:11:49.045 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.54185992199999999 2024-03-11T21:11:49.045 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.045 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.045 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:11:49.045 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:55.514337+0000", 2024-03-11T21:11:49.045 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012339800000000001 2024-03-11T21:11:49.045 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:11:49.045 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:11:49.045 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:11:49.045 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.046 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.046 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.15291.0:151257 10.6 e348/306 10:6eba6ab7:::benchmark_data_smithi023_124944_object42245:head v 348'9495, mlcod=348'9492)", 2024-03-11T21:11:49.046 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T21:09:54.973420+0000", 2024-03-11T21:11:49.046 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 114.021670165, 2024-03-11T21:11:49.046 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.54819602899999997, 2024-03-11T21:11:49.046 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:11:49.046 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:11:49.046 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:11:49.046 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.046 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:11:49.046 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.973420+0000", 2024-03-11T21:11:49.046 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:11:49.046 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.046 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.047 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:11:49.047 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.973420+0000", 2024-03-11T21:11:49.047 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:11:49.047 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.047 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.047 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:11:49.047 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.973420+0000", 2024-03-11T21:11:49.047 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2900000000000003e-07 2024-03-11T21:11:49.047 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.047 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.047 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:11:49.047 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.973426+0000", 2024-03-11T21:11:49.047 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5559999999999998e-06 2024-03-11T21:11:49.047 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.047 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.048 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:11:49.048 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:55.514148+0000", 2024-03-11T21:11:49.048 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.54072219600000004 2024-03-11T21:11:49.048 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.048 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.048 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:11:49.048 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:55.514181+0000", 2024-03-11T21:11:49.048 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3158999999999999e-05 2024-03-11T21:11:49.048 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.048 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.048 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:11:49.048 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:55.514297+0000", 2024-03-11T21:11:49.048 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000116185 2024-03-11T21:11:49.048 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.048 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.049 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:11:49.049 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:55.514368+0000", 2024-03-11T21:11:49.049 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1021999999999999e-05 2024-03-11T21:11:49.049 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.049 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.049 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:11:49.049 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:55.521464+0000", 2024-03-11T21:11:49.049 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0070957609999999999 2024-03-11T21:11:49.049 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.049 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.049 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:11:49.049 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:55.521616+0000", 2024-03-11T21:11:49.049 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000151421 2024-03-11T21:11:49.049 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:11:49.050 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:11:49.050 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:11:49.050 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.050 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.050 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.15291.0:151260 10.2 e348/306 10:4870f564:::benchmark_data_smithi023_124944_object42248:head v 348'9542, mlcod=348'9540)", 2024-03-11T21:11:49.050 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T21:09:54.976334+0000", 2024-03-11T21:11:49.050 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 114.018755809, 2024-03-11T21:11:49.050 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.544141129, 2024-03-11T21:11:49.050 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:11:49.050 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:11:49.050 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:11:49.050 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.050 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:11:49.050 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.976334+0000", 2024-03-11T21:11:49.050 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:11:49.051 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.051 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.051 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:11:49.051 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.976334+0000", 2024-03-11T21:11:49.051 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:11:49.051 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.051 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.051 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:11:49.051 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.976335+0000", 2024-03-11T21:11:49.051 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6599999999999996e-07 2024-03-11T21:11:49.051 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.051 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.051 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:11:49.051 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:54.976341+0000", 2024-03-11T21:11:49.051 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.494e-06 2024-03-11T21:11:49.052 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.052 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.052 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:11:49.052 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:55.514628+0000", 2024-03-11T21:11:49.052 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.53828673800000004 2024-03-11T21:11:49.052 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.052 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.052 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:11:49.052 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:55.514645+0000", 2024-03-11T21:11:49.052 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6660999999999999e-05 2024-03-11T21:11:49.052 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.052 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.052 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:11:49.052 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:55.514744+0000", 2024-03-11T21:11:49.052 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.9601999999999999e-05 2024-03-11T21:11:49.052 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.053 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.053 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:11:49.053 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:55.514769+0000", 2024-03-11T21:11:49.053 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4544000000000001e-05 2024-03-11T21:11:49.053 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.053 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.053 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:11:49.053 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:55.520385+0000", 2024-03-11T21:11:49.053 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0056158639999999999 2024-03-11T21:11:49.053 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.053 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.053 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:11:49.053 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:09:55.520475+0000", 2024-03-11T21:11:49.053 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.056e-05 2024-03-11T21:11:49.053 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:11:49.054 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:11:49.054 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:11:49.054 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.054 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.054 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.15291.0:182244 10.2 e354/306 10:46f10874:::benchmark_data_smithi023_124944_object73232:head v 354'11397, mlcod=353'11395)", 2024-03-11T21:11:49.054 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T21:10:28.518578+0000", 2024-03-11T21:11:49.054 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 80.476511532000004, 2024-03-11T21:11:49.054 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.031995238000000002, 2024-03-11T21:11:49.054 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:11:49.054 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:11:49.054 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:11:49.054 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.054 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:11:49.054 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:10:28.518578+0000", 2024-03-11T21:11:49.055 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:11:49.055 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.055 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.055 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:11:49.055 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:10:28.518578+0000", 2024-03-11T21:11:49.055 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:11:49.055 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.055 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.055 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:11:49.055 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:10:28.518589+0000", 2024-03-11T21:11:49.055 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0688e-05 2024-03-11T21:11:49.055 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.055 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.055 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:11:49.055 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:10:28.518595+0000", 2024-03-11T21:11:49.056 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4699999999999999e-06 2024-03-11T21:11:49.056 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.056 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.056 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:11:49.056 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:10:28.518606+0000", 2024-03-11T21:11:49.056 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0261000000000001e-05 2024-03-11T21:11:49.056 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.056 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.056 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:11:49.056 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:10:28.518612+0000", 2024-03-11T21:11:49.056 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.489e-06 2024-03-11T21:11:49.056 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.056 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.056 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:11:49.056 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:10:28.519150+0000", 2024-03-11T21:11:49.057 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00053803200000000003 2024-03-11T21:11:49.057 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.057 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.057 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:11:49.057 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:10:28.519173+0000", 2024-03-11T21:11:49.057 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2878e-05 2024-03-11T21:11:49.057 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.057 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.057 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:11:49.057 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:10:28.550394+0000", 2024-03-11T21:11:49.057 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.031221187000000001 2024-03-11T21:11:49.057 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.057 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.057 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:11:49.058 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:10:28.550574+0000", 2024-03-11T21:11:49.058 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000179233 2024-03-11T21:11:49.058 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:11:49.058 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:11:49.058 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:11:49.058 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.058 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.058 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.15291.0:184462 10.e e354/306 10:7848fecd:::benchmark_data_smithi023_124944_object75450:head v 354'11513, mlcod=354'11511)", 2024-03-11T21:11:49.058 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T21:10:30.663636+0000", 2024-03-11T21:11:49.058 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 78.331453823000004, 2024-03-11T21:11:49.058 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033833282999999999, 2024-03-11T21:11:49.058 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:11:49.058 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:11:49.058 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:11:49.058 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.059 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:11:49.059 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:10:30.663636+0000", 2024-03-11T21:11:49.059 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:11:49.059 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.059 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.059 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:11:49.059 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:10:30.663636+0000", 2024-03-11T21:11:49.059 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:11:49.059 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.059 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.059 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:11:49.059 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:10:30.663637+0000", 2024-03-11T21:11:49.059 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8899999999999998e-07 2024-03-11T21:11:49.059 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.060 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.060 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:11:49.060 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:10:30.663645+0000", 2024-03-11T21:11:49.060 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.7689999999999996e-06 2024-03-11T21:11:49.060 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.060 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.060 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:11:49.060 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:10:30.663659+0000", 2024-03-11T21:11:49.060 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3895e-05 2024-03-11T21:11:49.060 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.060 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.060 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:11:49.060 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:10:30.663667+0000", 2024-03-11T21:11:49.060 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5310000000000001e-06 2024-03-11T21:11:49.061 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.061 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.061 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:11:49.061 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:10:30.663743+0000", 2024-03-11T21:11:49.061 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5456000000000004e-05 2024-03-11T21:11:49.061 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.061 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.061 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:11:49.061 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:10:30.663759+0000", 2024-03-11T21:11:49.061 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6030000000000001e-05 2024-03-11T21:11:49.061 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.061 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.061 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:11:49.061 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:10:30.697387+0000", 2024-03-11T21:11:49.061 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033628225999999997 2024-03-11T21:11:49.062 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.062 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.062 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:11:49.062 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:10:30.697469+0000", 2024-03-11T21:11:49.062 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.2386999999999996e-05 2024-03-11T21:11:49.062 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:11:49.062 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:11:49.062 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:11:49.062 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.062 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.062 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.15291.0:187324 10.e e355/306 10:7f75802c:::benchmark_data_smithi023_124944_object78312:head v 355'11689, mlcod=355'11686)", 2024-03-11T21:11:49.062 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T21:10:33.473295+0000", 2024-03-11T21:11:49.062 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 75.521795212000001, 2024-03-11T21:11:49.062 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.032440625000000001, 2024-03-11T21:11:49.063 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:11:49.063 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:11:49.063 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:11:49.063 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.063 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:11:49.063 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:10:33.473295+0000", 2024-03-11T21:11:49.063 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:11:49.063 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.063 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.063 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:11:49.063 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:10:33.473295+0000", 2024-03-11T21:11:49.063 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:11:49.063 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.063 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.063 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:11:49.064 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:10:33.473295+0000", 2024-03-11T21:11:49.064 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3099999999999999e-07 2024-03-11T21:11:49.064 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.064 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.064 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:11:49.064 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:10:33.473300+0000", 2024-03-11T21:11:49.064 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9259999999999999e-06 2024-03-11T21:11:49.064 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.064 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.064 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:11:49.064 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:10:33.473306+0000", 2024-03-11T21:11:49.064 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2829999999999998e-06 2024-03-11T21:11:49.064 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.064 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.065 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:11:49.065 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:10:33.473311+0000", 2024-03-11T21:11:49.065 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9509999999999997e-06 2024-03-11T21:11:49.065 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.065 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.065 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:11:49.065 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:10:33.475872+0000", 2024-03-11T21:11:49.065 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002561341 2024-03-11T21:11:49.065 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.065 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.065 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:11:49.065 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:10:33.475949+0000", 2024-03-11T21:11:49.065 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6604e-05 2024-03-11T21:11:49.065 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.066 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.066 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:11:49.066 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:10:33.505542+0000", 2024-03-11T21:11:49.066 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029593181 2024-03-11T21:11:49.066 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.066 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.066 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:11:49.066 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:10:33.505735+0000", 2024-03-11T21:11:49.066 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019300799999999999 2024-03-11T21:11:49.066 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:11:49.066 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:11:49.066 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:11:49.066 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.066 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.066 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.15291.0:217376 10.e e362/306 10:7e84c8fa:::benchmark_data_smithi023_124944_object108364:head v 362'13595, mlcod=362'13589)", 2024-03-11T21:11:49.067 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-03-11T21:11:08.054928+0000", 2024-03-11T21:11:49.067 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 40.940161580000002, 2024-03-11T21:11:49.067 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033719027999999998, 2024-03-11T21:11:49.067 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-03-11T21:11:49.067 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:11:49.067 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-03-11T21:11:49.067 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.067 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-03-11T21:11:49.067 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:11:08.054928+0000", 2024-03-11T21:11:49.067 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:11:49.067 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.067 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.067 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-03-11T21:11:49.067 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:11:08.054928+0000", 2024-03-11T21:11:49.067 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-03-11T21:11:49.068 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.068 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.068 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-03-11T21:11:49.068 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:11:08.054929+0000", 2024-03-11T21:11:49.068 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8100000000000001e-07 2024-03-11T21:11:49.068 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.068 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.068 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-03-11T21:11:49.068 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:11:08.054934+0000", 2024-03-11T21:11:49.068 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0529999999999997e-06 2024-03-11T21:11:49.068 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.068 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.068 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-03-11T21:11:49.068 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:11:08.054945+0000", 2024-03-11T21:11:49.069 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1228e-05 2024-03-11T21:11:49.069 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.069 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.069 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-03-11T21:11:49.069 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:11:08.054952+0000", 2024-03-11T21:11:49.069 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8000000000000001e-06 2024-03-11T21:11:49.069 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.069 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.069 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-03-11T21:11:49.069 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:11:08.060346+0000", 2024-03-11T21:11:49.069 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0053939039999999997 2024-03-11T21:11:49.069 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.069 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.069 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-03-11T21:11:49.069 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:11:08.060363+0000", 2024-03-11T21:11:49.070 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6935e-05 2024-03-11T21:11:49.070 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.070 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.070 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-03-11T21:11:49.070 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:11:08.088601+0000", 2024-03-11T21:11:49.070 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.028237762 2024-03-11T21:11:49.070 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-11T21:11:49.070 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-03-11T21:11:49.070 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-03-11T21:11:49.070 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-03-11T21:11:08.088647+0000", 2024-03-11T21:11:49.070 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6365000000000001e-05 2024-03-11T21:11:49.070 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:11:49.070 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:11:49.070 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:11:49.070 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-11T21:11:49.071 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-03-11T21:11:49.071 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-03-11T21:11:49.071 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:49.091 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:49.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:49.093 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:11:49.193 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:11:49.222 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:11:49.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:49.225 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:11:49.326 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:11:49.358 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:11:49.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:49.361 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:11:49.387 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd pool application enable unique_pool_4 rados --yes-i-really-mean-it || true 2024-03-11T21:11:49.463 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:11:49.490 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:11:49.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:49.493 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:11:49.593 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:11:49.614 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:11:49.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:49.616 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:11:49.717 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:11:49.745 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:11:49.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:49.748 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:11:49.757 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:49 smithi146 bash[23809]: cluster 2024-03-11T21:11:48.356831+0000 mgr.y (mgr.24380) 778 : cluster 0 pgmap v890: 158 pgs: 3 creating+peering, 13 unknown, 142 active+clean; 582 KiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:11:49.757 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:49 smithi146 bash[23809]: cluster 2024-03-11T21:11:48.550009+0000 mon.a (mon.0) 1794 : cluster 0 osdmap e382: 8 total, 8 up, 8 in 2024-03-11T21:11:49.757 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:49 smithi146 bash[23809]: cluster 2024-03-11T21:11:48.570459+0000 mon.a (mon.0) 1795 : cluster 3 Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:11:49.848 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:49.852 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-11T21:11:49.858 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:49 smithi023 bash[21372]: cluster 2024-03-11T21:11:48.356831+0000 mgr.y (mgr.24380) 778 : cluster 0 pgmap v890: 158 pgs: 3 creating+peering, 13 unknown, 142 active+clean; 582 KiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:11:49.858 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:49 smithi023 bash[21372]: cluster 2024-03-11T21:11:48.550009+0000 mon.a (mon.0) 1794 : cluster 0 osdmap e382: 8 total, 8 up, 8 in 2024-03-11T21:11:49.858 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:49 smithi023 bash[21372]: cluster 2024-03-11T21:11:48.570459+0000 mon.a (mon.0) 1795 : cluster 3 Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:11:49.858 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:49 smithi023 bash[29146]: cluster 2024-03-11T21:11:48.356831+0000 mgr.y (mgr.24380) 778 : cluster 0 pgmap v890: 158 pgs: 3 creating+peering, 13 unknown, 142 active+clean; 582 KiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:11:49.859 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:49 smithi023 bash[29146]: cluster 2024-03-11T21:11:48.550009+0000 mon.a (mon.0) 1794 : cluster 0 osdmap e382: 8 total, 8 up, 8 in 2024-03-11T21:11:49.859 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:49 smithi023 bash[29146]: cluster 2024-03-11T21:11:48.570459+0000 mon.a (mon.0) 1795 : cluster 3 Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:11:49.863 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:11:49.873 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:49.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:49.876 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:11:49.977 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:11:50.003 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:11:50.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:50.005 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:11:50.106 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:11:50.134 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:11:50.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:50.136 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:11:50.237 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:11:50.266 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:11:50.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:50.268 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:11:50.369 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:50.397 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:50.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:50.405 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:11:50.506 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:11:50.533 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:11:50.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:50.535 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:11:50.636 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:11:50.663 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:11:50.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:50.665 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:11:50.766 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:11:50.794 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:11:50.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:50.796 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:11:50.896 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:11:50.922 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:11:50.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:50.924 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:11:51.025 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:11:51.055 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:11:51.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:51.057 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:11:51.157 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:51.186 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:51.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:51.188 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:11:51.289 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:51.318 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:51.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:51.320 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:11:51.421 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:51.433 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:51 smithi146 bash[23809]: cluster 2024-03-11T21:11:50.355923+0000 mon.a (mon.0) 1796 : cluster 0 osdmap e383: 8 total, 8 up, 8 in 2024-03-11T21:11:51.434 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:51 smithi146 bash[23809]: cluster 2024-03-11T21:11:50.357814+0000 mgr.y (mgr.24380) 779 : cluster 0 pgmap v893: 158 pgs: 3 creating+peering, 2 unknown, 153 active+clean; 582 KiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:11:51.434 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:51 smithi146 bash[23809]: audit 2024-03-11T21:11:51.121208+0000 mon.a (mon.0) 1797 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:11:51.434 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:51 smithi146 bash[23809]: audit 2024-03-11T21:11:51.122560+0000 mon.a (mon.0) 1798 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:11:51.449 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:51.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:51.452 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:11:51.552 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:11:51.579 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:11:51.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:51.582 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:11:51.649 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:51 smithi023 bash[29146]: cluster 2024-03-11T21:11:50.355923+0000 mon.a (mon.0) 1796 : cluster 0 osdmap e383: 8 total, 8 up, 8 in 2024-03-11T21:11:51.649 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:51 smithi023 bash[29146]: cluster 2024-03-11T21:11:50.357814+0000 mgr.y (mgr.24380) 779 : cluster 0 pgmap v893: 158 pgs: 3 creating+peering, 2 unknown, 153 active+clean; 582 KiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:11:51.650 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:51 smithi023 bash[29146]: audit 2024-03-11T21:11:51.121208+0000 mon.a (mon.0) 1797 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:11:51.650 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:51 smithi023 bash[29146]: audit 2024-03-11T21:11:51.122560+0000 mon.a (mon.0) 1798 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:11:51.650 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:51 smithi023 bash[21372]: cluster 2024-03-11T21:11:50.355923+0000 mon.a (mon.0) 1796 : cluster 0 osdmap e383: 8 total, 8 up, 8 in 2024-03-11T21:11:51.650 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:51 smithi023 bash[21372]: cluster 2024-03-11T21:11:50.357814+0000 mgr.y (mgr.24380) 779 : cluster 0 pgmap v893: 158 pgs: 3 creating+peering, 2 unknown, 153 active+clean; 582 KiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:11:51.650 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:51 smithi023 bash[21372]: audit 2024-03-11T21:11:51.121208+0000 mon.a (mon.0) 1797 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:11:51.650 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:51 smithi023 bash[21372]: audit 2024-03-11T21:11:51.122560+0000 mon.a (mon.0) 1798 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:11:51.684 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:51.710 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:51.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:51.712 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:11:51.813 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:51.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:51.843 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:51.843 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:11:51.939 INFO:teuthology.orchestra.run.smithi023.stdout:osd.0: {} 2024-03-11T21:11:51.944 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:51.961 INFO:teuthology.orchestra.run.smithi023.stdout:osd.1: {} 2024-03-11T21:11:51.967 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:51.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:51.970 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:11:51.986 INFO:teuthology.orchestra.run.smithi023.stdout:osd.2: {} 2024-03-11T21:11:52.020 INFO:teuthology.orchestra.run.smithi023.stdout:osd.3: {} 2024-03-11T21:11:52.046 INFO:teuthology.orchestra.run.smithi023.stdout:osd.4: {} 2024-03-11T21:11:52.070 INFO:teuthology.orchestra.run.smithi023.stdout:osd.5: {} 2024-03-11T21:11:52.071 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:11:52.089 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:11:52.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:52.091 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:11:52.095 INFO:teuthology.orchestra.run.smithi023.stdout:osd.6: {} 2024-03-11T21:11:52.121 INFO:teuthology.orchestra.run.smithi023.stdout:osd.7: {} 2024-03-11T21:11:52.122 INFO:teuthology.orchestra.run.smithi023.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:11:52.122 INFO:teuthology.orchestra.run.smithi023.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:11:52.122 INFO:teuthology.orchestra.run.smithi023.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:11:52.122 INFO:teuthology.orchestra.run.smithi023.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:11:52.122 INFO:teuthology.orchestra.run.smithi023.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:11:52.122 INFO:teuthology.orchestra.run.smithi023.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:11:52.122 INFO:teuthology.orchestra.run.smithi023.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:11:52.123 INFO:teuthology.orchestra.run.smithi023.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:11:52.193 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:11:52.211 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:11:52.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:52.213 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:11:52.314 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:52.339 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:52.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:52.341 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:11:52.379 INFO:teuthology.orchestra.run.smithi023.stderr:noscrub is unset 2024-03-11T21:11:52.443 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:11:52.461 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:11:52.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:52.463 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:11:52.470 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:52 smithi023 bash[21372]: audit 2024-03-11T21:11:51.924533+0000 mon.c (mon.2) 90 : audit 0 from='client.? 172.21.15.23:0/3293145645' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:11:52.470 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:52 smithi023 bash[21372]: audit 2024-03-11T21:11:52.081945+0000 mon.c (mon.2) 91 : audit 1 from='client.? 172.21.15.23:0/2336391519' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:11:52.471 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:52 smithi023 bash[21372]: audit 2024-03-11T21:11:52.082236+0000 mon.a (mon.0) 1799 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:11:52.471 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:52 smithi023 bash[29146]: audit 2024-03-11T21:11:51.924533+0000 mon.c (mon.2) 90 : audit 0 from='client.? 172.21.15.23:0/3293145645' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:11:52.471 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:52 smithi023 bash[29146]: audit 2024-03-11T21:11:52.081945+0000 mon.c (mon.2) 91 : audit 1 from='client.? 172.21.15.23:0/2336391519' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:11:52.471 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:52 smithi023 bash[29146]: audit 2024-03-11T21:11:52.082236+0000 mon.a (mon.0) 1799 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:11:52.564 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:11:52.584 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:11:52.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:52.586 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:11:52.687 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:11:52.699 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:52 smithi146 bash[23809]: audit 2024-03-11T21:11:51.924533+0000 mon.c (mon.2) 90 : audit 0 from='client.? 172.21.15.23:0/3293145645' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:11:52.699 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:52 smithi146 bash[23809]: audit 2024-03-11T21:11:52.081945+0000 mon.c (mon.2) 91 : audit 1 from='client.? 172.21.15.23:0/2336391519' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:11:52.699 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:52 smithi146 bash[23809]: audit 2024-03-11T21:11:52.082236+0000 mon.a (mon.0) 1799 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:11:52.715 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:11:52.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:52.718 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:11:52.818 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:11:52.846 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:11:52.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:52.849 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:11:52.950 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:11:52.978 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:11:52.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:52.980 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:11:53.081 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:53.109 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:53.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:53.112 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:11:53.212 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:11:53.242 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:11:53.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:53.245 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:11:53.345 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:53.374 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:53.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:53.376 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:11:53.403 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:53 smithi023 bash[21372]: cluster 2024-03-11T21:11:52.358854+0000 mgr.y (mgr.24380) 780 : cluster 0 pgmap v894: 158 pgs: 158 active+clean; 582 KiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 880 B/s rd, 0 op/s 2024-03-11T21:11:53.403 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:53 smithi023 bash[21372]: cluster 2024-03-11T21:11:52.363344+0000 mon.a (mon.0) 1800 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:11:53.477 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:53.504 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:53.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:53.507 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:11:53.527 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:53 smithi023 bash[21372]: audit 2024-03-11T21:11:52.375881+0000 mon.a (mon.0) 1801 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:11:53.527 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:53 smithi023 bash[21372]: cluster 2024-03-11T21:11:52.384411+0000 mon.a (mon.0) 1802 : cluster 0 osdmap e384: 8 total, 8 up, 8 in 2024-03-11T21:11:53.527 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:53 smithi023 bash[29146]: cluster 2024-03-11T21:11:52.358854+0000 mgr.y (mgr.24380) 780 : cluster 0 pgmap v894: 158 pgs: 158 active+clean; 582 KiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 880 B/s rd, 0 op/s 2024-03-11T21:11:53.528 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:53 smithi023 bash[29146]: cluster 2024-03-11T21:11:52.363344+0000 mon.a (mon.0) 1800 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:11:53.528 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:53 smithi023 bash[29146]: audit 2024-03-11T21:11:52.375881+0000 mon.a (mon.0) 1801 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:11:53.528 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:53 smithi023 bash[29146]: cluster 2024-03-11T21:11:52.384411+0000 mon.a (mon.0) 1802 : cluster 0 osdmap e384: 8 total, 8 up, 8 in 2024-03-11T21:11:53.607 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:11:53.619 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:53 smithi146 bash[23809]: cluster 2024-03-11T21:11:52.358854+0000 mgr.y (mgr.24380) 780 : cluster 0 pgmap v894: 158 pgs: 158 active+clean; 582 KiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 880 B/s rd, 0 op/s 2024-03-11T21:11:53.619 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:53 smithi146 bash[23809]: cluster 2024-03-11T21:11:52.363344+0000 mon.a (mon.0) 1800 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:11:53.619 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:53 smithi146 bash[23809]: audit 2024-03-11T21:11:52.375881+0000 mon.a (mon.0) 1801 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:11:53.619 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:53 smithi146 bash[23809]: cluster 2024-03-11T21:11:52.384411+0000 mon.a (mon.0) 1802 : cluster 0 osdmap e384: 8 total, 8 up, 8 in 2024-03-11T21:11:53.633 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:11:53.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:53.635 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:11:53.736 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:11:53.763 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:11:53.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:53.765 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:11:53.867 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:53.894 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:53.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:53.898 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:11:53.999 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:11:54.026 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:11:54.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:54.029 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:11:54.129 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:11:54.158 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:11:54.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:54.160 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:11:54.261 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:11:54.291 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:11:54.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:54.293 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:11:54.394 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:54.405 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:54 smithi146 bash[23809]: cluster 2024-03-11T21:11:53.951913+0000 osd.6 (osd.6) 362 : cluster 0 2.14 scrub starts 2024-03-11T21:11:54.405 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:54 smithi146 bash[23809]: cluster 2024-03-11T21:11:53.953489+0000 osd.6 (osd.6) 363 : cluster 0 2.14 scrub ok 2024-03-11T21:11:54.422 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:54.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:54.424 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:11:54.525 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:11:54.527 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:54 smithi023 bash[21372]: cluster 2024-03-11T21:11:53.951913+0000 osd.6 (osd.6) 362 : cluster 0 2.14 scrub starts 2024-03-11T21:11:54.527 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:54 smithi023 bash[21372]: cluster 2024-03-11T21:11:53.953489+0000 osd.6 (osd.6) 363 : cluster 0 2.14 scrub ok 2024-03-11T21:11:54.527 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:54 smithi023 bash[29146]: cluster 2024-03-11T21:11:53.951913+0000 osd.6 (osd.6) 362 : cluster 0 2.14 scrub starts 2024-03-11T21:11:54.527 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:54 smithi023 bash[29146]: cluster 2024-03-11T21:11:53.953489+0000 osd.6 (osd.6) 363 : cluster 0 2.14 scrub ok 2024-03-11T21:11:54.554 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:11:54.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:54.557 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:11:54.658 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:11:54.688 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:11:54.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:54.690 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:11:54.791 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:54.817 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:54.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:54.819 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:11:54.920 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:11:54.946 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:11:54.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:54.949 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:11:55.049 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:11:55.078 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:11:55.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:55.081 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:11:55.182 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:11:55.210 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:11:55.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:55.212 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:11:55.313 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:55.340 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:55.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:55.343 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:11:55.444 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:55.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:55 smithi146 bash[23809]: cluster 2024-03-11T21:11:53.944988+0000 osd.3 (osd.3) 428 : cluster 0 3.2 scrub starts 2024-03-11T21:11:55.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:55 smithi146 bash[23809]: cluster 2024-03-11T21:11:53.948808+0000 osd.3 (osd.3) 429 : cluster 0 3.2 scrub ok 2024-03-11T21:11:55.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:55 smithi146 bash[23809]: cluster 2024-03-11T21:11:54.359424+0000 mgr.y (mgr.24380) 781 : cluster 0 pgmap v896: 158 pgs: 158 active+clean; 582 KiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:11:55.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:55 smithi146 bash[23809]: cluster 2024-03-11T21:11:54.791956+0000 osd.1 (osd.1) 255 : cluster 0 6.4 scrub starts 2024-03-11T21:11:55.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:55 smithi146 bash[23809]: cluster 2024-03-11T21:11:54.793796+0000 osd.1 (osd.1) 256 : cluster 0 6.4 scrub ok 2024-03-11T21:11:55.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:55 smithi146 bash[23809]: cluster 2024-03-11T21:11:54.927640+0000 osd.6 (osd.6) 364 : cluster 0 4.5 scrub starts 2024-03-11T21:11:55.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:55 smithi146 bash[23809]: cluster 2024-03-11T21:11:54.930980+0000 osd.6 (osd.6) 365 : cluster 0 4.5 scrub ok 2024-03-11T21:11:55.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:55 smithi146 bash[23809]: cluster 2024-03-11T21:11:55.363416+0000 mon.a (mon.0) 1803 : cluster 0 osdmap e385: 8 total, 8 up, 8 in 2024-03-11T21:11:55.471 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:55.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:55.473 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:11:55.574 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:55.601 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:55.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:55.603 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:11:55.704 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:55.716 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:55 smithi023 bash[29146]: cluster 2024-03-11T21:11:53.944988+0000 osd.3 (osd.3) 428 : cluster 0 3.2 scrub starts 2024-03-11T21:11:55.716 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:55 smithi023 bash[29146]: cluster 2024-03-11T21:11:53.948808+0000 osd.3 (osd.3) 429 : cluster 0 3.2 scrub ok 2024-03-11T21:11:55.716 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:55 smithi023 bash[29146]: cluster 2024-03-11T21:11:54.359424+0000 mgr.y (mgr.24380) 781 : cluster 0 pgmap v896: 158 pgs: 158 active+clean; 582 KiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:11:55.716 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:55 smithi023 bash[29146]: cluster 2024-03-11T21:11:54.791956+0000 osd.1 (osd.1) 255 : cluster 0 6.4 scrub starts 2024-03-11T21:11:55.716 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:55 smithi023 bash[29146]: cluster 2024-03-11T21:11:54.793796+0000 osd.1 (osd.1) 256 : cluster 0 6.4 scrub ok 2024-03-11T21:11:55.716 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:55 smithi023 bash[29146]: cluster 2024-03-11T21:11:54.927640+0000 osd.6 (osd.6) 364 : cluster 0 4.5 scrub starts 2024-03-11T21:11:55.716 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:55 smithi023 bash[29146]: cluster 2024-03-11T21:11:54.930980+0000 osd.6 (osd.6) 365 : cluster 0 4.5 scrub ok 2024-03-11T21:11:55.717 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:55 smithi023 bash[29146]: cluster 2024-03-11T21:11:55.363416+0000 mon.a (mon.0) 1803 : cluster 0 osdmap e385: 8 total, 8 up, 8 in 2024-03-11T21:11:55.717 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:55 smithi023 bash[21372]: cluster 2024-03-11T21:11:53.944988+0000 osd.3 (osd.3) 428 : cluster 0 3.2 scrub starts 2024-03-11T21:11:55.717 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:55 smithi023 bash[21372]: cluster 2024-03-11T21:11:53.948808+0000 osd.3 (osd.3) 429 : cluster 0 3.2 scrub ok 2024-03-11T21:11:55.717 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:55 smithi023 bash[21372]: cluster 2024-03-11T21:11:54.359424+0000 mgr.y (mgr.24380) 781 : cluster 0 pgmap v896: 158 pgs: 158 active+clean; 582 KiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:11:55.717 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:55 smithi023 bash[21372]: cluster 2024-03-11T21:11:54.791956+0000 osd.1 (osd.1) 255 : cluster 0 6.4 scrub starts 2024-03-11T21:11:55.718 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:55 smithi023 bash[21372]: cluster 2024-03-11T21:11:54.793796+0000 osd.1 (osd.1) 256 : cluster 0 6.4 scrub ok 2024-03-11T21:11:55.718 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:55 smithi023 bash[21372]: cluster 2024-03-11T21:11:54.927640+0000 osd.6 (osd.6) 364 : cluster 0 4.5 scrub starts 2024-03-11T21:11:55.718 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:55 smithi023 bash[21372]: cluster 2024-03-11T21:11:54.930980+0000 osd.6 (osd.6) 365 : cluster 0 4.5 scrub ok 2024-03-11T21:11:55.718 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:55 smithi023 bash[21372]: cluster 2024-03-11T21:11:55.363416+0000 mon.a (mon.0) 1803 : cluster 0 osdmap e385: 8 total, 8 up, 8 in 2024-03-11T21:11:55.734 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:55.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:55.738 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:11:55.839 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:11:55.861 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:11:55.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:55.863 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:11:55.964 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:11:55.991 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:11:55.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:55.993 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:11:56.094 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:11:56.119 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:11:56.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:56.121 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:11:56.222 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:11:56.243 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:11:56.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:56.246 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:11:56.257 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:11:55 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:11:56.268 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-11T21:11:56.271 INFO:teuthology.orchestra.run.smithi023.stderr:dumped all 2024-03-11T21:11:56.346 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:11:56.367 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:11:56.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:56.369 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:11:56.470 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:56.498 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:56.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:56.501 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:11:56.602 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:11:56.630 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:11:56.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:56.632 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:11:56.733 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:11:56.745 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:56 smithi023 bash[21372]: cluster 2024-03-11T21:11:54.930320+0000 osd.3 (osd.3) 430 : cluster 0 4.19 scrub starts 2024-03-11T21:11:56.745 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:56 smithi023 bash[21372]: cluster 2024-03-11T21:11:54.932669+0000 osd.3 (osd.3) 431 : cluster 0 4.19 scrub ok 2024-03-11T21:11:56.745 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:56 smithi023 bash[21372]: cluster 2024-03-11T21:11:55.037818+0000 osd.7 (osd.7) 489 : cluster 0 1.0 scrub starts 2024-03-11T21:11:56.745 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:56 smithi023 bash[21372]: cluster 2024-03-11T21:11:55.042688+0000 osd.7 (osd.7) 490 : cluster 0 1.0 scrub ok 2024-03-11T21:11:56.745 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:56 smithi023 bash[21372]: cluster 2024-03-11T21:11:55.405067+0000 osd.5 (osd.5) 441 : cluster 0 3.5 scrub starts 2024-03-11T21:11:56.746 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:56 smithi023 bash[21372]: cluster 2024-03-11T21:11:55.408615+0000 osd.5 (osd.5) 442 : cluster 0 3.5 scrub ok 2024-03-11T21:11:56.746 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:56 smithi023 bash[21372]: cluster 2024-03-11T21:11:55.752794+0000 osd.1 (osd.1) 257 : cluster 0 1.3 scrub starts 2024-03-11T21:11:56.746 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:56 smithi023 bash[21372]: cluster 2024-03-11T21:11:55.755814+0000 osd.1 (osd.1) 258 : cluster 0 1.3 scrub ok 2024-03-11T21:11:56.746 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:56 smithi023 bash[21372]: cluster 2024-03-11T21:11:55.977164+0000 osd.6 (osd.6) 366 : cluster 0 1.7 scrub starts 2024-03-11T21:11:56.746 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:56 smithi023 bash[21372]: cluster 2024-03-11T21:11:55.979603+0000 osd.6 (osd.6) 367 : cluster 0 1.7 scrub ok 2024-03-11T21:11:56.746 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:56 smithi023 bash[29146]: cluster 2024-03-11T21:11:54.930320+0000 osd.3 (osd.3) 430 : cluster 0 4.19 scrub starts 2024-03-11T21:11:56.747 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:56 smithi023 bash[29146]: cluster 2024-03-11T21:11:54.932669+0000 osd.3 (osd.3) 431 : cluster 0 4.19 scrub ok 2024-03-11T21:11:56.747 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:56 smithi023 bash[29146]: cluster 2024-03-11T21:11:55.037818+0000 osd.7 (osd.7) 489 : cluster 0 1.0 scrub starts 2024-03-11T21:11:56.747 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:56 smithi023 bash[29146]: cluster 2024-03-11T21:11:55.042688+0000 osd.7 (osd.7) 490 : cluster 0 1.0 scrub ok 2024-03-11T21:11:56.747 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:56 smithi023 bash[29146]: cluster 2024-03-11T21:11:55.405067+0000 osd.5 (osd.5) 441 : cluster 0 3.5 scrub starts 2024-03-11T21:11:56.747 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:56 smithi023 bash[29146]: cluster 2024-03-11T21:11:55.408615+0000 osd.5 (osd.5) 442 : cluster 0 3.5 scrub ok 2024-03-11T21:11:56.747 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:56 smithi023 bash[29146]: cluster 2024-03-11T21:11:55.752794+0000 osd.1 (osd.1) 257 : cluster 0 1.3 scrub starts 2024-03-11T21:11:56.747 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:56 smithi023 bash[29146]: cluster 2024-03-11T21:11:55.755814+0000 osd.1 (osd.1) 258 : cluster 0 1.3 scrub ok 2024-03-11T21:11:56.747 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:56 smithi023 bash[29146]: cluster 2024-03-11T21:11:55.977164+0000 osd.6 (osd.6) 366 : cluster 0 1.7 scrub starts 2024-03-11T21:11:56.747 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:56 smithi023 bash[29146]: cluster 2024-03-11T21:11:55.979603+0000 osd.6 (osd.6) 367 : cluster 0 1.7 scrub ok 2024-03-11T21:11:56.757 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:56 smithi146 bash[23809]: cluster 2024-03-11T21:11:54.930320+0000 osd.3 (osd.3) 430 : cluster 0 4.19 scrub starts 2024-03-11T21:11:56.757 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:56 smithi146 bash[23809]: cluster 2024-03-11T21:11:54.932669+0000 osd.3 (osd.3) 431 : cluster 0 4.19 scrub ok 2024-03-11T21:11:56.757 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:56 smithi146 bash[23809]: cluster 2024-03-11T21:11:55.037818+0000 osd.7 (osd.7) 489 : cluster 0 1.0 scrub starts 2024-03-11T21:11:56.757 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:56 smithi146 bash[23809]: cluster 2024-03-11T21:11:55.042688+0000 osd.7 (osd.7) 490 : cluster 0 1.0 scrub ok 2024-03-11T21:11:56.757 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:56 smithi146 bash[23809]: cluster 2024-03-11T21:11:55.405067+0000 osd.5 (osd.5) 441 : cluster 0 3.5 scrub starts 2024-03-11T21:11:56.757 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:56 smithi146 bash[23809]: cluster 2024-03-11T21:11:55.408615+0000 osd.5 (osd.5) 442 : cluster 0 3.5 scrub ok 2024-03-11T21:11:56.758 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:56 smithi146 bash[23809]: cluster 2024-03-11T21:11:55.752794+0000 osd.1 (osd.1) 257 : cluster 0 1.3 scrub starts 2024-03-11T21:11:56.758 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:56 smithi146 bash[23809]: cluster 2024-03-11T21:11:55.755814+0000 osd.1 (osd.1) 258 : cluster 0 1.3 scrub ok 2024-03-11T21:11:56.758 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:56 smithi146 bash[23809]: cluster 2024-03-11T21:11:55.977164+0000 osd.6 (osd.6) 366 : cluster 0 1.7 scrub starts 2024-03-11T21:11:56.758 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:56 smithi146 bash[23809]: cluster 2024-03-11T21:11:55.979603+0000 osd.6 (osd.6) 367 : cluster 0 1.7 scrub ok 2024-03-11T21:11:56.762 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:11:56.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:56.767 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:11:56.868 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:11:56.896 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:11:56.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:56.898 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:11:56.999 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:57.025 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:57.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:57.028 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:11:57.129 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:57.156 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:57.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:57.158 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:11:57.229 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T21:11:57.229 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-11T21:11:57.259 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:11:57.287 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:11:57.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:57.289 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:11:57.292 INFO:teuthology.orchestra.run.smithi023.stdout:{"pg_ready":true,"pg_map":{"version":897,"stamp":"2024-03-11T21:11:55.357385+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596392,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":2933,"num_read_kb":2779,"num_write":1187,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":192,"num_bytes_recovered":2366300,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2644,"ondisk_log_size":2644,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":3471204,"kb_used_data":9844,"kb_used_omap":21,"kb_used_meta":3461226,"kb_avail":746490012,"statfs":{"total":767960285184,"available":764405772288,"internally_reserved":0,"allocated":10080256,"data_stored":6859018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22089,"internal_metadata":3544295863},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.810694"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":684,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:11:52.382662+0000","last_change":"2024-03-11T21:11:15.221894+0000","last_active":"2024-03-11T21:11:52.382662+0000","last_peered":"2024-03-11T21:11:52.382662+0000","last_clean":"2024-03-11T21:11:52.382662+0000","last_became_active":"2024-03-11T20:50:34.338390+0000","last_became_peered":"2024-03-11T20:50:34.338390+0000","last_unstale":"2024-03-11T21:11:52.382662+0000","last_undegraded":"2024-03-11T21:11:52.382662+0000","last_fullsized":"2024-03-11T21:11:52.382662+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:11:15.221783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:11:15.221783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:12:25.845909+0000","scrub_duration":127,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":688,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:11:54.932824+0000","last_change":"2024-03-11T21:11:54.932879+0000","last_active":"2024-03-11T21:11:54.932824+0000","last_peered":"2024-03-11T21:11:54.932824+0000","last_clean":"2024-03-11T21:11:54.932824+0000","last_became_active":"2024-03-11T20:50:32.210194+0000","last_became_peered":"2024-03-11T20:50:32.210194+0000","last_unstale":"2024-03-11T21:11:54.932824+0000","last_undegraded":"2024-03-11T21:11:54.932824+0000","last_fullsized":"2024-03-11T21:11:54.932824+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:11:54.932679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:11:54.932679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:13:23.945079+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":396,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-03-11T21:11:48.552558+0000","last_change":"2024-03-11T21:11:05.885010+0000","last_active":"2024-03-11T21:11:48.552558+0000","last_peered":"2024-03-11T21:11:48.552558+0000","last_clean":"2024-03-11T21:11:48.552558+0000","last_became_active":"2024-03-11T21:06:28.760306+0000","last_became_peered":"2024-03-11T21:06:28.760306+0000","last_unstale":"2024-03-11T21:11:48.552558+0000","last_undegraded":"2024-03-11T21:11:48.552558+0000","last_fullsized":"2024-03-11T21:11:48.552558+0000","mapping_epoch":297,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:11:05.884900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:11:05.884900+0000","last_clean_scrub_stamp":"2024-03-11T21:11:05.884900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:12:19.812750+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"358'30","reported_seq":740,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:11:52.384567+0000","last_change":"2024-03-11T21:10:52.029948+0000","last_active":"2024-03-11T21:11:52.384567+0000","last_peered":"2024-03-11T21:11:52.384567+0000","last_clean":"2024-03-11T21:11:52.384567+0000","last_became_active":"2024-03-11T20:50:30.101721+0000","last_became_peered":"2024-03-11T20:50:30.101721+0000","last_unstale":"2024-03-11T21:11:52.384567+0000","last_undegraded":"2024-03-11T21:11:52.384567+0000","last_fullsized":"2024-03-11T21:11:52.384567+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'30","last_scrub_stamp":"2024-03-11T21:10:52.029827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:10:52.029827+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:12:18.208429+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":523,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-03-11T21:11:50.582395+0000","last_change":"2024-03-11T21:11:00.484671+0000","last_active":"2024-03-11T21:11:50.582395+0000","last_peered":"2024-03-11T21:11:50.582395+0000","last_clean":"2024-03-11T21:11:50.582395+0000","last_became_active":"2024-03-11T21:05:17.316714+0000","last_became_peered":"2024-03-11T21:05:17.316714+0000","last_unstale":"2024-03-11T21:11:50.582395+0000","last_undegraded":"2024-03-11T21:11:50.582395+0000","last_fullsized":"2024-03-11T21:11:50.582395+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:11:00.484541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:55.545758+0000","last_clean_scrub_stamp":"2024-03-11T21:11:00.484541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:12:09.789390+0000","scrub_duration":59,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":521,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:11:52.916516+0000","last_change":"2024-03-11T21:09:53.477578+0000","last_active":"2024-03-11T21:11:52.916516+0000","last_peered":"2024-03-11T21:11:52.916516+0000","last_clean":"2024-03-11T21:11:52.916516+0000","last_became_active":"2024-03-11T21:05:17.319272+0000","last_became_peered":"2024-03-11T21:05:17.319272+0000","last_unstale":"2024-03-11T21:11:52.916516+0000","last_undegraded":"2024-03-11T21:11:52.916516+0000","last_fullsized":"2024-03-11T21:11:52.916516+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:53.477473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:54:13.660325+0000","last_clean_scrub_stamp":"2024-03-11T21:09:53.477473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:54.878262+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":690,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:11:52.384140+0000","last_change":"2024-03-11T21:09:49.933320+0000","last_active":"2024-03-11T21:11:52.384140+0000","last_peered":"2024-03-11T21:11:52.384140+0000","last_clean":"2024-03-11T21:11:52.384140+0000","last_became_active":"2024-03-11T20:50:28.058274+0000","last_became_peered":"2024-03-11T20:50:28.058274+0000","last_unstale":"2024-03-11T21:11:52.384140+0000","last_undegraded":"2024-03-11T21:11:52.384140+0000","last_fullsized":"2024-03-11T21:11:52.384140+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:49.933191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:09:49.933191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:53.045181+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"358'27","reported_seq":754,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-03-11T21:11:48.552436+0000","last_change":"2024-03-11T21:11:08.791120+0000","last_active":"2024-03-11T21:11:48.552436+0000","last_peered":"2024-03-11T21:11:48.552436+0000","last_clean":"2024-03-11T21:11:48.552436+0000","last_became_active":"2024-03-11T20:50:30.092638+0000","last_became_peered":"2024-03-11T20:50:30.092638+0000","last_unstale":"2024-03-11T21:11:48.552436+0000","last_undegraded":"2024-03-11T21:11:48.552436+0000","last_fullsized":"2024-03-11T21:11:48.552436+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'27","last_scrub_stamp":"2024-03-11T21:11:08.791004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:11:08.791004+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:12:16.243973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":711,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-03-11T21:11:50.582492+0000","last_change":"2024-03-11T21:11:19.401922+0000","last_active":"2024-03-11T21:11:50.582492+0000","last_peered":"2024-03-11T21:11:50.582492+0000","last_clean":"2024-03-11T21:11:50.582492+0000","last_became_active":"2024-03-11T21:00:48.460694+0000","last_became_peered":"2024-03-11T21:00:48.460694+0000","last_unstale":"2024-03-11T21:11:50.582492+0000","last_undegraded":"2024-03-11T21:11:50.582492+0000","last_fullsized":"2024-03-11T21:11:50.582492+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:11:19.401770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:11:19.401770+0000","last_clean_scrub_stamp":"2024-03-11T21:11:19.401770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:12:37.369492+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":505,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:11:52.382932+0000","last_change":"2024-03-11T21:09:29.891712+0000","last_active":"2024-03-11T21:11:52.382932+0000","last_peered":"2024-03-11T21:11:52.382932+0000","last_clean":"2024-03-11T21:11:52.382932+0000","last_became_active":"2024-03-11T21:05:17.316491+0000","last_became_peered":"2024-03-11T21:05:17.316491+0000","last_unstale":"2024-03-11T21:11:52.382932+0000","last_undegraded":"2024-03-11T21:11:52.382932+0000","last_fullsized":"2024-03-11T21:11:52.382932+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:29.891601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:03:36.230647+0000","last_clean_scrub_stamp":"2024-03-11T21:09:29.891601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:53.045035+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":710,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:11:53.029118+0000","last_change":"2024-03-11T21:09:52.421945+0000","last_active":"2024-03-11T21:11:53.029118+0000","last_peered":"2024-03-11T21:11:53.029118+0000","last_clean":"2024-03-11T21:11:53.029118+0000","last_became_active":"2024-03-11T20:50:28.079225+0000","last_became_peered":"2024-03-11T20:50:28.079225+0000","last_unstale":"2024-03-11T21:11:53.029118+0000","last_undegraded":"2024-03-11T21:11:53.029118+0000","last_fullsized":"2024-03-11T21:11:53.029118+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:52.421824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:09:52.421824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:54.026627+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"358'39","reported_seq":824,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-03-11T21:11:50.585303+0000","last_change":"2024-03-11T21:09:31.716392+0000","last_active":"2024-03-11T21:11:50.585303+0000","last_peered":"2024-03-11T21:11:50.585303+0000","last_clean":"2024-03-11T21:11:50.585303+0000","last_became_active":"2024-03-11T21:00:48.462943+0000","last_became_peered":"2024-03-11T21:00:48.462943+0000","last_unstale":"2024-03-11T21:11:50.585303+0000","last_undegraded":"2024-03-11T21:11:50.585303+0000","last_fullsized":"2024-03-11T21:11:50.585303+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'27","last_scrub_stamp":"2024-03-11T21:09:31.716290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:09:31.716290+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:51.330850+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":389,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-03-11T21:11:48.550728+0000","last_change":"2024-03-11T21:09:47.379851+0000","last_active":"2024-03-11T21:11:48.550728+0000","last_peered":"2024-03-11T21:11:48.550728+0000","last_clean":"2024-03-11T21:11:48.550728+0000","last_became_active":"2024-03-11T21:06:28.757201+0000","last_became_peered":"2024-03-11T21:06:28.757201+0000","last_unstale":"2024-03-11T21:11:48.550728+0000","last_undegraded":"2024-03-11T21:11:48.550728+0000","last_fullsized":"2024-03-11T21:11:48.550728+0000","mapping_epoch":297,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:47.379734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:09:47.379734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":467,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:11:52.383593+0000","last_change":"2024-03-11T21:08:01.360006+0000","last_active":"2024-03-11T21:11:52.383593+0000","last_peered":"2024-03-11T21:11:52.383593+0000","last_clean":"2024-03-11T21:11:52.383593+0000","last_became_active":"2024-03-11T21:05:17.313388+0000","last_became_peered":"2024-03-11T21:05:17.313388+0000","last_unstale":"2024-03-11T21:11:52.383593+0000","last_undegraded":"2024-03-11T21:11:52.383593+0000","last_fullsized":"2024-03-11T21:11:52.383593+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:08:01.359878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:53:50.599029+0000","last_clean_scrub_stamp":"2024-03-11T21:08:01.359878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:53.044798+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":718,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:11:53.028630+0000","last_change":"2024-03-11T21:09:57.535072+0000","last_active":"2024-03-11T21:11:53.028630+0000","last_peered":"2024-03-11T21:11:53.028630+0000","last_clean":"2024-03-11T21:11:53.028630+0000","last_became_active":"2024-03-11T20:50:28.068978+0000","last_became_peered":"2024-03-11T20:50:28.068978+0000","last_unstale":"2024-03-11T21:11:53.028630+0000","last_undegraded":"2024-03-11T21:11:53.028630+0000","last_fullsized":"2024-03-11T21:11:53.028630+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:57.534967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:54:07.155130+0000","last_clean_scrub_stamp":"2024-03-11T21:09:57.534967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:54.026590+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"358'29","reported_seq":753,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-03-11T21:11:50.584547+0000","last_change":"2024-03-11T21:10:48.516700+0000","last_active":"2024-03-11T21:11:50.584547+0000","last_peered":"2024-03-11T21:11:50.584547+0000","last_clean":"2024-03-11T21:11:50.584547+0000","last_became_active":"2024-03-11T21:00:27.952093+0000","last_became_peered":"2024-03-11T21:00:27.952093+0000","last_unstale":"2024-03-11T21:11:50.584547+0000","last_undegraded":"2024-03-11T21:11:50.584547+0000","last_fullsized":"2024-03-11T21:11:50.584547+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'29","last_scrub_stamp":"2024-03-11T21:10:48.516563+0000","last_deep_scrub":"110'17","last_deep_scrub_stamp":"2024-03-11T20:53:04.802488+0000","last_clean_scrub_stamp":"2024-03-11T21:10:48.516563+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:12:00.091300+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":678,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:11:52.384758+0000","last_change":"2024-03-11T21:08:18.541547+0000","last_active":"2024-03-11T21:11:52.384758+0000","last_peered":"2024-03-11T21:11:52.384758+0000","last_clean":"2024-03-11T21:11:52.384758+0000","last_became_active":"2024-03-11T20:50:34.334558+0000","last_became_peered":"2024-03-11T20:50:34.334558+0000","last_unstale":"2024-03-11T21:11:52.384758+0000","last_undegraded":"2024-03-11T21:11:52.384758+0000","last_fullsized":"2024-03-11T21:11:52.384758+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:08:18.541431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:55:59.892394+0000","last_clean_scrub_stamp":"2024-03-11T21:08:18.541431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:53.044779+0000","scrub_duration":163,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":63,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:11:52.382904+0000","last_change":"2024-03-11T21:10:38.442610+0000","last_active":"2024-03-11T21:11:52.382904+0000","last_peered":"2024-03-11T21:11:52.382904+0000","last_clean":"2024-03-11T21:11:52.382904+0000","last_became_active":"2024-03-11T21:10:38.441649+0000","last_became_peered":"2024-03-11T21:10:38.441649+0000","last_unstale":"2024-03-11T21:11:52.382904+0000","last_undegraded":"2024-03-11T21:11:52.382904+0000","last_fullsized":"2024-03-11T21:11:52.382904+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:35.734334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:09:35.734334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:53.045052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"77'1","reported_seq":704,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:11:52.383032+0000","last_change":"2024-03-11T21:11:05.190655+0000","last_active":"2024-03-11T21:11:52.383032+0000","last_peered":"2024-03-11T21:11:52.383032+0000","last_clean":"2024-03-11T21:11:52.383032+0000","last_became_active":"2024-03-11T20:50:28.068840+0000","last_became_peered":"2024-03-11T20:50:28.068840+0000","last_unstale":"2024-03-11T21:11:52.383032+0000","last_undegraded":"2024-03-11T21:11:52.383032+0000","last_fullsized":"2024-03-11T21:11:52.383032+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-03-11T21:11:05.190559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:11:05.190559+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:12:22.697771+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"357'25","reported_seq":570,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-03-11T21:11:50.355419+0000","last_change":"2024-03-11T21:11:09.683813+0000","last_active":"2024-03-11T21:11:50.355419+0000","last_peered":"2024-03-11T21:11:50.355419+0000","last_clean":"2024-03-11T21:11:50.355419+0000","last_became_active":"2024-03-11T21:05:17.443030+0000","last_became_peered":"2024-03-11T21:05:17.443030+0000","last_unstale":"2024-03-11T21:11:50.355419+0000","last_undegraded":"2024-03-11T21:11:50.355419+0000","last_fullsized":"2024-03-11T21:11:50.355419+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"357'25","last_scrub_stamp":"2024-03-11T21:11:09.683716+0000","last_deep_scrub":"207'19","last_deep_scrub_stamp":"2024-03-11T21:03:39.240089+0000","last_clean_scrub_stamp":"2024-03-11T21:11:09.683716+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:12:11.398886+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":478,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:11:52.918062+0000","last_change":"2024-03-11T21:09:51.529563+0000","last_active":"2024-03-11T21:11:52.918062+0000","last_peered":"2024-03-11T21:11:52.918062+0000","last_clean":"2024-03-11T21:11:52.918062+0000","last_became_active":"2024-03-11T21:05:17.313986+0000","last_became_peered":"2024-03-11T21:05:17.313986+0000","last_unstale":"2024-03-11T21:11:52.918062+0000","last_undegraded":"2024-03-11T21:11:52.918062+0000","last_fullsized":"2024-03-11T21:11:52.918062+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:51.529432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:33.399510+0000","last_clean_scrub_stamp":"2024-03-11T21:09:51.529432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:54.878304+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":494,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-03-11T21:11:50.357170+0000","last_change":"2024-03-11T21:10:55.542768+0000","last_active":"2024-03-11T21:11:50.357170+0000","last_peered":"2024-03-11T21:11:50.357170+0000","last_clean":"2024-03-11T21:11:50.357170+0000","last_became_active":"2024-03-11T21:08:05.298552+0000","last_became_peered":"2024-03-11T21:08:05.298552+0000","last_unstale":"2024-03-11T21:11:50.357170+0000","last_undegraded":"2024-03-11T21:11:50.357170+0000","last_fullsized":"2024-03-11T21:11:50.357170+0000","mapping_epoch":319,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:10:55.542644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:24.666325+0000","last_clean_scrub_stamp":"2024-03-11T21:10:55.542644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:12:12.365126+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":719,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:11:52.917150+0000","last_change":"2024-03-11T21:09:44.522913+0000","last_active":"2024-03-11T21:11:52.917150+0000","last_peered":"2024-03-11T21:11:52.917150+0000","last_clean":"2024-03-11T21:11:52.917150+0000","last_became_active":"2024-03-11T21:00:27.956935+0000","last_became_peered":"2024-03-11T21:00:27.956935+0000","last_unstale":"2024-03-11T21:11:52.917150+0000","last_undegraded":"2024-03-11T21:11:52.917150+0000","last_fullsized":"2024-03-11T21:11:52.917150+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:44.522800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:09:44.522800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:54.878164+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"357'9","reported_seq":726,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-03-11T21:11:48.551507+0000","last_change":"2024-03-11T21:10:52.845265+0000","last_active":"2024-03-11T21:11:48.551507+0000","last_peered":"2024-03-11T21:11:48.551507+0000","last_clean":"2024-03-11T21:11:48.551507+0000","last_became_active":"2024-03-11T21:00:27.955296+0000","last_became_peered":"2024-03-11T21:00:27.955296+0000","last_unstale":"2024-03-11T21:11:48.551507+0000","last_undegraded":"2024-03-11T21:11:48.551507+0000","last_fullsized":"2024-03-11T21:11:48.551507+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"357'9","last_scrub_stamp":"2024-03-11T21:10:52.845050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:10:52.845050+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:12:03.949558+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":13,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":417,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-03-11T21:11:50.358078+0000","last_change":"2024-03-11T21:11:20.728085+0000","last_active":"2024-03-11T21:11:50.358078+0000","last_peered":"2024-03-11T21:11:50.358078+0000","last_clean":"2024-03-11T21:11:50.358078+0000","last_became_active":"2024-03-11T21:00:48.464694+0000","last_became_peered":"2024-03-11T21:00:48.464694+0000","last_unstale":"2024-03-11T21:11:50.358078+0000","last_undegraded":"2024-03-11T21:11:50.358078+0000","last_fullsized":"2024-03-11T21:11:50.358078+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:11:20.727935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:11:20.727935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:12:45.672473+0000","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1f","version":"212'16","reported_seq":1004,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:11:52.916107+0000","last_change":"2024-03-11T21:09:47.536775+0000","last_active":"2024-03-11T21:11:52.916107+0000","last_peered":"2024-03-11T21:11:52.916107+0000","last_clean":"2024-03-11T21:11:52.916107+0000","last_became_active":"2024-03-11T21:00:48.464350+0000","last_became_peered":"2024-03-11T21:00:48.464350+0000","last_unstale":"2024-03-11T21:11:52.916107+0000","last_undegraded":"2024-03-11T21:11:52.916107+0000","last_fullsized":"2024-03-11T21:11:52.916107+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"212'16","last_scrub_stamp":"2024-03-11T21:09:47.536622+0000","last_deep_scrub":"212'16","last_deep_scrub_stamp":"2024-03-11T21:06:47.821221+0000","last_clean_scrub_stamp":"2024-03-11T21:09:47.536622+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:54.878062+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"362'2","reported_seq":690,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:11:52.383421+0000","last_change":"2024-03-11T21:11:03.482945+0000","last_active":"2024-03-11T21:11:52.383421+0000","last_peered":"2024-03-11T21:11:52.383421+0000","last_clean":"2024-03-11T21:11:52.383421+0000","last_became_active":"2024-03-11T20:50:28.068827+0000","last_became_peered":"2024-03-11T20:50:28.068827+0000","last_unstale":"2024-03-11T21:11:52.383421+0000","last_undegraded":"2024-03-11T21:11:52.383421+0000","last_fullsized":"2024-03-11T21:11:52.383421+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"362'2","last_scrub_stamp":"2024-03-11T21:11:03.482685+0000","last_deep_scrub":"362'2","last_deep_scrub_stamp":"2024-03-11T21:11:03.482685+0000","last_clean_scrub_stamp":"2024-03-11T21:11:03.482685+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:12:31.332073+0000","scrub_duration":335,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"357'21","reported_seq":760,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:11:52.382515+0000","last_change":"2024-03-11T21:08:21.296049+0000","last_active":"2024-03-11T21:11:52.382515+0000","last_peered":"2024-03-11T21:11:52.382515+0000","last_clean":"2024-03-11T21:11:52.382515+0000","last_became_active":"2024-03-11T21:00:48.467513+0000","last_became_peered":"2024-03-11T21:00:48.467513+0000","last_unstale":"2024-03-11T21:11:52.382515+0000","last_undegraded":"2024-03-11T21:11:52.382515+0000","last_fullsized":"2024-03-11T21:11:52.382515+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'15","last_scrub_stamp":"2024-03-11T21:08:21.295914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:08:21.295914+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:53.044965+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"105'2","reported_seq":709,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:11:53.029033+0000","last_change":"2024-03-11T21:11:01.833581+0000","last_active":"2024-03-11T21:11:53.029033+0000","last_peered":"2024-03-11T21:11:53.029033+0000","last_clean":"2024-03-11T21:11:53.029033+0000","last_became_active":"2024-03-11T20:50:34.327356+0000","last_became_peered":"2024-03-11T20:50:34.327356+0000","last_unstale":"2024-03-11T21:11:53.029033+0000","last_undegraded":"2024-03-11T21:11:53.029033+0000","last_fullsized":"2024-03-11T21:11:53.029033+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'2","last_scrub_stamp":"2024-03-11T21:11:01.833452+0000","last_deep_scrub":"105'2","last_deep_scrub_stamp":"2024-03-11T21:03:41.971918+0000","last_clean_scrub_stamp":"2024-03-11T21:11:01.833452+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:12:25.014272+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":396,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-03-11T21:11:48.552485+0000","last_change":"2024-03-11T21:09:54.403649+0000","last_active":"2024-03-11T21:11:48.552485+0000","last_peered":"2024-03-11T21:11:48.552485+0000","last_clean":"2024-03-11T21:11:48.552485+0000","last_became_active":"2024-03-11T21:06:28.759784+0000","last_became_peered":"2024-03-11T21:06:28.759784+0000","last_unstale":"2024-03-11T21:11:48.552485+0000","last_undegraded":"2024-03-11T21:11:48.552485+0000","last_fullsized":"2024-03-11T21:11:48.552485+0000","mapping_epoch":297,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:54.403136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:06:32.968650+0000","last_clean_scrub_stamp":"2024-03-11T21:09:54.403136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":711,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-03-11T21:11:50.582583+0000","last_change":"2024-03-11T21:10:01.677396+0000","last_active":"2024-03-11T21:11:50.582583+0000","last_peered":"2024-03-11T21:11:50.582583+0000","last_clean":"2024-03-11T21:11:50.582583+0000","last_became_active":"2024-03-11T21:00:27.947614+0000","last_became_peered":"2024-03-11T21:00:27.947614+0000","last_unstale":"2024-03-11T21:11:50.582583+0000","last_undegraded":"2024-03-11T21:11:50.582583+0000","last_fullsized":"2024-03-11T21:11:50.582583+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:10:01.677279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:08:27.756036+0000","last_clean_scrub_stamp":"2024-03-11T21:10:01.677279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:51.331137+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"358'39","reported_seq":465,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-03-11T21:11:50.354546+0000","last_change":"2024-03-11T21:10:47.277481+0000","last_active":"2024-03-11T21:11:50.354546+0000","last_peered":"2024-03-11T21:11:50.354546+0000","last_clean":"2024-03-11T21:11:50.354546+0000","last_became_active":"2024-03-11T21:00:48.457893+0000","last_became_peered":"2024-03-11T21:00:48.457893+0000","last_unstale":"2024-03-11T21:11:50.354546+0000","last_undegraded":"2024-03-11T21:11:50.354546+0000","last_fullsized":"2024-03-11T21:11:50.354546+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'39","last_scrub_stamp":"2024-03-11T21:10:47.277324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:10:47.277324+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:12:15.705510+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":685,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:11:52.383367+0000","last_change":"2024-03-11T21:10:49.083160+0000","last_active":"2024-03-11T21:11:52.383367+0000","last_peered":"2024-03-11T21:11:52.383367+0000","last_clean":"2024-03-11T21:11:52.383367+0000","last_became_active":"2024-03-11T20:50:34.337878+0000","last_became_peered":"2024-03-11T20:50:34.337878+0000","last_unstale":"2024-03-11T21:11:52.383367+0000","last_undegraded":"2024-03-11T21:11:52.383367+0000","last_fullsized":"2024-03-11T21:11:52.383367+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:10:49.083027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:52:11.462480+0000","last_clean_scrub_stamp":"2024-03-11T21:10:49.083027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:59.535242+0000","scrub_duration":119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":689,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:11:52.384157+0000","last_change":"2024-03-11T21:10:02.099820+0000","last_active":"2024-03-11T21:11:52.384157+0000","last_peered":"2024-03-11T21:11:52.384157+0000","last_clean":"2024-03-11T21:11:52.384157+0000","last_became_active":"2024-03-11T20:50:32.646323+0000","last_became_peered":"2024-03-11T20:50:32.646323+0000","last_unstale":"2024-03-11T21:11:52.384157+0000","last_undegraded":"2024-03-11T21:11:52.384157+0000","last_fullsized":"2024-03-11T21:11:52.384157+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:10:02.099653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:10:02.099653+0000","last_clean_scrub_stamp":"2024-03-11T21:10:02.099653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:53.045222+0000","scrub_duration":179,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":734,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:11:52.918187+0000","last_change":"2024-03-11T21:11:16.864655+0000","last_active":"2024-03-11T21:11:52.918187+0000","last_peered":"2024-03-11T21:11:52.918187+0000","last_clean":"2024-03-11T21:11:52.918187+0000","last_became_active":"2024-03-11T20:50:28.086513+0000","last_became_peered":"2024-03-11T20:50:28.086513+0000","last_unstale":"2024-03-11T21:11:52.918187+0000","last_undegraded":"2024-03-11T21:11:52.918187+0000","last_fullsized":"2024-03-11T21:11:52.918187+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:11:16.864254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:11:16.864254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:12:40.435103+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"357'25","reported_seq":778,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-03-11T21:11:50.584077+0000","last_change":"2024-03-11T21:09:49.809926+0000","last_active":"2024-03-11T21:11:50.584077+0000","last_peered":"2024-03-11T21:11:50.584077+0000","last_clean":"2024-03-11T21:11:50.584077+0000","last_became_active":"2024-03-11T21:00:48.464707+0000","last_became_peered":"2024-03-11T21:00:48.464707+0000","last_unstale":"2024-03-11T21:11:50.584077+0000","last_undegraded":"2024-03-11T21:11:50.584077+0000","last_fullsized":"2024-03-11T21:11:50.584077+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'19","last_scrub_stamp":"2024-03-11T21:09:49.809679+0000","last_deep_scrub":"145'13","last_deep_scrub_stamp":"2024-03-11T20:55:53.560287+0000","last_clean_scrub_stamp":"2024-03-11T21:09:49.809679+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:51.330903+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":708,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:11:53.027861+0000","last_change":"2024-03-11T21:11:07.805434+0000","last_active":"2024-03-11T21:11:53.027861+0000","last_peered":"2024-03-11T21:11:53.027861+0000","last_clean":"2024-03-11T21:11:53.027861+0000","last_became_active":"2024-03-11T21:00:27.973405+0000","last_became_peered":"2024-03-11T21:00:27.973405+0000","last_unstale":"2024-03-11T21:11:53.027861+0000","last_undegraded":"2024-03-11T21:11:53.027861+0000","last_fullsized":"2024-03-11T21:11:53.027861+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:11:07.805305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:11:07.805305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:12:08.310664+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":694,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:11:52.382624+0000","last_change":"2024-03-11T21:08:24.323683+0000","last_active":"2024-03-11T21:11:52.382624+0000","last_peered":"2024-03-11T21:11:52.382624+0000","last_clean":"2024-03-11T21:11:52.382624+0000","last_became_active":"2024-03-11T21:00:48.472812+0000","last_became_peered":"2024-03-11T21:00:48.472812+0000","last_unstale":"2024-03-11T21:11:52.382624+0000","last_undegraded":"2024-03-11T21:11:52.382624+0000","last_fullsized":"2024-03-11T21:11:52.382624+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:08:24.323601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:08:24.323601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:53.044838+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":710,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-03-11T21:11:50.584029+0000","last_change":"2024-03-11T21:09:37.712819+0000","last_active":"2024-03-11T21:11:50.584029+0000","last_peered":"2024-03-11T21:11:50.584029+0000","last_clean":"2024-03-11T21:11:50.584029+0000","last_became_active":"2024-03-11T20:50:28.075760+0000","last_became_peered":"2024-03-11T20:50:28.075760+0000","last_unstale":"2024-03-11T21:11:50.584029+0000","last_undegraded":"2024-03-11T21:11:50.584029+0000","last_fullsized":"2024-03-11T21:11:50.584029+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:37.712659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:58:55.977289+0000","last_clean_scrub_stamp":"2024-03-11T21:09:37.712659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:51.331001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"358'18","reported_seq":373,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-03-11T21:11:48.552412+0000","last_change":"2024-03-11T21:10:54.801465+0000","last_active":"2024-03-11T21:11:48.552412+0000","last_peered":"2024-03-11T21:11:48.552412+0000","last_clean":"2024-03-11T21:11:48.552412+0000","last_became_active":"2024-03-11T21:06:28.763971+0000","last_became_peered":"2024-03-11T21:06:28.763971+0000","last_unstale":"2024-03-11T21:11:48.552412+0000","last_undegraded":"2024-03-11T21:11:48.552412+0000","last_fullsized":"2024-03-11T21:11:48.552412+0000","mapping_epoch":297,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'18","last_scrub_stamp":"2024-03-11T21:10:54.801312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:10:54.801312+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:12:23.639619+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":25,"num_read_kb":22,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.11","version":"186'2","reported_seq":687,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:11:52.382685+0000","last_change":"2024-03-11T21:11:11.081301+0000","last_active":"2024-03-11T21:11:52.382685+0000","last_peered":"2024-03-11T21:11:52.382685+0000","last_clean":"2024-03-11T21:11:52.382685+0000","last_became_active":"2024-03-11T20:50:34.339326+0000","last_became_peered":"2024-03-11T20:50:34.339326+0000","last_unstale":"2024-03-11T21:11:52.382685+0000","last_undegraded":"2024-03-11T21:11:52.382685+0000","last_fullsized":"2024-03-11T21:11:52.382685+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"186'2","last_scrub_stamp":"2024-03-11T21:11:11.081135+0000","last_deep_scrub":"186'2","last_deep_scrub_stamp":"2024-03-11T20:59:04.534850+0000","last_clean_scrub_stamp":"2024-03-11T21:11:11.081135+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:12:38.689515+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":529,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:11:52.917508+0000","last_change":"2024-03-11T21:11:02.744744+0000","last_active":"2024-03-11T21:11:52.917508+0000","last_peered":"2024-03-11T21:11:52.917508+0000","last_clean":"2024-03-11T21:11:52.917508+0000","last_became_active":"2024-03-11T21:05:17.314174+0000","last_became_peered":"2024-03-11T21:05:17.314174+0000","last_unstale":"2024-03-11T21:11:52.917508+0000","last_undegraded":"2024-03-11T21:11:52.917508+0000","last_fullsized":"2024-03-11T21:11:52.917508+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:11:02.744642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:11:02.744642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:12:28.502080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":409,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-03-11T21:11:50.357731+0000","last_change":"2024-03-11T21:10:01.220880+0000","last_active":"2024-03-11T21:11:50.357731+0000","last_peered":"2024-03-11T21:11:50.357731+0000","last_clean":"2024-03-11T21:11:50.357731+0000","last_became_active":"2024-03-11T21:00:48.455510+0000","last_became_peered":"2024-03-11T21:00:48.455510+0000","last_unstale":"2024-03-11T21:11:50.357731+0000","last_undegraded":"2024-03-11T21:11:50.357731+0000","last_fullsized":"2024-03-11T21:11:50.357731+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:10:01.220751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:10:01.220751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:53.672923+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"358'35","reported_seq":514,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:11:53.028704+0000","last_change":"2024-03-11T21:11:16.867612+0000","last_active":"2024-03-11T21:11:53.028704+0000","last_peered":"2024-03-11T21:11:53.028704+0000","last_clean":"2024-03-11T21:11:53.028704+0000","last_became_active":"2024-03-11T21:05:17.314507+0000","last_became_peered":"2024-03-11T21:05:17.314507+0000","last_unstale":"2024-03-11T21:11:53.028704+0000","last_undegraded":"2024-03-11T21:11:53.028704+0000","last_fullsized":"2024-03-11T21:11:53.028704+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'35","last_scrub_stamp":"2024-03-11T21:11:16.867326+0000","last_deep_scrub":"161'15","last_deep_scrub_stamp":"2024-03-11T20:57:15.125548+0000","last_clean_scrub_stamp":"2024-03-11T21:11:16.867326+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:12:25.738404+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":715,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-03-11T21:11:48.551006+0000","last_change":"2024-03-11T21:10:02.497069+0000","last_active":"2024-03-11T21:11:48.551006+0000","last_peered":"2024-03-11T21:11:48.551006+0000","last_clean":"2024-03-11T21:11:48.551006+0000","last_became_active":"2024-03-11T21:00:48.462105+0000","last_became_peered":"2024-03-11T21:00:48.462105+0000","last_unstale":"2024-03-11T21:11:48.551006+0000","last_undegraded":"2024-03-11T21:11:48.551006+0000","last_fullsized":"2024-03-11T21:11:48.551006+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:10:02.496964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:10:02.496964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":417,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-03-11T21:11:50.355154+0000","last_change":"2024-03-11T21:11:21.665103+0000","last_active":"2024-03-11T21:11:50.355154+0000","last_peered":"2024-03-11T21:11:50.355154+0000","last_clean":"2024-03-11T21:11:50.355154+0000","last_became_active":"2024-03-11T21:00:48.459043+0000","last_became_peered":"2024-03-11T21:00:48.459043+0000","last_unstale":"2024-03-11T21:11:50.355154+0000","last_undegraded":"2024-03-11T21:11:50.355154+0000","last_fullsized":"2024-03-11T21:11:50.355154+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:11:21.664989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:23.137832+0000","last_clean_scrub_stamp":"2024-03-11T21:11:21.664989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:12:31.120616+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":735,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:11:53.953585+0000","last_change":"2024-03-11T21:11:53.953625+0000","last_active":"2024-03-11T21:11:53.953585+0000","last_peered":"2024-03-11T21:11:53.953585+0000","last_clean":"2024-03-11T21:11:53.953585+0000","last_became_active":"2024-03-11T20:50:28.082737+0000","last_became_peered":"2024-03-11T20:50:28.082737+0000","last_unstale":"2024-03-11T21:11:53.953585+0000","last_undegraded":"2024-03-11T21:11:53.953585+0000","last_fullsized":"2024-03-11T21:11:53.953585+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:11:53.953494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:28.404997+0000","last_clean_scrub_stamp":"2024-03-11T21:11:53.953494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:13:01.110794+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"358'21","reported_seq":749,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:11:53.029426+0000","last_change":"2024-03-11T21:11:05.817669+0000","last_active":"2024-03-11T21:11:53.029426+0000","last_peered":"2024-03-11T21:11:53.029426+0000","last_clean":"2024-03-11T21:11:53.029426+0000","last_became_active":"2024-03-11T21:00:27.967624+0000","last_became_peered":"2024-03-11T21:00:27.967624+0000","last_unstale":"2024-03-11T21:11:53.029426+0000","last_undegraded":"2024-03-11T21:11:53.029426+0000","last_fullsized":"2024-03-11T21:11:53.029426+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'21","last_scrub_stamp":"2024-03-11T21:11:05.817511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:11:05.817511+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:12:23.242575+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":361,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-03-11T21:11:50.585320+0000","last_change":"2024-03-11T21:10:51.719504+0000","last_active":"2024-03-11T21:11:50.585320+0000","last_peered":"2024-03-11T21:11:50.585320+0000","last_clean":"2024-03-11T21:11:50.585320+0000","last_became_active":"2024-03-11T21:08:05.299913+0000","last_became_peered":"2024-03-11T21:08:05.299913+0000","last_unstale":"2024-03-11T21:11:50.585320+0000","last_undegraded":"2024-03-11T21:11:50.585320+0000","last_fullsized":"2024-03-11T21:11:50.585320+0000","mapping_epoch":319,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:10:51.719385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:57:16.032302+0000","last_clean_scrub_stamp":"2024-03-11T21:10:51.719385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:12:19.761008+0000","scrub_duration":191,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"112'12","reported_seq":970,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-03-11T21:11:50.585072+0000","last_change":"2024-03-11T21:11:07.401850+0000","last_active":"2024-03-11T21:11:50.585072+0000","last_peered":"2024-03-11T21:11:50.585072+0000","last_clean":"2024-03-11T21:11:50.585072+0000","last_became_active":"2024-03-11T20:50:32.650064+0000","last_became_peered":"2024-03-11T20:50:32.650064+0000","last_unstale":"2024-03-11T21:11:50.585072+0000","last_undegraded":"2024-03-11T21:11:50.585072+0000","last_fullsized":"2024-03-11T21:11:50.585072+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'12","last_scrub_stamp":"2024-03-11T21:11:07.401721+0000","last_deep_scrub":"112'12","last_deep_scrub_stamp":"2024-03-11T20:53:11.843951+0000","last_clean_scrub_stamp":"2024-03-11T21:11:07.401721+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:12:22.182369+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":411,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-03-11T21:11:48.551266+0000","last_change":"2024-03-11T21:11:18.956954+0000","last_active":"2024-03-11T21:11:48.551266+0000","last_peered":"2024-03-11T21:11:48.551266+0000","last_clean":"2024-03-11T21:11:48.551266+0000","last_became_active":"2024-03-11T21:06:28.756041+0000","last_became_peered":"2024-03-11T21:06:28.756041+0000","last_unstale":"2024-03-11T21:11:48.551266+0000","last_undegraded":"2024-03-11T21:11:48.551266+0000","last_fullsized":"2024-03-11T21:11:48.551266+0000","mapping_epoch":297,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:11:18.956581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:06:52.862971+0000","last_clean_scrub_stamp":"2024-03-11T21:11:18.956581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:12:33.876556+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"357'25","reported_seq":471,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-03-11T21:11:48.552203+0000","last_change":"2024-03-11T21:11:17.944714+0000","last_active":"2024-03-11T21:11:48.552203+0000","last_peered":"2024-03-11T21:11:48.552203+0000","last_clean":"2024-03-11T21:11:48.552203+0000","last_became_active":"2024-03-11T21:06:28.758223+0000","last_became_peered":"2024-03-11T21:06:28.758223+0000","last_unstale":"2024-03-11T21:11:48.552203+0000","last_undegraded":"2024-03-11T21:11:48.552203+0000","last_fullsized":"2024-03-11T21:11:48.552203+0000","mapping_epoch":297,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"357'25","last_scrub_stamp":"2024-03-11T21:11:17.944545+0000","last_deep_scrub":"119'13","last_deep_scrub_stamp":"2024-03-11T20:54:15.443472+0000","last_clean_scrub_stamp":"2024-03-11T21:11:17.944545+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:12:20.096454+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":383,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-03-11T21:11:48.553361+0000","last_change":"2024-03-11T21:10:01.497745+0000","last_active":"2024-03-11T21:11:48.553361+0000","last_peered":"2024-03-11T21:11:48.553361+0000","last_clean":"2024-03-11T21:11:48.553361+0000","last_became_active":"2024-03-11T21:06:28.758634+0000","last_became_peered":"2024-03-11T21:06:28.758634+0000","last_unstale":"2024-03-11T21:11:48.553361+0000","last_undegraded":"2024-03-11T21:11:48.553361+0000","last_fullsized":"2024-03-11T21:11:48.553361+0000","mapping_epoch":297,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:10:01.497631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:10:01.497631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"212'17","reported_seq":981,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:11:52.384056+0000","last_change":"2024-03-11T21:08:20.327834+0000","last_active":"2024-03-11T21:11:52.384056+0000","last_peered":"2024-03-11T21:11:52.384056+0000","last_clean":"2024-03-11T21:11:52.384056+0000","last_became_active":"2024-03-11T21:00:48.467116+0000","last_became_peered":"2024-03-11T21:00:48.467116+0000","last_unstale":"2024-03-11T21:11:52.384056+0000","last_undegraded":"2024-03-11T21:11:52.384056+0000","last_fullsized":"2024-03-11T21:11:52.384056+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"212'17","last_scrub_stamp":"2024-03-11T21:08:20.327696+0000","last_deep_scrub":"110'13","last_deep_scrub_stamp":"2024-03-11T20:53:00.434532+0000","last_clean_scrub_stamp":"2024-03-11T21:08:20.327696+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":719,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-03-11T21:11:50.582512+0000","last_change":"2024-03-11T21:11:01.413740+0000","last_active":"2024-03-11T21:11:50.582512+0000","last_peered":"2024-03-11T21:11:50.582512+0000","last_clean":"2024-03-11T21:11:50.582512+0000","last_became_active":"2024-03-11T20:50:28.070325+0000","last_became_peered":"2024-03-11T20:50:28.070325+0000","last_unstale":"2024-03-11T21:11:50.582512+0000","last_undegraded":"2024-03-11T21:11:50.582512+0000","last_fullsized":"2024-03-11T21:11:50.582512+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:11:01.413630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:11:01.413630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:12:30.767060+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"358'32","reported_seq":764,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:11:53.028137+0000","last_change":"2024-03-11T21:10:56.860289+0000","last_active":"2024-03-11T21:11:53.028137+0000","last_peered":"2024-03-11T21:11:53.028137+0000","last_clean":"2024-03-11T21:11:53.028137+0000","last_became_active":"2024-03-11T21:00:27.967078+0000","last_became_peered":"2024-03-11T21:00:27.967078+0000","last_unstale":"2024-03-11T21:11:53.028137+0000","last_undegraded":"2024-03-11T21:11:53.028137+0000","last_fullsized":"2024-03-11T21:11:53.028137+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'32","last_scrub_stamp":"2024-03-11T21:10:56.860175+0000","last_deep_scrub":"116'16","last_deep_scrub_stamp":"2024-03-11T20:53:39.292118+0000","last_clean_scrub_stamp":"2024-03-11T21:10:56.860175+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:12:12.317656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":699,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:11:53.028528+0000","last_change":"2024-03-11T21:11:02.899594+0000","last_active":"2024-03-11T21:11:53.028528+0000","last_peered":"2024-03-11T21:11:53.028528+0000","last_clean":"2024-03-11T21:11:53.028528+0000","last_became_active":"2024-03-11T21:00:27.966997+0000","last_became_peered":"2024-03-11T21:00:27.966997+0000","last_unstale":"2024-03-11T21:11:53.028528+0000","last_undegraded":"2024-03-11T21:11:53.028528+0000","last_fullsized":"2024-03-11T21:11:53.028528+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:11:02.899468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:11:02.899468+0000","last_clean_scrub_stamp":"2024-03-11T21:11:02.899468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:12:11.273473+0000","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":701,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:11:52.384786+0000","last_change":"2024-03-11T21:09:24.812080+0000","last_active":"2024-03-11T21:11:52.384786+0000","last_peered":"2024-03-11T21:11:52.384786+0000","last_clean":"2024-03-11T21:11:52.384786+0000","last_became_active":"2024-03-11T21:00:48.468386+0000","last_became_peered":"2024-03-11T21:00:48.468386+0000","last_unstale":"2024-03-11T21:11:52.384786+0000","last_undegraded":"2024-03-11T21:11:52.384786+0000","last_fullsized":"2024-03-11T21:11:52.384786+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:24.811789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:09:24.811789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:53.045071+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"166'2","reported_seq":754,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:11:52.915943+0000","last_change":"2024-03-11T21:09:41.540284+0000","last_active":"2024-03-11T21:11:52.915943+0000","last_peered":"2024-03-11T21:11:52.915943+0000","last_clean":"2024-03-11T21:11:52.915943+0000","last_became_active":"2024-03-11T21:00:48.466071+0000","last_became_peered":"2024-03-11T21:00:48.466071+0000","last_unstale":"2024-03-11T21:11:52.915943+0000","last_undegraded":"2024-03-11T21:11:52.915943+0000","last_fullsized":"2024-03-11T21:11:52.915943+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'2","last_scrub_stamp":"2024-03-11T21:09:41.540185+0000","last_deep_scrub":"166'2","last_deep_scrub_stamp":"2024-03-11T20:57:39.868776+0000","last_clean_scrub_stamp":"2024-03-11T21:09:41.540185+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:54.878009+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1822,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"364'14","reported_seq":735,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:11:52.916382+0000","last_change":"2024-03-11T21:11:15.819520+0000","last_active":"2024-03-11T21:11:52.916382+0000","last_peered":"2024-03-11T21:11:52.916382+0000","last_clean":"2024-03-11T21:11:52.916382+0000","last_became_active":"2024-03-11T20:50:30.101572+0000","last_became_peered":"2024-03-11T20:50:30.101572+0000","last_unstale":"2024-03-11T21:11:52.916382+0000","last_undegraded":"2024-03-11T21:11:52.916382+0000","last_fullsized":"2024-03-11T21:11:52.916382+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"364'14","last_scrub_stamp":"2024-03-11T21:11:15.819245+0000","last_deep_scrub":"364'14","last_deep_scrub_stamp":"2024-03-11T21:11:15.819245+0000","last_clean_scrub_stamp":"2024-03-11T21:11:15.819245+0000","objects_scrubbed":2,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:12:31.518827+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":16,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":462,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:11:53.029041+0000","last_change":"2024-03-11T21:11:03.865348+0000","last_active":"2024-03-11T21:11:53.029041+0000","last_peered":"2024-03-11T21:11:53.029041+0000","last_clean":"2024-03-11T21:11:53.029041+0000","last_became_active":"2024-03-11T21:08:05.296676+0000","last_became_peered":"2024-03-11T21:08:05.296676+0000","last_unstale":"2024-03-11T21:11:53.029041+0000","last_undegraded":"2024-03-11T21:11:53.029041+0000","last_fullsized":"2024-03-11T21:11:53.029041+0000","mapping_epoch":319,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:11:03.865227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:11:03.865227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:12:04.006399+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":409,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-03-11T21:11:48.553420+0000","last_change":"2024-03-11T21:10:56.778830+0000","last_active":"2024-03-11T21:11:48.553420+0000","last_peered":"2024-03-11T21:11:48.553420+0000","last_clean":"2024-03-11T21:11:48.553420+0000","last_became_active":"2024-03-11T21:06:28.758133+0000","last_became_peered":"2024-03-11T21:06:28.758133+0000","last_unstale":"2024-03-11T21:11:48.553420+0000","last_undegraded":"2024-03-11T21:11:48.553420+0000","last_fullsized":"2024-03-11T21:11:48.553420+0000","mapping_epoch":297,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:10:56.778713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:54:09.537972+0000","last_clean_scrub_stamp":"2024-03-11T21:10:56.778713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:12:16.658224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":482,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-03-11T21:11:50.351809+0000","last_change":"2024-03-11T21:10:52.523855+0000","last_active":"2024-03-11T21:11:50.351809+0000","last_peered":"2024-03-11T21:11:50.351809+0000","last_clean":"2024-03-11T21:11:50.351809+0000","last_became_active":"2024-03-11T21:08:05.300426+0000","last_became_peered":"2024-03-11T21:08:05.300426+0000","last_unstale":"2024-03-11T21:11:50.351809+0000","last_undegraded":"2024-03-11T21:11:50.351809+0000","last_fullsized":"2024-03-11T21:11:50.351809+0000","mapping_epoch":319,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:10:52.523598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:18.682947+0000","last_clean_scrub_stamp":"2024-03-11T21:10:52.523598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:12:05.626155+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"358'32","reported_seq":756,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:11:53.027150+0000","last_change":"2024-03-11T21:10:50.019374+0000","last_active":"2024-03-11T21:11:53.027150+0000","last_peered":"2024-03-11T21:11:53.027150+0000","last_clean":"2024-03-11T21:11:53.027150+0000","last_became_active":"2024-03-11T21:00:27.970878+0000","last_became_peered":"2024-03-11T21:00:27.970878+0000","last_unstale":"2024-03-11T21:11:53.027150+0000","last_undegraded":"2024-03-11T21:11:53.027150+0000","last_fullsized":"2024-03-11T21:11:53.027150+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'32","last_scrub_stamp":"2024-03-11T21:10:50.019137+0000","last_deep_scrub":"262'24","last_deep_scrub_stamp":"2024-03-11T21:03:28.128719+0000","last_clean_scrub_stamp":"2024-03-11T21:10:50.019137+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:12:02.744582+0000","scrub_duration":179,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":13,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-03-11T21:11:48.552252+0000","last_change":"2024-03-11T21:11:47.554722+0000","last_active":"2024-03-11T21:11:48.552252+0000","last_peered":"2024-03-11T21:11:48.552252+0000","last_clean":"2024-03-11T21:11:48.552252+0000","last_became_active":"2024-03-11T21:11:47.554508+0000","last_became_peered":"2024-03-11T21:11:47.554508+0000","last_unstale":"2024-03-11T21:11:48.552252+0000","last_undegraded":"2024-03-11T21:11:48.552252+0000","last_fullsized":"2024-03-11T21:11:48.552252+0000","mapping_epoch":380,"log_start":"0'0","ondisk_log_start":"0'0","created":380,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:11:46.536415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:11:46.536415+0000","last_clean_scrub_stamp":"2024-03-11T21:11:46.536415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:12:51.333169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":706,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:11:52.383266+0000","last_change":"2024-03-11T21:09:51.110151+0000","last_active":"2024-03-11T21:11:52.383266+0000","last_peered":"2024-03-11T21:11:52.383266+0000","last_clean":"2024-03-11T21:11:52.383266+0000","last_became_active":"2024-03-11T21:00:48.466853+0000","last_became_peered":"2024-03-11T21:00:48.466853+0000","last_unstale":"2024-03-11T21:11:52.383266+0000","last_undegraded":"2024-03-11T21:11:52.383266+0000","last_fullsized":"2024-03-11T21:11:52.383266+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:51.109823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:52:53.450885+0000","last_clean_scrub_stamp":"2024-03-11T21:09:51.109823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:53.045163+0000","scrub_duration":139,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"77'2","reported_seq":207,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-03-11T21:11:48.552264+0000","last_change":"2024-03-11T21:11:03.794300+0000","last_active":"2024-03-11T21:11:48.552264+0000","last_peered":"2024-03-11T21:11:48.552264+0000","last_clean":"2024-03-11T21:11:48.552264+0000","last_became_active":"2024-03-11T21:06:28.759838+0000","last_became_peered":"2024-03-11T21:06:28.759838+0000","last_unstale":"2024-03-11T21:11:48.552264+0000","last_undegraded":"2024-03-11T21:11:48.552264+0000","last_fullsized":"2024-03-11T21:11:48.552264+0000","mapping_epoch":297,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'2","last_scrub_stamp":"2024-03-11T21:11:03.794167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:11:03.794167+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:12:31.863284+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"358'32","reported_seq":816,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:11:53.026572+0000","last_change":"2024-03-11T21:10:53.748734+0000","last_active":"2024-03-11T21:11:53.026572+0000","last_peered":"2024-03-11T21:11:53.026572+0000","last_clean":"2024-03-11T21:11:53.026572+0000","last_became_active":"2024-03-11T21:00:48.465254+0000","last_became_peered":"2024-03-11T21:00:48.465254+0000","last_unstale":"2024-03-11T21:11:53.026572+0000","last_undegraded":"2024-03-11T21:11:53.026572+0000","last_fullsized":"2024-03-11T21:11:53.026572+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'32","last_scrub_stamp":"2024-03-11T21:10:53.748601+0000","last_deep_scrub":"100'16","last_deep_scrub_stamp":"2024-03-11T20:56:08.036331+0000","last_clean_scrub_stamp":"2024-03-11T21:10:53.748601+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:56.698446+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"286'18","reported_seq":700,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-03-11T21:11:50.354185+0000","last_change":"2024-03-11T21:10:48.267171+0000","last_active":"2024-03-11T21:11:50.354185+0000","last_peered":"2024-03-11T21:11:50.354185+0000","last_clean":"2024-03-11T21:11:50.354185+0000","last_became_active":"2024-03-11T21:05:17.317915+0000","last_became_peered":"2024-03-11T21:05:17.317915+0000","last_unstale":"2024-03-11T21:11:50.354185+0000","last_undegraded":"2024-03-11T21:11:50.354185+0000","last_fullsized":"2024-03-11T21:11:50.354185+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"286'18","last_scrub_stamp":"2024-03-11T21:10:48.267041+0000","last_deep_scrub":"212'17","last_deep_scrub_stamp":"2024-03-11T21:03:38.258850+0000","last_clean_scrub_stamp":"2024-03-11T21:10:48.267041+0000","objects_scrubbed":1,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:12:05.174671+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":17,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:11:52.917590+0000","last_change":"2024-03-11T21:11:47.575844+0000","last_active":"2024-03-11T21:11:52.917590+0000","last_peered":"2024-03-11T21:11:52.917590+0000","last_clean":"2024-03-11T21:11:52.917590+0000","last_became_active":"2024-03-11T21:11:47.574539+0000","last_became_peered":"2024-03-11T21:11:47.574539+0000","last_unstale":"2024-03-11T21:11:52.917590+0000","last_undegraded":"2024-03-11T21:11:52.917590+0000","last_fullsized":"2024-03-11T21:11:52.917590+0000","mapping_epoch":380,"log_start":"0'0","ondisk_log_start":"0'0","created":380,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:11:46.536415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:11:46.536415+0000","last_clean_scrub_stamp":"2024-03-11T21:11:46.536415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:13:15.095087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"184'130","reported_seq":565,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:11:52.916324+0000","last_change":"2024-03-11T21:11:28.394320+0000","last_active":"2024-03-11T21:11:52.916324+0000","last_peered":"2024-03-11T21:11:52.916324+0000","last_clean":"2024-03-11T21:11:52.916324+0000","last_became_active":"2024-03-11T21:11:28.393886+0000","last_became_peered":"2024-03-11T21:11:28.393886+0000","last_unstale":"2024-03-11T21:11:52.916324+0000","last_undegraded":"2024-03-11T21:11:52.916324+0000","last_fullsized":"2024-03-11T21:11:52.916324+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":3,"last_scrub":"184'130","last_scrub_stamp":"2024-03-11T21:09:59.511442+0000","last_deep_scrub":"184'130","last_deep_scrub_stamp":"2024-03-11T20:58:56.415267+0000","last_clean_scrub_stamp":"2024-03-11T21:09:59.511442+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:55.919337+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":28,"num_write":16,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":161009,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":392,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-03-11T21:11:48.551117+0000","last_change":"2024-03-11T21:11:14.979182+0000","last_active":"2024-03-11T21:11:48.551117+0000","last_peered":"2024-03-11T21:11:48.551117+0000","last_clean":"2024-03-11T21:11:48.551117+0000","last_became_active":"2024-03-11T21:06:28.755689+0000","last_became_peered":"2024-03-11T21:06:28.755689+0000","last_unstale":"2024-03-11T21:11:48.551117+0000","last_undegraded":"2024-03-11T21:11:48.551117+0000","last_fullsized":"2024-03-11T21:11:48.551117+0000","mapping_epoch":297,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:11:14.978947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:57:21.568400+0000","last_clean_scrub_stamp":"2024-03-11T21:11:14.978947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:12:26.610251+0000","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":418,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-03-11T21:11:50.357542+0000","last_change":"2024-03-11T21:11:19.646045+0000","last_active":"2024-03-11T21:11:50.357542+0000","last_peered":"2024-03-11T21:11:50.357542+0000","last_clean":"2024-03-11T21:11:50.357542+0000","last_became_active":"2024-03-11T21:00:48.455733+0000","last_became_peered":"2024-03-11T21:00:48.455733+0000","last_unstale":"2024-03-11T21:11:50.357542+0000","last_undegraded":"2024-03-11T21:11:50.357542+0000","last_fullsized":"2024-03-11T21:11:50.357542+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:11:19.645830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:20.096874+0000","last_clean_scrub_stamp":"2024-03-11T21:11:19.645830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:12:22.896628+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"358'40","reported_seq":1023,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-03-11T21:11:50.584085+0000","last_change":"2024-03-11T21:10:42.616379+0000","last_active":"2024-03-11T21:11:50.584085+0000","last_peered":"2024-03-11T21:11:50.584085+0000","last_clean":"2024-03-11T21:11:50.584085+0000","last_became_active":"2024-03-11T20:50:30.094977+0000","last_became_peered":"2024-03-11T20:50:30.094977+0000","last_unstale":"2024-03-11T21:11:50.584085+0000","last_undegraded":"2024-03-11T21:11:50.584085+0000","last_fullsized":"2024-03-11T21:11:50.584085+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'40","last_scrub_stamp":"2024-03-11T21:10:42.616218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:10:42.616218+0000","objects_scrubbed":8,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:54.317245+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":61,"num_read_kb":51,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":418,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-03-11T21:11:50.354662+0000","last_change":"2024-03-11T21:11:06.619167+0000","last_active":"2024-03-11T21:11:50.354662+0000","last_peered":"2024-03-11T21:11:50.354662+0000","last_clean":"2024-03-11T21:11:50.354662+0000","last_became_active":"2024-03-11T21:00:48.457901+0000","last_became_peered":"2024-03-11T21:00:48.457901+0000","last_unstale":"2024-03-11T21:11:50.354662+0000","last_undegraded":"2024-03-11T21:11:50.354662+0000","last_fullsized":"2024-03-11T21:11:50.354662+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:11:06.619029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:52:10.160202+0000","last_clean_scrub_stamp":"2024-03-11T21:11:06.619029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:12:34.029559+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":17,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:11:52.382776+0000","last_change":"2024-03-11T21:11:47.554963+0000","last_active":"2024-03-11T21:11:52.382776+0000","last_peered":"2024-03-11T21:11:52.382776+0000","last_clean":"2024-03-11T21:11:52.382776+0000","last_became_active":"2024-03-11T21:11:47.554262+0000","last_became_peered":"2024-03-11T21:11:47.554262+0000","last_unstale":"2024-03-11T21:11:52.382776+0000","last_undegraded":"2024-03-11T21:11:52.382776+0000","last_fullsized":"2024-03-11T21:11:52.382776+0000","mapping_epoch":380,"log_start":"0'0","ondisk_log_start":"0'0","created":380,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:11:46.536415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:11:46.536415+0000","last_clean_scrub_stamp":"2024-03-11T21:11:46.536415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:12:50.391686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"184'130","reported_seq":558,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-03-11T21:11:50.581709+0000","last_change":"2024-03-11T21:11:28.413290+0000","last_active":"2024-03-11T21:11:50.581709+0000","last_peered":"2024-03-11T21:11:50.581709+0000","last_clean":"2024-03-11T21:11:50.581709+0000","last_became_active":"2024-03-11T21:11:28.412914+0000","last_became_peered":"2024-03-11T21:11:28.412914+0000","last_unstale":"2024-03-11T21:11:50.581709+0000","last_undegraded":"2024-03-11T21:11:50.581709+0000","last_fullsized":"2024-03-11T21:11:50.581709+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":4,"last_scrub":"184'130","last_scrub_stamp":"2024-03-11T21:09:59.511442+0000","last_deep_scrub":"184'130","last_deep_scrub_stamp":"2024-03-11T20:58:56.415267+0000","last_clean_scrub_stamp":"2024-03-11T21:09:59.511442+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":20,"num_read_kb":28,"num_write":17,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":161010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":686,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:11:52.384199+0000","last_change":"2024-03-11T21:09:39.887725+0000","last_active":"2024-03-11T21:11:52.384199+0000","last_peered":"2024-03-11T21:11:52.384199+0000","last_clean":"2024-03-11T21:11:52.384199+0000","last_became_active":"2024-03-11T21:00:27.959408+0000","last_became_peered":"2024-03-11T21:00:27.959408+0000","last_unstale":"2024-03-11T21:11:52.384199+0000","last_undegraded":"2024-03-11T21:11:52.384199+0000","last_fullsized":"2024-03-11T21:11:52.384199+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:39.887594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:09:39.887594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:53.045143+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":725,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-03-11T21:11:50.582701+0000","last_change":"2024-03-11T21:09:55.786453+0000","last_active":"2024-03-11T21:11:50.582701+0000","last_peered":"2024-03-11T21:11:50.582701+0000","last_clean":"2024-03-11T21:11:50.582701+0000","last_became_active":"2024-03-11T21:00:48.471212+0000","last_became_peered":"2024-03-11T21:00:48.471212+0000","last_unstale":"2024-03-11T21:11:50.582701+0000","last_undegraded":"2024-03-11T21:11:50.582701+0000","last_fullsized":"2024-03-11T21:11:50.582701+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:55.786349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:09:55.786349+0000","last_clean_scrub_stamp":"2024-03-11T21:09:55.786349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:51.331092+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"358'47","reported_seq":546,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-03-11T21:11:48.551382+0000","last_change":"2024-03-11T21:09:35.262587+0000","last_active":"2024-03-11T21:11:48.551382+0000","last_peered":"2024-03-11T21:11:48.551382+0000","last_clean":"2024-03-11T21:11:48.551382+0000","last_became_active":"2024-03-11T21:06:28.756995+0000","last_became_peered":"2024-03-11T21:06:28.756995+0000","last_unstale":"2024-03-11T21:11:48.551382+0000","last_undegraded":"2024-03-11T21:11:48.551382+0000","last_fullsized":"2024-03-11T21:11:48.551382+0000","mapping_epoch":297,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'37","last_scrub_stamp":"2024-03-11T21:09:35.261983+0000","last_deep_scrub":"182'27","last_deep_scrub_stamp":"2024-03-11T20:58:52.058851+0000","last_clean_scrub_stamp":"2024-03-11T21:09:35.261983+0000","objects_scrubbed":8,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":68,"num_read_kb":54,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":410,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-03-11T21:11:50.356295+0000","last_change":"2024-03-11T21:09:48.134846+0000","last_active":"2024-03-11T21:11:50.356295+0000","last_peered":"2024-03-11T21:11:50.356295+0000","last_clean":"2024-03-11T21:11:50.356295+0000","last_became_active":"2024-03-11T21:00:48.458008+0000","last_became_peered":"2024-03-11T21:00:48.458008+0000","last_unstale":"2024-03-11T21:11:50.356295+0000","last_undegraded":"2024-03-11T21:11:50.356295+0000","last_fullsized":"2024-03-11T21:11:50.356295+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:48.134729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:33.319221+0000","last_clean_scrub_stamp":"2024-03-11T21:09:48.134729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:53.672885+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":17,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:11:52.382582+0000","last_change":"2024-03-11T21:11:47.560291+0000","last_active":"2024-03-11T21:11:52.382582+0000","last_peered":"2024-03-11T21:11:52.382582+0000","last_clean":"2024-03-11T21:11:52.382582+0000","last_became_active":"2024-03-11T21:11:47.559993+0000","last_became_peered":"2024-03-11T21:11:47.559993+0000","last_unstale":"2024-03-11T21:11:52.382582+0000","last_undegraded":"2024-03-11T21:11:52.382582+0000","last_fullsized":"2024-03-11T21:11:52.382582+0000","mapping_epoch":380,"log_start":"0'0","ondisk_log_start":"0'0","created":380,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:11:46.536415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:11:46.536415+0000","last_clean_scrub_stamp":"2024-03-11T21:11:46.536415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:13:12.932615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"184'130","reported_seq":997,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:11:53.026209+0000","last_change":"2024-03-11T21:11:28.392203+0000","last_active":"2024-03-11T21:11:53.026209+0000","last_peered":"2024-03-11T21:11:53.026209+0000","last_clean":"2024-03-11T21:11:53.026209+0000","last_became_active":"2024-03-11T21:11:28.391741+0000","last_became_peered":"2024-03-11T21:11:28.391741+0000","last_unstale":"2024-03-11T21:11:53.026209+0000","last_undegraded":"2024-03-11T21:11:53.026209+0000","last_fullsized":"2024-03-11T21:11:53.026209+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":4,"last_scrub":"184'130","last_scrub_stamp":"2024-03-11T21:09:59.511442+0000","last_deep_scrub":"184'130","last_deep_scrub_stamp":"2024-03-11T20:58:56.415267+0000","last_clean_scrub_stamp":"2024-03-11T21:09:59.511442+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:55.010898+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":20,"num_read_kb":28,"num_write":17,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":161010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":708,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:11:53.028697+0000","last_change":"2024-03-11T21:11:20.989703+0000","last_active":"2024-03-11T21:11:53.028697+0000","last_peered":"2024-03-11T21:11:53.028697+0000","last_clean":"2024-03-11T21:11:53.028697+0000","last_became_active":"2024-03-11T20:50:34.321038+0000","last_became_peered":"2024-03-11T20:50:34.321038+0000","last_unstale":"2024-03-11T21:11:53.028697+0000","last_undegraded":"2024-03-11T21:11:53.028697+0000","last_fullsized":"2024-03-11T21:11:53.028697+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:11:20.989384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:11:20.989384+0000","last_clean_scrub_stamp":"2024-03-11T21:11:20.989384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:12:27.623922+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"358'50","reported_seq":460,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-03-11T21:11:50.351688+0000","last_change":"2024-03-11T21:10:02.247772+0000","last_active":"2024-03-11T21:11:50.351688+0000","last_peered":"2024-03-11T21:11:50.351688+0000","last_clean":"2024-03-11T21:11:50.351688+0000","last_became_active":"2024-03-11T21:00:48.456178+0000","last_became_peered":"2024-03-11T21:00:48.456178+0000","last_unstale":"2024-03-11T21:11:50.351688+0000","last_undegraded":"2024-03-11T21:11:50.351688+0000","last_fullsized":"2024-03-11T21:11:50.351688+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'38","last_scrub_stamp":"2024-03-11T21:10:02.247651+0000","last_deep_scrub":"109'26","last_deep_scrub_stamp":"2024-03-11T20:52:49.216378+0000","last_clean_scrub_stamp":"2024-03-11T21:10:02.247651+0000","objects_scrubbed":8,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:53.672905+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":58,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":150,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:11:52.383589+0000","last_change":"2024-03-11T21:10:55.016392+0000","last_active":"2024-03-11T21:11:52.383589+0000","last_peered":"2024-03-11T21:11:52.383589+0000","last_clean":"2024-03-11T21:11:52.383589+0000","last_became_active":"2024-03-11T21:08:05.308608+0000","last_became_peered":"2024-03-11T21:08:05.308608+0000","last_unstale":"2024-03-11T21:11:52.383589+0000","last_undegraded":"2024-03-11T21:11:52.383589+0000","last_fullsized":"2024-03-11T21:11:52.383589+0000","mapping_epoch":319,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:10:55.016272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:10:55.016272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:12:06.189069+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":410,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-03-11T21:11:50.359039+0000","last_change":"2024-03-11T21:11:01.676556+0000","last_active":"2024-03-11T21:11:50.359039+0000","last_peered":"2024-03-11T21:11:50.359039+0000","last_clean":"2024-03-11T21:11:50.359039+0000","last_became_active":"2024-03-11T21:00:48.473346+0000","last_became_peered":"2024-03-11T21:00:48.473346+0000","last_unstale":"2024-03-11T21:11:50.359039+0000","last_undegraded":"2024-03-11T21:11:50.359039+0000","last_fullsized":"2024-03-11T21:11:50.359039+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:11:01.676437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:11:01.676437+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:12:30.656233+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":17,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:11:52.383772+0000","last_change":"2024-03-11T21:11:47.562755+0000","last_active":"2024-03-11T21:11:52.383772+0000","last_peered":"2024-03-11T21:11:52.383772+0000","last_clean":"2024-03-11T21:11:52.383772+0000","last_became_active":"2024-03-11T21:11:47.562374+0000","last_became_peered":"2024-03-11T21:11:47.562374+0000","last_unstale":"2024-03-11T21:11:52.383772+0000","last_undegraded":"2024-03-11T21:11:52.383772+0000","last_fullsized":"2024-03-11T21:11:52.383772+0000","mapping_epoch":380,"log_start":"0'0","ondisk_log_start":"0'0","created":380,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:11:46.536415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:11:46.536415+0000","last_clean_scrub_stamp":"2024-03-11T21:11:46.536415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:12:54.040998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":692,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-03-11T21:11:50.585834+0000","last_change":"2024-03-11T21:09:32.825302+0000","last_active":"2024-03-11T21:11:50.585834+0000","last_peered":"2024-03-11T21:11:50.585834+0000","last_clean":"2024-03-11T21:11:50.585834+0000","last_became_active":"2024-03-11T21:00:27.962668+0000","last_became_peered":"2024-03-11T21:00:27.962668+0000","last_unstale":"2024-03-11T21:11:50.585834+0000","last_undegraded":"2024-03-11T21:11:50.585834+0000","last_fullsized":"2024-03-11T21:11:50.585834+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:32.825166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:09:32.825166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:51.330794+0000","scrub_duration":95,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"358'35","reported_seq":744,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:11:53.948992+0000","last_change":"2024-03-11T21:11:53.949072+0000","last_active":"2024-03-11T21:11:53.948992+0000","last_peered":"2024-03-11T21:11:53.948992+0000","last_clean":"2024-03-11T21:11:53.948992+0000","last_became_active":"2024-03-11T20:50:30.101747+0000","last_became_peered":"2024-03-11T20:50:30.101747+0000","last_unstale":"2024-03-11T21:11:53.948992+0000","last_undegraded":"2024-03-11T21:11:53.948992+0000","last_fullsized":"2024-03-11T21:11:53.948992+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'35","last_scrub_stamp":"2024-03-11T21:11:53.948819+0000","last_deep_scrub":"284'25","last_deep_scrub_stamp":"2024-03-11T21:04:53.217188+0000","last_clean_scrub_stamp":"2024-03-11T21:11:53.948819+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:12:57.522771+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":712,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-03-11T21:11:50.583390+0000","last_change":"2024-03-11T21:09:56.781956+0000","last_active":"2024-03-11T21:11:50.583390+0000","last_peered":"2024-03-11T21:11:50.583390+0000","last_clean":"2024-03-11T21:11:50.583390+0000","last_became_active":"2024-03-11T20:50:28.085503+0000","last_became_peered":"2024-03-11T20:50:28.085503+0000","last_unstale":"2024-03-11T21:11:50.583390+0000","last_undegraded":"2024-03-11T21:11:50.583390+0000","last_fullsized":"2024-03-11T21:11:50.583390+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:56.781841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:56:03.479180+0000","last_clean_scrub_stamp":"2024-03-11T21:09:56.781841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:51.331182+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"184'130","reported_seq":989,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:11:53.027406+0000","last_change":"2024-03-11T21:11:24.103689+0000","last_active":"2024-03-11T21:11:53.027406+0000","last_peered":"2024-03-11T21:11:53.027406+0000","last_clean":"2024-03-11T21:11:53.027406+0000","last_became_active":"2024-03-11T21:11:24.103207+0000","last_became_peered":"2024-03-11T21:11:24.103207+0000","last_unstale":"2024-03-11T21:11:53.027406+0000","last_undegraded":"2024-03-11T21:11:53.027406+0000","last_fullsized":"2024-03-11T21:11:53.027406+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"184'130","last_scrub_stamp":"2024-03-11T21:09:59.511442+0000","last_deep_scrub":"184'130","last_deep_scrub_stamp":"2024-03-11T20:58:56.415267+0000","last_clean_scrub_stamp":"2024-03-11T21:09:59.511442+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:54.026471+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53669,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":27,"num_write":16,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":161009,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":729,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:11:54.931179+0000","last_change":"2024-03-11T21:11:54.931292+0000","last_active":"2024-03-11T21:11:54.931179+0000","last_peered":"2024-03-11T21:11:54.931179+0000","last_clean":"2024-03-11T21:11:54.931179+0000","last_became_active":"2024-03-11T21:00:27.959467+0000","last_became_peered":"2024-03-11T21:00:27.959467+0000","last_unstale":"2024-03-11T21:11:54.931179+0000","last_undegraded":"2024-03-11T21:11:54.931179+0000","last_fullsized":"2024-03-11T21:11:54.931179+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:11:54.930990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:54:22.448812+0000","last_clean_scrub_stamp":"2024-03-11T21:11:54.930990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:13:06.584136+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":17,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:11:52.383768+0000","last_change":"2024-03-11T21:11:47.562662+0000","last_active":"2024-03-11T21:11:52.383768+0000","last_peered":"2024-03-11T21:11:52.383768+0000","last_clean":"2024-03-11T21:11:52.383768+0000","last_became_active":"2024-03-11T21:11:47.562305+0000","last_became_peered":"2024-03-11T21:11:47.562305+0000","last_unstale":"2024-03-11T21:11:52.383768+0000","last_undegraded":"2024-03-11T21:11:52.383768+0000","last_fullsized":"2024-03-11T21:11:52.383768+0000","mapping_epoch":380,"log_start":"0'0","ondisk_log_start":"0'0","created":380,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:11:46.536415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:11:46.536415+0000","last_clean_scrub_stamp":"2024-03-11T21:11:46.536415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:13:08.293266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"184'130","reported_seq":558,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-03-11T21:11:50.355427+0000","last_change":"2024-03-11T21:11:28.399733+0000","last_active":"2024-03-11T21:11:50.355427+0000","last_peered":"2024-03-11T21:11:50.355427+0000","last_clean":"2024-03-11T21:11:50.355427+0000","last_became_active":"2024-03-11T21:11:28.397947+0000","last_became_peered":"2024-03-11T21:11:28.397947+0000","last_unstale":"2024-03-11T21:11:50.355427+0000","last_undegraded":"2024-03-11T21:11:50.355427+0000","last_fullsized":"2024-03-11T21:11:50.355427+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":3,"last_scrub":"184'130","last_scrub_stamp":"2024-03-11T21:09:59.511442+0000","last_deep_scrub":"184'130","last_deep_scrub_stamp":"2024-03-11T20:58:56.415267+0000","last_clean_scrub_stamp":"2024-03-11T21:09:59.511442+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:50.608593+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":28,"num_write":17,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":161010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":410,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-03-11T21:11:50.356847+0000","last_change":"2024-03-11T21:09:56.170013+0000","last_active":"2024-03-11T21:11:50.356847+0000","last_peered":"2024-03-11T21:11:50.356847+0000","last_clean":"2024-03-11T21:11:50.356847+0000","last_became_active":"2024-03-11T21:00:48.464596+0000","last_became_peered":"2024-03-11T21:00:48.464596+0000","last_unstale":"2024-03-11T21:11:50.356847+0000","last_undegraded":"2024-03-11T21:11:50.356847+0000","last_fullsized":"2024-03-11T21:11:50.356847+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:56.169836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:09:56.169836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:53.672829+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"358'49","reported_seq":409,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-03-11T21:11:48.551690+0000","last_change":"2024-03-11T21:10:58.718141+0000","last_active":"2024-03-11T21:11:48.551690+0000","last_peered":"2024-03-11T21:11:48.551690+0000","last_clean":"2024-03-11T21:11:48.551690+0000","last_became_active":"2024-03-11T21:06:28.761186+0000","last_became_peered":"2024-03-11T21:06:28.761186+0000","last_unstale":"2024-03-11T21:11:48.551690+0000","last_undegraded":"2024-03-11T21:11:48.551690+0000","last_fullsized":"2024-03-11T21:11:48.551690+0000","mapping_epoch":297,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'49","last_scrub_stamp":"2024-03-11T21:10:58.718016+0000","last_deep_scrub":"100'21","last_deep_scrub_stamp":"2024-03-11T20:51:31.835302+0000","last_clean_scrub_stamp":"2024-03-11T21:10:58.718016+0000","objects_scrubbed":7,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:12:04.295700+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":55,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":448,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-03-11T21:11:50.352785+0000","last_change":"2024-03-11T21:11:07.632770+0000","last_active":"2024-03-11T21:11:50.352785+0000","last_peered":"2024-03-11T21:11:50.352785+0000","last_clean":"2024-03-11T21:11:50.352785+0000","last_became_active":"2024-03-11T21:08:05.301024+0000","last_became_peered":"2024-03-11T21:08:05.301024+0000","last_unstale":"2024-03-11T21:11:50.352785+0000","last_undegraded":"2024-03-11T21:11:50.352785+0000","last_fullsized":"2024-03-11T21:11:50.352785+0000","mapping_epoch":319,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:11:07.632642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:11:07.632642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:12:12.424478+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":719,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-03-11T21:11:48.553022+0000","last_change":"2024-03-11T21:11:15.884304+0000","last_active":"2024-03-11T21:11:48.553022+0000","last_peered":"2024-03-11T21:11:48.553022+0000","last_clean":"2024-03-11T21:11:48.553022+0000","last_became_active":"2024-03-11T21:00:48.463247+0000","last_became_peered":"2024-03-11T21:00:48.463247+0000","last_unstale":"2024-03-11T21:11:48.553022+0000","last_undegraded":"2024-03-11T21:11:48.553022+0000","last_fullsized":"2024-03-11T21:11:48.553022+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:11:15.883970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:26.839782+0000","last_clean_scrub_stamp":"2024-03-11T21:11:15.883970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:12:30.471956+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":15,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-03-11T21:11:50.357020+0000","last_change":"2024-03-11T21:11:47.552194+0000","last_active":"2024-03-11T21:11:50.357020+0000","last_peered":"2024-03-11T21:11:50.357020+0000","last_clean":"2024-03-11T21:11:50.357020+0000","last_became_active":"2024-03-11T21:11:47.551863+0000","last_became_peered":"2024-03-11T21:11:47.551863+0000","last_unstale":"2024-03-11T21:11:50.357020+0000","last_undegraded":"2024-03-11T21:11:50.357020+0000","last_fullsized":"2024-03-11T21:11:50.357020+0000","mapping_epoch":380,"log_start":"0'0","ondisk_log_start":"0'0","created":380,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:11:46.536415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:11:46.536415+0000","last_clean_scrub_stamp":"2024-03-11T21:11:46.536415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:13:06.780269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"184'130","reported_seq":565,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:11:52.916006+0000","last_change":"2024-03-11T21:11:28.402667+0000","last_active":"2024-03-11T21:11:52.916006+0000","last_peered":"2024-03-11T21:11:52.916006+0000","last_clean":"2024-03-11T21:11:52.916006+0000","last_became_active":"2024-03-11T21:11:28.402091+0000","last_became_peered":"2024-03-11T21:11:28.402091+0000","last_unstale":"2024-03-11T21:11:52.916006+0000","last_undegraded":"2024-03-11T21:11:52.916006+0000","last_fullsized":"2024-03-11T21:11:52.916006+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":3,"last_scrub":"184'130","last_scrub_stamp":"2024-03-11T21:09:59.511442+0000","last_deep_scrub":"184'130","last_deep_scrub_stamp":"2024-03-11T20:58:56.415267+0000","last_clean_scrub_stamp":"2024-03-11T21:09:59.511442+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:55.919533+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":28,"num_write":16,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":161009,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":418,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-03-11T21:11:50.354380+0000","last_change":"2024-03-11T21:11:10.720532+0000","last_active":"2024-03-11T21:11:50.354380+0000","last_peered":"2024-03-11T21:11:50.354380+0000","last_clean":"2024-03-11T21:11:50.354380+0000","last_became_active":"2024-03-11T21:00:48.469688+0000","last_became_peered":"2024-03-11T21:00:48.469688+0000","last_unstale":"2024-03-11T21:11:50.354380+0000","last_undegraded":"2024-03-11T21:11:50.354380+0000","last_fullsized":"2024-03-11T21:11:50.354380+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:11:10.720386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:11:10.720386+0000","last_clean_scrub_stamp":"2024-03-11T21:11:10.720386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:12:38.883835+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"145'3","reported_seq":722,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:11:52.917113+0000","last_change":"2024-03-11T21:10:55.753902+0000","last_active":"2024-03-11T21:11:52.917113+0000","last_peered":"2024-03-11T21:11:52.917113+0000","last_clean":"2024-03-11T21:11:52.917113+0000","last_became_active":"2024-03-11T20:50:32.195452+0000","last_became_peered":"2024-03-11T20:50:32.195452+0000","last_unstale":"2024-03-11T21:11:52.917113+0000","last_undegraded":"2024-03-11T21:11:52.917113+0000","last_fullsized":"2024-03-11T21:11:52.917113+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'3","last_scrub_stamp":"2024-03-11T21:10:55.753809+0000","last_deep_scrub":"145'3","last_deep_scrub_stamp":"2024-03-11T20:55:52.848043+0000","last_clean_scrub_stamp":"2024-03-11T21:10:55.753809+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:12:21.469593+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0.0024049620000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":713,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:11:53.028849+0000","last_change":"2024-03-11T21:09:55.530420+0000","last_active":"2024-03-11T21:11:53.028849+0000","last_peered":"2024-03-11T21:11:53.028849+0000","last_clean":"2024-03-11T21:11:53.028849+0000","last_became_active":"2024-03-11T21:00:27.966749+0000","last_became_peered":"2024-03-11T21:00:27.966749+0000","last_unstale":"2024-03-11T21:11:53.028849+0000","last_undegraded":"2024-03-11T21:11:53.028849+0000","last_fullsized":"2024-03-11T21:11:53.028849+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:55.530113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:09:55.530113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:54.026510+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"358'58","reported_seq":598,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-03-11T21:11:50.357349+0000","last_change":"2024-03-11T21:10:59.629789+0000","last_active":"2024-03-11T21:11:50.357349+0000","last_peered":"2024-03-11T21:11:50.357349+0000","last_clean":"2024-03-11T21:11:50.357349+0000","last_became_active":"2024-03-11T21:00:48.473227+0000","last_became_peered":"2024-03-11T21:00:48.473227+0000","last_unstale":"2024-03-11T21:11:50.357349+0000","last_undegraded":"2024-03-11T21:11:50.357349+0000","last_fullsized":"2024-03-11T21:11:50.357349+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'58","last_scrub_stamp":"2024-03-11T21:10:59.629631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:10:59.629631+0000","objects_scrubbed":10,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:12:09.434429+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":87,"num_read_kb":72,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"298'13","reported_seq":543,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-03-11T21:11:48.551672+0000","last_change":"2024-03-11T21:11:00.767241+0000","last_active":"2024-03-11T21:11:48.551672+0000","last_peered":"2024-03-11T21:11:48.551672+0000","last_clean":"2024-03-11T21:11:48.551672+0000","last_became_active":"2024-03-11T21:06:28.762397+0000","last_became_peered":"2024-03-11T21:06:28.762397+0000","last_unstale":"2024-03-11T21:11:48.551672+0000","last_undegraded":"2024-03-11T21:11:48.551672+0000","last_fullsized":"2024-03-11T21:11:48.551672+0000","mapping_epoch":297,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"298'13","last_scrub_stamp":"2024-03-11T21:11:00.767154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:11:00.767154+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:12:20.777271+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":17,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:11:53.029677+0000","last_change":"2024-03-11T21:11:47.576455+0000","last_active":"2024-03-11T21:11:53.029677+0000","last_peered":"2024-03-11T21:11:53.029677+0000","last_clean":"2024-03-11T21:11:53.029677+0000","last_became_active":"2024-03-11T21:11:47.575627+0000","last_became_peered":"2024-03-11T21:11:47.575627+0000","last_unstale":"2024-03-11T21:11:53.029677+0000","last_undegraded":"2024-03-11T21:11:53.029677+0000","last_fullsized":"2024-03-11T21:11:53.029677+0000","mapping_epoch":380,"log_start":"0'0","ondisk_log_start":"0'0","created":380,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:11:46.536415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:11:46.536415+0000","last_clean_scrub_stamp":"2024-03-11T21:11:46.536415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:12:54.534248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"184'130","reported_seq":565,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:11:52.916275+0000","last_change":"2024-03-11T21:11:28.397785+0000","last_active":"2024-03-11T21:11:52.916275+0000","last_peered":"2024-03-11T21:11:52.916275+0000","last_clean":"2024-03-11T21:11:52.916275+0000","last_became_active":"2024-03-11T21:11:28.397374+0000","last_became_peered":"2024-03-11T21:11:28.397374+0000","last_unstale":"2024-03-11T21:11:52.916275+0000","last_undegraded":"2024-03-11T21:11:52.916275+0000","last_fullsized":"2024-03-11T21:11:52.916275+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":3,"last_scrub":"184'130","last_scrub_stamp":"2024-03-11T21:09:59.511442+0000","last_deep_scrub":"184'130","last_deep_scrub_stamp":"2024-03-11T20:58:56.415267+0000","last_clean_scrub_stamp":"2024-03-11T21:09:59.511442+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:55.919445+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":28,"num_write":16,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":161010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":432,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:11:52.383876+0000","last_change":"2024-03-11T21:09:34.951119+0000","last_active":"2024-03-11T21:11:52.383876+0000","last_peered":"2024-03-11T21:11:52.383876+0000","last_clean":"2024-03-11T21:11:52.383876+0000","last_became_active":"2024-03-11T21:05:17.441521+0000","last_became_peered":"2024-03-11T21:05:17.441521+0000","last_unstale":"2024-03-11T21:11:52.383876+0000","last_undegraded":"2024-03-11T21:11:52.383876+0000","last_fullsized":"2024-03-11T21:11:52.383876+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:34.951014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:03:43.345257+0000","last_clean_scrub_stamp":"2024-03-11T21:09:34.951014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:53.045016+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":453,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:11:52.383934+0000","last_change":"2024-03-11T21:09:35.935667+0000","last_active":"2024-03-11T21:11:52.383934+0000","last_peered":"2024-03-11T21:11:52.383934+0000","last_clean":"2024-03-11T21:11:52.383934+0000","last_became_active":"2024-03-11T21:05:17.315844+0000","last_became_peered":"2024-03-11T21:05:17.315844+0000","last_unstale":"2024-03-11T21:11:52.383934+0000","last_undegraded":"2024-03-11T21:11:52.383934+0000","last_fullsized":"2024-03-11T21:11:52.383934+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:35.935559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:22.617999+0000","last_clean_scrub_stamp":"2024-03-11T21:09:35.935559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:53.045089+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"284'2","reported_seq":415,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-03-11T21:11:50.357535+0000","last_change":"2024-03-11T21:09:54.213683+0000","last_active":"2024-03-11T21:11:50.357535+0000","last_peered":"2024-03-11T21:11:50.357535+0000","last_clean":"2024-03-11T21:11:50.357535+0000","last_became_active":"2024-03-11T21:00:48.468682+0000","last_became_peered":"2024-03-11T21:00:48.468682+0000","last_unstale":"2024-03-11T21:11:50.357535+0000","last_undegraded":"2024-03-11T21:11:50.357535+0000","last_fullsized":"2024-03-11T21:11:50.357535+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"284'2","last_scrub_stamp":"2024-03-11T21:09:54.213473+0000","last_deep_scrub":"284'2","last_deep_scrub_stamp":"2024-03-11T21:06:47.645682+0000","last_clean_scrub_stamp":"2024-03-11T21:09:54.213473+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:53.672852+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"357'21","reported_seq":724,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:11:52.384729+0000","last_change":"2024-03-11T21:09:54.016666+0000","last_active":"2024-03-11T21:11:52.384729+0000","last_peered":"2024-03-11T21:11:52.384729+0000","last_clean":"2024-03-11T21:11:52.384729+0000","last_became_active":"2024-03-11T20:50:30.097669+0000","last_became_peered":"2024-03-11T20:50:30.097669+0000","last_unstale":"2024-03-11T21:11:52.384729+0000","last_undegraded":"2024-03-11T21:11:52.384729+0000","last_fullsized":"2024-03-11T21:11:52.384729+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'17","last_scrub_stamp":"2024-03-11T21:09:54.016577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:09:54.016577+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:53.045203+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":31,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"84'11","reported_seq":958,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:11:52.383188+0000","last_change":"2024-03-11T21:08:04.428744+0000","last_active":"2024-03-11T21:11:52.383188+0000","last_peered":"2024-03-11T21:11:52.383188+0000","last_clean":"2024-03-11T21:11:52.383188+0000","last_became_active":"2024-03-11T20:50:32.647041+0000","last_became_peered":"2024-03-11T20:50:32.647041+0000","last_unstale":"2024-03-11T21:11:52.383188+0000","last_undegraded":"2024-03-11T21:11:52.383188+0000","last_fullsized":"2024-03-11T21:11:52.383188+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:08:04.428606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:08:04.428606+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":83,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":15,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-03-11T21:11:50.358227+0000","last_change":"2024-03-11T21:11:47.571655+0000","last_active":"2024-03-11T21:11:50.358227+0000","last_peered":"2024-03-11T21:11:50.358227+0000","last_clean":"2024-03-11T21:11:50.358227+0000","last_became_active":"2024-03-11T21:11:47.570431+0000","last_became_peered":"2024-03-11T21:11:47.570431+0000","last_unstale":"2024-03-11T21:11:50.358227+0000","last_undegraded":"2024-03-11T21:11:50.358227+0000","last_fullsized":"2024-03-11T21:11:50.358227+0000","mapping_epoch":380,"log_start":"0'0","ondisk_log_start":"0'0","created":380,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:11:46.536415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:11:46.536415+0000","last_clean_scrub_stamp":"2024-03-11T21:11:46.536415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:13:11.644183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"184'130","reported_seq":560,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:11:52.383809+0000","last_change":"2024-03-11T21:11:28.400944+0000","last_active":"2024-03-11T21:11:52.383809+0000","last_peered":"2024-03-11T21:11:52.383809+0000","last_clean":"2024-03-11T21:11:52.383809+0000","last_became_active":"2024-03-11T21:11:28.393756+0000","last_became_peered":"2024-03-11T21:11:28.393756+0000","last_unstale":"2024-03-11T21:11:52.383809+0000","last_undegraded":"2024-03-11T21:11:52.383809+0000","last_fullsized":"2024-03-11T21:11:52.383809+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":3,"last_scrub":"184'130","last_scrub_stamp":"2024-03-11T21:09:59.511442+0000","last_deep_scrub":"184'130","last_deep_scrub_stamp":"2024-03-11T20:58:56.415267+0000","last_clean_scrub_stamp":"2024-03-11T21:09:59.511442+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:56.974943+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":28,"num_write":16,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":488690,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":708,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:11:53.027722+0000","last_change":"2024-03-11T21:10:02.768209+0000","last_active":"2024-03-11T21:11:53.027722+0000","last_peered":"2024-03-11T21:11:53.027722+0000","last_clean":"2024-03-11T21:11:53.027722+0000","last_became_active":"2024-03-11T20:50:34.326947+0000","last_became_peered":"2024-03-11T20:50:34.326947+0000","last_unstale":"2024-03-11T21:11:53.027722+0000","last_undegraded":"2024-03-11T21:11:53.027722+0000","last_fullsized":"2024-03-11T21:11:53.027722+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:10:02.768094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:06:43.271453+0000","last_clean_scrub_stamp":"2024-03-11T21:10:02.768094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:54.026544+0000","scrub_duration":219,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"120'6","reported_seq":1879,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:11:53.119334+0000","last_change":"2024-03-11T21:08:19.340719+0000","last_active":"2024-03-11T21:11:53.119334+0000","last_peered":"2024-03-11T21:11:53.119334+0000","last_clean":"2024-03-11T21:11:53.119334+0000","last_became_active":"2024-03-11T21:00:48.468997+0000","last_became_peered":"2024-03-11T21:00:48.468997+0000","last_unstale":"2024-03-11T21:11:53.119334+0000","last_undegraded":"2024-03-11T21:11:53.119334+0000","last_fullsized":"2024-03-11T21:11:53.119334+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"120'6","last_scrub_stamp":"2024-03-11T21:08:19.340643+0000","last_deep_scrub":"120'6","last_deep_scrub_stamp":"2024-03-11T20:54:26.468057+0000","last_clean_scrub_stamp":"2024-03-11T21:08:19.340643+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0035533370000000002,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1148,"num_read_kb":1143,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":727,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:11:52.918253+0000","last_change":"2024-03-11T21:10:47.579963+0000","last_active":"2024-03-11T21:11:52.918253+0000","last_peered":"2024-03-11T21:11:52.918253+0000","last_clean":"2024-03-11T21:11:52.918253+0000","last_became_active":"2024-03-11T20:50:28.071707+0000","last_became_peered":"2024-03-11T20:50:28.071707+0000","last_unstale":"2024-03-11T21:11:52.918253+0000","last_undegraded":"2024-03-11T21:11:52.918253+0000","last_fullsized":"2024-03-11T21:11:52.918253+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:10:47.579844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:23.417283+0000","last_clean_scrub_stamp":"2024-03-11T21:10:47.579844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:12:06.595102+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"358'36","reported_seq":825,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-03-11T21:11:48.552380+0000","last_change":"2024-03-11T21:10:55.759420+0000","last_active":"2024-03-11T21:11:48.552380+0000","last_peered":"2024-03-11T21:11:48.552380+0000","last_clean":"2024-03-11T21:11:48.552380+0000","last_became_active":"2024-03-11T21:00:48.461972+0000","last_became_peered":"2024-03-11T21:00:48.461972+0000","last_unstale":"2024-03-11T21:11:48.552380+0000","last_undegraded":"2024-03-11T21:11:48.552380+0000","last_fullsized":"2024-03-11T21:11:48.552380+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'36","last_scrub_stamp":"2024-03-11T21:10:55.759335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:10:55.759335+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:12:04.155539+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":480,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-03-11T21:11:50.582234+0000","last_change":"2024-03-11T21:09:51.796505+0000","last_active":"2024-03-11T21:11:50.582234+0000","last_peered":"2024-03-11T21:11:50.582234+0000","last_clean":"2024-03-11T21:11:50.582234+0000","last_became_active":"2024-03-11T21:05:17.315807+0000","last_became_peered":"2024-03-11T21:05:17.315807+0000","last_unstale":"2024-03-11T21:11:50.582234+0000","last_undegraded":"2024-03-11T21:11:50.582234+0000","last_fullsized":"2024-03-11T21:11:50.582234+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:51.796383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:09:51.796383+0000","last_clean_scrub_stamp":"2024-03-11T21:09:51.796383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:51.331047+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":17,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:11:52.916861+0000","last_change":"2024-03-11T21:11:47.567263+0000","last_active":"2024-03-11T21:11:52.916861+0000","last_peered":"2024-03-11T21:11:52.916861+0000","last_clean":"2024-03-11T21:11:52.916861+0000","last_became_active":"2024-03-11T21:11:47.562677+0000","last_became_peered":"2024-03-11T21:11:47.562677+0000","last_unstale":"2024-03-11T21:11:52.916861+0000","last_undegraded":"2024-03-11T21:11:52.916861+0000","last_fullsized":"2024-03-11T21:11:52.916861+0000","mapping_epoch":380,"log_start":"0'0","ondisk_log_start":"0'0","created":380,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:11:46.536415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:11:46.536415+0000","last_clean_scrub_stamp":"2024-03-11T21:11:46.536415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:13:11.190216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":677,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:11:52.383429+0000","last_change":"2024-03-11T21:09:40.876804+0000","last_active":"2024-03-11T21:11:52.383429+0000","last_peered":"2024-03-11T21:11:52.383429+0000","last_clean":"2024-03-11T21:11:52.383429+0000","last_became_active":"2024-03-11T20:50:34.335370+0000","last_became_peered":"2024-03-11T20:50:34.335370+0000","last_unstale":"2024-03-11T21:11:52.383429+0000","last_undegraded":"2024-03-11T21:11:52.383429+0000","last_fullsized":"2024-03-11T21:11:52.383429+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:40.876681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:09:40.876681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:53.045125+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"286'18","reported_seq":378,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:11:52.918063+0000","last_change":"2024-03-11T21:09:45.491799+0000","last_active":"2024-03-11T21:11:52.918063+0000","last_peered":"2024-03-11T21:11:52.918063+0000","last_clean":"2024-03-11T21:11:52.918063+0000","last_became_active":"2024-03-11T21:05:17.449213+0000","last_became_peered":"2024-03-11T21:05:17.449213+0000","last_unstale":"2024-03-11T21:11:52.918063+0000","last_undegraded":"2024-03-11T21:11:52.918063+0000","last_fullsized":"2024-03-11T21:11:52.918063+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"286'18","last_scrub_stamp":"2024-03-11T21:09:45.491546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:09:45.491546+0000","objects_scrubbed":1,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:54.878114+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":724,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:11:53.026983+0000","last_change":"2024-03-11T21:10:52.823384+0000","last_active":"2024-03-11T21:11:53.026983+0000","last_peered":"2024-03-11T21:11:53.026983+0000","last_clean":"2024-03-11T21:11:53.026983+0000","last_became_active":"2024-03-11T21:00:48.466102+0000","last_became_peered":"2024-03-11T21:00:48.466102+0000","last_unstale":"2024-03-11T21:11:53.026983+0000","last_undegraded":"2024-03-11T21:11:53.026983+0000","last_fullsized":"2024-03-11T21:11:53.026983+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:10:52.823180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:06:42.242801+0000","last_clean_scrub_stamp":"2024-03-11T21:10:52.823180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:12:10.873197+0000","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"357'24","reported_seq":424,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:11:53.027773+0000","last_change":"2024-03-11T21:10:57.865042+0000","last_active":"2024-03-11T21:11:53.027773+0000","last_peered":"2024-03-11T21:11:53.027773+0000","last_clean":"2024-03-11T21:11:53.027773+0000","last_became_active":"2024-03-11T21:08:05.298170+0000","last_became_peered":"2024-03-11T21:08:05.298170+0000","last_unstale":"2024-03-11T21:11:53.027773+0000","last_undegraded":"2024-03-11T21:11:53.027773+0000","last_fullsized":"2024-03-11T21:11:53.027773+0000","mapping_epoch":319,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":0,"last_scrub":"357'24","last_scrub_stamp":"2024-03-11T21:10:57.864805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:10:57.864805+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:12:07.587067+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":41,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":15,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-03-11T21:11:50.353048+0000","last_change":"2024-03-11T21:11:47.570843+0000","last_active":"2024-03-11T21:11:50.353048+0000","last_peered":"2024-03-11T21:11:50.353048+0000","last_clean":"2024-03-11T21:11:50.353048+0000","last_became_active":"2024-03-11T21:11:47.570517+0000","last_became_peered":"2024-03-11T21:11:47.570517+0000","last_unstale":"2024-03-11T21:11:50.353048+0000","last_undegraded":"2024-03-11T21:11:50.353048+0000","last_fullsized":"2024-03-11T21:11:50.353048+0000","mapping_epoch":380,"log_start":"0'0","ondisk_log_start":"0'0","created":380,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:11:46.536415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:11:46.536415+0000","last_clean_scrub_stamp":"2024-03-11T21:11:46.536415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:13:08.155576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"184'130","reported_seq":558,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-03-11T21:11:50.582242+0000","last_change":"2024-03-11T21:11:28.397635+0000","last_active":"2024-03-11T21:11:50.582242+0000","last_peered":"2024-03-11T21:11:50.582242+0000","last_clean":"2024-03-11T21:11:50.582242+0000","last_became_active":"2024-03-11T21:11:28.396897+0000","last_became_peered":"2024-03-11T21:11:28.396897+0000","last_unstale":"2024-03-11T21:11:50.582242+0000","last_undegraded":"2024-03-11T21:11:50.582242+0000","last_fullsized":"2024-03-11T21:11:50.582242+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":4,"last_scrub":"184'130","last_scrub_stamp":"2024-03-11T21:09:59.511442+0000","last_deep_scrub":"184'130","last_deep_scrub_stamp":"2024-03-11T20:58:56.415267+0000","last_clean_scrub_stamp":"2024-03-11T21:09:59.511442+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53669,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":27,"num_write":16,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":161009,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":712,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-03-11T21:11:50.585220+0000","last_change":"2024-03-11T21:10:50.652568+0000","last_active":"2024-03-11T21:11:50.585220+0000","last_peered":"2024-03-11T21:11:50.585220+0000","last_clean":"2024-03-11T21:11:50.585220+0000","last_became_active":"2024-03-11T21:00:48.460531+0000","last_became_peered":"2024-03-11T21:00:48.460531+0000","last_unstale":"2024-03-11T21:11:50.585220+0000","last_undegraded":"2024-03-11T21:11:50.585220+0000","last_fullsized":"2024-03-11T21:11:50.585220+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:10:50.652484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:27.775534+0000","last_clean_scrub_stamp":"2024-03-11T21:10:50.652484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:12:12.434661+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":417,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-03-11T21:11:50.358800+0000","last_change":"2024-03-11T21:09:53.196862+0000","last_active":"2024-03-11T21:11:50.358800+0000","last_peered":"2024-03-11T21:11:50.358800+0000","last_clean":"2024-03-11T21:11:50.358800+0000","last_became_active":"2024-03-11T21:00:48.469596+0000","last_became_peered":"2024-03-11T21:00:48.469596+0000","last_unstale":"2024-03-11T21:11:50.358800+0000","last_undegraded":"2024-03-11T21:11:50.358800+0000","last_fullsized":"2024-03-11T21:11:50.358800+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:53.196741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:09:53.196741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:53.672868+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":410,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-03-11T21:11:50.354712+0000","last_change":"2024-03-11T21:10:53.563996+0000","last_active":"2024-03-11T21:11:50.354712+0000","last_peered":"2024-03-11T21:11:50.354712+0000","last_clean":"2024-03-11T21:11:50.354712+0000","last_became_active":"2024-03-11T21:00:48.466084+0000","last_became_peered":"2024-03-11T21:00:48.466084+0000","last_unstale":"2024-03-11T21:11:50.354712+0000","last_undegraded":"2024-03-11T21:11:50.354712+0000","last_fullsized":"2024-03-11T21:11:50.354712+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:10:53.563854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:40.974147+0000","last_clean_scrub_stamp":"2024-03-11T21:10:53.563854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:12:20.012133+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"358'46","reported_seq":813,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:11:52.383049+0000","last_change":"2024-03-11T21:09:19.777457+0000","last_active":"2024-03-11T21:11:52.383049+0000","last_peered":"2024-03-11T21:11:52.383049+0000","last_clean":"2024-03-11T21:11:52.383049+0000","last_became_active":"2024-03-11T21:00:48.472191+0000","last_became_peered":"2024-03-11T21:00:48.472191+0000","last_unstale":"2024-03-11T21:11:52.383049+0000","last_undegraded":"2024-03-11T21:11:52.383049+0000","last_fullsized":"2024-03-11T21:11:52.383049+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'34","last_scrub_stamp":"2024-03-11T21:09:19.777317+0000","last_deep_scrub":"116'22","last_deep_scrub_stamp":"2024-03-11T20:53:47.364709+0000","last_clean_scrub_stamp":"2024-03-11T21:09:19.777317+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:53.044996+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":17,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:11:52.917385+0000","last_change":"2024-03-11T21:11:47.580286+0000","last_active":"2024-03-11T21:11:52.917385+0000","last_peered":"2024-03-11T21:11:52.917385+0000","last_clean":"2024-03-11T21:11:52.917385+0000","last_became_active":"2024-03-11T21:11:47.579025+0000","last_became_peered":"2024-03-11T21:11:47.579025+0000","last_unstale":"2024-03-11T21:11:52.917385+0000","last_undegraded":"2024-03-11T21:11:52.917385+0000","last_fullsized":"2024-03-11T21:11:52.917385+0000","mapping_epoch":380,"log_start":"0'0","ondisk_log_start":"0'0","created":380,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:11:46.536415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:11:46.536415+0000","last_clean_scrub_stamp":"2024-03-11T21:11:46.536415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:12:52.205691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"184'130","reported_seq":997,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:11:53.026195+0000","last_change":"2024-03-11T21:11:28.405619+0000","last_active":"2024-03-11T21:11:53.026195+0000","last_peered":"2024-03-11T21:11:53.026195+0000","last_clean":"2024-03-11T21:11:53.026195+0000","last_became_active":"2024-03-11T21:11:28.405250+0000","last_became_peered":"2024-03-11T21:11:28.405250+0000","last_unstale":"2024-03-11T21:11:53.026195+0000","last_undegraded":"2024-03-11T21:11:53.026195+0000","last_fullsized":"2024-03-11T21:11:53.026195+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":4,"last_scrub":"184'130","last_scrub_stamp":"2024-03-11T21:09:59.511442+0000","last_deep_scrub":"184'130","last_deep_scrub_stamp":"2024-03-11T20:58:56.415267+0000","last_clean_scrub_stamp":"2024-03-11T21:09:59.511442+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:55.010952+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":28,"num_write":16,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":161009,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":496,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-03-11T21:11:50.357215+0000","last_change":"2024-03-11T21:11:02.715877+0000","last_active":"2024-03-11T21:11:50.357215+0000","last_peered":"2024-03-11T21:11:50.357215+0000","last_clean":"2024-03-11T21:11:50.357215+0000","last_became_active":"2024-03-11T21:05:17.455236+0000","last_became_peered":"2024-03-11T21:05:17.455236+0000","last_unstale":"2024-03-11T21:11:50.357215+0000","last_undegraded":"2024-03-11T21:11:50.357215+0000","last_fullsized":"2024-03-11T21:11:50.357215+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:11:02.715745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:11:02.715745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:12:08.000615+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":418,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-03-11T21:11:50.355474+0000","last_change":"2024-03-11T21:11:18.625591+0000","last_active":"2024-03-11T21:11:50.355474+0000","last_peered":"2024-03-11T21:11:50.355474+0000","last_clean":"2024-03-11T21:11:50.355474+0000","last_became_active":"2024-03-11T21:00:48.461367+0000","last_became_peered":"2024-03-11T21:00:48.461367+0000","last_unstale":"2024-03-11T21:11:50.355474+0000","last_undegraded":"2024-03-11T21:11:50.355474+0000","last_fullsized":"2024-03-11T21:11:50.355474+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:11:18.625296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:03:37.601229+0000","last_clean_scrub_stamp":"2024-03-11T21:11:18.625296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:12:45.841157+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"358'25","reported_seq":732,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:11:52.383141+0000","last_change":"2024-03-11T21:08:28.368669+0000","last_active":"2024-03-11T21:11:52.383141+0000","last_peered":"2024-03-11T21:11:52.383141+0000","last_clean":"2024-03-11T21:11:52.383141+0000","last_became_active":"2024-03-11T21:00:27.967572+0000","last_became_peered":"2024-03-11T21:00:27.967572+0000","last_unstale":"2024-03-11T21:11:52.383141+0000","last_undegraded":"2024-03-11T21:11:52.383141+0000","last_fullsized":"2024-03-11T21:11:52.383141+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"267'19","last_scrub_stamp":"2024-03-11T21:08:28.368557+0000","last_deep_scrub":"267'19","last_deep_scrub_stamp":"2024-03-11T21:03:39.673317+0000","last_clean_scrub_stamp":"2024-03-11T21:08:28.368557+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:53.044859+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":483,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:11:52.384202+0000","last_change":"2024-03-11T21:11:20.159860+0000","last_active":"2024-03-11T21:11:52.384202+0000","last_peered":"2024-03-11T21:11:52.384202+0000","last_clean":"2024-03-11T21:11:52.384202+0000","last_became_active":"2024-03-11T21:05:17.316136+0000","last_became_peered":"2024-03-11T21:05:17.316136+0000","last_unstale":"2024-03-11T21:11:52.384202+0000","last_undegraded":"2024-03-11T21:11:52.384202+0000","last_fullsized":"2024-03-11T21:11:52.384202+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:11:20.159640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:35.670149+0000","last_clean_scrub_stamp":"2024-03-11T21:11:20.159640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:12:36.960690+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":17,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:11:52.383528+0000","last_change":"2024-03-11T21:11:47.556789+0000","last_active":"2024-03-11T21:11:52.383528+0000","last_peered":"2024-03-11T21:11:52.383528+0000","last_clean":"2024-03-11T21:11:52.383528+0000","last_became_active":"2024-03-11T21:11:47.556451+0000","last_became_peered":"2024-03-11T21:11:47.556451+0000","last_unstale":"2024-03-11T21:11:52.383528+0000","last_undegraded":"2024-03-11T21:11:52.383528+0000","last_fullsized":"2024-03-11T21:11:52.383528+0000","mapping_epoch":380,"log_start":"0'0","ondisk_log_start":"0'0","created":380,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:11:46.536415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:11:46.536415+0000","last_clean_scrub_stamp":"2024-03-11T21:11:46.536415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:12:59.526480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"184'130","reported_seq":558,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-03-11T21:11:50.581761+0000","last_change":"2024-03-11T21:11:28.399218+0000","last_active":"2024-03-11T21:11:50.581761+0000","last_peered":"2024-03-11T21:11:50.581761+0000","last_clean":"2024-03-11T21:11:50.581761+0000","last_became_active":"2024-03-11T21:11:28.398728+0000","last_became_peered":"2024-03-11T21:11:28.398728+0000","last_unstale":"2024-03-11T21:11:50.581761+0000","last_undegraded":"2024-03-11T21:11:50.581761+0000","last_fullsized":"2024-03-11T21:11:50.581761+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":4,"last_scrub":"184'130","last_scrub_stamp":"2024-03-11T21:09:59.511442+0000","last_deep_scrub":"184'130","last_deep_scrub_stamp":"2024-03-11T20:58:56.415267+0000","last_clean_scrub_stamp":"2024-03-11T21:09:59.511442+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":28,"num_write":16,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":423697,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":454,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:11:52.384613+0000","last_change":"2024-03-11T21:10:03.144979+0000","last_active":"2024-03-11T21:11:52.384613+0000","last_peered":"2024-03-11T21:11:52.384613+0000","last_clean":"2024-03-11T21:11:52.384613+0000","last_became_active":"2024-03-11T21:08:05.304963+0000","last_became_peered":"2024-03-11T21:08:05.304963+0000","last_unstale":"2024-03-11T21:11:52.384613+0000","last_undegraded":"2024-03-11T21:11:52.384613+0000","last_fullsized":"2024-03-11T21:11:52.384613+0000","mapping_epoch":319,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:10:03.144875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:10:03.144875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:53.045108+0000","scrub_duration":239,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"320'21","reported_seq":692,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-03-11T21:11:50.583639+0000","last_change":"2024-03-11T21:09:46.828876+0000","last_active":"2024-03-11T21:11:50.583639+0000","last_peered":"2024-03-11T21:11:50.583639+0000","last_clean":"2024-03-11T21:11:50.583639+0000","last_became_active":"2024-03-11T21:08:05.298887+0000","last_became_peered":"2024-03-11T21:08:05.298887+0000","last_unstale":"2024-03-11T21:11:50.583639+0000","last_undegraded":"2024-03-11T21:11:50.583639+0000","last_fullsized":"2024-03-11T21:11:50.583639+0000","mapping_epoch":319,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'21","last_scrub_stamp":"2024-03-11T21:09:46.828795+0000","last_deep_scrub":"146'14","last_deep_scrub_stamp":"2024-03-11T20:56:00.242211+0000","last_clean_scrub_stamp":"2024-03-11T21:09:46.828795+0000","objects_scrubbed":1,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:51.330955+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":720,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:11:53.028163+0000","last_change":"2024-03-11T21:11:14.837109+0000","last_active":"2024-03-11T21:11:53.028163+0000","last_peered":"2024-03-11T21:11:53.028163+0000","last_clean":"2024-03-11T21:11:53.028163+0000","last_became_active":"2024-03-11T21:00:27.970573+0000","last_became_peered":"2024-03-11T21:00:27.970573+0000","last_unstale":"2024-03-11T21:11:53.028163+0000","last_undegraded":"2024-03-11T21:11:53.028163+0000","last_fullsized":"2024-03-11T21:11:53.028163+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:11:14.836808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:11:14.836808+0000","last_clean_scrub_stamp":"2024-03-11T21:11:14.836808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:12:28.578986+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"358'60","reported_seq":880,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:11:52.915937+0000","last_change":"2024-03-11T21:09:48.540145+0000","last_active":"2024-03-11T21:11:52.915937+0000","last_peered":"2024-03-11T21:11:52.915937+0000","last_clean":"2024-03-11T21:11:52.915937+0000","last_became_active":"2024-03-11T21:00:48.469854+0000","last_became_peered":"2024-03-11T21:00:48.469854+0000","last_unstale":"2024-03-11T21:11:52.915937+0000","last_undegraded":"2024-03-11T21:11:52.915937+0000","last_fullsized":"2024-03-11T21:11:52.915937+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'44","last_scrub_stamp":"2024-03-11T21:09:48.539752+0000","last_deep_scrub":"161'28","last_deep_scrub_stamp":"2024-03-11T20:57:20.816278+0000","last_clean_scrub_stamp":"2024-03-11T21:09:48.539752+0000","objects_scrubbed":9,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:54.878214+0000","scrub_duration":59,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":80,"num_read_kb":69,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":17,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:11:53.029039+0000","last_change":"2024-03-11T21:11:47.579128+0000","last_active":"2024-03-11T21:11:53.029039+0000","last_peered":"2024-03-11T21:11:53.029039+0000","last_clean":"2024-03-11T21:11:53.029039+0000","last_became_active":"2024-03-11T21:11:47.578368+0000","last_became_peered":"2024-03-11T21:11:47.578368+0000","last_unstale":"2024-03-11T21:11:53.029039+0000","last_undegraded":"2024-03-11T21:11:53.029039+0000","last_fullsized":"2024-03-11T21:11:53.029039+0000","mapping_epoch":380,"log_start":"0'0","ondisk_log_start":"0'0","created":380,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:11:46.536415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:11:46.536415+0000","last_clean_scrub_stamp":"2024-03-11T21:11:46.536415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:12:59.138785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":699,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:11:53.027394+0000","last_change":"2024-03-11T21:09:53.436563+0000","last_active":"2024-03-11T21:11:53.027394+0000","last_peered":"2024-03-11T21:11:53.027394+0000","last_clean":"2024-03-11T21:11:53.027394+0000","last_became_active":"2024-03-11T20:50:34.327139+0000","last_became_peered":"2024-03-11T20:50:34.327139+0000","last_unstale":"2024-03-11T21:11:53.027394+0000","last_undegraded":"2024-03-11T21:11:53.027394+0000","last_fullsized":"2024-03-11T21:11:53.027394+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:53.436482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:09:53.436482+0000","last_clean_scrub_stamp":"2024-03-11T21:09:53.436482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:54.026826+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":156,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-03-11T21:11:50.585041+0000","last_change":"2024-03-11T21:10:57.527045+0000","last_active":"2024-03-11T21:11:50.585041+0000","last_peered":"2024-03-11T21:11:50.585041+0000","last_clean":"2024-03-11T21:11:50.585041+0000","last_became_active":"2024-03-11T21:08:05.303226+0000","last_became_peered":"2024-03-11T21:08:05.303226+0000","last_unstale":"2024-03-11T21:11:50.585041+0000","last_undegraded":"2024-03-11T21:11:50.585041+0000","last_fullsized":"2024-03-11T21:11:50.585041+0000","mapping_epoch":319,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:10:57.526936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:57:17.447191+0000","last_clean_scrub_stamp":"2024-03-11T21:10:57.526936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:12:19.908586+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"358'53","reported_seq":793,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:11:53.029416+0000","last_change":"2024-03-11T21:11:15.839953+0000","last_active":"2024-03-11T21:11:53.029416+0000","last_peered":"2024-03-11T21:11:53.029416+0000","last_clean":"2024-03-11T21:11:53.029416+0000","last_became_active":"2024-03-11T20:50:30.111819+0000","last_became_peered":"2024-03-11T20:50:30.111819+0000","last_unstale":"2024-03-11T21:11:53.029416+0000","last_undegraded":"2024-03-11T21:11:53.029416+0000","last_fullsized":"2024-03-11T21:11:53.029416+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'53","last_scrub_stamp":"2024-03-11T21:11:15.839681+0000","last_deep_scrub":"102'25","last_deep_scrub_stamp":"2024-03-11T20:51:35.187094+0000","last_clean_scrub_stamp":"2024-03-11T21:11:15.839681+0000","objects_scrubbed":8,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:12:33.041958+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":61,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":732,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:11:52.916488+0000","last_change":"2024-03-11T21:11:14.780887+0000","last_active":"2024-03-11T21:11:52.916488+0000","last_peered":"2024-03-11T21:11:52.916488+0000","last_clean":"2024-03-11T21:11:52.916488+0000","last_became_active":"2024-03-11T21:00:48.362389+0000","last_became_peered":"2024-03-11T21:00:48.362389+0000","last_unstale":"2024-03-11T21:11:52.916488+0000","last_undegraded":"2024-03-11T21:11:52.916488+0000","last_fullsized":"2024-03-11T21:11:52.916488+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:11:14.780672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:11:14.780672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:12:42.753588+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":17,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:11:53.028272+0000","last_change":"2024-03-11T21:11:47.572412+0000","last_active":"2024-03-11T21:11:53.028272+0000","last_peered":"2024-03-11T21:11:53.028272+0000","last_clean":"2024-03-11T21:11:53.028272+0000","last_became_active":"2024-03-11T21:11:47.571530+0000","last_became_peered":"2024-03-11T21:11:47.571530+0000","last_unstale":"2024-03-11T21:11:53.028272+0000","last_undegraded":"2024-03-11T21:11:53.028272+0000","last_fullsized":"2024-03-11T21:11:53.028272+0000","mapping_epoch":380,"log_start":"0'0","ondisk_log_start":"0'0","created":380,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:11:46.536415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:11:46.536415+0000","last_clean_scrub_stamp":"2024-03-11T21:11:46.536415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:13:05.170579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":399,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-03-11T21:11:48.551929+0000","last_change":"2024-03-11T21:11:21.007674+0000","last_active":"2024-03-11T21:11:48.551929+0000","last_peered":"2024-03-11T21:11:48.551929+0000","last_clean":"2024-03-11T21:11:48.551929+0000","last_became_active":"2024-03-11T21:06:28.758451+0000","last_became_peered":"2024-03-11T21:06:28.758451+0000","last_unstale":"2024-03-11T21:11:48.551929+0000","last_undegraded":"2024-03-11T21:11:48.551929+0000","last_fullsized":"2024-03-11T21:11:48.551929+0000","mapping_epoch":297,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:11:21.007497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:55:58.888890+0000","last_clean_scrub_stamp":"2024-03-11T21:11:21.007497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:12:40.311148+0000","scrub_duration":71,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":409,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-03-11T21:11:50.357659+0000","last_change":"2024-03-11T21:10:58.600491+0000","last_active":"2024-03-11T21:11:50.357659+0000","last_peered":"2024-03-11T21:11:50.357659+0000","last_clean":"2024-03-11T21:11:50.357659+0000","last_became_active":"2024-03-11T21:00:48.469015+0000","last_became_peered":"2024-03-11T21:00:48.469015+0000","last_unstale":"2024-03-11T21:11:50.357659+0000","last_undegraded":"2024-03-11T21:11:50.357659+0000","last_fullsized":"2024-03-11T21:11:50.357659+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:10:58.600345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:10:58.600345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:11:59.952066+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"358'30","reported_seq":752,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-03-11T21:11:50.584728+0000","last_change":"2024-03-11T21:11:12.404722+0000","last_active":"2024-03-11T21:11:50.584728+0000","last_peered":"2024-03-11T21:11:50.584728+0000","last_clean":"2024-03-11T21:11:50.584728+0000","last_became_active":"2024-03-11T20:50:30.109549+0000","last_became_peered":"2024-03-11T20:50:30.109549+0000","last_unstale":"2024-03-11T21:11:50.584728+0000","last_undegraded":"2024-03-11T21:11:50.584728+0000","last_fullsized":"2024-03-11T21:11:50.584728+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'30","last_scrub_stamp":"2024-03-11T21:11:12.404533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:11:12.404533+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:12:19.965700+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":738,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-03-11T21:11:48.551805+0000","last_change":"2024-03-11T21:09:49.344698+0000","last_active":"2024-03-11T21:11:48.551805+0000","last_peered":"2024-03-11T21:11:48.551805+0000","last_clean":"2024-03-11T21:11:48.551805+0000","last_became_active":"2024-03-11T21:02:37.517748+0000","last_became_peered":"2024-03-11T21:02:37.517748+0000","last_unstale":"2024-03-11T21:11:48.551805+0000","last_undegraded":"2024-03-11T21:11:48.551805+0000","last_fullsized":"2024-03-11T21:11:48.551805+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:49.344594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:52:51.931295+0000","last_clean_scrub_stamp":"2024-03-11T21:09:49.344594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":15,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-03-11T21:11:50.584217+0000","last_change":"2024-03-11T21:11:47.566051+0000","last_active":"2024-03-11T21:11:50.584217+0000","last_peered":"2024-03-11T21:11:50.584217+0000","last_clean":"2024-03-11T21:11:50.584217+0000","last_became_active":"2024-03-11T21:11:47.564750+0000","last_became_peered":"2024-03-11T21:11:47.564750+0000","last_unstale":"2024-03-11T21:11:50.584217+0000","last_undegraded":"2024-03-11T21:11:50.584217+0000","last_fullsized":"2024-03-11T21:11:50.584217+0000","mapping_epoch":380,"log_start":"0'0","ondisk_log_start":"0'0","created":380,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:11:46.536415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:11:46.536415+0000","last_clean_scrub_stamp":"2024-03-11T21:11:46.536415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:12:57.504951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":169,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-03-11T21:11:48.552343+0000","last_change":"2024-03-11T21:10:50.090657+0000","last_active":"2024-03-11T21:11:48.552343+0000","last_peered":"2024-03-11T21:11:48.552343+0000","last_clean":"2024-03-11T21:11:48.552343+0000","last_became_active":"2024-03-11T21:07:34.504579+0000","last_became_peered":"2024-03-11T21:07:34.504579+0000","last_unstale":"2024-03-11T21:11:48.552343+0000","last_undegraded":"2024-03-11T21:11:48.552343+0000","last_fullsized":"2024-03-11T21:11:48.552343+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:10:50.090575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:00:14.030599+0000","last_clean_scrub_stamp":"2024-03-11T21:10:50.090575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:12:10.607701+0000","scrub_duration":387,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":150,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:11:52.384440+0000","last_change":"2024-03-11T21:11:01.150241+0000","last_active":"2024-03-11T21:11:52.384440+0000","last_peered":"2024-03-11T21:11:52.384440+0000","last_clean":"2024-03-11T21:11:52.384440+0000","last_became_active":"2024-03-11T21:08:05.307984+0000","last_became_peered":"2024-03-11T21:08:05.307984+0000","last_unstale":"2024-03-11T21:11:52.384440+0000","last_undegraded":"2024-03-11T21:11:52.384440+0000","last_fullsized":"2024-03-11T21:11:52.384440+0000","mapping_epoch":319,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:11:01.150119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:11:01.150119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:12:24.345396+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.f","version":"358'39","reported_seq":769,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-11T21:11:53.029614+0000","last_change":"2024-03-11T21:10:48.857063+0000","last_active":"2024-03-11T21:11:53.029614+0000","last_peered":"2024-03-11T21:11:53.029614+0000","last_clean":"2024-03-11T21:11:53.029614+0000","last_became_active":"2024-03-11T21:00:27.963119+0000","last_became_peered":"2024-03-11T21:00:27.963119+0000","last_unstale":"2024-03-11T21:11:53.029614+0000","last_undegraded":"2024-03-11T21:11:53.029614+0000","last_fullsized":"2024-03-11T21:11:53.029614+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'39","last_scrub_stamp":"2024-03-11T21:10:48.856947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:10:48.856947+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:12:12.155046+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":703,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-03-11T21:11:50.582285+0000","last_change":"2024-03-11T21:11:18.372511+0000","last_active":"2024-03-11T21:11:50.582285+0000","last_peered":"2024-03-11T21:11:50.582285+0000","last_clean":"2024-03-11T21:11:50.582285+0000","last_became_active":"2024-03-11T20:50:32.652854+0000","last_became_peered":"2024-03-11T20:50:32.652854+0000","last_unstale":"2024-03-11T21:11:50.582285+0000","last_undegraded":"2024-03-11T21:11:50.582285+0000","last_fullsized":"2024-03-11T21:11:50.582285+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:11:18.372284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:59:00.946094+0000","last_clean_scrub_stamp":"2024-03-11T21:11:18.372284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:12:24.650806+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":20,"num_read_kb":20,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1822,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1430,"ondisk_log_size":1430,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1550,"num_read_kb":1307,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":171,"num_bytes_recovered":2228,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3632,"internal_metadata":0},"log_size":1066,"ondisk_log_size":1066,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":126,"ondisk_log_size":126,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1148,"num_read_kb":1143,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":68,"seq":292057776396,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":483732,"kb_used_data":1352,"kb_used_omap":1,"kb_used_meta":482366,"kb_avail":93261420,"statfs":{"total":95995035648,"available":95499694080,"internally_reserved":0,"allocated":1384448,"data_stored":970139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":493943242},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":57,"seq":244813136150,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":393156,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":392126,"kb_avail":93351996,"statfs":{"total":95995035648,"available":95592443904,"internally_reserved":0,"allocated":1040384,"data_stored":642469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":401537479},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":211,"seq":906238099590,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":362180,"kb_used_data":1012,"kb_used_omap":7,"kb_used_meta":361144,"kb_avail":93382972,"statfs":{"total":95995035648,"available":95624163328,"internally_reserved":0,"allocated":1036288,"data_stored":634474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8078,"internal_metadata":369811570},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738707,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":505592,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":504574,"kb_avail":93239560,"statfs":{"total":95995035648,"available":95477309440,"internally_reserved":0,"allocated":1024000,"data_stored":624919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":516684229},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378943,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":419776,"kb_used_data":1268,"kb_used_omap":1,"kb_used_meta":418494,"kb_avail":93325376,"statfs":{"total":95995035648,"available":95565185024,"internally_reserved":0,"allocated":1298432,"data_stored":904251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":428538309},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":28,"seq":120259084596,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":556160,"kb_used_data":1592,"kb_used_omap":1,"kb_used_meta":554558,"kb_avail":93188992,"statfs":{"total":95995035648,"available":95425527808,"internally_reserved":0,"allocated":1630208,"data_stored":1232307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":567867848},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":204,"seq":876173328522,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":294428,"kb_used_data":1036,"kb_used_omap":4,"kb_used_meta":293371,"kb_avail":93450724,"statfs":{"total":95995035648,"available":95693541376,"internally_reserved":0,"allocated":1060864,"data_stored":635088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4451,"internal_metadata":300412573},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":44,"seq":188978561311,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":456180,"kb_used_data":1568,"kb_used_omap":1,"kb_used_meta":454590,"kb_avail":93288972,"statfs":{"total":95995035648,"available":95527907328,"internally_reserved":0,"allocated":1605632,"data_stored":1215371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":465500613},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3632,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T21:11:57.390 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:57.414 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:57.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:57.416 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:11:57.517 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:11:57.546 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:11:57.548 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:57 smithi023 bash[29146]: audit 2024-03-11T21:11:55.902846+0000 mgr.y (mgr.24380) 782 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:11:57.548 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:57 smithi023 bash[29146]: cluster 2024-03-11T21:11:56.040142+0000 osd.7 (osd.7) 491 : cluster 0 1.2 scrub starts 2024-03-11T21:11:57.548 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:57 smithi023 bash[29146]: cluster 2024-03-11T21:11:56.041611+0000 osd.7 (osd.7) 492 : cluster 0 1.2 scrub ok 2024-03-11T21:11:57.548 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:57 smithi023 bash[29146]: cluster 2024-03-11T21:11:56.102763+0000 osd.0 (osd.0) 346 : cluster 0 6.18 scrub starts 2024-03-11T21:11:57.548 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:57 smithi023 bash[29146]: cluster 2024-03-11T21:11:56.158442+0000 osd.0 (osd.0) 347 : cluster 0 6.18 scrub ok 2024-03-11T21:11:57.548 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:57 smithi023 bash[29146]: audit 2024-03-11T21:11:56.260646+0000 mgr.y (mgr.24380) 783 : audit 0 from='client.15483 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:11:57.548 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:57 smithi023 bash[29146]: cluster 2024-03-11T21:11:56.360140+0000 mgr.y (mgr.24380) 784 : cluster 0 pgmap v898: 158 pgs: 158 active+clean; 582 KiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 851 B/s rd, 0 op/s 2024-03-11T21:11:57.548 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:57 smithi023 bash[29146]: cluster 2024-03-11T21:11:56.421709+0000 osd.5 (osd.5) 443 : cluster 0 1.a scrub starts 2024-03-11T21:11:57.548 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:57 smithi023 bash[29146]: cluster 2024-03-11T21:11:56.424034+0000 osd.5 (osd.5) 444 : cluster 0 1.a scrub ok 2024-03-11T21:11:57.549 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:57 smithi023 bash[29146]: cluster 2024-03-11T21:11:56.974609+0000 osd.6 (osd.6) 368 : cluster 0 1.5 scrub starts 2024-03-11T21:11:57.549 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:57 smithi023 bash[29146]: cluster 2024-03-11T21:11:56.977077+0000 osd.6 (osd.6) 369 : cluster 0 1.5 scrub ok 2024-03-11T21:11:57.549 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:57 smithi023 bash[21372]: audit 2024-03-11T21:11:55.902846+0000 mgr.y (mgr.24380) 782 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:11:57.549 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:57 smithi023 bash[21372]: cluster 2024-03-11T21:11:56.040142+0000 osd.7 (osd.7) 491 : cluster 0 1.2 scrub starts 2024-03-11T21:11:57.549 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:57 smithi023 bash[21372]: cluster 2024-03-11T21:11:56.041611+0000 osd.7 (osd.7) 492 : cluster 0 1.2 scrub ok 2024-03-11T21:11:57.549 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:57 smithi023 bash[21372]: cluster 2024-03-11T21:11:56.102763+0000 osd.0 (osd.0) 346 : cluster 0 6.18 scrub starts 2024-03-11T21:11:57.549 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:57 smithi023 bash[21372]: cluster 2024-03-11T21:11:56.158442+0000 osd.0 (osd.0) 347 : cluster 0 6.18 scrub ok 2024-03-11T21:11:57.549 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:57 smithi023 bash[21372]: audit 2024-03-11T21:11:56.260646+0000 mgr.y (mgr.24380) 783 : audit 0 from='client.15483 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:11:57.549 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:57 smithi023 bash[21372]: cluster 2024-03-11T21:11:56.360140+0000 mgr.y (mgr.24380) 784 : cluster 0 pgmap v898: 158 pgs: 158 active+clean; 582 KiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 851 B/s rd, 0 op/s 2024-03-11T21:11:57.549 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:57 smithi023 bash[21372]: cluster 2024-03-11T21:11:56.421709+0000 osd.5 (osd.5) 443 : cluster 0 1.a scrub starts 2024-03-11T21:11:57.549 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:57 smithi023 bash[21372]: cluster 2024-03-11T21:11:56.424034+0000 osd.5 (osd.5) 444 : cluster 0 1.a scrub ok 2024-03-11T21:11:57.550 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:57 smithi023 bash[21372]: cluster 2024-03-11T21:11:56.974609+0000 osd.6 (osd.6) 368 : cluster 0 1.5 scrub starts 2024-03-11T21:11:57.550 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:57 smithi023 bash[21372]: cluster 2024-03-11T21:11:56.977077+0000 osd.6 (osd.6) 369 : cluster 0 1.5 scrub ok 2024-03-11T21:11:57.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:57.550 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:11:57.651 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:11:57.677 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:11:57.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:57.679 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:11:57.682 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:11:57.682 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:11:57.757 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:57 smithi146 bash[23809]: audit 2024-03-11T21:11:55.902846+0000 mgr.y (mgr.24380) 782 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:11:57.757 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:57 smithi146 bash[23809]: cluster 2024-03-11T21:11:56.040142+0000 osd.7 (osd.7) 491 : cluster 0 1.2 scrub starts 2024-03-11T21:11:57.757 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:57 smithi146 bash[23809]: cluster 2024-03-11T21:11:56.041611+0000 osd.7 (osd.7) 492 : cluster 0 1.2 scrub ok 2024-03-11T21:11:57.758 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:57 smithi146 bash[23809]: cluster 2024-03-11T21:11:56.102763+0000 osd.0 (osd.0) 346 : cluster 0 6.18 scrub starts 2024-03-11T21:11:57.758 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:57 smithi146 bash[23809]: cluster 2024-03-11T21:11:56.158442+0000 osd.0 (osd.0) 347 : cluster 0 6.18 scrub ok 2024-03-11T21:11:57.758 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:57 smithi146 bash[23809]: audit 2024-03-11T21:11:56.260646+0000 mgr.y (mgr.24380) 783 : audit 0 from='client.15483 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:11:57.758 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:57 smithi146 bash[23809]: cluster 2024-03-11T21:11:56.360140+0000 mgr.y (mgr.24380) 784 : cluster 0 pgmap v898: 158 pgs: 158 active+clean; 582 KiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 851 B/s rd, 0 op/s 2024-03-11T21:11:57.758 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:57 smithi146 bash[23809]: cluster 2024-03-11T21:11:56.421709+0000 osd.5 (osd.5) 443 : cluster 0 1.a scrub starts 2024-03-11T21:11:57.758 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:57 smithi146 bash[23809]: cluster 2024-03-11T21:11:56.424034+0000 osd.5 (osd.5) 444 : cluster 0 1.a scrub ok 2024-03-11T21:11:57.758 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:57 smithi146 bash[23809]: cluster 2024-03-11T21:11:56.974609+0000 osd.6 (osd.6) 368 : cluster 0 1.5 scrub starts 2024-03-11T21:11:57.758 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:57 smithi146 bash[23809]: cluster 2024-03-11T21:11:56.977077+0000 osd.6 (osd.6) 369 : cluster 0 1.5 scrub ok 2024-03-11T21:11:57.781 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:57.807 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:57.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:57.809 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:11:57.910 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:57.938 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:57.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:57.940 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:11:58.041 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:58.068 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:58.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:58.070 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:11:58.171 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:11:58.202 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:11:58.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:58.205 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:11:58.306 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:58.334 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:58.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:58.336 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:11:58.437 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:58.462 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:58.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:58.465 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:11:58.527 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:11:58 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:11:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:11:58.527 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:58 smithi023 bash[21372]: cluster 2024-03-11T21:11:57.033948+0000 osd.7 (osd.7) 493 : cluster 0 1.9 scrub starts 2024-03-11T21:11:58.527 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:58 smithi023 bash[21372]: cluster 2024-03-11T21:11:57.035941+0000 osd.7 (osd.7) 494 : cluster 0 1.9 scrub ok 2024-03-11T21:11:58.527 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:58 smithi023 bash[21372]: cluster 2024-03-11T21:11:57.119723+0000 osd.0 (osd.0) 348 : cluster 0 3.3 scrub starts 2024-03-11T21:11:58.528 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:58 smithi023 bash[21372]: cluster 2024-03-11T21:11:57.124286+0000 osd.0 (osd.0) 349 : cluster 0 3.3 scrub ok 2024-03-11T21:11:58.528 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:58 smithi023 bash[21372]: cluster 2024-03-11T21:11:57.378403+0000 osd.5 (osd.5) 445 : cluster 0 1.8 scrub starts 2024-03-11T21:11:58.528 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:58 smithi023 bash[21372]: cluster 2024-03-11T21:11:57.380680+0000 osd.5 (osd.5) 446 : cluster 0 1.8 scrub ok 2024-03-11T21:11:58.528 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:58 smithi023 bash[21372]: cluster 2024-03-11T21:11:57.836116+0000 osd.1 (osd.1) 259 : cluster 0 2.6 scrub starts 2024-03-11T21:11:58.528 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:58 smithi023 bash[21372]: cluster 2024-03-11T21:11:57.838651+0000 osd.1 (osd.1) 260 : cluster 0 2.6 scrub ok 2024-03-11T21:11:58.528 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:58 smithi023 bash[21372]: cluster 2024-03-11T21:11:58.003633+0000 osd.6 (osd.6) 370 : cluster 0 1.6 scrub starts 2024-03-11T21:11:58.528 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:58 smithi023 bash[21372]: cluster 2024-03-11T21:11:58.005861+0000 osd.6 (osd.6) 371 : cluster 0 1.6 scrub ok 2024-03-11T21:11:58.528 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:58 smithi023 bash[29146]: cluster 2024-03-11T21:11:57.033948+0000 osd.7 (osd.7) 493 : cluster 0 1.9 scrub starts 2024-03-11T21:11:58.528 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:58 smithi023 bash[29146]: cluster 2024-03-11T21:11:57.035941+0000 osd.7 (osd.7) 494 : cluster 0 1.9 scrub ok 2024-03-11T21:11:58.528 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:58 smithi023 bash[29146]: cluster 2024-03-11T21:11:57.119723+0000 osd.0 (osd.0) 348 : cluster 0 3.3 scrub starts 2024-03-11T21:11:58.528 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:58 smithi023 bash[29146]: cluster 2024-03-11T21:11:57.124286+0000 osd.0 (osd.0) 349 : cluster 0 3.3 scrub ok 2024-03-11T21:11:58.529 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:58 smithi023 bash[29146]: cluster 2024-03-11T21:11:57.378403+0000 osd.5 (osd.5) 445 : cluster 0 1.8 scrub starts 2024-03-11T21:11:58.529 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:58 smithi023 bash[29146]: cluster 2024-03-11T21:11:57.380680+0000 osd.5 (osd.5) 446 : cluster 0 1.8 scrub ok 2024-03-11T21:11:58.529 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:58 smithi023 bash[29146]: cluster 2024-03-11T21:11:57.836116+0000 osd.1 (osd.1) 259 : cluster 0 2.6 scrub starts 2024-03-11T21:11:58.529 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:58 smithi023 bash[29146]: cluster 2024-03-11T21:11:57.838651+0000 osd.1 (osd.1) 260 : cluster 0 2.6 scrub ok 2024-03-11T21:11:58.529 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:58 smithi023 bash[29146]: cluster 2024-03-11T21:11:58.003633+0000 osd.6 (osd.6) 370 : cluster 0 1.6 scrub starts 2024-03-11T21:11:58.529 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:58 smithi023 bash[29146]: cluster 2024-03-11T21:11:58.005861+0000 osd.6 (osd.6) 371 : cluster 0 1.6 scrub ok 2024-03-11T21:11:58.565 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:11:58.577 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:58 smithi146 bash[23809]: cluster 2024-03-11T21:11:57.033948+0000 osd.7 (osd.7) 493 : cluster 0 1.9 scrub starts 2024-03-11T21:11:58.577 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:58 smithi146 bash[23809]: cluster 2024-03-11T21:11:57.035941+0000 osd.7 (osd.7) 494 : cluster 0 1.9 scrub ok 2024-03-11T21:11:58.577 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:58 smithi146 bash[23809]: cluster 2024-03-11T21:11:57.119723+0000 osd.0 (osd.0) 348 : cluster 0 3.3 scrub starts 2024-03-11T21:11:58.577 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:58 smithi146 bash[23809]: cluster 2024-03-11T21:11:57.124286+0000 osd.0 (osd.0) 349 : cluster 0 3.3 scrub ok 2024-03-11T21:11:58.577 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:58 smithi146 bash[23809]: cluster 2024-03-11T21:11:57.378403+0000 osd.5 (osd.5) 445 : cluster 0 1.8 scrub starts 2024-03-11T21:11:58.577 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:58 smithi146 bash[23809]: cluster 2024-03-11T21:11:57.380680+0000 osd.5 (osd.5) 446 : cluster 0 1.8 scrub ok 2024-03-11T21:11:58.577 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:58 smithi146 bash[23809]: cluster 2024-03-11T21:11:57.836116+0000 osd.1 (osd.1) 259 : cluster 0 2.6 scrub starts 2024-03-11T21:11:58.577 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:58 smithi146 bash[23809]: cluster 2024-03-11T21:11:57.838651+0000 osd.1 (osd.1) 260 : cluster 0 2.6 scrub ok 2024-03-11T21:11:58.578 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:58 smithi146 bash[23809]: cluster 2024-03-11T21:11:58.003633+0000 osd.6 (osd.6) 370 : cluster 0 1.6 scrub starts 2024-03-11T21:11:58.578 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:58 smithi146 bash[23809]: cluster 2024-03-11T21:11:58.005861+0000 osd.6 (osd.6) 371 : cluster 0 1.6 scrub ok 2024-03-11T21:11:58.594 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:11:58.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:58.596 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:11:58.697 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:11:58.726 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:11:58.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:58.729 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:11:58.830 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:58.859 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:58.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:58.861 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:11:58.962 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:58.990 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:11:58.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:58.992 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:11:59.093 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:59.122 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:59.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:59.124 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:11:59.225 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:59.247 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:59.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:59.249 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:11:59.350 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:11:59.375 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:11:59.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:59.377 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:11:59.478 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:59.487 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:59 smithi023 bash[21372]: cluster 2024-03-11T21:11:57.828516+0000 osd.3 (osd.3) 432 : cluster 0 1.4 scrub starts 2024-03-11T21:11:59.487 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:59 smithi023 bash[21372]: cluster 2024-03-11T21:11:57.831625+0000 osd.3 (osd.3) 433 : cluster 0 1.4 scrub ok 2024-03-11T21:11:59.487 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:59 smithi023 bash[21372]: cluster 2024-03-11T21:11:58.361051+0000 mgr.y (mgr.24380) 785 : cluster 0 pgmap v899: 158 pgs: 158 active+clean; 530 KiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2024-03-11T21:11:59.487 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:59 smithi023 bash[21372]: cluster 2024-03-11T21:11:58.412679+0000 osd.5 (osd.5) 447 : cluster 0 1.1 scrub starts 2024-03-11T21:11:59.487 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:59 smithi023 bash[21372]: cluster 2024-03-11T21:11:58.415120+0000 osd.5 (osd.5) 448 : cluster 0 1.1 scrub ok 2024-03-11T21:11:59.487 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:59 smithi023 bash[21372]: cluster 2024-03-11T21:11:58.862098+0000 osd.1 (osd.1) 261 : cluster 0 4.f scrub starts 2024-03-11T21:11:59.487 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:11:59 smithi023 bash[21372]: cluster 2024-03-11T21:11:58.864716+0000 osd.1 (osd.1) 262 : cluster 0 4.f scrub ok 2024-03-11T21:11:59.488 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:59 smithi023 bash[29146]: cluster 2024-03-11T21:11:57.828516+0000 osd.3 (osd.3) 432 : cluster 0 1.4 scrub starts 2024-03-11T21:11:59.488 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:59 smithi023 bash[29146]: cluster 2024-03-11T21:11:57.831625+0000 osd.3 (osd.3) 433 : cluster 0 1.4 scrub ok 2024-03-11T21:11:59.488 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:59 smithi023 bash[29146]: cluster 2024-03-11T21:11:58.361051+0000 mgr.y (mgr.24380) 785 : cluster 0 pgmap v899: 158 pgs: 158 active+clean; 530 KiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2024-03-11T21:11:59.488 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:59 smithi023 bash[29146]: cluster 2024-03-11T21:11:58.412679+0000 osd.5 (osd.5) 447 : cluster 0 1.1 scrub starts 2024-03-11T21:11:59.488 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:59 smithi023 bash[29146]: cluster 2024-03-11T21:11:58.415120+0000 osd.5 (osd.5) 448 : cluster 0 1.1 scrub ok 2024-03-11T21:11:59.488 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:59 smithi023 bash[29146]: cluster 2024-03-11T21:11:58.862098+0000 osd.1 (osd.1) 261 : cluster 0 4.f scrub starts 2024-03-11T21:11:59.488 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:11:59 smithi023 bash[29146]: cluster 2024-03-11T21:11:58.864716+0000 osd.1 (osd.1) 262 : cluster 0 4.f scrub ok 2024-03-11T21:11:59.500 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:11:59.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:59.503 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:11:59.603 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:59.630 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:11:59.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:59.632 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:11:59.733 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:11:59.757 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:59 smithi146 bash[23809]: cluster 2024-03-11T21:11:57.828516+0000 osd.3 (osd.3) 432 : cluster 0 1.4 scrub starts 2024-03-11T21:11:59.757 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:59 smithi146 bash[23809]: cluster 2024-03-11T21:11:57.831625+0000 osd.3 (osd.3) 433 : cluster 0 1.4 scrub ok 2024-03-11T21:11:59.757 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:59 smithi146 bash[23809]: cluster 2024-03-11T21:11:58.361051+0000 mgr.y (mgr.24380) 785 : cluster 0 pgmap v899: 158 pgs: 158 active+clean; 530 KiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2024-03-11T21:11:59.757 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:59 smithi146 bash[23809]: cluster 2024-03-11T21:11:58.412679+0000 osd.5 (osd.5) 447 : cluster 0 1.1 scrub starts 2024-03-11T21:11:59.757 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:59 smithi146 bash[23809]: cluster 2024-03-11T21:11:58.415120+0000 osd.5 (osd.5) 448 : cluster 0 1.1 scrub ok 2024-03-11T21:11:59.758 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:59 smithi146 bash[23809]: cluster 2024-03-11T21:11:58.862098+0000 osd.1 (osd.1) 261 : cluster 0 4.f scrub starts 2024-03-11T21:11:59.758 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:11:59 smithi146 bash[23809]: cluster 2024-03-11T21:11:58.864716+0000 osd.1 (osd.1) 262 : cluster 0 4.f scrub ok 2024-03-11T21:11:59.762 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:11:59.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:59.764 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:11:59.865 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:11:59.894 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:11:59.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:11:59.897 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:11:59.998 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:12:00.024 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:12:00.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:00.026 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:12:00.127 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:00.153 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:00.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:00.156 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:12:00.256 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:00.283 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:00.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:00.285 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:12:00.386 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:12:00.415 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:12:00.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:00.419 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:12:00.520 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:12:00.546 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:12:00.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:00.548 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:12:00.650 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:12:00.662 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:00 smithi023 bash[29146]: cluster 2024-03-11T21:11:59.374438+0000 osd.5 (osd.5) 449 : cluster 0 6.7 scrub starts 2024-03-11T21:12:00.663 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:00 smithi023 bash[21372]: cluster 2024-03-11T21:11:59.374438+0000 osd.5 (osd.5) 449 : cluster 0 6.7 scrub starts 2024-03-11T21:12:00.676 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:12:00.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:00.678 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:12:00.757 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:00 smithi146 bash[23809]: cluster 2024-03-11T21:11:59.374438+0000 osd.5 (osd.5) 449 : cluster 0 6.7 scrub starts 2024-03-11T21:12:00.779 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:00.806 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:00.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:00.808 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:12:00.909 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:12:00.934 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:12:00.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:00.936 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:12:01.037 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:12:01.061 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:12:01.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:01.064 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:12:01.165 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:12:01.193 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:12:01.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:01.195 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:12:01.296 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:12:01.323 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:12:01.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:01.325 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:12:01.426 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:12:01.456 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:12:01.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:01.458 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:12:01.559 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:01.586 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:01.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:01.588 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:12:01.689 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:12:01.698 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:01 smithi023 bash[21372]: cluster 2024-03-11T21:11:59.527782+0000 osd.5 (osd.5) 450 : cluster 0 6.7 scrub ok 2024-03-11T21:12:01.698 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:01 smithi023 bash[21372]: cluster 2024-03-11T21:11:59.976638+0000 osd.7 (osd.7) 495 : cluster 0 2.5 scrub starts 2024-03-11T21:12:01.698 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:01 smithi023 bash[21372]: cluster 2024-03-11T21:11:59.979073+0000 osd.7 (osd.7) 496 : cluster 0 2.5 scrub ok 2024-03-11T21:12:01.698 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:01 smithi023 bash[21372]: cluster 2024-03-11T21:12:00.362030+0000 mgr.y (mgr.24380) 786 : cluster 0 pgmap v900: 158 pgs: 158 active+clean; 483 KiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2024-03-11T21:12:01.698 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:01 smithi023 bash[21372]: cluster 2024-03-11T21:12:00.996099+0000 osd.6 (osd.6) 372 : cluster 0 2.11 scrub starts 2024-03-11T21:12:01.698 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:01 smithi023 bash[21372]: cluster 2024-03-11T21:12:00.998988+0000 osd.6 (osd.6) 373 : cluster 0 2.11 scrub ok 2024-03-11T21:12:01.698 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:01 smithi023 bash[29146]: cluster 2024-03-11T21:11:59.527782+0000 osd.5 (osd.5) 450 : cluster 0 6.7 scrub ok 2024-03-11T21:12:01.698 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:01 smithi023 bash[29146]: cluster 2024-03-11T21:11:59.976638+0000 osd.7 (osd.7) 495 : cluster 0 2.5 scrub starts 2024-03-11T21:12:01.698 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:01 smithi023 bash[29146]: cluster 2024-03-11T21:11:59.979073+0000 osd.7 (osd.7) 496 : cluster 0 2.5 scrub ok 2024-03-11T21:12:01.698 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:01 smithi023 bash[29146]: cluster 2024-03-11T21:12:00.362030+0000 mgr.y (mgr.24380) 786 : cluster 0 pgmap v900: 158 pgs: 158 active+clean; 483 KiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2024-03-11T21:12:01.699 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:01 smithi023 bash[29146]: cluster 2024-03-11T21:12:00.996099+0000 osd.6 (osd.6) 372 : cluster 0 2.11 scrub starts 2024-03-11T21:12:01.699 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:01 smithi023 bash[29146]: cluster 2024-03-11T21:12:00.998988+0000 osd.6 (osd.6) 373 : cluster 0 2.11 scrub ok 2024-03-11T21:12:01.711 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:12:01.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:01.715 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:12:01.757 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:01 smithi146 bash[23809]: cluster 2024-03-11T21:11:59.527782+0000 osd.5 (osd.5) 450 : cluster 0 6.7 scrub ok 2024-03-11T21:12:01.757 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:01 smithi146 bash[23809]: cluster 2024-03-11T21:11:59.976638+0000 osd.7 (osd.7) 495 : cluster 0 2.5 scrub starts 2024-03-11T21:12:01.757 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:01 smithi146 bash[23809]: cluster 2024-03-11T21:11:59.979073+0000 osd.7 (osd.7) 496 : cluster 0 2.5 scrub ok 2024-03-11T21:12:01.757 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:01 smithi146 bash[23809]: cluster 2024-03-11T21:12:00.362030+0000 mgr.y (mgr.24380) 786 : cluster 0 pgmap v900: 158 pgs: 158 active+clean; 483 KiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2024-03-11T21:12:01.757 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:01 smithi146 bash[23809]: cluster 2024-03-11T21:12:00.996099+0000 osd.6 (osd.6) 372 : cluster 0 2.11 scrub starts 2024-03-11T21:12:01.757 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:01 smithi146 bash[23809]: cluster 2024-03-11T21:12:00.998988+0000 osd.6 (osd.6) 373 : cluster 0 2.11 scrub ok 2024-03-11T21:12:01.816 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:01.844 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:01.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:01.846 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:12:01.894 INFO:teuthology.orchestra.run.smithi023.stdout:ERROR: (22) Invalid argument 2024-03-11T21:12:01.894 INFO:teuthology.orchestra.run.smithi023.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:12:01.946 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:01.972 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:01.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:01.975 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:12:02.075 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:12:02.103 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:12:02.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:02.105 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:12:02.206 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:02.234 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:02.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:02.237 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:12:02.304 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 6, 7, 1] out_osds: [] dead_osds: [] live_osds: [0, 2, 3, 5, 6, 7, 4, 1] 2024-03-11T21:12:02.305 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-11T21:12:02.305 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-03-11T21:12:02.305 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd dump -f json-pretty 2024-03-11T21:12:02.338 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:02.367 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:02.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:02.369 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:12:02.470 INFO:teuthology.orchestra.run.smithi023.stderr:enabled application 'rados' on pool 'unique_pool_4' 2024-03-11T21:12:02.470 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:02.497 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:02.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:02.499 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:12:02.600 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:02.611 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:02 smithi146 bash[23809]: cluster 2024-03-11T21:12:00.898798+0000 osd.3 (osd.3) 434 : cluster 0 6.1f scrub starts 2024-03-11T21:12:02.611 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:02 smithi146 bash[23809]: cluster 2024-03-11T21:12:01.082618+0000 osd.3 (osd.3) 435 : cluster 0 6.1f scrub ok 2024-03-11T21:12:02.611 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:02 smithi146 bash[23809]: cluster 2024-03-11T21:12:01.753222+0000 osd.1 (osd.1) 263 : cluster 0 4.4 scrub starts 2024-03-11T21:12:02.611 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:02 smithi146 bash[23809]: cluster 2024-03-11T21:12:01.754394+0000 osd.1 (osd.1) 264 : cluster 0 4.4 scrub ok 2024-03-11T21:12:02.611 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:02 smithi146 bash[23809]: cluster 2024-03-11T21:12:02.042439+0000 osd.6 (osd.6) 374 : cluster 0 4.1f scrub starts 2024-03-11T21:12:02.612 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:02 smithi146 bash[23809]: cluster 2024-03-11T21:12:02.044026+0000 osd.6 (osd.6) 375 : cluster 0 4.1f scrub ok 2024-03-11T21:12:02.612 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:02 smithi146 bash[23809]: audit 2024-03-11T21:12:02.132031+0000 mon.a (mon.0) 1804 : audit 1 from='client.? 172.21.15.23:0/67452671' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:12:02.625 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:02.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:02.628 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:12:02.729 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:12:02.740 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:02 smithi023 bash[21372]: cluster 2024-03-11T21:12:00.898798+0000 osd.3 (osd.3) 434 : cluster 0 6.1f scrub starts 2024-03-11T21:12:02.740 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:02 smithi023 bash[21372]: cluster 2024-03-11T21:12:01.082618+0000 osd.3 (osd.3) 435 : cluster 0 6.1f scrub ok 2024-03-11T21:12:02.740 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:02 smithi023 bash[21372]: cluster 2024-03-11T21:12:01.753222+0000 osd.1 (osd.1) 263 : cluster 0 4.4 scrub starts 2024-03-11T21:12:02.741 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:02 smithi023 bash[21372]: cluster 2024-03-11T21:12:01.754394+0000 osd.1 (osd.1) 264 : cluster 0 4.4 scrub ok 2024-03-11T21:12:02.741 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:02 smithi023 bash[21372]: cluster 2024-03-11T21:12:02.042439+0000 osd.6 (osd.6) 374 : cluster 0 4.1f scrub starts 2024-03-11T21:12:02.741 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:02 smithi023 bash[21372]: cluster 2024-03-11T21:12:02.044026+0000 osd.6 (osd.6) 375 : cluster 0 4.1f scrub ok 2024-03-11T21:12:02.741 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:02 smithi023 bash[21372]: audit 2024-03-11T21:12:02.132031+0000 mon.a (mon.0) 1804 : audit 1 from='client.? 172.21.15.23:0/67452671' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:12:02.741 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:02 smithi023 bash[29146]: cluster 2024-03-11T21:12:00.898798+0000 osd.3 (osd.3) 434 : cluster 0 6.1f scrub starts 2024-03-11T21:12:02.741 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:02 smithi023 bash[29146]: cluster 2024-03-11T21:12:01.082618+0000 osd.3 (osd.3) 435 : cluster 0 6.1f scrub ok 2024-03-11T21:12:02.741 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:02 smithi023 bash[29146]: cluster 2024-03-11T21:12:01.753222+0000 osd.1 (osd.1) 263 : cluster 0 4.4 scrub starts 2024-03-11T21:12:02.741 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:02 smithi023 bash[29146]: cluster 2024-03-11T21:12:01.754394+0000 osd.1 (osd.1) 264 : cluster 0 4.4 scrub ok 2024-03-11T21:12:02.741 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:02 smithi023 bash[29146]: cluster 2024-03-11T21:12:02.042439+0000 osd.6 (osd.6) 374 : cluster 0 4.1f scrub starts 2024-03-11T21:12:02.741 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:02 smithi023 bash[29146]: cluster 2024-03-11T21:12:02.044026+0000 osd.6 (osd.6) 375 : cluster 0 4.1f scrub ok 2024-03-11T21:12:02.741 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:02 smithi023 bash[29146]: audit 2024-03-11T21:12:02.132031+0000 mon.a (mon.0) 1804 : audit 1 from='client.? 172.21.15.23:0/67452671' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:12:02.757 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:12:02.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:02.759 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:12:02.815 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-03-11T21:12:02.860 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:12:02.882 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:12:02.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:02.885 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:12:02.986 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:03.013 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:03.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:03.015 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:12:03.117 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:12:03.144 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:12:03.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:03.146 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:12:03.247 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:03.274 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:03.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:03.276 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:12:03.377 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:12:03.621 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:12:03.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:03.623 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:12:03.724 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:12:03.737 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:03 smithi023 bash[29146]: cluster 2024-03-11T21:12:01.876006+0000 osd.3 (osd.3) 436 : cluster 0 4.1a scrub starts 2024-03-11T21:12:03.737 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:03 smithi023 bash[29146]: cluster 2024-03-11T21:12:01.877124+0000 osd.3 (osd.3) 437 : cluster 0 4.1a scrub ok 2024-03-11T21:12:03.737 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:03 smithi023 bash[29146]: cluster 2024-03-11T21:12:02.362933+0000 mgr.y (mgr.24380) 787 : cluster 0 pgmap v901: 158 pgs: 158 active+clean; 582 KiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2024-03-11T21:12:03.737 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:03 smithi023 bash[29146]: cluster 2024-03-11T21:12:02.369810+0000 osd.5 (osd.5) 451 : cluster 0 3.1c scrub starts 2024-03-11T21:12:03.738 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:03 smithi023 bash[29146]: cluster 2024-03-11T21:12:02.373119+0000 osd.5 (osd.5) 452 : cluster 0 3.1c scrub ok 2024-03-11T21:12:03.738 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:03 smithi023 bash[29146]: audit 2024-03-11T21:12:02.473689+0000 mon.a (mon.0) 1805 : audit 1 from='client.? 172.21.15.23:0/67452671' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-11T21:12:03.738 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:03 smithi023 bash[29146]: cluster 2024-03-11T21:12:02.486401+0000 mon.a (mon.0) 1806 : cluster 0 osdmap e386: 8 total, 8 up, 8 in 2024-03-11T21:12:03.738 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:03 smithi023 bash[21372]: cluster 2024-03-11T21:12:01.876006+0000 osd.3 (osd.3) 436 : cluster 0 4.1a scrub starts 2024-03-11T21:12:03.738 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:03 smithi023 bash[21372]: cluster 2024-03-11T21:12:01.877124+0000 osd.3 (osd.3) 437 : cluster 0 4.1a scrub ok 2024-03-11T21:12:03.738 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:03 smithi023 bash[21372]: cluster 2024-03-11T21:12:02.362933+0000 mgr.y (mgr.24380) 787 : cluster 0 pgmap v901: 158 pgs: 158 active+clean; 582 KiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2024-03-11T21:12:03.738 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:03 smithi023 bash[21372]: cluster 2024-03-11T21:12:02.369810+0000 osd.5 (osd.5) 451 : cluster 0 3.1c scrub starts 2024-03-11T21:12:03.738 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:03 smithi023 bash[21372]: cluster 2024-03-11T21:12:02.373119+0000 osd.5 (osd.5) 452 : cluster 0 3.1c scrub ok 2024-03-11T21:12:03.738 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:03 smithi023 bash[21372]: audit 2024-03-11T21:12:02.473689+0000 mon.a (mon.0) 1805 : audit 1 from='client.? 172.21.15.23:0/67452671' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-11T21:12:03.738 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:03 smithi023 bash[21372]: cluster 2024-03-11T21:12:02.486401+0000 mon.a (mon.0) 1806 : cluster 0 osdmap e386: 8 total, 8 up, 8 in 2024-03-11T21:12:03.753 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:12:03.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:03.755 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:12:03.757 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:03 smithi146 bash[23809]: cluster 2024-03-11T21:12:01.876006+0000 osd.3 (osd.3) 436 : cluster 0 4.1a scrub starts 2024-03-11T21:12:03.757 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:03 smithi146 bash[23809]: cluster 2024-03-11T21:12:01.877124+0000 osd.3 (osd.3) 437 : cluster 0 4.1a scrub ok 2024-03-11T21:12:03.757 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:03 smithi146 bash[23809]: cluster 2024-03-11T21:12:02.362933+0000 mgr.y (mgr.24380) 787 : cluster 0 pgmap v901: 158 pgs: 158 active+clean; 582 KiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2024-03-11T21:12:03.757 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:03 smithi146 bash[23809]: cluster 2024-03-11T21:12:02.369810+0000 osd.5 (osd.5) 451 : cluster 0 3.1c scrub starts 2024-03-11T21:12:03.757 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:03 smithi146 bash[23809]: cluster 2024-03-11T21:12:02.373119+0000 osd.5 (osd.5) 452 : cluster 0 3.1c scrub ok 2024-03-11T21:12:03.757 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:03 smithi146 bash[23809]: audit 2024-03-11T21:12:02.473689+0000 mon.a (mon.0) 1805 : audit 1 from='client.? 172.21.15.23:0/67452671' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-11T21:12:03.757 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:03 smithi146 bash[23809]: cluster 2024-03-11T21:12:02.486401+0000 mon.a (mon.0) 1806 : cluster 0 osdmap e386: 8 total, 8 up, 8 in 2024-03-11T21:12:03.856 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:12:03.885 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:12:03.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:03.888 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:12:03.951 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:12:03.951 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:12:03.989 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:12:04.010 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:12:04.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:04.012 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:12:04.113 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:04.141 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:04.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:04.144 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:12:04.245 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:12:04.274 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:12:04.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:04.276 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:12:04.377 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:12:04.403 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:12:04.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:04.406 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:12:04.506 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:12:04.527 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:04 smithi023 bash[21372]: cluster 2024-03-11T21:12:02.858407+0000 osd.3 (osd.3) 438 : cluster 0 4.0 scrub starts 2024-03-11T21:12:04.527 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:04 smithi023 bash[21372]: cluster 2024-03-11T21:12:02.860631+0000 osd.3 (osd.3) 439 : cluster 0 4.0 scrub ok 2024-03-11T21:12:04.527 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:04 smithi023 bash[21372]: cluster 2024-03-11T21:12:03.351374+0000 osd.5 (osd.5) 453 : cluster 0 3.16 scrub starts 2024-03-11T21:12:04.527 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:04 smithi023 bash[21372]: cluster 2024-03-11T21:12:03.355481+0000 osd.5 (osd.5) 454 : cluster 0 3.16 scrub ok 2024-03-11T21:12:04.527 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:04 smithi023 bash[29146]: cluster 2024-03-11T21:12:02.858407+0000 osd.3 (osd.3) 438 : cluster 0 4.0 scrub starts 2024-03-11T21:12:04.528 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:04 smithi023 bash[29146]: cluster 2024-03-11T21:12:02.860631+0000 osd.3 (osd.3) 439 : cluster 0 4.0 scrub ok 2024-03-11T21:12:04.528 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:04 smithi023 bash[29146]: cluster 2024-03-11T21:12:03.351374+0000 osd.5 (osd.5) 453 : cluster 0 3.16 scrub starts 2024-03-11T21:12:04.528 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:04 smithi023 bash[29146]: cluster 2024-03-11T21:12:03.355481+0000 osd.5 (osd.5) 454 : cluster 0 3.16 scrub ok 2024-03-11T21:12:04.534 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:12:04.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:04.537 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:12:04.553 DEBUG:teuthology.orchestra.run.smithi023:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_4 bench 90 write' 2024-03-11T21:12:04.556 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-03-11T21:12:04.556 INFO:teuthology.orchestra.run:waiting for 3000 2024-03-11T21:12:04.638 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:04.666 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:04.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:04.668 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:12:04.757 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:04 smithi146 bash[23809]: cluster 2024-03-11T21:12:02.858407+0000 osd.3 (osd.3) 438 : cluster 0 4.0 scrub starts 2024-03-11T21:12:04.757 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:04 smithi146 bash[23809]: cluster 2024-03-11T21:12:02.860631+0000 osd.3 (osd.3) 439 : cluster 0 4.0 scrub ok 2024-03-11T21:12:04.757 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:04 smithi146 bash[23809]: cluster 2024-03-11T21:12:03.351374+0000 osd.5 (osd.5) 453 : cluster 0 3.16 scrub starts 2024-03-11T21:12:04.757 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:04 smithi146 bash[23809]: cluster 2024-03-11T21:12:03.355481+0000 osd.5 (osd.5) 454 : cluster 0 3.16 scrub ok 2024-03-11T21:12:04.769 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:12:04.796 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:12:04.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:04.800 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:12:04.826 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:hints = 1 2024-03-11T21:12:04.826 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-03-11T21:12:04.826 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:Object prefix: benchmark_data_smithi023_142591 2024-03-11T21:12:04.827 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-11T21:12:04.827 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 0 0 0 0 0 0 - 0 2024-03-11T21:12:04.901 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:12:04.936 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:12:04.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:04.940 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:12:05.040 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:12:05.068 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:12:05.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:05.071 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:12:05.172 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:12:05.199 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:12:05.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:05.202 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:12:05.303 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:12:05.344 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:12:05.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:05.350 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:12:05.451 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:12:05.474 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:12:05.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:05.477 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:12:05.578 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:12:05.605 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:12:05.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:05.609 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:12:05.710 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:12:05.719 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:05 smithi146 bash[23809]: cluster 2024-03-11T21:12:03.901763+0000 osd.3 (osd.3) 440 : cluster 0 4.10 scrub starts 2024-03-11T21:12:05.719 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:05 smithi146 bash[23809]: cluster 2024-03-11T21:12:03.903554+0000 osd.3 (osd.3) 441 : cluster 0 4.10 scrub ok 2024-03-11T21:12:05.719 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:05 smithi146 bash[23809]: cluster 2024-03-11T21:12:04.296186+0000 osd.0 (osd.0) 350 : cluster 0 purged_snaps scrub starts 2024-03-11T21:12:05.719 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:05 smithi146 bash[23809]: cluster 2024-03-11T21:12:04.296269+0000 osd.0 (osd.0) 351 : cluster 0 purged_snaps scrub ok 2024-03-11T21:12:05.719 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:05 smithi146 bash[23809]: cluster 2024-03-11T21:12:04.363648+0000 mgr.y (mgr.24380) 788 : cluster 0 pgmap v903: 158 pgs: 158 active+clean; 582 KiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2024-03-11T21:12:05.719 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:05 smithi146 bash[23809]: cluster 2024-03-11T21:12:04.478086+0000 mon.a (mon.0) 1807 : cluster 1 Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-11T21:12:05.719 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:05 smithi146 bash[23809]: cluster 2024-03-11T21:12:04.497759+0000 mon.a (mon.0) 1808 : cluster 0 osdmap e387: 8 total, 8 up, 8 in 2024-03-11T21:12:05.719 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:05 smithi146 bash[23809]: cluster 2024-03-11T21:12:04.968664+0000 osd.6 (osd.6) 376 : cluster 0 4.e scrub starts 2024-03-11T21:12:05.719 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:05 smithi146 bash[23809]: cluster 2024-03-11T21:12:04.970856+0000 osd.6 (osd.6) 377 : cluster 0 4.e scrub ok 2024-03-11T21:12:05.733 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:12:05.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:05.736 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:12:05.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:05 smithi023 bash[21372]: cluster 2024-03-11T21:12:03.901763+0000 osd.3 (osd.3) 440 : cluster 0 4.10 scrub starts 2024-03-11T21:12:05.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:05 smithi023 bash[21372]: cluster 2024-03-11T21:12:03.903554+0000 osd.3 (osd.3) 441 : cluster 0 4.10 scrub ok 2024-03-11T21:12:05.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:05 smithi023 bash[21372]: cluster 2024-03-11T21:12:04.296186+0000 osd.0 (osd.0) 350 : cluster 0 purged_snaps scrub starts 2024-03-11T21:12:05.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:05 smithi023 bash[21372]: cluster 2024-03-11T21:12:04.296269+0000 osd.0 (osd.0) 351 : cluster 0 purged_snaps scrub ok 2024-03-11T21:12:05.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:05 smithi023 bash[21372]: cluster 2024-03-11T21:12:04.363648+0000 mgr.y (mgr.24380) 788 : cluster 0 pgmap v903: 158 pgs: 158 active+clean; 582 KiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2024-03-11T21:12:05.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:05 smithi023 bash[21372]: cluster 2024-03-11T21:12:04.478086+0000 mon.a (mon.0) 1807 : cluster 1 Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-11T21:12:05.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:05 smithi023 bash[21372]: cluster 2024-03-11T21:12:04.497759+0000 mon.a (mon.0) 1808 : cluster 0 osdmap e387: 8 total, 8 up, 8 in 2024-03-11T21:12:05.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:05 smithi023 bash[21372]: cluster 2024-03-11T21:12:04.968664+0000 osd.6 (osd.6) 376 : cluster 0 4.e scrub starts 2024-03-11T21:12:05.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:05 smithi023 bash[21372]: cluster 2024-03-11T21:12:04.970856+0000 osd.6 (osd.6) 377 : cluster 0 4.e scrub ok 2024-03-11T21:12:05.779 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:05 smithi023 bash[29146]: cluster 2024-03-11T21:12:03.901763+0000 osd.3 (osd.3) 440 : cluster 0 4.10 scrub starts 2024-03-11T21:12:05.779 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:05 smithi023 bash[29146]: cluster 2024-03-11T21:12:03.903554+0000 osd.3 (osd.3) 441 : cluster 0 4.10 scrub ok 2024-03-11T21:12:05.779 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:05 smithi023 bash[29146]: cluster 2024-03-11T21:12:04.296186+0000 osd.0 (osd.0) 350 : cluster 0 purged_snaps scrub starts 2024-03-11T21:12:05.779 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:05 smithi023 bash[29146]: cluster 2024-03-11T21:12:04.296269+0000 osd.0 (osd.0) 351 : cluster 0 purged_snaps scrub ok 2024-03-11T21:12:05.779 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:05 smithi023 bash[29146]: cluster 2024-03-11T21:12:04.363648+0000 mgr.y (mgr.24380) 788 : cluster 0 pgmap v903: 158 pgs: 158 active+clean; 582 KiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2024-03-11T21:12:05.779 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:05 smithi023 bash[29146]: cluster 2024-03-11T21:12:04.478086+0000 mon.a (mon.0) 1807 : cluster 1 Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-11T21:12:05.779 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:05 smithi023 bash[29146]: cluster 2024-03-11T21:12:04.497759+0000 mon.a (mon.0) 1808 : cluster 0 osdmap e387: 8 total, 8 up, 8 in 2024-03-11T21:12:05.779 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:05 smithi023 bash[29146]: cluster 2024-03-11T21:12:04.968664+0000 osd.6 (osd.6) 376 : cluster 0 4.e scrub starts 2024-03-11T21:12:05.779 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:05 smithi023 bash[29146]: cluster 2024-03-11T21:12:04.970856+0000 osd.6 (osd.6) 377 : cluster 0 4.e scrub ok 2024-03-11T21:12:05.827 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 1 15 2553 2538 158.617 158.625 0.00375664 0.00627688 2024-03-11T21:12:05.837 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:12:05.868 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:12:05.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:05.870 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:12:05.971 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:12:06.007 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:12:05 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:12:06.012 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:12:06.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:06.014 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:12:06.115 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:06.142 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:06.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:06.145 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:12:06.246 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:12:06.270 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:12:06.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:06.274 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:12:06.375 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:12:06.401 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:12:06.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:06.403 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:12:06.504 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:06.527 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:06 smithi023 bash[21372]: cluster 2024-03-11T21:12:05.167091+0000 osd.0 (osd.0) 352 : cluster 0 6.16 scrub starts 2024-03-11T21:12:06.527 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:06 smithi023 bash[21372]: cluster 2024-03-11T21:12:05.248300+0000 osd.0 (osd.0) 353 : cluster 0 6.16 scrub ok 2024-03-11T21:12:06.527 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:06 smithi023 bash[21372]: cluster 2024-03-11T21:12:06.008538+0000 osd.6 (osd.6) 378 : cluster 0 2.1a scrub starts 2024-03-11T21:12:06.527 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:06 smithi023 bash[21372]: cluster 2024-03-11T21:12:06.009687+0000 osd.6 (osd.6) 379 : cluster 0 2.1a scrub ok 2024-03-11T21:12:06.527 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:06 smithi023 bash[21372]: audit 2024-03-11T21:12:06.149916+0000 mon.a (mon.0) 1809 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:12:06.527 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:06 smithi023 bash[21372]: audit 2024-03-11T21:12:06.154755+0000 mon.a (mon.0) 1810 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:12:06.528 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:06 smithi023 bash[29146]: cluster 2024-03-11T21:12:05.167091+0000 osd.0 (osd.0) 352 : cluster 0 6.16 scrub starts 2024-03-11T21:12:06.528 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:06 smithi023 bash[29146]: cluster 2024-03-11T21:12:05.248300+0000 osd.0 (osd.0) 353 : cluster 0 6.16 scrub ok 2024-03-11T21:12:06.528 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:06 smithi023 bash[29146]: cluster 2024-03-11T21:12:06.008538+0000 osd.6 (osd.6) 378 : cluster 0 2.1a scrub starts 2024-03-11T21:12:06.528 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:06 smithi023 bash[29146]: cluster 2024-03-11T21:12:06.009687+0000 osd.6 (osd.6) 379 : cluster 0 2.1a scrub ok 2024-03-11T21:12:06.528 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:06 smithi023 bash[29146]: audit 2024-03-11T21:12:06.149916+0000 mon.a (mon.0) 1809 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:12:06.528 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:06 smithi023 bash[29146]: audit 2024-03-11T21:12:06.154755+0000 mon.a (mon.0) 1810 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:12:06.528 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:06.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:06.530 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:12:06.631 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:12:06.654 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:12:06.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:06.657 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:12:06.757 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:06 smithi146 bash[23809]: cluster 2024-03-11T21:12:05.167091+0000 osd.0 (osd.0) 352 : cluster 0 6.16 scrub starts 2024-03-11T21:12:06.757 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:06 smithi146 bash[23809]: cluster 2024-03-11T21:12:05.248300+0000 osd.0 (osd.0) 353 : cluster 0 6.16 scrub ok 2024-03-11T21:12:06.757 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:06 smithi146 bash[23809]: cluster 2024-03-11T21:12:06.008538+0000 osd.6 (osd.6) 378 : cluster 0 2.1a scrub starts 2024-03-11T21:12:06.757 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:06 smithi146 bash[23809]: cluster 2024-03-11T21:12:06.009687+0000 osd.6 (osd.6) 379 : cluster 0 2.1a scrub ok 2024-03-11T21:12:06.757 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:06 smithi146 bash[23809]: audit 2024-03-11T21:12:06.149916+0000 mon.a (mon.0) 1809 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:12:06.757 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:06 smithi146 bash[23809]: audit 2024-03-11T21:12:06.154755+0000 mon.a (mon.0) 1810 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:12:06.759 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:06.783 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:06.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:06.785 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:12:06.827 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 2 16 5277 5261 164.395 170.188 0.00479709 0.00606909 2024-03-11T21:12:06.885 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:12:06.912 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:12:06.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:06.915 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:12:07.016 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:12:07.040 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:12:07.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:07.043 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:12:07.143 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:12:07.170 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:12:07.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:07.173 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:12:07.274 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:07.298 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:07.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:07.303 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:12:07.404 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:12:07.428 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:12:07.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:07.431 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:12:07.532 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:12:07.558 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:12:07.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:07.562 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:12:07.663 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:12:07.673 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:07 smithi023 bash[29146]: audit 2024-03-11T21:12:05.913040+0000 mgr.y (mgr.24380) 789 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:12:07.674 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:07 smithi023 bash[29146]: cluster 2024-03-11T21:12:06.204556+0000 osd.0 (osd.0) 354 : cluster 0 4.b scrub starts 2024-03-11T21:12:07.674 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:07 smithi023 bash[29146]: cluster 2024-03-11T21:12:06.205724+0000 osd.0 (osd.0) 355 : cluster 0 4.b scrub ok 2024-03-11T21:12:07.674 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:07 smithi023 bash[29146]: cluster 2024-03-11T21:12:06.365127+0000 mgr.y (mgr.24380) 790 : cluster 0 pgmap v905: 158 pgs: 158 active+clean; 34 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 3.9 MiB/s wr, 47 op/s 2024-03-11T21:12:07.674 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:07 smithi023 bash[29146]: cluster 2024-03-11T21:12:06.375404+0000 osd.5 (osd.5) 455 : cluster 0 4.d scrub starts 2024-03-11T21:12:07.674 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:07 smithi023 bash[29146]: cluster 2024-03-11T21:12:06.377198+0000 osd.5 (osd.5) 456 : cluster 0 4.d scrub ok 2024-03-11T21:12:07.674 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:07 smithi023 bash[29146]: cluster 2024-03-11T21:12:06.855097+0000 osd.1 (osd.1) 265 : cluster 0 3.0 scrub starts 2024-03-11T21:12:07.674 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:07 smithi023 bash[29146]: cluster 2024-03-11T21:12:06.861173+0000 osd.1 (osd.1) 266 : cluster 0 3.0 scrub ok 2024-03-11T21:12:07.674 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:07 smithi023 bash[21372]: audit 2024-03-11T21:12:05.913040+0000 mgr.y (mgr.24380) 789 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:12:07.674 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:07 smithi023 bash[21372]: cluster 2024-03-11T21:12:06.204556+0000 osd.0 (osd.0) 354 : cluster 0 4.b scrub starts 2024-03-11T21:12:07.674 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:07 smithi023 bash[21372]: cluster 2024-03-11T21:12:06.205724+0000 osd.0 (osd.0) 355 : cluster 0 4.b scrub ok 2024-03-11T21:12:07.674 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:07 smithi023 bash[21372]: cluster 2024-03-11T21:12:06.365127+0000 mgr.y (mgr.24380) 790 : cluster 0 pgmap v905: 158 pgs: 158 active+clean; 34 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 3.9 MiB/s wr, 47 op/s 2024-03-11T21:12:07.675 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:07 smithi023 bash[21372]: cluster 2024-03-11T21:12:06.375404+0000 osd.5 (osd.5) 455 : cluster 0 4.d scrub starts 2024-03-11T21:12:07.675 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:07 smithi023 bash[21372]: cluster 2024-03-11T21:12:06.377198+0000 osd.5 (osd.5) 456 : cluster 0 4.d scrub ok 2024-03-11T21:12:07.675 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:07 smithi023 bash[21372]: cluster 2024-03-11T21:12:06.855097+0000 osd.1 (osd.1) 265 : cluster 0 3.0 scrub starts 2024-03-11T21:12:07.675 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:07 smithi023 bash[21372]: cluster 2024-03-11T21:12:06.861173+0000 osd.1 (osd.1) 266 : cluster 0 3.0 scrub ok 2024-03-11T21:12:07.689 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:12:07.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:07.692 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:12:07.757 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:07 smithi146 bash[23809]: audit 2024-03-11T21:12:05.913040+0000 mgr.y (mgr.24380) 789 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:12:07.757 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:07 smithi146 bash[23809]: cluster 2024-03-11T21:12:06.204556+0000 osd.0 (osd.0) 354 : cluster 0 4.b scrub starts 2024-03-11T21:12:07.757 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:07 smithi146 bash[23809]: cluster 2024-03-11T21:12:06.205724+0000 osd.0 (osd.0) 355 : cluster 0 4.b scrub ok 2024-03-11T21:12:07.757 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:07 smithi146 bash[23809]: cluster 2024-03-11T21:12:06.365127+0000 mgr.y (mgr.24380) 790 : cluster 0 pgmap v905: 158 pgs: 158 active+clean; 34 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 3.9 MiB/s wr, 47 op/s 2024-03-11T21:12:07.757 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:07 smithi146 bash[23809]: cluster 2024-03-11T21:12:06.375404+0000 osd.5 (osd.5) 455 : cluster 0 4.d scrub starts 2024-03-11T21:12:07.757 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:07 smithi146 bash[23809]: cluster 2024-03-11T21:12:06.377198+0000 osd.5 (osd.5) 456 : cluster 0 4.d scrub ok 2024-03-11T21:12:07.758 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:07 smithi146 bash[23809]: cluster 2024-03-11T21:12:06.855097+0000 osd.1 (osd.1) 265 : cluster 0 3.0 scrub starts 2024-03-11T21:12:07.758 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:07 smithi146 bash[23809]: cluster 2024-03-11T21:12:06.861173+0000 osd.1 (osd.1) 266 : cluster 0 3.0 scrub ok 2024-03-11T21:12:07.792 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:12:07.822 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:12:07.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:07.825 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:12:07.826 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 3 16 7902 7886 164.279 164.062 0.00909908 0.00607302 2024-03-11T21:12:07.926 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:07.950 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:07.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:07.952 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:12:08.053 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:12:08.081 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:12:08.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:08.083 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:12:08.184 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:12:08.209 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:12:08.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:08.211 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:12:08.312 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:12:08.341 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:12:08.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:08.343 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:12:08.444 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:08.475 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:08.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:08.477 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:12:08.578 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:12:08.588 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:12:08 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:12:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:12:08.588 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:08 smithi023 bash[21372]: cluster 2024-03-11T21:12:07.171658+0000 osd.0 (osd.0) 356 : cluster 0 4.1e scrub starts 2024-03-11T21:12:08.588 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:08 smithi023 bash[21372]: cluster 2024-03-11T21:12:07.174871+0000 osd.0 (osd.0) 357 : cluster 0 4.1e scrub ok 2024-03-11T21:12:08.588 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:08 smithi023 bash[21372]: cluster 2024-03-11T21:12:07.981709+0000 osd.6 (osd.6) 380 : cluster 0 3.a scrub starts 2024-03-11T21:12:08.588 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:08 smithi023 bash[21372]: cluster 2024-03-11T21:12:07.992031+0000 osd.6 (osd.6) 381 : cluster 0 3.a scrub ok 2024-03-11T21:12:08.589 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:08 smithi023 bash[29146]: cluster 2024-03-11T21:12:07.171658+0000 osd.0 (osd.0) 356 : cluster 0 4.1e scrub starts 2024-03-11T21:12:08.589 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:08 smithi023 bash[29146]: cluster 2024-03-11T21:12:07.174871+0000 osd.0 (osd.0) 357 : cluster 0 4.1e scrub ok 2024-03-11T21:12:08.589 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:08 smithi023 bash[29146]: cluster 2024-03-11T21:12:07.981709+0000 osd.6 (osd.6) 380 : cluster 0 3.a scrub starts 2024-03-11T21:12:08.589 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:08 smithi023 bash[29146]: cluster 2024-03-11T21:12:07.992031+0000 osd.6 (osd.6) 381 : cluster 0 3.a scrub ok 2024-03-11T21:12:08.607 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:12:08.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:08.610 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:12:08.711 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:08.721 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:08 smithi146 bash[23809]: cluster 2024-03-11T21:12:07.171658+0000 osd.0 (osd.0) 356 : cluster 0 4.1e scrub starts 2024-03-11T21:12:08.721 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:08 smithi146 bash[23809]: cluster 2024-03-11T21:12:07.174871+0000 osd.0 (osd.0) 357 : cluster 0 4.1e scrub ok 2024-03-11T21:12:08.721 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:08 smithi146 bash[23809]: cluster 2024-03-11T21:12:07.981709+0000 osd.6 (osd.6) 380 : cluster 0 3.a scrub starts 2024-03-11T21:12:08.721 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:08 smithi146 bash[23809]: cluster 2024-03-11T21:12:07.992031+0000 osd.6 (osd.6) 381 : cluster 0 3.a scrub ok 2024-03-11T21:12:08.748 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:08.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:08.750 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:12:08.827 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 4 16 10319 10303 160.972 151.062 0.00986242 0.00619964 2024-03-11T21:12:08.851 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:12:08.887 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:12:08.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:08.890 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:12:08.991 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:12:09.023 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:12:09.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:09.028 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:12:09.060 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:12:09.129 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:12:09.167 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:12:09.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:09.171 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:12:09.272 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:12:09.301 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:12:09.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:09.304 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:12:09.405 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:09.447 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:09.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:09.449 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:12:09.550 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:09.569 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:09.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:09.570 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:12:09.671 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:09.699 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:09.701 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:09 smithi146 bash[23809]: cluster 2024-03-11T21:12:08.107984+0000 osd.7 (osd.7) 497 : cluster 0 6.3 scrub starts 2024-03-11T21:12:09.701 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:09 smithi146 bash[23809]: cluster 2024-03-11T21:12:08.234817+0000 osd.7 (osd.7) 498 : cluster 0 6.3 scrub ok 2024-03-11T21:12:09.701 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:09 smithi146 bash[23809]: cluster 2024-03-11T21:12:08.365937+0000 mgr.y (mgr.24380) 791 : cluster 0 pgmap v906: 158 pgs: 158 active+clean; 112 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 14 MiB/s wr, 216 op/s 2024-03-11T21:12:09.701 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:09 smithi146 bash[23809]: cluster 2024-03-11T21:12:08.636662+0000 osd.2 (osd.2) 257 : cluster 0 purged_snaps scrub starts 2024-03-11T21:12:09.702 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:09 smithi146 bash[23809]: cluster 2024-03-11T21:12:08.636734+0000 osd.2 (osd.2) 258 : cluster 0 purged_snaps scrub ok 2024-03-11T21:12:09.702 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:09 smithi146 bash[23809]: audit 2024-03-11T21:12:08.670999+0000 mon.a (mon.0) 1811 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:12:09.702 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:09 smithi146 bash[23809]: cluster 2024-03-11T21:12:08.991904+0000 osd.6 (osd.6) 382 : cluster 0 4.18 scrub starts 2024-03-11T21:12:09.702 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:09 smithi146 bash[23809]: cluster 2024-03-11T21:12:08.997104+0000 osd.6 (osd.6) 383 : cluster 0 4.18 scrub ok 2024-03-11T21:12:09.702 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:09 smithi146 bash[23809]: cluster 2024-03-11T21:12:09.371930+0000 osd.5 (osd.5) 457 : cluster 0 2.16 scrub starts 2024-03-11T21:12:09.702 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:09 smithi146 bash[23809]: cluster 2024-03-11T21:12:09.374256+0000 osd.5 (osd.5) 458 : cluster 0 2.16 scrub ok 2024-03-11T21:12:09.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:09.703 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:12:09.804 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:12:09.827 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 5 16 12128 12112 151.388 113.062 0.00859305 0.00659786 2024-03-11T21:12:09.829 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:12:09.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:09.832 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:12:09.933 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:12:09.958 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:12:09.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:09.960 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:12:10.061 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:12:10.092 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:12:10.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:10.094 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:12:10.124 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:09 smithi023 bash[21372]: cluster 2024-03-11T21:12:08.107984+0000 osd.7 (osd.7) 497 : cluster 0 6.3 scrub starts 2024-03-11T21:12:10.124 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:09 smithi023 bash[21372]: cluster 2024-03-11T21:12:08.234817+0000 osd.7 (osd.7) 498 : cluster 0 6.3 scrub ok 2024-03-11T21:12:10.124 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:09 smithi023 bash[21372]: cluster 2024-03-11T21:12:08.365937+0000 mgr.y (mgr.24380) 791 : cluster 0 pgmap v906: 158 pgs: 158 active+clean; 112 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 14 MiB/s wr, 216 op/s 2024-03-11T21:12:10.124 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:09 smithi023 bash[21372]: cluster 2024-03-11T21:12:08.636662+0000 osd.2 (osd.2) 257 : cluster 0 purged_snaps scrub starts 2024-03-11T21:12:10.124 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:09 smithi023 bash[21372]: cluster 2024-03-11T21:12:08.636734+0000 osd.2 (osd.2) 258 : cluster 0 purged_snaps scrub ok 2024-03-11T21:12:10.124 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:09 smithi023 bash[21372]: audit 2024-03-11T21:12:08.670999+0000 mon.a (mon.0) 1811 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:12:10.124 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:09 smithi023 bash[21372]: cluster 2024-03-11T21:12:08.991904+0000 osd.6 (osd.6) 382 : cluster 0 4.18 scrub starts 2024-03-11T21:12:10.124 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:09 smithi023 bash[21372]: cluster 2024-03-11T21:12:08.997104+0000 osd.6 (osd.6) 383 : cluster 0 4.18 scrub ok 2024-03-11T21:12:10.124 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:09 smithi023 bash[21372]: cluster 2024-03-11T21:12:09.371930+0000 osd.5 (osd.5) 457 : cluster 0 2.16 scrub starts 2024-03-11T21:12:10.125 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:09 smithi023 bash[21372]: cluster 2024-03-11T21:12:09.374256+0000 osd.5 (osd.5) 458 : cluster 0 2.16 scrub ok 2024-03-11T21:12:10.125 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:09 smithi023 bash[29146]: cluster 2024-03-11T21:12:08.107984+0000 osd.7 (osd.7) 497 : cluster 0 6.3 scrub starts 2024-03-11T21:12:10.125 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:09 smithi023 bash[29146]: cluster 2024-03-11T21:12:08.234817+0000 osd.7 (osd.7) 498 : cluster 0 6.3 scrub ok 2024-03-11T21:12:10.125 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:09 smithi023 bash[29146]: cluster 2024-03-11T21:12:08.365937+0000 mgr.y (mgr.24380) 791 : cluster 0 pgmap v906: 158 pgs: 158 active+clean; 112 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 14 MiB/s wr, 216 op/s 2024-03-11T21:12:10.125 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:09 smithi023 bash[29146]: cluster 2024-03-11T21:12:08.636662+0000 osd.2 (osd.2) 257 : cluster 0 purged_snaps scrub starts 2024-03-11T21:12:10.125 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:09 smithi023 bash[29146]: cluster 2024-03-11T21:12:08.636734+0000 osd.2 (osd.2) 258 : cluster 0 purged_snaps scrub ok 2024-03-11T21:12:10.125 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:09 smithi023 bash[29146]: audit 2024-03-11T21:12:08.670999+0000 mon.a (mon.0) 1811 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:12:10.125 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:09 smithi023 bash[29146]: cluster 2024-03-11T21:12:08.991904+0000 osd.6 (osd.6) 382 : cluster 0 4.18 scrub starts 2024-03-11T21:12:10.125 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:09 smithi023 bash[29146]: cluster 2024-03-11T21:12:08.997104+0000 osd.6 (osd.6) 383 : cluster 0 4.18 scrub ok 2024-03-11T21:12:10.126 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:09 smithi023 bash[29146]: cluster 2024-03-11T21:12:09.371930+0000 osd.5 (osd.5) 457 : cluster 0 2.16 scrub starts 2024-03-11T21:12:10.126 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:09 smithi023 bash[29146]: cluster 2024-03-11T21:12:09.374256+0000 osd.5 (osd.5) 458 : cluster 0 2.16 scrub ok 2024-03-11T21:12:10.196 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:10.220 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:10.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:10.223 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:12:10.324 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:10.351 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:10.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:10.355 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:12:10.456 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:12:10.488 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:12:10.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:10.492 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:12:10.593 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:12:10.617 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:12:10.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:10.619 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:12:10.723 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:12:10.734 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:10 smithi023 bash[29146]: cluster 2024-03-11T21:12:08.857566+0000 osd.3 (osd.3) 442 : cluster 0 3.b scrub starts 2024-03-11T21:12:10.734 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:10 smithi023 bash[29146]: cluster 2024-03-11T21:12:08.861788+0000 osd.3 (osd.3) 443 : cluster 0 3.b scrub ok 2024-03-11T21:12:10.734 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:10 smithi023 bash[29146]: cluster 2024-03-11T21:12:09.115551+0000 osd.7 (osd.7) 499 : cluster 0 2.1c scrub starts 2024-03-11T21:12:10.734 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:10 smithi023 bash[29146]: cluster 2024-03-11T21:12:09.117888+0000 osd.7 (osd.7) 500 : cluster 0 2.1c scrub ok 2024-03-11T21:12:10.735 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:10 smithi023 bash[29146]: cluster 2024-03-11T21:12:09.513155+0000 mon.a (mon.0) 1812 : cluster 0 osdmap e388: 8 total, 8 up, 8 in 2024-03-11T21:12:10.735 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:10 smithi023 bash[29146]: cluster 2024-03-11T21:12:09.695253+0000 osd.1 (osd.1) 267 : cluster 0 purged_snaps scrub starts 2024-03-11T21:12:10.735 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:10 smithi023 bash[29146]: cluster 2024-03-11T21:12:09.695358+0000 osd.1 (osd.1) 268 : cluster 0 purged_snaps scrub ok 2024-03-11T21:12:10.735 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:10 smithi023 bash[29146]: cluster 2024-03-11T21:12:09.795376+0000 osd.1 (osd.1) 269 : cluster 0 2.15 scrub starts 2024-03-11T21:12:10.735 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:10 smithi023 bash[29146]: cluster 2024-03-11T21:12:09.798970+0000 osd.1 (osd.1) 270 : cluster 0 2.15 scrub ok 2024-03-11T21:12:10.735 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:10 smithi023 bash[29146]: cluster 2024-03-11T21:12:09.975547+0000 osd.6 (osd.6) 384 : cluster 0 6.1e scrub starts 2024-03-11T21:12:10.735 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:10 smithi023 bash[29146]: cluster 2024-03-11T21:12:09.993396+0000 osd.6 (osd.6) 385 : cluster 0 6.1e scrub ok 2024-03-11T21:12:10.735 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:10 smithi023 bash[29146]: cluster 2024-03-11T21:12:10.333645+0000 osd.5 (osd.5) 459 : cluster 0 4.1 scrub starts 2024-03-11T21:12:10.735 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:10 smithi023 bash[29146]: cluster 2024-03-11T21:12:10.335101+0000 osd.5 (osd.5) 460 : cluster 0 4.1 scrub ok 2024-03-11T21:12:10.735 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:10 smithi023 bash[29146]: cluster 2024-03-11T21:12:10.358375+0000 mon.a (mon.0) 1813 : cluster 0 osdmap e389: 8 total, 8 up, 8 in 2024-03-11T21:12:10.735 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:10 smithi023 bash[29146]: audit 2024-03-11T21:12:10.512715+0000 mon.a (mon.0) 1814 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:12:10.735 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:10 smithi023 bash[21372]: cluster 2024-03-11T21:12:08.857566+0000 osd.3 (osd.3) 442 : cluster 0 3.b scrub starts 2024-03-11T21:12:10.736 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:10 smithi023 bash[21372]: cluster 2024-03-11T21:12:08.861788+0000 osd.3 (osd.3) 443 : cluster 0 3.b scrub ok 2024-03-11T21:12:10.736 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:10 smithi023 bash[21372]: cluster 2024-03-11T21:12:09.115551+0000 osd.7 (osd.7) 499 : cluster 0 2.1c scrub starts 2024-03-11T21:12:10.736 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:10 smithi023 bash[21372]: cluster 2024-03-11T21:12:09.117888+0000 osd.7 (osd.7) 500 : cluster 0 2.1c scrub ok 2024-03-11T21:12:10.736 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:10 smithi023 bash[21372]: cluster 2024-03-11T21:12:09.513155+0000 mon.a (mon.0) 1812 : cluster 0 osdmap e388: 8 total, 8 up, 8 in 2024-03-11T21:12:10.736 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:10 smithi023 bash[21372]: cluster 2024-03-11T21:12:09.695253+0000 osd.1 (osd.1) 267 : cluster 0 purged_snaps scrub starts 2024-03-11T21:12:10.736 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:10 smithi023 bash[21372]: cluster 2024-03-11T21:12:09.695358+0000 osd.1 (osd.1) 268 : cluster 0 purged_snaps scrub ok 2024-03-11T21:12:10.736 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:10 smithi023 bash[21372]: cluster 2024-03-11T21:12:09.795376+0000 osd.1 (osd.1) 269 : cluster 0 2.15 scrub starts 2024-03-11T21:12:10.736 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:10 smithi023 bash[21372]: cluster 2024-03-11T21:12:09.798970+0000 osd.1 (osd.1) 270 : cluster 0 2.15 scrub ok 2024-03-11T21:12:10.736 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:10 smithi023 bash[21372]: cluster 2024-03-11T21:12:09.975547+0000 osd.6 (osd.6) 384 : cluster 0 6.1e scrub starts 2024-03-11T21:12:10.736 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:10 smithi023 bash[21372]: cluster 2024-03-11T21:12:09.993396+0000 osd.6 (osd.6) 385 : cluster 0 6.1e scrub ok 2024-03-11T21:12:10.736 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:10 smithi023 bash[21372]: cluster 2024-03-11T21:12:10.333645+0000 osd.5 (osd.5) 459 : cluster 0 4.1 scrub starts 2024-03-11T21:12:10.736 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:10 smithi023 bash[21372]: cluster 2024-03-11T21:12:10.335101+0000 osd.5 (osd.5) 460 : cluster 0 4.1 scrub ok 2024-03-11T21:12:10.736 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:10 smithi023 bash[21372]: cluster 2024-03-11T21:12:10.358375+0000 mon.a (mon.0) 1813 : cluster 0 osdmap e389: 8 total, 8 up, 8 in 2024-03-11T21:12:10.737 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:10 smithi023 bash[21372]: audit 2024-03-11T21:12:10.512715+0000 mon.a (mon.0) 1814 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:12:10.754 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:12:10.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:10.758 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:12:10.833 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 6 16 14358 14342 149.275 139.375 0.00179817 0.00668993 2024-03-11T21:12:10.859 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:12:10.884 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:12:10.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:10.887 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:12:10.987 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:10.995 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:10 smithi146 bash[23809]: cluster 2024-03-11T21:12:08.857566+0000 osd.3 (osd.3) 442 : cluster 0 3.b scrub starts 2024-03-11T21:12:10.996 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:10 smithi146 bash[23809]: cluster 2024-03-11T21:12:08.861788+0000 osd.3 (osd.3) 443 : cluster 0 3.b scrub ok 2024-03-11T21:12:10.996 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:10 smithi146 bash[23809]: cluster 2024-03-11T21:12:09.115551+0000 osd.7 (osd.7) 499 : cluster 0 2.1c scrub starts 2024-03-11T21:12:10.996 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:10 smithi146 bash[23809]: cluster 2024-03-11T21:12:09.117888+0000 osd.7 (osd.7) 500 : cluster 0 2.1c scrub ok 2024-03-11T21:12:10.996 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:10 smithi146 bash[23809]: cluster 2024-03-11T21:12:09.513155+0000 mon.a (mon.0) 1812 : cluster 0 osdmap e388: 8 total, 8 up, 8 in 2024-03-11T21:12:10.996 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:10 smithi146 bash[23809]: cluster 2024-03-11T21:12:09.695253+0000 osd.1 (osd.1) 267 : cluster 0 purged_snaps scrub starts 2024-03-11T21:12:10.996 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:10 smithi146 bash[23809]: cluster 2024-03-11T21:12:09.695358+0000 osd.1 (osd.1) 268 : cluster 0 purged_snaps scrub ok 2024-03-11T21:12:10.996 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:10 smithi146 bash[23809]: cluster 2024-03-11T21:12:09.795376+0000 osd.1 (osd.1) 269 : cluster 0 2.15 scrub starts 2024-03-11T21:12:10.996 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:10 smithi146 bash[23809]: cluster 2024-03-11T21:12:09.798970+0000 osd.1 (osd.1) 270 : cluster 0 2.15 scrub ok 2024-03-11T21:12:10.996 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:10 smithi146 bash[23809]: cluster 2024-03-11T21:12:09.975547+0000 osd.6 (osd.6) 384 : cluster 0 6.1e scrub starts 2024-03-11T21:12:10.996 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:10 smithi146 bash[23809]: cluster 2024-03-11T21:12:09.993396+0000 osd.6 (osd.6) 385 : cluster 0 6.1e scrub ok 2024-03-11T21:12:10.996 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:10 smithi146 bash[23809]: cluster 2024-03-11T21:12:10.333645+0000 osd.5 (osd.5) 459 : cluster 0 4.1 scrub starts 2024-03-11T21:12:10.996 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:10 smithi146 bash[23809]: cluster 2024-03-11T21:12:10.335101+0000 osd.5 (osd.5) 460 : cluster 0 4.1 scrub ok 2024-03-11T21:12:10.997 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:10 smithi146 bash[23809]: cluster 2024-03-11T21:12:10.358375+0000 mon.a (mon.0) 1813 : cluster 0 osdmap e389: 8 total, 8 up, 8 in 2024-03-11T21:12:10.997 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:10 smithi146 bash[23809]: audit 2024-03-11T21:12:10.512715+0000 mon.a (mon.0) 1814 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:12:11.006 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:11.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:11.008 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:12:11.109 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:12:11.134 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:12:11.135 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:12:11.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:11.142 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:12:11.242 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:12:11.281 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:12:11.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:11.284 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:12:11.385 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:12:11.410 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:12:11.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:11.412 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:12:11.512 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:12:11.537 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:12:11.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:11.539 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:12:11.640 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:12:11.664 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:12:11.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:11.666 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:12:11.767 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:12:11.799 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:12:11.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:11.801 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:12:11.832 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 7 16 14989 14973 133.593 39.4375 0.00369575 0.00742347 2024-03-11T21:12:11.902 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:12:11.914 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:11 smithi146 bash[23809]: cluster 2024-03-11T21:12:09.857338+0000 osd.3 (osd.3) 444 : cluster 0 5.0 scrub starts 2024-03-11T21:12:11.915 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:11 smithi146 bash[23809]: cluster 2024-03-11T21:12:09.860778+0000 osd.3 (osd.3) 445 : cluster 0 5.0 scrub ok 2024-03-11T21:12:11.917 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:11 smithi146 bash[23809]: cluster 2024-03-11T21:12:10.366422+0000 mgr.y (mgr.24380) 792 : cluster 0 pgmap v909: 158 pgs: 158 active+clean; 635 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.7 KiB/s rd, 106 MiB/s wr, 1.69k op/s 2024-03-11T21:12:11.917 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:11 smithi146 bash[23809]: audit 2024-03-11T21:12:10.521107+0000 mon.a (mon.0) 1815 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:12:11.937 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:12:11.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:11.939 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:12:12.027 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:11 smithi023 bash[21372]: cluster 2024-03-11T21:12:09.857338+0000 osd.3 (osd.3) 444 : cluster 0 5.0 scrub starts 2024-03-11T21:12:12.027 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:11 smithi023 bash[21372]: cluster 2024-03-11T21:12:09.860778+0000 osd.3 (osd.3) 445 : cluster 0 5.0 scrub ok 2024-03-11T21:12:12.027 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:11 smithi023 bash[21372]: cluster 2024-03-11T21:12:10.366422+0000 mgr.y (mgr.24380) 792 : cluster 0 pgmap v909: 158 pgs: 158 active+clean; 635 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.7 KiB/s rd, 106 MiB/s wr, 1.69k op/s 2024-03-11T21:12:12.027 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:11 smithi023 bash[21372]: audit 2024-03-11T21:12:10.521107+0000 mon.a (mon.0) 1815 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:12:12.027 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:11 smithi023 bash[29146]: cluster 2024-03-11T21:12:09.857338+0000 osd.3 (osd.3) 444 : cluster 0 5.0 scrub starts 2024-03-11T21:12:12.027 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:11 smithi023 bash[29146]: cluster 2024-03-11T21:12:09.860778+0000 osd.3 (osd.3) 445 : cluster 0 5.0 scrub ok 2024-03-11T21:12:12.028 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:11 smithi023 bash[29146]: cluster 2024-03-11T21:12:10.366422+0000 mgr.y (mgr.24380) 792 : cluster 0 pgmap v909: 158 pgs: 158 active+clean; 635 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.7 KiB/s rd, 106 MiB/s wr, 1.69k op/s 2024-03-11T21:12:12.028 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:11 smithi023 bash[29146]: audit 2024-03-11T21:12:10.521107+0000 mon.a (mon.0) 1815 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:12:12.040 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:12:12.066 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:12:12.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:12.069 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:12:12.170 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:12:12.195 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:12:12.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:12.198 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:12:12.299 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:12.325 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:12.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:12.331 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:12:12.432 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:12:12.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:12.465 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:12:12.465 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:12:12.566 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:12.594 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:12.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:12.597 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:12:12.698 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:12:12.722 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:12:12.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:12.725 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:12:12.777 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:12 smithi023 bash[21372]: audit 2024-03-11T21:12:11.554587+0000 mon.a (mon.0) 1816 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:12:12.777 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:12 smithi023 bash[21372]: audit 2024-03-11T21:12:11.555628+0000 mon.a (mon.0) 1817 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:12:12.777 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:12 smithi023 bash[21372]: audit 2024-03-11T21:12:11.565020+0000 mon.a (mon.0) 1818 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:12:12.777 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:12 smithi023 bash[21372]: cluster 2024-03-11T21:12:11.794657+0000 osd.1 (osd.1) 271 : cluster 0 2.d scrub starts 2024-03-11T21:12:12.777 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:12 smithi023 bash[21372]: cluster 2024-03-11T21:12:11.805933+0000 osd.1 (osd.1) 272 : cluster 0 2.d scrub ok 2024-03-11T21:12:12.777 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:12 smithi023 bash[21372]: cluster 2024-03-11T21:12:11.981715+0000 osd.6 (osd.6) 386 : cluster 0 2.7 scrub starts 2024-03-11T21:12:12.777 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:12 smithi023 bash[21372]: cluster 2024-03-11T21:12:11.984658+0000 osd.6 (osd.6) 387 : cluster 0 2.7 scrub ok 2024-03-11T21:12:12.777 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:12 smithi023 bash[21372]: cluster 2024-03-11T21:12:12.371632+0000 osd.5 (osd.5) 461 : cluster 0 2.2 scrub starts 2024-03-11T21:12:12.777 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:12 smithi023 bash[21372]: cluster 2024-03-11T21:12:12.373106+0000 osd.5 (osd.5) 462 : cluster 0 2.2 scrub ok 2024-03-11T21:12:12.778 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:12 smithi023 bash[29146]: audit 2024-03-11T21:12:11.554587+0000 mon.a (mon.0) 1816 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:12:12.778 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:12 smithi023 bash[29146]: audit 2024-03-11T21:12:11.555628+0000 mon.a (mon.0) 1817 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:12:12.778 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:12 smithi023 bash[29146]: audit 2024-03-11T21:12:11.565020+0000 mon.a (mon.0) 1818 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:12:12.778 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:12 smithi023 bash[29146]: cluster 2024-03-11T21:12:11.794657+0000 osd.1 (osd.1) 271 : cluster 0 2.d scrub starts 2024-03-11T21:12:12.778 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:12 smithi023 bash[29146]: cluster 2024-03-11T21:12:11.805933+0000 osd.1 (osd.1) 272 : cluster 0 2.d scrub ok 2024-03-11T21:12:12.778 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:12 smithi023 bash[29146]: cluster 2024-03-11T21:12:11.981715+0000 osd.6 (osd.6) 386 : cluster 0 2.7 scrub starts 2024-03-11T21:12:12.778 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:12 smithi023 bash[29146]: cluster 2024-03-11T21:12:11.984658+0000 osd.6 (osd.6) 387 : cluster 0 2.7 scrub ok 2024-03-11T21:12:12.778 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:12 smithi023 bash[29146]: cluster 2024-03-11T21:12:12.371632+0000 osd.5 (osd.5) 461 : cluster 0 2.2 scrub starts 2024-03-11T21:12:12.778 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:12 smithi023 bash[29146]: cluster 2024-03-11T21:12:12.373106+0000 osd.5 (osd.5) 462 : cluster 0 2.2 scrub ok 2024-03-11T21:12:12.826 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:12.831 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 8 16 17153 17137 133.799 135.25 0.00631742 0.00746694 2024-03-11T21:12:12.836 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:12 smithi146 bash[23809]: audit 2024-03-11T21:12:11.554587+0000 mon.a (mon.0) 1816 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:12:12.836 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:12 smithi146 bash[23809]: audit 2024-03-11T21:12:11.555628+0000 mon.a (mon.0) 1817 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:12:12.836 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:12 smithi146 bash[23809]: audit 2024-03-11T21:12:11.565020+0000 mon.a (mon.0) 1818 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:12:12.836 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:12 smithi146 bash[23809]: cluster 2024-03-11T21:12:11.794657+0000 osd.1 (osd.1) 271 : cluster 0 2.d scrub starts 2024-03-11T21:12:12.837 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:12 smithi146 bash[23809]: cluster 2024-03-11T21:12:11.805933+0000 osd.1 (osd.1) 272 : cluster 0 2.d scrub ok 2024-03-11T21:12:12.837 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:12 smithi146 bash[23809]: cluster 2024-03-11T21:12:11.981715+0000 osd.6 (osd.6) 386 : cluster 0 2.7 scrub starts 2024-03-11T21:12:12.837 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:12 smithi146 bash[23809]: cluster 2024-03-11T21:12:11.984658+0000 osd.6 (osd.6) 387 : cluster 0 2.7 scrub ok 2024-03-11T21:12:12.837 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:12 smithi146 bash[23809]: cluster 2024-03-11T21:12:12.371632+0000 osd.5 (osd.5) 461 : cluster 0 2.2 scrub starts 2024-03-11T21:12:12.837 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:12 smithi146 bash[23809]: cluster 2024-03-11T21:12:12.373106+0000 osd.5 (osd.5) 462 : cluster 0 2.2 scrub ok 2024-03-11T21:12:12.851 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:12.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:12.854 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:12:12.954 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:12.976 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:12.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:12.978 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:12:13.079 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:12:13.098 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:12:13.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:13.100 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:12:13.201 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:12:13.219 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:12:13.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:13.221 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:12:13.322 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:12:13.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:13.350 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:12:13.351 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:12:13.452 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:13.483 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:13.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:13.485 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:12:13.586 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:13.610 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:13.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:13.612 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:12:13.713 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:12:13.723 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:13 smithi146 bash[23809]: cluster 2024-03-11T21:12:12.098226+0000 osd.0 (osd.0) 358 : cluster 0 6.10 scrub starts 2024-03-11T21:12:13.723 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:13 smithi146 bash[23809]: cluster 2024-03-11T21:12:12.124195+0000 osd.7 (osd.7) 501 : cluster 0 2.1d scrub starts 2024-03-11T21:12:13.723 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:13 smithi146 bash[23809]: cluster 2024-03-11T21:12:12.128403+0000 osd.7 (osd.7) 502 : cluster 0 2.1d scrub ok 2024-03-11T21:12:13.723 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:13 smithi146 bash[23809]: cluster 2024-03-11T21:12:12.183515+0000 osd.0 (osd.0) 359 : cluster 0 6.10 scrub ok 2024-03-11T21:12:13.723 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:13 smithi146 bash[23809]: cluster 2024-03-11T21:12:12.367104+0000 mgr.y (mgr.24380) 793 : cluster 0 pgmap v910: 158 pgs: 158 active+clean; 841 MiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 107 MiB/s wr, 1.71k op/s 2024-03-11T21:12:13.738 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:12:13.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:13.741 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:12:13.777 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:13 smithi023 bash[21372]: cluster 2024-03-11T21:12:12.098226+0000 osd.0 (osd.0) 358 : cluster 0 6.10 scrub starts 2024-03-11T21:12:13.777 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:13 smithi023 bash[21372]: cluster 2024-03-11T21:12:12.124195+0000 osd.7 (osd.7) 501 : cluster 0 2.1d scrub starts 2024-03-11T21:12:13.777 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:13 smithi023 bash[21372]: cluster 2024-03-11T21:12:12.128403+0000 osd.7 (osd.7) 502 : cluster 0 2.1d scrub ok 2024-03-11T21:12:13.777 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:13 smithi023 bash[21372]: cluster 2024-03-11T21:12:12.183515+0000 osd.0 (osd.0) 359 : cluster 0 6.10 scrub ok 2024-03-11T21:12:13.777 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:13 smithi023 bash[21372]: cluster 2024-03-11T21:12:12.367104+0000 mgr.y (mgr.24380) 793 : cluster 0 pgmap v910: 158 pgs: 158 active+clean; 841 MiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 107 MiB/s wr, 1.71k op/s 2024-03-11T21:12:13.777 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:13 smithi023 bash[29146]: cluster 2024-03-11T21:12:12.098226+0000 osd.0 (osd.0) 358 : cluster 0 6.10 scrub starts 2024-03-11T21:12:13.777 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:13 smithi023 bash[29146]: cluster 2024-03-11T21:12:12.124195+0000 osd.7 (osd.7) 501 : cluster 0 2.1d scrub starts 2024-03-11T21:12:13.777 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:13 smithi023 bash[29146]: cluster 2024-03-11T21:12:12.128403+0000 osd.7 (osd.7) 502 : cluster 0 2.1d scrub ok 2024-03-11T21:12:13.778 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:13 smithi023 bash[29146]: cluster 2024-03-11T21:12:12.183515+0000 osd.0 (osd.0) 359 : cluster 0 6.10 scrub ok 2024-03-11T21:12:13.778 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:13 smithi023 bash[29146]: cluster 2024-03-11T21:12:12.367104+0000 mgr.y (mgr.24380) 793 : cluster 0 pgmap v910: 158 pgs: 158 active+clean; 841 MiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 107 MiB/s wr, 1.71k op/s 2024-03-11T21:12:13.832 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 9 16 18479 18463 128.142 82.875 0.00641466 0.00779525 2024-03-11T21:12:13.841 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:12:13.877 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:12:13.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:13.880 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:12:13.981 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:12:14.009 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:12:14.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:14.013 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:12:14.114 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:12:14.137 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:12:14.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:14.141 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:12:14.241 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:12:14.269 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:12:14.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:14.273 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:12:14.375 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:14.402 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:14.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:14.405 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:12:14.506 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:12:14.530 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:12:14.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:14.533 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:12:14.634 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:14.642 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:14 smithi146 bash[23809]: cluster 2024-03-11T21:12:13.053572+0000 osd.0 (osd.0) 360 : cluster 0 3.1b scrub starts 2024-03-11T21:12:14.642 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:14 smithi146 bash[23809]: cluster 2024-03-11T21:12:13.057920+0000 osd.0 (osd.0) 361 : cluster 0 3.1b scrub ok 2024-03-11T21:12:14.642 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:14 smithi146 bash[23809]: cluster 2024-03-11T21:12:13.097216+0000 osd.7 (osd.7) 503 : cluster 0 6.8 scrub starts 2024-03-11T21:12:14.643 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:14 smithi146 bash[23809]: cluster 2024-03-11T21:12:13.099213+0000 osd.7 (osd.7) 504 : cluster 0 6.8 scrub ok 2024-03-11T21:12:14.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:14.661 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:14.661 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:12:14.762 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:12:14.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:14 smithi023 bash[21372]: cluster 2024-03-11T21:12:13.053572+0000 osd.0 (osd.0) 360 : cluster 0 3.1b scrub starts 2024-03-11T21:12:14.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:14 smithi023 bash[21372]: cluster 2024-03-11T21:12:13.057920+0000 osd.0 (osd.0) 361 : cluster 0 3.1b scrub ok 2024-03-11T21:12:14.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:14 smithi023 bash[21372]: cluster 2024-03-11T21:12:13.097216+0000 osd.7 (osd.7) 503 : cluster 0 6.8 scrub starts 2024-03-11T21:12:14.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:14 smithi023 bash[21372]: cluster 2024-03-11T21:12:13.099213+0000 osd.7 (osd.7) 504 : cluster 0 6.8 scrub ok 2024-03-11T21:12:14.773 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:14 smithi023 bash[29146]: cluster 2024-03-11T21:12:13.053572+0000 osd.0 (osd.0) 360 : cluster 0 3.1b scrub starts 2024-03-11T21:12:14.773 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:14 smithi023 bash[29146]: cluster 2024-03-11T21:12:13.057920+0000 osd.0 (osd.0) 361 : cluster 0 3.1b scrub ok 2024-03-11T21:12:14.773 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:14 smithi023 bash[29146]: cluster 2024-03-11T21:12:13.097216+0000 osd.7 (osd.7) 503 : cluster 0 6.8 scrub starts 2024-03-11T21:12:14.773 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:14 smithi023 bash[29146]: cluster 2024-03-11T21:12:13.099213+0000 osd.7 (osd.7) 504 : cluster 0 6.8 scrub ok 2024-03-11T21:12:14.795 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:12:14.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:14.800 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:12:14.832 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 10 15 20513 20498 128.046 127.188 0.0106463 0.00780392 2024-03-11T21:12:14.901 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:12:14.924 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:12:14.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:14.926 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:12:15.027 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:12:15.058 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:12:15.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:15.061 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:12:15.161 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:15.186 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:15.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:15.189 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:12:15.290 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:12:15.319 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:12:15.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:15.321 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:12:15.421 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:12:15.447 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:12:15.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:15.451 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:12:15.552 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:12:15.578 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:12:15.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:15.581 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:12:15.682 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:15.705 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:15.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:15.707 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:12:15.777 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:15 smithi023 bash[21372]: cluster 2024-03-11T21:12:13.932906+0000 osd.3 (osd.3) 446 : cluster 0 4.14 scrub starts 2024-03-11T21:12:15.777 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:15 smithi023 bash[21372]: cluster 2024-03-11T21:12:13.934160+0000 osd.3 (osd.3) 447 : cluster 0 4.14 scrub ok 2024-03-11T21:12:15.777 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:15 smithi023 bash[21372]: cluster 2024-03-11T21:12:14.026550+0000 osd.0 (osd.0) 362 : cluster 0 3.6 scrub starts 2024-03-11T21:12:15.777 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:15 smithi023 bash[21372]: cluster 2024-03-11T21:12:14.031336+0000 osd.0 (osd.0) 363 : cluster 0 3.6 scrub ok 2024-03-11T21:12:15.777 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:15 smithi023 bash[21372]: cluster 2024-03-11T21:12:14.367534+0000 mgr.y (mgr.24380) 794 : cluster 0 pgmap v911: 158 pgs: 158 active+clean; 841 MiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 101 MiB/s wr, 1.61k op/s 2024-03-11T21:12:15.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:15 smithi023 bash[21372]: cluster 2024-03-11T21:12:14.748628+0000 osd.1 (osd.1) 273 : cluster 0 4.9 scrub starts 2024-03-11T21:12:15.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:15 smithi023 bash[21372]: cluster 2024-03-11T21:12:14.753085+0000 osd.1 (osd.1) 274 : cluster 0 4.9 scrub ok 2024-03-11T21:12:15.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:15 smithi023 bash[21372]: cluster 2024-03-11T21:12:15.311579+0000 osd.5 (osd.5) 463 : cluster 0 2.18 scrub starts 2024-03-11T21:12:15.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:15 smithi023 bash[21372]: cluster 2024-03-11T21:12:15.313019+0000 osd.5 (osd.5) 464 : cluster 0 2.18 scrub ok 2024-03-11T21:12:15.778 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:15 smithi023 bash[29146]: cluster 2024-03-11T21:12:13.932906+0000 osd.3 (osd.3) 446 : cluster 0 4.14 scrub starts 2024-03-11T21:12:15.778 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:15 smithi023 bash[29146]: cluster 2024-03-11T21:12:13.934160+0000 osd.3 (osd.3) 447 : cluster 0 4.14 scrub ok 2024-03-11T21:12:15.778 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:15 smithi023 bash[29146]: cluster 2024-03-11T21:12:14.026550+0000 osd.0 (osd.0) 362 : cluster 0 3.6 scrub starts 2024-03-11T21:12:15.778 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:15 smithi023 bash[29146]: cluster 2024-03-11T21:12:14.031336+0000 osd.0 (osd.0) 363 : cluster 0 3.6 scrub ok 2024-03-11T21:12:15.778 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:15 smithi023 bash[29146]: cluster 2024-03-11T21:12:14.367534+0000 mgr.y (mgr.24380) 794 : cluster 0 pgmap v911: 158 pgs: 158 active+clean; 841 MiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 101 MiB/s wr, 1.61k op/s 2024-03-11T21:12:15.779 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:15 smithi023 bash[29146]: cluster 2024-03-11T21:12:14.748628+0000 osd.1 (osd.1) 273 : cluster 0 4.9 scrub starts 2024-03-11T21:12:15.779 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:15 smithi023 bash[29146]: cluster 2024-03-11T21:12:14.753085+0000 osd.1 (osd.1) 274 : cluster 0 4.9 scrub ok 2024-03-11T21:12:15.779 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:15 smithi023 bash[29146]: cluster 2024-03-11T21:12:15.311579+0000 osd.5 (osd.5) 463 : cluster 0 2.18 scrub starts 2024-03-11T21:12:15.779 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:15 smithi023 bash[29146]: cluster 2024-03-11T21:12:15.313019+0000 osd.5 (osd.5) 464 : cluster 0 2.18 scrub ok 2024-03-11T21:12:15.808 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:12:15.833 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 11 16 22119 22103 125.519 100.312 0.00853964 0.00795929 2024-03-11T21:12:15.836 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:12:15.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:15.841 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:12:15.910 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:15 smithi146 bash[23809]: cluster 2024-03-11T21:12:13.932906+0000 osd.3 (osd.3) 446 : cluster 0 4.14 scrub starts 2024-03-11T21:12:15.910 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:15 smithi146 bash[23809]: cluster 2024-03-11T21:12:13.934160+0000 osd.3 (osd.3) 447 : cluster 0 4.14 scrub ok 2024-03-11T21:12:15.910 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:15 smithi146 bash[23809]: cluster 2024-03-11T21:12:14.026550+0000 osd.0 (osd.0) 362 : cluster 0 3.6 scrub starts 2024-03-11T21:12:15.910 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:15 smithi146 bash[23809]: cluster 2024-03-11T21:12:14.031336+0000 osd.0 (osd.0) 363 : cluster 0 3.6 scrub ok 2024-03-11T21:12:15.910 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:15 smithi146 bash[23809]: cluster 2024-03-11T21:12:14.367534+0000 mgr.y (mgr.24380) 794 : cluster 0 pgmap v911: 158 pgs: 158 active+clean; 841 MiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 101 MiB/s wr, 1.61k op/s 2024-03-11T21:12:15.910 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:15 smithi146 bash[23809]: cluster 2024-03-11T21:12:14.748628+0000 osd.1 (osd.1) 273 : cluster 0 4.9 scrub starts 2024-03-11T21:12:15.910 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:15 smithi146 bash[23809]: cluster 2024-03-11T21:12:14.753085+0000 osd.1 (osd.1) 274 : cluster 0 4.9 scrub ok 2024-03-11T21:12:15.910 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:15 smithi146 bash[23809]: cluster 2024-03-11T21:12:15.311579+0000 osd.5 (osd.5) 463 : cluster 0 2.18 scrub starts 2024-03-11T21:12:15.911 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:15 smithi146 bash[23809]: cluster 2024-03-11T21:12:15.313019+0000 osd.5 (osd.5) 464 : cluster 0 2.18 scrub ok 2024-03-11T21:12:15.942 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:12:15.951 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:12:15 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:12:15.967 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:12:15.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:15.970 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:12:16.071 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:12:16.096 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:12:16.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:16.098 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:12:16.199 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:12:16.227 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:12:16.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:16.230 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:12:16.331 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:12:16.359 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:12:16.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:16.362 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:12:16.463 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:16.492 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:16.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:16.495 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:12:16.596 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:12:16.635 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:12:16.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:16.640 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:12:16.741 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:16.752 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:16 smithi146 bash[23809]: cluster 2024-03-11T21:12:15.023986+0000 osd.0 (osd.0) 364 : cluster 0 3.1 scrub starts 2024-03-11T21:12:16.752 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:16 smithi146 bash[23809]: cluster 2024-03-11T21:12:15.033026+0000 osd.0 (osd.0) 365 : cluster 0 3.1 scrub ok 2024-03-11T21:12:16.752 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:16 smithi146 bash[23809]: cluster 2024-03-11T21:12:15.145034+0000 osd.7 (osd.7) 505 : cluster 0 3.e scrub starts 2024-03-11T21:12:16.752 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:16 smithi146 bash[23809]: cluster 2024-03-11T21:12:15.148486+0000 osd.7 (osd.7) 506 : cluster 0 3.e scrub ok 2024-03-11T21:12:16.752 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:16 smithi146 bash[23809]: cluster 2024-03-11T21:12:15.741175+0000 osd.1 (osd.1) 275 : cluster 0 2.10 scrub starts 2024-03-11T21:12:16.753 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:16 smithi146 bash[23809]: cluster 2024-03-11T21:12:15.742327+0000 osd.1 (osd.1) 276 : cluster 0 2.10 scrub ok 2024-03-11T21:12:16.763 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:16.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:16.766 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:12:16.832 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 12 16 24086 24070 125.303 122.938 0.00852845 0.00797427 2024-03-11T21:12:16.866 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:12:16.876 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:16 smithi023 bash[21372]: cluster 2024-03-11T21:12:15.023986+0000 osd.0 (osd.0) 364 : cluster 0 3.1 scrub starts 2024-03-11T21:12:16.876 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:16 smithi023 bash[21372]: cluster 2024-03-11T21:12:15.033026+0000 osd.0 (osd.0) 365 : cluster 0 3.1 scrub ok 2024-03-11T21:12:16.876 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:16 smithi023 bash[21372]: cluster 2024-03-11T21:12:15.145034+0000 osd.7 (osd.7) 505 : cluster 0 3.e scrub starts 2024-03-11T21:12:16.876 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:16 smithi023 bash[21372]: cluster 2024-03-11T21:12:15.148486+0000 osd.7 (osd.7) 506 : cluster 0 3.e scrub ok 2024-03-11T21:12:16.876 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:16 smithi023 bash[21372]: cluster 2024-03-11T21:12:15.741175+0000 osd.1 (osd.1) 275 : cluster 0 2.10 scrub starts 2024-03-11T21:12:16.876 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:16 smithi023 bash[21372]: cluster 2024-03-11T21:12:15.742327+0000 osd.1 (osd.1) 276 : cluster 0 2.10 scrub ok 2024-03-11T21:12:16.876 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:16 smithi023 bash[29146]: cluster 2024-03-11T21:12:15.023986+0000 osd.0 (osd.0) 364 : cluster 0 3.1 scrub starts 2024-03-11T21:12:16.877 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:16 smithi023 bash[29146]: cluster 2024-03-11T21:12:15.033026+0000 osd.0 (osd.0) 365 : cluster 0 3.1 scrub ok 2024-03-11T21:12:16.877 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:16 smithi023 bash[29146]: cluster 2024-03-11T21:12:15.145034+0000 osd.7 (osd.7) 505 : cluster 0 3.e scrub starts 2024-03-11T21:12:16.877 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:16 smithi023 bash[29146]: cluster 2024-03-11T21:12:15.148486+0000 osd.7 (osd.7) 506 : cluster 0 3.e scrub ok 2024-03-11T21:12:16.877 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:16 smithi023 bash[29146]: cluster 2024-03-11T21:12:15.741175+0000 osd.1 (osd.1) 275 : cluster 0 2.10 scrub starts 2024-03-11T21:12:16.877 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:16 smithi023 bash[29146]: cluster 2024-03-11T21:12:15.742327+0000 osd.1 (osd.1) 276 : cluster 0 2.10 scrub ok 2024-03-11T21:12:16.889 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:12:16.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:16.891 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:12:16.992 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:17.016 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:17.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:17.018 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:12:17.118 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:12:17.153 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:12:17.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:17.155 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:12:17.256 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:12:17.282 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:12:17.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:17.285 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:12:17.347 INFO:teuthology.orchestra.run.smithi023.stdout:osd.0: {} 2024-03-11T21:12:17.385 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:12:17.414 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:12:17.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:17.416 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:12:17.416 INFO:teuthology.orchestra.run.smithi023.stdout:osd.1: {} 2024-03-11T21:12:17.449 INFO:teuthology.orchestra.run.smithi023.stdout:osd.2: {} 2024-03-11T21:12:17.495 INFO:teuthology.orchestra.run.smithi023.stdout:osd.3: {} 2024-03-11T21:12:17.517 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:12:17.536 INFO:teuthology.orchestra.run.smithi023.stdout:osd.4: {} 2024-03-11T21:12:17.544 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:12:17.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:17.547 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:12:17.578 INFO:teuthology.orchestra.run.smithi023.stdout:osd.5: {} 2024-03-11T21:12:17.625 INFO:teuthology.orchestra.run.smithi023.stdout:osd.6: {} 2024-03-11T21:12:17.648 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:12:17.670 INFO:teuthology.orchestra.run.smithi023.stdout:osd.7: {} 2024-03-11T21:12:17.671 INFO:teuthology.orchestra.run.smithi023.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:12:17.671 INFO:teuthology.orchestra.run.smithi023.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:12:17.672 INFO:teuthology.orchestra.run.smithi023.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:12:17.672 INFO:teuthology.orchestra.run.smithi023.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:12:17.672 INFO:teuthology.orchestra.run.smithi023.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:12:17.672 INFO:teuthology.orchestra.run.smithi023.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:12:17.672 INFO:teuthology.orchestra.run.smithi023.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:12:17.672 INFO:teuthology.orchestra.run.smithi023.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:12:17.675 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:12:17.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:17.677 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:12:17.778 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:12:17.809 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:12:17.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:17.811 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:12:17.832 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 13 16 25777 25761 123.794 105.688 0.0115407 0.00807076 2024-03-11T21:12:17.912 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:12:17.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:17 smithi146 bash[23809]: audit 2024-03-11T21:12:15.920979+0000 mgr.y (mgr.24380) 795 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:12:17.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:17 smithi146 bash[23809]: cluster 2024-03-11T21:12:16.144651+0000 osd.7 (osd.7) 507 : cluster 0 3.11 scrub starts 2024-03-11T21:12:17.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:17 smithi146 bash[23809]: cluster 2024-03-11T21:12:16.148121+0000 osd.7 (osd.7) 508 : cluster 0 3.11 scrub ok 2024-03-11T21:12:17.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:17 smithi146 bash[23809]: cluster 2024-03-11T21:12:16.368683+0000 mgr.y (mgr.24380) 796 : cluster 0 pgmap v912: 158 pgs: 158 active+clean; 1.2 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 142 MiB/s wr, 2.27k op/s 2024-03-11T21:12:17.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:17 smithi146 bash[23809]: cluster 2024-03-11T21:12:16.892977+0000 osd.3 (osd.3) 448 : cluster 0 3.18 scrub starts 2024-03-11T21:12:17.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:17 smithi146 bash[23809]: cluster 2024-03-11T21:12:16.917495+0000 osd.3 (osd.3) 449 : cluster 0 3.18 scrub ok 2024-03-11T21:12:17.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:17 smithi146 bash[23809]: audit 2024-03-11T21:12:17.298336+0000 mon.c (mon.2) 92 : audit 0 from='client.? 172.21.15.23:0/3915995368' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:12:17.938 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:12:17.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:17.940 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:12:18.027 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:17 smithi023 bash[21372]: audit 2024-03-11T21:12:15.920979+0000 mgr.y (mgr.24380) 795 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:12:18.027 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:17 smithi023 bash[21372]: cluster 2024-03-11T21:12:16.144651+0000 osd.7 (osd.7) 507 : cluster 0 3.11 scrub starts 2024-03-11T21:12:18.027 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:17 smithi023 bash[21372]: cluster 2024-03-11T21:12:16.148121+0000 osd.7 (osd.7) 508 : cluster 0 3.11 scrub ok 2024-03-11T21:12:18.027 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:17 smithi023 bash[21372]: cluster 2024-03-11T21:12:16.368683+0000 mgr.y (mgr.24380) 796 : cluster 0 pgmap v912: 158 pgs: 158 active+clean; 1.2 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 142 MiB/s wr, 2.27k op/s 2024-03-11T21:12:18.027 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:17 smithi023 bash[21372]: cluster 2024-03-11T21:12:16.892977+0000 osd.3 (osd.3) 448 : cluster 0 3.18 scrub starts 2024-03-11T21:12:18.027 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:17 smithi023 bash[21372]: cluster 2024-03-11T21:12:16.917495+0000 osd.3 (osd.3) 449 : cluster 0 3.18 scrub ok 2024-03-11T21:12:18.027 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:17 smithi023 bash[21372]: audit 2024-03-11T21:12:17.298336+0000 mon.c (mon.2) 92 : audit 0 from='client.? 172.21.15.23:0/3915995368' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:12:18.028 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:17 smithi023 bash[29146]: audit 2024-03-11T21:12:15.920979+0000 mgr.y (mgr.24380) 795 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:12:18.028 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:17 smithi023 bash[29146]: cluster 2024-03-11T21:12:16.144651+0000 osd.7 (osd.7) 507 : cluster 0 3.11 scrub starts 2024-03-11T21:12:18.028 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:17 smithi023 bash[29146]: cluster 2024-03-11T21:12:16.148121+0000 osd.7 (osd.7) 508 : cluster 0 3.11 scrub ok 2024-03-11T21:12:18.028 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:17 smithi023 bash[29146]: cluster 2024-03-11T21:12:16.368683+0000 mgr.y (mgr.24380) 796 : cluster 0 pgmap v912: 158 pgs: 158 active+clean; 1.2 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 142 MiB/s wr, 2.27k op/s 2024-03-11T21:12:18.028 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:17 smithi023 bash[29146]: cluster 2024-03-11T21:12:16.892977+0000 osd.3 (osd.3) 448 : cluster 0 3.18 scrub starts 2024-03-11T21:12:18.028 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:17 smithi023 bash[29146]: cluster 2024-03-11T21:12:16.917495+0000 osd.3 (osd.3) 449 : cluster 0 3.18 scrub ok 2024-03-11T21:12:18.028 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:17 smithi023 bash[29146]: audit 2024-03-11T21:12:17.298336+0000 mon.c (mon.2) 92 : audit 0 from='client.? 172.21.15.23:0/3915995368' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:12:18.041 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:18.064 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:18.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:18.066 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:12:18.167 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:12:18.197 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:12:18.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:18.201 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:12:18.302 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:12:18.319 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:12:18.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:18.321 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:12:18.422 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:18.445 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:18.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:18.447 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:12:18.527 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:12:18 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:12:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:12:18.548 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:12:18.577 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:12:18.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:18.579 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:12:18.623 INFO:teuthology.orchestra.run.smithi023.stderr:nodeep-scrub is unset 2024-03-11T21:12:18.680 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:12:18.693 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:18 smithi146 bash[23809]: cluster 2024-03-11T21:12:17.082463+0000 osd.4 (osd.4) 113 : cluster 0 purged_snaps scrub starts 2024-03-11T21:12:18.693 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:18 smithi146 bash[23809]: cluster 2024-03-11T21:12:17.082549+0000 osd.4 (osd.4) 114 : cluster 0 purged_snaps scrub ok 2024-03-11T21:12:18.693 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:18 smithi146 bash[23809]: cluster 2024-03-11T21:12:17.587513+0000 mon.a (mon.0) 1819 : cluster 0 osdmap e390: 8 total, 8 up, 8 in 2024-03-11T21:12:18.693 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:18 smithi146 bash[23809]: audit 2024-03-11T21:12:17.741924+0000 mon.c (mon.2) 93 : audit 1 from='client.? 172.21.15.23:0/2769123531' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:12:18.693 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:18 smithi146 bash[23809]: audit 2024-03-11T21:12:17.742838+0000 mon.a (mon.0) 1820 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:12:18.693 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:18 smithi146 bash[23809]: cluster 2024-03-11T21:12:18.406500+0000 osd.5 (osd.5) 465 : cluster 0 2.3 scrub starts 2024-03-11T21:12:18.694 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:18 smithi146 bash[23809]: cluster 2024-03-11T21:12:18.408798+0000 osd.5 (osd.5) 466 : cluster 0 2.3 scrub ok 2024-03-11T21:12:18.713 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:12:18.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:18.717 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:12:18.819 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:18.833 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 14 16 26959 26943 120.229 73.875 0.00293117 0.00831068 2024-03-11T21:12:18.845 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:18.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:18.849 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:12:18.950 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:12:18.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:18.983 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:12:18.983 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:12:19.084 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:19.089 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:18 smithi023 bash[29146]: cluster 2024-03-11T21:12:17.082463+0000 osd.4 (osd.4) 113 : cluster 0 purged_snaps scrub starts 2024-03-11T21:12:19.089 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:18 smithi023 bash[29146]: cluster 2024-03-11T21:12:17.082549+0000 osd.4 (osd.4) 114 : cluster 0 purged_snaps scrub ok 2024-03-11T21:12:19.089 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:18 smithi023 bash[29146]: cluster 2024-03-11T21:12:17.587513+0000 mon.a (mon.0) 1819 : cluster 0 osdmap e390: 8 total, 8 up, 8 in 2024-03-11T21:12:19.089 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:18 smithi023 bash[29146]: audit 2024-03-11T21:12:17.741924+0000 mon.c (mon.2) 93 : audit 1 from='client.? 172.21.15.23:0/2769123531' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:12:19.090 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:18 smithi023 bash[29146]: audit 2024-03-11T21:12:17.742838+0000 mon.a (mon.0) 1820 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:12:19.090 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:18 smithi023 bash[29146]: cluster 2024-03-11T21:12:18.406500+0000 osd.5 (osd.5) 465 : cluster 0 2.3 scrub starts 2024-03-11T21:12:19.090 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:18 smithi023 bash[29146]: cluster 2024-03-11T21:12:18.408798+0000 osd.5 (osd.5) 466 : cluster 0 2.3 scrub ok 2024-03-11T21:12:19.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:18 smithi023 bash[21372]: cluster 2024-03-11T21:12:17.082463+0000 osd.4 (osd.4) 113 : cluster 0 purged_snaps scrub starts 2024-03-11T21:12:19.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:18 smithi023 bash[21372]: cluster 2024-03-11T21:12:17.082549+0000 osd.4 (osd.4) 114 : cluster 0 purged_snaps scrub ok 2024-03-11T21:12:19.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:18 smithi023 bash[21372]: cluster 2024-03-11T21:12:17.587513+0000 mon.a (mon.0) 1819 : cluster 0 osdmap e390: 8 total, 8 up, 8 in 2024-03-11T21:12:19.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:18 smithi023 bash[21372]: audit 2024-03-11T21:12:17.741924+0000 mon.c (mon.2) 93 : audit 1 from='client.? 172.21.15.23:0/2769123531' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:12:19.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:18 smithi023 bash[21372]: audit 2024-03-11T21:12:17.742838+0000 mon.a (mon.0) 1820 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:12:19.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:18 smithi023 bash[21372]: cluster 2024-03-11T21:12:18.406500+0000 osd.5 (osd.5) 465 : cluster 0 2.3 scrub starts 2024-03-11T21:12:19.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:18 smithi023 bash[21372]: cluster 2024-03-11T21:12:18.408798+0000 osd.5 (osd.5) 466 : cluster 0 2.3 scrub ok 2024-03-11T21:12:19.114 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:19.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:19.116 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:12:19.217 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:12:19.258 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:12:19.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:19.262 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:12:19.363 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:19.390 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:19.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:19.392 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:12:19.493 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:12:19.513 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:12:19.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:19.515 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:12:19.616 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:19.626 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:19 smithi146 bash[23809]: cluster 2024-03-11T21:12:18.369067+0000 mgr.y (mgr.24380) 797 : cluster 0 pgmap v914: 158 pgs: 158 active+clean; 1.3 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 638 B/s rd, 90 MiB/s wr, 1.44k op/s 2024-03-11T21:12:19.627 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:19 smithi146 bash[23809]: cluster 2024-03-11T21:12:18.593634+0000 mon.a (mon.0) 1821 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:12:19.627 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:19 smithi146 bash[23809]: cluster 2024-03-11T21:12:18.593661+0000 mon.a (mon.0) 1822 : cluster 1 Cluster is now healthy 2024-03-11T21:12:19.627 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:19 smithi146 bash[23809]: audit 2024-03-11T21:12:18.613179+0000 mon.a (mon.0) 1823 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:12:19.627 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:19 smithi146 bash[23809]: cluster 2024-03-11T21:12:18.631183+0000 mon.a (mon.0) 1824 : cluster 0 osdmap e391: 8 total, 8 up, 8 in 2024-03-11T21:12:19.627 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:19 smithi146 bash[23809]: cluster 2024-03-11T21:12:18.750002+0000 osd.1 (osd.1) 277 : cluster 0 6.e scrub starts 2024-03-11T21:12:19.627 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:19 smithi146 bash[23809]: cluster 2024-03-11T21:12:18.813931+0000 osd.1 (osd.1) 278 : cluster 0 6.e scrub ok 2024-03-11T21:12:19.639 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:19.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:19.641 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:12:19.742 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:12:19.768 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:12:19.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:19.770 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:12:19.833 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 15 16 28070 28054 116.843 69.4375 0.00398092 0.00836636 2024-03-11T21:12:19.871 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:12:19.901 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:12:19.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:19.904 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:12:20.005 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:20.041 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:20.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:20.044 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:12:20.145 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:12:20.173 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:12:20.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:20.177 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:12:20.278 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:12:20.292 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:19 smithi023 bash[21372]: cluster 2024-03-11T21:12:18.369067+0000 mgr.y (mgr.24380) 797 : cluster 0 pgmap v914: 158 pgs: 158 active+clean; 1.3 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 638 B/s rd, 90 MiB/s wr, 1.44k op/s 2024-03-11T21:12:20.292 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:19 smithi023 bash[21372]: cluster 2024-03-11T21:12:18.593634+0000 mon.a (mon.0) 1821 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:12:20.292 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:19 smithi023 bash[21372]: cluster 2024-03-11T21:12:18.593661+0000 mon.a (mon.0) 1822 : cluster 1 Cluster is now healthy 2024-03-11T21:12:20.292 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:19 smithi023 bash[21372]: audit 2024-03-11T21:12:18.613179+0000 mon.a (mon.0) 1823 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:12:20.292 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:19 smithi023 bash[21372]: cluster 2024-03-11T21:12:18.631183+0000 mon.a (mon.0) 1824 : cluster 0 osdmap e391: 8 total, 8 up, 8 in 2024-03-11T21:12:20.292 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:19 smithi023 bash[21372]: cluster 2024-03-11T21:12:18.750002+0000 osd.1 (osd.1) 277 : cluster 0 6.e scrub starts 2024-03-11T21:12:20.292 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:19 smithi023 bash[21372]: cluster 2024-03-11T21:12:18.813931+0000 osd.1 (osd.1) 278 : cluster 0 6.e scrub ok 2024-03-11T21:12:20.293 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:20 smithi023 bash[29146]: cluster 2024-03-11T21:12:18.369067+0000 mgr.y (mgr.24380) 797 : cluster 0 pgmap v914: 158 pgs: 158 active+clean; 1.3 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 638 B/s rd, 90 MiB/s wr, 1.44k op/s 2024-03-11T21:12:20.293 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:20 smithi023 bash[29146]: cluster 2024-03-11T21:12:18.593634+0000 mon.a (mon.0) 1821 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:12:20.293 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:20 smithi023 bash[29146]: cluster 2024-03-11T21:12:18.593661+0000 mon.a (mon.0) 1822 : cluster 1 Cluster is now healthy 2024-03-11T21:12:20.293 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:20 smithi023 bash[29146]: audit 2024-03-11T21:12:18.613179+0000 mon.a (mon.0) 1823 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:12:20.293 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:20 smithi023 bash[29146]: cluster 2024-03-11T21:12:18.631183+0000 mon.a (mon.0) 1824 : cluster 0 osdmap e391: 8 total, 8 up, 8 in 2024-03-11T21:12:20.293 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:20 smithi023 bash[29146]: cluster 2024-03-11T21:12:18.750002+0000 osd.1 (osd.1) 277 : cluster 0 6.e scrub starts 2024-03-11T21:12:20.293 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:20 smithi023 bash[29146]: cluster 2024-03-11T21:12:18.813931+0000 osd.1 (osd.1) 278 : cluster 0 6.e scrub ok 2024-03-11T21:12:20.302 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:12:20.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:20.306 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:12:20.407 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:12:20.438 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:12:20.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:20.441 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:12:20.542 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:20.566 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:20.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:20.569 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:12:20.670 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:12:20.695 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:12:20.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:20.697 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:12:20.798 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:12:20.830 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:12:20.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:20.833 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:12:20.833 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 16 15 29586 29571 115.466 94.8125 0.0162762 0.00865246 2024-03-11T21:12:20.934 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:12:20.946 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:20 smithi023 bash[21372]: cluster 2024-03-11T21:12:19.016237+0000 osd.0 (osd.0) 366 : cluster 0 6.a scrub starts 2024-03-11T21:12:20.946 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:20 smithi023 bash[21372]: cluster 2024-03-11T21:12:19.065988+0000 osd.0 (osd.0) 367 : cluster 0 6.a scrub ok 2024-03-11T21:12:20.946 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:20 smithi023 bash[21372]: cluster 2024-03-11T21:12:19.081323+0000 osd.7 (osd.7) 509 : cluster 0 6.14 scrub starts 2024-03-11T21:12:20.946 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:20 smithi023 bash[21372]: cluster 2024-03-11T21:12:19.082416+0000 osd.7 (osd.7) 510 : cluster 0 6.14 scrub ok 2024-03-11T21:12:20.946 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:20 smithi023 bash[29146]: cluster 2024-03-11T21:12:19.016237+0000 osd.0 (osd.0) 366 : cluster 0 6.a scrub starts 2024-03-11T21:12:20.947 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:20 smithi023 bash[29146]: cluster 2024-03-11T21:12:19.065988+0000 osd.0 (osd.0) 367 : cluster 0 6.a scrub ok 2024-03-11T21:12:20.947 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:20 smithi023 bash[29146]: cluster 2024-03-11T21:12:19.081323+0000 osd.7 (osd.7) 509 : cluster 0 6.14 scrub starts 2024-03-11T21:12:20.947 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:20 smithi023 bash[29146]: cluster 2024-03-11T21:12:19.082416+0000 osd.7 (osd.7) 510 : cluster 0 6.14 scrub ok 2024-03-11T21:12:20.961 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:12:20.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:20.964 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:12:21.006 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:20 smithi146 bash[23809]: cluster 2024-03-11T21:12:19.016237+0000 osd.0 (osd.0) 366 : cluster 0 6.a scrub starts 2024-03-11T21:12:21.006 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:20 smithi146 bash[23809]: cluster 2024-03-11T21:12:19.065988+0000 osd.0 (osd.0) 367 : cluster 0 6.a scrub ok 2024-03-11T21:12:21.007 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:20 smithi146 bash[23809]: cluster 2024-03-11T21:12:19.081323+0000 osd.7 (osd.7) 509 : cluster 0 6.14 scrub starts 2024-03-11T21:12:21.007 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:20 smithi146 bash[23809]: cluster 2024-03-11T21:12:19.082416+0000 osd.7 (osd.7) 510 : cluster 0 6.14 scrub ok 2024-03-11T21:12:21.064 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:21.091 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:21.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:21.093 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:12:21.194 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:12:21.226 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:12:21.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:21.230 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:12:21.331 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:21.350 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:21.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:21.352 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:12:21.452 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:12:21.487 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:12:21.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:21.490 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:12:21.591 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:12:21.609 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:12:21.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:21.611 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:12:21.712 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:12:21.724 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:21 smithi023 bash[21372]: cluster 2024-03-11T21:12:19.993550+0000 osd.1 (osd.1) 279 : cluster 0 3.4 scrub starts 2024-03-11T21:12:21.724 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:21 smithi023 bash[21372]: cluster 2024-03-11T21:12:19.998705+0000 osd.3 (osd.3) 450 : cluster 0 3.8 scrub starts 2024-03-11T21:12:21.724 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:21 smithi023 bash[21372]: cluster 2024-03-11T21:12:20.000149+0000 osd.1 (osd.1) 280 : cluster 0 3.4 scrub ok 2024-03-11T21:12:21.725 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:21 smithi023 bash[21372]: cluster 2024-03-11T21:12:20.007722+0000 osd.3 (osd.3) 451 : cluster 0 3.8 scrub ok 2024-03-11T21:12:21.725 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:21 smithi023 bash[21372]: cluster 2024-03-11T21:12:20.037318+0000 osd.7 (osd.7) 511 : cluster 0 3.9 scrub starts 2024-03-11T21:12:21.725 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:21 smithi023 bash[21372]: cluster 2024-03-11T21:12:20.041007+0000 osd.7 (osd.7) 512 : cluster 0 3.9 scrub ok 2024-03-11T21:12:21.725 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:21 smithi023 bash[21372]: cluster 2024-03-11T21:12:20.370654+0000 mgr.y (mgr.24380) 798 : cluster 0 pgmap v916: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 1.6 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 105 MiB/s wr, 1.68k op/s 2024-03-11T21:12:21.725 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:21 smithi023 bash[21372]: audit 2024-03-11T21:12:21.181124+0000 mon.a (mon.0) 1825 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:12:21.725 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:21 smithi023 bash[21372]: audit 2024-03-11T21:12:21.184738+0000 mon.a (mon.0) 1826 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:12:21.725 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:21 smithi023 bash[21372]: cluster 2024-03-11T21:12:21.396954+0000 osd.5 (osd.5) 467 : cluster 0 3.1d scrub starts 2024-03-11T21:12:21.725 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:21 smithi023 bash[21372]: cluster 2024-03-11T21:12:21.398522+0000 osd.5 (osd.5) 468 : cluster 0 3.1d scrub ok 2024-03-11T21:12:21.725 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:21 smithi023 bash[29146]: cluster 2024-03-11T21:12:19.993550+0000 osd.1 (osd.1) 279 : cluster 0 3.4 scrub starts 2024-03-11T21:12:21.725 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:21 smithi023 bash[29146]: cluster 2024-03-11T21:12:19.998705+0000 osd.3 (osd.3) 450 : cluster 0 3.8 scrub starts 2024-03-11T21:12:21.725 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:21 smithi023 bash[29146]: cluster 2024-03-11T21:12:20.000149+0000 osd.1 (osd.1) 280 : cluster 0 3.4 scrub ok 2024-03-11T21:12:21.726 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:21 smithi023 bash[29146]: cluster 2024-03-11T21:12:20.007722+0000 osd.3 (osd.3) 451 : cluster 0 3.8 scrub ok 2024-03-11T21:12:21.726 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:21 smithi023 bash[29146]: cluster 2024-03-11T21:12:20.037318+0000 osd.7 (osd.7) 511 : cluster 0 3.9 scrub starts 2024-03-11T21:12:21.726 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:21 smithi023 bash[29146]: cluster 2024-03-11T21:12:20.041007+0000 osd.7 (osd.7) 512 : cluster 0 3.9 scrub ok 2024-03-11T21:12:21.726 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:21 smithi023 bash[29146]: cluster 2024-03-11T21:12:20.370654+0000 mgr.y (mgr.24380) 798 : cluster 0 pgmap v916: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 1.6 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 105 MiB/s wr, 1.68k op/s 2024-03-11T21:12:21.726 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:21 smithi023 bash[29146]: audit 2024-03-11T21:12:21.181124+0000 mon.a (mon.0) 1825 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:12:21.726 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:21 smithi023 bash[29146]: audit 2024-03-11T21:12:21.184738+0000 mon.a (mon.0) 1826 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:12:21.726 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:21 smithi023 bash[29146]: cluster 2024-03-11T21:12:21.396954+0000 osd.5 (osd.5) 467 : cluster 0 3.1d scrub starts 2024-03-11T21:12:21.726 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:21 smithi023 bash[29146]: cluster 2024-03-11T21:12:21.398522+0000 osd.5 (osd.5) 468 : cluster 0 3.1d scrub ok 2024-03-11T21:12:21.746 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:12:21.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:21.748 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:12:21.833 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 17 16 30731 30715 112.88 71.5 0.00245502 0.00877723 2024-03-11T21:12:21.849 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:12:21.887 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:12:21.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:21.891 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:12:21.992 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:12:22.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:21 smithi146 bash[23809]: cluster 2024-03-11T21:12:19.993550+0000 osd.1 (osd.1) 279 : cluster 0 3.4 scrub starts 2024-03-11T21:12:22.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:21 smithi146 bash[23809]: cluster 2024-03-11T21:12:19.998705+0000 osd.3 (osd.3) 450 : cluster 0 3.8 scrub starts 2024-03-11T21:12:22.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:21 smithi146 bash[23809]: cluster 2024-03-11T21:12:20.000149+0000 osd.1 (osd.1) 280 : cluster 0 3.4 scrub ok 2024-03-11T21:12:22.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:21 smithi146 bash[23809]: cluster 2024-03-11T21:12:20.007722+0000 osd.3 (osd.3) 451 : cluster 0 3.8 scrub ok 2024-03-11T21:12:22.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:21 smithi146 bash[23809]: cluster 2024-03-11T21:12:20.037318+0000 osd.7 (osd.7) 511 : cluster 0 3.9 scrub starts 2024-03-11T21:12:22.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:21 smithi146 bash[23809]: cluster 2024-03-11T21:12:20.041007+0000 osd.7 (osd.7) 512 : cluster 0 3.9 scrub ok 2024-03-11T21:12:22.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:21 smithi146 bash[23809]: cluster 2024-03-11T21:12:20.370654+0000 mgr.y (mgr.24380) 798 : cluster 0 pgmap v916: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 1.6 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 105 MiB/s wr, 1.68k op/s 2024-03-11T21:12:22.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:21 smithi146 bash[23809]: audit 2024-03-11T21:12:21.181124+0000 mon.a (mon.0) 1825 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:12:22.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:21 smithi146 bash[23809]: audit 2024-03-11T21:12:21.184738+0000 mon.a (mon.0) 1826 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:12:22.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:21 smithi146 bash[23809]: cluster 2024-03-11T21:12:21.396954+0000 osd.5 (osd.5) 467 : cluster 0 3.1d scrub starts 2024-03-11T21:12:22.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:21 smithi146 bash[23809]: cluster 2024-03-11T21:12:21.398522+0000 osd.5 (osd.5) 468 : cluster 0 3.1d scrub ok 2024-03-11T21:12:22.020 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:12:22.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:22.024 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:12:22.125 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:12:22.158 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:12:22.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:22.162 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:12:22.263 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:12:22.289 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:12:22.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:22.291 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:12:22.392 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:12:22.424 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:12:22.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:22.427 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:12:22.528 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:12:22.556 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:12:22.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:22.560 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:12:22.661 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:12:22.681 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:12:22.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:22.683 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:12:22.785 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:22.796 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:22 smithi146 bash[23809]: cluster 2024-03-11T21:12:21.960743+0000 osd.6 (osd.6) 388 : cluster 0 5.2 scrub starts 2024-03-11T21:12:22.796 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:22 smithi146 bash[23809]: cluster 2024-03-11T21:12:21.997545+0000 osd.6 (osd.6) 389 : cluster 0 5.2 scrub ok 2024-03-11T21:12:22.814 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:22.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:22.816 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:12:22.833 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 18 15 32216 32201 111.768 92.875 0.00688446 0.00894137 2024-03-11T21:12:22.917 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:12:22.946 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:12:22.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:22.948 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:12:23.027 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:22 smithi023 bash[21372]: cluster 2024-03-11T21:12:21.960743+0000 osd.6 (osd.6) 388 : cluster 0 5.2 scrub starts 2024-03-11T21:12:23.027 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:22 smithi023 bash[21372]: cluster 2024-03-11T21:12:21.997545+0000 osd.6 (osd.6) 389 : cluster 0 5.2 scrub ok 2024-03-11T21:12:23.027 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:22 smithi023 bash[29146]: cluster 2024-03-11T21:12:21.960743+0000 osd.6 (osd.6) 388 : cluster 0 5.2 scrub starts 2024-03-11T21:12:23.027 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:22 smithi023 bash[29146]: cluster 2024-03-11T21:12:21.997545+0000 osd.6 (osd.6) 389 : cluster 0 5.2 scrub ok 2024-03-11T21:12:23.049 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:12:23.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:23.080 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:12:23.081 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:12:23.181 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:12:23.205 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:12:23.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:23.207 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:12:23.308 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:23.335 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:23.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:23.341 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:12:23.441 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:23.466 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:23.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:23.473 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:12:23.574 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:23.598 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:23.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:23.601 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:12:23.702 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:12:23.703 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T21:12:23.729 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:12:23.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:23.734 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:12:23.833 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 19 16 33793 33777 111.07 98.5 0.01677 0.00899665 2024-03-11T21:12:23.834 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:23.844 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:23 smithi146 bash[23809]: cluster 2024-03-11T21:12:22.371711+0000 mgr.y (mgr.24380) 799 : cluster 0 pgmap v917: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 1.8 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 121 MiB/s wr, 1.94k op/s 2024-03-11T21:12:23.844 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:23 smithi146 bash[23809]: cluster 2024-03-11T21:12:23.355467+0000 osd.5 (osd.5) 469 : cluster 0 6.1a scrub starts 2024-03-11T21:12:23.844 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:23 smithi146 bash[23809]: cluster 2024-03-11T21:12:23.401230+0000 osd.5 (osd.5) 470 : cluster 0 6.1a scrub ok 2024-03-11T21:12:23.859 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:23.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:23.862 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:12:23.963 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:12:23.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:23.991 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:12:23.991 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:12:24.006 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:23 smithi023 bash[21372]: cluster 2024-03-11T21:12:22.371711+0000 mgr.y (mgr.24380) 799 : cluster 0 pgmap v917: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 1.8 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 121 MiB/s wr, 1.94k op/s 2024-03-11T21:12:24.006 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:23 smithi023 bash[21372]: cluster 2024-03-11T21:12:23.355467+0000 osd.5 (osd.5) 469 : cluster 0 6.1a scrub starts 2024-03-11T21:12:24.006 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:23 smithi023 bash[21372]: cluster 2024-03-11T21:12:23.401230+0000 osd.5 (osd.5) 470 : cluster 0 6.1a scrub ok 2024-03-11T21:12:24.007 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:23 smithi023 bash[29146]: cluster 2024-03-11T21:12:22.371711+0000 mgr.y (mgr.24380) 799 : cluster 0 pgmap v917: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 1.8 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 121 MiB/s wr, 1.94k op/s 2024-03-11T21:12:24.007 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:23 smithi023 bash[29146]: cluster 2024-03-11T21:12:23.355467+0000 osd.5 (osd.5) 469 : cluster 0 6.1a scrub starts 2024-03-11T21:12:24.007 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:23 smithi023 bash[29146]: cluster 2024-03-11T21:12:23.401230+0000 osd.5 (osd.5) 470 : cluster 0 6.1a scrub ok 2024-03-11T21:12:24.091 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:12:24.117 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:12:24.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:24.120 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:12:24.221 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:24.248 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:24.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:24.251 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:12:24.351 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:24.375 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:24.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:24.377 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:12:24.478 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:24.504 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:24.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:24.507 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:12:24.608 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:12:24.637 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:12:24.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:24.640 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:12:24.741 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:12:24.756 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:24 smithi146 bash[23809]: cluster 2024-03-11T21:12:23.020396+0000 osd.0 (osd.0) 368 : cluster 0 3.1f scrub starts 2024-03-11T21:12:24.756 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:24 smithi146 bash[23809]: cluster 2024-03-11T21:12:23.022893+0000 osd.0 (osd.0) 369 : cluster 0 3.1f scrub ok 2024-03-11T21:12:24.771 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:12:24.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:24.775 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:12:24.833 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:2024-03-11T21:12:24.845360+0000 min lat: 0.00123771 max lat: 0.48987 avg lat: 0.00904237 2024-03-11T21:12:24.833 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-11T21:12:24.833 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 20 16 35394 35378 110.519 100.062 0.00594456 0.00904237 2024-03-11T21:12:24.876 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:12:24.887 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:24 smithi023 bash[21372]: cluster 2024-03-11T21:12:23.020396+0000 osd.0 (osd.0) 368 : cluster 0 3.1f scrub starts 2024-03-11T21:12:24.887 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:24 smithi023 bash[21372]: cluster 2024-03-11T21:12:23.022893+0000 osd.0 (osd.0) 369 : cluster 0 3.1f scrub ok 2024-03-11T21:12:24.888 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:24 smithi023 bash[29146]: cluster 2024-03-11T21:12:23.020396+0000 osd.0 (osd.0) 368 : cluster 0 3.1f scrub starts 2024-03-11T21:12:24.888 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:24 smithi023 bash[29146]: cluster 2024-03-11T21:12:23.022893+0000 osd.0 (osd.0) 369 : cluster 0 3.1f scrub ok 2024-03-11T21:12:24.901 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:12:24.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:24.904 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:12:25.005 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:12:25.033 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:12:25.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:25.036 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:12:25.137 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:12:25.160 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:12:25.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:25.163 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:12:25.264 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:12:25.293 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:12:25.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:25.295 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:12:25.396 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:25.422 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:25.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:25.427 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:12:25.528 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:12:25.552 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:12:25.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:25.554 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:12:25.655 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:25.665 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:25 smithi146 bash[23809]: cluster 2024-03-11T21:12:23.999151+0000 osd.0 (osd.0) 370 : cluster 0 4.16 scrub starts 2024-03-11T21:12:25.665 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:25 smithi146 bash[23809]: cluster 2024-03-11T21:12:24.000494+0000 osd.0 (osd.0) 371 : cluster 0 4.16 scrub ok 2024-03-11T21:12:25.665 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:25 smithi146 bash[23809]: cluster 2024-03-11T21:12:24.068694+0000 osd.7 (osd.7) 513 : cluster 0 6.12 deep-scrub starts 2024-03-11T21:12:25.665 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:25 smithi146 bash[23809]: cluster 2024-03-11T21:12:24.069895+0000 osd.7 (osd.7) 514 : cluster 0 6.12 deep-scrub ok 2024-03-11T21:12:25.665 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:25 smithi146 bash[23809]: cluster 2024-03-11T21:12:24.372176+0000 mgr.y (mgr.24380) 800 : cluster 0 pgmap v918: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 1.8 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 70 MiB/s wr, 1.12k op/s 2024-03-11T21:12:25.665 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:25 smithi146 bash[23809]: cluster 2024-03-11T21:12:24.978345+0000 osd.3 (osd.3) 452 : cluster 0 6.2 scrub starts 2024-03-11T21:12:25.665 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:25 smithi146 bash[23809]: cluster 2024-03-11T21:12:24.980133+0000 osd.3 (osd.3) 453 : cluster 0 6.2 scrub ok 2024-03-11T21:12:25.677 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:25.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:25.683 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:12:25.784 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:25.805 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:25.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:25.810 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:12:25.833 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 21 16 36887 36871 109.699 93.3125 0.00429579 0.00910871 2024-03-11T21:12:25.867 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-11T21:12:25.867 INFO:teuthology.orchestra.run.smithi023.stdout: "ops": [], 2024-03-11T21:12:25.867 INFO:teuthology.orchestra.run.smithi023.stdout: "complaint_time": 30, 2024-03-11T21:12:25.868 INFO:teuthology.orchestra.run.smithi023.stdout: "num_blocked_ops": 0 2024-03-11T21:12:25.868 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-11T21:12:25.887 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:25 smithi023 bash[21372]: cluster 2024-03-11T21:12:23.999151+0000 osd.0 (osd.0) 370 : cluster 0 4.16 scrub starts 2024-03-11T21:12:25.887 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:25 smithi023 bash[21372]: cluster 2024-03-11T21:12:24.000494+0000 osd.0 (osd.0) 371 : cluster 0 4.16 scrub ok 2024-03-11T21:12:25.887 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:25 smithi023 bash[21372]: cluster 2024-03-11T21:12:24.068694+0000 osd.7 (osd.7) 513 : cluster 0 6.12 deep-scrub starts 2024-03-11T21:12:25.887 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:25 smithi023 bash[21372]: cluster 2024-03-11T21:12:24.069895+0000 osd.7 (osd.7) 514 : cluster 0 6.12 deep-scrub ok 2024-03-11T21:12:25.887 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:25 smithi023 bash[21372]: cluster 2024-03-11T21:12:24.372176+0000 mgr.y (mgr.24380) 800 : cluster 0 pgmap v918: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 1.8 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 70 MiB/s wr, 1.12k op/s 2024-03-11T21:12:25.887 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:25 smithi023 bash[21372]: cluster 2024-03-11T21:12:24.978345+0000 osd.3 (osd.3) 452 : cluster 0 6.2 scrub starts 2024-03-11T21:12:25.887 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:25 smithi023 bash[21372]: cluster 2024-03-11T21:12:24.980133+0000 osd.3 (osd.3) 453 : cluster 0 6.2 scrub ok 2024-03-11T21:12:25.890 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:25 smithi023 bash[29146]: cluster 2024-03-11T21:12:23.999151+0000 osd.0 (osd.0) 370 : cluster 0 4.16 scrub starts 2024-03-11T21:12:25.890 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:25 smithi023 bash[29146]: cluster 2024-03-11T21:12:24.000494+0000 osd.0 (osd.0) 371 : cluster 0 4.16 scrub ok 2024-03-11T21:12:25.890 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:25 smithi023 bash[29146]: cluster 2024-03-11T21:12:24.068694+0000 osd.7 (osd.7) 513 : cluster 0 6.12 deep-scrub starts 2024-03-11T21:12:25.890 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:25 smithi023 bash[29146]: cluster 2024-03-11T21:12:24.069895+0000 osd.7 (osd.7) 514 : cluster 0 6.12 deep-scrub ok 2024-03-11T21:12:25.890 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:25 smithi023 bash[29146]: cluster 2024-03-11T21:12:24.372176+0000 mgr.y (mgr.24380) 800 : cluster 0 pgmap v918: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 1.8 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 70 MiB/s wr, 1.12k op/s 2024-03-11T21:12:25.890 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:25 smithi023 bash[29146]: cluster 2024-03-11T21:12:24.978345+0000 osd.3 (osd.3) 452 : cluster 0 6.2 scrub starts 2024-03-11T21:12:25.890 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:25 smithi023 bash[29146]: cluster 2024-03-11T21:12:24.980133+0000 osd.3 (osd.3) 453 : cluster 0 6.2 scrub ok 2024-03-11T21:12:25.911 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:12:25.937 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:12:25.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:25.941 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:12:26.009 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd set noscrub 2024-03-11T21:12:26.041 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:12:26.061 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:12:26.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:26.063 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:12:26.164 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:12:26.209 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:12:26.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:26.213 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:12:26.236 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-11T21:12:26.236 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-11T21:12:26.236 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 391, 2024-03-11T21:12:26.237 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "67ba00f8-dfe7-11ee-95c8-87774f69a715", 2024-03-11T21:12:26.237 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2024-03-11T20:40:13.664802+0000", 2024-03-11T21:12:26.237 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2024-03-11T21:12:18.593416+0000", 2024-03-11T21:12:26.237 INFO:teuthology.orchestra.run.smithi023.stdout: "last_up_change": "2024-03-11T21:00:47.324817+0000", 2024-03-11T21:12:26.237 INFO:teuthology.orchestra.run.smithi023.stdout: "last_in_change": "2024-03-11T20:57:23.560527+0000", 2024-03-11T21:12:26.237 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-03-11T21:12:26.237 INFO:teuthology.orchestra.run.smithi023.stdout: "flags_num": 5799936, 2024-03-11T21:12:26.237 INFO:teuthology.orchestra.run.smithi023.stdout: "flags_set": [ 2024-03-11T21:12:26.237 INFO:teuthology.orchestra.run.smithi023.stdout: "pglog_hardlimit", 2024-03-11T21:12:26.237 INFO:teuthology.orchestra.run.smithi023.stdout: "purged_snapdirs", 2024-03-11T21:12:26.237 INFO:teuthology.orchestra.run.smithi023.stdout: "recovery_deletes", 2024-03-11T21:12:26.237 INFO:teuthology.orchestra.run.smithi023.stdout: "sortbitwise" 2024-03-11T21:12:26.237 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-11T21:12:26.237 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_version": 20, 2024-03-11T21:12:26.238 INFO:teuthology.orchestra.run.smithi023.stdout: "full_ratio": 0.94999998807907104, 2024-03-11T21:12:26.238 INFO:teuthology.orchestra.run.smithi023.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-03-11T21:12:26.238 INFO:teuthology.orchestra.run.smithi023.stdout: "nearfull_ratio": 0.85000002384185791, 2024-03-11T21:12:26.238 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_snapshot": "", 2024-03-11T21:12:26.238 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_max": 11, 2024-03-11T21:12:26.238 INFO:teuthology.orchestra.run.smithi023.stdout: "max_osd": 8, 2024-03-11T21:12:26.238 INFO:teuthology.orchestra.run.smithi023.stdout: "require_min_compat_client": "luminous", 2024-03-11T21:12:26.238 INFO:teuthology.orchestra.run.smithi023.stdout: "min_compat_client": "luminous", 2024-03-11T21:12:26.238 INFO:teuthology.orchestra.run.smithi023.stdout: "require_osd_release": "squid", 2024-03-11T21:12:26.238 INFO:teuthology.orchestra.run.smithi023.stdout: "allow_crimson": false, 2024-03-11T21:12:26.238 INFO:teuthology.orchestra.run.smithi023.stdout: "pools": [ 2024-03-11T21:12:26.238 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:26.238 INFO:teuthology.orchestra.run.smithi023.stdout: "pool": 1, 2024-03-11T21:12:26.238 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_name": ".mgr", 2024-03-11T21:12:26.239 INFO:teuthology.orchestra.run.smithi023.stdout: "create_time": "2024-03-11T20:45:31.964570+0000", 2024-03-11T21:12:26.239 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-11T21:12:26.239 INFO:teuthology.orchestra.run.smithi023.stdout: "flags_names": "hashpspool", 2024-03-11T21:12:26.239 INFO:teuthology.orchestra.run.smithi023.stdout: "type": 1, 2024-03-11T21:12:26.239 INFO:teuthology.orchestra.run.smithi023.stdout: "size": 3, 2024-03-11T21:12:26.239 INFO:teuthology.orchestra.run.smithi023.stdout: "min_size": 2, 2024-03-11T21:12:26.239 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_rule": 0, 2024-03-11T21:12:26.239 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_count": 0, 2024-03-11T21:12:26.239 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_target": 0, 2024-03-11T21:12:26.239 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T21:12:26.239 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T21:12:26.239 INFO:teuthology.orchestra.run.smithi023.stdout: "object_hash": 2, 2024-03-11T21:12:26.239 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_autoscale_mode": "off", 2024-03-11T21:12:26.239 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num": 11, 2024-03-11T21:12:26.240 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_placement_num": 11, 2024-03-11T21:12:26.240 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_placement_num_target": 11, 2024-03-11T21:12:26.240 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num_target": 11, 2024-03-11T21:12:26.240 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num_pending": 11, 2024-03-11T21:12:26.240 INFO:teuthology.orchestra.run.smithi023.stdout: "last_pg_merge_meta": { 2024-03-11T21:12:26.240 INFO:teuthology.orchestra.run.smithi023.stdout: "source_pgid": "0.0", 2024-03-11T21:12:26.240 INFO:teuthology.orchestra.run.smithi023.stdout: "ready_epoch": 0, 2024-03-11T21:12:26.240 INFO:teuthology.orchestra.run.smithi023.stdout: "last_epoch_started": 0, 2024-03-11T21:12:26.240 INFO:teuthology.orchestra.run.smithi023.stdout: "last_epoch_clean": 0, 2024-03-11T21:12:26.240 INFO:teuthology.orchestra.run.smithi023.stdout: "source_version": "0'0", 2024-03-11T21:12:26.240 INFO:teuthology.orchestra.run.smithi023.stdout: "target_version": "0'0" 2024-03-11T21:12:26.240 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:26.240 INFO:teuthology.orchestra.run.smithi023.stdout: "last_change": "369", 2024-03-11T21:12:26.240 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend": "0", 2024-03-11T21:12:26.241 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T21:12:26.241 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend_preluminous": "367", 2024-03-11T21:12:26.241 INFO:teuthology.orchestra.run.smithi023.stdout: "auid": 0, 2024-03-11T21:12:26.241 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_mode": "selfmanaged", 2024-03-11T21:12:26.241 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_seq": 0, 2024-03-11T21:12:26.241 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_epoch": 0, 2024-03-11T21:12:26.241 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_snaps": [], 2024-03-11T21:12:26.241 INFO:teuthology.orchestra.run.smithi023.stdout: "removed_snaps": "[]", 2024-03-11T21:12:26.241 INFO:teuthology.orchestra.run.smithi023.stdout: "quota_max_bytes": 0, 2024-03-11T21:12:26.241 INFO:teuthology.orchestra.run.smithi023.stdout: "quota_max_objects": 0, 2024-03-11T21:12:26.241 INFO:teuthology.orchestra.run.smithi023.stdout: "tiers": [], 2024-03-11T21:12:26.241 INFO:teuthology.orchestra.run.smithi023.stdout: "tier_of": -1, 2024-03-11T21:12:26.241 INFO:teuthology.orchestra.run.smithi023.stdout: "read_tier": -1, 2024-03-11T21:12:26.241 INFO:teuthology.orchestra.run.smithi023.stdout: "write_tier": -1, 2024-03-11T21:12:26.242 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_mode": "none", 2024-03-11T21:12:26.242 INFO:teuthology.orchestra.run.smithi023.stdout: "target_max_bytes": 0, 2024-03-11T21:12:26.242 INFO:teuthology.orchestra.run.smithi023.stdout: "target_max_objects": 0, 2024-03-11T21:12:26.242 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T21:12:26.242 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T21:12:26.242 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T21:12:26.242 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_min_flush_age": 0, 2024-03-11T21:12:26.242 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_min_evict_age": 0, 2024-03-11T21:12:26.242 INFO:teuthology.orchestra.run.smithi023.stdout: "erasure_code_profile": "", 2024-03-11T21:12:26.242 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_params": { 2024-03-11T21:12:26.242 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "none" 2024-03-11T21:12:26.242 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:26.242 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_period": 0, 2024-03-11T21:12:26.242 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_count": 0, 2024-03-11T21:12:26.243 INFO:teuthology.orchestra.run.smithi023.stdout: "use_gmt_hitset": true, 2024-03-11T21:12:26.243 INFO:teuthology.orchestra.run.smithi023.stdout: "min_read_recency_for_promote": 0, 2024-03-11T21:12:26.243 INFO:teuthology.orchestra.run.smithi023.stdout: "min_write_recency_for_promote": 0, 2024-03-11T21:12:26.243 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T21:12:26.243 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_search_last_n": 0, 2024-03-11T21:12:26.243 INFO:teuthology.orchestra.run.smithi023.stdout: "grade_table": [], 2024-03-11T21:12:26.243 INFO:teuthology.orchestra.run.smithi023.stdout: "stripe_width": 0, 2024-03-11T21:12:26.243 INFO:teuthology.orchestra.run.smithi023.stdout: "expected_num_objects": 0, 2024-03-11T21:12:26.243 INFO:teuthology.orchestra.run.smithi023.stdout: "fast_read": false, 2024-03-11T21:12:26.243 INFO:teuthology.orchestra.run.smithi023.stdout: "options": { 2024-03-11T21:12:26.243 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num_max": 32, 2024-03-11T21:12:26.243 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num_min": 1 2024-03-11T21:12:26.243 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:26.243 INFO:teuthology.orchestra.run.smithi023.stdout: "application_metadata": { 2024-03-11T21:12:26.244 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr": {} 2024-03-11T21:12:26.244 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:26.244 INFO:teuthology.orchestra.run.smithi023.stdout: "read_balance": { 2024-03-11T21:12:26.244 INFO:teuthology.orchestra.run.smithi023.stdout: "score_type": "Fair distribution", 2024-03-11T21:12:26.244 INFO:teuthology.orchestra.run.smithi023.stdout: "score_acting": 1.6399999856948853, 2024-03-11T21:12:26.244 INFO:teuthology.orchestra.run.smithi023.stdout: "score_stable": 1.6399999856948853, 2024-03-11T21:12:26.244 INFO:teuthology.orchestra.run.smithi023.stdout: "optimal_score": 1.3300000429153442, 2024-03-11T21:12:26.244 INFO:teuthology.orchestra.run.smithi023.stdout: "raw_score_acting": 2.1800000667572021, 2024-03-11T21:12:26.244 INFO:teuthology.orchestra.run.smithi023.stdout: "raw_score_stable": 2.1800000667572021, 2024-03-11T21:12:26.244 INFO:teuthology.orchestra.run.smithi023.stdout: "primary_affinity_weighted": 0.75, 2024-03-11T21:12:26.244 INFO:teuthology.orchestra.run.smithi023.stdout: "average_primary_affinity": 1, 2024-03-11T21:12:26.244 INFO:teuthology.orchestra.run.smithi023.stdout: "average_primary_affinity_weighted": 0.75 2024-03-11T21:12:26.244 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:12:26.244 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:26.244 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:26.245 INFO:teuthology.orchestra.run.smithi023.stdout: "pool": 2, 2024-03-11T21:12:26.245 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_name": ".rgw.root", 2024-03-11T21:12:26.245 INFO:teuthology.orchestra.run.smithi023.stdout: "create_time": "2024-03-11T20:50:26.421486+0000", 2024-03-11T21:12:26.245 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-11T21:12:26.245 INFO:teuthology.orchestra.run.smithi023.stdout: "flags_names": "hashpspool", 2024-03-11T21:12:26.245 INFO:teuthology.orchestra.run.smithi023.stdout: "type": 1, 2024-03-11T21:12:26.245 INFO:teuthology.orchestra.run.smithi023.stdout: "size": 3, 2024-03-11T21:12:26.245 INFO:teuthology.orchestra.run.smithi023.stdout: "min_size": 2, 2024-03-11T21:12:26.245 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_rule": 0, 2024-03-11T21:12:26.245 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_count": 0, 2024-03-11T21:12:26.245 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_target": 0, 2024-03-11T21:12:26.245 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T21:12:26.245 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T21:12:26.245 INFO:teuthology.orchestra.run.smithi023.stdout: "object_hash": 2, 2024-03-11T21:12:26.245 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_autoscale_mode": "off", 2024-03-11T21:12:26.246 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num": 32, 2024-03-11T21:12:26.246 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_placement_num": 32, 2024-03-11T21:12:26.246 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_placement_num_target": 32, 2024-03-11T21:12:26.246 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num_target": 32, 2024-03-11T21:12:26.246 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num_pending": 32, 2024-03-11T21:12:26.246 INFO:teuthology.orchestra.run.smithi023.stdout: "last_pg_merge_meta": { 2024-03-11T21:12:26.246 INFO:teuthology.orchestra.run.smithi023.stdout: "source_pgid": "0.0", 2024-03-11T21:12:26.246 INFO:teuthology.orchestra.run.smithi023.stdout: "ready_epoch": 0, 2024-03-11T21:12:26.246 INFO:teuthology.orchestra.run.smithi023.stdout: "last_epoch_started": 0, 2024-03-11T21:12:26.246 INFO:teuthology.orchestra.run.smithi023.stdout: "last_epoch_clean": 0, 2024-03-11T21:12:26.246 INFO:teuthology.orchestra.run.smithi023.stdout: "source_version": "0'0", 2024-03-11T21:12:26.246 INFO:teuthology.orchestra.run.smithi023.stdout: "target_version": "0'0" 2024-03-11T21:12:26.246 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:26.246 INFO:teuthology.orchestra.run.smithi023.stdout: "last_change": "78", 2024-03-11T21:12:26.247 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend": "0", 2024-03-11T21:12:26.247 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T21:12:26.247 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T21:12:26.247 INFO:teuthology.orchestra.run.smithi023.stdout: "auid": 0, 2024-03-11T21:12:26.247 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_mode": "selfmanaged", 2024-03-11T21:12:26.247 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_seq": 0, 2024-03-11T21:12:26.247 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_epoch": 0, 2024-03-11T21:12:26.247 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_snaps": [], 2024-03-11T21:12:26.247 INFO:teuthology.orchestra.run.smithi023.stdout: "removed_snaps": "[]", 2024-03-11T21:12:26.247 INFO:teuthology.orchestra.run.smithi023.stdout: "quota_max_bytes": 0, 2024-03-11T21:12:26.247 INFO:teuthology.orchestra.run.smithi023.stdout: "quota_max_objects": 0, 2024-03-11T21:12:26.247 INFO:teuthology.orchestra.run.smithi023.stdout: "tiers": [], 2024-03-11T21:12:26.247 INFO:teuthology.orchestra.run.smithi023.stdout: "tier_of": -1, 2024-03-11T21:12:26.247 INFO:teuthology.orchestra.run.smithi023.stdout: "read_tier": -1, 2024-03-11T21:12:26.247 INFO:teuthology.orchestra.run.smithi023.stdout: "write_tier": -1, 2024-03-11T21:12:26.248 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_mode": "none", 2024-03-11T21:12:26.248 INFO:teuthology.orchestra.run.smithi023.stdout: "target_max_bytes": 0, 2024-03-11T21:12:26.248 INFO:teuthology.orchestra.run.smithi023.stdout: "target_max_objects": 0, 2024-03-11T21:12:26.248 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T21:12:26.248 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T21:12:26.248 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T21:12:26.248 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_min_flush_age": 0, 2024-03-11T21:12:26.248 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_min_evict_age": 0, 2024-03-11T21:12:26.248 INFO:teuthology.orchestra.run.smithi023.stdout: "erasure_code_profile": "", 2024-03-11T21:12:26.248 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_params": { 2024-03-11T21:12:26.248 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "none" 2024-03-11T21:12:26.248 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:26.248 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_period": 0, 2024-03-11T21:12:26.248 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_count": 0, 2024-03-11T21:12:26.248 INFO:teuthology.orchestra.run.smithi023.stdout: "use_gmt_hitset": true, 2024-03-11T21:12:26.248 INFO:teuthology.orchestra.run.smithi023.stdout: "min_read_recency_for_promote": 0, 2024-03-11T21:12:26.249 INFO:teuthology.orchestra.run.smithi023.stdout: "min_write_recency_for_promote": 0, 2024-03-11T21:12:26.249 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T21:12:26.249 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_search_last_n": 0, 2024-03-11T21:12:26.249 INFO:teuthology.orchestra.run.smithi023.stdout: "grade_table": [], 2024-03-11T21:12:26.249 INFO:teuthology.orchestra.run.smithi023.stdout: "stripe_width": 0, 2024-03-11T21:12:26.249 INFO:teuthology.orchestra.run.smithi023.stdout: "expected_num_objects": 0, 2024-03-11T21:12:26.250 INFO:teuthology.orchestra.run.smithi023.stdout: "fast_read": false, 2024-03-11T21:12:26.250 INFO:teuthology.orchestra.run.smithi023.stdout: "options": {}, 2024-03-11T21:12:26.250 INFO:teuthology.orchestra.run.smithi023.stdout: "application_metadata": { 2024-03-11T21:12:26.250 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw": {} 2024-03-11T21:12:26.250 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:26.250 INFO:teuthology.orchestra.run.smithi023.stdout: "read_balance": { 2024-03-11T21:12:26.250 INFO:teuthology.orchestra.run.smithi023.stdout: "score_type": "Fair distribution", 2024-03-11T21:12:26.250 INFO:teuthology.orchestra.run.smithi023.stdout: "score_acting": 1.5, 2024-03-11T21:12:26.250 INFO:teuthology.orchestra.run.smithi023.stdout: "score_stable": 1.5, 2024-03-11T21:12:26.250 INFO:teuthology.orchestra.run.smithi023.stdout: "optimal_score": 1.3300000429153442, 2024-03-11T21:12:26.250 INFO:teuthology.orchestra.run.smithi023.stdout: "raw_score_acting": 2, 2024-03-11T21:12:26.250 INFO:teuthology.orchestra.run.smithi023.stdout: "raw_score_stable": 2, 2024-03-11T21:12:26.250 INFO:teuthology.orchestra.run.smithi023.stdout: "primary_affinity_weighted": 0.75, 2024-03-11T21:12:26.250 INFO:teuthology.orchestra.run.smithi023.stdout: "average_primary_affinity": 1, 2024-03-11T21:12:26.251 INFO:teuthology.orchestra.run.smithi023.stdout: "average_primary_affinity_weighted": 0.75 2024-03-11T21:12:26.251 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:12:26.251 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:26.251 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:26.251 INFO:teuthology.orchestra.run.smithi023.stdout: "pool": 3, 2024-03-11T21:12:26.251 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_name": "default.rgw.log", 2024-03-11T21:12:26.251 INFO:teuthology.orchestra.run.smithi023.stdout: "create_time": "2024-03-11T20:50:28.592615+0000", 2024-03-11T21:12:26.251 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-11T21:12:26.251 INFO:teuthology.orchestra.run.smithi023.stdout: "flags_names": "hashpspool", 2024-03-11T21:12:26.251 INFO:teuthology.orchestra.run.smithi023.stdout: "type": 1, 2024-03-11T21:12:26.251 INFO:teuthology.orchestra.run.smithi023.stdout: "size": 3, 2024-03-11T21:12:26.251 INFO:teuthology.orchestra.run.smithi023.stdout: "min_size": 2, 2024-03-11T21:12:26.251 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_rule": 0, 2024-03-11T21:12:26.251 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_count": 0, 2024-03-11T21:12:26.252 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_target": 0, 2024-03-11T21:12:26.252 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T21:12:26.252 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T21:12:26.252 INFO:teuthology.orchestra.run.smithi023.stdout: "object_hash": 2, 2024-03-11T21:12:26.252 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_autoscale_mode": "off", 2024-03-11T21:12:26.252 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num": 32, 2024-03-11T21:12:26.252 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_placement_num": 32, 2024-03-11T21:12:26.252 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_placement_num_target": 32, 2024-03-11T21:12:26.252 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num_target": 32, 2024-03-11T21:12:26.252 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num_pending": 32, 2024-03-11T21:12:26.252 INFO:teuthology.orchestra.run.smithi023.stdout: "last_pg_merge_meta": { 2024-03-11T21:12:26.252 INFO:teuthology.orchestra.run.smithi023.stdout: "source_pgid": "0.0", 2024-03-11T21:12:26.252 INFO:teuthology.orchestra.run.smithi023.stdout: "ready_epoch": 0, 2024-03-11T21:12:26.252 INFO:teuthology.orchestra.run.smithi023.stdout: "last_epoch_started": 0, 2024-03-11T21:12:26.253 INFO:teuthology.orchestra.run.smithi023.stdout: "last_epoch_clean": 0, 2024-03-11T21:12:26.253 INFO:teuthology.orchestra.run.smithi023.stdout: "source_version": "0'0", 2024-03-11T21:12:26.253 INFO:teuthology.orchestra.run.smithi023.stdout: "target_version": "0'0" 2024-03-11T21:12:26.253 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:26.253 INFO:teuthology.orchestra.run.smithi023.stdout: "last_change": "80", 2024-03-11T21:12:26.253 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend": "0", 2024-03-11T21:12:26.253 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T21:12:26.253 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T21:12:26.253 INFO:teuthology.orchestra.run.smithi023.stdout: "auid": 0, 2024-03-11T21:12:26.253 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_mode": "selfmanaged", 2024-03-11T21:12:26.253 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_seq": 0, 2024-03-11T21:12:26.253 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_epoch": 0, 2024-03-11T21:12:26.253 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_snaps": [], 2024-03-11T21:12:26.253 INFO:teuthology.orchestra.run.smithi023.stdout: "removed_snaps": "[]", 2024-03-11T21:12:26.254 INFO:teuthology.orchestra.run.smithi023.stdout: "quota_max_bytes": 0, 2024-03-11T21:12:26.254 INFO:teuthology.orchestra.run.smithi023.stdout: "quota_max_objects": 0, 2024-03-11T21:12:26.254 INFO:teuthology.orchestra.run.smithi023.stdout: "tiers": [], 2024-03-11T21:12:26.254 INFO:teuthology.orchestra.run.smithi023.stdout: "tier_of": -1, 2024-03-11T21:12:26.254 INFO:teuthology.orchestra.run.smithi023.stdout: "read_tier": -1, 2024-03-11T21:12:26.254 INFO:teuthology.orchestra.run.smithi023.stdout: "write_tier": -1, 2024-03-11T21:12:26.254 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_mode": "none", 2024-03-11T21:12:26.254 INFO:teuthology.orchestra.run.smithi023.stdout: "target_max_bytes": 0, 2024-03-11T21:12:26.254 INFO:teuthology.orchestra.run.smithi023.stdout: "target_max_objects": 0, 2024-03-11T21:12:26.254 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T21:12:26.254 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T21:12:26.254 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T21:12:26.254 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_min_flush_age": 0, 2024-03-11T21:12:26.254 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_min_evict_age": 0, 2024-03-11T21:12:26.255 INFO:teuthology.orchestra.run.smithi023.stdout: "erasure_code_profile": "", 2024-03-11T21:12:26.255 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_params": { 2024-03-11T21:12:26.255 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "none" 2024-03-11T21:12:26.255 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:26.255 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_period": 0, 2024-03-11T21:12:26.255 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_count": 0, 2024-03-11T21:12:26.255 INFO:teuthology.orchestra.run.smithi023.stdout: "use_gmt_hitset": true, 2024-03-11T21:12:26.255 INFO:teuthology.orchestra.run.smithi023.stdout: "min_read_recency_for_promote": 0, 2024-03-11T21:12:26.255 INFO:teuthology.orchestra.run.smithi023.stdout: "min_write_recency_for_promote": 0, 2024-03-11T21:12:26.255 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T21:12:26.255 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_search_last_n": 0, 2024-03-11T21:12:26.255 INFO:teuthology.orchestra.run.smithi023.stdout: "grade_table": [], 2024-03-11T21:12:26.255 INFO:teuthology.orchestra.run.smithi023.stdout: "stripe_width": 0, 2024-03-11T21:12:26.255 INFO:teuthology.orchestra.run.smithi023.stdout: "expected_num_objects": 0, 2024-03-11T21:12:26.255 INFO:teuthology.orchestra.run.smithi023.stdout: "fast_read": false, 2024-03-11T21:12:26.256 INFO:teuthology.orchestra.run.smithi023.stdout: "options": {}, 2024-03-11T21:12:26.256 INFO:teuthology.orchestra.run.smithi023.stdout: "application_metadata": { 2024-03-11T21:12:26.256 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw": {} 2024-03-11T21:12:26.256 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:26.256 INFO:teuthology.orchestra.run.smithi023.stdout: "read_balance": { 2024-03-11T21:12:26.256 INFO:teuthology.orchestra.run.smithi023.stdout: "score_type": "Fair distribution", 2024-03-11T21:12:26.256 INFO:teuthology.orchestra.run.smithi023.stdout: "score_acting": 1.5, 2024-03-11T21:12:26.256 INFO:teuthology.orchestra.run.smithi023.stdout: "score_stable": 1.5, 2024-03-11T21:12:26.256 INFO:teuthology.orchestra.run.smithi023.stdout: "optimal_score": 1.3300000429153442, 2024-03-11T21:12:26.256 INFO:teuthology.orchestra.run.smithi023.stdout: "raw_score_acting": 2, 2024-03-11T21:12:26.256 INFO:teuthology.orchestra.run.smithi023.stdout: "raw_score_stable": 2, 2024-03-11T21:12:26.256 INFO:teuthology.orchestra.run.smithi023.stdout: "primary_affinity_weighted": 0.75, 2024-03-11T21:12:26.256 INFO:teuthology.orchestra.run.smithi023.stdout: "average_primary_affinity": 1, 2024-03-11T21:12:26.256 INFO:teuthology.orchestra.run.smithi023.stdout: "average_primary_affinity_weighted": 0.75 2024-03-11T21:12:26.257 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:12:26.257 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:26.257 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:26.257 INFO:teuthology.orchestra.run.smithi023.stdout: "pool": 4, 2024-03-11T21:12:26.257 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_name": "default.rgw.control", 2024-03-11T21:12:26.257 INFO:teuthology.orchestra.run.smithi023.stdout: "create_time": "2024-03-11T20:50:30.536371+0000", 2024-03-11T21:12:26.257 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-11T21:12:26.257 INFO:teuthology.orchestra.run.smithi023.stdout: "flags_names": "hashpspool", 2024-03-11T21:12:26.257 INFO:teuthology.orchestra.run.smithi023.stdout: "type": 1, 2024-03-11T21:12:26.257 INFO:teuthology.orchestra.run.smithi023.stdout: "size": 3, 2024-03-11T21:12:26.257 INFO:teuthology.orchestra.run.smithi023.stdout: "min_size": 2, 2024-03-11T21:12:26.257 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_rule": 0, 2024-03-11T21:12:26.257 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_count": 0, 2024-03-11T21:12:26.257 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_target": 0, 2024-03-11T21:12:26.258 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T21:12:26.258 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T21:12:26.258 INFO:teuthology.orchestra.run.smithi023.stdout: "object_hash": 2, 2024-03-11T21:12:26.258 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_autoscale_mode": "off", 2024-03-11T21:12:26.258 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num": 32, 2024-03-11T21:12:26.258 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_placement_num": 32, 2024-03-11T21:12:26.258 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_placement_num_target": 32, 2024-03-11T21:12:26.258 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num_target": 32, 2024-03-11T21:12:26.258 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num_pending": 32, 2024-03-11T21:12:26.258 INFO:teuthology.orchestra.run.smithi023.stdout: "last_pg_merge_meta": { 2024-03-11T21:12:26.258 INFO:teuthology.orchestra.run.smithi023.stdout: "source_pgid": "0.0", 2024-03-11T21:12:26.258 INFO:teuthology.orchestra.run.smithi023.stdout: "ready_epoch": 0, 2024-03-11T21:12:26.258 INFO:teuthology.orchestra.run.smithi023.stdout: "last_epoch_started": 0, 2024-03-11T21:12:26.258 INFO:teuthology.orchestra.run.smithi023.stdout: "last_epoch_clean": 0, 2024-03-11T21:12:26.259 INFO:teuthology.orchestra.run.smithi023.stdout: "source_version": "0'0", 2024-03-11T21:12:26.259 INFO:teuthology.orchestra.run.smithi023.stdout: "target_version": "0'0" 2024-03-11T21:12:26.259 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:26.259 INFO:teuthology.orchestra.run.smithi023.stdout: "last_change": "82", 2024-03-11T21:12:26.259 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend": "0", 2024-03-11T21:12:26.259 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T21:12:26.259 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T21:12:26.259 INFO:teuthology.orchestra.run.smithi023.stdout: "auid": 0, 2024-03-11T21:12:26.259 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_mode": "selfmanaged", 2024-03-11T21:12:26.259 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_seq": 0, 2024-03-11T21:12:26.259 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_epoch": 0, 2024-03-11T21:12:26.259 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_snaps": [], 2024-03-11T21:12:26.259 INFO:teuthology.orchestra.run.smithi023.stdout: "removed_snaps": "[]", 2024-03-11T21:12:26.259 INFO:teuthology.orchestra.run.smithi023.stdout: "quota_max_bytes": 0, 2024-03-11T21:12:26.259 INFO:teuthology.orchestra.run.smithi023.stdout: "quota_max_objects": 0, 2024-03-11T21:12:26.260 INFO:teuthology.orchestra.run.smithi023.stdout: "tiers": [], 2024-03-11T21:12:26.260 INFO:teuthology.orchestra.run.smithi023.stdout: "tier_of": -1, 2024-03-11T21:12:26.260 INFO:teuthology.orchestra.run.smithi023.stdout: "read_tier": -1, 2024-03-11T21:12:26.260 INFO:teuthology.orchestra.run.smithi023.stdout: "write_tier": -1, 2024-03-11T21:12:26.260 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_mode": "none", 2024-03-11T21:12:26.260 INFO:teuthology.orchestra.run.smithi023.stdout: "target_max_bytes": 0, 2024-03-11T21:12:26.260 INFO:teuthology.orchestra.run.smithi023.stdout: "target_max_objects": 0, 2024-03-11T21:12:26.260 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T21:12:26.260 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T21:12:26.260 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T21:12:26.260 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_min_flush_age": 0, 2024-03-11T21:12:26.260 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_min_evict_age": 0, 2024-03-11T21:12:26.260 INFO:teuthology.orchestra.run.smithi023.stdout: "erasure_code_profile": "", 2024-03-11T21:12:26.260 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_params": { 2024-03-11T21:12:26.261 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "none" 2024-03-11T21:12:26.261 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:26.261 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_period": 0, 2024-03-11T21:12:26.261 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_count": 0, 2024-03-11T21:12:26.261 INFO:teuthology.orchestra.run.smithi023.stdout: "use_gmt_hitset": true, 2024-03-11T21:12:26.261 INFO:teuthology.orchestra.run.smithi023.stdout: "min_read_recency_for_promote": 0, 2024-03-11T21:12:26.261 INFO:teuthology.orchestra.run.smithi023.stdout: "min_write_recency_for_promote": 0, 2024-03-11T21:12:26.261 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T21:12:26.261 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_search_last_n": 0, 2024-03-11T21:12:26.261 INFO:teuthology.orchestra.run.smithi023.stdout: "grade_table": [], 2024-03-11T21:12:26.261 INFO:teuthology.orchestra.run.smithi023.stdout: "stripe_width": 0, 2024-03-11T21:12:26.261 INFO:teuthology.orchestra.run.smithi023.stdout: "expected_num_objects": 0, 2024-03-11T21:12:26.261 INFO:teuthology.orchestra.run.smithi023.stdout: "fast_read": false, 2024-03-11T21:12:26.261 INFO:teuthology.orchestra.run.smithi023.stdout: "options": {}, 2024-03-11T21:12:26.261 INFO:teuthology.orchestra.run.smithi023.stdout: "application_metadata": { 2024-03-11T21:12:26.262 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw": {} 2024-03-11T21:12:26.262 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:26.262 INFO:teuthology.orchestra.run.smithi023.stdout: "read_balance": { 2024-03-11T21:12:26.262 INFO:teuthology.orchestra.run.smithi023.stdout: "score_type": "Fair distribution", 2024-03-11T21:12:26.262 INFO:teuthology.orchestra.run.smithi023.stdout: "score_acting": 1.8799999952316284, 2024-03-11T21:12:26.262 INFO:teuthology.orchestra.run.smithi023.stdout: "score_stable": 1.8799999952316284, 2024-03-11T21:12:26.262 INFO:teuthology.orchestra.run.smithi023.stdout: "optimal_score": 1.3300000429153442, 2024-03-11T21:12:26.262 INFO:teuthology.orchestra.run.smithi023.stdout: "raw_score_acting": 2.5, 2024-03-11T21:12:26.262 INFO:teuthology.orchestra.run.smithi023.stdout: "raw_score_stable": 2.5, 2024-03-11T21:12:26.262 INFO:teuthology.orchestra.run.smithi023.stdout: "primary_affinity_weighted": 0.75, 2024-03-11T21:12:26.262 INFO:teuthology.orchestra.run.smithi023.stdout: "average_primary_affinity": 1, 2024-03-11T21:12:26.262 INFO:teuthology.orchestra.run.smithi023.stdout: "average_primary_affinity_weighted": 0.75 2024-03-11T21:12:26.262 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:12:26.262 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:26.262 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:26.263 INFO:teuthology.orchestra.run.smithi023.stdout: "pool": 5, 2024-03-11T21:12:26.263 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_name": "datapool", 2024-03-11T21:12:26.263 INFO:teuthology.orchestra.run.smithi023.stdout: "create_time": "2024-03-11T20:50:31.135845+0000", 2024-03-11T21:12:26.263 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 8193, 2024-03-11T21:12:26.263 INFO:teuthology.orchestra.run.smithi023.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-03-11T21:12:26.263 INFO:teuthology.orchestra.run.smithi023.stdout: "type": 1, 2024-03-11T21:12:26.263 INFO:teuthology.orchestra.run.smithi023.stdout: "size": 3, 2024-03-11T21:12:26.263 INFO:teuthology.orchestra.run.smithi023.stdout: "min_size": 2, 2024-03-11T21:12:26.263 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_rule": 0, 2024-03-11T21:12:26.263 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_count": 0, 2024-03-11T21:12:26.263 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_target": 0, 2024-03-11T21:12:26.263 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T21:12:26.263 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T21:12:26.263 INFO:teuthology.orchestra.run.smithi023.stdout: "object_hash": 2, 2024-03-11T21:12:26.263 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_autoscale_mode": "off", 2024-03-11T21:12:26.264 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num": 3, 2024-03-11T21:12:26.264 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_placement_num": 3, 2024-03-11T21:12:26.264 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_placement_num_target": 3, 2024-03-11T21:12:26.264 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num_target": 3, 2024-03-11T21:12:26.264 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num_pending": 3, 2024-03-11T21:12:26.264 INFO:teuthology.orchestra.run.smithi023.stdout: "last_pg_merge_meta": { 2024-03-11T21:12:26.264 INFO:teuthology.orchestra.run.smithi023.stdout: "source_pgid": "0.0", 2024-03-11T21:12:26.264 INFO:teuthology.orchestra.run.smithi023.stdout: "ready_epoch": 0, 2024-03-11T21:12:26.264 INFO:teuthology.orchestra.run.smithi023.stdout: "last_epoch_started": 0, 2024-03-11T21:12:26.264 INFO:teuthology.orchestra.run.smithi023.stdout: "last_epoch_clean": 0, 2024-03-11T21:12:26.264 INFO:teuthology.orchestra.run.smithi023.stdout: "source_version": "0'0", 2024-03-11T21:12:26.264 INFO:teuthology.orchestra.run.smithi023.stdout: "target_version": "0'0" 2024-03-11T21:12:26.264 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:26.264 INFO:teuthology.orchestra.run.smithi023.stdout: "last_change": "87", 2024-03-11T21:12:26.264 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend": "0", 2024-03-11T21:12:26.265 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T21:12:26.265 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T21:12:26.265 INFO:teuthology.orchestra.run.smithi023.stdout: "auid": 0, 2024-03-11T21:12:26.265 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_mode": "selfmanaged", 2024-03-11T21:12:26.265 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_seq": 2, 2024-03-11T21:12:26.265 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_epoch": 87, 2024-03-11T21:12:26.265 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_snaps": [], 2024-03-11T21:12:26.265 INFO:teuthology.orchestra.run.smithi023.stdout: "removed_snaps": "[]", 2024-03-11T21:12:26.265 INFO:teuthology.orchestra.run.smithi023.stdout: "quota_max_bytes": 0, 2024-03-11T21:12:26.265 INFO:teuthology.orchestra.run.smithi023.stdout: "quota_max_objects": 0, 2024-03-11T21:12:26.265 INFO:teuthology.orchestra.run.smithi023.stdout: "tiers": [], 2024-03-11T21:12:26.265 INFO:teuthology.orchestra.run.smithi023.stdout: "tier_of": -1, 2024-03-11T21:12:26.265 INFO:teuthology.orchestra.run.smithi023.stdout: "read_tier": -1, 2024-03-11T21:12:26.265 INFO:teuthology.orchestra.run.smithi023.stdout: "write_tier": -1, 2024-03-11T21:12:26.265 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_mode": "none", 2024-03-11T21:12:26.266 INFO:teuthology.orchestra.run.smithi023.stdout: "target_max_bytes": 0, 2024-03-11T21:12:26.266 INFO:teuthology.orchestra.run.smithi023.stdout: "target_max_objects": 0, 2024-03-11T21:12:26.266 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T21:12:26.266 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T21:12:26.266 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T21:12:26.266 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_min_flush_age": 0, 2024-03-11T21:12:26.266 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_min_evict_age": 0, 2024-03-11T21:12:26.266 INFO:teuthology.orchestra.run.smithi023.stdout: "erasure_code_profile": "", 2024-03-11T21:12:26.266 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_params": { 2024-03-11T21:12:26.266 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "none" 2024-03-11T21:12:26.266 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:26.266 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_period": 0, 2024-03-11T21:12:26.266 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_count": 0, 2024-03-11T21:12:26.266 INFO:teuthology.orchestra.run.smithi023.stdout: "use_gmt_hitset": true, 2024-03-11T21:12:26.267 INFO:teuthology.orchestra.run.smithi023.stdout: "min_read_recency_for_promote": 0, 2024-03-11T21:12:26.267 INFO:teuthology.orchestra.run.smithi023.stdout: "min_write_recency_for_promote": 0, 2024-03-11T21:12:26.267 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T21:12:26.267 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_search_last_n": 0, 2024-03-11T21:12:26.267 INFO:teuthology.orchestra.run.smithi023.stdout: "grade_table": [], 2024-03-11T21:12:26.267 INFO:teuthology.orchestra.run.smithi023.stdout: "stripe_width": 0, 2024-03-11T21:12:26.267 INFO:teuthology.orchestra.run.smithi023.stdout: "expected_num_objects": 0, 2024-03-11T21:12:26.267 INFO:teuthology.orchestra.run.smithi023.stdout: "fast_read": false, 2024-03-11T21:12:26.267 INFO:teuthology.orchestra.run.smithi023.stdout: "options": {}, 2024-03-11T21:12:26.267 INFO:teuthology.orchestra.run.smithi023.stdout: "application_metadata": { 2024-03-11T21:12:26.267 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd": {} 2024-03-11T21:12:26.267 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:26.267 INFO:teuthology.orchestra.run.smithi023.stdout: "read_balance": { 2024-03-11T21:12:26.267 INFO:teuthology.orchestra.run.smithi023.stdout: "score_type": "Fair distribution", 2024-03-11T21:12:26.267 INFO:teuthology.orchestra.run.smithi023.stdout: "score_acting": 3.9900000095367432, 2024-03-11T21:12:26.268 INFO:teuthology.orchestra.run.smithi023.stdout: "score_stable": 3.9900000095367432, 2024-03-11T21:12:26.268 INFO:teuthology.orchestra.run.smithi023.stdout: "optimal_score": 1.1699999570846558, 2024-03-11T21:12:26.268 INFO:teuthology.orchestra.run.smithi023.stdout: "raw_score_acting": 4.6700000762939453, 2024-03-11T21:12:26.268 INFO:teuthology.orchestra.run.smithi023.stdout: "raw_score_stable": 4.6700000762939453, 2024-03-11T21:12:26.268 INFO:teuthology.orchestra.run.smithi023.stdout: "primary_affinity_weighted": 0.75, 2024-03-11T21:12:26.268 INFO:teuthology.orchestra.run.smithi023.stdout: "average_primary_affinity": 1, 2024-03-11T21:12:26.268 INFO:teuthology.orchestra.run.smithi023.stdout: "average_primary_affinity_weighted": 0.75 2024-03-11T21:12:26.268 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:12:26.268 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:26.268 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:26.268 INFO:teuthology.orchestra.run.smithi023.stdout: "pool": 6, 2024-03-11T21:12:26.268 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_name": "default.rgw.meta", 2024-03-11T21:12:26.268 INFO:teuthology.orchestra.run.smithi023.stdout: "create_time": "2024-03-11T20:50:33.233163+0000", 2024-03-11T21:12:26.268 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-11T21:12:26.269 INFO:teuthology.orchestra.run.smithi023.stdout: "flags_names": "hashpspool", 2024-03-11T21:12:26.269 INFO:teuthology.orchestra.run.smithi023.stdout: "type": 1, 2024-03-11T21:12:26.269 INFO:teuthology.orchestra.run.smithi023.stdout: "size": 3, 2024-03-11T21:12:26.269 INFO:teuthology.orchestra.run.smithi023.stdout: "min_size": 2, 2024-03-11T21:12:26.269 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_rule": 0, 2024-03-11T21:12:26.269 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_count": 0, 2024-03-11T21:12:26.269 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_target": 0, 2024-03-11T21:12:26.269 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T21:12:26.269 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T21:12:26.269 INFO:teuthology.orchestra.run.smithi023.stdout: "object_hash": 2, 2024-03-11T21:12:26.269 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_autoscale_mode": "off", 2024-03-11T21:12:26.269 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num": 32, 2024-03-11T21:12:26.269 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_placement_num": 32, 2024-03-11T21:12:26.269 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_placement_num_target": 32, 2024-03-11T21:12:26.269 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num_target": 32, 2024-03-11T21:12:26.270 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num_pending": 32, 2024-03-11T21:12:26.270 INFO:teuthology.orchestra.run.smithi023.stdout: "last_pg_merge_meta": { 2024-03-11T21:12:26.270 INFO:teuthology.orchestra.run.smithi023.stdout: "source_pgid": "0.0", 2024-03-11T21:12:26.270 INFO:teuthology.orchestra.run.smithi023.stdout: "ready_epoch": 0, 2024-03-11T21:12:26.270 INFO:teuthology.orchestra.run.smithi023.stdout: "last_epoch_started": 0, 2024-03-11T21:12:26.270 INFO:teuthology.orchestra.run.smithi023.stdout: "last_epoch_clean": 0, 2024-03-11T21:12:26.270 INFO:teuthology.orchestra.run.smithi023.stdout: "source_version": "0'0", 2024-03-11T21:12:26.270 INFO:teuthology.orchestra.run.smithi023.stdout: "target_version": "0'0" 2024-03-11T21:12:26.270 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:26.270 INFO:teuthology.orchestra.run.smithi023.stdout: "last_change": "84", 2024-03-11T21:12:26.270 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend": "0", 2024-03-11T21:12:26.270 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T21:12:26.270 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T21:12:26.270 INFO:teuthology.orchestra.run.smithi023.stdout: "auid": 0, 2024-03-11T21:12:26.270 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_mode": "selfmanaged", 2024-03-11T21:12:26.271 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_seq": 0, 2024-03-11T21:12:26.271 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_epoch": 0, 2024-03-11T21:12:26.271 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_snaps": [], 2024-03-11T21:12:26.271 INFO:teuthology.orchestra.run.smithi023.stdout: "removed_snaps": "[]", 2024-03-11T21:12:26.271 INFO:teuthology.orchestra.run.smithi023.stdout: "quota_max_bytes": 0, 2024-03-11T21:12:26.271 INFO:teuthology.orchestra.run.smithi023.stdout: "quota_max_objects": 0, 2024-03-11T21:12:26.271 INFO:teuthology.orchestra.run.smithi023.stdout: "tiers": [], 2024-03-11T21:12:26.271 INFO:teuthology.orchestra.run.smithi023.stdout: "tier_of": -1, 2024-03-11T21:12:26.271 INFO:teuthology.orchestra.run.smithi023.stdout: "read_tier": -1, 2024-03-11T21:12:26.271 INFO:teuthology.orchestra.run.smithi023.stdout: "write_tier": -1, 2024-03-11T21:12:26.271 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_mode": "none", 2024-03-11T21:12:26.271 INFO:teuthology.orchestra.run.smithi023.stdout: "target_max_bytes": 0, 2024-03-11T21:12:26.271 INFO:teuthology.orchestra.run.smithi023.stdout: "target_max_objects": 0, 2024-03-11T21:12:26.271 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T21:12:26.272 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T21:12:26.272 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T21:12:26.272 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_min_flush_age": 0, 2024-03-11T21:12:26.272 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_min_evict_age": 0, 2024-03-11T21:12:26.272 INFO:teuthology.orchestra.run.smithi023.stdout: "erasure_code_profile": "", 2024-03-11T21:12:26.272 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_params": { 2024-03-11T21:12:26.272 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "none" 2024-03-11T21:12:26.272 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:26.272 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_period": 0, 2024-03-11T21:12:26.272 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_count": 0, 2024-03-11T21:12:26.272 INFO:teuthology.orchestra.run.smithi023.stdout: "use_gmt_hitset": true, 2024-03-11T21:12:26.272 INFO:teuthology.orchestra.run.smithi023.stdout: "min_read_recency_for_promote": 0, 2024-03-11T21:12:26.272 INFO:teuthology.orchestra.run.smithi023.stdout: "min_write_recency_for_promote": 0, 2024-03-11T21:12:26.272 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T21:12:26.272 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_search_last_n": 0, 2024-03-11T21:12:26.273 INFO:teuthology.orchestra.run.smithi023.stdout: "grade_table": [], 2024-03-11T21:12:26.273 INFO:teuthology.orchestra.run.smithi023.stdout: "stripe_width": 0, 2024-03-11T21:12:26.273 INFO:teuthology.orchestra.run.smithi023.stdout: "expected_num_objects": 0, 2024-03-11T21:12:26.273 INFO:teuthology.orchestra.run.smithi023.stdout: "fast_read": false, 2024-03-11T21:12:26.273 INFO:teuthology.orchestra.run.smithi023.stdout: "options": { 2024-03-11T21:12:26.273 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_autoscale_bias": 4 2024-03-11T21:12:26.273 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:26.273 INFO:teuthology.orchestra.run.smithi023.stdout: "application_metadata": { 2024-03-11T21:12:26.273 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw": {} 2024-03-11T21:12:26.273 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:26.273 INFO:teuthology.orchestra.run.smithi023.stdout: "read_balance": { 2024-03-11T21:12:26.273 INFO:teuthology.orchestra.run.smithi023.stdout: "score_type": "Fair distribution", 2024-03-11T21:12:26.273 INFO:teuthology.orchestra.run.smithi023.stdout: "score_acting": 1.690000057220459, 2024-03-11T21:12:26.273 INFO:teuthology.orchestra.run.smithi023.stdout: "score_stable": 1.690000057220459, 2024-03-11T21:12:26.274 INFO:teuthology.orchestra.run.smithi023.stdout: "optimal_score": 1.3300000429153442, 2024-03-11T21:12:26.274 INFO:teuthology.orchestra.run.smithi023.stdout: "raw_score_acting": 2.25, 2024-03-11T21:12:26.274 INFO:teuthology.orchestra.run.smithi023.stdout: "raw_score_stable": 2.25, 2024-03-11T21:12:26.274 INFO:teuthology.orchestra.run.smithi023.stdout: "primary_affinity_weighted": 0.75, 2024-03-11T21:12:26.274 INFO:teuthology.orchestra.run.smithi023.stdout: "average_primary_affinity": 1, 2024-03-11T21:12:26.274 INFO:teuthology.orchestra.run.smithi023.stdout: "average_primary_affinity_weighted": 0.75 2024-03-11T21:12:26.274 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:12:26.274 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:26.274 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:26.274 INFO:teuthology.orchestra.run.smithi023.stdout: "pool": 11, 2024-03-11T21:12:26.274 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_name": "unique_pool_4", 2024-03-11T21:12:26.274 INFO:teuthology.orchestra.run.smithi023.stdout: "create_time": "2024-03-11T21:11:46.486253+0000", 2024-03-11T21:12:26.274 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-11T21:12:26.274 INFO:teuthology.orchestra.run.smithi023.stdout: "flags_names": "hashpspool", 2024-03-11T21:12:26.274 INFO:teuthology.orchestra.run.smithi023.stdout: "type": 1, 2024-03-11T21:12:26.275 INFO:teuthology.orchestra.run.smithi023.stdout: "size": 3, 2024-03-11T21:12:26.275 INFO:teuthology.orchestra.run.smithi023.stdout: "min_size": 2, 2024-03-11T21:12:26.275 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_rule": 0, 2024-03-11T21:12:26.275 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_count": 0, 2024-03-11T21:12:26.275 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_target": 0, 2024-03-11T21:12:26.275 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T21:12:26.275 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T21:12:26.275 INFO:teuthology.orchestra.run.smithi023.stdout: "object_hash": 2, 2024-03-11T21:12:26.275 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_autoscale_mode": "off", 2024-03-11T21:12:26.275 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num": 16, 2024-03-11T21:12:26.275 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_placement_num": 16, 2024-03-11T21:12:26.275 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_placement_num_target": 16, 2024-03-11T21:12:26.275 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num_target": 16, 2024-03-11T21:12:26.275 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num_pending": 16, 2024-03-11T21:12:26.275 INFO:teuthology.orchestra.run.smithi023.stdout: "last_pg_merge_meta": { 2024-03-11T21:12:26.276 INFO:teuthology.orchestra.run.smithi023.stdout: "source_pgid": "0.0", 2024-03-11T21:12:26.276 INFO:teuthology.orchestra.run.smithi023.stdout: "ready_epoch": 0, 2024-03-11T21:12:26.276 INFO:teuthology.orchestra.run.smithi023.stdout: "last_epoch_started": 0, 2024-03-11T21:12:26.276 INFO:teuthology.orchestra.run.smithi023.stdout: "last_epoch_clean": 0, 2024-03-11T21:12:26.276 INFO:teuthology.orchestra.run.smithi023.stdout: "source_version": "0'0", 2024-03-11T21:12:26.276 INFO:teuthology.orchestra.run.smithi023.stdout: "target_version": "0'0" 2024-03-11T21:12:26.276 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:26.276 INFO:teuthology.orchestra.run.smithi023.stdout: "last_change": "386", 2024-03-11T21:12:26.276 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend": "0", 2024-03-11T21:12:26.276 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T21:12:26.276 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T21:12:26.276 INFO:teuthology.orchestra.run.smithi023.stdout: "auid": 0, 2024-03-11T21:12:26.276 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_mode": "selfmanaged", 2024-03-11T21:12:26.276 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_seq": 0, 2024-03-11T21:12:26.276 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_epoch": 0, 2024-03-11T21:12:26.277 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_snaps": [], 2024-03-11T21:12:26.277 INFO:teuthology.orchestra.run.smithi023.stdout: "removed_snaps": "[]", 2024-03-11T21:12:26.277 INFO:teuthology.orchestra.run.smithi023.stdout: "quota_max_bytes": 0, 2024-03-11T21:12:26.277 INFO:teuthology.orchestra.run.smithi023.stdout: "quota_max_objects": 0, 2024-03-11T21:12:26.277 INFO:teuthology.orchestra.run.smithi023.stdout: "tiers": [], 2024-03-11T21:12:26.277 INFO:teuthology.orchestra.run.smithi023.stdout: "tier_of": -1, 2024-03-11T21:12:26.277 INFO:teuthology.orchestra.run.smithi023.stdout: "read_tier": -1, 2024-03-11T21:12:26.277 INFO:teuthology.orchestra.run.smithi023.stdout: "write_tier": -1, 2024-03-11T21:12:26.277 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_mode": "none", 2024-03-11T21:12:26.277 INFO:teuthology.orchestra.run.smithi023.stdout: "target_max_bytes": 0, 2024-03-11T21:12:26.277 INFO:teuthology.orchestra.run.smithi023.stdout: "target_max_objects": 0, 2024-03-11T21:12:26.277 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T21:12:26.277 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T21:12:26.277 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T21:12:26.277 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_min_flush_age": 0, 2024-03-11T21:12:26.278 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_min_evict_age": 0, 2024-03-11T21:12:26.278 INFO:teuthology.orchestra.run.smithi023.stdout: "erasure_code_profile": "", 2024-03-11T21:12:26.278 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_params": { 2024-03-11T21:12:26.278 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "none" 2024-03-11T21:12:26.278 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:26.278 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_period": 0, 2024-03-11T21:12:26.278 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_count": 0, 2024-03-11T21:12:26.278 INFO:teuthology.orchestra.run.smithi023.stdout: "use_gmt_hitset": true, 2024-03-11T21:12:26.278 INFO:teuthology.orchestra.run.smithi023.stdout: "min_read_recency_for_promote": 0, 2024-03-11T21:12:26.278 INFO:teuthology.orchestra.run.smithi023.stdout: "min_write_recency_for_promote": 0, 2024-03-11T21:12:26.278 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T21:12:26.278 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_search_last_n": 0, 2024-03-11T21:12:26.278 INFO:teuthology.orchestra.run.smithi023.stdout: "grade_table": [], 2024-03-11T21:12:26.278 INFO:teuthology.orchestra.run.smithi023.stdout: "stripe_width": 0, 2024-03-11T21:12:26.279 INFO:teuthology.orchestra.run.smithi023.stdout: "expected_num_objects": 0, 2024-03-11T21:12:26.279 INFO:teuthology.orchestra.run.smithi023.stdout: "fast_read": false, 2024-03-11T21:12:26.279 INFO:teuthology.orchestra.run.smithi023.stdout: "options": {}, 2024-03-11T21:12:26.279 INFO:teuthology.orchestra.run.smithi023.stdout: "application_metadata": { 2024-03-11T21:12:26.279 INFO:teuthology.orchestra.run.smithi023.stdout: "rados": {} 2024-03-11T21:12:26.279 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:26.279 INFO:teuthology.orchestra.run.smithi023.stdout: "read_balance": { 2024-03-11T21:12:26.279 INFO:teuthology.orchestra.run.smithi023.stdout: "score_type": "Fair distribution", 2024-03-11T21:12:26.279 INFO:teuthology.orchestra.run.smithi023.stdout: "score_acting": 1.8799999952316284, 2024-03-11T21:12:26.279 INFO:teuthology.orchestra.run.smithi023.stdout: "score_stable": 1.8799999952316284, 2024-03-11T21:12:26.279 INFO:teuthology.orchestra.run.smithi023.stdout: "optimal_score": 1.3300000429153442, 2024-03-11T21:12:26.279 INFO:teuthology.orchestra.run.smithi023.stdout: "raw_score_acting": 2.5, 2024-03-11T21:12:26.279 INFO:teuthology.orchestra.run.smithi023.stdout: "raw_score_stable": 2.5, 2024-03-11T21:12:26.279 INFO:teuthology.orchestra.run.smithi023.stdout: "primary_affinity_weighted": 0.75, 2024-03-11T21:12:26.279 INFO:teuthology.orchestra.run.smithi023.stdout: "average_primary_affinity": 1, 2024-03-11T21:12:26.280 INFO:teuthology.orchestra.run.smithi023.stdout: "average_primary_affinity_weighted": 0.75 2024-03-11T21:12:26.280 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:12:26.280 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:12:26.280 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-11T21:12:26.280 INFO:teuthology.orchestra.run.smithi023.stdout: "osds": [ 2024-03-11T21:12:26.280 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:26.280 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": 0, 2024-03-11T21:12:26.280 INFO:teuthology.orchestra.run.smithi023.stdout: "uuid": "3df72b4a-8fe4-4265-b4c6-af022e1e23e1", 2024-03-11T21:12:26.280 INFO:teuthology.orchestra.run.smithi023.stdout: "up": 1, 2024-03-11T21:12:26.280 INFO:teuthology.orchestra.run.smithi023.stdout: "in": 1, 2024-03-11T21:12:26.280 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 1, 2024-03-11T21:12:26.280 INFO:teuthology.orchestra.run.smithi023.stdout: "primary_affinity": 1, 2024-03-11T21:12:26.280 INFO:teuthology.orchestra.run.smithi023.stdout: "last_clean_begin": 0, 2024-03-11T21:12:26.280 INFO:teuthology.orchestra.run.smithi023.stdout: "last_clean_end": 0, 2024-03-11T21:12:26.280 INFO:teuthology.orchestra.run.smithi023.stdout: "up_from": 8, 2024-03-11T21:12:26.281 INFO:teuthology.orchestra.run.smithi023.stdout: "up_thru": 380, 2024-03-11T21:12:26.281 INFO:teuthology.orchestra.run.smithi023.stdout: "down_at": 0, 2024-03-11T21:12:26.281 INFO:teuthology.orchestra.run.smithi023.stdout: "lost_at": 0, 2024-03-11T21:12:26.281 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2024-03-11T21:12:26.281 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:12:26.281 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:26.281 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:12:26.281 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6802", 2024-03-11T21:12:26.281 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 2154627393 2024-03-11T21:12:26.281 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:26.281 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:26.281 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:12:26.281 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6803", 2024-03-11T21:12:26.281 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 2154627393 2024-03-11T21:12:26.281 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:12:26.281 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:12:26.282 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:26.282 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_addrs": { 2024-03-11T21:12:26.282 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:12:26.282 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:26.282 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:12:26.282 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6804", 2024-03-11T21:12:26.282 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 2154627393 2024-03-11T21:12:26.282 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:26.282 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:26.282 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:12:26.282 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6805", 2024-03-11T21:12:26.282 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 2154627393 2024-03-11T21:12:26.282 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:12:26.282 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:12:26.283 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:26.283 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_back_addrs": { 2024-03-11T21:12:26.283 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:12:26.283 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:26.283 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:12:26.283 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6808", 2024-03-11T21:12:26.283 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 2154627393 2024-03-11T21:12:26.283 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:26.283 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:26.283 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:12:26.283 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6809", 2024-03-11T21:12:26.283 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 2154627393 2024-03-11T21:12:26.283 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:12:26.283 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:12:26.284 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:26.284 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_front_addrs": { 2024-03-11T21:12:26.284 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:12:26.284 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:26.284 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:12:26.284 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6806", 2024-03-11T21:12:26.284 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 2154627393 2024-03-11T21:12:26.284 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:26.284 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:26.284 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:12:26.284 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6807", 2024-03-11T21:12:26.284 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 2154627393 2024-03-11T21:12:26.284 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:12:26.284 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:12:26.285 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:26.285 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6803/2154627393", 2024-03-11T21:12:26.285 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_addr": "172.21.15.23:6805/2154627393", 2024-03-11T21:12:26.285 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_back_addr": "172.21.15.23:6809/2154627393", 2024-03-11T21:12:26.285 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_front_addr": "172.21.15.23:6807/2154627393", 2024-03-11T21:12:26.285 INFO:teuthology.orchestra.run.smithi023.stdout: "state": [ 2024-03-11T21:12:26.285 INFO:teuthology.orchestra.run.smithi023.stdout: "exists", 2024-03-11T21:12:26.285 INFO:teuthology.orchestra.run.smithi023.stdout: "up" 2024-03-11T21:12:26.285 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:12:26.285 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:26.286 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:26.286 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": 1, 2024-03-11T21:12:26.286 INFO:teuthology.orchestra.run.smithi023.stdout: "uuid": "e8fd7500-9081-41a1-8077-a4d0f0e12a2c", 2024-03-11T21:12:26.286 INFO:teuthology.orchestra.run.smithi023.stdout: "up": 1, 2024-03-11T21:12:26.286 INFO:teuthology.orchestra.run.smithi023.stdout: "in": 1, 2024-03-11T21:12:26.286 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 1, 2024-03-11T21:12:26.286 INFO:teuthology.orchestra.run.smithi023.stdout: "primary_affinity": 1, 2024-03-11T21:12:26.286 INFO:teuthology.orchestra.run.smithi023.stdout: "last_clean_begin": 13, 2024-03-11T21:12:26.286 INFO:teuthology.orchestra.run.smithi023.stdout: "last_clean_end": 173, 2024-03-11T21:12:26.286 INFO:teuthology.orchestra.run.smithi023.stdout: "up_from": 211, 2024-03-11T21:12:26.286 INFO:teuthology.orchestra.run.smithi023.stdout: "up_thru": 380, 2024-03-11T21:12:26.286 INFO:teuthology.orchestra.run.smithi023.stdout: "down_at": 174, 2024-03-11T21:12:26.286 INFO:teuthology.orchestra.run.smithi023.stdout: "lost_at": 0, 2024-03-11T21:12:26.286 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2024-03-11T21:12:26.287 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:12:26.287 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:26.287 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:12:26.287 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6810", 2024-03-11T21:12:26.287 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 56585354 2024-03-11T21:12:26.287 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:26.287 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:26.287 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:12:26.287 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6811", 2024-03-11T21:12:26.287 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 56585354 2024-03-11T21:12:26.287 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:12:26.287 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:12:26.287 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:26.288 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_addrs": { 2024-03-11T21:12:26.288 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:12:26.288 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:26.288 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:12:26.288 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6812", 2024-03-11T21:12:26.288 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 56585354 2024-03-11T21:12:26.288 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:26.288 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:26.288 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:12:26.288 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6813", 2024-03-11T21:12:26.288 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 56585354 2024-03-11T21:12:26.288 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:12:26.288 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:12:26.288 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:26.289 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_back_addrs": { 2024-03-11T21:12:26.289 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:12:26.289 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:26.289 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:12:26.289 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6816", 2024-03-11T21:12:26.289 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 56585354 2024-03-11T21:12:26.289 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:26.289 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:26.289 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:12:26.289 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6817", 2024-03-11T21:12:26.289 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 56585354 2024-03-11T21:12:26.289 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:12:26.289 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:12:26.289 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:26.289 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_front_addrs": { 2024-03-11T21:12:26.290 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:12:26.290 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:26.290 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:12:26.290 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6814", 2024-03-11T21:12:26.290 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 56585354 2024-03-11T21:12:26.290 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:26.290 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:26.290 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:12:26.290 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6815", 2024-03-11T21:12:26.290 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 56585354 2024-03-11T21:12:26.290 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:12:26.290 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:12:26.290 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:26.290 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6811/56585354", 2024-03-11T21:12:26.291 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_addr": "172.21.15.23:6813/56585354", 2024-03-11T21:12:26.291 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_back_addr": "172.21.15.23:6817/56585354", 2024-03-11T21:12:26.291 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_front_addr": "172.21.15.23:6815/56585354", 2024-03-11T21:12:26.291 INFO:teuthology.orchestra.run.smithi023.stdout: "state": [ 2024-03-11T21:12:26.291 INFO:teuthology.orchestra.run.smithi023.stdout: "exists", 2024-03-11T21:12:26.291 INFO:teuthology.orchestra.run.smithi023.stdout: "up" 2024-03-11T21:12:26.291 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:12:26.291 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:26.291 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:26.291 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": 2, 2024-03-11T21:12:26.291 INFO:teuthology.orchestra.run.smithi023.stdout: "uuid": "dd85d52a-df48-4b03-8612-cecfd82b5987", 2024-03-11T21:12:26.291 INFO:teuthology.orchestra.run.smithi023.stdout: "up": 1, 2024-03-11T21:12:26.291 INFO:teuthology.orchestra.run.smithi023.stdout: "in": 1, 2024-03-11T21:12:26.291 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 1, 2024-03-11T21:12:26.292 INFO:teuthology.orchestra.run.smithi023.stdout: "primary_affinity": 0, 2024-03-11T21:12:26.292 INFO:teuthology.orchestra.run.smithi023.stdout: "last_clean_begin": 0, 2024-03-11T21:12:26.292 INFO:teuthology.orchestra.run.smithi023.stdout: "last_clean_end": 0, 2024-03-11T21:12:26.292 INFO:teuthology.orchestra.run.smithi023.stdout: "up_from": 19, 2024-03-11T21:12:26.292 INFO:teuthology.orchestra.run.smithi023.stdout: "up_thru": 306, 2024-03-11T21:12:26.292 INFO:teuthology.orchestra.run.smithi023.stdout: "down_at": 0, 2024-03-11T21:12:26.292 INFO:teuthology.orchestra.run.smithi023.stdout: "lost_at": 0, 2024-03-11T21:12:26.292 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2024-03-11T21:12:26.292 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:12:26.292 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:26.292 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:12:26.292 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6818", 2024-03-11T21:12:26.292 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1770748056 2024-03-11T21:12:26.292 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:26.293 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:26.293 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:12:26.293 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6819", 2024-03-11T21:12:26.293 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1770748056 2024-03-11T21:12:26.293 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:12:26.293 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:12:26.293 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:26.293 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_addrs": { 2024-03-11T21:12:26.293 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:12:26.293 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:26.293 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:12:26.293 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6820", 2024-03-11T21:12:26.293 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1770748056 2024-03-11T21:12:26.293 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:26.294 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:26.294 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:12:26.294 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6821", 2024-03-11T21:12:26.294 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1770748056 2024-03-11T21:12:26.294 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:12:26.294 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:12:26.294 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:26.294 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_back_addrs": { 2024-03-11T21:12:26.294 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:12:26.294 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:26.294 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:12:26.294 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6824", 2024-03-11T21:12:26.294 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1770748056 2024-03-11T21:12:26.294 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:26.295 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:26.295 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:12:26.295 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6825", 2024-03-11T21:12:26.295 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1770748056 2024-03-11T21:12:26.295 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:12:26.295 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:12:26.295 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:26.295 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_front_addrs": { 2024-03-11T21:12:26.295 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:12:26.295 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:26.295 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:12:26.295 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6822", 2024-03-11T21:12:26.295 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1770748056 2024-03-11T21:12:26.295 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:26.295 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:26.296 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:12:26.296 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6823", 2024-03-11T21:12:26.296 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1770748056 2024-03-11T21:12:26.296 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:12:26.296 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:12:26.296 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:26.296 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6819/1770748056", 2024-03-11T21:12:26.296 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_addr": "172.21.15.23:6821/1770748056", 2024-03-11T21:12:26.296 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_back_addr": "172.21.15.23:6825/1770748056", 2024-03-11T21:12:26.296 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_front_addr": "172.21.15.23:6823/1770748056", 2024-03-11T21:12:26.296 INFO:teuthology.orchestra.run.smithi023.stdout: "state": [ 2024-03-11T21:12:26.296 INFO:teuthology.orchestra.run.smithi023.stdout: "exists", 2024-03-11T21:12:26.296 INFO:teuthology.orchestra.run.smithi023.stdout: "up" 2024-03-11T21:12:26.296 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:12:26.297 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:26.297 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:26.297 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": 3, 2024-03-11T21:12:26.297 INFO:teuthology.orchestra.run.smithi023.stdout: "uuid": "e3021106-3779-4b94-9bbf-a7724544110f", 2024-03-11T21:12:26.297 INFO:teuthology.orchestra.run.smithi023.stdout: "up": 1, 2024-03-11T21:12:26.297 INFO:teuthology.orchestra.run.smithi023.stdout: "in": 1, 2024-03-11T21:12:26.297 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 1, 2024-03-11T21:12:26.297 INFO:teuthology.orchestra.run.smithi023.stdout: "primary_affinity": 1, 2024-03-11T21:12:26.297 INFO:teuthology.orchestra.run.smithi023.stdout: "last_clean_begin": 0, 2024-03-11T21:12:26.297 INFO:teuthology.orchestra.run.smithi023.stdout: "last_clean_end": 0, 2024-03-11T21:12:26.297 INFO:teuthology.orchestra.run.smithi023.stdout: "up_from": 28, 2024-03-11T21:12:26.297 INFO:teuthology.orchestra.run.smithi023.stdout: "up_thru": 380, 2024-03-11T21:12:26.297 INFO:teuthology.orchestra.run.smithi023.stdout: "down_at": 0, 2024-03-11T21:12:26.297 INFO:teuthology.orchestra.run.smithi023.stdout: "lost_at": 0, 2024-03-11T21:12:26.297 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2024-03-11T21:12:26.298 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:12:26.298 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:26.298 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:12:26.298 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6826", 2024-03-11T21:12:26.298 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 938496814 2024-03-11T21:12:26.298 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:26.298 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:26.298 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:12:26.298 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6827", 2024-03-11T21:12:26.298 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 938496814 2024-03-11T21:12:26.298 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:12:26.298 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:12:26.298 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:26.298 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_addrs": { 2024-03-11T21:12:26.299 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:12:26.299 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:26.299 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:12:26.299 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6828", 2024-03-11T21:12:26.299 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 938496814 2024-03-11T21:12:26.299 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:26.299 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:26.299 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:12:26.299 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6829", 2024-03-11T21:12:26.299 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 938496814 2024-03-11T21:12:26.299 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:12:26.299 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:12:26.299 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:26.299 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_back_addrs": { 2024-03-11T21:12:26.299 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:12:26.300 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:26.300 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:12:26.300 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6832", 2024-03-11T21:12:26.300 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 938496814 2024-03-11T21:12:26.300 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:26.300 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:26.300 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:12:26.300 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6833", 2024-03-11T21:12:26.300 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 938496814 2024-03-11T21:12:26.300 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:12:26.300 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:12:26.300 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:26.300 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_front_addrs": { 2024-03-11T21:12:26.300 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:12:26.301 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:26.301 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:12:26.301 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6830", 2024-03-11T21:12:26.301 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 938496814 2024-03-11T21:12:26.301 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:26.301 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:26.301 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:12:26.301 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6831", 2024-03-11T21:12:26.301 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 938496814 2024-03-11T21:12:26.301 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:12:26.301 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:12:26.301 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:26.301 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6827/938496814", 2024-03-11T21:12:26.301 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_addr": "172.21.15.23:6829/938496814", 2024-03-11T21:12:26.301 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_back_addr": "172.21.15.23:6833/938496814", 2024-03-11T21:12:26.302 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_front_addr": "172.21.15.23:6831/938496814", 2024-03-11T21:12:26.302 INFO:teuthology.orchestra.run.smithi023.stdout: "state": [ 2024-03-11T21:12:26.302 INFO:teuthology.orchestra.run.smithi023.stdout: "exists", 2024-03-11T21:12:26.302 INFO:teuthology.orchestra.run.smithi023.stdout: "up" 2024-03-11T21:12:26.302 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:12:26.302 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:26.302 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:26.302 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": 4, 2024-03-11T21:12:26.302 INFO:teuthology.orchestra.run.smithi023.stdout: "uuid": "4dcbc91e-7035-413d-97bc-c9d804eb16d7", 2024-03-11T21:12:26.302 INFO:teuthology.orchestra.run.smithi023.stdout: "up": 1, 2024-03-11T21:12:26.302 INFO:teuthology.orchestra.run.smithi023.stdout: "in": 1, 2024-03-11T21:12:26.302 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 1, 2024-03-11T21:12:26.302 INFO:teuthology.orchestra.run.smithi023.stdout: "primary_affinity": 0, 2024-03-11T21:12:26.303 INFO:teuthology.orchestra.run.smithi023.stdout: "last_clean_begin": 35, 2024-03-11T21:12:26.303 INFO:teuthology.orchestra.run.smithi023.stdout: "last_clean_end": 182, 2024-03-11T21:12:26.303 INFO:teuthology.orchestra.run.smithi023.stdout: "up_from": 204, 2024-03-11T21:12:26.303 INFO:teuthology.orchestra.run.smithi023.stdout: "up_thru": 248, 2024-03-11T21:12:26.303 INFO:teuthology.orchestra.run.smithi023.stdout: "down_at": 183, 2024-03-11T21:12:26.303 INFO:teuthology.orchestra.run.smithi023.stdout: "lost_at": 0, 2024-03-11T21:12:26.303 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2024-03-11T21:12:26.303 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:12:26.303 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:26.303 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:12:26.303 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6800", 2024-03-11T21:12:26.303 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 3858359434 2024-03-11T21:12:26.303 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:26.303 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:26.303 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:12:26.304 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6801", 2024-03-11T21:12:26.304 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 3858359434 2024-03-11T21:12:26.304 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:12:26.304 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:12:26.304 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:26.304 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_addrs": { 2024-03-11T21:12:26.304 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:12:26.304 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:26.304 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:12:26.304 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6802", 2024-03-11T21:12:26.304 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 3858359434 2024-03-11T21:12:26.304 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:26.304 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:26.304 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:12:26.304 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6803", 2024-03-11T21:12:26.305 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 3858359434 2024-03-11T21:12:26.305 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:12:26.305 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:12:26.305 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:26.305 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_back_addrs": { 2024-03-11T21:12:26.305 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:12:26.305 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:26.305 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:12:26.305 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6806", 2024-03-11T21:12:26.305 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 3858359434 2024-03-11T21:12:26.305 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:26.305 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:26.305 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:12:26.305 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6807", 2024-03-11T21:12:26.306 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 3858359434 2024-03-11T21:12:26.306 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:12:26.306 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:12:26.306 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:26.306 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_front_addrs": { 2024-03-11T21:12:26.306 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:12:26.306 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:26.306 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:12:26.306 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6804", 2024-03-11T21:12:26.306 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 3858359434 2024-03-11T21:12:26.306 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:26.306 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:26.306 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:12:26.306 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6805", 2024-03-11T21:12:26.307 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 3858359434 2024-03-11T21:12:26.307 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:12:26.307 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:12:26.307 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:26.307 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.146:6801/3858359434", 2024-03-11T21:12:26.307 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_addr": "172.21.15.146:6803/3858359434", 2024-03-11T21:12:26.307 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_back_addr": "172.21.15.146:6807/3858359434", 2024-03-11T21:12:26.307 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_front_addr": "172.21.15.146:6805/3858359434", 2024-03-11T21:12:26.307 INFO:teuthology.orchestra.run.smithi023.stdout: "state": [ 2024-03-11T21:12:26.307 INFO:teuthology.orchestra.run.smithi023.stdout: "exists", 2024-03-11T21:12:26.307 INFO:teuthology.orchestra.run.smithi023.stdout: "up" 2024-03-11T21:12:26.307 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:12:26.307 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:26.307 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:26.307 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": 5, 2024-03-11T21:12:26.308 INFO:teuthology.orchestra.run.smithi023.stdout: "uuid": "aefcb766-8bae-4ff4-b71e-6691c8ac02f1", 2024-03-11T21:12:26.308 INFO:teuthology.orchestra.run.smithi023.stdout: "up": 1, 2024-03-11T21:12:26.308 INFO:teuthology.orchestra.run.smithi023.stdout: "in": 1, 2024-03-11T21:12:26.308 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 1, 2024-03-11T21:12:26.308 INFO:teuthology.orchestra.run.smithi023.stdout: "primary_affinity": 1, 2024-03-11T21:12:26.308 INFO:teuthology.orchestra.run.smithi023.stdout: "last_clean_begin": 0, 2024-03-11T21:12:26.308 INFO:teuthology.orchestra.run.smithi023.stdout: "last_clean_end": 0, 2024-03-11T21:12:26.308 INFO:teuthology.orchestra.run.smithi023.stdout: "up_from": 44, 2024-03-11T21:12:26.308 INFO:teuthology.orchestra.run.smithi023.stdout: "up_thru": 380, 2024-03-11T21:12:26.308 INFO:teuthology.orchestra.run.smithi023.stdout: "down_at": 0, 2024-03-11T21:12:26.308 INFO:teuthology.orchestra.run.smithi023.stdout: "lost_at": 0, 2024-03-11T21:12:26.308 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2024-03-11T21:12:26.308 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:12:26.308 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:26.308 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:12:26.309 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6808", 2024-03-11T21:12:26.309 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 4003230814 2024-03-11T21:12:26.309 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:26.309 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:26.309 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:12:26.309 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6809", 2024-03-11T21:12:26.309 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 4003230814 2024-03-11T21:12:26.309 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:12:26.309 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:12:26.309 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:26.309 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_addrs": { 2024-03-11T21:12:26.309 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:12:26.309 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:26.309 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:12:26.310 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6810", 2024-03-11T21:12:26.310 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 4003230814 2024-03-11T21:12:26.310 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:26.310 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:26.310 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:12:26.310 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6811", 2024-03-11T21:12:26.310 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 4003230814 2024-03-11T21:12:26.310 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:12:26.310 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:12:26.310 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:26.310 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_back_addrs": { 2024-03-11T21:12:26.310 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:12:26.310 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:26.310 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:12:26.310 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6814", 2024-03-11T21:12:26.311 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 4003230814 2024-03-11T21:12:26.311 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:26.311 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:26.311 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:12:26.311 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6815", 2024-03-11T21:12:26.311 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 4003230814 2024-03-11T21:12:26.311 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:12:26.311 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:12:26.311 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:26.311 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_front_addrs": { 2024-03-11T21:12:26.311 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:12:26.311 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:26.311 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:12:26.311 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6812", 2024-03-11T21:12:26.311 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 4003230814 2024-03-11T21:12:26.312 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:26.312 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:26.312 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:12:26.312 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6813", 2024-03-11T21:12:26.312 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 4003230814 2024-03-11T21:12:26.312 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:12:26.312 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:12:26.312 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:26.312 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.146:6809/4003230814", 2024-03-11T21:12:26.312 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_addr": "172.21.15.146:6811/4003230814", 2024-03-11T21:12:26.312 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_back_addr": "172.21.15.146:6815/4003230814", 2024-03-11T21:12:26.312 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_front_addr": "172.21.15.146:6813/4003230814", 2024-03-11T21:12:26.312 INFO:teuthology.orchestra.run.smithi023.stdout: "state": [ 2024-03-11T21:12:26.312 INFO:teuthology.orchestra.run.smithi023.stdout: "exists", 2024-03-11T21:12:26.312 INFO:teuthology.orchestra.run.smithi023.stdout: "up" 2024-03-11T21:12:26.313 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:12:26.313 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:26.313 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:26.313 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": 6, 2024-03-11T21:12:26.313 INFO:teuthology.orchestra.run.smithi023.stdout: "uuid": "5cb151e8-1f9b-4617-8a86-adfb06bb9606", 2024-03-11T21:12:26.313 INFO:teuthology.orchestra.run.smithi023.stdout: "up": 1, 2024-03-11T21:12:26.313 INFO:teuthology.orchestra.run.smithi023.stdout: "in": 1, 2024-03-11T21:12:26.313 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 1, 2024-03-11T21:12:26.313 INFO:teuthology.orchestra.run.smithi023.stdout: "primary_affinity": 1, 2024-03-11T21:12:26.313 INFO:teuthology.orchestra.run.smithi023.stdout: "last_clean_begin": 0, 2024-03-11T21:12:26.313 INFO:teuthology.orchestra.run.smithi023.stdout: "last_clean_end": 0, 2024-03-11T21:12:26.313 INFO:teuthology.orchestra.run.smithi023.stdout: "up_from": 57, 2024-03-11T21:12:26.313 INFO:teuthology.orchestra.run.smithi023.stdout: "up_thru": 380, 2024-03-11T21:12:26.313 INFO:teuthology.orchestra.run.smithi023.stdout: "down_at": 0, 2024-03-11T21:12:26.313 INFO:teuthology.orchestra.run.smithi023.stdout: "lost_at": 0, 2024-03-11T21:12:26.314 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2024-03-11T21:12:26.314 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:12:26.314 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:26.314 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:12:26.314 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6816", 2024-03-11T21:12:26.314 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 3334092376 2024-03-11T21:12:26.314 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:26.314 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:26.314 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:12:26.314 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6817", 2024-03-11T21:12:26.314 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 3334092376 2024-03-11T21:12:26.314 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:12:26.314 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:12:26.314 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:26.315 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_addrs": { 2024-03-11T21:12:26.315 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:12:26.315 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:26.315 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:12:26.315 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6818", 2024-03-11T21:12:26.315 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 3334092376 2024-03-11T21:12:26.315 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:26.315 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:26.315 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:12:26.315 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6819", 2024-03-11T21:12:26.315 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 3334092376 2024-03-11T21:12:26.315 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:12:26.315 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:12:26.315 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:26.315 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_back_addrs": { 2024-03-11T21:12:26.316 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:12:26.316 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:26.316 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:12:26.316 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6822", 2024-03-11T21:12:26.316 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 3334092376 2024-03-11T21:12:26.316 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:26.316 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:26.316 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:12:26.316 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6823", 2024-03-11T21:12:26.316 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 3334092376 2024-03-11T21:12:26.316 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:12:26.316 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:12:26.316 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:26.316 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_front_addrs": { 2024-03-11T21:12:26.316 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:12:26.317 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:26.317 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:12:26.317 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6820", 2024-03-11T21:12:26.317 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 3334092376 2024-03-11T21:12:26.317 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:26.317 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:26.317 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:12:26.317 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6821", 2024-03-11T21:12:26.317 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 3334092376 2024-03-11T21:12:26.317 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:12:26.317 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:12:26.317 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:26.317 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.146:6817/3334092376", 2024-03-11T21:12:26.317 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_addr": "172.21.15.146:6819/3334092376", 2024-03-11T21:12:26.317 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_back_addr": "172.21.15.146:6823/3334092376", 2024-03-11T21:12:26.318 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_front_addr": "172.21.15.146:6821/3334092376", 2024-03-11T21:12:26.318 INFO:teuthology.orchestra.run.smithi023.stdout: "state": [ 2024-03-11T21:12:26.318 INFO:teuthology.orchestra.run.smithi023.stdout: "exists", 2024-03-11T21:12:26.318 INFO:teuthology.orchestra.run.smithi023.stdout: "up" 2024-03-11T21:12:26.318 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:12:26.318 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:26.318 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:26.318 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": 7, 2024-03-11T21:12:26.318 INFO:teuthology.orchestra.run.smithi023.stdout: "uuid": "bdbd11a2-fa09-4f82-93a2-6b2d9c750bfd", 2024-03-11T21:12:26.318 INFO:teuthology.orchestra.run.smithi023.stdout: "up": 1, 2024-03-11T21:12:26.318 INFO:teuthology.orchestra.run.smithi023.stdout: "in": 1, 2024-03-11T21:12:26.318 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 1, 2024-03-11T21:12:26.318 INFO:teuthology.orchestra.run.smithi023.stdout: "primary_affinity": 1, 2024-03-11T21:12:26.318 INFO:teuthology.orchestra.run.smithi023.stdout: "last_clean_begin": 0, 2024-03-11T21:12:26.319 INFO:teuthology.orchestra.run.smithi023.stdout: "last_clean_end": 0, 2024-03-11T21:12:26.319 INFO:teuthology.orchestra.run.smithi023.stdout: "up_from": 68, 2024-03-11T21:12:26.319 INFO:teuthology.orchestra.run.smithi023.stdout: "up_thru": 380, 2024-03-11T21:12:26.319 INFO:teuthology.orchestra.run.smithi023.stdout: "down_at": 0, 2024-03-11T21:12:26.319 INFO:teuthology.orchestra.run.smithi023.stdout: "lost_at": 0, 2024-03-11T21:12:26.319 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2024-03-11T21:12:26.319 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:12:26.319 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:26.319 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:12:26.319 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6824", 2024-03-11T21:12:26.319 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1678407923 2024-03-11T21:12:26.319 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:26.319 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:26.319 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:12:26.319 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6825", 2024-03-11T21:12:26.320 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1678407923 2024-03-11T21:12:26.320 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:12:26.320 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:12:26.320 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:26.320 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_addrs": { 2024-03-11T21:12:26.320 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:12:26.320 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:26.320 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:12:26.320 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6826", 2024-03-11T21:12:26.320 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1678407923 2024-03-11T21:12:26.320 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:26.320 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:26.320 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:12:26.320 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6827", 2024-03-11T21:12:26.320 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1678407923 2024-03-11T21:12:26.321 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:12:26.321 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:12:26.321 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:26.321 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_back_addrs": { 2024-03-11T21:12:26.321 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:12:26.321 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:26.321 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:12:26.321 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6830", 2024-03-11T21:12:26.321 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1678407923 2024-03-11T21:12:26.321 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:26.321 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:26.321 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:12:26.321 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6831", 2024-03-11T21:12:26.321 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1678407923 2024-03-11T21:12:26.321 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:12:26.321 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:12:26.322 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:26.322 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_front_addrs": { 2024-03-11T21:12:26.322 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:12:26.322 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:26.322 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:12:26.322 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6828", 2024-03-11T21:12:26.322 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1678407923 2024-03-11T21:12:26.322 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:26.322 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:26.322 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:12:26.322 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6829", 2024-03-11T21:12:26.322 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1678407923 2024-03-11T21:12:26.322 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:12:26.322 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:12:26.322 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:26.323 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.146:6825/1678407923", 2024-03-11T21:12:26.323 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_addr": "172.21.15.146:6827/1678407923", 2024-03-11T21:12:26.323 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_back_addr": "172.21.15.146:6831/1678407923", 2024-03-11T21:12:26.323 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_front_addr": "172.21.15.146:6829/1678407923", 2024-03-11T21:12:26.323 INFO:teuthology.orchestra.run.smithi023.stdout: "state": [ 2024-03-11T21:12:26.323 INFO:teuthology.orchestra.run.smithi023.stdout: "exists", 2024-03-11T21:12:26.323 INFO:teuthology.orchestra.run.smithi023.stdout: "up" 2024-03-11T21:12:26.323 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:12:26.323 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:12:26.323 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-11T21:12:26.323 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_xinfo": [ 2024-03-11T21:12:26.323 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:26.323 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": 0, 2024-03-11T21:12:26.323 INFO:teuthology.orchestra.run.smithi023.stdout: "down_stamp": "0.000000", 2024-03-11T21:12:26.324 INFO:teuthology.orchestra.run.smithi023.stdout: "laggy_probability": 0, 2024-03-11T21:12:26.324 INFO:teuthology.orchestra.run.smithi023.stdout: "laggy_interval": 0, 2024-03-11T21:12:26.324 INFO:teuthology.orchestra.run.smithi023.stdout: "features": 4540701547738038271, 2024-03-11T21:12:26.324 INFO:teuthology.orchestra.run.smithi023.stdout: "old_weight": 0, 2024-03-11T21:12:26.324 INFO:teuthology.orchestra.run.smithi023.stdout: "last_purged_snaps_scrub": "2024-03-11T21:12:04.296272+0000", 2024-03-11T21:12:26.324 INFO:teuthology.orchestra.run.smithi023.stdout: "dead_epoch": 0 2024-03-11T21:12:26.324 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:26.324 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:26.324 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": 1, 2024-03-11T21:12:26.324 INFO:teuthology.orchestra.run.smithi023.stdout: "down_stamp": "0.000000", 2024-03-11T21:12:26.324 INFO:teuthology.orchestra.run.smithi023.stdout: "laggy_probability": 0, 2024-03-11T21:12:26.324 INFO:teuthology.orchestra.run.smithi023.stdout: "laggy_interval": 0, 2024-03-11T21:12:26.324 INFO:teuthology.orchestra.run.smithi023.stdout: "features": 4540701547738038271, 2024-03-11T21:12:26.324 INFO:teuthology.orchestra.run.smithi023.stdout: "old_weight": 0, 2024-03-11T21:12:26.324 INFO:teuthology.orchestra.run.smithi023.stdout: "last_purged_snaps_scrub": "2024-03-11T21:12:09.695362+0000", 2024-03-11T21:12:26.325 INFO:teuthology.orchestra.run.smithi023.stdout: "dead_epoch": 173 2024-03-11T21:12:26.325 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:26.325 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:26.325 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": 2, 2024-03-11T21:12:26.325 INFO:teuthology.orchestra.run.smithi023.stdout: "down_stamp": "0.000000", 2024-03-11T21:12:26.325 INFO:teuthology.orchestra.run.smithi023.stdout: "laggy_probability": 0, 2024-03-11T21:12:26.325 INFO:teuthology.orchestra.run.smithi023.stdout: "laggy_interval": 0, 2024-03-11T21:12:26.325 INFO:teuthology.orchestra.run.smithi023.stdout: "features": 4540701547738038271, 2024-03-11T21:12:26.325 INFO:teuthology.orchestra.run.smithi023.stdout: "old_weight": 0, 2024-03-11T21:12:26.325 INFO:teuthology.orchestra.run.smithi023.stdout: "last_purged_snaps_scrub": "2024-03-11T21:12:08.636738+0000", 2024-03-11T21:12:26.325 INFO:teuthology.orchestra.run.smithi023.stdout: "dead_epoch": 0 2024-03-11T21:12:26.325 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:26.325 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:26.325 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": 3, 2024-03-11T21:12:26.326 INFO:teuthology.orchestra.run.smithi023.stdout: "down_stamp": "0.000000", 2024-03-11T21:12:26.326 INFO:teuthology.orchestra.run.smithi023.stdout: "laggy_probability": 0, 2024-03-11T21:12:26.326 INFO:teuthology.orchestra.run.smithi023.stdout: "laggy_interval": 0, 2024-03-11T21:12:26.326 INFO:teuthology.orchestra.run.smithi023.stdout: "features": 4540701547738038271, 2024-03-11T21:12:26.326 INFO:teuthology.orchestra.run.smithi023.stdout: "old_weight": 0, 2024-03-11T21:12:26.326 INFO:teuthology.orchestra.run.smithi023.stdout: "last_purged_snaps_scrub": "2024-03-11T21:11:42.955065+0000", 2024-03-11T21:12:26.326 INFO:teuthology.orchestra.run.smithi023.stdout: "dead_epoch": 0 2024-03-11T21:12:26.326 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:26.326 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:26.326 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": 4, 2024-03-11T21:12:26.326 INFO:teuthology.orchestra.run.smithi023.stdout: "down_stamp": "0.000000", 2024-03-11T21:12:26.326 INFO:teuthology.orchestra.run.smithi023.stdout: "laggy_probability": 0, 2024-03-11T21:12:26.326 INFO:teuthology.orchestra.run.smithi023.stdout: "laggy_interval": 0, 2024-03-11T21:12:26.326 INFO:teuthology.orchestra.run.smithi023.stdout: "features": 4540701547738038271, 2024-03-11T21:12:26.327 INFO:teuthology.orchestra.run.smithi023.stdout: "old_weight": 0, 2024-03-11T21:12:26.327 INFO:teuthology.orchestra.run.smithi023.stdout: "last_purged_snaps_scrub": "2024-03-11T21:12:17.082552+0000", 2024-03-11T21:12:26.327 INFO:teuthology.orchestra.run.smithi023.stdout: "dead_epoch": 182 2024-03-11T21:12:26.327 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:26.327 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:26.327 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": 5, 2024-03-11T21:12:26.327 INFO:teuthology.orchestra.run.smithi023.stdout: "down_stamp": "0.000000", 2024-03-11T21:12:26.327 INFO:teuthology.orchestra.run.smithi023.stdout: "laggy_probability": 0, 2024-03-11T21:12:26.327 INFO:teuthology.orchestra.run.smithi023.stdout: "laggy_interval": 0, 2024-03-11T21:12:26.327 INFO:teuthology.orchestra.run.smithi023.stdout: "features": 4540701547738038271, 2024-03-11T21:12:26.327 INFO:teuthology.orchestra.run.smithi023.stdout: "old_weight": 0, 2024-03-11T21:12:26.327 INFO:teuthology.orchestra.run.smithi023.stdout: "last_purged_snaps_scrub": "2024-03-11T21:11:29.695143+0000", 2024-03-11T21:12:26.327 INFO:teuthology.orchestra.run.smithi023.stdout: "dead_epoch": 0 2024-03-11T21:12:26.327 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:26.327 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:26.328 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": 6, 2024-03-11T21:12:26.328 INFO:teuthology.orchestra.run.smithi023.stdout: "down_stamp": "0.000000", 2024-03-11T21:12:26.328 INFO:teuthology.orchestra.run.smithi023.stdout: "laggy_probability": 0, 2024-03-11T21:12:26.328 INFO:teuthology.orchestra.run.smithi023.stdout: "laggy_interval": 0, 2024-03-11T21:12:26.328 INFO:teuthology.orchestra.run.smithi023.stdout: "features": 4540701547738038271, 2024-03-11T21:12:26.328 INFO:teuthology.orchestra.run.smithi023.stdout: "old_weight": 0, 2024-03-11T21:12:26.328 INFO:teuthology.orchestra.run.smithi023.stdout: "last_purged_snaps_scrub": "2024-03-11T21:11:11.806735+0000", 2024-03-11T21:12:26.328 INFO:teuthology.orchestra.run.smithi023.stdout: "dead_epoch": 0 2024-03-11T21:12:26.328 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:26.328 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:26.328 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": 7, 2024-03-11T21:12:26.328 INFO:teuthology.orchestra.run.smithi023.stdout: "down_stamp": "0.000000", 2024-03-11T21:12:26.328 INFO:teuthology.orchestra.run.smithi023.stdout: "laggy_probability": 0, 2024-03-11T21:12:26.328 INFO:teuthology.orchestra.run.smithi023.stdout: "laggy_interval": 0, 2024-03-11T21:12:26.328 INFO:teuthology.orchestra.run.smithi023.stdout: "features": 4540701547738038271, 2024-03-11T21:12:26.328 INFO:teuthology.orchestra.run.smithi023.stdout: "old_weight": 0, 2024-03-11T21:12:26.329 INFO:teuthology.orchestra.run.smithi023.stdout: "last_purged_snaps_scrub": "2024-03-11T21:11:42.713660+0000", 2024-03-11T21:12:26.329 INFO:teuthology.orchestra.run.smithi023.stdout: "dead_epoch": 0 2024-03-11T21:12:26.329 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:12:26.329 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-11T21:12:26.329 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_upmap": [ 2024-03-11T21:12:26.329 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:26.329 INFO:teuthology.orchestra.run.smithi023.stdout: "pgid": "6.a", 2024-03-11T21:12:26.329 INFO:teuthology.orchestra.run.smithi023.stdout: "osds": [ 2024-03-11T21:12:26.329 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2024-03-11T21:12:26.329 INFO:teuthology.orchestra.run.smithi023.stdout: 4, 2024-03-11T21:12:26.329 INFO:teuthology.orchestra.run.smithi023.stdout: 7 2024-03-11T21:12:26.329 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:12:26.329 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:12:26.329 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-11T21:12:26.329 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_upmap_items": [ 2024-03-11T21:12:26.330 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:26.330 INFO:teuthology.orchestra.run.smithi023.stdout: "pgid": "2.0", 2024-03-11T21:12:26.330 INFO:teuthology.orchestra.run.smithi023.stdout: "mappings": [ 2024-03-11T21:12:26.330 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:26.330 INFO:teuthology.orchestra.run.smithi023.stdout: "from": 7, 2024-03-11T21:12:26.330 INFO:teuthology.orchestra.run.smithi023.stdout: "to": 2 2024-03-11T21:12:26.330 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:12:26.330 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:12:26.330 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:26.330 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:26.330 INFO:teuthology.orchestra.run.smithi023.stdout: "pgid": "4.b", 2024-03-11T21:12:26.330 INFO:teuthology.orchestra.run.smithi023.stdout: "mappings": [ 2024-03-11T21:12:26.330 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:26.330 INFO:teuthology.orchestra.run.smithi023.stdout: "from": 1, 2024-03-11T21:12:26.331 INFO:teuthology.orchestra.run.smithi023.stdout: "to": 2 2024-03-11T21:12:26.331 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:12:26.331 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:12:26.331 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:26.331 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:26.331 INFO:teuthology.orchestra.run.smithi023.stdout: "pgid": "4.d", 2024-03-11T21:12:26.331 INFO:teuthology.orchestra.run.smithi023.stdout: "mappings": [ 2024-03-11T21:12:26.331 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:26.331 INFO:teuthology.orchestra.run.smithi023.stdout: "from": 1, 2024-03-11T21:12:26.331 INFO:teuthology.orchestra.run.smithi023.stdout: "to": 5 2024-03-11T21:12:26.331 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:12:26.331 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:12:26.331 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:26.331 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:26.331 INFO:teuthology.orchestra.run.smithi023.stdout: "pgid": "4.e", 2024-03-11T21:12:26.331 INFO:teuthology.orchestra.run.smithi023.stdout: "mappings": [ 2024-03-11T21:12:26.332 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:26.332 INFO:teuthology.orchestra.run.smithi023.stdout: "from": 1, 2024-03-11T21:12:26.332 INFO:teuthology.orchestra.run.smithi023.stdout: "to": 2 2024-03-11T21:12:26.332 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:12:26.332 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:12:26.332 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:26.332 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:26.332 INFO:teuthology.orchestra.run.smithi023.stdout: "pgid": "4.16", 2024-03-11T21:12:26.332 INFO:teuthology.orchestra.run.smithi023.stdout: "mappings": [ 2024-03-11T21:12:26.332 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:26.332 INFO:teuthology.orchestra.run.smithi023.stdout: "from": 3, 2024-03-11T21:12:26.332 INFO:teuthology.orchestra.run.smithi023.stdout: "to": 2 2024-03-11T21:12:26.332 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:12:26.332 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:12:26.333 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:26.333 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:26.333 INFO:teuthology.orchestra.run.smithi023.stdout: "pgid": "4.1d", 2024-03-11T21:12:26.333 INFO:teuthology.orchestra.run.smithi023.stdout: "mappings": [ 2024-03-11T21:12:26.333 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:26.333 INFO:teuthology.orchestra.run.smithi023.stdout: "from": 4, 2024-03-11T21:12:26.333 INFO:teuthology.orchestra.run.smithi023.stdout: "to": 2 2024-03-11T21:12:26.333 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:12:26.333 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:12:26.333 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:26.333 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:26.333 INFO:teuthology.orchestra.run.smithi023.stdout: "pgid": "6.a", 2024-03-11T21:12:26.333 INFO:teuthology.orchestra.run.smithi023.stdout: "mappings": [ 2024-03-11T21:12:26.333 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:26.334 INFO:teuthology.orchestra.run.smithi023.stdout: "from": 6, 2024-03-11T21:12:26.334 INFO:teuthology.orchestra.run.smithi023.stdout: "to": 1 2024-03-11T21:12:26.334 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:12:26.334 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:12:26.334 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:12:26.334 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-11T21:12:26.334 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_upmap_primaries": [], 2024-03-11T21:12:26.334 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_temp": [], 2024-03-11T21:12:26.334 INFO:teuthology.orchestra.run.smithi023.stdout: "primary_temp": [], 2024-03-11T21:12:26.334 INFO:teuthology.orchestra.run.smithi023.stdout: "blocklist": { 2024-03-11T21:12:26.334 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:0/1618334186": "2024-03-12T20:40:45.361537+0000", 2024-03-11T21:12:26.334 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:6801/2813578081": "2024-03-12T20:40:45.361537+0000", 2024-03-11T21:12:26.334 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:0/2869940180": "2024-03-12T20:51:33.790205+0000", 2024-03-11T21:12:26.334 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:6800/3037564930": "2024-03-12T20:51:33.790205+0000", 2024-03-11T21:12:26.335 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:0/3297567084": "2024-03-12T20:51:33.790205+0000", 2024-03-11T21:12:26.335 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:0/3708753659": "2024-03-12T20:40:45.361537+0000", 2024-03-11T21:12:26.335 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:0/2004146740": "2024-03-12T20:40:45.361537+0000", 2024-03-11T21:12:26.335 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:0/827653812": "2024-03-12T20:41:21.301516+0000", 2024-03-11T21:12:26.335 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:6800/1057699850": "2024-03-12T20:41:21.301516+0000", 2024-03-11T21:12:26.335 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:6800/2813578081": "2024-03-12T20:40:45.361537+0000", 2024-03-11T21:12:26.335 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:6801/1057699850": "2024-03-12T20:41:21.301516+0000", 2024-03-11T21:12:26.335 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:0/3833230375": "2024-03-12T20:41:21.301516+0000", 2024-03-11T21:12:26.335 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:6801/3037564930": "2024-03-12T20:51:33.790205+0000", 2024-03-11T21:12:26.335 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:0/1182809102": "2024-03-12T20:51:33.790205+0000", 2024-03-11T21:12:26.335 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:0/2122134187": "2024-03-12T20:51:33.790205+0000", 2024-03-11T21:12:26.335 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:0/2356701899": "2024-03-12T20:41:21.301516+0000", 2024-03-11T21:12:26.336 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:0/843974100": "2024-03-12T20:51:33.790205+0000" 2024-03-11T21:12:26.336 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:26.336 INFO:teuthology.orchestra.run.smithi023.stdout: "range_blocklist": {}, 2024-03-11T21:12:26.336 INFO:teuthology.orchestra.run.smithi023.stdout: "erasure_code_profiles": { 2024-03-11T21:12:26.336 INFO:teuthology.orchestra.run.smithi023.stdout: "default": { 2024-03-11T21:12:26.336 INFO:teuthology.orchestra.run.smithi023.stdout: "crush-failure-domain": "osd", 2024-03-11T21:12:26.336 INFO:teuthology.orchestra.run.smithi023.stdout: "k": "2", 2024-03-11T21:12:26.336 INFO:teuthology.orchestra.run.smithi023.stdout: "m": "1", 2024-03-11T21:12:26.336 INFO:teuthology.orchestra.run.smithi023.stdout: "plugin": "jerasure", 2024-03-11T21:12:26.336 INFO:teuthology.orchestra.run.smithi023.stdout: "technique": "reed_sol_van" 2024-03-11T21:12:26.336 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:12:26.336 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:26.336 INFO:teuthology.orchestra.run.smithi023.stdout: "removed_snaps_queue": [], 2024-03-11T21:12:26.336 INFO:teuthology.orchestra.run.smithi023.stdout: "new_removed_snaps": [], 2024-03-11T21:12:26.336 INFO:teuthology.orchestra.run.smithi023.stdout: "new_purged_snaps": [], 2024-03-11T21:12:26.336 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_node_flags": {}, 2024-03-11T21:12:26.337 INFO:teuthology.orchestra.run.smithi023.stdout: "device_class_flags": {}, 2024-03-11T21:12:26.337 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": { 2024-03-11T21:12:26.337 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode_enabled": false, 2024-03-11T21:12:26.337 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_bucket_count": 0, 2024-03-11T21:12:26.337 INFO:teuthology.orchestra.run.smithi023.stdout: "degraded_stretch_mode": 0, 2024-03-11T21:12:26.337 INFO:teuthology.orchestra.run.smithi023.stdout: "recovering_stretch_mode": 0, 2024-03-11T21:12:26.337 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode_bucket": 0 2024-03-11T21:12:26.337 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:12:26.337 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-11T21:12:26.337 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:12:26.338 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:12:25 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:12:26.366 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:12:26.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:26.370 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:12:26.470 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:26.498 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:26.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:26.502 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:12:26.603 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:12:26.630 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:12:26.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:26.635 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:12:26.736 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:12:26.756 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:26 smithi146 bash[23809]: cluster 2024-03-11T21:12:25.080772+0000 osd.7 (osd.7) 515 : cluster 0 2.8 scrub starts 2024-03-11T21:12:26.757 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:26 smithi146 bash[23809]: cluster 2024-03-11T21:12:25.082730+0000 osd.7 (osd.7) 516 : cluster 0 2.8 scrub ok 2024-03-11T21:12:26.757 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:26 smithi146 bash[23809]: audit 2024-03-11T21:12:26.239662+0000 mon.c (mon.2) 94 : audit 0 from='client.? 172.21.15.23:0/3946458650' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-11T21:12:26.760 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:12:26.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:26.762 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:12:26.833 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 22 16 38301 38285 108.73 88.375 0.0168372 0.00919101 2024-03-11T21:12:26.866 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:12:26.890 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:12:26.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:26.892 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:12:26.993 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:27.023 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:27.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:27.025 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:12:27.026 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:26 smithi023 bash[21372]: cluster 2024-03-11T21:12:25.080772+0000 osd.7 (osd.7) 515 : cluster 0 2.8 scrub starts 2024-03-11T21:12:27.026 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:26 smithi023 bash[21372]: cluster 2024-03-11T21:12:25.082730+0000 osd.7 (osd.7) 516 : cluster 0 2.8 scrub ok 2024-03-11T21:12:27.027 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:26 smithi023 bash[21372]: audit 2024-03-11T21:12:26.239662+0000 mon.c (mon.2) 94 : audit 0 from='client.? 172.21.15.23:0/3946458650' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-11T21:12:27.027 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:26 smithi023 bash[29146]: cluster 2024-03-11T21:12:25.080772+0000 osd.7 (osd.7) 515 : cluster 0 2.8 scrub starts 2024-03-11T21:12:27.027 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:26 smithi023 bash[29146]: cluster 2024-03-11T21:12:25.082730+0000 osd.7 (osd.7) 516 : cluster 0 2.8 scrub ok 2024-03-11T21:12:27.027 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:26 smithi023 bash[29146]: audit 2024-03-11T21:12:26.239662+0000 mon.c (mon.2) 94 : audit 0 from='client.? 172.21.15.23:0/3946458650' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-11T21:12:27.126 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:27.155 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:27.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:27.158 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:12:27.258 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:12:27.283 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:12:27.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:27.285 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:12:27.386 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:27.410 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:27.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:27.413 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:12:27.514 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:27.547 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:27.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:27.551 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:12:27.652 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:12:27.680 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:12:27.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:27.682 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:12:27.783 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:12:27.812 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:12:27.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:27.813 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:12:27.833 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 23 16 39446 39430 107.114 71.5625 0.00420025 0.00932411 2024-03-11T21:12:27.914 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:12:27.940 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:12:27.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:27.942 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:12:27.946 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:27 smithi023 bash[21372]: audit 2024-03-11T21:12:25.929027+0000 mgr.y (mgr.24380) 801 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:12:27.946 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:27 smithi023 bash[21372]: cluster 2024-03-11T21:12:26.011851+0000 osd.0 (osd.0) 372 : cluster 0 2.1f scrub starts 2024-03-11T21:12:27.946 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:27 smithi023 bash[21372]: cluster 2024-03-11T21:12:26.013226+0000 osd.0 (osd.0) 373 : cluster 0 2.1f scrub ok 2024-03-11T21:12:27.946 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:27 smithi023 bash[21372]: cluster 2024-03-11T21:12:26.372899+0000 mgr.y (mgr.24380) 802 : cluster 0 pgmap v919: 158 pgs: 158 active+clean; 2.1 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 1.1 KiB/s rd, 95 MiB/s wr, 1.51k op/s 2024-03-11T21:12:27.946 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:27 smithi023 bash[29146]: audit 2024-03-11T21:12:25.929027+0000 mgr.y (mgr.24380) 801 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:12:27.947 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:27 smithi023 bash[29146]: cluster 2024-03-11T21:12:26.011851+0000 osd.0 (osd.0) 372 : cluster 0 2.1f scrub starts 2024-03-11T21:12:27.947 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:27 smithi023 bash[29146]: cluster 2024-03-11T21:12:26.013226+0000 osd.0 (osd.0) 373 : cluster 0 2.1f scrub ok 2024-03-11T21:12:27.947 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:27 smithi023 bash[29146]: cluster 2024-03-11T21:12:26.372899+0000 mgr.y (mgr.24380) 802 : cluster 0 pgmap v919: 158 pgs: 158 active+clean; 2.1 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 1.1 KiB/s rd, 95 MiB/s wr, 1.51k op/s 2024-03-11T21:12:28.007 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:27 smithi146 bash[23809]: audit 2024-03-11T21:12:25.929027+0000 mgr.y (mgr.24380) 801 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:12:28.007 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:27 smithi146 bash[23809]: cluster 2024-03-11T21:12:26.011851+0000 osd.0 (osd.0) 372 : cluster 0 2.1f scrub starts 2024-03-11T21:12:28.007 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:27 smithi146 bash[23809]: cluster 2024-03-11T21:12:26.013226+0000 osd.0 (osd.0) 373 : cluster 0 2.1f scrub ok 2024-03-11T21:12:28.007 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:27 smithi146 bash[23809]: cluster 2024-03-11T21:12:26.372899+0000 mgr.y (mgr.24380) 802 : cluster 0 pgmap v919: 158 pgs: 158 active+clean; 2.1 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 1.1 KiB/s rd, 95 MiB/s wr, 1.51k op/s 2024-03-11T21:12:28.043 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:12:28.071 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:12:28.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:28.074 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:12:28.175 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:12:28.192 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:12:28.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:28.194 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:12:28.298 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:12:28.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:28.330 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:12:28.331 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:12:28.431 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:28.458 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:28.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:28.462 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:12:28.563 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:12:28.573 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:12:28 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:12:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:12:28.591 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:12:28.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:28.596 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:12:28.697 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:12:28.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:28 smithi146 bash[23809]: cluster 2024-03-11T21:12:26.993332+0000 osd.0 (osd.0) 374 : cluster 0 3.12 scrub starts 2024-03-11T21:12:28.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:28 smithi146 bash[23809]: cluster 2024-03-11T21:12:27.006357+0000 osd.1 (osd.1) 281 : cluster 0 3.1a scrub starts 2024-03-11T21:12:28.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:28 smithi146 bash[23809]: cluster 2024-03-11T21:12:27.006367+0000 osd.0 (osd.0) 375 : cluster 0 3.12 scrub ok 2024-03-11T21:12:28.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:28 smithi146 bash[23809]: cluster 2024-03-11T21:12:27.010000+0000 osd.1 (osd.1) 282 : cluster 0 3.1a scrub ok 2024-03-11T21:12:28.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:28 smithi146 bash[23809]: cluster 2024-03-11T21:12:27.121090+0000 osd.7 (osd.7) 517 : cluster 0 6.15 scrub starts 2024-03-11T21:12:28.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:28 smithi146 bash[23809]: cluster 2024-03-11T21:12:27.179421+0000 osd.7 (osd.7) 518 : cluster 0 6.15 scrub ok 2024-03-11T21:12:28.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:28 smithi146 bash[23809]: cluster 2024-03-11T21:12:28.473043+0000 osd.5 (osd.5) 471 : cluster 0 6.d scrub starts 2024-03-11T21:12:28.727 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:12:28.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:28.730 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:12:28.777 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:28 smithi023 bash[21372]: cluster 2024-03-11T21:12:26.993332+0000 osd.0 (osd.0) 374 : cluster 0 3.12 scrub starts 2024-03-11T21:12:28.777 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:28 smithi023 bash[21372]: cluster 2024-03-11T21:12:27.006357+0000 osd.1 (osd.1) 281 : cluster 0 3.1a scrub starts 2024-03-11T21:12:28.777 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:28 smithi023 bash[21372]: cluster 2024-03-11T21:12:27.006367+0000 osd.0 (osd.0) 375 : cluster 0 3.12 scrub ok 2024-03-11T21:12:28.777 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:28 smithi023 bash[21372]: cluster 2024-03-11T21:12:27.010000+0000 osd.1 (osd.1) 282 : cluster 0 3.1a scrub ok 2024-03-11T21:12:28.777 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:28 smithi023 bash[21372]: cluster 2024-03-11T21:12:27.121090+0000 osd.7 (osd.7) 517 : cluster 0 6.15 scrub starts 2024-03-11T21:12:28.777 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:28 smithi023 bash[21372]: cluster 2024-03-11T21:12:27.179421+0000 osd.7 (osd.7) 518 : cluster 0 6.15 scrub ok 2024-03-11T21:12:28.777 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:28 smithi023 bash[21372]: cluster 2024-03-11T21:12:28.473043+0000 osd.5 (osd.5) 471 : cluster 0 6.d scrub starts 2024-03-11T21:12:28.777 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:28 smithi023 bash[29146]: cluster 2024-03-11T21:12:26.993332+0000 osd.0 (osd.0) 374 : cluster 0 3.12 scrub starts 2024-03-11T21:12:28.778 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:28 smithi023 bash[29146]: cluster 2024-03-11T21:12:27.006357+0000 osd.1 (osd.1) 281 : cluster 0 3.1a scrub starts 2024-03-11T21:12:28.778 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:28 smithi023 bash[29146]: cluster 2024-03-11T21:12:27.006367+0000 osd.0 (osd.0) 375 : cluster 0 3.12 scrub ok 2024-03-11T21:12:28.778 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:28 smithi023 bash[29146]: cluster 2024-03-11T21:12:27.010000+0000 osd.1 (osd.1) 282 : cluster 0 3.1a scrub ok 2024-03-11T21:12:28.778 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:28 smithi023 bash[29146]: cluster 2024-03-11T21:12:27.121090+0000 osd.7 (osd.7) 517 : cluster 0 6.15 scrub starts 2024-03-11T21:12:28.778 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:28 smithi023 bash[29146]: cluster 2024-03-11T21:12:27.179421+0000 osd.7 (osd.7) 518 : cluster 0 6.15 scrub ok 2024-03-11T21:12:28.778 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:28 smithi023 bash[29146]: cluster 2024-03-11T21:12:28.473043+0000 osd.5 (osd.5) 471 : cluster 0 6.d scrub starts 2024-03-11T21:12:28.830 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:12:28.833 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 24 15 40765 40750 106.088 82.5 0.0143265 0.00941974 2024-03-11T21:12:28.863 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:12:28.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:28.867 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:12:28.968 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:12:28.991 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:12:28.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:28.993 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:12:29.094 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:12:29.119 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:12:29.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:29.121 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:12:29.222 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:12:29.247 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:12:29.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:29.252 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:12:29.353 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:12:29.370 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-03-11T21:12:29.377 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:12:29.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:29.381 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:12:29.482 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:12:29.505 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:12:29.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:29.510 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:12:29.610 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:29.636 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:29.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:29.644 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:12:29.742 INFO:tasks.thrashosds.thrasher:j is {'epoch': 391, 'fsid': '67ba00f8-dfe7-11ee-95c8-87774f69a715', 'created': '2024-03-11T20:40:13.664802+0000', 'modified': '2024-03-11T21:12:18.593416+0000', 'last_up_change': '2024-03-11T21:00:47.324817+0000', 'last_in_change': '2024-03-11T20:57:23.560527+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 20, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 11, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-11T20:45:31.964570+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 11, 'pg_placement_num': 11, 'pg_placement_num_target': 11, 'pg_num_target': 11, 'pg_num_pending': 11, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '369', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '367', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6399999856948853, 'score_stable': 1.6399999856948853, 'optimal_score': 1.3300000429153442, 'raw_score_acting': 2.180000066757202, 'raw_score_stable': 2.180000066757202, 'primary_affinity_weighted': 0.75, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.75}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2024-03-11T20:50:26.421486+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '78', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1.3300000429153442, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.75, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.75}}, {'pool': 3, 'pool_name': 'default.rgw.log', 'create_time': '2024-03-11T20:50:28.592615+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1.3300000429153442, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.75, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.75}}, {'pool': 4, 'pool_name': 'default.rgw.control', 'create_time': '2024-03-11T20:50:30.536371+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1.3300000429153442, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 0.75, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.75}}, {'pool': 5, 'pool_name': 'datapool', 'create_time': '2024-03-11T20:50:31.135845+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 87, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3.990000009536743, 'score_stable': 3.990000009536743, 'optimal_score': 1.1699999570846558, 'raw_score_acting': 4.670000076293945, 'raw_score_stable': 4.670000076293945, 'primary_affinity_weighted': 0.75, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.75}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-03-11T20:50:33.233163+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.690000057220459, 'score_stable': 1.690000057220459, 'optimal_score': 1.3300000429153442, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.75, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.75}}, {'pool': 11, 'pool_name': 'unique_pool_4', 'create_time': '2024-03-11T21:11:46.486253+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '386', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1.3300000429153442, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 0.75, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.75}}], 'osds': [{'osd': 0, 'uuid': '3df72b4a-8fe4-4265-b4c6-af022e1e23e1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 380, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6802', 'nonce': 2154627393}, {'type': 'v1', 'addr': '172.21.15.23:6803', 'nonce': 2154627393}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6804', 'nonce': 2154627393}, {'type': 'v1', 'addr': '172.21.15.23:6805', 'nonce': 2154627393}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6808', 'nonce': 2154627393}, {'type': 'v1', 'addr': '172.21.15.23:6809', 'nonce': 2154627393}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6806', 'nonce': 2154627393}, {'type': 'v1', 'addr': '172.21.15.23:6807', 'nonce': 2154627393}]}, 'public_addr': '172.21.15.23:6803/2154627393', 'cluster_addr': '172.21.15.23:6805/2154627393', 'heartbeat_back_addr': '172.21.15.23:6809/2154627393', 'heartbeat_front_addr': '172.21.15.23:6807/2154627393', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'e8fd7500-9081-41a1-8077-a4d0f0e12a2c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 13, 'last_clean_end': 173, 'up_from': 211, 'up_thru': 380, 'down_at': 174, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6810', 'nonce': 56585354}, {'type': 'v1', 'addr': '172.21.15.23:6811', 'nonce': 56585354}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6812', 'nonce': 56585354}, {'type': 'v1', 'addr': '172.21.15.23:6813', 'nonce': 56585354}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6816', 'nonce': 56585354}, {'type': 'v1', 'addr': '172.21.15.23:6817', 'nonce': 56585354}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6814', 'nonce': 56585354}, {'type': 'v1', 'addr': '172.21.15.23:6815', 'nonce': 56585354}]}, 'public_addr': '172.21.15.23:6811/56585354', 'cluster_addr': '172.21.15.23:6813/56585354', 'heartbeat_back_addr': '172.21.15.23:6817/56585354', 'heartbeat_front_addr': '172.21.15.23:6815/56585354', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'dd85d52a-df48-4b03-8612-cecfd82b5987', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 19, 'up_thru': 306, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6818', 'nonce': 1770748056}, {'type': 'v1', 'addr': '172.21.15.23:6819', 'nonce': 1770748056}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6820', 'nonce': 1770748056}, {'type': 'v1', 'addr': '172.21.15.23:6821', 'nonce': 1770748056}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6824', 'nonce': 1770748056}, {'type': 'v1', 'addr': '172.21.15.23:6825', 'nonce': 1770748056}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6822', 'nonce': 1770748056}, {'type': 'v1', 'addr': '172.21.15.23:6823', 'nonce': 1770748056}]}, 'public_addr': '172.21.15.23:6819/1770748056', 'cluster_addr': '172.21.15.23:6821/1770748056', 'heartbeat_back_addr': '172.21.15.23:6825/1770748056', 'heartbeat_front_addr': '172.21.15.23:6823/1770748056', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'e3021106-3779-4b94-9bbf-a7724544110f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 28, 'up_thru': 380, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6826', 'nonce': 938496814}, {'type': 'v1', 'addr': '172.21.15.23:6827', 'nonce': 938496814}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6828', 'nonce': 938496814}, {'type': 'v1', 'addr': '172.21.15.23:6829', 'nonce': 938496814}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6832', 'nonce': 938496814}, {'type': 'v1', 'addr': '172.21.15.23:6833', 'nonce': 938496814}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6830', 'nonce': 938496814}, {'type': 'v1', 'addr': '172.21.15.23:6831', 'nonce': 938496814}]}, 'public_addr': '172.21.15.23:6827/938496814', 'cluster_addr': '172.21.15.23:6829/938496814', 'heartbeat_back_addr': '172.21.15.23:6833/938496814', 'heartbeat_front_addr': '172.21.15.23:6831/938496814', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '4dcbc91e-7035-413d-97bc-c9d804eb16d7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 35, 'last_clean_end': 182, 'up_from': 204, 'up_thru': 248, 'down_at': 183, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6800', 'nonce': 3858359434}, {'type': 'v1', 'addr': '172.21.15.146:6801', 'nonce': 3858359434}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6802', 'nonce': 3858359434}, {'type': 'v1', 'addr': '172.21.15.146:6803', 'nonce': 3858359434}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6806', 'nonce': 3858359434}, {'type': 'v1', 'addr': '172.21.15.146:6807', 'nonce': 3858359434}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6804', 'nonce': 3858359434}, {'type': 'v1', 'addr': '172.21.15.146:6805', 'nonce': 3858359434}]}, 'public_addr': '172.21.15.146:6801/3858359434', 'cluster_addr': '172.21.15.146:6803/3858359434', 'heartbeat_back_addr': '172.21.15.146:6807/3858359434', 'heartbeat_front_addr': '172.21.15.146:6805/3858359434', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'aefcb766-8bae-4ff4-b71e-6691c8ac02f1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 44, 'up_thru': 380, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6808', 'nonce': 4003230814}, {'type': 'v1', 'addr': '172.21.15.146:6809', 'nonce': 4003230814}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6810', 'nonce': 4003230814}, {'type': 'v1', 'addr': '172.21.15.146:6811', 'nonce': 4003230814}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6814', 'nonce': 4003230814}, {'type': 'v1', 'addr': '172.21.15.146:6815', 'nonce': 4003230814}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6812', 'nonce': 4003230814}, {'type': 'v1', 'addr': '172.21.15.146:6813', 'nonce': 4003230814}]}, 'public_addr': '172.21.15.146:6809/4003230814', 'cluster_addr': '172.21.15.146:6811/4003230814', 'heartbeat_back_addr': '172.21.15.146:6815/4003230814', 'heartbeat_front_addr': '172.21.15.146:6813/4003230814', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '5cb151e8-1f9b-4617-8a86-adfb06bb9606', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 57, 'up_thru': 380, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6816', 'nonce': 3334092376}, {'type': 'v1', 'addr': '172.21.15.146:6817', 'nonce': 3334092376}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6818', 'nonce': 3334092376}, {'type': 'v1', 'addr': '172.21.15.146:6819', 'nonce': 3334092376}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6822', 'nonce': 3334092376}, {'type': 'v1', 'addr': '172.21.15.146:6823', 'nonce': 3334092376}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6820', 'nonce': 3334092376}, {'type': 'v1', 'addr': '172.21.15.146:6821', 'nonce': 3334092376}]}, 'public_addr': '172.21.15.146:6817/3334092376', 'cluster_addr': '172.21.15.146:6819/3334092376', 'heartbeat_back_addr': '172.21.15.146:6823/3334092376', 'heartbeat_front_addr': '172.21.15.146:6821/3334092376', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'bdbd11a2-fa09-4f82-93a2-6b2d9c750bfd', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 68, 'up_thru': 380, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6824', 'nonce': 1678407923}, {'type': 'v1', 'addr': '172.21.15.146:6825', 'nonce': 1678407923}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6826', 'nonce': 1678407923}, {'type': 'v1', 'addr': '172.21.15.146:6827', 'nonce': 1678407923}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6830', 'nonce': 1678407923}, {'type': 'v1', 'addr': '172.21.15.146:6831', 'nonce': 1678407923}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6828', 'nonce': 1678407923}, {'type': 'v1', 'addr': '172.21.15.146:6829', 'nonce': 1678407923}]}, 'public_addr': '172.21.15.146:6825/1678407923', 'cluster_addr': '172.21.15.146:6827/1678407923', 'heartbeat_back_addr': '172.21.15.146:6831/1678407923', 'heartbeat_front_addr': '172.21.15.146:6829/1678407923', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T21:12:04.296272+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T21:12:09.695362+0000', 'dead_epoch': 173}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T21:12:08.636738+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T21:11:42.955065+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T21:12:17.082552+0000', 'dead_epoch': 182}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T21:11:29.695143+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T21:11:11.806735+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T21:11:42.713660+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '6.a', 'osds': [0, 4, 7]}], 'pg_upmap_items': [{'pgid': '2.0', 'mappings': [{'from': 7, 'to': 2}]}, {'pgid': '4.b', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.d', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.e', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.16', 'mappings': [{'from': 3, 'to': 2}]}, {'pgid': '4.1d', 'mappings': [{'from': 4, 'to': 2}]}, {'pgid': '6.a', 'mappings': [{'from': 6, 'to': 1}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.23:0/1618334186': '2024-03-12T20:40:45.361537+0000', '172.21.15.23:6801/2813578081': '2024-03-12T20:40:45.361537+0000', '172.21.15.23:0/2869940180': '2024-03-12T20:51:33.790205+0000', '172.21.15.23:6800/3037564930': '2024-03-12T20:51:33.790205+0000', '172.21.15.23:0/3297567084': '2024-03-12T20:51:33.790205+0000', '172.21.15.23:0/3708753659': '2024-03-12T20:40:45.361537+0000', '172.21.15.23:0/2004146740': '2024-03-12T20:40:45.361537+0000', '172.21.15.23:0/827653812': '2024-03-12T20:41:21.301516+0000', '172.21.15.23:6800/1057699850': '2024-03-12T20:41:21.301516+0000', '172.21.15.23:6800/2813578081': '2024-03-12T20:40:45.361537+0000', '172.21.15.23:6801/1057699850': '2024-03-12T20:41:21.301516+0000', '172.21.15.23:0/3833230375': '2024-03-12T20:41:21.301516+0000', '172.21.15.23:6801/3037564930': '2024-03-12T20:51:33.790205+0000', '172.21.15.23:0/1182809102': '2024-03-12T20:51:33.790205+0000', '172.21.15.23:0/2122134187': '2024-03-12T20:51:33.790205+0000', '172.21.15.23:0/2356701899': '2024-03-12T20:41:21.301516+0000', '172.21.15.23:0/843974100': '2024-03-12T20:51:33.790205+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-03-11T21:12:29.743 INFO:tasks.thrashosds.thrasher:Clearing pg_upmap on 6.a 2024-03-11T21:12:29.743 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd rm-pg-upmap 6.a 2024-03-11T21:12:29.745 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:12:29.766 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:12:29.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:29.769 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:12:29.835 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 25 16 42152 42136 105.308 86.625 0.0140063 0.00948967 2024-03-11T21:12:29.870 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:12:29.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:29.908 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:12:29.908 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:12:30.006 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:29 smithi146 bash[23809]: cluster 2024-03-11T21:12:28.373594+0000 mgr.y (mgr.24380) 803 : cluster 0 pgmap v920: 158 pgs: 158 active+clean; 2.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 92 MiB/s wr, 1.47k op/s 2024-03-11T21:12:30.007 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:29 smithi146 bash[23809]: cluster 2024-03-11T21:12:28.598078+0000 osd.5 (osd.5) 472 : cluster 0 6.d scrub ok 2024-03-11T21:12:30.007 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:29 smithi146 bash[23809]: cluster 2024-03-11T21:12:29.428250+0000 osd.5 (osd.5) 473 : cluster 0 6.17 scrub starts 2024-03-11T21:12:30.009 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:30.026 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:29 smithi023 bash[21372]: cluster 2024-03-11T21:12:28.373594+0000 mgr.y (mgr.24380) 803 : cluster 0 pgmap v920: 158 pgs: 158 active+clean; 2.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 92 MiB/s wr, 1.47k op/s 2024-03-11T21:12:30.027 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:29 smithi023 bash[21372]: cluster 2024-03-11T21:12:28.598078+0000 osd.5 (osd.5) 472 : cluster 0 6.d scrub ok 2024-03-11T21:12:30.027 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:29 smithi023 bash[21372]: cluster 2024-03-11T21:12:29.428250+0000 osd.5 (osd.5) 473 : cluster 0 6.17 scrub starts 2024-03-11T21:12:30.027 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:29 smithi023 bash[29146]: cluster 2024-03-11T21:12:28.373594+0000 mgr.y (mgr.24380) 803 : cluster 0 pgmap v920: 158 pgs: 158 active+clean; 2.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 92 MiB/s wr, 1.47k op/s 2024-03-11T21:12:30.027 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:29 smithi023 bash[29146]: cluster 2024-03-11T21:12:28.598078+0000 osd.5 (osd.5) 472 : cluster 0 6.d scrub ok 2024-03-11T21:12:30.027 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:29 smithi023 bash[29146]: cluster 2024-03-11T21:12:29.428250+0000 osd.5 (osd.5) 473 : cluster 0 6.17 scrub starts 2024-03-11T21:12:30.034 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:30.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:30.037 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:12:30.138 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:12:30.167 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:12:30.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:30.170 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:12:30.271 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:12:30.306 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:12:30.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:30.310 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:12:30.411 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:30.441 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:30.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:30.443 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:12:30.546 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:30.570 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:30.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:30.574 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:12:30.674 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:12:30.708 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:12:30.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:30.712 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:12:30.813 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:12:30.823 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:30 smithi146 bash[23809]: cluster 2024-03-11T21:12:29.603512+0000 osd.5 (osd.5) 474 : cluster 0 6.17 scrub ok 2024-03-11T21:12:30.823 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:30 smithi146 bash[23809]: cluster 2024-03-11T21:12:30.036027+0000 osd.6 (osd.6) 390 : cluster 0 4.13 scrub starts 2024-03-11T21:12:30.823 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:30 smithi146 bash[23809]: cluster 2024-03-11T21:12:30.039443+0000 osd.6 (osd.6) 391 : cluster 0 4.13 scrub ok 2024-03-11T21:12:30.834 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 26 16 43654 43638 104.868 93.875 0.0109735 0.00953053 2024-03-11T21:12:30.841 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:12:30.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:30.845 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:12:30.946 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:12:30.971 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:12:30.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:30.972 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:12:31.026 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:30 smithi023 bash[21372]: cluster 2024-03-11T21:12:29.603512+0000 osd.5 (osd.5) 474 : cluster 0 6.17 scrub ok 2024-03-11T21:12:31.027 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:30 smithi023 bash[21372]: cluster 2024-03-11T21:12:30.036027+0000 osd.6 (osd.6) 390 : cluster 0 4.13 scrub starts 2024-03-11T21:12:31.027 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:30 smithi023 bash[21372]: cluster 2024-03-11T21:12:30.039443+0000 osd.6 (osd.6) 391 : cluster 0 4.13 scrub ok 2024-03-11T21:12:31.027 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:30 smithi023 bash[29146]: cluster 2024-03-11T21:12:29.603512+0000 osd.5 (osd.5) 474 : cluster 0 6.17 scrub ok 2024-03-11T21:12:31.027 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:30 smithi023 bash[29146]: cluster 2024-03-11T21:12:30.036027+0000 osd.6 (osd.6) 390 : cluster 0 4.13 scrub starts 2024-03-11T21:12:31.027 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:30 smithi023 bash[29146]: cluster 2024-03-11T21:12:30.039443+0000 osd.6 (osd.6) 391 : cluster 0 4.13 scrub ok 2024-03-11T21:12:31.074 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:31.098 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:31.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:31.101 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:12:31.202 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:31.229 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:31.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:31.232 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:12:31.333 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:12:31.362 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:12:31.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:31.366 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:12:31.467 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:31.491 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:31.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:31.494 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:12:31.594 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:31.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:31.629 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:31.629 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:12:31.730 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:12:31.753 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:12:31.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:31.758 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:12:31.834 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 27 16 45171 45155 104.495 94.8125 0.00506803 0.00956354 2024-03-11T21:12:31.859 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:12:31.886 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:12:31.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:31.888 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:12:31.989 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:31.998 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:31 smithi146 bash[23809]: cluster 2024-03-11T21:12:30.035554+0000 osd.3 (osd.3) 454 : cluster 0 4.1b scrub starts 2024-03-11T21:12:31.998 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:31 smithi146 bash[23809]: cluster 2024-03-11T21:12:30.037727+0000 osd.3 (osd.3) 455 : cluster 0 4.1b scrub ok 2024-03-11T21:12:31.999 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:31 smithi146 bash[23809]: cluster 2024-03-11T21:12:30.374362+0000 mgr.y (mgr.24380) 804 : cluster 0 pgmap v921: 158 pgs: 158 active+clean; 2.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.3 KiB/s rd, 103 MiB/s wr, 1.65k op/s 2024-03-11T21:12:32.025 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:32.026 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:31 smithi023 bash[21372]: cluster 2024-03-11T21:12:30.035554+0000 osd.3 (osd.3) 454 : cluster 0 4.1b scrub starts 2024-03-11T21:12:32.026 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:31 smithi023 bash[21372]: cluster 2024-03-11T21:12:30.037727+0000 osd.3 (osd.3) 455 : cluster 0 4.1b scrub ok 2024-03-11T21:12:32.026 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:31 smithi023 bash[21372]: cluster 2024-03-11T21:12:30.374362+0000 mgr.y (mgr.24380) 804 : cluster 0 pgmap v921: 158 pgs: 158 active+clean; 2.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.3 KiB/s rd, 103 MiB/s wr, 1.65k op/s 2024-03-11T21:12:32.027 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:31 smithi023 bash[29146]: cluster 2024-03-11T21:12:30.035554+0000 osd.3 (osd.3) 454 : cluster 0 4.1b scrub starts 2024-03-11T21:12:32.027 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:31 smithi023 bash[29146]: cluster 2024-03-11T21:12:30.037727+0000 osd.3 (osd.3) 455 : cluster 0 4.1b scrub ok 2024-03-11T21:12:32.027 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:31 smithi023 bash[29146]: cluster 2024-03-11T21:12:30.374362+0000 mgr.y (mgr.24380) 804 : cluster 0 pgmap v921: 158 pgs: 158 active+clean; 2.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.3 KiB/s rd, 103 MiB/s wr, 1.65k op/s 2024-03-11T21:12:32.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:32.033 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:12:32.134 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:12:32.159 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:12:32.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:32.162 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:12:32.252 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:12:32.263 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:32.284 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:32.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:32.287 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:12:32.390 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:12:32.418 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:12:32.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:32.423 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:12:32.524 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:32.554 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:32.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:32.561 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:12:32.662 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:32.684 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:32.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:32.687 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:12:32.788 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:12:32.798 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:32 smithi146 bash[23809]: cluster 2024-03-11T21:12:31.019567+0000 osd.0 (osd.0) 376 : cluster 0 4.3 scrub starts 2024-03-11T21:12:32.799 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:32 smithi146 bash[23809]: cluster 2024-03-11T21:12:31.022890+0000 osd.0 (osd.0) 377 : cluster 0 4.3 scrub ok 2024-03-11T21:12:32.799 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:32 smithi146 bash[23809]: cluster 2024-03-11T21:12:31.134723+0000 osd.7 (osd.7) 519 : cluster 0 3.f deep-scrub starts 2024-03-11T21:12:32.799 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:32 smithi146 bash[23809]: cluster 2024-03-11T21:12:31.144993+0000 osd.7 (osd.7) 520 : cluster 0 3.f deep-scrub ok 2024-03-11T21:12:32.799 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:32 smithi146 bash[23809]: cluster 2024-03-11T21:12:31.954301+0000 osd.3 (osd.3) 456 : cluster 0 4.1d scrub starts 2024-03-11T21:12:32.799 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:32 smithi146 bash[23809]: cluster 2024-03-11T21:12:31.955352+0000 osd.3 (osd.3) 457 : cluster 0 4.1d scrub ok 2024-03-11T21:12:32.799 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:32 smithi146 bash[23809]: cluster 2024-03-11T21:12:32.064482+0000 osd.6 (osd.6) 392 : cluster 0 3.10 scrub starts 2024-03-11T21:12:32.799 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:32 smithi146 bash[23809]: cluster 2024-03-11T21:12:32.066245+0000 osd.6 (osd.6) 393 : cluster 0 3.10 scrub ok 2024-03-11T21:12:32.799 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:32 smithi146 bash[23809]: cluster 2024-03-11T21:12:32.374981+0000 mgr.y (mgr.24380) 805 : cluster 0 pgmap v922: 158 pgs: 158 active+clean; 2.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 853 B/s rd, 85 MiB/s wr, 1.36k op/s 2024-03-11T21:12:32.810 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:12:32.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:32.814 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:12:32.834 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 28 16 46605 46589 103.964 89.625 0.00724629 0.00961302 2024-03-11T21:12:32.915 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:12:32.941 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:12:32.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:32.943 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:12:33.027 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:32 smithi023 bash[21372]: cluster 2024-03-11T21:12:31.019567+0000 osd.0 (osd.0) 376 : cluster 0 4.3 scrub starts 2024-03-11T21:12:33.027 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:32 smithi023 bash[21372]: cluster 2024-03-11T21:12:31.022890+0000 osd.0 (osd.0) 377 : cluster 0 4.3 scrub ok 2024-03-11T21:12:33.027 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:32 smithi023 bash[21372]: cluster 2024-03-11T21:12:31.134723+0000 osd.7 (osd.7) 519 : cluster 0 3.f deep-scrub starts 2024-03-11T21:12:33.027 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:32 smithi023 bash[21372]: cluster 2024-03-11T21:12:31.144993+0000 osd.7 (osd.7) 520 : cluster 0 3.f deep-scrub ok 2024-03-11T21:12:33.027 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:32 smithi023 bash[21372]: cluster 2024-03-11T21:12:31.954301+0000 osd.3 (osd.3) 456 : cluster 0 4.1d scrub starts 2024-03-11T21:12:33.027 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:32 smithi023 bash[21372]: cluster 2024-03-11T21:12:31.955352+0000 osd.3 (osd.3) 457 : cluster 0 4.1d scrub ok 2024-03-11T21:12:33.027 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:32 smithi023 bash[21372]: cluster 2024-03-11T21:12:32.064482+0000 osd.6 (osd.6) 392 : cluster 0 3.10 scrub starts 2024-03-11T21:12:33.027 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:32 smithi023 bash[21372]: cluster 2024-03-11T21:12:32.066245+0000 osd.6 (osd.6) 393 : cluster 0 3.10 scrub ok 2024-03-11T21:12:33.027 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:32 smithi023 bash[21372]: cluster 2024-03-11T21:12:32.374981+0000 mgr.y (mgr.24380) 805 : cluster 0 pgmap v922: 158 pgs: 158 active+clean; 2.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 853 B/s rd, 85 MiB/s wr, 1.36k op/s 2024-03-11T21:12:33.028 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:32 smithi023 bash[29146]: cluster 2024-03-11T21:12:31.019567+0000 osd.0 (osd.0) 376 : cluster 0 4.3 scrub starts 2024-03-11T21:12:33.028 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:32 smithi023 bash[29146]: cluster 2024-03-11T21:12:31.022890+0000 osd.0 (osd.0) 377 : cluster 0 4.3 scrub ok 2024-03-11T21:12:33.028 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:32 smithi023 bash[29146]: cluster 2024-03-11T21:12:31.134723+0000 osd.7 (osd.7) 519 : cluster 0 3.f deep-scrub starts 2024-03-11T21:12:33.028 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:32 smithi023 bash[29146]: cluster 2024-03-11T21:12:31.144993+0000 osd.7 (osd.7) 520 : cluster 0 3.f deep-scrub ok 2024-03-11T21:12:33.028 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:32 smithi023 bash[29146]: cluster 2024-03-11T21:12:31.954301+0000 osd.3 (osd.3) 456 : cluster 0 4.1d scrub starts 2024-03-11T21:12:33.028 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:32 smithi023 bash[29146]: cluster 2024-03-11T21:12:31.955352+0000 osd.3 (osd.3) 457 : cluster 0 4.1d scrub ok 2024-03-11T21:12:33.028 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:32 smithi023 bash[29146]: cluster 2024-03-11T21:12:32.064482+0000 osd.6 (osd.6) 392 : cluster 0 3.10 scrub starts 2024-03-11T21:12:33.028 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:32 smithi023 bash[29146]: cluster 2024-03-11T21:12:32.066245+0000 osd.6 (osd.6) 393 : cluster 0 3.10 scrub ok 2024-03-11T21:12:33.028 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:32 smithi023 bash[29146]: cluster 2024-03-11T21:12:32.374981+0000 mgr.y (mgr.24380) 805 : cluster 0 pgmap v922: 158 pgs: 158 active+clean; 2.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 853 B/s rd, 85 MiB/s wr, 1.36k op/s 2024-03-11T21:12:33.045 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:12:33.071 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:12:33.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:33.073 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:12:33.174 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:12:33.201 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:12:33.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:33.204 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:12:33.304 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:12:33.328 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:12:33.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:33.332 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:12:33.432 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:33.452 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:33.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:33.453 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:12:33.554 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:12:33.572 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:12:33.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:33.574 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:12:33.675 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:12:33.700 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:12:33.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:33.702 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:12:33.803 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:12:33.834 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 29 16 47471 47455 102.246 54.125 0.00528653 0.0097729 2024-03-11T21:12:33.836 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:12:33.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:33.839 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:12:33.940 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:12:33.970 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:12:33.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:33.974 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:12:34.006 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:33 smithi146 bash[23809]: cluster 2024-03-11T21:12:31.995982+0000 osd.0 (osd.0) 378 : cluster 0 3.17 scrub starts 2024-03-11T21:12:34.007 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:33 smithi146 bash[23809]: cluster 2024-03-11T21:12:32.003154+0000 osd.0 (osd.0) 379 : cluster 0 3.17 scrub ok 2024-03-11T21:12:34.007 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:33 smithi146 bash[23809]: cluster 2024-03-11T21:12:32.755194+0000 osd.6 (osd.6) 394 : cluster 0 purged_snaps scrub starts 2024-03-11T21:12:34.007 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:33 smithi146 bash[23809]: cluster 2024-03-11T21:12:32.755245+0000 osd.6 (osd.6) 395 : cluster 0 purged_snaps scrub ok 2024-03-11T21:12:34.007 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:33 smithi146 bash[23809]: cluster 2024-03-11T21:12:32.860745+0000 osd.1 (osd.1) 283 : cluster 0 4.1c deep-scrub starts 2024-03-11T21:12:34.007 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:33 smithi146 bash[23809]: cluster 2024-03-11T21:12:32.862529+0000 osd.1 (osd.1) 284 : cluster 0 4.1c deep-scrub ok 2024-03-11T21:12:34.007 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:33 smithi146 bash[23809]: cluster 2024-03-11T21:12:32.942093+0000 osd.3 (osd.3) 458 : cluster 0 4.17 scrub starts 2024-03-11T21:12:34.007 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:33 smithi146 bash[23809]: cluster 2024-03-11T21:12:32.946953+0000 osd.3 (osd.3) 459 : cluster 0 4.17 scrub ok 2024-03-11T21:12:34.027 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:33 smithi023 bash[21372]: cluster 2024-03-11T21:12:31.995982+0000 osd.0 (osd.0) 378 : cluster 0 3.17 scrub starts 2024-03-11T21:12:34.027 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:33 smithi023 bash[21372]: cluster 2024-03-11T21:12:32.003154+0000 osd.0 (osd.0) 379 : cluster 0 3.17 scrub ok 2024-03-11T21:12:34.027 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:33 smithi023 bash[21372]: cluster 2024-03-11T21:12:32.755194+0000 osd.6 (osd.6) 394 : cluster 0 purged_snaps scrub starts 2024-03-11T21:12:34.027 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:33 smithi023 bash[21372]: cluster 2024-03-11T21:12:32.755245+0000 osd.6 (osd.6) 395 : cluster 0 purged_snaps scrub ok 2024-03-11T21:12:34.027 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:33 smithi023 bash[21372]: cluster 2024-03-11T21:12:32.860745+0000 osd.1 (osd.1) 283 : cluster 0 4.1c deep-scrub starts 2024-03-11T21:12:34.027 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:33 smithi023 bash[21372]: cluster 2024-03-11T21:12:32.862529+0000 osd.1 (osd.1) 284 : cluster 0 4.1c deep-scrub ok 2024-03-11T21:12:34.027 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:33 smithi023 bash[21372]: cluster 2024-03-11T21:12:32.942093+0000 osd.3 (osd.3) 458 : cluster 0 4.17 scrub starts 2024-03-11T21:12:34.027 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:33 smithi023 bash[21372]: cluster 2024-03-11T21:12:32.946953+0000 osd.3 (osd.3) 459 : cluster 0 4.17 scrub ok 2024-03-11T21:12:34.028 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:33 smithi023 bash[29146]: cluster 2024-03-11T21:12:31.995982+0000 osd.0 (osd.0) 378 : cluster 0 3.17 scrub starts 2024-03-11T21:12:34.028 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:33 smithi023 bash[29146]: cluster 2024-03-11T21:12:32.003154+0000 osd.0 (osd.0) 379 : cluster 0 3.17 scrub ok 2024-03-11T21:12:34.028 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:33 smithi023 bash[29146]: cluster 2024-03-11T21:12:32.755194+0000 osd.6 (osd.6) 394 : cluster 0 purged_snaps scrub starts 2024-03-11T21:12:34.028 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:33 smithi023 bash[29146]: cluster 2024-03-11T21:12:32.755245+0000 osd.6 (osd.6) 395 : cluster 0 purged_snaps scrub ok 2024-03-11T21:12:34.028 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:33 smithi023 bash[29146]: cluster 2024-03-11T21:12:32.860745+0000 osd.1 (osd.1) 283 : cluster 0 4.1c deep-scrub starts 2024-03-11T21:12:34.028 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:33 smithi023 bash[29146]: cluster 2024-03-11T21:12:32.862529+0000 osd.1 (osd.1) 284 : cluster 0 4.1c deep-scrub ok 2024-03-11T21:12:34.028 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:33 smithi023 bash[29146]: cluster 2024-03-11T21:12:32.942093+0000 osd.3 (osd.3) 458 : cluster 0 4.17 scrub starts 2024-03-11T21:12:34.028 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:33 smithi023 bash[29146]: cluster 2024-03-11T21:12:32.946953+0000 osd.3 (osd.3) 459 : cluster 0 4.17 scrub ok 2024-03-11T21:12:34.076 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:12:34.101 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:12:34.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:34.105 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:12:34.206 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:34.235 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:34.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:34.239 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:12:34.281 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:12:34.282 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:12:34.282 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:12:34.340 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:34.432 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:34.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:34.435 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:12:34.536 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:12:34.565 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:12:34.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:34.569 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:12:34.670 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:12:34.699 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:12:34.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:34.702 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:12:34.802 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:34.829 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:34.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:34.833 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:12:34.834 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 30 16 48782 48766 101.569 81.9375 0.00463035 0.00983798 2024-03-11T21:12:34.935 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:12:34.943 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:34 smithi146 bash[23809]: cluster 2024-03-11T21:12:33.776684+0000 mon.a (mon.0) 1827 : cluster 0 osdmap e392: 8 total, 8 up, 8 in 2024-03-11T21:12:34.943 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:34 smithi146 bash[23809]: cluster 2024-03-11T21:12:33.822574+0000 osd.5 (osd.5) 475 : cluster 0 2.c scrub starts 2024-03-11T21:12:34.943 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:34 smithi146 bash[23809]: cluster 2024-03-11T21:12:33.830520+0000 osd.5 (osd.5) 476 : cluster 0 2.c scrub ok 2024-03-11T21:12:34.943 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:34 smithi146 bash[23809]: cluster 2024-03-11T21:12:34.375400+0000 mgr.y (mgr.24380) 806 : cluster 0 pgmap v924: 158 pgs: 158 active+clean; 2.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 89 MiB/s wr, 1.43k op/s 2024-03-11T21:12:34.962 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:12:34.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:34.966 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:12:35.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:34 smithi023 bash[21372]: cluster 2024-03-11T21:12:33.776684+0000 mon.a (mon.0) 1827 : cluster 0 osdmap e392: 8 total, 8 up, 8 in 2024-03-11T21:12:35.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:34 smithi023 bash[21372]: cluster 2024-03-11T21:12:33.822574+0000 osd.5 (osd.5) 475 : cluster 0 2.c scrub starts 2024-03-11T21:12:35.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:34 smithi023 bash[21372]: cluster 2024-03-11T21:12:33.830520+0000 osd.5 (osd.5) 476 : cluster 0 2.c scrub ok 2024-03-11T21:12:35.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:34 smithi023 bash[21372]: cluster 2024-03-11T21:12:34.375400+0000 mgr.y (mgr.24380) 806 : cluster 0 pgmap v924: 158 pgs: 158 active+clean; 2.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 89 MiB/s wr, 1.43k op/s 2024-03-11T21:12:35.028 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:34 smithi023 bash[29146]: cluster 2024-03-11T21:12:33.776684+0000 mon.a (mon.0) 1827 : cluster 0 osdmap e392: 8 total, 8 up, 8 in 2024-03-11T21:12:35.028 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:34 smithi023 bash[29146]: cluster 2024-03-11T21:12:33.822574+0000 osd.5 (osd.5) 475 : cluster 0 2.c scrub starts 2024-03-11T21:12:35.028 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:34 smithi023 bash[29146]: cluster 2024-03-11T21:12:33.830520+0000 osd.5 (osd.5) 476 : cluster 0 2.c scrub ok 2024-03-11T21:12:35.028 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:34 smithi023 bash[29146]: cluster 2024-03-11T21:12:34.375400+0000 mgr.y (mgr.24380) 806 : cluster 0 pgmap v924: 158 pgs: 158 active+clean; 2.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 89 MiB/s wr, 1.43k op/s 2024-03-11T21:12:35.067 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:12:35.092 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:12:35.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:35.097 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:12:35.198 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:12:35.223 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:12:35.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:35.225 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:12:35.326 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:35.362 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:35.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:35.364 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:12:35.465 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:12:35.487 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:12:35.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:35.488 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:12:35.589 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:12:35.622 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:12:35.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:35.625 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:12:35.726 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:12:35.743 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:12:35.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:35.745 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:12:35.835 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 31 16 49474 49458 99.6874 43.25 0.00217932 0.00998884 2024-03-11T21:12:35.846 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:12:35.873 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:12:35.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:35.875 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:12:35.976 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:12:36.013 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:12:36.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:36.016 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:12:36.116 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:12:36.125 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:12:35 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:12:36.137 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:12:36.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:36.139 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:12:36.240 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:12:36.277 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:12:36.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:36.280 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:12:36.381 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:12:36.421 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:12:36.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:36.426 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:12:36.506 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:36 smithi146 bash[23809]: cluster 2024-03-11T21:12:35.369786+0000 mon.a (mon.0) 1828 : cluster 0 osdmap e393: 8 total, 8 up, 8 in 2024-03-11T21:12:36.507 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:36 smithi146 bash[23809]: cluster 2024-03-11T21:12:35.487430+0000 osd.5 (osd.5) 477 : cluster 0 3.c deep-scrub starts 2024-03-11T21:12:36.507 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:36 smithi146 bash[23809]: cluster 2024-03-11T21:12:35.497193+0000 osd.5 (osd.5) 478 : cluster 0 3.c deep-scrub ok 2024-03-11T21:12:36.507 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:36 smithi146 bash[23809]: audit 2024-03-11T21:12:36.210171+0000 mon.a (mon.0) 1829 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:12:36.527 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:36.551 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:36.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:36.554 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:12:36.654 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:12:36.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:36 smithi023 bash[21372]: cluster 2024-03-11T21:12:35.369786+0000 mon.a (mon.0) 1828 : cluster 0 osdmap e393: 8 total, 8 up, 8 in 2024-03-11T21:12:36.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:36 smithi023 bash[21372]: cluster 2024-03-11T21:12:35.487430+0000 osd.5 (osd.5) 477 : cluster 0 3.c deep-scrub starts 2024-03-11T21:12:36.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:36 smithi023 bash[21372]: cluster 2024-03-11T21:12:35.497193+0000 osd.5 (osd.5) 478 : cluster 0 3.c deep-scrub ok 2024-03-11T21:12:36.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:36 smithi023 bash[21372]: audit 2024-03-11T21:12:36.210171+0000 mon.a (mon.0) 1829 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:12:36.671 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:36 smithi023 bash[29146]: cluster 2024-03-11T21:12:35.369786+0000 mon.a (mon.0) 1828 : cluster 0 osdmap e393: 8 total, 8 up, 8 in 2024-03-11T21:12:36.671 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:36 smithi023 bash[29146]: cluster 2024-03-11T21:12:35.487430+0000 osd.5 (osd.5) 477 : cluster 0 3.c deep-scrub starts 2024-03-11T21:12:36.671 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:36 smithi023 bash[29146]: cluster 2024-03-11T21:12:35.497193+0000 osd.5 (osd.5) 478 : cluster 0 3.c deep-scrub ok 2024-03-11T21:12:36.671 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:36 smithi023 bash[29146]: audit 2024-03-11T21:12:36.210171+0000 mon.a (mon.0) 1829 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:12:36.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:36.695 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:12:36.696 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:12:36.796 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:12:36.821 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:12:36.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:36.823 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:12:36.834 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 32 16 50102 50086 97.7988 39.25 0.0036032 0.0102114 2024-03-11T21:12:36.924 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:12:36.952 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:12:36.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:36.954 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:12:37.055 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:12:37.083 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:12:37.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:37.086 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:12:37.187 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:12:37.213 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:12:37.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:37.215 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:12:37.316 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:37.335 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:37.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:37.337 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:12:37.438 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:37.471 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:37.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:37.473 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:12:37.573 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:37.582 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:37 smithi146 bash[23809]: audit 2024-03-11T21:12:35.937159+0000 mgr.y (mgr.24380) 807 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:12:37.582 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:37 smithi146 bash[23809]: cluster 2024-03-11T21:12:36.376840+0000 mgr.y (mgr.24380) 808 : cluster 0 pgmap v926: 158 pgs: 158 active+clean; 3.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 95 MiB/s wr, 1.52k op/s 2024-03-11T21:12:37.582 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:37 smithi146 bash[23809]: cluster 2024-03-11T21:12:36.441245+0000 osd.5 (osd.5) 479 : cluster 0 4.15 scrub starts 2024-03-11T21:12:37.583 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:37 smithi146 bash[23809]: cluster 2024-03-11T21:12:36.443639+0000 osd.5 (osd.5) 480 : cluster 0 4.15 scrub ok 2024-03-11T21:12:37.595 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:37.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:37.597 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:12:37.653 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:37 smithi023 bash[21372]: audit 2024-03-11T21:12:35.937159+0000 mgr.y (mgr.24380) 807 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:12:37.653 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:37 smithi023 bash[21372]: cluster 2024-03-11T21:12:36.376840+0000 mgr.y (mgr.24380) 808 : cluster 0 pgmap v926: 158 pgs: 158 active+clean; 3.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 95 MiB/s wr, 1.52k op/s 2024-03-11T21:12:37.653 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:37 smithi023 bash[21372]: cluster 2024-03-11T21:12:36.441245+0000 osd.5 (osd.5) 479 : cluster 0 4.15 scrub starts 2024-03-11T21:12:37.654 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:37 smithi023 bash[21372]: cluster 2024-03-11T21:12:36.443639+0000 osd.5 (osd.5) 480 : cluster 0 4.15 scrub ok 2024-03-11T21:12:37.654 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:37 smithi023 bash[29146]: audit 2024-03-11T21:12:35.937159+0000 mgr.y (mgr.24380) 807 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:12:37.654 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:37 smithi023 bash[29146]: cluster 2024-03-11T21:12:36.376840+0000 mgr.y (mgr.24380) 808 : cluster 0 pgmap v926: 158 pgs: 158 active+clean; 3.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 95 MiB/s wr, 1.52k op/s 2024-03-11T21:12:37.654 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:37 smithi023 bash[29146]: cluster 2024-03-11T21:12:36.441245+0000 osd.5 (osd.5) 479 : cluster 0 4.15 scrub starts 2024-03-11T21:12:37.654 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:37 smithi023 bash[29146]: cluster 2024-03-11T21:12:36.443639+0000 osd.5 (osd.5) 480 : cluster 0 4.15 scrub ok 2024-03-11T21:12:37.698 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:12:37.732 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:12:37.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:37.734 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:12:37.835 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 33 16 50995 50979 96.5265 55.8125 0.0176635 0.0103541 2024-03-11T21:12:37.835 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:37.872 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:37.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:37.875 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:12:37.975 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:12:38.007 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:12:38.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:38.009 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:12:38.110 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:38.152 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:38.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:38.154 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:12:38.255 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:12:38.288 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:12:38.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:38.291 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:12:38.391 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:12:38.433 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:12:38.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:38.436 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:12:38.536 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:12:38.567 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:12:38.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:38.570 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:12:38.662 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:12:38 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:12:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:12:38.671 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:12:38.708 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:12:38.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:38.712 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:12:38.813 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:38.835 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 34 16 52246 52230 95.987 78.1875 0.00789673 0.0104118 2024-03-11T21:12:38.836 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:38.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:38.839 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:12:38.940 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:12:38.965 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:12:38.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:38.967 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:12:39.068 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:12:39.094 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:12:39.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:39.098 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:12:39.199 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:39.227 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:39.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:39.234 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:12:39.335 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:39.362 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:39.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:39.366 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:12:39.466 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:12:39.475 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:39 smithi146 bash[23809]: cluster 2024-03-11T21:12:38.377405+0000 mgr.y (mgr.24380) 809 : cluster 0 pgmap v927: 158 pgs: 158 active+clean; 3.0 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 65 MiB/s wr, 1.04k op/s 2024-03-11T21:12:39.476 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:39 smithi146 bash[23809]: cluster 2024-03-11T21:12:38.471047+0000 osd.5 (osd.5) 481 : cluster 0 4.8 scrub starts 2024-03-11T21:12:39.476 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:39 smithi146 bash[23809]: cluster 2024-03-11T21:12:38.473797+0000 osd.5 (osd.5) 482 : cluster 0 4.8 scrub ok 2024-03-11T21:12:39.487 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:12:39.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:39.489 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:12:39.590 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:12:39.614 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:12:39.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:39.617 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:12:39.718 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:12:39.756 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:12:39.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:39.763 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:12:39.835 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 35 16 53094 53078 94.7589 53 0.0126438 0.0105469 2024-03-11T21:12:39.864 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:12:39.894 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:12:39.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:39.898 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:12:39.999 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:12:40.020 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:12:40.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:40.022 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:12:40.083 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:39 smithi023 bash[21372]: cluster 2024-03-11T21:12:38.377405+0000 mgr.y (mgr.24380) 809 : cluster 0 pgmap v927: 158 pgs: 158 active+clean; 3.0 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 65 MiB/s wr, 1.04k op/s 2024-03-11T21:12:40.083 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:39 smithi023 bash[21372]: cluster 2024-03-11T21:12:38.471047+0000 osd.5 (osd.5) 481 : cluster 0 4.8 scrub starts 2024-03-11T21:12:40.083 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:39 smithi023 bash[21372]: cluster 2024-03-11T21:12:38.473797+0000 osd.5 (osd.5) 482 : cluster 0 4.8 scrub ok 2024-03-11T21:12:40.083 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:39 smithi023 bash[29146]: cluster 2024-03-11T21:12:38.377405+0000 mgr.y (mgr.24380) 809 : cluster 0 pgmap v927: 158 pgs: 158 active+clean; 3.0 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 65 MiB/s wr, 1.04k op/s 2024-03-11T21:12:40.083 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:39 smithi023 bash[29146]: cluster 2024-03-11T21:12:38.471047+0000 osd.5 (osd.5) 481 : cluster 0 4.8 scrub starts 2024-03-11T21:12:40.083 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:39 smithi023 bash[29146]: cluster 2024-03-11T21:12:38.473797+0000 osd.5 (osd.5) 482 : cluster 0 4.8 scrub ok 2024-03-11T21:12:40.123 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:12:40.164 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:12:40.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:40.167 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:12:40.268 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:12:40.290 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:12:40.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:40.293 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:12:40.393 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:12:40.412 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:12:40.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:40.414 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:12:40.510 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:40 smithi023 bash[21372]: cluster 2024-03-11T21:12:39.909619+0000 osd.1 (osd.1) 285 : cluster 0 2.0 scrub starts 2024-03-11T21:12:40.510 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:40 smithi023 bash[21372]: cluster 2024-03-11T21:12:39.915890+0000 osd.1 (osd.1) 286 : cluster 0 2.0 scrub ok 2024-03-11T21:12:40.510 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:40 smithi023 bash[21372]: cluster 2024-03-11T21:12:39.959466+0000 osd.3 (osd.3) 460 : cluster 0 5.1 scrub starts 2024-03-11T21:12:40.511 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:40 smithi023 bash[29146]: cluster 2024-03-11T21:12:39.909619+0000 osd.1 (osd.1) 285 : cluster 0 2.0 scrub starts 2024-03-11T21:12:40.511 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:40 smithi023 bash[29146]: cluster 2024-03-11T21:12:39.915890+0000 osd.1 (osd.1) 286 : cluster 0 2.0 scrub ok 2024-03-11T21:12:40.511 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:40 smithi023 bash[29146]: cluster 2024-03-11T21:12:39.959466+0000 osd.3 (osd.3) 460 : cluster 0 5.1 scrub starts 2024-03-11T21:12:40.515 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:40.539 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:40.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:40.540 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:12:40.641 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:12:40.662 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:12:40.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:40.664 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:12:40.756 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:40 smithi146 bash[23809]: cluster 2024-03-11T21:12:39.909619+0000 osd.1 (osd.1) 285 : cluster 0 2.0 scrub starts 2024-03-11T21:12:40.757 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:40 smithi146 bash[23809]: cluster 2024-03-11T21:12:39.915890+0000 osd.1 (osd.1) 286 : cluster 0 2.0 scrub ok 2024-03-11T21:12:40.757 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:40 smithi146 bash[23809]: cluster 2024-03-11T21:12:39.959466+0000 osd.3 (osd.3) 460 : cluster 0 5.1 scrub starts 2024-03-11T21:12:40.764 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:12:40.788 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:12:40.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:40.790 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:12:40.835 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 36 16 53808 53792 93.3664 44.625 0.00330632 0.01062 2024-03-11T21:12:40.890 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:12:40.912 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:12:40.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:40.914 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:12:41.015 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:12:41.041 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:12:41.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:41.043 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:12:41.144 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:12:41.176 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:12:41.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:41.178 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:12:41.279 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:12:41.318 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:12:41.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:41.322 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:12:41.423 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:12:41.461 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:12:41.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:41.466 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:12:41.567 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:12:41.576 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:41 smithi146 bash[23809]: cluster 2024-03-11T21:12:39.697792+0000 osd.5 (osd.5) 483 : cluster 0 2.12 deep-scrub starts 2024-03-11T21:12:41.577 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:41 smithi146 bash[23809]: cluster 2024-03-11T21:12:39.708020+0000 osd.5 (osd.5) 484 : cluster 0 2.12 deep-scrub ok 2024-03-11T21:12:41.577 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:41 smithi146 bash[23809]: cluster 2024-03-11T21:12:40.096680+0000 osd.3 (osd.3) 461 : cluster 0 5.1 scrub ok 2024-03-11T21:12:41.577 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:41 smithi146 bash[23809]: cluster 2024-03-11T21:12:40.378074+0000 mgr.y (mgr.24380) 810 : cluster 0 pgmap v928: 158 pgs: 158 active+clean; 3.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 71 MiB/s wr, 1.14k op/s 2024-03-11T21:12:41.594 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:12:41.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:41.599 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:12:41.699 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:12:41.710 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:41 smithi023 bash[21372]: cluster 2024-03-11T21:12:39.697792+0000 osd.5 (osd.5) 483 : cluster 0 2.12 deep-scrub starts 2024-03-11T21:12:41.710 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:41 smithi023 bash[21372]: cluster 2024-03-11T21:12:39.708020+0000 osd.5 (osd.5) 484 : cluster 0 2.12 deep-scrub ok 2024-03-11T21:12:41.710 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:41 smithi023 bash[21372]: cluster 2024-03-11T21:12:40.096680+0000 osd.3 (osd.3) 461 : cluster 0 5.1 scrub ok 2024-03-11T21:12:41.710 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:41 smithi023 bash[21372]: cluster 2024-03-11T21:12:40.378074+0000 mgr.y (mgr.24380) 810 : cluster 0 pgmap v928: 158 pgs: 158 active+clean; 3.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 71 MiB/s wr, 1.14k op/s 2024-03-11T21:12:41.710 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:41 smithi023 bash[29146]: cluster 2024-03-11T21:12:39.697792+0000 osd.5 (osd.5) 483 : cluster 0 2.12 deep-scrub starts 2024-03-11T21:12:41.711 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:41 smithi023 bash[29146]: cluster 2024-03-11T21:12:39.708020+0000 osd.5 (osd.5) 484 : cluster 0 2.12 deep-scrub ok 2024-03-11T21:12:41.711 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:41 smithi023 bash[29146]: cluster 2024-03-11T21:12:40.096680+0000 osd.3 (osd.3) 461 : cluster 0 5.1 scrub ok 2024-03-11T21:12:41.711 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:41 smithi023 bash[29146]: cluster 2024-03-11T21:12:40.378074+0000 mgr.y (mgr.24380) 810 : cluster 0 pgmap v928: 158 pgs: 158 active+clean; 3.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 71 MiB/s wr, 1.14k op/s 2024-03-11T21:12:41.729 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:12:41.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:41.732 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:12:41.833 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:12:41.835 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 37 16 54364 54348 91.7822 34.75 0.00433225 0.0108584 2024-03-11T21:12:41.856 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:12:41.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:41.858 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:12:41.959 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:12:41.987 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:12:41.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:41.989 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:12:42.090 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:42.115 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:42.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:42.117 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:12:42.218 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:42.246 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:42.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:42.248 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:12:42.349 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:42.370 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:42.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:42.372 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:12:42.472 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:12:42.501 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:12:42.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:42.504 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:12:42.604 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:12:42.616 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:42 smithi023 bash[21372]: cluster 2024-03-11T21:12:41.139878+0000 osd.0 (osd.0) 380 : cluster 0 6.0 scrub starts 2024-03-11T21:12:42.616 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:42 smithi023 bash[21372]: cluster 2024-03-11T21:12:41.214075+0000 osd.0 (osd.0) 381 : cluster 0 6.0 scrub ok 2024-03-11T21:12:42.617 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:42 smithi023 bash[21372]: cluster 2024-03-11T21:12:41.930734+0000 osd.1 (osd.1) 287 : cluster 0 3.19 scrub starts 2024-03-11T21:12:42.617 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:42 smithi023 bash[21372]: cluster 2024-03-11T21:12:41.940114+0000 osd.1 (osd.1) 288 : cluster 0 3.19 scrub ok 2024-03-11T21:12:42.617 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:42 smithi023 bash[29146]: cluster 2024-03-11T21:12:41.139878+0000 osd.0 (osd.0) 380 : cluster 0 6.0 scrub starts 2024-03-11T21:12:42.617 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:42 smithi023 bash[29146]: cluster 2024-03-11T21:12:41.214075+0000 osd.0 (osd.0) 381 : cluster 0 6.0 scrub ok 2024-03-11T21:12:42.617 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:42 smithi023 bash[29146]: cluster 2024-03-11T21:12:41.930734+0000 osd.1 (osd.1) 287 : cluster 0 3.19 scrub starts 2024-03-11T21:12:42.617 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:42 smithi023 bash[29146]: cluster 2024-03-11T21:12:41.940114+0000 osd.1 (osd.1) 288 : cluster 0 3.19 scrub ok 2024-03-11T21:12:42.640 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:12:42.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:42.643 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:12:42.744 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:12:42.756 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:42 smithi146 bash[23809]: cluster 2024-03-11T21:12:41.139878+0000 osd.0 (osd.0) 380 : cluster 0 6.0 scrub starts 2024-03-11T21:12:42.757 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:42 smithi146 bash[23809]: cluster 2024-03-11T21:12:41.214075+0000 osd.0 (osd.0) 381 : cluster 0 6.0 scrub ok 2024-03-11T21:12:42.757 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:42 smithi146 bash[23809]: cluster 2024-03-11T21:12:41.930734+0000 osd.1 (osd.1) 287 : cluster 0 3.19 scrub starts 2024-03-11T21:12:42.757 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:42 smithi146 bash[23809]: cluster 2024-03-11T21:12:41.940114+0000 osd.1 (osd.1) 288 : cluster 0 3.19 scrub ok 2024-03-11T21:12:42.775 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:12:42.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:42.781 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:12:42.835 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 38 16 54876 54860 90.2091 32 0.012519 0.01108 2024-03-11T21:12:42.881 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:12:42.910 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:12:42.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:42.918 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:12:43.019 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:12:43.047 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:12:43.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:43.052 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:12:43.153 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:43.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:43.184 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:43.184 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:12:43.285 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:12:43.315 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:12:43.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:43.318 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:12:43.419 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:12:43.451 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:12:43.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:43.455 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:12:43.556 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:12:43.582 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:12:43.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:43.585 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:12:43.642 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:43 smithi023 bash[21372]: cluster 2024-03-11T21:12:41.929537+0000 osd.0 (osd.0) 382 : cluster 0 4.6 scrub starts 2024-03-11T21:12:43.642 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:43 smithi023 bash[21372]: cluster 2024-03-11T21:12:41.942560+0000 osd.0 (osd.0) 383 : cluster 0 4.6 scrub ok 2024-03-11T21:12:43.642 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:43 smithi023 bash[21372]: cluster 2024-03-11T21:12:42.378809+0000 mgr.y (mgr.24380) 811 : cluster 0 pgmap v929: 158 pgs: 158 active+clean; 3.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 70 MiB/s wr, 1.12k op/s 2024-03-11T21:12:43.642 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:43 smithi023 bash[21372]: cluster 2024-03-11T21:12:42.922121+0000 osd.1 (osd.1) 289 : cluster 0 2.9 scrub starts 2024-03-11T21:12:43.642 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:43 smithi023 bash[21372]: cluster 2024-03-11T21:12:42.924352+0000 osd.1 (osd.1) 290 : cluster 0 2.9 scrub ok 2024-03-11T21:12:43.643 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:43 smithi023 bash[29146]: cluster 2024-03-11T21:12:41.929537+0000 osd.0 (osd.0) 382 : cluster 0 4.6 scrub starts 2024-03-11T21:12:43.643 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:43 smithi023 bash[29146]: cluster 2024-03-11T21:12:41.942560+0000 osd.0 (osd.0) 383 : cluster 0 4.6 scrub ok 2024-03-11T21:12:43.643 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:43 smithi023 bash[29146]: cluster 2024-03-11T21:12:42.378809+0000 mgr.y (mgr.24380) 811 : cluster 0 pgmap v929: 158 pgs: 158 active+clean; 3.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 70 MiB/s wr, 1.12k op/s 2024-03-11T21:12:43.643 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:43 smithi023 bash[29146]: cluster 2024-03-11T21:12:42.922121+0000 osd.1 (osd.1) 289 : cluster 0 2.9 scrub starts 2024-03-11T21:12:43.643 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:43 smithi023 bash[29146]: cluster 2024-03-11T21:12:42.924352+0000 osd.1 (osd.1) 290 : cluster 0 2.9 scrub ok 2024-03-11T21:12:43.686 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:43.704 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:43.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:43.706 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:12:43.756 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:43 smithi146 bash[23809]: cluster 2024-03-11T21:12:41.929537+0000 osd.0 (osd.0) 382 : cluster 0 4.6 scrub starts 2024-03-11T21:12:43.756 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:43 smithi146 bash[23809]: cluster 2024-03-11T21:12:41.942560+0000 osd.0 (osd.0) 383 : cluster 0 4.6 scrub ok 2024-03-11T21:12:43.756 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:43 smithi146 bash[23809]: cluster 2024-03-11T21:12:42.378809+0000 mgr.y (mgr.24380) 811 : cluster 0 pgmap v929: 158 pgs: 158 active+clean; 3.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 70 MiB/s wr, 1.12k op/s 2024-03-11T21:12:43.757 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:43 smithi146 bash[23809]: cluster 2024-03-11T21:12:42.922121+0000 osd.1 (osd.1) 289 : cluster 0 2.9 scrub starts 2024-03-11T21:12:43.757 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:43 smithi146 bash[23809]: cluster 2024-03-11T21:12:42.924352+0000 osd.1 (osd.1) 290 : cluster 0 2.9 scrub ok 2024-03-11T21:12:43.807 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:12:43.839 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 39 15 55905 55890 89.5466 64.375 0.0108797 0.0111614 2024-03-11T21:12:43.841 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:12:43.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:43.844 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:12:43.944 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:43.986 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:43.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:43.990 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:12:44.091 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:12:44.109 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:12:44.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:44.111 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:12:44.211 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:12:44.256 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 21:12:44 smithi146 bash[51661]: logger=cleanup t=2024-03-11T21:12:44.109586114Z level=info msg="Completed cleanup jobs" duration=55.797931ms 2024-03-11T21:12:44.256 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:12:44.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:44.261 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:12:44.362 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:12:44.388 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:12:44.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:44.392 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:12:44.492 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:12:44.518 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:12:44.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:44.522 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:12:44.623 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:12:44.642 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:12:44.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:44.644 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:12:44.745 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:12:44.756 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:44 smithi146 bash[23809]: cluster 2024-03-11T21:12:43.058423+0000 osd.7 (osd.7) 521 : cluster 0 3.13 scrub starts 2024-03-11T21:12:44.757 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:44 smithi146 bash[23809]: cluster 2024-03-11T21:12:43.060106+0000 osd.7 (osd.7) 522 : cluster 0 3.13 scrub ok 2024-03-11T21:12:44.757 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:44 smithi146 bash[23809]: cluster 2024-03-11T21:12:43.923708+0000 osd.1 (osd.1) 291 : cluster 0 2.4 scrub starts 2024-03-11T21:12:44.757 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:44 smithi146 bash[23809]: cluster 2024-03-11T21:12:43.931258+0000 osd.1 (osd.1) 292 : cluster 0 2.4 scrub ok 2024-03-11T21:12:44.757 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:44 smithi146 bash[23809]: cluster 2024-03-11T21:12:44.048394+0000 osd.6 (osd.6) 396 : cluster 0 2.17 scrub starts 2024-03-11T21:12:44.757 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:44 smithi146 bash[23809]: cluster 2024-03-11T21:12:44.053380+0000 osd.6 (osd.6) 397 : cluster 0 2.17 scrub ok 2024-03-11T21:12:44.778 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:12:44.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:44 smithi023 bash[21372]: cluster 2024-03-11T21:12:43.058423+0000 osd.7 (osd.7) 521 : cluster 0 3.13 scrub starts 2024-03-11T21:12:44.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:44 smithi023 bash[21372]: cluster 2024-03-11T21:12:43.060106+0000 osd.7 (osd.7) 522 : cluster 0 3.13 scrub ok 2024-03-11T21:12:44.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:44 smithi023 bash[21372]: cluster 2024-03-11T21:12:43.923708+0000 osd.1 (osd.1) 291 : cluster 0 2.4 scrub starts 2024-03-11T21:12:44.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:44 smithi023 bash[21372]: cluster 2024-03-11T21:12:43.931258+0000 osd.1 (osd.1) 292 : cluster 0 2.4 scrub ok 2024-03-11T21:12:44.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:44 smithi023 bash[21372]: cluster 2024-03-11T21:12:44.048394+0000 osd.6 (osd.6) 396 : cluster 0 2.17 scrub starts 2024-03-11T21:12:44.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:44 smithi023 bash[21372]: cluster 2024-03-11T21:12:44.053380+0000 osd.6 (osd.6) 397 : cluster 0 2.17 scrub ok 2024-03-11T21:12:44.781 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:44 smithi023 bash[29146]: cluster 2024-03-11T21:12:43.058423+0000 osd.7 (osd.7) 521 : cluster 0 3.13 scrub starts 2024-03-11T21:12:44.781 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:44 smithi023 bash[29146]: cluster 2024-03-11T21:12:43.060106+0000 osd.7 (osd.7) 522 : cluster 0 3.13 scrub ok 2024-03-11T21:12:44.781 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:44 smithi023 bash[29146]: cluster 2024-03-11T21:12:43.923708+0000 osd.1 (osd.1) 291 : cluster 0 2.4 scrub starts 2024-03-11T21:12:44.781 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:44 smithi023 bash[29146]: cluster 2024-03-11T21:12:43.931258+0000 osd.1 (osd.1) 292 : cluster 0 2.4 scrub ok 2024-03-11T21:12:44.781 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:44 smithi023 bash[29146]: cluster 2024-03-11T21:12:44.048394+0000 osd.6 (osd.6) 396 : cluster 0 2.17 scrub starts 2024-03-11T21:12:44.781 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:44 smithi023 bash[29146]: cluster 2024-03-11T21:12:44.053380+0000 osd.6 (osd.6) 397 : cluster 0 2.17 scrub ok 2024-03-11T21:12:44.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:44.782 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:12:44.835 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:2024-03-11T21:12:44.847715+0000 min lat: 0.00123771 max lat: 0.48987 avg lat: 0.0112415 2024-03-11T21:12:44.835 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-11T21:12:44.835 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 40 16 56926 56910 88.9017 63.75 0.00775676 0.0112415 2024-03-11T21:12:44.882 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:44.900 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:44.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:44.902 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:12:45.003 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:12:45.027 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:12:45.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:45.029 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:12:45.129 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:12:45.167 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:12:45.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:45.172 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:12:45.272 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:12:45.301 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:12:45.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:45.309 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:12:45.410 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:12:45.444 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:12:45.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:45.449 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:12:45.549 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:45.560 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:45 smithi146 bash[23809]: cluster 2024-03-11T21:12:44.379324+0000 mgr.y (mgr.24380) 812 : cluster 0 pgmap v930: 158 pgs: 158 active+clean; 3.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 65 MiB/s wr, 1.04k op/s 2024-03-11T21:12:45.574 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:45.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:45.577 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:12:45.678 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:12:45.704 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:45 smithi023 bash[21372]: cluster 2024-03-11T21:12:44.379324+0000 mgr.y (mgr.24380) 812 : cluster 0 pgmap v930: 158 pgs: 158 active+clean; 3.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 65 MiB/s wr, 1.04k op/s 2024-03-11T21:12:45.704 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:45 smithi023 bash[29146]: cluster 2024-03-11T21:12:44.379324+0000 mgr.y (mgr.24380) 812 : cluster 0 pgmap v930: 158 pgs: 158 active+clean; 3.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 65 MiB/s wr, 1.04k op/s 2024-03-11T21:12:45.706 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:12:45.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:45.713 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:12:45.814 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:12:45.835 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 41 16 57790 57774 88.0504 54 0.0177574 0.0113518 2024-03-11T21:12:45.849 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:12:45.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:45.853 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:12:45.954 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:12:45.980 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:12:45.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:45.985 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:12:46.086 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:46.094 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:12:45 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:12:46.104 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:46.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:46.106 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:12:46.207 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:46.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:46.242 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:46.242 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:12:46.345 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:46.372 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:46.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:46.374 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:12:46.475 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:12:46.502 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:12:46.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:46.508 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:12:46.609 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:12:46.618 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:46 smithi146 bash[23809]: cluster 2024-03-11T21:12:44.902972+0000 osd.0 (osd.0) 384 : cluster 0 2.f scrub starts 2024-03-11T21:12:46.619 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:46 smithi146 bash[23809]: cluster 2024-03-11T21:12:44.904266+0000 osd.0 (osd.0) 385 : cluster 0 2.f scrub ok 2024-03-11T21:12:46.619 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:46 smithi146 bash[23809]: cluster 2024-03-11T21:12:45.299877+0000 osd.3 (osd.3) 462 : cluster 0 6.f scrub starts 2024-03-11T21:12:46.619 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:46 smithi146 bash[23809]: cluster 2024-03-11T21:12:45.400181+0000 osd.5 (osd.5) 485 : cluster 0 6.19 deep-scrub starts 2024-03-11T21:12:46.619 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:46 smithi146 bash[23809]: cluster 2024-03-11T21:12:45.484644+0000 osd.5 (osd.5) 486 : cluster 0 6.19 deep-scrub ok 2024-03-11T21:12:46.619 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:46 smithi146 bash[23809]: cluster 2024-03-11T21:12:45.541612+0000 osd.3 (osd.3) 463 : cluster 0 6.f scrub ok 2024-03-11T21:12:46.619 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:46 smithi146 bash[23809]: cluster 2024-03-11T21:12:46.030734+0000 osd.6 (osd.6) 398 : cluster 0 4.a scrub starts 2024-03-11T21:12:46.619 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:46 smithi146 bash[23809]: cluster 2024-03-11T21:12:46.033763+0000 osd.6 (osd.6) 399 : cluster 0 4.a scrub ok 2024-03-11T21:12:46.633 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:12:46.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:46.636 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:12:46.736 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:46.757 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:46.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:46.759 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:12:46.776 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:46 smithi023 bash[21372]: cluster 2024-03-11T21:12:44.902972+0000 osd.0 (osd.0) 384 : cluster 0 2.f scrub starts 2024-03-11T21:12:46.776 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:46 smithi023 bash[21372]: cluster 2024-03-11T21:12:44.904266+0000 osd.0 (osd.0) 385 : cluster 0 2.f scrub ok 2024-03-11T21:12:46.777 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:46 smithi023 bash[21372]: cluster 2024-03-11T21:12:45.299877+0000 osd.3 (osd.3) 462 : cluster 0 6.f scrub starts 2024-03-11T21:12:46.777 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:46 smithi023 bash[21372]: cluster 2024-03-11T21:12:45.400181+0000 osd.5 (osd.5) 485 : cluster 0 6.19 deep-scrub starts 2024-03-11T21:12:46.777 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:46 smithi023 bash[21372]: cluster 2024-03-11T21:12:45.484644+0000 osd.5 (osd.5) 486 : cluster 0 6.19 deep-scrub ok 2024-03-11T21:12:46.777 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:46 smithi023 bash[21372]: cluster 2024-03-11T21:12:45.541612+0000 osd.3 (osd.3) 463 : cluster 0 6.f scrub ok 2024-03-11T21:12:46.777 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:46 smithi023 bash[21372]: cluster 2024-03-11T21:12:46.030734+0000 osd.6 (osd.6) 398 : cluster 0 4.a scrub starts 2024-03-11T21:12:46.777 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:46 smithi023 bash[21372]: cluster 2024-03-11T21:12:46.033763+0000 osd.6 (osd.6) 399 : cluster 0 4.a scrub ok 2024-03-11T21:12:46.777 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:46 smithi023 bash[29146]: cluster 2024-03-11T21:12:44.902972+0000 osd.0 (osd.0) 384 : cluster 0 2.f scrub starts 2024-03-11T21:12:46.777 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:46 smithi023 bash[29146]: cluster 2024-03-11T21:12:44.904266+0000 osd.0 (osd.0) 385 : cluster 0 2.f scrub ok 2024-03-11T21:12:46.777 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:46 smithi023 bash[29146]: cluster 2024-03-11T21:12:45.299877+0000 osd.3 (osd.3) 462 : cluster 0 6.f scrub starts 2024-03-11T21:12:46.777 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:46 smithi023 bash[29146]: cluster 2024-03-11T21:12:45.400181+0000 osd.5 (osd.5) 485 : cluster 0 6.19 deep-scrub starts 2024-03-11T21:12:46.777 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:46 smithi023 bash[29146]: cluster 2024-03-11T21:12:45.484644+0000 osd.5 (osd.5) 486 : cluster 0 6.19 deep-scrub ok 2024-03-11T21:12:46.777 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:46 smithi023 bash[29146]: cluster 2024-03-11T21:12:45.541612+0000 osd.3 (osd.3) 463 : cluster 0 6.f scrub ok 2024-03-11T21:12:46.777 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:46 smithi023 bash[29146]: cluster 2024-03-11T21:12:46.030734+0000 osd.6 (osd.6) 398 : cluster 0 4.a scrub starts 2024-03-11T21:12:46.778 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:46 smithi023 bash[29146]: cluster 2024-03-11T21:12:46.033763+0000 osd.6 (osd.6) 399 : cluster 0 4.a scrub ok 2024-03-11T21:12:46.835 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 42 16 58984 58968 87.7306 74.625 0.0142655 0.0113916 2024-03-11T21:12:46.860 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:12:46.882 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:12:46.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:46.884 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:12:46.985 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:12:47.013 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:12:47.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:47.017 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:12:47.118 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:12:47.157 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:12:47.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:47.161 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:12:47.261 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:12:47.286 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:12:47.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:47.289 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:12:47.390 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:47.415 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:47.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:47.418 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:12:47.519 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:12:47.541 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:12:47.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:47.543 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:12:47.644 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:47.654 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:47 smithi146 bash[23809]: cluster 2024-03-11T21:12:45.879568+0000 osd.0 (osd.0) 386 : cluster 0 2.13 scrub starts 2024-03-11T21:12:47.654 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:47 smithi146 bash[23809]: cluster 2024-03-11T21:12:45.881104+0000 osd.0 (osd.0) 387 : cluster 0 2.13 scrub ok 2024-03-11T21:12:47.654 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:47 smithi146 bash[23809]: audit 2024-03-11T21:12:45.945769+0000 mgr.y (mgr.24380) 813 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:12:47.655 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:47 smithi146 bash[23809]: cluster 2024-03-11T21:12:46.024216+0000 osd.5 (osd.5) 487 : cluster 0 purged_snaps scrub starts 2024-03-11T21:12:47.655 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:47 smithi146 bash[23809]: cluster 2024-03-11T21:12:46.024278+0000 osd.5 (osd.5) 488 : cluster 0 purged_snaps scrub ok 2024-03-11T21:12:47.655 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:47 smithi146 bash[23809]: cluster 2024-03-11T21:12:46.380243+0000 mgr.y (mgr.24380) 814 : cluster 0 pgmap v931: 158 pgs: 158 active+clean; 3.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 929 B/s rd, 50 MiB/s wr, 805 op/s 2024-03-11T21:12:47.655 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:47 smithi146 bash[23809]: cluster 2024-03-11T21:12:46.526821+0000 mon.a (mon.0) 1830 : cluster 0 osdmap e394: 8 total, 8 up, 8 in 2024-03-11T21:12:47.673 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:47.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:47.677 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:12:47.777 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:47 smithi023 bash[21372]: cluster 2024-03-11T21:12:45.879568+0000 osd.0 (osd.0) 386 : cluster 0 2.13 scrub starts 2024-03-11T21:12:47.777 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:47 smithi023 bash[21372]: cluster 2024-03-11T21:12:45.881104+0000 osd.0 (osd.0) 387 : cluster 0 2.13 scrub ok 2024-03-11T21:12:47.777 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:47 smithi023 bash[21372]: audit 2024-03-11T21:12:45.945769+0000 mgr.y (mgr.24380) 813 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:12:47.777 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:47 smithi023 bash[21372]: cluster 2024-03-11T21:12:46.024216+0000 osd.5 (osd.5) 487 : cluster 0 purged_snaps scrub starts 2024-03-11T21:12:47.777 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:47 smithi023 bash[21372]: cluster 2024-03-11T21:12:46.024278+0000 osd.5 (osd.5) 488 : cluster 0 purged_snaps scrub ok 2024-03-11T21:12:47.777 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:47 smithi023 bash[21372]: cluster 2024-03-11T21:12:46.380243+0000 mgr.y (mgr.24380) 814 : cluster 0 pgmap v931: 158 pgs: 158 active+clean; 3.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 929 B/s rd, 50 MiB/s wr, 805 op/s 2024-03-11T21:12:47.777 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:47 smithi023 bash[21372]: cluster 2024-03-11T21:12:46.526821+0000 mon.a (mon.0) 1830 : cluster 0 osdmap e394: 8 total, 8 up, 8 in 2024-03-11T21:12:47.777 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:47 smithi023 bash[29146]: cluster 2024-03-11T21:12:45.879568+0000 osd.0 (osd.0) 386 : cluster 0 2.13 scrub starts 2024-03-11T21:12:47.777 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:47 smithi023 bash[29146]: cluster 2024-03-11T21:12:45.881104+0000 osd.0 (osd.0) 387 : cluster 0 2.13 scrub ok 2024-03-11T21:12:47.777 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:47 smithi023 bash[29146]: audit 2024-03-11T21:12:45.945769+0000 mgr.y (mgr.24380) 813 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:12:47.778 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:47 smithi023 bash[29146]: cluster 2024-03-11T21:12:46.024216+0000 osd.5 (osd.5) 487 : cluster 0 purged_snaps scrub starts 2024-03-11T21:12:47.778 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:47 smithi023 bash[29146]: cluster 2024-03-11T21:12:46.024278+0000 osd.5 (osd.5) 488 : cluster 0 purged_snaps scrub ok 2024-03-11T21:12:47.778 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:47 smithi023 bash[29146]: cluster 2024-03-11T21:12:46.380243+0000 mgr.y (mgr.24380) 814 : cluster 0 pgmap v931: 158 pgs: 158 active+clean; 3.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 929 B/s rd, 50 MiB/s wr, 805 op/s 2024-03-11T21:12:47.778 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:47 smithi023 bash[29146]: cluster 2024-03-11T21:12:46.526821+0000 mon.a (mon.0) 1830 : cluster 0 osdmap e394: 8 total, 8 up, 8 in 2024-03-11T21:12:47.778 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:47.807 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:47.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:47.809 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:12:47.835 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 43 16 59880 59864 86.9927 56 0.00852183 0.0114895 2024-03-11T21:12:47.910 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:12:47.931 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:12:47.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:47.934 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:12:48.035 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:12:48.061 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:12:48.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:48.063 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:12:48.164 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:48.190 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:48.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:48.192 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:12:48.293 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:12:48.326 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:12:48.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:48.334 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:12:48.435 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:48.454 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:48.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:48.455 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:12:48.556 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:48.581 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:48.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:48.585 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:12:48.686 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:12:48.694 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:48 smithi023 bash[21372]: cluster 2024-03-11T21:12:47.831704+0000 osd.0 (osd.0) 388 : cluster 0 6.9 scrub starts 2024-03-11T21:12:48.695 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:12:48 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:12:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:12:48.695 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:48 smithi023 bash[29146]: cluster 2024-03-11T21:12:47.831704+0000 osd.0 (osd.0) 388 : cluster 0 6.9 scrub starts 2024-03-11T21:12:48.709 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:12:48.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:48.711 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:12:48.756 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:48 smithi146 bash[23809]: cluster 2024-03-11T21:12:47.831704+0000 osd.0 (osd.0) 388 : cluster 0 6.9 scrub starts 2024-03-11T21:12:48.813 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:48.834 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:48.835 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 44 16 60921 60905 86.4942 65.0625 0.00920837 0.0115553 2024-03-11T21:12:48.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:48.838 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:12:48.939 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:12:48.963 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:12:48.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:48.966 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:12:49.067 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:12:49.093 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:12:49.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:49.100 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:12:49.201 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:12:49.231 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:12:49.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:49.233 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:12:49.334 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:49.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:49.365 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:49.365 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:12:49.466 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:12:49.492 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:12:49.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:49.500 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:12:49.600 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:12:49.609 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:49 smithi146 bash[23809]: cluster 2024-03-11T21:12:47.959923+0000 osd.0 (osd.0) 389 : cluster 0 6.9 scrub ok 2024-03-11T21:12:49.609 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:49 smithi146 bash[23809]: cluster 2024-03-11T21:12:48.115018+0000 osd.7 (osd.7) 523 : cluster 0 3.15 scrub starts 2024-03-11T21:12:49.609 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:49 smithi146 bash[23809]: cluster 2024-03-11T21:12:48.118336+0000 osd.7 (osd.7) 524 : cluster 0 3.15 scrub ok 2024-03-11T21:12:49.609 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:49 smithi146 bash[23809]: cluster 2024-03-11T21:12:48.380842+0000 mgr.y (mgr.24380) 815 : cluster 0 pgmap v933: 158 pgs: 158 active+clean; 3.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 51 MiB/s wr, 821 op/s 2024-03-11T21:12:49.625 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:12:49.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:49.629 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:12:49.730 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:49.754 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:49.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:49.757 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:12:49.776 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:49 smithi023 bash[21372]: cluster 2024-03-11T21:12:47.959923+0000 osd.0 (osd.0) 389 : cluster 0 6.9 scrub ok 2024-03-11T21:12:49.777 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:49 smithi023 bash[21372]: cluster 2024-03-11T21:12:48.115018+0000 osd.7 (osd.7) 523 : cluster 0 3.15 scrub starts 2024-03-11T21:12:49.777 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:49 smithi023 bash[21372]: cluster 2024-03-11T21:12:48.118336+0000 osd.7 (osd.7) 524 : cluster 0 3.15 scrub ok 2024-03-11T21:12:49.777 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:49 smithi023 bash[21372]: cluster 2024-03-11T21:12:48.380842+0000 mgr.y (mgr.24380) 815 : cluster 0 pgmap v933: 158 pgs: 158 active+clean; 3.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 51 MiB/s wr, 821 op/s 2024-03-11T21:12:49.777 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:49 smithi023 bash[29146]: cluster 2024-03-11T21:12:47.959923+0000 osd.0 (osd.0) 389 : cluster 0 6.9 scrub ok 2024-03-11T21:12:49.777 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:49 smithi023 bash[29146]: cluster 2024-03-11T21:12:48.115018+0000 osd.7 (osd.7) 523 : cluster 0 3.15 scrub starts 2024-03-11T21:12:49.777 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:49 smithi023 bash[29146]: cluster 2024-03-11T21:12:48.118336+0000 osd.7 (osd.7) 524 : cluster 0 3.15 scrub ok 2024-03-11T21:12:49.777 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:49 smithi023 bash[29146]: cluster 2024-03-11T21:12:48.380842+0000 mgr.y (mgr.24380) 815 : cluster 0 pgmap v933: 158 pgs: 158 active+clean; 3.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 51 MiB/s wr, 821 op/s 2024-03-11T21:12:49.836 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 45 16 62130 62114 86.2512 75.5625 0.00956287 0.0115874 2024-03-11T21:12:49.857 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:12:49.902 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:12:49.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:49.908 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:12:50.008 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:12:50.031 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:12:50.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:50.034 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:12:50.135 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:12:50.159 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:12:50.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:50.163 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:12:50.264 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:12:50.288 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:12:50.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:50.290 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:12:50.391 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:12:50.418 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:12:50.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:50.422 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:12:50.523 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:12:50.556 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:12:50.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:50.564 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:12:50.664 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:12:50.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:50 smithi146 bash[23809]: cluster 2024-03-11T21:12:49.851935+0000 osd.1 (osd.1) 293 : cluster 0 4.7 deep-scrub starts 2024-03-11T21:12:50.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:50 smithi146 bash[23809]: cluster 2024-03-11T21:12:49.854660+0000 osd.1 (osd.1) 294 : cluster 0 4.7 deep-scrub ok 2024-03-11T21:12:50.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:50 smithi146 bash[23809]: cluster 2024-03-11T21:12:50.368191+0000 mon.a (mon.0) 1831 : cluster 0 osdmap e395: 8 total, 8 up, 8 in 2024-03-11T21:12:50.684 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:12:50.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:50.689 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:12:50.761 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-11T21:12:50.761 INFO:teuthology.orchestra.run.smithi023.stdout: "size": 20, 2024-03-11T21:12:50.761 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 600, 2024-03-11T21:12:50.761 INFO:teuthology.orchestra.run.smithi023.stdout: "ops": [ 2024-03-11T21:12:50.761 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.761 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_op(client.15498.0:28057 11.c 11:3f34bc98:::benchmark_data_smithi023_142591_object28056:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e391)", 2024-03-11T21:12:50.761 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T21:12:19.511226+0000", 2024-03-11T21:12:50.761 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 31.257399915000001, 2024-03-11T21:12:50.762 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.47578855199999998, 2024-03-11T21:12:50.762 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:12:50.762 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:12:50.762 INFO:teuthology.orchestra.run.smithi023.stdout: "client_info": { 2024-03-11T21:12:50.762 INFO:teuthology.orchestra.run.smithi023.stdout: "client": "client.15498", 2024-03-11T21:12:50.762 INFO:teuthology.orchestra.run.smithi023.stdout: "client_addr": "172.21.15.23:0/4267430685", 2024-03-11T21:12:50.762 INFO:teuthology.orchestra.run.smithi023.stdout: "tid": 28057 2024-03-11T21:12:50.763 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.763 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:12:50.763 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.763 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:12:50.763 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:19.511226+0000", 2024-03-11T21:12:50.763 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:12:50.763 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.763 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.763 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:12:50.763 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:19.511226+0000", 2024-03-11T21:12:50.763 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:12:50.763 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.764 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.764 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:12:50.764 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:19.511227+0000", 2024-03-11T21:12:50.764 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.731e-06 2024-03-11T21:12:50.764 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.764 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.764 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:12:50.764 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:19.511254+0000", 2024-03-11T21:12:50.764 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.6173000000000002e-05 2024-03-11T21:12:50.764 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.764 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.764 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:12:50.765 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:19.511266+0000", 2024-03-11T21:12:50.765 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.2140000000000001e-05 2024-03-11T21:12:50.765 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.765 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.765 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:12:50.765 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:19.511277+0000", 2024-03-11T21:12:50.765 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.1455e-05 2024-03-11T21:12:50.765 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.765 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.765 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:12:50.765 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:19.511343+0000", 2024-03-11T21:12:50.765 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.5370000000000006e-05 2024-03-11T21:12:50.765 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.766 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.766 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:12:50.766 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:19.511429+0000", 2024-03-11T21:12:50.766 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 8.6808e-05 2024-03-11T21:12:50.766 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.766 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.766 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for subops from 2,4", 2024-03-11T21:12:50.766 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:19.511554+0000", 2024-03-11T21:12:50.766 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00012433799999999999 2024-03-11T21:12:50.766 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.766 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.767 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "op_commit", 2024-03-11T21:12:50.767 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:19.512538+0000", 2024-03-11T21:12:50.767 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0009844840000000001 2024-03-11T21:12:50.767 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.767 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.767 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:12:50.767 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:19.513876+0000", 2024-03-11T21:12:50.767 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.001337838 2024-03-11T21:12:50.767 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.767 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.767 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:12:50.767 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:19.986820+0000", 2024-03-11T21:12:50.767 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.472944268 2024-03-11T21:12:50.768 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.768 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.768 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T21:12:50.768 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:19.986931+0000", 2024-03-11T21:12:50.768 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000110577 2024-03-11T21:12:50.768 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.768 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.768 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:12:50.768 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:19.987014+0000", 2024-03-11T21:12:50.768 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 8.3369999999999996e-05 2024-03-11T21:12:50.769 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:12:50.769 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:12:50.769 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:12:50.769 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.769 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.769 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_op(client.15498.0:28058 11.e 11:78adb2d9:::benchmark_data_smithi023_142591_object28057:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e391)", 2024-03-11T21:12:50.769 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T21:12:19.514086+0000", 2024-03-11T21:12:50.769 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 31.254539868999998, 2024-03-11T21:12:50.769 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.47877635299999999, 2024-03-11T21:12:50.769 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:12:50.769 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:12:50.770 INFO:teuthology.orchestra.run.smithi023.stdout: "client_info": { 2024-03-11T21:12:50.770 INFO:teuthology.orchestra.run.smithi023.stdout: "client": "client.15498", 2024-03-11T21:12:50.770 INFO:teuthology.orchestra.run.smithi023.stdout: "client_addr": "172.21.15.23:0/4267430685", 2024-03-11T21:12:50.770 INFO:teuthology.orchestra.run.smithi023.stdout: "tid": 28058 2024-03-11T21:12:50.770 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.770 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:12:50.770 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.770 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:12:50.770 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:19.514086+0000", 2024-03-11T21:12:50.770 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:12:50.770 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.770 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.771 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:12:50.771 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:19.514086+0000", 2024-03-11T21:12:50.771 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:12:50.771 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.771 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.771 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:12:50.772 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:19.514087+0000", 2024-03-11T21:12:50.772 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.1480000000000001e-06 2024-03-11T21:12:50.772 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.772 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.772 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:12:50.772 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:19.514107+0000", 2024-03-11T21:12:50.772 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.9925e-05 2024-03-11T21:12:50.772 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.772 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.772 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:12:50.772 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:19.514122+0000", 2024-03-11T21:12:50.773 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.4888999999999999e-05 2024-03-11T21:12:50.773 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.773 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.773 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:12:50.773 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:19.514130+0000", 2024-03-11T21:12:50.774 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.7640000000000005e-06 2024-03-11T21:12:50.774 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.774 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.774 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:12:50.774 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:19.514454+0000", 2024-03-11T21:12:50.774 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00032408099999999999 2024-03-11T21:12:50.774 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.774 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.774 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:12:50.775 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:19.514598+0000", 2024-03-11T21:12:50.775 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00014473000000000001 2024-03-11T21:12:50.775 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.775 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.775 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for subops from 4,7", 2024-03-11T21:12:50.775 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:19.514746+0000", 2024-03-11T21:12:50.775 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00014761400000000001 2024-03-11T21:12:50.775 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.775 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.775 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:12:50.776 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:19.516288+0000", 2024-03-11T21:12:50.776 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0015419609999999999 2024-03-11T21:12:50.776 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.776 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.776 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "op_commit", 2024-03-11T21:12:50.776 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:19.988205+0000", 2024-03-11T21:12:50.776 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.47191706999999999 2024-03-11T21:12:50.776 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.776 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.776 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:12:50.776 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:19.992739+0000", 2024-03-11T21:12:50.776 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0045336930000000001 2024-03-11T21:12:50.777 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.777 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.777 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T21:12:50.777 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:19.992810+0000", 2024-03-11T21:12:50.777 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.0915999999999997e-05 2024-03-11T21:12:50.777 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.777 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.777 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:12:50.777 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:19.992862+0000", 2024-03-11T21:12:50.777 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 5.2562000000000003e-05 2024-03-11T21:12:50.777 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:12:50.777 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:12:50.778 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:12:50.778 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.778 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.778 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_op(client.15498.0:28060 11.c 11:3481f6a4:::benchmark_data_smithi023_142591_object28059:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e391)", 2024-03-11T21:12:50.778 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T21:12:19.514957+0000", 2024-03-11T21:12:50.778 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 31.253668634, 2024-03-11T21:12:50.778 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.47570240000000003, 2024-03-11T21:12:50.778 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:12:50.778 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:12:50.778 INFO:teuthology.orchestra.run.smithi023.stdout: "client_info": { 2024-03-11T21:12:50.778 INFO:teuthology.orchestra.run.smithi023.stdout: "client": "client.15498", 2024-03-11T21:12:50.779 INFO:teuthology.orchestra.run.smithi023.stdout: "client_addr": "172.21.15.23:0/4267430685", 2024-03-11T21:12:50.779 INFO:teuthology.orchestra.run.smithi023.stdout: "tid": 28060 2024-03-11T21:12:50.779 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.779 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:12:50.779 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.779 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:12:50.779 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:19.514957+0000", 2024-03-11T21:12:50.779 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:12:50.779 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.779 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.779 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:12:50.780 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:19.514957+0000", 2024-03-11T21:12:50.780 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:12:50.780 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.780 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.780 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:12:50.780 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:19.514958+0000", 2024-03-11T21:12:50.780 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 9.9699999999999994e-07 2024-03-11T21:12:50.780 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.780 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.780 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:12:50.780 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:19.514978+0000", 2024-03-11T21:12:50.780 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.0171000000000001e-05 2024-03-11T21:12:50.780 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.781 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.781 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:12:50.781 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:19.514989+0000", 2024-03-11T21:12:50.781 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.1287000000000001e-05 2024-03-11T21:12:50.781 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.781 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.782 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:12:50.782 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:19.514997+0000", 2024-03-11T21:12:50.782 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.6040000000000004e-06 2024-03-11T21:12:50.782 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.782 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.782 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:12:50.782 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:19.515049+0000", 2024-03-11T21:12:50.782 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 5.1666000000000003e-05 2024-03-11T21:12:50.782 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.782 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.783 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:12:50.783 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:19.515170+0000", 2024-03-11T21:12:50.783 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00012134900000000001 2024-03-11T21:12:50.783 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.783 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.783 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for subops from 2,4", 2024-03-11T21:12:50.784 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:19.515301+0000", 2024-03-11T21:12:50.784 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000130882 2024-03-11T21:12:50.784 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.784 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.784 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:12:50.784 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:19.516861+0000", 2024-03-11T21:12:50.784 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0015602210000000001 2024-03-11T21:12:50.784 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.784 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.784 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "op_commit", 2024-03-11T21:12:50.784 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:19.987060+0000", 2024-03-11T21:12:50.784 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.47019871200000002 2024-03-11T21:12:50.785 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.785 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.785 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:12:50.785 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:19.990505+0000", 2024-03-11T21:12:50.785 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0034447950000000001 2024-03-11T21:12:50.785 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.785 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.785 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T21:12:50.785 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:19.990596+0000", 2024-03-11T21:12:50.785 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 9.0895000000000003e-05 2024-03-11T21:12:50.785 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.785 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.786 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:12:50.786 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:19.990659+0000", 2024-03-11T21:12:50.786 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.3820999999999998e-05 2024-03-11T21:12:50.786 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:12:50.786 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:12:50.786 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:12:50.786 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.786 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.786 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_repop(client.15498.0:28062 11.3 e391/380 11:c6a6d30c:::benchmark_data_smithi023_142591_object28061:head v 391'1710, mlcod=391'1707)", 2024-03-11T21:12:50.786 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T21:12:19.516886+0000", 2024-03-11T21:12:50.786 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 31.251739718, 2024-03-11T21:12:50.786 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.47384282, 2024-03-11T21:12:50.787 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:12:50.787 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:12:50.787 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:12:50.787 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.787 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:12:50.787 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:19.516886+0000", 2024-03-11T21:12:50.787 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:12:50.787 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.787 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.787 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:12:50.787 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:19.516886+0000", 2024-03-11T21:12:50.787 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:12:50.788 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.788 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.788 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:12:50.788 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:19.516887+0000", 2024-03-11T21:12:50.788 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.1900000000000002e-07 2024-03-11T21:12:50.788 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.788 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.788 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:12:50.788 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:19.516914+0000", 2024-03-11T21:12:50.788 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.7917000000000001e-05 2024-03-11T21:12:50.788 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.788 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.788 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:12:50.789 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:19.516924+0000", 2024-03-11T21:12:50.789 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 9.5019999999999998e-06 2024-03-11T21:12:50.789 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.789 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.789 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:12:50.789 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:19.516931+0000", 2024-03-11T21:12:50.789 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.8290000000000002e-06 2024-03-11T21:12:50.789 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.789 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.789 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:12:50.789 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:19.517022+0000", 2024-03-11T21:12:50.789 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 9.1119999999999995e-05 2024-03-11T21:12:50.789 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.790 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.790 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:12:50.790 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:19.517041+0000", 2024-03-11T21:12:50.790 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.8698999999999998e-05 2024-03-11T21:12:50.790 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.790 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.790 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T21:12:50.790 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:19.990681+0000", 2024-03-11T21:12:50.790 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.47364057100000001 2024-03-11T21:12:50.790 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.790 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.790 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:12:50.791 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:19.990729+0000", 2024-03-11T21:12:50.791 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 4.7562999999999998e-05 2024-03-11T21:12:50.791 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:12:50.791 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:12:50.791 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:12:50.791 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.791 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.791 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_repop_reply(client.15498.0:28057 11.c e391/380 ondisk, result = 0)", 2024-03-11T21:12:50.791 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T21:12:19.516960+0000", 2024-03-11T21:12:50.791 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 31.251666145000002, 2024-03-11T21:12:50.791 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.47006861, 2024-03-11T21:12:50.791 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:12:50.791 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "started", 2024-03-11T21:12:50.792 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:12:50.792 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.792 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:12:50.792 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:19.516960+0000", 2024-03-11T21:12:50.792 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:12:50.792 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.792 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.792 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:12:50.792 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:19.516960+0000", 2024-03-11T21:12:50.792 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:12:50.792 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.792 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.793 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:12:50.793 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:19.516960+0000", 2024-03-11T21:12:50.793 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 4.3599999999999999e-07 2024-03-11T21:12:50.793 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.793 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.793 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:12:50.793 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:19.516963+0000", 2024-03-11T21:12:50.793 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 3.36e-06 2024-03-11T21:12:50.793 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.793 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.793 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:12:50.793 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:19.516971+0000", 2024-03-11T21:12:50.793 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.994e-06 2024-03-11T21:12:50.794 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.794 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.794 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:12:50.794 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:19.516977+0000", 2024-03-11T21:12:50.794 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 5.5770000000000001e-06 2024-03-11T21:12:50.794 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.794 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.794 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:12:50.794 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:19.986790+0000", 2024-03-11T21:12:50.794 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.46981290799999997 2024-03-11T21:12:50.794 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.794 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.794 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:12:50.795 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:19.986808+0000", 2024-03-11T21:12:50.795 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.8553e-05 2024-03-11T21:12:50.795 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.795 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.795 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:12:50.795 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:19.987028+0000", 2024-03-11T21:12:50.795 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00021978199999999999 2024-03-11T21:12:50.795 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:12:50.795 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:12:50.795 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:12:50.795 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.795 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.795 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_repop(client.15498.0:28065 11.3 e391/380 11:c8127384:::benchmark_data_smithi023_142591_object28064:head v 391'1711, mlcod=391'1707)", 2024-03-11T21:12:50.796 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T21:12:19.517814+0000", 2024-03-11T21:12:50.796 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 31.250811458000001, 2024-03-11T21:12:50.796 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.47815042600000002, 2024-03-11T21:12:50.796 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:12:50.796 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:12:50.796 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:12:50.796 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.796 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:12:50.796 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:19.517814+0000", 2024-03-11T21:12:50.796 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:12:50.796 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.796 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.797 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:12:50.797 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:19.517814+0000", 2024-03-11T21:12:50.797 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:12:50.797 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.797 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.797 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:12:50.797 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:19.517815+0000", 2024-03-11T21:12:50.797 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.0900000000000001e-07 2024-03-11T21:12:50.797 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.797 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.797 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:12:50.797 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:19.517879+0000", 2024-03-11T21:12:50.798 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.4172999999999997e-05 2024-03-11T21:12:50.798 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.798 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.798 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:12:50.798 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:19.987379+0000", 2024-03-11T21:12:50.798 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.46949991600000002 2024-03-11T21:12:50.798 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.798 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.798 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:12:50.798 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:19.987393+0000", 2024-03-11T21:12:50.798 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.3574000000000001e-05 2024-03-11T21:12:50.798 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.799 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.799 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:12:50.799 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:19.989454+0000", 2024-03-11T21:12:50.799 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0020611520000000001 2024-03-11T21:12:50.799 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.799 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.799 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:12:50.799 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:19.989477+0000", 2024-03-11T21:12:50.799 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.3180000000000002e-05 2024-03-11T21:12:50.799 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.799 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.799 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T21:12:50.799 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:19.995902+0000", 2024-03-11T21:12:50.800 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0064248719999999999 2024-03-11T21:12:50.800 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.800 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.800 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:12:50.800 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:19.995965+0000", 2024-03-11T21:12:50.800 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.2849999999999996e-05 2024-03-11T21:12:50.800 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:12:50.800 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:12:50.800 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:12:50.800 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.800 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.800 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_repop(client.15498.0:28063 11.6 e391/380 11:6c3f9a56:::benchmark_data_smithi023_142591_object28062:head v 391'1760, mlcod=391'1758)", 2024-03-11T21:12:50.800 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T21:12:19.518029+0000", 2024-03-11T21:12:50.801 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 31.250596709, 2024-03-11T21:12:50.801 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.47720933100000001, 2024-03-11T21:12:50.801 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:12:50.801 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:12:50.801 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:12:50.801 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.801 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:12:50.801 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:19.518029+0000", 2024-03-11T21:12:50.801 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:12:50.801 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.801 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.801 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:12:50.802 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:19.518029+0000", 2024-03-11T21:12:50.802 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:12:50.802 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.802 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.802 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:12:50.802 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:19.518030+0000", 2024-03-11T21:12:50.802 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.8700000000000005e-07 2024-03-11T21:12:50.802 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.802 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.802 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:12:50.802 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:19.518052+0000", 2024-03-11T21:12:50.802 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.2569e-05 2024-03-11T21:12:50.803 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.803 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.803 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:12:50.803 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:19.986442+0000", 2024-03-11T21:12:50.803 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.46838985 2024-03-11T21:12:50.803 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.803 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.803 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:12:50.803 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:19.986464+0000", 2024-03-11T21:12:50.803 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.1803000000000001e-05 2024-03-11T21:12:50.803 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.803 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.804 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:12:50.804 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:19.988255+0000", 2024-03-11T21:12:50.804 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.001790687 2024-03-11T21:12:50.804 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.804 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.804 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:12:50.804 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:19.988290+0000", 2024-03-11T21:12:50.804 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 3.5079000000000001e-05 2024-03-11T21:12:50.804 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.804 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.804 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T21:12:50.804 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:19.995171+0000", 2024-03-11T21:12:50.804 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0068817560000000002 2024-03-11T21:12:50.805 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.805 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.805 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:12:50.805 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:19.995238+0000", 2024-03-11T21:12:50.805 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.69e-05 2024-03-11T21:12:50.805 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:12:50.805 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:12:50.805 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:12:50.805 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.806 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.806 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_op(client.15498.0:38778 11.c 11:3a37bc44:::benchmark_data_smithi023_142591_object38777:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e391)", 2024-03-11T21:12:50.806 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T21:12:27.143294+0000", 2024-03-11T21:12:50.806 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 23.625331323000001, 2024-03-11T21:12:50.806 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.28514709599999999, 2024-03-11T21:12:50.806 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:12:50.806 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:12:50.806 INFO:teuthology.orchestra.run.smithi023.stdout: "client_info": { 2024-03-11T21:12:50.806 INFO:teuthology.orchestra.run.smithi023.stdout: "client": "client.15498", 2024-03-11T21:12:50.806 INFO:teuthology.orchestra.run.smithi023.stdout: "client_addr": "172.21.15.23:0/4267430685", 2024-03-11T21:12:50.806 INFO:teuthology.orchestra.run.smithi023.stdout: "tid": 38778 2024-03-11T21:12:50.806 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.807 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:12:50.807 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.807 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:12:50.807 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:27.143294+0000", 2024-03-11T21:12:50.807 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:12:50.807 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.807 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.807 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:12:50.807 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:27.143294+0000", 2024-03-11T21:12:50.807 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:12:50.807 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.807 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.807 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:12:50.808 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:27.143296+0000", 2024-03-11T21:12:50.808 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.897e-06 2024-03-11T21:12:50.808 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.808 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.808 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:12:50.808 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:27.143326+0000", 2024-03-11T21:12:50.808 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 3.0040999999999999e-05 2024-03-11T21:12:50.808 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.808 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.808 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:12:50.808 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:27.143339+0000", 2024-03-11T21:12:50.808 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.26e-05 2024-03-11T21:12:50.809 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.809 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.809 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:12:50.809 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:27.143347+0000", 2024-03-11T21:12:50.809 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.9030000000000004e-06 2024-03-11T21:12:50.809 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.809 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.809 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:12:50.809 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:27.143395+0000", 2024-03-11T21:12:50.809 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 4.8136999999999997e-05 2024-03-11T21:12:50.809 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.809 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.809 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:12:50.810 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:27.143539+0000", 2024-03-11T21:12:50.810 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000143972 2024-03-11T21:12:50.810 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.810 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.810 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for subops from 2,4", 2024-03-11T21:12:50.810 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:27.143682+0000", 2024-03-11T21:12:50.810 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00014271600000000001 2024-03-11T21:12:50.810 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.810 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.810 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "op_commit", 2024-03-11T21:12:50.810 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:27.145664+0000", 2024-03-11T21:12:50.810 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0019827109999999998 2024-03-11T21:12:50.810 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.811 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.811 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:12:50.811 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:27.149818+0000", 2024-03-11T21:12:50.811 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0041540819999999999 2024-03-11T21:12:50.811 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.811 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.811 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:12:50.811 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:27.428211+0000", 2024-03-11T21:12:50.811 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.278392743 2024-03-11T21:12:50.811 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.811 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.811 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T21:12:50.812 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:27.428341+0000", 2024-03-11T21:12:50.812 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00013019400000000001 2024-03-11T21:12:50.812 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.812 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.812 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:12:50.812 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:27.428441+0000", 2024-03-11T21:12:50.812 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00010009999999999999 2024-03-11T21:12:50.812 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:12:50.812 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:12:50.812 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:12:50.812 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.812 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.812 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_op(client.15498.0:47415 11.c 11:35b86436:::benchmark_data_smithi023_142591_object47414:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e391)", 2024-03-11T21:12:50.813 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T21:12:33.530073+0000", 2024-03-11T21:12:50.813 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 17.238553104000001, 2024-03-11T21:12:50.813 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.28972703799999999, 2024-03-11T21:12:50.813 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:12:50.813 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:12:50.813 INFO:teuthology.orchestra.run.smithi023.stdout: "client_info": { 2024-03-11T21:12:50.813 INFO:teuthology.orchestra.run.smithi023.stdout: "client": "client.15498", 2024-03-11T21:12:50.813 INFO:teuthology.orchestra.run.smithi023.stdout: "client_addr": "172.21.15.23:0/4267430685", 2024-03-11T21:12:50.813 INFO:teuthology.orchestra.run.smithi023.stdout: "tid": 47415 2024-03-11T21:12:50.813 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.813 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:12:50.813 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.814 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:12:50.814 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:33.530073+0000", 2024-03-11T21:12:50.814 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:12:50.814 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.814 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.814 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:12:50.814 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:33.530073+0000", 2024-03-11T21:12:50.814 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:12:50.814 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.814 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.814 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:12:50.814 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:33.530075+0000", 2024-03-11T21:12:50.814 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.9939999999999999e-06 2024-03-11T21:12:50.815 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.815 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.815 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:12:50.815 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:33.530104+0000", 2024-03-11T21:12:50.815 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.9620000000000001e-05 2024-03-11T21:12:50.815 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.815 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.815 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:12:50.815 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:33.530124+0000", 2024-03-11T21:12:50.815 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.9884000000000001e-05 2024-03-11T21:12:50.815 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.815 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.816 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:12:50.816 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:33.530137+0000", 2024-03-11T21:12:50.816 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.2524000000000001e-05 2024-03-11T21:12:50.816 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.816 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.816 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:12:50.816 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:33.530214+0000", 2024-03-11T21:12:50.816 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.7052999999999996e-05 2024-03-11T21:12:50.816 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.816 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.816 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:12:50.816 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:33.530371+0000", 2024-03-11T21:12:50.816 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00015726999999999999 2024-03-11T21:12:50.817 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.817 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.817 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for subops from 2,4", 2024-03-11T21:12:50.817 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:33.530552+0000", 2024-03-11T21:12:50.817 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000181182 2024-03-11T21:12:50.817 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.817 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.817 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:12:50.817 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:33.534084+0000", 2024-03-11T21:12:50.817 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0035324029999999999 2024-03-11T21:12:50.817 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.817 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.817 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "op_commit", 2024-03-11T21:12:50.818 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:33.534115+0000", 2024-03-11T21:12:50.818 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 3.0620999999999997e-05 2024-03-11T21:12:50.818 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.818 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.818 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:12:50.818 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:33.819659+0000", 2024-03-11T21:12:50.818 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.285544042 2024-03-11T21:12:50.818 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.818 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.818 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T21:12:50.818 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:33.819744+0000", 2024-03-11T21:12:50.818 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 8.4361000000000001e-05 2024-03-11T21:12:50.819 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.819 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.819 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:12:50.819 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:33.819800+0000", 2024-03-11T21:12:50.819 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 5.6084e-05 2024-03-11T21:12:50.819 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:12:50.819 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:12:50.819 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:12:50.819 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.819 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.819 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_op(client.15498.0:52879 11.0 11:0713afe9:::benchmark_data_smithi023_142591_object52878:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e393)", 2024-03-11T21:12:50.819 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T21:12:39.363669+0000", 2024-03-11T21:12:50.819 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 11.404956890999999, 2024-03-11T21:12:50.820 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.334709959, 2024-03-11T21:12:50.820 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:12:50.820 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:12:50.820 INFO:teuthology.orchestra.run.smithi023.stdout: "client_info": { 2024-03-11T21:12:50.820 INFO:teuthology.orchestra.run.smithi023.stdout: "client": "client.15498", 2024-03-11T21:12:50.820 INFO:teuthology.orchestra.run.smithi023.stdout: "client_addr": "172.21.15.23:0/4267430685", 2024-03-11T21:12:50.820 INFO:teuthology.orchestra.run.smithi023.stdout: "tid": 52879 2024-03-11T21:12:50.820 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.820 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:12:50.820 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.820 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:12:50.820 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:39.363669+0000", 2024-03-11T21:12:50.821 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:12:50.821 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.821 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.821 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:12:50.821 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:39.363669+0000", 2024-03-11T21:12:50.821 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:12:50.821 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.821 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.821 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:12:50.821 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:39.363670+0000", 2024-03-11T21:12:50.821 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.2699999999999999e-06 2024-03-11T21:12:50.821 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.821 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.822 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:12:50.822 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:39.363731+0000", 2024-03-11T21:12:50.822 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.1230999999999997e-05 2024-03-11T21:12:50.822 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.822 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.822 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:12:50.822 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:39.363746+0000", 2024-03-11T21:12:50.822 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.4499e-05 2024-03-11T21:12:50.822 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.822 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.822 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:12:50.822 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:39.363755+0000", 2024-03-11T21:12:50.823 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 9.6330000000000008e-06 2024-03-11T21:12:50.823 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.823 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.824 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:12:50.824 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:39.363851+0000", 2024-03-11T21:12:50.824 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 9.5812999999999999e-05 2024-03-11T21:12:50.824 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.824 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.824 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:12:50.824 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:39.363940+0000", 2024-03-11T21:12:50.824 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 8.9130999999999998e-05 2024-03-11T21:12:50.824 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.824 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.825 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for subops from 3,7", 2024-03-11T21:12:50.825 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:39.364027+0000", 2024-03-11T21:12:50.825 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 8.6471999999999995e-05 2024-03-11T21:12:50.825 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.825 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.825 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "op_commit", 2024-03-11T21:12:50.825 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:39.365548+0000", 2024-03-11T21:12:50.825 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0015207510000000001 2024-03-11T21:12:50.825 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.825 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.825 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:12:50.826 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:39.369990+0000", 2024-03-11T21:12:50.826 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0044423780000000003 2024-03-11T21:12:50.826 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.826 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.826 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:12:50.826 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:39.698188+0000", 2024-03-11T21:12:50.826 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.32819785600000001 2024-03-11T21:12:50.826 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.827 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.827 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T21:12:50.827 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:39.698300+0000", 2024-03-11T21:12:50.827 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000112256 2024-03-11T21:12:50.827 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.827 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.827 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:12:50.827 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:39.698379+0000", 2024-03-11T21:12:50.827 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.8669000000000002e-05 2024-03-11T21:12:50.827 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:12:50.828 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:12:50.828 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:12:50.828 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.828 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.828 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:12:50.830 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_op(client.15498.0:52898 11.c 11:3e843369:::benchmark_data_smithi023_142591_object52897:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e393)", 2024-03-11T21:12:50.831 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T21:12:39.377790+0000", 2024-03-11T21:12:50.831 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 11.390836033999999, 2024-03-11T21:12:50.831 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.30676693300000002, 2024-03-11T21:12:50.831 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:12:50.831 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:12:50.831 INFO:teuthology.orchestra.run.smithi023.stdout: "client_info": { 2024-03-11T21:12:50.831 INFO:teuthology.orchestra.run.smithi023.stdout: "client": "client.15498", 2024-03-11T21:12:50.831 INFO:teuthology.orchestra.run.smithi023.stdout: "client_addr": "172.21.15.23:0/4267430685", 2024-03-11T21:12:50.832 INFO:teuthology.orchestra.run.smithi023.stdout: "tid": 52898 2024-03-11T21:12:50.832 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.832 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:12:50.832 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.832 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:12:50.832 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:39.377790+0000", 2024-03-11T21:12:50.832 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:12:50.832 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.832 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.833 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:12:50.833 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:39.377790+0000", 2024-03-11T21:12:50.833 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:12:50.833 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.833 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.833 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:12:50.833 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:39.377791+0000", 2024-03-11T21:12:50.833 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.189e-06 2024-03-11T21:12:50.834 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.834 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.834 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:12:50.834 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:39.377810+0000", 2024-03-11T21:12:50.834 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.9371000000000001e-05 2024-03-11T21:12:50.834 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.834 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.834 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:12:50.834 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:39.377821+0000", 2024-03-11T21:12:50.834 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.0385e-05 2024-03-11T21:12:50.835 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.835 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.835 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:12:50.835 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:39.377828+0000", 2024-03-11T21:12:50.835 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.3989999999999998e-06 2024-03-11T21:12:50.835 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.835 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.835 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:12:50.836 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:39.377877+0000", 2024-03-11T21:12:50.836 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 4.8816000000000002e-05 2024-03-11T21:12:50.836 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.836 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.836 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:12:50.836 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:39.378002+0000", 2024-03-11T21:12:50.836 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00012523300000000001 2024-03-11T21:12:50.836 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.836 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.836 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for subops from 2,4", 2024-03-11T21:12:50.837 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:39.378126+0000", 2024-03-11T21:12:50.837 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000124213 2024-03-11T21:12:50.837 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.837 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.837 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "op_commit", 2024-03-11T21:12:50.837 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:39.379891+0000", 2024-03-11T21:12:50.837 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.001764481 2024-03-11T21:12:50.837 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.837 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.837 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:12:50.837 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:39.380110+0000", 2024-03-11T21:12:50.838 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000219027 2024-03-11T21:12:50.838 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.838 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.838 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:12:50.838 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:39.684409+0000", 2024-03-11T21:12:50.838 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.30429898799999999 2024-03-11T21:12:50.838 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.838 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.838 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T21:12:50.838 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:39.684488+0000", 2024-03-11T21:12:50.839 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.9629999999999995e-05 2024-03-11T21:12:50.839 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.839 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.839 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:12:50.839 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:39.684557+0000", 2024-03-11T21:12:50.839 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.8201000000000007e-05 2024-03-11T21:12:50.839 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:12:50.839 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:12:50.839 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:12:50.840 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.840 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.840 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_op(client.15498.0:52900 11.e 11:714aa1b7:::benchmark_data_smithi023_142591_object52899:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e393)", 2024-03-11T21:12:50.840 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T21:12:39.379751+0000", 2024-03-11T21:12:50.840 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 11.388874945, 2024-03-11T21:12:50.840 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.31072443700000002, 2024-03-11T21:12:50.840 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:12:50.840 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:12:50.840 INFO:teuthology.orchestra.run.smithi023.stdout: "client_info": { 2024-03-11T21:12:50.841 INFO:teuthology.orchestra.run.smithi023.stdout: "client": "client.15498", 2024-03-11T21:12:50.841 INFO:teuthology.orchestra.run.smithi023.stdout: "client_addr": "172.21.15.23:0/4267430685", 2024-03-11T21:12:50.841 INFO:teuthology.orchestra.run.smithi023.stdout: "tid": 52900 2024-03-11T21:12:50.841 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.841 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:12:50.841 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.841 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:12:50.841 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:39.379751+0000", 2024-03-11T21:12:50.841 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:12:50.841 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.841 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.842 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:12:50.842 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:39.379751+0000", 2024-03-11T21:12:50.842 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:12:50.842 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.842 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.842 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:12:50.842 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:39.379752+0000", 2024-03-11T21:12:50.842 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.2780000000000001e-06 2024-03-11T21:12:50.843 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.843 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.843 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:12:50.843 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:39.379771+0000", 2024-03-11T21:12:50.843 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.927e-05 2024-03-11T21:12:50.843 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.843 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.843 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:12:50.843 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:39.379781+0000", 2024-03-11T21:12:50.843 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 9.9839999999999996e-06 2024-03-11T21:12:50.843 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.843 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.844 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:12:50.844 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:39.379787+0000", 2024-03-11T21:12:50.844 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.1190000000000002e-06 2024-03-11T21:12:50.844 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.844 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.844 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:12:50.844 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:39.379833+0000", 2024-03-11T21:12:50.844 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 4.5410000000000001e-05 2024-03-11T21:12:50.845 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.845 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.845 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:12:50.845 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:39.379924+0000", 2024-03-11T21:12:50.845 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 9.1629e-05 2024-03-11T21:12:50.845 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.845 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.845 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for subops from 4,7", 2024-03-11T21:12:50.845 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:39.380006+0000", 2024-03-11T21:12:50.845 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 8.1859000000000003e-05 2024-03-11T21:12:50.846 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.846 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.846 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:12:50.846 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:39.689614+0000", 2024-03-11T21:12:50.846 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.30960792500000001 2024-03-11T21:12:50.846 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.846 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.846 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:12:50.846 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:39.689664+0000", 2024-03-11T21:12:50.846 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 4.9619e-05 2024-03-11T21:12:50.847 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.847 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.847 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "op_commit", 2024-03-11T21:12:50.847 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:39.689765+0000", 2024-03-11T21:12:50.847 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000101477 2024-03-11T21:12:50.847 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.847 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.847 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T21:12:50.847 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:39.689933+0000", 2024-03-11T21:12:50.847 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00016740400000000001 2024-03-11T21:12:50.847 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.848 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.848 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:12:50.848 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:39.690475+0000", 2024-03-11T21:12:50.848 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00054246299999999997 2024-03-11T21:12:50.848 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:12:50.848 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:12:50.848 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:12:50.848 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.848 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.848 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_repop_reply(client.15498.0:52898 11.c e393/380 ondisk, result = 0)", 2024-03-11T21:12:50.848 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T21:12:39.380035+0000", 2024-03-11T21:12:50.848 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 11.388590485, 2024-03-11T21:12:50.848 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.304533579, 2024-03-11T21:12:50.849 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:12:50.849 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "started", 2024-03-11T21:12:50.849 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:12:50.849 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.849 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:12:50.849 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:39.380035+0000", 2024-03-11T21:12:50.849 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:12:50.849 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.849 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.849 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:12:50.849 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:39.380035+0000", 2024-03-11T21:12:50.850 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:12:50.850 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.850 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.850 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:12:50.850 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:39.380036+0000", 2024-03-11T21:12:50.850 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.0800000000000004e-07 2024-03-11T21:12:50.850 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.850 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.851 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:12:50.851 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:39.380040+0000", 2024-03-11T21:12:50.851 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 4.4139999999999996e-06 2024-03-11T21:12:50.851 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.851 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.851 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:12:50.851 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:39.380051+0000", 2024-03-11T21:12:50.851 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.03e-05 2024-03-11T21:12:50.851 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.851 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.852 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:12:50.852 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:39.380059+0000", 2024-03-11T21:12:50.852 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 8.8529999999999998e-06 2024-03-11T21:12:50.852 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.852 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.852 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:12:50.852 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:39.684373+0000", 2024-03-11T21:12:50.852 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.304313583 2024-03-11T21:12:50.852 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.852 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.853 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:12:50.853 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:39.684400+0000", 2024-03-11T21:12:50.853 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.6928e-05 2024-03-11T21:12:50.853 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.853 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.853 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:12:50.853 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:39.684569+0000", 2024-03-11T21:12:50.853 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00016879300000000001 2024-03-11T21:12:50.854 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:12:50.854 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:12:50.854 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:12:50.854 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.854 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.854 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_repop_reply(client.15498.0:52898 11.c e393/380 ondisk, result = 0)", 2024-03-11T21:12:50.854 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T21:12:39.380042+0000", 2024-03-11T21:12:50.854 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 11.388583175000001, 2024-03-11T21:12:50.854 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.303973092, 2024-03-11T21:12:50.854 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:12:50.854 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "started", 2024-03-11T21:12:50.855 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:12:50.855 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.855 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:12:50.855 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:39.380042+0000", 2024-03-11T21:12:50.855 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:12:50.855 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.855 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.856 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:12:50.856 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:39.380042+0000", 2024-03-11T21:12:50.856 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:12:50.856 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.856 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.856 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:12:50.856 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:39.380043+0000", 2024-03-11T21:12:50.856 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 4.4799999999999999e-07 2024-03-11T21:12:50.856 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.857 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.857 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:12:50.857 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:39.380046+0000", 2024-03-11T21:12:50.857 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.7800000000000001e-06 2024-03-11T21:12:50.857 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.857 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.857 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:12:50.857 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:39.380052+0000", 2024-03-11T21:12:50.857 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.5719999999999999e-06 2024-03-11T21:12:50.857 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.857 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.858 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:12:50.858 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:39.380059+0000", 2024-03-11T21:12:50.858 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.4160000000000001e-06 2024-03-11T21:12:50.858 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.858 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.858 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:12:50.858 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:39.380102+0000", 2024-03-11T21:12:50.858 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 4.2793000000000002e-05 2024-03-11T21:12:50.858 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.858 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.858 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:12:50.858 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:39.380106+0000", 2024-03-11T21:12:50.859 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 4.95e-06 2024-03-11T21:12:50.859 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.859 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.859 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:12:50.859 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:39.684016+0000", 2024-03-11T21:12:50.859 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.303909133 2024-03-11T21:12:50.859 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:12:50.859 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:12:50.859 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:12:50.859 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.859 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.859 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_repop(client.15498.0:52901 11.3 e393/380 11:ceb62fc9:::benchmark_data_smithi023_142591_object52900:head v 393'3230, mlcod=393'3228)", 2024-03-11T21:12:50.859 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T21:12:39.380315+0000", 2024-03-11T21:12:50.859 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 11.388310745, 2024-03-11T21:12:50.861 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.310070389, 2024-03-11T21:12:50.861 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:12:50.861 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:12:50.861 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:12:50.862 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.862 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:12:50.862 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:39.380315+0000", 2024-03-11T21:12:50.862 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:12:50.862 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.862 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.862 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:12:50.862 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:39.380315+0000", 2024-03-11T21:12:50.862 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:12:50.862 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.862 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.863 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:12:50.863 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:39.380316+0000", 2024-03-11T21:12:50.863 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.5000000000000002e-07 2024-03-11T21:12:50.863 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.863 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.863 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:12:50.863 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:39.380393+0000", 2024-03-11T21:12:50.863 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.6983000000000005e-05 2024-03-11T21:12:50.863 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.863 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.864 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:12:50.864 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:39.684036+0000", 2024-03-11T21:12:50.864 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.30364328699999998 2024-03-11T21:12:50.864 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.864 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.864 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:12:50.864 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:39.684049+0000", 2024-03-11T21:12:50.864 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.3402e-05 2024-03-11T21:12:50.865 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.865 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.865 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:12:50.865 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:39.684293+0000", 2024-03-11T21:12:50.865 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00024375699999999999 2024-03-11T21:12:50.865 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.866 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.866 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:12:50.866 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:39.684357+0000", 2024-03-11T21:12:50.866 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.3923000000000004e-05 2024-03-11T21:12:50.866 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.866 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.866 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T21:12:50.866 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:39.690296+0000", 2024-03-11T21:12:50.867 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0059389810000000003 2024-03-11T21:12:50.867 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.867 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.867 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:12:50.867 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:39.690385+0000", 2024-03-11T21:12:50.867 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 8.9406000000000004e-05 2024-03-11T21:12:50.867 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:12:50.867 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:12:50.867 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:12:50.867 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.868 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.868 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_repop_reply(client.15498.0:52900 11.e e393/380 ondisk, result = 0)", 2024-03-11T21:12:50.868 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T21:12:39.382274+0000", 2024-03-11T21:12:50.868 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 11.386351884, 2024-03-11T21:12:50.868 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.30735019099999999, 2024-03-11T21:12:50.868 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:12:50.868 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "started", 2024-03-11T21:12:50.868 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:12:50.868 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.868 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:12:50.869 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:39.382274+0000", 2024-03-11T21:12:50.869 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:12:50.869 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.869 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.869 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:12:50.869 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:39.382274+0000", 2024-03-11T21:12:50.869 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:12:50.869 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.869 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.869 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:12:50.869 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:39.382274+0000", 2024-03-11T21:12:50.869 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 5.5700000000000002e-07 2024-03-11T21:12:50.870 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.870 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.870 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:12:50.870 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:39.382278+0000", 2024-03-11T21:12:50.870 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 3.6279999999999998e-06 2024-03-11T21:12:50.870 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.870 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.870 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:12:50.870 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:39.683705+0000", 2024-03-11T21:12:50.870 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.30142687600000001 2024-03-11T21:12:50.870 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.871 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.871 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:12:50.871 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:39.683740+0000", 2024-03-11T21:12:50.871 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 3.5493000000000001e-05 2024-03-11T21:12:50.871 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.871 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.871 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:12:50.871 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:39.689594+0000", 2024-03-11T21:12:50.871 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0058538669999999996 2024-03-11T21:12:50.871 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.872 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.872 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:12:50.872 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:39.689606+0000", 2024-03-11T21:12:50.872 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.1823e-05 2024-03-11T21:12:50.872 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.872 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.872 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:12:50.873 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:39.689624+0000", 2024-03-11T21:12:50.873 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.7946999999999999e-05 2024-03-11T21:12:50.873 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:12:50.873 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:12:50.873 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:12:50.873 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.873 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.873 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_op(client.15498.0:54430 11.c 11:348f898e:::benchmark_data_smithi023_142591_object54429:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e393)", 2024-03-11T21:12:50.874 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T21:12:41.985746+0000", 2024-03-11T21:12:50.874 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 8.7828800329999996, 2024-03-11T21:12:50.874 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.29689212199999998, 2024-03-11T21:12:50.874 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:12:50.874 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:12:50.874 INFO:teuthology.orchestra.run.smithi023.stdout: "client_info": { 2024-03-11T21:12:50.874 INFO:teuthology.orchestra.run.smithi023.stdout: "client": "client.15498", 2024-03-11T21:12:50.874 INFO:teuthology.orchestra.run.smithi023.stdout: "client_addr": "172.21.15.23:0/4267430685", 2024-03-11T21:12:50.875 INFO:teuthology.orchestra.run.smithi023.stdout: "tid": 54430 2024-03-11T21:12:50.875 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.875 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:12:50.875 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.875 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:12:50.875 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:41.985746+0000", 2024-03-11T21:12:50.875 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:12:50.875 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.875 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.876 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:12:50.876 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:41.985746+0000", 2024-03-11T21:12:50.876 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:12:50.876 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.876 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.876 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:12:50.876 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:41.985747+0000", 2024-03-11T21:12:50.876 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.2720000000000001e-06 2024-03-11T21:12:50.876 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.876 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.876 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:12:50.877 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:41.985779+0000", 2024-03-11T21:12:50.877 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 3.2561000000000002e-05 2024-03-11T21:12:50.877 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.877 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.877 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:12:50.877 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:41.985795+0000", 2024-03-11T21:12:50.877 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.5092000000000001e-05 2024-03-11T21:12:50.877 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.877 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.878 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:12:50.878 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:41.985804+0000", 2024-03-11T21:12:50.878 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 9.6069999999999997e-06 2024-03-11T21:12:50.878 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.878 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.878 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:12:50.878 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:41.985877+0000", 2024-03-11T21:12:50.878 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.3085999999999998e-05 2024-03-11T21:12:50.878 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.878 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.879 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:12:50.879 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:41.986045+0000", 2024-03-11T21:12:50.879 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00016743899999999999 2024-03-11T21:12:50.879 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.879 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.879 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for subops from 2,4", 2024-03-11T21:12:50.879 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:41.986195+0000", 2024-03-11T21:12:50.880 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00014984199999999999 2024-03-11T21:12:50.880 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.880 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.880 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:12:50.880 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:42.276102+0000", 2024-03-11T21:12:50.880 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.28990709100000001 2024-03-11T21:12:50.880 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.880 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.880 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:12:50.881 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:42.276155+0000", 2024-03-11T21:12:50.881 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 5.3140000000000003e-05 2024-03-11T21:12:50.881 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.881 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.881 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "op_commit", 2024-03-11T21:12:50.881 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:42.282391+0000", 2024-03-11T21:12:50.881 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0062362989999999998 2024-03-11T21:12:50.881 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.881 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.882 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T21:12:50.882 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:42.282580+0000", 2024-03-11T21:12:50.882 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000189039 2024-03-11T21:12:50.882 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.882 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.882 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:12:50.882 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:42.282638+0000", 2024-03-11T21:12:50.883 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 5.7654000000000001e-05 2024-03-11T21:12:50.883 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:12:50.883 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:12:50.883 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:12:50.883 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.883 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.884 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_op(client.15498.0:54434 11.c 11:3d4da696:::benchmark_data_smithi023_142591_object54433:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e393)", 2024-03-11T21:12:50.884 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T21:12:41.988277+0000", 2024-03-11T21:12:50.884 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 8.7803490800000006, 2024-03-11T21:12:50.884 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.29447043299999998, 2024-03-11T21:12:50.884 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:12:50.884 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:12:50.884 INFO:teuthology.orchestra.run.smithi023.stdout: "client_info": { 2024-03-11T21:12:50.884 INFO:teuthology.orchestra.run.smithi023.stdout: "client": "client.15498", 2024-03-11T21:12:50.885 INFO:teuthology.orchestra.run.smithi023.stdout: "client_addr": "172.21.15.23:0/4267430685", 2024-03-11T21:12:50.885 INFO:teuthology.orchestra.run.smithi023.stdout: "tid": 54434 2024-03-11T21:12:50.885 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.885 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:12:50.885 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.885 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:12:50.886 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:41.988277+0000", 2024-03-11T21:12:50.886 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:12:50.886 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.886 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.886 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:12:50.886 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:41.988277+0000", 2024-03-11T21:12:50.886 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:12:50.887 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.887 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.887 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:12:50.887 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:41.988278+0000", 2024-03-11T21:12:50.887 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.5e-06 2024-03-11T21:12:50.887 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.887 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.888 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:12:50.888 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:41.988306+0000", 2024-03-11T21:12:50.888 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.8221e-05 2024-03-11T21:12:50.888 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.888 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.888 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:12:50.888 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:41.988322+0000", 2024-03-11T21:12:50.889 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.5840000000000001e-05 2024-03-11T21:12:50.889 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.889 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.889 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:12:50.889 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:41.988338+0000", 2024-03-11T21:12:50.889 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.6282000000000001e-05 2024-03-11T21:12:50.889 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.889 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.890 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:12:50.890 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:41.988409+0000", 2024-03-11T21:12:50.890 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.0992000000000001e-05 2024-03-11T21:12:50.890 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.890 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.890 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:12:50.890 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:41.988593+0000", 2024-03-11T21:12:50.891 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00018388599999999999 2024-03-11T21:12:50.891 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.891 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.891 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for subops from 2,4", 2024-03-11T21:12:50.891 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:41.988754+0000", 2024-03-11T21:12:50.891 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00016059000000000001 2024-03-11T21:12:50.891 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.891 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.892 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:12:50.892 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:42.276222+0000", 2024-03-11T21:12:50.892 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.28746792100000002 2024-03-11T21:12:50.892 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.892 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.892 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:12:50.892 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:42.276271+0000", 2024-03-11T21:12:50.892 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 4.8993999999999999e-05 2024-03-11T21:12:50.892 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.892 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.892 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "op_commit", 2024-03-11T21:12:50.893 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:42.282651+0000", 2024-03-11T21:12:50.893 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0063801480000000004 2024-03-11T21:12:50.893 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.893 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.893 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T21:12:50.893 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:42.282707+0000", 2024-03-11T21:12:50.893 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 5.5628999999999999e-05 2024-03-11T21:12:50.893 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.893 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.894 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:12:50.894 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:42.282747+0000", 2024-03-11T21:12:50.894 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 4.0429999999999997e-05 2024-03-11T21:12:50.894 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:12:50.894 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:12:50.894 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:12:50.894 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.894 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.894 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_repop_reply(client.15498.0:54430 11.c e393/380 ondisk, result = 0)", 2024-03-11T21:12:50.895 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T21:12:41.991050+0000", 2024-03-11T21:12:50.895 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 8.7775758600000007, 2024-03-11T21:12:50.895 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.28506156799999999, 2024-03-11T21:12:50.895 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:12:50.895 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "started", 2024-03-11T21:12:50.895 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:12:50.895 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.895 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:12:50.896 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:41.991050+0000", 2024-03-11T21:12:50.896 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:12:50.896 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.896 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.896 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:12:50.896 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:41.991050+0000", 2024-03-11T21:12:50.896 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:12:50.896 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.896 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.897 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:12:50.897 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:41.991051+0000", 2024-03-11T21:12:50.897 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 9.4399999999999998e-07 2024-03-11T21:12:50.897 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.897 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.897 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:12:50.897 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:41.991057+0000", 2024-03-11T21:12:50.897 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.1750000000000002e-06 2024-03-11T21:12:50.898 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.898 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.898 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:12:50.898 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:42.269110+0000", 2024-03-11T21:12:50.898 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.278052839 2024-03-11T21:12:50.898 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.898 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.898 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:12:50.899 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:42.269125+0000", 2024-03-11T21:12:50.899 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.5437000000000001e-05 2024-03-11T21:12:50.899 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.899 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.899 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:12:50.899 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:42.276083+0000", 2024-03-11T21:12:50.899 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0069577950000000001 2024-03-11T21:12:50.899 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.900 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.900 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:12:50.900 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:42.276094+0000", 2024-03-11T21:12:50.900 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.0511e-05 2024-03-11T21:12:50.900 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.900 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.900 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:12:50.900 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:42.276111+0000", 2024-03-11T21:12:50.901 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.7867e-05 2024-03-11T21:12:50.901 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:12:50.901 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:12:50.901 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:12:50.901 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.901 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.901 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_repop(client.15498.0:54437 11.b e393/380 11:d9750c30:::benchmark_data_smithi023_142591_object54436:head v 393'3411, mlcod=393'3409)", 2024-03-11T21:12:50.901 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T21:12:41.992810+0000", 2024-03-11T21:12:50.901 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 8.7758159039999999, 2024-03-11T21:12:50.901 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.29219232899999997, 2024-03-11T21:12:50.902 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:12:50.902 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:12:50.902 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:12:50.902 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.902 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:12:50.902 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:41.992810+0000", 2024-03-11T21:12:50.902 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:12:50.902 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.902 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.902 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:12:50.903 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:41.992810+0000", 2024-03-11T21:12:50.903 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:12:50.903 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.903 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.903 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:12:50.903 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:41.992811+0000", 2024-03-11T21:12:50.903 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.1570000000000001e-06 2024-03-11T21:12:50.903 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.904 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.904 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:12:50.904 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:41.992843+0000", 2024-03-11T21:12:50.904 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 3.2530999999999997e-05 2024-03-11T21:12:50.904 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.904 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.904 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:12:50.904 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:42.268662+0000", 2024-03-11T21:12:50.904 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.275818746 2024-03-11T21:12:50.904 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.905 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.905 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:12:50.905 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:42.268684+0000", 2024-03-11T21:12:50.905 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.1724000000000001e-05 2024-03-11T21:12:50.905 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.905 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.905 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:12:50.905 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:42.269368+0000", 2024-03-11T21:12:50.906 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00068443400000000004 2024-03-11T21:12:50.906 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.906 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.906 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:12:50.906 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:42.269426+0000", 2024-03-11T21:12:50.906 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 5.7293999999999997e-05 2024-03-11T21:12:50.906 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.907 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.907 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T21:12:50.907 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:42.284961+0000", 2024-03-11T21:12:50.907 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.015535307 2024-03-11T21:12:50.907 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:12:50.907 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:12:50.907 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:12:50.907 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:42.285002+0000", 2024-03-11T21:12:50.908 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 4.1136000000000001e-05 2024-03-11T21:12:50.908 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:12:50.908 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:12:50.908 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:12:50.908 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:12:50.908 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:12:50.908 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-11T21:12:50.909 INFO:teuthology.orchestra.run.smithi023.stdout:osd.0: {} 2024-03-11T21:12:50.909 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:50 smithi023 bash[21372]: cluster 2024-03-11T21:12:49.851935+0000 osd.1 (osd.1) 293 : cluster 0 4.7 deep-scrub starts 2024-03-11T21:12:50.909 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:50 smithi023 bash[21372]: cluster 2024-03-11T21:12:49.854660+0000 osd.1 (osd.1) 294 : cluster 0 4.7 deep-scrub ok 2024-03-11T21:12:50.910 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:50 smithi023 bash[21372]: cluster 2024-03-11T21:12:50.368191+0000 mon.a (mon.0) 1831 : cluster 0 osdmap e395: 8 total, 8 up, 8 in 2024-03-11T21:12:50.910 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:50 smithi023 bash[29146]: cluster 2024-03-11T21:12:49.851935+0000 osd.1 (osd.1) 293 : cluster 0 4.7 deep-scrub starts 2024-03-11T21:12:50.910 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:50 smithi023 bash[29146]: cluster 2024-03-11T21:12:49.854660+0000 osd.1 (osd.1) 294 : cluster 0 4.7 deep-scrub ok 2024-03-11T21:12:50.910 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:50 smithi023 bash[29146]: cluster 2024-03-11T21:12:50.368191+0000 mon.a (mon.0) 1831 : cluster 0 osdmap e395: 8 total, 8 up, 8 in 2024-03-11T21:12:50.911 INFO:teuthology.orchestra.run.smithi023.stdout:osd.1: {} 2024-03-11T21:12:50.911 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 46 15 63113 63098 85.7131 61.5 0.0120835 0.0116599 2024-03-11T21:12:50.911 INFO:teuthology.orchestra.run.smithi023.stdout:osd.2: {} 2024-03-11T21:12:50.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:50.913 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:12:50.914 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:12:50.956 INFO:teuthology.orchestra.run.smithi023.stdout:osd.3: {} 2024-03-11T21:12:51.014 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:12:51.027 INFO:teuthology.orchestra.run.smithi023.stdout:osd.4: {} 2024-03-11T21:12:51.042 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:12:51.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:51.048 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:12:51.068 INFO:teuthology.orchestra.run.smithi023.stdout:osd.5: {} 2024-03-11T21:12:51.115 INFO:teuthology.orchestra.run.smithi023.stdout:osd.6: {} 2024-03-11T21:12:51.149 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:12:51.166 INFO:teuthology.orchestra.run.smithi023.stdout:osd.7: {} 2024-03-11T21:12:51.170 INFO:teuthology.orchestra.run.smithi023.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:12:51.170 INFO:teuthology.orchestra.run.smithi023.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:12:51.171 INFO:teuthology.orchestra.run.smithi023.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:12:51.171 INFO:teuthology.orchestra.run.smithi023.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:12:51.171 INFO:teuthology.orchestra.run.smithi023.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:12:51.171 INFO:teuthology.orchestra.run.smithi023.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:12:51.171 INFO:teuthology.orchestra.run.smithi023.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:12:51.171 INFO:teuthology.orchestra.run.smithi023.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:12:51.174 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:12:51.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:51.177 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:12:51.278 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:12:51.312 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:12:51.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:51.316 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:12:51.417 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:12:51.448 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:12:51.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:51.451 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:12:51.528 INFO:teuthology.orchestra.run.smithi023.stderr:clear 6.a pg_upmap mapping 2024-03-11T21:12:51.545 INFO:teuthology.orchestra.run.smithi023.stderr:noscrub is set 2024-03-11T21:12:51.553 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:12:51.577 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:12:51.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:51.584 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:12:51.684 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:12:51.714 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:12:51.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:51.717 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:12:51.817 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:12:51.840 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 47 16 64169 64153 85.2841 65.9375 0.0140039 0.0117196 2024-03-11T21:12:51.847 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:12:51.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:51.851 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:12:51.951 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:12:51.978 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:12:51.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:51.980 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:12:52.006 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:51 smithi146 bash[23809]: cluster 2024-03-11T21:12:50.381314+0000 mgr.y (mgr.24380) 816 : cluster 0 pgmap v935: 158 pgs: 158 active+clean; 3.7 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 66 MiB/s wr, 1.05k op/s 2024-03-11T21:12:52.006 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:51 smithi146 bash[23809]: audit 2024-03-11T21:12:50.722771+0000 mon.a (mon.0) 1832 : audit 0 from='client.? 172.21.15.23:0/2730452234' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:12:52.006 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:51 smithi146 bash[23809]: cluster 2024-03-11T21:12:50.884479+0000 osd.1 (osd.1) 295 : cluster 0 4.12 deep-scrub starts 2024-03-11T21:12:52.006 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:51 smithi146 bash[23809]: cluster 2024-03-11T21:12:50.895372+0000 osd.1 (osd.1) 296 : cluster 0 4.12 deep-scrub ok 2024-03-11T21:12:52.006 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:51 smithi146 bash[23809]: cluster 2024-03-11T21:12:51.167955+0000 osd.3 (osd.3) 464 : cluster 0 6.c scrub starts 2024-03-11T21:12:52.006 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:51 smithi146 bash[23809]: cluster 2024-03-11T21:12:51.179443+0000 osd.3 (osd.3) 465 : cluster 0 6.c scrub ok 2024-03-11T21:12:52.006 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:51 smithi146 bash[23809]: audit 2024-03-11T21:12:51.202699+0000 mon.a (mon.0) 1833 : audit 1 from='client.? 172.21.15.23:0/853253862' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "6.a"}]: dispatch 2024-03-11T21:12:52.007 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:51 smithi146 bash[23809]: audit 2024-03-11T21:12:51.224869+0000 mon.c (mon.2) 95 : audit 1 from='client.? 172.21.15.23:0/1030673157' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:12:52.007 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:51 smithi146 bash[23809]: audit 2024-03-11T21:12:51.225555+0000 mon.a (mon.0) 1834 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:12:52.007 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:51 smithi146 bash[23809]: audit 2024-03-11T21:12:51.242358+0000 mon.a (mon.0) 1835 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:12:52.081 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:12:52.107 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:12:52.113 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:51 smithi023 bash[21372]: cluster 2024-03-11T21:12:50.381314+0000 mgr.y (mgr.24380) 816 : cluster 0 pgmap v935: 158 pgs: 158 active+clean; 3.7 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 66 MiB/s wr, 1.05k op/s 2024-03-11T21:12:52.113 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:51 smithi023 bash[21372]: audit 2024-03-11T21:12:50.722771+0000 mon.a (mon.0) 1832 : audit 0 from='client.? 172.21.15.23:0/2730452234' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:12:52.113 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:51 smithi023 bash[21372]: cluster 2024-03-11T21:12:50.884479+0000 osd.1 (osd.1) 295 : cluster 0 4.12 deep-scrub starts 2024-03-11T21:12:52.113 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:51 smithi023 bash[21372]: cluster 2024-03-11T21:12:50.895372+0000 osd.1 (osd.1) 296 : cluster 0 4.12 deep-scrub ok 2024-03-11T21:12:52.114 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:51 smithi023 bash[21372]: cluster 2024-03-11T21:12:51.167955+0000 osd.3 (osd.3) 464 : cluster 0 6.c scrub starts 2024-03-11T21:12:52.114 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:51 smithi023 bash[21372]: cluster 2024-03-11T21:12:51.179443+0000 osd.3 (osd.3) 465 : cluster 0 6.c scrub ok 2024-03-11T21:12:52.114 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:51 smithi023 bash[21372]: audit 2024-03-11T21:12:51.202699+0000 mon.a (mon.0) 1833 : audit 1 from='client.? 172.21.15.23:0/853253862' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "6.a"}]: dispatch 2024-03-11T21:12:52.114 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:51 smithi023 bash[21372]: audit 2024-03-11T21:12:51.224869+0000 mon.c (mon.2) 95 : audit 1 from='client.? 172.21.15.23:0/1030673157' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:12:52.114 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:51 smithi023 bash[21372]: audit 2024-03-11T21:12:51.225555+0000 mon.a (mon.0) 1834 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:12:52.114 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:51 smithi023 bash[21372]: audit 2024-03-11T21:12:51.242358+0000 mon.a (mon.0) 1835 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:12:52.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:52.116 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:12:52.117 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:51 smithi023 bash[29146]: cluster 2024-03-11T21:12:50.381314+0000 mgr.y (mgr.24380) 816 : cluster 0 pgmap v935: 158 pgs: 158 active+clean; 3.7 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 66 MiB/s wr, 1.05k op/s 2024-03-11T21:12:52.117 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:51 smithi023 bash[29146]: audit 2024-03-11T21:12:50.722771+0000 mon.a (mon.0) 1832 : audit 0 from='client.? 172.21.15.23:0/2730452234' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:12:52.117 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:51 smithi023 bash[29146]: cluster 2024-03-11T21:12:50.884479+0000 osd.1 (osd.1) 295 : cluster 0 4.12 deep-scrub starts 2024-03-11T21:12:52.117 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:51 smithi023 bash[29146]: cluster 2024-03-11T21:12:50.895372+0000 osd.1 (osd.1) 296 : cluster 0 4.12 deep-scrub ok 2024-03-11T21:12:52.117 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:51 smithi023 bash[29146]: cluster 2024-03-11T21:12:51.167955+0000 osd.3 (osd.3) 464 : cluster 0 6.c scrub starts 2024-03-11T21:12:52.117 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:51 smithi023 bash[29146]: cluster 2024-03-11T21:12:51.179443+0000 osd.3 (osd.3) 465 : cluster 0 6.c scrub ok 2024-03-11T21:12:52.117 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:51 smithi023 bash[29146]: audit 2024-03-11T21:12:51.202699+0000 mon.a (mon.0) 1833 : audit 1 from='client.? 172.21.15.23:0/853253862' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "6.a"}]: dispatch 2024-03-11T21:12:52.117 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:51 smithi023 bash[29146]: audit 2024-03-11T21:12:51.224869+0000 mon.c (mon.2) 95 : audit 1 from='client.? 172.21.15.23:0/1030673157' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:12:52.117 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:51 smithi023 bash[29146]: audit 2024-03-11T21:12:51.225555+0000 mon.a (mon.0) 1834 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:12:52.118 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:51 smithi023 bash[29146]: audit 2024-03-11T21:12:51.242358+0000 mon.a (mon.0) 1835 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:12:52.217 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:12:52.241 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:12:52.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:52.244 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:12:52.345 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:52.373 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:52.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:52.376 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:12:52.477 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:12:52.505 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:12:52.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:52.508 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:12:52.608 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:12:52.633 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:12:52.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:52.638 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:12:52.739 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:12:52.749 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:52 smithi023 bash[21372]: cluster 2024-03-11T21:12:51.526832+0000 mon.a (mon.0) 1836 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:12:52.749 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:52 smithi023 bash[21372]: audit 2024-03-11T21:12:51.529748+0000 mon.a (mon.0) 1837 : audit 1 from='client.? 172.21.15.23:0/853253862' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "6.a"}]': finished 2024-03-11T21:12:52.749 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:52 smithi023 bash[21372]: audit 2024-03-11T21:12:51.529831+0000 mon.a (mon.0) 1838 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:12:52.749 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:52 smithi023 bash[21372]: cluster 2024-03-11T21:12:51.543160+0000 mon.a (mon.0) 1839 : cluster 0 osdmap e396: 8 total, 8 up, 8 in 2024-03-11T21:12:52.750 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:52 smithi023 bash[21372]: cluster 2024-03-11T21:12:52.538390+0000 mon.a (mon.0) 1840 : cluster 0 osdmap e397: 8 total, 8 up, 8 in 2024-03-11T21:12:52.750 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:52 smithi023 bash[29146]: cluster 2024-03-11T21:12:51.526832+0000 mon.a (mon.0) 1836 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:12:52.750 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:52 smithi023 bash[29146]: audit 2024-03-11T21:12:51.529748+0000 mon.a (mon.0) 1837 : audit 1 from='client.? 172.21.15.23:0/853253862' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "6.a"}]': finished 2024-03-11T21:12:52.750 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:52 smithi023 bash[29146]: audit 2024-03-11T21:12:51.529831+0000 mon.a (mon.0) 1838 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:12:52.750 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:52 smithi023 bash[29146]: cluster 2024-03-11T21:12:51.543160+0000 mon.a (mon.0) 1839 : cluster 0 osdmap e396: 8 total, 8 up, 8 in 2024-03-11T21:12:52.751 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:52 smithi023 bash[29146]: cluster 2024-03-11T21:12:52.538390+0000 mon.a (mon.0) 1840 : cluster 0 osdmap e397: 8 total, 8 up, 8 in 2024-03-11T21:12:52.756 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:52 smithi146 bash[23809]: cluster 2024-03-11T21:12:51.526832+0000 mon.a (mon.0) 1836 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:12:52.756 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:52 smithi146 bash[23809]: audit 2024-03-11T21:12:51.529748+0000 mon.a (mon.0) 1837 : audit 1 from='client.? 172.21.15.23:0/853253862' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "6.a"}]': finished 2024-03-11T21:12:52.756 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:52 smithi146 bash[23809]: audit 2024-03-11T21:12:51.529831+0000 mon.a (mon.0) 1838 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:12:52.757 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:52 smithi146 bash[23809]: cluster 2024-03-11T21:12:51.543160+0000 mon.a (mon.0) 1839 : cluster 0 osdmap e396: 8 total, 8 up, 8 in 2024-03-11T21:12:52.757 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:52 smithi146 bash[23809]: cluster 2024-03-11T21:12:52.538390+0000 mon.a (mon.0) 1840 : cluster 0 osdmap e397: 8 total, 8 up, 8 in 2024-03-11T21:12:52.773 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:12:52.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:52.776 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:12:52.841 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 48 16 65314 65298 84.9981 71.5625 0.00680354 0.0117602 2024-03-11T21:12:52.877 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:12:52.909 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:12:52.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:52.914 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:12:53.014 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:12:53.048 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:12:53.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:53.050 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:12:53.151 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:12:53.179 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:12:53.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:53.181 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:12:53.282 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:12:53.308 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:12:53.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:53.311 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:12:53.411 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:12:53.445 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:12:53.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:53.448 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:12:53.549 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:53.578 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:53.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:53.583 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:12:53.684 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:12:53.693 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:53 smithi023 bash[21372]: cluster 2024-03-11T21:12:52.381750+0000 mgr.y (mgr.24380) 817 : cluster 0 pgmap v937: 158 pgs: 158 active+clean; 3.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 56 MiB/s wr, 889 op/s 2024-03-11T21:12:53.694 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:53 smithi023 bash[29146]: cluster 2024-03-11T21:12:52.381750+0000 mgr.y (mgr.24380) 817 : cluster 0 pgmap v937: 158 pgs: 158 active+clean; 3.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 56 MiB/s wr, 889 op/s 2024-03-11T21:12:53.731 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:12:53.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:53.733 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:12:53.834 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:12:53.840 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 49 16 66359 66343 84.5964 65.3125 0.00811636 0.0118157 2024-03-11T21:12:53.857 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:12:53.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:53.863 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:12:53.964 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:12:53.988 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:12:53.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:53.990 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:12:54.006 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:53 smithi146 bash[23809]: cluster 2024-03-11T21:12:52.381750+0000 mgr.y (mgr.24380) 817 : cluster 0 pgmap v937: 158 pgs: 158 active+clean; 3.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 56 MiB/s wr, 889 op/s 2024-03-11T21:12:54.091 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:54.117 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:54.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:54.119 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:12:54.220 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:12:54.257 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:12:54.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:54.262 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:12:54.363 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:12:54.392 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:12:54.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:54.395 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:12:54.495 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:12:54.518 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:12:54.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:54.520 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:12:54.621 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:12:54.647 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:12:54.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:54.653 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:12:54.754 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:12:54.778 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:12:54.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:54.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:12:54.840 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 50 16 67027 67011 83.7395 41.75 0.0200964 0.0119358 2024-03-11T21:12:54.882 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:54.900 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:54.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:54.902 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:12:54.963 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-11T21:12:55.003 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:55.033 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:55.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:55.039 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:12:55.139 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:12:55.181 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:12:55.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:55.184 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:12:55.284 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:12:55.309 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:12:55.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:55.311 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:12:55.411 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:12:55.443 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:12:55.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:55.446 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:12:55.547 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:12:55.582 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:12:55.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:55.585 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:12:55.686 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:12:55.714 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:12:55.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:55.717 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:12:55.776 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:55 smithi023 bash[21372]: cluster 2024-03-11T21:12:54.382301+0000 mgr.y (mgr.24380) 818 : cluster 0 pgmap v939: 158 pgs: 158 active+clean; 3.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 56 MiB/s wr, 891 op/s 2024-03-11T21:12:55.777 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:55 smithi023 bash[29146]: cluster 2024-03-11T21:12:54.382301+0000 mgr.y (mgr.24380) 818 : cluster 0 pgmap v939: 158 pgs: 158 active+clean; 3.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 56 MiB/s wr, 891 op/s 2024-03-11T21:12:55.818 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:12:55.840 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 51 15 67761 67746 82.9984 45.9375 0.0131334 0.0120422 2024-03-11T21:12:55.846 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:12:55.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:55.849 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:12:55.942 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:55 smithi146 bash[23809]: cluster 2024-03-11T21:12:54.382301+0000 mgr.y (mgr.24380) 818 : cluster 0 pgmap v939: 158 pgs: 158 active+clean; 3.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 56 MiB/s wr, 891 op/s 2024-03-11T21:12:55.950 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:55.959 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:12:55 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:12:55.973 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:55.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:55.979 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:12:56.080 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:12:56.108 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:12:56.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:56.114 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:12:56.215 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:12:56.244 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:12:56.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:56.248 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:12:56.349 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:12:56.378 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:12:56.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:56.383 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:12:56.484 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:56.510 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:56.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:56.514 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:12:56.614 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:12:56.648 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:12:56.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:56.653 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:12:56.754 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:12:56.785 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:12:56.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:56.790 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:12:56.841 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 52 16 68911 68895 82.7832 71.8125 0.0126617 0.0120747 2024-03-11T21:12:56.891 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:56.923 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:56.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:56.927 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:12:57.028 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:12:57.053 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:12:57.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:57.056 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:12:57.157 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:12:57.187 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:12:57.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:57.193 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:12:57.294 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:57.318 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:57.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:57.321 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:12:57.421 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:12:57.447 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:12:57.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:57.456 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:12:57.556 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:12:57.558 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T21:12:57.577 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:12:57.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:57.581 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:12:57.682 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:12:57.692 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:57 smithi023 bash[21372]: audit 2024-03-11T21:12:55.954221+0000 mgr.y (mgr.24380) 819 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:12:57.692 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:57 smithi023 bash[21372]: cluster 2024-03-11T21:12:56.383104+0000 mgr.y (mgr.24380) 820 : cluster 0 pgmap v940: 158 pgs: 158 active+clean; 4.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 850 B/s rd, 58 MiB/s wr, 930 op/s 2024-03-11T21:12:57.692 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:57 smithi023 bash[29146]: audit 2024-03-11T21:12:55.954221+0000 mgr.y (mgr.24380) 819 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:12:57.692 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:57 smithi023 bash[29146]: cluster 2024-03-11T21:12:56.383104+0000 mgr.y (mgr.24380) 820 : cluster 0 pgmap v940: 158 pgs: 158 active+clean; 4.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 850 B/s rd, 58 MiB/s wr, 930 op/s 2024-03-11T21:12:57.720 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:12:57.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:57.725 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:12:57.826 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:12:57.842 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 53 16 70025 70009 82.5344 69.625 0.0157564 0.0121112 2024-03-11T21:12:57.862 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:12:57.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:57.867 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:12:57.968 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:12:58.000 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:12:58.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:58.006 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:12:58.007 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:57 smithi146 bash[23809]: audit 2024-03-11T21:12:55.954221+0000 mgr.y (mgr.24380) 819 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:12:58.007 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:57 smithi146 bash[23809]: cluster 2024-03-11T21:12:56.383104+0000 mgr.y (mgr.24380) 820 : cluster 0 pgmap v940: 158 pgs: 158 active+clean; 4.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 850 B/s rd, 58 MiB/s wr, 930 op/s 2024-03-11T21:12:58.106 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:12:58.139 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:12:58.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:58.145 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:12:58.246 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:58.269 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:58.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:58.272 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:12:58.373 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:58.411 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:58.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:58.414 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:12:58.515 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:58.536 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:12:58.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:58.538 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:12:58.594 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:12:58 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:12:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:12:58.639 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:58.662 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:58.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:58.670 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:12:58.701 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-11T21:12:58.771 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:58.781 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:58 smithi146 bash[23809]: cluster 2024-03-11T21:12:58.205302+0000 osd.3 (osd.3) 466 : cluster 0 purged_snaps scrub starts 2024-03-11T21:12:58.781 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:58 smithi146 bash[23809]: cluster 2024-03-11T21:12:58.205356+0000 osd.3 (osd.3) 467 : cluster 0 purged_snaps scrub ok 2024-03-11T21:12:58.794 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:58.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:58.797 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:12:58.842 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 54 16 71169 71153 82.3299 71.5 0.00941099 0.0121411 2024-03-11T21:12:58.898 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:12:58.908 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:58 smithi023 bash[21372]: cluster 2024-03-11T21:12:58.205302+0000 osd.3 (osd.3) 466 : cluster 0 purged_snaps scrub starts 2024-03-11T21:12:58.909 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:58 smithi023 bash[21372]: cluster 2024-03-11T21:12:58.205356+0000 osd.3 (osd.3) 467 : cluster 0 purged_snaps scrub ok 2024-03-11T21:12:58.909 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:58 smithi023 bash[29146]: cluster 2024-03-11T21:12:58.205302+0000 osd.3 (osd.3) 466 : cluster 0 purged_snaps scrub starts 2024-03-11T21:12:58.909 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:58 smithi023 bash[29146]: cluster 2024-03-11T21:12:58.205356+0000 osd.3 (osd.3) 467 : cluster 0 purged_snaps scrub ok 2024-03-11T21:12:58.933 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:12:58.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:58.937 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:12:59.038 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:12:59.075 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:12:59.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:59.082 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:12:59.183 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:12:59.213 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:12:59.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:59.219 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:12:59.320 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:12:59.345 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:12:59.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:59.349 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:12:59.449 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:12:59.476 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:12:59.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:59.480 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:12:59.581 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:12:59.605 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:12:59.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:59.613 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:12:59.714 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:59.746 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:59.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:59.750 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:12:59.772 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:12:59.842 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 55 15 72261 72246 82.0743 68.3125 0.0221191 0.0121783 2024-03-11T21:12:59.851 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:59.860 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:59 smithi146 bash[23809]: cluster 2024-03-11T21:12:58.383733+0000 mgr.y (mgr.24380) 821 : cluster 0 pgmap v941: 158 pgs: 158 active+clean; 4.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 50 MiB/s wr, 807 op/s 2024-03-11T21:12:59.860 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:12:59 smithi146 bash[23809]: cluster 2024-03-11T21:12:58.619816+0000 mon.a (mon.0) 1841 : cluster 0 osdmap e398: 8 total, 8 up, 8 in 2024-03-11T21:12:59.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:12:59.893 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:12:59.893 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:12:59.994 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:00.026 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:59 smithi023 bash[21372]: cluster 2024-03-11T21:12:58.383733+0000 mgr.y (mgr.24380) 821 : cluster 0 pgmap v941: 158 pgs: 158 active+clean; 4.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 50 MiB/s wr, 807 op/s 2024-03-11T21:13:00.027 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:12:59 smithi023 bash[21372]: cluster 2024-03-11T21:12:58.619816+0000 mon.a (mon.0) 1841 : cluster 0 osdmap e398: 8 total, 8 up, 8 in 2024-03-11T21:13:00.027 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:59 smithi023 bash[29146]: cluster 2024-03-11T21:12:58.383733+0000 mgr.y (mgr.24380) 821 : cluster 0 pgmap v941: 158 pgs: 158 active+clean; 4.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 50 MiB/s wr, 807 op/s 2024-03-11T21:13:00.027 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:12:59 smithi023 bash[29146]: cluster 2024-03-11T21:12:58.619816+0000 mon.a (mon.0) 1841 : cluster 0 osdmap e398: 8 total, 8 up, 8 in 2024-03-11T21:13:00.030 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:00.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:00.034 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:13:00.135 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:13:00.161 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:13:00.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:00.163 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:13:00.264 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:00.286 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:00.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:00.288 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:13:00.389 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:00.419 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:00.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:00.421 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:13:00.522 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:00.553 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:00.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:00.558 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:13:00.659 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:00.691 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:00.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:00.696 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:13:00.797 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:00.815 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:00.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:00.817 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:13:00.842 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 56 16 73290 73274 81.756 64.25 0.00450549 0.0122105 2024-03-11T21:13:00.917 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:00.943 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:00.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:00.945 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:13:01.046 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:13:01.073 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:13:01.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:01.075 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:13:01.176 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:01.202 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:01.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:01.204 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:13:01.306 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:13:01.338 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:13:01.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:01.341 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:13:01.442 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:13:01.452 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:13:01 smithi146 bash[23809]: cluster 2024-03-11T21:13:00.375381+0000 mon.a (mon.0) 1842 : cluster 0 osdmap e399: 8 total, 8 up, 8 in 2024-03-11T21:13:01.453 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:13:01 smithi146 bash[23809]: cluster 2024-03-11T21:13:00.384181+0000 mgr.y (mgr.24380) 822 : cluster 0 pgmap v944: 158 pgs: 158 active+clean; 4.4 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.3 KiB/s rd, 76 MiB/s wr, 1.21k op/s 2024-03-11T21:13:01.466 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:13:01.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:01.468 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:13:01.569 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:01.589 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:01.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:01.591 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:13:01.688 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 6, 7, 1] out_osds: [] dead_osds: [] live_osds: [0, 2, 3, 5, 6, 7, 4, 1] 2024-03-11T21:13:01.688 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-11T21:13:01.689 INFO:tasks.thrashosds.thrasher:Removing osd 0, in_osds are: [0, 2, 3, 4, 5, 6, 7, 1] 2024-03-11T21:13:01.689 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd out 0 2024-03-11T21:13:01.693 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:01.699 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:13:01 smithi023 bash[21372]: cluster 2024-03-11T21:13:00.375381+0000 mon.a (mon.0) 1842 : cluster 0 osdmap e399: 8 total, 8 up, 8 in 2024-03-11T21:13:01.699 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:13:01 smithi023 bash[21372]: cluster 2024-03-11T21:13:00.384181+0000 mgr.y (mgr.24380) 822 : cluster 0 pgmap v944: 158 pgs: 158 active+clean; 4.4 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.3 KiB/s rd, 76 MiB/s wr, 1.21k op/s 2024-03-11T21:13:01.699 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:13:01 smithi023 bash[29146]: cluster 2024-03-11T21:13:00.375381+0000 mon.a (mon.0) 1842 : cluster 0 osdmap e399: 8 total, 8 up, 8 in 2024-03-11T21:13:01.699 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:13:01 smithi023 bash[29146]: cluster 2024-03-11T21:13:00.384181+0000 mgr.y (mgr.24380) 822 : cluster 0 pgmap v944: 158 pgs: 158 active+clean; 4.4 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.3 KiB/s rd, 76 MiB/s wr, 1.21k op/s 2024-03-11T21:13:01.717 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:01.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:01.725 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:13:01.812 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:13:01.826 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:13:01.846 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:13:01.847 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 57 16 73935 73919 81.0222 40.3125 0.00514363 0.0123352 2024-03-11T21:13:01.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:01.849 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:13:01.950 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:01.970 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:01.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:01.972 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:13:02.073 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:02.106 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:02.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:02.108 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:13:02.209 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:13:02.231 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:13:02.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:02.234 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:13:02.335 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:13:02.360 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:13:02.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:02.365 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:13:02.466 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:13:02.494 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:13:02.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:02.498 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:13:02.598 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:02.621 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:02.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:02.628 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:13:02.729 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:13:02.761 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:13:02.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:02.764 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:13:02.832 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:13:02.847 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 58 16 74772 74756 80.5271 52.3125 0.0260202 0.0124118 2024-03-11T21:13:02.865 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:02.891 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:02.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:02.894 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:13:02.995 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:03.024 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:03.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:03.027 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:13:03.128 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:13:03.162 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:13:03.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:03.167 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:13:03.268 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:03.292 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:03.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:03.297 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:13:03.398 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:03.417 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:03.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:03.418 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:13:03.519 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:13:03.528 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:13:03 smithi146 bash[23809]: cluster 2024-03-11T21:13:02.384814+0000 mgr.y (mgr.24380) 823 : cluster 0 pgmap v945: 158 pgs: 158 active+clean; 4.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 72 MiB/s wr, 1.15k op/s 2024-03-11T21:13:03.552 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:13:03.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:03.554 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:13:03.655 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:13:03.687 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:13:03.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:03.690 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:13:03.754 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:13:03 smithi023 bash[21372]: cluster 2024-03-11T21:13:02.384814+0000 mgr.y (mgr.24380) 823 : cluster 0 pgmap v945: 158 pgs: 158 active+clean; 4.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 72 MiB/s wr, 1.15k op/s 2024-03-11T21:13:03.757 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:13:03 smithi023 bash[29146]: cluster 2024-03-11T21:13:02.384814+0000 mgr.y (mgr.24380) 823 : cluster 0 pgmap v945: 158 pgs: 158 active+clean; 4.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 72 MiB/s wr, 1.15k op/s 2024-03-11T21:13:03.791 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:13:03.808 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:13:03.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:03.810 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:13:03.847 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 59 16 75735 75719 80.1823 60.1875 0.070572 0.0124576 2024-03-11T21:13:03.911 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:13:03.942 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:13:03.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:03.946 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:13:04.047 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:04.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:04.084 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:04.084 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:13:04.185 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:13:04.214 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:13:04.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:04.218 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:13:04.319 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:04.345 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:04.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:04.347 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:13:04.448 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:13:04.465 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:13:04.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:04.467 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:13:04.568 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:04.601 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:04.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:04.606 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:13:04.707 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:13:04.733 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:13:04.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:04.736 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:13:04.837 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:13:04.847 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:2024-03-11T21:13:04.859673+0000 min lat: 0.00123771 max lat: 0.48987 avg lat: 0.0125382 2024-03-11T21:13:04.847 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-11T21:13:04.847 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 60 16 76579 76563 79.7251 52.75 0.0104288 0.0125382 2024-03-11T21:13:04.861 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:13:04.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:04.865 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:13:04.966 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:04.989 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:04.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:04.991 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:13:05.092 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:13:05.128 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:13:05.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:05.131 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:13:05.232 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:13:05.251 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:13:05.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:05.253 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:13:05.353 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:13:05.397 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:13:05.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:05.400 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:13:05.500 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:05.528 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:05.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:05.533 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:13:05.634 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:13:05.665 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:13:05.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:05.668 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:13:05.756 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:13:05 smithi146 bash[23809]: cluster 2024-03-11T21:13:04.385447+0000 mgr.y (mgr.24380) 824 : cluster 0 pgmap v946: 158 pgs: 158 active+clean; 4.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 48 MiB/s wr, 765 op/s 2024-03-11T21:13:05.769 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:05.769 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:13:05 smithi023 bash[21372]: cluster 2024-03-11T21:13:04.385447+0000 mgr.y (mgr.24380) 824 : cluster 0 pgmap v946: 158 pgs: 158 active+clean; 4.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 48 MiB/s wr, 765 op/s 2024-03-11T21:13:05.770 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:13:05 smithi023 bash[29146]: cluster 2024-03-11T21:13:04.385447+0000 mgr.y (mgr.24380) 824 : cluster 0 pgmap v946: 158 pgs: 158 active+clean; 4.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 48 MiB/s wr, 765 op/s 2024-03-11T21:13:05.793 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:05.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:05.798 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:13:05.847 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 61 16 77452 77436 79.3127 54.5625 0.0075612 0.012603 2024-03-11T21:13:05.888 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:13:05.899 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:13:05.932 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:13:05.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:05.937 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:13:06.038 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:06.057 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:06.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:06.058 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:13:06.159 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:06.184 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:06.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:06.186 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:13:06.256 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:13:05 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:13:06.287 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:06.321 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:06.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:06.323 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:13:06.424 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:13:06.454 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:13:06.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:06.456 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:13:06.557 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:13:06.586 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:13:06.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:06.589 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:13:06.690 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:06.713 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:06.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:06.715 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:13:06.816 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:13:06.844 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:13:06.847 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 62 16 78243 78227 78.8308 49.4375 0.0100669 0.0126808 2024-03-11T21:13:06.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:06.851 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:13:06.952 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:06.976 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:06.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:06.980 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:13:07.081 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:07.101 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:07.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:07.103 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:13:07.204 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:13:07.231 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:13:07.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:07.238 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:13:07.339 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:07.382 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:07.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:07.385 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:13:07.486 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:13:07.496 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:13:07 smithi023 bash[29146]: audit 2024-03-11T21:13:05.964903+0000 mgr.y (mgr.24380) 825 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:13:07.496 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:13:07 smithi023 bash[29146]: audit 2024-03-11T21:13:06.274519+0000 mon.a (mon.0) 1843 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:13:07.496 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:13:07 smithi023 bash[29146]: audit 2024-03-11T21:13:06.277520+0000 mon.a (mon.0) 1844 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:13:07.496 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:13:07 smithi023 bash[29146]: cluster 2024-03-11T21:13:06.339910+0000 osd.7 (osd.7) 525 : cluster 0 purged_snaps scrub starts 2024-03-11T21:13:07.496 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:13:07 smithi023 bash[29146]: cluster 2024-03-11T21:13:06.339987+0000 osd.7 (osd.7) 526 : cluster 0 purged_snaps scrub ok 2024-03-11T21:13:07.496 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:13:07 smithi023 bash[29146]: cluster 2024-03-11T21:13:06.386211+0000 mgr.y (mgr.24380) 826 : cluster 0 pgmap v947: 158 pgs: 158 active+clean; 4.7 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 68 MiB/s wr, 1.09k op/s 2024-03-11T21:13:07.497 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:13:07 smithi023 bash[21372]: audit 2024-03-11T21:13:05.964903+0000 mgr.y (mgr.24380) 825 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:13:07.497 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:13:07 smithi023 bash[21372]: audit 2024-03-11T21:13:06.274519+0000 mon.a (mon.0) 1843 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:13:07.497 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:13:07 smithi023 bash[21372]: audit 2024-03-11T21:13:06.277520+0000 mon.a (mon.0) 1844 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:13:07.497 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:13:07 smithi023 bash[21372]: cluster 2024-03-11T21:13:06.339910+0000 osd.7 (osd.7) 525 : cluster 0 purged_snaps scrub starts 2024-03-11T21:13:07.497 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:13:07 smithi023 bash[21372]: cluster 2024-03-11T21:13:06.339987+0000 osd.7 (osd.7) 526 : cluster 0 purged_snaps scrub ok 2024-03-11T21:13:07.497 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:13:07 smithi023 bash[21372]: cluster 2024-03-11T21:13:06.386211+0000 mgr.y (mgr.24380) 826 : cluster 0 pgmap v947: 158 pgs: 158 active+clean; 4.7 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 68 MiB/s wr, 1.09k op/s 2024-03-11T21:13:07.519 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:13:07.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:07.524 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:13:07.625 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:07.635 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:13:07 smithi146 bash[23809]: audit 2024-03-11T21:13:05.964903+0000 mgr.y (mgr.24380) 825 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:13:07.635 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:13:07 smithi146 bash[23809]: audit 2024-03-11T21:13:06.274519+0000 mon.a (mon.0) 1843 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:13:07.635 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:13:07 smithi146 bash[23809]: audit 2024-03-11T21:13:06.277520+0000 mon.a (mon.0) 1844 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:13:07.635 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:13:07 smithi146 bash[23809]: cluster 2024-03-11T21:13:06.339910+0000 osd.7 (osd.7) 525 : cluster 0 purged_snaps scrub starts 2024-03-11T21:13:07.635 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:13:07 smithi146 bash[23809]: cluster 2024-03-11T21:13:06.339987+0000 osd.7 (osd.7) 526 : cluster 0 purged_snaps scrub ok 2024-03-11T21:13:07.635 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:13:07 smithi146 bash[23809]: cluster 2024-03-11T21:13:06.386211+0000 mgr.y (mgr.24380) 826 : cluster 0 pgmap v947: 158 pgs: 158 active+clean; 4.7 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 68 MiB/s wr, 1.09k op/s 2024-03-11T21:13:07.653 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:07.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:07.655 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:13:07.756 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:13:07.782 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:13:07.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:07.787 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:13:07.847 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 63 16 79170 79154 78.4992 57.9375 0.0123142 0.0127332 2024-03-11T21:13:07.888 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:13:07.920 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:13:07.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:07.923 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:13:08.023 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:13:08.050 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:13:08.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:08.054 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:13:08.155 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:13:08.182 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:13:08.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:08.185 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:13:08.286 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:08.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:08.317 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:08.318 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:13:08.338 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:13:08 smithi023 bash[21372]: cluster 2024-03-11T21:13:07.289418+0000 mon.a (mon.0) 1845 : cluster 0 osdmap e400: 8 total, 8 up, 8 in 2024-03-11T21:13:08.338 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:13:08 smithi023 bash[29146]: cluster 2024-03-11T21:13:07.289418+0000 mon.a (mon.0) 1845 : cluster 0 osdmap e400: 8 total, 8 up, 8 in 2024-03-11T21:13:08.418 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:08.442 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:08.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:08.446 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:13:08.492 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:13:08 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:13:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:13:08.546 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:08.556 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:13:08 smithi146 bash[23809]: cluster 2024-03-11T21:13:07.289418+0000 mon.a (mon.0) 1845 : cluster 0 osdmap e400: 8 total, 8 up, 8 in 2024-03-11T21:13:08.570 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:08.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:08.571 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:13:08.672 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:08.696 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:08.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:08.698 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:13:08.799 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:13:08.825 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:13:08.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:08.827 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:13:08.847 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 64 16 79735 79719 77.8245 35.3125 0.00813683 0.0127977 2024-03-11T21:13:08.928 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:13:08.958 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:13:08.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:08.961 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:13:09.061 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:13:09.093 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:13:09.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:09.098 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:13:09.199 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:09.227 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:09.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:09.235 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:13:09.335 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:09.356 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:09.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:09.359 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:13:09.460 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:13:09.470 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:13:09 smithi146 bash[23809]: cluster 2024-03-11T21:13:08.386709+0000 mgr.y (mgr.24380) 827 : cluster 0 pgmap v949: 158 pgs: 158 active+clean; 4.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 638 B/s rd, 45 MiB/s wr, 722 op/s 2024-03-11T21:13:09.484 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:13:09.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:09.487 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:13:09.526 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:13:09 smithi023 bash[29146]: cluster 2024-03-11T21:13:08.386709+0000 mgr.y (mgr.24380) 827 : cluster 0 pgmap v949: 158 pgs: 158 active+clean; 4.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 638 B/s rd, 45 MiB/s wr, 722 op/s 2024-03-11T21:13:09.527 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:13:09 smithi023 bash[21372]: cluster 2024-03-11T21:13:08.386709+0000 mgr.y (mgr.24380) 827 : cluster 0 pgmap v949: 158 pgs: 158 active+clean; 4.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 638 B/s rd, 45 MiB/s wr, 722 op/s 2024-03-11T21:13:09.588 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:13:09.622 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:13:09.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:09.625 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:13:09.726 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:13:09.751 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:13:09.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:09.753 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:13:09.847 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 65 16 80469 80453 77.333 45.875 0.00386625 0.0128862 2024-03-11T21:13:09.854 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:09.878 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:09.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:09.880 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:13:09.981 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:10.015 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:10.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:10.020 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:13:10.121 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:13:10.153 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:13:10.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:10.158 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:13:10.259 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:13:10.290 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:13:10.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:10.293 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:13:10.394 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:13:10.412 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:13:10.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:10.415 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:13:10.515 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:13:10.543 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:13:10.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:10.549 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:13:10.650 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:10.684 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:10.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:10.690 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:13:10.790 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:10.815 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:10.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:10.821 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:13:10.848 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 66 16 81323 81307 76.97 53.375 0.011585 0.0129868 2024-03-11T21:13:10.922 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:13:10.944 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:13:10.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:10.946 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:13:11.047 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:13:11.068 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:13:11.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:11.070 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:13:11.171 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:13:11.199 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:13:11.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:11.203 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:13:11.304 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:11.330 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:11.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:11.335 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:13:11.436 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:13:11.458 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:13:11.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:11.462 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:13:11.563 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:11.582 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:11.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:11.583 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:13:11.684 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:13:11.691 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:13:11 smithi146 bash[23809]: cluster 2024-03-11T21:13:10.378572+0000 mon.a (mon.0) 1846 : cluster 0 osdmap e401: 8 total, 8 up, 8 in 2024-03-11T21:13:11.692 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:13:11 smithi146 bash[23809]: cluster 2024-03-11T21:13:10.387834+0000 mgr.y (mgr.24380) 828 : cluster 0 pgmap v951: 158 pgs: 158 active+clean; 4.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 61 MiB/s wr, 969 op/s 2024-03-11T21:13:11.703 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:13:11.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:11.705 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:13:11.806 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:13:11.832 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:13:11.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:11.834 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:13:11.848 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 67 16 81932 81916 76.3893 38.0625 0.00357828 0.0130228 2024-03-11T21:13:11.935 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:13:11.949 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:13:11 smithi023 bash[21372]: cluster 2024-03-11T21:13:10.378572+0000 mon.a (mon.0) 1846 : cluster 0 osdmap e401: 8 total, 8 up, 8 in 2024-03-11T21:13:11.949 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:13:11 smithi023 bash[21372]: cluster 2024-03-11T21:13:10.387834+0000 mgr.y (mgr.24380) 828 : cluster 0 pgmap v951: 158 pgs: 158 active+clean; 4.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 61 MiB/s wr, 969 op/s 2024-03-11T21:13:11.949 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:13:11 smithi023 bash[29146]: cluster 2024-03-11T21:13:10.378572+0000 mon.a (mon.0) 1846 : cluster 0 osdmap e401: 8 total, 8 up, 8 in 2024-03-11T21:13:11.950 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:13:11 smithi023 bash[29146]: cluster 2024-03-11T21:13:10.387834+0000 mgr.y (mgr.24380) 828 : cluster 0 pgmap v951: 158 pgs: 158 active+clean; 4.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 61 MiB/s wr, 969 op/s 2024-03-11T21:13:11.962 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:13:11.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:11.966 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:13:12.067 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:12.099 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:12.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:12.104 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:13:12.205 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:13:12.225 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:13:12.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:12.227 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:13:12.328 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:13:12.346 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:13:12.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:12.347 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:13:12.448 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:12.485 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:12.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:12.487 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:13:12.506 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:13:12 smithi146 bash[23809]: audit 2024-03-11T21:13:11.606922+0000 mon.a (mon.0) 1847 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:13:12.506 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:13:12 smithi146 bash[23809]: audit 2024-03-11T21:13:12.233429+0000 mon.a (mon.0) 1848 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:13:12.506 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:13:12 smithi146 bash[23809]: audit 2024-03-11T21:13:12.246597+0000 mon.a (mon.0) 1849 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:13:12.520 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:13:12 smithi023 bash[21372]: audit 2024-03-11T21:13:11.606922+0000 mon.a (mon.0) 1847 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:13:12.520 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:13:12 smithi023 bash[21372]: audit 2024-03-11T21:13:12.233429+0000 mon.a (mon.0) 1848 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:13:12.520 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:13:12 smithi023 bash[21372]: audit 2024-03-11T21:13:12.246597+0000 mon.a (mon.0) 1849 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:13:12.588 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:12.628 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:13:12 smithi023 bash[29146]: audit 2024-03-11T21:13:11.606922+0000 mon.a (mon.0) 1847 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:13:12.628 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:13:12 smithi023 bash[29146]: audit 2024-03-11T21:13:12.233429+0000 mon.a (mon.0) 1848 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:13:12.628 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:13:12 smithi023 bash[29146]: audit 2024-03-11T21:13:12.246597+0000 mon.a (mon.0) 1849 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:13:12.630 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:12.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:12.637 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:13:12.738 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:13:12.767 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:13:12.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:12.773 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:13:12.848 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 68 16 82613 82597 75.8919 42.5625 0.00484699 0.0131606 2024-03-11T21:13:12.874 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:12.892 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:12.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:12.894 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:13:12.995 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:13.019 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:13.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:13.021 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:13:13.122 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:13.158 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:13.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:13.162 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:13:13.262 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:13:13.293 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:13:13.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:13.300 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:13:13.401 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:13.423 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:13.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:13.428 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:13:13.528 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:13:13.548 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:13:13.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:13.550 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:13:13.593 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:13:13 smithi023 bash[21372]: cluster 2024-03-11T21:13:12.388476+0000 mgr.y (mgr.24380) 829 : cluster 0 pgmap v952: 158 pgs: 158 active+clean; 4.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 58 MiB/s wr, 928 op/s 2024-03-11T21:13:13.593 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:13:13 smithi023 bash[21372]: audit 2024-03-11T21:13:12.980167+0000 mon.a (mon.0) 1850 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:13:13.593 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:13:13 smithi023 bash[21372]: audit 2024-03-11T21:13:12.981667+0000 mon.a (mon.0) 1851 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:13:13.593 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:13:13 smithi023 bash[21372]: audit 2024-03-11T21:13:12.990942+0000 mon.a (mon.0) 1852 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:13:13.593 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:13:13 smithi023 bash[29146]: cluster 2024-03-11T21:13:12.388476+0000 mgr.y (mgr.24380) 829 : cluster 0 pgmap v952: 158 pgs: 158 active+clean; 4.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 58 MiB/s wr, 928 op/s 2024-03-11T21:13:13.593 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:13:13 smithi023 bash[29146]: audit 2024-03-11T21:13:12.980167+0000 mon.a (mon.0) 1850 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:13:13.593 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:13:13 smithi023 bash[29146]: audit 2024-03-11T21:13:12.981667+0000 mon.a (mon.0) 1851 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:13:13.593 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:13:13 smithi023 bash[29146]: audit 2024-03-11T21:13:12.990942+0000 mon.a (mon.0) 1852 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:13:13.651 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:13.660 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:13:13 smithi146 bash[23809]: cluster 2024-03-11T21:13:12.388476+0000 mgr.y (mgr.24380) 829 : cluster 0 pgmap v952: 158 pgs: 158 active+clean; 4.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 58 MiB/s wr, 928 op/s 2024-03-11T21:13:13.660 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:13:13 smithi146 bash[23809]: audit 2024-03-11T21:13:12.980167+0000 mon.a (mon.0) 1850 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:13:13.660 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:13:13 smithi146 bash[23809]: audit 2024-03-11T21:13:12.981667+0000 mon.a (mon.0) 1851 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:13:13.660 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:13:13 smithi146 bash[23809]: audit 2024-03-11T21:13:12.990942+0000 mon.a (mon.0) 1852 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:13:13.672 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:13.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:13.674 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:13:13.775 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:13.807 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:13.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:13.812 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:13:13.848 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 69 16 83135 83119 75.265 32.625 0.0280376 0.01328 2024-03-11T21:13:13.914 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:13.948 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:13.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:13.951 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:13:14.052 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:14.076 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:14.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:14.081 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:13:14.182 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:14.211 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:14.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:14.214 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:13:14.315 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:13:14.339 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:13:14.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:14.342 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:13:14.442 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:14.471 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:14.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:14.475 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:13:14.576 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:14.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:14.606 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:14.606 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:13:14.707 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:13:14.725 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:13:14.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:14.727 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:13:14.828 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:13:14.848 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 70 16 84020 84004 74.9799 55.3125 0.00516347 0.0133008 2024-03-11T21:13:14.852 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:13:14.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:14.854 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:13:14.955 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:13:14.982 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:13:14.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:14.984 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:13:15.085 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:15.116 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:15.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:15.119 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:13:15.220 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:13:15.247 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:13:15.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:15.249 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:13:15.350 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:13:15.376 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:13:15.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:15.382 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:13:15.483 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:15.492 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:13:15 smithi146 bash[23809]: cluster 2024-03-11T21:13:14.389058+0000 mgr.y (mgr.24380) 830 : cluster 0 pgmap v953: 158 pgs: 158 active+clean; 4.9 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 36 MiB/s wr, 580 op/s 2024-03-11T21:13:15.505 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:15.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:15.509 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:13:15.610 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:15.614 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:13:15 smithi023 bash[21372]: cluster 2024-03-11T21:13:14.389058+0000 mgr.y (mgr.24380) 830 : cluster 0 pgmap v953: 158 pgs: 158 active+clean; 4.9 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 36 MiB/s wr, 580 op/s 2024-03-11T21:13:15.614 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:13:15 smithi023 bash[29146]: cluster 2024-03-11T21:13:14.389058+0000 mgr.y (mgr.24380) 830 : cluster 0 pgmap v953: 158 pgs: 158 active+clean; 4.9 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 36 MiB/s wr, 580 op/s 2024-03-11T21:13:15.635 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:15.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:15.637 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:13:15.738 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:15.758 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:15.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:15.759 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:13:15.848 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 71 16 84684 84668 74.5084 41.5 0.00514338 0.013383 2024-03-11T21:13:15.860 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:13:15.887 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:13:15.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:15.890 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:13:15.991 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:16.006 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:13:15 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:13:16.024 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:16.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:16.032 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:13:16.132 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:16.167 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:16.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:16.170 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:13:16.271 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:16.295 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:16.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:16.299 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:13:16.400 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:16.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:16.426 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:16.427 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:13:16.527 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:13:16.558 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:13:16.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:16.562 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:13:16.663 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:16.687 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:16.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:16.689 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:13:16.790 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:16.818 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:16.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:16.820 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:13:16.848 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 72 16 85275 85259 73.9866 36.9375 0.0225908 0.0135111 2024-03-11T21:13:16.921 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:13:16.955 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:13:16.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:16.961 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:13:17.061 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:13:17.085 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:13:17.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:17.089 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:13:17.189 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:17.220 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:17.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:17.222 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:13:17.323 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:17.353 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:17.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:17.356 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:13:17.456 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:17.493 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:17.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:17.497 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:13:17.598 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:17.628 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:17.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:17.632 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:13:17.733 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:17.743 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:13:17 smithi146 bash[23809]: audit 2024-03-11T21:13:15.977496+0000 mgr.y (mgr.24380) 831 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:13:17.744 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:13:17 smithi146 bash[23809]: cluster 2024-03-11T21:13:16.389817+0000 mgr.y (mgr.24380) 832 : cluster 0 pgmap v954: 158 pgs: 158 active+clean; 5.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.1 KiB/s rd, 51 MiB/s wr, 818 op/s 2024-03-11T21:13:17.762 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:17.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:17.767 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:13:17.776 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:13:17 smithi023 bash[21372]: audit 2024-03-11T21:13:15.977496+0000 mgr.y (mgr.24380) 831 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:13:17.776 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:13:17 smithi023 bash[21372]: cluster 2024-03-11T21:13:16.389817+0000 mgr.y (mgr.24380) 832 : cluster 0 pgmap v954: 158 pgs: 158 active+clean; 5.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.1 KiB/s rd, 51 MiB/s wr, 818 op/s 2024-03-11T21:13:17.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:13:17 smithi023 bash[29146]: audit 2024-03-11T21:13:15.977496+0000 mgr.y (mgr.24380) 831 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:13:17.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:13:17 smithi023 bash[29146]: cluster 2024-03-11T21:13:16.389817+0000 mgr.y (mgr.24380) 832 : cluster 0 pgmap v954: 158 pgs: 158 active+clean; 5.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.1 KiB/s rd, 51 MiB/s wr, 818 op/s 2024-03-11T21:13:17.848 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 73 16 86288 86272 73.8403 63.3125 0.0130795 0.0135379 2024-03-11T21:13:17.868 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:13:17.892 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:13:17.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:17.900 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:13:18.000 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:13:18.027 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:13:18.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:18.032 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:13:18.133 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:13:18.165 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:13:18.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:18.171 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:13:18.272 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:13:18.301 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:13:18.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:18.305 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:13:18.406 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:13:18.427 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:13:18.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:18.429 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:13:18.530 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:13:18.555 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:13:18.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:18.557 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:13:18.658 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:18.681 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:18.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:18.683 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:13:18.784 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:13:18.819 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:13:18.819 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:13:18 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:13:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:13:18.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:18.822 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:13:18.848 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 74 16 86894 86878 73.3544 37.875 0.03863 0.0136257 2024-03-11T21:13:18.923 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:13:18.952 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:13:18.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:18.955 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:13:19.056 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:13:19.091 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:13:19.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:19.092 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:13:19.193 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:19.214 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:19.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:19.216 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:13:19.317 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:13:19.344 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:13:19.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:19.347 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:13:19.448 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:13:19.475 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:13:19.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:19.477 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:13:19.506 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:13:19 smithi146 bash[23809]: cluster 2024-03-11T21:13:18.381373+0000 osd.0 (osd.0) 390 : cluster 0 purged_snaps scrub starts 2024-03-11T21:13:19.506 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:13:19 smithi146 bash[23809]: cluster 2024-03-11T21:13:18.381423+0000 osd.0 (osd.0) 391 : cluster 0 purged_snaps scrub ok 2024-03-11T21:13:19.506 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:13:19 smithi146 bash[23809]: cluster 2024-03-11T21:13:18.390318+0000 mgr.y (mgr.24380) 833 : cluster 0 pgmap v955: 158 pgs: 158 active+clean; 5.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 46 MiB/s wr, 730 op/s 2024-03-11T21:13:19.506 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:13:19 smithi146 bash[23809]: cluster 2024-03-11T21:13:18.436559+0000 mon.a (mon.0) 1853 : cluster 0 osdmap e402: 8 total, 8 up, 8 in 2024-03-11T21:13:19.578 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:19.612 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:19.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:19.614 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:13:19.715 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:13:19.738 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:13:19.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:19.745 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:13:19.776 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:13:19 smithi023 bash[21372]: cluster 2024-03-11T21:13:18.381373+0000 osd.0 (osd.0) 390 : cluster 0 purged_snaps scrub starts 2024-03-11T21:13:19.776 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:13:19 smithi023 bash[21372]: cluster 2024-03-11T21:13:18.381423+0000 osd.0 (osd.0) 391 : cluster 0 purged_snaps scrub ok 2024-03-11T21:13:19.776 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:13:19 smithi023 bash[21372]: cluster 2024-03-11T21:13:18.390318+0000 mgr.y (mgr.24380) 833 : cluster 0 pgmap v955: 158 pgs: 158 active+clean; 5.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 46 MiB/s wr, 730 op/s 2024-03-11T21:13:19.777 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:13:19 smithi023 bash[21372]: cluster 2024-03-11T21:13:18.436559+0000 mon.a (mon.0) 1853 : cluster 0 osdmap e402: 8 total, 8 up, 8 in 2024-03-11T21:13:19.777 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:13:19 smithi023 bash[29146]: cluster 2024-03-11T21:13:18.381373+0000 osd.0 (osd.0) 390 : cluster 0 purged_snaps scrub starts 2024-03-11T21:13:19.777 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:13:19 smithi023 bash[29146]: cluster 2024-03-11T21:13:18.381423+0000 osd.0 (osd.0) 391 : cluster 0 purged_snaps scrub ok 2024-03-11T21:13:19.777 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:13:19 smithi023 bash[29146]: cluster 2024-03-11T21:13:18.390318+0000 mgr.y (mgr.24380) 833 : cluster 0 pgmap v955: 158 pgs: 158 active+clean; 5.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 46 MiB/s wr, 730 op/s 2024-03-11T21:13:19.777 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:13:19 smithi023 bash[29146]: cluster 2024-03-11T21:13:18.436559+0000 mon.a (mon.0) 1853 : cluster 0 osdmap e402: 8 total, 8 up, 8 in 2024-03-11T21:13:19.845 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:19.848 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 75 16 87781 87765 73.1155 55.4375 0.0160411 0.0136724 2024-03-11T21:13:19.889 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:19.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:19.894 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:13:19.995 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:13:20.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:20.027 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:13:20.028 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:13:20.128 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:13:20.153 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:13:20.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:20.158 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:13:20.259 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:20.283 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:20.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:20.288 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:13:20.350 INFO:teuthology.orchestra.run.smithi023.stdout:ERROR: (22) Invalid argument 2024-03-11T21:13:20.350 INFO:teuthology.orchestra.run.smithi023.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:13:20.382 INFO:teuthology.orchestra.run.smithi023.stdout:osd.0: {} 2024-03-11T21:13:20.389 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:20.430 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:20.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:20.433 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:13:20.453 INFO:teuthology.orchestra.run.smithi023.stdout:osd.1: {} 2024-03-11T21:13:20.518 INFO:teuthology.orchestra.run.smithi023.stdout:osd.2: {} 2024-03-11T21:13:20.534 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:13:20.552 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:13:20.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:20.555 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:13:20.583 INFO:teuthology.orchestra.run.smithi023.stdout:osd.3: {} 2024-03-11T21:13:20.622 INFO:teuthology.orchestra.run.smithi023.stdout:osd.4: {} 2024-03-11T21:13:20.655 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:13:20.660 INFO:teuthology.orchestra.run.smithi023.stdout:osd.5: {} 2024-03-11T21:13:20.666 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:13:20 smithi023 bash[21372]: audit 2024-03-11T21:13:20.319167+0000 mon.a (mon.0) 1854 : audit 0 from='client.? 172.21.15.23:0/4126637532' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:13:20.669 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:13:20 smithi023 bash[29146]: audit 2024-03-11T21:13:20.319167+0000 mon.a (mon.0) 1854 : audit 0 from='client.? 172.21.15.23:0/4126637532' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:13:20.690 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:13:20.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:20.693 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:13:20.727 INFO:teuthology.orchestra.run.smithi023.stdout:osd.6: {} 2024-03-11T21:13:20.756 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:13:20 smithi146 bash[23809]: audit 2024-03-11T21:13:20.319167+0000 mon.a (mon.0) 1854 : audit 0 from='client.? 172.21.15.23:0/4126637532' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:13:20.769 INFO:teuthology.orchestra.run.smithi023.stdout:osd.7: {} 2024-03-11T21:13:20.774 INFO:teuthology.orchestra.run.smithi023.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:13:20.774 INFO:teuthology.orchestra.run.smithi023.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:13:20.774 INFO:teuthology.orchestra.run.smithi023.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:13:20.775 INFO:teuthology.orchestra.run.smithi023.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:13:20.775 INFO:teuthology.orchestra.run.smithi023.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:13:20.775 INFO:teuthology.orchestra.run.smithi023.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:13:20.775 INFO:teuthology.orchestra.run.smithi023.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:13:20.775 INFO:teuthology.orchestra.run.smithi023.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:13:20.795 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:13:20.826 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:13:20.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:20.828 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:13:20.848 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 76 16 88644 88628 72.8631 53.9375 0.011842 0.0137191 2024-03-11T21:13:20.930 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:13:20.952 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:13:20.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:20.954 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:13:21.055 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:13:21.069 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:13:21 smithi023 bash[21372]: debug 2024-03-11T21:13:21.013+0000 7f55248dd700 -1 mon.a@0(leader).osd e402 definitely_dead 0 2024-03-11T21:13:21.084 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:13:21.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:21.087 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:13:21.187 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:21.220 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:21.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:21.222 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:13:21.322 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:21.355 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:21.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:21.359 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:13:21.451 INFO:teuthology.orchestra.run.smithi023.stderr:marked out osd.0. 2024-03-11T21:13:21.452 INFO:teuthology.orchestra.run.smithi023.stderr:nodeep-scrub is set 2024-03-11T21:13:21.460 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:13:21.480 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:13:21.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:21.484 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:13:21.506 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:13:21 smithi146 bash[23809]: cluster 2024-03-11T21:13:20.390994+0000 mgr.y (mgr.24380) 834 : cluster 0 pgmap v957: 158 pgs: 158 active+clean; 5.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 41 MiB/s wr, 661 op/s 2024-03-11T21:13:21.506 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:13:21 smithi146 bash[23809]: audit 2024-03-11T21:13:20.753951+0000 mon.b (mon.1) 46 : audit 1 from='client.? 172.21.15.23:0/3926029014' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:13:21.506 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:13:21 smithi146 bash[23809]: audit 2024-03-11T21:13:20.755901+0000 mon.a (mon.0) 1855 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:13:21.506 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:13:21 smithi146 bash[23809]: audit 2024-03-11T21:13:21.016289+0000 mon.b (mon.1) 47 : audit 1 from='client.? 172.21.15.23:0/2113160464' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["0"]}]: dispatch 2024-03-11T21:13:21.506 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:13:21 smithi146 bash[23809]: audit 2024-03-11T21:13:21.016437+0000 mon.a (mon.0) 1856 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["0"]}]: dispatch 2024-03-11T21:13:21.506 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:13:21 smithi146 bash[23809]: cluster 2024-03-11T21:13:21.016603+0000 mon.a (mon.0) 1857 : cluster 1 Client client.admin marked osd.0 out, while it was still marked up 2024-03-11T21:13:21.506 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:13:21 smithi146 bash[23809]: audit 2024-03-11T21:13:21.303528+0000 mon.a (mon.0) 1858 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:13:21.585 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:13:21.605 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:13:21.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:21.607 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:13:21.708 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:13:21.718 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:13:21 smithi023 bash[21372]: cluster 2024-03-11T21:13:20.390994+0000 mgr.y (mgr.24380) 834 : cluster 0 pgmap v957: 158 pgs: 158 active+clean; 5.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 41 MiB/s wr, 661 op/s 2024-03-11T21:13:21.718 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:13:21 smithi023 bash[21372]: audit 2024-03-11T21:13:20.753951+0000 mon.b (mon.1) 46 : audit 1 from='client.? 172.21.15.23:0/3926029014' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:13:21.718 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:13:21 smithi023 bash[21372]: audit 2024-03-11T21:13:20.755901+0000 mon.a (mon.0) 1855 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:13:21.718 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:13:21 smithi023 bash[21372]: audit 2024-03-11T21:13:21.016289+0000 mon.b (mon.1) 47 : audit 1 from='client.? 172.21.15.23:0/2113160464' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["0"]}]: dispatch 2024-03-11T21:13:21.718 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:13:21 smithi023 bash[21372]: audit 2024-03-11T21:13:21.016437+0000 mon.a (mon.0) 1856 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["0"]}]: dispatch 2024-03-11T21:13:21.718 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:13:21 smithi023 bash[21372]: cluster 2024-03-11T21:13:21.016603+0000 mon.a (mon.0) 1857 : cluster 1 Client client.admin marked osd.0 out, while it was still marked up 2024-03-11T21:13:21.718 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:13:21 smithi023 bash[21372]: audit 2024-03-11T21:13:21.303528+0000 mon.a (mon.0) 1858 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:13:21.718 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:13:21 smithi023 bash[29146]: cluster 2024-03-11T21:13:20.390994+0000 mgr.y (mgr.24380) 834 : cluster 0 pgmap v957: 158 pgs: 158 active+clean; 5.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 41 MiB/s wr, 661 op/s 2024-03-11T21:13:21.719 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:13:21 smithi023 bash[29146]: audit 2024-03-11T21:13:20.753951+0000 mon.b (mon.1) 46 : audit 1 from='client.? 172.21.15.23:0/3926029014' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:13:21.719 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:13:21 smithi023 bash[29146]: audit 2024-03-11T21:13:20.755901+0000 mon.a (mon.0) 1855 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:13:21.719 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:13:21 smithi023 bash[29146]: audit 2024-03-11T21:13:21.016289+0000 mon.b (mon.1) 47 : audit 1 from='client.? 172.21.15.23:0/2113160464' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["0"]}]: dispatch 2024-03-11T21:13:21.719 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:13:21 smithi023 bash[29146]: audit 2024-03-11T21:13:21.016437+0000 mon.a (mon.0) 1856 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["0"]}]: dispatch 2024-03-11T21:13:21.719 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:13:21 smithi023 bash[29146]: cluster 2024-03-11T21:13:21.016603+0000 mon.a (mon.0) 1857 : cluster 1 Client client.admin marked osd.0 out, while it was still marked up 2024-03-11T21:13:21.719 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:13:21 smithi023 bash[29146]: audit 2024-03-11T21:13:21.303528+0000 mon.a (mon.0) 1858 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:13:21.728 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:13:21.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:21.730 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:13:21.831 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:21.848 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 77 16 88904 88888 72.128 16.25 0.00408903 0.0138054 2024-03-11T21:13:21.857 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:21.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:21.859 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:13:21.960 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:21.977 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-03-11T21:13:21.986 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:21.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:21.988 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:13:22.089 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:13:22.115 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:13:22.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:22.118 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:13:22.219 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:22.246 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:22.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:22.248 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:13:22.349 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:22.375 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:22.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:22.377 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:13:22.478 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:22.494 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:13:22 smithi146 bash[23809]: cluster 2024-03-11T21:13:21.440956+0000 mon.a (mon.0) 1859 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:13:22.494 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:13:22 smithi146 bash[23809]: audit 2024-03-11T21:13:21.452726+0000 mon.a (mon.0) 1860 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:13:22.494 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:13:22 smithi146 bash[23809]: audit 2024-03-11T21:13:21.452849+0000 mon.a (mon.0) 1861 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["0"]}]': finished 2024-03-11T21:13:22.494 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:13:22 smithi146 bash[23809]: cluster 2024-03-11T21:13:21.463042+0000 mon.a (mon.0) 1862 : cluster 0 osdmap e403: 8 total, 8 up, 7 in 2024-03-11T21:13:22.517 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:22.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:22.520 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:13:22.621 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:13:22.647 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:13:22.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:22.650 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:13:22.751 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:13:22.761 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:13:22 smithi023 bash[29146]: cluster 2024-03-11T21:13:21.440956+0000 mon.a (mon.0) 1859 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:13:22.761 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:13:22 smithi023 bash[29146]: audit 2024-03-11T21:13:21.452726+0000 mon.a (mon.0) 1860 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:13:22.761 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:13:22 smithi023 bash[29146]: audit 2024-03-11T21:13:21.452849+0000 mon.a (mon.0) 1861 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["0"]}]': finished 2024-03-11T21:13:22.761 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:13:22 smithi023 bash[29146]: cluster 2024-03-11T21:13:21.463042+0000 mon.a (mon.0) 1862 : cluster 0 osdmap e403: 8 total, 8 up, 7 in 2024-03-11T21:13:22.761 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:13:22 smithi023 bash[21372]: cluster 2024-03-11T21:13:21.440956+0000 mon.a (mon.0) 1859 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:13:22.762 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:13:22 smithi023 bash[21372]: audit 2024-03-11T21:13:21.452726+0000 mon.a (mon.0) 1860 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:13:22.762 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:13:22 smithi023 bash[21372]: audit 2024-03-11T21:13:21.452849+0000 mon.a (mon.0) 1861 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["0"]}]': finished 2024-03-11T21:13:22.762 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:13:22 smithi023 bash[21372]: cluster 2024-03-11T21:13:21.463042+0000 mon.a (mon.0) 1862 : cluster 0 osdmap e403: 8 total, 8 up, 7 in 2024-03-11T21:13:22.775 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:13:22.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:22.778 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:13:22.849 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 78 16 88904 88888 71.2034 0 - 0.0138054 2024-03-11T21:13:22.879 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:13:22.906 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:13:22.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:22.908 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:13:23.009 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:13:23.033 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:13:23.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:23.035 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:13:23.136 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:13:23.158 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:13:23.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:23.160 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:13:23.261 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:13:23.287 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:13:23.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:23.289 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:13:23.390 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:23.417 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:23.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:23.419 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:13:23.520 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:23.546 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:23.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:23.549 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:13:23.642 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:13:23 smithi023 bash[21372]: cluster 2024-03-11T21:13:22.391424+0000 mgr.y (mgr.24380) 835 : cluster 0 pgmap v959: 158 pgs: 158 active+clean; 5.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 52 MiB/s wr, 833 op/s 2024-03-11T21:13:23.642 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:13:23 smithi023 bash[21372]: cluster 2024-03-11T21:13:22.478219+0000 mon.a (mon.0) 1863 : cluster 0 osdmap e404: 8 total, 8 up, 7 in 2024-03-11T21:13:23.649 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:23.675 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:23.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:23.681 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:13:23.756 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:13:23 smithi146 bash[23809]: cluster 2024-03-11T21:13:22.391424+0000 mgr.y (mgr.24380) 835 : cluster 0 pgmap v959: 158 pgs: 158 active+clean; 5.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 52 MiB/s wr, 833 op/s 2024-03-11T21:13:23.756 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:13:23 smithi146 bash[23809]: cluster 2024-03-11T21:13:22.478219+0000 mon.a (mon.0) 1863 : cluster 0 osdmap e404: 8 total, 8 up, 7 in 2024-03-11T21:13:23.782 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:23.812 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:23.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:23.815 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:13:23.849 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 79 16 89077 89061 70.4391 5.40625 0.013897 0.0141889 2024-03-11T21:13:23.917 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:13:23.928 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:13:23 smithi023 bash[29146]: cluster 2024-03-11T21:13:22.391424+0000 mgr.y (mgr.24380) 835 : cluster 0 pgmap v959: 158 pgs: 158 active+clean; 5.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 52 MiB/s wr, 833 op/s 2024-03-11T21:13:23.928 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:13:23 smithi023 bash[29146]: cluster 2024-03-11T21:13:22.478219+0000 mon.a (mon.0) 1863 : cluster 0 osdmap e404: 8 total, 8 up, 7 in 2024-03-11T21:13:23.956 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:13:23.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:23.959 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:13:24.060 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:13:24.086 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:13:24.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:24.087 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:13:24.188 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:24.212 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:24.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:24.215 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:13:24.316 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:13:24.338 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:13:24.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:24.341 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:13:24.442 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:13:24.467 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:13:24.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:24.469 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:13:24.570 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:24.581 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:13:24 smithi023 bash[21372]: cluster 2024-03-11T21:13:23.485524+0000 mon.a (mon.0) 1864 : cluster 0 osdmap e405: 8 total, 8 up, 7 in 2024-03-11T21:13:24.581 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:13:24 smithi023 bash[21372]: cluster 2024-03-11T21:13:23.486601+0000 osd.6 (osd.6) 400 : cluster 0 11.1 starting backfill to osd.2 from (0'0,0'0] MAX to 402'5532 2024-03-11T21:13:24.581 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:13:24 smithi023 bash[21372]: cluster 2024-03-11T21:13:23.493646+0000 osd.3 (osd.3) 468 : cluster 0 11.b starting backfill to osd.4 from (0'0,0'0] MAX to 402'5541 2024-03-11T21:13:24.581 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:13:24 smithi023 bash[21372]: cluster 2024-03-11T21:13:23.499699+0000 osd.3 (osd.3) 469 : cluster 0 11.9 starting backfill to osd.1 from (0'0,0'0] MAX to 402'5509 2024-03-11T21:13:24.581 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:13:24 smithi023 bash[21372]: cluster 2024-03-11T21:13:23.501760+0000 osd.3 (osd.3) 470 : cluster 0 11.2 starting backfill to osd.7 from (0'0,0'0] MAX to 402'5620 2024-03-11T21:13:24.581 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:13:24 smithi023 bash[21372]: cluster 2024-03-11T21:13:24.482958+0000 mon.a (mon.0) 1865 : cluster 0 osdmap e406: 8 total, 8 up, 7 in 2024-03-11T21:13:24.582 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:13:24 smithi023 bash[29146]: cluster 2024-03-11T21:13:23.485524+0000 mon.a (mon.0) 1864 : cluster 0 osdmap e405: 8 total, 8 up, 7 in 2024-03-11T21:13:24.582 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:13:24 smithi023 bash[29146]: cluster 2024-03-11T21:13:23.486601+0000 osd.6 (osd.6) 400 : cluster 0 11.1 starting backfill to osd.2 from (0'0,0'0] MAX to 402'5532 2024-03-11T21:13:24.582 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:13:24 smithi023 bash[29146]: cluster 2024-03-11T21:13:23.493646+0000 osd.3 (osd.3) 468 : cluster 0 11.b starting backfill to osd.4 from (0'0,0'0] MAX to 402'5541 2024-03-11T21:13:24.582 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:13:24 smithi023 bash[29146]: cluster 2024-03-11T21:13:23.499699+0000 osd.3 (osd.3) 469 : cluster 0 11.9 starting backfill to osd.1 from (0'0,0'0] MAX to 402'5509 2024-03-11T21:13:24.582 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:13:24 smithi023 bash[29146]: cluster 2024-03-11T21:13:23.501760+0000 osd.3 (osd.3) 470 : cluster 0 11.2 starting backfill to osd.7 from (0'0,0'0] MAX to 402'5620 2024-03-11T21:13:24.582 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:13:24 smithi023 bash[29146]: cluster 2024-03-11T21:13:24.482958+0000 mon.a (mon.0) 1865 : cluster 0 osdmap e406: 8 total, 8 up, 7 in 2024-03-11T21:13:24.606 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:24.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:24.611 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:13:24.712 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:13:24.735 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:13:24.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:24.740 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:13:24.781 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:13:24 smithi146 bash[23809]: cluster 2024-03-11T21:13:23.485524+0000 mon.a (mon.0) 1864 : cluster 0 osdmap e405: 8 total, 8 up, 7 in 2024-03-11T21:13:24.781 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:13:24 smithi146 bash[23809]: cluster 2024-03-11T21:13:23.486601+0000 osd.6 (osd.6) 400 : cluster 0 11.1 starting backfill to osd.2 from (0'0,0'0] MAX to 402'5532 2024-03-11T21:13:24.781 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:13:24 smithi146 bash[23809]: cluster 2024-03-11T21:13:23.493646+0000 osd.3 (osd.3) 468 : cluster 0 11.b starting backfill to osd.4 from (0'0,0'0] MAX to 402'5541 2024-03-11T21:13:24.781 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:13:24 smithi146 bash[23809]: cluster 2024-03-11T21:13:23.499699+0000 osd.3 (osd.3) 469 : cluster 0 11.9 starting backfill to osd.1 from (0'0,0'0] MAX to 402'5509 2024-03-11T21:13:24.782 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:13:24 smithi146 bash[23809]: cluster 2024-03-11T21:13:23.501760+0000 osd.3 (osd.3) 470 : cluster 0 11.2 starting backfill to osd.7 from (0'0,0'0] MAX to 402'5620 2024-03-11T21:13:24.782 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:13:24 smithi146 bash[23809]: cluster 2024-03-11T21:13:24.482958+0000 mon.a (mon.0) 1865 : cluster 0 osdmap e406: 8 total, 8 up, 7 in 2024-03-11T21:13:24.841 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:13:24.849 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:2024-03-11T21:13:24.861719+0000 min lat: 0.00123771 max lat: 2.28205 avg lat: 0.0142508 2024-03-11T21:13:24.849 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-11T21:13:24.849 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 80 16 89828 89812 70.1454 46.9375 0.0152715 0.0142508 2024-03-11T21:13:24.883 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:13:24.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:24.885 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:13:24.986 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:25.009 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:25.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:25.011 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:13:25.112 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:13:25.146 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:13:25.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:25.149 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:13:25.250 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:13:25.279 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:13:25.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:25.284 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:13:25.385 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:25.415 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:25.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:25.417 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:13:25.518 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:13:25.577 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:13:25.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:25.581 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:13:25.682 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:13:25.707 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:13:25.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:25.711 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:13:25.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:13:25 smithi146 bash[23809]: cluster 2024-03-11T21:13:24.391892+0000 mgr.y (mgr.24380) 836 : cluster 0 pgmap v962: 158 pgs: 1 remapped+peering, 157 active+clean; 5.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 687 B/s rd, 40 MiB/s wr, 634 op/s; 0 B/s, 1 objects/s recovering 2024-03-11T21:13:25.756 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:13:25 smithi146 bash[23809]: cluster 2024-03-11T21:13:25.487918+0000 mon.a (mon.0) 1866 : cluster 0 osdmap e407: 8 total, 8 up, 7 in 2024-03-11T21:13:25.776 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:13:25 smithi023 bash[21372]: cluster 2024-03-11T21:13:24.391892+0000 mgr.y (mgr.24380) 836 : cluster 0 pgmap v962: 158 pgs: 1 remapped+peering, 157 active+clean; 5.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 687 B/s rd, 40 MiB/s wr, 634 op/s; 0 B/s, 1 objects/s recovering 2024-03-11T21:13:25.776 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:13:25 smithi023 bash[21372]: cluster 2024-03-11T21:13:25.487918+0000 mon.a (mon.0) 1866 : cluster 0 osdmap e407: 8 total, 8 up, 7 in 2024-03-11T21:13:25.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:13:25 smithi023 bash[29146]: cluster 2024-03-11T21:13:24.391892+0000 mgr.y (mgr.24380) 836 : cluster 0 pgmap v962: 158 pgs: 1 remapped+peering, 157 active+clean; 5.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 687 B/s rd, 40 MiB/s wr, 634 op/s; 0 B/s, 1 objects/s recovering 2024-03-11T21:13:25.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:13:25 smithi023 bash[29146]: cluster 2024-03-11T21:13:25.487918+0000 mon.a (mon.0) 1866 : cluster 0 osdmap e407: 8 total, 8 up, 7 in 2024-03-11T21:13:25.812 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:25.832 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:25.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:25.834 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:13:25.849 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 81 16 90539 90523 69.828 44.4375 0.00398261 0.0143082 2024-03-11T21:13:25.935 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:13:25.966 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:13:25.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:25.969 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:13:26.070 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:13:26.104 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:13:26.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:26.109 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:13:26.210 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:13:26.248 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:13:26.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:26.251 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:13:26.255 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:13:25 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:13:26.352 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:26.378 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:26.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:26.380 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:13:26.481 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:13:26.517 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:13:26.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:26.522 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:13:26.623 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:26.668 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:26.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:26.671 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:13:26.693 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T21:13:26.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:13:26 smithi146 bash[23809]: cluster 2024-03-11T21:13:26.509661+0000 mon.a (mon.0) 1867 : cluster 3 Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-11T21:13:26.772 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:13:26.776 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:13:26 smithi023 bash[21372]: cluster 2024-03-11T21:13:26.509661+0000 mon.a (mon.0) 1867 : cluster 3 Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-11T21:13:26.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:13:26 smithi023 bash[29146]: cluster 2024-03-11T21:13:26.509661+0000 mon.a (mon.0) 1867 : cluster 3 Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-11T21:13:26.797 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:13:26.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:26.800 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:13:26.833 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd unset noscrub 2024-03-11T21:13:26.849 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 82 16 91465 91449 69.682 57.875 0.00812202 0.014344 2024-03-11T21:13:26.901 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:13:26.955 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:13:26.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:26.959 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:13:27.060 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:27.087 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:27.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:27.090 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:13:27.190 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:13:27.220 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:13:27.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:27.223 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:13:27.324 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:27.350 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:27.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:27.353 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:13:27.453 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:27.491 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:27.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:27.495 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:13:27.596 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:13:27.642 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:13:27.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:27.646 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:13:27.746 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:13:27.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:13:27 smithi146 bash[23809]: audit 2024-03-11T21:13:25.985919+0000 mgr.y (mgr.24380) 837 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:13:27.756 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:13:27 smithi146 bash[23809]: cluster 2024-03-11T21:13:26.392528+0000 mgr.y (mgr.24380) 838 : cluster 0 pgmap v965: 158 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 3 peering, 1 remapped+peering, 150 active+clean; 5.5 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 33 MiB/s wr, 535 op/s; 21754/270204 objects misplaced (8.051%); 12 MiB/s, 212 objects/s recovering 2024-03-11T21:13:27.757 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:13:27 smithi023 bash[21372]: audit 2024-03-11T21:13:25.985919+0000 mgr.y (mgr.24380) 837 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:13:27.757 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:13:27 smithi023 bash[21372]: cluster 2024-03-11T21:13:26.392528+0000 mgr.y (mgr.24380) 838 : cluster 0 pgmap v965: 158 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 3 peering, 1 remapped+peering, 150 active+clean; 5.5 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 33 MiB/s wr, 535 op/s; 21754/270204 objects misplaced (8.051%); 12 MiB/s, 212 objects/s recovering 2024-03-11T21:13:27.757 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:13:27 smithi023 bash[29146]: audit 2024-03-11T21:13:25.985919+0000 mgr.y (mgr.24380) 837 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:13:27.757 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:13:27 smithi023 bash[29146]: cluster 2024-03-11T21:13:26.392528+0000 mgr.y (mgr.24380) 838 : cluster 0 pgmap v965: 158 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 3 peering, 1 remapped+peering, 150 active+clean; 5.5 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 33 MiB/s wr, 535 op/s; 21754/270204 objects misplaced (8.051%); 12 MiB/s, 212 objects/s recovering 2024-03-11T21:13:27.778 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:13:27.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:27.784 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:13:27.849 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 83 15 92318 92303 69.4855 53.375 0.0221877 0.0143858 2024-03-11T21:13:27.882 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:13:27.886 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:27.925 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:27.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:27.932 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:13:28.033 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:28.057 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:28.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:28.061 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:13:28.162 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:13:28.188 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:13:28.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:28.193 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:13:28.294 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:13:28.330 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:13:28.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:28.338 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:13:28.439 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:28.471 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:28.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:28.475 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:13:28.576 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:28.604 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:28.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:28.607 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:13:28.708 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:13:28.718 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:13:28 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:13:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:13:28.736 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:13:28.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:28.738 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:13:28.839 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:28.849 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 84 16 93224 93208 69.3316 56.5625 0.0133223 0.0144184 2024-03-11T21:13:28.870 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:28.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:28.874 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:13:28.975 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:13:29.010 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:13:29.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:29.013 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:13:29.114 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:29.139 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:29.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:29.142 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:13:29.243 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:29.270 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:29.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:29.272 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:13:29.373 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:13:29.401 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:13:29.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:29.411 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:13:29.512 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:29.536 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:29.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:29.538 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:13:29.639 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:13:29.648 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:13:29 smithi023 bash[21372]: cluster 2024-03-11T21:13:28.393145+0000 mgr.y (mgr.24380) 839 : cluster 0 pgmap v966: 158 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 3 peering, 150 active+clean; 5.5 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 31 MiB/s wr, 496 op/s; 27227/272586 objects misplaced (9.988%); 9.1 MiB/s, 155 objects/s recovering 2024-03-11T21:13:29.648 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:13:29 smithi023 bash[29146]: cluster 2024-03-11T21:13:28.393145+0000 mgr.y (mgr.24380) 839 : cluster 0 pgmap v966: 158 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 3 peering, 150 active+clean; 5.5 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 31 MiB/s wr, 496 op/s; 27227/272586 objects misplaced (9.988%); 9.1 MiB/s, 155 objects/s recovering 2024-03-11T21:13:29.673 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:13:29.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:29.677 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:13:29.778 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:13:29.810 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:13:29.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:29.813 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:13:29.840 INFO:tasks.thrashosds.thrasher:in_osds: [2, 3, 4, 5, 6, 7, 1] out_osds: [0] dead_osds: [] live_osds: [0, 2, 3, 5, 6, 7, 4, 1] 2024-03-11T21:13:29.840 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-11T21:13:29.841 INFO:tasks.thrashosds.thrasher:Shrinking pool unique_pool_4 2024-03-11T21:13:29.841 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool unique_pool_4 by 10 min 8 2024-03-11T21:13:29.841 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T21:13:29.849 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 85 16 94125 94109 69.1785 56.3125 0.010347 0.0144497 2024-03-11T21:13:29.913 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:13:29.962 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:13:29.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:29.965 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:13:30.006 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:13:29 smithi146 bash[23809]: cluster 2024-03-11T21:13:28.393145+0000 mgr.y (mgr.24380) 839 : cluster 0 pgmap v966: 158 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 3 peering, 150 active+clean; 5.5 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 31 MiB/s wr, 496 op/s; 27227/272586 objects misplaced (9.988%); 9.1 MiB/s, 155 objects/s recovering 2024-03-11T21:13:30.066 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:30.097 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:30.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:30.100 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:13:30.201 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:30.226 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:30.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:30.230 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:13:30.330 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:13:30.363 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:13:30.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:30.365 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:13:30.466 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:30.491 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:30.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:30.495 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:13:30.596 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:13:30.629 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:13:30.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:30.632 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:13:30.732 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:13:30.758 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:13:30.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:30.762 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:13:30.850 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 86 15 95024 95009 69.0281 56.25 0.0336079 0.014482 2024-03-11T21:13:30.863 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:13:30.893 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:13:30.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:30.896 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:13:30.949 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:13:30.952 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:13:30.997 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:31.022 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:31.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:31.029 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:13:31.129 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:13:31.154 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:13:31.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:31.158 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:13:31.258 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:31.281 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:31.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:31.284 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:13:31.385 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:13:31.405 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:13:31.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:31.407 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:13:31.508 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:13:31.535 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:13:31.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:31.537 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:13:31.638 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:31.647 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:13:31 smithi146 bash[23809]: cluster 2024-03-11T21:13:30.393856+0000 mgr.y (mgr.24380) 840 : cluster 0 pgmap v967: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+remapped, 3 active+remapped+backfilling, 1 peering, 152 active+clean; 5.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 592 B/s rd, 40 MiB/s wr, 645 op/s; 26573/278931 objects misplaced (9.527%); 20 MiB/s, 333 objects/s recovering 2024-03-11T21:13:31.667 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:31.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:31.674 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:13:31.775 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:13:31.776 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:13:31 smithi023 bash[21372]: cluster 2024-03-11T21:13:30.393856+0000 mgr.y (mgr.24380) 840 : cluster 0 pgmap v967: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+remapped, 3 active+remapped+backfilling, 1 peering, 152 active+clean; 5.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 592 B/s rd, 40 MiB/s wr, 645 op/s; 26573/278931 objects misplaced (9.527%); 20 MiB/s, 333 objects/s recovering 2024-03-11T21:13:31.777 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:13:31 smithi023 bash[29146]: cluster 2024-03-11T21:13:30.393856+0000 mgr.y (mgr.24380) 840 : cluster 0 pgmap v967: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+remapped, 3 active+remapped+backfilling, 1 peering, 152 active+clean; 5.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 592 B/s rd, 40 MiB/s wr, 645 op/s; 26573/278931 objects misplaced (9.527%); 20 MiB/s, 333 objects/s recovering 2024-03-11T21:13:31.804 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:13:31.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:31.808 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:13:31.850 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 87 16 95643 95627 68.6787 38.625 0.0155562 0.0145555 2024-03-11T21:13:31.909 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:31.930 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:31.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:31.932 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:13:32.032 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:32.057 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:32.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:32.060 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:13:32.160 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:13:32.186 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:13:32.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:32.188 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:13:32.289 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:13:32.320 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:13:32.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:32.325 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:13:32.426 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:13:32.457 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:13:32.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:32.461 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:13:32.562 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:13:32.597 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:13:32.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:32.601 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:13:32.702 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:13:32.726 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:13:32.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:32.729 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:13:32.830 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:13:32.839 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:13:32 smithi023 bash[21372]: cluster 2024-03-11T21:13:31.747577+0000 osd.2 (osd.2) 259 : cluster 0 purged_snaps scrub starts 2024-03-11T21:13:32.839 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:13:32 smithi023 bash[21372]: cluster 2024-03-11T21:13:31.747645+0000 osd.2 (osd.2) 260 : cluster 0 purged_snaps scrub ok 2024-03-11T21:13:32.840 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:13:32 smithi023 bash[29146]: cluster 2024-03-11T21:13:31.747577+0000 osd.2 (osd.2) 259 : cluster 0 purged_snaps scrub starts 2024-03-11T21:13:32.840 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:13:32 smithi023 bash[29146]: cluster 2024-03-11T21:13:31.747645+0000 osd.2 (osd.2) 260 : cluster 0 purged_snaps scrub ok 2024-03-11T21:13:32.850 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 88 16 96386 96370 68.4259 46.4375 0.0106935 0.014608 2024-03-11T21:13:32.855 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:13:32.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:32.861 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:13:32.962 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:13:32.972 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:13:32 smithi146 bash[23809]: cluster 2024-03-11T21:13:31.747577+0000 osd.2 (osd.2) 259 : cluster 0 purged_snaps scrub starts 2024-03-11T21:13:32.972 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:13:32 smithi146 bash[23809]: cluster 2024-03-11T21:13:31.747645+0000 osd.2 (osd.2) 260 : cluster 0 purged_snaps scrub ok 2024-03-11T21:13:32.991 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:13:32.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:32.996 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:13:33.097 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:13:33.123 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:13:33.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:33.128 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:13:33.229 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:13:33.257 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:13:33.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:33.260 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:13:33.361 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:13:33.387 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:13:33.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:33.391 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:13:33.492 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:13:33.524 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:13:33.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:33.527 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:13:33.629 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:33.639 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:13:33 smithi146 bash[23809]: cluster 2024-03-11T21:13:32.041919+0000 osd.1 (osd.1) 297 : cluster 0 purged_snaps scrub starts 2024-03-11T21:13:33.640 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:13:33 smithi146 bash[23809]: cluster 2024-03-11T21:13:32.042084+0000 osd.1 (osd.1) 298 : cluster 0 purged_snaps scrub ok 2024-03-11T21:13:33.640 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:13:33 smithi146 bash[23809]: cluster 2024-03-11T21:13:32.394578+0000 mgr.y (mgr.24380) 841 : cluster 0 pgmap v968: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+remapped, 3 active+remapped+backfilling, 153 active+clean; 5.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 511 B/s rd, 43 MiB/s wr, 695 op/s; 25993/282234 objects misplaced (9.210%); 24 MiB/s, 390 objects/s recovering 2024-03-11T21:13:33.640 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:13:33 smithi146 bash[23809]: cluster 2024-03-11T21:13:32.595386+0000 mon.a (mon.0) 1868 : cluster 0 osdmap e408: 8 total, 8 up, 7 in 2024-03-11T21:13:33.653 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:33.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:33.660 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:13:33.761 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:13:33.776 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:13:33 smithi023 bash[21372]: cluster 2024-03-11T21:13:32.041919+0000 osd.1 (osd.1) 297 : cluster 0 purged_snaps scrub starts 2024-03-11T21:13:33.776 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:13:33 smithi023 bash[21372]: cluster 2024-03-11T21:13:32.042084+0000 osd.1 (osd.1) 298 : cluster 0 purged_snaps scrub ok 2024-03-11T21:13:33.776 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:13:33 smithi023 bash[21372]: cluster 2024-03-11T21:13:32.394578+0000 mgr.y (mgr.24380) 841 : cluster 0 pgmap v968: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+remapped, 3 active+remapped+backfilling, 153 active+clean; 5.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 511 B/s rd, 43 MiB/s wr, 695 op/s; 25993/282234 objects misplaced (9.210%); 24 MiB/s, 390 objects/s recovering 2024-03-11T21:13:33.776 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:13:33 smithi023 bash[21372]: cluster 2024-03-11T21:13:32.595386+0000 mon.a (mon.0) 1868 : cluster 0 osdmap e408: 8 total, 8 up, 7 in 2024-03-11T21:13:33.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:13:33 smithi023 bash[29146]: cluster 2024-03-11T21:13:32.041919+0000 osd.1 (osd.1) 297 : cluster 0 purged_snaps scrub starts 2024-03-11T21:13:33.777 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:13:33 smithi023 bash[29146]: cluster 2024-03-11T21:13:32.042084+0000 osd.1 (osd.1) 298 : cluster 0 purged_snaps scrub ok 2024-03-11T21:13:33.777 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:13:33 smithi023 bash[29146]: cluster 2024-03-11T21:13:32.394578+0000 mgr.y (mgr.24380) 841 : cluster 0 pgmap v968: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+remapped, 3 active+remapped+backfilling, 153 active+clean; 5.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 511 B/s rd, 43 MiB/s wr, 695 op/s; 25993/282234 objects misplaced (9.210%); 24 MiB/s, 390 objects/s recovering 2024-03-11T21:13:33.777 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:13:33 smithi023 bash[29146]: cluster 2024-03-11T21:13:32.595386+0000 mon.a (mon.0) 1868 : cluster 0 osdmap e408: 8 total, 8 up, 7 in 2024-03-11T21:13:33.787 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:13:33.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:33.789 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:13:33.850 INFO:tasks.radosbench.radosbench.0.smithi023.stdout: 89 16 97196 97180 68.2259 50.625 0.0183011 0.0146521 2024-03-11T21:13:33.890 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:33.918 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:33.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:33.920 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:13:34.008 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:13:34.021 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:34.046 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:34.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:34.050 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:13:34.150 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:13:34.171 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:13:34.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:34.174 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:13:34.274 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:13:34.294 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:13:34.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:34.297 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:13:34.397 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:34.418 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:34.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:34.420 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:13:34.521 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:34.545 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:34.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:34.547 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:13:34.648 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:34.683 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:34.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:34.690 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:13:34.790 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:13:34.831 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:13:34.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:34.833 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:13:34.850 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:Total time run: 90.0094 2024-03-11T21:13:34.850 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:Total writes made: 97641 2024-03-11T21:13:34.850 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:Write size: 65536 2024-03-11T21:13:34.850 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:Object size: 65536 2024-03-11T21:13:34.850 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:Bandwidth (MB/sec): 67.7992 2024-03-11T21:13:34.850 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:Stddev Bandwidth: 32.7659 2024-03-11T21:13:34.850 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:Max bandwidth (MB/sec): 170.188 2024-03-11T21:13:34.850 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:Min bandwidth (MB/sec): 0 2024-03-11T21:13:34.850 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:Average IOPS: 1084 2024-03-11T21:13:34.850 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:Stddev IOPS: 524.266 2024-03-11T21:13:34.851 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:Max IOPS: 2723 2024-03-11T21:13:34.851 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:Min IOPS: 0 2024-03-11T21:13:34.851 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:Average Latency(s): 0.014745 2024-03-11T21:13:34.851 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:Stddev Latency(s): 0.0379901 2024-03-11T21:13:34.851 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:Max latency(s): 2.28205 2024-03-11T21:13:34.851 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:Min latency(s): 0.00123771 2024-03-11T21:13:34.855 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:Cleaning up (deleting benchmark objects) 2024-03-11T21:13:34.934 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:34.960 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:34.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:34.964 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:13:35.064 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:35.093 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:35.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:35.097 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:13:35.198 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:13:35.239 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:13:35.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:35.241 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:13:35.341 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:35.379 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:35.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:35.391 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:13:35.492 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:35.519 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:35.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:35.521 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:13:35.621 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:35.649 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:35.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:35.652 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:13:35.753 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:13:35.756 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:13:35 smithi146 bash[23809]: cluster 2024-03-11T21:13:34.395069+0000 mgr.y (mgr.24380) 842 : cluster 0 pgmap v970: 158 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 153 active+clean; 5.8 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 344 B/s rd, 36 MiB/s wr, 577 op/s; 25895/285246 objects misplaced (9.078%); 17 MiB/s, 268 objects/s recovering 2024-03-11T21:13:35.764 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:13:35 smithi023 bash[29146]: cluster 2024-03-11T21:13:34.395069+0000 mgr.y (mgr.24380) 842 : cluster 0 pgmap v970: 158 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 153 active+clean; 5.8 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 344 B/s rd, 36 MiB/s wr, 577 op/s; 25895/285246 objects misplaced (9.078%); 17 MiB/s, 268 objects/s recovering 2024-03-11T21:13:35.764 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:13:35 smithi023 bash[21372]: cluster 2024-03-11T21:13:34.395069+0000 mgr.y (mgr.24380) 842 : cluster 0 pgmap v970: 158 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 153 active+clean; 5.8 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 344 B/s rd, 36 MiB/s wr, 577 op/s; 25895/285246 objects misplaced (9.078%); 17 MiB/s, 268 objects/s recovering 2024-03-11T21:13:35.781 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:13:35.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:35.786 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:13:35.887 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:13:35.923 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:13:35.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:35.926 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:13:36.027 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:13:36.056 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:13:36.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:36.060 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:13:36.161 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:13:36.187 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:13:36.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:36.190 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:13:36.255 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:13:35 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:13:36.290 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:13:36.325 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:13:36.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:36.330 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:13:36.433 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:37.562 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:37.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:37.566 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:13:37.667 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:13:37.678 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:13:37 smithi023 bash[21372]: audit 2024-03-11T21:13:35.995481+0000 mgr.y (mgr.24380) 843 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:13:37.678 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:13:37 smithi023 bash[21372]: audit 2024-03-11T21:13:36.345451+0000 mon.a (mon.0) 1869 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:13:37.678 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:13:37 smithi023 bash[21372]: audit 2024-03-11T21:13:36.347788+0000 mon.a (mon.0) 1870 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:13:37.678 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:13:37 smithi023 bash[21372]: cluster 2024-03-11T21:13:36.395990+0000 mgr.y (mgr.24380) 844 : cluster 0 pgmap v971: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+remapped, 3 active+remapped+backfilling, 153 active+clean; 5.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.1 KiB/s rd, 46 MiB/s wr, 766 op/s; 23831/291450 objects misplaced (8.177%); 31 MiB/s, 502 objects/s recovering 2024-03-11T21:13:37.678 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:13:37 smithi023 bash[29146]: audit 2024-03-11T21:13:35.995481+0000 mgr.y (mgr.24380) 843 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:13:37.679 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:13:37 smithi023 bash[29146]: audit 2024-03-11T21:13:36.345451+0000 mon.a (mon.0) 1869 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:13:37.679 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:13:37 smithi023 bash[29146]: audit 2024-03-11T21:13:36.347788+0000 mon.a (mon.0) 1870 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:13:37.679 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:13:37 smithi023 bash[29146]: cluster 2024-03-11T21:13:36.395990+0000 mgr.y (mgr.24380) 844 : cluster 0 pgmap v971: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+remapped, 3 active+remapped+backfilling, 153 active+clean; 5.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.1 KiB/s rd, 46 MiB/s wr, 766 op/s; 23831/291450 objects misplaced (8.177%); 31 MiB/s, 502 objects/s recovering 2024-03-11T21:13:37.707 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:13:37.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:37.710 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:13:37.756 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:13:37 smithi146 bash[23809]: audit 2024-03-11T21:13:35.995481+0000 mgr.y (mgr.24380) 843 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:13:37.756 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:13:37 smithi146 bash[23809]: audit 2024-03-11T21:13:36.345451+0000 mon.a (mon.0) 1869 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:13:37.756 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:13:37 smithi146 bash[23809]: audit 2024-03-11T21:13:36.347788+0000 mon.a (mon.0) 1870 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:13:37.756 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:13:37 smithi146 bash[23809]: cluster 2024-03-11T21:13:36.395990+0000 mgr.y (mgr.24380) 844 : cluster 0 pgmap v971: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+remapped, 3 active+remapped+backfilling, 153 active+clean; 5.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.1 KiB/s rd, 46 MiB/s wr, 766 op/s; 23831/291450 objects misplaced (8.177%); 31 MiB/s, 502 objects/s recovering 2024-03-11T21:13:37.811 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:13:37.862 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:13:37.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:37.865 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:13:37.965 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:13:38.005 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:13:38.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:38.012 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:13:38.113 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:38.171 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:38.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:38.176 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:13:38.277 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:13:38.299 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:13:38.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:38.302 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:13:38.403 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:13:38.427 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:13:38.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:38.434 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:13:38.535 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:38.558 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:38.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:38.560 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:13:38.661 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:38.673 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:13:38 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:13:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:13:38.692 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:38.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:38.696 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:13:38.796 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:13:38.829 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:13:38.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:38.833 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:13:38.934 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:38.957 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:38.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:38.959 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:13:39.060 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:39.088 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:39.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:39.093 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:13:39.194 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:13:39.224 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:13:39.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:39.231 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:13:39.332 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:13:39.361 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:13:39.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:39.363 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:13:39.464 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:13:39.496 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:13:39.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:39.499 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:13:39.600 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:39.610 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:13:39 smithi023 bash[21372]: cluster 2024-03-11T21:13:38.396619+0000 mgr.y (mgr.24380) 845 : cluster 0 pgmap v972: 158 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 153 active+clean; 5.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.1 KiB/s rd, 44 MiB/s wr, 790 op/s; 23749/290766 objects misplaced (8.168%); 31 MiB/s, 503 objects/s recovering 2024-03-11T21:13:39.610 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:13:39 smithi023 bash[21372]: cluster 2024-03-11T21:13:38.657211+0000 osd.6 (osd.6) 401 : cluster 0 purged_snaps scrub starts 2024-03-11T21:13:39.610 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:13:39 smithi023 bash[21372]: cluster 2024-03-11T21:13:38.657288+0000 osd.6 (osd.6) 402 : cluster 0 purged_snaps scrub ok 2024-03-11T21:13:39.610 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:13:39 smithi023 bash[29146]: cluster 2024-03-11T21:13:38.396619+0000 mgr.y (mgr.24380) 845 : cluster 0 pgmap v972: 158 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 153 active+clean; 5.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.1 KiB/s rd, 44 MiB/s wr, 790 op/s; 23749/290766 objects misplaced (8.168%); 31 MiB/s, 503 objects/s recovering 2024-03-11T21:13:39.611 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:13:39 smithi023 bash[29146]: cluster 2024-03-11T21:13:38.657211+0000 osd.6 (osd.6) 401 : cluster 0 purged_snaps scrub starts 2024-03-11T21:13:39.611 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:13:39 smithi023 bash[29146]: cluster 2024-03-11T21:13:38.657288+0000 osd.6 (osd.6) 402 : cluster 0 purged_snaps scrub ok 2024-03-11T21:13:39.628 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:39.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:39.634 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:13:39.734 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:39.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:13:39 smithi146 bash[23809]: cluster 2024-03-11T21:13:38.396619+0000 mgr.y (mgr.24380) 845 : cluster 0 pgmap v972: 158 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 153 active+clean; 5.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.1 KiB/s rd, 44 MiB/s wr, 790 op/s; 23749/290766 objects misplaced (8.168%); 31 MiB/s, 503 objects/s recovering 2024-03-11T21:13:39.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:13:39 smithi146 bash[23809]: cluster 2024-03-11T21:13:38.657211+0000 osd.6 (osd.6) 401 : cluster 0 purged_snaps scrub starts 2024-03-11T21:13:39.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:13:39 smithi146 bash[23809]: cluster 2024-03-11T21:13:38.657288+0000 osd.6 (osd.6) 402 : cluster 0 purged_snaps scrub ok 2024-03-11T21:13:39.763 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:39.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:39.766 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:13:39.866 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:39.896 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:39.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:39.899 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:13:40.000 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:13:40.026 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:13:40.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:40.027 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:13:40.128 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:13:40.162 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:13:40.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:40.168 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:13:40.269 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:40.293 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:40.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:40.298 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:13:40.399 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:13:40.422 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:13:40.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:40.425 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:13:40.526 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:40.536 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:13:40 smithi146 bash[23809]: cluster 2024-03-11T21:13:39.462923+0000 mon.a (mon.0) 1871 : cluster 0 osdmap e409: 8 total, 8 up, 7 in 2024-03-11T21:13:40.536 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:13:40 smithi146 bash[23809]: cluster 2024-03-11T21:13:39.968433+0000 osd.4 (osd.4) 115 : cluster 0 purged_snaps scrub starts 2024-03-11T21:13:40.536 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:13:40 smithi146 bash[23809]: cluster 2024-03-11T21:13:39.968517+0000 osd.4 (osd.4) 116 : cluster 0 purged_snaps scrub ok 2024-03-11T21:13:40.555 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:40.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:40.562 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:13:40.663 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:40.705 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:40.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:40.711 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:13:40.776 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:13:40 smithi023 bash[21372]: cluster 2024-03-11T21:13:39.462923+0000 mon.a (mon.0) 1871 : cluster 0 osdmap e409: 8 total, 8 up, 7 in 2024-03-11T21:13:40.776 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:13:40 smithi023 bash[21372]: cluster 2024-03-11T21:13:39.968433+0000 osd.4 (osd.4) 115 : cluster 0 purged_snaps scrub starts 2024-03-11T21:13:40.776 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:13:40 smithi023 bash[21372]: cluster 2024-03-11T21:13:39.968517+0000 osd.4 (osd.4) 116 : cluster 0 purged_snaps scrub ok 2024-03-11T21:13:40.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:13:40 smithi023 bash[29146]: cluster 2024-03-11T21:13:39.462923+0000 mon.a (mon.0) 1871 : cluster 0 osdmap e409: 8 total, 8 up, 7 in 2024-03-11T21:13:40.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:13:40 smithi023 bash[29146]: cluster 2024-03-11T21:13:39.968433+0000 osd.4 (osd.4) 115 : cluster 0 purged_snaps scrub starts 2024-03-11T21:13:40.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:13:40 smithi023 bash[29146]: cluster 2024-03-11T21:13:39.968517+0000 osd.4 (osd.4) 116 : cluster 0 purged_snaps scrub ok 2024-03-11T21:13:40.811 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:40.843 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:40.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:40.846 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:13:40.947 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:13:40.972 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:13:40.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:40.974 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:13:41.075 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:41.094 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:41.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:41.096 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:13:41.197 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:13:41.255 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:13:41.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:41.259 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:13:41.360 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:41.384 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:41.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:41.389 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:13:41.490 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:41.514 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:41.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:41.517 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:13:41.526 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:13:41 smithi023 bash[21372]: cluster 2024-03-11T21:13:40.397328+0000 mgr.y (mgr.24380) 846 : cluster 0 pgmap v974: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+remapped, 3 active+remapped+backfilling, 153 active+clean; 5.8 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 1.4 KiB/s rd, 29 MiB/s wr, 869 op/s; 21689/283971 objects misplaced (7.638%); 33 MiB/s, 528 objects/s recovering 2024-03-11T21:13:41.526 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:13:41 smithi023 bash[21372]: cluster 2024-03-11T21:13:40.496722+0000 mon.a (mon.0) 1872 : cluster 0 osdmap e410: 8 total, 8 up, 7 in 2024-03-11T21:13:41.526 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:13:41 smithi023 bash[29146]: cluster 2024-03-11T21:13:40.397328+0000 mgr.y (mgr.24380) 846 : cluster 0 pgmap v974: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+remapped, 3 active+remapped+backfilling, 153 active+clean; 5.8 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 1.4 KiB/s rd, 29 MiB/s wr, 869 op/s; 21689/283971 objects misplaced (7.638%); 33 MiB/s, 528 objects/s recovering 2024-03-11T21:13:41.526 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:13:41 smithi023 bash[29146]: cluster 2024-03-11T21:13:40.496722+0000 mon.a (mon.0) 1872 : cluster 0 osdmap e410: 8 total, 8 up, 7 in 2024-03-11T21:13:41.617 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:41.655 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:41.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:41.657 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:13:41.757 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:13:41 smithi146 bash[23809]: cluster 2024-03-11T21:13:40.397328+0000 mgr.y (mgr.24380) 846 : cluster 0 pgmap v974: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+remapped, 3 active+remapped+backfilling, 153 active+clean; 5.8 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 1.4 KiB/s rd, 29 MiB/s wr, 869 op/s; 21689/283971 objects misplaced (7.638%); 33 MiB/s, 528 objects/s recovering 2024-03-11T21:13:41.757 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:13:41 smithi146 bash[23809]: cluster 2024-03-11T21:13:40.496722+0000 mon.a (mon.0) 1872 : cluster 0 osdmap e410: 8 total, 8 up, 7 in 2024-03-11T21:13:41.758 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:41.791 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:41.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:41.795 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:13:41.896 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:13:41.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:41.924 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:13:41.924 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:13:42.025 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:42.064 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:42.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:42.068 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:13:42.169 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:42.200 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:42.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:42.206 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:13:42.307 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:13:42.336 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:13:42.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:42.341 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:13:42.442 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:13:42.482 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:13:42.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:42.487 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:13:42.588 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:13:42.630 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:13:42.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:42.633 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:13:42.734 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:13:42.762 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:13:42.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:42.767 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:13:42.867 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:13:42.903 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:13:42.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:42.906 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:13:43.007 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:13:43.042 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:13:43.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:43.047 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:13:43.148 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:43.182 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:43.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:43.185 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:13:43.286 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:43.311 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:43.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:43.315 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:13:43.416 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:13:43.447 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:13:43.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:43.449 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:13:43.550 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:13:43.587 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:13:43.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:43.589 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:13:43.690 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:43.699 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:13:43 smithi146 bash[23809]: cluster 2024-03-11T21:13:42.397842+0000 mgr.y (mgr.24380) 847 : cluster 0 pgmap v976: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+remapped, 3 active+remapped+backfilling, 153 active+clean; 5.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.4 KiB/s rd, 22 MiB/s wr, 888 op/s; 20767/280506 objects misplaced (7.403%); 38 MiB/s, 610 objects/s recovering 2024-03-11T21:13:43.718 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:43.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:43.721 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:13:43.776 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:13:43 smithi023 bash[21372]: cluster 2024-03-11T21:13:42.397842+0000 mgr.y (mgr.24380) 847 : cluster 0 pgmap v976: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+remapped, 3 active+remapped+backfilling, 153 active+clean; 5.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.4 KiB/s rd, 22 MiB/s wr, 888 op/s; 20767/280506 objects misplaced (7.403%); 38 MiB/s, 610 objects/s recovering 2024-03-11T21:13:43.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:13:43 smithi023 bash[29146]: cluster 2024-03-11T21:13:42.397842+0000 mgr.y (mgr.24380) 847 : cluster 0 pgmap v976: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+remapped, 3 active+remapped+backfilling, 153 active+clean; 5.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.4 KiB/s rd, 22 MiB/s wr, 888 op/s; 20767/280506 objects misplaced (7.403%); 38 MiB/s, 610 objects/s recovering 2024-03-11T21:13:43.822 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:43.851 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:43.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:43.853 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:13:43.954 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:43.974 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:43.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:43.976 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:13:44.077 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:13:44.108 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:13:44.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:44.112 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:13:44.212 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:44.243 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:44.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:44.245 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:13:44.346 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:13:44.387 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:13:44.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:44.393 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:13:44.494 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:44.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:44.526 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:44.526 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:13:44.627 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:13:44.661 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:13:44.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:44.666 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:13:44.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:13:44 smithi146 bash[23809]: cluster 2024-03-11T21:13:44.500467+0000 mon.a (mon.0) 1873 : cluster 3 Health check failed: Degraded data redundancy: 80/276999 objects degraded (0.029%), 1 pg degraded (PG_DEGRADED) 2024-03-11T21:13:44.767 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:13:44.776 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:13:44 smithi023 bash[21372]: cluster 2024-03-11T21:13:44.500467+0000 mon.a (mon.0) 1873 : cluster 3 Health check failed: Degraded data redundancy: 80/276999 objects degraded (0.029%), 1 pg degraded (PG_DEGRADED) 2024-03-11T21:13:44.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:13:44 smithi023 bash[29146]: cluster 2024-03-11T21:13:44.500467+0000 mon.a (mon.0) 1873 : cluster 3 Health check failed: Degraded data redundancy: 80/276999 objects degraded (0.029%), 1 pg degraded (PG_DEGRADED) 2024-03-11T21:13:44.793 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:13:44.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:44.796 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:13:44.897 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:13:44.923 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:13:44.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:44.928 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:13:45.029 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:13:45.055 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:13:45.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:45.060 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:13:45.161 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:13:45.186 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:13:45.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:45.189 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:13:45.290 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:13:45.322 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:13:45.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:45.325 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:13:45.426 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:13:45.457 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:13:45.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:45.462 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:13:45.562 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:45.593 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:45.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:45.597 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:13:45.655 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:13:45 smithi023 bash[21372]: cluster 2024-03-11T21:13:44.400140+0000 mgr.y (mgr.24380) 848 : cluster 0 pgmap v977: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 153 active+clean; 5.6 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 639 B/s rd, 3.1 MiB/s wr, 702 op/s; 80/276999 objects degraded (0.029%); 20585/276999 objects misplaced (7.431%); 18 MiB/s, 292 objects/s recovering 2024-03-11T21:13:45.656 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:13:45 smithi023 bash[29146]: cluster 2024-03-11T21:13:44.400140+0000 mgr.y (mgr.24380) 848 : cluster 0 pgmap v977: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 153 active+clean; 5.6 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 639 B/s rd, 3.1 MiB/s wr, 702 op/s; 80/276999 objects degraded (0.029%); 20585/276999 objects misplaced (7.431%); 18 MiB/s, 292 objects/s recovering 2024-03-11T21:13:45.697 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:45.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:13:45 smithi146 bash[23809]: cluster 2024-03-11T21:13:44.400140+0000 mgr.y (mgr.24380) 848 : cluster 0 pgmap v977: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 153 active+clean; 5.6 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 639 B/s rd, 3.1 MiB/s wr, 702 op/s; 80/276999 objects degraded (0.029%); 20585/276999 objects misplaced (7.431%); 18 MiB/s, 292 objects/s recovering 2024-03-11T21:13:45.717 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:45.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:45.722 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:13:45.822 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:13:45.862 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:13:45.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:45.865 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:13:45.966 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:45.991 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:45.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:45.993 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:13:46.094 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:13:46.120 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:13:46.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:46.126 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:13:46.227 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:46.236 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:13:46 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:13:46.257 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:46.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:46.261 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:13:46.362 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:13:46.389 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:13:46.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:46.393 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:13:46.494 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:13:46.526 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:13:46.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:46.529 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:13:46.630 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:13:46.652 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:13:46.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:46.655 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:13:46.756 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:46.785 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:46.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:46.789 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:13:46.890 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:13:46.908 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:13:46.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:46.910 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:13:47.011 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:13:47.031 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:13:47.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:47.033 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:13:47.133 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:13:47.155 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:13:47.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:47.157 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:13:47.258 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:47.298 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:47.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:47.300 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:13:47.401 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:13:47.436 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:13:47.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:47.440 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:13:47.540 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:13:47.576 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:13:47.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:47.582 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:13:47.682 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:13:47.705 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:13:47.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:47.708 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:13:47.808 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:13:47.817 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:13:47 smithi146 bash[23809]: audit 2024-03-11T21:13:46.004779+0000 mgr.y (mgr.24380) 849 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:13:47.817 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:13:47 smithi146 bash[23809]: cluster 2024-03-11T21:13:46.400878+0000 mgr.y (mgr.24380) 850 : cluster 0 pgmap v978: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfilling, 153 active+clean; 5.4 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 993 op/s; 80/266928 objects degraded (0.030%); 17820/266928 objects misplaced (6.676%); 34 MiB/s, 546 objects/s recovering 2024-03-11T21:13:47.829 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:13:47.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:47.831 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:13:47.932 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:13:47.940 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:13:47 smithi023 bash[21372]: audit 2024-03-11T21:13:46.004779+0000 mgr.y (mgr.24380) 849 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:13:47.940 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:13:47 smithi023 bash[21372]: cluster 2024-03-11T21:13:46.400878+0000 mgr.y (mgr.24380) 850 : cluster 0 pgmap v978: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfilling, 153 active+clean; 5.4 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 993 op/s; 80/266928 objects degraded (0.030%); 17820/266928 objects misplaced (6.676%); 34 MiB/s, 546 objects/s recovering 2024-03-11T21:13:47.940 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:13:47 smithi023 bash[29146]: audit 2024-03-11T21:13:46.004779+0000 mgr.y (mgr.24380) 849 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:13:47.940 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:13:47 smithi023 bash[29146]: cluster 2024-03-11T21:13:46.400878+0000 mgr.y (mgr.24380) 850 : cluster 0 pgmap v978: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfilling, 153 active+clean; 5.4 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 993 op/s; 80/266928 objects degraded (0.030%); 17820/266928 objects misplaced (6.676%); 34 MiB/s, 546 objects/s recovering 2024-03-11T21:13:47.955 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:13:47.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:47.957 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:13:48.057 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:13:48.093 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:13:48.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:48.098 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:13:48.199 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:13:48.226 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:13:48.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:48.229 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:13:48.330 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:13:48.356 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:13:48.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:48.359 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:13:48.460 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:13:48.495 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:13:48.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:48.499 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:13:48.525 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:13:48 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:13:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:13:48.600 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:48.619 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:48.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:48.621 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:13:48.722 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:13:48.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:48.751 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:13:48.751 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:13:48.852 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:13:48.872 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:13:48.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:48.873 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:13:48.974 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:48.993 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:48.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:48.995 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:13:49.096 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:13:49.126 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:13:49.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:49.130 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:13:49.231 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:13:49.270 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:13:49.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:49.274 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:13:49.375 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:49.405 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:49.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:49.410 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:13:49.511 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:49.538 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:49.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:49.545 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:13:49.646 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:13:49.675 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:13:49.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:49.678 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:13:49.779 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:13:49.819 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:13:49.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:49.822 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:13:49.923 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:13:49.931 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:13:49 smithi023 bash[21372]: cluster 2024-03-11T21:13:48.407283+0000 mgr.y (mgr.24380) 851 : cluster 0 pgmap v979: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 153 active+clean; 5.4 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.00k op/s; 269/263868 objects degraded (0.102%); 17545/263868 objects misplaced (6.649%); 31 MiB/s, 498 objects/s recovering 2024-03-11T21:13:49.932 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:13:49 smithi023 bash[21372]: cluster 2024-03-11T21:13:48.818924+0000 osd.5 (osd.5) 489 : cluster 0 purged_snaps scrub starts 2024-03-11T21:13:49.932 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:13:49 smithi023 bash[21372]: cluster 2024-03-11T21:13:48.818989+0000 osd.5 (osd.5) 490 : cluster 0 purged_snaps scrub ok 2024-03-11T21:13:49.932 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:13:49 smithi023 bash[29146]: cluster 2024-03-11T21:13:48.407283+0000 mgr.y (mgr.24380) 851 : cluster 0 pgmap v979: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 153 active+clean; 5.4 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.00k op/s; 269/263868 objects degraded (0.102%); 17545/263868 objects misplaced (6.649%); 31 MiB/s, 498 objects/s recovering 2024-03-11T21:13:49.932 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:13:49 smithi023 bash[29146]: cluster 2024-03-11T21:13:48.818924+0000 osd.5 (osd.5) 489 : cluster 0 purged_snaps scrub starts 2024-03-11T21:13:49.932 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:13:49 smithi023 bash[29146]: cluster 2024-03-11T21:13:48.818989+0000 osd.5 (osd.5) 490 : cluster 0 purged_snaps scrub ok 2024-03-11T21:13:49.947 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:13:49.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:49.950 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:13:50.006 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:13:49 smithi146 bash[23809]: cluster 2024-03-11T21:13:48.407283+0000 mgr.y (mgr.24380) 851 : cluster 0 pgmap v979: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 153 active+clean; 5.4 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.00k op/s; 269/263868 objects degraded (0.102%); 17545/263868 objects misplaced (6.649%); 31 MiB/s, 498 objects/s recovering 2024-03-11T21:13:50.006 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:13:49 smithi146 bash[23809]: cluster 2024-03-11T21:13:48.818924+0000 osd.5 (osd.5) 489 : cluster 0 purged_snaps scrub starts 2024-03-11T21:13:50.006 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:13:49 smithi146 bash[23809]: cluster 2024-03-11T21:13:48.818989+0000 osd.5 (osd.5) 490 : cluster 0 purged_snaps scrub ok 2024-03-11T21:13:50.051 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:50.086 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:50.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:50.089 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:13:50.190 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:50.220 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:50.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:50.225 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:13:50.325 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:50.354 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:50.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:50.358 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:13:50.459 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:50.496 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:50.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:50.500 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:13:50.600 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:50.630 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:50.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:50.635 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:13:50.736 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:13:50.737 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:13:50 smithi023 bash[21372]: cluster 2024-03-11T21:13:49.587503+0000 mon.a (mon.0) 1874 : cluster 0 osdmap e411: 8 total, 8 up, 7 in 2024-03-11T21:13:50.737 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:13:50 smithi023 bash[21372]: cluster 2024-03-11T21:13:50.380793+0000 mon.a (mon.0) 1875 : cluster 3 Health check update: Degraded data redundancy: 269/263868 objects degraded (0.102%), 1 pg degraded (PG_DEGRADED) 2024-03-11T21:13:50.770 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:13:50.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:50.773 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:13:50.874 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:50.884 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:13:50 smithi146 bash[23809]: cluster 2024-03-11T21:13:49.587503+0000 mon.a (mon.0) 1874 : cluster 0 osdmap e411: 8 total, 8 up, 7 in 2024-03-11T21:13:50.884 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:13:50 smithi146 bash[23809]: cluster 2024-03-11T21:13:50.380793+0000 mon.a (mon.0) 1875 : cluster 3 Health check update: Degraded data redundancy: 269/263868 objects degraded (0.102%), 1 pg degraded (PG_DEGRADED) 2024-03-11T21:13:50.902 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:50.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:50.905 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:13:51.006 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:13:51.031 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:13:51.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:51.033 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:13:51.044 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:13:50 smithi023 bash[29146]: cluster 2024-03-11T21:13:49.587503+0000 mon.a (mon.0) 1874 : cluster 0 osdmap e411: 8 total, 8 up, 7 in 2024-03-11T21:13:51.044 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:13:50 smithi023 bash[29146]: cluster 2024-03-11T21:13:50.380793+0000 mon.a (mon.0) 1875 : cluster 3 Health check update: Degraded data redundancy: 269/263868 objects degraded (0.102%), 1 pg degraded (PG_DEGRADED) 2024-03-11T21:13:51.134 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:51.165 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:51.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:51.174 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:13:51.275 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:51.298 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:51.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:51.302 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:13:51.403 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:13:51.429 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:13:51.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:51.436 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:13:51.537 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:13:51.563 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:13:51.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:51.565 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:13:51.666 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:13:51.692 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:13:51.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:51.695 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:13:51.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:13:51 smithi146 bash[23809]: cluster 2024-03-11T21:13:50.407922+0000 mgr.y (mgr.24380) 852 : cluster 0 pgmap v981: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfilling, 153 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 778 op/s; 269/258183 objects degraded (0.104%); 15804/258183 objects misplaced (6.121%); 23 MiB/s, 367 objects/s recovering 2024-03-11T21:13:51.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:13:51 smithi146 bash[23809]: audit 2024-03-11T21:13:51.371304+0000 mon.a (mon.0) 1876 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:13:51.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:13:51 smithi146 bash[23809]: audit 2024-03-11T21:13:51.376201+0000 mon.a (mon.0) 1877 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:13:51.796 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:13:51.823 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:13:51.826 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:13:51 smithi023 bash[29146]: cluster 2024-03-11T21:13:50.407922+0000 mgr.y (mgr.24380) 852 : cluster 0 pgmap v981: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfilling, 153 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 778 op/s; 269/258183 objects degraded (0.104%); 15804/258183 objects misplaced (6.121%); 23 MiB/s, 367 objects/s recovering 2024-03-11T21:13:51.826 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:13:51 smithi023 bash[29146]: audit 2024-03-11T21:13:51.371304+0000 mon.a (mon.0) 1876 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:13:51.826 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:13:51 smithi023 bash[29146]: audit 2024-03-11T21:13:51.376201+0000 mon.a (mon.0) 1877 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:13:51.826 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:13:51 smithi023 bash[21372]: cluster 2024-03-11T21:13:50.407922+0000 mgr.y (mgr.24380) 852 : cluster 0 pgmap v981: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfilling, 153 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 778 op/s; 269/258183 objects degraded (0.104%); 15804/258183 objects misplaced (6.121%); 23 MiB/s, 367 objects/s recovering 2024-03-11T21:13:51.826 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:13:51 smithi023 bash[21372]: audit 2024-03-11T21:13:51.371304+0000 mon.a (mon.0) 1876 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:13:51.826 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:13:51 smithi023 bash[21372]: audit 2024-03-11T21:13:51.376201+0000 mon.a (mon.0) 1877 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:13:51.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:51.827 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:13:51.928 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:13:51.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:51.957 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:13:51.957 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:13:52.058 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:52.085 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:52.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:52.089 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:13:52.190 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:13:52.223 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:13:52.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:52.229 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:13:52.330 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:52.352 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:52.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:52.354 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:13:52.455 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:13:52.486 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:13:52.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:52.490 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:13:52.590 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:13:52.618 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:13:52.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:52.623 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:13:52.724 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:13:52.772 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:13:52.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:52.777 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:13:52.878 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:52.898 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:52.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:52.901 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:13:52.904 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-11T21:13:52.904 INFO:teuthology.orchestra.run.smithi023.stdout: "ops": [], 2024-03-11T21:13:52.904 INFO:teuthology.orchestra.run.smithi023.stdout: "complaint_time": 30, 2024-03-11T21:13:52.904 INFO:teuthology.orchestra.run.smithi023.stdout: "num_blocked_ops": 0 2024-03-11T21:13:52.904 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-11T21:13:52.915 INFO:teuthology.orchestra.run.smithi023.stdout:osd.0: {} 2024-03-11T21:13:52.994 INFO:teuthology.orchestra.run.smithi023.stdout:osd.1: {} 2024-03-11T21:13:53.002 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:53.021 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:53.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:53.024 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:13:53.063 INFO:teuthology.orchestra.run.smithi023.stdout:osd.2: {} 2024-03-11T21:13:53.124 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:53.131 INFO:teuthology.orchestra.run.smithi023.stdout:osd.3: {} 2024-03-11T21:13:53.166 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:53.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:53.168 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:13:53.193 INFO:teuthology.orchestra.run.smithi023.stdout:osd.4: {} 2024-03-11T21:13:53.235 INFO:teuthology.orchestra.run.smithi023.stdout:osd.5: {} 2024-03-11T21:13:53.268 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:53.274 INFO:teuthology.orchestra.run.smithi023.stdout:osd.6: {} 2024-03-11T21:13:53.293 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:53.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:53.295 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:13:53.316 INFO:teuthology.orchestra.run.smithi023.stdout:osd.7: {} 2024-03-11T21:13:53.318 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-11T21:13:53.326 INFO:teuthology.orchestra.run.smithi023.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:13:53.326 INFO:teuthology.orchestra.run.smithi023.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:13:53.326 INFO:teuthology.orchestra.run.smithi023.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:13:53.326 INFO:teuthology.orchestra.run.smithi023.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:13:53.326 INFO:teuthology.orchestra.run.smithi023.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:13:53.326 INFO:teuthology.orchestra.run.smithi023.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:13:53.326 INFO:teuthology.orchestra.run.smithi023.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:13:53.326 INFO:teuthology.orchestra.run.smithi023.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:13:53.326 INFO:teuthology.orchestra.run.smithi023.stderr:dumped all 2024-03-11T21:13:53.396 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:53.417 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:53.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:53.419 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:13:53.520 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:53.554 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:53.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:53.560 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:13:53.603 INFO:teuthology.orchestra.run.smithi023.stderr:noscrub is unset 2024-03-11T21:13:53.660 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:53.686 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:53.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:53.693 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:13:53.794 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:13:53.821 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:13:53.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:53.827 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:13:53.928 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:13:53.939 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:13:53 smithi146 bash[23809]: cluster 2024-03-11T21:13:52.408628+0000 mgr.y (mgr.24380) 853 : cluster 0 pgmap v982: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfilling, 153 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1022 B/s rd, 0 B/s wr, 836 op/s; 269/255405 objects degraded (0.105%); 14898/255405 objects misplaced (5.833%); 28 MiB/s, 446 objects/s recovering 2024-03-11T21:13:53.939 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:13:53 smithi146 bash[23809]: audit 2024-03-11T21:13:52.866167+0000 mon.c (mon.2) 96 : audit 0 from='client.? 172.21.15.23:0/435375326' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:13:53.939 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:13:53 smithi146 bash[23809]: audit 2024-03-11T21:13:53.316351+0000 mon.c (mon.2) 97 : audit 1 from='client.? 172.21.15.23:0/3631959431' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:13:53.939 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:13:53 smithi146 bash[23809]: audit 2024-03-11T21:13:53.316846+0000 mon.a (mon.0) 1878 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:13:53.952 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:13:53.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:53.956 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:13:54.026 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:13:53 smithi023 bash[21372]: cluster 2024-03-11T21:13:52.408628+0000 mgr.y (mgr.24380) 853 : cluster 0 pgmap v982: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfilling, 153 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1022 B/s rd, 0 B/s wr, 836 op/s; 269/255405 objects degraded (0.105%); 14898/255405 objects misplaced (5.833%); 28 MiB/s, 446 objects/s recovering 2024-03-11T21:13:54.026 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:13:53 smithi023 bash[21372]: audit 2024-03-11T21:13:52.866167+0000 mon.c (mon.2) 96 : audit 0 from='client.? 172.21.15.23:0/435375326' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:13:54.026 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:13:53 smithi023 bash[21372]: audit 2024-03-11T21:13:53.316351+0000 mon.c (mon.2) 97 : audit 1 from='client.? 172.21.15.23:0/3631959431' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:13:54.026 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:13:53 smithi023 bash[21372]: audit 2024-03-11T21:13:53.316846+0000 mon.a (mon.0) 1878 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:13:54.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:13:53 smithi023 bash[29146]: cluster 2024-03-11T21:13:52.408628+0000 mgr.y (mgr.24380) 853 : cluster 0 pgmap v982: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfilling, 153 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1022 B/s rd, 0 B/s wr, 836 op/s; 269/255405 objects degraded (0.105%); 14898/255405 objects misplaced (5.833%); 28 MiB/s, 446 objects/s recovering 2024-03-11T21:13:54.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:13:53 smithi023 bash[29146]: audit 2024-03-11T21:13:52.866167+0000 mon.c (mon.2) 96 : audit 0 from='client.? 172.21.15.23:0/435375326' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:13:54.027 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:13:53 smithi023 bash[29146]: audit 2024-03-11T21:13:53.316351+0000 mon.c (mon.2) 97 : audit 1 from='client.? 172.21.15.23:0/3631959431' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:13:54.027 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:13:53 smithi023 bash[29146]: audit 2024-03-11T21:13:53.316846+0000 mon.a (mon.0) 1878 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:13:54.056 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:13:54.089 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:13:54.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:54.093 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:13:54.193 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:13:54.216 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:13:54.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:54.220 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:13:54.320 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:13:54.345 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:13:54.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:54.347 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:13:54.448 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:13:54.475 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:13:54.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:54.478 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:13:54.579 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:54.602 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:54.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:54.605 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:13:54.706 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:54.730 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:54.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:54.733 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:13:54.834 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:54.871 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:54.874 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:13:54 smithi023 bash[21372]: audit 2024-03-11T21:13:53.311476+0000 mgr.y (mgr.24380) 854 : audit 0 from='client.25436 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:13:54.875 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:13:54 smithi023 bash[21372]: cluster 2024-03-11T21:13:53.587779+0000 mon.a (mon.0) 1879 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:13:54.875 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:13:54 smithi023 bash[21372]: audit 2024-03-11T21:13:53.591986+0000 mon.a (mon.0) 1880 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:13:54.875 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:13:54 smithi023 bash[21372]: cluster 2024-03-11T21:13:53.598345+0000 mon.a (mon.0) 1881 : cluster 0 osdmap e412: 8 total, 8 up, 7 in 2024-03-11T21:13:54.875 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:13:54 smithi023 bash[29146]: audit 2024-03-11T21:13:53.311476+0000 mgr.y (mgr.24380) 854 : audit 0 from='client.25436 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:13:54.875 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:13:54 smithi023 bash[29146]: cluster 2024-03-11T21:13:53.587779+0000 mon.a (mon.0) 1879 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:13:54.875 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:13:54 smithi023 bash[29146]: audit 2024-03-11T21:13:53.591986+0000 mon.a (mon.0) 1880 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:13:54.875 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:13:54 smithi023 bash[29146]: cluster 2024-03-11T21:13:53.598345+0000 mon.a (mon.0) 1881 : cluster 0 osdmap e412: 8 total, 8 up, 7 in 2024-03-11T21:13:54.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:54.877 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:13:54.977 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:54.987 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:13:54 smithi146 bash[23809]: audit 2024-03-11T21:13:53.311476+0000 mgr.y (mgr.24380) 854 : audit 0 from='client.25436 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:13:54.987 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:13:54 smithi146 bash[23809]: cluster 2024-03-11T21:13:53.587779+0000 mon.a (mon.0) 1879 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:13:54.987 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:13:54 smithi146 bash[23809]: audit 2024-03-11T21:13:53.591986+0000 mon.a (mon.0) 1880 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:13:54.987 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:13:54 smithi146 bash[23809]: cluster 2024-03-11T21:13:53.598345+0000 mon.a (mon.0) 1881 : cluster 0 osdmap e412: 8 total, 8 up, 7 in 2024-03-11T21:13:54.999 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:55.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:55.007 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:13:55.108 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:13:55.129 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:13:55.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:55.132 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:13:55.233 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:55.263 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:55.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:55.265 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:13:55.365 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:13:55.405 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:13:55.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:55.409 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:13:55.510 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:13:55.537 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:13:55.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:55.538 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:13:55.639 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:55.672 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:55.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:55.674 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:13:55.775 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:13:55.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:13:55 smithi023 bash[29146]: cluster 2024-03-11T21:13:54.409136+0000 mgr.y (mgr.24380) 855 : cluster 0 pgmap v984: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 153 active+clean; 5.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 604 op/s; 419/252429 objects degraded (0.166%); 14648/252429 objects misplaced (5.803%); 18 MiB/s, 294 objects/s recovering 2024-03-11T21:13:55.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:13:55 smithi023 bash[29146]: cluster 2024-03-11T21:13:55.285471+0000 osd.5 (osd.5) 491 : cluster 0 1.a scrub starts 2024-03-11T21:13:55.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:13:55 smithi023 bash[29146]: cluster 2024-03-11T21:13:55.286748+0000 osd.5 (osd.5) 492 : cluster 0 1.a scrub ok 2024-03-11T21:13:55.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:13:55 smithi023 bash[29146]: cluster 2024-03-11T21:13:55.381897+0000 mon.a (mon.0) 1882 : cluster 3 Health check update: Degraded data redundancy: 419/252429 objects degraded (0.166%), 1 pg degraded (PG_DEGRADED) 2024-03-11T21:13:55.776 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:13:55 smithi023 bash[21372]: cluster 2024-03-11T21:13:54.409136+0000 mgr.y (mgr.24380) 855 : cluster 0 pgmap v984: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 153 active+clean; 5.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 604 op/s; 419/252429 objects degraded (0.166%); 14648/252429 objects misplaced (5.803%); 18 MiB/s, 294 objects/s recovering 2024-03-11T21:13:55.776 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:13:55 smithi023 bash[21372]: cluster 2024-03-11T21:13:55.285471+0000 osd.5 (osd.5) 491 : cluster 0 1.a scrub starts 2024-03-11T21:13:55.777 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:13:55 smithi023 bash[21372]: cluster 2024-03-11T21:13:55.286748+0000 osd.5 (osd.5) 492 : cluster 0 1.a scrub ok 2024-03-11T21:13:55.777 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:13:55 smithi023 bash[21372]: cluster 2024-03-11T21:13:55.381897+0000 mon.a (mon.0) 1882 : cluster 3 Health check update: Degraded data redundancy: 419/252429 objects degraded (0.166%), 1 pg degraded (PG_DEGRADED) 2024-03-11T21:13:55.799 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:13:55.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:55.803 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:13:55.904 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:13:55.937 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:13:55.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:55.942 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:13:56.001 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:13:55 smithi146 bash[23809]: cluster 2024-03-11T21:13:54.409136+0000 mgr.y (mgr.24380) 855 : cluster 0 pgmap v984: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 153 active+clean; 5.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 604 op/s; 419/252429 objects degraded (0.166%); 14648/252429 objects misplaced (5.803%); 18 MiB/s, 294 objects/s recovering 2024-03-11T21:13:56.001 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:13:55 smithi146 bash[23809]: cluster 2024-03-11T21:13:55.285471+0000 osd.5 (osd.5) 491 : cluster 0 1.a scrub starts 2024-03-11T21:13:56.001 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:13:55 smithi146 bash[23809]: cluster 2024-03-11T21:13:55.286748+0000 osd.5 (osd.5) 492 : cluster 0 1.a scrub ok 2024-03-11T21:13:56.001 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:13:55 smithi146 bash[23809]: cluster 2024-03-11T21:13:55.381897+0000 mon.a (mon.0) 1882 : cluster 3 Health check update: Degraded data redundancy: 419/252429 objects degraded (0.166%), 1 pg degraded (PG_DEGRADED) 2024-03-11T21:13:56.043 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:56.071 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:56.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:56.075 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:13:56.175 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:56.205 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:56.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:56.209 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:13:56.255 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:13:56 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:13:56.310 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:56.336 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:56.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:56.340 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:13:56.441 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:56.472 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:56.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:56.475 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:13:56.576 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:13:56.605 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:13:56.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:56.609 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:13:56.710 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:56.737 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:56.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:56.740 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:13:56.776 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:13:56 smithi023 bash[21372]: cluster 2024-03-11T21:13:56.615331+0000 mon.a (mon.0) 1883 : cluster 1 Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-11T21:13:56.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:13:56 smithi023 bash[29146]: cluster 2024-03-11T21:13:56.615331+0000 mon.a (mon.0) 1883 : cluster 1 Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-11T21:13:56.841 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:13:56.852 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:13:56 smithi146 bash[23809]: cluster 2024-03-11T21:13:56.615331+0000 mon.a (mon.0) 1883 : cluster 1 Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-11T21:13:56.870 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:13:56.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:56.874 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:13:56.975 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:57.003 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:57.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:57.008 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:13:57.109 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:57.138 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:57.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:57.140 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:13:57.241 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:13:57.275 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:13:57.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:57.282 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:13:57.383 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:13:57.404 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:13:57.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:57.409 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:13:57.510 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:57.541 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:57.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:57.546 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:13:57.646 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:57.670 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:57.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:57.672 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:13:57.773 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:13:57.797 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:13:57.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:57.799 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:13:57.900 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:13:57.909 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:13:57 smithi146 bash[23809]: audit 2024-03-11T21:13:56.012718+0000 mgr.y (mgr.24380) 856 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:13:57.910 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:13:57 smithi146 bash[23809]: cluster 2024-03-11T21:13:56.410068+0000 mgr.y (mgr.24380) 857 : cluster 0 pgmap v985: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfilling, 153 active+clean; 4.9 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 868 op/s; 419/243045 objects degraded (0.172%); 11982/243045 objects misplaced (4.930%); 35 MiB/s, 562 objects/s recovering 2024-03-11T21:13:57.922 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:13:57.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:57.925 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:13:58.026 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:13:58.053 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:13:58.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:58.059 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:13:58.077 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:13:57 smithi023 bash[21372]: audit 2024-03-11T21:13:56.012718+0000 mgr.y (mgr.24380) 856 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:13:58.078 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:13:57 smithi023 bash[21372]: cluster 2024-03-11T21:13:56.410068+0000 mgr.y (mgr.24380) 857 : cluster 0 pgmap v985: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfilling, 153 active+clean; 4.9 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 868 op/s; 419/243045 objects degraded (0.172%); 11982/243045 objects misplaced (4.930%); 35 MiB/s, 562 objects/s recovering 2024-03-11T21:13:58.160 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:13:58.185 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:13:58.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:58.187 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:13:58.288 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:13:58.318 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:13:58.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:58.321 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:13:58.338 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:13:58 smithi023 bash[29146]: audit 2024-03-11T21:13:56.012718+0000 mgr.y (mgr.24380) 856 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:13:58.338 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:13:58 smithi023 bash[29146]: cluster 2024-03-11T21:13:56.410068+0000 mgr.y (mgr.24380) 857 : cluster 0 pgmap v985: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfilling, 153 active+clean; 4.9 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 868 op/s; 419/243045 objects degraded (0.172%); 11982/243045 objects misplaced (4.930%); 35 MiB/s, 562 objects/s recovering 2024-03-11T21:13:58.422 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:58.448 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:58.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:58.452 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:13:58.553 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:13:58.587 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:13:58.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:58.592 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:13:58.692 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:13:58.715 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:13:58.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:58.718 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:13:58.801 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:13:58 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:13:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:13:58.802 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:13:58 smithi023 bash[29146]: cluster 2024-03-11T21:13:58.303851+0000 osd.5 (osd.5) 493 : cluster 0 11.7 scrub starts 2024-03-11T21:13:58.819 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:13:58.845 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:13:58.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:58.849 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:13:58.950 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:13:58.959 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:13:58 smithi146 bash[23809]: cluster 2024-03-11T21:13:58.303851+0000 osd.5 (osd.5) 493 : cluster 0 11.7 scrub starts 2024-03-11T21:13:58.971 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:13:58.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:58.973 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:13:59.074 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:13:59.094 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:13:59.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:59.095 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:13:59.196 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:13:59.206 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:13:58 smithi023 bash[21372]: cluster 2024-03-11T21:13:58.303851+0000 osd.5 (osd.5) 493 : cluster 0 11.7 scrub starts 2024-03-11T21:13:59.219 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:13:59.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:59.221 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:13:59.322 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:59.343 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:59.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:59.345 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:13:59.446 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:13:59.471 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:13:59.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:59.473 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:13:59.574 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:59.596 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:13:59.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:59.598 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:13:59.642 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-03-11T21:13:59.699 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:59.733 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:13:59.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:59.737 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:13:59.776 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:13:59 smithi023 bash[21372]: cluster 2024-03-11T21:13:58.411152+0000 mgr.y (mgr.24380) 858 : cluster 0 pgmap v986: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfilling, 153 active+clean; 4.9 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 911 op/s; 594/239769 objects degraded (0.248%); 11707/239769 objects misplaced (4.883%); 33 MiB/s, 521 objects/s recovering 2024-03-11T21:13:59.776 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:13:59 smithi023 bash[21372]: cluster 2024-03-11T21:13:58.689587+0000 mon.a (mon.0) 1884 : cluster 0 osdmap e413: 8 total, 8 up, 7 in 2024-03-11T21:13:59.776 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:13:59 smithi023 bash[21372]: cluster 2024-03-11T21:13:59.168938+0000 osd.5 (osd.5) 494 : cluster 0 11.7 scrub ok 2024-03-11T21:13:59.776 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:13:59 smithi023 bash[21372]: cluster 2024-03-11T21:13:59.490873+0000 osd.3 (osd.3) 471 : cluster 0 purged_snaps scrub starts 2024-03-11T21:13:59.776 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:13:59 smithi023 bash[21372]: cluster 2024-03-11T21:13:59.490936+0000 osd.3 (osd.3) 472 : cluster 0 purged_snaps scrub ok 2024-03-11T21:13:59.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:13:59 smithi023 bash[29146]: cluster 2024-03-11T21:13:58.411152+0000 mgr.y (mgr.24380) 858 : cluster 0 pgmap v986: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfilling, 153 active+clean; 4.9 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 911 op/s; 594/239769 objects degraded (0.248%); 11707/239769 objects misplaced (4.883%); 33 MiB/s, 521 objects/s recovering 2024-03-11T21:13:59.777 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:13:59 smithi023 bash[29146]: cluster 2024-03-11T21:13:58.689587+0000 mon.a (mon.0) 1884 : cluster 0 osdmap e413: 8 total, 8 up, 7 in 2024-03-11T21:13:59.777 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:13:59 smithi023 bash[29146]: cluster 2024-03-11T21:13:59.168938+0000 osd.5 (osd.5) 494 : cluster 0 11.7 scrub ok 2024-03-11T21:13:59.777 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:13:59 smithi023 bash[29146]: cluster 2024-03-11T21:13:59.490873+0000 osd.3 (osd.3) 471 : cluster 0 purged_snaps scrub starts 2024-03-11T21:13:59.777 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:13:59 smithi023 bash[29146]: cluster 2024-03-11T21:13:59.490936+0000 osd.3 (osd.3) 472 : cluster 0 purged_snaps scrub ok 2024-03-11T21:13:59.837 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:13:59.879 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:13:59.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:13:59.882 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:13:59.983 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:13:59.991 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:13:59 smithi146 bash[23809]: cluster 2024-03-11T21:13:58.411152+0000 mgr.y (mgr.24380) 858 : cluster 0 pgmap v986: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfilling, 153 active+clean; 4.9 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 911 op/s; 594/239769 objects degraded (0.248%); 11707/239769 objects misplaced (4.883%); 33 MiB/s, 521 objects/s recovering 2024-03-11T21:13:59.991 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:13:59 smithi146 bash[23809]: cluster 2024-03-11T21:13:58.689587+0000 mon.a (mon.0) 1884 : cluster 0 osdmap e413: 8 total, 8 up, 7 in 2024-03-11T21:13:59.991 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:13:59 smithi146 bash[23809]: cluster 2024-03-11T21:13:59.168938+0000 osd.5 (osd.5) 494 : cluster 0 11.7 scrub ok 2024-03-11T21:13:59.991 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:13:59 smithi146 bash[23809]: cluster 2024-03-11T21:13:59.490873+0000 osd.3 (osd.3) 471 : cluster 0 purged_snaps scrub starts 2024-03-11T21:13:59.991 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:13:59 smithi146 bash[23809]: cluster 2024-03-11T21:13:59.490936+0000 osd.3 (osd.3) 472 : cluster 0 purged_snaps scrub ok 2024-03-11T21:14:00.009 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:14:00.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:00.014 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:14:00.115 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:00.138 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:00.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:00.139 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:14:00.240 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:14:00.265 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:14:00.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:00.269 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:14:00.370 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:00.396 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:00.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:00.400 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:14:00.502 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:14:00.534 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:14:00.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:00.538 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:14:00.639 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:00.667 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:00.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:00.670 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:14:00.771 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:00.800 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:00.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:00.802 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:14:00.903 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:00.942 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:00.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:00.947 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:14:01.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:00 smithi146 bash[23809]: cluster 2024-03-11T21:13:59.691601+0000 mon.a (mon.0) 1885 : cluster 0 osdmap e414: 8 total, 8 up, 7 in 2024-03-11T21:14:01.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:00 smithi146 bash[23809]: cluster 2024-03-11T21:14:00.270733+0000 osd.5 (osd.5) 495 : cluster 0 1.1 scrub starts 2024-03-11T21:14:01.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:00 smithi146 bash[23809]: cluster 2024-03-11T21:14:00.274122+0000 osd.5 (osd.5) 496 : cluster 0 1.1 scrub ok 2024-03-11T21:14:01.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:00 smithi146 bash[23809]: cluster 2024-03-11T21:14:00.383388+0000 mon.a (mon.0) 1886 : cluster 3 Health check update: Degraded data redundancy: 594/239769 objects degraded (0.248%), 1 pg degraded (PG_DEGRADED) 2024-03-11T21:14:01.026 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:00 smithi023 bash[21372]: cluster 2024-03-11T21:13:59.691601+0000 mon.a (mon.0) 1885 : cluster 0 osdmap e414: 8 total, 8 up, 7 in 2024-03-11T21:14:01.026 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:00 smithi023 bash[21372]: cluster 2024-03-11T21:14:00.270733+0000 osd.5 (osd.5) 495 : cluster 0 1.1 scrub starts 2024-03-11T21:14:01.026 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:00 smithi023 bash[21372]: cluster 2024-03-11T21:14:00.274122+0000 osd.5 (osd.5) 496 : cluster 0 1.1 scrub ok 2024-03-11T21:14:01.026 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:00 smithi023 bash[21372]: cluster 2024-03-11T21:14:00.383388+0000 mon.a (mon.0) 1886 : cluster 3 Health check update: Degraded data redundancy: 594/239769 objects degraded (0.248%), 1 pg degraded (PG_DEGRADED) 2024-03-11T21:14:01.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:00 smithi023 bash[29146]: cluster 2024-03-11T21:13:59.691601+0000 mon.a (mon.0) 1885 : cluster 0 osdmap e414: 8 total, 8 up, 7 in 2024-03-11T21:14:01.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:00 smithi023 bash[29146]: cluster 2024-03-11T21:14:00.270733+0000 osd.5 (osd.5) 495 : cluster 0 1.1 scrub starts 2024-03-11T21:14:01.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:00 smithi023 bash[29146]: cluster 2024-03-11T21:14:00.274122+0000 osd.5 (osd.5) 496 : cluster 0 1.1 scrub ok 2024-03-11T21:14:01.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:00 smithi023 bash[29146]: cluster 2024-03-11T21:14:00.383388+0000 mon.a (mon.0) 1886 : cluster 3 Health check update: Degraded data redundancy: 594/239769 objects degraded (0.248%), 1 pg degraded (PG_DEGRADED) 2024-03-11T21:14:01.048 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:14:01.068 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:14:01.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:01.071 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:14:01.172 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:01.211 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:01.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:01.212 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:14:01.313 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:01.342 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:01.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:01.345 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:14:01.446 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:01.468 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:01.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:01.469 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:14:01.570 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:14:01.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:01.611 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:14:01.611 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:14:01.712 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:01.743 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:01.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:01.749 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:14:01.850 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:14:01.878 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:14:01.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:01.881 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:14:01.981 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:01.990 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:01 smithi146 bash[23809]: cluster 2024-03-11T21:14:00.411908+0000 mgr.y (mgr.24380) 859 : cluster 0 pgmap v989: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfilling, 154 active+clean; 4.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 883 op/s; 594/234396 objects degraded (0.253%); 9265/234396 objects misplaced (3.953%); 41 MiB/s, 658 objects/s recovering 2024-03-11T21:14:01.990 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:01 smithi146 bash[23809]: cluster 2024-03-11T21:14:01.255030+0000 osd.5 (osd.5) 497 : cluster 0 4.d scrub starts 2024-03-11T21:14:01.991 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:01 smithi146 bash[23809]: cluster 2024-03-11T21:14:01.256601+0000 osd.5 (osd.5) 498 : cluster 0 4.d scrub ok 2024-03-11T21:14:02.006 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:02.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:02.008 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:14:02.109 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:14:02.137 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:14:02.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:02.141 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:14:02.242 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:14:02.251 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:01 smithi023 bash[21372]: cluster 2024-03-11T21:14:00.411908+0000 mgr.y (mgr.24380) 859 : cluster 0 pgmap v989: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfilling, 154 active+clean; 4.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 883 op/s; 594/234396 objects degraded (0.253%); 9265/234396 objects misplaced (3.953%); 41 MiB/s, 658 objects/s recovering 2024-03-11T21:14:02.251 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:01 smithi023 bash[21372]: cluster 2024-03-11T21:14:01.255030+0000 osd.5 (osd.5) 497 : cluster 0 4.d scrub starts 2024-03-11T21:14:02.251 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:01 smithi023 bash[21372]: cluster 2024-03-11T21:14:01.256601+0000 osd.5 (osd.5) 498 : cluster 0 4.d scrub ok 2024-03-11T21:14:02.252 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:01 smithi023 bash[29146]: cluster 2024-03-11T21:14:00.411908+0000 mgr.y (mgr.24380) 859 : cluster 0 pgmap v989: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfilling, 154 active+clean; 4.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 883 op/s; 594/234396 objects degraded (0.253%); 9265/234396 objects misplaced (3.953%); 41 MiB/s, 658 objects/s recovering 2024-03-11T21:14:02.252 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:01 smithi023 bash[29146]: cluster 2024-03-11T21:14:01.255030+0000 osd.5 (osd.5) 497 : cluster 0 4.d scrub starts 2024-03-11T21:14:02.252 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:01 smithi023 bash[29146]: cluster 2024-03-11T21:14:01.256601+0000 osd.5 (osd.5) 498 : cluster 0 4.d scrub ok 2024-03-11T21:14:02.271 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:14:02.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:02.274 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:14:02.375 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:14:02.417 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:14:02.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:02.422 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:14:02.523 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:02.550 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:02.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:02.554 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:14:02.655 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:14:02.681 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:14:02.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:02.690 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:14:02.791 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:02.810 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:02 smithi023 bash[21372]: cluster 2024-03-11T21:14:02.113715+0000 osd.6 (osd.6) 403 : cluster 0 1.7 scrub starts 2024-03-11T21:14:02.810 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:02 smithi023 bash[21372]: cluster 2024-03-11T21:14:02.114858+0000 osd.6 (osd.6) 404 : cluster 0 1.7 scrub ok 2024-03-11T21:14:02.810 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:02 smithi023 bash[21372]: cluster 2024-03-11T21:14:02.225121+0000 osd.5 (osd.5) 499 : cluster 0 6.7 scrub starts 2024-03-11T21:14:02.810 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:02 smithi023 bash[21372]: cluster 2024-03-11T21:14:02.373704+0000 osd.5 (osd.5) 500 : cluster 0 6.7 scrub ok 2024-03-11T21:14:02.810 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:02 smithi023 bash[29146]: cluster 2024-03-11T21:14:02.113715+0000 osd.6 (osd.6) 403 : cluster 0 1.7 scrub starts 2024-03-11T21:14:02.810 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:02 smithi023 bash[29146]: cluster 2024-03-11T21:14:02.114858+0000 osd.6 (osd.6) 404 : cluster 0 1.7 scrub ok 2024-03-11T21:14:02.811 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:02 smithi023 bash[29146]: cluster 2024-03-11T21:14:02.225121+0000 osd.5 (osd.5) 499 : cluster 0 6.7 scrub starts 2024-03-11T21:14:02.811 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:02 smithi023 bash[29146]: cluster 2024-03-11T21:14:02.373704+0000 osd.5 (osd.5) 500 : cluster 0 6.7 scrub ok 2024-03-11T21:14:02.831 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:02.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:02.838 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:14:02.907 INFO:teuthology.orchestra.run.smithi023.stdout:{"pg_ready":true,"pg_map":{"version":982,"stamp":"2024-03-11T21:13:52.408081+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5566962112,"num_objects":85135,"num_object_clones":0,"num_object_copies":255405,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":269,"num_objects_misplaced":14898,"num_objects_unfound":0,"num_objects_dirty":85135,"num_whiteouts":0,"num_read":2830,"num_read_kb":2582,"num_write":111355,"num_write_kb":6249070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12774,"num_bytes_recovered":820910254,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71725,"ondisk_log_size":71725,"up":474,"acting":473,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":18920356,"kb_used_data":15373536,"kb_used_omap":20,"kb_used_meta":3546667,"kb_avail":637295708,"statfs":{"total":671965249536,"available":652590804992,"internally_reserved":0,"allocated":15742500864,"data_stored":15740011965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20494,"internal_metadata":3631788018},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1,5,2,4,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-548339712,"num_objects":-8367,"num_object_clones":0,"num_object_copies":-25101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":269,"num_objects_misplaced":-5869,"num_objects_unfound":0,"num_objects_dirty":-8367,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":8367,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4473,"num_bytes_recovered":293142528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.010586"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":739,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:49.594854+0000","last_change":"2024-03-11T21:11:15.221894+0000","last_active":"2024-03-11T21:13:49.594854+0000","last_peered":"2024-03-11T21:13:49.594854+0000","last_clean":"2024-03-11T21:13:49.594854+0000","last_became_active":"2024-03-11T20:50:34.338390+0000","last_became_peered":"2024-03-11T20:50:34.338390+0000","last_unstale":"2024-03-11T21:13:49.594854+0000","last_undegraded":"2024-03-11T21:13:49.594854+0000","last_fullsized":"2024-03-11T21:13:49.594854+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:11:15.221783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:11:15.221783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":127,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":749,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:49.598521+0000","last_change":"2024-03-11T21:13:22.516577+0000","last_active":"2024-03-11T21:13:49.598521+0000","last_peered":"2024-03-11T21:13:49.598521+0000","last_clean":"2024-03-11T21:13:49.598521+0000","last_became_active":"2024-03-11T21:13:22.516099+0000","last_became_peered":"2024-03-11T21:13:22.516099+0000","last_unstale":"2024-03-11T21:13:49.598521+0000","last_undegraded":"2024-03-11T21:13:49.598521+0000","last_fullsized":"2024-03-11T21:13:49.598521+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:11:54.932679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:11:54.932679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":344,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:49.591654+0000","last_change":"2024-03-11T21:13:22.518484+0000","last_active":"2024-03-11T21:13:49.591654+0000","last_peered":"2024-03-11T21:13:49.591654+0000","last_clean":"2024-03-11T21:13:49.591654+0000","last_became_active":"2024-03-11T21:13:22.518057+0000","last_became_peered":"2024-03-11T21:13:22.518057+0000","last_unstale":"2024-03-11T21:13:49.591654+0000","last_undegraded":"2024-03-11T21:13:49.591654+0000","last_fullsized":"2024-03-11T21:13:49.591654+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:12:26.013231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:11:05.884900+0000","last_clean_scrub_stamp":"2024-03-11T21:12:26.013231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1e","version":"358'30","reported_seq":795,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:49.592501+0000","last_change":"2024-03-11T21:10:52.029948+0000","last_active":"2024-03-11T21:13:49.592501+0000","last_peered":"2024-03-11T21:13:49.592501+0000","last_clean":"2024-03-11T21:13:49.592501+0000","last_became_active":"2024-03-11T20:50:30.101721+0000","last_became_peered":"2024-03-11T20:50:30.101721+0000","last_unstale":"2024-03-11T21:13:49.592501+0000","last_undegraded":"2024-03-11T21:13:49.592501+0000","last_fullsized":"2024-03-11T21:13:49.592501+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'30","last_scrub_stamp":"2024-03-11T21:10:52.029827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:10:52.029827+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":588,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:49.593068+0000","last_change":"2024-03-11T21:12:23.401320+0000","last_active":"2024-03-11T21:13:49.593068+0000","last_peered":"2024-03-11T21:13:49.593068+0000","last_clean":"2024-03-11T21:13:49.593068+0000","last_became_active":"2024-03-11T21:05:17.316714+0000","last_became_peered":"2024-03-11T21:05:17.316714+0000","last_unstale":"2024-03-11T21:13:49.593068+0000","last_undegraded":"2024-03-11T21:13:49.593068+0000","last_fullsized":"2024-03-11T21:13:49.593068+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:12:23.401235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:55.545758+0000","last_clean_scrub_stamp":"2024-03-11T21:12:23.401235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":584,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:49.584661+0000","last_change":"2024-03-11T21:12:08.997233+0000","last_active":"2024-03-11T21:13:49.584661+0000","last_peered":"2024-03-11T21:13:49.584661+0000","last_clean":"2024-03-11T21:13:49.584661+0000","last_became_active":"2024-03-11T21:05:17.319272+0000","last_became_peered":"2024-03-11T21:05:17.319272+0000","last_unstale":"2024-03-11T21:13:49.584661+0000","last_undegraded":"2024-03-11T21:13:49.584661+0000","last_fullsized":"2024-03-11T21:13:49.584661+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:12:08.997113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:54:13.660325+0000","last_clean_scrub_stamp":"2024-03-11T21:12:08.997113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":751,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:49.592232+0000","last_change":"2024-03-11T21:13:22.521682+0000","last_active":"2024-03-11T21:13:49.592232+0000","last_peered":"2024-03-11T21:13:49.592232+0000","last_clean":"2024-03-11T21:13:49.592232+0000","last_became_active":"2024-03-11T21:13:22.521379+0000","last_became_peered":"2024-03-11T21:13:22.521379+0000","last_unstale":"2024-03-11T21:13:49.592232+0000","last_undegraded":"2024-03-11T21:13:49.592232+0000","last_fullsized":"2024-03-11T21:13:49.592232+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:49.933191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:09:49.933191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"358'27","reported_seq":733,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:49.595565+0000","last_change":"2024-03-11T21:13:22.532269+0000","last_active":"2024-03-11T21:13:49.595565+0000","last_peered":"2024-03-11T21:13:49.595565+0000","last_clean":"2024-03-11T21:13:49.595565+0000","last_became_active":"2024-03-11T21:13:22.516346+0000","last_became_peered":"2024-03-11T21:13:22.516346+0000","last_unstale":"2024-03-11T21:13:49.595565+0000","last_undegraded":"2024-03-11T21:13:49.595565+0000","last_fullsized":"2024-03-11T21:13:49.595565+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'27","last_scrub_stamp":"2024-03-11T21:12:23.022901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:12:23.022901+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":776,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:49.593162+0000","last_change":"2024-03-11T21:12:45.484750+0000","last_active":"2024-03-11T21:13:49.593162+0000","last_peered":"2024-03-11T21:13:49.593162+0000","last_clean":"2024-03-11T21:13:49.593162+0000","last_became_active":"2024-03-11T21:00:48.460694+0000","last_became_peered":"2024-03-11T21:00:48.460694+0000","last_unstale":"2024-03-11T21:13:49.593162+0000","last_undegraded":"2024-03-11T21:13:49.593162+0000","last_fullsized":"2024-03-11T21:13:49.593162+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:12:45.484649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:12:45.484649+0000","last_clean_scrub_stamp":"2024-03-11T21:12:45.484649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:13:50.487913+0000","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":568,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:49.595042+0000","last_change":"2024-03-11T21:12:30.037865+0000","last_active":"2024-03-11T21:13:49.595042+0000","last_peered":"2024-03-11T21:13:49.595042+0000","last_clean":"2024-03-11T21:13:49.595042+0000","last_became_active":"2024-03-11T21:05:17.316491+0000","last_became_peered":"2024-03-11T21:05:17.316491+0000","last_unstale":"2024-03-11T21:13:49.595042+0000","last_undegraded":"2024-03-11T21:13:49.595042+0000","last_fullsized":"2024-03-11T21:13:49.595042+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:12:30.037733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:03:36.230647+0000","last_clean_scrub_stamp":"2024-03-11T21:12:30.037733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":779,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:49.591396+0000","last_change":"2024-03-11T21:13:22.523411+0000","last_active":"2024-03-11T21:13:49.591396+0000","last_peered":"2024-03-11T21:13:49.591396+0000","last_clean":"2024-03-11T21:13:49.591396+0000","last_became_active":"2024-03-11T21:13:22.522924+0000","last_became_peered":"2024-03-11T21:13:22.522924+0000","last_unstale":"2024-03-11T21:13:49.591396+0000","last_undegraded":"2024-03-11T21:13:49.591396+0000","last_fullsized":"2024-03-11T21:13:49.591396+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:12:12.128410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:12:12.128410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"358'39","reported_seq":889,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:49.593705+0000","last_change":"2024-03-11T21:12:02.373289+0000","last_active":"2024-03-11T21:13:49.593705+0000","last_peered":"2024-03-11T21:13:49.593705+0000","last_clean":"2024-03-11T21:13:49.593705+0000","last_became_active":"2024-03-11T21:00:48.462943+0000","last_became_peered":"2024-03-11T21:00:48.462943+0000","last_unstale":"2024-03-11T21:13:49.593705+0000","last_undegraded":"2024-03-11T21:13:49.593705+0000","last_fullsized":"2024-03-11T21:13:49.593705+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'39","last_scrub_stamp":"2024-03-11T21:12:02.373124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:12:02.373124+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":232,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-11T21:13:40.494309+0000","last_change":"2024-03-11T21:13:22.503894+0000","last_active":"2024-03-11T21:13:40.494309+0000","last_peered":"2024-03-11T21:13:40.494309+0000","last_clean":"2024-03-11T21:13:40.494309+0000","last_became_active":"2024-03-11T21:13:22.503488+0000","last_became_peered":"2024-03-11T21:13:22.503488+0000","last_unstale":"2024-03-11T21:13:40.494309+0000","last_undegraded":"2024-03-11T21:13:40.494309+0000","last_fullsized":"2024-03-11T21:13:40.494309+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:11:56.158450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:11:56.158450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":536,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:49.592495+0000","last_change":"2024-03-11T21:13:22.513782+0000","last_active":"2024-03-11T21:13:49.592495+0000","last_peered":"2024-03-11T21:13:49.592495+0000","last_clean":"2024-03-11T21:13:49.592495+0000","last_became_active":"2024-03-11T21:13:22.513191+0000","last_became_peered":"2024-03-11T21:13:22.513191+0000","last_unstale":"2024-03-11T21:13:49.592495+0000","last_undegraded":"2024-03-11T21:13:49.592495+0000","last_fullsized":"2024-03-11T21:13:49.592495+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:12:01.877128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:53:50.599029+0000","last_clean_scrub_stamp":"2024-03-11T21:12:01.877128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":781,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:49.590804+0000","last_change":"2024-03-11T21:12:09.118041+0000","last_active":"2024-03-11T21:13:49.590804+0000","last_peered":"2024-03-11T21:13:49.590804+0000","last_clean":"2024-03-11T21:13:49.590804+0000","last_became_active":"2024-03-11T20:50:28.068978+0000","last_became_peered":"2024-03-11T20:50:28.068978+0000","last_unstale":"2024-03-11T21:13:49.590804+0000","last_undegraded":"2024-03-11T21:13:49.590804+0000","last_fullsized":"2024-03-11T21:13:49.590804+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:12:09.117895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:54:07.155130+0000","last_clean_scrub_stamp":"2024-03-11T21:12:09.117895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"358'29","reported_seq":818,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:49.594676+0000","last_change":"2024-03-11T21:12:21.398653+0000","last_active":"2024-03-11T21:13:49.594676+0000","last_peered":"2024-03-11T21:13:49.594676+0000","last_clean":"2024-03-11T21:13:49.594676+0000","last_became_active":"2024-03-11T21:00:27.952093+0000","last_became_peered":"2024-03-11T21:00:27.952093+0000","last_unstale":"2024-03-11T21:13:49.594676+0000","last_undegraded":"2024-03-11T21:13:49.594676+0000","last_fullsized":"2024-03-11T21:13:49.594676+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'29","last_scrub_stamp":"2024-03-11T21:12:21.398527+0000","last_deep_scrub":"110'17","last_deep_scrub_stamp":"2024-03-11T20:53:04.802488+0000","last_clean_scrub_stamp":"2024-03-11T21:12:21.398527+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":741,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:49.593141+0000","last_change":"2024-03-11T21:12:01.082740+0000","last_active":"2024-03-11T21:13:49.593141+0000","last_peered":"2024-03-11T21:13:49.593141+0000","last_clean":"2024-03-11T21:13:49.593141+0000","last_became_active":"2024-03-11T20:50:34.334558+0000","last_became_peered":"2024-03-11T20:50:34.334558+0000","last_unstale":"2024-03-11T21:13:49.593141+0000","last_undegraded":"2024-03-11T21:13:49.593141+0000","last_fullsized":"2024-03-11T21:13:49.593141+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:12:01.082623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:55:59.892394+0000","last_clean_scrub_stamp":"2024-03-11T21:12:01.082623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":183,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":126,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:49.591571+0000","last_change":"2024-03-11T21:12:31.955513+0000","last_active":"2024-03-11T21:13:49.591571+0000","last_peered":"2024-03-11T21:13:49.591571+0000","last_clean":"2024-03-11T21:13:49.591571+0000","last_became_active":"2024-03-11T21:10:38.441649+0000","last_became_peered":"2024-03-11T21:10:38.441649+0000","last_unstale":"2024-03-11T21:13:49.591571+0000","last_undegraded":"2024-03-11T21:13:49.591571+0000","last_fullsized":"2024-03-11T21:13:49.591571+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:12:31.955357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:12:31.955357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"77'1","reported_seq":759,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:49.595121+0000","last_change":"2024-03-11T21:11:05.190655+0000","last_active":"2024-03-11T21:13:49.595121+0000","last_peered":"2024-03-11T21:13:49.595121+0000","last_clean":"2024-03-11T21:13:49.595121+0000","last_became_active":"2024-03-11T20:50:28.068840+0000","last_became_peered":"2024-03-11T20:50:28.068840+0000","last_unstale":"2024-03-11T21:13:49.595121+0000","last_undegraded":"2024-03-11T21:13:49.595121+0000","last_fullsized":"2024-03-11T21:13:49.595121+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-03-11T21:11:05.190559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:11:05.190559+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"357'25","reported_seq":633,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-11T21:13:40.494473+0000","last_change":"2024-03-11T21:12:27.010140+0000","last_active":"2024-03-11T21:13:40.494473+0000","last_peered":"2024-03-11T21:13:40.494473+0000","last_clean":"2024-03-11T21:13:40.494473+0000","last_became_active":"2024-03-11T21:05:17.443030+0000","last_became_peered":"2024-03-11T21:05:17.443030+0000","last_unstale":"2024-03-11T21:13:40.494473+0000","last_undegraded":"2024-03-11T21:13:40.494473+0000","last_fullsized":"2024-03-11T21:13:40.494473+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"357'25","last_scrub_stamp":"2024-03-11T21:12:27.010006+0000","last_deep_scrub":"207'19","last_deep_scrub_stamp":"2024-03-11T21:03:39.240089+0000","last_clean_scrub_stamp":"2024-03-11T21:12:27.010006+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:13:48.019725+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":541,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:49.586021+0000","last_change":"2024-03-11T21:12:09.993519+0000","last_active":"2024-03-11T21:13:49.586021+0000","last_peered":"2024-03-11T21:13:49.586021+0000","last_clean":"2024-03-11T21:13:49.586021+0000","last_became_active":"2024-03-11T21:05:17.313986+0000","last_became_peered":"2024-03-11T21:05:17.313986+0000","last_unstale":"2024-03-11T21:13:49.586021+0000","last_undegraded":"2024-03-11T21:13:49.586021+0000","last_fullsized":"2024-03-11T21:13:49.586021+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:12:09.993400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:33.399510+0000","last_clean_scrub_stamp":"2024-03-11T21:12:09.993400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":557,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-11T21:13:40.494953+0000","last_change":"2024-03-11T21:12:32.862629+0000","last_active":"2024-03-11T21:13:40.494953+0000","last_peered":"2024-03-11T21:13:40.494953+0000","last_clean":"2024-03-11T21:13:40.494953+0000","last_became_active":"2024-03-11T21:08:05.298552+0000","last_became_peered":"2024-03-11T21:08:05.298552+0000","last_unstale":"2024-03-11T21:13:40.494953+0000","last_undegraded":"2024-03-11T21:13:40.494953+0000","last_fullsized":"2024-03-11T21:13:40.494953+0000","mapping_epoch":319,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:12:32.862538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:12:32.862538+0000","last_clean_scrub_stamp":"2024-03-11T21:12:32.862538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":782,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:49.585021+0000","last_change":"2024-03-11T21:12:06.009990+0000","last_active":"2024-03-11T21:13:49.585021+0000","last_peered":"2024-03-11T21:13:49.585021+0000","last_clean":"2024-03-11T21:13:49.585021+0000","last_became_active":"2024-03-11T21:00:27.956935+0000","last_became_peered":"2024-03-11T21:00:27.956935+0000","last_unstale":"2024-03-11T21:13:49.585021+0000","last_undegraded":"2024-03-11T21:13:49.585021+0000","last_fullsized":"2024-03-11T21:13:49.585021+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:12:06.009691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:12:06.009691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"357'9","reported_seq":698,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:49.591227+0000","last_change":"2024-03-11T21:13:22.560163+0000","last_active":"2024-03-11T21:13:49.591227+0000","last_peered":"2024-03-11T21:13:49.591227+0000","last_clean":"2024-03-11T21:13:49.591227+0000","last_became_active":"2024-03-11T21:13:22.517846+0000","last_became_peered":"2024-03-11T21:13:22.517846+0000","last_unstale":"2024-03-11T21:13:49.591227+0000","last_undegraded":"2024-03-11T21:13:49.591227+0000","last_fullsized":"2024-03-11T21:13:49.591227+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"357'9","last_scrub_stamp":"2024-03-11T21:12:13.057926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:12:13.057926+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":16,"num_read_kb":12,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":472,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-11T21:13:40.494692+0000","last_change":"2024-03-11T21:11:20.728085+0000","last_active":"2024-03-11T21:13:40.494692+0000","last_peered":"2024-03-11T21:13:40.494692+0000","last_clean":"2024-03-11T21:13:40.494692+0000","last_became_active":"2024-03-11T21:00:48.464694+0000","last_became_peered":"2024-03-11T21:00:48.464694+0000","last_unstale":"2024-03-11T21:13:40.494692+0000","last_undegraded":"2024-03-11T21:13:40.494692+0000","last_fullsized":"2024-03-11T21:13:40.494692+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:11:20.727935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:11:20.727935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1f","version":"212'16","reported_seq":1090,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:49.586474+0000","last_change":"2024-03-11T21:12:02.044146+0000","last_active":"2024-03-11T21:13:49.586474+0000","last_peered":"2024-03-11T21:13:49.586474+0000","last_clean":"2024-03-11T21:13:49.586474+0000","last_became_active":"2024-03-11T21:00:48.464350+0000","last_became_peered":"2024-03-11T21:00:48.464350+0000","last_unstale":"2024-03-11T21:13:49.586474+0000","last_undegraded":"2024-03-11T21:13:49.586474+0000","last_fullsized":"2024-03-11T21:13:49.586474+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"212'16","last_scrub_stamp":"2024-03-11T21:12:02.044031+0000","last_deep_scrub":"212'16","last_deep_scrub_stamp":"2024-03-11T21:06:47.821221+0000","last_clean_scrub_stamp":"2024-03-11T21:12:02.044031+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"362'2","reported_seq":755,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:49.598675+0000","last_change":"2024-03-11T21:13:22.561576+0000","last_active":"2024-03-11T21:13:49.598675+0000","last_peered":"2024-03-11T21:13:49.598675+0000","last_clean":"2024-03-11T21:13:49.598675+0000","last_became_active":"2024-03-11T21:13:22.514643+0000","last_became_peered":"2024-03-11T21:13:22.514643+0000","last_unstale":"2024-03-11T21:13:49.598675+0000","last_undegraded":"2024-03-11T21:13:49.598675+0000","last_fullsized":"2024-03-11T21:13:49.598675+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"362'2","last_scrub_stamp":"2024-03-11T21:11:03.482685+0000","last_deep_scrub":"362'2","last_deep_scrub_stamp":"2024-03-11T21:11:03.482685+0000","last_clean_scrub_stamp":"2024-03-11T21:11:03.482685+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":335,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"357'21","reported_seq":839,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:49.590672+0000","last_change":"2024-03-11T21:13:22.778375+0000","last_active":"2024-03-11T21:13:49.590672+0000","last_peered":"2024-03-11T21:13:49.590672+0000","last_clean":"2024-03-11T21:13:49.590672+0000","last_became_active":"2024-03-11T21:13:22.513831+0000","last_became_peered":"2024-03-11T21:13:22.513831+0000","last_unstale":"2024-03-11T21:13:49.590672+0000","last_undegraded":"2024-03-11T21:13:49.590672+0000","last_fullsized":"2024-03-11T21:13:49.590672+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"357'21","last_scrub_stamp":"2024-03-11T21:12:16.917505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:12:16.917505+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"105'2","reported_seq":764,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:49.590990+0000","last_change":"2024-03-11T21:11:01.833581+0000","last_active":"2024-03-11T21:13:49.590990+0000","last_peered":"2024-03-11T21:13:49.590990+0000","last_clean":"2024-03-11T21:13:49.590990+0000","last_became_active":"2024-03-11T20:50:34.327356+0000","last_became_peered":"2024-03-11T20:50:34.327356+0000","last_unstale":"2024-03-11T21:13:49.590990+0000","last_undegraded":"2024-03-11T21:13:49.590990+0000","last_fullsized":"2024-03-11T21:13:49.590990+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'2","last_scrub_stamp":"2024-03-11T21:11:01.833452+0000","last_deep_scrub":"105'2","last_deep_scrub_stamp":"2024-03-11T21:03:41.971918+0000","last_clean_scrub_stamp":"2024-03-11T21:11:01.833452+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":343,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:49.591781+0000","last_change":"2024-03-11T21:13:22.529303+0000","last_active":"2024-03-11T21:13:49.591781+0000","last_peered":"2024-03-11T21:13:49.591781+0000","last_clean":"2024-03-11T21:13:49.591781+0000","last_became_active":"2024-03-11T21:13:22.528954+0000","last_became_peered":"2024-03-11T21:13:22.528954+0000","last_unstale":"2024-03-11T21:13:49.591781+0000","last_undegraded":"2024-03-11T21:13:49.591781+0000","last_fullsized":"2024-03-11T21:13:49.591781+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:12:07.174875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:06:32.968650+0000","last_clean_scrub_stamp":"2024-03-11T21:12:07.174875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":776,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:49.593047+0000","last_change":"2024-03-11T21:12:15.313176+0000","last_active":"2024-03-11T21:13:49.593047+0000","last_peered":"2024-03-11T21:13:49.593047+0000","last_clean":"2024-03-11T21:13:49.593047+0000","last_became_active":"2024-03-11T21:00:27.947614+0000","last_became_peered":"2024-03-11T21:00:27.947614+0000","last_unstale":"2024-03-11T21:13:49.593047+0000","last_undegraded":"2024-03-11T21:13:49.593047+0000","last_fullsized":"2024-03-11T21:13:49.593047+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:12:15.313027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:08:27.756036+0000","last_clean_scrub_stamp":"2024-03-11T21:12:15.313027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"358'39","reported_seq":528,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-11T21:13:40.494528+0000","last_change":"2024-03-11T21:12:41.940357+0000","last_active":"2024-03-11T21:13:40.494528+0000","last_peered":"2024-03-11T21:13:40.494528+0000","last_clean":"2024-03-11T21:13:40.494528+0000","last_became_active":"2024-03-11T21:00:48.457893+0000","last_became_peered":"2024-03-11T21:00:48.457893+0000","last_unstale":"2024-03-11T21:13:40.494528+0000","last_undegraded":"2024-03-11T21:13:40.494528+0000","last_fullsized":"2024-03-11T21:13:40.494528+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'39","last_scrub_stamp":"2024-03-11T21:12:41.940134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:12:41.940134+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:14:10.232072+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":745,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:49.595796+0000","last_change":"2024-03-11T21:13:22.514763+0000","last_active":"2024-03-11T21:13:49.595796+0000","last_peered":"2024-03-11T21:13:49.595796+0000","last_clean":"2024-03-11T21:13:49.595796+0000","last_became_active":"2024-03-11T21:13:22.513912+0000","last_became_peered":"2024-03-11T21:13:22.513912+0000","last_unstale":"2024-03-11T21:13:49.595796+0000","last_undegraded":"2024-03-11T21:13:49.595796+0000","last_fullsized":"2024-03-11T21:13:49.595796+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:10:49.083027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:52:11.462480+0000","last_clean_scrub_stamp":"2024-03-11T21:10:49.083027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":743,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:49.599187+0000","last_change":"2024-03-11T21:10:02.099820+0000","last_active":"2024-03-11T21:13:49.599187+0000","last_peered":"2024-03-11T21:13:49.599187+0000","last_clean":"2024-03-11T21:13:49.599187+0000","last_became_active":"2024-03-11T20:50:32.646323+0000","last_became_peered":"2024-03-11T20:50:32.646323+0000","last_unstale":"2024-03-11T21:13:49.599187+0000","last_undegraded":"2024-03-11T21:13:49.599187+0000","last_fullsized":"2024-03-11T21:13:49.599187+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:10:02.099653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:10:02.099653+0000","last_clean_scrub_stamp":"2024-03-11T21:10:02.099653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":179,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":797,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:49.586358+0000","last_change":"2024-03-11T21:12:44.053469+0000","last_active":"2024-03-11T21:13:49.586358+0000","last_peered":"2024-03-11T21:13:49.586358+0000","last_clean":"2024-03-11T21:13:49.586358+0000","last_became_active":"2024-03-11T20:50:28.086513+0000","last_became_peered":"2024-03-11T20:50:28.086513+0000","last_unstale":"2024-03-11T21:13:49.586358+0000","last_undegraded":"2024-03-11T21:13:49.586358+0000","last_fullsized":"2024-03-11T21:13:49.586358+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:12:44.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:12:44.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:13:57.726059+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"357'25","reported_seq":843,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:49.594804+0000","last_change":"2024-03-11T21:12:03.355731+0000","last_active":"2024-03-11T21:13:49.594804+0000","last_peered":"2024-03-11T21:13:49.594804+0000","last_clean":"2024-03-11T21:13:49.594804+0000","last_became_active":"2024-03-11T21:00:48.464707+0000","last_became_peered":"2024-03-11T21:00:48.464707+0000","last_unstale":"2024-03-11T21:13:49.594804+0000","last_undegraded":"2024-03-11T21:13:49.594804+0000","last_fullsized":"2024-03-11T21:13:49.594804+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"357'25","last_scrub_stamp":"2024-03-11T21:12:03.355486+0000","last_deep_scrub":"145'13","last_deep_scrub_stamp":"2024-03-11T20:55:53.560287+0000","last_clean_scrub_stamp":"2024-03-11T21:12:03.355486+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":770,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:49.591209+0000","last_change":"2024-03-11T21:12:24.070009+0000","last_active":"2024-03-11T21:13:49.591209+0000","last_peered":"2024-03-11T21:13:49.591209+0000","last_clean":"2024-03-11T21:13:49.591209+0000","last_became_active":"2024-03-11T21:00:27.973405+0000","last_became_peered":"2024-03-11T21:00:27.973405+0000","last_unstale":"2024-03-11T21:13:49.591209+0000","last_undegraded":"2024-03-11T21:13:49.591209+0000","last_fullsized":"2024-03-11T21:13:49.591209+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:12:24.069900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:12:24.069900+0000","last_clean_scrub_stamp":"2024-03-11T21:12:24.069900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:13:51.665128+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":757,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:49.590891+0000","last_change":"2024-03-11T21:12:03.903669+0000","last_active":"2024-03-11T21:13:49.590891+0000","last_peered":"2024-03-11T21:13:49.590891+0000","last_clean":"2024-03-11T21:13:49.590891+0000","last_became_active":"2024-03-11T21:00:48.472812+0000","last_became_peered":"2024-03-11T21:00:48.472812+0000","last_unstale":"2024-03-11T21:13:49.590891+0000","last_undegraded":"2024-03-11T21:13:49.590891+0000","last_fullsized":"2024-03-11T21:13:49.590891+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:12:03.903559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:12:03.903559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":775,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:49.594711+0000","last_change":"2024-03-11T21:12:09.374347+0000","last_active":"2024-03-11T21:13:49.594711+0000","last_peered":"2024-03-11T21:13:49.594711+0000","last_clean":"2024-03-11T21:13:49.594711+0000","last_became_active":"2024-03-11T20:50:28.075760+0000","last_became_peered":"2024-03-11T20:50:28.075760+0000","last_unstale":"2024-03-11T21:13:49.594711+0000","last_undegraded":"2024-03-11T21:13:49.594711+0000","last_fullsized":"2024-03-11T21:13:49.594711+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:12:09.374261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:58:55.977289+0000","last_clean_scrub_stamp":"2024-03-11T21:12:09.374261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"358'18","reported_seq":348,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:49.595742+0000","last_change":"2024-03-11T21:13:22.535023+0000","last_active":"2024-03-11T21:13:49.595742+0000","last_peered":"2024-03-11T21:13:49.595742+0000","last_clean":"2024-03-11T21:13:49.595742+0000","last_became_active":"2024-03-11T21:13:22.521494+0000","last_became_peered":"2024-03-11T21:13:22.521494+0000","last_unstale":"2024-03-11T21:13:49.595742+0000","last_undegraded":"2024-03-11T21:13:49.595742+0000","last_fullsized":"2024-03-11T21:13:49.595742+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'18","last_scrub_stamp":"2024-03-11T21:12:32.003163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:12:32.003163+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":24,"num_read_kb":21,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"186'2","reported_seq":751,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:49.598321+0000","last_change":"2024-03-11T21:13:22.564435+0000","last_active":"2024-03-11T21:13:49.598321+0000","last_peered":"2024-03-11T21:13:49.598321+0000","last_clean":"2024-03-11T21:13:49.598321+0000","last_became_active":"2024-03-11T21:13:22.524574+0000","last_became_peered":"2024-03-11T21:13:22.524574+0000","last_unstale":"2024-03-11T21:13:49.598321+0000","last_undegraded":"2024-03-11T21:13:49.598321+0000","last_fullsized":"2024-03-11T21:13:49.598321+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"186'2","last_scrub_stamp":"2024-03-11T21:11:11.081135+0000","last_deep_scrub":"186'2","last_deep_scrub_stamp":"2024-03-11T20:59:04.534850+0000","last_clean_scrub_stamp":"2024-03-11T21:11:11.081135+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":592,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:49.589148+0000","last_change":"2024-03-11T21:12:30.039588+0000","last_active":"2024-03-11T21:13:49.589148+0000","last_peered":"2024-03-11T21:13:49.589148+0000","last_clean":"2024-03-11T21:13:49.589148+0000","last_became_active":"2024-03-11T21:05:17.314174+0000","last_became_peered":"2024-03-11T21:05:17.314174+0000","last_unstale":"2024-03-11T21:13:49.589148+0000","last_undegraded":"2024-03-11T21:13:49.589148+0000","last_fullsized":"2024-03-11T21:13:49.589148+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:12:30.039450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:12:30.039450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:13:52.686426+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":478,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-11T21:13:40.494565+0000","last_change":"2024-03-11T21:13:22.509985+0000","last_active":"2024-03-11T21:13:40.494565+0000","last_peered":"2024-03-11T21:13:40.494565+0000","last_clean":"2024-03-11T21:13:40.494565+0000","last_became_active":"2024-03-11T21:13:22.509622+0000","last_became_peered":"2024-03-11T21:13:22.509622+0000","last_unstale":"2024-03-11T21:13:40.494565+0000","last_undegraded":"2024-03-11T21:13:40.494565+0000","last_fullsized":"2024-03-11T21:13:40.494565+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:12:09.798977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:12:09.798977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"358'35","reported_seq":569,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:49.590908+0000","last_change":"2024-03-11T21:11:16.867612+0000","last_active":"2024-03-11T21:13:49.590908+0000","last_peered":"2024-03-11T21:13:49.590908+0000","last_clean":"2024-03-11T21:13:49.590908+0000","last_became_active":"2024-03-11T21:05:17.314507+0000","last_became_peered":"2024-03-11T21:05:17.314507+0000","last_unstale":"2024-03-11T21:13:49.590908+0000","last_undegraded":"2024-03-11T21:13:49.590908+0000","last_fullsized":"2024-03-11T21:13:49.590908+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'35","last_scrub_stamp":"2024-03-11T21:11:16.867326+0000","last_deep_scrub":"161'15","last_deep_scrub_stamp":"2024-03-11T20:57:15.125548+0000","last_clean_scrub_stamp":"2024-03-11T21:11:16.867326+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":342,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:49.592939+0000","last_change":"2024-03-11T21:13:22.511095+0000","last_active":"2024-03-11T21:13:49.592939+0000","last_peered":"2024-03-11T21:13:49.592939+0000","last_clean":"2024-03-11T21:13:49.592939+0000","last_became_active":"2024-03-11T21:13:22.510676+0000","last_became_peered":"2024-03-11T21:13:22.510676+0000","last_unstale":"2024-03-11T21:13:49.592939+0000","last_undegraded":"2024-03-11T21:13:49.592939+0000","last_fullsized":"2024-03-11T21:13:49.592939+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:12:12.183563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:12:12.183563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":480,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-11T21:13:40.494206+0000","last_change":"2024-03-11T21:12:50.895561+0000","last_active":"2024-03-11T21:13:40.494206+0000","last_peered":"2024-03-11T21:13:40.494206+0000","last_clean":"2024-03-11T21:13:40.494206+0000","last_became_active":"2024-03-11T21:00:48.459043+0000","last_became_peered":"2024-03-11T21:00:48.459043+0000","last_unstale":"2024-03-11T21:13:40.494206+0000","last_undegraded":"2024-03-11T21:13:40.494206+0000","last_fullsized":"2024-03-11T21:13:40.494206+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:12:50.895379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:12:50.895379+0000","last_clean_scrub_stamp":"2024-03-11T21:12:50.895379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:14:03.686288+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":789,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:49.585247+0000","last_change":"2024-03-11T21:11:53.953625+0000","last_active":"2024-03-11T21:13:49.585247+0000","last_peered":"2024-03-11T21:13:49.585247+0000","last_clean":"2024-03-11T21:13:49.585247+0000","last_became_active":"2024-03-11T20:50:28.082737+0000","last_became_peered":"2024-03-11T20:50:28.082737+0000","last_unstale":"2024-03-11T21:13:49.585247+0000","last_undegraded":"2024-03-11T21:13:49.585247+0000","last_fullsized":"2024-03-11T21:13:49.585247+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:11:53.953494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:28.404997+0000","last_clean_scrub_stamp":"2024-03-11T21:11:53.953494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"358'21","reported_seq":812,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:49.591540+0000","last_change":"2024-03-11T21:12:48.118469+0000","last_active":"2024-03-11T21:13:49.591540+0000","last_peered":"2024-03-11T21:13:49.591540+0000","last_clean":"2024-03-11T21:13:49.591540+0000","last_became_active":"2024-03-11T21:00:27.967624+0000","last_became_peered":"2024-03-11T21:00:27.967624+0000","last_unstale":"2024-03-11T21:13:49.591540+0000","last_undegraded":"2024-03-11T21:13:49.591540+0000","last_fullsized":"2024-03-11T21:13:49.591540+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'21","last_scrub_stamp":"2024-03-11T21:12:48.118343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:12:48.118343+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:13:58.609085+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":426,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:49.595330+0000","last_change":"2024-03-11T21:12:29.603649+0000","last_active":"2024-03-11T21:13:49.595330+0000","last_peered":"2024-03-11T21:13:49.595330+0000","last_clean":"2024-03-11T21:13:49.595330+0000","last_became_active":"2024-03-11T21:08:05.299913+0000","last_became_peered":"2024-03-11T21:08:05.299913+0000","last_unstale":"2024-03-11T21:13:49.595330+0000","last_undegraded":"2024-03-11T21:13:49.595330+0000","last_fullsized":"2024-03-11T21:13:49.595330+0000","mapping_epoch":319,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:12:29.603519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:57:16.032302+0000","last_clean_scrub_stamp":"2024-03-11T21:12:29.603519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":179,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"112'12","reported_seq":1059,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:49.594904+0000","last_change":"2024-03-11T21:12:36.443795+0000","last_active":"2024-03-11T21:13:49.594904+0000","last_peered":"2024-03-11T21:13:49.594904+0000","last_clean":"2024-03-11T21:13:49.594904+0000","last_became_active":"2024-03-11T20:50:32.650064+0000","last_became_peered":"2024-03-11T20:50:32.650064+0000","last_unstale":"2024-03-11T21:13:49.594904+0000","last_undegraded":"2024-03-11T21:13:49.594904+0000","last_fullsized":"2024-03-11T21:13:49.594904+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'12","last_scrub_stamp":"2024-03-11T21:12:36.443649+0000","last_deep_scrub":"112'12","last_deep_scrub_stamp":"2024-03-11T20:53:11.843951+0000","last_clean_scrub_stamp":"2024-03-11T21:12:36.443649+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:13:56.252437+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":233,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:49.594964+0000","last_change":"2024-03-11T21:13:22.514616+0000","last_active":"2024-03-11T21:13:49.594964+0000","last_peered":"2024-03-11T21:13:49.594964+0000","last_clean":"2024-03-11T21:13:49.594964+0000","last_became_active":"2024-03-11T21:13:22.512230+0000","last_became_peered":"2024-03-11T21:13:22.512230+0000","last_unstale":"2024-03-11T21:13:49.594964+0000","last_undegraded":"2024-03-11T21:13:49.594964+0000","last_fullsized":"2024-03-11T21:13:49.594964+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:12:45.881111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:06:52.862971+0000","last_clean_scrub_stamp":"2024-03-11T21:12:45.881111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"357'25","reported_seq":446,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:49.590924+0000","last_change":"2024-03-11T21:13:22.561247+0000","last_active":"2024-03-11T21:13:49.590924+0000","last_peered":"2024-03-11T21:13:49.590924+0000","last_clean":"2024-03-11T21:13:49.590924+0000","last_became_active":"2024-03-11T21:13:22.518229+0000","last_became_peered":"2024-03-11T21:13:22.518229+0000","last_unstale":"2024-03-11T21:13:49.590924+0000","last_undegraded":"2024-03-11T21:13:49.590924+0000","last_fullsized":"2024-03-11T21:13:49.590924+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"357'25","last_scrub_stamp":"2024-03-11T21:12:27.006374+0000","last_deep_scrub":"119'13","last_deep_scrub_stamp":"2024-03-11T20:54:15.443472+0000","last_clean_scrub_stamp":"2024-03-11T21:12:27.006374+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":342,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:49.591553+0000","last_change":"2024-03-11T21:13:22.528681+0000","last_active":"2024-03-11T21:13:49.591553+0000","last_peered":"2024-03-11T21:13:49.591553+0000","last_clean":"2024-03-11T21:13:49.591553+0000","last_became_active":"2024-03-11T21:13:22.528209+0000","last_became_peered":"2024-03-11T21:13:22.528209+0000","last_unstale":"2024-03-11T21:13:49.591553+0000","last_undegraded":"2024-03-11T21:13:49.591553+0000","last_fullsized":"2024-03-11T21:13:49.591553+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:12:05.248307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:12:05.248307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"212'17","reported_seq":1066,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:49.591310+0000","last_change":"2024-03-11T21:12:13.934238+0000","last_active":"2024-03-11T21:13:49.591310+0000","last_peered":"2024-03-11T21:13:49.591310+0000","last_clean":"2024-03-11T21:13:49.591310+0000","last_became_active":"2024-03-11T21:00:48.467116+0000","last_became_peered":"2024-03-11T21:00:48.467116+0000","last_unstale":"2024-03-11T21:13:49.591310+0000","last_undegraded":"2024-03-11T21:13:49.591310+0000","last_fullsized":"2024-03-11T21:13:49.591310+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"212'17","last_scrub_stamp":"2024-03-11T21:12:13.934164+0000","last_deep_scrub":"110'13","last_deep_scrub_stamp":"2024-03-11T20:53:00.434532+0000","last_clean_scrub_stamp":"2024-03-11T21:12:13.934164+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":784,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:49.593215+0000","last_change":"2024-03-11T21:12:39.708354+0000","last_active":"2024-03-11T21:13:49.593215+0000","last_peered":"2024-03-11T21:13:49.593215+0000","last_clean":"2024-03-11T21:13:49.593215+0000","last_became_active":"2024-03-11T20:50:28.070325+0000","last_became_peered":"2024-03-11T20:50:28.070325+0000","last_unstale":"2024-03-11T21:13:49.593215+0000","last_undegraded":"2024-03-11T21:13:49.593215+0000","last_fullsized":"2024-03-11T21:13:49.593215+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:12:39.708033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:12:39.708033+0000","last_clean_scrub_stamp":"2024-03-11T21:12:39.708033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:13:51.845494+0000","scrub_duration":239,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"358'32","reported_seq":827,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:49.590853+0000","last_change":"2024-03-11T21:12:43.060210+0000","last_active":"2024-03-11T21:13:49.590853+0000","last_peered":"2024-03-11T21:13:49.590853+0000","last_clean":"2024-03-11T21:13:49.590853+0000","last_became_active":"2024-03-11T21:00:27.967078+0000","last_became_peered":"2024-03-11T21:00:27.967078+0000","last_unstale":"2024-03-11T21:13:49.590853+0000","last_undegraded":"2024-03-11T21:13:49.590853+0000","last_fullsized":"2024-03-11T21:13:49.590853+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'32","last_scrub_stamp":"2024-03-11T21:12:43.060110+0000","last_deep_scrub":"116'16","last_deep_scrub_stamp":"2024-03-11T20:53:39.292118+0000","last_clean_scrub_stamp":"2024-03-11T21:12:43.060110+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:14:05.424881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":762,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:49.591120+0000","last_change":"2024-03-11T21:12:27.179524+0000","last_active":"2024-03-11T21:13:49.591120+0000","last_peered":"2024-03-11T21:13:49.591120+0000","last_clean":"2024-03-11T21:13:49.591120+0000","last_became_active":"2024-03-11T21:00:27.966997+0000","last_became_peered":"2024-03-11T21:00:27.966997+0000","last_unstale":"2024-03-11T21:13:49.591120+0000","last_undegraded":"2024-03-11T21:13:49.591120+0000","last_fullsized":"2024-03-11T21:13:49.591120+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:12:27.179425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:11:02.899468+0000","last_clean_scrub_stamp":"2024-03-11T21:12:27.179425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":59,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":764,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:49.593230+0000","last_change":"2024-03-11T21:12:32.947079+0000","last_active":"2024-03-11T21:13:49.593230+0000","last_peered":"2024-03-11T21:13:49.593230+0000","last_clean":"2024-03-11T21:13:49.593230+0000","last_became_active":"2024-03-11T21:00:48.468386+0000","last_became_peered":"2024-03-11T21:00:48.468386+0000","last_unstale":"2024-03-11T21:13:49.593230+0000","last_undegraded":"2024-03-11T21:13:49.593230+0000","last_fullsized":"2024-03-11T21:13:49.593230+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:12:32.946959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:12:32.946959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"166'2","reported_seq":817,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:49.588571+0000","last_change":"2024-03-11T21:12:00.999251+0000","last_active":"2024-03-11T21:13:49.588571+0000","last_peered":"2024-03-11T21:13:49.588571+0000","last_clean":"2024-03-11T21:13:49.588571+0000","last_became_active":"2024-03-11T21:00:48.466071+0000","last_became_peered":"2024-03-11T21:00:48.466071+0000","last_unstale":"2024-03-11T21:13:49.588571+0000","last_undegraded":"2024-03-11T21:13:49.588571+0000","last_fullsized":"2024-03-11T21:13:49.588571+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'2","last_scrub_stamp":"2024-03-11T21:12:00.999002+0000","last_deep_scrub":"166'2","last_deep_scrub_stamp":"2024-03-11T20:57:39.868776+0000","last_clean_scrub_stamp":"2024-03-11T21:12:00.999002+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1822,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"364'14","reported_seq":810,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:49.584560+0000","last_change":"2024-03-11T21:13:22.527910+0000","last_active":"2024-03-11T21:13:49.584560+0000","last_peered":"2024-03-11T21:13:49.584560+0000","last_clean":"2024-03-11T21:13:49.584560+0000","last_became_active":"2024-03-11T21:13:22.509157+0000","last_became_peered":"2024-03-11T21:13:22.509157+0000","last_unstale":"2024-03-11T21:13:49.584560+0000","last_undegraded":"2024-03-11T21:13:49.584560+0000","last_fullsized":"2024-03-11T21:13:49.584560+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"364'14","last_scrub_stamp":"2024-03-11T21:12:32.066251+0000","last_deep_scrub":"364'14","last_deep_scrub_stamp":"2024-03-11T21:11:15.819245+0000","last_clean_scrub_stamp":"2024-03-11T21:12:32.066251+0000","objects_scrubbed":2,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:13:56.990100+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":16,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":525,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:49.591013+0000","last_change":"2024-03-11T21:12:19.082499+0000","last_active":"2024-03-11T21:13:49.591013+0000","last_peered":"2024-03-11T21:13:49.591013+0000","last_clean":"2024-03-11T21:13:49.591013+0000","last_became_active":"2024-03-11T21:08:05.296676+0000","last_became_peered":"2024-03-11T21:08:05.296676+0000","last_unstale":"2024-03-11T21:13:49.591013+0000","last_undegraded":"2024-03-11T21:13:49.591013+0000","last_fullsized":"2024-03-11T21:13:49.591013+0000","mapping_epoch":319,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:12:19.082419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:12:19.082419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":25,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:49.591961+0000","last_change":"2024-03-11T21:13:22.527433+0000","last_active":"2024-03-11T21:13:49.591961+0000","last_peered":"2024-03-11T21:13:49.591961+0000","last_clean":"2024-03-11T21:13:49.591961+0000","last_became_active":"2024-03-11T21:13:22.527054+0000","last_became_peered":"2024-03-11T21:13:22.527054+0000","last_unstale":"2024-03-11T21:13:49.591961+0000","last_undegraded":"2024-03-11T21:13:49.591961+0000","last_fullsized":"2024-03-11T21:13:49.591961+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:12:24.000498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:54:09.537972+0000","last_clean_scrub_stamp":"2024-03-11T21:12:24.000498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":551,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-11T21:13:40.494028+0000","last_change":"2024-03-11T21:13:22.500667+0000","last_active":"2024-03-11T21:13:40.494028+0000","last_peered":"2024-03-11T21:13:40.494028+0000","last_clean":"2024-03-11T21:13:40.494028+0000","last_became_active":"2024-03-11T21:13:22.496976+0000","last_became_peered":"2024-03-11T21:13:22.496976+0000","last_unstale":"2024-03-11T21:13:40.494028+0000","last_undegraded":"2024-03-11T21:13:40.494028+0000","last_fullsized":"2024-03-11T21:13:40.494028+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:12:15.742332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:18.682947+0000","last_clean_scrub_stamp":"2024-03-11T21:12:15.742332+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"358'32","reported_seq":819,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:49.590288+0000","last_change":"2024-03-11T21:12:16.148325+0000","last_active":"2024-03-11T21:13:49.590288+0000","last_peered":"2024-03-11T21:13:49.590288+0000","last_clean":"2024-03-11T21:13:49.590288+0000","last_became_active":"2024-03-11T21:00:27.970878+0000","last_became_peered":"2024-03-11T21:00:27.970878+0000","last_unstale":"2024-03-11T21:13:49.590288+0000","last_undegraded":"2024-03-11T21:13:49.590288+0000","last_fullsized":"2024-03-11T21:13:49.590288+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'32","last_scrub_stamp":"2024-03-11T21:12:16.148129+0000","last_deep_scrub":"262'24","last_deep_scrub_stamp":"2024-03-11T21:03:28.128719+0000","last_clean_scrub_stamp":"2024-03-11T21:12:16.148129+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.6","version":"410'6794","reported_seq":7748,"reported_epoch":410,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-03-11T21:13:47.666055+0000","last_change":"2024-03-11T21:13:40.441341+0000","last_active":"2024-03-11T21:13:47.666055+0000","last_peered":"2024-03-11T21:13:47.666055+0000","last_clean":"2024-03-11T21:13:21.447262+0000","last_became_active":"2024-03-11T21:13:23.706053+0000","last_became_peered":"2024-03-11T21:13:23.706053+0000","last_unstale":"2024-03-11T21:13:47.666055+0000","last_undegraded":"2024-03-11T21:13:40.431611+0000","last_fullsized":"2024-03-11T21:13:23.487871+0000","mapping_epoch":404,"log_start":"391'2637","ondisk_log_start":"391'2637","created":380,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:11:46.536415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:11:46.536415+0000","last_clean_scrub_stamp":"2024-03-11T21:11:46.536415+0000","objects_scrubbed":0,"log_size":4157,"log_dups_size":0,"ondisk_log_size":4157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":352321536,"num_objects":5376,"num_object_clones":0,"num_object_copies":16128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":269,"num_objects_misplaced":4836,"num_objects_unfound":0,"num_objects_dirty":5376,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6794,"num_write_kb":389440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":441,"num_bytes_recovered":28901376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2],"avail_no_missing":["1","0","2"],"object_location_counts":[{"shards":"0,1,2","objects":4836},{"shards":"1,2","objects":271}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":761,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:49.595295+0000","last_change":"2024-03-11T21:09:51.110151+0000","last_active":"2024-03-11T21:13:49.595295+0000","last_peered":"2024-03-11T21:13:49.595295+0000","last_clean":"2024-03-11T21:13:49.595295+0000","last_became_active":"2024-03-11T21:00:48.466853+0000","last_became_peered":"2024-03-11T21:00:48.466853+0000","last_unstale":"2024-03-11T21:13:49.595295+0000","last_undegraded":"2024-03-11T21:13:49.595295+0000","last_fullsized":"2024-03-11T21:13:49.595295+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:51.109823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:52:53.450885+0000","last_clean_scrub_stamp":"2024-03-11T21:09:51.109823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":139,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"77'2","reported_seq":126,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:49.591532+0000","last_change":"2024-03-11T21:13:22.564041+0000","last_active":"2024-03-11T21:13:49.591532+0000","last_peered":"2024-03-11T21:13:49.591532+0000","last_clean":"2024-03-11T21:13:49.591532+0000","last_became_active":"2024-03-11T21:13:22.527317+0000","last_became_peered":"2024-03-11T21:13:22.527317+0000","last_unstale":"2024-03-11T21:13:49.591532+0000","last_undegraded":"2024-03-11T21:13:49.591532+0000","last_fullsized":"2024-03-11T21:13:49.591532+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'2","last_scrub_stamp":"2024-03-11T21:12:44.904275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:12:44.904275+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:14:10.157825+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"358'32","reported_seq":879,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:49.591642+0000","last_change":"2024-03-11T21:12:15.148614+0000","last_active":"2024-03-11T21:13:49.591642+0000","last_peered":"2024-03-11T21:13:49.591642+0000","last_clean":"2024-03-11T21:13:49.591642+0000","last_became_active":"2024-03-11T21:00:48.465254+0000","last_became_peered":"2024-03-11T21:00:48.465254+0000","last_unstale":"2024-03-11T21:13:49.591642+0000","last_undegraded":"2024-03-11T21:13:49.591642+0000","last_fullsized":"2024-03-11T21:13:49.591642+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'32","last_scrub_stamp":"2024-03-11T21:12:15.148495+0000","last_deep_scrub":"100'16","last_deep_scrub_stamp":"2024-03-11T20:56:08.036331+0000","last_clean_scrub_stamp":"2024-03-11T21:12:15.148495+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"286'18","reported_seq":785,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-11T21:13:40.494210+0000","last_change":"2024-03-11T21:12:14.753256+0000","last_active":"2024-03-11T21:13:40.494210+0000","last_peered":"2024-03-11T21:13:40.494210+0000","last_clean":"2024-03-11T21:13:40.494210+0000","last_became_active":"2024-03-11T21:05:17.317915+0000","last_became_peered":"2024-03-11T21:05:17.317915+0000","last_unstale":"2024-03-11T21:13:40.494210+0000","last_undegraded":"2024-03-11T21:13:40.494210+0000","last_fullsized":"2024-03-11T21:13:40.494210+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"286'18","last_scrub_stamp":"2024-03-11T21:12:14.753092+0000","last_deep_scrub":"212'17","last_deep_scrub_stamp":"2024-03-11T21:03:38.258850+0000","last_clean_scrub_stamp":"2024-03-11T21:12:14.753092+0000","objects_scrubbed":1,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:13:44.640806+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.d","version":"411'6914","reported_seq":6987,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:50.378767+0000","last_change":"2024-03-11T21:11:47.575844+0000","last_active":"2024-03-11T21:13:50.378767+0000","last_peered":"2024-03-11T21:13:50.378767+0000","last_clean":"2024-03-11T21:13:50.378767+0000","last_became_active":"2024-03-11T21:11:47.574539+0000","last_became_peered":"2024-03-11T21:11:47.574539+0000","last_unstale":"2024-03-11T21:13:50.378767+0000","last_undegraded":"2024-03-11T21:13:50.378767+0000","last_fullsized":"2024-03-11T21:13:50.378767+0000","mapping_epoch":380,"log_start":"391'2131","ondisk_log_start":"391'2131","created":380,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:11:46.536415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:11:46.536415+0000","last_clean_scrub_stamp":"2024-03-11T21:11:46.536415+0000","objects_scrubbed":0,"log_size":4783,"log_dups_size":0,"ondisk_log_size":4783,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":346226712,"num_objects":5284,"num_object_clones":0,"num_object_copies":15852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5284,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6914,"num_write_kb":390273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"184'130","reported_seq":628,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:49.586272+0000","last_change":"2024-03-11T21:11:55.979852+0000","last_active":"2024-03-11T21:13:49.586272+0000","last_peered":"2024-03-11T21:13:49.586272+0000","last_clean":"2024-03-11T21:13:49.586272+0000","last_became_active":"2024-03-11T21:11:28.393886+0000","last_became_peered":"2024-03-11T21:11:28.393886+0000","last_unstale":"2024-03-11T21:13:49.586272+0000","last_undegraded":"2024-03-11T21:13:49.586272+0000","last_fullsized":"2024-03-11T21:13:49.586272+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":3,"last_scrub":"184'130","last_scrub_stamp":"2024-03-11T21:11:55.979618+0000","last_deep_scrub":"184'130","last_deep_scrub_stamp":"2024-03-11T20:58:56.415267+0000","last_clean_scrub_stamp":"2024-03-11T21:11:55.979618+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":345,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:49.590488+0000","last_change":"2024-03-11T21:13:22.515636+0000","last_active":"2024-03-11T21:13:49.590488+0000","last_peered":"2024-03-11T21:13:49.590488+0000","last_clean":"2024-03-11T21:13:49.590488+0000","last_became_active":"2024-03-11T21:13:22.514731+0000","last_became_peered":"2024-03-11T21:13:22.514731+0000","last_unstale":"2024-03-11T21:13:49.590488+0000","last_undegraded":"2024-03-11T21:13:49.590488+0000","last_fullsized":"2024-03-11T21:13:49.590488+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:12:41.214079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:57:21.568400+0000","last_clean_scrub_stamp":"2024-03-11T21:12:41.214079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:14:08.726647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":487,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-11T21:13:40.495036+0000","last_change":"2024-03-11T21:13:22.499724+0000","last_active":"2024-03-11T21:13:40.495036+0000","last_peered":"2024-03-11T21:13:40.495036+0000","last_clean":"2024-03-11T21:13:40.495036+0000","last_became_active":"2024-03-11T21:13:22.498571+0000","last_became_peered":"2024-03-11T21:13:22.498571+0000","last_unstale":"2024-03-11T21:13:40.495036+0000","last_undegraded":"2024-03-11T21:13:40.495036+0000","last_fullsized":"2024-03-11T21:13:40.495036+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:12:43.931266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:20.096874+0000","last_clean_scrub_stamp":"2024-03-11T21:12:43.931266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:13:54.685583+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"358'40","reported_seq":1112,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:49.594957+0000","last_change":"2024-03-11T21:11:55.408780+0000","last_active":"2024-03-11T21:13:49.594957+0000","last_peered":"2024-03-11T21:13:49.594957+0000","last_clean":"2024-03-11T21:13:49.594957+0000","last_became_active":"2024-03-11T20:50:30.094977+0000","last_became_peered":"2024-03-11T20:50:30.094977+0000","last_unstale":"2024-03-11T21:13:49.594957+0000","last_undegraded":"2024-03-11T21:13:49.594957+0000","last_fullsized":"2024-03-11T21:13:49.594957+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'40","last_scrub_stamp":"2024-03-11T21:11:55.408619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:11:55.408619+0000","objects_scrubbed":8,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":61,"num_read_kb":51,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":473,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-11T21:13:40.494072+0000","last_change":"2024-03-11T21:11:06.619167+0000","last_active":"2024-03-11T21:13:40.494072+0000","last_peered":"2024-03-11T21:13:40.494072+0000","last_clean":"2024-03-11T21:13:40.494072+0000","last_became_active":"2024-03-11T21:00:48.457901+0000","last_became_peered":"2024-03-11T21:00:48.457901+0000","last_unstale":"2024-03-11T21:13:40.494072+0000","last_undegraded":"2024-03-11T21:13:40.494072+0000","last_fullsized":"2024-03-11T21:13:40.494072+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:11:06.619029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:52:10.160202+0000","last_clean_scrub_stamp":"2024-03-11T21:11:06.619029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.b","version":"411'6930","reported_seq":15711,"reported_epoch":411,"state":"active+remapped+backfilling","last_fresh":"2024-03-11T21:13:50.292678+0000","last_change":"2024-03-11T21:13:23.633758+0000","last_active":"2024-03-11T21:13:50.292678+0000","last_peered":"2024-03-11T21:13:50.292678+0000","last_clean":"2024-03-11T21:13:21.435561+0000","last_became_active":"2024-03-11T21:13:23.603860+0000","last_became_peered":"2024-03-11T21:13:23.603860+0000","last_unstale":"2024-03-11T21:13:50.292678+0000","last_undegraded":"2024-03-11T21:13:50.292678+0000","last_fullsized":"2024-03-11T21:13:50.292678+0000","mapping_epoch":404,"log_start":"391'2894","ondisk_log_start":"391'2894","created":380,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:11:46.536415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:11:46.536415+0000","last_clean_scrub_stamp":"2024-03-11T21:11:46.536415+0000","objects_scrubbed":0,"log_size":4036,"log_dups_size":0,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":347471872,"num_objects":5302,"num_object_clones":0,"num_object_copies":15906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1271,"num_objects_unfound":0,"num_objects_dirty":5302,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6930,"num_write_kb":391424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4352,"num_bytes_recovered":285212672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,0],"avail_no_missing":["3","0","1"],"object_location_counts":[{"shards":"0,1,3","objects":5302}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"184'130","reported_seq":632,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:49.592819+0000","last_change":"2024-03-11T21:13:25.500506+0000","last_active":"2024-03-11T21:13:49.592819+0000","last_peered":"2024-03-11T21:13:49.592819+0000","last_clean":"2024-03-11T21:13:49.592819+0000","last_became_active":"2024-03-11T21:13:25.500198+0000","last_became_peered":"2024-03-11T21:13:25.500198+0000","last_unstale":"2024-03-11T21:13:49.592819+0000","last_undegraded":"2024-03-11T21:13:49.592819+0000","last_fullsized":"2024-03-11T21:13:49.592819+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":4,"last_scrub":"184'130","last_scrub_stamp":"2024-03-11T21:11:58.415125+0000","last_deep_scrub":"184'130","last_deep_scrub_stamp":"2024-03-11T20:58:56.415267+0000","last_clean_scrub_stamp":"2024-03-11T21:11:58.415125+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":741,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:49.592321+0000","last_change":"2024-03-11T21:09:39.887725+0000","last_active":"2024-03-11T21:13:49.592321+0000","last_peered":"2024-03-11T21:13:49.592321+0000","last_clean":"2024-03-11T21:13:49.592321+0000","last_became_active":"2024-03-11T21:00:27.959408+0000","last_became_peered":"2024-03-11T21:00:27.959408+0000","last_unstale":"2024-03-11T21:13:49.592321+0000","last_undegraded":"2024-03-11T21:13:49.592321+0000","last_fullsized":"2024-03-11T21:13:49.592321+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:09:39.887594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:09:39.887594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":790,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:49.593253+0000","last_change":"2024-03-11T21:12:12.373265+0000","last_active":"2024-03-11T21:13:49.593253+0000","last_peered":"2024-03-11T21:13:49.593253+0000","last_clean":"2024-03-11T21:13:49.593253+0000","last_became_active":"2024-03-11T21:00:48.471212+0000","last_became_peered":"2024-03-11T21:00:48.471212+0000","last_unstale":"2024-03-11T21:13:49.593253+0000","last_undegraded":"2024-03-11T21:13:49.593253+0000","last_fullsized":"2024-03-11T21:13:49.593253+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:12:12.373115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:09:55.786349+0000","last_clean_scrub_stamp":"2024-03-11T21:12:12.373115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"358'47","reported_seq":539,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:49.588979+0000","last_change":"2024-03-11T21:13:22.552852+0000","last_active":"2024-03-11T21:13:49.588979+0000","last_peered":"2024-03-11T21:13:49.588979+0000","last_clean":"2024-03-11T21:13:49.588979+0000","last_became_active":"2024-03-11T21:13:22.509979+0000","last_became_peered":"2024-03-11T21:13:22.509979+0000","last_unstale":"2024-03-11T21:13:49.588979+0000","last_undegraded":"2024-03-11T21:13:49.588979+0000","last_fullsized":"2024-03-11T21:13:49.588979+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'47","last_scrub_stamp":"2024-03-11T21:11:57.124294+0000","last_deep_scrub":"182'27","last_deep_scrub_stamp":"2024-03-11T20:58:52.058851+0000","last_clean_scrub_stamp":"2024-03-11T21:11:57.124294+0000","objects_scrubbed":8,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":67,"num_read_kb":53,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":473,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-11T21:13:40.494210+0000","last_change":"2024-03-11T21:12:01.754493+0000","last_active":"2024-03-11T21:13:40.494210+0000","last_peered":"2024-03-11T21:13:40.494210+0000","last_clean":"2024-03-11T21:13:40.494210+0000","last_became_active":"2024-03-11T21:00:48.458008+0000","last_became_peered":"2024-03-11T21:00:48.458008+0000","last_unstale":"2024-03-11T21:13:40.494210+0000","last_undegraded":"2024-03-11T21:13:40.494210+0000","last_fullsized":"2024-03-11T21:13:40.494210+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:12:01.754401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:33.319221+0000","last_clean_scrub_stamp":"2024-03-11T21:12:01.754401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.8","version":"411'7078","reported_seq":7150,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:50.295807+0000","last_change":"2024-03-11T21:11:47.560291+0000","last_active":"2024-03-11T21:13:50.295807+0000","last_peered":"2024-03-11T21:13:50.295807+0000","last_clean":"2024-03-11T21:13:50.295807+0000","last_became_active":"2024-03-11T21:11:47.559993+0000","last_became_peered":"2024-03-11T21:11:47.559993+0000","last_unstale":"2024-03-11T21:13:50.295807+0000","last_undegraded":"2024-03-11T21:13:50.295807+0000","last_fullsized":"2024-03-11T21:13:50.295807+0000","mapping_epoch":380,"log_start":"392'3117","ondisk_log_start":"392'3117","created":380,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:11:46.536415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:11:46.536415+0000","last_clean_scrub_stamp":"2024-03-11T21:11:46.536415+0000","objects_scrubbed":0,"log_size":3961,"log_dups_size":0,"ondisk_log_size":3961,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":351666176,"num_objects":5366,"num_object_clones":0,"num_object_copies":16098,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5366,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7078,"num_write_kb":398208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"184'130","reported_seq":1069,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:49.591101+0000","last_change":"2024-03-11T21:13:25.498839+0000","last_active":"2024-03-11T21:13:49.591101+0000","last_peered":"2024-03-11T21:13:49.591101+0000","last_clean":"2024-03-11T21:13:49.591101+0000","last_became_active":"2024-03-11T21:13:25.498453+0000","last_became_peered":"2024-03-11T21:13:25.498453+0000","last_unstale":"2024-03-11T21:13:49.591101+0000","last_undegraded":"2024-03-11T21:13:49.591101+0000","last_fullsized":"2024-03-11T21:13:49.591101+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":4,"last_scrub":"184'130","last_scrub_stamp":"2024-03-11T21:11:56.041618+0000","last_deep_scrub":"184'130","last_deep_scrub_stamp":"2024-03-11T20:58:56.415267+0000","last_clean_scrub_stamp":"2024-03-11T21:11:56.041618+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":763,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:49.591149+0000","last_change":"2024-03-11T21:11:20.989703+0000","last_active":"2024-03-11T21:13:49.591149+0000","last_peered":"2024-03-11T21:13:49.591149+0000","last_clean":"2024-03-11T21:13:49.591149+0000","last_became_active":"2024-03-11T20:50:34.321038+0000","last_became_peered":"2024-03-11T20:50:34.321038+0000","last_unstale":"2024-03-11T21:13:49.591149+0000","last_undegraded":"2024-03-11T21:13:49.591149+0000","last_fullsized":"2024-03-11T21:13:49.591149+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:11:20.989384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:11:20.989384+0000","last_clean_scrub_stamp":"2024-03-11T21:11:20.989384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"358'50","reported_seq":523,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-11T21:13:40.493847+0000","last_change":"2024-03-11T21:12:06.861343+0000","last_active":"2024-03-11T21:13:40.493847+0000","last_peered":"2024-03-11T21:13:40.493847+0000","last_clean":"2024-03-11T21:13:40.493847+0000","last_became_active":"2024-03-11T21:00:48.456178+0000","last_became_peered":"2024-03-11T21:00:48.456178+0000","last_unstale":"2024-03-11T21:13:40.493847+0000","last_undegraded":"2024-03-11T21:13:40.493847+0000","last_fullsized":"2024-03-11T21:13:40.493847+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'50","last_scrub_stamp":"2024-03-11T21:12:06.861183+0000","last_deep_scrub":"109'26","last_deep_scrub_stamp":"2024-03-11T20:52:49.216378+0000","last_clean_scrub_stamp":"2024-03-11T21:12:06.861183+0000","objects_scrubbed":8,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":58,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":211,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:49.598811+0000","last_change":"2024-03-11T21:13:22.515096+0000","last_active":"2024-03-11T21:13:49.598811+0000","last_peered":"2024-03-11T21:13:49.598811+0000","last_clean":"2024-03-11T21:13:49.598811+0000","last_became_active":"2024-03-11T21:13:22.512525+0000","last_became_peered":"2024-03-11T21:13:22.512525+0000","last_unstale":"2024-03-11T21:13:49.598811+0000","last_undegraded":"2024-03-11T21:13:49.598811+0000","last_fullsized":"2024-03-11T21:13:49.598811+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:10:55.016272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:10:55.016272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":479,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-11T21:13:40.494584+0000","last_change":"2024-03-11T21:13:22.516009+0000","last_active":"2024-03-11T21:13:40.494584+0000","last_peered":"2024-03-11T21:13:40.494584+0000","last_clean":"2024-03-11T21:13:40.494584+0000","last_became_active":"2024-03-11T21:13:22.515562+0000","last_became_peered":"2024-03-11T21:13:22.515562+0000","last_unstale":"2024-03-11T21:13:40.494584+0000","last_undegraded":"2024-03-11T21:13:40.494584+0000","last_fullsized":"2024-03-11T21:13:40.494584+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:12:49.854667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:12:49.854667+0000","last_clean_scrub_stamp":"2024-03-11T21:12:49.854667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:14:06.307772+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"411'6895","reported_seq":6967,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:50.270778+0000","last_change":"2024-03-11T21:11:47.562755+0000","last_active":"2024-03-11T21:13:50.270778+0000","last_peered":"2024-03-11T21:13:50.270778+0000","last_clean":"2024-03-11T21:13:50.270778+0000","last_became_active":"2024-03-11T21:11:47.562374+0000","last_became_peered":"2024-03-11T21:11:47.562374+0000","last_unstale":"2024-03-11T21:13:50.270778+0000","last_undegraded":"2024-03-11T21:13:50.270778+0000","last_fullsized":"2024-03-11T21:13:50.270778+0000","mapping_epoch":380,"log_start":"391'2884","ondisk_log_start":"391'2884","created":380,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:11:46.536415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:11:46.536415+0000","last_clean_scrub_stamp":"2024-03-11T21:11:46.536415+0000","objects_scrubbed":0,"log_size":4011,"log_dups_size":0,"ondisk_log_size":4011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":343474176,"num_objects":5241,"num_object_clones":0,"num_object_copies":15723,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6895,"num_write_kb":388352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":757,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:49.595909+0000","last_change":"2024-03-11T21:11:59.527905+0000","last_active":"2024-03-11T21:13:49.595909+0000","last_peered":"2024-03-11T21:13:49.595909+0000","last_clean":"2024-03-11T21:13:49.595909+0000","last_became_active":"2024-03-11T21:00:27.962668+0000","last_became_peered":"2024-03-11T21:00:27.962668+0000","last_unstale":"2024-03-11T21:13:49.595909+0000","last_undegraded":"2024-03-11T21:13:49.595909+0000","last_fullsized":"2024-03-11T21:13:49.595909+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:11:59.527788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:11:59.527788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":155,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"358'35","reported_seq":799,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:49.593402+0000","last_change":"2024-03-11T21:11:53.949072+0000","last_active":"2024-03-11T21:13:49.593402+0000","last_peered":"2024-03-11T21:13:49.593402+0000","last_clean":"2024-03-11T21:13:49.593402+0000","last_became_active":"2024-03-11T20:50:30.101747+0000","last_became_peered":"2024-03-11T20:50:30.101747+0000","last_unstale":"2024-03-11T21:13:49.593402+0000","last_undegraded":"2024-03-11T21:13:49.593402+0000","last_fullsized":"2024-03-11T21:13:49.593402+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'35","last_scrub_stamp":"2024-03-11T21:11:53.948819+0000","last_deep_scrub":"284'25","last_deep_scrub_stamp":"2024-03-11T21:04:53.217188+0000","last_clean_scrub_stamp":"2024-03-11T21:11:53.948819+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":777,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:49.593072+0000","last_change":"2024-03-11T21:12:18.409027+0000","last_active":"2024-03-11T21:13:49.593072+0000","last_peered":"2024-03-11T21:13:49.593072+0000","last_clean":"2024-03-11T21:13:49.593072+0000","last_became_active":"2024-03-11T20:50:28.085503+0000","last_became_peered":"2024-03-11T20:50:28.085503+0000","last_unstale":"2024-03-11T21:13:49.593072+0000","last_undegraded":"2024-03-11T21:13:49.593072+0000","last_fullsized":"2024-03-11T21:13:49.593072+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:12:18.408807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:56:03.479180+0000","last_clean_scrub_stamp":"2024-03-11T21:12:18.408807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"184'130","reported_seq":1061,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:49.590187+0000","last_change":"2024-03-11T21:13:25.504672+0000","last_active":"2024-03-11T21:13:49.590187+0000","last_peered":"2024-03-11T21:13:49.590187+0000","last_clean":"2024-03-11T21:13:49.590187+0000","last_became_active":"2024-03-11T21:13:25.504354+0000","last_became_peered":"2024-03-11T21:13:25.504354+0000","last_unstale":"2024-03-11T21:13:49.590187+0000","last_undegraded":"2024-03-11T21:13:49.590187+0000","last_fullsized":"2024-03-11T21:13:49.590187+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"184'130","last_scrub_stamp":"2024-03-11T21:11:55.042705+0000","last_deep_scrub":"184'130","last_deep_scrub_stamp":"2024-03-11T20:58:56.415267+0000","last_clean_scrub_stamp":"2024-03-11T21:11:55.042705+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":784,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:49.588067+0000","last_change":"2024-03-11T21:11:54.931292+0000","last_active":"2024-03-11T21:13:49.588067+0000","last_peered":"2024-03-11T21:13:49.588067+0000","last_clean":"2024-03-11T21:13:49.588067+0000","last_became_active":"2024-03-11T21:00:27.959467+0000","last_became_peered":"2024-03-11T21:00:27.959467+0000","last_unstale":"2024-03-11T21:13:49.588067+0000","last_undegraded":"2024-03-11T21:13:49.588067+0000","last_fullsized":"2024-03-11T21:13:49.588067+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:11:54.930990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:54:22.448812+0000","last_clean_scrub_stamp":"2024-03-11T21:11:54.930990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.9","version":"411'6819","reported_seq":13372,"reported_epoch":411,"state":"active+remapped+backfilling","last_fresh":"2024-03-11T21:13:50.285960+0000","last_change":"2024-03-11T21:13:23.597208+0000","last_active":"2024-03-11T21:13:50.285960+0000","last_peered":"2024-03-11T21:13:50.285960+0000","last_clean":"2024-03-11T21:13:21.566728+0000","last_became_active":"2024-03-11T21:13:23.587951+0000","last_became_peered":"2024-03-11T21:13:23.587951+0000","last_unstale":"2024-03-11T21:13:50.285960+0000","last_undegraded":"2024-03-11T21:13:50.285960+0000","last_fullsized":"2024-03-11T21:13:50.285960+0000","mapping_epoch":404,"log_start":"391'2862","ondisk_log_start":"391'2862","created":380,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:11:46.536415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:11:46.536415+0000","last_clean_scrub_stamp":"2024-03-11T21:11:46.536415+0000","objects_scrubbed":0,"log_size":3957,"log_dups_size":0,"ondisk_log_size":3957,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":345309184,"num_objects":5269,"num_object_clones":0,"num_object_copies":15807,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2303,"num_objects_unfound":0,"num_objects_dirty":5269,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6819,"num_write_kb":386816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3238,"num_bytes_recovered":212205568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,0],"avail_no_missing":["3","0","6"],"object_location_counts":[{"shards":"0,3,6","objects":5269}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"184'130","reported_seq":621,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-11T21:13:40.495046+0000","last_change":"2024-03-11T21:11:55.755937+0000","last_active":"2024-03-11T21:13:40.495046+0000","last_peered":"2024-03-11T21:13:40.495046+0000","last_clean":"2024-03-11T21:13:40.495046+0000","last_became_active":"2024-03-11T21:11:28.397947+0000","last_became_peered":"2024-03-11T21:11:28.397947+0000","last_unstale":"2024-03-11T21:13:40.495046+0000","last_undegraded":"2024-03-11T21:13:40.495046+0000","last_fullsized":"2024-03-11T21:13:40.495046+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":3,"last_scrub":"184'130","last_scrub_stamp":"2024-03-11T21:11:55.755820+0000","last_deep_scrub":"184'130","last_deep_scrub_stamp":"2024-03-11T20:58:56.415267+0000","last_clean_scrub_stamp":"2024-03-11T21:11:55.755820+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":473,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-11T21:13:40.494666+0000","last_change":"2024-03-11T21:11:54.793910+0000","last_active":"2024-03-11T21:13:40.494666+0000","last_peered":"2024-03-11T21:13:40.494666+0000","last_clean":"2024-03-11T21:13:40.494666+0000","last_became_active":"2024-03-11T21:00:48.464596+0000","last_became_peered":"2024-03-11T21:00:48.464596+0000","last_unstale":"2024-03-11T21:13:40.494666+0000","last_undegraded":"2024-03-11T21:13:40.494666+0000","last_fullsized":"2024-03-11T21:13:40.494666+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:11:54.793800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:11:54.793800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"358'49","reported_seq":392,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:49.599008+0000","last_change":"2024-03-11T21:13:22.789906+0000","last_active":"2024-03-11T21:13:49.599008+0000","last_peered":"2024-03-11T21:13:49.599008+0000","last_clean":"2024-03-11T21:13:49.599008+0000","last_became_active":"2024-03-11T21:13:22.523826+0000","last_became_peered":"2024-03-11T21:13:22.523826+0000","last_unstale":"2024-03-11T21:13:49.599008+0000","last_undegraded":"2024-03-11T21:13:49.599008+0000","last_fullsized":"2024-03-11T21:13:49.599008+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'49","last_scrub_stamp":"2024-03-11T21:12:15.033032+0000","last_deep_scrub":"100'21","last_deep_scrub_stamp":"2024-03-11T20:51:31.835302+0000","last_clean_scrub_stamp":"2024-03-11T21:12:15.033032+0000","objects_scrubbed":7,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":54,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":517,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-11T21:13:40.494426+0000","last_change":"2024-03-11T21:13:22.506333+0000","last_active":"2024-03-11T21:13:40.494426+0000","last_peered":"2024-03-11T21:13:40.494426+0000","last_clean":"2024-03-11T21:13:40.494426+0000","last_became_active":"2024-03-11T21:13:22.505962+0000","last_became_peered":"2024-03-11T21:13:22.505962+0000","last_unstale":"2024-03-11T21:13:40.494426+0000","last_undegraded":"2024-03-11T21:13:40.494426+0000","last_fullsized":"2024-03-11T21:13:40.494426+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:12:39.915896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:12:39.915896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:13:43.818776+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":344,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-11T21:13:40.495040+0000","last_change":"2024-03-11T21:13:22.528408+0000","last_active":"2024-03-11T21:13:40.495040+0000","last_peered":"2024-03-11T21:13:40.495040+0000","last_clean":"2024-03-11T21:13:40.495040+0000","last_became_active":"2024-03-11T21:13:22.527326+0000","last_became_peered":"2024-03-11T21:13:22.527326+0000","last_unstale":"2024-03-11T21:13:40.495040+0000","last_undegraded":"2024-03-11T21:13:40.495040+0000","last_fullsized":"2024-03-11T21:13:40.495040+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:12:41.942568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:26.839782+0000","last_clean_scrub_stamp":"2024-03-11T21:12:41.942568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:14:08.881724+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.c","version":"410'6813","reported_seq":6883,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-11T21:13:47.625228+0000","last_change":"2024-03-11T21:11:47.552194+0000","last_active":"2024-03-11T21:13:47.625228+0000","last_peered":"2024-03-11T21:13:47.625228+0000","last_clean":"2024-03-11T21:13:47.625228+0000","last_became_active":"2024-03-11T21:11:47.551863+0000","last_became_peered":"2024-03-11T21:11:47.551863+0000","last_unstale":"2024-03-11T21:13:47.625228+0000","last_undegraded":"2024-03-11T21:13:47.625228+0000","last_fullsized":"2024-03-11T21:13:47.625228+0000","mapping_epoch":380,"log_start":"391'2680","ondisk_log_start":"391'2680","created":380,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:11:46.536415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:11:46.536415+0000","last_clean_scrub_stamp":"2024-03-11T21:11:46.536415+0000","objects_scrubbed":0,"log_size":4133,"log_dups_size":0,"ondisk_log_size":4133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":353697792,"num_objects":5397,"num_object_clones":0,"num_object_copies":16191,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5397,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6813,"num_write_kb":390720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"184'130","reported_seq":637,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:49.585937+0000","last_change":"2024-03-11T21:13:25.508307+0000","last_active":"2024-03-11T21:13:49.585937+0000","last_peered":"2024-03-11T21:13:49.585937+0000","last_clean":"2024-03-11T21:13:49.585937+0000","last_became_active":"2024-03-11T21:13:25.507914+0000","last_became_peered":"2024-03-11T21:13:25.507914+0000","last_unstale":"2024-03-11T21:13:49.585937+0000","last_undegraded":"2024-03-11T21:13:49.585937+0000","last_fullsized":"2024-03-11T21:13:49.585937+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":3,"last_scrub":"184'130","last_scrub_stamp":"2024-03-11T21:11:58.005866+0000","last_deep_scrub":"184'130","last_deep_scrub_stamp":"2024-03-11T20:58:56.415267+0000","last_clean_scrub_stamp":"2024-03-11T21:11:58.005866+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":473,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-11T21:13:40.494255+0000","last_change":"2024-03-11T21:11:10.720532+0000","last_active":"2024-03-11T21:13:40.494255+0000","last_peered":"2024-03-11T21:13:40.494255+0000","last_clean":"2024-03-11T21:13:40.494255+0000","last_became_active":"2024-03-11T21:00:48.469688+0000","last_became_peered":"2024-03-11T21:00:48.469688+0000","last_unstale":"2024-03-11T21:13:40.494255+0000","last_undegraded":"2024-03-11T21:13:40.494255+0000","last_fullsized":"2024-03-11T21:13:40.494255+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:11:10.720386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:11:10.720386+0000","last_clean_scrub_stamp":"2024-03-11T21:11:10.720386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"145'3","reported_seq":795,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:49.584986+0000","last_change":"2024-03-11T21:13:22.525767+0000","last_active":"2024-03-11T21:13:49.584986+0000","last_peered":"2024-03-11T21:13:49.584986+0000","last_clean":"2024-03-11T21:13:49.584986+0000","last_became_active":"2024-03-11T21:13:22.510700+0000","last_became_peered":"2024-03-11T21:13:22.510700+0000","last_unstale":"2024-03-11T21:13:49.584986+0000","last_undegraded":"2024-03-11T21:13:49.584986+0000","last_fullsized":"2024-03-11T21:13:49.584986+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'3","last_scrub_stamp":"2024-03-11T21:12:21.997550+0000","last_deep_scrub":"145'3","last_deep_scrub_stamp":"2024-03-11T20:55:52.848043+0000","last_clean_scrub_stamp":"2024-03-11T21:12:21.997550+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0.0024049620000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":782,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:49.591300+0000","last_change":"2024-03-11T21:13:22.522518+0000","last_active":"2024-03-11T21:13:49.591300+0000","last_peered":"2024-03-11T21:13:49.591300+0000","last_clean":"2024-03-11T21:13:49.591300+0000","last_became_active":"2024-03-11T21:13:22.518684+0000","last_became_peered":"2024-03-11T21:13:22.518684+0000","last_unstale":"2024-03-11T21:13:49.591300+0000","last_undegraded":"2024-03-11T21:13:49.591300+0000","last_fullsized":"2024-03-11T21:13:49.591300+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:11:59.979083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:11:59.979083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"358'58","reported_seq":684,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-11T21:13:40.494979+0000","last_change":"2024-03-11T21:12:20.000340+0000","last_active":"2024-03-11T21:13:40.494979+0000","last_peered":"2024-03-11T21:13:40.494979+0000","last_clean":"2024-03-11T21:13:40.494979+0000","last_became_active":"2024-03-11T21:00:48.473227+0000","last_became_peered":"2024-03-11T21:00:48.473227+0000","last_unstale":"2024-03-11T21:13:40.494979+0000","last_undegraded":"2024-03-11T21:13:40.494979+0000","last_fullsized":"2024-03-11T21:13:40.494979+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'58","last_scrub_stamp":"2024-03-11T21:12:20.000157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:12:20.000157+0000","objects_scrubbed":10,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":87,"num_read_kb":72,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"405'14","reported_seq":494,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:49.592966+0000","last_change":"2024-03-11T21:13:22.563300+0000","last_active":"2024-03-11T21:13:49.592966+0000","last_peered":"2024-03-11T21:13:49.592966+0000","last_clean":"2024-03-11T21:13:49.592966+0000","last_became_active":"2024-03-11T21:13:22.504504+0000","last_became_peered":"2024-03-11T21:13:22.504504+0000","last_unstale":"2024-03-11T21:13:49.592966+0000","last_undegraded":"2024-03-11T21:13:49.592966+0000","last_fullsized":"2024-03-11T21:13:49.592966+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"298'13","last_scrub_stamp":"2024-03-11T21:12:31.022898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:12:31.022898+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:14:00.907145+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.f","version":"411'6819","reported_seq":6891,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:50.024908+0000","last_change":"2024-03-11T21:11:47.576455+0000","last_active":"2024-03-11T21:13:50.024908+0000","last_peered":"2024-03-11T21:13:50.024908+0000","last_clean":"2024-03-11T21:13:50.024908+0000","last_became_active":"2024-03-11T21:11:47.575627+0000","last_became_peered":"2024-03-11T21:11:47.575627+0000","last_unstale":"2024-03-11T21:13:50.024908+0000","last_undegraded":"2024-03-11T21:13:50.024908+0000","last_fullsized":"2024-03-11T21:13:50.024908+0000","mapping_epoch":380,"log_start":"391'2578","ondisk_log_start":"391'2578","created":380,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:11:46.536415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:11:46.536415+0000","last_clean_scrub_stamp":"2024-03-11T21:11:46.536415+0000","objects_scrubbed":0,"log_size":4241,"log_dups_size":0,"ondisk_log_size":4241,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":343212032,"num_objects":5237,"num_object_clones":0,"num_object_copies":15711,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5237,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6819,"num_write_kb":385792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"184'130","reported_seq":628,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:49.587700+0000","last_change":"2024-03-11T21:11:56.977406+0000","last_active":"2024-03-11T21:13:49.587700+0000","last_peered":"2024-03-11T21:13:49.587700+0000","last_clean":"2024-03-11T21:13:49.587700+0000","last_became_active":"2024-03-11T21:11:28.397374+0000","last_became_peered":"2024-03-11T21:11:28.397374+0000","last_unstale":"2024-03-11T21:13:49.587700+0000","last_undegraded":"2024-03-11T21:13:49.587700+0000","last_fullsized":"2024-03-11T21:13:49.587700+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":3,"last_scrub":"184'130","last_scrub_stamp":"2024-03-11T21:11:56.977088+0000","last_deep_scrub":"184'130","last_deep_scrub_stamp":"2024-03-11T20:58:56.415267+0000","last_clean_scrub_stamp":"2024-03-11T21:11:56.977088+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":495,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:49.593563+0000","last_change":"2024-03-11T21:12:24.980257+0000","last_active":"2024-03-11T21:13:49.593563+0000","last_peered":"2024-03-11T21:13:49.593563+0000","last_clean":"2024-03-11T21:13:49.593563+0000","last_became_active":"2024-03-11T21:05:17.441521+0000","last_became_peered":"2024-03-11T21:05:17.441521+0000","last_unstale":"2024-03-11T21:13:49.593563+0000","last_undegraded":"2024-03-11T21:13:49.593563+0000","last_fullsized":"2024-03-11T21:13:49.593563+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:12:24.980140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:03:43.345257+0000","last_clean_scrub_stamp":"2024-03-11T21:12:24.980140+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":516,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:49.599107+0000","last_change":"2024-03-11T21:12:40.097209+0000","last_active":"2024-03-11T21:13:49.599107+0000","last_peered":"2024-03-11T21:13:49.599107+0000","last_clean":"2024-03-11T21:13:49.599107+0000","last_became_active":"2024-03-11T21:05:17.315844+0000","last_became_peered":"2024-03-11T21:05:17.315844+0000","last_unstale":"2024-03-11T21:13:49.599107+0000","last_undegraded":"2024-03-11T21:13:49.599107+0000","last_fullsized":"2024-03-11T21:13:49.599107+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:12:40.096694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:22.617999+0000","last_clean_scrub_stamp":"2024-03-11T21:12:40.096694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:13:50.559981+0000","scrub_duration":139,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"284'2","reported_seq":478,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-11T21:13:40.495295+0000","last_change":"2024-03-11T21:11:57.838884+0000","last_active":"2024-03-11T21:13:40.495295+0000","last_peered":"2024-03-11T21:13:40.495295+0000","last_clean":"2024-03-11T21:13:40.495295+0000","last_became_active":"2024-03-11T21:00:48.468682+0000","last_became_peered":"2024-03-11T21:00:48.468682+0000","last_unstale":"2024-03-11T21:13:40.495295+0000","last_undegraded":"2024-03-11T21:13:40.495295+0000","last_fullsized":"2024-03-11T21:13:40.495295+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"284'2","last_scrub_stamp":"2024-03-11T21:11:57.838662+0000","last_deep_scrub":"284'2","last_deep_scrub_stamp":"2024-03-11T21:06:47.645682+0000","last_clean_scrub_stamp":"2024-03-11T21:11:57.838662+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"357'21","reported_seq":797,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:49.592998+0000","last_change":"2024-03-11T21:13:22.784011+0000","last_active":"2024-03-11T21:13:49.592998+0000","last_peered":"2024-03-11T21:13:49.592998+0000","last_clean":"2024-03-11T21:13:49.592998+0000","last_became_active":"2024-03-11T21:13:22.525457+0000","last_became_peered":"2024-03-11T21:13:22.525457+0000","last_unstale":"2024-03-11T21:13:49.592998+0000","last_undegraded":"2024-03-11T21:13:49.592998+0000","last_fullsized":"2024-03-11T21:13:49.592998+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'17","last_scrub_stamp":"2024-03-11T21:09:54.016577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:09:54.016577+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":31,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"405'12","reported_seq":1054,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:49.591141+0000","last_change":"2024-03-11T21:13:22.540306+0000","last_active":"2024-03-11T21:13:49.591141+0000","last_peered":"2024-03-11T21:13:49.591141+0000","last_clean":"2024-03-11T21:13:49.591141+0000","last_became_active":"2024-03-11T21:13:22.512081+0000","last_became_peered":"2024-03-11T21:13:22.512081+0000","last_unstale":"2024-03-11T21:13:49.591141+0000","last_undegraded":"2024-03-11T21:13:49.591141+0000","last_fullsized":"2024-03-11T21:13:49.591141+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:12:02.860638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:12:02.860638+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.e","version":"410'6772","reported_seq":6842,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-11T21:13:47.670967+0000","last_change":"2024-03-11T21:11:47.571655+0000","last_active":"2024-03-11T21:13:47.670967+0000","last_peered":"2024-03-11T21:13:47.670967+0000","last_clean":"2024-03-11T21:13:47.670967+0000","last_became_active":"2024-03-11T21:11:47.570431+0000","last_became_peered":"2024-03-11T21:11:47.570431+0000","last_unstale":"2024-03-11T21:13:47.670967+0000","last_undegraded":"2024-03-11T21:13:47.670967+0000","last_fullsized":"2024-03-11T21:13:47.670967+0000","mapping_epoch":380,"log_start":"391'2600","ondisk_log_start":"391'2600","created":380,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:11:46.536415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:11:46.536415+0000","last_clean_scrub_stamp":"2024-03-11T21:11:46.536415+0000","objects_scrubbed":0,"log_size":4172,"log_dups_size":0,"ondisk_log_size":4172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":352845824,"num_objects":5384,"num_object_clones":0,"num_object_copies":16152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5384,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6772,"num_write_kb":388992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"184'130","reported_seq":623,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:49.591061+0000","last_change":"2024-03-11T21:11:57.831886+0000","last_active":"2024-03-11T21:13:49.591061+0000","last_peered":"2024-03-11T21:13:49.591061+0000","last_clean":"2024-03-11T21:13:49.591061+0000","last_became_active":"2024-03-11T21:11:28.393756+0000","last_became_peered":"2024-03-11T21:11:28.393756+0000","last_unstale":"2024-03-11T21:13:49.591061+0000","last_undegraded":"2024-03-11T21:13:49.591061+0000","last_fullsized":"2024-03-11T21:13:49.591061+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":3,"last_scrub":"184'130","last_scrub_stamp":"2024-03-11T21:11:57.831636+0000","last_deep_scrub":"184'130","last_deep_scrub_stamp":"2024-03-11T20:58:56.415267+0000","last_clean_scrub_stamp":"2024-03-11T21:11:57.831636+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":771,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:49.590625+0000","last_change":"2024-03-11T21:12:08.234963+0000","last_active":"2024-03-11T21:13:49.590625+0000","last_peered":"2024-03-11T21:13:49.590625+0000","last_clean":"2024-03-11T21:13:49.590625+0000","last_became_active":"2024-03-11T20:50:34.326947+0000","last_became_peered":"2024-03-11T20:50:34.326947+0000","last_unstale":"2024-03-11T21:13:49.590625+0000","last_undegraded":"2024-03-11T21:13:49.590625+0000","last_fullsized":"2024-03-11T21:13:49.590625+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:12:08.234826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:06:43.271453+0000","last_clean_scrub_stamp":"2024-03-11T21:12:08.234826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":127,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"120'6","reported_seq":2057,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:49.591298+0000","last_change":"2024-03-11T21:12:09.860900+0000","last_active":"2024-03-11T21:13:49.591298+0000","last_peered":"2024-03-11T21:13:49.591298+0000","last_clean":"2024-03-11T21:13:49.591298+0000","last_became_active":"2024-03-11T21:00:48.468997+0000","last_became_peered":"2024-03-11T21:00:48.468997+0000","last_unstale":"2024-03-11T21:13:49.591298+0000","last_undegraded":"2024-03-11T21:13:49.591298+0000","last_fullsized":"2024-03-11T21:13:49.591298+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"120'6","last_scrub_stamp":"2024-03-11T21:12:09.860783+0000","last_deep_scrub":"120'6","last_deep_scrub_stamp":"2024-03-11T20:54:26.468057+0000","last_clean_scrub_stamp":"2024-03-11T21:12:09.860783+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0035533370000000002,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1263,"num_read_kb":1258,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":790,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:49.586460+0000","last_change":"2024-03-11T21:12:11.984791+0000","last_active":"2024-03-11T21:13:49.586460+0000","last_peered":"2024-03-11T21:13:49.586460+0000","last_clean":"2024-03-11T21:13:49.586460+0000","last_became_active":"2024-03-11T20:50:28.071707+0000","last_became_peered":"2024-03-11T20:50:28.071707+0000","last_unstale":"2024-03-11T21:13:49.586460+0000","last_undegraded":"2024-03-11T21:13:49.586460+0000","last_fullsized":"2024-03-11T21:13:49.586460+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:12:11.984666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:23.417283+0000","last_clean_scrub_stamp":"2024-03-11T21:12:11.984666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"358'36","reported_seq":805,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-11T21:13:40.495510+0000","last_change":"2024-03-11T21:13:22.549343+0000","last_active":"2024-03-11T21:13:40.495510+0000","last_peered":"2024-03-11T21:13:40.495510+0000","last_clean":"2024-03-11T21:13:40.495510+0000","last_became_active":"2024-03-11T21:13:22.527950+0000","last_became_peered":"2024-03-11T21:13:22.527950+0000","last_unstale":"2024-03-11T21:13:40.495510+0000","last_undegraded":"2024-03-11T21:13:40.495510+0000","last_fullsized":"2024-03-11T21:13:40.495510+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'36","last_scrub_stamp":"2024-03-11T21:12:14.031345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:12:14.031345+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":47,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":545,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:49.593063+0000","last_change":"2024-03-11T21:12:10.335229+0000","last_active":"2024-03-11T21:13:49.593063+0000","last_peered":"2024-03-11T21:13:49.593063+0000","last_clean":"2024-03-11T21:13:49.593063+0000","last_became_active":"2024-03-11T21:05:17.315807+0000","last_became_peered":"2024-03-11T21:05:17.315807+0000","last_unstale":"2024-03-11T21:13:49.593063+0000","last_undegraded":"2024-03-11T21:13:49.593063+0000","last_fullsized":"2024-03-11T21:13:49.593063+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:12:10.335107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:09:51.796383+0000","last_clean_scrub_stamp":"2024-03-11T21:12:10.335107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.1","version":"411'6913","reported_seq":15979,"reported_epoch":411,"state":"active+remapped+backfilling","last_fresh":"2024-03-11T21:13:50.425846+0000","last_change":"2024-03-11T21:13:23.614010+0000","last_active":"2024-03-11T21:13:50.425846+0000","last_peered":"2024-03-11T21:13:50.425846+0000","last_clean":"2024-03-11T21:13:21.428994+0000","last_became_active":"2024-03-11T21:13:23.602491+0000","last_became_peered":"2024-03-11T21:13:23.602491+0000","last_unstale":"2024-03-11T21:13:50.425846+0000","last_undegraded":"2024-03-11T21:13:50.425846+0000","last_fullsized":"2024-03-11T21:13:50.425846+0000","mapping_epoch":404,"log_start":"391'2071","ondisk_log_start":"391'2071","created":380,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:11:46.536415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:11:46.536415+0000","last_clean_scrub_stamp":"2024-03-11T21:11:46.536415+0000","objects_scrubbed":0,"log_size":4842,"log_dups_size":0,"ondisk_log_size":4842,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":344129536,"num_objects":5251,"num_object_clones":0,"num_object_copies":15753,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1143,"num_objects_unfound":0,"num_objects_dirty":5251,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6913,"num_write_kb":389248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4495,"num_bytes_recovered":294584320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,0],"avail_no_missing":["6","0","3"],"object_location_counts":[{"shards":"0,3,6","objects":5251}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":740,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:49.590874+0000","last_change":"2024-03-11T21:12:51.179642+0000","last_active":"2024-03-11T21:13:49.590874+0000","last_peered":"2024-03-11T21:13:49.590874+0000","last_clean":"2024-03-11T21:13:49.590874+0000","last_became_active":"2024-03-11T20:50:34.335370+0000","last_became_peered":"2024-03-11T20:50:34.335370+0000","last_unstale":"2024-03-11T21:13:49.590874+0000","last_undegraded":"2024-03-11T21:13:49.590874+0000","last_fullsized":"2024-03-11T21:13:49.590874+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:12:51.179514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:12:51.179514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:13:52.389496+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"286'18","reported_seq":464,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:49.585847+0000","last_change":"2024-03-11T21:12:04.970976+0000","last_active":"2024-03-11T21:13:49.585847+0000","last_peered":"2024-03-11T21:13:49.585847+0000","last_clean":"2024-03-11T21:13:49.585847+0000","last_became_active":"2024-03-11T21:05:17.449213+0000","last_became_peered":"2024-03-11T21:05:17.449213+0000","last_unstale":"2024-03-11T21:13:49.585847+0000","last_undegraded":"2024-03-11T21:13:49.585847+0000","last_fullsized":"2024-03-11T21:13:49.585847+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"286'18","last_scrub_stamp":"2024-03-11T21:12:04.970865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:12:04.970865+0000","objects_scrubbed":1,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":793,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:49.590289+0000","last_change":"2024-03-11T21:13:22.526160+0000","last_active":"2024-03-11T21:13:49.590289+0000","last_peered":"2024-03-11T21:13:49.590289+0000","last_clean":"2024-03-11T21:13:49.590289+0000","last_became_active":"2024-03-11T21:13:22.523670+0000","last_became_peered":"2024-03-11T21:13:22.523670+0000","last_unstale":"2024-03-11T21:13:49.590289+0000","last_undegraded":"2024-03-11T21:13:49.590289+0000","last_fullsized":"2024-03-11T21:13:49.590289+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:12:25.082734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:06:42.242801+0000","last_clean_scrub_stamp":"2024-03-11T21:12:25.082734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"357'24","reported_seq":487,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:49.590439+0000","last_change":"2024-03-11T21:12:20.041186+0000","last_active":"2024-03-11T21:13:49.590439+0000","last_peered":"2024-03-11T21:13:49.590439+0000","last_clean":"2024-03-11T21:13:49.590439+0000","last_became_active":"2024-03-11T21:08:05.298170+0000","last_became_peered":"2024-03-11T21:08:05.298170+0000","last_unstale":"2024-03-11T21:13:49.590439+0000","last_undegraded":"2024-03-11T21:13:49.590439+0000","last_fullsized":"2024-03-11T21:13:49.590439+0000","mapping_epoch":319,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":0,"last_scrub":"357'24","last_scrub_stamp":"2024-03-11T21:12:20.041015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:12:20.041015+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":41,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.0","version":"410'6864","reported_seq":6935,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-11T21:13:47.671442+0000","last_change":"2024-03-11T21:11:47.570843+0000","last_active":"2024-03-11T21:13:47.671442+0000","last_peered":"2024-03-11T21:13:47.671442+0000","last_clean":"2024-03-11T21:13:47.671442+0000","last_became_active":"2024-03-11T21:11:47.570517+0000","last_became_peered":"2024-03-11T21:11:47.570517+0000","last_unstale":"2024-03-11T21:13:47.671442+0000","last_undegraded":"2024-03-11T21:13:47.671442+0000","last_fullsized":"2024-03-11T21:13:47.671442+0000","mapping_epoch":380,"log_start":"391'2673","ondisk_log_start":"391'2673","created":380,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:11:46.536415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:11:46.536415+0000","last_clean_scrub_stamp":"2024-03-11T21:11:46.536415+0000","objects_scrubbed":0,"log_size":4191,"log_dups_size":0,"ondisk_log_size":4191,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":357171200,"num_objects":5450,"num_object_clones":0,"num_object_copies":16350,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5450,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6864,"num_write_kb":394048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"184'130","reported_seq":623,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:49.593639+0000","last_change":"2024-03-11T21:11:56.424263+0000","last_active":"2024-03-11T21:13:49.593639+0000","last_peered":"2024-03-11T21:13:49.593639+0000","last_clean":"2024-03-11T21:13:49.593639+0000","last_became_active":"2024-03-11T21:11:28.396897+0000","last_became_peered":"2024-03-11T21:11:28.396897+0000","last_unstale":"2024-03-11T21:13:49.593639+0000","last_undegraded":"2024-03-11T21:13:49.593639+0000","last_fullsized":"2024-03-11T21:13:49.593639+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":4,"last_scrub":"184'130","last_scrub_stamp":"2024-03-11T21:11:56.424044+0000","last_deep_scrub":"184'130","last_deep_scrub_stamp":"2024-03-11T20:58:56.415267+0000","last_clean_scrub_stamp":"2024-03-11T21:11:56.424044+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":783,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:49.595005+0000","last_change":"2024-03-11T21:13:22.512005+0000","last_active":"2024-03-11T21:13:49.595005+0000","last_peered":"2024-03-11T21:13:49.595005+0000","last_clean":"2024-03-11T21:13:49.595005+0000","last_became_active":"2024-03-11T21:13:22.511157+0000","last_became_peered":"2024-03-11T21:13:22.511157+0000","last_unstale":"2024-03-11T21:13:49.595005+0000","last_undegraded":"2024-03-11T21:13:49.595005+0000","last_fullsized":"2024-03-11T21:13:49.595005+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:12:28.598084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:27.775534+0000","last_clean_scrub_stamp":"2024-03-11T21:12:28.598084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":127,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":480,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-11T21:13:40.494857+0000","last_change":"2024-03-11T21:11:58.865003+0000","last_active":"2024-03-11T21:13:40.494857+0000","last_peered":"2024-03-11T21:13:40.494857+0000","last_clean":"2024-03-11T21:13:40.494857+0000","last_became_active":"2024-03-11T21:00:48.469596+0000","last_became_peered":"2024-03-11T21:00:48.469596+0000","last_unstale":"2024-03-11T21:13:40.494857+0000","last_undegraded":"2024-03-11T21:13:40.494857+0000","last_fullsized":"2024-03-11T21:13:40.494857+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:11:58.864737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:11:58.864737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":473,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-11T21:13:40.494862+0000","last_change":"2024-03-11T21:12:42.924462+0000","last_active":"2024-03-11T21:13:40.494862+0000","last_peered":"2024-03-11T21:13:40.494862+0000","last_clean":"2024-03-11T21:13:40.494862+0000","last_became_active":"2024-03-11T21:00:48.466084+0000","last_became_peered":"2024-03-11T21:00:48.466084+0000","last_unstale":"2024-03-11T21:13:40.494862+0000","last_undegraded":"2024-03-11T21:13:40.494862+0000","last_fullsized":"2024-03-11T21:13:40.494862+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:12:42.924357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:40.974147+0000","last_clean_scrub_stamp":"2024-03-11T21:12:42.924357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"358'46","reported_seq":876,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:49.591026+0000","last_change":"2024-03-11T21:12:20.007909+0000","last_active":"2024-03-11T21:13:49.591026+0000","last_peered":"2024-03-11T21:13:49.591026+0000","last_clean":"2024-03-11T21:13:49.591026+0000","last_became_active":"2024-03-11T21:00:48.472191+0000","last_became_peered":"2024-03-11T21:00:48.472191+0000","last_unstale":"2024-03-11T21:13:49.591026+0000","last_undegraded":"2024-03-11T21:13:49.591026+0000","last_fullsized":"2024-03-11T21:13:49.591026+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'46","last_scrub_stamp":"2024-03-11T21:12:20.007731+0000","last_deep_scrub":"116'22","last_deep_scrub_stamp":"2024-03-11T20:53:47.364709+0000","last_clean_scrub_stamp":"2024-03-11T21:12:20.007731+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3","version":"411'6815","reported_seq":6887,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:50.316037+0000","last_change":"2024-03-11T21:11:47.580286+0000","last_active":"2024-03-11T21:13:50.316037+0000","last_peered":"2024-03-11T21:13:50.316037+0000","last_clean":"2024-03-11T21:13:50.316037+0000","last_became_active":"2024-03-11T21:11:47.579025+0000","last_became_peered":"2024-03-11T21:11:47.579025+0000","last_unstale":"2024-03-11T21:13:50.316037+0000","last_undegraded":"2024-03-11T21:13:50.316037+0000","last_fullsized":"2024-03-11T21:13:50.316037+0000","mapping_epoch":380,"log_start":"391'1987","ondisk_log_start":"391'1987","created":380,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:11:46.536415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:11:46.536415+0000","last_clean_scrub_stamp":"2024-03-11T21:11:46.536415+0000","objects_scrubbed":0,"log_size":4828,"log_dups_size":0,"ondisk_log_size":4828,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":341770240,"num_objects":5215,"num_object_clones":0,"num_object_copies":15645,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5215,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6815,"num_write_kb":384960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"184'130","reported_seq":1060,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:49.590199+0000","last_change":"2024-03-11T21:11:57.036171+0000","last_active":"2024-03-11T21:13:49.590199+0000","last_peered":"2024-03-11T21:13:49.590199+0000","last_clean":"2024-03-11T21:13:49.590199+0000","last_became_active":"2024-03-11T21:11:28.405250+0000","last_became_peered":"2024-03-11T21:11:28.405250+0000","last_unstale":"2024-03-11T21:13:49.590199+0000","last_undegraded":"2024-03-11T21:13:49.590199+0000","last_fullsized":"2024-03-11T21:13:49.590199+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":4,"last_scrub":"184'130","last_scrub_stamp":"2024-03-11T21:11:57.035951+0000","last_deep_scrub":"184'130","last_deep_scrub_stamp":"2024-03-11T20:58:56.415267+0000","last_clean_scrub_stamp":"2024-03-11T21:11:57.035951+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":559,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-11T21:13:40.495066+0000","last_change":"2024-03-11T21:12:18.814077+0000","last_active":"2024-03-11T21:13:40.495066+0000","last_peered":"2024-03-11T21:13:40.495066+0000","last_clean":"2024-03-11T21:13:40.495066+0000","last_became_active":"2024-03-11T21:05:17.455236+0000","last_became_peered":"2024-03-11T21:05:17.455236+0000","last_unstale":"2024-03-11T21:13:40.495066+0000","last_undegraded":"2024-03-11T21:13:40.495066+0000","last_fullsized":"2024-03-11T21:13:40.495066+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:12:18.813941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:12:18.813941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":473,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-11T21:13:40.494909+0000","last_change":"2024-03-11T21:11:18.625591+0000","last_active":"2024-03-11T21:13:40.494909+0000","last_peered":"2024-03-11T21:13:40.494909+0000","last_clean":"2024-03-11T21:13:40.494909+0000","last_became_active":"2024-03-11T21:00:48.461367+0000","last_became_peered":"2024-03-11T21:00:48.461367+0000","last_unstale":"2024-03-11T21:13:40.494909+0000","last_undegraded":"2024-03-11T21:13:40.494909+0000","last_fullsized":"2024-03-11T21:13:40.494909+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:11:18.625296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:03:37.601229+0000","last_clean_scrub_stamp":"2024-03-11T21:11:18.625296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"358'25","reported_seq":811,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:49.595209+0000","last_change":"2024-03-11T21:13:22.544913+0000","last_active":"2024-03-11T21:13:49.595209+0000","last_peered":"2024-03-11T21:13:49.595209+0000","last_clean":"2024-03-11T21:13:49.595209+0000","last_became_active":"2024-03-11T21:13:22.507677+0000","last_became_peered":"2024-03-11T21:13:22.507677+0000","last_unstale":"2024-03-11T21:13:49.595209+0000","last_undegraded":"2024-03-11T21:13:49.595209+0000","last_fullsized":"2024-03-11T21:13:49.595209+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'25","last_scrub_stamp":"2024-03-11T21:12:08.861796+0000","last_deep_scrub":"267'19","last_deep_scrub_stamp":"2024-03-11T21:03:39.673317+0000","last_clean_scrub_stamp":"2024-03-11T21:12:08.861796+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":538,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:49.591216+0000","last_change":"2024-03-11T21:11:20.159860+0000","last_active":"2024-03-11T21:13:49.591216+0000","last_peered":"2024-03-11T21:13:49.591216+0000","last_clean":"2024-03-11T21:13:49.591216+0000","last_became_active":"2024-03-11T21:05:17.316136+0000","last_became_peered":"2024-03-11T21:05:17.316136+0000","last_unstale":"2024-03-11T21:13:49.591216+0000","last_undegraded":"2024-03-11T21:13:49.591216+0000","last_fullsized":"2024-03-11T21:13:49.591216+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:11:20.159640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:35.670149+0000","last_clean_scrub_stamp":"2024-03-11T21:11:20.159640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.2","version":"411'6985","reported_seq":7064,"reported_epoch":411,"state":"active+remapped+backfill_toofull","last_fresh":"2024-03-11T21:13:50.284393+0000","last_change":"2024-03-11T21:13:23.628182+0000","last_active":"2024-03-11T21:13:50.284393+0000","last_peered":"2024-03-11T21:13:50.284393+0000","last_clean":"2024-03-11T21:13:21.453051+0000","last_became_active":"2024-03-11T21:13:23.605311+0000","last_became_peered":"2024-03-11T21:13:23.605311+0000","last_unstale":"2024-03-11T21:13:50.284393+0000","last_undegraded":"2024-03-11T21:13:50.284393+0000","last_fullsized":"2024-03-11T21:13:50.284393+0000","mapping_epoch":404,"log_start":"391'2973","ondisk_log_start":"391'2973","created":380,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:11:46.536415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:11:46.536415+0000","last_clean_scrub_stamp":"2024-03-11T21:11:46.536415+0000","objects_scrubbed":0,"log_size":4012,"log_dups_size":0,"ondisk_log_size":4012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350289920,"num_objects":5345,"num_object_clones":0,"num_object_copies":16035,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5345,"num_objects_unfound":0,"num_objects_dirty":5345,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6985,"num_write_kb":394560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,0],"avail_no_missing":["3","0","2"],"object_location_counts":[{"shards":"0,2,3","objects":5345}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"184'130","reported_seq":623,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:49.593191+0000","last_change":"2024-03-11T21:11:57.380857+0000","last_active":"2024-03-11T21:13:49.593191+0000","last_peered":"2024-03-11T21:13:49.593191+0000","last_clean":"2024-03-11T21:13:49.593191+0000","last_became_active":"2024-03-11T21:11:28.398728+0000","last_became_peered":"2024-03-11T21:11:28.398728+0000","last_unstale":"2024-03-11T21:13:49.593191+0000","last_undegraded":"2024-03-11T21:13:49.593191+0000","last_fullsized":"2024-03-11T21:13:49.593191+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":4,"last_scrub":"184'130","last_scrub_stamp":"2024-03-11T21:11:57.380689+0000","last_deep_scrub":"184'130","last_deep_scrub_stamp":"2024-03-11T20:58:56.415267+0000","last_clean_scrub_stamp":"2024-03-11T21:11:57.380689+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":517,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:49.592842+0000","last_change":"2024-03-11T21:12:45.541739+0000","last_active":"2024-03-11T21:13:49.592842+0000","last_peered":"2024-03-11T21:13:49.592842+0000","last_clean":"2024-03-11T21:13:49.592842+0000","last_became_active":"2024-03-11T21:08:05.304963+0000","last_became_peered":"2024-03-11T21:08:05.304963+0000","last_unstale":"2024-03-11T21:13:49.592842+0000","last_undegraded":"2024-03-11T21:13:49.592842+0000","last_fullsized":"2024-03-11T21:13:49.592842+0000","mapping_epoch":319,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:12:45.541619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:12:45.541619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:14:07.443684+0000","scrub_duration":247,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"320'21","reported_seq":781,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:49.593707+0000","last_change":"2024-03-11T21:12:06.377586+0000","last_active":"2024-03-11T21:13:49.593707+0000","last_peered":"2024-03-11T21:13:49.593707+0000","last_clean":"2024-03-11T21:13:49.593707+0000","last_became_active":"2024-03-11T21:08:05.298887+0000","last_became_peered":"2024-03-11T21:08:05.298887+0000","last_unstale":"2024-03-11T21:13:49.593707+0000","last_undegraded":"2024-03-11T21:13:49.593707+0000","last_fullsized":"2024-03-11T21:13:49.593707+0000","mapping_epoch":319,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'21","last_scrub_stamp":"2024-03-11T21:12:06.377204+0000","last_deep_scrub":"146'14","last_deep_scrub_stamp":"2024-03-11T20:56:00.242211+0000","last_clean_scrub_stamp":"2024-03-11T21:12:06.377204+0000","objects_scrubbed":1,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":775,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:49.590718+0000","last_change":"2024-03-11T21:11:14.837109+0000","last_active":"2024-03-11T21:13:49.590718+0000","last_peered":"2024-03-11T21:13:49.590718+0000","last_clean":"2024-03-11T21:13:49.590718+0000","last_became_active":"2024-03-11T21:00:27.970573+0000","last_became_peered":"2024-03-11T21:00:27.970573+0000","last_unstale":"2024-03-11T21:13:49.590718+0000","last_undegraded":"2024-03-11T21:13:49.590718+0000","last_fullsized":"2024-03-11T21:13:49.590718+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:11:14.836808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:11:14.836808+0000","last_clean_scrub_stamp":"2024-03-11T21:11:14.836808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"358'60","reported_seq":943,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:49.585088+0000","last_change":"2024-03-11T21:12:07.992182+0000","last_active":"2024-03-11T21:13:49.585088+0000","last_peered":"2024-03-11T21:13:49.585088+0000","last_clean":"2024-03-11T21:13:49.585088+0000","last_became_active":"2024-03-11T21:00:48.469854+0000","last_became_peered":"2024-03-11T21:00:48.469854+0000","last_unstale":"2024-03-11T21:13:49.585088+0000","last_undegraded":"2024-03-11T21:13:49.585088+0000","last_fullsized":"2024-03-11T21:13:49.585088+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'60","last_scrub_stamp":"2024-03-11T21:12:07.992039+0000","last_deep_scrub":"161'28","last_deep_scrub_stamp":"2024-03-11T20:57:20.816278+0000","last_clean_scrub_stamp":"2024-03-11T21:12:07.992039+0000","objects_scrubbed":9,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":80,"num_read_kb":69,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.5","version":"411'7036","reported_seq":7108,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:50.007260+0000","last_change":"2024-03-11T21:11:47.579128+0000","last_active":"2024-03-11T21:13:50.007260+0000","last_peered":"2024-03-11T21:13:50.007260+0000","last_clean":"2024-03-11T21:13:50.007260+0000","last_became_active":"2024-03-11T21:11:47.578368+0000","last_became_peered":"2024-03-11T21:11:47.578368+0000","last_unstale":"2024-03-11T21:13:50.007260+0000","last_undegraded":"2024-03-11T21:13:50.007260+0000","last_fullsized":"2024-03-11T21:13:50.007260+0000","mapping_epoch":380,"log_start":"391'2719","ondisk_log_start":"391'2719","created":380,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:11:46.536415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:11:46.536415+0000","last_clean_scrub_stamp":"2024-03-11T21:11:46.536415+0000","objects_scrubbed":0,"log_size":4317,"log_dups_size":0,"ondisk_log_size":4317,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":348913664,"num_objects":5324,"num_object_clones":0,"num_object_copies":15972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5324,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7036,"num_write_kb":395520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":762,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:49.590005+0000","last_change":"2024-03-11T21:12:13.099306+0000","last_active":"2024-03-11T21:13:49.590005+0000","last_peered":"2024-03-11T21:13:49.590005+0000","last_clean":"2024-03-11T21:13:49.590005+0000","last_became_active":"2024-03-11T20:50:34.327139+0000","last_became_peered":"2024-03-11T20:50:34.327139+0000","last_unstale":"2024-03-11T21:13:49.590005+0000","last_undegraded":"2024-03-11T21:13:49.590005+0000","last_fullsized":"2024-03-11T21:13:49.590005+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:12:13.099217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:09:53.436482+0000","last_clean_scrub_stamp":"2024-03-11T21:12:13.099217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":227,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:49.593657+0000","last_change":"2024-03-11T21:13:22.514309+0000","last_active":"2024-03-11T21:13:49.593657+0000","last_peered":"2024-03-11T21:13:49.593657+0000","last_clean":"2024-03-11T21:13:49.593657+0000","last_became_active":"2024-03-11T21:13:22.513882+0000","last_became_peered":"2024-03-11T21:13:22.513882+0000","last_unstale":"2024-03-11T21:13:49.593657+0000","last_undegraded":"2024-03-11T21:13:49.593657+0000","last_fullsized":"2024-03-11T21:13:49.593657+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:12:33.830534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:57:17.447191+0000","last_clean_scrub_stamp":"2024-03-11T21:12:33.830534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":291,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"358'53","reported_seq":848,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:49.591436+0000","last_change":"2024-03-11T21:11:15.839953+0000","last_active":"2024-03-11T21:13:49.591436+0000","last_peered":"2024-03-11T21:13:49.591436+0000","last_clean":"2024-03-11T21:13:49.591436+0000","last_became_active":"2024-03-11T20:50:30.111819+0000","last_became_peered":"2024-03-11T20:50:30.111819+0000","last_unstale":"2024-03-11T21:13:49.591436+0000","last_undegraded":"2024-03-11T21:13:49.591436+0000","last_fullsized":"2024-03-11T21:13:49.591436+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'53","last_scrub_stamp":"2024-03-11T21:11:15.839681+0000","last_deep_scrub":"102'25","last_deep_scrub_stamp":"2024-03-11T20:51:35.187094+0000","last_clean_scrub_stamp":"2024-03-11T21:11:15.839681+0000","objects_scrubbed":8,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":61,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":795,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:49.584917+0000","last_change":"2024-03-11T21:12:46.033878+0000","last_active":"2024-03-11T21:13:49.584917+0000","last_peered":"2024-03-11T21:13:49.584917+0000","last_clean":"2024-03-11T21:13:49.584917+0000","last_became_active":"2024-03-11T21:00:48.362389+0000","last_became_peered":"2024-03-11T21:00:48.362389+0000","last_unstale":"2024-03-11T21:13:49.584917+0000","last_undegraded":"2024-03-11T21:13:49.584917+0000","last_fullsized":"2024-03-11T21:13:49.584917+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:12:46.033771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:12:46.033771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.4","version":"411'6901","reported_seq":6973,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:49.973145+0000","last_change":"2024-03-11T21:11:47.572412+0000","last_active":"2024-03-11T21:13:49.973145+0000","last_peered":"2024-03-11T21:13:49.973145+0000","last_clean":"2024-03-11T21:13:49.973145+0000","last_became_active":"2024-03-11T21:11:47.571530+0000","last_became_peered":"2024-03-11T21:11:47.571530+0000","last_unstale":"2024-03-11T21:13:49.973145+0000","last_undegraded":"2024-03-11T21:13:49.973145+0000","last_fullsized":"2024-03-11T21:13:49.973145+0000","mapping_epoch":380,"log_start":"391'2596","ondisk_log_start":"391'2596","created":380,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:11:46.536415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:11:46.536415+0000","last_clean_scrub_stamp":"2024-03-11T21:11:46.536415+0000","objects_scrubbed":0,"log_size":4305,"log_dups_size":0,"ondisk_log_size":4305,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":342818816,"num_objects":5231,"num_object_clones":0,"num_object_copies":15693,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5231,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6901,"num_write_kb":388224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":343,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:49.590638+0000","last_change":"2024-03-11T21:13:22.518429+0000","last_active":"2024-03-11T21:13:49.590638+0000","last_peered":"2024-03-11T21:13:49.590638+0000","last_clean":"2024-03-11T21:13:49.590638+0000","last_became_active":"2024-03-11T21:13:22.515452+0000","last_became_peered":"2024-03-11T21:13:22.515452+0000","last_unstale":"2024-03-11T21:13:49.590638+0000","last_undegraded":"2024-03-11T21:13:49.590638+0000","last_fullsized":"2024-03-11T21:13:49.590638+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:12:47.959930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:55:58.888890+0000","last_clean_scrub_stamp":"2024-03-11T21:12:47.959930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":472,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-11T21:13:40.494022+0000","last_change":"2024-03-11T21:12:11.806426+0000","last_active":"2024-03-11T21:13:40.494022+0000","last_peered":"2024-03-11T21:13:40.494022+0000","last_clean":"2024-03-11T21:13:40.494022+0000","last_became_active":"2024-03-11T21:00:48.469015+0000","last_became_peered":"2024-03-11T21:00:48.469015+0000","last_unstale":"2024-03-11T21:13:40.494022+0000","last_undegraded":"2024-03-11T21:13:40.494022+0000","last_fullsized":"2024-03-11T21:13:40.494022+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:12:11.805941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:12:11.805941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"393'35","reported_seq":822,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:49.593523+0000","last_change":"2024-03-11T21:12:35.497436+0000","last_active":"2024-03-11T21:13:49.593523+0000","last_peered":"2024-03-11T21:13:49.593523+0000","last_clean":"2024-03-11T21:13:49.593523+0000","last_became_active":"2024-03-11T20:50:30.109549+0000","last_became_peered":"2024-03-11T20:50:30.109549+0000","last_unstale":"2024-03-11T21:13:49.593523+0000","last_undegraded":"2024-03-11T21:13:49.593523+0000","last_fullsized":"2024-03-11T21:13:49.593523+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"393'35","last_scrub_stamp":"2024-03-11T21:12:35.497207+0000","last_deep_scrub":"393'35","last_deep_scrub_stamp":"2024-03-11T21:12:35.497207+0000","last_clean_scrub_stamp":"2024-03-11T21:12:35.497207+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":25,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:49.601179+0000","last_change":"2024-03-11T21:13:22.513201+0000","last_active":"2024-03-11T21:13:49.601179+0000","last_peered":"2024-03-11T21:13:49.601179+0000","last_clean":"2024-03-11T21:13:49.601179+0000","last_became_active":"2024-03-11T21:13:22.512704+0000","last_became_peered":"2024-03-11T21:13:22.512704+0000","last_unstale":"2024-03-11T21:13:49.601179+0000","last_undegraded":"2024-03-11T21:13:49.601179+0000","last_fullsized":"2024-03-11T21:13:49.601179+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:12:06.205727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:52:51.931295+0000","last_clean_scrub_stamp":"2024-03-11T21:12:06.205727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"411'6999","reported_seq":7071,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:50.645469+0000","last_change":"2024-03-11T21:11:47.566051+0000","last_active":"2024-03-11T21:13:50.645469+0000","last_peered":"2024-03-11T21:13:50.645469+0000","last_clean":"2024-03-11T21:13:50.645469+0000","last_became_active":"2024-03-11T21:11:47.564750+0000","last_became_peered":"2024-03-11T21:11:47.564750+0000","last_unstale":"2024-03-11T21:13:50.645469+0000","last_undegraded":"2024-03-11T21:13:50.645469+0000","last_fullsized":"2024-03-11T21:13:50.645469+0000","mapping_epoch":380,"log_start":"391'1878","ondisk_log_start":"391'1878","created":380,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:11:46.536415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:11:46.536415+0000","last_clean_scrub_stamp":"2024-03-11T21:11:46.536415+0000","objects_scrubbed":0,"log_size":5121,"log_dups_size":0,"ondisk_log_size":5121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:13:55.269911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":345047040,"num_objects":5265,"num_object_clones":0,"num_object_copies":15795,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5265,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6999,"num_write_kb":392448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":45,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:49.592882+0000","last_change":"2024-03-11T21:13:22.514888+0000","last_active":"2024-03-11T21:13:49.592882+0000","last_peered":"2024-03-11T21:13:49.592882+0000","last_clean":"2024-03-11T21:13:49.592882+0000","last_became_active":"2024-03-11T21:13:22.513923+0000","last_became_peered":"2024-03-11T21:13:22.513923+0000","last_unstale":"2024-03-11T21:13:49.592882+0000","last_undegraded":"2024-03-11T21:13:49.592882+0000","last_fullsized":"2024-03-11T21:13:49.592882+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:12:19.065992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:00:14.030599+0000","last_clean_scrub_stamp":"2024-03-11T21:12:19.065992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":205,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:49.592414+0000","last_change":"2024-03-11T21:11:01.150241+0000","last_active":"2024-03-11T21:13:49.592414+0000","last_peered":"2024-03-11T21:13:49.592414+0000","last_clean":"2024-03-11T21:13:49.592414+0000","last_became_active":"2024-03-11T21:08:05.307984+0000","last_became_peered":"2024-03-11T21:08:05.307984+0000","last_unstale":"2024-03-11T21:13:49.592414+0000","last_undegraded":"2024-03-11T21:13:49.592414+0000","last_fullsized":"2024-03-11T21:13:49.592414+0000","mapping_epoch":319,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:11:01.150119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:11:01.150119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.f","version":"391'46","reported_seq":861,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:49.592200+0000","last_change":"2024-03-11T21:13:22.572416+0000","last_active":"2024-03-11T21:13:49.592200+0000","last_peered":"2024-03-11T21:13:49.592200+0000","last_clean":"2024-03-11T21:13:49.592200+0000","last_became_active":"2024-03-11T21:13:22.523216+0000","last_became_peered":"2024-03-11T21:13:22.523216+0000","last_unstale":"2024-03-11T21:13:49.592200+0000","last_undegraded":"2024-03-11T21:13:49.592200+0000","last_fullsized":"2024-03-11T21:13:49.592200+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"391'46","last_scrub_stamp":"2024-03-11T21:12:31.145006+0000","last_deep_scrub":"391'46","last_deep_scrub_stamp":"2024-03-11T21:12:31.145006+0000","last_clean_scrub_stamp":"2024-03-11T21:12:31.145006+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:13:52.119494+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":768,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-03-11T21:13:49.592879+0000","last_change":"2024-03-11T21:12:38.473882+0000","last_active":"2024-03-11T21:13:49.592879+0000","last_peered":"2024-03-11T21:13:49.592879+0000","last_clean":"2024-03-11T21:13:49.592879+0000","last_became_active":"2024-03-11T20:50:32.652854+0000","last_became_peered":"2024-03-11T20:50:32.652854+0000","last_unstale":"2024-03-11T21:13:49.592879+0000","last_undegraded":"2024-03-11T21:13:49.592879+0000","last_fullsized":"2024-03-11T21:13:49.592879+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:12:38.473801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:59:00.946094+0000","last_clean_scrub_stamp":"2024-03-11T21:12:38.473801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":5566365720,"num_objects":84937,"num_object_clones":0,"num_object_copies":254811,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":269,"num_objects_misplaced":14898,"num_objects_unfound":0,"num_objects_dirty":84937,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":110347,"num_write_kb":6249025,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12526,"num_bytes_recovered":820903936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17417908224,"data_stored":17417896008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69067,"ondisk_log_size":69067,"up":48,"acting":47,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":20,"num_read_kb":20,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":4731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1430,"ondisk_log_size":1430,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1543,"num_read_kb":1300,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":228,"num_bytes_recovered":3548,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":466944,"data_stored":12426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3632,"internal_metadata":0},"log_size":1078,"ondisk_log_size":1078,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1263,"num_read_kb":1258,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":797,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":68,"seq":292057776419,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2603852,"kb_used_data":2036804,"kb_used_omap":1,"kb_used_meta":567038,"kb_avail":91141300,"statfs":{"total":95995035648,"available":93328691200,"internally_reserved":0,"allocated":2085687296,"data_stored":2085336591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":580647370},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":57,"seq":244813136173,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3211092,"kb_used_data":2706440,"kb_used_omap":1,"kb_used_meta":504574,"kb_avail":90534060,"statfs":{"total":95995035648,"available":92706877440,"internally_reserved":0,"allocated":2771394560,"data_stored":2771048335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":516684231},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":211,"seq":906238099613,"num_pgs":73,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3371936,"kb_used_data":2934424,"kb_used_omap":7,"kb_used_meta":437496,"kb_avail":90373216,"statfs":{"total":95995035648,"available":92542173184,"internally_reserved":0,"allocated":3004850176,"data_stored":3004480017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8078,"internal_metadata":447996018},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,2,4,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":0,"seq":34359738731,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378966,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2819164,"kb_used_data":2292368,"kb_used_omap":1,"kb_used_meta":526782,"kb_avail":90925988,"statfs":{"total":95995035648,"available":93108211712,"internally_reserved":0,"allocated":2347384832,"data_stored":2347030131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":539425221},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":28,"seq":120259084619,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3027708,"kb_used_data":2400948,"kb_used_omap":1,"kb_used_meta":626750,"kb_avail":90717444,"statfs":{"total":95995035648,"available":92894662656,"internally_reserved":0,"allocated":2458570752,"data_stored":2458236455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":641792456},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":204,"seq":876173328546,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2009840,"kb_used_data":1644260,"kb_used_omap":4,"kb_used_meta":365563,"kb_avail":91735312,"statfs":{"total":95995035648,"available":93936959488,"internally_reserved":0,"allocated":1683722240,"data_stored":1683348093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4451,"internal_metadata":374337181},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":44,"seq":188978561335,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1876764,"kb_used_data":1358292,"kb_used_omap":1,"kb_used_meta":518462,"kb_avail":91868388,"statfs":{"total":95995035648,"available":94073229312,"internally_reserved":0,"allocated":1390891008,"data_stored":1390532343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":530905541},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3632,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1743716352,"data_stored":1743716352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3002859520,"data_stored":3002859520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2344226816,"data_stored":2344222744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2432499712,"data_stored":2432499712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1676546048,"data_stored":1676541976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1383464960,"data_stored":1383464960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2760183808,"data_stored":2760179736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2074411008,"data_stored":2074411008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T21:14:02.915 INFO:tasks.cephadm.ceph_manager.ceph:would drop below min_pgs, proj 6, currently 16 2024-03-11T21:14:02.939 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:14:02.975 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:14:02.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:02.979 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:14:03.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:02 smithi146 bash[23809]: cluster 2024-03-11T21:14:02.113715+0000 osd.6 (osd.6) 403 : cluster 0 1.7 scrub starts 2024-03-11T21:14:03.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:02 smithi146 bash[23809]: cluster 2024-03-11T21:14:02.114858+0000 osd.6 (osd.6) 404 : cluster 0 1.7 scrub ok 2024-03-11T21:14:03.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:02 smithi146 bash[23809]: cluster 2024-03-11T21:14:02.225121+0000 osd.5 (osd.5) 499 : cluster 0 6.7 scrub starts 2024-03-11T21:14:03.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:02 smithi146 bash[23809]: cluster 2024-03-11T21:14:02.373704+0000 osd.5 (osd.5) 500 : cluster 0 6.7 scrub ok 2024-03-11T21:14:03.079 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:14:03.106 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:14:03.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:03.109 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:14:03.210 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:03.236 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:03.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:03.239 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:14:03.340 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:14:03.368 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:14:03.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:03.372 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:14:03.473 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:14:03.499 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:14:03.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:03.506 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:14:03.607 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:14:03.633 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:14:03.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:03.636 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:14:03.737 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:14:03.760 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:14:03.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:03.764 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:14:03.865 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:14:03.875 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:03 smithi023 bash[21372]: cluster 2024-03-11T21:14:02.415023+0000 mgr.y (mgr.24380) 860 : cluster 0 pgmap v990: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfilling, 155 active+clean; 4.7 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 857 op/s; 594/231873 objects degraded (0.256%); 8900/231873 objects misplaced (3.838%); 38 MiB/s, 604 objects/s recovering 2024-03-11T21:14:03.875 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:03 smithi023 bash[29146]: cluster 2024-03-11T21:14:02.415023+0000 mgr.y (mgr.24380) 860 : cluster 0 pgmap v990: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfilling, 155 active+clean; 4.7 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 857 op/s; 594/231873 objects degraded (0.256%); 8900/231873 objects misplaced (3.838%); 38 MiB/s, 604 objects/s recovering 2024-03-11T21:14:03.911 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:14:03.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:03.916 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:14:04.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:03 smithi146 bash[23809]: cluster 2024-03-11T21:14:02.415023+0000 mgr.y (mgr.24380) 860 : cluster 0 pgmap v990: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfilling, 155 active+clean; 4.7 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 857 op/s; 594/231873 objects degraded (0.256%); 8900/231873 objects misplaced (3.838%); 38 MiB/s, 604 objects/s recovering 2024-03-11T21:14:04.016 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:04.044 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:04.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:04.049 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:14:04.150 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:04.175 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:04.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:04.180 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:14:04.281 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:14:04.312 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:14:04.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:04.316 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:14:04.416 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:14:04.446 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:14:04.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:04.449 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:14:04.550 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:14:04.574 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:14:04.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:04.576 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:14:04.677 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:14:04.704 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:14:04.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:04.707 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:14:04.808 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:04.817 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T21:14:04.818 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:04 smithi023 bash[21372]: cluster 2024-03-11T21:14:04.184818+0000 osd.6 (osd.6) 405 : cluster 0 3.3 scrub starts 2024-03-11T21:14:04.818 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:04 smithi023 bash[21372]: cluster 2024-03-11T21:14:04.199755+0000 osd.6 (osd.6) 406 : cluster 0 3.3 scrub ok 2024-03-11T21:14:04.819 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:04 smithi023 bash[29146]: cluster 2024-03-11T21:14:04.184818+0000 osd.6 (osd.6) 405 : cluster 0 3.3 scrub starts 2024-03-11T21:14:04.819 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:04 smithi023 bash[29146]: cluster 2024-03-11T21:14:04.199755+0000 osd.6 (osd.6) 406 : cluster 0 3.3 scrub ok 2024-03-11T21:14:04.842 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:04.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:04.846 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:14:04.917 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-11T21:14:04.947 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:14:04.955 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:04 smithi146 bash[23809]: cluster 2024-03-11T21:14:04.184818+0000 osd.6 (osd.6) 405 : cluster 0 3.3 scrub starts 2024-03-11T21:14:04.955 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:04 smithi146 bash[23809]: cluster 2024-03-11T21:14:04.199755+0000 osd.6 (osd.6) 406 : cluster 0 3.3 scrub ok 2024-03-11T21:14:04.968 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:14:04.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:04.972 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:14:05.073 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:05.112 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:05.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:05.117 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:14:05.218 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:05.254 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:05.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:05.259 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:14:05.360 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:05.387 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:05.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:05.392 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:14:05.493 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:05.523 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:05.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:05.529 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:14:05.630 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:05.654 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:05.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:05.655 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:14:05.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:05 smithi146 bash[23809]: cluster 2024-03-11T21:14:04.415673+0000 mgr.y (mgr.24380) 861 : cluster 0 pgmap v991: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfilling, 155 active+clean; 4.6 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 585 op/s; 655/229008 objects degraded (0.286%); 8665/229008 objects misplaced (3.784%); 22 MiB/s, 347 objects/s recovering 2024-03-11T21:14:05.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:05 smithi146 bash[23809]: cluster 2024-03-11T21:14:05.158566+0000 osd.6 (osd.6) 407 : cluster 0 11.3 scrub starts 2024-03-11T21:14:05.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:05 smithi146 bash[23809]: cluster 2024-03-11T21:14:05.384208+0000 mon.a (mon.0) 1887 : cluster 3 Health check update: Degraded data redundancy: 655/229008 objects degraded (0.286%), 1 pg degraded (PG_DEGRADED) 2024-03-11T21:14:05.756 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:05.778 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:05.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:05.780 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:14:05.881 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:05.914 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:05.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:05.920 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:14:05.951 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:14:06.021 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:14:06.026 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:05 smithi023 bash[21372]: cluster 2024-03-11T21:14:04.415673+0000 mgr.y (mgr.24380) 861 : cluster 0 pgmap v991: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfilling, 155 active+clean; 4.6 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 585 op/s; 655/229008 objects degraded (0.286%); 8665/229008 objects misplaced (3.784%); 22 MiB/s, 347 objects/s recovering 2024-03-11T21:14:06.026 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:05 smithi023 bash[21372]: cluster 2024-03-11T21:14:05.158566+0000 osd.6 (osd.6) 407 : cluster 0 11.3 scrub starts 2024-03-11T21:14:06.026 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:05 smithi023 bash[21372]: cluster 2024-03-11T21:14:05.384208+0000 mon.a (mon.0) 1887 : cluster 3 Health check update: Degraded data redundancy: 655/229008 objects degraded (0.286%), 1 pg degraded (PG_DEGRADED) 2024-03-11T21:14:06.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:05 smithi023 bash[29146]: cluster 2024-03-11T21:14:04.415673+0000 mgr.y (mgr.24380) 861 : cluster 0 pgmap v991: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfilling, 155 active+clean; 4.6 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 585 op/s; 655/229008 objects degraded (0.286%); 8665/229008 objects misplaced (3.784%); 22 MiB/s, 347 objects/s recovering 2024-03-11T21:14:06.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:05 smithi023 bash[29146]: cluster 2024-03-11T21:14:05.158566+0000 osd.6 (osd.6) 407 : cluster 0 11.3 scrub starts 2024-03-11T21:14:06.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:05 smithi023 bash[29146]: cluster 2024-03-11T21:14:05.384208+0000 mon.a (mon.0) 1887 : cluster 3 Health check update: Degraded data redundancy: 655/229008 objects degraded (0.286%), 1 pg degraded (PG_DEGRADED) 2024-03-11T21:14:06.029 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:14:06 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:14:06.047 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:14:06.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:06.050 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:14:06.150 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:14:06.172 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:14:06.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:06.180 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:14:06.280 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:06.310 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:06.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:06.312 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:14:06.413 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:14:06.439 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:14:06.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:06.443 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:14:06.544 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:14:06.570 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:14:06.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:06.576 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:14:06.677 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:06.715 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:06.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:06.718 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:14:06.818 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:06.847 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:06.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:06.852 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:14:06.953 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:06.976 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:06.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:06.978 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:14:07.079 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:07.098 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:07.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:07.100 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:14:07.201 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:14:07.221 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:14:07.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:07.224 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:14:07.325 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:07.345 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:07.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:07.347 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:14:07.448 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:14:07.469 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:14:07.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:07.471 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:14:07.526 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:07 smithi023 bash[21372]: audit 2024-03-11T21:14:06.020660+0000 mgr.y (mgr.24380) 862 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:14:07.593 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:07 smithi023 bash[21372]: cluster 2024-03-11T21:14:06.376426+0000 osd.6 (osd.6) 408 : cluster 0 11.3 scrub ok 2024-03-11T21:14:07.593 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:07 smithi023 bash[21372]: audit 2024-03-11T21:14:06.402889+0000 mon.a (mon.0) 1888 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:14:07.593 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:07 smithi023 bash[21372]: audit 2024-03-11T21:14:06.404748+0000 mon.a (mon.0) 1889 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:14:07.593 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:07 smithi023 bash[21372]: cluster 2024-03-11T21:14:06.416559+0000 mgr.y (mgr.24380) 863 : cluster 0 pgmap v992: 158 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfilling, 154 active+clean; 4.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 925 op/s; 655/217566 objects degraded (0.301%); 7235/217566 objects misplaced (3.325%); 30 MiB/s, 483 objects/s recovering 2024-03-11T21:14:07.593 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:07 smithi023 bash[21372]: cluster 2024-03-11T21:14:06.770344+0000 mon.a (mon.0) 1890 : cluster 0 osdmap e415: 8 total, 8 up, 7 in 2024-03-11T21:14:07.593 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:07 smithi023 bash[29146]: audit 2024-03-11T21:14:06.020660+0000 mgr.y (mgr.24380) 862 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:14:07.593 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:07 smithi023 bash[29146]: cluster 2024-03-11T21:14:06.376426+0000 osd.6 (osd.6) 408 : cluster 0 11.3 scrub ok 2024-03-11T21:14:07.593 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:07 smithi023 bash[29146]: audit 2024-03-11T21:14:06.402889+0000 mon.a (mon.0) 1888 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:14:07.594 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:07 smithi023 bash[29146]: audit 2024-03-11T21:14:06.404748+0000 mon.a (mon.0) 1889 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:14:07.594 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:07 smithi023 bash[29146]: cluster 2024-03-11T21:14:06.416559+0000 mgr.y (mgr.24380) 863 : cluster 0 pgmap v992: 158 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfilling, 154 active+clean; 4.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 925 op/s; 655/217566 objects degraded (0.301%); 7235/217566 objects misplaced (3.325%); 30 MiB/s, 483 objects/s recovering 2024-03-11T21:14:07.594 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:07 smithi023 bash[29146]: cluster 2024-03-11T21:14:06.770344+0000 mon.a (mon.0) 1890 : cluster 0 osdmap e415: 8 total, 8 up, 7 in 2024-03-11T21:14:07.594 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:14:07.603 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:07 smithi146 bash[23809]: audit 2024-03-11T21:14:06.020660+0000 mgr.y (mgr.24380) 862 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:14:07.603 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:07 smithi146 bash[23809]: cluster 2024-03-11T21:14:06.376426+0000 osd.6 (osd.6) 408 : cluster 0 11.3 scrub ok 2024-03-11T21:14:07.603 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:07 smithi146 bash[23809]: audit 2024-03-11T21:14:06.402889+0000 mon.a (mon.0) 1888 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:14:07.603 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:07 smithi146 bash[23809]: audit 2024-03-11T21:14:06.404748+0000 mon.a (mon.0) 1889 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:14:07.603 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:07 smithi146 bash[23809]: cluster 2024-03-11T21:14:06.416559+0000 mgr.y (mgr.24380) 863 : cluster 0 pgmap v992: 158 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfilling, 154 active+clean; 4.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 925 op/s; 655/217566 objects degraded (0.301%); 7235/217566 objects misplaced (3.325%); 30 MiB/s, 483 objects/s recovering 2024-03-11T21:14:07.603 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:07 smithi146 bash[23809]: cluster 2024-03-11T21:14:06.770344+0000 mon.a (mon.0) 1890 : cluster 0 osdmap e415: 8 total, 8 up, 7 in 2024-03-11T21:14:07.614 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:14:07.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:07.616 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:14:07.717 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:07.743 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:07.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:07.746 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:14:07.847 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:07.870 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:07.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:07.871 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:14:07.916 INFO:tasks.thrashosds.thrasher:in_osds: [2, 3, 4, 5, 6, 7, 1] out_osds: [0] dead_osds: [] live_osds: [0, 2, 3, 5, 6, 7, 4, 1] 2024-03-11T21:14:07.916 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-11T21:14:07.916 INFO:tasks.thrashosds.thrasher:inject_pause on osd.6 2024-03-11T21:14:07.916 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-03-11T21:14:07.916 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-03-11T21:14:07.917 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph daemon osd.6 config set filestore_inject_stall 3 2024-03-11T21:14:07.972 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:07.991 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:07.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:07.993 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:14:08.094 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:08.115 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:08.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:08.117 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:14:08.218 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:14:08.256 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:14:08.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:08.260 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:14:08.361 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:08.387 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:08.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:08.390 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:14:08.491 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:14:08.502 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:14:08 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:14:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:14:08.520 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:14:08.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:08.523 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:14:08.623 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:08.649 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:08.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:08.654 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:14:08.755 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:14:08.779 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:14:08.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:08.781 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:14:08.882 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:08.914 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:08.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:08.918 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:14:09.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:08 smithi146 bash[23809]: cluster 2024-03-11T21:14:07.769985+0000 mon.a (mon.0) 1891 : cluster 0 osdmap e416: 8 total, 8 up, 7 in 2024-03-11T21:14:09.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:08 smithi146 bash[23809]: cluster 2024-03-11T21:14:08.072598+0000 osd.6 (osd.6) 409 : cluster 0 2.14 scrub starts 2024-03-11T21:14:09.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:08 smithi146 bash[23809]: cluster 2024-03-11T21:14:08.074259+0000 osd.6 (osd.6) 410 : cluster 0 2.14 scrub ok 2024-03-11T21:14:09.006 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:08 smithi146 bash[23809]: cluster 2024-03-11T21:14:08.417069+0000 mgr.y (mgr.24380) 864 : cluster 0 pgmap v995: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 2 active+remapped+backfilling, 154 active+clean; 4.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 833 op/s; 818/214398 objects degraded (0.382%); 6917/214398 objects misplaced (3.226%); 15 MiB/s, 233 objects/s recovering 2024-03-11T21:14:09.013 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:14:09.013 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:14:09.019 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:09.026 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:08 smithi023 bash[21372]: cluster 2024-03-11T21:14:07.769985+0000 mon.a (mon.0) 1891 : cluster 0 osdmap e416: 8 total, 8 up, 7 in 2024-03-11T21:14:09.026 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:08 smithi023 bash[21372]: cluster 2024-03-11T21:14:08.072598+0000 osd.6 (osd.6) 409 : cluster 0 2.14 scrub starts 2024-03-11T21:14:09.026 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:08 smithi023 bash[21372]: cluster 2024-03-11T21:14:08.074259+0000 osd.6 (osd.6) 410 : cluster 0 2.14 scrub ok 2024-03-11T21:14:09.026 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:08 smithi023 bash[21372]: cluster 2024-03-11T21:14:08.417069+0000 mgr.y (mgr.24380) 864 : cluster 0 pgmap v995: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 2 active+remapped+backfilling, 154 active+clean; 4.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 833 op/s; 818/214398 objects degraded (0.382%); 6917/214398 objects misplaced (3.226%); 15 MiB/s, 233 objects/s recovering 2024-03-11T21:14:09.027 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:08 smithi023 bash[29146]: cluster 2024-03-11T21:14:07.769985+0000 mon.a (mon.0) 1891 : cluster 0 osdmap e416: 8 total, 8 up, 7 in 2024-03-11T21:14:09.027 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:08 smithi023 bash[29146]: cluster 2024-03-11T21:14:08.072598+0000 osd.6 (osd.6) 409 : cluster 0 2.14 scrub starts 2024-03-11T21:14:09.027 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:08 smithi023 bash[29146]: cluster 2024-03-11T21:14:08.074259+0000 osd.6 (osd.6) 410 : cluster 0 2.14 scrub ok 2024-03-11T21:14:09.027 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:08 smithi023 bash[29146]: cluster 2024-03-11T21:14:08.417069+0000 mgr.y (mgr.24380) 864 : cluster 0 pgmap v995: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 2 active+remapped+backfilling, 154 active+clean; 4.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 833 op/s; 818/214398 objects degraded (0.382%); 6917/214398 objects misplaced (3.226%); 15 MiB/s, 233 objects/s recovering 2024-03-11T21:14:09.070 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:09.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:09.072 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:14:09.173 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:09.204 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:09.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:09.208 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:14:09.309 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:09.342 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:09.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:09.345 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:14:09.446 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:09.482 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:09.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:09.485 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:14:09.586 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:09.615 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:09.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:09.620 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:14:09.721 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:09.743 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:09.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:09.747 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:14:09.848 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:09.857 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:09 smithi146 bash[23809]: cluster 2024-03-11T21:14:09.328602+0000 osd.5 (osd.5) 501 : cluster 0 3.1c scrub starts 2024-03-11T21:14:09.857 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:09 smithi146 bash[23809]: cluster 2024-03-11T21:14:09.337820+0000 osd.5 (osd.5) 502 : cluster 0 3.1c scrub ok 2024-03-11T21:14:09.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:09.880 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:09.880 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:14:09.981 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:10.007 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:10.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:10.012 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:14:10.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:09 smithi023 bash[21372]: cluster 2024-03-11T21:14:09.328602+0000 osd.5 (osd.5) 501 : cluster 0 3.1c scrub starts 2024-03-11T21:14:10.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:09 smithi023 bash[21372]: cluster 2024-03-11T21:14:09.337820+0000 osd.5 (osd.5) 502 : cluster 0 3.1c scrub ok 2024-03-11T21:14:10.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:09 smithi023 bash[29146]: cluster 2024-03-11T21:14:09.328602+0000 osd.5 (osd.5) 501 : cluster 0 3.1c scrub starts 2024-03-11T21:14:10.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:09 smithi023 bash[29146]: cluster 2024-03-11T21:14:09.337820+0000 osd.5 (osd.5) 502 : cluster 0 3.1c scrub ok 2024-03-11T21:14:10.112 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:10.135 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:10.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:10.137 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:14:10.238 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:10.273 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:10.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:10.278 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:14:10.379 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:14:10.404 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:14:10.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:10.407 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:14:10.508 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:10.538 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:10.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:10.544 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:14:10.645 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:10.673 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:10.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:10.681 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:14:10.782 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:10.809 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:10.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:10.813 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:14:10.914 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:10.939 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:10.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:10.942 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:14:11.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:10 smithi023 bash[29146]: cluster 2024-03-11T21:14:10.125475+0000 osd.6 (osd.6) 411 : cluster 0 4.5 scrub starts 2024-03-11T21:14:11.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:10 smithi023 bash[29146]: cluster 2024-03-11T21:14:10.129198+0000 osd.6 (osd.6) 412 : cluster 0 4.5 scrub ok 2024-03-11T21:14:11.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:10 smithi023 bash[29146]: cluster 2024-03-11T21:14:10.385652+0000 mon.a (mon.0) 1892 : cluster 3 Health check update: Degraded data redundancy: 818/214398 objects degraded (0.382%), 1 pg degraded (PG_DEGRADED) 2024-03-11T21:14:11.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:10 smithi023 bash[29146]: cluster 2024-03-11T21:14:10.417892+0000 mgr.y (mgr.24380) 865 : cluster 0 pgmap v996: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 1 active+remapped+backfilling, 155 active+clean; 4.2 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 968 op/s; 818/208647 objects degraded (0.392%); 5719/208647 objects misplaced (2.741%); 20 MiB/s, 323 objects/s recovering 2024-03-11T21:14:11.026 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:10 smithi023 bash[21372]: cluster 2024-03-11T21:14:10.125475+0000 osd.6 (osd.6) 411 : cluster 0 4.5 scrub starts 2024-03-11T21:14:11.026 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:10 smithi023 bash[21372]: cluster 2024-03-11T21:14:10.129198+0000 osd.6 (osd.6) 412 : cluster 0 4.5 scrub ok 2024-03-11T21:14:11.026 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:10 smithi023 bash[21372]: cluster 2024-03-11T21:14:10.385652+0000 mon.a (mon.0) 1892 : cluster 3 Health check update: Degraded data redundancy: 818/214398 objects degraded (0.382%), 1 pg degraded (PG_DEGRADED) 2024-03-11T21:14:11.026 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:10 smithi023 bash[21372]: cluster 2024-03-11T21:14:10.417892+0000 mgr.y (mgr.24380) 865 : cluster 0 pgmap v996: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 1 active+remapped+backfilling, 155 active+clean; 4.2 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 968 op/s; 818/208647 objects degraded (0.392%); 5719/208647 objects misplaced (2.741%); 20 MiB/s, 323 objects/s recovering 2024-03-11T21:14:11.043 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:11.053 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:10 smithi146 bash[23809]: cluster 2024-03-11T21:14:10.125475+0000 osd.6 (osd.6) 411 : cluster 0 4.5 scrub starts 2024-03-11T21:14:11.053 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:10 smithi146 bash[23809]: cluster 2024-03-11T21:14:10.129198+0000 osd.6 (osd.6) 412 : cluster 0 4.5 scrub ok 2024-03-11T21:14:11.053 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:10 smithi146 bash[23809]: cluster 2024-03-11T21:14:10.385652+0000 mon.a (mon.0) 1892 : cluster 3 Health check update: Degraded data redundancy: 818/214398 objects degraded (0.382%), 1 pg degraded (PG_DEGRADED) 2024-03-11T21:14:11.053 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:10 smithi146 bash[23809]: cluster 2024-03-11T21:14:10.417892+0000 mgr.y (mgr.24380) 865 : cluster 0 pgmap v996: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 1 active+remapped+backfilling, 155 active+clean; 4.2 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 968 op/s; 818/208647 objects degraded (0.392%); 5719/208647 objects misplaced (2.741%); 20 MiB/s, 323 objects/s recovering 2024-03-11T21:14:11.070 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:11.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:11.072 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:14:11.173 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:14:11.207 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:14:11.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:11.208 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:14:11.309 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:11.329 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:11.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:11.332 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:14:11.432 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:11.451 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:11.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:11.453 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:14:11.554 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:14:11.574 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:14:11.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:11.576 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:14:11.677 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:14:11.705 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:14:11.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:11.709 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:14:11.810 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:11.835 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:11.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:11.837 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:14:11.938 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:11.969 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:11.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:11.971 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:14:12.072 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:12.106 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:12.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:12.109 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:14:12.209 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:12.238 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:12.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:12.241 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:14:12.342 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:14:12.368 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:14:12.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:12.372 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:14:12.472 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:14:12.499 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:14:12.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:12.505 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:14:12.606 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:12.645 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:12.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:12.648 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:14:12.749 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:12.773 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:12.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:12.776 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:14:12.877 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:14:12.914 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:14:12.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:12.916 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:14:12.982 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:14:13.017 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:13.043 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:13.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:13.046 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:14:13.146 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:13.191 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:13.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:13.194 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:14:13.295 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:13.338 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:13.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:13.341 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:14:13.441 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:13.465 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:13.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:13.468 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:14:13.569 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:14:13.577 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:13 smithi146 bash[23809]: cluster 2024-03-11T21:14:12.416525+0000 osd.5 (osd.5) 503 : cluster 0 3.5 scrub starts 2024-03-11T21:14:13.577 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:13 smithi146 bash[23809]: cluster 2024-03-11T21:14:12.418577+0000 mgr.y (mgr.24380) 866 : cluster 0 pgmap v997: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfilling, 156 active+clean; 4.2 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 975 op/s; 818/205629 objects degraded (0.398%); 5719/205629 objects misplaced (2.781%); 19 MiB/s, 301 objects/s recovering 2024-03-11T21:14:13.578 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:13 smithi146 bash[23809]: cluster 2024-03-11T21:14:12.419187+0000 osd.5 (osd.5) 504 : cluster 0 3.5 scrub ok 2024-03-11T21:14:13.578 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:13 smithi146 bash[23809]: audit 2024-03-11T21:14:13.070775+0000 mon.a (mon.0) 1893 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:14:13.578 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:13 smithi146 bash[23809]: cluster 2024-03-11T21:14:13.147543+0000 osd.6 (osd.6) 413 : cluster 0 2.11 scrub starts 2024-03-11T21:14:13.578 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:13 smithi146 bash[23809]: cluster 2024-03-11T21:14:13.149576+0000 osd.6 (osd.6) 414 : cluster 0 2.11 scrub ok 2024-03-11T21:14:13.579 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:13 smithi023 bash[29146]: cluster 2024-03-11T21:14:12.416525+0000 osd.5 (osd.5) 503 : cluster 0 3.5 scrub starts 2024-03-11T21:14:13.580 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:13 smithi023 bash[29146]: cluster 2024-03-11T21:14:12.418577+0000 mgr.y (mgr.24380) 866 : cluster 0 pgmap v997: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfilling, 156 active+clean; 4.2 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 975 op/s; 818/205629 objects degraded (0.398%); 5719/205629 objects misplaced (2.781%); 19 MiB/s, 301 objects/s recovering 2024-03-11T21:14:13.580 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:13 smithi023 bash[29146]: cluster 2024-03-11T21:14:12.419187+0000 osd.5 (osd.5) 504 : cluster 0 3.5 scrub ok 2024-03-11T21:14:13.580 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:13 smithi023 bash[29146]: audit 2024-03-11T21:14:13.070775+0000 mon.a (mon.0) 1893 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:14:13.580 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:13 smithi023 bash[29146]: cluster 2024-03-11T21:14:13.147543+0000 osd.6 (osd.6) 413 : cluster 0 2.11 scrub starts 2024-03-11T21:14:13.580 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:13 smithi023 bash[29146]: cluster 2024-03-11T21:14:13.149576+0000 osd.6 (osd.6) 414 : cluster 0 2.11 scrub ok 2024-03-11T21:14:13.589 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:14:13.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:13.595 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:14:13.696 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:14:13.718 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:14:13.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:13.719 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:14:13.820 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:14:13.843 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:14:13.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:13.847 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:14:13.948 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:13.969 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:13.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:13.971 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:14:14.026 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:13 smithi023 bash[21372]: cluster 2024-03-11T21:14:12.416525+0000 osd.5 (osd.5) 503 : cluster 0 3.5 scrub starts 2024-03-11T21:14:14.026 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:13 smithi023 bash[21372]: cluster 2024-03-11T21:14:12.418577+0000 mgr.y (mgr.24380) 866 : cluster 0 pgmap v997: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfilling, 156 active+clean; 4.2 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 975 op/s; 818/205629 objects degraded (0.398%); 5719/205629 objects misplaced (2.781%); 19 MiB/s, 301 objects/s recovering 2024-03-11T21:14:14.026 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:13 smithi023 bash[21372]: cluster 2024-03-11T21:14:12.419187+0000 osd.5 (osd.5) 504 : cluster 0 3.5 scrub ok 2024-03-11T21:14:14.026 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:13 smithi023 bash[21372]: audit 2024-03-11T21:14:13.070775+0000 mon.a (mon.0) 1893 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:14:14.026 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:13 smithi023 bash[21372]: cluster 2024-03-11T21:14:13.147543+0000 osd.6 (osd.6) 413 : cluster 0 2.11 scrub starts 2024-03-11T21:14:14.026 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:13 smithi023 bash[21372]: cluster 2024-03-11T21:14:13.149576+0000 osd.6 (osd.6) 414 : cluster 0 2.11 scrub ok 2024-03-11T21:14:14.072 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:14.095 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:14.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:14.097 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:14:14.198 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:14.225 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:14.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:14.227 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:14:14.330 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:14:14.362 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:14:14.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:14.365 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:14:14.463 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:14 smithi023 bash[29146]: cluster 2024-03-11T21:14:13.438524+0000 osd.5 (osd.5) 505 : cluster 0 2.2 scrub starts 2024-03-11T21:14:14.464 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:14 smithi023 bash[29146]: cluster 2024-03-11T21:14:13.439917+0000 osd.5 (osd.5) 506 : cluster 0 2.2 scrub ok 2024-03-11T21:14:14.466 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:14.496 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:14.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:14.499 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:14:14.600 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:14:14.632 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:14:14.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:14.637 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:14:14.738 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:14.747 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:14 smithi146 bash[23809]: cluster 2024-03-11T21:14:13.438524+0000 osd.5 (osd.5) 505 : cluster 0 2.2 scrub starts 2024-03-11T21:14:14.747 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:14 smithi146 bash[23809]: cluster 2024-03-11T21:14:13.439917+0000 osd.5 (osd.5) 506 : cluster 0 2.2 scrub ok 2024-03-11T21:14:14.762 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:14 smithi023 bash[21372]: cluster 2024-03-11T21:14:13.438524+0000 osd.5 (osd.5) 505 : cluster 0 2.2 scrub starts 2024-03-11T21:14:14.762 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:14 smithi023 bash[21372]: cluster 2024-03-11T21:14:13.439917+0000 osd.5 (osd.5) 506 : cluster 0 2.2 scrub ok 2024-03-11T21:14:14.762 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:14.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:14.768 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:14:14.869 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:14:14.895 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:14:14.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:14.899 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:14:14.999 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:14:15.027 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:14:15.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:15.031 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:14:15.132 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:15.157 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:15.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:15.160 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:14:15.261 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:14:15.287 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:14:15.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:15.296 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:14:15.396 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:14:15.414 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:14:15.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:15.416 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:14:15.517 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:14:15.541 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:14:15.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:15.543 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:14:15.644 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:15.669 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:15.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:15.672 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:14:15.693 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:15 smithi146 bash[23809]: cluster 2024-03-11T21:14:14.419235+0000 mgr.y (mgr.24380) 867 : cluster 0 pgmap v998: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfilling, 156 active+clean; 4.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 635 op/s; 879/202314 objects degraded (0.434%); 5484/202314 objects misplaced (2.711%); 11 MiB/s, 174 objects/s recovering 2024-03-11T21:14:15.693 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:15 smithi146 bash[23809]: cluster 2024-03-11T21:14:15.386387+0000 mon.a (mon.0) 1894 : cluster 3 Health check update: Degraded data redundancy: 879/202314 objects degraded (0.434%), 1 pg degraded (PG_DEGRADED) 2024-03-11T21:14:15.773 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:14:15.798 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:14:15.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:15.800 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:14:15.901 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:15.929 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:15.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:15.931 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:14:16.026 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:15 smithi023 bash[21372]: cluster 2024-03-11T21:14:14.419235+0000 mgr.y (mgr.24380) 867 : cluster 0 pgmap v998: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfilling, 156 active+clean; 4.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 635 op/s; 879/202314 objects degraded (0.434%); 5484/202314 objects misplaced (2.711%); 11 MiB/s, 174 objects/s recovering 2024-03-11T21:14:16.026 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:15 smithi023 bash[21372]: cluster 2024-03-11T21:14:15.386387+0000 mon.a (mon.0) 1894 : cluster 3 Health check update: Degraded data redundancy: 879/202314 objects degraded (0.434%), 1 pg degraded (PG_DEGRADED) 2024-03-11T21:14:16.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:15 smithi023 bash[29146]: cluster 2024-03-11T21:14:14.419235+0000 mgr.y (mgr.24380) 867 : cluster 0 pgmap v998: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfilling, 156 active+clean; 4.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 635 op/s; 879/202314 objects degraded (0.434%); 5484/202314 objects misplaced (2.711%); 11 MiB/s, 174 objects/s recovering 2024-03-11T21:14:16.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:15 smithi023 bash[29146]: cluster 2024-03-11T21:14:15.386387+0000 mon.a (mon.0) 1894 : cluster 3 Health check update: Degraded data redundancy: 879/202314 objects degraded (0.434%), 1 pg degraded (PG_DEGRADED) 2024-03-11T21:14:16.032 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:16.062 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:16.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:16.067 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:14:16.168 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:14:16.202 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:14:16.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:16.205 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:14:16.255 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:14:16 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:14:16.306 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:14:16.345 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:14:16.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:16.348 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:14:16.449 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:16.477 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:16.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:16.480 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:14:16.581 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:16.615 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:16.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:16.617 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:14:16.717 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:16.745 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:16.751 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:16 smithi023 bash[21372]: cluster 2024-03-11T21:14:15.613256+0000 osd.6 (osd.6) 415 : cluster 0 1.6 scrub starts 2024-03-11T21:14:16.751 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:16 smithi023 bash[21372]: cluster 2024-03-11T21:14:15.621859+0000 osd.6 (osd.6) 416 : cluster 0 1.6 scrub ok 2024-03-11T21:14:16.751 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:16 smithi023 bash[29146]: cluster 2024-03-11T21:14:15.613256+0000 osd.6 (osd.6) 415 : cluster 0 1.6 scrub starts 2024-03-11T21:14:16.751 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:16 smithi023 bash[29146]: cluster 2024-03-11T21:14:15.621859+0000 osd.6 (osd.6) 416 : cluster 0 1.6 scrub ok 2024-03-11T21:14:16.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:16.752 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:14:16.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:16 smithi146 bash[23809]: cluster 2024-03-11T21:14:15.613256+0000 osd.6 (osd.6) 415 : cluster 0 1.6 scrub starts 2024-03-11T21:14:16.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:16 smithi146 bash[23809]: cluster 2024-03-11T21:14:15.621859+0000 osd.6 (osd.6) 416 : cluster 0 1.6 scrub ok 2024-03-11T21:14:16.853 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:16.875 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:16.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:16.881 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:14:16.982 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:14:17.020 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:14:17.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:17.023 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:14:17.124 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:14:17.152 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:14:17.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:17.154 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:14:17.255 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:14:17.285 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:14:17.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:17.289 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:14:17.390 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:17.421 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:17.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:17.424 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:14:17.524 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:14:17.525 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:17 smithi023 bash[21372]: cluster 2024-03-11T21:14:15.619085+0000 osd.5 (osd.5) 507 : cluster 0 1.8 scrub starts 2024-03-11T21:14:17.526 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:17 smithi023 bash[21372]: cluster 2024-03-11T21:14:15.624412+0000 osd.5 (osd.5) 508 : cluster 0 1.8 scrub ok 2024-03-11T21:14:17.526 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:17 smithi023 bash[21372]: audit 2024-03-11T21:14:16.028661+0000 mgr.y (mgr.24380) 868 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:14:17.526 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:17 smithi023 bash[21372]: cluster 2024-03-11T21:14:16.420184+0000 mgr.y (mgr.24380) 869 : cluster 0 pgmap v999: 158 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing, 1 active+recovering+undersized+degraded+remapped, 155 active+clean; 3.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 840 op/s; 879/193242 objects degraded (0.455%); 4747/193242 objects misplaced (2.457%); 13 MiB/s, 210 objects/s recovering 2024-03-11T21:14:17.526 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:17 smithi023 bash[29146]: cluster 2024-03-11T21:14:15.619085+0000 osd.5 (osd.5) 507 : cluster 0 1.8 scrub starts 2024-03-11T21:14:17.526 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:17 smithi023 bash[29146]: cluster 2024-03-11T21:14:15.624412+0000 osd.5 (osd.5) 508 : cluster 0 1.8 scrub ok 2024-03-11T21:14:17.526 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:17 smithi023 bash[29146]: audit 2024-03-11T21:14:16.028661+0000 mgr.y (mgr.24380) 868 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:14:17.526 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:17 smithi023 bash[29146]: cluster 2024-03-11T21:14:16.420184+0000 mgr.y (mgr.24380) 869 : cluster 0 pgmap v999: 158 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing, 1 active+recovering+undersized+degraded+remapped, 155 active+clean; 3.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 840 op/s; 879/193242 objects degraded (0.455%); 4747/193242 objects misplaced (2.457%); 13 MiB/s, 210 objects/s recovering 2024-03-11T21:14:17.560 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:14:17.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:17.563 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:14:17.664 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:17.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:17.693 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:17.694 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:14:17.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:17 smithi146 bash[23809]: cluster 2024-03-11T21:14:15.619085+0000 osd.5 (osd.5) 507 : cluster 0 1.8 scrub starts 2024-03-11T21:14:17.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:17 smithi146 bash[23809]: cluster 2024-03-11T21:14:15.624412+0000 osd.5 (osd.5) 508 : cluster 0 1.8 scrub ok 2024-03-11T21:14:17.756 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:17 smithi146 bash[23809]: audit 2024-03-11T21:14:16.028661+0000 mgr.y (mgr.24380) 868 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:14:17.756 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:17 smithi146 bash[23809]: cluster 2024-03-11T21:14:16.420184+0000 mgr.y (mgr.24380) 869 : cluster 0 pgmap v999: 158 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing, 1 active+recovering+undersized+degraded+remapped, 155 active+clean; 3.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 840 op/s; 879/193242 objects degraded (0.455%); 4747/193242 objects misplaced (2.457%); 13 MiB/s, 210 objects/s recovering 2024-03-11T21:14:17.798 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:17.825 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:17.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:17.828 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:14:17.929 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:17.974 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:17.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:17.977 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:14:18.078 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:18.102 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:18.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:18.104 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:14:18.205 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:18.239 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:18.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:18.243 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:14:18.344 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:14:18.364 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:14:18.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:18.366 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:14:18.458 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:14:18 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:14:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:14:18.467 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:14:18.506 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:14:18.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:18.510 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:14:18.611 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:14:18.633 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:14:18.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:18.635 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:14:18.736 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:14:18.765 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:14:18.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:18.767 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:14:18.868 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:14:18.887 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:14:18.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:18.889 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:14:18.989 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:19.026 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:19.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:19.028 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:14:19.129 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:19.146 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-03-11T21:14:19.146 INFO:teuthology.orchestra.run.smithi146.stdout: "success": "" 2024-03-11T21:14:19.146 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-03-11T21:14:19.164 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:19.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:19.174 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:14:19.275 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:14:19.312 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:14:19.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:19.317 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:14:19.418 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:14:19.448 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:14:19.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:19.455 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:14:19.556 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:14:19.587 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:14:19.591 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:19 smithi023 bash[21372]: cluster 2024-03-11T21:14:18.421118+0000 mgr.y (mgr.24380) 870 : cluster 0 pgmap v1000: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfilling, 1 active+clean+scrubbing, 155 active+clean; 3.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 961 B/s rd, 0 B/s wr, 761 op/s; 961/190086 objects degraded (0.506%); 4504/190086 objects misplaced (2.369%); 12 MiB/s, 191 objects/s recovering 2024-03-11T21:14:19.591 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:19 smithi023 bash[29146]: cluster 2024-03-11T21:14:18.421118+0000 mgr.y (mgr.24380) 870 : cluster 0 pgmap v1000: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfilling, 1 active+clean+scrubbing, 155 active+clean; 3.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 961 B/s rd, 0 B/s wr, 761 op/s; 961/190086 objects degraded (0.506%); 4504/190086 objects misplaced (2.369%); 12 MiB/s, 191 objects/s recovering 2024-03-11T21:14:19.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:19.592 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:14:19.693 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:19.718 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:19.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:19.721 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:14:19.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:19 smithi146 bash[23809]: cluster 2024-03-11T21:14:18.421118+0000 mgr.y (mgr.24380) 870 : cluster 0 pgmap v1000: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfilling, 1 active+clean+scrubbing, 155 active+clean; 3.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 961 B/s rd, 0 B/s wr, 761 op/s; 961/190086 objects degraded (0.506%); 4504/190086 objects misplaced (2.369%); 12 MiB/s, 191 objects/s recovering 2024-03-11T21:14:19.823 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:14:19.844 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:14:19.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:19.846 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:14:19.947 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:14:19.972 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:14:19.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:19.977 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:14:20.078 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:20.110 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:20.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:20.112 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:14:20.213 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:14:20.238 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:14:20.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:20.242 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:14:20.343 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:20.368 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:20.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:20.373 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:14:20.474 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:20.504 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:20.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:20.507 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:14:20.608 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:14:20.619 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:20 smithi146 bash[23809]: cluster 2024-03-11T21:14:19.436373+0000 osd.5 (osd.5) 509 : cluster 0 2.18 scrub starts 2024-03-11T21:14:20.619 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:20 smithi146 bash[23809]: cluster 2024-03-11T21:14:19.440385+0000 osd.5 (osd.5) 510 : cluster 0 2.18 scrub ok 2024-03-11T21:14:20.619 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:20 smithi146 bash[23809]: cluster 2024-03-11T21:14:20.387484+0000 mon.a (mon.0) 1895 : cluster 3 Health check update: Degraded data redundancy: 961/190086 objects degraded (0.506%), 1 pg degraded (PG_DEGRADED) 2024-03-11T21:14:20.656 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:14:20.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:20.663 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:14:20.764 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:14:20.774 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:20 smithi023 bash[21372]: cluster 2024-03-11T21:14:19.436373+0000 osd.5 (osd.5) 509 : cluster 0 2.18 scrub starts 2024-03-11T21:14:20.774 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:20 smithi023 bash[21372]: cluster 2024-03-11T21:14:19.440385+0000 osd.5 (osd.5) 510 : cluster 0 2.18 scrub ok 2024-03-11T21:14:20.774 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:20 smithi023 bash[21372]: cluster 2024-03-11T21:14:20.387484+0000 mon.a (mon.0) 1895 : cluster 3 Health check update: Degraded data redundancy: 961/190086 objects degraded (0.506%), 1 pg degraded (PG_DEGRADED) 2024-03-11T21:14:20.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:20 smithi023 bash[29146]: cluster 2024-03-11T21:14:19.436373+0000 osd.5 (osd.5) 509 : cluster 0 2.18 scrub starts 2024-03-11T21:14:20.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:20 smithi023 bash[29146]: cluster 2024-03-11T21:14:19.440385+0000 osd.5 (osd.5) 510 : cluster 0 2.18 scrub ok 2024-03-11T21:14:20.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:20 smithi023 bash[29146]: cluster 2024-03-11T21:14:20.387484+0000 mon.a (mon.0) 1895 : cluster 3 Health check update: Degraded data redundancy: 961/190086 objects degraded (0.506%), 1 pg degraded (PG_DEGRADED) 2024-03-11T21:14:20.810 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:14:20.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:20.815 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:14:20.916 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:20.935 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:20.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:20.937 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:14:21.038 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:21.065 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:21.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:21.068 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:14:21.168 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:14:21.197 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:14:21.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:21.202 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:14:21.303 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:21.350 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:21.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:21.357 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:14:21.458 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:21.498 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:21.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:21.501 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:14:21.602 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:14:21.631 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:14:21.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:21.633 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:14:21.734 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:21.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:21 smithi146 bash[23809]: cluster 2024-03-11T21:14:20.422030+0000 mgr.y (mgr.24380) 871 : cluster 0 pgmap v1001: 158 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 155 active+clean; 3.7 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 870 op/s; 961/183105 objects degraded (0.525%); 3963/183105 objects misplaced (2.164%); 13 MiB/s, 209 objects/s recovering 2024-03-11T21:14:21.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:21 smithi146 bash[23809]: audit 2024-03-11T21:14:20.985304+0000 mon.a (mon.0) 1896 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:14:21.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:21 smithi146 bash[23809]: audit 2024-03-11T21:14:20.996449+0000 mon.a (mon.0) 1897 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:14:21.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:21 smithi146 bash[23809]: cluster 2024-03-11T21:14:21.299669+0000 osd.6 (osd.6) 417 : cluster 0 11.d scrub starts 2024-03-11T21:14:21.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:21 smithi146 bash[23809]: audit 2024-03-11T21:14:21.432643+0000 mon.a (mon.0) 1898 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:14:21.766 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:21.767 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:21 smithi023 bash[21372]: cluster 2024-03-11T21:14:20.422030+0000 mgr.y (mgr.24380) 871 : cluster 0 pgmap v1001: 158 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 155 active+clean; 3.7 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 870 op/s; 961/183105 objects degraded (0.525%); 3963/183105 objects misplaced (2.164%); 13 MiB/s, 209 objects/s recovering 2024-03-11T21:14:21.767 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:21 smithi023 bash[21372]: audit 2024-03-11T21:14:20.985304+0000 mon.a (mon.0) 1896 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:14:21.767 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:21 smithi023 bash[21372]: audit 2024-03-11T21:14:20.996449+0000 mon.a (mon.0) 1897 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:14:21.767 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:21 smithi023 bash[21372]: cluster 2024-03-11T21:14:21.299669+0000 osd.6 (osd.6) 417 : cluster 0 11.d scrub starts 2024-03-11T21:14:21.767 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:21 smithi023 bash[21372]: audit 2024-03-11T21:14:21.432643+0000 mon.a (mon.0) 1898 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:14:21.767 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:21 smithi023 bash[29146]: cluster 2024-03-11T21:14:20.422030+0000 mgr.y (mgr.24380) 871 : cluster 0 pgmap v1001: 158 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 155 active+clean; 3.7 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 870 op/s; 961/183105 objects degraded (0.525%); 3963/183105 objects misplaced (2.164%); 13 MiB/s, 209 objects/s recovering 2024-03-11T21:14:21.767 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:21 smithi023 bash[29146]: audit 2024-03-11T21:14:20.985304+0000 mon.a (mon.0) 1896 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:14:21.767 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:21 smithi023 bash[29146]: audit 2024-03-11T21:14:20.996449+0000 mon.a (mon.0) 1897 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:14:21.767 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:21 smithi023 bash[29146]: cluster 2024-03-11T21:14:21.299669+0000 osd.6 (osd.6) 417 : cluster 0 11.d scrub starts 2024-03-11T21:14:21.767 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:21 smithi023 bash[29146]: audit 2024-03-11T21:14:21.432643+0000 mon.a (mon.0) 1898 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:14:21.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:21.768 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:14:21.869 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:21.886 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:21.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:21.891 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:14:21.992 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:14:22.027 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:14:22.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:22.034 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:14:22.135 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:14:22.174 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:14:22.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:22.179 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:14:22.280 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:22.305 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:22.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:22.308 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:14:22.410 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:22.439 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:22.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:22.441 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:14:22.542 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:22.551 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:22 smithi146 bash[23809]: cluster 2024-03-11T21:14:22.339711+0000 osd.6 (osd.6) 418 : cluster 0 4.e scrub starts 2024-03-11T21:14:22.551 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:22 smithi146 bash[23809]: cluster 2024-03-11T21:14:22.345544+0000 osd.6 (osd.6) 419 : cluster 0 4.e scrub ok 2024-03-11T21:14:22.576 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:22.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:22.582 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:14:22.602 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:22 smithi023 bash[21372]: cluster 2024-03-11T21:14:22.339711+0000 osd.6 (osd.6) 418 : cluster 0 4.e scrub starts 2024-03-11T21:14:22.602 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:22 smithi023 bash[21372]: cluster 2024-03-11T21:14:22.345544+0000 osd.6 (osd.6) 419 : cluster 0 4.e scrub ok 2024-03-11T21:14:22.683 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:22.705 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:22.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:22.708 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:14:22.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:22 smithi023 bash[29146]: cluster 2024-03-11T21:14:22.339711+0000 osd.6 (osd.6) 418 : cluster 0 4.e scrub starts 2024-03-11T21:14:22.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:22 smithi023 bash[29146]: cluster 2024-03-11T21:14:22.345544+0000 osd.6 (osd.6) 419 : cluster 0 4.e scrub ok 2024-03-11T21:14:22.808 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:22.841 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:22.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:22.844 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:14:22.945 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:14:22.966 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:14:22.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:22.969 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:14:23.070 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:23.095 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:23.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:23.097 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:14:23.198 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:23.228 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:23.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:23.234 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:14:23.335 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:14:23.364 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:14:23.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:23.366 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:14:23.467 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:14:23.498 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:14:23.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:23.501 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:14:23.602 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:14:23.612 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:23 smithi146 bash[23809]: cluster 2024-03-11T21:14:22.371230+0000 osd.5 (osd.5) 511 : cluster 0 3.16 scrub starts 2024-03-11T21:14:23.613 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:23 smithi146 bash[23809]: cluster 2024-03-11T21:14:22.394389+0000 osd.5 (osd.5) 512 : cluster 0 3.16 scrub ok 2024-03-11T21:14:23.613 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:23 smithi146 bash[23809]: cluster 2024-03-11T21:14:22.422700+0000 mgr.y (mgr.24380) 872 : cluster 0 pgmap v1002: 158 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 156 active+clean; 3.6 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 804 op/s; 961/179688 objects degraded (0.535%); 3963/179688 objects misplaced (2.205%); 7.5 MiB/s, 120 objects/s recovering 2024-03-11T21:14:23.613 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:23 smithi146 bash[23809]: cluster 2024-03-11T21:14:22.505774+0000 osd.6 (osd.6) 420 : cluster 0 11.d scrub ok 2024-03-11T21:14:23.631 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:14:23.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:23.633 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:14:23.734 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:23.767 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:23.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:23.771 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:14:23.777 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:23 smithi023 bash[21372]: cluster 2024-03-11T21:14:22.371230+0000 osd.5 (osd.5) 511 : cluster 0 3.16 scrub starts 2024-03-11T21:14:23.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:23 smithi023 bash[21372]: cluster 2024-03-11T21:14:22.394389+0000 osd.5 (osd.5) 512 : cluster 0 3.16 scrub ok 2024-03-11T21:14:23.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:23 smithi023 bash[21372]: cluster 2024-03-11T21:14:22.422700+0000 mgr.y (mgr.24380) 872 : cluster 0 pgmap v1002: 158 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 156 active+clean; 3.6 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 804 op/s; 961/179688 objects degraded (0.535%); 3963/179688 objects misplaced (2.205%); 7.5 MiB/s, 120 objects/s recovering 2024-03-11T21:14:23.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:23 smithi023 bash[21372]: cluster 2024-03-11T21:14:22.505774+0000 osd.6 (osd.6) 420 : cluster 0 11.d scrub ok 2024-03-11T21:14:23.778 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:23 smithi023 bash[29146]: cluster 2024-03-11T21:14:22.371230+0000 osd.5 (osd.5) 511 : cluster 0 3.16 scrub starts 2024-03-11T21:14:23.778 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:23 smithi023 bash[29146]: cluster 2024-03-11T21:14:22.394389+0000 osd.5 (osd.5) 512 : cluster 0 3.16 scrub ok 2024-03-11T21:14:23.778 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:23 smithi023 bash[29146]: cluster 2024-03-11T21:14:22.422700+0000 mgr.y (mgr.24380) 872 : cluster 0 pgmap v1002: 158 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 156 active+clean; 3.6 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 804 op/s; 961/179688 objects degraded (0.535%); 3963/179688 objects misplaced (2.205%); 7.5 MiB/s, 120 objects/s recovering 2024-03-11T21:14:23.778 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:23 smithi023 bash[29146]: cluster 2024-03-11T21:14:22.505774+0000 osd.6 (osd.6) 420 : cluster 0 11.d scrub ok 2024-03-11T21:14:23.872 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:14:23.902 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:14:23.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:23.908 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:14:24.009 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:14:24.047 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:14:24.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:24.053 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:14:24.154 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:24.179 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:24.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:24.182 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:14:24.283 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:24.316 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:24.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:24.318 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:14:24.419 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:14:24.441 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:14:24.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:24.444 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:14:24.545 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:14:24.583 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:14:24.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:24.585 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:14:24.686 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:24.696 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:24 smithi023 bash[21372]: cluster 2024-03-11T21:14:24.330789+0000 osd.6 (osd.6) 421 : cluster 0 4.1f scrub starts 2024-03-11T21:14:24.696 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:24 smithi023 bash[21372]: cluster 2024-03-11T21:14:24.332708+0000 osd.6 (osd.6) 422 : cluster 0 4.1f scrub ok 2024-03-11T21:14:24.697 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:24 smithi023 bash[29146]: cluster 2024-03-11T21:14:24.330789+0000 osd.6 (osd.6) 421 : cluster 0 4.1f scrub starts 2024-03-11T21:14:24.697 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:24 smithi023 bash[29146]: cluster 2024-03-11T21:14:24.332708+0000 osd.6 (osd.6) 422 : cluster 0 4.1f scrub ok 2024-03-11T21:14:24.710 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:24.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:24.714 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:14:24.816 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:14:24.845 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:14:24.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:24.849 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:14:24.950 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:14:24.962 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:24 smithi146 bash[23809]: cluster 2024-03-11T21:14:24.330789+0000 osd.6 (osd.6) 421 : cluster 0 4.1f scrub starts 2024-03-11T21:14:24.962 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:24 smithi146 bash[23809]: cluster 2024-03-11T21:14:24.332708+0000 osd.6 (osd.6) 422 : cluster 0 4.1f scrub ok 2024-03-11T21:14:24.981 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:14:24.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:24.988 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:14:25.089 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:14:25.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:25.123 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:14:25.124 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:14:25.224 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:25.246 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:25.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:25.249 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:14:25.343 INFO:tasks.thrashosds.thrasher:in_osds: [2, 3, 4, 5, 6, 7, 1] out_osds: [0] dead_osds: [] live_osds: [0, 2, 3, 5, 6, 7, 4, 1] 2024-03-11T21:14:25.344 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-11T21:14:25.344 INFO:tasks.thrashosds.thrasher:Adding osd 0 2024-03-11T21:14:25.344 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd in 0 2024-03-11T21:14:25.350 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:14:25.416 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:14:25.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:25.422 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:14:25.522 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:25.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:25.551 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:25.551 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:14:25.619 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:25 smithi023 bash[21372]: cluster 2024-03-11T21:14:24.424462+0000 mgr.y (mgr.24380) 873 : cluster 0 pgmap v1003: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfilling, 156 active+clean; 3.6 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 813 op/s; 1086/176349 objects degraded (0.616%); 3689/176349 objects misplaced (2.092%); 8.3 MiB/s, 132 objects/s recovering 2024-03-11T21:14:25.620 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:25 smithi023 bash[21372]: cluster 2024-03-11T21:14:25.391540+0000 mon.a (mon.0) 1899 : cluster 3 Health check update: Degraded data redundancy: 1086/176349 objects degraded (0.616%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-11T21:14:25.652 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:25.689 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:25.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:25.695 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:14:25.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:25 smithi023 bash[29146]: cluster 2024-03-11T21:14:24.424462+0000 mgr.y (mgr.24380) 873 : cluster 0 pgmap v1003: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfilling, 156 active+clean; 3.6 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 813 op/s; 1086/176349 objects degraded (0.616%); 3689/176349 objects misplaced (2.092%); 8.3 MiB/s, 132 objects/s recovering 2024-03-11T21:14:25.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:25 smithi023 bash[29146]: cluster 2024-03-11T21:14:25.391540+0000 mon.a (mon.0) 1899 : cluster 3 Health check update: Degraded data redundancy: 1086/176349 objects degraded (0.616%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-11T21:14:25.796 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:25.815 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:25.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:25.817 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:14:25.918 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:25.939 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:25.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:25.941 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:14:26.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:25 smithi146 bash[23809]: cluster 2024-03-11T21:14:24.424462+0000 mgr.y (mgr.24380) 873 : cluster 0 pgmap v1003: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfilling, 156 active+clean; 3.6 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 813 op/s; 1086/176349 objects degraded (0.616%); 3689/176349 objects misplaced (2.092%); 8.3 MiB/s, 132 objects/s recovering 2024-03-11T21:14:26.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:25 smithi146 bash[23809]: cluster 2024-03-11T21:14:25.391540+0000 mon.a (mon.0) 1899 : cluster 3 Health check update: Degraded data redundancy: 1086/176349 objects degraded (0.616%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-11T21:14:26.042 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:26.072 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:26.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:26.074 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:14:26.174 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:26.200 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:26.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:26.206 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:14:26.307 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:14:26.316 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:14:26 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:14:26.346 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:14:26.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:26.349 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:14:26.450 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:26.480 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:26.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:26.484 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:14:26.584 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:14:26.609 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:14:26.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:26.614 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:14:26.715 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:26.738 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:26.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:26.741 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:14:26.841 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:26.849 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:26 smithi146 bash[23809]: cluster 2024-03-11T21:14:26.356599+0000 osd.5 (osd.5) 513 : cluster 0 2.3 scrub starts 2024-03-11T21:14:26.849 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:26 smithi146 bash[23809]: cluster 2024-03-11T21:14:26.360734+0000 osd.5 (osd.5) 514 : cluster 0 2.3 scrub ok 2024-03-11T21:14:26.861 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:26.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:26.863 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:14:26.964 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:26.982 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:26.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:26.983 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:14:27.026 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:26 smithi023 bash[21372]: cluster 2024-03-11T21:14:26.356599+0000 osd.5 (osd.5) 513 : cluster 0 2.3 scrub starts 2024-03-11T21:14:27.026 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:26 smithi023 bash[21372]: cluster 2024-03-11T21:14:26.360734+0000 osd.5 (osd.5) 514 : cluster 0 2.3 scrub ok 2024-03-11T21:14:27.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:26 smithi023 bash[29146]: cluster 2024-03-11T21:14:26.356599+0000 osd.5 (osd.5) 513 : cluster 0 2.3 scrub starts 2024-03-11T21:14:27.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:26 smithi023 bash[29146]: cluster 2024-03-11T21:14:26.360734+0000 osd.5 (osd.5) 514 : cluster 0 2.3 scrub ok 2024-03-11T21:14:27.084 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:27.107 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:27.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:27.109 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:14:27.210 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:14:27.240 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:14:27.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:27.242 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:14:27.343 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:27.368 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:27.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:27.371 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:14:27.471 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:14:27.496 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:14:27.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:27.498 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:14:27.599 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:27.610 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:27 smithi146 bash[23809]: cluster 2024-03-11T21:14:25.978376+0000 osd.7 (osd.7) 527 : cluster 0 6.1c scrub starts 2024-03-11T21:14:27.610 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:27 smithi146 bash[23809]: cluster 2024-03-11T21:14:25.984708+0000 osd.7 (osd.7) 528 : cluster 0 6.1c scrub ok 2024-03-11T21:14:27.610 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:27 smithi146 bash[23809]: audit 2024-03-11T21:14:26.038994+0000 mgr.y (mgr.24380) 874 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:14:27.610 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:27 smithi146 bash[23809]: cluster 2024-03-11T21:14:26.425449+0000 mgr.y (mgr.24380) 875 : cluster 0 pgmap v1004: 158 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 156 active+clean; 3.4 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.02k op/s; 1086/165441 objects degraded (0.656%); 3145/165441 objects misplaced (1.901%); 10 MiB/s, 160 objects/s recovering 2024-03-11T21:14:27.611 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:27 smithi146 bash[23809]: cluster 2024-03-11T21:14:26.596485+0000 mon.a (mon.0) 1900 : cluster 0 osdmap e417: 8 total, 8 up, 7 in 2024-03-11T21:14:27.623 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:27.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:27.625 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:14:27.726 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:27.756 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:27.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:27.759 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:14:27.860 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:14:27.870 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:27 smithi023 bash[21372]: cluster 2024-03-11T21:14:25.978376+0000 osd.7 (osd.7) 527 : cluster 0 6.1c scrub starts 2024-03-11T21:14:27.870 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:27 smithi023 bash[21372]: cluster 2024-03-11T21:14:25.984708+0000 osd.7 (osd.7) 528 : cluster 0 6.1c scrub ok 2024-03-11T21:14:27.870 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:27 smithi023 bash[21372]: audit 2024-03-11T21:14:26.038994+0000 mgr.y (mgr.24380) 874 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:14:27.870 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:27 smithi023 bash[21372]: cluster 2024-03-11T21:14:26.425449+0000 mgr.y (mgr.24380) 875 : cluster 0 pgmap v1004: 158 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 156 active+clean; 3.4 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.02k op/s; 1086/165441 objects degraded (0.656%); 3145/165441 objects misplaced (1.901%); 10 MiB/s, 160 objects/s recovering 2024-03-11T21:14:27.870 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:27 smithi023 bash[21372]: cluster 2024-03-11T21:14:26.596485+0000 mon.a (mon.0) 1900 : cluster 0 osdmap e417: 8 total, 8 up, 7 in 2024-03-11T21:14:27.871 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:27 smithi023 bash[29146]: cluster 2024-03-11T21:14:25.978376+0000 osd.7 (osd.7) 527 : cluster 0 6.1c scrub starts 2024-03-11T21:14:27.871 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:27 smithi023 bash[29146]: cluster 2024-03-11T21:14:25.984708+0000 osd.7 (osd.7) 528 : cluster 0 6.1c scrub ok 2024-03-11T21:14:27.871 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:27 smithi023 bash[29146]: audit 2024-03-11T21:14:26.038994+0000 mgr.y (mgr.24380) 874 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:14:27.871 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:27 smithi023 bash[29146]: cluster 2024-03-11T21:14:26.425449+0000 mgr.y (mgr.24380) 875 : cluster 0 pgmap v1004: 158 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 156 active+clean; 3.4 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.02k op/s; 1086/165441 objects degraded (0.656%); 3145/165441 objects misplaced (1.901%); 10 MiB/s, 160 objects/s recovering 2024-03-11T21:14:27.871 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:27 smithi023 bash[29146]: cluster 2024-03-11T21:14:26.596485+0000 mon.a (mon.0) 1900 : cluster 0 osdmap e417: 8 total, 8 up, 7 in 2024-03-11T21:14:27.885 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:14:27.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:27.888 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:14:27.989 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:14:28.013 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:14:28.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:28.019 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:14:28.119 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:28.144 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:28.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:28.147 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:14:28.247 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:28.302 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:28.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:28.303 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:14:28.404 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:28.439 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:28.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:28.445 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:14:28.525 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:14:28 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:14:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:14:28.546 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:14:28.579 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:14:28.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:28.581 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:14:28.682 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:14:28.691 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:28 smithi146 bash[23809]: cluster 2024-03-11T21:14:27.355999+0000 osd.3 (osd.3) 473 : cluster 0 2.1e scrub starts 2024-03-11T21:14:28.691 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:28 smithi146 bash[23809]: cluster 2024-03-11T21:14:27.357344+0000 osd.3 (osd.3) 474 : cluster 0 2.1e scrub ok 2024-03-11T21:14:28.691 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:28 smithi146 bash[23809]: cluster 2024-03-11T21:14:27.601610+0000 mon.a (mon.0) 1901 : cluster 0 osdmap e418: 8 total, 8 up, 7 in 2024-03-11T21:14:28.691 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:28 smithi146 bash[23809]: cluster 2024-03-11T21:14:28.372076+0000 osd.5 (osd.5) 515 : cluster 0 6.a scrub starts 2024-03-11T21:14:28.691 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:28 smithi146 bash[23809]: cluster 2024-03-11T21:14:28.376239+0000 osd.5 (osd.5) 516 : cluster 0 6.a scrub ok 2024-03-11T21:14:28.712 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:14:28.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:28.715 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:14:28.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:28 smithi023 bash[21372]: cluster 2024-03-11T21:14:27.355999+0000 osd.3 (osd.3) 473 : cluster 0 2.1e scrub starts 2024-03-11T21:14:28.776 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:28 smithi023 bash[21372]: cluster 2024-03-11T21:14:27.357344+0000 osd.3 (osd.3) 474 : cluster 0 2.1e scrub ok 2024-03-11T21:14:28.776 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:28 smithi023 bash[21372]: cluster 2024-03-11T21:14:27.601610+0000 mon.a (mon.0) 1901 : cluster 0 osdmap e418: 8 total, 8 up, 7 in 2024-03-11T21:14:28.776 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:28 smithi023 bash[21372]: cluster 2024-03-11T21:14:28.372076+0000 osd.5 (osd.5) 515 : cluster 0 6.a scrub starts 2024-03-11T21:14:28.776 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:28 smithi023 bash[21372]: cluster 2024-03-11T21:14:28.376239+0000 osd.5 (osd.5) 516 : cluster 0 6.a scrub ok 2024-03-11T21:14:28.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:28 smithi023 bash[29146]: cluster 2024-03-11T21:14:27.355999+0000 osd.3 (osd.3) 473 : cluster 0 2.1e scrub starts 2024-03-11T21:14:28.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:28 smithi023 bash[29146]: cluster 2024-03-11T21:14:27.357344+0000 osd.3 (osd.3) 474 : cluster 0 2.1e scrub ok 2024-03-11T21:14:28.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:28 smithi023 bash[29146]: cluster 2024-03-11T21:14:27.601610+0000 mon.a (mon.0) 1901 : cluster 0 osdmap e418: 8 total, 8 up, 7 in 2024-03-11T21:14:28.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:28 smithi023 bash[29146]: cluster 2024-03-11T21:14:28.372076+0000 osd.5 (osd.5) 515 : cluster 0 6.a scrub starts 2024-03-11T21:14:28.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:28 smithi023 bash[29146]: cluster 2024-03-11T21:14:28.376239+0000 osd.5 (osd.5) 516 : cluster 0 6.a scrub ok 2024-03-11T21:14:28.816 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:28.842 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:28.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:28.844 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:14:28.944 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:28.970 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:28.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:28.972 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:14:29.073 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:29.097 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:29.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:29.100 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:14:29.201 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:14:29.228 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:14:29.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:29.232 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:14:29.303 INFO:teuthology.orchestra.run.smithi023.stdout:ERROR: (22) Invalid argument 2024-03-11T21:14:29.303 INFO:teuthology.orchestra.run.smithi023.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:14:29.333 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:29.373 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:29.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:29.376 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:14:29.476 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:14:29.507 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:14:29.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:29.510 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:14:29.610 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:29.629 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:29.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:29.631 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:14:29.665 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:29 smithi023 bash[21372]: cluster 2024-03-11T21:14:28.008582+0000 osd.7 (osd.7) 529 : cluster 0 3.14 scrub starts 2024-03-11T21:14:29.665 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:29 smithi023 bash[21372]: cluster 2024-03-11T21:14:28.012398+0000 osd.7 (osd.7) 530 : cluster 0 3.14 scrub ok 2024-03-11T21:14:29.665 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:29 smithi023 bash[21372]: cluster 2024-03-11T21:14:28.377179+0000 osd.3 (osd.3) 475 : cluster 0 6.6 scrub starts 2024-03-11T21:14:29.666 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:29 smithi023 bash[21372]: cluster 2024-03-11T21:14:28.402489+0000 osd.3 (osd.3) 476 : cluster 0 6.6 scrub ok 2024-03-11T21:14:29.666 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:29 smithi023 bash[21372]: cluster 2024-03-11T21:14:28.426172+0000 mgr.y (mgr.24380) 876 : cluster 0 pgmap v1007: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfilling, 156 active+clean; 3.3 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 861 op/s; 1196/162432 objects degraded (0.736%); 2855/162432 objects misplaced (1.758%); 6.5 MiB/s, 104 objects/s recovering 2024-03-11T21:14:29.731 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:14:29.741 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:29 smithi146 bash[23809]: cluster 2024-03-11T21:14:28.008582+0000 osd.7 (osd.7) 529 : cluster 0 3.14 scrub starts 2024-03-11T21:14:29.741 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:29 smithi146 bash[23809]: cluster 2024-03-11T21:14:28.012398+0000 osd.7 (osd.7) 530 : cluster 0 3.14 scrub ok 2024-03-11T21:14:29.741 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:29 smithi146 bash[23809]: cluster 2024-03-11T21:14:28.377179+0000 osd.3 (osd.3) 475 : cluster 0 6.6 scrub starts 2024-03-11T21:14:29.741 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:29 smithi146 bash[23809]: cluster 2024-03-11T21:14:28.402489+0000 osd.3 (osd.3) 476 : cluster 0 6.6 scrub ok 2024-03-11T21:14:29.741 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:29 smithi146 bash[23809]: cluster 2024-03-11T21:14:28.426172+0000 mgr.y (mgr.24380) 876 : cluster 0 pgmap v1007: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfilling, 156 active+clean; 3.3 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 861 op/s; 1196/162432 objects degraded (0.736%); 2855/162432 objects misplaced (1.758%); 6.5 MiB/s, 104 objects/s recovering 2024-03-11T21:14:29.755 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:14:29.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:29.757 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:14:29.858 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:29.886 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:29.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:29.889 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:14:29.990 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:30.028 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:29 smithi023 bash[29146]: cluster 2024-03-11T21:14:28.008582+0000 osd.7 (osd.7) 529 : cluster 0 3.14 scrub starts 2024-03-11T21:14:30.028 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:29 smithi023 bash[29146]: cluster 2024-03-11T21:14:28.012398+0000 osd.7 (osd.7) 530 : cluster 0 3.14 scrub ok 2024-03-11T21:14:30.028 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:29 smithi023 bash[29146]: cluster 2024-03-11T21:14:28.377179+0000 osd.3 (osd.3) 475 : cluster 0 6.6 scrub starts 2024-03-11T21:14:30.028 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:29 smithi023 bash[29146]: cluster 2024-03-11T21:14:28.402489+0000 osd.3 (osd.3) 476 : cluster 0 6.6 scrub ok 2024-03-11T21:14:30.028 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:29 smithi023 bash[29146]: cluster 2024-03-11T21:14:28.426172+0000 mgr.y (mgr.24380) 876 : cluster 0 pgmap v1007: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfilling, 156 active+clean; 3.3 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 861 op/s; 1196/162432 objects degraded (0.736%); 2855/162432 objects misplaced (1.758%); 6.5 MiB/s, 104 objects/s recovering 2024-03-11T21:14:30.034 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:30.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:30.037 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:14:30.138 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:30.160 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:30.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:30.164 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:14:30.266 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:14:30.287 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:14:30.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:30.289 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:14:30.390 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:30.422 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:30.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:30.425 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:14:30.527 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:30.556 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:30.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:30.563 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:14:30.664 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:14:30.683 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:14:30.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:30.685 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:14:30.785 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:14:30.803 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:14:30.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:30.804 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:14:30.905 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:30.917 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:30 smithi146 bash[23809]: cluster 2024-03-11T21:14:29.023501+0000 osd.7 (osd.7) 531 : cluster 0 6.5 scrub starts 2024-03-11T21:14:30.918 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:30 smithi146 bash[23809]: cluster 2024-03-11T21:14:29.105417+0000 osd.7 (osd.7) 532 : cluster 0 6.5 scrub ok 2024-03-11T21:14:30.918 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:30 smithi146 bash[23809]: cluster 2024-03-11T21:14:29.226898+0000 osd.7 (osd.7) 533 : cluster 0 purged_snaps scrub starts 2024-03-11T21:14:30.918 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:30 smithi146 bash[23809]: cluster 2024-03-11T21:14:29.226968+0000 osd.7 (osd.7) 534 : cluster 0 purged_snaps scrub ok 2024-03-11T21:14:30.918 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:30 smithi146 bash[23809]: cluster 2024-03-11T21:14:29.628580+0000 mon.a (mon.0) 1902 : cluster 0 osdmap e419: 8 total, 8 up, 7 in 2024-03-11T21:14:30.918 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:30 smithi146 bash[23809]: audit 2024-03-11T21:14:30.059317+0000 mon.a (mon.0) 1903 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:14:30.918 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:30 smithi146 bash[23809]: audit 2024-03-11T21:14:30.069072+0000 mon.a (mon.0) 1904 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:14:30.918 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:30 smithi146 bash[23809]: cluster 2024-03-11T21:14:30.369526+0000 osd.5 (osd.5) 517 : cluster 0 2.16 scrub starts 2024-03-11T21:14:30.918 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:30 smithi146 bash[23809]: cluster 2024-03-11T21:14:30.370875+0000 osd.5 (osd.5) 518 : cluster 0 2.16 scrub ok 2024-03-11T21:14:30.918 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:30 smithi146 bash[23809]: cluster 2024-03-11T21:14:30.392415+0000 mon.a (mon.0) 1905 : cluster 3 Health check update: Degraded data redundancy: 1196/162432 objects degraded (0.736%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-11T21:14:30.935 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:30.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:30.941 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:14:31.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:30 smithi023 bash[29146]: cluster 2024-03-11T21:14:29.023501+0000 osd.7 (osd.7) 531 : cluster 0 6.5 scrub starts 2024-03-11T21:14:31.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:30 smithi023 bash[29146]: cluster 2024-03-11T21:14:29.105417+0000 osd.7 (osd.7) 532 : cluster 0 6.5 scrub ok 2024-03-11T21:14:31.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:30 smithi023 bash[29146]: cluster 2024-03-11T21:14:29.226898+0000 osd.7 (osd.7) 533 : cluster 0 purged_snaps scrub starts 2024-03-11T21:14:31.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:30 smithi023 bash[29146]: cluster 2024-03-11T21:14:29.226968+0000 osd.7 (osd.7) 534 : cluster 0 purged_snaps scrub ok 2024-03-11T21:14:31.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:30 smithi023 bash[29146]: cluster 2024-03-11T21:14:29.628580+0000 mon.a (mon.0) 1902 : cluster 0 osdmap e419: 8 total, 8 up, 7 in 2024-03-11T21:14:31.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:30 smithi023 bash[29146]: audit 2024-03-11T21:14:30.059317+0000 mon.a (mon.0) 1903 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:14:31.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:30 smithi023 bash[29146]: audit 2024-03-11T21:14:30.069072+0000 mon.a (mon.0) 1904 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:14:31.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:30 smithi023 bash[29146]: cluster 2024-03-11T21:14:30.369526+0000 osd.5 (osd.5) 517 : cluster 0 2.16 scrub starts 2024-03-11T21:14:31.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:30 smithi023 bash[29146]: cluster 2024-03-11T21:14:30.370875+0000 osd.5 (osd.5) 518 : cluster 0 2.16 scrub ok 2024-03-11T21:14:31.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:30 smithi023 bash[29146]: cluster 2024-03-11T21:14:30.392415+0000 mon.a (mon.0) 1905 : cluster 3 Health check update: Degraded data redundancy: 1196/162432 objects degraded (0.736%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-11T21:14:31.027 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:30 smithi023 bash[21372]: cluster 2024-03-11T21:14:29.023501+0000 osd.7 (osd.7) 531 : cluster 0 6.5 scrub starts 2024-03-11T21:14:31.027 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:30 smithi023 bash[21372]: cluster 2024-03-11T21:14:29.105417+0000 osd.7 (osd.7) 532 : cluster 0 6.5 scrub ok 2024-03-11T21:14:31.027 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:30 smithi023 bash[21372]: cluster 2024-03-11T21:14:29.226898+0000 osd.7 (osd.7) 533 : cluster 0 purged_snaps scrub starts 2024-03-11T21:14:31.027 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:30 smithi023 bash[21372]: cluster 2024-03-11T21:14:29.226968+0000 osd.7 (osd.7) 534 : cluster 0 purged_snaps scrub ok 2024-03-11T21:14:31.027 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:30 smithi023 bash[21372]: cluster 2024-03-11T21:14:29.628580+0000 mon.a (mon.0) 1902 : cluster 0 osdmap e419: 8 total, 8 up, 7 in 2024-03-11T21:14:31.027 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:30 smithi023 bash[21372]: audit 2024-03-11T21:14:30.059317+0000 mon.a (mon.0) 1903 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:14:31.027 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:30 smithi023 bash[21372]: audit 2024-03-11T21:14:30.069072+0000 mon.a (mon.0) 1904 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:14:31.027 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:30 smithi023 bash[21372]: cluster 2024-03-11T21:14:30.369526+0000 osd.5 (osd.5) 517 : cluster 0 2.16 scrub starts 2024-03-11T21:14:31.027 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:30 smithi023 bash[21372]: cluster 2024-03-11T21:14:30.370875+0000 osd.5 (osd.5) 518 : cluster 0 2.16 scrub ok 2024-03-11T21:14:31.027 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:30 smithi023 bash[21372]: cluster 2024-03-11T21:14:30.392415+0000 mon.a (mon.0) 1905 : cluster 3 Health check update: Degraded data redundancy: 1196/162432 objects degraded (0.736%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-11T21:14:31.042 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:31.062 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:31.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:31.064 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:14:31.165 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:31.191 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:31.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:31.198 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:14:31.299 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:31.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:31.329 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:31.329 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:14:31.430 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:14:31.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:31.455 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:14:31.455 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:14:31.556 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:14:31.585 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:14:31.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:31.588 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:14:31.689 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:31.716 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:31.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:31.720 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:14:31.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:31 smithi146 bash[23809]: cluster 2024-03-11T21:14:30.001712+0000 osd.7 (osd.7) 535 : cluster 0 2.b scrub starts 2024-03-11T21:14:31.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:31 smithi146 bash[23809]: cluster 2024-03-11T21:14:30.002869+0000 osd.7 (osd.7) 536 : cluster 0 2.b scrub ok 2024-03-11T21:14:31.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:31 smithi146 bash[23809]: cluster 2024-03-11T21:14:30.427001+0000 mgr.y (mgr.24380) 877 : cluster 0 pgmap v1009: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 3.2 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.09k op/s; 1196/156792 objects degraded (0.763%); 2716/156792 objects misplaced (1.732%); 8.6 MiB/s, 137 objects/s recovering 2024-03-11T21:14:31.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:31 smithi146 bash[23809]: audit 2024-03-11T21:14:30.736391+0000 mon.a (mon.0) 1906 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:14:31.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:31 smithi146 bash[23809]: audit 2024-03-11T21:14:30.737109+0000 mon.a (mon.0) 1907 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:14:31.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:31 smithi146 bash[23809]: audit 2024-03-11T21:14:30.743433+0000 mon.a (mon.0) 1908 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:14:31.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:31 smithi146 bash[23809]: cluster 2024-03-11T21:14:31.244601+0000 osd.6 (osd.6) 423 : cluster 0 11.1 scrub starts 2024-03-11T21:14:31.776 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:31 smithi023 bash[21372]: cluster 2024-03-11T21:14:30.001712+0000 osd.7 (osd.7) 535 : cluster 0 2.b scrub starts 2024-03-11T21:14:31.776 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:31 smithi023 bash[21372]: cluster 2024-03-11T21:14:30.002869+0000 osd.7 (osd.7) 536 : cluster 0 2.b scrub ok 2024-03-11T21:14:31.776 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:31 smithi023 bash[21372]: cluster 2024-03-11T21:14:30.427001+0000 mgr.y (mgr.24380) 877 : cluster 0 pgmap v1009: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 3.2 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.09k op/s; 1196/156792 objects degraded (0.763%); 2716/156792 objects misplaced (1.732%); 8.6 MiB/s, 137 objects/s recovering 2024-03-11T21:14:31.776 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:31 smithi023 bash[21372]: audit 2024-03-11T21:14:30.736391+0000 mon.a (mon.0) 1906 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:14:31.776 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:31 smithi023 bash[21372]: audit 2024-03-11T21:14:30.737109+0000 mon.a (mon.0) 1907 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:14:31.776 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:31 smithi023 bash[21372]: audit 2024-03-11T21:14:30.743433+0000 mon.a (mon.0) 1908 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:14:31.776 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:31 smithi023 bash[21372]: cluster 2024-03-11T21:14:31.244601+0000 osd.6 (osd.6) 423 : cluster 0 11.1 scrub starts 2024-03-11T21:14:31.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:31 smithi023 bash[29146]: cluster 2024-03-11T21:14:30.001712+0000 osd.7 (osd.7) 535 : cluster 0 2.b scrub starts 2024-03-11T21:14:31.777 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:31 smithi023 bash[29146]: cluster 2024-03-11T21:14:30.002869+0000 osd.7 (osd.7) 536 : cluster 0 2.b scrub ok 2024-03-11T21:14:31.777 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:31 smithi023 bash[29146]: cluster 2024-03-11T21:14:30.427001+0000 mgr.y (mgr.24380) 877 : cluster 0 pgmap v1009: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 3.2 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.09k op/s; 1196/156792 objects degraded (0.763%); 2716/156792 objects misplaced (1.732%); 8.6 MiB/s, 137 objects/s recovering 2024-03-11T21:14:31.777 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:31 smithi023 bash[29146]: audit 2024-03-11T21:14:30.736391+0000 mon.a (mon.0) 1906 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:14:31.777 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:31 smithi023 bash[29146]: audit 2024-03-11T21:14:30.737109+0000 mon.a (mon.0) 1907 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:14:31.777 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:31 smithi023 bash[29146]: audit 2024-03-11T21:14:30.743433+0000 mon.a (mon.0) 1908 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:14:31.777 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:31 smithi023 bash[29146]: cluster 2024-03-11T21:14:31.244601+0000 osd.6 (osd.6) 423 : cluster 0 11.1 scrub starts 2024-03-11T21:14:31.821 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:14:31.844 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:14:31.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:31.847 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:14:31.947 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:31.973 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:31.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:31.976 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:14:32.077 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:32.094 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:32.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:32.096 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:14:32.154 INFO:teuthology.orchestra.run.smithi023.stdout:osd.0: {} 2024-03-11T21:14:32.191 INFO:teuthology.orchestra.run.smithi023.stdout:osd.1: {} 2024-03-11T21:14:32.197 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:32.227 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:32.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:32.233 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:14:32.238 INFO:teuthology.orchestra.run.smithi023.stdout:osd.2: {} 2024-03-11T21:14:32.282 INFO:teuthology.orchestra.run.smithi023.stdout:osd.3: {} 2024-03-11T21:14:32.327 INFO:teuthology.orchestra.run.smithi023.stdout:osd.4: {} 2024-03-11T21:14:32.333 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:32.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:32.365 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:32.365 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:14:32.374 INFO:teuthology.orchestra.run.smithi023.stdout:osd.5: {} 2024-03-11T21:14:32.410 INFO:teuthology.orchestra.run.smithi023.stdout:osd.6: {} 2024-03-11T21:14:32.464 INFO:teuthology.orchestra.run.smithi023.stdout:osd.7: {} 2024-03-11T21:14:32.466 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:14:32.467 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-11T21:14:32.471 INFO:teuthology.orchestra.run.smithi023.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:14:32.471 INFO:teuthology.orchestra.run.smithi023.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:14:32.471 INFO:teuthology.orchestra.run.smithi023.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:14:32.471 INFO:teuthology.orchestra.run.smithi023.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:14:32.471 INFO:teuthology.orchestra.run.smithi023.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:14:32.471 INFO:teuthology.orchestra.run.smithi023.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:14:32.471 INFO:teuthology.orchestra.run.smithi023.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:14:32.471 INFO:teuthology.orchestra.run.smithi023.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:14:32.501 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:14:32.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:32.506 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:14:32.607 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:14:32.647 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:14:32.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:32.651 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:14:32.670 INFO:teuthology.orchestra.run.smithi023.stderr:nodeep-scrub is unset 2024-03-11T21:14:32.727 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:32 smithi023 bash[21372]: audit 2024-03-11T21:14:32.124391+0000 mon.c (mon.2) 98 : audit 0 from='client.? 172.21.15.23:0/1793135980' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:14:32.727 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:32 smithi023 bash[21372]: cluster 2024-03-11T21:14:32.228406+0000 osd.6 (osd.6) 424 : cluster 0 2.1a scrub starts 2024-03-11T21:14:32.727 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:32 smithi023 bash[21372]: cluster 2024-03-11T21:14:32.229733+0000 osd.6 (osd.6) 425 : cluster 0 2.1a scrub ok 2024-03-11T21:14:32.728 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:32 smithi023 bash[21372]: cluster 2024-03-11T21:14:32.360190+0000 osd.6 (osd.6) 426 : cluster 0 11.1 scrub ok 2024-03-11T21:14:32.728 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:32 smithi023 bash[21372]: cluster 2024-03-11T21:14:32.439882+0000 osd.5 (osd.5) 519 : cluster 0 4.1 scrub starts 2024-03-11T21:14:32.728 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:32 smithi023 bash[21372]: cluster 2024-03-11T21:14:32.441309+0000 osd.5 (osd.5) 520 : cluster 0 4.1 scrub ok 2024-03-11T21:14:32.728 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:32 smithi023 bash[21372]: audit 2024-03-11T21:14:32.563554+0000 mon.a (mon.0) 1909 : audit 1 from='client.? 172.21.15.23:0/2584980648' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:14:32.728 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:32 smithi023 bash[29146]: audit 2024-03-11T21:14:32.124391+0000 mon.c (mon.2) 98 : audit 0 from='client.? 172.21.15.23:0/1793135980' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:14:32.728 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:32 smithi023 bash[29146]: cluster 2024-03-11T21:14:32.228406+0000 osd.6 (osd.6) 424 : cluster 0 2.1a scrub starts 2024-03-11T21:14:32.728 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:32 smithi023 bash[29146]: cluster 2024-03-11T21:14:32.229733+0000 osd.6 (osd.6) 425 : cluster 0 2.1a scrub ok 2024-03-11T21:14:32.728 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:32 smithi023 bash[29146]: cluster 2024-03-11T21:14:32.360190+0000 osd.6 (osd.6) 426 : cluster 0 11.1 scrub ok 2024-03-11T21:14:32.728 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:32 smithi023 bash[29146]: cluster 2024-03-11T21:14:32.439882+0000 osd.5 (osd.5) 519 : cluster 0 4.1 scrub starts 2024-03-11T21:14:32.728 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:32 smithi023 bash[29146]: cluster 2024-03-11T21:14:32.441309+0000 osd.5 (osd.5) 520 : cluster 0 4.1 scrub ok 2024-03-11T21:14:32.728 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:32 smithi023 bash[29146]: audit 2024-03-11T21:14:32.563554+0000 mon.a (mon.0) 1909 : audit 1 from='client.? 172.21.15.23:0/2584980648' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:14:32.752 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:14:32.777 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:14:32.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:32.783 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:14:32.884 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:14:32.893 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:32 smithi146 bash[23809]: audit 2024-03-11T21:14:32.124391+0000 mon.c (mon.2) 98 : audit 0 from='client.? 172.21.15.23:0/1793135980' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:14:32.893 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:32 smithi146 bash[23809]: cluster 2024-03-11T21:14:32.228406+0000 osd.6 (osd.6) 424 : cluster 0 2.1a scrub starts 2024-03-11T21:14:32.893 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:32 smithi146 bash[23809]: cluster 2024-03-11T21:14:32.229733+0000 osd.6 (osd.6) 425 : cluster 0 2.1a scrub ok 2024-03-11T21:14:32.893 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:32 smithi146 bash[23809]: cluster 2024-03-11T21:14:32.360190+0000 osd.6 (osd.6) 426 : cluster 0 11.1 scrub ok 2024-03-11T21:14:32.893 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:32 smithi146 bash[23809]: cluster 2024-03-11T21:14:32.439882+0000 osd.5 (osd.5) 519 : cluster 0 4.1 scrub starts 2024-03-11T21:14:32.893 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:32 smithi146 bash[23809]: cluster 2024-03-11T21:14:32.441309+0000 osd.5 (osd.5) 520 : cluster 0 4.1 scrub ok 2024-03-11T21:14:32.893 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:32 smithi146 bash[23809]: audit 2024-03-11T21:14:32.563554+0000 mon.a (mon.0) 1909 : audit 1 from='client.? 172.21.15.23:0/2584980648' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:14:32.910 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:14:32.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:32.914 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:14:33.015 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:33.046 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:33.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:33.052 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:14:33.153 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:14:33.183 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:14:33.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:33.186 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:14:33.287 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:33.315 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:33.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:33.318 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:14:33.418 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:33.450 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:33.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:33.454 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:14:33.554 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:14:33.581 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:14:33.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:33.588 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:14:33.689 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:33.716 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:33.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:33.720 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:14:33.821 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:33.831 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:33 smithi146 bash[23809]: cluster 2024-03-11T21:14:32.328718+0000 osd.3 (osd.3) 477 : cluster 0 6.b scrub starts 2024-03-11T21:14:33.831 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:33 smithi146 bash[23809]: cluster 2024-03-11T21:14:32.427741+0000 mgr.y (mgr.24380) 878 : cluster 0 pgmap v1010: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 3.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 627 op/s; 1196/154152 objects degraded (0.776%); 2716/154152 objects misplaced (1.762%); 3.3 MiB/s, 52 objects/s recovering 2024-03-11T21:14:33.831 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:33 smithi146 bash[23809]: cluster 2024-03-11T21:14:32.546839+0000 osd.3 (osd.3) 478 : cluster 0 6.b scrub ok 2024-03-11T21:14:33.831 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:33 smithi146 bash[23809]: cluster 2024-03-11T21:14:32.643111+0000 mon.a (mon.0) 1910 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:14:33.831 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:33 smithi146 bash[23809]: audit 2024-03-11T21:14:32.670002+0000 mon.a (mon.0) 1911 : audit 1 from='client.? 172.21.15.23:0/2584980648' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:14:33.831 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:33 smithi146 bash[23809]: cluster 2024-03-11T21:14:32.678941+0000 mon.a (mon.0) 1912 : cluster 0 osdmap e420: 8 total, 8 up, 7 in 2024-03-11T21:14:33.857 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:33.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:33.861 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:14:33.945 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:33 smithi023 bash[21372]: cluster 2024-03-11T21:14:32.328718+0000 osd.3 (osd.3) 477 : cluster 0 6.b scrub starts 2024-03-11T21:14:33.945 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:33 smithi023 bash[21372]: cluster 2024-03-11T21:14:32.427741+0000 mgr.y (mgr.24380) 878 : cluster 0 pgmap v1010: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 3.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 627 op/s; 1196/154152 objects degraded (0.776%); 2716/154152 objects misplaced (1.762%); 3.3 MiB/s, 52 objects/s recovering 2024-03-11T21:14:33.945 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:33 smithi023 bash[21372]: cluster 2024-03-11T21:14:32.546839+0000 osd.3 (osd.3) 478 : cluster 0 6.b scrub ok 2024-03-11T21:14:33.945 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:33 smithi023 bash[21372]: cluster 2024-03-11T21:14:32.643111+0000 mon.a (mon.0) 1910 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:14:33.945 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:33 smithi023 bash[21372]: audit 2024-03-11T21:14:32.670002+0000 mon.a (mon.0) 1911 : audit 1 from='client.? 172.21.15.23:0/2584980648' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:14:33.945 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:33 smithi023 bash[21372]: cluster 2024-03-11T21:14:32.678941+0000 mon.a (mon.0) 1912 : cluster 0 osdmap e420: 8 total, 8 up, 7 in 2024-03-11T21:14:33.946 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:33 smithi023 bash[29146]: cluster 2024-03-11T21:14:32.328718+0000 osd.3 (osd.3) 477 : cluster 0 6.b scrub starts 2024-03-11T21:14:33.946 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:33 smithi023 bash[29146]: cluster 2024-03-11T21:14:32.427741+0000 mgr.y (mgr.24380) 878 : cluster 0 pgmap v1010: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 3.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 627 op/s; 1196/154152 objects degraded (0.776%); 2716/154152 objects misplaced (1.762%); 3.3 MiB/s, 52 objects/s recovering 2024-03-11T21:14:33.946 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:33 smithi023 bash[29146]: cluster 2024-03-11T21:14:32.546839+0000 osd.3 (osd.3) 478 : cluster 0 6.b scrub ok 2024-03-11T21:14:33.946 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:33 smithi023 bash[29146]: cluster 2024-03-11T21:14:32.643111+0000 mon.a (mon.0) 1910 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:14:33.946 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:33 smithi023 bash[29146]: audit 2024-03-11T21:14:32.670002+0000 mon.a (mon.0) 1911 : audit 1 from='client.? 172.21.15.23:0/2584980648' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:14:33.946 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:33 smithi023 bash[29146]: cluster 2024-03-11T21:14:32.678941+0000 mon.a (mon.0) 1912 : cluster 0 osdmap e420: 8 total, 8 up, 7 in 2024-03-11T21:14:33.961 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:14:33.981 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:14:33.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:33.983 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:14:34.084 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:34.122 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:34.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:34.127 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:14:34.228 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:34.254 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:34.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:34.257 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:14:34.358 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:14:34.390 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:14:34.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:34.393 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:14:34.494 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:34.523 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:34.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:34.527 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:14:34.628 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:34.657 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:34.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:34.659 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:14:34.760 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:14:34.786 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:14:34.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:34.788 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:14:34.889 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:34.898 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:34 smithi146 bash[23809]: cluster 2024-03-11T21:14:32.976716+0000 osd.7 (osd.7) 537 : cluster 0 3.d scrub starts 2024-03-11T21:14:34.898 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:34 smithi146 bash[23809]: cluster 2024-03-11T21:14:32.981343+0000 osd.7 (osd.7) 538 : cluster 0 3.d scrub ok 2024-03-11T21:14:34.899 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:34 smithi146 bash[23809]: cluster 2024-03-11T21:14:34.387043+0000 osd.0 (osd.0) 392 : cluster 0 purged_snaps scrub starts 2024-03-11T21:14:34.899 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:34 smithi146 bash[23809]: cluster 2024-03-11T21:14:34.387132+0000 osd.0 (osd.0) 393 : cluster 0 purged_snaps scrub ok 2024-03-11T21:14:34.899 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:34 smithi146 bash[23809]: cluster 2024-03-11T21:14:34.515235+0000 osd.5 (osd.5) 521 : cluster 0 3.1d scrub starts 2024-03-11T21:14:34.899 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:34 smithi146 bash[23809]: cluster 2024-03-11T21:14:34.520122+0000 osd.5 (osd.5) 522 : cluster 0 3.1d scrub ok 2024-03-11T21:14:34.917 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:34.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:34.921 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:14:35.022 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:14:35.060 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:14:35.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:35.068 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:14:35.169 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:14:35.196 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:14:35.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:35.202 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:14:35.275 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:34 smithi023 bash[21372]: cluster 2024-03-11T21:14:32.976716+0000 osd.7 (osd.7) 537 : cluster 0 3.d scrub starts 2024-03-11T21:14:35.276 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:34 smithi023 bash[21372]: cluster 2024-03-11T21:14:32.981343+0000 osd.7 (osd.7) 538 : cluster 0 3.d scrub ok 2024-03-11T21:14:35.276 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:34 smithi023 bash[21372]: cluster 2024-03-11T21:14:34.387043+0000 osd.0 (osd.0) 392 : cluster 0 purged_snaps scrub starts 2024-03-11T21:14:35.276 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:34 smithi023 bash[21372]: cluster 2024-03-11T21:14:34.387132+0000 osd.0 (osd.0) 393 : cluster 0 purged_snaps scrub ok 2024-03-11T21:14:35.276 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:34 smithi023 bash[21372]: cluster 2024-03-11T21:14:34.515235+0000 osd.5 (osd.5) 521 : cluster 0 3.1d scrub starts 2024-03-11T21:14:35.276 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:34 smithi023 bash[21372]: cluster 2024-03-11T21:14:34.520122+0000 osd.5 (osd.5) 522 : cluster 0 3.1d scrub ok 2024-03-11T21:14:35.276 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:34 smithi023 bash[29146]: cluster 2024-03-11T21:14:32.976716+0000 osd.7 (osd.7) 537 : cluster 0 3.d scrub starts 2024-03-11T21:14:35.276 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:34 smithi023 bash[29146]: cluster 2024-03-11T21:14:32.981343+0000 osd.7 (osd.7) 538 : cluster 0 3.d scrub ok 2024-03-11T21:14:35.276 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:34 smithi023 bash[29146]: cluster 2024-03-11T21:14:34.387043+0000 osd.0 (osd.0) 392 : cluster 0 purged_snaps scrub starts 2024-03-11T21:14:35.276 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:34 smithi023 bash[29146]: cluster 2024-03-11T21:14:34.387132+0000 osd.0 (osd.0) 393 : cluster 0 purged_snaps scrub ok 2024-03-11T21:14:35.276 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:34 smithi023 bash[29146]: cluster 2024-03-11T21:14:34.515235+0000 osd.5 (osd.5) 521 : cluster 0 3.1d scrub starts 2024-03-11T21:14:35.276 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:34 smithi023 bash[29146]: cluster 2024-03-11T21:14:34.520122+0000 osd.5 (osd.5) 522 : cluster 0 3.1d scrub ok 2024-03-11T21:14:35.303 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:14:35.330 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:14:35.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:35.333 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:14:35.434 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:35.460 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:35.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:35.463 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:14:35.564 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:35.592 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:35.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:35.598 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:14:35.699 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:35.733 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:35.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:35.736 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:14:35.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:35 smithi023 bash[21372]: cluster 2024-03-11T21:14:34.108440+0000 osd.7 (osd.7) 539 : cluster 0 1.2 scrub starts 2024-03-11T21:14:35.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:35 smithi023 bash[21372]: cluster 2024-03-11T21:14:34.111897+0000 osd.7 (osd.7) 540 : cluster 0 1.2 scrub ok 2024-03-11T21:14:35.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:35 smithi023 bash[21372]: cluster 2024-03-11T21:14:34.428358+0000 mgr.y (mgr.24380) 879 : cluster 0 pgmap v1012: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 3.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 749 B/s rd, 0 B/s wr, 709 op/s; 1311/150912 objects degraded (0.869%); 2491/150912 objects misplaced (1.651%); 3.9 MiB/s, 62 objects/s recovering 2024-03-11T21:14:35.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:35 smithi023 bash[21372]: cluster 2024-03-11T21:14:34.674066+0000 mon.a (mon.0) 1913 : cluster 0 osdmap e421: 8 total, 8 up, 7 in 2024-03-11T21:14:35.776 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:35 smithi023 bash[21372]: cluster 2024-03-11T21:14:35.396129+0000 mon.a (mon.0) 1914 : cluster 3 Health check update: Degraded data redundancy: 1311/150912 objects degraded (0.869%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-11T21:14:35.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:35 smithi023 bash[29146]: cluster 2024-03-11T21:14:34.108440+0000 osd.7 (osd.7) 539 : cluster 0 1.2 scrub starts 2024-03-11T21:14:35.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:35 smithi023 bash[29146]: cluster 2024-03-11T21:14:34.111897+0000 osd.7 (osd.7) 540 : cluster 0 1.2 scrub ok 2024-03-11T21:14:35.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:35 smithi023 bash[29146]: cluster 2024-03-11T21:14:34.428358+0000 mgr.y (mgr.24380) 879 : cluster 0 pgmap v1012: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 3.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 749 B/s rd, 0 B/s wr, 709 op/s; 1311/150912 objects degraded (0.869%); 2491/150912 objects misplaced (1.651%); 3.9 MiB/s, 62 objects/s recovering 2024-03-11T21:14:35.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:35 smithi023 bash[29146]: cluster 2024-03-11T21:14:34.674066+0000 mon.a (mon.0) 1913 : cluster 0 osdmap e421: 8 total, 8 up, 7 in 2024-03-11T21:14:35.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:35 smithi023 bash[29146]: cluster 2024-03-11T21:14:35.396129+0000 mon.a (mon.0) 1914 : cluster 3 Health check update: Degraded data redundancy: 1311/150912 objects degraded (0.869%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-11T21:14:35.837 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:14:35.867 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:14:35.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:35.872 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:14:35.933 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:14:35.973 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:14:35.998 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:14:36.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:36.004 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:14:36.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:35 smithi146 bash[23809]: cluster 2024-03-11T21:14:34.108440+0000 osd.7 (osd.7) 539 : cluster 0 1.2 scrub starts 2024-03-11T21:14:36.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:35 smithi146 bash[23809]: cluster 2024-03-11T21:14:34.111897+0000 osd.7 (osd.7) 540 : cluster 0 1.2 scrub ok 2024-03-11T21:14:36.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:35 smithi146 bash[23809]: cluster 2024-03-11T21:14:34.428358+0000 mgr.y (mgr.24380) 879 : cluster 0 pgmap v1012: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 3.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 749 B/s rd, 0 B/s wr, 709 op/s; 1311/150912 objects degraded (0.869%); 2491/150912 objects misplaced (1.651%); 3.9 MiB/s, 62 objects/s recovering 2024-03-11T21:14:36.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:35 smithi146 bash[23809]: cluster 2024-03-11T21:14:34.674066+0000 mon.a (mon.0) 1913 : cluster 0 osdmap e421: 8 total, 8 up, 7 in 2024-03-11T21:14:36.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:35 smithi146 bash[23809]: cluster 2024-03-11T21:14:35.396129+0000 mon.a (mon.0) 1914 : cluster 3 Health check update: Degraded data redundancy: 1311/150912 objects degraded (0.869%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-11T21:14:36.104 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:14:36.129 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:14:36.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:36.131 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:14:36.232 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:14:36.263 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:14:36.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:36.264 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:14:36.364 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:36.374 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:14:36 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:14:36.394 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:36.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:36.399 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:14:36.500 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:36.528 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:36.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:36.531 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:14:36.632 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:14:36.661 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:14:36.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:36.664 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:14:36.764 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:36.774 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:36 smithi023 bash[21372]: cluster 2024-03-11T21:14:35.350436+0000 osd.3 (osd.3) 479 : cluster 0 3.7 scrub starts 2024-03-11T21:14:36.774 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:36 smithi023 bash[21372]: cluster 2024-03-11T21:14:35.355702+0000 osd.3 (osd.3) 480 : cluster 0 3.7 scrub ok 2024-03-11T21:14:36.774 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:36 smithi023 bash[21372]: cluster 2024-03-11T21:14:36.141809+0000 osd.6 (osd.6) 427 : cluster 0 1.5 scrub starts 2024-03-11T21:14:36.774 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:36 smithi023 bash[21372]: cluster 2024-03-11T21:14:36.148412+0000 osd.6 (osd.6) 428 : cluster 0 1.5 scrub ok 2024-03-11T21:14:36.774 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:36 smithi023 bash[21372]: audit 2024-03-11T21:14:36.458556+0000 mon.a (mon.0) 1915 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:14:36.774 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:36 smithi023 bash[21372]: audit 2024-03-11T21:14:36.460385+0000 mon.a (mon.0) 1916 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:14:36.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:36 smithi023 bash[29146]: cluster 2024-03-11T21:14:35.350436+0000 osd.3 (osd.3) 479 : cluster 0 3.7 scrub starts 2024-03-11T21:14:36.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:36 smithi023 bash[29146]: cluster 2024-03-11T21:14:35.355702+0000 osd.3 (osd.3) 480 : cluster 0 3.7 scrub ok 2024-03-11T21:14:36.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:36 smithi023 bash[29146]: cluster 2024-03-11T21:14:36.141809+0000 osd.6 (osd.6) 427 : cluster 0 1.5 scrub starts 2024-03-11T21:14:36.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:36 smithi023 bash[29146]: cluster 2024-03-11T21:14:36.148412+0000 osd.6 (osd.6) 428 : cluster 0 1.5 scrub ok 2024-03-11T21:14:36.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:36 smithi023 bash[29146]: audit 2024-03-11T21:14:36.458556+0000 mon.a (mon.0) 1915 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:14:36.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:36 smithi023 bash[29146]: audit 2024-03-11T21:14:36.460385+0000 mon.a (mon.0) 1916 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:14:36.790 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:36.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:36.792 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:14:36.893 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:14:36.922 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:14:36.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:36.927 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:14:36.927 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:36 smithi146 bash[23809]: cluster 2024-03-11T21:14:35.350436+0000 osd.3 (osd.3) 479 : cluster 0 3.7 scrub starts 2024-03-11T21:14:36.927 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:36 smithi146 bash[23809]: cluster 2024-03-11T21:14:35.355702+0000 osd.3 (osd.3) 480 : cluster 0 3.7 scrub ok 2024-03-11T21:14:36.927 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:36 smithi146 bash[23809]: cluster 2024-03-11T21:14:36.141809+0000 osd.6 (osd.6) 427 : cluster 0 1.5 scrub starts 2024-03-11T21:14:36.927 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:36 smithi146 bash[23809]: cluster 2024-03-11T21:14:36.148412+0000 osd.6 (osd.6) 428 : cluster 0 1.5 scrub ok 2024-03-11T21:14:36.928 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:36 smithi146 bash[23809]: audit 2024-03-11T21:14:36.458556+0000 mon.a (mon.0) 1915 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:14:36.928 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:36 smithi146 bash[23809]: audit 2024-03-11T21:14:36.460385+0000 mon.a (mon.0) 1916 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:14:37.027 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:37.058 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:37.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:37.061 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:14:37.162 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:14:37.180 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:14:37.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:37.182 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:14:37.283 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:14:37.304 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:14:37.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:37.306 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:14:37.407 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:37.431 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:37.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:37.433 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:14:37.534 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:37.562 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:37.562 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T21:14:37.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:37.564 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:14:37.577 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd set noscrub 2024-03-11T21:14:37.665 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:14:37.699 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:14:37.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:37.703 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:14:37.804 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:37.814 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:37 smithi023 bash[29146]: audit 2024-03-11T21:14:36.048582+0000 mgr.y (mgr.24380) 880 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:14:37.814 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:37 smithi023 bash[29146]: cluster 2024-03-11T21:14:36.429234+0000 mgr.y (mgr.24380) 881 : cluster 0 pgmap v1014: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 2.8 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.08k op/s; 1311/140349 objects degraded (0.934%); 2491/140349 objects misplaced (1.775%); 2.2 MiB/s, 35 objects/s recovering 2024-03-11T21:14:37.814 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:37 smithi023 bash[21372]: audit 2024-03-11T21:14:36.048582+0000 mgr.y (mgr.24380) 880 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:14:37.814 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:37 smithi023 bash[21372]: cluster 2024-03-11T21:14:36.429234+0000 mgr.y (mgr.24380) 881 : cluster 0 pgmap v1014: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 2.8 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.08k op/s; 1311/140349 objects degraded (0.934%); 2491/140349 objects misplaced (1.775%); 2.2 MiB/s, 35 objects/s recovering 2024-03-11T21:14:37.834 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:37.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:37.838 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:14:37.938 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:37.965 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:37.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:37.968 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:14:38.007 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:37 smithi146 bash[23809]: audit 2024-03-11T21:14:36.048582+0000 mgr.y (mgr.24380) 880 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:14:38.007 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:37 smithi146 bash[23809]: cluster 2024-03-11T21:14:36.429234+0000 mgr.y (mgr.24380) 881 : cluster 0 pgmap v1014: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 2.8 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.08k op/s; 1311/140349 objects degraded (0.934%); 2491/140349 objects misplaced (1.775%); 2.2 MiB/s, 35 objects/s recovering 2024-03-11T21:14:38.069 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:38.095 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:38.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:38.098 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:14:38.198 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:38.236 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:38.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:38.240 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:14:38.341 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:38.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:38.367 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:38.367 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:14:38.468 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:14:38.503 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:14:38.564 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:14:38 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:14:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:14:38.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:38.565 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:14:38.666 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:14:38.696 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:14:38.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:38.698 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:14:38.799 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:14:38.837 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:14:38.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:38.839 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:14:38.940 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:38.967 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:38.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:38.973 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:14:39.074 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:14:39.104 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:14:39.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:39.110 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:14:39.210 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:39.237 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:39.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:39.240 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:14:39.340 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:14:39.373 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:14:39.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:39.375 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:14:39.475 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:39.503 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:39.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:39.509 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:14:39.610 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:39.637 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:39.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:39.640 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:14:39.741 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:39.762 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:39.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:39.765 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:14:39.866 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:14:39.873 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:39 smithi023 bash[21372]: cluster 2024-03-11T21:14:38.107498+0000 osd.7 (osd.7) 541 : cluster 0 11.f scrub starts 2024-03-11T21:14:39.873 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:39 smithi023 bash[21372]: cluster 2024-03-11T21:14:38.429865+0000 mgr.y (mgr.24380) 882 : cluster 0 pgmap v1015: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 2.8 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 844 op/s; 1481/136521 objects degraded (1.085%); 2172/136521 objects misplaced (1.591%); 2.0 MiB/s, 32 objects/s recovering 2024-03-11T21:14:39.873 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:39 smithi023 bash[21372]: cluster 2024-03-11T21:14:39.072267+0000 osd.6 (osd.6) 429 : cluster 0 2.7 scrub starts 2024-03-11T21:14:39.873 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:39 smithi023 bash[21372]: cluster 2024-03-11T21:14:39.076754+0000 osd.6 (osd.6) 430 : cluster 0 2.7 scrub ok 2024-03-11T21:14:39.874 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:39 smithi023 bash[29146]: cluster 2024-03-11T21:14:38.107498+0000 osd.7 (osd.7) 541 : cluster 0 11.f scrub starts 2024-03-11T21:14:39.874 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:39 smithi023 bash[29146]: cluster 2024-03-11T21:14:38.429865+0000 mgr.y (mgr.24380) 882 : cluster 0 pgmap v1015: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 2.8 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 844 op/s; 1481/136521 objects degraded (1.085%); 2172/136521 objects misplaced (1.591%); 2.0 MiB/s, 32 objects/s recovering 2024-03-11T21:14:39.874 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:39 smithi023 bash[29146]: cluster 2024-03-11T21:14:39.072267+0000 osd.6 (osd.6) 429 : cluster 0 2.7 scrub starts 2024-03-11T21:14:39.874 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:39 smithi023 bash[29146]: cluster 2024-03-11T21:14:39.076754+0000 osd.6 (osd.6) 430 : cluster 0 2.7 scrub ok 2024-03-11T21:14:39.890 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:14:39.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:39.894 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:14:39.995 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:14:40.006 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:39 smithi146 bash[23809]: cluster 2024-03-11T21:14:38.107498+0000 osd.7 (osd.7) 541 : cluster 0 11.f scrub starts 2024-03-11T21:14:40.007 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:39 smithi146 bash[23809]: cluster 2024-03-11T21:14:38.429865+0000 mgr.y (mgr.24380) 882 : cluster 0 pgmap v1015: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 2.8 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 844 op/s; 1481/136521 objects degraded (1.085%); 2172/136521 objects misplaced (1.591%); 2.0 MiB/s, 32 objects/s recovering 2024-03-11T21:14:40.007 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:39 smithi146 bash[23809]: cluster 2024-03-11T21:14:39.072267+0000 osd.6 (osd.6) 429 : cluster 0 2.7 scrub starts 2024-03-11T21:14:40.007 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:39 smithi146 bash[23809]: cluster 2024-03-11T21:14:39.076754+0000 osd.6 (osd.6) 430 : cluster 0 2.7 scrub ok 2024-03-11T21:14:40.024 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:14:40.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:40.025 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:14:40.126 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:40.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:40.159 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:40.159 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:14:40.208 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:14:40.260 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:14:40.283 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:14:40.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:40.285 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:14:40.386 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:40.408 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:40.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:40.409 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:14:40.510 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:40.535 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:40.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:40.537 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:14:40.638 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:40.684 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:40.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:40.689 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:14:40.789 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:40.813 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:40.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:40.816 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:14:40.916 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:14:40.925 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:40 smithi023 bash[21372]: cluster 2024-03-11T21:14:38.946967+0000 osd.7 (osd.7) 542 : cluster 0 11.f scrub ok 2024-03-11T21:14:40.926 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:40 smithi023 bash[21372]: cluster 2024-03-11T21:14:40.397562+0000 mon.a (mon.0) 1917 : cluster 3 Health check update: Degraded data redundancy: 1481/136521 objects degraded (1.085%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-11T21:14:40.926 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:40 smithi023 bash[21372]: cluster 2024-03-11T21:14:40.628865+0000 osd.3 (osd.3) 481 : cluster 0 4.11 scrub starts 2024-03-11T21:14:40.926 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:40 smithi023 bash[21372]: cluster 2024-03-11T21:14:40.631040+0000 osd.3 (osd.3) 482 : cluster 0 4.11 scrub ok 2024-03-11T21:14:40.926 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:40 smithi023 bash[29146]: cluster 2024-03-11T21:14:38.946967+0000 osd.7 (osd.7) 542 : cluster 0 11.f scrub ok 2024-03-11T21:14:40.926 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:40 smithi023 bash[29146]: cluster 2024-03-11T21:14:40.397562+0000 mon.a (mon.0) 1917 : cluster 3 Health check update: Degraded data redundancy: 1481/136521 objects degraded (1.085%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-11T21:14:40.926 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:40 smithi023 bash[29146]: cluster 2024-03-11T21:14:40.628865+0000 osd.3 (osd.3) 481 : cluster 0 4.11 scrub starts 2024-03-11T21:14:40.926 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:40 smithi023 bash[29146]: cluster 2024-03-11T21:14:40.631040+0000 osd.3 (osd.3) 482 : cluster 0 4.11 scrub ok 2024-03-11T21:14:40.943 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:14:40.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:40.946 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:14:41.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:40 smithi146 bash[23809]: cluster 2024-03-11T21:14:38.946967+0000 osd.7 (osd.7) 542 : cluster 0 11.f scrub ok 2024-03-11T21:14:41.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:40 smithi146 bash[23809]: cluster 2024-03-11T21:14:40.397562+0000 mon.a (mon.0) 1917 : cluster 3 Health check update: Degraded data redundancy: 1481/136521 objects degraded (1.085%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-11T21:14:41.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:40 smithi146 bash[23809]: cluster 2024-03-11T21:14:40.628865+0000 osd.3 (osd.3) 481 : cluster 0 4.11 scrub starts 2024-03-11T21:14:41.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:40 smithi146 bash[23809]: cluster 2024-03-11T21:14:40.631040+0000 osd.3 (osd.3) 482 : cluster 0 4.11 scrub ok 2024-03-11T21:14:41.048 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:41.089 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:41.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:41.093 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:14:41.194 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:14:41.223 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:14:41.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:41.226 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:14:41.327 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:14:41.351 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:14:41.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:41.355 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:14:41.456 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:14:41.484 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:14:41.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:41.487 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:14:41.588 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:14:41.617 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:14:41.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:41.621 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:14:41.721 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:41.749 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:41.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:41.754 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:14:41.854 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:41.895 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:41.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:41.900 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:14:42.000 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:42.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:41 smithi146 bash[23809]: cluster 2024-03-11T21:14:40.120887+0000 osd.7 (osd.7) 543 : cluster 0 1.0 deep-scrub starts 2024-03-11T21:14:42.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:41 smithi146 bash[23809]: cluster 2024-03-11T21:14:40.126333+0000 osd.7 (osd.7) 544 : cluster 0 1.0 deep-scrub ok 2024-03-11T21:14:42.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:41 smithi146 bash[23809]: cluster 2024-03-11T21:14:40.430378+0000 mgr.y (mgr.24380) 883 : cluster 0 pgmap v1016: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 2.7 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 854 op/s; 1481/133662 objects degraded (1.108%); 2172/133662 objects misplaced (1.625%); 2.0 MiB/s, 32 objects/s recovering 2024-03-11T21:14:42.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:41 smithi146 bash[23809]: cluster 2024-03-11T21:14:41.096053+0000 osd.6 (osd.6) 431 : cluster 0 3.a scrub starts 2024-03-11T21:14:42.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:41 smithi146 bash[23809]: cluster 2024-03-11T21:14:41.101189+0000 osd.6 (osd.6) 432 : cluster 0 3.a scrub ok 2024-03-11T21:14:42.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:41 smithi146 bash[23809]: cluster 2024-03-11T21:14:41.624527+0000 osd.3 (osd.3) 483 : cluster 0 6.13 scrub starts 2024-03-11T21:14:42.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:41 smithi023 bash[29146]: cluster 2024-03-11T21:14:40.120887+0000 osd.7 (osd.7) 543 : cluster 0 1.0 deep-scrub starts 2024-03-11T21:14:42.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:41 smithi023 bash[29146]: cluster 2024-03-11T21:14:40.126333+0000 osd.7 (osd.7) 544 : cluster 0 1.0 deep-scrub ok 2024-03-11T21:14:42.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:41 smithi023 bash[29146]: cluster 2024-03-11T21:14:40.430378+0000 mgr.y (mgr.24380) 883 : cluster 0 pgmap v1016: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 2.7 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 854 op/s; 1481/133662 objects degraded (1.108%); 2172/133662 objects misplaced (1.625%); 2.0 MiB/s, 32 objects/s recovering 2024-03-11T21:14:42.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:41 smithi023 bash[29146]: cluster 2024-03-11T21:14:41.096053+0000 osd.6 (osd.6) 431 : cluster 0 3.a scrub starts 2024-03-11T21:14:42.027 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:41 smithi023 bash[29146]: cluster 2024-03-11T21:14:41.101189+0000 osd.6 (osd.6) 432 : cluster 0 3.a scrub ok 2024-03-11T21:14:42.027 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:41 smithi023 bash[29146]: cluster 2024-03-11T21:14:41.624527+0000 osd.3 (osd.3) 483 : cluster 0 6.13 scrub starts 2024-03-11T21:14:42.027 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:41 smithi023 bash[21372]: cluster 2024-03-11T21:14:40.120887+0000 osd.7 (osd.7) 543 : cluster 0 1.0 deep-scrub starts 2024-03-11T21:14:42.027 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:41 smithi023 bash[21372]: cluster 2024-03-11T21:14:40.126333+0000 osd.7 (osd.7) 544 : cluster 0 1.0 deep-scrub ok 2024-03-11T21:14:42.027 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:41 smithi023 bash[21372]: cluster 2024-03-11T21:14:40.430378+0000 mgr.y (mgr.24380) 883 : cluster 0 pgmap v1016: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 2.7 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 854 op/s; 1481/133662 objects degraded (1.108%); 2172/133662 objects misplaced (1.625%); 2.0 MiB/s, 32 objects/s recovering 2024-03-11T21:14:42.027 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:41 smithi023 bash[21372]: cluster 2024-03-11T21:14:41.096053+0000 osd.6 (osd.6) 431 : cluster 0 3.a scrub starts 2024-03-11T21:14:42.027 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:41 smithi023 bash[21372]: cluster 2024-03-11T21:14:41.101189+0000 osd.6 (osd.6) 432 : cluster 0 3.a scrub ok 2024-03-11T21:14:42.027 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:41 smithi023 bash[21372]: cluster 2024-03-11T21:14:41.624527+0000 osd.3 (osd.3) 483 : cluster 0 6.13 scrub starts 2024-03-11T21:14:42.031 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:42.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:42.037 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:14:42.138 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:14:42.157 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:14:42.159 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:14:42.172 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:14:42.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:42.183 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:14:42.283 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:14:42.302 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:14:42.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:42.304 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:14:42.405 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:14:42.432 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:14:42.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:42.439 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:14:42.540 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:14:42.565 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:14:42.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:42.567 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:14:42.668 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:42.686 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:42.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:42.688 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:14:42.789 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:14:42.835 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:14:42.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:42.837 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:14:42.938 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:42.959 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:42.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:42.962 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:14:43.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:42 smithi146 bash[23809]: cluster 2024-03-11T21:14:41.639707+0000 osd.5 (osd.5) 523 : cluster 0 6.10 scrub starts 2024-03-11T21:14:43.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:42 smithi146 bash[23809]: cluster 2024-03-11T21:14:41.641013+0000 osd.5 (osd.5) 524 : cluster 0 6.10 scrub ok 2024-03-11T21:14:43.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:42 smithi146 bash[23809]: cluster 2024-03-11T21:14:41.715484+0000 osd.3 (osd.3) 484 : cluster 0 6.13 scrub ok 2024-03-11T21:14:43.017 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:42 smithi023 bash[21372]: cluster 2024-03-11T21:14:41.639707+0000 osd.5 (osd.5) 523 : cluster 0 6.10 scrub starts 2024-03-11T21:14:43.017 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:42 smithi023 bash[21372]: cluster 2024-03-11T21:14:41.641013+0000 osd.5 (osd.5) 524 : cluster 0 6.10 scrub ok 2024-03-11T21:14:43.017 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:42 smithi023 bash[21372]: cluster 2024-03-11T21:14:41.715484+0000 osd.3 (osd.3) 484 : cluster 0 6.13 scrub ok 2024-03-11T21:14:43.017 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:42 smithi023 bash[29146]: cluster 2024-03-11T21:14:41.639707+0000 osd.5 (osd.5) 523 : cluster 0 6.10 scrub starts 2024-03-11T21:14:43.017 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:42 smithi023 bash[29146]: cluster 2024-03-11T21:14:41.641013+0000 osd.5 (osd.5) 524 : cluster 0 6.10 scrub ok 2024-03-11T21:14:43.017 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:42 smithi023 bash[29146]: cluster 2024-03-11T21:14:41.715484+0000 osd.3 (osd.3) 484 : cluster 0 6.13 scrub ok 2024-03-11T21:14:43.063 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:14:43.097 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:14:43.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:43.100 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:14:43.201 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:14:43.223 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:14:43.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:43.229 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:14:43.330 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:43.355 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:43.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:43.362 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:14:43.463 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:43.499 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:43.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:43.504 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:14:43.605 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:43.647 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:43.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:43.651 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:14:43.752 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:43.773 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:43.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:43.776 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:14:43.876 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:14:43.905 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:14:43.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:43.910 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:14:44.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:43 smithi146 bash[23809]: cluster 2024-03-11T21:14:42.431087+0000 mgr.y (mgr.24380) 884 : cluster 0 pgmap v1017: 158 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+degraded+remapped, 156 active+clean; 2.5 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 984 op/s; 1481/125382 objects degraded (1.181%); 2172/125382 objects misplaced (1.732%); 1.7 MiB/s, 26 objects/s recovering 2024-03-11T21:14:44.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:43 smithi146 bash[23809]: cluster 2024-03-11T21:14:42.712146+0000 osd.3 (osd.3) 485 : cluster 0 6.11 scrub starts 2024-03-11T21:14:44.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:43 smithi146 bash[23809]: cluster 2024-03-11T21:14:42.714311+0000 osd.3 (osd.3) 486 : cluster 0 6.11 scrub ok 2024-03-11T21:14:44.011 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:14:44.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:43 smithi023 bash[21372]: cluster 2024-03-11T21:14:42.431087+0000 mgr.y (mgr.24380) 884 : cluster 0 pgmap v1017: 158 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+degraded+remapped, 156 active+clean; 2.5 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 984 op/s; 1481/125382 objects degraded (1.181%); 2172/125382 objects misplaced (1.732%); 1.7 MiB/s, 26 objects/s recovering 2024-03-11T21:14:44.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:43 smithi023 bash[21372]: cluster 2024-03-11T21:14:42.712146+0000 osd.3 (osd.3) 485 : cluster 0 6.11 scrub starts 2024-03-11T21:14:44.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:43 smithi023 bash[21372]: cluster 2024-03-11T21:14:42.714311+0000 osd.3 (osd.3) 486 : cluster 0 6.11 scrub ok 2024-03-11T21:14:44.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:43 smithi023 bash[29146]: cluster 2024-03-11T21:14:42.431087+0000 mgr.y (mgr.24380) 884 : cluster 0 pgmap v1017: 158 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+degraded+remapped, 156 active+clean; 2.5 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 984 op/s; 1481/125382 objects degraded (1.181%); 2172/125382 objects misplaced (1.732%); 1.7 MiB/s, 26 objects/s recovering 2024-03-11T21:14:44.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:43 smithi023 bash[29146]: cluster 2024-03-11T21:14:42.712146+0000 osd.3 (osd.3) 485 : cluster 0 6.11 scrub starts 2024-03-11T21:14:44.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:43 smithi023 bash[29146]: cluster 2024-03-11T21:14:42.714311+0000 osd.3 (osd.3) 486 : cluster 0 6.11 scrub ok 2024-03-11T21:14:44.028 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:14:44.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:44.030 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:14:44.131 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:44.152 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:44.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:44.154 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:14:44.255 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:14:44.282 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:14:44.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:44.289 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:14:44.390 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:44.431 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:44.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:44.433 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:14:44.534 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:44.565 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:44.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:44.567 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:14:44.668 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:44.695 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:44.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:44.698 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:14:44.799 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:44.824 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:44.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:44.830 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:14:44.931 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:44.956 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:44.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:44.962 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:14:45.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:44 smithi023 bash[21372]: cluster 2024-03-11T21:14:43.253069+0000 osd.4 (osd.4) 117 : cluster 0 purged_snaps scrub starts 2024-03-11T21:14:45.026 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:44 smithi023 bash[21372]: cluster 2024-03-11T21:14:43.253149+0000 osd.4 (osd.4) 118 : cluster 0 purged_snaps scrub ok 2024-03-11T21:14:45.026 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:44 smithi023 bash[21372]: cluster 2024-03-11T21:14:43.902295+0000 mon.a (mon.0) 1918 : cluster 0 osdmap e422: 8 total, 8 up, 7 in 2024-03-11T21:14:45.026 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:44 smithi023 bash[21372]: cluster 2024-03-11T21:14:44.094634+0000 osd.7 (osd.7) 545 : cluster 0 11.4 scrub starts 2024-03-11T21:14:45.026 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:44 smithi023 bash[21372]: cluster 2024-03-11T21:14:44.431533+0000 mgr.y (mgr.24380) 885 : cluster 0 pgmap v1019: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 156 active+clean; 2.5 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.01k op/s; 1642/121287 objects degraded (1.354%); 1815/121287 objects misplaced (1.496%); 2.2 MiB/s, 35 objects/s recovering 2024-03-11T21:14:45.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:44 smithi023 bash[29146]: cluster 2024-03-11T21:14:43.253069+0000 osd.4 (osd.4) 117 : cluster 0 purged_snaps scrub starts 2024-03-11T21:14:45.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:44 smithi023 bash[29146]: cluster 2024-03-11T21:14:43.253149+0000 osd.4 (osd.4) 118 : cluster 0 purged_snaps scrub ok 2024-03-11T21:14:45.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:44 smithi023 bash[29146]: cluster 2024-03-11T21:14:43.902295+0000 mon.a (mon.0) 1918 : cluster 0 osdmap e422: 8 total, 8 up, 7 in 2024-03-11T21:14:45.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:44 smithi023 bash[29146]: cluster 2024-03-11T21:14:44.094634+0000 osd.7 (osd.7) 545 : cluster 0 11.4 scrub starts 2024-03-11T21:14:45.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:44 smithi023 bash[29146]: cluster 2024-03-11T21:14:44.431533+0000 mgr.y (mgr.24380) 885 : cluster 0 pgmap v1019: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 156 active+clean; 2.5 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.01k op/s; 1642/121287 objects degraded (1.354%); 1815/121287 objects misplaced (1.496%); 2.2 MiB/s, 35 objects/s recovering 2024-03-11T21:14:45.063 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:45.095 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:45.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:45.101 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:14:45.202 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:45.230 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:45.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:45.232 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:14:45.255 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:44 smithi146 bash[23809]: cluster 2024-03-11T21:14:43.253069+0000 osd.4 (osd.4) 117 : cluster 0 purged_snaps scrub starts 2024-03-11T21:14:45.255 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:44 smithi146 bash[23809]: cluster 2024-03-11T21:14:43.253149+0000 osd.4 (osd.4) 118 : cluster 0 purged_snaps scrub ok 2024-03-11T21:14:45.255 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:44 smithi146 bash[23809]: cluster 2024-03-11T21:14:43.902295+0000 mon.a (mon.0) 1918 : cluster 0 osdmap e422: 8 total, 8 up, 7 in 2024-03-11T21:14:45.255 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:44 smithi146 bash[23809]: cluster 2024-03-11T21:14:44.094634+0000 osd.7 (osd.7) 545 : cluster 0 11.4 scrub starts 2024-03-11T21:14:45.255 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:44 smithi146 bash[23809]: cluster 2024-03-11T21:14:44.431533+0000 mgr.y (mgr.24380) 885 : cluster 0 pgmap v1019: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 156 active+clean; 2.5 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.01k op/s; 1642/121287 objects degraded (1.354%); 1815/121287 objects misplaced (1.496%); 2.2 MiB/s, 35 objects/s recovering 2024-03-11T21:14:45.333 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:45.360 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:45.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:45.364 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:14:45.465 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:45.489 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:45.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:45.491 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:14:45.592 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:14:45.615 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:14:45.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:45.617 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:14:45.718 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:14:45.745 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:14:45.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:45.749 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:14:45.849 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:45.880 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:45.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:45.886 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:14:45.988 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:46.015 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:46.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:46.018 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:14:46.119 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:14:46.141 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:14:46.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:46.145 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:14:46.246 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:14:46.254 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:14:46 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:14:46.255 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:45 smithi146 bash[23809]: cluster 2024-03-11T21:14:45.031174+0000 osd.7 (osd.7) 546 : cluster 0 11.4 scrub ok 2024-03-11T21:14:46.255 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:45 smithi146 bash[23809]: cluster 2024-03-11T21:14:45.050562+0000 osd.6 (osd.6) 433 : cluster 0 6.1e scrub starts 2024-03-11T21:14:46.255 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:45 smithi146 bash[23809]: cluster 2024-03-11T21:14:45.081200+0000 osd.6 (osd.6) 434 : cluster 0 6.1e scrub ok 2024-03-11T21:14:46.255 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:45 smithi146 bash[23809]: cluster 2024-03-11T21:14:45.107858+0000 osd.7 (osd.7) 547 : cluster 0 11.5 deep-scrub starts 2024-03-11T21:14:46.255 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:45 smithi146 bash[23809]: cluster 2024-03-11T21:14:45.398448+0000 mon.a (mon.0) 1919 : cluster 3 Health check update: Degraded data redundancy: 1642/121287 objects degraded (1.354%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-11T21:14:46.272 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:14:46.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:46.276 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:14:46.376 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:46.400 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:46.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:46.405 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:14:46.506 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:46.525 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:46.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:46.527 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:14:46.537 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:46 smithi023 bash[21372]: cluster 2024-03-11T21:14:45.031174+0000 osd.7 (osd.7) 546 : cluster 0 11.4 scrub ok 2024-03-11T21:14:46.537 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:46 smithi023 bash[21372]: cluster 2024-03-11T21:14:45.050562+0000 osd.6 (osd.6) 433 : cluster 0 6.1e scrub starts 2024-03-11T21:14:46.537 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:46 smithi023 bash[21372]: cluster 2024-03-11T21:14:45.081200+0000 osd.6 (osd.6) 434 : cluster 0 6.1e scrub ok 2024-03-11T21:14:46.537 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:46 smithi023 bash[21372]: cluster 2024-03-11T21:14:45.107858+0000 osd.7 (osd.7) 547 : cluster 0 11.5 deep-scrub starts 2024-03-11T21:14:46.537 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:46 smithi023 bash[21372]: cluster 2024-03-11T21:14:45.398448+0000 mon.a (mon.0) 1919 : cluster 3 Health check update: Degraded data redundancy: 1642/121287 objects degraded (1.354%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-11T21:14:46.537 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:46 smithi023 bash[29146]: cluster 2024-03-11T21:14:45.031174+0000 osd.7 (osd.7) 546 : cluster 0 11.4 scrub ok 2024-03-11T21:14:46.537 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:46 smithi023 bash[29146]: cluster 2024-03-11T21:14:45.050562+0000 osd.6 (osd.6) 433 : cluster 0 6.1e scrub starts 2024-03-11T21:14:46.537 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:46 smithi023 bash[29146]: cluster 2024-03-11T21:14:45.081200+0000 osd.6 (osd.6) 434 : cluster 0 6.1e scrub ok 2024-03-11T21:14:46.537 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:46 smithi023 bash[29146]: cluster 2024-03-11T21:14:45.107858+0000 osd.7 (osd.7) 547 : cluster 0 11.5 deep-scrub starts 2024-03-11T21:14:46.538 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:46 smithi023 bash[29146]: cluster 2024-03-11T21:14:45.398448+0000 mon.a (mon.0) 1919 : cluster 3 Health check update: Degraded data redundancy: 1642/121287 objects degraded (1.354%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-11T21:14:46.627 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:46.647 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:46.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:46.651 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:14:46.751 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:14:46.785 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:14:46.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:46.790 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:14:46.891 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:14:46.927 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:14:46.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:46.929 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:14:47.030 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:47.055 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:47.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:47.057 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:14:47.158 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:14:47.169 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:46 smithi146 bash[23809]: cluster 2024-03-11T21:14:45.553883+0000 osd.5 (osd.5) 525 : cluster 0 6.17 scrub starts 2024-03-11T21:14:47.169 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:46 smithi146 bash[23809]: cluster 2024-03-11T21:14:45.728248+0000 osd.5 (osd.5) 526 : cluster 0 6.17 scrub ok 2024-03-11T21:14:47.169 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:46 smithi146 bash[23809]: cluster 2024-03-11T21:14:46.003665+0000 osd.6 (osd.6) 435 : cluster 0 4.18 scrub starts 2024-03-11T21:14:47.169 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:46 smithi146 bash[23809]: cluster 2024-03-11T21:14:46.005478+0000 osd.6 (osd.6) 436 : cluster 0 4.18 scrub ok 2024-03-11T21:14:47.169 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:46 smithi146 bash[23809]: audit 2024-03-11T21:14:46.058953+0000 mgr.y (mgr.24380) 886 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:14:47.169 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:46 smithi146 bash[23809]: cluster 2024-03-11T21:14:46.111705+0000 osd.7 (osd.7) 548 : cluster 0 1.9 scrub starts 2024-03-11T21:14:47.170 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:46 smithi146 bash[23809]: cluster 2024-03-11T21:14:46.117290+0000 osd.7 (osd.7) 549 : cluster 0 1.9 scrub ok 2024-03-11T21:14:47.170 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:46 smithi146 bash[23809]: cluster 2024-03-11T21:14:46.432376+0000 mgr.y (mgr.24380) 887 : cluster 0 pgmap v1020: 158 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+degraded+remapped, 156 active+clean; 2.2 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 993 op/s; 1642/110550 objects degraded (1.485%); 1815/110550 objects misplaced (1.642%); 2.2 MiB/s, 34 objects/s recovering 2024-03-11T21:14:47.191 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:14:47.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:47.193 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:14:47.275 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:46 smithi023 bash[21372]: cluster 2024-03-11T21:14:45.553883+0000 osd.5 (osd.5) 525 : cluster 0 6.17 scrub starts 2024-03-11T21:14:47.276 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:46 smithi023 bash[21372]: cluster 2024-03-11T21:14:45.728248+0000 osd.5 (osd.5) 526 : cluster 0 6.17 scrub ok 2024-03-11T21:14:47.276 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:46 smithi023 bash[21372]: cluster 2024-03-11T21:14:46.003665+0000 osd.6 (osd.6) 435 : cluster 0 4.18 scrub starts 2024-03-11T21:14:47.276 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:46 smithi023 bash[21372]: cluster 2024-03-11T21:14:46.005478+0000 osd.6 (osd.6) 436 : cluster 0 4.18 scrub ok 2024-03-11T21:14:47.276 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:46 smithi023 bash[21372]: audit 2024-03-11T21:14:46.058953+0000 mgr.y (mgr.24380) 886 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:14:47.276 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:46 smithi023 bash[21372]: cluster 2024-03-11T21:14:46.111705+0000 osd.7 (osd.7) 548 : cluster 0 1.9 scrub starts 2024-03-11T21:14:47.276 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:46 smithi023 bash[21372]: cluster 2024-03-11T21:14:46.117290+0000 osd.7 (osd.7) 549 : cluster 0 1.9 scrub ok 2024-03-11T21:14:47.276 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:46 smithi023 bash[21372]: cluster 2024-03-11T21:14:46.432376+0000 mgr.y (mgr.24380) 887 : cluster 0 pgmap v1020: 158 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+degraded+remapped, 156 active+clean; 2.2 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 993 op/s; 1642/110550 objects degraded (1.485%); 1815/110550 objects misplaced (1.642%); 2.2 MiB/s, 34 objects/s recovering 2024-03-11T21:14:47.276 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:46 smithi023 bash[29146]: cluster 2024-03-11T21:14:45.553883+0000 osd.5 (osd.5) 525 : cluster 0 6.17 scrub starts 2024-03-11T21:14:47.276 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:46 smithi023 bash[29146]: cluster 2024-03-11T21:14:45.728248+0000 osd.5 (osd.5) 526 : cluster 0 6.17 scrub ok 2024-03-11T21:14:47.276 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:46 smithi023 bash[29146]: cluster 2024-03-11T21:14:46.003665+0000 osd.6 (osd.6) 435 : cluster 0 4.18 scrub starts 2024-03-11T21:14:47.277 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:46 smithi023 bash[29146]: cluster 2024-03-11T21:14:46.005478+0000 osd.6 (osd.6) 436 : cluster 0 4.18 scrub ok 2024-03-11T21:14:47.277 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:46 smithi023 bash[29146]: audit 2024-03-11T21:14:46.058953+0000 mgr.y (mgr.24380) 886 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:14:47.277 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:46 smithi023 bash[29146]: cluster 2024-03-11T21:14:46.111705+0000 osd.7 (osd.7) 548 : cluster 0 1.9 scrub starts 2024-03-11T21:14:47.277 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:46 smithi023 bash[29146]: cluster 2024-03-11T21:14:46.117290+0000 osd.7 (osd.7) 549 : cluster 0 1.9 scrub ok 2024-03-11T21:14:47.277 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:46 smithi023 bash[29146]: cluster 2024-03-11T21:14:46.432376+0000 mgr.y (mgr.24380) 887 : cluster 0 pgmap v1020: 158 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+degraded+remapped, 156 active+clean; 2.2 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 993 op/s; 1642/110550 objects degraded (1.485%); 1815/110550 objects misplaced (1.642%); 2.2 MiB/s, 34 objects/s recovering 2024-03-11T21:14:47.293 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:47.321 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:47.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:47.324 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:14:47.425 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:47.459 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:47.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:47.466 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:14:47.566 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:47.591 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:47.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:47.594 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:14:47.695 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:47.719 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:47.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:47.721 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:14:47.822 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:14:47.844 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:14:47.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:47.846 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:14:47.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:47 smithi023 bash[21372]: cluster 2024-03-11T21:14:47.703264+0000 osd.3 (osd.3) 487 : cluster 0 3.1e scrub starts 2024-03-11T21:14:47.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:47 smithi023 bash[21372]: cluster 2024-03-11T21:14:47.708869+0000 osd.3 (osd.3) 488 : cluster 0 3.1e scrub ok 2024-03-11T21:14:47.947 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:14:47.957 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:47 smithi023 bash[29146]: cluster 2024-03-11T21:14:47.703264+0000 osd.3 (osd.3) 487 : cluster 0 3.1e scrub starts 2024-03-11T21:14:47.957 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:47 smithi023 bash[29146]: cluster 2024-03-11T21:14:47.708869+0000 osd.3 (osd.3) 488 : cluster 0 3.1e scrub ok 2024-03-11T21:14:47.978 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:14:47.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:47.983 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:14:48.085 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:14:48.112 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:14:48.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:48.116 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:14:48.217 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:48.227 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:47 smithi146 bash[23809]: cluster 2024-03-11T21:14:47.703264+0000 osd.3 (osd.3) 487 : cluster 0 3.1e scrub starts 2024-03-11T21:14:48.227 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:47 smithi146 bash[23809]: cluster 2024-03-11T21:14:47.708869+0000 osd.3 (osd.3) 488 : cluster 0 3.1e scrub ok 2024-03-11T21:14:48.242 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:48.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:48.248 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:14:48.349 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:14:48.374 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:14:48.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:48.378 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:14:48.479 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:14:48.517 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:14:48.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:48.521 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:14:48.622 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:14:48.632 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:14:48 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:14:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:14:48.649 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:14:48.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:48.657 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:14:48.757 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:48.784 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:48.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:48.788 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:14:48.889 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:14:48.918 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:14:48.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:48.923 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:14:49.024 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:49.034 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:48 smithi146 bash[23809]: cluster 2024-03-11T21:14:48.157418+0000 osd.7 (osd.7) 550 : cluster 0 4.1e scrub starts 2024-03-11T21:14:49.034 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:48 smithi146 bash[23809]: cluster 2024-03-11T21:14:48.161539+0000 osd.7 (osd.7) 551 : cluster 0 4.1e scrub ok 2024-03-11T21:14:49.034 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:48 smithi146 bash[23809]: cluster 2024-03-11T21:14:48.433045+0000 mgr.y (mgr.24380) 888 : cluster 0 pgmap v1021: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 156 active+clean; 2.2 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 978 op/s; 1730/107178 objects degraded (1.614%); 1549/107178 objects misplaced (1.445%); 2.3 MiB/s, 37 objects/s recovering 2024-03-11T21:14:49.034 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:48 smithi146 bash[23809]: cluster 2024-03-11T21:14:48.511684+0000 osd.5 (osd.5) 527 : cluster 0 2.c scrub starts 2024-03-11T21:14:49.035 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:48 smithi146 bash[23809]: cluster 2024-03-11T21:14:48.520677+0000 osd.5 (osd.5) 528 : cluster 0 2.c scrub ok 2024-03-11T21:14:49.049 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:49.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:49.052 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:14:49.152 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:49.181 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:49.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:49.185 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:14:49.185 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:48 smithi023 bash[21372]: cluster 2024-03-11T21:14:48.157418+0000 osd.7 (osd.7) 550 : cluster 0 4.1e scrub starts 2024-03-11T21:14:49.185 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:48 smithi023 bash[21372]: cluster 2024-03-11T21:14:48.161539+0000 osd.7 (osd.7) 551 : cluster 0 4.1e scrub ok 2024-03-11T21:14:49.185 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:48 smithi023 bash[21372]: cluster 2024-03-11T21:14:48.433045+0000 mgr.y (mgr.24380) 888 : cluster 0 pgmap v1021: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 156 active+clean; 2.2 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 978 op/s; 1730/107178 objects degraded (1.614%); 1549/107178 objects misplaced (1.445%); 2.3 MiB/s, 37 objects/s recovering 2024-03-11T21:14:49.185 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:48 smithi023 bash[21372]: cluster 2024-03-11T21:14:48.511684+0000 osd.5 (osd.5) 527 : cluster 0 2.c scrub starts 2024-03-11T21:14:49.185 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:48 smithi023 bash[21372]: cluster 2024-03-11T21:14:48.520677+0000 osd.5 (osd.5) 528 : cluster 0 2.c scrub ok 2024-03-11T21:14:49.185 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:48 smithi023 bash[21372]: debug 2024-03-11T21:14:48.944+0000 7f55248dd700 -1 mon.a@0(leader).osd e422 definitely_dead 0 2024-03-11T21:14:49.185 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:48 smithi023 bash[29146]: cluster 2024-03-11T21:14:48.157418+0000 osd.7 (osd.7) 550 : cluster 0 4.1e scrub starts 2024-03-11T21:14:49.185 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:48 smithi023 bash[29146]: cluster 2024-03-11T21:14:48.161539+0000 osd.7 (osd.7) 551 : cluster 0 4.1e scrub ok 2024-03-11T21:14:49.186 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:48 smithi023 bash[29146]: cluster 2024-03-11T21:14:48.433045+0000 mgr.y (mgr.24380) 888 : cluster 0 pgmap v1021: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 156 active+clean; 2.2 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 978 op/s; 1730/107178 objects degraded (1.614%); 1549/107178 objects misplaced (1.445%); 2.3 MiB/s, 37 objects/s recovering 2024-03-11T21:14:49.186 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:48 smithi023 bash[29146]: cluster 2024-03-11T21:14:48.511684+0000 osd.5 (osd.5) 527 : cluster 0 2.c scrub starts 2024-03-11T21:14:49.186 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:48 smithi023 bash[29146]: cluster 2024-03-11T21:14:48.520677+0000 osd.5 (osd.5) 528 : cluster 0 2.c scrub ok 2024-03-11T21:14:49.285 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:14:49.321 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:14:49.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:49.323 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:14:49.424 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:14:49.451 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:14:49.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:49.454 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:14:49.555 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:14:49.580 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:14:49.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:49.583 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:14:49.683 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:14:49.708 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:14:49.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:49.712 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:14:49.813 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:49.836 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:49.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:49.840 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:14:49.941 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:14:49.972 INFO:teuthology.orchestra.run.smithi023.stderr:marked in osd.0. 2024-03-11T21:14:49.977 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:14:49.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:49.980 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:14:50.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:49 smithi146 bash[23809]: audit 2024-03-11T21:14:48.950125+0000 mon.c (mon.2) 99 : audit 0 from='client.? 172.21.15.23:0/1427107069' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["0"]}]: dispatch 2024-03-11T21:14:50.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:49 smithi146 bash[23809]: audit 2024-03-11T21:14:48.950584+0000 mon.a (mon.0) 1920 : audit 0 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["0"]}]: dispatch 2024-03-11T21:14:50.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:49 smithi146 bash[23809]: cluster 2024-03-11T21:14:49.020507+0000 osd.6 (osd.6) 437 : cluster 0 5.2 scrub starts 2024-03-11T21:14:50.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:49 smithi146 bash[23809]: cluster 2024-03-11T21:14:49.024162+0000 osd.6 (osd.6) 438 : cluster 0 5.2 scrub ok 2024-03-11T21:14:50.081 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:50.101 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:50.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:50.105 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:14:50.206 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:50.216 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:49 smithi023 bash[21372]: audit 2024-03-11T21:14:48.950125+0000 mon.c (mon.2) 99 : audit 0 from='client.? 172.21.15.23:0/1427107069' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["0"]}]: dispatch 2024-03-11T21:14:50.216 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:49 smithi023 bash[21372]: audit 2024-03-11T21:14:48.950584+0000 mon.a (mon.0) 1920 : audit 0 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["0"]}]: dispatch 2024-03-11T21:14:50.216 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:49 smithi023 bash[21372]: cluster 2024-03-11T21:14:49.020507+0000 osd.6 (osd.6) 437 : cluster 0 5.2 scrub starts 2024-03-11T21:14:50.216 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:49 smithi023 bash[21372]: cluster 2024-03-11T21:14:49.024162+0000 osd.6 (osd.6) 438 : cluster 0 5.2 scrub ok 2024-03-11T21:14:50.217 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:49 smithi023 bash[29146]: audit 2024-03-11T21:14:48.950125+0000 mon.c (mon.2) 99 : audit 0 from='client.? 172.21.15.23:0/1427107069' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["0"]}]: dispatch 2024-03-11T21:14:50.217 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:49 smithi023 bash[29146]: audit 2024-03-11T21:14:48.950584+0000 mon.a (mon.0) 1920 : audit 0 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["0"]}]: dispatch 2024-03-11T21:14:50.217 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:49 smithi023 bash[29146]: cluster 2024-03-11T21:14:49.020507+0000 osd.6 (osd.6) 437 : cluster 0 5.2 scrub starts 2024-03-11T21:14:50.217 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:49 smithi023 bash[29146]: cluster 2024-03-11T21:14:49.024162+0000 osd.6 (osd.6) 438 : cluster 0 5.2 scrub ok 2024-03-11T21:14:50.231 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:50.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:50.233 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:14:50.334 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:50.352 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:50.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:50.354 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:14:50.455 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:50.477 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:50.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:50.480 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:14:50.581 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:50.602 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:50.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:50.604 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:14:50.705 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:14:50.729 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:14:50.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:50.731 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:14:50.832 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:50.856 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:50.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:50.858 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:14:50.959 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:14:50.987 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:14:50.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:50.989 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:14:51.090 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:51.100 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:50 smithi146 bash[23809]: audit 2024-03-11T21:14:49.962294+0000 mon.a (mon.0) 1921 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["0"]}]': finished 2024-03-11T21:14:51.100 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:50 smithi146 bash[23809]: cluster 2024-03-11T21:14:49.970376+0000 mon.a (mon.0) 1922 : cluster 0 osdmap e423: 8 total, 8 up, 8 in 2024-03-11T21:14:51.100 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:50 smithi146 bash[23809]: cluster 2024-03-11T21:14:50.399627+0000 mon.a (mon.0) 1923 : cluster 3 Health check update: Degraded data redundancy: 1730/107178 objects degraded (1.614%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-11T21:14:51.100 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:50 smithi146 bash[23809]: cluster 2024-03-11T21:14:50.414497+0000 mon.a (mon.0) 1924 : cluster 0 osdmap e424: 8 total, 8 up, 8 in 2024-03-11T21:14:51.100 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:50 smithi146 bash[23809]: cluster 2024-03-11T21:14:50.433469+0000 mgr.y (mgr.24380) 889 : cluster 0 pgmap v1024: 158 pgs: 6 peering, 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 149 active+clean; 2.1 GiB data, 9.9 GiB used, 616 GiB / 626 GiB avail; 783 B/s rd, 0 B/s wr, 847 op/s; 1730/104691 objects degraded (1.652%); 1549/104691 objects misplaced (1.480%); 1.7 MiB/s, 27 objects/s recovering 2024-03-11T21:14:51.100 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:50 smithi146 bash[23809]: cluster 2024-03-11T21:14:50.662735+0000 osd.1 (osd.1) 299 : cluster 0 4.2 scrub starts 2024-03-11T21:14:51.100 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:50 smithi146 bash[23809]: cluster 2024-03-11T21:14:50.664074+0000 osd.1 (osd.1) 300 : cluster 0 4.2 scrub ok 2024-03-11T21:14:51.112 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:51.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:51.114 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:14:51.215 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:51.236 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:51.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:51.238 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:14:51.339 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:14:51.350 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:51 smithi023 bash[29146]: audit 2024-03-11T21:14:49.962294+0000 mon.a (mon.0) 1921 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["0"]}]': finished 2024-03-11T21:14:51.351 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:51 smithi023 bash[29146]: cluster 2024-03-11T21:14:49.970376+0000 mon.a (mon.0) 1922 : cluster 0 osdmap e423: 8 total, 8 up, 8 in 2024-03-11T21:14:51.351 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:51 smithi023 bash[29146]: cluster 2024-03-11T21:14:50.399627+0000 mon.a (mon.0) 1923 : cluster 3 Health check update: Degraded data redundancy: 1730/107178 objects degraded (1.614%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-11T21:14:51.351 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:51 smithi023 bash[29146]: cluster 2024-03-11T21:14:50.414497+0000 mon.a (mon.0) 1924 : cluster 0 osdmap e424: 8 total, 8 up, 8 in 2024-03-11T21:14:51.351 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:51 smithi023 bash[29146]: cluster 2024-03-11T21:14:50.433469+0000 mgr.y (mgr.24380) 889 : cluster 0 pgmap v1024: 158 pgs: 6 peering, 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 149 active+clean; 2.1 GiB data, 9.9 GiB used, 616 GiB / 626 GiB avail; 783 B/s rd, 0 B/s wr, 847 op/s; 1730/104691 objects degraded (1.652%); 1549/104691 objects misplaced (1.480%); 1.7 MiB/s, 27 objects/s recovering 2024-03-11T21:14:51.351 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:51 smithi023 bash[29146]: cluster 2024-03-11T21:14:50.662735+0000 osd.1 (osd.1) 299 : cluster 0 4.2 scrub starts 2024-03-11T21:14:51.351 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:51 smithi023 bash[29146]: cluster 2024-03-11T21:14:50.664074+0000 osd.1 (osd.1) 300 : cluster 0 4.2 scrub ok 2024-03-11T21:14:51.351 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:51 smithi023 bash[21372]: audit 2024-03-11T21:14:49.962294+0000 mon.a (mon.0) 1921 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["0"]}]': finished 2024-03-11T21:14:51.351 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:51 smithi023 bash[21372]: cluster 2024-03-11T21:14:49.970376+0000 mon.a (mon.0) 1922 : cluster 0 osdmap e423: 8 total, 8 up, 8 in 2024-03-11T21:14:51.351 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:51 smithi023 bash[21372]: cluster 2024-03-11T21:14:50.399627+0000 mon.a (mon.0) 1923 : cluster 3 Health check update: Degraded data redundancy: 1730/107178 objects degraded (1.614%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-11T21:14:51.351 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:51 smithi023 bash[21372]: cluster 2024-03-11T21:14:50.414497+0000 mon.a (mon.0) 1924 : cluster 0 osdmap e424: 8 total, 8 up, 8 in 2024-03-11T21:14:51.351 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:51 smithi023 bash[21372]: cluster 2024-03-11T21:14:50.433469+0000 mgr.y (mgr.24380) 889 : cluster 0 pgmap v1024: 158 pgs: 6 peering, 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 149 active+clean; 2.1 GiB data, 9.9 GiB used, 616 GiB / 626 GiB avail; 783 B/s rd, 0 B/s wr, 847 op/s; 1730/104691 objects degraded (1.652%); 1549/104691 objects misplaced (1.480%); 1.7 MiB/s, 27 objects/s recovering 2024-03-11T21:14:51.351 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:51 smithi023 bash[21372]: cluster 2024-03-11T21:14:50.662735+0000 osd.1 (osd.1) 299 : cluster 0 4.2 scrub starts 2024-03-11T21:14:51.351 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:51 smithi023 bash[21372]: cluster 2024-03-11T21:14:50.664074+0000 osd.1 (osd.1) 300 : cluster 0 4.2 scrub ok 2024-03-11T21:14:51.365 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:14:51.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:51.367 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:14:51.468 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:51.486 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:51.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:51.488 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:14:51.589 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:14:51.619 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:14:51.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:51.625 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:14:51.726 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:51.749 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:51.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:51.751 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:14:51.851 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:14:51.886 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:14:51.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:51.891 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:14:51.992 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:52.031 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:52.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:52.032 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:14:52.133 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:52.152 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:52.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:52.154 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:14:52.255 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:52.285 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:52.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:52.290 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:14:52.391 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:52.415 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:52.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:52.417 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:14:52.466 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:52 smithi023 bash[21372]: cluster 2024-03-11T21:14:51.039488+0000 osd.6 (osd.6) 439 : cluster 0 4.a scrub starts 2024-03-11T21:14:52.466 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:52 smithi023 bash[21372]: cluster 2024-03-11T21:14:51.040900+0000 osd.6 (osd.6) 440 : cluster 0 4.a scrub ok 2024-03-11T21:14:52.466 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:52 smithi023 bash[21372]: cluster 2024-03-11T21:14:51.409957+0000 osd.3 (osd.3) 489 : cluster 0 11.b starting backfill to osd.0 from (0'0,0'0] MAX to 422'10232 2024-03-11T21:14:52.467 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:52 smithi023 bash[21372]: cluster 2024-03-11T21:14:51.409998+0000 osd.1 (osd.1) 301 : cluster 0 11.6 starting backfill to osd.0 from (0'0,402'5546] MAX to 422'10186 2024-03-11T21:14:52.467 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:52 smithi023 bash[21372]: cluster 2024-03-11T21:14:51.411044+0000 osd.3 (osd.3) 490 : cluster 0 11.2 starting backfill to osd.0 from (0'0,0'0] MAX to 422'10311 2024-03-11T21:14:52.467 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:52 smithi023 bash[21372]: cluster 2024-03-11T21:14:51.411370+0000 osd.6 (osd.6) 441 : cluster 0 11.1 starting backfill to osd.0 from (0'0,0'0] MAX to 422'10137 2024-03-11T21:14:52.467 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:52 smithi023 bash[21372]: cluster 2024-03-11T21:14:51.417419+0000 mon.a (mon.0) 1925 : cluster 0 osdmap e425: 8 total, 8 up, 8 in 2024-03-11T21:14:52.467 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:52 smithi023 bash[21372]: cluster 2024-03-11T21:14:51.417884+0000 osd.3 (osd.3) 491 : cluster 0 11.9 starting backfill to osd.0 from (0'0,0'0] MAX to 422'10111 2024-03-11T21:14:52.467 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:52 smithi023 bash[21372]: audit 2024-03-11T21:14:51.482561+0000 mon.a (mon.0) 1926 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:14:52.518 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:14:52.544 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:14:52.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:52.548 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:14:52.649 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:52.673 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:52.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:52.676 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:14:52.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:52 smithi023 bash[29146]: cluster 2024-03-11T21:14:51.039488+0000 osd.6 (osd.6) 439 : cluster 0 4.a scrub starts 2024-03-11T21:14:52.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:52 smithi023 bash[29146]: cluster 2024-03-11T21:14:51.040900+0000 osd.6 (osd.6) 440 : cluster 0 4.a scrub ok 2024-03-11T21:14:52.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:52 smithi023 bash[29146]: cluster 2024-03-11T21:14:51.409957+0000 osd.3 (osd.3) 489 : cluster 0 11.b starting backfill to osd.0 from (0'0,0'0] MAX to 422'10232 2024-03-11T21:14:52.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:52 smithi023 bash[29146]: cluster 2024-03-11T21:14:51.409998+0000 osd.1 (osd.1) 301 : cluster 0 11.6 starting backfill to osd.0 from (0'0,402'5546] MAX to 422'10186 2024-03-11T21:14:52.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:52 smithi023 bash[29146]: cluster 2024-03-11T21:14:51.411044+0000 osd.3 (osd.3) 490 : cluster 0 11.2 starting backfill to osd.0 from (0'0,0'0] MAX to 422'10311 2024-03-11T21:14:52.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:52 smithi023 bash[29146]: cluster 2024-03-11T21:14:51.411370+0000 osd.6 (osd.6) 441 : cluster 0 11.1 starting backfill to osd.0 from (0'0,0'0] MAX to 422'10137 2024-03-11T21:14:52.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:52 smithi023 bash[29146]: cluster 2024-03-11T21:14:51.417419+0000 mon.a (mon.0) 1925 : cluster 0 osdmap e425: 8 total, 8 up, 8 in 2024-03-11T21:14:52.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:52 smithi023 bash[29146]: cluster 2024-03-11T21:14:51.417884+0000 osd.3 (osd.3) 491 : cluster 0 11.9 starting backfill to osd.0 from (0'0,0'0] MAX to 422'10111 2024-03-11T21:14:52.777 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:52 smithi023 bash[29146]: audit 2024-03-11T21:14:51.482561+0000 mon.a (mon.0) 1926 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:14:52.777 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:14:52.785 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:52 smithi146 bash[23809]: cluster 2024-03-11T21:14:51.039488+0000 osd.6 (osd.6) 439 : cluster 0 4.a scrub starts 2024-03-11T21:14:52.785 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:52 smithi146 bash[23809]: cluster 2024-03-11T21:14:51.040900+0000 osd.6 (osd.6) 440 : cluster 0 4.a scrub ok 2024-03-11T21:14:52.785 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:52 smithi146 bash[23809]: cluster 2024-03-11T21:14:51.409957+0000 osd.3 (osd.3) 489 : cluster 0 11.b starting backfill to osd.0 from (0'0,0'0] MAX to 422'10232 2024-03-11T21:14:52.785 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:52 smithi146 bash[23809]: cluster 2024-03-11T21:14:51.409998+0000 osd.1 (osd.1) 301 : cluster 0 11.6 starting backfill to osd.0 from (0'0,402'5546] MAX to 422'10186 2024-03-11T21:14:52.785 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:52 smithi146 bash[23809]: cluster 2024-03-11T21:14:51.411044+0000 osd.3 (osd.3) 490 : cluster 0 11.2 starting backfill to osd.0 from (0'0,0'0] MAX to 422'10311 2024-03-11T21:14:52.786 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:52 smithi146 bash[23809]: cluster 2024-03-11T21:14:51.411370+0000 osd.6 (osd.6) 441 : cluster 0 11.1 starting backfill to osd.0 from (0'0,0'0] MAX to 422'10137 2024-03-11T21:14:52.786 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:52 smithi146 bash[23809]: cluster 2024-03-11T21:14:51.417419+0000 mon.a (mon.0) 1925 : cluster 0 osdmap e425: 8 total, 8 up, 8 in 2024-03-11T21:14:52.786 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:52 smithi146 bash[23809]: cluster 2024-03-11T21:14:51.417884+0000 osd.3 (osd.3) 491 : cluster 0 11.9 starting backfill to osd.0 from (0'0,0'0] MAX to 422'10111 2024-03-11T21:14:52.786 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:52 smithi146 bash[23809]: audit 2024-03-11T21:14:51.482561+0000 mon.a (mon.0) 1926 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:14:52.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:52.806 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:14:52.806 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:14:52.907 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:52.933 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:52.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:52.935 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:14:53.036 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:14:53.062 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:14:53.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:53.067 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:14:53.168 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:53.196 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:53.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:53.198 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:14:53.299 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:53.332 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:53.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:53.336 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:14:53.437 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:53.465 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:53.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:53.468 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:14:53.569 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:14:53.598 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:14:53.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:53.600 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:14:53.701 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:14:53.711 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:53 smithi023 bash[29146]: cluster 2024-03-11T21:14:51.603672+0000 osd.5 (osd.5) 529 : cluster 0 6.d scrub starts 2024-03-11T21:14:53.712 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:53 smithi023 bash[29146]: cluster 2024-03-11T21:14:51.855132+0000 osd.5 (osd.5) 530 : cluster 0 6.d scrub ok 2024-03-11T21:14:53.712 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:53 smithi023 bash[29146]: cluster 2024-03-11T21:14:52.231282+0000 osd.7 (osd.7) 552 : cluster 0 2.1c scrub starts 2024-03-11T21:14:53.712 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:53 smithi023 bash[29146]: cluster 2024-03-11T21:14:52.238824+0000 osd.7 (osd.7) 553 : cluster 0 2.1c scrub ok 2024-03-11T21:14:53.712 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:53 smithi023 bash[29146]: cluster 2024-03-11T21:14:52.423452+0000 mon.a (mon.0) 1927 : cluster 0 osdmap e426: 8 total, 8 up, 8 in 2024-03-11T21:14:53.712 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:53 smithi023 bash[29146]: cluster 2024-03-11T21:14:52.433968+0000 mgr.y (mgr.24380) 890 : cluster 0 pgmap v1027: 158 pgs: 2 activating, 6 remapped+peering, 6 peering, 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+degraded+remapped, 142 active+clean; 2.0 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 787 op/s; 1730/97740 objects degraded (1.770%); 1549/97740 objects misplaced (1.585%); 156 B/s, 7 objects/s recovering 2024-03-11T21:14:53.712 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:53 smithi023 bash[29146]: cluster 2024-03-11T21:14:53.424766+0000 mon.a (mon.0) 1928 : cluster 0 osdmap e427: 8 total, 8 up, 8 in 2024-03-11T21:14:53.712 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:53 smithi023 bash[21372]: cluster 2024-03-11T21:14:51.603672+0000 osd.5 (osd.5) 529 : cluster 0 6.d scrub starts 2024-03-11T21:14:53.712 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:53 smithi023 bash[21372]: cluster 2024-03-11T21:14:51.855132+0000 osd.5 (osd.5) 530 : cluster 0 6.d scrub ok 2024-03-11T21:14:53.712 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:53 smithi023 bash[21372]: cluster 2024-03-11T21:14:52.231282+0000 osd.7 (osd.7) 552 : cluster 0 2.1c scrub starts 2024-03-11T21:14:53.712 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:53 smithi023 bash[21372]: cluster 2024-03-11T21:14:52.238824+0000 osd.7 (osd.7) 553 : cluster 0 2.1c scrub ok 2024-03-11T21:14:53.712 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:53 smithi023 bash[21372]: cluster 2024-03-11T21:14:52.423452+0000 mon.a (mon.0) 1927 : cluster 0 osdmap e426: 8 total, 8 up, 8 in 2024-03-11T21:14:53.712 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:53 smithi023 bash[21372]: cluster 2024-03-11T21:14:52.433968+0000 mgr.y (mgr.24380) 890 : cluster 0 pgmap v1027: 158 pgs: 2 activating, 6 remapped+peering, 6 peering, 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+degraded+remapped, 142 active+clean; 2.0 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 787 op/s; 1730/97740 objects degraded (1.770%); 1549/97740 objects misplaced (1.585%); 156 B/s, 7 objects/s recovering 2024-03-11T21:14:53.713 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:53 smithi023 bash[21372]: cluster 2024-03-11T21:14:53.424766+0000 mon.a (mon.0) 1928 : cluster 0 osdmap e427: 8 total, 8 up, 8 in 2024-03-11T21:14:53.730 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:14:53.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:53.734 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:14:53.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:53 smithi146 bash[23809]: cluster 2024-03-11T21:14:51.603672+0000 osd.5 (osd.5) 529 : cluster 0 6.d scrub starts 2024-03-11T21:14:53.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:53 smithi146 bash[23809]: cluster 2024-03-11T21:14:51.855132+0000 osd.5 (osd.5) 530 : cluster 0 6.d scrub ok 2024-03-11T21:14:53.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:53 smithi146 bash[23809]: cluster 2024-03-11T21:14:52.231282+0000 osd.7 (osd.7) 552 : cluster 0 2.1c scrub starts 2024-03-11T21:14:53.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:53 smithi146 bash[23809]: cluster 2024-03-11T21:14:52.238824+0000 osd.7 (osd.7) 553 : cluster 0 2.1c scrub ok 2024-03-11T21:14:53.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:53 smithi146 bash[23809]: cluster 2024-03-11T21:14:52.423452+0000 mon.a (mon.0) 1927 : cluster 0 osdmap e426: 8 total, 8 up, 8 in 2024-03-11T21:14:53.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:53 smithi146 bash[23809]: cluster 2024-03-11T21:14:52.433968+0000 mgr.y (mgr.24380) 890 : cluster 0 pgmap v1027: 158 pgs: 2 activating, 6 remapped+peering, 6 peering, 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+degraded+remapped, 142 active+clean; 2.0 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 787 op/s; 1730/97740 objects degraded (1.770%); 1549/97740 objects misplaced (1.585%); 156 B/s, 7 objects/s recovering 2024-03-11T21:14:53.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:53 smithi146 bash[23809]: cluster 2024-03-11T21:14:53.424766+0000 mon.a (mon.0) 1928 : cluster 0 osdmap e427: 8 total, 8 up, 8 in 2024-03-11T21:14:53.835 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:14:53.863 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:14:53.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:53.868 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:14:53.969 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:14:54.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:54.005 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:14:54.005 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:14:54.106 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:14:54.133 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:14:54.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:54.140 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:14:54.242 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:54.264 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:54.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:54.266 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:14:54.367 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:54.406 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:54.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:54.410 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:14:54.511 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:54.544 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:54.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:54.550 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:14:54.651 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:54.662 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:54 smithi023 bash[21372]: cluster 2024-03-11T21:14:53.046222+0000 osd.6 (osd.6) 442 : cluster 0 4.13 scrub starts 2024-03-11T21:14:54.662 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:54 smithi023 bash[21372]: cluster 2024-03-11T21:14:53.058980+0000 osd.6 (osd.6) 443 : cluster 0 4.13 scrub ok 2024-03-11T21:14:54.662 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:54 smithi023 bash[21372]: cluster 2024-03-11T21:14:53.227755+0000 osd.7 (osd.7) 554 : cluster 0 11.5 deep-scrub ok 2024-03-11T21:14:54.662 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:54 smithi023 bash[29146]: cluster 2024-03-11T21:14:53.046222+0000 osd.6 (osd.6) 442 : cluster 0 4.13 scrub starts 2024-03-11T21:14:54.662 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:54 smithi023 bash[29146]: cluster 2024-03-11T21:14:53.058980+0000 osd.6 (osd.6) 443 : cluster 0 4.13 scrub ok 2024-03-11T21:14:54.663 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:54 smithi023 bash[29146]: cluster 2024-03-11T21:14:53.227755+0000 osd.7 (osd.7) 554 : cluster 0 11.5 deep-scrub ok 2024-03-11T21:14:54.684 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:54.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:54.687 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:14:54.754 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:54 smithi146 bash[23809]: cluster 2024-03-11T21:14:53.046222+0000 osd.6 (osd.6) 442 : cluster 0 4.13 scrub starts 2024-03-11T21:14:54.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:54 smithi146 bash[23809]: cluster 2024-03-11T21:14:53.058980+0000 osd.6 (osd.6) 443 : cluster 0 4.13 scrub ok 2024-03-11T21:14:54.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:54 smithi146 bash[23809]: cluster 2024-03-11T21:14:53.227755+0000 osd.7 (osd.7) 554 : cluster 0 11.5 deep-scrub ok 2024-03-11T21:14:54.789 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:14:54.813 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:14:54.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:54.815 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:14:54.916 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:54.936 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:54.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:54.938 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:14:55.039 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:55.078 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:55.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:55.082 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:14:55.182 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:55.211 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:55.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:55.216 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:14:55.316 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:14:55.334 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:14:55.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:55.336 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:14:55.437 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:55.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:55.468 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:55.468 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:14:55.569 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:55.591 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:55.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:55.596 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:14:55.697 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:55.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:55 smithi146 bash[23809]: cluster 2024-03-11T21:14:54.051009+0000 osd.6 (osd.6) 444 : cluster 0 3.10 scrub starts 2024-03-11T21:14:55.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:55 smithi146 bash[23809]: cluster 2024-03-11T21:14:54.058796+0000 osd.6 (osd.6) 445 : cluster 0 3.10 scrub ok 2024-03-11T21:14:55.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:55 smithi146 bash[23809]: cluster 2024-03-11T21:14:54.437366+0000 mgr.y (mgr.24380) 891 : cluster 0 pgmap v1029: 158 pgs: 2 activating, 6 remapped+peering, 6 peering, 1 active+clean+scrubbing+deep, 1 active+recovering+degraded+remapped, 142 active+clean; 1.9 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 744 op/s; 1400/95715 objects degraded (1.463%); 2.2 MiB/s, 43 objects/s recovering 2024-03-11T21:14:55.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:55 smithi146 bash[23809]: cluster 2024-03-11T21:14:55.400748+0000 mon.a (mon.0) 1929 : cluster 3 Health check update: Degraded data redundancy: 1400/95715 objects degraded (1.463%), 1 pg degraded (PG_DEGRADED) 2024-03-11T21:14:55.721 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:55.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:55.724 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:14:55.743 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:55 smithi023 bash[29146]: cluster 2024-03-11T21:14:54.051009+0000 osd.6 (osd.6) 444 : cluster 0 3.10 scrub starts 2024-03-11T21:14:55.743 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:55 smithi023 bash[29146]: cluster 2024-03-11T21:14:54.058796+0000 osd.6 (osd.6) 445 : cluster 0 3.10 scrub ok 2024-03-11T21:14:55.743 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:55 smithi023 bash[29146]: cluster 2024-03-11T21:14:54.437366+0000 mgr.y (mgr.24380) 891 : cluster 0 pgmap v1029: 158 pgs: 2 activating, 6 remapped+peering, 6 peering, 1 active+clean+scrubbing+deep, 1 active+recovering+degraded+remapped, 142 active+clean; 1.9 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 744 op/s; 1400/95715 objects degraded (1.463%); 2.2 MiB/s, 43 objects/s recovering 2024-03-11T21:14:55.743 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:55 smithi023 bash[21372]: cluster 2024-03-11T21:14:54.051009+0000 osd.6 (osd.6) 444 : cluster 0 3.10 scrub starts 2024-03-11T21:14:55.743 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:55 smithi023 bash[21372]: cluster 2024-03-11T21:14:54.058796+0000 osd.6 (osd.6) 445 : cluster 0 3.10 scrub ok 2024-03-11T21:14:55.744 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:55 smithi023 bash[21372]: cluster 2024-03-11T21:14:54.437366+0000 mgr.y (mgr.24380) 891 : cluster 0 pgmap v1029: 158 pgs: 2 activating, 6 remapped+peering, 6 peering, 1 active+clean+scrubbing+deep, 1 active+recovering+degraded+remapped, 142 active+clean; 1.9 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 744 op/s; 1400/95715 objects degraded (1.463%); 2.2 MiB/s, 43 objects/s recovering 2024-03-11T21:14:55.744 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:55 smithi023 bash[21372]: cluster 2024-03-11T21:14:55.400748+0000 mon.a (mon.0) 1929 : cluster 3 Health check update: Degraded data redundancy: 1400/95715 objects degraded (1.463%), 1 pg degraded (PG_DEGRADED) 2024-03-11T21:14:55.825 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:14:55.843 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:14:55.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:55.844 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:14:55.945 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:55.985 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:55.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:55.994 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:14:56.094 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:56.098 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:55 smithi023 bash[29146]: cluster 2024-03-11T21:14:55.400748+0000 mon.a (mon.0) 1929 : cluster 3 Health check update: Degraded data redundancy: 1400/95715 objects degraded (1.463%), 1 pg degraded (PG_DEGRADED) 2024-03-11T21:14:56.151 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:56.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:56.154 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:14:56.254 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:14:56 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:14:56.255 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:56.289 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:56.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:56.292 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:14:56.393 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:56.419 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:56.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:56.426 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:14:56.526 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:14:56.552 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:14:56.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:56.559 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:14:56.659 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:56.687 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:56.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:56.693 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:14:56.754 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:56 smithi146 bash[23809]: cluster 2024-03-11T21:14:54.595057+0000 osd.5 (osd.5) 531 : cluster 0 3.c scrub starts 2024-03-11T21:14:56.754 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:56 smithi146 bash[23809]: cluster 2024-03-11T21:14:54.597290+0000 osd.5 (osd.5) 532 : cluster 0 3.c scrub ok 2024-03-11T21:14:56.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:56 smithi023 bash[21372]: cluster 2024-03-11T21:14:54.595057+0000 osd.5 (osd.5) 531 : cluster 0 3.c scrub starts 2024-03-11T21:14:56.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:56 smithi023 bash[21372]: cluster 2024-03-11T21:14:54.597290+0000 osd.5 (osd.5) 532 : cluster 0 3.c scrub ok 2024-03-11T21:14:56.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:56 smithi023 bash[29146]: cluster 2024-03-11T21:14:54.595057+0000 osd.5 (osd.5) 531 : cluster 0 3.c scrub starts 2024-03-11T21:14:56.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:56 smithi023 bash[29146]: cluster 2024-03-11T21:14:54.597290+0000 osd.5 (osd.5) 532 : cluster 0 3.c scrub ok 2024-03-11T21:14:56.794 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:56.820 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:56.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:56.822 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:14:56.923 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:14:56.966 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:14:56.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:56.968 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:14:57.069 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:14:57.095 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:14:57.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:57.103 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:14:57.122 INFO:tasks.thrashosds.thrasher:Added osd 0 2024-03-11T21:14:57.203 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:57.231 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:57.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:57.236 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:14:57.337 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:57.361 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:57.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:57.365 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:14:57.465 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:14:57.473 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:57 smithi146 bash[23809]: audit 2024-03-11T21:14:56.068998+0000 mgr.y (mgr.24380) 892 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:14:57.473 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:57 smithi146 bash[23809]: cluster 2024-03-11T21:14:56.230207+0000 osd.7 (osd.7) 555 : cluster 0 2.5 scrub starts 2024-03-11T21:14:57.473 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:57 smithi146 bash[23809]: cluster 2024-03-11T21:14:56.231550+0000 osd.7 (osd.7) 556 : cluster 0 2.5 scrub ok 2024-03-11T21:14:57.473 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:57 smithi146 bash[23809]: cluster 2024-03-11T21:14:56.438557+0000 mgr.y (mgr.24380) 893 : cluster 0 pgmap v1030: 158 pgs: 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 153 active+clean; 1.8 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 874 op/s; 1400/88968 objects degraded (1.574%); 6516/88968 objects misplaced (7.324%); 9.3 MiB/s, 155 objects/s recovering 2024-03-11T21:14:57.473 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:57 smithi146 bash[23809]: cluster 2024-03-11T21:14:56.461354+0000 mon.a (mon.0) 1930 : cluster 3 Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-11T21:14:57.484 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:14:57.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:57.486 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:14:57.587 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:57.626 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:57.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:57.628 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:14:57.729 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:57.753 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:57.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:57.756 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:14:57.856 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:14:57.866 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:57 smithi023 bash[21372]: audit 2024-03-11T21:14:56.068998+0000 mgr.y (mgr.24380) 892 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:14:57.866 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:57 smithi023 bash[21372]: cluster 2024-03-11T21:14:56.230207+0000 osd.7 (osd.7) 555 : cluster 0 2.5 scrub starts 2024-03-11T21:14:57.866 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:57 smithi023 bash[21372]: cluster 2024-03-11T21:14:56.231550+0000 osd.7 (osd.7) 556 : cluster 0 2.5 scrub ok 2024-03-11T21:14:57.866 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:57 smithi023 bash[21372]: cluster 2024-03-11T21:14:56.438557+0000 mgr.y (mgr.24380) 893 : cluster 0 pgmap v1030: 158 pgs: 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 153 active+clean; 1.8 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 874 op/s; 1400/88968 objects degraded (1.574%); 6516/88968 objects misplaced (7.324%); 9.3 MiB/s, 155 objects/s recovering 2024-03-11T21:14:57.866 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:57 smithi023 bash[21372]: cluster 2024-03-11T21:14:56.461354+0000 mon.a (mon.0) 1930 : cluster 3 Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-11T21:14:57.866 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:57 smithi023 bash[29146]: audit 2024-03-11T21:14:56.068998+0000 mgr.y (mgr.24380) 892 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:14:57.866 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:57 smithi023 bash[29146]: cluster 2024-03-11T21:14:56.230207+0000 osd.7 (osd.7) 555 : cluster 0 2.5 scrub starts 2024-03-11T21:14:57.866 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:57 smithi023 bash[29146]: cluster 2024-03-11T21:14:56.231550+0000 osd.7 (osd.7) 556 : cluster 0 2.5 scrub ok 2024-03-11T21:14:57.867 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:57 smithi023 bash[29146]: cluster 2024-03-11T21:14:56.438557+0000 mgr.y (mgr.24380) 893 : cluster 0 pgmap v1030: 158 pgs: 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 153 active+clean; 1.8 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 874 op/s; 1400/88968 objects degraded (1.574%); 6516/88968 objects misplaced (7.324%); 9.3 MiB/s, 155 objects/s recovering 2024-03-11T21:14:57.867 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:57 smithi023 bash[29146]: cluster 2024-03-11T21:14:56.461354+0000 mon.a (mon.0) 1930 : cluster 3 Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-11T21:14:57.883 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:14:57.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:57.891 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:14:57.991 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:14:58.028 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:14:58.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:58.029 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:14:58.130 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:58.170 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:58.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:58.174 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:14:58.274 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:14:58.299 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:14:58.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:58.306 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:14:58.407 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:14:58.417 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:14:58 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:14:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:14:58.440 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:14:58.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:58.444 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:14:58.545 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:14:58.552 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:58 smithi146 bash[23809]: cluster 2024-03-11T21:14:56.960146+0000 osd.6 (osd.6) 446 : cluster 0 2.17 scrub starts 2024-03-11T21:14:58.552 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:58 smithi146 bash[23809]: cluster 2024-03-11T21:14:56.962539+0000 osd.6 (osd.6) 447 : cluster 0 2.17 scrub ok 2024-03-11T21:14:58.567 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:14:58.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:58.571 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:14:58.672 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:58.697 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:58.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:58.700 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:14:58.799 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:58 smithi023 bash[21372]: cluster 2024-03-11T21:14:56.960146+0000 osd.6 (osd.6) 446 : cluster 0 2.17 scrub starts 2024-03-11T21:14:58.799 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:58 smithi023 bash[21372]: cluster 2024-03-11T21:14:56.962539+0000 osd.6 (osd.6) 447 : cluster 0 2.17 scrub ok 2024-03-11T21:14:58.799 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:58 smithi023 bash[29146]: cluster 2024-03-11T21:14:56.960146+0000 osd.6 (osd.6) 446 : cluster 0 2.17 scrub starts 2024-03-11T21:14:58.800 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:58 smithi023 bash[29146]: cluster 2024-03-11T21:14:56.962539+0000 osd.6 (osd.6) 447 : cluster 0 2.17 scrub ok 2024-03-11T21:14:58.801 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:58.824 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:58.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:58.828 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:14:58.929 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:58.959 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:58.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:58.964 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:14:59.064 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:59.093 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:14:59.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:59.099 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:14:59.200 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:14:59.225 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:14:59.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:59.229 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:14:59.330 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:59.351 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:14:59.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:59.356 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:14:59.456 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:59.477 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:14:59.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:59.479 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:14:59.580 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:14:59.603 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:14:59.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:59.609 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:14:59.709 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:59.735 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:14:59.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:59.738 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:14:59.754 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:14:59 smithi146 bash[23809]: cluster 2024-03-11T21:14:58.439260+0000 mgr.y (mgr.24380) 894 : cluster 0 pgmap v1031: 158 pgs: 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 153 active+clean; 1.7 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 727 B/s rd, 0 B/s wr, 555 op/s; 1150/86028 objects degraded (1.337%); 6516/86028 objects misplaced (7.574%); 10 MiB/s, 163 objects/s recovering 2024-03-11T21:14:59.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:14:59 smithi023 bash[21372]: cluster 2024-03-11T21:14:58.439260+0000 mgr.y (mgr.24380) 894 : cluster 0 pgmap v1031: 158 pgs: 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 153 active+clean; 1.7 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 727 B/s rd, 0 B/s wr, 555 op/s; 1150/86028 objects degraded (1.337%); 6516/86028 objects misplaced (7.574%); 10 MiB/s, 163 objects/s recovering 2024-03-11T21:14:59.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:14:59 smithi023 bash[29146]: cluster 2024-03-11T21:14:58.439260+0000 mgr.y (mgr.24380) 894 : cluster 0 pgmap v1031: 158 pgs: 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 153 active+clean; 1.7 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 727 B/s rd, 0 B/s wr, 555 op/s; 1150/86028 objects degraded (1.337%); 6516/86028 objects misplaced (7.574%); 10 MiB/s, 163 objects/s recovering 2024-03-11T21:14:59.839 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:14:59.876 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:14:59.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:14:59.879 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:14:59.980 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:15:00.005 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:15:00.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:00.008 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:15:00.109 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:15:00.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:00.135 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:15:00.136 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:15:00.236 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:15:00.276 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:15:00.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:00.283 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:15:00.383 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:15:00.415 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:15:00.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:00.420 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:15:00.521 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:15:00.550 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:15:00.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:00.553 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:15:00.654 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:15:00.693 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:15:00.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:00.696 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:15:00.797 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:00.807 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:00 smithi146 bash[23809]: cluster 2024-03-11T21:14:58.652865+0000 osd.5 (osd.5) 533 : cluster 0 4.8 scrub starts 2024-03-11T21:15:00.807 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:00 smithi146 bash[23809]: cluster 2024-03-11T21:14:58.654225+0000 osd.5 (osd.5) 534 : cluster 0 4.8 scrub ok 2024-03-11T21:15:00.807 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:00 smithi146 bash[23809]: cluster 2024-03-11T21:14:58.803880+0000 osd.5 (osd.5) 535 : cluster 0 purged_snaps scrub starts 2024-03-11T21:15:00.807 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:00 smithi146 bash[23809]: cluster 2024-03-11T21:14:58.803947+0000 osd.5 (osd.5) 536 : cluster 0 purged_snaps scrub ok 2024-03-11T21:15:00.807 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:00 smithi146 bash[23809]: cluster 2024-03-11T21:14:59.261543+0000 osd.2 (osd.2) 261 : cluster 0 purged_snaps scrub starts 2024-03-11T21:15:00.807 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:00 smithi146 bash[23809]: cluster 2024-03-11T21:14:59.261589+0000 osd.2 (osd.2) 262 : cluster 0 purged_snaps scrub ok 2024-03-11T21:15:00.807 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:00 smithi146 bash[23809]: cluster 2024-03-11T21:14:59.513870+0000 mon.a (mon.0) 1931 : cluster 0 osdmap e428: 8 total, 8 up, 8 in 2024-03-11T21:15:00.807 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:00 smithi146 bash[23809]: cluster 2024-03-11T21:15:00.401951+0000 mon.a (mon.0) 1932 : cluster 3 Health check update: Degraded data redundancy: 1150/86028 objects degraded (1.337%), 1 pg degraded (PG_DEGRADED) 2024-03-11T21:15:00.820 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:00.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:00.822 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:15:00.923 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:15:00.935 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:00 smithi023 bash[21372]: cluster 2024-03-11T21:14:58.652865+0000 osd.5 (osd.5) 533 : cluster 0 4.8 scrub starts 2024-03-11T21:15:00.936 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:00 smithi023 bash[21372]: cluster 2024-03-11T21:14:58.654225+0000 osd.5 (osd.5) 534 : cluster 0 4.8 scrub ok 2024-03-11T21:15:00.936 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:00 smithi023 bash[21372]: cluster 2024-03-11T21:14:58.803880+0000 osd.5 (osd.5) 535 : cluster 0 purged_snaps scrub starts 2024-03-11T21:15:00.936 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:00 smithi023 bash[21372]: cluster 2024-03-11T21:14:58.803947+0000 osd.5 (osd.5) 536 : cluster 0 purged_snaps scrub ok 2024-03-11T21:15:00.936 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:00 smithi023 bash[21372]: cluster 2024-03-11T21:14:59.261543+0000 osd.2 (osd.2) 261 : cluster 0 purged_snaps scrub starts 2024-03-11T21:15:00.936 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:00 smithi023 bash[21372]: cluster 2024-03-11T21:14:59.261589+0000 osd.2 (osd.2) 262 : cluster 0 purged_snaps scrub ok 2024-03-11T21:15:00.936 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:00 smithi023 bash[21372]: cluster 2024-03-11T21:14:59.513870+0000 mon.a (mon.0) 1931 : cluster 0 osdmap e428: 8 total, 8 up, 8 in 2024-03-11T21:15:00.936 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:00 smithi023 bash[21372]: cluster 2024-03-11T21:15:00.401951+0000 mon.a (mon.0) 1932 : cluster 3 Health check update: Degraded data redundancy: 1150/86028 objects degraded (1.337%), 1 pg degraded (PG_DEGRADED) 2024-03-11T21:15:00.936 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:00 smithi023 bash[29146]: cluster 2024-03-11T21:14:58.652865+0000 osd.5 (osd.5) 533 : cluster 0 4.8 scrub starts 2024-03-11T21:15:00.936 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:00 smithi023 bash[29146]: cluster 2024-03-11T21:14:58.654225+0000 osd.5 (osd.5) 534 : cluster 0 4.8 scrub ok 2024-03-11T21:15:00.936 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:00 smithi023 bash[29146]: cluster 2024-03-11T21:14:58.803880+0000 osd.5 (osd.5) 535 : cluster 0 purged_snaps scrub starts 2024-03-11T21:15:00.936 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:00 smithi023 bash[29146]: cluster 2024-03-11T21:14:58.803947+0000 osd.5 (osd.5) 536 : cluster 0 purged_snaps scrub ok 2024-03-11T21:15:00.936 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:00 smithi023 bash[29146]: cluster 2024-03-11T21:14:59.261543+0000 osd.2 (osd.2) 261 : cluster 0 purged_snaps scrub starts 2024-03-11T21:15:00.937 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:00 smithi023 bash[29146]: cluster 2024-03-11T21:14:59.261589+0000 osd.2 (osd.2) 262 : cluster 0 purged_snaps scrub ok 2024-03-11T21:15:00.937 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:00 smithi023 bash[29146]: cluster 2024-03-11T21:14:59.513870+0000 mon.a (mon.0) 1931 : cluster 0 osdmap e428: 8 total, 8 up, 8 in 2024-03-11T21:15:00.937 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:00 smithi023 bash[29146]: cluster 2024-03-11T21:15:00.401951+0000 mon.a (mon.0) 1932 : cluster 3 Health check update: Degraded data redundancy: 1150/86028 objects degraded (1.337%), 1 pg degraded (PG_DEGRADED) 2024-03-11T21:15:00.956 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:15:00.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:00.961 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:15:01.062 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:15:01.080 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:15:01.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:01.082 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:15:01.183 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:15:01.210 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:15:01.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:01.212 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:15:01.313 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:01.339 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:01.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:01.341 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:15:01.442 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:15:01.460 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:15:01.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:01.462 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:15:01.563 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:15:01.587 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:15:01.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:01.590 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:15:01.690 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:15:01.727 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:15:01.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:01.731 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:15:01.736 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:01 smithi023 bash[21372]: cluster 2024-03-11T21:15:00.009731+0000 osd.1 (osd.1) 302 : cluster 0 purged_snaps scrub starts 2024-03-11T21:15:01.736 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:01 smithi023 bash[21372]: cluster 2024-03-11T21:15:00.009807+0000 osd.1 (osd.1) 303 : cluster 0 purged_snaps scrub ok 2024-03-11T21:15:01.736 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:01 smithi023 bash[21372]: cluster 2024-03-11T21:15:00.439903+0000 mgr.y (mgr.24380) 895 : cluster 0 pgmap v1033: 158 pgs: 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 153 active+clean; 1.7 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 584 op/s; 1150/83706 objects degraded (1.374%); 6516/83706 objects misplaced (7.784%); 8.9 MiB/s, 143 objects/s recovering 2024-03-11T21:15:01.736 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:01 smithi023 bash[21372]: cluster 2024-03-11T21:15:00.533615+0000 mon.a (mon.0) 1933 : cluster 0 osdmap e429: 8 total, 8 up, 8 in 2024-03-11T21:15:01.737 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:01 smithi023 bash[29146]: cluster 2024-03-11T21:15:00.009731+0000 osd.1 (osd.1) 302 : cluster 0 purged_snaps scrub starts 2024-03-11T21:15:01.737 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:01 smithi023 bash[29146]: cluster 2024-03-11T21:15:00.009807+0000 osd.1 (osd.1) 303 : cluster 0 purged_snaps scrub ok 2024-03-11T21:15:01.737 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:01 smithi023 bash[29146]: cluster 2024-03-11T21:15:00.439903+0000 mgr.y (mgr.24380) 895 : cluster 0 pgmap v1033: 158 pgs: 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 153 active+clean; 1.7 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 584 op/s; 1150/83706 objects degraded (1.374%); 6516/83706 objects misplaced (7.784%); 8.9 MiB/s, 143 objects/s recovering 2024-03-11T21:15:01.737 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:01 smithi023 bash[29146]: cluster 2024-03-11T21:15:00.533615+0000 mon.a (mon.0) 1933 : cluster 0 osdmap e429: 8 total, 8 up, 8 in 2024-03-11T21:15:01.832 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:15:01.854 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:15:01.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:01.856 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:15:01.957 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:15:01.970 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:01 smithi146 bash[23809]: cluster 2024-03-11T21:15:00.009731+0000 osd.1 (osd.1) 302 : cluster 0 purged_snaps scrub starts 2024-03-11T21:15:01.970 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:01 smithi146 bash[23809]: cluster 2024-03-11T21:15:00.009807+0000 osd.1 (osd.1) 303 : cluster 0 purged_snaps scrub ok 2024-03-11T21:15:01.970 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:01 smithi146 bash[23809]: cluster 2024-03-11T21:15:00.439903+0000 mgr.y (mgr.24380) 895 : cluster 0 pgmap v1033: 158 pgs: 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 153 active+clean; 1.7 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 584 op/s; 1150/83706 objects degraded (1.374%); 6516/83706 objects misplaced (7.784%); 8.9 MiB/s, 143 objects/s recovering 2024-03-11T21:15:01.970 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:01 smithi146 bash[23809]: cluster 2024-03-11T21:15:00.533615+0000 mon.a (mon.0) 1933 : cluster 0 osdmap e429: 8 total, 8 up, 8 in 2024-03-11T21:15:01.987 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:15:01.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:01.992 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:15:02.092 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:15:02.124 INFO:tasks.thrashosds.thrasher:in_osds: [2, 3, 4, 5, 6, 7, 1, 0] out_osds: [] dead_osds: [] live_osds: [0, 2, 3, 5, 6, 7, 4, 1] 2024-03-11T21:15:02.124 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-11T21:15:02.124 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-03-11T21:15:02.124 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd dump -f json-pretty 2024-03-11T21:15:02.125 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:15:02.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:02.132 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:15:02.233 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:15:02.275 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:15:02.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:02.279 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:15:02.379 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:15:02.405 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:15:02.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:02.409 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:15:02.510 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:15:02.557 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:15:02.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:02.566 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:15:02.667 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:15:02.708 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-11T21:15:02.709 INFO:teuthology.orchestra.run.smithi023.stdout: "ops": [ 2024-03-11T21:15:02.709 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:02.709 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_repop(client.15498.0:171787 11.d e429/380 11:bbadea01:::benchmark_data_smithi023_142591_object74143:head v 429'10754, mlcod=429'10751)", 2024-03-11T21:15:02.709 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T21:15:02.715094+0000", 2024-03-11T21:15:02.709 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 0.0063030300000000003, 2024-03-11T21:15:02.709 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0063258280000000004, 2024-03-11T21:15:02.709 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:15:02.709 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "started", 2024-03-11T21:15:02.709 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:15:02.709 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:02.709 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:15:02.709 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:15:02.715094+0000", 2024-03-11T21:15:02.709 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:15:02.710 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:02.710 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:02.710 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:15:02.710 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:15:02.715094+0000", 2024-03-11T21:15:02.710 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:15:02.710 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:02.710 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:02.710 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:15:02.710 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:15:02.715095+0000", 2024-03-11T21:15:02.710 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.7999999999999995e-07 2024-03-11T21:15:02.710 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:02.710 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:02.710 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:15:02.710 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:15:02.715101+0000", 2024-03-11T21:15:02.710 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 5.9259999999999997e-06 2024-03-11T21:15:02.711 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:02.711 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:02.711 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:15:02.711 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:15:02.715112+0000", 2024-03-11T21:15:02.711 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.0564000000000001e-05 2024-03-11T21:15:02.711 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:02.711 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:02.711 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:15:02.711 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:15:02.715119+0000", 2024-03-11T21:15:02.711 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.8549999999999996e-06 2024-03-11T21:15:02.711 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:02.711 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:02.711 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:15:02.711 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:15:02.721256+0000", 2024-03-11T21:15:02.711 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0061372950000000001 2024-03-11T21:15:02.712 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:02.712 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:02.712 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:15:02.712 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:15:02.721276+0000", 2024-03-11T21:15:02.712 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.9984999999999999e-05 2024-03-11T21:15:02.712 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:15:02.712 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:15:02.712 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:15:02.712 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:15:02.712 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-11T21:15:02.712 INFO:teuthology.orchestra.run.smithi023.stdout: "num_ops": 1 2024-03-11T21:15:02.712 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-11T21:15:02.713 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:15:02.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:02.714 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:15:02.754 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:02 smithi146 bash[23809]: cluster 2024-03-11T21:15:00.686679+0000 osd.5 (osd.5) 537 : cluster 0 6.1a scrub starts 2024-03-11T21:15:02.754 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:02 smithi146 bash[23809]: cluster 2024-03-11T21:15:00.705902+0000 osd.5 (osd.5) 538 : cluster 0 6.1a scrub ok 2024-03-11T21:15:02.815 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:15:02.824 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:02 smithi023 bash[21372]: cluster 2024-03-11T21:15:00.686679+0000 osd.5 (osd.5) 537 : cluster 0 6.1a scrub starts 2024-03-11T21:15:02.824 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:02 smithi023 bash[21372]: cluster 2024-03-11T21:15:00.705902+0000 osd.5 (osd.5) 538 : cluster 0 6.1a scrub ok 2024-03-11T21:15:02.825 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:02 smithi023 bash[29146]: cluster 2024-03-11T21:15:00.686679+0000 osd.5 (osd.5) 537 : cluster 0 6.1a scrub starts 2024-03-11T21:15:02.825 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:02 smithi023 bash[29146]: cluster 2024-03-11T21:15:00.705902+0000 osd.5 (osd.5) 538 : cluster 0 6.1a scrub ok 2024-03-11T21:15:02.846 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:15:02.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:02.850 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:15:02.950 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:15:02.974 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:15:02.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:02.980 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:15:03.081 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:03.103 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:03.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:03.111 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:15:03.212 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:15:03.230 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:15:03.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:03.232 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:15:03.332 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:15:03.356 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:15:03.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:03.358 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:15:03.459 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:03.483 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:03.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:03.485 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:15:03.586 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:15:03.619 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:15:03.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:03.626 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:15:03.727 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:15:03.754 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:03 smithi146 bash[23809]: cluster 2024-03-11T21:15:02.213005+0000 osd.7 (osd.7) 557 : cluster 0 3.e scrub starts 2024-03-11T21:15:03.754 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:03 smithi146 bash[23809]: cluster 2024-03-11T21:15:02.224027+0000 osd.7 (osd.7) 558 : cluster 0 3.e scrub ok 2024-03-11T21:15:03.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:03 smithi146 bash[23809]: cluster 2024-03-11T21:15:02.440753+0000 mgr.y (mgr.24380) 896 : cluster 0 pgmap v1035: 158 pgs: 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 153 active+clean; 1.5 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 800 op/s; 1150/76521 objects degraded (1.503%); 4703/76521 objects misplaced (6.146%); 15 MiB/s, 243 objects/s recovering 2024-03-11T21:15:03.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:03 smithi146 bash[23809]: cluster 2024-03-11T21:15:02.721897+0000 osd.5 (osd.5) 539 : cluster 0 6.19 scrub starts 2024-03-11T21:15:03.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:03 smithi146 bash[23809]: cluster 2024-03-11T21:15:02.818188+0000 osd.5 (osd.5) 540 : cluster 0 6.19 scrub ok 2024-03-11T21:15:03.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:03 smithi146 bash[23809]: cluster 2024-03-11T21:15:03.074488+0000 osd.6 (osd.6) 448 : cluster 0 purged_snaps scrub starts 2024-03-11T21:15:03.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:03 smithi146 bash[23809]: cluster 2024-03-11T21:15:03.074599+0000 osd.6 (osd.6) 449 : cluster 0 purged_snaps scrub ok 2024-03-11T21:15:03.756 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:15:03.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:03.763 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:15:03.864 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:15:03.898 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:15:03.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:03.901 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:15:04.002 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:15:04.022 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:15:04.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:04.024 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:15:04.125 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:15:04.148 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:15:04.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:04.150 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:15:04.251 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:15:04.282 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:15:04.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:04.289 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:15:04.390 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:15:04.430 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:15:04.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:04.431 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:15:04.452 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:04 smithi023 bash[21372]: cluster 2024-03-11T21:15:02.213005+0000 osd.7 (osd.7) 557 : cluster 0 3.e scrub starts 2024-03-11T21:15:04.452 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:04 smithi023 bash[21372]: cluster 2024-03-11T21:15:02.224027+0000 osd.7 (osd.7) 558 : cluster 0 3.e scrub ok 2024-03-11T21:15:04.453 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:04 smithi023 bash[21372]: cluster 2024-03-11T21:15:02.440753+0000 mgr.y (mgr.24380) 896 : cluster 0 pgmap v1035: 158 pgs: 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 153 active+clean; 1.5 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 800 op/s; 1150/76521 objects degraded (1.503%); 4703/76521 objects misplaced (6.146%); 15 MiB/s, 243 objects/s recovering 2024-03-11T21:15:04.453 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:04 smithi023 bash[21372]: cluster 2024-03-11T21:15:02.721897+0000 osd.5 (osd.5) 539 : cluster 0 6.19 scrub starts 2024-03-11T21:15:04.453 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:04 smithi023 bash[21372]: cluster 2024-03-11T21:15:02.818188+0000 osd.5 (osd.5) 540 : cluster 0 6.19 scrub ok 2024-03-11T21:15:04.453 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:04 smithi023 bash[21372]: cluster 2024-03-11T21:15:03.074488+0000 osd.6 (osd.6) 448 : cluster 0 purged_snaps scrub starts 2024-03-11T21:15:04.453 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:04 smithi023 bash[21372]: cluster 2024-03-11T21:15:03.074599+0000 osd.6 (osd.6) 449 : cluster 0 purged_snaps scrub ok 2024-03-11T21:15:04.532 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:15:04.572 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:15:04.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:04.577 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:15:04.678 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:15:04.689 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:04 smithi146 bash[23809]: cluster 2024-03-11T21:15:03.590557+0000 mon.a (mon.0) 1934 : cluster 0 osdmap e430: 8 total, 8 up, 8 in 2024-03-11T21:15:04.709 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:15:04.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:04.712 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:15:04.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:04 smithi023 bash[21372]: cluster 2024-03-11T21:15:03.590557+0000 mon.a (mon.0) 1934 : cluster 0 osdmap e430: 8 total, 8 up, 8 in 2024-03-11T21:15:04.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:04 smithi023 bash[29146]: cluster 2024-03-11T21:15:02.213005+0000 osd.7 (osd.7) 557 : cluster 0 3.e scrub starts 2024-03-11T21:15:04.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:04 smithi023 bash[29146]: cluster 2024-03-11T21:15:02.224027+0000 osd.7 (osd.7) 558 : cluster 0 3.e scrub ok 2024-03-11T21:15:04.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:04 smithi023 bash[29146]: cluster 2024-03-11T21:15:02.440753+0000 mgr.y (mgr.24380) 896 : cluster 0 pgmap v1035: 158 pgs: 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 153 active+clean; 1.5 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 800 op/s; 1150/76521 objects degraded (1.503%); 4703/76521 objects misplaced (6.146%); 15 MiB/s, 243 objects/s recovering 2024-03-11T21:15:04.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:04 smithi023 bash[29146]: cluster 2024-03-11T21:15:02.721897+0000 osd.5 (osd.5) 539 : cluster 0 6.19 scrub starts 2024-03-11T21:15:04.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:04 smithi023 bash[29146]: cluster 2024-03-11T21:15:02.818188+0000 osd.5 (osd.5) 540 : cluster 0 6.19 scrub ok 2024-03-11T21:15:04.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:04 smithi023 bash[29146]: cluster 2024-03-11T21:15:03.074488+0000 osd.6 (osd.6) 448 : cluster 0 purged_snaps scrub starts 2024-03-11T21:15:04.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:04 smithi023 bash[29146]: cluster 2024-03-11T21:15:03.074599+0000 osd.6 (osd.6) 449 : cluster 0 purged_snaps scrub ok 2024-03-11T21:15:04.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:04 smithi023 bash[29146]: cluster 2024-03-11T21:15:03.590557+0000 mon.a (mon.0) 1934 : cluster 0 osdmap e430: 8 total, 8 up, 8 in 2024-03-11T21:15:04.813 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:04.843 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:04.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:04.847 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:15:04.948 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:15:04.968 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:15:04.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:04.970 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:15:05.071 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:15:05.093 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:15:05.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:05.095 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:15:05.196 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:05.227 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:05.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:05.232 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:15:05.333 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:15:05.370 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:15:05.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:05.372 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:15:05.473 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:15:05.499 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:15:05.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:05.501 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:15:05.602 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:15:05.632 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:15:05.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:05.634 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:15:05.735 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:15:05.743 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:05 smithi146 bash[23809]: cluster 2024-03-11T21:15:04.237148+0000 osd.7 (osd.7) 559 : cluster 0 6.8 scrub starts 2024-03-11T21:15:05.743 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:05 smithi146 bash[23809]: cluster 2024-03-11T21:15:04.238986+0000 osd.7 (osd.7) 560 : cluster 0 6.8 scrub ok 2024-03-11T21:15:05.743 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:05 smithi146 bash[23809]: cluster 2024-03-11T21:15:04.441262+0000 mgr.y (mgr.24380) 897 : cluster 0 pgmap v1037: 158 pgs: 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 153 active+clean; 1.5 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 677 op/s; 907/73848 objects degraded (1.228%); 4703/73848 objects misplaced (6.368%); 12 MiB/s, 197 objects/s recovering 2024-03-11T21:15:05.743 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:05 smithi146 bash[23809]: cluster 2024-03-11T21:15:05.403385+0000 mon.a (mon.0) 1935 : cluster 3 Health check update: Degraded data redundancy: 907/73848 objects degraded (1.228%), 1 pg degraded (PG_DEGRADED) 2024-03-11T21:15:05.765 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:15:05.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:05.768 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:15:05.869 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:15:05.880 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:05 smithi023 bash[21372]: cluster 2024-03-11T21:15:04.237148+0000 osd.7 (osd.7) 559 : cluster 0 6.8 scrub starts 2024-03-11T21:15:05.880 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:05 smithi023 bash[21372]: cluster 2024-03-11T21:15:04.238986+0000 osd.7 (osd.7) 560 : cluster 0 6.8 scrub ok 2024-03-11T21:15:05.880 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:05 smithi023 bash[21372]: cluster 2024-03-11T21:15:04.441262+0000 mgr.y (mgr.24380) 897 : cluster 0 pgmap v1037: 158 pgs: 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 153 active+clean; 1.5 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 677 op/s; 907/73848 objects degraded (1.228%); 4703/73848 objects misplaced (6.368%); 12 MiB/s, 197 objects/s recovering 2024-03-11T21:15:05.880 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:05 smithi023 bash[21372]: cluster 2024-03-11T21:15:05.403385+0000 mon.a (mon.0) 1935 : cluster 3 Health check update: Degraded data redundancy: 907/73848 objects degraded (1.228%), 1 pg degraded (PG_DEGRADED) 2024-03-11T21:15:05.880 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:05 smithi023 bash[29146]: cluster 2024-03-11T21:15:04.237148+0000 osd.7 (osd.7) 559 : cluster 0 6.8 scrub starts 2024-03-11T21:15:05.880 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:05 smithi023 bash[29146]: cluster 2024-03-11T21:15:04.238986+0000 osd.7 (osd.7) 560 : cluster 0 6.8 scrub ok 2024-03-11T21:15:05.880 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:05 smithi023 bash[29146]: cluster 2024-03-11T21:15:04.441262+0000 mgr.y (mgr.24380) 897 : cluster 0 pgmap v1037: 158 pgs: 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 153 active+clean; 1.5 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 677 op/s; 907/73848 objects degraded (1.228%); 4703/73848 objects misplaced (6.368%); 12 MiB/s, 197 objects/s recovering 2024-03-11T21:15:05.881 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:05 smithi023 bash[29146]: cluster 2024-03-11T21:15:05.403385+0000 mon.a (mon.0) 1935 : cluster 3 Health check update: Degraded data redundancy: 907/73848 objects degraded (1.228%), 1 pg degraded (PG_DEGRADED) 2024-03-11T21:15:05.904 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:15:05.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:05.910 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:15:05.911 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-03-11T21:15:06.011 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:15:06.035 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:15:06.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:06.040 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:15:06.141 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:15:06.187 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:15:06.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:06.191 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:15:06.292 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:15:06.303 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:15:06 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:15:06.318 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:15:06.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:06.321 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:15:06.422 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:15:06.445 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:15:06.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:06.448 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:15:06.548 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:15:06.573 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:15:06.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:06.578 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:15:06.679 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:15:06.715 INFO:teuthology.orchestra.run.smithi023.stdout:osd.0: {} 2024-03-11T21:15:06.716 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:15:06.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:06.719 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:15:06.789 INFO:teuthology.orchestra.run.smithi023.stdout:osd.1: {} 2024-03-11T21:15:06.820 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:15:06.830 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:06 smithi023 bash[21372]: cluster 2024-03-11T21:15:05.127231+0000 osd.7 (osd.7) 561 : cluster 0 3.11 scrub starts 2024-03-11T21:15:06.830 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:06 smithi023 bash[21372]: cluster 2024-03-11T21:15:05.131136+0000 osd.7 (osd.7) 562 : cluster 0 3.11 scrub ok 2024-03-11T21:15:06.830 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:06 smithi023 bash[21372]: cluster 2024-03-11T21:15:05.721247+0000 osd.5 (osd.5) 541 : cluster 0 2.12 scrub starts 2024-03-11T21:15:06.831 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:06 smithi023 bash[21372]: cluster 2024-03-11T21:15:05.725646+0000 osd.5 (osd.5) 542 : cluster 0 2.12 scrub ok 2024-03-11T21:15:06.831 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:06 smithi023 bash[21372]: audit 2024-03-11T21:15:06.517931+0000 mon.a (mon.0) 1936 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:15:06.831 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:06 smithi023 bash[21372]: audit 2024-03-11T21:15:06.522393+0000 mon.a (mon.0) 1937 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:15:06.831 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:06 smithi023 bash[29146]: cluster 2024-03-11T21:15:05.127231+0000 osd.7 (osd.7) 561 : cluster 0 3.11 scrub starts 2024-03-11T21:15:06.831 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:06 smithi023 bash[29146]: cluster 2024-03-11T21:15:05.131136+0000 osd.7 (osd.7) 562 : cluster 0 3.11 scrub ok 2024-03-11T21:15:06.831 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:06 smithi023 bash[29146]: cluster 2024-03-11T21:15:05.721247+0000 osd.5 (osd.5) 541 : cluster 0 2.12 scrub starts 2024-03-11T21:15:06.831 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:06 smithi023 bash[29146]: cluster 2024-03-11T21:15:05.725646+0000 osd.5 (osd.5) 542 : cluster 0 2.12 scrub ok 2024-03-11T21:15:06.831 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:06 smithi023 bash[29146]: audit 2024-03-11T21:15:06.517931+0000 mon.a (mon.0) 1936 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:15:06.831 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:06 smithi023 bash[29146]: audit 2024-03-11T21:15:06.522393+0000 mon.a (mon.0) 1937 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:15:06.842 INFO:teuthology.orchestra.run.smithi023.stdout:osd.2: {} 2024-03-11T21:15:06.872 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:15:06.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:06.877 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:15:06.880 INFO:teuthology.orchestra.run.smithi023.stdout:osd.3: {} 2024-03-11T21:15:06.924 INFO:teuthology.orchestra.run.smithi023.stdout:osd.4: {} 2024-03-11T21:15:06.963 INFO:teuthology.orchestra.run.smithi023.stdout:osd.5: {} 2024-03-11T21:15:06.978 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:15:07.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:06 smithi146 bash[23809]: cluster 2024-03-11T21:15:05.127231+0000 osd.7 (osd.7) 561 : cluster 0 3.11 scrub starts 2024-03-11T21:15:07.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:06 smithi146 bash[23809]: cluster 2024-03-11T21:15:05.131136+0000 osd.7 (osd.7) 562 : cluster 0 3.11 scrub ok 2024-03-11T21:15:07.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:06 smithi146 bash[23809]: cluster 2024-03-11T21:15:05.721247+0000 osd.5 (osd.5) 541 : cluster 0 2.12 scrub starts 2024-03-11T21:15:07.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:06 smithi146 bash[23809]: cluster 2024-03-11T21:15:05.725646+0000 osd.5 (osd.5) 542 : cluster 0 2.12 scrub ok 2024-03-11T21:15:07.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:06 smithi146 bash[23809]: audit 2024-03-11T21:15:06.517931+0000 mon.a (mon.0) 1936 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:15:07.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:06 smithi146 bash[23809]: audit 2024-03-11T21:15:06.522393+0000 mon.a (mon.0) 1937 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:15:07.024 INFO:teuthology.orchestra.run.smithi023.stdout:osd.6: {} 2024-03-11T21:15:07.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:07.030 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:15:07.031 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:15:07.056 INFO:teuthology.orchestra.run.smithi023.stdout:osd.7: {} 2024-03-11T21:15:07.060 INFO:teuthology.orchestra.run.smithi023.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:15:07.060 INFO:teuthology.orchestra.run.smithi023.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:15:07.060 INFO:teuthology.orchestra.run.smithi023.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:15:07.061 INFO:teuthology.orchestra.run.smithi023.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:15:07.061 INFO:teuthology.orchestra.run.smithi023.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:15:07.061 INFO:teuthology.orchestra.run.smithi023.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:15:07.061 INFO:teuthology.orchestra.run.smithi023.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:15:07.061 INFO:teuthology.orchestra.run.smithi023.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:15:07.131 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:15:07.155 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:15:07.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:07.157 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:15:07.258 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:15:07.286 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:15:07.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:07.290 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:15:07.391 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:07.413 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:07.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:07.416 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:15:07.516 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:15:07.518 INFO:teuthology.orchestra.run.smithi023.stderr:noscrub is set 2024-03-11T21:15:07.539 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:15:07.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:07.546 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:15:07.647 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:15:07.677 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:15:07.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:07.680 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:15:07.754 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:07 smithi146 bash[23809]: audit 2024-03-11T21:15:06.078117+0000 mgr.y (mgr.24380) 898 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:15:07.754 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:07 smithi146 bash[23809]: cluster 2024-03-11T21:15:06.133456+0000 osd.7 (osd.7) 563 : cluster 0 2.1d scrub starts 2024-03-11T21:15:07.754 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:07 smithi146 bash[23809]: cluster 2024-03-11T21:15:06.134455+0000 osd.7 (osd.7) 564 : cluster 0 2.1d scrub ok 2024-03-11T21:15:07.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:07 smithi146 bash[23809]: cluster 2024-03-11T21:15:06.442252+0000 mgr.y (mgr.24380) 899 : cluster 0 pgmap v1038: 158 pgs: 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 153 active+clean; 1.4 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 919 op/s; 907/66930 objects degraded (1.355%); 3160/66930 objects misplaced (4.721%); 20 MiB/s, 318 objects/s recovering 2024-03-11T21:15:07.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:07 smithi146 bash[23809]: audit 2024-03-11T21:15:06.656142+0000 mon.c (mon.2) 100 : audit 0 from='client.? 172.21.15.23:0/1453710666' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:15:07.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:07 smithi146 bash[23809]: audit 2024-03-11T21:15:07.064263+0000 mon.a (mon.0) 1938 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:15:07.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:07 smithi146 bash[23809]: audit 2024-03-11T21:15:07.064673+0000 mon.b (mon.1) 48 : audit 1 from='client.? 172.21.15.23:0/2217847972' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:15:07.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:07 smithi146 bash[23809]: cluster 2024-03-11T21:15:07.515034+0000 mon.a (mon.0) 1939 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:15:07.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:07 smithi146 bash[23809]: audit 2024-03-11T21:15:07.518252+0000 mon.a (mon.0) 1940 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:15:07.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:07 smithi146 bash[23809]: cluster 2024-03-11T21:15:07.523474+0000 mon.a (mon.0) 1941 : cluster 0 osdmap e431: 8 total, 8 up, 8 in 2024-03-11T21:15:07.780 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:15:07.799 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:15:07.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:07.801 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:15:07.902 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:15:07.911 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:07 smithi023 bash[21372]: audit 2024-03-11T21:15:06.078117+0000 mgr.y (mgr.24380) 898 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:15:07.911 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:07 smithi023 bash[21372]: cluster 2024-03-11T21:15:06.133456+0000 osd.7 (osd.7) 563 : cluster 0 2.1d scrub starts 2024-03-11T21:15:07.911 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:07 smithi023 bash[21372]: cluster 2024-03-11T21:15:06.134455+0000 osd.7 (osd.7) 564 : cluster 0 2.1d scrub ok 2024-03-11T21:15:07.911 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:07 smithi023 bash[21372]: cluster 2024-03-11T21:15:06.442252+0000 mgr.y (mgr.24380) 899 : cluster 0 pgmap v1038: 158 pgs: 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 153 active+clean; 1.4 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 919 op/s; 907/66930 objects degraded (1.355%); 3160/66930 objects misplaced (4.721%); 20 MiB/s, 318 objects/s recovering 2024-03-11T21:15:07.911 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:07 smithi023 bash[21372]: audit 2024-03-11T21:15:06.656142+0000 mon.c (mon.2) 100 : audit 0 from='client.? 172.21.15.23:0/1453710666' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:15:07.911 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:07 smithi023 bash[21372]: audit 2024-03-11T21:15:07.064263+0000 mon.a (mon.0) 1938 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:15:07.911 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:07 smithi023 bash[21372]: audit 2024-03-11T21:15:07.064673+0000 mon.b (mon.1) 48 : audit 1 from='client.? 172.21.15.23:0/2217847972' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:15:07.911 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:07 smithi023 bash[21372]: cluster 2024-03-11T21:15:07.515034+0000 mon.a (mon.0) 1939 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:15:07.911 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:07 smithi023 bash[21372]: audit 2024-03-11T21:15:07.518252+0000 mon.a (mon.0) 1940 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:15:07.911 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:07 smithi023 bash[21372]: cluster 2024-03-11T21:15:07.523474+0000 mon.a (mon.0) 1941 : cluster 0 osdmap e431: 8 total, 8 up, 8 in 2024-03-11T21:15:07.912 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:07 smithi023 bash[29146]: audit 2024-03-11T21:15:06.078117+0000 mgr.y (mgr.24380) 898 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:15:07.912 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:07 smithi023 bash[29146]: cluster 2024-03-11T21:15:06.133456+0000 osd.7 (osd.7) 563 : cluster 0 2.1d scrub starts 2024-03-11T21:15:07.912 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:07 smithi023 bash[29146]: cluster 2024-03-11T21:15:06.134455+0000 osd.7 (osd.7) 564 : cluster 0 2.1d scrub ok 2024-03-11T21:15:07.912 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:07 smithi023 bash[29146]: cluster 2024-03-11T21:15:06.442252+0000 mgr.y (mgr.24380) 899 : cluster 0 pgmap v1038: 158 pgs: 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 153 active+clean; 1.4 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 919 op/s; 907/66930 objects degraded (1.355%); 3160/66930 objects misplaced (4.721%); 20 MiB/s, 318 objects/s recovering 2024-03-11T21:15:07.912 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:07 smithi023 bash[29146]: audit 2024-03-11T21:15:06.656142+0000 mon.c (mon.2) 100 : audit 0 from='client.? 172.21.15.23:0/1453710666' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:15:07.912 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:07 smithi023 bash[29146]: audit 2024-03-11T21:15:07.064263+0000 mon.a (mon.0) 1938 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:15:07.912 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:07 smithi023 bash[29146]: audit 2024-03-11T21:15:07.064673+0000 mon.b (mon.1) 48 : audit 1 from='client.? 172.21.15.23:0/2217847972' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:15:07.912 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:07 smithi023 bash[29146]: cluster 2024-03-11T21:15:07.515034+0000 mon.a (mon.0) 1939 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:15:07.912 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:07 smithi023 bash[29146]: audit 2024-03-11T21:15:07.518252+0000 mon.a (mon.0) 1940 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:15:07.912 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:07 smithi023 bash[29146]: cluster 2024-03-11T21:15:07.523474+0000 mon.a (mon.0) 1941 : cluster 0 osdmap e431: 8 total, 8 up, 8 in 2024-03-11T21:15:07.923 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:15:07.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:07.925 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:15:08.026 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:15:08.050 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:15:08.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:08.052 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:15:08.153 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:15:08.177 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:15:08.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:08.180 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:15:08.280 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:15:08.306 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:15:08.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:08.308 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:15:08.409 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:15:08.421 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:15:08 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:15:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:15:08.433 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:15:08.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:08.435 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:15:08.536 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:15:08.558 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:15:08.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:08.563 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:15:08.664 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:08.694 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:08.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:08.698 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:15:08.799 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:15:08.819 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:15:08.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:08.822 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:15:08.841 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:08 smithi023 bash[21372]: cluster 2024-03-11T21:15:07.705490+0000 osd.3 (osd.3) 492 : cluster 0 purged_snaps scrub starts 2024-03-11T21:15:08.841 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:08 smithi023 bash[21372]: cluster 2024-03-11T21:15:07.705549+0000 osd.3 (osd.3) 493 : cluster 0 purged_snaps scrub ok 2024-03-11T21:15:08.841 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:08 smithi023 bash[21372]: cluster 2024-03-11T21:15:08.522322+0000 mon.a (mon.0) 1942 : cluster 0 osdmap e432: 8 total, 8 up, 8 in 2024-03-11T21:15:08.841 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:08 smithi023 bash[29146]: cluster 2024-03-11T21:15:07.705490+0000 osd.3 (osd.3) 492 : cluster 0 purged_snaps scrub starts 2024-03-11T21:15:08.841 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:08 smithi023 bash[29146]: cluster 2024-03-11T21:15:07.705549+0000 osd.3 (osd.3) 493 : cluster 0 purged_snaps scrub ok 2024-03-11T21:15:08.841 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:08 smithi023 bash[29146]: cluster 2024-03-11T21:15:08.522322+0000 mon.a (mon.0) 1942 : cluster 0 osdmap e432: 8 total, 8 up, 8 in 2024-03-11T21:15:08.922 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:15:08.932 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:08 smithi146 bash[23809]: cluster 2024-03-11T21:15:07.705490+0000 osd.3 (osd.3) 492 : cluster 0 purged_snaps scrub starts 2024-03-11T21:15:08.932 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:08 smithi146 bash[23809]: cluster 2024-03-11T21:15:07.705549+0000 osd.3 (osd.3) 493 : cluster 0 purged_snaps scrub ok 2024-03-11T21:15:08.932 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:08 smithi146 bash[23809]: cluster 2024-03-11T21:15:08.522322+0000 mon.a (mon.0) 1942 : cluster 0 osdmap e432: 8 total, 8 up, 8 in 2024-03-11T21:15:08.945 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:15:08.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:08.948 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:15:09.049 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:09.070 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:09.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:09.072 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:15:09.173 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:15:09.193 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:15:09.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:09.195 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:15:09.296 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:15:09.330 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:15:09.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:09.334 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:15:09.435 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:15:09.462 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:15:09.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:09.465 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:15:09.565 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:15:09.590 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:15:09.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:09.592 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:15:09.693 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:15:09.725 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:15:09.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:09.727 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:15:09.828 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:15:09.862 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:15:09.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:09.868 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:15:09.969 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:15:09.998 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:15:10.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:10.001 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:15:10.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:09 smithi146 bash[23809]: cluster 2024-03-11T21:15:08.442895+0000 mgr.y (mgr.24380) 900 : cluster 0 pgmap v1040: 158 pgs: 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 153 active+clean; 1.3 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 715 op/s; 678/64308 objects degraded (1.054%); 3160/64308 objects misplaced (4.914%); 19 MiB/s, 307 objects/s recovering 2024-03-11T21:15:10.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:09 smithi146 bash[23809]: cluster 2024-03-11T21:15:09.536881+0000 mon.a (mon.0) 1943 : cluster 0 osdmap e433: 8 total, 8 up, 8 in 2024-03-11T21:15:10.102 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:15:10.139 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:15:10.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:10.144 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:15:10.205 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:09 smithi023 bash[21372]: cluster 2024-03-11T21:15:08.442895+0000 mgr.y (mgr.24380) 900 : cluster 0 pgmap v1040: 158 pgs: 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 153 active+clean; 1.3 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 715 op/s; 678/64308 objects degraded (1.054%); 3160/64308 objects misplaced (4.914%); 19 MiB/s, 307 objects/s recovering 2024-03-11T21:15:10.205 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:09 smithi023 bash[21372]: cluster 2024-03-11T21:15:09.536881+0000 mon.a (mon.0) 1943 : cluster 0 osdmap e433: 8 total, 8 up, 8 in 2024-03-11T21:15:10.206 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:09 smithi023 bash[29146]: cluster 2024-03-11T21:15:08.442895+0000 mgr.y (mgr.24380) 900 : cluster 0 pgmap v1040: 158 pgs: 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 153 active+clean; 1.3 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 715 op/s; 678/64308 objects degraded (1.054%); 3160/64308 objects misplaced (4.914%); 19 MiB/s, 307 objects/s recovering 2024-03-11T21:15:10.206 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:09 smithi023 bash[29146]: cluster 2024-03-11T21:15:09.536881+0000 mon.a (mon.0) 1943 : cluster 0 osdmap e433: 8 total, 8 up, 8 in 2024-03-11T21:15:10.244 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:15:10.288 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:15:10.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:10.291 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:15:10.392 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:15:10.432 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:15:10.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:10.436 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:15:10.537 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:10.573 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:10.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:10.577 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:15:10.678 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:15:10.712 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:15:10.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:10.718 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:15:10.819 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:10.828 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:10 smithi146 bash[23809]: cluster 2024-03-11T21:15:10.404304+0000 mon.a (mon.0) 1944 : cluster 3 Health check update: Degraded data redundancy: 678/64308 objects degraded (1.054%), 1 pg degraded (PG_DEGRADED) 2024-03-11T21:15:10.843 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:10.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:10.846 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:15:10.947 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:15:10.957 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:10 smithi023 bash[21372]: cluster 2024-03-11T21:15:10.404304+0000 mon.a (mon.0) 1944 : cluster 3 Health check update: Degraded data redundancy: 678/64308 objects degraded (1.054%), 1 pg degraded (PG_DEGRADED) 2024-03-11T21:15:10.958 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:10 smithi023 bash[29146]: cluster 2024-03-11T21:15:10.404304+0000 mon.a (mon.0) 1944 : cluster 3 Health check update: Degraded data redundancy: 678/64308 objects degraded (1.054%), 1 pg degraded (PG_DEGRADED) 2024-03-11T21:15:10.972 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:15:10.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:10.975 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:15:11.075 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:15:11.105 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:15:11.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:11.111 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:15:11.212 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:15:11.255 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:15:11.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:11.257 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:15:11.358 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:15:11.394 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:15:11.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:11.398 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:15:11.498 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:15:11.523 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:15:11.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:11.528 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:15:11.629 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:15:11.654 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:15:11.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:11.657 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:15:11.758 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:15:11.769 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:11 smithi023 bash[21372]: cluster 2024-03-11T21:15:10.443581+0000 mgr.y (mgr.24380) 901 : cluster 0 pgmap v1043: 158 pgs: 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 153 active+clean; 1.3 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 620 op/s; 678/62688 objects degraded (1.082%); 3160/62688 objects misplaced (5.041%); 13 MiB/s, 208 objects/s recovering 2024-03-11T21:15:11.769 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:11 smithi023 bash[29146]: cluster 2024-03-11T21:15:10.443581+0000 mgr.y (mgr.24380) 901 : cluster 0 pgmap v1043: 158 pgs: 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 153 active+clean; 1.3 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 620 op/s; 678/62688 objects degraded (1.082%); 3160/62688 objects misplaced (5.041%); 13 MiB/s, 208 objects/s recovering 2024-03-11T21:15:11.785 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:15:11.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:11.789 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:15:11.890 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:15:11.900 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:11 smithi146 bash[23809]: cluster 2024-03-11T21:15:10.443581+0000 mgr.y (mgr.24380) 901 : cluster 0 pgmap v1043: 158 pgs: 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 153 active+clean; 1.3 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 620 op/s; 678/62688 objects degraded (1.082%); 3160/62688 objects misplaced (5.041%); 13 MiB/s, 208 objects/s recovering 2024-03-11T21:15:11.914 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:15:11.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:11.919 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:15:12.020 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:15:12.052 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:15:12.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:12.055 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:15:12.155 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:12.181 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:12.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:12.184 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:15:12.284 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:15:12.326 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:15:12.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:12.329 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:15:12.430 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:15:12.463 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:15:12.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:12.467 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:15:12.567 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:12.592 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:12.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:12.597 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:15:12.698 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:15:12.739 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:15:12.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:12.741 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:15:12.781 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T21:15:12.802 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-11T21:15:12.841 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:15:12.919 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:15:12.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:12.923 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:15:13.024 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:15:13.045 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:15:13.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:13.047 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:15:13.148 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:15:13.175 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:15:13.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:13.177 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:15:13.278 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:15:13.307 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:15:13.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:13.315 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:15:13.316 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:15:13.316 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:15:13.417 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:15:13.443 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:15:13.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:13.449 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:15:13.549 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:15:13.580 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:15:13.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:13.583 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:15:13.684 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:15:13.725 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:15:13.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:13.730 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:15:13.754 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:13 smithi146 bash[23809]: cluster 2024-03-11T21:15:12.444466+0000 mgr.y (mgr.24380) 902 : cluster 0 pgmap v1044: 158 pgs: 2 active+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 155 active+clean; 1.2 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 534 op/s; 678/57324 objects degraded (1.183%); 2309/57324 objects misplaced (4.028%); 6.7 MiB/s, 106 objects/s recovering 2024-03-11T21:15:13.832 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:15:13.859 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:15:13.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:13.864 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:15:13.965 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:15:13.975 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:13 smithi023 bash[21372]: cluster 2024-03-11T21:15:12.444466+0000 mgr.y (mgr.24380) 902 : cluster 0 pgmap v1044: 158 pgs: 2 active+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 155 active+clean; 1.2 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 534 op/s; 678/57324 objects degraded (1.183%); 2309/57324 objects misplaced (4.028%); 6.7 MiB/s, 106 objects/s recovering 2024-03-11T21:15:13.975 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:13 smithi023 bash[29146]: cluster 2024-03-11T21:15:12.444466+0000 mgr.y (mgr.24380) 902 : cluster 0 pgmap v1044: 158 pgs: 2 active+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 155 active+clean; 1.2 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 534 op/s; 678/57324 objects degraded (1.183%); 2309/57324 objects misplaced (4.028%); 6.7 MiB/s, 106 objects/s recovering 2024-03-11T21:15:13.995 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:15:13.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:13.999 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:15:14.100 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:15:14.132 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:15:14.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:14.134 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:15:14.235 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:15:14.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:14.277 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:15:14.277 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:15:14.378 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:15:14.620 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:15:14.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:14.625 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:15:14.726 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:15:14.747 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:15:14.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:14.750 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:15:14.851 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:15:14.880 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:15:14.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:14.882 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:15:14.983 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:15:15.012 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:15:15.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:15.016 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:15:15.117 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:15:15.159 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:15:15.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:15.163 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:15:15.264 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:15:15.295 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:15:15.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:15.298 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:15:15.398 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:15.429 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:15.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:15.431 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:15:15.531 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:15.561 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:15.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:15.566 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:15:15.666 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:15.695 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:15.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:15.706 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:15:15.806 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:15:15.824 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:15:15.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:15.826 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:15:15.927 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:15:15.934 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:15 smithi146 bash[23809]: cluster 2024-03-11T21:15:14.445130+0000 mgr.y (mgr.24380) 903 : cluster 0 pgmap v1045: 158 pgs: 2 active+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 155 active+clean; 1.1 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 739 B/s rd, 0 B/s wr, 596 op/s; 373/54561 objects degraded (0.684%); 2309/54561 objects misplaced (4.232%); 8.5 MiB/s, 136 objects/s recovering 2024-03-11T21:15:15.935 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:15 smithi146 bash[23809]: cluster 2024-03-11T21:15:15.405809+0000 mon.a (mon.0) 1945 : cluster 3 Health check update: Degraded data redundancy: 373/54561 objects degraded (0.684%), 1 pg degraded (PG_DEGRADED) 2024-03-11T21:15:15.945 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:15:15.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:15.947 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:15:16.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:15 smithi023 bash[29146]: cluster 2024-03-11T21:15:14.445130+0000 mgr.y (mgr.24380) 903 : cluster 0 pgmap v1045: 158 pgs: 2 active+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 155 active+clean; 1.1 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 739 B/s rd, 0 B/s wr, 596 op/s; 373/54561 objects degraded (0.684%); 2309/54561 objects misplaced (4.232%); 8.5 MiB/s, 136 objects/s recovering 2024-03-11T21:15:16.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:15 smithi023 bash[29146]: cluster 2024-03-11T21:15:15.405809+0000 mon.a (mon.0) 1945 : cluster 3 Health check update: Degraded data redundancy: 373/54561 objects degraded (0.684%), 1 pg degraded (PG_DEGRADED) 2024-03-11T21:15:16.026 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:15 smithi023 bash[21372]: cluster 2024-03-11T21:15:14.445130+0000 mgr.y (mgr.24380) 903 : cluster 0 pgmap v1045: 158 pgs: 2 active+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 155 active+clean; 1.1 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 739 B/s rd, 0 B/s wr, 596 op/s; 373/54561 objects degraded (0.684%); 2309/54561 objects misplaced (4.232%); 8.5 MiB/s, 136 objects/s recovering 2024-03-11T21:15:16.026 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:15 smithi023 bash[21372]: cluster 2024-03-11T21:15:15.405809+0000 mon.a (mon.0) 1945 : cluster 3 Health check update: Degraded data redundancy: 373/54561 objects degraded (0.684%), 1 pg degraded (PG_DEGRADED) 2024-03-11T21:15:16.047 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:16.069 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:16.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:16.071 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:15:16.172 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:15:16.204 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:15:16.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:16.208 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:15:16.254 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:15:16 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:15:16.309 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:15:16.344 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:15:16.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:16.348 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:15:16.448 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:15:16.478 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:15:16.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:16.479 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:15:16.580 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:16.606 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:16.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:16.609 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:15:16.710 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:15:16.732 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:15:16.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:16.737 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:15:16.837 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:16.873 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:16.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:16.876 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:15:16.977 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:15:17.006 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:15:17.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:17.016 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:15:17.117 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:15:17.143 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:15:17.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:17.147 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:15:17.248 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:15:17.271 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:15:17.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:17.275 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:15:17.376 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:15:17.381 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:15:17.384 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:15:17.424 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:15:17.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:17.431 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:15:17.532 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:15:17.575 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:15:17.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:17.577 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:15:17.677 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:15:17.696 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:15:17.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:17.698 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:15:17.798 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:15:17.808 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:17 smithi146 bash[23809]: audit 2024-03-11T21:15:16.088902+0000 mgr.y (mgr.24380) 904 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:15:17.808 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:17 smithi146 bash[23809]: cluster 2024-03-11T21:15:16.449214+0000 mgr.y (mgr.24380) 905 : cluster 0 pgmap v1046: 158 pgs: 2 active+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 155 active+clean; 886 MiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 884 op/s; 373/43101 objects degraded (0.865%); 1664/43101 objects misplaced (3.861%); 5.6 MiB/s, 89 objects/s recovering 2024-03-11T21:15:17.832 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:15:17.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:17.836 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:15:17.936 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:17.975 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:17.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:17.980 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:15:18.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:17 smithi023 bash[29146]: audit 2024-03-11T21:15:16.088902+0000 mgr.y (mgr.24380) 904 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:15:18.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:17 smithi023 bash[29146]: cluster 2024-03-11T21:15:16.449214+0000 mgr.y (mgr.24380) 905 : cluster 0 pgmap v1046: 158 pgs: 2 active+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 155 active+clean; 886 MiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 884 op/s; 373/43101 objects degraded (0.865%); 1664/43101 objects misplaced (3.861%); 5.6 MiB/s, 89 objects/s recovering 2024-03-11T21:15:18.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:17 smithi023 bash[21372]: audit 2024-03-11T21:15:16.088902+0000 mgr.y (mgr.24380) 904 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:15:18.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:17 smithi023 bash[21372]: cluster 2024-03-11T21:15:16.449214+0000 mgr.y (mgr.24380) 905 : cluster 0 pgmap v1046: 158 pgs: 2 active+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 155 active+clean; 886 MiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 884 op/s; 373/43101 objects degraded (0.865%); 1664/43101 objects misplaced (3.861%); 5.6 MiB/s, 89 objects/s recovering 2024-03-11T21:15:18.081 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:15:18.100 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:15:18.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:18.102 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:15:18.203 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:18.223 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:18.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:18.225 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:15:18.325 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:15:18.357 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:15:18.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:18.363 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:15:18.464 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:15:18.497 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:15:18.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:18.501 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:15:18.594 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:15:18 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:15:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:15:18.602 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:15:18.626 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:15:18.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:18.631 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:15:18.731 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:18.757 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:18.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:18.761 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:15:18.861 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:15:18.890 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:15:18.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:18.892 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:15:18.993 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:19.012 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:19.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:19.014 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:15:19.115 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:15:19.150 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:15:19.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:19.153 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:15:19.254 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:15:19.296 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:15:19.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:19.299 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:15:19.399 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:15:19.428 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:15:19.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:19.431 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:15:19.532 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:15:19.556 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:15:19.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:19.559 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:15:19.660 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:15:19.678 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:15:19.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:19.680 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:15:19.781 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:15:19.795 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:19 smithi146 bash[23809]: cluster 2024-03-11T21:15:18.449856+0000 mgr.y (mgr.24380) 906 : cluster 0 pgmap v1047: 158 pgs: 2 active+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 155 active+clean; 813 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 830 op/s; 129/39609 objects degraded (0.326%); 1664/39609 objects misplaced (4.201%); 6.0 MiB/s, 96 objects/s recovering 2024-03-11T21:15:19.807 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:15:19.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:19.815 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:15:19.916 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:15:19.942 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:15:19.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:19.946 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:15:20.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:19 smithi023 bash[21372]: cluster 2024-03-11T21:15:18.449856+0000 mgr.y (mgr.24380) 906 : cluster 0 pgmap v1047: 158 pgs: 2 active+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 155 active+clean; 813 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 830 op/s; 129/39609 objects degraded (0.326%); 1664/39609 objects misplaced (4.201%); 6.0 MiB/s, 96 objects/s recovering 2024-03-11T21:15:20.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:19 smithi023 bash[29146]: cluster 2024-03-11T21:15:18.449856+0000 mgr.y (mgr.24380) 906 : cluster 0 pgmap v1047: 158 pgs: 2 active+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 155 active+clean; 813 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 830 op/s; 129/39609 objects degraded (0.326%); 1664/39609 objects misplaced (4.201%); 6.0 MiB/s, 96 objects/s recovering 2024-03-11T21:15:20.047 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:15:20.071 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:15:20.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:20.073 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:15:20.173 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:15:20.191 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:15:20.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:20.193 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:15:20.294 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:15:20.322 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:15:20.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:20.324 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:15:20.425 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:15:20.453 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:15:20.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:20.456 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:15:20.556 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:15:20.582 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:15:20.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:20.586 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:15:20.686 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:15:20.720 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:15:20.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:20.723 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:15:20.824 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:15:20.845 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:15:20.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:20.848 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:15:20.948 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:15:20.958 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:20 smithi023 bash[21372]: cluster 2024-03-11T21:15:20.407107+0000 mon.a (mon.0) 1946 : cluster 3 Health check update: Degraded data redundancy: 129/39609 objects degraded (0.326%), 1 pg degraded (PG_DEGRADED) 2024-03-11T21:15:20.959 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:20 smithi023 bash[29146]: cluster 2024-03-11T21:15:20.407107+0000 mon.a (mon.0) 1946 : cluster 3 Health check update: Degraded data redundancy: 129/39609 objects degraded (0.326%), 1 pg degraded (PG_DEGRADED) 2024-03-11T21:15:20.980 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:15:20.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:20.984 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:15:21.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:20 smithi146 bash[23809]: cluster 2024-03-11T21:15:20.407107+0000 mon.a (mon.0) 1946 : cluster 3 Health check update: Degraded data redundancy: 129/39609 objects degraded (0.326%), 1 pg degraded (PG_DEGRADED) 2024-03-11T21:15:21.085 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:15:21.122 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:15:21.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:21.126 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:15:21.227 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:15:21.251 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:15:21.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:21.254 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:15:21.355 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:15:21.379 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:15:21.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:21.382 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:15:21.483 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:15:21.509 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:15:21.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:21.512 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:15:21.613 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:15:21.637 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:15:21.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:21.640 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:15:21.740 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:15:21.758 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:15:21.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:21.760 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:15:21.861 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:15:21.871 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:21 smithi146 bash[23809]: cluster 2024-03-11T21:15:20.450591+0000 mgr.y (mgr.24380) 907 : cluster 0 pgmap v1048: 158 pgs: 2 active+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 155 active+clean; 765 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 938 B/s rd, 0 B/s wr, 826 op/s; 129/37281 objects degraded (0.346%); 1664/37281 objects misplaced (4.463%); 5.5 MiB/s, 87 objects/s recovering 2024-03-11T21:15:21.871 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:21 smithi146 bash[23809]: audit 2024-03-11T21:15:21.554064+0000 mon.a (mon.0) 1947 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:15:21.871 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:21 smithi146 bash[23809]: audit 2024-03-11T21:15:21.556386+0000 mon.a (mon.0) 1948 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:15:21.886 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:15:21.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:21.888 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:15:21.909 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:21 smithi023 bash[21372]: cluster 2024-03-11T21:15:20.450591+0000 mgr.y (mgr.24380) 907 : cluster 0 pgmap v1048: 158 pgs: 2 active+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 155 active+clean; 765 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 938 B/s rd, 0 B/s wr, 826 op/s; 129/37281 objects degraded (0.346%); 1664/37281 objects misplaced (4.463%); 5.5 MiB/s, 87 objects/s recovering 2024-03-11T21:15:21.909 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:21 smithi023 bash[21372]: audit 2024-03-11T21:15:21.554064+0000 mon.a (mon.0) 1947 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:15:21.909 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:21 smithi023 bash[21372]: audit 2024-03-11T21:15:21.556386+0000 mon.a (mon.0) 1948 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:15:21.910 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:21 smithi023 bash[29146]: cluster 2024-03-11T21:15:20.450591+0000 mgr.y (mgr.24380) 907 : cluster 0 pgmap v1048: 158 pgs: 2 active+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 155 active+clean; 765 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 938 B/s rd, 0 B/s wr, 826 op/s; 129/37281 objects degraded (0.346%); 1664/37281 objects misplaced (4.463%); 5.5 MiB/s, 87 objects/s recovering 2024-03-11T21:15:21.910 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:21 smithi023 bash[29146]: audit 2024-03-11T21:15:21.554064+0000 mon.a (mon.0) 1947 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:15:21.910 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:21 smithi023 bash[29146]: audit 2024-03-11T21:15:21.556386+0000 mon.a (mon.0) 1948 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:15:21.989 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:15:22.011 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:15:22.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:22.013 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:15:22.114 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:22.146 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:22.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:22.148 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:15:22.250 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:15:22.276 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:15:22.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:22.284 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:15:22.385 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:15:22.413 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:15:22.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:22.414 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:15:22.515 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:22.539 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:22.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:22.545 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:15:22.646 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:15:22.676 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:15:22.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:22.681 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:15:22.782 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:15:22.814 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:15:22.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:22.817 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:15:22.918 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:22.952 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:22.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:22.955 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:15:23.056 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:15:23.092 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:15:23.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:23.096 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:15:23.197 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:15:23.221 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:15:23.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:23.225 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:15:23.325 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:15:23.350 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:15:23.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:23.354 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:15:23.455 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:15:23.487 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:15:23.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:23.490 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:15:23.591 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:15:23.617 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:15:23.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:23.620 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:15:23.721 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:15:23.745 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:15:23.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:23.749 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:15:23.850 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:15:23.859 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:23 smithi146 bash[23809]: cluster 2024-03-11T21:15:22.451805+0000 mgr.y (mgr.24380) 908 : cluster 0 pgmap v1049: 158 pgs: 2 active+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 155 active+clean; 619 MiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 900 op/s; 129/30303 objects degraded (0.426%); 1154/30303 objects misplaced (3.808%); 5.0 MiB/s, 79 objects/s recovering 2024-03-11T21:15:23.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:23.891 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:15:23.891 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:15:23.992 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:15:24.004 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:23 smithi023 bash[21372]: cluster 2024-03-11T21:15:22.451805+0000 mgr.y (mgr.24380) 908 : cluster 0 pgmap v1049: 158 pgs: 2 active+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 155 active+clean; 619 MiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 900 op/s; 129/30303 objects degraded (0.426%); 1154/30303 objects misplaced (3.808%); 5.0 MiB/s, 79 objects/s recovering 2024-03-11T21:15:24.004 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:23 smithi023 bash[29146]: cluster 2024-03-11T21:15:22.451805+0000 mgr.y (mgr.24380) 908 : cluster 0 pgmap v1049: 158 pgs: 2 active+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 155 active+clean; 619 MiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 900 op/s; 129/30303 objects degraded (0.426%); 1154/30303 objects misplaced (3.808%); 5.0 MiB/s, 79 objects/s recovering 2024-03-11T21:15:24.031 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:15:24.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:24.036 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:15:24.137 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:15:24.167 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:15:24.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:24.169 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:15:24.270 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:15:24.288 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:15:24.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:24.290 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:15:24.391 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:15:24.417 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:15:24.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:24.420 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:15:24.521 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:15:24.554 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:15:24.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:24.561 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:15:24.661 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:15:24.688 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:15:24.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:24.691 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:15:24.792 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:15:24.817 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:15:24.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:24.821 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:15:24.921 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:15:24.923 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:24 smithi023 bash[21372]: cluster 2024-03-11T21:15:24.452408+0000 mgr.y (mgr.24380) 909 : cluster 0 pgmap v1050: 158 pgs: 1 active+recovering+remapped, 2 active+remapped+backfill_toofull, 155 active+clean; 553 MiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 839 op/s; 1154/27120 objects misplaced (4.255%); 3.5 MiB/s, 71 objects/s recovering 2024-03-11T21:15:24.923 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:24 smithi023 bash[21372]: cluster 2024-03-11T21:15:24.726097+0000 mon.a (mon.0) 1949 : cluster 1 Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 129/30303 objects degraded (0.426%), 1 pg degraded) 2024-03-11T21:15:24.946 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:15:24.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:24.951 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:15:25.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:24 smithi146 bash[23809]: cluster 2024-03-11T21:15:24.452408+0000 mgr.y (mgr.24380) 909 : cluster 0 pgmap v1050: 158 pgs: 1 active+recovering+remapped, 2 active+remapped+backfill_toofull, 155 active+clean; 553 MiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 839 op/s; 1154/27120 objects misplaced (4.255%); 3.5 MiB/s, 71 objects/s recovering 2024-03-11T21:15:25.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:24 smithi146 bash[23809]: cluster 2024-03-11T21:15:24.726097+0000 mon.a (mon.0) 1949 : cluster 1 Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 129/30303 objects degraded (0.426%), 1 pg degraded) 2024-03-11T21:15:25.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:24 smithi023 bash[29146]: cluster 2024-03-11T21:15:24.452408+0000 mgr.y (mgr.24380) 909 : cluster 0 pgmap v1050: 158 pgs: 1 active+recovering+remapped, 2 active+remapped+backfill_toofull, 155 active+clean; 553 MiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 839 op/s; 1154/27120 objects misplaced (4.255%); 3.5 MiB/s, 71 objects/s recovering 2024-03-11T21:15:25.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:24 smithi023 bash[29146]: cluster 2024-03-11T21:15:24.726097+0000 mon.a (mon.0) 1949 : cluster 1 Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 129/30303 objects degraded (0.426%), 1 pg degraded) 2024-03-11T21:15:25.052 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:25.080 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:25.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:25.085 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:15:25.186 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:15:25.209 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:15:25.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:25.211 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:15:25.312 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:15:25.336 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:15:25.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:25.340 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:15:25.441 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:25.466 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:25.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:25.468 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:15:25.569 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:25.592 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:25.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:25.594 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:15:25.694 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:15:25.730 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:15:25.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:25.735 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:15:25.837 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:25.865 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:25.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:25.868 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:15:25.969 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:15:25.995 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:15:25.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:25.999 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:15:26.100 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:15:26.108 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:15:26 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:15:26.120 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:15:26.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:26.125 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:15:26.225 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:15:26.251 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:15:26.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:26.254 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:15:26.355 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:26.377 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:26.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:26.382 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:15:26.483 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:15:26.504 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:15:26.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:26.506 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:15:26.607 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:26.645 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:26.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:26.648 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:15:26.749 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:15:26.773 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:15:26.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:26.775 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:15:26.876 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:15:26.902 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:15:26.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:26.905 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:15:27.006 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:27.032 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:27.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:27.035 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:15:27.136 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:15:27.153 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:15:27.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:27.155 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:15:27.256 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:15:27.289 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:15:27.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:27.291 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:15:27.392 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:15:27.416 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:15:27.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:27.418 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:15:27.519 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:15:27.528 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:27 smithi146 bash[23809]: audit 2024-03-11T21:15:26.096338+0000 mgr.y (mgr.24380) 910 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:15:27.529 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:27 smithi146 bash[23809]: cluster 2024-03-11T21:15:26.453339+0000 mgr.y (mgr.24380) 911 : cluster 0 pgmap v1051: 158 pgs: 1 active+recovering+remapped, 2 active+remapped+backfill_toofull, 155 active+clean; 339 MiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.05k op/s; 576/16833 objects misplaced (3.422%); 1.9 MiB/s, 46 objects/s recovering 2024-03-11T21:15:27.543 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:15:27.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:27.547 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:15:27.648 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:15:27.658 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:27 smithi023 bash[21372]: audit 2024-03-11T21:15:26.096338+0000 mgr.y (mgr.24380) 910 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:15:27.659 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:27 smithi023 bash[21372]: cluster 2024-03-11T21:15:26.453339+0000 mgr.y (mgr.24380) 911 : cluster 0 pgmap v1051: 158 pgs: 1 active+recovering+remapped, 2 active+remapped+backfill_toofull, 155 active+clean; 339 MiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.05k op/s; 576/16833 objects misplaced (3.422%); 1.9 MiB/s, 46 objects/s recovering 2024-03-11T21:15:27.659 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:27 smithi023 bash[29146]: audit 2024-03-11T21:15:26.096338+0000 mgr.y (mgr.24380) 910 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:15:27.659 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:27 smithi023 bash[29146]: cluster 2024-03-11T21:15:26.453339+0000 mgr.y (mgr.24380) 911 : cluster 0 pgmap v1051: 158 pgs: 1 active+recovering+remapped, 2 active+remapped+backfill_toofull, 155 active+clean; 339 MiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.05k op/s; 576/16833 objects misplaced (3.422%); 1.9 MiB/s, 46 objects/s recovering 2024-03-11T21:15:27.674 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:15:27.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:27.675 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:15:27.776 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:15:27.806 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:15:27.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:27.810 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:15:27.911 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:15:27.939 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:15:27.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:27.941 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:15:28.042 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:15:28.062 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:15:28.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:28.064 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:15:28.164 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:15:28.185 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:15:28.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:28.191 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:15:28.291 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:15:28.332 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:15:28.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:28.336 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:15:28.437 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:28.465 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:28.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:28.468 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:15:28.525 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:15:28 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:15:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:15:28.569 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:15:28.597 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:15:28.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:28.602 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:15:28.703 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:15:28.735 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:15:28.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:28.741 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:15:28.842 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:15:28.870 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:15:28.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:28.874 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:15:28.975 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:15:29.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:29.007 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:15:29.007 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:15:29.108 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:15:29.132 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:15:29.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:29.134 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:15:29.234 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:15:29.254 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:15:29.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:29.255 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:15:29.356 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:15:29.386 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:15:29.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:29.392 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:15:29.493 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:15:29.528 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:15:29.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:29.531 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:15:29.632 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:15:29.664 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:15:29.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:29.668 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:15:29.754 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:29 smithi146 bash[23809]: cluster 2024-03-11T21:15:28.453812+0000 mgr.y (mgr.24380) 912 : cluster 0 pgmap v1052: 158 pgs: 1 active+recovering+remapped, 2 active+remapped+backfill_toofull, 155 active+clean; 268 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 824 op/s; 576/13425 objects misplaced (4.291%); 1.9 MiB/s, 70 objects/s recovering 2024-03-11T21:15:29.769 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:15:29.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:29 smithi023 bash[21372]: cluster 2024-03-11T21:15:28.453812+0000 mgr.y (mgr.24380) 912 : cluster 0 pgmap v1052: 158 pgs: 1 active+recovering+remapped, 2 active+remapped+backfill_toofull, 155 active+clean; 268 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 824 op/s; 576/13425 objects misplaced (4.291%); 1.9 MiB/s, 70 objects/s recovering 2024-03-11T21:15:29.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:29 smithi023 bash[29146]: cluster 2024-03-11T21:15:28.453812+0000 mgr.y (mgr.24380) 912 : cluster 0 pgmap v1052: 158 pgs: 1 active+recovering+remapped, 2 active+remapped+backfill_toofull, 155 active+clean; 268 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 824 op/s; 576/13425 objects misplaced (4.291%); 1.9 MiB/s, 70 objects/s recovering 2024-03-11T21:15:29.796 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:15:29.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:29.801 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:15:29.902 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:15:29.922 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:15:29.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:29.924 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:15:30.025 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:15:30.048 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:15:30.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:30.050 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:15:30.151 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:15:30.175 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:15:30.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:30.177 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:15:30.278 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:15:30.319 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:15:30.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:30.322 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:15:30.423 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:15:30.450 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:15:30.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:30.453 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:15:30.554 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:15:30.582 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:15:30.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:30.585 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:15:30.686 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:30.721 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:30.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:30.729 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:15:30.829 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:15:30.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:30.859 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:15:30.859 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:15:30.960 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:15:30.991 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:15:30.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:30.993 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:15:31.094 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:15:31.117 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:15:31.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:31.119 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:15:31.220 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:15:31.263 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:15:31.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:31.268 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:15:31.369 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:15:31.389 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:15:31.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:31.391 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:15:31.492 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:31.520 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:31.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:31.524 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:15:31.625 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:15:31.647 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:15:31.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:31.650 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:15:31.731 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:Removed 97641 objects 2024-03-11T21:15:31.731 INFO:tasks.radosbench.radosbench.0.smithi023.stdout:Clean up completed and total clean up time :116.876 2024-03-11T21:15:31.750 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:15:31.754 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:31 smithi146 bash[23809]: cluster 2024-03-11T21:15:30.454457+0000 mgr.y (mgr.24380) 913 : cluster 0 pgmap v1053: 158 pgs: 1 active+recovering+remapped, 2 active+remapped+backfill_toofull, 155 active+clean; 220 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 791 op/s; 576/11121 objects misplaced (5.179%); 688 KiB/s, 49 objects/s recovering 2024-03-11T21:15:31.754 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:31 smithi146 bash[23809]: audit 2024-03-11T21:15:30.785403+0000 mon.a (mon.0) 1950 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:15:31.768 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:15:31.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:31.769 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:15:31.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:31 smithi023 bash[29146]: cluster 2024-03-11T21:15:30.454457+0000 mgr.y (mgr.24380) 913 : cluster 0 pgmap v1053: 158 pgs: 1 active+recovering+remapped, 2 active+remapped+backfill_toofull, 155 active+clean; 220 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 791 op/s; 576/11121 objects misplaced (5.179%); 688 KiB/s, 49 objects/s recovering 2024-03-11T21:15:31.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:31 smithi023 bash[29146]: audit 2024-03-11T21:15:30.785403+0000 mon.a (mon.0) 1950 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:15:31.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:31 smithi023 bash[21372]: cluster 2024-03-11T21:15:30.454457+0000 mgr.y (mgr.24380) 913 : cluster 0 pgmap v1053: 158 pgs: 1 active+recovering+remapped, 2 active+remapped+backfill_toofull, 155 active+clean; 220 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 791 op/s; 576/11121 objects misplaced (5.179%); 688 KiB/s, 49 objects/s recovering 2024-03-11T21:15:31.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:31 smithi023 bash[21372]: audit 2024-03-11T21:15:30.785403+0000 mon.a (mon.0) 1950 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:15:31.871 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:15:31.896 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:15:31.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:31.898 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:15:31.999 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:15:32.002 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_4 2024-03-11T21:15:32.002 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd pool rm unique_pool_4 unique_pool_4 --yes-i-really-really-mean-it 2024-03-11T21:15:32.024 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:15:32.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:32.027 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:15:32.127 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:15:32.149 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:15:32.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:32.152 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:15:32.252 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:15:32.283 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:15:32.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:32.285 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:15:32.386 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:15:32.412 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:15:32.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:32.414 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:15:32.515 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:15:32.528 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:32 smithi146 bash[23809]: audit 2024-03-11T21:15:31.768383+0000 mon.a (mon.0) 1951 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:15:32.528 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:32 smithi146 bash[23809]: audit 2024-03-11T21:15:31.769118+0000 mon.a (mon.0) 1952 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:15:32.528 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:32 smithi146 bash[23809]: audit 2024-03-11T21:15:31.775904+0000 mon.a (mon.0) 1953 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:15:32.543 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:15:32.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:32.546 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:15:32.647 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:15:32.670 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:15:32.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:32.673 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:15:32.774 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:15:32.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:32 smithi023 bash[21372]: audit 2024-03-11T21:15:31.768383+0000 mon.a (mon.0) 1951 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:15:32.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:32 smithi023 bash[21372]: audit 2024-03-11T21:15:31.769118+0000 mon.a (mon.0) 1952 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:15:32.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:32 smithi023 bash[21372]: audit 2024-03-11T21:15:31.775904+0000 mon.a (mon.0) 1953 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:15:32.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:32 smithi023 bash[29146]: audit 2024-03-11T21:15:31.768383+0000 mon.a (mon.0) 1951 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:15:32.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:32 smithi023 bash[29146]: audit 2024-03-11T21:15:31.769118+0000 mon.a (mon.0) 1952 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:15:32.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:32 smithi023 bash[29146]: audit 2024-03-11T21:15:31.775904+0000 mon.a (mon.0) 1953 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:15:32.800 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:15:32.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:32.802 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:15:32.903 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:15:32.931 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:15:32.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:32.933 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:15:33.034 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:15:33.058 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:15:33.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:33.061 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:15:33.161 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:15:33.184 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:15:33.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:33.186 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:15:33.287 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:15:33.310 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-11T21:15:33.311 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-11T21:15:33.311 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 433, 2024-03-11T21:15:33.311 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "67ba00f8-dfe7-11ee-95c8-87774f69a715", 2024-03-11T21:15:33.311 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2024-03-11T20:40:13.664802+0000", 2024-03-11T21:15:33.311 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2024-03-11T21:15:09.519848+0000", 2024-03-11T21:15:33.311 INFO:teuthology.orchestra.run.smithi023.stdout: "last_up_change": "2024-03-11T21:00:47.324817+0000", 2024-03-11T21:15:33.311 INFO:teuthology.orchestra.run.smithi023.stdout: "last_in_change": "2024-03-11T21:14:49.940617+0000", 2024-03-11T21:15:33.311 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-03-11T21:15:33.311 INFO:teuthology.orchestra.run.smithi023.stdout: "flags_num": 5801984, 2024-03-11T21:15:33.311 INFO:teuthology.orchestra.run.smithi023.stdout: "flags_set": [ 2024-03-11T21:15:33.311 INFO:teuthology.orchestra.run.smithi023.stdout: "noscrub", 2024-03-11T21:15:33.311 INFO:teuthology.orchestra.run.smithi023.stdout: "pglog_hardlimit", 2024-03-11T21:15:33.311 INFO:teuthology.orchestra.run.smithi023.stdout: "purged_snapdirs", 2024-03-11T21:15:33.312 INFO:teuthology.orchestra.run.smithi023.stdout: "recovery_deletes", 2024-03-11T21:15:33.312 INFO:teuthology.orchestra.run.smithi023.stdout: "sortbitwise" 2024-03-11T21:15:33.312 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-11T21:15:33.312 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_version": 20, 2024-03-11T21:15:33.312 INFO:teuthology.orchestra.run.smithi023.stdout: "full_ratio": 0.94999998807907104, 2024-03-11T21:15:33.312 INFO:teuthology.orchestra.run.smithi023.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-03-11T21:15:33.312 INFO:teuthology.orchestra.run.smithi023.stdout: "nearfull_ratio": 0.85000002384185791, 2024-03-11T21:15:33.312 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_snapshot": "", 2024-03-11T21:15:33.312 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_max": 11, 2024-03-11T21:15:33.312 INFO:teuthology.orchestra.run.smithi023.stdout: "max_osd": 8, 2024-03-11T21:15:33.312 INFO:teuthology.orchestra.run.smithi023.stdout: "require_min_compat_client": "luminous", 2024-03-11T21:15:33.312 INFO:teuthology.orchestra.run.smithi023.stdout: "min_compat_client": "luminous", 2024-03-11T21:15:33.312 INFO:teuthology.orchestra.run.smithi023.stdout: "require_osd_release": "squid", 2024-03-11T21:15:33.313 INFO:teuthology.orchestra.run.smithi023.stdout: "allow_crimson": false, 2024-03-11T21:15:33.313 INFO:teuthology.orchestra.run.smithi023.stdout: "pools": [ 2024-03-11T21:15:33.313 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:33.313 INFO:teuthology.orchestra.run.smithi023.stdout: "pool": 1, 2024-03-11T21:15:33.313 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_name": ".mgr", 2024-03-11T21:15:33.313 INFO:teuthology.orchestra.run.smithi023.stdout: "create_time": "2024-03-11T20:45:31.964570+0000", 2024-03-11T21:15:33.313 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-11T21:15:33.313 INFO:teuthology.orchestra.run.smithi023.stdout: "flags_names": "hashpspool", 2024-03-11T21:15:33.313 INFO:teuthology.orchestra.run.smithi023.stdout: "type": 1, 2024-03-11T21:15:33.313 INFO:teuthology.orchestra.run.smithi023.stdout: "size": 3, 2024-03-11T21:15:33.313 INFO:teuthology.orchestra.run.smithi023.stdout: "min_size": 2, 2024-03-11T21:15:33.313 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_rule": 0, 2024-03-11T21:15:33.313 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_count": 0, 2024-03-11T21:15:33.313 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_target": 0, 2024-03-11T21:15:33.314 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T21:15:33.314 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T21:15:33.314 INFO:teuthology.orchestra.run.smithi023.stdout: "object_hash": 2, 2024-03-11T21:15:33.314 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_autoscale_mode": "off", 2024-03-11T21:15:33.314 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num": 11, 2024-03-11T21:15:33.314 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_placement_num": 11, 2024-03-11T21:15:33.314 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_placement_num_target": 11, 2024-03-11T21:15:33.314 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num_target": 11, 2024-03-11T21:15:33.314 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num_pending": 11, 2024-03-11T21:15:33.314 INFO:teuthology.orchestra.run.smithi023.stdout: "last_pg_merge_meta": { 2024-03-11T21:15:33.314 INFO:teuthology.orchestra.run.smithi023.stdout: "source_pgid": "0.0", 2024-03-11T21:15:33.314 INFO:teuthology.orchestra.run.smithi023.stdout: "ready_epoch": 0, 2024-03-11T21:15:33.314 INFO:teuthology.orchestra.run.smithi023.stdout: "last_epoch_started": 0, 2024-03-11T21:15:33.315 INFO:teuthology.orchestra.run.smithi023.stdout: "last_epoch_clean": 0, 2024-03-11T21:15:33.315 INFO:teuthology.orchestra.run.smithi023.stdout: "source_version": "0'0", 2024-03-11T21:15:33.315 INFO:teuthology.orchestra.run.smithi023.stdout: "target_version": "0'0" 2024-03-11T21:15:33.315 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.315 INFO:teuthology.orchestra.run.smithi023.stdout: "last_change": "369", 2024-03-11T21:15:33.315 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend": "0", 2024-03-11T21:15:33.315 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T21:15:33.315 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend_preluminous": "367", 2024-03-11T21:15:33.315 INFO:teuthology.orchestra.run.smithi023.stdout: "auid": 0, 2024-03-11T21:15:33.315 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_mode": "selfmanaged", 2024-03-11T21:15:33.315 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_seq": 0, 2024-03-11T21:15:33.315 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_epoch": 0, 2024-03-11T21:15:33.315 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_snaps": [], 2024-03-11T21:15:33.316 INFO:teuthology.orchestra.run.smithi023.stdout: "removed_snaps": "[]", 2024-03-11T21:15:33.316 INFO:teuthology.orchestra.run.smithi023.stdout: "quota_max_bytes": 0, 2024-03-11T21:15:33.316 INFO:teuthology.orchestra.run.smithi023.stdout: "quota_max_objects": 0, 2024-03-11T21:15:33.316 INFO:teuthology.orchestra.run.smithi023.stdout: "tiers": [], 2024-03-11T21:15:33.316 INFO:teuthology.orchestra.run.smithi023.stdout: "tier_of": -1, 2024-03-11T21:15:33.316 INFO:teuthology.orchestra.run.smithi023.stdout: "read_tier": -1, 2024-03-11T21:15:33.316 INFO:teuthology.orchestra.run.smithi023.stdout: "write_tier": -1, 2024-03-11T21:15:33.316 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_mode": "none", 2024-03-11T21:15:33.316 INFO:teuthology.orchestra.run.smithi023.stdout: "target_max_bytes": 0, 2024-03-11T21:15:33.316 INFO:teuthology.orchestra.run.smithi023.stdout: "target_max_objects": 0, 2024-03-11T21:15:33.316 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T21:15:33.316 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T21:15:33.316 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T21:15:33.316 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_min_flush_age": 0, 2024-03-11T21:15:33.317 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_min_evict_age": 0, 2024-03-11T21:15:33.317 INFO:teuthology.orchestra.run.smithi023.stdout: "erasure_code_profile": "", 2024-03-11T21:15:33.317 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_params": { 2024-03-11T21:15:33.317 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "none" 2024-03-11T21:15:33.317 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.317 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_period": 0, 2024-03-11T21:15:33.317 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_count": 0, 2024-03-11T21:15:33.317 INFO:teuthology.orchestra.run.smithi023.stdout: "use_gmt_hitset": true, 2024-03-11T21:15:33.317 INFO:teuthology.orchestra.run.smithi023.stdout: "min_read_recency_for_promote": 0, 2024-03-11T21:15:33.317 INFO:teuthology.orchestra.run.smithi023.stdout: "min_write_recency_for_promote": 0, 2024-03-11T21:15:33.317 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T21:15:33.317 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_search_last_n": 0, 2024-03-11T21:15:33.317 INFO:teuthology.orchestra.run.smithi023.stdout: "grade_table": [], 2024-03-11T21:15:33.317 INFO:teuthology.orchestra.run.smithi023.stdout: "stripe_width": 0, 2024-03-11T21:15:33.318 INFO:teuthology.orchestra.run.smithi023.stdout: "expected_num_objects": 0, 2024-03-11T21:15:33.318 INFO:teuthology.orchestra.run.smithi023.stdout: "fast_read": false, 2024-03-11T21:15:33.318 INFO:teuthology.orchestra.run.smithi023.stdout: "options": { 2024-03-11T21:15:33.318 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num_max": 32, 2024-03-11T21:15:33.318 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num_min": 1 2024-03-11T21:15:33.318 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.318 INFO:teuthology.orchestra.run.smithi023.stdout: "application_metadata": { 2024-03-11T21:15:33.318 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr": {} 2024-03-11T21:15:33.318 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.318 INFO:teuthology.orchestra.run.smithi023.stdout: "read_balance": { 2024-03-11T21:15:33.318 INFO:teuthology.orchestra.run.smithi023.stdout: "score_type": "Fair distribution", 2024-03-11T21:15:33.318 INFO:teuthology.orchestra.run.smithi023.stdout: "score_acting": 1.6399999856948853, 2024-03-11T21:15:33.319 INFO:teuthology.orchestra.run.smithi023.stdout: "score_stable": 1.6399999856948853, 2024-03-11T21:15:33.319 INFO:teuthology.orchestra.run.smithi023.stdout: "optimal_score": 1.3300000429153442, 2024-03-11T21:15:33.319 INFO:teuthology.orchestra.run.smithi023.stdout: "raw_score_acting": 2.1800000667572021, 2024-03-11T21:15:33.319 INFO:teuthology.orchestra.run.smithi023.stdout: "raw_score_stable": 2.1800000667572021, 2024-03-11T21:15:33.319 INFO:teuthology.orchestra.run.smithi023.stdout: "primary_affinity_weighted": 0.75, 2024-03-11T21:15:33.319 INFO:teuthology.orchestra.run.smithi023.stdout: "average_primary_affinity": 1, 2024-03-11T21:15:33.319 INFO:teuthology.orchestra.run.smithi023.stdout: "average_primary_affinity_weighted": 0.75 2024-03-11T21:15:33.319 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:15:33.319 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.319 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:33.319 INFO:teuthology.orchestra.run.smithi023.stdout: "pool": 2, 2024-03-11T21:15:33.319 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_name": ".rgw.root", 2024-03-11T21:15:33.319 INFO:teuthology.orchestra.run.smithi023.stdout: "create_time": "2024-03-11T20:50:26.421486+0000", 2024-03-11T21:15:33.319 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-11T21:15:33.319 INFO:teuthology.orchestra.run.smithi023.stdout: "flags_names": "hashpspool", 2024-03-11T21:15:33.320 INFO:teuthology.orchestra.run.smithi023.stdout: "type": 1, 2024-03-11T21:15:33.320 INFO:teuthology.orchestra.run.smithi023.stdout: "size": 3, 2024-03-11T21:15:33.320 INFO:teuthology.orchestra.run.smithi023.stdout: "min_size": 2, 2024-03-11T21:15:33.320 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_rule": 0, 2024-03-11T21:15:33.320 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_count": 0, 2024-03-11T21:15:33.320 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_target": 0, 2024-03-11T21:15:33.320 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T21:15:33.320 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T21:15:33.320 INFO:teuthology.orchestra.run.smithi023.stdout: "object_hash": 2, 2024-03-11T21:15:33.320 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_autoscale_mode": "off", 2024-03-11T21:15:33.320 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num": 32, 2024-03-11T21:15:33.320 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_placement_num": 32, 2024-03-11T21:15:33.320 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_placement_num_target": 32, 2024-03-11T21:15:33.320 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num_target": 32, 2024-03-11T21:15:33.320 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num_pending": 32, 2024-03-11T21:15:33.321 INFO:teuthology.orchestra.run.smithi023.stdout: "last_pg_merge_meta": { 2024-03-11T21:15:33.321 INFO:teuthology.orchestra.run.smithi023.stdout: "source_pgid": "0.0", 2024-03-11T21:15:33.321 INFO:teuthology.orchestra.run.smithi023.stdout: "ready_epoch": 0, 2024-03-11T21:15:33.321 INFO:teuthology.orchestra.run.smithi023.stdout: "last_epoch_started": 0, 2024-03-11T21:15:33.321 INFO:teuthology.orchestra.run.smithi023.stdout: "last_epoch_clean": 0, 2024-03-11T21:15:33.321 INFO:teuthology.orchestra.run.smithi023.stdout: "source_version": "0'0", 2024-03-11T21:15:33.321 INFO:teuthology.orchestra.run.smithi023.stdout: "target_version": "0'0" 2024-03-11T21:15:33.321 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.321 INFO:teuthology.orchestra.run.smithi023.stdout: "last_change": "78", 2024-03-11T21:15:33.321 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend": "0", 2024-03-11T21:15:33.321 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T21:15:33.321 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T21:15:33.321 INFO:teuthology.orchestra.run.smithi023.stdout: "auid": 0, 2024-03-11T21:15:33.321 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_mode": "selfmanaged", 2024-03-11T21:15:33.321 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_seq": 0, 2024-03-11T21:15:33.322 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_epoch": 0, 2024-03-11T21:15:33.322 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_snaps": [], 2024-03-11T21:15:33.322 INFO:teuthology.orchestra.run.smithi023.stdout: "removed_snaps": "[]", 2024-03-11T21:15:33.322 INFO:teuthology.orchestra.run.smithi023.stdout: "quota_max_bytes": 0, 2024-03-11T21:15:33.322 INFO:teuthology.orchestra.run.smithi023.stdout: "quota_max_objects": 0, 2024-03-11T21:15:33.322 INFO:teuthology.orchestra.run.smithi023.stdout: "tiers": [], 2024-03-11T21:15:33.322 INFO:teuthology.orchestra.run.smithi023.stdout: "tier_of": -1, 2024-03-11T21:15:33.322 INFO:teuthology.orchestra.run.smithi023.stdout: "read_tier": -1, 2024-03-11T21:15:33.322 INFO:teuthology.orchestra.run.smithi023.stdout: "write_tier": -1, 2024-03-11T21:15:33.322 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_mode": "none", 2024-03-11T21:15:33.322 INFO:teuthology.orchestra.run.smithi023.stdout: "target_max_bytes": 0, 2024-03-11T21:15:33.322 INFO:teuthology.orchestra.run.smithi023.stdout: "target_max_objects": 0, 2024-03-11T21:15:33.322 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T21:15:33.322 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T21:15:33.323 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T21:15:33.323 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_min_flush_age": 0, 2024-03-11T21:15:33.323 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_min_evict_age": 0, 2024-03-11T21:15:33.323 INFO:teuthology.orchestra.run.smithi023.stdout: "erasure_code_profile": "", 2024-03-11T21:15:33.323 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_params": { 2024-03-11T21:15:33.323 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "none" 2024-03-11T21:15:33.323 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.323 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_period": 0, 2024-03-11T21:15:33.323 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_count": 0, 2024-03-11T21:15:33.323 INFO:teuthology.orchestra.run.smithi023.stdout: "use_gmt_hitset": true, 2024-03-11T21:15:33.323 INFO:teuthology.orchestra.run.smithi023.stdout: "min_read_recency_for_promote": 0, 2024-03-11T21:15:33.323 INFO:teuthology.orchestra.run.smithi023.stdout: "min_write_recency_for_promote": 0, 2024-03-11T21:15:33.323 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T21:15:33.323 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_search_last_n": 0, 2024-03-11T21:15:33.323 INFO:teuthology.orchestra.run.smithi023.stdout: "grade_table": [], 2024-03-11T21:15:33.324 INFO:teuthology.orchestra.run.smithi023.stdout: "stripe_width": 0, 2024-03-11T21:15:33.324 INFO:teuthology.orchestra.run.smithi023.stdout: "expected_num_objects": 0, 2024-03-11T21:15:33.324 INFO:teuthology.orchestra.run.smithi023.stdout: "fast_read": false, 2024-03-11T21:15:33.324 INFO:teuthology.orchestra.run.smithi023.stdout: "options": {}, 2024-03-11T21:15:33.324 INFO:teuthology.orchestra.run.smithi023.stdout: "application_metadata": { 2024-03-11T21:15:33.324 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw": {} 2024-03-11T21:15:33.324 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.324 INFO:teuthology.orchestra.run.smithi023.stdout: "read_balance": { 2024-03-11T21:15:33.324 INFO:teuthology.orchestra.run.smithi023.stdout: "score_type": "Fair distribution", 2024-03-11T21:15:33.324 INFO:teuthology.orchestra.run.smithi023.stdout: "score_acting": 1.5, 2024-03-11T21:15:33.324 INFO:teuthology.orchestra.run.smithi023.stdout: "score_stable": 1.5, 2024-03-11T21:15:33.324 INFO:teuthology.orchestra.run.smithi023.stdout: "optimal_score": 1.3300000429153442, 2024-03-11T21:15:33.324 INFO:teuthology.orchestra.run.smithi023.stdout: "raw_score_acting": 2, 2024-03-11T21:15:33.324 INFO:teuthology.orchestra.run.smithi023.stdout: "raw_score_stable": 2, 2024-03-11T21:15:33.325 INFO:teuthology.orchestra.run.smithi023.stdout: "primary_affinity_weighted": 0.75, 2024-03-11T21:15:33.325 INFO:teuthology.orchestra.run.smithi023.stdout: "average_primary_affinity": 1, 2024-03-11T21:15:33.325 INFO:teuthology.orchestra.run.smithi023.stdout: "average_primary_affinity_weighted": 0.75 2024-03-11T21:15:33.325 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:15:33.325 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.325 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:33.325 INFO:teuthology.orchestra.run.smithi023.stdout: "pool": 3, 2024-03-11T21:15:33.325 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_name": "default.rgw.log", 2024-03-11T21:15:33.325 INFO:teuthology.orchestra.run.smithi023.stdout: "create_time": "2024-03-11T20:50:28.592615+0000", 2024-03-11T21:15:33.325 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-11T21:15:33.325 INFO:teuthology.orchestra.run.smithi023.stdout: "flags_names": "hashpspool", 2024-03-11T21:15:33.325 INFO:teuthology.orchestra.run.smithi023.stdout: "type": 1, 2024-03-11T21:15:33.325 INFO:teuthology.orchestra.run.smithi023.stdout: "size": 3, 2024-03-11T21:15:33.325 INFO:teuthology.orchestra.run.smithi023.stdout: "min_size": 2, 2024-03-11T21:15:33.325 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_rule": 0, 2024-03-11T21:15:33.326 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_count": 0, 2024-03-11T21:15:33.326 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_target": 0, 2024-03-11T21:15:33.326 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T21:15:33.326 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T21:15:33.326 INFO:teuthology.orchestra.run.smithi023.stdout: "object_hash": 2, 2024-03-11T21:15:33.326 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_autoscale_mode": "off", 2024-03-11T21:15:33.326 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num": 32, 2024-03-11T21:15:33.326 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_placement_num": 32, 2024-03-11T21:15:33.326 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_placement_num_target": 32, 2024-03-11T21:15:33.326 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num_target": 32, 2024-03-11T21:15:33.326 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num_pending": 32, 2024-03-11T21:15:33.326 INFO:teuthology.orchestra.run.smithi023.stdout: "last_pg_merge_meta": { 2024-03-11T21:15:33.326 INFO:teuthology.orchestra.run.smithi023.stdout: "source_pgid": "0.0", 2024-03-11T21:15:33.327 INFO:teuthology.orchestra.run.smithi023.stdout: "ready_epoch": 0, 2024-03-11T21:15:33.327 INFO:teuthology.orchestra.run.smithi023.stdout: "last_epoch_started": 0, 2024-03-11T21:15:33.327 INFO:teuthology.orchestra.run.smithi023.stdout: "last_epoch_clean": 0, 2024-03-11T21:15:33.327 INFO:teuthology.orchestra.run.smithi023.stdout: "source_version": "0'0", 2024-03-11T21:15:33.327 INFO:teuthology.orchestra.run.smithi023.stdout: "target_version": "0'0" 2024-03-11T21:15:33.327 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.327 INFO:teuthology.orchestra.run.smithi023.stdout: "last_change": "80", 2024-03-11T21:15:33.327 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend": "0", 2024-03-11T21:15:33.327 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T21:15:33.327 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T21:15:33.327 INFO:teuthology.orchestra.run.smithi023.stdout: "auid": 0, 2024-03-11T21:15:33.327 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_mode": "selfmanaged", 2024-03-11T21:15:33.327 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_seq": 0, 2024-03-11T21:15:33.327 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_epoch": 0, 2024-03-11T21:15:33.327 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_snaps": [], 2024-03-11T21:15:33.328 INFO:teuthology.orchestra.run.smithi023.stdout: "removed_snaps": "[]", 2024-03-11T21:15:33.328 INFO:teuthology.orchestra.run.smithi023.stdout: "quota_max_bytes": 0, 2024-03-11T21:15:33.328 INFO:teuthology.orchestra.run.smithi023.stdout: "quota_max_objects": 0, 2024-03-11T21:15:33.328 INFO:teuthology.orchestra.run.smithi023.stdout: "tiers": [], 2024-03-11T21:15:33.328 INFO:teuthology.orchestra.run.smithi023.stdout: "tier_of": -1, 2024-03-11T21:15:33.328 INFO:teuthology.orchestra.run.smithi023.stdout: "read_tier": -1, 2024-03-11T21:15:33.328 INFO:teuthology.orchestra.run.smithi023.stdout: "write_tier": -1, 2024-03-11T21:15:33.328 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_mode": "none", 2024-03-11T21:15:33.328 INFO:teuthology.orchestra.run.smithi023.stdout: "target_max_bytes": 0, 2024-03-11T21:15:33.328 INFO:teuthology.orchestra.run.smithi023.stdout: "target_max_objects": 0, 2024-03-11T21:15:33.328 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T21:15:33.328 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T21:15:33.328 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T21:15:33.328 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_min_flush_age": 0, 2024-03-11T21:15:33.328 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_min_evict_age": 0, 2024-03-11T21:15:33.329 INFO:teuthology.orchestra.run.smithi023.stdout: "erasure_code_profile": "", 2024-03-11T21:15:33.329 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_params": { 2024-03-11T21:15:33.329 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "none" 2024-03-11T21:15:33.329 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.329 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_period": 0, 2024-03-11T21:15:33.329 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_count": 0, 2024-03-11T21:15:33.329 INFO:teuthology.orchestra.run.smithi023.stdout: "use_gmt_hitset": true, 2024-03-11T21:15:33.329 INFO:teuthology.orchestra.run.smithi023.stdout: "min_read_recency_for_promote": 0, 2024-03-11T21:15:33.329 INFO:teuthology.orchestra.run.smithi023.stdout: "min_write_recency_for_promote": 0, 2024-03-11T21:15:33.329 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T21:15:33.329 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_search_last_n": 0, 2024-03-11T21:15:33.329 INFO:teuthology.orchestra.run.smithi023.stdout: "grade_table": [], 2024-03-11T21:15:33.329 INFO:teuthology.orchestra.run.smithi023.stdout: "stripe_width": 0, 2024-03-11T21:15:33.329 INFO:teuthology.orchestra.run.smithi023.stdout: "expected_num_objects": 0, 2024-03-11T21:15:33.330 INFO:teuthology.orchestra.run.smithi023.stdout: "fast_read": false, 2024-03-11T21:15:33.330 INFO:teuthology.orchestra.run.smithi023.stdout: "options": {}, 2024-03-11T21:15:33.330 INFO:teuthology.orchestra.run.smithi023.stdout: "application_metadata": { 2024-03-11T21:15:33.330 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw": {} 2024-03-11T21:15:33.330 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.330 INFO:teuthology.orchestra.run.smithi023.stdout: "read_balance": { 2024-03-11T21:15:33.330 INFO:teuthology.orchestra.run.smithi023.stdout: "score_type": "Fair distribution", 2024-03-11T21:15:33.330 INFO:teuthology.orchestra.run.smithi023.stdout: "score_acting": 1.5, 2024-03-11T21:15:33.330 INFO:teuthology.orchestra.run.smithi023.stdout: "score_stable": 1.5, 2024-03-11T21:15:33.330 INFO:teuthology.orchestra.run.smithi023.stdout: "optimal_score": 1.3300000429153442, 2024-03-11T21:15:33.330 INFO:teuthology.orchestra.run.smithi023.stdout: "raw_score_acting": 2, 2024-03-11T21:15:33.330 INFO:teuthology.orchestra.run.smithi023.stdout: "raw_score_stable": 2, 2024-03-11T21:15:33.330 INFO:teuthology.orchestra.run.smithi023.stdout: "primary_affinity_weighted": 0.75, 2024-03-11T21:15:33.330 INFO:teuthology.orchestra.run.smithi023.stdout: "average_primary_affinity": 1, 2024-03-11T21:15:33.331 INFO:teuthology.orchestra.run.smithi023.stdout: "average_primary_affinity_weighted": 0.75 2024-03-11T21:15:33.331 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:15:33.331 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.331 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:33.331 INFO:teuthology.orchestra.run.smithi023.stdout: "pool": 4, 2024-03-11T21:15:33.331 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_name": "default.rgw.control", 2024-03-11T21:15:33.331 INFO:teuthology.orchestra.run.smithi023.stdout: "create_time": "2024-03-11T20:50:30.536371+0000", 2024-03-11T21:15:33.331 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-11T21:15:33.331 INFO:teuthology.orchestra.run.smithi023.stdout: "flags_names": "hashpspool", 2024-03-11T21:15:33.331 INFO:teuthology.orchestra.run.smithi023.stdout: "type": 1, 2024-03-11T21:15:33.331 INFO:teuthology.orchestra.run.smithi023.stdout: "size": 3, 2024-03-11T21:15:33.331 INFO:teuthology.orchestra.run.smithi023.stdout: "min_size": 2, 2024-03-11T21:15:33.331 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_rule": 0, 2024-03-11T21:15:33.332 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_count": 0, 2024-03-11T21:15:33.332 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_target": 0, 2024-03-11T21:15:33.332 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T21:15:33.332 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T21:15:33.332 INFO:teuthology.orchestra.run.smithi023.stdout: "object_hash": 2, 2024-03-11T21:15:33.332 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_autoscale_mode": "off", 2024-03-11T21:15:33.332 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num": 32, 2024-03-11T21:15:33.332 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_placement_num": 32, 2024-03-11T21:15:33.332 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_placement_num_target": 32, 2024-03-11T21:15:33.332 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num_target": 32, 2024-03-11T21:15:33.332 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num_pending": 32, 2024-03-11T21:15:33.332 INFO:teuthology.orchestra.run.smithi023.stdout: "last_pg_merge_meta": { 2024-03-11T21:15:33.332 INFO:teuthology.orchestra.run.smithi023.stdout: "source_pgid": "0.0", 2024-03-11T21:15:33.332 INFO:teuthology.orchestra.run.smithi023.stdout: "ready_epoch": 0, 2024-03-11T21:15:33.332 INFO:teuthology.orchestra.run.smithi023.stdout: "last_epoch_started": 0, 2024-03-11T21:15:33.333 INFO:teuthology.orchestra.run.smithi023.stdout: "last_epoch_clean": 0, 2024-03-11T21:15:33.333 INFO:teuthology.orchestra.run.smithi023.stdout: "source_version": "0'0", 2024-03-11T21:15:33.333 INFO:teuthology.orchestra.run.smithi023.stdout: "target_version": "0'0" 2024-03-11T21:15:33.333 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.333 INFO:teuthology.orchestra.run.smithi023.stdout: "last_change": "82", 2024-03-11T21:15:33.333 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend": "0", 2024-03-11T21:15:33.333 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T21:15:33.333 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T21:15:33.333 INFO:teuthology.orchestra.run.smithi023.stdout: "auid": 0, 2024-03-11T21:15:33.333 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_mode": "selfmanaged", 2024-03-11T21:15:33.333 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_seq": 0, 2024-03-11T21:15:33.333 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_epoch": 0, 2024-03-11T21:15:33.333 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_snaps": [], 2024-03-11T21:15:33.333 INFO:teuthology.orchestra.run.smithi023.stdout: "removed_snaps": "[]", 2024-03-11T21:15:33.333 INFO:teuthology.orchestra.run.smithi023.stdout: "quota_max_bytes": 0, 2024-03-11T21:15:33.334 INFO:teuthology.orchestra.run.smithi023.stdout: "quota_max_objects": 0, 2024-03-11T21:15:33.334 INFO:teuthology.orchestra.run.smithi023.stdout: "tiers": [], 2024-03-11T21:15:33.334 INFO:teuthology.orchestra.run.smithi023.stdout: "tier_of": -1, 2024-03-11T21:15:33.334 INFO:teuthology.orchestra.run.smithi023.stdout: "read_tier": -1, 2024-03-11T21:15:33.334 INFO:teuthology.orchestra.run.smithi023.stdout: "write_tier": -1, 2024-03-11T21:15:33.334 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_mode": "none", 2024-03-11T21:15:33.334 INFO:teuthology.orchestra.run.smithi023.stdout: "target_max_bytes": 0, 2024-03-11T21:15:33.334 INFO:teuthology.orchestra.run.smithi023.stdout: "target_max_objects": 0, 2024-03-11T21:15:33.334 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T21:15:33.334 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T21:15:33.334 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T21:15:33.334 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_min_flush_age": 0, 2024-03-11T21:15:33.334 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_min_evict_age": 0, 2024-03-11T21:15:33.335 INFO:teuthology.orchestra.run.smithi023.stdout: "erasure_code_profile": "", 2024-03-11T21:15:33.335 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_params": { 2024-03-11T21:15:33.335 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "none" 2024-03-11T21:15:33.335 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.335 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_period": 0, 2024-03-11T21:15:33.335 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_count": 0, 2024-03-11T21:15:33.335 INFO:teuthology.orchestra.run.smithi023.stdout: "use_gmt_hitset": true, 2024-03-11T21:15:33.335 INFO:teuthology.orchestra.run.smithi023.stdout: "min_read_recency_for_promote": 0, 2024-03-11T21:15:33.335 INFO:teuthology.orchestra.run.smithi023.stdout: "min_write_recency_for_promote": 0, 2024-03-11T21:15:33.335 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T21:15:33.335 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_search_last_n": 0, 2024-03-11T21:15:33.335 INFO:teuthology.orchestra.run.smithi023.stdout: "grade_table": [], 2024-03-11T21:15:33.335 INFO:teuthology.orchestra.run.smithi023.stdout: "stripe_width": 0, 2024-03-11T21:15:33.335 INFO:teuthology.orchestra.run.smithi023.stdout: "expected_num_objects": 0, 2024-03-11T21:15:33.336 INFO:teuthology.orchestra.run.smithi023.stdout: "fast_read": false, 2024-03-11T21:15:33.336 INFO:teuthology.orchestra.run.smithi023.stdout: "options": {}, 2024-03-11T21:15:33.336 INFO:teuthology.orchestra.run.smithi023.stdout: "application_metadata": { 2024-03-11T21:15:33.336 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw": {} 2024-03-11T21:15:33.336 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.336 INFO:teuthology.orchestra.run.smithi023.stdout: "read_balance": { 2024-03-11T21:15:33.336 INFO:teuthology.orchestra.run.smithi023.stdout: "score_type": "Fair distribution", 2024-03-11T21:15:33.336 INFO:teuthology.orchestra.run.smithi023.stdout: "score_acting": 1.8799999952316284, 2024-03-11T21:15:33.336 INFO:teuthology.orchestra.run.smithi023.stdout: "score_stable": 1.8799999952316284, 2024-03-11T21:15:33.336 INFO:teuthology.orchestra.run.smithi023.stdout: "optimal_score": 1.3300000429153442, 2024-03-11T21:15:33.336 INFO:teuthology.orchestra.run.smithi023.stdout: "raw_score_acting": 2.5, 2024-03-11T21:15:33.336 INFO:teuthology.orchestra.run.smithi023.stdout: "raw_score_stable": 2.5, 2024-03-11T21:15:33.336 INFO:teuthology.orchestra.run.smithi023.stdout: "primary_affinity_weighted": 0.75, 2024-03-11T21:15:33.336 INFO:teuthology.orchestra.run.smithi023.stdout: "average_primary_affinity": 1, 2024-03-11T21:15:33.336 INFO:teuthology.orchestra.run.smithi023.stdout: "average_primary_affinity_weighted": 0.75 2024-03-11T21:15:33.337 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:15:33.337 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.337 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:33.337 INFO:teuthology.orchestra.run.smithi023.stdout: "pool": 5, 2024-03-11T21:15:33.337 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_name": "datapool", 2024-03-11T21:15:33.337 INFO:teuthology.orchestra.run.smithi023.stdout: "create_time": "2024-03-11T20:50:31.135845+0000", 2024-03-11T21:15:33.337 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 8193, 2024-03-11T21:15:33.337 INFO:teuthology.orchestra.run.smithi023.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-03-11T21:15:33.337 INFO:teuthology.orchestra.run.smithi023.stdout: "type": 1, 2024-03-11T21:15:33.337 INFO:teuthology.orchestra.run.smithi023.stdout: "size": 3, 2024-03-11T21:15:33.337 INFO:teuthology.orchestra.run.smithi023.stdout: "min_size": 2, 2024-03-11T21:15:33.337 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_rule": 0, 2024-03-11T21:15:33.337 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_count": 0, 2024-03-11T21:15:33.337 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_target": 0, 2024-03-11T21:15:33.337 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T21:15:33.338 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T21:15:33.338 INFO:teuthology.orchestra.run.smithi023.stdout: "object_hash": 2, 2024-03-11T21:15:33.338 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_autoscale_mode": "off", 2024-03-11T21:15:33.338 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num": 3, 2024-03-11T21:15:33.338 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_placement_num": 3, 2024-03-11T21:15:33.338 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_placement_num_target": 3, 2024-03-11T21:15:33.338 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num_target": 3, 2024-03-11T21:15:33.338 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num_pending": 3, 2024-03-11T21:15:33.338 INFO:teuthology.orchestra.run.smithi023.stdout: "last_pg_merge_meta": { 2024-03-11T21:15:33.338 INFO:teuthology.orchestra.run.smithi023.stdout: "source_pgid": "0.0", 2024-03-11T21:15:33.338 INFO:teuthology.orchestra.run.smithi023.stdout: "ready_epoch": 0, 2024-03-11T21:15:33.338 INFO:teuthology.orchestra.run.smithi023.stdout: "last_epoch_started": 0, 2024-03-11T21:15:33.338 INFO:teuthology.orchestra.run.smithi023.stdout: "last_epoch_clean": 0, 2024-03-11T21:15:33.339 INFO:teuthology.orchestra.run.smithi023.stdout: "source_version": "0'0", 2024-03-11T21:15:33.339 INFO:teuthology.orchestra.run.smithi023.stdout: "target_version": "0'0" 2024-03-11T21:15:33.339 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.339 INFO:teuthology.orchestra.run.smithi023.stdout: "last_change": "87", 2024-03-11T21:15:33.339 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend": "0", 2024-03-11T21:15:33.339 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T21:15:33.339 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T21:15:33.339 INFO:teuthology.orchestra.run.smithi023.stdout: "auid": 0, 2024-03-11T21:15:33.339 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_mode": "selfmanaged", 2024-03-11T21:15:33.339 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_seq": 2, 2024-03-11T21:15:33.339 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_epoch": 87, 2024-03-11T21:15:33.339 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_snaps": [], 2024-03-11T21:15:33.339 INFO:teuthology.orchestra.run.smithi023.stdout: "removed_snaps": "[]", 2024-03-11T21:15:33.339 INFO:teuthology.orchestra.run.smithi023.stdout: "quota_max_bytes": 0, 2024-03-11T21:15:33.339 INFO:teuthology.orchestra.run.smithi023.stdout: "quota_max_objects": 0, 2024-03-11T21:15:33.340 INFO:teuthology.orchestra.run.smithi023.stdout: "tiers": [], 2024-03-11T21:15:33.340 INFO:teuthology.orchestra.run.smithi023.stdout: "tier_of": -1, 2024-03-11T21:15:33.340 INFO:teuthology.orchestra.run.smithi023.stdout: "read_tier": -1, 2024-03-11T21:15:33.340 INFO:teuthology.orchestra.run.smithi023.stdout: "write_tier": -1, 2024-03-11T21:15:33.340 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_mode": "none", 2024-03-11T21:15:33.340 INFO:teuthology.orchestra.run.smithi023.stdout: "target_max_bytes": 0, 2024-03-11T21:15:33.340 INFO:teuthology.orchestra.run.smithi023.stdout: "target_max_objects": 0, 2024-03-11T21:15:33.340 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T21:15:33.340 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T21:15:33.340 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T21:15:33.340 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_min_flush_age": 0, 2024-03-11T21:15:33.340 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_min_evict_age": 0, 2024-03-11T21:15:33.340 INFO:teuthology.orchestra.run.smithi023.stdout: "erasure_code_profile": "", 2024-03-11T21:15:33.340 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_params": { 2024-03-11T21:15:33.340 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "none" 2024-03-11T21:15:33.340 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.341 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_period": 0, 2024-03-11T21:15:33.341 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_count": 0, 2024-03-11T21:15:33.341 INFO:teuthology.orchestra.run.smithi023.stdout: "use_gmt_hitset": true, 2024-03-11T21:15:33.341 INFO:teuthology.orchestra.run.smithi023.stdout: "min_read_recency_for_promote": 0, 2024-03-11T21:15:33.341 INFO:teuthology.orchestra.run.smithi023.stdout: "min_write_recency_for_promote": 0, 2024-03-11T21:15:33.341 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T21:15:33.341 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_search_last_n": 0, 2024-03-11T21:15:33.341 INFO:teuthology.orchestra.run.smithi023.stdout: "grade_table": [], 2024-03-11T21:15:33.341 INFO:teuthology.orchestra.run.smithi023.stdout: "stripe_width": 0, 2024-03-11T21:15:33.341 INFO:teuthology.orchestra.run.smithi023.stdout: "expected_num_objects": 0, 2024-03-11T21:15:33.341 INFO:teuthology.orchestra.run.smithi023.stdout: "fast_read": false, 2024-03-11T21:15:33.341 INFO:teuthology.orchestra.run.smithi023.stdout: "options": {}, 2024-03-11T21:15:33.341 INFO:teuthology.orchestra.run.smithi023.stdout: "application_metadata": { 2024-03-11T21:15:33.341 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd": {} 2024-03-11T21:15:33.341 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.342 INFO:teuthology.orchestra.run.smithi023.stdout: "read_balance": { 2024-03-11T21:15:33.342 INFO:teuthology.orchestra.run.smithi023.stdout: "score_type": "Fair distribution", 2024-03-11T21:15:33.342 INFO:teuthology.orchestra.run.smithi023.stdout: "score_acting": 3.9900000095367432, 2024-03-11T21:15:33.342 INFO:teuthology.orchestra.run.smithi023.stdout: "score_stable": 3.9900000095367432, 2024-03-11T21:15:33.342 INFO:teuthology.orchestra.run.smithi023.stdout: "optimal_score": 1.1699999570846558, 2024-03-11T21:15:33.342 INFO:teuthology.orchestra.run.smithi023.stdout: "raw_score_acting": 4.6700000762939453, 2024-03-11T21:15:33.342 INFO:teuthology.orchestra.run.smithi023.stdout: "raw_score_stable": 4.6700000762939453, 2024-03-11T21:15:33.342 INFO:teuthology.orchestra.run.smithi023.stdout: "primary_affinity_weighted": 0.75, 2024-03-11T21:15:33.342 INFO:teuthology.orchestra.run.smithi023.stdout: "average_primary_affinity": 1, 2024-03-11T21:15:33.342 INFO:teuthology.orchestra.run.smithi023.stdout: "average_primary_affinity_weighted": 0.75 2024-03-11T21:15:33.342 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:15:33.342 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.342 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:33.342 INFO:teuthology.orchestra.run.smithi023.stdout: "pool": 6, 2024-03-11T21:15:33.343 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_name": "default.rgw.meta", 2024-03-11T21:15:33.343 INFO:teuthology.orchestra.run.smithi023.stdout: "create_time": "2024-03-11T20:50:33.233163+0000", 2024-03-11T21:15:33.343 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-11T21:15:33.343 INFO:teuthology.orchestra.run.smithi023.stdout: "flags_names": "hashpspool", 2024-03-11T21:15:33.343 INFO:teuthology.orchestra.run.smithi023.stdout: "type": 1, 2024-03-11T21:15:33.343 INFO:teuthology.orchestra.run.smithi023.stdout: "size": 3, 2024-03-11T21:15:33.343 INFO:teuthology.orchestra.run.smithi023.stdout: "min_size": 2, 2024-03-11T21:15:33.343 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_rule": 0, 2024-03-11T21:15:33.343 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_count": 0, 2024-03-11T21:15:33.343 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_target": 0, 2024-03-11T21:15:33.343 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T21:15:33.343 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T21:15:33.343 INFO:teuthology.orchestra.run.smithi023.stdout: "object_hash": 2, 2024-03-11T21:15:33.343 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_autoscale_mode": "off", 2024-03-11T21:15:33.344 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num": 32, 2024-03-11T21:15:33.344 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_placement_num": 32, 2024-03-11T21:15:33.344 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_placement_num_target": 32, 2024-03-11T21:15:33.344 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num_target": 32, 2024-03-11T21:15:33.344 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num_pending": 32, 2024-03-11T21:15:33.344 INFO:teuthology.orchestra.run.smithi023.stdout: "last_pg_merge_meta": { 2024-03-11T21:15:33.344 INFO:teuthology.orchestra.run.smithi023.stdout: "source_pgid": "0.0", 2024-03-11T21:15:33.344 INFO:teuthology.orchestra.run.smithi023.stdout: "ready_epoch": 0, 2024-03-11T21:15:33.344 INFO:teuthology.orchestra.run.smithi023.stdout: "last_epoch_started": 0, 2024-03-11T21:15:33.344 INFO:teuthology.orchestra.run.smithi023.stdout: "last_epoch_clean": 0, 2024-03-11T21:15:33.344 INFO:teuthology.orchestra.run.smithi023.stdout: "source_version": "0'0", 2024-03-11T21:15:33.344 INFO:teuthology.orchestra.run.smithi023.stdout: "target_version": "0'0" 2024-03-11T21:15:33.344 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.344 INFO:teuthology.orchestra.run.smithi023.stdout: "last_change": "84", 2024-03-11T21:15:33.345 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend": "0", 2024-03-11T21:15:33.345 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T21:15:33.345 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T21:15:33.345 INFO:teuthology.orchestra.run.smithi023.stdout: "auid": 0, 2024-03-11T21:15:33.345 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_mode": "selfmanaged", 2024-03-11T21:15:33.345 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_seq": 0, 2024-03-11T21:15:33.345 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_epoch": 0, 2024-03-11T21:15:33.345 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_snaps": [], 2024-03-11T21:15:33.345 INFO:teuthology.orchestra.run.smithi023.stdout: "removed_snaps": "[]", 2024-03-11T21:15:33.345 INFO:teuthology.orchestra.run.smithi023.stdout: "quota_max_bytes": 0, 2024-03-11T21:15:33.345 INFO:teuthology.orchestra.run.smithi023.stdout: "quota_max_objects": 0, 2024-03-11T21:15:33.345 INFO:teuthology.orchestra.run.smithi023.stdout: "tiers": [], 2024-03-11T21:15:33.345 INFO:teuthology.orchestra.run.smithi023.stdout: "tier_of": -1, 2024-03-11T21:15:33.345 INFO:teuthology.orchestra.run.smithi023.stdout: "read_tier": -1, 2024-03-11T21:15:33.346 INFO:teuthology.orchestra.run.smithi023.stdout: "write_tier": -1, 2024-03-11T21:15:33.346 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_mode": "none", 2024-03-11T21:15:33.346 INFO:teuthology.orchestra.run.smithi023.stdout: "target_max_bytes": 0, 2024-03-11T21:15:33.346 INFO:teuthology.orchestra.run.smithi023.stdout: "target_max_objects": 0, 2024-03-11T21:15:33.346 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T21:15:33.346 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T21:15:33.346 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T21:15:33.346 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_min_flush_age": 0, 2024-03-11T21:15:33.346 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_min_evict_age": 0, 2024-03-11T21:15:33.346 INFO:teuthology.orchestra.run.smithi023.stdout: "erasure_code_profile": "", 2024-03-11T21:15:33.346 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_params": { 2024-03-11T21:15:33.346 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "none" 2024-03-11T21:15:33.346 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.346 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_period": 0, 2024-03-11T21:15:33.347 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_count": 0, 2024-03-11T21:15:33.347 INFO:teuthology.orchestra.run.smithi023.stdout: "use_gmt_hitset": true, 2024-03-11T21:15:33.347 INFO:teuthology.orchestra.run.smithi023.stdout: "min_read_recency_for_promote": 0, 2024-03-11T21:15:33.347 INFO:teuthology.orchestra.run.smithi023.stdout: "min_write_recency_for_promote": 0, 2024-03-11T21:15:33.347 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T21:15:33.347 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_search_last_n": 0, 2024-03-11T21:15:33.347 INFO:teuthology.orchestra.run.smithi023.stdout: "grade_table": [], 2024-03-11T21:15:33.347 INFO:teuthology.orchestra.run.smithi023.stdout: "stripe_width": 0, 2024-03-11T21:15:33.347 INFO:teuthology.orchestra.run.smithi023.stdout: "expected_num_objects": 0, 2024-03-11T21:15:33.347 INFO:teuthology.orchestra.run.smithi023.stdout: "fast_read": false, 2024-03-11T21:15:33.347 INFO:teuthology.orchestra.run.smithi023.stdout: "options": { 2024-03-11T21:15:33.347 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_autoscale_bias": 4 2024-03-11T21:15:33.347 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.347 INFO:teuthology.orchestra.run.smithi023.stdout: "application_metadata": { 2024-03-11T21:15:33.348 INFO:teuthology.orchestra.run.smithi023.stdout: "rgw": {} 2024-03-11T21:15:33.348 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.348 INFO:teuthology.orchestra.run.smithi023.stdout: "read_balance": { 2024-03-11T21:15:33.348 INFO:teuthology.orchestra.run.smithi023.stdout: "score_type": "Fair distribution", 2024-03-11T21:15:33.348 INFO:teuthology.orchestra.run.smithi023.stdout: "score_acting": 1.690000057220459, 2024-03-11T21:15:33.348 INFO:teuthology.orchestra.run.smithi023.stdout: "score_stable": 1.690000057220459, 2024-03-11T21:15:33.348 INFO:teuthology.orchestra.run.smithi023.stdout: "optimal_score": 1.3300000429153442, 2024-03-11T21:15:33.348 INFO:teuthology.orchestra.run.smithi023.stdout: "raw_score_acting": 2.25, 2024-03-11T21:15:33.348 INFO:teuthology.orchestra.run.smithi023.stdout: "raw_score_stable": 2.25, 2024-03-11T21:15:33.348 INFO:teuthology.orchestra.run.smithi023.stdout: "primary_affinity_weighted": 0.75, 2024-03-11T21:15:33.348 INFO:teuthology.orchestra.run.smithi023.stdout: "average_primary_affinity": 1, 2024-03-11T21:15:33.348 INFO:teuthology.orchestra.run.smithi023.stdout: "average_primary_affinity_weighted": 0.75 2024-03-11T21:15:33.348 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:15:33.348 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.349 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:33.349 INFO:teuthology.orchestra.run.smithi023.stdout: "pool": 11, 2024-03-11T21:15:33.349 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_name": "unique_pool_4", 2024-03-11T21:15:33.349 INFO:teuthology.orchestra.run.smithi023.stdout: "create_time": "2024-03-11T21:11:46.486253+0000", 2024-03-11T21:15:33.349 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-11T21:15:33.349 INFO:teuthology.orchestra.run.smithi023.stdout: "flags_names": "hashpspool", 2024-03-11T21:15:33.349 INFO:teuthology.orchestra.run.smithi023.stdout: "type": 1, 2024-03-11T21:15:33.349 INFO:teuthology.orchestra.run.smithi023.stdout: "size": 3, 2024-03-11T21:15:33.349 INFO:teuthology.orchestra.run.smithi023.stdout: "min_size": 2, 2024-03-11T21:15:33.349 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_rule": 0, 2024-03-11T21:15:33.349 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_count": 0, 2024-03-11T21:15:33.349 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_target": 0, 2024-03-11T21:15:33.349 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T21:15:33.350 INFO:teuthology.orchestra.run.smithi023.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T21:15:33.350 INFO:teuthology.orchestra.run.smithi023.stdout: "object_hash": 2, 2024-03-11T21:15:33.350 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_autoscale_mode": "off", 2024-03-11T21:15:33.350 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num": 16, 2024-03-11T21:15:33.350 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_placement_num": 16, 2024-03-11T21:15:33.350 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_placement_num_target": 16, 2024-03-11T21:15:33.350 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num_target": 16, 2024-03-11T21:15:33.350 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num_pending": 16, 2024-03-11T21:15:33.350 INFO:teuthology.orchestra.run.smithi023.stdout: "last_pg_merge_meta": { 2024-03-11T21:15:33.350 INFO:teuthology.orchestra.run.smithi023.stdout: "source_pgid": "0.0", 2024-03-11T21:15:33.350 INFO:teuthology.orchestra.run.smithi023.stdout: "ready_epoch": 0, 2024-03-11T21:15:33.350 INFO:teuthology.orchestra.run.smithi023.stdout: "last_epoch_started": 0, 2024-03-11T21:15:33.350 INFO:teuthology.orchestra.run.smithi023.stdout: "last_epoch_clean": 0, 2024-03-11T21:15:33.350 INFO:teuthology.orchestra.run.smithi023.stdout: "source_version": "0'0", 2024-03-11T21:15:33.351 INFO:teuthology.orchestra.run.smithi023.stdout: "target_version": "0'0" 2024-03-11T21:15:33.351 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.351 INFO:teuthology.orchestra.run.smithi023.stdout: "last_change": "386", 2024-03-11T21:15:33.351 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend": "0", 2024-03-11T21:15:33.351 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T21:15:33.351 INFO:teuthology.orchestra.run.smithi023.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T21:15:33.351 INFO:teuthology.orchestra.run.smithi023.stdout: "auid": 0, 2024-03-11T21:15:33.351 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_mode": "selfmanaged", 2024-03-11T21:15:33.351 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_seq": 0, 2024-03-11T21:15:33.351 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_epoch": 0, 2024-03-11T21:15:33.351 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_snaps": [], 2024-03-11T21:15:33.351 INFO:teuthology.orchestra.run.smithi023.stdout: "removed_snaps": "[]", 2024-03-11T21:15:33.351 INFO:teuthology.orchestra.run.smithi023.stdout: "quota_max_bytes": 0, 2024-03-11T21:15:33.351 INFO:teuthology.orchestra.run.smithi023.stdout: "quota_max_objects": 0, 2024-03-11T21:15:33.351 INFO:teuthology.orchestra.run.smithi023.stdout: "tiers": [], 2024-03-11T21:15:33.352 INFO:teuthology.orchestra.run.smithi023.stdout: "tier_of": -1, 2024-03-11T21:15:33.352 INFO:teuthology.orchestra.run.smithi023.stdout: "read_tier": -1, 2024-03-11T21:15:33.352 INFO:teuthology.orchestra.run.smithi023.stdout: "write_tier": -1, 2024-03-11T21:15:33.352 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_mode": "none", 2024-03-11T21:15:33.352 INFO:teuthology.orchestra.run.smithi023.stdout: "target_max_bytes": 0, 2024-03-11T21:15:33.352 INFO:teuthology.orchestra.run.smithi023.stdout: "target_max_objects": 0, 2024-03-11T21:15:33.352 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T21:15:33.352 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T21:15:33.352 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T21:15:33.352 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_min_flush_age": 0, 2024-03-11T21:15:33.352 INFO:teuthology.orchestra.run.smithi023.stdout: "cache_min_evict_age": 0, 2024-03-11T21:15:33.352 INFO:teuthology.orchestra.run.smithi023.stdout: "erasure_code_profile": "", 2024-03-11T21:15:33.352 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_params": { 2024-03-11T21:15:33.352 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "none" 2024-03-11T21:15:33.352 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.353 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_period": 0, 2024-03-11T21:15:33.353 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_count": 0, 2024-03-11T21:15:33.353 INFO:teuthology.orchestra.run.smithi023.stdout: "use_gmt_hitset": true, 2024-03-11T21:15:33.353 INFO:teuthology.orchestra.run.smithi023.stdout: "min_read_recency_for_promote": 0, 2024-03-11T21:15:33.353 INFO:teuthology.orchestra.run.smithi023.stdout: "min_write_recency_for_promote": 0, 2024-03-11T21:15:33.353 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T21:15:33.353 INFO:teuthology.orchestra.run.smithi023.stdout: "hit_set_search_last_n": 0, 2024-03-11T21:15:33.353 INFO:teuthology.orchestra.run.smithi023.stdout: "grade_table": [], 2024-03-11T21:15:33.353 INFO:teuthology.orchestra.run.smithi023.stdout: "stripe_width": 0, 2024-03-11T21:15:33.353 INFO:teuthology.orchestra.run.smithi023.stdout: "expected_num_objects": 0, 2024-03-11T21:15:33.353 INFO:teuthology.orchestra.run.smithi023.stdout: "fast_read": false, 2024-03-11T21:15:33.353 INFO:teuthology.orchestra.run.smithi023.stdout: "options": {}, 2024-03-11T21:15:33.353 INFO:teuthology.orchestra.run.smithi023.stdout: "application_metadata": { 2024-03-11T21:15:33.353 INFO:teuthology.orchestra.run.smithi023.stdout: "rados": {} 2024-03-11T21:15:33.354 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.354 INFO:teuthology.orchestra.run.smithi023.stdout: "read_balance": { 2024-03-11T21:15:33.354 INFO:teuthology.orchestra.run.smithi023.stdout: "score_type": "Fair distribution", 2024-03-11T21:15:33.354 INFO:teuthology.orchestra.run.smithi023.stdout: "score_acting": 1.8799999952316284, 2024-03-11T21:15:33.354 INFO:teuthology.orchestra.run.smithi023.stdout: "score_stable": 1.8799999952316284, 2024-03-11T21:15:33.354 INFO:teuthology.orchestra.run.smithi023.stdout: "optimal_score": 1.3300000429153442, 2024-03-11T21:15:33.354 INFO:teuthology.orchestra.run.smithi023.stdout: "raw_score_acting": 2.5, 2024-03-11T21:15:33.354 INFO:teuthology.orchestra.run.smithi023.stdout: "raw_score_stable": 2.5, 2024-03-11T21:15:33.354 INFO:teuthology.orchestra.run.smithi023.stdout: "primary_affinity_weighted": 0.75, 2024-03-11T21:15:33.354 INFO:teuthology.orchestra.run.smithi023.stdout: "average_primary_affinity": 1, 2024-03-11T21:15:33.354 INFO:teuthology.orchestra.run.smithi023.stdout: "average_primary_affinity_weighted": 0.75 2024-03-11T21:15:33.354 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:15:33.354 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:15:33.354 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-11T21:15:33.355 INFO:teuthology.orchestra.run.smithi023.stdout: "osds": [ 2024-03-11T21:15:33.355 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:33.355 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": 0, 2024-03-11T21:15:33.355 INFO:teuthology.orchestra.run.smithi023.stdout: "uuid": "3df72b4a-8fe4-4265-b4c6-af022e1e23e1", 2024-03-11T21:15:33.355 INFO:teuthology.orchestra.run.smithi023.stdout: "up": 1, 2024-03-11T21:15:33.355 INFO:teuthology.orchestra.run.smithi023.stdout: "in": 1, 2024-03-11T21:15:33.355 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 1, 2024-03-11T21:15:33.355 INFO:teuthology.orchestra.run.smithi023.stdout: "primary_affinity": 1, 2024-03-11T21:15:33.355 INFO:teuthology.orchestra.run.smithi023.stdout: "last_clean_begin": 0, 2024-03-11T21:15:33.355 INFO:teuthology.orchestra.run.smithi023.stdout: "last_clean_end": 0, 2024-03-11T21:15:33.355 INFO:teuthology.orchestra.run.smithi023.stdout: "up_from": 8, 2024-03-11T21:15:33.355 INFO:teuthology.orchestra.run.smithi023.stdout: "up_thru": 423, 2024-03-11T21:15:33.355 INFO:teuthology.orchestra.run.smithi023.stdout: "down_at": 0, 2024-03-11T21:15:33.355 INFO:teuthology.orchestra.run.smithi023.stdout: "lost_at": 0, 2024-03-11T21:15:33.356 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2024-03-11T21:15:33.356 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:15:33.356 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:33.356 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:15:33.356 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6802", 2024-03-11T21:15:33.356 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 2154627393 2024-03-11T21:15:33.356 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.356 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:33.356 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:15:33.356 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6803", 2024-03-11T21:15:33.356 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 2154627393 2024-03-11T21:15:33.356 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:15:33.356 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:15:33.356 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.356 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_addrs": { 2024-03-11T21:15:33.357 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:15:33.357 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:33.357 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:15:33.357 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6804", 2024-03-11T21:15:33.357 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 2154627393 2024-03-11T21:15:33.357 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.357 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:33.357 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:15:33.357 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6805", 2024-03-11T21:15:33.357 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 2154627393 2024-03-11T21:15:33.357 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:15:33.357 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:15:33.357 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.357 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_back_addrs": { 2024-03-11T21:15:33.357 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:15:33.358 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:33.358 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:15:33.358 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6808", 2024-03-11T21:15:33.358 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 2154627393 2024-03-11T21:15:33.358 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.358 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:33.358 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:15:33.358 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6809", 2024-03-11T21:15:33.358 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 2154627393 2024-03-11T21:15:33.358 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:15:33.358 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:15:33.358 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.358 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_front_addrs": { 2024-03-11T21:15:33.358 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:15:33.359 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:33.359 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:15:33.359 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6806", 2024-03-11T21:15:33.359 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 2154627393 2024-03-11T21:15:33.359 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.359 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:33.359 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:15:33.359 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6807", 2024-03-11T21:15:33.359 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 2154627393 2024-03-11T21:15:33.359 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:15:33.359 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:15:33.359 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.359 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6803/2154627393", 2024-03-11T21:15:33.359 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_addr": "172.21.15.23:6805/2154627393", 2024-03-11T21:15:33.360 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_back_addr": "172.21.15.23:6809/2154627393", 2024-03-11T21:15:33.360 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_front_addr": "172.21.15.23:6807/2154627393", 2024-03-11T21:15:33.360 INFO:teuthology.orchestra.run.smithi023.stdout: "state": [ 2024-03-11T21:15:33.360 INFO:teuthology.orchestra.run.smithi023.stdout: "exists", 2024-03-11T21:15:33.360 INFO:teuthology.orchestra.run.smithi023.stdout: "up" 2024-03-11T21:15:33.360 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:15:33.360 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.360 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:33.360 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": 1, 2024-03-11T21:15:33.360 INFO:teuthology.orchestra.run.smithi023.stdout: "uuid": "e8fd7500-9081-41a1-8077-a4d0f0e12a2c", 2024-03-11T21:15:33.360 INFO:teuthology.orchestra.run.smithi023.stdout: "up": 1, 2024-03-11T21:15:33.360 INFO:teuthology.orchestra.run.smithi023.stdout: "in": 1, 2024-03-11T21:15:33.360 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 1, 2024-03-11T21:15:33.360 INFO:teuthology.orchestra.run.smithi023.stdout: "primary_affinity": 1, 2024-03-11T21:15:33.361 INFO:teuthology.orchestra.run.smithi023.stdout: "last_clean_begin": 13, 2024-03-11T21:15:33.361 INFO:teuthology.orchestra.run.smithi023.stdout: "last_clean_end": 173, 2024-03-11T21:15:33.361 INFO:teuthology.orchestra.run.smithi023.stdout: "up_from": 211, 2024-03-11T21:15:33.361 INFO:teuthology.orchestra.run.smithi023.stdout: "up_thru": 424, 2024-03-11T21:15:33.361 INFO:teuthology.orchestra.run.smithi023.stdout: "down_at": 174, 2024-03-11T21:15:33.361 INFO:teuthology.orchestra.run.smithi023.stdout: "lost_at": 0, 2024-03-11T21:15:33.361 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2024-03-11T21:15:33.361 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:15:33.361 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:33.361 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:15:33.361 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6810", 2024-03-11T21:15:33.361 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 56585354 2024-03-11T21:15:33.361 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.361 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:33.361 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:15:33.362 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6811", 2024-03-11T21:15:33.362 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 56585354 2024-03-11T21:15:33.362 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:15:33.362 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:15:33.362 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.362 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_addrs": { 2024-03-11T21:15:33.362 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:15:33.362 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:33.362 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:15:33.362 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6812", 2024-03-11T21:15:33.362 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 56585354 2024-03-11T21:15:33.362 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.362 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:33.363 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:15:33.363 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6813", 2024-03-11T21:15:33.363 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 56585354 2024-03-11T21:15:33.363 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:15:33.363 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:15:33.363 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.363 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_back_addrs": { 2024-03-11T21:15:33.363 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:15:33.363 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:33.363 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:15:33.363 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6816", 2024-03-11T21:15:33.363 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 56585354 2024-03-11T21:15:33.363 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.363 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:33.363 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:15:33.364 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6817", 2024-03-11T21:15:33.364 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 56585354 2024-03-11T21:15:33.364 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:15:33.364 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:15:33.364 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.364 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_front_addrs": { 2024-03-11T21:15:33.364 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:15:33.364 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:33.364 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:15:33.364 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6814", 2024-03-11T21:15:33.364 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 56585354 2024-03-11T21:15:33.364 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.364 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:33.364 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:15:33.364 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6815", 2024-03-11T21:15:33.365 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 56585354 2024-03-11T21:15:33.365 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:15:33.365 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:15:33.365 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.365 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6811/56585354", 2024-03-11T21:15:33.365 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_addr": "172.21.15.23:6813/56585354", 2024-03-11T21:15:33.365 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_back_addr": "172.21.15.23:6817/56585354", 2024-03-11T21:15:33.365 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_front_addr": "172.21.15.23:6815/56585354", 2024-03-11T21:15:33.365 INFO:teuthology.orchestra.run.smithi023.stdout: "state": [ 2024-03-11T21:15:33.365 INFO:teuthology.orchestra.run.smithi023.stdout: "exists", 2024-03-11T21:15:33.365 INFO:teuthology.orchestra.run.smithi023.stdout: "up" 2024-03-11T21:15:33.365 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:15:33.365 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.365 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:33.366 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": 2, 2024-03-11T21:15:33.366 INFO:teuthology.orchestra.run.smithi023.stdout: "uuid": "dd85d52a-df48-4b03-8612-cecfd82b5987", 2024-03-11T21:15:33.366 INFO:teuthology.orchestra.run.smithi023.stdout: "up": 1, 2024-03-11T21:15:33.366 INFO:teuthology.orchestra.run.smithi023.stdout: "in": 1, 2024-03-11T21:15:33.366 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 1, 2024-03-11T21:15:33.366 INFO:teuthology.orchestra.run.smithi023.stdout: "primary_affinity": 0, 2024-03-11T21:15:33.366 INFO:teuthology.orchestra.run.smithi023.stdout: "last_clean_begin": 0, 2024-03-11T21:15:33.366 INFO:teuthology.orchestra.run.smithi023.stdout: "last_clean_end": 0, 2024-03-11T21:15:33.366 INFO:teuthology.orchestra.run.smithi023.stdout: "up_from": 19, 2024-03-11T21:15:33.366 INFO:teuthology.orchestra.run.smithi023.stdout: "up_thru": 306, 2024-03-11T21:15:33.366 INFO:teuthology.orchestra.run.smithi023.stdout: "down_at": 0, 2024-03-11T21:15:33.366 INFO:teuthology.orchestra.run.smithi023.stdout: "lost_at": 0, 2024-03-11T21:15:33.366 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2024-03-11T21:15:33.366 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:15:33.366 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:33.367 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:15:33.367 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6818", 2024-03-11T21:15:33.367 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1770748056 2024-03-11T21:15:33.367 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.367 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:33.367 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:15:33.367 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6819", 2024-03-11T21:15:33.367 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1770748056 2024-03-11T21:15:33.367 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:15:33.367 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:15:33.367 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.367 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_addrs": { 2024-03-11T21:15:33.367 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:15:33.368 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:33.368 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:15:33.368 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6820", 2024-03-11T21:15:33.368 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1770748056 2024-03-11T21:15:33.368 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.368 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:33.368 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:15:33.368 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6821", 2024-03-11T21:15:33.368 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1770748056 2024-03-11T21:15:33.368 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:15:33.368 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:15:33.368 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.368 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_back_addrs": { 2024-03-11T21:15:33.368 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:15:33.368 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:33.369 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:15:33.369 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6824", 2024-03-11T21:15:33.369 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1770748056 2024-03-11T21:15:33.369 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.369 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:33.369 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:15:33.369 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6825", 2024-03-11T21:15:33.369 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1770748056 2024-03-11T21:15:33.369 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:15:33.369 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:15:33.369 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.369 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_front_addrs": { 2024-03-11T21:15:33.369 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:15:33.369 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:33.369 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:15:33.369 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6822", 2024-03-11T21:15:33.370 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1770748056 2024-03-11T21:15:33.370 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.370 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:33.370 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:15:33.370 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6823", 2024-03-11T21:15:33.370 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1770748056 2024-03-11T21:15:33.370 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:15:33.370 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:15:33.370 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.370 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6819/1770748056", 2024-03-11T21:15:33.370 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_addr": "172.21.15.23:6821/1770748056", 2024-03-11T21:15:33.370 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_back_addr": "172.21.15.23:6825/1770748056", 2024-03-11T21:15:33.370 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_front_addr": "172.21.15.23:6823/1770748056", 2024-03-11T21:15:33.370 INFO:teuthology.orchestra.run.smithi023.stdout: "state": [ 2024-03-11T21:15:33.371 INFO:teuthology.orchestra.run.smithi023.stdout: "exists", 2024-03-11T21:15:33.371 INFO:teuthology.orchestra.run.smithi023.stdout: "up" 2024-03-11T21:15:33.371 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:15:33.371 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.371 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:33.371 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": 3, 2024-03-11T21:15:33.371 INFO:teuthology.orchestra.run.smithi023.stdout: "uuid": "e3021106-3779-4b94-9bbf-a7724544110f", 2024-03-11T21:15:33.371 INFO:teuthology.orchestra.run.smithi023.stdout: "up": 1, 2024-03-11T21:15:33.371 INFO:teuthology.orchestra.run.smithi023.stdout: "in": 1, 2024-03-11T21:15:33.371 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 1, 2024-03-11T21:15:33.371 INFO:teuthology.orchestra.run.smithi023.stdout: "primary_affinity": 1, 2024-03-11T21:15:33.371 INFO:teuthology.orchestra.run.smithi023.stdout: "last_clean_begin": 0, 2024-03-11T21:15:33.371 INFO:teuthology.orchestra.run.smithi023.stdout: "last_clean_end": 0, 2024-03-11T21:15:33.371 INFO:teuthology.orchestra.run.smithi023.stdout: "up_from": 28, 2024-03-11T21:15:33.372 INFO:teuthology.orchestra.run.smithi023.stdout: "up_thru": 432, 2024-03-11T21:15:33.372 INFO:teuthology.orchestra.run.smithi023.stdout: "down_at": 0, 2024-03-11T21:15:33.372 INFO:teuthology.orchestra.run.smithi023.stdout: "lost_at": 0, 2024-03-11T21:15:33.372 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2024-03-11T21:15:33.372 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:15:33.372 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:33.372 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:15:33.372 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6826", 2024-03-11T21:15:33.372 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 938496814 2024-03-11T21:15:33.372 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.372 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:33.372 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:15:33.372 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6827", 2024-03-11T21:15:33.372 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 938496814 2024-03-11T21:15:33.372 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:15:33.372 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:15:33.373 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.373 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_addrs": { 2024-03-11T21:15:33.373 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:15:33.373 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:33.373 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:15:33.373 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6828", 2024-03-11T21:15:33.373 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 938496814 2024-03-11T21:15:33.373 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.373 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:33.373 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:15:33.373 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6829", 2024-03-11T21:15:33.373 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 938496814 2024-03-11T21:15:33.373 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:15:33.373 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:15:33.373 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.374 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_back_addrs": { 2024-03-11T21:15:33.374 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:15:33.374 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:33.374 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:15:33.374 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6832", 2024-03-11T21:15:33.374 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 938496814 2024-03-11T21:15:33.374 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.374 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:33.374 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:15:33.374 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6833", 2024-03-11T21:15:33.374 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 938496814 2024-03-11T21:15:33.374 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:15:33.374 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:15:33.374 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.375 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_front_addrs": { 2024-03-11T21:15:33.375 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:15:33.375 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:33.375 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:15:33.375 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6830", 2024-03-11T21:15:33.375 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 938496814 2024-03-11T21:15:33.375 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.375 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:33.375 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:15:33.375 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6831", 2024-03-11T21:15:33.375 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 938496814 2024-03-11T21:15:33.375 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:15:33.375 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:15:33.375 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.375 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6827/938496814", 2024-03-11T21:15:33.375 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_addr": "172.21.15.23:6829/938496814", 2024-03-11T21:15:33.376 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_back_addr": "172.21.15.23:6833/938496814", 2024-03-11T21:15:33.376 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_front_addr": "172.21.15.23:6831/938496814", 2024-03-11T21:15:33.376 INFO:teuthology.orchestra.run.smithi023.stdout: "state": [ 2024-03-11T21:15:33.376 INFO:teuthology.orchestra.run.smithi023.stdout: "exists", 2024-03-11T21:15:33.376 INFO:teuthology.orchestra.run.smithi023.stdout: "up" 2024-03-11T21:15:33.376 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:15:33.376 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.376 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:33.376 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": 4, 2024-03-11T21:15:33.376 INFO:teuthology.orchestra.run.smithi023.stdout: "uuid": "4dcbc91e-7035-413d-97bc-c9d804eb16d7", 2024-03-11T21:15:33.376 INFO:teuthology.orchestra.run.smithi023.stdout: "up": 1, 2024-03-11T21:15:33.376 INFO:teuthology.orchestra.run.smithi023.stdout: "in": 1, 2024-03-11T21:15:33.376 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 1, 2024-03-11T21:15:33.377 INFO:teuthology.orchestra.run.smithi023.stdout: "primary_affinity": 0, 2024-03-11T21:15:33.377 INFO:teuthology.orchestra.run.smithi023.stdout: "last_clean_begin": 35, 2024-03-11T21:15:33.377 INFO:teuthology.orchestra.run.smithi023.stdout: "last_clean_end": 182, 2024-03-11T21:15:33.377 INFO:teuthology.orchestra.run.smithi023.stdout: "up_from": 204, 2024-03-11T21:15:33.377 INFO:teuthology.orchestra.run.smithi023.stdout: "up_thru": 248, 2024-03-11T21:15:33.377 INFO:teuthology.orchestra.run.smithi023.stdout: "down_at": 183, 2024-03-11T21:15:33.377 INFO:teuthology.orchestra.run.smithi023.stdout: "lost_at": 0, 2024-03-11T21:15:33.377 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2024-03-11T21:15:33.377 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:15:33.377 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:33.377 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:15:33.377 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6800", 2024-03-11T21:15:33.377 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 3858359434 2024-03-11T21:15:33.377 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.377 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:33.378 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:15:33.378 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6801", 2024-03-11T21:15:33.378 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 3858359434 2024-03-11T21:15:33.378 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:15:33.378 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:15:33.378 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.378 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_addrs": { 2024-03-11T21:15:33.378 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:15:33.378 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:33.378 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:15:33.378 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6802", 2024-03-11T21:15:33.378 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 3858359434 2024-03-11T21:15:33.378 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.378 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:33.379 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:15:33.379 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6803", 2024-03-11T21:15:33.379 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 3858359434 2024-03-11T21:15:33.379 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:15:33.379 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:15:33.379 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.379 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_back_addrs": { 2024-03-11T21:15:33.379 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:15:33.379 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:33.379 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:15:33.379 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6806", 2024-03-11T21:15:33.379 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 3858359434 2024-03-11T21:15:33.379 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.379 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:33.379 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:15:33.380 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6807", 2024-03-11T21:15:33.380 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 3858359434 2024-03-11T21:15:33.380 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:15:33.380 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:15:33.380 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.380 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_front_addrs": { 2024-03-11T21:15:33.380 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:15:33.380 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:33.380 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:15:33.380 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6804", 2024-03-11T21:15:33.380 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 3858359434 2024-03-11T21:15:33.380 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.380 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:33.380 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:15:33.381 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6805", 2024-03-11T21:15:33.381 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 3858359434 2024-03-11T21:15:33.381 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:15:33.381 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:15:33.381 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.381 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.146:6801/3858359434", 2024-03-11T21:15:33.381 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_addr": "172.21.15.146:6803/3858359434", 2024-03-11T21:15:33.381 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_back_addr": "172.21.15.146:6807/3858359434", 2024-03-11T21:15:33.381 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_front_addr": "172.21.15.146:6805/3858359434", 2024-03-11T21:15:33.381 INFO:teuthology.orchestra.run.smithi023.stdout: "state": [ 2024-03-11T21:15:33.381 INFO:teuthology.orchestra.run.smithi023.stdout: "exists", 2024-03-11T21:15:33.381 INFO:teuthology.orchestra.run.smithi023.stdout: "up" 2024-03-11T21:15:33.381 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:15:33.381 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.382 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:33.382 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": 5, 2024-03-11T21:15:33.382 INFO:teuthology.orchestra.run.smithi023.stdout: "uuid": "aefcb766-8bae-4ff4-b71e-6691c8ac02f1", 2024-03-11T21:15:33.382 INFO:teuthology.orchestra.run.smithi023.stdout: "up": 1, 2024-03-11T21:15:33.382 INFO:teuthology.orchestra.run.smithi023.stdout: "in": 1, 2024-03-11T21:15:33.382 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 1, 2024-03-11T21:15:33.382 INFO:teuthology.orchestra.run.smithi023.stdout: "primary_affinity": 1, 2024-03-11T21:15:33.382 INFO:teuthology.orchestra.run.smithi023.stdout: "last_clean_begin": 0, 2024-03-11T21:15:33.382 INFO:teuthology.orchestra.run.smithi023.stdout: "last_clean_end": 0, 2024-03-11T21:15:33.382 INFO:teuthology.orchestra.run.smithi023.stdout: "up_from": 44, 2024-03-11T21:15:33.382 INFO:teuthology.orchestra.run.smithi023.stdout: "up_thru": 426, 2024-03-11T21:15:33.382 INFO:teuthology.orchestra.run.smithi023.stdout: "down_at": 0, 2024-03-11T21:15:33.382 INFO:teuthology.orchestra.run.smithi023.stdout: "lost_at": 0, 2024-03-11T21:15:33.383 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2024-03-11T21:15:33.383 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:15:33.383 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:33.383 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:15:33.383 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6808", 2024-03-11T21:15:33.383 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 4003230814 2024-03-11T21:15:33.383 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.383 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:33.383 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:15:33.383 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6809", 2024-03-11T21:15:33.383 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 4003230814 2024-03-11T21:15:33.383 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:15:33.383 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:15:33.383 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.384 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_addrs": { 2024-03-11T21:15:33.384 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:15:33.384 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:33.384 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:15:33.384 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6810", 2024-03-11T21:15:33.384 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 4003230814 2024-03-11T21:15:33.384 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.384 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:33.384 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:15:33.384 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6811", 2024-03-11T21:15:33.384 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 4003230814 2024-03-11T21:15:33.384 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:15:33.384 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:15:33.384 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.385 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_back_addrs": { 2024-03-11T21:15:33.385 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:15:33.385 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:33.385 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:15:33.385 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6814", 2024-03-11T21:15:33.385 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 4003230814 2024-03-11T21:15:33.385 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.385 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:33.385 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:15:33.385 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6815", 2024-03-11T21:15:33.385 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 4003230814 2024-03-11T21:15:33.385 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:15:33.385 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:15:33.385 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.386 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_front_addrs": { 2024-03-11T21:15:33.386 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:15:33.386 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:33.386 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:15:33.386 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6812", 2024-03-11T21:15:33.386 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 4003230814 2024-03-11T21:15:33.386 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.386 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:33.386 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:15:33.386 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6813", 2024-03-11T21:15:33.386 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 4003230814 2024-03-11T21:15:33.386 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:15:33.386 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:15:33.386 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.387 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.146:6809/4003230814", 2024-03-11T21:15:33.387 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_addr": "172.21.15.146:6811/4003230814", 2024-03-11T21:15:33.387 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_back_addr": "172.21.15.146:6815/4003230814", 2024-03-11T21:15:33.387 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_front_addr": "172.21.15.146:6813/4003230814", 2024-03-11T21:15:33.387 INFO:teuthology.orchestra.run.smithi023.stdout: "state": [ 2024-03-11T21:15:33.387 INFO:teuthology.orchestra.run.smithi023.stdout: "exists", 2024-03-11T21:15:33.387 INFO:teuthology.orchestra.run.smithi023.stdout: "up" 2024-03-11T21:15:33.387 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:15:33.387 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.387 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:33.387 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": 6, 2024-03-11T21:15:33.387 INFO:teuthology.orchestra.run.smithi023.stdout: "uuid": "5cb151e8-1f9b-4617-8a86-adfb06bb9606", 2024-03-11T21:15:33.387 INFO:teuthology.orchestra.run.smithi023.stdout: "up": 1, 2024-03-11T21:15:33.388 INFO:teuthology.orchestra.run.smithi023.stdout: "in": 1, 2024-03-11T21:15:33.388 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 1, 2024-03-11T21:15:33.388 INFO:teuthology.orchestra.run.smithi023.stdout: "primary_affinity": 1, 2024-03-11T21:15:33.388 INFO:teuthology.orchestra.run.smithi023.stdout: "last_clean_begin": 0, 2024-03-11T21:15:33.388 INFO:teuthology.orchestra.run.smithi023.stdout: "last_clean_end": 0, 2024-03-11T21:15:33.388 INFO:teuthology.orchestra.run.smithi023.stdout: "up_from": 57, 2024-03-11T21:15:33.388 INFO:teuthology.orchestra.run.smithi023.stdout: "up_thru": 426, 2024-03-11T21:15:33.388 INFO:teuthology.orchestra.run.smithi023.stdout: "down_at": 0, 2024-03-11T21:15:33.388 INFO:teuthology.orchestra.run.smithi023.stdout: "lost_at": 0, 2024-03-11T21:15:33.388 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2024-03-11T21:15:33.388 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:15:33.388 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:33.388 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:15:33.388 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6816", 2024-03-11T21:15:33.389 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 3334092376 2024-03-11T21:15:33.389 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.389 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:33.389 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:15:33.389 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6817", 2024-03-11T21:15:33.389 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 3334092376 2024-03-11T21:15:33.389 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:15:33.389 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:15:33.389 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.389 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_addrs": { 2024-03-11T21:15:33.389 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:15:33.389 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:33.389 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:15:33.389 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6818", 2024-03-11T21:15:33.390 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 3334092376 2024-03-11T21:15:33.390 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.390 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:33.390 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:15:33.390 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6819", 2024-03-11T21:15:33.390 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 3334092376 2024-03-11T21:15:33.390 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:15:33.390 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:15:33.390 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.390 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_back_addrs": { 2024-03-11T21:15:33.390 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:15:33.390 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:33.390 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:15:33.390 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6822", 2024-03-11T21:15:33.391 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 3334092376 2024-03-11T21:15:33.391 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.391 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:33.391 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:15:33.391 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6823", 2024-03-11T21:15:33.391 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 3334092376 2024-03-11T21:15:33.391 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:15:33.391 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:15:33.391 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.391 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_front_addrs": { 2024-03-11T21:15:33.391 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:15:33.391 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:33.391 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:15:33.391 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6820", 2024-03-11T21:15:33.391 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 3334092376 2024-03-11T21:15:33.392 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.392 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:33.392 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:15:33.392 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6821", 2024-03-11T21:15:33.392 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 3334092376 2024-03-11T21:15:33.392 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:15:33.392 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:15:33.392 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.392 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.146:6817/3334092376", 2024-03-11T21:15:33.392 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_addr": "172.21.15.146:6819/3334092376", 2024-03-11T21:15:33.392 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_back_addr": "172.21.15.146:6823/3334092376", 2024-03-11T21:15:33.392 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_front_addr": "172.21.15.146:6821/3334092376", 2024-03-11T21:15:33.392 INFO:teuthology.orchestra.run.smithi023.stdout: "state": [ 2024-03-11T21:15:33.392 INFO:teuthology.orchestra.run.smithi023.stdout: "exists", 2024-03-11T21:15:33.393 INFO:teuthology.orchestra.run.smithi023.stdout: "up" 2024-03-11T21:15:33.393 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:15:33.393 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.393 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:33.393 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": 7, 2024-03-11T21:15:33.393 INFO:teuthology.orchestra.run.smithi023.stdout: "uuid": "bdbd11a2-fa09-4f82-93a2-6b2d9c750bfd", 2024-03-11T21:15:33.393 INFO:teuthology.orchestra.run.smithi023.stdout: "up": 1, 2024-03-11T21:15:33.393 INFO:teuthology.orchestra.run.smithi023.stdout: "in": 1, 2024-03-11T21:15:33.393 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 1, 2024-03-11T21:15:33.393 INFO:teuthology.orchestra.run.smithi023.stdout: "primary_affinity": 1, 2024-03-11T21:15:33.393 INFO:teuthology.orchestra.run.smithi023.stdout: "last_clean_begin": 0, 2024-03-11T21:15:33.393 INFO:teuthology.orchestra.run.smithi023.stdout: "last_clean_end": 0, 2024-03-11T21:15:33.393 INFO:teuthology.orchestra.run.smithi023.stdout: "up_from": 68, 2024-03-11T21:15:33.393 INFO:teuthology.orchestra.run.smithi023.stdout: "up_thru": 426, 2024-03-11T21:15:33.393 INFO:teuthology.orchestra.run.smithi023.stdout: "down_at": 0, 2024-03-11T21:15:33.394 INFO:teuthology.orchestra.run.smithi023.stdout: "lost_at": 0, 2024-03-11T21:15:33.394 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2024-03-11T21:15:33.394 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:15:33.394 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:33.394 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:15:33.394 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6824", 2024-03-11T21:15:33.394 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1678407923 2024-03-11T21:15:33.394 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.394 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:33.394 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:15:33.394 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6825", 2024-03-11T21:15:33.394 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1678407923 2024-03-11T21:15:33.394 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:15:33.394 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:15:33.395 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.395 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_addrs": { 2024-03-11T21:15:33.395 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:15:33.395 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:33.395 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:15:33.395 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6826", 2024-03-11T21:15:33.395 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1678407923 2024-03-11T21:15:33.395 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.395 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:33.395 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:15:33.395 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6827", 2024-03-11T21:15:33.395 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1678407923 2024-03-11T21:15:33.395 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:15:33.395 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:15:33.396 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.396 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_back_addrs": { 2024-03-11T21:15:33.396 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:15:33.396 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:33.396 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:15:33.396 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6830", 2024-03-11T21:15:33.396 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1678407923 2024-03-11T21:15:33.396 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.396 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:33.396 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:15:33.396 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6831", 2024-03-11T21:15:33.396 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1678407923 2024-03-11T21:15:33.396 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:15:33.396 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:15:33.397 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.397 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_front_addrs": { 2024-03-11T21:15:33.397 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-11T21:15:33.397 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:33.397 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-11T21:15:33.397 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6828", 2024-03-11T21:15:33.397 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1678407923 2024-03-11T21:15:33.397 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.397 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:33.397 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-11T21:15:33.397 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.146:6829", 2024-03-11T21:15:33.397 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1678407923 2024-03-11T21:15:33.397 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:15:33.397 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:15:33.397 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.398 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.146:6825/1678407923", 2024-03-11T21:15:33.398 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_addr": "172.21.15.146:6827/1678407923", 2024-03-11T21:15:33.398 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_back_addr": "172.21.15.146:6831/1678407923", 2024-03-11T21:15:33.398 INFO:teuthology.orchestra.run.smithi023.stdout: "heartbeat_front_addr": "172.21.15.146:6829/1678407923", 2024-03-11T21:15:33.398 INFO:teuthology.orchestra.run.smithi023.stdout: "state": [ 2024-03-11T21:15:33.398 INFO:teuthology.orchestra.run.smithi023.stdout: "exists", 2024-03-11T21:15:33.398 INFO:teuthology.orchestra.run.smithi023.stdout: "up" 2024-03-11T21:15:33.398 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:15:33.398 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:15:33.398 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-11T21:15:33.398 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_xinfo": [ 2024-03-11T21:15:33.398 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:33.398 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": 0, 2024-03-11T21:15:33.398 INFO:teuthology.orchestra.run.smithi023.stdout: "down_stamp": "0.000000", 2024-03-11T21:15:33.399 INFO:teuthology.orchestra.run.smithi023.stdout: "laggy_probability": 0, 2024-03-11T21:15:33.399 INFO:teuthology.orchestra.run.smithi023.stdout: "laggy_interval": 0, 2024-03-11T21:15:33.399 INFO:teuthology.orchestra.run.smithi023.stdout: "features": 4540701547738038271, 2024-03-11T21:15:33.399 INFO:teuthology.orchestra.run.smithi023.stdout: "old_weight": 0, 2024-03-11T21:15:33.399 INFO:teuthology.orchestra.run.smithi023.stdout: "last_purged_snaps_scrub": "2024-03-11T21:14:34.387211+0000", 2024-03-11T21:15:33.399 INFO:teuthology.orchestra.run.smithi023.stdout: "dead_epoch": 0 2024-03-11T21:15:33.399 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.399 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:33.399 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": 1, 2024-03-11T21:15:33.399 INFO:teuthology.orchestra.run.smithi023.stdout: "down_stamp": "0.000000", 2024-03-11T21:15:33.399 INFO:teuthology.orchestra.run.smithi023.stdout: "laggy_probability": 0, 2024-03-11T21:15:33.399 INFO:teuthology.orchestra.run.smithi023.stdout: "laggy_interval": 0, 2024-03-11T21:15:33.399 INFO:teuthology.orchestra.run.smithi023.stdout: "features": 4540701547738038271, 2024-03-11T21:15:33.399 INFO:teuthology.orchestra.run.smithi023.stdout: "old_weight": 0, 2024-03-11T21:15:33.400 INFO:teuthology.orchestra.run.smithi023.stdout: "last_purged_snaps_scrub": "2024-03-11T21:15:00.009812+0000", 2024-03-11T21:15:33.400 INFO:teuthology.orchestra.run.smithi023.stdout: "dead_epoch": 173 2024-03-11T21:15:33.400 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.400 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:33.400 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": 2, 2024-03-11T21:15:33.400 INFO:teuthology.orchestra.run.smithi023.stdout: "down_stamp": "0.000000", 2024-03-11T21:15:33.400 INFO:teuthology.orchestra.run.smithi023.stdout: "laggy_probability": 0, 2024-03-11T21:15:33.400 INFO:teuthology.orchestra.run.smithi023.stdout: "laggy_interval": 0, 2024-03-11T21:15:33.400 INFO:teuthology.orchestra.run.smithi023.stdout: "features": 4540701547738038271, 2024-03-11T21:15:33.400 INFO:teuthology.orchestra.run.smithi023.stdout: "old_weight": 0, 2024-03-11T21:15:33.400 INFO:teuthology.orchestra.run.smithi023.stdout: "last_purged_snaps_scrub": "2024-03-11T21:14:59.261591+0000", 2024-03-11T21:15:33.400 INFO:teuthology.orchestra.run.smithi023.stdout: "dead_epoch": 0 2024-03-11T21:15:33.400 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.400 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:33.400 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": 3, 2024-03-11T21:15:33.401 INFO:teuthology.orchestra.run.smithi023.stdout: "down_stamp": "0.000000", 2024-03-11T21:15:33.401 INFO:teuthology.orchestra.run.smithi023.stdout: "laggy_probability": 0, 2024-03-11T21:15:33.401 INFO:teuthology.orchestra.run.smithi023.stdout: "laggy_interval": 0, 2024-03-11T21:15:33.401 INFO:teuthology.orchestra.run.smithi023.stdout: "features": 4540701547738038271, 2024-03-11T21:15:33.401 INFO:teuthology.orchestra.run.smithi023.stdout: "old_weight": 0, 2024-03-11T21:15:33.401 INFO:teuthology.orchestra.run.smithi023.stdout: "last_purged_snaps_scrub": "2024-03-11T21:15:07.705552+0000", 2024-03-11T21:15:33.401 INFO:teuthology.orchestra.run.smithi023.stdout: "dead_epoch": 0 2024-03-11T21:15:33.401 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.401 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:33.401 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": 4, 2024-03-11T21:15:33.401 INFO:teuthology.orchestra.run.smithi023.stdout: "down_stamp": "0.000000", 2024-03-11T21:15:33.401 INFO:teuthology.orchestra.run.smithi023.stdout: "laggy_probability": 0, 2024-03-11T21:15:33.401 INFO:teuthology.orchestra.run.smithi023.stdout: "laggy_interval": 0, 2024-03-11T21:15:33.401 INFO:teuthology.orchestra.run.smithi023.stdout: "features": 4540701547738038271, 2024-03-11T21:15:33.402 INFO:teuthology.orchestra.run.smithi023.stdout: "old_weight": 0, 2024-03-11T21:15:33.402 INFO:teuthology.orchestra.run.smithi023.stdout: "last_purged_snaps_scrub": "2024-03-11T21:14:43.253151+0000", 2024-03-11T21:15:33.402 INFO:teuthology.orchestra.run.smithi023.stdout: "dead_epoch": 182 2024-03-11T21:15:33.402 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.402 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:33.402 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": 5, 2024-03-11T21:15:33.402 INFO:teuthology.orchestra.run.smithi023.stdout: "down_stamp": "0.000000", 2024-03-11T21:15:33.402 INFO:teuthology.orchestra.run.smithi023.stdout: "laggy_probability": 0, 2024-03-11T21:15:33.402 INFO:teuthology.orchestra.run.smithi023.stdout: "laggy_interval": 0, 2024-03-11T21:15:33.402 INFO:teuthology.orchestra.run.smithi023.stdout: "features": 4540701547738038271, 2024-03-11T21:15:33.402 INFO:teuthology.orchestra.run.smithi023.stdout: "old_weight": 0, 2024-03-11T21:15:33.402 INFO:teuthology.orchestra.run.smithi023.stdout: "last_purged_snaps_scrub": "2024-03-11T21:14:58.803952+0000", 2024-03-11T21:15:33.402 INFO:teuthology.orchestra.run.smithi023.stdout: "dead_epoch": 0 2024-03-11T21:15:33.402 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.403 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:33.403 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": 6, 2024-03-11T21:15:33.403 INFO:teuthology.orchestra.run.smithi023.stdout: "down_stamp": "0.000000", 2024-03-11T21:15:33.403 INFO:teuthology.orchestra.run.smithi023.stdout: "laggy_probability": 0, 2024-03-11T21:15:33.403 INFO:teuthology.orchestra.run.smithi023.stdout: "laggy_interval": 0, 2024-03-11T21:15:33.403 INFO:teuthology.orchestra.run.smithi023.stdout: "features": 4540701547738038271, 2024-03-11T21:15:33.403 INFO:teuthology.orchestra.run.smithi023.stdout: "old_weight": 0, 2024-03-11T21:15:33.403 INFO:teuthology.orchestra.run.smithi023.stdout: "last_purged_snaps_scrub": "2024-03-11T21:15:03.074602+0000", 2024-03-11T21:15:33.403 INFO:teuthology.orchestra.run.smithi023.stdout: "dead_epoch": 0 2024-03-11T21:15:33.403 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.403 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:33.403 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": 7, 2024-03-11T21:15:33.403 INFO:teuthology.orchestra.run.smithi023.stdout: "down_stamp": "0.000000", 2024-03-11T21:15:33.403 INFO:teuthology.orchestra.run.smithi023.stdout: "laggy_probability": 0, 2024-03-11T21:15:33.403 INFO:teuthology.orchestra.run.smithi023.stdout: "laggy_interval": 0, 2024-03-11T21:15:33.404 INFO:teuthology.orchestra.run.smithi023.stdout: "features": 4540701547738038271, 2024-03-11T21:15:33.404 INFO:teuthology.orchestra.run.smithi023.stdout: "old_weight": 0, 2024-03-11T21:15:33.404 INFO:teuthology.orchestra.run.smithi023.stdout: "last_purged_snaps_scrub": "2024-03-11T21:14:29.226970+0000", 2024-03-11T21:15:33.404 INFO:teuthology.orchestra.run.smithi023.stdout: "dead_epoch": 0 2024-03-11T21:15:33.404 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:15:33.404 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-11T21:15:33.404 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_upmap": [], 2024-03-11T21:15:33.404 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_upmap_items": [ 2024-03-11T21:15:33.404 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:33.404 INFO:teuthology.orchestra.run.smithi023.stdout: "pgid": "2.0", 2024-03-11T21:15:33.404 INFO:teuthology.orchestra.run.smithi023.stdout: "mappings": [ 2024-03-11T21:15:33.404 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:33.404 INFO:teuthology.orchestra.run.smithi023.stdout: "from": 7, 2024-03-11T21:15:33.404 INFO:teuthology.orchestra.run.smithi023.stdout: "to": 2 2024-03-11T21:15:33.405 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:15:33.405 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:15:33.405 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.405 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:33.405 INFO:teuthology.orchestra.run.smithi023.stdout: "pgid": "4.b", 2024-03-11T21:15:33.405 INFO:teuthology.orchestra.run.smithi023.stdout: "mappings": [ 2024-03-11T21:15:33.405 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:33.405 INFO:teuthology.orchestra.run.smithi023.stdout: "from": 1, 2024-03-11T21:15:33.405 INFO:teuthology.orchestra.run.smithi023.stdout: "to": 2 2024-03-11T21:15:33.405 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:15:33.405 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:15:33.405 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.405 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:33.405 INFO:teuthology.orchestra.run.smithi023.stdout: "pgid": "4.d", 2024-03-11T21:15:33.405 INFO:teuthology.orchestra.run.smithi023.stdout: "mappings": [ 2024-03-11T21:15:33.405 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:33.406 INFO:teuthology.orchestra.run.smithi023.stdout: "from": 1, 2024-03-11T21:15:33.406 INFO:teuthology.orchestra.run.smithi023.stdout: "to": 5 2024-03-11T21:15:33.406 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:15:33.406 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:15:33.406 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.406 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:33.406 INFO:teuthology.orchestra.run.smithi023.stdout: "pgid": "4.e", 2024-03-11T21:15:33.406 INFO:teuthology.orchestra.run.smithi023.stdout: "mappings": [ 2024-03-11T21:15:33.406 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:33.406 INFO:teuthology.orchestra.run.smithi023.stdout: "from": 1, 2024-03-11T21:15:33.406 INFO:teuthology.orchestra.run.smithi023.stdout: "to": 2 2024-03-11T21:15:33.406 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:15:33.406 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:15:33.406 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.407 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:33.407 INFO:teuthology.orchestra.run.smithi023.stdout: "pgid": "4.16", 2024-03-11T21:15:33.407 INFO:teuthology.orchestra.run.smithi023.stdout: "mappings": [ 2024-03-11T21:15:33.407 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:33.407 INFO:teuthology.orchestra.run.smithi023.stdout: "from": 3, 2024-03-11T21:15:33.407 INFO:teuthology.orchestra.run.smithi023.stdout: "to": 2 2024-03-11T21:15:33.407 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:15:33.407 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:15:33.407 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.407 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:33.407 INFO:teuthology.orchestra.run.smithi023.stdout: "pgid": "4.1d", 2024-03-11T21:15:33.407 INFO:teuthology.orchestra.run.smithi023.stdout: "mappings": [ 2024-03-11T21:15:33.407 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:33.407 INFO:teuthology.orchestra.run.smithi023.stdout: "from": 4, 2024-03-11T21:15:33.408 INFO:teuthology.orchestra.run.smithi023.stdout: "to": 2 2024-03-11T21:15:33.408 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:15:33.408 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:15:33.408 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.408 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:33.408 INFO:teuthology.orchestra.run.smithi023.stdout: "pgid": "6.a", 2024-03-11T21:15:33.408 INFO:teuthology.orchestra.run.smithi023.stdout: "mappings": [ 2024-03-11T21:15:33.408 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:33.408 INFO:teuthology.orchestra.run.smithi023.stdout: "from": 6, 2024-03-11T21:15:33.408 INFO:teuthology.orchestra.run.smithi023.stdout: "to": 1 2024-03-11T21:15:33.408 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:15:33.408 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:15:33.408 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:15:33.408 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-11T21:15:33.408 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_upmap_primaries": [], 2024-03-11T21:15:33.409 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_temp": [ 2024-03-11T21:15:33.409 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:33.409 INFO:teuthology.orchestra.run.smithi023.stdout: "pgid": "11.1", 2024-03-11T21:15:33.409 INFO:teuthology.orchestra.run.smithi023.stdout: "osds": [ 2024-03-11T21:15:33.409 INFO:teuthology.orchestra.run.smithi023.stdout: 6, 2024-03-11T21:15:33.409 INFO:teuthology.orchestra.run.smithi023.stdout: 3, 2024-03-11T21:15:33.409 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2024-03-11T21:15:33.409 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:15:33.409 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.409 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:33.409 INFO:teuthology.orchestra.run.smithi023.stdout: "pgid": "11.2", 2024-03-11T21:15:33.409 INFO:teuthology.orchestra.run.smithi023.stdout: "osds": [ 2024-03-11T21:15:33.409 INFO:teuthology.orchestra.run.smithi023.stdout: 3, 2024-03-11T21:15:33.409 INFO:teuthology.orchestra.run.smithi023.stdout: 2, 2024-03-11T21:15:33.409 INFO:teuthology.orchestra.run.smithi023.stdout: 7 2024-03-11T21:15:33.410 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:15:33.410 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.410 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:33.410 INFO:teuthology.orchestra.run.smithi023.stdout: "pgid": "11.6", 2024-03-11T21:15:33.410 INFO:teuthology.orchestra.run.smithi023.stdout: "osds": [ 2024-03-11T21:15:33.410 INFO:teuthology.orchestra.run.smithi023.stdout: 1, 2024-03-11T21:15:33.410 INFO:teuthology.orchestra.run.smithi023.stdout: 2, 2024-03-11T21:15:33.410 INFO:teuthology.orchestra.run.smithi023.stdout: 4 2024-03-11T21:15:33.410 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:15:33.410 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:15:33.410 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-11T21:15:33.410 INFO:teuthology.orchestra.run.smithi023.stdout: "primary_temp": [], 2024-03-11T21:15:33.410 INFO:teuthology.orchestra.run.smithi023.stdout: "blocklist": { 2024-03-11T21:15:33.410 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:0/1618334186": "2024-03-12T20:40:45.361537+0000", 2024-03-11T21:15:33.411 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:6801/2813578081": "2024-03-12T20:40:45.361537+0000", 2024-03-11T21:15:33.411 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:0/2869940180": "2024-03-12T20:51:33.790205+0000", 2024-03-11T21:15:33.411 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:6800/3037564930": "2024-03-12T20:51:33.790205+0000", 2024-03-11T21:15:33.411 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:0/3297567084": "2024-03-12T20:51:33.790205+0000", 2024-03-11T21:15:33.411 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:0/3708753659": "2024-03-12T20:40:45.361537+0000", 2024-03-11T21:15:33.411 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:0/2004146740": "2024-03-12T20:40:45.361537+0000", 2024-03-11T21:15:33.411 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:0/827653812": "2024-03-12T20:41:21.301516+0000", 2024-03-11T21:15:33.411 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:6800/1057699850": "2024-03-12T20:41:21.301516+0000", 2024-03-11T21:15:33.411 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:6800/2813578081": "2024-03-12T20:40:45.361537+0000", 2024-03-11T21:15:33.411 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:6801/1057699850": "2024-03-12T20:41:21.301516+0000", 2024-03-11T21:15:33.411 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:0/3833230375": "2024-03-12T20:41:21.301516+0000", 2024-03-11T21:15:33.411 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:6801/3037564930": "2024-03-12T20:51:33.790205+0000", 2024-03-11T21:15:33.411 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:0/1182809102": "2024-03-12T20:51:33.790205+0000", 2024-03-11T21:15:33.411 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:0/2122134187": "2024-03-12T20:51:33.790205+0000", 2024-03-11T21:15:33.412 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:0/2356701899": "2024-03-12T20:41:21.301516+0000", 2024-03-11T21:15:33.412 INFO:teuthology.orchestra.run.smithi023.stdout: "172.21.15.23:0/843974100": "2024-03-12T20:51:33.790205+0000" 2024-03-11T21:15:33.412 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.412 INFO:teuthology.orchestra.run.smithi023.stdout: "range_blocklist": {}, 2024-03-11T21:15:33.412 INFO:teuthology.orchestra.run.smithi023.stdout: "erasure_code_profiles": { 2024-03-11T21:15:33.412 INFO:teuthology.orchestra.run.smithi023.stdout: "default": { 2024-03-11T21:15:33.412 INFO:teuthology.orchestra.run.smithi023.stdout: "crush-failure-domain": "osd", 2024-03-11T21:15:33.412 INFO:teuthology.orchestra.run.smithi023.stdout: "k": "2", 2024-03-11T21:15:33.412 INFO:teuthology.orchestra.run.smithi023.stdout: "m": "1", 2024-03-11T21:15:33.412 INFO:teuthology.orchestra.run.smithi023.stdout: "plugin": "jerasure", 2024-03-11T21:15:33.412 INFO:teuthology.orchestra.run.smithi023.stdout: "technique": "reed_sol_van" 2024-03-11T21:15:33.412 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:15:33.412 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:33.412 INFO:teuthology.orchestra.run.smithi023.stdout: "removed_snaps_queue": [], 2024-03-11T21:15:33.412 INFO:teuthology.orchestra.run.smithi023.stdout: "new_removed_snaps": [], 2024-03-11T21:15:33.413 INFO:teuthology.orchestra.run.smithi023.stdout: "new_purged_snaps": [], 2024-03-11T21:15:33.413 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_node_flags": {}, 2024-03-11T21:15:33.413 INFO:teuthology.orchestra.run.smithi023.stdout: "device_class_flags": {}, 2024-03-11T21:15:33.413 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": { 2024-03-11T21:15:33.413 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode_enabled": false, 2024-03-11T21:15:33.413 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_bucket_count": 0, 2024-03-11T21:15:33.413 INFO:teuthology.orchestra.run.smithi023.stdout: "degraded_stretch_mode": 0, 2024-03-11T21:15:33.413 INFO:teuthology.orchestra.run.smithi023.stdout: "recovering_stretch_mode": 0, 2024-03-11T21:15:33.413 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode_bucket": 0 2024-03-11T21:15:33.413 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:15:33.413 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-11T21:15:33.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:33.414 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:15:33.415 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:15:33.462 INFO:teuthology.orchestra.run.smithi023.stdout:osd.0: {} 2024-03-11T21:15:33.484 INFO:teuthology.orchestra.run.smithi023.stdout:osd.1: {} 2024-03-11T21:15:33.506 INFO:teuthology.orchestra.run.smithi023.stdout:osd.2: {} 2024-03-11T21:15:33.516 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:15:33.524 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:33 smithi023 bash[21372]: cluster 2024-03-11T21:15:32.455162+0000 mgr.y (mgr.24380) 914 : cluster 0 pgmap v1054: 158 pgs: 1 active+recovering+remapped, 2 active+remapped+backfill_toofull, 155 active+clean; 76 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 918 op/s; 76/4230 objects misplaced (1.797%); 688 KiB/s, 49 objects/s recovering 2024-03-11T21:15:33.524 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:33 smithi023 bash[21372]: audit 2024-03-11T21:15:33.320660+0000 mon.c (mon.2) 101 : audit 0 from='client.? 172.21.15.23:0/3189732734' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-11T21:15:33.525 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:33 smithi023 bash[21372]: audit 2024-03-11T21:15:33.453205+0000 mon.a (mon.0) 1954 : audit 0 from='client.? 172.21.15.23:0/423716824' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:15:33.525 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:33 smithi023 bash[29146]: cluster 2024-03-11T21:15:32.455162+0000 mgr.y (mgr.24380) 914 : cluster 0 pgmap v1054: 158 pgs: 1 active+recovering+remapped, 2 active+remapped+backfill_toofull, 155 active+clean; 76 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 918 op/s; 76/4230 objects misplaced (1.797%); 688 KiB/s, 49 objects/s recovering 2024-03-11T21:15:33.525 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:33 smithi023 bash[29146]: audit 2024-03-11T21:15:33.320660+0000 mon.c (mon.2) 101 : audit 0 from='client.? 172.21.15.23:0/3189732734' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-11T21:15:33.525 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:33 smithi023 bash[29146]: audit 2024-03-11T21:15:33.453205+0000 mon.a (mon.0) 1954 : audit 0 from='client.? 172.21.15.23:0/423716824' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:15:33.530 INFO:teuthology.orchestra.run.smithi023.stdout:osd.3: {} 2024-03-11T21:15:33.535 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:15:33.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:33.538 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:15:33.557 INFO:teuthology.orchestra.run.smithi023.stdout:osd.4: {} 2024-03-11T21:15:33.584 INFO:teuthology.orchestra.run.smithi023.stdout:osd.5: {} 2024-03-11T21:15:33.610 INFO:teuthology.orchestra.run.smithi023.stdout:osd.6: {} 2024-03-11T21:15:33.638 INFO:teuthology.orchestra.run.smithi023.stdout:osd.7: {} 2024-03-11T21:15:33.639 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:33.640 INFO:teuthology.orchestra.run.smithi023.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:15:33.640 INFO:teuthology.orchestra.run.smithi023.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:15:33.640 INFO:teuthology.orchestra.run.smithi023.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:15:33.640 INFO:teuthology.orchestra.run.smithi023.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:15:33.640 INFO:teuthology.orchestra.run.smithi023.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:15:33.640 INFO:teuthology.orchestra.run.smithi023.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:15:33.640 INFO:teuthology.orchestra.run.smithi023.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:15:33.640 INFO:teuthology.orchestra.run.smithi023.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:15:33.667 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:33.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:33.669 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:15:33.770 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:15:33.790 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:15:33.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:33.792 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:15:33.846 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-11T21:15:33.846 INFO:teuthology.orchestra.run.smithi023.stdout: "ops": [], 2024-03-11T21:15:33.846 INFO:teuthology.orchestra.run.smithi023.stdout: "complaint_time": 30, 2024-03-11T21:15:33.846 INFO:teuthology.orchestra.run.smithi023.stdout: "num_blocked_ops": 0 2024-03-11T21:15:33.846 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-11T21:15:33.893 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:33.904 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:33 smithi146 bash[23809]: cluster 2024-03-11T21:15:32.455162+0000 mgr.y (mgr.24380) 914 : cluster 0 pgmap v1054: 158 pgs: 1 active+recovering+remapped, 2 active+remapped+backfill_toofull, 155 active+clean; 76 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 918 op/s; 76/4230 objects misplaced (1.797%); 688 KiB/s, 49 objects/s recovering 2024-03-11T21:15:33.904 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:33 smithi146 bash[23809]: audit 2024-03-11T21:15:33.320660+0000 mon.c (mon.2) 101 : audit 0 from='client.? 172.21.15.23:0/3189732734' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-11T21:15:33.904 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:33 smithi146 bash[23809]: audit 2024-03-11T21:15:33.453205+0000 mon.a (mon.0) 1954 : audit 0 from='client.? 172.21.15.23:0/423716824' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:15:33.917 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:33.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:33.922 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:15:34.023 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:15:34.048 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:15:34.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:34.051 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:15:34.151 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:15:34.175 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:15:34.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:34.178 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:15:34.279 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:15:34.298 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:15:34.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:34.300 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:15:34.400 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:15:34.423 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:15:34.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:34.425 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:15:34.526 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:15:34.539 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:34 smithi023 bash[21372]: audit 2024-03-11T21:15:34.292900+0000 mon.c (mon.2) 102 : audit 1 from='client.? 172.21.15.23:0/137810352' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:15:34.539 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:34 smithi023 bash[21372]: audit 2024-03-11T21:15:34.293182+0000 mon.a (mon.0) 1955 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:15:34.539 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:34 smithi023 bash[29146]: audit 2024-03-11T21:15:34.292900+0000 mon.c (mon.2) 102 : audit 1 from='client.? 172.21.15.23:0/137810352' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:15:34.539 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:34 smithi023 bash[29146]: audit 2024-03-11T21:15:34.293182+0000 mon.a (mon.0) 1955 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:15:34.541 INFO:teuthology.orchestra.run.smithi023.stderr:nodeep-scrub is set 2024-03-11T21:15:34.550 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:15:34.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:34.553 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:15:34.653 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:15:34.672 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:15:34.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:34.675 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:15:34.754 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:34 smithi146 bash[23809]: audit 2024-03-11T21:15:34.292900+0000 mon.c (mon.2) 102 : audit 1 from='client.? 172.21.15.23:0/137810352' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:15:34.754 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:34 smithi146 bash[23809]: audit 2024-03-11T21:15:34.293182+0000 mon.a (mon.0) 1955 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:15:34.776 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:15:34.797 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:15:34.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:34.799 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:15:34.900 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:15:34.922 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:15:34.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:34.924 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:15:35.025 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:15:35.047 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:15:35.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:35.049 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:15:35.150 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:15:35.176 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:15:35.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:35.178 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:15:35.279 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:15:35.306 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:15:35.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:35.308 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:15:35.409 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:15:35.436 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:15:35.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:35.438 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:15:35.539 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:15:35.567 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:15:35.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:35.569 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:15:35.671 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:15:35.696 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:15:35.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:35.698 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:15:35.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:35 smithi146 bash[23809]: cluster 2024-03-11T21:15:34.455741+0000 mgr.y (mgr.24380) 915 : cluster 0 pgmap v1055: 158 pgs: 1 active+recovering+remapped, 2 active+remapped+backfill_toofull, 155 active+clean; 33 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 782 op/s; 76/2154 objects misplaced (3.528%); 688 KiB/s, 92 objects/s recovering 2024-03-11T21:15:35.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:35 smithi146 bash[23809]: cluster 2024-03-11T21:15:34.530372+0000 mon.a (mon.0) 1956 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:15:35.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:35 smithi146 bash[23809]: audit 2024-03-11T21:15:34.542609+0000 mon.a (mon.0) 1957 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:15:35.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:35 smithi146 bash[23809]: cluster 2024-03-11T21:15:34.549992+0000 mon.a (mon.0) 1958 : cluster 0 osdmap e434: 8 total, 8 up, 8 in 2024-03-11T21:15:35.799 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:15:35.811 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:35 smithi023 bash[21372]: cluster 2024-03-11T21:15:34.455741+0000 mgr.y (mgr.24380) 915 : cluster 0 pgmap v1055: 158 pgs: 1 active+recovering+remapped, 2 active+remapped+backfill_toofull, 155 active+clean; 33 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 782 op/s; 76/2154 objects misplaced (3.528%); 688 KiB/s, 92 objects/s recovering 2024-03-11T21:15:35.811 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:35 smithi023 bash[21372]: cluster 2024-03-11T21:15:34.530372+0000 mon.a (mon.0) 1956 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:15:35.811 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:35 smithi023 bash[21372]: audit 2024-03-11T21:15:34.542609+0000 mon.a (mon.0) 1957 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:15:35.811 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:35 smithi023 bash[21372]: cluster 2024-03-11T21:15:34.549992+0000 mon.a (mon.0) 1958 : cluster 0 osdmap e434: 8 total, 8 up, 8 in 2024-03-11T21:15:35.811 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:35 smithi023 bash[29146]: cluster 2024-03-11T21:15:34.455741+0000 mgr.y (mgr.24380) 915 : cluster 0 pgmap v1055: 158 pgs: 1 active+recovering+remapped, 2 active+remapped+backfill_toofull, 155 active+clean; 33 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 782 op/s; 76/2154 objects misplaced (3.528%); 688 KiB/s, 92 objects/s recovering 2024-03-11T21:15:35.811 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:35 smithi023 bash[29146]: cluster 2024-03-11T21:15:34.530372+0000 mon.a (mon.0) 1956 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:15:35.811 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:35 smithi023 bash[29146]: audit 2024-03-11T21:15:34.542609+0000 mon.a (mon.0) 1957 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:15:35.811 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:35 smithi023 bash[29146]: cluster 2024-03-11T21:15:34.549992+0000 mon.a (mon.0) 1958 : cluster 0 osdmap e434: 8 total, 8 up, 8 in 2024-03-11T21:15:35.824 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:15:35.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:35.826 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:15:35.927 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:15:35.956 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:15:35.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:35.959 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:15:36.060 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:15:36.087 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:15:36.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:36.089 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:15:36.190 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:36.202 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:15:36 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:15:36.219 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:36.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:36.221 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:15:36.322 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:36.350 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:36.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:36.353 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:15:36.453 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:15:36.477 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:15:36.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:36.479 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:15:36.580 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:15:36.601 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:15:36.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:36.603 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:15:36.704 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:36.730 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:36.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:36.732 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:15:36.833 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:15:36.859 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:15:36.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:36.861 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:15:36.962 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:15:36.992 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:15:36.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:36.994 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:15:37.095 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:15:37.123 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:15:37.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:37.125 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:15:37.226 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:15:37.255 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:15:37.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:37.257 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:15:37.358 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:15:37.384 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:15:37.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:37.386 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:15:37.487 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:15:37.514 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:15:37.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:37.516 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:15:37.617 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:15:37.644 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:15:37.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:37.646 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:15:37.747 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:15:37.754 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:37 smithi146 bash[23809]: audit 2024-03-11T21:15:36.100951+0000 mgr.y (mgr.24380) 916 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:15:37.754 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:37 smithi146 bash[23809]: cluster 2024-03-11T21:15:36.457659+0000 mgr.y (mgr.24380) 917 : cluster 0 pgmap v1057: 158 pgs: 1 active+recovering+remapped, 2 active+remapped+backfill_toofull, 155 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 542 op/s; 0 B/s, 79 objects/s recovering 2024-03-11T21:15:37.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:37 smithi146 bash[23809]: audit 2024-03-11T21:15:36.585167+0000 mon.a (mon.0) 1959 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:15:37.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:37 smithi146 bash[23809]: audit 2024-03-11T21:15:36.586576+0000 mon.a (mon.0) 1960 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:15:37.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:37 smithi146 bash[23809]: cluster 2024-03-11T21:15:37.549648+0000 mon.a (mon.0) 1961 : cluster 0 osdmap e435: 8 total, 8 up, 8 in 2024-03-11T21:15:37.776 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:15:37.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:37.779 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:15:37.880 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:37.907 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:37.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:37.910 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:15:38.011 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:15:38.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:37 smithi023 bash[21372]: audit 2024-03-11T21:15:36.100951+0000 mgr.y (mgr.24380) 916 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:15:38.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:37 smithi023 bash[21372]: cluster 2024-03-11T21:15:36.457659+0000 mgr.y (mgr.24380) 917 : cluster 0 pgmap v1057: 158 pgs: 1 active+recovering+remapped, 2 active+remapped+backfill_toofull, 155 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 542 op/s; 0 B/s, 79 objects/s recovering 2024-03-11T21:15:38.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:37 smithi023 bash[21372]: audit 2024-03-11T21:15:36.585167+0000 mon.a (mon.0) 1959 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:15:38.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:37 smithi023 bash[21372]: audit 2024-03-11T21:15:36.586576+0000 mon.a (mon.0) 1960 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:15:38.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:37 smithi023 bash[21372]: cluster 2024-03-11T21:15:37.549648+0000 mon.a (mon.0) 1961 : cluster 0 osdmap e435: 8 total, 8 up, 8 in 2024-03-11T21:15:38.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:37 smithi023 bash[29146]: audit 2024-03-11T21:15:36.100951+0000 mgr.y (mgr.24380) 916 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:15:38.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:37 smithi023 bash[29146]: cluster 2024-03-11T21:15:36.457659+0000 mgr.y (mgr.24380) 917 : cluster 0 pgmap v1057: 158 pgs: 1 active+recovering+remapped, 2 active+remapped+backfill_toofull, 155 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 542 op/s; 0 B/s, 79 objects/s recovering 2024-03-11T21:15:38.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:37 smithi023 bash[29146]: audit 2024-03-11T21:15:36.585167+0000 mon.a (mon.0) 1959 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:15:38.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:37 smithi023 bash[29146]: audit 2024-03-11T21:15:36.586576+0000 mon.a (mon.0) 1960 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:15:38.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:37 smithi023 bash[29146]: cluster 2024-03-11T21:15:37.549648+0000 mon.a (mon.0) 1961 : cluster 0 osdmap e435: 8 total, 8 up, 8 in 2024-03-11T21:15:38.035 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:15:38.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:38.038 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:15:38.138 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:15:38.167 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:15:38.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:38.169 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:15:38.270 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:38.296 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:38.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:38.297 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:15:38.398 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:15:38.426 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:15:38.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:38.429 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:15:38.530 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:15:38.563 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:15:38.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:38.566 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:15:38.594 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:15:38 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:15:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:15:38.595 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-03-11T21:15:38.600 INFO:tasks.thrashosds.thrasher:j is {'epoch': 433, 'fsid': '67ba00f8-dfe7-11ee-95c8-87774f69a715', 'created': '2024-03-11T20:40:13.664802+0000', 'modified': '2024-03-11T21:15:09.519848+0000', 'last_up_change': '2024-03-11T21:00:47.324817+0000', 'last_in_change': '2024-03-11T21:14:49.940617+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 20, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 11, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-11T20:45:31.964570+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 11, 'pg_placement_num': 11, 'pg_placement_num_target': 11, 'pg_num_target': 11, 'pg_num_pending': 11, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '369', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '367', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6399999856948853, 'score_stable': 1.6399999856948853, 'optimal_score': 1.3300000429153442, 'raw_score_acting': 2.180000066757202, 'raw_score_stable': 2.180000066757202, 'primary_affinity_weighted': 0.75, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.75}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2024-03-11T20:50:26.421486+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '78', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1.3300000429153442, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.75, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.75}}, {'pool': 3, 'pool_name': 'default.rgw.log', 'create_time': '2024-03-11T20:50:28.592615+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1.3300000429153442, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.75, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.75}}, {'pool': 4, 'pool_name': 'default.rgw.control', 'create_time': '2024-03-11T20:50:30.536371+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1.3300000429153442, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 0.75, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.75}}, {'pool': 5, 'pool_name': 'datapool', 'create_time': '2024-03-11T20:50:31.135845+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 87, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3.990000009536743, 'score_stable': 3.990000009536743, 'optimal_score': 1.1699999570846558, 'raw_score_acting': 4.670000076293945, 'raw_score_stable': 4.670000076293945, 'primary_affinity_weighted': 0.75, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.75}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-03-11T20:50:33.233163+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.690000057220459, 'score_stable': 1.690000057220459, 'optimal_score': 1.3300000429153442, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.75, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.75}}, {'pool': 11, 'pool_name': 'unique_pool_4', 'create_time': '2024-03-11T21:11:46.486253+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '386', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1.3300000429153442, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 0.75, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.75}}], 'osds': [{'osd': 0, 'uuid': '3df72b4a-8fe4-4265-b4c6-af022e1e23e1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 423, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6802', 'nonce': 2154627393}, {'type': 'v1', 'addr': '172.21.15.23:6803', 'nonce': 2154627393}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6804', 'nonce': 2154627393}, {'type': 'v1', 'addr': '172.21.15.23:6805', 'nonce': 2154627393}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6808', 'nonce': 2154627393}, {'type': 'v1', 'addr': '172.21.15.23:6809', 'nonce': 2154627393}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6806', 'nonce': 2154627393}, {'type': 'v1', 'addr': '172.21.15.23:6807', 'nonce': 2154627393}]}, 'public_addr': '172.21.15.23:6803/2154627393', 'cluster_addr': '172.21.15.23:6805/2154627393', 'heartbeat_back_addr': '172.21.15.23:6809/2154627393', 'heartbeat_front_addr': '172.21.15.23:6807/2154627393', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'e8fd7500-9081-41a1-8077-a4d0f0e12a2c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 13, 'last_clean_end': 173, 'up_from': 211, 'up_thru': 424, 'down_at': 174, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6810', 'nonce': 56585354}, {'type': 'v1', 'addr': '172.21.15.23:6811', 'nonce': 56585354}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6812', 'nonce': 56585354}, {'type': 'v1', 'addr': '172.21.15.23:6813', 'nonce': 56585354}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6816', 'nonce': 56585354}, {'type': 'v1', 'addr': '172.21.15.23:6817', 'nonce': 56585354}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6814', 'nonce': 56585354}, {'type': 'v1', 'addr': '172.21.15.23:6815', 'nonce': 56585354}]}, 'public_addr': '172.21.15.23:6811/56585354', 'cluster_addr': '172.21.15.23:6813/56585354', 'heartbeat_back_addr': '172.21.15.23:6817/56585354', 'heartbeat_front_addr': '172.21.15.23:6815/56585354', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'dd85d52a-df48-4b03-8612-cecfd82b5987', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 19, 'up_thru': 306, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6818', 'nonce': 1770748056}, {'type': 'v1', 'addr': '172.21.15.23:6819', 'nonce': 1770748056}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6820', 'nonce': 1770748056}, {'type': 'v1', 'addr': '172.21.15.23:6821', 'nonce': 1770748056}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6824', 'nonce': 1770748056}, {'type': 'v1', 'addr': '172.21.15.23:6825', 'nonce': 1770748056}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6822', 'nonce': 1770748056}, {'type': 'v1', 'addr': '172.21.15.23:6823', 'nonce': 1770748056}]}, 'public_addr': '172.21.15.23:6819/1770748056', 'cluster_addr': '172.21.15.23:6821/1770748056', 'heartbeat_back_addr': '172.21.15.23:6825/1770748056', 'heartbeat_front_addr': '172.21.15.23:6823/1770748056', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'e3021106-3779-4b94-9bbf-a7724544110f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 28, 'up_thru': 432, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6826', 'nonce': 938496814}, {'type': 'v1', 'addr': '172.21.15.23:6827', 'nonce': 938496814}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6828', 'nonce': 938496814}, {'type': 'v1', 'addr': '172.21.15.23:6829', 'nonce': 938496814}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6832', 'nonce': 938496814}, {'type': 'v1', 'addr': '172.21.15.23:6833', 'nonce': 938496814}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6830', 'nonce': 938496814}, {'type': 'v1', 'addr': '172.21.15.23:6831', 'nonce': 938496814}]}, 'public_addr': '172.21.15.23:6827/938496814', 'cluster_addr': '172.21.15.23:6829/938496814', 'heartbeat_back_addr': '172.21.15.23:6833/938496814', 'heartbeat_front_addr': '172.21.15.23:6831/938496814', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '4dcbc91e-7035-413d-97bc-c9d804eb16d7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 35, 'last_clean_end': 182, 'up_from': 204, 'up_thru': 248, 'down_at': 183, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6800', 'nonce': 3858359434}, {'type': 'v1', 'addr': '172.21.15.146:6801', 'nonce': 3858359434}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6802', 'nonce': 3858359434}, {'type': 'v1', 'addr': '172.21.15.146:6803', 'nonce': 3858359434}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6806', 'nonce': 3858359434}, {'type': 'v1', 'addr': '172.21.15.146:6807', 'nonce': 3858359434}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6804', 'nonce': 3858359434}, {'type': 'v1', 'addr': '172.21.15.146:6805', 'nonce': 3858359434}]}, 'public_addr': '172.21.15.146:6801/3858359434', 'cluster_addr': '172.21.15.146:6803/3858359434', 'heartbeat_back_addr': '172.21.15.146:6807/3858359434', 'heartbeat_front_addr': '172.21.15.146:6805/3858359434', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'aefcb766-8bae-4ff4-b71e-6691c8ac02f1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 44, 'up_thru': 426, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6808', 'nonce': 4003230814}, {'type': 'v1', 'addr': '172.21.15.146:6809', 'nonce': 4003230814}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6810', 'nonce': 4003230814}, {'type': 'v1', 'addr': '172.21.15.146:6811', 'nonce': 4003230814}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6814', 'nonce': 4003230814}, {'type': 'v1', 'addr': '172.21.15.146:6815', 'nonce': 4003230814}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6812', 'nonce': 4003230814}, {'type': 'v1', 'addr': '172.21.15.146:6813', 'nonce': 4003230814}]}, 'public_addr': '172.21.15.146:6809/4003230814', 'cluster_addr': '172.21.15.146:6811/4003230814', 'heartbeat_back_addr': '172.21.15.146:6815/4003230814', 'heartbeat_front_addr': '172.21.15.146:6813/4003230814', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '5cb151e8-1f9b-4617-8a86-adfb06bb9606', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 57, 'up_thru': 426, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6816', 'nonce': 3334092376}, {'type': 'v1', 'addr': '172.21.15.146:6817', 'nonce': 3334092376}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6818', 'nonce': 3334092376}, {'type': 'v1', 'addr': '172.21.15.146:6819', 'nonce': 3334092376}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6822', 'nonce': 3334092376}, {'type': 'v1', 'addr': '172.21.15.146:6823', 'nonce': 3334092376}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6820', 'nonce': 3334092376}, {'type': 'v1', 'addr': '172.21.15.146:6821', 'nonce': 3334092376}]}, 'public_addr': '172.21.15.146:6817/3334092376', 'cluster_addr': '172.21.15.146:6819/3334092376', 'heartbeat_back_addr': '172.21.15.146:6823/3334092376', 'heartbeat_front_addr': '172.21.15.146:6821/3334092376', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'bdbd11a2-fa09-4f82-93a2-6b2d9c750bfd', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 68, 'up_thru': 426, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6824', 'nonce': 1678407923}, {'type': 'v1', 'addr': '172.21.15.146:6825', 'nonce': 1678407923}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6826', 'nonce': 1678407923}, {'type': 'v1', 'addr': '172.21.15.146:6827', 'nonce': 1678407923}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6830', 'nonce': 1678407923}, {'type': 'v1', 'addr': '172.21.15.146:6831', 'nonce': 1678407923}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6828', 'nonce': 1678407923}, {'type': 'v1', 'addr': '172.21.15.146:6829', 'nonce': 1678407923}]}, 'public_addr': '172.21.15.146:6825/1678407923', 'cluster_addr': '172.21.15.146:6827/1678407923', 'heartbeat_back_addr': '172.21.15.146:6831/1678407923', 'heartbeat_front_addr': '172.21.15.146:6829/1678407923', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T21:14:34.387211+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T21:15:00.009812+0000', 'dead_epoch': 173}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T21:14:59.261591+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T21:15:07.705552+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T21:14:43.253151+0000', 'dead_epoch': 182}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T21:14:58.803952+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T21:15:03.074602+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T21:14:29.226970+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '2.0', 'mappings': [{'from': 7, 'to': 2}]}, {'pgid': '4.b', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.d', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.e', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.16', 'mappings': [{'from': 3, 'to': 2}]}, {'pgid': '4.1d', 'mappings': [{'from': 4, 'to': 2}]}, {'pgid': '6.a', 'mappings': [{'from': 6, 'to': 1}]}], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '11.1', 'osds': [6, 3, 2]}, {'pgid': '11.2', 'osds': [3, 2, 7]}, {'pgid': '11.6', 'osds': [1, 2, 4]}], 'primary_temp': [], 'blocklist': {'172.21.15.23:0/1618334186': '2024-03-12T20:40:45.361537+0000', '172.21.15.23:6801/2813578081': '2024-03-12T20:40:45.361537+0000', '172.21.15.23:0/2869940180': '2024-03-12T20:51:33.790205+0000', '172.21.15.23:6800/3037564930': '2024-03-12T20:51:33.790205+0000', '172.21.15.23:0/3297567084': '2024-03-12T20:51:33.790205+0000', '172.21.15.23:0/3708753659': '2024-03-12T20:40:45.361537+0000', '172.21.15.23:0/2004146740': '2024-03-12T20:40:45.361537+0000', '172.21.15.23:0/827653812': '2024-03-12T20:41:21.301516+0000', '172.21.15.23:6800/1057699850': '2024-03-12T20:41:21.301516+0000', '172.21.15.23:6800/2813578081': '2024-03-12T20:40:45.361537+0000', '172.21.15.23:6801/1057699850': '2024-03-12T20:41:21.301516+0000', '172.21.15.23:0/3833230375': '2024-03-12T20:41:21.301516+0000', '172.21.15.23:6801/3037564930': '2024-03-12T20:51:33.790205+0000', '172.21.15.23:0/1182809102': '2024-03-12T20:51:33.790205+0000', '172.21.15.23:0/2122134187': '2024-03-12T20:51:33.790205+0000', '172.21.15.23:0/2356701899': '2024-03-12T20:41:21.301516+0000', '172.21.15.23:0/843974100': '2024-03-12T20:51:33.790205+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-03-11T21:15:38.600 INFO:tasks.thrashosds.thrasher:No pg_upmap entries; doing nothing 2024-03-11T21:15:38.667 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:15:38.693 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:15:38.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:38.695 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:15:38.796 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:15:38.817 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:15:38.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:38.819 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:15:38.920 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:15:38.948 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:15:38.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:38.951 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:15:39.052 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:39.081 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:39.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:39.083 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:15:39.184 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:15:39.212 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:15:39.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:39.214 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:15:39.315 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:15:39.344 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:15:39.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:39.346 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:15:39.447 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:15:39.476 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:15:39.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:39.478 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:15:39.579 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:15:39.606 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:15:39.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:39.609 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:15:39.709 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:15:39.722 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:39 smithi146 bash[23809]: cluster 2024-03-11T21:15:38.458302+0000 mgr.y (mgr.24380) 918 : cluster 0 pgmap v1059: 158 pgs: 1 active+recovering+remapped, 2 active+remapped+backfill_toofull, 155 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 439 op/s; 0 B/s, 63 objects/s recovering 2024-03-11T21:15:39.722 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:39 smithi146 bash[23809]: cluster 2024-03-11T21:15:38.550502+0000 mon.a (mon.0) 1962 : cluster 0 osdmap e436: 8 total, 8 up, 8 in 2024-03-11T21:15:39.739 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:15:39.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:39.741 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:15:39.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:39 smithi023 bash[21372]: cluster 2024-03-11T21:15:38.458302+0000 mgr.y (mgr.24380) 918 : cluster 0 pgmap v1059: 158 pgs: 1 active+recovering+remapped, 2 active+remapped+backfill_toofull, 155 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 439 op/s; 0 B/s, 63 objects/s recovering 2024-03-11T21:15:39.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:39 smithi023 bash[21372]: cluster 2024-03-11T21:15:38.550502+0000 mon.a (mon.0) 1962 : cluster 0 osdmap e436: 8 total, 8 up, 8 in 2024-03-11T21:15:39.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:39 smithi023 bash[29146]: cluster 2024-03-11T21:15:38.458302+0000 mgr.y (mgr.24380) 918 : cluster 0 pgmap v1059: 158 pgs: 1 active+recovering+remapped, 2 active+remapped+backfill_toofull, 155 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 439 op/s; 0 B/s, 63 objects/s recovering 2024-03-11T21:15:39.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:39 smithi023 bash[29146]: cluster 2024-03-11T21:15:38.550502+0000 mon.a (mon.0) 1962 : cluster 0 osdmap e436: 8 total, 8 up, 8 in 2024-03-11T21:15:39.842 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:15:39.869 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:15:39.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:39.871 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:15:39.972 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:15:40.003 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:15:40.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:40.005 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:15:40.106 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:15:40.133 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:15:40.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:40.136 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:15:40.237 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:15:40.266 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:15:40.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:40.269 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:15:40.370 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:15:40.399 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:15:40.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:40.401 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:15:40.502 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:15:40.532 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:15:40.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:40.535 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:15:40.565 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T21:15:40.608 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd unset noscrub 2024-03-11T21:15:40.636 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:40.662 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:40.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:40.665 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:15:40.766 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:15:40.787 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:15:40.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:40.790 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:15:40.890 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:15:40.917 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:15:40.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:40.919 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:15:41.020 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:15:41.046 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:15:41.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:41.048 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:15:41.083 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:15:41.149 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:15:41.176 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:15:41.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:41.179 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:15:41.279 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:15:41.307 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:15:41.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:41.309 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:15:41.410 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:15:41.439 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:15:41.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:41.441 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:15:41.542 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:15:41.569 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:15:41.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:41.571 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:15:41.672 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:15:41.699 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:15:41.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:41.701 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:15:41.802 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:41.813 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:41 smithi146 bash[23809]: cluster 2024-03-11T21:15:40.459149+0000 mgr.y (mgr.24380) 919 : cluster 0 pgmap v1061: 158 pgs: 1 active+recovering+remapped, 2 active+remapped+backfill_toofull, 155 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 87 op/s 2024-03-11T21:15:41.829 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:41.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:41.831 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:15:41.932 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:15:41.944 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:41 smithi023 bash[21372]: cluster 2024-03-11T21:15:40.459149+0000 mgr.y (mgr.24380) 919 : cluster 0 pgmap v1061: 158 pgs: 1 active+recovering+remapped, 2 active+remapped+backfill_toofull, 155 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 87 op/s 2024-03-11T21:15:41.944 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:41 smithi023 bash[29146]: cluster 2024-03-11T21:15:40.459149+0000 mgr.y (mgr.24380) 919 : cluster 0 pgmap v1061: 158 pgs: 1 active+recovering+remapped, 2 active+remapped+backfill_toofull, 155 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 87 op/s 2024-03-11T21:15:41.958 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:15:41.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:41.961 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:15:42.062 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:15:42.089 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:15:42.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:42.092 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:15:42.192 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:15:42.219 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:15:42.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:42.222 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:15:42.322 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:15:42.350 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:15:42.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:42.353 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:15:42.453 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:42.482 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:42.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:42.484 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:15:42.585 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:15:42.608 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:15:42.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:42.611 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:15:42.711 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:42.740 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:42.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:42.742 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:15:42.843 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:15:42.870 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:15:42.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:42.872 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:15:42.973 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:15:43.001 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:15:43.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:43.003 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:15:43.104 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:15:43.124 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:15:43.130 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:15:43.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:43.132 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:15:43.233 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:15:43.260 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:15:43.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:43.262 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:15:43.363 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:15:43.392 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:15:43.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:43.395 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:15:43.496 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:15:43.523 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:15:43.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:43.526 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:15:43.602 INFO:tasks.thrashosds.thrasher:in_osds: [2, 3, 4, 5, 6, 7, 1, 0] out_osds: [] dead_osds: [] live_osds: [0, 2, 3, 5, 6, 7, 4, 1] 2024-03-11T21:15:43.602 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-11T21:15:43.626 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:15:43.654 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:15:43.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:43.656 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:15:43.758 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:15:43.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:43 smithi146 bash[23809]: cluster 2024-03-11T21:15:42.460813+0000 mgr.y (mgr.24380) 920 : cluster 0 pgmap v1062: 158 pgs: 2 active+remapped+backfill_toofull, 156 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 66 op/s; 0 B/s, 95 objects/s recovering 2024-03-11T21:15:43.786 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:15:43.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:43.788 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:15:43.889 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:15:43.916 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:15:43.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:43.919 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:15:44.020 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:15:44.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:43 smithi023 bash[21372]: cluster 2024-03-11T21:15:42.460813+0000 mgr.y (mgr.24380) 920 : cluster 0 pgmap v1062: 158 pgs: 2 active+remapped+backfill_toofull, 156 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 66 op/s; 0 B/s, 95 objects/s recovering 2024-03-11T21:15:44.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:43 smithi023 bash[29146]: cluster 2024-03-11T21:15:42.460813+0000 mgr.y (mgr.24380) 920 : cluster 0 pgmap v1062: 158 pgs: 2 active+remapped+backfill_toofull, 156 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 66 op/s; 0 B/s, 95 objects/s recovering 2024-03-11T21:15:44.049 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:15:44.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:44.051 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:15:44.152 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:15:44.173 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:15:44.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:44.175 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:15:44.276 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:15:44.303 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:15:44.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:44.305 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:15:44.406 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:15:44.432 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:15:44.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:44.435 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:15:44.536 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:15:44.564 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:15:44.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:44.566 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:15:44.666 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:15:44.695 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:15:44.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:44.698 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:15:44.799 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:15:44.827 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:15:44.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:44.830 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:15:44.930 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:15:44.958 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:15:44.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:44.960 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:15:45.061 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:15:45.088 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:15:45.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:45.090 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:15:45.169 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:15:45.170 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:15:45.191 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:15:45.219 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:15:45.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:45.221 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:15:45.322 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:15:45.350 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:15:45.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:45.353 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:15:45.454 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:15:45.480 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:15:45.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:45.482 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:15:45.582 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:15:45.611 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:15:45.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:45.614 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:15:45.714 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:15:45.742 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:15:45.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:45.744 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:15:45.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:45 smithi023 bash[21372]: cluster 2024-03-11T21:15:44.461971+0000 mgr.y (mgr.24380) 921 : cluster 0 pgmap v1063: 158 pgs: 2 active+remapped+backfill_toofull, 156 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 0 B/s, 94 objects/s recovering 2024-03-11T21:15:45.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:45 smithi023 bash[29146]: cluster 2024-03-11T21:15:44.461971+0000 mgr.y (mgr.24380) 921 : cluster 0 pgmap v1063: 158 pgs: 2 active+remapped+backfill_toofull, 156 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 0 B/s, 94 objects/s recovering 2024-03-11T21:15:45.845 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:15:45.872 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:15:45.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:45.874 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:15:45.975 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:15:46.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:45 smithi146 bash[23809]: cluster 2024-03-11T21:15:44.461971+0000 mgr.y (mgr.24380) 921 : cluster 0 pgmap v1063: 158 pgs: 2 active+remapped+backfill_toofull, 156 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 0 B/s, 94 objects/s recovering 2024-03-11T21:15:46.005 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:15:46.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:46.007 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:15:46.108 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:15:46.135 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:15:46.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:46.137 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:15:46.238 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:15:46.263 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:15:46.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:46.266 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:15:46.367 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:15:46.378 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:15:46 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:15:46.395 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:15:46.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:46.397 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:15:46.498 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:15:46.524 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:15:46.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:46.527 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:15:46.627 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:15:46.655 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:15:46.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:46.658 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:15:46.759 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:15:46.780 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:15:46.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:46.782 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:15:46.883 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:15:46.905 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:15:46.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:46.907 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:15:47.008 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:15:47.035 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:15:47.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:47.038 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:15:47.139 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:47.166 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:47.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:47.169 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:15:47.269 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:15:47.297 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:15:47.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:47.299 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:15:47.400 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:47.426 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:47.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:47.429 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:15:47.529 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:15:47.558 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:15:47.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:47.561 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:15:47.661 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:15:47.688 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:15:47.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:47.690 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:15:47.791 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:15:47.803 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:47 smithi023 bash[29146]: audit 2024-03-11T21:15:46.108602+0000 mgr.y (mgr.24380) 922 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:15:47.803 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:47 smithi023 bash[29146]: cluster 2024-03-11T21:15:46.463729+0000 mgr.y (mgr.24380) 923 : cluster 0 pgmap v1064: 158 pgs: 2 active+remapped+backfill_toofull, 156 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 0 B/s, 84 objects/s recovering 2024-03-11T21:15:47.803 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:47 smithi023 bash[21372]: audit 2024-03-11T21:15:46.108602+0000 mgr.y (mgr.24380) 922 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:15:47.803 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:47 smithi023 bash[21372]: cluster 2024-03-11T21:15:46.463729+0000 mgr.y (mgr.24380) 923 : cluster 0 pgmap v1064: 158 pgs: 2 active+remapped+backfill_toofull, 156 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 0 B/s, 84 objects/s recovering 2024-03-11T21:15:47.821 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:15:47.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:47.823 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:15:47.923 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:15:47.935 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:47 smithi146 bash[23809]: audit 2024-03-11T21:15:46.108602+0000 mgr.y (mgr.24380) 922 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:15:47.935 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:47 smithi146 bash[23809]: cluster 2024-03-11T21:15:46.463729+0000 mgr.y (mgr.24380) 923 : cluster 0 pgmap v1064: 158 pgs: 2 active+remapped+backfill_toofull, 156 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 0 B/s, 84 objects/s recovering 2024-03-11T21:15:47.952 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:15:47.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:47.954 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:15:48.055 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:15:48.085 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:15:48.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:48.087 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:15:48.188 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:15:48.215 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:15:48.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:48.218 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:15:48.319 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:48.347 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:48.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:48.349 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:15:48.450 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:15:48.480 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:15:48.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:48.482 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:15:48.525 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:15:48 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:15:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:15:48.583 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:15:48.611 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:15:48.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:48.613 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:15:48.713 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:15:48.740 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:15:48.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:48.742 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:15:48.843 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:15:48.865 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:15:48.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:48.867 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:15:48.968 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:15:48.995 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:15:48.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:48.997 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:15:49.098 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:15:49.122 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:15:49.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:49.124 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:15:49.225 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:15:49.251 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:15:49.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:49.254 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:15:49.354 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:15:49.382 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:15:49.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:49.385 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:15:49.486 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:15:49.510 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:15:49.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:49.511 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:15:49.601 INFO:teuthology.orchestra.run.smithi023.stderr:pool 'unique_pool_4' removed 2024-03-11T21:15:49.613 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:15:49.641 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:15:49.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:49.646 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:15:49.747 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:49.769 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:49.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:49.772 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:15:49.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:49 smithi023 bash[21372]: cluster 2024-03-11T21:15:48.464701+0000 mgr.y (mgr.24380) 924 : cluster 0 pgmap v1065: 158 pgs: 2 active+remapped+backfill_toofull, 156 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 0 B/s, 75 objects/s recovering 2024-03-11T21:15:49.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:49 smithi023 bash[21372]: audit 2024-03-11T21:15:49.008235+0000 mon.a (mon.0) 1963 : audit 1 from='client.? 172.21.15.23:0/1728225833' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T21:15:49.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:49 smithi023 bash[29146]: cluster 2024-03-11T21:15:48.464701+0000 mgr.y (mgr.24380) 924 : cluster 0 pgmap v1065: 158 pgs: 2 active+remapped+backfill_toofull, 156 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 0 B/s, 75 objects/s recovering 2024-03-11T21:15:49.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:49 smithi023 bash[29146]: audit 2024-03-11T21:15:49.008235+0000 mon.a (mon.0) 1963 : audit 1 from='client.? 172.21.15.23:0/1728225833' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T21:15:49.874 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:15:49.895 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:15:49.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:49.897 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:15:49.998 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:50.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:49 smithi146 bash[23809]: cluster 2024-03-11T21:15:48.464701+0000 mgr.y (mgr.24380) 924 : cluster 0 pgmap v1065: 158 pgs: 2 active+remapped+backfill_toofull, 156 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 0 B/s, 75 objects/s recovering 2024-03-11T21:15:50.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:49 smithi146 bash[23809]: audit 2024-03-11T21:15:49.008235+0000 mon.a (mon.0) 1963 : audit 1 from='client.? 172.21.15.23:0/1728225833' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T21:15:50.023 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:50.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:50.025 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:15:50.126 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:15:50.152 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:15:50.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:50.154 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:15:50.255 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:50.283 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:50.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:50.285 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:15:50.386 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:15:50.415 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:15:50.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:50.418 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:15:50.519 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:15:50.547 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:15:50.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:50.549 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:15:50.651 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:15:50.672 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:15:50.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:50.674 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:15:50.751 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:50 smithi023 bash[21372]: cluster 2024-03-11T21:15:49.305572+0000 osd.4 (osd.4) 119 : cluster 0 purged_snaps scrub starts 2024-03-11T21:15:50.751 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:50 smithi023 bash[21372]: cluster 2024-03-11T21:15:49.305679+0000 osd.4 (osd.4) 120 : cluster 0 purged_snaps scrub ok 2024-03-11T21:15:50.751 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:50 smithi023 bash[21372]: audit 2024-03-11T21:15:49.610942+0000 mon.a (mon.0) 1964 : audit 1 from='client.? 172.21.15.23:0/1728225833' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-03-11T21:15:50.751 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:50 smithi023 bash[21372]: cluster 2024-03-11T21:15:49.618779+0000 mon.a (mon.0) 1965 : cluster 0 osdmap e437: 8 total, 8 up, 8 in 2024-03-11T21:15:50.751 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:50 smithi023 bash[29146]: cluster 2024-03-11T21:15:49.305572+0000 osd.4 (osd.4) 119 : cluster 0 purged_snaps scrub starts 2024-03-11T21:15:50.751 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:50 smithi023 bash[29146]: cluster 2024-03-11T21:15:49.305679+0000 osd.4 (osd.4) 120 : cluster 0 purged_snaps scrub ok 2024-03-11T21:15:50.751 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:50 smithi023 bash[29146]: audit 2024-03-11T21:15:49.610942+0000 mon.a (mon.0) 1964 : audit 1 from='client.? 172.21.15.23:0/1728225833' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-03-11T21:15:50.751 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:50 smithi023 bash[29146]: cluster 2024-03-11T21:15:49.618779+0000 mon.a (mon.0) 1965 : cluster 0 osdmap e437: 8 total, 8 up, 8 in 2024-03-11T21:15:50.775 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:15:50.803 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:15:50.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:50.805 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:15:50.906 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:50.917 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:50 smithi146 bash[23809]: cluster 2024-03-11T21:15:49.305572+0000 osd.4 (osd.4) 119 : cluster 0 purged_snaps scrub starts 2024-03-11T21:15:50.917 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:50 smithi146 bash[23809]: cluster 2024-03-11T21:15:49.305679+0000 osd.4 (osd.4) 120 : cluster 0 purged_snaps scrub ok 2024-03-11T21:15:50.917 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:50 smithi146 bash[23809]: audit 2024-03-11T21:15:49.610942+0000 mon.a (mon.0) 1964 : audit 1 from='client.? 172.21.15.23:0/1728225833' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-03-11T21:15:50.917 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:50 smithi146 bash[23809]: cluster 2024-03-11T21:15:49.618779+0000 mon.a (mon.0) 1965 : cluster 0 osdmap e437: 8 total, 8 up, 8 in 2024-03-11T21:15:50.933 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:50.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:50.935 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:15:51.036 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:15:51.064 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:15:51.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:51.066 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:15:51.167 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:15:51.194 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:15:51.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:51.196 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:15:51.297 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:51.324 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:51.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:51.326 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:15:51.427 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:51.452 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:51.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:51.454 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:15:51.555 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:15:51.583 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:15:51.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:51.585 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:15:51.686 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:15:51.715 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:15:51.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:51.717 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:15:51.754 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:51 smithi146 bash[23809]: cluster 2024-03-11T21:15:50.465252+0000 mgr.y (mgr.24380) 925 : cluster 0 pgmap v1067: 142 pgs: 142 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-03-11T21:15:51.754 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:51 smithi146 bash[23809]: cluster 2024-03-11T21:15:50.632682+0000 mon.a (mon.0) 1966 : cluster 1 Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull) 2024-03-11T21:15:51.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:51 smithi146 bash[23809]: audit 2024-03-11T21:15:51.623578+0000 mon.a (mon.0) 1967 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:15:51.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:51 smithi146 bash[23809]: audit 2024-03-11T21:15:51.624787+0000 mon.a (mon.0) 1968 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:15:51.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:51 smithi023 bash[21372]: cluster 2024-03-11T21:15:50.465252+0000 mgr.y (mgr.24380) 925 : cluster 0 pgmap v1067: 142 pgs: 142 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-03-11T21:15:51.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:51 smithi023 bash[21372]: cluster 2024-03-11T21:15:50.632682+0000 mon.a (mon.0) 1966 : cluster 1 Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull) 2024-03-11T21:15:51.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:51 smithi023 bash[21372]: audit 2024-03-11T21:15:51.623578+0000 mon.a (mon.0) 1967 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:15:51.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:51 smithi023 bash[21372]: audit 2024-03-11T21:15:51.624787+0000 mon.a (mon.0) 1968 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:15:51.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:51 smithi023 bash[29146]: cluster 2024-03-11T21:15:50.465252+0000 mgr.y (mgr.24380) 925 : cluster 0 pgmap v1067: 142 pgs: 142 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-03-11T21:15:51.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:51 smithi023 bash[29146]: cluster 2024-03-11T21:15:50.632682+0000 mon.a (mon.0) 1966 : cluster 1 Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull) 2024-03-11T21:15:51.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:51 smithi023 bash[29146]: audit 2024-03-11T21:15:51.623578+0000 mon.a (mon.0) 1967 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:15:51.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:51 smithi023 bash[29146]: audit 2024-03-11T21:15:51.624787+0000 mon.a (mon.0) 1968 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:15:51.818 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:51.847 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:51.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:51.849 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:15:51.950 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:15:51.977 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:15:51.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:51.980 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:15:52.081 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:15:52.109 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:15:52.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:52.111 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:15:52.212 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:15:52.240 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:15:52.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:52.242 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:15:52.343 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:15:52.372 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:15:52.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:52.374 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:15:52.475 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:52.502 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:15:52.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:52.504 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:15:52.605 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:15:52.635 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:15:52.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:52.638 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:15:52.739 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:15:52.768 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:15:52.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:52.770 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:15:52.871 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:15:52.898 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:15:52.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:52.900 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:15:53.001 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:15:53.028 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:15:53.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:53.030 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:15:53.131 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:15:53.159 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:15:53.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:53.162 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:15:53.262 DEBUG:teuthology.orchestra.run.smithi146:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:15:53.291 INFO:teuthology.orchestra.run.smithi146.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:15:53.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:53.293 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi146 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:15:53.394 DEBUG:teuthology.orchestra.run.smithi023:> sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:15:53.422 INFO:teuthology.orchestra.run.smithi023.stderr:Error response from daemon: Cannot kill container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:15:53.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:15:53.424 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi023 with status 1: 'sudo docker kill -s 1 ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:15:53.481 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-03-11T21:15:53.491 INFO:tasks.thrashosds:joining thrashosds 2024-03-11T21:15:53.491 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-03-11T21:15:53.491 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T21:15:53.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:53 smithi023 bash[21372]: cluster 2024-03-11T21:15:52.466059+0000 mgr.y (mgr.24380) 926 : cluster 0 pgmap v1068: 142 pgs: 142 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-03-11T21:15:53.669 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:53 smithi023 bash[29146]: cluster 2024-03-11T21:15:52.466059+0000 mgr.y (mgr.24380) 926 : cluster 0 pgmap v1068: 142 pgs: 142 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-03-11T21:15:54.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:53 smithi146 bash[23809]: cluster 2024-03-11T21:15:52.466059+0000 mgr.y (mgr.24380) 926 : cluster 0 pgmap v1068: 142 pgs: 142 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-03-11T21:15:54.351 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-11T21:15:54.351 INFO:teuthology.orchestra.run.smithi023.stdout: "size": 20, 2024-03-11T21:15:54.351 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 600, 2024-03-11T21:15:54.351 INFO:teuthology.orchestra.run.smithi023.stdout: "ops": [ 2024-03-11T21:15:54.351 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.352 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_op(client.15291.0:36114 10.5 10:afeeceab:::benchmark_data_smithi023_124944_object36113:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e316)", 2024-03-11T21:15:54.352 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T21:07:56.357102+0000", 2024-03-11T21:15:54.352 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 478.00525257800001, 2024-03-11T21:15:54.352 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.29702194500000001, 2024-03-11T21:15:54.352 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:15:54.352 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:15:54.352 INFO:teuthology.orchestra.run.smithi023.stdout: "client_info": { 2024-03-11T21:15:54.352 INFO:teuthology.orchestra.run.smithi023.stdout: "client": "client.15291", 2024-03-11T21:15:54.352 INFO:teuthology.orchestra.run.smithi023.stdout: "client_addr": "172.21.15.23:0/729656300", 2024-03-11T21:15:54.352 INFO:teuthology.orchestra.run.smithi023.stdout: "tid": 36114 2024-03-11T21:15:54.352 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.352 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:15:54.352 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.353 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:15:54.353 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:07:56.357102+0000", 2024-03-11T21:15:54.353 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:15:54.353 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.353 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.353 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:15:54.353 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:07:56.357102+0000", 2024-03-11T21:15:54.353 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:15:54.353 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.353 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.353 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:15:54.353 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:07:56.357103+0000", 2024-03-11T21:15:54.353 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.2020000000000001e-06 2024-03-11T21:15:54.353 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.354 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.354 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:15:54.354 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:07:56.357123+0000", 2024-03-11T21:15:54.354 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.9905e-05 2024-03-11T21:15:54.354 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.354 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.354 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:15:54.354 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:07:56.357133+0000", 2024-03-11T21:15:54.354 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 9.9790000000000005e-06 2024-03-11T21:15:54.355 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.355 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.355 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:15:54.355 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:07:56.357138+0000", 2024-03-11T21:15:54.355 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 5.0710000000000002e-06 2024-03-11T21:15:54.355 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.355 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.355 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:15:54.355 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:07:56.357183+0000", 2024-03-11T21:15:54.355 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 4.4190999999999998e-05 2024-03-11T21:15:54.355 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.355 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.355 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:15:54.355 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:07:56.357286+0000", 2024-03-11T21:15:54.356 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000103942 2024-03-11T21:15:54.356 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.356 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.356 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for subops from 0,3", 2024-03-11T21:15:54.356 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:07:56.357394+0000", 2024-03-11T21:15:54.356 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000107849 2024-03-11T21:15:54.356 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.356 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.356 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "op_commit", 2024-03-11T21:15:54.356 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:07:56.358570+0000", 2024-03-11T21:15:54.356 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.001175253 2024-03-11T21:15:54.356 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.356 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.356 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:15:54.357 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:07:56.360099+0000", 2024-03-11T21:15:54.357 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0015296789999999999 2024-03-11T21:15:54.357 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.357 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.357 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:15:54.357 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:07:56.653554+0000", 2024-03-11T21:15:54.357 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.29345461299999998 2024-03-11T21:15:54.357 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.357 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.357 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T21:15:54.357 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:07:56.653974+0000", 2024-03-11T21:15:54.357 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00042009700000000003 2024-03-11T21:15:54.357 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.357 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.358 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:15:54.358 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:07:56.654124+0000", 2024-03-11T21:15:54.358 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00015016399999999999 2024-03-11T21:15:54.358 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:15:54.358 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:15:54.358 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:15:54.358 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.358 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.358 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_op(client.15291.0:36121 10.5 10:a834e12d:::benchmark_data_smithi023_124944_object36120:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e316)", 2024-03-11T21:15:54.358 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T21:07:56.362220+0000", 2024-03-11T21:15:54.359 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 478.00013448800001, 2024-03-11T21:15:54.359 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.29291954100000001, 2024-03-11T21:15:54.359 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:15:54.359 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:15:54.359 INFO:teuthology.orchestra.run.smithi023.stdout: "client_info": { 2024-03-11T21:15:54.359 INFO:teuthology.orchestra.run.smithi023.stdout: "client": "client.15291", 2024-03-11T21:15:54.359 INFO:teuthology.orchestra.run.smithi023.stdout: "client_addr": "172.21.15.23:0/729656300", 2024-03-11T21:15:54.359 INFO:teuthology.orchestra.run.smithi023.stdout: "tid": 36121 2024-03-11T21:15:54.359 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.359 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:15:54.359 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.359 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:15:54.359 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:07:56.362220+0000", 2024-03-11T21:15:54.359 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:15:54.359 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.359 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.360 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:15:54.360 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:07:56.362220+0000", 2024-03-11T21:15:54.360 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:15:54.360 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.360 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.360 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:15:54.360 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:07:56.362222+0000", 2024-03-11T21:15:54.360 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.669e-06 2024-03-11T21:15:54.360 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.360 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.360 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:15:54.360 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:07:56.362244+0000", 2024-03-11T21:15:54.360 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.1911000000000001e-05 2024-03-11T21:15:54.360 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.360 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.361 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:15:54.361 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:07:56.362259+0000", 2024-03-11T21:15:54.361 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.4791e-05 2024-03-11T21:15:54.361 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.361 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.361 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:15:54.361 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:07:56.362267+0000", 2024-03-11T21:15:54.361 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.9719999999999992e-06 2024-03-11T21:15:54.361 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.361 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.361 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:15:54.361 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:07:56.362317+0000", 2024-03-11T21:15:54.361 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 4.9932000000000002e-05 2024-03-11T21:15:54.361 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.361 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.362 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:15:54.362 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:07:56.362418+0000", 2024-03-11T21:15:54.362 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00010139799999999999 2024-03-11T21:15:54.362 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.362 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.362 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for subops from 0,3", 2024-03-11T21:15:54.362 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:07:56.362516+0000", 2024-03-11T21:15:54.362 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 9.8024999999999997e-05 2024-03-11T21:15:54.362 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.362 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.362 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "op_commit", 2024-03-11T21:15:54.362 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:07:56.363889+0000", 2024-03-11T21:15:54.362 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0013730000000000001 2024-03-11T21:15:54.362 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.363 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.363 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:15:54.363 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:07:56.364083+0000", 2024-03-11T21:15:54.363 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000193597 2024-03-11T21:15:54.363 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.363 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.363 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:15:54.363 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:07:56.654319+0000", 2024-03-11T21:15:54.363 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.29023663599999999 2024-03-11T21:15:54.363 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.363 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.363 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T21:15:54.363 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:07:56.655045+0000", 2024-03-11T21:15:54.363 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000725988 2024-03-11T21:15:54.363 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.363 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.365 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:15:54.365 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:07:56.655140+0000", 2024-03-11T21:15:54.365 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 9.4622000000000003e-05 2024-03-11T21:15:54.365 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:15:54.365 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:15:54.366 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:15:54.366 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.366 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.366 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_op(client.15291.0:36131 10.5 10:a34df68a:::benchmark_data_smithi023_124944_object36130:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e316)", 2024-03-11T21:15:54.366 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T21:07:56.379043+0000", 2024-03-11T21:15:54.366 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 477.98331151500003, 2024-03-11T21:15:54.366 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.27728798399999999, 2024-03-11T21:15:54.366 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:15:54.366 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:15:54.366 INFO:teuthology.orchestra.run.smithi023.stdout: "client_info": { 2024-03-11T21:15:54.366 INFO:teuthology.orchestra.run.smithi023.stdout: "client": "client.15291", 2024-03-11T21:15:54.367 INFO:teuthology.orchestra.run.smithi023.stdout: "client_addr": "172.21.15.23:0/729656300", 2024-03-11T21:15:54.367 INFO:teuthology.orchestra.run.smithi023.stdout: "tid": 36131 2024-03-11T21:15:54.367 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.367 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:15:54.367 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.367 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:15:54.367 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:07:56.379043+0000", 2024-03-11T21:15:54.367 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:15:54.367 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.367 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.367 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:15:54.367 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:07:56.379043+0000", 2024-03-11T21:15:54.367 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:15:54.368 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.368 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.368 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:15:54.368 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:07:56.379044+0000", 2024-03-11T21:15:54.368 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.28e-06 2024-03-11T21:15:54.368 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.368 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.368 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:15:54.368 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:07:56.379064+0000", 2024-03-11T21:15:54.368 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.9349000000000001e-05 2024-03-11T21:15:54.368 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.368 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.368 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:15:54.368 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:07:56.379075+0000", 2024-03-11T21:15:54.369 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.1409999999999999e-05 2024-03-11T21:15:54.369 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.369 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.369 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:15:54.369 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:07:56.379081+0000", 2024-03-11T21:15:54.369 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 5.7189999999999998e-06 2024-03-11T21:15:54.369 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.369 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.369 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:15:54.369 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:07:56.379134+0000", 2024-03-11T21:15:54.369 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 5.2941e-05 2024-03-11T21:15:54.369 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.369 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.369 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:15:54.370 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:07:56.379239+0000", 2024-03-11T21:15:54.370 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000104589 2024-03-11T21:15:54.370 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.370 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.370 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for subops from 0,3", 2024-03-11T21:15:54.370 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:07:56.379339+0000", 2024-03-11T21:15:54.370 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00010030399999999999 2024-03-11T21:15:54.370 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.370 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.370 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "op_commit", 2024-03-11T21:15:54.370 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:07:56.380690+0000", 2024-03-11T21:15:54.370 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0013515249999999999 2024-03-11T21:15:54.371 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.371 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.371 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:15:54.371 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:07:56.380986+0000", 2024-03-11T21:15:54.371 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000295474 2024-03-11T21:15:54.371 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.371 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.371 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:15:54.371 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:07:56.655246+0000", 2024-03-11T21:15:54.371 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.27426044100000002 2024-03-11T21:15:54.371 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.371 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.371 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T21:15:54.371 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:07:56.656022+0000", 2024-03-11T21:15:54.371 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00077543200000000005 2024-03-11T21:15:54.372 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.372 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.372 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:15:54.372 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:07:56.656331+0000", 2024-03-11T21:15:54.372 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00030951999999999998 2024-03-11T21:15:54.372 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:15:54.372 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:15:54.372 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:15:54.372 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.372 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.372 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_op(client.15291.0:45062 10.5 10:a3d6b3da:::benchmark_data_smithi023_124944_object45061:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e318)", 2024-03-11T21:15:54.372 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T21:08:03.238438+0000", 2024-03-11T21:15:54.372 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 471.12391708299998, 2024-03-11T21:15:54.372 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.30270804099999998, 2024-03-11T21:15:54.373 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:15:54.373 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:15:54.373 INFO:teuthology.orchestra.run.smithi023.stdout: "client_info": { 2024-03-11T21:15:54.373 INFO:teuthology.orchestra.run.smithi023.stdout: "client": "client.15291", 2024-03-11T21:15:54.373 INFO:teuthology.orchestra.run.smithi023.stdout: "client_addr": "172.21.15.23:0/729656300", 2024-03-11T21:15:54.373 INFO:teuthology.orchestra.run.smithi023.stdout: "tid": 45062 2024-03-11T21:15:54.373 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.373 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:15:54.373 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.373 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:15:54.373 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.238438+0000", 2024-03-11T21:15:54.373 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:15:54.373 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.373 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.373 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:15:54.374 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.238438+0000", 2024-03-11T21:15:54.374 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:15:54.374 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.374 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.374 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:15:54.374 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.238439+0000", 2024-03-11T21:15:54.374 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.302e-06 2024-03-11T21:15:54.374 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.374 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.374 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:15:54.374 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.238460+0000", 2024-03-11T21:15:54.374 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.1270000000000001e-05 2024-03-11T21:15:54.374 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.374 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.374 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:15:54.375 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.238471+0000", 2024-03-11T21:15:54.375 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.1117999999999999e-05 2024-03-11T21:15:54.375 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.375 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.375 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:15:54.375 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.238479+0000", 2024-03-11T21:15:54.375 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.3300000000000001e-06 2024-03-11T21:15:54.375 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.375 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.375 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:15:54.375 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.238530+0000", 2024-03-11T21:15:54.375 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 5.1811000000000003e-05 2024-03-11T21:15:54.375 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.375 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.375 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:15:54.376 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.238649+0000", 2024-03-11T21:15:54.376 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00011900000000000001 2024-03-11T21:15:54.376 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.376 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.376 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for subops from 0,3", 2024-03-11T21:15:54.376 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.238781+0000", 2024-03-11T21:15:54.376 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00013152500000000001 2024-03-11T21:15:54.376 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.376 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.376 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "op_commit", 2024-03-11T21:15:54.376 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.240785+0000", 2024-03-11T21:15:54.376 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00200431 2024-03-11T21:15:54.376 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.376 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.376 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:15:54.376 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.540829+0000", 2024-03-11T21:15:54.377 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.300043485 2024-03-11T21:15:54.377 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.377 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.377 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:15:54.377 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.540961+0000", 2024-03-11T21:15:54.377 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000132373 2024-03-11T21:15:54.377 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.377 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.377 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T21:15:54.377 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.541091+0000", 2024-03-11T21:15:54.377 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00012990199999999999 2024-03-11T21:15:54.377 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.378 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.378 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:15:54.378 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.541146+0000", 2024-03-11T21:15:54.378 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 5.4614999999999998e-05 2024-03-11T21:15:54.378 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:15:54.378 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:15:54.378 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:15:54.378 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.378 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.378 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_op(client.15291.0:45065 10.5 10:a028b1f5:::benchmark_data_smithi023_124944_object45064:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e318)", 2024-03-11T21:15:54.378 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T21:08:03.242285+0000", 2024-03-11T21:15:54.378 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 471.12006999200003, 2024-03-11T21:15:54.378 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.30386189400000002, 2024-03-11T21:15:54.378 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:15:54.379 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:15:54.379 INFO:teuthology.orchestra.run.smithi023.stdout: "client_info": { 2024-03-11T21:15:54.379 INFO:teuthology.orchestra.run.smithi023.stdout: "client": "client.15291", 2024-03-11T21:15:54.379 INFO:teuthology.orchestra.run.smithi023.stdout: "client_addr": "172.21.15.23:0/729656300", 2024-03-11T21:15:54.379 INFO:teuthology.orchestra.run.smithi023.stdout: "tid": 45065 2024-03-11T21:15:54.379 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.379 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:15:54.379 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.379 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:15:54.380 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.242285+0000", 2024-03-11T21:15:54.380 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:15:54.380 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.380 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.380 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:15:54.380 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.242285+0000", 2024-03-11T21:15:54.380 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:15:54.380 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.380 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.380 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:15:54.380 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.242286+0000", 2024-03-11T21:15:54.380 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.6509999999999999e-06 2024-03-11T21:15:54.381 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.381 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.381 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:15:54.381 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.242305+0000", 2024-03-11T21:15:54.381 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.9097999999999999e-05 2024-03-11T21:15:54.381 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.381 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.381 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:15:54.381 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.242315+0000", 2024-03-11T21:15:54.381 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 9.7040000000000006e-06 2024-03-11T21:15:54.382 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.382 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.382 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:15:54.382 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.242322+0000", 2024-03-11T21:15:54.382 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.3659999999999997e-06 2024-03-11T21:15:54.382 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.382 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.382 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:15:54.382 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.242387+0000", 2024-03-11T21:15:54.383 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.5164000000000002e-05 2024-03-11T21:15:54.383 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.383 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.383 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:15:54.383 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.242498+0000", 2024-03-11T21:15:54.383 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000111625 2024-03-11T21:15:54.383 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.383 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.383 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for subops from 0,3", 2024-03-11T21:15:54.383 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.242599+0000", 2024-03-11T21:15:54.383 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00010038400000000001 2024-03-11T21:15:54.383 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.383 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.384 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:15:54.384 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.540897+0000", 2024-03-11T21:15:54.384 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.29829849000000003 2024-03-11T21:15:54.384 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.384 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.384 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:15:54.384 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.541195+0000", 2024-03-11T21:15:54.384 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00029787200000000002 2024-03-11T21:15:54.384 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.384 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.385 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "op_commit", 2024-03-11T21:15:54.385 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.546006+0000", 2024-03-11T21:15:54.385 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00481132 2024-03-11T21:15:54.385 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.385 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.385 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T21:15:54.385 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.546083+0000", 2024-03-11T21:15:54.385 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.6267999999999996e-05 2024-03-11T21:15:54.385 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.385 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.385 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:15:54.385 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.546147+0000", 2024-03-11T21:15:54.386 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.3952e-05 2024-03-11T21:15:54.386 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:15:54.386 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:15:54.386 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:15:54.386 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.386 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.386 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_op(client.15291.0:45066 10.3 10:c8f8b29c:::benchmark_data_smithi023_124944_object45065:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e318)", 2024-03-11T21:15:54.386 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T21:08:03.244472+0000", 2024-03-11T21:15:54.386 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 471.11788223299999, 2024-03-11T21:15:54.386 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.29773306900000002, 2024-03-11T21:15:54.387 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:15:54.387 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:15:54.387 INFO:teuthology.orchestra.run.smithi023.stdout: "client_info": { 2024-03-11T21:15:54.387 INFO:teuthology.orchestra.run.smithi023.stdout: "client": "client.15291", 2024-03-11T21:15:54.387 INFO:teuthology.orchestra.run.smithi023.stdout: "client_addr": "172.21.15.23:0/729656300", 2024-03-11T21:15:54.387 INFO:teuthology.orchestra.run.smithi023.stdout: "tid": 45066 2024-03-11T21:15:54.387 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.387 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:15:54.387 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.387 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:15:54.387 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.244472+0000", 2024-03-11T21:15:54.388 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:15:54.388 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.388 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.388 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:15:54.388 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.244472+0000", 2024-03-11T21:15:54.388 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:15:54.388 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.388 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.388 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:15:54.388 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.244474+0000", 2024-03-11T21:15:54.388 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.5850000000000001e-06 2024-03-11T21:15:54.389 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.389 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.389 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:15:54.389 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.244501+0000", 2024-03-11T21:15:54.389 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.7251999999999999e-05 2024-03-11T21:15:54.389 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.389 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.389 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:15:54.389 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.244515+0000", 2024-03-11T21:15:54.389 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.3946e-05 2024-03-11T21:15:54.389 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.389 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.389 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:15:54.389 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.244524+0000", 2024-03-11T21:15:54.389 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 8.3809999999999999e-06 2024-03-11T21:15:54.390 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.390 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.390 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:15:54.390 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.244676+0000", 2024-03-11T21:15:54.390 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00015238400000000001 2024-03-11T21:15:54.390 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.390 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.390 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:15:54.390 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.244837+0000", 2024-03-11T21:15:54.390 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00016081900000000001 2024-03-11T21:15:54.390 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.390 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.390 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for subops from 3,5", 2024-03-11T21:15:54.391 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.245039+0000", 2024-03-11T21:15:54.391 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00020164799999999999 2024-03-11T21:15:54.391 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.391 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.391 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:15:54.391 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.246993+0000", 2024-03-11T21:15:54.391 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0019543830000000001 2024-03-11T21:15:54.391 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.391 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.391 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "op_commit", 2024-03-11T21:15:54.391 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.247209+0000", 2024-03-11T21:15:54.391 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00021650700000000001 2024-03-11T21:15:54.391 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.391 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.392 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:15:54.392 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.542068+0000", 2024-03-11T21:15:54.392 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.29485842499999998 2024-03-11T21:15:54.392 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.392 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.392 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T21:15:54.392 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.542165+0000", 2024-03-11T21:15:54.392 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 9.6874999999999997e-05 2024-03-11T21:15:54.392 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.392 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.393 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:15:54.393 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.542206+0000", 2024-03-11T21:15:54.393 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 4.0864e-05 2024-03-11T21:15:54.393 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:15:54.393 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:15:54.393 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:15:54.393 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.393 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.393 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_op(client.15291.0:45067 10.5 10:a13e9247:::benchmark_data_smithi023_124944_object45066:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e318)", 2024-03-11T21:15:54.393 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T21:08:03.244550+0000", 2024-03-11T21:15:54.393 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 471.11780427100001, 2024-03-11T21:15:54.393 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.30287361499999998, 2024-03-11T21:15:54.393 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:15:54.393 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:15:54.394 INFO:teuthology.orchestra.run.smithi023.stdout: "client_info": { 2024-03-11T21:15:54.394 INFO:teuthology.orchestra.run.smithi023.stdout: "client": "client.15291", 2024-03-11T21:15:54.394 INFO:teuthology.orchestra.run.smithi023.stdout: "client_addr": "172.21.15.23:0/729656300", 2024-03-11T21:15:54.394 INFO:teuthology.orchestra.run.smithi023.stdout: "tid": 45067 2024-03-11T21:15:54.394 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.394 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:15:54.394 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.394 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:15:54.394 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.244550+0000", 2024-03-11T21:15:54.394 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:15:54.394 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.394 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.394 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:15:54.394 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.244550+0000", 2024-03-11T21:15:54.395 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:15:54.395 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.395 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.395 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:15:54.395 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.244551+0000", 2024-03-11T21:15:54.395 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.2699999999999999e-07 2024-03-11T21:15:54.395 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.395 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.395 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:15:54.395 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.244572+0000", 2024-03-11T21:15:54.395 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.0678000000000001e-05 2024-03-11T21:15:54.395 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.395 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.395 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:15:54.395 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.244581+0000", 2024-03-11T21:15:54.396 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 8.8720000000000008e-06 2024-03-11T21:15:54.396 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.396 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.396 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:15:54.396 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.244588+0000", 2024-03-11T21:15:54.396 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.0430000000000002e-06 2024-03-11T21:15:54.396 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.396 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.396 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:15:54.396 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.247066+0000", 2024-03-11T21:15:54.396 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0024784070000000002 2024-03-11T21:15:54.396 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.397 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.397 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:15:54.397 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.247201+0000", 2024-03-11T21:15:54.397 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000134585 2024-03-11T21:15:54.397 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.397 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.397 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for subops from 0,3", 2024-03-11T21:15:54.397 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.537674+0000", 2024-03-11T21:15:54.397 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.29047304600000001 2024-03-11T21:15:54.397 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.397 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.397 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "op_commit", 2024-03-11T21:15:54.397 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.546160+0000", 2024-03-11T21:15:54.397 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0084859849999999997 2024-03-11T21:15:54.397 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.398 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.398 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:15:54.398 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.546892+0000", 2024-03-11T21:15:54.398 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00073225700000000005 2024-03-11T21:15:54.398 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.398 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.398 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:15:54.398 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.547326+0000", 2024-03-11T21:15:54.398 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000434435 2024-03-11T21:15:54.399 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.399 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.399 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T21:15:54.399 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.547384+0000", 2024-03-11T21:15:54.399 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 5.7858000000000001e-05 2024-03-11T21:15:54.399 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.399 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.399 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:15:54.399 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.547424+0000", 2024-03-11T21:15:54.399 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 3.9721999999999997e-05 2024-03-11T21:15:54.399 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:15:54.399 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:15:54.399 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:15:54.399 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.400 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.400 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_op(client.15291.0:45070 10.3 10:c860362f:::benchmark_data_smithi023_124944_object45069:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e318)", 2024-03-11T21:15:54.400 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T21:08:03.248420+0000", 2024-03-11T21:15:54.400 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 471.11393467699997, 2024-03-11T21:15:54.400 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.298756935, 2024-03-11T21:15:54.400 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:15:54.400 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:15:54.400 INFO:teuthology.orchestra.run.smithi023.stdout: "client_info": { 2024-03-11T21:15:54.400 INFO:teuthology.orchestra.run.smithi023.stdout: "client": "client.15291", 2024-03-11T21:15:54.400 INFO:teuthology.orchestra.run.smithi023.stdout: "client_addr": "172.21.15.23:0/729656300", 2024-03-11T21:15:54.400 INFO:teuthology.orchestra.run.smithi023.stdout: "tid": 45070 2024-03-11T21:15:54.401 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.401 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:15:54.401 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.401 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:15:54.401 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.248420+0000", 2024-03-11T21:15:54.401 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:15:54.401 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.401 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.401 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:15:54.401 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.248420+0000", 2024-03-11T21:15:54.401 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:15:54.401 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.402 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.402 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:15:54.402 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.248422+0000", 2024-03-11T21:15:54.402 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.0600000000000002e-06 2024-03-11T21:15:54.402 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.402 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.402 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:15:54.402 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.248460+0000", 2024-03-11T21:15:54.402 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 3.7824999999999998e-05 2024-03-11T21:15:54.402 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.402 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.402 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:15:54.402 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.535541+0000", 2024-03-11T21:15:54.403 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.28708099100000001 2024-03-11T21:15:54.403 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.403 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.403 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:15:54.403 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.535577+0000", 2024-03-11T21:15:54.403 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 3.5615999999999999e-05 2024-03-11T21:15:54.403 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.403 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.403 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:15:54.403 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.536442+0000", 2024-03-11T21:15:54.403 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00086580000000000001 2024-03-11T21:15:54.403 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.404 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.404 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:15:54.404 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.536875+0000", 2024-03-11T21:15:54.404 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000432397 2024-03-11T21:15:54.404 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.404 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.404 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for subops from 3,5", 2024-03-11T21:15:54.404 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.537295+0000", 2024-03-11T21:15:54.404 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00042025 2024-03-11T21:15:54.404 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.404 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.404 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "op_commit", 2024-03-11T21:15:54.404 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.542220+0000", 2024-03-11T21:15:54.405 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0049246589999999996 2024-03-11T21:15:54.405 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.405 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.405 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:15:54.405 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.544296+0000", 2024-03-11T21:15:54.405 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.002076479 2024-03-11T21:15:54.405 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.405 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.405 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:15:54.405 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.547068+0000", 2024-03-11T21:15:54.405 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.002772189 2024-03-11T21:15:54.405 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.405 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.405 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T21:15:54.406 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.547132+0000", 2024-03-11T21:15:54.406 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.4035999999999995e-05 2024-03-11T21:15:54.406 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.406 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.406 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:15:54.406 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.547177+0000", 2024-03-11T21:15:54.406 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 4.4632999999999998e-05 2024-03-11T21:15:54.406 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:15:54.406 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:15:54.406 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:15:54.406 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.406 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.406 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_repop_reply(client.15291.0:45062 10.5 e318/306 ondisk, result = 0)", 2024-03-11T21:15:54.407 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T21:08:03.255834+0000", 2024-03-11T21:15:54.407 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 471.10652095699999, 2024-03-11T21:15:54.407 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.28500816299999998, 2024-03-11T21:15:54.407 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:15:54.407 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "started", 2024-03-11T21:15:54.407 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:15:54.407 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.407 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:15:54.407 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.255834+0000", 2024-03-11T21:15:54.407 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:15:54.407 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.407 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.407 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:15:54.407 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.255834+0000", 2024-03-11T21:15:54.407 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:15:54.408 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.408 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.408 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:15:54.408 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.255840+0000", 2024-03-11T21:15:54.408 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 5.8409999999999996e-06 2024-03-11T21:15:54.408 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.408 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.408 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:15:54.408 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.255845+0000", 2024-03-11T21:15:54.408 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 5.6810000000000003e-06 2024-03-11T21:15:54.408 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.408 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.408 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:15:54.408 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.535738+0000", 2024-03-11T21:15:54.408 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.27989238300000002 2024-03-11T21:15:54.408 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.409 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.409 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:15:54.409 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.535747+0000", 2024-03-11T21:15:54.409 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 9.149e-06 2024-03-11T21:15:54.409 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.409 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.409 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:15:54.409 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.540805+0000", 2024-03-11T21:15:54.409 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.005058585 2024-03-11T21:15:54.409 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.409 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.409 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:15:54.409 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.540818+0000", 2024-03-11T21:15:54.409 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.289e-05 2024-03-11T21:15:54.410 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.410 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.410 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:15:54.410 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.540842+0000", 2024-03-11T21:15:54.410 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.3634000000000001e-05 2024-03-11T21:15:54.410 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:15:54.410 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:15:54.410 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:15:54.410 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.410 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.410 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_repop(client.15291.0:45080 10.9 e318/306 10:9c9e49b5:::benchmark_data_smithi023_124944_object45079:head v 318'2801, mlcod=318'2799)", 2024-03-11T21:15:54.410 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T21:08:03.258296+0000", 2024-03-11T21:15:54.410 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 471.10405873799999, 2024-03-11T21:15:54.410 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.281699225, 2024-03-11T21:15:54.411 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:15:54.411 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:15:54.411 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:15:54.411 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.411 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:15:54.411 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.258296+0000", 2024-03-11T21:15:54.411 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:15:54.411 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.411 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.411 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:15:54.411 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.258296+0000", 2024-03-11T21:15:54.411 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:15:54.411 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.411 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.411 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:15:54.412 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.258299+0000", 2024-03-11T21:15:54.412 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 3.388e-06 2024-03-11T21:15:54.412 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.412 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.412 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:15:54.412 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.258339+0000", 2024-03-11T21:15:54.412 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 3.9169999999999999e-05 2024-03-11T21:15:54.412 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.412 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.412 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:15:54.412 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.535739+0000", 2024-03-11T21:15:54.412 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.27740067200000001 2024-03-11T21:15:54.412 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.412 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.413 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:15:54.413 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.535750+0000", 2024-03-11T21:15:54.413 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.0841e-05 2024-03-11T21:15:54.413 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.413 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.413 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:15:54.413 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.536073+0000", 2024-03-11T21:15:54.413 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00032279299999999998 2024-03-11T21:15:54.413 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.413 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.413 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:15:54.413 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.536093+0000", 2024-03-11T21:15:54.413 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.0530000000000002e-05 2024-03-11T21:15:54.413 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.414 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.414 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T21:15:54.414 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.539948+0000", 2024-03-11T21:15:54.414 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0038548639999999999 2024-03-11T21:15:54.414 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.414 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.414 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:15:54.414 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.539995+0000", 2024-03-11T21:15:54.414 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 4.6966999999999999e-05 2024-03-11T21:15:54.414 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:15:54.414 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:15:54.414 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:15:54.414 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.414 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.415 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_op(client.15291.0:45109 10.5 10:a5649fa4:::benchmark_data_smithi023_124944_object45108:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e318)", 2024-03-11T21:15:54.415 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T21:08:03.548069+0000", 2024-03-11T21:15:54.415 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 470.814285246, 2024-03-11T21:15:54.415 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.245461016, 2024-03-11T21:15:54.415 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:15:54.415 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:15:54.415 INFO:teuthology.orchestra.run.smithi023.stdout: "client_info": { 2024-03-11T21:15:54.415 INFO:teuthology.orchestra.run.smithi023.stdout: "client": "client.15291", 2024-03-11T21:15:54.415 INFO:teuthology.orchestra.run.smithi023.stdout: "client_addr": "172.21.15.23:0/729656300", 2024-03-11T21:15:54.415 INFO:teuthology.orchestra.run.smithi023.stdout: "tid": 45109 2024-03-11T21:15:54.415 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.415 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:15:54.415 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.415 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:15:54.416 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.548069+0000", 2024-03-11T21:15:54.416 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:15:54.416 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.416 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.416 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:15:54.416 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.548069+0000", 2024-03-11T21:15:54.416 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:15:54.416 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.416 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.416 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:15:54.416 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.548071+0000", 2024-03-11T21:15:54.416 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.097e-06 2024-03-11T21:15:54.416 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.417 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.417 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:15:54.417 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.548095+0000", 2024-03-11T21:15:54.417 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.4893e-05 2024-03-11T21:15:54.417 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.417 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.417 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:15:54.417 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.773244+0000", 2024-03-11T21:15:54.417 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.225148719 2024-03-11T21:15:54.417 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.417 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.417 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:15:54.417 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.773284+0000", 2024-03-11T21:15:54.417 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 3.9433e-05 2024-03-11T21:15:54.417 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.418 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.418 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:15:54.418 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.775090+0000", 2024-03-11T21:15:54.418 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.001806001 2024-03-11T21:15:54.418 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.418 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.418 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:15:54.418 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.775418+0000", 2024-03-11T21:15:54.419 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00032868099999999999 2024-03-11T21:15:54.419 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.419 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.419 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "waiting for subops from 0,3", 2024-03-11T21:15:54.419 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.775836+0000", 2024-03-11T21:15:54.419 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00041763400000000001 2024-03-11T21:15:54.419 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.419 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.419 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "op_commit", 2024-03-11T21:15:54.419 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.780021+0000", 2024-03-11T21:15:54.419 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0041853100000000002 2024-03-11T21:15:54.419 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.419 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.419 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:15:54.420 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.785472+0000", 2024-03-11T21:15:54.420 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0054502739999999997 2024-03-11T21:15:54.420 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.420 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.420 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:15:54.420 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.793289+0000", 2024-03-11T21:15:54.420 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0078170159999999995 2024-03-11T21:15:54.420 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.420 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.420 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T21:15:54.421 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.793432+0000", 2024-03-11T21:15:54.421 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000143745 2024-03-11T21:15:54.421 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.421 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.421 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:15:54.421 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.793531+0000", 2024-03-11T21:15:54.421 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 9.8213000000000003e-05 2024-03-11T21:15:54.421 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:15:54.421 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:15:54.421 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:15:54.421 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.421 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.421 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_repop(client.15291.0:45103 10.8 e318/306 10:196c74ce:::benchmark_data_smithi023_124944_object45102:head v 318'2855, mlcod=318'2853)", 2024-03-11T21:15:54.422 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T21:08:03.548073+0000", 2024-03-11T21:15:54.422 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 470.81428211000002, 2024-03-11T21:15:54.422 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.242017859, 2024-03-11T21:15:54.422 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:15:54.422 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:15:54.422 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:15:54.422 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.422 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:15:54.422 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.548073+0000", 2024-03-11T21:15:54.422 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:15:54.422 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.423 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.423 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:15:54.423 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.548073+0000", 2024-03-11T21:15:54.423 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:15:54.423 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.423 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.423 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:15:54.423 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.548073+0000", 2024-03-11T21:15:54.423 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.0999999999999998e-07 2024-03-11T21:15:54.423 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.423 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.424 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:15:54.424 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.548103+0000", 2024-03-11T21:15:54.424 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.9651000000000001e-05 2024-03-11T21:15:54.424 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.424 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.424 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:15:54.424 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.773233+0000", 2024-03-11T21:15:54.424 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.22513018900000001 2024-03-11T21:15:54.424 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.424 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.424 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:15:54.425 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.773268+0000", 2024-03-11T21:15:54.425 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 3.5102999999999999e-05 2024-03-11T21:15:54.425 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.425 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.425 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:15:54.425 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.773422+0000", 2024-03-11T21:15:54.425 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000153969 2024-03-11T21:15:54.425 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.425 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.425 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:15:54.425 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.773482+0000", 2024-03-11T21:15:54.425 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 5.9695999999999999e-05 2024-03-11T21:15:54.425 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.426 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.426 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T21:15:54.426 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.789989+0000", 2024-03-11T21:15:54.426 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.016507404999999999 2024-03-11T21:15:54.426 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.426 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.426 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:15:54.426 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:08:03.790090+0000", 2024-03-11T21:15:54.426 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000101136 2024-03-11T21:15:54.426 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:15:54.427 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:15:54.427 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:15:54.427 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.427 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.427 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_repop(client.15291.0:108986 10.8 e334/306 10:162e1d4d:::benchmark_data_smithi023_124944_object108985:head v 334'6811, mlcod=334'6809)", 2024-03-11T21:15:54.427 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T21:09:06.179659+0000", 2024-03-11T21:15:54.427 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 408.18269580999998, 2024-03-11T21:15:54.427 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.24230507100000001, 2024-03-11T21:15:54.427 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:15:54.427 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:15:54.427 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:15:54.427 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.427 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:15:54.428 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:09:06.179659+0000", 2024-03-11T21:15:54.428 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:15:54.428 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.428 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.428 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:15:54.428 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:09:06.179659+0000", 2024-03-11T21:15:54.428 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:15:54.428 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.428 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.428 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:15:54.428 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:09:06.179660+0000", 2024-03-11T21:15:54.428 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.2500000000000005e-07 2024-03-11T21:15:54.429 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.429 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.429 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:15:54.429 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:09:06.179695+0000", 2024-03-11T21:15:54.429 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 3.5398000000000002e-05 2024-03-11T21:15:54.429 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.429 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.429 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:15:54.429 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:09:06.179704+0000", 2024-03-11T21:15:54.429 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 8.7450000000000001e-06 2024-03-11T21:15:54.429 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.429 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.429 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:15:54.429 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:09:06.179710+0000", 2024-03-11T21:15:54.430 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 5.9850000000000003e-06 2024-03-11T21:15:54.430 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.430 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.430 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:15:54.430 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:09:06.411298+0000", 2024-03-11T21:15:54.430 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.23158834 2024-03-11T21:15:54.430 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.430 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.430 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:15:54.430 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:09:06.411360+0000", 2024-03-11T21:15:54.430 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.1846000000000004e-05 2024-03-11T21:15:54.430 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.430 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.431 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T21:15:54.431 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:09:06.421873+0000", 2024-03-11T21:15:54.431 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.010513201 2024-03-11T21:15:54.431 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.431 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.431 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:15:54.431 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:09:06.421964+0000", 2024-03-11T21:15:54.431 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 9.0931000000000001e-05 2024-03-11T21:15:54.431 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:15:54.431 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:15:54.431 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:15:54.431 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.431 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.431 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_repop(client.15498.0:38778 11.c e391/380 11:3a37bc44:::benchmark_data_smithi023_142591_object38777:head v 391'2473, mlcod=391'2471)", 2024-03-11T21:15:54.431 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T21:12:27.148958+0000", 2024-03-11T21:15:54.432 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 207.213397222, 2024-03-11T21:15:54.432 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.27684184499999998, 2024-03-11T21:15:54.432 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:15:54.432 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:15:54.432 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:15:54.432 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.432 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:15:54.432 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:27.148958+0000", 2024-03-11T21:15:54.432 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:15:54.432 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.432 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.433 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:15:54.433 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:27.148958+0000", 2024-03-11T21:15:54.433 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:15:54.433 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.433 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.433 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:15:54.433 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:27.148958+0000", 2024-03-11T21:15:54.433 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.1399999999999996e-07 2024-03-11T21:15:54.433 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.433 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.433 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:15:54.433 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:27.148976+0000", 2024-03-11T21:15:54.433 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.7793999999999999e-05 2024-03-11T21:15:54.433 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.434 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.434 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:15:54.434 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:27.148984+0000", 2024-03-11T21:15:54.434 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 8.4230000000000005e-06 2024-03-11T21:15:54.434 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.434 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.434 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:15:54.434 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:27.148990+0000", 2024-03-11T21:15:54.434 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 5.1440000000000002e-06 2024-03-11T21:15:54.434 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.435 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.435 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:15:54.435 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:27.149045+0000", 2024-03-11T21:15:54.435 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 5.5411999999999997e-05 2024-03-11T21:15:54.435 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.435 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.435 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:15:54.435 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:27.149058+0000", 2024-03-11T21:15:54.435 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.2607e-05 2024-03-11T21:15:54.435 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.435 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.435 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T21:15:54.436 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:27.425645+0000", 2024-03-11T21:15:54.436 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.276587052 2024-03-11T21:15:54.436 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.437 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.437 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:15:54.437 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:27.425799+0000", 2024-03-11T21:15:54.437 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000154699 2024-03-11T21:15:54.437 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:15:54.437 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:15:54.437 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:15:54.437 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.437 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.437 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_repop(client.15498.0:38791 11.c e391/380 11:3c61b285:::benchmark_data_smithi023_142591_object38790:head v 391'2474, mlcod=391'2471)", 2024-03-11T21:15:54.437 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T21:12:27.151711+0000", 2024-03-11T21:15:54.437 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 207.21064417599999, 2024-03-11T21:15:54.437 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.27447223399999998, 2024-03-11T21:15:54.437 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:15:54.437 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:15:54.438 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:15:54.438 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.438 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:15:54.438 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:27.151711+0000", 2024-03-11T21:15:54.438 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:15:54.438 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.438 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.438 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:15:54.438 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:27.151711+0000", 2024-03-11T21:15:54.438 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:15:54.438 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.438 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.438 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:15:54.438 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:27.151711+0000", 2024-03-11T21:15:54.438 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.8599999999999998e-07 2024-03-11T21:15:54.439 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.439 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.439 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:15:54.439 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:27.151746+0000", 2024-03-11T21:15:54.439 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 3.4282999999999997e-05 2024-03-11T21:15:54.439 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.439 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.439 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:15:54.439 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:27.413569+0000", 2024-03-11T21:15:54.439 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.26182353899999999 2024-03-11T21:15:54.439 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.439 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.439 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:15:54.439 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:27.413596+0000", 2024-03-11T21:15:54.439 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.6570000000000001e-05 2024-03-11T21:15:54.440 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.440 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.440 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:15:54.440 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:27.416413+0000", 2024-03-11T21:15:54.440 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0028173299999999998 2024-03-11T21:15:54.440 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.440 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.440 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:15:54.440 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:27.416463+0000", 2024-03-11T21:15:54.440 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 5.0179999999999997e-05 2024-03-11T21:15:54.440 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.440 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.440 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T21:15:54.440 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:27.425805+0000", 2024-03-11T21:15:54.440 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0093416650000000007 2024-03-11T21:15:54.441 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.441 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.441 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:15:54.441 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:27.426183+0000", 2024-03-11T21:15:54.441 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000377981 2024-03-11T21:15:54.441 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:15:54.441 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:15:54.441 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:15:54.441 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.441 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.441 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_repop(client.15498.0:38789 11.d e391/380 11:b7f02166:::benchmark_data_smithi023_142591_object38788:head v 391'2378, mlcod=391'2375)", 2024-03-11T21:15:54.441 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T21:12:27.154407+0000", 2024-03-11T21:15:54.441 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 207.20794771800001, 2024-03-11T21:15:54.441 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.26938732599999998, 2024-03-11T21:15:54.442 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:15:54.442 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:15:54.442 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:15:54.442 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.442 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:15:54.442 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:27.154407+0000", 2024-03-11T21:15:54.442 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:15:54.442 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.442 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.442 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:15:54.442 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:27.154407+0000", 2024-03-11T21:15:54.442 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:15:54.442 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.442 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.443 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:15:54.443 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:27.154408+0000", 2024-03-11T21:15:54.443 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.7599999999999996e-07 2024-03-11T21:15:54.443 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.443 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.443 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:15:54.443 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:27.154480+0000", 2024-03-11T21:15:54.443 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.2410000000000006e-05 2024-03-11T21:15:54.443 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.443 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.443 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:15:54.443 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:27.413555+0000", 2024-03-11T21:15:54.443 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.25907460599999999 2024-03-11T21:15:54.443 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.443 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.444 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:15:54.444 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:27.413580+0000", 2024-03-11T21:15:54.444 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.5596e-05 2024-03-11T21:15:54.444 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.444 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.444 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:15:54.444 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:27.413729+0000", 2024-03-11T21:15:54.444 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00014908099999999999 2024-03-11T21:15:54.444 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.444 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.444 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:15:54.444 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:27.413793+0000", 2024-03-11T21:15:54.444 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 6.3316000000000002e-05 2024-03-11T21:15:54.444 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.444 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.445 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T21:15:54.445 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:27.423706+0000", 2024-03-11T21:15:54.445 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0099127299999999998 2024-03-11T21:15:54.445 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.445 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.445 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:15:54.445 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:27.423794+0000", 2024-03-11T21:15:54.445 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 8.8811000000000001e-05 2024-03-11T21:15:54.445 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:15:54.445 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:15:54.445 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:15:54.445 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.445 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.445 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_repop(client.15498.0:47406 11.d e391/380 11:bc1d0c58:::benchmark_data_smithi023_142591_object47405:head v 391'2918, mlcod=391'2916)", 2024-03-11T21:15:54.446 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T21:12:33.525013+0000", 2024-03-11T21:15:54.446 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 200.83734135399999, 2024-03-11T21:15:54.446 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.28924738900000002, 2024-03-11T21:15:54.446 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:15:54.446 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:15:54.446 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:15:54.446 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.446 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:15:54.446 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:33.525013+0000", 2024-03-11T21:15:54.446 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:15:54.446 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.446 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.446 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:15:54.446 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:33.525013+0000", 2024-03-11T21:15:54.446 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:15:54.447 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.447 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.447 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:15:54.447 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:33.525015+0000", 2024-03-11T21:15:54.447 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.652e-06 2024-03-11T21:15:54.447 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.447 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.447 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:15:54.447 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:33.525063+0000", 2024-03-11T21:15:54.447 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 4.7583999999999997e-05 2024-03-11T21:15:54.447 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.447 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.447 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:15:54.447 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:33.525084+0000", 2024-03-11T21:15:54.447 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.0917999999999999e-05 2024-03-11T21:15:54.448 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.448 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.448 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:15:54.448 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:33.525097+0000", 2024-03-11T21:15:54.448 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.323e-05 2024-03-11T21:15:54.448 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.448 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.448 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:15:54.448 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:33.525238+0000", 2024-03-11T21:15:54.448 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00014094300000000001 2024-03-11T21:15:54.448 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.448 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.448 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:15:54.448 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:33.811850+0000", 2024-03-11T21:15:54.448 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.28661266099999999 2024-03-11T21:15:54.449 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.449 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.449 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T21:15:54.449 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:33.814187+0000", 2024-03-11T21:15:54.449 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0023363490000000002 2024-03-11T21:15:54.449 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.449 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.449 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:15:54.449 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:33.814261+0000", 2024-03-11T21:15:54.449 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 7.4052000000000002e-05 2024-03-11T21:15:54.449 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:15:54.449 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:15:54.449 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:15:54.449 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.450 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.450 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_repop(client.15498.0:47412 11.6 e391/380 11:64f17ae9:::benchmark_data_smithi023_142591_object47411:head v 391'2944, mlcod=391'2942)", 2024-03-11T21:15:54.450 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T21:12:33.527646+0000", 2024-03-11T21:15:54.450 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 200.83470840800001, 2024-03-11T21:15:54.450 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.30094739100000001, 2024-03-11T21:15:54.450 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:15:54.450 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:15:54.450 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:15:54.450 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.450 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:15:54.450 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:33.527646+0000", 2024-03-11T21:15:54.450 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:15:54.450 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.450 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.451 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:15:54.451 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:33.527646+0000", 2024-03-11T21:15:54.451 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:15:54.451 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.451 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.451 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:15:54.451 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:33.527648+0000", 2024-03-11T21:15:54.451 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.6080000000000001e-06 2024-03-11T21:15:54.451 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.451 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.451 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:15:54.451 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:33.527689+0000", 2024-03-11T21:15:54.451 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 4.066e-05 2024-03-11T21:15:54.451 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.451 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.452 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:15:54.452 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:33.811850+0000", 2024-03-11T21:15:54.452 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.28416170000000002 2024-03-11T21:15:54.452 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.452 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.452 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:15:54.452 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:33.811879+0000", 2024-03-11T21:15:54.452 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.8846000000000001e-05 2024-03-11T21:15:54.452 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.452 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.452 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:15:54.452 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:33.811993+0000", 2024-03-11T21:15:54.452 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.00011390100000000001 2024-03-11T21:15:54.452 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.453 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.453 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:15:54.453 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:33.812020+0000", 2024-03-11T21:15:54.453 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.6653999999999999e-05 2024-03-11T21:15:54.453 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.453 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.453 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T21:15:54.453 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:33.828450+0000", 2024-03-11T21:15:54.453 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.016430450999999999 2024-03-11T21:15:54.453 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.453 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.453 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:15:54.453 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:33.828594+0000", 2024-03-11T21:15:54.453 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000143571 2024-03-11T21:15:54.453 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:15:54.454 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:15:54.454 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:15:54.454 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.454 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.454 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "osd_repop(client.15498.0:47415 11.c e391/380 11:35b86436:::benchmark_data_smithi023_142591_object47414:head v 391'3015, mlcod=391'3013)", 2024-03-11T21:15:54.454 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T21:12:33.530728+0000", 2024-03-11T21:15:54.454 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 200.83162668599999, 2024-03-11T21:15:54.454 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.28829812900000001, 2024-03-11T21:15:54.454 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:15:54.454 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:15:54.454 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:15:54.454 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.454 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:15:54.454 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:33.530728+0000", 2024-03-11T21:15:54.455 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:15:54.455 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.455 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.455 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:15:54.455 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:33.530728+0000", 2024-03-11T21:15:54.455 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:15:54.455 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.455 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.455 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:15:54.455 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:33.530730+0000", 2024-03-11T21:15:54.455 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.5260000000000001e-06 2024-03-11T21:15:54.455 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.455 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.455 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:15:54.456 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:33.530768+0000", 2024-03-11T21:15:54.456 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 3.8581999999999998e-05 2024-03-11T21:15:54.456 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.456 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.456 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:15:54.456 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:33.811861+0000", 2024-03-11T21:15:54.456 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.281092913 2024-03-11T21:15:54.456 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.456 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.456 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:15:54.456 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:33.811879+0000", 2024-03-11T21:15:54.456 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.8221000000000001e-05 2024-03-11T21:15:54.456 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.456 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.456 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:15:54.457 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:33.811981+0000", 2024-03-11T21:15:54.457 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000101764 2024-03-11T21:15:54.457 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.457 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.457 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:15:54.457 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:33.812016+0000", 2024-03-11T21:15:54.457 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 3.5160000000000002e-05 2024-03-11T21:15:54.457 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.457 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.457 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "commit_sent", 2024-03-11T21:15:54.457 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:33.818931+0000", 2024-03-11T21:15:54.457 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.0069151899999999999 2024-03-11T21:15:54.457 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.457 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.457 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:15:54.457 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:12:33.819026+0000", 2024-03-11T21:15:54.458 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 9.4772999999999995e-05 2024-03-11T21:15:54.458 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:15:54.458 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:15:54.458 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:15:54.458 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.458 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.458 INFO:teuthology.orchestra.run.smithi023.stdout: "description": "pg_backfill(progress 11.1 e 410/410 lb 11:8a9262c1:::benchmark_data_smithi023_142591_object60644:head)", 2024-03-11T21:15:54.458 INFO:teuthology.orchestra.run.smithi023.stdout: "initiated_at": "2024-03-11T21:13:46.880489+0000", 2024-03-11T21:15:54.458 INFO:teuthology.orchestra.run.smithi023.stdout: "age": 127.481865351, 2024-03-11T21:15:54.458 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.25832654599999999, 2024-03-11T21:15:54.458 INFO:teuthology.orchestra.run.smithi023.stdout: "type_data": { 2024-03-11T21:15:54.458 INFO:teuthology.orchestra.run.smithi023.stdout: "flag_point": "started", 2024-03-11T21:15:54.458 INFO:teuthology.orchestra.run.smithi023.stdout: "events": [ 2024-03-11T21:15:54.458 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.459 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "initiated", 2024-03-11T21:15:54.459 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:13:46.880489+0000", 2024-03-11T21:15:54.459 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:15:54.459 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.459 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.459 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "header_read", 2024-03-11T21:15:54.459 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:13:46.880489+0000", 2024-03-11T21:15:54.459 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0 2024-03-11T21:15:54.459 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.459 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.459 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "throttled", 2024-03-11T21:15:54.459 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:13:46.880490+0000", 2024-03-11T21:15:54.459 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 2.9799999999999999e-07 2024-03-11T21:15:54.459 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.459 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.460 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "all_read", 2024-03-11T21:15:54.460 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:13:46.880494+0000", 2024-03-11T21:15:54.460 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 4.4109999999999998e-06 2024-03-11T21:15:54.460 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.460 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.460 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "dispatched", 2024-03-11T21:15:54.460 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:13:46.880499+0000", 2024-03-11T21:15:54.460 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 4.8269999999999997e-06 2024-03-11T21:15:54.460 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.460 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.460 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "queued_for_pg", 2024-03-11T21:15:54.460 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:13:46.880502+0000", 2024-03-11T21:15:54.460 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 3.4520000000000002e-06 2024-03-11T21:15:54.460 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.460 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.460 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "reached_pg", 2024-03-11T21:15:54.461 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:13:46.880705+0000", 2024-03-11T21:15:54.461 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.000203119 2024-03-11T21:15:54.461 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.461 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.461 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "started", 2024-03-11T21:15:54.461 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:13:46.880720+0000", 2024-03-11T21:15:54.461 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 1.453e-05 2024-03-11T21:15:54.461 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T21:15:54.461 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-11T21:15:54.461 INFO:teuthology.orchestra.run.smithi023.stdout: "event": "done", 2024-03-11T21:15:54.461 INFO:teuthology.orchestra.run.smithi023.stdout: "time": "2024-03-11T21:13:47.138816+0000", 2024-03-11T21:15:54.461 INFO:teuthology.orchestra.run.smithi023.stdout: "duration": 0.25809590900000001 2024-03-11T21:15:54.461 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:15:54.461 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:15:54.461 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:15:54.462 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T21:15:54.462 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T21:15:54.462 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-11T21:15:55.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:55 smithi023 bash[21372]: cluster 2024-03-11T21:15:54.466776+0000 mgr.y (mgr.24380) 927 : cluster 0 pgmap v1069: 142 pgs: 142 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-03-11T21:15:55.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:55 smithi023 bash[29146]: cluster 2024-03-11T21:15:54.466776+0000 mgr.y (mgr.24380) 927 : cluster 0 pgmap v1069: 142 pgs: 142 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-03-11T21:15:55.909 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-11T21:15:56.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:55 smithi146 bash[23809]: cluster 2024-03-11T21:15:54.466776+0000 mgr.y (mgr.24380) 927 : cluster 0 pgmap v1069: 142 pgs: 142 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-03-11T21:15:56.292 INFO:teuthology.orchestra.run.smithi023.stdout:osd.0: {} 2024-03-11T21:15:56.313 INFO:teuthology.orchestra.run.smithi023.stdout:osd.1: {} 2024-03-11T21:15:56.339 INFO:teuthology.orchestra.run.smithi023.stdout:osd.2: {} 2024-03-11T21:15:56.362 INFO:teuthology.orchestra.run.smithi023.stdout:osd.3: {} 2024-03-11T21:15:56.389 INFO:teuthology.orchestra.run.smithi023.stdout:osd.4: {} 2024-03-11T21:15:56.414 INFO:teuthology.orchestra.run.smithi023.stdout:osd.5: {} 2024-03-11T21:15:56.439 INFO:teuthology.orchestra.run.smithi023.stdout:osd.6: {} 2024-03-11T21:15:56.464 INFO:teuthology.orchestra.run.smithi023.stdout:osd.7: {} 2024-03-11T21:15:56.465 INFO:teuthology.orchestra.run.smithi023.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:15:56.465 INFO:teuthology.orchestra.run.smithi023.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:15:56.465 INFO:teuthology.orchestra.run.smithi023.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:15:56.465 INFO:teuthology.orchestra.run.smithi023.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:15:56.466 INFO:teuthology.orchestra.run.smithi023.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:15:56.466 INFO:teuthology.orchestra.run.smithi023.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:15:56.466 INFO:teuthology.orchestra.run.smithi023.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:15:56.466 INFO:teuthology.orchestra.run.smithi023.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:15:56.504 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:15:56 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:15:56.663 INFO:teuthology.orchestra.run.smithi023.stderr:noscrub is unset 2024-03-11T21:15:56.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:56 smithi023 bash[21372]: audit 2024-03-11T21:15:56.284193+0000 mon.a (mon.0) 1969 : audit 0 from='client.? 172.21.15.23:0/1267766933' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:15:56.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:56 smithi023 bash[21372]: audit 2024-03-11T21:15:56.584135+0000 mon.a (mon.0) 1970 : audit 1 from='client.? 172.21.15.23:0/3162888045' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:15:56.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:56 smithi023 bash[29146]: audit 2024-03-11T21:15:56.284193+0000 mon.a (mon.0) 1969 : audit 0 from='client.? 172.21.15.23:0/1267766933' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:15:56.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:56 smithi023 bash[29146]: audit 2024-03-11T21:15:56.584135+0000 mon.a (mon.0) 1970 : audit 1 from='client.? 172.21.15.23:0/3162888045' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:15:57.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:56 smithi146 bash[23809]: audit 2024-03-11T21:15:56.284193+0000 mon.a (mon.0) 1969 : audit 0 from='client.? 172.21.15.23:0/1267766933' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:15:57.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:56 smithi146 bash[23809]: audit 2024-03-11T21:15:56.584135+0000 mon.a (mon.0) 1970 : audit 1 from='client.? 172.21.15.23:0/3162888045' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:15:57.915 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:57 smithi023 bash[21372]: cluster 2024-03-11T21:15:55.833240+0000 osd.7 (osd.7) 565 : cluster 0 purged_snaps scrub starts 2024-03-11T21:15:57.916 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:57 smithi023 bash[21372]: cluster 2024-03-11T21:15:55.833352+0000 osd.7 (osd.7) 566 : cluster 0 purged_snaps scrub ok 2024-03-11T21:15:57.916 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:57 smithi023 bash[21372]: audit 2024-03-11T21:15:56.116365+0000 mgr.y (mgr.24380) 928 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:15:57.916 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:57 smithi023 bash[21372]: cluster 2024-03-11T21:15:56.467696+0000 mgr.y (mgr.24380) 929 : cluster 0 pgmap v1070: 142 pgs: 142 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-03-11T21:15:57.916 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:57 smithi023 bash[21372]: cluster 2024-03-11T21:15:56.652378+0000 mon.a (mon.0) 1971 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:15:57.916 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:57 smithi023 bash[21372]: audit 2024-03-11T21:15:56.673021+0000 mon.a (mon.0) 1972 : audit 1 from='client.? 172.21.15.23:0/3162888045' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:15:57.916 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:57 smithi023 bash[21372]: cluster 2024-03-11T21:15:56.696257+0000 mon.a (mon.0) 1973 : cluster 0 osdmap e438: 8 total, 8 up, 8 in 2024-03-11T21:15:57.916 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:57 smithi023 bash[29146]: cluster 2024-03-11T21:15:55.833240+0000 osd.7 (osd.7) 565 : cluster 0 purged_snaps scrub starts 2024-03-11T21:15:57.916 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:57 smithi023 bash[29146]: cluster 2024-03-11T21:15:55.833352+0000 osd.7 (osd.7) 566 : cluster 0 purged_snaps scrub ok 2024-03-11T21:15:57.916 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:57 smithi023 bash[29146]: audit 2024-03-11T21:15:56.116365+0000 mgr.y (mgr.24380) 928 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:15:57.917 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:57 smithi023 bash[29146]: cluster 2024-03-11T21:15:56.467696+0000 mgr.y (mgr.24380) 929 : cluster 0 pgmap v1070: 142 pgs: 142 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-03-11T21:15:57.917 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:57 smithi023 bash[29146]: cluster 2024-03-11T21:15:56.652378+0000 mon.a (mon.0) 1971 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:15:57.917 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:57 smithi023 bash[29146]: audit 2024-03-11T21:15:56.673021+0000 mon.a (mon.0) 1972 : audit 1 from='client.? 172.21.15.23:0/3162888045' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:15:57.917 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:57 smithi023 bash[29146]: cluster 2024-03-11T21:15:56.696257+0000 mon.a (mon.0) 1973 : cluster 0 osdmap e438: 8 total, 8 up, 8 in 2024-03-11T21:15:58.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:57 smithi146 bash[23809]: cluster 2024-03-11T21:15:55.833240+0000 osd.7 (osd.7) 565 : cluster 0 purged_snaps scrub starts 2024-03-11T21:15:58.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:57 smithi146 bash[23809]: cluster 2024-03-11T21:15:55.833352+0000 osd.7 (osd.7) 566 : cluster 0 purged_snaps scrub ok 2024-03-11T21:15:58.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:57 smithi146 bash[23809]: audit 2024-03-11T21:15:56.116365+0000 mgr.y (mgr.24380) 928 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:15:58.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:57 smithi146 bash[23809]: cluster 2024-03-11T21:15:56.467696+0000 mgr.y (mgr.24380) 929 : cluster 0 pgmap v1070: 142 pgs: 142 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-03-11T21:15:58.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:57 smithi146 bash[23809]: cluster 2024-03-11T21:15:56.652378+0000 mon.a (mon.0) 1971 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:15:58.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:57 smithi146 bash[23809]: audit 2024-03-11T21:15:56.673021+0000 mon.a (mon.0) 1972 : audit 1 from='client.? 172.21.15.23:0/3162888045' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:15:58.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:57 smithi146 bash[23809]: cluster 2024-03-11T21:15:56.696257+0000 mon.a (mon.0) 1973 : cluster 0 osdmap e438: 8 total, 8 up, 8 in 2024-03-11T21:15:58.679 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:15:58 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:15:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:15:59.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:58 smithi146 bash[23809]: cluster 2024-03-11T21:15:58.037747+0000 osd.1 (osd.1) 304 : cluster 0 6.1 scrub starts 2024-03-11T21:15:59.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:58 smithi146 bash[23809]: cluster 2024-03-11T21:15:58.118277+0000 osd.1 (osd.1) 305 : cluster 0 6.1 scrub ok 2024-03-11T21:15:59.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:58 smithi146 bash[23809]: cluster 2024-03-11T21:15:58.552413+0000 osd.0 (osd.0) 394 : cluster 0 6.18 scrub starts 2024-03-11T21:15:59.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:58 smithi023 bash[21372]: cluster 2024-03-11T21:15:58.037747+0000 osd.1 (osd.1) 304 : cluster 0 6.1 scrub starts 2024-03-11T21:15:59.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:58 smithi023 bash[21372]: cluster 2024-03-11T21:15:58.118277+0000 osd.1 (osd.1) 305 : cluster 0 6.1 scrub ok 2024-03-11T21:15:59.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:58 smithi023 bash[21372]: cluster 2024-03-11T21:15:58.552413+0000 osd.0 (osd.0) 394 : cluster 0 6.18 scrub starts 2024-03-11T21:15:59.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:58 smithi023 bash[29146]: cluster 2024-03-11T21:15:58.037747+0000 osd.1 (osd.1) 304 : cluster 0 6.1 scrub starts 2024-03-11T21:15:59.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:58 smithi023 bash[29146]: cluster 2024-03-11T21:15:58.118277+0000 osd.1 (osd.1) 305 : cluster 0 6.1 scrub ok 2024-03-11T21:15:59.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:58 smithi023 bash[29146]: cluster 2024-03-11T21:15:58.552413+0000 osd.0 (osd.0) 394 : cluster 0 6.18 scrub starts 2024-03-11T21:15:59.395 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:15:59.921 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd unset noscrub 2024-03-11T21:15:59.934 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:59 smithi023 bash[21372]: cluster 2024-03-11T21:15:58.468628+0000 mgr.y (mgr.24380) 930 : cluster 0 pgmap v1072: 142 pgs: 142 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T21:15:59.934 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:59 smithi023 bash[21372]: cluster 2024-03-11T21:15:58.636030+0000 osd.0 (osd.0) 395 : cluster 0 6.18 scrub ok 2024-03-11T21:15:59.934 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:59 smithi023 bash[21372]: cluster 2024-03-11T21:15:58.880509+0000 osd.6 (osd.6) 450 : cluster 0 2.7 scrub starts 2024-03-11T21:15:59.934 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:59 smithi023 bash[21372]: cluster 2024-03-11T21:15:58.883172+0000 osd.6 (osd.6) 451 : cluster 0 2.7 scrub ok 2024-03-11T21:15:59.934 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:59 smithi023 bash[21372]: cluster 2024-03-11T21:15:59.043110+0000 osd.1 (osd.1) 306 : cluster 0 6.1d scrub starts 2024-03-11T21:15:59.934 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:15:59 smithi023 bash[21372]: cluster 2024-03-11T21:15:59.207510+0000 osd.1 (osd.1) 307 : cluster 0 6.1d scrub ok 2024-03-11T21:15:59.935 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:59 smithi023 bash[29146]: cluster 2024-03-11T21:15:58.468628+0000 mgr.y (mgr.24380) 930 : cluster 0 pgmap v1072: 142 pgs: 142 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T21:15:59.935 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:59 smithi023 bash[29146]: cluster 2024-03-11T21:15:58.636030+0000 osd.0 (osd.0) 395 : cluster 0 6.18 scrub ok 2024-03-11T21:15:59.935 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:59 smithi023 bash[29146]: cluster 2024-03-11T21:15:58.880509+0000 osd.6 (osd.6) 450 : cluster 0 2.7 scrub starts 2024-03-11T21:15:59.935 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:59 smithi023 bash[29146]: cluster 2024-03-11T21:15:58.883172+0000 osd.6 (osd.6) 451 : cluster 0 2.7 scrub ok 2024-03-11T21:15:59.935 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:59 smithi023 bash[29146]: cluster 2024-03-11T21:15:59.043110+0000 osd.1 (osd.1) 306 : cluster 0 6.1d scrub starts 2024-03-11T21:15:59.935 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:15:59 smithi023 bash[29146]: cluster 2024-03-11T21:15:59.207510+0000 osd.1 (osd.1) 307 : cluster 0 6.1d scrub ok 2024-03-11T21:16:00.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:59 smithi146 bash[23809]: cluster 2024-03-11T21:15:58.468628+0000 mgr.y (mgr.24380) 930 : cluster 0 pgmap v1072: 142 pgs: 142 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T21:16:00.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:59 smithi146 bash[23809]: cluster 2024-03-11T21:15:58.636030+0000 osd.0 (osd.0) 395 : cluster 0 6.18 scrub ok 2024-03-11T21:16:00.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:59 smithi146 bash[23809]: cluster 2024-03-11T21:15:58.880509+0000 osd.6 (osd.6) 450 : cluster 0 2.7 scrub starts 2024-03-11T21:16:00.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:59 smithi146 bash[23809]: cluster 2024-03-11T21:15:58.883172+0000 osd.6 (osd.6) 451 : cluster 0 2.7 scrub ok 2024-03-11T21:16:00.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:59 smithi146 bash[23809]: cluster 2024-03-11T21:15:59.043110+0000 osd.1 (osd.1) 306 : cluster 0 6.1d scrub starts 2024-03-11T21:16:00.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:15:59 smithi146 bash[23809]: cluster 2024-03-11T21:15:59.207510+0000 osd.1 (osd.1) 307 : cluster 0 6.1d scrub ok 2024-03-11T21:16:01.754 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:01 smithi146 bash[23809]: cluster 2024-03-11T21:15:59.920720+0000 osd.5 (osd.5) 543 : cluster 0 3.1d scrub starts 2024-03-11T21:16:01.754 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:01 smithi146 bash[23809]: cluster 2024-03-11T21:15:59.925450+0000 osd.5 (osd.5) 544 : cluster 0 3.1d scrub ok 2024-03-11T21:16:01.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:01 smithi146 bash[23809]: cluster 2024-03-11T21:16:00.096529+0000 osd.7 (osd.7) 567 : cluster 0 6.3 scrub starts 2024-03-11T21:16:01.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:01 smithi146 bash[23809]: cluster 2024-03-11T21:16:00.232084+0000 osd.7 (osd.7) 568 : cluster 0 6.3 scrub ok 2024-03-11T21:16:01.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:01 smithi146 bash[23809]: cluster 2024-03-11T21:16:00.469248+0000 mgr.y (mgr.24380) 931 : cluster 0 pgmap v1073: 142 pgs: 142 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:16:01.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:01 smithi146 bash[23809]: cluster 2024-03-11T21:16:00.506356+0000 osd.0 (osd.0) 396 : cluster 0 3.1b scrub starts 2024-03-11T21:16:01.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:01 smithi146 bash[23809]: cluster 2024-03-11T21:16:00.510575+0000 osd.0 (osd.0) 397 : cluster 0 3.1b scrub ok 2024-03-11T21:16:01.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:01 smithi023 bash[29146]: cluster 2024-03-11T21:15:59.920720+0000 osd.5 (osd.5) 543 : cluster 0 3.1d scrub starts 2024-03-11T21:16:01.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:01 smithi023 bash[29146]: cluster 2024-03-11T21:15:59.925450+0000 osd.5 (osd.5) 544 : cluster 0 3.1d scrub ok 2024-03-11T21:16:01.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:01 smithi023 bash[29146]: cluster 2024-03-11T21:16:00.096529+0000 osd.7 (osd.7) 567 : cluster 0 6.3 scrub starts 2024-03-11T21:16:01.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:01 smithi023 bash[29146]: cluster 2024-03-11T21:16:00.232084+0000 osd.7 (osd.7) 568 : cluster 0 6.3 scrub ok 2024-03-11T21:16:01.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:01 smithi023 bash[29146]: cluster 2024-03-11T21:16:00.469248+0000 mgr.y (mgr.24380) 931 : cluster 0 pgmap v1073: 142 pgs: 142 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:16:01.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:01 smithi023 bash[29146]: cluster 2024-03-11T21:16:00.506356+0000 osd.0 (osd.0) 396 : cluster 0 3.1b scrub starts 2024-03-11T21:16:01.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:01 smithi023 bash[29146]: cluster 2024-03-11T21:16:00.510575+0000 osd.0 (osd.0) 397 : cluster 0 3.1b scrub ok 2024-03-11T21:16:01.776 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:01 smithi023 bash[21372]: cluster 2024-03-11T21:15:59.920720+0000 osd.5 (osd.5) 543 : cluster 0 3.1d scrub starts 2024-03-11T21:16:01.776 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:01 smithi023 bash[21372]: cluster 2024-03-11T21:15:59.925450+0000 osd.5 (osd.5) 544 : cluster 0 3.1d scrub ok 2024-03-11T21:16:01.776 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:01 smithi023 bash[21372]: cluster 2024-03-11T21:16:00.096529+0000 osd.7 (osd.7) 567 : cluster 0 6.3 scrub starts 2024-03-11T21:16:01.776 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:01 smithi023 bash[21372]: cluster 2024-03-11T21:16:00.232084+0000 osd.7 (osd.7) 568 : cluster 0 6.3 scrub ok 2024-03-11T21:16:01.776 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:01 smithi023 bash[21372]: cluster 2024-03-11T21:16:00.469248+0000 mgr.y (mgr.24380) 931 : cluster 0 pgmap v1073: 142 pgs: 142 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:16:01.776 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:01 smithi023 bash[21372]: cluster 2024-03-11T21:16:00.506356+0000 osd.0 (osd.0) 396 : cluster 0 3.1b scrub starts 2024-03-11T21:16:01.776 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:01 smithi023 bash[21372]: cluster 2024-03-11T21:16:00.510575+0000 osd.0 (osd.0) 397 : cluster 0 3.1b scrub ok 2024-03-11T21:16:01.956 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:16:02.754 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:02 smithi146 bash[23809]: cluster 2024-03-11T21:16:01.424247+0000 osd.3 (osd.3) 494 : cluster 0 2.1 scrub starts 2024-03-11T21:16:02.754 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:02 smithi146 bash[23809]: cluster 2024-03-11T21:16:01.426305+0000 osd.3 (osd.3) 495 : cluster 0 2.1 scrub ok 2024-03-11T21:16:02.754 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:02 smithi146 bash[23809]: cluster 2024-03-11T21:16:01.473303+0000 osd.0 (osd.0) 398 : cluster 0 purged_snaps scrub starts 2024-03-11T21:16:02.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:02 smithi146 bash[23809]: cluster 2024-03-11T21:16:01.473456+0000 osd.0 (osd.0) 399 : cluster 0 purged_snaps scrub ok 2024-03-11T21:16:02.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:02 smithi146 bash[23809]: cluster 2024-03-11T21:16:01.486591+0000 osd.0 (osd.0) 400 : cluster 0 4.b scrub starts 2024-03-11T21:16:02.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:02 smithi146 bash[23809]: cluster 2024-03-11T21:16:01.488656+0000 osd.0 (osd.0) 401 : cluster 0 4.b scrub ok 2024-03-11T21:16:02.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:02 smithi023 bash[21372]: cluster 2024-03-11T21:16:01.424247+0000 osd.3 (osd.3) 494 : cluster 0 2.1 scrub starts 2024-03-11T21:16:02.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:02 smithi023 bash[21372]: cluster 2024-03-11T21:16:01.426305+0000 osd.3 (osd.3) 495 : cluster 0 2.1 scrub ok 2024-03-11T21:16:02.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:02 smithi023 bash[21372]: cluster 2024-03-11T21:16:01.473303+0000 osd.0 (osd.0) 398 : cluster 0 purged_snaps scrub starts 2024-03-11T21:16:02.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:02 smithi023 bash[21372]: cluster 2024-03-11T21:16:01.473456+0000 osd.0 (osd.0) 399 : cluster 0 purged_snaps scrub ok 2024-03-11T21:16:02.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:02 smithi023 bash[21372]: cluster 2024-03-11T21:16:01.486591+0000 osd.0 (osd.0) 400 : cluster 0 4.b scrub starts 2024-03-11T21:16:02.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:02 smithi023 bash[21372]: cluster 2024-03-11T21:16:01.488656+0000 osd.0 (osd.0) 401 : cluster 0 4.b scrub ok 2024-03-11T21:16:02.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:02 smithi023 bash[29146]: cluster 2024-03-11T21:16:01.424247+0000 osd.3 (osd.3) 494 : cluster 0 2.1 scrub starts 2024-03-11T21:16:02.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:02 smithi023 bash[29146]: cluster 2024-03-11T21:16:01.426305+0000 osd.3 (osd.3) 495 : cluster 0 2.1 scrub ok 2024-03-11T21:16:02.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:02 smithi023 bash[29146]: cluster 2024-03-11T21:16:01.473303+0000 osd.0 (osd.0) 398 : cluster 0 purged_snaps scrub starts 2024-03-11T21:16:02.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:02 smithi023 bash[29146]: cluster 2024-03-11T21:16:01.473456+0000 osd.0 (osd.0) 399 : cluster 0 purged_snaps scrub ok 2024-03-11T21:16:02.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:02 smithi023 bash[29146]: cluster 2024-03-11T21:16:01.486591+0000 osd.0 (osd.0) 400 : cluster 0 4.b scrub starts 2024-03-11T21:16:02.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:02 smithi023 bash[29146]: cluster 2024-03-11T21:16:01.488656+0000 osd.0 (osd.0) 401 : cluster 0 4.b scrub ok 2024-03-11T21:16:03.360 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-11T21:16:03.363 INFO:teuthology.orchestra.run.smithi023.stderr:dumped all 2024-03-11T21:16:03.754 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:03 smithi146 bash[23809]: cluster 2024-03-11T21:16:02.456199+0000 mon.a (mon.0) 1974 : cluster 0 osdmap e439: 8 total, 8 up, 8 in 2024-03-11T21:16:03.754 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:03 smithi146 bash[23809]: cluster 2024-03-11T21:16:02.470003+0000 mgr.y (mgr.24380) 932 : cluster 0 pgmap v1075: 142 pgs: 142 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:16:03.754 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:03 smithi146 bash[23809]: cluster 2024-03-11T21:16:02.490078+0000 osd.0 (osd.0) 402 : cluster 0 3.1f scrub starts 2024-03-11T21:16:03.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:03 smithi146 bash[23809]: cluster 2024-03-11T21:16:02.493264+0000 osd.0 (osd.0) 403 : cluster 0 3.1f scrub ok 2024-03-11T21:16:03.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:03 smithi146 bash[23809]: cluster 2024-03-11T21:16:02.950788+0000 osd.1 (osd.1) 308 : cluster 0 2.a scrub starts 2024-03-11T21:16:03.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:03 smithi146 bash[23809]: cluster 2024-03-11T21:16:02.952687+0000 osd.1 (osd.1) 309 : cluster 0 2.a scrub ok 2024-03-11T21:16:03.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:03 smithi023 bash[21372]: cluster 2024-03-11T21:16:02.456199+0000 mon.a (mon.0) 1974 : cluster 0 osdmap e439: 8 total, 8 up, 8 in 2024-03-11T21:16:03.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:03 smithi023 bash[21372]: cluster 2024-03-11T21:16:02.470003+0000 mgr.y (mgr.24380) 932 : cluster 0 pgmap v1075: 142 pgs: 142 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:16:03.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:03 smithi023 bash[21372]: cluster 2024-03-11T21:16:02.490078+0000 osd.0 (osd.0) 402 : cluster 0 3.1f scrub starts 2024-03-11T21:16:03.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:03 smithi023 bash[21372]: cluster 2024-03-11T21:16:02.493264+0000 osd.0 (osd.0) 403 : cluster 0 3.1f scrub ok 2024-03-11T21:16:03.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:03 smithi023 bash[21372]: cluster 2024-03-11T21:16:02.950788+0000 osd.1 (osd.1) 308 : cluster 0 2.a scrub starts 2024-03-11T21:16:03.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:03 smithi023 bash[21372]: cluster 2024-03-11T21:16:02.952687+0000 osd.1 (osd.1) 309 : cluster 0 2.a scrub ok 2024-03-11T21:16:03.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:03 smithi023 bash[29146]: cluster 2024-03-11T21:16:02.456199+0000 mon.a (mon.0) 1974 : cluster 0 osdmap e439: 8 total, 8 up, 8 in 2024-03-11T21:16:03.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:03 smithi023 bash[29146]: cluster 2024-03-11T21:16:02.470003+0000 mgr.y (mgr.24380) 932 : cluster 0 pgmap v1075: 142 pgs: 142 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:16:03.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:03 smithi023 bash[29146]: cluster 2024-03-11T21:16:02.490078+0000 osd.0 (osd.0) 402 : cluster 0 3.1f scrub starts 2024-03-11T21:16:03.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:03 smithi023 bash[29146]: cluster 2024-03-11T21:16:02.493264+0000 osd.0 (osd.0) 403 : cluster 0 3.1f scrub ok 2024-03-11T21:16:03.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:03 smithi023 bash[29146]: cluster 2024-03-11T21:16:02.950788+0000 osd.1 (osd.1) 308 : cluster 0 2.a scrub starts 2024-03-11T21:16:03.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:03 smithi023 bash[29146]: cluster 2024-03-11T21:16:02.952687+0000 osd.1 (osd.1) 309 : cluster 0 2.a scrub ok 2024-03-11T21:16:04.754 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:04 smithi146 bash[23809]: audit 2024-03-11T21:16:03.333466+0000 mgr.y (mgr.24380) 933 : audit 0 from='client.25505 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:16:04.754 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:04 smithi146 bash[23809]: cluster 2024-03-11T21:16:03.432274+0000 osd.3 (osd.3) 496 : cluster 0 2.19 scrub starts 2024-03-11T21:16:04.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:04 smithi146 bash[23809]: cluster 2024-03-11T21:16:03.434496+0000 osd.3 (osd.3) 497 : cluster 0 2.19 scrub ok 2024-03-11T21:16:04.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:04 smithi146 bash[23809]: cluster 2024-03-11T21:16:03.525601+0000 osd.0 (osd.0) 404 : cluster 0 4.16 scrub starts 2024-03-11T21:16:04.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:04 smithi146 bash[23809]: cluster 2024-03-11T21:16:03.527140+0000 osd.0 (osd.0) 405 : cluster 0 4.16 scrub ok 2024-03-11T21:16:04.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:04 smithi023 bash[29146]: audit 2024-03-11T21:16:03.333466+0000 mgr.y (mgr.24380) 933 : audit 0 from='client.25505 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:16:04.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:04 smithi023 bash[29146]: cluster 2024-03-11T21:16:03.432274+0000 osd.3 (osd.3) 496 : cluster 0 2.19 scrub starts 2024-03-11T21:16:04.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:04 smithi023 bash[29146]: cluster 2024-03-11T21:16:03.434496+0000 osd.3 (osd.3) 497 : cluster 0 2.19 scrub ok 2024-03-11T21:16:04.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:04 smithi023 bash[29146]: cluster 2024-03-11T21:16:03.525601+0000 osd.0 (osd.0) 404 : cluster 0 4.16 scrub starts 2024-03-11T21:16:04.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:04 smithi023 bash[29146]: cluster 2024-03-11T21:16:03.527140+0000 osd.0 (osd.0) 405 : cluster 0 4.16 scrub ok 2024-03-11T21:16:04.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:04 smithi023 bash[21372]: audit 2024-03-11T21:16:03.333466+0000 mgr.y (mgr.24380) 933 : audit 0 from='client.25505 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:16:04.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:04 smithi023 bash[21372]: cluster 2024-03-11T21:16:03.432274+0000 osd.3 (osd.3) 496 : cluster 0 2.19 scrub starts 2024-03-11T21:16:04.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:04 smithi023 bash[21372]: cluster 2024-03-11T21:16:03.434496+0000 osd.3 (osd.3) 497 : cluster 0 2.19 scrub ok 2024-03-11T21:16:04.776 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:04 smithi023 bash[21372]: cluster 2024-03-11T21:16:03.525601+0000 osd.0 (osd.0) 404 : cluster 0 4.16 scrub starts 2024-03-11T21:16:04.776 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:04 smithi023 bash[21372]: cluster 2024-03-11T21:16:03.527140+0000 osd.0 (osd.0) 405 : cluster 0 4.16 scrub ok 2024-03-11T21:16:05.013 INFO:teuthology.orchestra.run.smithi023.stdout:{"pg_ready":true,"pg_map":{"version":1075,"stamp":"2024-03-11T21:16:02.469512+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596392,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":2959,"num_read_kb":2711,"num_write":1008,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":287,"num_bytes_recovered":7056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2660,"ondisk_log_size":2660,"up":426,"acting":426,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":426,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":4488340,"kb_used_data":7996,"kb_used_omap":21,"kb_used_meta":4480234,"kb_avail":745472876,"statfs":{"total":767960285184,"available":763364225024,"internally_reserved":0,"allocated":8187904,"data_stored":5565112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22089,"internal_metadata":4587760055},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003213"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":794,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:57.054259+0000","last_change":"2024-03-11T21:11:15.221894+0000","last_active":"2024-03-11T21:15:57.054259+0000","last_peered":"2024-03-11T21:15:57.054259+0000","last_clean":"2024-03-11T21:15:57.054259+0000","last_became_active":"2024-03-11T20:50:34.338390+0000","last_became_peered":"2024-03-11T20:50:34.338390+0000","last_unstale":"2024-03-11T21:15:57.054259+0000","last_undegraded":"2024-03-11T21:15:57.054259+0000","last_fullsized":"2024-03-11T21:15:57.054259+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:11:15.221783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:11:15.221783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:15:59.287309+0000","scrub_duration":127,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":810,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:57.054275+0000","last_change":"2024-03-11T21:14:50.709735+0000","last_active":"2024-03-11T21:15:57.054275+0000","last_peered":"2024-03-11T21:15:57.054275+0000","last_clean":"2024-03-11T21:15:57.054275+0000","last_became_active":"2024-03-11T21:14:50.707251+0000","last_became_peered":"2024-03-11T21:14:50.707251+0000","last_unstale":"2024-03-11T21:15:57.054275+0000","last_undegraded":"2024-03-11T21:15:57.054275+0000","last_fullsized":"2024-03-11T21:15:57.054275+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:11:54.932679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:11:54.932679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:15:59.287502+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":39,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:56.685025+0000","last_change":"2024-03-11T21:14:50.433773+0000","last_active":"2024-03-11T21:15:56.685025+0000","last_peered":"2024-03-11T21:15:56.685025+0000","last_clean":"2024-03-11T21:15:56.685025+0000","last_became_active":"2024-03-11T21:14:50.431630+0000","last_became_peered":"2024-03-11T21:14:50.431630+0000","last_unstale":"2024-03-11T21:15:56.685025+0000","last_undegraded":"2024-03-11T21:15:56.685025+0000","last_fullsized":"2024-03-11T21:15:56.685025+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:12:26.013231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:11:05.884900+0000","last_clean_scrub_stamp":"2024-03-11T21:12:26.013231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"358'30","reported_seq":858,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:57.056378+0000","last_change":"2024-03-11T21:14:47.709010+0000","last_active":"2024-03-11T21:15:57.056378+0000","last_peered":"2024-03-11T21:15:57.056378+0000","last_clean":"2024-03-11T21:15:57.056378+0000","last_became_active":"2024-03-11T20:50:30.101721+0000","last_became_peered":"2024-03-11T20:50:30.101721+0000","last_unstale":"2024-03-11T21:15:57.056378+0000","last_undegraded":"2024-03-11T21:15:57.056378+0000","last_fullsized":"2024-03-11T21:15:57.056378+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'30","last_scrub_stamp":"2024-03-11T21:14:47.708876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:14:47.708876+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":651,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:57.902443+0000","last_change":"2024-03-11T21:15:00.706025+0000","last_active":"2024-03-11T21:15:57.902443+0000","last_peered":"2024-03-11T21:15:57.902443+0000","last_clean":"2024-03-11T21:15:57.902443+0000","last_became_active":"2024-03-11T21:05:17.316714+0000","last_became_peered":"2024-03-11T21:05:17.316714+0000","last_unstale":"2024-03-11T21:15:57.902443+0000","last_undegraded":"2024-03-11T21:15:57.902443+0000","last_fullsized":"2024-03-11T21:15:57.902443+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:15:00.705906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:55.545758+0000","last_clean_scrub_stamp":"2024-03-11T21:15:00.705906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:16:13.804673+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":646,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:57.102943+0000","last_change":"2024-03-11T21:14:46.005608+0000","last_active":"2024-03-11T21:15:57.102943+0000","last_peered":"2024-03-11T21:15:57.102943+0000","last_clean":"2024-03-11T21:15:57.102943+0000","last_became_active":"2024-03-11T21:05:17.319272+0000","last_became_peered":"2024-03-11T21:05:17.319272+0000","last_unstale":"2024-03-11T21:15:57.102943+0000","last_undegraded":"2024-03-11T21:15:57.102943+0000","last_fullsized":"2024-03-11T21:15:57.102943+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:14:46.005485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:54:13.660325+0000","last_clean_scrub_stamp":"2024-03-11T21:14:46.005485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:16:01.881682+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":820,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:57.055972+0000","last_change":"2024-03-11T21:14:50.435796+0000","last_active":"2024-03-11T21:15:57.055972+0000","last_peered":"2024-03-11T21:15:57.055972+0000","last_clean":"2024-03-11T21:15:57.055972+0000","last_became_active":"2024-03-11T21:14:50.435212+0000","last_became_peered":"2024-03-11T21:14:50.435212+0000","last_unstale":"2024-03-11T21:15:57.055972+0000","last_undegraded":"2024-03-11T21:15:57.055972+0000","last_fullsized":"2024-03-11T21:15:57.055972+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:14:27.357348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:14:27.357348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"358'27","reported_seq":51,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:56.684916+0000","last_change":"2024-03-11T21:14:50.732289+0000","last_active":"2024-03-11T21:15:56.684916+0000","last_peered":"2024-03-11T21:15:56.684916+0000","last_clean":"2024-03-11T21:15:56.684916+0000","last_became_active":"2024-03-11T21:14:50.704536+0000","last_became_peered":"2024-03-11T21:14:50.704536+0000","last_unstale":"2024-03-11T21:15:56.684916+0000","last_undegraded":"2024-03-11T21:15:56.684916+0000","last_fullsized":"2024-03-11T21:15:56.684916+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'27","last_scrub_stamp":"2024-03-11T21:12:23.022901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:12:23.022901+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":839,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:57.903335+0000","last_change":"2024-03-11T21:15:02.818324+0000","last_active":"2024-03-11T21:15:57.903335+0000","last_peered":"2024-03-11T21:15:57.903335+0000","last_clean":"2024-03-11T21:15:57.903335+0000","last_became_active":"2024-03-11T21:00:48.460694+0000","last_became_peered":"2024-03-11T21:00:48.460694+0000","last_unstale":"2024-03-11T21:15:57.903335+0000","last_undegraded":"2024-03-11T21:15:57.903335+0000","last_fullsized":"2024-03-11T21:15:57.903335+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:15:02.818197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:12:45.484649+0000","last_clean_scrub_stamp":"2024-03-11T21:15:02.818197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:16:32.442369+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":623,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:57.054570+0000","last_change":"2024-03-11T21:12:30.037865+0000","last_active":"2024-03-11T21:15:57.054570+0000","last_peered":"2024-03-11T21:15:57.054570+0000","last_clean":"2024-03-11T21:15:57.054570+0000","last_became_active":"2024-03-11T21:05:17.316491+0000","last_became_peered":"2024-03-11T21:05:17.316491+0000","last_unstale":"2024-03-11T21:15:57.054570+0000","last_undegraded":"2024-03-11T21:15:57.054570+0000","last_fullsized":"2024-03-11T21:15:57.054570+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:12:30.037733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:03:36.230647+0000","last_clean_scrub_stamp":"2024-03-11T21:12:30.037733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:15:59.288586+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":847,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:58.124335+0000","last_change":"2024-03-11T21:15:06.134539+0000","last_active":"2024-03-11T21:15:58.124335+0000","last_peered":"2024-03-11T21:15:58.124335+0000","last_clean":"2024-03-11T21:15:58.124335+0000","last_became_active":"2024-03-11T21:14:50.446747+0000","last_became_peered":"2024-03-11T21:14:50.446747+0000","last_unstale":"2024-03-11T21:15:58.124335+0000","last_undegraded":"2024-03-11T21:15:58.124335+0000","last_fullsized":"2024-03-11T21:15:58.124335+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:15:06.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:15:06.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:16:29.894605+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"358'39","reported_seq":952,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:57.904390+0000","last_change":"2024-03-11T21:14:09.337950+0000","last_active":"2024-03-11T21:15:57.904390+0000","last_peered":"2024-03-11T21:15:57.904390+0000","last_clean":"2024-03-11T21:15:57.904390+0000","last_became_active":"2024-03-11T21:00:48.462943+0000","last_became_peered":"2024-03-11T21:00:48.462943+0000","last_unstale":"2024-03-11T21:15:57.904390+0000","last_undegraded":"2024-03-11T21:15:57.904390+0000","last_fullsized":"2024-03-11T21:15:57.904390+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'39","last_scrub_stamp":"2024-03-11T21:14:09.337824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:14:09.337824+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:16:02.900487+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":47,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:58.636153+0000","last_change":"2024-03-11T21:15:58.636189+0000","last_active":"2024-03-11T21:15:58.636153+0000","last_peered":"2024-03-11T21:15:58.636153+0000","last_clean":"2024-03-11T21:15:58.636153+0000","last_became_active":"2024-03-11T21:14:50.430442+0000","last_became_peered":"2024-03-11T21:14:50.430442+0000","last_unstale":"2024-03-11T21:15:58.636153+0000","last_undegraded":"2024-03-11T21:15:58.636153+0000","last_fullsized":"2024-03-11T21:15:58.636153+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:15:58.636037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:15:58.636037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:17:26.086619+0000","scrub_duration":87,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":597,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:57.054136+0000","last_change":"2024-03-11T21:14:50.437983+0000","last_active":"2024-03-11T21:15:57.054136+0000","last_peered":"2024-03-11T21:15:57.054136+0000","last_clean":"2024-03-11T21:15:57.054136+0000","last_became_active":"2024-03-11T21:14:50.437512+0000","last_became_peered":"2024-03-11T21:14:50.437512+0000","last_unstale":"2024-03-11T21:15:57.054136+0000","last_undegraded":"2024-03-11T21:15:57.054136+0000","last_fullsized":"2024-03-11T21:15:57.054136+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:12:01.877128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:53:50.599029+0000","last_clean_scrub_stamp":"2024-03-11T21:12:01.877128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:15:59.287989+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":843,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:58.123951+0000","last_change":"2024-03-11T21:14:52.238965+0000","last_active":"2024-03-11T21:15:58.123951+0000","last_peered":"2024-03-11T21:15:58.123951+0000","last_clean":"2024-03-11T21:15:58.123951+0000","last_became_active":"2024-03-11T20:50:28.068978+0000","last_became_peered":"2024-03-11T20:50:28.068978+0000","last_unstale":"2024-03-11T21:15:58.123951+0000","last_undegraded":"2024-03-11T21:15:58.123951+0000","last_fullsized":"2024-03-11T21:15:58.123951+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:14:52.238831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:54:07.155130+0000","last_clean_scrub_stamp":"2024-03-11T21:14:52.238831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:16:13.158500+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"358'29","reported_seq":889,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:59.925646+0000","last_change":"2024-03-11T21:15:59.925729+0000","last_active":"2024-03-11T21:15:59.925646+0000","last_peered":"2024-03-11T21:15:59.925646+0000","last_clean":"2024-03-11T21:15:59.925646+0000","last_became_active":"2024-03-11T21:00:27.952093+0000","last_became_peered":"2024-03-11T21:00:27.952093+0000","last_unstale":"2024-03-11T21:15:59.925646+0000","last_undegraded":"2024-03-11T21:15:59.925646+0000","last_fullsized":"2024-03-11T21:15:59.925646+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'29","last_scrub_stamp":"2024-03-11T21:15:59.925461+0000","last_deep_scrub":"110'17","last_deep_scrub_stamp":"2024-03-11T20:53:04.802488+0000","last_clean_scrub_stamp":"2024-03-11T21:15:59.925461+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:17:16.526006+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":796,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:57.056280+0000","last_change":"2024-03-11T21:12:01.082740+0000","last_active":"2024-03-11T21:15:57.056280+0000","last_peered":"2024-03-11T21:15:57.056280+0000","last_clean":"2024-03-11T21:15:57.056280+0000","last_became_active":"2024-03-11T20:50:34.334558+0000","last_became_peered":"2024-03-11T20:50:34.334558+0000","last_unstale":"2024-03-11T21:15:57.056280+0000","last_undegraded":"2024-03-11T21:15:57.056280+0000","last_fullsized":"2024-03-11T21:15:57.056280+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:12:01.082623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:55:59.892394+0000","last_clean_scrub_stamp":"2024-03-11T21:12:01.082623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:15:59.288280+0000","scrub_duration":183,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":181,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:57.054819+0000","last_change":"2024-03-11T21:12:31.955513+0000","last_active":"2024-03-11T21:15:57.054819+0000","last_peered":"2024-03-11T21:15:57.054819+0000","last_clean":"2024-03-11T21:15:57.054819+0000","last_became_active":"2024-03-11T21:10:38.441649+0000","last_became_peered":"2024-03-11T21:10:38.441649+0000","last_unstale":"2024-03-11T21:15:57.054819+0000","last_undegraded":"2024-03-11T21:15:57.054819+0000","last_fullsized":"2024-03-11T21:15:57.054819+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:12:31.955357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:12:31.955357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:15:59.288866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"77'1","reported_seq":814,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:57.054699+0000","last_change":"2024-03-11T21:11:05.190655+0000","last_active":"2024-03-11T21:15:57.054699+0000","last_peered":"2024-03-11T21:15:57.054699+0000","last_clean":"2024-03-11T21:15:57.054699+0000","last_became_active":"2024-03-11T20:50:28.068840+0000","last_became_peered":"2024-03-11T20:50:28.068840+0000","last_unstale":"2024-03-11T21:15:57.054699+0000","last_undegraded":"2024-03-11T21:15:57.054699+0000","last_fullsized":"2024-03-11T21:15:57.054699+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-03-11T21:11:05.190559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:11:05.190559+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:15:59.287132+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"357'25","reported_seq":690,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:56.688269+0000","last_change":"2024-03-11T21:12:27.010140+0000","last_active":"2024-03-11T21:15:56.688269+0000","last_peered":"2024-03-11T21:15:56.688269+0000","last_clean":"2024-03-11T21:15:56.688269+0000","last_became_active":"2024-03-11T21:05:17.443030+0000","last_became_peered":"2024-03-11T21:05:17.443030+0000","last_unstale":"2024-03-11T21:15:56.688269+0000","last_undegraded":"2024-03-11T21:15:56.688269+0000","last_fullsized":"2024-03-11T21:15:56.688269+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"357'25","last_scrub_stamp":"2024-03-11T21:12:27.010006+0000","last_deep_scrub":"207'19","last_deep_scrub_stamp":"2024-03-11T21:03:39.240089+0000","last_clean_scrub_stamp":"2024-03-11T21:12:27.010006+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":603,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:57.104749+0000","last_change":"2024-03-11T21:14:45.081293+0000","last_active":"2024-03-11T21:15:57.104749+0000","last_peered":"2024-03-11T21:15:57.104749+0000","last_clean":"2024-03-11T21:15:57.104749+0000","last_became_active":"2024-03-11T21:05:17.313986+0000","last_became_peered":"2024-03-11T21:05:17.313986+0000","last_unstale":"2024-03-11T21:15:57.104749+0000","last_undegraded":"2024-03-11T21:15:57.104749+0000","last_fullsized":"2024-03-11T21:15:57.104749+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:14:45.081205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:33.399510+0000","last_clean_scrub_stamp":"2024-03-11T21:14:45.081205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:16:01.881542+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":614,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:56.688582+0000","last_change":"2024-03-11T21:12:32.862629+0000","last_active":"2024-03-11T21:15:56.688582+0000","last_peered":"2024-03-11T21:15:56.688582+0000","last_clean":"2024-03-11T21:15:56.688582+0000","last_became_active":"2024-03-11T21:08:05.298552+0000","last_became_peered":"2024-03-11T21:08:05.298552+0000","last_unstale":"2024-03-11T21:15:56.688582+0000","last_undegraded":"2024-03-11T21:15:56.688582+0000","last_fullsized":"2024-03-11T21:15:56.688582+0000","mapping_epoch":319,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:12:32.862538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:12:32.862538+0000","last_clean_scrub_stamp":"2024-03-11T21:12:32.862538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":844,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:57.103641+0000","last_change":"2024-03-11T21:14:32.229812+0000","last_active":"2024-03-11T21:15:57.103641+0000","last_peered":"2024-03-11T21:15:57.103641+0000","last_clean":"2024-03-11T21:15:57.103641+0000","last_became_active":"2024-03-11T21:00:27.956935+0000","last_became_peered":"2024-03-11T21:00:27.956935+0000","last_unstale":"2024-03-11T21:15:57.103641+0000","last_undegraded":"2024-03-11T21:15:57.103641+0000","last_fullsized":"2024-03-11T21:15:57.103641+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:14:32.229737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:14:32.229737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:16:01.881364+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"357'9","reported_seq":53,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:16:00.510745+0000","last_change":"2024-03-11T21:16:00.510817+0000","last_active":"2024-03-11T21:16:00.510745+0000","last_peered":"2024-03-11T21:16:00.510745+0000","last_clean":"2024-03-11T21:16:00.510745+0000","last_became_active":"2024-03-11T21:14:50.430436+0000","last_became_peered":"2024-03-11T21:14:50.430436+0000","last_unstale":"2024-03-11T21:16:00.510745+0000","last_undegraded":"2024-03-11T21:16:00.510745+0000","last_fullsized":"2024-03-11T21:16:00.510745+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"357'9","last_scrub_stamp":"2024-03-11T21:16:00.510585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:16:00.510585+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:17:16.576666+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":16,"num_read_kb":12,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":529,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:56.688707+0000","last_change":"2024-03-11T21:11:20.728085+0000","last_active":"2024-03-11T21:15:56.688707+0000","last_peered":"2024-03-11T21:15:56.688707+0000","last_clean":"2024-03-11T21:15:56.688707+0000","last_became_active":"2024-03-11T21:00:48.464694+0000","last_became_peered":"2024-03-11T21:00:48.464694+0000","last_unstale":"2024-03-11T21:15:56.688707+0000","last_undegraded":"2024-03-11T21:15:56.688707+0000","last_fullsized":"2024-03-11T21:15:56.688707+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:11:20.727935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:11:20.727935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1f","version":"212'16","reported_seq":1178,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:57.104865+0000","last_change":"2024-03-11T21:14:24.332827+0000","last_active":"2024-03-11T21:15:57.104865+0000","last_peered":"2024-03-11T21:15:57.104865+0000","last_clean":"2024-03-11T21:15:57.104865+0000","last_became_active":"2024-03-11T21:00:48.464350+0000","last_became_peered":"2024-03-11T21:00:48.464350+0000","last_unstale":"2024-03-11T21:15:57.104865+0000","last_undegraded":"2024-03-11T21:15:57.104865+0000","last_fullsized":"2024-03-11T21:15:57.104865+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"212'16","last_scrub_stamp":"2024-03-11T21:14:24.332714+0000","last_deep_scrub":"212'16","last_deep_scrub_stamp":"2024-03-11T21:06:47.821221+0000","last_clean_scrub_stamp":"2024-03-11T21:14:24.332714+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:16:01.881501+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"362'2","reported_seq":820,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:57.054479+0000","last_change":"2024-03-11T21:14:50.456801+0000","last_active":"2024-03-11T21:15:57.054479+0000","last_peered":"2024-03-11T21:15:57.054479+0000","last_clean":"2024-03-11T21:15:57.054479+0000","last_became_active":"2024-03-11T21:14:50.436938+0000","last_became_peered":"2024-03-11T21:14:50.436938+0000","last_unstale":"2024-03-11T21:15:57.054479+0000","last_undegraded":"2024-03-11T21:15:57.054479+0000","last_fullsized":"2024-03-11T21:15:57.054479+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"362'2","last_scrub_stamp":"2024-03-11T21:11:03.482685+0000","last_deep_scrub":"362'2","last_deep_scrub_stamp":"2024-03-11T21:11:03.482685+0000","last_clean_scrub_stamp":"2024-03-11T21:11:03.482685+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:15:59.287014+0000","scrub_duration":335,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"357'21","reported_seq":910,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:57.053835+0000","last_change":"2024-03-11T21:14:50.464039+0000","last_active":"2024-03-11T21:15:57.053835+0000","last_peered":"2024-03-11T21:15:57.053835+0000","last_clean":"2024-03-11T21:15:57.053835+0000","last_became_active":"2024-03-11T21:14:50.437313+0000","last_became_peered":"2024-03-11T21:14:50.437313+0000","last_unstale":"2024-03-11T21:15:57.053835+0000","last_undegraded":"2024-03-11T21:15:57.053835+0000","last_fullsized":"2024-03-11T21:15:57.053835+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"357'21","last_scrub_stamp":"2024-03-11T21:12:16.917505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:12:16.917505+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:15:59.288778+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"105'2","reported_seq":826,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:58.124550+0000","last_change":"2024-03-11T21:14:25.985077+0000","last_active":"2024-03-11T21:15:58.124550+0000","last_peered":"2024-03-11T21:15:58.124550+0000","last_clean":"2024-03-11T21:15:58.124550+0000","last_became_active":"2024-03-11T20:50:34.327356+0000","last_became_peered":"2024-03-11T20:50:34.327356+0000","last_unstale":"2024-03-11T21:15:58.124550+0000","last_undegraded":"2024-03-11T21:15:58.124550+0000","last_fullsized":"2024-03-11T21:15:58.124550+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'2","last_scrub_stamp":"2024-03-11T21:14:25.984725+0000","last_deep_scrub":"105'2","last_deep_scrub_stamp":"2024-03-11T21:03:41.971918+0000","last_clean_scrub_stamp":"2024-03-11T21:14:25.984725+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:15:59.185979+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":39,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:56.684942+0000","last_change":"2024-03-11T21:14:50.432402+0000","last_active":"2024-03-11T21:15:56.684942+0000","last_peered":"2024-03-11T21:15:56.684942+0000","last_clean":"2024-03-11T21:15:56.684942+0000","last_became_active":"2024-03-11T21:14:50.431785+0000","last_became_peered":"2024-03-11T21:14:50.431785+0000","last_unstale":"2024-03-11T21:15:56.684942+0000","last_undegraded":"2024-03-11T21:15:56.684942+0000","last_fullsized":"2024-03-11T21:15:56.684942+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:14:48.161544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:06:32.968650+0000","last_clean_scrub_stamp":"2024-03-11T21:14:48.161544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:16:00.434672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":839,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:57.903102+0000","last_change":"2024-03-11T21:14:19.440511+0000","last_active":"2024-03-11T21:15:57.903102+0000","last_peered":"2024-03-11T21:15:57.903102+0000","last_clean":"2024-03-11T21:15:57.903102+0000","last_became_active":"2024-03-11T21:00:27.947614+0000","last_became_peered":"2024-03-11T21:00:27.947614+0000","last_unstale":"2024-03-11T21:15:57.903102+0000","last_undegraded":"2024-03-11T21:15:57.903102+0000","last_fullsized":"2024-03-11T21:15:57.903102+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:14:19.440393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:08:27.756036+0000","last_clean_scrub_stamp":"2024-03-11T21:14:19.440393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:16:02.901024+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"358'39","reported_seq":585,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:56.687903+0000","last_change":"2024-03-11T21:12:41.940357+0000","last_active":"2024-03-11T21:15:56.687903+0000","last_peered":"2024-03-11T21:15:56.687903+0000","last_clean":"2024-03-11T21:15:56.687903+0000","last_became_active":"2024-03-11T21:00:48.457893+0000","last_became_peered":"2024-03-11T21:00:48.457893+0000","last_unstale":"2024-03-11T21:15:56.687903+0000","last_undegraded":"2024-03-11T21:15:56.687903+0000","last_fullsized":"2024-03-11T21:15:56.687903+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'39","last_scrub_stamp":"2024-03-11T21:12:41.940134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:12:41.940134+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":814,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:57.055332+0000","last_change":"2024-03-11T21:14:50.438605+0000","last_active":"2024-03-11T21:15:57.055332+0000","last_peered":"2024-03-11T21:15:57.055332+0000","last_clean":"2024-03-11T21:15:57.055332+0000","last_became_active":"2024-03-11T21:14:50.438215+0000","last_became_peered":"2024-03-11T21:14:50.438215+0000","last_unstale":"2024-03-11T21:15:57.055332+0000","last_undegraded":"2024-03-11T21:15:57.055332+0000","last_fullsized":"2024-03-11T21:15:57.055332+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:14:41.715491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:52:11.462480+0000","last_clean_scrub_stamp":"2024-03-11T21:14:41.715491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":806,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:57.056083+0000","last_change":"2024-03-11T21:14:40.631379+0000","last_active":"2024-03-11T21:15:57.056083+0000","last_peered":"2024-03-11T21:15:57.056083+0000","last_clean":"2024-03-11T21:15:57.056083+0000","last_became_active":"2024-03-11T20:50:32.646323+0000","last_became_peered":"2024-03-11T20:50:32.646323+0000","last_unstale":"2024-03-11T21:15:57.056083+0000","last_undegraded":"2024-03-11T21:15:57.056083+0000","last_fullsized":"2024-03-11T21:15:57.056083+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:14:40.631052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:10:02.099653+0000","last_clean_scrub_stamp":"2024-03-11T21:14:40.631052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:16:06.108657+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":859,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:57.104438+0000","last_change":"2024-03-11T21:14:56.962742+0000","last_active":"2024-03-11T21:15:57.104438+0000","last_peered":"2024-03-11T21:15:57.104438+0000","last_clean":"2024-03-11T21:15:57.104438+0000","last_became_active":"2024-03-11T20:50:28.086513+0000","last_became_peered":"2024-03-11T20:50:28.086513+0000","last_unstale":"2024-03-11T21:15:57.104438+0000","last_undegraded":"2024-03-11T21:15:57.104438+0000","last_fullsized":"2024-03-11T21:15:57.104438+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:14:56.962561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:14:56.962561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:16:25.762485+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"357'25","reported_seq":906,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:57.904261+0000","last_change":"2024-03-11T21:14:22.394537+0000","last_active":"2024-03-11T21:15:57.904261+0000","last_peered":"2024-03-11T21:15:57.904261+0000","last_clean":"2024-03-11T21:15:57.904261+0000","last_became_active":"2024-03-11T21:00:48.464707+0000","last_became_peered":"2024-03-11T21:00:48.464707+0000","last_unstale":"2024-03-11T21:15:57.904261+0000","last_undegraded":"2024-03-11T21:15:57.904261+0000","last_fullsized":"2024-03-11T21:15:57.904261+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"357'25","last_scrub_stamp":"2024-03-11T21:14:22.394398+0000","last_deep_scrub":"145'13","last_deep_scrub_stamp":"2024-03-11T20:55:53.560287+0000","last_clean_scrub_stamp":"2024-03-11T21:14:22.394398+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:16:02.900977+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":824,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:58.124034+0000","last_change":"2024-03-11T21:12:24.070009+0000","last_active":"2024-03-11T21:15:58.124034+0000","last_peered":"2024-03-11T21:15:58.124034+0000","last_clean":"2024-03-11T21:15:58.124034+0000","last_became_active":"2024-03-11T21:00:27.973405+0000","last_became_peered":"2024-03-11T21:00:27.973405+0000","last_unstale":"2024-03-11T21:15:58.124034+0000","last_undegraded":"2024-03-11T21:15:58.124034+0000","last_fullsized":"2024-03-11T21:15:58.124034+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:12:24.069900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:12:24.069900+0000","last_clean_scrub_stamp":"2024-03-11T21:12:24.069900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:15:59.185477+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":811,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:57.053971+0000","last_change":"2024-03-11T21:12:03.903669+0000","last_active":"2024-03-11T21:15:57.053971+0000","last_peered":"2024-03-11T21:15:57.053971+0000","last_clean":"2024-03-11T21:15:57.053971+0000","last_became_active":"2024-03-11T21:00:48.472812+0000","last_became_peered":"2024-03-11T21:00:48.472812+0000","last_unstale":"2024-03-11T21:15:57.053971+0000","last_undegraded":"2024-03-11T21:15:57.053971+0000","last_fullsized":"2024-03-11T21:15:57.053971+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:12:03.903559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:12:03.903559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:15:59.287693+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":838,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:57.904175+0000","last_change":"2024-03-11T21:14:30.370952+0000","last_active":"2024-03-11T21:15:57.904175+0000","last_peered":"2024-03-11T21:15:57.904175+0000","last_clean":"2024-03-11T21:15:57.904175+0000","last_became_active":"2024-03-11T20:50:28.075760+0000","last_became_peered":"2024-03-11T20:50:28.075760+0000","last_unstale":"2024-03-11T21:15:57.904175+0000","last_undegraded":"2024-03-11T21:15:57.904175+0000","last_fullsized":"2024-03-11T21:15:57.904175+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:14:30.370879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:58:55.977289+0000","last_clean_scrub_stamp":"2024-03-11T21:14:30.370879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:16:02.900777+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"358'18","reported_seq":47,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:56.684892+0000","last_change":"2024-03-11T21:14:50.444579+0000","last_active":"2024-03-11T21:15:56.684892+0000","last_peered":"2024-03-11T21:15:56.684892+0000","last_clean":"2024-03-11T21:15:56.684892+0000","last_became_active":"2024-03-11T21:14:50.427230+0000","last_became_peered":"2024-03-11T21:14:50.427230+0000","last_unstale":"2024-03-11T21:15:56.684892+0000","last_undegraded":"2024-03-11T21:15:56.684892+0000","last_fullsized":"2024-03-11T21:15:56.684892+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'18","last_scrub_stamp":"2024-03-11T21:12:32.003163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:12:32.003163+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":24,"num_read_kb":21,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.11","version":"186'2","reported_seq":824,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:57.053875+0000","last_change":"2024-03-11T21:14:50.442973+0000","last_active":"2024-03-11T21:15:57.053875+0000","last_peered":"2024-03-11T21:15:57.053875+0000","last_clean":"2024-03-11T21:15:57.053875+0000","last_became_active":"2024-03-11T21:14:50.436281+0000","last_became_peered":"2024-03-11T21:14:50.436281+0000","last_unstale":"2024-03-11T21:15:57.053875+0000","last_undegraded":"2024-03-11T21:15:57.053875+0000","last_fullsized":"2024-03-11T21:15:57.053875+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"186'2","last_scrub_stamp":"2024-03-11T21:14:42.714318+0000","last_deep_scrub":"186'2","last_deep_scrub_stamp":"2024-03-11T20:59:04.534850+0000","last_clean_scrub_stamp":"2024-03-11T21:14:42.714318+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:15:59.289371+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":654,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:57.103709+0000","last_change":"2024-03-11T21:14:53.059102+0000","last_active":"2024-03-11T21:15:57.103709+0000","last_peered":"2024-03-11T21:15:57.103709+0000","last_clean":"2024-03-11T21:15:57.103709+0000","last_became_active":"2024-03-11T21:05:17.314174+0000","last_became_peered":"2024-03-11T21:05:17.314174+0000","last_unstale":"2024-03-11T21:15:57.103709+0000","last_undegraded":"2024-03-11T21:15:57.103709+0000","last_fullsized":"2024-03-11T21:15:57.103709+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:14:53.058986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:14:53.058986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:16:01.866113+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":541,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:56.688575+0000","last_change":"2024-03-11T21:14:50.419909+0000","last_active":"2024-03-11T21:15:56.688575+0000","last_peered":"2024-03-11T21:15:56.688575+0000","last_clean":"2024-03-11T21:15:56.688575+0000","last_became_active":"2024-03-11T21:14:50.419556+0000","last_became_peered":"2024-03-11T21:14:50.419556+0000","last_unstale":"2024-03-11T21:15:56.688575+0000","last_undegraded":"2024-03-11T21:15:56.688575+0000","last_fullsized":"2024-03-11T21:15:56.688575+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:12:09.798977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:12:09.798977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"358'35","reported_seq":631,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:58.122595+0000","last_change":"2024-03-11T21:14:28.012591+0000","last_active":"2024-03-11T21:15:58.122595+0000","last_peered":"2024-03-11T21:15:58.122595+0000","last_clean":"2024-03-11T21:15:58.122595+0000","last_became_active":"2024-03-11T21:05:17.314507+0000","last_became_peered":"2024-03-11T21:05:17.314507+0000","last_unstale":"2024-03-11T21:15:58.122595+0000","last_undegraded":"2024-03-11T21:15:58.122595+0000","last_fullsized":"2024-03-11T21:15:58.122595+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'35","last_scrub_stamp":"2024-03-11T21:14:28.012406+0000","last_deep_scrub":"161'15","last_deep_scrub_stamp":"2024-03-11T20:57:15.125548+0000","last_clean_scrub_stamp":"2024-03-11T21:14:28.012406+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:15:59.186378+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":39,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:56.684185+0000","last_change":"2024-03-11T21:14:50.420476+0000","last_active":"2024-03-11T21:15:56.684185+0000","last_peered":"2024-03-11T21:15:56.684185+0000","last_clean":"2024-03-11T21:15:56.684185+0000","last_became_active":"2024-03-11T21:14:50.420063+0000","last_became_peered":"2024-03-11T21:14:50.420063+0000","last_unstale":"2024-03-11T21:15:56.684185+0000","last_undegraded":"2024-03-11T21:15:56.684185+0000","last_fullsized":"2024-03-11T21:15:56.684185+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:14:41.641017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:14:41.641017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:16:02.909553+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":537,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:56.688098+0000","last_change":"2024-03-11T21:12:50.895561+0000","last_active":"2024-03-11T21:15:56.688098+0000","last_peered":"2024-03-11T21:15:56.688098+0000","last_clean":"2024-03-11T21:15:56.688098+0000","last_became_active":"2024-03-11T21:00:48.459043+0000","last_became_peered":"2024-03-11T21:00:48.459043+0000","last_unstale":"2024-03-11T21:15:56.688098+0000","last_undegraded":"2024-03-11T21:15:56.688098+0000","last_fullsized":"2024-03-11T21:15:56.688098+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:12:50.895379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:12:50.895379+0000","last_clean_scrub_stamp":"2024-03-11T21:12:50.895379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":851,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:57.103824+0000","last_change":"2024-03-11T21:14:08.074400+0000","last_active":"2024-03-11T21:15:57.103824+0000","last_peered":"2024-03-11T21:15:57.103824+0000","last_clean":"2024-03-11T21:15:57.103824+0000","last_became_active":"2024-03-11T20:50:28.082737+0000","last_became_peered":"2024-03-11T20:50:28.082737+0000","last_unstale":"2024-03-11T21:15:57.103824+0000","last_undegraded":"2024-03-11T21:15:57.103824+0000","last_fullsized":"2024-03-11T21:15:57.103824+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:14:08.074267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:28.404997+0000","last_clean_scrub_stamp":"2024-03-11T21:14:08.074267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:16:01.881162+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"358'21","reported_seq":866,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:58.124611+0000","last_change":"2024-03-11T21:12:48.118469+0000","last_active":"2024-03-11T21:15:58.124611+0000","last_peered":"2024-03-11T21:15:58.124611+0000","last_clean":"2024-03-11T21:15:58.124611+0000","last_became_active":"2024-03-11T21:00:27.967624+0000","last_became_peered":"2024-03-11T21:00:27.967624+0000","last_unstale":"2024-03-11T21:15:58.124611+0000","last_undegraded":"2024-03-11T21:15:58.124611+0000","last_fullsized":"2024-03-11T21:15:58.124611+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'21","last_scrub_stamp":"2024-03-11T21:12:48.118343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:12:48.118343+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:15:59.185585+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":489,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:57.902558+0000","last_change":"2024-03-11T21:14:45.728377+0000","last_active":"2024-03-11T21:15:57.902558+0000","last_peered":"2024-03-11T21:15:57.902558+0000","last_clean":"2024-03-11T21:15:57.902558+0000","last_became_active":"2024-03-11T21:08:05.299913+0000","last_became_peered":"2024-03-11T21:08:05.299913+0000","last_unstale":"2024-03-11T21:15:57.902558+0000","last_undegraded":"2024-03-11T21:15:57.902558+0000","last_fullsized":"2024-03-11T21:15:57.902558+0000","mapping_epoch":319,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:14:45.728255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:57:16.032302+0000","last_clean_scrub_stamp":"2024-03-11T21:14:45.728255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:16:02.900901+0000","scrub_duration":179,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"112'12","reported_seq":1140,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:57.904662+0000","last_change":"2024-03-11T21:12:36.443795+0000","last_active":"2024-03-11T21:15:57.904662+0000","last_peered":"2024-03-11T21:15:57.904662+0000","last_clean":"2024-03-11T21:15:57.904662+0000","last_became_active":"2024-03-11T20:50:32.650064+0000","last_became_peered":"2024-03-11T20:50:32.650064+0000","last_unstale":"2024-03-11T21:15:57.904662+0000","last_undegraded":"2024-03-11T21:15:57.904662+0000","last_fullsized":"2024-03-11T21:15:57.904662+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'12","last_scrub_stamp":"2024-03-11T21:12:36.443649+0000","last_deep_scrub":"112'12","last_deep_scrub_stamp":"2024-03-11T20:53:11.843951+0000","last_clean_scrub_stamp":"2024-03-11T21:12:36.443649+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:16:02.900330+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":39,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:56.684508+0000","last_change":"2024-03-11T21:14:50.434421+0000","last_active":"2024-03-11T21:15:56.684508+0000","last_peered":"2024-03-11T21:15:56.684508+0000","last_clean":"2024-03-11T21:15:56.684508+0000","last_became_active":"2024-03-11T21:14:50.434124+0000","last_became_peered":"2024-03-11T21:14:50.434124+0000","last_unstale":"2024-03-11T21:15:56.684508+0000","last_undegraded":"2024-03-11T21:15:56.684508+0000","last_fullsized":"2024-03-11T21:15:56.684508+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:12:45.881111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:06:52.862971+0000","last_clean_scrub_stamp":"2024-03-11T21:12:45.881111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"357'25","reported_seq":49,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:56.684453+0000","last_change":"2024-03-11T21:14:50.461865+0000","last_active":"2024-03-11T21:15:56.684453+0000","last_peered":"2024-03-11T21:15:56.684453+0000","last_clean":"2024-03-11T21:15:56.684453+0000","last_became_active":"2024-03-11T21:14:50.434105+0000","last_became_peered":"2024-03-11T21:14:50.434105+0000","last_unstale":"2024-03-11T21:15:56.684453+0000","last_undegraded":"2024-03-11T21:15:56.684453+0000","last_fullsized":"2024-03-11T21:15:56.684453+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"357'25","last_scrub_stamp":"2024-03-11T21:12:27.006374+0000","last_deep_scrub":"119'13","last_deep_scrub_stamp":"2024-03-11T20:54:15.443472+0000","last_clean_scrub_stamp":"2024-03-11T21:12:27.006374+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":39,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:56.684954+0000","last_change":"2024-03-11T21:14:50.429220+0000","last_active":"2024-03-11T21:15:56.684954+0000","last_peered":"2024-03-11T21:15:56.684954+0000","last_clean":"2024-03-11T21:15:56.684954+0000","last_became_active":"2024-03-11T21:14:50.428898+0000","last_became_peered":"2024-03-11T21:14:50.428898+0000","last_unstale":"2024-03-11T21:15:56.684954+0000","last_undegraded":"2024-03-11T21:15:56.684954+0000","last_fullsized":"2024-03-11T21:15:56.684954+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:12:05.248307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:12:05.248307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"212'17","reported_seq":1147,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:57.056394+0000","last_change":"2024-03-11T21:12:13.934238+0000","last_active":"2024-03-11T21:15:57.056394+0000","last_peered":"2024-03-11T21:15:57.056394+0000","last_clean":"2024-03-11T21:15:57.056394+0000","last_became_active":"2024-03-11T21:00:48.467116+0000","last_became_peered":"2024-03-11T21:00:48.467116+0000","last_unstale":"2024-03-11T21:15:57.056394+0000","last_undegraded":"2024-03-11T21:15:57.056394+0000","last_fullsized":"2024-03-11T21:15:57.056394+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"212'17","last_scrub_stamp":"2024-03-11T21:12:13.934164+0000","last_deep_scrub":"110'13","last_deep_scrub_stamp":"2024-03-11T20:53:00.434532+0000","last_clean_scrub_stamp":"2024-03-11T21:12:13.934164+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":846,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:57.903821+0000","last_change":"2024-03-11T21:15:05.725868+0000","last_active":"2024-03-11T21:15:57.903821+0000","last_peered":"2024-03-11T21:15:57.903821+0000","last_clean":"2024-03-11T21:15:57.903821+0000","last_became_active":"2024-03-11T20:50:28.070325+0000","last_became_peered":"2024-03-11T20:50:28.070325+0000","last_unstale":"2024-03-11T21:15:57.903821+0000","last_undegraded":"2024-03-11T21:15:57.903821+0000","last_fullsized":"2024-03-11T21:15:57.903821+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:15:05.725652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:12:39.708033+0000","last_clean_scrub_stamp":"2024-03-11T21:15:05.725652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:16:29.768194+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"358'32","reported_seq":880,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:58.124020+0000","last_change":"2024-03-11T21:12:43.060210+0000","last_active":"2024-03-11T21:15:58.124020+0000","last_peered":"2024-03-11T21:15:58.124020+0000","last_clean":"2024-03-11T21:15:58.124020+0000","last_became_active":"2024-03-11T21:00:27.967078+0000","last_became_peered":"2024-03-11T21:00:27.967078+0000","last_unstale":"2024-03-11T21:15:58.124020+0000","last_undegraded":"2024-03-11T21:15:58.124020+0000","last_fullsized":"2024-03-11T21:15:58.124020+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'32","last_scrub_stamp":"2024-03-11T21:12:43.060110+0000","last_deep_scrub":"116'16","last_deep_scrub_stamp":"2024-03-11T20:53:39.292118+0000","last_clean_scrub_stamp":"2024-03-11T21:12:43.060110+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:15:59.185686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":816,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:58.123775+0000","last_change":"2024-03-11T21:12:27.179524+0000","last_active":"2024-03-11T21:15:58.123775+0000","last_peered":"2024-03-11T21:15:58.123775+0000","last_clean":"2024-03-11T21:15:58.123775+0000","last_became_active":"2024-03-11T21:00:27.966997+0000","last_became_peered":"2024-03-11T21:00:27.966997+0000","last_unstale":"2024-03-11T21:15:58.123775+0000","last_undegraded":"2024-03-11T21:15:58.123775+0000","last_fullsized":"2024-03-11T21:15:58.123775+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:12:27.179425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:11:02.899468+0000","last_clean_scrub_stamp":"2024-03-11T21:12:27.179425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:15:59.185285+0000","scrub_duration":59,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":819,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:57.056378+0000","last_change":"2024-03-11T21:12:32.947079+0000","last_active":"2024-03-11T21:15:57.056378+0000","last_peered":"2024-03-11T21:15:57.056378+0000","last_clean":"2024-03-11T21:15:57.056378+0000","last_became_active":"2024-03-11T21:00:48.468386+0000","last_became_peered":"2024-03-11T21:00:48.468386+0000","last_unstale":"2024-03-11T21:15:57.056378+0000","last_undegraded":"2024-03-11T21:15:57.056378+0000","last_fullsized":"2024-03-11T21:15:57.056378+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:12:32.946959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:12:32.946959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:15:59.288672+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"166'2","reported_seq":878,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:57.102959+0000","last_change":"2024-03-11T21:14:13.159243+0000","last_active":"2024-03-11T21:15:57.102959+0000","last_peered":"2024-03-11T21:15:57.102959+0000","last_clean":"2024-03-11T21:15:57.102959+0000","last_became_active":"2024-03-11T21:00:48.466071+0000","last_became_peered":"2024-03-11T21:00:48.466071+0000","last_unstale":"2024-03-11T21:15:57.102959+0000","last_undegraded":"2024-03-11T21:15:57.102959+0000","last_fullsized":"2024-03-11T21:15:57.102959+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'2","last_scrub_stamp":"2024-03-11T21:14:13.149583+0000","last_deep_scrub":"166'2","last_deep_scrub_stamp":"2024-03-11T20:57:39.868776+0000","last_clean_scrub_stamp":"2024-03-11T21:14:13.149583+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:16:01.881323+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1822,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"364'14","reported_seq":883,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:57.102719+0000","last_change":"2024-03-11T21:14:54.058932+0000","last_active":"2024-03-11T21:15:57.102719+0000","last_peered":"2024-03-11T21:15:57.102719+0000","last_clean":"2024-03-11T21:15:57.102719+0000","last_became_active":"2024-03-11T21:14:50.420602+0000","last_became_peered":"2024-03-11T21:14:50.420602+0000","last_unstale":"2024-03-11T21:15:57.102719+0000","last_undegraded":"2024-03-11T21:15:57.102719+0000","last_fullsized":"2024-03-11T21:15:57.102719+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"364'14","last_scrub_stamp":"2024-03-11T21:14:54.058804+0000","last_deep_scrub":"364'14","last_deep_scrub_stamp":"2024-03-11T21:11:15.819245+0000","last_clean_scrub_stamp":"2024-03-11T21:14:54.058804+0000","objects_scrubbed":2,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:16:17.359804+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":16,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":579,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:58.122398+0000","last_change":"2024-03-11T21:12:19.082499+0000","last_active":"2024-03-11T21:15:58.122398+0000","last_peered":"2024-03-11T21:15:58.122398+0000","last_clean":"2024-03-11T21:15:58.122398+0000","last_became_active":"2024-03-11T21:08:05.296676+0000","last_became_peered":"2024-03-11T21:08:05.296676+0000","last_unstale":"2024-03-11T21:15:58.122398+0000","last_undegraded":"2024-03-11T21:15:58.122398+0000","last_fullsized":"2024-03-11T21:15:58.122398+0000","mapping_epoch":319,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:12:19.082419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:12:19.082419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:15:59.185089+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":39,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:56.685189+0000","last_change":"2024-03-11T21:14:50.708892+0000","last_active":"2024-03-11T21:15:56.685189+0000","last_peered":"2024-03-11T21:15:56.685189+0000","last_clean":"2024-03-11T21:15:56.685189+0000","last_became_active":"2024-03-11T21:14:50.708563+0000","last_became_peered":"2024-03-11T21:14:50.708563+0000","last_unstale":"2024-03-11T21:15:56.685189+0000","last_undegraded":"2024-03-11T21:15:56.685189+0000","last_fullsized":"2024-03-11T21:15:56.685189+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:12:24.000498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:54:09.537972+0000","last_clean_scrub_stamp":"2024-03-11T21:12:24.000498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":614,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:56.687743+0000","last_change":"2024-03-11T21:14:50.702434+0000","last_active":"2024-03-11T21:15:56.687743+0000","last_peered":"2024-03-11T21:15:56.687743+0000","last_clean":"2024-03-11T21:15:56.687743+0000","last_became_active":"2024-03-11T21:14:50.701560+0000","last_became_peered":"2024-03-11T21:14:50.701560+0000","last_unstale":"2024-03-11T21:15:56.687743+0000","last_undegraded":"2024-03-11T21:15:56.687743+0000","last_fullsized":"2024-03-11T21:15:56.687743+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:12:15.742332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:18.682947+0000","last_clean_scrub_stamp":"2024-03-11T21:12:15.742332+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"358'32","reported_seq":880,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:58.123037+0000","last_change":"2024-03-11T21:15:05.131380+0000","last_active":"2024-03-11T21:15:58.123037+0000","last_peered":"2024-03-11T21:15:58.123037+0000","last_clean":"2024-03-11T21:15:58.123037+0000","last_became_active":"2024-03-11T21:00:27.970878+0000","last_became_peered":"2024-03-11T21:00:27.970878+0000","last_unstale":"2024-03-11T21:15:58.123037+0000","last_undegraded":"2024-03-11T21:15:58.123037+0000","last_fullsized":"2024-03-11T21:15:58.123037+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'32","last_scrub_stamp":"2024-03-11T21:15:05.131148+0000","last_deep_scrub":"262'24","last_deep_scrub_stamp":"2024-03-11T21:03:28.128719+0000","last_clean_scrub_stamp":"2024-03-11T21:15:05.131148+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:16:29.338505+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":824,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:57.055094+0000","last_change":"2024-03-11T21:14:32.547043+0000","last_active":"2024-03-11T21:15:57.055094+0000","last_peered":"2024-03-11T21:15:57.055094+0000","last_clean":"2024-03-11T21:15:57.055094+0000","last_became_active":"2024-03-11T21:00:48.466853+0000","last_became_peered":"2024-03-11T21:00:48.466853+0000","last_unstale":"2024-03-11T21:15:57.055094+0000","last_undegraded":"2024-03-11T21:15:57.055094+0000","last_fullsized":"2024-03-11T21:15:57.055094+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:14:32.546846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:52:53.450885+0000","last_clean_scrub_stamp":"2024-03-11T21:14:32.546846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":219,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"77'2","reported_seq":45,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:56.684986+0000","last_change":"2024-03-11T21:14:50.464213+0000","last_active":"2024-03-11T21:15:56.684986+0000","last_peered":"2024-03-11T21:15:56.684986+0000","last_clean":"2024-03-11T21:15:56.684986+0000","last_became_active":"2024-03-11T21:14:50.431939+0000","last_became_peered":"2024-03-11T21:14:50.431939+0000","last_unstale":"2024-03-11T21:15:56.684986+0000","last_undegraded":"2024-03-11T21:15:56.684986+0000","last_fullsized":"2024-03-11T21:15:56.684986+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'2","last_scrub_stamp":"2024-03-11T21:12:44.904275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:12:44.904275+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"358'32","reported_seq":941,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:58.125201+0000","last_change":"2024-03-11T21:15:02.224193+0000","last_active":"2024-03-11T21:15:58.125201+0000","last_peered":"2024-03-11T21:15:58.125201+0000","last_clean":"2024-03-11T21:15:58.125201+0000","last_became_active":"2024-03-11T21:00:48.465254+0000","last_became_peered":"2024-03-11T21:00:48.465254+0000","last_unstale":"2024-03-11T21:15:58.125201+0000","last_undegraded":"2024-03-11T21:15:58.125201+0000","last_fullsized":"2024-03-11T21:15:58.125201+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'32","last_scrub_stamp":"2024-03-11T21:15:02.224034+0000","last_deep_scrub":"100'16","last_deep_scrub_stamp":"2024-03-11T20:56:08.036331+0000","last_clean_scrub_stamp":"2024-03-11T21:15:02.224034+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:16:19.312372+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"286'18","reported_seq":868,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:56.687725+0000","last_change":"2024-03-11T21:12:14.753256+0000","last_active":"2024-03-11T21:15:56.687725+0000","last_peered":"2024-03-11T21:15:56.687725+0000","last_clean":"2024-03-11T21:15:56.687725+0000","last_became_active":"2024-03-11T21:05:17.317915+0000","last_became_peered":"2024-03-11T21:05:17.317915+0000","last_unstale":"2024-03-11T21:15:56.687725+0000","last_undegraded":"2024-03-11T21:15:56.687725+0000","last_fullsized":"2024-03-11T21:15:56.687725+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"286'18","last_scrub_stamp":"2024-03-11T21:12:14.753092+0000","last_deep_scrub":"212'17","last_deep_scrub_stamp":"2024-03-11T21:03:38.258850+0000","last_clean_scrub_stamp":"2024-03-11T21:12:14.753092+0000","objects_scrubbed":1,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"184'130","reported_seq":690,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:57.104430+0000","last_change":"2024-03-11T21:14:02.114980+0000","last_active":"2024-03-11T21:15:57.104430+0000","last_peered":"2024-03-11T21:15:57.104430+0000","last_clean":"2024-03-11T21:15:57.104430+0000","last_became_active":"2024-03-11T21:11:28.393886+0000","last_became_peered":"2024-03-11T21:11:28.393886+0000","last_unstale":"2024-03-11T21:15:57.104430+0000","last_undegraded":"2024-03-11T21:15:57.104430+0000","last_fullsized":"2024-03-11T21:15:57.104430+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":3,"last_scrub":"184'130","last_scrub_stamp":"2024-03-11T21:14:02.114865+0000","last_deep_scrub":"184'130","last_deep_scrub_stamp":"2024-03-11T20:58:56.415267+0000","last_clean_scrub_stamp":"2024-03-11T21:14:02.114865+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:16:01.881222+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":39,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:56.684302+0000","last_change":"2024-03-11T21:14:50.683872+0000","last_active":"2024-03-11T21:15:56.684302+0000","last_peered":"2024-03-11T21:15:56.684302+0000","last_clean":"2024-03-11T21:15:56.684302+0000","last_became_active":"2024-03-11T21:14:50.683519+0000","last_became_peered":"2024-03-11T21:14:50.683519+0000","last_unstale":"2024-03-11T21:15:56.684302+0000","last_undegraded":"2024-03-11T21:15:56.684302+0000","last_fullsized":"2024-03-11T21:15:56.684302+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:12:41.214079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:57:21.568400+0000","last_clean_scrub_stamp":"2024-03-11T21:12:41.214079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":550,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:56.688840+0000","last_change":"2024-03-11T21:14:50.421646+0000","last_active":"2024-03-11T21:15:56.688840+0000","last_peered":"2024-03-11T21:15:56.688840+0000","last_clean":"2024-03-11T21:15:56.688840+0000","last_became_active":"2024-03-11T21:14:50.421319+0000","last_became_peered":"2024-03-11T21:14:50.421319+0000","last_unstale":"2024-03-11T21:15:56.688840+0000","last_undegraded":"2024-03-11T21:15:56.688840+0000","last_fullsized":"2024-03-11T21:15:56.688840+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:12:43.931266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:20.096874+0000","last_clean_scrub_stamp":"2024-03-11T21:12:43.931266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"358'40","reported_seq":1201,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:57.903904+0000","last_change":"2024-03-11T21:14:12.419360+0000","last_active":"2024-03-11T21:15:57.903904+0000","last_peered":"2024-03-11T21:15:57.903904+0000","last_clean":"2024-03-11T21:15:57.903904+0000","last_became_active":"2024-03-11T20:50:30.094977+0000","last_became_peered":"2024-03-11T20:50:30.094977+0000","last_unstale":"2024-03-11T21:15:57.903904+0000","last_undegraded":"2024-03-11T21:15:57.903904+0000","last_fullsized":"2024-03-11T21:15:57.903904+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'40","last_scrub_stamp":"2024-03-11T21:14:12.419197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:14:12.419197+0000","objects_scrubbed":8,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:16:02.900731+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":61,"num_read_kb":51,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":538,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:56.687936+0000","last_change":"2024-03-11T21:14:50.664203+0000","last_active":"2024-03-11T21:15:56.687936+0000","last_peered":"2024-03-11T21:15:56.687936+0000","last_clean":"2024-03-11T21:15:56.687936+0000","last_became_active":"2024-03-11T21:00:48.457901+0000","last_became_peered":"2024-03-11T21:00:48.457901+0000","last_unstale":"2024-03-11T21:15:56.687936+0000","last_undegraded":"2024-03-11T21:15:56.687936+0000","last_fullsized":"2024-03-11T21:15:56.687936+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:14:50.664081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:52:10.160202+0000","last_clean_scrub_stamp":"2024-03-11T21:14:50.664081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:16:14.300074+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"184'130","reported_seq":704,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:57.902684+0000","last_change":"2024-03-11T21:14:53.438624+0000","last_active":"2024-03-11T21:15:57.902684+0000","last_peered":"2024-03-11T21:15:57.902684+0000","last_clean":"2024-03-11T21:15:57.902684+0000","last_became_active":"2024-03-11T21:14:53.438231+0000","last_became_peered":"2024-03-11T21:14:53.438231+0000","last_unstale":"2024-03-11T21:15:57.902684+0000","last_undegraded":"2024-03-11T21:15:57.902684+0000","last_fullsized":"2024-03-11T21:15:57.902684+0000","mapping_epoch":426,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":427,"parent":"0.0","parent_split_bits":4,"last_scrub":"184'130","last_scrub_stamp":"2024-03-11T21:14:00.274131+0000","last_deep_scrub":"184'130","last_deep_scrub_stamp":"2024-03-11T20:58:56.415267+0000","last_clean_scrub_stamp":"2024-03-11T21:14:00.274131+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:16:02.900578+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":804,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:57.056100+0000","last_change":"2024-03-11T21:14:28.402647+0000","last_active":"2024-03-11T21:15:57.056100+0000","last_peered":"2024-03-11T21:15:57.056100+0000","last_clean":"2024-03-11T21:15:57.056100+0000","last_became_active":"2024-03-11T21:00:27.959408+0000","last_became_peered":"2024-03-11T21:00:27.959408+0000","last_unstale":"2024-03-11T21:15:57.056100+0000","last_undegraded":"2024-03-11T21:15:57.056100+0000","last_fullsized":"2024-03-11T21:15:57.056100+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:14:28.402497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:14:28.402497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":853,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:57.904009+0000","last_change":"2024-03-11T21:14:13.439997+0000","last_active":"2024-03-11T21:15:57.904009+0000","last_peered":"2024-03-11T21:15:57.904009+0000","last_clean":"2024-03-11T21:15:57.904009+0000","last_became_active":"2024-03-11T21:00:48.471212+0000","last_became_peered":"2024-03-11T21:00:48.471212+0000","last_unstale":"2024-03-11T21:15:57.904009+0000","last_undegraded":"2024-03-11T21:15:57.904009+0000","last_fullsized":"2024-03-11T21:15:57.904009+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:14:13.439921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:09:55.786349+0000","last_clean_scrub_stamp":"2024-03-11T21:14:13.439921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:16:02.900529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"358'47","reported_seq":57,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:56.684601+0000","last_change":"2024-03-11T21:14:50.451736+0000","last_active":"2024-03-11T21:15:56.684601+0000","last_peered":"2024-03-11T21:15:56.684601+0000","last_clean":"2024-03-11T21:15:56.684601+0000","last_became_active":"2024-03-11T21:14:50.418097+0000","last_became_peered":"2024-03-11T21:14:50.418097+0000","last_unstale":"2024-03-11T21:15:56.684601+0000","last_undegraded":"2024-03-11T21:15:56.684601+0000","last_fullsized":"2024-03-11T21:15:56.684601+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'47","last_scrub_stamp":"2024-03-11T21:14:04.199762+0000","last_deep_scrub":"182'27","last_deep_scrub_stamp":"2024-03-11T20:58:52.058851+0000","last_clean_scrub_stamp":"2024-03-11T21:14:04.199762+0000","objects_scrubbed":8,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":67,"num_read_kb":53,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":530,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:56.688270+0000","last_change":"2024-03-11T21:12:01.754493+0000","last_active":"2024-03-11T21:15:56.688270+0000","last_peered":"2024-03-11T21:15:56.688270+0000","last_clean":"2024-03-11T21:15:56.688270+0000","last_became_active":"2024-03-11T21:00:48.458008+0000","last_became_peered":"2024-03-11T21:00:48.458008+0000","last_unstale":"2024-03-11T21:15:56.688270+0000","last_undegraded":"2024-03-11T21:15:56.688270+0000","last_fullsized":"2024-03-11T21:15:56.688270+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:12:01.754401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:33.319221+0000","last_clean_scrub_stamp":"2024-03-11T21:12:01.754401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"184'130","reported_seq":1140,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:58.122339+0000","last_change":"2024-03-11T21:14:53.434048+0000","last_active":"2024-03-11T21:15:58.122339+0000","last_peered":"2024-03-11T21:15:58.122339+0000","last_clean":"2024-03-11T21:15:58.122339+0000","last_became_active":"2024-03-11T21:14:53.433609+0000","last_became_peered":"2024-03-11T21:14:53.433609+0000","last_unstale":"2024-03-11T21:15:58.122339+0000","last_undegraded":"2024-03-11T21:15:58.122339+0000","last_fullsized":"2024-03-11T21:15:58.122339+0000","mapping_epoch":426,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":427,"parent":"0.0","parent_split_bits":4,"last_scrub":"184'130","last_scrub_stamp":"2024-03-11T21:14:34.111918+0000","last_deep_scrub":"184'130","last_deep_scrub_stamp":"2024-03-11T20:58:56.415267+0000","last_clean_scrub_stamp":"2024-03-11T21:14:34.111918+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:15:59.185871+0000","scrub_duration":147,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":825,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:58.123874+0000","last_change":"2024-03-11T21:14:29.105552+0000","last_active":"2024-03-11T21:15:58.123874+0000","last_peered":"2024-03-11T21:15:58.123874+0000","last_clean":"2024-03-11T21:15:58.123874+0000","last_became_active":"2024-03-11T20:50:34.321038+0000","last_became_peered":"2024-03-11T20:50:34.321038+0000","last_unstale":"2024-03-11T21:15:58.123874+0000","last_undegraded":"2024-03-11T21:15:58.123874+0000","last_fullsized":"2024-03-11T21:15:58.123874+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:14:29.105424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:11:20.989384+0000","last_clean_scrub_stamp":"2024-03-11T21:14:29.105424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:15:59.185771+0000","scrub_duration":87,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"358'50","reported_seq":580,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:56.687608+0000","last_change":"2024-03-11T21:12:06.861343+0000","last_active":"2024-03-11T21:15:56.687608+0000","last_peered":"2024-03-11T21:15:56.687608+0000","last_clean":"2024-03-11T21:15:56.687608+0000","last_became_active":"2024-03-11T21:00:48.456178+0000","last_became_peered":"2024-03-11T21:00:48.456178+0000","last_unstale":"2024-03-11T21:15:56.687608+0000","last_undegraded":"2024-03-11T21:15:56.687608+0000","last_fullsized":"2024-03-11T21:15:56.687608+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'50","last_scrub_stamp":"2024-03-11T21:12:06.861183+0000","last_deep_scrub":"109'26","last_deep_scrub_stamp":"2024-03-11T20:52:49.216378+0000","last_clean_scrub_stamp":"2024-03-11T21:12:06.861183+0000","objects_scrubbed":8,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":58,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":272,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:57.055019+0000","last_change":"2024-03-11T21:14:50.710933+0000","last_active":"2024-03-11T21:15:57.055019+0000","last_peered":"2024-03-11T21:15:57.055019+0000","last_clean":"2024-03-11T21:15:57.055019+0000","last_became_active":"2024-03-11T21:14:50.708456+0000","last_became_peered":"2024-03-11T21:14:50.708456+0000","last_unstale":"2024-03-11T21:15:57.055019+0000","last_undegraded":"2024-03-11T21:15:57.055019+0000","last_fullsized":"2024-03-11T21:15:57.055019+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:10:55.016272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:10:55.016272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:15:59.286900+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":542,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:56.689063+0000","last_change":"2024-03-11T21:14:50.422600+0000","last_active":"2024-03-11T21:15:56.689063+0000","last_peered":"2024-03-11T21:15:56.689063+0000","last_clean":"2024-03-11T21:15:56.689063+0000","last_became_active":"2024-03-11T21:14:50.422215+0000","last_became_peered":"2024-03-11T21:14:50.422215+0000","last_unstale":"2024-03-11T21:15:56.689063+0000","last_undegraded":"2024-03-11T21:15:56.689063+0000","last_fullsized":"2024-03-11T21:15:56.689063+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:12:49.854667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:12:49.854667+0000","last_clean_scrub_stamp":"2024-03-11T21:12:49.854667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":820,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:57.904855+0000","last_change":"2024-03-11T21:14:02.373791+0000","last_active":"2024-03-11T21:15:57.904855+0000","last_peered":"2024-03-11T21:15:57.904855+0000","last_clean":"2024-03-11T21:15:57.904855+0000","last_became_active":"2024-03-11T21:00:27.962668+0000","last_became_peered":"2024-03-11T21:00:27.962668+0000","last_unstale":"2024-03-11T21:15:57.904855+0000","last_undegraded":"2024-03-11T21:15:57.904855+0000","last_fullsized":"2024-03-11T21:15:57.904855+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:14:02.373709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:14:02.373709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:16:02.900381+0000","scrub_duration":151,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"358'35","reported_seq":854,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:57.054477+0000","last_change":"2024-03-11T21:11:53.949072+0000","last_active":"2024-03-11T21:15:57.054477+0000","last_peered":"2024-03-11T21:15:57.054477+0000","last_clean":"2024-03-11T21:15:57.054477+0000","last_became_active":"2024-03-11T20:50:30.101747+0000","last_became_peered":"2024-03-11T20:50:30.101747+0000","last_unstale":"2024-03-11T21:15:57.054477+0000","last_undegraded":"2024-03-11T21:15:57.054477+0000","last_fullsized":"2024-03-11T21:15:57.054477+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'35","last_scrub_stamp":"2024-03-11T21:11:53.948819+0000","last_deep_scrub":"284'25","last_deep_scrub_stamp":"2024-03-11T21:04:53.217188+0000","last_clean_scrub_stamp":"2024-03-11T21:11:53.948819+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:15:59.287605+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":840,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:57.903135+0000","last_change":"2024-03-11T21:14:26.360865+0000","last_active":"2024-03-11T21:15:57.903135+0000","last_peered":"2024-03-11T21:15:57.903135+0000","last_clean":"2024-03-11T21:15:57.903135+0000","last_became_active":"2024-03-11T20:50:28.085503+0000","last_became_peered":"2024-03-11T20:50:28.085503+0000","last_unstale":"2024-03-11T21:15:57.903135+0000","last_undegraded":"2024-03-11T21:15:57.903135+0000","last_fullsized":"2024-03-11T21:15:57.903135+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:14:26.360741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:56:03.479180+0000","last_clean_scrub_stamp":"2024-03-11T21:14:26.360741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:16:02.900817+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"184'130","reported_seq":1132,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:58.122310+0000","last_change":"2024-03-11T21:14:53.436551+0000","last_active":"2024-03-11T21:15:58.122310+0000","last_peered":"2024-03-11T21:15:58.122310+0000","last_clean":"2024-03-11T21:15:58.122310+0000","last_became_active":"2024-03-11T21:14:53.436241+0000","last_became_peered":"2024-03-11T21:14:53.436241+0000","last_unstale":"2024-03-11T21:15:58.122310+0000","last_undegraded":"2024-03-11T21:15:58.122310+0000","last_fullsized":"2024-03-11T21:15:58.122310+0000","mapping_epoch":426,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":427,"parent":"0.0","parent_split_bits":0,"last_scrub":"184'130","last_scrub_stamp":"2024-03-11T21:14:40.126341+0000","last_deep_scrub":"184'130","last_deep_scrub_stamp":"2024-03-11T21:14:40.126341+0000","last_clean_scrub_stamp":"2024-03-11T21:14:40.126341+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:15:59.186175+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":846,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:57.104573+0000","last_change":"2024-03-11T21:14:10.129313+0000","last_active":"2024-03-11T21:15:57.104573+0000","last_peered":"2024-03-11T21:15:57.104573+0000","last_clean":"2024-03-11T21:15:57.104573+0000","last_became_active":"2024-03-11T21:00:27.959467+0000","last_became_peered":"2024-03-11T21:00:27.959467+0000","last_unstale":"2024-03-11T21:15:57.104573+0000","last_undegraded":"2024-03-11T21:15:57.104573+0000","last_fullsized":"2024-03-11T21:15:57.104573+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:14:10.129204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:54:22.448812+0000","last_clean_scrub_stamp":"2024-03-11T21:14:10.129204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:16:01.881454+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.3","version":"184'130","reported_seq":678,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:56.688228+0000","last_change":"2024-03-11T21:11:55.755937+0000","last_active":"2024-03-11T21:15:56.688228+0000","last_peered":"2024-03-11T21:15:56.688228+0000","last_clean":"2024-03-11T21:15:56.688228+0000","last_became_active":"2024-03-11T21:11:28.397947+0000","last_became_peered":"2024-03-11T21:11:28.397947+0000","last_unstale":"2024-03-11T21:15:56.688228+0000","last_undegraded":"2024-03-11T21:15:56.688228+0000","last_fullsized":"2024-03-11T21:15:56.688228+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":3,"last_scrub":"184'130","last_scrub_stamp":"2024-03-11T21:11:55.755820+0000","last_deep_scrub":"184'130","last_deep_scrub_stamp":"2024-03-11T20:58:56.415267+0000","last_clean_scrub_stamp":"2024-03-11T21:11:55.755820+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":530,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:56.688463+0000","last_change":"2024-03-11T21:11:54.793910+0000","last_active":"2024-03-11T21:15:56.688463+0000","last_peered":"2024-03-11T21:15:56.688463+0000","last_clean":"2024-03-11T21:15:56.688463+0000","last_became_active":"2024-03-11T21:00:48.464596+0000","last_became_peered":"2024-03-11T21:00:48.464596+0000","last_unstale":"2024-03-11T21:15:56.688463+0000","last_undegraded":"2024-03-11T21:15:56.688463+0000","last_fullsized":"2024-03-11T21:15:56.688463+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:11:54.793800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:11:54.793800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"358'49","reported_seq":55,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:56.684295+0000","last_change":"2024-03-11T21:14:50.468244+0000","last_active":"2024-03-11T21:15:56.684295+0000","last_peered":"2024-03-11T21:15:56.684295+0000","last_clean":"2024-03-11T21:15:56.684295+0000","last_became_active":"2024-03-11T21:14:50.433732+0000","last_became_peered":"2024-03-11T21:14:50.433732+0000","last_unstale":"2024-03-11T21:15:56.684295+0000","last_undegraded":"2024-03-11T21:15:56.684295+0000","last_fullsized":"2024-03-11T21:15:56.684295+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'49","last_scrub_stamp":"2024-03-11T21:12:15.033032+0000","last_deep_scrub":"100'21","last_deep_scrub_stamp":"2024-03-11T20:51:31.835302+0000","last_clean_scrub_stamp":"2024-03-11T21:12:15.033032+0000","objects_scrubbed":7,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":54,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":580,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:56.687957+0000","last_change":"2024-03-11T21:14:50.702476+0000","last_active":"2024-03-11T21:15:56.687957+0000","last_peered":"2024-03-11T21:15:56.687957+0000","last_clean":"2024-03-11T21:15:56.687957+0000","last_became_active":"2024-03-11T21:14:50.701590+0000","last_became_peered":"2024-03-11T21:14:50.701590+0000","last_unstale":"2024-03-11T21:15:56.687957+0000","last_undegraded":"2024-03-11T21:15:56.687957+0000","last_fullsized":"2024-03-11T21:15:56.687957+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:12:39.915896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:12:39.915896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":39,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:56.685139+0000","last_change":"2024-03-11T21:14:50.708970+0000","last_active":"2024-03-11T21:15:56.685139+0000","last_peered":"2024-03-11T21:15:56.685139+0000","last_clean":"2024-03-11T21:15:56.685139+0000","last_became_active":"2024-03-11T21:14:50.708638+0000","last_became_peered":"2024-03-11T21:14:50.708638+0000","last_unstale":"2024-03-11T21:15:56.685139+0000","last_undegraded":"2024-03-11T21:15:56.685139+0000","last_fullsized":"2024-03-11T21:15:56.685139+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:12:41.942568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:26.839782+0000","last_clean_scrub_stamp":"2024-03-11T21:12:41.942568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"184'130","reported_seq":708,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:57.104531+0000","last_change":"2024-03-11T21:14:53.456006+0000","last_active":"2024-03-11T21:15:57.104531+0000","last_peered":"2024-03-11T21:15:57.104531+0000","last_clean":"2024-03-11T21:15:57.104531+0000","last_became_active":"2024-03-11T21:14:53.455686+0000","last_became_peered":"2024-03-11T21:14:53.455686+0000","last_unstale":"2024-03-11T21:15:57.104531+0000","last_undegraded":"2024-03-11T21:15:57.104531+0000","last_fullsized":"2024-03-11T21:15:57.104531+0000","mapping_epoch":426,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":427,"parent":"0.0","parent_split_bits":3,"last_scrub":"184'130","last_scrub_stamp":"2024-03-11T21:14:15.621867+0000","last_deep_scrub":"184'130","last_deep_scrub_stamp":"2024-03-11T20:58:56.415267+0000","last_clean_scrub_stamp":"2024-03-11T21:14:15.621867+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:16:01.881275+0000","scrub_duration":187,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":530,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:56.687783+0000","last_change":"2024-03-11T21:11:10.720532+0000","last_active":"2024-03-11T21:15:56.687783+0000","last_peered":"2024-03-11T21:15:56.687783+0000","last_clean":"2024-03-11T21:15:56.687783+0000","last_became_active":"2024-03-11T21:00:48.469688+0000","last_became_peered":"2024-03-11T21:00:48.469688+0000","last_unstale":"2024-03-11T21:15:56.687783+0000","last_undegraded":"2024-03-11T21:15:56.687783+0000","last_fullsized":"2024-03-11T21:15:56.687783+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:11:10.720386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:11:10.720386+0000","last_clean_scrub_stamp":"2024-03-11T21:11:10.720386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"145'3","reported_seq":867,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:57.103408+0000","last_change":"2024-03-11T21:14:50.441186+0000","last_active":"2024-03-11T21:15:57.103408+0000","last_peered":"2024-03-11T21:15:57.103408+0000","last_clean":"2024-03-11T21:15:57.103408+0000","last_became_active":"2024-03-11T21:14:50.427126+0000","last_became_peered":"2024-03-11T21:14:50.427126+0000","last_unstale":"2024-03-11T21:15:57.103408+0000","last_undegraded":"2024-03-11T21:15:57.103408+0000","last_fullsized":"2024-03-11T21:15:57.103408+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'3","last_scrub_stamp":"2024-03-11T21:14:49.024170+0000","last_deep_scrub":"145'3","last_deep_scrub_stamp":"2024-03-11T20:55:52.848043+0000","last_clean_scrub_stamp":"2024-03-11T21:14:49.024170+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:16:01.881638+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0024049620000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":850,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:58.124165+0000","last_change":"2024-03-11T21:14:56.231678+0000","last_active":"2024-03-11T21:15:58.124165+0000","last_peered":"2024-03-11T21:15:58.124165+0000","last_clean":"2024-03-11T21:15:58.124165+0000","last_became_active":"2024-03-11T21:14:50.446250+0000","last_became_peered":"2024-03-11T21:14:50.446250+0000","last_unstale":"2024-03-11T21:15:58.124165+0000","last_undegraded":"2024-03-11T21:15:58.124165+0000","last_fullsized":"2024-03-11T21:15:58.124165+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:14:56.231557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:14:56.231557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:16:12.427137+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"358'58","reported_seq":767,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:56.688660+0000","last_change":"2024-03-11T21:12:20.000340+0000","last_active":"2024-03-11T21:15:56.688660+0000","last_peered":"2024-03-11T21:15:56.688660+0000","last_clean":"2024-03-11T21:15:56.688660+0000","last_became_active":"2024-03-11T21:00:48.473227+0000","last_became_peered":"2024-03-11T21:00:48.473227+0000","last_unstale":"2024-03-11T21:15:56.688660+0000","last_undegraded":"2024-03-11T21:15:56.688660+0000","last_fullsized":"2024-03-11T21:15:56.688660+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'58","last_scrub_stamp":"2024-03-11T21:12:20.000157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:12:20.000157+0000","objects_scrubbed":10,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":87,"num_read_kb":72,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"424'15","reported_seq":58,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:56.684758+0000","last_change":"2024-03-11T21:14:50.447580+0000","last_active":"2024-03-11T21:15:56.684758+0000","last_peered":"2024-03-11T21:15:56.684758+0000","last_clean":"2024-03-11T21:15:56.684758+0000","last_became_active":"2024-03-11T21:14:50.431548+0000","last_became_peered":"2024-03-11T21:14:50.431548+0000","last_unstale":"2024-03-11T21:15:56.684758+0000","last_undegraded":"2024-03-11T21:15:56.684758+0000","last_fullsized":"2024-03-11T21:15:56.684758+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"298'13","last_scrub_stamp":"2024-03-11T21:12:31.022898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:12:31.022898+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"184'130","reported_seq":690,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:57.103995+0000","last_change":"2024-03-11T21:14:36.148536+0000","last_active":"2024-03-11T21:15:57.103995+0000","last_peered":"2024-03-11T21:15:57.103995+0000","last_clean":"2024-03-11T21:15:57.103995+0000","last_became_active":"2024-03-11T21:11:28.397374+0000","last_became_peered":"2024-03-11T21:11:28.397374+0000","last_unstale":"2024-03-11T21:15:57.103995+0000","last_undegraded":"2024-03-11T21:15:57.103995+0000","last_fullsized":"2024-03-11T21:15:57.103995+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":3,"last_scrub":"184'130","last_scrub_stamp":"2024-03-11T21:14:36.148419+0000","last_deep_scrub":"184'130","last_deep_scrub_stamp":"2024-03-11T20:58:56.415267+0000","last_clean_scrub_stamp":"2024-03-11T21:14:36.148419+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:16:04.406981+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":550,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:57.054783+0000","last_change":"2024-03-11T21:12:24.980257+0000","last_active":"2024-03-11T21:15:57.054783+0000","last_peered":"2024-03-11T21:15:57.054783+0000","last_clean":"2024-03-11T21:15:57.054783+0000","last_became_active":"2024-03-11T21:05:17.441521+0000","last_became_peered":"2024-03-11T21:05:17.441521+0000","last_unstale":"2024-03-11T21:15:57.054783+0000","last_undegraded":"2024-03-11T21:15:57.054783+0000","last_fullsized":"2024-03-11T21:15:57.054783+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:12:24.980140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:03:43.345257+0000","last_clean_scrub_stamp":"2024-03-11T21:12:24.980140+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:15:59.288954+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":571,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:57.055763+0000","last_change":"2024-03-11T21:12:40.097209+0000","last_active":"2024-03-11T21:15:57.055763+0000","last_peered":"2024-03-11T21:15:57.055763+0000","last_clean":"2024-03-11T21:15:57.055763+0000","last_became_active":"2024-03-11T21:05:17.315844+0000","last_became_peered":"2024-03-11T21:05:17.315844+0000","last_unstale":"2024-03-11T21:15:57.055763+0000","last_undegraded":"2024-03-11T21:15:57.055763+0000","last_fullsized":"2024-03-11T21:15:57.055763+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:12:40.096694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:22.617999+0000","last_clean_scrub_stamp":"2024-03-11T21:12:40.096694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:15:59.289040+0000","scrub_duration":139,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"284'2","reported_seq":535,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:56.688918+0000","last_change":"2024-03-11T21:11:57.838884+0000","last_active":"2024-03-11T21:15:56.688918+0000","last_peered":"2024-03-11T21:15:56.688918+0000","last_clean":"2024-03-11T21:15:56.688918+0000","last_became_active":"2024-03-11T21:00:48.468682+0000","last_became_peered":"2024-03-11T21:00:48.468682+0000","last_unstale":"2024-03-11T21:15:56.688918+0000","last_undegraded":"2024-03-11T21:15:56.688918+0000","last_fullsized":"2024-03-11T21:15:56.688918+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"284'2","last_scrub_stamp":"2024-03-11T21:11:57.838662+0000","last_deep_scrub":"284'2","last_deep_scrub_stamp":"2024-03-11T21:06:47.645682+0000","last_clean_scrub_stamp":"2024-03-11T21:11:57.838662+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"357'21","reported_seq":878,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:57.055886+0000","last_change":"2024-03-11T21:14:50.454247+0000","last_active":"2024-03-11T21:15:57.055886+0000","last_peered":"2024-03-11T21:15:57.055886+0000","last_clean":"2024-03-11T21:15:57.055886+0000","last_became_active":"2024-03-11T21:14:50.435780+0000","last_became_peered":"2024-03-11T21:14:50.435780+0000","last_unstale":"2024-03-11T21:15:57.055886+0000","last_undegraded":"2024-03-11T21:15:57.055886+0000","last_fullsized":"2024-03-11T21:15:57.055886+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"357'21","last_scrub_stamp":"2024-03-11T21:14:35.355709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:14:35.355709+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:15:59.289499+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":31,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"424'13","reported_seq":1146,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:57.054319+0000","last_change":"2024-03-11T21:14:50.446981+0000","last_active":"2024-03-11T21:15:57.054319+0000","last_peered":"2024-03-11T21:15:57.054319+0000","last_clean":"2024-03-11T21:15:57.054319+0000","last_became_active":"2024-03-11T21:14:50.428398+0000","last_became_peered":"2024-03-11T21:14:50.428398+0000","last_unstale":"2024-03-11T21:15:57.054319+0000","last_undegraded":"2024-03-11T21:15:57.054319+0000","last_fullsized":"2024-03-11T21:15:57.054319+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'11","last_scrub_stamp":"2024-03-11T21:12:02.860638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:12:02.860638+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"184'130","reported_seq":678,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:57.055190+0000","last_change":"2024-03-11T21:11:57.831886+0000","last_active":"2024-03-11T21:15:57.055190+0000","last_peered":"2024-03-11T21:15:57.055190+0000","last_clean":"2024-03-11T21:15:57.055190+0000","last_became_active":"2024-03-11T21:11:28.393756+0000","last_became_peered":"2024-03-11T21:11:28.393756+0000","last_unstale":"2024-03-11T21:15:57.055190+0000","last_undegraded":"2024-03-11T21:15:57.055190+0000","last_fullsized":"2024-03-11T21:15:57.055190+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":3,"last_scrub":"184'130","last_scrub_stamp":"2024-03-11T21:11:57.831636+0000","last_deep_scrub":"184'130","last_deep_scrub_stamp":"2024-03-11T20:58:56.415267+0000","last_clean_scrub_stamp":"2024-03-11T21:11:57.831636+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:15:59.288188+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":833,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:16:00.232279+0000","last_change":"2024-03-11T21:16:00.232356+0000","last_active":"2024-03-11T21:16:00.232279+0000","last_peered":"2024-03-11T21:16:00.232279+0000","last_clean":"2024-03-11T21:16:00.232279+0000","last_became_active":"2024-03-11T20:50:34.326947+0000","last_became_peered":"2024-03-11T20:50:34.326947+0000","last_unstale":"2024-03-11T21:16:00.232279+0000","last_undegraded":"2024-03-11T21:16:00.232279+0000","last_fullsized":"2024-03-11T21:16:00.232279+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:16:00.232096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:06:43.271453+0000","last_clean_scrub_stamp":"2024-03-11T21:16:00.232096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:17:06.856754+0000","scrub_duration":135,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"120'6","reported_seq":2242,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:59.726205+0000","last_change":"2024-03-11T21:12:09.860900+0000","last_active":"2024-03-11T21:15:59.726205+0000","last_peered":"2024-03-11T21:15:59.726205+0000","last_clean":"2024-03-11T21:15:59.726205+0000","last_became_active":"2024-03-11T21:00:48.468997+0000","last_became_peered":"2024-03-11T21:00:48.468997+0000","last_unstale":"2024-03-11T21:15:59.726205+0000","last_undegraded":"2024-03-11T21:15:59.726205+0000","last_fullsized":"2024-03-11T21:15:59.726205+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"120'6","last_scrub_stamp":"2024-03-11T21:12:09.860783+0000","last_deep_scrub":"120'6","last_deep_scrub_stamp":"2024-03-11T20:54:26.468057+0000","last_clean_scrub_stamp":"2024-03-11T21:12:09.860783+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0035533370000000002,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1393,"num_read_kb":1388,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":860,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:58.883335+0000","last_change":"2024-03-11T21:15:58.883390+0000","last_active":"2024-03-11T21:15:58.883335+0000","last_peered":"2024-03-11T21:15:58.883335+0000","last_clean":"2024-03-11T21:15:58.883335+0000","last_became_active":"2024-03-11T20:50:28.071707+0000","last_became_peered":"2024-03-11T20:50:28.071707+0000","last_unstale":"2024-03-11T21:15:58.883335+0000","last_undegraded":"2024-03-11T21:15:58.883335+0000","last_fullsized":"2024-03-11T21:15:58.883335+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:15:58.883185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:23.417283+0000","last_clean_scrub_stamp":"2024-03-11T21:15:58.883185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:17:27.728855+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"358'36","reported_seq":53,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:56.685086+0000","last_change":"2024-03-11T21:14:50.453810+0000","last_active":"2024-03-11T21:15:56.685086+0000","last_peered":"2024-03-11T21:15:56.685086+0000","last_clean":"2024-03-11T21:15:56.685086+0000","last_became_active":"2024-03-11T21:14:50.418699+0000","last_became_peered":"2024-03-11T21:14:50.418699+0000","last_unstale":"2024-03-11T21:15:56.685086+0000","last_undegraded":"2024-03-11T21:15:56.685086+0000","last_fullsized":"2024-03-11T21:15:56.685086+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'36","last_scrub_stamp":"2024-03-11T21:12:14.031345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:12:14.031345+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":47,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":608,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:57.902402+0000","last_change":"2024-03-11T21:14:32.441436+0000","last_active":"2024-03-11T21:15:57.902402+0000","last_peered":"2024-03-11T21:15:57.902402+0000","last_clean":"2024-03-11T21:15:57.902402+0000","last_became_active":"2024-03-11T21:05:17.315807+0000","last_became_peered":"2024-03-11T21:05:17.315807+0000","last_unstale":"2024-03-11T21:15:57.902402+0000","last_undegraded":"2024-03-11T21:15:57.902402+0000","last_fullsized":"2024-03-11T21:15:57.902402+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:14:32.441315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:09:51.796383+0000","last_clean_scrub_stamp":"2024-03-11T21:14:32.441315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:16:02.900858+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":795,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:57.054817+0000","last_change":"2024-03-11T21:12:51.179642+0000","last_active":"2024-03-11T21:15:57.054817+0000","last_peered":"2024-03-11T21:15:57.054817+0000","last_clean":"2024-03-11T21:15:57.054817+0000","last_became_active":"2024-03-11T20:50:34.335370+0000","last_became_peered":"2024-03-11T20:50:34.335370+0000","last_unstale":"2024-03-11T21:15:57.054817+0000","last_undegraded":"2024-03-11T21:15:57.054817+0000","last_fullsized":"2024-03-11T21:15:57.054817+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:12:51.179514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:12:51.179514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:15:59.289169+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"286'18","reported_seq":552,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:57.104591+0000","last_change":"2024-03-11T21:14:22.345691+0000","last_active":"2024-03-11T21:15:57.104591+0000","last_peered":"2024-03-11T21:15:57.104591+0000","last_clean":"2024-03-11T21:15:57.104591+0000","last_became_active":"2024-03-11T21:05:17.449213+0000","last_became_peered":"2024-03-11T21:05:17.449213+0000","last_unstale":"2024-03-11T21:15:57.104591+0000","last_undegraded":"2024-03-11T21:15:57.104591+0000","last_fullsized":"2024-03-11T21:15:57.104591+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"286'18","last_scrub_stamp":"2024-03-11T21:14:22.345553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:14:22.345553+0000","objects_scrubbed":1,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:16:01.881413+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":853,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:58.123332+0000","last_change":"2024-03-11T21:14:50.433691+0000","last_active":"2024-03-11T21:15:58.123332+0000","last_peered":"2024-03-11T21:15:58.123332+0000","last_clean":"2024-03-11T21:15:58.123332+0000","last_became_active":"2024-03-11T21:14:50.433372+0000","last_became_peered":"2024-03-11T21:14:50.433372+0000","last_unstale":"2024-03-11T21:15:58.123332+0000","last_undegraded":"2024-03-11T21:15:58.123332+0000","last_fullsized":"2024-03-11T21:15:58.123332+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:12:25.082734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:06:42.242801+0000","last_clean_scrub_stamp":"2024-03-11T21:12:25.082734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:15:59.184997+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"357'24","reported_seq":541,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:58.122367+0000","last_change":"2024-03-11T21:12:20.041186+0000","last_active":"2024-03-11T21:15:58.122367+0000","last_peered":"2024-03-11T21:15:58.122367+0000","last_clean":"2024-03-11T21:15:58.122367+0000","last_became_active":"2024-03-11T21:08:05.298170+0000","last_became_peered":"2024-03-11T21:08:05.298170+0000","last_unstale":"2024-03-11T21:15:58.122367+0000","last_undegraded":"2024-03-11T21:15:58.122367+0000","last_fullsized":"2024-03-11T21:15:58.122367+0000","mapping_epoch":319,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":0,"last_scrub":"357'24","last_scrub_stamp":"2024-03-11T21:12:20.041015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:12:20.041015+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:15:59.185390+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":41,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"184'130","reported_seq":686,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:57.903674+0000","last_change":"2024-03-11T21:13:55.286844+0000","last_active":"2024-03-11T21:15:57.903674+0000","last_peered":"2024-03-11T21:15:57.903674+0000","last_clean":"2024-03-11T21:15:57.903674+0000","last_became_active":"2024-03-11T21:11:28.396897+0000","last_became_peered":"2024-03-11T21:11:28.396897+0000","last_unstale":"2024-03-11T21:15:57.903674+0000","last_undegraded":"2024-03-11T21:15:57.903674+0000","last_fullsized":"2024-03-11T21:15:57.903674+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":4,"last_scrub":"184'130","last_scrub_stamp":"2024-03-11T21:13:55.286755+0000","last_deep_scrub":"184'130","last_deep_scrub_stamp":"2024-03-11T20:58:56.415267+0000","last_clean_scrub_stamp":"2024-03-11T21:13:55.286755+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:16:02.900628+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":852,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:57.904813+0000","last_change":"2024-03-11T21:14:51.855267+0000","last_active":"2024-03-11T21:15:57.904813+0000","last_peered":"2024-03-11T21:15:57.904813+0000","last_clean":"2024-03-11T21:15:57.904813+0000","last_became_active":"2024-03-11T21:14:50.434097+0000","last_became_peered":"2024-03-11T21:14:50.434097+0000","last_unstale":"2024-03-11T21:15:57.904813+0000","last_undegraded":"2024-03-11T21:15:57.904813+0000","last_fullsized":"2024-03-11T21:15:57.904813+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:14:51.855140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:27.775534+0000","last_clean_scrub_stamp":"2024-03-11T21:14:51.855140+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:16:02.901109+0000","scrub_duration":251,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":537,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:56.689005+0000","last_change":"2024-03-11T21:11:58.865003+0000","last_active":"2024-03-11T21:15:56.689005+0000","last_peered":"2024-03-11T21:15:56.689005+0000","last_clean":"2024-03-11T21:15:56.689005+0000","last_became_active":"2024-03-11T21:00:48.469596+0000","last_became_peered":"2024-03-11T21:00:48.469596+0000","last_unstale":"2024-03-11T21:15:56.689005+0000","last_undegraded":"2024-03-11T21:15:56.689005+0000","last_fullsized":"2024-03-11T21:15:56.689005+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:11:58.864737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:11:58.864737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":530,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:56.688059+0000","last_change":"2024-03-11T21:12:42.924462+0000","last_active":"2024-03-11T21:15:56.688059+0000","last_peered":"2024-03-11T21:15:56.688059+0000","last_clean":"2024-03-11T21:15:56.688059+0000","last_became_active":"2024-03-11T21:00:48.466084+0000","last_became_peered":"2024-03-11T21:00:48.466084+0000","last_unstale":"2024-03-11T21:15:56.688059+0000","last_undegraded":"2024-03-11T21:15:56.688059+0000","last_fullsized":"2024-03-11T21:15:56.688059+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:12:42.924357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:40.974147+0000","last_clean_scrub_stamp":"2024-03-11T21:12:42.924357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"358'46","reported_seq":931,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:57.054104+0000","last_change":"2024-03-11T21:12:20.007909+0000","last_active":"2024-03-11T21:15:57.054104+0000","last_peered":"2024-03-11T21:15:57.054104+0000","last_clean":"2024-03-11T21:15:57.054104+0000","last_became_active":"2024-03-11T21:00:48.472191+0000","last_became_peered":"2024-03-11T21:00:48.472191+0000","last_unstale":"2024-03-11T21:15:57.054104+0000","last_undegraded":"2024-03-11T21:15:57.054104+0000","last_fullsized":"2024-03-11T21:15:57.054104+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'46","last_scrub_stamp":"2024-03-11T21:12:20.007731+0000","last_deep_scrub":"116'22","last_deep_scrub_stamp":"2024-03-11T20:53:47.364709+0000","last_clean_scrub_stamp":"2024-03-11T21:12:20.007731+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:15:59.288091+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"184'130","reported_seq":1122,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:58.122901+0000","last_change":"2024-03-11T21:14:46.117398+0000","last_active":"2024-03-11T21:15:58.122901+0000","last_peered":"2024-03-11T21:15:58.122901+0000","last_clean":"2024-03-11T21:15:58.122901+0000","last_became_active":"2024-03-11T21:11:28.405250+0000","last_became_peered":"2024-03-11T21:11:28.405250+0000","last_unstale":"2024-03-11T21:15:58.122901+0000","last_undegraded":"2024-03-11T21:15:58.122901+0000","last_fullsized":"2024-03-11T21:15:58.122901+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":4,"last_scrub":"184'130","last_scrub_stamp":"2024-03-11T21:14:46.117302+0000","last_deep_scrub":"184'130","last_deep_scrub_stamp":"2024-03-11T20:58:56.415267+0000","last_clean_scrub_stamp":"2024-03-11T21:14:46.117302+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:16:00.618690+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":616,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:56.688762+0000","last_change":"2024-03-11T21:12:18.814077+0000","last_active":"2024-03-11T21:15:56.688762+0000","last_peered":"2024-03-11T21:15:56.688762+0000","last_clean":"2024-03-11T21:15:56.688762+0000","last_became_active":"2024-03-11T21:05:17.455236+0000","last_became_peered":"2024-03-11T21:05:17.455236+0000","last_unstale":"2024-03-11T21:15:56.688762+0000","last_undegraded":"2024-03-11T21:15:56.688762+0000","last_fullsized":"2024-03-11T21:15:56.688762+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:12:18.813941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:12:18.813941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":530,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:56.688421+0000","last_change":"2024-03-11T21:11:18.625591+0000","last_active":"2024-03-11T21:15:56.688421+0000","last_peered":"2024-03-11T21:15:56.688421+0000","last_clean":"2024-03-11T21:15:56.688421+0000","last_became_active":"2024-03-11T21:00:48.461367+0000","last_became_peered":"2024-03-11T21:00:48.461367+0000","last_unstale":"2024-03-11T21:15:56.688421+0000","last_undegraded":"2024-03-11T21:15:56.688421+0000","last_fullsized":"2024-03-11T21:15:56.688421+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:11:18.625296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:03:37.601229+0000","last_clean_scrub_stamp":"2024-03-11T21:11:18.625296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"358'25","reported_seq":882,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:57.055513+0000","last_change":"2024-03-11T21:14:50.459402+0000","last_active":"2024-03-11T21:15:57.055513+0000","last_peered":"2024-03-11T21:15:57.055513+0000","last_clean":"2024-03-11T21:15:57.055513+0000","last_became_active":"2024-03-11T21:14:50.436328+0000","last_became_peered":"2024-03-11T21:14:50.436328+0000","last_unstale":"2024-03-11T21:15:57.055513+0000","last_undegraded":"2024-03-11T21:15:57.055513+0000","last_fullsized":"2024-03-11T21:15:57.055513+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'25","last_scrub_stamp":"2024-03-11T21:12:08.861796+0000","last_deep_scrub":"267'19","last_deep_scrub_stamp":"2024-03-11T21:03:39.673317+0000","last_clean_scrub_stamp":"2024-03-11T21:12:08.861796+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:15:59.287795+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":593,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:57.055552+0000","last_change":"2024-03-11T21:11:20.159860+0000","last_active":"2024-03-11T21:15:57.055552+0000","last_peered":"2024-03-11T21:15:57.055552+0000","last_clean":"2024-03-11T21:15:57.055552+0000","last_became_active":"2024-03-11T21:05:17.316136+0000","last_became_peered":"2024-03-11T21:05:17.316136+0000","last_unstale":"2024-03-11T21:15:57.055552+0000","last_undegraded":"2024-03-11T21:15:57.055552+0000","last_fullsized":"2024-03-11T21:15:57.055552+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:11:20.159640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:35.670149+0000","last_clean_scrub_stamp":"2024-03-11T21:11:20.159640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:15:59.287401+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"184'130","reported_seq":686,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:57.903298+0000","last_change":"2024-03-11T21:14:15.624671+0000","last_active":"2024-03-11T21:15:57.903298+0000","last_peered":"2024-03-11T21:15:57.903298+0000","last_clean":"2024-03-11T21:15:57.903298+0000","last_became_active":"2024-03-11T21:11:28.398728+0000","last_became_peered":"2024-03-11T21:11:28.398728+0000","last_unstale":"2024-03-11T21:15:57.903298+0000","last_undegraded":"2024-03-11T21:15:57.903298+0000","last_fullsized":"2024-03-11T21:15:57.903298+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":4,"last_scrub":"184'130","last_scrub_stamp":"2024-03-11T21:14:15.624423+0000","last_deep_scrub":"184'130","last_deep_scrub_stamp":"2024-03-11T20:58:56.415267+0000","last_clean_scrub_stamp":"2024-03-11T21:14:15.624423+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:16:02.900674+0000","scrub_duration":191,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":572,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:57.055472+0000","last_change":"2024-03-11T21:12:45.541739+0000","last_active":"2024-03-11T21:15:57.055472+0000","last_peered":"2024-03-11T21:15:57.055472+0000","last_clean":"2024-03-11T21:15:57.055472+0000","last_became_active":"2024-03-11T21:08:05.304963+0000","last_became_peered":"2024-03-11T21:08:05.304963+0000","last_unstale":"2024-03-11T21:15:57.055472+0000","last_undegraded":"2024-03-11T21:15:57.055472+0000","last_fullsized":"2024-03-11T21:15:57.055472+0000","mapping_epoch":319,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:12:45.541619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:12:45.541619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:15:59.289262+0000","scrub_duration":247,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"320'21","reported_seq":870,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:57.902646+0000","last_change":"2024-03-11T21:14:01.256726+0000","last_active":"2024-03-11T21:15:57.902646+0000","last_peered":"2024-03-11T21:15:57.902646+0000","last_clean":"2024-03-11T21:15:57.902646+0000","last_became_active":"2024-03-11T21:08:05.298887+0000","last_became_peered":"2024-03-11T21:08:05.298887+0000","last_unstale":"2024-03-11T21:15:57.902646+0000","last_undegraded":"2024-03-11T21:15:57.902646+0000","last_fullsized":"2024-03-11T21:15:57.902646+0000","mapping_epoch":319,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'21","last_scrub_stamp":"2024-03-11T21:14:01.256608+0000","last_deep_scrub":"146'14","last_deep_scrub_stamp":"2024-03-11T20:56:00.242211+0000","last_clean_scrub_stamp":"2024-03-11T21:14:01.256608+0000","objects_scrubbed":1,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:16:02.900434+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":837,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:58.123785+0000","last_change":"2024-03-11T21:14:30.002979+0000","last_active":"2024-03-11T21:15:58.123785+0000","last_peered":"2024-03-11T21:15:58.123785+0000","last_clean":"2024-03-11T21:15:58.123785+0000","last_became_active":"2024-03-11T21:00:27.970573+0000","last_became_peered":"2024-03-11T21:00:27.970573+0000","last_unstale":"2024-03-11T21:15:58.123785+0000","last_undegraded":"2024-03-11T21:15:58.123785+0000","last_fullsized":"2024-03-11T21:15:58.123785+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:14:30.002875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:11:14.836808+0000","last_clean_scrub_stamp":"2024-03-11T21:14:30.002875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:15:59.186071+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"358'60","reported_seq":1005,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:57.103642+0000","last_change":"2024-03-11T21:14:41.101346+0000","last_active":"2024-03-11T21:15:57.103642+0000","last_peered":"2024-03-11T21:15:57.103642+0000","last_clean":"2024-03-11T21:15:57.103642+0000","last_became_active":"2024-03-11T21:00:48.469854+0000","last_became_peered":"2024-03-11T21:00:48.469854+0000","last_unstale":"2024-03-11T21:15:57.103642+0000","last_undegraded":"2024-03-11T21:15:57.103642+0000","last_fullsized":"2024-03-11T21:15:57.103642+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'60","last_scrub_stamp":"2024-03-11T21:14:41.101197+0000","last_deep_scrub":"161'28","last_deep_scrub_stamp":"2024-03-11T20:57:20.816278+0000","last_clean_scrub_stamp":"2024-03-11T21:14:41.101197+0000","objects_scrubbed":9,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:16:01.881590+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":80,"num_read_kb":69,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":824,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:58.122929+0000","last_change":"2024-03-11T21:15:04.239092+0000","last_active":"2024-03-11T21:15:58.122929+0000","last_peered":"2024-03-11T21:15:58.122929+0000","last_clean":"2024-03-11T21:15:58.122929+0000","last_became_active":"2024-03-11T20:50:34.327139+0000","last_became_peered":"2024-03-11T20:50:34.327139+0000","last_unstale":"2024-03-11T21:15:58.122929+0000","last_undegraded":"2024-03-11T21:15:58.122929+0000","last_fullsized":"2024-03-11T21:15:58.122929+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:15:04.238990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:09:53.436482+0000","last_clean_scrub_stamp":"2024-03-11T21:15:04.238990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:16:32.272189+0000","scrub_duration":87,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":296,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:57.904212+0000","last_change":"2024-03-11T21:14:50.702080+0000","last_active":"2024-03-11T21:15:57.904212+0000","last_peered":"2024-03-11T21:15:57.904212+0000","last_clean":"2024-03-11T21:15:57.904212+0000","last_became_active":"2024-03-11T21:14:50.701717+0000","last_became_peered":"2024-03-11T21:14:50.701717+0000","last_unstale":"2024-03-11T21:15:57.904212+0000","last_undegraded":"2024-03-11T21:15:57.904212+0000","last_fullsized":"2024-03-11T21:15:57.904212+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:14:48.520684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:57:17.447191+0000","last_clean_scrub_stamp":"2024-03-11T21:14:48.520684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:16:16.588992+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"358'53","reported_seq":910,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:58.124448+0000","last_change":"2024-03-11T21:14:32.981494+0000","last_active":"2024-03-11T21:15:58.124448+0000","last_peered":"2024-03-11T21:15:58.124448+0000","last_clean":"2024-03-11T21:15:58.124448+0000","last_became_active":"2024-03-11T20:50:30.111819+0000","last_became_peered":"2024-03-11T20:50:30.111819+0000","last_unstale":"2024-03-11T21:15:58.124448+0000","last_undegraded":"2024-03-11T21:15:58.124448+0000","last_fullsized":"2024-03-11T21:15:58.124448+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'53","last_scrub_stamp":"2024-03-11T21:14:32.981350+0000","last_deep_scrub":"102'25","last_deep_scrub_stamp":"2024-03-11T20:51:35.187094+0000","last_clean_scrub_stamp":"2024-03-11T21:14:32.981350+0000","objects_scrubbed":8,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:15:59.186276+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":61,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":857,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:57.103332+0000","last_change":"2024-03-11T21:14:51.040991+0000","last_active":"2024-03-11T21:15:57.103332+0000","last_peered":"2024-03-11T21:15:57.103332+0000","last_clean":"2024-03-11T21:15:57.103332+0000","last_became_active":"2024-03-11T21:00:48.362389+0000","last_became_peered":"2024-03-11T21:00:48.362389+0000","last_unstale":"2024-03-11T21:15:57.103332+0000","last_undegraded":"2024-03-11T21:15:57.103332+0000","last_fullsized":"2024-03-11T21:15:57.103332+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:14:51.040904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:14:51.040904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:16:06.166510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":39,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:56.684215+0000","last_change":"2024-03-11T21:14:50.707520+0000","last_active":"2024-03-11T21:15:56.684215+0000","last_peered":"2024-03-11T21:15:56.684215+0000","last_clean":"2024-03-11T21:15:56.684215+0000","last_became_active":"2024-03-11T21:14:50.706718+0000","last_became_peered":"2024-03-11T21:14:50.706718+0000","last_unstale":"2024-03-11T21:15:56.684215+0000","last_undegraded":"2024-03-11T21:15:56.684215+0000","last_fullsized":"2024-03-11T21:15:56.684215+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:12:47.959930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:55:58.888890+0000","last_clean_scrub_stamp":"2024-03-11T21:12:47.959930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":529,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:56.688394+0000","last_change":"2024-03-11T21:12:11.806426+0000","last_active":"2024-03-11T21:15:56.688394+0000","last_peered":"2024-03-11T21:15:56.688394+0000","last_clean":"2024-03-11T21:15:56.688394+0000","last_became_active":"2024-03-11T21:00:48.469015+0000","last_became_peered":"2024-03-11T21:00:48.469015+0000","last_unstale":"2024-03-11T21:15:56.688394+0000","last_undegraded":"2024-03-11T21:15:56.688394+0000","last_fullsized":"2024-03-11T21:15:56.688394+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:12:11.805941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:12:11.805941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"393'35","reported_seq":885,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:57.904080+0000","last_change":"2024-03-11T21:14:54.597458+0000","last_active":"2024-03-11T21:15:57.904080+0000","last_peered":"2024-03-11T21:15:57.904080+0000","last_clean":"2024-03-11T21:15:57.904080+0000","last_became_active":"2024-03-11T20:50:30.109549+0000","last_became_peered":"2024-03-11T20:50:30.109549+0000","last_unstale":"2024-03-11T21:15:57.904080+0000","last_undegraded":"2024-03-11T21:15:57.904080+0000","last_fullsized":"2024-03-11T21:15:57.904080+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"393'35","last_scrub_stamp":"2024-03-11T21:14:54.597298+0000","last_deep_scrub":"393'35","last_deep_scrub_stamp":"2024-03-11T21:12:35.497207+0000","last_clean_scrub_stamp":"2024-03-11T21:14:54.597298+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:16:12.710689+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":39,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:56.684763+0000","last_change":"2024-03-11T21:14:50.682831+0000","last_active":"2024-03-11T21:15:56.684763+0000","last_peered":"2024-03-11T21:15:56.684763+0000","last_clean":"2024-03-11T21:15:56.684763+0000","last_became_active":"2024-03-11T21:14:50.682439+0000","last_became_peered":"2024-03-11T21:14:50.682439+0000","last_unstale":"2024-03-11T21:15:56.684763+0000","last_undegraded":"2024-03-11T21:15:56.684763+0000","last_fullsized":"2024-03-11T21:15:56.684763+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:12:06.205727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:52:51.931295+0000","last_clean_scrub_stamp":"2024-03-11T21:12:06.205727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":114,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:57.903514+0000","last_change":"2024-03-11T21:14:50.427952+0000","last_active":"2024-03-11T21:15:57.903514+0000","last_peered":"2024-03-11T21:15:57.903514+0000","last_clean":"2024-03-11T21:15:57.903514+0000","last_became_active":"2024-03-11T21:14:50.427625+0000","last_became_peered":"2024-03-11T21:14:50.427625+0000","last_unstale":"2024-03-11T21:15:57.903514+0000","last_undegraded":"2024-03-11T21:15:57.903514+0000","last_fullsized":"2024-03-11T21:15:57.903514+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:14:28.376246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:00:14.030599+0000","last_clean_scrub_stamp":"2024-03-11T21:14:28.376246+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:16:02.901064+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":260,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:57.056160+0000","last_change":"2024-03-11T21:11:01.150241+0000","last_active":"2024-03-11T21:15:57.056160+0000","last_peered":"2024-03-11T21:15:57.056160+0000","last_clean":"2024-03-11T21:15:57.056160+0000","last_became_active":"2024-03-11T21:08:05.307984+0000","last_became_peered":"2024-03-11T21:08:05.307984+0000","last_unstale":"2024-03-11T21:15:57.056160+0000","last_undegraded":"2024-03-11T21:15:57.056160+0000","last_fullsized":"2024-03-11T21:15:57.056160+0000","mapping_epoch":319,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:11:01.150119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:11:01.150119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:15:59.287221+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.f","version":"391'46","reported_seq":937,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:58.124742+0000","last_change":"2024-03-11T21:14:50.458622+0000","last_active":"2024-03-11T21:15:58.124742+0000","last_peered":"2024-03-11T21:15:58.124742+0000","last_clean":"2024-03-11T21:15:58.124742+0000","last_became_active":"2024-03-11T21:14:50.427841+0000","last_became_peered":"2024-03-11T21:14:50.427841+0000","last_unstale":"2024-03-11T21:15:58.124742+0000","last_undegraded":"2024-03-11T21:15:58.124742+0000","last_fullsized":"2024-03-11T21:15:58.124742+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"391'46","last_scrub_stamp":"2024-03-11T21:12:31.145006+0000","last_deep_scrub":"391'46","last_deep_scrub_stamp":"2024-03-11T21:12:31.145006+0000","last_clean_scrub_stamp":"2024-03-11T21:12:31.145006+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:15:59.185200+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":831,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-11T21:15:57.902800+0000","last_change":"2024-03-11T21:14:58.654308+0000","last_active":"2024-03-11T21:15:57.902800+0000","last_peered":"2024-03-11T21:15:57.902800+0000","last_clean":"2024-03-11T21:15:57.902800+0000","last_became_active":"2024-03-11T20:50:32.652854+0000","last_became_peered":"2024-03-11T20:50:32.652854+0000","last_unstale":"2024-03-11T21:15:57.902800+0000","last_undegraded":"2024-03-11T21:15:57.902800+0000","last_fullsized":"2024-03-11T21:15:57.902800+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:14:58.654229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:59:00.946094+0000","last_clean_scrub_stamp":"2024-03-11T21:14:58.654229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:16:04.388711+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":20,"num_read_kb":20,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":2006,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1430,"ondisk_log_size":1430,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1543,"num_read_kb":1300,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":263,"num_bytes_recovered":4208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3632,"internal_metadata":0},"log_size":1078,"ondisk_log_size":1078,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1393,"num_read_kb":1388,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":68,"seq":292057776445,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":627876,"kb_used_data":1112,"kb_used_omap":1,"kb_used_meta":626750,"kb_avail":93117276,"statfs":{"total":95995035648,"available":95352090624,"internally_reserved":0,"allocated":1138688,"data_stored":802220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":641792458},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":57,"seq":244813136199,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":537296,"kb_used_data":776,"kb_used_omap":1,"kb_used_meta":536510,"kb_avail":93207856,"statfs":{"total":95995035648,"available":95444844544,"internally_reserved":0,"allocated":794624,"data_stored":474550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":549386695},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":211,"seq":906238099639,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":510492,"kb_used_data":784,"kb_used_omap":7,"kb_used_meta":509688,"kb_avail":93234660,"statfs":{"total":95995035648,"available":95472291840,"internally_reserved":0,"allocated":802816,"data_stored":474796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8078,"internal_metadata":521920626},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738757,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":595612,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":594814,"kb_avail":93149540,"statfs":{"total":95995035648,"available":95385128960,"internally_reserved":0,"allocated":802816,"data_stored":473482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":609089989},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378992,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":577808,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":576766,"kb_avail":93167344,"statfs":{"total":95995035648,"available":95403360256,"internally_reserved":0,"allocated":1052672,"data_stored":736332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":590608837},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":28,"seq":120259084645,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":668372,"kb_used_data":1352,"kb_used_omap":1,"kb_used_meta":667006,"kb_avail":93076780,"statfs":{"total":95995035648,"available":95310622720,"internally_reserved":0,"allocated":1384448,"data_stored":1064388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":683014600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":204,"seq":876173328572,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":424688,"kb_used_data":808,"kb_used_omap":4,"kb_used_meta":423867,"kb_avail":93320464,"statfs":{"total":95995035648,"available":95560155136,"internally_reserved":0,"allocated":827392,"data_stored":475410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4451,"internal_metadata":434040477},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":44,"seq":188978561361,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":546196,"kb_used_data":1352,"kb_used_omap":1,"kb_used_meta":544830,"kb_avail":93198956,"statfs":{"total":95995035648,"available":95435730944,"internally_reserved":0,"allocated":1384448,"data_stored":1063934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":557906373},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3632,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T21:16:05.019 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-03-11T21:16:05.019 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd pool set .mgr pg_num 21 2024-03-11T21:16:05.401 INFO:teuthology.orchestra.run.smithi023.stdout:ERROR: (22) Invalid argument 2024-03-11T21:16:05.401 INFO:teuthology.orchestra.run.smithi023.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:16:05.468 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:16:05.485 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:05 smithi023 bash[29146]: cluster 2024-03-11T21:16:03.870591+0000 osd.6 (osd.6) 452 : cluster 0 2.14 scrub starts 2024-03-11T21:16:05.485 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:05 smithi023 bash[29146]: cluster 2024-03-11T21:16:03.873049+0000 osd.6 (osd.6) 453 : cluster 0 2.14 scrub ok 2024-03-11T21:16:05.485 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:05 smithi023 bash[29146]: cluster 2024-03-11T21:16:03.931455+0000 osd.5 (osd.5) 545 : cluster 0 4.15 scrub starts 2024-03-11T21:16:05.485 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:05 smithi023 bash[29146]: cluster 2024-03-11T21:16:03.934540+0000 osd.5 (osd.5) 546 : cluster 0 4.15 scrub ok 2024-03-11T21:16:05.485 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:05 smithi023 bash[29146]: cluster 2024-03-11T21:16:04.072092+0000 osd.7 (osd.7) 569 : cluster 0 2.8 scrub starts 2024-03-11T21:16:05.486 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:05 smithi023 bash[29146]: cluster 2024-03-11T21:16:04.074659+0000 osd.7 (osd.7) 570 : cluster 0 2.8 scrub ok 2024-03-11T21:16:05.486 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:05 smithi023 bash[29146]: cluster 2024-03-11T21:16:04.470862+0000 mgr.y (mgr.24380) 934 : cluster 0 pgmap v1076: 142 pgs: 142 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:16:05.486 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:05 smithi023 bash[21372]: cluster 2024-03-11T21:16:03.870591+0000 osd.6 (osd.6) 452 : cluster 0 2.14 scrub starts 2024-03-11T21:16:05.486 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:05 smithi023 bash[21372]: cluster 2024-03-11T21:16:03.873049+0000 osd.6 (osd.6) 453 : cluster 0 2.14 scrub ok 2024-03-11T21:16:05.486 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:05 smithi023 bash[21372]: cluster 2024-03-11T21:16:03.931455+0000 osd.5 (osd.5) 545 : cluster 0 4.15 scrub starts 2024-03-11T21:16:05.486 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:05 smithi023 bash[21372]: cluster 2024-03-11T21:16:03.934540+0000 osd.5 (osd.5) 546 : cluster 0 4.15 scrub ok 2024-03-11T21:16:05.486 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:05 smithi023 bash[21372]: cluster 2024-03-11T21:16:04.072092+0000 osd.7 (osd.7) 569 : cluster 0 2.8 scrub starts 2024-03-11T21:16:05.486 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:05 smithi023 bash[21372]: cluster 2024-03-11T21:16:04.074659+0000 osd.7 (osd.7) 570 : cluster 0 2.8 scrub ok 2024-03-11T21:16:05.486 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:05 smithi023 bash[21372]: cluster 2024-03-11T21:16:04.470862+0000 mgr.y (mgr.24380) 934 : cluster 0 pgmap v1076: 142 pgs: 142 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:16:05.754 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:05 smithi146 bash[23809]: cluster 2024-03-11T21:16:03.870591+0000 osd.6 (osd.6) 452 : cluster 0 2.14 scrub starts 2024-03-11T21:16:05.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:05 smithi146 bash[23809]: cluster 2024-03-11T21:16:03.873049+0000 osd.6 (osd.6) 453 : cluster 0 2.14 scrub ok 2024-03-11T21:16:05.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:05 smithi146 bash[23809]: cluster 2024-03-11T21:16:03.931455+0000 osd.5 (osd.5) 545 : cluster 0 4.15 scrub starts 2024-03-11T21:16:05.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:05 smithi146 bash[23809]: cluster 2024-03-11T21:16:03.934540+0000 osd.5 (osd.5) 546 : cluster 0 4.15 scrub ok 2024-03-11T21:16:05.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:05 smithi146 bash[23809]: cluster 2024-03-11T21:16:04.072092+0000 osd.7 (osd.7) 569 : cluster 0 2.8 scrub starts 2024-03-11T21:16:05.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:05 smithi146 bash[23809]: cluster 2024-03-11T21:16:04.074659+0000 osd.7 (osd.7) 570 : cluster 0 2.8 scrub ok 2024-03-11T21:16:05.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:05 smithi146 bash[23809]: cluster 2024-03-11T21:16:04.470862+0000 mgr.y (mgr.24380) 934 : cluster 0 pgmap v1076: 142 pgs: 142 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:16:06.477 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:16:06 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:16:06.754 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:06 smithi146 bash[23809]: cluster 2024-03-11T21:16:04.898192+0000 osd.6 (osd.6) 454 : cluster 0 1.7 scrub starts 2024-03-11T21:16:06.754 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:06 smithi146 bash[23809]: cluster 2024-03-11T21:16:04.901005+0000 osd.6 (osd.6) 455 : cluster 0 1.7 scrub ok 2024-03-11T21:16:06.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:06 smithi146 bash[23809]: cluster 2024-03-11T21:16:04.901830+0000 osd.5 (osd.5) 547 : cluster 0 6.7 scrub starts 2024-03-11T21:16:06.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:06 smithi146 bash[23809]: cluster 2024-03-11T21:16:05.039027+0000 osd.7 (osd.7) 571 : cluster 0 6.14 scrub starts 2024-03-11T21:16:06.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:06 smithi146 bash[23809]: cluster 2024-03-11T21:16:05.040722+0000 osd.7 (osd.7) 572 : cluster 0 6.14 scrub ok 2024-03-11T21:16:06.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:06 smithi146 bash[23809]: cluster 2024-03-11T21:16:05.079414+0000 osd.5 (osd.5) 548 : cluster 0 6.7 scrub ok 2024-03-11T21:16:06.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:06 smithi146 bash[23809]: cluster 2024-03-11T21:16:05.516888+0000 osd.0 (osd.0) 406 : cluster 0 6.16 scrub starts 2024-03-11T21:16:06.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:06 smithi146 bash[23809]: cluster 2024-03-11T21:16:05.567301+0000 osd.0 (osd.0) 407 : cluster 0 6.16 scrub ok 2024-03-11T21:16:06.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:06 smithi146 bash[23809]: cluster 2024-03-11T21:16:06.005405+0000 osd.1 (osd.1) 310 : cluster 0 2.6 scrub starts 2024-03-11T21:16:06.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:06 smithi146 bash[23809]: cluster 2024-03-11T21:16:06.008999+0000 osd.1 (osd.1) 311 : cluster 0 2.6 scrub ok 2024-03-11T21:16:06.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:06 smithi023 bash[29146]: cluster 2024-03-11T21:16:04.898192+0000 osd.6 (osd.6) 454 : cluster 0 1.7 scrub starts 2024-03-11T21:16:06.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:06 smithi023 bash[29146]: cluster 2024-03-11T21:16:04.901005+0000 osd.6 (osd.6) 455 : cluster 0 1.7 scrub ok 2024-03-11T21:16:06.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:06 smithi023 bash[29146]: cluster 2024-03-11T21:16:04.901830+0000 osd.5 (osd.5) 547 : cluster 0 6.7 scrub starts 2024-03-11T21:16:06.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:06 smithi023 bash[29146]: cluster 2024-03-11T21:16:05.039027+0000 osd.7 (osd.7) 571 : cluster 0 6.14 scrub starts 2024-03-11T21:16:06.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:06 smithi023 bash[29146]: cluster 2024-03-11T21:16:05.040722+0000 osd.7 (osd.7) 572 : cluster 0 6.14 scrub ok 2024-03-11T21:16:06.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:06 smithi023 bash[29146]: cluster 2024-03-11T21:16:05.079414+0000 osd.5 (osd.5) 548 : cluster 0 6.7 scrub ok 2024-03-11T21:16:06.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:06 smithi023 bash[29146]: cluster 2024-03-11T21:16:05.516888+0000 osd.0 (osd.0) 406 : cluster 0 6.16 scrub starts 2024-03-11T21:16:06.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:06 smithi023 bash[29146]: cluster 2024-03-11T21:16:05.567301+0000 osd.0 (osd.0) 407 : cluster 0 6.16 scrub ok 2024-03-11T21:16:06.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:06 smithi023 bash[29146]: cluster 2024-03-11T21:16:06.005405+0000 osd.1 (osd.1) 310 : cluster 0 2.6 scrub starts 2024-03-11T21:16:06.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:06 smithi023 bash[29146]: cluster 2024-03-11T21:16:06.008999+0000 osd.1 (osd.1) 311 : cluster 0 2.6 scrub ok 2024-03-11T21:16:06.776 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:06 smithi023 bash[21372]: cluster 2024-03-11T21:16:04.898192+0000 osd.6 (osd.6) 454 : cluster 0 1.7 scrub starts 2024-03-11T21:16:06.776 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:06 smithi023 bash[21372]: cluster 2024-03-11T21:16:04.901005+0000 osd.6 (osd.6) 455 : cluster 0 1.7 scrub ok 2024-03-11T21:16:06.776 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:06 smithi023 bash[21372]: cluster 2024-03-11T21:16:04.901830+0000 osd.5 (osd.5) 547 : cluster 0 6.7 scrub starts 2024-03-11T21:16:06.776 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:06 smithi023 bash[21372]: cluster 2024-03-11T21:16:05.039027+0000 osd.7 (osd.7) 571 : cluster 0 6.14 scrub starts 2024-03-11T21:16:06.776 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:06 smithi023 bash[21372]: cluster 2024-03-11T21:16:05.040722+0000 osd.7 (osd.7) 572 : cluster 0 6.14 scrub ok 2024-03-11T21:16:06.777 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:06 smithi023 bash[21372]: cluster 2024-03-11T21:16:05.079414+0000 osd.5 (osd.5) 548 : cluster 0 6.7 scrub ok 2024-03-11T21:16:06.777 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:06 smithi023 bash[21372]: cluster 2024-03-11T21:16:05.516888+0000 osd.0 (osd.0) 406 : cluster 0 6.16 scrub starts 2024-03-11T21:16:06.777 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:06 smithi023 bash[21372]: cluster 2024-03-11T21:16:05.567301+0000 osd.0 (osd.0) 407 : cluster 0 6.16 scrub ok 2024-03-11T21:16:06.777 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:06 smithi023 bash[21372]: cluster 2024-03-11T21:16:06.005405+0000 osd.1 (osd.1) 310 : cluster 0 2.6 scrub starts 2024-03-11T21:16:06.777 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:06 smithi023 bash[21372]: cluster 2024-03-11T21:16:06.008999+0000 osd.1 (osd.1) 311 : cluster 0 2.6 scrub ok 2024-03-11T21:16:06.829 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-03-11T21:16:07.754 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:07 smithi146 bash[23809]: cluster 2024-03-11T21:16:05.919982+0000 osd.5 (osd.5) 549 : cluster 0 4.d scrub starts 2024-03-11T21:16:07.754 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:07 smithi146 bash[23809]: cluster 2024-03-11T21:16:05.923334+0000 osd.5 (osd.5) 550 : cluster 0 4.d scrub ok 2024-03-11T21:16:07.754 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:07 smithi146 bash[23809]: audit 2024-03-11T21:16:06.127638+0000 mgr.y (mgr.24380) 935 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:16:07.754 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:07 smithi146 bash[23809]: cluster 2024-03-11T21:16:06.472340+0000 mgr.y (mgr.24380) 936 : cluster 0 pgmap v1077: 142 pgs: 142 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T21:16:07.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:07 smithi146 bash[23809]: audit 2024-03-11T21:16:06.640670+0000 mon.a (mon.0) 1975 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:16:07.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:07 smithi023 bash[21372]: cluster 2024-03-11T21:16:05.919982+0000 osd.5 (osd.5) 549 : cluster 0 4.d scrub starts 2024-03-11T21:16:07.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:07 smithi023 bash[21372]: cluster 2024-03-11T21:16:05.923334+0000 osd.5 (osd.5) 550 : cluster 0 4.d scrub ok 2024-03-11T21:16:07.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:07 smithi023 bash[21372]: audit 2024-03-11T21:16:06.127638+0000 mgr.y (mgr.24380) 935 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:16:07.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:07 smithi023 bash[21372]: cluster 2024-03-11T21:16:06.472340+0000 mgr.y (mgr.24380) 936 : cluster 0 pgmap v1077: 142 pgs: 142 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T21:16:07.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:07 smithi023 bash[21372]: audit 2024-03-11T21:16:06.640670+0000 mon.a (mon.0) 1975 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:16:07.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:07 smithi023 bash[29146]: cluster 2024-03-11T21:16:05.919982+0000 osd.5 (osd.5) 549 : cluster 0 4.d scrub starts 2024-03-11T21:16:07.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:07 smithi023 bash[29146]: cluster 2024-03-11T21:16:05.923334+0000 osd.5 (osd.5) 550 : cluster 0 4.d scrub ok 2024-03-11T21:16:07.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:07 smithi023 bash[29146]: audit 2024-03-11T21:16:06.127638+0000 mgr.y (mgr.24380) 935 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:16:07.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:07 smithi023 bash[29146]: cluster 2024-03-11T21:16:06.472340+0000 mgr.y (mgr.24380) 936 : cluster 0 pgmap v1077: 142 pgs: 142 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T21:16:07.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:07 smithi023 bash[29146]: audit 2024-03-11T21:16:06.640670+0000 mon.a (mon.0) 1975 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:16:08.739 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:16:08 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:16:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:16:08.740 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:08 smithi023 bash[21372]: cluster 2024-03-11T21:16:06.884587+0000 osd.5 (osd.5) 551 : cluster 0 3.1c scrub starts 2024-03-11T21:16:08.740 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:08 smithi023 bash[21372]: cluster 2024-03-11T21:16:06.889064+0000 osd.5 (osd.5) 552 : cluster 0 3.1c scrub ok 2024-03-11T21:16:08.740 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:08 smithi023 bash[21372]: cluster 2024-03-11T21:16:07.466644+0000 osd.0 (osd.0) 408 : cluster 0 3.6 scrub starts 2024-03-11T21:16:08.740 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:08 smithi023 bash[21372]: cluster 2024-03-11T21:16:07.471191+0000 osd.0 (osd.0) 409 : cluster 0 3.6 scrub ok 2024-03-11T21:16:08.740 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:08 smithi023 bash[21372]: cluster 2024-03-11T21:16:07.966234+0000 osd.1 (osd.1) 312 : cluster 0 1.3 scrub starts 2024-03-11T21:16:08.740 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:08 smithi023 bash[21372]: cluster 2024-03-11T21:16:07.967333+0000 osd.6 (osd.6) 456 : cluster 0 1.6 scrub starts 2024-03-11T21:16:08.740 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:08 smithi023 bash[21372]: cluster 2024-03-11T21:16:07.969957+0000 osd.1 (osd.1) 313 : cluster 0 1.3 scrub ok 2024-03-11T21:16:08.740 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:08 smithi023 bash[21372]: cluster 2024-03-11T21:16:07.970011+0000 osd.6 (osd.6) 457 : cluster 0 1.6 scrub ok 2024-03-11T21:16:08.741 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:08 smithi023 bash[29146]: cluster 2024-03-11T21:16:06.884587+0000 osd.5 (osd.5) 551 : cluster 0 3.1c scrub starts 2024-03-11T21:16:08.741 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:08 smithi023 bash[29146]: cluster 2024-03-11T21:16:06.889064+0000 osd.5 (osd.5) 552 : cluster 0 3.1c scrub ok 2024-03-11T21:16:08.741 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:08 smithi023 bash[29146]: cluster 2024-03-11T21:16:07.466644+0000 osd.0 (osd.0) 408 : cluster 0 3.6 scrub starts 2024-03-11T21:16:08.741 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:08 smithi023 bash[29146]: cluster 2024-03-11T21:16:07.471191+0000 osd.0 (osd.0) 409 : cluster 0 3.6 scrub ok 2024-03-11T21:16:08.741 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:08 smithi023 bash[29146]: cluster 2024-03-11T21:16:07.966234+0000 osd.1 (osd.1) 312 : cluster 0 1.3 scrub starts 2024-03-11T21:16:08.741 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:08 smithi023 bash[29146]: cluster 2024-03-11T21:16:07.967333+0000 osd.6 (osd.6) 456 : cluster 0 1.6 scrub starts 2024-03-11T21:16:08.741 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:08 smithi023 bash[29146]: cluster 2024-03-11T21:16:07.969957+0000 osd.1 (osd.1) 313 : cluster 0 1.3 scrub ok 2024-03-11T21:16:08.741 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:08 smithi023 bash[29146]: cluster 2024-03-11T21:16:07.970011+0000 osd.6 (osd.6) 457 : cluster 0 1.6 scrub ok 2024-03-11T21:16:09.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:08 smithi146 bash[23809]: cluster 2024-03-11T21:16:06.884587+0000 osd.5 (osd.5) 551 : cluster 0 3.1c scrub starts 2024-03-11T21:16:09.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:08 smithi146 bash[23809]: cluster 2024-03-11T21:16:06.889064+0000 osd.5 (osd.5) 552 : cluster 0 3.1c scrub ok 2024-03-11T21:16:09.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:08 smithi146 bash[23809]: cluster 2024-03-11T21:16:07.466644+0000 osd.0 (osd.0) 408 : cluster 0 3.6 scrub starts 2024-03-11T21:16:09.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:08 smithi146 bash[23809]: cluster 2024-03-11T21:16:07.471191+0000 osd.0 (osd.0) 409 : cluster 0 3.6 scrub ok 2024-03-11T21:16:09.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:08 smithi146 bash[23809]: cluster 2024-03-11T21:16:07.966234+0000 osd.1 (osd.1) 312 : cluster 0 1.3 scrub starts 2024-03-11T21:16:09.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:08 smithi146 bash[23809]: cluster 2024-03-11T21:16:07.967333+0000 osd.6 (osd.6) 456 : cluster 0 1.6 scrub starts 2024-03-11T21:16:09.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:08 smithi146 bash[23809]: cluster 2024-03-11T21:16:07.969957+0000 osd.1 (osd.1) 313 : cluster 0 1.3 scrub ok 2024-03-11T21:16:09.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:08 smithi146 bash[23809]: cluster 2024-03-11T21:16:07.970011+0000 osd.6 (osd.6) 457 : cluster 0 1.6 scrub ok 2024-03-11T21:16:09.544 INFO:teuthology.orchestra.run.smithi023.stderr:noscrub is unset 2024-03-11T21:16:10.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:09 smithi146 bash[23809]: cluster 2024-03-11T21:16:07.917784+0000 osd.5 (osd.5) 553 : cluster 0 2.2 scrub starts 2024-03-11T21:16:10.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:09 smithi146 bash[23809]: cluster 2024-03-11T21:16:07.920011+0000 osd.5 (osd.5) 554 : cluster 0 2.2 scrub ok 2024-03-11T21:16:10.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:09 smithi146 bash[23809]: cluster 2024-03-11T21:16:08.437355+0000 osd.3 (osd.3) 498 : cluster 0 2.1b scrub starts 2024-03-11T21:16:10.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:09 smithi146 bash[23809]: cluster 2024-03-11T21:16:08.440066+0000 osd.3 (osd.3) 499 : cluster 0 2.1b scrub ok 2024-03-11T21:16:10.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:09 smithi146 bash[23809]: cluster 2024-03-11T21:16:08.473257+0000 mgr.y (mgr.24380) 937 : cluster 0 pgmap v1078: 142 pgs: 142 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:16:10.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:09 smithi146 bash[23809]: cluster 2024-03-11T21:16:08.969562+0000 osd.1 (osd.1) 314 : cluster 0 4.4 scrub starts 2024-03-11T21:16:10.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:09 smithi146 bash[23809]: cluster 2024-03-11T21:16:08.970768+0000 osd.1 (osd.1) 315 : cluster 0 4.4 scrub ok 2024-03-11T21:16:10.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:09 smithi146 bash[23809]: audit 2024-03-11T21:16:09.189492+0000 mon.c (mon.2) 103 : audit 1 from='client.? 172.21.15.23:0/4251054247' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:16:10.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:09 smithi146 bash[23809]: audit 2024-03-11T21:16:09.190189+0000 mon.a (mon.0) 1976 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:16:10.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:09 smithi023 bash[21372]: cluster 2024-03-11T21:16:07.917784+0000 osd.5 (osd.5) 553 : cluster 0 2.2 scrub starts 2024-03-11T21:16:10.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:09 smithi023 bash[21372]: cluster 2024-03-11T21:16:07.920011+0000 osd.5 (osd.5) 554 : cluster 0 2.2 scrub ok 2024-03-11T21:16:10.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:09 smithi023 bash[21372]: cluster 2024-03-11T21:16:08.437355+0000 osd.3 (osd.3) 498 : cluster 0 2.1b scrub starts 2024-03-11T21:16:10.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:09 smithi023 bash[21372]: cluster 2024-03-11T21:16:08.440066+0000 osd.3 (osd.3) 499 : cluster 0 2.1b scrub ok 2024-03-11T21:16:10.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:09 smithi023 bash[21372]: cluster 2024-03-11T21:16:08.473257+0000 mgr.y (mgr.24380) 937 : cluster 0 pgmap v1078: 142 pgs: 142 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:16:10.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:09 smithi023 bash[21372]: cluster 2024-03-11T21:16:08.969562+0000 osd.1 (osd.1) 314 : cluster 0 4.4 scrub starts 2024-03-11T21:16:10.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:09 smithi023 bash[21372]: cluster 2024-03-11T21:16:08.970768+0000 osd.1 (osd.1) 315 : cluster 0 4.4 scrub ok 2024-03-11T21:16:10.026 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:09 smithi023 bash[21372]: audit 2024-03-11T21:16:09.189492+0000 mon.c (mon.2) 103 : audit 1 from='client.? 172.21.15.23:0/4251054247' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:16:10.026 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:09 smithi023 bash[21372]: audit 2024-03-11T21:16:09.190189+0000 mon.a (mon.0) 1976 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:16:10.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:09 smithi023 bash[29146]: cluster 2024-03-11T21:16:07.917784+0000 osd.5 (osd.5) 553 : cluster 0 2.2 scrub starts 2024-03-11T21:16:10.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:09 smithi023 bash[29146]: cluster 2024-03-11T21:16:07.920011+0000 osd.5 (osd.5) 554 : cluster 0 2.2 scrub ok 2024-03-11T21:16:10.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:09 smithi023 bash[29146]: cluster 2024-03-11T21:16:08.437355+0000 osd.3 (osd.3) 498 : cluster 0 2.1b scrub starts 2024-03-11T21:16:10.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:09 smithi023 bash[29146]: cluster 2024-03-11T21:16:08.440066+0000 osd.3 (osd.3) 499 : cluster 0 2.1b scrub ok 2024-03-11T21:16:10.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:09 smithi023 bash[29146]: cluster 2024-03-11T21:16:08.473257+0000 mgr.y (mgr.24380) 937 : cluster 0 pgmap v1078: 142 pgs: 142 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:16:10.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:09 smithi023 bash[29146]: cluster 2024-03-11T21:16:08.969562+0000 osd.1 (osd.1) 314 : cluster 0 4.4 scrub starts 2024-03-11T21:16:10.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:09 smithi023 bash[29146]: cluster 2024-03-11T21:16:08.970768+0000 osd.1 (osd.1) 315 : cluster 0 4.4 scrub ok 2024-03-11T21:16:10.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:09 smithi023 bash[29146]: audit 2024-03-11T21:16:09.189492+0000 mon.c (mon.2) 103 : audit 1 from='client.? 172.21.15.23:0/4251054247' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:16:10.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:09 smithi023 bash[29146]: audit 2024-03-11T21:16:09.190189+0000 mon.a (mon.0) 1976 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:16:10.356 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-11T21:16:10.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:10 smithi023 bash[21372]: cluster 2024-03-11T21:16:09.051853+0000 osd.7 (osd.7) 573 : cluster 0 3.f scrub starts 2024-03-11T21:16:10.776 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:10 smithi023 bash[21372]: cluster 2024-03-11T21:16:09.057256+0000 osd.7 (osd.7) 574 : cluster 0 3.f scrub ok 2024-03-11T21:16:10.776 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:10 smithi023 bash[21372]: cluster 2024-03-11T21:16:09.467129+0000 osd.3 (osd.3) 500 : cluster 0 2.e scrub starts 2024-03-11T21:16:10.776 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:10 smithi023 bash[21372]: cluster 2024-03-11T21:16:09.469663+0000 osd.3 (osd.3) 501 : cluster 0 2.e scrub ok 2024-03-11T21:16:10.776 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:10 smithi023 bash[21372]: audit 2024-03-11T21:16:09.530903+0000 mon.a (mon.0) 1977 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:16:10.776 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:10 smithi023 bash[21372]: cluster 2024-03-11T21:16:09.545114+0000 mon.a (mon.0) 1978 : cluster 0 osdmap e440: 8 total, 8 up, 8 in 2024-03-11T21:16:10.776 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:10 smithi023 bash[21372]: cluster 2024-03-11T21:16:10.005409+0000 osd.1 (osd.1) 316 : cluster 0 3.0 scrub starts 2024-03-11T21:16:10.776 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:10 smithi023 bash[21372]: cluster 2024-03-11T21:16:10.012191+0000 osd.1 (osd.1) 317 : cluster 0 3.0 scrub ok 2024-03-11T21:16:10.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:10 smithi023 bash[29146]: cluster 2024-03-11T21:16:09.051853+0000 osd.7 (osd.7) 573 : cluster 0 3.f scrub starts 2024-03-11T21:16:10.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:10 smithi023 bash[29146]: cluster 2024-03-11T21:16:09.057256+0000 osd.7 (osd.7) 574 : cluster 0 3.f scrub ok 2024-03-11T21:16:10.777 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:10 smithi023 bash[29146]: cluster 2024-03-11T21:16:09.467129+0000 osd.3 (osd.3) 500 : cluster 0 2.e scrub starts 2024-03-11T21:16:10.777 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:10 smithi023 bash[29146]: cluster 2024-03-11T21:16:09.469663+0000 osd.3 (osd.3) 501 : cluster 0 2.e scrub ok 2024-03-11T21:16:10.777 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:10 smithi023 bash[29146]: audit 2024-03-11T21:16:09.530903+0000 mon.a (mon.0) 1977 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:16:10.777 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:10 smithi023 bash[29146]: cluster 2024-03-11T21:16:09.545114+0000 mon.a (mon.0) 1978 : cluster 0 osdmap e440: 8 total, 8 up, 8 in 2024-03-11T21:16:10.777 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:10 smithi023 bash[29146]: cluster 2024-03-11T21:16:10.005409+0000 osd.1 (osd.1) 316 : cluster 0 3.0 scrub starts 2024-03-11T21:16:10.777 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:10 smithi023 bash[29146]: cluster 2024-03-11T21:16:10.012191+0000 osd.1 (osd.1) 317 : cluster 0 3.0 scrub ok 2024-03-11T21:16:10.829 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:16:11.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:10 smithi146 bash[23809]: cluster 2024-03-11T21:16:09.051853+0000 osd.7 (osd.7) 573 : cluster 0 3.f scrub starts 2024-03-11T21:16:11.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:10 smithi146 bash[23809]: cluster 2024-03-11T21:16:09.057256+0000 osd.7 (osd.7) 574 : cluster 0 3.f scrub ok 2024-03-11T21:16:11.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:10 smithi146 bash[23809]: cluster 2024-03-11T21:16:09.467129+0000 osd.3 (osd.3) 500 : cluster 0 2.e scrub starts 2024-03-11T21:16:11.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:10 smithi146 bash[23809]: cluster 2024-03-11T21:16:09.469663+0000 osd.3 (osd.3) 501 : cluster 0 2.e scrub ok 2024-03-11T21:16:11.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:10 smithi146 bash[23809]: audit 2024-03-11T21:16:09.530903+0000 mon.a (mon.0) 1977 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:16:11.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:10 smithi146 bash[23809]: cluster 2024-03-11T21:16:09.545114+0000 mon.a (mon.0) 1978 : cluster 0 osdmap e440: 8 total, 8 up, 8 in 2024-03-11T21:16:11.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:10 smithi146 bash[23809]: cluster 2024-03-11T21:16:10.005409+0000 osd.1 (osd.1) 316 : cluster 0 3.0 scrub starts 2024-03-11T21:16:11.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:10 smithi146 bash[23809]: cluster 2024-03-11T21:16:10.012191+0000 osd.1 (osd.1) 317 : cluster 0 3.0 scrub ok 2024-03-11T21:16:11.851 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:16:11.869 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:11 smithi023 bash[21372]: cluster 2024-03-11T21:16:10.004175+0000 osd.5 (osd.5) 555 : cluster 0 1.1 scrub starts 2024-03-11T21:16:11.870 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:11 smithi023 bash[21372]: cluster 2024-03-11T21:16:10.007035+0000 osd.5 (osd.5) 556 : cluster 0 1.1 scrub ok 2024-03-11T21:16:11.870 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:11 smithi023 bash[21372]: cluster 2024-03-11T21:16:10.415762+0000 osd.0 (osd.0) 410 : cluster 0 3.17 scrub starts 2024-03-11T21:16:11.870 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:11 smithi023 bash[21372]: cluster 2024-03-11T21:16:10.419063+0000 osd.0 (osd.0) 411 : cluster 0 3.17 scrub ok 2024-03-11T21:16:11.870 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:11 smithi023 bash[21372]: cluster 2024-03-11T21:16:10.471754+0000 osd.3 (osd.3) 502 : cluster 0 6.1b scrub starts 2024-03-11T21:16:11.870 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:11 smithi023 bash[21372]: cluster 2024-03-11T21:16:10.474109+0000 mgr.y (mgr.24380) 938 : cluster 0 pgmap v1080: 142 pgs: 142 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 638 B/s rd, 0 op/s 2024-03-11T21:16:11.870 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:11 smithi023 bash[21372]: cluster 2024-03-11T21:16:10.650602+0000 osd.3 (osd.3) 503 : cluster 0 6.1b scrub ok 2024-03-11T21:16:11.870 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:11 smithi023 bash[21372]: cluster 2024-03-11T21:16:10.985185+0000 osd.6 (osd.6) 458 : cluster 0 2.11 scrub starts 2024-03-11T21:16:11.870 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:11 smithi023 bash[21372]: cluster 2024-03-11T21:16:10.988474+0000 osd.6 (osd.6) 459 : cluster 0 2.11 scrub ok 2024-03-11T21:16:11.870 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:11 smithi023 bash[21372]: cluster 2024-03-11T21:16:10.998402+0000 osd.1 (osd.1) 318 : cluster 0 2.15 scrub starts 2024-03-11T21:16:11.870 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:11 smithi023 bash[21372]: cluster 2024-03-11T21:16:11.000823+0000 osd.1 (osd.1) 319 : cluster 0 2.15 scrub ok 2024-03-11T21:16:11.870 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:11 smithi023 bash[29146]: cluster 2024-03-11T21:16:10.004175+0000 osd.5 (osd.5) 555 : cluster 0 1.1 scrub starts 2024-03-11T21:16:11.871 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:11 smithi023 bash[29146]: cluster 2024-03-11T21:16:10.007035+0000 osd.5 (osd.5) 556 : cluster 0 1.1 scrub ok 2024-03-11T21:16:11.871 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:11 smithi023 bash[29146]: cluster 2024-03-11T21:16:10.415762+0000 osd.0 (osd.0) 410 : cluster 0 3.17 scrub starts 2024-03-11T21:16:11.871 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:11 smithi023 bash[29146]: cluster 2024-03-11T21:16:10.419063+0000 osd.0 (osd.0) 411 : cluster 0 3.17 scrub ok 2024-03-11T21:16:11.871 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:11 smithi023 bash[29146]: cluster 2024-03-11T21:16:10.471754+0000 osd.3 (osd.3) 502 : cluster 0 6.1b scrub starts 2024-03-11T21:16:11.871 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:11 smithi023 bash[29146]: cluster 2024-03-11T21:16:10.474109+0000 mgr.y (mgr.24380) 938 : cluster 0 pgmap v1080: 142 pgs: 142 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 638 B/s rd, 0 op/s 2024-03-11T21:16:11.871 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:11 smithi023 bash[29146]: cluster 2024-03-11T21:16:10.650602+0000 osd.3 (osd.3) 503 : cluster 0 6.1b scrub ok 2024-03-11T21:16:11.871 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:11 smithi023 bash[29146]: cluster 2024-03-11T21:16:10.985185+0000 osd.6 (osd.6) 458 : cluster 0 2.11 scrub starts 2024-03-11T21:16:11.871 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:11 smithi023 bash[29146]: cluster 2024-03-11T21:16:10.988474+0000 osd.6 (osd.6) 459 : cluster 0 2.11 scrub ok 2024-03-11T21:16:11.871 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:11 smithi023 bash[29146]: cluster 2024-03-11T21:16:10.998402+0000 osd.1 (osd.1) 318 : cluster 0 2.15 scrub starts 2024-03-11T21:16:11.871 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:11 smithi023 bash[29146]: cluster 2024-03-11T21:16:11.000823+0000 osd.1 (osd.1) 319 : cluster 0 2.15 scrub ok 2024-03-11T21:16:12.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:11 smithi146 bash[23809]: cluster 2024-03-11T21:16:10.004175+0000 osd.5 (osd.5) 555 : cluster 0 1.1 scrub starts 2024-03-11T21:16:12.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:11 smithi146 bash[23809]: cluster 2024-03-11T21:16:10.007035+0000 osd.5 (osd.5) 556 : cluster 0 1.1 scrub ok 2024-03-11T21:16:12.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:11 smithi146 bash[23809]: cluster 2024-03-11T21:16:10.415762+0000 osd.0 (osd.0) 410 : cluster 0 3.17 scrub starts 2024-03-11T21:16:12.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:11 smithi146 bash[23809]: cluster 2024-03-11T21:16:10.419063+0000 osd.0 (osd.0) 411 : cluster 0 3.17 scrub ok 2024-03-11T21:16:12.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:11 smithi146 bash[23809]: cluster 2024-03-11T21:16:10.471754+0000 osd.3 (osd.3) 502 : cluster 0 6.1b scrub starts 2024-03-11T21:16:12.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:11 smithi146 bash[23809]: cluster 2024-03-11T21:16:10.474109+0000 mgr.y (mgr.24380) 938 : cluster 0 pgmap v1080: 142 pgs: 142 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 638 B/s rd, 0 op/s 2024-03-11T21:16:12.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:11 smithi146 bash[23809]: cluster 2024-03-11T21:16:10.650602+0000 osd.3 (osd.3) 503 : cluster 0 6.1b scrub ok 2024-03-11T21:16:12.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:11 smithi146 bash[23809]: cluster 2024-03-11T21:16:10.985185+0000 osd.6 (osd.6) 458 : cluster 0 2.11 scrub starts 2024-03-11T21:16:12.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:11 smithi146 bash[23809]: cluster 2024-03-11T21:16:10.988474+0000 osd.6 (osd.6) 459 : cluster 0 2.11 scrub ok 2024-03-11T21:16:12.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:11 smithi146 bash[23809]: cluster 2024-03-11T21:16:10.998402+0000 osd.1 (osd.1) 318 : cluster 0 2.15 scrub starts 2024-03-11T21:16:12.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:11 smithi146 bash[23809]: cluster 2024-03-11T21:16:11.000823+0000 osd.1 (osd.1) 319 : cluster 0 2.15 scrub ok 2024-03-11T21:16:12.856 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:12 smithi023 bash[21372]: cluster 2024-03-11T21:16:11.405364+0000 osd.0 (osd.0) 412 : cluster 0 3.1 scrub starts 2024-03-11T21:16:12.856 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:12 smithi023 bash[21372]: cluster 2024-03-11T21:16:11.410263+0000 osd.0 (osd.0) 413 : cluster 0 3.1 scrub ok 2024-03-11T21:16:12.856 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:12 smithi023 bash[21372]: cluster 2024-03-11T21:16:12.027403+0000 osd.5 (osd.5) 557 : cluster 0 1.a scrub starts 2024-03-11T21:16:12.856 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:12 smithi023 bash[21372]: cluster 2024-03-11T21:16:12.030094+0000 osd.5 (osd.5) 558 : cluster 0 1.a scrub ok 2024-03-11T21:16:12.856 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:12 smithi023 bash[21372]: cluster 2024-03-11T21:16:12.363276+0000 osd.0 (osd.0) 414 : cluster 0 4.3 scrub starts 2024-03-11T21:16:12.856 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:12 smithi023 bash[21372]: cluster 2024-03-11T21:16:12.367119+0000 osd.0 (osd.0) 415 : cluster 0 4.3 scrub ok 2024-03-11T21:16:12.856 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:12 smithi023 bash[29146]: cluster 2024-03-11T21:16:11.405364+0000 osd.0 (osd.0) 412 : cluster 0 3.1 scrub starts 2024-03-11T21:16:12.856 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:12 smithi023 bash[29146]: cluster 2024-03-11T21:16:11.410263+0000 osd.0 (osd.0) 413 : cluster 0 3.1 scrub ok 2024-03-11T21:16:12.856 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:12 smithi023 bash[29146]: cluster 2024-03-11T21:16:12.027403+0000 osd.5 (osd.5) 557 : cluster 0 1.a scrub starts 2024-03-11T21:16:12.856 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:12 smithi023 bash[29146]: cluster 2024-03-11T21:16:12.030094+0000 osd.5 (osd.5) 558 : cluster 0 1.a scrub ok 2024-03-11T21:16:12.856 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:12 smithi023 bash[29146]: cluster 2024-03-11T21:16:12.363276+0000 osd.0 (osd.0) 414 : cluster 0 4.3 scrub starts 2024-03-11T21:16:12.857 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:12 smithi023 bash[29146]: cluster 2024-03-11T21:16:12.367119+0000 osd.0 (osd.0) 415 : cluster 0 4.3 scrub ok 2024-03-11T21:16:13.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:12 smithi146 bash[23809]: cluster 2024-03-11T21:16:11.405364+0000 osd.0 (osd.0) 412 : cluster 0 3.1 scrub starts 2024-03-11T21:16:13.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:12 smithi146 bash[23809]: cluster 2024-03-11T21:16:11.410263+0000 osd.0 (osd.0) 413 : cluster 0 3.1 scrub ok 2024-03-11T21:16:13.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:12 smithi146 bash[23809]: cluster 2024-03-11T21:16:12.027403+0000 osd.5 (osd.5) 557 : cluster 0 1.a scrub starts 2024-03-11T21:16:13.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:12 smithi146 bash[23809]: cluster 2024-03-11T21:16:12.030094+0000 osd.5 (osd.5) 558 : cluster 0 1.a scrub ok 2024-03-11T21:16:13.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:12 smithi146 bash[23809]: cluster 2024-03-11T21:16:12.363276+0000 osd.0 (osd.0) 414 : cluster 0 4.3 scrub starts 2024-03-11T21:16:13.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:12 smithi146 bash[23809]: cluster 2024-03-11T21:16:12.367119+0000 osd.0 (osd.0) 415 : cluster 0 4.3 scrub ok 2024-03-11T21:16:14.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:13 smithi146 bash[23809]: cluster 2024-03-11T21:16:12.078763+0000 osd.7 (osd.7) 575 : cluster 0 6.15 scrub starts 2024-03-11T21:16:14.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:13 smithi146 bash[23809]: cluster 2024-03-11T21:16:12.188103+0000 osd.7 (osd.7) 576 : cluster 0 6.15 scrub ok 2024-03-11T21:16:14.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:13 smithi146 bash[23809]: cluster 2024-03-11T21:16:12.475521+0000 mgr.y (mgr.24380) 939 : cluster 0 pgmap v1081: 142 pgs: 142 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-03-11T21:16:14.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:13 smithi146 bash[23809]: cluster 2024-03-11T21:16:12.702925+0000 osd.5 (osd.5) 559 : cluster 0 purged_snaps scrub starts 2024-03-11T21:16:14.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:13 smithi146 bash[23809]: cluster 2024-03-11T21:16:12.703034+0000 osd.5 (osd.5) 560 : cluster 0 purged_snaps scrub ok 2024-03-11T21:16:14.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:13 smithi146 bash[23809]: cluster 2024-03-11T21:16:12.940992+0000 osd.6 (osd.6) 460 : cluster 0 2.1a scrub starts 2024-03-11T21:16:14.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:13 smithi146 bash[23809]: cluster 2024-03-11T21:16:12.943359+0000 osd.6 (osd.6) 461 : cluster 0 2.1a scrub ok 2024-03-11T21:16:14.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:13 smithi023 bash[21372]: cluster 2024-03-11T21:16:12.078763+0000 osd.7 (osd.7) 575 : cluster 0 6.15 scrub starts 2024-03-11T21:16:14.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:13 smithi023 bash[21372]: cluster 2024-03-11T21:16:12.188103+0000 osd.7 (osd.7) 576 : cluster 0 6.15 scrub ok 2024-03-11T21:16:14.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:13 smithi023 bash[21372]: cluster 2024-03-11T21:16:12.475521+0000 mgr.y (mgr.24380) 939 : cluster 0 pgmap v1081: 142 pgs: 142 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-03-11T21:16:14.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:13 smithi023 bash[21372]: cluster 2024-03-11T21:16:12.702925+0000 osd.5 (osd.5) 559 : cluster 0 purged_snaps scrub starts 2024-03-11T21:16:14.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:13 smithi023 bash[21372]: cluster 2024-03-11T21:16:12.703034+0000 osd.5 (osd.5) 560 : cluster 0 purged_snaps scrub ok 2024-03-11T21:16:14.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:13 smithi023 bash[21372]: cluster 2024-03-11T21:16:12.940992+0000 osd.6 (osd.6) 460 : cluster 0 2.1a scrub starts 2024-03-11T21:16:14.026 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:13 smithi023 bash[21372]: cluster 2024-03-11T21:16:12.943359+0000 osd.6 (osd.6) 461 : cluster 0 2.1a scrub ok 2024-03-11T21:16:14.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:13 smithi023 bash[29146]: cluster 2024-03-11T21:16:12.078763+0000 osd.7 (osd.7) 575 : cluster 0 6.15 scrub starts 2024-03-11T21:16:14.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:13 smithi023 bash[29146]: cluster 2024-03-11T21:16:12.188103+0000 osd.7 (osd.7) 576 : cluster 0 6.15 scrub ok 2024-03-11T21:16:14.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:13 smithi023 bash[29146]: cluster 2024-03-11T21:16:12.475521+0000 mgr.y (mgr.24380) 939 : cluster 0 pgmap v1081: 142 pgs: 142 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-03-11T21:16:14.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:13 smithi023 bash[29146]: cluster 2024-03-11T21:16:12.702925+0000 osd.5 (osd.5) 559 : cluster 0 purged_snaps scrub starts 2024-03-11T21:16:14.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:13 smithi023 bash[29146]: cluster 2024-03-11T21:16:12.703034+0000 osd.5 (osd.5) 560 : cluster 0 purged_snaps scrub ok 2024-03-11T21:16:14.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:13 smithi023 bash[29146]: cluster 2024-03-11T21:16:12.940992+0000 osd.6 (osd.6) 460 : cluster 0 2.1a scrub starts 2024-03-11T21:16:14.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:13 smithi023 bash[29146]: cluster 2024-03-11T21:16:12.943359+0000 osd.6 (osd.6) 461 : cluster 0 2.1a scrub ok 2024-03-11T21:16:15.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:14 smithi146 bash[23809]: cluster 2024-03-11T21:16:13.496051+0000 osd.3 (osd.3) 504 : cluster 0 4.c scrub starts 2024-03-11T21:16:15.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:14 smithi146 bash[23809]: cluster 2024-03-11T21:16:13.498302+0000 osd.3 (osd.3) 505 : cluster 0 4.c scrub ok 2024-03-11T21:16:15.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:14 smithi146 bash[23809]: cluster 2024-03-11T21:16:13.614816+0000 mon.a (mon.0) 1979 : cluster 0 osdmap e441: 8 total, 8 up, 8 in 2024-03-11T21:16:15.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:14 smithi146 bash[23809]: cluster 2024-03-11T21:16:13.891856+0000 osd.6 (osd.6) 462 : cluster 0 4.e scrub starts 2024-03-11T21:16:15.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:14 smithi146 bash[23809]: cluster 2024-03-11T21:16:13.894850+0000 osd.6 (osd.6) 463 : cluster 0 4.e scrub ok 2024-03-11T21:16:15.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:14 smithi146 bash[23809]: cluster 2024-03-11T21:16:14.371939+0000 osd.0 (osd.0) 416 : cluster 0 2.f scrub starts 2024-03-11T21:16:15.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:14 smithi146 bash[23809]: cluster 2024-03-11T21:16:14.375443+0000 osd.0 (osd.0) 417 : cluster 0 2.f scrub ok 2024-03-11T21:16:15.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:14 smithi023 bash[21372]: cluster 2024-03-11T21:16:13.496051+0000 osd.3 (osd.3) 504 : cluster 0 4.c scrub starts 2024-03-11T21:16:15.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:14 smithi023 bash[21372]: cluster 2024-03-11T21:16:13.498302+0000 osd.3 (osd.3) 505 : cluster 0 4.c scrub ok 2024-03-11T21:16:15.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:14 smithi023 bash[21372]: cluster 2024-03-11T21:16:13.614816+0000 mon.a (mon.0) 1979 : cluster 0 osdmap e441: 8 total, 8 up, 8 in 2024-03-11T21:16:15.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:14 smithi023 bash[21372]: cluster 2024-03-11T21:16:13.891856+0000 osd.6 (osd.6) 462 : cluster 0 4.e scrub starts 2024-03-11T21:16:15.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:14 smithi023 bash[21372]: cluster 2024-03-11T21:16:13.894850+0000 osd.6 (osd.6) 463 : cluster 0 4.e scrub ok 2024-03-11T21:16:15.026 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:14 smithi023 bash[21372]: cluster 2024-03-11T21:16:14.371939+0000 osd.0 (osd.0) 416 : cluster 0 2.f scrub starts 2024-03-11T21:16:15.026 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:14 smithi023 bash[21372]: cluster 2024-03-11T21:16:14.375443+0000 osd.0 (osd.0) 417 : cluster 0 2.f scrub ok 2024-03-11T21:16:15.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:14 smithi023 bash[29146]: cluster 2024-03-11T21:16:13.496051+0000 osd.3 (osd.3) 504 : cluster 0 4.c scrub starts 2024-03-11T21:16:15.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:14 smithi023 bash[29146]: cluster 2024-03-11T21:16:13.498302+0000 osd.3 (osd.3) 505 : cluster 0 4.c scrub ok 2024-03-11T21:16:15.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:14 smithi023 bash[29146]: cluster 2024-03-11T21:16:13.614816+0000 mon.a (mon.0) 1979 : cluster 0 osdmap e441: 8 total, 8 up, 8 in 2024-03-11T21:16:15.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:14 smithi023 bash[29146]: cluster 2024-03-11T21:16:13.891856+0000 osd.6 (osd.6) 462 : cluster 0 4.e scrub starts 2024-03-11T21:16:15.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:14 smithi023 bash[29146]: cluster 2024-03-11T21:16:13.894850+0000 osd.6 (osd.6) 463 : cluster 0 4.e scrub ok 2024-03-11T21:16:15.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:14 smithi023 bash[29146]: cluster 2024-03-11T21:16:14.371939+0000 osd.0 (osd.0) 416 : cluster 0 2.f scrub starts 2024-03-11T21:16:15.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:14 smithi023 bash[29146]: cluster 2024-03-11T21:16:14.375443+0000 osd.0 (osd.0) 417 : cluster 0 2.f scrub ok 2024-03-11T21:16:15.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:15 smithi023 bash[29146]: cluster 2024-03-11T21:16:14.069936+0000 osd.7 (osd.7) 577 : cluster 0 3.9 scrub starts 2024-03-11T21:16:15.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:15 smithi023 bash[29146]: cluster 2024-03-11T21:16:14.074326+0000 osd.7 (osd.7) 578 : cluster 0 3.9 scrub ok 2024-03-11T21:16:15.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:15 smithi023 bash[29146]: cluster 2024-03-11T21:16:14.476222+0000 mgr.y (mgr.24380) 940 : cluster 0 pgmap v1083: 142 pgs: 142 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-11T21:16:15.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:15 smithi023 bash[29146]: cluster 2024-03-11T21:16:14.479698+0000 osd.3 (osd.3) 506 : cluster 0 4.19 scrub starts 2024-03-11T21:16:15.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:15 smithi023 bash[29146]: cluster 2024-03-11T21:16:14.482390+0000 osd.3 (osd.3) 507 : cluster 0 4.19 scrub ok 2024-03-11T21:16:15.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:15 smithi023 bash[29146]: cluster 2024-03-11T21:16:15.003122+0000 osd.1 (osd.1) 320 : cluster 0 6.4 scrub starts 2024-03-11T21:16:15.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:15 smithi023 bash[29146]: cluster 2024-03-11T21:16:15.005782+0000 osd.1 (osd.1) 321 : cluster 0 6.4 scrub ok 2024-03-11T21:16:15.776 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:15 smithi023 bash[21372]: cluster 2024-03-11T21:16:14.069936+0000 osd.7 (osd.7) 577 : cluster 0 3.9 scrub starts 2024-03-11T21:16:15.776 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:15 smithi023 bash[21372]: cluster 2024-03-11T21:16:14.074326+0000 osd.7 (osd.7) 578 : cluster 0 3.9 scrub ok 2024-03-11T21:16:15.776 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:15 smithi023 bash[21372]: cluster 2024-03-11T21:16:14.476222+0000 mgr.y (mgr.24380) 940 : cluster 0 pgmap v1083: 142 pgs: 142 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-11T21:16:15.776 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:15 smithi023 bash[21372]: cluster 2024-03-11T21:16:14.479698+0000 osd.3 (osd.3) 506 : cluster 0 4.19 scrub starts 2024-03-11T21:16:15.776 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:15 smithi023 bash[21372]: cluster 2024-03-11T21:16:14.482390+0000 osd.3 (osd.3) 507 : cluster 0 4.19 scrub ok 2024-03-11T21:16:15.776 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:15 smithi023 bash[21372]: cluster 2024-03-11T21:16:15.003122+0000 osd.1 (osd.1) 320 : cluster 0 6.4 scrub starts 2024-03-11T21:16:15.776 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:15 smithi023 bash[21372]: cluster 2024-03-11T21:16:15.005782+0000 osd.1 (osd.1) 321 : cluster 0 6.4 scrub ok 2024-03-11T21:16:16.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:15 smithi146 bash[23809]: cluster 2024-03-11T21:16:14.069936+0000 osd.7 (osd.7) 577 : cluster 0 3.9 scrub starts 2024-03-11T21:16:16.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:15 smithi146 bash[23809]: cluster 2024-03-11T21:16:14.074326+0000 osd.7 (osd.7) 578 : cluster 0 3.9 scrub ok 2024-03-11T21:16:16.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:15 smithi146 bash[23809]: cluster 2024-03-11T21:16:14.476222+0000 mgr.y (mgr.24380) 940 : cluster 0 pgmap v1083: 142 pgs: 142 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-11T21:16:16.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:15 smithi146 bash[23809]: cluster 2024-03-11T21:16:14.479698+0000 osd.3 (osd.3) 506 : cluster 0 4.19 scrub starts 2024-03-11T21:16:16.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:15 smithi146 bash[23809]: cluster 2024-03-11T21:16:14.482390+0000 osd.3 (osd.3) 507 : cluster 0 4.19 scrub ok 2024-03-11T21:16:16.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:15 smithi146 bash[23809]: cluster 2024-03-11T21:16:15.003122+0000 osd.1 (osd.1) 320 : cluster 0 6.4 scrub starts 2024-03-11T21:16:16.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:15 smithi146 bash[23809]: cluster 2024-03-11T21:16:15.005782+0000 osd.1 (osd.1) 321 : cluster 0 6.4 scrub ok 2024-03-11T21:16:16.504 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:16:16 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:16:16.650 INFO:teuthology.orchestra.run.smithi023.stderr:set pool 1 pg_num to 21 2024-03-11T21:16:16.767 INFO:teuthology.orchestra.run.smithi023.stdout:ERROR: (22) Invalid argument 2024-03-11T21:16:16.767 INFO:teuthology.orchestra.run.smithi023.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:16:16.868 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:16 smithi023 bash[29146]: audit 2024-03-11T21:16:15.886859+0000 mon.c (mon.2) 104 : audit 1 from='client.? 172.21.15.23:0/2346664653' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-03-11T21:16:16.869 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:16 smithi023 bash[29146]: audit 2024-03-11T21:16:15.887376+0000 mon.a (mon.0) 1980 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-03-11T21:16:16.869 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:16 smithi023 bash[29146]: cluster 2024-03-11T21:16:15.958092+0000 osd.1 (osd.1) 322 : cluster 0 3.4 scrub starts 2024-03-11T21:16:16.869 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:16 smithi023 bash[29146]: cluster 2024-03-11T21:16:15.961587+0000 osd.1 (osd.1) 323 : cluster 0 3.4 scrub ok 2024-03-11T21:16:16.869 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:16 smithi023 bash[29146]: cluster 2024-03-11T21:16:16.371863+0000 osd.0 (osd.0) 418 : cluster 0 3.12 scrub starts 2024-03-11T21:16:16.869 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:16 smithi023 bash[29146]: cluster 2024-03-11T21:16:16.375624+0000 osd.0 (osd.0) 419 : cluster 0 3.12 scrub ok 2024-03-11T21:16:16.869 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:16 smithi023 bash[21372]: audit 2024-03-11T21:16:15.886859+0000 mon.c (mon.2) 104 : audit 1 from='client.? 172.21.15.23:0/2346664653' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-03-11T21:16:16.869 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:16 smithi023 bash[21372]: audit 2024-03-11T21:16:15.887376+0000 mon.a (mon.0) 1980 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-03-11T21:16:16.870 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:16 smithi023 bash[21372]: cluster 2024-03-11T21:16:15.958092+0000 osd.1 (osd.1) 322 : cluster 0 3.4 scrub starts 2024-03-11T21:16:16.870 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:16 smithi023 bash[21372]: cluster 2024-03-11T21:16:15.961587+0000 osd.1 (osd.1) 323 : cluster 0 3.4 scrub ok 2024-03-11T21:16:16.870 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:16 smithi023 bash[21372]: cluster 2024-03-11T21:16:16.371863+0000 osd.0 (osd.0) 418 : cluster 0 3.12 scrub starts 2024-03-11T21:16:16.870 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:16 smithi023 bash[21372]: cluster 2024-03-11T21:16:16.375624+0000 osd.0 (osd.0) 419 : cluster 0 3.12 scrub ok 2024-03-11T21:16:17.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:16 smithi146 bash[23809]: audit 2024-03-11T21:16:15.886859+0000 mon.c (mon.2) 104 : audit 1 from='client.? 172.21.15.23:0/2346664653' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-03-11T21:16:17.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:16 smithi146 bash[23809]: audit 2024-03-11T21:16:15.887376+0000 mon.a (mon.0) 1980 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-03-11T21:16:17.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:16 smithi146 bash[23809]: cluster 2024-03-11T21:16:15.958092+0000 osd.1 (osd.1) 322 : cluster 0 3.4 scrub starts 2024-03-11T21:16:17.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:16 smithi146 bash[23809]: cluster 2024-03-11T21:16:15.961587+0000 osd.1 (osd.1) 323 : cluster 0 3.4 scrub ok 2024-03-11T21:16:17.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:16 smithi146 bash[23809]: cluster 2024-03-11T21:16:16.371863+0000 osd.0 (osd.0) 418 : cluster 0 3.12 scrub starts 2024-03-11T21:16:17.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:16 smithi146 bash[23809]: cluster 2024-03-11T21:16:16.375624+0000 osd.0 (osd.0) 419 : cluster 0 3.12 scrub ok 2024-03-11T21:16:17.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:17 smithi023 bash[29146]: audit 2024-03-11T21:16:16.131746+0000 mgr.y (mgr.24380) 941 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:16:17.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:17 smithi023 bash[29146]: cluster 2024-03-11T21:16:16.338138+0000 osd.2 (osd.2) 263 : cluster 0 purged_snaps scrub starts 2024-03-11T21:16:17.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:17 smithi023 bash[29146]: cluster 2024-03-11T21:16:16.338239+0000 osd.2 (osd.2) 264 : cluster 0 purged_snaps scrub ok 2024-03-11T21:16:17.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:17 smithi023 bash[29146]: cluster 2024-03-11T21:16:16.477720+0000 mgr.y (mgr.24380) 942 : cluster 0 pgmap v1084: 142 pgs: 142 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T21:16:17.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:17 smithi023 bash[29146]: cluster 2024-03-11T21:16:16.514239+0000 osd.3 (osd.3) 508 : cluster 0 3.2 scrub starts 2024-03-11T21:16:17.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:17 smithi023 bash[29146]: cluster 2024-03-11T21:16:16.518998+0000 osd.3 (osd.3) 509 : cluster 0 3.2 scrub ok 2024-03-11T21:16:17.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:17 smithi023 bash[29146]: audit 2024-03-11T21:16:16.650915+0000 mon.a (mon.0) 1981 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-03-11T21:16:17.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:17 smithi023 bash[29146]: cluster 2024-03-11T21:16:16.654025+0000 mon.a (mon.0) 1982 : cluster 0 osdmap e442: 8 total, 8 up, 8 in 2024-03-11T21:16:17.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:17 smithi023 bash[29146]: cluster 2024-03-11T21:16:16.988947+0000 osd.1 (osd.1) 324 : cluster 0 4.f scrub starts 2024-03-11T21:16:17.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:17 smithi023 bash[29146]: cluster 2024-03-11T21:16:16.991345+0000 osd.1 (osd.1) 325 : cluster 0 4.f scrub ok 2024-03-11T21:16:17.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:17 smithi023 bash[29146]: cluster 2024-03-11T21:16:17.265663+0000 osd.1 (osd.1) 326 : cluster 0 purged_snaps scrub starts 2024-03-11T21:16:17.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:17 smithi023 bash[29146]: cluster 2024-03-11T21:16:17.265801+0000 osd.1 (osd.1) 327 : cluster 0 purged_snaps scrub ok 2024-03-11T21:16:17.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:17 smithi023 bash[29146]: cluster 2024-03-11T21:16:17.404699+0000 osd.0 (osd.0) 420 : cluster 0 2.13 scrub starts 2024-03-11T21:16:17.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:17 smithi023 bash[29146]: cluster 2024-03-11T21:16:17.407314+0000 osd.0 (osd.0) 421 : cluster 0 2.13 scrub ok 2024-03-11T21:16:17.777 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:17 smithi023 bash[21372]: audit 2024-03-11T21:16:16.131746+0000 mgr.y (mgr.24380) 941 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:16:17.777 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:17 smithi023 bash[21372]: cluster 2024-03-11T21:16:16.338138+0000 osd.2 (osd.2) 263 : cluster 0 purged_snaps scrub starts 2024-03-11T21:16:17.777 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:17 smithi023 bash[21372]: cluster 2024-03-11T21:16:16.338239+0000 osd.2 (osd.2) 264 : cluster 0 purged_snaps scrub ok 2024-03-11T21:16:17.777 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:17 smithi023 bash[21372]: cluster 2024-03-11T21:16:16.477720+0000 mgr.y (mgr.24380) 942 : cluster 0 pgmap v1084: 142 pgs: 142 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T21:16:17.777 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:17 smithi023 bash[21372]: cluster 2024-03-11T21:16:16.514239+0000 osd.3 (osd.3) 508 : cluster 0 3.2 scrub starts 2024-03-11T21:16:17.777 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:17 smithi023 bash[21372]: cluster 2024-03-11T21:16:16.518998+0000 osd.3 (osd.3) 509 : cluster 0 3.2 scrub ok 2024-03-11T21:16:17.777 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:17 smithi023 bash[21372]: audit 2024-03-11T21:16:16.650915+0000 mon.a (mon.0) 1981 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-03-11T21:16:17.777 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:17 smithi023 bash[21372]: cluster 2024-03-11T21:16:16.654025+0000 mon.a (mon.0) 1982 : cluster 0 osdmap e442: 8 total, 8 up, 8 in 2024-03-11T21:16:17.777 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:17 smithi023 bash[21372]: cluster 2024-03-11T21:16:16.988947+0000 osd.1 (osd.1) 324 : cluster 0 4.f scrub starts 2024-03-11T21:16:17.777 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:17 smithi023 bash[21372]: cluster 2024-03-11T21:16:16.991345+0000 osd.1 (osd.1) 325 : cluster 0 4.f scrub ok 2024-03-11T21:16:17.777 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:17 smithi023 bash[21372]: cluster 2024-03-11T21:16:17.265663+0000 osd.1 (osd.1) 326 : cluster 0 purged_snaps scrub starts 2024-03-11T21:16:17.777 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:17 smithi023 bash[21372]: cluster 2024-03-11T21:16:17.265801+0000 osd.1 (osd.1) 327 : cluster 0 purged_snaps scrub ok 2024-03-11T21:16:17.777 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:17 smithi023 bash[21372]: cluster 2024-03-11T21:16:17.404699+0000 osd.0 (osd.0) 420 : cluster 0 2.13 scrub starts 2024-03-11T21:16:17.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:17 smithi023 bash[21372]: cluster 2024-03-11T21:16:17.407314+0000 osd.0 (osd.0) 421 : cluster 0 2.13 scrub ok 2024-03-11T21:16:17.928 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-03-11T21:16:18.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:17 smithi146 bash[23809]: audit 2024-03-11T21:16:16.131746+0000 mgr.y (mgr.24380) 941 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:16:18.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:17 smithi146 bash[23809]: cluster 2024-03-11T21:16:16.338138+0000 osd.2 (osd.2) 263 : cluster 0 purged_snaps scrub starts 2024-03-11T21:16:18.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:17 smithi146 bash[23809]: cluster 2024-03-11T21:16:16.338239+0000 osd.2 (osd.2) 264 : cluster 0 purged_snaps scrub ok 2024-03-11T21:16:18.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:17 smithi146 bash[23809]: cluster 2024-03-11T21:16:16.477720+0000 mgr.y (mgr.24380) 942 : cluster 0 pgmap v1084: 142 pgs: 142 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T21:16:18.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:17 smithi146 bash[23809]: cluster 2024-03-11T21:16:16.514239+0000 osd.3 (osd.3) 508 : cluster 0 3.2 scrub starts 2024-03-11T21:16:18.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:17 smithi146 bash[23809]: cluster 2024-03-11T21:16:16.518998+0000 osd.3 (osd.3) 509 : cluster 0 3.2 scrub ok 2024-03-11T21:16:18.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:17 smithi146 bash[23809]: audit 2024-03-11T21:16:16.650915+0000 mon.a (mon.0) 1981 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-03-11T21:16:18.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:17 smithi146 bash[23809]: cluster 2024-03-11T21:16:16.654025+0000 mon.a (mon.0) 1982 : cluster 0 osdmap e442: 8 total, 8 up, 8 in 2024-03-11T21:16:18.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:17 smithi146 bash[23809]: cluster 2024-03-11T21:16:16.988947+0000 osd.1 (osd.1) 324 : cluster 0 4.f scrub starts 2024-03-11T21:16:18.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:17 smithi146 bash[23809]: cluster 2024-03-11T21:16:16.991345+0000 osd.1 (osd.1) 325 : cluster 0 4.f scrub ok 2024-03-11T21:16:18.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:17 smithi146 bash[23809]: cluster 2024-03-11T21:16:17.265663+0000 osd.1 (osd.1) 326 : cluster 0 purged_snaps scrub starts 2024-03-11T21:16:18.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:17 smithi146 bash[23809]: cluster 2024-03-11T21:16:17.265801+0000 osd.1 (osd.1) 327 : cluster 0 purged_snaps scrub ok 2024-03-11T21:16:18.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:17 smithi146 bash[23809]: cluster 2024-03-11T21:16:17.404699+0000 osd.0 (osd.0) 420 : cluster 0 2.13 scrub starts 2024-03-11T21:16:18.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:17 smithi146 bash[23809]: cluster 2024-03-11T21:16:17.407314+0000 osd.0 (osd.0) 421 : cluster 0 2.13 scrub ok 2024-03-11T21:16:18.306 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:16:18.703 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:16:18 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:16:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:16:19.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:18 smithi146 bash[23809]: cluster 2024-03-11T21:16:16.998086+0000 osd.7 (osd.7) 579 : cluster 0 6.12 scrub starts 2024-03-11T21:16:19.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:18 smithi146 bash[23809]: cluster 2024-03-11T21:16:17.031936+0000 osd.7 (osd.7) 580 : cluster 0 6.12 scrub ok 2024-03-11T21:16:19.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:18 smithi146 bash[23809]: cluster 2024-03-11T21:16:17.684236+0000 mon.a (mon.0) 1983 : cluster 0 osdmap e443: 8 total, 8 up, 8 in 2024-03-11T21:16:19.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:18 smithi146 bash[23809]: cluster 2024-03-11T21:16:18.383316+0000 osd.0 (osd.0) 422 : cluster 0 2.1f scrub starts 2024-03-11T21:16:19.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:18 smithi146 bash[23809]: cluster 2024-03-11T21:16:18.385478+0000 osd.0 (osd.0) 423 : cluster 0 2.1f scrub ok 2024-03-11T21:16:19.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:18 smithi146 bash[23809]: audit 2024-03-11T21:16:18.480692+0000 mon.a (mon.0) 1984 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-03-11T21:16:19.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:18 smithi023 bash[21372]: cluster 2024-03-11T21:16:16.998086+0000 osd.7 (osd.7) 579 : cluster 0 6.12 scrub starts 2024-03-11T21:16:19.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:18 smithi023 bash[21372]: cluster 2024-03-11T21:16:17.031936+0000 osd.7 (osd.7) 580 : cluster 0 6.12 scrub ok 2024-03-11T21:16:19.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:18 smithi023 bash[21372]: cluster 2024-03-11T21:16:17.684236+0000 mon.a (mon.0) 1983 : cluster 0 osdmap e443: 8 total, 8 up, 8 in 2024-03-11T21:16:19.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:18 smithi023 bash[21372]: cluster 2024-03-11T21:16:18.383316+0000 osd.0 (osd.0) 422 : cluster 0 2.1f scrub starts 2024-03-11T21:16:19.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:18 smithi023 bash[21372]: cluster 2024-03-11T21:16:18.385478+0000 osd.0 (osd.0) 423 : cluster 0 2.1f scrub ok 2024-03-11T21:16:19.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:18 smithi023 bash[21372]: audit 2024-03-11T21:16:18.480692+0000 mon.a (mon.0) 1984 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-03-11T21:16:19.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:18 smithi023 bash[29146]: cluster 2024-03-11T21:16:16.998086+0000 osd.7 (osd.7) 579 : cluster 0 6.12 scrub starts 2024-03-11T21:16:19.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:18 smithi023 bash[29146]: cluster 2024-03-11T21:16:17.031936+0000 osd.7 (osd.7) 580 : cluster 0 6.12 scrub ok 2024-03-11T21:16:19.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:18 smithi023 bash[29146]: cluster 2024-03-11T21:16:17.684236+0000 mon.a (mon.0) 1983 : cluster 0 osdmap e443: 8 total, 8 up, 8 in 2024-03-11T21:16:19.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:18 smithi023 bash[29146]: cluster 2024-03-11T21:16:18.383316+0000 osd.0 (osd.0) 422 : cluster 0 2.1f scrub starts 2024-03-11T21:16:19.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:18 smithi023 bash[29146]: cluster 2024-03-11T21:16:18.385478+0000 osd.0 (osd.0) 423 : cluster 0 2.1f scrub ok 2024-03-11T21:16:19.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:18 smithi023 bash[29146]: audit 2024-03-11T21:16:18.480692+0000 mon.a (mon.0) 1984 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-03-11T21:16:20.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:19 smithi146 bash[23809]: cluster 2024-03-11T21:16:18.478639+0000 mgr.y (mgr.24380) 943 : cluster 0 pgmap v1087: 142 pgs: 142 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:16:20.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:19 smithi146 bash[23809]: cluster 2024-03-11T21:16:18.604921+0000 osd.3 (osd.3) 510 : cluster 0 1.4 scrub starts 2024-03-11T21:16:20.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:19 smithi146 bash[23809]: cluster 2024-03-11T21:16:18.608274+0000 osd.3 (osd.3) 511 : cluster 0 1.4 scrub ok 2024-03-11T21:16:20.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:19 smithi146 bash[23809]: audit 2024-03-11T21:16:18.685077+0000 mon.a (mon.0) 1985 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-03-11T21:16:20.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:19 smithi146 bash[23809]: cluster 2024-03-11T21:16:18.701334+0000 mon.a (mon.0) 1986 : cluster 0 osdmap e444: 8 total, 8 up, 8 in 2024-03-11T21:16:20.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:19 smithi146 bash[23809]: cluster 2024-03-11T21:16:18.752274+0000 osd.6 (osd.6) 464 : cluster 0 4.5 scrub starts 2024-03-11T21:16:20.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:19 smithi146 bash[23809]: cluster 2024-03-11T21:16:18.754926+0000 osd.6 (osd.6) 465 : cluster 0 4.5 scrub ok 2024-03-11T21:16:20.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:19 smithi146 bash[23809]: cluster 2024-03-11T21:16:19.004300+0000 osd.1 (osd.1) 328 : cluster 0 2.d scrub starts 2024-03-11T21:16:20.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:19 smithi146 bash[23809]: cluster 2024-03-11T21:16:19.007179+0000 osd.1 (osd.1) 329 : cluster 0 2.d scrub ok 2024-03-11T21:16:20.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:19 smithi146 bash[23809]: cluster 2024-03-11T21:16:19.337171+0000 osd.0 (osd.0) 424 : cluster 0 6.9 scrub starts 2024-03-11T21:16:20.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:19 smithi146 bash[23809]: cluster 2024-03-11T21:16:19.448582+0000 osd.0 (osd.0) 425 : cluster 0 6.9 scrub ok 2024-03-11T21:16:20.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:19 smithi146 bash[23809]: cluster 2024-03-11T21:16:19.703968+0000 mon.a (mon.0) 1987 : cluster 0 osdmap e445: 8 total, 8 up, 8 in 2024-03-11T21:16:20.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:19 smithi023 bash[21372]: cluster 2024-03-11T21:16:18.478639+0000 mgr.y (mgr.24380) 943 : cluster 0 pgmap v1087: 142 pgs: 142 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:16:20.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:19 smithi023 bash[21372]: cluster 2024-03-11T21:16:18.604921+0000 osd.3 (osd.3) 510 : cluster 0 1.4 scrub starts 2024-03-11T21:16:20.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:19 smithi023 bash[21372]: cluster 2024-03-11T21:16:18.608274+0000 osd.3 (osd.3) 511 : cluster 0 1.4 scrub ok 2024-03-11T21:16:20.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:19 smithi023 bash[21372]: audit 2024-03-11T21:16:18.685077+0000 mon.a (mon.0) 1985 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-03-11T21:16:20.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:19 smithi023 bash[21372]: cluster 2024-03-11T21:16:18.701334+0000 mon.a (mon.0) 1986 : cluster 0 osdmap e444: 8 total, 8 up, 8 in 2024-03-11T21:16:20.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:19 smithi023 bash[21372]: cluster 2024-03-11T21:16:18.752274+0000 osd.6 (osd.6) 464 : cluster 0 4.5 scrub starts 2024-03-11T21:16:20.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:19 smithi023 bash[21372]: cluster 2024-03-11T21:16:18.754926+0000 osd.6 (osd.6) 465 : cluster 0 4.5 scrub ok 2024-03-11T21:16:20.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:19 smithi023 bash[21372]: cluster 2024-03-11T21:16:19.004300+0000 osd.1 (osd.1) 328 : cluster 0 2.d scrub starts 2024-03-11T21:16:20.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:19 smithi023 bash[21372]: cluster 2024-03-11T21:16:19.007179+0000 osd.1 (osd.1) 329 : cluster 0 2.d scrub ok 2024-03-11T21:16:20.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:19 smithi023 bash[21372]: cluster 2024-03-11T21:16:19.337171+0000 osd.0 (osd.0) 424 : cluster 0 6.9 scrub starts 2024-03-11T21:16:20.026 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:19 smithi023 bash[21372]: cluster 2024-03-11T21:16:19.448582+0000 osd.0 (osd.0) 425 : cluster 0 6.9 scrub ok 2024-03-11T21:16:20.026 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:19 smithi023 bash[21372]: cluster 2024-03-11T21:16:19.703968+0000 mon.a (mon.0) 1987 : cluster 0 osdmap e445: 8 total, 8 up, 8 in 2024-03-11T21:16:20.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:19 smithi023 bash[29146]: cluster 2024-03-11T21:16:18.478639+0000 mgr.y (mgr.24380) 943 : cluster 0 pgmap v1087: 142 pgs: 142 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:16:20.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:19 smithi023 bash[29146]: cluster 2024-03-11T21:16:18.604921+0000 osd.3 (osd.3) 510 : cluster 0 1.4 scrub starts 2024-03-11T21:16:20.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:19 smithi023 bash[29146]: cluster 2024-03-11T21:16:18.608274+0000 osd.3 (osd.3) 511 : cluster 0 1.4 scrub ok 2024-03-11T21:16:20.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:19 smithi023 bash[29146]: audit 2024-03-11T21:16:18.685077+0000 mon.a (mon.0) 1985 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-03-11T21:16:20.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:19 smithi023 bash[29146]: cluster 2024-03-11T21:16:18.701334+0000 mon.a (mon.0) 1986 : cluster 0 osdmap e444: 8 total, 8 up, 8 in 2024-03-11T21:16:20.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:19 smithi023 bash[29146]: cluster 2024-03-11T21:16:18.752274+0000 osd.6 (osd.6) 464 : cluster 0 4.5 scrub starts 2024-03-11T21:16:20.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:19 smithi023 bash[29146]: cluster 2024-03-11T21:16:18.754926+0000 osd.6 (osd.6) 465 : cluster 0 4.5 scrub ok 2024-03-11T21:16:20.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:19 smithi023 bash[29146]: cluster 2024-03-11T21:16:19.004300+0000 osd.1 (osd.1) 328 : cluster 0 2.d scrub starts 2024-03-11T21:16:20.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:19 smithi023 bash[29146]: cluster 2024-03-11T21:16:19.007179+0000 osd.1 (osd.1) 329 : cluster 0 2.d scrub ok 2024-03-11T21:16:20.027 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:19 smithi023 bash[29146]: cluster 2024-03-11T21:16:19.337171+0000 osd.0 (osd.0) 424 : cluster 0 6.9 scrub starts 2024-03-11T21:16:20.027 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:19 smithi023 bash[29146]: cluster 2024-03-11T21:16:19.448582+0000 osd.0 (osd.0) 425 : cluster 0 6.9 scrub ok 2024-03-11T21:16:20.027 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:19 smithi023 bash[29146]: cluster 2024-03-11T21:16:19.703968+0000 mon.a (mon.0) 1987 : cluster 0 osdmap e445: 8 total, 8 up, 8 in 2024-03-11T21:16:21.006 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:20 smithi146 bash[23809]: cluster 2024-03-11T21:16:19.761330+0000 osd.6 (osd.6) 466 : cluster 0 1.5 scrub starts 2024-03-11T21:16:21.006 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:20 smithi146 bash[23809]: cluster 2024-03-11T21:16:19.762885+0000 osd.6 (osd.6) 467 : cluster 0 1.5 scrub ok 2024-03-11T21:16:21.006 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:20 smithi146 bash[23809]: cluster 2024-03-11T21:16:19.973580+0000 osd.5 (osd.5) 561 : cluster 0 1.1 scrub starts 2024-03-11T21:16:21.006 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:20 smithi146 bash[23809]: cluster 2024-03-11T21:16:19.976151+0000 osd.5 (osd.5) 562 : cluster 0 1.1 scrub ok 2024-03-11T21:16:21.006 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:20 smithi146 bash[23809]: cluster 2024-03-11T21:16:20.034895+0000 osd.1 (osd.1) 330 : cluster 0 1.3 scrub starts 2024-03-11T21:16:21.007 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:20 smithi146 bash[23809]: cluster 2024-03-11T21:16:20.037234+0000 osd.1 (osd.1) 331 : cluster 0 1.3 scrub ok 2024-03-11T21:16:21.007 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:20 smithi146 bash[23809]: audit 2024-03-11T21:16:20.481886+0000 mon.a (mon.0) 1988 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-03-11T21:16:21.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:20 smithi023 bash[21372]: cluster 2024-03-11T21:16:19.761330+0000 osd.6 (osd.6) 466 : cluster 0 1.5 scrub starts 2024-03-11T21:16:21.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:20 smithi023 bash[21372]: cluster 2024-03-11T21:16:19.762885+0000 osd.6 (osd.6) 467 : cluster 0 1.5 scrub ok 2024-03-11T21:16:21.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:20 smithi023 bash[21372]: cluster 2024-03-11T21:16:19.973580+0000 osd.5 (osd.5) 561 : cluster 0 1.1 scrub starts 2024-03-11T21:16:21.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:20 smithi023 bash[21372]: cluster 2024-03-11T21:16:19.976151+0000 osd.5 (osd.5) 562 : cluster 0 1.1 scrub ok 2024-03-11T21:16:21.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:20 smithi023 bash[21372]: cluster 2024-03-11T21:16:20.034895+0000 osd.1 (osd.1) 330 : cluster 0 1.3 scrub starts 2024-03-11T21:16:21.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:20 smithi023 bash[21372]: cluster 2024-03-11T21:16:20.037234+0000 osd.1 (osd.1) 331 : cluster 0 1.3 scrub ok 2024-03-11T21:16:21.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:20 smithi023 bash[21372]: audit 2024-03-11T21:16:20.481886+0000 mon.a (mon.0) 1988 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-03-11T21:16:21.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:20 smithi023 bash[29146]: cluster 2024-03-11T21:16:19.761330+0000 osd.6 (osd.6) 466 : cluster 0 1.5 scrub starts 2024-03-11T21:16:21.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:20 smithi023 bash[29146]: cluster 2024-03-11T21:16:19.762885+0000 osd.6 (osd.6) 467 : cluster 0 1.5 scrub ok 2024-03-11T21:16:21.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:20 smithi023 bash[29146]: cluster 2024-03-11T21:16:19.973580+0000 osd.5 (osd.5) 561 : cluster 0 1.1 scrub starts 2024-03-11T21:16:21.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:20 smithi023 bash[29146]: cluster 2024-03-11T21:16:19.976151+0000 osd.5 (osd.5) 562 : cluster 0 1.1 scrub ok 2024-03-11T21:16:21.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:20 smithi023 bash[29146]: cluster 2024-03-11T21:16:20.034895+0000 osd.1 (osd.1) 330 : cluster 0 1.3 scrub starts 2024-03-11T21:16:21.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:20 smithi023 bash[29146]: cluster 2024-03-11T21:16:20.037234+0000 osd.1 (osd.1) 331 : cluster 0 1.3 scrub ok 2024-03-11T21:16:21.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:20 smithi023 bash[29146]: audit 2024-03-11T21:16:20.481886+0000 mon.a (mon.0) 1988 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-03-11T21:16:21.730 INFO:teuthology.orchestra.run.smithi023.stderr:nodeep-scrub is unset 2024-03-11T21:16:22.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:21 smithi146 bash[23809]: cluster 2024-03-11T21:16:19.996479+0000 osd.7 (osd.7) 581 : cluster 0 1.0 scrub starts 2024-03-11T21:16:22.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:21 smithi146 bash[23809]: cluster 2024-03-11T21:16:20.001720+0000 osd.7 (osd.7) 582 : cluster 0 1.0 scrub ok 2024-03-11T21:16:22.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:21 smithi146 bash[23809]: cluster 2024-03-11T21:16:20.479808+0000 mgr.y (mgr.24380) 944 : cluster 0 pgmap v1090: 152 pgs: 8 unknown, 144 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail 2024-03-11T21:16:22.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:21 smithi146 bash[23809]: cluster 2024-03-11T21:16:20.616109+0000 osd.3 (osd.3) 512 : cluster 0 1.4 scrub starts 2024-03-11T21:16:22.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:21 smithi146 bash[23809]: cluster 2024-03-11T21:16:20.617775+0000 osd.3 (osd.3) 513 : cluster 0 1.4 scrub ok 2024-03-11T21:16:22.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:21 smithi146 bash[23809]: audit 2024-03-11T21:16:20.735078+0000 mon.a (mon.0) 1989 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-03-11T21:16:22.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:21 smithi146 bash[23809]: cluster 2024-03-11T21:16:20.745315+0000 mon.a (mon.0) 1990 : cluster 0 osdmap e446: 8 total, 8 up, 8 in 2024-03-11T21:16:22.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:21 smithi146 bash[23809]: cluster 2024-03-11T21:16:20.768272+0000 osd.6 (osd.6) 468 : cluster 0 1.7 scrub starts 2024-03-11T21:16:22.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:21 smithi146 bash[23809]: cluster 2024-03-11T21:16:20.774772+0000 osd.6 (osd.6) 469 : cluster 0 1.7 scrub ok 2024-03-11T21:16:22.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:21 smithi146 bash[23809]: cluster 2024-03-11T21:16:20.993695+0000 osd.5 (osd.5) 563 : cluster 0 1.8 scrub starts 2024-03-11T21:16:22.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:21 smithi146 bash[23809]: cluster 2024-03-11T21:16:20.996799+0000 osd.5 (osd.5) 564 : cluster 0 1.8 scrub ok 2024-03-11T21:16:22.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:21 smithi146 bash[23809]: cluster 2024-03-11T21:16:21.313358+0000 osd.0 (osd.0) 426 : cluster 0 6.0 scrub starts 2024-03-11T21:16:22.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:21 smithi146 bash[23809]: cluster 2024-03-11T21:16:21.363476+0000 osd.0 (osd.0) 427 : cluster 0 6.0 scrub ok 2024-03-11T21:16:22.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:21 smithi146 bash[23809]: audit 2024-03-11T21:16:21.527091+0000 mon.a (mon.0) 1991 : audit 1 from='client.? 172.21.15.23:0/1637164466' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:16:22.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:21 smithi146 bash[23809]: audit 2024-03-11T21:16:21.672639+0000 mon.a (mon.0) 1992 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:16:22.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:21 smithi023 bash[21372]: cluster 2024-03-11T21:16:19.996479+0000 osd.7 (osd.7) 581 : cluster 0 1.0 scrub starts 2024-03-11T21:16:22.026 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:21 smithi023 bash[21372]: cluster 2024-03-11T21:16:20.001720+0000 osd.7 (osd.7) 582 : cluster 0 1.0 scrub ok 2024-03-11T21:16:22.026 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:21 smithi023 bash[21372]: cluster 2024-03-11T21:16:20.479808+0000 mgr.y (mgr.24380) 944 : cluster 0 pgmap v1090: 152 pgs: 8 unknown, 144 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail 2024-03-11T21:16:22.026 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:21 smithi023 bash[21372]: cluster 2024-03-11T21:16:20.616109+0000 osd.3 (osd.3) 512 : cluster 0 1.4 scrub starts 2024-03-11T21:16:22.026 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:21 smithi023 bash[21372]: cluster 2024-03-11T21:16:20.617775+0000 osd.3 (osd.3) 513 : cluster 0 1.4 scrub ok 2024-03-11T21:16:22.026 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:21 smithi023 bash[21372]: audit 2024-03-11T21:16:20.735078+0000 mon.a (mon.0) 1989 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-03-11T21:16:22.026 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:21 smithi023 bash[21372]: cluster 2024-03-11T21:16:20.745315+0000 mon.a (mon.0) 1990 : cluster 0 osdmap e446: 8 total, 8 up, 8 in 2024-03-11T21:16:22.026 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:21 smithi023 bash[21372]: cluster 2024-03-11T21:16:20.768272+0000 osd.6 (osd.6) 468 : cluster 0 1.7 scrub starts 2024-03-11T21:16:22.026 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:21 smithi023 bash[21372]: cluster 2024-03-11T21:16:20.774772+0000 osd.6 (osd.6) 469 : cluster 0 1.7 scrub ok 2024-03-11T21:16:22.026 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:21 smithi023 bash[21372]: cluster 2024-03-11T21:16:20.993695+0000 osd.5 (osd.5) 563 : cluster 0 1.8 scrub starts 2024-03-11T21:16:22.026 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:21 smithi023 bash[21372]: cluster 2024-03-11T21:16:20.996799+0000 osd.5 (osd.5) 564 : cluster 0 1.8 scrub ok 2024-03-11T21:16:22.026 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:21 smithi023 bash[21372]: cluster 2024-03-11T21:16:21.313358+0000 osd.0 (osd.0) 426 : cluster 0 6.0 scrub starts 2024-03-11T21:16:22.026 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:21 smithi023 bash[21372]: cluster 2024-03-11T21:16:21.363476+0000 osd.0 (osd.0) 427 : cluster 0 6.0 scrub ok 2024-03-11T21:16:22.027 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:21 smithi023 bash[21372]: audit 2024-03-11T21:16:21.527091+0000 mon.a (mon.0) 1991 : audit 1 from='client.? 172.21.15.23:0/1637164466' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:16:22.027 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:21 smithi023 bash[21372]: audit 2024-03-11T21:16:21.672639+0000 mon.a (mon.0) 1992 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:16:22.027 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:21 smithi023 bash[29146]: cluster 2024-03-11T21:16:19.996479+0000 osd.7 (osd.7) 581 : cluster 0 1.0 scrub starts 2024-03-11T21:16:22.027 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:21 smithi023 bash[29146]: cluster 2024-03-11T21:16:20.001720+0000 osd.7 (osd.7) 582 : cluster 0 1.0 scrub ok 2024-03-11T21:16:22.027 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:21 smithi023 bash[29146]: cluster 2024-03-11T21:16:20.479808+0000 mgr.y (mgr.24380) 944 : cluster 0 pgmap v1090: 152 pgs: 8 unknown, 144 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail 2024-03-11T21:16:22.027 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:21 smithi023 bash[29146]: cluster 2024-03-11T21:16:20.616109+0000 osd.3 (osd.3) 512 : cluster 0 1.4 scrub starts 2024-03-11T21:16:22.027 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:21 smithi023 bash[29146]: cluster 2024-03-11T21:16:20.617775+0000 osd.3 (osd.3) 513 : cluster 0 1.4 scrub ok 2024-03-11T21:16:22.027 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:21 smithi023 bash[29146]: audit 2024-03-11T21:16:20.735078+0000 mon.a (mon.0) 1989 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-03-11T21:16:22.027 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:21 smithi023 bash[29146]: cluster 2024-03-11T21:16:20.745315+0000 mon.a (mon.0) 1990 : cluster 0 osdmap e446: 8 total, 8 up, 8 in 2024-03-11T21:16:22.027 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:21 smithi023 bash[29146]: cluster 2024-03-11T21:16:20.768272+0000 osd.6 (osd.6) 468 : cluster 0 1.7 scrub starts 2024-03-11T21:16:22.027 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:21 smithi023 bash[29146]: cluster 2024-03-11T21:16:20.774772+0000 osd.6 (osd.6) 469 : cluster 0 1.7 scrub ok 2024-03-11T21:16:22.027 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:21 smithi023 bash[29146]: cluster 2024-03-11T21:16:20.993695+0000 osd.5 (osd.5) 563 : cluster 0 1.8 scrub starts 2024-03-11T21:16:22.028 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:21 smithi023 bash[29146]: cluster 2024-03-11T21:16:20.996799+0000 osd.5 (osd.5) 564 : cluster 0 1.8 scrub ok 2024-03-11T21:16:22.028 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:21 smithi023 bash[29146]: cluster 2024-03-11T21:16:21.313358+0000 osd.0 (osd.0) 426 : cluster 0 6.0 scrub starts 2024-03-11T21:16:22.028 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:21 smithi023 bash[29146]: cluster 2024-03-11T21:16:21.363476+0000 osd.0 (osd.0) 427 : cluster 0 6.0 scrub ok 2024-03-11T21:16:22.028 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:21 smithi023 bash[29146]: audit 2024-03-11T21:16:21.527091+0000 mon.a (mon.0) 1991 : audit 1 from='client.? 172.21.15.23:0/1637164466' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:16:22.028 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:21 smithi023 bash[29146]: audit 2024-03-11T21:16:21.672639+0000 mon.a (mon.0) 1992 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:16:22.888 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-03-11T21:16:22.888 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd pool set .mgr pgp_num 21 2024-03-11T21:16:22.902 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:22 smithi023 bash[21372]: cluster 2024-03-11T21:16:21.001086+0000 osd.7 (osd.7) 583 : cluster 0 1.2 scrub starts 2024-03-11T21:16:22.902 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:22 smithi023 bash[21372]: cluster 2024-03-11T21:16:21.003521+0000 osd.7 (osd.7) 584 : cluster 0 1.2 scrub ok 2024-03-11T21:16:22.902 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:22 smithi023 bash[21372]: cluster 2024-03-11T21:16:21.722423+0000 osd.6 (osd.6) 470 : cluster 0 1.6 scrub starts 2024-03-11T21:16:22.902 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:22 smithi023 bash[21372]: cluster 2024-03-11T21:16:21.725066+0000 osd.6 (osd.6) 471 : cluster 0 1.6 scrub ok 2024-03-11T21:16:22.902 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:22 smithi023 bash[21372]: cluster 2024-03-11T21:16:21.735418+0000 mon.a (mon.0) 1993 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:16:22.902 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:22 smithi023 bash[21372]: cluster 2024-03-11T21:16:21.735534+0000 mon.a (mon.0) 1994 : cluster 1 Cluster is now healthy 2024-03-11T21:16:22.902 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:22 smithi023 bash[21372]: audit 2024-03-11T21:16:21.739142+0000 mon.a (mon.0) 1995 : audit 1 from='client.? 172.21.15.23:0/1637164466' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:16:22.902 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:22 smithi023 bash[21372]: cluster 2024-03-11T21:16:21.746039+0000 mon.a (mon.0) 1996 : cluster 0 osdmap e447: 8 total, 8 up, 8 in 2024-03-11T21:16:22.902 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:22 smithi023 bash[21372]: cluster 2024-03-11T21:16:22.364743+0000 osd.0 (osd.0) 428 : cluster 0 4.6 scrub starts 2024-03-11T21:16:22.902 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:22 smithi023 bash[21372]: cluster 2024-03-11T21:16:22.367093+0000 osd.0 (osd.0) 429 : cluster 0 4.6 scrub ok 2024-03-11T21:16:22.903 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:22 smithi023 bash[21372]: cluster 2024-03-11T21:16:22.748480+0000 mon.a (mon.0) 1997 : cluster 0 osdmap e448: 8 total, 8 up, 8 in 2024-03-11T21:16:22.903 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:22 smithi023 bash[29146]: cluster 2024-03-11T21:16:21.001086+0000 osd.7 (osd.7) 583 : cluster 0 1.2 scrub starts 2024-03-11T21:16:22.903 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:22 smithi023 bash[29146]: cluster 2024-03-11T21:16:21.003521+0000 osd.7 (osd.7) 584 : cluster 0 1.2 scrub ok 2024-03-11T21:16:22.903 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:22 smithi023 bash[29146]: cluster 2024-03-11T21:16:21.722423+0000 osd.6 (osd.6) 470 : cluster 0 1.6 scrub starts 2024-03-11T21:16:22.903 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:22 smithi023 bash[29146]: cluster 2024-03-11T21:16:21.725066+0000 osd.6 (osd.6) 471 : cluster 0 1.6 scrub ok 2024-03-11T21:16:22.903 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:22 smithi023 bash[29146]: cluster 2024-03-11T21:16:21.735418+0000 mon.a (mon.0) 1993 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:16:22.903 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:22 smithi023 bash[29146]: cluster 2024-03-11T21:16:21.735534+0000 mon.a (mon.0) 1994 : cluster 1 Cluster is now healthy 2024-03-11T21:16:22.903 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:22 smithi023 bash[29146]: audit 2024-03-11T21:16:21.739142+0000 mon.a (mon.0) 1995 : audit 1 from='client.? 172.21.15.23:0/1637164466' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:16:22.903 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:22 smithi023 bash[29146]: cluster 2024-03-11T21:16:21.746039+0000 mon.a (mon.0) 1996 : cluster 0 osdmap e447: 8 total, 8 up, 8 in 2024-03-11T21:16:22.903 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:22 smithi023 bash[29146]: cluster 2024-03-11T21:16:22.364743+0000 osd.0 (osd.0) 428 : cluster 0 4.6 scrub starts 2024-03-11T21:16:22.903 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:22 smithi023 bash[29146]: cluster 2024-03-11T21:16:22.367093+0000 osd.0 (osd.0) 429 : cluster 0 4.6 scrub ok 2024-03-11T21:16:22.904 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:22 smithi023 bash[29146]: cluster 2024-03-11T21:16:22.748480+0000 mon.a (mon.0) 1997 : cluster 0 osdmap e448: 8 total, 8 up, 8 in 2024-03-11T21:16:23.013 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:16:23.254 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:22 smithi146 bash[23809]: cluster 2024-03-11T21:16:21.001086+0000 osd.7 (osd.7) 583 : cluster 0 1.2 scrub starts 2024-03-11T21:16:23.255 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:22 smithi146 bash[23809]: cluster 2024-03-11T21:16:21.003521+0000 osd.7 (osd.7) 584 : cluster 0 1.2 scrub ok 2024-03-11T21:16:23.255 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:22 smithi146 bash[23809]: cluster 2024-03-11T21:16:21.722423+0000 osd.6 (osd.6) 470 : cluster 0 1.6 scrub starts 2024-03-11T21:16:23.255 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:22 smithi146 bash[23809]: cluster 2024-03-11T21:16:21.725066+0000 osd.6 (osd.6) 471 : cluster 0 1.6 scrub ok 2024-03-11T21:16:23.255 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:22 smithi146 bash[23809]: cluster 2024-03-11T21:16:21.735418+0000 mon.a (mon.0) 1993 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:16:23.255 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:22 smithi146 bash[23809]: cluster 2024-03-11T21:16:21.735534+0000 mon.a (mon.0) 1994 : cluster 1 Cluster is now healthy 2024-03-11T21:16:23.255 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:22 smithi146 bash[23809]: audit 2024-03-11T21:16:21.739142+0000 mon.a (mon.0) 1995 : audit 1 from='client.? 172.21.15.23:0/1637164466' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:16:23.255 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:22 smithi146 bash[23809]: cluster 2024-03-11T21:16:21.746039+0000 mon.a (mon.0) 1996 : cluster 0 osdmap e447: 8 total, 8 up, 8 in 2024-03-11T21:16:23.255 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:22 smithi146 bash[23809]: cluster 2024-03-11T21:16:22.364743+0000 osd.0 (osd.0) 428 : cluster 0 4.6 scrub starts 2024-03-11T21:16:23.256 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:22 smithi146 bash[23809]: cluster 2024-03-11T21:16:22.367093+0000 osd.0 (osd.0) 429 : cluster 0 4.6 scrub ok 2024-03-11T21:16:23.256 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:22 smithi146 bash[23809]: cluster 2024-03-11T21:16:22.748480+0000 mon.a (mon.0) 1997 : cluster 0 osdmap e448: 8 total, 8 up, 8 in 2024-03-11T21:16:24.023 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:23 smithi023 bash[21372]: cluster 2024-03-11T21:16:22.480744+0000 mgr.y (mgr.24380) 945 : cluster 0 pgmap v1093: 152 pgs: 1 peering, 3 unknown, 148 active+clean; 476 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:16:24.023 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:23 smithi023 bash[21372]: cluster 2024-03-11T21:16:22.759795+0000 mon.a (mon.0) 1998 : cluster 3 Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-03-11T21:16:24.023 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:23 smithi023 bash[29146]: cluster 2024-03-11T21:16:22.480744+0000 mgr.y (mgr.24380) 945 : cluster 0 pgmap v1093: 152 pgs: 1 peering, 3 unknown, 148 active+clean; 476 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:16:24.023 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:23 smithi023 bash[29146]: cluster 2024-03-11T21:16:22.759795+0000 mon.a (mon.0) 1998 : cluster 3 Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-03-11T21:16:24.254 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:23 smithi146 bash[23809]: cluster 2024-03-11T21:16:22.480744+0000 mgr.y (mgr.24380) 945 : cluster 0 pgmap v1093: 152 pgs: 1 peering, 3 unknown, 148 active+clean; 476 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:16:24.254 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:23 smithi146 bash[23809]: cluster 2024-03-11T21:16:22.759795+0000 mon.a (mon.0) 1998 : cluster 3 Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-03-11T21:16:25.254 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:24 smithi146 bash[23809]: cluster 2024-03-11T21:16:23.799281+0000 mon.a (mon.0) 1999 : cluster 0 osdmap e449: 8 total, 8 up, 8 in 2024-03-11T21:16:25.254 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:24 smithi146 bash[23809]: cluster 2024-03-11T21:16:24.020115+0000 osd.1 (osd.1) 332 : cluster 0 2.10 scrub starts 2024-03-11T21:16:25.254 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:24 smithi146 bash[23809]: cluster 2024-03-11T21:16:24.021439+0000 osd.1 (osd.1) 333 : cluster 0 2.10 scrub ok 2024-03-11T21:16:25.254 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:24 smithi146 bash[23809]: cluster 2024-03-11T21:16:24.481550+0000 mgr.y (mgr.24380) 946 : cluster 0 pgmap v1096: 152 pgs: 2 active+undersized+remapped, 1 peering, 1 unknown, 148 active+clean; 476 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:16:25.275 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:24 smithi023 bash[21372]: cluster 2024-03-11T21:16:23.799281+0000 mon.a (mon.0) 1999 : cluster 0 osdmap e449: 8 total, 8 up, 8 in 2024-03-11T21:16:25.275 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:24 smithi023 bash[21372]: cluster 2024-03-11T21:16:24.020115+0000 osd.1 (osd.1) 332 : cluster 0 2.10 scrub starts 2024-03-11T21:16:25.275 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:24 smithi023 bash[21372]: cluster 2024-03-11T21:16:24.021439+0000 osd.1 (osd.1) 333 : cluster 0 2.10 scrub ok 2024-03-11T21:16:25.275 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:24 smithi023 bash[21372]: cluster 2024-03-11T21:16:24.481550+0000 mgr.y (mgr.24380) 946 : cluster 0 pgmap v1096: 152 pgs: 2 active+undersized+remapped, 1 peering, 1 unknown, 148 active+clean; 476 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:16:25.275 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:24 smithi023 bash[29146]: cluster 2024-03-11T21:16:23.799281+0000 mon.a (mon.0) 1999 : cluster 0 osdmap e449: 8 total, 8 up, 8 in 2024-03-11T21:16:25.275 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:24 smithi023 bash[29146]: cluster 2024-03-11T21:16:24.020115+0000 osd.1 (osd.1) 332 : cluster 0 2.10 scrub starts 2024-03-11T21:16:25.276 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:24 smithi023 bash[29146]: cluster 2024-03-11T21:16:24.021439+0000 osd.1 (osd.1) 333 : cluster 0 2.10 scrub ok 2024-03-11T21:16:25.276 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:24 smithi023 bash[29146]: cluster 2024-03-11T21:16:24.481550+0000 mgr.y (mgr.24380) 946 : cluster 0 pgmap v1096: 152 pgs: 2 active+undersized+remapped, 1 peering, 1 unknown, 148 active+clean; 476 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:16:26.030 INFO:teuthology.orchestra.run.smithi023.stdout:ERROR: (22) Invalid argument 2024-03-11T21:16:26.030 INFO:teuthology.orchestra.run.smithi023.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:16:26.054 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:25 smithi023 bash[21372]: cluster 2024-03-11T21:16:23.936730+0000 osd.7 (osd.7) 585 : cluster 0 3.15 scrub starts 2024-03-11T21:16:26.054 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:25 smithi023 bash[21372]: cluster 2024-03-11T21:16:23.940622+0000 osd.7 (osd.7) 586 : cluster 0 3.15 scrub ok 2024-03-11T21:16:26.054 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:25 smithi023 bash[21372]: cluster 2024-03-11T21:16:24.812967+0000 mon.a (mon.0) 2000 : cluster 0 osdmap e450: 8 total, 8 up, 8 in 2024-03-11T21:16:26.054 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:25 smithi023 bash[21372]: cluster 2024-03-11T21:16:24.949718+0000 osd.7 (osd.7) 587 : cluster 0 1.e deep-scrub starts 2024-03-11T21:16:26.054 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:25 smithi023 bash[21372]: cluster 2024-03-11T21:16:24.952246+0000 osd.7 (osd.7) 588 : cluster 0 1.e deep-scrub ok 2024-03-11T21:16:26.054 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:25 smithi023 bash[21372]: cluster 2024-03-11T21:16:24.969909+0000 osd.6 (osd.6) 472 : cluster 0 purged_snaps scrub starts 2024-03-11T21:16:26.054 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:25 smithi023 bash[21372]: cluster 2024-03-11T21:16:24.969998+0000 osd.6 (osd.6) 473 : cluster 0 purged_snaps scrub ok 2024-03-11T21:16:26.055 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:25 smithi023 bash[21372]: cluster 2024-03-11T21:16:24.995263+0000 osd.1 (osd.1) 334 : cluster 0 1.b scrub starts 2024-03-11T21:16:26.055 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:25 smithi023 bash[21372]: cluster 2024-03-11T21:16:24.997706+0000 osd.1 (osd.1) 335 : cluster 0 1.b scrub ok 2024-03-11T21:16:26.055 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:25 smithi023 bash[21372]: cluster 2024-03-11T21:16:25.021104+0000 osd.5 (osd.5) 565 : cluster 0 1.10 scrub starts 2024-03-11T21:16:26.055 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:25 smithi023 bash[21372]: cluster 2024-03-11T21:16:25.023474+0000 osd.5 (osd.5) 566 : cluster 0 1.10 scrub ok 2024-03-11T21:16:26.055 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:25 smithi023 bash[21372]: cluster 2024-03-11T21:16:25.359903+0000 osd.0 (osd.0) 430 : cluster 0 1.11 scrub starts 2024-03-11T21:16:26.055 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:25 smithi023 bash[21372]: cluster 2024-03-11T21:16:25.362162+0000 osd.0 (osd.0) 431 : cluster 0 1.11 scrub ok 2024-03-11T21:16:26.055 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:25 smithi023 bash[21372]: cluster 2024-03-11T21:16:25.435969+0000 mon.a (mon.0) 2001 : cluster 0 osdmap e451: 8 total, 8 up, 8 in 2024-03-11T21:16:26.055 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:25 smithi023 bash[29146]: cluster 2024-03-11T21:16:23.936730+0000 osd.7 (osd.7) 585 : cluster 0 3.15 scrub starts 2024-03-11T21:16:26.055 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:25 smithi023 bash[29146]: cluster 2024-03-11T21:16:23.940622+0000 osd.7 (osd.7) 586 : cluster 0 3.15 scrub ok 2024-03-11T21:16:26.055 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:25 smithi023 bash[29146]: cluster 2024-03-11T21:16:24.812967+0000 mon.a (mon.0) 2000 : cluster 0 osdmap e450: 8 total, 8 up, 8 in 2024-03-11T21:16:26.056 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:25 smithi023 bash[29146]: cluster 2024-03-11T21:16:24.949718+0000 osd.7 (osd.7) 587 : cluster 0 1.e deep-scrub starts 2024-03-11T21:16:26.056 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:25 smithi023 bash[29146]: cluster 2024-03-11T21:16:24.952246+0000 osd.7 (osd.7) 588 : cluster 0 1.e deep-scrub ok 2024-03-11T21:16:26.056 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:25 smithi023 bash[29146]: cluster 2024-03-11T21:16:24.969909+0000 osd.6 (osd.6) 472 : cluster 0 purged_snaps scrub starts 2024-03-11T21:16:26.056 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:25 smithi023 bash[29146]: cluster 2024-03-11T21:16:24.969998+0000 osd.6 (osd.6) 473 : cluster 0 purged_snaps scrub ok 2024-03-11T21:16:26.056 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:25 smithi023 bash[29146]: cluster 2024-03-11T21:16:24.995263+0000 osd.1 (osd.1) 334 : cluster 0 1.b scrub starts 2024-03-11T21:16:26.056 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:25 smithi023 bash[29146]: cluster 2024-03-11T21:16:24.997706+0000 osd.1 (osd.1) 335 : cluster 0 1.b scrub ok 2024-03-11T21:16:26.056 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:25 smithi023 bash[29146]: cluster 2024-03-11T21:16:25.021104+0000 osd.5 (osd.5) 565 : cluster 0 1.10 scrub starts 2024-03-11T21:16:26.056 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:25 smithi023 bash[29146]: cluster 2024-03-11T21:16:25.023474+0000 osd.5 (osd.5) 566 : cluster 0 1.10 scrub ok 2024-03-11T21:16:26.056 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:25 smithi023 bash[29146]: cluster 2024-03-11T21:16:25.359903+0000 osd.0 (osd.0) 430 : cluster 0 1.11 scrub starts 2024-03-11T21:16:26.056 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:25 smithi023 bash[29146]: cluster 2024-03-11T21:16:25.362162+0000 osd.0 (osd.0) 431 : cluster 0 1.11 scrub ok 2024-03-11T21:16:26.056 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:25 smithi023 bash[29146]: cluster 2024-03-11T21:16:25.435969+0000 mon.a (mon.0) 2001 : cluster 0 osdmap e451: 8 total, 8 up, 8 in 2024-03-11T21:16:26.128 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:25 smithi146 bash[23809]: cluster 2024-03-11T21:16:23.936730+0000 osd.7 (osd.7) 585 : cluster 0 3.15 scrub starts 2024-03-11T21:16:26.128 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:25 smithi146 bash[23809]: cluster 2024-03-11T21:16:23.940622+0000 osd.7 (osd.7) 586 : cluster 0 3.15 scrub ok 2024-03-11T21:16:26.128 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:25 smithi146 bash[23809]: cluster 2024-03-11T21:16:24.812967+0000 mon.a (mon.0) 2000 : cluster 0 osdmap e450: 8 total, 8 up, 8 in 2024-03-11T21:16:26.128 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:25 smithi146 bash[23809]: cluster 2024-03-11T21:16:24.949718+0000 osd.7 (osd.7) 587 : cluster 0 1.e deep-scrub starts 2024-03-11T21:16:26.129 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:25 smithi146 bash[23809]: cluster 2024-03-11T21:16:24.952246+0000 osd.7 (osd.7) 588 : cluster 0 1.e deep-scrub ok 2024-03-11T21:16:26.129 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:25 smithi146 bash[23809]: cluster 2024-03-11T21:16:24.969909+0000 osd.6 (osd.6) 472 : cluster 0 purged_snaps scrub starts 2024-03-11T21:16:26.129 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:25 smithi146 bash[23809]: cluster 2024-03-11T21:16:24.969998+0000 osd.6 (osd.6) 473 : cluster 0 purged_snaps scrub ok 2024-03-11T21:16:26.129 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:25 smithi146 bash[23809]: cluster 2024-03-11T21:16:24.995263+0000 osd.1 (osd.1) 334 : cluster 0 1.b scrub starts 2024-03-11T21:16:26.129 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:25 smithi146 bash[23809]: cluster 2024-03-11T21:16:24.997706+0000 osd.1 (osd.1) 335 : cluster 0 1.b scrub ok 2024-03-11T21:16:26.129 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:25 smithi146 bash[23809]: cluster 2024-03-11T21:16:25.021104+0000 osd.5 (osd.5) 565 : cluster 0 1.10 scrub starts 2024-03-11T21:16:26.129 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:25 smithi146 bash[23809]: cluster 2024-03-11T21:16:25.023474+0000 osd.5 (osd.5) 566 : cluster 0 1.10 scrub ok 2024-03-11T21:16:26.129 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:25 smithi146 bash[23809]: cluster 2024-03-11T21:16:25.359903+0000 osd.0 (osd.0) 430 : cluster 0 1.11 scrub starts 2024-03-11T21:16:26.129 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:25 smithi146 bash[23809]: cluster 2024-03-11T21:16:25.362162+0000 osd.0 (osd.0) 431 : cluster 0 1.11 scrub ok 2024-03-11T21:16:26.129 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:25 smithi146 bash[23809]: cluster 2024-03-11T21:16:25.435969+0000 mon.a (mon.0) 2001 : cluster 0 osdmap e451: 8 total, 8 up, 8 in 2024-03-11T21:16:26.504 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:16:26 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:16:26.859 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-11T21:16:27.254 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:26 smithi146 bash[23809]: cluster 2024-03-11T21:16:25.612968+0000 osd.3 (osd.3) 514 : cluster 0 1.c scrub starts 2024-03-11T21:16:27.255 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:26 smithi146 bash[23809]: cluster 2024-03-11T21:16:25.617532+0000 osd.3 (osd.3) 515 : cluster 0 1.c scrub ok 2024-03-11T21:16:27.255 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:26 smithi146 bash[23809]: cluster 2024-03-11T21:16:25.725747+0000 osd.6 (osd.6) 474 : cluster 0 1.f scrub starts 2024-03-11T21:16:27.255 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:26 smithi146 bash[23809]: cluster 2024-03-11T21:16:25.728858+0000 osd.6 (osd.6) 475 : cluster 0 1.f scrub ok 2024-03-11T21:16:27.255 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:26 smithi146 bash[23809]: cluster 2024-03-11T21:16:25.962574+0000 osd.7 (osd.7) 589 : cluster 0 1.d scrub starts 2024-03-11T21:16:27.255 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:26 smithi146 bash[23809]: cluster 2024-03-11T21:16:25.962704+0000 osd.1 (osd.1) 336 : cluster 0 1.13 scrub starts 2024-03-11T21:16:27.255 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:26 smithi146 bash[23809]: cluster 2024-03-11T21:16:25.964133+0000 osd.1 (osd.1) 337 : cluster 0 1.13 scrub ok 2024-03-11T21:16:27.255 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:26 smithi146 bash[23809]: cluster 2024-03-11T21:16:25.965264+0000 osd.7 (osd.7) 590 : cluster 0 1.d scrub ok 2024-03-11T21:16:27.255 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:26 smithi146 bash[23809]: cluster 2024-03-11T21:16:26.017420+0000 osd.5 (osd.5) 567 : cluster 0 1.12 scrub starts 2024-03-11T21:16:27.255 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:26 smithi146 bash[23809]: cluster 2024-03-11T21:16:26.020018+0000 osd.5 (osd.5) 568 : cluster 0 1.12 scrub ok 2024-03-11T21:16:27.255 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:26 smithi146 bash[23809]: audit 2024-03-11T21:16:26.140501+0000 mgr.y (mgr.24380) 947 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:16:27.255 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:26 smithi146 bash[23809]: cluster 2024-03-11T21:16:26.338703+0000 osd.0 (osd.0) 432 : cluster 0 3.3 deep-scrub starts 2024-03-11T21:16:27.255 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:26 smithi146 bash[23809]: cluster 2024-03-11T21:16:26.345048+0000 osd.0 (osd.0) 433 : cluster 0 3.3 deep-scrub ok 2024-03-11T21:16:27.256 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:26 smithi146 bash[23809]: cluster 2024-03-11T21:16:26.483028+0000 mgr.y (mgr.24380) 948 : cluster 0 pgmap v1099: 152 pgs: 2 active+undersized+remapped, 150 active+clean; 626 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 7.0 KiB/s rd, 46 KiB/s wr, 8 op/s; 33 KiB/s, 0 objects/s recovering 2024-03-11T21:16:27.275 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:26 smithi023 bash[21372]: cluster 2024-03-11T21:16:25.612968+0000 osd.3 (osd.3) 514 : cluster 0 1.c scrub starts 2024-03-11T21:16:27.276 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:26 smithi023 bash[21372]: cluster 2024-03-11T21:16:25.617532+0000 osd.3 (osd.3) 515 : cluster 0 1.c scrub ok 2024-03-11T21:16:27.276 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:26 smithi023 bash[21372]: cluster 2024-03-11T21:16:25.725747+0000 osd.6 (osd.6) 474 : cluster 0 1.f scrub starts 2024-03-11T21:16:27.276 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:26 smithi023 bash[21372]: cluster 2024-03-11T21:16:25.728858+0000 osd.6 (osd.6) 475 : cluster 0 1.f scrub ok 2024-03-11T21:16:27.276 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:26 smithi023 bash[21372]: cluster 2024-03-11T21:16:25.962574+0000 osd.7 (osd.7) 589 : cluster 0 1.d scrub starts 2024-03-11T21:16:27.276 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:26 smithi023 bash[21372]: cluster 2024-03-11T21:16:25.962704+0000 osd.1 (osd.1) 336 : cluster 0 1.13 scrub starts 2024-03-11T21:16:27.276 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:26 smithi023 bash[21372]: cluster 2024-03-11T21:16:25.964133+0000 osd.1 (osd.1) 337 : cluster 0 1.13 scrub ok 2024-03-11T21:16:27.276 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:26 smithi023 bash[21372]: cluster 2024-03-11T21:16:25.965264+0000 osd.7 (osd.7) 590 : cluster 0 1.d scrub ok 2024-03-11T21:16:27.276 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:26 smithi023 bash[21372]: cluster 2024-03-11T21:16:26.017420+0000 osd.5 (osd.5) 567 : cluster 0 1.12 scrub starts 2024-03-11T21:16:27.276 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:26 smithi023 bash[21372]: cluster 2024-03-11T21:16:26.020018+0000 osd.5 (osd.5) 568 : cluster 0 1.12 scrub ok 2024-03-11T21:16:27.276 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:26 smithi023 bash[21372]: audit 2024-03-11T21:16:26.140501+0000 mgr.y (mgr.24380) 947 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:16:27.276 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:26 smithi023 bash[21372]: cluster 2024-03-11T21:16:26.338703+0000 osd.0 (osd.0) 432 : cluster 0 3.3 deep-scrub starts 2024-03-11T21:16:27.277 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:26 smithi023 bash[21372]: cluster 2024-03-11T21:16:26.345048+0000 osd.0 (osd.0) 433 : cluster 0 3.3 deep-scrub ok 2024-03-11T21:16:27.277 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:26 smithi023 bash[21372]: cluster 2024-03-11T21:16:26.483028+0000 mgr.y (mgr.24380) 948 : cluster 0 pgmap v1099: 152 pgs: 2 active+undersized+remapped, 150 active+clean; 626 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 7.0 KiB/s rd, 46 KiB/s wr, 8 op/s; 33 KiB/s, 0 objects/s recovering 2024-03-11T21:16:27.277 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:26 smithi023 bash[29146]: cluster 2024-03-11T21:16:25.612968+0000 osd.3 (osd.3) 514 : cluster 0 1.c scrub starts 2024-03-11T21:16:27.277 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:26 smithi023 bash[29146]: cluster 2024-03-11T21:16:25.617532+0000 osd.3 (osd.3) 515 : cluster 0 1.c scrub ok 2024-03-11T21:16:27.277 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:26 smithi023 bash[29146]: cluster 2024-03-11T21:16:25.725747+0000 osd.6 (osd.6) 474 : cluster 0 1.f scrub starts 2024-03-11T21:16:27.277 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:26 smithi023 bash[29146]: cluster 2024-03-11T21:16:25.728858+0000 osd.6 (osd.6) 475 : cluster 0 1.f scrub ok 2024-03-11T21:16:27.277 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:26 smithi023 bash[29146]: cluster 2024-03-11T21:16:25.962574+0000 osd.7 (osd.7) 589 : cluster 0 1.d scrub starts 2024-03-11T21:16:27.277 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:26 smithi023 bash[29146]: cluster 2024-03-11T21:16:25.962704+0000 osd.1 (osd.1) 336 : cluster 0 1.13 scrub starts 2024-03-11T21:16:27.277 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:26 smithi023 bash[29146]: cluster 2024-03-11T21:16:25.964133+0000 osd.1 (osd.1) 337 : cluster 0 1.13 scrub ok 2024-03-11T21:16:27.277 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:26 smithi023 bash[29146]: cluster 2024-03-11T21:16:25.965264+0000 osd.7 (osd.7) 590 : cluster 0 1.d scrub ok 2024-03-11T21:16:27.277 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:26 smithi023 bash[29146]: cluster 2024-03-11T21:16:26.017420+0000 osd.5 (osd.5) 567 : cluster 0 1.12 scrub starts 2024-03-11T21:16:27.277 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:26 smithi023 bash[29146]: cluster 2024-03-11T21:16:26.020018+0000 osd.5 (osd.5) 568 : cluster 0 1.12 scrub ok 2024-03-11T21:16:27.278 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:26 smithi023 bash[29146]: audit 2024-03-11T21:16:26.140501+0000 mgr.y (mgr.24380) 947 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:16:27.278 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:26 smithi023 bash[29146]: cluster 2024-03-11T21:16:26.338703+0000 osd.0 (osd.0) 432 : cluster 0 3.3 deep-scrub starts 2024-03-11T21:16:27.278 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:26 smithi023 bash[29146]: cluster 2024-03-11T21:16:26.345048+0000 osd.0 (osd.0) 433 : cluster 0 3.3 deep-scrub ok 2024-03-11T21:16:27.278 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:26 smithi023 bash[29146]: cluster 2024-03-11T21:16:26.483028+0000 mgr.y (mgr.24380) 948 : cluster 0 pgmap v1099: 152 pgs: 2 active+undersized+remapped, 150 active+clean; 626 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 7.0 KiB/s rd, 46 KiB/s wr, 8 op/s; 33 KiB/s, 0 objects/s recovering 2024-03-11T21:16:27.334 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:16:28.254 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:27 smithi146 bash[23809]: cluster 2024-03-11T21:16:26.805332+0000 mon.a (mon.0) 2002 : cluster 1 Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-03-11T21:16:28.254 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:27 smithi146 bash[23809]: cluster 2024-03-11T21:16:26.805388+0000 mon.a (mon.0) 2003 : cluster 1 Cluster is now healthy 2024-03-11T21:16:28.254 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:27 smithi146 bash[23809]: cluster 2024-03-11T21:16:26.990505+0000 osd.7 (osd.7) 591 : cluster 0 1.14 scrub starts 2024-03-11T21:16:28.254 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:27 smithi146 bash[23809]: cluster 2024-03-11T21:16:26.992836+0000 osd.7 (osd.7) 592 : cluster 0 1.14 scrub ok 2024-03-11T21:16:28.275 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:27 smithi023 bash[21372]: cluster 2024-03-11T21:16:26.805332+0000 mon.a (mon.0) 2002 : cluster 1 Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-03-11T21:16:28.275 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:27 smithi023 bash[21372]: cluster 2024-03-11T21:16:26.805388+0000 mon.a (mon.0) 2003 : cluster 1 Cluster is now healthy 2024-03-11T21:16:28.275 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:27 smithi023 bash[21372]: cluster 2024-03-11T21:16:26.990505+0000 osd.7 (osd.7) 591 : cluster 0 1.14 scrub starts 2024-03-11T21:16:28.276 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:27 smithi023 bash[21372]: cluster 2024-03-11T21:16:26.992836+0000 osd.7 (osd.7) 592 : cluster 0 1.14 scrub ok 2024-03-11T21:16:28.276 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:27 smithi023 bash[29146]: cluster 2024-03-11T21:16:26.805332+0000 mon.a (mon.0) 2002 : cluster 1 Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-03-11T21:16:28.276 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:27 smithi023 bash[29146]: cluster 2024-03-11T21:16:26.805388+0000 mon.a (mon.0) 2003 : cluster 1 Cluster is now healthy 2024-03-11T21:16:28.276 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:27 smithi023 bash[29146]: cluster 2024-03-11T21:16:26.990505+0000 osd.7 (osd.7) 591 : cluster 0 1.14 scrub starts 2024-03-11T21:16:28.276 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:27 smithi023 bash[29146]: cluster 2024-03-11T21:16:26.992836+0000 osd.7 (osd.7) 592 : cluster 0 1.14 scrub ok 2024-03-11T21:16:28.634 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:16:28 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:16:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:16:29.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:28 smithi023 bash[29146]: cluster 2024-03-11T21:16:28.483638+0000 mgr.y (mgr.24380) 949 : cluster 0 pgmap v1100: 152 pgs: 152 active+clean; 626 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 4.9 KiB/s rd, 32 KiB/s wr, 5 op/s; 23 KiB/s, 0 objects/s recovering 2024-03-11T21:16:29.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:28 smithi023 bash[21372]: cluster 2024-03-11T21:16:28.483638+0000 mgr.y (mgr.24380) 949 : cluster 0 pgmap v1100: 152 pgs: 152 active+clean; 626 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 4.9 KiB/s rd, 32 KiB/s wr, 5 op/s; 23 KiB/s, 0 objects/s recovering 2024-03-11T21:16:29.254 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:28 smithi146 bash[23809]: cluster 2024-03-11T21:16:28.483638+0000 mgr.y (mgr.24380) 949 : cluster 0 pgmap v1100: 152 pgs: 152 active+clean; 626 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 4.9 KiB/s rd, 32 KiB/s wr, 5 op/s; 23 KiB/s, 0 objects/s recovering 2024-03-11T21:16:30.254 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:29 smithi146 bash[23809]: cluster 2024-03-11T21:16:28.712446+0000 osd.6 (osd.6) 476 : cluster 0 4.1f scrub starts 2024-03-11T21:16:30.254 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:29 smithi146 bash[23809]: cluster 2024-03-11T21:16:28.715532+0000 osd.6 (osd.6) 477 : cluster 0 4.1f scrub ok 2024-03-11T21:16:30.254 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:29 smithi146 bash[23809]: cluster 2024-03-11T21:16:28.923501+0000 osd.3 (osd.3) 516 : cluster 0 purged_snaps scrub starts 2024-03-11T21:16:30.254 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:29 smithi146 bash[23809]: cluster 2024-03-11T21:16:28.923620+0000 osd.3 (osd.3) 517 : cluster 0 purged_snaps scrub ok 2024-03-11T21:16:30.255 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:29 smithi146 bash[23809]: cluster 2024-03-11T21:16:29.012946+0000 osd.7 (osd.7) 593 : cluster 0 3.13 scrub starts 2024-03-11T21:16:30.255 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:29 smithi146 bash[23809]: cluster 2024-03-11T21:16:29.017317+0000 osd.7 (osd.7) 594 : cluster 0 3.13 scrub ok 2024-03-11T21:16:30.275 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:29 smithi023 bash[21372]: cluster 2024-03-11T21:16:28.712446+0000 osd.6 (osd.6) 476 : cluster 0 4.1f scrub starts 2024-03-11T21:16:30.275 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:29 smithi023 bash[21372]: cluster 2024-03-11T21:16:28.715532+0000 osd.6 (osd.6) 477 : cluster 0 4.1f scrub ok 2024-03-11T21:16:30.275 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:29 smithi023 bash[21372]: cluster 2024-03-11T21:16:28.923501+0000 osd.3 (osd.3) 516 : cluster 0 purged_snaps scrub starts 2024-03-11T21:16:30.276 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:29 smithi023 bash[21372]: cluster 2024-03-11T21:16:28.923620+0000 osd.3 (osd.3) 517 : cluster 0 purged_snaps scrub ok 2024-03-11T21:16:30.276 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:29 smithi023 bash[21372]: cluster 2024-03-11T21:16:29.012946+0000 osd.7 (osd.7) 593 : cluster 0 3.13 scrub starts 2024-03-11T21:16:30.276 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:29 smithi023 bash[21372]: cluster 2024-03-11T21:16:29.017317+0000 osd.7 (osd.7) 594 : cluster 0 3.13 scrub ok 2024-03-11T21:16:30.276 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:29 smithi023 bash[29146]: cluster 2024-03-11T21:16:28.712446+0000 osd.6 (osd.6) 476 : cluster 0 4.1f scrub starts 2024-03-11T21:16:30.276 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:29 smithi023 bash[29146]: cluster 2024-03-11T21:16:28.715532+0000 osd.6 (osd.6) 477 : cluster 0 4.1f scrub ok 2024-03-11T21:16:30.277 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:29 smithi023 bash[29146]: cluster 2024-03-11T21:16:28.923501+0000 osd.3 (osd.3) 516 : cluster 0 purged_snaps scrub starts 2024-03-11T21:16:30.277 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:29 smithi023 bash[29146]: cluster 2024-03-11T21:16:28.923620+0000 osd.3 (osd.3) 517 : cluster 0 purged_snaps scrub ok 2024-03-11T21:16:30.277 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:29 smithi023 bash[29146]: cluster 2024-03-11T21:16:29.012946+0000 osd.7 (osd.7) 593 : cluster 0 3.13 scrub starts 2024-03-11T21:16:30.277 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:29 smithi023 bash[29146]: cluster 2024-03-11T21:16:29.017317+0000 osd.7 (osd.7) 594 : cluster 0 3.13 scrub ok 2024-03-11T21:16:31.254 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:30 smithi146 bash[23809]: cluster 2024-03-11T21:16:29.846196+0000 mon.a (mon.0) 2004 : cluster 0 osdmap e452: 8 total, 8 up, 8 in 2024-03-11T21:16:31.254 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:30 smithi146 bash[23809]: cluster 2024-03-11T21:16:30.368616+0000 osd.0 (osd.0) 434 : cluster 0 4.1e scrub starts 2024-03-11T21:16:31.254 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:30 smithi146 bash[23809]: cluster 2024-03-11T21:16:30.370849+0000 osd.0 (osd.0) 435 : cluster 0 4.1e scrub ok 2024-03-11T21:16:31.254 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:30 smithi146 bash[23809]: cluster 2024-03-11T21:16:30.441249+0000 mon.a (mon.0) 2005 : cluster 0 osdmap e453: 8 total, 8 up, 8 in 2024-03-11T21:16:31.254 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:30 smithi146 bash[23809]: audit 2024-03-11T21:16:30.441858+0000 mon.a (mon.0) 2006 : audit 1 from='client.? 172.21.15.23:0/3275271803' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]: dispatch 2024-03-11T21:16:31.254 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:30 smithi146 bash[23809]: cluster 2024-03-11T21:16:30.484219+0000 mgr.y (mgr.24380) 950 : cluster 0 pgmap v1103: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 901 B/s rd, 0 op/s 2024-03-11T21:16:31.275 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:30 smithi023 bash[21372]: cluster 2024-03-11T21:16:29.846196+0000 mon.a (mon.0) 2004 : cluster 0 osdmap e452: 8 total, 8 up, 8 in 2024-03-11T21:16:31.275 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:30 smithi023 bash[21372]: cluster 2024-03-11T21:16:30.368616+0000 osd.0 (osd.0) 434 : cluster 0 4.1e scrub starts 2024-03-11T21:16:31.275 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:30 smithi023 bash[21372]: cluster 2024-03-11T21:16:30.370849+0000 osd.0 (osd.0) 435 : cluster 0 4.1e scrub ok 2024-03-11T21:16:31.275 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:30 smithi023 bash[21372]: cluster 2024-03-11T21:16:30.441249+0000 mon.a (mon.0) 2005 : cluster 0 osdmap e453: 8 total, 8 up, 8 in 2024-03-11T21:16:31.275 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:30 smithi023 bash[21372]: audit 2024-03-11T21:16:30.441858+0000 mon.a (mon.0) 2006 : audit 1 from='client.? 172.21.15.23:0/3275271803' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]: dispatch 2024-03-11T21:16:31.275 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:30 smithi023 bash[21372]: cluster 2024-03-11T21:16:30.484219+0000 mgr.y (mgr.24380) 950 : cluster 0 pgmap v1103: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 901 B/s rd, 0 op/s 2024-03-11T21:16:31.275 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:30 smithi023 bash[29146]: cluster 2024-03-11T21:16:29.846196+0000 mon.a (mon.0) 2004 : cluster 0 osdmap e452: 8 total, 8 up, 8 in 2024-03-11T21:16:31.276 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:30 smithi023 bash[29146]: cluster 2024-03-11T21:16:30.368616+0000 osd.0 (osd.0) 434 : cluster 0 4.1e scrub starts 2024-03-11T21:16:31.276 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:30 smithi023 bash[29146]: cluster 2024-03-11T21:16:30.370849+0000 osd.0 (osd.0) 435 : cluster 0 4.1e scrub ok 2024-03-11T21:16:31.276 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:30 smithi023 bash[29146]: cluster 2024-03-11T21:16:30.441249+0000 mon.a (mon.0) 2005 : cluster 0 osdmap e453: 8 total, 8 up, 8 in 2024-03-11T21:16:31.276 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:30 smithi023 bash[29146]: audit 2024-03-11T21:16:30.441858+0000 mon.a (mon.0) 2006 : audit 1 from='client.? 172.21.15.23:0/3275271803' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]: dispatch 2024-03-11T21:16:31.276 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:30 smithi023 bash[29146]: cluster 2024-03-11T21:16:30.484219+0000 mgr.y (mgr.24380) 950 : cluster 0 pgmap v1103: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 901 B/s rd, 0 op/s 2024-03-11T21:16:31.435 INFO:teuthology.orchestra.run.smithi023.stderr:set pool 1 pgp_num to 21 2024-03-11T21:16:31.741 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:16:32.238 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-03-11T21:16:32.525 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:32 smithi023 bash[29146]: cluster 2024-03-11T21:16:30.644112+0000 osd.6 (osd.6) 478 : cluster 0 6.1e deep-scrub starts 2024-03-11T21:16:32.525 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:32 smithi023 bash[29146]: cluster 2024-03-11T21:16:30.676867+0000 osd.6 (osd.6) 479 : cluster 0 6.1e deep-scrub ok 2024-03-11T21:16:32.525 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:32 smithi023 bash[29146]: cluster 2024-03-11T21:16:31.329354+0000 osd.0 (osd.0) 436 : cluster 0 6.10 scrub starts 2024-03-11T21:16:32.525 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:32 smithi023 bash[29146]: cluster 2024-03-11T21:16:31.411757+0000 osd.0 (osd.0) 437 : cluster 0 6.10 scrub ok 2024-03-11T21:16:32.525 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:32 smithi023 bash[29146]: audit 2024-03-11T21:16:31.443105+0000 mon.a (mon.0) 2007 : audit 1 from='client.? 172.21.15.23:0/3275271803' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]': finished 2024-03-11T21:16:32.525 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:32 smithi023 bash[29146]: cluster 2024-03-11T21:16:31.451022+0000 mon.a (mon.0) 2008 : cluster 0 osdmap e454: 8 total, 8 up, 8 in 2024-03-11T21:16:32.525 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:32 smithi023 bash[29146]: cluster 2024-03-11T21:16:31.713554+0000 osd.3 (osd.3) 518 : cluster 0 4.10 scrub starts 2024-03-11T21:16:32.525 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:32 smithi023 bash[29146]: cluster 2024-03-11T21:16:31.715915+0000 osd.3 (osd.3) 519 : cluster 0 4.10 scrub ok 2024-03-11T21:16:32.525 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:32 smithi023 bash[29146]: audit 2024-03-11T21:16:31.817755+0000 mon.a (mon.0) 2009 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:16:32.526 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:32 smithi023 bash[21372]: cluster 2024-03-11T21:16:30.644112+0000 osd.6 (osd.6) 478 : cluster 0 6.1e deep-scrub starts 2024-03-11T21:16:32.526 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:32 smithi023 bash[21372]: cluster 2024-03-11T21:16:30.676867+0000 osd.6 (osd.6) 479 : cluster 0 6.1e deep-scrub ok 2024-03-11T21:16:32.526 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:32 smithi023 bash[21372]: cluster 2024-03-11T21:16:31.329354+0000 osd.0 (osd.0) 436 : cluster 0 6.10 scrub starts 2024-03-11T21:16:32.526 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:32 smithi023 bash[21372]: cluster 2024-03-11T21:16:31.411757+0000 osd.0 (osd.0) 437 : cluster 0 6.10 scrub ok 2024-03-11T21:16:32.526 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:32 smithi023 bash[21372]: audit 2024-03-11T21:16:31.443105+0000 mon.a (mon.0) 2007 : audit 1 from='client.? 172.21.15.23:0/3275271803' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]': finished 2024-03-11T21:16:32.526 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:32 smithi023 bash[21372]: cluster 2024-03-11T21:16:31.451022+0000 mon.a (mon.0) 2008 : cluster 0 osdmap e454: 8 total, 8 up, 8 in 2024-03-11T21:16:32.526 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:32 smithi023 bash[21372]: cluster 2024-03-11T21:16:31.713554+0000 osd.3 (osd.3) 518 : cluster 0 4.10 scrub starts 2024-03-11T21:16:32.526 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:32 smithi023 bash[21372]: cluster 2024-03-11T21:16:31.715915+0000 osd.3 (osd.3) 519 : cluster 0 4.10 scrub ok 2024-03-11T21:16:32.526 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:32 smithi023 bash[21372]: audit 2024-03-11T21:16:31.817755+0000 mon.a (mon.0) 2009 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:16:32.754 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:32 smithi146 bash[23809]: cluster 2024-03-11T21:16:30.644112+0000 osd.6 (osd.6) 478 : cluster 0 6.1e deep-scrub starts 2024-03-11T21:16:32.754 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:32 smithi146 bash[23809]: cluster 2024-03-11T21:16:30.676867+0000 osd.6 (osd.6) 479 : cluster 0 6.1e deep-scrub ok 2024-03-11T21:16:32.754 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:32 smithi146 bash[23809]: cluster 2024-03-11T21:16:31.329354+0000 osd.0 (osd.0) 436 : cluster 0 6.10 scrub starts 2024-03-11T21:16:32.754 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:32 smithi146 bash[23809]: cluster 2024-03-11T21:16:31.411757+0000 osd.0 (osd.0) 437 : cluster 0 6.10 scrub ok 2024-03-11T21:16:32.754 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:32 smithi146 bash[23809]: audit 2024-03-11T21:16:31.443105+0000 mon.a (mon.0) 2007 : audit 1 from='client.? 172.21.15.23:0/3275271803' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]': finished 2024-03-11T21:16:32.754 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:32 smithi146 bash[23809]: cluster 2024-03-11T21:16:31.451022+0000 mon.a (mon.0) 2008 : cluster 0 osdmap e454: 8 total, 8 up, 8 in 2024-03-11T21:16:32.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:32 smithi146 bash[23809]: cluster 2024-03-11T21:16:31.713554+0000 osd.3 (osd.3) 518 : cluster 0 4.10 scrub starts 2024-03-11T21:16:32.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:32 smithi146 bash[23809]: cluster 2024-03-11T21:16:31.715915+0000 osd.3 (osd.3) 519 : cluster 0 4.10 scrub ok 2024-03-11T21:16:32.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:32 smithi146 bash[23809]: audit 2024-03-11T21:16:31.817755+0000 mon.a (mon.0) 2009 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:16:33.754 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:33 smithi146 bash[23809]: cluster 2024-03-11T21:16:31.592580+0000 osd.6 (osd.6) 480 : cluster 0 3.a scrub starts 2024-03-11T21:16:33.754 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:33 smithi146 bash[23809]: cluster 2024-03-11T21:16:31.599354+0000 osd.6 (osd.6) 481 : cluster 0 3.a scrub ok 2024-03-11T21:16:33.754 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:33 smithi146 bash[23809]: cluster 2024-03-11T21:16:32.108088+0000 osd.7 (osd.7) 595 : cluster 0 6.5 scrub starts 2024-03-11T21:16:33.754 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:33 smithi146 bash[23809]: cluster 2024-03-11T21:16:32.201417+0000 osd.7 (osd.7) 596 : cluster 0 6.5 scrub ok 2024-03-11T21:16:33.754 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:33 smithi146 bash[23809]: cluster 2024-03-11T21:16:32.484583+0000 mgr.y (mgr.24380) 951 : cluster 0 pgmap v1105: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail 2024-03-11T21:16:33.754 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:33 smithi146 bash[23809]: cluster 2024-03-11T21:16:32.667422+0000 osd.3 (osd.3) 520 : cluster 0 3.b scrub starts 2024-03-11T21:16:33.754 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:33 smithi146 bash[23809]: cluster 2024-03-11T21:16:32.671240+0000 osd.3 (osd.3) 521 : cluster 0 3.b scrub ok 2024-03-11T21:16:33.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:33 smithi023 bash[21372]: cluster 2024-03-11T21:16:31.592580+0000 osd.6 (osd.6) 480 : cluster 0 3.a scrub starts 2024-03-11T21:16:33.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:33 smithi023 bash[21372]: cluster 2024-03-11T21:16:31.599354+0000 osd.6 (osd.6) 481 : cluster 0 3.a scrub ok 2024-03-11T21:16:33.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:33 smithi023 bash[21372]: cluster 2024-03-11T21:16:32.108088+0000 osd.7 (osd.7) 595 : cluster 0 6.5 scrub starts 2024-03-11T21:16:33.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:33 smithi023 bash[21372]: cluster 2024-03-11T21:16:32.201417+0000 osd.7 (osd.7) 596 : cluster 0 6.5 scrub ok 2024-03-11T21:16:33.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:33 smithi023 bash[21372]: cluster 2024-03-11T21:16:32.484583+0000 mgr.y (mgr.24380) 951 : cluster 0 pgmap v1105: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail 2024-03-11T21:16:33.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:33 smithi023 bash[21372]: cluster 2024-03-11T21:16:32.667422+0000 osd.3 (osd.3) 520 : cluster 0 3.b scrub starts 2024-03-11T21:16:33.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:33 smithi023 bash[21372]: cluster 2024-03-11T21:16:32.671240+0000 osd.3 (osd.3) 521 : cluster 0 3.b scrub ok 2024-03-11T21:16:33.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:33 smithi023 bash[29146]: cluster 2024-03-11T21:16:31.592580+0000 osd.6 (osd.6) 480 : cluster 0 3.a scrub starts 2024-03-11T21:16:33.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:33 smithi023 bash[29146]: cluster 2024-03-11T21:16:31.599354+0000 osd.6 (osd.6) 481 : cluster 0 3.a scrub ok 2024-03-11T21:16:33.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:33 smithi023 bash[29146]: cluster 2024-03-11T21:16:32.108088+0000 osd.7 (osd.7) 595 : cluster 0 6.5 scrub starts 2024-03-11T21:16:33.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:33 smithi023 bash[29146]: cluster 2024-03-11T21:16:32.201417+0000 osd.7 (osd.7) 596 : cluster 0 6.5 scrub ok 2024-03-11T21:16:33.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:33 smithi023 bash[29146]: cluster 2024-03-11T21:16:32.484583+0000 mgr.y (mgr.24380) 951 : cluster 0 pgmap v1105: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail 2024-03-11T21:16:33.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:33 smithi023 bash[29146]: cluster 2024-03-11T21:16:32.667422+0000 osd.3 (osd.3) 520 : cluster 0 3.b scrub starts 2024-03-11T21:16:33.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:33 smithi023 bash[29146]: cluster 2024-03-11T21:16:32.671240+0000 osd.3 (osd.3) 521 : cluster 0 3.b scrub ok 2024-03-11T21:16:34.754 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:34 smithi146 bash[23809]: cluster 2024-03-11T21:16:33.090326+0000 osd.7 (osd.7) 597 : cluster 0 6.1c scrub starts 2024-03-11T21:16:34.754 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:34 smithi146 bash[23809]: cluster 2024-03-11T21:16:33.092463+0000 osd.7 (osd.7) 598 : cluster 0 6.1c scrub ok 2024-03-11T21:16:34.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:34 smithi023 bash[21372]: cluster 2024-03-11T21:16:33.090326+0000 osd.7 (osd.7) 597 : cluster 0 6.1c scrub starts 2024-03-11T21:16:34.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:34 smithi023 bash[21372]: cluster 2024-03-11T21:16:33.092463+0000 osd.7 (osd.7) 598 : cluster 0 6.1c scrub ok 2024-03-11T21:16:34.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:34 smithi023 bash[29146]: cluster 2024-03-11T21:16:33.090326+0000 osd.7 (osd.7) 597 : cluster 0 6.1c scrub starts 2024-03-11T21:16:34.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:34 smithi023 bash[29146]: cluster 2024-03-11T21:16:33.092463+0000 osd.7 (osd.7) 598 : cluster 0 6.1c scrub ok 2024-03-11T21:16:35.130 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-03-11T21:16:35.131 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:16:35.754 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:35 smithi146 bash[23809]: cluster 2024-03-11T21:16:34.485412+0000 mgr.y (mgr.24380) 952 : cluster 0 pgmap v1106: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail 2024-03-11T21:16:35.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:35 smithi023 bash[21372]: cluster 2024-03-11T21:16:34.485412+0000 mgr.y (mgr.24380) 952 : cluster 0 pgmap v1106: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail 2024-03-11T21:16:35.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:35 smithi023 bash[29146]: cluster 2024-03-11T21:16:34.485412+0000 mgr.y (mgr.24380) 952 : cluster 0 pgmap v1106: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail 2024-03-11T21:16:35.889 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-03-11T21:16:36.463 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:16:36 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:16:36.754 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:36 smithi146 bash[23809]: cluster 2024-03-11T21:16:34.623927+0000 osd.6 (osd.6) 482 : cluster 0 5.2 deep-scrub starts 2024-03-11T21:16:36.754 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:36 smithi146 bash[23809]: cluster 2024-03-11T21:16:34.671314+0000 osd.6 (osd.6) 483 : cluster 0 5.2 deep-scrub ok 2024-03-11T21:16:36.754 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:36 smithi146 bash[23809]: cluster 2024-03-11T21:16:35.079703+0000 osd.7 (osd.7) 599 : cluster 0 2.b scrub starts 2024-03-11T21:16:36.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:36 smithi146 bash[23809]: cluster 2024-03-11T21:16:35.081817+0000 osd.7 (osd.7) 600 : cluster 0 2.b scrub ok 2024-03-11T21:16:36.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:36 smithi146 bash[23809]: cluster 2024-03-11T21:16:35.668926+0000 osd.3 (osd.3) 522 : cluster 0 4.0 scrub starts 2024-03-11T21:16:36.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:36 smithi146 bash[23809]: cluster 2024-03-11T21:16:35.672364+0000 osd.3 (osd.3) 523 : cluster 0 4.0 scrub ok 2024-03-11T21:16:36.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:36 smithi023 bash[21372]: cluster 2024-03-11T21:16:34.623927+0000 osd.6 (osd.6) 482 : cluster 0 5.2 deep-scrub starts 2024-03-11T21:16:36.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:36 smithi023 bash[21372]: cluster 2024-03-11T21:16:34.671314+0000 osd.6 (osd.6) 483 : cluster 0 5.2 deep-scrub ok 2024-03-11T21:16:36.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:36 smithi023 bash[21372]: cluster 2024-03-11T21:16:35.079703+0000 osd.7 (osd.7) 599 : cluster 0 2.b scrub starts 2024-03-11T21:16:36.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:36 smithi023 bash[21372]: cluster 2024-03-11T21:16:35.081817+0000 osd.7 (osd.7) 600 : cluster 0 2.b scrub ok 2024-03-11T21:16:36.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:36 smithi023 bash[21372]: cluster 2024-03-11T21:16:35.668926+0000 osd.3 (osd.3) 522 : cluster 0 4.0 scrub starts 2024-03-11T21:16:36.776 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:36 smithi023 bash[21372]: cluster 2024-03-11T21:16:35.672364+0000 osd.3 (osd.3) 523 : cluster 0 4.0 scrub ok 2024-03-11T21:16:36.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:36 smithi023 bash[29146]: cluster 2024-03-11T21:16:34.623927+0000 osd.6 (osd.6) 482 : cluster 0 5.2 deep-scrub starts 2024-03-11T21:16:36.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:36 smithi023 bash[29146]: cluster 2024-03-11T21:16:34.671314+0000 osd.6 (osd.6) 483 : cluster 0 5.2 deep-scrub ok 2024-03-11T21:16:36.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:36 smithi023 bash[29146]: cluster 2024-03-11T21:16:35.079703+0000 osd.7 (osd.7) 599 : cluster 0 2.b scrub starts 2024-03-11T21:16:36.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:36 smithi023 bash[29146]: cluster 2024-03-11T21:16:35.081817+0000 osd.7 (osd.7) 600 : cluster 0 2.b scrub ok 2024-03-11T21:16:36.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:36 smithi023 bash[29146]: cluster 2024-03-11T21:16:35.668926+0000 osd.3 (osd.3) 522 : cluster 0 4.0 scrub starts 2024-03-11T21:16:36.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:36 smithi023 bash[29146]: cluster 2024-03-11T21:16:35.672364+0000 osd.3 (osd.3) 523 : cluster 0 4.0 scrub ok 2024-03-11T21:16:37.028 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:16:37.754 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:37 smithi146 bash[23809]: cluster 2024-03-11T21:16:35.664637+0000 osd.6 (osd.6) 484 : cluster 0 4.18 scrub starts 2024-03-11T21:16:37.754 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:37 smithi146 bash[23809]: cluster 2024-03-11T21:16:35.667317+0000 osd.6 (osd.6) 485 : cluster 0 4.18 scrub ok 2024-03-11T21:16:37.754 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:37 smithi146 bash[23809]: audit 2024-03-11T21:16:36.151813+0000 mgr.y (mgr.24380) 953 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:16:37.754 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:37 smithi146 bash[23809]: cluster 2024-03-11T21:16:36.487031+0000 mgr.y (mgr.24380) 954 : cluster 0 pgmap v1107: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail 2024-03-11T21:16:37.754 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:37 smithi146 bash[23809]: audit 2024-03-11T21:16:36.714955+0000 mon.a (mon.0) 2010 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:16:37.754 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:37 smithi146 bash[23809]: audit 2024-03-11T21:16:36.717441+0000 mon.a (mon.0) 2011 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:16:37.754 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:37 smithi146 bash[23809]: cluster 2024-03-11T21:16:36.904943+0000 osd.1 (osd.1) 338 : cluster 0 6.e scrub starts 2024-03-11T21:16:37.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:37 smithi146 bash[23809]: cluster 2024-03-11T21:16:36.987022+0000 osd.1 (osd.1) 339 : cluster 0 6.e scrub ok 2024-03-11T21:16:37.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:37 smithi023 bash[21372]: cluster 2024-03-11T21:16:35.664637+0000 osd.6 (osd.6) 484 : cluster 0 4.18 scrub starts 2024-03-11T21:16:37.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:37 smithi023 bash[21372]: cluster 2024-03-11T21:16:35.667317+0000 osd.6 (osd.6) 485 : cluster 0 4.18 scrub ok 2024-03-11T21:16:37.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:37 smithi023 bash[21372]: audit 2024-03-11T21:16:36.151813+0000 mgr.y (mgr.24380) 953 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:16:37.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:37 smithi023 bash[21372]: cluster 2024-03-11T21:16:36.487031+0000 mgr.y (mgr.24380) 954 : cluster 0 pgmap v1107: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail 2024-03-11T21:16:37.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:37 smithi023 bash[21372]: audit 2024-03-11T21:16:36.714955+0000 mon.a (mon.0) 2010 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:16:37.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:37 smithi023 bash[21372]: audit 2024-03-11T21:16:36.717441+0000 mon.a (mon.0) 2011 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:16:37.776 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:37 smithi023 bash[21372]: cluster 2024-03-11T21:16:36.904943+0000 osd.1 (osd.1) 338 : cluster 0 6.e scrub starts 2024-03-11T21:16:37.776 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:37 smithi023 bash[21372]: cluster 2024-03-11T21:16:36.987022+0000 osd.1 (osd.1) 339 : cluster 0 6.e scrub ok 2024-03-11T21:16:37.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:37 smithi023 bash[29146]: cluster 2024-03-11T21:16:35.664637+0000 osd.6 (osd.6) 484 : cluster 0 4.18 scrub starts 2024-03-11T21:16:37.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:37 smithi023 bash[29146]: cluster 2024-03-11T21:16:35.667317+0000 osd.6 (osd.6) 485 : cluster 0 4.18 scrub ok 2024-03-11T21:16:37.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:37 smithi023 bash[29146]: audit 2024-03-11T21:16:36.151813+0000 mgr.y (mgr.24380) 953 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:16:37.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:37 smithi023 bash[29146]: cluster 2024-03-11T21:16:36.487031+0000 mgr.y (mgr.24380) 954 : cluster 0 pgmap v1107: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail 2024-03-11T21:16:37.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:37 smithi023 bash[29146]: audit 2024-03-11T21:16:36.714955+0000 mon.a (mon.0) 2010 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:16:37.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:37 smithi023 bash[29146]: audit 2024-03-11T21:16:36.717441+0000 mon.a (mon.0) 2011 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:16:37.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:37 smithi023 bash[29146]: cluster 2024-03-11T21:16:36.904943+0000 osd.1 (osd.1) 338 : cluster 0 6.e scrub starts 2024-03-11T21:16:37.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:37 smithi023 bash[29146]: cluster 2024-03-11T21:16:36.987022+0000 osd.1 (osd.1) 339 : cluster 0 6.e scrub ok 2024-03-11T21:16:38.508 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:16:38 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:16:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:16:39.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:39 smithi023 bash[21372]: cluster 2024-03-11T21:16:37.617790+0000 osd.6 (osd.6) 486 : cluster 0 4.13 scrub starts 2024-03-11T21:16:39.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:39 smithi023 bash[21372]: cluster 2024-03-11T21:16:37.620539+0000 osd.6 (osd.6) 487 : cluster 0 4.13 scrub ok 2024-03-11T21:16:39.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:39 smithi023 bash[21372]: cluster 2024-03-11T21:16:38.487986+0000 mgr.y (mgr.24380) 955 : cluster 0 pgmap v1108: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:16:39.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:39 smithi023 bash[21372]: cluster 2024-03-11T21:16:38.574810+0000 osd.6 (osd.6) 488 : cluster 0 4.a scrub starts 2024-03-11T21:16:39.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:39 smithi023 bash[21372]: cluster 2024-03-11T21:16:38.577389+0000 osd.6 (osd.6) 489 : cluster 0 4.a scrub ok 2024-03-11T21:16:39.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:39 smithi023 bash[21372]: cluster 2024-03-11T21:16:38.914993+0000 osd.1 (osd.1) 340 : cluster 0 4.1c scrub starts 2024-03-11T21:16:39.776 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:39 smithi023 bash[21372]: cluster 2024-03-11T21:16:38.917565+0000 osd.1 (osd.1) 341 : cluster 0 4.1c scrub ok 2024-03-11T21:16:39.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:39 smithi023 bash[29146]: cluster 2024-03-11T21:16:37.617790+0000 osd.6 (osd.6) 486 : cluster 0 4.13 scrub starts 2024-03-11T21:16:39.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:39 smithi023 bash[29146]: cluster 2024-03-11T21:16:37.620539+0000 osd.6 (osd.6) 487 : cluster 0 4.13 scrub ok 2024-03-11T21:16:39.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:39 smithi023 bash[29146]: cluster 2024-03-11T21:16:38.487986+0000 mgr.y (mgr.24380) 955 : cluster 0 pgmap v1108: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:16:39.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:39 smithi023 bash[29146]: cluster 2024-03-11T21:16:38.574810+0000 osd.6 (osd.6) 488 : cluster 0 4.a scrub starts 2024-03-11T21:16:39.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:39 smithi023 bash[29146]: cluster 2024-03-11T21:16:38.577389+0000 osd.6 (osd.6) 489 : cluster 0 4.a scrub ok 2024-03-11T21:16:39.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:39 smithi023 bash[29146]: cluster 2024-03-11T21:16:38.914993+0000 osd.1 (osd.1) 340 : cluster 0 4.1c scrub starts 2024-03-11T21:16:39.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:39 smithi023 bash[29146]: cluster 2024-03-11T21:16:38.917565+0000 osd.1 (osd.1) 341 : cluster 0 4.1c scrub ok 2024-03-11T21:16:40.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:39 smithi146 bash[23809]: cluster 2024-03-11T21:16:37.617790+0000 osd.6 (osd.6) 486 : cluster 0 4.13 scrub starts 2024-03-11T21:16:40.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:39 smithi146 bash[23809]: cluster 2024-03-11T21:16:37.620539+0000 osd.6 (osd.6) 487 : cluster 0 4.13 scrub ok 2024-03-11T21:16:40.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:39 smithi146 bash[23809]: cluster 2024-03-11T21:16:38.487986+0000 mgr.y (mgr.24380) 955 : cluster 0 pgmap v1108: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:16:40.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:39 smithi146 bash[23809]: cluster 2024-03-11T21:16:38.574810+0000 osd.6 (osd.6) 488 : cluster 0 4.a scrub starts 2024-03-11T21:16:40.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:39 smithi146 bash[23809]: cluster 2024-03-11T21:16:38.577389+0000 osd.6 (osd.6) 489 : cluster 0 4.a scrub ok 2024-03-11T21:16:40.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:39 smithi146 bash[23809]: cluster 2024-03-11T21:16:38.914993+0000 osd.1 (osd.1) 340 : cluster 0 4.1c scrub starts 2024-03-11T21:16:40.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:39 smithi146 bash[23809]: cluster 2024-03-11T21:16:38.917565+0000 osd.1 (osd.1) 341 : cluster 0 4.1c scrub ok 2024-03-11T21:16:40.685 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:16:40.800 INFO:teuthology.orchestra.run.smithi023.stdout:mon.a: {} 2024-03-11T21:16:40.815 INFO:teuthology.orchestra.run.smithi023.stdout:mon.b: {} 2024-03-11T21:16:40.827 INFO:teuthology.orchestra.run.smithi023.stdout:mon.c: {} 2024-03-11T21:16:40.828 INFO:teuthology.orchestra.run.smithi023.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-11T21:16:40.828 INFO:teuthology.orchestra.run.smithi023.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-11T21:16:40.828 INFO:teuthology.orchestra.run.smithi023.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-11T21:16:41.734 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:41 smithi023 bash[21372]: cluster 2024-03-11T21:16:40.488835+0000 mgr.y (mgr.24380) 956 : cluster 0 pgmap v1109: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:16:41.734 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:41 smithi023 bash[21372]: cluster 2024-03-11T21:16:40.611585+0000 osd.3 (osd.3) 524 : cluster 0 4.1a deep-scrub starts 2024-03-11T21:16:41.734 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:41 smithi023 bash[21372]: cluster 2024-03-11T21:16:40.613423+0000 osd.3 (osd.3) 525 : cluster 0 4.1a deep-scrub ok 2024-03-11T21:16:41.735 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:41 smithi023 bash[21372]: audit 2024-03-11T21:16:40.801101+0000 mon.c (mon.2) 105 : audit 0 from='client.? 172.21.15.23:0/1441175701' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T21:16:41.735 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:41 smithi023 bash[21372]: cluster 2024-03-11T21:16:40.923443+0000 osd.1 (osd.1) 342 : cluster 0 2.9 deep-scrub starts 2024-03-11T21:16:41.735 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:41 smithi023 bash[21372]: cluster 2024-03-11T21:16:40.924948+0000 osd.1 (osd.1) 343 : cluster 0 2.9 deep-scrub ok 2024-03-11T21:16:41.735 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:41 smithi023 bash[29146]: cluster 2024-03-11T21:16:40.488835+0000 mgr.y (mgr.24380) 956 : cluster 0 pgmap v1109: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:16:41.735 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:41 smithi023 bash[29146]: cluster 2024-03-11T21:16:40.611585+0000 osd.3 (osd.3) 524 : cluster 0 4.1a deep-scrub starts 2024-03-11T21:16:41.736 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:41 smithi023 bash[29146]: cluster 2024-03-11T21:16:40.613423+0000 osd.3 (osd.3) 525 : cluster 0 4.1a deep-scrub ok 2024-03-11T21:16:41.736 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:41 smithi023 bash[29146]: audit 2024-03-11T21:16:40.801101+0000 mon.c (mon.2) 105 : audit 0 from='client.? 172.21.15.23:0/1441175701' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T21:16:41.736 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:41 smithi023 bash[29146]: cluster 2024-03-11T21:16:40.923443+0000 osd.1 (osd.1) 342 : cluster 0 2.9 deep-scrub starts 2024-03-11T21:16:41.736 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:41 smithi023 bash[29146]: cluster 2024-03-11T21:16:40.924948+0000 osd.1 (osd.1) 343 : cluster 0 2.9 deep-scrub ok 2024-03-11T21:16:41.738 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd reweight 0 1 2024-03-11T21:16:41.747 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:41 smithi146 bash[23809]: cluster 2024-03-11T21:16:40.488835+0000 mgr.y (mgr.24380) 956 : cluster 0 pgmap v1109: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:16:41.747 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:41 smithi146 bash[23809]: cluster 2024-03-11T21:16:40.611585+0000 osd.3 (osd.3) 524 : cluster 0 4.1a deep-scrub starts 2024-03-11T21:16:41.747 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:41 smithi146 bash[23809]: cluster 2024-03-11T21:16:40.613423+0000 osd.3 (osd.3) 525 : cluster 0 4.1a deep-scrub ok 2024-03-11T21:16:41.747 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:41 smithi146 bash[23809]: audit 2024-03-11T21:16:40.801101+0000 mon.c (mon.2) 105 : audit 0 from='client.? 172.21.15.23:0/1441175701' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T21:16:41.747 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:41 smithi146 bash[23809]: cluster 2024-03-11T21:16:40.923443+0000 osd.1 (osd.1) 342 : cluster 0 2.9 deep-scrub starts 2024-03-11T21:16:41.747 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:41 smithi146 bash[23809]: cluster 2024-03-11T21:16:40.924948+0000 osd.1 (osd.1) 343 : cluster 0 2.9 deep-scrub ok 2024-03-11T21:16:42.754 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:42 smithi146 bash[23809]: cluster 2024-03-11T21:16:41.581642+0000 osd.3 (osd.3) 526 : cluster 0 3.8 scrub starts 2024-03-11T21:16:42.754 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:42 smithi146 bash[23809]: cluster 2024-03-11T21:16:41.586809+0000 osd.3 (osd.3) 527 : cluster 0 3.8 scrub ok 2024-03-11T21:16:42.757 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:42 smithi023 bash[29146]: cluster 2024-03-11T21:16:41.581642+0000 osd.3 (osd.3) 526 : cluster 0 3.8 scrub starts 2024-03-11T21:16:42.757 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:42 smithi023 bash[29146]: cluster 2024-03-11T21:16:41.586809+0000 osd.3 (osd.3) 527 : cluster 0 3.8 scrub ok 2024-03-11T21:16:42.757 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:42 smithi023 bash[21372]: cluster 2024-03-11T21:16:41.581642+0000 osd.3 (osd.3) 526 : cluster 0 3.8 scrub starts 2024-03-11T21:16:42.757 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:42 smithi023 bash[21372]: cluster 2024-03-11T21:16:41.586809+0000 osd.3 (osd.3) 527 : cluster 0 3.8 scrub ok 2024-03-11T21:16:43.730 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-03-11T21:16:43.730 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:16:43.741 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:43 smithi146 bash[23809]: cluster 2024-03-11T21:16:42.048121+0000 osd.5 (osd.5) 569 : cluster 0 3.5 scrub starts 2024-03-11T21:16:43.741 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:43 smithi146 bash[23809]: cluster 2024-03-11T21:16:42.052823+0000 osd.5 (osd.5) 570 : cluster 0 3.5 scrub ok 2024-03-11T21:16:43.742 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:43 smithi146 bash[23809]: cluster 2024-03-11T21:16:42.489625+0000 mgr.y (mgr.24380) 957 : cluster 0 pgmap v1110: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 927 B/s rd, 0 op/s 2024-03-11T21:16:43.742 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:43 smithi146 bash[23809]: cluster 2024-03-11T21:16:42.609800+0000 osd.3 (osd.3) 528 : cluster 0 6.1f scrub starts 2024-03-11T21:16:43.742 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:43 smithi146 bash[23809]: cluster 2024-03-11T21:16:42.817201+0000 osd.3 (osd.3) 529 : cluster 0 6.1f scrub ok 2024-03-11T21:16:43.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:43 smithi023 bash[29146]: cluster 2024-03-11T21:16:42.048121+0000 osd.5 (osd.5) 569 : cluster 0 3.5 scrub starts 2024-03-11T21:16:43.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:43 smithi023 bash[29146]: cluster 2024-03-11T21:16:42.052823+0000 osd.5 (osd.5) 570 : cluster 0 3.5 scrub ok 2024-03-11T21:16:43.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:43 smithi023 bash[29146]: cluster 2024-03-11T21:16:42.489625+0000 mgr.y (mgr.24380) 957 : cluster 0 pgmap v1110: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 927 B/s rd, 0 op/s 2024-03-11T21:16:43.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:43 smithi023 bash[29146]: cluster 2024-03-11T21:16:42.609800+0000 osd.3 (osd.3) 528 : cluster 0 6.1f scrub starts 2024-03-11T21:16:43.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:43 smithi023 bash[29146]: cluster 2024-03-11T21:16:42.817201+0000 osd.3 (osd.3) 529 : cluster 0 6.1f scrub ok 2024-03-11T21:16:43.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:43 smithi023 bash[21372]: cluster 2024-03-11T21:16:42.048121+0000 osd.5 (osd.5) 569 : cluster 0 3.5 scrub starts 2024-03-11T21:16:43.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:43 smithi023 bash[21372]: cluster 2024-03-11T21:16:42.052823+0000 osd.5 (osd.5) 570 : cluster 0 3.5 scrub ok 2024-03-11T21:16:43.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:43 smithi023 bash[21372]: cluster 2024-03-11T21:16:42.489625+0000 mgr.y (mgr.24380) 957 : cluster 0 pgmap v1110: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 927 B/s rd, 0 op/s 2024-03-11T21:16:43.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:43 smithi023 bash[21372]: cluster 2024-03-11T21:16:42.609800+0000 osd.3 (osd.3) 528 : cluster 0 6.1f scrub starts 2024-03-11T21:16:43.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:43 smithi023 bash[21372]: cluster 2024-03-11T21:16:42.817201+0000 osd.3 (osd.3) 529 : cluster 0 6.1f scrub ok 2024-03-11T21:16:44.391 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-03-11T21:16:44.754 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:44 smithi146 bash[23809]: cluster 2024-03-11T21:16:42.604048+0000 osd.6 (osd.6) 490 : cluster 0 3.10 scrub starts 2024-03-11T21:16:44.754 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:44 smithi146 bash[23809]: cluster 2024-03-11T21:16:42.606005+0000 osd.6 (osd.6) 491 : cluster 0 3.10 scrub ok 2024-03-11T21:16:44.754 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:44 smithi146 bash[23809]: cluster 2024-03-11T21:16:43.016896+0000 osd.5 (osd.5) 571 : cluster 0 2.16 scrub starts 2024-03-11T21:16:44.754 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:44 smithi146 bash[23809]: cluster 2024-03-11T21:16:43.019364+0000 osd.5 (osd.5) 572 : cluster 0 2.16 scrub ok 2024-03-11T21:16:44.754 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:44 smithi146 bash[23809]: audit 2024-03-11T21:16:43.711753+0000 mon.a (mon.0) 2012 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:16:44.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:44 smithi146 bash[23809]: audit 2024-03-11T21:16:43.732773+0000 mon.a (mon.0) 2013 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:16:44.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:44 smithi146 bash[23809]: audit 2024-03-11T21:16:43.735401+0000 mon.a (mon.0) 2014 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:16:44.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:44 smithi146 bash[23809]: audit 2024-03-11T21:16:43.737066+0000 mon.a (mon.0) 2015 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:16:44.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:44 smithi146 bash[23809]: audit 2024-03-11T21:16:43.754718+0000 mon.a (mon.0) 2016 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:16:44.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:44 smithi146 bash[23809]: cluster 2024-03-11T21:16:43.989371+0000 osd.1 (osd.1) 344 : cluster 0 4.9 scrub starts 2024-03-11T21:16:44.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:44 smithi146 bash[23809]: cluster 2024-03-11T21:16:43.992670+0000 osd.1 (osd.1) 345 : cluster 0 4.9 scrub ok 2024-03-11T21:16:44.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:44 smithi023 bash[21372]: cluster 2024-03-11T21:16:42.604048+0000 osd.6 (osd.6) 490 : cluster 0 3.10 scrub starts 2024-03-11T21:16:44.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:44 smithi023 bash[21372]: cluster 2024-03-11T21:16:42.606005+0000 osd.6 (osd.6) 491 : cluster 0 3.10 scrub ok 2024-03-11T21:16:44.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:44 smithi023 bash[21372]: cluster 2024-03-11T21:16:43.016896+0000 osd.5 (osd.5) 571 : cluster 0 2.16 scrub starts 2024-03-11T21:16:44.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:44 smithi023 bash[21372]: cluster 2024-03-11T21:16:43.019364+0000 osd.5 (osd.5) 572 : cluster 0 2.16 scrub ok 2024-03-11T21:16:44.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:44 smithi023 bash[21372]: audit 2024-03-11T21:16:43.711753+0000 mon.a (mon.0) 2012 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:16:44.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:44 smithi023 bash[21372]: audit 2024-03-11T21:16:43.732773+0000 mon.a (mon.0) 2013 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:16:44.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:44 smithi023 bash[21372]: audit 2024-03-11T21:16:43.735401+0000 mon.a (mon.0) 2014 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:16:44.776 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:44 smithi023 bash[21372]: audit 2024-03-11T21:16:43.737066+0000 mon.a (mon.0) 2015 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:16:44.776 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:44 smithi023 bash[21372]: audit 2024-03-11T21:16:43.754718+0000 mon.a (mon.0) 2016 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:16:44.776 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:44 smithi023 bash[21372]: cluster 2024-03-11T21:16:43.989371+0000 osd.1 (osd.1) 344 : cluster 0 4.9 scrub starts 2024-03-11T21:16:44.776 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:44 smithi023 bash[21372]: cluster 2024-03-11T21:16:43.992670+0000 osd.1 (osd.1) 345 : cluster 0 4.9 scrub ok 2024-03-11T21:16:44.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:44 smithi023 bash[29146]: cluster 2024-03-11T21:16:42.604048+0000 osd.6 (osd.6) 490 : cluster 0 3.10 scrub starts 2024-03-11T21:16:44.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:44 smithi023 bash[29146]: cluster 2024-03-11T21:16:42.606005+0000 osd.6 (osd.6) 491 : cluster 0 3.10 scrub ok 2024-03-11T21:16:44.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:44 smithi023 bash[29146]: cluster 2024-03-11T21:16:43.016896+0000 osd.5 (osd.5) 571 : cluster 0 2.16 scrub starts 2024-03-11T21:16:44.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:44 smithi023 bash[29146]: cluster 2024-03-11T21:16:43.019364+0000 osd.5 (osd.5) 572 : cluster 0 2.16 scrub ok 2024-03-11T21:16:44.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:44 smithi023 bash[29146]: audit 2024-03-11T21:16:43.711753+0000 mon.a (mon.0) 2012 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:16:44.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:44 smithi023 bash[29146]: audit 2024-03-11T21:16:43.732773+0000 mon.a (mon.0) 2013 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:16:44.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:44 smithi023 bash[29146]: audit 2024-03-11T21:16:43.735401+0000 mon.a (mon.0) 2014 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:16:44.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:44 smithi023 bash[29146]: audit 2024-03-11T21:16:43.737066+0000 mon.a (mon.0) 2015 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:16:44.777 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:44 smithi023 bash[29146]: audit 2024-03-11T21:16:43.754718+0000 mon.a (mon.0) 2016 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:16:44.777 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:44 smithi023 bash[29146]: cluster 2024-03-11T21:16:43.989371+0000 osd.1 (osd.1) 344 : cluster 0 4.9 scrub starts 2024-03-11T21:16:44.777 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:44 smithi023 bash[29146]: cluster 2024-03-11T21:16:43.992670+0000 osd.1 (osd.1) 345 : cluster 0 4.9 scrub ok 2024-03-11T21:16:45.754 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:45 smithi146 bash[23809]: cluster 2024-03-11T21:16:43.998153+0000 osd.5 (osd.5) 573 : cluster 0 2.3 scrub starts 2024-03-11T21:16:45.754 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:45 smithi146 bash[23809]: cluster 2024-03-11T21:16:44.000258+0000 osd.5 (osd.5) 574 : cluster 0 2.3 scrub ok 2024-03-11T21:16:45.754 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:45 smithi146 bash[23809]: cluster 2024-03-11T21:16:44.119891+0000 osd.7 (osd.7) 601 : cluster 0 3.d scrub starts 2024-03-11T21:16:45.754 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:45 smithi146 bash[23809]: cluster 2024-03-11T21:16:44.125276+0000 osd.7 (osd.7) 602 : cluster 0 3.d scrub ok 2024-03-11T21:16:45.754 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:45 smithi146 bash[23809]: cluster 2024-03-11T21:16:44.490525+0000 mgr.y (mgr.24380) 958 : cluster 0 pgmap v1111: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:16:45.754 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:45 smithi146 bash[23809]: cluster 2024-03-11T21:16:45.017017+0000 osd.1 (osd.1) 346 : cluster 0 3.1a scrub starts 2024-03-11T21:16:45.754 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:45 smithi146 bash[23809]: cluster 2024-03-11T21:16:45.022030+0000 osd.1 (osd.1) 347 : cluster 0 3.1a scrub ok 2024-03-11T21:16:45.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:45 smithi023 bash[21372]: cluster 2024-03-11T21:16:43.998153+0000 osd.5 (osd.5) 573 : cluster 0 2.3 scrub starts 2024-03-11T21:16:45.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:45 smithi023 bash[21372]: cluster 2024-03-11T21:16:44.000258+0000 osd.5 (osd.5) 574 : cluster 0 2.3 scrub ok 2024-03-11T21:16:45.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:45 smithi023 bash[21372]: cluster 2024-03-11T21:16:44.119891+0000 osd.7 (osd.7) 601 : cluster 0 3.d scrub starts 2024-03-11T21:16:45.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:45 smithi023 bash[21372]: cluster 2024-03-11T21:16:44.125276+0000 osd.7 (osd.7) 602 : cluster 0 3.d scrub ok 2024-03-11T21:16:45.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:45 smithi023 bash[21372]: cluster 2024-03-11T21:16:44.490525+0000 mgr.y (mgr.24380) 958 : cluster 0 pgmap v1111: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:16:45.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:45 smithi023 bash[21372]: cluster 2024-03-11T21:16:45.017017+0000 osd.1 (osd.1) 346 : cluster 0 3.1a scrub starts 2024-03-11T21:16:45.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:45 smithi023 bash[21372]: cluster 2024-03-11T21:16:45.022030+0000 osd.1 (osd.1) 347 : cluster 0 3.1a scrub ok 2024-03-11T21:16:45.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:45 smithi023 bash[29146]: cluster 2024-03-11T21:16:43.998153+0000 osd.5 (osd.5) 573 : cluster 0 2.3 scrub starts 2024-03-11T21:16:45.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:45 smithi023 bash[29146]: cluster 2024-03-11T21:16:44.000258+0000 osd.5 (osd.5) 574 : cluster 0 2.3 scrub ok 2024-03-11T21:16:45.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:45 smithi023 bash[29146]: cluster 2024-03-11T21:16:44.119891+0000 osd.7 (osd.7) 601 : cluster 0 3.d scrub starts 2024-03-11T21:16:45.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:45 smithi023 bash[29146]: cluster 2024-03-11T21:16:44.125276+0000 osd.7 (osd.7) 602 : cluster 0 3.d scrub ok 2024-03-11T21:16:45.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:45 smithi023 bash[29146]: cluster 2024-03-11T21:16:44.490525+0000 mgr.y (mgr.24380) 958 : cluster 0 pgmap v1111: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:16:45.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:45 smithi023 bash[29146]: cluster 2024-03-11T21:16:45.017017+0000 osd.1 (osd.1) 346 : cluster 0 3.1a scrub starts 2024-03-11T21:16:45.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:45 smithi023 bash[29146]: cluster 2024-03-11T21:16:45.022030+0000 osd.1 (osd.1) 347 : cluster 0 3.1a scrub ok 2024-03-11T21:16:46.497 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:16:46 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:16:46.754 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:46 smithi146 bash[23809]: cluster 2024-03-11T21:16:44.669715+0000 osd.6 (osd.6) 492 : cluster 0 2.17 scrub starts 2024-03-11T21:16:46.754 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:46 smithi146 bash[23809]: cluster 2024-03-11T21:16:44.672022+0000 osd.6 (osd.6) 493 : cluster 0 2.17 scrub ok 2024-03-11T21:16:46.754 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:46 smithi146 bash[23809]: cluster 2024-03-11T21:16:45.140176+0000 osd.7 (osd.7) 603 : cluster 0 3.14 scrub starts 2024-03-11T21:16:46.754 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:46 smithi146 bash[23809]: cluster 2024-03-11T21:16:45.144301+0000 osd.7 (osd.7) 604 : cluster 0 3.14 scrub ok 2024-03-11T21:16:46.754 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:46 smithi146 bash[23809]: cluster 2024-03-11T21:16:45.601758+0000 osd.3 (osd.3) 530 : cluster 0 4.14 scrub starts 2024-03-11T21:16:46.754 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:46 smithi146 bash[23809]: cluster 2024-03-11T21:16:45.605007+0000 osd.3 (osd.3) 531 : cluster 0 4.14 scrub ok 2024-03-11T21:16:46.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:46 smithi023 bash[29146]: cluster 2024-03-11T21:16:44.669715+0000 osd.6 (osd.6) 492 : cluster 0 2.17 scrub starts 2024-03-11T21:16:46.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:46 smithi023 bash[29146]: cluster 2024-03-11T21:16:44.672022+0000 osd.6 (osd.6) 493 : cluster 0 2.17 scrub ok 2024-03-11T21:16:46.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:46 smithi023 bash[29146]: cluster 2024-03-11T21:16:45.140176+0000 osd.7 (osd.7) 603 : cluster 0 3.14 scrub starts 2024-03-11T21:16:46.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:46 smithi023 bash[29146]: cluster 2024-03-11T21:16:45.144301+0000 osd.7 (osd.7) 604 : cluster 0 3.14 scrub ok 2024-03-11T21:16:46.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:46 smithi023 bash[29146]: cluster 2024-03-11T21:16:45.601758+0000 osd.3 (osd.3) 530 : cluster 0 4.14 scrub starts 2024-03-11T21:16:46.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:46 smithi023 bash[29146]: cluster 2024-03-11T21:16:45.605007+0000 osd.3 (osd.3) 531 : cluster 0 4.14 scrub ok 2024-03-11T21:16:46.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:46 smithi023 bash[21372]: cluster 2024-03-11T21:16:44.669715+0000 osd.6 (osd.6) 492 : cluster 0 2.17 scrub starts 2024-03-11T21:16:46.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:46 smithi023 bash[21372]: cluster 2024-03-11T21:16:44.672022+0000 osd.6 (osd.6) 493 : cluster 0 2.17 scrub ok 2024-03-11T21:16:46.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:46 smithi023 bash[21372]: cluster 2024-03-11T21:16:45.140176+0000 osd.7 (osd.7) 603 : cluster 0 3.14 scrub starts 2024-03-11T21:16:46.776 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:46 smithi023 bash[21372]: cluster 2024-03-11T21:16:45.144301+0000 osd.7 (osd.7) 604 : cluster 0 3.14 scrub ok 2024-03-11T21:16:46.776 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:46 smithi023 bash[21372]: cluster 2024-03-11T21:16:45.601758+0000 osd.3 (osd.3) 530 : cluster 0 4.14 scrub starts 2024-03-11T21:16:46.776 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:46 smithi023 bash[21372]: cluster 2024-03-11T21:16:45.605007+0000 osd.3 (osd.3) 531 : cluster 0 4.14 scrub ok 2024-03-11T21:16:47.190 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:16:48.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:47 smithi146 bash[23809]: audit 2024-03-11T21:16:46.163003+0000 mgr.y (mgr.24380) 959 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:16:48.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:47 smithi146 bash[23809]: cluster 2024-03-11T21:16:46.492119+0000 mgr.y (mgr.24380) 960 : cluster 0 pgmap v1112: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:16:48.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:47 smithi146 bash[23809]: cluster 2024-03-11T21:16:46.577366+0000 osd.3 (osd.3) 532 : cluster 0 5.0 scrub starts 2024-03-11T21:16:48.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:47 smithi146 bash[23809]: cluster 2024-03-11T21:16:46.580894+0000 osd.3 (osd.3) 533 : cluster 0 5.0 scrub ok 2024-03-11T21:16:48.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:47 smithi023 bash[21372]: audit 2024-03-11T21:16:46.163003+0000 mgr.y (mgr.24380) 959 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:16:48.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:47 smithi023 bash[21372]: cluster 2024-03-11T21:16:46.492119+0000 mgr.y (mgr.24380) 960 : cluster 0 pgmap v1112: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:16:48.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:47 smithi023 bash[21372]: cluster 2024-03-11T21:16:46.577366+0000 osd.3 (osd.3) 532 : cluster 0 5.0 scrub starts 2024-03-11T21:16:48.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:47 smithi023 bash[21372]: cluster 2024-03-11T21:16:46.580894+0000 osd.3 (osd.3) 533 : cluster 0 5.0 scrub ok 2024-03-11T21:16:48.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:47 smithi023 bash[29146]: audit 2024-03-11T21:16:46.163003+0000 mgr.y (mgr.24380) 959 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:16:48.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:47 smithi023 bash[29146]: cluster 2024-03-11T21:16:46.492119+0000 mgr.y (mgr.24380) 960 : cluster 0 pgmap v1112: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:16:48.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:47 smithi023 bash[29146]: cluster 2024-03-11T21:16:46.577366+0000 osd.3 (osd.3) 532 : cluster 0 5.0 scrub starts 2024-03-11T21:16:48.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:47 smithi023 bash[29146]: cluster 2024-03-11T21:16:46.580894+0000 osd.3 (osd.3) 533 : cluster 0 5.0 scrub ok 2024-03-11T21:16:48.614 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:16:48 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:16:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:16:48.614 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:48 smithi023 bash[21372]: cluster 2024-03-11T21:16:47.984928+0000 osd.1 (osd.1) 348 : cluster 0 4.7 deep-scrub starts 2024-03-11T21:16:48.614 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:48 smithi023 bash[21372]: cluster 2024-03-11T21:16:47.986859+0000 osd.1 (osd.1) 349 : cluster 0 4.7 deep-scrub ok 2024-03-11T21:16:48.614 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:48 smithi023 bash[29146]: cluster 2024-03-11T21:16:47.984928+0000 osd.1 (osd.1) 348 : cluster 0 4.7 deep-scrub starts 2024-03-11T21:16:48.614 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:48 smithi023 bash[29146]: cluster 2024-03-11T21:16:47.986859+0000 osd.1 (osd.1) 349 : cluster 0 4.7 deep-scrub ok 2024-03-11T21:16:49.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:48 smithi146 bash[23809]: cluster 2024-03-11T21:16:47.984928+0000 osd.1 (osd.1) 348 : cluster 0 4.7 deep-scrub starts 2024-03-11T21:16:49.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:48 smithi146 bash[23809]: cluster 2024-03-11T21:16:47.986859+0000 osd.1 (osd.1) 349 : cluster 0 4.7 deep-scrub ok 2024-03-11T21:16:49.178 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:16:49.966 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:49 smithi023 bash[21372]: cluster 2024-03-11T21:16:48.493095+0000 mgr.y (mgr.24380) 961 : cluster 0 pgmap v1113: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:16:49.967 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:49 smithi023 bash[29146]: cluster 2024-03-11T21:16:48.493095+0000 mgr.y (mgr.24380) 961 : cluster 0 pgmap v1113: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:16:50.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:49 smithi146 bash[23809]: cluster 2024-03-11T21:16:48.493095+0000 mgr.y (mgr.24380) 961 : cluster 0 pgmap v1113: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:16:50.577 INFO:teuthology.orchestra.run.smithi023.stderr:reweighted osd.0 to 1 (10000) 2024-03-11T21:16:51.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:50 smithi146 bash[23809]: cluster 2024-03-11T21:16:48.927545+0000 osd.5 (osd.5) 575 : cluster 0 4.1 scrub starts 2024-03-11T21:16:51.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:50 smithi146 bash[23809]: cluster 2024-03-11T21:16:48.930075+0000 osd.5 (osd.5) 576 : cluster 0 4.1 scrub ok 2024-03-11T21:16:51.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:50 smithi146 bash[23809]: cluster 2024-03-11T21:16:49.194446+0000 osd.7 (osd.7) 605 : cluster 0 1.9 scrub starts 2024-03-11T21:16:51.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:50 smithi146 bash[23809]: cluster 2024-03-11T21:16:49.196710+0000 osd.7 (osd.7) 606 : cluster 0 1.9 scrub ok 2024-03-11T21:16:51.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:50 smithi146 bash[23809]: cluster 2024-03-11T21:16:50.050482+0000 osd.1 (osd.1) 350 : cluster 0 2.0 scrub starts 2024-03-11T21:16:51.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:50 smithi146 bash[23809]: cluster 2024-03-11T21:16:50.053657+0000 osd.1 (osd.1) 351 : cluster 0 2.0 scrub ok 2024-03-11T21:16:51.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:50 smithi146 bash[23809]: audit 2024-03-11T21:16:50.358165+0000 mon.c (mon.2) 106 : audit 1 from='client.? 172.21.15.23:0/1638152861' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-11T21:16:51.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:50 smithi146 bash[23809]: audit 2024-03-11T21:16:50.358693+0000 mon.a (mon.0) 2017 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-11T21:16:51.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:50 smithi023 bash[29146]: cluster 2024-03-11T21:16:48.927545+0000 osd.5 (osd.5) 575 : cluster 0 4.1 scrub starts 2024-03-11T21:16:51.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:50 smithi023 bash[29146]: cluster 2024-03-11T21:16:48.930075+0000 osd.5 (osd.5) 576 : cluster 0 4.1 scrub ok 2024-03-11T21:16:51.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:50 smithi023 bash[29146]: cluster 2024-03-11T21:16:49.194446+0000 osd.7 (osd.7) 605 : cluster 0 1.9 scrub starts 2024-03-11T21:16:51.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:50 smithi023 bash[29146]: cluster 2024-03-11T21:16:49.196710+0000 osd.7 (osd.7) 606 : cluster 0 1.9 scrub ok 2024-03-11T21:16:51.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:50 smithi023 bash[29146]: cluster 2024-03-11T21:16:50.050482+0000 osd.1 (osd.1) 350 : cluster 0 2.0 scrub starts 2024-03-11T21:16:51.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:50 smithi023 bash[29146]: cluster 2024-03-11T21:16:50.053657+0000 osd.1 (osd.1) 351 : cluster 0 2.0 scrub ok 2024-03-11T21:16:51.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:50 smithi023 bash[29146]: audit 2024-03-11T21:16:50.358165+0000 mon.c (mon.2) 106 : audit 1 from='client.? 172.21.15.23:0/1638152861' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-11T21:16:51.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:50 smithi023 bash[29146]: audit 2024-03-11T21:16:50.358693+0000 mon.a (mon.0) 2017 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-11T21:16:51.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:50 smithi023 bash[21372]: cluster 2024-03-11T21:16:48.927545+0000 osd.5 (osd.5) 575 : cluster 0 4.1 scrub starts 2024-03-11T21:16:51.026 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:50 smithi023 bash[21372]: cluster 2024-03-11T21:16:48.930075+0000 osd.5 (osd.5) 576 : cluster 0 4.1 scrub ok 2024-03-11T21:16:51.026 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:50 smithi023 bash[21372]: cluster 2024-03-11T21:16:49.194446+0000 osd.7 (osd.7) 605 : cluster 0 1.9 scrub starts 2024-03-11T21:16:51.026 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:50 smithi023 bash[21372]: cluster 2024-03-11T21:16:49.196710+0000 osd.7 (osd.7) 606 : cluster 0 1.9 scrub ok 2024-03-11T21:16:51.026 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:50 smithi023 bash[21372]: cluster 2024-03-11T21:16:50.050482+0000 osd.1 (osd.1) 350 : cluster 0 2.0 scrub starts 2024-03-11T21:16:51.026 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:50 smithi023 bash[21372]: cluster 2024-03-11T21:16:50.053657+0000 osd.1 (osd.1) 351 : cluster 0 2.0 scrub ok 2024-03-11T21:16:51.026 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:50 smithi023 bash[21372]: audit 2024-03-11T21:16:50.358165+0000 mon.c (mon.2) 106 : audit 1 from='client.? 172.21.15.23:0/1638152861' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-11T21:16:51.026 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:50 smithi023 bash[21372]: audit 2024-03-11T21:16:50.358693+0000 mon.a (mon.0) 2017 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-11T21:16:51.354 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd primary-affinity 0 1 2024-03-11T21:16:51.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:51 smithi023 bash[21372]: cluster 2024-03-11T21:16:50.494001+0000 mgr.y (mgr.24380) 962 : cluster 0 pgmap v1114: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:16:51.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:51 smithi023 bash[21372]: audit 2024-03-11T21:16:50.571390+0000 mon.a (mon.0) 2018 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-11T21:16:51.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:51 smithi023 bash[21372]: cluster 2024-03-11T21:16:50.579933+0000 mon.a (mon.0) 2019 : cluster 0 osdmap e455: 8 total, 8 up, 8 in 2024-03-11T21:16:51.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:51 smithi023 bash[29146]: cluster 2024-03-11T21:16:50.494001+0000 mgr.y (mgr.24380) 962 : cluster 0 pgmap v1114: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:16:51.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:51 smithi023 bash[29146]: audit 2024-03-11T21:16:50.571390+0000 mon.a (mon.0) 2018 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-11T21:16:51.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:51 smithi023 bash[29146]: cluster 2024-03-11T21:16:50.579933+0000 mon.a (mon.0) 2019 : cluster 0 osdmap e455: 8 total, 8 up, 8 in 2024-03-11T21:16:52.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:51 smithi146 bash[23809]: cluster 2024-03-11T21:16:50.494001+0000 mgr.y (mgr.24380) 962 : cluster 0 pgmap v1114: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:16:52.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:51 smithi146 bash[23809]: audit 2024-03-11T21:16:50.571390+0000 mon.a (mon.0) 2018 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-11T21:16:52.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:51 smithi146 bash[23809]: cluster 2024-03-11T21:16:50.579933+0000 mon.a (mon.0) 2019 : cluster 0 osdmap e455: 8 total, 8 up, 8 in 2024-03-11T21:16:52.232 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-03-11T21:16:52.233 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:16:52.986 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:52 smithi146 bash[23809]: cluster 2024-03-11T21:16:51.641376+0000 osd.3 (osd.3) 534 : cluster 0 4.1b deep-scrub starts 2024-03-11T21:16:52.987 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:52 smithi146 bash[23809]: cluster 2024-03-11T21:16:51.642945+0000 osd.3 (osd.3) 535 : cluster 0 4.1b deep-scrub ok 2024-03-11T21:16:52.987 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:52 smithi146 bash[23809]: audit 2024-03-11T21:16:51.751997+0000 mon.a (mon.0) 2020 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:16:52.987 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:52 smithi146 bash[23809]: cluster 2024-03-11T21:16:52.061843+0000 osd.1 (osd.1) 352 : cluster 0 2.4 scrub starts 2024-03-11T21:16:52.987 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:52 smithi146 bash[23809]: cluster 2024-03-11T21:16:52.064135+0000 osd.1 (osd.1) 353 : cluster 0 2.4 scrub ok 2024-03-11T21:16:52.988 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-11T21:16:53.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:52 smithi023 bash[21372]: cluster 2024-03-11T21:16:51.641376+0000 osd.3 (osd.3) 534 : cluster 0 4.1b deep-scrub starts 2024-03-11T21:16:53.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:52 smithi023 bash[21372]: cluster 2024-03-11T21:16:51.642945+0000 osd.3 (osd.3) 535 : cluster 0 4.1b deep-scrub ok 2024-03-11T21:16:53.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:52 smithi023 bash[21372]: audit 2024-03-11T21:16:51.751997+0000 mon.a (mon.0) 2020 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:16:53.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:52 smithi023 bash[21372]: cluster 2024-03-11T21:16:52.061843+0000 osd.1 (osd.1) 352 : cluster 0 2.4 scrub starts 2024-03-11T21:16:53.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:52 smithi023 bash[21372]: cluster 2024-03-11T21:16:52.064135+0000 osd.1 (osd.1) 353 : cluster 0 2.4 scrub ok 2024-03-11T21:16:53.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:52 smithi023 bash[29146]: cluster 2024-03-11T21:16:51.641376+0000 osd.3 (osd.3) 534 : cluster 0 4.1b deep-scrub starts 2024-03-11T21:16:53.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:52 smithi023 bash[29146]: cluster 2024-03-11T21:16:51.642945+0000 osd.3 (osd.3) 535 : cluster 0 4.1b deep-scrub ok 2024-03-11T21:16:53.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:52 smithi023 bash[29146]: audit 2024-03-11T21:16:51.751997+0000 mon.a (mon.0) 2020 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:16:53.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:52 smithi023 bash[29146]: cluster 2024-03-11T21:16:52.061843+0000 osd.1 (osd.1) 352 : cluster 0 2.4 scrub starts 2024-03-11T21:16:53.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:52 smithi023 bash[29146]: cluster 2024-03-11T21:16:52.064135+0000 osd.1 (osd.1) 353 : cluster 0 2.4 scrub ok 2024-03-11T21:16:54.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:53 smithi146 bash[23809]: cluster 2024-03-11T21:16:52.495342+0000 mgr.y (mgr.24380) 963 : cluster 0 pgmap v1116: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:16:54.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:53 smithi146 bash[23809]: cluster 2024-03-11T21:16:52.600073+0000 osd.3 (osd.3) 536 : cluster 0 4.17 scrub starts 2024-03-11T21:16:54.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:53 smithi146 bash[23809]: cluster 2024-03-11T21:16:52.602149+0000 osd.3 (osd.3) 537 : cluster 0 4.17 scrub ok 2024-03-11T21:16:54.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:53 smithi146 bash[23809]: cluster 2024-03-11T21:16:53.029894+0000 osd.5 (osd.5) 577 : cluster 0 6.17 scrub starts 2024-03-11T21:16:54.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:53 smithi146 bash[23809]: cluster 2024-03-11T21:16:53.031588+0000 osd.1 (osd.1) 354 : cluster 0 4.12 scrub starts 2024-03-11T21:16:54.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:53 smithi146 bash[23809]: cluster 2024-03-11T21:16:53.033956+0000 osd.1 (osd.1) 355 : cluster 0 4.12 scrub ok 2024-03-11T21:16:54.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:53 smithi146 bash[23809]: cluster 2024-03-11T21:16:53.220976+0000 osd.5 (osd.5) 578 : cluster 0 6.17 scrub ok 2024-03-11T21:16:54.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:53 smithi023 bash[21372]: cluster 2024-03-11T21:16:52.495342+0000 mgr.y (mgr.24380) 963 : cluster 0 pgmap v1116: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:16:54.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:53 smithi023 bash[21372]: cluster 2024-03-11T21:16:52.600073+0000 osd.3 (osd.3) 536 : cluster 0 4.17 scrub starts 2024-03-11T21:16:54.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:53 smithi023 bash[21372]: cluster 2024-03-11T21:16:52.602149+0000 osd.3 (osd.3) 537 : cluster 0 4.17 scrub ok 2024-03-11T21:16:54.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:53 smithi023 bash[21372]: cluster 2024-03-11T21:16:53.029894+0000 osd.5 (osd.5) 577 : cluster 0 6.17 scrub starts 2024-03-11T21:16:54.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:53 smithi023 bash[21372]: cluster 2024-03-11T21:16:53.031588+0000 osd.1 (osd.1) 354 : cluster 0 4.12 scrub starts 2024-03-11T21:16:54.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:53 smithi023 bash[21372]: cluster 2024-03-11T21:16:53.033956+0000 osd.1 (osd.1) 355 : cluster 0 4.12 scrub ok 2024-03-11T21:16:54.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:53 smithi023 bash[21372]: cluster 2024-03-11T21:16:53.220976+0000 osd.5 (osd.5) 578 : cluster 0 6.17 scrub ok 2024-03-11T21:16:54.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:53 smithi023 bash[29146]: cluster 2024-03-11T21:16:52.495342+0000 mgr.y (mgr.24380) 963 : cluster 0 pgmap v1116: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:16:54.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:53 smithi023 bash[29146]: cluster 2024-03-11T21:16:52.600073+0000 osd.3 (osd.3) 536 : cluster 0 4.17 scrub starts 2024-03-11T21:16:54.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:53 smithi023 bash[29146]: cluster 2024-03-11T21:16:52.602149+0000 osd.3 (osd.3) 537 : cluster 0 4.17 scrub ok 2024-03-11T21:16:54.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:53 smithi023 bash[29146]: cluster 2024-03-11T21:16:53.029894+0000 osd.5 (osd.5) 577 : cluster 0 6.17 scrub starts 2024-03-11T21:16:54.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:53 smithi023 bash[29146]: cluster 2024-03-11T21:16:53.031588+0000 osd.1 (osd.1) 354 : cluster 0 4.12 scrub starts 2024-03-11T21:16:54.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:53 smithi023 bash[29146]: cluster 2024-03-11T21:16:53.033956+0000 osd.1 (osd.1) 355 : cluster 0 4.12 scrub ok 2024-03-11T21:16:54.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:53 smithi023 bash[29146]: cluster 2024-03-11T21:16:53.220976+0000 osd.5 (osd.5) 578 : cluster 0 6.17 scrub ok 2024-03-11T21:16:55.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:54 smithi146 bash[23809]: cluster 2024-03-11T21:16:54.054005+0000 osd.5 (osd.5) 579 : cluster 0 3.16 scrub starts 2024-03-11T21:16:55.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:54 smithi146 bash[23809]: cluster 2024-03-11T21:16:54.057921+0000 osd.5 (osd.5) 580 : cluster 0 3.16 scrub ok 2024-03-11T21:16:55.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:54 smithi146 bash[23809]: cluster 2024-03-11T21:16:54.059132+0000 osd.1 (osd.1) 356 : cluster 0 3.19 scrub starts 2024-03-11T21:16:55.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:54 smithi146 bash[23809]: cluster 2024-03-11T21:16:54.063945+0000 osd.1 (osd.1) 357 : cluster 0 3.19 scrub ok 2024-03-11T21:16:55.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:54 smithi023 bash[21372]: cluster 2024-03-11T21:16:54.054005+0000 osd.5 (osd.5) 579 : cluster 0 3.16 scrub starts 2024-03-11T21:16:55.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:54 smithi023 bash[21372]: cluster 2024-03-11T21:16:54.057921+0000 osd.5 (osd.5) 580 : cluster 0 3.16 scrub ok 2024-03-11T21:16:55.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:54 smithi023 bash[21372]: cluster 2024-03-11T21:16:54.059132+0000 osd.1 (osd.1) 356 : cluster 0 3.19 scrub starts 2024-03-11T21:16:55.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:54 smithi023 bash[21372]: cluster 2024-03-11T21:16:54.063945+0000 osd.1 (osd.1) 357 : cluster 0 3.19 scrub ok 2024-03-11T21:16:55.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:54 smithi023 bash[29146]: cluster 2024-03-11T21:16:54.054005+0000 osd.5 (osd.5) 579 : cluster 0 3.16 scrub starts 2024-03-11T21:16:55.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:54 smithi023 bash[29146]: cluster 2024-03-11T21:16:54.057921+0000 osd.5 (osd.5) 580 : cluster 0 3.16 scrub ok 2024-03-11T21:16:55.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:54 smithi023 bash[29146]: cluster 2024-03-11T21:16:54.059132+0000 osd.1 (osd.1) 356 : cluster 0 3.19 scrub starts 2024-03-11T21:16:55.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:54 smithi023 bash[29146]: cluster 2024-03-11T21:16:54.063945+0000 osd.1 (osd.1) 357 : cluster 0 3.19 scrub ok 2024-03-11T21:16:56.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:55 smithi146 bash[23809]: cluster 2024-03-11T21:16:54.222420+0000 osd.7 (osd.7) 607 : cluster 0 2.5 scrub starts 2024-03-11T21:16:56.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:55 smithi146 bash[23809]: cluster 2024-03-11T21:16:54.225258+0000 osd.7 (osd.7) 608 : cluster 0 2.5 scrub ok 2024-03-11T21:16:56.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:55 smithi146 bash[23809]: cluster 2024-03-11T21:16:54.496370+0000 mgr.y (mgr.24380) 964 : cluster 0 pgmap v1117: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:16:56.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:55 smithi146 bash[23809]: cluster 2024-03-11T21:16:54.548443+0000 osd.3 (osd.3) 538 : cluster 0 3.18 scrub starts 2024-03-11T21:16:56.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:55 smithi146 bash[23809]: cluster 2024-03-11T21:16:54.552652+0000 osd.3 (osd.3) 539 : cluster 0 3.18 scrub ok 2024-03-11T21:16:56.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:55 smithi146 bash[23809]: cluster 2024-03-11T21:16:55.038758+0000 osd.5 (osd.5) 581 : cluster 0 2.18 scrub starts 2024-03-11T21:16:56.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:55 smithi146 bash[23809]: cluster 2024-03-11T21:16:55.041249+0000 osd.5 (osd.5) 582 : cluster 0 2.18 scrub ok 2024-03-11T21:16:56.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:55 smithi146 bash[23809]: cluster 2024-03-11T21:16:55.459198+0000 mon.a (mon.0) 2021 : cluster 0 osdmap e456: 8 total, 8 up, 8 in 2024-03-11T21:16:56.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:55 smithi023 bash[29146]: cluster 2024-03-11T21:16:54.222420+0000 osd.7 (osd.7) 607 : cluster 0 2.5 scrub starts 2024-03-11T21:16:56.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:55 smithi023 bash[29146]: cluster 2024-03-11T21:16:54.225258+0000 osd.7 (osd.7) 608 : cluster 0 2.5 scrub ok 2024-03-11T21:16:56.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:55 smithi023 bash[29146]: cluster 2024-03-11T21:16:54.496370+0000 mgr.y (mgr.24380) 964 : cluster 0 pgmap v1117: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:16:56.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:55 smithi023 bash[29146]: cluster 2024-03-11T21:16:54.548443+0000 osd.3 (osd.3) 538 : cluster 0 3.18 scrub starts 2024-03-11T21:16:56.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:55 smithi023 bash[29146]: cluster 2024-03-11T21:16:54.552652+0000 osd.3 (osd.3) 539 : cluster 0 3.18 scrub ok 2024-03-11T21:16:56.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:55 smithi023 bash[29146]: cluster 2024-03-11T21:16:55.038758+0000 osd.5 (osd.5) 581 : cluster 0 2.18 scrub starts 2024-03-11T21:16:56.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:55 smithi023 bash[29146]: cluster 2024-03-11T21:16:55.041249+0000 osd.5 (osd.5) 582 : cluster 0 2.18 scrub ok 2024-03-11T21:16:56.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:55 smithi023 bash[29146]: cluster 2024-03-11T21:16:55.459198+0000 mon.a (mon.0) 2021 : cluster 0 osdmap e456: 8 total, 8 up, 8 in 2024-03-11T21:16:56.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:55 smithi023 bash[21372]: cluster 2024-03-11T21:16:54.222420+0000 osd.7 (osd.7) 607 : cluster 0 2.5 scrub starts 2024-03-11T21:16:56.026 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:55 smithi023 bash[21372]: cluster 2024-03-11T21:16:54.225258+0000 osd.7 (osd.7) 608 : cluster 0 2.5 scrub ok 2024-03-11T21:16:56.026 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:55 smithi023 bash[21372]: cluster 2024-03-11T21:16:54.496370+0000 mgr.y (mgr.24380) 964 : cluster 0 pgmap v1117: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:16:56.026 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:55 smithi023 bash[21372]: cluster 2024-03-11T21:16:54.548443+0000 osd.3 (osd.3) 538 : cluster 0 3.18 scrub starts 2024-03-11T21:16:56.026 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:55 smithi023 bash[21372]: cluster 2024-03-11T21:16:54.552652+0000 osd.3 (osd.3) 539 : cluster 0 3.18 scrub ok 2024-03-11T21:16:56.026 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:55 smithi023 bash[21372]: cluster 2024-03-11T21:16:55.038758+0000 osd.5 (osd.5) 581 : cluster 0 2.18 scrub starts 2024-03-11T21:16:56.026 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:55 smithi023 bash[21372]: cluster 2024-03-11T21:16:55.041249+0000 osd.5 (osd.5) 582 : cluster 0 2.18 scrub ok 2024-03-11T21:16:56.026 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:55 smithi023 bash[21372]: cluster 2024-03-11T21:16:55.459198+0000 mon.a (mon.0) 2021 : cluster 0 osdmap e456: 8 total, 8 up, 8 in 2024-03-11T21:16:56.144 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:16:56.503 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:16:56 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:16:57.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:56 smithi146 bash[23809]: cluster 2024-03-11T21:16:55.180519+0000 osd.7 (osd.7) 609 : cluster 0 2.1c scrub starts 2024-03-11T21:16:57.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:56 smithi146 bash[23809]: cluster 2024-03-11T21:16:55.182988+0000 osd.7 (osd.7) 610 : cluster 0 2.1c scrub ok 2024-03-11T21:16:57.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:56 smithi146 bash[23809]: cluster 2024-03-11T21:16:55.557896+0000 osd.3 (osd.3) 540 : cluster 0 4.1d scrub starts 2024-03-11T21:16:57.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:56 smithi146 bash[23809]: cluster 2024-03-11T21:16:55.563156+0000 osd.3 (osd.3) 541 : cluster 0 4.1d scrub ok 2024-03-11T21:16:57.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:56 smithi146 bash[23809]: cluster 2024-03-11T21:16:56.084904+0000 osd.5 (osd.5) 583 : cluster 0 6.a scrub starts 2024-03-11T21:16:57.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:56 smithi146 bash[23809]: cluster 2024-03-11T21:16:56.132434+0000 osd.5 (osd.5) 584 : cluster 0 6.a scrub ok 2024-03-11T21:16:57.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:56 smithi023 bash[29146]: cluster 2024-03-11T21:16:55.180519+0000 osd.7 (osd.7) 609 : cluster 0 2.1c scrub starts 2024-03-11T21:16:57.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:56 smithi023 bash[29146]: cluster 2024-03-11T21:16:55.182988+0000 osd.7 (osd.7) 610 : cluster 0 2.1c scrub ok 2024-03-11T21:16:57.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:56 smithi023 bash[29146]: cluster 2024-03-11T21:16:55.557896+0000 osd.3 (osd.3) 540 : cluster 0 4.1d scrub starts 2024-03-11T21:16:57.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:56 smithi023 bash[29146]: cluster 2024-03-11T21:16:55.563156+0000 osd.3 (osd.3) 541 : cluster 0 4.1d scrub ok 2024-03-11T21:16:57.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:56 smithi023 bash[29146]: cluster 2024-03-11T21:16:56.084904+0000 osd.5 (osd.5) 583 : cluster 0 6.a scrub starts 2024-03-11T21:16:57.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:56 smithi023 bash[29146]: cluster 2024-03-11T21:16:56.132434+0000 osd.5 (osd.5) 584 : cluster 0 6.a scrub ok 2024-03-11T21:16:57.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:56 smithi023 bash[21372]: cluster 2024-03-11T21:16:55.180519+0000 osd.7 (osd.7) 609 : cluster 0 2.1c scrub starts 2024-03-11T21:16:57.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:56 smithi023 bash[21372]: cluster 2024-03-11T21:16:55.182988+0000 osd.7 (osd.7) 610 : cluster 0 2.1c scrub ok 2024-03-11T21:16:57.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:56 smithi023 bash[21372]: cluster 2024-03-11T21:16:55.557896+0000 osd.3 (osd.3) 540 : cluster 0 4.1d scrub starts 2024-03-11T21:16:57.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:56 smithi023 bash[21372]: cluster 2024-03-11T21:16:55.563156+0000 osd.3 (osd.3) 541 : cluster 0 4.1d scrub ok 2024-03-11T21:16:57.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:56 smithi023 bash[21372]: cluster 2024-03-11T21:16:56.084904+0000 osd.5 (osd.5) 583 : cluster 0 6.a scrub starts 2024-03-11T21:16:57.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:56 smithi023 bash[21372]: cluster 2024-03-11T21:16:56.132434+0000 osd.5 (osd.5) 584 : cluster 0 6.a scrub ok 2024-03-11T21:16:57.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:57 smithi023 bash[21372]: audit 2024-03-11T21:16:56.173189+0000 mgr.y (mgr.24380) 965 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:16:57.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:57 smithi023 bash[21372]: cluster 2024-03-11T21:16:56.198192+0000 osd.7 (osd.7) 611 : cluster 0 3.e scrub starts 2024-03-11T21:16:57.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:57 smithi023 bash[21372]: cluster 2024-03-11T21:16:56.203929+0000 osd.7 (osd.7) 612 : cluster 0 3.e scrub ok 2024-03-11T21:16:57.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:57 smithi023 bash[21372]: cluster 2024-03-11T21:16:56.497793+0000 mgr.y (mgr.24380) 966 : cluster 0 pgmap v1119: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:16:57.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:57 smithi023 bash[21372]: cluster 2024-03-11T21:16:57.127677+0000 osd.5 (osd.5) 585 : cluster 0 6.d scrub starts 2024-03-11T21:16:57.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:57 smithi023 bash[21372]: cluster 2024-03-11T21:16:57.310893+0000 osd.5 (osd.5) 586 : cluster 0 6.d scrub ok 2024-03-11T21:16:57.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:57 smithi023 bash[29146]: audit 2024-03-11T21:16:56.173189+0000 mgr.y (mgr.24380) 965 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:16:57.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:57 smithi023 bash[29146]: cluster 2024-03-11T21:16:56.198192+0000 osd.7 (osd.7) 611 : cluster 0 3.e scrub starts 2024-03-11T21:16:57.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:57 smithi023 bash[29146]: cluster 2024-03-11T21:16:56.203929+0000 osd.7 (osd.7) 612 : cluster 0 3.e scrub ok 2024-03-11T21:16:57.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:57 smithi023 bash[29146]: cluster 2024-03-11T21:16:56.497793+0000 mgr.y (mgr.24380) 966 : cluster 0 pgmap v1119: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:16:57.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:57 smithi023 bash[29146]: cluster 2024-03-11T21:16:57.127677+0000 osd.5 (osd.5) 585 : cluster 0 6.d scrub starts 2024-03-11T21:16:57.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:57 smithi023 bash[29146]: cluster 2024-03-11T21:16:57.310893+0000 osd.5 (osd.5) 586 : cluster 0 6.d scrub ok 2024-03-11T21:16:57.812 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:16:58.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:57 smithi146 bash[23809]: audit 2024-03-11T21:16:56.173189+0000 mgr.y (mgr.24380) 965 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:16:58.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:57 smithi146 bash[23809]: cluster 2024-03-11T21:16:56.198192+0000 osd.7 (osd.7) 611 : cluster 0 3.e scrub starts 2024-03-11T21:16:58.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:57 smithi146 bash[23809]: cluster 2024-03-11T21:16:56.203929+0000 osd.7 (osd.7) 612 : cluster 0 3.e scrub ok 2024-03-11T21:16:58.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:57 smithi146 bash[23809]: cluster 2024-03-11T21:16:56.497793+0000 mgr.y (mgr.24380) 966 : cluster 0 pgmap v1119: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:16:58.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:57 smithi146 bash[23809]: cluster 2024-03-11T21:16:57.127677+0000 osd.5 (osd.5) 585 : cluster 0 6.d scrub starts 2024-03-11T21:16:58.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:57 smithi146 bash[23809]: cluster 2024-03-11T21:16:57.310893+0000 osd.5 (osd.5) 586 : cluster 0 6.d scrub ok 2024-03-11T21:16:58.524 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:16:58 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:16:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:16:58.899 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:58 smithi146 bash[23809]: cluster 2024-03-11T21:16:57.244571+0000 osd.7 (osd.7) 613 : cluster 0 3.11 scrub starts 2024-03-11T21:16:58.899 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:58 smithi146 bash[23809]: cluster 2024-03-11T21:16:57.248982+0000 osd.7 (osd.7) 614 : cluster 0 3.11 scrub ok 2024-03-11T21:16:59.007 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:58 smithi023 bash[21372]: cluster 2024-03-11T21:16:57.244571+0000 osd.7 (osd.7) 613 : cluster 0 3.11 scrub starts 2024-03-11T21:16:59.007 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:58 smithi023 bash[21372]: cluster 2024-03-11T21:16:57.248982+0000 osd.7 (osd.7) 614 : cluster 0 3.11 scrub ok 2024-03-11T21:16:59.007 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:58 smithi023 bash[29146]: cluster 2024-03-11T21:16:57.244571+0000 osd.7 (osd.7) 613 : cluster 0 3.11 scrub starts 2024-03-11T21:16:59.007 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:58 smithi023 bash[29146]: cluster 2024-03-11T21:16:57.248982+0000 osd.7 (osd.7) 614 : cluster 0 3.11 scrub ok 2024-03-11T21:16:59.666 INFO:teuthology.orchestra.run.smithi023.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-03-11T21:17:00.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:59 smithi146 bash[23809]: cluster 2024-03-11T21:16:58.227259+0000 osd.7 (osd.7) 615 : cluster 0 2.1d deep-scrub starts 2024-03-11T21:17:00.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:59 smithi146 bash[23809]: cluster 2024-03-11T21:16:58.229330+0000 osd.7 (osd.7) 616 : cluster 0 2.1d deep-scrub ok 2024-03-11T21:17:00.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:59 smithi146 bash[23809]: cluster 2024-03-11T21:16:58.498749+0000 mgr.y (mgr.24380) 967 : cluster 0 pgmap v1120: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:17:00.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:59 smithi146 bash[23809]: cluster 2024-03-11T21:16:59.050468+0000 osd.1 (osd.1) 358 : cluster 0 4.2 deep-scrub starts 2024-03-11T21:17:00.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:59 smithi146 bash[23809]: cluster 2024-03-11T21:16:59.052083+0000 osd.1 (osd.1) 359 : cluster 0 4.2 deep-scrub ok 2024-03-11T21:17:00.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:59 smithi146 bash[23809]: audit 2024-03-11T21:16:59.442461+0000 mon.c (mon.2) 107 : audit 1 from='client.? 172.21.15.23:0/3331819923' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-03-11T21:17:00.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:16:59 smithi146 bash[23809]: audit 2024-03-11T21:16:59.442935+0000 mon.a (mon.0) 2022 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-03-11T21:17:00.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:59 smithi023 bash[21372]: cluster 2024-03-11T21:16:58.227259+0000 osd.7 (osd.7) 615 : cluster 0 2.1d deep-scrub starts 2024-03-11T21:17:00.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:59 smithi023 bash[21372]: cluster 2024-03-11T21:16:58.229330+0000 osd.7 (osd.7) 616 : cluster 0 2.1d deep-scrub ok 2024-03-11T21:17:00.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:59 smithi023 bash[21372]: cluster 2024-03-11T21:16:58.498749+0000 mgr.y (mgr.24380) 967 : cluster 0 pgmap v1120: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:17:00.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:59 smithi023 bash[21372]: cluster 2024-03-11T21:16:59.050468+0000 osd.1 (osd.1) 358 : cluster 0 4.2 deep-scrub starts 2024-03-11T21:17:00.026 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:59 smithi023 bash[21372]: cluster 2024-03-11T21:16:59.052083+0000 osd.1 (osd.1) 359 : cluster 0 4.2 deep-scrub ok 2024-03-11T21:17:00.026 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:59 smithi023 bash[21372]: audit 2024-03-11T21:16:59.442461+0000 mon.c (mon.2) 107 : audit 1 from='client.? 172.21.15.23:0/3331819923' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-03-11T21:17:00.026 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:16:59 smithi023 bash[21372]: audit 2024-03-11T21:16:59.442935+0000 mon.a (mon.0) 2022 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-03-11T21:17:00.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:59 smithi023 bash[29146]: cluster 2024-03-11T21:16:58.227259+0000 osd.7 (osd.7) 615 : cluster 0 2.1d deep-scrub starts 2024-03-11T21:17:00.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:59 smithi023 bash[29146]: cluster 2024-03-11T21:16:58.229330+0000 osd.7 (osd.7) 616 : cluster 0 2.1d deep-scrub ok 2024-03-11T21:17:00.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:59 smithi023 bash[29146]: cluster 2024-03-11T21:16:58.498749+0000 mgr.y (mgr.24380) 967 : cluster 0 pgmap v1120: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:17:00.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:59 smithi023 bash[29146]: cluster 2024-03-11T21:16:59.050468+0000 osd.1 (osd.1) 358 : cluster 0 4.2 deep-scrub starts 2024-03-11T21:17:00.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:59 smithi023 bash[29146]: cluster 2024-03-11T21:16:59.052083+0000 osd.1 (osd.1) 359 : cluster 0 4.2 deep-scrub ok 2024-03-11T21:17:00.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:59 smithi023 bash[29146]: audit 2024-03-11T21:16:59.442461+0000 mon.c (mon.2) 107 : audit 1 from='client.? 172.21.15.23:0/3331819923' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-03-11T21:17:00.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:16:59 smithi023 bash[29146]: audit 2024-03-11T21:16:59.442935+0000 mon.a (mon.0) 2022 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-03-11T21:17:00.437 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd reweight 2 1 2024-03-11T21:17:00.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:00 smithi023 bash[21372]: cluster 2024-03-11T21:16:59.212855+0000 osd.7 (osd.7) 617 : cluster 0 6.8 scrub starts 2024-03-11T21:17:00.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:00 smithi023 bash[21372]: cluster 2024-03-11T21:16:59.214981+0000 osd.7 (osd.7) 618 : cluster 0 6.8 scrub ok 2024-03-11T21:17:00.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:00 smithi023 bash[21372]: audit 2024-03-11T21:16:59.657876+0000 mon.a (mon.0) 2023 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-03-11T21:17:00.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:00 smithi023 bash[21372]: cluster 2024-03-11T21:16:59.668716+0000 mon.a (mon.0) 2024 : cluster 0 osdmap e457: 8 total, 8 up, 8 in 2024-03-11T21:17:00.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:00 smithi023 bash[21372]: cluster 2024-03-11T21:17:00.168166+0000 osd.5 (osd.5) 587 : cluster 0 4.8 scrub starts 2024-03-11T21:17:00.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:00 smithi023 bash[21372]: cluster 2024-03-11T21:17:00.171974+0000 osd.5 (osd.5) 588 : cluster 0 4.8 scrub ok 2024-03-11T21:17:00.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:00 smithi023 bash[21372]: cluster 2024-03-11T21:17:00.456659+0000 mon.a (mon.0) 2025 : cluster 0 osdmap e458: 8 total, 8 up, 8 in 2024-03-11T21:17:00.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:00 smithi023 bash[29146]: cluster 2024-03-11T21:16:59.212855+0000 osd.7 (osd.7) 617 : cluster 0 6.8 scrub starts 2024-03-11T21:17:00.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:00 smithi023 bash[29146]: cluster 2024-03-11T21:16:59.214981+0000 osd.7 (osd.7) 618 : cluster 0 6.8 scrub ok 2024-03-11T21:17:00.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:00 smithi023 bash[29146]: audit 2024-03-11T21:16:59.657876+0000 mon.a (mon.0) 2023 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-03-11T21:17:00.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:00 smithi023 bash[29146]: cluster 2024-03-11T21:16:59.668716+0000 mon.a (mon.0) 2024 : cluster 0 osdmap e457: 8 total, 8 up, 8 in 2024-03-11T21:17:00.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:00 smithi023 bash[29146]: cluster 2024-03-11T21:17:00.168166+0000 osd.5 (osd.5) 587 : cluster 0 4.8 scrub starts 2024-03-11T21:17:00.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:00 smithi023 bash[29146]: cluster 2024-03-11T21:17:00.171974+0000 osd.5 (osd.5) 588 : cluster 0 4.8 scrub ok 2024-03-11T21:17:00.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:00 smithi023 bash[29146]: cluster 2024-03-11T21:17:00.456659+0000 mon.a (mon.0) 2025 : cluster 0 osdmap e458: 8 total, 8 up, 8 in 2024-03-11T21:17:00.927 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-03-11T21:17:00.927 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:17:00.938 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:00 smithi146 bash[23809]: cluster 2024-03-11T21:16:59.212855+0000 osd.7 (osd.7) 617 : cluster 0 6.8 scrub starts 2024-03-11T21:17:00.938 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:00 smithi146 bash[23809]: cluster 2024-03-11T21:16:59.214981+0000 osd.7 (osd.7) 618 : cluster 0 6.8 scrub ok 2024-03-11T21:17:00.938 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:00 smithi146 bash[23809]: audit 2024-03-11T21:16:59.657876+0000 mon.a (mon.0) 2023 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-03-11T21:17:00.939 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:00 smithi146 bash[23809]: cluster 2024-03-11T21:16:59.668716+0000 mon.a (mon.0) 2024 : cluster 0 osdmap e457: 8 total, 8 up, 8 in 2024-03-11T21:17:00.939 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:00 smithi146 bash[23809]: cluster 2024-03-11T21:17:00.168166+0000 osd.5 (osd.5) 587 : cluster 0 4.8 scrub starts 2024-03-11T21:17:00.939 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:00 smithi146 bash[23809]: cluster 2024-03-11T21:17:00.171974+0000 osd.5 (osd.5) 588 : cluster 0 4.8 scrub ok 2024-03-11T21:17:00.939 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:00 smithi146 bash[23809]: cluster 2024-03-11T21:17:00.456659+0000 mon.a (mon.0) 2025 : cluster 0 osdmap e458: 8 total, 8 up, 8 in 2024-03-11T21:17:01.694 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-03-11T21:17:01.965 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:01 smithi023 bash[21372]: cluster 2024-03-11T21:17:00.499528+0000 mgr.y (mgr.24380) 968 : cluster 0 pgmap v1123: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:17:01.965 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:01 smithi023 bash[29146]: cluster 2024-03-11T21:17:00.499528+0000 mgr.y (mgr.24380) 968 : cluster 0 pgmap v1123: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:17:02.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:01 smithi146 bash[23809]: cluster 2024-03-11T21:17:00.499528+0000 mgr.y (mgr.24380) 968 : cluster 0 pgmap v1123: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:17:03.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:02 smithi146 bash[23809]: cluster 2024-03-11T21:17:01.587610+0000 osd.3 (osd.3) 542 : cluster 0 6.2 scrub starts 2024-03-11T21:17:03.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:02 smithi146 bash[23809]: cluster 2024-03-11T21:17:01.590115+0000 osd.3 (osd.3) 543 : cluster 0 6.2 scrub ok 2024-03-11T21:17:03.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:02 smithi146 bash[23809]: cluster 2024-03-11T21:17:02.117764+0000 osd.5 (osd.5) 589 : cluster 0 3.c scrub starts 2024-03-11T21:17:03.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:02 smithi146 bash[23809]: cluster 2024-03-11T21:17:02.122140+0000 osd.5 (osd.5) 590 : cluster 0 3.c scrub ok 2024-03-11T21:17:03.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:02 smithi023 bash[21372]: cluster 2024-03-11T21:17:01.587610+0000 osd.3 (osd.3) 542 : cluster 0 6.2 scrub starts 2024-03-11T21:17:03.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:02 smithi023 bash[21372]: cluster 2024-03-11T21:17:01.590115+0000 osd.3 (osd.3) 543 : cluster 0 6.2 scrub ok 2024-03-11T21:17:03.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:02 smithi023 bash[21372]: cluster 2024-03-11T21:17:02.117764+0000 osd.5 (osd.5) 589 : cluster 0 3.c scrub starts 2024-03-11T21:17:03.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:02 smithi023 bash[21372]: cluster 2024-03-11T21:17:02.122140+0000 osd.5 (osd.5) 590 : cluster 0 3.c scrub ok 2024-03-11T21:17:03.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:02 smithi023 bash[29146]: cluster 2024-03-11T21:17:01.587610+0000 osd.3 (osd.3) 542 : cluster 0 6.2 scrub starts 2024-03-11T21:17:03.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:02 smithi023 bash[29146]: cluster 2024-03-11T21:17:01.590115+0000 osd.3 (osd.3) 543 : cluster 0 6.2 scrub ok 2024-03-11T21:17:03.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:02 smithi023 bash[29146]: cluster 2024-03-11T21:17:02.117764+0000 osd.5 (osd.5) 589 : cluster 0 3.c scrub starts 2024-03-11T21:17:03.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:02 smithi023 bash[29146]: cluster 2024-03-11T21:17:02.122140+0000 osd.5 (osd.5) 590 : cluster 0 3.c scrub ok 2024-03-11T21:17:04.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:03 smithi146 bash[23809]: cluster 2024-03-11T21:17:02.500904+0000 mgr.y (mgr.24380) 969 : cluster 0 pgmap v1124: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-11T21:17:04.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:03 smithi146 bash[23809]: cluster 2024-03-11T21:17:02.546981+0000 osd.3 (osd.3) 544 : cluster 0 5.1 scrub starts 2024-03-11T21:17:04.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:03 smithi146 bash[23809]: cluster 2024-03-11T21:17:02.549381+0000 osd.3 (osd.3) 545 : cluster 0 5.1 scrub ok 2024-03-11T21:17:04.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:03 smithi023 bash[21372]: cluster 2024-03-11T21:17:02.500904+0000 mgr.y (mgr.24380) 969 : cluster 0 pgmap v1124: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-11T21:17:04.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:03 smithi023 bash[21372]: cluster 2024-03-11T21:17:02.546981+0000 osd.3 (osd.3) 544 : cluster 0 5.1 scrub starts 2024-03-11T21:17:04.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:03 smithi023 bash[21372]: cluster 2024-03-11T21:17:02.549381+0000 osd.3 (osd.3) 545 : cluster 0 5.1 scrub ok 2024-03-11T21:17:04.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:03 smithi023 bash[29146]: cluster 2024-03-11T21:17:02.500904+0000 mgr.y (mgr.24380) 969 : cluster 0 pgmap v1124: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-11T21:17:04.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:03 smithi023 bash[29146]: cluster 2024-03-11T21:17:02.546981+0000 osd.3 (osd.3) 544 : cluster 0 5.1 scrub starts 2024-03-11T21:17:04.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:03 smithi023 bash[29146]: cluster 2024-03-11T21:17:02.549381+0000 osd.3 (osd.3) 545 : cluster 0 5.1 scrub ok 2024-03-11T21:17:05.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:04 smithi146 bash[23809]: cluster 2024-03-11T21:17:04.133398+0000 osd.5 (osd.5) 591 : cluster 0 6.1a scrub starts 2024-03-11T21:17:05.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:04 smithi146 bash[23809]: cluster 2024-03-11T21:17:04.172165+0000 osd.5 (osd.5) 592 : cluster 0 6.1a scrub ok 2024-03-11T21:17:05.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:04 smithi023 bash[21372]: cluster 2024-03-11T21:17:04.133398+0000 osd.5 (osd.5) 591 : cluster 0 6.1a scrub starts 2024-03-11T21:17:05.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:04 smithi023 bash[21372]: cluster 2024-03-11T21:17:04.172165+0000 osd.5 (osd.5) 592 : cluster 0 6.1a scrub ok 2024-03-11T21:17:05.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:04 smithi023 bash[29146]: cluster 2024-03-11T21:17:04.133398+0000 osd.5 (osd.5) 591 : cluster 0 6.1a scrub starts 2024-03-11T21:17:05.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:04 smithi023 bash[29146]: cluster 2024-03-11T21:17:04.172165+0000 osd.5 (osd.5) 592 : cluster 0 6.1a scrub ok 2024-03-11T21:17:05.235 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:17:06.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:05 smithi146 bash[23809]: cluster 2024-03-11T21:17:04.501817+0000 mgr.y (mgr.24380) 970 : cluster 0 pgmap v1125: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:17:06.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:05 smithi146 bash[23809]: cluster 2024-03-11T21:17:04.542178+0000 osd.3 (osd.3) 546 : cluster 0 6.c deep-scrub starts 2024-03-11T21:17:06.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:05 smithi146 bash[23809]: cluster 2024-03-11T21:17:04.544805+0000 osd.3 (osd.3) 547 : cluster 0 6.c deep-scrub ok 2024-03-11T21:17:06.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:05 smithi023 bash[29146]: cluster 2024-03-11T21:17:04.501817+0000 mgr.y (mgr.24380) 970 : cluster 0 pgmap v1125: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:17:06.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:05 smithi023 bash[29146]: cluster 2024-03-11T21:17:04.542178+0000 osd.3 (osd.3) 546 : cluster 0 6.c deep-scrub starts 2024-03-11T21:17:06.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:05 smithi023 bash[29146]: cluster 2024-03-11T21:17:04.544805+0000 osd.3 (osd.3) 547 : cluster 0 6.c deep-scrub ok 2024-03-11T21:17:06.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:05 smithi023 bash[21372]: cluster 2024-03-11T21:17:04.501817+0000 mgr.y (mgr.24380) 970 : cluster 0 pgmap v1125: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:17:06.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:05 smithi023 bash[21372]: cluster 2024-03-11T21:17:04.542178+0000 osd.3 (osd.3) 546 : cluster 0 6.c deep-scrub starts 2024-03-11T21:17:06.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:05 smithi023 bash[21372]: cluster 2024-03-11T21:17:04.544805+0000 osd.3 (osd.3) 547 : cluster 0 6.c deep-scrub ok 2024-03-11T21:17:06.484 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:17:06.503 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:17:06 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:17:07.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:06 smithi146 bash[23809]: cluster 2024-03-11T21:17:05.586633+0000 osd.3 (osd.3) 548 : cluster 0 6.f scrub starts 2024-03-11T21:17:07.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:06 smithi146 bash[23809]: cluster 2024-03-11T21:17:05.829947+0000 osd.3 (osd.3) 549 : cluster 0 6.f scrub ok 2024-03-11T21:17:07.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:06 smithi023 bash[21372]: cluster 2024-03-11T21:17:05.586633+0000 osd.3 (osd.3) 548 : cluster 0 6.f scrub starts 2024-03-11T21:17:07.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:06 smithi023 bash[21372]: cluster 2024-03-11T21:17:05.829947+0000 osd.3 (osd.3) 549 : cluster 0 6.f scrub ok 2024-03-11T21:17:07.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:06 smithi023 bash[29146]: cluster 2024-03-11T21:17:05.586633+0000 osd.3 (osd.3) 548 : cluster 0 6.f scrub starts 2024-03-11T21:17:07.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:06 smithi023 bash[29146]: cluster 2024-03-11T21:17:05.829947+0000 osd.3 (osd.3) 549 : cluster 0 6.f scrub ok 2024-03-11T21:17:08.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:07 smithi146 bash[23809]: audit 2024-03-11T21:17:06.184431+0000 mgr.y (mgr.24380) 971 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:17:08.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:07 smithi146 bash[23809]: cluster 2024-03-11T21:17:06.502798+0000 mgr.y (mgr.24380) 972 : cluster 0 pgmap v1126: 152 pgs: 1 active+clean+scrubbing, 151 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:17:08.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:07 smithi146 bash[23809]: audit 2024-03-11T21:17:06.783626+0000 mon.a (mon.0) 2026 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:17:08.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:07 smithi146 bash[23809]: cluster 2024-03-11T21:17:07.053523+0000 osd.5 (osd.5) 593 : cluster 0 2.c scrub starts 2024-03-11T21:17:08.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:07 smithi146 bash[23809]: cluster 2024-03-11T21:17:07.056298+0000 osd.5 (osd.5) 594 : cluster 0 2.c scrub ok 2024-03-11T21:17:08.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:07 smithi023 bash[21372]: audit 2024-03-11T21:17:06.184431+0000 mgr.y (mgr.24380) 971 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:17:08.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:07 smithi023 bash[21372]: cluster 2024-03-11T21:17:06.502798+0000 mgr.y (mgr.24380) 972 : cluster 0 pgmap v1126: 152 pgs: 1 active+clean+scrubbing, 151 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:17:08.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:07 smithi023 bash[21372]: audit 2024-03-11T21:17:06.783626+0000 mon.a (mon.0) 2026 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:17:08.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:07 smithi023 bash[21372]: cluster 2024-03-11T21:17:07.053523+0000 osd.5 (osd.5) 593 : cluster 0 2.c scrub starts 2024-03-11T21:17:08.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:07 smithi023 bash[21372]: cluster 2024-03-11T21:17:07.056298+0000 osd.5 (osd.5) 594 : cluster 0 2.c scrub ok 2024-03-11T21:17:08.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:07 smithi023 bash[29146]: audit 2024-03-11T21:17:06.184431+0000 mgr.y (mgr.24380) 971 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:17:08.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:07 smithi023 bash[29146]: cluster 2024-03-11T21:17:06.502798+0000 mgr.y (mgr.24380) 972 : cluster 0 pgmap v1126: 152 pgs: 1 active+clean+scrubbing, 151 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:17:08.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:07 smithi023 bash[29146]: audit 2024-03-11T21:17:06.783626+0000 mon.a (mon.0) 2026 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:17:08.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:07 smithi023 bash[29146]: cluster 2024-03-11T21:17:07.053523+0000 osd.5 (osd.5) 593 : cluster 0 2.c scrub starts 2024-03-11T21:17:08.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:07 smithi023 bash[29146]: cluster 2024-03-11T21:17:07.056298+0000 osd.5 (osd.5) 594 : cluster 0 2.c scrub ok 2024-03-11T21:17:08.729 INFO:teuthology.orchestra.run.smithi023.stderr:reweighted osd.2 to 1 (10000) 2024-03-11T21:17:08.731 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:17:08 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:17:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:17:09.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:08 smithi146 bash[23809]: cluster 2024-03-11T21:17:07.538458+0000 osd.3 (osd.3) 550 : cluster 0 6.6 scrub starts 2024-03-11T21:17:09.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:08 smithi146 bash[23809]: cluster 2024-03-11T21:17:07.571290+0000 osd.3 (osd.3) 551 : cluster 0 6.6 scrub ok 2024-03-11T21:17:09.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:08 smithi146 bash[23809]: cluster 2024-03-11T21:17:08.011309+0000 osd.1 (osd.1) 360 : cluster 0 6.1d scrub starts 2024-03-11T21:17:09.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:08 smithi146 bash[23809]: cluster 2024-03-11T21:17:08.157317+0000 osd.1 (osd.1) 361 : cluster 0 6.1d scrub ok 2024-03-11T21:17:09.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:08 smithi146 bash[23809]: cluster 2024-03-11T21:17:08.335336+0000 osd.0 (osd.0) 438 : cluster 0 3.1f scrub starts 2024-03-11T21:17:09.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:08 smithi146 bash[23809]: cluster 2024-03-11T21:17:08.338490+0000 osd.0 (osd.0) 439 : cluster 0 3.1f scrub ok 2024-03-11T21:17:09.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:08 smithi146 bash[23809]: audit 2024-03-11T21:17:08.496570+0000 mon.a (mon.0) 2027 : audit 1 from='client.? 172.21.15.23:0/255580527' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-11T21:17:09.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:08 smithi023 bash[21372]: cluster 2024-03-11T21:17:07.538458+0000 osd.3 (osd.3) 550 : cluster 0 6.6 scrub starts 2024-03-11T21:17:09.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:08 smithi023 bash[21372]: cluster 2024-03-11T21:17:07.571290+0000 osd.3 (osd.3) 551 : cluster 0 6.6 scrub ok 2024-03-11T21:17:09.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:08 smithi023 bash[21372]: cluster 2024-03-11T21:17:08.011309+0000 osd.1 (osd.1) 360 : cluster 0 6.1d scrub starts 2024-03-11T21:17:09.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:08 smithi023 bash[21372]: cluster 2024-03-11T21:17:08.157317+0000 osd.1 (osd.1) 361 : cluster 0 6.1d scrub ok 2024-03-11T21:17:09.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:08 smithi023 bash[21372]: cluster 2024-03-11T21:17:08.335336+0000 osd.0 (osd.0) 438 : cluster 0 3.1f scrub starts 2024-03-11T21:17:09.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:08 smithi023 bash[21372]: cluster 2024-03-11T21:17:08.338490+0000 osd.0 (osd.0) 439 : cluster 0 3.1f scrub ok 2024-03-11T21:17:09.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:08 smithi023 bash[21372]: audit 2024-03-11T21:17:08.496570+0000 mon.a (mon.0) 2027 : audit 1 from='client.? 172.21.15.23:0/255580527' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-11T21:17:09.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:08 smithi023 bash[29146]: cluster 2024-03-11T21:17:07.538458+0000 osd.3 (osd.3) 550 : cluster 0 6.6 scrub starts 2024-03-11T21:17:09.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:08 smithi023 bash[29146]: cluster 2024-03-11T21:17:07.571290+0000 osd.3 (osd.3) 551 : cluster 0 6.6 scrub ok 2024-03-11T21:17:09.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:08 smithi023 bash[29146]: cluster 2024-03-11T21:17:08.011309+0000 osd.1 (osd.1) 360 : cluster 0 6.1d scrub starts 2024-03-11T21:17:09.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:08 smithi023 bash[29146]: cluster 2024-03-11T21:17:08.157317+0000 osd.1 (osd.1) 361 : cluster 0 6.1d scrub ok 2024-03-11T21:17:09.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:08 smithi023 bash[29146]: cluster 2024-03-11T21:17:08.335336+0000 osd.0 (osd.0) 438 : cluster 0 3.1f scrub starts 2024-03-11T21:17:09.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:08 smithi023 bash[29146]: cluster 2024-03-11T21:17:08.338490+0000 osd.0 (osd.0) 439 : cluster 0 3.1f scrub ok 2024-03-11T21:17:09.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:08 smithi023 bash[29146]: audit 2024-03-11T21:17:08.496570+0000 mon.a (mon.0) 2027 : audit 1 from='client.? 172.21.15.23:0/255580527' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-11T21:17:09.542 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-03-11T21:17:09.542 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:17:09.603 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd primary-affinity 2 1 2024-03-11T21:17:10.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:09 smithi146 bash[23809]: cluster 2024-03-11T21:17:08.503439+0000 mgr.y (mgr.24380) 973 : cluster 0 pgmap v1127: 152 pgs: 1 active+clean+scrubbing, 151 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T21:17:10.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:09 smithi146 bash[23809]: cluster 2024-03-11T21:17:08.565126+0000 osd.3 (osd.3) 552 : cluster 0 6.b scrub starts 2024-03-11T21:17:10.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:09 smithi146 bash[23809]: audit 2024-03-11T21:17:08.731702+0000 mon.a (mon.0) 2028 : audit 1 from='client.? 172.21.15.23:0/255580527' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-11T21:17:10.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:09 smithi146 bash[23809]: cluster 2024-03-11T21:17:08.741561+0000 mon.a (mon.0) 2029 : cluster 0 osdmap e459: 8 total, 8 up, 8 in 2024-03-11T21:17:10.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:09 smithi146 bash[23809]: cluster 2024-03-11T21:17:08.771042+0000 osd.3 (osd.3) 553 : cluster 0 6.b scrub ok 2024-03-11T21:17:10.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:09 smithi023 bash[21372]: cluster 2024-03-11T21:17:08.503439+0000 mgr.y (mgr.24380) 973 : cluster 0 pgmap v1127: 152 pgs: 1 active+clean+scrubbing, 151 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T21:17:10.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:09 smithi023 bash[21372]: cluster 2024-03-11T21:17:08.565126+0000 osd.3 (osd.3) 552 : cluster 0 6.b scrub starts 2024-03-11T21:17:10.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:09 smithi023 bash[21372]: audit 2024-03-11T21:17:08.731702+0000 mon.a (mon.0) 2028 : audit 1 from='client.? 172.21.15.23:0/255580527' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-11T21:17:10.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:09 smithi023 bash[21372]: cluster 2024-03-11T21:17:08.741561+0000 mon.a (mon.0) 2029 : cluster 0 osdmap e459: 8 total, 8 up, 8 in 2024-03-11T21:17:10.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:09 smithi023 bash[21372]: cluster 2024-03-11T21:17:08.771042+0000 osd.3 (osd.3) 553 : cluster 0 6.b scrub ok 2024-03-11T21:17:10.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:09 smithi023 bash[29146]: cluster 2024-03-11T21:17:08.503439+0000 mgr.y (mgr.24380) 973 : cluster 0 pgmap v1127: 152 pgs: 1 active+clean+scrubbing, 151 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T21:17:10.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:09 smithi023 bash[29146]: cluster 2024-03-11T21:17:08.565126+0000 osd.3 (osd.3) 552 : cluster 0 6.b scrub starts 2024-03-11T21:17:10.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:09 smithi023 bash[29146]: audit 2024-03-11T21:17:08.731702+0000 mon.a (mon.0) 2028 : audit 1 from='client.? 172.21.15.23:0/255580527' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-11T21:17:10.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:09 smithi023 bash[29146]: cluster 2024-03-11T21:17:08.741561+0000 mon.a (mon.0) 2029 : cluster 0 osdmap e459: 8 total, 8 up, 8 in 2024-03-11T21:17:10.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:09 smithi023 bash[29146]: cluster 2024-03-11T21:17:08.771042+0000 osd.3 (osd.3) 553 : cluster 0 6.b scrub ok 2024-03-11T21:17:10.321 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-03-11T21:17:11.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:10 smithi146 bash[23809]: cluster 2024-03-11T21:17:09.205547+0000 osd.7 (osd.7) 619 : cluster 0 6.3 scrub starts 2024-03-11T21:17:11.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:10 smithi146 bash[23809]: cluster 2024-03-11T21:17:09.366931+0000 osd.7 (osd.7) 620 : cluster 0 6.3 scrub ok 2024-03-11T21:17:11.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:10 smithi146 bash[23809]: cluster 2024-03-11T21:17:09.632388+0000 osd.6 (osd.6) 494 : cluster 0 2.14 scrub starts 2024-03-11T21:17:11.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:10 smithi146 bash[23809]: cluster 2024-03-11T21:17:09.633944+0000 osd.6 (osd.6) 495 : cluster 0 2.14 scrub ok 2024-03-11T21:17:11.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:10 smithi146 bash[23809]: cluster 2024-03-11T21:17:10.466486+0000 mon.a (mon.0) 2030 : cluster 0 osdmap e460: 8 total, 8 up, 8 in 2024-03-11T21:17:11.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:10 smithi023 bash[21372]: cluster 2024-03-11T21:17:09.205547+0000 osd.7 (osd.7) 619 : cluster 0 6.3 scrub starts 2024-03-11T21:17:11.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:10 smithi023 bash[21372]: cluster 2024-03-11T21:17:09.366931+0000 osd.7 (osd.7) 620 : cluster 0 6.3 scrub ok 2024-03-11T21:17:11.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:10 smithi023 bash[21372]: cluster 2024-03-11T21:17:09.632388+0000 osd.6 (osd.6) 494 : cluster 0 2.14 scrub starts 2024-03-11T21:17:11.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:10 smithi023 bash[21372]: cluster 2024-03-11T21:17:09.633944+0000 osd.6 (osd.6) 495 : cluster 0 2.14 scrub ok 2024-03-11T21:17:11.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:10 smithi023 bash[21372]: cluster 2024-03-11T21:17:10.466486+0000 mon.a (mon.0) 2030 : cluster 0 osdmap e460: 8 total, 8 up, 8 in 2024-03-11T21:17:11.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:10 smithi023 bash[29146]: cluster 2024-03-11T21:17:09.205547+0000 osd.7 (osd.7) 619 : cluster 0 6.3 scrub starts 2024-03-11T21:17:11.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:10 smithi023 bash[29146]: cluster 2024-03-11T21:17:09.366931+0000 osd.7 (osd.7) 620 : cluster 0 6.3 scrub ok 2024-03-11T21:17:11.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:10 smithi023 bash[29146]: cluster 2024-03-11T21:17:09.632388+0000 osd.6 (osd.6) 494 : cluster 0 2.14 scrub starts 2024-03-11T21:17:11.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:10 smithi023 bash[29146]: cluster 2024-03-11T21:17:09.633944+0000 osd.6 (osd.6) 495 : cluster 0 2.14 scrub ok 2024-03-11T21:17:11.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:10 smithi023 bash[29146]: cluster 2024-03-11T21:17:10.466486+0000 mon.a (mon.0) 2030 : cluster 0 osdmap e460: 8 total, 8 up, 8 in 2024-03-11T21:17:12.006 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:11 smithi146 bash[23809]: cluster 2024-03-11T21:17:10.504076+0000 mgr.y (mgr.24380) 974 : cluster 0 pgmap v1130: 152 pgs: 1 active+clean+scrubbing, 151 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:17:12.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:11 smithi023 bash[29146]: cluster 2024-03-11T21:17:10.504076+0000 mgr.y (mgr.24380) 974 : cluster 0 pgmap v1130: 152 pgs: 1 active+clean+scrubbing, 151 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:17:12.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:11 smithi023 bash[21372]: cluster 2024-03-11T21:17:10.504076+0000 mgr.y (mgr.24380) 974 : cluster 0 pgmap v1130: 152 pgs: 1 active+clean+scrubbing, 151 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:17:13.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:12 smithi023 bash[21372]: cluster 2024-03-11T21:17:12.102791+0000 osd.5 (osd.5) 595 : cluster 0 2.12 scrub starts 2024-03-11T21:17:13.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:12 smithi023 bash[21372]: cluster 2024-03-11T21:17:12.106589+0000 osd.5 (osd.5) 596 : cluster 0 2.12 scrub ok 2024-03-11T21:17:13.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:12 smithi023 bash[29146]: cluster 2024-03-11T21:17:12.102791+0000 osd.5 (osd.5) 595 : cluster 0 2.12 scrub starts 2024-03-11T21:17:13.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:12 smithi023 bash[29146]: cluster 2024-03-11T21:17:12.106589+0000 osd.5 (osd.5) 596 : cluster 0 2.12 scrub ok 2024-03-11T21:17:13.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:12 smithi146 bash[23809]: cluster 2024-03-11T21:17:12.102791+0000 osd.5 (osd.5) 595 : cluster 0 2.12 scrub starts 2024-03-11T21:17:13.254 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:12 smithi146 bash[23809]: cluster 2024-03-11T21:17:12.106589+0000 osd.5 (osd.5) 596 : cluster 0 2.12 scrub ok 2024-03-11T21:17:14.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:13 smithi023 bash[21372]: cluster 2024-03-11T21:17:12.505471+0000 mgr.y (mgr.24380) 975 : cluster 0 pgmap v1131: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:17:14.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:13 smithi023 bash[21372]: cluster 2024-03-11T21:17:13.065638+0000 osd.5 (osd.5) 597 : cluster 0 6.19 scrub starts 2024-03-11T21:17:14.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:13 smithi023 bash[21372]: cluster 2024-03-11T21:17:13.174135+0000 osd.5 (osd.5) 598 : cluster 0 6.19 scrub ok 2024-03-11T21:17:14.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:13 smithi023 bash[29146]: cluster 2024-03-11T21:17:12.505471+0000 mgr.y (mgr.24380) 975 : cluster 0 pgmap v1131: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:17:14.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:13 smithi023 bash[29146]: cluster 2024-03-11T21:17:13.065638+0000 osd.5 (osd.5) 597 : cluster 0 6.19 scrub starts 2024-03-11T21:17:14.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:13 smithi023 bash[29146]: cluster 2024-03-11T21:17:13.174135+0000 osd.5 (osd.5) 598 : cluster 0 6.19 scrub ok 2024-03-11T21:17:14.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:13 smithi146 bash[23809]: cluster 2024-03-11T21:17:12.505471+0000 mgr.y (mgr.24380) 975 : cluster 0 pgmap v1131: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:17:14.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:13 smithi146 bash[23809]: cluster 2024-03-11T21:17:13.065638+0000 osd.5 (osd.5) 597 : cluster 0 6.19 scrub starts 2024-03-11T21:17:14.254 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:13 smithi146 bash[23809]: cluster 2024-03-11T21:17:13.174135+0000 osd.5 (osd.5) 598 : cluster 0 6.19 scrub ok 2024-03-11T21:17:14.406 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:17:15.122 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:17:16.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:15 smithi023 bash[21372]: cluster 2024-03-11T21:17:13.908635+0000 osd.7 (osd.7) 621 : cluster 0 purged_snaps scrub starts 2024-03-11T21:17:16.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:15 smithi023 bash[21372]: cluster 2024-03-11T21:17:13.908746+0000 osd.7 (osd.7) 622 : cluster 0 purged_snaps scrub ok 2024-03-11T21:17:16.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:15 smithi023 bash[21372]: cluster 2024-03-11T21:17:14.506116+0000 mgr.y (mgr.24380) 976 : cluster 0 pgmap v1132: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:17:16.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:15 smithi023 bash[21372]: cluster 2024-03-11T21:17:14.812035+0000 mon.a (mon.0) 2031 : cluster 0 osdmap e461: 8 total, 8 up, 8 in 2024-03-11T21:17:16.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:15 smithi023 bash[21372]: cluster 2024-03-11T21:17:15.033030+0000 osd.5 (osd.5) 599 : cluster 0 6.7 scrub starts 2024-03-11T21:17:16.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:15 smithi023 bash[21372]: cluster 2024-03-11T21:17:15.176780+0000 osd.5 (osd.5) 600 : cluster 0 6.7 scrub ok 2024-03-11T21:17:16.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:15 smithi023 bash[21372]: cluster 2024-03-11T21:17:15.315523+0000 osd.0 (osd.0) 440 : cluster 0 4.16 scrub starts 2024-03-11T21:17:16.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:15 smithi023 bash[21372]: cluster 2024-03-11T21:17:15.318008+0000 osd.0 (osd.0) 441 : cluster 0 4.16 scrub ok 2024-03-11T21:17:16.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:15 smithi023 bash[21372]: cluster 2024-03-11T21:17:15.469754+0000 mon.a (mon.0) 2032 : cluster 0 osdmap e462: 8 total, 8 up, 8 in 2024-03-11T21:17:16.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:15 smithi023 bash[29146]: cluster 2024-03-11T21:17:13.908635+0000 osd.7 (osd.7) 621 : cluster 0 purged_snaps scrub starts 2024-03-11T21:17:16.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:15 smithi023 bash[29146]: cluster 2024-03-11T21:17:13.908746+0000 osd.7 (osd.7) 622 : cluster 0 purged_snaps scrub ok 2024-03-11T21:17:16.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:15 smithi023 bash[29146]: cluster 2024-03-11T21:17:14.506116+0000 mgr.y (mgr.24380) 976 : cluster 0 pgmap v1132: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:17:16.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:15 smithi023 bash[29146]: cluster 2024-03-11T21:17:14.812035+0000 mon.a (mon.0) 2031 : cluster 0 osdmap e461: 8 total, 8 up, 8 in 2024-03-11T21:17:16.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:15 smithi023 bash[29146]: cluster 2024-03-11T21:17:15.033030+0000 osd.5 (osd.5) 599 : cluster 0 6.7 scrub starts 2024-03-11T21:17:16.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:15 smithi023 bash[29146]: cluster 2024-03-11T21:17:15.176780+0000 osd.5 (osd.5) 600 : cluster 0 6.7 scrub ok 2024-03-11T21:17:16.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:15 smithi023 bash[29146]: cluster 2024-03-11T21:17:15.315523+0000 osd.0 (osd.0) 440 : cluster 0 4.16 scrub starts 2024-03-11T21:17:16.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:15 smithi023 bash[29146]: cluster 2024-03-11T21:17:15.318008+0000 osd.0 (osd.0) 441 : cluster 0 4.16 scrub ok 2024-03-11T21:17:16.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:15 smithi023 bash[29146]: cluster 2024-03-11T21:17:15.469754+0000 mon.a (mon.0) 2032 : cluster 0 osdmap e462: 8 total, 8 up, 8 in 2024-03-11T21:17:16.179 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:15 smithi146 bash[23809]: cluster 2024-03-11T21:17:13.908635+0000 osd.7 (osd.7) 621 : cluster 0 purged_snaps scrub starts 2024-03-11T21:17:16.179 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:15 smithi146 bash[23809]: cluster 2024-03-11T21:17:13.908746+0000 osd.7 (osd.7) 622 : cluster 0 purged_snaps scrub ok 2024-03-11T21:17:16.180 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:15 smithi146 bash[23809]: cluster 2024-03-11T21:17:14.506116+0000 mgr.y (mgr.24380) 976 : cluster 0 pgmap v1132: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:17:16.180 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:15 smithi146 bash[23809]: cluster 2024-03-11T21:17:14.812035+0000 mon.a (mon.0) 2031 : cluster 0 osdmap e461: 8 total, 8 up, 8 in 2024-03-11T21:17:16.180 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:15 smithi146 bash[23809]: cluster 2024-03-11T21:17:15.033030+0000 osd.5 (osd.5) 599 : cluster 0 6.7 scrub starts 2024-03-11T21:17:16.180 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:15 smithi146 bash[23809]: cluster 2024-03-11T21:17:15.176780+0000 osd.5 (osd.5) 600 : cluster 0 6.7 scrub ok 2024-03-11T21:17:16.180 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:15 smithi146 bash[23809]: cluster 2024-03-11T21:17:15.315523+0000 osd.0 (osd.0) 440 : cluster 0 4.16 scrub starts 2024-03-11T21:17:16.180 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:15 smithi146 bash[23809]: cluster 2024-03-11T21:17:15.318008+0000 osd.0 (osd.0) 441 : cluster 0 4.16 scrub ok 2024-03-11T21:17:16.180 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:15 smithi146 bash[23809]: cluster 2024-03-11T21:17:15.469754+0000 mon.a (mon.0) 2032 : cluster 0 osdmap e462: 8 total, 8 up, 8 in 2024-03-11T21:17:16.503 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:17:16 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:17:17.754 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:17 smithi146 bash[23809]: cluster 2024-03-11T21:17:15.680370+0000 osd.6 (osd.6) 496 : cluster 0 2.1a deep-scrub starts 2024-03-11T21:17:17.754 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:17 smithi146 bash[23809]: cluster 2024-03-11T21:17:15.682869+0000 osd.6 (osd.6) 497 : cluster 0 2.1a deep-scrub ok 2024-03-11T21:17:17.754 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:17 smithi146 bash[23809]: cluster 2024-03-11T21:17:15.784750+0000 osd.5 (osd.5) 601 : cluster 0 purged_snaps scrub starts 2024-03-11T21:17:17.754 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:17 smithi146 bash[23809]: cluster 2024-03-11T21:17:15.784874+0000 osd.5 (osd.5) 602 : cluster 0 purged_snaps scrub ok 2024-03-11T21:17:17.754 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:17 smithi146 bash[23809]: audit 2024-03-11T21:17:16.192194+0000 mgr.y (mgr.24380) 977 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:17:17.754 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:17 smithi146 bash[23809]: cluster 2024-03-11T21:17:16.198857+0000 osd.7 (osd.7) 623 : cluster 0 6.14 scrub starts 2024-03-11T21:17:17.757 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:17 smithi146 bash[23809]: cluster 2024-03-11T21:17:16.203396+0000 osd.7 (osd.7) 624 : cluster 0 6.14 scrub ok 2024-03-11T21:17:17.757 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:17 smithi146 bash[23809]: cluster 2024-03-11T21:17:16.475524+0000 mon.a (mon.0) 2033 : cluster 0 osdmap e463: 8 total, 8 up, 8 in 2024-03-11T21:17:17.757 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:17 smithi146 bash[23809]: cluster 2024-03-11T21:17:16.506741+0000 mgr.y (mgr.24380) 978 : cluster 0 pgmap v1136: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-11T21:17:17.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:17 smithi023 bash[21372]: cluster 2024-03-11T21:17:15.680370+0000 osd.6 (osd.6) 496 : cluster 0 2.1a deep-scrub starts 2024-03-11T21:17:17.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:17 smithi023 bash[21372]: cluster 2024-03-11T21:17:15.682869+0000 osd.6 (osd.6) 497 : cluster 0 2.1a deep-scrub ok 2024-03-11T21:17:17.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:17 smithi023 bash[21372]: cluster 2024-03-11T21:17:15.784750+0000 osd.5 (osd.5) 601 : cluster 0 purged_snaps scrub starts 2024-03-11T21:17:17.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:17 smithi023 bash[21372]: cluster 2024-03-11T21:17:15.784874+0000 osd.5 (osd.5) 602 : cluster 0 purged_snaps scrub ok 2024-03-11T21:17:17.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:17 smithi023 bash[21372]: audit 2024-03-11T21:17:16.192194+0000 mgr.y (mgr.24380) 977 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:17:17.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:17 smithi023 bash[21372]: cluster 2024-03-11T21:17:16.198857+0000 osd.7 (osd.7) 623 : cluster 0 6.14 scrub starts 2024-03-11T21:17:17.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:17 smithi023 bash[21372]: cluster 2024-03-11T21:17:16.203396+0000 osd.7 (osd.7) 624 : cluster 0 6.14 scrub ok 2024-03-11T21:17:17.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:17 smithi023 bash[21372]: cluster 2024-03-11T21:17:16.475524+0000 mon.a (mon.0) 2033 : cluster 0 osdmap e463: 8 total, 8 up, 8 in 2024-03-11T21:17:17.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:17 smithi023 bash[21372]: cluster 2024-03-11T21:17:16.506741+0000 mgr.y (mgr.24380) 978 : cluster 0 pgmap v1136: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-11T21:17:17.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:17 smithi023 bash[29146]: cluster 2024-03-11T21:17:15.680370+0000 osd.6 (osd.6) 496 : cluster 0 2.1a deep-scrub starts 2024-03-11T21:17:17.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:17 smithi023 bash[29146]: cluster 2024-03-11T21:17:15.682869+0000 osd.6 (osd.6) 497 : cluster 0 2.1a deep-scrub ok 2024-03-11T21:17:17.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:17 smithi023 bash[29146]: cluster 2024-03-11T21:17:15.784750+0000 osd.5 (osd.5) 601 : cluster 0 purged_snaps scrub starts 2024-03-11T21:17:17.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:17 smithi023 bash[29146]: cluster 2024-03-11T21:17:15.784874+0000 osd.5 (osd.5) 602 : cluster 0 purged_snaps scrub ok 2024-03-11T21:17:17.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:17 smithi023 bash[29146]: audit 2024-03-11T21:17:16.192194+0000 mgr.y (mgr.24380) 977 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:17:17.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:17 smithi023 bash[29146]: cluster 2024-03-11T21:17:16.198857+0000 osd.7 (osd.7) 623 : cluster 0 6.14 scrub starts 2024-03-11T21:17:17.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:17 smithi023 bash[29146]: cluster 2024-03-11T21:17:16.203396+0000 osd.7 (osd.7) 624 : cluster 0 6.14 scrub ok 2024-03-11T21:17:17.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:17 smithi023 bash[29146]: cluster 2024-03-11T21:17:16.475524+0000 mon.a (mon.0) 2033 : cluster 0 osdmap e463: 8 total, 8 up, 8 in 2024-03-11T21:17:17.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:17 smithi023 bash[29146]: cluster 2024-03-11T21:17:16.506741+0000 mgr.y (mgr.24380) 978 : cluster 0 pgmap v1136: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-11T21:17:18.193 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-03-11T21:17:18.193 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:17:18.498 INFO:teuthology.orchestra.run.smithi023.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-03-11T21:17:18.710 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:17:18 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:17:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:17:18.710 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:18 smithi023 bash[21372]: cluster 2024-03-11T21:17:16.680742+0000 osd.6 (osd.6) 498 : cluster 0 2.11 scrub starts 2024-03-11T21:17:18.710 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:18 smithi023 bash[21372]: cluster 2024-03-11T21:17:16.684054+0000 osd.6 (osd.6) 499 : cluster 0 2.11 scrub ok 2024-03-11T21:17:18.710 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:18 smithi023 bash[21372]: cluster 2024-03-11T21:17:16.768724+0000 osd.0 (osd.0) 442 : cluster 0 purged_snaps scrub starts 2024-03-11T21:17:18.710 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:18 smithi023 bash[21372]: cluster 2024-03-11T21:17:16.768831+0000 osd.0 (osd.0) 443 : cluster 0 purged_snaps scrub ok 2024-03-11T21:17:18.710 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:18 smithi023 bash[21372]: cluster 2024-03-11T21:17:17.090949+0000 osd.5 (osd.5) 603 : cluster 0 3.1c scrub starts 2024-03-11T21:17:18.710 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:18 smithi023 bash[21372]: cluster 2024-03-11T21:17:17.095959+0000 osd.5 (osd.5) 604 : cluster 0 3.1c scrub ok 2024-03-11T21:17:18.710 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:18 smithi023 bash[21372]: cluster 2024-03-11T21:17:17.255047+0000 osd.4 (osd.4) 121 : cluster 0 purged_snaps scrub starts 2024-03-11T21:17:18.710 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:18 smithi023 bash[21372]: cluster 2024-03-11T21:17:17.255194+0000 osd.4 (osd.4) 122 : cluster 0 purged_snaps scrub ok 2024-03-11T21:17:18.711 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:18 smithi023 bash[21372]: cluster 2024-03-11T21:17:17.495272+0000 mon.a (mon.0) 2034 : cluster 0 osdmap e464: 8 total, 8 up, 8 in 2024-03-11T21:17:18.711 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:18 smithi023 bash[21372]: audit 2024-03-11T21:17:17.672046+0000 mon.c (mon.2) 108 : audit 1 from='client.? 172.21.15.23:0/3915556913' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-03-11T21:17:18.711 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:18 smithi023 bash[21372]: audit 2024-03-11T21:17:17.672761+0000 mon.a (mon.0) 2035 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-03-11T21:17:18.711 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:18 smithi023 bash[29146]: cluster 2024-03-11T21:17:16.680742+0000 osd.6 (osd.6) 498 : cluster 0 2.11 scrub starts 2024-03-11T21:17:18.711 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:18 smithi023 bash[29146]: cluster 2024-03-11T21:17:16.684054+0000 osd.6 (osd.6) 499 : cluster 0 2.11 scrub ok 2024-03-11T21:17:18.711 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:18 smithi023 bash[29146]: cluster 2024-03-11T21:17:16.768724+0000 osd.0 (osd.0) 442 : cluster 0 purged_snaps scrub starts 2024-03-11T21:17:18.711 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:18 smithi023 bash[29146]: cluster 2024-03-11T21:17:16.768831+0000 osd.0 (osd.0) 443 : cluster 0 purged_snaps scrub ok 2024-03-11T21:17:18.711 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:18 smithi023 bash[29146]: cluster 2024-03-11T21:17:17.090949+0000 osd.5 (osd.5) 603 : cluster 0 3.1c scrub starts 2024-03-11T21:17:18.711 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:18 smithi023 bash[29146]: cluster 2024-03-11T21:17:17.095959+0000 osd.5 (osd.5) 604 : cluster 0 3.1c scrub ok 2024-03-11T21:17:18.711 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:18 smithi023 bash[29146]: cluster 2024-03-11T21:17:17.255047+0000 osd.4 (osd.4) 121 : cluster 0 purged_snaps scrub starts 2024-03-11T21:17:18.711 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:18 smithi023 bash[29146]: cluster 2024-03-11T21:17:17.255194+0000 osd.4 (osd.4) 122 : cluster 0 purged_snaps scrub ok 2024-03-11T21:17:18.712 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:18 smithi023 bash[29146]: cluster 2024-03-11T21:17:17.495272+0000 mon.a (mon.0) 2034 : cluster 0 osdmap e464: 8 total, 8 up, 8 in 2024-03-11T21:17:18.712 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:18 smithi023 bash[29146]: audit 2024-03-11T21:17:17.672046+0000 mon.c (mon.2) 108 : audit 1 from='client.? 172.21.15.23:0/3915556913' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-03-11T21:17:18.712 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:18 smithi023 bash[29146]: audit 2024-03-11T21:17:17.672761+0000 mon.a (mon.0) 2035 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-03-11T21:17:18.753 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:18 smithi146 bash[23809]: cluster 2024-03-11T21:17:16.680742+0000 osd.6 (osd.6) 498 : cluster 0 2.11 scrub starts 2024-03-11T21:17:18.753 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:18 smithi146 bash[23809]: cluster 2024-03-11T21:17:16.684054+0000 osd.6 (osd.6) 499 : cluster 0 2.11 scrub ok 2024-03-11T21:17:18.754 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:18 smithi146 bash[23809]: cluster 2024-03-11T21:17:16.768724+0000 osd.0 (osd.0) 442 : cluster 0 purged_snaps scrub starts 2024-03-11T21:17:18.754 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:18 smithi146 bash[23809]: cluster 2024-03-11T21:17:16.768831+0000 osd.0 (osd.0) 443 : cluster 0 purged_snaps scrub ok 2024-03-11T21:17:18.754 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:18 smithi146 bash[23809]: cluster 2024-03-11T21:17:17.090949+0000 osd.5 (osd.5) 603 : cluster 0 3.1c scrub starts 2024-03-11T21:17:18.754 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:18 smithi146 bash[23809]: cluster 2024-03-11T21:17:17.095959+0000 osd.5 (osd.5) 604 : cluster 0 3.1c scrub ok 2024-03-11T21:17:18.754 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:18 smithi146 bash[23809]: cluster 2024-03-11T21:17:17.255047+0000 osd.4 (osd.4) 121 : cluster 0 purged_snaps scrub starts 2024-03-11T21:17:18.754 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:18 smithi146 bash[23809]: cluster 2024-03-11T21:17:17.255194+0000 osd.4 (osd.4) 122 : cluster 0 purged_snaps scrub ok 2024-03-11T21:17:18.754 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:18 smithi146 bash[23809]: cluster 2024-03-11T21:17:17.495272+0000 mon.a (mon.0) 2034 : cluster 0 osdmap e464: 8 total, 8 up, 8 in 2024-03-11T21:17:18.754 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:18 smithi146 bash[23809]: audit 2024-03-11T21:17:17.672046+0000 mon.c (mon.2) 108 : audit 1 from='client.? 172.21.15.23:0/3915556913' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-03-11T21:17:18.754 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:18 smithi146 bash[23809]: audit 2024-03-11T21:17:17.672761+0000 mon.a (mon.0) 2035 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-03-11T21:17:19.018 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-11T21:17:19.379 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd reweight 3 1 2024-03-11T21:17:19.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:19 smithi023 bash[21372]: cluster 2024-03-11T21:17:18.206071+0000 osd.7 (osd.7) 625 : cluster 0 2.8 scrub starts 2024-03-11T21:17:19.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:19 smithi023 bash[21372]: cluster 2024-03-11T21:17:18.208227+0000 osd.7 (osd.7) 626 : cluster 0 2.8 scrub ok 2024-03-11T21:17:19.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:19 smithi023 bash[21372]: cluster 2024-03-11T21:17:18.277153+0000 osd.0 (osd.0) 444 : cluster 0 3.1b scrub starts 2024-03-11T21:17:19.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:19 smithi023 bash[21372]: cluster 2024-03-11T21:17:18.280228+0000 osd.0 (osd.0) 445 : cluster 0 3.1b scrub ok 2024-03-11T21:17:19.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:19 smithi023 bash[21372]: audit 2024-03-11T21:17:18.491815+0000 mon.a (mon.0) 2036 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-03-11T21:17:19.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:19 smithi023 bash[21372]: cluster 2024-03-11T21:17:18.499428+0000 mon.a (mon.0) 2037 : cluster 0 osdmap e465: 8 total, 8 up, 8 in 2024-03-11T21:17:19.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:19 smithi023 bash[21372]: cluster 2024-03-11T21:17:18.507291+0000 mgr.y (mgr.24380) 979 : cluster 0 pgmap v1139: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-11T21:17:19.776 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:19 smithi023 bash[21372]: cluster 2024-03-11T21:17:19.504712+0000 mon.a (mon.0) 2038 : cluster 0 osdmap e466: 8 total, 8 up, 8 in 2024-03-11T21:17:19.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:19 smithi023 bash[29146]: cluster 2024-03-11T21:17:18.206071+0000 osd.7 (osd.7) 625 : cluster 0 2.8 scrub starts 2024-03-11T21:17:19.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:19 smithi023 bash[29146]: cluster 2024-03-11T21:17:18.208227+0000 osd.7 (osd.7) 626 : cluster 0 2.8 scrub ok 2024-03-11T21:17:19.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:19 smithi023 bash[29146]: cluster 2024-03-11T21:17:18.277153+0000 osd.0 (osd.0) 444 : cluster 0 3.1b scrub starts 2024-03-11T21:17:19.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:19 smithi023 bash[29146]: cluster 2024-03-11T21:17:18.280228+0000 osd.0 (osd.0) 445 : cluster 0 3.1b scrub ok 2024-03-11T21:17:19.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:19 smithi023 bash[29146]: audit 2024-03-11T21:17:18.491815+0000 mon.a (mon.0) 2036 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-03-11T21:17:19.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:19 smithi023 bash[29146]: cluster 2024-03-11T21:17:18.499428+0000 mon.a (mon.0) 2037 : cluster 0 osdmap e465: 8 total, 8 up, 8 in 2024-03-11T21:17:19.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:19 smithi023 bash[29146]: cluster 2024-03-11T21:17:18.507291+0000 mgr.y (mgr.24380) 979 : cluster 0 pgmap v1139: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-11T21:17:19.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:19 smithi023 bash[29146]: cluster 2024-03-11T21:17:19.504712+0000 mon.a (mon.0) 2038 : cluster 0 osdmap e466: 8 total, 8 up, 8 in 2024-03-11T21:17:20.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:19 smithi146 bash[23809]: cluster 2024-03-11T21:17:18.206071+0000 osd.7 (osd.7) 625 : cluster 0 2.8 scrub starts 2024-03-11T21:17:20.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:19 smithi146 bash[23809]: cluster 2024-03-11T21:17:18.208227+0000 osd.7 (osd.7) 626 : cluster 0 2.8 scrub ok 2024-03-11T21:17:20.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:19 smithi146 bash[23809]: cluster 2024-03-11T21:17:18.277153+0000 osd.0 (osd.0) 444 : cluster 0 3.1b scrub starts 2024-03-11T21:17:20.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:19 smithi146 bash[23809]: cluster 2024-03-11T21:17:18.280228+0000 osd.0 (osd.0) 445 : cluster 0 3.1b scrub ok 2024-03-11T21:17:20.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:19 smithi146 bash[23809]: audit 2024-03-11T21:17:18.491815+0000 mon.a (mon.0) 2036 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-03-11T21:17:20.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:19 smithi146 bash[23809]: cluster 2024-03-11T21:17:18.499428+0000 mon.a (mon.0) 2037 : cluster 0 osdmap e465: 8 total, 8 up, 8 in 2024-03-11T21:17:20.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:19 smithi146 bash[23809]: cluster 2024-03-11T21:17:18.507291+0000 mgr.y (mgr.24380) 979 : cluster 0 pgmap v1139: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-11T21:17:20.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:19 smithi146 bash[23809]: cluster 2024-03-11T21:17:19.504712+0000 mon.a (mon.0) 2038 : cluster 0 osdmap e466: 8 total, 8 up, 8 in 2024-03-11T21:17:20.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:20 smithi023 bash[21372]: cluster 2024-03-11T21:17:19.303849+0000 osd.0 (osd.0) 446 : cluster 0 4.3 scrub starts 2024-03-11T21:17:20.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:20 smithi023 bash[21372]: cluster 2024-03-11T21:17:19.307148+0000 osd.0 (osd.0) 447 : cluster 0 4.3 scrub ok 2024-03-11T21:17:20.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:20 smithi023 bash[21372]: cluster 2024-03-11T21:17:19.888146+0000 osd.1 (osd.1) 362 : cluster 0 6.1 scrub starts 2024-03-11T21:17:20.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:20 smithi023 bash[21372]: cluster 2024-03-11T21:17:19.970515+0000 osd.1 (osd.1) 363 : cluster 0 6.1 scrub ok 2024-03-11T21:17:20.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:20 smithi023 bash[29146]: cluster 2024-03-11T21:17:19.303849+0000 osd.0 (osd.0) 446 : cluster 0 4.3 scrub starts 2024-03-11T21:17:20.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:20 smithi023 bash[29146]: cluster 2024-03-11T21:17:19.307148+0000 osd.0 (osd.0) 447 : cluster 0 4.3 scrub ok 2024-03-11T21:17:20.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:20 smithi023 bash[29146]: cluster 2024-03-11T21:17:19.888146+0000 osd.1 (osd.1) 362 : cluster 0 6.1 scrub starts 2024-03-11T21:17:20.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:20 smithi023 bash[29146]: cluster 2024-03-11T21:17:19.970515+0000 osd.1 (osd.1) 363 : cluster 0 6.1 scrub ok 2024-03-11T21:17:21.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:20 smithi146 bash[23809]: cluster 2024-03-11T21:17:19.303849+0000 osd.0 (osd.0) 446 : cluster 0 4.3 scrub starts 2024-03-11T21:17:21.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:20 smithi146 bash[23809]: cluster 2024-03-11T21:17:19.307148+0000 osd.0 (osd.0) 447 : cluster 0 4.3 scrub ok 2024-03-11T21:17:21.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:20 smithi146 bash[23809]: cluster 2024-03-11T21:17:19.888146+0000 osd.1 (osd.1) 362 : cluster 0 6.1 scrub starts 2024-03-11T21:17:21.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:20 smithi146 bash[23809]: cluster 2024-03-11T21:17:19.970515+0000 osd.1 (osd.1) 363 : cluster 0 6.1 scrub ok 2024-03-11T21:17:21.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:21 smithi023 bash[21372]: cluster 2024-03-11T21:17:20.269804+0000 osd.2 (osd.2) 265 : cluster 0 1.14 scrub starts 2024-03-11T21:17:21.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:21 smithi023 bash[21372]: cluster 2024-03-11T21:17:20.272772+0000 osd.2 (osd.2) 266 : cluster 0 1.14 scrub ok 2024-03-11T21:17:21.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:21 smithi023 bash[21372]: cluster 2024-03-11T21:17:20.508296+0000 mgr.y (mgr.24380) 980 : cluster 0 pgmap v1141: 152 pgs: 152 active+clean; 626 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 5.4 KiB/s rd, 45 KiB/s wr, 6 op/s; 32 KiB/s, 0 objects/s recovering 2024-03-11T21:17:21.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:21 smithi023 bash[29146]: cluster 2024-03-11T21:17:20.269804+0000 osd.2 (osd.2) 265 : cluster 0 1.14 scrub starts 2024-03-11T21:17:21.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:21 smithi023 bash[29146]: cluster 2024-03-11T21:17:20.272772+0000 osd.2 (osd.2) 266 : cluster 0 1.14 scrub ok 2024-03-11T21:17:21.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:21 smithi023 bash[29146]: cluster 2024-03-11T21:17:20.508296+0000 mgr.y (mgr.24380) 980 : cluster 0 pgmap v1141: 152 pgs: 152 active+clean; 626 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 5.4 KiB/s rd, 45 KiB/s wr, 6 op/s; 32 KiB/s, 0 objects/s recovering 2024-03-11T21:17:22.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:21 smithi146 bash[23809]: cluster 2024-03-11T21:17:20.269804+0000 osd.2 (osd.2) 265 : cluster 0 1.14 scrub starts 2024-03-11T21:17:22.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:21 smithi146 bash[23809]: cluster 2024-03-11T21:17:20.272772+0000 osd.2 (osd.2) 266 : cluster 0 1.14 scrub ok 2024-03-11T21:17:22.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:21 smithi146 bash[23809]: cluster 2024-03-11T21:17:20.508296+0000 mgr.y (mgr.24380) 980 : cluster 0 pgmap v1141: 152 pgs: 152 active+clean; 626 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 5.4 KiB/s rd, 45 KiB/s wr, 6 op/s; 32 KiB/s, 0 objects/s recovering 2024-03-11T21:17:23.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:22 smithi146 bash[23809]: cluster 2024-03-11T21:17:21.182687+0000 osd.7 (osd.7) 627 : cluster 0 6.12 scrub starts 2024-03-11T21:17:23.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:22 smithi146 bash[23809]: cluster 2024-03-11T21:17:21.215918+0000 osd.7 (osd.7) 628 : cluster 0 6.12 scrub ok 2024-03-11T21:17:23.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:22 smithi146 bash[23809]: cluster 2024-03-11T21:17:21.305868+0000 osd.2 (osd.2) 267 : cluster 0 1.5 scrub starts 2024-03-11T21:17:23.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:22 smithi146 bash[23809]: cluster 2024-03-11T21:17:21.309177+0000 osd.2 (osd.2) 268 : cluster 0 1.5 scrub ok 2024-03-11T21:17:23.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:22 smithi146 bash[23809]: cluster 2024-03-11T21:17:21.338294+0000 osd.0 (osd.0) 448 : cluster 0 2.1f scrub starts 2024-03-11T21:17:23.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:22 smithi146 bash[23809]: cluster 2024-03-11T21:17:21.340889+0000 osd.0 (osd.0) 449 : cluster 0 2.1f scrub ok 2024-03-11T21:17:23.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:22 smithi146 bash[23809]: audit 2024-03-11T21:17:21.817047+0000 mon.a (mon.0) 2039 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:17:23.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:22 smithi023 bash[21372]: cluster 2024-03-11T21:17:21.182687+0000 osd.7 (osd.7) 627 : cluster 0 6.12 scrub starts 2024-03-11T21:17:23.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:22 smithi023 bash[21372]: cluster 2024-03-11T21:17:21.215918+0000 osd.7 (osd.7) 628 : cluster 0 6.12 scrub ok 2024-03-11T21:17:23.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:22 smithi023 bash[21372]: cluster 2024-03-11T21:17:21.305868+0000 osd.2 (osd.2) 267 : cluster 0 1.5 scrub starts 2024-03-11T21:17:23.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:22 smithi023 bash[21372]: cluster 2024-03-11T21:17:21.309177+0000 osd.2 (osd.2) 268 : cluster 0 1.5 scrub ok 2024-03-11T21:17:23.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:22 smithi023 bash[21372]: cluster 2024-03-11T21:17:21.338294+0000 osd.0 (osd.0) 448 : cluster 0 2.1f scrub starts 2024-03-11T21:17:23.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:22 smithi023 bash[21372]: cluster 2024-03-11T21:17:21.340889+0000 osd.0 (osd.0) 449 : cluster 0 2.1f scrub ok 2024-03-11T21:17:23.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:22 smithi023 bash[21372]: audit 2024-03-11T21:17:21.817047+0000 mon.a (mon.0) 2039 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:17:23.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:22 smithi023 bash[29146]: cluster 2024-03-11T21:17:21.182687+0000 osd.7 (osd.7) 627 : cluster 0 6.12 scrub starts 2024-03-11T21:17:23.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:22 smithi023 bash[29146]: cluster 2024-03-11T21:17:21.215918+0000 osd.7 (osd.7) 628 : cluster 0 6.12 scrub ok 2024-03-11T21:17:23.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:22 smithi023 bash[29146]: cluster 2024-03-11T21:17:21.305868+0000 osd.2 (osd.2) 267 : cluster 0 1.5 scrub starts 2024-03-11T21:17:23.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:22 smithi023 bash[29146]: cluster 2024-03-11T21:17:21.309177+0000 osd.2 (osd.2) 268 : cluster 0 1.5 scrub ok 2024-03-11T21:17:23.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:22 smithi023 bash[29146]: cluster 2024-03-11T21:17:21.338294+0000 osd.0 (osd.0) 448 : cluster 0 2.1f scrub starts 2024-03-11T21:17:23.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:22 smithi023 bash[29146]: cluster 2024-03-11T21:17:21.340889+0000 osd.0 (osd.0) 449 : cluster 0 2.1f scrub ok 2024-03-11T21:17:23.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:22 smithi023 bash[29146]: audit 2024-03-11T21:17:21.817047+0000 mon.a (mon.0) 2039 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:17:23.814 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:17:23.836 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:23 smithi146 bash[23809]: cluster 2024-03-11T21:17:22.113389+0000 osd.5 (osd.5) 605 : cluster 0 3.1d scrub starts 2024-03-11T21:17:23.836 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:23 smithi146 bash[23809]: cluster 2024-03-11T21:17:22.117840+0000 osd.5 (osd.5) 606 : cluster 0 3.1d scrub ok 2024-03-11T21:17:23.836 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:23 smithi146 bash[23809]: cluster 2024-03-11T21:17:22.509691+0000 mgr.y (mgr.24380) 981 : cluster 0 pgmap v1142: 152 pgs: 152 active+clean; 626 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 4.5 KiB/s rd, 30 KiB/s wr, 5 op/s; 22 KiB/s, 0 objects/s recovering 2024-03-11T21:17:24.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:23 smithi023 bash[21372]: cluster 2024-03-11T21:17:22.113389+0000 osd.5 (osd.5) 605 : cluster 0 3.1d scrub starts 2024-03-11T21:17:24.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:23 smithi023 bash[21372]: cluster 2024-03-11T21:17:22.117840+0000 osd.5 (osd.5) 606 : cluster 0 3.1d scrub ok 2024-03-11T21:17:24.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:23 smithi023 bash[21372]: cluster 2024-03-11T21:17:22.509691+0000 mgr.y (mgr.24380) 981 : cluster 0 pgmap v1142: 152 pgs: 152 active+clean; 626 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 4.5 KiB/s rd, 30 KiB/s wr, 5 op/s; 22 KiB/s, 0 objects/s recovering 2024-03-11T21:17:24.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:23 smithi023 bash[29146]: cluster 2024-03-11T21:17:22.113389+0000 osd.5 (osd.5) 605 : cluster 0 3.1d scrub starts 2024-03-11T21:17:24.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:23 smithi023 bash[29146]: cluster 2024-03-11T21:17:22.117840+0000 osd.5 (osd.5) 606 : cluster 0 3.1d scrub ok 2024-03-11T21:17:24.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:23 smithi023 bash[29146]: cluster 2024-03-11T21:17:22.509691+0000 mgr.y (mgr.24380) 981 : cluster 0 pgmap v1142: 152 pgs: 152 active+clean; 626 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 4.5 KiB/s rd, 30 KiB/s wr, 5 op/s; 22 KiB/s, 0 objects/s recovering 2024-03-11T21:17:24.186 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:17:24.971 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:24 smithi146 bash[23809]: cluster 2024-03-11T21:17:23.361730+0000 osd.0 (osd.0) 450 : cluster 0 3.6 scrub starts 2024-03-11T21:17:24.971 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:24 smithi146 bash[23809]: cluster 2024-03-11T21:17:23.366116+0000 osd.0 (osd.0) 451 : cluster 0 3.6 scrub ok 2024-03-11T21:17:25.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:24 smithi023 bash[21372]: cluster 2024-03-11T21:17:23.361730+0000 osd.0 (osd.0) 450 : cluster 0 3.6 scrub starts 2024-03-11T21:17:25.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:24 smithi023 bash[21372]: cluster 2024-03-11T21:17:23.366116+0000 osd.0 (osd.0) 451 : cluster 0 3.6 scrub ok 2024-03-11T21:17:25.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:24 smithi023 bash[29146]: cluster 2024-03-11T21:17:23.361730+0000 osd.0 (osd.0) 450 : cluster 0 3.6 scrub starts 2024-03-11T21:17:25.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:24 smithi023 bash[29146]: cluster 2024-03-11T21:17:23.366116+0000 osd.0 (osd.0) 451 : cluster 0 3.6 scrub ok 2024-03-11T21:17:25.753 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:25 smithi146 bash[23809]: cluster 2024-03-11T21:17:24.361948+0000 osd.0 (osd.0) 452 : cluster 0 6.9 scrub starts 2024-03-11T21:17:25.753 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:25 smithi146 bash[23809]: cluster 2024-03-11T21:17:24.500345+0000 osd.0 (osd.0) 453 : cluster 0 6.9 scrub ok 2024-03-11T21:17:25.753 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:25 smithi146 bash[23809]: cluster 2024-03-11T21:17:24.510598+0000 mgr.y (mgr.24380) 982 : cluster 0 pgmap v1143: 152 pgs: 152 active+clean; 626 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 3.8 KiB/s rd, 26 KiB/s wr, 4 op/s; 19 KiB/s, 0 objects/s recovering 2024-03-11T21:17:26.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:25 smithi023 bash[21372]: cluster 2024-03-11T21:17:24.361948+0000 osd.0 (osd.0) 452 : cluster 0 6.9 scrub starts 2024-03-11T21:17:26.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:25 smithi023 bash[21372]: cluster 2024-03-11T21:17:24.500345+0000 osd.0 (osd.0) 453 : cluster 0 6.9 scrub ok 2024-03-11T21:17:26.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:25 smithi023 bash[21372]: cluster 2024-03-11T21:17:24.510598+0000 mgr.y (mgr.24380) 982 : cluster 0 pgmap v1143: 152 pgs: 152 active+clean; 626 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 3.8 KiB/s rd, 26 KiB/s wr, 4 op/s; 19 KiB/s, 0 objects/s recovering 2024-03-11T21:17:26.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:25 smithi023 bash[29146]: cluster 2024-03-11T21:17:24.361948+0000 osd.0 (osd.0) 452 : cluster 0 6.9 scrub starts 2024-03-11T21:17:26.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:25 smithi023 bash[29146]: cluster 2024-03-11T21:17:24.500345+0000 osd.0 (osd.0) 453 : cluster 0 6.9 scrub ok 2024-03-11T21:17:26.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:25 smithi023 bash[29146]: cluster 2024-03-11T21:17:24.510598+0000 mgr.y (mgr.24380) 982 : cluster 0 pgmap v1143: 152 pgs: 152 active+clean; 626 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 3.8 KiB/s rd, 26 KiB/s wr, 4 op/s; 19 KiB/s, 0 objects/s recovering 2024-03-11T21:17:26.503 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:17:26 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:17:26.969 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:26 smithi023 bash[21372]: cluster 2024-03-11T21:17:25.487392+0000 osd.3 (osd.3) 554 : cluster 0 3.1e scrub starts 2024-03-11T21:17:26.970 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:26 smithi023 bash[21372]: cluster 2024-03-11T21:17:25.489983+0000 osd.3 (osd.3) 555 : cluster 0 3.1e scrub ok 2024-03-11T21:17:26.970 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:26 smithi023 bash[29146]: cluster 2024-03-11T21:17:25.487392+0000 osd.3 (osd.3) 554 : cluster 0 3.1e scrub starts 2024-03-11T21:17:26.970 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:26 smithi023 bash[29146]: cluster 2024-03-11T21:17:25.489983+0000 osd.3 (osd.3) 555 : cluster 0 3.1e scrub ok 2024-03-11T21:17:27.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:26 smithi146 bash[23809]: cluster 2024-03-11T21:17:25.487392+0000 osd.3 (osd.3) 554 : cluster 0 3.1e scrub starts 2024-03-11T21:17:27.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:26 smithi146 bash[23809]: cluster 2024-03-11T21:17:25.489983+0000 osd.3 (osd.3) 555 : cluster 0 3.1e scrub ok 2024-03-11T21:17:27.231 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-03-11T21:17:27.231 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:17:27.602 INFO:teuthology.orchestra.run.smithi023.stderr:reweighted osd.3 to 1 (10000) 2024-03-11T21:17:27.753 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:27 smithi146 bash[23809]: audit 2024-03-11T21:17:26.203480+0000 mgr.y (mgr.24380) 983 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:17:27.754 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:27 smithi146 bash[23809]: cluster 2024-03-11T21:17:26.388241+0000 osd.2 (osd.2) 269 : cluster 0 2.e scrub starts 2024-03-11T21:17:27.754 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:27 smithi146 bash[23809]: cluster 2024-03-11T21:17:26.390477+0000 osd.2 (osd.2) 270 : cluster 0 2.e scrub ok 2024-03-11T21:17:27.754 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:27 smithi146 bash[23809]: cluster 2024-03-11T21:17:26.512137+0000 mgr.y (mgr.24380) 984 : cluster 0 pgmap v1144: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T21:17:27.754 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:27 smithi146 bash[23809]: cluster 2024-03-11T21:17:26.710654+0000 osd.6 (osd.6) 500 : cluster 0 4.e scrub starts 2024-03-11T21:17:27.754 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:27 smithi146 bash[23809]: cluster 2024-03-11T21:17:26.712987+0000 osd.6 (osd.6) 501 : cluster 0 4.e scrub ok 2024-03-11T21:17:27.754 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:27 smithi146 bash[23809]: audit 2024-03-11T21:17:27.382106+0000 mon.a (mon.0) 2040 : audit 1 from='client.? 172.21.15.23:0/317400761' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-11T21:17:28.012 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-03-11T21:17:28.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:27 smithi023 bash[29146]: audit 2024-03-11T21:17:26.203480+0000 mgr.y (mgr.24380) 983 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:17:28.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:27 smithi023 bash[29146]: cluster 2024-03-11T21:17:26.388241+0000 osd.2 (osd.2) 269 : cluster 0 2.e scrub starts 2024-03-11T21:17:28.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:27 smithi023 bash[29146]: cluster 2024-03-11T21:17:26.390477+0000 osd.2 (osd.2) 270 : cluster 0 2.e scrub ok 2024-03-11T21:17:28.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:27 smithi023 bash[29146]: cluster 2024-03-11T21:17:26.512137+0000 mgr.y (mgr.24380) 984 : cluster 0 pgmap v1144: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T21:17:28.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:27 smithi023 bash[29146]: cluster 2024-03-11T21:17:26.710654+0000 osd.6 (osd.6) 500 : cluster 0 4.e scrub starts 2024-03-11T21:17:28.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:27 smithi023 bash[29146]: cluster 2024-03-11T21:17:26.712987+0000 osd.6 (osd.6) 501 : cluster 0 4.e scrub ok 2024-03-11T21:17:28.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:27 smithi023 bash[29146]: audit 2024-03-11T21:17:27.382106+0000 mon.a (mon.0) 2040 : audit 1 from='client.? 172.21.15.23:0/317400761' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-11T21:17:28.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:27 smithi023 bash[21372]: audit 2024-03-11T21:17:26.203480+0000 mgr.y (mgr.24380) 983 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:17:28.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:27 smithi023 bash[21372]: cluster 2024-03-11T21:17:26.388241+0000 osd.2 (osd.2) 269 : cluster 0 2.e scrub starts 2024-03-11T21:17:28.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:27 smithi023 bash[21372]: cluster 2024-03-11T21:17:26.390477+0000 osd.2 (osd.2) 270 : cluster 0 2.e scrub ok 2024-03-11T21:17:28.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:27 smithi023 bash[21372]: cluster 2024-03-11T21:17:26.512137+0000 mgr.y (mgr.24380) 984 : cluster 0 pgmap v1144: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T21:17:28.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:27 smithi023 bash[21372]: cluster 2024-03-11T21:17:26.710654+0000 osd.6 (osd.6) 500 : cluster 0 4.e scrub starts 2024-03-11T21:17:28.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:27 smithi023 bash[21372]: cluster 2024-03-11T21:17:26.712987+0000 osd.6 (osd.6) 501 : cluster 0 4.e scrub ok 2024-03-11T21:17:28.026 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:27 smithi023 bash[21372]: audit 2024-03-11T21:17:27.382106+0000 mon.a (mon.0) 2040 : audit 1 from='client.? 172.21.15.23:0/317400761' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-11T21:17:28.447 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd primary-affinity 3 1 2024-03-11T21:17:28.623 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:17:28 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:17:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:17:28.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:28 smithi023 bash[21372]: cluster 2024-03-11T21:17:27.123967+0000 osd.7 (osd.7) 629 : cluster 0 3.f scrub starts 2024-03-11T21:17:28.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:28 smithi023 bash[21372]: cluster 2024-03-11T21:17:27.125900+0000 osd.7 (osd.7) 630 : cluster 0 3.f scrub ok 2024-03-11T21:17:28.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:28 smithi023 bash[21372]: cluster 2024-03-11T21:17:27.346234+0000 osd.2 (osd.2) 271 : cluster 0 3.9 deep-scrub starts 2024-03-11T21:17:28.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:28 smithi023 bash[21372]: cluster 2024-03-11T21:17:27.355087+0000 osd.2 (osd.2) 272 : cluster 0 3.9 deep-scrub ok 2024-03-11T21:17:28.776 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:28 smithi023 bash[21372]: cluster 2024-03-11T21:17:27.513611+0000 osd.3 (osd.3) 556 : cluster 0 6.13 scrub starts 2024-03-11T21:17:28.776 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:28 smithi023 bash[21372]: audit 2024-03-11T21:17:27.605168+0000 mon.a (mon.0) 2041 : audit 1 from='client.? 172.21.15.23:0/317400761' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-11T21:17:28.776 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:28 smithi023 bash[21372]: cluster 2024-03-11T21:17:27.612130+0000 mon.a (mon.0) 2042 : cluster 0 osdmap e467: 8 total, 8 up, 8 in 2024-03-11T21:17:28.776 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:28 smithi023 bash[21372]: cluster 2024-03-11T21:17:27.672923+0000 osd.3 (osd.3) 557 : cluster 0 6.13 scrub ok 2024-03-11T21:17:28.776 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:28 smithi023 bash[21372]: cluster 2024-03-11T21:17:27.869514+0000 osd.1 (osd.1) 364 : cluster 0 2.15 scrub starts 2024-03-11T21:17:28.776 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:28 smithi023 bash[21372]: cluster 2024-03-11T21:17:27.871942+0000 osd.1 (osd.1) 365 : cluster 0 2.15 scrub ok 2024-03-11T21:17:28.776 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:28 smithi023 bash[21372]: cluster 2024-03-11T21:17:28.405953+0000 osd.0 (osd.0) 454 : cluster 0 6.18 scrub starts 2024-03-11T21:17:28.776 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:28 smithi023 bash[21372]: cluster 2024-03-11T21:17:28.491511+0000 osd.0 (osd.0) 455 : cluster 0 6.18 scrub ok 2024-03-11T21:17:28.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:28 smithi023 bash[29146]: cluster 2024-03-11T21:17:27.123967+0000 osd.7 (osd.7) 629 : cluster 0 3.f scrub starts 2024-03-11T21:17:28.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:28 smithi023 bash[29146]: cluster 2024-03-11T21:17:27.125900+0000 osd.7 (osd.7) 630 : cluster 0 3.f scrub ok 2024-03-11T21:17:28.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:28 smithi023 bash[29146]: cluster 2024-03-11T21:17:27.346234+0000 osd.2 (osd.2) 271 : cluster 0 3.9 deep-scrub starts 2024-03-11T21:17:28.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:28 smithi023 bash[29146]: cluster 2024-03-11T21:17:27.355087+0000 osd.2 (osd.2) 272 : cluster 0 3.9 deep-scrub ok 2024-03-11T21:17:28.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:28 smithi023 bash[29146]: cluster 2024-03-11T21:17:27.513611+0000 osd.3 (osd.3) 556 : cluster 0 6.13 scrub starts 2024-03-11T21:17:28.777 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:28 smithi023 bash[29146]: audit 2024-03-11T21:17:27.605168+0000 mon.a (mon.0) 2041 : audit 1 from='client.? 172.21.15.23:0/317400761' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-11T21:17:28.777 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:28 smithi023 bash[29146]: cluster 2024-03-11T21:17:27.612130+0000 mon.a (mon.0) 2042 : cluster 0 osdmap e467: 8 total, 8 up, 8 in 2024-03-11T21:17:28.777 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:28 smithi023 bash[29146]: cluster 2024-03-11T21:17:27.672923+0000 osd.3 (osd.3) 557 : cluster 0 6.13 scrub ok 2024-03-11T21:17:28.777 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:28 smithi023 bash[29146]: cluster 2024-03-11T21:17:27.869514+0000 osd.1 (osd.1) 364 : cluster 0 2.15 scrub starts 2024-03-11T21:17:28.777 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:28 smithi023 bash[29146]: cluster 2024-03-11T21:17:27.871942+0000 osd.1 (osd.1) 365 : cluster 0 2.15 scrub ok 2024-03-11T21:17:28.777 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:28 smithi023 bash[29146]: cluster 2024-03-11T21:17:28.405953+0000 osd.0 (osd.0) 454 : cluster 0 6.18 scrub starts 2024-03-11T21:17:28.777 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:28 smithi023 bash[29146]: cluster 2024-03-11T21:17:28.491511+0000 osd.0 (osd.0) 455 : cluster 0 6.18 scrub ok 2024-03-11T21:17:29.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:28 smithi146 bash[23809]: cluster 2024-03-11T21:17:27.123967+0000 osd.7 (osd.7) 629 : cluster 0 3.f scrub starts 2024-03-11T21:17:29.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:28 smithi146 bash[23809]: cluster 2024-03-11T21:17:27.125900+0000 osd.7 (osd.7) 630 : cluster 0 3.f scrub ok 2024-03-11T21:17:29.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:28 smithi146 bash[23809]: cluster 2024-03-11T21:17:27.346234+0000 osd.2 (osd.2) 271 : cluster 0 3.9 deep-scrub starts 2024-03-11T21:17:29.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:28 smithi146 bash[23809]: cluster 2024-03-11T21:17:27.355087+0000 osd.2 (osd.2) 272 : cluster 0 3.9 deep-scrub ok 2024-03-11T21:17:29.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:28 smithi146 bash[23809]: cluster 2024-03-11T21:17:27.513611+0000 osd.3 (osd.3) 556 : cluster 0 6.13 scrub starts 2024-03-11T21:17:29.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:28 smithi146 bash[23809]: audit 2024-03-11T21:17:27.605168+0000 mon.a (mon.0) 2041 : audit 1 from='client.? 172.21.15.23:0/317400761' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-11T21:17:29.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:28 smithi146 bash[23809]: cluster 2024-03-11T21:17:27.612130+0000 mon.a (mon.0) 2042 : cluster 0 osdmap e467: 8 total, 8 up, 8 in 2024-03-11T21:17:29.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:28 smithi146 bash[23809]: cluster 2024-03-11T21:17:27.672923+0000 osd.3 (osd.3) 557 : cluster 0 6.13 scrub ok 2024-03-11T21:17:29.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:28 smithi146 bash[23809]: cluster 2024-03-11T21:17:27.869514+0000 osd.1 (osd.1) 364 : cluster 0 2.15 scrub starts 2024-03-11T21:17:29.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:28 smithi146 bash[23809]: cluster 2024-03-11T21:17:27.871942+0000 osd.1 (osd.1) 365 : cluster 0 2.15 scrub ok 2024-03-11T21:17:29.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:28 smithi146 bash[23809]: cluster 2024-03-11T21:17:28.405953+0000 osd.0 (osd.0) 454 : cluster 0 6.18 scrub starts 2024-03-11T21:17:29.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:28 smithi146 bash[23809]: cluster 2024-03-11T21:17:28.491511+0000 osd.0 (osd.0) 455 : cluster 0 6.18 scrub ok 2024-03-11T21:17:30.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:29 smithi146 bash[23809]: cluster 2024-03-11T21:17:28.512733+0000 mgr.y (mgr.24380) 985 : cluster 0 pgmap v1146: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1022 B/s rd, 0 op/s 2024-03-11T21:17:30.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:29 smithi146 bash[23809]: cluster 2024-03-11T21:17:28.686270+0000 osd.6 (osd.6) 502 : cluster 0 4.5 scrub starts 2024-03-11T21:17:30.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:29 smithi146 bash[23809]: cluster 2024-03-11T21:17:28.688728+0000 osd.6 (osd.6) 503 : cluster 0 4.5 scrub ok 2024-03-11T21:17:30.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:29 smithi146 bash[23809]: cluster 2024-03-11T21:17:28.881660+0000 osd.1 (osd.1) 366 : cluster 0 2.6 scrub starts 2024-03-11T21:17:30.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:29 smithi146 bash[23809]: cluster 2024-03-11T21:17:28.885203+0000 osd.1 (osd.1) 367 : cluster 0 2.6 scrub ok 2024-03-11T21:17:30.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:29 smithi023 bash[21372]: cluster 2024-03-11T21:17:28.512733+0000 mgr.y (mgr.24380) 985 : cluster 0 pgmap v1146: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1022 B/s rd, 0 op/s 2024-03-11T21:17:30.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:29 smithi023 bash[21372]: cluster 2024-03-11T21:17:28.686270+0000 osd.6 (osd.6) 502 : cluster 0 4.5 scrub starts 2024-03-11T21:17:30.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:29 smithi023 bash[21372]: cluster 2024-03-11T21:17:28.688728+0000 osd.6 (osd.6) 503 : cluster 0 4.5 scrub ok 2024-03-11T21:17:30.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:29 smithi023 bash[21372]: cluster 2024-03-11T21:17:28.881660+0000 osd.1 (osd.1) 366 : cluster 0 2.6 scrub starts 2024-03-11T21:17:30.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:29 smithi023 bash[21372]: cluster 2024-03-11T21:17:28.885203+0000 osd.1 (osd.1) 367 : cluster 0 2.6 scrub ok 2024-03-11T21:17:30.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:29 smithi023 bash[29146]: cluster 2024-03-11T21:17:28.512733+0000 mgr.y (mgr.24380) 985 : cluster 0 pgmap v1146: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1022 B/s rd, 0 op/s 2024-03-11T21:17:30.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:29 smithi023 bash[29146]: cluster 2024-03-11T21:17:28.686270+0000 osd.6 (osd.6) 502 : cluster 0 4.5 scrub starts 2024-03-11T21:17:30.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:29 smithi023 bash[29146]: cluster 2024-03-11T21:17:28.688728+0000 osd.6 (osd.6) 503 : cluster 0 4.5 scrub ok 2024-03-11T21:17:30.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:29 smithi023 bash[29146]: cluster 2024-03-11T21:17:28.881660+0000 osd.1 (osd.1) 366 : cluster 0 2.6 scrub starts 2024-03-11T21:17:30.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:29 smithi023 bash[29146]: cluster 2024-03-11T21:17:28.885203+0000 osd.1 (osd.1) 367 : cluster 0 2.6 scrub ok 2024-03-11T21:17:31.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:30 smithi146 bash[23809]: cluster 2024-03-11T21:17:29.123863+0000 osd.7 (osd.7) 631 : cluster 0 6.15 deep-scrub starts 2024-03-11T21:17:31.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:30 smithi146 bash[23809]: cluster 2024-03-11T21:17:29.228412+0000 osd.7 (osd.7) 632 : cluster 0 6.15 deep-scrub ok 2024-03-11T21:17:31.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:30 smithi146 bash[23809]: cluster 2024-03-11T21:17:29.391039+0000 osd.2 (osd.2) 273 : cluster 0 4.d deep-scrub starts 2024-03-11T21:17:31.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:30 smithi146 bash[23809]: cluster 2024-03-11T21:17:29.394601+0000 osd.2 (osd.2) 274 : cluster 0 4.d deep-scrub ok 2024-03-11T21:17:31.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:30 smithi146 bash[23809]: cluster 2024-03-11T21:17:29.895867+0000 osd.1 (osd.1) 368 : cluster 0 4.4 scrub starts 2024-03-11T21:17:31.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:30 smithi146 bash[23809]: cluster 2024-03-11T21:17:29.898431+0000 osd.1 (osd.1) 369 : cluster 0 4.4 scrub ok 2024-03-11T21:17:31.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:30 smithi146 bash[23809]: cluster 2024-03-11T21:17:30.474914+0000 mon.a (mon.0) 2043 : cluster 0 osdmap e468: 8 total, 8 up, 8 in 2024-03-11T21:17:31.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:30 smithi023 bash[21372]: cluster 2024-03-11T21:17:29.123863+0000 osd.7 (osd.7) 631 : cluster 0 6.15 deep-scrub starts 2024-03-11T21:17:31.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:30 smithi023 bash[21372]: cluster 2024-03-11T21:17:29.228412+0000 osd.7 (osd.7) 632 : cluster 0 6.15 deep-scrub ok 2024-03-11T21:17:31.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:30 smithi023 bash[21372]: cluster 2024-03-11T21:17:29.391039+0000 osd.2 (osd.2) 273 : cluster 0 4.d deep-scrub starts 2024-03-11T21:17:31.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:30 smithi023 bash[21372]: cluster 2024-03-11T21:17:29.394601+0000 osd.2 (osd.2) 274 : cluster 0 4.d deep-scrub ok 2024-03-11T21:17:31.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:30 smithi023 bash[21372]: cluster 2024-03-11T21:17:29.895867+0000 osd.1 (osd.1) 368 : cluster 0 4.4 scrub starts 2024-03-11T21:17:31.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:30 smithi023 bash[21372]: cluster 2024-03-11T21:17:29.898431+0000 osd.1 (osd.1) 369 : cluster 0 4.4 scrub ok 2024-03-11T21:17:31.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:30 smithi023 bash[21372]: cluster 2024-03-11T21:17:30.474914+0000 mon.a (mon.0) 2043 : cluster 0 osdmap e468: 8 total, 8 up, 8 in 2024-03-11T21:17:31.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:30 smithi023 bash[29146]: cluster 2024-03-11T21:17:29.123863+0000 osd.7 (osd.7) 631 : cluster 0 6.15 deep-scrub starts 2024-03-11T21:17:31.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:30 smithi023 bash[29146]: cluster 2024-03-11T21:17:29.228412+0000 osd.7 (osd.7) 632 : cluster 0 6.15 deep-scrub ok 2024-03-11T21:17:31.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:30 smithi023 bash[29146]: cluster 2024-03-11T21:17:29.391039+0000 osd.2 (osd.2) 273 : cluster 0 4.d deep-scrub starts 2024-03-11T21:17:31.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:30 smithi023 bash[29146]: cluster 2024-03-11T21:17:29.394601+0000 osd.2 (osd.2) 274 : cluster 0 4.d deep-scrub ok 2024-03-11T21:17:31.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:30 smithi023 bash[29146]: cluster 2024-03-11T21:17:29.895867+0000 osd.1 (osd.1) 368 : cluster 0 4.4 scrub starts 2024-03-11T21:17:31.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:30 smithi023 bash[29146]: cluster 2024-03-11T21:17:29.898431+0000 osd.1 (osd.1) 369 : cluster 0 4.4 scrub ok 2024-03-11T21:17:31.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:30 smithi023 bash[29146]: cluster 2024-03-11T21:17:30.474914+0000 mon.a (mon.0) 2043 : cluster 0 osdmap e468: 8 total, 8 up, 8 in 2024-03-11T21:17:32.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:31 smithi146 bash[23809]: cluster 2024-03-11T21:17:30.513543+0000 mgr.y (mgr.24380) 986 : cluster 0 pgmap v1148: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail 2024-03-11T21:17:32.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:31 smithi023 bash[21372]: cluster 2024-03-11T21:17:30.513543+0000 mgr.y (mgr.24380) 986 : cluster 0 pgmap v1148: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail 2024-03-11T21:17:32.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:31 smithi023 bash[29146]: cluster 2024-03-11T21:17:30.513543+0000 mgr.y (mgr.24380) 986 : cluster 0 pgmap v1148: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail 2024-03-11T21:17:32.805 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:17:33.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:32 smithi146 bash[23809]: cluster 2024-03-11T21:17:31.378632+0000 osd.2 (osd.2) 275 : cluster 0 2.1 scrub starts 2024-03-11T21:17:33.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:32 smithi146 bash[23809]: cluster 2024-03-11T21:17:31.381256+0000 osd.2 (osd.2) 276 : cluster 0 2.1 scrub ok 2024-03-11T21:17:33.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:32 smithi146 bash[23809]: cluster 2024-03-11T21:17:31.454693+0000 osd.3 (osd.3) 558 : cluster 0 6.11 scrub starts 2024-03-11T21:17:33.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:32 smithi146 bash[23809]: cluster 2024-03-11T21:17:31.506029+0000 osd.3 (osd.3) 559 : cluster 0 6.11 scrub ok 2024-03-11T21:17:33.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:32 smithi146 bash[23809]: cluster 2024-03-11T21:17:31.726866+0000 osd.6 (osd.6) 504 : cluster 0 2.7 scrub starts 2024-03-11T21:17:33.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:32 smithi146 bash[23809]: cluster 2024-03-11T21:17:31.729600+0000 osd.6 (osd.6) 505 : cluster 0 2.7 scrub ok 2024-03-11T21:17:33.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:32 smithi146 bash[23809]: cluster 2024-03-11T21:17:31.831780+0000 osd.1 (osd.1) 370 : cluster 0 1.3 scrub starts 2024-03-11T21:17:33.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:32 smithi146 bash[23809]: cluster 2024-03-11T21:17:31.834664+0000 osd.1 (osd.1) 371 : cluster 0 1.3 scrub ok 2024-03-11T21:17:33.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:32 smithi023 bash[21372]: cluster 2024-03-11T21:17:31.378632+0000 osd.2 (osd.2) 275 : cluster 0 2.1 scrub starts 2024-03-11T21:17:33.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:32 smithi023 bash[21372]: cluster 2024-03-11T21:17:31.381256+0000 osd.2 (osd.2) 276 : cluster 0 2.1 scrub ok 2024-03-11T21:17:33.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:32 smithi023 bash[21372]: cluster 2024-03-11T21:17:31.454693+0000 osd.3 (osd.3) 558 : cluster 0 6.11 scrub starts 2024-03-11T21:17:33.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:32 smithi023 bash[21372]: cluster 2024-03-11T21:17:31.506029+0000 osd.3 (osd.3) 559 : cluster 0 6.11 scrub ok 2024-03-11T21:17:33.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:32 smithi023 bash[21372]: cluster 2024-03-11T21:17:31.726866+0000 osd.6 (osd.6) 504 : cluster 0 2.7 scrub starts 2024-03-11T21:17:33.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:32 smithi023 bash[21372]: cluster 2024-03-11T21:17:31.729600+0000 osd.6 (osd.6) 505 : cluster 0 2.7 scrub ok 2024-03-11T21:17:33.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:32 smithi023 bash[21372]: cluster 2024-03-11T21:17:31.831780+0000 osd.1 (osd.1) 370 : cluster 0 1.3 scrub starts 2024-03-11T21:17:33.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:32 smithi023 bash[21372]: cluster 2024-03-11T21:17:31.834664+0000 osd.1 (osd.1) 371 : cluster 0 1.3 scrub ok 2024-03-11T21:17:33.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:32 smithi023 bash[29146]: cluster 2024-03-11T21:17:31.378632+0000 osd.2 (osd.2) 275 : cluster 0 2.1 scrub starts 2024-03-11T21:17:33.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:32 smithi023 bash[29146]: cluster 2024-03-11T21:17:31.381256+0000 osd.2 (osd.2) 276 : cluster 0 2.1 scrub ok 2024-03-11T21:17:33.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:32 smithi023 bash[29146]: cluster 2024-03-11T21:17:31.454693+0000 osd.3 (osd.3) 558 : cluster 0 6.11 scrub starts 2024-03-11T21:17:33.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:32 smithi023 bash[29146]: cluster 2024-03-11T21:17:31.506029+0000 osd.3 (osd.3) 559 : cluster 0 6.11 scrub ok 2024-03-11T21:17:33.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:32 smithi023 bash[29146]: cluster 2024-03-11T21:17:31.726866+0000 osd.6 (osd.6) 504 : cluster 0 2.7 scrub starts 2024-03-11T21:17:33.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:32 smithi023 bash[29146]: cluster 2024-03-11T21:17:31.729600+0000 osd.6 (osd.6) 505 : cluster 0 2.7 scrub ok 2024-03-11T21:17:33.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:32 smithi023 bash[29146]: cluster 2024-03-11T21:17:31.831780+0000 osd.1 (osd.1) 370 : cluster 0 1.3 scrub starts 2024-03-11T21:17:33.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:32 smithi023 bash[29146]: cluster 2024-03-11T21:17:31.834664+0000 osd.1 (osd.1) 371 : cluster 0 1.3 scrub ok 2024-03-11T21:17:33.248 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:17:33.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:33 smithi146 bash[23809]: cluster 2024-03-11T21:17:32.064750+0000 osd.7 (osd.7) 633 : cluster 0 1.0 scrub starts 2024-03-11T21:17:33.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:33 smithi146 bash[23809]: cluster 2024-03-11T21:17:32.069862+0000 osd.7 (osd.7) 634 : cluster 0 1.0 scrub ok 2024-03-11T21:17:33.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:33 smithi146 bash[23809]: cluster 2024-03-11T21:17:32.515051+0000 mgr.y (mgr.24380) 987 : cluster 0 pgmap v1149: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail 2024-03-11T21:17:33.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:33 smithi146 bash[23809]: cluster 2024-03-11T21:17:33.319214+0000 osd.5 (osd.5) 607 : cluster 0 1.a scrub starts 2024-03-11T21:17:33.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:33 smithi146 bash[23809]: cluster 2024-03-11T21:17:33.321743+0000 osd.5 (osd.5) 608 : cluster 0 1.a scrub ok 2024-03-11T21:17:33.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:33 smithi146 bash[23809]: cluster 2024-03-11T21:17:33.424609+0000 osd.0 (osd.0) 456 : cluster 0 3.12 scrub starts 2024-03-11T21:17:33.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:33 smithi146 bash[23809]: cluster 2024-03-11T21:17:33.428324+0000 osd.0 (osd.0) 457 : cluster 0 3.12 scrub ok 2024-03-11T21:17:34.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:33 smithi023 bash[21372]: cluster 2024-03-11T21:17:32.064750+0000 osd.7 (osd.7) 633 : cluster 0 1.0 scrub starts 2024-03-11T21:17:34.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:33 smithi023 bash[21372]: cluster 2024-03-11T21:17:32.069862+0000 osd.7 (osd.7) 634 : cluster 0 1.0 scrub ok 2024-03-11T21:17:34.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:33 smithi023 bash[21372]: cluster 2024-03-11T21:17:32.515051+0000 mgr.y (mgr.24380) 987 : cluster 0 pgmap v1149: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail 2024-03-11T21:17:34.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:33 smithi023 bash[21372]: cluster 2024-03-11T21:17:33.319214+0000 osd.5 (osd.5) 607 : cluster 0 1.a scrub starts 2024-03-11T21:17:34.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:33 smithi023 bash[21372]: cluster 2024-03-11T21:17:33.321743+0000 osd.5 (osd.5) 608 : cluster 0 1.a scrub ok 2024-03-11T21:17:34.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:33 smithi023 bash[21372]: cluster 2024-03-11T21:17:33.424609+0000 osd.0 (osd.0) 456 : cluster 0 3.12 scrub starts 2024-03-11T21:17:34.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:33 smithi023 bash[21372]: cluster 2024-03-11T21:17:33.428324+0000 osd.0 (osd.0) 457 : cluster 0 3.12 scrub ok 2024-03-11T21:17:34.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:33 smithi023 bash[29146]: cluster 2024-03-11T21:17:32.064750+0000 osd.7 (osd.7) 633 : cluster 0 1.0 scrub starts 2024-03-11T21:17:34.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:33 smithi023 bash[29146]: cluster 2024-03-11T21:17:32.069862+0000 osd.7 (osd.7) 634 : cluster 0 1.0 scrub ok 2024-03-11T21:17:34.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:33 smithi023 bash[29146]: cluster 2024-03-11T21:17:32.515051+0000 mgr.y (mgr.24380) 987 : cluster 0 pgmap v1149: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail 2024-03-11T21:17:34.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:33 smithi023 bash[29146]: cluster 2024-03-11T21:17:33.319214+0000 osd.5 (osd.5) 607 : cluster 0 1.a scrub starts 2024-03-11T21:17:34.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:33 smithi023 bash[29146]: cluster 2024-03-11T21:17:33.321743+0000 osd.5 (osd.5) 608 : cluster 0 1.a scrub ok 2024-03-11T21:17:34.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:33 smithi023 bash[29146]: cluster 2024-03-11T21:17:33.424609+0000 osd.0 (osd.0) 456 : cluster 0 3.12 scrub starts 2024-03-11T21:17:34.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:33 smithi023 bash[29146]: cluster 2024-03-11T21:17:33.428324+0000 osd.0 (osd.0) 457 : cluster 0 3.12 scrub ok 2024-03-11T21:17:34.774 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:34 smithi023 bash[21372]: cluster 2024-03-11T21:17:33.482978+0000 osd.3 (osd.3) 560 : cluster 0 3.7 scrub starts 2024-03-11T21:17:34.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:34 smithi023 bash[21372]: cluster 2024-03-11T21:17:33.486678+0000 osd.3 (osd.3) 561 : cluster 0 3.7 scrub ok 2024-03-11T21:17:34.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:34 smithi023 bash[21372]: cluster 2024-03-11T21:17:34.380435+0000 osd.0 (osd.0) 458 : cluster 0 2.f scrub starts 2024-03-11T21:17:34.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:34 smithi023 bash[21372]: cluster 2024-03-11T21:17:34.383949+0000 osd.0 (osd.0) 459 : cluster 0 2.f scrub ok 2024-03-11T21:17:34.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:34 smithi023 bash[29146]: cluster 2024-03-11T21:17:33.482978+0000 osd.3 (osd.3) 560 : cluster 0 3.7 scrub starts 2024-03-11T21:17:34.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:34 smithi023 bash[29146]: cluster 2024-03-11T21:17:33.486678+0000 osd.3 (osd.3) 561 : cluster 0 3.7 scrub ok 2024-03-11T21:17:34.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:34 smithi023 bash[29146]: cluster 2024-03-11T21:17:34.380435+0000 osd.0 (osd.0) 458 : cluster 0 2.f scrub starts 2024-03-11T21:17:34.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:34 smithi023 bash[29146]: cluster 2024-03-11T21:17:34.383949+0000 osd.0 (osd.0) 459 : cluster 0 2.f scrub ok 2024-03-11T21:17:35.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:34 smithi146 bash[23809]: cluster 2024-03-11T21:17:33.482978+0000 osd.3 (osd.3) 560 : cluster 0 3.7 scrub starts 2024-03-11T21:17:35.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:34 smithi146 bash[23809]: cluster 2024-03-11T21:17:33.486678+0000 osd.3 (osd.3) 561 : cluster 0 3.7 scrub ok 2024-03-11T21:17:35.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:34 smithi146 bash[23809]: cluster 2024-03-11T21:17:34.380435+0000 osd.0 (osd.0) 458 : cluster 0 2.f scrub starts 2024-03-11T21:17:35.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:34 smithi146 bash[23809]: cluster 2024-03-11T21:17:34.383949+0000 osd.0 (osd.0) 459 : cluster 0 2.f scrub ok 2024-03-11T21:17:35.774 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:35 smithi023 bash[21372]: cluster 2024-03-11T21:17:34.515701+0000 mgr.y (mgr.24380) 988 : cluster 0 pgmap v1150: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:17:35.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:35 smithi023 bash[21372]: cluster 2024-03-11T21:17:34.788706+0000 osd.1 (osd.1) 372 : cluster 0 2.a scrub starts 2024-03-11T21:17:35.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:35 smithi023 bash[21372]: cluster 2024-03-11T21:17:34.791695+0000 osd.1 (osd.1) 373 : cluster 0 2.a scrub ok 2024-03-11T21:17:35.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:35 smithi023 bash[29146]: cluster 2024-03-11T21:17:34.515701+0000 mgr.y (mgr.24380) 988 : cluster 0 pgmap v1150: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:17:35.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:35 smithi023 bash[29146]: cluster 2024-03-11T21:17:34.788706+0000 osd.1 (osd.1) 372 : cluster 0 2.a scrub starts 2024-03-11T21:17:35.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:35 smithi023 bash[29146]: cluster 2024-03-11T21:17:34.791695+0000 osd.1 (osd.1) 373 : cluster 0 2.a scrub ok 2024-03-11T21:17:35.879 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-03-11T21:17:35.880 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:17:35.935 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:35 smithi146 bash[23809]: cluster 2024-03-11T21:17:34.515701+0000 mgr.y (mgr.24380) 988 : cluster 0 pgmap v1150: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:17:35.935 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:35 smithi146 bash[23809]: cluster 2024-03-11T21:17:34.788706+0000 osd.1 (osd.1) 372 : cluster 0 2.a scrub starts 2024-03-11T21:17:35.935 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:35 smithi146 bash[23809]: cluster 2024-03-11T21:17:34.791695+0000 osd.1 (osd.1) 373 : cluster 0 2.a scrub ok 2024-03-11T21:17:36.503 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:17:36 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:17:36.688 INFO:teuthology.orchestra.run.smithi023.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-03-11T21:17:36.742 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-03-11T21:17:37.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:36 smithi146 bash[23809]: cluster 2024-03-11T21:17:35.034918+0000 osd.7 (osd.7) 635 : cluster 0 1.2 scrub starts 2024-03-11T21:17:37.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:36 smithi146 bash[23809]: cluster 2024-03-11T21:17:35.037516+0000 osd.7 (osd.7) 636 : cluster 0 1.2 scrub ok 2024-03-11T21:17:37.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:36 smithi146 bash[23809]: audit 2024-03-11T21:17:36.663966+0000 mon.a (mon.0) 2044 : audit 1 from='client.? 172.21.15.23:0/2521714646' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-03-11T21:17:37.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:36 smithi023 bash[21372]: cluster 2024-03-11T21:17:35.034918+0000 osd.7 (osd.7) 635 : cluster 0 1.2 scrub starts 2024-03-11T21:17:37.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:36 smithi023 bash[21372]: cluster 2024-03-11T21:17:35.037516+0000 osd.7 (osd.7) 636 : cluster 0 1.2 scrub ok 2024-03-11T21:17:37.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:36 smithi023 bash[21372]: audit 2024-03-11T21:17:36.663966+0000 mon.a (mon.0) 2044 : audit 1 from='client.? 172.21.15.23:0/2521714646' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-03-11T21:17:37.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:36 smithi023 bash[29146]: cluster 2024-03-11T21:17:35.034918+0000 osd.7 (osd.7) 635 : cluster 0 1.2 scrub starts 2024-03-11T21:17:37.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:36 smithi023 bash[29146]: cluster 2024-03-11T21:17:35.037516+0000 osd.7 (osd.7) 636 : cluster 0 1.2 scrub ok 2024-03-11T21:17:37.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:36 smithi023 bash[29146]: audit 2024-03-11T21:17:36.663966+0000 mon.a (mon.0) 2044 : audit 1 from='client.? 172.21.15.23:0/2521714646' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-03-11T21:17:37.506 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd reweight 5 1 2024-03-11T21:17:37.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:37 smithi023 bash[21372]: audit 2024-03-11T21:17:36.212170+0000 mgr.y (mgr.24380) 989 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:17:37.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:37 smithi023 bash[21372]: cluster 2024-03-11T21:17:36.517140+0000 mgr.y (mgr.24380) 990 : cluster 0 pgmap v1151: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T21:17:37.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:37 smithi023 bash[21372]: audit 2024-03-11T21:17:36.698746+0000 mon.a (mon.0) 2045 : audit 1 from='client.? 172.21.15.23:0/2521714646' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-03-11T21:17:37.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:37 smithi023 bash[21372]: cluster 2024-03-11T21:17:36.711145+0000 mon.a (mon.0) 2046 : cluster 0 osdmap e469: 8 total, 8 up, 8 in 2024-03-11T21:17:37.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:37 smithi023 bash[21372]: audit 2024-03-11T21:17:36.849821+0000 mon.a (mon.0) 2047 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:17:37.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:37 smithi023 bash[21372]: cluster 2024-03-11T21:17:37.282095+0000 osd.5 (osd.5) 609 : cluster 0 4.15 scrub starts 2024-03-11T21:17:37.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:37 smithi023 bash[21372]: cluster 2024-03-11T21:17:37.286222+0000 osd.5 (osd.5) 610 : cluster 0 4.15 scrub ok 2024-03-11T21:17:37.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:37 smithi023 bash[21372]: cluster 2024-03-11T21:17:37.407435+0000 osd.0 (osd.0) 460 : cluster 0 4.b scrub starts 2024-03-11T21:17:37.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:37 smithi023 bash[21372]: cluster 2024-03-11T21:17:37.409891+0000 osd.0 (osd.0) 461 : cluster 0 4.b scrub ok 2024-03-11T21:17:37.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:37 smithi023 bash[29146]: audit 2024-03-11T21:17:36.212170+0000 mgr.y (mgr.24380) 989 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:17:37.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:37 smithi023 bash[29146]: cluster 2024-03-11T21:17:36.517140+0000 mgr.y (mgr.24380) 990 : cluster 0 pgmap v1151: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T21:17:37.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:37 smithi023 bash[29146]: audit 2024-03-11T21:17:36.698746+0000 mon.a (mon.0) 2045 : audit 1 from='client.? 172.21.15.23:0/2521714646' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-03-11T21:17:37.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:37 smithi023 bash[29146]: cluster 2024-03-11T21:17:36.711145+0000 mon.a (mon.0) 2046 : cluster 0 osdmap e469: 8 total, 8 up, 8 in 2024-03-11T21:17:37.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:37 smithi023 bash[29146]: audit 2024-03-11T21:17:36.849821+0000 mon.a (mon.0) 2047 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:17:37.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:37 smithi023 bash[29146]: cluster 2024-03-11T21:17:37.282095+0000 osd.5 (osd.5) 609 : cluster 0 4.15 scrub starts 2024-03-11T21:17:37.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:37 smithi023 bash[29146]: cluster 2024-03-11T21:17:37.286222+0000 osd.5 (osd.5) 610 : cluster 0 4.15 scrub ok 2024-03-11T21:17:37.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:37 smithi023 bash[29146]: cluster 2024-03-11T21:17:37.407435+0000 osd.0 (osd.0) 460 : cluster 0 4.b scrub starts 2024-03-11T21:17:37.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:37 smithi023 bash[29146]: cluster 2024-03-11T21:17:37.409891+0000 osd.0 (osd.0) 461 : cluster 0 4.b scrub ok 2024-03-11T21:17:38.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:37 smithi146 bash[23809]: audit 2024-03-11T21:17:36.212170+0000 mgr.y (mgr.24380) 989 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:17:38.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:37 smithi146 bash[23809]: cluster 2024-03-11T21:17:36.517140+0000 mgr.y (mgr.24380) 990 : cluster 0 pgmap v1151: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T21:17:38.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:37 smithi146 bash[23809]: audit 2024-03-11T21:17:36.698746+0000 mon.a (mon.0) 2045 : audit 1 from='client.? 172.21.15.23:0/2521714646' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-03-11T21:17:38.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:37 smithi146 bash[23809]: cluster 2024-03-11T21:17:36.711145+0000 mon.a (mon.0) 2046 : cluster 0 osdmap e469: 8 total, 8 up, 8 in 2024-03-11T21:17:38.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:37 smithi146 bash[23809]: audit 2024-03-11T21:17:36.849821+0000 mon.a (mon.0) 2047 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:17:38.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:37 smithi146 bash[23809]: cluster 2024-03-11T21:17:37.282095+0000 osd.5 (osd.5) 609 : cluster 0 4.15 scrub starts 2024-03-11T21:17:38.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:37 smithi146 bash[23809]: cluster 2024-03-11T21:17:37.286222+0000 osd.5 (osd.5) 610 : cluster 0 4.15 scrub ok 2024-03-11T21:17:38.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:37 smithi146 bash[23809]: cluster 2024-03-11T21:17:37.407435+0000 osd.0 (osd.0) 460 : cluster 0 4.b scrub starts 2024-03-11T21:17:38.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:37 smithi146 bash[23809]: cluster 2024-03-11T21:17:37.409891+0000 osd.0 (osd.0) 461 : cluster 0 4.b scrub ok 2024-03-11T21:17:38.774 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:17:38 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:17:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:17:40.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:39 smithi146 bash[23809]: cluster 2024-03-11T21:17:38.518073+0000 mgr.y (mgr.24380) 991 : cluster 0 pgmap v1153: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:17:40.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:39 smithi146 bash[23809]: cluster 2024-03-11T21:17:38.679918+0000 osd.1 (osd.1) 374 : cluster 0 3.0 scrub starts 2024-03-11T21:17:40.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:39 smithi146 bash[23809]: cluster 2024-03-11T21:17:38.686502+0000 osd.1 (osd.1) 375 : cluster 0 3.0 scrub ok 2024-03-11T21:17:40.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:39 smithi023 bash[21372]: cluster 2024-03-11T21:17:38.518073+0000 mgr.y (mgr.24380) 991 : cluster 0 pgmap v1153: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:17:40.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:39 smithi023 bash[21372]: cluster 2024-03-11T21:17:38.679918+0000 osd.1 (osd.1) 374 : cluster 0 3.0 scrub starts 2024-03-11T21:17:40.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:39 smithi023 bash[21372]: cluster 2024-03-11T21:17:38.686502+0000 osd.1 (osd.1) 375 : cluster 0 3.0 scrub ok 2024-03-11T21:17:40.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:39 smithi023 bash[29146]: cluster 2024-03-11T21:17:38.518073+0000 mgr.y (mgr.24380) 991 : cluster 0 pgmap v1153: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:17:40.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:39 smithi023 bash[29146]: cluster 2024-03-11T21:17:38.679918+0000 osd.1 (osd.1) 374 : cluster 0 3.0 scrub starts 2024-03-11T21:17:40.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:39 smithi023 bash[29146]: cluster 2024-03-11T21:17:38.686502+0000 osd.1 (osd.1) 375 : cluster 0 3.0 scrub ok 2024-03-11T21:17:41.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:40 smithi146 bash[23809]: cluster 2024-03-11T21:17:39.517208+0000 osd.3 (osd.3) 562 : cluster 0 2.1e scrub starts 2024-03-11T21:17:41.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:40 smithi146 bash[23809]: cluster 2024-03-11T21:17:39.519639+0000 osd.3 (osd.3) 563 : cluster 0 2.1e scrub ok 2024-03-11T21:17:41.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:40 smithi146 bash[23809]: cluster 2024-03-11T21:17:40.379855+0000 osd.5 (osd.5) 611 : cluster 0 2.2 scrub starts 2024-03-11T21:17:41.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:40 smithi146 bash[23809]: cluster 2024-03-11T21:17:40.382353+0000 osd.5 (osd.5) 612 : cluster 0 2.2 scrub ok 2024-03-11T21:17:41.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:40 smithi146 bash[23809]: cluster 2024-03-11T21:17:40.476755+0000 mon.a (mon.0) 2048 : cluster 0 osdmap e470: 8 total, 8 up, 8 in 2024-03-11T21:17:41.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:40 smithi023 bash[29146]: cluster 2024-03-11T21:17:39.517208+0000 osd.3 (osd.3) 562 : cluster 0 2.1e scrub starts 2024-03-11T21:17:41.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:40 smithi023 bash[29146]: cluster 2024-03-11T21:17:39.519639+0000 osd.3 (osd.3) 563 : cluster 0 2.1e scrub ok 2024-03-11T21:17:41.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:40 smithi023 bash[29146]: cluster 2024-03-11T21:17:40.379855+0000 osd.5 (osd.5) 611 : cluster 0 2.2 scrub starts 2024-03-11T21:17:41.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:40 smithi023 bash[29146]: cluster 2024-03-11T21:17:40.382353+0000 osd.5 (osd.5) 612 : cluster 0 2.2 scrub ok 2024-03-11T21:17:41.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:40 smithi023 bash[29146]: cluster 2024-03-11T21:17:40.476755+0000 mon.a (mon.0) 2048 : cluster 0 osdmap e470: 8 total, 8 up, 8 in 2024-03-11T21:17:41.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:40 smithi023 bash[21372]: cluster 2024-03-11T21:17:39.517208+0000 osd.3 (osd.3) 562 : cluster 0 2.1e scrub starts 2024-03-11T21:17:41.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:40 smithi023 bash[21372]: cluster 2024-03-11T21:17:39.519639+0000 osd.3 (osd.3) 563 : cluster 0 2.1e scrub ok 2024-03-11T21:17:41.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:40 smithi023 bash[21372]: cluster 2024-03-11T21:17:40.379855+0000 osd.5 (osd.5) 611 : cluster 0 2.2 scrub starts 2024-03-11T21:17:41.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:40 smithi023 bash[21372]: cluster 2024-03-11T21:17:40.382353+0000 osd.5 (osd.5) 612 : cluster 0 2.2 scrub ok 2024-03-11T21:17:41.026 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:40 smithi023 bash[21372]: cluster 2024-03-11T21:17:40.476755+0000 mon.a (mon.0) 2048 : cluster 0 osdmap e470: 8 total, 8 up, 8 in 2024-03-11T21:17:41.551 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:17:42.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:41 smithi146 bash[23809]: cluster 2024-03-11T21:17:40.518789+0000 mgr.y (mgr.24380) 992 : cluster 0 pgmap v1155: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:17:42.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:41 smithi146 bash[23809]: cluster 2024-03-11T21:17:40.889154+0000 osd.6 (osd.6) 506 : cluster 0 4.1f scrub starts 2024-03-11T21:17:42.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:41 smithi146 bash[23809]: cluster 2024-03-11T21:17:40.894368+0000 osd.6 (osd.6) 507 : cluster 0 4.1f scrub ok 2024-03-11T21:17:42.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:41 smithi146 bash[23809]: cluster 2024-03-11T21:17:41.339795+0000 osd.0 (osd.0) 462 : cluster 0 2.13 scrub starts 2024-03-11T21:17:42.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:41 smithi146 bash[23809]: cluster 2024-03-11T21:17:41.342197+0000 osd.0 (osd.0) 463 : cluster 0 2.13 scrub ok 2024-03-11T21:17:42.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:41 smithi023 bash[21372]: cluster 2024-03-11T21:17:40.518789+0000 mgr.y (mgr.24380) 992 : cluster 0 pgmap v1155: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:17:42.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:41 smithi023 bash[21372]: cluster 2024-03-11T21:17:40.889154+0000 osd.6 (osd.6) 506 : cluster 0 4.1f scrub starts 2024-03-11T21:17:42.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:41 smithi023 bash[21372]: cluster 2024-03-11T21:17:40.894368+0000 osd.6 (osd.6) 507 : cluster 0 4.1f scrub ok 2024-03-11T21:17:42.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:41 smithi023 bash[21372]: cluster 2024-03-11T21:17:41.339795+0000 osd.0 (osd.0) 462 : cluster 0 2.13 scrub starts 2024-03-11T21:17:42.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:41 smithi023 bash[21372]: cluster 2024-03-11T21:17:41.342197+0000 osd.0 (osd.0) 463 : cluster 0 2.13 scrub ok 2024-03-11T21:17:42.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:41 smithi023 bash[29146]: cluster 2024-03-11T21:17:40.518789+0000 mgr.y (mgr.24380) 992 : cluster 0 pgmap v1155: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:17:42.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:41 smithi023 bash[29146]: cluster 2024-03-11T21:17:40.889154+0000 osd.6 (osd.6) 506 : cluster 0 4.1f scrub starts 2024-03-11T21:17:42.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:41 smithi023 bash[29146]: cluster 2024-03-11T21:17:40.894368+0000 osd.6 (osd.6) 507 : cluster 0 4.1f scrub ok 2024-03-11T21:17:42.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:41 smithi023 bash[29146]: cluster 2024-03-11T21:17:41.339795+0000 osd.0 (osd.0) 462 : cluster 0 2.13 scrub starts 2024-03-11T21:17:42.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:41 smithi023 bash[29146]: cluster 2024-03-11T21:17:41.342197+0000 osd.0 (osd.0) 463 : cluster 0 2.13 scrub ok 2024-03-11T21:17:42.301 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:17:43.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:42 smithi146 bash[23809]: cluster 2024-03-11T21:17:41.798960+0000 osd.1 (osd.1) 376 : cluster 0 1.13 scrub starts 2024-03-11T21:17:43.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:42 smithi146 bash[23809]: cluster 2024-03-11T21:17:41.801570+0000 osd.1 (osd.1) 377 : cluster 0 1.13 scrub ok 2024-03-11T21:17:43.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:42 smithi023 bash[29146]: cluster 2024-03-11T21:17:41.798960+0000 osd.1 (osd.1) 376 : cluster 0 1.13 scrub starts 2024-03-11T21:17:43.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:42 smithi023 bash[29146]: cluster 2024-03-11T21:17:41.801570+0000 osd.1 (osd.1) 377 : cluster 0 1.13 scrub ok 2024-03-11T21:17:43.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:42 smithi023 bash[21372]: cluster 2024-03-11T21:17:41.798960+0000 osd.1 (osd.1) 376 : cluster 0 1.13 scrub starts 2024-03-11T21:17:43.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:42 smithi023 bash[21372]: cluster 2024-03-11T21:17:41.801570+0000 osd.1 (osd.1) 377 : cluster 0 1.13 scrub ok 2024-03-11T21:17:44.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:43 smithi146 bash[23809]: cluster 2024-03-11T21:17:42.520135+0000 mgr.y (mgr.24380) 993 : cluster 0 pgmap v1156: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:17:44.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:43 smithi146 bash[23809]: cluster 2024-03-11T21:17:42.874360+0000 osd.6 (osd.6) 508 : cluster 0 1.6 scrub starts 2024-03-11T21:17:44.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:43 smithi146 bash[23809]: cluster 2024-03-11T21:17:42.876322+0000 osd.6 (osd.6) 509 : cluster 0 1.6 scrub ok 2024-03-11T21:17:44.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:43 smithi023 bash[29146]: cluster 2024-03-11T21:17:42.520135+0000 mgr.y (mgr.24380) 993 : cluster 0 pgmap v1156: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:17:44.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:43 smithi023 bash[29146]: cluster 2024-03-11T21:17:42.874360+0000 osd.6 (osd.6) 508 : cluster 0 1.6 scrub starts 2024-03-11T21:17:44.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:43 smithi023 bash[29146]: cluster 2024-03-11T21:17:42.876322+0000 osd.6 (osd.6) 509 : cluster 0 1.6 scrub ok 2024-03-11T21:17:44.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:43 smithi023 bash[21372]: cluster 2024-03-11T21:17:42.520135+0000 mgr.y (mgr.24380) 993 : cluster 0 pgmap v1156: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:17:44.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:43 smithi023 bash[21372]: cluster 2024-03-11T21:17:42.874360+0000 osd.6 (osd.6) 508 : cluster 0 1.6 scrub starts 2024-03-11T21:17:44.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:43 smithi023 bash[21372]: cluster 2024-03-11T21:17:42.876322+0000 osd.6 (osd.6) 509 : cluster 0 1.6 scrub ok 2024-03-11T21:17:44.707 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-03-11T21:17:44.707 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:17:44.935 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:44 smithi146 bash[23809]: cluster 2024-03-11T21:17:43.113494+0000 osd.7 (osd.7) 637 : cluster 0 6.1c scrub starts 2024-03-11T21:17:44.935 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:44 smithi146 bash[23809]: cluster 2024-03-11T21:17:43.115380+0000 osd.7 (osd.7) 638 : cluster 0 6.1c scrub ok 2024-03-11T21:17:44.935 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:44 smithi146 bash[23809]: audit 2024-03-11T21:17:43.835726+0000 mon.a (mon.0) 2049 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:17:44.935 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:44 smithi146 bash[23809]: cluster 2024-03-11T21:17:43.849781+0000 osd.2 (osd.2) 277 : cluster 0 purged_snaps scrub starts 2024-03-11T21:17:44.935 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:44 smithi146 bash[23809]: cluster 2024-03-11T21:17:43.849900+0000 osd.2 (osd.2) 278 : cluster 0 purged_snaps scrub ok 2024-03-11T21:17:44.935 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:44 smithi146 bash[23809]: cluster 2024-03-11T21:17:44.377956+0000 osd.0 (osd.0) 464 : cluster 0 1.11 deep-scrub starts 2024-03-11T21:17:44.935 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:44 smithi146 bash[23809]: cluster 2024-03-11T21:17:44.379266+0000 osd.0 (osd.0) 465 : cluster 0 1.11 deep-scrub ok 2024-03-11T21:17:44.935 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:44 smithi146 bash[23809]: cluster 2024-03-11T21:17:44.423635+0000 osd.5 (osd.5) 613 : cluster 0 1.10 scrub starts 2024-03-11T21:17:44.935 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:44 smithi146 bash[23809]: cluster 2024-03-11T21:17:44.424760+0000 osd.5 (osd.5) 614 : cluster 0 1.10 scrub ok 2024-03-11T21:17:44.935 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:44 smithi146 bash[23809]: audit 2024-03-11T21:17:44.569982+0000 mon.a (mon.0) 2050 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:17:44.935 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:44 smithi146 bash[23809]: audit 2024-03-11T21:17:44.571619+0000 mon.a (mon.0) 2051 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:17:44.936 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:44 smithi146 bash[23809]: audit 2024-03-11T21:17:44.586477+0000 mon.a (mon.0) 2052 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:17:45.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:44 smithi023 bash[29146]: cluster 2024-03-11T21:17:43.113494+0000 osd.7 (osd.7) 637 : cluster 0 6.1c scrub starts 2024-03-11T21:17:45.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:44 smithi023 bash[29146]: cluster 2024-03-11T21:17:43.115380+0000 osd.7 (osd.7) 638 : cluster 0 6.1c scrub ok 2024-03-11T21:17:45.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:44 smithi023 bash[29146]: audit 2024-03-11T21:17:43.835726+0000 mon.a (mon.0) 2049 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:17:45.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:44 smithi023 bash[29146]: cluster 2024-03-11T21:17:43.849781+0000 osd.2 (osd.2) 277 : cluster 0 purged_snaps scrub starts 2024-03-11T21:17:45.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:44 smithi023 bash[29146]: cluster 2024-03-11T21:17:43.849900+0000 osd.2 (osd.2) 278 : cluster 0 purged_snaps scrub ok 2024-03-11T21:17:45.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:44 smithi023 bash[29146]: cluster 2024-03-11T21:17:44.377956+0000 osd.0 (osd.0) 464 : cluster 0 1.11 deep-scrub starts 2024-03-11T21:17:45.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:44 smithi023 bash[29146]: cluster 2024-03-11T21:17:44.379266+0000 osd.0 (osd.0) 465 : cluster 0 1.11 deep-scrub ok 2024-03-11T21:17:45.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:44 smithi023 bash[29146]: cluster 2024-03-11T21:17:44.423635+0000 osd.5 (osd.5) 613 : cluster 0 1.10 scrub starts 2024-03-11T21:17:45.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:44 smithi023 bash[29146]: cluster 2024-03-11T21:17:44.424760+0000 osd.5 (osd.5) 614 : cluster 0 1.10 scrub ok 2024-03-11T21:17:45.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:44 smithi023 bash[29146]: audit 2024-03-11T21:17:44.569982+0000 mon.a (mon.0) 2050 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:17:45.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:44 smithi023 bash[29146]: audit 2024-03-11T21:17:44.571619+0000 mon.a (mon.0) 2051 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:17:45.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:44 smithi023 bash[29146]: audit 2024-03-11T21:17:44.586477+0000 mon.a (mon.0) 2052 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:17:45.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:44 smithi023 bash[21372]: cluster 2024-03-11T21:17:43.113494+0000 osd.7 (osd.7) 637 : cluster 0 6.1c scrub starts 2024-03-11T21:17:45.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:44 smithi023 bash[21372]: cluster 2024-03-11T21:17:43.115380+0000 osd.7 (osd.7) 638 : cluster 0 6.1c scrub ok 2024-03-11T21:17:45.026 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:44 smithi023 bash[21372]: audit 2024-03-11T21:17:43.835726+0000 mon.a (mon.0) 2049 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:17:45.026 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:44 smithi023 bash[21372]: cluster 2024-03-11T21:17:43.849781+0000 osd.2 (osd.2) 277 : cluster 0 purged_snaps scrub starts 2024-03-11T21:17:45.026 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:44 smithi023 bash[21372]: cluster 2024-03-11T21:17:43.849900+0000 osd.2 (osd.2) 278 : cluster 0 purged_snaps scrub ok 2024-03-11T21:17:45.026 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:44 smithi023 bash[21372]: cluster 2024-03-11T21:17:44.377956+0000 osd.0 (osd.0) 464 : cluster 0 1.11 deep-scrub starts 2024-03-11T21:17:45.026 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:44 smithi023 bash[21372]: cluster 2024-03-11T21:17:44.379266+0000 osd.0 (osd.0) 465 : cluster 0 1.11 deep-scrub ok 2024-03-11T21:17:45.026 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:44 smithi023 bash[21372]: cluster 2024-03-11T21:17:44.423635+0000 osd.5 (osd.5) 613 : cluster 0 1.10 scrub starts 2024-03-11T21:17:45.026 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:44 smithi023 bash[21372]: cluster 2024-03-11T21:17:44.424760+0000 osd.5 (osd.5) 614 : cluster 0 1.10 scrub ok 2024-03-11T21:17:45.026 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:44 smithi023 bash[21372]: audit 2024-03-11T21:17:44.569982+0000 mon.a (mon.0) 2050 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:17:45.026 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:44 smithi023 bash[21372]: audit 2024-03-11T21:17:44.571619+0000 mon.a (mon.0) 2051 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:17:45.026 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:44 smithi023 bash[21372]: audit 2024-03-11T21:17:44.586477+0000 mon.a (mon.0) 2052 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:17:45.567 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-11T21:17:46.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:45 smithi146 bash[23809]: cluster 2024-03-11T21:17:44.085043+0000 osd.7 (osd.7) 639 : cluster 0 1.d scrub starts 2024-03-11T21:17:46.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:45 smithi146 bash[23809]: cluster 2024-03-11T21:17:44.087092+0000 osd.7 (osd.7) 640 : cluster 0 1.d scrub ok 2024-03-11T21:17:46.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:45 smithi146 bash[23809]: cluster 2024-03-11T21:17:44.520740+0000 mgr.y (mgr.24380) 994 : cluster 0 pgmap v1157: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:17:46.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:45 smithi146 bash[23809]: cluster 2024-03-11T21:17:44.794013+0000 mon.a (mon.0) 2053 : cluster 0 osdmap e471: 8 total, 8 up, 8 in 2024-03-11T21:17:46.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:45 smithi146 bash[23809]: cluster 2024-03-11T21:17:44.930891+0000 osd.6 (osd.6) 510 : cluster 0 1.f scrub starts 2024-03-11T21:17:46.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:45 smithi146 bash[23809]: cluster 2024-03-11T21:17:44.935028+0000 osd.6 (osd.6) 511 : cluster 0 1.f scrub ok 2024-03-11T21:17:46.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:45 smithi146 bash[23809]: cluster 2024-03-11T21:17:45.076470+0000 osd.1 (osd.1) 378 : cluster 0 purged_snaps scrub starts 2024-03-11T21:17:46.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:45 smithi146 bash[23809]: cluster 2024-03-11T21:17:45.076645+0000 osd.1 (osd.1) 379 : cluster 0 purged_snaps scrub ok 2024-03-11T21:17:46.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:45 smithi146 bash[23809]: cluster 2024-03-11T21:17:45.437514+0000 osd.5 (osd.5) 615 : cluster 0 1.1 deep-scrub starts 2024-03-11T21:17:46.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:45 smithi146 bash[23809]: cluster 2024-03-11T21:17:45.440102+0000 osd.5 (osd.5) 616 : cluster 0 1.1 deep-scrub ok 2024-03-11T21:17:46.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:45 smithi146 bash[23809]: cluster 2024-03-11T21:17:45.481148+0000 mon.a (mon.0) 2054 : cluster 0 osdmap e472: 8 total, 8 up, 8 in 2024-03-11T21:17:46.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:45 smithi146 bash[23809]: audit 2024-03-11T21:17:45.522155+0000 mon.a (mon.0) 2055 : audit 1 from='client.? 172.21.15.23:0/3257622485' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-11T21:17:46.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:45 smithi023 bash[21372]: cluster 2024-03-11T21:17:44.085043+0000 osd.7 (osd.7) 639 : cluster 0 1.d scrub starts 2024-03-11T21:17:46.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:45 smithi023 bash[21372]: cluster 2024-03-11T21:17:44.087092+0000 osd.7 (osd.7) 640 : cluster 0 1.d scrub ok 2024-03-11T21:17:46.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:45 smithi023 bash[21372]: cluster 2024-03-11T21:17:44.520740+0000 mgr.y (mgr.24380) 994 : cluster 0 pgmap v1157: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:17:46.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:45 smithi023 bash[21372]: cluster 2024-03-11T21:17:44.794013+0000 mon.a (mon.0) 2053 : cluster 0 osdmap e471: 8 total, 8 up, 8 in 2024-03-11T21:17:46.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:45 smithi023 bash[21372]: cluster 2024-03-11T21:17:44.930891+0000 osd.6 (osd.6) 510 : cluster 0 1.f scrub starts 2024-03-11T21:17:46.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:45 smithi023 bash[21372]: cluster 2024-03-11T21:17:44.935028+0000 osd.6 (osd.6) 511 : cluster 0 1.f scrub ok 2024-03-11T21:17:46.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:45 smithi023 bash[21372]: cluster 2024-03-11T21:17:45.076470+0000 osd.1 (osd.1) 378 : cluster 0 purged_snaps scrub starts 2024-03-11T21:17:46.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:45 smithi023 bash[21372]: cluster 2024-03-11T21:17:45.076645+0000 osd.1 (osd.1) 379 : cluster 0 purged_snaps scrub ok 2024-03-11T21:17:46.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:45 smithi023 bash[21372]: cluster 2024-03-11T21:17:45.437514+0000 osd.5 (osd.5) 615 : cluster 0 1.1 deep-scrub starts 2024-03-11T21:17:46.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:45 smithi023 bash[21372]: cluster 2024-03-11T21:17:45.440102+0000 osd.5 (osd.5) 616 : cluster 0 1.1 deep-scrub ok 2024-03-11T21:17:46.026 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:45 smithi023 bash[21372]: cluster 2024-03-11T21:17:45.481148+0000 mon.a (mon.0) 2054 : cluster 0 osdmap e472: 8 total, 8 up, 8 in 2024-03-11T21:17:46.026 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:45 smithi023 bash[21372]: audit 2024-03-11T21:17:45.522155+0000 mon.a (mon.0) 2055 : audit 1 from='client.? 172.21.15.23:0/3257622485' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-11T21:17:46.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:45 smithi023 bash[29146]: cluster 2024-03-11T21:17:44.085043+0000 osd.7 (osd.7) 639 : cluster 0 1.d scrub starts 2024-03-11T21:17:46.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:45 smithi023 bash[29146]: cluster 2024-03-11T21:17:44.087092+0000 osd.7 (osd.7) 640 : cluster 0 1.d scrub ok 2024-03-11T21:17:46.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:45 smithi023 bash[29146]: cluster 2024-03-11T21:17:44.520740+0000 mgr.y (mgr.24380) 994 : cluster 0 pgmap v1157: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:17:46.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:45 smithi023 bash[29146]: cluster 2024-03-11T21:17:44.794013+0000 mon.a (mon.0) 2053 : cluster 0 osdmap e471: 8 total, 8 up, 8 in 2024-03-11T21:17:46.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:45 smithi023 bash[29146]: cluster 2024-03-11T21:17:44.930891+0000 osd.6 (osd.6) 510 : cluster 0 1.f scrub starts 2024-03-11T21:17:46.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:45 smithi023 bash[29146]: cluster 2024-03-11T21:17:44.935028+0000 osd.6 (osd.6) 511 : cluster 0 1.f scrub ok 2024-03-11T21:17:46.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:45 smithi023 bash[29146]: cluster 2024-03-11T21:17:45.076470+0000 osd.1 (osd.1) 378 : cluster 0 purged_snaps scrub starts 2024-03-11T21:17:46.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:45 smithi023 bash[29146]: cluster 2024-03-11T21:17:45.076645+0000 osd.1 (osd.1) 379 : cluster 0 purged_snaps scrub ok 2024-03-11T21:17:46.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:45 smithi023 bash[29146]: cluster 2024-03-11T21:17:45.437514+0000 osd.5 (osd.5) 615 : cluster 0 1.1 deep-scrub starts 2024-03-11T21:17:46.027 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:45 smithi023 bash[29146]: cluster 2024-03-11T21:17:45.440102+0000 osd.5 (osd.5) 616 : cluster 0 1.1 deep-scrub ok 2024-03-11T21:17:46.027 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:45 smithi023 bash[29146]: cluster 2024-03-11T21:17:45.481148+0000 mon.a (mon.0) 2054 : cluster 0 osdmap e472: 8 total, 8 up, 8 in 2024-03-11T21:17:46.027 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:45 smithi023 bash[29146]: audit 2024-03-11T21:17:45.522155+0000 mon.a (mon.0) 2055 : audit 1 from='client.? 172.21.15.23:0/3257622485' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-11T21:17:46.495 INFO:teuthology.orchestra.run.smithi023.stderr:reweighted osd.5 to 1 (10000) 2024-03-11T21:17:46.503 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:17:46 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:17:47.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:46 smithi146 bash[23809]: cluster 2024-03-11T21:17:45.929293+0000 osd.1 (osd.1) 380 : cluster 0 6.4 scrub starts 2024-03-11T21:17:47.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:46 smithi146 bash[23809]: cluster 2024-03-11T21:17:45.931759+0000 osd.1 (osd.1) 381 : cluster 0 6.4 scrub ok 2024-03-11T21:17:47.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:46 smithi146 bash[23809]: cluster 2024-03-11T21:17:46.436991+0000 osd.0 (osd.0) 466 : cluster 0 3.17 deep-scrub starts 2024-03-11T21:17:47.254 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:46 smithi146 bash[23809]: cluster 2024-03-11T21:17:46.445776+0000 osd.0 (osd.0) 467 : cluster 0 3.17 deep-scrub ok 2024-03-11T21:17:47.254 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:46 smithi146 bash[23809]: audit 2024-03-11T21:17:46.497930+0000 mon.a (mon.0) 2056 : audit 1 from='client.? 172.21.15.23:0/3257622485' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-11T21:17:47.254 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:46 smithi146 bash[23809]: cluster 2024-03-11T21:17:46.504220+0000 mon.a (mon.0) 2057 : cluster 0 osdmap e473: 8 total, 8 up, 8 in 2024-03-11T21:17:47.275 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:46 smithi023 bash[21372]: cluster 2024-03-11T21:17:45.929293+0000 osd.1 (osd.1) 380 : cluster 0 6.4 scrub starts 2024-03-11T21:17:47.275 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:46 smithi023 bash[21372]: cluster 2024-03-11T21:17:45.931759+0000 osd.1 (osd.1) 381 : cluster 0 6.4 scrub ok 2024-03-11T21:17:47.275 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:46 smithi023 bash[21372]: cluster 2024-03-11T21:17:46.436991+0000 osd.0 (osd.0) 466 : cluster 0 3.17 deep-scrub starts 2024-03-11T21:17:47.275 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:46 smithi023 bash[21372]: cluster 2024-03-11T21:17:46.445776+0000 osd.0 (osd.0) 467 : cluster 0 3.17 deep-scrub ok 2024-03-11T21:17:47.275 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:46 smithi023 bash[21372]: audit 2024-03-11T21:17:46.497930+0000 mon.a (mon.0) 2056 : audit 1 from='client.? 172.21.15.23:0/3257622485' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-11T21:17:47.275 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:46 smithi023 bash[21372]: cluster 2024-03-11T21:17:46.504220+0000 mon.a (mon.0) 2057 : cluster 0 osdmap e473: 8 total, 8 up, 8 in 2024-03-11T21:17:47.275 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:46 smithi023 bash[29146]: cluster 2024-03-11T21:17:45.929293+0000 osd.1 (osd.1) 380 : cluster 0 6.4 scrub starts 2024-03-11T21:17:47.275 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:46 smithi023 bash[29146]: cluster 2024-03-11T21:17:45.931759+0000 osd.1 (osd.1) 381 : cluster 0 6.4 scrub ok 2024-03-11T21:17:47.275 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:46 smithi023 bash[29146]: cluster 2024-03-11T21:17:46.436991+0000 osd.0 (osd.0) 466 : cluster 0 3.17 deep-scrub starts 2024-03-11T21:17:47.276 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:46 smithi023 bash[29146]: cluster 2024-03-11T21:17:46.445776+0000 osd.0 (osd.0) 467 : cluster 0 3.17 deep-scrub ok 2024-03-11T21:17:47.276 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:46 smithi023 bash[29146]: audit 2024-03-11T21:17:46.497930+0000 mon.a (mon.0) 2056 : audit 1 from='client.? 172.21.15.23:0/3257622485' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-11T21:17:47.276 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:46 smithi023 bash[29146]: cluster 2024-03-11T21:17:46.504220+0000 mon.a (mon.0) 2057 : cluster 0 osdmap e473: 8 total, 8 up, 8 in 2024-03-11T21:17:47.372 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd primary-affinity 5 1 2024-03-11T21:17:48.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:47 smithi146 bash[23809]: audit 2024-03-11T21:17:46.220186+0000 mgr.y (mgr.24380) 995 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:17:48.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:47 smithi146 bash[23809]: cluster 2024-03-11T21:17:46.521205+0000 mgr.y (mgr.24380) 996 : cluster 0 pgmap v1161: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-11T21:17:48.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:47 smithi146 bash[23809]: cluster 2024-03-11T21:17:46.912318+0000 osd.6 (osd.6) 512 : cluster 0 1.7 scrub starts 2024-03-11T21:17:48.254 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:47 smithi146 bash[23809]: cluster 2024-03-11T21:17:46.914825+0000 osd.6 (osd.6) 513 : cluster 0 1.7 scrub ok 2024-03-11T21:17:48.274 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:47 smithi023 bash[29146]: audit 2024-03-11T21:17:46.220186+0000 mgr.y (mgr.24380) 995 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:17:48.275 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:47 smithi023 bash[29146]: cluster 2024-03-11T21:17:46.521205+0000 mgr.y (mgr.24380) 996 : cluster 0 pgmap v1161: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-11T21:17:48.275 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:47 smithi023 bash[29146]: cluster 2024-03-11T21:17:46.912318+0000 osd.6 (osd.6) 512 : cluster 0 1.7 scrub starts 2024-03-11T21:17:48.275 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:47 smithi023 bash[29146]: cluster 2024-03-11T21:17:46.914825+0000 osd.6 (osd.6) 513 : cluster 0 1.7 scrub ok 2024-03-11T21:17:48.275 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:47 smithi023 bash[21372]: audit 2024-03-11T21:17:46.220186+0000 mgr.y (mgr.24380) 995 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:17:48.275 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:47 smithi023 bash[21372]: cluster 2024-03-11T21:17:46.521205+0000 mgr.y (mgr.24380) 996 : cluster 0 pgmap v1161: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-11T21:17:48.275 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:47 smithi023 bash[21372]: cluster 2024-03-11T21:17:46.912318+0000 osd.6 (osd.6) 512 : cluster 0 1.7 scrub starts 2024-03-11T21:17:48.275 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:47 smithi023 bash[21372]: cluster 2024-03-11T21:17:46.914825+0000 osd.6 (osd.6) 513 : cluster 0 1.7 scrub ok 2024-03-11T21:17:48.774 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:17:48 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:17:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:17:49.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:48 smithi146 bash[23809]: cluster 2024-03-11T21:17:47.262825+0000 osd.2 (osd.2) 279 : cluster 0 4.1c scrub starts 2024-03-11T21:17:49.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:48 smithi146 bash[23809]: cluster 2024-03-11T21:17:47.265347+0000 osd.2 (osd.2) 280 : cluster 0 4.1c scrub ok 2024-03-11T21:17:49.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:48 smithi146 bash[23809]: cluster 2024-03-11T21:17:48.438696+0000 osd.5 (osd.5) 617 : cluster 0 1.8 deep-scrub starts 2024-03-11T21:17:49.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:48 smithi146 bash[23809]: cluster 2024-03-11T21:17:48.442494+0000 osd.5 (osd.5) 618 : cluster 0 1.8 deep-scrub ok 2024-03-11T21:17:49.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:48 smithi023 bash[21372]: cluster 2024-03-11T21:17:47.262825+0000 osd.2 (osd.2) 279 : cluster 0 4.1c scrub starts 2024-03-11T21:17:49.275 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:48 smithi023 bash[21372]: cluster 2024-03-11T21:17:47.265347+0000 osd.2 (osd.2) 280 : cluster 0 4.1c scrub ok 2024-03-11T21:17:49.275 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:48 smithi023 bash[21372]: cluster 2024-03-11T21:17:48.438696+0000 osd.5 (osd.5) 617 : cluster 0 1.8 deep-scrub starts 2024-03-11T21:17:49.275 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:48 smithi023 bash[21372]: cluster 2024-03-11T21:17:48.442494+0000 osd.5 (osd.5) 618 : cluster 0 1.8 deep-scrub ok 2024-03-11T21:17:49.275 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:48 smithi023 bash[29146]: cluster 2024-03-11T21:17:47.262825+0000 osd.2 (osd.2) 279 : cluster 0 4.1c scrub starts 2024-03-11T21:17:49.275 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:48 smithi023 bash[29146]: cluster 2024-03-11T21:17:47.265347+0000 osd.2 (osd.2) 280 : cluster 0 4.1c scrub ok 2024-03-11T21:17:49.275 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:48 smithi023 bash[29146]: cluster 2024-03-11T21:17:48.438696+0000 osd.5 (osd.5) 617 : cluster 0 1.8 deep-scrub starts 2024-03-11T21:17:49.275 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:48 smithi023 bash[29146]: cluster 2024-03-11T21:17:48.442494+0000 osd.5 (osd.5) 618 : cluster 0 1.8 deep-scrub ok 2024-03-11T21:17:50.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:49 smithi146 bash[23809]: cluster 2024-03-11T21:17:48.522181+0000 mgr.y (mgr.24380) 997 : cluster 0 pgmap v1162: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:17:50.254 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:49 smithi146 bash[23809]: cluster 2024-03-11T21:17:48.528670+0000 osd.6 (osd.6) 514 : cluster 0 purged_snaps scrub starts 2024-03-11T21:17:50.254 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:49 smithi146 bash[23809]: cluster 2024-03-11T21:17:48.528764+0000 osd.6 (osd.6) 515 : cluster 0 purged_snaps scrub ok 2024-03-11T21:17:50.254 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:49 smithi146 bash[23809]: cluster 2024-03-11T21:17:48.827829+0000 mon.a (mon.0) 2058 : cluster 0 osdmap e474: 8 total, 8 up, 8 in 2024-03-11T21:17:50.254 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:49 smithi146 bash[23809]: cluster 2024-03-11T21:17:49.414065+0000 osd.0 (osd.0) 468 : cluster 0 6.16 deep-scrub starts 2024-03-11T21:17:50.254 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:49 smithi146 bash[23809]: cluster 2024-03-11T21:17:49.459134+0000 osd.5 (osd.5) 619 : cluster 0 1.12 scrub starts 2024-03-11T21:17:50.254 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:49 smithi146 bash[23809]: cluster 2024-03-11T21:17:49.461698+0000 osd.5 (osd.5) 620 : cluster 0 1.12 scrub ok 2024-03-11T21:17:50.254 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:49 smithi146 bash[23809]: cluster 2024-03-11T21:17:49.501922+0000 osd.0 (osd.0) 469 : cluster 0 6.16 deep-scrub ok 2024-03-11T21:17:50.275 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:49 smithi023 bash[21372]: cluster 2024-03-11T21:17:48.522181+0000 mgr.y (mgr.24380) 997 : cluster 0 pgmap v1162: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:17:50.275 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:49 smithi023 bash[21372]: cluster 2024-03-11T21:17:48.528670+0000 osd.6 (osd.6) 514 : cluster 0 purged_snaps scrub starts 2024-03-11T21:17:50.275 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:49 smithi023 bash[21372]: cluster 2024-03-11T21:17:48.528764+0000 osd.6 (osd.6) 515 : cluster 0 purged_snaps scrub ok 2024-03-11T21:17:50.275 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:49 smithi023 bash[21372]: cluster 2024-03-11T21:17:48.827829+0000 mon.a (mon.0) 2058 : cluster 0 osdmap e474: 8 total, 8 up, 8 in 2024-03-11T21:17:50.275 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:49 smithi023 bash[21372]: cluster 2024-03-11T21:17:49.414065+0000 osd.0 (osd.0) 468 : cluster 0 6.16 deep-scrub starts 2024-03-11T21:17:50.275 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:49 smithi023 bash[21372]: cluster 2024-03-11T21:17:49.459134+0000 osd.5 (osd.5) 619 : cluster 0 1.12 scrub starts 2024-03-11T21:17:50.275 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:49 smithi023 bash[21372]: cluster 2024-03-11T21:17:49.461698+0000 osd.5 (osd.5) 620 : cluster 0 1.12 scrub ok 2024-03-11T21:17:50.275 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:49 smithi023 bash[21372]: cluster 2024-03-11T21:17:49.501922+0000 osd.0 (osd.0) 469 : cluster 0 6.16 deep-scrub ok 2024-03-11T21:17:50.276 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:49 smithi023 bash[29146]: cluster 2024-03-11T21:17:48.522181+0000 mgr.y (mgr.24380) 997 : cluster 0 pgmap v1162: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:17:50.276 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:49 smithi023 bash[29146]: cluster 2024-03-11T21:17:48.528670+0000 osd.6 (osd.6) 514 : cluster 0 purged_snaps scrub starts 2024-03-11T21:17:50.276 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:49 smithi023 bash[29146]: cluster 2024-03-11T21:17:48.528764+0000 osd.6 (osd.6) 515 : cluster 0 purged_snaps scrub ok 2024-03-11T21:17:50.276 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:49 smithi023 bash[29146]: cluster 2024-03-11T21:17:48.827829+0000 mon.a (mon.0) 2058 : cluster 0 osdmap e474: 8 total, 8 up, 8 in 2024-03-11T21:17:50.276 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:49 smithi023 bash[29146]: cluster 2024-03-11T21:17:49.414065+0000 osd.0 (osd.0) 468 : cluster 0 6.16 deep-scrub starts 2024-03-11T21:17:50.276 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:49 smithi023 bash[29146]: cluster 2024-03-11T21:17:49.459134+0000 osd.5 (osd.5) 619 : cluster 0 1.12 scrub starts 2024-03-11T21:17:50.276 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:49 smithi023 bash[29146]: cluster 2024-03-11T21:17:49.461698+0000 osd.5 (osd.5) 620 : cluster 0 1.12 scrub ok 2024-03-11T21:17:50.277 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:49 smithi023 bash[29146]: cluster 2024-03-11T21:17:49.501922+0000 osd.0 (osd.0) 469 : cluster 0 6.16 deep-scrub ok 2024-03-11T21:17:50.362 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:17:51.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:50 smithi146 bash[23809]: cluster 2024-03-11T21:17:49.160807+0000 osd.7 (osd.7) 641 : cluster 0 1.e scrub starts 2024-03-11T21:17:51.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:50 smithi146 bash[23809]: cluster 2024-03-11T21:17:49.163484+0000 osd.7 (osd.7) 642 : cluster 0 1.e scrub ok 2024-03-11T21:17:51.254 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:50 smithi146 bash[23809]: cluster 2024-03-11T21:17:50.207638+0000 osd.7 (osd.7) 643 : cluster 0 3.13 scrub starts 2024-03-11T21:17:51.254 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:50 smithi146 bash[23809]: cluster 2024-03-11T21:17:50.211823+0000 osd.7 (osd.7) 644 : cluster 0 3.13 scrub ok 2024-03-11T21:17:51.254 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:50 smithi146 bash[23809]: cluster 2024-03-11T21:17:50.416110+0000 osd.0 (osd.0) 470 : cluster 0 3.1 scrub starts 2024-03-11T21:17:51.254 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:50 smithi146 bash[23809]: cluster 2024-03-11T21:17:50.420835+0000 osd.0 (osd.0) 471 : cluster 0 3.1 scrub ok 2024-03-11T21:17:51.254 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:50 smithi146 bash[23809]: cluster 2024-03-11T21:17:50.522897+0000 mgr.y (mgr.24380) 998 : cluster 0 pgmap v1164: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 892 B/s rd, 0 op/s 2024-03-11T21:17:51.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:50 smithi023 bash[21372]: cluster 2024-03-11T21:17:49.160807+0000 osd.7 (osd.7) 641 : cluster 0 1.e scrub starts 2024-03-11T21:17:51.275 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:50 smithi023 bash[21372]: cluster 2024-03-11T21:17:49.163484+0000 osd.7 (osd.7) 642 : cluster 0 1.e scrub ok 2024-03-11T21:17:51.275 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:50 smithi023 bash[21372]: cluster 2024-03-11T21:17:50.207638+0000 osd.7 (osd.7) 643 : cluster 0 3.13 scrub starts 2024-03-11T21:17:51.275 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:50 smithi023 bash[21372]: cluster 2024-03-11T21:17:50.211823+0000 osd.7 (osd.7) 644 : cluster 0 3.13 scrub ok 2024-03-11T21:17:51.275 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:50 smithi023 bash[21372]: cluster 2024-03-11T21:17:50.416110+0000 osd.0 (osd.0) 470 : cluster 0 3.1 scrub starts 2024-03-11T21:17:51.275 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:50 smithi023 bash[21372]: cluster 2024-03-11T21:17:50.420835+0000 osd.0 (osd.0) 471 : cluster 0 3.1 scrub ok 2024-03-11T21:17:51.275 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:50 smithi023 bash[21372]: cluster 2024-03-11T21:17:50.522897+0000 mgr.y (mgr.24380) 998 : cluster 0 pgmap v1164: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 892 B/s rd, 0 op/s 2024-03-11T21:17:51.275 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:50 smithi023 bash[29146]: cluster 2024-03-11T21:17:49.160807+0000 osd.7 (osd.7) 641 : cluster 0 1.e scrub starts 2024-03-11T21:17:51.275 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:50 smithi023 bash[29146]: cluster 2024-03-11T21:17:49.163484+0000 osd.7 (osd.7) 642 : cluster 0 1.e scrub ok 2024-03-11T21:17:51.276 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:50 smithi023 bash[29146]: cluster 2024-03-11T21:17:50.207638+0000 osd.7 (osd.7) 643 : cluster 0 3.13 scrub starts 2024-03-11T21:17:51.276 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:50 smithi023 bash[29146]: cluster 2024-03-11T21:17:50.211823+0000 osd.7 (osd.7) 644 : cluster 0 3.13 scrub ok 2024-03-11T21:17:51.276 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:50 smithi023 bash[29146]: cluster 2024-03-11T21:17:50.416110+0000 osd.0 (osd.0) 470 : cluster 0 3.1 scrub starts 2024-03-11T21:17:51.276 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:50 smithi023 bash[29146]: cluster 2024-03-11T21:17:50.420835+0000 osd.0 (osd.0) 471 : cluster 0 3.1 scrub ok 2024-03-11T21:17:51.276 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:50 smithi023 bash[29146]: cluster 2024-03-11T21:17:50.522897+0000 mgr.y (mgr.24380) 998 : cluster 0 pgmap v1164: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 892 B/s rd, 0 op/s 2024-03-11T21:17:52.163 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:17:52.186 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:51 smithi023 bash[21372]: cluster 2024-03-11T21:17:50.492094+0000 osd.3 (osd.3) 564 : cluster 0 4.11 scrub starts 2024-03-11T21:17:52.186 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:51 smithi023 bash[21372]: cluster 2024-03-11T21:17:50.495603+0000 osd.3 (osd.3) 565 : cluster 0 4.11 scrub ok 2024-03-11T21:17:52.186 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:51 smithi023 bash[21372]: cluster 2024-03-11T21:17:50.934550+0000 osd.6 (osd.6) 516 : cluster 0 5.2 scrub starts 2024-03-11T21:17:52.186 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:51 smithi023 bash[21372]: cluster 2024-03-11T21:17:50.982612+0000 osd.6 (osd.6) 517 : cluster 0 5.2 scrub ok 2024-03-11T21:17:52.186 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:51 smithi023 bash[29146]: cluster 2024-03-11T21:17:50.492094+0000 osd.3 (osd.3) 564 : cluster 0 4.11 scrub starts 2024-03-11T21:17:52.186 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:51 smithi023 bash[29146]: cluster 2024-03-11T21:17:50.495603+0000 osd.3 (osd.3) 565 : cluster 0 4.11 scrub ok 2024-03-11T21:17:52.186 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:51 smithi023 bash[29146]: cluster 2024-03-11T21:17:50.934550+0000 osd.6 (osd.6) 516 : cluster 0 5.2 scrub starts 2024-03-11T21:17:52.186 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:51 smithi023 bash[29146]: cluster 2024-03-11T21:17:50.982612+0000 osd.6 (osd.6) 517 : cluster 0 5.2 scrub ok 2024-03-11T21:17:52.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:51 smithi146 bash[23809]: cluster 2024-03-11T21:17:50.492094+0000 osd.3 (osd.3) 564 : cluster 0 4.11 scrub starts 2024-03-11T21:17:52.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:51 smithi146 bash[23809]: cluster 2024-03-11T21:17:50.495603+0000 osd.3 (osd.3) 565 : cluster 0 4.11 scrub ok 2024-03-11T21:17:52.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:51 smithi146 bash[23809]: cluster 2024-03-11T21:17:50.934550+0000 osd.6 (osd.6) 516 : cluster 0 5.2 scrub starts 2024-03-11T21:17:52.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:51 smithi146 bash[23809]: cluster 2024-03-11T21:17:50.982612+0000 osd.6 (osd.6) 517 : cluster 0 5.2 scrub ok 2024-03-11T21:17:53.188 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:52 smithi023 bash[21372]: audit 2024-03-11T21:17:51.863509+0000 mon.a (mon.0) 2059 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:17:53.188 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:52 smithi023 bash[21372]: cluster 2024-03-11T21:17:52.524238+0000 mgr.y (mgr.24380) 999 : cluster 0 pgmap v1165: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-11T21:17:53.189 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:52 smithi023 bash[29146]: audit 2024-03-11T21:17:51.863509+0000 mon.a (mon.0) 2059 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:17:53.189 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:52 smithi023 bash[29146]: cluster 2024-03-11T21:17:52.524238+0000 mgr.y (mgr.24380) 999 : cluster 0 pgmap v1165: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-11T21:17:53.225 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:52 smithi146 bash[23809]: audit 2024-03-11T21:17:51.863509+0000 mon.a (mon.0) 2059 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:17:53.226 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:52 smithi146 bash[23809]: cluster 2024-03-11T21:17:52.524238+0000 mgr.y (mgr.24380) 999 : cluster 0 pgmap v1165: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-11T21:17:53.409 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-03-11T21:17:53.409 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:17:54.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:53 smithi146 bash[23809]: cluster 2024-03-11T21:17:52.478932+0000 osd.3 (osd.3) 566 : cluster 0 2.1b deep-scrub starts 2024-03-11T21:17:54.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:53 smithi146 bash[23809]: cluster 2024-03-11T21:17:52.483974+0000 osd.3 (osd.3) 567 : cluster 0 2.1b deep-scrub ok 2024-03-11T21:17:54.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:53 smithi146 bash[23809]: cluster 2024-03-11T21:17:52.977428+0000 osd.1 (osd.1) 382 : cluster 0 4.f scrub starts 2024-03-11T21:17:54.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:53 smithi146 bash[23809]: cluster 2024-03-11T21:17:52.980142+0000 osd.1 (osd.1) 383 : cluster 0 4.f scrub ok 2024-03-11T21:17:54.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:53 smithi146 bash[23809]: cluster 2024-03-11T21:17:53.181301+0000 osd.7 (osd.7) 645 : cluster 0 3.15 scrub starts 2024-03-11T21:17:54.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:53 smithi146 bash[23809]: cluster 2024-03-11T21:17:53.185602+0000 osd.7 (osd.7) 646 : cluster 0 3.15 scrub ok 2024-03-11T21:17:54.111 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:53 smithi023 bash[21372]: cluster 2024-03-11T21:17:52.478932+0000 osd.3 (osd.3) 566 : cluster 0 2.1b deep-scrub starts 2024-03-11T21:17:54.111 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:53 smithi023 bash[21372]: cluster 2024-03-11T21:17:52.483974+0000 osd.3 (osd.3) 567 : cluster 0 2.1b deep-scrub ok 2024-03-11T21:17:54.111 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:53 smithi023 bash[21372]: cluster 2024-03-11T21:17:52.977428+0000 osd.1 (osd.1) 382 : cluster 0 4.f scrub starts 2024-03-11T21:17:54.111 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:53 smithi023 bash[21372]: cluster 2024-03-11T21:17:52.980142+0000 osd.1 (osd.1) 383 : cluster 0 4.f scrub ok 2024-03-11T21:17:54.111 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:53 smithi023 bash[21372]: cluster 2024-03-11T21:17:53.181301+0000 osd.7 (osd.7) 645 : cluster 0 3.15 scrub starts 2024-03-11T21:17:54.112 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:53 smithi023 bash[21372]: cluster 2024-03-11T21:17:53.185602+0000 osd.7 (osd.7) 646 : cluster 0 3.15 scrub ok 2024-03-11T21:17:54.112 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:53 smithi023 bash[29146]: cluster 2024-03-11T21:17:52.478932+0000 osd.3 (osd.3) 566 : cluster 0 2.1b deep-scrub starts 2024-03-11T21:17:54.112 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:53 smithi023 bash[29146]: cluster 2024-03-11T21:17:52.483974+0000 osd.3 (osd.3) 567 : cluster 0 2.1b deep-scrub ok 2024-03-11T21:17:54.112 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:53 smithi023 bash[29146]: cluster 2024-03-11T21:17:52.977428+0000 osd.1 (osd.1) 382 : cluster 0 4.f scrub starts 2024-03-11T21:17:54.112 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:53 smithi023 bash[29146]: cluster 2024-03-11T21:17:52.980142+0000 osd.1 (osd.1) 383 : cluster 0 4.f scrub ok 2024-03-11T21:17:54.112 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:53 smithi023 bash[29146]: cluster 2024-03-11T21:17:53.181301+0000 osd.7 (osd.7) 645 : cluster 0 3.15 scrub starts 2024-03-11T21:17:54.112 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:53 smithi023 bash[29146]: cluster 2024-03-11T21:17:53.185602+0000 osd.7 (osd.7) 646 : cluster 0 3.15 scrub ok 2024-03-11T21:17:54.178 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-03-11T21:17:55.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:54 smithi146 bash[23809]: cluster 2024-03-11T21:17:53.327595+0000 osd.3 (osd.3) 568 : cluster 0 purged_snaps scrub starts 2024-03-11T21:17:55.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:54 smithi146 bash[23809]: cluster 2024-03-11T21:17:53.327674+0000 osd.3 (osd.3) 569 : cluster 0 purged_snaps scrub ok 2024-03-11T21:17:55.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:54 smithi146 bash[23809]: cluster 2024-03-11T21:17:53.520045+0000 osd.3 (osd.3) 570 : cluster 0 3.2 deep-scrub starts 2024-03-11T21:17:55.254 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:54 smithi146 bash[23809]: cluster 2024-03-11T21:17:53.522661+0000 osd.3 (osd.3) 571 : cluster 0 3.2 deep-scrub ok 2024-03-11T21:17:55.254 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:54 smithi146 bash[23809]: cluster 2024-03-11T21:17:53.844417+0000 mon.a (mon.0) 2060 : cluster 0 osdmap e475: 8 total, 8 up, 8 in 2024-03-11T21:17:55.254 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:54 smithi146 bash[23809]: cluster 2024-03-11T21:17:54.016839+0000 osd.1 (osd.1) 384 : cluster 0 3.4 scrub starts 2024-03-11T21:17:55.254 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:54 smithi146 bash[23809]: cluster 2024-03-11T21:17:54.022250+0000 osd.1 (osd.1) 385 : cluster 0 3.4 scrub ok 2024-03-11T21:17:55.254 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:54 smithi146 bash[23809]: cluster 2024-03-11T21:17:54.427412+0000 osd.0 (osd.0) 472 : cluster 0 4.6 scrub starts 2024-03-11T21:17:55.254 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:54 smithi146 bash[23809]: cluster 2024-03-11T21:17:54.431390+0000 osd.0 (osd.0) 473 : cluster 0 4.6 scrub ok 2024-03-11T21:17:55.254 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:54 smithi146 bash[23809]: cluster 2024-03-11T21:17:54.525044+0000 mgr.y (mgr.24380) 1000 : cluster 0 pgmap v1167: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:17:55.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:54 smithi023 bash[21372]: cluster 2024-03-11T21:17:53.327595+0000 osd.3 (osd.3) 568 : cluster 0 purged_snaps scrub starts 2024-03-11T21:17:55.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:54 smithi023 bash[21372]: cluster 2024-03-11T21:17:53.327674+0000 osd.3 (osd.3) 569 : cluster 0 purged_snaps scrub ok 2024-03-11T21:17:55.275 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:54 smithi023 bash[21372]: cluster 2024-03-11T21:17:53.520045+0000 osd.3 (osd.3) 570 : cluster 0 3.2 deep-scrub starts 2024-03-11T21:17:55.275 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:54 smithi023 bash[21372]: cluster 2024-03-11T21:17:53.522661+0000 osd.3 (osd.3) 571 : cluster 0 3.2 deep-scrub ok 2024-03-11T21:17:55.275 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:54 smithi023 bash[21372]: cluster 2024-03-11T21:17:53.844417+0000 mon.a (mon.0) 2060 : cluster 0 osdmap e475: 8 total, 8 up, 8 in 2024-03-11T21:17:55.275 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:54 smithi023 bash[21372]: cluster 2024-03-11T21:17:54.016839+0000 osd.1 (osd.1) 384 : cluster 0 3.4 scrub starts 2024-03-11T21:17:55.275 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:54 smithi023 bash[21372]: cluster 2024-03-11T21:17:54.022250+0000 osd.1 (osd.1) 385 : cluster 0 3.4 scrub ok 2024-03-11T21:17:55.275 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:54 smithi023 bash[21372]: cluster 2024-03-11T21:17:54.427412+0000 osd.0 (osd.0) 472 : cluster 0 4.6 scrub starts 2024-03-11T21:17:55.275 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:54 smithi023 bash[21372]: cluster 2024-03-11T21:17:54.431390+0000 osd.0 (osd.0) 473 : cluster 0 4.6 scrub ok 2024-03-11T21:17:55.275 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:54 smithi023 bash[21372]: cluster 2024-03-11T21:17:54.525044+0000 mgr.y (mgr.24380) 1000 : cluster 0 pgmap v1167: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:17:55.275 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:54 smithi023 bash[29146]: cluster 2024-03-11T21:17:53.327595+0000 osd.3 (osd.3) 568 : cluster 0 purged_snaps scrub starts 2024-03-11T21:17:55.275 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:54 smithi023 bash[29146]: cluster 2024-03-11T21:17:53.327674+0000 osd.3 (osd.3) 569 : cluster 0 purged_snaps scrub ok 2024-03-11T21:17:55.275 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:54 smithi023 bash[29146]: cluster 2024-03-11T21:17:53.520045+0000 osd.3 (osd.3) 570 : cluster 0 3.2 deep-scrub starts 2024-03-11T21:17:55.275 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:54 smithi023 bash[29146]: cluster 2024-03-11T21:17:53.522661+0000 osd.3 (osd.3) 571 : cluster 0 3.2 deep-scrub ok 2024-03-11T21:17:55.276 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:54 smithi023 bash[29146]: cluster 2024-03-11T21:17:53.844417+0000 mon.a (mon.0) 2060 : cluster 0 osdmap e475: 8 total, 8 up, 8 in 2024-03-11T21:17:55.276 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:54 smithi023 bash[29146]: cluster 2024-03-11T21:17:54.016839+0000 osd.1 (osd.1) 384 : cluster 0 3.4 scrub starts 2024-03-11T21:17:55.276 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:54 smithi023 bash[29146]: cluster 2024-03-11T21:17:54.022250+0000 osd.1 (osd.1) 385 : cluster 0 3.4 scrub ok 2024-03-11T21:17:55.276 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:54 smithi023 bash[29146]: cluster 2024-03-11T21:17:54.427412+0000 osd.0 (osd.0) 472 : cluster 0 4.6 scrub starts 2024-03-11T21:17:55.276 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:54 smithi023 bash[29146]: cluster 2024-03-11T21:17:54.431390+0000 osd.0 (osd.0) 473 : cluster 0 4.6 scrub ok 2024-03-11T21:17:55.276 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:54 smithi023 bash[29146]: cluster 2024-03-11T21:17:54.525044+0000 mgr.y (mgr.24380) 1000 : cluster 0 pgmap v1167: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:17:55.881 INFO:teuthology.orchestra.run.smithi023.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-03-11T21:17:56.211 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:55 smithi146 bash[23809]: cluster 2024-03-11T21:17:55.010753+0000 osd.1 (osd.1) 386 : cluster 0 2.d scrub starts 2024-03-11T21:17:56.211 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:55 smithi146 bash[23809]: cluster 2024-03-11T21:17:55.012470+0000 osd.1 (osd.1) 387 : cluster 0 2.d scrub ok 2024-03-11T21:17:56.211 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:55 smithi146 bash[23809]: cluster 2024-03-11T21:17:55.381167+0000 osd.0 (osd.0) 474 : cluster 0 4.1e scrub starts 2024-03-11T21:17:56.211 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:55 smithi146 bash[23809]: cluster 2024-03-11T21:17:55.383401+0000 osd.0 (osd.0) 475 : cluster 0 4.1e scrub ok 2024-03-11T21:17:56.212 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:55 smithi146 bash[23809]: audit 2024-03-11T21:17:55.387960+0000 mon.a (mon.0) 2061 : audit 1 from='client.? 172.21.15.23:0/3760215949' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-03-11T21:17:56.212 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:55 smithi146 bash[23809]: cluster 2024-03-11T21:17:55.424164+0000 osd.5 (osd.5) 621 : cluster 0 3.5 scrub starts 2024-03-11T21:17:56.212 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:55 smithi146 bash[23809]: cluster 2024-03-11T21:17:55.428118+0000 osd.5 (osd.5) 622 : cluster 0 3.5 scrub ok 2024-03-11T21:17:56.275 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:55 smithi023 bash[21372]: cluster 2024-03-11T21:17:55.010753+0000 osd.1 (osd.1) 386 : cluster 0 2.d scrub starts 2024-03-11T21:17:56.275 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:55 smithi023 bash[21372]: cluster 2024-03-11T21:17:55.012470+0000 osd.1 (osd.1) 387 : cluster 0 2.d scrub ok 2024-03-11T21:17:56.275 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:55 smithi023 bash[21372]: cluster 2024-03-11T21:17:55.381167+0000 osd.0 (osd.0) 474 : cluster 0 4.1e scrub starts 2024-03-11T21:17:56.275 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:55 smithi023 bash[21372]: cluster 2024-03-11T21:17:55.383401+0000 osd.0 (osd.0) 475 : cluster 0 4.1e scrub ok 2024-03-11T21:17:56.275 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:55 smithi023 bash[21372]: audit 2024-03-11T21:17:55.387960+0000 mon.a (mon.0) 2061 : audit 1 from='client.? 172.21.15.23:0/3760215949' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-03-11T21:17:56.275 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:55 smithi023 bash[21372]: cluster 2024-03-11T21:17:55.424164+0000 osd.5 (osd.5) 621 : cluster 0 3.5 scrub starts 2024-03-11T21:17:56.275 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:55 smithi023 bash[21372]: cluster 2024-03-11T21:17:55.428118+0000 osd.5 (osd.5) 622 : cluster 0 3.5 scrub ok 2024-03-11T21:17:56.275 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:55 smithi023 bash[29146]: cluster 2024-03-11T21:17:55.010753+0000 osd.1 (osd.1) 386 : cluster 0 2.d scrub starts 2024-03-11T21:17:56.275 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:55 smithi023 bash[29146]: cluster 2024-03-11T21:17:55.012470+0000 osd.1 (osd.1) 387 : cluster 0 2.d scrub ok 2024-03-11T21:17:56.276 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:55 smithi023 bash[29146]: cluster 2024-03-11T21:17:55.381167+0000 osd.0 (osd.0) 474 : cluster 0 4.1e scrub starts 2024-03-11T21:17:56.276 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:55 smithi023 bash[29146]: cluster 2024-03-11T21:17:55.383401+0000 osd.0 (osd.0) 475 : cluster 0 4.1e scrub ok 2024-03-11T21:17:56.276 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:55 smithi023 bash[29146]: audit 2024-03-11T21:17:55.387960+0000 mon.a (mon.0) 2061 : audit 1 from='client.? 172.21.15.23:0/3760215949' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-03-11T21:17:56.276 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:55 smithi023 bash[29146]: cluster 2024-03-11T21:17:55.424164+0000 osd.5 (osd.5) 621 : cluster 0 3.5 scrub starts 2024-03-11T21:17:56.276 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:55 smithi023 bash[29146]: cluster 2024-03-11T21:17:55.428118+0000 osd.5 (osd.5) 622 : cluster 0 3.5 scrub ok 2024-03-11T21:17:56.503 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:17:56 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:17:56.783 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd reweight 6 1 2024-03-11T21:17:57.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:56 smithi146 bash[23809]: cluster 2024-03-11T21:17:55.153521+0000 osd.2 (osd.2) 281 : cluster 0 2.10 scrub starts 2024-03-11T21:17:57.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:56 smithi146 bash[23809]: cluster 2024-03-11T21:17:55.154556+0000 osd.2 (osd.2) 282 : cluster 0 2.10 scrub ok 2024-03-11T21:17:57.254 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:56 smithi146 bash[23809]: audit 2024-03-11T21:17:55.886305+0000 mon.a (mon.0) 2062 : audit 1 from='client.? 172.21.15.23:0/3760215949' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-03-11T21:17:57.254 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:56 smithi146 bash[23809]: cluster 2024-03-11T21:17:55.896156+0000 mon.a (mon.0) 2063 : cluster 0 osdmap e476: 8 total, 8 up, 8 in 2024-03-11T21:17:57.254 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:56 smithi146 bash[23809]: cluster 2024-03-11T21:17:55.996089+0000 osd.6 (osd.6) 518 : cluster 0 6.1e scrub starts 2024-03-11T21:17:57.254 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:56 smithi146 bash[23809]: cluster 2024-03-11T21:17:56.034254+0000 osd.6 (osd.6) 519 : cluster 0 6.1e scrub ok 2024-03-11T21:17:57.254 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:56 smithi146 bash[23809]: audit 2024-03-11T21:17:56.224135+0000 mgr.y (mgr.24380) 1001 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:17:57.254 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:56 smithi146 bash[23809]: cluster 2024-03-11T21:17:56.375325+0000 osd.0 (osd.0) 476 : cluster 0 6.10 scrub starts 2024-03-11T21:17:57.254 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:56 smithi146 bash[23809]: cluster 2024-03-11T21:17:56.458825+0000 osd.0 (osd.0) 477 : cluster 0 6.10 scrub ok 2024-03-11T21:17:57.254 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:56 smithi146 bash[23809]: cluster 2024-03-11T21:17:56.525952+0000 mgr.y (mgr.24380) 1002 : cluster 0 pgmap v1169: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-11T21:17:57.275 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:56 smithi023 bash[21372]: cluster 2024-03-11T21:17:55.153521+0000 osd.2 (osd.2) 281 : cluster 0 2.10 scrub starts 2024-03-11T21:17:57.275 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:56 smithi023 bash[21372]: cluster 2024-03-11T21:17:55.154556+0000 osd.2 (osd.2) 282 : cluster 0 2.10 scrub ok 2024-03-11T21:17:57.275 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:56 smithi023 bash[21372]: audit 2024-03-11T21:17:55.886305+0000 mon.a (mon.0) 2062 : audit 1 from='client.? 172.21.15.23:0/3760215949' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-03-11T21:17:57.275 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:56 smithi023 bash[21372]: cluster 2024-03-11T21:17:55.896156+0000 mon.a (mon.0) 2063 : cluster 0 osdmap e476: 8 total, 8 up, 8 in 2024-03-11T21:17:57.275 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:56 smithi023 bash[21372]: cluster 2024-03-11T21:17:55.996089+0000 osd.6 (osd.6) 518 : cluster 0 6.1e scrub starts 2024-03-11T21:17:57.275 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:56 smithi023 bash[21372]: cluster 2024-03-11T21:17:56.034254+0000 osd.6 (osd.6) 519 : cluster 0 6.1e scrub ok 2024-03-11T21:17:57.275 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:56 smithi023 bash[21372]: audit 2024-03-11T21:17:56.224135+0000 mgr.y (mgr.24380) 1001 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:17:57.275 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:56 smithi023 bash[21372]: cluster 2024-03-11T21:17:56.375325+0000 osd.0 (osd.0) 476 : cluster 0 6.10 scrub starts 2024-03-11T21:17:57.275 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:56 smithi023 bash[21372]: cluster 2024-03-11T21:17:56.458825+0000 osd.0 (osd.0) 477 : cluster 0 6.10 scrub ok 2024-03-11T21:17:57.275 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:56 smithi023 bash[21372]: cluster 2024-03-11T21:17:56.525952+0000 mgr.y (mgr.24380) 1002 : cluster 0 pgmap v1169: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-11T21:17:57.276 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:56 smithi023 bash[29146]: cluster 2024-03-11T21:17:55.153521+0000 osd.2 (osd.2) 281 : cluster 0 2.10 scrub starts 2024-03-11T21:17:57.276 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:56 smithi023 bash[29146]: cluster 2024-03-11T21:17:55.154556+0000 osd.2 (osd.2) 282 : cluster 0 2.10 scrub ok 2024-03-11T21:17:57.276 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:56 smithi023 bash[29146]: audit 2024-03-11T21:17:55.886305+0000 mon.a (mon.0) 2062 : audit 1 from='client.? 172.21.15.23:0/3760215949' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-03-11T21:17:57.276 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:56 smithi023 bash[29146]: cluster 2024-03-11T21:17:55.896156+0000 mon.a (mon.0) 2063 : cluster 0 osdmap e476: 8 total, 8 up, 8 in 2024-03-11T21:17:57.276 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:56 smithi023 bash[29146]: cluster 2024-03-11T21:17:55.996089+0000 osd.6 (osd.6) 518 : cluster 0 6.1e scrub starts 2024-03-11T21:17:57.276 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:56 smithi023 bash[29146]: cluster 2024-03-11T21:17:56.034254+0000 osd.6 (osd.6) 519 : cluster 0 6.1e scrub ok 2024-03-11T21:17:57.276 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:56 smithi023 bash[29146]: audit 2024-03-11T21:17:56.224135+0000 mgr.y (mgr.24380) 1001 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:17:57.276 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:56 smithi023 bash[29146]: cluster 2024-03-11T21:17:56.375325+0000 osd.0 (osd.0) 476 : cluster 0 6.10 scrub starts 2024-03-11T21:17:57.276 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:56 smithi023 bash[29146]: cluster 2024-03-11T21:17:56.458825+0000 osd.0 (osd.0) 477 : cluster 0 6.10 scrub ok 2024-03-11T21:17:57.276 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:56 smithi023 bash[29146]: cluster 2024-03-11T21:17:56.525952+0000 mgr.y (mgr.24380) 1002 : cluster 0 pgmap v1169: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-11T21:17:58.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:57 smithi146 bash[23809]: cluster 2024-03-11T21:17:57.369536+0000 osd.0 (osd.0) 478 : cluster 0 6.0 scrub starts 2024-03-11T21:17:58.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:57 smithi146 bash[23809]: cluster 2024-03-11T21:17:57.452775+0000 osd.0 (osd.0) 479 : cluster 0 6.0 scrub ok 2024-03-11T21:17:58.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:57 smithi023 bash[21372]: cluster 2024-03-11T21:17:57.369536+0000 osd.0 (osd.0) 478 : cluster 0 6.0 scrub starts 2024-03-11T21:17:58.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:57 smithi023 bash[21372]: cluster 2024-03-11T21:17:57.452775+0000 osd.0 (osd.0) 479 : cluster 0 6.0 scrub ok 2024-03-11T21:17:58.275 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:57 smithi023 bash[29146]: cluster 2024-03-11T21:17:57.369536+0000 osd.0 (osd.0) 478 : cluster 0 6.0 scrub starts 2024-03-11T21:17:58.275 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:57 smithi023 bash[29146]: cluster 2024-03-11T21:17:57.452775+0000 osd.0 (osd.0) 479 : cluster 0 6.0 scrub ok 2024-03-11T21:17:58.774 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:17:58 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:17:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:17:58.976 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:17:59.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:58 smithi146 bash[23809]: cluster 2024-03-11T21:17:58.526961+0000 mgr.y (mgr.24380) 1003 : cluster 0 pgmap v1170: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:17:59.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:58 smithi023 bash[21372]: cluster 2024-03-11T21:17:58.526961+0000 mgr.y (mgr.24380) 1003 : cluster 0 pgmap v1170: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:17:59.274 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:58 smithi023 bash[29146]: cluster 2024-03-11T21:17:58.526961+0000 mgr.y (mgr.24380) 1003 : cluster 0 pgmap v1170: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:18:00.182 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:59 smithi146 bash[23809]: cluster 2024-03-11T21:17:59.352810+0000 osd.0 (osd.0) 480 : cluster 0 3.3 deep-scrub starts 2024-03-11T21:18:00.182 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:59 smithi146 bash[23809]: cluster 2024-03-11T21:17:59.358899+0000 osd.0 (osd.0) 481 : cluster 0 3.3 deep-scrub ok 2024-03-11T21:18:00.182 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:59 smithi146 bash[23809]: cluster 2024-03-11T21:17:59.401469+0000 osd.5 (osd.5) 623 : cluster 0 2.16 deep-scrub starts 2024-03-11T21:18:00.182 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:17:59 smithi146 bash[23809]: cluster 2024-03-11T21:17:59.403858+0000 osd.5 (osd.5) 624 : cluster 0 2.16 deep-scrub ok 2024-03-11T21:18:00.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:59 smithi023 bash[21372]: cluster 2024-03-11T21:17:59.352810+0000 osd.0 (osd.0) 480 : cluster 0 3.3 deep-scrub starts 2024-03-11T21:18:00.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:59 smithi023 bash[21372]: cluster 2024-03-11T21:17:59.358899+0000 osd.0 (osd.0) 481 : cluster 0 3.3 deep-scrub ok 2024-03-11T21:18:00.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:59 smithi023 bash[21372]: cluster 2024-03-11T21:17:59.401469+0000 osd.5 (osd.5) 623 : cluster 0 2.16 deep-scrub starts 2024-03-11T21:18:00.275 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:17:59 smithi023 bash[21372]: cluster 2024-03-11T21:17:59.403858+0000 osd.5 (osd.5) 624 : cluster 0 2.16 deep-scrub ok 2024-03-11T21:18:00.275 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:59 smithi023 bash[29146]: cluster 2024-03-11T21:17:59.352810+0000 osd.0 (osd.0) 480 : cluster 0 3.3 deep-scrub starts 2024-03-11T21:18:00.275 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:59 smithi023 bash[29146]: cluster 2024-03-11T21:17:59.358899+0000 osd.0 (osd.0) 481 : cluster 0 3.3 deep-scrub ok 2024-03-11T21:18:00.275 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:59 smithi023 bash[29146]: cluster 2024-03-11T21:17:59.401469+0000 osd.5 (osd.5) 623 : cluster 0 2.16 deep-scrub starts 2024-03-11T21:18:00.275 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:17:59 smithi023 bash[29146]: cluster 2024-03-11T21:17:59.403858+0000 osd.5 (osd.5) 624 : cluster 0 2.16 deep-scrub ok 2024-03-11T21:18:01.581 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:18:01.753 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:01 smithi146 bash[23809]: cluster 2024-03-11T21:18:00.035973+0000 osd.6 (osd.6) 520 : cluster 0 3.a scrub starts 2024-03-11T21:18:01.753 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:01 smithi146 bash[23809]: cluster 2024-03-11T21:18:00.041365+0000 osd.6 (osd.6) 521 : cluster 0 3.a scrub ok 2024-03-11T21:18:01.753 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:01 smithi146 bash[23809]: cluster 2024-03-11T21:18:00.528155+0000 mgr.y (mgr.24380) 1004 : cluster 0 pgmap v1171: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:18:01.753 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:01 smithi146 bash[23809]: cluster 2024-03-11T21:18:01.042166+0000 osd.1 (osd.1) 388 : cluster 0 4.7 scrub starts 2024-03-11T21:18:01.753 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:01 smithi146 bash[23809]: cluster 2024-03-11T21:18:01.044885+0000 osd.1 (osd.1) 389 : cluster 0 4.7 scrub ok 2024-03-11T21:18:01.774 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:01 smithi023 bash[21372]: cluster 2024-03-11T21:18:00.035973+0000 osd.6 (osd.6) 520 : cluster 0 3.a scrub starts 2024-03-11T21:18:01.774 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:01 smithi023 bash[21372]: cluster 2024-03-11T21:18:00.041365+0000 osd.6 (osd.6) 521 : cluster 0 3.a scrub ok 2024-03-11T21:18:01.774 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:01 smithi023 bash[21372]: cluster 2024-03-11T21:18:00.528155+0000 mgr.y (mgr.24380) 1004 : cluster 0 pgmap v1171: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:18:01.774 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:01 smithi023 bash[21372]: cluster 2024-03-11T21:18:01.042166+0000 osd.1 (osd.1) 388 : cluster 0 4.7 scrub starts 2024-03-11T21:18:01.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:01 smithi023 bash[21372]: cluster 2024-03-11T21:18:01.044885+0000 osd.1 (osd.1) 389 : cluster 0 4.7 scrub ok 2024-03-11T21:18:01.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:01 smithi023 bash[29146]: cluster 2024-03-11T21:18:00.035973+0000 osd.6 (osd.6) 520 : cluster 0 3.a scrub starts 2024-03-11T21:18:01.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:01 smithi023 bash[29146]: cluster 2024-03-11T21:18:00.041365+0000 osd.6 (osd.6) 521 : cluster 0 3.a scrub ok 2024-03-11T21:18:01.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:01 smithi023 bash[29146]: cluster 2024-03-11T21:18:00.528155+0000 mgr.y (mgr.24380) 1004 : cluster 0 pgmap v1171: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:18:01.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:01 smithi023 bash[29146]: cluster 2024-03-11T21:18:01.042166+0000 osd.1 (osd.1) 388 : cluster 0 4.7 scrub starts 2024-03-11T21:18:01.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:01 smithi023 bash[29146]: cluster 2024-03-11T21:18:01.044885+0000 osd.1 (osd.1) 389 : cluster 0 4.7 scrub ok 2024-03-11T21:18:02.030 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-03-11T21:18:02.030 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:18:02.747 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:02 smithi023 bash[21372]: cluster 2024-03-11T21:18:01.067035+0000 osd.6 (osd.6) 522 : cluster 0 2.17 scrub starts 2024-03-11T21:18:02.748 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:02 smithi023 bash[21372]: cluster 2024-03-11T21:18:01.068842+0000 osd.6 (osd.6) 523 : cluster 0 2.17 scrub ok 2024-03-11T21:18:02.748 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:02 smithi023 bash[21372]: cluster 2024-03-11T21:18:01.640239+0000 osd.3 (osd.3) 572 : cluster 0 1.4 deep-scrub starts 2024-03-11T21:18:02.748 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:02 smithi023 bash[21372]: cluster 2024-03-11T21:18:01.642187+0000 osd.3 (osd.3) 573 : cluster 0 1.4 deep-scrub ok 2024-03-11T21:18:02.748 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:02 smithi023 bash[21372]: cluster 2024-03-11T21:18:02.085002+0000 osd.1 (osd.1) 390 : cluster 0 6.e scrub starts 2024-03-11T21:18:02.748 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:02 smithi023 bash[21372]: cluster 2024-03-11T21:18:02.154935+0000 osd.1 (osd.1) 391 : cluster 0 6.e scrub ok 2024-03-11T21:18:02.748 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:02 smithi023 bash[29146]: cluster 2024-03-11T21:18:01.067035+0000 osd.6 (osd.6) 522 : cluster 0 2.17 scrub starts 2024-03-11T21:18:02.748 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:02 smithi023 bash[29146]: cluster 2024-03-11T21:18:01.068842+0000 osd.6 (osd.6) 523 : cluster 0 2.17 scrub ok 2024-03-11T21:18:02.748 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:02 smithi023 bash[29146]: cluster 2024-03-11T21:18:01.640239+0000 osd.3 (osd.3) 572 : cluster 0 1.4 deep-scrub starts 2024-03-11T21:18:02.748 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:02 smithi023 bash[29146]: cluster 2024-03-11T21:18:01.642187+0000 osd.3 (osd.3) 573 : cluster 0 1.4 deep-scrub ok 2024-03-11T21:18:02.748 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:02 smithi023 bash[29146]: cluster 2024-03-11T21:18:02.085002+0000 osd.1 (osd.1) 390 : cluster 0 6.e scrub starts 2024-03-11T21:18:02.748 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:02 smithi023 bash[29146]: cluster 2024-03-11T21:18:02.154935+0000 osd.1 (osd.1) 391 : cluster 0 6.e scrub ok 2024-03-11T21:18:02.753 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:02 smithi146 bash[23809]: cluster 2024-03-11T21:18:01.067035+0000 osd.6 (osd.6) 522 : cluster 0 2.17 scrub starts 2024-03-11T21:18:02.753 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:02 smithi146 bash[23809]: cluster 2024-03-11T21:18:01.068842+0000 osd.6 (osd.6) 523 : cluster 0 2.17 scrub ok 2024-03-11T21:18:02.753 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:02 smithi146 bash[23809]: cluster 2024-03-11T21:18:01.640239+0000 osd.3 (osd.3) 572 : cluster 0 1.4 deep-scrub starts 2024-03-11T21:18:02.753 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:02 smithi146 bash[23809]: cluster 2024-03-11T21:18:01.642187+0000 osd.3 (osd.3) 573 : cluster 0 1.4 deep-scrub ok 2024-03-11T21:18:02.753 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:02 smithi146 bash[23809]: cluster 2024-03-11T21:18:02.085002+0000 osd.1 (osd.1) 390 : cluster 0 6.e scrub starts 2024-03-11T21:18:02.753 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:02 smithi146 bash[23809]: cluster 2024-03-11T21:18:02.154935+0000 osd.1 (osd.1) 391 : cluster 0 6.e scrub ok 2024-03-11T21:18:02.824 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-03-11T21:18:03.753 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:03 smithi146 bash[23809]: cluster 2024-03-11T21:18:02.529583+0000 mgr.y (mgr.24380) 1005 : cluster 0 pgmap v1172: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:18:03.753 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:03 smithi146 bash[23809]: cluster 2024-03-11T21:18:02.686615+0000 osd.3 (osd.3) 574 : cluster 0 4.19 scrub starts 2024-03-11T21:18:03.753 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:03 smithi146 bash[23809]: cluster 2024-03-11T21:18:02.689074+0000 osd.3 (osd.3) 575 : cluster 0 4.19 scrub ok 2024-03-11T21:18:03.774 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:03 smithi023 bash[21372]: cluster 2024-03-11T21:18:02.529583+0000 mgr.y (mgr.24380) 1005 : cluster 0 pgmap v1172: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:18:03.774 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:03 smithi023 bash[21372]: cluster 2024-03-11T21:18:02.686615+0000 osd.3 (osd.3) 574 : cluster 0 4.19 scrub starts 2024-03-11T21:18:03.774 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:03 smithi023 bash[21372]: cluster 2024-03-11T21:18:02.689074+0000 osd.3 (osd.3) 575 : cluster 0 4.19 scrub ok 2024-03-11T21:18:03.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:03 smithi023 bash[29146]: cluster 2024-03-11T21:18:02.529583+0000 mgr.y (mgr.24380) 1005 : cluster 0 pgmap v1172: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:18:03.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:03 smithi023 bash[29146]: cluster 2024-03-11T21:18:02.686615+0000 osd.3 (osd.3) 574 : cluster 0 4.19 scrub starts 2024-03-11T21:18:03.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:03 smithi023 bash[29146]: cluster 2024-03-11T21:18:02.689074+0000 osd.3 (osd.3) 575 : cluster 0 4.19 scrub ok 2024-03-11T21:18:04.753 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:04 smithi146 bash[23809]: cluster 2024-03-11T21:18:03.039546+0000 osd.2 (osd.2) 283 : cluster 0 2.0 scrub starts 2024-03-11T21:18:04.753 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:04 smithi146 bash[23809]: cluster 2024-03-11T21:18:03.043607+0000 osd.2 (osd.2) 284 : cluster 0 2.0 scrub ok 2024-03-11T21:18:04.753 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:04 smithi146 bash[23809]: cluster 2024-03-11T21:18:03.436660+0000 osd.5 (osd.5) 625 : cluster 0 3.16 scrub starts 2024-03-11T21:18:04.753 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:04 smithi146 bash[23809]: cluster 2024-03-11T21:18:03.440575+0000 osd.5 (osd.5) 626 : cluster 0 3.16 scrub ok 2024-03-11T21:18:04.754 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:04 smithi146 bash[23809]: cluster 2024-03-11T21:18:04.006551+0000 osd.1 (osd.1) 392 : cluster 0 1.b scrub starts 2024-03-11T21:18:04.754 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:04 smithi146 bash[23809]: cluster 2024-03-11T21:18:04.009553+0000 osd.1 (osd.1) 393 : cluster 0 1.b scrub ok 2024-03-11T21:18:04.774 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:04 smithi023 bash[21372]: cluster 2024-03-11T21:18:03.039546+0000 osd.2 (osd.2) 283 : cluster 0 2.0 scrub starts 2024-03-11T21:18:04.774 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:04 smithi023 bash[21372]: cluster 2024-03-11T21:18:03.043607+0000 osd.2 (osd.2) 284 : cluster 0 2.0 scrub ok 2024-03-11T21:18:04.774 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:04 smithi023 bash[21372]: cluster 2024-03-11T21:18:03.436660+0000 osd.5 (osd.5) 625 : cluster 0 3.16 scrub starts 2024-03-11T21:18:04.774 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:04 smithi023 bash[21372]: cluster 2024-03-11T21:18:03.440575+0000 osd.5 (osd.5) 626 : cluster 0 3.16 scrub ok 2024-03-11T21:18:04.774 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:04 smithi023 bash[21372]: cluster 2024-03-11T21:18:04.006551+0000 osd.1 (osd.1) 392 : cluster 0 1.b scrub starts 2024-03-11T21:18:04.774 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:04 smithi023 bash[21372]: cluster 2024-03-11T21:18:04.009553+0000 osd.1 (osd.1) 393 : cluster 0 1.b scrub ok 2024-03-11T21:18:04.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:04 smithi023 bash[29146]: cluster 2024-03-11T21:18:03.039546+0000 osd.2 (osd.2) 283 : cluster 0 2.0 scrub starts 2024-03-11T21:18:04.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:04 smithi023 bash[29146]: cluster 2024-03-11T21:18:03.043607+0000 osd.2 (osd.2) 284 : cluster 0 2.0 scrub ok 2024-03-11T21:18:04.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:04 smithi023 bash[29146]: cluster 2024-03-11T21:18:03.436660+0000 osd.5 (osd.5) 625 : cluster 0 3.16 scrub starts 2024-03-11T21:18:04.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:04 smithi023 bash[29146]: cluster 2024-03-11T21:18:03.440575+0000 osd.5 (osd.5) 626 : cluster 0 3.16 scrub ok 2024-03-11T21:18:04.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:04 smithi023 bash[29146]: cluster 2024-03-11T21:18:04.006551+0000 osd.1 (osd.1) 392 : cluster 0 1.b scrub starts 2024-03-11T21:18:04.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:04 smithi023 bash[29146]: cluster 2024-03-11T21:18:04.009553+0000 osd.1 (osd.1) 393 : cluster 0 1.b scrub ok 2024-03-11T21:18:05.515 INFO:teuthology.orchestra.run.smithi023.stderr:reweighted osd.6 to 1 (10000) 2024-03-11T21:18:05.774 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:05 smithi023 bash[21372]: cluster 2024-03-11T21:18:04.093952+0000 osd.6 (osd.6) 524 : cluster 0 4.18 scrub starts 2024-03-11T21:18:05.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:05 smithi023 bash[21372]: cluster 2024-03-11T21:18:04.096377+0000 osd.6 (osd.6) 525 : cluster 0 4.18 scrub ok 2024-03-11T21:18:05.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:05 smithi023 bash[21372]: cluster 2024-03-11T21:18:04.480221+0000 osd.5 (osd.5) 627 : cluster 0 2.3 scrub starts 2024-03-11T21:18:05.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:05 smithi023 bash[21372]: cluster 2024-03-11T21:18:04.482349+0000 osd.5 (osd.5) 628 : cluster 0 2.3 scrub ok 2024-03-11T21:18:05.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:05 smithi023 bash[21372]: cluster 2024-03-11T21:18:04.530603+0000 mgr.y (mgr.24380) 1006 : cluster 0 pgmap v1173: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:18:05.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:05 smithi023 bash[21372]: audit 2024-03-11T21:18:04.925597+0000 mon.c (mon.2) 109 : audit 1 from='client.? 172.21.15.23:0/3297073855' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-11T21:18:05.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:05 smithi023 bash[21372]: audit 2024-03-11T21:18:04.926126+0000 mon.a (mon.0) 2064 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-11T21:18:05.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:05 smithi023 bash[29146]: cluster 2024-03-11T21:18:04.093952+0000 osd.6 (osd.6) 524 : cluster 0 4.18 scrub starts 2024-03-11T21:18:05.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:05 smithi023 bash[29146]: cluster 2024-03-11T21:18:04.096377+0000 osd.6 (osd.6) 525 : cluster 0 4.18 scrub ok 2024-03-11T21:18:05.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:05 smithi023 bash[29146]: cluster 2024-03-11T21:18:04.480221+0000 osd.5 (osd.5) 627 : cluster 0 2.3 scrub starts 2024-03-11T21:18:05.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:05 smithi023 bash[29146]: cluster 2024-03-11T21:18:04.482349+0000 osd.5 (osd.5) 628 : cluster 0 2.3 scrub ok 2024-03-11T21:18:05.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:05 smithi023 bash[29146]: cluster 2024-03-11T21:18:04.530603+0000 mgr.y (mgr.24380) 1006 : cluster 0 pgmap v1173: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:18:05.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:05 smithi023 bash[29146]: audit 2024-03-11T21:18:04.925597+0000 mon.c (mon.2) 109 : audit 1 from='client.? 172.21.15.23:0/3297073855' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-11T21:18:05.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:05 smithi023 bash[29146]: audit 2024-03-11T21:18:04.926126+0000 mon.a (mon.0) 2064 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-11T21:18:06.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:05 smithi146 bash[23809]: cluster 2024-03-11T21:18:04.093952+0000 osd.6 (osd.6) 524 : cluster 0 4.18 scrub starts 2024-03-11T21:18:06.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:05 smithi146 bash[23809]: cluster 2024-03-11T21:18:04.096377+0000 osd.6 (osd.6) 525 : cluster 0 4.18 scrub ok 2024-03-11T21:18:06.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:05 smithi146 bash[23809]: cluster 2024-03-11T21:18:04.480221+0000 osd.5 (osd.5) 627 : cluster 0 2.3 scrub starts 2024-03-11T21:18:06.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:05 smithi146 bash[23809]: cluster 2024-03-11T21:18:04.482349+0000 osd.5 (osd.5) 628 : cluster 0 2.3 scrub ok 2024-03-11T21:18:06.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:05 smithi146 bash[23809]: cluster 2024-03-11T21:18:04.530603+0000 mgr.y (mgr.24380) 1006 : cluster 0 pgmap v1173: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:18:06.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:05 smithi146 bash[23809]: audit 2024-03-11T21:18:04.925597+0000 mon.c (mon.2) 109 : audit 1 from='client.? 172.21.15.23:0/3297073855' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-11T21:18:06.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:05 smithi146 bash[23809]: audit 2024-03-11T21:18:04.926126+0000 mon.a (mon.0) 2064 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-11T21:18:06.361 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd primary-affinity 6 1 2024-03-11T21:18:06.503 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:18:06 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:18:06.774 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:06 smithi023 bash[21372]: audit 2024-03-11T21:18:05.509361+0000 mon.a (mon.0) 2065 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-11T21:18:06.774 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:06 smithi023 bash[21372]: cluster 2024-03-11T21:18:05.520752+0000 mon.a (mon.0) 2066 : cluster 0 osdmap e477: 8 total, 8 up, 8 in 2024-03-11T21:18:06.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:06 smithi023 bash[29146]: audit 2024-03-11T21:18:05.509361+0000 mon.a (mon.0) 2065 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-11T21:18:06.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:06 smithi023 bash[29146]: cluster 2024-03-11T21:18:05.520752+0000 mon.a (mon.0) 2066 : cluster 0 osdmap e477: 8 total, 8 up, 8 in 2024-03-11T21:18:07.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:06 smithi146 bash[23809]: audit 2024-03-11T21:18:05.509361+0000 mon.a (mon.0) 2065 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-11T21:18:07.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:06 smithi146 bash[23809]: cluster 2024-03-11T21:18:05.520752+0000 mon.a (mon.0) 2066 : cluster 0 osdmap e477: 8 total, 8 up, 8 in 2024-03-11T21:18:07.616 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:18:08.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:07 smithi146 bash[23809]: cluster 2024-03-11T21:18:06.021893+0000 osd.7 (osd.7) 647 : cluster 0 1.9 scrub starts 2024-03-11T21:18:08.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:07 smithi146 bash[23809]: cluster 2024-03-11T21:18:06.024568+0000 osd.7 (osd.7) 648 : cluster 0 1.9 scrub ok 2024-03-11T21:18:08.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:07 smithi146 bash[23809]: audit 2024-03-11T21:18:06.235549+0000 mgr.y (mgr.24380) 1007 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:18:08.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:07 smithi146 bash[23809]: cluster 2024-03-11T21:18:06.532156+0000 mgr.y (mgr.24380) 1008 : cluster 0 pgmap v1175: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:18:08.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:07 smithi146 bash[23809]: audit 2024-03-11T21:18:06.896605+0000 mon.a (mon.0) 2067 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:18:08.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:07 smithi023 bash[21372]: cluster 2024-03-11T21:18:06.021893+0000 osd.7 (osd.7) 647 : cluster 0 1.9 scrub starts 2024-03-11T21:18:08.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:07 smithi023 bash[21372]: cluster 2024-03-11T21:18:06.024568+0000 osd.7 (osd.7) 648 : cluster 0 1.9 scrub ok 2024-03-11T21:18:08.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:07 smithi023 bash[21372]: audit 2024-03-11T21:18:06.235549+0000 mgr.y (mgr.24380) 1007 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:18:08.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:07 smithi023 bash[21372]: cluster 2024-03-11T21:18:06.532156+0000 mgr.y (mgr.24380) 1008 : cluster 0 pgmap v1175: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:18:08.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:07 smithi023 bash[21372]: audit 2024-03-11T21:18:06.896605+0000 mon.a (mon.0) 2067 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:18:08.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:07 smithi023 bash[29146]: cluster 2024-03-11T21:18:06.021893+0000 osd.7 (osd.7) 647 : cluster 0 1.9 scrub starts 2024-03-11T21:18:08.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:07 smithi023 bash[29146]: cluster 2024-03-11T21:18:06.024568+0000 osd.7 (osd.7) 648 : cluster 0 1.9 scrub ok 2024-03-11T21:18:08.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:07 smithi023 bash[29146]: audit 2024-03-11T21:18:06.235549+0000 mgr.y (mgr.24380) 1007 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:18:08.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:07 smithi023 bash[29146]: cluster 2024-03-11T21:18:06.532156+0000 mgr.y (mgr.24380) 1008 : cluster 0 pgmap v1175: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:18:08.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:07 smithi023 bash[29146]: audit 2024-03-11T21:18:06.896605+0000 mon.a (mon.0) 2067 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:18:08.774 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:18:08 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:18:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:18:08.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:08 smithi023 bash[21372]: cluster 2024-03-11T21:18:06.989437+0000 osd.7 (osd.7) 649 : cluster 0 2.b scrub starts 2024-03-11T21:18:08.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:08 smithi023 bash[21372]: cluster 2024-03-11T21:18:06.991900+0000 osd.7 (osd.7) 650 : cluster 0 2.b scrub ok 2024-03-11T21:18:08.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:08 smithi023 bash[21372]: cluster 2024-03-11T21:18:07.918404+0000 osd.1 (osd.1) 394 : cluster 0 3.1a deep-scrub starts 2024-03-11T21:18:08.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:08 smithi023 bash[21372]: cluster 2024-03-11T21:18:07.922994+0000 osd.1 (osd.1) 395 : cluster 0 3.1a deep-scrub ok 2024-03-11T21:18:08.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:08 smithi023 bash[29146]: cluster 2024-03-11T21:18:06.989437+0000 osd.7 (osd.7) 649 : cluster 0 2.b scrub starts 2024-03-11T21:18:08.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:08 smithi023 bash[29146]: cluster 2024-03-11T21:18:06.991900+0000 osd.7 (osd.7) 650 : cluster 0 2.b scrub ok 2024-03-11T21:18:08.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:08 smithi023 bash[29146]: cluster 2024-03-11T21:18:07.918404+0000 osd.1 (osd.1) 394 : cluster 0 3.1a deep-scrub starts 2024-03-11T21:18:08.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:08 smithi023 bash[29146]: cluster 2024-03-11T21:18:07.922994+0000 osd.1 (osd.1) 395 : cluster 0 3.1a deep-scrub ok 2024-03-11T21:18:08.814 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:08 smithi146 bash[23809]: cluster 2024-03-11T21:18:06.989437+0000 osd.7 (osd.7) 649 : cluster 0 2.b scrub starts 2024-03-11T21:18:08.814 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:08 smithi146 bash[23809]: cluster 2024-03-11T21:18:06.991900+0000 osd.7 (osd.7) 650 : cluster 0 2.b scrub ok 2024-03-11T21:18:08.814 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:08 smithi146 bash[23809]: cluster 2024-03-11T21:18:07.918404+0000 osd.1 (osd.1) 394 : cluster 0 3.1a deep-scrub starts 2024-03-11T21:18:08.815 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:08 smithi146 bash[23809]: cluster 2024-03-11T21:18:07.922994+0000 osd.1 (osd.1) 395 : cluster 0 3.1a deep-scrub ok 2024-03-11T21:18:10.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:09 smithi146 bash[23809]: cluster 2024-03-11T21:18:07.988827+0000 osd.7 (osd.7) 651 : cluster 0 3.14 scrub starts 2024-03-11T21:18:10.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:09 smithi146 bash[23809]: cluster 2024-03-11T21:18:07.993176+0000 osd.7 (osd.7) 652 : cluster 0 3.14 scrub ok 2024-03-11T21:18:10.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:09 smithi146 bash[23809]: cluster 2024-03-11T21:18:08.533045+0000 mgr.y (mgr.24380) 1009 : cluster 0 pgmap v1176: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:18:10.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:09 smithi146 bash[23809]: cluster 2024-03-11T21:18:08.919067+0000 osd.1 (osd.1) 396 : cluster 0 4.9 scrub starts 2024-03-11T21:18:10.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:09 smithi146 bash[23809]: cluster 2024-03-11T21:18:08.922232+0000 osd.1 (osd.1) 397 : cluster 0 4.9 scrub ok 2024-03-11T21:18:10.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:09 smithi023 bash[21372]: cluster 2024-03-11T21:18:07.988827+0000 osd.7 (osd.7) 651 : cluster 0 3.14 scrub starts 2024-03-11T21:18:10.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:09 smithi023 bash[21372]: cluster 2024-03-11T21:18:07.993176+0000 osd.7 (osd.7) 652 : cluster 0 3.14 scrub ok 2024-03-11T21:18:10.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:09 smithi023 bash[21372]: cluster 2024-03-11T21:18:08.533045+0000 mgr.y (mgr.24380) 1009 : cluster 0 pgmap v1176: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:18:10.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:09 smithi023 bash[21372]: cluster 2024-03-11T21:18:08.919067+0000 osd.1 (osd.1) 396 : cluster 0 4.9 scrub starts 2024-03-11T21:18:10.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:09 smithi023 bash[21372]: cluster 2024-03-11T21:18:08.922232+0000 osd.1 (osd.1) 397 : cluster 0 4.9 scrub ok 2024-03-11T21:18:10.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:09 smithi023 bash[29146]: cluster 2024-03-11T21:18:07.988827+0000 osd.7 (osd.7) 651 : cluster 0 3.14 scrub starts 2024-03-11T21:18:10.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:09 smithi023 bash[29146]: cluster 2024-03-11T21:18:07.993176+0000 osd.7 (osd.7) 652 : cluster 0 3.14 scrub ok 2024-03-11T21:18:10.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:09 smithi023 bash[29146]: cluster 2024-03-11T21:18:08.533045+0000 mgr.y (mgr.24380) 1009 : cluster 0 pgmap v1176: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:18:10.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:09 smithi023 bash[29146]: cluster 2024-03-11T21:18:08.919067+0000 osd.1 (osd.1) 396 : cluster 0 4.9 scrub starts 2024-03-11T21:18:10.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:09 smithi023 bash[29146]: cluster 2024-03-11T21:18:08.922232+0000 osd.1 (osd.1) 397 : cluster 0 4.9 scrub ok 2024-03-11T21:18:10.848 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-03-11T21:18:10.848 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:18:11.157 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:18:11.629 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-11T21:18:11.753 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:11 smithi146 bash[23809]: cluster 2024-03-11T21:18:10.004561+0000 osd.7 (osd.7) 653 : cluster 0 3.d scrub starts 2024-03-11T21:18:11.753 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:11 smithi146 bash[23809]: cluster 2024-03-11T21:18:10.011553+0000 osd.7 (osd.7) 654 : cluster 0 3.d scrub ok 2024-03-11T21:18:11.753 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:11 smithi146 bash[23809]: cluster 2024-03-11T21:18:10.533707+0000 mgr.y (mgr.24380) 1010 : cluster 0 pgmap v1177: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:18:11.774 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:11 smithi023 bash[21372]: cluster 2024-03-11T21:18:10.004561+0000 osd.7 (osd.7) 653 : cluster 0 3.d scrub starts 2024-03-11T21:18:11.774 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:11 smithi023 bash[21372]: cluster 2024-03-11T21:18:10.011553+0000 osd.7 (osd.7) 654 : cluster 0 3.d scrub ok 2024-03-11T21:18:11.774 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:11 smithi023 bash[21372]: cluster 2024-03-11T21:18:10.533707+0000 mgr.y (mgr.24380) 1010 : cluster 0 pgmap v1177: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:18:11.774 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:11 smithi023 bash[29146]: cluster 2024-03-11T21:18:10.004561+0000 osd.7 (osd.7) 653 : cluster 0 3.d scrub starts 2024-03-11T21:18:11.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:11 smithi023 bash[29146]: cluster 2024-03-11T21:18:10.011553+0000 osd.7 (osd.7) 654 : cluster 0 3.d scrub ok 2024-03-11T21:18:11.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:11 smithi023 bash[29146]: cluster 2024-03-11T21:18:10.533707+0000 mgr.y (mgr.24380) 1010 : cluster 0 pgmap v1177: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:18:12.524 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:12 smithi023 bash[21372]: cluster 2024-03-11T21:18:11.003702+0000 osd.2 (osd.2) 285 : cluster 0 6.17 scrub starts 2024-03-11T21:18:12.524 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:12 smithi023 bash[21372]: cluster 2024-03-11T21:18:11.223719+0000 osd.2 (osd.2) 286 : cluster 0 6.17 scrub ok 2024-03-11T21:18:12.524 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:12 smithi023 bash[29146]: cluster 2024-03-11T21:18:11.003702+0000 osd.2 (osd.2) 285 : cluster 0 6.17 scrub starts 2024-03-11T21:18:12.525 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:12 smithi023 bash[29146]: cluster 2024-03-11T21:18:11.223719+0000 osd.2 (osd.2) 286 : cluster 0 6.17 scrub ok 2024-03-11T21:18:12.753 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:12 smithi146 bash[23809]: cluster 2024-03-11T21:18:11.003702+0000 osd.2 (osd.2) 285 : cluster 0 6.17 scrub starts 2024-03-11T21:18:12.753 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:12 smithi146 bash[23809]: cluster 2024-03-11T21:18:11.223719+0000 osd.2 (osd.2) 286 : cluster 0 6.17 scrub ok 2024-03-11T21:18:13.524 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:13 smithi023 bash[21372]: cluster 2024-03-11T21:18:12.017199+0000 osd.7 (osd.7) 655 : cluster 0 6.5 scrub starts 2024-03-11T21:18:13.525 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:13 smithi023 bash[21372]: cluster 2024-03-11T21:18:12.106365+0000 osd.7 (osd.7) 656 : cluster 0 6.5 scrub ok 2024-03-11T21:18:13.525 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:13 smithi023 bash[21372]: cluster 2024-03-11T21:18:12.305650+0000 osd.6 (osd.6) 526 : cluster 0 4.13 scrub starts 2024-03-11T21:18:13.525 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:13 smithi023 bash[21372]: cluster 2024-03-11T21:18:12.308034+0000 osd.6 (osd.6) 527 : cluster 0 4.13 scrub ok 2024-03-11T21:18:13.525 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:13 smithi023 bash[21372]: cluster 2024-03-11T21:18:12.534593+0000 mgr.y (mgr.24380) 1011 : cluster 0 pgmap v1178: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:18:13.525 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:13 smithi023 bash[29146]: cluster 2024-03-11T21:18:12.017199+0000 osd.7 (osd.7) 655 : cluster 0 6.5 scrub starts 2024-03-11T21:18:13.525 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:13 smithi023 bash[29146]: cluster 2024-03-11T21:18:12.106365+0000 osd.7 (osd.7) 656 : cluster 0 6.5 scrub ok 2024-03-11T21:18:13.525 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:13 smithi023 bash[29146]: cluster 2024-03-11T21:18:12.305650+0000 osd.6 (osd.6) 526 : cluster 0 4.13 scrub starts 2024-03-11T21:18:13.525 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:13 smithi023 bash[29146]: cluster 2024-03-11T21:18:12.308034+0000 osd.6 (osd.6) 527 : cluster 0 4.13 scrub ok 2024-03-11T21:18:13.525 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:13 smithi023 bash[29146]: cluster 2024-03-11T21:18:12.534593+0000 mgr.y (mgr.24380) 1011 : cluster 0 pgmap v1178: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:18:13.753 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:13 smithi146 bash[23809]: cluster 2024-03-11T21:18:12.017199+0000 osd.7 (osd.7) 655 : cluster 0 6.5 scrub starts 2024-03-11T21:18:13.753 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:13 smithi146 bash[23809]: cluster 2024-03-11T21:18:12.106365+0000 osd.7 (osd.7) 656 : cluster 0 6.5 scrub ok 2024-03-11T21:18:13.753 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:13 smithi146 bash[23809]: cluster 2024-03-11T21:18:12.305650+0000 osd.6 (osd.6) 526 : cluster 0 4.13 scrub starts 2024-03-11T21:18:13.753 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:13 smithi146 bash[23809]: cluster 2024-03-11T21:18:12.308034+0000 osd.6 (osd.6) 527 : cluster 0 4.13 scrub ok 2024-03-11T21:18:13.753 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:13 smithi146 bash[23809]: cluster 2024-03-11T21:18:12.534593+0000 mgr.y (mgr.24380) 1011 : cluster 0 pgmap v1178: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:18:14.504 INFO:teuthology.orchestra.run.smithi023.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-03-11T21:18:14.774 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:14 smithi023 bash[21372]: cluster 2024-03-11T21:18:13.427706+0000 osd.0 (osd.0) 482 : cluster 0 3.1f scrub starts 2024-03-11T21:18:14.774 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:14 smithi023 bash[21372]: cluster 2024-03-11T21:18:13.430397+0000 osd.0 (osd.0) 483 : cluster 0 3.1f scrub ok 2024-03-11T21:18:14.774 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:14 smithi023 bash[21372]: audit 2024-03-11T21:18:14.423408+0000 mon.a (mon.0) 2068 : audit 1 from='client.? 172.21.15.23:0/3123608244' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-03-11T21:18:14.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:14 smithi023 bash[29146]: cluster 2024-03-11T21:18:13.427706+0000 osd.0 (osd.0) 482 : cluster 0 3.1f scrub starts 2024-03-11T21:18:14.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:14 smithi023 bash[29146]: cluster 2024-03-11T21:18:13.430397+0000 osd.0 (osd.0) 483 : cluster 0 3.1f scrub ok 2024-03-11T21:18:14.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:14 smithi023 bash[29146]: audit 2024-03-11T21:18:14.423408+0000 mon.a (mon.0) 2068 : audit 1 from='client.? 172.21.15.23:0/3123608244' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-03-11T21:18:15.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:14 smithi146 bash[23809]: cluster 2024-03-11T21:18:13.427706+0000 osd.0 (osd.0) 482 : cluster 0 3.1f scrub starts 2024-03-11T21:18:15.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:14 smithi146 bash[23809]: cluster 2024-03-11T21:18:13.430397+0000 osd.0 (osd.0) 483 : cluster 0 3.1f scrub ok 2024-03-11T21:18:15.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:14 smithi146 bash[23809]: audit 2024-03-11T21:18:14.423408+0000 mon.a (mon.0) 2068 : audit 1 from='client.? 172.21.15.23:0/3123608244' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-03-11T21:18:15.329 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd reweight 7 1 2024-03-11T21:18:15.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:15 smithi023 bash[21372]: cluster 2024-03-11T21:18:13.677033+0000 osd.5 (osd.5) 629 : cluster 0 2.18 scrub starts 2024-03-11T21:18:15.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:15 smithi023 bash[21372]: cluster 2024-03-11T21:18:13.679462+0000 osd.5 (osd.5) 630 : cluster 0 2.18 scrub ok 2024-03-11T21:18:15.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:15 smithi023 bash[21372]: cluster 2024-03-11T21:18:14.079362+0000 osd.7 (osd.7) 657 : cluster 0 2.1d deep-scrub starts 2024-03-11T21:18:15.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:15 smithi023 bash[21372]: cluster 2024-03-11T21:18:14.081569+0000 osd.7 (osd.7) 658 : cluster 0 2.1d deep-scrub ok 2024-03-11T21:18:15.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:15 smithi023 bash[21372]: cluster 2024-03-11T21:18:14.297584+0000 osd.6 (osd.6) 528 : cluster 0 3.10 scrub starts 2024-03-11T21:18:15.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:15 smithi023 bash[21372]: cluster 2024-03-11T21:18:14.301147+0000 osd.6 (osd.6) 529 : cluster 0 3.10 scrub ok 2024-03-11T21:18:15.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:15 smithi023 bash[21372]: audit 2024-03-11T21:18:14.513737+0000 mon.a (mon.0) 2069 : audit 1 from='client.? 172.21.15.23:0/3123608244' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-03-11T21:18:15.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:15 smithi023 bash[21372]: cluster 2024-03-11T21:18:14.527917+0000 mon.a (mon.0) 2070 : cluster 0 osdmap e478: 8 total, 8 up, 8 in 2024-03-11T21:18:15.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:15 smithi023 bash[21372]: cluster 2024-03-11T21:18:14.535154+0000 mgr.y (mgr.24380) 1012 : cluster 0 pgmap v1180: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T21:18:15.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:15 smithi023 bash[21372]: cluster 2024-03-11T21:18:14.834800+0000 osd.1 (osd.1) 398 : cluster 0 2.9 scrub starts 2024-03-11T21:18:15.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:15 smithi023 bash[21372]: cluster 2024-03-11T21:18:14.837161+0000 osd.1 (osd.1) 399 : cluster 0 2.9 scrub ok 2024-03-11T21:18:15.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:15 smithi023 bash[29146]: cluster 2024-03-11T21:18:13.677033+0000 osd.5 (osd.5) 629 : cluster 0 2.18 scrub starts 2024-03-11T21:18:15.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:15 smithi023 bash[29146]: cluster 2024-03-11T21:18:13.679462+0000 osd.5 (osd.5) 630 : cluster 0 2.18 scrub ok 2024-03-11T21:18:15.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:15 smithi023 bash[29146]: cluster 2024-03-11T21:18:14.079362+0000 osd.7 (osd.7) 657 : cluster 0 2.1d deep-scrub starts 2024-03-11T21:18:15.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:15 smithi023 bash[29146]: cluster 2024-03-11T21:18:14.081569+0000 osd.7 (osd.7) 658 : cluster 0 2.1d deep-scrub ok 2024-03-11T21:18:15.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:15 smithi023 bash[29146]: cluster 2024-03-11T21:18:14.297584+0000 osd.6 (osd.6) 528 : cluster 0 3.10 scrub starts 2024-03-11T21:18:15.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:15 smithi023 bash[29146]: cluster 2024-03-11T21:18:14.301147+0000 osd.6 (osd.6) 529 : cluster 0 3.10 scrub ok 2024-03-11T21:18:15.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:15 smithi023 bash[29146]: audit 2024-03-11T21:18:14.513737+0000 mon.a (mon.0) 2069 : audit 1 from='client.? 172.21.15.23:0/3123608244' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-03-11T21:18:15.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:15 smithi023 bash[29146]: cluster 2024-03-11T21:18:14.527917+0000 mon.a (mon.0) 2070 : cluster 0 osdmap e478: 8 total, 8 up, 8 in 2024-03-11T21:18:15.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:15 smithi023 bash[29146]: cluster 2024-03-11T21:18:14.535154+0000 mgr.y (mgr.24380) 1012 : cluster 0 pgmap v1180: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T21:18:15.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:15 smithi023 bash[29146]: cluster 2024-03-11T21:18:14.834800+0000 osd.1 (osd.1) 398 : cluster 0 2.9 scrub starts 2024-03-11T21:18:15.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:15 smithi023 bash[29146]: cluster 2024-03-11T21:18:14.837161+0000 osd.1 (osd.1) 399 : cluster 0 2.9 scrub ok 2024-03-11T21:18:16.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:15 smithi146 bash[23809]: cluster 2024-03-11T21:18:13.677033+0000 osd.5 (osd.5) 629 : cluster 0 2.18 scrub starts 2024-03-11T21:18:16.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:15 smithi146 bash[23809]: cluster 2024-03-11T21:18:13.679462+0000 osd.5 (osd.5) 630 : cluster 0 2.18 scrub ok 2024-03-11T21:18:16.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:15 smithi146 bash[23809]: cluster 2024-03-11T21:18:14.079362+0000 osd.7 (osd.7) 657 : cluster 0 2.1d deep-scrub starts 2024-03-11T21:18:16.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:15 smithi146 bash[23809]: cluster 2024-03-11T21:18:14.081569+0000 osd.7 (osd.7) 658 : cluster 0 2.1d deep-scrub ok 2024-03-11T21:18:16.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:15 smithi146 bash[23809]: cluster 2024-03-11T21:18:14.297584+0000 osd.6 (osd.6) 528 : cluster 0 3.10 scrub starts 2024-03-11T21:18:16.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:15 smithi146 bash[23809]: cluster 2024-03-11T21:18:14.301147+0000 osd.6 (osd.6) 529 : cluster 0 3.10 scrub ok 2024-03-11T21:18:16.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:15 smithi146 bash[23809]: audit 2024-03-11T21:18:14.513737+0000 mon.a (mon.0) 2069 : audit 1 from='client.? 172.21.15.23:0/3123608244' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-03-11T21:18:16.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:15 smithi146 bash[23809]: cluster 2024-03-11T21:18:14.527917+0000 mon.a (mon.0) 2070 : cluster 0 osdmap e478: 8 total, 8 up, 8 in 2024-03-11T21:18:16.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:15 smithi146 bash[23809]: cluster 2024-03-11T21:18:14.535154+0000 mgr.y (mgr.24380) 1012 : cluster 0 pgmap v1180: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T21:18:16.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:15 smithi146 bash[23809]: cluster 2024-03-11T21:18:14.834800+0000 osd.1 (osd.1) 398 : cluster 0 2.9 scrub starts 2024-03-11T21:18:16.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:15 smithi146 bash[23809]: cluster 2024-03-11T21:18:14.837161+0000 osd.1 (osd.1) 399 : cluster 0 2.9 scrub ok 2024-03-11T21:18:16.503 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:18:16 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:18:16.818 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:18:17.798 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:17 smithi023 bash[21372]: cluster 2024-03-11T21:18:16.082520+0000 osd.7 (osd.7) 659 : cluster 0 2.1c scrub starts 2024-03-11T21:18:17.799 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:17 smithi023 bash[21372]: cluster 2024-03-11T21:18:16.085489+0000 osd.7 (osd.7) 660 : cluster 0 2.1c scrub ok 2024-03-11T21:18:17.799 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:17 smithi023 bash[21372]: audit 2024-03-11T21:18:16.244035+0000 mgr.y (mgr.24380) 1013 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:18:17.799 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:17 smithi023 bash[21372]: cluster 2024-03-11T21:18:16.536902+0000 mgr.y (mgr.24380) 1014 : cluster 0 pgmap v1181: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:18:17.799 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:17 smithi023 bash[21372]: cluster 2024-03-11T21:18:16.670683+0000 osd.3 (osd.3) 576 : cluster 0 4.c scrub starts 2024-03-11T21:18:17.799 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:17 smithi023 bash[21372]: cluster 2024-03-11T21:18:16.673345+0000 osd.3 (osd.3) 577 : cluster 0 4.c scrub ok 2024-03-11T21:18:17.799 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:17 smithi023 bash[21372]: cluster 2024-03-11T21:18:16.858346+0000 osd.1 (osd.1) 400 : cluster 0 4.12 deep-scrub starts 2024-03-11T21:18:17.799 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:17 smithi023 bash[21372]: cluster 2024-03-11T21:18:16.860121+0000 osd.1 (osd.1) 401 : cluster 0 4.12 deep-scrub ok 2024-03-11T21:18:17.799 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:17 smithi023 bash[29146]: cluster 2024-03-11T21:18:16.082520+0000 osd.7 (osd.7) 659 : cluster 0 2.1c scrub starts 2024-03-11T21:18:17.799 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:17 smithi023 bash[29146]: cluster 2024-03-11T21:18:16.085489+0000 osd.7 (osd.7) 660 : cluster 0 2.1c scrub ok 2024-03-11T21:18:17.799 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:17 smithi023 bash[29146]: audit 2024-03-11T21:18:16.244035+0000 mgr.y (mgr.24380) 1013 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:18:17.799 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:17 smithi023 bash[29146]: cluster 2024-03-11T21:18:16.536902+0000 mgr.y (mgr.24380) 1014 : cluster 0 pgmap v1181: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:18:17.800 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:17 smithi023 bash[29146]: cluster 2024-03-11T21:18:16.670683+0000 osd.3 (osd.3) 576 : cluster 0 4.c scrub starts 2024-03-11T21:18:17.800 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:17 smithi023 bash[29146]: cluster 2024-03-11T21:18:16.673345+0000 osd.3 (osd.3) 577 : cluster 0 4.c scrub ok 2024-03-11T21:18:17.800 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:17 smithi023 bash[29146]: cluster 2024-03-11T21:18:16.858346+0000 osd.1 (osd.1) 400 : cluster 0 4.12 deep-scrub starts 2024-03-11T21:18:17.800 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:17 smithi023 bash[29146]: cluster 2024-03-11T21:18:16.860121+0000 osd.1 (osd.1) 401 : cluster 0 4.12 deep-scrub ok 2024-03-11T21:18:18.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:17 smithi146 bash[23809]: cluster 2024-03-11T21:18:16.082520+0000 osd.7 (osd.7) 659 : cluster 0 2.1c scrub starts 2024-03-11T21:18:18.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:17 smithi146 bash[23809]: cluster 2024-03-11T21:18:16.085489+0000 osd.7 (osd.7) 660 : cluster 0 2.1c scrub ok 2024-03-11T21:18:18.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:17 smithi146 bash[23809]: audit 2024-03-11T21:18:16.244035+0000 mgr.y (mgr.24380) 1013 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:18:18.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:17 smithi146 bash[23809]: cluster 2024-03-11T21:18:16.536902+0000 mgr.y (mgr.24380) 1014 : cluster 0 pgmap v1181: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:18:18.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:17 smithi146 bash[23809]: cluster 2024-03-11T21:18:16.670683+0000 osd.3 (osd.3) 576 : cluster 0 4.c scrub starts 2024-03-11T21:18:18.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:17 smithi146 bash[23809]: cluster 2024-03-11T21:18:16.673345+0000 osd.3 (osd.3) 577 : cluster 0 4.c scrub ok 2024-03-11T21:18:18.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:17 smithi146 bash[23809]: cluster 2024-03-11T21:18:16.858346+0000 osd.1 (osd.1) 400 : cluster 0 4.12 deep-scrub starts 2024-03-11T21:18:18.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:17 smithi146 bash[23809]: cluster 2024-03-11T21:18:16.860121+0000 osd.1 (osd.1) 401 : cluster 0 4.12 deep-scrub ok 2024-03-11T21:18:18.689 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:18:18 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:18:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:18:18.689 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:18 smithi023 bash[21372]: cluster 2024-03-11T21:18:16.766827+0000 osd.5 (osd.5) 631 : cluster 0 6.1a scrub starts 2024-03-11T21:18:18.690 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:18 smithi023 bash[21372]: cluster 2024-03-11T21:18:16.806963+0000 osd.5 (osd.5) 632 : cluster 0 6.1a scrub ok 2024-03-11T21:18:18.690 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:18 smithi023 bash[21372]: cluster 2024-03-11T21:18:17.062255+0000 osd.2 (osd.2) 287 : cluster 0 2.c scrub starts 2024-03-11T21:18:18.690 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:18 smithi023 bash[21372]: cluster 2024-03-11T21:18:17.064760+0000 osd.2 (osd.2) 288 : cluster 0 2.c scrub ok 2024-03-11T21:18:18.690 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:18 smithi023 bash[21372]: cluster 2024-03-11T21:18:17.112376+0000 osd.7 (osd.7) 661 : cluster 0 3.e scrub starts 2024-03-11T21:18:18.690 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:18 smithi023 bash[21372]: cluster 2024-03-11T21:18:17.116439+0000 osd.7 (osd.7) 662 : cluster 0 3.e scrub ok 2024-03-11T21:18:18.690 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:18 smithi023 bash[21372]: cluster 2024-03-11T21:18:17.843483+0000 osd.1 (osd.1) 402 : cluster 0 2.4 scrub starts 2024-03-11T21:18:18.690 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:18 smithi023 bash[21372]: cluster 2024-03-11T21:18:17.845200+0000 osd.1 (osd.1) 403 : cluster 0 2.4 scrub ok 2024-03-11T21:18:18.690 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:18 smithi023 bash[29146]: cluster 2024-03-11T21:18:16.766827+0000 osd.5 (osd.5) 631 : cluster 0 6.1a scrub starts 2024-03-11T21:18:18.691 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:18 smithi023 bash[29146]: cluster 2024-03-11T21:18:16.806963+0000 osd.5 (osd.5) 632 : cluster 0 6.1a scrub ok 2024-03-11T21:18:18.691 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:18 smithi023 bash[29146]: cluster 2024-03-11T21:18:17.062255+0000 osd.2 (osd.2) 287 : cluster 0 2.c scrub starts 2024-03-11T21:18:18.691 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:18 smithi023 bash[29146]: cluster 2024-03-11T21:18:17.064760+0000 osd.2 (osd.2) 288 : cluster 0 2.c scrub ok 2024-03-11T21:18:18.691 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:18 smithi023 bash[29146]: cluster 2024-03-11T21:18:17.112376+0000 osd.7 (osd.7) 661 : cluster 0 3.e scrub starts 2024-03-11T21:18:18.691 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:18 smithi023 bash[29146]: cluster 2024-03-11T21:18:17.116439+0000 osd.7 (osd.7) 662 : cluster 0 3.e scrub ok 2024-03-11T21:18:18.691 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:18 smithi023 bash[29146]: cluster 2024-03-11T21:18:17.843483+0000 osd.1 (osd.1) 402 : cluster 0 2.4 scrub starts 2024-03-11T21:18:18.691 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:18 smithi023 bash[29146]: cluster 2024-03-11T21:18:17.845200+0000 osd.1 (osd.1) 403 : cluster 0 2.4 scrub ok 2024-03-11T21:18:19.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:18 smithi146 bash[23809]: cluster 2024-03-11T21:18:16.766827+0000 osd.5 (osd.5) 631 : cluster 0 6.1a scrub starts 2024-03-11T21:18:19.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:18 smithi146 bash[23809]: cluster 2024-03-11T21:18:16.806963+0000 osd.5 (osd.5) 632 : cluster 0 6.1a scrub ok 2024-03-11T21:18:19.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:18 smithi146 bash[23809]: cluster 2024-03-11T21:18:17.062255+0000 osd.2 (osd.2) 287 : cluster 0 2.c scrub starts 2024-03-11T21:18:19.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:18 smithi146 bash[23809]: cluster 2024-03-11T21:18:17.064760+0000 osd.2 (osd.2) 288 : cluster 0 2.c scrub ok 2024-03-11T21:18:19.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:18 smithi146 bash[23809]: cluster 2024-03-11T21:18:17.112376+0000 osd.7 (osd.7) 661 : cluster 0 3.e scrub starts 2024-03-11T21:18:19.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:18 smithi146 bash[23809]: cluster 2024-03-11T21:18:17.116439+0000 osd.7 (osd.7) 662 : cluster 0 3.e scrub ok 2024-03-11T21:18:19.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:18 smithi146 bash[23809]: cluster 2024-03-11T21:18:17.843483+0000 osd.1 (osd.1) 402 : cluster 0 2.4 scrub starts 2024-03-11T21:18:19.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:18 smithi146 bash[23809]: cluster 2024-03-11T21:18:17.845200+0000 osd.1 (osd.1) 403 : cluster 0 2.4 scrub ok 2024-03-11T21:18:19.773 INFO:teuthology.orchestra.run.smithi023.stdout:ERROR: (22) Invalid argument 2024-03-11T21:18:19.773 INFO:teuthology.orchestra.run.smithi023.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:18:19.888 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:18:20.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:19 smithi146 bash[23809]: cluster 2024-03-11T21:18:18.080556+0000 osd.7 (osd.7) 663 : cluster 0 6.3 scrub starts 2024-03-11T21:18:20.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:19 smithi146 bash[23809]: cluster 2024-03-11T21:18:18.243513+0000 osd.7 (osd.7) 664 : cluster 0 6.3 scrub ok 2024-03-11T21:18:20.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:19 smithi146 bash[23809]: cluster 2024-03-11T21:18:18.537839+0000 mgr.y (mgr.24380) 1015 : cluster 0 pgmap v1182: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:18:20.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:19 smithi146 bash[23809]: cluster 2024-03-11T21:18:18.673800+0000 osd.7 (osd.7) 665 : cluster 0 purged_snaps scrub starts 2024-03-11T21:18:20.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:19 smithi146 bash[23809]: cluster 2024-03-11T21:18:18.673935+0000 osd.7 (osd.7) 666 : cluster 0 purged_snaps scrub ok 2024-03-11T21:18:20.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:19 smithi146 bash[23809]: cluster 2024-03-11T21:18:19.333060+0000 osd.6 (osd.6) 530 : cluster 0 4.a scrub starts 2024-03-11T21:18:20.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:19 smithi146 bash[23809]: cluster 2024-03-11T21:18:19.335610+0000 osd.6 (osd.6) 531 : cluster 0 4.a scrub ok 2024-03-11T21:18:20.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:19 smithi023 bash[21372]: cluster 2024-03-11T21:18:18.080556+0000 osd.7 (osd.7) 663 : cluster 0 6.3 scrub starts 2024-03-11T21:18:20.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:19 smithi023 bash[21372]: cluster 2024-03-11T21:18:18.243513+0000 osd.7 (osd.7) 664 : cluster 0 6.3 scrub ok 2024-03-11T21:18:20.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:19 smithi023 bash[21372]: cluster 2024-03-11T21:18:18.537839+0000 mgr.y (mgr.24380) 1015 : cluster 0 pgmap v1182: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:18:20.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:19 smithi023 bash[21372]: cluster 2024-03-11T21:18:18.673800+0000 osd.7 (osd.7) 665 : cluster 0 purged_snaps scrub starts 2024-03-11T21:18:20.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:19 smithi023 bash[21372]: cluster 2024-03-11T21:18:18.673935+0000 osd.7 (osd.7) 666 : cluster 0 purged_snaps scrub ok 2024-03-11T21:18:20.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:19 smithi023 bash[21372]: cluster 2024-03-11T21:18:19.333060+0000 osd.6 (osd.6) 530 : cluster 0 4.a scrub starts 2024-03-11T21:18:20.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:19 smithi023 bash[21372]: cluster 2024-03-11T21:18:19.335610+0000 osd.6 (osd.6) 531 : cluster 0 4.a scrub ok 2024-03-11T21:18:20.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:19 smithi023 bash[29146]: cluster 2024-03-11T21:18:18.080556+0000 osd.7 (osd.7) 663 : cluster 0 6.3 scrub starts 2024-03-11T21:18:20.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:19 smithi023 bash[29146]: cluster 2024-03-11T21:18:18.243513+0000 osd.7 (osd.7) 664 : cluster 0 6.3 scrub ok 2024-03-11T21:18:20.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:19 smithi023 bash[29146]: cluster 2024-03-11T21:18:18.537839+0000 mgr.y (mgr.24380) 1015 : cluster 0 pgmap v1182: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:18:20.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:19 smithi023 bash[29146]: cluster 2024-03-11T21:18:18.673800+0000 osd.7 (osd.7) 665 : cluster 0 purged_snaps scrub starts 2024-03-11T21:18:20.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:19 smithi023 bash[29146]: cluster 2024-03-11T21:18:18.673935+0000 osd.7 (osd.7) 666 : cluster 0 purged_snaps scrub ok 2024-03-11T21:18:20.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:19 smithi023 bash[29146]: cluster 2024-03-11T21:18:19.333060+0000 osd.6 (osd.6) 530 : cluster 0 4.a scrub starts 2024-03-11T21:18:20.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:19 smithi023 bash[29146]: cluster 2024-03-11T21:18:19.335610+0000 osd.6 (osd.6) 531 : cluster 0 4.a scrub ok 2024-03-11T21:18:20.888 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:20 smithi023 bash[21372]: cluster 2024-03-11T21:18:19.588125+0000 mon.a (mon.0) 2071 : cluster 0 osdmap e479: 8 total, 8 up, 8 in 2024-03-11T21:18:20.888 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:20 smithi023 bash[21372]: cluster 2024-03-11T21:18:19.690461+0000 osd.5 (osd.5) 633 : cluster 0 4.1 deep-scrub starts 2024-03-11T21:18:20.888 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:20 smithi023 bash[21372]: cluster 2024-03-11T21:18:19.695789+0000 osd.5 (osd.5) 634 : cluster 0 4.1 deep-scrub ok 2024-03-11T21:18:20.888 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:20 smithi023 bash[21372]: cluster 2024-03-11T21:18:20.334792+0000 osd.6 (osd.6) 532 : cluster 0 2.1a scrub starts 2024-03-11T21:18:20.888 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:20 smithi023 bash[21372]: cluster 2024-03-11T21:18:20.337178+0000 osd.6 (osd.6) 533 : cluster 0 2.1a scrub ok 2024-03-11T21:18:20.888 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:20 smithi023 bash[21372]: cluster 2024-03-11T21:18:20.506557+0000 osd.5 (osd.5) 635 : cluster 0 purged_snaps scrub starts 2024-03-11T21:18:20.888 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:20 smithi023 bash[21372]: cluster 2024-03-11T21:18:20.506681+0000 osd.5 (osd.5) 636 : cluster 0 purged_snaps scrub ok 2024-03-11T21:18:20.889 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:20 smithi023 bash[29146]: cluster 2024-03-11T21:18:19.588125+0000 mon.a (mon.0) 2071 : cluster 0 osdmap e479: 8 total, 8 up, 8 in 2024-03-11T21:18:20.889 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:20 smithi023 bash[29146]: cluster 2024-03-11T21:18:19.690461+0000 osd.5 (osd.5) 633 : cluster 0 4.1 deep-scrub starts 2024-03-11T21:18:20.889 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:20 smithi023 bash[29146]: cluster 2024-03-11T21:18:19.695789+0000 osd.5 (osd.5) 634 : cluster 0 4.1 deep-scrub ok 2024-03-11T21:18:20.889 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:20 smithi023 bash[29146]: cluster 2024-03-11T21:18:20.334792+0000 osd.6 (osd.6) 532 : cluster 0 2.1a scrub starts 2024-03-11T21:18:20.889 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:20 smithi023 bash[29146]: cluster 2024-03-11T21:18:20.337178+0000 osd.6 (osd.6) 533 : cluster 0 2.1a scrub ok 2024-03-11T21:18:20.889 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:20 smithi023 bash[29146]: cluster 2024-03-11T21:18:20.506557+0000 osd.5 (osd.5) 635 : cluster 0 purged_snaps scrub starts 2024-03-11T21:18:20.889 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:20 smithi023 bash[29146]: cluster 2024-03-11T21:18:20.506681+0000 osd.5 (osd.5) 636 : cluster 0 purged_snaps scrub ok 2024-03-11T21:18:20.890 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-03-11T21:18:21.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:20 smithi146 bash[23809]: cluster 2024-03-11T21:18:19.588125+0000 mon.a (mon.0) 2071 : cluster 0 osdmap e479: 8 total, 8 up, 8 in 2024-03-11T21:18:21.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:20 smithi146 bash[23809]: cluster 2024-03-11T21:18:19.690461+0000 osd.5 (osd.5) 633 : cluster 0 4.1 deep-scrub starts 2024-03-11T21:18:21.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:20 smithi146 bash[23809]: cluster 2024-03-11T21:18:19.695789+0000 osd.5 (osd.5) 634 : cluster 0 4.1 deep-scrub ok 2024-03-11T21:18:21.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:20 smithi146 bash[23809]: cluster 2024-03-11T21:18:20.334792+0000 osd.6 (osd.6) 532 : cluster 0 2.1a scrub starts 2024-03-11T21:18:21.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:20 smithi146 bash[23809]: cluster 2024-03-11T21:18:20.337178+0000 osd.6 (osd.6) 533 : cluster 0 2.1a scrub ok 2024-03-11T21:18:21.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:20 smithi146 bash[23809]: cluster 2024-03-11T21:18:20.506557+0000 osd.5 (osd.5) 635 : cluster 0 purged_snaps scrub starts 2024-03-11T21:18:21.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:20 smithi146 bash[23809]: cluster 2024-03-11T21:18:20.506681+0000 osd.5 (osd.5) 636 : cluster 0 purged_snaps scrub ok 2024-03-11T21:18:22.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:21 smithi146 bash[23809]: cluster 2024-03-11T21:18:20.056648+0000 osd.2 (osd.2) 289 : cluster 0 6.14 scrub starts 2024-03-11T21:18:22.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:21 smithi146 bash[23809]: cluster 2024-03-11T21:18:20.060075+0000 osd.2 (osd.2) 290 : cluster 0 6.14 scrub ok 2024-03-11T21:18:22.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:21 smithi146 bash[23809]: cluster 2024-03-11T21:18:20.087604+0000 osd.7 (osd.7) 667 : cluster 0 6.8 scrub starts 2024-03-11T21:18:22.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:21 smithi146 bash[23809]: cluster 2024-03-11T21:18:20.090070+0000 osd.7 (osd.7) 668 : cluster 0 6.8 scrub ok 2024-03-11T21:18:22.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:21 smithi146 bash[23809]: cluster 2024-03-11T21:18:20.538451+0000 mgr.y (mgr.24380) 1016 : cluster 0 pgmap v1184: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:18:22.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:21 smithi146 bash[23809]: cluster 2024-03-11T21:18:20.661480+0000 osd.5 (osd.5) 637 : cluster 0 2.12 scrub starts 2024-03-11T21:18:22.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:21 smithi146 bash[23809]: cluster 2024-03-11T21:18:20.664206+0000 osd.5 (osd.5) 638 : cluster 0 2.12 scrub ok 2024-03-11T21:18:22.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:21 smithi146 bash[23809]: cluster 2024-03-11T21:18:21.502945+0000 mon.a (mon.0) 2072 : cluster 0 osdmap e480: 8 total, 8 up, 8 in 2024-03-11T21:18:22.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:21 smithi023 bash[21372]: cluster 2024-03-11T21:18:20.056648+0000 osd.2 (osd.2) 289 : cluster 0 6.14 scrub starts 2024-03-11T21:18:22.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:21 smithi023 bash[21372]: cluster 2024-03-11T21:18:20.060075+0000 osd.2 (osd.2) 290 : cluster 0 6.14 scrub ok 2024-03-11T21:18:22.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:21 smithi023 bash[21372]: cluster 2024-03-11T21:18:20.087604+0000 osd.7 (osd.7) 667 : cluster 0 6.8 scrub starts 2024-03-11T21:18:22.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:21 smithi023 bash[21372]: cluster 2024-03-11T21:18:20.090070+0000 osd.7 (osd.7) 668 : cluster 0 6.8 scrub ok 2024-03-11T21:18:22.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:21 smithi023 bash[21372]: cluster 2024-03-11T21:18:20.538451+0000 mgr.y (mgr.24380) 1016 : cluster 0 pgmap v1184: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:18:22.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:21 smithi023 bash[21372]: cluster 2024-03-11T21:18:20.661480+0000 osd.5 (osd.5) 637 : cluster 0 2.12 scrub starts 2024-03-11T21:18:22.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:21 smithi023 bash[21372]: cluster 2024-03-11T21:18:20.664206+0000 osd.5 (osd.5) 638 : cluster 0 2.12 scrub ok 2024-03-11T21:18:22.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:21 smithi023 bash[21372]: cluster 2024-03-11T21:18:21.502945+0000 mon.a (mon.0) 2072 : cluster 0 osdmap e480: 8 total, 8 up, 8 in 2024-03-11T21:18:22.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:21 smithi023 bash[29146]: cluster 2024-03-11T21:18:20.056648+0000 osd.2 (osd.2) 289 : cluster 0 6.14 scrub starts 2024-03-11T21:18:22.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:21 smithi023 bash[29146]: cluster 2024-03-11T21:18:20.060075+0000 osd.2 (osd.2) 290 : cluster 0 6.14 scrub ok 2024-03-11T21:18:22.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:21 smithi023 bash[29146]: cluster 2024-03-11T21:18:20.087604+0000 osd.7 (osd.7) 667 : cluster 0 6.8 scrub starts 2024-03-11T21:18:22.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:21 smithi023 bash[29146]: cluster 2024-03-11T21:18:20.090070+0000 osd.7 (osd.7) 668 : cluster 0 6.8 scrub ok 2024-03-11T21:18:22.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:21 smithi023 bash[29146]: cluster 2024-03-11T21:18:20.538451+0000 mgr.y (mgr.24380) 1016 : cluster 0 pgmap v1184: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:18:22.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:21 smithi023 bash[29146]: cluster 2024-03-11T21:18:20.661480+0000 osd.5 (osd.5) 637 : cluster 0 2.12 scrub starts 2024-03-11T21:18:22.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:21 smithi023 bash[29146]: cluster 2024-03-11T21:18:20.664206+0000 osd.5 (osd.5) 638 : cluster 0 2.12 scrub ok 2024-03-11T21:18:22.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:21 smithi023 bash[29146]: cluster 2024-03-11T21:18:21.502945+0000 mon.a (mon.0) 2072 : cluster 0 osdmap e480: 8 total, 8 up, 8 in 2024-03-11T21:18:22.986 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:22 smithi023 bash[29146]: cluster 2024-03-11T21:18:21.370827+0000 osd.4 (osd.4) 123 : cluster 0 purged_snaps scrub starts 2024-03-11T21:18:22.986 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:22 smithi023 bash[29146]: cluster 2024-03-11T21:18:21.370969+0000 osd.4 (osd.4) 124 : cluster 0 purged_snaps scrub ok 2024-03-11T21:18:22.986 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:22 smithi023 bash[29146]: audit 2024-03-11T21:18:21.928319+0000 mon.a (mon.0) 2073 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:18:22.986 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:22 smithi023 bash[21372]: cluster 2024-03-11T21:18:21.370827+0000 osd.4 (osd.4) 123 : cluster 0 purged_snaps scrub starts 2024-03-11T21:18:22.986 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:22 smithi023 bash[21372]: cluster 2024-03-11T21:18:21.370969+0000 osd.4 (osd.4) 124 : cluster 0 purged_snaps scrub ok 2024-03-11T21:18:22.986 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:22 smithi023 bash[21372]: audit 2024-03-11T21:18:21.928319+0000 mon.a (mon.0) 2073 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:18:23.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:22 smithi146 bash[23809]: cluster 2024-03-11T21:18:21.370827+0000 osd.4 (osd.4) 123 : cluster 0 purged_snaps scrub starts 2024-03-11T21:18:23.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:22 smithi146 bash[23809]: cluster 2024-03-11T21:18:21.370969+0000 osd.4 (osd.4) 124 : cluster 0 purged_snaps scrub ok 2024-03-11T21:18:23.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:22 smithi146 bash[23809]: audit 2024-03-11T21:18:21.928319+0000 mon.a (mon.0) 2073 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:18:23.624 INFO:teuthology.orchestra.run.smithi023.stderr:reweighted osd.7 to 1 (10000) 2024-03-11T21:18:24.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:23 smithi146 bash[23809]: cluster 2024-03-11T21:18:22.539176+0000 mgr.y (mgr.24380) 1017 : cluster 0 pgmap v1186: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:18:24.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:23 smithi146 bash[23809]: cluster 2024-03-11T21:18:22.918611+0000 osd.1 (osd.1) 404 : cluster 0 3.19 scrub starts 2024-03-11T21:18:24.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:23 smithi146 bash[23809]: cluster 2024-03-11T21:18:22.923824+0000 osd.1 (osd.1) 405 : cluster 0 3.19 scrub ok 2024-03-11T21:18:24.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:23 smithi146 bash[23809]: audit 2024-03-11T21:18:23.425270+0000 mon.c (mon.2) 110 : audit 1 from='client.? 172.21.15.23:0/3292246503' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-11T21:18:24.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:23 smithi146 bash[23809]: audit 2024-03-11T21:18:23.425737+0000 mon.a (mon.0) 2074 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-11T21:18:24.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:23 smithi023 bash[21372]: cluster 2024-03-11T21:18:22.539176+0000 mgr.y (mgr.24380) 1017 : cluster 0 pgmap v1186: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:18:24.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:23 smithi023 bash[21372]: cluster 2024-03-11T21:18:22.918611+0000 osd.1 (osd.1) 404 : cluster 0 3.19 scrub starts 2024-03-11T21:18:24.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:23 smithi023 bash[21372]: cluster 2024-03-11T21:18:22.923824+0000 osd.1 (osd.1) 405 : cluster 0 3.19 scrub ok 2024-03-11T21:18:24.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:23 smithi023 bash[21372]: audit 2024-03-11T21:18:23.425270+0000 mon.c (mon.2) 110 : audit 1 from='client.? 172.21.15.23:0/3292246503' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-11T21:18:24.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:23 smithi023 bash[21372]: audit 2024-03-11T21:18:23.425737+0000 mon.a (mon.0) 2074 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-11T21:18:24.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:23 smithi023 bash[29146]: cluster 2024-03-11T21:18:22.539176+0000 mgr.y (mgr.24380) 1017 : cluster 0 pgmap v1186: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:18:24.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:23 smithi023 bash[29146]: cluster 2024-03-11T21:18:22.918611+0000 osd.1 (osd.1) 404 : cluster 0 3.19 scrub starts 2024-03-11T21:18:24.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:23 smithi023 bash[29146]: cluster 2024-03-11T21:18:22.923824+0000 osd.1 (osd.1) 405 : cluster 0 3.19 scrub ok 2024-03-11T21:18:24.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:23 smithi023 bash[29146]: audit 2024-03-11T21:18:23.425270+0000 mon.c (mon.2) 110 : audit 1 from='client.? 172.21.15.23:0/3292246503' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-11T21:18:24.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:23 smithi023 bash[29146]: audit 2024-03-11T21:18:23.425737+0000 mon.a (mon.0) 2074 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-11T21:18:24.442 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd primary-affinity 7 1 2024-03-11T21:18:24.774 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:24 smithi023 bash[21372]: audit 2024-03-11T21:18:23.620497+0000 mon.a (mon.0) 2075 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-11T21:18:24.774 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:24 smithi023 bash[21372]: cluster 2024-03-11T21:18:23.629889+0000 mon.a (mon.0) 2076 : cluster 0 osdmap e481: 8 total, 8 up, 8 in 2024-03-11T21:18:24.774 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:24 smithi023 bash[21372]: cluster 2024-03-11T21:18:23.668251+0000 osd.5 (osd.5) 639 : cluster 0 6.a scrub starts 2024-03-11T21:18:24.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:24 smithi023 bash[21372]: cluster 2024-03-11T21:18:23.695764+0000 osd.5 (osd.5) 640 : cluster 0 6.a scrub ok 2024-03-11T21:18:24.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:24 smithi023 bash[29146]: audit 2024-03-11T21:18:23.620497+0000 mon.a (mon.0) 2075 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-11T21:18:24.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:24 smithi023 bash[29146]: cluster 2024-03-11T21:18:23.629889+0000 mon.a (mon.0) 2076 : cluster 0 osdmap e481: 8 total, 8 up, 8 in 2024-03-11T21:18:24.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:24 smithi023 bash[29146]: cluster 2024-03-11T21:18:23.668251+0000 osd.5 (osd.5) 639 : cluster 0 6.a scrub starts 2024-03-11T21:18:24.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:24 smithi023 bash[29146]: cluster 2024-03-11T21:18:23.695764+0000 osd.5 (osd.5) 640 : cluster 0 6.a scrub ok 2024-03-11T21:18:25.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:24 smithi146 bash[23809]: audit 2024-03-11T21:18:23.620497+0000 mon.a (mon.0) 2075 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-11T21:18:25.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:24 smithi146 bash[23809]: cluster 2024-03-11T21:18:23.629889+0000 mon.a (mon.0) 2076 : cluster 0 osdmap e481: 8 total, 8 up, 8 in 2024-03-11T21:18:25.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:24 smithi146 bash[23809]: cluster 2024-03-11T21:18:23.668251+0000 osd.5 (osd.5) 639 : cluster 0 6.a scrub starts 2024-03-11T21:18:25.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:24 smithi146 bash[23809]: cluster 2024-03-11T21:18:23.695764+0000 osd.5 (osd.5) 640 : cluster 0 6.a scrub ok 2024-03-11T21:18:25.670 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:18:26.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:25 smithi146 bash[23809]: cluster 2024-03-11T21:18:24.539880+0000 mgr.y (mgr.24380) 1018 : cluster 0 pgmap v1188: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:18:26.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:25 smithi023 bash[21372]: cluster 2024-03-11T21:18:24.539880+0000 mgr.y (mgr.24380) 1018 : cluster 0 pgmap v1188: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:18:26.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:25 smithi023 bash[29146]: cluster 2024-03-11T21:18:24.539880+0000 mgr.y (mgr.24380) 1018 : cluster 0 pgmap v1188: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:18:26.503 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:18:26 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:18:27.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:26 smithi146 bash[23809]: cluster 2024-03-11T21:18:25.535111+0000 osd.3 (osd.3) 578 : cluster 0 2.19 scrub starts 2024-03-11T21:18:27.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:26 smithi146 bash[23809]: cluster 2024-03-11T21:18:25.538320+0000 osd.3 (osd.3) 579 : cluster 0 2.19 scrub ok 2024-03-11T21:18:27.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:26 smithi146 bash[23809]: cluster 2024-03-11T21:18:26.318587+0000 osd.0 (osd.0) 484 : cluster 0 3.1b scrub starts 2024-03-11T21:18:27.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:26 smithi146 bash[23809]: cluster 2024-03-11T21:18:26.322079+0000 osd.0 (osd.0) 485 : cluster 0 3.1b scrub ok 2024-03-11T21:18:27.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:26 smithi023 bash[21372]: cluster 2024-03-11T21:18:25.535111+0000 osd.3 (osd.3) 578 : cluster 0 2.19 scrub starts 2024-03-11T21:18:27.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:26 smithi023 bash[21372]: cluster 2024-03-11T21:18:25.538320+0000 osd.3 (osd.3) 579 : cluster 0 2.19 scrub ok 2024-03-11T21:18:27.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:26 smithi023 bash[21372]: cluster 2024-03-11T21:18:26.318587+0000 osd.0 (osd.0) 484 : cluster 0 3.1b scrub starts 2024-03-11T21:18:27.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:26 smithi023 bash[21372]: cluster 2024-03-11T21:18:26.322079+0000 osd.0 (osd.0) 485 : cluster 0 3.1b scrub ok 2024-03-11T21:18:27.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:26 smithi023 bash[29146]: cluster 2024-03-11T21:18:25.535111+0000 osd.3 (osd.3) 578 : cluster 0 2.19 scrub starts 2024-03-11T21:18:27.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:26 smithi023 bash[29146]: cluster 2024-03-11T21:18:25.538320+0000 osd.3 (osd.3) 579 : cluster 0 2.19 scrub ok 2024-03-11T21:18:27.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:26 smithi023 bash[29146]: cluster 2024-03-11T21:18:26.318587+0000 osd.0 (osd.0) 484 : cluster 0 3.1b scrub starts 2024-03-11T21:18:27.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:26 smithi023 bash[29146]: cluster 2024-03-11T21:18:26.322079+0000 osd.0 (osd.0) 485 : cluster 0 3.1b scrub ok 2024-03-11T21:18:28.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:27 smithi146 bash[23809]: cluster 2024-03-11T21:18:26.057019+0000 osd.2 (osd.2) 291 : cluster 0 1.14 deep-scrub starts 2024-03-11T21:18:28.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:27 smithi146 bash[23809]: cluster 2024-03-11T21:18:26.059230+0000 osd.2 (osd.2) 292 : cluster 0 1.14 deep-scrub ok 2024-03-11T21:18:28.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:27 smithi146 bash[23809]: audit 2024-03-11T21:18:26.252008+0000 mgr.y (mgr.24380) 1019 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:18:28.006 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:27 smithi146 bash[23809]: cluster 2024-03-11T21:18:26.541381+0000 mgr.y (mgr.24380) 1020 : cluster 0 pgmap v1189: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-11T21:18:28.006 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:27 smithi146 bash[23809]: cluster 2024-03-11T21:18:26.580189+0000 osd.3 (osd.3) 580 : cluster 0 4.10 scrub starts 2024-03-11T21:18:28.006 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:27 smithi146 bash[23809]: cluster 2024-03-11T21:18:26.582717+0000 osd.3 (osd.3) 581 : cluster 0 4.10 scrub ok 2024-03-11T21:18:28.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:27 smithi023 bash[21372]: cluster 2024-03-11T21:18:26.057019+0000 osd.2 (osd.2) 291 : cluster 0 1.14 deep-scrub starts 2024-03-11T21:18:28.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:27 smithi023 bash[21372]: cluster 2024-03-11T21:18:26.059230+0000 osd.2 (osd.2) 292 : cluster 0 1.14 deep-scrub ok 2024-03-11T21:18:28.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:27 smithi023 bash[21372]: audit 2024-03-11T21:18:26.252008+0000 mgr.y (mgr.24380) 1019 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:18:28.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:27 smithi023 bash[21372]: cluster 2024-03-11T21:18:26.541381+0000 mgr.y (mgr.24380) 1020 : cluster 0 pgmap v1189: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-11T21:18:28.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:27 smithi023 bash[21372]: cluster 2024-03-11T21:18:26.580189+0000 osd.3 (osd.3) 580 : cluster 0 4.10 scrub starts 2024-03-11T21:18:28.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:27 smithi023 bash[21372]: cluster 2024-03-11T21:18:26.582717+0000 osd.3 (osd.3) 581 : cluster 0 4.10 scrub ok 2024-03-11T21:18:28.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:27 smithi023 bash[29146]: cluster 2024-03-11T21:18:26.057019+0000 osd.2 (osd.2) 291 : cluster 0 1.14 deep-scrub starts 2024-03-11T21:18:28.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:27 smithi023 bash[29146]: cluster 2024-03-11T21:18:26.059230+0000 osd.2 (osd.2) 292 : cluster 0 1.14 deep-scrub ok 2024-03-11T21:18:28.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:27 smithi023 bash[29146]: audit 2024-03-11T21:18:26.252008+0000 mgr.y (mgr.24380) 1019 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:18:28.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:27 smithi023 bash[29146]: cluster 2024-03-11T21:18:26.541381+0000 mgr.y (mgr.24380) 1020 : cluster 0 pgmap v1189: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-11T21:18:28.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:27 smithi023 bash[29146]: cluster 2024-03-11T21:18:26.580189+0000 osd.3 (osd.3) 580 : cluster 0 4.10 scrub starts 2024-03-11T21:18:28.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:27 smithi023 bash[29146]: cluster 2024-03-11T21:18:26.582717+0000 osd.3 (osd.3) 581 : cluster 0 4.10 scrub ok 2024-03-11T21:18:28.602 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:18:28 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:18:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:18:28.740 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:18:28.787 INFO:teuthology.orchestra.run.smithi023.stdout:ERROR: (22) Invalid argument 2024-03-11T21:18:28.788 INFO:teuthology.orchestra.run.smithi023.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:18:29.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:28 smithi146 bash[23809]: cluster 2024-03-11T21:18:27.265072+0000 osd.7 (osd.7) 669 : cluster 0 2.5 scrub starts 2024-03-11T21:18:29.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:28 smithi146 bash[23809]: cluster 2024-03-11T21:18:27.267374+0000 osd.7 (osd.7) 670 : cluster 0 2.5 scrub ok 2024-03-11T21:18:29.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:28 smithi146 bash[23809]: cluster 2024-03-11T21:18:27.618425+0000 osd.5 (osd.5) 641 : cluster 0 3.c scrub starts 2024-03-11T21:18:29.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:28 smithi146 bash[23809]: cluster 2024-03-11T21:18:27.622333+0000 osd.5 (osd.5) 642 : cluster 0 3.c scrub ok 2024-03-11T21:18:29.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:28 smithi146 bash[23809]: cluster 2024-03-11T21:18:27.963412+0000 osd.1 (osd.1) 406 : cluster 0 6.1d scrub starts 2024-03-11T21:18:29.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:28 smithi146 bash[23809]: cluster 2024-03-11T21:18:28.129230+0000 osd.1 (osd.1) 407 : cluster 0 6.1d scrub ok 2024-03-11T21:18:29.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:28 smithi023 bash[21372]: cluster 2024-03-11T21:18:27.265072+0000 osd.7 (osd.7) 669 : cluster 0 2.5 scrub starts 2024-03-11T21:18:29.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:28 smithi023 bash[21372]: cluster 2024-03-11T21:18:27.267374+0000 osd.7 (osd.7) 670 : cluster 0 2.5 scrub ok 2024-03-11T21:18:29.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:28 smithi023 bash[21372]: cluster 2024-03-11T21:18:27.618425+0000 osd.5 (osd.5) 641 : cluster 0 3.c scrub starts 2024-03-11T21:18:29.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:28 smithi023 bash[21372]: cluster 2024-03-11T21:18:27.622333+0000 osd.5 (osd.5) 642 : cluster 0 3.c scrub ok 2024-03-11T21:18:29.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:28 smithi023 bash[21372]: cluster 2024-03-11T21:18:27.963412+0000 osd.1 (osd.1) 406 : cluster 0 6.1d scrub starts 2024-03-11T21:18:29.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:28 smithi023 bash[21372]: cluster 2024-03-11T21:18:28.129230+0000 osd.1 (osd.1) 407 : cluster 0 6.1d scrub ok 2024-03-11T21:18:29.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:28 smithi023 bash[29146]: cluster 2024-03-11T21:18:27.265072+0000 osd.7 (osd.7) 669 : cluster 0 2.5 scrub starts 2024-03-11T21:18:29.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:28 smithi023 bash[29146]: cluster 2024-03-11T21:18:27.267374+0000 osd.7 (osd.7) 670 : cluster 0 2.5 scrub ok 2024-03-11T21:18:29.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:28 smithi023 bash[29146]: cluster 2024-03-11T21:18:27.618425+0000 osd.5 (osd.5) 641 : cluster 0 3.c scrub starts 2024-03-11T21:18:29.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:28 smithi023 bash[29146]: cluster 2024-03-11T21:18:27.622333+0000 osd.5 (osd.5) 642 : cluster 0 3.c scrub ok 2024-03-11T21:18:29.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:28 smithi023 bash[29146]: cluster 2024-03-11T21:18:27.963412+0000 osd.1 (osd.1) 406 : cluster 0 6.1d scrub starts 2024-03-11T21:18:29.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:28 smithi023 bash[29146]: cluster 2024-03-11T21:18:28.129230+0000 osd.1 (osd.1) 407 : cluster 0 6.1d scrub ok 2024-03-11T21:18:30.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:29 smithi146 bash[23809]: cluster 2024-03-11T21:18:28.542315+0000 mgr.y (mgr.24380) 1021 : cluster 0 pgmap v1190: 152 pgs: 1 active+clean+scrubbing, 151 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:18:30.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:29 smithi146 bash[23809]: cluster 2024-03-11T21:18:28.998072+0000 osd.1 (osd.1) 408 : cluster 0 4.2 scrub starts 2024-03-11T21:18:30.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:29 smithi146 bash[23809]: cluster 2024-03-11T21:18:29.000737+0000 osd.1 (osd.1) 409 : cluster 0 4.2 scrub ok 2024-03-11T21:18:30.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:29 smithi023 bash[29146]: cluster 2024-03-11T21:18:28.542315+0000 mgr.y (mgr.24380) 1021 : cluster 0 pgmap v1190: 152 pgs: 1 active+clean+scrubbing, 151 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:18:30.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:29 smithi023 bash[29146]: cluster 2024-03-11T21:18:28.998072+0000 osd.1 (osd.1) 408 : cluster 0 4.2 scrub starts 2024-03-11T21:18:30.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:29 smithi023 bash[29146]: cluster 2024-03-11T21:18:29.000737+0000 osd.1 (osd.1) 409 : cluster 0 4.2 scrub ok 2024-03-11T21:18:30.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:29 smithi023 bash[21372]: cluster 2024-03-11T21:18:28.542315+0000 mgr.y (mgr.24380) 1021 : cluster 0 pgmap v1190: 152 pgs: 1 active+clean+scrubbing, 151 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:18:30.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:29 smithi023 bash[21372]: cluster 2024-03-11T21:18:28.998072+0000 osd.1 (osd.1) 408 : cluster 0 4.2 scrub starts 2024-03-11T21:18:30.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:29 smithi023 bash[21372]: cluster 2024-03-11T21:18:29.000737+0000 osd.1 (osd.1) 409 : cluster 0 4.2 scrub ok 2024-03-11T21:18:30.449 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-03-11T21:18:32.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:31 smithi146 bash[23809]: cluster 2024-03-11T21:18:30.315807+0000 osd.7 (osd.7) 671 : cluster 0 3.11 scrub starts 2024-03-11T21:18:32.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:31 smithi146 bash[23809]: cluster 2024-03-11T21:18:30.320863+0000 osd.7 (osd.7) 672 : cluster 0 3.11 scrub ok 2024-03-11T21:18:32.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:31 smithi146 bash[23809]: cluster 2024-03-11T21:18:30.542960+0000 mgr.y (mgr.24380) 1022 : cluster 0 pgmap v1191: 152 pgs: 1 active+clean+scrubbing, 151 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 566 B/s rd, 0 op/s 2024-03-11T21:18:32.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:31 smithi146 bash[23809]: cluster 2024-03-11T21:18:30.662222+0000 osd.5 (osd.5) 643 : cluster 0 6.d scrub starts 2024-03-11T21:18:32.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:31 smithi146 bash[23809]: cluster 2024-03-11T21:18:30.784218+0000 osd.5 (osd.5) 644 : cluster 0 6.d scrub ok 2024-03-11T21:18:32.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:31 smithi146 bash[23809]: cluster 2024-03-11T21:18:30.955433+0000 osd.1 (osd.1) 410 : cluster 0 6.1 scrub starts 2024-03-11T21:18:32.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:31 smithi146 bash[23809]: cluster 2024-03-11T21:18:31.036943+0000 osd.1 (osd.1) 411 : cluster 0 6.1 scrub ok 2024-03-11T21:18:32.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:31 smithi146 bash[23809]: cluster 2024-03-11T21:18:31.401610+0000 osd.0 (osd.0) 486 : cluster 0 4.3 scrub starts 2024-03-11T21:18:32.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:31 smithi146 bash[23809]: cluster 2024-03-11T21:18:31.405678+0000 osd.0 (osd.0) 487 : cluster 0 4.3 scrub ok 2024-03-11T21:18:32.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:31 smithi023 bash[21372]: cluster 2024-03-11T21:18:30.315807+0000 osd.7 (osd.7) 671 : cluster 0 3.11 scrub starts 2024-03-11T21:18:32.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:31 smithi023 bash[21372]: cluster 2024-03-11T21:18:30.320863+0000 osd.7 (osd.7) 672 : cluster 0 3.11 scrub ok 2024-03-11T21:18:32.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:31 smithi023 bash[21372]: cluster 2024-03-11T21:18:30.542960+0000 mgr.y (mgr.24380) 1022 : cluster 0 pgmap v1191: 152 pgs: 1 active+clean+scrubbing, 151 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 566 B/s rd, 0 op/s 2024-03-11T21:18:32.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:31 smithi023 bash[21372]: cluster 2024-03-11T21:18:30.662222+0000 osd.5 (osd.5) 643 : cluster 0 6.d scrub starts 2024-03-11T21:18:32.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:31 smithi023 bash[21372]: cluster 2024-03-11T21:18:30.784218+0000 osd.5 (osd.5) 644 : cluster 0 6.d scrub ok 2024-03-11T21:18:32.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:31 smithi023 bash[21372]: cluster 2024-03-11T21:18:30.955433+0000 osd.1 (osd.1) 410 : cluster 0 6.1 scrub starts 2024-03-11T21:18:32.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:31 smithi023 bash[21372]: cluster 2024-03-11T21:18:31.036943+0000 osd.1 (osd.1) 411 : cluster 0 6.1 scrub ok 2024-03-11T21:18:32.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:31 smithi023 bash[21372]: cluster 2024-03-11T21:18:31.401610+0000 osd.0 (osd.0) 486 : cluster 0 4.3 scrub starts 2024-03-11T21:18:32.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:31 smithi023 bash[21372]: cluster 2024-03-11T21:18:31.405678+0000 osd.0 (osd.0) 487 : cluster 0 4.3 scrub ok 2024-03-11T21:18:32.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:31 smithi023 bash[29146]: cluster 2024-03-11T21:18:30.315807+0000 osd.7 (osd.7) 671 : cluster 0 3.11 scrub starts 2024-03-11T21:18:32.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:31 smithi023 bash[29146]: cluster 2024-03-11T21:18:30.320863+0000 osd.7 (osd.7) 672 : cluster 0 3.11 scrub ok 2024-03-11T21:18:32.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:31 smithi023 bash[29146]: cluster 2024-03-11T21:18:30.542960+0000 mgr.y (mgr.24380) 1022 : cluster 0 pgmap v1191: 152 pgs: 1 active+clean+scrubbing, 151 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 566 B/s rd, 0 op/s 2024-03-11T21:18:32.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:31 smithi023 bash[29146]: cluster 2024-03-11T21:18:30.662222+0000 osd.5 (osd.5) 643 : cluster 0 6.d scrub starts 2024-03-11T21:18:32.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:31 smithi023 bash[29146]: cluster 2024-03-11T21:18:30.784218+0000 osd.5 (osd.5) 644 : cluster 0 6.d scrub ok 2024-03-11T21:18:32.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:31 smithi023 bash[29146]: cluster 2024-03-11T21:18:30.955433+0000 osd.1 (osd.1) 410 : cluster 0 6.1 scrub starts 2024-03-11T21:18:32.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:31 smithi023 bash[29146]: cluster 2024-03-11T21:18:31.036943+0000 osd.1 (osd.1) 411 : cluster 0 6.1 scrub ok 2024-03-11T21:18:32.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:31 smithi023 bash[29146]: cluster 2024-03-11T21:18:31.401610+0000 osd.0 (osd.0) 486 : cluster 0 4.3 scrub starts 2024-03-11T21:18:32.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:31 smithi023 bash[29146]: cluster 2024-03-11T21:18:31.405678+0000 osd.0 (osd.0) 487 : cluster 0 4.3 scrub ok 2024-03-11T21:18:33.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:32 smithi146 bash[23809]: cluster 2024-03-11T21:18:31.174233+0000 osd.2 (osd.2) 293 : cluster 0 3.9 scrub starts 2024-03-11T21:18:33.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:32 smithi146 bash[23809]: cluster 2024-03-11T21:18:31.178744+0000 osd.2 (osd.2) 294 : cluster 0 3.9 scrub ok 2024-03-11T21:18:33.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:32 smithi146 bash[23809]: cluster 2024-03-11T21:18:31.315844+0000 osd.7 (osd.7) 673 : cluster 0 2.8 scrub starts 2024-03-11T21:18:33.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:32 smithi146 bash[23809]: cluster 2024-03-11T21:18:31.318698+0000 osd.7 (osd.7) 674 : cluster 0 2.8 scrub ok 2024-03-11T21:18:33.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:32 smithi146 bash[23809]: cluster 2024-03-11T21:18:32.296800+0000 osd.6 (osd.6) 534 : cluster 0 4.e scrub starts 2024-03-11T21:18:33.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:32 smithi146 bash[23809]: cluster 2024-03-11T21:18:32.300035+0000 osd.6 (osd.6) 535 : cluster 0 4.e scrub ok 2024-03-11T21:18:33.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:32 smithi023 bash[21372]: cluster 2024-03-11T21:18:31.174233+0000 osd.2 (osd.2) 293 : cluster 0 3.9 scrub starts 2024-03-11T21:18:33.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:32 smithi023 bash[21372]: cluster 2024-03-11T21:18:31.178744+0000 osd.2 (osd.2) 294 : cluster 0 3.9 scrub ok 2024-03-11T21:18:33.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:32 smithi023 bash[21372]: cluster 2024-03-11T21:18:31.315844+0000 osd.7 (osd.7) 673 : cluster 0 2.8 scrub starts 2024-03-11T21:18:33.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:32 smithi023 bash[21372]: cluster 2024-03-11T21:18:31.318698+0000 osd.7 (osd.7) 674 : cluster 0 2.8 scrub ok 2024-03-11T21:18:33.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:32 smithi023 bash[21372]: cluster 2024-03-11T21:18:32.296800+0000 osd.6 (osd.6) 534 : cluster 0 4.e scrub starts 2024-03-11T21:18:33.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:32 smithi023 bash[21372]: cluster 2024-03-11T21:18:32.300035+0000 osd.6 (osd.6) 535 : cluster 0 4.e scrub ok 2024-03-11T21:18:33.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:32 smithi023 bash[29146]: cluster 2024-03-11T21:18:31.174233+0000 osd.2 (osd.2) 293 : cluster 0 3.9 scrub starts 2024-03-11T21:18:33.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:32 smithi023 bash[29146]: cluster 2024-03-11T21:18:31.178744+0000 osd.2 (osd.2) 294 : cluster 0 3.9 scrub ok 2024-03-11T21:18:33.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:32 smithi023 bash[29146]: cluster 2024-03-11T21:18:31.315844+0000 osd.7 (osd.7) 673 : cluster 0 2.8 scrub starts 2024-03-11T21:18:33.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:32 smithi023 bash[29146]: cluster 2024-03-11T21:18:31.318698+0000 osd.7 (osd.7) 674 : cluster 0 2.8 scrub ok 2024-03-11T21:18:33.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:32 smithi023 bash[29146]: cluster 2024-03-11T21:18:32.296800+0000 osd.6 (osd.6) 534 : cluster 0 4.e scrub starts 2024-03-11T21:18:33.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:32 smithi023 bash[29146]: cluster 2024-03-11T21:18:32.300035+0000 osd.6 (osd.6) 535 : cluster 0 4.e scrub ok 2024-03-11T21:18:33.730 INFO:teuthology.orchestra.run.smithi023.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-03-11T21:18:34.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:33 smithi146 bash[23809]: cluster 2024-03-11T21:18:32.543619+0000 mgr.y (mgr.24380) 1023 : cluster 0 pgmap v1192: 152 pgs: 2 active+clean+scrubbing, 150 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:18:34.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:33 smithi146 bash[23809]: audit 2024-03-11T21:18:32.779108+0000 mon.a (mon.0) 2077 : audit 1 from='client.? 172.21.15.23:0/134990663' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-03-11T21:18:34.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:33 smithi146 bash[23809]: cluster 2024-03-11T21:18:33.322385+0000 osd.6 (osd.6) 536 : cluster 0 4.5 deep-scrub starts 2024-03-11T21:18:34.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:33 smithi146 bash[23809]: cluster 2024-03-11T21:18:33.325153+0000 osd.6 (osd.6) 537 : cluster 0 4.5 deep-scrub ok 2024-03-11T21:18:34.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:33 smithi023 bash[29146]: cluster 2024-03-11T21:18:32.543619+0000 mgr.y (mgr.24380) 1023 : cluster 0 pgmap v1192: 152 pgs: 2 active+clean+scrubbing, 150 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:18:34.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:33 smithi023 bash[29146]: audit 2024-03-11T21:18:32.779108+0000 mon.a (mon.0) 2077 : audit 1 from='client.? 172.21.15.23:0/134990663' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-03-11T21:18:34.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:33 smithi023 bash[29146]: cluster 2024-03-11T21:18:33.322385+0000 osd.6 (osd.6) 536 : cluster 0 4.5 deep-scrub starts 2024-03-11T21:18:34.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:33 smithi023 bash[29146]: cluster 2024-03-11T21:18:33.325153+0000 osd.6 (osd.6) 537 : cluster 0 4.5 deep-scrub ok 2024-03-11T21:18:34.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:33 smithi023 bash[21372]: cluster 2024-03-11T21:18:32.543619+0000 mgr.y (mgr.24380) 1023 : cluster 0 pgmap v1192: 152 pgs: 2 active+clean+scrubbing, 150 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:18:34.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:33 smithi023 bash[21372]: audit 2024-03-11T21:18:32.779108+0000 mon.a (mon.0) 2077 : audit 1 from='client.? 172.21.15.23:0/134990663' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-03-11T21:18:34.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:33 smithi023 bash[21372]: cluster 2024-03-11T21:18:33.322385+0000 osd.6 (osd.6) 536 : cluster 0 4.5 deep-scrub starts 2024-03-11T21:18:34.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:33 smithi023 bash[21372]: cluster 2024-03-11T21:18:33.325153+0000 osd.6 (osd.6) 537 : cluster 0 4.5 deep-scrub ok 2024-03-11T21:18:34.582 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd reweight 4 1 2024-03-11T21:18:35.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:34 smithi146 bash[23809]: cluster 2024-03-11T21:18:33.602345+0000 osd.3 (osd.3) 582 : cluster 0 1.c scrub starts 2024-03-11T21:18:35.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:34 smithi146 bash[23809]: cluster 2024-03-11T21:18:33.605498+0000 osd.3 (osd.3) 583 : cluster 0 1.c scrub ok 2024-03-11T21:18:35.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:34 smithi146 bash[23809]: audit 2024-03-11T21:18:33.736620+0000 mon.a (mon.0) 2078 : audit 1 from='client.? 172.21.15.23:0/134990663' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-03-11T21:18:35.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:34 smithi146 bash[23809]: cluster 2024-03-11T21:18:33.742608+0000 mon.a (mon.0) 2079 : cluster 0 osdmap e482: 8 total, 8 up, 8 in 2024-03-11T21:18:35.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:34 smithi146 bash[23809]: cluster 2024-03-11T21:18:34.265369+0000 osd.0 (osd.0) 488 : cluster 0 purged_snaps scrub starts 2024-03-11T21:18:35.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:34 smithi146 bash[23809]: cluster 2024-03-11T21:18:34.265468+0000 osd.0 (osd.0) 489 : cluster 0 purged_snaps scrub ok 2024-03-11T21:18:35.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:34 smithi146 bash[23809]: cluster 2024-03-11T21:18:34.349130+0000 osd.6 (osd.6) 538 : cluster 0 2.14 scrub starts 2024-03-11T21:18:35.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:34 smithi146 bash[23809]: cluster 2024-03-11T21:18:34.351491+0000 osd.6 (osd.6) 539 : cluster 0 2.14 scrub ok 2024-03-11T21:18:35.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:34 smithi023 bash[21372]: cluster 2024-03-11T21:18:33.602345+0000 osd.3 (osd.3) 582 : cluster 0 1.c scrub starts 2024-03-11T21:18:35.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:34 smithi023 bash[21372]: cluster 2024-03-11T21:18:33.605498+0000 osd.3 (osd.3) 583 : cluster 0 1.c scrub ok 2024-03-11T21:18:35.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:34 smithi023 bash[21372]: audit 2024-03-11T21:18:33.736620+0000 mon.a (mon.0) 2078 : audit 1 from='client.? 172.21.15.23:0/134990663' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-03-11T21:18:35.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:34 smithi023 bash[21372]: cluster 2024-03-11T21:18:33.742608+0000 mon.a (mon.0) 2079 : cluster 0 osdmap e482: 8 total, 8 up, 8 in 2024-03-11T21:18:35.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:34 smithi023 bash[21372]: cluster 2024-03-11T21:18:34.265369+0000 osd.0 (osd.0) 488 : cluster 0 purged_snaps scrub starts 2024-03-11T21:18:35.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:34 smithi023 bash[21372]: cluster 2024-03-11T21:18:34.265468+0000 osd.0 (osd.0) 489 : cluster 0 purged_snaps scrub ok 2024-03-11T21:18:35.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:34 smithi023 bash[21372]: cluster 2024-03-11T21:18:34.349130+0000 osd.6 (osd.6) 538 : cluster 0 2.14 scrub starts 2024-03-11T21:18:35.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:34 smithi023 bash[21372]: cluster 2024-03-11T21:18:34.351491+0000 osd.6 (osd.6) 539 : cluster 0 2.14 scrub ok 2024-03-11T21:18:35.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:34 smithi023 bash[29146]: cluster 2024-03-11T21:18:33.602345+0000 osd.3 (osd.3) 582 : cluster 0 1.c scrub starts 2024-03-11T21:18:35.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:34 smithi023 bash[29146]: cluster 2024-03-11T21:18:33.605498+0000 osd.3 (osd.3) 583 : cluster 0 1.c scrub ok 2024-03-11T21:18:35.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:34 smithi023 bash[29146]: audit 2024-03-11T21:18:33.736620+0000 mon.a (mon.0) 2078 : audit 1 from='client.? 172.21.15.23:0/134990663' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-03-11T21:18:35.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:34 smithi023 bash[29146]: cluster 2024-03-11T21:18:33.742608+0000 mon.a (mon.0) 2079 : cluster 0 osdmap e482: 8 total, 8 up, 8 in 2024-03-11T21:18:35.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:34 smithi023 bash[29146]: cluster 2024-03-11T21:18:34.265369+0000 osd.0 (osd.0) 488 : cluster 0 purged_snaps scrub starts 2024-03-11T21:18:35.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:34 smithi023 bash[29146]: cluster 2024-03-11T21:18:34.265468+0000 osd.0 (osd.0) 489 : cluster 0 purged_snaps scrub ok 2024-03-11T21:18:35.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:34 smithi023 bash[29146]: cluster 2024-03-11T21:18:34.349130+0000 osd.6 (osd.6) 538 : cluster 0 2.14 scrub starts 2024-03-11T21:18:35.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:34 smithi023 bash[29146]: cluster 2024-03-11T21:18:34.351491+0000 osd.6 (osd.6) 539 : cluster 0 2.14 scrub ok 2024-03-11T21:18:36.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:35 smithi146 bash[23809]: cluster 2024-03-11T21:18:34.544390+0000 mgr.y (mgr.24380) 1024 : cluster 0 pgmap v1194: 152 pgs: 1 active+clean+scrubbing, 151 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:18:36.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:35 smithi146 bash[23809]: cluster 2024-03-11T21:18:34.641374+0000 osd.3 (osd.3) 584 : cluster 0 6.1b scrub starts 2024-03-11T21:18:36.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:35 smithi146 bash[23809]: cluster 2024-03-11T21:18:34.765427+0000 mon.a (mon.0) 2080 : cluster 0 osdmap e483: 8 total, 8 up, 8 in 2024-03-11T21:18:36.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:35 smithi146 bash[23809]: cluster 2024-03-11T21:18:34.839608+0000 osd.3 (osd.3) 585 : cluster 0 6.1b scrub ok 2024-03-11T21:18:36.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:35 smithi023 bash[21372]: cluster 2024-03-11T21:18:34.544390+0000 mgr.y (mgr.24380) 1024 : cluster 0 pgmap v1194: 152 pgs: 1 active+clean+scrubbing, 151 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:18:36.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:35 smithi023 bash[21372]: cluster 2024-03-11T21:18:34.641374+0000 osd.3 (osd.3) 584 : cluster 0 6.1b scrub starts 2024-03-11T21:18:36.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:35 smithi023 bash[21372]: cluster 2024-03-11T21:18:34.765427+0000 mon.a (mon.0) 2080 : cluster 0 osdmap e483: 8 total, 8 up, 8 in 2024-03-11T21:18:36.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:35 smithi023 bash[21372]: cluster 2024-03-11T21:18:34.839608+0000 osd.3 (osd.3) 585 : cluster 0 6.1b scrub ok 2024-03-11T21:18:36.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:35 smithi023 bash[29146]: cluster 2024-03-11T21:18:34.544390+0000 mgr.y (mgr.24380) 1024 : cluster 0 pgmap v1194: 152 pgs: 1 active+clean+scrubbing, 151 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:18:36.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:35 smithi023 bash[29146]: cluster 2024-03-11T21:18:34.641374+0000 osd.3 (osd.3) 584 : cluster 0 6.1b scrub starts 2024-03-11T21:18:36.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:35 smithi023 bash[29146]: cluster 2024-03-11T21:18:34.765427+0000 mon.a (mon.0) 2080 : cluster 0 osdmap e483: 8 total, 8 up, 8 in 2024-03-11T21:18:36.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:35 smithi023 bash[29146]: cluster 2024-03-11T21:18:34.839608+0000 osd.3 (osd.3) 585 : cluster 0 6.1b scrub ok 2024-03-11T21:18:36.244 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:18:36.502 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:18:36 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:18:37.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:36 smithi023 bash[29146]: cluster 2024-03-11T21:18:35.300319+0000 osd.7 (osd.7) 675 : cluster 0 1.0 deep-scrub starts 2024-03-11T21:18:37.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:36 smithi023 bash[29146]: cluster 2024-03-11T21:18:35.305676+0000 osd.7 (osd.7) 676 : cluster 0 1.0 deep-scrub ok 2024-03-11T21:18:37.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:36 smithi023 bash[29146]: cluster 2024-03-11T21:18:35.594440+0000 osd.3 (osd.3) 586 : cluster 0 4.0 scrub starts 2024-03-11T21:18:37.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:36 smithi023 bash[29146]: cluster 2024-03-11T21:18:35.597823+0000 osd.3 (osd.3) 587 : cluster 0 4.0 scrub ok 2024-03-11T21:18:37.026 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:36 smithi023 bash[21372]: cluster 2024-03-11T21:18:35.300319+0000 osd.7 (osd.7) 675 : cluster 0 1.0 deep-scrub starts 2024-03-11T21:18:37.026 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:36 smithi023 bash[21372]: cluster 2024-03-11T21:18:35.305676+0000 osd.7 (osd.7) 676 : cluster 0 1.0 deep-scrub ok 2024-03-11T21:18:37.026 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:36 smithi023 bash[21372]: cluster 2024-03-11T21:18:35.594440+0000 osd.3 (osd.3) 586 : cluster 0 4.0 scrub starts 2024-03-11T21:18:37.026 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:36 smithi023 bash[21372]: cluster 2024-03-11T21:18:35.597823+0000 osd.3 (osd.3) 587 : cluster 0 4.0 scrub ok 2024-03-11T21:18:37.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:36 smithi146 bash[23809]: cluster 2024-03-11T21:18:35.300319+0000 osd.7 (osd.7) 675 : cluster 0 1.0 deep-scrub starts 2024-03-11T21:18:37.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:36 smithi146 bash[23809]: cluster 2024-03-11T21:18:35.305676+0000 osd.7 (osd.7) 676 : cluster 0 1.0 deep-scrub ok 2024-03-11T21:18:37.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:36 smithi146 bash[23809]: cluster 2024-03-11T21:18:35.594440+0000 osd.3 (osd.3) 586 : cluster 0 4.0 scrub starts 2024-03-11T21:18:37.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:36 smithi146 bash[23809]: cluster 2024-03-11T21:18:35.597823+0000 osd.3 (osd.3) 587 : cluster 0 4.0 scrub ok 2024-03-11T21:18:38.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:37 smithi023 bash[21372]: cluster 2024-03-11T21:18:36.153303+0000 osd.2 (osd.2) 295 : cluster 0 6.f scrub starts 2024-03-11T21:18:38.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:37 smithi023 bash[21372]: audit 2024-03-11T21:18:36.262853+0000 mgr.y (mgr.24380) 1025 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:18:38.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:37 smithi023 bash[21372]: cluster 2024-03-11T21:18:36.358503+0000 osd.2 (osd.2) 296 : cluster 0 6.f scrub ok 2024-03-11T21:18:38.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:37 smithi023 bash[21372]: cluster 2024-03-11T21:18:36.546015+0000 mgr.y (mgr.24380) 1026 : cluster 0 pgmap v1196: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:18:38.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:37 smithi023 bash[21372]: cluster 2024-03-11T21:18:36.674708+0000 osd.5 (osd.5) 645 : cluster 0 4.8 scrub starts 2024-03-11T21:18:38.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:37 smithi023 bash[21372]: cluster 2024-03-11T21:18:36.677133+0000 osd.5 (osd.5) 646 : cluster 0 4.8 scrub ok 2024-03-11T21:18:38.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:37 smithi023 bash[21372]: audit 2024-03-11T21:18:36.960822+0000 mon.a (mon.0) 2081 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:18:38.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:37 smithi023 bash[29146]: cluster 2024-03-11T21:18:36.153303+0000 osd.2 (osd.2) 295 : cluster 0 6.f scrub starts 2024-03-11T21:18:38.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:37 smithi023 bash[29146]: audit 2024-03-11T21:18:36.262853+0000 mgr.y (mgr.24380) 1025 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:18:38.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:37 smithi023 bash[29146]: cluster 2024-03-11T21:18:36.358503+0000 osd.2 (osd.2) 296 : cluster 0 6.f scrub ok 2024-03-11T21:18:38.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:37 smithi023 bash[29146]: cluster 2024-03-11T21:18:36.546015+0000 mgr.y (mgr.24380) 1026 : cluster 0 pgmap v1196: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:18:38.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:37 smithi023 bash[29146]: cluster 2024-03-11T21:18:36.674708+0000 osd.5 (osd.5) 645 : cluster 0 4.8 scrub starts 2024-03-11T21:18:38.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:37 smithi023 bash[29146]: cluster 2024-03-11T21:18:36.677133+0000 osd.5 (osd.5) 646 : cluster 0 4.8 scrub ok 2024-03-11T21:18:38.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:37 smithi023 bash[29146]: audit 2024-03-11T21:18:36.960822+0000 mon.a (mon.0) 2081 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:18:38.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:37 smithi146 bash[23809]: cluster 2024-03-11T21:18:36.153303+0000 osd.2 (osd.2) 295 : cluster 0 6.f scrub starts 2024-03-11T21:18:38.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:37 smithi146 bash[23809]: audit 2024-03-11T21:18:36.262853+0000 mgr.y (mgr.24380) 1025 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:18:38.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:37 smithi146 bash[23809]: cluster 2024-03-11T21:18:36.358503+0000 osd.2 (osd.2) 296 : cluster 0 6.f scrub ok 2024-03-11T21:18:38.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:37 smithi146 bash[23809]: cluster 2024-03-11T21:18:36.546015+0000 mgr.y (mgr.24380) 1026 : cluster 0 pgmap v1196: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:18:38.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:37 smithi146 bash[23809]: cluster 2024-03-11T21:18:36.674708+0000 osd.5 (osd.5) 645 : cluster 0 4.8 scrub starts 2024-03-11T21:18:38.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:37 smithi146 bash[23809]: cluster 2024-03-11T21:18:36.677133+0000 osd.5 (osd.5) 646 : cluster 0 4.8 scrub ok 2024-03-11T21:18:38.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:37 smithi146 bash[23809]: audit 2024-03-11T21:18:36.960822+0000 mon.a (mon.0) 2081 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:18:38.524 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:18:38 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:18:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:18:39.142 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:38 smithi023 bash[21372]: cluster 2024-03-11T21:18:37.661079+0000 osd.5 (osd.5) 647 : cluster 0 6.7 scrub starts 2024-03-11T21:18:39.142 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:38 smithi023 bash[21372]: cluster 2024-03-11T21:18:37.798651+0000 osd.5 (osd.5) 648 : cluster 0 6.7 scrub ok 2024-03-11T21:18:39.142 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:38 smithi023 bash[21372]: cluster 2024-03-11T21:18:37.971092+0000 osd.1 (osd.1) 412 : cluster 0 2.6 scrub starts 2024-03-11T21:18:39.142 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:38 smithi023 bash[21372]: cluster 2024-03-11T21:18:37.974162+0000 osd.1 (osd.1) 413 : cluster 0 2.6 scrub ok 2024-03-11T21:18:39.143 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:38 smithi023 bash[29146]: cluster 2024-03-11T21:18:37.661079+0000 osd.5 (osd.5) 647 : cluster 0 6.7 scrub starts 2024-03-11T21:18:39.143 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:38 smithi023 bash[29146]: cluster 2024-03-11T21:18:37.798651+0000 osd.5 (osd.5) 648 : cluster 0 6.7 scrub ok 2024-03-11T21:18:39.143 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:38 smithi023 bash[29146]: cluster 2024-03-11T21:18:37.971092+0000 osd.1 (osd.1) 412 : cluster 0 2.6 scrub starts 2024-03-11T21:18:39.143 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:38 smithi023 bash[29146]: cluster 2024-03-11T21:18:37.974162+0000 osd.1 (osd.1) 413 : cluster 0 2.6 scrub ok 2024-03-11T21:18:39.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:38 smithi146 bash[23809]: cluster 2024-03-11T21:18:37.661079+0000 osd.5 (osd.5) 647 : cluster 0 6.7 scrub starts 2024-03-11T21:18:39.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:38 smithi146 bash[23809]: cluster 2024-03-11T21:18:37.798651+0000 osd.5 (osd.5) 648 : cluster 0 6.7 scrub ok 2024-03-11T21:18:39.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:38 smithi146 bash[23809]: cluster 2024-03-11T21:18:37.971092+0000 osd.1 (osd.1) 412 : cluster 0 2.6 scrub starts 2024-03-11T21:18:39.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:38 smithi146 bash[23809]: cluster 2024-03-11T21:18:37.974162+0000 osd.1 (osd.1) 413 : cluster 0 2.6 scrub ok 2024-03-11T21:18:39.310 INFO:teuthology.orchestra.run.smithi023.stdout:ERROR: (22) Invalid argument 2024-03-11T21:18:39.310 INFO:teuthology.orchestra.run.smithi023.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:18:39.315 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:18:40.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:39 smithi146 bash[23809]: cluster 2024-03-11T21:18:38.546996+0000 mgr.y (mgr.24380) 1027 : cluster 0 pgmap v1197: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:18:40.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:39 smithi146 bash[23809]: cluster 2024-03-11T21:18:38.615345+0000 osd.5 (osd.5) 649 : cluster 0 3.1c scrub starts 2024-03-11T21:18:40.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:39 smithi146 bash[23809]: cluster 2024-03-11T21:18:38.620413+0000 osd.5 (osd.5) 650 : cluster 0 3.1c scrub ok 2024-03-11T21:18:40.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:39 smithi146 bash[23809]: cluster 2024-03-11T21:18:39.249983+0000 osd.0 (osd.0) 490 : cluster 0 4.16 scrub starts 2024-03-11T21:18:40.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:39 smithi146 bash[23809]: cluster 2024-03-11T21:18:39.251949+0000 osd.0 (osd.0) 491 : cluster 0 4.16 scrub ok 2024-03-11T21:18:40.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:39 smithi023 bash[21372]: cluster 2024-03-11T21:18:38.546996+0000 mgr.y (mgr.24380) 1027 : cluster 0 pgmap v1197: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:18:40.275 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:39 smithi023 bash[21372]: cluster 2024-03-11T21:18:38.615345+0000 osd.5 (osd.5) 649 : cluster 0 3.1c scrub starts 2024-03-11T21:18:40.275 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:39 smithi023 bash[21372]: cluster 2024-03-11T21:18:38.620413+0000 osd.5 (osd.5) 650 : cluster 0 3.1c scrub ok 2024-03-11T21:18:40.275 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:39 smithi023 bash[21372]: cluster 2024-03-11T21:18:39.249983+0000 osd.0 (osd.0) 490 : cluster 0 4.16 scrub starts 2024-03-11T21:18:40.275 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:39 smithi023 bash[21372]: cluster 2024-03-11T21:18:39.251949+0000 osd.0 (osd.0) 491 : cluster 0 4.16 scrub ok 2024-03-11T21:18:40.275 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:39 smithi023 bash[29146]: cluster 2024-03-11T21:18:38.546996+0000 mgr.y (mgr.24380) 1027 : cluster 0 pgmap v1197: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:18:40.276 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:39 smithi023 bash[29146]: cluster 2024-03-11T21:18:38.615345+0000 osd.5 (osd.5) 649 : cluster 0 3.1c scrub starts 2024-03-11T21:18:40.276 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:39 smithi023 bash[29146]: cluster 2024-03-11T21:18:38.620413+0000 osd.5 (osd.5) 650 : cluster 0 3.1c scrub ok 2024-03-11T21:18:40.276 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:39 smithi023 bash[29146]: cluster 2024-03-11T21:18:39.249983+0000 osd.0 (osd.0) 490 : cluster 0 4.16 scrub starts 2024-03-11T21:18:40.276 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:39 smithi023 bash[29146]: cluster 2024-03-11T21:18:39.251949+0000 osd.0 (osd.0) 491 : cluster 0 4.16 scrub ok 2024-03-11T21:18:41.774 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:41 smithi023 bash[21372]: cluster 2024-03-11T21:18:40.548128+0000 mgr.y (mgr.24380) 1028 : cluster 0 pgmap v1198: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:18:41.774 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:41 smithi023 bash[21372]: cluster 2024-03-11T21:18:41.073143+0000 osd.1 (osd.1) 414 : cluster 0 2.15 scrub starts 2024-03-11T21:18:41.774 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:41 smithi023 bash[21372]: cluster 2024-03-11T21:18:41.074909+0000 osd.1 (osd.1) 415 : cluster 0 2.15 scrub ok 2024-03-11T21:18:41.774 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:41 smithi023 bash[29146]: cluster 2024-03-11T21:18:40.548128+0000 mgr.y (mgr.24380) 1028 : cluster 0 pgmap v1198: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:18:41.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:41 smithi023 bash[29146]: cluster 2024-03-11T21:18:41.073143+0000 osd.1 (osd.1) 414 : cluster 0 2.15 scrub starts 2024-03-11T21:18:41.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:41 smithi023 bash[29146]: cluster 2024-03-11T21:18:41.074909+0000 osd.1 (osd.1) 415 : cluster 0 2.15 scrub ok 2024-03-11T21:18:42.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:41 smithi146 bash[23809]: cluster 2024-03-11T21:18:40.548128+0000 mgr.y (mgr.24380) 1028 : cluster 0 pgmap v1198: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:18:42.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:41 smithi146 bash[23809]: cluster 2024-03-11T21:18:41.073143+0000 osd.1 (osd.1) 414 : cluster 0 2.15 scrub starts 2024-03-11T21:18:42.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:41 smithi146 bash[23809]: cluster 2024-03-11T21:18:41.074909+0000 osd.1 (osd.1) 415 : cluster 0 2.15 scrub ok 2024-03-11T21:18:42.774 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:42 smithi023 bash[21372]: cluster 2024-03-11T21:18:41.528440+0000 osd.3 (osd.3) 588 : cluster 0 4.14 scrub starts 2024-03-11T21:18:42.774 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:42 smithi023 bash[21372]: cluster 2024-03-11T21:18:41.531435+0000 osd.3 (osd.3) 589 : cluster 0 4.14 scrub ok 2024-03-11T21:18:42.774 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:42 smithi023 bash[21372]: cluster 2024-03-11T21:18:41.591191+0000 osd.5 (osd.5) 651 : cluster 0 6.19 deep-scrub starts 2024-03-11T21:18:42.774 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:42 smithi023 bash[29146]: cluster 2024-03-11T21:18:41.528440+0000 osd.3 (osd.3) 588 : cluster 0 4.14 scrub starts 2024-03-11T21:18:42.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:42 smithi023 bash[29146]: cluster 2024-03-11T21:18:41.531435+0000 osd.3 (osd.3) 589 : cluster 0 4.14 scrub ok 2024-03-11T21:18:42.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:42 smithi023 bash[29146]: cluster 2024-03-11T21:18:41.591191+0000 osd.5 (osd.5) 651 : cluster 0 6.19 deep-scrub starts 2024-03-11T21:18:43.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:42 smithi146 bash[23809]: cluster 2024-03-11T21:18:41.528440+0000 osd.3 (osd.3) 588 : cluster 0 4.14 scrub starts 2024-03-11T21:18:43.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:42 smithi146 bash[23809]: cluster 2024-03-11T21:18:41.531435+0000 osd.3 (osd.3) 589 : cluster 0 4.14 scrub ok 2024-03-11T21:18:43.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:42 smithi146 bash[23809]: cluster 2024-03-11T21:18:41.591191+0000 osd.5 (osd.5) 651 : cluster 0 6.19 deep-scrub starts 2024-03-11T21:18:43.530 INFO:teuthology.orchestra.run.smithi023.stderr:reweighted osd.4 to 1 (10000) 2024-03-11T21:18:44.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:43 smithi146 bash[23809]: cluster 2024-03-11T21:18:41.711855+0000 osd.5 (osd.5) 652 : cluster 0 6.19 deep-scrub ok 2024-03-11T21:18:44.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:43 smithi146 bash[23809]: cluster 2024-03-11T21:18:42.111216+0000 osd.2 (osd.2) 297 : cluster 0 1.5 deep-scrub starts 2024-03-11T21:18:44.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:43 smithi146 bash[23809]: cluster 2024-03-11T21:18:42.113931+0000 osd.2 (osd.2) 298 : cluster 0 1.5 deep-scrub ok 2024-03-11T21:18:44.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:43 smithi146 bash[23809]: cluster 2024-03-11T21:18:42.286120+0000 osd.7 (osd.7) 677 : cluster 0 3.f scrub starts 2024-03-11T21:18:44.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:43 smithi146 bash[23809]: cluster 2024-03-11T21:18:42.289246+0000 osd.7 (osd.7) 678 : cluster 0 3.f scrub ok 2024-03-11T21:18:44.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:43 smithi146 bash[23809]: cluster 2024-03-11T21:18:42.549327+0000 mgr.y (mgr.24380) 1029 : cluster 0 pgmap v1199: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T21:18:44.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:43 smithi146 bash[23809]: audit 2024-03-11T21:18:43.270232+0000 mon.c (mon.2) 111 : audit 1 from='client.? 172.21.15.23:0/3829405530' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-11T21:18:44.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:43 smithi146 bash[23809]: audit 2024-03-11T21:18:43.270876+0000 mon.a (mon.0) 2082 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-11T21:18:44.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:43 smithi023 bash[21372]: cluster 2024-03-11T21:18:41.711855+0000 osd.5 (osd.5) 652 : cluster 0 6.19 deep-scrub ok 2024-03-11T21:18:44.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:43 smithi023 bash[21372]: cluster 2024-03-11T21:18:42.111216+0000 osd.2 (osd.2) 297 : cluster 0 1.5 deep-scrub starts 2024-03-11T21:18:44.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:43 smithi023 bash[21372]: cluster 2024-03-11T21:18:42.113931+0000 osd.2 (osd.2) 298 : cluster 0 1.5 deep-scrub ok 2024-03-11T21:18:44.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:43 smithi023 bash[21372]: cluster 2024-03-11T21:18:42.286120+0000 osd.7 (osd.7) 677 : cluster 0 3.f scrub starts 2024-03-11T21:18:44.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:43 smithi023 bash[21372]: cluster 2024-03-11T21:18:42.289246+0000 osd.7 (osd.7) 678 : cluster 0 3.f scrub ok 2024-03-11T21:18:44.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:43 smithi023 bash[21372]: cluster 2024-03-11T21:18:42.549327+0000 mgr.y (mgr.24380) 1029 : cluster 0 pgmap v1199: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T21:18:44.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:43 smithi023 bash[21372]: audit 2024-03-11T21:18:43.270232+0000 mon.c (mon.2) 111 : audit 1 from='client.? 172.21.15.23:0/3829405530' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-11T21:18:44.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:43 smithi023 bash[21372]: audit 2024-03-11T21:18:43.270876+0000 mon.a (mon.0) 2082 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-11T21:18:44.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:43 smithi023 bash[29146]: cluster 2024-03-11T21:18:41.711855+0000 osd.5 (osd.5) 652 : cluster 0 6.19 deep-scrub ok 2024-03-11T21:18:44.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:43 smithi023 bash[29146]: cluster 2024-03-11T21:18:42.111216+0000 osd.2 (osd.2) 297 : cluster 0 1.5 deep-scrub starts 2024-03-11T21:18:44.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:43 smithi023 bash[29146]: cluster 2024-03-11T21:18:42.113931+0000 osd.2 (osd.2) 298 : cluster 0 1.5 deep-scrub ok 2024-03-11T21:18:44.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:43 smithi023 bash[29146]: cluster 2024-03-11T21:18:42.286120+0000 osd.7 (osd.7) 677 : cluster 0 3.f scrub starts 2024-03-11T21:18:44.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:43 smithi023 bash[29146]: cluster 2024-03-11T21:18:42.289246+0000 osd.7 (osd.7) 678 : cluster 0 3.f scrub ok 2024-03-11T21:18:44.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:43 smithi023 bash[29146]: cluster 2024-03-11T21:18:42.549327+0000 mgr.y (mgr.24380) 1029 : cluster 0 pgmap v1199: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T21:18:44.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:43 smithi023 bash[29146]: audit 2024-03-11T21:18:43.270232+0000 mon.c (mon.2) 111 : audit 1 from='client.? 172.21.15.23:0/3829405530' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-11T21:18:44.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:43 smithi023 bash[29146]: audit 2024-03-11T21:18:43.270876+0000 mon.a (mon.0) 2082 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-11T21:18:44.364 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd primary-affinity 4 1 2024-03-11T21:18:44.774 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:44 smithi023 bash[21372]: cluster 2024-03-11T21:18:43.502120+0000 osd.3 (osd.3) 590 : cluster 0 3.b scrub starts 2024-03-11T21:18:44.774 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:44 smithi023 bash[21372]: cluster 2024-03-11T21:18:43.506123+0000 osd.3 (osd.3) 591 : cluster 0 3.b scrub ok 2024-03-11T21:18:44.774 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:44 smithi023 bash[21372]: audit 2024-03-11T21:18:43.527933+0000 mon.a (mon.0) 2083 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-11T21:18:44.774 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:44 smithi023 bash[21372]: cluster 2024-03-11T21:18:43.537467+0000 mon.a (mon.0) 2084 : cluster 0 osdmap e484: 8 total, 8 up, 8 in 2024-03-11T21:18:44.774 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:44 smithi023 bash[29146]: cluster 2024-03-11T21:18:43.502120+0000 osd.3 (osd.3) 590 : cluster 0 3.b scrub starts 2024-03-11T21:18:44.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:44 smithi023 bash[29146]: cluster 2024-03-11T21:18:43.506123+0000 osd.3 (osd.3) 591 : cluster 0 3.b scrub ok 2024-03-11T21:18:44.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:44 smithi023 bash[29146]: audit 2024-03-11T21:18:43.527933+0000 mon.a (mon.0) 2083 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-11T21:18:44.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:44 smithi023 bash[29146]: cluster 2024-03-11T21:18:43.537467+0000 mon.a (mon.0) 2084 : cluster 0 osdmap e484: 8 total, 8 up, 8 in 2024-03-11T21:18:45.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:44 smithi146 bash[23809]: cluster 2024-03-11T21:18:43.502120+0000 osd.3 (osd.3) 590 : cluster 0 3.b scrub starts 2024-03-11T21:18:45.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:44 smithi146 bash[23809]: cluster 2024-03-11T21:18:43.506123+0000 osd.3 (osd.3) 591 : cluster 0 3.b scrub ok 2024-03-11T21:18:45.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:44 smithi146 bash[23809]: audit 2024-03-11T21:18:43.527933+0000 mon.a (mon.0) 2083 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-11T21:18:45.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:44 smithi146 bash[23809]: cluster 2024-03-11T21:18:43.537467+0000 mon.a (mon.0) 2084 : cluster 0 osdmap e484: 8 total, 8 up, 8 in 2024-03-11T21:18:46.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:45 smithi146 bash[23809]: cluster 2024-03-11T21:18:44.527544+0000 osd.3 (osd.3) 592 : cluster 0 3.8 scrub starts 2024-03-11T21:18:46.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:45 smithi146 bash[23809]: cluster 2024-03-11T21:18:44.531819+0000 osd.3 (osd.3) 593 : cluster 0 3.8 scrub ok 2024-03-11T21:18:46.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:45 smithi146 bash[23809]: cluster 2024-03-11T21:18:44.550078+0000 mgr.y (mgr.24380) 1030 : cluster 0 pgmap v1201: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T21:18:46.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:45 smithi146 bash[23809]: audit 2024-03-11T21:18:44.679641+0000 mon.a (mon.0) 2085 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:18:46.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:45 smithi146 bash[23809]: cluster 2024-03-11T21:18:45.050022+0000 osd.1 (osd.1) 416 : cluster 0 3.0 scrub starts 2024-03-11T21:18:46.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:45 smithi146 bash[23809]: cluster 2024-03-11T21:18:45.055847+0000 osd.1 (osd.1) 417 : cluster 0 3.0 scrub ok 2024-03-11T21:18:46.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:45 smithi146 bash[23809]: audit 2024-03-11T21:18:45.421498+0000 mon.a (mon.0) 2086 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:18:46.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:45 smithi146 bash[23809]: audit 2024-03-11T21:18:45.423247+0000 mon.a (mon.0) 2087 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:18:46.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:45 smithi146 bash[23809]: audit 2024-03-11T21:18:45.435323+0000 mon.a (mon.0) 2088 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:18:46.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:45 smithi023 bash[21372]: cluster 2024-03-11T21:18:44.527544+0000 osd.3 (osd.3) 592 : cluster 0 3.8 scrub starts 2024-03-11T21:18:46.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:45 smithi023 bash[21372]: cluster 2024-03-11T21:18:44.531819+0000 osd.3 (osd.3) 593 : cluster 0 3.8 scrub ok 2024-03-11T21:18:46.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:45 smithi023 bash[21372]: cluster 2024-03-11T21:18:44.550078+0000 mgr.y (mgr.24380) 1030 : cluster 0 pgmap v1201: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T21:18:46.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:45 smithi023 bash[21372]: audit 2024-03-11T21:18:44.679641+0000 mon.a (mon.0) 2085 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:18:46.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:45 smithi023 bash[21372]: cluster 2024-03-11T21:18:45.050022+0000 osd.1 (osd.1) 416 : cluster 0 3.0 scrub starts 2024-03-11T21:18:46.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:45 smithi023 bash[21372]: cluster 2024-03-11T21:18:45.055847+0000 osd.1 (osd.1) 417 : cluster 0 3.0 scrub ok 2024-03-11T21:18:46.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:45 smithi023 bash[21372]: audit 2024-03-11T21:18:45.421498+0000 mon.a (mon.0) 2086 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:18:46.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:45 smithi023 bash[21372]: audit 2024-03-11T21:18:45.423247+0000 mon.a (mon.0) 2087 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:18:46.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:45 smithi023 bash[21372]: audit 2024-03-11T21:18:45.435323+0000 mon.a (mon.0) 2088 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:18:46.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:45 smithi023 bash[29146]: cluster 2024-03-11T21:18:44.527544+0000 osd.3 (osd.3) 592 : cluster 0 3.8 scrub starts 2024-03-11T21:18:46.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:45 smithi023 bash[29146]: cluster 2024-03-11T21:18:44.531819+0000 osd.3 (osd.3) 593 : cluster 0 3.8 scrub ok 2024-03-11T21:18:46.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:45 smithi023 bash[29146]: cluster 2024-03-11T21:18:44.550078+0000 mgr.y (mgr.24380) 1030 : cluster 0 pgmap v1201: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T21:18:46.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:45 smithi023 bash[29146]: audit 2024-03-11T21:18:44.679641+0000 mon.a (mon.0) 2085 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:18:46.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:45 smithi023 bash[29146]: cluster 2024-03-11T21:18:45.050022+0000 osd.1 (osd.1) 416 : cluster 0 3.0 scrub starts 2024-03-11T21:18:46.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:45 smithi023 bash[29146]: cluster 2024-03-11T21:18:45.055847+0000 osd.1 (osd.1) 417 : cluster 0 3.0 scrub ok 2024-03-11T21:18:46.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:45 smithi023 bash[29146]: audit 2024-03-11T21:18:45.421498+0000 mon.a (mon.0) 2086 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:18:46.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:45 smithi023 bash[29146]: audit 2024-03-11T21:18:45.423247+0000 mon.a (mon.0) 2087 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:18:46.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:45 smithi023 bash[29146]: audit 2024-03-11T21:18:45.435323+0000 mon.a (mon.0) 2088 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:18:46.752 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:18:46 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:18:48.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:47 smithi146 bash[23809]: audit 2024-03-11T21:18:46.274203+0000 mgr.y (mgr.24380) 1031 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:18:48.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:47 smithi146 bash[23809]: cluster 2024-03-11T21:18:46.533895+0000 osd.3 (osd.3) 594 : cluster 0 4.1a scrub starts 2024-03-11T21:18:48.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:47 smithi146 bash[23809]: cluster 2024-03-11T21:18:46.536332+0000 osd.3 (osd.3) 595 : cluster 0 4.1a scrub ok 2024-03-11T21:18:48.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:47 smithi146 bash[23809]: cluster 2024-03-11T21:18:46.551632+0000 mgr.y (mgr.24380) 1032 : cluster 0 pgmap v1202: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:18:48.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:47 smithi146 bash[23809]: cluster 2024-03-11T21:18:46.563953+0000 osd.5 (osd.5) 653 : cluster 0 3.1d scrub starts 2024-03-11T21:18:48.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:47 smithi146 bash[23809]: cluster 2024-03-11T21:18:46.568167+0000 osd.5 (osd.5) 654 : cluster 0 3.1d scrub ok 2024-03-11T21:18:48.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:47 smithi023 bash[21372]: audit 2024-03-11T21:18:46.274203+0000 mgr.y (mgr.24380) 1031 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:18:48.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:47 smithi023 bash[21372]: cluster 2024-03-11T21:18:46.533895+0000 osd.3 (osd.3) 594 : cluster 0 4.1a scrub starts 2024-03-11T21:18:48.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:47 smithi023 bash[21372]: cluster 2024-03-11T21:18:46.536332+0000 osd.3 (osd.3) 595 : cluster 0 4.1a scrub ok 2024-03-11T21:18:48.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:47 smithi023 bash[21372]: cluster 2024-03-11T21:18:46.551632+0000 mgr.y (mgr.24380) 1032 : cluster 0 pgmap v1202: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:18:48.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:47 smithi023 bash[21372]: cluster 2024-03-11T21:18:46.563953+0000 osd.5 (osd.5) 653 : cluster 0 3.1d scrub starts 2024-03-11T21:18:48.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:47 smithi023 bash[21372]: cluster 2024-03-11T21:18:46.568167+0000 osd.5 (osd.5) 654 : cluster 0 3.1d scrub ok 2024-03-11T21:18:48.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:47 smithi023 bash[29146]: audit 2024-03-11T21:18:46.274203+0000 mgr.y (mgr.24380) 1031 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:18:48.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:47 smithi023 bash[29146]: cluster 2024-03-11T21:18:46.533895+0000 osd.3 (osd.3) 594 : cluster 0 4.1a scrub starts 2024-03-11T21:18:48.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:47 smithi023 bash[29146]: cluster 2024-03-11T21:18:46.536332+0000 osd.3 (osd.3) 595 : cluster 0 4.1a scrub ok 2024-03-11T21:18:48.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:47 smithi023 bash[29146]: cluster 2024-03-11T21:18:46.551632+0000 mgr.y (mgr.24380) 1032 : cluster 0 pgmap v1202: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:18:48.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:47 smithi023 bash[29146]: cluster 2024-03-11T21:18:46.563953+0000 osd.5 (osd.5) 653 : cluster 0 3.1d scrub starts 2024-03-11T21:18:48.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:47 smithi023 bash[29146]: cluster 2024-03-11T21:18:46.568167+0000 osd.5 (osd.5) 654 : cluster 0 3.1d scrub ok 2024-03-11T21:18:48.774 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:18:48 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:18:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:18:48.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:48 smithi023 bash[21372]: cluster 2024-03-11T21:18:48.058604+0000 osd.1 (osd.1) 418 : cluster 0 4.4 scrub starts 2024-03-11T21:18:48.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:48 smithi023 bash[21372]: cluster 2024-03-11T21:18:48.061243+0000 osd.1 (osd.1) 419 : cluster 0 4.4 scrub ok 2024-03-11T21:18:48.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:48 smithi023 bash[29146]: cluster 2024-03-11T21:18:48.058604+0000 osd.1 (osd.1) 418 : cluster 0 4.4 scrub starts 2024-03-11T21:18:48.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:48 smithi023 bash[29146]: cluster 2024-03-11T21:18:48.061243+0000 osd.1 (osd.1) 419 : cluster 0 4.4 scrub ok 2024-03-11T21:18:49.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:48 smithi146 bash[23809]: cluster 2024-03-11T21:18:48.058604+0000 osd.1 (osd.1) 418 : cluster 0 4.4 scrub starts 2024-03-11T21:18:49.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:48 smithi146 bash[23809]: cluster 2024-03-11T21:18:48.061243+0000 osd.1 (osd.1) 419 : cluster 0 4.4 scrub ok 2024-03-11T21:18:49.159 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:18:50.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:49 smithi146 bash[23809]: cluster 2024-03-11T21:18:48.552509+0000 mgr.y (mgr.24380) 1033 : cluster 0 pgmap v1203: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:18:50.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:49 smithi146 bash[23809]: cluster 2024-03-11T21:18:48.575988+0000 osd.3 (osd.3) 596 : cluster 0 3.18 scrub starts 2024-03-11T21:18:50.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:49 smithi146 bash[23809]: cluster 2024-03-11T21:18:48.580458+0000 osd.3 (osd.3) 597 : cluster 0 3.18 scrub ok 2024-03-11T21:18:50.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:49 smithi146 bash[23809]: cluster 2024-03-11T21:18:49.099072+0000 osd.1 (osd.1) 420 : cluster 0 2.a scrub starts 2024-03-11T21:18:50.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:49 smithi146 bash[23809]: cluster 2024-03-11T21:18:49.102037+0000 osd.1 (osd.1) 421 : cluster 0 2.a scrub ok 2024-03-11T21:18:50.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:49 smithi023 bash[21372]: cluster 2024-03-11T21:18:48.552509+0000 mgr.y (mgr.24380) 1033 : cluster 0 pgmap v1203: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:18:50.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:49 smithi023 bash[21372]: cluster 2024-03-11T21:18:48.575988+0000 osd.3 (osd.3) 596 : cluster 0 3.18 scrub starts 2024-03-11T21:18:50.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:49 smithi023 bash[21372]: cluster 2024-03-11T21:18:48.580458+0000 osd.3 (osd.3) 597 : cluster 0 3.18 scrub ok 2024-03-11T21:18:50.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:49 smithi023 bash[21372]: cluster 2024-03-11T21:18:49.099072+0000 osd.1 (osd.1) 420 : cluster 0 2.a scrub starts 2024-03-11T21:18:50.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:49 smithi023 bash[21372]: cluster 2024-03-11T21:18:49.102037+0000 osd.1 (osd.1) 421 : cluster 0 2.a scrub ok 2024-03-11T21:18:50.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:49 smithi023 bash[29146]: cluster 2024-03-11T21:18:48.552509+0000 mgr.y (mgr.24380) 1033 : cluster 0 pgmap v1203: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:18:50.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:49 smithi023 bash[29146]: cluster 2024-03-11T21:18:48.575988+0000 osd.3 (osd.3) 596 : cluster 0 3.18 scrub starts 2024-03-11T21:18:50.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:49 smithi023 bash[29146]: cluster 2024-03-11T21:18:48.580458+0000 osd.3 (osd.3) 597 : cluster 0 3.18 scrub ok 2024-03-11T21:18:50.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:49 smithi023 bash[29146]: cluster 2024-03-11T21:18:49.099072+0000 osd.1 (osd.1) 420 : cluster 0 2.a scrub starts 2024-03-11T21:18:50.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:49 smithi023 bash[29146]: cluster 2024-03-11T21:18:49.102037+0000 osd.1 (osd.1) 421 : cluster 0 2.a scrub ok 2024-03-11T21:18:50.638 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:50 smithi023 bash[21372]: cluster 2024-03-11T21:18:49.546055+0000 osd.3 (osd.3) 598 : cluster 0 4.1b scrub starts 2024-03-11T21:18:50.638 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:50 smithi023 bash[21372]: cluster 2024-03-11T21:18:49.548419+0000 osd.3 (osd.3) 599 : cluster 0 4.1b scrub ok 2024-03-11T21:18:50.638 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:50 smithi023 bash[29146]: cluster 2024-03-11T21:18:49.546055+0000 osd.3 (osd.3) 598 : cluster 0 4.1b scrub starts 2024-03-11T21:18:50.638 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:50 smithi023 bash[29146]: cluster 2024-03-11T21:18:49.548419+0000 osd.3 (osd.3) 599 : cluster 0 4.1b scrub ok 2024-03-11T21:18:51.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:50 smithi146 bash[23809]: cluster 2024-03-11T21:18:49.546055+0000 osd.3 (osd.3) 598 : cluster 0 4.1b scrub starts 2024-03-11T21:18:51.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:50 smithi146 bash[23809]: cluster 2024-03-11T21:18:49.548419+0000 osd.3 (osd.3) 599 : cluster 0 4.1b scrub ok 2024-03-11T21:18:52.000 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:51 smithi023 bash[21372]: cluster 2024-03-11T21:18:50.553158+0000 mgr.y (mgr.24380) 1034 : cluster 0 pgmap v1204: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:18:52.000 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:51 smithi023 bash[29146]: cluster 2024-03-11T21:18:50.553158+0000 mgr.y (mgr.24380) 1034 : cluster 0 pgmap v1204: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:18:52.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:51 smithi146 bash[23809]: cluster 2024-03-11T21:18:50.553158+0000 mgr.y (mgr.24380) 1034 : cluster 0 pgmap v1204: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:18:52.615 INFO:teuthology.orchestra.run.smithi023.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-03-11T21:18:53.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:52 smithi146 bash[23809]: cluster 2024-03-11T21:18:51.251723+0000 osd.0 (osd.0) 492 : cluster 0 3.6 scrub starts 2024-03-11T21:18:53.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:52 smithi146 bash[23809]: cluster 2024-03-11T21:18:51.254686+0000 osd.0 (osd.0) 493 : cluster 0 3.6 scrub ok 2024-03-11T21:18:53.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:52 smithi146 bash[23809]: cluster 2024-03-11T21:18:51.535747+0000 osd.3 (osd.3) 600 : cluster 0 5.0 scrub starts 2024-03-11T21:18:53.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:52 smithi146 bash[23809]: cluster 2024-03-11T21:18:51.539385+0000 osd.3 (osd.3) 601 : cluster 0 5.0 scrub ok 2024-03-11T21:18:53.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:52 smithi146 bash[23809]: audit 2024-03-11T21:18:51.993626+0000 mon.a (mon.0) 2089 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:18:53.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:52 smithi146 bash[23809]: audit 2024-03-11T21:18:52.413981+0000 mon.c (mon.2) 112 : audit 1 from='client.? 172.21.15.23:0/2490870598' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-03-11T21:18:53.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:52 smithi146 bash[23809]: audit 2024-03-11T21:18:52.414548+0000 mon.a (mon.0) 2090 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-03-11T21:18:53.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:52 smithi146 bash[23809]: cluster 2024-03-11T21:18:52.487738+0000 osd.6 (osd.6) 540 : cluster 0 2.11 scrub starts 2024-03-11T21:18:53.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:52 smithi146 bash[23809]: cluster 2024-03-11T21:18:52.490944+0000 osd.6 (osd.6) 541 : cluster 0 2.11 scrub ok 2024-03-11T21:18:53.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:52 smithi023 bash[21372]: cluster 2024-03-11T21:18:51.251723+0000 osd.0 (osd.0) 492 : cluster 0 3.6 scrub starts 2024-03-11T21:18:53.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:52 smithi023 bash[21372]: cluster 2024-03-11T21:18:51.254686+0000 osd.0 (osd.0) 493 : cluster 0 3.6 scrub ok 2024-03-11T21:18:53.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:52 smithi023 bash[21372]: cluster 2024-03-11T21:18:51.535747+0000 osd.3 (osd.3) 600 : cluster 0 5.0 scrub starts 2024-03-11T21:18:53.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:52 smithi023 bash[21372]: cluster 2024-03-11T21:18:51.539385+0000 osd.3 (osd.3) 601 : cluster 0 5.0 scrub ok 2024-03-11T21:18:53.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:52 smithi023 bash[21372]: audit 2024-03-11T21:18:51.993626+0000 mon.a (mon.0) 2089 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:18:53.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:52 smithi023 bash[21372]: audit 2024-03-11T21:18:52.413981+0000 mon.c (mon.2) 112 : audit 1 from='client.? 172.21.15.23:0/2490870598' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-03-11T21:18:53.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:52 smithi023 bash[21372]: audit 2024-03-11T21:18:52.414548+0000 mon.a (mon.0) 2090 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-03-11T21:18:53.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:52 smithi023 bash[21372]: cluster 2024-03-11T21:18:52.487738+0000 osd.6 (osd.6) 540 : cluster 0 2.11 scrub starts 2024-03-11T21:18:53.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:52 smithi023 bash[21372]: cluster 2024-03-11T21:18:52.490944+0000 osd.6 (osd.6) 541 : cluster 0 2.11 scrub ok 2024-03-11T21:18:53.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:52 smithi023 bash[29146]: cluster 2024-03-11T21:18:51.251723+0000 osd.0 (osd.0) 492 : cluster 0 3.6 scrub starts 2024-03-11T21:18:53.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:52 smithi023 bash[29146]: cluster 2024-03-11T21:18:51.254686+0000 osd.0 (osd.0) 493 : cluster 0 3.6 scrub ok 2024-03-11T21:18:53.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:52 smithi023 bash[29146]: cluster 2024-03-11T21:18:51.535747+0000 osd.3 (osd.3) 600 : cluster 0 5.0 scrub starts 2024-03-11T21:18:53.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:52 smithi023 bash[29146]: cluster 2024-03-11T21:18:51.539385+0000 osd.3 (osd.3) 601 : cluster 0 5.0 scrub ok 2024-03-11T21:18:53.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:52 smithi023 bash[29146]: audit 2024-03-11T21:18:51.993626+0000 mon.a (mon.0) 2089 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:18:53.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:52 smithi023 bash[29146]: audit 2024-03-11T21:18:52.413981+0000 mon.c (mon.2) 112 : audit 1 from='client.? 172.21.15.23:0/2490870598' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-03-11T21:18:53.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:52 smithi023 bash[29146]: audit 2024-03-11T21:18:52.414548+0000 mon.a (mon.0) 2090 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-03-11T21:18:53.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:52 smithi023 bash[29146]: cluster 2024-03-11T21:18:52.487738+0000 osd.6 (osd.6) 540 : cluster 0 2.11 scrub starts 2024-03-11T21:18:53.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:52 smithi023 bash[29146]: cluster 2024-03-11T21:18:52.490944+0000 osd.6 (osd.6) 541 : cluster 0 2.11 scrub ok 2024-03-11T21:18:53.450 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd reweight 1 1 2024-03-11T21:18:53.774 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:53 smithi023 bash[21372]: cluster 2024-03-11T21:18:52.515343+0000 osd.5 (osd.5) 655 : cluster 0 1.a scrub starts 2024-03-11T21:18:53.774 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:53 smithi023 bash[21372]: cluster 2024-03-11T21:18:52.517865+0000 osd.5 (osd.5) 656 : cluster 0 1.a scrub ok 2024-03-11T21:18:53.774 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:53 smithi023 bash[21372]: cluster 2024-03-11T21:18:52.554288+0000 mgr.y (mgr.24380) 1035 : cluster 0 pgmap v1205: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:18:53.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:53 smithi023 bash[21372]: cluster 2024-03-11T21:18:52.558249+0000 osd.3 (osd.3) 602 : cluster 0 6.1f deep-scrub starts 2024-03-11T21:18:53.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:53 smithi023 bash[21372]: audit 2024-03-11T21:18:52.600342+0000 mon.a (mon.0) 2091 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-03-11T21:18:53.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:53 smithi023 bash[21372]: cluster 2024-03-11T21:18:52.608314+0000 mon.a (mon.0) 2092 : cluster 0 osdmap e485: 8 total, 8 up, 8 in 2024-03-11T21:18:53.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:53 smithi023 bash[21372]: cluster 2024-03-11T21:18:52.792318+0000 osd.3 (osd.3) 603 : cluster 0 6.1f deep-scrub ok 2024-03-11T21:18:53.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:53 smithi023 bash[21372]: cluster 2024-03-11T21:18:53.471748+0000 osd.6 (osd.6) 542 : cluster 0 purged_snaps scrub starts 2024-03-11T21:18:53.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:53 smithi023 bash[21372]: cluster 2024-03-11T21:18:53.471889+0000 osd.6 (osd.6) 543 : cluster 0 purged_snaps scrub ok 2024-03-11T21:18:53.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:53 smithi023 bash[29146]: cluster 2024-03-11T21:18:52.515343+0000 osd.5 (osd.5) 655 : cluster 0 1.a scrub starts 2024-03-11T21:18:53.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:53 smithi023 bash[29146]: cluster 2024-03-11T21:18:52.517865+0000 osd.5 (osd.5) 656 : cluster 0 1.a scrub ok 2024-03-11T21:18:53.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:53 smithi023 bash[29146]: cluster 2024-03-11T21:18:52.554288+0000 mgr.y (mgr.24380) 1035 : cluster 0 pgmap v1205: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:18:53.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:53 smithi023 bash[29146]: cluster 2024-03-11T21:18:52.558249+0000 osd.3 (osd.3) 602 : cluster 0 6.1f deep-scrub starts 2024-03-11T21:18:53.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:53 smithi023 bash[29146]: audit 2024-03-11T21:18:52.600342+0000 mon.a (mon.0) 2091 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-03-11T21:18:53.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:53 smithi023 bash[29146]: cluster 2024-03-11T21:18:52.608314+0000 mon.a (mon.0) 2092 : cluster 0 osdmap e485: 8 total, 8 up, 8 in 2024-03-11T21:18:53.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:53 smithi023 bash[29146]: cluster 2024-03-11T21:18:52.792318+0000 osd.3 (osd.3) 603 : cluster 0 6.1f deep-scrub ok 2024-03-11T21:18:53.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:53 smithi023 bash[29146]: cluster 2024-03-11T21:18:53.471748+0000 osd.6 (osd.6) 542 : cluster 0 purged_snaps scrub starts 2024-03-11T21:18:53.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:53 smithi023 bash[29146]: cluster 2024-03-11T21:18:53.471889+0000 osd.6 (osd.6) 543 : cluster 0 purged_snaps scrub ok 2024-03-11T21:18:54.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:53 smithi146 bash[23809]: cluster 2024-03-11T21:18:52.515343+0000 osd.5 (osd.5) 655 : cluster 0 1.a scrub starts 2024-03-11T21:18:54.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:53 smithi146 bash[23809]: cluster 2024-03-11T21:18:52.517865+0000 osd.5 (osd.5) 656 : cluster 0 1.a scrub ok 2024-03-11T21:18:54.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:53 smithi146 bash[23809]: cluster 2024-03-11T21:18:52.554288+0000 mgr.y (mgr.24380) 1035 : cluster 0 pgmap v1205: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:18:54.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:53 smithi146 bash[23809]: cluster 2024-03-11T21:18:52.558249+0000 osd.3 (osd.3) 602 : cluster 0 6.1f deep-scrub starts 2024-03-11T21:18:54.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:53 smithi146 bash[23809]: audit 2024-03-11T21:18:52.600342+0000 mon.a (mon.0) 2091 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-03-11T21:18:54.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:53 smithi146 bash[23809]: cluster 2024-03-11T21:18:52.608314+0000 mon.a (mon.0) 2092 : cluster 0 osdmap e485: 8 total, 8 up, 8 in 2024-03-11T21:18:54.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:53 smithi146 bash[23809]: cluster 2024-03-11T21:18:52.792318+0000 osd.3 (osd.3) 603 : cluster 0 6.1f deep-scrub ok 2024-03-11T21:18:54.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:53 smithi146 bash[23809]: cluster 2024-03-11T21:18:53.471748+0000 osd.6 (osd.6) 542 : cluster 0 purged_snaps scrub starts 2024-03-11T21:18:54.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:53 smithi146 bash[23809]: cluster 2024-03-11T21:18:53.471889+0000 osd.6 (osd.6) 543 : cluster 0 purged_snaps scrub ok 2024-03-11T21:18:55.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:54 smithi146 bash[23809]: cluster 2024-03-11T21:18:53.639057+0000 mon.a (mon.0) 2093 : cluster 0 osdmap e486: 8 total, 8 up, 8 in 2024-03-11T21:18:55.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:54 smithi146 bash[23809]: cluster 2024-03-11T21:18:54.176266+0000 osd.3 (osd.3) 604 : cluster 0 purged_snaps scrub starts 2024-03-11T21:18:55.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:54 smithi146 bash[23809]: cluster 2024-03-11T21:18:54.176375+0000 osd.3 (osd.3) 605 : cluster 0 purged_snaps scrub ok 2024-03-11T21:18:55.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:54 smithi146 bash[23809]: cluster 2024-03-11T21:18:54.448759+0000 osd.5 (osd.5) 657 : cluster 0 4.15 deep-scrub starts 2024-03-11T21:18:55.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:54 smithi146 bash[23809]: cluster 2024-03-11T21:18:54.451787+0000 osd.5 (osd.5) 658 : cluster 0 4.15 deep-scrub ok 2024-03-11T21:18:55.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:54 smithi023 bash[21372]: cluster 2024-03-11T21:18:53.639057+0000 mon.a (mon.0) 2093 : cluster 0 osdmap e486: 8 total, 8 up, 8 in 2024-03-11T21:18:55.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:54 smithi023 bash[21372]: cluster 2024-03-11T21:18:54.176266+0000 osd.3 (osd.3) 604 : cluster 0 purged_snaps scrub starts 2024-03-11T21:18:55.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:54 smithi023 bash[21372]: cluster 2024-03-11T21:18:54.176375+0000 osd.3 (osd.3) 605 : cluster 0 purged_snaps scrub ok 2024-03-11T21:18:55.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:54 smithi023 bash[21372]: cluster 2024-03-11T21:18:54.448759+0000 osd.5 (osd.5) 657 : cluster 0 4.15 deep-scrub starts 2024-03-11T21:18:55.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:54 smithi023 bash[21372]: cluster 2024-03-11T21:18:54.451787+0000 osd.5 (osd.5) 658 : cluster 0 4.15 deep-scrub ok 2024-03-11T21:18:55.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:54 smithi023 bash[29146]: cluster 2024-03-11T21:18:53.639057+0000 mon.a (mon.0) 2093 : cluster 0 osdmap e486: 8 total, 8 up, 8 in 2024-03-11T21:18:55.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:54 smithi023 bash[29146]: cluster 2024-03-11T21:18:54.176266+0000 osd.3 (osd.3) 604 : cluster 0 purged_snaps scrub starts 2024-03-11T21:18:55.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:54 smithi023 bash[29146]: cluster 2024-03-11T21:18:54.176375+0000 osd.3 (osd.3) 605 : cluster 0 purged_snaps scrub ok 2024-03-11T21:18:55.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:54 smithi023 bash[29146]: cluster 2024-03-11T21:18:54.448759+0000 osd.5 (osd.5) 657 : cluster 0 4.15 deep-scrub starts 2024-03-11T21:18:55.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:54 smithi023 bash[29146]: cluster 2024-03-11T21:18:54.451787+0000 osd.5 (osd.5) 658 : cluster 0 4.15 deep-scrub ok 2024-03-11T21:18:56.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:55 smithi146 bash[23809]: cluster 2024-03-11T21:18:54.248465+0000 osd.4 (osd.4) 125 : cluster 0 1.e scrub starts 2024-03-11T21:18:56.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:55 smithi146 bash[23809]: cluster 2024-03-11T21:18:54.251233+0000 osd.4 (osd.4) 126 : cluster 0 1.e scrub ok 2024-03-11T21:18:56.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:55 smithi146 bash[23809]: cluster 2024-03-11T21:18:54.275384+0000 osd.7 (osd.7) 679 : cluster 0 1.d scrub starts 2024-03-11T21:18:56.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:55 smithi146 bash[23809]: cluster 2024-03-11T21:18:54.278078+0000 osd.7 (osd.7) 680 : cluster 0 1.d scrub ok 2024-03-11T21:18:56.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:55 smithi146 bash[23809]: cluster 2024-03-11T21:18:54.532908+0000 osd.6 (osd.6) 544 : cluster 0 4.1f scrub starts 2024-03-11T21:18:56.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:55 smithi146 bash[23809]: cluster 2024-03-11T21:18:54.536229+0000 osd.6 (osd.6) 545 : cluster 0 4.1f scrub ok 2024-03-11T21:18:56.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:55 smithi146 bash[23809]: cluster 2024-03-11T21:18:54.554933+0000 mgr.y (mgr.24380) 1036 : cluster 0 pgmap v1208: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:18:56.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:55 smithi146 bash[23809]: cluster 2024-03-11T21:18:54.666982+0000 mon.a (mon.0) 2094 : cluster 0 osdmap e487: 8 total, 8 up, 8 in 2024-03-11T21:18:56.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:55 smithi023 bash[21372]: cluster 2024-03-11T21:18:54.248465+0000 osd.4 (osd.4) 125 : cluster 0 1.e scrub starts 2024-03-11T21:18:56.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:55 smithi023 bash[21372]: cluster 2024-03-11T21:18:54.251233+0000 osd.4 (osd.4) 126 : cluster 0 1.e scrub ok 2024-03-11T21:18:56.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:55 smithi023 bash[21372]: cluster 2024-03-11T21:18:54.275384+0000 osd.7 (osd.7) 679 : cluster 0 1.d scrub starts 2024-03-11T21:18:56.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:55 smithi023 bash[21372]: cluster 2024-03-11T21:18:54.278078+0000 osd.7 (osd.7) 680 : cluster 0 1.d scrub ok 2024-03-11T21:18:56.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:55 smithi023 bash[21372]: cluster 2024-03-11T21:18:54.532908+0000 osd.6 (osd.6) 544 : cluster 0 4.1f scrub starts 2024-03-11T21:18:56.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:55 smithi023 bash[21372]: cluster 2024-03-11T21:18:54.536229+0000 osd.6 (osd.6) 545 : cluster 0 4.1f scrub ok 2024-03-11T21:18:56.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:55 smithi023 bash[21372]: cluster 2024-03-11T21:18:54.554933+0000 mgr.y (mgr.24380) 1036 : cluster 0 pgmap v1208: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:18:56.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:55 smithi023 bash[21372]: cluster 2024-03-11T21:18:54.666982+0000 mon.a (mon.0) 2094 : cluster 0 osdmap e487: 8 total, 8 up, 8 in 2024-03-11T21:18:56.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:55 smithi023 bash[29146]: cluster 2024-03-11T21:18:54.248465+0000 osd.4 (osd.4) 125 : cluster 0 1.e scrub starts 2024-03-11T21:18:56.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:55 smithi023 bash[29146]: cluster 2024-03-11T21:18:54.251233+0000 osd.4 (osd.4) 126 : cluster 0 1.e scrub ok 2024-03-11T21:18:56.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:55 smithi023 bash[29146]: cluster 2024-03-11T21:18:54.275384+0000 osd.7 (osd.7) 679 : cluster 0 1.d scrub starts 2024-03-11T21:18:56.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:55 smithi023 bash[29146]: cluster 2024-03-11T21:18:54.278078+0000 osd.7 (osd.7) 680 : cluster 0 1.d scrub ok 2024-03-11T21:18:56.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:55 smithi023 bash[29146]: cluster 2024-03-11T21:18:54.532908+0000 osd.6 (osd.6) 544 : cluster 0 4.1f scrub starts 2024-03-11T21:18:56.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:55 smithi023 bash[29146]: cluster 2024-03-11T21:18:54.536229+0000 osd.6 (osd.6) 545 : cluster 0 4.1f scrub ok 2024-03-11T21:18:56.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:55 smithi023 bash[29146]: cluster 2024-03-11T21:18:54.554933+0000 mgr.y (mgr.24380) 1036 : cluster 0 pgmap v1208: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:18:56.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:55 smithi023 bash[29146]: cluster 2024-03-11T21:18:54.666982+0000 mon.a (mon.0) 2094 : cluster 0 osdmap e487: 8 total, 8 up, 8 in 2024-03-11T21:18:56.654 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:18:56 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:18:57.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:56 smithi146 bash[23809]: cluster 2024-03-11T21:18:55.209173+0000 osd.4 (osd.4) 127 : cluster 0 1.5 scrub starts 2024-03-11T21:18:57.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:56 smithi146 bash[23809]: cluster 2024-03-11T21:18:55.211576+0000 osd.4 (osd.4) 128 : cluster 0 1.5 scrub ok 2024-03-11T21:18:57.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:56 smithi146 bash[23809]: cluster 2024-03-11T21:18:55.348216+0000 osd.2 (osd.2) 299 : cluster 0 2.e scrub starts 2024-03-11T21:18:57.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:56 smithi146 bash[23809]: cluster 2024-03-11T21:18:55.350569+0000 osd.2 (osd.2) 300 : cluster 0 2.e scrub ok 2024-03-11T21:18:57.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:56 smithi023 bash[21372]: cluster 2024-03-11T21:18:55.209173+0000 osd.4 (osd.4) 127 : cluster 0 1.5 scrub starts 2024-03-11T21:18:57.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:56 smithi023 bash[21372]: cluster 2024-03-11T21:18:55.211576+0000 osd.4 (osd.4) 128 : cluster 0 1.5 scrub ok 2024-03-11T21:18:57.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:56 smithi023 bash[21372]: cluster 2024-03-11T21:18:55.348216+0000 osd.2 (osd.2) 299 : cluster 0 2.e scrub starts 2024-03-11T21:18:57.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:56 smithi023 bash[21372]: cluster 2024-03-11T21:18:55.350569+0000 osd.2 (osd.2) 300 : cluster 0 2.e scrub ok 2024-03-11T21:18:57.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:56 smithi023 bash[29146]: cluster 2024-03-11T21:18:55.209173+0000 osd.4 (osd.4) 127 : cluster 0 1.5 scrub starts 2024-03-11T21:18:57.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:56 smithi023 bash[29146]: cluster 2024-03-11T21:18:55.211576+0000 osd.4 (osd.4) 128 : cluster 0 1.5 scrub ok 2024-03-11T21:18:57.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:56 smithi023 bash[29146]: cluster 2024-03-11T21:18:55.348216+0000 osd.2 (osd.2) 299 : cluster 0 2.e scrub starts 2024-03-11T21:18:57.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:56 smithi023 bash[29146]: cluster 2024-03-11T21:18:55.350569+0000 osd.2 (osd.2) 300 : cluster 0 2.e scrub ok 2024-03-11T21:18:58.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:57 smithi146 bash[23809]: cluster 2024-03-11T21:18:56.177217+0000 osd.4 (osd.4) 129 : cluster 0 6.2 scrub starts 2024-03-11T21:18:58.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:57 smithi146 bash[23809]: cluster 2024-03-11T21:18:56.179749+0000 osd.4 (osd.4) 130 : cluster 0 6.2 scrub ok 2024-03-11T21:18:58.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:57 smithi146 bash[23809]: audit 2024-03-11T21:18:56.285579+0000 mgr.y (mgr.24380) 1037 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:18:58.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:57 smithi146 bash[23809]: cluster 2024-03-11T21:18:56.499229+0000 osd.3 (osd.3) 606 : cluster 0 4.17 scrub starts 2024-03-11T21:18:58.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:57 smithi146 bash[23809]: cluster 2024-03-11T21:18:56.501609+0000 osd.3 (osd.3) 607 : cluster 0 4.17 scrub ok 2024-03-11T21:18:58.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:57 smithi146 bash[23809]: cluster 2024-03-11T21:18:56.556428+0000 mgr.y (mgr.24380) 1038 : cluster 0 pgmap v1210: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-11T21:18:58.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:57 smithi146 bash[23809]: cluster 2024-03-11T21:18:57.116967+0000 osd.1 (osd.1) 422 : cluster 0 1.3 scrub starts 2024-03-11T21:18:58.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:57 smithi146 bash[23809]: cluster 2024-03-11T21:18:57.119629+0000 osd.1 (osd.1) 423 : cluster 0 1.3 scrub ok 2024-03-11T21:18:58.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:57 smithi146 bash[23809]: cluster 2024-03-11T21:18:57.354928+0000 osd.5 (osd.5) 659 : cluster 0 1.10 scrub starts 2024-03-11T21:18:58.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:57 smithi146 bash[23809]: cluster 2024-03-11T21:18:57.357448+0000 osd.5 (osd.5) 660 : cluster 0 1.10 scrub ok 2024-03-11T21:18:58.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:57 smithi023 bash[21372]: cluster 2024-03-11T21:18:56.177217+0000 osd.4 (osd.4) 129 : cluster 0 6.2 scrub starts 2024-03-11T21:18:58.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:57 smithi023 bash[21372]: cluster 2024-03-11T21:18:56.179749+0000 osd.4 (osd.4) 130 : cluster 0 6.2 scrub ok 2024-03-11T21:18:58.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:57 smithi023 bash[21372]: audit 2024-03-11T21:18:56.285579+0000 mgr.y (mgr.24380) 1037 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:18:58.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:57 smithi023 bash[21372]: cluster 2024-03-11T21:18:56.499229+0000 osd.3 (osd.3) 606 : cluster 0 4.17 scrub starts 2024-03-11T21:18:58.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:57 smithi023 bash[21372]: cluster 2024-03-11T21:18:56.501609+0000 osd.3 (osd.3) 607 : cluster 0 4.17 scrub ok 2024-03-11T21:18:58.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:57 smithi023 bash[21372]: cluster 2024-03-11T21:18:56.556428+0000 mgr.y (mgr.24380) 1038 : cluster 0 pgmap v1210: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-11T21:18:58.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:57 smithi023 bash[21372]: cluster 2024-03-11T21:18:57.116967+0000 osd.1 (osd.1) 422 : cluster 0 1.3 scrub starts 2024-03-11T21:18:58.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:57 smithi023 bash[21372]: cluster 2024-03-11T21:18:57.119629+0000 osd.1 (osd.1) 423 : cluster 0 1.3 scrub ok 2024-03-11T21:18:58.026 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:57 smithi023 bash[21372]: cluster 2024-03-11T21:18:57.354928+0000 osd.5 (osd.5) 659 : cluster 0 1.10 scrub starts 2024-03-11T21:18:58.026 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:57 smithi023 bash[21372]: cluster 2024-03-11T21:18:57.357448+0000 osd.5 (osd.5) 660 : cluster 0 1.10 scrub ok 2024-03-11T21:18:58.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:57 smithi023 bash[29146]: cluster 2024-03-11T21:18:56.177217+0000 osd.4 (osd.4) 129 : cluster 0 6.2 scrub starts 2024-03-11T21:18:58.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:57 smithi023 bash[29146]: cluster 2024-03-11T21:18:56.179749+0000 osd.4 (osd.4) 130 : cluster 0 6.2 scrub ok 2024-03-11T21:18:58.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:57 smithi023 bash[29146]: audit 2024-03-11T21:18:56.285579+0000 mgr.y (mgr.24380) 1037 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:18:58.027 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:57 smithi023 bash[29146]: cluster 2024-03-11T21:18:56.499229+0000 osd.3 (osd.3) 606 : cluster 0 4.17 scrub starts 2024-03-11T21:18:58.027 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:57 smithi023 bash[29146]: cluster 2024-03-11T21:18:56.501609+0000 osd.3 (osd.3) 607 : cluster 0 4.17 scrub ok 2024-03-11T21:18:58.027 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:57 smithi023 bash[29146]: cluster 2024-03-11T21:18:56.556428+0000 mgr.y (mgr.24380) 1038 : cluster 0 pgmap v1210: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-11T21:18:58.027 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:57 smithi023 bash[29146]: cluster 2024-03-11T21:18:57.116967+0000 osd.1 (osd.1) 422 : cluster 0 1.3 scrub starts 2024-03-11T21:18:58.027 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:57 smithi023 bash[29146]: cluster 2024-03-11T21:18:57.119629+0000 osd.1 (osd.1) 423 : cluster 0 1.3 scrub ok 2024-03-11T21:18:58.027 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:57 smithi023 bash[29146]: cluster 2024-03-11T21:18:57.354928+0000 osd.5 (osd.5) 659 : cluster 0 1.10 scrub starts 2024-03-11T21:18:58.027 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:57 smithi023 bash[29146]: cluster 2024-03-11T21:18:57.357448+0000 osd.5 (osd.5) 660 : cluster 0 1.10 scrub ok 2024-03-11T21:18:58.262 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:18:58.693 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:18:58 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:18:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:18:59.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:58 smithi146 bash[23809]: cluster 2024-03-11T21:18:57.024468+0000 osd.2 (osd.2) 301 : cluster 0 purged_snaps scrub starts 2024-03-11T21:18:59.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:58 smithi146 bash[23809]: cluster 2024-03-11T21:18:57.024585+0000 osd.2 (osd.2) 302 : cluster 0 purged_snaps scrub ok 2024-03-11T21:18:59.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:58 smithi146 bash[23809]: cluster 2024-03-11T21:18:57.300982+0000 osd.7 (osd.7) 681 : cluster 0 6.12 scrub starts 2024-03-11T21:18:59.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:58 smithi146 bash[23809]: cluster 2024-03-11T21:18:57.346008+0000 osd.7 (osd.7) 682 : cluster 0 6.12 scrub ok 2024-03-11T21:18:59.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:58 smithi146 bash[23809]: cluster 2024-03-11T21:18:57.676386+0000 mon.a (mon.0) 2095 : cluster 0 osdmap e488: 8 total, 8 up, 8 in 2024-03-11T21:18:59.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:58 smithi146 bash[23809]: cluster 2024-03-11T21:18:58.321056+0000 osd.0 (osd.0) 494 : cluster 0 6.18 scrub starts 2024-03-11T21:18:59.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:58 smithi146 bash[23809]: cluster 2024-03-11T21:18:58.381842+0000 osd.0 (osd.0) 495 : cluster 0 6.18 scrub ok 2024-03-11T21:18:59.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:58 smithi023 bash[21372]: cluster 2024-03-11T21:18:57.024468+0000 osd.2 (osd.2) 301 : cluster 0 purged_snaps scrub starts 2024-03-11T21:18:59.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:58 smithi023 bash[21372]: cluster 2024-03-11T21:18:57.024585+0000 osd.2 (osd.2) 302 : cluster 0 purged_snaps scrub ok 2024-03-11T21:18:59.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:58 smithi023 bash[21372]: cluster 2024-03-11T21:18:57.300982+0000 osd.7 (osd.7) 681 : cluster 0 6.12 scrub starts 2024-03-11T21:18:59.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:58 smithi023 bash[21372]: cluster 2024-03-11T21:18:57.346008+0000 osd.7 (osd.7) 682 : cluster 0 6.12 scrub ok 2024-03-11T21:18:59.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:58 smithi023 bash[21372]: cluster 2024-03-11T21:18:57.676386+0000 mon.a (mon.0) 2095 : cluster 0 osdmap e488: 8 total, 8 up, 8 in 2024-03-11T21:18:59.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:58 smithi023 bash[21372]: cluster 2024-03-11T21:18:58.321056+0000 osd.0 (osd.0) 494 : cluster 0 6.18 scrub starts 2024-03-11T21:18:59.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:58 smithi023 bash[21372]: cluster 2024-03-11T21:18:58.381842+0000 osd.0 (osd.0) 495 : cluster 0 6.18 scrub ok 2024-03-11T21:18:59.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:58 smithi023 bash[29146]: cluster 2024-03-11T21:18:57.024468+0000 osd.2 (osd.2) 301 : cluster 0 purged_snaps scrub starts 2024-03-11T21:18:59.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:58 smithi023 bash[29146]: cluster 2024-03-11T21:18:57.024585+0000 osd.2 (osd.2) 302 : cluster 0 purged_snaps scrub ok 2024-03-11T21:18:59.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:58 smithi023 bash[29146]: cluster 2024-03-11T21:18:57.300982+0000 osd.7 (osd.7) 681 : cluster 0 6.12 scrub starts 2024-03-11T21:18:59.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:58 smithi023 bash[29146]: cluster 2024-03-11T21:18:57.346008+0000 osd.7 (osd.7) 682 : cluster 0 6.12 scrub ok 2024-03-11T21:18:59.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:58 smithi023 bash[29146]: cluster 2024-03-11T21:18:57.676386+0000 mon.a (mon.0) 2095 : cluster 0 osdmap e488: 8 total, 8 up, 8 in 2024-03-11T21:18:59.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:58 smithi023 bash[29146]: cluster 2024-03-11T21:18:58.321056+0000 osd.0 (osd.0) 494 : cluster 0 6.18 scrub starts 2024-03-11T21:18:59.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:58 smithi023 bash[29146]: cluster 2024-03-11T21:18:58.381842+0000 osd.0 (osd.0) 495 : cluster 0 6.18 scrub ok 2024-03-11T21:19:00.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:59 smithi146 bash[23809]: cluster 2024-03-11T21:18:58.141619+0000 osd.4 (osd.4) 131 : cluster 0 5.1 scrub starts 2024-03-11T21:19:00.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:59 smithi146 bash[23809]: cluster 2024-03-11T21:18:58.145229+0000 osd.4 (osd.4) 132 : cluster 0 5.1 scrub ok 2024-03-11T21:19:00.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:59 smithi146 bash[23809]: cluster 2024-03-11T21:18:58.557252+0000 mgr.y (mgr.24380) 1039 : cluster 0 pgmap v1212: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 515 B/s rd, 0 op/s 2024-03-11T21:19:00.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:59 smithi146 bash[23809]: cluster 2024-03-11T21:18:59.320826+0000 osd.0 (osd.0) 496 : cluster 0 2.13 scrub starts 2024-03-11T21:19:00.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:18:59 smithi146 bash[23809]: cluster 2024-03-11T21:18:59.323076+0000 osd.0 (osd.0) 497 : cluster 0 2.13 scrub ok 2024-03-11T21:19:00.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:59 smithi023 bash[21372]: cluster 2024-03-11T21:18:58.141619+0000 osd.4 (osd.4) 131 : cluster 0 5.1 scrub starts 2024-03-11T21:19:00.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:59 smithi023 bash[21372]: cluster 2024-03-11T21:18:58.145229+0000 osd.4 (osd.4) 132 : cluster 0 5.1 scrub ok 2024-03-11T21:19:00.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:59 smithi023 bash[21372]: cluster 2024-03-11T21:18:58.557252+0000 mgr.y (mgr.24380) 1039 : cluster 0 pgmap v1212: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 515 B/s rd, 0 op/s 2024-03-11T21:19:00.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:59 smithi023 bash[21372]: cluster 2024-03-11T21:18:59.320826+0000 osd.0 (osd.0) 496 : cluster 0 2.13 scrub starts 2024-03-11T21:19:00.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:18:59 smithi023 bash[21372]: cluster 2024-03-11T21:18:59.323076+0000 osd.0 (osd.0) 497 : cluster 0 2.13 scrub ok 2024-03-11T21:19:00.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:59 smithi023 bash[29146]: cluster 2024-03-11T21:18:58.141619+0000 osd.4 (osd.4) 131 : cluster 0 5.1 scrub starts 2024-03-11T21:19:00.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:59 smithi023 bash[29146]: cluster 2024-03-11T21:18:58.145229+0000 osd.4 (osd.4) 132 : cluster 0 5.1 scrub ok 2024-03-11T21:19:00.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:59 smithi023 bash[29146]: cluster 2024-03-11T21:18:58.557252+0000 mgr.y (mgr.24380) 1039 : cluster 0 pgmap v1212: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 515 B/s rd, 0 op/s 2024-03-11T21:19:00.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:59 smithi023 bash[29146]: cluster 2024-03-11T21:18:59.320826+0000 osd.0 (osd.0) 496 : cluster 0 2.13 scrub starts 2024-03-11T21:19:00.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:18:59 smithi023 bash[29146]: cluster 2024-03-11T21:18:59.323076+0000 osd.0 (osd.0) 497 : cluster 0 2.13 scrub ok 2024-03-11T21:19:01.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:00 smithi146 bash[23809]: cluster 2024-03-11T21:18:59.320024+0000 osd.7 (osd.7) 683 : cluster 0 6.15 scrub starts 2024-03-11T21:19:01.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:00 smithi146 bash[23809]: cluster 2024-03-11T21:18:59.409397+0000 osd.7 (osd.7) 684 : cluster 0 6.15 scrub ok 2024-03-11T21:19:01.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:00 smithi146 bash[23809]: cluster 2024-03-11T21:19:00.122750+0000 osd.4 (osd.4) 133 : cluster 0 2.f scrub starts 2024-03-11T21:19:01.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:00 smithi023 bash[21372]: cluster 2024-03-11T21:18:59.320024+0000 osd.7 (osd.7) 683 : cluster 0 6.15 scrub starts 2024-03-11T21:19:01.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:00 smithi023 bash[21372]: cluster 2024-03-11T21:18:59.409397+0000 osd.7 (osd.7) 684 : cluster 0 6.15 scrub ok 2024-03-11T21:19:01.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:00 smithi023 bash[21372]: cluster 2024-03-11T21:19:00.122750+0000 osd.4 (osd.4) 133 : cluster 0 2.f scrub starts 2024-03-11T21:19:01.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:00 smithi023 bash[29146]: cluster 2024-03-11T21:18:59.320024+0000 osd.7 (osd.7) 683 : cluster 0 6.15 scrub starts 2024-03-11T21:19:01.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:00 smithi023 bash[29146]: cluster 2024-03-11T21:18:59.409397+0000 osd.7 (osd.7) 684 : cluster 0 6.15 scrub ok 2024-03-11T21:19:01.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:00 smithi023 bash[29146]: cluster 2024-03-11T21:19:00.122750+0000 osd.4 (osd.4) 133 : cluster 0 2.f scrub starts 2024-03-11T21:19:02.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:01 smithi146 bash[23809]: cluster 2024-03-11T21:19:00.125681+0000 osd.4 (osd.4) 134 : cluster 0 2.f scrub ok 2024-03-11T21:19:02.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:01 smithi146 bash[23809]: cluster 2024-03-11T21:19:00.278462+0000 osd.7 (osd.7) 685 : cluster 0 1.2 scrub starts 2024-03-11T21:19:02.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:01 smithi146 bash[23809]: cluster 2024-03-11T21:19:00.280835+0000 osd.7 (osd.7) 686 : cluster 0 1.2 scrub ok 2024-03-11T21:19:02.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:01 smithi146 bash[23809]: cluster 2024-03-11T21:19:00.558000+0000 mgr.y (mgr.24380) 1040 : cluster 0 pgmap v1213: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 443 B/s rd, 0 op/s 2024-03-11T21:19:02.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:01 smithi146 bash[23809]: cluster 2024-03-11T21:19:00.568029+0000 osd.3 (osd.3) 608 : cluster 0 4.1d scrub starts 2024-03-11T21:19:02.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:01 smithi146 bash[23809]: cluster 2024-03-11T21:19:00.569311+0000 osd.3 (osd.3) 609 : cluster 0 4.1d scrub ok 2024-03-11T21:19:02.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:01 smithi146 bash[23809]: cluster 2024-03-11T21:19:01.406879+0000 osd.5 (osd.5) 661 : cluster 0 1.1 scrub starts 2024-03-11T21:19:02.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:01 smithi146 bash[23809]: cluster 2024-03-11T21:19:01.409137+0000 osd.5 (osd.5) 662 : cluster 0 1.1 scrub ok 2024-03-11T21:19:02.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:01 smithi146 bash[23809]: audit 2024-03-11T21:19:01.601648+0000 mon.a (mon.0) 2096 : audit 1 from='client.? 172.21.15.23:0/3593544374' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-11T21:19:02.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:01 smithi023 bash[21372]: cluster 2024-03-11T21:19:00.125681+0000 osd.4 (osd.4) 134 : cluster 0 2.f scrub ok 2024-03-11T21:19:02.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:01 smithi023 bash[21372]: cluster 2024-03-11T21:19:00.278462+0000 osd.7 (osd.7) 685 : cluster 0 1.2 scrub starts 2024-03-11T21:19:02.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:01 smithi023 bash[21372]: cluster 2024-03-11T21:19:00.280835+0000 osd.7 (osd.7) 686 : cluster 0 1.2 scrub ok 2024-03-11T21:19:02.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:01 smithi023 bash[21372]: cluster 2024-03-11T21:19:00.558000+0000 mgr.y (mgr.24380) 1040 : cluster 0 pgmap v1213: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 443 B/s rd, 0 op/s 2024-03-11T21:19:02.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:01 smithi023 bash[21372]: cluster 2024-03-11T21:19:00.568029+0000 osd.3 (osd.3) 608 : cluster 0 4.1d scrub starts 2024-03-11T21:19:02.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:01 smithi023 bash[21372]: cluster 2024-03-11T21:19:00.569311+0000 osd.3 (osd.3) 609 : cluster 0 4.1d scrub ok 2024-03-11T21:19:02.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:01 smithi023 bash[21372]: cluster 2024-03-11T21:19:01.406879+0000 osd.5 (osd.5) 661 : cluster 0 1.1 scrub starts 2024-03-11T21:19:02.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:01 smithi023 bash[21372]: cluster 2024-03-11T21:19:01.409137+0000 osd.5 (osd.5) 662 : cluster 0 1.1 scrub ok 2024-03-11T21:19:02.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:01 smithi023 bash[21372]: audit 2024-03-11T21:19:01.601648+0000 mon.a (mon.0) 2096 : audit 1 from='client.? 172.21.15.23:0/3593544374' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-11T21:19:02.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:01 smithi023 bash[29146]: cluster 2024-03-11T21:19:00.125681+0000 osd.4 (osd.4) 134 : cluster 0 2.f scrub ok 2024-03-11T21:19:02.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:01 smithi023 bash[29146]: cluster 2024-03-11T21:19:00.278462+0000 osd.7 (osd.7) 685 : cluster 0 1.2 scrub starts 2024-03-11T21:19:02.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:01 smithi023 bash[29146]: cluster 2024-03-11T21:19:00.280835+0000 osd.7 (osd.7) 686 : cluster 0 1.2 scrub ok 2024-03-11T21:19:02.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:01 smithi023 bash[29146]: cluster 2024-03-11T21:19:00.558000+0000 mgr.y (mgr.24380) 1040 : cluster 0 pgmap v1213: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 443 B/s rd, 0 op/s 2024-03-11T21:19:02.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:01 smithi023 bash[29146]: cluster 2024-03-11T21:19:00.568029+0000 osd.3 (osd.3) 608 : cluster 0 4.1d scrub starts 2024-03-11T21:19:02.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:01 smithi023 bash[29146]: cluster 2024-03-11T21:19:00.569311+0000 osd.3 (osd.3) 609 : cluster 0 4.1d scrub ok 2024-03-11T21:19:02.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:01 smithi023 bash[29146]: cluster 2024-03-11T21:19:01.406879+0000 osd.5 (osd.5) 661 : cluster 0 1.1 scrub starts 2024-03-11T21:19:02.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:01 smithi023 bash[29146]: cluster 2024-03-11T21:19:01.409137+0000 osd.5 (osd.5) 662 : cluster 0 1.1 scrub ok 2024-03-11T21:19:02.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:01 smithi023 bash[29146]: audit 2024-03-11T21:19:01.601648+0000 mon.a (mon.0) 2096 : audit 1 from='client.? 172.21.15.23:0/3593544374' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-11T21:19:02.503 INFO:teuthology.orchestra.run.smithi023.stderr:reweighted osd.1 to 1 (10000) 2024-03-11T21:19:02.774 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:02 smithi023 bash[29146]: cluster 2024-03-11T21:19:01.276811+0000 osd.7 (osd.7) 687 : cluster 0 3.15 scrub starts 2024-03-11T21:19:02.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:02 smithi023 bash[29146]: cluster 2024-03-11T21:19:01.280786+0000 osd.7 (osd.7) 688 : cluster 0 3.15 scrub ok 2024-03-11T21:19:02.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:02 smithi023 bash[29146]: cluster 2024-03-11T21:19:01.323796+0000 osd.2 (osd.2) 303 : cluster 0 2.1 deep-scrub starts 2024-03-11T21:19:02.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:02 smithi023 bash[29146]: cluster 2024-03-11T21:19:01.325109+0000 osd.2 (osd.2) 304 : cluster 0 2.1 deep-scrub ok 2024-03-11T21:19:02.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:02 smithi023 bash[29146]: cluster 2024-03-11T21:19:02.347619+0000 osd.0 (osd.0) 498 : cluster 0 3.12 scrub starts 2024-03-11T21:19:02.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:02 smithi023 bash[29146]: cluster 2024-03-11T21:19:02.351645+0000 osd.0 (osd.0) 499 : cluster 0 3.12 scrub ok 2024-03-11T21:19:02.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:02 smithi023 bash[29146]: audit 2024-03-11T21:19:02.512338+0000 mon.a (mon.0) 2097 : audit 1 from='client.? 172.21.15.23:0/3593544374' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-11T21:19:02.776 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:02 smithi023 bash[29146]: cluster 2024-03-11T21:19:02.522470+0000 mon.a (mon.0) 2098 : cluster 0 osdmap e489: 8 total, 8 up, 8 in 2024-03-11T21:19:02.776 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:02 smithi023 bash[21372]: cluster 2024-03-11T21:19:01.276811+0000 osd.7 (osd.7) 687 : cluster 0 3.15 scrub starts 2024-03-11T21:19:02.776 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:02 smithi023 bash[21372]: cluster 2024-03-11T21:19:01.280786+0000 osd.7 (osd.7) 688 : cluster 0 3.15 scrub ok 2024-03-11T21:19:02.776 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:02 smithi023 bash[21372]: cluster 2024-03-11T21:19:01.323796+0000 osd.2 (osd.2) 303 : cluster 0 2.1 deep-scrub starts 2024-03-11T21:19:02.776 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:02 smithi023 bash[21372]: cluster 2024-03-11T21:19:01.325109+0000 osd.2 (osd.2) 304 : cluster 0 2.1 deep-scrub ok 2024-03-11T21:19:02.776 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:02 smithi023 bash[21372]: cluster 2024-03-11T21:19:02.347619+0000 osd.0 (osd.0) 498 : cluster 0 3.12 scrub starts 2024-03-11T21:19:02.776 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:02 smithi023 bash[21372]: cluster 2024-03-11T21:19:02.351645+0000 osd.0 (osd.0) 499 : cluster 0 3.12 scrub ok 2024-03-11T21:19:02.777 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:02 smithi023 bash[21372]: audit 2024-03-11T21:19:02.512338+0000 mon.a (mon.0) 2097 : audit 1 from='client.? 172.21.15.23:0/3593544374' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-11T21:19:02.777 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:02 smithi023 bash[21372]: cluster 2024-03-11T21:19:02.522470+0000 mon.a (mon.0) 2098 : cluster 0 osdmap e489: 8 total, 8 up, 8 in 2024-03-11T21:19:03.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:02 smithi146 bash[23809]: cluster 2024-03-11T21:19:01.276811+0000 osd.7 (osd.7) 687 : cluster 0 3.15 scrub starts 2024-03-11T21:19:03.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:02 smithi146 bash[23809]: cluster 2024-03-11T21:19:01.280786+0000 osd.7 (osd.7) 688 : cluster 0 3.15 scrub ok 2024-03-11T21:19:03.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:02 smithi146 bash[23809]: cluster 2024-03-11T21:19:01.323796+0000 osd.2 (osd.2) 303 : cluster 0 2.1 deep-scrub starts 2024-03-11T21:19:03.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:02 smithi146 bash[23809]: cluster 2024-03-11T21:19:01.325109+0000 osd.2 (osd.2) 304 : cluster 0 2.1 deep-scrub ok 2024-03-11T21:19:03.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:02 smithi146 bash[23809]: cluster 2024-03-11T21:19:02.347619+0000 osd.0 (osd.0) 498 : cluster 0 3.12 scrub starts 2024-03-11T21:19:03.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:02 smithi146 bash[23809]: cluster 2024-03-11T21:19:02.351645+0000 osd.0 (osd.0) 499 : cluster 0 3.12 scrub ok 2024-03-11T21:19:03.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:02 smithi146 bash[23809]: audit 2024-03-11T21:19:02.512338+0000 mon.a (mon.0) 2097 : audit 1 from='client.? 172.21.15.23:0/3593544374' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-11T21:19:03.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:02 smithi146 bash[23809]: cluster 2024-03-11T21:19:02.522470+0000 mon.a (mon.0) 2098 : cluster 0 osdmap e489: 8 total, 8 up, 8 in 2024-03-11T21:19:03.465 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd primary-affinity 1 1 2024-03-11T21:19:04.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:03 smithi146 bash[23809]: cluster 2024-03-11T21:19:02.558515+0000 mgr.y (mgr.24380) 1041 : cluster 0 pgmap v1215: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T21:19:04.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:03 smithi146 bash[23809]: cluster 2024-03-11T21:19:02.623627+0000 osd.3 (osd.3) 610 : cluster 0 6.b scrub starts 2024-03-11T21:19:04.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:03 smithi146 bash[23809]: cluster 2024-03-11T21:19:02.797558+0000 osd.3 (osd.3) 611 : cluster 0 6.b scrub ok 2024-03-11T21:19:04.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:03 smithi146 bash[23809]: cluster 2024-03-11T21:19:03.511624+0000 osd.6 (osd.6) 546 : cluster 0 2.7 scrub starts 2024-03-11T21:19:04.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:03 smithi146 bash[23809]: cluster 2024-03-11T21:19:03.516808+0000 osd.6 (osd.6) 547 : cluster 0 2.7 scrub ok 2024-03-11T21:19:04.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:03 smithi023 bash[29146]: cluster 2024-03-11T21:19:02.558515+0000 mgr.y (mgr.24380) 1041 : cluster 0 pgmap v1215: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T21:19:04.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:03 smithi023 bash[29146]: cluster 2024-03-11T21:19:02.623627+0000 osd.3 (osd.3) 610 : cluster 0 6.b scrub starts 2024-03-11T21:19:04.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:03 smithi023 bash[29146]: cluster 2024-03-11T21:19:02.797558+0000 osd.3 (osd.3) 611 : cluster 0 6.b scrub ok 2024-03-11T21:19:04.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:03 smithi023 bash[29146]: cluster 2024-03-11T21:19:03.511624+0000 osd.6 (osd.6) 546 : cluster 0 2.7 scrub starts 2024-03-11T21:19:04.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:03 smithi023 bash[29146]: cluster 2024-03-11T21:19:03.516808+0000 osd.6 (osd.6) 547 : cluster 0 2.7 scrub ok 2024-03-11T21:19:04.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:03 smithi023 bash[21372]: cluster 2024-03-11T21:19:02.558515+0000 mgr.y (mgr.24380) 1041 : cluster 0 pgmap v1215: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T21:19:04.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:03 smithi023 bash[21372]: cluster 2024-03-11T21:19:02.623627+0000 osd.3 (osd.3) 610 : cluster 0 6.b scrub starts 2024-03-11T21:19:04.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:03 smithi023 bash[21372]: cluster 2024-03-11T21:19:02.797558+0000 osd.3 (osd.3) 611 : cluster 0 6.b scrub ok 2024-03-11T21:19:04.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:03 smithi023 bash[21372]: cluster 2024-03-11T21:19:03.511624+0000 osd.6 (osd.6) 546 : cluster 0 2.7 scrub starts 2024-03-11T21:19:04.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:03 smithi023 bash[21372]: cluster 2024-03-11T21:19:03.516808+0000 osd.6 (osd.6) 547 : cluster 0 2.7 scrub ok 2024-03-11T21:19:05.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:04 smithi146 bash[23809]: cluster 2024-03-11T21:19:03.927628+0000 osd.1 (osd.1) 424 : cluster 0 purged_snaps scrub starts 2024-03-11T21:19:05.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:04 smithi146 bash[23809]: cluster 2024-03-11T21:19:03.927786+0000 osd.1 (osd.1) 425 : cluster 0 purged_snaps scrub ok 2024-03-11T21:19:05.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:04 smithi146 bash[23809]: cluster 2024-03-11T21:19:03.956503+0000 osd.1 (osd.1) 426 : cluster 0 1.13 scrub starts 2024-03-11T21:19:05.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:04 smithi146 bash[23809]: cluster 2024-03-11T21:19:03.958836+0000 osd.1 (osd.1) 427 : cluster 0 1.13 scrub ok 2024-03-11T21:19:05.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:04 smithi146 bash[23809]: cluster 2024-03-11T21:19:04.416615+0000 osd.5 (osd.5) 663 : cluster 0 1.8 deep-scrub starts 2024-03-11T21:19:05.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:04 smithi146 bash[23809]: cluster 2024-03-11T21:19:04.421361+0000 osd.5 (osd.5) 664 : cluster 0 1.8 deep-scrub ok 2024-03-11T21:19:05.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:04 smithi146 bash[23809]: cluster 2024-03-11T21:19:04.524181+0000 osd.6 (osd.6) 548 : cluster 0 5.2 scrub starts 2024-03-11T21:19:05.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:04 smithi023 bash[21372]: cluster 2024-03-11T21:19:03.927628+0000 osd.1 (osd.1) 424 : cluster 0 purged_snaps scrub starts 2024-03-11T21:19:05.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:04 smithi023 bash[21372]: cluster 2024-03-11T21:19:03.927786+0000 osd.1 (osd.1) 425 : cluster 0 purged_snaps scrub ok 2024-03-11T21:19:05.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:04 smithi023 bash[21372]: cluster 2024-03-11T21:19:03.956503+0000 osd.1 (osd.1) 426 : cluster 0 1.13 scrub starts 2024-03-11T21:19:05.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:04 smithi023 bash[21372]: cluster 2024-03-11T21:19:03.958836+0000 osd.1 (osd.1) 427 : cluster 0 1.13 scrub ok 2024-03-11T21:19:05.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:04 smithi023 bash[21372]: cluster 2024-03-11T21:19:04.416615+0000 osd.5 (osd.5) 663 : cluster 0 1.8 deep-scrub starts 2024-03-11T21:19:05.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:04 smithi023 bash[21372]: cluster 2024-03-11T21:19:04.421361+0000 osd.5 (osd.5) 664 : cluster 0 1.8 deep-scrub ok 2024-03-11T21:19:05.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:04 smithi023 bash[21372]: cluster 2024-03-11T21:19:04.524181+0000 osd.6 (osd.6) 548 : cluster 0 5.2 scrub starts 2024-03-11T21:19:05.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:04 smithi023 bash[29146]: cluster 2024-03-11T21:19:03.927628+0000 osd.1 (osd.1) 424 : cluster 0 purged_snaps scrub starts 2024-03-11T21:19:05.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:04 smithi023 bash[29146]: cluster 2024-03-11T21:19:03.927786+0000 osd.1 (osd.1) 425 : cluster 0 purged_snaps scrub ok 2024-03-11T21:19:05.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:04 smithi023 bash[29146]: cluster 2024-03-11T21:19:03.956503+0000 osd.1 (osd.1) 426 : cluster 0 1.13 scrub starts 2024-03-11T21:19:05.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:04 smithi023 bash[29146]: cluster 2024-03-11T21:19:03.958836+0000 osd.1 (osd.1) 427 : cluster 0 1.13 scrub ok 2024-03-11T21:19:05.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:04 smithi023 bash[29146]: cluster 2024-03-11T21:19:04.416615+0000 osd.5 (osd.5) 663 : cluster 0 1.8 deep-scrub starts 2024-03-11T21:19:05.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:04 smithi023 bash[29146]: cluster 2024-03-11T21:19:04.421361+0000 osd.5 (osd.5) 664 : cluster 0 1.8 deep-scrub ok 2024-03-11T21:19:05.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:04 smithi023 bash[29146]: cluster 2024-03-11T21:19:04.524181+0000 osd.6 (osd.6) 548 : cluster 0 5.2 scrub starts 2024-03-11T21:19:06.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:05 smithi146 bash[23809]: cluster 2024-03-11T21:19:04.559592+0000 mgr.y (mgr.24380) 1042 : cluster 0 pgmap v1216: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:19:06.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:05 smithi146 bash[23809]: cluster 2024-03-11T21:19:04.573367+0000 osd.6 (osd.6) 549 : cluster 0 5.2 scrub ok 2024-03-11T21:19:06.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:05 smithi146 bash[23809]: cluster 2024-03-11T21:19:04.767930+0000 mon.a (mon.0) 2099 : cluster 0 osdmap e490: 8 total, 8 up, 8 in 2024-03-11T21:19:06.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:05 smithi146 bash[23809]: cluster 2024-03-11T21:19:05.332478+0000 osd.0 (osd.0) 500 : cluster 0 2.1f scrub starts 2024-03-11T21:19:06.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:05 smithi146 bash[23809]: cluster 2024-03-11T21:19:05.337050+0000 osd.0 (osd.0) 501 : cluster 0 2.1f scrub ok 2024-03-11T21:19:06.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:05 smithi146 bash[23809]: cluster 2024-03-11T21:19:05.466141+0000 osd.5 (osd.5) 665 : cluster 0 2.2 scrub starts 2024-03-11T21:19:06.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:05 smithi146 bash[23809]: cluster 2024-03-11T21:19:05.468532+0000 osd.5 (osd.5) 666 : cluster 0 2.2 scrub ok 2024-03-11T21:19:06.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:05 smithi023 bash[21372]: cluster 2024-03-11T21:19:04.559592+0000 mgr.y (mgr.24380) 1042 : cluster 0 pgmap v1216: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:19:06.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:05 smithi023 bash[21372]: cluster 2024-03-11T21:19:04.573367+0000 osd.6 (osd.6) 549 : cluster 0 5.2 scrub ok 2024-03-11T21:19:06.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:05 smithi023 bash[21372]: cluster 2024-03-11T21:19:04.767930+0000 mon.a (mon.0) 2099 : cluster 0 osdmap e490: 8 total, 8 up, 8 in 2024-03-11T21:19:06.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:05 smithi023 bash[21372]: cluster 2024-03-11T21:19:05.332478+0000 osd.0 (osd.0) 500 : cluster 0 2.1f scrub starts 2024-03-11T21:19:06.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:05 smithi023 bash[21372]: cluster 2024-03-11T21:19:05.337050+0000 osd.0 (osd.0) 501 : cluster 0 2.1f scrub ok 2024-03-11T21:19:06.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:05 smithi023 bash[21372]: cluster 2024-03-11T21:19:05.466141+0000 osd.5 (osd.5) 665 : cluster 0 2.2 scrub starts 2024-03-11T21:19:06.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:05 smithi023 bash[21372]: cluster 2024-03-11T21:19:05.468532+0000 osd.5 (osd.5) 666 : cluster 0 2.2 scrub ok 2024-03-11T21:19:06.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:05 smithi023 bash[29146]: cluster 2024-03-11T21:19:04.559592+0000 mgr.y (mgr.24380) 1042 : cluster 0 pgmap v1216: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:19:06.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:05 smithi023 bash[29146]: cluster 2024-03-11T21:19:04.573367+0000 osd.6 (osd.6) 549 : cluster 0 5.2 scrub ok 2024-03-11T21:19:06.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:05 smithi023 bash[29146]: cluster 2024-03-11T21:19:04.767930+0000 mon.a (mon.0) 2099 : cluster 0 osdmap e490: 8 total, 8 up, 8 in 2024-03-11T21:19:06.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:05 smithi023 bash[29146]: cluster 2024-03-11T21:19:05.332478+0000 osd.0 (osd.0) 500 : cluster 0 2.1f scrub starts 2024-03-11T21:19:06.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:05 smithi023 bash[29146]: cluster 2024-03-11T21:19:05.337050+0000 osd.0 (osd.0) 501 : cluster 0 2.1f scrub ok 2024-03-11T21:19:06.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:05 smithi023 bash[29146]: cluster 2024-03-11T21:19:05.466141+0000 osd.5 (osd.5) 665 : cluster 0 2.2 scrub starts 2024-03-11T21:19:06.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:05 smithi023 bash[29146]: cluster 2024-03-11T21:19:05.468532+0000 osd.5 (osd.5) 666 : cluster 0 2.2 scrub ok 2024-03-11T21:19:06.749 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:19:06 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:19:07.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:06 smithi146 bash[23809]: cluster 2024-03-11T21:19:05.160640+0000 osd.4 (osd.4) 135 : cluster 0 4.d scrub starts 2024-03-11T21:19:07.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:06 smithi146 bash[23809]: cluster 2024-03-11T21:19:05.163534+0000 osd.4 (osd.4) 136 : cluster 0 4.d scrub ok 2024-03-11T21:19:07.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:06 smithi146 bash[23809]: cluster 2024-03-11T21:19:05.729497+0000 osd.3 (osd.3) 612 : cluster 0 6.c scrub starts 2024-03-11T21:19:07.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:06 smithi146 bash[23809]: cluster 2024-03-11T21:19:05.731428+0000 osd.3 (osd.3) 613 : cluster 0 6.c scrub ok 2024-03-11T21:19:07.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:06 smithi146 bash[23809]: cluster 2024-03-11T21:19:06.382842+0000 osd.0 (osd.0) 502 : cluster 0 1.11 scrub starts 2024-03-11T21:19:07.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:06 smithi146 bash[23809]: cluster 2024-03-11T21:19:06.385453+0000 osd.0 (osd.0) 503 : cluster 0 1.11 scrub ok 2024-03-11T21:19:07.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:06 smithi023 bash[21372]: cluster 2024-03-11T21:19:05.160640+0000 osd.4 (osd.4) 135 : cluster 0 4.d scrub starts 2024-03-11T21:19:07.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:06 smithi023 bash[21372]: cluster 2024-03-11T21:19:05.163534+0000 osd.4 (osd.4) 136 : cluster 0 4.d scrub ok 2024-03-11T21:19:07.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:06 smithi023 bash[21372]: cluster 2024-03-11T21:19:05.729497+0000 osd.3 (osd.3) 612 : cluster 0 6.c scrub starts 2024-03-11T21:19:07.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:06 smithi023 bash[21372]: cluster 2024-03-11T21:19:05.731428+0000 osd.3 (osd.3) 613 : cluster 0 6.c scrub ok 2024-03-11T21:19:07.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:06 smithi023 bash[21372]: cluster 2024-03-11T21:19:06.382842+0000 osd.0 (osd.0) 502 : cluster 0 1.11 scrub starts 2024-03-11T21:19:07.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:06 smithi023 bash[21372]: cluster 2024-03-11T21:19:06.385453+0000 osd.0 (osd.0) 503 : cluster 0 1.11 scrub ok 2024-03-11T21:19:07.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:06 smithi023 bash[29146]: cluster 2024-03-11T21:19:05.160640+0000 osd.4 (osd.4) 135 : cluster 0 4.d scrub starts 2024-03-11T21:19:07.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:06 smithi023 bash[29146]: cluster 2024-03-11T21:19:05.163534+0000 osd.4 (osd.4) 136 : cluster 0 4.d scrub ok 2024-03-11T21:19:07.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:06 smithi023 bash[29146]: cluster 2024-03-11T21:19:05.729497+0000 osd.3 (osd.3) 612 : cluster 0 6.c scrub starts 2024-03-11T21:19:07.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:06 smithi023 bash[29146]: cluster 2024-03-11T21:19:05.731428+0000 osd.3 (osd.3) 613 : cluster 0 6.c scrub ok 2024-03-11T21:19:07.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:06 smithi023 bash[29146]: cluster 2024-03-11T21:19:06.382842+0000 osd.0 (osd.0) 502 : cluster 0 1.11 scrub starts 2024-03-11T21:19:07.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:06 smithi023 bash[29146]: cluster 2024-03-11T21:19:06.385453+0000 osd.0 (osd.0) 503 : cluster 0 1.11 scrub ok 2024-03-11T21:19:08.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:07 smithi146 bash[23809]: audit 2024-03-11T21:19:06.296931+0000 mgr.y (mgr.24380) 1043 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:19:08.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:07 smithi146 bash[23809]: cluster 2024-03-11T21:19:06.561301+0000 mgr.y (mgr.24380) 1044 : cluster 0 pgmap v1218: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:19:08.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:07 smithi146 bash[23809]: cluster 2024-03-11T21:19:06.996902+0000 osd.1 (osd.1) 428 : cluster 0 6.4 scrub starts 2024-03-11T21:19:08.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:07 smithi146 bash[23809]: cluster 2024-03-11T21:19:06.999382+0000 osd.1 (osd.1) 429 : cluster 0 6.4 scrub ok 2024-03-11T21:19:08.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:07 smithi146 bash[23809]: audit 2024-03-11T21:19:07.020114+0000 mon.a (mon.0) 2100 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:19:08.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:07 smithi023 bash[21372]: audit 2024-03-11T21:19:06.296931+0000 mgr.y (mgr.24380) 1043 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:19:08.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:07 smithi023 bash[21372]: cluster 2024-03-11T21:19:06.561301+0000 mgr.y (mgr.24380) 1044 : cluster 0 pgmap v1218: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:19:08.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:07 smithi023 bash[21372]: cluster 2024-03-11T21:19:06.996902+0000 osd.1 (osd.1) 428 : cluster 0 6.4 scrub starts 2024-03-11T21:19:08.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:07 smithi023 bash[21372]: cluster 2024-03-11T21:19:06.999382+0000 osd.1 (osd.1) 429 : cluster 0 6.4 scrub ok 2024-03-11T21:19:08.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:07 smithi023 bash[21372]: audit 2024-03-11T21:19:07.020114+0000 mon.a (mon.0) 2100 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:19:08.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:07 smithi023 bash[29146]: audit 2024-03-11T21:19:06.296931+0000 mgr.y (mgr.24380) 1043 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:19:08.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:07 smithi023 bash[29146]: cluster 2024-03-11T21:19:06.561301+0000 mgr.y (mgr.24380) 1044 : cluster 0 pgmap v1218: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:19:08.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:07 smithi023 bash[29146]: cluster 2024-03-11T21:19:06.996902+0000 osd.1 (osd.1) 428 : cluster 0 6.4 scrub starts 2024-03-11T21:19:08.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:07 smithi023 bash[29146]: cluster 2024-03-11T21:19:06.999382+0000 osd.1 (osd.1) 429 : cluster 0 6.4 scrub ok 2024-03-11T21:19:08.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:07 smithi023 bash[29146]: audit 2024-03-11T21:19:07.020114+0000 mon.a (mon.0) 2100 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:19:08.263 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:19:08.774 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:19:08 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:19:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:19:09.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:08 smithi146 bash[23809]: cluster 2024-03-11T21:19:07.176796+0000 osd.4 (osd.4) 137 : cluster 0 6.e scrub starts 2024-03-11T21:19:09.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:08 smithi146 bash[23809]: cluster 2024-03-11T21:19:07.241642+0000 osd.7 (osd.7) 689 : cluster 0 3.13 scrub starts 2024-03-11T21:19:09.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:08 smithi146 bash[23809]: cluster 2024-03-11T21:19:07.246099+0000 osd.7 (osd.7) 690 : cluster 0 3.13 scrub ok 2024-03-11T21:19:09.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:08 smithi146 bash[23809]: cluster 2024-03-11T21:19:07.246709+0000 osd.4 (osd.4) 138 : cluster 0 6.e scrub ok 2024-03-11T21:19:09.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:08 smithi146 bash[23809]: cluster 2024-03-11T21:19:07.987689+0000 osd.1 (osd.1) 430 : cluster 0 2.d scrub starts 2024-03-11T21:19:09.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:08 smithi146 bash[23809]: cluster 2024-03-11T21:19:07.990277+0000 osd.1 (osd.1) 431 : cluster 0 2.d scrub ok 2024-03-11T21:19:09.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:08 smithi023 bash[21372]: cluster 2024-03-11T21:19:07.176796+0000 osd.4 (osd.4) 137 : cluster 0 6.e scrub starts 2024-03-11T21:19:09.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:08 smithi023 bash[21372]: cluster 2024-03-11T21:19:07.241642+0000 osd.7 (osd.7) 689 : cluster 0 3.13 scrub starts 2024-03-11T21:19:09.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:08 smithi023 bash[21372]: cluster 2024-03-11T21:19:07.246099+0000 osd.7 (osd.7) 690 : cluster 0 3.13 scrub ok 2024-03-11T21:19:09.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:08 smithi023 bash[21372]: cluster 2024-03-11T21:19:07.246709+0000 osd.4 (osd.4) 138 : cluster 0 6.e scrub ok 2024-03-11T21:19:09.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:08 smithi023 bash[21372]: cluster 2024-03-11T21:19:07.987689+0000 osd.1 (osd.1) 430 : cluster 0 2.d scrub starts 2024-03-11T21:19:09.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:08 smithi023 bash[21372]: cluster 2024-03-11T21:19:07.990277+0000 osd.1 (osd.1) 431 : cluster 0 2.d scrub ok 2024-03-11T21:19:09.275 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:08 smithi023 bash[29146]: cluster 2024-03-11T21:19:07.176796+0000 osd.4 (osd.4) 137 : cluster 0 6.e scrub starts 2024-03-11T21:19:09.275 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:08 smithi023 bash[29146]: cluster 2024-03-11T21:19:07.241642+0000 osd.7 (osd.7) 689 : cluster 0 3.13 scrub starts 2024-03-11T21:19:09.275 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:08 smithi023 bash[29146]: cluster 2024-03-11T21:19:07.246099+0000 osd.7 (osd.7) 690 : cluster 0 3.13 scrub ok 2024-03-11T21:19:09.275 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:08 smithi023 bash[29146]: cluster 2024-03-11T21:19:07.246709+0000 osd.4 (osd.4) 138 : cluster 0 6.e scrub ok 2024-03-11T21:19:09.275 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:08 smithi023 bash[29146]: cluster 2024-03-11T21:19:07.987689+0000 osd.1 (osd.1) 430 : cluster 0 2.d scrub starts 2024-03-11T21:19:09.275 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:08 smithi023 bash[29146]: cluster 2024-03-11T21:19:07.990277+0000 osd.1 (osd.1) 431 : cluster 0 2.d scrub ok 2024-03-11T21:19:10.232 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:09 smithi023 bash[21372]: cluster 2024-03-11T21:19:08.326033+0000 osd.2 (osd.2) 305 : cluster 0 4.1c deep-scrub starts 2024-03-11T21:19:10.233 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:09 smithi023 bash[21372]: cluster 2024-03-11T21:19:08.327553+0000 osd.2 (osd.2) 306 : cluster 0 4.1c deep-scrub ok 2024-03-11T21:19:10.233 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:09 smithi023 bash[21372]: cluster 2024-03-11T21:19:08.562222+0000 mgr.y (mgr.24380) 1045 : cluster 0 pgmap v1219: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:19:10.233 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:09 smithi023 bash[21372]: cluster 2024-03-11T21:19:09.469733+0000 osd.6 (osd.6) 550 : cluster 0 1.f scrub starts 2024-03-11T21:19:10.233 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:09 smithi023 bash[21372]: cluster 2024-03-11T21:19:09.472418+0000 osd.6 (osd.6) 551 : cluster 0 1.f scrub ok 2024-03-11T21:19:10.233 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:09 smithi023 bash[21372]: cluster 2024-03-11T21:19:09.565296+0000 osd.5 (osd.5) 667 : cluster 0 2.16 scrub starts 2024-03-11T21:19:10.233 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:09 smithi023 bash[21372]: cluster 2024-03-11T21:19:09.567598+0000 osd.5 (osd.5) 668 : cluster 0 2.16 scrub ok 2024-03-11T21:19:10.233 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:09 smithi023 bash[29146]: cluster 2024-03-11T21:19:08.326033+0000 osd.2 (osd.2) 305 : cluster 0 4.1c deep-scrub starts 2024-03-11T21:19:10.233 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:09 smithi023 bash[29146]: cluster 2024-03-11T21:19:08.327553+0000 osd.2 (osd.2) 306 : cluster 0 4.1c deep-scrub ok 2024-03-11T21:19:10.233 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:09 smithi023 bash[29146]: cluster 2024-03-11T21:19:08.562222+0000 mgr.y (mgr.24380) 1045 : cluster 0 pgmap v1219: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:19:10.233 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:09 smithi023 bash[29146]: cluster 2024-03-11T21:19:09.469733+0000 osd.6 (osd.6) 550 : cluster 0 1.f scrub starts 2024-03-11T21:19:10.234 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:09 smithi023 bash[29146]: cluster 2024-03-11T21:19:09.472418+0000 osd.6 (osd.6) 551 : cluster 0 1.f scrub ok 2024-03-11T21:19:10.234 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:09 smithi023 bash[29146]: cluster 2024-03-11T21:19:09.565296+0000 osd.5 (osd.5) 667 : cluster 0 2.16 scrub starts 2024-03-11T21:19:10.234 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:09 smithi023 bash[29146]: cluster 2024-03-11T21:19:09.567598+0000 osd.5 (osd.5) 668 : cluster 0 2.16 scrub ok 2024-03-11T21:19:10.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:09 smithi146 bash[23809]: cluster 2024-03-11T21:19:08.326033+0000 osd.2 (osd.2) 305 : cluster 0 4.1c deep-scrub starts 2024-03-11T21:19:10.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:09 smithi146 bash[23809]: cluster 2024-03-11T21:19:08.327553+0000 osd.2 (osd.2) 306 : cluster 0 4.1c deep-scrub ok 2024-03-11T21:19:10.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:09 smithi146 bash[23809]: cluster 2024-03-11T21:19:08.562222+0000 mgr.y (mgr.24380) 1045 : cluster 0 pgmap v1219: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:19:10.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:09 smithi146 bash[23809]: cluster 2024-03-11T21:19:09.469733+0000 osd.6 (osd.6) 550 : cluster 0 1.f scrub starts 2024-03-11T21:19:10.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:09 smithi146 bash[23809]: cluster 2024-03-11T21:19:09.472418+0000 osd.6 (osd.6) 551 : cluster 0 1.f scrub ok 2024-03-11T21:19:10.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:09 smithi146 bash[23809]: cluster 2024-03-11T21:19:09.565296+0000 osd.5 (osd.5) 667 : cluster 0 2.16 scrub starts 2024-03-11T21:19:10.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:09 smithi146 bash[23809]: cluster 2024-03-11T21:19:09.567598+0000 osd.5 (osd.5) 668 : cluster 0 2.16 scrub ok 2024-03-11T21:19:11.111 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:10 smithi023 bash[21372]: cluster 2024-03-11T21:19:09.266889+0000 osd.7 (osd.7) 691 : cluster 0 6.1c scrub starts 2024-03-11T21:19:11.111 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:10 smithi023 bash[21372]: cluster 2024-03-11T21:19:09.270130+0000 osd.7 (osd.7) 692 : cluster 0 6.1c scrub ok 2024-03-11T21:19:11.112 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:10 smithi023 bash[29146]: cluster 2024-03-11T21:19:09.266889+0000 osd.7 (osd.7) 691 : cluster 0 6.1c scrub starts 2024-03-11T21:19:11.112 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:10 smithi023 bash[29146]: cluster 2024-03-11T21:19:09.270130+0000 osd.7 (osd.7) 692 : cluster 0 6.1c scrub ok 2024-03-11T21:19:11.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:10 smithi146 bash[23809]: cluster 2024-03-11T21:19:09.266889+0000 osd.7 (osd.7) 691 : cluster 0 6.1c scrub starts 2024-03-11T21:19:11.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:10 smithi146 bash[23809]: cluster 2024-03-11T21:19:09.270130+0000 osd.7 (osd.7) 692 : cluster 0 6.1c scrub ok 2024-03-11T21:19:11.809 INFO:teuthology.orchestra.run.smithi023.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-03-11T21:19:12.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:11 smithi146 bash[23809]: cluster 2024-03-11T21:19:10.181418+0000 osd.4 (osd.4) 139 : cluster 0 3.3 scrub starts 2024-03-11T21:19:12.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:11 smithi146 bash[23809]: cluster 2024-03-11T21:19:10.186796+0000 osd.4 (osd.4) 140 : cluster 0 3.3 scrub ok 2024-03-11T21:19:12.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:11 smithi146 bash[23809]: cluster 2024-03-11T21:19:10.356386+0000 osd.2 (osd.2) 307 : cluster 0 2.10 deep-scrub starts 2024-03-11T21:19:12.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:11 smithi146 bash[23809]: cluster 2024-03-11T21:19:10.359112+0000 osd.2 (osd.2) 308 : cluster 0 2.10 deep-scrub ok 2024-03-11T21:19:12.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:11 smithi146 bash[23809]: cluster 2024-03-11T21:19:10.563256+0000 mgr.y (mgr.24380) 1046 : cluster 0 pgmap v1220: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 636 B/s rd, 0 op/s 2024-03-11T21:19:12.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:11 smithi146 bash[23809]: cluster 2024-03-11T21:19:10.688201+0000 osd.3 (osd.3) 614 : cluster 0 6.6 deep-scrub starts 2024-03-11T21:19:12.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:11 smithi146 bash[23809]: cluster 2024-03-11T21:19:10.721014+0000 osd.3 (osd.3) 615 : cluster 0 6.6 deep-scrub ok 2024-03-11T21:19:12.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:11 smithi146 bash[23809]: audit 2024-03-11T21:19:11.552426+0000 mon.a (mon.0) 2101 : audit 1 from='client.? 172.21.15.23:0/3495271350' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-03-11T21:19:12.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:11 smithi023 bash[21372]: cluster 2024-03-11T21:19:10.181418+0000 osd.4 (osd.4) 139 : cluster 0 3.3 scrub starts 2024-03-11T21:19:12.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:11 smithi023 bash[21372]: cluster 2024-03-11T21:19:10.186796+0000 osd.4 (osd.4) 140 : cluster 0 3.3 scrub ok 2024-03-11T21:19:12.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:11 smithi023 bash[21372]: cluster 2024-03-11T21:19:10.356386+0000 osd.2 (osd.2) 307 : cluster 0 2.10 deep-scrub starts 2024-03-11T21:19:12.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:11 smithi023 bash[21372]: cluster 2024-03-11T21:19:10.359112+0000 osd.2 (osd.2) 308 : cluster 0 2.10 deep-scrub ok 2024-03-11T21:19:12.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:11 smithi023 bash[21372]: cluster 2024-03-11T21:19:10.563256+0000 mgr.y (mgr.24380) 1046 : cluster 0 pgmap v1220: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 636 B/s rd, 0 op/s 2024-03-11T21:19:12.275 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:11 smithi023 bash[21372]: cluster 2024-03-11T21:19:10.688201+0000 osd.3 (osd.3) 614 : cluster 0 6.6 deep-scrub starts 2024-03-11T21:19:12.275 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:11 smithi023 bash[21372]: cluster 2024-03-11T21:19:10.721014+0000 osd.3 (osd.3) 615 : cluster 0 6.6 deep-scrub ok 2024-03-11T21:19:12.275 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:11 smithi023 bash[21372]: audit 2024-03-11T21:19:11.552426+0000 mon.a (mon.0) 2101 : audit 1 from='client.? 172.21.15.23:0/3495271350' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-03-11T21:19:12.275 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:11 smithi023 bash[29146]: cluster 2024-03-11T21:19:10.181418+0000 osd.4 (osd.4) 139 : cluster 0 3.3 scrub starts 2024-03-11T21:19:12.275 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:11 smithi023 bash[29146]: cluster 2024-03-11T21:19:10.186796+0000 osd.4 (osd.4) 140 : cluster 0 3.3 scrub ok 2024-03-11T21:19:12.275 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:11 smithi023 bash[29146]: cluster 2024-03-11T21:19:10.356386+0000 osd.2 (osd.2) 307 : cluster 0 2.10 deep-scrub starts 2024-03-11T21:19:12.275 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:11 smithi023 bash[29146]: cluster 2024-03-11T21:19:10.359112+0000 osd.2 (osd.2) 308 : cluster 0 2.10 deep-scrub ok 2024-03-11T21:19:12.275 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:11 smithi023 bash[29146]: cluster 2024-03-11T21:19:10.563256+0000 mgr.y (mgr.24380) 1046 : cluster 0 pgmap v1220: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 636 B/s rd, 0 op/s 2024-03-11T21:19:12.275 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:11 smithi023 bash[29146]: cluster 2024-03-11T21:19:10.688201+0000 osd.3 (osd.3) 614 : cluster 0 6.6 deep-scrub starts 2024-03-11T21:19:12.275 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:11 smithi023 bash[29146]: cluster 2024-03-11T21:19:10.721014+0000 osd.3 (osd.3) 615 : cluster 0 6.6 deep-scrub ok 2024-03-11T21:19:12.275 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:11 smithi023 bash[29146]: audit 2024-03-11T21:19:11.552426+0000 mon.a (mon.0) 2101 : audit 1 from='client.? 172.21.15.23:0/3495271350' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-03-11T21:19:12.678 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-03-11T21:19:12.678 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-03-11T21:19:12.681 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-03-11T21:19:12.681 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-03-11T21:19:12.682 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-11T21:19:12.682 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd dump --format=json 2024-03-11T21:19:13.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:12 smithi023 bash[21372]: cluster 2024-03-11T21:19:11.216747+0000 osd.4 (osd.4) 141 : cluster 0 4.9 scrub starts 2024-03-11T21:19:13.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:12 smithi023 bash[21372]: cluster 2024-03-11T21:19:11.219983+0000 osd.4 (osd.4) 142 : cluster 0 4.9 scrub ok 2024-03-11T21:19:13.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:12 smithi023 bash[21372]: audit 2024-03-11T21:19:11.818611+0000 mon.a (mon.0) 2102 : audit 1 from='client.? 172.21.15.23:0/3495271350' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-03-11T21:19:13.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:12 smithi023 bash[21372]: cluster 2024-03-11T21:19:11.828837+0000 mon.a (mon.0) 2103 : cluster 0 osdmap e491: 8 total, 8 up, 8 in 2024-03-11T21:19:13.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:12 smithi023 bash[21372]: cluster 2024-03-11T21:19:11.962149+0000 osd.1 (osd.1) 432 : cluster 0 3.4 scrub starts 2024-03-11T21:19:13.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:12 smithi023 bash[21372]: cluster 2024-03-11T21:19:11.965916+0000 osd.1 (osd.1) 433 : cluster 0 3.4 scrub ok 2024-03-11T21:19:13.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:12 smithi023 bash[21372]: cluster 2024-03-11T21:19:12.564019+0000 mgr.y (mgr.24380) 1047 : cluster 0 pgmap v1222: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:19:13.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:12 smithi023 bash[29146]: cluster 2024-03-11T21:19:11.216747+0000 osd.4 (osd.4) 141 : cluster 0 4.9 scrub starts 2024-03-11T21:19:13.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:12 smithi023 bash[29146]: cluster 2024-03-11T21:19:11.219983+0000 osd.4 (osd.4) 142 : cluster 0 4.9 scrub ok 2024-03-11T21:19:13.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:12 smithi023 bash[29146]: audit 2024-03-11T21:19:11.818611+0000 mon.a (mon.0) 2102 : audit 1 from='client.? 172.21.15.23:0/3495271350' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-03-11T21:19:13.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:12 smithi023 bash[29146]: cluster 2024-03-11T21:19:11.828837+0000 mon.a (mon.0) 2103 : cluster 0 osdmap e491: 8 total, 8 up, 8 in 2024-03-11T21:19:13.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:12 smithi023 bash[29146]: cluster 2024-03-11T21:19:11.962149+0000 osd.1 (osd.1) 432 : cluster 0 3.4 scrub starts 2024-03-11T21:19:13.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:12 smithi023 bash[29146]: cluster 2024-03-11T21:19:11.965916+0000 osd.1 (osd.1) 433 : cluster 0 3.4 scrub ok 2024-03-11T21:19:13.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:12 smithi023 bash[29146]: cluster 2024-03-11T21:19:12.564019+0000 mgr.y (mgr.24380) 1047 : cluster 0 pgmap v1222: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:19:13.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:12 smithi146 bash[23809]: cluster 2024-03-11T21:19:11.216747+0000 osd.4 (osd.4) 141 : cluster 0 4.9 scrub starts 2024-03-11T21:19:13.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:12 smithi146 bash[23809]: cluster 2024-03-11T21:19:11.219983+0000 osd.4 (osd.4) 142 : cluster 0 4.9 scrub ok 2024-03-11T21:19:13.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:12 smithi146 bash[23809]: audit 2024-03-11T21:19:11.818611+0000 mon.a (mon.0) 2102 : audit 1 from='client.? 172.21.15.23:0/3495271350' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-03-11T21:19:13.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:12 smithi146 bash[23809]: cluster 2024-03-11T21:19:11.828837+0000 mon.a (mon.0) 2103 : cluster 0 osdmap e491: 8 total, 8 up, 8 in 2024-03-11T21:19:13.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:12 smithi146 bash[23809]: cluster 2024-03-11T21:19:11.962149+0000 osd.1 (osd.1) 432 : cluster 0 3.4 scrub starts 2024-03-11T21:19:13.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:12 smithi146 bash[23809]: cluster 2024-03-11T21:19:11.965916+0000 osd.1 (osd.1) 433 : cluster 0 3.4 scrub ok 2024-03-11T21:19:13.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:12 smithi146 bash[23809]: cluster 2024-03-11T21:19:12.564019+0000 mgr.y (mgr.24380) 1047 : cluster 0 pgmap v1222: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:19:14.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:13 smithi146 bash[23809]: cluster 2024-03-11T21:19:12.646050+0000 osd.3 (osd.3) 616 : cluster 0 3.1e scrub starts 2024-03-11T21:19:14.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:13 smithi146 bash[23809]: cluster 2024-03-11T21:19:12.650381+0000 osd.3 (osd.3) 617 : cluster 0 3.1e scrub ok 2024-03-11T21:19:14.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:13 smithi146 bash[23809]: cluster 2024-03-11T21:19:13.458119+0000 osd.0 (osd.0) 504 : cluster 0 6.9 scrub starts 2024-03-11T21:19:14.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:13 smithi023 bash[21372]: cluster 2024-03-11T21:19:12.646050+0000 osd.3 (osd.3) 616 : cluster 0 3.1e scrub starts 2024-03-11T21:19:14.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:13 smithi023 bash[21372]: cluster 2024-03-11T21:19:12.650381+0000 osd.3 (osd.3) 617 : cluster 0 3.1e scrub ok 2024-03-11T21:19:14.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:13 smithi023 bash[21372]: cluster 2024-03-11T21:19:13.458119+0000 osd.0 (osd.0) 504 : cluster 0 6.9 scrub starts 2024-03-11T21:19:14.274 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:13 smithi023 bash[29146]: cluster 2024-03-11T21:19:12.646050+0000 osd.3 (osd.3) 616 : cluster 0 3.1e scrub starts 2024-03-11T21:19:14.274 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:13 smithi023 bash[29146]: cluster 2024-03-11T21:19:12.650381+0000 osd.3 (osd.3) 617 : cluster 0 3.1e scrub ok 2024-03-11T21:19:14.274 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:13 smithi023 bash[29146]: cluster 2024-03-11T21:19:13.458119+0000 osd.0 (osd.0) 504 : cluster 0 6.9 scrub starts 2024-03-11T21:19:15.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:14 smithi146 bash[23809]: cluster 2024-03-11T21:19:13.617694+0000 osd.0 (osd.0) 505 : cluster 0 6.9 scrub ok 2024-03-11T21:19:15.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:14 smithi146 bash[23809]: cluster 2024-03-11T21:19:13.637096+0000 osd.3 (osd.3) 618 : cluster 0 6.13 scrub starts 2024-03-11T21:19:15.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:14 smithi146 bash[23809]: cluster 2024-03-11T21:19:13.801296+0000 osd.3 (osd.3) 619 : cluster 0 6.13 scrub ok 2024-03-11T21:19:15.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:14 smithi146 bash[23809]: cluster 2024-03-11T21:19:14.345252+0000 osd.7 (osd.7) 693 : cluster 0 2.b scrub starts 2024-03-11T21:19:15.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:14 smithi146 bash[23809]: cluster 2024-03-11T21:19:14.347824+0000 osd.7 (osd.7) 694 : cluster 0 2.b scrub ok 2024-03-11T21:19:15.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:14 smithi146 bash[23809]: cluster 2024-03-11T21:19:14.457566+0000 osd.0 (osd.0) 506 : cluster 0 3.17 scrub starts 2024-03-11T21:19:15.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:14 smithi146 bash[23809]: cluster 2024-03-11T21:19:14.460861+0000 osd.0 (osd.0) 507 : cluster 0 3.17 scrub ok 2024-03-11T21:19:15.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:14 smithi146 bash[23809]: cluster 2024-03-11T21:19:14.564873+0000 mgr.y (mgr.24380) 1048 : cluster 0 pgmap v1223: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T21:19:15.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:14 smithi023 bash[21372]: cluster 2024-03-11T21:19:13.617694+0000 osd.0 (osd.0) 505 : cluster 0 6.9 scrub ok 2024-03-11T21:19:15.275 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:14 smithi023 bash[21372]: cluster 2024-03-11T21:19:13.637096+0000 osd.3 (osd.3) 618 : cluster 0 6.13 scrub starts 2024-03-11T21:19:15.275 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:14 smithi023 bash[21372]: cluster 2024-03-11T21:19:13.801296+0000 osd.3 (osd.3) 619 : cluster 0 6.13 scrub ok 2024-03-11T21:19:15.275 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:14 smithi023 bash[21372]: cluster 2024-03-11T21:19:14.345252+0000 osd.7 (osd.7) 693 : cluster 0 2.b scrub starts 2024-03-11T21:19:15.275 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:14 smithi023 bash[21372]: cluster 2024-03-11T21:19:14.347824+0000 osd.7 (osd.7) 694 : cluster 0 2.b scrub ok 2024-03-11T21:19:15.275 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:14 smithi023 bash[21372]: cluster 2024-03-11T21:19:14.457566+0000 osd.0 (osd.0) 506 : cluster 0 3.17 scrub starts 2024-03-11T21:19:15.275 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:14 smithi023 bash[21372]: cluster 2024-03-11T21:19:14.460861+0000 osd.0 (osd.0) 507 : cluster 0 3.17 scrub ok 2024-03-11T21:19:15.275 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:14 smithi023 bash[21372]: cluster 2024-03-11T21:19:14.564873+0000 mgr.y (mgr.24380) 1048 : cluster 0 pgmap v1223: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T21:19:15.275 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:14 smithi023 bash[29146]: cluster 2024-03-11T21:19:13.617694+0000 osd.0 (osd.0) 505 : cluster 0 6.9 scrub ok 2024-03-11T21:19:15.275 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:14 smithi023 bash[29146]: cluster 2024-03-11T21:19:13.637096+0000 osd.3 (osd.3) 618 : cluster 0 6.13 scrub starts 2024-03-11T21:19:15.275 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:14 smithi023 bash[29146]: cluster 2024-03-11T21:19:13.801296+0000 osd.3 (osd.3) 619 : cluster 0 6.13 scrub ok 2024-03-11T21:19:15.275 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:14 smithi023 bash[29146]: cluster 2024-03-11T21:19:14.345252+0000 osd.7 (osd.7) 693 : cluster 0 2.b scrub starts 2024-03-11T21:19:15.276 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:14 smithi023 bash[29146]: cluster 2024-03-11T21:19:14.347824+0000 osd.7 (osd.7) 694 : cluster 0 2.b scrub ok 2024-03-11T21:19:15.276 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:14 smithi023 bash[29146]: cluster 2024-03-11T21:19:14.457566+0000 osd.0 (osd.0) 506 : cluster 0 3.17 scrub starts 2024-03-11T21:19:15.276 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:14 smithi023 bash[29146]: cluster 2024-03-11T21:19:14.460861+0000 osd.0 (osd.0) 507 : cluster 0 3.17 scrub ok 2024-03-11T21:19:15.276 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:14 smithi023 bash[29146]: cluster 2024-03-11T21:19:14.564873+0000 mgr.y (mgr.24380) 1048 : cluster 0 pgmap v1223: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T21:19:16.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:15 smithi146 bash[23809]: cluster 2024-03-11T21:19:14.613775+0000 osd.3 (osd.3) 620 : cluster 0 3.7 scrub starts 2024-03-11T21:19:16.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:15 smithi146 bash[23809]: cluster 2024-03-11T21:19:14.618537+0000 osd.3 (osd.3) 621 : cluster 0 3.7 scrub ok 2024-03-11T21:19:16.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:15 smithi146 bash[23809]: cluster 2024-03-11T21:19:14.663788+0000 osd.5 (osd.5) 669 : cluster 0 3.16 scrub starts 2024-03-11T21:19:16.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:15 smithi146 bash[23809]: cluster 2024-03-11T21:19:14.667578+0000 osd.5 (osd.5) 670 : cluster 0 3.16 scrub ok 2024-03-11T21:19:16.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:15 smithi146 bash[23809]: cluster 2024-03-11T21:19:14.923792+0000 osd.1 (osd.1) 434 : cluster 0 4.f scrub starts 2024-03-11T21:19:16.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:15 smithi146 bash[23809]: cluster 2024-03-11T21:19:14.926683+0000 osd.1 (osd.1) 435 : cluster 0 4.f scrub ok 2024-03-11T21:19:16.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:15 smithi146 bash[23809]: cluster 2024-03-11T21:19:15.393831+0000 osd.6 (osd.6) 552 : cluster 0 1.6 deep-scrub starts 2024-03-11T21:19:16.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:15 smithi146 bash[23809]: cluster 2024-03-11T21:19:15.396633+0000 osd.6 (osd.6) 553 : cluster 0 1.6 deep-scrub ok 2024-03-11T21:19:16.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:15 smithi023 bash[21372]: cluster 2024-03-11T21:19:14.613775+0000 osd.3 (osd.3) 620 : cluster 0 3.7 scrub starts 2024-03-11T21:19:16.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:15 smithi023 bash[21372]: cluster 2024-03-11T21:19:14.618537+0000 osd.3 (osd.3) 621 : cluster 0 3.7 scrub ok 2024-03-11T21:19:16.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:15 smithi023 bash[21372]: cluster 2024-03-11T21:19:14.663788+0000 osd.5 (osd.5) 669 : cluster 0 3.16 scrub starts 2024-03-11T21:19:16.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:15 smithi023 bash[21372]: cluster 2024-03-11T21:19:14.667578+0000 osd.5 (osd.5) 670 : cluster 0 3.16 scrub ok 2024-03-11T21:19:16.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:15 smithi023 bash[21372]: cluster 2024-03-11T21:19:14.923792+0000 osd.1 (osd.1) 434 : cluster 0 4.f scrub starts 2024-03-11T21:19:16.275 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:15 smithi023 bash[21372]: cluster 2024-03-11T21:19:14.926683+0000 osd.1 (osd.1) 435 : cluster 0 4.f scrub ok 2024-03-11T21:19:16.275 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:15 smithi023 bash[21372]: cluster 2024-03-11T21:19:15.393831+0000 osd.6 (osd.6) 552 : cluster 0 1.6 deep-scrub starts 2024-03-11T21:19:16.275 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:15 smithi023 bash[21372]: cluster 2024-03-11T21:19:15.396633+0000 osd.6 (osd.6) 553 : cluster 0 1.6 deep-scrub ok 2024-03-11T21:19:16.275 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:15 smithi023 bash[29146]: cluster 2024-03-11T21:19:14.613775+0000 osd.3 (osd.3) 620 : cluster 0 3.7 scrub starts 2024-03-11T21:19:16.275 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:15 smithi023 bash[29146]: cluster 2024-03-11T21:19:14.618537+0000 osd.3 (osd.3) 621 : cluster 0 3.7 scrub ok 2024-03-11T21:19:16.275 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:15 smithi023 bash[29146]: cluster 2024-03-11T21:19:14.663788+0000 osd.5 (osd.5) 669 : cluster 0 3.16 scrub starts 2024-03-11T21:19:16.275 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:15 smithi023 bash[29146]: cluster 2024-03-11T21:19:14.667578+0000 osd.5 (osd.5) 670 : cluster 0 3.16 scrub ok 2024-03-11T21:19:16.275 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:15 smithi023 bash[29146]: cluster 2024-03-11T21:19:14.923792+0000 osd.1 (osd.1) 434 : cluster 0 4.f scrub starts 2024-03-11T21:19:16.275 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:15 smithi023 bash[29146]: cluster 2024-03-11T21:19:14.926683+0000 osd.1 (osd.1) 435 : cluster 0 4.f scrub ok 2024-03-11T21:19:16.276 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:15 smithi023 bash[29146]: cluster 2024-03-11T21:19:15.393831+0000 osd.6 (osd.6) 552 : cluster 0 1.6 deep-scrub starts 2024-03-11T21:19:16.276 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:15 smithi023 bash[29146]: cluster 2024-03-11T21:19:15.396633+0000 osd.6 (osd.6) 553 : cluster 0 1.6 deep-scrub ok 2024-03-11T21:19:16.752 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:19:16 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:19:17.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:16 smithi146 bash[23809]: audit 2024-03-11T21:19:16.303588+0000 mgr.y (mgr.24380) 1049 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:19:17.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:16 smithi146 bash[23809]: cluster 2024-03-11T21:19:16.566390+0000 mgr.y (mgr.24380) 1050 : cluster 0 pgmap v1224: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:19:17.274 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:16 smithi023 bash[29146]: audit 2024-03-11T21:19:16.303588+0000 mgr.y (mgr.24380) 1049 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:19:17.274 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:16 smithi023 bash[29146]: cluster 2024-03-11T21:19:16.566390+0000 mgr.y (mgr.24380) 1050 : cluster 0 pgmap v1224: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:19:17.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:16 smithi023 bash[21372]: audit 2024-03-11T21:19:16.303588+0000 mgr.y (mgr.24380) 1049 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:19:17.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:16 smithi023 bash[21372]: cluster 2024-03-11T21:19:16.566390+0000 mgr.y (mgr.24380) 1050 : cluster 0 pgmap v1224: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:19:17.483 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:19:18.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:17 smithi146 bash[23809]: cluster 2024-03-11T21:19:17.340691+0000 osd.7 (osd.7) 695 : cluster 0 1.9 scrub starts 2024-03-11T21:19:18.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:17 smithi146 bash[23809]: cluster 2024-03-11T21:19:17.343868+0000 osd.7 (osd.7) 696 : cluster 0 1.9 scrub ok 2024-03-11T21:19:18.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:17 smithi146 bash[23809]: cluster 2024-03-11T21:19:17.443145+0000 osd.6 (osd.6) 554 : cluster 0 1.7 scrub starts 2024-03-11T21:19:18.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:17 smithi146 bash[23809]: cluster 2024-03-11T21:19:17.445613+0000 osd.6 (osd.6) 555 : cluster 0 1.7 scrub ok 2024-03-11T21:19:18.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:17 smithi023 bash[21372]: cluster 2024-03-11T21:19:17.340691+0000 osd.7 (osd.7) 695 : cluster 0 1.9 scrub starts 2024-03-11T21:19:18.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:17 smithi023 bash[21372]: cluster 2024-03-11T21:19:17.343868+0000 osd.7 (osd.7) 696 : cluster 0 1.9 scrub ok 2024-03-11T21:19:18.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:17 smithi023 bash[21372]: cluster 2024-03-11T21:19:17.443145+0000 osd.6 (osd.6) 554 : cluster 0 1.7 scrub starts 2024-03-11T21:19:18.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:17 smithi023 bash[21372]: cluster 2024-03-11T21:19:17.445613+0000 osd.6 (osd.6) 555 : cluster 0 1.7 scrub ok 2024-03-11T21:19:18.274 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:17 smithi023 bash[29146]: cluster 2024-03-11T21:19:17.340691+0000 osd.7 (osd.7) 695 : cluster 0 1.9 scrub starts 2024-03-11T21:19:18.274 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:17 smithi023 bash[29146]: cluster 2024-03-11T21:19:17.343868+0000 osd.7 (osd.7) 696 : cluster 0 1.9 scrub ok 2024-03-11T21:19:18.274 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:17 smithi023 bash[29146]: cluster 2024-03-11T21:19:17.443145+0000 osd.6 (osd.6) 554 : cluster 0 1.7 scrub starts 2024-03-11T21:19:18.274 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:17 smithi023 bash[29146]: cluster 2024-03-11T21:19:17.445613+0000 osd.6 (osd.6) 555 : cluster 0 1.7 scrub ok 2024-03-11T21:19:18.718 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:19:18 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:19:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:19:19.000 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:18 smithi023 bash[21372]: cluster 2024-03-11T21:19:18.567225+0000 mgr.y (mgr.24380) 1051 : cluster 0 pgmap v1225: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:19:19.000 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:18 smithi023 bash[29146]: cluster 2024-03-11T21:19:18.567225+0000 mgr.y (mgr.24380) 1051 : cluster 0 pgmap v1225: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:19:19.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:18 smithi146 bash[23809]: cluster 2024-03-11T21:19:18.567225+0000 mgr.y (mgr.24380) 1051 : cluster 0 pgmap v1225: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:19:20.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:19 smithi023 bash[21372]: cluster 2024-03-11T21:19:18.524958+0000 osd.3 (osd.3) 622 : cluster 0 6.11 scrub starts 2024-03-11T21:19:20.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:19 smithi023 bash[21372]: cluster 2024-03-11T21:19:18.567499+0000 osd.3 (osd.3) 623 : cluster 0 6.11 scrub ok 2024-03-11T21:19:20.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:19 smithi023 bash[21372]: cluster 2024-03-11T21:19:18.716208+0000 osd.5 (osd.5) 671 : cluster 0 1.12 scrub starts 2024-03-11T21:19:20.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:19 smithi023 bash[21372]: cluster 2024-03-11T21:19:18.719004+0000 osd.5 (osd.5) 672 : cluster 0 1.12 scrub ok 2024-03-11T21:19:20.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:19 smithi023 bash[29146]: cluster 2024-03-11T21:19:18.524958+0000 osd.3 (osd.3) 622 : cluster 0 6.11 scrub starts 2024-03-11T21:19:20.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:19 smithi023 bash[29146]: cluster 2024-03-11T21:19:18.567499+0000 osd.3 (osd.3) 623 : cluster 0 6.11 scrub ok 2024-03-11T21:19:20.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:19 smithi023 bash[29146]: cluster 2024-03-11T21:19:18.716208+0000 osd.5 (osd.5) 671 : cluster 0 1.12 scrub starts 2024-03-11T21:19:20.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:19 smithi023 bash[29146]: cluster 2024-03-11T21:19:18.719004+0000 osd.5 (osd.5) 672 : cluster 0 1.12 scrub ok 2024-03-11T21:19:20.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:19 smithi146 bash[23809]: cluster 2024-03-11T21:19:18.524958+0000 osd.3 (osd.3) 622 : cluster 0 6.11 scrub starts 2024-03-11T21:19:20.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:19 smithi146 bash[23809]: cluster 2024-03-11T21:19:18.567499+0000 osd.3 (osd.3) 623 : cluster 0 6.11 scrub ok 2024-03-11T21:19:20.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:19 smithi146 bash[23809]: cluster 2024-03-11T21:19:18.716208+0000 osd.5 (osd.5) 671 : cluster 0 1.12 scrub starts 2024-03-11T21:19:20.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:19 smithi146 bash[23809]: cluster 2024-03-11T21:19:18.719004+0000 osd.5 (osd.5) 672 : cluster 0 1.12 scrub ok 2024-03-11T21:19:20.790 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-11T21:19:20.791 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":491,"fsid":"67ba00f8-dfe7-11ee-95c8-87774f69a715","created":"2024-03-11T20:40:13.664802+0000","modified":"2024-03-11T21:19:11.812614+0000","last_up_change":"2024-03-11T21:00:47.324817+0000","last_in_change":"2024-03-11T21:14:49.940617+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":20,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T20:45:31.964570+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":21,"pg_placement_num":21,"pg_placement_num_target":21,"pg_num_target":21,"pg_num_pending":21,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"454","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"444","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8999999761581421,"score_stable":1.8999999761581421,"optimal_score":1,"raw_score_acting":1.8999999761581421,"raw_score_stable":1.8999999761581421,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-03-11T20:50:26.421486+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"default.rgw.log","create_time":"2024-03-11T20:50:28.592615+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.control","create_time":"2024-03-11T20:50:30.536371+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"datapool","create_time":"2024-03-11T20:50:31.135845+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":87,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-03-11T20:50:33.233163+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3df72b4a-8fe4-4265-b4c6-af022e1e23e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":449,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":2154627393},{"type":"v1","addr":"172.21.15.23:6803","nonce":2154627393}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":2154627393},{"type":"v1","addr":"172.21.15.23:6805","nonce":2154627393}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":2154627393},{"type":"v1","addr":"172.21.15.23:6809","nonce":2154627393}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":2154627393},{"type":"v1","addr":"172.21.15.23:6807","nonce":2154627393}]},"public_addr":"172.21.15.23:6803/2154627393","cluster_addr":"172.21.15.23:6805/2154627393","heartbeat_back_addr":"172.21.15.23:6809/2154627393","heartbeat_front_addr":"172.21.15.23:6807/2154627393","state":["exists","up"]},{"osd":1,"uuid":"e8fd7500-9081-41a1-8077-a4d0f0e12a2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":173,"up_from":211,"up_thru":449,"down_at":174,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":56585354},{"type":"v1","addr":"172.21.15.23:6811","nonce":56585354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":56585354},{"type":"v1","addr":"172.21.15.23:6813","nonce":56585354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":56585354},{"type":"v1","addr":"172.21.15.23:6817","nonce":56585354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":56585354},{"type":"v1","addr":"172.21.15.23:6815","nonce":56585354}]},"public_addr":"172.21.15.23:6811/56585354","cluster_addr":"172.21.15.23:6813/56585354","heartbeat_back_addr":"172.21.15.23:6817/56585354","heartbeat_front_addr":"172.21.15.23:6815/56585354","state":["exists","up"]},{"osd":2,"uuid":"dd85d52a-df48-4b03-8612-cecfd82b5987","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":465,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":1770748056},{"type":"v1","addr":"172.21.15.23:6819","nonce":1770748056}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":1770748056},{"type":"v1","addr":"172.21.15.23:6821","nonce":1770748056}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":1770748056},{"type":"v1","addr":"172.21.15.23:6825","nonce":1770748056}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":1770748056},{"type":"v1","addr":"172.21.15.23:6823","nonce":1770748056}]},"public_addr":"172.21.15.23:6819/1770748056","cluster_addr":"172.21.15.23:6821/1770748056","heartbeat_back_addr":"172.21.15.23:6825/1770748056","heartbeat_front_addr":"172.21.15.23:6823/1770748056","state":["exists","up"]},{"osd":3,"uuid":"e3021106-3779-4b94-9bbf-a7724544110f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":28,"up_thru":449,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":938496814},{"type":"v1","addr":"172.21.15.23:6827","nonce":938496814}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":938496814},{"type":"v1","addr":"172.21.15.23:6829","nonce":938496814}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":938496814},{"type":"v1","addr":"172.21.15.23:6833","nonce":938496814}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":938496814},{"type":"v1","addr":"172.21.15.23:6831","nonce":938496814}]},"public_addr":"172.21.15.23:6827/938496814","cluster_addr":"172.21.15.23:6829/938496814","heartbeat_back_addr":"172.21.15.23:6833/938496814","heartbeat_front_addr":"172.21.15.23:6831/938496814","state":["exists","up"]},{"osd":4,"uuid":"4dcbc91e-7035-413d-97bc-c9d804eb16d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":35,"last_clean_end":182,"up_from":204,"up_thru":485,"down_at":183,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":3858359434},{"type":"v1","addr":"172.21.15.146:6801","nonce":3858359434}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":3858359434},{"type":"v1","addr":"172.21.15.146:6803","nonce":3858359434}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":3858359434},{"type":"v1","addr":"172.21.15.146:6807","nonce":3858359434}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":3858359434},{"type":"v1","addr":"172.21.15.146:6805","nonce":3858359434}]},"public_addr":"172.21.15.146:6801/3858359434","cluster_addr":"172.21.15.146:6803/3858359434","heartbeat_back_addr":"172.21.15.146:6807/3858359434","heartbeat_front_addr":"172.21.15.146:6805/3858359434","state":["exists","up"]},{"osd":5,"uuid":"aefcb766-8bae-4ff4-b71e-6691c8ac02f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":449,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":4003230814},{"type":"v1","addr":"172.21.15.146:6809","nonce":4003230814}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":4003230814},{"type":"v1","addr":"172.21.15.146:6811","nonce":4003230814}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":4003230814},{"type":"v1","addr":"172.21.15.146:6815","nonce":4003230814}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":4003230814},{"type":"v1","addr":"172.21.15.146:6813","nonce":4003230814}]},"public_addr":"172.21.15.146:6809/4003230814","cluster_addr":"172.21.15.146:6811/4003230814","heartbeat_back_addr":"172.21.15.146:6815/4003230814","heartbeat_front_addr":"172.21.15.146:6813/4003230814","state":["exists","up"]},{"osd":6,"uuid":"5cb151e8-1f9b-4617-8a86-adfb06bb9606","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":449,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":3334092376},{"type":"v1","addr":"172.21.15.146:6817","nonce":3334092376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3334092376},{"type":"v1","addr":"172.21.15.146:6819","nonce":3334092376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":3334092376},{"type":"v1","addr":"172.21.15.146:6823","nonce":3334092376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3334092376},{"type":"v1","addr":"172.21.15.146:6821","nonce":3334092376}]},"public_addr":"172.21.15.146:6817/3334092376","cluster_addr":"172.21.15.146:6819/3334092376","heartbeat_back_addr":"172.21.15.146:6823/3334092376","heartbeat_front_addr":"172.21.15.146:6821/3334092376","state":["exists","up"]},{"osd":7,"uuid":"bdbd11a2-fa09-4f82-93a2-6b2d9c750bfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":449,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":1678407923},{"type":"v1","addr":"172.21.15.146:6825","nonce":1678407923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":1678407923},{"type":"v1","addr":"172.21.15.146:6827","nonce":1678407923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":1678407923},{"type":"v1","addr":"172.21.15.146:6831","nonce":1678407923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":1678407923},{"type":"v1","addr":"172.21.15.146:6829","nonce":1678407923}]},"public_addr":"172.21.15.146:6825/1678407923","cluster_addr":"172.21.15.146:6827/1678407923","heartbeat_back_addr":"172.21.15.146:6831/1678407923","heartbeat_front_addr":"172.21.15.146:6829/1678407923","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:18:34.265472+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:19:03.927791+0000","dead_epoch":173},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:18:57.024592+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:18:54.176379+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:18:21.370975+0000","dead_epoch":182},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:18:20.506687+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:18:53.471898+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:18:18.673940+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":7,"to":2}]},{"pgid":"4.b","mappings":[{"from":1,"to":2}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]},{"pgid":"4.e","mappings":[{"from":1,"to":2}]},{"pgid":"4.16","mappings":[{"from":3,"to":2}]},{"pgid":"4.1d","mappings":[{"from":4,"to":2}]},{"pgid":"6.a","mappings":[{"from":6,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:0/1618334186":"2024-03-12T20:40:45.361537+0000","172.21.15.23:6801/2813578081":"2024-03-12T20:40:45.361537+0000","172.21.15.23:0/2869940180":"2024-03-12T20:51:33.790205+0000","172.21.15.23:6800/3037564930":"2024-03-12T20:51:33.790205+0000","172.21.15.23:0/3297567084":"2024-03-12T20:51:33.790205+0000","172.21.15.23:0/3708753659":"2024-03-12T20:40:45.361537+0000","172.21.15.23:0/2004146740":"2024-03-12T20:40:45.361537+0000","172.21.15.23:0/827653812":"2024-03-12T20:41:21.301516+0000","172.21.15.23:6800/1057699850":"2024-03-12T20:41:21.301516+0000","172.21.15.23:6800/2813578081":"2024-03-12T20:40:45.361537+0000","172.21.15.23:6801/1057699850":"2024-03-12T20:41:21.301516+0000","172.21.15.23:0/3833230375":"2024-03-12T20:41:21.301516+0000","172.21.15.23:6801/3037564930":"2024-03-12T20:51:33.790205+0000","172.21.15.23:0/1182809102":"2024-03-12T20:51:33.790205+0000","172.21.15.23:0/2122134187":"2024-03-12T20:51:33.790205+0000","172.21.15.23:0/2356701899":"2024-03-12T20:41:21.301516+0000","172.21.15.23:0/843974100":"2024-03-12T20:51:33.790205+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T21:19:21.608 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-11T21:19:21.608 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd dump --format=json 2024-03-11T21:19:21.752 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:21 smithi146 bash[23809]: cluster 2024-03-11T21:19:19.529416+0000 osd.6 (osd.6) 556 : cluster 0 3.a scrub starts 2024-03-11T21:19:21.752 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:21 smithi146 bash[23809]: cluster 2024-03-11T21:19:19.534443+0000 osd.6 (osd.6) 557 : cluster 0 3.a scrub ok 2024-03-11T21:19:21.753 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:21 smithi146 bash[23809]: cluster 2024-03-11T21:19:20.567824+0000 mgr.y (mgr.24380) 1052 : cluster 0 pgmap v1226: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:19:21.753 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:21 smithi146 bash[23809]: audit 2024-03-11T21:19:20.800977+0000 mon.c (mon.2) 113 : audit 0 from='client.? 172.21.15.23:0/96740076' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T21:19:21.753 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:21 smithi146 bash[23809]: cluster 2024-03-11T21:19:20.893420+0000 osd.1 (osd.1) 436 : cluster 0 4.7 scrub starts 2024-03-11T21:19:21.753 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:21 smithi146 bash[23809]: cluster 2024-03-11T21:19:20.895210+0000 osd.1 (osd.1) 437 : cluster 0 4.7 scrub ok 2024-03-11T21:19:21.774 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:21 smithi023 bash[21372]: cluster 2024-03-11T21:19:19.529416+0000 osd.6 (osd.6) 556 : cluster 0 3.a scrub starts 2024-03-11T21:19:21.774 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:21 smithi023 bash[21372]: cluster 2024-03-11T21:19:19.534443+0000 osd.6 (osd.6) 557 : cluster 0 3.a scrub ok 2024-03-11T21:19:21.774 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:21 smithi023 bash[21372]: cluster 2024-03-11T21:19:20.567824+0000 mgr.y (mgr.24380) 1052 : cluster 0 pgmap v1226: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:19:21.774 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:21 smithi023 bash[21372]: audit 2024-03-11T21:19:20.800977+0000 mon.c (mon.2) 113 : audit 0 from='client.? 172.21.15.23:0/96740076' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T21:19:21.774 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:21 smithi023 bash[21372]: cluster 2024-03-11T21:19:20.893420+0000 osd.1 (osd.1) 436 : cluster 0 4.7 scrub starts 2024-03-11T21:19:21.774 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:21 smithi023 bash[21372]: cluster 2024-03-11T21:19:20.895210+0000 osd.1 (osd.1) 437 : cluster 0 4.7 scrub ok 2024-03-11T21:19:21.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:21 smithi023 bash[29146]: cluster 2024-03-11T21:19:19.529416+0000 osd.6 (osd.6) 556 : cluster 0 3.a scrub starts 2024-03-11T21:19:21.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:21 smithi023 bash[29146]: cluster 2024-03-11T21:19:19.534443+0000 osd.6 (osd.6) 557 : cluster 0 3.a scrub ok 2024-03-11T21:19:21.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:21 smithi023 bash[29146]: cluster 2024-03-11T21:19:20.567824+0000 mgr.y (mgr.24380) 1052 : cluster 0 pgmap v1226: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:19:21.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:21 smithi023 bash[29146]: audit 2024-03-11T21:19:20.800977+0000 mon.c (mon.2) 113 : audit 0 from='client.? 172.21.15.23:0/96740076' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T21:19:21.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:21 smithi023 bash[29146]: cluster 2024-03-11T21:19:20.893420+0000 osd.1 (osd.1) 436 : cluster 0 4.7 scrub starts 2024-03-11T21:19:21.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:21 smithi023 bash[29146]: cluster 2024-03-11T21:19:20.895210+0000 osd.1 (osd.1) 437 : cluster 0 4.7 scrub ok 2024-03-11T21:19:22.774 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:22 smithi023 bash[21372]: cluster 2024-03-11T21:19:20.562928+0000 osd.6 (osd.6) 558 : cluster 0 2.17 scrub starts 2024-03-11T21:19:22.774 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:22 smithi023 bash[21372]: cluster 2024-03-11T21:19:20.564683+0000 osd.6 (osd.6) 559 : cluster 0 2.17 scrub ok 2024-03-11T21:19:22.774 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:22 smithi023 bash[21372]: cluster 2024-03-11T21:19:21.301863+0000 osd.4 (osd.4) 143 : cluster 0 6.1e scrub starts 2024-03-11T21:19:22.774 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:22 smithi023 bash[21372]: cluster 2024-03-11T21:19:21.380333+0000 osd.4 (osd.4) 144 : cluster 0 6.1e scrub ok 2024-03-11T21:19:22.774 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:22 smithi023 bash[21372]: cluster 2024-03-11T21:19:21.515180+0000 osd.3 (osd.3) 624 : cluster 0 2.1e scrub starts 2024-03-11T21:19:22.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:22 smithi023 bash[21372]: cluster 2024-03-11T21:19:21.518339+0000 osd.3 (osd.3) 625 : cluster 0 2.1e scrub ok 2024-03-11T21:19:22.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:22 smithi023 bash[21372]: audit 2024-03-11T21:19:22.053172+0000 mon.a (mon.0) 2104 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:19:22.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:22 smithi023 bash[29146]: cluster 2024-03-11T21:19:20.562928+0000 osd.6 (osd.6) 558 : cluster 0 2.17 scrub starts 2024-03-11T21:19:22.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:22 smithi023 bash[29146]: cluster 2024-03-11T21:19:20.564683+0000 osd.6 (osd.6) 559 : cluster 0 2.17 scrub ok 2024-03-11T21:19:22.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:22 smithi023 bash[29146]: cluster 2024-03-11T21:19:21.301863+0000 osd.4 (osd.4) 143 : cluster 0 6.1e scrub starts 2024-03-11T21:19:22.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:22 smithi023 bash[29146]: cluster 2024-03-11T21:19:21.380333+0000 osd.4 (osd.4) 144 : cluster 0 6.1e scrub ok 2024-03-11T21:19:22.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:22 smithi023 bash[29146]: cluster 2024-03-11T21:19:21.515180+0000 osd.3 (osd.3) 624 : cluster 0 2.1e scrub starts 2024-03-11T21:19:22.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:22 smithi023 bash[29146]: cluster 2024-03-11T21:19:21.518339+0000 osd.3 (osd.3) 625 : cluster 0 2.1e scrub ok 2024-03-11T21:19:22.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:22 smithi023 bash[29146]: audit 2024-03-11T21:19:22.053172+0000 mon.a (mon.0) 2104 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:19:23.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:22 smithi146 bash[23809]: cluster 2024-03-11T21:19:20.562928+0000 osd.6 (osd.6) 558 : cluster 0 2.17 scrub starts 2024-03-11T21:19:23.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:22 smithi146 bash[23809]: cluster 2024-03-11T21:19:20.564683+0000 osd.6 (osd.6) 559 : cluster 0 2.17 scrub ok 2024-03-11T21:19:23.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:22 smithi146 bash[23809]: cluster 2024-03-11T21:19:21.301863+0000 osd.4 (osd.4) 143 : cluster 0 6.1e scrub starts 2024-03-11T21:19:23.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:22 smithi146 bash[23809]: cluster 2024-03-11T21:19:21.380333+0000 osd.4 (osd.4) 144 : cluster 0 6.1e scrub ok 2024-03-11T21:19:23.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:22 smithi146 bash[23809]: cluster 2024-03-11T21:19:21.515180+0000 osd.3 (osd.3) 624 : cluster 0 2.1e scrub starts 2024-03-11T21:19:23.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:22 smithi146 bash[23809]: cluster 2024-03-11T21:19:21.518339+0000 osd.3 (osd.3) 625 : cluster 0 2.1e scrub ok 2024-03-11T21:19:23.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:22 smithi146 bash[23809]: audit 2024-03-11T21:19:22.053172+0000 mon.a (mon.0) 2104 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:19:23.774 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:23 smithi023 bash[21372]: cluster 2024-03-11T21:19:22.264068+0000 osd.4 (osd.4) 145 : cluster 0 4.c scrub starts 2024-03-11T21:19:23.774 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:23 smithi023 bash[21372]: cluster 2024-03-11T21:19:22.266660+0000 osd.4 (osd.4) 146 : cluster 0 4.c scrub ok 2024-03-11T21:19:23.774 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:23 smithi023 bash[21372]: cluster 2024-03-11T21:19:22.382192+0000 osd.0 (osd.0) 508 : cluster 0 4.b deep-scrub starts 2024-03-11T21:19:23.774 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:23 smithi023 bash[21372]: cluster 2024-03-11T21:19:22.384663+0000 osd.0 (osd.0) 509 : cluster 0 4.b deep-scrub ok 2024-03-11T21:19:23.774 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:23 smithi023 bash[21372]: cluster 2024-03-11T21:19:22.569086+0000 mgr.y (mgr.24380) 1053 : cluster 0 pgmap v1227: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 952 B/s rd, 0 op/s 2024-03-11T21:19:23.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:23 smithi023 bash[29146]: cluster 2024-03-11T21:19:22.264068+0000 osd.4 (osd.4) 145 : cluster 0 4.c scrub starts 2024-03-11T21:19:23.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:23 smithi023 bash[29146]: cluster 2024-03-11T21:19:22.266660+0000 osd.4 (osd.4) 146 : cluster 0 4.c scrub ok 2024-03-11T21:19:23.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:23 smithi023 bash[29146]: cluster 2024-03-11T21:19:22.382192+0000 osd.0 (osd.0) 508 : cluster 0 4.b deep-scrub starts 2024-03-11T21:19:23.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:23 smithi023 bash[29146]: cluster 2024-03-11T21:19:22.384663+0000 osd.0 (osd.0) 509 : cluster 0 4.b deep-scrub ok 2024-03-11T21:19:23.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:23 smithi023 bash[29146]: cluster 2024-03-11T21:19:22.569086+0000 mgr.y (mgr.24380) 1053 : cluster 0 pgmap v1227: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 952 B/s rd, 0 op/s 2024-03-11T21:19:24.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:23 smithi146 bash[23809]: cluster 2024-03-11T21:19:22.264068+0000 osd.4 (osd.4) 145 : cluster 0 4.c scrub starts 2024-03-11T21:19:24.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:23 smithi146 bash[23809]: cluster 2024-03-11T21:19:22.266660+0000 osd.4 (osd.4) 146 : cluster 0 4.c scrub ok 2024-03-11T21:19:24.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:23 smithi146 bash[23809]: cluster 2024-03-11T21:19:22.382192+0000 osd.0 (osd.0) 508 : cluster 0 4.b deep-scrub starts 2024-03-11T21:19:24.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:23 smithi146 bash[23809]: cluster 2024-03-11T21:19:22.384663+0000 osd.0 (osd.0) 509 : cluster 0 4.b deep-scrub ok 2024-03-11T21:19:24.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:23 smithi146 bash[23809]: cluster 2024-03-11T21:19:22.569086+0000 mgr.y (mgr.24380) 1053 : cluster 0 pgmap v1227: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 952 B/s rd, 0 op/s 2024-03-11T21:19:24.774 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:24 smithi023 bash[21372]: cluster 2024-03-11T21:19:22.729749+0000 osd.5 (osd.5) 673 : cluster 0 3.5 scrub starts 2024-03-11T21:19:24.774 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:24 smithi023 bash[21372]: cluster 2024-03-11T21:19:22.734604+0000 osd.5 (osd.5) 674 : cluster 0 3.5 scrub ok 2024-03-11T21:19:24.774 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:24 smithi023 bash[21372]: cluster 2024-03-11T21:19:23.421526+0000 osd.0 (osd.0) 510 : cluster 0 6.16 scrub starts 2024-03-11T21:19:24.774 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:24 smithi023 bash[21372]: cluster 2024-03-11T21:19:23.504270+0000 osd.0 (osd.0) 511 : cluster 0 6.16 scrub ok 2024-03-11T21:19:24.774 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:24 smithi023 bash[21372]: cluster 2024-03-11T21:19:23.525128+0000 osd.3 (osd.3) 626 : cluster 0 3.2 scrub starts 2024-03-11T21:19:24.774 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:24 smithi023 bash[21372]: cluster 2024-03-11T21:19:23.530789+0000 osd.3 (osd.3) 627 : cluster 0 3.2 scrub ok 2024-03-11T21:19:24.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:24 smithi023 bash[29146]: cluster 2024-03-11T21:19:22.729749+0000 osd.5 (osd.5) 673 : cluster 0 3.5 scrub starts 2024-03-11T21:19:24.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:24 smithi023 bash[29146]: cluster 2024-03-11T21:19:22.734604+0000 osd.5 (osd.5) 674 : cluster 0 3.5 scrub ok 2024-03-11T21:19:24.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:24 smithi023 bash[29146]: cluster 2024-03-11T21:19:23.421526+0000 osd.0 (osd.0) 510 : cluster 0 6.16 scrub starts 2024-03-11T21:19:24.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:24 smithi023 bash[29146]: cluster 2024-03-11T21:19:23.504270+0000 osd.0 (osd.0) 511 : cluster 0 6.16 scrub ok 2024-03-11T21:19:24.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:24 smithi023 bash[29146]: cluster 2024-03-11T21:19:23.525128+0000 osd.3 (osd.3) 626 : cluster 0 3.2 scrub starts 2024-03-11T21:19:24.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:24 smithi023 bash[29146]: cluster 2024-03-11T21:19:23.530789+0000 osd.3 (osd.3) 627 : cluster 0 3.2 scrub ok 2024-03-11T21:19:25.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:24 smithi146 bash[23809]: cluster 2024-03-11T21:19:22.729749+0000 osd.5 (osd.5) 673 : cluster 0 3.5 scrub starts 2024-03-11T21:19:25.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:24 smithi146 bash[23809]: cluster 2024-03-11T21:19:22.734604+0000 osd.5 (osd.5) 674 : cluster 0 3.5 scrub ok 2024-03-11T21:19:25.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:24 smithi146 bash[23809]: cluster 2024-03-11T21:19:23.421526+0000 osd.0 (osd.0) 510 : cluster 0 6.16 scrub starts 2024-03-11T21:19:25.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:24 smithi146 bash[23809]: cluster 2024-03-11T21:19:23.504270+0000 osd.0 (osd.0) 511 : cluster 0 6.16 scrub ok 2024-03-11T21:19:25.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:24 smithi146 bash[23809]: cluster 2024-03-11T21:19:23.525128+0000 osd.3 (osd.3) 626 : cluster 0 3.2 scrub starts 2024-03-11T21:19:25.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:24 smithi146 bash[23809]: cluster 2024-03-11T21:19:23.530789+0000 osd.3 (osd.3) 627 : cluster 0 3.2 scrub ok 2024-03-11T21:19:26.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:25 smithi146 bash[23809]: cluster 2024-03-11T21:19:24.393501+0000 osd.0 (osd.0) 512 : cluster 0 3.1 scrub starts 2024-03-11T21:19:26.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:25 smithi146 bash[23809]: cluster 2024-03-11T21:19:24.399040+0000 osd.0 (osd.0) 513 : cluster 0 3.1 scrub ok 2024-03-11T21:19:26.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:25 smithi146 bash[23809]: cluster 2024-03-11T21:19:24.534614+0000 osd.3 (osd.3) 628 : cluster 0 1.4 scrub starts 2024-03-11T21:19:26.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:25 smithi146 bash[23809]: cluster 2024-03-11T21:19:24.537503+0000 osd.3 (osd.3) 629 : cluster 0 1.4 scrub ok 2024-03-11T21:19:26.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:25 smithi146 bash[23809]: cluster 2024-03-11T21:19:24.569976+0000 mgr.y (mgr.24380) 1054 : cluster 0 pgmap v1228: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:19:26.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:25 smithi023 bash[21372]: cluster 2024-03-11T21:19:24.393501+0000 osd.0 (osd.0) 512 : cluster 0 3.1 scrub starts 2024-03-11T21:19:26.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:25 smithi023 bash[21372]: cluster 2024-03-11T21:19:24.399040+0000 osd.0 (osd.0) 513 : cluster 0 3.1 scrub ok 2024-03-11T21:19:26.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:25 smithi023 bash[21372]: cluster 2024-03-11T21:19:24.534614+0000 osd.3 (osd.3) 628 : cluster 0 1.4 scrub starts 2024-03-11T21:19:26.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:25 smithi023 bash[21372]: cluster 2024-03-11T21:19:24.537503+0000 osd.3 (osd.3) 629 : cluster 0 1.4 scrub ok 2024-03-11T21:19:26.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:25 smithi023 bash[21372]: cluster 2024-03-11T21:19:24.569976+0000 mgr.y (mgr.24380) 1054 : cluster 0 pgmap v1228: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:19:26.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:25 smithi023 bash[29146]: cluster 2024-03-11T21:19:24.393501+0000 osd.0 (osd.0) 512 : cluster 0 3.1 scrub starts 2024-03-11T21:19:26.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:25 smithi023 bash[29146]: cluster 2024-03-11T21:19:24.399040+0000 osd.0 (osd.0) 513 : cluster 0 3.1 scrub ok 2024-03-11T21:19:26.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:25 smithi023 bash[29146]: cluster 2024-03-11T21:19:24.534614+0000 osd.3 (osd.3) 628 : cluster 0 1.4 scrub starts 2024-03-11T21:19:26.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:25 smithi023 bash[29146]: cluster 2024-03-11T21:19:24.537503+0000 osd.3 (osd.3) 629 : cluster 0 1.4 scrub ok 2024-03-11T21:19:26.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:25 smithi023 bash[29146]: cluster 2024-03-11T21:19:24.569976+0000 mgr.y (mgr.24380) 1054 : cluster 0 pgmap v1228: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:19:26.405 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:19:26.752 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:19:26 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:19:26.752 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:26 smithi146 bash[23809]: cluster 2024-03-11T21:19:25.286285+0000 osd.7 (osd.7) 697 : cluster 0 3.d scrub starts 2024-03-11T21:19:26.753 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:26 smithi146 bash[23809]: cluster 2024-03-11T21:19:25.291443+0000 osd.7 (osd.7) 698 : cluster 0 3.d scrub ok 2024-03-11T21:19:26.753 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:26 smithi146 bash[23809]: cluster 2024-03-11T21:19:25.506015+0000 osd.3 (osd.3) 630 : cluster 0 2.1b deep-scrub starts 2024-03-11T21:19:26.753 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:26 smithi146 bash[23809]: cluster 2024-03-11T21:19:25.509645+0000 osd.3 (osd.3) 631 : cluster 0 2.1b deep-scrub ok 2024-03-11T21:19:26.774 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:26 smithi023 bash[21372]: cluster 2024-03-11T21:19:25.286285+0000 osd.7 (osd.7) 697 : cluster 0 3.d scrub starts 2024-03-11T21:19:26.774 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:26 smithi023 bash[21372]: cluster 2024-03-11T21:19:25.291443+0000 osd.7 (osd.7) 698 : cluster 0 3.d scrub ok 2024-03-11T21:19:26.774 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:26 smithi023 bash[21372]: cluster 2024-03-11T21:19:25.506015+0000 osd.3 (osd.3) 630 : cluster 0 2.1b deep-scrub starts 2024-03-11T21:19:26.774 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:26 smithi023 bash[21372]: cluster 2024-03-11T21:19:25.509645+0000 osd.3 (osd.3) 631 : cluster 0 2.1b deep-scrub ok 2024-03-11T21:19:26.774 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:26 smithi023 bash[29146]: cluster 2024-03-11T21:19:25.286285+0000 osd.7 (osd.7) 697 : cluster 0 3.d scrub starts 2024-03-11T21:19:26.774 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:26 smithi023 bash[29146]: cluster 2024-03-11T21:19:25.291443+0000 osd.7 (osd.7) 698 : cluster 0 3.d scrub ok 2024-03-11T21:19:26.774 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:26 smithi023 bash[29146]: cluster 2024-03-11T21:19:25.506015+0000 osd.3 (osd.3) 630 : cluster 0 2.1b deep-scrub starts 2024-03-11T21:19:26.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:26 smithi023 bash[29146]: cluster 2024-03-11T21:19:25.509645+0000 osd.3 (osd.3) 631 : cluster 0 2.1b deep-scrub ok 2024-03-11T21:19:28.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:27 smithi146 bash[23809]: cluster 2024-03-11T21:19:26.122181+0000 osd.2 (osd.2) 309 : cluster 0 2.0 deep-scrub starts 2024-03-11T21:19:28.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:27 smithi146 bash[23809]: cluster 2024-03-11T21:19:26.126360+0000 osd.2 (osd.2) 310 : cluster 0 2.0 deep-scrub ok 2024-03-11T21:19:28.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:27 smithi146 bash[23809]: cluster 2024-03-11T21:19:26.295692+0000 osd.7 (osd.7) 699 : cluster 0 3.e scrub starts 2024-03-11T21:19:28.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:27 smithi146 bash[23809]: cluster 2024-03-11T21:19:26.300180+0000 osd.7 (osd.7) 700 : cluster 0 3.e scrub ok 2024-03-11T21:19:28.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:27 smithi146 bash[23809]: audit 2024-03-11T21:19:26.314909+0000 mgr.y (mgr.24380) 1055 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:19:28.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:27 smithi146 bash[23809]: cluster 2024-03-11T21:19:26.416812+0000 osd.0 (osd.0) 514 : cluster 0 6.10 scrub starts 2024-03-11T21:19:28.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:27 smithi146 bash[23809]: cluster 2024-03-11T21:19:26.481942+0000 osd.0 (osd.0) 515 : cluster 0 6.10 scrub ok 2024-03-11T21:19:28.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:27 smithi146 bash[23809]: cluster 2024-03-11T21:19:26.546466+0000 osd.6 (osd.6) 560 : cluster 0 3.10 scrub starts 2024-03-11T21:19:28.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:27 smithi146 bash[23809]: cluster 2024-03-11T21:19:26.550092+0000 osd.6 (osd.6) 561 : cluster 0 3.10 scrub ok 2024-03-11T21:19:28.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:27 smithi146 bash[23809]: cluster 2024-03-11T21:19:26.571593+0000 mgr.y (mgr.24380) 1056 : cluster 0 pgmap v1229: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:19:28.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:27 smithi023 bash[21372]: cluster 2024-03-11T21:19:26.122181+0000 osd.2 (osd.2) 309 : cluster 0 2.0 deep-scrub starts 2024-03-11T21:19:28.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:27 smithi023 bash[21372]: cluster 2024-03-11T21:19:26.126360+0000 osd.2 (osd.2) 310 : cluster 0 2.0 deep-scrub ok 2024-03-11T21:19:28.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:27 smithi023 bash[21372]: cluster 2024-03-11T21:19:26.295692+0000 osd.7 (osd.7) 699 : cluster 0 3.e scrub starts 2024-03-11T21:19:28.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:27 smithi023 bash[21372]: cluster 2024-03-11T21:19:26.300180+0000 osd.7 (osd.7) 700 : cluster 0 3.e scrub ok 2024-03-11T21:19:28.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:27 smithi023 bash[21372]: audit 2024-03-11T21:19:26.314909+0000 mgr.y (mgr.24380) 1055 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:19:28.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:27 smithi023 bash[21372]: cluster 2024-03-11T21:19:26.416812+0000 osd.0 (osd.0) 514 : cluster 0 6.10 scrub starts 2024-03-11T21:19:28.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:27 smithi023 bash[21372]: cluster 2024-03-11T21:19:26.481942+0000 osd.0 (osd.0) 515 : cluster 0 6.10 scrub ok 2024-03-11T21:19:28.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:27 smithi023 bash[21372]: cluster 2024-03-11T21:19:26.546466+0000 osd.6 (osd.6) 560 : cluster 0 3.10 scrub starts 2024-03-11T21:19:28.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:27 smithi023 bash[21372]: cluster 2024-03-11T21:19:26.550092+0000 osd.6 (osd.6) 561 : cluster 0 3.10 scrub ok 2024-03-11T21:19:28.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:27 smithi023 bash[21372]: cluster 2024-03-11T21:19:26.571593+0000 mgr.y (mgr.24380) 1056 : cluster 0 pgmap v1229: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:19:28.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:27 smithi023 bash[29146]: cluster 2024-03-11T21:19:26.122181+0000 osd.2 (osd.2) 309 : cluster 0 2.0 deep-scrub starts 2024-03-11T21:19:28.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:27 smithi023 bash[29146]: cluster 2024-03-11T21:19:26.126360+0000 osd.2 (osd.2) 310 : cluster 0 2.0 deep-scrub ok 2024-03-11T21:19:28.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:27 smithi023 bash[29146]: cluster 2024-03-11T21:19:26.295692+0000 osd.7 (osd.7) 699 : cluster 0 3.e scrub starts 2024-03-11T21:19:28.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:27 smithi023 bash[29146]: cluster 2024-03-11T21:19:26.300180+0000 osd.7 (osd.7) 700 : cluster 0 3.e scrub ok 2024-03-11T21:19:28.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:27 smithi023 bash[29146]: audit 2024-03-11T21:19:26.314909+0000 mgr.y (mgr.24380) 1055 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:19:28.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:27 smithi023 bash[29146]: cluster 2024-03-11T21:19:26.416812+0000 osd.0 (osd.0) 514 : cluster 0 6.10 scrub starts 2024-03-11T21:19:28.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:27 smithi023 bash[29146]: cluster 2024-03-11T21:19:26.481942+0000 osd.0 (osd.0) 515 : cluster 0 6.10 scrub ok 2024-03-11T21:19:28.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:27 smithi023 bash[29146]: cluster 2024-03-11T21:19:26.546466+0000 osd.6 (osd.6) 560 : cluster 0 3.10 scrub starts 2024-03-11T21:19:28.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:27 smithi023 bash[29146]: cluster 2024-03-11T21:19:26.550092+0000 osd.6 (osd.6) 561 : cluster 0 3.10 scrub ok 2024-03-11T21:19:28.026 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:27 smithi023 bash[29146]: cluster 2024-03-11T21:19:26.571593+0000 mgr.y (mgr.24380) 1056 : cluster 0 pgmap v1229: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:19:28.774 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:19:28 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:19:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:19:28.774 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:28 smithi023 bash[21372]: cluster 2024-03-11T21:19:26.677966+0000 osd.5 (osd.5) 675 : cluster 0 2.18 scrub starts 2024-03-11T21:19:28.774 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:28 smithi023 bash[21372]: cluster 2024-03-11T21:19:26.680573+0000 osd.5 (osd.5) 676 : cluster 0 2.18 scrub ok 2024-03-11T21:19:28.774 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:28 smithi023 bash[21372]: cluster 2024-03-11T21:19:27.110379+0000 osd.2 (osd.2) 311 : cluster 0 2.c scrub starts 2024-03-11T21:19:28.774 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:28 smithi023 bash[21372]: cluster 2024-03-11T21:19:27.112780+0000 osd.2 (osd.2) 312 : cluster 0 2.c scrub ok 2024-03-11T21:19:28.774 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:28 smithi023 bash[21372]: cluster 2024-03-11T21:19:27.320995+0000 osd.4 (osd.4) 147 : cluster 0 4.18 scrub starts 2024-03-11T21:19:28.774 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:28 smithi023 bash[21372]: cluster 2024-03-11T21:19:27.323775+0000 osd.4 (osd.4) 148 : cluster 0 4.18 scrub ok 2024-03-11T21:19:28.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:28 smithi023 bash[21372]: cluster 2024-03-11T21:19:27.500178+0000 osd.3 (osd.3) 632 : cluster 0 4.11 scrub starts 2024-03-11T21:19:28.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:28 smithi023 bash[21372]: cluster 2024-03-11T21:19:27.502369+0000 osd.3 (osd.3) 633 : cluster 0 4.11 scrub ok 2024-03-11T21:19:28.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:28 smithi023 bash[29146]: cluster 2024-03-11T21:19:26.677966+0000 osd.5 (osd.5) 675 : cluster 0 2.18 scrub starts 2024-03-11T21:19:28.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:28 smithi023 bash[29146]: cluster 2024-03-11T21:19:26.680573+0000 osd.5 (osd.5) 676 : cluster 0 2.18 scrub ok 2024-03-11T21:19:28.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:28 smithi023 bash[29146]: cluster 2024-03-11T21:19:27.110379+0000 osd.2 (osd.2) 311 : cluster 0 2.c scrub starts 2024-03-11T21:19:28.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:28 smithi023 bash[29146]: cluster 2024-03-11T21:19:27.112780+0000 osd.2 (osd.2) 312 : cluster 0 2.c scrub ok 2024-03-11T21:19:28.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:28 smithi023 bash[29146]: cluster 2024-03-11T21:19:27.320995+0000 osd.4 (osd.4) 147 : cluster 0 4.18 scrub starts 2024-03-11T21:19:28.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:28 smithi023 bash[29146]: cluster 2024-03-11T21:19:27.323775+0000 osd.4 (osd.4) 148 : cluster 0 4.18 scrub ok 2024-03-11T21:19:28.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:28 smithi023 bash[29146]: cluster 2024-03-11T21:19:27.500178+0000 osd.3 (osd.3) 632 : cluster 0 4.11 scrub starts 2024-03-11T21:19:28.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:28 smithi023 bash[29146]: cluster 2024-03-11T21:19:27.502369+0000 osd.3 (osd.3) 633 : cluster 0 4.11 scrub ok 2024-03-11T21:19:29.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:28 smithi146 bash[23809]: cluster 2024-03-11T21:19:26.677966+0000 osd.5 (osd.5) 675 : cluster 0 2.18 scrub starts 2024-03-11T21:19:29.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:28 smithi146 bash[23809]: cluster 2024-03-11T21:19:26.680573+0000 osd.5 (osd.5) 676 : cluster 0 2.18 scrub ok 2024-03-11T21:19:29.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:28 smithi146 bash[23809]: cluster 2024-03-11T21:19:27.110379+0000 osd.2 (osd.2) 311 : cluster 0 2.c scrub starts 2024-03-11T21:19:29.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:28 smithi146 bash[23809]: cluster 2024-03-11T21:19:27.112780+0000 osd.2 (osd.2) 312 : cluster 0 2.c scrub ok 2024-03-11T21:19:29.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:28 smithi146 bash[23809]: cluster 2024-03-11T21:19:27.320995+0000 osd.4 (osd.4) 147 : cluster 0 4.18 scrub starts 2024-03-11T21:19:29.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:28 smithi146 bash[23809]: cluster 2024-03-11T21:19:27.323775+0000 osd.4 (osd.4) 148 : cluster 0 4.18 scrub ok 2024-03-11T21:19:29.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:28 smithi146 bash[23809]: cluster 2024-03-11T21:19:27.500178+0000 osd.3 (osd.3) 632 : cluster 0 4.11 scrub starts 2024-03-11T21:19:29.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:28 smithi146 bash[23809]: cluster 2024-03-11T21:19:27.502369+0000 osd.3 (osd.3) 633 : cluster 0 4.11 scrub ok 2024-03-11T21:19:29.774 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:29 smithi023 bash[21372]: cluster 2024-03-11T21:19:28.254354+0000 osd.7 (osd.7) 701 : cluster 0 6.8 scrub starts 2024-03-11T21:19:29.774 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:29 smithi023 bash[21372]: cluster 2024-03-11T21:19:28.257059+0000 osd.7 (osd.7) 702 : cluster 0 6.8 scrub ok 2024-03-11T21:19:29.774 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:29 smithi023 bash[21372]: cluster 2024-03-11T21:19:28.468491+0000 osd.3 (osd.3) 634 : cluster 0 4.19 scrub starts 2024-03-11T21:19:29.774 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:29 smithi023 bash[21372]: cluster 2024-03-11T21:19:28.470549+0000 osd.3 (osd.3) 635 : cluster 0 4.19 scrub ok 2024-03-11T21:19:29.774 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:29 smithi023 bash[21372]: cluster 2024-03-11T21:19:28.572532+0000 mgr.y (mgr.24380) 1057 : cluster 0 pgmap v1230: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:19:29.774 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:29 smithi023 bash[29146]: cluster 2024-03-11T21:19:28.254354+0000 osd.7 (osd.7) 701 : cluster 0 6.8 scrub starts 2024-03-11T21:19:29.774 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:29 smithi023 bash[29146]: cluster 2024-03-11T21:19:28.257059+0000 osd.7 (osd.7) 702 : cluster 0 6.8 scrub ok 2024-03-11T21:19:29.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:29 smithi023 bash[29146]: cluster 2024-03-11T21:19:28.468491+0000 osd.3 (osd.3) 634 : cluster 0 4.19 scrub starts 2024-03-11T21:19:29.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:29 smithi023 bash[29146]: cluster 2024-03-11T21:19:28.470549+0000 osd.3 (osd.3) 635 : cluster 0 4.19 scrub ok 2024-03-11T21:19:29.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:29 smithi023 bash[29146]: cluster 2024-03-11T21:19:28.572532+0000 mgr.y (mgr.24380) 1057 : cluster 0 pgmap v1230: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:19:29.881 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-11T21:19:29.881 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":491,"fsid":"67ba00f8-dfe7-11ee-95c8-87774f69a715","created":"2024-03-11T20:40:13.664802+0000","modified":"2024-03-11T21:19:11.812614+0000","last_up_change":"2024-03-11T21:00:47.324817+0000","last_in_change":"2024-03-11T21:14:49.940617+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":20,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T20:45:31.964570+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":21,"pg_placement_num":21,"pg_placement_num_target":21,"pg_num_target":21,"pg_num_pending":21,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"454","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"444","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8999999761581421,"score_stable":1.8999999761581421,"optimal_score":1,"raw_score_acting":1.8999999761581421,"raw_score_stable":1.8999999761581421,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-03-11T20:50:26.421486+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"default.rgw.log","create_time":"2024-03-11T20:50:28.592615+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.control","create_time":"2024-03-11T20:50:30.536371+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"datapool","create_time":"2024-03-11T20:50:31.135845+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":87,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-03-11T20:50:33.233163+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3df72b4a-8fe4-4265-b4c6-af022e1e23e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":449,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":2154627393},{"type":"v1","addr":"172.21.15.23:6803","nonce":2154627393}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":2154627393},{"type":"v1","addr":"172.21.15.23:6805","nonce":2154627393}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":2154627393},{"type":"v1","addr":"172.21.15.23:6809","nonce":2154627393}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":2154627393},{"type":"v1","addr":"172.21.15.23:6807","nonce":2154627393}]},"public_addr":"172.21.15.23:6803/2154627393","cluster_addr":"172.21.15.23:6805/2154627393","heartbeat_back_addr":"172.21.15.23:6809/2154627393","heartbeat_front_addr":"172.21.15.23:6807/2154627393","state":["exists","up"]},{"osd":1,"uuid":"e8fd7500-9081-41a1-8077-a4d0f0e12a2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":173,"up_from":211,"up_thru":449,"down_at":174,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":56585354},{"type":"v1","addr":"172.21.15.23:6811","nonce":56585354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":56585354},{"type":"v1","addr":"172.21.15.23:6813","nonce":56585354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":56585354},{"type":"v1","addr":"172.21.15.23:6817","nonce":56585354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":56585354},{"type":"v1","addr":"172.21.15.23:6815","nonce":56585354}]},"public_addr":"172.21.15.23:6811/56585354","cluster_addr":"172.21.15.23:6813/56585354","heartbeat_back_addr":"172.21.15.23:6817/56585354","heartbeat_front_addr":"172.21.15.23:6815/56585354","state":["exists","up"]},{"osd":2,"uuid":"dd85d52a-df48-4b03-8612-cecfd82b5987","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":465,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":1770748056},{"type":"v1","addr":"172.21.15.23:6819","nonce":1770748056}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":1770748056},{"type":"v1","addr":"172.21.15.23:6821","nonce":1770748056}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":1770748056},{"type":"v1","addr":"172.21.15.23:6825","nonce":1770748056}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":1770748056},{"type":"v1","addr":"172.21.15.23:6823","nonce":1770748056}]},"public_addr":"172.21.15.23:6819/1770748056","cluster_addr":"172.21.15.23:6821/1770748056","heartbeat_back_addr":"172.21.15.23:6825/1770748056","heartbeat_front_addr":"172.21.15.23:6823/1770748056","state":["exists","up"]},{"osd":3,"uuid":"e3021106-3779-4b94-9bbf-a7724544110f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":28,"up_thru":449,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":938496814},{"type":"v1","addr":"172.21.15.23:6827","nonce":938496814}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":938496814},{"type":"v1","addr":"172.21.15.23:6829","nonce":938496814}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":938496814},{"type":"v1","addr":"172.21.15.23:6833","nonce":938496814}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":938496814},{"type":"v1","addr":"172.21.15.23:6831","nonce":938496814}]},"public_addr":"172.21.15.23:6827/938496814","cluster_addr":"172.21.15.23:6829/938496814","heartbeat_back_addr":"172.21.15.23:6833/938496814","heartbeat_front_addr":"172.21.15.23:6831/938496814","state":["exists","up"]},{"osd":4,"uuid":"4dcbc91e-7035-413d-97bc-c9d804eb16d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":35,"last_clean_end":182,"up_from":204,"up_thru":485,"down_at":183,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":3858359434},{"type":"v1","addr":"172.21.15.146:6801","nonce":3858359434}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":3858359434},{"type":"v1","addr":"172.21.15.146:6803","nonce":3858359434}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":3858359434},{"type":"v1","addr":"172.21.15.146:6807","nonce":3858359434}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":3858359434},{"type":"v1","addr":"172.21.15.146:6805","nonce":3858359434}]},"public_addr":"172.21.15.146:6801/3858359434","cluster_addr":"172.21.15.146:6803/3858359434","heartbeat_back_addr":"172.21.15.146:6807/3858359434","heartbeat_front_addr":"172.21.15.146:6805/3858359434","state":["exists","up"]},{"osd":5,"uuid":"aefcb766-8bae-4ff4-b71e-6691c8ac02f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":449,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":4003230814},{"type":"v1","addr":"172.21.15.146:6809","nonce":4003230814}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":4003230814},{"type":"v1","addr":"172.21.15.146:6811","nonce":4003230814}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":4003230814},{"type":"v1","addr":"172.21.15.146:6815","nonce":4003230814}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":4003230814},{"type":"v1","addr":"172.21.15.146:6813","nonce":4003230814}]},"public_addr":"172.21.15.146:6809/4003230814","cluster_addr":"172.21.15.146:6811/4003230814","heartbeat_back_addr":"172.21.15.146:6815/4003230814","heartbeat_front_addr":"172.21.15.146:6813/4003230814","state":["exists","up"]},{"osd":6,"uuid":"5cb151e8-1f9b-4617-8a86-adfb06bb9606","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":449,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":3334092376},{"type":"v1","addr":"172.21.15.146:6817","nonce":3334092376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3334092376},{"type":"v1","addr":"172.21.15.146:6819","nonce":3334092376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":3334092376},{"type":"v1","addr":"172.21.15.146:6823","nonce":3334092376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3334092376},{"type":"v1","addr":"172.21.15.146:6821","nonce":3334092376}]},"public_addr":"172.21.15.146:6817/3334092376","cluster_addr":"172.21.15.146:6819/3334092376","heartbeat_back_addr":"172.21.15.146:6823/3334092376","heartbeat_front_addr":"172.21.15.146:6821/3334092376","state":["exists","up"]},{"osd":7,"uuid":"bdbd11a2-fa09-4f82-93a2-6b2d9c750bfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":449,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":1678407923},{"type":"v1","addr":"172.21.15.146:6825","nonce":1678407923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":1678407923},{"type":"v1","addr":"172.21.15.146:6827","nonce":1678407923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":1678407923},{"type":"v1","addr":"172.21.15.146:6831","nonce":1678407923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":1678407923},{"type":"v1","addr":"172.21.15.146:6829","nonce":1678407923}]},"public_addr":"172.21.15.146:6825/1678407923","cluster_addr":"172.21.15.146:6827/1678407923","heartbeat_back_addr":"172.21.15.146:6831/1678407923","heartbeat_front_addr":"172.21.15.146:6829/1678407923","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:18:34.265472+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:19:03.927791+0000","dead_epoch":173},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:18:57.024592+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:18:54.176379+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:18:21.370975+0000","dead_epoch":182},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:18:20.506687+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:18:53.471898+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:18:18.673940+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":7,"to":2}]},{"pgid":"4.b","mappings":[{"from":1,"to":2}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]},{"pgid":"4.e","mappings":[{"from":1,"to":2}]},{"pgid":"4.16","mappings":[{"from":3,"to":2}]},{"pgid":"4.1d","mappings":[{"from":4,"to":2}]},{"pgid":"6.a","mappings":[{"from":6,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:0/1618334186":"2024-03-12T20:40:45.361537+0000","172.21.15.23:6801/2813578081":"2024-03-12T20:40:45.361537+0000","172.21.15.23:0/2869940180":"2024-03-12T20:51:33.790205+0000","172.21.15.23:6800/3037564930":"2024-03-12T20:51:33.790205+0000","172.21.15.23:0/3297567084":"2024-03-12T20:51:33.790205+0000","172.21.15.23:0/3708753659":"2024-03-12T20:40:45.361537+0000","172.21.15.23:0/2004146740":"2024-03-12T20:40:45.361537+0000","172.21.15.23:0/827653812":"2024-03-12T20:41:21.301516+0000","172.21.15.23:6800/1057699850":"2024-03-12T20:41:21.301516+0000","172.21.15.23:6800/2813578081":"2024-03-12T20:40:45.361537+0000","172.21.15.23:6801/1057699850":"2024-03-12T20:41:21.301516+0000","172.21.15.23:0/3833230375":"2024-03-12T20:41:21.301516+0000","172.21.15.23:6801/3037564930":"2024-03-12T20:51:33.790205+0000","172.21.15.23:0/1182809102":"2024-03-12T20:51:33.790205+0000","172.21.15.23:0/2122134187":"2024-03-12T20:51:33.790205+0000","172.21.15.23:0/2356701899":"2024-03-12T20:41:21.301516+0000","172.21.15.23:0/843974100":"2024-03-12T20:51:33.790205+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T21:19:30.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:29 smithi146 bash[23809]: cluster 2024-03-11T21:19:28.254354+0000 osd.7 (osd.7) 701 : cluster 0 6.8 scrub starts 2024-03-11T21:19:30.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:29 smithi146 bash[23809]: cluster 2024-03-11T21:19:28.257059+0000 osd.7 (osd.7) 702 : cluster 0 6.8 scrub ok 2024-03-11T21:19:30.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:29 smithi146 bash[23809]: cluster 2024-03-11T21:19:28.468491+0000 osd.3 (osd.3) 634 : cluster 0 4.19 scrub starts 2024-03-11T21:19:30.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:29 smithi146 bash[23809]: cluster 2024-03-11T21:19:28.470549+0000 osd.3 (osd.3) 635 : cluster 0 4.19 scrub ok 2024-03-11T21:19:30.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:29 smithi146 bash[23809]: cluster 2024-03-11T21:19:28.572532+0000 mgr.y (mgr.24380) 1057 : cluster 0 pgmap v1230: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:19:30.617 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-11T21:19:30.618 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-11T21:19:30.618 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-11T21:19:30.618 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-11T21:19:30.619 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-11T21:19:30.619 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-11T21:19:30.620 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-03-11T21:19:30.620 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-03-11T21:19:31.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:30 smithi146 bash[23809]: cluster 2024-03-11T21:19:29.247698+0000 osd.7 (osd.7) 703 : cluster 0 2.1c scrub starts 2024-03-11T21:19:31.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:30 smithi146 bash[23809]: cluster 2024-03-11T21:19:29.250327+0000 osd.7 (osd.7) 704 : cluster 0 2.1c scrub ok 2024-03-11T21:19:31.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:30 smithi146 bash[23809]: cluster 2024-03-11T21:19:29.430111+0000 osd.0 (osd.0) 516 : cluster 0 4.6 scrub starts 2024-03-11T21:19:31.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:30 smithi146 bash[23809]: cluster 2024-03-11T21:19:29.432401+0000 osd.0 (osd.0) 517 : cluster 0 4.6 scrub ok 2024-03-11T21:19:31.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:30 smithi146 bash[23809]: audit 2024-03-11T21:19:29.891268+0000 mon.a (mon.0) 2105 : audit 0 from='client.? 172.21.15.23:0/2984674709' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T21:19:31.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:30 smithi023 bash[29146]: cluster 2024-03-11T21:19:29.247698+0000 osd.7 (osd.7) 703 : cluster 0 2.1c scrub starts 2024-03-11T21:19:31.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:30 smithi023 bash[29146]: cluster 2024-03-11T21:19:29.250327+0000 osd.7 (osd.7) 704 : cluster 0 2.1c scrub ok 2024-03-11T21:19:31.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:30 smithi023 bash[29146]: cluster 2024-03-11T21:19:29.430111+0000 osd.0 (osd.0) 516 : cluster 0 4.6 scrub starts 2024-03-11T21:19:31.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:30 smithi023 bash[29146]: cluster 2024-03-11T21:19:29.432401+0000 osd.0 (osd.0) 517 : cluster 0 4.6 scrub ok 2024-03-11T21:19:31.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:30 smithi023 bash[29146]: audit 2024-03-11T21:19:29.891268+0000 mon.a (mon.0) 2105 : audit 0 from='client.? 172.21.15.23:0/2984674709' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T21:19:31.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:30 smithi023 bash[21372]: cluster 2024-03-11T21:19:29.247698+0000 osd.7 (osd.7) 703 : cluster 0 2.1c scrub starts 2024-03-11T21:19:31.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:30 smithi023 bash[21372]: cluster 2024-03-11T21:19:29.250327+0000 osd.7 (osd.7) 704 : cluster 0 2.1c scrub ok 2024-03-11T21:19:31.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:30 smithi023 bash[21372]: cluster 2024-03-11T21:19:29.430111+0000 osd.0 (osd.0) 516 : cluster 0 4.6 scrub starts 2024-03-11T21:19:31.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:30 smithi023 bash[21372]: cluster 2024-03-11T21:19:29.432401+0000 osd.0 (osd.0) 517 : cluster 0 4.6 scrub ok 2024-03-11T21:19:31.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:30 smithi023 bash[21372]: audit 2024-03-11T21:19:29.891268+0000 mon.a (mon.0) 2105 : audit 0 from='client.? 172.21.15.23:0/2984674709' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T21:19:32.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:31 smithi146 bash[23809]: cluster 2024-03-11T21:19:30.365785+0000 osd.4 (osd.4) 149 : cluster 0 6.1a scrub starts 2024-03-11T21:19:32.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:31 smithi146 bash[23809]: cluster 2024-03-11T21:19:30.434741+0000 osd.4 (osd.4) 150 : cluster 0 6.1a scrub ok 2024-03-11T21:19:32.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:31 smithi146 bash[23809]: cluster 2024-03-11T21:19:30.456716+0000 osd.0 (osd.0) 518 : cluster 0 4.1e scrub starts 2024-03-11T21:19:32.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:31 smithi146 bash[23809]: cluster 2024-03-11T21:19:30.460231+0000 osd.0 (osd.0) 519 : cluster 0 4.1e scrub ok 2024-03-11T21:19:32.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:31 smithi146 bash[23809]: cluster 2024-03-11T21:19:30.573547+0000 mgr.y (mgr.24380) 1058 : cluster 0 pgmap v1231: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:19:32.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:31 smithi146 bash[23809]: cluster 2024-03-11T21:19:30.927671+0000 osd.1 (osd.1) 438 : cluster 0 1.b scrub starts 2024-03-11T21:19:32.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:31 smithi146 bash[23809]: cluster 2024-03-11T21:19:30.930920+0000 osd.1 (osd.1) 439 : cluster 0 1.b scrub ok 2024-03-11T21:19:32.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:31 smithi023 bash[29146]: cluster 2024-03-11T21:19:30.365785+0000 osd.4 (osd.4) 149 : cluster 0 6.1a scrub starts 2024-03-11T21:19:32.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:31 smithi023 bash[29146]: cluster 2024-03-11T21:19:30.434741+0000 osd.4 (osd.4) 150 : cluster 0 6.1a scrub ok 2024-03-11T21:19:32.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:31 smithi023 bash[29146]: cluster 2024-03-11T21:19:30.456716+0000 osd.0 (osd.0) 518 : cluster 0 4.1e scrub starts 2024-03-11T21:19:32.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:31 smithi023 bash[29146]: cluster 2024-03-11T21:19:30.460231+0000 osd.0 (osd.0) 519 : cluster 0 4.1e scrub ok 2024-03-11T21:19:32.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:31 smithi023 bash[29146]: cluster 2024-03-11T21:19:30.573547+0000 mgr.y (mgr.24380) 1058 : cluster 0 pgmap v1231: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:19:32.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:31 smithi023 bash[29146]: cluster 2024-03-11T21:19:30.927671+0000 osd.1 (osd.1) 438 : cluster 0 1.b scrub starts 2024-03-11T21:19:32.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:31 smithi023 bash[29146]: cluster 2024-03-11T21:19:30.930920+0000 osd.1 (osd.1) 439 : cluster 0 1.b scrub ok 2024-03-11T21:19:32.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:31 smithi023 bash[21372]: cluster 2024-03-11T21:19:30.365785+0000 osd.4 (osd.4) 149 : cluster 0 6.1a scrub starts 2024-03-11T21:19:32.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:31 smithi023 bash[21372]: cluster 2024-03-11T21:19:30.434741+0000 osd.4 (osd.4) 150 : cluster 0 6.1a scrub ok 2024-03-11T21:19:32.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:31 smithi023 bash[21372]: cluster 2024-03-11T21:19:30.456716+0000 osd.0 (osd.0) 518 : cluster 0 4.1e scrub starts 2024-03-11T21:19:32.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:31 smithi023 bash[21372]: cluster 2024-03-11T21:19:30.460231+0000 osd.0 (osd.0) 519 : cluster 0 4.1e scrub ok 2024-03-11T21:19:32.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:31 smithi023 bash[21372]: cluster 2024-03-11T21:19:30.573547+0000 mgr.y (mgr.24380) 1058 : cluster 0 pgmap v1231: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:19:32.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:31 smithi023 bash[21372]: cluster 2024-03-11T21:19:30.927671+0000 osd.1 (osd.1) 438 : cluster 0 1.b scrub starts 2024-03-11T21:19:32.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:31 smithi023 bash[21372]: cluster 2024-03-11T21:19:30.930920+0000 osd.1 (osd.1) 439 : cluster 0 1.b scrub ok 2024-03-11T21:19:33.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:32 smithi146 bash[23809]: cluster 2024-03-11T21:19:30.705800+0000 osd.5 (osd.5) 677 : cluster 0 2.3 scrub starts 2024-03-11T21:19:33.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:32 smithi146 bash[23809]: cluster 2024-03-11T21:19:30.708041+0000 osd.5 (osd.5) 678 : cluster 0 2.3 scrub ok 2024-03-11T21:19:33.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:32 smithi146 bash[23809]: cluster 2024-03-11T21:19:31.019929+0000 osd.2 (osd.2) 313 : cluster 0 6.14 scrub starts 2024-03-11T21:19:33.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:32 smithi146 bash[23809]: cluster 2024-03-11T21:19:31.021859+0000 osd.2 (osd.2) 314 : cluster 0 6.14 scrub ok 2024-03-11T21:19:33.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:32 smithi146 bash[23809]: cluster 2024-03-11T21:19:31.488873+0000 osd.0 (osd.0) 520 : cluster 0 6.0 scrub starts 2024-03-11T21:19:33.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:32 smithi146 bash[23809]: cluster 2024-03-11T21:19:31.577693+0000 osd.0 (osd.0) 521 : cluster 0 6.0 scrub ok 2024-03-11T21:19:33.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:32 smithi023 bash[21372]: cluster 2024-03-11T21:19:30.705800+0000 osd.5 (osd.5) 677 : cluster 0 2.3 scrub starts 2024-03-11T21:19:33.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:32 smithi023 bash[21372]: cluster 2024-03-11T21:19:30.708041+0000 osd.5 (osd.5) 678 : cluster 0 2.3 scrub ok 2024-03-11T21:19:33.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:32 smithi023 bash[21372]: cluster 2024-03-11T21:19:31.019929+0000 osd.2 (osd.2) 313 : cluster 0 6.14 scrub starts 2024-03-11T21:19:33.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:32 smithi023 bash[21372]: cluster 2024-03-11T21:19:31.021859+0000 osd.2 (osd.2) 314 : cluster 0 6.14 scrub ok 2024-03-11T21:19:33.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:32 smithi023 bash[21372]: cluster 2024-03-11T21:19:31.488873+0000 osd.0 (osd.0) 520 : cluster 0 6.0 scrub starts 2024-03-11T21:19:33.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:32 smithi023 bash[21372]: cluster 2024-03-11T21:19:31.577693+0000 osd.0 (osd.0) 521 : cluster 0 6.0 scrub ok 2024-03-11T21:19:33.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:32 smithi023 bash[29146]: cluster 2024-03-11T21:19:30.705800+0000 osd.5 (osd.5) 677 : cluster 0 2.3 scrub starts 2024-03-11T21:19:33.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:32 smithi023 bash[29146]: cluster 2024-03-11T21:19:30.708041+0000 osd.5 (osd.5) 678 : cluster 0 2.3 scrub ok 2024-03-11T21:19:33.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:32 smithi023 bash[29146]: cluster 2024-03-11T21:19:31.019929+0000 osd.2 (osd.2) 313 : cluster 0 6.14 scrub starts 2024-03-11T21:19:33.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:32 smithi023 bash[29146]: cluster 2024-03-11T21:19:31.021859+0000 osd.2 (osd.2) 314 : cluster 0 6.14 scrub ok 2024-03-11T21:19:33.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:32 smithi023 bash[29146]: cluster 2024-03-11T21:19:31.488873+0000 osd.0 (osd.0) 520 : cluster 0 6.0 scrub starts 2024-03-11T21:19:33.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:32 smithi023 bash[29146]: cluster 2024-03-11T21:19:31.577693+0000 osd.0 (osd.0) 521 : cluster 0 6.0 scrub ok 2024-03-11T21:19:34.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:33 smithi146 bash[23809]: cluster 2024-03-11T21:19:32.574881+0000 mgr.y (mgr.24380) 1059 : cluster 0 pgmap v1232: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:19:34.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:33 smithi146 bash[23809]: cluster 2024-03-11T21:19:32.651024+0000 osd.6 (osd.6) 562 : cluster 0 2.1a scrub starts 2024-03-11T21:19:34.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:33 smithi146 bash[23809]: cluster 2024-03-11T21:19:32.653521+0000 osd.6 (osd.6) 563 : cluster 0 2.1a scrub ok 2024-03-11T21:19:34.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:33 smithi146 bash[23809]: cluster 2024-03-11T21:19:32.739560+0000 osd.5 (osd.5) 679 : cluster 0 6.a deep-scrub starts 2024-03-11T21:19:34.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:33 smithi146 bash[23809]: cluster 2024-03-11T21:19:32.786925+0000 osd.5 (osd.5) 680 : cluster 0 6.a deep-scrub ok 2024-03-11T21:19:34.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:33 smithi023 bash[21372]: cluster 2024-03-11T21:19:32.574881+0000 mgr.y (mgr.24380) 1059 : cluster 0 pgmap v1232: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:19:34.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:33 smithi023 bash[21372]: cluster 2024-03-11T21:19:32.651024+0000 osd.6 (osd.6) 562 : cluster 0 2.1a scrub starts 2024-03-11T21:19:34.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:33 smithi023 bash[21372]: cluster 2024-03-11T21:19:32.653521+0000 osd.6 (osd.6) 563 : cluster 0 2.1a scrub ok 2024-03-11T21:19:34.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:33 smithi023 bash[21372]: cluster 2024-03-11T21:19:32.739560+0000 osd.5 (osd.5) 679 : cluster 0 6.a deep-scrub starts 2024-03-11T21:19:34.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:33 smithi023 bash[21372]: cluster 2024-03-11T21:19:32.786925+0000 osd.5 (osd.5) 680 : cluster 0 6.a deep-scrub ok 2024-03-11T21:19:34.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:33 smithi023 bash[29146]: cluster 2024-03-11T21:19:32.574881+0000 mgr.y (mgr.24380) 1059 : cluster 0 pgmap v1232: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:19:34.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:33 smithi023 bash[29146]: cluster 2024-03-11T21:19:32.651024+0000 osd.6 (osd.6) 562 : cluster 0 2.1a scrub starts 2024-03-11T21:19:34.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:33 smithi023 bash[29146]: cluster 2024-03-11T21:19:32.653521+0000 osd.6 (osd.6) 563 : cluster 0 2.1a scrub ok 2024-03-11T21:19:34.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:33 smithi023 bash[29146]: cluster 2024-03-11T21:19:32.739560+0000 osd.5 (osd.5) 679 : cluster 0 6.a deep-scrub starts 2024-03-11T21:19:34.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:33 smithi023 bash[29146]: cluster 2024-03-11T21:19:32.786925+0000 osd.5 (osd.5) 680 : cluster 0 6.a deep-scrub ok 2024-03-11T21:19:35.548 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:19:35.548 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:19:35.549 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:19:35.549 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:19:35.549 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:19:35.549 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:19:35.550 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:19:35.550 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:19:36.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:35 smithi146 bash[23809]: cluster 2024-03-11T21:19:34.213762+0000 osd.7 (osd.7) 705 : cluster 0 2.1d scrub starts 2024-03-11T21:19:36.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:35 smithi146 bash[23809]: cluster 2024-03-11T21:19:34.216019+0000 osd.7 (osd.7) 706 : cluster 0 2.1d scrub ok 2024-03-11T21:19:36.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:35 smithi146 bash[23809]: cluster 2024-03-11T21:19:34.369028+0000 osd.4 (osd.4) 151 : cluster 0 3.14 scrub starts 2024-03-11T21:19:36.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:35 smithi146 bash[23809]: cluster 2024-03-11T21:19:34.373213+0000 osd.4 (osd.4) 152 : cluster 0 3.14 scrub ok 2024-03-11T21:19:36.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:35 smithi146 bash[23809]: cluster 2024-03-11T21:19:34.575775+0000 mgr.y (mgr.24380) 1060 : cluster 0 pgmap v1233: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:19:36.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:35 smithi146 bash[23809]: cluster 2024-03-11T21:19:35.030159+0000 osd.1 (osd.1) 440 : cluster 0 4.12 scrub starts 2024-03-11T21:19:36.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:35 smithi146 bash[23809]: cluster 2024-03-11T21:19:35.032624+0000 osd.1 (osd.1) 441 : cluster 0 4.12 scrub ok 2024-03-11T21:19:36.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:35 smithi023 bash[21372]: cluster 2024-03-11T21:19:34.213762+0000 osd.7 (osd.7) 705 : cluster 0 2.1d scrub starts 2024-03-11T21:19:36.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:35 smithi023 bash[21372]: cluster 2024-03-11T21:19:34.216019+0000 osd.7 (osd.7) 706 : cluster 0 2.1d scrub ok 2024-03-11T21:19:36.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:35 smithi023 bash[21372]: cluster 2024-03-11T21:19:34.369028+0000 osd.4 (osd.4) 151 : cluster 0 3.14 scrub starts 2024-03-11T21:19:36.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:35 smithi023 bash[21372]: cluster 2024-03-11T21:19:34.373213+0000 osd.4 (osd.4) 152 : cluster 0 3.14 scrub ok 2024-03-11T21:19:36.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:35 smithi023 bash[21372]: cluster 2024-03-11T21:19:34.575775+0000 mgr.y (mgr.24380) 1060 : cluster 0 pgmap v1233: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:19:36.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:35 smithi023 bash[21372]: cluster 2024-03-11T21:19:35.030159+0000 osd.1 (osd.1) 440 : cluster 0 4.12 scrub starts 2024-03-11T21:19:36.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:35 smithi023 bash[21372]: cluster 2024-03-11T21:19:35.032624+0000 osd.1 (osd.1) 441 : cluster 0 4.12 scrub ok 2024-03-11T21:19:36.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:35 smithi023 bash[29146]: cluster 2024-03-11T21:19:34.213762+0000 osd.7 (osd.7) 705 : cluster 0 2.1d scrub starts 2024-03-11T21:19:36.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:35 smithi023 bash[29146]: cluster 2024-03-11T21:19:34.216019+0000 osd.7 (osd.7) 706 : cluster 0 2.1d scrub ok 2024-03-11T21:19:36.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:35 smithi023 bash[29146]: cluster 2024-03-11T21:19:34.369028+0000 osd.4 (osd.4) 151 : cluster 0 3.14 scrub starts 2024-03-11T21:19:36.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:35 smithi023 bash[29146]: cluster 2024-03-11T21:19:34.373213+0000 osd.4 (osd.4) 152 : cluster 0 3.14 scrub ok 2024-03-11T21:19:36.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:35 smithi023 bash[29146]: cluster 2024-03-11T21:19:34.575775+0000 mgr.y (mgr.24380) 1060 : cluster 0 pgmap v1233: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:19:36.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:35 smithi023 bash[29146]: cluster 2024-03-11T21:19:35.030159+0000 osd.1 (osd.1) 440 : cluster 0 4.12 scrub starts 2024-03-11T21:19:36.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:35 smithi023 bash[29146]: cluster 2024-03-11T21:19:35.032624+0000 osd.1 (osd.1) 441 : cluster 0 4.12 scrub ok 2024-03-11T21:19:36.630 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:19:36 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:19:37.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:36 smithi146 bash[23809]: cluster 2024-03-11T21:19:36.075732+0000 osd.1 (osd.1) 442 : cluster 0 6.1d scrub starts 2024-03-11T21:19:37.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:36 smithi146 bash[23809]: cluster 2024-03-11T21:19:36.241323+0000 osd.1 (osd.1) 443 : cluster 0 6.1d scrub ok 2024-03-11T21:19:37.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:36 smithi023 bash[21372]: cluster 2024-03-11T21:19:36.075732+0000 osd.1 (osd.1) 442 : cluster 0 6.1d scrub starts 2024-03-11T21:19:37.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:36 smithi023 bash[21372]: cluster 2024-03-11T21:19:36.241323+0000 osd.1 (osd.1) 443 : cluster 0 6.1d scrub ok 2024-03-11T21:19:37.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:36 smithi023 bash[29146]: cluster 2024-03-11T21:19:36.075732+0000 osd.1 (osd.1) 442 : cluster 0 6.1d scrub starts 2024-03-11T21:19:37.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:36 smithi023 bash[29146]: cluster 2024-03-11T21:19:36.241323+0000 osd.1 (osd.1) 443 : cluster 0 6.1d scrub ok 2024-03-11T21:19:37.774 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:37 smithi023 bash[21372]: audit 2024-03-11T21:19:36.326253+0000 mgr.y (mgr.24380) 1061 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:19:37.774 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:37 smithi023 bash[21372]: cluster 2024-03-11T21:19:36.388224+0000 osd.4 (osd.4) 153 : cluster 0 4.1 scrub starts 2024-03-11T21:19:37.774 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:37 smithi023 bash[21372]: cluster 2024-03-11T21:19:36.390638+0000 osd.4 (osd.4) 154 : cluster 0 4.1 scrub ok 2024-03-11T21:19:37.774 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:37 smithi023 bash[21372]: cluster 2024-03-11T21:19:36.577444+0000 mgr.y (mgr.24380) 1062 : cluster 0 pgmap v1234: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:19:37.774 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:37 smithi023 bash[21372]: cluster 2024-03-11T21:19:36.657484+0000 osd.5 (osd.5) 681 : cluster 0 2.12 scrub starts 2024-03-11T21:19:37.774 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:37 smithi023 bash[21372]: cluster 2024-03-11T21:19:36.659646+0000 osd.5 (osd.5) 682 : cluster 0 2.12 scrub ok 2024-03-11T21:19:37.774 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:37 smithi023 bash[21372]: audit 2024-03-11T21:19:37.089572+0000 mon.a (mon.0) 2106 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:19:37.774 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:37 smithi023 bash[29146]: audit 2024-03-11T21:19:36.326253+0000 mgr.y (mgr.24380) 1061 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:19:37.774 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:37 smithi023 bash[29146]: cluster 2024-03-11T21:19:36.388224+0000 osd.4 (osd.4) 153 : cluster 0 4.1 scrub starts 2024-03-11T21:19:37.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:37 smithi023 bash[29146]: cluster 2024-03-11T21:19:36.390638+0000 osd.4 (osd.4) 154 : cluster 0 4.1 scrub ok 2024-03-11T21:19:37.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:37 smithi023 bash[29146]: cluster 2024-03-11T21:19:36.577444+0000 mgr.y (mgr.24380) 1062 : cluster 0 pgmap v1234: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:19:37.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:37 smithi023 bash[29146]: cluster 2024-03-11T21:19:36.657484+0000 osd.5 (osd.5) 681 : cluster 0 2.12 scrub starts 2024-03-11T21:19:37.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:37 smithi023 bash[29146]: cluster 2024-03-11T21:19:36.659646+0000 osd.5 (osd.5) 682 : cluster 0 2.12 scrub ok 2024-03-11T21:19:37.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:37 smithi023 bash[29146]: audit 2024-03-11T21:19:37.089572+0000 mon.a (mon.0) 2106 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:19:38.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:37 smithi146 bash[23809]: audit 2024-03-11T21:19:36.326253+0000 mgr.y (mgr.24380) 1061 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:19:38.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:37 smithi146 bash[23809]: cluster 2024-03-11T21:19:36.388224+0000 osd.4 (osd.4) 153 : cluster 0 4.1 scrub starts 2024-03-11T21:19:38.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:37 smithi146 bash[23809]: cluster 2024-03-11T21:19:36.390638+0000 osd.4 (osd.4) 154 : cluster 0 4.1 scrub ok 2024-03-11T21:19:38.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:37 smithi146 bash[23809]: cluster 2024-03-11T21:19:36.577444+0000 mgr.y (mgr.24380) 1062 : cluster 0 pgmap v1234: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:19:38.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:37 smithi146 bash[23809]: cluster 2024-03-11T21:19:36.657484+0000 osd.5 (osd.5) 681 : cluster 0 2.12 scrub starts 2024-03-11T21:19:38.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:37 smithi146 bash[23809]: cluster 2024-03-11T21:19:36.659646+0000 osd.5 (osd.5) 682 : cluster 0 2.12 scrub ok 2024-03-11T21:19:38.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:37 smithi146 bash[23809]: audit 2024-03-11T21:19:37.089572+0000 mon.a (mon.0) 2106 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:19:38.524 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:19:38 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:19:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:19:39.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:38 smithi146 bash[23809]: cluster 2024-03-11T21:19:37.181239+0000 osd.7 (osd.7) 707 : cluster 0 6.5 scrub starts 2024-03-11T21:19:39.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:38 smithi146 bash[23809]: cluster 2024-03-11T21:19:37.270213+0000 osd.7 (osd.7) 708 : cluster 0 6.5 scrub ok 2024-03-11T21:19:39.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:38 smithi146 bash[23809]: cluster 2024-03-11T21:19:37.457794+0000 osd.3 (osd.3) 636 : cluster 0 2.19 scrub starts 2024-03-11T21:19:39.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:38 smithi146 bash[23809]: cluster 2024-03-11T21:19:37.460790+0000 osd.3 (osd.3) 637 : cluster 0 2.19 scrub ok 2024-03-11T21:19:39.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:38 smithi146 bash[23809]: cluster 2024-03-11T21:19:38.139378+0000 osd.1 (osd.1) 444 : cluster 0 2.4 scrub starts 2024-03-11T21:19:39.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:38 smithi146 bash[23809]: cluster 2024-03-11T21:19:38.141027+0000 osd.1 (osd.1) 445 : cluster 0 2.4 scrub ok 2024-03-11T21:19:39.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:38 smithi023 bash[21372]: cluster 2024-03-11T21:19:37.181239+0000 osd.7 (osd.7) 707 : cluster 0 6.5 scrub starts 2024-03-11T21:19:39.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:38 smithi023 bash[21372]: cluster 2024-03-11T21:19:37.270213+0000 osd.7 (osd.7) 708 : cluster 0 6.5 scrub ok 2024-03-11T21:19:39.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:38 smithi023 bash[21372]: cluster 2024-03-11T21:19:37.457794+0000 osd.3 (osd.3) 636 : cluster 0 2.19 scrub starts 2024-03-11T21:19:39.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:38 smithi023 bash[21372]: cluster 2024-03-11T21:19:37.460790+0000 osd.3 (osd.3) 637 : cluster 0 2.19 scrub ok 2024-03-11T21:19:39.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:38 smithi023 bash[21372]: cluster 2024-03-11T21:19:38.139378+0000 osd.1 (osd.1) 444 : cluster 0 2.4 scrub starts 2024-03-11T21:19:39.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:38 smithi023 bash[21372]: cluster 2024-03-11T21:19:38.141027+0000 osd.1 (osd.1) 445 : cluster 0 2.4 scrub ok 2024-03-11T21:19:39.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:38 smithi023 bash[29146]: cluster 2024-03-11T21:19:37.181239+0000 osd.7 (osd.7) 707 : cluster 0 6.5 scrub starts 2024-03-11T21:19:39.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:38 smithi023 bash[29146]: cluster 2024-03-11T21:19:37.270213+0000 osd.7 (osd.7) 708 : cluster 0 6.5 scrub ok 2024-03-11T21:19:39.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:38 smithi023 bash[29146]: cluster 2024-03-11T21:19:37.457794+0000 osd.3 (osd.3) 636 : cluster 0 2.19 scrub starts 2024-03-11T21:19:39.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:38 smithi023 bash[29146]: cluster 2024-03-11T21:19:37.460790+0000 osd.3 (osd.3) 637 : cluster 0 2.19 scrub ok 2024-03-11T21:19:39.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:38 smithi023 bash[29146]: cluster 2024-03-11T21:19:38.139378+0000 osd.1 (osd.1) 444 : cluster 0 2.4 scrub starts 2024-03-11T21:19:39.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:38 smithi023 bash[29146]: cluster 2024-03-11T21:19:38.141027+0000 osd.1 (osd.1) 445 : cluster 0 2.4 scrub ok 2024-03-11T21:19:40.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:39 smithi146 bash[23809]: cluster 2024-03-11T21:19:38.483020+0000 osd.7 (osd.7) 709 : cluster 0 purged_snaps scrub starts 2024-03-11T21:19:40.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:39 smithi146 bash[23809]: cluster 2024-03-11T21:19:38.483112+0000 osd.7 (osd.7) 710 : cluster 0 purged_snaps scrub ok 2024-03-11T21:19:40.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:39 smithi146 bash[23809]: cluster 2024-03-11T21:19:38.578374+0000 mgr.y (mgr.24380) 1063 : cluster 0 pgmap v1235: 152 pgs: 1 active+clean+scrubbing, 151 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:19:40.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:39 smithi146 bash[23809]: cluster 2024-03-11T21:19:38.689021+0000 mon.a (mon.0) 2107 : cluster 0 osdmap e492: 8 total, 8 up, 8 in 2024-03-11T21:19:40.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:39 smithi023 bash[21372]: cluster 2024-03-11T21:19:38.483020+0000 osd.7 (osd.7) 709 : cluster 0 purged_snaps scrub starts 2024-03-11T21:19:40.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:39 smithi023 bash[21372]: cluster 2024-03-11T21:19:38.483112+0000 osd.7 (osd.7) 710 : cluster 0 purged_snaps scrub ok 2024-03-11T21:19:40.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:39 smithi023 bash[21372]: cluster 2024-03-11T21:19:38.578374+0000 mgr.y (mgr.24380) 1063 : cluster 0 pgmap v1235: 152 pgs: 1 active+clean+scrubbing, 151 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:19:40.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:39 smithi023 bash[21372]: cluster 2024-03-11T21:19:38.689021+0000 mon.a (mon.0) 2107 : cluster 0 osdmap e492: 8 total, 8 up, 8 in 2024-03-11T21:19:40.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:39 smithi023 bash[29146]: cluster 2024-03-11T21:19:38.483020+0000 osd.7 (osd.7) 709 : cluster 0 purged_snaps scrub starts 2024-03-11T21:19:40.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:39 smithi023 bash[29146]: cluster 2024-03-11T21:19:38.483112+0000 osd.7 (osd.7) 710 : cluster 0 purged_snaps scrub ok 2024-03-11T21:19:40.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:39 smithi023 bash[29146]: cluster 2024-03-11T21:19:38.578374+0000 mgr.y (mgr.24380) 1063 : cluster 0 pgmap v1235: 152 pgs: 1 active+clean+scrubbing, 151 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:19:40.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:39 smithi023 bash[29146]: cluster 2024-03-11T21:19:38.689021+0000 mon.a (mon.0) 2107 : cluster 0 osdmap e492: 8 total, 8 up, 8 in 2024-03-11T21:19:40.774 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:40 smithi023 bash[29146]: cluster 2024-03-11T21:19:39.036335+0000 osd.2 (osd.2) 315 : cluster 0 1.14 scrub starts 2024-03-11T21:19:40.774 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:40 smithi023 bash[29146]: cluster 2024-03-11T21:19:39.039225+0000 osd.2 (osd.2) 316 : cluster 0 1.14 scrub ok 2024-03-11T21:19:40.774 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:40 smithi023 bash[29146]: cluster 2024-03-11T21:19:39.266615+0000 osd.7 (osd.7) 711 : cluster 0 2.8 scrub starts 2024-03-11T21:19:40.774 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:40 smithi023 bash[29146]: cluster 2024-03-11T21:19:39.269006+0000 osd.7 (osd.7) 712 : cluster 0 2.8 scrub ok 2024-03-11T21:19:40.774 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:40 smithi023 bash[29146]: cluster 2024-03-11T21:19:40.427079+0000 osd.5 (osd.5) 683 : cluster 0 purged_snaps scrub starts 2024-03-11T21:19:40.774 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:40 smithi023 bash[29146]: cluster 2024-03-11T21:19:40.427178+0000 osd.5 (osd.5) 684 : cluster 0 purged_snaps scrub ok 2024-03-11T21:19:40.774 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:40 smithi023 bash[29146]: cluster 2024-03-11T21:19:40.433050+0000 osd.0 (osd.0) 522 : cluster 0 purged_snaps scrub starts 2024-03-11T21:19:40.775 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:40 smithi023 bash[29146]: cluster 2024-03-11T21:19:40.433149+0000 osd.0 (osd.0) 523 : cluster 0 purged_snaps scrub ok 2024-03-11T21:19:40.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:40 smithi023 bash[21372]: cluster 2024-03-11T21:19:39.036335+0000 osd.2 (osd.2) 315 : cluster 0 1.14 scrub starts 2024-03-11T21:19:40.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:40 smithi023 bash[21372]: cluster 2024-03-11T21:19:39.039225+0000 osd.2 (osd.2) 316 : cluster 0 1.14 scrub ok 2024-03-11T21:19:40.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:40 smithi023 bash[21372]: cluster 2024-03-11T21:19:39.266615+0000 osd.7 (osd.7) 711 : cluster 0 2.8 scrub starts 2024-03-11T21:19:40.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:40 smithi023 bash[21372]: cluster 2024-03-11T21:19:39.269006+0000 osd.7 (osd.7) 712 : cluster 0 2.8 scrub ok 2024-03-11T21:19:40.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:40 smithi023 bash[21372]: cluster 2024-03-11T21:19:40.427079+0000 osd.5 (osd.5) 683 : cluster 0 purged_snaps scrub starts 2024-03-11T21:19:40.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:40 smithi023 bash[21372]: cluster 2024-03-11T21:19:40.427178+0000 osd.5 (osd.5) 684 : cluster 0 purged_snaps scrub ok 2024-03-11T21:19:40.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:40 smithi023 bash[21372]: cluster 2024-03-11T21:19:40.433050+0000 osd.0 (osd.0) 522 : cluster 0 purged_snaps scrub starts 2024-03-11T21:19:40.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:40 smithi023 bash[21372]: cluster 2024-03-11T21:19:40.433149+0000 osd.0 (osd.0) 523 : cluster 0 purged_snaps scrub ok 2024-03-11T21:19:41.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:40 smithi146 bash[23809]: cluster 2024-03-11T21:19:39.036335+0000 osd.2 (osd.2) 315 : cluster 0 1.14 scrub starts 2024-03-11T21:19:41.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:40 smithi146 bash[23809]: cluster 2024-03-11T21:19:39.039225+0000 osd.2 (osd.2) 316 : cluster 0 1.14 scrub ok 2024-03-11T21:19:41.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:40 smithi146 bash[23809]: cluster 2024-03-11T21:19:39.266615+0000 osd.7 (osd.7) 711 : cluster 0 2.8 scrub starts 2024-03-11T21:19:41.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:40 smithi146 bash[23809]: cluster 2024-03-11T21:19:39.269006+0000 osd.7 (osd.7) 712 : cluster 0 2.8 scrub ok 2024-03-11T21:19:41.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:40 smithi146 bash[23809]: cluster 2024-03-11T21:19:40.427079+0000 osd.5 (osd.5) 683 : cluster 0 purged_snaps scrub starts 2024-03-11T21:19:41.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:40 smithi146 bash[23809]: cluster 2024-03-11T21:19:40.427178+0000 osd.5 (osd.5) 684 : cluster 0 purged_snaps scrub ok 2024-03-11T21:19:41.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:40 smithi146 bash[23809]: cluster 2024-03-11T21:19:40.433050+0000 osd.0 (osd.0) 522 : cluster 0 purged_snaps scrub starts 2024-03-11T21:19:41.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:40 smithi146 bash[23809]: cluster 2024-03-11T21:19:40.433149+0000 osd.0 (osd.0) 523 : cluster 0 purged_snaps scrub ok 2024-03-11T21:19:42.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:41 smithi146 bash[23809]: cluster 2024-03-11T21:19:40.253767+0000 osd.7 (osd.7) 713 : cluster 0 6.3 scrub starts 2024-03-11T21:19:42.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:41 smithi146 bash[23809]: cluster 2024-03-11T21:19:40.393134+0000 osd.7 (osd.7) 714 : cluster 0 6.3 scrub ok 2024-03-11T21:19:42.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:41 smithi146 bash[23809]: cluster 2024-03-11T21:19:40.579068+0000 mgr.y (mgr.24380) 1064 : cluster 0 pgmap v1237: 152 pgs: 1 active+clean+scrubbing, 151 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:19:42.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:41 smithi146 bash[23809]: cluster 2024-03-11T21:19:40.664715+0000 osd.6 (osd.6) 564 : cluster 0 4.5 scrub starts 2024-03-11T21:19:42.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:41 smithi146 bash[23809]: cluster 2024-03-11T21:19:40.667366+0000 osd.6 (osd.6) 565 : cluster 0 4.5 scrub ok 2024-03-11T21:19:42.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:41 smithi146 bash[23809]: cluster 2024-03-11T21:19:40.685815+0000 mon.a (mon.0) 2108 : cluster 0 osdmap e493: 8 total, 8 up, 8 in 2024-03-11T21:19:42.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:41 smithi023 bash[21372]: cluster 2024-03-11T21:19:40.253767+0000 osd.7 (osd.7) 713 : cluster 0 6.3 scrub starts 2024-03-11T21:19:42.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:41 smithi023 bash[21372]: cluster 2024-03-11T21:19:40.393134+0000 osd.7 (osd.7) 714 : cluster 0 6.3 scrub ok 2024-03-11T21:19:42.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:41 smithi023 bash[21372]: cluster 2024-03-11T21:19:40.579068+0000 mgr.y (mgr.24380) 1064 : cluster 0 pgmap v1237: 152 pgs: 1 active+clean+scrubbing, 151 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:19:42.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:41 smithi023 bash[21372]: cluster 2024-03-11T21:19:40.664715+0000 osd.6 (osd.6) 564 : cluster 0 4.5 scrub starts 2024-03-11T21:19:42.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:41 smithi023 bash[21372]: cluster 2024-03-11T21:19:40.667366+0000 osd.6 (osd.6) 565 : cluster 0 4.5 scrub ok 2024-03-11T21:19:42.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:41 smithi023 bash[21372]: cluster 2024-03-11T21:19:40.685815+0000 mon.a (mon.0) 2108 : cluster 0 osdmap e493: 8 total, 8 up, 8 in 2024-03-11T21:19:42.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:41 smithi023 bash[29146]: cluster 2024-03-11T21:19:40.253767+0000 osd.7 (osd.7) 713 : cluster 0 6.3 scrub starts 2024-03-11T21:19:42.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:41 smithi023 bash[29146]: cluster 2024-03-11T21:19:40.393134+0000 osd.7 (osd.7) 714 : cluster 0 6.3 scrub ok 2024-03-11T21:19:42.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:41 smithi023 bash[29146]: cluster 2024-03-11T21:19:40.579068+0000 mgr.y (mgr.24380) 1064 : cluster 0 pgmap v1237: 152 pgs: 1 active+clean+scrubbing, 151 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:19:42.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:41 smithi023 bash[29146]: cluster 2024-03-11T21:19:40.664715+0000 osd.6 (osd.6) 564 : cluster 0 4.5 scrub starts 2024-03-11T21:19:42.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:41 smithi023 bash[29146]: cluster 2024-03-11T21:19:40.667366+0000 osd.6 (osd.6) 565 : cluster 0 4.5 scrub ok 2024-03-11T21:19:42.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:41 smithi023 bash[29146]: cluster 2024-03-11T21:19:40.685815+0000 mon.a (mon.0) 2108 : cluster 0 osdmap e493: 8 total, 8 up, 8 in 2024-03-11T21:19:43.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:42 smithi146 bash[23809]: cluster 2024-03-11T21:19:41.659146+0000 osd.4 (osd.4) 155 : cluster 0 purged_snaps scrub starts 2024-03-11T21:19:43.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:42 smithi146 bash[23809]: cluster 2024-03-11T21:19:41.659287+0000 osd.4 (osd.4) 156 : cluster 0 purged_snaps scrub ok 2024-03-11T21:19:43.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:42 smithi146 bash[23809]: cluster 2024-03-11T21:19:41.723936+0000 osd.5 (osd.5) 685 : cluster 0 3.c scrub starts 2024-03-11T21:19:43.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:42 smithi146 bash[23809]: cluster 2024-03-11T21:19:41.727872+0000 osd.5 (osd.5) 686 : cluster 0 3.c scrub ok 2024-03-11T21:19:43.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:42 smithi146 bash[23809]: cluster 2024-03-11T21:19:42.536624+0000 mon.a (mon.0) 2109 : cluster 0 osdmap e494: 8 total, 8 up, 8 in 2024-03-11T21:19:43.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:42 smithi023 bash[21372]: cluster 2024-03-11T21:19:41.659146+0000 osd.4 (osd.4) 155 : cluster 0 purged_snaps scrub starts 2024-03-11T21:19:43.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:42 smithi023 bash[21372]: cluster 2024-03-11T21:19:41.659287+0000 osd.4 (osd.4) 156 : cluster 0 purged_snaps scrub ok 2024-03-11T21:19:43.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:42 smithi023 bash[21372]: cluster 2024-03-11T21:19:41.723936+0000 osd.5 (osd.5) 685 : cluster 0 3.c scrub starts 2024-03-11T21:19:43.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:42 smithi023 bash[21372]: cluster 2024-03-11T21:19:41.727872+0000 osd.5 (osd.5) 686 : cluster 0 3.c scrub ok 2024-03-11T21:19:43.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:42 smithi023 bash[21372]: cluster 2024-03-11T21:19:42.536624+0000 mon.a (mon.0) 2109 : cluster 0 osdmap e494: 8 total, 8 up, 8 in 2024-03-11T21:19:43.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:42 smithi023 bash[29146]: cluster 2024-03-11T21:19:41.659146+0000 osd.4 (osd.4) 155 : cluster 0 purged_snaps scrub starts 2024-03-11T21:19:43.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:42 smithi023 bash[29146]: cluster 2024-03-11T21:19:41.659287+0000 osd.4 (osd.4) 156 : cluster 0 purged_snaps scrub ok 2024-03-11T21:19:43.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:42 smithi023 bash[29146]: cluster 2024-03-11T21:19:41.723936+0000 osd.5 (osd.5) 685 : cluster 0 3.c scrub starts 2024-03-11T21:19:43.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:42 smithi023 bash[29146]: cluster 2024-03-11T21:19:41.727872+0000 osd.5 (osd.5) 686 : cluster 0 3.c scrub ok 2024-03-11T21:19:43.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:42 smithi023 bash[29146]: cluster 2024-03-11T21:19:42.536624+0000 mon.a (mon.0) 2109 : cluster 0 osdmap e494: 8 total, 8 up, 8 in 2024-03-11T21:19:43.764 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:43 smithi023 bash[21372]: cluster 2024-03-11T21:19:42.224787+0000 osd.7 (osd.7) 715 : cluster 0 2.5 scrub starts 2024-03-11T21:19:43.765 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:43 smithi023 bash[21372]: cluster 2024-03-11T21:19:42.227186+0000 osd.7 (osd.7) 716 : cluster 0 2.5 scrub ok 2024-03-11T21:19:43.766 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:43 smithi023 bash[29146]: cluster 2024-03-11T21:19:42.224787+0000 osd.7 (osd.7) 715 : cluster 0 2.5 scrub starts 2024-03-11T21:19:43.766 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:43 smithi023 bash[29146]: cluster 2024-03-11T21:19:42.227186+0000 osd.7 (osd.7) 716 : cluster 0 2.5 scrub ok 2024-03-11T21:19:43.766 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:43 smithi023 bash[29146]: cluster 2024-03-11T21:19:42.433629+0000 osd.3 (osd.3) 638 : cluster 0 4.10 deep-scrub starts 2024-03-11T21:19:43.766 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:43 smithi023 bash[29146]: cluster 2024-03-11T21:19:42.436139+0000 osd.3 (osd.3) 639 : cluster 0 4.10 deep-scrub ok 2024-03-11T21:19:43.766 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:43 smithi023 bash[29146]: cluster 2024-03-11T21:19:42.579775+0000 mgr.y (mgr.24380) 1065 : cluster 0 pgmap v1240: 152 pgs: 1 active+clean+scrubbing, 151 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:19:43.766 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:43 smithi023 bash[29146]: cluster 2024-03-11T21:19:43.158152+0000 osd.1 (osd.1) 446 : cluster 0 6.1 deep-scrub starts 2024-03-11T21:19:43.766 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:43 smithi023 bash[29146]: cluster 2024-03-11T21:19:43.246782+0000 osd.1 (osd.1) 447 : cluster 0 6.1 deep-scrub ok 2024-03-11T21:19:44.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:43 smithi146 bash[23809]: cluster 2024-03-11T21:19:42.224787+0000 osd.7 (osd.7) 715 : cluster 0 2.5 scrub starts 2024-03-11T21:19:44.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:43 smithi146 bash[23809]: cluster 2024-03-11T21:19:42.227186+0000 osd.7 (osd.7) 716 : cluster 0 2.5 scrub ok 2024-03-11T21:19:44.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:43 smithi146 bash[23809]: cluster 2024-03-11T21:19:42.433629+0000 osd.3 (osd.3) 638 : cluster 0 4.10 deep-scrub starts 2024-03-11T21:19:44.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:43 smithi146 bash[23809]: cluster 2024-03-11T21:19:42.436139+0000 osd.3 (osd.3) 639 : cluster 0 4.10 deep-scrub ok 2024-03-11T21:19:44.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:43 smithi146 bash[23809]: cluster 2024-03-11T21:19:42.579775+0000 mgr.y (mgr.24380) 1065 : cluster 0 pgmap v1240: 152 pgs: 1 active+clean+scrubbing, 151 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:19:44.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:43 smithi146 bash[23809]: cluster 2024-03-11T21:19:43.158152+0000 osd.1 (osd.1) 446 : cluster 0 6.1 deep-scrub starts 2024-03-11T21:19:44.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:43 smithi146 bash[23809]: cluster 2024-03-11T21:19:43.246782+0000 osd.1 (osd.1) 447 : cluster 0 6.1 deep-scrub ok 2024-03-11T21:19:44.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:43 smithi023 bash[21372]: cluster 2024-03-11T21:19:42.433629+0000 osd.3 (osd.3) 638 : cluster 0 4.10 deep-scrub starts 2024-03-11T21:19:44.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:43 smithi023 bash[21372]: cluster 2024-03-11T21:19:42.436139+0000 osd.3 (osd.3) 639 : cluster 0 4.10 deep-scrub ok 2024-03-11T21:19:44.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:43 smithi023 bash[21372]: cluster 2024-03-11T21:19:42.579775+0000 mgr.y (mgr.24380) 1065 : cluster 0 pgmap v1240: 152 pgs: 1 active+clean+scrubbing, 151 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:19:44.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:43 smithi023 bash[21372]: cluster 2024-03-11T21:19:43.158152+0000 osd.1 (osd.1) 446 : cluster 0 6.1 deep-scrub starts 2024-03-11T21:19:44.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:43 smithi023 bash[21372]: cluster 2024-03-11T21:19:43.246782+0000 osd.1 (osd.1) 447 : cluster 0 6.1 deep-scrub ok 2024-03-11T21:19:45.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:44 smithi146 bash[23809]: cluster 2024-03-11T21:19:44.137903+0000 osd.1 (osd.1) 448 : cluster 0 2.9 scrub starts 2024-03-11T21:19:45.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:44 smithi146 bash[23809]: cluster 2024-03-11T21:19:44.140433+0000 osd.1 (osd.1) 449 : cluster 0 2.9 scrub ok 2024-03-11T21:19:45.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:44 smithi023 bash[29146]: cluster 2024-03-11T21:19:44.137903+0000 osd.1 (osd.1) 448 : cluster 0 2.9 scrub starts 2024-03-11T21:19:45.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:44 smithi023 bash[29146]: cluster 2024-03-11T21:19:44.140433+0000 osd.1 (osd.1) 449 : cluster 0 2.9 scrub ok 2024-03-11T21:19:45.124 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:44 smithi023 bash[21372]: cluster 2024-03-11T21:19:44.137903+0000 osd.1 (osd.1) 448 : cluster 0 2.9 scrub starts 2024-03-11T21:19:45.124 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:44 smithi023 bash[21372]: cluster 2024-03-11T21:19:44.140433+0000 osd.1 (osd.1) 449 : cluster 0 2.9 scrub ok 2024-03-11T21:19:46.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:45 smithi146 bash[23809]: cluster 2024-03-11T21:19:44.435986+0000 osd.4 (osd.4) 157 : cluster 0 3.1a scrub starts 2024-03-11T21:19:46.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:45 smithi146 bash[23809]: cluster 2024-03-11T21:19:44.440043+0000 osd.4 (osd.4) 158 : cluster 0 3.1a scrub ok 2024-03-11T21:19:46.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:45 smithi146 bash[23809]: cluster 2024-03-11T21:19:44.580635+0000 mgr.y (mgr.24380) 1066 : cluster 0 pgmap v1241: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:19:46.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:45 smithi146 bash[23809]: cluster 2024-03-11T21:19:45.104570+0000 osd.1 (osd.1) 450 : cluster 0 4.2 scrub starts 2024-03-11T21:19:46.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:45 smithi146 bash[23809]: cluster 2024-03-11T21:19:45.107175+0000 osd.1 (osd.1) 451 : cluster 0 4.2 scrub ok 2024-03-11T21:19:46.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:45 smithi146 bash[23809]: audit 2024-03-11T21:19:45.529317+0000 mon.a (mon.0) 2110 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:19:46.023 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:45 smithi023 bash[29146]: cluster 2024-03-11T21:19:44.435986+0000 osd.4 (osd.4) 157 : cluster 0 3.1a scrub starts 2024-03-11T21:19:46.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:45 smithi023 bash[29146]: cluster 2024-03-11T21:19:44.440043+0000 osd.4 (osd.4) 158 : cluster 0 3.1a scrub ok 2024-03-11T21:19:46.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:45 smithi023 bash[29146]: cluster 2024-03-11T21:19:44.580635+0000 mgr.y (mgr.24380) 1066 : cluster 0 pgmap v1241: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:19:46.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:45 smithi023 bash[29146]: cluster 2024-03-11T21:19:45.104570+0000 osd.1 (osd.1) 450 : cluster 0 4.2 scrub starts 2024-03-11T21:19:46.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:45 smithi023 bash[29146]: cluster 2024-03-11T21:19:45.107175+0000 osd.1 (osd.1) 451 : cluster 0 4.2 scrub ok 2024-03-11T21:19:46.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:45 smithi023 bash[29146]: audit 2024-03-11T21:19:45.529317+0000 mon.a (mon.0) 2110 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:19:46.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:45 smithi023 bash[21372]: cluster 2024-03-11T21:19:44.435986+0000 osd.4 (osd.4) 157 : cluster 0 3.1a scrub starts 2024-03-11T21:19:46.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:45 smithi023 bash[21372]: cluster 2024-03-11T21:19:44.440043+0000 osd.4 (osd.4) 158 : cluster 0 3.1a scrub ok 2024-03-11T21:19:46.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:45 smithi023 bash[21372]: cluster 2024-03-11T21:19:44.580635+0000 mgr.y (mgr.24380) 1066 : cluster 0 pgmap v1241: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:19:46.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:45 smithi023 bash[21372]: cluster 2024-03-11T21:19:45.104570+0000 osd.1 (osd.1) 450 : cluster 0 4.2 scrub starts 2024-03-11T21:19:46.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:45 smithi023 bash[21372]: cluster 2024-03-11T21:19:45.107175+0000 osd.1 (osd.1) 451 : cluster 0 4.2 scrub ok 2024-03-11T21:19:46.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:45 smithi023 bash[21372]: audit 2024-03-11T21:19:45.529317+0000 mon.a (mon.0) 2110 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:19:46.713 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:19:46 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:19:47.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:46 smithi146 bash[23809]: audit 2024-03-11T21:19:46.262122+0000 mon.a (mon.0) 2111 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:19:47.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:46 smithi146 bash[23809]: audit 2024-03-11T21:19:46.263925+0000 mon.a (mon.0) 2112 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:19:47.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:46 smithi146 bash[23809]: audit 2024-03-11T21:19:46.282179+0000 mon.a (mon.0) 2113 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:19:47.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:46 smithi023 bash[21372]: audit 2024-03-11T21:19:46.262122+0000 mon.a (mon.0) 2111 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:19:47.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:46 smithi023 bash[21372]: audit 2024-03-11T21:19:46.263925+0000 mon.a (mon.0) 2112 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:19:47.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:46 smithi023 bash[21372]: audit 2024-03-11T21:19:46.282179+0000 mon.a (mon.0) 2113 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:19:47.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:46 smithi023 bash[29146]: audit 2024-03-11T21:19:46.262122+0000 mon.a (mon.0) 2111 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:19:47.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:46 smithi023 bash[29146]: audit 2024-03-11T21:19:46.263925+0000 mon.a (mon.0) 2112 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:19:47.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:46 smithi023 bash[29146]: audit 2024-03-11T21:19:46.282179+0000 mon.a (mon.0) 2113 : audit 1 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' 2024-03-11T21:19:47.989 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:47 smithi023 bash[21372]: audit 2024-03-11T21:19:46.335695+0000 mgr.y (mgr.24380) 1067 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:19:47.989 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:47 smithi023 bash[21372]: cluster 2024-03-11T21:19:46.489070+0000 osd.3 (osd.3) 640 : cluster 0 1.c scrub starts 2024-03-11T21:19:47.989 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:47 smithi023 bash[21372]: cluster 2024-03-11T21:19:46.491118+0000 osd.3 (osd.3) 641 : cluster 0 1.c scrub ok 2024-03-11T21:19:47.989 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:47 smithi023 bash[21372]: cluster 2024-03-11T21:19:46.582164+0000 mgr.y (mgr.24380) 1068 : cluster 0 pgmap v1242: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-11T21:19:47.990 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:47 smithi023 bash[29146]: audit 2024-03-11T21:19:46.335695+0000 mgr.y (mgr.24380) 1067 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:19:47.990 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:47 smithi023 bash[29146]: cluster 2024-03-11T21:19:46.489070+0000 osd.3 (osd.3) 640 : cluster 0 1.c scrub starts 2024-03-11T21:19:47.990 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:47 smithi023 bash[29146]: cluster 2024-03-11T21:19:46.491118+0000 osd.3 (osd.3) 641 : cluster 0 1.c scrub ok 2024-03-11T21:19:47.990 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:47 smithi023 bash[29146]: cluster 2024-03-11T21:19:46.582164+0000 mgr.y (mgr.24380) 1068 : cluster 0 pgmap v1242: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-11T21:19:48.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:47 smithi146 bash[23809]: audit 2024-03-11T21:19:46.335695+0000 mgr.y (mgr.24380) 1067 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:19:48.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:47 smithi146 bash[23809]: cluster 2024-03-11T21:19:46.489070+0000 osd.3 (osd.3) 640 : cluster 0 1.c scrub starts 2024-03-11T21:19:48.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:47 smithi146 bash[23809]: cluster 2024-03-11T21:19:46.491118+0000 osd.3 (osd.3) 641 : cluster 0 1.c scrub ok 2024-03-11T21:19:48.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:47 smithi146 bash[23809]: cluster 2024-03-11T21:19:46.582164+0000 mgr.y (mgr.24380) 1068 : cluster 0 pgmap v1242: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-11T21:19:48.728 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:19:48 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:19:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:19:48.992 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:48 smithi023 bash[21372]: cluster 2024-03-11T21:19:47.250795+0000 osd.7 (osd.7) 717 : cluster 0 3.11 deep-scrub starts 2024-03-11T21:19:48.993 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:48 smithi023 bash[21372]: cluster 2024-03-11T21:19:47.256192+0000 osd.7 (osd.7) 718 : cluster 0 3.11 deep-scrub ok 2024-03-11T21:19:48.993 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:48 smithi023 bash[21372]: cluster 2024-03-11T21:19:47.427778+0000 osd.4 (osd.4) 159 : cluster 0 3.9 deep-scrub starts 2024-03-11T21:19:48.993 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:48 smithi023 bash[21372]: cluster 2024-03-11T21:19:47.432480+0000 osd.4 (osd.4) 160 : cluster 0 3.9 deep-scrub ok 2024-03-11T21:19:48.993 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:48 smithi023 bash[21372]: cluster 2024-03-11T21:19:47.617504+0000 osd.5 (osd.5) 687 : cluster 0 4.8 scrub starts 2024-03-11T21:19:48.993 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:48 smithi023 bash[21372]: cluster 2024-03-11T21:19:47.619905+0000 osd.5 (osd.5) 688 : cluster 0 4.8 scrub ok 2024-03-11T21:19:48.993 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:48 smithi023 bash[21372]: cluster 2024-03-11T21:19:47.696945+0000 osd.0 (osd.0) 524 : cluster 0 3.1f scrub starts 2024-03-11T21:19:48.993 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:48 smithi023 bash[21372]: cluster 2024-03-11T21:19:47.699219+0000 osd.0 (osd.0) 525 : cluster 0 3.1f scrub ok 2024-03-11T21:19:48.993 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:48 smithi023 bash[29146]: cluster 2024-03-11T21:19:47.250795+0000 osd.7 (osd.7) 717 : cluster 0 3.11 deep-scrub starts 2024-03-11T21:19:48.993 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:48 smithi023 bash[29146]: cluster 2024-03-11T21:19:47.256192+0000 osd.7 (osd.7) 718 : cluster 0 3.11 deep-scrub ok 2024-03-11T21:19:48.993 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:48 smithi023 bash[29146]: cluster 2024-03-11T21:19:47.427778+0000 osd.4 (osd.4) 159 : cluster 0 3.9 deep-scrub starts 2024-03-11T21:19:48.994 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:48 smithi023 bash[29146]: cluster 2024-03-11T21:19:47.432480+0000 osd.4 (osd.4) 160 : cluster 0 3.9 deep-scrub ok 2024-03-11T21:19:48.994 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:48 smithi023 bash[29146]: cluster 2024-03-11T21:19:47.617504+0000 osd.5 (osd.5) 687 : cluster 0 4.8 scrub starts 2024-03-11T21:19:48.994 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:48 smithi023 bash[29146]: cluster 2024-03-11T21:19:47.619905+0000 osd.5 (osd.5) 688 : cluster 0 4.8 scrub ok 2024-03-11T21:19:48.994 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:48 smithi023 bash[29146]: cluster 2024-03-11T21:19:47.696945+0000 osd.0 (osd.0) 524 : cluster 0 3.1f scrub starts 2024-03-11T21:19:48.994 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:48 smithi023 bash[29146]: cluster 2024-03-11T21:19:47.699219+0000 osd.0 (osd.0) 525 : cluster 0 3.1f scrub ok 2024-03-11T21:19:49.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:48 smithi146 bash[23809]: cluster 2024-03-11T21:19:47.250795+0000 osd.7 (osd.7) 717 : cluster 0 3.11 deep-scrub starts 2024-03-11T21:19:49.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:48 smithi146 bash[23809]: cluster 2024-03-11T21:19:47.256192+0000 osd.7 (osd.7) 718 : cluster 0 3.11 deep-scrub ok 2024-03-11T21:19:49.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:48 smithi146 bash[23809]: cluster 2024-03-11T21:19:47.427778+0000 osd.4 (osd.4) 159 : cluster 0 3.9 deep-scrub starts 2024-03-11T21:19:49.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:48 smithi146 bash[23809]: cluster 2024-03-11T21:19:47.432480+0000 osd.4 (osd.4) 160 : cluster 0 3.9 deep-scrub ok 2024-03-11T21:19:49.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:48 smithi146 bash[23809]: cluster 2024-03-11T21:19:47.617504+0000 osd.5 (osd.5) 687 : cluster 0 4.8 scrub starts 2024-03-11T21:19:49.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:48 smithi146 bash[23809]: cluster 2024-03-11T21:19:47.619905+0000 osd.5 (osd.5) 688 : cluster 0 4.8 scrub ok 2024-03-11T21:19:49.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:48 smithi146 bash[23809]: cluster 2024-03-11T21:19:47.696945+0000 osd.0 (osd.0) 524 : cluster 0 3.1f scrub starts 2024-03-11T21:19:49.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:48 smithi146 bash[23809]: cluster 2024-03-11T21:19:47.699219+0000 osd.0 (osd.0) 525 : cluster 0 3.1f scrub ok 2024-03-11T21:19:50.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:49 smithi146 bash[23809]: cluster 2024-03-11T21:19:48.463261+0000 osd.4 (osd.4) 161 : cluster 0 4.13 deep-scrub starts 2024-03-11T21:19:50.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:49 smithi146 bash[23809]: cluster 2024-03-11T21:19:48.465604+0000 osd.4 (osd.4) 162 : cluster 0 4.13 deep-scrub ok 2024-03-11T21:19:50.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:49 smithi146 bash[23809]: cluster 2024-03-11T21:19:48.583255+0000 mgr.y (mgr.24380) 1069 : cluster 0 pgmap v1243: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:19:50.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:49 smithi023 bash[21372]: cluster 2024-03-11T21:19:48.463261+0000 osd.4 (osd.4) 161 : cluster 0 4.13 deep-scrub starts 2024-03-11T21:19:50.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:49 smithi023 bash[21372]: cluster 2024-03-11T21:19:48.465604+0000 osd.4 (osd.4) 162 : cluster 0 4.13 deep-scrub ok 2024-03-11T21:19:50.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:49 smithi023 bash[21372]: cluster 2024-03-11T21:19:48.583255+0000 mgr.y (mgr.24380) 1069 : cluster 0 pgmap v1243: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:19:50.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:49 smithi023 bash[29146]: cluster 2024-03-11T21:19:48.463261+0000 osd.4 (osd.4) 161 : cluster 0 4.13 deep-scrub starts 2024-03-11T21:19:50.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:49 smithi023 bash[29146]: cluster 2024-03-11T21:19:48.465604+0000 osd.4 (osd.4) 162 : cluster 0 4.13 deep-scrub ok 2024-03-11T21:19:50.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:49 smithi023 bash[29146]: cluster 2024-03-11T21:19:48.583255+0000 mgr.y (mgr.24380) 1069 : cluster 0 pgmap v1243: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:19:50.299 INFO:teuthology.orchestra.run.smithi023.stdout:906238099685 2024-03-11T21:19:50.300 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-11T21:19:51.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:50 smithi146 bash[23809]: cluster 2024-03-11T21:19:49.764791+0000 osd.6 (osd.6) 566 : cluster 0 4.a scrub starts 2024-03-11T21:19:51.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:50 smithi146 bash[23809]: cluster 2024-03-11T21:19:49.767561+0000 osd.6 (osd.6) 567 : cluster 0 4.a scrub ok 2024-03-11T21:19:51.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:50 smithi146 bash[23809]: cluster 2024-03-11T21:19:50.114452+0000 osd.1 (osd.1) 452 : cluster 0 2.15 scrub starts 2024-03-11T21:19:51.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:50 smithi146 bash[23809]: cluster 2024-03-11T21:19:50.116911+0000 osd.1 (osd.1) 453 : cluster 0 2.15 scrub ok 2024-03-11T21:19:51.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:50 smithi023 bash[21372]: cluster 2024-03-11T21:19:49.764791+0000 osd.6 (osd.6) 566 : cluster 0 4.a scrub starts 2024-03-11T21:19:51.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:50 smithi023 bash[21372]: cluster 2024-03-11T21:19:49.767561+0000 osd.6 (osd.6) 567 : cluster 0 4.a scrub ok 2024-03-11T21:19:51.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:50 smithi023 bash[21372]: cluster 2024-03-11T21:19:50.114452+0000 osd.1 (osd.1) 452 : cluster 0 2.15 scrub starts 2024-03-11T21:19:51.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:50 smithi023 bash[21372]: cluster 2024-03-11T21:19:50.116911+0000 osd.1 (osd.1) 453 : cluster 0 2.15 scrub ok 2024-03-11T21:19:51.274 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:50 smithi023 bash[29146]: cluster 2024-03-11T21:19:49.764791+0000 osd.6 (osd.6) 566 : cluster 0 4.a scrub starts 2024-03-11T21:19:51.274 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:50 smithi023 bash[29146]: cluster 2024-03-11T21:19:49.767561+0000 osd.6 (osd.6) 567 : cluster 0 4.a scrub ok 2024-03-11T21:19:51.275 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:50 smithi023 bash[29146]: cluster 2024-03-11T21:19:50.114452+0000 osd.1 (osd.1) 452 : cluster 0 2.15 scrub starts 2024-03-11T21:19:51.275 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:50 smithi023 bash[29146]: cluster 2024-03-11T21:19:50.116911+0000 osd.1 (osd.1) 453 : cluster 0 2.15 scrub ok 2024-03-11T21:19:51.890 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:51 smithi023 bash[21372]: cluster 2024-03-11T21:19:50.505001+0000 osd.4 (osd.4) 163 : cluster 0 6.17 scrub starts 2024-03-11T21:19:51.890 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:51 smithi023 bash[21372]: cluster 2024-03-11T21:19:50.583818+0000 mgr.y (mgr.24380) 1070 : cluster 0 pgmap v1244: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T21:19:51.890 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:51 smithi023 bash[21372]: cluster 2024-03-11T21:19:50.693370+0000 osd.4 (osd.4) 164 : cluster 0 6.17 scrub ok 2024-03-11T21:19:52.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:51 smithi146 bash[23809]: cluster 2024-03-11T21:19:50.505001+0000 osd.4 (osd.4) 163 : cluster 0 6.17 scrub starts 2024-03-11T21:19:52.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:51 smithi146 bash[23809]: cluster 2024-03-11T21:19:50.583818+0000 mgr.y (mgr.24380) 1070 : cluster 0 pgmap v1244: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T21:19:52.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:51 smithi146 bash[23809]: cluster 2024-03-11T21:19:50.693370+0000 osd.4 (osd.4) 164 : cluster 0 6.17 scrub ok 2024-03-11T21:19:52.274 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:51 smithi023 bash[29146]: cluster 2024-03-11T21:19:50.505001+0000 osd.4 (osd.4) 163 : cluster 0 6.17 scrub starts 2024-03-11T21:19:52.274 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:51 smithi023 bash[29146]: cluster 2024-03-11T21:19:50.583818+0000 mgr.y (mgr.24380) 1070 : cluster 0 pgmap v1244: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T21:19:52.274 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:51 smithi023 bash[29146]: cluster 2024-03-11T21:19:50.693370+0000 osd.4 (osd.4) 164 : cluster 0 6.17 scrub ok 2024-03-11T21:19:53.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:52 smithi023 bash[21372]: cluster 2024-03-11T21:19:51.674802+0000 osd.0 (osd.0) 526 : cluster 0 4.3 scrub starts 2024-03-11T21:19:53.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:52 smithi023 bash[21372]: cluster 2024-03-11T21:19:51.676973+0000 osd.0 (osd.0) 527 : cluster 0 4.3 scrub ok 2024-03-11T21:19:53.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:52 smithi023 bash[21372]: audit 2024-03-11T21:19:52.121782+0000 mon.a (mon.0) 2114 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:19:53.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:52 smithi023 bash[29146]: cluster 2024-03-11T21:19:51.674802+0000 osd.0 (osd.0) 526 : cluster 0 4.3 scrub starts 2024-03-11T21:19:53.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:52 smithi023 bash[29146]: cluster 2024-03-11T21:19:51.676973+0000 osd.0 (osd.0) 527 : cluster 0 4.3 scrub ok 2024-03-11T21:19:53.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:52 smithi023 bash[29146]: audit 2024-03-11T21:19:52.121782+0000 mon.a (mon.0) 2114 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:19:53.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:52 smithi146 bash[23809]: cluster 2024-03-11T21:19:51.674802+0000 osd.0 (osd.0) 526 : cluster 0 4.3 scrub starts 2024-03-11T21:19:53.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:52 smithi146 bash[23809]: cluster 2024-03-11T21:19:51.676973+0000 osd.0 (osd.0) 527 : cluster 0 4.3 scrub ok 2024-03-11T21:19:53.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:52 smithi146 bash[23809]: audit 2024-03-11T21:19:52.121782+0000 mon.a (mon.0) 2114 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:19:53.585 INFO:teuthology.orchestra.run.smithi023.stdout:81604379038 2024-03-11T21:19:53.586 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-11T21:19:53.635 INFO:teuthology.orchestra.run.smithi023.stdout:876173328618 2024-03-11T21:19:53.636 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-11T21:19:53.841 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:53 smithi023 bash[21372]: cluster 2024-03-11T21:19:52.584632+0000 mgr.y (mgr.24380) 1071 : cluster 0 pgmap v1245: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1018 B/s rd, 0 op/s 2024-03-11T21:19:53.841 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:53 smithi023 bash[21372]: cluster 2024-03-11T21:19:52.693169+0000 osd.0 (osd.0) 528 : cluster 0 4.16 scrub starts 2024-03-11T21:19:53.841 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:53 smithi023 bash[21372]: cluster 2024-03-11T21:19:52.695662+0000 osd.0 (osd.0) 529 : cluster 0 4.16 scrub ok 2024-03-11T21:19:53.841 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:53 smithi023 bash[21372]: cluster 2024-03-11T21:19:53.193670+0000 osd.1 (osd.1) 454 : cluster 0 3.19 scrub starts 2024-03-11T21:19:53.841 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:53 smithi023 bash[21372]: cluster 2024-03-11T21:19:53.198382+0000 osd.1 (osd.1) 455 : cluster 0 3.19 scrub ok 2024-03-11T21:19:53.842 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:53 smithi023 bash[29146]: cluster 2024-03-11T21:19:52.584632+0000 mgr.y (mgr.24380) 1071 : cluster 0 pgmap v1245: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1018 B/s rd, 0 op/s 2024-03-11T21:19:53.842 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:53 smithi023 bash[29146]: cluster 2024-03-11T21:19:52.693169+0000 osd.0 (osd.0) 528 : cluster 0 4.16 scrub starts 2024-03-11T21:19:53.842 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:53 smithi023 bash[29146]: cluster 2024-03-11T21:19:52.695662+0000 osd.0 (osd.0) 529 : cluster 0 4.16 scrub ok 2024-03-11T21:19:53.842 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:53 smithi023 bash[29146]: cluster 2024-03-11T21:19:53.193670+0000 osd.1 (osd.1) 454 : cluster 0 3.19 scrub starts 2024-03-11T21:19:53.842 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:53 smithi023 bash[29146]: cluster 2024-03-11T21:19:53.198382+0000 osd.1 (osd.1) 455 : cluster 0 3.19 scrub ok 2024-03-11T21:19:54.011 INFO:teuthology.orchestra.run.smithi023.stdout:292057776492 2024-03-11T21:19:54.012 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-03-11T21:19:54.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:53 smithi146 bash[23809]: cluster 2024-03-11T21:19:52.584632+0000 mgr.y (mgr.24380) 1071 : cluster 0 pgmap v1245: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1018 B/s rd, 0 op/s 2024-03-11T21:19:54.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:53 smithi146 bash[23809]: cluster 2024-03-11T21:19:52.693169+0000 osd.0 (osd.0) 528 : cluster 0 4.16 scrub starts 2024-03-11T21:19:54.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:53 smithi146 bash[23809]: cluster 2024-03-11T21:19:52.695662+0000 osd.0 (osd.0) 529 : cluster 0 4.16 scrub ok 2024-03-11T21:19:54.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:53 smithi146 bash[23809]: cluster 2024-03-11T21:19:53.193670+0000 osd.1 (osd.1) 454 : cluster 0 3.19 scrub starts 2024-03-11T21:19:54.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:53 smithi146 bash[23809]: cluster 2024-03-11T21:19:53.198382+0000 osd.1 (osd.1) 455 : cluster 0 3.19 scrub ok 2024-03-11T21:19:55.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:54 smithi023 bash[21372]: cluster 2024-03-11T21:19:53.762688+0000 osd.6 (osd.6) 568 : cluster 0 2.14 scrub starts 2024-03-11T21:19:55.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:54 smithi023 bash[21372]: cluster 2024-03-11T21:19:53.765208+0000 osd.6 (osd.6) 569 : cluster 0 2.14 scrub ok 2024-03-11T21:19:55.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:54 smithi023 bash[29146]: cluster 2024-03-11T21:19:53.762688+0000 osd.6 (osd.6) 568 : cluster 0 2.14 scrub starts 2024-03-11T21:19:55.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:54 smithi023 bash[29146]: cluster 2024-03-11T21:19:53.765208+0000 osd.6 (osd.6) 569 : cluster 0 2.14 scrub ok 2024-03-11T21:19:55.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:54 smithi146 bash[23809]: cluster 2024-03-11T21:19:53.762688+0000 osd.6 (osd.6) 568 : cluster 0 2.14 scrub starts 2024-03-11T21:19:55.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:54 smithi146 bash[23809]: cluster 2024-03-11T21:19:53.765208+0000 osd.6 (osd.6) 569 : cluster 0 2.14 scrub ok 2024-03-11T21:19:55.779 INFO:teuthology.orchestra.run.smithi023.stdout:188978561408 2024-03-11T21:19:55.779 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-11T21:19:55.783 INFO:teuthology.orchestra.run.smithi023.stdout:244813136246 2024-03-11T21:19:55.783 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-03-11T21:19:56.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:55 smithi023 bash[21372]: cluster 2024-03-11T21:19:54.472758+0000 osd.3 (osd.3) 642 : cluster 0 3.8 scrub starts 2024-03-11T21:19:56.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:55 smithi023 bash[21372]: cluster 2024-03-11T21:19:54.479730+0000 osd.3 (osd.3) 643 : cluster 0 3.8 scrub ok 2024-03-11T21:19:56.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:55 smithi023 bash[21372]: cluster 2024-03-11T21:19:54.585808+0000 mgr.y (mgr.24380) 1072 : cluster 0 pgmap v1246: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T21:19:56.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:55 smithi023 bash[29146]: cluster 2024-03-11T21:19:54.472758+0000 osd.3 (osd.3) 642 : cluster 0 3.8 scrub starts 2024-03-11T21:19:56.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:55 smithi023 bash[29146]: cluster 2024-03-11T21:19:54.479730+0000 osd.3 (osd.3) 643 : cluster 0 3.8 scrub ok 2024-03-11T21:19:56.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:55 smithi023 bash[29146]: cluster 2024-03-11T21:19:54.585808+0000 mgr.y (mgr.24380) 1072 : cluster 0 pgmap v1246: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T21:19:56.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:55 smithi146 bash[23809]: cluster 2024-03-11T21:19:54.472758+0000 osd.3 (osd.3) 642 : cluster 0 3.8 scrub starts 2024-03-11T21:19:56.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:55 smithi146 bash[23809]: cluster 2024-03-11T21:19:54.479730+0000 osd.3 (osd.3) 643 : cluster 0 3.8 scrub ok 2024-03-11T21:19:56.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:55 smithi146 bash[23809]: cluster 2024-03-11T21:19:54.585808+0000 mgr.y (mgr.24380) 1072 : cluster 0 pgmap v1246: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T21:19:56.607 INFO:teuthology.orchestra.run.smithi023.stdout:120259084692 2024-03-11T21:19:56.607 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-11T21:19:56.752 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:19:56 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:19:56.816 INFO:teuthology.orchestra.run.smithi023.stdout:34359738804 2024-03-11T21:19:56.817 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-11T21:19:57.023 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:56 smithi023 bash[21372]: cluster 2024-03-11T21:19:55.363108+0000 osd.4 (osd.4) 165 : cluster 0 4.e scrub starts 2024-03-11T21:19:57.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:56 smithi023 bash[21372]: cluster 2024-03-11T21:19:55.366195+0000 osd.4 (osd.4) 166 : cluster 0 4.e scrub ok 2024-03-11T21:19:57.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:56 smithi023 bash[29146]: cluster 2024-03-11T21:19:55.363108+0000 osd.4 (osd.4) 165 : cluster 0 4.e scrub starts 2024-03-11T21:19:57.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:56 smithi023 bash[29146]: cluster 2024-03-11T21:19:55.366195+0000 osd.4 (osd.4) 166 : cluster 0 4.e scrub ok 2024-03-11T21:19:57.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:56 smithi146 bash[23809]: cluster 2024-03-11T21:19:55.363108+0000 osd.4 (osd.4) 165 : cluster 0 4.e scrub starts 2024-03-11T21:19:57.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:56 smithi146 bash[23809]: cluster 2024-03-11T21:19:55.366195+0000 osd.4 (osd.4) 166 : cluster 0 4.e scrub ok 2024-03-11T21:19:58.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:57 smithi146 bash[23809]: audit 2024-03-11T21:19:56.347005+0000 mgr.y (mgr.24380) 1073 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:19:58.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:57 smithi146 bash[23809]: cluster 2024-03-11T21:19:56.410084+0000 osd.4 (osd.4) 167 : cluster 0 4.1a deep-scrub starts 2024-03-11T21:19:58.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:57 smithi146 bash[23809]: cluster 2024-03-11T21:19:56.412623+0000 osd.4 (osd.4) 168 : cluster 0 4.1a deep-scrub ok 2024-03-11T21:19:58.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:57 smithi146 bash[23809]: cluster 2024-03-11T21:19:56.490956+0000 osd.3 (osd.3) 644 : cluster 0 3.18 scrub starts 2024-03-11T21:19:58.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:57 smithi146 bash[23809]: cluster 2024-03-11T21:19:56.494963+0000 osd.3 (osd.3) 645 : cluster 0 3.18 scrub ok 2024-03-11T21:19:58.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:57 smithi146 bash[23809]: cluster 2024-03-11T21:19:56.587160+0000 mgr.y (mgr.24380) 1074 : cluster 0 pgmap v1247: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:19:58.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:57 smithi146 bash[23809]: cluster 2024-03-11T21:19:56.621551+0000 osd.5 (osd.5) 689 : cluster 0 3.1d scrub starts 2024-03-11T21:19:58.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:57 smithi146 bash[23809]: cluster 2024-03-11T21:19:56.625910+0000 osd.5 (osd.5) 690 : cluster 0 3.1d scrub ok 2024-03-11T21:19:58.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:57 smithi023 bash[21372]: audit 2024-03-11T21:19:56.347005+0000 mgr.y (mgr.24380) 1073 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:19:58.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:57 smithi023 bash[21372]: cluster 2024-03-11T21:19:56.410084+0000 osd.4 (osd.4) 167 : cluster 0 4.1a deep-scrub starts 2024-03-11T21:19:58.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:57 smithi023 bash[21372]: cluster 2024-03-11T21:19:56.412623+0000 osd.4 (osd.4) 168 : cluster 0 4.1a deep-scrub ok 2024-03-11T21:19:58.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:57 smithi023 bash[21372]: cluster 2024-03-11T21:19:56.490956+0000 osd.3 (osd.3) 644 : cluster 0 3.18 scrub starts 2024-03-11T21:19:58.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:57 smithi023 bash[21372]: cluster 2024-03-11T21:19:56.494963+0000 osd.3 (osd.3) 645 : cluster 0 3.18 scrub ok 2024-03-11T21:19:58.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:57 smithi023 bash[21372]: cluster 2024-03-11T21:19:56.587160+0000 mgr.y (mgr.24380) 1074 : cluster 0 pgmap v1247: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:19:58.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:57 smithi023 bash[21372]: cluster 2024-03-11T21:19:56.621551+0000 osd.5 (osd.5) 689 : cluster 0 3.1d scrub starts 2024-03-11T21:19:58.275 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:57 smithi023 bash[21372]: cluster 2024-03-11T21:19:56.625910+0000 osd.5 (osd.5) 690 : cluster 0 3.1d scrub ok 2024-03-11T21:19:58.275 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:57 smithi023 bash[29146]: audit 2024-03-11T21:19:56.347005+0000 mgr.y (mgr.24380) 1073 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:19:58.275 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:57 smithi023 bash[29146]: cluster 2024-03-11T21:19:56.410084+0000 osd.4 (osd.4) 167 : cluster 0 4.1a deep-scrub starts 2024-03-11T21:19:58.275 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:57 smithi023 bash[29146]: cluster 2024-03-11T21:19:56.412623+0000 osd.4 (osd.4) 168 : cluster 0 4.1a deep-scrub ok 2024-03-11T21:19:58.275 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:57 smithi023 bash[29146]: cluster 2024-03-11T21:19:56.490956+0000 osd.3 (osd.3) 644 : cluster 0 3.18 scrub starts 2024-03-11T21:19:58.275 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:57 smithi023 bash[29146]: cluster 2024-03-11T21:19:56.494963+0000 osd.3 (osd.3) 645 : cluster 0 3.18 scrub ok 2024-03-11T21:19:58.275 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:57 smithi023 bash[29146]: cluster 2024-03-11T21:19:56.587160+0000 mgr.y (mgr.24380) 1074 : cluster 0 pgmap v1247: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:19:58.275 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:57 smithi023 bash[29146]: cluster 2024-03-11T21:19:56.621551+0000 osd.5 (osd.5) 689 : cluster 0 3.1d scrub starts 2024-03-11T21:19:58.275 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:57 smithi023 bash[29146]: cluster 2024-03-11T21:19:56.625910+0000 osd.5 (osd.5) 690 : cluster 0 3.1d scrub ok 2024-03-11T21:19:58.773 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:19:58 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:19:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:19:59.391 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:19:59.392 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:19:59.392 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:19:59.392 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:20:00.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:19:59 smithi146 bash[23809]: cluster 2024-03-11T21:19:58.588100+0000 mgr.y (mgr.24380) 1075 : cluster 0 pgmap v1248: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:20:00.273 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:19:59 smithi023 bash[29146]: cluster 2024-03-11T21:19:58.588100+0000 mgr.y (mgr.24380) 1075 : cluster 0 pgmap v1248: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:20:00.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:19:59 smithi023 bash[21372]: cluster 2024-03-11T21:19:58.588100+0000 mgr.y (mgr.24380) 1075 : cluster 0 pgmap v1248: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:20:00.414 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:20:00.415 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:20:00.416 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:20:01.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:00 smithi146 bash[23809]: cluster 2024-03-11T21:20:00.000123+0000 mon.a (mon.0) 2115 : cluster 1 overall HEALTH_OK 2024-03-11T21:20:01.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:00 smithi146 bash[23809]: cluster 2024-03-11T21:20:00.573606+0000 osd.5 (osd.5) 691 : cluster 0 1.a scrub starts 2024-03-11T21:20:01.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:00 smithi146 bash[23809]: cluster 2024-03-11T21:20:00.576202+0000 osd.5 (osd.5) 692 : cluster 0 1.a scrub ok 2024-03-11T21:20:01.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:00 smithi146 bash[23809]: cluster 2024-03-11T21:20:00.589094+0000 mgr.y (mgr.24380) 1076 : cluster 0 pgmap v1249: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:20:01.273 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:00 smithi023 bash[21372]: cluster 2024-03-11T21:20:00.000123+0000 mon.a (mon.0) 2115 : cluster 1 overall HEALTH_OK 2024-03-11T21:20:01.273 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:00 smithi023 bash[21372]: cluster 2024-03-11T21:20:00.573606+0000 osd.5 (osd.5) 691 : cluster 0 1.a scrub starts 2024-03-11T21:20:01.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:00 smithi023 bash[21372]: cluster 2024-03-11T21:20:00.576202+0000 osd.5 (osd.5) 692 : cluster 0 1.a scrub ok 2024-03-11T21:20:01.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:00 smithi023 bash[21372]: cluster 2024-03-11T21:20:00.589094+0000 mgr.y (mgr.24380) 1076 : cluster 0 pgmap v1249: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:20:01.274 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:00 smithi023 bash[29146]: cluster 2024-03-11T21:20:00.000123+0000 mon.a (mon.0) 2115 : cluster 1 overall HEALTH_OK 2024-03-11T21:20:01.274 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:00 smithi023 bash[29146]: cluster 2024-03-11T21:20:00.573606+0000 osd.5 (osd.5) 691 : cluster 0 1.a scrub starts 2024-03-11T21:20:01.274 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:00 smithi023 bash[29146]: cluster 2024-03-11T21:20:00.576202+0000 osd.5 (osd.5) 692 : cluster 0 1.a scrub ok 2024-03-11T21:20:01.274 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:00 smithi023 bash[29146]: cluster 2024-03-11T21:20:00.589094+0000 mgr.y (mgr.24380) 1076 : cluster 0 pgmap v1249: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:20:01.433 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:20:02.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:01 smithi146 bash[23809]: cluster 2024-03-11T21:20:00.530746+0000 osd.3 (osd.3) 646 : cluster 0 purged_snaps scrub starts 2024-03-11T21:20:02.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:01 smithi146 bash[23809]: cluster 2024-03-11T21:20:00.530829+0000 osd.3 (osd.3) 647 : cluster 0 purged_snaps scrub ok 2024-03-11T21:20:02.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:01 smithi146 bash[23809]: cluster 2024-03-11T21:20:00.688208+0000 osd.0 (osd.0) 530 : cluster 0 3.1b deep-scrub starts 2024-03-11T21:20:02.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:01 smithi146 bash[23809]: cluster 2024-03-11T21:20:00.693167+0000 osd.0 (osd.0) 531 : cluster 0 3.1b deep-scrub ok 2024-03-11T21:20:02.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:01 smithi146 bash[23809]: cluster 2024-03-11T21:20:01.536788+0000 mon.a (mon.0) 2116 : cluster 0 osdmap e495: 8 total, 8 up, 8 in 2024-03-11T21:20:02.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:01 smithi023 bash[21372]: cluster 2024-03-11T21:20:00.530746+0000 osd.3 (osd.3) 646 : cluster 0 purged_snaps scrub starts 2024-03-11T21:20:02.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:01 smithi023 bash[21372]: cluster 2024-03-11T21:20:00.530829+0000 osd.3 (osd.3) 647 : cluster 0 purged_snaps scrub ok 2024-03-11T21:20:02.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:01 smithi023 bash[21372]: cluster 2024-03-11T21:20:00.688208+0000 osd.0 (osd.0) 530 : cluster 0 3.1b deep-scrub starts 2024-03-11T21:20:02.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:01 smithi023 bash[21372]: cluster 2024-03-11T21:20:00.693167+0000 osd.0 (osd.0) 531 : cluster 0 3.1b deep-scrub ok 2024-03-11T21:20:02.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:01 smithi023 bash[21372]: cluster 2024-03-11T21:20:01.536788+0000 mon.a (mon.0) 2116 : cluster 0 osdmap e495: 8 total, 8 up, 8 in 2024-03-11T21:20:02.274 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:01 smithi023 bash[29146]: cluster 2024-03-11T21:20:00.530746+0000 osd.3 (osd.3) 646 : cluster 0 purged_snaps scrub starts 2024-03-11T21:20:02.274 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:01 smithi023 bash[29146]: cluster 2024-03-11T21:20:00.530829+0000 osd.3 (osd.3) 647 : cluster 0 purged_snaps scrub ok 2024-03-11T21:20:02.274 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:01 smithi023 bash[29146]: cluster 2024-03-11T21:20:00.688208+0000 osd.0 (osd.0) 530 : cluster 0 3.1b deep-scrub starts 2024-03-11T21:20:02.274 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:01 smithi023 bash[29146]: cluster 2024-03-11T21:20:00.693167+0000 osd.0 (osd.0) 531 : cluster 0 3.1b deep-scrub ok 2024-03-11T21:20:02.274 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:01 smithi023 bash[29146]: cluster 2024-03-11T21:20:01.536788+0000 mon.a (mon.0) 2116 : cluster 0 osdmap e495: 8 total, 8 up, 8 in 2024-03-11T21:20:03.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:02 smithi146 bash[23809]: cluster 2024-03-11T21:20:02.589644+0000 mgr.y (mgr.24380) 1077 : cluster 0 pgmap v1251: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:20:03.273 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:02 smithi023 bash[21372]: cluster 2024-03-11T21:20:02.589644+0000 mgr.y (mgr.24380) 1077 : cluster 0 pgmap v1251: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:20:03.274 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:02 smithi023 bash[29146]: cluster 2024-03-11T21:20:02.589644+0000 mgr.y (mgr.24380) 1077 : cluster 0 pgmap v1251: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:20:03.989 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:03 smithi023 bash[21372]: cluster 2024-03-11T21:20:02.530028+0000 osd.3 (osd.3) 648 : cluster 0 6.1b deep-scrub starts 2024-03-11T21:20:03.989 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:03 smithi023 bash[21372]: cluster 2024-03-11T21:20:02.741783+0000 osd.3 (osd.3) 649 : cluster 0 6.1b deep-scrub ok 2024-03-11T21:20:03.989 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:03 smithi023 bash[29146]: cluster 2024-03-11T21:20:02.530028+0000 osd.3 (osd.3) 648 : cluster 0 6.1b deep-scrub starts 2024-03-11T21:20:03.989 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:03 smithi023 bash[29146]: cluster 2024-03-11T21:20:02.741783+0000 osd.3 (osd.3) 649 : cluster 0 6.1b deep-scrub ok 2024-03-11T21:20:04.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:03 smithi146 bash[23809]: cluster 2024-03-11T21:20:02.530028+0000 osd.3 (osd.3) 648 : cluster 0 6.1b deep-scrub starts 2024-03-11T21:20:04.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:03 smithi146 bash[23809]: cluster 2024-03-11T21:20:02.741783+0000 osd.3 (osd.3) 649 : cluster 0 6.1b deep-scrub ok 2024-03-11T21:20:04.904 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:04 smithi023 bash[29146]: cluster 2024-03-11T21:20:03.867333+0000 osd.2 (osd.2) 317 : cluster 0 2.e scrub starts 2024-03-11T21:20:04.904 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:04 smithi023 bash[29146]: cluster 2024-03-11T21:20:03.869249+0000 osd.2 (osd.2) 318 : cluster 0 2.e scrub ok 2024-03-11T21:20:04.904 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:04 smithi023 bash[29146]: cluster 2024-03-11T21:20:04.590377+0000 mgr.y (mgr.24380) 1078 : cluster 0 pgmap v1252: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 716 B/s rd, 0 op/s 2024-03-11T21:20:04.904 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:04 smithi023 bash[29146]: cluster 2024-03-11T21:20:04.598386+0000 osd.5 (osd.5) 693 : cluster 0 3.1c scrub starts 2024-03-11T21:20:04.905 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:04 smithi023 bash[29146]: cluster 2024-03-11T21:20:04.602965+0000 osd.5 (osd.5) 694 : cluster 0 3.1c scrub ok 2024-03-11T21:20:04.905 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:04 smithi023 bash[21372]: cluster 2024-03-11T21:20:03.867333+0000 osd.2 (osd.2) 317 : cluster 0 2.e scrub starts 2024-03-11T21:20:04.905 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:04 smithi023 bash[21372]: cluster 2024-03-11T21:20:03.869249+0000 osd.2 (osd.2) 318 : cluster 0 2.e scrub ok 2024-03-11T21:20:04.905 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:04 smithi023 bash[21372]: cluster 2024-03-11T21:20:04.590377+0000 mgr.y (mgr.24380) 1078 : cluster 0 pgmap v1252: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 716 B/s rd, 0 op/s 2024-03-11T21:20:04.905 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:04 smithi023 bash[21372]: cluster 2024-03-11T21:20:04.598386+0000 osd.5 (osd.5) 693 : cluster 0 3.1c scrub starts 2024-03-11T21:20:04.905 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:04 smithi023 bash[21372]: cluster 2024-03-11T21:20:04.602965+0000 osd.5 (osd.5) 694 : cluster 0 3.1c scrub ok 2024-03-11T21:20:05.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:04 smithi146 bash[23809]: cluster 2024-03-11T21:20:03.867333+0000 osd.2 (osd.2) 317 : cluster 0 2.e scrub starts 2024-03-11T21:20:05.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:04 smithi146 bash[23809]: cluster 2024-03-11T21:20:03.869249+0000 osd.2 (osd.2) 318 : cluster 0 2.e scrub ok 2024-03-11T21:20:05.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:04 smithi146 bash[23809]: cluster 2024-03-11T21:20:04.590377+0000 mgr.y (mgr.24380) 1078 : cluster 0 pgmap v1252: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 716 B/s rd, 0 op/s 2024-03-11T21:20:05.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:04 smithi146 bash[23809]: cluster 2024-03-11T21:20:04.598386+0000 osd.5 (osd.5) 693 : cluster 0 3.1c scrub starts 2024-03-11T21:20:05.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:04 smithi146 bash[23809]: cluster 2024-03-11T21:20:04.602965+0000 osd.5 (osd.5) 694 : cluster 0 3.1c scrub ok 2024-03-11T21:20:06.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:05 smithi023 bash[21372]: cluster 2024-03-11T21:20:04.512644+0000 osd.3 (osd.3) 650 : cluster 0 4.0 scrub starts 2024-03-11T21:20:06.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:05 smithi023 bash[21372]: cluster 2024-03-11T21:20:04.515600+0000 osd.3 (osd.3) 651 : cluster 0 4.0 scrub ok 2024-03-11T21:20:06.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:05 smithi023 bash[29146]: cluster 2024-03-11T21:20:04.512644+0000 osd.3 (osd.3) 650 : cluster 0 4.0 scrub starts 2024-03-11T21:20:06.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:05 smithi023 bash[29146]: cluster 2024-03-11T21:20:04.515600+0000 osd.3 (osd.3) 651 : cluster 0 4.0 scrub ok 2024-03-11T21:20:06.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:05 smithi146 bash[23809]: cluster 2024-03-11T21:20:04.512644+0000 osd.3 (osd.3) 650 : cluster 0 4.0 scrub starts 2024-03-11T21:20:06.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:05 smithi146 bash[23809]: cluster 2024-03-11T21:20:04.515600+0000 osd.3 (osd.3) 651 : cluster 0 4.0 scrub ok 2024-03-11T21:20:06.752 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:20:06 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:20:06.997 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:06 smithi023 bash[21372]: cluster 2024-03-11T21:20:05.530371+0000 osd.3 (osd.3) 652 : cluster 0 5.0 deep-scrub starts 2024-03-11T21:20:06.997 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:06 smithi023 bash[21372]: cluster 2024-03-11T21:20:05.533472+0000 osd.3 (osd.3) 653 : cluster 0 5.0 deep-scrub ok 2024-03-11T21:20:06.997 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:06 smithi023 bash[21372]: cluster 2024-03-11T21:20:06.042551+0000 osd.1 (osd.1) 456 : cluster 0 2.6 scrub starts 2024-03-11T21:20:06.997 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:06 smithi023 bash[21372]: cluster 2024-03-11T21:20:06.046071+0000 osd.1 (osd.1) 457 : cluster 0 2.6 scrub ok 2024-03-11T21:20:06.997 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:06 smithi023 bash[21372]: audit 2024-03-11T21:20:06.358405+0000 mgr.y (mgr.24380) 1079 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:20:06.997 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:06 smithi023 bash[21372]: cluster 2024-03-11T21:20:06.591905+0000 mgr.y (mgr.24380) 1080 : cluster 0 pgmap v1253: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:20:06.998 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:06 smithi023 bash[29146]: cluster 2024-03-11T21:20:05.530371+0000 osd.3 (osd.3) 652 : cluster 0 5.0 deep-scrub starts 2024-03-11T21:20:06.998 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:06 smithi023 bash[29146]: cluster 2024-03-11T21:20:05.533472+0000 osd.3 (osd.3) 653 : cluster 0 5.0 deep-scrub ok 2024-03-11T21:20:06.998 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:06 smithi023 bash[29146]: cluster 2024-03-11T21:20:06.042551+0000 osd.1 (osd.1) 456 : cluster 0 2.6 scrub starts 2024-03-11T21:20:06.998 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:06 smithi023 bash[29146]: cluster 2024-03-11T21:20:06.046071+0000 osd.1 (osd.1) 457 : cluster 0 2.6 scrub ok 2024-03-11T21:20:06.998 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:06 smithi023 bash[29146]: audit 2024-03-11T21:20:06.358405+0000 mgr.y (mgr.24380) 1079 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:20:06.998 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:06 smithi023 bash[29146]: cluster 2024-03-11T21:20:06.591905+0000 mgr.y (mgr.24380) 1080 : cluster 0 pgmap v1253: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:20:07.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:06 smithi146 bash[23809]: cluster 2024-03-11T21:20:05.530371+0000 osd.3 (osd.3) 652 : cluster 0 5.0 deep-scrub starts 2024-03-11T21:20:07.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:06 smithi146 bash[23809]: cluster 2024-03-11T21:20:05.533472+0000 osd.3 (osd.3) 653 : cluster 0 5.0 deep-scrub ok 2024-03-11T21:20:07.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:06 smithi146 bash[23809]: cluster 2024-03-11T21:20:06.042551+0000 osd.1 (osd.1) 456 : cluster 0 2.6 scrub starts 2024-03-11T21:20:07.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:06 smithi146 bash[23809]: cluster 2024-03-11T21:20:06.046071+0000 osd.1 (osd.1) 457 : cluster 0 2.6 scrub ok 2024-03-11T21:20:07.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:06 smithi146 bash[23809]: audit 2024-03-11T21:20:06.358405+0000 mgr.y (mgr.24380) 1079 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:20:07.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:06 smithi146 bash[23809]: cluster 2024-03-11T21:20:06.591905+0000 mgr.y (mgr.24380) 1080 : cluster 0 pgmap v1253: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:20:08.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:07 smithi146 bash[23809]: audit 2024-03-11T21:20:07.154622+0000 mon.a (mon.0) 2117 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:20:08.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:07 smithi146 bash[23809]: cluster 2024-03-11T21:20:07.604996+0000 osd.5 (osd.5) 695 : cluster 0 6.d scrub starts 2024-03-11T21:20:08.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:07 smithi023 bash[21372]: audit 2024-03-11T21:20:07.154622+0000 mon.a (mon.0) 2117 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:20:08.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:07 smithi023 bash[21372]: cluster 2024-03-11T21:20:07.604996+0000 osd.5 (osd.5) 695 : cluster 0 6.d scrub starts 2024-03-11T21:20:08.282 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:07 smithi023 bash[29146]: audit 2024-03-11T21:20:07.154622+0000 mon.a (mon.0) 2117 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:20:08.282 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:07 smithi023 bash[29146]: cluster 2024-03-11T21:20:07.604996+0000 osd.5 (osd.5) 695 : cluster 0 6.d scrub starts 2024-03-11T21:20:08.773 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:20:08 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:20:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:20:09.231 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:08 smithi023 bash[21372]: cluster 2024-03-11T21:20:07.526929+0000 osd.4 (osd.4) 169 : cluster 0 4.1b scrub starts 2024-03-11T21:20:09.231 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:08 smithi023 bash[21372]: cluster 2024-03-11T21:20:07.529304+0000 osd.4 (osd.4) 170 : cluster 0 4.1b scrub ok 2024-03-11T21:20:09.231 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:08 smithi023 bash[21372]: cluster 2024-03-11T21:20:07.761652+0000 osd.5 (osd.5) 696 : cluster 0 6.d scrub ok 2024-03-11T21:20:09.231 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:08 smithi023 bash[21372]: cluster 2024-03-11T21:20:08.332490+0000 osd.7 (osd.7) 719 : cluster 0 1.0 scrub starts 2024-03-11T21:20:09.231 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:08 smithi023 bash[21372]: cluster 2024-03-11T21:20:08.337665+0000 osd.7 (osd.7) 720 : cluster 0 1.0 scrub ok 2024-03-11T21:20:09.232 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:08 smithi023 bash[21372]: cluster 2024-03-11T21:20:08.592738+0000 mgr.y (mgr.24380) 1081 : cluster 0 pgmap v1254: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:20:09.232 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:08 smithi023 bash[29146]: cluster 2024-03-11T21:20:07.526929+0000 osd.4 (osd.4) 169 : cluster 0 4.1b scrub starts 2024-03-11T21:20:09.232 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:08 smithi023 bash[29146]: cluster 2024-03-11T21:20:07.529304+0000 osd.4 (osd.4) 170 : cluster 0 4.1b scrub ok 2024-03-11T21:20:09.232 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:08 smithi023 bash[29146]: cluster 2024-03-11T21:20:07.761652+0000 osd.5 (osd.5) 696 : cluster 0 6.d scrub ok 2024-03-11T21:20:09.232 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:08 smithi023 bash[29146]: cluster 2024-03-11T21:20:08.332490+0000 osd.7 (osd.7) 719 : cluster 0 1.0 scrub starts 2024-03-11T21:20:09.232 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:08 smithi023 bash[29146]: cluster 2024-03-11T21:20:08.337665+0000 osd.7 (osd.7) 720 : cluster 0 1.0 scrub ok 2024-03-11T21:20:09.232 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:08 smithi023 bash[29146]: cluster 2024-03-11T21:20:08.592738+0000 mgr.y (mgr.24380) 1081 : cluster 0 pgmap v1254: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:20:09.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:08 smithi146 bash[23809]: cluster 2024-03-11T21:20:07.526929+0000 osd.4 (osd.4) 169 : cluster 0 4.1b scrub starts 2024-03-11T21:20:09.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:08 smithi146 bash[23809]: cluster 2024-03-11T21:20:07.529304+0000 osd.4 (osd.4) 170 : cluster 0 4.1b scrub ok 2024-03-11T21:20:09.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:08 smithi146 bash[23809]: cluster 2024-03-11T21:20:07.761652+0000 osd.5 (osd.5) 696 : cluster 0 6.d scrub ok 2024-03-11T21:20:09.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:08 smithi146 bash[23809]: cluster 2024-03-11T21:20:08.332490+0000 osd.7 (osd.7) 719 : cluster 0 1.0 scrub starts 2024-03-11T21:20:09.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:08 smithi146 bash[23809]: cluster 2024-03-11T21:20:08.337665+0000 osd.7 (osd.7) 720 : cluster 0 1.0 scrub ok 2024-03-11T21:20:09.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:08 smithi146 bash[23809]: cluster 2024-03-11T21:20:08.592738+0000 mgr.y (mgr.24380) 1081 : cluster 0 pgmap v1254: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:20:10.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:09 smithi146 bash[23809]: cluster 2024-03-11T21:20:09.367021+0000 osd.7 (osd.7) 721 : cluster 0 3.f scrub starts 2024-03-11T21:20:10.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:09 smithi146 bash[23809]: cluster 2024-03-11T21:20:09.371748+0000 osd.7 (osd.7) 722 : cluster 0 3.f scrub ok 2024-03-11T21:20:10.273 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:09 smithi023 bash[21372]: cluster 2024-03-11T21:20:09.367021+0000 osd.7 (osd.7) 721 : cluster 0 3.f scrub starts 2024-03-11T21:20:10.273 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:09 smithi023 bash[21372]: cluster 2024-03-11T21:20:09.371748+0000 osd.7 (osd.7) 722 : cluster 0 3.f scrub ok 2024-03-11T21:20:10.274 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:09 smithi023 bash[29146]: cluster 2024-03-11T21:20:09.367021+0000 osd.7 (osd.7) 721 : cluster 0 3.f scrub starts 2024-03-11T21:20:10.274 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:09 smithi023 bash[29146]: cluster 2024-03-11T21:20:09.371748+0000 osd.7 (osd.7) 722 : cluster 0 3.f scrub ok 2024-03-11T21:20:11.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:10 smithi146 bash[23809]: cluster 2024-03-11T21:20:09.838292+0000 osd.2 (osd.2) 319 : cluster 0 6.f scrub starts 2024-03-11T21:20:11.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:10 smithi146 bash[23809]: cluster 2024-03-11T21:20:10.593806+0000 mgr.y (mgr.24380) 1082 : cluster 0 pgmap v1255: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:20:11.273 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:10 smithi023 bash[21372]: cluster 2024-03-11T21:20:09.838292+0000 osd.2 (osd.2) 319 : cluster 0 6.f scrub starts 2024-03-11T21:20:11.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:10 smithi023 bash[21372]: cluster 2024-03-11T21:20:10.593806+0000 mgr.y (mgr.24380) 1082 : cluster 0 pgmap v1255: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:20:11.274 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:10 smithi023 bash[29146]: cluster 2024-03-11T21:20:09.838292+0000 osd.2 (osd.2) 319 : cluster 0 6.f scrub starts 2024-03-11T21:20:11.274 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:10 smithi023 bash[29146]: cluster 2024-03-11T21:20:10.593806+0000 mgr.y (mgr.24380) 1082 : cluster 0 pgmap v1255: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:20:12.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:11 smithi146 bash[23809]: cluster 2024-03-11T21:20:10.061488+0000 osd.2 (osd.2) 320 : cluster 0 6.f scrub ok 2024-03-11T21:20:12.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:11 smithi146 bash[23809]: cluster 2024-03-11T21:20:10.490223+0000 osd.4 (osd.4) 171 : cluster 0 5.1 scrub starts 2024-03-11T21:20:12.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:11 smithi146 bash[23809]: cluster 2024-03-11T21:20:10.492727+0000 osd.4 (osd.4) 172 : cluster 0 5.1 scrub ok 2024-03-11T21:20:12.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:11 smithi146 bash[23809]: cluster 2024-03-11T21:20:10.677766+0000 osd.5 (osd.5) 697 : cluster 0 4.15 scrub starts 2024-03-11T21:20:12.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:11 smithi146 bash[23809]: cluster 2024-03-11T21:20:10.680714+0000 osd.5 (osd.5) 698 : cluster 0 4.15 scrub ok 2024-03-11T21:20:12.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:11 smithi023 bash[21372]: cluster 2024-03-11T21:20:10.061488+0000 osd.2 (osd.2) 320 : cluster 0 6.f scrub ok 2024-03-11T21:20:12.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:11 smithi023 bash[21372]: cluster 2024-03-11T21:20:10.490223+0000 osd.4 (osd.4) 171 : cluster 0 5.1 scrub starts 2024-03-11T21:20:12.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:11 smithi023 bash[21372]: cluster 2024-03-11T21:20:10.492727+0000 osd.4 (osd.4) 172 : cluster 0 5.1 scrub ok 2024-03-11T21:20:12.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:11 smithi023 bash[21372]: cluster 2024-03-11T21:20:10.677766+0000 osd.5 (osd.5) 697 : cluster 0 4.15 scrub starts 2024-03-11T21:20:12.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:11 smithi023 bash[21372]: cluster 2024-03-11T21:20:10.680714+0000 osd.5 (osd.5) 698 : cluster 0 4.15 scrub ok 2024-03-11T21:20:12.274 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:11 smithi023 bash[29146]: cluster 2024-03-11T21:20:10.061488+0000 osd.2 (osd.2) 320 : cluster 0 6.f scrub ok 2024-03-11T21:20:12.274 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:11 smithi023 bash[29146]: cluster 2024-03-11T21:20:10.490223+0000 osd.4 (osd.4) 171 : cluster 0 5.1 scrub starts 2024-03-11T21:20:12.274 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:11 smithi023 bash[29146]: cluster 2024-03-11T21:20:10.492727+0000 osd.4 (osd.4) 172 : cluster 0 5.1 scrub ok 2024-03-11T21:20:12.274 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:11 smithi023 bash[29146]: cluster 2024-03-11T21:20:10.677766+0000 osd.5 (osd.5) 697 : cluster 0 4.15 scrub starts 2024-03-11T21:20:12.274 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:11 smithi023 bash[29146]: cluster 2024-03-11T21:20:10.680714+0000 osd.5 (osd.5) 698 : cluster 0 4.15 scrub ok 2024-03-11T21:20:13.231 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:12 smithi023 bash[21372]: cluster 2024-03-11T21:20:11.505961+0000 osd.4 (osd.4) 173 : cluster 0 6.e scrub starts 2024-03-11T21:20:13.231 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:12 smithi023 bash[21372]: cluster 2024-03-11T21:20:11.575627+0000 osd.4 (osd.4) 174 : cluster 0 6.e scrub ok 2024-03-11T21:20:13.231 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:12 smithi023 bash[21372]: cluster 2024-03-11T21:20:12.595164+0000 mgr.y (mgr.24380) 1083 : cluster 0 pgmap v1256: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 925 B/s rd, 0 op/s 2024-03-11T21:20:13.232 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:12 smithi023 bash[29146]: cluster 2024-03-11T21:20:11.505961+0000 osd.4 (osd.4) 173 : cluster 0 6.e scrub starts 2024-03-11T21:20:13.232 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:12 smithi023 bash[29146]: cluster 2024-03-11T21:20:11.575627+0000 osd.4 (osd.4) 174 : cluster 0 6.e scrub ok 2024-03-11T21:20:13.232 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:12 smithi023 bash[29146]: cluster 2024-03-11T21:20:12.595164+0000 mgr.y (mgr.24380) 1083 : cluster 0 pgmap v1256: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 925 B/s rd, 0 op/s 2024-03-11T21:20:13.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:12 smithi146 bash[23809]: cluster 2024-03-11T21:20:11.505961+0000 osd.4 (osd.4) 173 : cluster 0 6.e scrub starts 2024-03-11T21:20:13.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:12 smithi146 bash[23809]: cluster 2024-03-11T21:20:11.575627+0000 osd.4 (osd.4) 174 : cluster 0 6.e scrub ok 2024-03-11T21:20:13.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:12 smithi146 bash[23809]: cluster 2024-03-11T21:20:12.595164+0000 mgr.y (mgr.24380) 1083 : cluster 0 pgmap v1256: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 925 B/s rd, 0 op/s 2024-03-11T21:20:13.654 INFO:teuthology.orchestra.run.smithi023.stdout:188978561412 2024-03-11T21:20:13.856 INFO:teuthology.orchestra.run.smithi023.stdout:876173328623 2024-03-11T21:20:14.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:13 smithi146 bash[23809]: audit 2024-03-11T21:20:13.669360+0000 mon.a (mon.0) 2118 : audit 0 from='client.? 172.21.15.23:0/1768096598' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-11T21:20:14.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:13 smithi146 bash[23809]: audit 2024-03-11T21:20:13.870901+0000 mon.c (mon.2) 114 : audit 0 from='client.? 172.21.15.23:0/1805766993' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-11T21:20:14.273 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:13 smithi023 bash[21372]: audit 2024-03-11T21:20:13.669360+0000 mon.a (mon.0) 2118 : audit 0 from='client.? 172.21.15.23:0/1768096598' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-11T21:20:14.273 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:13 smithi023 bash[21372]: audit 2024-03-11T21:20:13.870901+0000 mon.c (mon.2) 114 : audit 0 from='client.? 172.21.15.23:0/1805766993' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-11T21:20:14.274 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:13 smithi023 bash[29146]: audit 2024-03-11T21:20:13.669360+0000 mon.a (mon.0) 2118 : audit 0 from='client.? 172.21.15.23:0/1768096598' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-11T21:20:14.274 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:13 smithi023 bash[29146]: audit 2024-03-11T21:20:13.870901+0000 mon.c (mon.2) 114 : audit 0 from='client.? 172.21.15.23:0/1805766993' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-11T21:20:14.435 INFO:teuthology.orchestra.run.smithi023.stdout:34359738808 2024-03-11T21:20:14.811 INFO:teuthology.orchestra.run.smithi023.stdout:81604379043 2024-03-11T21:20:15.023 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:14 smithi023 bash[21372]: cluster 2024-03-11T21:20:13.550829+0000 osd.3 (osd.3) 654 : cluster 0 4.1d deep-scrub starts 2024-03-11T21:20:15.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:14 smithi023 bash[21372]: cluster 2024-03-11T21:20:13.552110+0000 osd.3 (osd.3) 655 : cluster 0 4.1d deep-scrub ok 2024-03-11T21:20:15.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:14 smithi023 bash[21372]: cluster 2024-03-11T21:20:13.682638+0000 osd.5 (osd.5) 699 : cluster 0 1.1 deep-scrub starts 2024-03-11T21:20:15.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:14 smithi023 bash[21372]: cluster 2024-03-11T21:20:13.684872+0000 osd.5 (osd.5) 700 : cluster 0 1.1 deep-scrub ok 2024-03-11T21:20:15.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:14 smithi023 bash[21372]: cluster 2024-03-11T21:20:13.921318+0000 osd.1 (osd.1) 458 : cluster 0 4.4 scrub starts 2024-03-11T21:20:15.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:14 smithi023 bash[21372]: cluster 2024-03-11T21:20:13.922304+0000 osd.1 (osd.1) 459 : cluster 0 4.4 scrub ok 2024-03-11T21:20:15.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:14 smithi023 bash[21372]: audit 2024-03-11T21:20:14.449891+0000 mon.a (mon.0) 2119 : audit 0 from='client.? 172.21.15.23:0/2665844872' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-11T21:20:15.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:14 smithi023 bash[21372]: cluster 2024-03-11T21:20:14.595732+0000 mgr.y (mgr.24380) 1084 : cluster 0 pgmap v1257: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:20:15.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:14 smithi023 bash[21372]: audit 2024-03-11T21:20:14.826023+0000 mon.c (mon.2) 115 : audit 0 from='client.? 172.21.15.23:0/3901626047' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-11T21:20:15.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:14 smithi023 bash[29146]: cluster 2024-03-11T21:20:13.550829+0000 osd.3 (osd.3) 654 : cluster 0 4.1d deep-scrub starts 2024-03-11T21:20:15.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:14 smithi023 bash[29146]: cluster 2024-03-11T21:20:13.552110+0000 osd.3 (osd.3) 655 : cluster 0 4.1d deep-scrub ok 2024-03-11T21:20:15.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:14 smithi023 bash[29146]: cluster 2024-03-11T21:20:13.682638+0000 osd.5 (osd.5) 699 : cluster 0 1.1 deep-scrub starts 2024-03-11T21:20:15.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:14 smithi023 bash[29146]: cluster 2024-03-11T21:20:13.684872+0000 osd.5 (osd.5) 700 : cluster 0 1.1 deep-scrub ok 2024-03-11T21:20:15.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:14 smithi023 bash[29146]: cluster 2024-03-11T21:20:13.921318+0000 osd.1 (osd.1) 458 : cluster 0 4.4 scrub starts 2024-03-11T21:20:15.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:14 smithi023 bash[29146]: cluster 2024-03-11T21:20:13.922304+0000 osd.1 (osd.1) 459 : cluster 0 4.4 scrub ok 2024-03-11T21:20:15.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:14 smithi023 bash[29146]: audit 2024-03-11T21:20:14.449891+0000 mon.a (mon.0) 2119 : audit 0 from='client.? 172.21.15.23:0/2665844872' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-11T21:20:15.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:14 smithi023 bash[29146]: cluster 2024-03-11T21:20:14.595732+0000 mgr.y (mgr.24380) 1084 : cluster 0 pgmap v1257: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:20:15.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:14 smithi023 bash[29146]: audit 2024-03-11T21:20:14.826023+0000 mon.c (mon.2) 115 : audit 0 from='client.? 172.21.15.23:0/3901626047' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-11T21:20:15.100 INFO:teuthology.orchestra.run.smithi023.stdout:292057776496 2024-03-11T21:20:15.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:14 smithi146 bash[23809]: cluster 2024-03-11T21:20:13.550829+0000 osd.3 (osd.3) 654 : cluster 0 4.1d deep-scrub starts 2024-03-11T21:20:15.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:14 smithi146 bash[23809]: cluster 2024-03-11T21:20:13.552110+0000 osd.3 (osd.3) 655 : cluster 0 4.1d deep-scrub ok 2024-03-11T21:20:15.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:14 smithi146 bash[23809]: cluster 2024-03-11T21:20:13.682638+0000 osd.5 (osd.5) 699 : cluster 0 1.1 deep-scrub starts 2024-03-11T21:20:15.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:14 smithi146 bash[23809]: cluster 2024-03-11T21:20:13.684872+0000 osd.5 (osd.5) 700 : cluster 0 1.1 deep-scrub ok 2024-03-11T21:20:15.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:14 smithi146 bash[23809]: cluster 2024-03-11T21:20:13.921318+0000 osd.1 (osd.1) 458 : cluster 0 4.4 scrub starts 2024-03-11T21:20:15.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:14 smithi146 bash[23809]: cluster 2024-03-11T21:20:13.922304+0000 osd.1 (osd.1) 459 : cluster 0 4.4 scrub ok 2024-03-11T21:20:15.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:14 smithi146 bash[23809]: audit 2024-03-11T21:20:14.449891+0000 mon.a (mon.0) 2119 : audit 0 from='client.? 172.21.15.23:0/2665844872' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-11T21:20:15.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:14 smithi146 bash[23809]: cluster 2024-03-11T21:20:14.595732+0000 mgr.y (mgr.24380) 1084 : cluster 0 pgmap v1257: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:20:15.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:14 smithi146 bash[23809]: audit 2024-03-11T21:20:14.826023+0000 mon.c (mon.2) 115 : audit 0 from='client.? 172.21.15.23:0/3901626047' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-11T21:20:16.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:15 smithi146 bash[23809]: cluster 2024-03-11T21:20:14.595327+0000 osd.3 (osd.3) 656 : cluster 0 4.14 scrub starts 2024-03-11T21:20:16.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:15 smithi146 bash[23809]: cluster 2024-03-11T21:20:14.597354+0000 osd.3 (osd.3) 657 : cluster 0 4.14 scrub ok 2024-03-11T21:20:16.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:15 smithi146 bash[23809]: cluster 2024-03-11T21:20:14.967807+0000 osd.1 (osd.1) 460 : cluster 0 1.3 scrub starts 2024-03-11T21:20:16.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:15 smithi146 bash[23809]: cluster 2024-03-11T21:20:14.969439+0000 osd.1 (osd.1) 461 : cluster 0 1.3 scrub ok 2024-03-11T21:20:16.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:15 smithi146 bash[23809]: audit 2024-03-11T21:20:15.114950+0000 mon.a (mon.0) 2120 : audit 0 from='client.? 172.21.15.23:0/1882522886' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-11T21:20:16.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:15 smithi146 bash[23809]: cluster 2024-03-11T21:20:15.307911+0000 osd.7 (osd.7) 723 : cluster 0 1.d deep-scrub starts 2024-03-11T21:20:16.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:15 smithi146 bash[23809]: cluster 2024-03-11T21:20:15.310481+0000 osd.7 (osd.7) 724 : cluster 0 1.d deep-scrub ok 2024-03-11T21:20:16.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:15 smithi146 bash[23809]: cluster 2024-03-11T21:20:15.376469+0000 osd.6 (osd.6) 570 : cluster 0 purged_snaps scrub starts 2024-03-11T21:20:16.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:15 smithi146 bash[23809]: cluster 2024-03-11T21:20:15.376591+0000 osd.6 (osd.6) 571 : cluster 0 purged_snaps scrub ok 2024-03-11T21:20:16.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:15 smithi023 bash[21372]: cluster 2024-03-11T21:20:14.595327+0000 osd.3 (osd.3) 656 : cluster 0 4.14 scrub starts 2024-03-11T21:20:16.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:15 smithi023 bash[21372]: cluster 2024-03-11T21:20:14.597354+0000 osd.3 (osd.3) 657 : cluster 0 4.14 scrub ok 2024-03-11T21:20:16.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:15 smithi023 bash[21372]: cluster 2024-03-11T21:20:14.967807+0000 osd.1 (osd.1) 460 : cluster 0 1.3 scrub starts 2024-03-11T21:20:16.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:15 smithi023 bash[21372]: cluster 2024-03-11T21:20:14.969439+0000 osd.1 (osd.1) 461 : cluster 0 1.3 scrub ok 2024-03-11T21:20:16.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:15 smithi023 bash[21372]: audit 2024-03-11T21:20:15.114950+0000 mon.a (mon.0) 2120 : audit 0 from='client.? 172.21.15.23:0/1882522886' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-11T21:20:16.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:15 smithi023 bash[21372]: cluster 2024-03-11T21:20:15.307911+0000 osd.7 (osd.7) 723 : cluster 0 1.d deep-scrub starts 2024-03-11T21:20:16.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:15 smithi023 bash[21372]: cluster 2024-03-11T21:20:15.310481+0000 osd.7 (osd.7) 724 : cluster 0 1.d deep-scrub ok 2024-03-11T21:20:16.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:15 smithi023 bash[21372]: cluster 2024-03-11T21:20:15.376469+0000 osd.6 (osd.6) 570 : cluster 0 purged_snaps scrub starts 2024-03-11T21:20:16.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:15 smithi023 bash[21372]: cluster 2024-03-11T21:20:15.376591+0000 osd.6 (osd.6) 571 : cluster 0 purged_snaps scrub ok 2024-03-11T21:20:16.274 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:15 smithi023 bash[29146]: cluster 2024-03-11T21:20:14.595327+0000 osd.3 (osd.3) 656 : cluster 0 4.14 scrub starts 2024-03-11T21:20:16.275 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:15 smithi023 bash[29146]: cluster 2024-03-11T21:20:14.597354+0000 osd.3 (osd.3) 657 : cluster 0 4.14 scrub ok 2024-03-11T21:20:16.275 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:15 smithi023 bash[29146]: cluster 2024-03-11T21:20:14.967807+0000 osd.1 (osd.1) 460 : cluster 0 1.3 scrub starts 2024-03-11T21:20:16.275 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:15 smithi023 bash[29146]: cluster 2024-03-11T21:20:14.969439+0000 osd.1 (osd.1) 461 : cluster 0 1.3 scrub ok 2024-03-11T21:20:16.275 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:15 smithi023 bash[29146]: audit 2024-03-11T21:20:15.114950+0000 mon.a (mon.0) 2120 : audit 0 from='client.? 172.21.15.23:0/1882522886' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-11T21:20:16.275 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:15 smithi023 bash[29146]: cluster 2024-03-11T21:20:15.307911+0000 osd.7 (osd.7) 723 : cluster 0 1.d deep-scrub starts 2024-03-11T21:20:16.275 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:15 smithi023 bash[29146]: cluster 2024-03-11T21:20:15.310481+0000 osd.7 (osd.7) 724 : cluster 0 1.d deep-scrub ok 2024-03-11T21:20:16.275 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:15 smithi023 bash[29146]: cluster 2024-03-11T21:20:15.376469+0000 osd.6 (osd.6) 570 : cluster 0 purged_snaps scrub starts 2024-03-11T21:20:16.275 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:15 smithi023 bash[29146]: cluster 2024-03-11T21:20:15.376591+0000 osd.6 (osd.6) 571 : cluster 0 purged_snaps scrub ok 2024-03-11T21:20:16.524 INFO:tasks.cephadm.ceph_manager.ceph:need seq 876173328618 got 876173328623 for osd.4 2024-03-11T21:20:16.524 DEBUG:teuthology.parallel:result is None 2024-03-11T21:20:16.752 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:20:16 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:20:16.844 INFO:teuthology.orchestra.run.smithi023.stdout:244813136250 2024-03-11T21:20:16.877 INFO:teuthology.orchestra.run.smithi023.stdout:906238099691 2024-03-11T21:20:17.040 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:17 smithi023 bash[29146]: cluster 2024-03-11T21:20:15.683907+0000 osd.5 (osd.5) 701 : cluster 0 6.7 scrub starts 2024-03-11T21:20:17.040 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:17 smithi023 bash[29146]: cluster 2024-03-11T21:20:15.856445+0000 osd.5 (osd.5) 702 : cluster 0 6.7 scrub ok 2024-03-11T21:20:17.040 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:17 smithi023 bash[29146]: cluster 2024-03-11T21:20:15.910900+0000 osd.2 (osd.2) 321 : cluster 0 4.1c scrub starts 2024-03-11T21:20:17.040 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:17 smithi023 bash[29146]: cluster 2024-03-11T21:20:15.913682+0000 osd.2 (osd.2) 322 : cluster 0 4.1c scrub ok 2024-03-11T21:20:17.040 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:17 smithi023 bash[29146]: cluster 2024-03-11T21:20:15.996389+0000 mon.a (mon.0) 2121 : cluster 0 osdmap e496: 8 total, 8 up, 8 in 2024-03-11T21:20:17.040 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:17 smithi023 bash[29146]: audit 2024-03-11T21:20:16.363187+0000 mgr.y (mgr.24380) 1085 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:20:17.040 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:17 smithi023 bash[29146]: cluster 2024-03-11T21:20:16.556779+0000 osd.0 (osd.0) 532 : cluster 0 3.6 scrub starts 2024-03-11T21:20:17.040 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:17 smithi023 bash[29146]: cluster 2024-03-11T21:20:16.560983+0000 osd.0 (osd.0) 533 : cluster 0 3.6 scrub ok 2024-03-11T21:20:17.040 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:17 smithi023 bash[29146]: cluster 2024-03-11T21:20:16.596192+0000 mgr.y (mgr.24380) 1086 : cluster 0 pgmap v1259: 152 pgs: 1 active+clean+scrubbing, 151 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:20:17.041 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:17 smithi023 bash[29146]: audit 2024-03-11T21:20:16.858999+0000 mon.a (mon.0) 2122 : audit 0 from='client.? 172.21.15.23:0/826944141' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-11T21:20:17.041 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:17 smithi023 bash[29146]: audit 2024-03-11T21:20:16.892824+0000 mon.a (mon.0) 2123 : audit 0 from='client.? 172.21.15.23:0/2842618213' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-11T21:20:17.041 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:17 smithi023 bash[21372]: cluster 2024-03-11T21:20:15.683907+0000 osd.5 (osd.5) 701 : cluster 0 6.7 scrub starts 2024-03-11T21:20:17.041 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:17 smithi023 bash[21372]: cluster 2024-03-11T21:20:15.856445+0000 osd.5 (osd.5) 702 : cluster 0 6.7 scrub ok 2024-03-11T21:20:17.041 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:17 smithi023 bash[21372]: cluster 2024-03-11T21:20:15.910900+0000 osd.2 (osd.2) 321 : cluster 0 4.1c scrub starts 2024-03-11T21:20:17.041 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:17 smithi023 bash[21372]: cluster 2024-03-11T21:20:15.913682+0000 osd.2 (osd.2) 322 : cluster 0 4.1c scrub ok 2024-03-11T21:20:17.041 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:17 smithi023 bash[21372]: cluster 2024-03-11T21:20:15.996389+0000 mon.a (mon.0) 2121 : cluster 0 osdmap e496: 8 total, 8 up, 8 in 2024-03-11T21:20:17.041 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:17 smithi023 bash[21372]: audit 2024-03-11T21:20:16.363187+0000 mgr.y (mgr.24380) 1085 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:20:17.041 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:17 smithi023 bash[21372]: cluster 2024-03-11T21:20:16.556779+0000 osd.0 (osd.0) 532 : cluster 0 3.6 scrub starts 2024-03-11T21:20:17.042 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:17 smithi023 bash[21372]: cluster 2024-03-11T21:20:16.560983+0000 osd.0 (osd.0) 533 : cluster 0 3.6 scrub ok 2024-03-11T21:20:17.042 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:17 smithi023 bash[21372]: cluster 2024-03-11T21:20:16.596192+0000 mgr.y (mgr.24380) 1086 : cluster 0 pgmap v1259: 152 pgs: 1 active+clean+scrubbing, 151 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:20:17.042 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:17 smithi023 bash[21372]: audit 2024-03-11T21:20:16.858999+0000 mon.a (mon.0) 2122 : audit 0 from='client.? 172.21.15.23:0/826944141' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-11T21:20:17.042 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:17 smithi023 bash[21372]: audit 2024-03-11T21:20:16.892824+0000 mon.a (mon.0) 2123 : audit 0 from='client.? 172.21.15.23:0/2842618213' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-11T21:20:17.043 INFO:tasks.cephadm.ceph_manager.ceph:need seq 188978561408 got 188978561412 for osd.5 2024-03-11T21:20:17.043 DEBUG:teuthology.parallel:result is None 2024-03-11T21:20:17.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:17 smithi146 bash[23809]: cluster 2024-03-11T21:20:15.683907+0000 osd.5 (osd.5) 701 : cluster 0 6.7 scrub starts 2024-03-11T21:20:17.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:17 smithi146 bash[23809]: cluster 2024-03-11T21:20:15.856445+0000 osd.5 (osd.5) 702 : cluster 0 6.7 scrub ok 2024-03-11T21:20:17.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:17 smithi146 bash[23809]: cluster 2024-03-11T21:20:15.910900+0000 osd.2 (osd.2) 321 : cluster 0 4.1c scrub starts 2024-03-11T21:20:17.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:17 smithi146 bash[23809]: cluster 2024-03-11T21:20:15.913682+0000 osd.2 (osd.2) 322 : cluster 0 4.1c scrub ok 2024-03-11T21:20:17.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:17 smithi146 bash[23809]: cluster 2024-03-11T21:20:15.996389+0000 mon.a (mon.0) 2121 : cluster 0 osdmap e496: 8 total, 8 up, 8 in 2024-03-11T21:20:17.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:17 smithi146 bash[23809]: audit 2024-03-11T21:20:16.363187+0000 mgr.y (mgr.24380) 1085 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:20:17.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:17 smithi146 bash[23809]: cluster 2024-03-11T21:20:16.556779+0000 osd.0 (osd.0) 532 : cluster 0 3.6 scrub starts 2024-03-11T21:20:17.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:17 smithi146 bash[23809]: cluster 2024-03-11T21:20:16.560983+0000 osd.0 (osd.0) 533 : cluster 0 3.6 scrub ok 2024-03-11T21:20:17.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:17 smithi146 bash[23809]: cluster 2024-03-11T21:20:16.596192+0000 mgr.y (mgr.24380) 1086 : cluster 0 pgmap v1259: 152 pgs: 1 active+clean+scrubbing, 151 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:20:17.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:17 smithi146 bash[23809]: audit 2024-03-11T21:20:16.858999+0000 mon.a (mon.0) 2122 : audit 0 from='client.? 172.21.15.23:0/826944141' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-11T21:20:17.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:17 smithi146 bash[23809]: audit 2024-03-11T21:20:16.892824+0000 mon.a (mon.0) 2123 : audit 0 from='client.? 172.21.15.23:0/2842618213' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-11T21:20:17.384 INFO:teuthology.orchestra.run.smithi023.stdout:120259084697 2024-03-11T21:20:18.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:18 smithi146 bash[23809]: cluster 2024-03-11T21:20:16.566026+0000 osd.3 (osd.3) 658 : cluster 0 6.c scrub starts 2024-03-11T21:20:18.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:18 smithi146 bash[23809]: cluster 2024-03-11T21:20:16.567873+0000 osd.3 (osd.3) 659 : cluster 0 6.c scrub ok 2024-03-11T21:20:18.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:18 smithi146 bash[23809]: cluster 2024-03-11T21:20:16.568255+0000 osd.4 (osd.4) 175 : cluster 0 6.2 scrub starts 2024-03-11T21:20:18.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:18 smithi146 bash[23809]: cluster 2024-03-11T21:20:16.570568+0000 osd.4 (osd.4) 176 : cluster 0 6.2 scrub ok 2024-03-11T21:20:18.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:18 smithi146 bash[23809]: audit 2024-03-11T21:20:17.398516+0000 mon.c (mon.2) 116 : audit 0 from='client.? 172.21.15.23:0/3673094212' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-11T21:20:18.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:18 smithi023 bash[21372]: cluster 2024-03-11T21:20:16.566026+0000 osd.3 (osd.3) 658 : cluster 0 6.c scrub starts 2024-03-11T21:20:18.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:18 smithi023 bash[21372]: cluster 2024-03-11T21:20:16.567873+0000 osd.3 (osd.3) 659 : cluster 0 6.c scrub ok 2024-03-11T21:20:18.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:18 smithi023 bash[21372]: cluster 2024-03-11T21:20:16.568255+0000 osd.4 (osd.4) 175 : cluster 0 6.2 scrub starts 2024-03-11T21:20:18.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:18 smithi023 bash[21372]: cluster 2024-03-11T21:20:16.570568+0000 osd.4 (osd.4) 176 : cluster 0 6.2 scrub ok 2024-03-11T21:20:18.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:18 smithi023 bash[21372]: audit 2024-03-11T21:20:17.398516+0000 mon.c (mon.2) 116 : audit 0 from='client.? 172.21.15.23:0/3673094212' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-11T21:20:18.274 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:18 smithi023 bash[29146]: cluster 2024-03-11T21:20:16.566026+0000 osd.3 (osd.3) 658 : cluster 0 6.c scrub starts 2024-03-11T21:20:18.275 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:18 smithi023 bash[29146]: cluster 2024-03-11T21:20:16.567873+0000 osd.3 (osd.3) 659 : cluster 0 6.c scrub ok 2024-03-11T21:20:18.275 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:18 smithi023 bash[29146]: cluster 2024-03-11T21:20:16.568255+0000 osd.4 (osd.4) 175 : cluster 0 6.2 scrub starts 2024-03-11T21:20:18.275 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:18 smithi023 bash[29146]: cluster 2024-03-11T21:20:16.570568+0000 osd.4 (osd.4) 176 : cluster 0 6.2 scrub ok 2024-03-11T21:20:18.275 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:18 smithi023 bash[29146]: audit 2024-03-11T21:20:17.398516+0000 mon.c (mon.2) 116 : audit 0 from='client.? 172.21.15.23:0/3673094212' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-11T21:20:18.773 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:20:18 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:20:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:20:19.021 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738804 got 34359738808 for osd.0 2024-03-11T21:20:19.021 DEBUG:teuthology.parallel:result is None 2024-03-11T21:20:19.088 INFO:tasks.cephadm.ceph_manager.ceph:need seq 81604379038 got 81604379043 for osd.2 2024-03-11T21:20:19.088 DEBUG:teuthology.parallel:result is None 2024-03-11T21:20:19.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:19 smithi023 bash[21372]: cluster 2024-03-11T21:20:17.524553+0000 osd.4 (osd.4) 177 : cluster 0 2.f scrub starts 2024-03-11T21:20:19.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:19 smithi023 bash[21372]: cluster 2024-03-11T21:20:17.528096+0000 osd.4 (osd.4) 178 : cluster 0 2.f scrub ok 2024-03-11T21:20:19.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:19 smithi023 bash[21372]: cluster 2024-03-11T21:20:18.596980+0000 mgr.y (mgr.24380) 1087 : cluster 0 pgmap v1260: 152 pgs: 1 active+clean+scrubbing, 151 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:20:19.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:19 smithi023 bash[21372]: cluster 2024-03-11T21:20:18.913938+0000 osd.2 (osd.2) 323 : cluster 0 purged_snaps scrub starts 2024-03-11T21:20:19.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:19 smithi023 bash[21372]: cluster 2024-03-11T21:20:18.914029+0000 osd.2 (osd.2) 324 : cluster 0 purged_snaps scrub ok 2024-03-11T21:20:19.274 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:19 smithi023 bash[29146]: cluster 2024-03-11T21:20:17.524553+0000 osd.4 (osd.4) 177 : cluster 0 2.f scrub starts 2024-03-11T21:20:19.274 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:19 smithi023 bash[29146]: cluster 2024-03-11T21:20:17.528096+0000 osd.4 (osd.4) 178 : cluster 0 2.f scrub ok 2024-03-11T21:20:19.274 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:19 smithi023 bash[29146]: cluster 2024-03-11T21:20:18.596980+0000 mgr.y (mgr.24380) 1087 : cluster 0 pgmap v1260: 152 pgs: 1 active+clean+scrubbing, 151 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:20:19.275 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:19 smithi023 bash[29146]: cluster 2024-03-11T21:20:18.913938+0000 osd.2 (osd.2) 323 : cluster 0 purged_snaps scrub starts 2024-03-11T21:20:19.275 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:19 smithi023 bash[29146]: cluster 2024-03-11T21:20:18.914029+0000 osd.2 (osd.2) 324 : cluster 0 purged_snaps scrub ok 2024-03-11T21:20:19.298 INFO:tasks.cephadm.ceph_manager.ceph:need seq 292057776492 got 292057776496 for osd.7 2024-03-11T21:20:19.298 DEBUG:teuthology.parallel:result is None 2024-03-11T21:20:19.502 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:19 smithi146 bash[23809]: cluster 2024-03-11T21:20:17.524553+0000 osd.4 (osd.4) 177 : cluster 0 2.f scrub starts 2024-03-11T21:20:19.502 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:19 smithi146 bash[23809]: cluster 2024-03-11T21:20:17.528096+0000 osd.4 (osd.4) 178 : cluster 0 2.f scrub ok 2024-03-11T21:20:19.502 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:19 smithi146 bash[23809]: cluster 2024-03-11T21:20:18.596980+0000 mgr.y (mgr.24380) 1087 : cluster 0 pgmap v1260: 152 pgs: 1 active+clean+scrubbing, 151 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:20:19.502 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:19 smithi146 bash[23809]: cluster 2024-03-11T21:20:18.913938+0000 osd.2 (osd.2) 323 : cluster 0 purged_snaps scrub starts 2024-03-11T21:20:19.502 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:19 smithi146 bash[23809]: cluster 2024-03-11T21:20:18.914029+0000 osd.2 (osd.2) 324 : cluster 0 purged_snaps scrub ok 2024-03-11T21:20:20.266 INFO:tasks.cephadm.ceph_manager.ceph:need seq 906238099685 got 906238099691 for osd.1 2024-03-11T21:20:20.266 DEBUG:teuthology.parallel:result is None 2024-03-11T21:20:20.273 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:20 smithi023 bash[29146]: cluster 2024-03-11T21:20:18.789940+0000 osd.6 (osd.6) 572 : cluster 0 2.7 scrub starts 2024-03-11T21:20:20.273 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:20 smithi023 bash[29146]: cluster 2024-03-11T21:20:18.792559+0000 osd.6 (osd.6) 573 : cluster 0 2.7 scrub ok 2024-03-11T21:20:20.273 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:20 smithi023 bash[29146]: cluster 2024-03-11T21:20:18.833452+0000 osd.1 (osd.1) 462 : cluster 0 purged_snaps scrub starts 2024-03-11T21:20:20.273 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:20 smithi023 bash[29146]: cluster 2024-03-11T21:20:18.833596+0000 osd.1 (osd.1) 463 : cluster 0 purged_snaps scrub ok 2024-03-11T21:20:20.274 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:20 smithi023 bash[29146]: cluster 2024-03-11T21:20:19.055051+0000 mon.a (mon.0) 2124 : cluster 0 osdmap e497: 8 total, 8 up, 8 in 2024-03-11T21:20:20.274 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:20 smithi023 bash[29146]: cluster 2024-03-11T21:20:19.080550+0000 osd.1 (osd.1) 464 : cluster 0 3.0 scrub starts 2024-03-11T21:20:20.274 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:20 smithi023 bash[29146]: cluster 2024-03-11T21:20:19.085529+0000 osd.1 (osd.1) 465 : cluster 0 3.0 scrub ok 2024-03-11T21:20:20.274 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:20 smithi023 bash[29146]: cluster 2024-03-11T21:20:19.436901+0000 osd.7 (osd.7) 725 : cluster 0 3.13 deep-scrub starts 2024-03-11T21:20:20.274 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:20 smithi023 bash[29146]: cluster 2024-03-11T21:20:19.441732+0000 osd.7 (osd.7) 726 : cluster 0 3.13 deep-scrub ok 2024-03-11T21:20:20.274 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:20 smithi023 bash[29146]: cluster 2024-03-11T21:20:19.602363+0000 osd.5 (osd.5) 703 : cluster 0 6.19 scrub starts 2024-03-11T21:20:20.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:20 smithi023 bash[21372]: cluster 2024-03-11T21:20:18.789940+0000 osd.6 (osd.6) 572 : cluster 0 2.7 scrub starts 2024-03-11T21:20:20.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:20 smithi023 bash[21372]: cluster 2024-03-11T21:20:18.792559+0000 osd.6 (osd.6) 573 : cluster 0 2.7 scrub ok 2024-03-11T21:20:20.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:20 smithi023 bash[21372]: cluster 2024-03-11T21:20:18.833452+0000 osd.1 (osd.1) 462 : cluster 0 purged_snaps scrub starts 2024-03-11T21:20:20.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:20 smithi023 bash[21372]: cluster 2024-03-11T21:20:18.833596+0000 osd.1 (osd.1) 463 : cluster 0 purged_snaps scrub ok 2024-03-11T21:20:20.275 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:20 smithi023 bash[21372]: cluster 2024-03-11T21:20:19.055051+0000 mon.a (mon.0) 2124 : cluster 0 osdmap e497: 8 total, 8 up, 8 in 2024-03-11T21:20:20.275 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:20 smithi023 bash[21372]: cluster 2024-03-11T21:20:19.080550+0000 osd.1 (osd.1) 464 : cluster 0 3.0 scrub starts 2024-03-11T21:20:20.275 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:20 smithi023 bash[21372]: cluster 2024-03-11T21:20:19.085529+0000 osd.1 (osd.1) 465 : cluster 0 3.0 scrub ok 2024-03-11T21:20:20.275 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:20 smithi023 bash[21372]: cluster 2024-03-11T21:20:19.436901+0000 osd.7 (osd.7) 725 : cluster 0 3.13 deep-scrub starts 2024-03-11T21:20:20.275 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:20 smithi023 bash[21372]: cluster 2024-03-11T21:20:19.441732+0000 osd.7 (osd.7) 726 : cluster 0 3.13 deep-scrub ok 2024-03-11T21:20:20.275 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:20 smithi023 bash[21372]: cluster 2024-03-11T21:20:19.602363+0000 osd.5 (osd.5) 703 : cluster 0 6.19 scrub starts 2024-03-11T21:20:20.502 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:20 smithi146 bash[23809]: cluster 2024-03-11T21:20:18.789940+0000 osd.6 (osd.6) 572 : cluster 0 2.7 scrub starts 2024-03-11T21:20:20.502 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:20 smithi146 bash[23809]: cluster 2024-03-11T21:20:18.792559+0000 osd.6 (osd.6) 573 : cluster 0 2.7 scrub ok 2024-03-11T21:20:20.502 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:20 smithi146 bash[23809]: cluster 2024-03-11T21:20:18.833452+0000 osd.1 (osd.1) 462 : cluster 0 purged_snaps scrub starts 2024-03-11T21:20:20.503 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:20 smithi146 bash[23809]: cluster 2024-03-11T21:20:18.833596+0000 osd.1 (osd.1) 463 : cluster 0 purged_snaps scrub ok 2024-03-11T21:20:20.503 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:20 smithi146 bash[23809]: cluster 2024-03-11T21:20:19.055051+0000 mon.a (mon.0) 2124 : cluster 0 osdmap e497: 8 total, 8 up, 8 in 2024-03-11T21:20:20.503 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:20 smithi146 bash[23809]: cluster 2024-03-11T21:20:19.080550+0000 osd.1 (osd.1) 464 : cluster 0 3.0 scrub starts 2024-03-11T21:20:20.503 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:20 smithi146 bash[23809]: cluster 2024-03-11T21:20:19.085529+0000 osd.1 (osd.1) 465 : cluster 0 3.0 scrub ok 2024-03-11T21:20:20.503 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:20 smithi146 bash[23809]: cluster 2024-03-11T21:20:19.436901+0000 osd.7 (osd.7) 725 : cluster 0 3.13 deep-scrub starts 2024-03-11T21:20:20.503 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:20 smithi146 bash[23809]: cluster 2024-03-11T21:20:19.441732+0000 osd.7 (osd.7) 726 : cluster 0 3.13 deep-scrub ok 2024-03-11T21:20:20.503 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:20 smithi146 bash[23809]: cluster 2024-03-11T21:20:19.602363+0000 osd.5 (osd.5) 703 : cluster 0 6.19 scrub starts 2024-03-11T21:20:20.816 INFO:tasks.cephadm.ceph_manager.ceph:need seq 120259084692 got 120259084697 for osd.3 2024-03-11T21:20:20.816 DEBUG:teuthology.parallel:result is None 2024-03-11T21:20:20.890 INFO:tasks.cephadm.ceph_manager.ceph:need seq 244813136246 got 244813136250 for osd.6 2024-03-11T21:20:20.890 DEBUG:teuthology.parallel:result is None 2024-03-11T21:20:20.890 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-03-11T21:20:20.893 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T21:20:21.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:21 smithi023 bash[21372]: cluster 2024-03-11T21:20:19.572139+0000 osd.4 (osd.4) 179 : cluster 0 1.e scrub starts 2024-03-11T21:20:21.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:21 smithi023 bash[21372]: cluster 2024-03-11T21:20:19.574626+0000 osd.4 (osd.4) 180 : cluster 0 1.e scrub ok 2024-03-11T21:20:21.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:21 smithi023 bash[21372]: cluster 2024-03-11T21:20:19.717012+0000 osd.5 (osd.5) 704 : cluster 0 6.19 scrub ok 2024-03-11T21:20:21.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:21 smithi023 bash[21372]: cluster 2024-03-11T21:20:20.121355+0000 osd.1 (osd.1) 466 : cluster 0 2.a scrub starts 2024-03-11T21:20:21.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:21 smithi023 bash[21372]: cluster 2024-03-11T21:20:20.123563+0000 osd.1 (osd.1) 467 : cluster 0 2.a scrub ok 2024-03-11T21:20:21.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:21 smithi023 bash[21372]: cluster 2024-03-11T21:20:20.597621+0000 mgr.y (mgr.24380) 1088 : cluster 0 pgmap v1262: 152 pgs: 1 active+clean+scrubbing, 151 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:20:21.274 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:21 smithi023 bash[29146]: cluster 2024-03-11T21:20:19.572139+0000 osd.4 (osd.4) 179 : cluster 0 1.e scrub starts 2024-03-11T21:20:21.274 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:21 smithi023 bash[29146]: cluster 2024-03-11T21:20:19.574626+0000 osd.4 (osd.4) 180 : cluster 0 1.e scrub ok 2024-03-11T21:20:21.275 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:21 smithi023 bash[29146]: cluster 2024-03-11T21:20:19.717012+0000 osd.5 (osd.5) 704 : cluster 0 6.19 scrub ok 2024-03-11T21:20:21.275 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:21 smithi023 bash[29146]: cluster 2024-03-11T21:20:20.121355+0000 osd.1 (osd.1) 466 : cluster 0 2.a scrub starts 2024-03-11T21:20:21.275 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:21 smithi023 bash[29146]: cluster 2024-03-11T21:20:20.123563+0000 osd.1 (osd.1) 467 : cluster 0 2.a scrub ok 2024-03-11T21:20:21.275 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:21 smithi023 bash[29146]: cluster 2024-03-11T21:20:20.597621+0000 mgr.y (mgr.24380) 1088 : cluster 0 pgmap v1262: 152 pgs: 1 active+clean+scrubbing, 151 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:20:21.502 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:21 smithi146 bash[23809]: cluster 2024-03-11T21:20:19.572139+0000 osd.4 (osd.4) 179 : cluster 0 1.e scrub starts 2024-03-11T21:20:21.502 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:21 smithi146 bash[23809]: cluster 2024-03-11T21:20:19.574626+0000 osd.4 (osd.4) 180 : cluster 0 1.e scrub ok 2024-03-11T21:20:21.502 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:21 smithi146 bash[23809]: cluster 2024-03-11T21:20:19.717012+0000 osd.5 (osd.5) 704 : cluster 0 6.19 scrub ok 2024-03-11T21:20:21.502 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:21 smithi146 bash[23809]: cluster 2024-03-11T21:20:20.121355+0000 osd.1 (osd.1) 466 : cluster 0 2.a scrub starts 2024-03-11T21:20:21.502 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:21 smithi146 bash[23809]: cluster 2024-03-11T21:20:20.123563+0000 osd.1 (osd.1) 467 : cluster 0 2.a scrub ok 2024-03-11T21:20:21.502 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:21 smithi146 bash[23809]: cluster 2024-03-11T21:20:20.597621+0000 mgr.y (mgr.24380) 1088 : cluster 0 pgmap v1262: 152 pgs: 1 active+clean+scrubbing, 151 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:20:22.502 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:22 smithi146 bash[23809]: cluster 2024-03-11T21:20:20.619727+0000 osd.3 (osd.3) 660 : cluster 0 3.b deep-scrub starts 2024-03-11T21:20:22.502 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:22 smithi146 bash[23809]: cluster 2024-03-11T21:20:20.624069+0000 osd.3 (osd.3) 661 : cluster 0 3.b deep-scrub ok 2024-03-11T21:20:22.502 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:22 smithi146 bash[23809]: cluster 2024-03-11T21:20:20.641855+0000 osd.5 (osd.5) 705 : cluster 0 2.2 scrub starts 2024-03-11T21:20:22.502 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:22 smithi146 bash[23809]: cluster 2024-03-11T21:20:20.644220+0000 osd.5 (osd.5) 706 : cluster 0 2.2 scrub ok 2024-03-11T21:20:22.523 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:22 smithi023 bash[21372]: cluster 2024-03-11T21:20:20.619727+0000 osd.3 (osd.3) 660 : cluster 0 3.b deep-scrub starts 2024-03-11T21:20:22.524 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:22 smithi023 bash[21372]: cluster 2024-03-11T21:20:20.624069+0000 osd.3 (osd.3) 661 : cluster 0 3.b deep-scrub ok 2024-03-11T21:20:22.524 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:22 smithi023 bash[21372]: cluster 2024-03-11T21:20:20.641855+0000 osd.5 (osd.5) 705 : cluster 0 2.2 scrub starts 2024-03-11T21:20:22.524 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:22 smithi023 bash[21372]: cluster 2024-03-11T21:20:20.644220+0000 osd.5 (osd.5) 706 : cluster 0 2.2 scrub ok 2024-03-11T21:20:22.524 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:22 smithi023 bash[29146]: cluster 2024-03-11T21:20:20.619727+0000 osd.3 (osd.3) 660 : cluster 0 3.b deep-scrub starts 2024-03-11T21:20:22.524 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:22 smithi023 bash[29146]: cluster 2024-03-11T21:20:20.624069+0000 osd.3 (osd.3) 661 : cluster 0 3.b deep-scrub ok 2024-03-11T21:20:22.524 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:22 smithi023 bash[29146]: cluster 2024-03-11T21:20:20.641855+0000 osd.5 (osd.5) 705 : cluster 0 2.2 scrub starts 2024-03-11T21:20:22.524 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:22 smithi023 bash[29146]: cluster 2024-03-11T21:20:20.644220+0000 osd.5 (osd.5) 706 : cluster 0 2.2 scrub ok 2024-03-11T21:20:23.502 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:23 smithi146 bash[23809]: cluster 2024-03-11T21:20:21.902669+0000 osd.6 (osd.6) 574 : cluster 0 1.f deep-scrub starts 2024-03-11T21:20:23.502 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:23 smithi146 bash[23809]: cluster 2024-03-11T21:20:21.905286+0000 osd.6 (osd.6) 575 : cluster 0 1.f deep-scrub ok 2024-03-11T21:20:23.502 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:23 smithi146 bash[23809]: audit 2024-03-11T21:20:22.187682+0000 mon.a (mon.0) 2125 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:20:23.502 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:23 smithi146 bash[23809]: cluster 2024-03-11T21:20:22.598925+0000 mgr.y (mgr.24380) 1089 : cluster 0 pgmap v1263: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:20:23.523 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:23 smithi023 bash[21372]: cluster 2024-03-11T21:20:21.902669+0000 osd.6 (osd.6) 574 : cluster 0 1.f deep-scrub starts 2024-03-11T21:20:23.523 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:23 smithi023 bash[21372]: cluster 2024-03-11T21:20:21.905286+0000 osd.6 (osd.6) 575 : cluster 0 1.f deep-scrub ok 2024-03-11T21:20:23.524 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:23 smithi023 bash[21372]: audit 2024-03-11T21:20:22.187682+0000 mon.a (mon.0) 2125 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:20:23.524 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:23 smithi023 bash[21372]: cluster 2024-03-11T21:20:22.598925+0000 mgr.y (mgr.24380) 1089 : cluster 0 pgmap v1263: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:20:23.524 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:23 smithi023 bash[29146]: cluster 2024-03-11T21:20:21.902669+0000 osd.6 (osd.6) 574 : cluster 0 1.f deep-scrub starts 2024-03-11T21:20:23.524 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:23 smithi023 bash[29146]: cluster 2024-03-11T21:20:21.905286+0000 osd.6 (osd.6) 575 : cluster 0 1.f deep-scrub ok 2024-03-11T21:20:23.524 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:23 smithi023 bash[29146]: audit 2024-03-11T21:20:22.187682+0000 mon.a (mon.0) 2125 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:20:23.524 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:23 smithi023 bash[29146]: cluster 2024-03-11T21:20:22.598925+0000 mgr.y (mgr.24380) 1089 : cluster 0 pgmap v1263: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:20:24.502 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:24 smithi146 bash[23809]: cluster 2024-03-11T21:20:22.582941+0000 osd.3 (osd.3) 662 : cluster 0 4.17 scrub starts 2024-03-11T21:20:24.502 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:24 smithi146 bash[23809]: cluster 2024-03-11T21:20:22.585273+0000 osd.3 (osd.3) 663 : cluster 0 4.17 scrub ok 2024-03-11T21:20:24.502 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:24 smithi146 bash[23809]: cluster 2024-03-11T21:20:22.652065+0000 osd.4 (osd.4) 181 : cluster 0 1.5 scrub starts 2024-03-11T21:20:24.502 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:24 smithi146 bash[23809]: cluster 2024-03-11T21:20:22.654643+0000 osd.4 (osd.4) 182 : cluster 0 1.5 scrub ok 2024-03-11T21:20:24.502 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:24 smithi146 bash[23809]: cluster 2024-03-11T21:20:23.448319+0000 osd.7 (osd.7) 727 : cluster 0 3.15 scrub starts 2024-03-11T21:20:24.502 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:24 smithi146 bash[23809]: cluster 2024-03-11T21:20:23.452457+0000 osd.7 (osd.7) 728 : cluster 0 3.15 scrub ok 2024-03-11T21:20:24.502 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:24 smithi146 bash[23809]: cluster 2024-03-11T21:20:23.514732+0000 osd.0 (osd.0) 534 : cluster 0 6.18 scrub starts 2024-03-11T21:20:24.503 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:24 smithi146 bash[23809]: cluster 2024-03-11T21:20:23.597606+0000 osd.0 (osd.0) 535 : cluster 0 6.18 scrub ok 2024-03-11T21:20:24.524 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:24 smithi023 bash[21372]: cluster 2024-03-11T21:20:22.582941+0000 osd.3 (osd.3) 662 : cluster 0 4.17 scrub starts 2024-03-11T21:20:24.524 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:24 smithi023 bash[21372]: cluster 2024-03-11T21:20:22.585273+0000 osd.3 (osd.3) 663 : cluster 0 4.17 scrub ok 2024-03-11T21:20:24.524 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:24 smithi023 bash[21372]: cluster 2024-03-11T21:20:22.652065+0000 osd.4 (osd.4) 181 : cluster 0 1.5 scrub starts 2024-03-11T21:20:24.524 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:24 smithi023 bash[21372]: cluster 2024-03-11T21:20:22.654643+0000 osd.4 (osd.4) 182 : cluster 0 1.5 scrub ok 2024-03-11T21:20:24.524 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:24 smithi023 bash[21372]: cluster 2024-03-11T21:20:23.448319+0000 osd.7 (osd.7) 727 : cluster 0 3.15 scrub starts 2024-03-11T21:20:24.524 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:24 smithi023 bash[21372]: cluster 2024-03-11T21:20:23.452457+0000 osd.7 (osd.7) 728 : cluster 0 3.15 scrub ok 2024-03-11T21:20:24.524 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:24 smithi023 bash[21372]: cluster 2024-03-11T21:20:23.514732+0000 osd.0 (osd.0) 534 : cluster 0 6.18 scrub starts 2024-03-11T21:20:24.524 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:24 smithi023 bash[21372]: cluster 2024-03-11T21:20:23.597606+0000 osd.0 (osd.0) 535 : cluster 0 6.18 scrub ok 2024-03-11T21:20:24.524 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:24 smithi023 bash[29146]: cluster 2024-03-11T21:20:22.582941+0000 osd.3 (osd.3) 662 : cluster 0 4.17 scrub starts 2024-03-11T21:20:24.524 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:24 smithi023 bash[29146]: cluster 2024-03-11T21:20:22.585273+0000 osd.3 (osd.3) 663 : cluster 0 4.17 scrub ok 2024-03-11T21:20:24.524 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:24 smithi023 bash[29146]: cluster 2024-03-11T21:20:22.652065+0000 osd.4 (osd.4) 181 : cluster 0 1.5 scrub starts 2024-03-11T21:20:24.524 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:24 smithi023 bash[29146]: cluster 2024-03-11T21:20:22.654643+0000 osd.4 (osd.4) 182 : cluster 0 1.5 scrub ok 2024-03-11T21:20:24.525 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:24 smithi023 bash[29146]: cluster 2024-03-11T21:20:23.448319+0000 osd.7 (osd.7) 727 : cluster 0 3.15 scrub starts 2024-03-11T21:20:24.525 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:24 smithi023 bash[29146]: cluster 2024-03-11T21:20:23.452457+0000 osd.7 (osd.7) 728 : cluster 0 3.15 scrub ok 2024-03-11T21:20:24.525 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:24 smithi023 bash[29146]: cluster 2024-03-11T21:20:23.514732+0000 osd.0 (osd.0) 534 : cluster 0 6.18 scrub starts 2024-03-11T21:20:24.525 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:24 smithi023 bash[29146]: cluster 2024-03-11T21:20:23.597606+0000 osd.0 (osd.0) 535 : cluster 0 6.18 scrub ok 2024-03-11T21:20:25.502 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:25 smithi146 bash[23809]: cluster 2024-03-11T21:20:23.654135+0000 osd.5 (osd.5) 707 : cluster 0 1.10 scrub starts 2024-03-11T21:20:25.502 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:25 smithi146 bash[23809]: cluster 2024-03-11T21:20:23.656808+0000 osd.5 (osd.5) 708 : cluster 0 1.10 scrub ok 2024-03-11T21:20:25.502 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:25 smithi146 bash[23809]: cluster 2024-03-11T21:20:24.066475+0000 osd.1 (osd.1) 468 : cluster 0 2.d scrub starts 2024-03-11T21:20:25.502 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:25 smithi146 bash[23809]: cluster 2024-03-11T21:20:24.069138+0000 osd.1 (osd.1) 469 : cluster 0 2.d scrub ok 2024-03-11T21:20:25.502 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:25 smithi146 bash[23809]: cluster 2024-03-11T21:20:24.599985+0000 mgr.y (mgr.24380) 1090 : cluster 0 pgmap v1264: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 594 B/s rd, 0 op/s 2024-03-11T21:20:25.523 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:25 smithi023 bash[21372]: cluster 2024-03-11T21:20:23.654135+0000 osd.5 (osd.5) 707 : cluster 0 1.10 scrub starts 2024-03-11T21:20:25.524 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:25 smithi023 bash[21372]: cluster 2024-03-11T21:20:23.656808+0000 osd.5 (osd.5) 708 : cluster 0 1.10 scrub ok 2024-03-11T21:20:25.524 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:25 smithi023 bash[21372]: cluster 2024-03-11T21:20:24.066475+0000 osd.1 (osd.1) 468 : cluster 0 2.d scrub starts 2024-03-11T21:20:25.524 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:25 smithi023 bash[21372]: cluster 2024-03-11T21:20:24.069138+0000 osd.1 (osd.1) 469 : cluster 0 2.d scrub ok 2024-03-11T21:20:25.524 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:25 smithi023 bash[21372]: cluster 2024-03-11T21:20:24.599985+0000 mgr.y (mgr.24380) 1090 : cluster 0 pgmap v1264: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 594 B/s rd, 0 op/s 2024-03-11T21:20:25.524 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:25 smithi023 bash[29146]: cluster 2024-03-11T21:20:23.654135+0000 osd.5 (osd.5) 707 : cluster 0 1.10 scrub starts 2024-03-11T21:20:25.524 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:25 smithi023 bash[29146]: cluster 2024-03-11T21:20:23.656808+0000 osd.5 (osd.5) 708 : cluster 0 1.10 scrub ok 2024-03-11T21:20:25.524 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:25 smithi023 bash[29146]: cluster 2024-03-11T21:20:24.066475+0000 osd.1 (osd.1) 468 : cluster 0 2.d scrub starts 2024-03-11T21:20:25.524 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:25 smithi023 bash[29146]: cluster 2024-03-11T21:20:24.069138+0000 osd.1 (osd.1) 469 : cluster 0 2.d scrub ok 2024-03-11T21:20:25.524 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:25 smithi023 bash[29146]: cluster 2024-03-11T21:20:24.599985+0000 mgr.y (mgr.24380) 1090 : cluster 0 pgmap v1264: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 594 B/s rd, 0 op/s 2024-03-11T21:20:25.690 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:20:26.358 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:26 smithi146 bash[23809]: cluster 2024-03-11T21:20:24.677729+0000 osd.4 (osd.4) 183 : cluster 0 3.3 scrub starts 2024-03-11T21:20:26.358 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:26 smithi146 bash[23809]: cluster 2024-03-11T21:20:24.683037+0000 osd.4 (osd.4) 184 : cluster 0 3.3 scrub ok 2024-03-11T21:20:26.359 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:26 smithi146 bash[23809]: cluster 2024-03-11T21:20:25.044828+0000 osd.1 (osd.1) 470 : cluster 0 1.13 scrub starts 2024-03-11T21:20:26.359 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:26 smithi146 bash[23809]: cluster 2024-03-11T21:20:25.047488+0000 osd.1 (osd.1) 471 : cluster 0 1.13 scrub ok 2024-03-11T21:20:26.359 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:26 smithi146 bash[23809]: cluster 2024-03-11T21:20:25.478986+0000 osd.7 (osd.7) 729 : cluster 0 6.12 scrub starts 2024-03-11T21:20:26.359 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:26 smithi146 bash[23809]: cluster 2024-03-11T21:20:25.511991+0000 osd.7 (osd.7) 730 : cluster 0 6.12 scrub ok 2024-03-11T21:20:26.523 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:26 smithi023 bash[21372]: cluster 2024-03-11T21:20:24.677729+0000 osd.4 (osd.4) 183 : cluster 0 3.3 scrub starts 2024-03-11T21:20:26.524 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:26 smithi023 bash[21372]: cluster 2024-03-11T21:20:24.683037+0000 osd.4 (osd.4) 184 : cluster 0 3.3 scrub ok 2024-03-11T21:20:26.524 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:26 smithi023 bash[21372]: cluster 2024-03-11T21:20:25.044828+0000 osd.1 (osd.1) 470 : cluster 0 1.13 scrub starts 2024-03-11T21:20:26.524 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:26 smithi023 bash[21372]: cluster 2024-03-11T21:20:25.047488+0000 osd.1 (osd.1) 471 : cluster 0 1.13 scrub ok 2024-03-11T21:20:26.524 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:26 smithi023 bash[21372]: cluster 2024-03-11T21:20:25.478986+0000 osd.7 (osd.7) 729 : cluster 0 6.12 scrub starts 2024-03-11T21:20:26.524 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:26 smithi023 bash[21372]: cluster 2024-03-11T21:20:25.511991+0000 osd.7 (osd.7) 730 : cluster 0 6.12 scrub ok 2024-03-11T21:20:26.524 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:26 smithi023 bash[29146]: cluster 2024-03-11T21:20:24.677729+0000 osd.4 (osd.4) 183 : cluster 0 3.3 scrub starts 2024-03-11T21:20:26.524 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:26 smithi023 bash[29146]: cluster 2024-03-11T21:20:24.683037+0000 osd.4 (osd.4) 184 : cluster 0 3.3 scrub ok 2024-03-11T21:20:26.524 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:26 smithi023 bash[29146]: cluster 2024-03-11T21:20:25.044828+0000 osd.1 (osd.1) 470 : cluster 0 1.13 scrub starts 2024-03-11T21:20:26.524 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:26 smithi023 bash[29146]: cluster 2024-03-11T21:20:25.047488+0000 osd.1 (osd.1) 471 : cluster 0 1.13 scrub ok 2024-03-11T21:20:26.525 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:26 smithi023 bash[29146]: cluster 2024-03-11T21:20:25.478986+0000 osd.7 (osd.7) 729 : cluster 0 6.12 scrub starts 2024-03-11T21:20:26.525 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:26 smithi023 bash[29146]: cluster 2024-03-11T21:20:25.511991+0000 osd.7 (osd.7) 730 : cluster 0 6.12 scrub ok 2024-03-11T21:20:26.752 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:20:26 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:20:27.249 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:27 smithi023 bash[21372]: cluster 2024-03-11T21:20:25.652978+0000 osd.5 (osd.5) 709 : cluster 0 2.16 deep-scrub starts 2024-03-11T21:20:27.249 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:27 smithi023 bash[21372]: cluster 2024-03-11T21:20:25.655256+0000 osd.5 (osd.5) 710 : cluster 0 2.16 deep-scrub ok 2024-03-11T21:20:27.249 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:27 smithi023 bash[21372]: cluster 2024-03-11T21:20:25.918898+0000 osd.6 (osd.6) 576 : cluster 0 2.11 scrub starts 2024-03-11T21:20:27.250 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:27 smithi023 bash[21372]: cluster 2024-03-11T21:20:25.922300+0000 osd.6 (osd.6) 577 : cluster 0 2.11 scrub ok 2024-03-11T21:20:27.250 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:27 smithi023 bash[21372]: audit 2024-03-11T21:20:26.372496+0000 mgr.y (mgr.24380) 1091 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:20:27.250 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:27 smithi023 bash[21372]: cluster 2024-03-11T21:20:26.601765+0000 mgr.y (mgr.24380) 1092 : cluster 0 pgmap v1265: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:20:27.250 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:27 smithi023 bash[29146]: cluster 2024-03-11T21:20:25.652978+0000 osd.5 (osd.5) 709 : cluster 0 2.16 deep-scrub starts 2024-03-11T21:20:27.250 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:27 smithi023 bash[29146]: cluster 2024-03-11T21:20:25.655256+0000 osd.5 (osd.5) 710 : cluster 0 2.16 deep-scrub ok 2024-03-11T21:20:27.250 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:27 smithi023 bash[29146]: cluster 2024-03-11T21:20:25.918898+0000 osd.6 (osd.6) 576 : cluster 0 2.11 scrub starts 2024-03-11T21:20:27.250 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:27 smithi023 bash[29146]: cluster 2024-03-11T21:20:25.922300+0000 osd.6 (osd.6) 577 : cluster 0 2.11 scrub ok 2024-03-11T21:20:27.250 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:27 smithi023 bash[29146]: audit 2024-03-11T21:20:26.372496+0000 mgr.y (mgr.24380) 1091 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:20:27.250 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:27 smithi023 bash[29146]: cluster 2024-03-11T21:20:26.601765+0000 mgr.y (mgr.24380) 1092 : cluster 0 pgmap v1265: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:20:27.502 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:27 smithi146 bash[23809]: cluster 2024-03-11T21:20:25.652978+0000 osd.5 (osd.5) 709 : cluster 0 2.16 deep-scrub starts 2024-03-11T21:20:27.502 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:27 smithi146 bash[23809]: cluster 2024-03-11T21:20:25.655256+0000 osd.5 (osd.5) 710 : cluster 0 2.16 deep-scrub ok 2024-03-11T21:20:27.502 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:27 smithi146 bash[23809]: cluster 2024-03-11T21:20:25.918898+0000 osd.6 (osd.6) 576 : cluster 0 2.11 scrub starts 2024-03-11T21:20:27.502 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:27 smithi146 bash[23809]: cluster 2024-03-11T21:20:25.922300+0000 osd.6 (osd.6) 577 : cluster 0 2.11 scrub ok 2024-03-11T21:20:27.502 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:27 smithi146 bash[23809]: audit 2024-03-11T21:20:26.372496+0000 mgr.y (mgr.24380) 1091 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:20:27.502 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:27 smithi146 bash[23809]: cluster 2024-03-11T21:20:26.601765+0000 mgr.y (mgr.24380) 1092 : cluster 0 pgmap v1265: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:20:28.666 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:20:28 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:20:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:20:29.088 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-11T21:20:29.091 INFO:teuthology.orchestra.run.smithi023.stderr:dumped all 2024-03-11T21:20:29.828 INFO:teuthology.orchestra.run.smithi023.stdout:{"pg_ready":true,"pg_map":{"version":1266,"stamp":"2024-03-11T21:20:28.602088+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596392,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":3220,"num_read_kb":2972,"num_write":1008,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":281,"num_bytes_recovered":6634,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3975,"ondisk_log_size":3975,"up":456,"acting":456,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":456,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":4500480,"kb_used_data":7676,"kb_used_omap":21,"kb_used_meta":4492714,"kb_avail":745460736,"statfs":{"total":767960285184,"available":763351793664,"internally_reserved":0,"allocated":7860224,"data_stored":5234128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22089,"internal_metadata":4600539575},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.005691"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":938,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.050640+0000","last_change":"2024-03-11T21:20:02.741954+0000","last_active":"2024-03-11T21:20:19.050640+0000","last_peered":"2024-03-11T21:20:19.050640+0000","last_clean":"2024-03-11T21:20:19.050640+0000","last_became_active":"2024-03-11T20:50:34.338390+0000","last_became_peered":"2024-03-11T20:50:34.338390+0000","last_unstale":"2024-03-11T21:20:19.050640+0000","last_undegraded":"2024-03-11T21:20:19.050640+0000","last_fullsized":"2024-03-11T21:20:19.050640+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:20:02.741789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:20:02.741789+0000","last_clean_scrub_stamp":"2024-03-11T21:20:02.741789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:03.379591+0000","scrub_duration":211,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":954,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.050468+0000","last_change":"2024-03-11T21:19:28.470700+0000","last_active":"2024-03-11T21:20:19.050468+0000","last_peered":"2024-03-11T21:20:19.050468+0000","last_clean":"2024-03-11T21:20:19.050468+0000","last_became_active":"2024-03-11T21:14:50.707251+0000","last_became_peered":"2024-03-11T21:14:50.707251+0000","last_unstale":"2024-03-11T21:20:19.050468+0000","last_undegraded":"2024-03-11T21:20:19.050468+0000","last_fullsized":"2024-03-11T21:20:19.050468+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:19:28.470554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:19:28.470554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:52.566247+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":182,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.266349+0000","last_change":"2024-03-11T21:19:05.337275+0000","last_active":"2024-03-11T21:20:19.266349+0000","last_peered":"2024-03-11T21:20:19.266349+0000","last_clean":"2024-03-11T21:20:19.266349+0000","last_became_active":"2024-03-11T21:14:50.431630+0000","last_became_peered":"2024-03-11T21:14:50.431630+0000","last_unstale":"2024-03-11T21:20:19.266349+0000","last_undegraded":"2024-03-11T21:20:19.266349+0000","last_fullsized":"2024-03-11T21:20:19.266349+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:19:05.337059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:11:05.884900+0000","last_clean_scrub_stamp":"2024-03-11T21:19:05.337059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:28.708101+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"358'30","reported_seq":994,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.052515+0000","last_change":"2024-03-11T21:19:12.650724+0000","last_active":"2024-03-11T21:20:19.052515+0000","last_peered":"2024-03-11T21:20:19.052515+0000","last_clean":"2024-03-11T21:20:19.052515+0000","last_became_active":"2024-03-11T20:50:30.101721+0000","last_became_peered":"2024-03-11T20:50:30.101721+0000","last_unstale":"2024-03-11T21:20:19.052515+0000","last_undegraded":"2024-03-11T21:20:19.052515+0000","last_fullsized":"2024-03-11T21:20:19.052515+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'30","last_scrub_stamp":"2024-03-11T21:19:12.650411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:19:12.650411+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:33.733558+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":474,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.266189+0000","last_change":"2024-03-11T21:19:30.434951+0000","last_active":"2024-03-11T21:20:19.266189+0000","last_peered":"2024-03-11T21:20:19.266189+0000","last_clean":"2024-03-11T21:20:19.266189+0000","last_became_active":"2024-03-11T21:18:53.662597+0000","last_became_peered":"2024-03-11T21:18:53.662597+0000","last_unstale":"2024-03-11T21:20:19.266189+0000","last_undegraded":"2024-03-11T21:20:19.266189+0000","last_fullsized":"2024-03-11T21:20:19.266189+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:19:30.434749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:55.545758+0000","last_clean_scrub_stamp":"2024-03-11T21:19:30.434749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:40.186016+0000","scrub_duration":71,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":488,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.265153+0000","last_change":"2024-03-11T21:19:27.324018+0000","last_active":"2024-03-11T21:20:19.265153+0000","last_peered":"2024-03-11T21:20:19.265153+0000","last_clean":"2024-03-11T21:20:19.265153+0000","last_became_active":"2024-03-11T21:18:53.658414+0000","last_became_peered":"2024-03-11T21:18:53.658414+0000","last_unstale":"2024-03-11T21:20:19.265153+0000","last_undegraded":"2024-03-11T21:20:19.265153+0000","last_fullsized":"2024-03-11T21:20:19.265153+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:19:27.323785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:54:13.660325+0000","last_clean_scrub_stamp":"2024-03-11T21:19:27.323785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:36.393830+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":956,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.052281+0000","last_change":"2024-03-11T21:19:21.518793+0000","last_active":"2024-03-11T21:20:19.052281+0000","last_peered":"2024-03-11T21:20:19.052281+0000","last_clean":"2024-03-11T21:20:19.052281+0000","last_became_active":"2024-03-11T21:14:50.435212+0000","last_became_peered":"2024-03-11T21:14:50.435212+0000","last_unstale":"2024-03-11T21:20:19.052281+0000","last_undegraded":"2024-03-11T21:20:19.052281+0000","last_fullsized":"2024-03-11T21:20:19.052281+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:19:21.518361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:19:21.518361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:35.540077+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"358'27","reported_seq":202,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.265233+0000","last_change":"2024-03-11T21:19:47.699324+0000","last_active":"2024-03-11T21:20:19.265233+0000","last_peered":"2024-03-11T21:20:19.265233+0000","last_clean":"2024-03-11T21:20:19.265233+0000","last_became_active":"2024-03-11T21:14:50.704536+0000","last_became_peered":"2024-03-11T21:14:50.704536+0000","last_unstale":"2024-03-11T21:20:19.265233+0000","last_undegraded":"2024-03-11T21:20:19.265233+0000","last_fullsized":"2024-03-11T21:20:19.265233+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'27","last_scrub_stamp":"2024-03-11T21:19:47.699225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:19:47.699225+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:06.025650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":983,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.717256+0000","last_change":"2024-03-11T21:20:19.717374+0000","last_active":"2024-03-11T21:20:19.717256+0000","last_peered":"2024-03-11T21:20:19.717256+0000","last_clean":"2024-03-11T21:20:19.717256+0000","last_became_active":"2024-03-11T21:00:48.460694+0000","last_became_peered":"2024-03-11T21:00:48.460694+0000","last_unstale":"2024-03-11T21:20:19.717256+0000","last_undegraded":"2024-03-11T21:20:19.717256+0000","last_fullsized":"2024-03-11T21:20:19.717256+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:20:19.717024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:41.711864+0000","last_clean_scrub_stamp":"2024-03-11T21:20:19.717024+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:29.793290+0000","scrub_duration":119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":470,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.266322+0000","last_change":"2024-03-11T21:20:07.529538+0000","last_active":"2024-03-11T21:20:19.266322+0000","last_peered":"2024-03-11T21:20:19.266322+0000","last_clean":"2024-03-11T21:20:19.266322+0000","last_became_active":"2024-03-11T21:18:53.658449+0000","last_became_peered":"2024-03-11T21:18:53.658449+0000","last_unstale":"2024-03-11T21:20:19.266322+0000","last_undegraded":"2024-03-11T21:20:19.266322+0000","last_fullsized":"2024-03-11T21:20:19.266322+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:20:07.529314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:16:51.642949+0000","last_clean_scrub_stamp":"2024-03-11T21:20:07.529314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:29.676605+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":990,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.039876+0000","last_change":"2024-03-11T21:19:34.216253+0000","last_active":"2024-03-11T21:20:19.039876+0000","last_peered":"2024-03-11T21:20:19.039876+0000","last_clean":"2024-03-11T21:20:19.039876+0000","last_became_active":"2024-03-11T21:14:50.446747+0000","last_became_peered":"2024-03-11T21:14:50.446747+0000","last_unstale":"2024-03-11T21:20:19.039876+0000","last_undegraded":"2024-03-11T21:20:19.039876+0000","last_fullsized":"2024-03-11T21:20:19.039876+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:19:34.216030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:14.081581+0000","last_clean_scrub_stamp":"2024-03-11T21:19:34.216030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:48.045501+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"358'39","reported_seq":1103,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.070086+0000","last_change":"2024-03-11T21:20:04.603226+0000","last_active":"2024-03-11T21:20:19.070086+0000","last_peered":"2024-03-11T21:20:19.070086+0000","last_clean":"2024-03-11T21:20:19.070086+0000","last_became_active":"2024-03-11T21:00:48.462943+0000","last_became_peered":"2024-03-11T21:00:48.462943+0000","last_unstale":"2024-03-11T21:20:19.070086+0000","last_undegraded":"2024-03-11T21:20:19.070086+0000","last_fullsized":"2024-03-11T21:20:19.070086+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'39","last_scrub_stamp":"2024-03-11T21:20:04.602978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:20:04.602978+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:07.921975+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":190,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:23.597768+0000","last_change":"2024-03-11T21:20:23.597814+0000","last_active":"2024-03-11T21:20:23.597768+0000","last_peered":"2024-03-11T21:20:23.597768+0000","last_clean":"2024-03-11T21:20:23.597768+0000","last_became_active":"2024-03-11T21:14:50.430442+0000","last_became_peered":"2024-03-11T21:14:50.430442+0000","last_unstale":"2024-03-11T21:20:23.597768+0000","last_undegraded":"2024-03-11T21:20:23.597768+0000","last_fullsized":"2024-03-11T21:20:23.597768+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:20:23.597614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:20:23.597614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:37.477258+0000","scrub_duration":83,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":601,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.265442+0000","last_change":"2024-03-11T21:19:56.412872+0000","last_active":"2024-03-11T21:20:19.265442+0000","last_peered":"2024-03-11T21:20:19.265442+0000","last_clean":"2024-03-11T21:20:19.265442+0000","last_became_active":"2024-03-11T21:18:53.654504+0000","last_became_peered":"2024-03-11T21:18:53.654504+0000","last_unstale":"2024-03-11T21:20:19.265442+0000","last_undegraded":"2024-03-11T21:20:19.265442+0000","last_fullsized":"2024-03-11T21:20:19.265442+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:19:56.412634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:19:56.412634+0000","last_clean_scrub_stamp":"2024-03-11T21:19:56.412634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:17.957822+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":986,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.039062+0000","last_change":"2024-03-11T21:19:29.250554+0000","last_active":"2024-03-11T21:20:19.039062+0000","last_peered":"2024-03-11T21:20:19.039062+0000","last_clean":"2024-03-11T21:20:19.039062+0000","last_became_active":"2024-03-11T20:50:28.068978+0000","last_became_peered":"2024-03-11T20:50:28.068978+0000","last_unstale":"2024-03-11T21:20:19.039062+0000","last_undegraded":"2024-03-11T21:20:19.039062+0000","last_fullsized":"2024-03-11T21:20:19.039062+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:19:29.250337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:54:07.155130+0000","last_clean_scrub_stamp":"2024-03-11T21:19:29.250337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:43.643589+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"358'29","reported_seq":1032,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.069888+0000","last_change":"2024-03-11T21:19:56.626174+0000","last_active":"2024-03-11T21:20:19.069888+0000","last_peered":"2024-03-11T21:20:19.069888+0000","last_clean":"2024-03-11T21:20:19.069888+0000","last_became_active":"2024-03-11T21:00:27.952093+0000","last_became_peered":"2024-03-11T21:00:27.952093+0000","last_unstale":"2024-03-11T21:20:19.069888+0000","last_undegraded":"2024-03-11T21:20:19.069888+0000","last_fullsized":"2024-03-11T21:20:19.069888+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'29","last_scrub_stamp":"2024-03-11T21:19:56.625921+0000","last_deep_scrub":"110'17","last_deep_scrub_stamp":"2024-03-11T20:53:04.802488+0000","last_clean_scrub_stamp":"2024-03-11T21:19:56.625921+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:12.539568+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":932,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.052789+0000","last_change":"2024-03-11T21:18:52.792441+0000","last_active":"2024-03-11T21:20:19.052789+0000","last_peered":"2024-03-11T21:20:19.052789+0000","last_clean":"2024-03-11T21:20:19.052789+0000","last_became_active":"2024-03-11T20:50:34.334558+0000","last_became_peered":"2024-03-11T20:50:34.334558+0000","last_unstale":"2024-03-11T21:20:19.052789+0000","last_undegraded":"2024-03-11T21:20:19.052789+0000","last_fullsized":"2024-03-11T21:20:19.052789+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:52.792323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:52.792323+0000","last_clean_scrub_stamp":"2024-03-11T21:18:52.792323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":235,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":324,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.051154+0000","last_change":"2024-03-11T21:20:13.552271+0000","last_active":"2024-03-11T21:20:19.051154+0000","last_peered":"2024-03-11T21:20:19.051154+0000","last_clean":"2024-03-11T21:20:19.051154+0000","last_became_active":"2024-03-11T21:10:38.441649+0000","last_became_peered":"2024-03-11T21:10:38.441649+0000","last_unstale":"2024-03-11T21:20:19.051154+0000","last_undegraded":"2024-03-11T21:20:19.051154+0000","last_fullsized":"2024-03-11T21:20:19.051154+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:20:13.552119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:20:13.552119+0000","last_clean_scrub_stamp":"2024-03-11T21:20:13.552119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:18.473429+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"474'2","reported_seq":958,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.050807+0000","last_change":"2024-03-11T21:19:25.509890+0000","last_active":"2024-03-11T21:20:19.050807+0000","last_peered":"2024-03-11T21:20:19.050807+0000","last_clean":"2024-03-11T21:20:19.050807+0000","last_became_active":"2024-03-11T20:50:28.068840+0000","last_became_peered":"2024-03-11T20:50:28.068840+0000","last_unstale":"2024-03-11T21:20:19.050807+0000","last_undegraded":"2024-03-11T21:20:19.050807+0000","last_fullsized":"2024-03-11T21:20:19.050807+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"474'2","last_scrub_stamp":"2024-03-11T21:19:25.509656+0000","last_deep_scrub":"474'2","last_deep_scrub_stamp":"2024-03-11T21:19:25.509656+0000","last_clean_scrub_stamp":"2024-03-11T21:19:25.509656+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:49.390181+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"357'25","reported_seq":550,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.265900+0000","last_change":"2024-03-11T21:19:44.440292+0000","last_active":"2024-03-11T21:20:19.265900+0000","last_peered":"2024-03-11T21:20:19.265900+0000","last_clean":"2024-03-11T21:20:19.265900+0000","last_became_active":"2024-03-11T21:18:53.659269+0000","last_became_peered":"2024-03-11T21:18:53.659269+0000","last_unstale":"2024-03-11T21:20:19.265900+0000","last_undegraded":"2024-03-11T21:20:19.265900+0000","last_fullsized":"2024-03-11T21:20:19.265900+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"357'25","last_scrub_stamp":"2024-03-11T21:19:44.440056+0000","last_deep_scrub":"357'25","last_deep_scrub_stamp":"2024-03-11T21:18:07.923009+0000","last_clean_scrub_stamp":"2024-03-11T21:19:44.440056+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:47.852416+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":469,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.267260+0000","last_change":"2024-03-11T21:19:21.380497+0000","last_active":"2024-03-11T21:20:19.267260+0000","last_peered":"2024-03-11T21:20:19.267260+0000","last_clean":"2024-03-11T21:20:19.267260+0000","last_became_active":"2024-03-11T21:18:53.662773+0000","last_became_peered":"2024-03-11T21:18:53.662773+0000","last_unstale":"2024-03-11T21:20:19.267260+0000","last_undegraded":"2024-03-11T21:20:19.267260+0000","last_fullsized":"2024-03-11T21:20:19.267260+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:19:21.380339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:16:30.676879+0000","last_clean_scrub_stamp":"2024-03-11T21:19:21.380339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:46.086669+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":699,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.076425+0000","last_change":"2024-03-11T21:20:15.913955+0000","last_active":"2024-03-11T21:20:19.076425+0000","last_peered":"2024-03-11T21:20:19.076425+0000","last_clean":"2024-03-11T21:20:19.076425+0000","last_became_active":"2024-03-11T21:17:19.512017+0000","last_became_peered":"2024-03-11T21:17:19.512017+0000","last_unstale":"2024-03-11T21:20:19.076425+0000","last_undegraded":"2024-03-11T21:20:19.076425+0000","last_fullsized":"2024-03-11T21:20:19.076425+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:20:15.913692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:19:08.327562+0000","last_clean_scrub_stamp":"2024-03-11T21:20:15.913692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:27.295349+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":993,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.082029+0000","last_change":"2024-03-11T21:19:32.653757+0000","last_active":"2024-03-11T21:20:19.082029+0000","last_peered":"2024-03-11T21:20:19.082029+0000","last_clean":"2024-03-11T21:20:19.082029+0000","last_became_active":"2024-03-11T21:00:27.956935+0000","last_became_peered":"2024-03-11T21:00:27.956935+0000","last_unstale":"2024-03-11T21:20:19.082029+0000","last_undegraded":"2024-03-11T21:20:19.082029+0000","last_fullsized":"2024-03-11T21:20:19.082029+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:19:32.653534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:17:15.682879+0000","last_clean_scrub_stamp":"2024-03-11T21:19:32.653534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:56.262588+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"494'11","reported_seq":198,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.264324+0000","last_change":"2024-03-11T21:20:00.693286+0000","last_active":"2024-03-11T21:20:19.264324+0000","last_peered":"2024-03-11T21:20:19.264324+0000","last_clean":"2024-03-11T21:20:19.264324+0000","last_became_active":"2024-03-11T21:14:50.430436+0000","last_became_peered":"2024-03-11T21:14:50.430436+0000","last_unstale":"2024-03-11T21:20:19.264324+0000","last_undegraded":"2024-03-11T21:20:19.264324+0000","last_fullsized":"2024-03-11T21:20:19.264324+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"494'11","last_scrub_stamp":"2024-03-11T21:20:00.693172+0000","last_deep_scrub":"494'11","last_deep_scrub_stamp":"2024-03-11T21:20:00.693172+0000","last_clean_scrub_stamp":"2024-03-11T21:20:00.693172+0000","objects_scrubbed":2,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:06.241988+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":16,"num_read_kb":12,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":681,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.046838+0000","last_change":"2024-03-11T21:19:36.241438+0000","last_active":"2024-03-11T21:20:19.046838+0000","last_peered":"2024-03-11T21:20:19.046838+0000","last_clean":"2024-03-11T21:20:19.046838+0000","last_became_active":"2024-03-11T21:00:48.464694+0000","last_became_peered":"2024-03-11T21:00:48.464694+0000","last_unstale":"2024-03-11T21:20:19.046838+0000","last_undegraded":"2024-03-11T21:20:19.046838+0000","last_fullsized":"2024-03-11T21:20:19.046838+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:19:36.241328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:19:36.241328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:01.965733+0000","scrub_duration":167,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1f","version":"212'16","reported_seq":1373,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.558725+0000","last_change":"2024-03-11T21:18:54.536467+0000","last_active":"2024-03-11T21:20:19.558725+0000","last_peered":"2024-03-11T21:20:19.558725+0000","last_clean":"2024-03-11T21:20:19.558725+0000","last_became_active":"2024-03-11T21:00:48.464350+0000","last_became_peered":"2024-03-11T21:00:48.464350+0000","last_unstale":"2024-03-11T21:20:19.558725+0000","last_undegraded":"2024-03-11T21:20:19.558725+0000","last_fullsized":"2024-03-11T21:20:19.558725+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"212'16","last_scrub_stamp":"2024-03-11T21:18:54.536239+0000","last_deep_scrub":"212'16","last_deep_scrub_stamp":"2024-03-11T21:06:47.821221+0000","last_clean_scrub_stamp":"2024-03-11T21:18:54.536239+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"362'2","reported_seq":964,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.050683+0000","last_change":"2024-03-11T21:19:37.461036+0000","last_active":"2024-03-11T21:20:19.050683+0000","last_peered":"2024-03-11T21:20:19.050683+0000","last_clean":"2024-03-11T21:20:19.050683+0000","last_became_active":"2024-03-11T21:14:50.436938+0000","last_became_peered":"2024-03-11T21:14:50.436938+0000","last_unstale":"2024-03-11T21:20:19.050683+0000","last_undegraded":"2024-03-11T21:20:19.050683+0000","last_fullsized":"2024-03-11T21:20:19.050683+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"362'2","last_scrub_stamp":"2024-03-11T21:19:37.460804+0000","last_deep_scrub":"362'2","last_deep_scrub_stamp":"2024-03-11T21:11:03.482685+0000","last_clean_scrub_stamp":"2024-03-11T21:19:37.460804+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:50.526401+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"357'21","reported_seq":1054,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.049683+0000","last_change":"2024-03-11T21:19:56.495228+0000","last_active":"2024-03-11T21:20:19.049683+0000","last_peered":"2024-03-11T21:20:19.049683+0000","last_clean":"2024-03-11T21:20:19.049683+0000","last_became_active":"2024-03-11T21:14:50.437313+0000","last_became_peered":"2024-03-11T21:14:50.437313+0000","last_unstale":"2024-03-11T21:20:19.049683+0000","last_undegraded":"2024-03-11T21:20:19.049683+0000","last_fullsized":"2024-03-11T21:20:19.049683+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"357'21","last_scrub_stamp":"2024-03-11T21:19:56.494977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:19:56.494977+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:24.063441+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"105'2","reported_seq":969,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.039657+0000","last_change":"2024-03-11T21:19:09.270385+0000","last_active":"2024-03-11T21:20:19.039657+0000","last_peered":"2024-03-11T21:20:19.039657+0000","last_clean":"2024-03-11T21:20:19.039657+0000","last_became_active":"2024-03-11T20:50:34.327356+0000","last_became_peered":"2024-03-11T20:50:34.327356+0000","last_unstale":"2024-03-11T21:20:19.039657+0000","last_undegraded":"2024-03-11T21:20:19.039657+0000","last_fullsized":"2024-03-11T21:20:19.039657+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'2","last_scrub_stamp":"2024-03-11T21:19:09.270142+0000","last_deep_scrub":"105'2","last_deep_scrub_stamp":"2024-03-11T21:03:41.971918+0000","last_clean_scrub_stamp":"2024-03-11T21:19:09.270142+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:32.785376+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":182,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.265006+0000","last_change":"2024-03-11T21:19:30.460578+0000","last_active":"2024-03-11T21:20:19.265006+0000","last_peered":"2024-03-11T21:20:19.265006+0000","last_clean":"2024-03-11T21:20:19.265006+0000","last_became_active":"2024-03-11T21:14:50.431785+0000","last_became_peered":"2024-03-11T21:14:50.431785+0000","last_unstale":"2024-03-11T21:20:19.265006+0000","last_undegraded":"2024-03-11T21:20:19.265006+0000","last_fullsized":"2024-03-11T21:20:19.265006+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:19:30.460241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:06:32.968650+0000","last_clean_scrub_stamp":"2024-03-11T21:19:30.460241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:36.454682+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":983,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.068722+0000","last_change":"2024-03-11T21:19:26.680850+0000","last_active":"2024-03-11T21:20:19.068722+0000","last_peered":"2024-03-11T21:20:19.068722+0000","last_clean":"2024-03-11T21:20:19.068722+0000","last_became_active":"2024-03-11T21:00:27.947614+0000","last_became_peered":"2024-03-11T21:00:27.947614+0000","last_unstale":"2024-03-11T21:20:19.068722+0000","last_undegraded":"2024-03-11T21:20:19.068722+0000","last_fullsized":"2024-03-11T21:20:19.068722+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:19:26.680583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:08:27.756036+0000","last_clean_scrub_stamp":"2024-03-11T21:19:26.680583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:50.445805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"358'39","reported_seq":729,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.045287+0000","last_change":"2024-03-11T21:19:53.198728+0000","last_active":"2024-03-11T21:20:19.045287+0000","last_peered":"2024-03-11T21:20:19.045287+0000","last_clean":"2024-03-11T21:20:19.045287+0000","last_became_active":"2024-03-11T21:00:48.457893+0000","last_became_peered":"2024-03-11T21:00:48.457893+0000","last_unstale":"2024-03-11T21:20:19.045287+0000","last_undegraded":"2024-03-11T21:20:19.045287+0000","last_fullsized":"2024-03-11T21:20:19.045287+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'39","last_scrub_stamp":"2024-03-11T21:19:53.198395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:19:53.198395+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:14.967931+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.14","version":"184'130","reported_seq":1057,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.075408+0000","last_change":"2024-03-11T21:19:39.039534+0000","last_active":"2024-03-11T21:20:19.075408+0000","last_peered":"2024-03-11T21:20:19.075408+0000","last_clean":"2024-03-11T21:20:19.075408+0000","last_became_active":"2024-03-11T21:17:19.512515+0000","last_became_peered":"2024-03-11T21:17:19.512515+0000","last_unstale":"2024-03-11T21:20:19.075408+0000","last_undegraded":"2024-03-11T21:20:19.075408+0000","last_fullsized":"2024-03-11T21:20:19.075408+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":444,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":5,"last_scrub":"184'130","last_scrub_stamp":"2024-03-11T21:19:39.039237+0000","last_deep_scrub":"184'130","last_deep_scrub_stamp":"2024-03-11T21:18:26.059235+0000","last_clean_scrub_stamp":"2024-03-11T21:19:39.039237+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:40.158443+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":950,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.051017+0000","last_change":"2024-03-11T21:19:13.801462+0000","last_active":"2024-03-11T21:20:19.051017+0000","last_peered":"2024-03-11T21:20:19.051017+0000","last_clean":"2024-03-11T21:20:19.051017+0000","last_became_active":"2024-03-11T21:14:50.438215+0000","last_became_peered":"2024-03-11T21:14:50.438215+0000","last_unstale":"2024-03-11T21:20:19.051017+0000","last_undegraded":"2024-03-11T21:20:19.051017+0000","last_fullsized":"2024-03-11T21:20:19.051017+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:19:13.801303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:52:11.462480+0000","last_clean_scrub_stamp":"2024-03-11T21:19:13.801303+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:35.890821+0000","scrub_duration":167,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":942,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.051105+0000","last_change":"2024-03-11T21:19:27.502595+0000","last_active":"2024-03-11T21:20:19.051105+0000","last_peered":"2024-03-11T21:20:19.051105+0000","last_clean":"2024-03-11T21:20:19.051105+0000","last_became_active":"2024-03-11T20:50:32.646323+0000","last_became_peered":"2024-03-11T20:50:32.646323+0000","last_unstale":"2024-03-11T21:20:19.051105+0000","last_undegraded":"2024-03-11T21:20:19.051105+0000","last_fullsized":"2024-03-11T21:20:19.051105+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:19:27.502382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:10:02.099653+0000","last_clean_scrub_stamp":"2024-03-11T21:19:27.502382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:48.649120+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":1001,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.083270+0000","last_change":"2024-03-11T21:19:20.564830+0000","last_active":"2024-03-11T21:20:19.083270+0000","last_peered":"2024-03-11T21:20:19.083270+0000","last_clean":"2024-03-11T21:20:19.083270+0000","last_became_active":"2024-03-11T20:50:28.086513+0000","last_became_peered":"2024-03-11T20:50:28.086513+0000","last_unstale":"2024-03-11T21:20:19.083270+0000","last_undegraded":"2024-03-11T21:20:19.083270+0000","last_fullsized":"2024-03-11T21:20:19.083270+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:19:20.564688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:19:20.564688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:34.169330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"357'25","reported_seq":1050,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.069897+0000","last_change":"2024-03-11T21:19:14.667835+0000","last_active":"2024-03-11T21:20:19.069897+0000","last_peered":"2024-03-11T21:20:19.069897+0000","last_clean":"2024-03-11T21:20:19.069897+0000","last_became_active":"2024-03-11T21:00:48.464707+0000","last_became_peered":"2024-03-11T21:00:48.464707+0000","last_unstale":"2024-03-11T21:20:19.069897+0000","last_undegraded":"2024-03-11T21:20:19.069897+0000","last_fullsized":"2024-03-11T21:20:19.069897+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"357'25","last_scrub_stamp":"2024-03-11T21:19:14.667589+0000","last_deep_scrub":"145'13","last_deep_scrub_stamp":"2024-03-11T20:55:53.560287+0000","last_clean_scrub_stamp":"2024-03-11T21:19:14.667589+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:28.971110+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":968,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.038693+0000","last_change":"2024-03-11T21:18:57.346335+0000","last_active":"2024-03-11T21:20:19.038693+0000","last_peered":"2024-03-11T21:20:19.038693+0000","last_clean":"2024-03-11T21:20:19.038693+0000","last_became_active":"2024-03-11T21:00:27.973405+0000","last_became_peered":"2024-03-11T21:00:27.973405+0000","last_unstale":"2024-03-11T21:20:19.038693+0000","last_undegraded":"2024-03-11T21:20:19.038693+0000","last_fullsized":"2024-03-11T21:20:19.038693+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:57.346019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:12:24.069900+0000","last_clean_scrub_stamp":"2024-03-11T21:18:57.346019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":955,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.049736+0000","last_change":"2024-03-11T21:19:42.436445+0000","last_active":"2024-03-11T21:20:19.049736+0000","last_peered":"2024-03-11T21:20:19.049736+0000","last_clean":"2024-03-11T21:20:19.049736+0000","last_became_active":"2024-03-11T21:00:48.472812+0000","last_became_peered":"2024-03-11T21:00:48.472812+0000","last_unstale":"2024-03-11T21:20:19.049736+0000","last_undegraded":"2024-03-11T21:20:19.049736+0000","last_fullsized":"2024-03-11T21:20:19.049736+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:19:42.436153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:19:42.436153+0000","last_clean_scrub_stamp":"2024-03-11T21:19:42.436153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:01.371751+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":990,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:25.655410+0000","last_change":"2024-03-11T21:20:25.655469+0000","last_active":"2024-03-11T21:20:25.655410+0000","last_peered":"2024-03-11T21:20:25.655410+0000","last_clean":"2024-03-11T21:20:25.655410+0000","last_became_active":"2024-03-11T20:50:28.075760+0000","last_became_peered":"2024-03-11T20:50:28.075760+0000","last_unstale":"2024-03-11T21:20:25.655410+0000","last_undegraded":"2024-03-11T21:20:25.655410+0000","last_fullsized":"2024-03-11T21:20:25.655410+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:20:25.655266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:20:25.655266+0000","last_clean_scrub_stamp":"2024-03-11T21:20:25.655266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:51.410555+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"472'21","reported_seq":193,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.265235+0000","last_change":"2024-03-11T21:19:14.461073+0000","last_active":"2024-03-11T21:20:19.265235+0000","last_peered":"2024-03-11T21:20:19.265235+0000","last_clean":"2024-03-11T21:20:19.265235+0000","last_became_active":"2024-03-11T21:14:50.427230+0000","last_became_peered":"2024-03-11T21:14:50.427230+0000","last_unstale":"2024-03-11T21:20:19.265235+0000","last_undegraded":"2024-03-11T21:20:19.265235+0000","last_fullsized":"2024-03-11T21:20:19.265235+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"472'21","last_scrub_stamp":"2024-03-11T21:19:14.460869+0000","last_deep_scrub":"472'21","last_deep_scrub_stamp":"2024-03-11T21:17:46.445788+0000","last_clean_scrub_stamp":"2024-03-11T21:19:14.460869+0000","objects_scrubbed":3,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:44.390729+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":24,"num_read_kb":21,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.11","version":"186'2","reported_seq":960,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.049954+0000","last_change":"2024-03-11T21:19:18.567773+0000","last_active":"2024-03-11T21:20:19.049954+0000","last_peered":"2024-03-11T21:20:19.049954+0000","last_clean":"2024-03-11T21:20:19.049954+0000","last_became_active":"2024-03-11T21:14:50.436281+0000","last_became_peered":"2024-03-11T21:14:50.436281+0000","last_unstale":"2024-03-11T21:20:19.049954+0000","last_undegraded":"2024-03-11T21:20:19.049954+0000","last_fullsized":"2024-03-11T21:20:19.049954+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"186'2","last_scrub_stamp":"2024-03-11T21:19:18.567510+0000","last_deep_scrub":"186'2","last_deep_scrub_stamp":"2024-03-11T20:59:04.534850+0000","last_clean_scrub_stamp":"2024-03-11T21:19:18.567510+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:43.662594+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":480,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.265932+0000","last_change":"2024-03-11T21:19:48.465834+0000","last_active":"2024-03-11T21:20:19.265932+0000","last_peered":"2024-03-11T21:20:19.265932+0000","last_clean":"2024-03-11T21:20:19.265932+0000","last_became_active":"2024-03-11T21:18:53.659060+0000","last_became_peered":"2024-03-11T21:18:53.659060+0000","last_unstale":"2024-03-11T21:20:19.265932+0000","last_undegraded":"2024-03-11T21:20:19.265932+0000","last_fullsized":"2024-03-11T21:20:19.265932+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:19:48.465615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:19:48.465615+0000","last_clean_scrub_stamp":"2024-03-11T21:19:48.465615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:06.059928+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":693,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.046562+0000","last_change":"2024-03-11T21:19:50.117199+0000","last_active":"2024-03-11T21:20:19.046562+0000","last_peered":"2024-03-11T21:20:19.046562+0000","last_clean":"2024-03-11T21:20:19.046562+0000","last_became_active":"2024-03-11T21:14:50.419556+0000","last_became_peered":"2024-03-11T21:14:50.419556+0000","last_unstale":"2024-03-11T21:20:19.046562+0000","last_undegraded":"2024-03-11T21:20:19.046562+0000","last_fullsized":"2024-03-11T21:20:19.046562+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:19:50.116922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:19:50.116922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:57.501009+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"358'35","reported_seq":494,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.266597+0000","last_change":"2024-03-11T21:19:34.373507+0000","last_active":"2024-03-11T21:20:19.266597+0000","last_peered":"2024-03-11T21:20:19.266597+0000","last_clean":"2024-03-11T21:20:19.266597+0000","last_became_active":"2024-03-11T21:18:53.660393+0000","last_became_peered":"2024-03-11T21:18:53.660393+0000","last_unstale":"2024-03-11T21:20:19.266597+0000","last_undegraded":"2024-03-11T21:20:19.266597+0000","last_fullsized":"2024-03-11T21:20:19.266597+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'35","last_scrub_stamp":"2024-03-11T21:19:34.373224+0000","last_deep_scrub":"161'15","last_deep_scrub_stamp":"2024-03-11T20:57:15.125548+0000","last_clean_scrub_stamp":"2024-03-11T21:19:34.373224+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:40.415748+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":182,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.263895+0000","last_change":"2024-03-11T21:19:26.482060+0000","last_active":"2024-03-11T21:20:19.263895+0000","last_peered":"2024-03-11T21:20:19.263895+0000","last_clean":"2024-03-11T21:20:19.263895+0000","last_became_active":"2024-03-11T21:14:50.420063+0000","last_became_peered":"2024-03-11T21:14:50.420063+0000","last_unstale":"2024-03-11T21:20:19.263895+0000","last_undegraded":"2024-03-11T21:20:19.263895+0000","last_fullsized":"2024-03-11T21:20:19.263895+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:19:26.481947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:19:26.481947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:32.923430+0000","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":681,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.045735+0000","last_change":"2024-03-11T21:19:35.032865+0000","last_active":"2024-03-11T21:20:19.045735+0000","last_peered":"2024-03-11T21:20:19.045735+0000","last_clean":"2024-03-11T21:20:19.045735+0000","last_became_active":"2024-03-11T21:00:48.459043+0000","last_became_peered":"2024-03-11T21:00:48.459043+0000","last_unstale":"2024-03-11T21:20:19.045735+0000","last_undegraded":"2024-03-11T21:20:19.045735+0000","last_fullsized":"2024-03-11T21:20:19.045735+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:19:35.032635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:16.860126+0000","last_clean_scrub_stamp":"2024-03-11T21:19:35.032635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:40.973406+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":1001,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.082174+0000","last_change":"2024-03-11T21:19:53.765481+0000","last_active":"2024-03-11T21:20:19.082174+0000","last_peered":"2024-03-11T21:20:19.082174+0000","last_clean":"2024-03-11T21:20:19.082174+0000","last_became_active":"2024-03-11T20:50:28.082737+0000","last_became_peered":"2024-03-11T20:50:28.082737+0000","last_unstale":"2024-03-11T21:20:19.082174+0000","last_undegraded":"2024-03-11T21:20:19.082174+0000","last_fullsized":"2024-03-11T21:20:19.082174+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:19:53.765222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:28.404997+0000","last_clean_scrub_stamp":"2024-03-11T21:19:53.765222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:18.382549+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"358'21","reported_seq":1018,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:23.452717+0000","last_change":"2024-03-11T21:20:23.452877+0000","last_active":"2024-03-11T21:20:23.452717+0000","last_peered":"2024-03-11T21:20:23.452717+0000","last_clean":"2024-03-11T21:20:23.452717+0000","last_became_active":"2024-03-11T21:00:27.967624+0000","last_became_peered":"2024-03-11T21:00:27.967624+0000","last_unstale":"2024-03-11T21:20:23.452717+0000","last_undegraded":"2024-03-11T21:20:23.452717+0000","last_fullsized":"2024-03-11T21:20:23.452717+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'21","last_scrub_stamp":"2024-03-11T21:20:23.452480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:20:23.452480+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:27.384101+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.10","version":"184'130","reported_seq":1237,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:23.656997+0000","last_change":"2024-03-11T21:20:23.657061+0000","last_active":"2024-03-11T21:20:23.656997+0000","last_peered":"2024-03-11T21:20:23.656997+0000","last_clean":"2024-03-11T21:20:23.656997+0000","last_became_active":"2024-03-11T21:16:24.829565+0000","last_became_peered":"2024-03-11T21:16:24.829565+0000","last_unstale":"2024-03-11T21:20:23.656997+0000","last_undegraded":"2024-03-11T21:20:23.656997+0000","last_fullsized":"2024-03-11T21:20:23.656997+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":444,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":5,"last_scrub":"184'130","last_scrub_stamp":"2024-03-11T21:20:23.656822+0000","last_deep_scrub":"184'130","last_deep_scrub_stamp":"2024-03-11T21:14:40.126341+0000","last_clean_scrub_stamp":"2024-03-11T21:20:23.656822+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:33.498464+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":430,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.267650+0000","last_change":"2024-03-11T21:19:50.693660+0000","last_active":"2024-03-11T21:20:19.267650+0000","last_peered":"2024-03-11T21:20:19.267650+0000","last_clean":"2024-03-11T21:20:19.267650+0000","last_became_active":"2024-03-11T21:18:53.664053+0000","last_became_peered":"2024-03-11T21:18:53.664053+0000","last_unstale":"2024-03-11T21:20:19.267650+0000","last_undegraded":"2024-03-11T21:20:19.267650+0000","last_fullsized":"2024-03-11T21:20:19.267650+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:19:50.693383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:57:16.032302+0000","last_clean_scrub_stamp":"2024-03-11T21:19:50.693383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:04.453078+0000","scrub_duration":187,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.15","version":"112'12","reported_seq":1345,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.558543+0000","last_change":"2024-03-11T21:20:10.680986+0000","last_active":"2024-03-11T21:20:19.558543+0000","last_peered":"2024-03-11T21:20:19.558543+0000","last_clean":"2024-03-11T21:20:19.558543+0000","last_became_active":"2024-03-11T20:50:32.650064+0000","last_became_peered":"2024-03-11T20:50:32.650064+0000","last_unstale":"2024-03-11T21:20:19.558543+0000","last_undegraded":"2024-03-11T21:20:19.558543+0000","last_fullsized":"2024-03-11T21:20:19.558543+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'12","last_scrub_stamp":"2024-03-11T21:20:10.680727+0000","last_deep_scrub":"112'12","last_deep_scrub_stamp":"2024-03-11T21:18:54.451798+0000","last_clean_scrub_stamp":"2024-03-11T21:20:10.680727+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:35.922921+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":182,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.264150+0000","last_change":"2024-03-11T21:18:59.323303+0000","last_active":"2024-03-11T21:20:19.264150+0000","last_peered":"2024-03-11T21:20:19.264150+0000","last_clean":"2024-03-11T21:20:19.264150+0000","last_became_active":"2024-03-11T21:14:50.434124+0000","last_became_peered":"2024-03-11T21:14:50.434124+0000","last_unstale":"2024-03-11T21:20:19.264150+0000","last_undegraded":"2024-03-11T21:20:19.264150+0000","last_fullsized":"2024-03-11T21:20:19.264150+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:59.323091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:06:52.862971+0000","last_clean_scrub_stamp":"2024-03-11T21:18:59.323091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"357'25","reported_seq":192,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.264102+0000","last_change":"2024-03-11T21:19:02.351926+0000","last_active":"2024-03-11T21:20:19.264102+0000","last_peered":"2024-03-11T21:20:19.264102+0000","last_clean":"2024-03-11T21:20:19.264102+0000","last_became_active":"2024-03-11T21:14:50.434105+0000","last_became_peered":"2024-03-11T21:14:50.434105+0000","last_unstale":"2024-03-11T21:20:19.264102+0000","last_undegraded":"2024-03-11T21:20:19.264102+0000","last_fullsized":"2024-03-11T21:20:19.264102+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"357'25","last_scrub_stamp":"2024-03-11T21:19:02.351660+0000","last_deep_scrub":"119'13","last_deep_scrub_stamp":"2024-03-11T20:54:15.443472+0000","last_clean_scrub_stamp":"2024-03-11T21:19:02.351660+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:29.441477+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.11","version":"184'130","reported_seq":805,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.263335+0000","last_change":"2024-03-11T21:19:06.385686+0000","last_active":"2024-03-11T21:20:19.263335+0000","last_peered":"2024-03-11T21:20:19.263335+0000","last_clean":"2024-03-11T21:20:19.263335+0000","last_became_active":"2024-03-11T21:16:24.825311+0000","last_became_peered":"2024-03-11T21:16:24.825311+0000","last_unstale":"2024-03-11T21:20:19.263335+0000","last_undegraded":"2024-03-11T21:20:19.263335+0000","last_fullsized":"2024-03-11T21:20:19.263335+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":444,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":5,"last_scrub":"184'130","last_scrub_stamp":"2024-03-11T21:19:06.385464+0000","last_deep_scrub":"184'130","last_deep_scrub_stamp":"2024-03-11T21:17:44.379271+0000","last_clean_scrub_stamp":"2024-03-11T21:19:06.385464+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:33.430704+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":182,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.265056+0000","last_change":"2024-03-11T21:19:23.504455+0000","last_active":"2024-03-11T21:20:19.265056+0000","last_peered":"2024-03-11T21:20:19.265056+0000","last_clean":"2024-03-11T21:20:19.265056+0000","last_became_active":"2024-03-11T21:14:50.428898+0000","last_became_peered":"2024-03-11T21:14:50.428898+0000","last_unstale":"2024-03-11T21:20:19.265056+0000","last_undegraded":"2024-03-11T21:20:19.265056+0000","last_fullsized":"2024-03-11T21:20:19.265056+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:19:23.504278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:17:49.501931+0000","last_clean_scrub_stamp":"2024-03-11T21:19:23.504278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:40.240239+0000","scrub_duration":87,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"212'17","reported_seq":1344,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.556386+0000","last_change":"2024-03-11T21:20:14.597474+0000","last_active":"2024-03-11T21:20:19.556386+0000","last_peered":"2024-03-11T21:20:19.556386+0000","last_clean":"2024-03-11T21:20:19.556386+0000","last_became_active":"2024-03-11T21:00:48.467116+0000","last_became_peered":"2024-03-11T21:00:48.467116+0000","last_unstale":"2024-03-11T21:20:19.556386+0000","last_undegraded":"2024-03-11T21:20:19.556386+0000","last_fullsized":"2024-03-11T21:20:19.556386+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"212'17","last_scrub_stamp":"2024-03-11T21:20:14.597360+0000","last_deep_scrub":"110'13","last_deep_scrub_stamp":"2024-03-11T20:53:00.434532+0000","last_clean_scrub_stamp":"2024-03-11T21:20:14.597360+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:32.296804+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":990,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.069082+0000","last_change":"2024-03-11T21:19:36.659874+0000","last_active":"2024-03-11T21:20:19.069082+0000","last_peered":"2024-03-11T21:20:19.069082+0000","last_clean":"2024-03-11T21:20:19.069082+0000","last_became_active":"2024-03-11T20:50:28.070325+0000","last_became_peered":"2024-03-11T20:50:28.070325+0000","last_unstale":"2024-03-11T21:20:19.069082+0000","last_undegraded":"2024-03-11T21:20:19.069082+0000","last_fullsized":"2024-03-11T21:20:19.069082+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:19:36.659656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:12:39.708033+0000","last_clean_scrub_stamp":"2024-03-11T21:19:36.659656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:49.794595+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"358'32","reported_seq":1032,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.441902+0000","last_change":"2024-03-11T21:20:19.441983+0000","last_active":"2024-03-11T21:20:19.441902+0000","last_peered":"2024-03-11T21:20:19.441902+0000","last_clean":"2024-03-11T21:20:19.441902+0000","last_became_active":"2024-03-11T21:00:27.967078+0000","last_became_peered":"2024-03-11T21:00:27.967078+0000","last_unstale":"2024-03-11T21:20:19.441902+0000","last_undegraded":"2024-03-11T21:20:19.441902+0000","last_fullsized":"2024-03-11T21:20:19.441902+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'32","last_scrub_stamp":"2024-03-11T21:20:19.441743+0000","last_deep_scrub":"358'32","last_deep_scrub_stamp":"2024-03-11T21:20:19.441743+0000","last_clean_scrub_stamp":"2024-03-11T21:20:19.441743+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:49.341484+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.12","version":"184'130","reported_seq":1235,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.068509+0000","last_change":"2024-03-11T21:19:18.719229+0000","last_active":"2024-03-11T21:20:19.068509+0000","last_peered":"2024-03-11T21:20:19.068509+0000","last_clean":"2024-03-11T21:20:19.068509+0000","last_became_active":"2024-03-11T21:16:24.831501+0000","last_became_peered":"2024-03-11T21:16:24.831501+0000","last_unstale":"2024-03-11T21:20:19.068509+0000","last_undegraded":"2024-03-11T21:20:19.068509+0000","last_fullsized":"2024-03-11T21:20:19.068509+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":444,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":5,"last_scrub":"184'130","last_scrub_stamp":"2024-03-11T21:19:18.719015+0000","last_deep_scrub":"184'130","last_deep_scrub_stamp":"2024-03-11T20:58:56.415267+0000","last_clean_scrub_stamp":"2024-03-11T21:19:18.719015+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:39.551998+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":960,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.039417+0000","last_change":"2024-03-11T21:18:59.409588+0000","last_active":"2024-03-11T21:20:19.039417+0000","last_peered":"2024-03-11T21:20:19.039417+0000","last_clean":"2024-03-11T21:20:19.039417+0000","last_became_active":"2024-03-11T21:00:27.966997+0000","last_became_peered":"2024-03-11T21:00:27.966997+0000","last_unstale":"2024-03-11T21:20:19.039417+0000","last_undegraded":"2024-03-11T21:20:19.039417+0000","last_fullsized":"2024-03-11T21:20:19.039417+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:59.409404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:17:29.228431+0000","last_clean_scrub_stamp":"2024-03-11T21:18:59.409404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:26.350359+0000","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":963,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:22.585435+0000","last_change":"2024-03-11T21:20:22.585490+0000","last_active":"2024-03-11T21:20:22.585435+0000","last_peered":"2024-03-11T21:20:22.585435+0000","last_clean":"2024-03-11T21:20:22.585435+0000","last_became_active":"2024-03-11T21:00:48.468386+0000","last_became_peered":"2024-03-11T21:00:48.468386+0000","last_unstale":"2024-03-11T21:20:22.585435+0000","last_undegraded":"2024-03-11T21:20:22.585435+0000","last_fullsized":"2024-03-11T21:20:22.585435+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:20:22.585283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:20:22.585283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:46.208993+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"166'2","reported_seq":1020,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.081283+0000","last_change":"2024-03-11T21:18:52.491258+0000","last_active":"2024-03-11T21:20:19.081283+0000","last_peered":"2024-03-11T21:20:19.081283+0000","last_clean":"2024-03-11T21:20:19.081283+0000","last_became_active":"2024-03-11T21:00:48.466071+0000","last_became_peered":"2024-03-11T21:00:48.466071+0000","last_unstale":"2024-03-11T21:20:19.081283+0000","last_undegraded":"2024-03-11T21:20:19.081283+0000","last_fullsized":"2024-03-11T21:20:19.081283+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'2","last_scrub_stamp":"2024-03-11T21:18:52.490954+0000","last_deep_scrub":"166'2","last_deep_scrub_stamp":"2024-03-11T20:57:39.868776+0000","last_clean_scrub_stamp":"2024-03-11T21:18:52.490954+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1822,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"364'14","reported_seq":1025,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.081039+0000","last_change":"2024-03-11T21:19:26.550324+0000","last_active":"2024-03-11T21:20:19.081039+0000","last_peered":"2024-03-11T21:20:19.081039+0000","last_clean":"2024-03-11T21:20:19.081039+0000","last_became_active":"2024-03-11T21:14:50.420602+0000","last_became_peered":"2024-03-11T21:14:50.420602+0000","last_unstale":"2024-03-11T21:20:19.081039+0000","last_undegraded":"2024-03-11T21:20:19.081039+0000","last_fullsized":"2024-03-11T21:20:19.081039+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"364'14","last_scrub_stamp":"2024-03-11T21:19:26.550103+0000","last_deep_scrub":"364'14","last_deep_scrub_stamp":"2024-03-11T21:11:15.819245+0000","last_clean_scrub_stamp":"2024-03-11T21:19:26.550103+0000","objects_scrubbed":2,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:41.517513+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":16,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.13","version":"184'130","reported_seq":854,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:25.047669+0000","last_change":"2024-03-11T21:20:25.047733+0000","last_active":"2024-03-11T21:20:25.047669+0000","last_peered":"2024-03-11T21:20:25.047669+0000","last_clean":"2024-03-11T21:20:25.047669+0000","last_became_active":"2024-03-11T21:16:24.840090+0000","last_became_peered":"2024-03-11T21:16:24.840090+0000","last_unstale":"2024-03-11T21:20:25.047669+0000","last_undegraded":"2024-03-11T21:20:25.047669+0000","last_fullsized":"2024-03-11T21:20:25.047669+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":444,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":5,"last_scrub":"184'130","last_scrub_stamp":"2024-03-11T21:20:25.047504+0000","last_deep_scrub":"184'130","last_deep_scrub_stamp":"2024-03-11T20:58:56.415267+0000","last_clean_scrub_stamp":"2024-03-11T21:20:25.047504+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:45.794232+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":667,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.076466+0000","last_change":"2024-03-11T21:19:31.021948+0000","last_active":"2024-03-11T21:20:19.076466+0000","last_peered":"2024-03-11T21:20:19.076466+0000","last_clean":"2024-03-11T21:20:19.076466+0000","last_became_active":"2024-03-11T21:17:19.512977+0000","last_became_peered":"2024-03-11T21:17:19.512977+0000","last_unstale":"2024-03-11T21:20:19.076466+0000","last_undegraded":"2024-03-11T21:20:19.076466+0000","last_fullsized":"2024-03-11T21:20:19.076466+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:19:31.021863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:19:31.021863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:49.852332+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":190,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.266227+0000","last_change":"2024-03-11T21:19:52.695880+0000","last_active":"2024-03-11T21:20:19.266227+0000","last_peered":"2024-03-11T21:20:19.266227+0000","last_clean":"2024-03-11T21:20:19.266227+0000","last_became_active":"2024-03-11T21:14:50.708563+0000","last_became_peered":"2024-03-11T21:14:50.708563+0000","last_unstale":"2024-03-11T21:20:19.266227+0000","last_undegraded":"2024-03-11T21:20:19.266227+0000","last_fullsized":"2024-03-11T21:20:19.266227+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:19:52.695673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:54:09.537972+0000","last_clean_scrub_stamp":"2024-03-11T21:19:52.695673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:02.884010+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":680,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.074284+0000","last_change":"2024-03-11T21:19:10.359343+0000","last_active":"2024-03-11T21:20:19.074284+0000","last_peered":"2024-03-11T21:20:19.074284+0000","last_clean":"2024-03-11T21:20:19.074284+0000","last_became_active":"2024-03-11T21:17:19.980445+0000","last_became_peered":"2024-03-11T21:17:19.980445+0000","last_unstale":"2024-03-11T21:20:19.074284+0000","last_undegraded":"2024-03-11T21:20:19.074284+0000","last_fullsized":"2024-03-11T21:20:19.074284+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:19:10.359123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:19:10.359123+0000","last_clean_scrub_stamp":"2024-03-11T21:19:10.359123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:29.257375+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"358'32","reported_seq":1024,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.037688+0000","last_change":"2024-03-11T21:19:47.256470+0000","last_active":"2024-03-11T21:20:19.037688+0000","last_peered":"2024-03-11T21:20:19.037688+0000","last_clean":"2024-03-11T21:20:19.037688+0000","last_became_active":"2024-03-11T21:00:27.970878+0000","last_became_peered":"2024-03-11T21:00:27.970878+0000","last_unstale":"2024-03-11T21:20:19.037688+0000","last_undegraded":"2024-03-11T21:20:19.037688+0000","last_fullsized":"2024-03-11T21:20:19.037688+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'32","last_scrub_stamp":"2024-03-11T21:19:47.256207+0000","last_deep_scrub":"358'32","last_deep_scrub_stamp":"2024-03-11T21:19:47.256207+0000","last_clean_scrub_stamp":"2024-03-11T21:19:47.256207+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:11.721751+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"184'130","reported_seq":852,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.051157+0000","last_change":"2024-03-11T21:19:46.491231+0000","last_active":"2024-03-11T21:20:19.051157+0000","last_peered":"2024-03-11T21:20:19.051157+0000","last_clean":"2024-03-11T21:20:19.051157+0000","last_became_active":"2024-03-11T21:16:24.827641+0000","last_became_peered":"2024-03-11T21:16:24.827641+0000","last_unstale":"2024-03-11T21:20:19.051157+0000","last_undegraded":"2024-03-11T21:20:19.051157+0000","last_fullsized":"2024-03-11T21:20:19.051157+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":444,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":4,"last_scrub":"184'130","last_scrub_stamp":"2024-03-11T21:19:46.491123+0000","last_deep_scrub":"184'130","last_deep_scrub_stamp":"2024-03-11T20:58:56.415267+0000","last_clean_scrub_stamp":"2024-03-11T21:19:46.491123+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:58.642722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":960,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.051069+0000","last_change":"2024-03-11T21:19:02.797661+0000","last_active":"2024-03-11T21:20:19.051069+0000","last_peered":"2024-03-11T21:20:19.051069+0000","last_clean":"2024-03-11T21:20:19.051069+0000","last_became_active":"2024-03-11T21:00:48.466853+0000","last_became_peered":"2024-03-11T21:00:48.466853+0000","last_unstale":"2024-03-11T21:20:19.051069+0000","last_undegraded":"2024-03-11T21:20:19.051069+0000","last_fullsized":"2024-03-11T21:20:19.051069+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:19:02.797564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:52:53.450885+0000","last_clean_scrub_stamp":"2024-03-11T21:19:02.797564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:23.095214+0000","scrub_duration":175,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"77'2","reported_seq":477,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.267772+0000","last_change":"2024-03-11T21:20:17.528399+0000","last_active":"2024-03-11T21:20:19.267772+0000","last_peered":"2024-03-11T21:20:19.267772+0000","last_clean":"2024-03-11T21:20:19.267772+0000","last_became_active":"2024-03-11T21:18:53.659656+0000","last_became_peered":"2024-03-11T21:18:53.659656+0000","last_unstale":"2024-03-11T21:20:19.267772+0000","last_undegraded":"2024-03-11T21:20:19.267772+0000","last_fullsized":"2024-03-11T21:20:19.267772+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'2","last_scrub_stamp":"2024-03-11T21:20:17.528107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:20:17.528107+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:18.203802+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"358'32","reported_seq":1085,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.039942+0000","last_change":"2024-03-11T21:19:26.300612+0000","last_active":"2024-03-11T21:20:19.039942+0000","last_peered":"2024-03-11T21:20:19.039942+0000","last_clean":"2024-03-11T21:20:19.039942+0000","last_became_active":"2024-03-11T21:00:48.465254+0000","last_became_peered":"2024-03-11T21:00:48.465254+0000","last_unstale":"2024-03-11T21:20:19.039942+0000","last_undegraded":"2024-03-11T21:20:19.039942+0000","last_fullsized":"2024-03-11T21:20:19.039942+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'32","last_scrub_stamp":"2024-03-11T21:19:26.300201+0000","last_deep_scrub":"100'16","last_deep_scrub_stamp":"2024-03-11T20:56:08.036331+0000","last_clean_scrub_stamp":"2024-03-11T21:19:26.300201+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:38.534730+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"486'19","reported_seq":665,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.559478+0000","last_change":"2024-03-11T21:19:11.220225+0000","last_active":"2024-03-11T21:20:19.559478+0000","last_peered":"2024-03-11T21:20:19.559478+0000","last_clean":"2024-03-11T21:20:19.559478+0000","last_became_active":"2024-03-11T21:18:53.658334+0000","last_became_peered":"2024-03-11T21:18:53.658334+0000","last_unstale":"2024-03-11T21:20:19.559478+0000","last_undegraded":"2024-03-11T21:20:19.559478+0000","last_fullsized":"2024-03-11T21:20:19.559478+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"486'19","last_scrub_stamp":"2024-03-11T21:19:11.219995+0000","last_deep_scrub":"212'17","last_deep_scrub_stamp":"2024-03-11T21:03:38.258850+0000","last_clean_scrub_stamp":"2024-03-11T21:19:11.219995+0000","objects_scrubbed":1,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:31.957980+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"184'130","reported_seq":849,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.082971+0000","last_change":"2024-03-11T21:19:17.445933+0000","last_active":"2024-03-11T21:20:19.082971+0000","last_peered":"2024-03-11T21:20:19.082971+0000","last_clean":"2024-03-11T21:20:19.082971+0000","last_became_active":"2024-03-11T21:16:19.734553+0000","last_became_peered":"2024-03-11T21:16:19.734553+0000","last_unstale":"2024-03-11T21:20:19.082971+0000","last_undegraded":"2024-03-11T21:20:19.082971+0000","last_fullsized":"2024-03-11T21:20:19.082971+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":3,"last_scrub":"184'130","last_scrub_stamp":"2024-03-11T21:19:17.445630+0000","last_deep_scrub":"184'130","last_deep_scrub_stamp":"2024-03-11T20:58:56.415267+0000","last_clean_scrub_stamp":"2024-03-11T21:19:17.445630+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:40.521827+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":182,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.264035+0000","last_change":"2024-03-11T21:19:31.577859+0000","last_active":"2024-03-11T21:20:19.264035+0000","last_peered":"2024-03-11T21:20:19.264035+0000","last_clean":"2024-03-11T21:20:19.264035+0000","last_became_active":"2024-03-11T21:14:50.683519+0000","last_became_peered":"2024-03-11T21:14:50.683519+0000","last_unstale":"2024-03-11T21:20:19.264035+0000","last_undegraded":"2024-03-11T21:20:19.264035+0000","last_fullsized":"2024-03-11T21:20:19.264035+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:19:31.577700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:57:21.568400+0000","last_clean_scrub_stamp":"2024-03-11T21:19:31.577700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:34.661958+0000","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":694,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.046660+0000","last_change":"2024-03-11T21:19:38.141168+0000","last_active":"2024-03-11T21:20:19.046660+0000","last_peered":"2024-03-11T21:20:19.046660+0000","last_clean":"2024-03-11T21:20:19.046660+0000","last_became_active":"2024-03-11T21:14:50.421319+0000","last_became_peered":"2024-03-11T21:14:50.421319+0000","last_unstale":"2024-03-11T21:20:19.046660+0000","last_undegraded":"2024-03-11T21:20:19.046660+0000","last_fullsized":"2024-03-11T21:20:19.046660+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:19:38.141032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:20.096874+0000","last_clean_scrub_stamp":"2024-03-11T21:19:38.141032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:48.260710+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"358'40","reported_seq":1398,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.558616+0000","last_change":"2024-03-11T21:19:22.734864+0000","last_active":"2024-03-11T21:20:19.558616+0000","last_peered":"2024-03-11T21:20:19.558616+0000","last_clean":"2024-03-11T21:20:19.558616+0000","last_became_active":"2024-03-11T20:50:30.094977+0000","last_became_peered":"2024-03-11T20:50:30.094977+0000","last_unstale":"2024-03-11T21:20:19.558616+0000","last_undegraded":"2024-03-11T21:20:19.558616+0000","last_fullsized":"2024-03-11T21:20:19.558616+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'40","last_scrub_stamp":"2024-03-11T21:19:22.734616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:19:22.734616+0000","objects_scrubbed":8,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:48.748907+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":61,"num_read_kb":51,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":682,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.045675+0000","last_change":"2024-03-11T21:19:45.107526+0000","last_active":"2024-03-11T21:20:19.045675+0000","last_peered":"2024-03-11T21:20:19.045675+0000","last_clean":"2024-03-11T21:20:19.045675+0000","last_became_active":"2024-03-11T21:00:48.457901+0000","last_became_peered":"2024-03-11T21:00:48.457901+0000","last_unstale":"2024-03-11T21:20:19.045675+0000","last_undegraded":"2024-03-11T21:20:19.045675+0000","last_fullsized":"2024-03-11T21:20:19.045675+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:19:45.107196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:16:59.052089+0000","last_clean_scrub_stamp":"2024-03-11T21:19:45.107196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:45.753171+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"184'130","reported_seq":873,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.068302+0000","last_change":"2024-03-11T21:20:13.685146+0000","last_active":"2024-03-11T21:20:19.068302+0000","last_peered":"2024-03-11T21:20:19.068302+0000","last_clean":"2024-03-11T21:20:19.068302+0000","last_became_active":"2024-03-11T21:16:19.715221+0000","last_became_peered":"2024-03-11T21:16:19.715221+0000","last_unstale":"2024-03-11T21:20:19.068302+0000","last_undegraded":"2024-03-11T21:20:19.068302+0000","last_fullsized":"2024-03-11T21:20:19.068302+0000","mapping_epoch":426,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":4,"last_scrub":"184'130","last_scrub_stamp":"2024-03-11T21:20:13.684884+0000","last_deep_scrub":"184'130","last_deep_scrub_stamp":"2024-03-11T21:20:13.684884+0000","last_clean_scrub_stamp":"2024-03-11T21:20:13.684884+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:24.842309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":940,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.052324+0000","last_change":"2024-03-11T21:19:10.721288+0000","last_active":"2024-03-11T21:20:19.052324+0000","last_peered":"2024-03-11T21:20:19.052324+0000","last_clean":"2024-03-11T21:20:19.052324+0000","last_became_active":"2024-03-11T21:00:27.959408+0000","last_became_peered":"2024-03-11T21:00:27.959408+0000","last_unstale":"2024-03-11T21:20:19.052324+0000","last_undegraded":"2024-03-11T21:20:19.052324+0000","last_fullsized":"2024-03-11T21:20:19.052324+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:19:10.721029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:19:10.721029+0000","last_clean_scrub_stamp":"2024-03-11T21:19:10.721029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1005,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:20.644381+0000","last_change":"2024-03-11T21:20:20.644436+0000","last_active":"2024-03-11T21:20:20.644381+0000","last_peered":"2024-03-11T21:20:20.644381+0000","last_clean":"2024-03-11T21:20:20.644381+0000","last_became_active":"2024-03-11T21:00:48.471212+0000","last_became_peered":"2024-03-11T21:00:48.471212+0000","last_unstale":"2024-03-11T21:20:20.644381+0000","last_undegraded":"2024-03-11T21:20:20.644381+0000","last_fullsized":"2024-03-11T21:20:20.644381+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:20:20.644234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:09:55.786349+0000","last_clean_scrub_stamp":"2024-03-11T21:20:20.644234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:44.957758+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"358'47","reported_seq":545,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:24.683210+0000","last_change":"2024-03-11T21:20:24.683297+0000","last_active":"2024-03-11T21:20:24.683210+0000","last_peered":"2024-03-11T21:20:24.683210+0000","last_clean":"2024-03-11T21:20:24.683210+0000","last_became_active":"2024-03-11T21:18:53.661301+0000","last_became_peered":"2024-03-11T21:18:53.661301+0000","last_unstale":"2024-03-11T21:20:24.683210+0000","last_undegraded":"2024-03-11T21:20:24.683210+0000","last_fullsized":"2024-03-11T21:20:24.683210+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'47","last_scrub_stamp":"2024-03-11T21:20:24.683048+0000","last_deep_scrub":"358'47","last_deep_scrub_stamp":"2024-03-11T21:17:59.358910+0000","last_clean_scrub_stamp":"2024-03-11T21:20:24.683048+0000","objects_scrubbed":8,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:42.618834+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":67,"num_read_kb":53,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":682,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.045878+0000","last_change":"2024-03-11T21:20:13.922389+0000","last_active":"2024-03-11T21:20:19.045878+0000","last_peered":"2024-03-11T21:20:19.045878+0000","last_clean":"2024-03-11T21:20:19.045878+0000","last_became_active":"2024-03-11T21:00:48.458008+0000","last_became_peered":"2024-03-11T21:00:48.458008+0000","last_unstale":"2024-03-11T21:20:19.045878+0000","last_undegraded":"2024-03-11T21:20:19.045878+0000","last_fullsized":"2024-03-11T21:20:19.045878+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:20:13.922307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:33.319221+0000","last_clean_scrub_stamp":"2024-03-11T21:20:13.922307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:21.960346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"184'130","reported_seq":1293,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.038528+0000","last_change":"2024-03-11T21:19:00.281119+0000","last_active":"2024-03-11T21:20:19.038528+0000","last_peered":"2024-03-11T21:20:19.038528+0000","last_clean":"2024-03-11T21:20:19.038528+0000","last_became_active":"2024-03-11T21:16:19.731039+0000","last_became_peered":"2024-03-11T21:16:19.731039+0000","last_unstale":"2024-03-11T21:20:19.038528+0000","last_undegraded":"2024-03-11T21:20:19.038528+0000","last_fullsized":"2024-03-11T21:20:19.038528+0000","mapping_epoch":426,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":4,"last_scrub":"184'130","last_scrub_stamp":"2024-03-11T21:19:00.280848+0000","last_deep_scrub":"184'130","last_deep_scrub_stamp":"2024-03-11T20:58:56.415267+0000","last_clean_scrub_stamp":"2024-03-11T21:19:00.280848+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:26.157013+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":969,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.039511+0000","last_change":"2024-03-11T21:19:37.270379+0000","last_active":"2024-03-11T21:20:19.039511+0000","last_peered":"2024-03-11T21:20:19.039511+0000","last_clean":"2024-03-11T21:20:19.039511+0000","last_became_active":"2024-03-11T20:50:34.321038+0000","last_became_peered":"2024-03-11T20:50:34.321038+0000","last_unstale":"2024-03-11T21:20:19.039511+0000","last_undegraded":"2024-03-11T21:20:19.039511+0000","last_fullsized":"2024-03-11T21:20:19.039511+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:19:37.270220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:11:20.989384+0000","last_clean_scrub_stamp":"2024-03-11T21:19:37.270220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:58.269312+0000","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"358'50","reported_seq":732,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.085703+0000","last_change":"2024-03-11T21:20:19.085797+0000","last_active":"2024-03-11T21:20:19.085703+0000","last_peered":"2024-03-11T21:20:19.085703+0000","last_clean":"2024-03-11T21:20:19.085703+0000","last_became_active":"2024-03-11T21:00:48.456178+0000","last_became_peered":"2024-03-11T21:00:48.456178+0000","last_unstale":"2024-03-11T21:20:19.085703+0000","last_undegraded":"2024-03-11T21:20:19.085703+0000","last_fullsized":"2024-03-11T21:20:19.085703+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'50","last_scrub_stamp":"2024-03-11T21:20:19.085539+0000","last_deep_scrub":"109'26","last_deep_scrub_stamp":"2024-03-11T20:52:49.216378+0000","last_clean_scrub_stamp":"2024-03-11T21:20:19.085539+0000","objects_scrubbed":8,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:43.206239+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":58,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":676,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.075312+0000","last_change":"2024-03-11T21:19:01.325208+0000","last_active":"2024-03-11T21:20:19.075312+0000","last_peered":"2024-03-11T21:20:19.075312+0000","last_clean":"2024-03-11T21:20:19.075312+0000","last_became_active":"2024-03-11T21:17:19.978546+0000","last_became_peered":"2024-03-11T21:17:19.978546+0000","last_unstale":"2024-03-11T21:20:19.075312+0000","last_undegraded":"2024-03-11T21:20:19.075312+0000","last_fullsized":"2024-03-11T21:20:19.075312+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:19:01.325113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:19:01.325113+0000","last_clean_scrub_stamp":"2024-03-11T21:19:01.325113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:30.351313+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":686,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.046341+0000","last_change":"2024-03-11T21:19:20.895344+0000","last_active":"2024-03-11T21:20:19.046341+0000","last_peered":"2024-03-11T21:20:19.046341+0000","last_clean":"2024-03-11T21:20:19.046341+0000","last_became_active":"2024-03-11T21:14:50.422215+0000","last_became_peered":"2024-03-11T21:14:50.422215+0000","last_unstale":"2024-03-11T21:20:19.046341+0000","last_undegraded":"2024-03-11T21:20:19.046341+0000","last_fullsized":"2024-03-11T21:20:19.046341+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:19:20.895217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:16:47.986863+0000","last_clean_scrub_stamp":"2024-03-11T21:19:20.895217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:41.983836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":973,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.070547+0000","last_change":"2024-03-11T21:20:15.856629+0000","last_active":"2024-03-11T21:20:19.070547+0000","last_peered":"2024-03-11T21:20:19.070547+0000","last_clean":"2024-03-11T21:20:19.070547+0000","last_became_active":"2024-03-11T21:00:27.962668+0000","last_became_peered":"2024-03-11T21:00:27.962668+0000","last_unstale":"2024-03-11T21:20:19.070547+0000","last_undegraded":"2024-03-11T21:20:19.070547+0000","last_fullsized":"2024-03-11T21:20:19.070547+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:20:15.856455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:20:15.856455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:35.867768+0000","scrub_duration":171,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"358'35","reported_seq":998,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.050251+0000","last_change":"2024-03-11T21:19:23.531256+0000","last_active":"2024-03-11T21:20:19.050251+0000","last_peered":"2024-03-11T21:20:19.050251+0000","last_clean":"2024-03-11T21:20:19.050251+0000","last_became_active":"2024-03-11T20:50:30.101747+0000","last_became_peered":"2024-03-11T20:50:30.101747+0000","last_unstale":"2024-03-11T21:20:19.050251+0000","last_undegraded":"2024-03-11T21:20:19.050251+0000","last_fullsized":"2024-03-11T21:20:19.050251+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'35","last_scrub_stamp":"2024-03-11T21:19:23.530814+0000","last_deep_scrub":"358'35","last_deep_scrub_stamp":"2024-03-11T21:17:53.522668+0000","last_clean_scrub_stamp":"2024-03-11T21:19:23.530814+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:40.955124+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":984,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.069114+0000","last_change":"2024-03-11T21:19:30.708268+0000","last_active":"2024-03-11T21:20:19.069114+0000","last_peered":"2024-03-11T21:20:19.069114+0000","last_clean":"2024-03-11T21:20:19.069114+0000","last_became_active":"2024-03-11T20:50:28.085503+0000","last_became_peered":"2024-03-11T20:50:28.085503+0000","last_unstale":"2024-03-11T21:20:19.069114+0000","last_undegraded":"2024-03-11T21:20:19.069114+0000","last_fullsized":"2024-03-11T21:20:19.069114+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:19:30.708052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:56:03.479180+0000","last_clean_scrub_stamp":"2024-03-11T21:19:30.708052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:45.500639+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"184'130","reported_seq":1294,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.037758+0000","last_change":"2024-03-11T21:20:08.337922+0000","last_active":"2024-03-11T21:20:19.037758+0000","last_peered":"2024-03-11T21:20:19.037758+0000","last_clean":"2024-03-11T21:20:19.037758+0000","last_became_active":"2024-03-11T21:16:19.728863+0000","last_became_peered":"2024-03-11T21:16:19.728863+0000","last_unstale":"2024-03-11T21:20:19.037758+0000","last_undegraded":"2024-03-11T21:20:19.037758+0000","last_fullsized":"2024-03-11T21:20:19.037758+0000","mapping_epoch":426,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":0,"last_scrub":"184'130","last_scrub_stamp":"2024-03-11T21:20:08.337679+0000","last_deep_scrub":"184'130","last_deep_scrub_stamp":"2024-03-11T21:18:35.305688+0000","last_clean_scrub_stamp":"2024-03-11T21:20:08.337679+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:20.325099+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":996,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.082948+0000","last_change":"2024-03-11T21:19:40.667610+0000","last_active":"2024-03-11T21:20:19.082948+0000","last_peered":"2024-03-11T21:20:19.082948+0000","last_clean":"2024-03-11T21:20:19.082948+0000","last_became_active":"2024-03-11T21:00:27.959467+0000","last_became_peered":"2024-03-11T21:00:27.959467+0000","last_unstale":"2024-03-11T21:20:19.082948+0000","last_undegraded":"2024-03-11T21:20:19.082948+0000","last_fullsized":"2024-03-11T21:20:19.082948+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:19:40.667380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:33.325163+0000","last_clean_scrub_stamp":"2024-03-11T21:19:40.667380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:59.839698+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.3","version":"184'130","reported_seq":847,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.045897+0000","last_change":"2024-03-11T21:20:14.969557+0000","last_active":"2024-03-11T21:20:19.045897+0000","last_peered":"2024-03-11T21:20:19.045897+0000","last_clean":"2024-03-11T21:20:19.045897+0000","last_became_active":"2024-03-11T21:16:19.722183+0000","last_became_peered":"2024-03-11T21:16:19.722183+0000","last_unstale":"2024-03-11T21:20:19.045897+0000","last_undegraded":"2024-03-11T21:20:19.045897+0000","last_fullsized":"2024-03-11T21:20:19.045897+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":3,"last_scrub":"184'130","last_scrub_stamp":"2024-03-11T21:20:14.969443+0000","last_deep_scrub":"184'130","last_deep_scrub_stamp":"2024-03-11T20:58:56.415267+0000","last_clean_scrub_stamp":"2024-03-11T21:20:14.969443+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:28.970466+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":674,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.046271+0000","last_change":"2024-03-11T21:19:06.999646+0000","last_active":"2024-03-11T21:20:19.046271+0000","last_peered":"2024-03-11T21:20:19.046271+0000","last_clean":"2024-03-11T21:20:19.046271+0000","last_became_active":"2024-03-11T21:00:48.464596+0000","last_became_peered":"2024-03-11T21:00:48.464596+0000","last_unstale":"2024-03-11T21:20:19.046271+0000","last_undegraded":"2024-03-11T21:20:19.046271+0000","last_fullsized":"2024-03-11T21:20:19.046271+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:19:06.999396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:19:06.999396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:25.215887+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"358'49","reported_seq":198,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.264187+0000","last_change":"2024-03-11T21:19:24.399278+0000","last_active":"2024-03-11T21:20:19.264187+0000","last_peered":"2024-03-11T21:20:19.264187+0000","last_clean":"2024-03-11T21:20:19.264187+0000","last_became_active":"2024-03-11T21:14:50.433732+0000","last_became_peered":"2024-03-11T21:14:50.433732+0000","last_unstale":"2024-03-11T21:20:19.264187+0000","last_undegraded":"2024-03-11T21:20:19.264187+0000","last_fullsized":"2024-03-11T21:20:19.264187+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'49","last_scrub_stamp":"2024-03-11T21:19:24.399051+0000","last_deep_scrub":"100'21","last_deep_scrub_stamp":"2024-03-11T20:51:31.835302+0000","last_clean_scrub_stamp":"2024-03-11T21:19:24.399051+0000","objects_scrubbed":7,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:54.269687+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":54,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":647,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.074784+0000","last_change":"2024-03-11T21:19:26.126592+0000","last_active":"2024-03-11T21:20:19.074784+0000","last_peered":"2024-03-11T21:20:19.074784+0000","last_clean":"2024-03-11T21:20:19.074784+0000","last_became_active":"2024-03-11T21:17:19.982830+0000","last_became_peered":"2024-03-11T21:17:19.982830+0000","last_unstale":"2024-03-11T21:20:19.074784+0000","last_undegraded":"2024-03-11T21:20:19.074784+0000","last_fullsized":"2024-03-11T21:20:19.074784+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:19:26.126371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:19:26.126371+0000","last_clean_scrub_stamp":"2024-03-11T21:19:26.126371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:41.028999+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":182,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.266016+0000","last_change":"2024-03-11T21:19:29.432627+0000","last_active":"2024-03-11T21:20:19.266016+0000","last_peered":"2024-03-11T21:20:19.266016+0000","last_clean":"2024-03-11T21:20:19.266016+0000","last_became_active":"2024-03-11T21:14:50.708638+0000","last_became_peered":"2024-03-11T21:14:50.708638+0000","last_unstale":"2024-03-11T21:20:19.266016+0000","last_undegraded":"2024-03-11T21:20:19.266016+0000","last_fullsized":"2024-03-11T21:20:19.266016+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:19:29.432412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:26.839782+0000","last_clean_scrub_stamp":"2024-03-11T21:19:29.432412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:57.505798+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"184'130","reported_seq":867,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.083074+0000","last_change":"2024-03-11T21:19:15.396876+0000","last_active":"2024-03-11T21:20:19.083074+0000","last_peered":"2024-03-11T21:20:19.083074+0000","last_clean":"2024-03-11T21:20:19.083074+0000","last_became_active":"2024-03-11T21:16:19.741626+0000","last_became_peered":"2024-03-11T21:16:19.741626+0000","last_unstale":"2024-03-11T21:20:19.083074+0000","last_undegraded":"2024-03-11T21:20:19.083074+0000","last_fullsized":"2024-03-11T21:20:19.083074+0000","mapping_epoch":426,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":3,"last_scrub":"184'130","last_scrub_stamp":"2024-03-11T21:19:15.396644+0000","last_deep_scrub":"184'130","last_deep_scrub_stamp":"2024-03-11T21:19:15.396644+0000","last_clean_scrub_stamp":"2024-03-11T21:19:15.396644+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:31.849427+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":682,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.045261+0000","last_change":"2024-03-11T21:19:43.246952+0000","last_active":"2024-03-11T21:20:19.045261+0000","last_peered":"2024-03-11T21:20:19.045261+0000","last_clean":"2024-03-11T21:20:19.045261+0000","last_became_active":"2024-03-11T21:00:48.469688+0000","last_became_peered":"2024-03-11T21:00:48.469688+0000","last_unstale":"2024-03-11T21:20:19.045261+0000","last_undegraded":"2024-03-11T21:20:19.045261+0000","last_fullsized":"2024-03-11T21:20:19.045261+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:19:43.246787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:19:43.246787+0000","last_clean_scrub_stamp":"2024-03-11T21:19:43.246787+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:57.534597+0000","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"145'3","reported_seq":1009,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.081914+0000","last_change":"2024-03-11T21:19:04.573624+0000","last_active":"2024-03-11T21:20:19.081914+0000","last_peered":"2024-03-11T21:20:19.081914+0000","last_clean":"2024-03-11T21:20:19.081914+0000","last_became_active":"2024-03-11T21:14:50.427126+0000","last_became_peered":"2024-03-11T21:14:50.427126+0000","last_unstale":"2024-03-11T21:20:19.081914+0000","last_undegraded":"2024-03-11T21:20:19.081914+0000","last_fullsized":"2024-03-11T21:20:19.081914+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'3","last_scrub_stamp":"2024-03-11T21:19:04.573378+0000","last_deep_scrub":"145'3","last_deep_scrub_stamp":"2024-03-11T21:16:34.671325+0000","last_clean_scrub_stamp":"2024-03-11T21:19:04.573378+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:21.440660+0000","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0.0024049620000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":994,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.039777+0000","last_change":"2024-03-11T21:19:42.227486+0000","last_active":"2024-03-11T21:20:19.039777+0000","last_peered":"2024-03-11T21:20:19.039777+0000","last_clean":"2024-03-11T21:20:19.039777+0000","last_became_active":"2024-03-11T21:14:50.446250+0000","last_became_peered":"2024-03-11T21:14:50.446250+0000","last_unstale":"2024-03-11T21:20:19.039777+0000","last_undegraded":"2024-03-11T21:20:19.039777+0000","last_fullsized":"2024-03-11T21:20:19.039777+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:19:42.227198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:19:42.227198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:56.920091+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"358'58","reported_seq":965,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.556673+0000","last_change":"2024-03-11T21:19:11.966085+0000","last_active":"2024-03-11T21:20:19.556673+0000","last_peered":"2024-03-11T21:20:19.556673+0000","last_clean":"2024-03-11T21:20:19.556673+0000","last_became_active":"2024-03-11T21:00:48.473227+0000","last_became_peered":"2024-03-11T21:00:48.473227+0000","last_unstale":"2024-03-11T21:20:19.556673+0000","last_undegraded":"2024-03-11T21:20:19.556673+0000","last_fullsized":"2024-03-11T21:20:19.556673+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'58","last_scrub_stamp":"2024-03-11T21:19:11.965922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:19:11.965922+0000","objects_scrubbed":10,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":87,"num_read_kb":72,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"424'15","reported_seq":262,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.556803+0000","last_change":"2024-03-11T21:19:51.677093+0000","last_active":"2024-03-11T21:20:19.556803+0000","last_peered":"2024-03-11T21:20:19.556803+0000","last_clean":"2024-03-11T21:20:19.556803+0000","last_became_active":"2024-03-11T21:14:50.431548+0000","last_became_peered":"2024-03-11T21:14:50.431548+0000","last_unstale":"2024-03-11T21:20:19.556803+0000","last_undegraded":"2024-03-11T21:20:19.556803+0000","last_fullsized":"2024-03-11T21:20:19.556803+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"424'15","last_scrub_stamp":"2024-03-11T21:19:51.676978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:19:51.676978+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:06.116410+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"184'130","reported_seq":576,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:22.654809+0000","last_change":"2024-03-11T21:20:22.654878+0000","last_active":"2024-03-11T21:20:22.654809+0000","last_peered":"2024-03-11T21:20:22.654809+0000","last_clean":"2024-03-11T21:20:22.654809+0000","last_became_active":"2024-03-11T21:18:53.662257+0000","last_became_peered":"2024-03-11T21:18:53.662257+0000","last_unstale":"2024-03-11T21:20:22.654809+0000","last_undegraded":"2024-03-11T21:20:22.654809+0000","last_fullsized":"2024-03-11T21:20:22.654809+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":3,"last_scrub":"184'130","last_scrub_stamp":"2024-03-11T21:20:22.654657+0000","last_deep_scrub":"184'130","last_deep_scrub_stamp":"2024-03-11T21:18:42.113943+0000","last_clean_scrub_stamp":"2024-03-11T21:20:22.654657+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:45.402190+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":439,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.265875+0000","last_change":"2024-03-11T21:20:16.570827+0000","last_active":"2024-03-11T21:20:19.265875+0000","last_peered":"2024-03-11T21:20:19.265875+0000","last_clean":"2024-03-11T21:20:19.265875+0000","last_became_active":"2024-03-11T21:18:53.658111+0000","last_became_peered":"2024-03-11T21:18:53.658111+0000","last_unstale":"2024-03-11T21:20:19.265875+0000","last_undegraded":"2024-03-11T21:20:19.265875+0000","last_fullsized":"2024-03-11T21:20:19.265875+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:20:16.570578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:03:43.345257+0000","last_clean_scrub_stamp":"2024-03-11T21:20:16.570578+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:21.286404+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":460,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.265598+0000","last_change":"2024-03-11T21:20:10.492980+0000","last_active":"2024-03-11T21:20:19.265598+0000","last_peered":"2024-03-11T21:20:19.265598+0000","last_clean":"2024-03-11T21:20:19.265598+0000","last_became_active":"2024-03-11T21:18:53.659495+0000","last_became_peered":"2024-03-11T21:18:53.659495+0000","last_unstale":"2024-03-11T21:20:19.265598+0000","last_undegraded":"2024-03-11T21:20:19.265598+0000","last_fullsized":"2024-03-11T21:20:19.265598+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:20:10.492743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:22.617999+0000","last_clean_scrub_stamp":"2024-03-11T21:20:10.492743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:12.391301+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"284'2","reported_seq":687,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.046270+0000","last_change":"2024-03-11T21:20:06.046409+0000","last_active":"2024-03-11T21:20:19.046270+0000","last_peered":"2024-03-11T21:20:19.046270+0000","last_clean":"2024-03-11T21:20:19.046270+0000","last_became_active":"2024-03-11T21:00:48.468682+0000","last_became_peered":"2024-03-11T21:00:48.468682+0000","last_unstale":"2024-03-11T21:20:19.046270+0000","last_undegraded":"2024-03-11T21:20:19.046270+0000","last_fullsized":"2024-03-11T21:20:19.046270+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"284'2","last_scrub_stamp":"2024-03-11T21:20:06.046086+0000","last_deep_scrub":"284'2","last_deep_scrub_stamp":"2024-03-11T21:06:47.645682+0000","last_clean_scrub_stamp":"2024-03-11T21:20:06.046086+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:23.271395+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"357'21","reported_seq":1014,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.052560+0000","last_change":"2024-03-11T21:19:14.618789+0000","last_active":"2024-03-11T21:20:19.052560+0000","last_peered":"2024-03-11T21:20:19.052560+0000","last_clean":"2024-03-11T21:20:19.052560+0000","last_became_active":"2024-03-11T21:14:50.435780+0000","last_became_peered":"2024-03-11T21:14:50.435780+0000","last_unstale":"2024-03-11T21:20:19.052560+0000","last_undegraded":"2024-03-11T21:20:19.052560+0000","last_fullsized":"2024-03-11T21:20:19.052560+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"357'21","last_scrub_stamp":"2024-03-11T21:19:14.618547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:19:14.618547+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:27.633037+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":31,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"424'13","reported_seq":1343,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.556037+0000","last_change":"2024-03-11T21:20:04.515837+0000","last_active":"2024-03-11T21:20:19.556037+0000","last_peered":"2024-03-11T21:20:19.556037+0000","last_clean":"2024-03-11T21:20:19.556037+0000","last_became_active":"2024-03-11T21:14:50.428398+0000","last_became_peered":"2024-03-11T21:14:50.428398+0000","last_unstale":"2024-03-11T21:20:19.556037+0000","last_undegraded":"2024-03-11T21:20:19.556037+0000","last_fullsized":"2024-03-11T21:20:19.556037+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"424'13","last_scrub_stamp":"2024-03-11T21:20:04.515611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:20:04.515611+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:14.281190+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"184'130","reported_seq":839,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.051440+0000","last_change":"2024-03-11T21:19:24.537746+0000","last_active":"2024-03-11T21:20:19.051440+0000","last_peered":"2024-03-11T21:20:19.051440+0000","last_clean":"2024-03-11T21:20:19.051440+0000","last_became_active":"2024-03-11T21:16:19.723401+0000","last_became_peered":"2024-03-11T21:16:19.723401+0000","last_unstale":"2024-03-11T21:20:19.051440+0000","last_undegraded":"2024-03-11T21:20:19.051440+0000","last_fullsized":"2024-03-11T21:20:19.051440+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":3,"last_scrub":"184'130","last_scrub_stamp":"2024-03-11T21:19:24.537515+0000","last_deep_scrub":"184'130","last_deep_scrub_stamp":"2024-03-11T21:18:01.642193+0000","last_clean_scrub_stamp":"2024-03-11T21:19:24.537515+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:33.665293+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":977,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.038864+0000","last_change":"2024-03-11T21:19:40.393255+0000","last_active":"2024-03-11T21:20:19.038864+0000","last_peered":"2024-03-11T21:20:19.038864+0000","last_clean":"2024-03-11T21:20:19.038864+0000","last_became_active":"2024-03-11T20:50:34.326947+0000","last_became_peered":"2024-03-11T20:50:34.326947+0000","last_unstale":"2024-03-11T21:20:19.038864+0000","last_undegraded":"2024-03-11T21:20:19.038864+0000","last_fullsized":"2024-03-11T21:20:19.038864+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:19:40.393138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:06:43.271453+0000","last_clean_scrub_stamp":"2024-03-11T21:19:40.393138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:54.482595+0000","scrub_duration":143,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"120'6","reported_seq":2650,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:24.298301+0000","last_change":"2024-03-11T21:20:05.533807+0000","last_active":"2024-03-11T21:20:24.298301+0000","last_peered":"2024-03-11T21:20:24.298301+0000","last_clean":"2024-03-11T21:20:24.298301+0000","last_became_active":"2024-03-11T21:00:48.468997+0000","last_became_peered":"2024-03-11T21:00:48.468997+0000","last_unstale":"2024-03-11T21:20:24.298301+0000","last_undegraded":"2024-03-11T21:20:24.298301+0000","last_fullsized":"2024-03-11T21:20:24.298301+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"120'6","last_scrub_stamp":"2024-03-11T21:20:05.533483+0000","last_deep_scrub":"120'6","last_deep_scrub_stamp":"2024-03-11T21:20:05.533483+0000","last_clean_scrub_stamp":"2024-03-11T21:20:05.533483+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:25.933555+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0035533370000000002,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1657,"num_read_kb":1652,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1002,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.083325+0000","last_change":"2024-03-11T21:20:18.792820+0000","last_active":"2024-03-11T21:20:19.083325+0000","last_peered":"2024-03-11T21:20:19.083325+0000","last_clean":"2024-03-11T21:20:19.083325+0000","last_became_active":"2024-03-11T20:50:28.071707+0000","last_became_peered":"2024-03-11T20:50:28.071707+0000","last_unstale":"2024-03-11T21:20:19.083325+0000","last_undegraded":"2024-03-11T21:20:19.083325+0000","last_fullsized":"2024-03-11T21:20:19.083325+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:20:18.792571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:23.417283+0000","last_clean_scrub_stamp":"2024-03-11T21:20:18.792571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:36.806503+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"358'36","reported_seq":204,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.265204+0000","last_change":"2024-03-11T21:20:16.561101+0000","last_active":"2024-03-11T21:20:19.265204+0000","last_peered":"2024-03-11T21:20:19.265204+0000","last_clean":"2024-03-11T21:20:19.265204+0000","last_became_active":"2024-03-11T21:14:50.418699+0000","last_became_peered":"2024-03-11T21:14:50.418699+0000","last_unstale":"2024-03-11T21:20:19.265204+0000","last_undegraded":"2024-03-11T21:20:19.265204+0000","last_fullsized":"2024-03-11T21:20:19.265204+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'36","last_scrub_stamp":"2024-03-11T21:20:16.560988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:20:16.560988+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:19.815766+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":47,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":473,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.265651+0000","last_change":"2024-03-11T21:19:36.390872+0000","last_active":"2024-03-11T21:20:19.265651+0000","last_peered":"2024-03-11T21:20:19.265651+0000","last_clean":"2024-03-11T21:20:19.265651+0000","last_became_active":"2024-03-11T21:18:53.663029+0000","last_became_peered":"2024-03-11T21:18:53.663029+0000","last_unstale":"2024-03-11T21:20:19.265651+0000","last_undegraded":"2024-03-11T21:20:19.265651+0000","last_fullsized":"2024-03-11T21:20:19.265651+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:19:36.390649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:19.695800+0000","last_clean_scrub_stamp":"2024-03-11T21:19:36.390649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:58.975394+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"184'130","reported_seq":846,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.045776+0000","last_change":"2024-03-11T21:19:30.931124+0000","last_active":"2024-03-11T21:20:19.045776+0000","last_peered":"2024-03-11T21:20:19.045776+0000","last_clean":"2024-03-11T21:20:19.045776+0000","last_became_active":"2024-03-11T21:16:24.835115+0000","last_became_peered":"2024-03-11T21:16:24.835115+0000","last_unstale":"2024-03-11T21:20:19.045776+0000","last_undegraded":"2024-03-11T21:20:19.045776+0000","last_fullsized":"2024-03-11T21:20:19.045776+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":444,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":4,"last_scrub":"184'130","last_scrub_stamp":"2024-03-11T21:19:30.930932+0000","last_deep_scrub":"184'130","last_deep_scrub_stamp":"2024-03-11T20:58:56.415267+0000","last_clean_scrub_stamp":"2024-03-11T21:19:30.930932+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:51.628370+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":939,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.051371+0000","last_change":"2024-03-11T21:20:16.567982+0000","last_active":"2024-03-11T21:20:19.051371+0000","last_peered":"2024-03-11T21:20:19.051371+0000","last_clean":"2024-03-11T21:20:19.051371+0000","last_became_active":"2024-03-11T20:50:34.335370+0000","last_became_peered":"2024-03-11T20:50:34.335370+0000","last_unstale":"2024-03-11T21:20:19.051371+0000","last_undegraded":"2024-03-11T21:20:19.051371+0000","last_fullsized":"2024-03-11T21:20:19.051371+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:20:16.567878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:17:04.544817+0000","last_clean_scrub_stamp":"2024-03-11T21:20:16.567878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:28.103994+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"486'19","reported_seq":682,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.558717+0000","last_change":"2024-03-11T21:19:55.366549+0000","last_active":"2024-03-11T21:20:19.558717+0000","last_peered":"2024-03-11T21:20:19.558717+0000","last_clean":"2024-03-11T21:20:19.558717+0000","last_became_active":"2024-03-11T21:18:53.667732+0000","last_became_peered":"2024-03-11T21:18:53.667732+0000","last_unstale":"2024-03-11T21:20:19.558717+0000","last_undegraded":"2024-03-11T21:20:19.558717+0000","last_fullsized":"2024-03-11T21:20:19.558717+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"486'19","last_scrub_stamp":"2024-03-11T21:19:55.366206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:19:55.366206+0000","objects_scrubbed":1,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:22.205065+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":1005,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.037826+0000","last_change":"2024-03-11T21:19:39.269234+0000","last_active":"2024-03-11T21:20:19.037826+0000","last_peered":"2024-03-11T21:20:19.037826+0000","last_clean":"2024-03-11T21:20:19.037826+0000","last_became_active":"2024-03-11T21:14:50.433372+0000","last_became_peered":"2024-03-11T21:14:50.433372+0000","last_unstale":"2024-03-11T21:20:19.037826+0000","last_undegraded":"2024-03-11T21:20:19.037826+0000","last_fullsized":"2024-03-11T21:20:19.037826+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:19:39.269015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:06:42.242801+0000","last_clean_scrub_stamp":"2024-03-11T21:19:39.269015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:53.783342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"466'29","reported_seq":417,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.266122+0000","last_change":"2024-03-11T21:19:47.432737+0000","last_active":"2024-03-11T21:20:19.266122+0000","last_peered":"2024-03-11T21:20:19.266122+0000","last_clean":"2024-03-11T21:20:19.266122+0000","last_became_active":"2024-03-11T21:18:53.662913+0000","last_became_peered":"2024-03-11T21:18:53.662913+0000","last_unstale":"2024-03-11T21:20:19.266122+0000","last_undegraded":"2024-03-11T21:20:19.266122+0000","last_fullsized":"2024-03-11T21:20:19.266122+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"466'29","last_scrub_stamp":"2024-03-11T21:19:47.432491+0000","last_deep_scrub":"466'29","last_deep_scrub_stamp":"2024-03-11T21:19:47.432491+0000","last_clean_scrub_stamp":"2024-03-11T21:19:47.432491+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:55.029516+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":40,"num_read_kb":31,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"184'130","reported_seq":842,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.068882+0000","last_change":"2024-03-11T21:20:00.576511+0000","last_active":"2024-03-11T21:20:19.068882+0000","last_peered":"2024-03-11T21:20:19.068882+0000","last_clean":"2024-03-11T21:20:19.068882+0000","last_became_active":"2024-03-11T21:11:28.396897+0000","last_became_peered":"2024-03-11T21:11:28.396897+0000","last_unstale":"2024-03-11T21:20:19.068882+0000","last_undegraded":"2024-03-11T21:20:19.068882+0000","last_fullsized":"2024-03-11T21:20:19.068882+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":4,"last_scrub":"184'130","last_scrub_stamp":"2024-03-11T21:20:00.576212+0000","last_deep_scrub":"184'130","last_deep_scrub_stamp":"2024-03-11T20:58:56.415267+0000","last_clean_scrub_stamp":"2024-03-11T21:20:00.576212+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:22.709835+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":996,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.070247+0000","last_change":"2024-03-11T21:20:07.762049+0000","last_active":"2024-03-11T21:20:19.070247+0000","last_peered":"2024-03-11T21:20:19.070247+0000","last_clean":"2024-03-11T21:20:19.070247+0000","last_became_active":"2024-03-11T21:14:50.434097+0000","last_became_peered":"2024-03-11T21:14:50.434097+0000","last_unstale":"2024-03-11T21:20:19.070247+0000","last_undegraded":"2024-03-11T21:20:19.070247+0000","last_fullsized":"2024-03-11T21:20:19.070247+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:20:07.761674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:27.775534+0000","last_clean_scrub_stamp":"2024-03-11T21:20:07.761674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:18.920560+0000","scrub_duration":159,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":681,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.046561+0000","last_change":"2024-03-11T21:19:14.926930+0000","last_active":"2024-03-11T21:20:19.046561+0000","last_peered":"2024-03-11T21:20:19.046561+0000","last_clean":"2024-03-11T21:20:19.046561+0000","last_became_active":"2024-03-11T21:00:48.469596+0000","last_became_peered":"2024-03-11T21:00:48.469596+0000","last_unstale":"2024-03-11T21:20:19.046561+0000","last_undegraded":"2024-03-11T21:20:19.046561+0000","last_fullsized":"2024-03-11T21:20:19.046561+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:19:14.926693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:19:14.926693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:33.263951+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":674,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.045412+0000","last_change":"2024-03-11T21:19:44.140736+0000","last_active":"2024-03-11T21:20:19.045412+0000","last_peered":"2024-03-11T21:20:19.045412+0000","last_clean":"2024-03-11T21:20:19.045412+0000","last_became_active":"2024-03-11T21:00:48.466084+0000","last_became_peered":"2024-03-11T21:00:48.466084+0000","last_unstale":"2024-03-11T21:20:19.045412+0000","last_undegraded":"2024-03-11T21:20:19.045412+0000","last_fullsized":"2024-03-11T21:20:19.045412+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:19:44.140446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:16:40.924953+0000","last_clean_scrub_stamp":"2024-03-11T21:19:44.140446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:46.051422+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"358'46","reported_seq":1075,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.049868+0000","last_change":"2024-03-11T21:19:54.480119+0000","last_active":"2024-03-11T21:20:19.049868+0000","last_peered":"2024-03-11T21:20:19.049868+0000","last_clean":"2024-03-11T21:20:19.049868+0000","last_became_active":"2024-03-11T21:00:48.472191+0000","last_became_peered":"2024-03-11T21:00:48.472191+0000","last_unstale":"2024-03-11T21:20:19.049868+0000","last_undegraded":"2024-03-11T21:20:19.049868+0000","last_fullsized":"2024-03-11T21:20:19.049868+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'46","last_scrub_stamp":"2024-03-11T21:19:54.479744+0000","last_deep_scrub":"116'22","last_deep_scrub_stamp":"2024-03-11T20:53:47.364709+0000","last_clean_scrub_stamp":"2024-03-11T21:19:54.479744+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:58.051113+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"184'130","reported_seq":1270,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.037652+0000","last_change":"2024-03-11T21:19:17.344102+0000","last_active":"2024-03-11T21:20:19.037652+0000","last_peered":"2024-03-11T21:20:19.037652+0000","last_clean":"2024-03-11T21:20:19.037652+0000","last_became_active":"2024-03-11T21:11:28.405250+0000","last_became_peered":"2024-03-11T21:11:28.405250+0000","last_unstale":"2024-03-11T21:20:19.037652+0000","last_undegraded":"2024-03-11T21:20:19.037652+0000","last_fullsized":"2024-03-11T21:20:19.037652+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":4,"last_scrub":"184'130","last_scrub_stamp":"2024-03-11T21:19:17.343882+0000","last_deep_scrub":"184'130","last_deep_scrub_stamp":"2024-03-11T20:58:56.415267+0000","last_clean_scrub_stamp":"2024-03-11T21:19:17.343882+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:27.116532+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":476,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.267093+0000","last_change":"2024-03-11T21:20:11.575856+0000","last_active":"2024-03-11T21:20:19.267093+0000","last_peered":"2024-03-11T21:20:19.267093+0000","last_clean":"2024-03-11T21:20:19.267093+0000","last_became_active":"2024-03-11T21:18:53.661803+0000","last_became_peered":"2024-03-11T21:18:53.661803+0000","last_unstale":"2024-03-11T21:20:19.267093+0000","last_undegraded":"2024-03-11T21:20:19.267093+0000","last_fullsized":"2024-03-11T21:20:19.267093+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:20:11.575635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:20:11.575635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:23.706717+0000","scrub_duration":71,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":682,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:20.123661+0000","last_change":"2024-03-11T21:20:20.123703+0000","last_active":"2024-03-11T21:20:20.123661+0000","last_peered":"2024-03-11T21:20:20.123661+0000","last_clean":"2024-03-11T21:20:20.123661+0000","last_became_active":"2024-03-11T21:00:48.461367+0000","last_became_peered":"2024-03-11T21:00:48.461367+0000","last_unstale":"2024-03-11T21:20:20.123661+0000","last_undegraded":"2024-03-11T21:20:20.123661+0000","last_fullsized":"2024-03-11T21:20:20.123661+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:20:20.123568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:03:37.601229+0000","last_clean_scrub_stamp":"2024-03-11T21:20:20.123568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:32.356002+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"358'25","reported_seq":1026,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:20.624294+0000","last_change":"2024-03-11T21:20:20.624418+0000","last_active":"2024-03-11T21:20:20.624294+0000","last_peered":"2024-03-11T21:20:20.624294+0000","last_clean":"2024-03-11T21:20:20.624294+0000","last_became_active":"2024-03-11T21:14:50.436328+0000","last_became_peered":"2024-03-11T21:14:50.436328+0000","last_unstale":"2024-03-11T21:20:20.624294+0000","last_undegraded":"2024-03-11T21:20:20.624294+0000","last_fullsized":"2024-03-11T21:20:20.624294+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'25","last_scrub_stamp":"2024-03-11T21:20:20.624084+0000","last_deep_scrub":"358'25","last_deep_scrub_stamp":"2024-03-11T21:20:20.624084+0000","last_clean_scrub_stamp":"2024-03-11T21:20:20.624084+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:21.095797+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":483,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.266524+0000","last_change":"2024-03-11T21:19:22.266951+0000","last_active":"2024-03-11T21:20:19.266524+0000","last_peered":"2024-03-11T21:20:19.266524+0000","last_clean":"2024-03-11T21:20:19.266524+0000","last_became_active":"2024-03-11T21:18:53.655669+0000","last_became_peered":"2024-03-11T21:18:53.655669+0000","last_unstale":"2024-03-11T21:20:19.266524+0000","last_undegraded":"2024-03-11T21:20:19.266524+0000","last_fullsized":"2024-03-11T21:20:19.266524+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:19:22.266671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:35.670149+0000","last_clean_scrub_stamp":"2024-03-11T21:19:22.266671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:31.484072+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"184'130","reported_seq":834,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.068706+0000","last_change":"2024-03-11T21:19:04.421601+0000","last_active":"2024-03-11T21:20:19.068706+0000","last_peered":"2024-03-11T21:20:19.068706+0000","last_clean":"2024-03-11T21:20:19.068706+0000","last_became_active":"2024-03-11T21:11:28.398728+0000","last_became_peered":"2024-03-11T21:11:28.398728+0000","last_unstale":"2024-03-11T21:20:19.068706+0000","last_undegraded":"2024-03-11T21:20:19.068706+0000","last_fullsized":"2024-03-11T21:20:19.068706+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":4,"last_scrub":"184'130","last_scrub_stamp":"2024-03-11T21:19:04.421375+0000","last_deep_scrub":"184'130","last_deep_scrub_stamp":"2024-03-11T21:19:04.421375+0000","last_clean_scrub_stamp":"2024-03-11T21:19:04.421375+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:30.624387+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":664,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.075500+0000","last_change":"2024-03-11T21:20:10.061606+0000","last_active":"2024-03-11T21:20:19.075500+0000","last_peered":"2024-03-11T21:20:19.075500+0000","last_clean":"2024-03-11T21:20:19.075500+0000","last_became_active":"2024-03-11T21:17:19.515538+0000","last_became_peered":"2024-03-11T21:17:19.515538+0000","last_unstale":"2024-03-11T21:20:19.075500+0000","last_undegraded":"2024-03-11T21:20:19.075500+0000","last_fullsized":"2024-03-11T21:20:19.075500+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:20:10.061493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:20:10.061493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:16.575596+0000","scrub_duration":223,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"486'23","reported_seq":693,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.559446+0000","last_change":"2024-03-11T21:19:05.163849+0000","last_active":"2024-03-11T21:20:19.559446+0000","last_peered":"2024-03-11T21:20:19.559446+0000","last_clean":"2024-03-11T21:20:19.559446+0000","last_became_active":"2024-03-11T21:18:53.663691+0000","last_became_peered":"2024-03-11T21:18:53.663691+0000","last_unstale":"2024-03-11T21:20:19.559446+0000","last_undegraded":"2024-03-11T21:20:19.559446+0000","last_fullsized":"2024-03-11T21:20:19.559446+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"486'23","last_scrub_stamp":"2024-03-11T21:19:05.163546+0000","last_deep_scrub":"466'22","last_deep_scrub_stamp":"2024-03-11T21:17:29.394614+0000","last_clean_scrub_stamp":"2024-03-11T21:19:05.163546+0000","objects_scrubbed":1,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:30.995515+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":981,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.038930+0000","last_change":"2024-03-11T21:19:14.348051+0000","last_active":"2024-03-11T21:20:19.038930+0000","last_peered":"2024-03-11T21:20:19.038930+0000","last_clean":"2024-03-11T21:20:19.038930+0000","last_became_active":"2024-03-11T21:00:27.970573+0000","last_became_peered":"2024-03-11T21:00:27.970573+0000","last_unstale":"2024-03-11T21:20:19.038930+0000","last_undegraded":"2024-03-11T21:20:19.038930+0000","last_fullsized":"2024-03-11T21:20:19.038930+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:19:14.347834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:11:14.836808+0000","last_clean_scrub_stamp":"2024-03-11T21:19:14.347834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:43.922964+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"358'60","reported_seq":1147,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.082240+0000","last_change":"2024-03-11T21:19:19.534707+0000","last_active":"2024-03-11T21:20:19.082240+0000","last_peered":"2024-03-11T21:20:19.082240+0000","last_clean":"2024-03-11T21:20:19.082240+0000","last_became_active":"2024-03-11T21:00:48.469854+0000","last_became_peered":"2024-03-11T21:00:48.469854+0000","last_unstale":"2024-03-11T21:20:19.082240+0000","last_undegraded":"2024-03-11T21:20:19.082240+0000","last_fullsized":"2024-03-11T21:20:19.082240+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'60","last_scrub_stamp":"2024-03-11T21:19:19.534453+0000","last_deep_scrub":"161'28","last_deep_scrub_stamp":"2024-03-11T20:57:20.816278+0000","last_clean_scrub_stamp":"2024-03-11T21:19:19.534453+0000","objects_scrubbed":9,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:34.244400+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":80,"num_read_kb":69,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.f","version":"184'130","reported_seq":864,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:21.905502+0000","last_change":"2024-03-11T21:20:21.905579+0000","last_active":"2024-03-11T21:20:21.905502+0000","last_peered":"2024-03-11T21:20:21.905502+0000","last_clean":"2024-03-11T21:20:21.905502+0000","last_became_active":"2024-03-11T21:16:24.834591+0000","last_became_peered":"2024-03-11T21:16:24.834591+0000","last_unstale":"2024-03-11T21:20:21.905502+0000","last_undegraded":"2024-03-11T21:20:21.905502+0000","last_fullsized":"2024-03-11T21:20:21.905502+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":444,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":4,"last_scrub":"184'130","last_scrub_stamp":"2024-03-11T21:20:21.905300+0000","last_deep_scrub":"184'130","last_deep_scrub_stamp":"2024-03-11T21:20:21.905300+0000","last_clean_scrub_stamp":"2024-03-11T21:20:21.905300+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:22.380704+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":968,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.037569+0000","last_change":"2024-03-11T21:19:28.257284+0000","last_active":"2024-03-11T21:20:19.037569+0000","last_peered":"2024-03-11T21:20:19.037569+0000","last_clean":"2024-03-11T21:20:19.037569+0000","last_became_active":"2024-03-11T20:50:34.327139+0000","last_became_peered":"2024-03-11T20:50:34.327139+0000","last_unstale":"2024-03-11T21:20:19.037569+0000","last_undegraded":"2024-03-11T21:20:19.037569+0000","last_fullsized":"2024-03-11T21:20:19.037569+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:19:28.257070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:09:53.436482+0000","last_clean_scrub_stamp":"2024-03-11T21:19:28.257070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:30.893734+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":675,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.076168+0000","last_change":"2024-03-11T21:19:27.113085+0000","last_active":"2024-03-11T21:20:19.076168+0000","last_peered":"2024-03-11T21:20:19.076168+0000","last_clean":"2024-03-11T21:20:19.076168+0000","last_became_active":"2024-03-11T21:17:19.980258+0000","last_became_peered":"2024-03-11T21:17:19.980258+0000","last_unstale":"2024-03-11T21:20:19.076168+0000","last_undegraded":"2024-03-11T21:20:19.076168+0000","last_fullsized":"2024-03-11T21:20:19.076168+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:19:27.112791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:57:17.447191+0000","last_clean_scrub_stamp":"2024-03-11T21:19:27.112791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:28.266969+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"358'53","reported_seq":1054,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.040163+0000","last_change":"2024-03-11T21:19:25.291742+0000","last_active":"2024-03-11T21:20:19.040163+0000","last_peered":"2024-03-11T21:20:19.040163+0000","last_clean":"2024-03-11T21:20:19.040163+0000","last_became_active":"2024-03-11T20:50:30.111819+0000","last_became_peered":"2024-03-11T20:50:30.111819+0000","last_unstale":"2024-03-11T21:20:19.040163+0000","last_undegraded":"2024-03-11T21:20:19.040163+0000","last_fullsized":"2024-03-11T21:20:19.040163+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'53","last_scrub_stamp":"2024-03-11T21:19:25.291454+0000","last_deep_scrub":"102'25","last_deep_scrub_stamp":"2024-03-11T20:51:35.187094+0000","last_clean_scrub_stamp":"2024-03-11T21:19:25.291454+0000","objects_scrubbed":8,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:41.693362+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":61,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":999,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.081762+0000","last_change":"2024-03-11T21:19:49.767793+0000","last_active":"2024-03-11T21:20:19.081762+0000","last_peered":"2024-03-11T21:20:19.081762+0000","last_clean":"2024-03-11T21:20:19.081762+0000","last_became_active":"2024-03-11T21:00:48.362389+0000","last_became_peered":"2024-03-11T21:00:48.362389+0000","last_unstale":"2024-03-11T21:20:19.081762+0000","last_undegraded":"2024-03-11T21:20:19.081762+0000","last_fullsized":"2024-03-11T21:20:19.081762+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:19:49.767576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:19:49.767576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:09.061465+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"184'130","reported_seq":721,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.574835+0000","last_change":"2024-03-11T21:20:19.574958+0000","last_active":"2024-03-11T21:20:19.574835+0000","last_peered":"2024-03-11T21:20:19.574835+0000","last_clean":"2024-03-11T21:20:19.574835+0000","last_became_active":"2024-03-11T21:18:53.660113+0000","last_became_peered":"2024-03-11T21:18:53.660113+0000","last_unstale":"2024-03-11T21:20:19.574835+0000","last_undegraded":"2024-03-11T21:20:19.574835+0000","last_fullsized":"2024-03-11T21:20:19.574835+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":444,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":4,"last_scrub":"184'130","last_scrub_stamp":"2024-03-11T21:20:19.574640+0000","last_deep_scrub":"184'130","last_deep_scrub_stamp":"2024-03-11T21:16:24.952261+0000","last_clean_scrub_stamp":"2024-03-11T21:20:19.574640+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:24.727860+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":182,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.264004+0000","last_change":"2024-03-11T21:19:13.617810+0000","last_active":"2024-03-11T21:20:19.264004+0000","last_peered":"2024-03-11T21:20:19.264004+0000","last_clean":"2024-03-11T21:20:19.264004+0000","last_became_active":"2024-03-11T21:14:50.706718+0000","last_became_peered":"2024-03-11T21:14:50.706718+0000","last_unstale":"2024-03-11T21:20:19.264004+0000","last_undegraded":"2024-03-11T21:20:19.264004+0000","last_fullsized":"2024-03-11T21:20:19.264004+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:19:13.617698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:55:58.888890+0000","last_clean_scrub_stamp":"2024-03-11T21:19:13.617698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:38.668712+0000","scrub_duration":159,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":681,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:24.069348+0000","last_change":"2024-03-11T21:20:24.069459+0000","last_active":"2024-03-11T21:20:24.069348+0000","last_peered":"2024-03-11T21:20:24.069348+0000","last_clean":"2024-03-11T21:20:24.069348+0000","last_became_active":"2024-03-11T21:00:48.469015+0000","last_became_peered":"2024-03-11T21:00:48.469015+0000","last_unstale":"2024-03-11T21:20:24.069348+0000","last_undegraded":"2024-03-11T21:20:24.069348+0000","last_fullsized":"2024-03-11T21:20:24.069348+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:20:24.069148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:20:24.069148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:40.926396+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"393'35","reported_seq":1029,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.069855+0000","last_change":"2024-03-11T21:19:41.728127+0000","last_active":"2024-03-11T21:20:19.069855+0000","last_peered":"2024-03-11T21:20:19.069855+0000","last_clean":"2024-03-11T21:20:19.069855+0000","last_became_active":"2024-03-11T20:50:30.109549+0000","last_became_peered":"2024-03-11T20:50:30.109549+0000","last_unstale":"2024-03-11T21:20:19.069855+0000","last_undegraded":"2024-03-11T21:20:19.069855+0000","last_fullsized":"2024-03-11T21:20:19.069855+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"393'35","last_scrub_stamp":"2024-03-11T21:19:41.727883+0000","last_deep_scrub":"393'35","last_deep_scrub_stamp":"2024-03-11T21:12:35.497207+0000","last_clean_scrub_stamp":"2024-03-11T21:19:41.727883+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:48.190520+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":182,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.264591+0000","last_change":"2024-03-11T21:19:22.384897+0000","last_active":"2024-03-11T21:20:19.264591+0000","last_peered":"2024-03-11T21:20:19.264591+0000","last_clean":"2024-03-11T21:20:19.264591+0000","last_became_active":"2024-03-11T21:14:50.682439+0000","last_became_peered":"2024-03-11T21:14:50.682439+0000","last_unstale":"2024-03-11T21:20:19.264591+0000","last_undegraded":"2024-03-11T21:20:19.264591+0000","last_fullsized":"2024-03-11T21:20:19.264591+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:19:22.384677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:19:22.384677+0000","last_clean_scrub_stamp":"2024-03-11T21:19:22.384677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:32.406155+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"184'130","reported_seq":669,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.039171+0000","last_change":"2024-03-11T21:20:15.310738+0000","last_active":"2024-03-11T21:20:19.039171+0000","last_peered":"2024-03-11T21:20:19.039171+0000","last_clean":"2024-03-11T21:20:19.039171+0000","last_became_active":"2024-03-11T21:16:24.829946+0000","last_became_peered":"2024-03-11T21:16:24.829946+0000","last_unstale":"2024-03-11T21:20:19.039171+0000","last_undegraded":"2024-03-11T21:20:19.039171+0000","last_fullsized":"2024-03-11T21:20:19.039171+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":444,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":4,"last_scrub":"184'130","last_scrub_stamp":"2024-03-11T21:20:15.310493+0000","last_deep_scrub":"184'130","last_deep_scrub_stamp":"2024-03-11T21:20:15.310493+0000","last_clean_scrub_stamp":"2024-03-11T21:20:15.310493+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:43.111774+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":258,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.068544+0000","last_change":"2024-03-11T21:19:32.787154+0000","last_active":"2024-03-11T21:20:19.068544+0000","last_peered":"2024-03-11T21:20:19.068544+0000","last_clean":"2024-03-11T21:20:19.068544+0000","last_became_active":"2024-03-11T21:14:50.427625+0000","last_became_peered":"2024-03-11T21:14:50.427625+0000","last_unstale":"2024-03-11T21:20:19.068544+0000","last_undegraded":"2024-03-11T21:20:19.068544+0000","last_fullsized":"2024-03-11T21:20:19.068544+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:19:32.786935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:19:32.786935+0000","last_clean_scrub_stamp":"2024-03-11T21:19:32.786935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:42.720188+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":675,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.075478+0000","last_change":"2024-03-11T21:20:03.869426+0000","last_active":"2024-03-11T21:20:19.075478+0000","last_peered":"2024-03-11T21:20:19.075478+0000","last_clean":"2024-03-11T21:20:19.075478+0000","last_became_active":"2024-03-11T21:17:19.515138+0000","last_became_peered":"2024-03-11T21:17:19.515138+0000","last_unstale":"2024-03-11T21:20:19.075478+0000","last_undegraded":"2024-03-11T21:20:19.075478+0000","last_fullsized":"2024-03-11T21:20:19.075478+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:20:03.869258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:20:03.869258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:08.713930+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"391'46","reported_seq":1089,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.040120+0000","last_change":"2024-03-11T21:20:09.372004+0000","last_active":"2024-03-11T21:20:19.040120+0000","last_peered":"2024-03-11T21:20:19.040120+0000","last_clean":"2024-03-11T21:20:19.040120+0000","last_became_active":"2024-03-11T21:14:50.427841+0000","last_became_peered":"2024-03-11T21:14:50.427841+0000","last_unstale":"2024-03-11T21:20:19.040120+0000","last_undegraded":"2024-03-11T21:20:19.040120+0000","last_fullsized":"2024-03-11T21:20:19.040120+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"391'46","last_scrub_stamp":"2024-03-11T21:20:09.371762+0000","last_deep_scrub":"391'46","last_deep_scrub_stamp":"2024-03-11T21:12:31.145006+0000","last_clean_scrub_stamp":"2024-03-11T21:20:09.371762+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:27.902715+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":975,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.068403+0000","last_change":"2024-03-11T21:19:47.620205+0000","last_active":"2024-03-11T21:20:19.068403+0000","last_peered":"2024-03-11T21:20:19.068403+0000","last_clean":"2024-03-11T21:20:19.068403+0000","last_became_active":"2024-03-11T20:50:32.652854+0000","last_became_peered":"2024-03-11T20:50:32.652854+0000","last_unstale":"2024-03-11T21:20:19.068403+0000","last_undegraded":"2024-03-11T21:20:19.068403+0000","last_fullsized":"2024-03-11T21:20:19.068403+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:19:47.619918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:59:00.946094+0000","last_clean_scrub_stamp":"2024-03-11T21:19:47.619918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:08.341012+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":20,"num_read_kb":20,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1914,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2730,"ondisk_log_size":2730,"up":63,"acting":63,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1540,"num_read_kb":1297,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":255,"num_bytes_recovered":3878,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3632,"internal_metadata":0},"log_size":1088,"ondisk_log_size":1088,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":134,"ondisk_log_size":134,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1657,"num_read_kb":1652,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":68,"seq":292057776499,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":631672,"kb_used_data":752,"kb_used_omap":1,"kb_used_meta":630910,"kb_avail":93113480,"statfs":{"total":95995035648,"available":95348203520,"internally_reserved":0,"allocated":770048,"data_stored":433167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":646052298},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":57,"seq":244813136253,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":537576,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":536510,"kb_avail":93207576,"statfs":{"total":95995035648,"available":95444557824,"internally_reserved":0,"allocated":1081344,"data_stored":760857,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":549386695},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":211,"seq":906238099694,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":510448,"kb_used_data":744,"kb_used_omap":7,"kb_used_meta":509688,"kb_avail":93234704,"statfs":{"total":95995035648,"available":95472336896,"internally_reserved":0,"allocated":761856,"data_stored":433423,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8078,"internal_metadata":521920626},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738811,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":595888,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":594814,"kb_avail":93149264,"statfs":{"total":95995035648,"available":95384846336,"internally_reserved":0,"allocated":1089536,"data_stored":759789,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":609089989},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604379046,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":577768,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":576766,"kb_avail":93167384,"statfs":{"total":95995035648,"available":95403401216,"internally_reserved":0,"allocated":1011712,"data_stored":694959,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":590608837},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":28,"seq":120259084699,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":672488,"kb_used_data":1312,"kb_used_omap":1,"kb_used_meta":671166,"kb_avail":93072664,"statfs":{"total":95995035648,"available":95306407936,"internally_reserved":0,"allocated":1343488,"data_stored":1023015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":687274440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":204,"seq":876173328626,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":424648,"kb_used_data":768,"kb_used_omap":4,"kb_used_meta":423867,"kb_avail":93320504,"statfs":{"total":95995035648,"available":95560196096,"internally_reserved":0,"allocated":786432,"data_stored":434037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4451,"internal_metadata":434040477},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":44,"seq":188978561415,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":549992,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":548990,"kb_avail":93195160,"statfs":{"total":95995035648,"available":95431843840,"internally_reserved":0,"allocated":1015808,"data_stored":694881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":562166213},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3632,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T21:20:29.835 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T21:20:30.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:29 smithi146 bash[23809]: cluster 2024-03-11T21:20:28.390299+0000 osd.7 (osd.7) 731 : cluster 0 1.2 scrub starts 2024-03-11T21:20:30.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:29 smithi146 bash[23809]: cluster 2024-03-11T21:20:28.392773+0000 osd.7 (osd.7) 732 : cluster 0 1.2 scrub ok 2024-03-11T21:20:30.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:29 smithi146 bash[23809]: cluster 2024-03-11T21:20:28.602774+0000 mgr.y (mgr.24380) 1093 : cluster 0 pgmap v1266: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:20:30.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:29 smithi146 bash[23809]: cluster 2024-03-11T21:20:28.610919+0000 osd.3 (osd.3) 664 : cluster 0 6.6 scrub starts 2024-03-11T21:20:30.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:29 smithi146 bash[23809]: cluster 2024-03-11T21:20:28.643779+0000 osd.3 (osd.3) 665 : cluster 0 6.6 scrub ok 2024-03-11T21:20:30.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:29 smithi146 bash[23809]: cluster 2024-03-11T21:20:28.767052+0000 osd.2 (osd.2) 325 : cluster 0 2.c scrub starts 2024-03-11T21:20:30.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:29 smithi146 bash[23809]: cluster 2024-03-11T21:20:28.768733+0000 osd.2 (osd.2) 326 : cluster 0 2.c scrub ok 2024-03-11T21:20:30.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:29 smithi146 bash[23809]: cluster 2024-03-11T21:20:28.916633+0000 osd.6 (osd.6) 578 : cluster 0 4.1f scrub starts 2024-03-11T21:20:30.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:29 smithi146 bash[23809]: cluster 2024-03-11T21:20:28.919498+0000 osd.6 (osd.6) 579 : cluster 0 4.1f scrub ok 2024-03-11T21:20:30.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:29 smithi146 bash[23809]: cluster 2024-03-11T21:20:28.974428+0000 osd.1 (osd.1) 472 : cluster 0 3.4 scrub starts 2024-03-11T21:20:30.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:29 smithi146 bash[23809]: cluster 2024-03-11T21:20:28.978762+0000 osd.1 (osd.1) 473 : cluster 0 3.4 scrub ok 2024-03-11T21:20:30.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:29 smithi146 bash[23809]: cluster 2024-03-11T21:20:29.494912+0000 osd.0 (osd.0) 536 : cluster 0 2.13 scrub starts 2024-03-11T21:20:30.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:29 smithi146 bash[23809]: cluster 2024-03-11T21:20:29.497254+0000 osd.0 (osd.0) 537 : cluster 0 2.13 scrub ok 2024-03-11T21:20:30.023 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:29 smithi023 bash[21372]: cluster 2024-03-11T21:20:28.390299+0000 osd.7 (osd.7) 731 : cluster 0 1.2 scrub starts 2024-03-11T21:20:30.023 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:29 smithi023 bash[21372]: cluster 2024-03-11T21:20:28.392773+0000 osd.7 (osd.7) 732 : cluster 0 1.2 scrub ok 2024-03-11T21:20:30.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:29 smithi023 bash[21372]: cluster 2024-03-11T21:20:28.602774+0000 mgr.y (mgr.24380) 1093 : cluster 0 pgmap v1266: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:20:30.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:29 smithi023 bash[21372]: cluster 2024-03-11T21:20:28.610919+0000 osd.3 (osd.3) 664 : cluster 0 6.6 scrub starts 2024-03-11T21:20:30.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:29 smithi023 bash[21372]: cluster 2024-03-11T21:20:28.643779+0000 osd.3 (osd.3) 665 : cluster 0 6.6 scrub ok 2024-03-11T21:20:30.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:29 smithi023 bash[21372]: cluster 2024-03-11T21:20:28.767052+0000 osd.2 (osd.2) 325 : cluster 0 2.c scrub starts 2024-03-11T21:20:30.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:29 smithi023 bash[21372]: cluster 2024-03-11T21:20:28.768733+0000 osd.2 (osd.2) 326 : cluster 0 2.c scrub ok 2024-03-11T21:20:30.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:29 smithi023 bash[21372]: cluster 2024-03-11T21:20:28.916633+0000 osd.6 (osd.6) 578 : cluster 0 4.1f scrub starts 2024-03-11T21:20:30.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:29 smithi023 bash[21372]: cluster 2024-03-11T21:20:28.919498+0000 osd.6 (osd.6) 579 : cluster 0 4.1f scrub ok 2024-03-11T21:20:30.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:29 smithi023 bash[21372]: cluster 2024-03-11T21:20:28.974428+0000 osd.1 (osd.1) 472 : cluster 0 3.4 scrub starts 2024-03-11T21:20:30.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:29 smithi023 bash[21372]: cluster 2024-03-11T21:20:28.978762+0000 osd.1 (osd.1) 473 : cluster 0 3.4 scrub ok 2024-03-11T21:20:30.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:29 smithi023 bash[21372]: cluster 2024-03-11T21:20:29.494912+0000 osd.0 (osd.0) 536 : cluster 0 2.13 scrub starts 2024-03-11T21:20:30.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:29 smithi023 bash[21372]: cluster 2024-03-11T21:20:29.497254+0000 osd.0 (osd.0) 537 : cluster 0 2.13 scrub ok 2024-03-11T21:20:30.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:29 smithi023 bash[29146]: cluster 2024-03-11T21:20:28.390299+0000 osd.7 (osd.7) 731 : cluster 0 1.2 scrub starts 2024-03-11T21:20:30.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:29 smithi023 bash[29146]: cluster 2024-03-11T21:20:28.392773+0000 osd.7 (osd.7) 732 : cluster 0 1.2 scrub ok 2024-03-11T21:20:30.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:29 smithi023 bash[29146]: cluster 2024-03-11T21:20:28.602774+0000 mgr.y (mgr.24380) 1093 : cluster 0 pgmap v1266: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:20:30.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:29 smithi023 bash[29146]: cluster 2024-03-11T21:20:28.610919+0000 osd.3 (osd.3) 664 : cluster 0 6.6 scrub starts 2024-03-11T21:20:30.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:29 smithi023 bash[29146]: cluster 2024-03-11T21:20:28.643779+0000 osd.3 (osd.3) 665 : cluster 0 6.6 scrub ok 2024-03-11T21:20:30.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:29 smithi023 bash[29146]: cluster 2024-03-11T21:20:28.767052+0000 osd.2 (osd.2) 325 : cluster 0 2.c scrub starts 2024-03-11T21:20:30.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:29 smithi023 bash[29146]: cluster 2024-03-11T21:20:28.768733+0000 osd.2 (osd.2) 326 : cluster 0 2.c scrub ok 2024-03-11T21:20:30.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:29 smithi023 bash[29146]: cluster 2024-03-11T21:20:28.916633+0000 osd.6 (osd.6) 578 : cluster 0 4.1f scrub starts 2024-03-11T21:20:30.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:29 smithi023 bash[29146]: cluster 2024-03-11T21:20:28.919498+0000 osd.6 (osd.6) 579 : cluster 0 4.1f scrub ok 2024-03-11T21:20:30.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:29 smithi023 bash[29146]: cluster 2024-03-11T21:20:28.974428+0000 osd.1 (osd.1) 472 : cluster 0 3.4 scrub starts 2024-03-11T21:20:30.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:29 smithi023 bash[29146]: cluster 2024-03-11T21:20:28.978762+0000 osd.1 (osd.1) 473 : cluster 0 3.4 scrub ok 2024-03-11T21:20:30.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:29 smithi023 bash[29146]: cluster 2024-03-11T21:20:29.494912+0000 osd.0 (osd.0) 536 : cluster 0 2.13 scrub starts 2024-03-11T21:20:30.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:29 smithi023 bash[29146]: cluster 2024-03-11T21:20:29.497254+0000 osd.0 (osd.0) 537 : cluster 0 2.13 scrub ok 2024-03-11T21:20:31.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:30 smithi146 bash[23809]: audit 2024-03-11T21:20:29.083326+0000 mgr.y (mgr.24380) 1094 : audit 0 from='client.25751 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:20:31.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:30 smithi146 bash[23809]: cluster 2024-03-11T21:20:29.414032+0000 osd.7 (osd.7) 733 : cluster 0 6.15 scrub starts 2024-03-11T21:20:31.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:30 smithi146 bash[23809]: cluster 2024-03-11T21:20:29.515328+0000 osd.7 (osd.7) 734 : cluster 0 6.15 scrub ok 2024-03-11T21:20:31.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:30 smithi146 bash[23809]: cluster 2024-03-11T21:20:29.644534+0000 osd.5 (osd.5) 711 : cluster 0 3.16 scrub starts 2024-03-11T21:20:31.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:30 smithi146 bash[23809]: cluster 2024-03-11T21:20:29.648474+0000 osd.5 (osd.5) 712 : cluster 0 3.16 scrub ok 2024-03-11T21:20:31.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:30 smithi146 bash[23809]: cluster 2024-03-11T21:20:29.725376+0000 osd.2 (osd.2) 327 : cluster 0 2.10 scrub starts 2024-03-11T21:20:31.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:30 smithi146 bash[23809]: cluster 2024-03-11T21:20:29.727917+0000 osd.2 (osd.2) 328 : cluster 0 2.10 scrub ok 2024-03-11T21:20:31.023 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:30 smithi023 bash[21372]: audit 2024-03-11T21:20:29.083326+0000 mgr.y (mgr.24380) 1094 : audit 0 from='client.25751 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:20:31.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:30 smithi023 bash[21372]: cluster 2024-03-11T21:20:29.414032+0000 osd.7 (osd.7) 733 : cluster 0 6.15 scrub starts 2024-03-11T21:20:31.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:30 smithi023 bash[21372]: cluster 2024-03-11T21:20:29.515328+0000 osd.7 (osd.7) 734 : cluster 0 6.15 scrub ok 2024-03-11T21:20:31.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:30 smithi023 bash[21372]: cluster 2024-03-11T21:20:29.644534+0000 osd.5 (osd.5) 711 : cluster 0 3.16 scrub starts 2024-03-11T21:20:31.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:30 smithi023 bash[21372]: cluster 2024-03-11T21:20:29.648474+0000 osd.5 (osd.5) 712 : cluster 0 3.16 scrub ok 2024-03-11T21:20:31.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:30 smithi023 bash[21372]: cluster 2024-03-11T21:20:29.725376+0000 osd.2 (osd.2) 327 : cluster 0 2.10 scrub starts 2024-03-11T21:20:31.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:30 smithi023 bash[21372]: cluster 2024-03-11T21:20:29.727917+0000 osd.2 (osd.2) 328 : cluster 0 2.10 scrub ok 2024-03-11T21:20:31.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:30 smithi023 bash[29146]: audit 2024-03-11T21:20:29.083326+0000 mgr.y (mgr.24380) 1094 : audit 0 from='client.25751 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:20:31.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:30 smithi023 bash[29146]: cluster 2024-03-11T21:20:29.414032+0000 osd.7 (osd.7) 733 : cluster 0 6.15 scrub starts 2024-03-11T21:20:31.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:30 smithi023 bash[29146]: cluster 2024-03-11T21:20:29.515328+0000 osd.7 (osd.7) 734 : cluster 0 6.15 scrub ok 2024-03-11T21:20:31.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:30 smithi023 bash[29146]: cluster 2024-03-11T21:20:29.644534+0000 osd.5 (osd.5) 711 : cluster 0 3.16 scrub starts 2024-03-11T21:20:31.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:30 smithi023 bash[29146]: cluster 2024-03-11T21:20:29.648474+0000 osd.5 (osd.5) 712 : cluster 0 3.16 scrub ok 2024-03-11T21:20:31.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:30 smithi023 bash[29146]: cluster 2024-03-11T21:20:29.725376+0000 osd.2 (osd.2) 327 : cluster 0 2.10 scrub starts 2024-03-11T21:20:31.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:30 smithi023 bash[29146]: cluster 2024-03-11T21:20:29.727917+0000 osd.2 (osd.2) 328 : cluster 0 2.10 scrub ok 2024-03-11T21:20:32.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:31 smithi146 bash[23809]: cluster 2024-03-11T21:20:30.603933+0000 mgr.y (mgr.24380) 1095 : cluster 0 pgmap v1267: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 885 B/s rd, 0 op/s 2024-03-11T21:20:32.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:31 smithi146 bash[23809]: cluster 2024-03-11T21:20:30.741286+0000 osd.2 (osd.2) 329 : cluster 0 2.1 scrub starts 2024-03-11T21:20:32.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:31 smithi146 bash[23809]: cluster 2024-03-11T21:20:30.743843+0000 osd.2 (osd.2) 330 : cluster 0 2.1 scrub ok 2024-03-11T21:20:32.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:31 smithi146 bash[23809]: cluster 2024-03-11T21:20:30.967756+0000 osd.1 (osd.1) 474 : cluster 0 6.4 scrub starts 2024-03-11T21:20:32.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:31 smithi146 bash[23809]: cluster 2024-03-11T21:20:30.970354+0000 osd.1 (osd.1) 475 : cluster 0 6.4 scrub ok 2024-03-11T21:20:32.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:31 smithi146 bash[23809]: cluster 2024-03-11T21:20:30.983000+0000 osd.6 (osd.6) 580 : cluster 0 5.2 scrub starts 2024-03-11T21:20:32.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:31 smithi146 bash[23809]: cluster 2024-03-11T21:20:31.030580+0000 osd.6 (osd.6) 581 : cluster 0 5.2 scrub ok 2024-03-11T21:20:32.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:31 smithi023 bash[21372]: cluster 2024-03-11T21:20:30.603933+0000 mgr.y (mgr.24380) 1095 : cluster 0 pgmap v1267: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 885 B/s rd, 0 op/s 2024-03-11T21:20:32.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:31 smithi023 bash[21372]: cluster 2024-03-11T21:20:30.741286+0000 osd.2 (osd.2) 329 : cluster 0 2.1 scrub starts 2024-03-11T21:20:32.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:31 smithi023 bash[21372]: cluster 2024-03-11T21:20:30.743843+0000 osd.2 (osd.2) 330 : cluster 0 2.1 scrub ok 2024-03-11T21:20:32.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:31 smithi023 bash[21372]: cluster 2024-03-11T21:20:30.967756+0000 osd.1 (osd.1) 474 : cluster 0 6.4 scrub starts 2024-03-11T21:20:32.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:31 smithi023 bash[21372]: cluster 2024-03-11T21:20:30.970354+0000 osd.1 (osd.1) 475 : cluster 0 6.4 scrub ok 2024-03-11T21:20:32.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:31 smithi023 bash[21372]: cluster 2024-03-11T21:20:30.983000+0000 osd.6 (osd.6) 580 : cluster 0 5.2 scrub starts 2024-03-11T21:20:32.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:31 smithi023 bash[21372]: cluster 2024-03-11T21:20:31.030580+0000 osd.6 (osd.6) 581 : cluster 0 5.2 scrub ok 2024-03-11T21:20:32.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:31 smithi023 bash[29146]: cluster 2024-03-11T21:20:30.603933+0000 mgr.y (mgr.24380) 1095 : cluster 0 pgmap v1267: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 885 B/s rd, 0 op/s 2024-03-11T21:20:32.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:31 smithi023 bash[29146]: cluster 2024-03-11T21:20:30.741286+0000 osd.2 (osd.2) 329 : cluster 0 2.1 scrub starts 2024-03-11T21:20:32.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:31 smithi023 bash[29146]: cluster 2024-03-11T21:20:30.743843+0000 osd.2 (osd.2) 330 : cluster 0 2.1 scrub ok 2024-03-11T21:20:32.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:31 smithi023 bash[29146]: cluster 2024-03-11T21:20:30.967756+0000 osd.1 (osd.1) 474 : cluster 0 6.4 scrub starts 2024-03-11T21:20:32.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:31 smithi023 bash[29146]: cluster 2024-03-11T21:20:30.970354+0000 osd.1 (osd.1) 475 : cluster 0 6.4 scrub ok 2024-03-11T21:20:32.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:31 smithi023 bash[29146]: cluster 2024-03-11T21:20:30.983000+0000 osd.6 (osd.6) 580 : cluster 0 5.2 scrub starts 2024-03-11T21:20:32.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:31 smithi023 bash[29146]: cluster 2024-03-11T21:20:31.030580+0000 osd.6 (osd.6) 581 : cluster 0 5.2 scrub ok 2024-03-11T21:20:33.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:32 smithi146 bash[23809]: cluster 2024-03-11T21:20:31.735819+0000 osd.4 (osd.4) 185 : cluster 0 4.d scrub starts 2024-03-11T21:20:33.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:32 smithi146 bash[23809]: cluster 2024-03-11T21:20:31.738772+0000 osd.4 (osd.4) 186 : cluster 0 4.d scrub ok 2024-03-11T21:20:33.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:32 smithi146 bash[23809]: cluster 2024-03-11T21:20:32.524243+0000 osd.0 (osd.0) 538 : cluster 0 2.1f scrub starts 2024-03-11T21:20:33.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:32 smithi146 bash[23809]: cluster 2024-03-11T21:20:32.526857+0000 osd.0 (osd.0) 539 : cluster 0 2.1f scrub ok 2024-03-11T21:20:33.023 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:32 smithi023 bash[21372]: cluster 2024-03-11T21:20:31.735819+0000 osd.4 (osd.4) 185 : cluster 0 4.d scrub starts 2024-03-11T21:20:33.023 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:32 smithi023 bash[21372]: cluster 2024-03-11T21:20:31.738772+0000 osd.4 (osd.4) 186 : cluster 0 4.d scrub ok 2024-03-11T21:20:33.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:32 smithi023 bash[21372]: cluster 2024-03-11T21:20:32.524243+0000 osd.0 (osd.0) 538 : cluster 0 2.1f scrub starts 2024-03-11T21:20:33.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:32 smithi023 bash[21372]: cluster 2024-03-11T21:20:32.526857+0000 osd.0 (osd.0) 539 : cluster 0 2.1f scrub ok 2024-03-11T21:20:33.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:32 smithi023 bash[29146]: cluster 2024-03-11T21:20:31.735819+0000 osd.4 (osd.4) 185 : cluster 0 4.d scrub starts 2024-03-11T21:20:33.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:32 smithi023 bash[29146]: cluster 2024-03-11T21:20:31.738772+0000 osd.4 (osd.4) 186 : cluster 0 4.d scrub ok 2024-03-11T21:20:33.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:32 smithi023 bash[29146]: cluster 2024-03-11T21:20:32.524243+0000 osd.0 (osd.0) 538 : cluster 0 2.1f scrub starts 2024-03-11T21:20:33.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:32 smithi023 bash[29146]: cluster 2024-03-11T21:20:32.526857+0000 osd.0 (osd.0) 539 : cluster 0 2.1f scrub ok 2024-03-11T21:20:34.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:33 smithi146 bash[23809]: cluster 2024-03-11T21:20:32.400279+0000 osd.7 (osd.7) 735 : cluster 0 1.9 deep-scrub starts 2024-03-11T21:20:34.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:33 smithi146 bash[23809]: cluster 2024-03-11T21:20:32.402856+0000 osd.7 (osd.7) 736 : cluster 0 1.9 deep-scrub ok 2024-03-11T21:20:34.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:33 smithi146 bash[23809]: cluster 2024-03-11T21:20:32.604834+0000 mgr.y (mgr.24380) 1096 : cluster 0 pgmap v1268: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:20:34.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:33 smithi146 bash[23809]: cluster 2024-03-11T21:20:32.655484+0000 osd.3 (osd.3) 666 : cluster 0 6.1f deep-scrub starts 2024-03-11T21:20:34.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:33 smithi146 bash[23809]: cluster 2024-03-11T21:20:32.863579+0000 osd.3 (osd.3) 667 : cluster 0 6.1f deep-scrub ok 2024-03-11T21:20:34.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:33 smithi146 bash[23809]: cluster 2024-03-11T21:20:32.950172+0000 osd.6 (osd.6) 582 : cluster 0 1.6 scrub starts 2024-03-11T21:20:34.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:33 smithi146 bash[23809]: cluster 2024-03-11T21:20:32.952711+0000 osd.6 (osd.6) 583 : cluster 0 1.6 scrub ok 2024-03-11T21:20:34.023 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:33 smithi023 bash[21372]: cluster 2024-03-11T21:20:32.400279+0000 osd.7 (osd.7) 735 : cluster 0 1.9 deep-scrub starts 2024-03-11T21:20:34.023 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:33 smithi023 bash[21372]: cluster 2024-03-11T21:20:32.402856+0000 osd.7 (osd.7) 736 : cluster 0 1.9 deep-scrub ok 2024-03-11T21:20:34.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:33 smithi023 bash[21372]: cluster 2024-03-11T21:20:32.604834+0000 mgr.y (mgr.24380) 1096 : cluster 0 pgmap v1268: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:20:34.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:33 smithi023 bash[21372]: cluster 2024-03-11T21:20:32.655484+0000 osd.3 (osd.3) 666 : cluster 0 6.1f deep-scrub starts 2024-03-11T21:20:34.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:33 smithi023 bash[21372]: cluster 2024-03-11T21:20:32.863579+0000 osd.3 (osd.3) 667 : cluster 0 6.1f deep-scrub ok 2024-03-11T21:20:34.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:33 smithi023 bash[21372]: cluster 2024-03-11T21:20:32.950172+0000 osd.6 (osd.6) 582 : cluster 0 1.6 scrub starts 2024-03-11T21:20:34.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:33 smithi023 bash[21372]: cluster 2024-03-11T21:20:32.952711+0000 osd.6 (osd.6) 583 : cluster 0 1.6 scrub ok 2024-03-11T21:20:34.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:33 smithi023 bash[29146]: cluster 2024-03-11T21:20:32.400279+0000 osd.7 (osd.7) 735 : cluster 0 1.9 deep-scrub starts 2024-03-11T21:20:34.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:33 smithi023 bash[29146]: cluster 2024-03-11T21:20:32.402856+0000 osd.7 (osd.7) 736 : cluster 0 1.9 deep-scrub ok 2024-03-11T21:20:34.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:33 smithi023 bash[29146]: cluster 2024-03-11T21:20:32.604834+0000 mgr.y (mgr.24380) 1096 : cluster 0 pgmap v1268: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:20:34.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:33 smithi023 bash[29146]: cluster 2024-03-11T21:20:32.655484+0000 osd.3 (osd.3) 666 : cluster 0 6.1f deep-scrub starts 2024-03-11T21:20:34.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:33 smithi023 bash[29146]: cluster 2024-03-11T21:20:32.863579+0000 osd.3 (osd.3) 667 : cluster 0 6.1f deep-scrub ok 2024-03-11T21:20:34.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:33 smithi023 bash[29146]: cluster 2024-03-11T21:20:32.950172+0000 osd.6 (osd.6) 582 : cluster 0 1.6 scrub starts 2024-03-11T21:20:34.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:33 smithi023 bash[29146]: cluster 2024-03-11T21:20:32.952711+0000 osd.6 (osd.6) 583 : cluster 0 1.6 scrub ok 2024-03-11T21:20:34.629 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:20:35.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:34 smithi146 bash[23809]: cluster 2024-03-11T21:20:33.730393+0000 osd.4 (osd.4) 187 : cluster 0 4.c scrub starts 2024-03-11T21:20:35.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:34 smithi146 bash[23809]: cluster 2024-03-11T21:20:33.733033+0000 osd.4 (osd.4) 188 : cluster 0 4.c scrub ok 2024-03-11T21:20:35.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:34 smithi146 bash[23809]: cluster 2024-03-11T21:20:34.529023+0000 osd.0 (osd.0) 540 : cluster 0 3.12 scrub starts 2024-03-11T21:20:35.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:34 smithi146 bash[23809]: cluster 2024-03-11T21:20:34.533339+0000 osd.0 (osd.0) 541 : cluster 0 3.12 scrub ok 2024-03-11T21:20:35.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:34 smithi146 bash[23809]: cluster 2024-03-11T21:20:34.607152+0000 osd.5 (osd.5) 713 : cluster 0 1.8 scrub starts 2024-03-11T21:20:35.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:34 smithi146 bash[23809]: cluster 2024-03-11T21:20:34.610349+0000 osd.5 (osd.5) 714 : cluster 0 1.8 scrub ok 2024-03-11T21:20:35.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:34 smithi023 bash[29146]: cluster 2024-03-11T21:20:33.730393+0000 osd.4 (osd.4) 187 : cluster 0 4.c scrub starts 2024-03-11T21:20:35.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:34 smithi023 bash[29146]: cluster 2024-03-11T21:20:33.733033+0000 osd.4 (osd.4) 188 : cluster 0 4.c scrub ok 2024-03-11T21:20:35.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:34 smithi023 bash[29146]: cluster 2024-03-11T21:20:34.529023+0000 osd.0 (osd.0) 540 : cluster 0 3.12 scrub starts 2024-03-11T21:20:35.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:34 smithi023 bash[29146]: cluster 2024-03-11T21:20:34.533339+0000 osd.0 (osd.0) 541 : cluster 0 3.12 scrub ok 2024-03-11T21:20:35.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:34 smithi023 bash[29146]: cluster 2024-03-11T21:20:34.607152+0000 osd.5 (osd.5) 713 : cluster 0 1.8 scrub starts 2024-03-11T21:20:35.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:34 smithi023 bash[29146]: cluster 2024-03-11T21:20:34.610349+0000 osd.5 (osd.5) 714 : cluster 0 1.8 scrub ok 2024-03-11T21:20:35.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:34 smithi023 bash[21372]: cluster 2024-03-11T21:20:33.730393+0000 osd.4 (osd.4) 187 : cluster 0 4.c scrub starts 2024-03-11T21:20:35.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:34 smithi023 bash[21372]: cluster 2024-03-11T21:20:33.733033+0000 osd.4 (osd.4) 188 : cluster 0 4.c scrub ok 2024-03-11T21:20:35.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:34 smithi023 bash[21372]: cluster 2024-03-11T21:20:34.529023+0000 osd.0 (osd.0) 540 : cluster 0 3.12 scrub starts 2024-03-11T21:20:35.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:34 smithi023 bash[21372]: cluster 2024-03-11T21:20:34.533339+0000 osd.0 (osd.0) 541 : cluster 0 3.12 scrub ok 2024-03-11T21:20:35.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:34 smithi023 bash[21372]: cluster 2024-03-11T21:20:34.607152+0000 osd.5 (osd.5) 713 : cluster 0 1.8 scrub starts 2024-03-11T21:20:35.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:34 smithi023 bash[21372]: cluster 2024-03-11T21:20:34.610349+0000 osd.5 (osd.5) 714 : cluster 0 1.8 scrub ok 2024-03-11T21:20:36.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:35 smithi146 bash[23809]: cluster 2024-03-11T21:20:34.605626+0000 mgr.y (mgr.24380) 1097 : cluster 0 pgmap v1269: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:20:36.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:35 smithi146 bash[23809]: cluster 2024-03-11T21:20:34.682723+0000 osd.4 (osd.4) 189 : cluster 0 4.9 scrub starts 2024-03-11T21:20:36.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:35 smithi146 bash[23809]: cluster 2024-03-11T21:20:34.685656+0000 osd.4 (osd.4) 190 : cluster 0 4.9 scrub ok 2024-03-11T21:20:36.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:35 smithi146 bash[23809]: cluster 2024-03-11T21:20:35.506833+0000 osd.0 (osd.0) 542 : cluster 0 4.b scrub starts 2024-03-11T21:20:36.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:35 smithi146 bash[23809]: cluster 2024-03-11T21:20:35.508727+0000 osd.0 (osd.0) 543 : cluster 0 4.b scrub ok 2024-03-11T21:20:36.017 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:35 smithi023 bash[21372]: cluster 2024-03-11T21:20:34.605626+0000 mgr.y (mgr.24380) 1097 : cluster 0 pgmap v1269: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:20:36.017 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:35 smithi023 bash[21372]: cluster 2024-03-11T21:20:34.682723+0000 osd.4 (osd.4) 189 : cluster 0 4.9 scrub starts 2024-03-11T21:20:36.017 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:35 smithi023 bash[21372]: cluster 2024-03-11T21:20:34.685656+0000 osd.4 (osd.4) 190 : cluster 0 4.9 scrub ok 2024-03-11T21:20:36.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:35 smithi023 bash[21372]: cluster 2024-03-11T21:20:35.506833+0000 osd.0 (osd.0) 542 : cluster 0 4.b scrub starts 2024-03-11T21:20:36.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:35 smithi023 bash[21372]: cluster 2024-03-11T21:20:35.508727+0000 osd.0 (osd.0) 543 : cluster 0 4.b scrub ok 2024-03-11T21:20:36.018 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:35 smithi023 bash[29146]: cluster 2024-03-11T21:20:34.605626+0000 mgr.y (mgr.24380) 1097 : cluster 0 pgmap v1269: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:20:36.018 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:35 smithi023 bash[29146]: cluster 2024-03-11T21:20:34.682723+0000 osd.4 (osd.4) 189 : cluster 0 4.9 scrub starts 2024-03-11T21:20:36.018 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:35 smithi023 bash[29146]: cluster 2024-03-11T21:20:34.685656+0000 osd.4 (osd.4) 190 : cluster 0 4.9 scrub ok 2024-03-11T21:20:36.018 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:35 smithi023 bash[29146]: cluster 2024-03-11T21:20:35.506833+0000 osd.0 (osd.0) 542 : cluster 0 4.b scrub starts 2024-03-11T21:20:36.018 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:35 smithi023 bash[29146]: cluster 2024-03-11T21:20:35.508727+0000 osd.0 (osd.0) 543 : cluster 0 4.b scrub ok 2024-03-11T21:20:36.751 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:20:36 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:20:37.023 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:36 smithi023 bash[29146]: cluster 2024-03-11T21:20:35.640020+0000 osd.3 (osd.3) 668 : cluster 0 6.b scrub starts 2024-03-11T21:20:37.023 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:36 smithi023 bash[29146]: cluster 2024-03-11T21:20:35.846326+0000 osd.3 (osd.3) 669 : cluster 0 6.b scrub ok 2024-03-11T21:20:37.023 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:36 smithi023 bash[29146]: cluster 2024-03-11T21:20:35.992139+0000 osd.1 (osd.1) 476 : cluster 0 4.f scrub starts 2024-03-11T21:20:37.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:36 smithi023 bash[29146]: cluster 2024-03-11T21:20:35.993797+0000 osd.1 (osd.1) 477 : cluster 0 4.f scrub ok 2024-03-11T21:20:37.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:36 smithi023 bash[21372]: cluster 2024-03-11T21:20:35.640020+0000 osd.3 (osd.3) 668 : cluster 0 6.b scrub starts 2024-03-11T21:20:37.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:36 smithi023 bash[21372]: cluster 2024-03-11T21:20:35.846326+0000 osd.3 (osd.3) 669 : cluster 0 6.b scrub ok 2024-03-11T21:20:37.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:36 smithi023 bash[21372]: cluster 2024-03-11T21:20:35.992139+0000 osd.1 (osd.1) 476 : cluster 0 4.f scrub starts 2024-03-11T21:20:37.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:36 smithi023 bash[21372]: cluster 2024-03-11T21:20:35.993797+0000 osd.1 (osd.1) 477 : cluster 0 4.f scrub ok 2024-03-11T21:20:37.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:36 smithi146 bash[23809]: cluster 2024-03-11T21:20:35.640020+0000 osd.3 (osd.3) 668 : cluster 0 6.b scrub starts 2024-03-11T21:20:37.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:36 smithi146 bash[23809]: cluster 2024-03-11T21:20:35.846326+0000 osd.3 (osd.3) 669 : cluster 0 6.b scrub ok 2024-03-11T21:20:37.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:36 smithi146 bash[23809]: cluster 2024-03-11T21:20:35.992139+0000 osd.1 (osd.1) 476 : cluster 0 4.f scrub starts 2024-03-11T21:20:37.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:36 smithi146 bash[23809]: cluster 2024-03-11T21:20:35.993797+0000 osd.1 (osd.1) 477 : cluster 0 4.f scrub ok 2024-03-11T21:20:38.023 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:37 smithi023 bash[29146]: audit 2024-03-11T21:20:36.378953+0000 mgr.y (mgr.24380) 1098 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:20:38.023 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:37 smithi023 bash[29146]: cluster 2024-03-11T21:20:36.598315+0000 osd.3 (osd.3) 670 : cluster 0 3.7 scrub starts 2024-03-11T21:20:38.023 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:37 smithi023 bash[29146]: cluster 2024-03-11T21:20:36.602578+0000 osd.3 (osd.3) 671 : cluster 0 3.7 scrub ok 2024-03-11T21:20:38.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:37 smithi023 bash[29146]: cluster 2024-03-11T21:20:36.607154+0000 mgr.y (mgr.24380) 1099 : cluster 0 pgmap v1270: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 3.0 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-11T21:20:38.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:37 smithi023 bash[29146]: cluster 2024-03-11T21:20:36.745083+0000 osd.4 (osd.4) 191 : cluster 0 4.18 scrub starts 2024-03-11T21:20:38.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:37 smithi023 bash[29146]: cluster 2024-03-11T21:20:36.747392+0000 osd.4 (osd.4) 192 : cluster 0 4.18 scrub ok 2024-03-11T21:20:38.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:37 smithi023 bash[29146]: audit 2024-03-11T21:20:37.219107+0000 mon.a (mon.0) 2126 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:20:38.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:37 smithi023 bash[21372]: audit 2024-03-11T21:20:36.378953+0000 mgr.y (mgr.24380) 1098 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:20:38.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:37 smithi023 bash[21372]: cluster 2024-03-11T21:20:36.598315+0000 osd.3 (osd.3) 670 : cluster 0 3.7 scrub starts 2024-03-11T21:20:38.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:37 smithi023 bash[21372]: cluster 2024-03-11T21:20:36.602578+0000 osd.3 (osd.3) 671 : cluster 0 3.7 scrub ok 2024-03-11T21:20:38.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:37 smithi023 bash[21372]: cluster 2024-03-11T21:20:36.607154+0000 mgr.y (mgr.24380) 1099 : cluster 0 pgmap v1270: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 3.0 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-11T21:20:38.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:37 smithi023 bash[21372]: cluster 2024-03-11T21:20:36.745083+0000 osd.4 (osd.4) 191 : cluster 0 4.18 scrub starts 2024-03-11T21:20:38.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:37 smithi023 bash[21372]: cluster 2024-03-11T21:20:36.747392+0000 osd.4 (osd.4) 192 : cluster 0 4.18 scrub ok 2024-03-11T21:20:38.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:37 smithi023 bash[21372]: audit 2024-03-11T21:20:37.219107+0000 mon.a (mon.0) 2126 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:20:38.028 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-11T21:20:38.034 INFO:teuthology.orchestra.run.smithi023.stderr:dumped all 2024-03-11T21:20:38.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:37 smithi146 bash[23809]: audit 2024-03-11T21:20:36.378953+0000 mgr.y (mgr.24380) 1098 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:20:38.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:37 smithi146 bash[23809]: cluster 2024-03-11T21:20:36.598315+0000 osd.3 (osd.3) 670 : cluster 0 3.7 scrub starts 2024-03-11T21:20:38.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:37 smithi146 bash[23809]: cluster 2024-03-11T21:20:36.602578+0000 osd.3 (osd.3) 671 : cluster 0 3.7 scrub ok 2024-03-11T21:20:38.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:37 smithi146 bash[23809]: cluster 2024-03-11T21:20:36.607154+0000 mgr.y (mgr.24380) 1099 : cluster 0 pgmap v1270: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 3.0 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-11T21:20:38.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:37 smithi146 bash[23809]: cluster 2024-03-11T21:20:36.745083+0000 osd.4 (osd.4) 191 : cluster 0 4.18 scrub starts 2024-03-11T21:20:38.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:37 smithi146 bash[23809]: cluster 2024-03-11T21:20:36.747392+0000 osd.4 (osd.4) 192 : cluster 0 4.18 scrub ok 2024-03-11T21:20:38.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:37 smithi146 bash[23809]: audit 2024-03-11T21:20:37.219107+0000 mon.a (mon.0) 2126 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:20:38.757 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:20:38 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:20:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:20:38.880 INFO:teuthology.orchestra.run.smithi023.stdout:{"pg_ready":true,"pg_map":{"version":1270,"stamp":"2024-03-11T21:20:36.605876+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596392,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":3251,"num_read_kb":3003,"num_write":1022,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":281,"num_bytes_recovered":6634,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3989,"ondisk_log_size":3989,"up":456,"acting":456,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":456,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":4500476,"kb_used_data":7676,"kb_used_omap":21,"kb_used_meta":4492714,"kb_avail":745460740,"statfs":{"total":767960285184,"available":763351797760,"internally_reserved":0,"allocated":7860224,"data_stored":5234128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22089,"internal_metadata":4600539575},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":36,"num_read_kb":36,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006604"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":938,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.050640+0000","last_change":"2024-03-11T21:20:02.741954+0000","last_active":"2024-03-11T21:20:19.050640+0000","last_peered":"2024-03-11T21:20:19.050640+0000","last_clean":"2024-03-11T21:20:19.050640+0000","last_became_active":"2024-03-11T20:50:34.338390+0000","last_became_peered":"2024-03-11T20:50:34.338390+0000","last_unstale":"2024-03-11T21:20:19.050640+0000","last_undegraded":"2024-03-11T21:20:19.050640+0000","last_fullsized":"2024-03-11T21:20:19.050640+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:20:02.741789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:20:02.741789+0000","last_clean_scrub_stamp":"2024-03-11T21:20:02.741789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:03.379591+0000","scrub_duration":211,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":954,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.050468+0000","last_change":"2024-03-11T21:19:28.470700+0000","last_active":"2024-03-11T21:20:19.050468+0000","last_peered":"2024-03-11T21:20:19.050468+0000","last_clean":"2024-03-11T21:20:19.050468+0000","last_became_active":"2024-03-11T21:14:50.707251+0000","last_became_peered":"2024-03-11T21:14:50.707251+0000","last_unstale":"2024-03-11T21:20:19.050468+0000","last_undegraded":"2024-03-11T21:20:19.050468+0000","last_fullsized":"2024-03-11T21:20:19.050468+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:19:28.470554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:19:28.470554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:52.566247+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":190,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:32.527019+0000","last_change":"2024-03-11T21:20:32.527077+0000","last_active":"2024-03-11T21:20:32.527019+0000","last_peered":"2024-03-11T21:20:32.527019+0000","last_clean":"2024-03-11T21:20:32.527019+0000","last_became_active":"2024-03-11T21:14:50.431630+0000","last_became_peered":"2024-03-11T21:14:50.431630+0000","last_unstale":"2024-03-11T21:20:32.527019+0000","last_undegraded":"2024-03-11T21:20:32.527019+0000","last_fullsized":"2024-03-11T21:20:32.527019+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:20:32.526867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:11:05.884900+0000","last_clean_scrub_stamp":"2024-03-11T21:20:32.526867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:57.491573+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"358'30","reported_seq":994,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.052515+0000","last_change":"2024-03-11T21:19:12.650724+0000","last_active":"2024-03-11T21:20:19.052515+0000","last_peered":"2024-03-11T21:20:19.052515+0000","last_clean":"2024-03-11T21:20:19.052515+0000","last_became_active":"2024-03-11T20:50:30.101721+0000","last_became_peered":"2024-03-11T20:50:30.101721+0000","last_unstale":"2024-03-11T21:20:19.052515+0000","last_undegraded":"2024-03-11T21:20:19.052515+0000","last_fullsized":"2024-03-11T21:20:19.052515+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'30","last_scrub_stamp":"2024-03-11T21:19:12.650411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:19:12.650411+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:33.733558+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":474,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.266189+0000","last_change":"2024-03-11T21:19:30.434951+0000","last_active":"2024-03-11T21:20:19.266189+0000","last_peered":"2024-03-11T21:20:19.266189+0000","last_clean":"2024-03-11T21:20:19.266189+0000","last_became_active":"2024-03-11T21:18:53.662597+0000","last_became_peered":"2024-03-11T21:18:53.662597+0000","last_unstale":"2024-03-11T21:20:19.266189+0000","last_undegraded":"2024-03-11T21:20:19.266189+0000","last_fullsized":"2024-03-11T21:20:19.266189+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:19:30.434749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:55.545758+0000","last_clean_scrub_stamp":"2024-03-11T21:19:30.434749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:40.186016+0000","scrub_duration":71,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":488,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.265153+0000","last_change":"2024-03-11T21:19:27.324018+0000","last_active":"2024-03-11T21:20:19.265153+0000","last_peered":"2024-03-11T21:20:19.265153+0000","last_clean":"2024-03-11T21:20:19.265153+0000","last_became_active":"2024-03-11T21:18:53.658414+0000","last_became_peered":"2024-03-11T21:18:53.658414+0000","last_unstale":"2024-03-11T21:20:19.265153+0000","last_undegraded":"2024-03-11T21:20:19.265153+0000","last_fullsized":"2024-03-11T21:20:19.265153+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:19:27.323785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:54:13.660325+0000","last_clean_scrub_stamp":"2024-03-11T21:19:27.323785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:36.393830+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":957,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.052343+0000","last_change":"2024-03-11T21:19:21.518793+0000","last_active":"2024-03-11T21:20:19.052343+0000","last_peered":"2024-03-11T21:20:19.052343+0000","last_clean":"2024-03-11T21:20:19.052343+0000","last_became_active":"2024-03-11T21:14:50.435212+0000","last_became_peered":"2024-03-11T21:14:50.435212+0000","last_unstale":"2024-03-11T21:20:19.052343+0000","last_undegraded":"2024-03-11T21:20:19.052343+0000","last_fullsized":"2024-03-11T21:20:19.052343+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:19:21.518361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:19:21.518361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"358'27","reported_seq":202,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.265233+0000","last_change":"2024-03-11T21:19:47.699324+0000","last_active":"2024-03-11T21:20:19.265233+0000","last_peered":"2024-03-11T21:20:19.265233+0000","last_clean":"2024-03-11T21:20:19.265233+0000","last_became_active":"2024-03-11T21:14:50.704536+0000","last_became_peered":"2024-03-11T21:14:50.704536+0000","last_unstale":"2024-03-11T21:20:19.265233+0000","last_undegraded":"2024-03-11T21:20:19.265233+0000","last_fullsized":"2024-03-11T21:20:19.265233+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'27","last_scrub_stamp":"2024-03-11T21:19:47.699225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:19:47.699225+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:06.025650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":983,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.717256+0000","last_change":"2024-03-11T21:20:19.717374+0000","last_active":"2024-03-11T21:20:19.717256+0000","last_peered":"2024-03-11T21:20:19.717256+0000","last_clean":"2024-03-11T21:20:19.717256+0000","last_became_active":"2024-03-11T21:00:48.460694+0000","last_became_peered":"2024-03-11T21:00:48.460694+0000","last_unstale":"2024-03-11T21:20:19.717256+0000","last_undegraded":"2024-03-11T21:20:19.717256+0000","last_fullsized":"2024-03-11T21:20:19.717256+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:20:19.717024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:41.711864+0000","last_clean_scrub_stamp":"2024-03-11T21:20:19.717024+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:29.793290+0000","scrub_duration":119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":470,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.266322+0000","last_change":"2024-03-11T21:20:07.529538+0000","last_active":"2024-03-11T21:20:19.266322+0000","last_peered":"2024-03-11T21:20:19.266322+0000","last_clean":"2024-03-11T21:20:19.266322+0000","last_became_active":"2024-03-11T21:18:53.658449+0000","last_became_peered":"2024-03-11T21:18:53.658449+0000","last_unstale":"2024-03-11T21:20:19.266322+0000","last_undegraded":"2024-03-11T21:20:19.266322+0000","last_fullsized":"2024-03-11T21:20:19.266322+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:20:07.529314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:16:51.642949+0000","last_clean_scrub_stamp":"2024-03-11T21:20:07.529314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:29.676605+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":991,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.039988+0000","last_change":"2024-03-11T21:19:34.216253+0000","last_active":"2024-03-11T21:20:19.039988+0000","last_peered":"2024-03-11T21:20:19.039988+0000","last_clean":"2024-03-11T21:20:19.039988+0000","last_became_active":"2024-03-11T21:14:50.446747+0000","last_became_peered":"2024-03-11T21:14:50.446747+0000","last_unstale":"2024-03-11T21:20:19.039988+0000","last_undegraded":"2024-03-11T21:20:19.039988+0000","last_fullsized":"2024-03-11T21:20:19.039988+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:19:34.216030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:14.081581+0000","last_clean_scrub_stamp":"2024-03-11T21:19:34.216030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:48.045501+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"358'39","reported_seq":1104,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.070224+0000","last_change":"2024-03-11T21:20:04.603226+0000","last_active":"2024-03-11T21:20:19.070224+0000","last_peered":"2024-03-11T21:20:19.070224+0000","last_clean":"2024-03-11T21:20:19.070224+0000","last_became_active":"2024-03-11T21:00:48.462943+0000","last_became_peered":"2024-03-11T21:00:48.462943+0000","last_unstale":"2024-03-11T21:20:19.070224+0000","last_undegraded":"2024-03-11T21:20:19.070224+0000","last_fullsized":"2024-03-11T21:20:19.070224+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'39","last_scrub_stamp":"2024-03-11T21:20:04.602978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:20:04.602978+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:07.921975+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":190,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:23.597768+0000","last_change":"2024-03-11T21:20:23.597814+0000","last_active":"2024-03-11T21:20:23.597768+0000","last_peered":"2024-03-11T21:20:23.597768+0000","last_clean":"2024-03-11T21:20:23.597768+0000","last_became_active":"2024-03-11T21:14:50.430442+0000","last_became_peered":"2024-03-11T21:14:50.430442+0000","last_unstale":"2024-03-11T21:20:23.597768+0000","last_undegraded":"2024-03-11T21:20:23.597768+0000","last_fullsized":"2024-03-11T21:20:23.597768+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:20:23.597614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:20:23.597614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:37.477258+0000","scrub_duration":83,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":601,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.265442+0000","last_change":"2024-03-11T21:19:56.412872+0000","last_active":"2024-03-11T21:20:19.265442+0000","last_peered":"2024-03-11T21:20:19.265442+0000","last_clean":"2024-03-11T21:20:19.265442+0000","last_became_active":"2024-03-11T21:18:53.654504+0000","last_became_peered":"2024-03-11T21:18:53.654504+0000","last_unstale":"2024-03-11T21:20:19.265442+0000","last_undegraded":"2024-03-11T21:20:19.265442+0000","last_fullsized":"2024-03-11T21:20:19.265442+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:19:56.412634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:19:56.412634+0000","last_clean_scrub_stamp":"2024-03-11T21:19:56.412634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:17.957822+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":987,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.039193+0000","last_change":"2024-03-11T21:19:29.250554+0000","last_active":"2024-03-11T21:20:19.039193+0000","last_peered":"2024-03-11T21:20:19.039193+0000","last_clean":"2024-03-11T21:20:19.039193+0000","last_became_active":"2024-03-11T20:50:28.068978+0000","last_became_peered":"2024-03-11T20:50:28.068978+0000","last_unstale":"2024-03-11T21:20:19.039193+0000","last_undegraded":"2024-03-11T21:20:19.039193+0000","last_fullsized":"2024-03-11T21:20:19.039193+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:19:29.250337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:54:07.155130+0000","last_clean_scrub_stamp":"2024-03-11T21:19:29.250337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:43.643589+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"358'29","reported_seq":1033,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.070050+0000","last_change":"2024-03-11T21:19:56.626174+0000","last_active":"2024-03-11T21:20:19.070050+0000","last_peered":"2024-03-11T21:20:19.070050+0000","last_clean":"2024-03-11T21:20:19.070050+0000","last_became_active":"2024-03-11T21:00:27.952093+0000","last_became_peered":"2024-03-11T21:00:27.952093+0000","last_unstale":"2024-03-11T21:20:19.070050+0000","last_undegraded":"2024-03-11T21:20:19.070050+0000","last_fullsized":"2024-03-11T21:20:19.070050+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'29","last_scrub_stamp":"2024-03-11T21:19:56.625921+0000","last_deep_scrub":"110'17","last_deep_scrub_stamp":"2024-03-11T20:53:04.802488+0000","last_clean_scrub_stamp":"2024-03-11T21:19:56.625921+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:12.539568+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":940,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:32.863679+0000","last_change":"2024-03-11T21:20:32.863711+0000","last_active":"2024-03-11T21:20:32.863679+0000","last_peered":"2024-03-11T21:20:32.863679+0000","last_clean":"2024-03-11T21:20:32.863679+0000","last_became_active":"2024-03-11T20:50:34.334558+0000","last_became_peered":"2024-03-11T20:50:34.334558+0000","last_unstale":"2024-03-11T21:20:32.863679+0000","last_undegraded":"2024-03-11T21:20:32.863679+0000","last_fullsized":"2024-03-11T21:20:32.863679+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:20:32.863584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:20:32.863584+0000","last_clean_scrub_stamp":"2024-03-11T21:20:32.863584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:56.489535+0000","scrub_duration":211,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":324,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.051154+0000","last_change":"2024-03-11T21:20:13.552271+0000","last_active":"2024-03-11T21:20:19.051154+0000","last_peered":"2024-03-11T21:20:19.051154+0000","last_clean":"2024-03-11T21:20:19.051154+0000","last_became_active":"2024-03-11T21:10:38.441649+0000","last_became_peered":"2024-03-11T21:10:38.441649+0000","last_unstale":"2024-03-11T21:20:19.051154+0000","last_undegraded":"2024-03-11T21:20:19.051154+0000","last_fullsized":"2024-03-11T21:20:19.051154+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:20:13.552119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:20:13.552119+0000","last_clean_scrub_stamp":"2024-03-11T21:20:13.552119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:18.473429+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"474'2","reported_seq":959,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.050876+0000","last_change":"2024-03-11T21:19:25.509890+0000","last_active":"2024-03-11T21:20:19.050876+0000","last_peered":"2024-03-11T21:20:19.050876+0000","last_clean":"2024-03-11T21:20:19.050876+0000","last_became_active":"2024-03-11T20:50:28.068840+0000","last_became_peered":"2024-03-11T20:50:28.068840+0000","last_unstale":"2024-03-11T21:20:19.050876+0000","last_undegraded":"2024-03-11T21:20:19.050876+0000","last_fullsized":"2024-03-11T21:20:19.050876+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"474'2","last_scrub_stamp":"2024-03-11T21:19:25.509656+0000","last_deep_scrub":"474'2","last_deep_scrub_stamp":"2024-03-11T21:19:25.509656+0000","last_clean_scrub_stamp":"2024-03-11T21:19:25.509656+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:49.390181+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"497'27","reported_seq":553,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:35.961361+0000","last_change":"2024-03-11T21:19:44.440292+0000","last_active":"2024-03-11T21:20:35.961361+0000","last_peered":"2024-03-11T21:20:35.961361+0000","last_clean":"2024-03-11T21:20:35.961361+0000","last_became_active":"2024-03-11T21:18:53.659269+0000","last_became_peered":"2024-03-11T21:18:53.659269+0000","last_unstale":"2024-03-11T21:20:35.961361+0000","last_undegraded":"2024-03-11T21:20:35.961361+0000","last_fullsized":"2024-03-11T21:20:35.961361+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"357'25","last_scrub_stamp":"2024-03-11T21:19:44.440056+0000","last_deep_scrub":"357'25","last_deep_scrub_stamp":"2024-03-11T21:18:07.923009+0000","last_clean_scrub_stamp":"2024-03-11T21:19:44.440056+0000","objects_scrubbed":4,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:47.852416+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":37,"num_read_kb":31,"num_write":24,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":469,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.267260+0000","last_change":"2024-03-11T21:19:21.380497+0000","last_active":"2024-03-11T21:20:19.267260+0000","last_peered":"2024-03-11T21:20:19.267260+0000","last_clean":"2024-03-11T21:20:19.267260+0000","last_became_active":"2024-03-11T21:18:53.662773+0000","last_became_peered":"2024-03-11T21:18:53.662773+0000","last_unstale":"2024-03-11T21:20:19.267260+0000","last_undegraded":"2024-03-11T21:20:19.267260+0000","last_fullsized":"2024-03-11T21:20:19.267260+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:19:21.380339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:16:30.676879+0000","last_clean_scrub_stamp":"2024-03-11T21:19:21.380339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:46.086669+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":699,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.076425+0000","last_change":"2024-03-11T21:20:15.913955+0000","last_active":"2024-03-11T21:20:19.076425+0000","last_peered":"2024-03-11T21:20:19.076425+0000","last_clean":"2024-03-11T21:20:19.076425+0000","last_became_active":"2024-03-11T21:17:19.512017+0000","last_became_peered":"2024-03-11T21:17:19.512017+0000","last_unstale":"2024-03-11T21:20:19.076425+0000","last_undegraded":"2024-03-11T21:20:19.076425+0000","last_fullsized":"2024-03-11T21:20:19.076425+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:20:15.913692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:19:08.327562+0000","last_clean_scrub_stamp":"2024-03-11T21:20:15.913692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:27.295349+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":994,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.082145+0000","last_change":"2024-03-11T21:19:32.653757+0000","last_active":"2024-03-11T21:20:19.082145+0000","last_peered":"2024-03-11T21:20:19.082145+0000","last_clean":"2024-03-11T21:20:19.082145+0000","last_became_active":"2024-03-11T21:00:27.956935+0000","last_became_peered":"2024-03-11T21:00:27.956935+0000","last_unstale":"2024-03-11T21:20:19.082145+0000","last_undegraded":"2024-03-11T21:20:19.082145+0000","last_fullsized":"2024-03-11T21:20:19.082145+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:19:32.653534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:17:15.682879+0000","last_clean_scrub_stamp":"2024-03-11T21:19:32.653534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:56.262588+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"494'11","reported_seq":198,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.264324+0000","last_change":"2024-03-11T21:20:00.693286+0000","last_active":"2024-03-11T21:20:19.264324+0000","last_peered":"2024-03-11T21:20:19.264324+0000","last_clean":"2024-03-11T21:20:19.264324+0000","last_became_active":"2024-03-11T21:14:50.430436+0000","last_became_peered":"2024-03-11T21:14:50.430436+0000","last_unstale":"2024-03-11T21:20:19.264324+0000","last_undegraded":"2024-03-11T21:20:19.264324+0000","last_fullsized":"2024-03-11T21:20:19.264324+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"494'11","last_scrub_stamp":"2024-03-11T21:20:00.693172+0000","last_deep_scrub":"494'11","last_deep_scrub_stamp":"2024-03-11T21:20:00.693172+0000","last_clean_scrub_stamp":"2024-03-11T21:20:00.693172+0000","objects_scrubbed":2,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:06.241988+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":16,"num_read_kb":12,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":681,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.046838+0000","last_change":"2024-03-11T21:19:36.241438+0000","last_active":"2024-03-11T21:20:19.046838+0000","last_peered":"2024-03-11T21:20:19.046838+0000","last_clean":"2024-03-11T21:20:19.046838+0000","last_became_active":"2024-03-11T21:00:48.464694+0000","last_became_peered":"2024-03-11T21:00:48.464694+0000","last_unstale":"2024-03-11T21:20:19.046838+0000","last_undegraded":"2024-03-11T21:20:19.046838+0000","last_fullsized":"2024-03-11T21:20:19.046838+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:19:36.241328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:19:36.241328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:01.965733+0000","scrub_duration":167,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1f","version":"212'16","reported_seq":1383,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:29.567033+0000","last_change":"2024-03-11T21:20:28.919830+0000","last_active":"2024-03-11T21:20:29.567033+0000","last_peered":"2024-03-11T21:20:29.567033+0000","last_clean":"2024-03-11T21:20:29.567033+0000","last_became_active":"2024-03-11T21:00:48.464350+0000","last_became_peered":"2024-03-11T21:00:48.464350+0000","last_unstale":"2024-03-11T21:20:29.567033+0000","last_undegraded":"2024-03-11T21:20:29.567033+0000","last_fullsized":"2024-03-11T21:20:29.567033+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"212'16","last_scrub_stamp":"2024-03-11T21:20:28.919512+0000","last_deep_scrub":"212'16","last_deep_scrub_stamp":"2024-03-11T21:06:47.821221+0000","last_clean_scrub_stamp":"2024-03-11T21:20:28.919512+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:32.248736+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"362'2","reported_seq":964,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.050683+0000","last_change":"2024-03-11T21:19:37.461036+0000","last_active":"2024-03-11T21:20:19.050683+0000","last_peered":"2024-03-11T21:20:19.050683+0000","last_clean":"2024-03-11T21:20:19.050683+0000","last_became_active":"2024-03-11T21:14:50.436938+0000","last_became_peered":"2024-03-11T21:14:50.436938+0000","last_unstale":"2024-03-11T21:20:19.050683+0000","last_undegraded":"2024-03-11T21:20:19.050683+0000","last_fullsized":"2024-03-11T21:20:19.050683+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"362'2","last_scrub_stamp":"2024-03-11T21:19:37.460804+0000","last_deep_scrub":"362'2","last_deep_scrub_stamp":"2024-03-11T21:11:03.482685+0000","last_clean_scrub_stamp":"2024-03-11T21:19:37.460804+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:50.526401+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"357'21","reported_seq":1054,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.049683+0000","last_change":"2024-03-11T21:19:56.495228+0000","last_active":"2024-03-11T21:20:19.049683+0000","last_peered":"2024-03-11T21:20:19.049683+0000","last_clean":"2024-03-11T21:20:19.049683+0000","last_became_active":"2024-03-11T21:14:50.437313+0000","last_became_peered":"2024-03-11T21:14:50.437313+0000","last_unstale":"2024-03-11T21:20:19.049683+0000","last_undegraded":"2024-03-11T21:20:19.049683+0000","last_fullsized":"2024-03-11T21:20:19.049683+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"357'21","last_scrub_stamp":"2024-03-11T21:19:56.494977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:19:56.494977+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:24.063441+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"105'2","reported_seq":970,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.039793+0000","last_change":"2024-03-11T21:19:09.270385+0000","last_active":"2024-03-11T21:20:19.039793+0000","last_peered":"2024-03-11T21:20:19.039793+0000","last_clean":"2024-03-11T21:20:19.039793+0000","last_became_active":"2024-03-11T20:50:34.327356+0000","last_became_peered":"2024-03-11T20:50:34.327356+0000","last_unstale":"2024-03-11T21:20:19.039793+0000","last_undegraded":"2024-03-11T21:20:19.039793+0000","last_fullsized":"2024-03-11T21:20:19.039793+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'2","last_scrub_stamp":"2024-03-11T21:19:09.270142+0000","last_deep_scrub":"105'2","last_deep_scrub_stamp":"2024-03-11T21:03:41.971918+0000","last_clean_scrub_stamp":"2024-03-11T21:19:09.270142+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":182,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.265006+0000","last_change":"2024-03-11T21:19:30.460578+0000","last_active":"2024-03-11T21:20:19.265006+0000","last_peered":"2024-03-11T21:20:19.265006+0000","last_clean":"2024-03-11T21:20:19.265006+0000","last_became_active":"2024-03-11T21:14:50.431785+0000","last_became_peered":"2024-03-11T21:14:50.431785+0000","last_unstale":"2024-03-11T21:20:19.265006+0000","last_undegraded":"2024-03-11T21:20:19.265006+0000","last_fullsized":"2024-03-11T21:20:19.265006+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:19:30.460241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:06:32.968650+0000","last_clean_scrub_stamp":"2024-03-11T21:19:30.460241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:36.454682+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":983,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.068722+0000","last_change":"2024-03-11T21:19:26.680850+0000","last_active":"2024-03-11T21:20:19.068722+0000","last_peered":"2024-03-11T21:20:19.068722+0000","last_clean":"2024-03-11T21:20:19.068722+0000","last_became_active":"2024-03-11T21:00:27.947614+0000","last_became_peered":"2024-03-11T21:00:27.947614+0000","last_unstale":"2024-03-11T21:20:19.068722+0000","last_undegraded":"2024-03-11T21:20:19.068722+0000","last_fullsized":"2024-03-11T21:20:19.068722+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:19:26.680583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:08:27.756036+0000","last_clean_scrub_stamp":"2024-03-11T21:19:26.680583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:50.445805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"358'39","reported_seq":729,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.045287+0000","last_change":"2024-03-11T21:19:53.198728+0000","last_active":"2024-03-11T21:20:19.045287+0000","last_peered":"2024-03-11T21:20:19.045287+0000","last_clean":"2024-03-11T21:20:19.045287+0000","last_became_active":"2024-03-11T21:00:48.457893+0000","last_became_peered":"2024-03-11T21:00:48.457893+0000","last_unstale":"2024-03-11T21:20:19.045287+0000","last_undegraded":"2024-03-11T21:20:19.045287+0000","last_fullsized":"2024-03-11T21:20:19.045287+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'39","last_scrub_stamp":"2024-03-11T21:19:53.198395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:19:53.198395+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:14.967931+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.14","version":"184'130","reported_seq":1057,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.075408+0000","last_change":"2024-03-11T21:19:39.039534+0000","last_active":"2024-03-11T21:20:19.075408+0000","last_peered":"2024-03-11T21:20:19.075408+0000","last_clean":"2024-03-11T21:20:19.075408+0000","last_became_active":"2024-03-11T21:17:19.512515+0000","last_became_peered":"2024-03-11T21:17:19.512515+0000","last_unstale":"2024-03-11T21:20:19.075408+0000","last_undegraded":"2024-03-11T21:20:19.075408+0000","last_fullsized":"2024-03-11T21:20:19.075408+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":444,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":5,"last_scrub":"184'130","last_scrub_stamp":"2024-03-11T21:19:39.039237+0000","last_deep_scrub":"184'130","last_deep_scrub_stamp":"2024-03-11T21:18:26.059235+0000","last_clean_scrub_stamp":"2024-03-11T21:19:39.039237+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:40.158443+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":950,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.051017+0000","last_change":"2024-03-11T21:19:13.801462+0000","last_active":"2024-03-11T21:20:19.051017+0000","last_peered":"2024-03-11T21:20:19.051017+0000","last_clean":"2024-03-11T21:20:19.051017+0000","last_became_active":"2024-03-11T21:14:50.438215+0000","last_became_peered":"2024-03-11T21:14:50.438215+0000","last_unstale":"2024-03-11T21:20:19.051017+0000","last_undegraded":"2024-03-11T21:20:19.051017+0000","last_fullsized":"2024-03-11T21:20:19.051017+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:19:13.801303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:52:11.462480+0000","last_clean_scrub_stamp":"2024-03-11T21:19:13.801303+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:35.890821+0000","scrub_duration":167,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":942,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.051105+0000","last_change":"2024-03-11T21:19:27.502595+0000","last_active":"2024-03-11T21:20:19.051105+0000","last_peered":"2024-03-11T21:20:19.051105+0000","last_clean":"2024-03-11T21:20:19.051105+0000","last_became_active":"2024-03-11T20:50:32.646323+0000","last_became_peered":"2024-03-11T20:50:32.646323+0000","last_unstale":"2024-03-11T21:20:19.051105+0000","last_undegraded":"2024-03-11T21:20:19.051105+0000","last_fullsized":"2024-03-11T21:20:19.051105+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:19:27.502382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:10:02.099653+0000","last_clean_scrub_stamp":"2024-03-11T21:19:27.502382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:48.649120+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":1001,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.083270+0000","last_change":"2024-03-11T21:19:20.564830+0000","last_active":"2024-03-11T21:20:19.083270+0000","last_peered":"2024-03-11T21:20:19.083270+0000","last_clean":"2024-03-11T21:20:19.083270+0000","last_became_active":"2024-03-11T20:50:28.086513+0000","last_became_peered":"2024-03-11T20:50:28.086513+0000","last_unstale":"2024-03-11T21:20:19.083270+0000","last_undegraded":"2024-03-11T21:20:19.083270+0000","last_fullsized":"2024-03-11T21:20:19.083270+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:19:20.564688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:19:20.564688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:34.169330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"357'25","reported_seq":1058,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:29.648659+0000","last_change":"2024-03-11T21:20:29.648736+0000","last_active":"2024-03-11T21:20:29.648659+0000","last_peered":"2024-03-11T21:20:29.648659+0000","last_clean":"2024-03-11T21:20:29.648659+0000","last_became_active":"2024-03-11T21:00:48.464707+0000","last_became_peered":"2024-03-11T21:00:48.464707+0000","last_unstale":"2024-03-11T21:20:29.648659+0000","last_undegraded":"2024-03-11T21:20:29.648659+0000","last_fullsized":"2024-03-11T21:20:29.648659+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"357'25","last_scrub_stamp":"2024-03-11T21:20:29.648487+0000","last_deep_scrub":"145'13","last_deep_scrub_stamp":"2024-03-11T20:55:53.560287+0000","last_clean_scrub_stamp":"2024-03-11T21:20:29.648487+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:32.244505+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":976,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:25.512177+0000","last_change":"2024-03-11T21:20:25.512241+0000","last_active":"2024-03-11T21:20:25.512177+0000","last_peered":"2024-03-11T21:20:25.512177+0000","last_clean":"2024-03-11T21:20:25.512177+0000","last_became_active":"2024-03-11T21:00:27.973405+0000","last_became_peered":"2024-03-11T21:00:27.973405+0000","last_unstale":"2024-03-11T21:20:25.512177+0000","last_undegraded":"2024-03-11T21:20:25.512177+0000","last_fullsized":"2024-03-11T21:20:25.512177+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:20:25.512001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:12:24.069900+0000","last_clean_scrub_stamp":"2024-03-11T21:20:25.512001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:42.656173+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":955,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.049736+0000","last_change":"2024-03-11T21:19:42.436445+0000","last_active":"2024-03-11T21:20:19.049736+0000","last_peered":"2024-03-11T21:20:19.049736+0000","last_clean":"2024-03-11T21:20:19.049736+0000","last_became_active":"2024-03-11T21:00:48.472812+0000","last_became_peered":"2024-03-11T21:00:48.472812+0000","last_unstale":"2024-03-11T21:20:19.049736+0000","last_undegraded":"2024-03-11T21:20:19.049736+0000","last_fullsized":"2024-03-11T21:20:19.049736+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:19:42.436153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:19:42.436153+0000","last_clean_scrub_stamp":"2024-03-11T21:19:42.436153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:01.371751+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":990,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:25.655410+0000","last_change":"2024-03-11T21:20:25.655469+0000","last_active":"2024-03-11T21:20:25.655410+0000","last_peered":"2024-03-11T21:20:25.655410+0000","last_clean":"2024-03-11T21:20:25.655410+0000","last_became_active":"2024-03-11T20:50:28.075760+0000","last_became_peered":"2024-03-11T20:50:28.075760+0000","last_unstale":"2024-03-11T21:20:25.655410+0000","last_undegraded":"2024-03-11T21:20:25.655410+0000","last_fullsized":"2024-03-11T21:20:25.655410+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:20:25.655266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:20:25.655266+0000","last_clean_scrub_stamp":"2024-03-11T21:20:25.655266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:51.410555+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"497'23","reported_seq":196,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:35.885665+0000","last_change":"2024-03-11T21:19:14.461073+0000","last_active":"2024-03-11T21:20:35.885665+0000","last_peered":"2024-03-11T21:20:35.885665+0000","last_clean":"2024-03-11T21:20:35.885665+0000","last_became_active":"2024-03-11T21:14:50.427230+0000","last_became_peered":"2024-03-11T21:14:50.427230+0000","last_unstale":"2024-03-11T21:20:35.885665+0000","last_undegraded":"2024-03-11T21:20:35.885665+0000","last_fullsized":"2024-03-11T21:20:35.885665+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"472'21","last_scrub_stamp":"2024-03-11T21:19:14.460869+0000","last_deep_scrub":"472'21","last_deep_scrub_stamp":"2024-03-11T21:17:46.445788+0000","last_clean_scrub_stamp":"2024-03-11T21:19:14.460869+0000","objects_scrubbed":3,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:44.390729+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":27,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.11","version":"186'2","reported_seq":960,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.049954+0000","last_change":"2024-03-11T21:19:18.567773+0000","last_active":"2024-03-11T21:20:19.049954+0000","last_peered":"2024-03-11T21:20:19.049954+0000","last_clean":"2024-03-11T21:20:19.049954+0000","last_became_active":"2024-03-11T21:14:50.436281+0000","last_became_peered":"2024-03-11T21:14:50.436281+0000","last_unstale":"2024-03-11T21:20:19.049954+0000","last_undegraded":"2024-03-11T21:20:19.049954+0000","last_fullsized":"2024-03-11T21:20:19.049954+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"186'2","last_scrub_stamp":"2024-03-11T21:19:18.567510+0000","last_deep_scrub":"186'2","last_deep_scrub_stamp":"2024-03-11T20:59:04.534850+0000","last_clean_scrub_stamp":"2024-03-11T21:19:18.567510+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:43.662594+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":480,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.265932+0000","last_change":"2024-03-11T21:19:48.465834+0000","last_active":"2024-03-11T21:20:19.265932+0000","last_peered":"2024-03-11T21:20:19.265932+0000","last_clean":"2024-03-11T21:20:19.265932+0000","last_became_active":"2024-03-11T21:18:53.659060+0000","last_became_peered":"2024-03-11T21:18:53.659060+0000","last_unstale":"2024-03-11T21:20:19.265932+0000","last_undegraded":"2024-03-11T21:20:19.265932+0000","last_fullsized":"2024-03-11T21:20:19.265932+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:19:48.465615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:19:48.465615+0000","last_clean_scrub_stamp":"2024-03-11T21:19:48.465615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:06.059928+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":693,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.046562+0000","last_change":"2024-03-11T21:19:50.117199+0000","last_active":"2024-03-11T21:20:19.046562+0000","last_peered":"2024-03-11T21:20:19.046562+0000","last_clean":"2024-03-11T21:20:19.046562+0000","last_became_active":"2024-03-11T21:14:50.419556+0000","last_became_peered":"2024-03-11T21:14:50.419556+0000","last_unstale":"2024-03-11T21:20:19.046562+0000","last_undegraded":"2024-03-11T21:20:19.046562+0000","last_fullsized":"2024-03-11T21:20:19.046562+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:19:50.116922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:19:50.116922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:57.501009+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"497'37","reported_seq":497,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:35.894397+0000","last_change":"2024-03-11T21:19:34.373507+0000","last_active":"2024-03-11T21:20:35.894397+0000","last_peered":"2024-03-11T21:20:35.894397+0000","last_clean":"2024-03-11T21:20:35.894397+0000","last_became_active":"2024-03-11T21:18:53.660393+0000","last_became_peered":"2024-03-11T21:18:53.660393+0000","last_unstale":"2024-03-11T21:20:35.894397+0000","last_undegraded":"2024-03-11T21:20:35.894397+0000","last_fullsized":"2024-03-11T21:20:35.894397+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'35","last_scrub_stamp":"2024-03-11T21:19:34.373224+0000","last_deep_scrub":"161'15","last_deep_scrub_stamp":"2024-03-11T20:57:15.125548+0000","last_clean_scrub_stamp":"2024-03-11T21:19:34.373224+0000","objects_scrubbed":5,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:40.415748+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":46,"num_read_kb":41,"num_write":32,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":182,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.263895+0000","last_change":"2024-03-11T21:19:26.482060+0000","last_active":"2024-03-11T21:20:19.263895+0000","last_peered":"2024-03-11T21:20:19.263895+0000","last_clean":"2024-03-11T21:20:19.263895+0000","last_became_active":"2024-03-11T21:14:50.420063+0000","last_became_peered":"2024-03-11T21:14:50.420063+0000","last_unstale":"2024-03-11T21:20:19.263895+0000","last_undegraded":"2024-03-11T21:20:19.263895+0000","last_fullsized":"2024-03-11T21:20:19.263895+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:19:26.481947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:19:26.481947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:32.923430+0000","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":681,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.045735+0000","last_change":"2024-03-11T21:19:35.032865+0000","last_active":"2024-03-11T21:20:19.045735+0000","last_peered":"2024-03-11T21:20:19.045735+0000","last_clean":"2024-03-11T21:20:19.045735+0000","last_became_active":"2024-03-11T21:00:48.459043+0000","last_became_peered":"2024-03-11T21:00:48.459043+0000","last_unstale":"2024-03-11T21:20:19.045735+0000","last_undegraded":"2024-03-11T21:20:19.045735+0000","last_fullsized":"2024-03-11T21:20:19.045735+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:19:35.032635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:16.860126+0000","last_clean_scrub_stamp":"2024-03-11T21:19:35.032635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:40.973406+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":1001,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.082174+0000","last_change":"2024-03-11T21:19:53.765481+0000","last_active":"2024-03-11T21:20:19.082174+0000","last_peered":"2024-03-11T21:20:19.082174+0000","last_clean":"2024-03-11T21:20:19.082174+0000","last_became_active":"2024-03-11T20:50:28.082737+0000","last_became_peered":"2024-03-11T20:50:28.082737+0000","last_unstale":"2024-03-11T21:20:19.082174+0000","last_undegraded":"2024-03-11T21:20:19.082174+0000","last_fullsized":"2024-03-11T21:20:19.082174+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:19:53.765222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:28.404997+0000","last_clean_scrub_stamp":"2024-03-11T21:19:53.765222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:18.382549+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"358'21","reported_seq":1018,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:23.452717+0000","last_change":"2024-03-11T21:20:23.452877+0000","last_active":"2024-03-11T21:20:23.452717+0000","last_peered":"2024-03-11T21:20:23.452717+0000","last_clean":"2024-03-11T21:20:23.452717+0000","last_became_active":"2024-03-11T21:00:27.967624+0000","last_became_peered":"2024-03-11T21:00:27.967624+0000","last_unstale":"2024-03-11T21:20:23.452717+0000","last_undegraded":"2024-03-11T21:20:23.452717+0000","last_fullsized":"2024-03-11T21:20:23.452717+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'21","last_scrub_stamp":"2024-03-11T21:20:23.452480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:20:23.452480+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:27.384101+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.10","version":"184'130","reported_seq":1237,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:23.656997+0000","last_change":"2024-03-11T21:20:23.657061+0000","last_active":"2024-03-11T21:20:23.656997+0000","last_peered":"2024-03-11T21:20:23.656997+0000","last_clean":"2024-03-11T21:20:23.656997+0000","last_became_active":"2024-03-11T21:16:24.829565+0000","last_became_peered":"2024-03-11T21:16:24.829565+0000","last_unstale":"2024-03-11T21:20:23.656997+0000","last_undegraded":"2024-03-11T21:20:23.656997+0000","last_fullsized":"2024-03-11T21:20:23.656997+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":444,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":5,"last_scrub":"184'130","last_scrub_stamp":"2024-03-11T21:20:23.656822+0000","last_deep_scrub":"184'130","last_deep_scrub_stamp":"2024-03-11T21:14:40.126341+0000","last_clean_scrub_stamp":"2024-03-11T21:20:23.656822+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:33.498464+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":430,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.267650+0000","last_change":"2024-03-11T21:19:50.693660+0000","last_active":"2024-03-11T21:20:19.267650+0000","last_peered":"2024-03-11T21:20:19.267650+0000","last_clean":"2024-03-11T21:20:19.267650+0000","last_became_active":"2024-03-11T21:18:53.664053+0000","last_became_peered":"2024-03-11T21:18:53.664053+0000","last_unstale":"2024-03-11T21:20:19.267650+0000","last_undegraded":"2024-03-11T21:20:19.267650+0000","last_fullsized":"2024-03-11T21:20:19.267650+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:19:50.693383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:57:16.032302+0000","last_clean_scrub_stamp":"2024-03-11T21:19:50.693383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:04.453078+0000","scrub_duration":187,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.15","version":"112'12","reported_seq":1347,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:29.566620+0000","last_change":"2024-03-11T21:20:10.680986+0000","last_active":"2024-03-11T21:20:29.566620+0000","last_peered":"2024-03-11T21:20:29.566620+0000","last_clean":"2024-03-11T21:20:29.566620+0000","last_became_active":"2024-03-11T20:50:32.650064+0000","last_became_peered":"2024-03-11T20:50:32.650064+0000","last_unstale":"2024-03-11T21:20:29.566620+0000","last_undegraded":"2024-03-11T21:20:29.566620+0000","last_fullsized":"2024-03-11T21:20:29.566620+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'12","last_scrub_stamp":"2024-03-11T21:20:10.680727+0000","last_deep_scrub":"112'12","last_deep_scrub_stamp":"2024-03-11T21:18:54.451798+0000","last_clean_scrub_stamp":"2024-03-11T21:20:10.680727+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:35.922921+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":190,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:29.497432+0000","last_change":"2024-03-11T21:20:29.497489+0000","last_active":"2024-03-11T21:20:29.497432+0000","last_peered":"2024-03-11T21:20:29.497432+0000","last_clean":"2024-03-11T21:20:29.497432+0000","last_became_active":"2024-03-11T21:14:50.434124+0000","last_became_peered":"2024-03-11T21:14:50.434124+0000","last_unstale":"2024-03-11T21:20:29.497432+0000","last_undegraded":"2024-03-11T21:20:29.497432+0000","last_fullsized":"2024-03-11T21:20:29.497432+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:20:29.497266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:06:52.862971+0000","last_clean_scrub_stamp":"2024-03-11T21:20:29.497266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:35.248395+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"357'25","reported_seq":200,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:34.533544+0000","last_change":"2024-03-11T21:20:34.533641+0000","last_active":"2024-03-11T21:20:34.533544+0000","last_peered":"2024-03-11T21:20:34.533544+0000","last_clean":"2024-03-11T21:20:34.533544+0000","last_became_active":"2024-03-11T21:14:50.434105+0000","last_became_peered":"2024-03-11T21:14:50.434105+0000","last_unstale":"2024-03-11T21:20:34.533544+0000","last_undegraded":"2024-03-11T21:20:34.533544+0000","last_fullsized":"2024-03-11T21:20:34.533544+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"357'25","last_scrub_stamp":"2024-03-11T21:20:34.533350+0000","last_deep_scrub":"119'13","last_deep_scrub_stamp":"2024-03-11T20:54:15.443472+0000","last_clean_scrub_stamp":"2024-03-11T21:20:34.533350+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:36.262773+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.11","version":"184'130","reported_seq":805,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.263335+0000","last_change":"2024-03-11T21:19:06.385686+0000","last_active":"2024-03-11T21:20:19.263335+0000","last_peered":"2024-03-11T21:20:19.263335+0000","last_clean":"2024-03-11T21:20:19.263335+0000","last_became_active":"2024-03-11T21:16:24.825311+0000","last_became_peered":"2024-03-11T21:16:24.825311+0000","last_unstale":"2024-03-11T21:20:19.263335+0000","last_undegraded":"2024-03-11T21:20:19.263335+0000","last_fullsized":"2024-03-11T21:20:19.263335+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":444,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":5,"last_scrub":"184'130","last_scrub_stamp":"2024-03-11T21:19:06.385464+0000","last_deep_scrub":"184'130","last_deep_scrub_stamp":"2024-03-11T21:17:44.379271+0000","last_clean_scrub_stamp":"2024-03-11T21:19:06.385464+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:33.430704+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":182,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.265056+0000","last_change":"2024-03-11T21:19:23.504455+0000","last_active":"2024-03-11T21:20:19.265056+0000","last_peered":"2024-03-11T21:20:19.265056+0000","last_clean":"2024-03-11T21:20:19.265056+0000","last_became_active":"2024-03-11T21:14:50.428898+0000","last_became_peered":"2024-03-11T21:14:50.428898+0000","last_unstale":"2024-03-11T21:20:19.265056+0000","last_undegraded":"2024-03-11T21:20:19.265056+0000","last_fullsized":"2024-03-11T21:20:19.265056+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:19:23.504278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:17:49.501931+0000","last_clean_scrub_stamp":"2024-03-11T21:19:23.504278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:40.240239+0000","scrub_duration":87,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"212'17","reported_seq":1346,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:29.564519+0000","last_change":"2024-03-11T21:20:14.597474+0000","last_active":"2024-03-11T21:20:29.564519+0000","last_peered":"2024-03-11T21:20:29.564519+0000","last_clean":"2024-03-11T21:20:29.564519+0000","last_became_active":"2024-03-11T21:00:48.467116+0000","last_became_peered":"2024-03-11T21:00:48.467116+0000","last_unstale":"2024-03-11T21:20:29.564519+0000","last_undegraded":"2024-03-11T21:20:29.564519+0000","last_fullsized":"2024-03-11T21:20:29.564519+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"212'17","last_scrub_stamp":"2024-03-11T21:20:14.597360+0000","last_deep_scrub":"110'13","last_deep_scrub_stamp":"2024-03-11T20:53:00.434532+0000","last_clean_scrub_stamp":"2024-03-11T21:20:14.597360+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:32.296804+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":990,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.069082+0000","last_change":"2024-03-11T21:19:36.659874+0000","last_active":"2024-03-11T21:20:19.069082+0000","last_peered":"2024-03-11T21:20:19.069082+0000","last_clean":"2024-03-11T21:20:19.069082+0000","last_became_active":"2024-03-11T20:50:28.070325+0000","last_became_peered":"2024-03-11T20:50:28.070325+0000","last_unstale":"2024-03-11T21:20:19.069082+0000","last_undegraded":"2024-03-11T21:20:19.069082+0000","last_fullsized":"2024-03-11T21:20:19.069082+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:19:36.659656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:12:39.708033+0000","last_clean_scrub_stamp":"2024-03-11T21:19:36.659656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:49.794595+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"358'32","reported_seq":1032,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.441902+0000","last_change":"2024-03-11T21:20:19.441983+0000","last_active":"2024-03-11T21:20:19.441902+0000","last_peered":"2024-03-11T21:20:19.441902+0000","last_clean":"2024-03-11T21:20:19.441902+0000","last_became_active":"2024-03-11T21:00:27.967078+0000","last_became_peered":"2024-03-11T21:00:27.967078+0000","last_unstale":"2024-03-11T21:20:19.441902+0000","last_undegraded":"2024-03-11T21:20:19.441902+0000","last_fullsized":"2024-03-11T21:20:19.441902+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'32","last_scrub_stamp":"2024-03-11T21:20:19.441743+0000","last_deep_scrub":"358'32","last_deep_scrub_stamp":"2024-03-11T21:20:19.441743+0000","last_clean_scrub_stamp":"2024-03-11T21:20:19.441743+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:49.341484+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.12","version":"184'130","reported_seq":1235,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.068509+0000","last_change":"2024-03-11T21:19:18.719229+0000","last_active":"2024-03-11T21:20:19.068509+0000","last_peered":"2024-03-11T21:20:19.068509+0000","last_clean":"2024-03-11T21:20:19.068509+0000","last_became_active":"2024-03-11T21:16:24.831501+0000","last_became_peered":"2024-03-11T21:16:24.831501+0000","last_unstale":"2024-03-11T21:20:19.068509+0000","last_undegraded":"2024-03-11T21:20:19.068509+0000","last_fullsized":"2024-03-11T21:20:19.068509+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":444,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":5,"last_scrub":"184'130","last_scrub_stamp":"2024-03-11T21:19:18.719015+0000","last_deep_scrub":"184'130","last_deep_scrub_stamp":"2024-03-11T20:58:56.415267+0000","last_clean_scrub_stamp":"2024-03-11T21:19:18.719015+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:39.551998+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":968,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:29.515456+0000","last_change":"2024-03-11T21:20:29.515496+0000","last_active":"2024-03-11T21:20:29.515456+0000","last_peered":"2024-03-11T21:20:29.515456+0000","last_clean":"2024-03-11T21:20:29.515456+0000","last_became_active":"2024-03-11T21:00:27.966997+0000","last_became_peered":"2024-03-11T21:00:27.966997+0000","last_unstale":"2024-03-11T21:20:29.515456+0000","last_undegraded":"2024-03-11T21:20:29.515456+0000","last_fullsized":"2024-03-11T21:20:29.515456+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:20:29.515336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:17:29.228431+0000","last_clean_scrub_stamp":"2024-03-11T21:20:29.515336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:58.711743+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":963,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:22.585435+0000","last_change":"2024-03-11T21:20:22.585490+0000","last_active":"2024-03-11T21:20:22.585435+0000","last_peered":"2024-03-11T21:20:22.585435+0000","last_clean":"2024-03-11T21:20:22.585435+0000","last_became_active":"2024-03-11T21:00:48.468386+0000","last_became_peered":"2024-03-11T21:00:48.468386+0000","last_unstale":"2024-03-11T21:20:22.585435+0000","last_undegraded":"2024-03-11T21:20:22.585435+0000","last_fullsized":"2024-03-11T21:20:22.585435+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:20:22.585283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:20:22.585283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:46.208993+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"166'2","reported_seq":1028,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:25.922466+0000","last_change":"2024-03-11T21:20:25.922536+0000","last_active":"2024-03-11T21:20:25.922466+0000","last_peered":"2024-03-11T21:20:25.922466+0000","last_clean":"2024-03-11T21:20:25.922466+0000","last_became_active":"2024-03-11T21:00:48.466071+0000","last_became_peered":"2024-03-11T21:00:48.466071+0000","last_unstale":"2024-03-11T21:20:25.922466+0000","last_undegraded":"2024-03-11T21:20:25.922466+0000","last_fullsized":"2024-03-11T21:20:25.922466+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'2","last_scrub_stamp":"2024-03-11T21:20:25.922313+0000","last_deep_scrub":"166'2","last_deep_scrub_stamp":"2024-03-11T20:57:39.868776+0000","last_clean_scrub_stamp":"2024-03-11T21:20:25.922313+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:28.623872+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1822,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"364'14","reported_seq":1025,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.081039+0000","last_change":"2024-03-11T21:19:26.550324+0000","last_active":"2024-03-11T21:20:19.081039+0000","last_peered":"2024-03-11T21:20:19.081039+0000","last_clean":"2024-03-11T21:20:19.081039+0000","last_became_active":"2024-03-11T21:14:50.420602+0000","last_became_peered":"2024-03-11T21:14:50.420602+0000","last_unstale":"2024-03-11T21:20:19.081039+0000","last_undegraded":"2024-03-11T21:20:19.081039+0000","last_fullsized":"2024-03-11T21:20:19.081039+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"364'14","last_scrub_stamp":"2024-03-11T21:19:26.550103+0000","last_deep_scrub":"364'14","last_deep_scrub_stamp":"2024-03-11T21:11:15.819245+0000","last_clean_scrub_stamp":"2024-03-11T21:19:26.550103+0000","objects_scrubbed":2,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:41.517513+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":16,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.13","version":"184'130","reported_seq":854,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:25.047669+0000","last_change":"2024-03-11T21:20:25.047733+0000","last_active":"2024-03-11T21:20:25.047669+0000","last_peered":"2024-03-11T21:20:25.047669+0000","last_clean":"2024-03-11T21:20:25.047669+0000","last_became_active":"2024-03-11T21:16:24.840090+0000","last_became_peered":"2024-03-11T21:16:24.840090+0000","last_unstale":"2024-03-11T21:20:25.047669+0000","last_undegraded":"2024-03-11T21:20:25.047669+0000","last_fullsized":"2024-03-11T21:20:25.047669+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":444,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":5,"last_scrub":"184'130","last_scrub_stamp":"2024-03-11T21:20:25.047504+0000","last_deep_scrub":"184'130","last_deep_scrub_stamp":"2024-03-11T20:58:56.415267+0000","last_clean_scrub_stamp":"2024-03-11T21:20:25.047504+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:45.794232+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":667,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.076466+0000","last_change":"2024-03-11T21:19:31.021948+0000","last_active":"2024-03-11T21:20:19.076466+0000","last_peered":"2024-03-11T21:20:19.076466+0000","last_clean":"2024-03-11T21:20:19.076466+0000","last_became_active":"2024-03-11T21:17:19.512977+0000","last_became_peered":"2024-03-11T21:17:19.512977+0000","last_unstale":"2024-03-11T21:20:19.076466+0000","last_undegraded":"2024-03-11T21:20:19.076466+0000","last_fullsized":"2024-03-11T21:20:19.076466+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:19:31.021863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:19:31.021863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:49.852332+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":190,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.266227+0000","last_change":"2024-03-11T21:19:52.695880+0000","last_active":"2024-03-11T21:20:19.266227+0000","last_peered":"2024-03-11T21:20:19.266227+0000","last_clean":"2024-03-11T21:20:19.266227+0000","last_became_active":"2024-03-11T21:14:50.708563+0000","last_became_peered":"2024-03-11T21:14:50.708563+0000","last_unstale":"2024-03-11T21:20:19.266227+0000","last_undegraded":"2024-03-11T21:20:19.266227+0000","last_fullsized":"2024-03-11T21:20:19.266227+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:19:52.695673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:54:09.537972+0000","last_clean_scrub_stamp":"2024-03-11T21:19:52.695673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:02.884010+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":688,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:29.728117+0000","last_change":"2024-03-11T21:20:29.728181+0000","last_active":"2024-03-11T21:20:29.728117+0000","last_peered":"2024-03-11T21:20:29.728117+0000","last_clean":"2024-03-11T21:20:29.728117+0000","last_became_active":"2024-03-11T21:17:19.980445+0000","last_became_peered":"2024-03-11T21:17:19.980445+0000","last_unstale":"2024-03-11T21:20:29.728117+0000","last_undegraded":"2024-03-11T21:20:29.728117+0000","last_fullsized":"2024-03-11T21:20:29.728117+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:20:29.727931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:19:10.359123+0000","last_clean_scrub_stamp":"2024-03-11T21:20:29.727931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:45.534602+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"358'32","reported_seq":1024,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.037688+0000","last_change":"2024-03-11T21:19:47.256470+0000","last_active":"2024-03-11T21:20:19.037688+0000","last_peered":"2024-03-11T21:20:19.037688+0000","last_clean":"2024-03-11T21:20:19.037688+0000","last_became_active":"2024-03-11T21:00:27.970878+0000","last_became_peered":"2024-03-11T21:00:27.970878+0000","last_unstale":"2024-03-11T21:20:19.037688+0000","last_undegraded":"2024-03-11T21:20:19.037688+0000","last_fullsized":"2024-03-11T21:20:19.037688+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'32","last_scrub_stamp":"2024-03-11T21:19:47.256207+0000","last_deep_scrub":"358'32","last_deep_scrub_stamp":"2024-03-11T21:19:47.256207+0000","last_clean_scrub_stamp":"2024-03-11T21:19:47.256207+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:11.721751+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"184'130","reported_seq":852,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.051157+0000","last_change":"2024-03-11T21:19:46.491231+0000","last_active":"2024-03-11T21:20:19.051157+0000","last_peered":"2024-03-11T21:20:19.051157+0000","last_clean":"2024-03-11T21:20:19.051157+0000","last_became_active":"2024-03-11T21:16:24.827641+0000","last_became_peered":"2024-03-11T21:16:24.827641+0000","last_unstale":"2024-03-11T21:20:19.051157+0000","last_undegraded":"2024-03-11T21:20:19.051157+0000","last_fullsized":"2024-03-11T21:20:19.051157+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":444,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":4,"last_scrub":"184'130","last_scrub_stamp":"2024-03-11T21:19:46.491123+0000","last_deep_scrub":"184'130","last_deep_scrub_stamp":"2024-03-11T20:58:56.415267+0000","last_clean_scrub_stamp":"2024-03-11T21:19:46.491123+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:58.642722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":968,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:35.846442+0000","last_change":"2024-03-11T21:20:35.846469+0000","last_active":"2024-03-11T21:20:35.846442+0000","last_peered":"2024-03-11T21:20:35.846442+0000","last_clean":"2024-03-11T21:20:35.846442+0000","last_became_active":"2024-03-11T21:00:48.466853+0000","last_became_peered":"2024-03-11T21:00:48.466853+0000","last_unstale":"2024-03-11T21:20:35.846442+0000","last_undegraded":"2024-03-11T21:20:35.846442+0000","last_fullsized":"2024-03-11T21:20:35.846442+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:20:35.846331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:52:53.450885+0000","last_clean_scrub_stamp":"2024-03-11T21:20:35.846331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:52.559817+0000","scrub_duration":207,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"77'2","reported_seq":477,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.267772+0000","last_change":"2024-03-11T21:20:17.528399+0000","last_active":"2024-03-11T21:20:19.267772+0000","last_peered":"2024-03-11T21:20:19.267772+0000","last_clean":"2024-03-11T21:20:19.267772+0000","last_became_active":"2024-03-11T21:18:53.659656+0000","last_became_peered":"2024-03-11T21:18:53.659656+0000","last_unstale":"2024-03-11T21:20:19.267772+0000","last_undegraded":"2024-03-11T21:20:19.267772+0000","last_fullsized":"2024-03-11T21:20:19.267772+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'2","last_scrub_stamp":"2024-03-11T21:20:17.528107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:20:17.528107+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:18.203802+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"358'32","reported_seq":1085,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.039942+0000","last_change":"2024-03-11T21:19:26.300612+0000","last_active":"2024-03-11T21:20:19.039942+0000","last_peered":"2024-03-11T21:20:19.039942+0000","last_clean":"2024-03-11T21:20:19.039942+0000","last_became_active":"2024-03-11T21:00:48.465254+0000","last_became_peered":"2024-03-11T21:00:48.465254+0000","last_unstale":"2024-03-11T21:20:19.039942+0000","last_undegraded":"2024-03-11T21:20:19.039942+0000","last_fullsized":"2024-03-11T21:20:19.039942+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'32","last_scrub_stamp":"2024-03-11T21:19:26.300201+0000","last_deep_scrub":"100'16","last_deep_scrub_stamp":"2024-03-11T20:56:08.036331+0000","last_clean_scrub_stamp":"2024-03-11T21:19:26.300201+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:38.534730+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"486'19","reported_seq":675,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:34.685825+0000","last_change":"2024-03-11T21:20:34.685894+0000","last_active":"2024-03-11T21:20:34.685825+0000","last_peered":"2024-03-11T21:20:34.685825+0000","last_clean":"2024-03-11T21:20:34.685825+0000","last_became_active":"2024-03-11T21:18:53.658334+0000","last_became_peered":"2024-03-11T21:18:53.658334+0000","last_unstale":"2024-03-11T21:20:34.685825+0000","last_undegraded":"2024-03-11T21:20:34.685825+0000","last_fullsized":"2024-03-11T21:20:34.685825+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"486'19","last_scrub_stamp":"2024-03-11T21:20:34.685669+0000","last_deep_scrub":"212'17","last_deep_scrub_stamp":"2024-03-11T21:03:38.258850+0000","last_clean_scrub_stamp":"2024-03-11T21:20:34.685669+0000","objects_scrubbed":1,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:45.202519+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"184'130","reported_seq":849,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.082971+0000","last_change":"2024-03-11T21:19:17.445933+0000","last_active":"2024-03-11T21:20:19.082971+0000","last_peered":"2024-03-11T21:20:19.082971+0000","last_clean":"2024-03-11T21:20:19.082971+0000","last_became_active":"2024-03-11T21:16:19.734553+0000","last_became_peered":"2024-03-11T21:16:19.734553+0000","last_unstale":"2024-03-11T21:20:19.082971+0000","last_undegraded":"2024-03-11T21:20:19.082971+0000","last_fullsized":"2024-03-11T21:20:19.082971+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":3,"last_scrub":"184'130","last_scrub_stamp":"2024-03-11T21:19:17.445630+0000","last_deep_scrub":"184'130","last_deep_scrub_stamp":"2024-03-11T20:58:56.415267+0000","last_clean_scrub_stamp":"2024-03-11T21:19:17.445630+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:40.521827+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":182,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.264035+0000","last_change":"2024-03-11T21:19:31.577859+0000","last_active":"2024-03-11T21:20:19.264035+0000","last_peered":"2024-03-11T21:20:19.264035+0000","last_clean":"2024-03-11T21:20:19.264035+0000","last_became_active":"2024-03-11T21:14:50.683519+0000","last_became_peered":"2024-03-11T21:14:50.683519+0000","last_unstale":"2024-03-11T21:20:19.264035+0000","last_undegraded":"2024-03-11T21:20:19.264035+0000","last_fullsized":"2024-03-11T21:20:19.264035+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:19:31.577700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:57:21.568400+0000","last_clean_scrub_stamp":"2024-03-11T21:19:31.577700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:34.661958+0000","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":694,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.046660+0000","last_change":"2024-03-11T21:19:38.141168+0000","last_active":"2024-03-11T21:20:19.046660+0000","last_peered":"2024-03-11T21:20:19.046660+0000","last_clean":"2024-03-11T21:20:19.046660+0000","last_became_active":"2024-03-11T21:14:50.421319+0000","last_became_peered":"2024-03-11T21:14:50.421319+0000","last_unstale":"2024-03-11T21:20:19.046660+0000","last_undegraded":"2024-03-11T21:20:19.046660+0000","last_fullsized":"2024-03-11T21:20:19.046660+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:19:38.141032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:20.096874+0000","last_clean_scrub_stamp":"2024-03-11T21:19:38.141032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:48.260710+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"358'40","reported_seq":1400,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:29.566876+0000","last_change":"2024-03-11T21:19:22.734864+0000","last_active":"2024-03-11T21:20:29.566876+0000","last_peered":"2024-03-11T21:20:29.566876+0000","last_clean":"2024-03-11T21:20:29.566876+0000","last_became_active":"2024-03-11T20:50:30.094977+0000","last_became_peered":"2024-03-11T20:50:30.094977+0000","last_unstale":"2024-03-11T21:20:29.566876+0000","last_undegraded":"2024-03-11T21:20:29.566876+0000","last_fullsized":"2024-03-11T21:20:29.566876+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'40","last_scrub_stamp":"2024-03-11T21:19:22.734616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:19:22.734616+0000","objects_scrubbed":8,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:48.748907+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":61,"num_read_kb":51,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":682,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.045675+0000","last_change":"2024-03-11T21:19:45.107526+0000","last_active":"2024-03-11T21:20:19.045675+0000","last_peered":"2024-03-11T21:20:19.045675+0000","last_clean":"2024-03-11T21:20:19.045675+0000","last_became_active":"2024-03-11T21:00:48.457901+0000","last_became_peered":"2024-03-11T21:00:48.457901+0000","last_unstale":"2024-03-11T21:20:19.045675+0000","last_undegraded":"2024-03-11T21:20:19.045675+0000","last_fullsized":"2024-03-11T21:20:19.045675+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:19:45.107196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:16:59.052089+0000","last_clean_scrub_stamp":"2024-03-11T21:19:45.107196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:45.753171+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"184'130","reported_seq":873,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.068302+0000","last_change":"2024-03-11T21:20:13.685146+0000","last_active":"2024-03-11T21:20:19.068302+0000","last_peered":"2024-03-11T21:20:19.068302+0000","last_clean":"2024-03-11T21:20:19.068302+0000","last_became_active":"2024-03-11T21:16:19.715221+0000","last_became_peered":"2024-03-11T21:16:19.715221+0000","last_unstale":"2024-03-11T21:20:19.068302+0000","last_undegraded":"2024-03-11T21:20:19.068302+0000","last_fullsized":"2024-03-11T21:20:19.068302+0000","mapping_epoch":426,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":4,"last_scrub":"184'130","last_scrub_stamp":"2024-03-11T21:20:13.684884+0000","last_deep_scrub":"184'130","last_deep_scrub_stamp":"2024-03-11T21:20:13.684884+0000","last_clean_scrub_stamp":"2024-03-11T21:20:13.684884+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:24.842309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":948,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:28.643948+0000","last_change":"2024-03-11T21:20:28.644007+0000","last_active":"2024-03-11T21:20:28.643948+0000","last_peered":"2024-03-11T21:20:28.643948+0000","last_clean":"2024-03-11T21:20:28.643948+0000","last_became_active":"2024-03-11T21:00:27.959408+0000","last_became_peered":"2024-03-11T21:00:27.959408+0000","last_unstale":"2024-03-11T21:20:28.643948+0000","last_undegraded":"2024-03-11T21:20:28.643948+0000","last_fullsized":"2024-03-11T21:20:28.643948+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:20:28.643793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:19:10.721029+0000","last_clean_scrub_stamp":"2024-03-11T21:20:28.643793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:54.701280+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1005,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:20.644381+0000","last_change":"2024-03-11T21:20:20.644436+0000","last_active":"2024-03-11T21:20:20.644381+0000","last_peered":"2024-03-11T21:20:20.644381+0000","last_clean":"2024-03-11T21:20:20.644381+0000","last_became_active":"2024-03-11T21:00:48.471212+0000","last_became_peered":"2024-03-11T21:00:48.471212+0000","last_unstale":"2024-03-11T21:20:20.644381+0000","last_undegraded":"2024-03-11T21:20:20.644381+0000","last_fullsized":"2024-03-11T21:20:20.644381+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:20:20.644234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:09:55.786349+0000","last_clean_scrub_stamp":"2024-03-11T21:20:20.644234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:44.957758+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"497'49","reported_seq":548,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:35.908994+0000","last_change":"2024-03-11T21:20:24.683297+0000","last_active":"2024-03-11T21:20:35.908994+0000","last_peered":"2024-03-11T21:20:35.908994+0000","last_clean":"2024-03-11T21:20:35.908994+0000","last_became_active":"2024-03-11T21:18:53.661301+0000","last_became_peered":"2024-03-11T21:18:53.661301+0000","last_unstale":"2024-03-11T21:20:35.908994+0000","last_undegraded":"2024-03-11T21:20:35.908994+0000","last_fullsized":"2024-03-11T21:20:35.908994+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'47","last_scrub_stamp":"2024-03-11T21:20:24.683048+0000","last_deep_scrub":"358'47","last_deep_scrub_stamp":"2024-03-11T21:17:59.358910+0000","last_clean_scrub_stamp":"2024-03-11T21:20:24.683048+0000","objects_scrubbed":8,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:42.618834+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":56,"num_write":44,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":682,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.045878+0000","last_change":"2024-03-11T21:20:13.922389+0000","last_active":"2024-03-11T21:20:19.045878+0000","last_peered":"2024-03-11T21:20:19.045878+0000","last_clean":"2024-03-11T21:20:19.045878+0000","last_became_active":"2024-03-11T21:00:48.458008+0000","last_became_peered":"2024-03-11T21:00:48.458008+0000","last_unstale":"2024-03-11T21:20:19.045878+0000","last_undegraded":"2024-03-11T21:20:19.045878+0000","last_fullsized":"2024-03-11T21:20:19.045878+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:20:13.922307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:33.319221+0000","last_clean_scrub_stamp":"2024-03-11T21:20:13.922307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:21.960346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"184'130","reported_seq":1301,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:28.393000+0000","last_change":"2024-03-11T21:20:28.393137+0000","last_active":"2024-03-11T21:20:28.393000+0000","last_peered":"2024-03-11T21:20:28.393000+0000","last_clean":"2024-03-11T21:20:28.393000+0000","last_became_active":"2024-03-11T21:16:19.731039+0000","last_became_peered":"2024-03-11T21:16:19.731039+0000","last_unstale":"2024-03-11T21:20:28.393000+0000","last_undegraded":"2024-03-11T21:20:28.393000+0000","last_fullsized":"2024-03-11T21:20:28.393000+0000","mapping_epoch":426,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":4,"last_scrub":"184'130","last_scrub_stamp":"2024-03-11T21:20:28.392785+0000","last_deep_scrub":"184'130","last_deep_scrub_stamp":"2024-03-11T20:58:56.415267+0000","last_clean_scrub_stamp":"2024-03-11T21:20:28.392785+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:29.626414+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":969,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.039511+0000","last_change":"2024-03-11T21:19:37.270379+0000","last_active":"2024-03-11T21:20:19.039511+0000","last_peered":"2024-03-11T21:20:19.039511+0000","last_clean":"2024-03-11T21:20:19.039511+0000","last_became_active":"2024-03-11T20:50:34.321038+0000","last_became_peered":"2024-03-11T20:50:34.321038+0000","last_unstale":"2024-03-11T21:20:19.039511+0000","last_undegraded":"2024-03-11T21:20:19.039511+0000","last_fullsized":"2024-03-11T21:20:19.039511+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:19:37.270220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:11:20.989384+0000","last_clean_scrub_stamp":"2024-03-11T21:19:37.270220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:58.269312+0000","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"358'50","reported_seq":732,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.085703+0000","last_change":"2024-03-11T21:20:19.085797+0000","last_active":"2024-03-11T21:20:19.085703+0000","last_peered":"2024-03-11T21:20:19.085703+0000","last_clean":"2024-03-11T21:20:19.085703+0000","last_became_active":"2024-03-11T21:00:48.456178+0000","last_became_peered":"2024-03-11T21:00:48.456178+0000","last_unstale":"2024-03-11T21:20:19.085703+0000","last_undegraded":"2024-03-11T21:20:19.085703+0000","last_fullsized":"2024-03-11T21:20:19.085703+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'50","last_scrub_stamp":"2024-03-11T21:20:19.085539+0000","last_deep_scrub":"109'26","last_deep_scrub_stamp":"2024-03-11T20:52:49.216378+0000","last_clean_scrub_stamp":"2024-03-11T21:20:19.085539+0000","objects_scrubbed":8,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:43.206239+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":58,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":684,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:30.744050+0000","last_change":"2024-03-11T21:20:30.744139+0000","last_active":"2024-03-11T21:20:30.744050+0000","last_peered":"2024-03-11T21:20:30.744050+0000","last_clean":"2024-03-11T21:20:30.744050+0000","last_became_active":"2024-03-11T21:17:19.978546+0000","last_became_peered":"2024-03-11T21:17:19.978546+0000","last_unstale":"2024-03-11T21:20:30.744050+0000","last_undegraded":"2024-03-11T21:20:30.744050+0000","last_fullsized":"2024-03-11T21:20:30.744050+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:20:30.743858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:19:01.325113+0000","last_clean_scrub_stamp":"2024-03-11T21:20:30.743858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:49.021497+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":686,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.046341+0000","last_change":"2024-03-11T21:19:20.895344+0000","last_active":"2024-03-11T21:20:19.046341+0000","last_peered":"2024-03-11T21:20:19.046341+0000","last_clean":"2024-03-11T21:20:19.046341+0000","last_became_active":"2024-03-11T21:14:50.422215+0000","last_became_peered":"2024-03-11T21:14:50.422215+0000","last_unstale":"2024-03-11T21:20:19.046341+0000","last_undegraded":"2024-03-11T21:20:19.046341+0000","last_fullsized":"2024-03-11T21:20:19.046341+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:19:20.895217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:16:47.986863+0000","last_clean_scrub_stamp":"2024-03-11T21:19:20.895217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:41.983836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":973,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.070547+0000","last_change":"2024-03-11T21:20:15.856629+0000","last_active":"2024-03-11T21:20:19.070547+0000","last_peered":"2024-03-11T21:20:19.070547+0000","last_clean":"2024-03-11T21:20:19.070547+0000","last_became_active":"2024-03-11T21:00:27.962668+0000","last_became_peered":"2024-03-11T21:00:27.962668+0000","last_unstale":"2024-03-11T21:20:19.070547+0000","last_undegraded":"2024-03-11T21:20:19.070547+0000","last_fullsized":"2024-03-11T21:20:19.070547+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:20:15.856455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:20:15.856455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:35.867768+0000","scrub_duration":171,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"358'35","reported_seq":998,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.050251+0000","last_change":"2024-03-11T21:19:23.531256+0000","last_active":"2024-03-11T21:20:19.050251+0000","last_peered":"2024-03-11T21:20:19.050251+0000","last_clean":"2024-03-11T21:20:19.050251+0000","last_became_active":"2024-03-11T20:50:30.101747+0000","last_became_peered":"2024-03-11T20:50:30.101747+0000","last_unstale":"2024-03-11T21:20:19.050251+0000","last_undegraded":"2024-03-11T21:20:19.050251+0000","last_fullsized":"2024-03-11T21:20:19.050251+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'35","last_scrub_stamp":"2024-03-11T21:19:23.530814+0000","last_deep_scrub":"358'35","last_deep_scrub_stamp":"2024-03-11T21:17:53.522668+0000","last_clean_scrub_stamp":"2024-03-11T21:19:23.530814+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:40.955124+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":984,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.069114+0000","last_change":"2024-03-11T21:19:30.708268+0000","last_active":"2024-03-11T21:20:19.069114+0000","last_peered":"2024-03-11T21:20:19.069114+0000","last_clean":"2024-03-11T21:20:19.069114+0000","last_became_active":"2024-03-11T20:50:28.085503+0000","last_became_peered":"2024-03-11T20:50:28.085503+0000","last_unstale":"2024-03-11T21:20:19.069114+0000","last_undegraded":"2024-03-11T21:20:19.069114+0000","last_fullsized":"2024-03-11T21:20:19.069114+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:19:30.708052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:56:03.479180+0000","last_clean_scrub_stamp":"2024-03-11T21:19:30.708052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:45.500639+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"184'130","reported_seq":1294,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.037758+0000","last_change":"2024-03-11T21:20:08.337922+0000","last_active":"2024-03-11T21:20:19.037758+0000","last_peered":"2024-03-11T21:20:19.037758+0000","last_clean":"2024-03-11T21:20:19.037758+0000","last_became_active":"2024-03-11T21:16:19.728863+0000","last_became_peered":"2024-03-11T21:16:19.728863+0000","last_unstale":"2024-03-11T21:20:19.037758+0000","last_undegraded":"2024-03-11T21:20:19.037758+0000","last_fullsized":"2024-03-11T21:20:19.037758+0000","mapping_epoch":426,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":0,"last_scrub":"184'130","last_scrub_stamp":"2024-03-11T21:20:08.337679+0000","last_deep_scrub":"184'130","last_deep_scrub_stamp":"2024-03-11T21:18:35.305688+0000","last_clean_scrub_stamp":"2024-03-11T21:20:08.337679+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:20.325099+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":996,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.082948+0000","last_change":"2024-03-11T21:19:40.667610+0000","last_active":"2024-03-11T21:20:19.082948+0000","last_peered":"2024-03-11T21:20:19.082948+0000","last_clean":"2024-03-11T21:20:19.082948+0000","last_became_active":"2024-03-11T21:00:27.959467+0000","last_became_peered":"2024-03-11T21:00:27.959467+0000","last_unstale":"2024-03-11T21:20:19.082948+0000","last_undegraded":"2024-03-11T21:20:19.082948+0000","last_fullsized":"2024-03-11T21:20:19.082948+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:19:40.667380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:33.325163+0000","last_clean_scrub_stamp":"2024-03-11T21:19:40.667380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:59.839698+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.3","version":"184'130","reported_seq":847,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.045897+0000","last_change":"2024-03-11T21:20:14.969557+0000","last_active":"2024-03-11T21:20:19.045897+0000","last_peered":"2024-03-11T21:20:19.045897+0000","last_clean":"2024-03-11T21:20:19.045897+0000","last_became_active":"2024-03-11T21:16:19.722183+0000","last_became_peered":"2024-03-11T21:16:19.722183+0000","last_unstale":"2024-03-11T21:20:19.045897+0000","last_undegraded":"2024-03-11T21:20:19.045897+0000","last_fullsized":"2024-03-11T21:20:19.045897+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":3,"last_scrub":"184'130","last_scrub_stamp":"2024-03-11T21:20:14.969443+0000","last_deep_scrub":"184'130","last_deep_scrub_stamp":"2024-03-11T20:58:56.415267+0000","last_clean_scrub_stamp":"2024-03-11T21:20:14.969443+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:28.970466+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":682,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:30.970522+0000","last_change":"2024-03-11T21:20:30.970578+0000","last_active":"2024-03-11T21:20:30.970522+0000","last_peered":"2024-03-11T21:20:30.970522+0000","last_clean":"2024-03-11T21:20:30.970522+0000","last_became_active":"2024-03-11T21:00:48.464596+0000","last_became_peered":"2024-03-11T21:00:48.464596+0000","last_unstale":"2024-03-11T21:20:30.970522+0000","last_undegraded":"2024-03-11T21:20:30.970522+0000","last_fullsized":"2024-03-11T21:20:30.970522+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:20:30.970368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:20:30.970368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:53.330811+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"497'53","reported_seq":204,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:35.939291+0000","last_change":"2024-03-11T21:19:24.399278+0000","last_active":"2024-03-11T21:20:35.939291+0000","last_peered":"2024-03-11T21:20:35.939291+0000","last_clean":"2024-03-11T21:20:35.939291+0000","last_became_active":"2024-03-11T21:14:50.433732+0000","last_became_peered":"2024-03-11T21:14:50.433732+0000","last_unstale":"2024-03-11T21:20:35.939291+0000","last_undegraded":"2024-03-11T21:20:35.939291+0000","last_fullsized":"2024-03-11T21:20:35.939291+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'49","last_scrub_stamp":"2024-03-11T21:19:24.399051+0000","last_deep_scrub":"100'21","last_deep_scrub_stamp":"2024-03-11T20:51:31.835302+0000","last_clean_scrub_stamp":"2024-03-11T21:19:24.399051+0000","objects_scrubbed":7,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:54.269687+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":67,"num_read_kb":60,"num_write":46,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":647,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.074784+0000","last_change":"2024-03-11T21:19:26.126592+0000","last_active":"2024-03-11T21:20:19.074784+0000","last_peered":"2024-03-11T21:20:19.074784+0000","last_clean":"2024-03-11T21:20:19.074784+0000","last_became_active":"2024-03-11T21:17:19.982830+0000","last_became_peered":"2024-03-11T21:17:19.982830+0000","last_unstale":"2024-03-11T21:20:19.074784+0000","last_undegraded":"2024-03-11T21:20:19.074784+0000","last_fullsized":"2024-03-11T21:20:19.074784+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:19:26.126371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:19:26.126371+0000","last_clean_scrub_stamp":"2024-03-11T21:19:26.126371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:41.028999+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":182,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.266016+0000","last_change":"2024-03-11T21:19:29.432627+0000","last_active":"2024-03-11T21:20:19.266016+0000","last_peered":"2024-03-11T21:20:19.266016+0000","last_clean":"2024-03-11T21:20:19.266016+0000","last_became_active":"2024-03-11T21:14:50.708638+0000","last_became_peered":"2024-03-11T21:14:50.708638+0000","last_unstale":"2024-03-11T21:20:19.266016+0000","last_undegraded":"2024-03-11T21:20:19.266016+0000","last_fullsized":"2024-03-11T21:20:19.266016+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:19:29.432412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:26.839782+0000","last_clean_scrub_stamp":"2024-03-11T21:19:29.432412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:57.505798+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"184'130","reported_seq":875,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:32.952915+0000","last_change":"2024-03-11T21:20:32.953048+0000","last_active":"2024-03-11T21:20:32.952915+0000","last_peered":"2024-03-11T21:20:32.952915+0000","last_clean":"2024-03-11T21:20:32.952915+0000","last_became_active":"2024-03-11T21:16:19.741626+0000","last_became_peered":"2024-03-11T21:16:19.741626+0000","last_unstale":"2024-03-11T21:20:32.952915+0000","last_undegraded":"2024-03-11T21:20:32.952915+0000","last_fullsized":"2024-03-11T21:20:32.952915+0000","mapping_epoch":426,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":3,"last_scrub":"184'130","last_scrub_stamp":"2024-03-11T21:20:32.952721+0000","last_deep_scrub":"184'130","last_deep_scrub_stamp":"2024-03-11T21:19:15.396644+0000","last_clean_scrub_stamp":"2024-03-11T21:20:32.952721+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:41.088200+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":682,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.045261+0000","last_change":"2024-03-11T21:19:43.246952+0000","last_active":"2024-03-11T21:20:19.045261+0000","last_peered":"2024-03-11T21:20:19.045261+0000","last_clean":"2024-03-11T21:20:19.045261+0000","last_became_active":"2024-03-11T21:00:48.469688+0000","last_became_peered":"2024-03-11T21:00:48.469688+0000","last_unstale":"2024-03-11T21:20:19.045261+0000","last_undegraded":"2024-03-11T21:20:19.045261+0000","last_fullsized":"2024-03-11T21:20:19.045261+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:19:43.246787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:19:43.246787+0000","last_clean_scrub_stamp":"2024-03-11T21:19:43.246787+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:57.534597+0000","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"145'3","reported_seq":1017,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:31.030753+0000","last_change":"2024-03-11T21:20:31.030824+0000","last_active":"2024-03-11T21:20:31.030753+0000","last_peered":"2024-03-11T21:20:31.030753+0000","last_clean":"2024-03-11T21:20:31.030753+0000","last_became_active":"2024-03-11T21:14:50.427126+0000","last_became_peered":"2024-03-11T21:14:50.427126+0000","last_unstale":"2024-03-11T21:20:31.030753+0000","last_undegraded":"2024-03-11T21:20:31.030753+0000","last_fullsized":"2024-03-11T21:20:31.030753+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'3","last_scrub_stamp":"2024-03-11T21:20:31.030591+0000","last_deep_scrub":"145'3","last_deep_scrub_stamp":"2024-03-11T21:16:34.671325+0000","last_clean_scrub_stamp":"2024-03-11T21:20:31.030591+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:40.237473+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0.0024049620000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":994,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.039777+0000","last_change":"2024-03-11T21:19:42.227486+0000","last_active":"2024-03-11T21:20:19.039777+0000","last_peered":"2024-03-11T21:20:19.039777+0000","last_clean":"2024-03-11T21:20:19.039777+0000","last_became_active":"2024-03-11T21:14:50.446250+0000","last_became_peered":"2024-03-11T21:14:50.446250+0000","last_unstale":"2024-03-11T21:20:19.039777+0000","last_undegraded":"2024-03-11T21:20:19.039777+0000","last_fullsized":"2024-03-11T21:20:19.039777+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:19:42.227198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:19:42.227198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:56.920091+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"358'58","reported_seq":974,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:28.978965+0000","last_change":"2024-03-11T21:20:28.978965+0000","last_active":"2024-03-11T21:20:28.978965+0000","last_peered":"2024-03-11T21:20:28.978965+0000","last_clean":"2024-03-11T21:20:28.978965+0000","last_became_active":"2024-03-11T21:00:48.473227+0000","last_became_peered":"2024-03-11T21:00:48.473227+0000","last_unstale":"2024-03-11T21:20:28.978965+0000","last_undegraded":"2024-03-11T21:20:28.978965+0000","last_fullsized":"2024-03-11T21:20:28.978965+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'58","last_scrub_stamp":"2024-03-11T21:20:28.978769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:20:28.978769+0000","objects_scrubbed":10,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:38.910616+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":87,"num_read_kb":72,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"424'15","reported_seq":264,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:29.564357+0000","last_change":"2024-03-11T21:19:51.677093+0000","last_active":"2024-03-11T21:20:29.564357+0000","last_peered":"2024-03-11T21:20:29.564357+0000","last_clean":"2024-03-11T21:20:29.564357+0000","last_became_active":"2024-03-11T21:14:50.431548+0000","last_became_peered":"2024-03-11T21:14:50.431548+0000","last_unstale":"2024-03-11T21:20:29.564357+0000","last_undegraded":"2024-03-11T21:20:29.564357+0000","last_fullsized":"2024-03-11T21:20:29.564357+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"424'15","last_scrub_stamp":"2024-03-11T21:19:51.676978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:19:51.676978+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:06.116410+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"184'130","reported_seq":576,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:22.654809+0000","last_change":"2024-03-11T21:20:22.654878+0000","last_active":"2024-03-11T21:20:22.654809+0000","last_peered":"2024-03-11T21:20:22.654809+0000","last_clean":"2024-03-11T21:20:22.654809+0000","last_became_active":"2024-03-11T21:18:53.662257+0000","last_became_peered":"2024-03-11T21:18:53.662257+0000","last_unstale":"2024-03-11T21:20:22.654809+0000","last_undegraded":"2024-03-11T21:20:22.654809+0000","last_fullsized":"2024-03-11T21:20:22.654809+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":3,"last_scrub":"184'130","last_scrub_stamp":"2024-03-11T21:20:22.654657+0000","last_deep_scrub":"184'130","last_deep_scrub_stamp":"2024-03-11T21:18:42.113943+0000","last_clean_scrub_stamp":"2024-03-11T21:20:22.654657+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:45.402190+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":439,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.265875+0000","last_change":"2024-03-11T21:20:16.570827+0000","last_active":"2024-03-11T21:20:19.265875+0000","last_peered":"2024-03-11T21:20:19.265875+0000","last_clean":"2024-03-11T21:20:19.265875+0000","last_became_active":"2024-03-11T21:18:53.658111+0000","last_became_peered":"2024-03-11T21:18:53.658111+0000","last_unstale":"2024-03-11T21:20:19.265875+0000","last_undegraded":"2024-03-11T21:20:19.265875+0000","last_fullsized":"2024-03-11T21:20:19.265875+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:20:16.570578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:03:43.345257+0000","last_clean_scrub_stamp":"2024-03-11T21:20:16.570578+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:21.286404+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":460,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.265598+0000","last_change":"2024-03-11T21:20:10.492980+0000","last_active":"2024-03-11T21:20:19.265598+0000","last_peered":"2024-03-11T21:20:19.265598+0000","last_clean":"2024-03-11T21:20:19.265598+0000","last_became_active":"2024-03-11T21:18:53.659495+0000","last_became_peered":"2024-03-11T21:18:53.659495+0000","last_unstale":"2024-03-11T21:20:19.265598+0000","last_undegraded":"2024-03-11T21:20:19.265598+0000","last_fullsized":"2024-03-11T21:20:19.265598+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:20:10.492743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:22.617999+0000","last_clean_scrub_stamp":"2024-03-11T21:20:10.492743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:12.391301+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"284'2","reported_seq":687,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.046270+0000","last_change":"2024-03-11T21:20:06.046409+0000","last_active":"2024-03-11T21:20:19.046270+0000","last_peered":"2024-03-11T21:20:19.046270+0000","last_clean":"2024-03-11T21:20:19.046270+0000","last_became_active":"2024-03-11T21:00:48.468682+0000","last_became_peered":"2024-03-11T21:00:48.468682+0000","last_unstale":"2024-03-11T21:20:19.046270+0000","last_undegraded":"2024-03-11T21:20:19.046270+0000","last_fullsized":"2024-03-11T21:20:19.046270+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"284'2","last_scrub_stamp":"2024-03-11T21:20:06.046086+0000","last_deep_scrub":"284'2","last_deep_scrub_stamp":"2024-03-11T21:06:47.645682+0000","last_clean_scrub_stamp":"2024-03-11T21:20:06.046086+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:23.271395+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"357'21","reported_seq":1014,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.052560+0000","last_change":"2024-03-11T21:19:14.618789+0000","last_active":"2024-03-11T21:20:19.052560+0000","last_peered":"2024-03-11T21:20:19.052560+0000","last_clean":"2024-03-11T21:20:19.052560+0000","last_became_active":"2024-03-11T21:14:50.435780+0000","last_became_peered":"2024-03-11T21:14:50.435780+0000","last_unstale":"2024-03-11T21:20:19.052560+0000","last_undegraded":"2024-03-11T21:20:19.052560+0000","last_fullsized":"2024-03-11T21:20:19.052560+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"357'21","last_scrub_stamp":"2024-03-11T21:19:14.618547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:19:14.618547+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:27.633037+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":31,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"424'13","reported_seq":1345,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:29.564382+0000","last_change":"2024-03-11T21:20:04.515837+0000","last_active":"2024-03-11T21:20:29.564382+0000","last_peered":"2024-03-11T21:20:29.564382+0000","last_clean":"2024-03-11T21:20:29.564382+0000","last_became_active":"2024-03-11T21:14:50.428398+0000","last_became_peered":"2024-03-11T21:14:50.428398+0000","last_unstale":"2024-03-11T21:20:29.564382+0000","last_undegraded":"2024-03-11T21:20:29.564382+0000","last_fullsized":"2024-03-11T21:20:29.564382+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"424'13","last_scrub_stamp":"2024-03-11T21:20:04.515611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:20:04.515611+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:14.281190+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"184'130","reported_seq":839,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.051440+0000","last_change":"2024-03-11T21:19:24.537746+0000","last_active":"2024-03-11T21:20:19.051440+0000","last_peered":"2024-03-11T21:20:19.051440+0000","last_clean":"2024-03-11T21:20:19.051440+0000","last_became_active":"2024-03-11T21:16:19.723401+0000","last_became_peered":"2024-03-11T21:16:19.723401+0000","last_unstale":"2024-03-11T21:20:19.051440+0000","last_undegraded":"2024-03-11T21:20:19.051440+0000","last_fullsized":"2024-03-11T21:20:19.051440+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":3,"last_scrub":"184'130","last_scrub_stamp":"2024-03-11T21:19:24.537515+0000","last_deep_scrub":"184'130","last_deep_scrub_stamp":"2024-03-11T21:18:01.642193+0000","last_clean_scrub_stamp":"2024-03-11T21:19:24.537515+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:33.665293+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":977,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.038864+0000","last_change":"2024-03-11T21:19:40.393255+0000","last_active":"2024-03-11T21:20:19.038864+0000","last_peered":"2024-03-11T21:20:19.038864+0000","last_clean":"2024-03-11T21:20:19.038864+0000","last_became_active":"2024-03-11T20:50:34.326947+0000","last_became_peered":"2024-03-11T20:50:34.326947+0000","last_unstale":"2024-03-11T21:20:19.038864+0000","last_undegraded":"2024-03-11T21:20:19.038864+0000","last_fullsized":"2024-03-11T21:20:19.038864+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:19:40.393138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:06:43.271453+0000","last_clean_scrub_stamp":"2024-03-11T21:19:40.393138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:54.482595+0000","scrub_duration":143,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"120'6","reported_seq":2660,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:34.321893+0000","last_change":"2024-03-11T21:20:05.533807+0000","last_active":"2024-03-11T21:20:34.321893+0000","last_peered":"2024-03-11T21:20:34.321893+0000","last_clean":"2024-03-11T21:20:34.321893+0000","last_became_active":"2024-03-11T21:00:48.468997+0000","last_became_peered":"2024-03-11T21:00:48.468997+0000","last_unstale":"2024-03-11T21:20:34.321893+0000","last_undegraded":"2024-03-11T21:20:34.321893+0000","last_fullsized":"2024-03-11T21:20:34.321893+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"120'6","last_scrub_stamp":"2024-03-11T21:20:05.533483+0000","last_deep_scrub":"120'6","last_deep_scrub_stamp":"2024-03-11T21:20:05.533483+0000","last_clean_scrub_stamp":"2024-03-11T21:20:05.533483+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:25.933555+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0035533370000000002,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1667,"num_read_kb":1662,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1002,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.083325+0000","last_change":"2024-03-11T21:20:18.792820+0000","last_active":"2024-03-11T21:20:19.083325+0000","last_peered":"2024-03-11T21:20:19.083325+0000","last_clean":"2024-03-11T21:20:19.083325+0000","last_became_active":"2024-03-11T20:50:28.071707+0000","last_became_peered":"2024-03-11T20:50:28.071707+0000","last_unstale":"2024-03-11T21:20:19.083325+0000","last_undegraded":"2024-03-11T21:20:19.083325+0000","last_fullsized":"2024-03-11T21:20:19.083325+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:20:18.792571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:23.417283+0000","last_clean_scrub_stamp":"2024-03-11T21:20:18.792571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:36.806503+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"497'38","reported_seq":207,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:35.911684+0000","last_change":"2024-03-11T21:20:16.561101+0000","last_active":"2024-03-11T21:20:35.911684+0000","last_peered":"2024-03-11T21:20:35.911684+0000","last_clean":"2024-03-11T21:20:35.911684+0000","last_became_active":"2024-03-11T21:14:50.418699+0000","last_became_peered":"2024-03-11T21:14:50.418699+0000","last_unstale":"2024-03-11T21:20:35.911684+0000","last_undegraded":"2024-03-11T21:20:35.911684+0000","last_fullsized":"2024-03-11T21:20:35.911684+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'36","last_scrub_stamp":"2024-03-11T21:20:16.560988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:29.034183+0000","last_clean_scrub_stamp":"2024-03-11T21:20:16.560988+0000","objects_scrubbed":6,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:19.815766+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":56,"num_read_kb":50,"num_write":38,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":473,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.265651+0000","last_change":"2024-03-11T21:19:36.390872+0000","last_active":"2024-03-11T21:20:19.265651+0000","last_peered":"2024-03-11T21:20:19.265651+0000","last_clean":"2024-03-11T21:20:19.265651+0000","last_became_active":"2024-03-11T21:18:53.663029+0000","last_became_peered":"2024-03-11T21:18:53.663029+0000","last_unstale":"2024-03-11T21:20:19.265651+0000","last_undegraded":"2024-03-11T21:20:19.265651+0000","last_fullsized":"2024-03-11T21:20:19.265651+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:19:36.390649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:19.695800+0000","last_clean_scrub_stamp":"2024-03-11T21:19:36.390649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:58.975394+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"184'130","reported_seq":846,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.045776+0000","last_change":"2024-03-11T21:19:30.931124+0000","last_active":"2024-03-11T21:20:19.045776+0000","last_peered":"2024-03-11T21:20:19.045776+0000","last_clean":"2024-03-11T21:20:19.045776+0000","last_became_active":"2024-03-11T21:16:24.835115+0000","last_became_peered":"2024-03-11T21:16:24.835115+0000","last_unstale":"2024-03-11T21:20:19.045776+0000","last_undegraded":"2024-03-11T21:20:19.045776+0000","last_fullsized":"2024-03-11T21:20:19.045776+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":444,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":4,"last_scrub":"184'130","last_scrub_stamp":"2024-03-11T21:19:30.930932+0000","last_deep_scrub":"184'130","last_deep_scrub_stamp":"2024-03-11T20:58:56.415267+0000","last_clean_scrub_stamp":"2024-03-11T21:19:30.930932+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:51.628370+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":939,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.051371+0000","last_change":"2024-03-11T21:20:16.567982+0000","last_active":"2024-03-11T21:20:19.051371+0000","last_peered":"2024-03-11T21:20:19.051371+0000","last_clean":"2024-03-11T21:20:19.051371+0000","last_became_active":"2024-03-11T20:50:34.335370+0000","last_became_peered":"2024-03-11T20:50:34.335370+0000","last_unstale":"2024-03-11T21:20:19.051371+0000","last_undegraded":"2024-03-11T21:20:19.051371+0000","last_fullsized":"2024-03-11T21:20:19.051371+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:20:16.567878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:17:04.544817+0000","last_clean_scrub_stamp":"2024-03-11T21:20:16.567878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:28.103994+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"486'19","reported_seq":684,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:29.566556+0000","last_change":"2024-03-11T21:19:55.366549+0000","last_active":"2024-03-11T21:20:29.566556+0000","last_peered":"2024-03-11T21:20:29.566556+0000","last_clean":"2024-03-11T21:20:29.566556+0000","last_became_active":"2024-03-11T21:18:53.667732+0000","last_became_peered":"2024-03-11T21:18:53.667732+0000","last_unstale":"2024-03-11T21:20:29.566556+0000","last_undegraded":"2024-03-11T21:20:29.566556+0000","last_fullsized":"2024-03-11T21:20:29.566556+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"486'19","last_scrub_stamp":"2024-03-11T21:19:55.366206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:19:55.366206+0000","objects_scrubbed":1,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:22.205065+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":1005,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.037826+0000","last_change":"2024-03-11T21:19:39.269234+0000","last_active":"2024-03-11T21:20:19.037826+0000","last_peered":"2024-03-11T21:20:19.037826+0000","last_clean":"2024-03-11T21:20:19.037826+0000","last_became_active":"2024-03-11T21:14:50.433372+0000","last_became_peered":"2024-03-11T21:14:50.433372+0000","last_unstale":"2024-03-11T21:20:19.037826+0000","last_undegraded":"2024-03-11T21:20:19.037826+0000","last_fullsized":"2024-03-11T21:20:19.037826+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:19:39.269015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:06:42.242801+0000","last_clean_scrub_stamp":"2024-03-11T21:19:39.269015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:53.783342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"466'29","reported_seq":417,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.266122+0000","last_change":"2024-03-11T21:19:47.432737+0000","last_active":"2024-03-11T21:20:19.266122+0000","last_peered":"2024-03-11T21:20:19.266122+0000","last_clean":"2024-03-11T21:20:19.266122+0000","last_became_active":"2024-03-11T21:18:53.662913+0000","last_became_peered":"2024-03-11T21:18:53.662913+0000","last_unstale":"2024-03-11T21:20:19.266122+0000","last_undegraded":"2024-03-11T21:20:19.266122+0000","last_fullsized":"2024-03-11T21:20:19.266122+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"466'29","last_scrub_stamp":"2024-03-11T21:19:47.432491+0000","last_deep_scrub":"466'29","last_deep_scrub_stamp":"2024-03-11T21:19:47.432491+0000","last_clean_scrub_stamp":"2024-03-11T21:19:47.432491+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:55.029516+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":40,"num_read_kb":31,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"184'130","reported_seq":842,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.068882+0000","last_change":"2024-03-11T21:20:00.576511+0000","last_active":"2024-03-11T21:20:19.068882+0000","last_peered":"2024-03-11T21:20:19.068882+0000","last_clean":"2024-03-11T21:20:19.068882+0000","last_became_active":"2024-03-11T21:11:28.396897+0000","last_became_peered":"2024-03-11T21:11:28.396897+0000","last_unstale":"2024-03-11T21:20:19.068882+0000","last_undegraded":"2024-03-11T21:20:19.068882+0000","last_fullsized":"2024-03-11T21:20:19.068882+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":4,"last_scrub":"184'130","last_scrub_stamp":"2024-03-11T21:20:00.576212+0000","last_deep_scrub":"184'130","last_deep_scrub_stamp":"2024-03-11T20:58:56.415267+0000","last_clean_scrub_stamp":"2024-03-11T21:20:00.576212+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:22.709835+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":996,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.070247+0000","last_change":"2024-03-11T21:20:07.762049+0000","last_active":"2024-03-11T21:20:19.070247+0000","last_peered":"2024-03-11T21:20:19.070247+0000","last_clean":"2024-03-11T21:20:19.070247+0000","last_became_active":"2024-03-11T21:14:50.434097+0000","last_became_peered":"2024-03-11T21:14:50.434097+0000","last_unstale":"2024-03-11T21:20:19.070247+0000","last_undegraded":"2024-03-11T21:20:19.070247+0000","last_fullsized":"2024-03-11T21:20:19.070247+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:20:07.761674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:02:27.775534+0000","last_clean_scrub_stamp":"2024-03-11T21:20:07.761674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:18.920560+0000","scrub_duration":159,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":681,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.046561+0000","last_change":"2024-03-11T21:19:14.926930+0000","last_active":"2024-03-11T21:20:19.046561+0000","last_peered":"2024-03-11T21:20:19.046561+0000","last_clean":"2024-03-11T21:20:19.046561+0000","last_became_active":"2024-03-11T21:00:48.469596+0000","last_became_peered":"2024-03-11T21:00:48.469596+0000","last_unstale":"2024-03-11T21:20:19.046561+0000","last_undegraded":"2024-03-11T21:20:19.046561+0000","last_fullsized":"2024-03-11T21:20:19.046561+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:19:14.926693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:19:14.926693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:33.263951+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":674,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.045412+0000","last_change":"2024-03-11T21:19:44.140736+0000","last_active":"2024-03-11T21:20:19.045412+0000","last_peered":"2024-03-11T21:20:19.045412+0000","last_clean":"2024-03-11T21:20:19.045412+0000","last_became_active":"2024-03-11T21:00:48.466084+0000","last_became_peered":"2024-03-11T21:00:48.466084+0000","last_unstale":"2024-03-11T21:20:19.045412+0000","last_undegraded":"2024-03-11T21:20:19.045412+0000","last_fullsized":"2024-03-11T21:20:19.045412+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:19:44.140446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:16:40.924953+0000","last_clean_scrub_stamp":"2024-03-11T21:19:44.140446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:46.051422+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"358'46","reported_seq":1075,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.049868+0000","last_change":"2024-03-11T21:19:54.480119+0000","last_active":"2024-03-11T21:20:19.049868+0000","last_peered":"2024-03-11T21:20:19.049868+0000","last_clean":"2024-03-11T21:20:19.049868+0000","last_became_active":"2024-03-11T21:00:48.472191+0000","last_became_peered":"2024-03-11T21:00:48.472191+0000","last_unstale":"2024-03-11T21:20:19.049868+0000","last_undegraded":"2024-03-11T21:20:19.049868+0000","last_fullsized":"2024-03-11T21:20:19.049868+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'46","last_scrub_stamp":"2024-03-11T21:19:54.479744+0000","last_deep_scrub":"116'22","last_deep_scrub_stamp":"2024-03-11T20:53:47.364709+0000","last_clean_scrub_stamp":"2024-03-11T21:19:54.479744+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:58.051113+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"184'130","reported_seq":1278,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:32.403033+0000","last_change":"2024-03-11T21:20:32.403101+0000","last_active":"2024-03-11T21:20:32.403033+0000","last_peered":"2024-03-11T21:20:32.403033+0000","last_clean":"2024-03-11T21:20:32.403033+0000","last_became_active":"2024-03-11T21:11:28.405250+0000","last_became_peered":"2024-03-11T21:11:28.405250+0000","last_unstale":"2024-03-11T21:20:32.403033+0000","last_undegraded":"2024-03-11T21:20:32.403033+0000","last_fullsized":"2024-03-11T21:20:32.403033+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":4,"last_scrub":"184'130","last_scrub_stamp":"2024-03-11T21:20:32.402867+0000","last_deep_scrub":"184'130","last_deep_scrub_stamp":"2024-03-11T21:20:32.402867+0000","last_clean_scrub_stamp":"2024-03-11T21:20:32.402867+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:56.064818+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":476,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.267093+0000","last_change":"2024-03-11T21:20:11.575856+0000","last_active":"2024-03-11T21:20:19.267093+0000","last_peered":"2024-03-11T21:20:19.267093+0000","last_clean":"2024-03-11T21:20:19.267093+0000","last_became_active":"2024-03-11T21:18:53.661803+0000","last_became_peered":"2024-03-11T21:18:53.661803+0000","last_unstale":"2024-03-11T21:20:19.267093+0000","last_undegraded":"2024-03-11T21:20:19.267093+0000","last_fullsized":"2024-03-11T21:20:19.267093+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:20:11.575635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:20:11.575635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:23.706717+0000","scrub_duration":71,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":682,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:20.123661+0000","last_change":"2024-03-11T21:20:20.123703+0000","last_active":"2024-03-11T21:20:20.123661+0000","last_peered":"2024-03-11T21:20:20.123661+0000","last_clean":"2024-03-11T21:20:20.123661+0000","last_became_active":"2024-03-11T21:00:48.461367+0000","last_became_peered":"2024-03-11T21:00:48.461367+0000","last_unstale":"2024-03-11T21:20:20.123661+0000","last_undegraded":"2024-03-11T21:20:20.123661+0000","last_fullsized":"2024-03-11T21:20:20.123661+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:20:20.123568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:03:37.601229+0000","last_clean_scrub_stamp":"2024-03-11T21:20:20.123568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:32.356002+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"358'25","reported_seq":1026,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:20.624294+0000","last_change":"2024-03-11T21:20:20.624418+0000","last_active":"2024-03-11T21:20:20.624294+0000","last_peered":"2024-03-11T21:20:20.624294+0000","last_clean":"2024-03-11T21:20:20.624294+0000","last_became_active":"2024-03-11T21:14:50.436328+0000","last_became_peered":"2024-03-11T21:14:50.436328+0000","last_unstale":"2024-03-11T21:20:20.624294+0000","last_undegraded":"2024-03-11T21:20:20.624294+0000","last_fullsized":"2024-03-11T21:20:20.624294+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'25","last_scrub_stamp":"2024-03-11T21:20:20.624084+0000","last_deep_scrub":"358'25","last_deep_scrub_stamp":"2024-03-11T21:20:20.624084+0000","last_clean_scrub_stamp":"2024-03-11T21:20:20.624084+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:21.095797+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":491,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:33.733203+0000","last_change":"2024-03-11T21:20:33.733263+0000","last_active":"2024-03-11T21:20:33.733203+0000","last_peered":"2024-03-11T21:20:33.733203+0000","last_clean":"2024-03-11T21:20:33.733203+0000","last_became_active":"2024-03-11T21:18:53.655669+0000","last_became_peered":"2024-03-11T21:18:53.655669+0000","last_unstale":"2024-03-11T21:20:33.733203+0000","last_undegraded":"2024-03-11T21:20:33.733203+0000","last_fullsized":"2024-03-11T21:20:33.733203+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:20:33.733043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:35.670149+0000","last_clean_scrub_stamp":"2024-03-11T21:20:33.733043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:52.154952+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"184'130","reported_seq":842,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:34.610511+0000","last_change":"2024-03-11T21:20:34.610582+0000","last_active":"2024-03-11T21:20:34.610511+0000","last_peered":"2024-03-11T21:20:34.610511+0000","last_clean":"2024-03-11T21:20:34.610511+0000","last_became_active":"2024-03-11T21:11:28.398728+0000","last_became_peered":"2024-03-11T21:11:28.398728+0000","last_unstale":"2024-03-11T21:20:34.610511+0000","last_undegraded":"2024-03-11T21:20:34.610511+0000","last_fullsized":"2024-03-11T21:20:34.610511+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":4,"last_scrub":"184'130","last_scrub_stamp":"2024-03-11T21:20:34.610363+0000","last_deep_scrub":"184'130","last_deep_scrub_stamp":"2024-03-11T21:19:04.421375+0000","last_clean_scrub_stamp":"2024-03-11T21:20:34.610363+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:59.872744+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":664,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.075500+0000","last_change":"2024-03-11T21:20:10.061606+0000","last_active":"2024-03-11T21:20:19.075500+0000","last_peered":"2024-03-11T21:20:19.075500+0000","last_clean":"2024-03-11T21:20:19.075500+0000","last_became_active":"2024-03-11T21:17:19.515538+0000","last_became_peered":"2024-03-11T21:17:19.515538+0000","last_unstale":"2024-03-11T21:20:19.075500+0000","last_undegraded":"2024-03-11T21:20:19.075500+0000","last_fullsized":"2024-03-11T21:20:19.075500+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:20:10.061493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:33.234063+0000","last_clean_scrub_stamp":"2024-03-11T21:20:10.061493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:16.575596+0000","scrub_duration":223,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"486'23","reported_seq":703,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:31.739054+0000","last_change":"2024-03-11T21:20:31.739054+0000","last_active":"2024-03-11T21:20:31.739054+0000","last_peered":"2024-03-11T21:20:31.739054+0000","last_clean":"2024-03-11T21:20:31.739054+0000","last_became_active":"2024-03-11T21:18:53.663691+0000","last_became_peered":"2024-03-11T21:18:53.663691+0000","last_unstale":"2024-03-11T21:20:31.739054+0000","last_undegraded":"2024-03-11T21:20:31.739054+0000","last_fullsized":"2024-03-11T21:20:31.739054+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"486'23","last_scrub_stamp":"2024-03-11T21:20:31.738783+0000","last_deep_scrub":"466'22","last_deep_scrub_stamp":"2024-03-11T21:17:29.394614+0000","last_clean_scrub_stamp":"2024-03-11T21:20:31.738783+0000","objects_scrubbed":1,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:44.762544+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":981,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.038930+0000","last_change":"2024-03-11T21:19:14.348051+0000","last_active":"2024-03-11T21:20:19.038930+0000","last_peered":"2024-03-11T21:20:19.038930+0000","last_clean":"2024-03-11T21:20:19.038930+0000","last_became_active":"2024-03-11T21:00:27.970573+0000","last_became_peered":"2024-03-11T21:00:27.970573+0000","last_unstale":"2024-03-11T21:20:19.038930+0000","last_undegraded":"2024-03-11T21:20:19.038930+0000","last_fullsized":"2024-03-11T21:20:19.038930+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:19:14.347834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:11:14.836808+0000","last_clean_scrub_stamp":"2024-03-11T21:19:14.347834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:43.922964+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"358'60","reported_seq":1147,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.082240+0000","last_change":"2024-03-11T21:19:19.534707+0000","last_active":"2024-03-11T21:20:19.082240+0000","last_peered":"2024-03-11T21:20:19.082240+0000","last_clean":"2024-03-11T21:20:19.082240+0000","last_became_active":"2024-03-11T21:00:48.469854+0000","last_became_peered":"2024-03-11T21:00:48.469854+0000","last_unstale":"2024-03-11T21:20:19.082240+0000","last_undegraded":"2024-03-11T21:20:19.082240+0000","last_fullsized":"2024-03-11T21:20:19.082240+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'60","last_scrub_stamp":"2024-03-11T21:19:19.534453+0000","last_deep_scrub":"161'28","last_deep_scrub_stamp":"2024-03-11T20:57:20.816278+0000","last_clean_scrub_stamp":"2024-03-11T21:19:19.534453+0000","objects_scrubbed":9,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:34.244400+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":80,"num_read_kb":69,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.f","version":"184'130","reported_seq":864,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:21.905502+0000","last_change":"2024-03-11T21:20:21.905579+0000","last_active":"2024-03-11T21:20:21.905502+0000","last_peered":"2024-03-11T21:20:21.905502+0000","last_clean":"2024-03-11T21:20:21.905502+0000","last_became_active":"2024-03-11T21:16:24.834591+0000","last_became_peered":"2024-03-11T21:16:24.834591+0000","last_unstale":"2024-03-11T21:20:21.905502+0000","last_undegraded":"2024-03-11T21:20:21.905502+0000","last_fullsized":"2024-03-11T21:20:21.905502+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":444,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":4,"last_scrub":"184'130","last_scrub_stamp":"2024-03-11T21:20:21.905300+0000","last_deep_scrub":"184'130","last_deep_scrub_stamp":"2024-03-11T21:20:21.905300+0000","last_clean_scrub_stamp":"2024-03-11T21:20:21.905300+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:22.380704+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":968,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.037569+0000","last_change":"2024-03-11T21:19:28.257284+0000","last_active":"2024-03-11T21:20:19.037569+0000","last_peered":"2024-03-11T21:20:19.037569+0000","last_clean":"2024-03-11T21:20:19.037569+0000","last_became_active":"2024-03-11T20:50:34.327139+0000","last_became_peered":"2024-03-11T20:50:34.327139+0000","last_unstale":"2024-03-11T21:20:19.037569+0000","last_undegraded":"2024-03-11T21:20:19.037569+0000","last_fullsized":"2024-03-11T21:20:19.037569+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:19:28.257070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:09:53.436482+0000","last_clean_scrub_stamp":"2024-03-11T21:19:28.257070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:30.893734+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":683,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:28.768827+0000","last_change":"2024-03-11T21:20:28.768875+0000","last_active":"2024-03-11T21:20:28.768827+0000","last_peered":"2024-03-11T21:20:28.768827+0000","last_clean":"2024-03-11T21:20:28.768827+0000","last_became_active":"2024-03-11T21:17:19.980258+0000","last_became_peered":"2024-03-11T21:17:19.980258+0000","last_unstale":"2024-03-11T21:20:28.768827+0000","last_undegraded":"2024-03-11T21:20:28.768827+0000","last_fullsized":"2024-03-11T21:20:28.768827+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:20:28.768737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:57:17.447191+0000","last_clean_scrub_stamp":"2024-03-11T21:20:28.768737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:53.151786+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"358'53","reported_seq":1054,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.040163+0000","last_change":"2024-03-11T21:19:25.291742+0000","last_active":"2024-03-11T21:20:19.040163+0000","last_peered":"2024-03-11T21:20:19.040163+0000","last_clean":"2024-03-11T21:20:19.040163+0000","last_became_active":"2024-03-11T20:50:30.111819+0000","last_became_peered":"2024-03-11T20:50:30.111819+0000","last_unstale":"2024-03-11T21:20:19.040163+0000","last_undegraded":"2024-03-11T21:20:19.040163+0000","last_fullsized":"2024-03-11T21:20:19.040163+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'53","last_scrub_stamp":"2024-03-11T21:19:25.291454+0000","last_deep_scrub":"102'25","last_deep_scrub_stamp":"2024-03-11T20:51:35.187094+0000","last_clean_scrub_stamp":"2024-03-11T21:19:25.291454+0000","objects_scrubbed":8,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:41.693362+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":61,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":999,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.081762+0000","last_change":"2024-03-11T21:19:49.767793+0000","last_active":"2024-03-11T21:20:19.081762+0000","last_peered":"2024-03-11T21:20:19.081762+0000","last_clean":"2024-03-11T21:20:19.081762+0000","last_became_active":"2024-03-11T21:00:48.362389+0000","last_became_peered":"2024-03-11T21:00:48.362389+0000","last_unstale":"2024-03-11T21:20:19.081762+0000","last_undegraded":"2024-03-11T21:20:19.081762+0000","last_fullsized":"2024-03-11T21:20:19.081762+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:19:49.767576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:31.136392+0000","last_clean_scrub_stamp":"2024-03-11T21:19:49.767576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:09.061465+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"184'130","reported_seq":721,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.574835+0000","last_change":"2024-03-11T21:20:19.574958+0000","last_active":"2024-03-11T21:20:19.574835+0000","last_peered":"2024-03-11T21:20:19.574835+0000","last_clean":"2024-03-11T21:20:19.574835+0000","last_became_active":"2024-03-11T21:18:53.660113+0000","last_became_peered":"2024-03-11T21:18:53.660113+0000","last_unstale":"2024-03-11T21:20:19.574835+0000","last_undegraded":"2024-03-11T21:20:19.574835+0000","last_fullsized":"2024-03-11T21:20:19.574835+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":444,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":4,"last_scrub":"184'130","last_scrub_stamp":"2024-03-11T21:20:19.574640+0000","last_deep_scrub":"184'130","last_deep_scrub_stamp":"2024-03-11T21:16:24.952261+0000","last_clean_scrub_stamp":"2024-03-11T21:20:19.574640+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:24.727860+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":182,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.264004+0000","last_change":"2024-03-11T21:19:13.617810+0000","last_active":"2024-03-11T21:20:19.264004+0000","last_peered":"2024-03-11T21:20:19.264004+0000","last_clean":"2024-03-11T21:20:19.264004+0000","last_became_active":"2024-03-11T21:14:50.706718+0000","last_became_peered":"2024-03-11T21:14:50.706718+0000","last_unstale":"2024-03-11T21:20:19.264004+0000","last_undegraded":"2024-03-11T21:20:19.264004+0000","last_fullsized":"2024-03-11T21:20:19.264004+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:19:13.617698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:55:58.888890+0000","last_clean_scrub_stamp":"2024-03-11T21:19:13.617698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:38.668712+0000","scrub_duration":159,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":681,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:24.069348+0000","last_change":"2024-03-11T21:20:24.069459+0000","last_active":"2024-03-11T21:20:24.069348+0000","last_peered":"2024-03-11T21:20:24.069348+0000","last_clean":"2024-03-11T21:20:24.069348+0000","last_became_active":"2024-03-11T21:00:48.469015+0000","last_became_peered":"2024-03-11T21:00:48.469015+0000","last_unstale":"2024-03-11T21:20:24.069348+0000","last_undegraded":"2024-03-11T21:20:24.069348+0000","last_fullsized":"2024-03-11T21:20:24.069348+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:20:24.069148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:20:24.069148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:40.926396+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"393'35","reported_seq":1029,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.069855+0000","last_change":"2024-03-11T21:19:41.728127+0000","last_active":"2024-03-11T21:20:19.069855+0000","last_peered":"2024-03-11T21:20:19.069855+0000","last_clean":"2024-03-11T21:20:19.069855+0000","last_became_active":"2024-03-11T20:50:30.109549+0000","last_became_peered":"2024-03-11T20:50:30.109549+0000","last_unstale":"2024-03-11T21:20:19.069855+0000","last_undegraded":"2024-03-11T21:20:19.069855+0000","last_fullsized":"2024-03-11T21:20:19.069855+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"393'35","last_scrub_stamp":"2024-03-11T21:19:41.727883+0000","last_deep_scrub":"393'35","last_deep_scrub_stamp":"2024-03-11T21:12:35.497207+0000","last_clean_scrub_stamp":"2024-03-11T21:19:41.727883+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:48.190520+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":190,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:35.508824+0000","last_change":"2024-03-11T21:20:35.508856+0000","last_active":"2024-03-11T21:20:35.508824+0000","last_peered":"2024-03-11T21:20:35.508824+0000","last_clean":"2024-03-11T21:20:35.508824+0000","last_became_active":"2024-03-11T21:14:50.682439+0000","last_became_peered":"2024-03-11T21:14:50.682439+0000","last_unstale":"2024-03-11T21:20:35.508824+0000","last_undegraded":"2024-03-11T21:20:35.508824+0000","last_fullsized":"2024-03-11T21:20:35.508824+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:20:35.508732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:19:22.384677+0000","last_clean_scrub_stamp":"2024-03-11T21:20:35.508732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:22:03.179267+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"184'130","reported_seq":669,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.039171+0000","last_change":"2024-03-11T21:20:15.310738+0000","last_active":"2024-03-11T21:20:19.039171+0000","last_peered":"2024-03-11T21:20:19.039171+0000","last_clean":"2024-03-11T21:20:19.039171+0000","last_became_active":"2024-03-11T21:16:24.829946+0000","last_became_peered":"2024-03-11T21:16:24.829946+0000","last_unstale":"2024-03-11T21:20:19.039171+0000","last_undegraded":"2024-03-11T21:20:19.039171+0000","last_fullsized":"2024-03-11T21:20:19.039171+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":444,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":4,"last_scrub":"184'130","last_scrub_stamp":"2024-03-11T21:20:15.310493+0000","last_deep_scrub":"184'130","last_deep_scrub_stamp":"2024-03-11T21:20:15.310493+0000","last_clean_scrub_stamp":"2024-03-11T21:20:15.310493+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:43.111774+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":258,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.068544+0000","last_change":"2024-03-11T21:19:32.787154+0000","last_active":"2024-03-11T21:20:19.068544+0000","last_peered":"2024-03-11T21:20:19.068544+0000","last_clean":"2024-03-11T21:20:19.068544+0000","last_became_active":"2024-03-11T21:14:50.427625+0000","last_became_peered":"2024-03-11T21:14:50.427625+0000","last_unstale":"2024-03-11T21:20:19.068544+0000","last_undegraded":"2024-03-11T21:20:19.068544+0000","last_fullsized":"2024-03-11T21:20:19.068544+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:19:32.786935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:19:32.786935+0000","last_clean_scrub_stamp":"2024-03-11T21:19:32.786935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:20:42.720188+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":675,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.075478+0000","last_change":"2024-03-11T21:20:03.869426+0000","last_active":"2024-03-11T21:20:19.075478+0000","last_peered":"2024-03-11T21:20:19.075478+0000","last_clean":"2024-03-11T21:20:19.075478+0000","last_became_active":"2024-03-11T21:17:19.515138+0000","last_became_peered":"2024-03-11T21:17:19.515138+0000","last_unstale":"2024-03-11T21:20:19.075478+0000","last_undegraded":"2024-03-11T21:20:19.075478+0000","last_fullsized":"2024-03-11T21:20:19.075478+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:20:03.869258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:50:26.989698+0000","last_clean_scrub_stamp":"2024-03-11T21:20:03.869258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:08.713930+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"391'46","reported_seq":1089,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.040120+0000","last_change":"2024-03-11T21:20:09.372004+0000","last_active":"2024-03-11T21:20:19.040120+0000","last_peered":"2024-03-11T21:20:19.040120+0000","last_clean":"2024-03-11T21:20:19.040120+0000","last_became_active":"2024-03-11T21:14:50.427841+0000","last_became_peered":"2024-03-11T21:14:50.427841+0000","last_unstale":"2024-03-11T21:20:19.040120+0000","last_undegraded":"2024-03-11T21:20:19.040120+0000","last_fullsized":"2024-03-11T21:20:19.040120+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"391'46","last_scrub_stamp":"2024-03-11T21:20:09.371762+0000","last_deep_scrub":"391'46","last_deep_scrub_stamp":"2024-03-11T21:12:31.145006+0000","last_clean_scrub_stamp":"2024-03-11T21:20:09.371762+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:27.902715+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":975,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-03-11T21:20:19.068403+0000","last_change":"2024-03-11T21:19:47.620205+0000","last_active":"2024-03-11T21:20:19.068403+0000","last_peered":"2024-03-11T21:20:19.068403+0000","last_clean":"2024-03-11T21:20:19.068403+0000","last_became_active":"2024-03-11T20:50:32.652854+0000","last_became_peered":"2024-03-11T20:50:32.652854+0000","last_unstale":"2024-03-11T21:20:19.068403+0000","last_undegraded":"2024-03-11T21:20:19.068403+0000","last_fullsized":"2024-03-11T21:20:19.068403+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:19:47.619918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:59:00.946094+0000","last_clean_scrub_stamp":"2024-03-11T21:19:47.619918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:21:08.341012+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":20,"num_read_kb":20,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1914,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2730,"ondisk_log_size":2730,"up":63,"acting":63,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1561,"num_read_kb":1318,"num_write":1003,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":255,"num_bytes_recovered":3878,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3632,"internal_metadata":0},"log_size":1102,"ondisk_log_size":1102,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":134,"ondisk_log_size":134,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1667,"num_read_kb":1662,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":68,"seq":292057776501,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":631672,"kb_used_data":752,"kb_used_omap":1,"kb_used_meta":630910,"kb_avail":93113480,"statfs":{"total":95995035648,"available":95348203520,"internally_reserved":0,"allocated":770048,"data_stored":433167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":646052298},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":57,"seq":244813136255,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":537576,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":536510,"kb_avail":93207576,"statfs":{"total":95995035648,"available":95444557824,"internally_reserved":0,"allocated":1081344,"data_stored":760857,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":549386695},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":211,"seq":906238099695,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":510448,"kb_used_data":744,"kb_used_omap":7,"kb_used_meta":509688,"kb_avail":93234704,"statfs":{"total":95995035648,"available":95472336896,"internally_reserved":0,"allocated":761856,"data_stored":433423,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8078,"internal_metadata":521920626},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738813,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":595888,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":594814,"kb_avail":93149264,"statfs":{"total":95995035648,"available":95384846336,"internally_reserved":0,"allocated":1089536,"data_stored":759789,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":609089989},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604379048,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":577764,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":576766,"kb_avail":93167388,"statfs":{"total":95995035648,"available":95403405312,"internally_reserved":0,"allocated":1011712,"data_stored":694959,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":590608837},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":28,"seq":120259084701,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":672488,"kb_used_data":1312,"kb_used_omap":1,"kb_used_meta":671166,"kb_avail":93072664,"statfs":{"total":95995035648,"available":95306407936,"internally_reserved":0,"allocated":1343488,"data_stored":1023015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":687274440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":204,"seq":876173328628,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":424648,"kb_used_data":768,"kb_used_omap":4,"kb_used_meta":423867,"kb_avail":93320504,"statfs":{"total":95995035648,"available":95560196096,"internally_reserved":0,"allocated":786432,"data_stored":434037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4451,"internal_metadata":434040477},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":44,"seq":188978561417,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":549992,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":548990,"kb_avail":93195160,"statfs":{"total":95995035648,"available":95431843840,"internally_reserved":0,"allocated":1015808,"data_stored":694881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":562166213},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3632,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T21:20:38.887 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-03-11T21:20:38.887 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 -- ceph config rm mgr mgr_debug_aggressive_pg_num_changes 2024-03-11T21:20:39.023 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:38 smithi023 bash[29146]: cluster 2024-03-11T21:20:37.379523+0000 osd.7 (osd.7) 737 : cluster 0 6.8 scrub starts 2024-03-11T21:20:39.023 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:38 smithi023 bash[29146]: cluster 2024-03-11T21:20:37.382270+0000 osd.7 (osd.7) 738 : cluster 0 6.8 scrub ok 2024-03-11T21:20:39.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:38 smithi023 bash[29146]: cluster 2024-03-11T21:20:37.956556+0000 osd.6 (osd.6) 584 : cluster 0 2.17 scrub starts 2024-03-11T21:20:39.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:38 smithi023 bash[29146]: cluster 2024-03-11T21:20:37.959078+0000 osd.6 (osd.6) 585 : cluster 0 2.17 scrub ok 2024-03-11T21:20:39.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:38 smithi023 bash[21372]: cluster 2024-03-11T21:20:37.379523+0000 osd.7 (osd.7) 737 : cluster 0 6.8 scrub starts 2024-03-11T21:20:39.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:38 smithi023 bash[21372]: cluster 2024-03-11T21:20:37.382270+0000 osd.7 (osd.7) 738 : cluster 0 6.8 scrub ok 2024-03-11T21:20:39.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:38 smithi023 bash[21372]: cluster 2024-03-11T21:20:37.956556+0000 osd.6 (osd.6) 584 : cluster 0 2.17 scrub starts 2024-03-11T21:20:39.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:38 smithi023 bash[21372]: cluster 2024-03-11T21:20:37.959078+0000 osd.6 (osd.6) 585 : cluster 0 2.17 scrub ok 2024-03-11T21:20:39.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:38 smithi146 bash[23809]: cluster 2024-03-11T21:20:37.379523+0000 osd.7 (osd.7) 737 : cluster 0 6.8 scrub starts 2024-03-11T21:20:39.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:38 smithi146 bash[23809]: cluster 2024-03-11T21:20:37.382270+0000 osd.7 (osd.7) 738 : cluster 0 6.8 scrub ok 2024-03-11T21:20:39.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:38 smithi146 bash[23809]: cluster 2024-03-11T21:20:37.956556+0000 osd.6 (osd.6) 584 : cluster 0 2.17 scrub starts 2024-03-11T21:20:39.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:38 smithi146 bash[23809]: cluster 2024-03-11T21:20:37.959078+0000 osd.6 (osd.6) 585 : cluster 0 2.17 scrub ok 2024-03-11T21:20:40.251 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:39 smithi146 bash[23809]: audit 2024-03-11T21:20:38.004755+0000 mgr.y (mgr.24380) 1100 : audit 0 from='client.15879 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:20:40.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:39 smithi146 bash[23809]: cluster 2024-03-11T21:20:38.608053+0000 mgr.y (mgr.24380) 1101 : cluster 0 pgmap v1271: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 7.1 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-11T21:20:40.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:39 smithi146 bash[23809]: cluster 2024-03-11T21:20:39.450894+0000 osd.0 (osd.0) 544 : cluster 0 6.10 scrub starts 2024-03-11T21:20:40.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:39 smithi146 bash[23809]: cluster 2024-03-11T21:20:39.535998+0000 osd.0 (osd.0) 545 : cluster 0 6.10 scrub ok 2024-03-11T21:20:40.273 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:39 smithi023 bash[21372]: audit 2024-03-11T21:20:38.004755+0000 mgr.y (mgr.24380) 1100 : audit 0 from='client.15879 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:20:40.273 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:39 smithi023 bash[21372]: cluster 2024-03-11T21:20:38.608053+0000 mgr.y (mgr.24380) 1101 : cluster 0 pgmap v1271: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 7.1 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-11T21:20:40.273 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:39 smithi023 bash[21372]: cluster 2024-03-11T21:20:39.450894+0000 osd.0 (osd.0) 544 : cluster 0 6.10 scrub starts 2024-03-11T21:20:40.273 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:39 smithi023 bash[21372]: cluster 2024-03-11T21:20:39.535998+0000 osd.0 (osd.0) 545 : cluster 0 6.10 scrub ok 2024-03-11T21:20:40.273 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:39 smithi023 bash[29146]: audit 2024-03-11T21:20:38.004755+0000 mgr.y (mgr.24380) 1100 : audit 0 from='client.15879 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:20:40.274 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:39 smithi023 bash[29146]: cluster 2024-03-11T21:20:38.608053+0000 mgr.y (mgr.24380) 1101 : cluster 0 pgmap v1271: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 7.1 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-11T21:20:40.274 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:39 smithi023 bash[29146]: cluster 2024-03-11T21:20:39.450894+0000 osd.0 (osd.0) 544 : cluster 0 6.10 scrub starts 2024-03-11T21:20:40.274 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:39 smithi023 bash[29146]: cluster 2024-03-11T21:20:39.535998+0000 osd.0 (osd.0) 545 : cluster 0 6.10 scrub ok 2024-03-11T21:20:42.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:41 smithi146 bash[23809]: cluster 2024-03-11T21:20:40.609082+0000 mgr.y (mgr.24380) 1102 : cluster 0 pgmap v1272: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 27 op/s 2024-03-11T21:20:42.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:41 smithi146 bash[23809]: cluster 2024-03-11T21:20:40.805786+0000 osd.2 (osd.2) 331 : cluster 0 1.14 deep-scrub starts 2024-03-11T21:20:42.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:41 smithi146 bash[23809]: cluster 2024-03-11T21:20:40.808536+0000 osd.2 (osd.2) 332 : cluster 0 1.14 deep-scrub ok 2024-03-11T21:20:42.023 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:41 smithi023 bash[21372]: cluster 2024-03-11T21:20:40.609082+0000 mgr.y (mgr.24380) 1102 : cluster 0 pgmap v1272: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 27 op/s 2024-03-11T21:20:42.023 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:41 smithi023 bash[21372]: cluster 2024-03-11T21:20:40.805786+0000 osd.2 (osd.2) 331 : cluster 0 1.14 deep-scrub starts 2024-03-11T21:20:42.023 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:41 smithi023 bash[21372]: cluster 2024-03-11T21:20:40.808536+0000 osd.2 (osd.2) 332 : cluster 0 1.14 deep-scrub ok 2024-03-11T21:20:42.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:41 smithi023 bash[29146]: cluster 2024-03-11T21:20:40.609082+0000 mgr.y (mgr.24380) 1102 : cluster 0 pgmap v1272: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 27 op/s 2024-03-11T21:20:42.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:41 smithi023 bash[29146]: cluster 2024-03-11T21:20:40.805786+0000 osd.2 (osd.2) 331 : cluster 0 1.14 deep-scrub starts 2024-03-11T21:20:42.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:41 smithi023 bash[29146]: cluster 2024-03-11T21:20:40.808536+0000 osd.2 (osd.2) 332 : cluster 0 1.14 deep-scrub ok 2024-03-11T21:20:43.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:42 smithi146 bash[23809]: cluster 2024-03-11T21:20:40.880687+0000 osd.6 (osd.6) 586 : cluster 0 3.a scrub starts 2024-03-11T21:20:43.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:42 smithi146 bash[23809]: cluster 2024-03-11T21:20:40.885821+0000 osd.6 (osd.6) 587 : cluster 0 3.a scrub ok 2024-03-11T21:20:43.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:42 smithi146 bash[23809]: cluster 2024-03-11T21:20:41.371630+0000 osd.7 (osd.7) 739 : cluster 0 6.1c scrub starts 2024-03-11T21:20:43.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:42 smithi146 bash[23809]: cluster 2024-03-11T21:20:41.375246+0000 osd.7 (osd.7) 740 : cluster 0 6.1c scrub ok 2024-03-11T21:20:43.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:42 smithi146 bash[23809]: cluster 2024-03-11T21:20:41.806795+0000 osd.4 (osd.4) 193 : cluster 0 6.1a deep-scrub starts 2024-03-11T21:20:43.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:42 smithi146 bash[23809]: cluster 2024-03-11T21:20:41.875920+0000 osd.4 (osd.4) 194 : cluster 0 6.1a deep-scrub ok 2024-03-11T21:20:43.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:42 smithi146 bash[23809]: cluster 2024-03-11T21:20:41.916260+0000 osd.6 (osd.6) 588 : cluster 0 1.7 scrub starts 2024-03-11T21:20:43.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:42 smithi146 bash[23809]: cluster 2024-03-11T21:20:41.918774+0000 osd.6 (osd.6) 589 : cluster 0 1.7 scrub ok 2024-03-11T21:20:43.023 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:42 smithi023 bash[21372]: cluster 2024-03-11T21:20:40.880687+0000 osd.6 (osd.6) 586 : cluster 0 3.a scrub starts 2024-03-11T21:20:43.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:42 smithi023 bash[21372]: cluster 2024-03-11T21:20:40.885821+0000 osd.6 (osd.6) 587 : cluster 0 3.a scrub ok 2024-03-11T21:20:43.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:42 smithi023 bash[21372]: cluster 2024-03-11T21:20:41.371630+0000 osd.7 (osd.7) 739 : cluster 0 6.1c scrub starts 2024-03-11T21:20:43.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:42 smithi023 bash[21372]: cluster 2024-03-11T21:20:41.375246+0000 osd.7 (osd.7) 740 : cluster 0 6.1c scrub ok 2024-03-11T21:20:43.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:42 smithi023 bash[21372]: cluster 2024-03-11T21:20:41.806795+0000 osd.4 (osd.4) 193 : cluster 0 6.1a deep-scrub starts 2024-03-11T21:20:43.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:42 smithi023 bash[21372]: cluster 2024-03-11T21:20:41.875920+0000 osd.4 (osd.4) 194 : cluster 0 6.1a deep-scrub ok 2024-03-11T21:20:43.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:42 smithi023 bash[21372]: cluster 2024-03-11T21:20:41.916260+0000 osd.6 (osd.6) 588 : cluster 0 1.7 scrub starts 2024-03-11T21:20:43.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:42 smithi023 bash[21372]: cluster 2024-03-11T21:20:41.918774+0000 osd.6 (osd.6) 589 : cluster 0 1.7 scrub ok 2024-03-11T21:20:43.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:42 smithi023 bash[29146]: cluster 2024-03-11T21:20:40.880687+0000 osd.6 (osd.6) 586 : cluster 0 3.a scrub starts 2024-03-11T21:20:43.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:42 smithi023 bash[29146]: cluster 2024-03-11T21:20:40.885821+0000 osd.6 (osd.6) 587 : cluster 0 3.a scrub ok 2024-03-11T21:20:43.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:42 smithi023 bash[29146]: cluster 2024-03-11T21:20:41.371630+0000 osd.7 (osd.7) 739 : cluster 0 6.1c scrub starts 2024-03-11T21:20:43.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:42 smithi023 bash[29146]: cluster 2024-03-11T21:20:41.375246+0000 osd.7 (osd.7) 740 : cluster 0 6.1c scrub ok 2024-03-11T21:20:43.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:42 smithi023 bash[29146]: cluster 2024-03-11T21:20:41.806795+0000 osd.4 (osd.4) 193 : cluster 0 6.1a deep-scrub starts 2024-03-11T21:20:43.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:42 smithi023 bash[29146]: cluster 2024-03-11T21:20:41.875920+0000 osd.4 (osd.4) 194 : cluster 0 6.1a deep-scrub ok 2024-03-11T21:20:43.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:42 smithi023 bash[29146]: cluster 2024-03-11T21:20:41.916260+0000 osd.6 (osd.6) 588 : cluster 0 1.7 scrub starts 2024-03-11T21:20:43.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:42 smithi023 bash[29146]: cluster 2024-03-11T21:20:41.918774+0000 osd.6 (osd.6) 589 : cluster 0 1.7 scrub ok 2024-03-11T21:20:43.679 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:20:44.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:43 smithi146 bash[23809]: cluster 2024-03-11T21:20:42.272116+0000 osd.0 (osd.0) 546 : cluster 0 purged_snaps scrub starts 2024-03-11T21:20:44.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:43 smithi146 bash[23809]: cluster 2024-03-11T21:20:42.272234+0000 osd.0 (osd.0) 547 : cluster 0 purged_snaps scrub ok 2024-03-11T21:20:44.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:43 smithi146 bash[23809]: cluster 2024-03-11T21:20:42.407880+0000 osd.0 (osd.0) 548 : cluster 0 1.11 deep-scrub starts 2024-03-11T21:20:44.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:43 smithi146 bash[23809]: cluster 2024-03-11T21:20:42.410796+0000 osd.0 (osd.0) 549 : cluster 0 1.11 deep-scrub ok 2024-03-11T21:20:44.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:43 smithi146 bash[23809]: cluster 2024-03-11T21:20:42.570821+0000 mon.a (mon.0) 2127 : cluster 0 osdmap e498: 8 total, 8 up, 8 in 2024-03-11T21:20:44.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:43 smithi146 bash[23809]: cluster 2024-03-11T21:20:42.609954+0000 mgr.y (mgr.24380) 1103 : cluster 0 pgmap v1274: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 44 KiB/s rd, 0 B/s wr, 72 op/s 2024-03-11T21:20:44.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:43 smithi146 bash[23809]: cluster 2024-03-11T21:20:42.905184+0000 osd.6 (osd.6) 590 : cluster 0 3.10 scrub starts 2024-03-11T21:20:44.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:43 smithi146 bash[23809]: cluster 2024-03-11T21:20:42.908961+0000 osd.6 (osd.6) 591 : cluster 0 3.10 scrub ok 2024-03-11T21:20:44.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:43 smithi023 bash[21372]: cluster 2024-03-11T21:20:42.272116+0000 osd.0 (osd.0) 546 : cluster 0 purged_snaps scrub starts 2024-03-11T21:20:44.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:43 smithi023 bash[21372]: cluster 2024-03-11T21:20:42.272234+0000 osd.0 (osd.0) 547 : cluster 0 purged_snaps scrub ok 2024-03-11T21:20:44.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:43 smithi023 bash[21372]: cluster 2024-03-11T21:20:42.407880+0000 osd.0 (osd.0) 548 : cluster 0 1.11 deep-scrub starts 2024-03-11T21:20:44.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:43 smithi023 bash[21372]: cluster 2024-03-11T21:20:42.410796+0000 osd.0 (osd.0) 549 : cluster 0 1.11 deep-scrub ok 2024-03-11T21:20:44.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:43 smithi023 bash[21372]: cluster 2024-03-11T21:20:42.570821+0000 mon.a (mon.0) 2127 : cluster 0 osdmap e498: 8 total, 8 up, 8 in 2024-03-11T21:20:44.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:43 smithi023 bash[21372]: cluster 2024-03-11T21:20:42.609954+0000 mgr.y (mgr.24380) 1103 : cluster 0 pgmap v1274: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 44 KiB/s rd, 0 B/s wr, 72 op/s 2024-03-11T21:20:44.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:43 smithi023 bash[21372]: cluster 2024-03-11T21:20:42.905184+0000 osd.6 (osd.6) 590 : cluster 0 3.10 scrub starts 2024-03-11T21:20:44.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:43 smithi023 bash[21372]: cluster 2024-03-11T21:20:42.908961+0000 osd.6 (osd.6) 591 : cluster 0 3.10 scrub ok 2024-03-11T21:20:44.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:43 smithi023 bash[29146]: cluster 2024-03-11T21:20:42.272116+0000 osd.0 (osd.0) 546 : cluster 0 purged_snaps scrub starts 2024-03-11T21:20:44.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:43 smithi023 bash[29146]: cluster 2024-03-11T21:20:42.272234+0000 osd.0 (osd.0) 547 : cluster 0 purged_snaps scrub ok 2024-03-11T21:20:44.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:43 smithi023 bash[29146]: cluster 2024-03-11T21:20:42.407880+0000 osd.0 (osd.0) 548 : cluster 0 1.11 deep-scrub starts 2024-03-11T21:20:44.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:43 smithi023 bash[29146]: cluster 2024-03-11T21:20:42.410796+0000 osd.0 (osd.0) 549 : cluster 0 1.11 deep-scrub ok 2024-03-11T21:20:44.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:43 smithi023 bash[29146]: cluster 2024-03-11T21:20:42.570821+0000 mon.a (mon.0) 2127 : cluster 0 osdmap e498: 8 total, 8 up, 8 in 2024-03-11T21:20:44.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:43 smithi023 bash[29146]: cluster 2024-03-11T21:20:42.609954+0000 mgr.y (mgr.24380) 1103 : cluster 0 pgmap v1274: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 44 KiB/s rd, 0 B/s wr, 72 op/s 2024-03-11T21:20:44.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:43 smithi023 bash[29146]: cluster 2024-03-11T21:20:42.905184+0000 osd.6 (osd.6) 590 : cluster 0 3.10 scrub starts 2024-03-11T21:20:44.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:43 smithi023 bash[29146]: cluster 2024-03-11T21:20:42.908961+0000 osd.6 (osd.6) 591 : cluster 0 3.10 scrub ok 2024-03-11T21:20:44.866 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:44 smithi023 bash[21372]: cluster 2024-03-11T21:20:42.632570+0000 osd.5 (osd.5) 715 : cluster 0 1.12 scrub starts 2024-03-11T21:20:44.866 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:44 smithi023 bash[21372]: cluster 2024-03-11T21:20:42.636510+0000 osd.5 (osd.5) 716 : cluster 0 1.12 scrub ok 2024-03-11T21:20:44.867 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:44 smithi023 bash[21372]: cluster 2024-03-11T21:20:43.371336+0000 osd.7 (osd.7) 741 : cluster 0 3.e scrub starts 2024-03-11T21:20:44.867 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:44 smithi023 bash[21372]: cluster 2024-03-11T21:20:43.375515+0000 osd.7 (osd.7) 742 : cluster 0 3.e scrub ok 2024-03-11T21:20:44.867 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:44 smithi023 bash[21372]: cluster 2024-03-11T21:20:43.611555+0000 osd.3 (osd.3) 672 : cluster 0 1.4 scrub starts 2024-03-11T21:20:44.867 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:44 smithi023 bash[21372]: cluster 2024-03-11T21:20:43.614348+0000 osd.3 (osd.3) 673 : cluster 0 1.4 scrub ok 2024-03-11T21:20:44.867 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:44 smithi023 bash[21372]: cluster 2024-03-11T21:20:43.805258+0000 osd.4 (osd.4) 195 : cluster 0 3.14 scrub starts 2024-03-11T21:20:44.867 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:44 smithi023 bash[21372]: cluster 2024-03-11T21:20:43.809549+0000 osd.4 (osd.4) 196 : cluster 0 3.14 scrub ok 2024-03-11T21:20:44.867 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:44 smithi023 bash[29146]: cluster 2024-03-11T21:20:42.632570+0000 osd.5 (osd.5) 715 : cluster 0 1.12 scrub starts 2024-03-11T21:20:44.867 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:44 smithi023 bash[29146]: cluster 2024-03-11T21:20:42.636510+0000 osd.5 (osd.5) 716 : cluster 0 1.12 scrub ok 2024-03-11T21:20:44.867 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:44 smithi023 bash[29146]: cluster 2024-03-11T21:20:43.371336+0000 osd.7 (osd.7) 741 : cluster 0 3.e scrub starts 2024-03-11T21:20:44.867 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:44 smithi023 bash[29146]: cluster 2024-03-11T21:20:43.375515+0000 osd.7 (osd.7) 742 : cluster 0 3.e scrub ok 2024-03-11T21:20:44.867 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:44 smithi023 bash[29146]: cluster 2024-03-11T21:20:43.611555+0000 osd.3 (osd.3) 672 : cluster 0 1.4 scrub starts 2024-03-11T21:20:44.867 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:44 smithi023 bash[29146]: cluster 2024-03-11T21:20:43.614348+0000 osd.3 (osd.3) 673 : cluster 0 1.4 scrub ok 2024-03-11T21:20:44.868 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:44 smithi023 bash[29146]: cluster 2024-03-11T21:20:43.805258+0000 osd.4 (osd.4) 195 : cluster 0 3.14 scrub starts 2024-03-11T21:20:44.868 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:44 smithi023 bash[29146]: cluster 2024-03-11T21:20:43.809549+0000 osd.4 (osd.4) 196 : cluster 0 3.14 scrub ok 2024-03-11T21:20:45.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:44 smithi146 bash[23809]: cluster 2024-03-11T21:20:42.632570+0000 osd.5 (osd.5) 715 : cluster 0 1.12 scrub starts 2024-03-11T21:20:45.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:44 smithi146 bash[23809]: cluster 2024-03-11T21:20:42.636510+0000 osd.5 (osd.5) 716 : cluster 0 1.12 scrub ok 2024-03-11T21:20:45.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:44 smithi146 bash[23809]: cluster 2024-03-11T21:20:43.371336+0000 osd.7 (osd.7) 741 : cluster 0 3.e scrub starts 2024-03-11T21:20:45.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:44 smithi146 bash[23809]: cluster 2024-03-11T21:20:43.375515+0000 osd.7 (osd.7) 742 : cluster 0 3.e scrub ok 2024-03-11T21:20:45.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:44 smithi146 bash[23809]: cluster 2024-03-11T21:20:43.611555+0000 osd.3 (osd.3) 672 : cluster 0 1.4 scrub starts 2024-03-11T21:20:45.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:44 smithi146 bash[23809]: cluster 2024-03-11T21:20:43.614348+0000 osd.3 (osd.3) 673 : cluster 0 1.4 scrub ok 2024-03-11T21:20:45.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:44 smithi146 bash[23809]: cluster 2024-03-11T21:20:43.805258+0000 osd.4 (osd.4) 195 : cluster 0 3.14 scrub starts 2024-03-11T21:20:45.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:44 smithi146 bash[23809]: cluster 2024-03-11T21:20:43.809549+0000 osd.4 (osd.4) 196 : cluster 0 3.14 scrub ok 2024-03-11T21:20:46.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:45 smithi146 bash[23809]: cluster 2024-03-11T21:20:44.340012+0000 osd.7 (osd.7) 743 : cluster 0 3.d scrub starts 2024-03-11T21:20:46.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:45 smithi146 bash[23809]: cluster 2024-03-11T21:20:44.347249+0000 osd.7 (osd.7) 744 : cluster 0 3.d scrub ok 2024-03-11T21:20:46.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:45 smithi146 bash[23809]: cluster 2024-03-11T21:20:44.406227+0000 osd.0 (osd.0) 550 : cluster 0 6.0 scrub starts 2024-03-11T21:20:46.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:45 smithi146 bash[23809]: cluster 2024-03-11T21:20:44.488517+0000 osd.0 (osd.0) 551 : cluster 0 6.0 scrub ok 2024-03-11T21:20:46.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:45 smithi146 bash[23809]: cluster 2024-03-11T21:20:44.590511+0000 osd.3 (osd.3) 674 : cluster 0 3.1e scrub starts 2024-03-11T21:20:46.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:45 smithi146 bash[23809]: cluster 2024-03-11T21:20:44.594504+0000 osd.3 (osd.3) 675 : cluster 0 3.1e scrub ok 2024-03-11T21:20:46.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:45 smithi146 bash[23809]: cluster 2024-03-11T21:20:44.610852+0000 mgr.y (mgr.24380) 1104 : cluster 0 pgmap v1275: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 44 KiB/s rd, 0 B/s wr, 72 op/s 2024-03-11T21:20:46.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:45 smithi146 bash[23809]: cluster 2024-03-11T21:20:44.877251+0000 osd.2 (osd.2) 333 : cluster 0 2.0 scrub starts 2024-03-11T21:20:46.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:45 smithi146 bash[23809]: cluster 2024-03-11T21:20:44.879971+0000 osd.2 (osd.2) 334 : cluster 0 2.0 scrub ok 2024-03-11T21:20:46.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:45 smithi023 bash[21372]: cluster 2024-03-11T21:20:44.340012+0000 osd.7 (osd.7) 743 : cluster 0 3.d scrub starts 2024-03-11T21:20:46.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:45 smithi023 bash[21372]: cluster 2024-03-11T21:20:44.347249+0000 osd.7 (osd.7) 744 : cluster 0 3.d scrub ok 2024-03-11T21:20:46.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:45 smithi023 bash[21372]: cluster 2024-03-11T21:20:44.406227+0000 osd.0 (osd.0) 550 : cluster 0 6.0 scrub starts 2024-03-11T21:20:46.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:45 smithi023 bash[21372]: cluster 2024-03-11T21:20:44.488517+0000 osd.0 (osd.0) 551 : cluster 0 6.0 scrub ok 2024-03-11T21:20:46.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:45 smithi023 bash[21372]: cluster 2024-03-11T21:20:44.590511+0000 osd.3 (osd.3) 674 : cluster 0 3.1e scrub starts 2024-03-11T21:20:46.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:45 smithi023 bash[21372]: cluster 2024-03-11T21:20:44.594504+0000 osd.3 (osd.3) 675 : cluster 0 3.1e scrub ok 2024-03-11T21:20:46.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:45 smithi023 bash[21372]: cluster 2024-03-11T21:20:44.610852+0000 mgr.y (mgr.24380) 1104 : cluster 0 pgmap v1275: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 44 KiB/s rd, 0 B/s wr, 72 op/s 2024-03-11T21:20:46.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:45 smithi023 bash[21372]: cluster 2024-03-11T21:20:44.877251+0000 osd.2 (osd.2) 333 : cluster 0 2.0 scrub starts 2024-03-11T21:20:46.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:45 smithi023 bash[21372]: cluster 2024-03-11T21:20:44.879971+0000 osd.2 (osd.2) 334 : cluster 0 2.0 scrub ok 2024-03-11T21:20:46.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:45 smithi023 bash[29146]: cluster 2024-03-11T21:20:44.340012+0000 osd.7 (osd.7) 743 : cluster 0 3.d scrub starts 2024-03-11T21:20:46.024 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:45 smithi023 bash[29146]: cluster 2024-03-11T21:20:44.347249+0000 osd.7 (osd.7) 744 : cluster 0 3.d scrub ok 2024-03-11T21:20:46.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:45 smithi023 bash[29146]: cluster 2024-03-11T21:20:44.406227+0000 osd.0 (osd.0) 550 : cluster 0 6.0 scrub starts 2024-03-11T21:20:46.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:45 smithi023 bash[29146]: cluster 2024-03-11T21:20:44.488517+0000 osd.0 (osd.0) 551 : cluster 0 6.0 scrub ok 2024-03-11T21:20:46.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:45 smithi023 bash[29146]: cluster 2024-03-11T21:20:44.590511+0000 osd.3 (osd.3) 674 : cluster 0 3.1e scrub starts 2024-03-11T21:20:46.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:45 smithi023 bash[29146]: cluster 2024-03-11T21:20:44.594504+0000 osd.3 (osd.3) 675 : cluster 0 3.1e scrub ok 2024-03-11T21:20:46.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:45 smithi023 bash[29146]: cluster 2024-03-11T21:20:44.610852+0000 mgr.y (mgr.24380) 1104 : cluster 0 pgmap v1275: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 44 KiB/s rd, 0 B/s wr, 72 op/s 2024-03-11T21:20:46.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:45 smithi023 bash[29146]: cluster 2024-03-11T21:20:44.877251+0000 osd.2 (osd.2) 333 : cluster 0 2.0 scrub starts 2024-03-11T21:20:46.025 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:45 smithi023 bash[29146]: cluster 2024-03-11T21:20:44.879971+0000 osd.2 (osd.2) 334 : cluster 0 2.0 scrub ok 2024-03-11T21:20:46.751 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:20:46 smithi146 bash[49028]: debug there is no tcmu-runner data available 2024-03-11T21:20:46.752 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:46 smithi146 bash[23809]: cluster 2024-03-11T21:20:45.375017+0000 osd.0 (osd.0) 552 : cluster 0 4.1e scrub starts 2024-03-11T21:20:46.752 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:46 smithi146 bash[23809]: cluster 2024-03-11T21:20:45.377620+0000 osd.0 (osd.0) 553 : cluster 0 4.1e scrub ok 2024-03-11T21:20:46.752 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:46 smithi146 bash[23809]: audit 2024-03-11T21:20:46.373600+0000 mon.a (mon.0) 2128 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:20:47.023 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:46 smithi023 bash[29146]: cluster 2024-03-11T21:20:45.375017+0000 osd.0 (osd.0) 552 : cluster 0 4.1e scrub starts 2024-03-11T21:20:47.023 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:46 smithi023 bash[29146]: cluster 2024-03-11T21:20:45.377620+0000 osd.0 (osd.0) 553 : cluster 0 4.1e scrub ok 2024-03-11T21:20:47.023 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:46 smithi023 bash[29146]: audit 2024-03-11T21:20:46.373600+0000 mon.a (mon.0) 2128 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:20:47.023 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:46 smithi023 bash[21372]: cluster 2024-03-11T21:20:45.375017+0000 osd.0 (osd.0) 552 : cluster 0 4.1e scrub starts 2024-03-11T21:20:47.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:46 smithi023 bash[21372]: cluster 2024-03-11T21:20:45.377620+0000 osd.0 (osd.0) 553 : cluster 0 4.1e scrub ok 2024-03-11T21:20:47.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:46 smithi023 bash[21372]: audit 2024-03-11T21:20:46.373600+0000 mon.a (mon.0) 2128 : audit 0 from='mgr.24380 172.21.15.23:0/3422054905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:20:47.523 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:20:47 smithi023 bash[21627]: [11/Mar/2024:21:20:47] ENGINE Bus STOPPING 2024-03-11T21:20:47.524 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:20:47 smithi023 bash[21627]: [11/Mar/2024:21:20:47] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-11T21:20:47.524 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:20:47 smithi023 bash[21627]: [11/Mar/2024:21:20:47] ENGINE Bus STOPPED 2024-03-11T21:20:47.524 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:20:47 smithi023 bash[21627]: [11/Mar/2024:21:20:47] ENGINE Bus STARTING 2024-03-11T21:20:47.524 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:20:47 smithi023 bash[21627]: [11/Mar/2024:21:20:47] ENGINE Serving on http://:::9283 2024-03-11T21:20:47.524 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:20:47 smithi023 bash[21627]: [11/Mar/2024:21:20:47] ENGINE Bus STARTED 2024-03-11T21:20:47.877 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:47 smithi023 bash[21372]: cluster 2024-03-11T21:20:46.351173+0000 osd.7 (osd.7) 745 : cluster 0 2.1c scrub starts 2024-03-11T21:20:47.877 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:47 smithi023 bash[21372]: cluster 2024-03-11T21:20:46.353697+0000 osd.7 (osd.7) 746 : cluster 0 2.1c scrub ok 2024-03-11T21:20:47.877 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:47 smithi023 bash[21372]: audit 2024-03-11T21:20:46.383408+0000 mgr.y (mgr.24380) 1105 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:20:47.877 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:47 smithi023 bash[21372]: cluster 2024-03-11T21:20:46.417084+0000 osd.0 (osd.0) 554 : cluster 0 6.9 scrub starts 2024-03-11T21:20:47.877 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:47 smithi023 bash[21372]: cluster 2024-03-11T21:20:46.530833+0000 osd.7 (osd.7) 747 : cluster 0 purged_snaps scrub starts 2024-03-11T21:20:47.877 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:47 smithi023 bash[21372]: cluster 2024-03-11T21:20:46.530971+0000 osd.7 (osd.7) 748 : cluster 0 purged_snaps scrub ok 2024-03-11T21:20:47.877 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:47 smithi023 bash[21372]: cluster 2024-03-11T21:20:46.534500+0000 osd.0 (osd.0) 555 : cluster 0 6.9 scrub ok 2024-03-11T21:20:47.877 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:47 smithi023 bash[21372]: cluster 2024-03-11T21:20:46.595024+0000 osd.5 (osd.5) 717 : cluster 0 6.a scrub starts 2024-03-11T21:20:47.877 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:47 smithi023 bash[21372]: cluster 2024-03-11T21:20:46.611987+0000 mgr.y (mgr.24380) 1106 : cluster 0 pgmap v1276: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 68 op/s 2024-03-11T21:20:47.877 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:47 smithi023 bash[21372]: cluster 2024-03-11T21:20:46.635808+0000 mon.a (mon.0) 2129 : cluster 0 osdmap e499: 8 total, 8 up, 8 in 2024-03-11T21:20:47.877 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:47 smithi023 bash[21372]: audit 2024-03-11T21:20:47.097516+0000 mon.a (mon.0) 2130 : audit 1 from='client.? 172.21.15.23:0/3016354538' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-03-11T21:20:47.877 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:47 smithi023 bash[21372]: audit 2024-03-11T21:20:47.105540+0000 mon.a (mon.0) 2131 : audit 1 from='client.? 172.21.15.23:0/3016354538' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-03-11T21:20:47.879 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-11T21:20:47.889 INFO:tasks.cephadm:Teardown begin 2024-03-11T21:20:47.890 DEBUG:teuthology.orchestra.run.smithi023:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-11T21:20:47.890 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:47 smithi023 bash[29146]: cluster 2024-03-11T21:20:46.351173+0000 osd.7 (osd.7) 745 : cluster 0 2.1c scrub starts 2024-03-11T21:20:47.890 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:47 smithi023 bash[29146]: cluster 2024-03-11T21:20:46.353697+0000 osd.7 (osd.7) 746 : cluster 0 2.1c scrub ok 2024-03-11T21:20:47.891 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:47 smithi023 bash[29146]: audit 2024-03-11T21:20:46.383408+0000 mgr.y (mgr.24380) 1105 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:20:47.891 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:47 smithi023 bash[29146]: cluster 2024-03-11T21:20:46.417084+0000 osd.0 (osd.0) 554 : cluster 0 6.9 scrub starts 2024-03-11T21:20:47.891 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:47 smithi023 bash[29146]: cluster 2024-03-11T21:20:46.530833+0000 osd.7 (osd.7) 747 : cluster 0 purged_snaps scrub starts 2024-03-11T21:20:47.891 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:47 smithi023 bash[29146]: cluster 2024-03-11T21:20:46.530971+0000 osd.7 (osd.7) 748 : cluster 0 purged_snaps scrub ok 2024-03-11T21:20:47.891 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:47 smithi023 bash[29146]: cluster 2024-03-11T21:20:46.534500+0000 osd.0 (osd.0) 555 : cluster 0 6.9 scrub ok 2024-03-11T21:20:47.891 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:47 smithi023 bash[29146]: cluster 2024-03-11T21:20:46.595024+0000 osd.5 (osd.5) 717 : cluster 0 6.a scrub starts 2024-03-11T21:20:47.891 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:47 smithi023 bash[29146]: cluster 2024-03-11T21:20:46.611987+0000 mgr.y (mgr.24380) 1106 : cluster 0 pgmap v1276: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 68 op/s 2024-03-11T21:20:47.891 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:47 smithi023 bash[29146]: cluster 2024-03-11T21:20:46.635808+0000 mon.a (mon.0) 2129 : cluster 0 osdmap e499: 8 total, 8 up, 8 in 2024-03-11T21:20:47.891 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:47 smithi023 bash[29146]: audit 2024-03-11T21:20:47.097516+0000 mon.a (mon.0) 2130 : audit 1 from='client.? 172.21.15.23:0/3016354538' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-03-11T21:20:47.891 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:47 smithi023 bash[29146]: audit 2024-03-11T21:20:47.105540+0000 mon.a (mon.0) 2131 : audit 1 from='client.? 172.21.15.23:0/3016354538' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-03-11T21:20:47.914 DEBUG:teuthology.orchestra.run.smithi146:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-11T21:20:47.927 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:47 smithi146 bash[23809]: cluster 2024-03-11T21:20:46.351173+0000 osd.7 (osd.7) 745 : cluster 0 2.1c scrub starts 2024-03-11T21:20:47.927 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:47 smithi146 bash[23809]: cluster 2024-03-11T21:20:46.353697+0000 osd.7 (osd.7) 746 : cluster 0 2.1c scrub ok 2024-03-11T21:20:47.927 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:47 smithi146 bash[23809]: audit 2024-03-11T21:20:46.383408+0000 mgr.y (mgr.24380) 1105 : audit 0 from='client.24403 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:20:47.927 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:47 smithi146 bash[23809]: cluster 2024-03-11T21:20:46.417084+0000 osd.0 (osd.0) 554 : cluster 0 6.9 scrub starts 2024-03-11T21:20:47.927 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:47 smithi146 bash[23809]: cluster 2024-03-11T21:20:46.530833+0000 osd.7 (osd.7) 747 : cluster 0 purged_snaps scrub starts 2024-03-11T21:20:47.927 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:47 smithi146 bash[23809]: cluster 2024-03-11T21:20:46.530971+0000 osd.7 (osd.7) 748 : cluster 0 purged_snaps scrub ok 2024-03-11T21:20:47.927 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:47 smithi146 bash[23809]: cluster 2024-03-11T21:20:46.534500+0000 osd.0 (osd.0) 555 : cluster 0 6.9 scrub ok 2024-03-11T21:20:47.927 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:47 smithi146 bash[23809]: cluster 2024-03-11T21:20:46.595024+0000 osd.5 (osd.5) 717 : cluster 0 6.a scrub starts 2024-03-11T21:20:47.927 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:47 smithi146 bash[23809]: cluster 2024-03-11T21:20:46.611987+0000 mgr.y (mgr.24380) 1106 : cluster 0 pgmap v1276: 152 pgs: 152 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 68 op/s 2024-03-11T21:20:47.927 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:47 smithi146 bash[23809]: cluster 2024-03-11T21:20:46.635808+0000 mon.a (mon.0) 2129 : cluster 0 osdmap e499: 8 total, 8 up, 8 in 2024-03-11T21:20:47.928 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:47 smithi146 bash[23809]: audit 2024-03-11T21:20:47.097516+0000 mon.a (mon.0) 2130 : audit 1 from='client.? 172.21.15.23:0/3016354538' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-03-11T21:20:47.928 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:47 smithi146 bash[23809]: audit 2024-03-11T21:20:47.105540+0000 mon.a (mon.0) 2131 : audit 1 from='client.? 172.21.15.23:0/3016354538' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-03-11T21:20:47.959 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-11T21:20:47.959 DEBUG:teuthology.orchestra.run.smithi023:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-11T21:20:47.996 DEBUG:teuthology.orchestra.run.smithi146:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-11T21:20:48.012 INFO:tasks.cephadm:Stopping all daemons... 2024-03-11T21:20:48.012 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-03-11T21:20:48.012 DEBUG:teuthology.orchestra.run.smithi023:> sudo systemctl stop ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@mon.a 2024-03-11T21:20:48.207 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:48 smithi023 systemd[1]: Stopping Ceph mon.a for 67ba00f8-dfe7-11ee-95c8-87774f69a715... 2024-03-11T21:20:48.334 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:48 smithi023 bash[21372]: debug 2024-03-11T21:20:48.217+0000 7f552a0e8700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-03-11T21:20:48.334 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:48 smithi023 bash[21372]: debug 2024-03-11T21:20:48.217+0000 7f552a0e8700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-03-11T21:20:48.334 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:20:48 smithi023 bash[21627]: [11/Mar/2024:21:20:48] ENGINE Bus STOPPING 2024-03-11T21:20:48.335 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:20:48 smithi023 bash[21627]: [11/Mar/2024:21:20:48] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-11T21:20:48.335 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:20:48 smithi023 bash[21627]: [11/Mar/2024:21:20:48] ENGINE Bus STOPPED 2024-03-11T21:20:48.335 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:20:48 smithi023 bash[21627]: [11/Mar/2024:21:20:48] ENGINE Bus STARTING 2024-03-11T21:20:48.773 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:20:48 smithi023 bash[21627]: ::ffff:172.21.15.146 - - [11/Mar/2024:21:20:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:20:48.774 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:20:48 smithi023 bash[21627]: [11/Mar/2024:21:20:48] ENGINE Serving on http://:::9283 2024-03-11T21:20:48.774 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:20:48 smithi023 bash[21627]: [11/Mar/2024:21:20:48] ENGINE Bus STARTED 2024-03-11T21:20:49.191 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 11 21:20:48 smithi023 bash[168933]: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-mon-a 2024-03-11T21:20:49.195 DEBUG:teuthology.orchestra.run.smithi023:> sudo pkill -f 'journalctl -f -n 0 -u ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@mon.a.service' 2024-03-11T21:20:49.220 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T21:20:49.221 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-03-11T21:20:49.221 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-03-11T21:20:49.221 DEBUG:teuthology.orchestra.run.smithi023:> sudo systemctl stop ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@mon.c 2024-03-11T21:20:49.443 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:49 smithi023 systemd[1]: Stopping Ceph mon.c for 67ba00f8-dfe7-11ee-95c8-87774f69a715... 2024-03-11T21:20:49.443 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:49 smithi023 bash[29146]: debug 2024-03-11T21:20:49.401+0000 7fedb69b4700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-03-11T21:20:49.443 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:49 smithi023 bash[29146]: debug 2024-03-11T21:20:49.401+0000 7fedb69b4700 -1 mon.c@2(peon) e3 *** Got Signal Terminated *** 2024-03-11T21:20:50.319 DEBUG:teuthology.orchestra.run.smithi023:> sudo pkill -f 'journalctl -f -n 0 -u ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@mon.c.service' 2024-03-11T21:20:50.330 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:50 smithi023 bash[169044]: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-mon-c 2024-03-11T21:20:50.330 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:50 smithi023 bash[169104]: Error response from daemon: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-mon-c 2024-03-11T21:20:50.330 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:50 smithi023 systemd[1]: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@mon.c.service: Deactivated successfully. 2024-03-11T21:20:50.330 INFO:journalctl@ceph.mon.c.smithi023.stdout:Mar 11 21:20:50 smithi023 systemd[1]: Stopped Ceph mon.c for 67ba00f8-dfe7-11ee-95c8-87774f69a715. 2024-03-11T21:20:50.349 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T21:20:50.349 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-03-11T21:20:50.349 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-03-11T21:20:50.349 DEBUG:teuthology.orchestra.run.smithi146:> sudo systemctl stop ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@mon.b 2024-03-11T21:20:50.726 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:50 smithi146 systemd[1]: Stopping Ceph mon.b for 67ba00f8-dfe7-11ee-95c8-87774f69a715... 2024-03-11T21:20:50.726 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:50 smithi146 bash[23809]: debug 2024-03-11T21:20:50.613+0000 7f82d6bef700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-03-11T21:20:50.726 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:50 smithi146 bash[23809]: debug 2024-03-11T21:20:50.613+0000 7f82d6bef700 -1 mon.b@1(peon) e3 *** Got Signal Terminated *** 2024-03-11T21:20:51.732 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:51 smithi146 bash[109433]: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-mon-b 2024-03-11T21:20:51.732 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 11 21:20:51 smithi146 bash[109496]: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-mon-b 2024-03-11T21:20:51.734 DEBUG:teuthology.orchestra.run.smithi146:> sudo pkill -f 'journalctl -f -n 0 -u ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@mon.b.service' 2024-03-11T21:20:51.765 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T21:20:51.766 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-03-11T21:20:51.766 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-03-11T21:20:51.766 DEBUG:teuthology.orchestra.run.smithi023:> sudo systemctl stop ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@mgr.y 2024-03-11T21:20:52.103 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:20:51 smithi023 systemd[1]: Stopping Ceph mgr.y for 67ba00f8-dfe7-11ee-95c8-87774f69a715... 2024-03-11T21:20:52.103 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:20:51 smithi023 bash[21627]: debug 2024-03-11T21:20:51.941+0000 7f6430a95700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-03-11T21:20:52.103 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 21:20:51 smithi023 bash[21627]: debug 2024-03-11T21:20:51.941+0000 7f6430a95700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-03-11T21:20:52.830 DEBUG:teuthology.orchestra.run.smithi023:> sudo pkill -f 'journalctl -f -n 0 -u ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@mgr.y.service' 2024-03-11T21:20:52.856 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T21:20:52.856 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-03-11T21:20:52.856 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-03-11T21:20:52.856 DEBUG:teuthology.orchestra.run.smithi146:> sudo systemctl stop ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@mgr.x 2024-03-11T21:20:53.132 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Mar 11 21:20:52 smithi146 systemd[1]: Stopping Ceph mgr.x for 67ba00f8-dfe7-11ee-95c8-87774f69a715... 2024-03-11T21:20:54.064 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Mar 11 21:20:53 smithi146 bash[109546]: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-mgr-x 2024-03-11T21:20:54.119 DEBUG:teuthology.orchestra.run.smithi146:> sudo pkill -f 'journalctl -f -n 0 -u ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@mgr.x.service' 2024-03-11T21:20:54.144 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T21:20:54.145 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-03-11T21:20:54.145 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-11T21:20:54.145 DEBUG:teuthology.orchestra.run.smithi023:> sudo systemctl stop ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@osd.0 2024-03-11T21:20:54.524 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 11 21:20:54 smithi023 systemd[1]: Stopping Ceph osd.0 for 67ba00f8-dfe7-11ee-95c8-87774f69a715... 2024-03-11T21:20:54.524 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 11 21:20:54 smithi023 bash[32169]: debug 2024-03-11T21:20:54.305+0000 7f3a1666a700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-03-11T21:20:54.524 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 11 21:20:54 smithi023 bash[32169]: debug 2024-03-11T21:20:54.305+0000 7f3a1666a700 -1 osd.0 499 *** Got signal Terminated *** 2024-03-11T21:20:54.524 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 11 21:20:54 smithi023 bash[32169]: debug 2024-03-11T21:20:54.305+0000 7f3a1666a700 -1 osd.0 499 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T21:21:00.700 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 11 21:21:00 smithi023 bash[169238]: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd-0 2024-03-11T21:21:00.700 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 11 21:21:00 smithi023 bash[169301]: Error response from daemon: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd-0 2024-03-11T21:21:02.787 DEBUG:teuthology.orchestra.run.smithi023:> sudo pkill -f 'journalctl -f -n 0 -u ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@osd.0.service' 2024-03-11T21:21:02.816 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T21:21:02.817 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-11T21:21:02.817 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-11T21:21:02.817 DEBUG:teuthology.orchestra.run.smithi023:> sudo systemctl stop ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@osd.1 2024-03-11T21:21:03.273 INFO:journalctl@ceph.osd.1.smithi023.stdout:Mar 11 21:21:02 smithi023 systemd[1]: Stopping Ceph osd.1 for 67ba00f8-dfe7-11ee-95c8-87774f69a715... 2024-03-11T21:21:03.274 INFO:journalctl@ceph.osd.1.smithi023.stdout:Mar 11 21:21:03 smithi023 bash[95887]: debug 2024-03-11T21:21:03.009+0000 7fb877407700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-03-11T21:21:03.274 INFO:journalctl@ceph.osd.1.smithi023.stdout:Mar 11 21:21:03 smithi023 bash[95887]: debug 2024-03-11T21:21:03.009+0000 7fb877407700 -1 osd.1 499 *** Got signal Terminated *** 2024-03-11T21:21:03.274 INFO:journalctl@ceph.osd.1.smithi023.stdout:Mar 11 21:21:03 smithi023 bash[95887]: debug 2024-03-11T21:21:03.009+0000 7fb877407700 -1 osd.1 499 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T21:21:09.556 INFO:journalctl@ceph.osd.1.smithi023.stdout:Mar 11 21:21:09 smithi023 bash[169439]: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd-1 2024-03-11T21:21:09.556 INFO:journalctl@ceph.osd.1.smithi023.stdout:Mar 11 21:21:09 smithi023 bash[169503]: Error response from daemon: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd-1 2024-03-11T21:21:09.752 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Mar 11 21:21:09 smithi146 bash[52406]: ts=2024-03-11T21:21:09.509Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=node msg="Unable to refresh target groups" err="Get \"http://172.21.15.23:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.23:8765: connect: connection refused" 2024-03-11T21:21:09.752 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Mar 11 21:21:09 smithi146 bash[52406]: ts=2024-03-11T21:21:09.509Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=nvmeof msg="Unable to refresh target groups" err="Get \"http://172.21.15.23:8765/sd/prometheus/sd-config?service=nvmeof\": dial tcp 172.21.15.23:8765: connect: connection refused" 2024-03-11T21:21:09.752 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Mar 11 21:21:09 smithi146 bash[52406]: ts=2024-03-11T21:21:09.517Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.23:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.23:8765: connect: connection refused" 2024-03-11T21:21:09.752 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Mar 11 21:21:09 smithi146 bash[52406]: ts=2024-03-11T21:21:09.521Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.23:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.23:8765: connect: connection refused" 2024-03-11T21:21:09.752 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Mar 11 21:21:09 smithi146 bash[52406]: ts=2024-03-11T21:21:09.521Z caller=refresh.go:99 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://172.21.15.23:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.23:8765: connect: connection refused" 2024-03-11T21:21:11.425 DEBUG:teuthology.orchestra.run.smithi023:> sudo pkill -f 'journalctl -f -n 0 -u ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@osd.1.service' 2024-03-11T21:21:11.454 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T21:21:11.455 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-11T21:21:11.455 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-11T21:21:11.455 DEBUG:teuthology.orchestra.run.smithi023:> sudo systemctl stop ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@osd.2 2024-03-11T21:21:11.774 INFO:journalctl@ceph.osd.2.smithi023.stdout:Mar 11 21:21:11 smithi023 systemd[1]: Stopping Ceph osd.2 for 67ba00f8-dfe7-11ee-95c8-87774f69a715... 2024-03-11T21:21:11.774 INFO:journalctl@ceph.osd.2.smithi023.stdout:Mar 11 21:21:11 smithi023 bash[43065]: debug 2024-03-11T21:21:11.641+0000 7f0ba0f34700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-03-11T21:21:11.774 INFO:journalctl@ceph.osd.2.smithi023.stdout:Mar 11 21:21:11 smithi023 bash[43065]: debug 2024-03-11T21:21:11.641+0000 7f0ba0f34700 -1 osd.2 499 *** Got signal Terminated *** 2024-03-11T21:21:11.774 INFO:journalctl@ceph.osd.2.smithi023.stdout:Mar 11 21:21:11 smithi023 bash[43065]: debug 2024-03-11T21:21:11.641+0000 7f0ba0f34700 -1 osd.2 499 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T21:21:17.954 INFO:journalctl@ceph.osd.2.smithi023.stdout:Mar 11 21:21:17 smithi023 bash[169634]: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd-2 2024-03-11T21:21:17.954 INFO:journalctl@ceph.osd.2.smithi023.stdout:Mar 11 21:21:17 smithi023 bash[169701]: Error response from daemon: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd-2 2024-03-11T21:21:19.703 DEBUG:teuthology.orchestra.run.smithi023:> sudo pkill -f 'journalctl -f -n 0 -u ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@osd.2.service' 2024-03-11T21:21:19.732 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T21:21:19.732 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-11T21:21:19.733 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-03-11T21:21:19.733 DEBUG:teuthology.orchestra.run.smithi023:> sudo systemctl stop ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@osd.3 2024-03-11T21:21:20.023 INFO:journalctl@ceph.osd.3.smithi023.stdout:Mar 11 21:21:19 smithi023 systemd[1]: Stopping Ceph osd.3 for 67ba00f8-dfe7-11ee-95c8-87774f69a715... 2024-03-11T21:21:20.024 INFO:journalctl@ceph.osd.3.smithi023.stdout:Mar 11 21:21:19 smithi023 bash[48706]: debug 2024-03-11T21:21:19.933+0000 7f2aeed94700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-03-11T21:21:20.024 INFO:journalctl@ceph.osd.3.smithi023.stdout:Mar 11 21:21:19 smithi023 bash[48706]: debug 2024-03-11T21:21:19.933+0000 7f2aeed94700 -1 osd.3 499 *** Got signal Terminated *** 2024-03-11T21:21:20.024 INFO:journalctl@ceph.osd.3.smithi023.stdout:Mar 11 21:21:19 smithi023 bash[48706]: debug 2024-03-11T21:21:19.933+0000 7f2aeed94700 -1 osd.3 499 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T21:21:20.251 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 11 21:21:19 smithi146 bash[68242]: debug 2024-03-11T21:21:19.905+0000 7f6db0bde700 -1 osd.4 499 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-11T21:20:58.280427+0000 front 2024-03-11T21:20:58.280453+0000 (oldest deadline 2024-03-11T21:21:19.880954+0000) 2024-03-11T21:21:20.863 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 11 21:21:20 smithi146 bash[32911]: debug 2024-03-11T21:21:20.613+0000 7f0b6ce00700 -1 osd.5 499 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-11T21:20:57.974158+0000 front 2024-03-11T21:20:57.974293+0000 (oldest deadline 2024-03-11T21:21:20.174723+0000) 2024-03-11T21:21:21.251 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 11 21:21:20 smithi146 bash[68242]: debug 2024-03-11T21:21:20.873+0000 7f6db0bde700 -1 osd.4 499 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-11T21:20:58.280427+0000 front 2024-03-11T21:20:58.280453+0000 (oldest deadline 2024-03-11T21:21:19.880954+0000) 2024-03-11T21:21:21.251 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 11 21:21:21 smithi146 bash[38675]: debug 2024-03-11T21:21:21.033+0000 7fb0d14ce700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-11T21:20:55.673190+0000 front 2024-03-11T21:20:55.672758+0000 (oldest deadline 2024-03-11T21:21:20.372648+0000) 2024-03-11T21:21:21.751 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 11 21:21:21 smithi146 bash[32911]: debug 2024-03-11T21:21:21.585+0000 7f0b6ce00700 -1 osd.5 499 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-11T21:20:57.974158+0000 front 2024-03-11T21:20:57.974293+0000 (oldest deadline 2024-03-11T21:21:20.174723+0000) 2024-03-11T21:21:21.752 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 11 21:21:21 smithi146 bash[44452]: debug 2024-03-11T21:21:21.433+0000 7f5d99d2d700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-11T21:20:55.488114+0000 front 2024-03-11T21:20:55.488176+0000 (oldest deadline 2024-03-11T21:21:21.387970+0000) 2024-03-11T21:21:22.251 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 11 21:21:21 smithi146 bash[68242]: debug 2024-03-11T21:21:21.869+0000 7f6db0bde700 -1 osd.4 499 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-11T21:20:58.280427+0000 front 2024-03-11T21:20:58.280453+0000 (oldest deadline 2024-03-11T21:21:19.880954+0000) 2024-03-11T21:21:22.251 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 11 21:21:22 smithi146 bash[38675]: debug 2024-03-11T21:21:22.025+0000 7fb0d14ce700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-11T21:20:55.673190+0000 front 2024-03-11T21:20:55.672758+0000 (oldest deadline 2024-03-11T21:21:20.372648+0000) 2024-03-11T21:21:22.751 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 11 21:21:22 smithi146 bash[32911]: debug 2024-03-11T21:21:22.621+0000 7f0b6ce00700 -1 osd.5 499 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-11T21:20:57.974158+0000 front 2024-03-11T21:20:57.974293+0000 (oldest deadline 2024-03-11T21:21:20.174723+0000) 2024-03-11T21:21:22.751 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 11 21:21:22 smithi146 bash[44452]: debug 2024-03-11T21:21:22.425+0000 7f5d99d2d700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-11T21:20:55.488114+0000 front 2024-03-11T21:20:55.488176+0000 (oldest deadline 2024-03-11T21:21:21.387970+0000) 2024-03-11T21:21:23.023 INFO:journalctl@ceph.osd.3.smithi023.stdout:Mar 11 21:21:22 smithi023 bash[48706]: debug 2024-03-11T21:21:22.657+0000 7f2aeb388700 -1 osd.3 499 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-11T21:20:57.325096+0000 front 2024-03-11T21:20:57.325156+0000 (oldest deadline 2024-03-11T21:21:22.025233+0000) 2024-03-11T21:21:23.251 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 11 21:21:22 smithi146 bash[68242]: debug 2024-03-11T21:21:22.893+0000 7f6db0bde700 -1 osd.4 499 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-11T21:20:58.280427+0000 front 2024-03-11T21:20:58.280453+0000 (oldest deadline 2024-03-11T21:21:19.880954+0000) 2024-03-11T21:21:23.251 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 11 21:21:23 smithi146 bash[38675]: debug 2024-03-11T21:21:23.049+0000 7fb0d14ce700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-11T21:20:55.673190+0000 front 2024-03-11T21:20:55.672758+0000 (oldest deadline 2024-03-11T21:21:20.372648+0000) 2024-03-11T21:21:23.751 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 11 21:21:23 smithi146 bash[32911]: debug 2024-03-11T21:21:23.653+0000 7f0b6ce00700 -1 osd.5 499 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-11T21:20:57.974158+0000 front 2024-03-11T21:20:57.974293+0000 (oldest deadline 2024-03-11T21:21:20.174723+0000) 2024-03-11T21:21:23.752 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 11 21:21:23 smithi146 bash[44452]: debug 2024-03-11T21:21:23.413+0000 7f5d99d2d700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-11T21:20:55.488114+0000 front 2024-03-11T21:20:55.488176+0000 (oldest deadline 2024-03-11T21:21:21.387970+0000) 2024-03-11T21:21:24.023 INFO:journalctl@ceph.osd.3.smithi023.stdout:Mar 11 21:21:23 smithi023 bash[48706]: debug 2024-03-11T21:21:23.701+0000 7f2aeb388700 -1 osd.3 499 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-11T21:20:57.325096+0000 front 2024-03-11T21:20:57.325156+0000 (oldest deadline 2024-03-11T21:21:22.025233+0000) 2024-03-11T21:21:24.251 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 11 21:21:23 smithi146 bash[68242]: debug 2024-03-11T21:21:23.865+0000 7f6db0bde700 -1 osd.4 499 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-11T21:20:58.280427+0000 front 2024-03-11T21:20:58.280453+0000 (oldest deadline 2024-03-11T21:21:19.880954+0000) 2024-03-11T21:21:24.251 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 11 21:21:24 smithi146 bash[38675]: debug 2024-03-11T21:21:24.033+0000 7fb0d14ce700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-11T21:20:55.673190+0000 front 2024-03-11T21:20:55.672758+0000 (oldest deadline 2024-03-11T21:21:20.372648+0000) 2024-03-11T21:21:24.751 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 11 21:21:24 smithi146 bash[44452]: debug 2024-03-11T21:21:24.441+0000 7f5d99d2d700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-11T21:20:55.488114+0000 front 2024-03-11T21:20:55.488176+0000 (oldest deadline 2024-03-11T21:21:21.387970+0000) 2024-03-11T21:21:24.752 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 11 21:21:24 smithi146 bash[32911]: debug 2024-03-11T21:21:24.637+0000 7f0b6ce00700 -1 osd.5 499 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-11T21:20:57.974158+0000 front 2024-03-11T21:20:57.974293+0000 (oldest deadline 2024-03-11T21:21:20.174723+0000) 2024-03-11T21:21:25.023 INFO:journalctl@ceph.osd.3.smithi023.stdout:Mar 11 21:21:24 smithi023 bash[48706]: debug 2024-03-11T21:21:24.669+0000 7f2aeb388700 -1 osd.3 499 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-11T21:20:57.325096+0000 front 2024-03-11T21:20:57.325156+0000 (oldest deadline 2024-03-11T21:21:22.025233+0000) 2024-03-11T21:21:25.251 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 11 21:21:24 smithi146 bash[68242]: debug 2024-03-11T21:21:24.825+0000 7f6db0bde700 -1 osd.4 499 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-11T21:20:58.280427+0000 front 2024-03-11T21:20:58.280453+0000 (oldest deadline 2024-03-11T21:21:19.880954+0000) 2024-03-11T21:21:25.251 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 11 21:21:25 smithi146 bash[38675]: debug 2024-03-11T21:21:25.025+0000 7fb0d14ce700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-11T21:20:55.673190+0000 front 2024-03-11T21:20:55.672758+0000 (oldest deadline 2024-03-11T21:21:20.372648+0000) 2024-03-11T21:21:25.751 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 11 21:21:25 smithi146 bash[44452]: debug 2024-03-11T21:21:25.473+0000 7f5d99d2d700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-11T21:20:55.488114+0000 front 2024-03-11T21:20:55.488176+0000 (oldest deadline 2024-03-11T21:21:21.387970+0000) 2024-03-11T21:21:25.752 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 11 21:21:25 smithi146 bash[32911]: debug 2024-03-11T21:21:25.677+0000 7f0b6ce00700 -1 osd.5 499 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-11T21:20:57.974158+0000 front 2024-03-11T21:20:57.974293+0000 (oldest deadline 2024-03-11T21:21:20.174723+0000) 2024-03-11T21:21:26.251 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 11 21:21:25 smithi146 bash[68242]: debug 2024-03-11T21:21:25.821+0000 7f6db0bde700 -1 osd.4 499 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-11T21:20:58.280427+0000 front 2024-03-11T21:20:58.280453+0000 (oldest deadline 2024-03-11T21:21:19.880954+0000) 2024-03-11T21:21:26.251 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 11 21:21:26 smithi146 bash[38675]: debug 2024-03-11T21:21:26.013+0000 7fb0d14ce700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-11T21:20:55.673190+0000 front 2024-03-11T21:20:55.672758+0000 (oldest deadline 2024-03-11T21:21:20.372648+0000) 2024-03-11T21:21:26.523 INFO:journalctl@ceph.osd.3.smithi023.stdout:Mar 11 21:21:26 smithi023 bash[169835]: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd-3 2024-03-11T21:21:26.523 INFO:journalctl@ceph.osd.3.smithi023.stdout:Mar 11 21:21:26 smithi023 bash[169899]: Error response from daemon: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd-3 2024-03-11T21:21:26.751 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 11 21:21:26 smithi146 bash[44452]: debug 2024-03-11T21:21:26.481+0000 7f5d99d2d700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-11T21:20:55.488114+0000 front 2024-03-11T21:20:55.488176+0000 (oldest deadline 2024-03-11T21:21:21.387970+0000) 2024-03-11T21:21:26.751 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 11 21:21:26 smithi146 bash[32911]: debug 2024-03-11T21:21:26.725+0000 7f0b6ce00700 -1 osd.5 499 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-11T21:20:57.974158+0000 front 2024-03-11T21:20:57.974293+0000 (oldest deadline 2024-03-11T21:21:20.174723+0000) 2024-03-11T21:21:27.251 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 11 21:21:26 smithi146 bash[68242]: debug 2024-03-11T21:21:26.777+0000 7f6db0bde700 -1 osd.4 499 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-11T21:20:58.280427+0000 front 2024-03-11T21:20:58.280453+0000 (oldest deadline 2024-03-11T21:21:19.880954+0000) 2024-03-11T21:21:27.251 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 11 21:21:27 smithi146 bash[38675]: debug 2024-03-11T21:21:26.997+0000 7fb0d14ce700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-11T21:20:55.673190+0000 front 2024-03-11T21:20:55.672758+0000 (oldest deadline 2024-03-11T21:21:20.372648+0000) 2024-03-11T21:21:27.718 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 11 21:21:27 smithi146 bash[44452]: debug 2024-03-11T21:21:27.469+0000 7f5d99d2d700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-11T21:20:55.488114+0000 front 2024-03-11T21:20:55.488176+0000 (oldest deadline 2024-03-11T21:21:21.387970+0000) 2024-03-11T21:21:27.994 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 11 21:21:27 smithi146 bash[68242]: debug 2024-03-11T21:21:27.785+0000 7f6db0bde700 -1 osd.4 499 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-11T21:20:58.280427+0000 front 2024-03-11T21:20:58.280453+0000 (oldest deadline 2024-03-11T21:21:19.880954+0000) 2024-03-11T21:21:27.994 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 11 21:21:27 smithi146 bash[32911]: debug 2024-03-11T21:21:27.729+0000 7f0b6ce00700 -1 osd.5 499 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-11T21:20:57.974158+0000 front 2024-03-11T21:20:57.974293+0000 (oldest deadline 2024-03-11T21:21:20.174723+0000) 2024-03-11T21:21:28.167 DEBUG:teuthology.orchestra.run.smithi023:> sudo pkill -f 'journalctl -f -n 0 -u ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@osd.3.service' 2024-03-11T21:21:28.195 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T21:21:28.195 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-03-11T21:21:28.196 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-03-11T21:21:28.196 DEBUG:teuthology.orchestra.run.smithi146:> sudo systemctl stop ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@osd.4 2024-03-11T21:21:28.251 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 11 21:21:28 smithi146 systemd[1]: Stopping Ceph osd.4 for 67ba00f8-dfe7-11ee-95c8-87774f69a715... 2024-03-11T21:21:28.251 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 11 21:21:28 smithi146 bash[38675]: debug 2024-03-11T21:21:28.009+0000 7fb0d14ce700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-11T21:20:55.673190+0000 front 2024-03-11T21:20:55.672758+0000 (oldest deadline 2024-03-11T21:21:20.372648+0000) 2024-03-11T21:21:28.488 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 11 21:21:28 smithi146 bash[68242]: debug 2024-03-11T21:21:28.373+0000 7f6db45ea700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-03-11T21:21:28.488 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 11 21:21:28 smithi146 bash[68242]: debug 2024-03-11T21:21:28.373+0000 7f6db45ea700 -1 osd.4 499 *** Got signal Terminated *** 2024-03-11T21:21:28.489 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 11 21:21:28 smithi146 bash[68242]: debug 2024-03-11T21:21:28.373+0000 7f6db45ea700 -1 osd.4 499 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T21:21:28.725 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 11 21:21:28 smithi146 bash[44452]: debug 2024-03-11T21:21:28.501+0000 7f5d99d2d700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-11T21:20:55.488114+0000 front 2024-03-11T21:20:55.488176+0000 (oldest deadline 2024-03-11T21:21:21.387970+0000) 2024-03-11T21:21:29.001 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 11 21:21:28 smithi146 bash[32911]: debug 2024-03-11T21:21:28.741+0000 7f0b6ce00700 -1 osd.5 499 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-11T21:20:57.974158+0000 front 2024-03-11T21:20:57.974293+0000 (oldest deadline 2024-03-11T21:21:20.174723+0000) 2024-03-11T21:21:29.001 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 11 21:21:28 smithi146 bash[68242]: debug 2024-03-11T21:21:28.785+0000 7f6db0bde700 -1 osd.4 499 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-11T21:20:58.280427+0000 front 2024-03-11T21:20:58.280453+0000 (oldest deadline 2024-03-11T21:21:19.880954+0000) 2024-03-11T21:21:29.001 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 11 21:21:28 smithi146 bash[38675]: debug 2024-03-11T21:21:28.961+0000 7fb0d14ce700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-11T21:20:55.673190+0000 front 2024-03-11T21:20:55.672758+0000 (oldest deadline 2024-03-11T21:21:20.372648+0000) 2024-03-11T21:21:29.752 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 11 21:21:29 smithi146 bash[44452]: debug 2024-03-11T21:21:29.477+0000 7f5d99d2d700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-11T21:20:55.488114+0000 front 2024-03-11T21:20:55.488176+0000 (oldest deadline 2024-03-11T21:21:21.387970+0000) 2024-03-11T21:21:30.251 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 11 21:21:29 smithi146 bash[68242]: debug 2024-03-11T21:21:29.773+0000 7f6db0bde700 -1 osd.4 499 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-11T21:20:58.280427+0000 front 2024-03-11T21:20:58.280453+0000 (oldest deadline 2024-03-11T21:21:19.880954+0000) 2024-03-11T21:21:30.251 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 11 21:21:30 smithi146 bash[38675]: debug 2024-03-11T21:21:29.997+0000 7fb0d14ce700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-11T21:20:55.673190+0000 front 2024-03-11T21:20:55.672758+0000 (oldest deadline 2024-03-11T21:21:20.372648+0000) 2024-03-11T21:21:30.252 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 11 21:21:29 smithi146 bash[32911]: debug 2024-03-11T21:21:29.777+0000 7f0b6ce00700 -1 osd.5 499 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-11T21:20:57.974158+0000 front 2024-03-11T21:20:57.974293+0000 (oldest deadline 2024-03-11T21:21:20.174723+0000) 2024-03-11T21:21:30.252 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 11 21:21:29 smithi146 bash[32911]: debug 2024-03-11T21:21:29.777+0000 7f0b6ce00700 -1 osd.5 499 heartbeat_check: no reply from 172.21.15.23:6814 osd.1 since back 2024-03-11T21:21:07.777019+0000 front 2024-03-11T21:21:07.776857+0000 (oldest deadline 2024-03-11T21:21:28.876529+0000) 2024-03-11T21:21:30.751 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 11 21:21:30 smithi146 bash[32911]: debug 2024-03-11T21:21:30.729+0000 7f0b6ce00700 -1 osd.5 499 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-11T21:20:57.974158+0000 front 2024-03-11T21:20:57.974293+0000 (oldest deadline 2024-03-11T21:21:20.174723+0000) 2024-03-11T21:21:30.751 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 11 21:21:30 smithi146 bash[32911]: debug 2024-03-11T21:21:30.729+0000 7f0b6ce00700 -1 osd.5 499 heartbeat_check: no reply from 172.21.15.23:6814 osd.1 since back 2024-03-11T21:21:07.777019+0000 front 2024-03-11T21:21:07.776857+0000 (oldest deadline 2024-03-11T21:21:28.876529+0000) 2024-03-11T21:21:30.751 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 11 21:21:30 smithi146 bash[44452]: debug 2024-03-11T21:21:30.493+0000 7f5d99d2d700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-11T21:20:55.488114+0000 front 2024-03-11T21:20:55.488176+0000 (oldest deadline 2024-03-11T21:21:21.387970+0000) 2024-03-11T21:21:30.752 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 11 21:21:30 smithi146 bash[44452]: debug 2024-03-11T21:21:30.493+0000 7f5d99d2d700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.23:6814 osd.1 since back 2024-03-11T21:21:06.089672+0000 front 2024-03-11T21:21:06.089360+0000 (oldest deadline 2024-03-11T21:21:29.589425+0000) 2024-03-11T21:21:31.251 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 11 21:21:30 smithi146 bash[68242]: debug 2024-03-11T21:21:30.813+0000 7f6db0bde700 -1 osd.4 499 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-11T21:20:58.280427+0000 front 2024-03-11T21:20:58.280453+0000 (oldest deadline 2024-03-11T21:21:19.880954+0000) 2024-03-11T21:21:31.251 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 11 21:21:31 smithi146 bash[38675]: debug 2024-03-11T21:21:31.009+0000 7fb0d14ce700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-11T21:20:55.673190+0000 front 2024-03-11T21:20:55.672758+0000 (oldest deadline 2024-03-11T21:21:20.372648+0000) 2024-03-11T21:21:31.251 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 11 21:21:31 smithi146 bash[38675]: debug 2024-03-11T21:21:31.009+0000 7fb0d14ce700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.23:6814 osd.1 since back 2024-03-11T21:21:06.775540+0000 front 2024-03-11T21:21:06.775088+0000 (oldest deadline 2024-03-11T21:21:30.875096+0000) 2024-03-11T21:21:31.751 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 11 21:21:31 smithi146 bash[44452]: debug 2024-03-11T21:21:31.469+0000 7f5d99d2d700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-11T21:20:55.488114+0000 front 2024-03-11T21:20:55.488176+0000 (oldest deadline 2024-03-11T21:21:21.387970+0000) 2024-03-11T21:21:31.751 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 11 21:21:31 smithi146 bash[44452]: debug 2024-03-11T21:21:31.469+0000 7f5d99d2d700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.23:6814 osd.1 since back 2024-03-11T21:21:06.089672+0000 front 2024-03-11T21:21:06.089360+0000 (oldest deadline 2024-03-11T21:21:29.589425+0000) 2024-03-11T21:21:32.251 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 11 21:21:31 smithi146 bash[68242]: debug 2024-03-11T21:21:31.829+0000 7f6db0bde700 -1 osd.4 499 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-11T21:20:58.280427+0000 front 2024-03-11T21:20:58.280453+0000 (oldest deadline 2024-03-11T21:21:19.880954+0000) 2024-03-11T21:21:32.251 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 11 21:21:31 smithi146 bash[68242]: debug 2024-03-11T21:21:31.829+0000 7f6db0bde700 -1 osd.4 499 heartbeat_check: no reply from 172.21.15.23:6814 osd.1 since back 2024-03-11T21:21:07.984136+0000 front 2024-03-11T21:21:07.984039+0000 (oldest deadline 2024-03-11T21:21:30.884115+0000) 2024-03-11T21:21:32.252 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 11 21:21:31 smithi146 bash[32911]: debug 2024-03-11T21:21:31.773+0000 7f0b6ce00700 -1 osd.5 499 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-11T21:20:57.974158+0000 front 2024-03-11T21:20:57.974293+0000 (oldest deadline 2024-03-11T21:21:20.174723+0000) 2024-03-11T21:21:32.252 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 11 21:21:31 smithi146 bash[32911]: debug 2024-03-11T21:21:31.773+0000 7f0b6ce00700 -1 osd.5 499 heartbeat_check: no reply from 172.21.15.23:6814 osd.1 since back 2024-03-11T21:21:07.777019+0000 front 2024-03-11T21:21:07.776857+0000 (oldest deadline 2024-03-11T21:21:28.876529+0000) 2024-03-11T21:21:32.252 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 11 21:21:32 smithi146 bash[38675]: debug 2024-03-11T21:21:31.997+0000 7fb0d14ce700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-11T21:20:55.673190+0000 front 2024-03-11T21:20:55.672758+0000 (oldest deadline 2024-03-11T21:21:20.372648+0000) 2024-03-11T21:21:32.252 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 11 21:21:32 smithi146 bash[38675]: debug 2024-03-11T21:21:31.997+0000 7fb0d14ce700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.23:6814 osd.1 since back 2024-03-11T21:21:06.775540+0000 front 2024-03-11T21:21:06.775088+0000 (oldest deadline 2024-03-11T21:21:30.875096+0000) 2024-03-11T21:21:32.751 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 11 21:21:32 smithi146 bash[32911]: debug 2024-03-11T21:21:32.733+0000 7f0b6ce00700 -1 osd.5 499 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-11T21:20:57.974158+0000 front 2024-03-11T21:20:57.974293+0000 (oldest deadline 2024-03-11T21:21:20.174723+0000) 2024-03-11T21:21:32.751 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 11 21:21:32 smithi146 bash[32911]: debug 2024-03-11T21:21:32.733+0000 7f0b6ce00700 -1 osd.5 499 heartbeat_check: no reply from 172.21.15.23:6814 osd.1 since back 2024-03-11T21:21:07.777019+0000 front 2024-03-11T21:21:07.776857+0000 (oldest deadline 2024-03-11T21:21:28.876529+0000) 2024-03-11T21:21:32.751 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 11 21:21:32 smithi146 bash[44452]: debug 2024-03-11T21:21:32.489+0000 7f5d99d2d700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-11T21:20:55.488114+0000 front 2024-03-11T21:20:55.488176+0000 (oldest deadline 2024-03-11T21:21:21.387970+0000) 2024-03-11T21:21:32.751 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 11 21:21:32 smithi146 bash[44452]: debug 2024-03-11T21:21:32.489+0000 7f5d99d2d700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.23:6814 osd.1 since back 2024-03-11T21:21:06.089672+0000 front 2024-03-11T21:21:06.089360+0000 (oldest deadline 2024-03-11T21:21:29.589425+0000) 2024-03-11T21:21:33.251 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 11 21:21:32 smithi146 bash[68242]: debug 2024-03-11T21:21:32.825+0000 7f6db0bde700 -1 osd.4 499 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-11T21:20:58.280427+0000 front 2024-03-11T21:20:58.280453+0000 (oldest deadline 2024-03-11T21:21:19.880954+0000) 2024-03-11T21:21:33.251 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 11 21:21:32 smithi146 bash[68242]: debug 2024-03-11T21:21:32.825+0000 7f6db0bde700 -1 osd.4 499 heartbeat_check: no reply from 172.21.15.23:6814 osd.1 since back 2024-03-11T21:21:07.984136+0000 front 2024-03-11T21:21:07.984039+0000 (oldest deadline 2024-03-11T21:21:30.884115+0000) 2024-03-11T21:21:33.251 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 11 21:21:33 smithi146 bash[38675]: debug 2024-03-11T21:21:32.997+0000 7fb0d14ce700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-11T21:20:55.673190+0000 front 2024-03-11T21:20:55.672758+0000 (oldest deadline 2024-03-11T21:21:20.372648+0000) 2024-03-11T21:21:33.251 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 11 21:21:33 smithi146 bash[38675]: debug 2024-03-11T21:21:32.997+0000 7fb0d14ce700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.23:6814 osd.1 since back 2024-03-11T21:21:06.775540+0000 front 2024-03-11T21:21:06.775088+0000 (oldest deadline 2024-03-11T21:21:30.875096+0000) 2024-03-11T21:21:33.751 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 11 21:21:33 smithi146 bash[32911]: debug 2024-03-11T21:21:33.701+0000 7f0b6ce00700 -1 osd.5 499 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-11T21:20:57.974158+0000 front 2024-03-11T21:20:57.974293+0000 (oldest deadline 2024-03-11T21:21:20.174723+0000) 2024-03-11T21:21:33.751 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 11 21:21:33 smithi146 bash[32911]: debug 2024-03-11T21:21:33.701+0000 7f0b6ce00700 -1 osd.5 499 heartbeat_check: no reply from 172.21.15.23:6814 osd.1 since back 2024-03-11T21:21:07.777019+0000 front 2024-03-11T21:21:07.776857+0000 (oldest deadline 2024-03-11T21:21:28.876529+0000) 2024-03-11T21:21:33.751 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 11 21:21:33 smithi146 bash[44452]: debug 2024-03-11T21:21:33.493+0000 7f5d99d2d700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-11T21:20:55.488114+0000 front 2024-03-11T21:20:55.488176+0000 (oldest deadline 2024-03-11T21:21:21.387970+0000) 2024-03-11T21:21:33.751 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 11 21:21:33 smithi146 bash[44452]: debug 2024-03-11T21:21:33.493+0000 7f5d99d2d700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.23:6814 osd.1 since back 2024-03-11T21:21:06.089672+0000 front 2024-03-11T21:21:06.089360+0000 (oldest deadline 2024-03-11T21:21:29.589425+0000) 2024-03-11T21:21:34.251 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 11 21:21:33 smithi146 bash[38675]: debug 2024-03-11T21:21:33.949+0000 7fb0d14ce700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-11T21:20:55.673190+0000 front 2024-03-11T21:20:55.672758+0000 (oldest deadline 2024-03-11T21:21:20.372648+0000) 2024-03-11T21:21:34.251 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 11 21:21:33 smithi146 bash[38675]: debug 2024-03-11T21:21:33.949+0000 7fb0d14ce700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.23:6814 osd.1 since back 2024-03-11T21:21:06.775540+0000 front 2024-03-11T21:21:06.775088+0000 (oldest deadline 2024-03-11T21:21:30.875096+0000) 2024-03-11T21:21:34.646 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 11 21:21:34 smithi146 bash[112337]: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd-4 2024-03-11T21:21:34.647 INFO:journalctl@ceph.osd.4.smithi146.stdout:Mar 11 21:21:34 smithi146 bash[112400]: Error response from daemon: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd-4 2024-03-11T21:21:34.647 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 11 21:21:34 smithi146 bash[44452]: debug 2024-03-11T21:21:34.453+0000 7f5d99d2d700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-11T21:20:55.488114+0000 front 2024-03-11T21:20:55.488176+0000 (oldest deadline 2024-03-11T21:21:21.387970+0000) 2024-03-11T21:21:34.647 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 11 21:21:34 smithi146 bash[44452]: debug 2024-03-11T21:21:34.453+0000 7f5d99d2d700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.23:6814 osd.1 since back 2024-03-11T21:21:06.089672+0000 front 2024-03-11T21:21:06.089360+0000 (oldest deadline 2024-03-11T21:21:29.589425+0000) 2024-03-11T21:21:35.001 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 11 21:21:34 smithi146 bash[32911]: debug 2024-03-11T21:21:34.741+0000 7f0b6ce00700 -1 osd.5 499 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-11T21:20:57.974158+0000 front 2024-03-11T21:20:57.974293+0000 (oldest deadline 2024-03-11T21:21:20.174723+0000) 2024-03-11T21:21:35.001 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 11 21:21:34 smithi146 bash[32911]: debug 2024-03-11T21:21:34.741+0000 7f0b6ce00700 -1 osd.5 499 heartbeat_check: no reply from 172.21.15.23:6814 osd.1 since back 2024-03-11T21:21:07.777019+0000 front 2024-03-11T21:21:07.776857+0000 (oldest deadline 2024-03-11T21:21:28.876529+0000) 2024-03-11T21:21:35.002 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 11 21:21:34 smithi146 bash[38675]: debug 2024-03-11T21:21:34.905+0000 7fb0d14ce700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-11T21:20:55.673190+0000 front 2024-03-11T21:20:55.672758+0000 (oldest deadline 2024-03-11T21:21:20.372648+0000) 2024-03-11T21:21:35.002 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 11 21:21:34 smithi146 bash[38675]: debug 2024-03-11T21:21:34.905+0000 7fb0d14ce700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.23:6814 osd.1 since back 2024-03-11T21:21:06.775540+0000 front 2024-03-11T21:21:06.775088+0000 (oldest deadline 2024-03-11T21:21:30.875096+0000) 2024-03-11T21:21:35.751 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 11 21:21:35 smithi146 bash[44452]: debug 2024-03-11T21:21:35.457+0000 7f5d99d2d700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-11T21:20:55.488114+0000 front 2024-03-11T21:20:55.488176+0000 (oldest deadline 2024-03-11T21:21:21.387970+0000) 2024-03-11T21:21:35.751 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 11 21:21:35 smithi146 bash[44452]: debug 2024-03-11T21:21:35.457+0000 7f5d99d2d700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.23:6814 osd.1 since back 2024-03-11T21:21:06.089672+0000 front 2024-03-11T21:21:06.089360+0000 (oldest deadline 2024-03-11T21:21:29.589425+0000) 2024-03-11T21:21:36.112 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 11 21:21:35 smithi146 bash[38675]: debug 2024-03-11T21:21:35.901+0000 7fb0d14ce700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-11T21:20:55.673190+0000 front 2024-03-11T21:20:55.672758+0000 (oldest deadline 2024-03-11T21:21:20.372648+0000) 2024-03-11T21:21:36.112 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 11 21:21:35 smithi146 bash[38675]: debug 2024-03-11T21:21:35.901+0000 7fb0d14ce700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.23:6814 osd.1 since back 2024-03-11T21:21:06.775540+0000 front 2024-03-11T21:21:06.775088+0000 (oldest deadline 2024-03-11T21:21:30.875096+0000) 2024-03-11T21:21:36.112 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 11 21:21:35 smithi146 bash[32911]: debug 2024-03-11T21:21:35.793+0000 7f0b6ce00700 -1 osd.5 499 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-11T21:20:57.974158+0000 front 2024-03-11T21:20:57.974293+0000 (oldest deadline 2024-03-11T21:21:20.174723+0000) 2024-03-11T21:21:36.112 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 11 21:21:35 smithi146 bash[32911]: debug 2024-03-11T21:21:35.793+0000 7f0b6ce00700 -1 osd.5 499 heartbeat_check: no reply from 172.21.15.23:6814 osd.1 since back 2024-03-11T21:21:07.777019+0000 front 2024-03-11T21:21:07.776857+0000 (oldest deadline 2024-03-11T21:21:28.876529+0000) 2024-03-11T21:21:36.578 DEBUG:teuthology.orchestra.run.smithi146:> sudo pkill -f 'journalctl -f -n 0 -u ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@osd.4.service' 2024-03-11T21:21:36.608 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T21:21:36.608 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-03-11T21:21:36.608 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-03-11T21:21:36.608 DEBUG:teuthology.orchestra.run.smithi146:> sudo systemctl stop ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@osd.5 2024-03-11T21:21:36.749 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 11 21:21:36 smithi146 systemd[1]: Stopping Ceph osd.5 for 67ba00f8-dfe7-11ee-95c8-87774f69a715... 2024-03-11T21:21:36.750 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 11 21:21:36 smithi146 bash[44452]: debug 2024-03-11T21:21:36.469+0000 7f5d99d2d700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-11T21:20:55.488114+0000 front 2024-03-11T21:20:55.488176+0000 (oldest deadline 2024-03-11T21:21:21.387970+0000) 2024-03-11T21:21:36.750 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 11 21:21:36 smithi146 bash[44452]: debug 2024-03-11T21:21:36.469+0000 7f5d99d2d700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.23:6814 osd.1 since back 2024-03-11T21:21:06.089672+0000 front 2024-03-11T21:21:06.089360+0000 (oldest deadline 2024-03-11T21:21:29.589425+0000) 2024-03-11T21:21:36.904 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 11 21:21:36 smithi146 bash[32911]: debug 2024-03-11T21:21:36.761+0000 7f0b6ce00700 -1 osd.5 499 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-11T21:20:57.974158+0000 front 2024-03-11T21:20:57.974293+0000 (oldest deadline 2024-03-11T21:21:20.174723+0000) 2024-03-11T21:21:36.905 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 11 21:21:36 smithi146 bash[32911]: debug 2024-03-11T21:21:36.761+0000 7f0b6ce00700 -1 osd.5 499 heartbeat_check: no reply from 172.21.15.23:6814 osd.1 since back 2024-03-11T21:21:07.777019+0000 front 2024-03-11T21:21:07.776857+0000 (oldest deadline 2024-03-11T21:21:28.876529+0000) 2024-03-11T21:21:36.905 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 11 21:21:36 smithi146 bash[32911]: debug 2024-03-11T21:21:36.773+0000 7f0b74814700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-03-11T21:21:36.905 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 11 21:21:36 smithi146 bash[32911]: debug 2024-03-11T21:21:36.773+0000 7f0b74814700 -1 osd.5 499 *** Got signal Terminated *** 2024-03-11T21:21:36.905 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 11 21:21:36 smithi146 bash[32911]: debug 2024-03-11T21:21:36.773+0000 7f0b74814700 -1 osd.5 499 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T21:21:37.001 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 11 21:21:36 smithi146 bash[38675]: debug 2024-03-11T21:21:36.917+0000 7fb0d14ce700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-11T21:20:55.673190+0000 front 2024-03-11T21:20:55.672758+0000 (oldest deadline 2024-03-11T21:21:20.372648+0000) 2024-03-11T21:21:37.001 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 11 21:21:36 smithi146 bash[38675]: debug 2024-03-11T21:21:36.917+0000 7fb0d14ce700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.23:6814 osd.1 since back 2024-03-11T21:21:06.775540+0000 front 2024-03-11T21:21:06.775088+0000 (oldest deadline 2024-03-11T21:21:30.875096+0000) 2024-03-11T21:21:37.751 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 11 21:21:37 smithi146 bash[44452]: debug 2024-03-11T21:21:37.501+0000 7f5d99d2d700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-11T21:20:55.488114+0000 front 2024-03-11T21:20:55.488176+0000 (oldest deadline 2024-03-11T21:21:21.387970+0000) 2024-03-11T21:21:37.751 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 11 21:21:37 smithi146 bash[44452]: debug 2024-03-11T21:21:37.501+0000 7f5d99d2d700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.23:6814 osd.1 since back 2024-03-11T21:21:06.089672+0000 front 2024-03-11T21:21:06.089360+0000 (oldest deadline 2024-03-11T21:21:29.589425+0000) 2024-03-11T21:21:38.251 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 11 21:21:37 smithi146 bash[32911]: debug 2024-03-11T21:21:37.781+0000 7f0b6ce00700 -1 osd.5 499 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-11T21:20:57.974158+0000 front 2024-03-11T21:20:57.974293+0000 (oldest deadline 2024-03-11T21:21:20.174723+0000) 2024-03-11T21:21:38.251 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 11 21:21:37 smithi146 bash[32911]: debug 2024-03-11T21:21:37.781+0000 7f0b6ce00700 -1 osd.5 499 heartbeat_check: no reply from 172.21.15.23:6814 osd.1 since back 2024-03-11T21:21:07.777019+0000 front 2024-03-11T21:21:07.776857+0000 (oldest deadline 2024-03-11T21:21:28.876529+0000) 2024-03-11T21:21:38.251 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 11 21:21:37 smithi146 bash[38675]: debug 2024-03-11T21:21:37.953+0000 7fb0d14ce700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-11T21:20:55.673190+0000 front 2024-03-11T21:20:55.672758+0000 (oldest deadline 2024-03-11T21:21:20.372648+0000) 2024-03-11T21:21:38.251 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 11 21:21:37 smithi146 bash[38675]: debug 2024-03-11T21:21:37.953+0000 7fb0d14ce700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.23:6814 osd.1 since back 2024-03-11T21:21:06.775540+0000 front 2024-03-11T21:21:06.775088+0000 (oldest deadline 2024-03-11T21:21:30.875096+0000) 2024-03-11T21:21:38.751 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 11 21:21:38 smithi146 bash[44452]: debug 2024-03-11T21:21:38.469+0000 7f5d99d2d700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-11T21:20:55.488114+0000 front 2024-03-11T21:20:55.488176+0000 (oldest deadline 2024-03-11T21:21:21.387970+0000) 2024-03-11T21:21:38.751 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 11 21:21:38 smithi146 bash[44452]: debug 2024-03-11T21:21:38.469+0000 7f5d99d2d700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.23:6814 osd.1 since back 2024-03-11T21:21:06.089672+0000 front 2024-03-11T21:21:06.089360+0000 (oldest deadline 2024-03-11T21:21:29.589425+0000) 2024-03-11T21:21:39.251 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 11 21:21:38 smithi146 bash[32911]: debug 2024-03-11T21:21:38.805+0000 7f0b6ce00700 -1 osd.5 499 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-11T21:20:57.974158+0000 front 2024-03-11T21:20:57.974293+0000 (oldest deadline 2024-03-11T21:21:20.174723+0000) 2024-03-11T21:21:39.251 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 11 21:21:38 smithi146 bash[32911]: debug 2024-03-11T21:21:38.805+0000 7f0b6ce00700 -1 osd.5 499 heartbeat_check: no reply from 172.21.15.23:6814 osd.1 since back 2024-03-11T21:21:07.777019+0000 front 2024-03-11T21:21:07.776857+0000 (oldest deadline 2024-03-11T21:21:28.876529+0000) 2024-03-11T21:21:39.252 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 11 21:21:38 smithi146 bash[38675]: debug 2024-03-11T21:21:38.973+0000 7fb0d14ce700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-11T21:20:55.673190+0000 front 2024-03-11T21:20:55.672758+0000 (oldest deadline 2024-03-11T21:21:20.372648+0000) 2024-03-11T21:21:39.252 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 11 21:21:38 smithi146 bash[38675]: debug 2024-03-11T21:21:38.973+0000 7fb0d14ce700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.23:6814 osd.1 since back 2024-03-11T21:21:06.775540+0000 front 2024-03-11T21:21:06.775088+0000 (oldest deadline 2024-03-11T21:21:30.875096+0000) 2024-03-11T21:21:39.751 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 11 21:21:39 smithi146 bash[44452]: debug 2024-03-11T21:21:39.465+0000 7f5d99d2d700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-11T21:20:55.488114+0000 front 2024-03-11T21:20:55.488176+0000 (oldest deadline 2024-03-11T21:21:21.387970+0000) 2024-03-11T21:21:39.751 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 11 21:21:39 smithi146 bash[44452]: debug 2024-03-11T21:21:39.465+0000 7f5d99d2d700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.23:6814 osd.1 since back 2024-03-11T21:21:06.089672+0000 front 2024-03-11T21:21:06.089360+0000 (oldest deadline 2024-03-11T21:21:29.589425+0000) 2024-03-11T21:21:39.751 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 11 21:21:39 smithi146 bash[44452]: debug 2024-03-11T21:21:39.465+0000 7f5d99d2d700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.23:6822 osd.2 since back 2024-03-11T21:21:16.293816+0000 front 2024-03-11T21:21:16.293862+0000 (oldest deadline 2024-03-11T21:21:38.593400+0000) 2024-03-11T21:21:40.251 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 11 21:21:39 smithi146 bash[32911]: debug 2024-03-11T21:21:39.785+0000 7f0b6ce00700 -1 osd.5 499 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-11T21:20:57.974158+0000 front 2024-03-11T21:20:57.974293+0000 (oldest deadline 2024-03-11T21:21:20.174723+0000) 2024-03-11T21:21:40.251 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 11 21:21:39 smithi146 bash[32911]: debug 2024-03-11T21:21:39.785+0000 7f0b6ce00700 -1 osd.5 499 heartbeat_check: no reply from 172.21.15.23:6814 osd.1 since back 2024-03-11T21:21:07.777019+0000 front 2024-03-11T21:21:07.776857+0000 (oldest deadline 2024-03-11T21:21:28.876529+0000) 2024-03-11T21:21:40.251 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 11 21:21:39 smithi146 bash[38675]: debug 2024-03-11T21:21:39.993+0000 7fb0d14ce700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-11T21:20:55.673190+0000 front 2024-03-11T21:20:55.672758+0000 (oldest deadline 2024-03-11T21:21:20.372648+0000) 2024-03-11T21:21:40.251 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 11 21:21:39 smithi146 bash[38675]: debug 2024-03-11T21:21:39.993+0000 7fb0d14ce700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.23:6814 osd.1 since back 2024-03-11T21:21:06.775540+0000 front 2024-03-11T21:21:06.775088+0000 (oldest deadline 2024-03-11T21:21:30.875096+0000) 2024-03-11T21:21:40.252 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 11 21:21:39 smithi146 bash[38675]: debug 2024-03-11T21:21:39.993+0000 7fb0d14ce700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.23:6822 osd.2 since back 2024-03-11T21:21:13.776564+0000 front 2024-03-11T21:21:13.776827+0000 (oldest deadline 2024-03-11T21:21:39.076442+0000) 2024-03-11T21:21:40.751 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 11 21:21:40 smithi146 bash[44452]: debug 2024-03-11T21:21:40.441+0000 7f5d99d2d700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-11T21:20:55.488114+0000 front 2024-03-11T21:20:55.488176+0000 (oldest deadline 2024-03-11T21:21:21.387970+0000) 2024-03-11T21:21:40.751 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 11 21:21:40 smithi146 bash[44452]: debug 2024-03-11T21:21:40.441+0000 7f5d99d2d700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.23:6814 osd.1 since back 2024-03-11T21:21:06.089672+0000 front 2024-03-11T21:21:06.089360+0000 (oldest deadline 2024-03-11T21:21:29.589425+0000) 2024-03-11T21:21:40.751 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 11 21:21:40 smithi146 bash[44452]: debug 2024-03-11T21:21:40.441+0000 7f5d99d2d700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.23:6822 osd.2 since back 2024-03-11T21:21:16.293816+0000 front 2024-03-11T21:21:16.293862+0000 (oldest deadline 2024-03-11T21:21:38.593400+0000) 2024-03-11T21:21:41.251 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 11 21:21:40 smithi146 bash[32911]: debug 2024-03-11T21:21:40.829+0000 7f0b6ce00700 -1 osd.5 499 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-11T21:20:57.974158+0000 front 2024-03-11T21:20:57.974293+0000 (oldest deadline 2024-03-11T21:21:20.174723+0000) 2024-03-11T21:21:41.251 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 11 21:21:40 smithi146 bash[32911]: debug 2024-03-11T21:21:40.829+0000 7f0b6ce00700 -1 osd.5 499 heartbeat_check: no reply from 172.21.15.23:6814 osd.1 since back 2024-03-11T21:21:07.777019+0000 front 2024-03-11T21:21:07.776857+0000 (oldest deadline 2024-03-11T21:21:28.876529+0000) 2024-03-11T21:21:41.251 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 11 21:21:40 smithi146 bash[32911]: debug 2024-03-11T21:21:40.829+0000 7f0b6ce00700 -1 osd.5 499 heartbeat_check: no reply from 172.21.15.23:6822 osd.2 since back 2024-03-11T21:21:15.878422+0000 front 2024-03-11T21:21:15.878617+0000 (oldest deadline 2024-03-11T21:21:40.478942+0000) 2024-03-11T21:21:41.252 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 11 21:21:41 smithi146 bash[38675]: debug 2024-03-11T21:21:41.001+0000 7fb0d14ce700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-11T21:20:55.673190+0000 front 2024-03-11T21:20:55.672758+0000 (oldest deadline 2024-03-11T21:21:20.372648+0000) 2024-03-11T21:21:41.252 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 11 21:21:41 smithi146 bash[38675]: debug 2024-03-11T21:21:41.001+0000 7fb0d14ce700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.23:6814 osd.1 since back 2024-03-11T21:21:06.775540+0000 front 2024-03-11T21:21:06.775088+0000 (oldest deadline 2024-03-11T21:21:30.875096+0000) 2024-03-11T21:21:41.252 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 11 21:21:41 smithi146 bash[38675]: debug 2024-03-11T21:21:41.001+0000 7fb0d14ce700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.23:6822 osd.2 since back 2024-03-11T21:21:13.776564+0000 front 2024-03-11T21:21:13.776827+0000 (oldest deadline 2024-03-11T21:21:39.076442+0000) 2024-03-11T21:21:41.751 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 11 21:21:41 smithi146 bash[44452]: debug 2024-03-11T21:21:41.473+0000 7f5d99d2d700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-11T21:20:55.488114+0000 front 2024-03-11T21:20:55.488176+0000 (oldest deadline 2024-03-11T21:21:21.387970+0000) 2024-03-11T21:21:41.751 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 11 21:21:41 smithi146 bash[44452]: debug 2024-03-11T21:21:41.473+0000 7f5d99d2d700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.23:6814 osd.1 since back 2024-03-11T21:21:06.089672+0000 front 2024-03-11T21:21:06.089360+0000 (oldest deadline 2024-03-11T21:21:29.589425+0000) 2024-03-11T21:21:41.751 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 11 21:21:41 smithi146 bash[44452]: debug 2024-03-11T21:21:41.473+0000 7f5d99d2d700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.23:6822 osd.2 since back 2024-03-11T21:21:16.293816+0000 front 2024-03-11T21:21:16.293862+0000 (oldest deadline 2024-03-11T21:21:38.593400+0000) 2024-03-11T21:21:42.267 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 11 21:21:42 smithi146 bash[38675]: debug 2024-03-11T21:21:42.029+0000 7fb0d14ce700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-11T21:20:55.673190+0000 front 2024-03-11T21:20:55.672758+0000 (oldest deadline 2024-03-11T21:21:20.372648+0000) 2024-03-11T21:21:42.267 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 11 21:21:42 smithi146 bash[38675]: debug 2024-03-11T21:21:42.029+0000 7fb0d14ce700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.23:6814 osd.1 since back 2024-03-11T21:21:06.775540+0000 front 2024-03-11T21:21:06.775088+0000 (oldest deadline 2024-03-11T21:21:30.875096+0000) 2024-03-11T21:21:42.267 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 11 21:21:42 smithi146 bash[38675]: debug 2024-03-11T21:21:42.029+0000 7fb0d14ce700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.23:6822 osd.2 since back 2024-03-11T21:21:13.776564+0000 front 2024-03-11T21:21:13.776827+0000 (oldest deadline 2024-03-11T21:21:39.076442+0000) 2024-03-11T21:21:42.751 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 11 21:21:42 smithi146 bash[44452]: debug 2024-03-11T21:21:42.428+0000 7f5d99d2d700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-11T21:20:55.488114+0000 front 2024-03-11T21:20:55.488176+0000 (oldest deadline 2024-03-11T21:21:21.387970+0000) 2024-03-11T21:21:42.751 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 11 21:21:42 smithi146 bash[44452]: debug 2024-03-11T21:21:42.428+0000 7f5d99d2d700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.23:6814 osd.1 since back 2024-03-11T21:21:06.089672+0000 front 2024-03-11T21:21:06.089360+0000 (oldest deadline 2024-03-11T21:21:29.589425+0000) 2024-03-11T21:21:42.751 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 11 21:21:42 smithi146 bash[44452]: debug 2024-03-11T21:21:42.428+0000 7f5d99d2d700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.23:6822 osd.2 since back 2024-03-11T21:21:16.293816+0000 front 2024-03-11T21:21:16.293862+0000 (oldest deadline 2024-03-11T21:21:38.593400+0000) 2024-03-11T21:21:43.168 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 11 21:21:42 smithi146 bash[112534]: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd-5 2024-03-11T21:21:43.169 INFO:journalctl@ceph.osd.5.smithi146.stdout:Mar 11 21:21:43 smithi146 bash[112604]: Error response from daemon: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd-5 2024-03-11T21:21:43.169 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 11 21:21:43 smithi146 bash[38675]: debug 2024-03-11T21:21:43.060+0000 7fb0d14ce700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-11T21:20:55.673190+0000 front 2024-03-11T21:20:55.672758+0000 (oldest deadline 2024-03-11T21:21:20.372648+0000) 2024-03-11T21:21:43.169 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 11 21:21:43 smithi146 bash[38675]: debug 2024-03-11T21:21:43.060+0000 7fb0d14ce700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.23:6814 osd.1 since back 2024-03-11T21:21:06.775540+0000 front 2024-03-11T21:21:06.775088+0000 (oldest deadline 2024-03-11T21:21:30.875096+0000) 2024-03-11T21:21:43.169 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 11 21:21:43 smithi146 bash[38675]: debug 2024-03-11T21:21:43.060+0000 7fb0d14ce700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.23:6822 osd.2 since back 2024-03-11T21:21:13.776564+0000 front 2024-03-11T21:21:13.776827+0000 (oldest deadline 2024-03-11T21:21:39.076442+0000) 2024-03-11T21:21:43.751 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 11 21:21:43 smithi146 bash[44452]: debug 2024-03-11T21:21:43.456+0000 7f5d99d2d700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-11T21:20:55.488114+0000 front 2024-03-11T21:20:55.488176+0000 (oldest deadline 2024-03-11T21:21:21.387970+0000) 2024-03-11T21:21:43.751 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 11 21:21:43 smithi146 bash[44452]: debug 2024-03-11T21:21:43.456+0000 7f5d99d2d700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.23:6814 osd.1 since back 2024-03-11T21:21:06.089672+0000 front 2024-03-11T21:21:06.089360+0000 (oldest deadline 2024-03-11T21:21:29.589425+0000) 2024-03-11T21:21:43.751 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 11 21:21:43 smithi146 bash[44452]: debug 2024-03-11T21:21:43.456+0000 7f5d99d2d700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.23:6822 osd.2 since back 2024-03-11T21:21:16.293816+0000 front 2024-03-11T21:21:16.293862+0000 (oldest deadline 2024-03-11T21:21:38.593400+0000) 2024-03-11T21:21:44.239 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 11 21:21:44 smithi146 bash[38675]: debug 2024-03-11T21:21:44.108+0000 7fb0d14ce700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-11T21:20:55.673190+0000 front 2024-03-11T21:20:55.672758+0000 (oldest deadline 2024-03-11T21:21:20.372648+0000) 2024-03-11T21:21:44.240 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 11 21:21:44 smithi146 bash[38675]: debug 2024-03-11T21:21:44.108+0000 7fb0d14ce700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.23:6814 osd.1 since back 2024-03-11T21:21:06.775540+0000 front 2024-03-11T21:21:06.775088+0000 (oldest deadline 2024-03-11T21:21:30.875096+0000) 2024-03-11T21:21:44.240 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 11 21:21:44 smithi146 bash[38675]: debug 2024-03-11T21:21:44.108+0000 7fb0d14ce700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.23:6822 osd.2 since back 2024-03-11T21:21:13.776564+0000 front 2024-03-11T21:21:13.776827+0000 (oldest deadline 2024-03-11T21:21:39.076442+0000) 2024-03-11T21:21:44.501 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 11 21:21:44 smithi146 bash[44452]: debug 2024-03-11T21:21:44.444+0000 7f5d99d2d700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-11T21:20:55.488114+0000 front 2024-03-11T21:20:55.488176+0000 (oldest deadline 2024-03-11T21:21:21.387970+0000) 2024-03-11T21:21:44.518 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 11 21:21:44 smithi146 bash[44452]: debug 2024-03-11T21:21:44.444+0000 7f5d99d2d700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.23:6814 osd.1 since back 2024-03-11T21:21:06.089672+0000 front 2024-03-11T21:21:06.089360+0000 (oldest deadline 2024-03-11T21:21:29.589425+0000) 2024-03-11T21:21:44.518 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 11 21:21:44 smithi146 bash[44452]: debug 2024-03-11T21:21:44.444+0000 7f5d99d2d700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.23:6822 osd.2 since back 2024-03-11T21:21:16.293816+0000 front 2024-03-11T21:21:16.293862+0000 (oldest deadline 2024-03-11T21:21:38.593400+0000) 2024-03-11T21:21:44.869 DEBUG:teuthology.orchestra.run.smithi146:> sudo pkill -f 'journalctl -f -n 0 -u ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@osd.5.service' 2024-03-11T21:21:44.895 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T21:21:44.895 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-03-11T21:21:44.896 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-03-11T21:21:44.896 DEBUG:teuthology.orchestra.run.smithi146:> sudo systemctl stop ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@osd.6 2024-03-11T21:21:45.251 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 11 21:21:44 smithi146 systemd[1]: Stopping Ceph osd.6 for 67ba00f8-dfe7-11ee-95c8-87774f69a715... 2024-03-11T21:21:45.251 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 11 21:21:45 smithi146 bash[38675]: debug 2024-03-11T21:21:45.076+0000 7fb0d8ee2700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-03-11T21:21:45.251 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 11 21:21:45 smithi146 bash[38675]: debug 2024-03-11T21:21:45.076+0000 7fb0d8ee2700 -1 osd.6 499 *** Got signal Terminated *** 2024-03-11T21:21:45.251 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 11 21:21:45 smithi146 bash[38675]: debug 2024-03-11T21:21:45.076+0000 7fb0d8ee2700 -1 osd.6 499 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T21:21:45.251 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 11 21:21:45 smithi146 bash[38675]: debug 2024-03-11T21:21:45.096+0000 7fb0d14ce700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-11T21:20:55.673190+0000 front 2024-03-11T21:20:55.672758+0000 (oldest deadline 2024-03-11T21:21:20.372648+0000) 2024-03-11T21:21:45.252 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 11 21:21:45 smithi146 bash[38675]: debug 2024-03-11T21:21:45.096+0000 7fb0d14ce700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.23:6814 osd.1 since back 2024-03-11T21:21:06.775540+0000 front 2024-03-11T21:21:06.775088+0000 (oldest deadline 2024-03-11T21:21:30.875096+0000) 2024-03-11T21:21:45.252 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 11 21:21:45 smithi146 bash[38675]: debug 2024-03-11T21:21:45.096+0000 7fb0d14ce700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.23:6822 osd.2 since back 2024-03-11T21:21:13.776564+0000 front 2024-03-11T21:21:13.776827+0000 (oldest deadline 2024-03-11T21:21:39.076442+0000) 2024-03-11T21:21:45.751 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 11 21:21:45 smithi146 bash[44452]: debug 2024-03-11T21:21:45.476+0000 7f5d99d2d700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-11T21:20:55.488114+0000 front 2024-03-11T21:20:55.488176+0000 (oldest deadline 2024-03-11T21:21:21.387970+0000) 2024-03-11T21:21:45.751 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 11 21:21:45 smithi146 bash[44452]: debug 2024-03-11T21:21:45.476+0000 7f5d99d2d700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.23:6814 osd.1 since back 2024-03-11T21:21:06.089672+0000 front 2024-03-11T21:21:06.089360+0000 (oldest deadline 2024-03-11T21:21:29.589425+0000) 2024-03-11T21:21:45.751 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 11 21:21:45 smithi146 bash[44452]: debug 2024-03-11T21:21:45.476+0000 7f5d99d2d700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.23:6822 osd.2 since back 2024-03-11T21:21:16.293816+0000 front 2024-03-11T21:21:16.293862+0000 (oldest deadline 2024-03-11T21:21:38.593400+0000) 2024-03-11T21:21:46.432 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 11 21:21:46 smithi146 bash[38675]: debug 2024-03-11T21:21:46.068+0000 7fb0d14ce700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-11T21:20:55.673190+0000 front 2024-03-11T21:20:55.672758+0000 (oldest deadline 2024-03-11T21:21:20.372648+0000) 2024-03-11T21:21:46.432 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 11 21:21:46 smithi146 bash[38675]: debug 2024-03-11T21:21:46.068+0000 7fb0d14ce700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.23:6814 osd.1 since back 2024-03-11T21:21:06.775540+0000 front 2024-03-11T21:21:06.775088+0000 (oldest deadline 2024-03-11T21:21:30.875096+0000) 2024-03-11T21:21:46.432 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 11 21:21:46 smithi146 bash[38675]: debug 2024-03-11T21:21:46.068+0000 7fb0d14ce700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.23:6822 osd.2 since back 2024-03-11T21:21:13.776564+0000 front 2024-03-11T21:21:13.776827+0000 (oldest deadline 2024-03-11T21:21:39.076442+0000) 2024-03-11T21:21:46.751 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 11 21:21:46 smithi146 bash[44452]: debug 2024-03-11T21:21:46.448+0000 7f5d99d2d700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-11T21:20:55.488114+0000 front 2024-03-11T21:20:55.488176+0000 (oldest deadline 2024-03-11T21:21:21.387970+0000) 2024-03-11T21:21:46.751 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 11 21:21:46 smithi146 bash[44452]: debug 2024-03-11T21:21:46.448+0000 7f5d99d2d700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.23:6814 osd.1 since back 2024-03-11T21:21:06.089672+0000 front 2024-03-11T21:21:06.089360+0000 (oldest deadline 2024-03-11T21:21:29.589425+0000) 2024-03-11T21:21:46.752 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 11 21:21:46 smithi146 bash[44452]: debug 2024-03-11T21:21:46.448+0000 7f5d99d2d700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.23:6822 osd.2 since back 2024-03-11T21:21:16.293816+0000 front 2024-03-11T21:21:16.293862+0000 (oldest deadline 2024-03-11T21:21:38.593400+0000) 2024-03-11T21:21:47.425 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 11 21:21:47 smithi146 bash[38675]: debug 2024-03-11T21:21:47.060+0000 7fb0d14ce700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-11T21:20:55.673190+0000 front 2024-03-11T21:20:55.672758+0000 (oldest deadline 2024-03-11T21:21:20.372648+0000) 2024-03-11T21:21:47.426 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 11 21:21:47 smithi146 bash[38675]: debug 2024-03-11T21:21:47.060+0000 7fb0d14ce700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.23:6814 osd.1 since back 2024-03-11T21:21:06.775540+0000 front 2024-03-11T21:21:06.775088+0000 (oldest deadline 2024-03-11T21:21:30.875096+0000) 2024-03-11T21:21:47.426 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 11 21:21:47 smithi146 bash[38675]: debug 2024-03-11T21:21:47.060+0000 7fb0d14ce700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.23:6822 osd.2 since back 2024-03-11T21:21:13.776564+0000 front 2024-03-11T21:21:13.776827+0000 (oldest deadline 2024-03-11T21:21:39.076442+0000) 2024-03-11T21:21:47.751 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 11 21:21:47 smithi146 bash[44452]: debug 2024-03-11T21:21:47.440+0000 7f5d99d2d700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-11T21:20:55.488114+0000 front 2024-03-11T21:20:55.488176+0000 (oldest deadline 2024-03-11T21:21:21.387970+0000) 2024-03-11T21:21:47.751 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 11 21:21:47 smithi146 bash[44452]: debug 2024-03-11T21:21:47.440+0000 7f5d99d2d700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.23:6814 osd.1 since back 2024-03-11T21:21:06.089672+0000 front 2024-03-11T21:21:06.089360+0000 (oldest deadline 2024-03-11T21:21:29.589425+0000) 2024-03-11T21:21:47.751 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 11 21:21:47 smithi146 bash[44452]: debug 2024-03-11T21:21:47.440+0000 7f5d99d2d700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.23:6822 osd.2 since back 2024-03-11T21:21:16.293816+0000 front 2024-03-11T21:21:16.293862+0000 (oldest deadline 2024-03-11T21:21:38.593400+0000) 2024-03-11T21:21:48.421 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 11 21:21:48 smithi146 bash[38675]: debug 2024-03-11T21:21:48.040+0000 7fb0d14ce700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-11T21:20:55.673190+0000 front 2024-03-11T21:20:55.672758+0000 (oldest deadline 2024-03-11T21:21:20.372648+0000) 2024-03-11T21:21:48.421 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 11 21:21:48 smithi146 bash[38675]: debug 2024-03-11T21:21:48.040+0000 7fb0d14ce700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.23:6814 osd.1 since back 2024-03-11T21:21:06.775540+0000 front 2024-03-11T21:21:06.775088+0000 (oldest deadline 2024-03-11T21:21:30.875096+0000) 2024-03-11T21:21:48.421 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 11 21:21:48 smithi146 bash[38675]: debug 2024-03-11T21:21:48.040+0000 7fb0d14ce700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.23:6822 osd.2 since back 2024-03-11T21:21:13.776564+0000 front 2024-03-11T21:21:13.776827+0000 (oldest deadline 2024-03-11T21:21:39.076442+0000) 2024-03-11T21:21:48.421 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 11 21:21:48 smithi146 bash[38675]: debug 2024-03-11T21:21:48.040+0000 7fb0d14ce700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.23:6830 osd.3 since back 2024-03-11T21:21:24.878677+0000 front 2024-03-11T21:21:24.878736+0000 (oldest deadline 2024-03-11T21:21:47.778538+0000) 2024-03-11T21:21:48.751 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 11 21:21:48 smithi146 bash[44452]: debug 2024-03-11T21:21:48.436+0000 7f5d99d2d700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-11T21:20:55.488114+0000 front 2024-03-11T21:20:55.488176+0000 (oldest deadline 2024-03-11T21:21:21.387970+0000) 2024-03-11T21:21:48.751 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 11 21:21:48 smithi146 bash[44452]: debug 2024-03-11T21:21:48.436+0000 7f5d99d2d700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.23:6814 osd.1 since back 2024-03-11T21:21:06.089672+0000 front 2024-03-11T21:21:06.089360+0000 (oldest deadline 2024-03-11T21:21:29.589425+0000) 2024-03-11T21:21:48.751 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 11 21:21:48 smithi146 bash[44452]: debug 2024-03-11T21:21:48.436+0000 7f5d99d2d700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.23:6822 osd.2 since back 2024-03-11T21:21:16.293816+0000 front 2024-03-11T21:21:16.293862+0000 (oldest deadline 2024-03-11T21:21:38.593400+0000) 2024-03-11T21:21:49.251 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 11 21:21:49 smithi146 bash[38675]: debug 2024-03-11T21:21:48.996+0000 7fb0d14ce700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-11T21:20:55.673190+0000 front 2024-03-11T21:20:55.672758+0000 (oldest deadline 2024-03-11T21:21:20.372648+0000) 2024-03-11T21:21:49.251 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 11 21:21:49 smithi146 bash[38675]: debug 2024-03-11T21:21:48.996+0000 7fb0d14ce700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.23:6814 osd.1 since back 2024-03-11T21:21:06.775540+0000 front 2024-03-11T21:21:06.775088+0000 (oldest deadline 2024-03-11T21:21:30.875096+0000) 2024-03-11T21:21:49.251 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 11 21:21:49 smithi146 bash[38675]: debug 2024-03-11T21:21:48.996+0000 7fb0d14ce700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.23:6822 osd.2 since back 2024-03-11T21:21:13.776564+0000 front 2024-03-11T21:21:13.776827+0000 (oldest deadline 2024-03-11T21:21:39.076442+0000) 2024-03-11T21:21:49.251 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 11 21:21:49 smithi146 bash[38675]: debug 2024-03-11T21:21:48.996+0000 7fb0d14ce700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.23:6830 osd.3 since back 2024-03-11T21:21:24.878677+0000 front 2024-03-11T21:21:24.878736+0000 (oldest deadline 2024-03-11T21:21:47.778538+0000) 2024-03-11T21:21:49.751 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 11 21:21:49 smithi146 bash[44452]: debug 2024-03-11T21:21:49.428+0000 7f5d99d2d700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-11T21:20:55.488114+0000 front 2024-03-11T21:20:55.488176+0000 (oldest deadline 2024-03-11T21:21:21.387970+0000) 2024-03-11T21:21:49.751 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 11 21:21:49 smithi146 bash[44452]: debug 2024-03-11T21:21:49.428+0000 7f5d99d2d700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.23:6814 osd.1 since back 2024-03-11T21:21:06.089672+0000 front 2024-03-11T21:21:06.089360+0000 (oldest deadline 2024-03-11T21:21:29.589425+0000) 2024-03-11T21:21:49.751 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 11 21:21:49 smithi146 bash[44452]: debug 2024-03-11T21:21:49.428+0000 7f5d99d2d700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.23:6822 osd.2 since back 2024-03-11T21:21:16.293816+0000 front 2024-03-11T21:21:16.293862+0000 (oldest deadline 2024-03-11T21:21:38.593400+0000) 2024-03-11T21:21:49.751 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 11 21:21:49 smithi146 bash[44452]: debug 2024-03-11T21:21:49.428+0000 7f5d99d2d700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.23:6830 osd.3 since back 2024-03-11T21:21:23.195467+0000 front 2024-03-11T21:21:23.195543+0000 (oldest deadline 2024-03-11T21:21:48.495326+0000) 2024-03-11T21:21:50.251 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 11 21:21:49 smithi146 bash[38675]: debug 2024-03-11T21:21:49.988+0000 7fb0d14ce700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-11T21:20:55.673190+0000 front 2024-03-11T21:20:55.672758+0000 (oldest deadline 2024-03-11T21:21:20.372648+0000) 2024-03-11T21:21:50.251 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 11 21:21:49 smithi146 bash[38675]: debug 2024-03-11T21:21:49.988+0000 7fb0d14ce700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.23:6814 osd.1 since back 2024-03-11T21:21:06.775540+0000 front 2024-03-11T21:21:06.775088+0000 (oldest deadline 2024-03-11T21:21:30.875096+0000) 2024-03-11T21:21:50.251 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 11 21:21:49 smithi146 bash[38675]: debug 2024-03-11T21:21:49.988+0000 7fb0d14ce700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.23:6822 osd.2 since back 2024-03-11T21:21:13.776564+0000 front 2024-03-11T21:21:13.776827+0000 (oldest deadline 2024-03-11T21:21:39.076442+0000) 2024-03-11T21:21:50.251 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 11 21:21:49 smithi146 bash[38675]: debug 2024-03-11T21:21:49.988+0000 7fb0d14ce700 -1 osd.6 499 heartbeat_check: no reply from 172.21.15.23:6830 osd.3 since back 2024-03-11T21:21:24.878677+0000 front 2024-03-11T21:21:24.878736+0000 (oldest deadline 2024-03-11T21:21:47.778538+0000) 2024-03-11T21:21:50.751 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 11 21:21:50 smithi146 bash[44452]: debug 2024-03-11T21:21:50.448+0000 7f5d99d2d700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-11T21:20:55.488114+0000 front 2024-03-11T21:20:55.488176+0000 (oldest deadline 2024-03-11T21:21:21.387970+0000) 2024-03-11T21:21:50.751 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 11 21:21:50 smithi146 bash[44452]: debug 2024-03-11T21:21:50.448+0000 7f5d99d2d700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.23:6814 osd.1 since back 2024-03-11T21:21:06.089672+0000 front 2024-03-11T21:21:06.089360+0000 (oldest deadline 2024-03-11T21:21:29.589425+0000) 2024-03-11T21:21:50.751 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 11 21:21:50 smithi146 bash[44452]: debug 2024-03-11T21:21:50.448+0000 7f5d99d2d700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.23:6822 osd.2 since back 2024-03-11T21:21:16.293816+0000 front 2024-03-11T21:21:16.293862+0000 (oldest deadline 2024-03-11T21:21:38.593400+0000) 2024-03-11T21:21:50.751 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 11 21:21:50 smithi146 bash[44452]: debug 2024-03-11T21:21:50.448+0000 7f5d99d2d700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.23:6830 osd.3 since back 2024-03-11T21:21:23.195467+0000 front 2024-03-11T21:21:23.195543+0000 (oldest deadline 2024-03-11T21:21:48.495326+0000) 2024-03-11T21:21:51.416 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 11 21:21:51 smithi146 bash[112739]: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd-6 2024-03-11T21:21:51.416 INFO:journalctl@ceph.osd.6.smithi146.stdout:Mar 11 21:21:51 smithi146 bash[112802]: Error response from daemon: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd-6 2024-03-11T21:21:51.751 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 11 21:21:51 smithi146 bash[44452]: debug 2024-03-11T21:21:51.452+0000 7f5d99d2d700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-11T21:20:55.488114+0000 front 2024-03-11T21:20:55.488176+0000 (oldest deadline 2024-03-11T21:21:21.387970+0000) 2024-03-11T21:21:51.751 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 11 21:21:51 smithi146 bash[44452]: debug 2024-03-11T21:21:51.452+0000 7f5d99d2d700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.23:6814 osd.1 since back 2024-03-11T21:21:06.089672+0000 front 2024-03-11T21:21:06.089360+0000 (oldest deadline 2024-03-11T21:21:29.589425+0000) 2024-03-11T21:21:51.751 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 11 21:21:51 smithi146 bash[44452]: debug 2024-03-11T21:21:51.452+0000 7f5d99d2d700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.23:6822 osd.2 since back 2024-03-11T21:21:16.293816+0000 front 2024-03-11T21:21:16.293862+0000 (oldest deadline 2024-03-11T21:21:38.593400+0000) 2024-03-11T21:21:51.751 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 11 21:21:51 smithi146 bash[44452]: debug 2024-03-11T21:21:51.452+0000 7f5d99d2d700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.23:6830 osd.3 since back 2024-03-11T21:21:23.195467+0000 front 2024-03-11T21:21:23.195543+0000 (oldest deadline 2024-03-11T21:21:48.495326+0000) 2024-03-11T21:21:52.501 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 11 21:21:52 smithi146 bash[44452]: debug 2024-03-11T21:21:52.500+0000 7f5d99d2d700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-11T21:20:55.488114+0000 front 2024-03-11T21:20:55.488176+0000 (oldest deadline 2024-03-11T21:21:21.387970+0000) 2024-03-11T21:21:52.501 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 11 21:21:52 smithi146 bash[44452]: debug 2024-03-11T21:21:52.500+0000 7f5d99d2d700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.23:6814 osd.1 since back 2024-03-11T21:21:06.089672+0000 front 2024-03-11T21:21:06.089360+0000 (oldest deadline 2024-03-11T21:21:29.589425+0000) 2024-03-11T21:21:52.501 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 11 21:21:52 smithi146 bash[44452]: debug 2024-03-11T21:21:52.500+0000 7f5d99d2d700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.23:6822 osd.2 since back 2024-03-11T21:21:16.293816+0000 front 2024-03-11T21:21:16.293862+0000 (oldest deadline 2024-03-11T21:21:38.593400+0000) 2024-03-11T21:21:52.501 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 11 21:21:52 smithi146 bash[44452]: debug 2024-03-11T21:21:52.500+0000 7f5d99d2d700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.23:6830 osd.3 since back 2024-03-11T21:21:23.195467+0000 front 2024-03-11T21:21:23.195543+0000 (oldest deadline 2024-03-11T21:21:48.495326+0000) 2024-03-11T21:21:53.319 DEBUG:teuthology.orchestra.run.smithi146:> sudo pkill -f 'journalctl -f -n 0 -u ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@osd.6.service' 2024-03-11T21:21:53.347 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T21:21:53.347 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-03-11T21:21:53.347 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-03-11T21:21:53.347 DEBUG:teuthology.orchestra.run.smithi146:> sudo systemctl stop ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@osd.7 2024-03-11T21:21:53.751 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 11 21:21:53 smithi146 systemd[1]: Stopping Ceph osd.7 for 67ba00f8-dfe7-11ee-95c8-87774f69a715... 2024-03-11T21:21:53.751 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 11 21:21:53 smithi146 bash[44452]: debug 2024-03-11T21:21:53.456+0000 7f5d99d2d700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-11T21:20:55.488114+0000 front 2024-03-11T21:20:55.488176+0000 (oldest deadline 2024-03-11T21:21:21.387970+0000) 2024-03-11T21:21:53.751 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 11 21:21:53 smithi146 bash[44452]: debug 2024-03-11T21:21:53.456+0000 7f5d99d2d700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.23:6814 osd.1 since back 2024-03-11T21:21:06.089672+0000 front 2024-03-11T21:21:06.089360+0000 (oldest deadline 2024-03-11T21:21:29.589425+0000) 2024-03-11T21:21:53.752 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 11 21:21:53 smithi146 bash[44452]: debug 2024-03-11T21:21:53.456+0000 7f5d99d2d700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.23:6822 osd.2 since back 2024-03-11T21:21:16.293816+0000 front 2024-03-11T21:21:16.293862+0000 (oldest deadline 2024-03-11T21:21:38.593400+0000) 2024-03-11T21:21:53.752 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 11 21:21:53 smithi146 bash[44452]: debug 2024-03-11T21:21:53.456+0000 7f5d99d2d700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.23:6830 osd.3 since back 2024-03-11T21:21:23.195467+0000 front 2024-03-11T21:21:23.195543+0000 (oldest deadline 2024-03-11T21:21:48.495326+0000) 2024-03-11T21:21:53.752 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 11 21:21:53 smithi146 bash[44452]: debug 2024-03-11T21:21:53.552+0000 7f5da1741700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-03-11T21:21:53.752 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 11 21:21:53 smithi146 bash[44452]: debug 2024-03-11T21:21:53.552+0000 7f5da1741700 -1 osd.7 499 *** Got signal Terminated *** 2024-03-11T21:21:53.752 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 11 21:21:53 smithi146 bash[44452]: debug 2024-03-11T21:21:53.552+0000 7f5da1741700 -1 osd.7 499 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T21:21:54.751 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 11 21:21:54 smithi146 bash[44452]: debug 2024-03-11T21:21:54.508+0000 7f5d99d2d700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-11T21:20:55.488114+0000 front 2024-03-11T21:20:55.488176+0000 (oldest deadline 2024-03-11T21:21:21.387970+0000) 2024-03-11T21:21:54.752 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 11 21:21:54 smithi146 bash[44452]: debug 2024-03-11T21:21:54.508+0000 7f5d99d2d700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.23:6814 osd.1 since back 2024-03-11T21:21:06.089672+0000 front 2024-03-11T21:21:06.089360+0000 (oldest deadline 2024-03-11T21:21:29.589425+0000) 2024-03-11T21:21:54.752 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 11 21:21:54 smithi146 bash[44452]: debug 2024-03-11T21:21:54.508+0000 7f5d99d2d700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.23:6822 osd.2 since back 2024-03-11T21:21:16.293816+0000 front 2024-03-11T21:21:16.293862+0000 (oldest deadline 2024-03-11T21:21:38.593400+0000) 2024-03-11T21:21:54.752 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 11 21:21:54 smithi146 bash[44452]: debug 2024-03-11T21:21:54.508+0000 7f5d99d2d700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.23:6830 osd.3 since back 2024-03-11T21:21:23.195467+0000 front 2024-03-11T21:21:23.195543+0000 (oldest deadline 2024-03-11T21:21:48.495326+0000) 2024-03-11T21:21:55.751 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 11 21:21:55 smithi146 bash[44452]: debug 2024-03-11T21:21:55.488+0000 7f5d99d2d700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-11T21:20:55.488114+0000 front 2024-03-11T21:20:55.488176+0000 (oldest deadline 2024-03-11T21:21:21.387970+0000) 2024-03-11T21:21:55.751 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 11 21:21:55 smithi146 bash[44452]: debug 2024-03-11T21:21:55.488+0000 7f5d99d2d700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.23:6814 osd.1 since back 2024-03-11T21:21:06.089672+0000 front 2024-03-11T21:21:06.089360+0000 (oldest deadline 2024-03-11T21:21:29.589425+0000) 2024-03-11T21:21:55.751 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 11 21:21:55 smithi146 bash[44452]: debug 2024-03-11T21:21:55.488+0000 7f5d99d2d700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.23:6822 osd.2 since back 2024-03-11T21:21:16.293816+0000 front 2024-03-11T21:21:16.293862+0000 (oldest deadline 2024-03-11T21:21:38.593400+0000) 2024-03-11T21:21:55.751 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 11 21:21:55 smithi146 bash[44452]: debug 2024-03-11T21:21:55.488+0000 7f5d99d2d700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.23:6830 osd.3 since back 2024-03-11T21:21:23.195467+0000 front 2024-03-11T21:21:23.195543+0000 (oldest deadline 2024-03-11T21:21:48.495326+0000) 2024-03-11T21:21:56.751 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 11 21:21:56 smithi146 bash[44452]: debug 2024-03-11T21:21:56.484+0000 7f5d99d2d700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-11T21:20:55.488114+0000 front 2024-03-11T21:20:55.488176+0000 (oldest deadline 2024-03-11T21:21:21.387970+0000) 2024-03-11T21:21:56.751 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 11 21:21:56 smithi146 bash[44452]: debug 2024-03-11T21:21:56.484+0000 7f5d99d2d700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.23:6814 osd.1 since back 2024-03-11T21:21:06.089672+0000 front 2024-03-11T21:21:06.089360+0000 (oldest deadline 2024-03-11T21:21:29.589425+0000) 2024-03-11T21:21:56.751 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 11 21:21:56 smithi146 bash[44452]: debug 2024-03-11T21:21:56.484+0000 7f5d99d2d700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.23:6822 osd.2 since back 2024-03-11T21:21:16.293816+0000 front 2024-03-11T21:21:16.293862+0000 (oldest deadline 2024-03-11T21:21:38.593400+0000) 2024-03-11T21:21:56.751 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 11 21:21:56 smithi146 bash[44452]: debug 2024-03-11T21:21:56.484+0000 7f5d99d2d700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.23:6830 osd.3 since back 2024-03-11T21:21:23.195467+0000 front 2024-03-11T21:21:23.195543+0000 (oldest deadline 2024-03-11T21:21:48.495326+0000) 2024-03-11T21:21:56.751 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 11 21:21:56 smithi146 bash[44452]: debug 2024-03-11T21:21:56.484+0000 7f5d99d2d700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.146:6804 osd.4 since back 2024-03-11T21:21:31.996738+0000 front 2024-03-11T21:21:31.996876+0000 (oldest deadline 2024-03-11T21:21:56.096476+0000) 2024-03-11T21:21:58.001 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 11 21:21:57 smithi146 bash[44452]: debug 2024-03-11T21:21:57.524+0000 7f5d99d2d700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-11T21:20:55.488114+0000 front 2024-03-11T21:20:55.488176+0000 (oldest deadline 2024-03-11T21:21:21.387970+0000) 2024-03-11T21:21:58.001 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 11 21:21:57 smithi146 bash[44452]: debug 2024-03-11T21:21:57.524+0000 7f5d99d2d700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.23:6814 osd.1 since back 2024-03-11T21:21:06.089672+0000 front 2024-03-11T21:21:06.089360+0000 (oldest deadline 2024-03-11T21:21:29.589425+0000) 2024-03-11T21:21:58.001 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 11 21:21:57 smithi146 bash[44452]: debug 2024-03-11T21:21:57.524+0000 7f5d99d2d700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.23:6822 osd.2 since back 2024-03-11T21:21:16.293816+0000 front 2024-03-11T21:21:16.293862+0000 (oldest deadline 2024-03-11T21:21:38.593400+0000) 2024-03-11T21:21:58.001 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 11 21:21:57 smithi146 bash[44452]: debug 2024-03-11T21:21:57.524+0000 7f5d99d2d700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.23:6830 osd.3 since back 2024-03-11T21:21:23.195467+0000 front 2024-03-11T21:21:23.195543+0000 (oldest deadline 2024-03-11T21:21:48.495326+0000) 2024-03-11T21:21:58.001 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 11 21:21:57 smithi146 bash[44452]: debug 2024-03-11T21:21:57.524+0000 7f5d99d2d700 -1 osd.7 499 heartbeat_check: no reply from 172.21.15.146:6804 osd.4 since back 2024-03-11T21:21:31.996738+0000 front 2024-03-11T21:21:31.996876+0000 (oldest deadline 2024-03-11T21:21:56.096476+0000) 2024-03-11T21:22:00.253 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 11 21:21:59 smithi146 bash[112937]: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd-7 2024-03-11T21:22:00.253 INFO:journalctl@ceph.osd.7.smithi146.stdout:Mar 11 21:22:00 smithi146 bash[113003]: Error response from daemon: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-osd-7 2024-03-11T21:22:01.886 DEBUG:teuthology.orchestra.run.smithi146:> sudo pkill -f 'journalctl -f -n 0 -u ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@osd.7.service' 2024-03-11T21:22:01.913 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T21:22:01.914 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-03-11T21:22:01.914 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopping rgw.foo.a... 2024-03-11T21:22:01.914 DEBUG:teuthology.orchestra.run.smithi023:> sudo systemctl stop ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@rgw.foo.a 2024-03-11T21:22:02.273 INFO:journalctl@ceph.rgw.foo.a.smithi023.stdout:Mar 11 21:22:01 smithi023 systemd[1]: Stopping Ceph rgw.foo.a for 67ba00f8-dfe7-11ee-95c8-87774f69a715... 2024-03-11T21:22:02.273 INFO:journalctl@ceph.rgw.foo.a.smithi023.stdout:Mar 11 21:22:02 smithi023 bash[53100]: debug 2024-03-11T21:22:02.069+0000 7f9fd4841700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/radosgw -n client.rgw.foo.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-03-11T21:22:02.273 INFO:journalctl@ceph.rgw.foo.a.smithi023.stdout:Mar 11 21:22:02 smithi023 bash[53100]: debug 2024-03-11T21:22:02.069+0000 7f9fe30b9ac0 -1 shutting down 2024-03-11T21:22:09.751 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Mar 11 21:22:09 smithi146 bash[52406]: ts=2024-03-11T21:22:09.509Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=node msg="Unable to refresh target groups" err="Get \"http://172.21.15.23:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.23:8765: connect: connection refused" 2024-03-11T21:22:09.751 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Mar 11 21:22:09 smithi146 bash[52406]: ts=2024-03-11T21:22:09.509Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=nvmeof msg="Unable to refresh target groups" err="Get \"http://172.21.15.23:8765/sd/prometheus/sd-config?service=nvmeof\": dial tcp 172.21.15.23:8765: connect: connection refused" 2024-03-11T21:22:09.752 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Mar 11 21:22:09 smithi146 bash[52406]: ts=2024-03-11T21:22:09.517Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.23:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.23:8765: connect: connection refused" 2024-03-11T21:22:09.752 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Mar 11 21:22:09 smithi146 bash[52406]: ts=2024-03-11T21:22:09.521Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.23:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.23:8765: connect: connection refused" 2024-03-11T21:22:09.752 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Mar 11 21:22:09 smithi146 bash[52406]: ts=2024-03-11T21:22:09.521Z caller=refresh.go:99 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://172.21.15.23:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.23:8765: connect: connection refused" 2024-03-11T21:22:12.773 INFO:journalctl@ceph.rgw.foo.a.smithi023.stdout:Mar 11 21:22:12 smithi023 bash[170031]: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-rgw-foo-a 2024-03-11T21:22:12.819 DEBUG:teuthology.orchestra.run.smithi023:> sudo pkill -f 'journalctl -f -n 0 -u ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@rgw.foo.a.service' 2024-03-11T21:22:12.844 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T21:22:12.845 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopped rgw.foo.a 2024-03-11T21:22:12.845 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-03-11T21:22:12.845 DEBUG:teuthology.orchestra.run.smithi146:> sudo systemctl stop ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@prometheus.a 2024-03-11T21:22:13.223 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Mar 11 21:22:12 smithi146 systemd[1]: Stopping Ceph prometheus.a for 67ba00f8-dfe7-11ee-95c8-87774f69a715... 2024-03-11T21:22:13.224 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Mar 11 21:22:13 smithi146 bash[52406]: ts=2024-03-11T21:22:13.006Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-11T21:22:13.224 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Mar 11 21:22:13 smithi146 bash[52406]: ts=2024-03-11T21:22:13.007Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-03-11T21:22:13.224 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Mar 11 21:22:13 smithi146 bash[52406]: ts=2024-03-11T21:22:13.007Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-03-11T21:22:13.224 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Mar 11 21:22:13 smithi146 bash[52406]: ts=2024-03-11T21:22:13.007Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-11T21:22:13.224 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Mar 11 21:22:13 smithi146 bash[52406]: ts=2024-03-11T21:22:13.007Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-03-11T21:22:13.224 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Mar 11 21:22:13 smithi146 bash[52406]: ts=2024-03-11T21:22:13.007Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-03-11T21:22:13.224 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Mar 11 21:22:13 smithi146 bash[52406]: ts=2024-03-11T21:22:13.007Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-03-11T21:22:13.224 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Mar 11 21:22:13 smithi146 bash[52406]: ts=2024-03-11T21:22:13.007Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-03-11T21:22:13.224 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Mar 11 21:22:13 smithi146 bash[52406]: ts=2024-03-11T21:22:13.007Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-03-11T21:22:13.224 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Mar 11 21:22:13 smithi146 bash[52406]: ts=2024-03-11T21:22:13.008Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-03-11T21:22:13.224 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Mar 11 21:22:13 smithi146 bash[52406]: ts=2024-03-11T21:22:13.008Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-03-11T21:22:13.224 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Mar 11 21:22:13 smithi146 bash[52406]: ts=2024-03-11T21:22:13.008Z caller=main.go:1147 level=info msg="See you next time!" 2024-03-11T21:22:13.804 DEBUG:teuthology.orchestra.run.smithi146:> sudo pkill -f 'journalctl -f -n 0 -u ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@prometheus.a.service' 2024-03-11T21:22:13.825 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T21:22:13.825 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-03-11T21:22:13.826 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 --force --keep-logs 2024-03-11T21:22:13.997 INFO:teuthology.orchestra.run.smithi023.stdout:Deleting cluster with fsid: 67ba00f8-dfe7-11ee-95c8-87774f69a715 2024-03-11T21:22:18.872 INFO:journalctl@ceph.alertmanager.a.smithi023.stdout:Mar 11 21:22:18 smithi023 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:22:18.873 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 11 21:22:18 smithi023 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:22:19.129 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 11 21:22:19 smithi023 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:22:19.130 INFO:journalctl@ceph.alertmanager.a.smithi023.stdout:Mar 11 21:22:19 smithi023 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:22:19.385 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 11 21:22:19 smithi023 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:22:19.386 INFO:journalctl@ceph.alertmanager.a.smithi023.stdout:Mar 11 21:22:19 smithi023 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:22:19.691 INFO:journalctl@ceph.alertmanager.a.smithi023.stdout:Mar 11 21:22:19 smithi023 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:22:19.692 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 11 21:22:19 smithi023 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:22:20.022 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 11 21:22:19 smithi023 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:22:20.023 INFO:journalctl@ceph.alertmanager.a.smithi023.stdout:Mar 11 21:22:19 smithi023 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:22:20.348 INFO:journalctl@ceph.alertmanager.a.smithi023.stdout:Mar 11 21:22:20 smithi023 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:22:20.348 INFO:journalctl@ceph.alertmanager.a.smithi023.stdout:Mar 11 21:22:20 smithi023 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:22:20.348 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 11 21:22:20 smithi023 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:22:20.348 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 11 21:22:20 smithi023 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:22:20.646 INFO:journalctl@ceph.alertmanager.a.smithi023.stdout:Mar 11 21:22:20 smithi023 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:22:20.646 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 11 21:22:20 smithi023 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:22:21.013 INFO:journalctl@ceph.alertmanager.a.smithi023.stdout:Mar 11 21:22:20 smithi023 systemd[1]: Stopping Ceph alertmanager.a for 67ba00f8-dfe7-11ee-95c8-87774f69a715... 2024-03-11T21:22:21.013 INFO:journalctl@ceph.alertmanager.a.smithi023.stdout:Mar 11 21:22:20 smithi023 bash[62486]: ts=2024-03-11T21:22:20.785Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-03-11T21:22:21.740 INFO:journalctl@ceph.alertmanager.a.smithi023.stdout:Mar 11 21:22:21 smithi023 bash[170436]: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-alertmanager-a 2024-03-11T21:22:21.740 INFO:journalctl@ceph.alertmanager.a.smithi023.stdout:Mar 11 21:22:21 smithi023 bash[170495]: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-alertmanager-a 2024-03-11T21:22:21.740 INFO:journalctl@ceph.alertmanager.a.smithi023.stdout:Mar 11 21:22:21 smithi023 systemd[1]: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-03-11T21:22:21.740 INFO:journalctl@ceph.alertmanager.a.smithi023.stdout:Mar 11 21:22:21 smithi023 systemd[1]: Stopped Ceph alertmanager.a for 67ba00f8-dfe7-11ee-95c8-87774f69a715. 2024-03-11T21:22:21.995 INFO:journalctl@ceph.alertmanager.a.smithi023.stdout:Mar 11 21:22:21 smithi023 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:22:21.995 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 11 21:22:21 smithi023 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:22:21.995 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 11 21:22:21 smithi023 systemd[1]: Stopping Ceph node-exporter.a for 67ba00f8-dfe7-11ee-95c8-87774f69a715... 2024-03-11T21:22:22.772 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 11 21:22:22 smithi023 bash[170563]: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-node-exporter-a 2024-03-11T21:22:22.773 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 11 21:22:22 smithi023 systemd[1]: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-03-11T21:22:22.773 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 11 21:22:22 smithi023 bash[170625]: Error response from daemon: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-node-exporter-a 2024-03-11T21:22:22.773 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 11 21:22:22 smithi023 systemd[1]: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@node-exporter.a.service: Failed with result 'exit-code'. 2024-03-11T21:22:22.773 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 11 21:22:22 smithi023 systemd[1]: Stopped Ceph node-exporter.a for 67ba00f8-dfe7-11ee-95c8-87774f69a715. 2024-03-11T21:22:23.142 INFO:journalctl@ceph.node-exporter.a.smithi023.stdout:Mar 11 21:22:22 smithi023 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:22:23.440 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 --force --keep-logs 2024-03-11T21:22:23.606 INFO:teuthology.orchestra.run.smithi146.stdout:Deleting cluster with fsid: 67ba00f8-dfe7-11ee-95c8-87774f69a715 2024-03-11T21:22:28.500 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:22:28 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:22:28.501 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Mar 11 21:22:28 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:22:28.501 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 21:22:28 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:22:28.806 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:22:28 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:22:28.806 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Mar 11 21:22:28 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:22:28.807 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 21:22:28 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:22:29.078 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Mar 11 21:22:29 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:22:29.079 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 21:22:29 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:22:29.079 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:22:29 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:22:29.354 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Mar 11 21:22:29 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:22:29.354 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 21:22:29 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:22:29.355 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:22:29 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:22:29.669 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:22:29 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:22:29.669 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Mar 11 21:22:29 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:22:29.670 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 21:22:29 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:22:29.920 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Mar 11 21:22:29 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:22:29.921 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 21:22:29 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:22:29.921 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:22:29 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:22:30.250 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:22:29 smithi146 systemd[1]: Stopping Ceph iscsi.iscsi.a for 67ba00f8-dfe7-11ee-95c8-87774f69a715... 2024-03-11T21:22:30.251 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:22:30 smithi146 bash[49028]: debug Shutdown received 2024-03-11T21:22:40.750 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:22:40 smithi146 bash[113470]: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-iscsi-iscsi-a 2024-03-11T21:22:41.066 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Mar 11 21:22:41 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:22:41.067 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 21:22:41 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:22:41.067 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:22:40 smithi146 systemd[1]: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@iscsi.iscsi.a.service: Main process exited, code=exited, status=137/n/a 2024-03-11T21:22:41.067 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:22:40 smithi146 bash[113536]: Error response from daemon: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-iscsi-iscsi-a 2024-03-11T21:22:41.067 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:22:40 smithi146 systemd[1]: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@iscsi.iscsi.a.service: Failed with result 'exit-code'. 2024-03-11T21:22:41.068 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:22:40 smithi146 systemd[1]: Stopped Ceph iscsi.iscsi.a for 67ba00f8-dfe7-11ee-95c8-87774f69a715. 2024-03-11T21:22:41.068 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Mar 11 21:22:41 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:22:42.379 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 21:22:42 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:22:42.379 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Mar 11 21:22:42 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:22:42.639 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Mar 11 21:22:42 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:22:42.639 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Mar 11 21:22:42 smithi146 systemd[1]: Stopping Ceph node-exporter.b for 67ba00f8-dfe7-11ee-95c8-87774f69a715... 2024-03-11T21:22:42.640 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 21:22:42 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:22:43.451 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Mar 11 21:22:43 smithi146 bash[113739]: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-node-exporter-b 2024-03-11T21:22:43.451 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Mar 11 21:22:43 smithi146 bash[113800]: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-node-exporter-b 2024-03-11T21:22:43.451 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Mar 11 21:22:43 smithi146 systemd[1]: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-03-11T21:22:43.451 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Mar 11 21:22:43 smithi146 systemd[1]: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@node-exporter.b.service: Failed with result 'exit-code'. 2024-03-11T21:22:43.452 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Mar 11 21:22:43 smithi146 systemd[1]: Stopped Ceph node-exporter.b for 67ba00f8-dfe7-11ee-95c8-87774f69a715. 2024-03-11T21:22:43.727 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Mar 11 21:22:43 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:22:43.728 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 21:22:43 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:22:43.728 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 21:22:43 smithi146 systemd[1]: Stopping Ceph grafana.a for 67ba00f8-dfe7-11ee-95c8-87774f69a715... 2024-03-11T21:22:44.500 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 21:22:44 smithi146 bash[113869]: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-grafana-a 2024-03-11T21:22:44.500 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 21:22:44 smithi146 systemd[1]: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@grafana.a.service: Main process exited, code=exited, status=143/n/a 2024-03-11T21:22:44.500 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 21:22:44 smithi146 bash[113931]: Error response from daemon: No such container: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715-grafana-a 2024-03-11T21:22:44.500 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 21:22:44 smithi146 systemd[1]: ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-03-11T21:22:44.501 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 21:22:44 smithi146 systemd[1]: Stopped Ceph grafana.a for 67ba00f8-dfe7-11ee-95c8-87774f69a715. 2024-03-11T21:22:44.824 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Mar 11 21:22:44 smithi146 systemd[1]: /etc/systemd/system/ceph-67ba00f8-dfe7-11ee-95c8-87774f69a715@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-03-11T21:22:45.119 DEBUG:teuthology.orchestra.run.smithi023:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-11T21:22:45.134 DEBUG:teuthology.orchestra.run.smithi146:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-11T21:22:45.147 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-11T21:22:45.149 DEBUG:teuthology.misc:Transferring archived files from smithi023:/var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591745/remote/smithi023/crash 2024-03-11T21:22:45.150 DEBUG:teuthology.orchestra.run.smithi023:> sudo tar c -f - -C /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/crash -- . 2024-03-11T21:22:45.202 INFO:teuthology.orchestra.run.smithi023.stderr:tar: /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/crash: Cannot open: No such file or directory 2024-03-11T21:22:45.202 INFO:teuthology.orchestra.run.smithi023.stderr:tar: Error is not recoverable: exiting now 2024-03-11T21:22:45.204 DEBUG:teuthology.misc:Transferring archived files from smithi146:/var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591745/remote/smithi146/crash 2024-03-11T21:22:45.205 DEBUG:teuthology.orchestra.run.smithi146:> sudo tar c -f - -C /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/crash -- . 2024-03-11T21:22:45.227 INFO:teuthology.orchestra.run.smithi146.stderr:tar: /var/lib/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/crash: Cannot open: No such file or directory 2024-03-11T21:22:45.227 INFO:teuthology.orchestra.run.smithi146.stderr:tar: Error is not recoverable: exiting now 2024-03-11T21:22:45.228 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-11T21:22:45.228 DEBUG:teuthology.orchestra.run.smithi023:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-03-11T21:22:45.318 INFO:teuthology.orchestra.run.smithi023.stdout:2024-03-11T21:00:00.000279+0000 mon.a (mon.0) 1176 : cluster 3 [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-03-11T21:22:45.319 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-03-11T21:22:45.319 DEBUG:teuthology.orchestra.run.smithi023:> sudo egrep '\[SEC\]' /var/log/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-03-11T21:22:45.333 DEBUG:teuthology.orchestra.run.smithi023:> sudo egrep '\[ERR\]' /var/log/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-03-11T21:22:45.388 DEBUG:teuthology.orchestra.run.smithi023:> sudo egrep '\[WRN\]' /var/log/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-03-11T21:22:45.443 INFO:teuthology.orchestra.run.smithi023.stdout:2024-03-11T21:00:00.000279+0000 mon.a (mon.0) 1176 : cluster 3 [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-03-11T21:22:45.444 INFO:tasks.cephadm:Compressing logs... 2024-03-11T21:22:45.444 DEBUG:teuthology.orchestra.run.smithi023:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-11T21:22:45.489 DEBUG:teuthology.orchestra.run.smithi146:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-11T21:22:45.497 INFO:teuthology.orchestra.run.smithi023.stderr:find: '/var/log/rbd-target-api': No such file or directory 2024-03-11T21:22:45.502 INFO:teuthology.orchestra.run.smithi146.stderr:find: '/var/log/rbd-target-api': No such file or directory 2024-03-11T21:22:45.503 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-11T21:22:45.508 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/ceph-mgr.y.log 2024-03-11T21:22:45.508 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/ceph-osd.0.log 2024-03-11T21:22:45.508 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/ceph-osd.2.log 2024-03-11T21:22:45.509 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/ceph-mgr.y.log: /var/log/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/ceph-mon.a.log 2024-03-11T21:22:45.510 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-11T21:22:45.511 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/objectstore_tool.6.log 2024-03-11T21:22:45.511 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/ceph.cephadm.log 2024-03-11T21:22:45.511 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/ceph-mon.a.log: /var/log/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/objectstore_tool.6.log: gzip -5 --verbose -- /var/log/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/ceph-osd.3.log 2024-03-11T21:22:45.511 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/ceph-osd.1.log 2024-03-11T21:22:45.511 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/objectstore_tool.7.log 2024-03-11T21:22:45.511 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/ceph.log 2024-03-11T21:22:45.513 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/objectstore_tool.7.log: gzip -5 --verbose -- /var/log/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/ceph-volume.log 2024-03-11T21:22:45.513 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/ceph-mon.c.log 2024-03-11T21:22:45.513 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/ceph.audit.log 2024-03-11T21:22:45.513 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/ceph-client.rgw.foo.a.log 2024-03-11T21:22:45.513 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/ceph.audit.log: /var/log/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/ceph-client.rgw.foo.a.log: 57.9% -- replaced with /var/log/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/ceph-client.rgw.foo.a.log.gz 2024-03-11T21:22:45.516 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/ceph-osd.7.log 2024-03-11T21:22:45.517 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/ceph-osd.6.log 2024-03-11T21:22:45.517 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/ceph.cephadm.log 2024-03-11T21:22:45.518 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/ceph-osd.5.log 2024-03-11T21:22:45.518 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/objectstore_tool.7.log 2024-03-11T21:22:45.519 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/ceph.cephadm.log: /var/log/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/tcmu-runner.log 2024-03-11T21:22:45.519 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/objectstore_tool.7.log: gzip -5 --verbose -- /var/log/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/ceph-mgr.x.log 2024-03-11T21:22:45.520 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/tcmu-runner.log: gzip -5 --verbose -- /var/log/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/ceph.log 2024-03-11T21:22:45.520 INFO:teuthology.orchestra.run.smithi146.stderr: 67.6% -- replaced with /var/log/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/tcmu-runner.log.gz 2024-03-11T21:22:45.520 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/ceph-volume.log 2024-03-11T21:22:45.521 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/ceph.audit.log 2024-03-11T21:22:45.521 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/ceph-mon.b.log 2024-03-11T21:22:45.521 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/ceph-osd.4.log 2024-03-11T21:22:45.527 INFO:teuthology.orchestra.run.smithi023.stderr: 91.8% -- replaced with /var/log/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/ceph.log.gz 2024-03-11T21:22:45.528 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/ceph-mon.b.log: /var/log/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/ceph-osd.4.log: 92.9% -- replaced with /var/log/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/ceph-mgr.x.log.gz 2024-03-11T21:22:45.549 INFO:teuthology.orchestra.run.smithi023.stderr: 89.1% -- replaced with /var/log/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/ceph.cephadm.log.gz 2024-03-11T21:22:45.557 INFO:teuthology.orchestra.run.smithi023.stderr: 94.3% -- replaced with /var/log/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/objectstore_tool.6.log.gz 2024-03-11T21:22:45.583 INFO:teuthology.orchestra.run.smithi146.stderr: 81.7% -- replaced with /var/log/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/ceph.cephadm.log.gz 2024-03-11T21:22:45.638 INFO:teuthology.orchestra.run.smithi023.stderr: 93.8% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-11T21:22:45.720 INFO:teuthology.orchestra.run.smithi023.stderr: 94.0% -- replaced with /var/log/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/ceph.audit.log.gz 2024-03-11T21:22:45.728 INFO:teuthology.orchestra.run.smithi146.stderr: 91.2% -- replaced with /var/log/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/ceph-volume.log.gz 2024-03-11T21:22:45.736 INFO:teuthology.orchestra.run.smithi146.stderr: 90.2% -- replaced with /var/log/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/ceph.audit.log.gz 2024-03-11T21:22:45.748 INFO:teuthology.orchestra.run.smithi146.stderr: 87.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-11T21:22:45.785 INFO:teuthology.orchestra.run.smithi146.stderr: 94.5% -- replaced with /var/log/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/objectstore_tool.7.log.gz 2024-03-11T21:22:45.799 INFO:teuthology.orchestra.run.smithi146.stderr: 85.3% -- replaced with /var/log/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/ceph.log.gz 2024-03-11T21:22:45.815 INFO:teuthology.orchestra.run.smithi023.stderr: 91.1% -- replaced with /var/log/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/ceph-volume.log.gz 2024-03-11T21:22:46.540 INFO:teuthology.orchestra.run.smithi023.stderr: 89.9% -- replaced with /var/log/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/ceph-mgr.y.log.gz 2024-03-11T21:22:47.557 INFO:teuthology.orchestra.run.smithi023.stderr: 94.8% -- replaced with /var/log/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/objectstore_tool.7.log.gz 2024-03-11T21:22:47.718 INFO:teuthology.orchestra.run.smithi146.stderr: 91.6% -- replaced with /var/log/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/ceph-mon.b.log.gz 2024-03-11T21:22:47.904 INFO:teuthology.orchestra.run.smithi023.stderr: 91.7% -- replaced with /var/log/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/ceph-mon.c.log.gz 2024-03-11T21:22:50.947 INFO:teuthology.orchestra.run.smithi023.stderr: 90.5% -- replaced with /var/log/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/ceph-mon.a.log.gz 2024-03-11T21:24:38.031 INFO:teuthology.orchestra.run.smithi146.stderr: 94.5% -- replaced with /var/log/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/ceph-osd.4.log.gz 2024-03-11T21:25:50.099 INFO:teuthology.orchestra.run.smithi023.stderr: 93.7% -- replaced with /var/log/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/ceph-osd.2.log.gz 2024-03-11T21:26:39.451 INFO:teuthology.orchestra.run.smithi146.stderr: 94.0% -- replaced with /var/log/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/ceph-osd.7.log.gz 2024-03-11T21:26:51.014 INFO:teuthology.orchestra.run.smithi146.stderr: 93.7% -- replaced with /var/log/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/ceph-osd.5.log.gz 2024-03-11T21:26:58.963 INFO:teuthology.orchestra.run.smithi146.stderr: 93.8% -- replaced with /var/log/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/ceph-osd.6.log.gz 2024-03-11T21:26:58.965 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-11T21:26:58.965 INFO:teuthology.orchestra.run.smithi146.stderr:real 4m13.472s 2024-03-11T21:26:58.965 INFO:teuthology.orchestra.run.smithi146.stderr:user 5m56.800s 2024-03-11T21:26:58.965 INFO:teuthology.orchestra.run.smithi146.stderr:sys 0m15.459s 2024-03-11T21:27:12.139 INFO:teuthology.orchestra.run.smithi023.stderr: 93.5% -- replaced with /var/log/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/ceph-osd.0.log.gz 2024-03-11T21:28:09.841 INFO:teuthology.orchestra.run.smithi023.stderr: 92.8% -- replaced with /var/log/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/ceph-osd.1.log.gz 2024-03-11T21:28:39.439 INFO:teuthology.orchestra.run.smithi023.stderr: 92.7% -- replaced with /var/log/ceph/67ba00f8-dfe7-11ee-95c8-87774f69a715/ceph-osd.3.log.gz 2024-03-11T21:28:39.440 INFO:teuthology.orchestra.run.smithi023.stderr: 2024-03-11T21:28:39.440 INFO:teuthology.orchestra.run.smithi023.stderr:real 5m53.951s 2024-03-11T21:28:39.440 INFO:teuthology.orchestra.run.smithi023.stderr:user 8m11.006s 2024-03-11T21:28:39.441 INFO:teuthology.orchestra.run.smithi023.stderr:sys 0m20.032s 2024-03-11T21:28:39.442 INFO:tasks.cephadm:Archiving logs... 2024-03-11T21:28:39.442 DEBUG:teuthology.misc:Transferring archived files from smithi023:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591745/remote/smithi023/log 2024-03-11T21:28:39.443 DEBUG:teuthology.orchestra.run.smithi023:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-11T21:29:17.329 DEBUG:teuthology.misc:Transferring archived files from smithi146:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591745/remote/smithi146/log 2024-03-11T21:29:17.330 DEBUG:teuthology.orchestra.run.smithi146:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-11T21:29:46.845 INFO:tasks.cephadm:Removing cluster... 2024-03-11T21:29:46.846 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 --force 2024-03-11T21:29:47.028 INFO:teuthology.orchestra.run.smithi023.stdout:Deleting cluster with fsid: 67ba00f8-dfe7-11ee-95c8-87774f69a715 2024-03-11T21:29:48.999 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 67ba00f8-dfe7-11ee-95c8-87774f69a715 --force 2024-03-11T21:29:49.226 INFO:teuthology.orchestra.run.smithi146.stdout:Deleting cluster with fsid: 67ba00f8-dfe7-11ee-95c8-87774f69a715 2024-03-11T21:29:51.055 INFO:tasks.cephadm:Removing cephadm ... 2024-03-11T21:29:51.055 DEBUG:teuthology.orchestra.run.smithi023:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-11T21:29:51.079 DEBUG:teuthology.orchestra.run.smithi146:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-11T21:29:51.083 INFO:tasks.cephadm:Teardown complete 2024-03-11T21:29:51.084 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-11T21:29:51.095 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-11T21:29:51.096 DEBUG:teuthology.orchestra.run.smithi023:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-11T21:29:51.123 DEBUG:teuthology.orchestra.run.smithi146:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-11T21:29:51.200 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-11T21:29:51.209 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-11T21:29:51.209 DEBUG:teuthology.orchestra.run.smithi023:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-11T21:29:51.211 DEBUG:teuthology.orchestra.run.smithi146:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-11T21:29:51.255 INFO:teuthology.orchestra.run.smithi023.stdout: remote refid st t when poll reach delay offset jitter 2024-03-11T21:29:51.255 INFO:teuthology.orchestra.run.smithi023.stdout:============================================================================== 2024-03-11T21:29:51.255 INFO:teuthology.orchestra.run.smithi023.stdout:*hv01.front.sepi 44.4.53.2 2 u 66 128 377 0.096 -1.331 1.293 2024-03-11T21:29:51.255 INFO:teuthology.orchestra.run.smithi023.stdout:+hv02.front.sepi 63.231.80.2 3 u 77 128 377 0.124 +2.473 0.473 2024-03-11T21:29:51.255 INFO:teuthology.orchestra.run.smithi023.stdout:+hv03.front.sepi 74.6.168.72 3 u 77 128 377 0.087 +0.465 0.154 2024-03-11T21:29:51.255 INFO:teuthology.orchestra.run.smithi023.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2024-03-11T21:29:51.260 INFO:teuthology.orchestra.run.smithi146.stdout: remote refid st t when poll reach delay offset jitter 2024-03-11T21:29:51.260 INFO:teuthology.orchestra.run.smithi146.stdout:============================================================================== 2024-03-11T21:29:51.260 INFO:teuthology.orchestra.run.smithi146.stdout:*hv01.front.sepi 44.4.53.2 2 u 68 128 377 0.095 -5.438 0.906 2024-03-11T21:29:51.260 INFO:teuthology.orchestra.run.smithi146.stdout:+hv02.front.sepi 63.231.80.2 3 u 71 128 377 0.100 +0.386 0.469 2024-03-11T21:29:51.260 INFO:teuthology.orchestra.run.smithi146.stdout:+hv03.front.sepi 74.6.168.72 3 u 75 128 377 0.091 -1.034 0.645 2024-03-11T21:29:51.260 INFO:teuthology.orchestra.run.smithi146.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2024-03-11T21:29:51.261 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-11T21:29:51.271 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-11T21:29:51.272 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-11T21:29:51.284 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-11T21:29:51.294 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-11T21:29:51.303 INFO:teuthology.task.internal:Duration was 3592.101498 seconds 2024-03-11T21:29:51.303 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-11T21:29:51.312 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-11T21:29:51.312 DEBUG:teuthology.orchestra.run.smithi023:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-11T21:29:51.315 DEBUG:teuthology.orchestra.run.smithi146:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-11T21:29:51.420 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-11T21:29:51.421 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi023.front.sepia.ceph.com 2024-03-11T21:29:51.421 DEBUG:teuthology.orchestra.run.smithi023:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-11T21:29:51.444 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi146.front.sepia.ceph.com 2024-03-11T21:29:51.444 DEBUG:teuthology.orchestra.run.smithi146:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-11T21:29:51.463 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-11T21:29:51.464 DEBUG:teuthology.orchestra.run.smithi023:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-11T21:29:51.486 DEBUG:teuthology.orchestra.run.smithi146:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-11T21:29:51.650 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-11T21:29:51.650 DEBUG:teuthology.orchestra.run.smithi023:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-11T21:29:51.652 DEBUG:teuthology.orchestra.run.smithi146:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-11T21:29:52.114 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-11T21:29:52.126 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-11T21:29:52.126 DEBUG:teuthology.orchestra.run.smithi023:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-11T21:29:52.171 DEBUG:teuthology.orchestra.run.smithi146:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-11T21:29:52.200 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-11T21:29:52.210 DEBUG:teuthology.orchestra.run.smithi023:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-11T21:29:52.215 DEBUG:teuthology.orchestra.run.smithi146:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-11T21:29:52.236 INFO:teuthology.orchestra.run.smithi023.stdout:kernel.core_pattern = core 2024-03-11T21:29:52.258 INFO:teuthology.orchestra.run.smithi146.stdout:kernel.core_pattern = core 2024-03-11T21:29:52.283 DEBUG:teuthology.orchestra.run.smithi023:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-11T21:29:52.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:52.303 DEBUG:teuthology.orchestra.run.smithi146:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-11T21:29:52.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:52.328 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-11T21:29:52.339 INFO:teuthology.task.internal:Transferring archived files... 2024-03-11T21:29:52.340 DEBUG:teuthology.misc:Transferring archived files from smithi023:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591745/remote/smithi023 2024-03-11T21:29:52.340 DEBUG:teuthology.orchestra.run.smithi023:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-11T21:29:52.385 DEBUG:teuthology.misc:Transferring archived files from smithi146:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591745/remote/smithi146 2024-03-11T21:29:52.386 DEBUG:teuthology.orchestra.run.smithi146:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-11T21:29:52.419 INFO:teuthology.task.internal:Removing archive directory... 2024-03-11T21:29:52.419 DEBUG:teuthology.orchestra.run.smithi023:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-11T21:29:52.421 DEBUG:teuthology.orchestra.run.smithi146:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-11T21:29:52.457 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-11T21:29:52.468 INFO:teuthology.task.internal:Not uploading archives. 2024-03-11T21:29:52.469 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-11T21:29:52.478 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-11T21:29:52.478 DEBUG:teuthology.orchestra.run.smithi023:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-11T21:29:52.481 DEBUG:teuthology.orchestra.run.smithi146:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-11T21:29:52.485 INFO:teuthology.orchestra.run.smithi023.stdout: 658624 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 11 21:29 /home/ubuntu/cephtest 2024-03-11T21:29:52.505 INFO:teuthology.orchestra.run.smithi146.stdout: 658640 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 11 21:29 /home/ubuntu/cephtest 2024-03-11T21:29:52.506 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-11T21:29:52.518 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-11T21:29:52.539 INFO:teuthology.nuke:Checking targets against current locks 2024-03-11T21:29:52.562 DEBUG:teuthology.nuke:shortname: smithi023 2024-03-11T21:29:52.562 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-11T21:29:52.587 DEBUG:teuthology.nuke:shortname: smithi146 2024-03-11T21:29:52.588 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-11T21:29:52.593 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi023.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591745', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-11 20:19:40.036957', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJktKkTPugdekffXcRoSm8ht+LzmjuwT01HtyylK2Lkt4rGHd5AhDfOFO5u8Qxy8J+yrw66+sXAXaphPkzYzA2o='} 2024-03-11T21:29:52.607 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi146.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591745', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-11 20:19:40.038821', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLU0075hNvzK2/URwJNjADYTv8KNoaw+s3EBLHJ9uB5xFJFgRPGgtjwlpMbn7H+EVSEXyoMcqEK+wNcEL+QGLXo='} 2024-03-11T21:29:52.649 INFO:teuthology.orchestra.console.smithi023:Power off 2024-03-11T21:29:52.649 DEBUG:teuthology.orchestra.console.smithi023:pexpect command: ipmitool -H smithi023.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-11T21:29:52.668 INFO:teuthology.orchestra.console.smithi146:Power off 2024-03-11T21:29:52.668 DEBUG:teuthology.orchestra.console.smithi146:pexpect command: ipmitool -H smithi146.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-11T21:29:52.677 DEBUG:teuthology.orchestra.console.smithi023:power off output: Chassis Power Control: Down/Off 2024-03-11T21:29:52.678 DEBUG:teuthology.orchestra.console.smithi023:pexpect command: ipmitool -H smithi023.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T21:29:52.694 DEBUG:teuthology.orchestra.console.smithi146:power off output: Chassis Power Control: Down/Off 2024-03-11T21:29:52.694 DEBUG:teuthology.orchestra.console.smithi146:pexpect command: ipmitool -H smithi146.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T21:29:52.708 DEBUG:teuthology.orchestra.console.smithi023:check power output: Chassis Power is on 2024-03-11T21:29:52.722 DEBUG:teuthology.orchestra.console.smithi146:check power output: Chassis Power is on 2024-03-11T21:29:56.709 DEBUG:teuthology.orchestra.console.smithi023:pexpect command: ipmitool -H smithi023.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T21:29:56.724 DEBUG:teuthology.orchestra.console.smithi146:pexpect command: ipmitool -H smithi146.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T21:29:56.819 DEBUG:teuthology.orchestra.console.smithi023:check power output: Chassis Power is on 2024-03-11T21:29:56.833 DEBUG:teuthology.orchestra.console.smithi146:check power output: Chassis Power is on 2024-03-11T21:30:00.822 DEBUG:teuthology.orchestra.console.smithi023:pexpect command: ipmitool -H smithi023.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T21:30:00.835 DEBUG:teuthology.orchestra.console.smithi146:pexpect command: ipmitool -H smithi146.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T21:30:00.933 DEBUG:teuthology.orchestra.console.smithi023:check power output: Chassis Power is on 2024-03-11T21:30:00.944 DEBUG:teuthology.orchestra.console.smithi146:check power output: Chassis Power is off 2024-03-11T21:30:01.044 INFO:teuthology.orchestra.console.smithi146:Power off completed 2024-03-11T21:30:04.934 DEBUG:teuthology.orchestra.console.smithi023:pexpect command: ipmitool -H smithi023.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T21:30:05.047 DEBUG:teuthology.orchestra.console.smithi023:check power output: Chassis Power is off 2024-03-11T21:30:05.147 INFO:teuthology.orchestra.console.smithi023:Power off completed 2024-03-11T21:30:05.251 INFO:teuthology.run:Summary data: description: orch/cephadm/thrash/{0-distro/ubuntu_22.04 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async root} duration: 3592.1014976501465 failure_reason: '"2024-03-11T21:00:00.000279+0000 mon.a (mon.0) 1176 : cluster 3 [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set" in cluster log' flavor: default owner: scheduled_teuthology@teuthology success: false 2024-03-11T21:30:05.252 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-11T21:30:05.338 INFO:teuthology.run:FAIL